2023-10-05T19:16:00.934 INFO:root:teuthology version: 0.0.1.dev194+g54e62bc 2023-10-05T19:16:00.934 DEBUG:teuthology.run:Teuthology command: teuthology --description rgw/singleton/{all/radosgw-admin frontend/beast ignore-pg-availability objectstore/bluestore-bitmap overrides rgw_pool_type/ec-profile supported-random-distro$/{ubuntu_latest}} --name yuriw-2023-10-05_14:30:58-rgw-wip-yuri2-testing-2023-10-04-1333-quincy-distro-default-smithi --archive /home/teuthworker/archive/yuriw-2023-10-05_14:30:58-rgw-wip-yuri2-testing-2023-10-04-1333-quincy-distro-default-smithi/7411744 --verbose --owner scheduled_yuriw@teuthology -- /home/teuthworker/archive/yuriw-2023-10-05_14:30:58-rgw-wip-yuri2-testing-2023-10-04-1333-quincy-distro-default-smithi/7411744/orig.config.yaml 2023-10-05T19:16:00.986 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2023-10-05T19:16:01.069 INFO:teuthology.run:Config: archive_path: /home/teuthworker/archive/yuriw-2023-10-05_14:30:58-rgw-wip-yuri2-testing-2023-10-04-1333-quincy-distro-default-smithi/7411744 branch: wip-yuri2-testing-2023-10-04-1333-quincy description: rgw/singleton/{all/radosgw-admin frontend/beast ignore-pg-availability objectstore/bluestore-bitmap overrides rgw_pool_type/ec-profile supported-random-distro$/{ubuntu_latest}} email: yweinste@redhat.com first_in_suite: false job_id: '7411744' kernel: kdb: true sha1: distro ktype: distro last_in_suite: false machine_type: smithi name: yuriw-2023-10-05_14:30:58-rgw-wip-yuri2-testing-2023-10-04-1333-quincy-distro-default-smithi no_nested_subset: false nuke-on-error: true openstack: - volumes: count: 3 size: 10 os_type: ubuntu os_version: '20.04' overrides: admin_socket: branch: wip-yuri2-testing-2023-10-04-1333-quincy ceph: conf: client: debug rgw: 20 setgroup: ceph setuser: ceph mgr: debug mgr: 20 debug ms: 1 mon: debug mon: 20 debug ms: 1 debug paxos: 20 osd: bdev async discard: true bdev enable discard: true bluestore allocator: bitmap bluestore block size: 96636764160 bluestore fsck on mount: true debug bluefs: 1/20 debug bluestore: 1/20 debug ms: 1 debug osd: 20 debug rocksdb: 4/10 mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd failsafe full ratio: 0.95 osd objectstore: bluestore flavor: default fs: xfs log-ignorelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) - \(PG_AVAILABILITY\) - \(PG_DEGRADED\) sha1: 487ffcb5ddb60d29c5688d1a457e812b1b697b90 wait-for-scrub: false ceph-deploy: bluestore: true conf: client: log file: /var/log/ceph/ceph-$name.$pid.log mon: {} osd: bdev async discard: true bdev enable discard: true bluestore block size: 96636764160 bluestore fsck on mount: true debug bluefs: 1/20 debug bluestore: 1/20 debug rocksdb: 4/10 mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd failsafe full ratio: 0.95 osd objectstore: bluestore fs: xfs install: ceph: flavor: default sha1: 487ffcb5ddb60d29c5688d1a457e812b1b697b90 rgw: ec-data-pool: true erasure_code_profile: crush-failure-domain: osd k: 3 m: 1 name: testprofile frontend: beast s3tests: slow_backend: true thrashosds: bdev_inject_crash: 2 bdev_inject_crash_probability: 0.5 workunit: branch: wip-yuri2-testing-2023-10-04-1333-quincy sha1: 487ffcb5ddb60d29c5688d1a457e812b1b697b90 owner: scheduled_yuriw@teuthology priority: 75 repo: https://github.com/ceph/ceph-ci.git roles: - - mon.a - osd.0 - - mgr.x - client.0 - osd.1 - osd.2 - osd.3 seed: 880 sha1: 487ffcb5ddb60d29c5688d1a457e812b1b697b90 sleep_before_teardown: 0 subset: 111/12000 suite: rgw suite_branch: wip-yuri2-testing-2023-10-04-1333-quincy suite_path: /home/teuthworker/src/github.com_ceph_ceph-c_487ffcb5ddb60d29c5688d1a457e812b1b697b90/qa suite_relpath: qa suite_repo: https://github.com/ceph/ceph-ci.git suite_sha1: 487ffcb5ddb60d29c5688d1a457e812b1b697b90 targets: smithi118.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBFyT/r7xuKfmg50w0OdQebr4A/ptJo1Zk6DyZNZNZM/q5rW+sznh+wvHqeT3e4YtCdZLaXyvMms8o8u8H5nXOYM= smithi167.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBFyT/r7xuKfmg50w0OdQebr4A/ptJo1Zk6DyZNZNZM/q5rW+sznh+wvHqeT3e4YtCdZLaXyvMms8o8u8H5nXOYM= tasks: - install: null - ceph: conf: client: debug ms: 1 rgw gc obj min wait: 15 osd: debug ms: 1 debug objclass: 20 - rgw: client.0: null - radosgw-admin: null - radosgw-admin-rest: null teuthology: fragments_dropped: [] meta: {} postmerge: [] teuthology_branch: main teuthology_sha1: 54e62bcbac4e53d9685e08328b790d3b20d71cae timestamp: 2023-10-05_14:30:58 tube: smithi user: yuriw verbose: true worker_log: /home/teuthworker/archive/worker_logs/dispatcher.smithi.2830 2023-10-05T19:16:01.070 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/github.com_ceph_ceph-c_487ffcb5ddb60d29c5688d1a457e812b1b697b90/qa; will attempt to use it 2023-10-05T19:16:01.071 INFO:teuthology.run:Found tasks at /home/teuthworker/src/github.com_ceph_ceph-c_487ffcb5ddb60d29c5688d1a457e812b1b697b90/qa/tasks 2023-10-05T19:16:01.071 INFO:teuthology.run_tasks:Running task internal.check_packages... 2023-10-05T19:16:01.072 INFO:teuthology.task.internal:Checking packages... 2023-10-05T19:16:01.097 INFO:teuthology.task.internal:Checking packages for os_type 'ubuntu', flavor 'default' and ceph hash '487ffcb5ddb60d29c5688d1a457e812b1b697b90' 2023-10-05T19:16:01.097 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2023-10-05T19:16:01.097 INFO:teuthology.packaging:ref: None 2023-10-05T19:16:01.097 INFO:teuthology.packaging:tag: None 2023-10-05T19:16:01.097 INFO:teuthology.packaging:branch: wip-yuri2-testing-2023-10-04-1333-quincy 2023-10-05T19:16:01.097 INFO:teuthology.packaging:sha1: 487ffcb5ddb60d29c5688d1a457e812b1b697b90 2023-10-05T19:16:01.097 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=ubuntu%2F20.04%2Fx86_64&ref=wip-yuri2-testing-2023-10-04-1333-quincy 2023-10-05T19:16:01.242 INFO:teuthology.task.internal:Found packages for ceph version 17.2.6-1278-g487ffcb5-1focal 2023-10-05T19:16:01.243 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep... 2023-10-05T19:16:01.260 INFO:teuthology.task.internal:no buildpackages task found 2023-10-05T19:16:01.261 INFO:teuthology.run_tasks:Running task internal.save_config... 2023-10-05T19:16:01.296 INFO:teuthology.task.internal:Saving configuration 2023-10-05T19:16:01.308 INFO:teuthology.run_tasks:Running task internal.check_lock... 2023-10-05T19:16:01.325 INFO:teuthology.task.internal.check_lock:Checking locks... 2023-10-05T19:16:01.349 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi118.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/yuriw-2023-10-05_14:30:58-rgw-wip-yuri2-testing-2023-10-04-1333-quincy-distro-default-smithi/7411744', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'ubuntu', 'os_version': '20.04', 'arch': 'x86_64', 'locked': True, 'locked_since': '2023-10-05 19:09:12.712230', 'locked_by': 'scheduled_yuriw@teuthology', 'mac_address': None, 'ssh_pub_key': 'ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIGETOy3xONfbx7kUFCNiUtRLNfkcrcEsWzPdq6Np/6Bw'} 2023-10-05T19:16:01.373 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi167.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/yuriw-2023-10-05_14:30:58-rgw-wip-yuri2-testing-2023-10-04-1333-quincy-distro-default-smithi/7411744', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'ubuntu', 'os_version': '20.04', 'arch': 'x86_64', 'locked': True, 'locked_since': '2023-10-05 19:09:12.710580', 'locked_by': 'scheduled_yuriw@teuthology', 'mac_address': None, 'ssh_pub_key': 'ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIGETOy3xONfbx7kUFCNiUtRLNfkcrcEsWzPdq6Np/6Bw'} 2023-10-05T19:16:01.373 INFO:teuthology.run_tasks:Running task internal.add_remotes... 2023-10-05T19:16:01.382 INFO:teuthology.task.internal:roles: ubuntu@smithi118.front.sepia.ceph.com - ['mon.a', 'osd.0'] 2023-10-05T19:16:01.382 INFO:teuthology.task.internal:roles: ubuntu@smithi167.front.sepia.ceph.com - ['mgr.x', 'client.0', 'osd.1', 'osd.2', 'osd.3'] 2023-10-05T19:16:01.383 INFO:teuthology.run_tasks:Running task console_log... 2023-10-05T19:16:01.480 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7fe0e5582040>, signals=[15]) 2023-10-05T19:16:01.481 INFO:teuthology.run_tasks:Running task internal.connect... 2023-10-05T19:16:01.489 INFO:teuthology.task.internal:Opening connections... 2023-10-05T19:16:01.489 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi118.front.sepia.ceph.com 2023-10-05T19:16:01.490 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi118.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-10-05T19:16:01.538 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi167.front.sepia.ceph.com 2023-10-05T19:16:01.539 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi167.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-10-05T19:16:01.586 INFO:teuthology.run_tasks:Running task internal.push_inventory... 2023-10-05T19:16:01.595 DEBUG:teuthology.orchestra.run.smithi118:> uname -m 2023-10-05T19:16:01.615 INFO:teuthology.orchestra.run.smithi118.stdout:x86_64 2023-10-05T19:16:01.616 DEBUG:teuthology.orchestra.run.smithi118:> cat /etc/os-release 2023-10-05T19:16:01.663 INFO:teuthology.orchestra.run.smithi118.stdout:NAME="Ubuntu" 2023-10-05T19:16:01.663 INFO:teuthology.orchestra.run.smithi118.stdout:VERSION="20.04.4 LTS (Focal Fossa)" 2023-10-05T19:16:01.663 INFO:teuthology.orchestra.run.smithi118.stdout:ID=ubuntu 2023-10-05T19:16:01.663 INFO:teuthology.orchestra.run.smithi118.stdout:ID_LIKE=debian 2023-10-05T19:16:01.663 INFO:teuthology.orchestra.run.smithi118.stdout:PRETTY_NAME="Ubuntu 20.04.4 LTS" 2023-10-05T19:16:01.663 INFO:teuthology.orchestra.run.smithi118.stdout:VERSION_ID="20.04" 2023-10-05T19:16:01.663 INFO:teuthology.orchestra.run.smithi118.stdout:HOME_URL="https://www.ubuntu.com/" 2023-10-05T19:16:01.663 INFO:teuthology.orchestra.run.smithi118.stdout:SUPPORT_URL="https://help.ubuntu.com/" 2023-10-05T19:16:01.663 INFO:teuthology.orchestra.run.smithi118.stdout:BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/" 2023-10-05T19:16:01.663 INFO:teuthology.orchestra.run.smithi118.stdout:PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy" 2023-10-05T19:16:01.664 INFO:teuthology.orchestra.run.smithi118.stdout:VERSION_CODENAME=focal 2023-10-05T19:16:01.664 INFO:teuthology.orchestra.run.smithi118.stdout:UBUNTU_CODENAME=focal 2023-10-05T19:16:01.664 INFO:teuthology.lock.ops:Updating smithi118.front.sepia.ceph.com on lock server 2023-10-05T19:16:01.688 DEBUG:teuthology.orchestra.run.smithi167:> uname -m 2023-10-05T19:16:01.695 INFO:teuthology.orchestra.run.smithi167.stdout:x86_64 2023-10-05T19:16:01.695 DEBUG:teuthology.orchestra.run.smithi167:> cat /etc/os-release 2023-10-05T19:16:01.744 INFO:teuthology.orchestra.run.smithi167.stdout:NAME="Ubuntu" 2023-10-05T19:16:01.744 INFO:teuthology.orchestra.run.smithi167.stdout:VERSION="20.04.4 LTS (Focal Fossa)" 2023-10-05T19:16:01.744 INFO:teuthology.orchestra.run.smithi167.stdout:ID=ubuntu 2023-10-05T19:16:01.744 INFO:teuthology.orchestra.run.smithi167.stdout:ID_LIKE=debian 2023-10-05T19:16:01.744 INFO:teuthology.orchestra.run.smithi167.stdout:PRETTY_NAME="Ubuntu 20.04.4 LTS" 2023-10-05T19:16:01.744 INFO:teuthology.orchestra.run.smithi167.stdout:VERSION_ID="20.04" 2023-10-05T19:16:01.744 INFO:teuthology.orchestra.run.smithi167.stdout:HOME_URL="https://www.ubuntu.com/" 2023-10-05T19:16:01.745 INFO:teuthology.orchestra.run.smithi167.stdout:SUPPORT_URL="https://help.ubuntu.com/" 2023-10-05T19:16:01.745 INFO:teuthology.orchestra.run.smithi167.stdout:BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/" 2023-10-05T19:16:01.745 INFO:teuthology.orchestra.run.smithi167.stdout:PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy" 2023-10-05T19:16:01.745 INFO:teuthology.orchestra.run.smithi167.stdout:VERSION_CODENAME=focal 2023-10-05T19:16:01.745 INFO:teuthology.orchestra.run.smithi167.stdout:UBUNTU_CODENAME=focal 2023-10-05T19:16:01.745 INFO:teuthology.lock.ops:Updating smithi167.front.sepia.ceph.com on lock server 2023-10-05T19:16:01.763 INFO:teuthology.run_tasks:Running task internal.serialize_remote_roles... 2023-10-05T19:16:01.771 INFO:teuthology.run_tasks:Running task internal.check_conflict... 2023-10-05T19:16:01.789 INFO:teuthology.task.internal:Checking for old test directory... 2023-10-05T19:16:01.789 DEBUG:teuthology.orchestra.run.smithi118:> test '!' -e /home/ubuntu/cephtest 2023-10-05T19:16:01.791 DEBUG:teuthology.orchestra.run.smithi167:> test '!' -e /home/ubuntu/cephtest 2023-10-05T19:16:01.797 INFO:teuthology.run_tasks:Running task internal.check_ceph_data... 2023-10-05T19:16:01.814 INFO:teuthology.task.internal:Checking for non-empty /var/lib/ceph... 2023-10-05T19:16:01.814 DEBUG:teuthology.orchestra.run.smithi118:> test -z $(ls -A /var/lib/ceph) 2023-10-05T19:16:01.816 DEBUG:teuthology.orchestra.run.smithi167:> test -z $(ls -A /var/lib/ceph) 2023-10-05T19:16:01.852 INFO:teuthology.run_tasks:Running task internal.vm_setup... 2023-10-05T19:16:01.908 INFO:teuthology.run_tasks:Running task kernel... 2023-10-05T19:16:01.924 INFO:teuthology.task.kernel:normalize config orig: {'kdb': True, 'sha1': 'distro'} 2023-10-05T19:16:01.924 INFO:teuthology.task.kernel:config {'mon.a': {'kdb': True, 'sha1': 'distro'}, 'mgr.x': {'kdb': True, 'sha1': 'distro'}}, timeout 300 2023-10-05T19:16:01.925 DEBUG:teuthology.orchestra.run.smithi118:> test -f /run/.containerenv -o -f /.dockerenv 2023-10-05T19:16:01.925 DEBUG:teuthology.orchestra.run.smithi167:> test -f /run/.containerenv -o -f /.dockerenv 2023-10-05T19:16:01.930 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-10-05T19:16:01.930 DEBUG:teuthology.orchestra.run.smithi118:> uname -r 2023-10-05T19:16:01.931 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-10-05T19:16:01.931 DEBUG:teuthology.orchestra.run.smithi167:> uname -r 2023-10-05T19:16:01.937 INFO:teuthology.orchestra.run.smithi118.stdout:5.4.0-124-generic 2023-10-05T19:16:01.937 INFO:teuthology.task.kernel:Running kernel on smithi118: 5.4.0-124-generic 2023-10-05T19:16:01.938 DEBUG:teuthology.orchestra.run.smithi118:> sudo apt-get clean 2023-10-05T19:16:01.939 INFO:teuthology.orchestra.run.smithi167.stdout:5.4.0-124-generic 2023-10-05T19:16:01.939 INFO:teuthology.task.kernel:Running kernel on smithi167: 5.4.0-124-generic 2023-10-05T19:16:01.939 DEBUG:teuthology.orchestra.run.smithi167:> sudo apt-get clean 2023-10-05T19:16:02.084 DEBUG:teuthology.orchestra.run.smithi167:> sudo apt-get update 2023-10-05T19:16:02.173 DEBUG:teuthology.orchestra.run.smithi118:> sudo apt-get update 2023-10-05T19:16:02.256 INFO:teuthology.orchestra.run.smithi167.stdout:Get:1 http://security.ubuntu.com/ubuntu focal-security InRelease [114 kB] 2023-10-05T19:16:02.327 INFO:teuthology.orchestra.run.smithi118.stdout:Get:1 http://security.ubuntu.com/ubuntu focal-security InRelease [114 kB] 2023-10-05T19:16:02.438 INFO:teuthology.orchestra.run.smithi118.stdout:Hit:2 http://archive.ubuntu.com/ubuntu focal InRelease 2023-10-05T19:16:02.520 INFO:teuthology.orchestra.run.smithi118.stdout:Get:3 http://archive.ubuntu.com/ubuntu focal-updates InRelease [114 kB] 2023-10-05T19:16:02.575 INFO:teuthology.orchestra.run.smithi167.stdout:Get:2 http://security.ubuntu.com/ubuntu focal-security/main amd64 Packages [2,495 kB] 2023-10-05T19:16:02.614 INFO:teuthology.orchestra.run.smithi118.stdout:Get:4 http://security.ubuntu.com/ubuntu focal-security/main i386 Packages [659 kB] 2023-10-05T19:16:02.718 INFO:teuthology.orchestra.run.smithi167.stdout:Get:3 http://security.ubuntu.com/ubuntu focal-security/main i386 Packages [659 kB] 2023-10-05T19:16:02.718 INFO:teuthology.orchestra.run.smithi118.stdout:Get:5 http://security.ubuntu.com/ubuntu focal-security/main amd64 Packages [2,495 kB] 2023-10-05T19:16:02.729 INFO:teuthology.orchestra.run.smithi167.stdout:Get:4 http://security.ubuntu.com/ubuntu focal-security/main Translation-en [389 kB] 2023-10-05T19:16:02.732 INFO:teuthology.orchestra.run.smithi167.stdout:Get:5 http://security.ubuntu.com/ubuntu focal-security/main amd64 c-n-f Metadata [13.2 kB] 2023-10-05T19:16:02.733 INFO:teuthology.orchestra.run.smithi167.stdout:Get:6 http://security.ubuntu.com/ubuntu focal-security/restricted i386 Packages [33.3 kB] 2023-10-05T19:16:02.733 INFO:teuthology.orchestra.run.smithi167.stdout:Get:7 http://security.ubuntu.com/ubuntu focal-security/restricted amd64 Packages [2,257 kB] 2023-10-05T19:16:02.748 INFO:teuthology.orchestra.run.smithi167.stdout:Get:8 http://security.ubuntu.com/ubuntu focal-security/restricted Translation-en [316 kB] 2023-10-05T19:16:02.750 INFO:teuthology.orchestra.run.smithi167.stdout:Get:9 http://security.ubuntu.com/ubuntu focal-security/restricted amd64 c-n-f Metadata [572 B] 2023-10-05T19:16:02.750 INFO:teuthology.orchestra.run.smithi167.stdout:Get:10 http://security.ubuntu.com/ubuntu focal-security/universe i386 Packages [616 kB] 2023-10-05T19:16:02.754 INFO:teuthology.orchestra.run.smithi167.stdout:Get:11 http://security.ubuntu.com/ubuntu focal-security/universe amd64 Packages [890 kB] 2023-10-05T19:16:02.758 INFO:teuthology.orchestra.run.smithi118.stdout:Get:6 http://security.ubuntu.com/ubuntu focal-security/main Translation-en [389 kB] 2023-10-05T19:16:02.760 INFO:teuthology.orchestra.run.smithi167.stdout:Get:12 http://security.ubuntu.com/ubuntu focal-security/universe Translation-en [186 kB] 2023-10-05T19:16:02.761 INFO:teuthology.orchestra.run.smithi167.stdout:Get:13 http://security.ubuntu.com/ubuntu focal-security/universe amd64 c-n-f Metadata [19.1 kB] 2023-10-05T19:16:02.762 INFO:teuthology.orchestra.run.smithi118.stdout:Get:7 http://security.ubuntu.com/ubuntu focal-security/main amd64 c-n-f Metadata [13.2 kB] 2023-10-05T19:16:02.762 INFO:teuthology.orchestra.run.smithi118.stdout:Get:8 http://security.ubuntu.com/ubuntu focal-security/restricted amd64 Packages [2,257 kB] 2023-10-05T19:16:02.764 INFO:teuthology.orchestra.run.smithi167.stdout:Get:14 http://security.ubuntu.com/ubuntu focal-security/multiverse amd64 Packages [23.6 kB] 2023-10-05T19:16:02.764 INFO:teuthology.orchestra.run.smithi167.stdout:Get:15 http://security.ubuntu.com/ubuntu focal-security/multiverse i386 Packages [7,196 B] 2023-10-05T19:16:02.765 INFO:teuthology.orchestra.run.smithi167.stdout:Get:16 http://security.ubuntu.com/ubuntu focal-security/multiverse Translation-en [5,504 B] 2023-10-05T19:16:02.777 INFO:teuthology.orchestra.run.smithi118.stdout:Get:9 http://security.ubuntu.com/ubuntu focal-security/restricted i386 Packages [33.3 kB] 2023-10-05T19:16:02.777 INFO:teuthology.orchestra.run.smithi118.stdout:Get:10 http://security.ubuntu.com/ubuntu focal-security/restricted Translation-en [316 kB] 2023-10-05T19:16:02.780 INFO:teuthology.orchestra.run.smithi118.stdout:Get:11 http://security.ubuntu.com/ubuntu focal-security/restricted amd64 c-n-f Metadata [572 B] 2023-10-05T19:16:02.780 INFO:teuthology.orchestra.run.smithi118.stdout:Get:12 http://security.ubuntu.com/ubuntu focal-security/universe i386 Packages [616 kB] 2023-10-05T19:16:02.780 INFO:teuthology.orchestra.run.smithi167.stdout:Get:17 http://security.ubuntu.com/ubuntu focal-security/multiverse amd64 c-n-f Metadata [548 B] 2023-10-05T19:16:02.786 INFO:teuthology.orchestra.run.smithi118.stdout:Get:13 http://security.ubuntu.com/ubuntu focal-security/universe amd64 Packages [890 kB] 2023-10-05T19:16:02.790 INFO:teuthology.orchestra.run.smithi118.stdout:Get:14 http://security.ubuntu.com/ubuntu focal-security/universe Translation-en [186 kB] 2023-10-05T19:16:02.791 INFO:teuthology.orchestra.run.smithi118.stdout:Get:15 http://security.ubuntu.com/ubuntu focal-security/universe amd64 c-n-f Metadata [19.1 kB] 2023-10-05T19:16:02.794 INFO:teuthology.orchestra.run.smithi118.stdout:Get:16 http://security.ubuntu.com/ubuntu focal-security/multiverse amd64 Packages [23.6 kB] 2023-10-05T19:16:02.794 INFO:teuthology.orchestra.run.smithi118.stdout:Get:17 http://security.ubuntu.com/ubuntu focal-security/multiverse i386 Packages [7,196 B] 2023-10-05T19:16:02.811 INFO:teuthology.orchestra.run.smithi118.stdout:Get:18 http://security.ubuntu.com/ubuntu focal-security/multiverse Translation-en [5,504 B] 2023-10-05T19:16:02.811 INFO:teuthology.orchestra.run.smithi118.stdout:Get:19 http://security.ubuntu.com/ubuntu focal-security/multiverse amd64 c-n-f Metadata [548 B] 2023-10-05T19:16:02.858 INFO:teuthology.orchestra.run.smithi167.stdout:Hit:18 http://archive.ubuntu.com/ubuntu focal InRelease 2023-10-05T19:16:02.863 INFO:teuthology.orchestra.run.smithi118.stdout:Get:20 http://archive.ubuntu.com/ubuntu focal-backports InRelease [108 kB] 2023-10-05T19:16:02.939 INFO:teuthology.orchestra.run.smithi167.stdout:Get:19 http://archive.ubuntu.com/ubuntu focal-updates InRelease [114 kB] 2023-10-05T19:16:03.005 INFO:teuthology.orchestra.run.smithi118.stdout:Get:21 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 Packages [2,889 kB] 2023-10-05T19:16:03.284 INFO:teuthology.orchestra.run.smithi167.stdout:Get:20 http://archive.ubuntu.com/ubuntu focal-backports InRelease [108 kB] 2023-10-05T19:16:03.385 INFO:teuthology.orchestra.run.smithi118.stdout:Get:22 http://archive.ubuntu.com/ubuntu focal-updates/main i386 Packages [893 kB] 2023-10-05T19:16:03.414 INFO:teuthology.orchestra.run.smithi118.stdout:Get:23 http://archive.ubuntu.com/ubuntu focal-updates/main Translation-en [473 kB] 2023-10-05T19:16:03.425 INFO:teuthology.orchestra.run.smithi167.stdout:Get:21 http://archive.ubuntu.com/ubuntu focal-updates/main i386 Packages [893 kB] 2023-10-05T19:16:03.427 INFO:teuthology.orchestra.run.smithi118.stdout:Get:24 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 c-n-f Metadata [17.1 kB] 2023-10-05T19:16:03.428 INFO:teuthology.orchestra.run.smithi118.stdout:Get:25 http://archive.ubuntu.com/ubuntu focal-updates/restricted amd64 Packages [2,383 kB] 2023-10-05T19:16:03.489 INFO:teuthology.orchestra.run.smithi118.stdout:Get:26 http://archive.ubuntu.com/ubuntu focal-updates/restricted i386 Packages [34.6 kB] 2023-10-05T19:16:03.490 INFO:teuthology.orchestra.run.smithi118.stdout:Get:27 http://archive.ubuntu.com/ubuntu focal-updates/restricted Translation-en [334 kB] 2023-10-05T19:16:03.509 INFO:teuthology.orchestra.run.smithi118.stdout:Get:28 http://archive.ubuntu.com/ubuntu focal-updates/restricted amd64 c-n-f Metadata [572 B] 2023-10-05T19:16:03.674 INFO:teuthology.orchestra.run.smithi118.stdout:Get:29 http://archive.ubuntu.com/ubuntu focal-updates/universe i386 Packages [749 kB] 2023-10-05T19:16:03.674 INFO:teuthology.orchestra.run.smithi167.stdout:Get:22 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 Packages [2,889 kB] 2023-10-05T19:16:03.836 INFO:teuthology.orchestra.run.smithi167.stdout:Get:23 http://archive.ubuntu.com/ubuntu focal-updates/main Translation-en [473 kB] 2023-10-05T19:16:03.848 INFO:teuthology.orchestra.run.smithi167.stdout:Get:24 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 c-n-f Metadata [17.1 kB] 2023-10-05T19:16:03.849 INFO:teuthology.orchestra.run.smithi167.stdout:Get:25 http://archive.ubuntu.com/ubuntu focal-updates/restricted i386 Packages [34.6 kB] 2023-10-05T19:16:03.849 INFO:teuthology.orchestra.run.smithi167.stdout:Get:26 http://archive.ubuntu.com/ubuntu focal-updates/restricted amd64 Packages [2,383 kB] 2023-10-05T19:16:03.912 INFO:teuthology.orchestra.run.smithi167.stdout:Get:27 http://archive.ubuntu.com/ubuntu focal-updates/restricted Translation-en [334 kB] 2023-10-05T19:16:03.930 INFO:teuthology.orchestra.run.smithi167.stdout:Get:28 http://archive.ubuntu.com/ubuntu focal-updates/restricted amd64 c-n-f Metadata [572 B] 2023-10-05T19:16:04.094 INFO:teuthology.orchestra.run.smithi167.stdout:Get:29 http://archive.ubuntu.com/ubuntu focal-updates/universe i386 Packages [749 kB] 2023-10-05T19:16:04.260 INFO:teuthology.orchestra.run.smithi118.stdout:Get:30 http://archive.ubuntu.com/ubuntu focal-updates/universe amd64 Packages [1,121 kB] 2023-10-05T19:16:04.369 INFO:teuthology.orchestra.run.smithi118.stdout:Get:31 http://archive.ubuntu.com/ubuntu focal-updates/universe Translation-en [268 kB] 2023-10-05T19:16:04.390 INFO:teuthology.orchestra.run.smithi118.stdout:Get:32 http://archive.ubuntu.com/ubuntu focal-updates/universe amd64 c-n-f Metadata [25.6 kB] 2023-10-05T19:16:04.392 INFO:teuthology.orchestra.run.smithi118.stdout:Get:33 http://archive.ubuntu.com/ubuntu focal-updates/multiverse amd64 Packages [25.8 kB] 2023-10-05T19:16:04.393 INFO:teuthology.orchestra.run.smithi118.stdout:Get:34 http://archive.ubuntu.com/ubuntu focal-updates/multiverse i386 Packages [8,436 B] 2023-10-05T19:16:04.394 INFO:teuthology.orchestra.run.smithi118.stdout:Get:35 http://archive.ubuntu.com/ubuntu focal-updates/multiverse Translation-en [7,484 B] 2023-10-05T19:16:04.394 INFO:teuthology.orchestra.run.smithi118.stdout:Get:36 http://archive.ubuntu.com/ubuntu focal-updates/multiverse amd64 c-n-f Metadata [620 B] 2023-10-05T19:16:04.395 INFO:teuthology.orchestra.run.smithi118.stdout:Get:37 http://archive.ubuntu.com/ubuntu focal-backports/main amd64 Packages [45.7 kB] 2023-10-05T19:16:04.398 INFO:teuthology.orchestra.run.smithi118.stdout:Get:38 http://archive.ubuntu.com/ubuntu focal-backports/main i386 Packages [36.1 kB] 2023-10-05T19:16:04.400 INFO:teuthology.orchestra.run.smithi118.stdout:Get:39 http://archive.ubuntu.com/ubuntu focal-backports/main amd64 c-n-f Metadata [1,420 B] 2023-10-05T19:16:04.564 INFO:teuthology.orchestra.run.smithi118.stdout:Get:40 http://archive.ubuntu.com/ubuntu focal-backports/universe amd64 Packages [25.0 kB] 2023-10-05T19:16:04.679 INFO:teuthology.orchestra.run.smithi167.stdout:Get:30 http://archive.ubuntu.com/ubuntu focal-updates/universe amd64 Packages [1,121 kB] 2023-10-05T19:16:04.748 INFO:teuthology.orchestra.run.smithi118.stdout:Get:41 http://archive.ubuntu.com/ubuntu focal-backports/universe i386 Packages [13.8 kB] 2023-10-05T19:16:04.770 INFO:teuthology.orchestra.run.smithi118.stdout:Get:42 http://archive.ubuntu.com/ubuntu focal-backports/universe amd64 c-n-f Metadata [880 B] 2023-10-05T19:16:04.789 INFO:teuthology.orchestra.run.smithi167.stdout:Get:31 http://archive.ubuntu.com/ubuntu focal-updates/universe Translation-en [268 kB] 2023-10-05T19:16:04.809 INFO:teuthology.orchestra.run.smithi167.stdout:Get:32 http://archive.ubuntu.com/ubuntu focal-updates/universe amd64 c-n-f Metadata [25.6 kB] 2023-10-05T19:16:04.811 INFO:teuthology.orchestra.run.smithi167.stdout:Get:33 http://archive.ubuntu.com/ubuntu focal-updates/multiverse i386 Packages [8,436 B] 2023-10-05T19:16:04.812 INFO:teuthology.orchestra.run.smithi167.stdout:Get:34 http://archive.ubuntu.com/ubuntu focal-updates/multiverse amd64 Packages [25.8 kB] 2023-10-05T19:16:04.814 INFO:teuthology.orchestra.run.smithi167.stdout:Get:35 http://archive.ubuntu.com/ubuntu focal-updates/multiverse Translation-en [7,484 B] 2023-10-05T19:16:04.815 INFO:teuthology.orchestra.run.smithi167.stdout:Get:36 http://archive.ubuntu.com/ubuntu focal-updates/multiverse amd64 c-n-f Metadata [620 B] 2023-10-05T19:16:04.815 INFO:teuthology.orchestra.run.smithi167.stdout:Get:37 http://archive.ubuntu.com/ubuntu focal-backports/main amd64 Packages [45.7 kB] 2023-10-05T19:16:04.817 INFO:teuthology.orchestra.run.smithi167.stdout:Get:38 http://archive.ubuntu.com/ubuntu focal-backports/main i386 Packages [36.1 kB] 2023-10-05T19:16:04.820 INFO:teuthology.orchestra.run.smithi167.stdout:Get:39 http://archive.ubuntu.com/ubuntu focal-backports/main amd64 c-n-f Metadata [1,420 B] 2023-10-05T19:16:04.984 INFO:teuthology.orchestra.run.smithi167.stdout:Get:40 http://archive.ubuntu.com/ubuntu focal-backports/universe i386 Packages [13.8 kB] 2023-10-05T19:16:05.148 INFO:teuthology.orchestra.run.smithi167.stdout:Get:41 http://archive.ubuntu.com/ubuntu focal-backports/universe amd64 Packages [25.0 kB] 2023-10-05T19:16:05.200 INFO:teuthology.orchestra.run.smithi167.stdout:Get:42 http://archive.ubuntu.com/ubuntu focal-backports/universe amd64 c-n-f Metadata [880 B] 2023-10-05T19:16:06.743 INFO:teuthology.orchestra.run.smithi118.stdout:Fetched 17.6 MB in 3s (6,993 kB/s) 2023-10-05T19:16:07.075 INFO:teuthology.orchestra.run.smithi167.stdout:Fetched 17.6 MB in 3s (5,789 kB/s) 2023-10-05T19:16:08.206 INFO:teuthology.orchestra.run.smithi118.stdout:Reading package lists... 2023-10-05T19:16:08.232 DEBUG:teuthology.orchestra.run.smithi118:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y install linux-image-generic 2023-10-05T19:16:08.300 INFO:teuthology.orchestra.run.smithi118.stdout:Reading package lists... 2023-10-05T19:16:08.500 INFO:teuthology.orchestra.run.smithi118.stdout:Building dependency tree... 2023-10-05T19:16:08.501 INFO:teuthology.orchestra.run.smithi118.stdout:Reading state information... 2023-10-05T19:16:08.573 INFO:teuthology.orchestra.run.smithi167.stdout:Reading package lists... 2023-10-05T19:16:08.599 DEBUG:teuthology.orchestra.run.smithi167:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y install linux-image-generic 2023-10-05T19:16:08.667 INFO:teuthology.orchestra.run.smithi167.stdout:Reading package lists... 2023-10-05T19:16:08.688 INFO:teuthology.orchestra.run.smithi118.stdout:The following additional packages will be installed: 2023-10-05T19:16:08.689 INFO:teuthology.orchestra.run.smithi118.stdout: linux-generic linux-headers-5.4.0-164 linux-headers-5.4.0-164-generic 2023-10-05T19:16:08.689 INFO:teuthology.orchestra.run.smithi118.stdout: linux-headers-generic linux-image-5.4.0-164-generic 2023-10-05T19:16:08.689 INFO:teuthology.orchestra.run.smithi118.stdout: linux-modules-5.4.0-164-generic linux-modules-extra-5.4.0-164-generic 2023-10-05T19:16:08.690 INFO:teuthology.orchestra.run.smithi118.stdout:Suggested packages: 2023-10-05T19:16:08.690 INFO:teuthology.orchestra.run.smithi118.stdout: fdutils linux-doc | linux-source-5.4.0 linux-tools 2023-10-05T19:16:08.690 INFO:teuthology.orchestra.run.smithi118.stdout:Recommended packages: 2023-10-05T19:16:08.690 INFO:teuthology.orchestra.run.smithi118.stdout: thermald 2023-10-05T19:16:08.759 INFO:teuthology.orchestra.run.smithi118.stdout:The following NEW packages will be installed: 2023-10-05T19:16:08.759 INFO:teuthology.orchestra.run.smithi118.stdout: linux-headers-5.4.0-164 linux-headers-5.4.0-164-generic 2023-10-05T19:16:08.759 INFO:teuthology.orchestra.run.smithi118.stdout: linux-image-5.4.0-164-generic linux-modules-5.4.0-164-generic 2023-10-05T19:16:08.759 INFO:teuthology.orchestra.run.smithi118.stdout: linux-modules-extra-5.4.0-164-generic 2023-10-05T19:16:08.760 INFO:teuthology.orchestra.run.smithi118.stdout:The following packages will be upgraded: 2023-10-05T19:16:08.761 INFO:teuthology.orchestra.run.smithi118.stdout: linux-generic linux-headers-generic linux-image-generic 2023-10-05T19:16:08.816 INFO:teuthology.orchestra.run.smithi118.stdout:3 upgraded, 5 newly installed, 0 to remove and 271 not upgraded. 2023-10-05T19:16:08.817 INFO:teuthology.orchestra.run.smithi118.stdout:Need to get 77.0 MB of archives. 2023-10-05T19:16:08.817 INFO:teuthology.orchestra.run.smithi118.stdout:After this operation, 380 MB of additional disk space will be used. 2023-10-05T19:16:08.817 INFO:teuthology.orchestra.run.smithi118.stdout:Get:1 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 linux-modules-5.4.0-164-generic amd64 5.4.0-164.181 [14.9 MB] 2023-10-05T19:16:08.869 INFO:teuthology.orchestra.run.smithi167.stdout:Building dependency tree... 2023-10-05T19:16:08.870 INFO:teuthology.orchestra.run.smithi167.stdout:Reading state information... 2023-10-05T19:16:09.059 INFO:teuthology.orchestra.run.smithi167.stdout:The following additional packages will be installed: 2023-10-05T19:16:09.059 INFO:teuthology.orchestra.run.smithi167.stdout: linux-generic linux-headers-5.4.0-164 linux-headers-5.4.0-164-generic 2023-10-05T19:16:09.059 INFO:teuthology.orchestra.run.smithi167.stdout: linux-headers-generic linux-image-5.4.0-164-generic 2023-10-05T19:16:09.060 INFO:teuthology.orchestra.run.smithi167.stdout: linux-modules-5.4.0-164-generic linux-modules-extra-5.4.0-164-generic 2023-10-05T19:16:09.060 INFO:teuthology.orchestra.run.smithi167.stdout:Suggested packages: 2023-10-05T19:16:09.060 INFO:teuthology.orchestra.run.smithi167.stdout: fdutils linux-doc | linux-source-5.4.0 linux-tools 2023-10-05T19:16:09.061 INFO:teuthology.orchestra.run.smithi167.stdout:Recommended packages: 2023-10-05T19:16:09.061 INFO:teuthology.orchestra.run.smithi167.stdout: thermald 2023-10-05T19:16:09.111 INFO:teuthology.orchestra.run.smithi167.stdout:The following NEW packages will be installed: 2023-10-05T19:16:09.111 INFO:teuthology.orchestra.run.smithi167.stdout: linux-headers-5.4.0-164 linux-headers-5.4.0-164-generic 2023-10-05T19:16:09.111 INFO:teuthology.orchestra.run.smithi167.stdout: linux-image-5.4.0-164-generic linux-modules-5.4.0-164-generic 2023-10-05T19:16:09.111 INFO:teuthology.orchestra.run.smithi167.stdout: linux-modules-extra-5.4.0-164-generic 2023-10-05T19:16:09.112 INFO:teuthology.orchestra.run.smithi167.stdout:The following packages will be upgraded: 2023-10-05T19:16:09.113 INFO:teuthology.orchestra.run.smithi167.stdout: linux-generic linux-headers-generic linux-image-generic 2023-10-05T19:16:09.143 INFO:teuthology.orchestra.run.smithi118.stdout:Get:2 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 linux-image-5.4.0-164-generic amd64 5.4.0-164.181 [10.5 MB] 2023-10-05T19:16:09.235 INFO:teuthology.orchestra.run.smithi118.stdout:Get:3 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 linux-modules-extra-5.4.0-164-generic amd64 5.4.0-164.181 [39.2 MB] 2023-10-05T19:16:09.293 INFO:teuthology.orchestra.run.smithi167.stdout:3 upgraded, 5 newly installed, 0 to remove and 271 not upgraded. 2023-10-05T19:16:09.293 INFO:teuthology.orchestra.run.smithi167.stdout:Need to get 77.0 MB of archives. 2023-10-05T19:16:09.294 INFO:teuthology.orchestra.run.smithi167.stdout:After this operation, 380 MB of additional disk space will be used. 2023-10-05T19:16:09.294 INFO:teuthology.orchestra.run.smithi167.stdout:Get:1 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 linux-modules-5.4.0-164-generic amd64 5.4.0-164.181 [14.9 MB] 2023-10-05T19:16:09.725 INFO:teuthology.orchestra.run.smithi118.stdout:Get:4 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 linux-generic amd64 5.4.0.164.161 [1,896 B] 2023-10-05T19:16:09.726 INFO:teuthology.orchestra.run.smithi118.stdout:Get:5 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 linux-image-generic amd64 5.4.0.164.161 [2,592 B] 2023-10-05T19:16:09.726 INFO:teuthology.orchestra.run.smithi118.stdout:Get:6 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 linux-headers-5.4.0-164 all 5.4.0-164.181 [11.0 MB] 2023-10-05T19:16:09.820 INFO:teuthology.orchestra.run.smithi118.stdout:Get:7 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 linux-headers-5.4.0-164-generic amd64 5.4.0-164.181 [1,360 kB] 2023-10-05T19:16:09.831 INFO:teuthology.orchestra.run.smithi118.stdout:Get:8 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 linux-headers-generic amd64 5.4.0.164.161 [2,468 B] 2023-10-05T19:16:10.457 INFO:teuthology.orchestra.run.smithi167.stdout:Get:2 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 linux-image-5.4.0-164-generic amd64 5.4.0-164.181 [10.5 MB] 2023-10-05T19:16:10.508 INFO:teuthology.orchestra.run.smithi118.stdout:Fetched 77.0 MB in 1s (72.4 MB/s) 2023-10-05T19:16:10.674 INFO:teuthology.orchestra.run.smithi118.stdout:Selecting previously unselected package linux-modules-5.4.0-164-generic. 2023-10-05T19:16:10.726 INFO:teuthology.orchestra.run.smithi167.stdout:Get:3 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 linux-modules-extra-5.4.0-164-generic amd64 5.4.0-164.181 [39.2 MB] 2023-10-05T19:16:11.996 INFO:teuthology.orchestra.run.smithi167.stdout:Get:4 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 linux-generic amd64 5.4.0.164.161 [1,896 B] 2023-10-05T19:16:11.996 INFO:teuthology.orchestra.run.smithi167.stdout:Get:5 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 linux-image-generic amd64 5.4.0.164.161 [2,592 B] 2023-10-05T19:16:11.997 INFO:teuthology.orchestra.run.smithi167.stdout:Get:6 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 linux-headers-5.4.0-164 all 5.4.0-164.181 [11.0 MB] 2023-10-05T19:16:12.282 INFO:teuthology.orchestra.run.smithi167.stdout:Get:7 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 linux-headers-5.4.0-164-generic amd64 5.4.0-164.181 [1,360 kB] 2023-10-05T19:16:12.318 INFO:teuthology.orchestra.run.smithi167.stdout:Get:8 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 linux-headers-generic amd64 5.4.0.164.161 [2,468 B] 2023-10-05T19:16:12.391 INFO:teuthology.orchestra.run.smithi118.stdout:(Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 104373 files and directories currently installed.) 2023-10-05T19:16:12.396 INFO:teuthology.orchestra.run.smithi118.stdout:Preparing to unpack .../0-linux-modules-5.4.0-164-generic_5.4.0-164.181_amd64.deb ... 2023-10-05T19:16:12.472 INFO:teuthology.orchestra.run.smithi118.stdout:Unpacking linux-modules-5.4.0-164-generic (5.4.0-164.181) ... 2023-10-05T19:16:13.035 INFO:teuthology.orchestra.run.smithi167.stdout:Fetched 77.0 MB in 3s (24.1 MB/s) 2023-10-05T19:16:13.230 INFO:teuthology.orchestra.run.smithi167.stdout:Selecting previously unselected package linux-modules-5.4.0-164-generic. 2023-10-05T19:16:14.382 INFO:teuthology.orchestra.run.smithi118.stdout:Selecting previously unselected package linux-image-5.4.0-164-generic. 2023-10-05T19:16:14.395 INFO:teuthology.orchestra.run.smithi118.stdout:Preparing to unpack .../1-linux-image-5.4.0-164-generic_5.4.0-164.181_amd64.deb ... 2023-10-05T19:16:14.482 INFO:teuthology.orchestra.run.smithi118.stdout:Unpacking linux-image-5.4.0-164-generic (5.4.0-164.181) ... 2023-10-05T19:16:14.942 INFO:teuthology.orchestra.run.smithi167.stdout:(Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 104373 files and directories currently installed.) 2023-10-05T19:16:14.948 INFO:teuthology.orchestra.run.smithi167.stdout:Preparing to unpack .../0-linux-modules-5.4.0-164-generic_5.4.0-164.181_amd64.deb ... 2023-10-05T19:16:15.044 INFO:teuthology.orchestra.run.smithi167.stdout:Unpacking linux-modules-5.4.0-164-generic (5.4.0-164.181) ... 2023-10-05T19:16:15.560 INFO:teuthology.orchestra.run.smithi118.stdout:Selecting previously unselected package linux-modules-extra-5.4.0-164-generic. 2023-10-05T19:16:15.574 INFO:teuthology.orchestra.run.smithi118.stdout:Preparing to unpack .../2-linux-modules-extra-5.4.0-164-generic_5.4.0-164.181_amd64.deb ... 2023-10-05T19:16:15.619 INFO:teuthology.orchestra.run.smithi118.stdout:Unpacking linux-modules-extra-5.4.0-164-generic (5.4.0-164.181) ... 2023-10-05T19:16:16.920 INFO:teuthology.orchestra.run.smithi167.stdout:Selecting previously unselected package linux-image-5.4.0-164-generic. 2023-10-05T19:16:16.937 INFO:teuthology.orchestra.run.smithi167.stdout:Preparing to unpack .../1-linux-image-5.4.0-164-generic_5.4.0-164.181_amd64.deb ... 2023-10-05T19:16:17.048 INFO:teuthology.orchestra.run.smithi167.stdout:Unpacking linux-image-5.4.0-164-generic (5.4.0-164.181) ... 2023-10-05T19:16:18.148 INFO:teuthology.orchestra.run.smithi167.stdout:Selecting previously unselected package linux-modules-extra-5.4.0-164-generic. 2023-10-05T19:16:18.308 INFO:teuthology.orchestra.run.smithi167.stdout:Preparing to unpack .../2-linux-modules-extra-5.4.0-164-generic_5.4.0-164.181_amd64.deb ... 2023-10-05T19:16:18.308 INFO:teuthology.orchestra.run.smithi167.stdout:Unpacking linux-modules-extra-5.4.0-164-generic (5.4.0-164.181) ... 2023-10-05T19:16:20.335 INFO:teuthology.orchestra.run.smithi118.stdout:Preparing to unpack .../3-linux-generic_5.4.0.164.161_amd64.deb ... 2023-10-05T19:16:20.462 INFO:teuthology.orchestra.run.smithi118.stdout:Unpacking linux-generic (5.4.0.164.161) over (5.4.0.124.125) ... 2023-10-05T19:16:20.799 INFO:teuthology.orchestra.run.smithi118.stdout:Preparing to unpack .../4-linux-image-generic_5.4.0.164.161_amd64.deb ... 2023-10-05T19:16:20.924 INFO:teuthology.orchestra.run.smithi118.stdout:Unpacking linux-image-generic (5.4.0.164.161) over (5.4.0.124.125) ... 2023-10-05T19:16:21.244 INFO:teuthology.orchestra.run.smithi118.stdout:Selecting previously unselected package linux-headers-5.4.0-164. 2023-10-05T19:16:21.258 INFO:teuthology.orchestra.run.smithi118.stdout:Preparing to unpack .../5-linux-headers-5.4.0-164_5.4.0-164.181_all.deb ... 2023-10-05T19:16:21.302 INFO:teuthology.orchestra.run.smithi118.stdout:Unpacking linux-headers-5.4.0-164 (5.4.0-164.181) ... 2023-10-05T19:16:23.005 INFO:teuthology.orchestra.run.smithi167.stdout:Preparing to unpack .../3-linux-generic_5.4.0.164.161_amd64.deb ... 2023-10-05T19:16:23.131 INFO:teuthology.orchestra.run.smithi167.stdout:Unpacking linux-generic (5.4.0.164.161) over (5.4.0.124.125) ... 2023-10-05T19:16:23.450 INFO:teuthology.orchestra.run.smithi167.stdout:Preparing to unpack .../4-linux-image-generic_5.4.0.164.161_amd64.deb ... 2023-10-05T19:16:23.593 INFO:teuthology.orchestra.run.smithi167.stdout:Unpacking linux-image-generic (5.4.0.164.161) over (5.4.0.124.125) ... 2023-10-05T19:16:23.930 INFO:teuthology.orchestra.run.smithi167.stdout:Selecting previously unselected package linux-headers-5.4.0-164. 2023-10-05T19:16:23.945 INFO:teuthology.orchestra.run.smithi167.stdout:Preparing to unpack .../5-linux-headers-5.4.0-164_5.4.0-164.181_all.deb ... 2023-10-05T19:16:23.988 INFO:teuthology.orchestra.run.smithi167.stdout:Unpacking linux-headers-5.4.0-164 (5.4.0-164.181) ... 2023-10-05T19:16:27.004 INFO:teuthology.orchestra.run.smithi118.stdout:Selecting previously unselected package linux-headers-5.4.0-164-generic. 2023-10-05T19:16:27.026 INFO:teuthology.orchestra.run.smithi118.stdout:Preparing to unpack .../6-linux-headers-5.4.0-164-generic_5.4.0-164.181_amd64.deb ... 2023-10-05T19:16:27.070 INFO:teuthology.orchestra.run.smithi118.stdout:Unpacking linux-headers-5.4.0-164-generic (5.4.0-164.181) ... 2023-10-05T19:16:29.354 INFO:teuthology.orchestra.run.smithi167.stdout:Selecting previously unselected package linux-headers-5.4.0-164-generic. 2023-10-05T19:16:29.374 INFO:teuthology.orchestra.run.smithi167.stdout:Preparing to unpack .../6-linux-headers-5.4.0-164-generic_5.4.0-164.181_amd64.deb ... 2023-10-05T19:16:29.412 INFO:teuthology.orchestra.run.smithi167.stdout:Unpacking linux-headers-5.4.0-164-generic (5.4.0-164.181) ... 2023-10-05T19:16:29.767 INFO:teuthology.orchestra.run.smithi118.stdout:Preparing to unpack .../7-linux-headers-generic_5.4.0.164.161_amd64.deb ... 2023-10-05T19:16:29.894 INFO:teuthology.orchestra.run.smithi118.stdout:Unpacking linux-headers-generic (5.4.0.164.161) over (5.4.0.124.125) ... 2023-10-05T19:16:30.220 INFO:teuthology.orchestra.run.smithi118.stdout:Setting up linux-headers-5.4.0-164 (5.4.0-164.181) ... 2023-10-05T19:16:30.366 INFO:teuthology.orchestra.run.smithi118.stdout:Setting up linux-headers-5.4.0-164-generic (5.4.0-164.181) ... 2023-10-05T19:16:30.500 INFO:teuthology.orchestra.run.smithi118.stdout:Setting up linux-modules-5.4.0-164-generic (5.4.0-164.181) ... 2023-10-05T19:16:31.710 INFO:teuthology.orchestra.run.smithi118.stdout:Setting up linux-image-5.4.0-164-generic (5.4.0-164.181) ... 2023-10-05T19:16:32.051 INFO:teuthology.orchestra.run.smithi167.stdout:Preparing to unpack .../7-linux-headers-generic_5.4.0.164.161_amd64.deb ... 2023-10-05T19:16:32.171 INFO:teuthology.orchestra.run.smithi167.stdout:Unpacking linux-headers-generic (5.4.0.164.161) over (5.4.0.124.125) ... 2023-10-05T19:16:32.497 INFO:teuthology.orchestra.run.smithi167.stdout:Setting up linux-headers-5.4.0-164 (5.4.0-164.181) ... 2023-10-05T19:16:32.626 INFO:teuthology.orchestra.run.smithi167.stdout:Setting up linux-headers-5.4.0-164-generic (5.4.0-164.181) ... 2023-10-05T19:16:32.752 INFO:teuthology.orchestra.run.smithi167.stdout:Setting up linux-modules-5.4.0-164-generic (5.4.0-164.181) ... 2023-10-05T19:16:32.865 INFO:teuthology.orchestra.run.smithi118.stdout:I: /vmlinuz is now a symlink to boot/vmlinuz-5.4.0-164-generic 2023-10-05T19:16:32.866 INFO:teuthology.orchestra.run.smithi118.stdout:I: /initrd.img is now a symlink to boot/initrd.img-5.4.0-164-generic 2023-10-05T19:16:33.003 INFO:teuthology.orchestra.run.smithi118.stdout:Setting up linux-headers-generic (5.4.0.164.161) ... 2023-10-05T19:16:33.129 INFO:teuthology.orchestra.run.smithi118.stdout:Setting up linux-modules-extra-5.4.0-164-generic (5.4.0-164.181) ... 2023-10-05T19:16:34.648 INFO:teuthology.orchestra.run.smithi118.stdout:Setting up linux-image-generic (5.4.0.164.161) ... 2023-10-05T19:16:34.774 INFO:teuthology.orchestra.run.smithi118.stdout:Setting up linux-generic (5.4.0.164.161) ... 2023-10-05T19:16:34.909 INFO:teuthology.orchestra.run.smithi118.stdout:Processing triggers for linux-image-5.4.0-164-generic (5.4.0-164.181) ... 2023-10-05T19:16:34.972 INFO:teuthology.orchestra.run.smithi118.stdout:/etc/kernel/postinst.d/initramfs-tools: 2023-10-05T19:16:34.973 INFO:teuthology.orchestra.run.smithi118.stdout:update-initramfs: Generating /boot/initrd.img-5.4.0-164-generic 2023-10-05T19:16:35.230 INFO:teuthology.orchestra.run.smithi167.stdout:Setting up linux-image-5.4.0-164-generic (5.4.0-164.181) ... 2023-10-05T19:16:36.694 INFO:teuthology.orchestra.run.smithi167.stdout:I: /vmlinuz is now a symlink to boot/vmlinuz-5.4.0-164-generic 2023-10-05T19:16:36.694 INFO:teuthology.orchestra.run.smithi167.stdout:I: /initrd.img is now a symlink to boot/initrd.img-5.4.0-164-generic 2023-10-05T19:16:36.857 INFO:teuthology.orchestra.run.smithi167.stdout:Setting up linux-headers-generic (5.4.0.164.161) ... 2023-10-05T19:16:36.983 INFO:teuthology.orchestra.run.smithi167.stdout:Setting up linux-modules-extra-5.4.0-164-generic (5.4.0-164.181) ... 2023-10-05T19:16:38.201 INFO:teuthology.orchestra.run.smithi167.stdout:Setting up linux-image-generic (5.4.0.164.161) ... 2023-10-05T19:16:38.327 INFO:teuthology.orchestra.run.smithi167.stdout:Setting up linux-generic (5.4.0.164.161) ... 2023-10-05T19:16:38.470 INFO:teuthology.orchestra.run.smithi167.stdout:Processing triggers for linux-image-5.4.0-164-generic (5.4.0-164.181) ... 2023-10-05T19:16:38.530 INFO:teuthology.orchestra.run.smithi167.stdout:/etc/kernel/postinst.d/initramfs-tools: 2023-10-05T19:16:38.530 INFO:teuthology.orchestra.run.smithi167.stdout:update-initramfs: Generating /boot/initrd.img-5.4.0-164-generic 2023-10-05T19:16:57.379 INFO:teuthology.orchestra.run.smithi118.stdout:/etc/kernel/postinst.d/zz-update-grub: 2023-10-05T19:16:57.379 INFO:teuthology.orchestra.run.smithi118.stdout:Sourcing file `/etc/default/grub' 2023-10-05T19:16:57.385 INFO:teuthology.orchestra.run.smithi118.stdout:Sourcing file `/etc/default/grub.d/init-select.cfg' 2023-10-05T19:16:57.399 INFO:teuthology.orchestra.run.smithi118.stdout:Generating grub configuration file ... 2023-10-05T19:16:57.694 INFO:teuthology.orchestra.run.smithi118.stdout:Found linux image: /boot/vmlinuz-5.4.0-164-generic 2023-10-05T19:16:57.714 INFO:teuthology.orchestra.run.smithi118.stdout:Found initrd image: /boot/initrd.img-5.4.0-164-generic 2023-10-05T19:16:58.070 INFO:teuthology.orchestra.run.smithi118.stdout:Found linux image: /boot/vmlinuz-5.4.0-124-generic 2023-10-05T19:16:58.082 INFO:teuthology.orchestra.run.smithi118.stdout:Found initrd image: /boot/initrd.img-5.4.0-124-generic 2023-10-05T19:16:59.989 INFO:teuthology.orchestra.run.smithi118.stdout:done 2023-10-05T19:17:00.391 DEBUG:teuthology.orchestra.run.smithi118:> dpkg -s linux-image-generic 2023-10-05T19:17:00.439 INFO:teuthology.orchestra.run.smithi118.stdout:Package: linux-image-generic 2023-10-05T19:17:00.440 INFO:teuthology.orchestra.run.smithi118.stdout:Status: install ok installed 2023-10-05T19:17:00.440 INFO:teuthology.orchestra.run.smithi118.stdout:Priority: optional 2023-10-05T19:17:00.440 INFO:teuthology.orchestra.run.smithi118.stdout:Section: kernel 2023-10-05T19:17:00.440 INFO:teuthology.orchestra.run.smithi118.stdout:Installed-Size: 20 2023-10-05T19:17:00.440 INFO:teuthology.orchestra.run.smithi118.stdout:Maintainer: Ubuntu Kernel Team 2023-10-05T19:17:00.440 INFO:teuthology.orchestra.run.smithi118.stdout:Architecture: amd64 2023-10-05T19:17:00.440 INFO:teuthology.orchestra.run.smithi118.stdout:Source: linux-meta 2023-10-05T19:17:00.440 INFO:teuthology.orchestra.run.smithi118.stdout:Version: 5.4.0.164.161 2023-10-05T19:17:00.440 INFO:teuthology.orchestra.run.smithi118.stdout:Provides: virtualbox-guest-modules (= 6.1.38-dfsg-3~ubuntu1.20.04.1), wireguard-modules (= 1.0.20201112-1~20.04.1), zfs-modules (= 0.8.3-1ubuntu12.15) 2023-10-05T19:17:00.440 INFO:teuthology.orchestra.run.smithi118.stdout:Depends: linux-image-5.4.0-164-generic, linux-modules-extra-5.4.0-164-generic, linux-firmware, intel-microcode, amd64-microcode 2023-10-05T19:17:00.440 INFO:teuthology.orchestra.run.smithi118.stdout:Recommends: thermald 2023-10-05T19:17:00.441 INFO:teuthology.orchestra.run.smithi118.stdout:Description: Generic Linux kernel image 2023-10-05T19:17:00.441 INFO:teuthology.orchestra.run.smithi118.stdout: This package will always depend on the latest generic kernel image 2023-10-05T19:17:00.441 INFO:teuthology.orchestra.run.smithi118.stdout: available. 2023-10-05T19:17:00.441 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.4.0-124-generic Expected: 5.4.0-164-generic 2023-10-05T19:17:00.441 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2023-10-05T19:17:00.441 DEBUG:teuthology.task.kernel:src is distro, skipping download 2023-10-05T19:17:00.441 INFO:teuthology.task.kernel:Installing distro kernel on mon.a... 2023-10-05T19:17:00.441 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi118.front.sepia.ceph.com, path=None, version=distro) 2023-10-05T19:17:00.441 DEBUG:teuthology.orchestra.run.smithi118:> sudo apt-get clean 2023-10-05T19:17:00.488 INFO:teuthology.orchestra.run.smithi167.stdout:/etc/kernel/postinst.d/zz-update-grub: 2023-10-05T19:17:00.488 INFO:teuthology.orchestra.run.smithi167.stdout:Sourcing file `/etc/default/grub' 2023-10-05T19:17:00.500 INFO:teuthology.orchestra.run.smithi167.stdout:Sourcing file `/etc/default/grub.d/init-select.cfg' 2023-10-05T19:17:00.521 INFO:teuthology.orchestra.run.smithi167.stdout:Generating grub configuration file ... 2023-10-05T19:17:00.522 DEBUG:teuthology.orchestra.run.smithi118:> sudo apt-get update 2023-10-05T19:17:00.782 INFO:teuthology.orchestra.run.smithi118.stdout:Hit:1 http://security.ubuntu.com/ubuntu focal-security InRelease 2023-10-05T19:17:00.782 INFO:teuthology.orchestra.run.smithi118.stdout:Hit:2 http://archive.ubuntu.com/ubuntu focal InRelease 2023-10-05T19:17:00.853 INFO:teuthology.orchestra.run.smithi118.stdout:Hit:3 http://archive.ubuntu.com/ubuntu focal-updates InRelease 2023-10-05T19:17:00.914 INFO:teuthology.orchestra.run.smithi167.stdout:Found linux image: /boot/vmlinuz-5.4.0-164-generic 2023-10-05T19:17:00.934 INFO:teuthology.orchestra.run.smithi118.stdout:Hit:4 http://archive.ubuntu.com/ubuntu focal-backports InRelease 2023-10-05T19:17:00.935 INFO:teuthology.orchestra.run.smithi167.stdout:Found initrd image: /boot/initrd.img-5.4.0-164-generic 2023-10-05T19:17:01.354 INFO:teuthology.orchestra.run.smithi167.stdout:Found linux image: /boot/vmlinuz-5.4.0-124-generic 2023-10-05T19:17:01.365 INFO:teuthology.orchestra.run.smithi167.stdout:Found initrd image: /boot/initrd.img-5.4.0-124-generic 2023-10-05T19:17:02.518 INFO:teuthology.orchestra.run.smithi118.stdout:Reading package lists... 2023-10-05T19:17:02.544 DEBUG:teuthology.orchestra.run.smithi118:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y install linux-image-generic 2023-10-05T19:17:02.614 INFO:teuthology.orchestra.run.smithi118.stdout:Reading package lists... 2023-10-05T19:17:02.793 INFO:teuthology.orchestra.run.smithi118.stdout:Building dependency tree... 2023-10-05T19:17:02.793 INFO:teuthology.orchestra.run.smithi118.stdout:Reading state information... 2023-10-05T19:17:03.000 INFO:teuthology.orchestra.run.smithi118.stdout:linux-image-generic is already the newest version (5.4.0.164.161). 2023-10-05T19:17:03.000 INFO:teuthology.orchestra.run.smithi118.stdout:0 upgraded, 0 newly installed, 0 to remove and 271 not upgraded. 2023-10-05T19:17:03.002 DEBUG:teuthology.orchestra.run.smithi118:> dpkg -s linux-image-generic 2023-10-05T19:17:03.020 INFO:teuthology.orchestra.run.smithi118.stdout:Package: linux-image-generic 2023-10-05T19:17:03.020 INFO:teuthology.orchestra.run.smithi118.stdout:Status: install ok installed 2023-10-05T19:17:03.020 INFO:teuthology.orchestra.run.smithi118.stdout:Priority: optional 2023-10-05T19:17:03.020 INFO:teuthology.orchestra.run.smithi118.stdout:Section: kernel 2023-10-05T19:17:03.020 INFO:teuthology.orchestra.run.smithi118.stdout:Installed-Size: 20 2023-10-05T19:17:03.020 INFO:teuthology.orchestra.run.smithi118.stdout:Maintainer: Ubuntu Kernel Team 2023-10-05T19:17:03.020 INFO:teuthology.orchestra.run.smithi118.stdout:Architecture: amd64 2023-10-05T19:17:03.020 INFO:teuthology.orchestra.run.smithi118.stdout:Source: linux-meta 2023-10-05T19:17:03.020 INFO:teuthology.orchestra.run.smithi118.stdout:Version: 5.4.0.164.161 2023-10-05T19:17:03.020 INFO:teuthology.orchestra.run.smithi118.stdout:Provides: virtualbox-guest-modules (= 6.1.38-dfsg-3~ubuntu1.20.04.1), wireguard-modules (= 1.0.20201112-1~20.04.1), zfs-modules (= 0.8.3-1ubuntu12.15) 2023-10-05T19:17:03.021 INFO:teuthology.orchestra.run.smithi118.stdout:Depends: linux-image-5.4.0-164-generic, linux-modules-extra-5.4.0-164-generic, linux-firmware, intel-microcode, amd64-microcode 2023-10-05T19:17:03.021 INFO:teuthology.orchestra.run.smithi118.stdout:Recommends: thermald 2023-10-05T19:17:03.021 INFO:teuthology.orchestra.run.smithi118.stdout:Description: Generic Linux kernel image 2023-10-05T19:17:03.021 INFO:teuthology.orchestra.run.smithi118.stdout: This package will always depend on the latest generic kernel image 2023-10-05T19:17:03.021 INFO:teuthology.orchestra.run.smithi118.stdout: available. 2023-10-05T19:17:03.021 DEBUG:teuthology.orchestra.run.smithi118:> mktemp 2023-10-05T19:17:03.027 INFO:teuthology.orchestra.run.smithi118.stdout:/tmp/tmp.upIPq8CJkU 2023-10-05T19:17:03.027 DEBUG:teuthology.orchestra.run.smithi118:> sudo cp /boot/grub/grub.cfg /tmp/tmp.upIPq8CJkU 2023-10-05T19:17:03.066 DEBUG:teuthology.orchestra.run.smithi118:> sudo chmod 0666 /tmp/tmp.upIPq8CJkU 2023-10-05T19:17:03.112 DEBUG:teuthology.orchestra.remote:smithi118:/tmp/tmp.upIPq8CJkU is 9KB 2023-10-05T19:17:03.125 DEBUG:teuthology.orchestra.run.smithi118:> rm -fr /tmp/tmp.upIPq8CJkU 2023-10-05T19:17:03.132 DEBUG:teuthology.orchestra.run.smithi118:> sudo rm -f -- /etc/grub.d/01_ceph_kernel 2023-10-05T19:17:03.160 INFO:teuthology.orchestra.run.smithi167.stdout:done 2023-10-05T19:17:03.187 DEBUG:teuthology.orchestra.run.smithi118:> set -ex 2023-10-05T19:17:03.187 DEBUG:teuthology.orchestra.run.smithi118:> sudo dd of=/etc/grub.d/01_ceph_kernel 2023-10-05T19:17:03.187 DEBUG:teuthology.orchestra.run.smithi118:> sudo chmod 755 /etc/grub.d/01_ceph_kernel 2023-10-05T19:17:03.222 INFO:teuthology.task.kernel:Distro Kernel Version: 5.4.0-164-generic 2023-10-05T19:17:03.222 DEBUG:teuthology.orchestra.run.smithi118:> sudo update-grub 2023-10-05T19:17:03.580 DEBUG:teuthology.orchestra.run.smithi167:> dpkg -s linux-image-generic 2023-10-05T19:17:03.620 INFO:teuthology.orchestra.run.smithi167.stdout:Package: linux-image-generic 2023-10-05T19:17:03.620 INFO:teuthology.orchestra.run.smithi167.stdout:Status: install ok installed 2023-10-05T19:17:03.620 INFO:teuthology.orchestra.run.smithi167.stdout:Priority: optional 2023-10-05T19:17:03.620 INFO:teuthology.orchestra.run.smithi167.stdout:Section: kernel 2023-10-05T19:17:03.620 INFO:teuthology.orchestra.run.smithi167.stdout:Installed-Size: 20 2023-10-05T19:17:03.620 INFO:teuthology.orchestra.run.smithi167.stdout:Maintainer: Ubuntu Kernel Team 2023-10-05T19:17:03.620 INFO:teuthology.orchestra.run.smithi167.stdout:Architecture: amd64 2023-10-05T19:17:03.620 INFO:teuthology.orchestra.run.smithi167.stdout:Source: linux-meta 2023-10-05T19:17:03.620 INFO:teuthology.orchestra.run.smithi167.stdout:Version: 5.4.0.164.161 2023-10-05T19:17:03.620 INFO:teuthology.orchestra.run.smithi167.stdout:Provides: virtualbox-guest-modules (= 6.1.38-dfsg-3~ubuntu1.20.04.1), wireguard-modules (= 1.0.20201112-1~20.04.1), zfs-modules (= 0.8.3-1ubuntu12.15) 2023-10-05T19:17:03.621 INFO:teuthology.orchestra.run.smithi167.stdout:Depends: linux-image-5.4.0-164-generic, linux-modules-extra-5.4.0-164-generic, linux-firmware, intel-microcode, amd64-microcode 2023-10-05T19:17:03.621 INFO:teuthology.orchestra.run.smithi167.stdout:Recommends: thermald 2023-10-05T19:17:03.621 INFO:teuthology.orchestra.run.smithi167.stdout:Description: Generic Linux kernel image 2023-10-05T19:17:03.621 INFO:teuthology.orchestra.run.smithi167.stdout: This package will always depend on the latest generic kernel image 2023-10-05T19:17:03.621 INFO:teuthology.orchestra.run.smithi167.stdout: available. 2023-10-05T19:17:03.621 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.4.0-124-generic Expected: 5.4.0-164-generic 2023-10-05T19:17:03.621 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2023-10-05T19:17:03.621 DEBUG:teuthology.task.kernel:src is distro, skipping download 2023-10-05T19:17:03.621 INFO:teuthology.task.kernel:Installing distro kernel on mgr.x... 2023-10-05T19:17:03.621 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi167.front.sepia.ceph.com, path=None, version=distro) 2023-10-05T19:17:03.622 DEBUG:teuthology.orchestra.run.smithi167:> sudo apt-get clean 2023-10-05T19:17:03.729 DEBUG:teuthology.orchestra.run.smithi167:> sudo apt-get update 2023-10-05T19:17:03.886 INFO:teuthology.orchestra.run.smithi167.stdout:Hit:1 http://security.ubuntu.com/ubuntu focal-security InRelease 2023-10-05T19:17:04.000 INFO:teuthology.orchestra.run.smithi167.stdout:Hit:2 http://archive.ubuntu.com/ubuntu focal InRelease 2023-10-05T19:17:04.020 INFO:teuthology.orchestra.run.smithi167.stdout:Hit:3 http://archive.ubuntu.com/ubuntu focal-updates InRelease 2023-10-05T19:17:04.040 INFO:teuthology.orchestra.run.smithi167.stdout:Hit:4 http://archive.ubuntu.com/ubuntu focal-backports InRelease 2023-10-05T19:17:04.426 INFO:teuthology.orchestra.run.smithi118.stderr:Sourcing file `/etc/default/grub' 2023-10-05T19:17:04.433 INFO:teuthology.orchestra.run.smithi118.stderr:Sourcing file `/etc/default/grub.d/init-select.cfg' 2023-10-05T19:17:04.445 INFO:teuthology.orchestra.run.smithi118.stderr:Generating grub configuration file ... 2023-10-05T19:17:04.704 INFO:teuthology.orchestra.run.smithi118.stderr:Found linux image: /boot/vmlinuz-5.4.0-164-generic 2023-10-05T19:17:04.716 INFO:teuthology.orchestra.run.smithi118.stderr:Found initrd image: /boot/initrd.img-5.4.0-164-generic 2023-10-05T19:17:05.072 INFO:teuthology.orchestra.run.smithi118.stderr:Found linux image: /boot/vmlinuz-5.4.0-124-generic 2023-10-05T19:17:05.083 INFO:teuthology.orchestra.run.smithi118.stderr:Found initrd image: /boot/initrd.img-5.4.0-124-generic 2023-10-05T19:17:05.661 INFO:teuthology.orchestra.run.smithi167.stdout:Reading package lists... 2023-10-05T19:17:05.687 DEBUG:teuthology.orchestra.run.smithi167:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y install linux-image-generic 2023-10-05T19:17:05.753 INFO:teuthology.orchestra.run.smithi167.stdout:Reading package lists... 2023-10-05T19:17:05.824 INFO:teuthology.orchestra.run.smithi118.stderr:done 2023-10-05T19:17:05.826 DEBUG:teuthology.orchestra.run.smithi118:> sudo shutdown -r now 2023-10-05T19:17:05.964 INFO:teuthology.orchestra.run.smithi167.stdout:Building dependency tree... 2023-10-05T19:17:05.964 INFO:teuthology.orchestra.run.smithi167.stdout:Reading state information... 2023-10-05T19:17:06.191 INFO:teuthology.orchestra.run.smithi167.stdout:linux-image-generic is already the newest version (5.4.0.164.161). 2023-10-05T19:17:06.191 INFO:teuthology.orchestra.run.smithi167.stdout:0 upgraded, 0 newly installed, 0 to remove and 271 not upgraded. 2023-10-05T19:17:06.193 DEBUG:teuthology.orchestra.run.smithi167:> dpkg -s linux-image-generic 2023-10-05T19:17:06.215 INFO:teuthology.orchestra.run.smithi167.stdout:Package: linux-image-generic 2023-10-05T19:17:06.215 INFO:teuthology.orchestra.run.smithi167.stdout:Status: install ok installed 2023-10-05T19:17:06.216 INFO:teuthology.orchestra.run.smithi167.stdout:Priority: optional 2023-10-05T19:17:06.216 INFO:teuthology.orchestra.run.smithi167.stdout:Section: kernel 2023-10-05T19:17:06.216 INFO:teuthology.orchestra.run.smithi167.stdout:Installed-Size: 20 2023-10-05T19:17:06.216 INFO:teuthology.orchestra.run.smithi167.stdout:Maintainer: Ubuntu Kernel Team 2023-10-05T19:17:06.216 INFO:teuthology.orchestra.run.smithi167.stdout:Architecture: amd64 2023-10-05T19:17:06.216 INFO:teuthology.orchestra.run.smithi167.stdout:Source: linux-meta 2023-10-05T19:17:06.216 INFO:teuthology.orchestra.run.smithi167.stdout:Version: 5.4.0.164.161 2023-10-05T19:17:06.216 INFO:teuthology.orchestra.run.smithi167.stdout:Provides: virtualbox-guest-modules (= 6.1.38-dfsg-3~ubuntu1.20.04.1), wireguard-modules (= 1.0.20201112-1~20.04.1), zfs-modules (= 0.8.3-1ubuntu12.15) 2023-10-05T19:17:06.216 INFO:teuthology.orchestra.run.smithi167.stdout:Depends: linux-image-5.4.0-164-generic, linux-modules-extra-5.4.0-164-generic, linux-firmware, intel-microcode, amd64-microcode 2023-10-05T19:17:06.216 INFO:teuthology.orchestra.run.smithi167.stdout:Recommends: thermald 2023-10-05T19:17:06.217 INFO:teuthology.orchestra.run.smithi167.stdout:Description: Generic Linux kernel image 2023-10-05T19:17:06.217 INFO:teuthology.orchestra.run.smithi167.stdout: This package will always depend on the latest generic kernel image 2023-10-05T19:17:06.217 INFO:teuthology.orchestra.run.smithi167.stdout: available. 2023-10-05T19:17:06.217 DEBUG:teuthology.orchestra.run.smithi167:> mktemp 2023-10-05T19:17:06.263 INFO:teuthology.orchestra.run.smithi167.stdout:/tmp/tmp.uSoZ6ZwZuV 2023-10-05T19:17:06.264 DEBUG:teuthology.orchestra.run.smithi167:> sudo cp /boot/grub/grub.cfg /tmp/tmp.uSoZ6ZwZuV 2023-10-05T19:17:06.279 DEBUG:teuthology.orchestra.run.smithi167:> sudo chmod 0666 /tmp/tmp.uSoZ6ZwZuV 2023-10-05T19:17:06.365 DEBUG:teuthology.orchestra.remote:smithi167:/tmp/tmp.uSoZ6ZwZuV is 9KB 2023-10-05T19:17:06.379 DEBUG:teuthology.orchestra.run.smithi167:> rm -fr /tmp/tmp.uSoZ6ZwZuV 2023-10-05T19:17:06.386 DEBUG:teuthology.orchestra.run.smithi167:> sudo rm -f -- /etc/grub.d/01_ceph_kernel 2023-10-05T19:17:06.445 DEBUG:teuthology.orchestra.run.smithi167:> set -ex 2023-10-05T19:17:06.445 DEBUG:teuthology.orchestra.run.smithi167:> sudo dd of=/etc/grub.d/01_ceph_kernel 2023-10-05T19:17:06.445 DEBUG:teuthology.orchestra.run.smithi167:> sudo chmod 755 /etc/grub.d/01_ceph_kernel 2023-10-05T19:17:06.480 INFO:teuthology.task.kernel:Distro Kernel Version: 5.4.0-164-generic 2023-10-05T19:17:06.480 DEBUG:teuthology.orchestra.run.smithi167:> sudo update-grub 2023-10-05T19:17:07.813 INFO:teuthology.orchestra.run.smithi167.stderr:Sourcing file `/etc/default/grub' 2023-10-05T19:17:07.819 INFO:teuthology.orchestra.run.smithi167.stderr:Sourcing file `/etc/default/grub.d/init-select.cfg' 2023-10-05T19:17:07.828 INFO:teuthology.orchestra.run.smithi167.stderr:Generating grub configuration file ... 2023-10-05T19:17:08.141 INFO:teuthology.orchestra.run.smithi167.stderr:Found linux image: /boot/vmlinuz-5.4.0-164-generic 2023-10-05T19:17:08.153 INFO:teuthology.orchestra.run.smithi167.stderr:Found initrd image: /boot/initrd.img-5.4.0-164-generic 2023-10-05T19:17:08.586 INFO:teuthology.orchestra.run.smithi167.stderr:Found linux image: /boot/vmlinuz-5.4.0-124-generic 2023-10-05T19:17:08.598 INFO:teuthology.orchestra.run.smithi167.stderr:Found initrd image: /boot/initrd.img-5.4.0-124-generic 2023-10-05T19:17:09.299 INFO:teuthology.orchestra.run.smithi167.stderr:done 2023-10-05T19:17:09.300 DEBUG:teuthology.orchestra.run.smithi167:> sudo shutdown -r now 2023-10-05T19:17:35.854 INFO:teuthology.task.kernel:Checking client mon.a for new kernel version... 2023-10-05T19:17:35.855 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi118.front.sepia.ceph.com' 2023-10-05T19:17:35.855 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi118.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-10-05T19:17:39.306 INFO:teuthology.task.kernel:Checking client mgr.x for new kernel version... 2023-10-05T19:17:39.306 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi167.front.sepia.ceph.com' 2023-10-05T19:17:39.307 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi167.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-10-05T19:17:54.174 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.118 2023-10-05T19:17:57.754 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.167 2023-10-05T19:18:03.181 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi118.front.sepia.ceph.com' 2023-10-05T19:18:03.182 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi118.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-10-05T19:18:06.234 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.118 2023-10-05T19:18:06.755 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi167.front.sepia.ceph.com' 2023-10-05T19:18:06.756 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi167.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-10-05T19:18:09.818 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.167 2023-10-05T19:18:18.238 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi118.front.sepia.ceph.com' 2023-10-05T19:18:18.869 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi118.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-10-05T19:18:21.822 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi167.front.sepia.ceph.com' 2023-10-05T19:18:21.823 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi167.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-10-05T19:18:21.946 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.118 2023-10-05T19:18:36.957 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi118.front.sepia.ceph.com' 2023-10-05T19:18:36.958 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi118.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-10-05T19:19:08.627 DEBUG:teuthology.orchestra.run.smithi118:> true 2023-10-05T19:19:09.224 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi118.front.sepia.ceph.com' 2023-10-05T19:19:09.224 INFO:teuthology.task.kernel:Checking kernel version of mon.a, want "5.4.0-164-generic"... 2023-10-05T19:19:09.225 DEBUG:teuthology.orchestra.run.smithi118:> uname -r 2023-10-05T19:19:09.231 INFO:teuthology.orchestra.run.smithi118.stdout:5.4.0-164-generic 2023-10-05T19:19:09.231 DEBUG:teuthology.task.kernel:current kernel version is 5.4.0-164-generic vs 5.4.0-164-generic 2023-10-05T19:19:09.231 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2023-10-05T19:19:09.231 DEBUG:teuthology.task.kernel:Distro of this test job: ubuntu 2023-10-05T19:19:10.232 INFO:teuthology.task.kernel:Enabling kdb on mon.a... 2023-10-05T19:19:10.233 DEBUG:teuthology.orchestra.run.smithi118:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2023-10-05T19:19:10.315 INFO:teuthology.orchestra.run.smithi118.stdout:ttyS1 2023-10-05T19:19:10.342 DEBUG:teuthology.parallel:result is None 2023-10-05T19:19:21.825 DEBUG:teuthology.orchestra.remote:timed out 2023-10-05T19:19:36.826 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi167.front.sepia.ceph.com' 2023-10-05T19:19:36.827 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi167.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-10-05T19:19:37.041 DEBUG:teuthology.orchestra.run.smithi167:> true 2023-10-05T19:19:37.606 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi167.front.sepia.ceph.com' 2023-10-05T19:19:37.606 INFO:teuthology.task.kernel:Checking kernel version of mgr.x, want "5.4.0-164-generic"... 2023-10-05T19:19:37.606 DEBUG:teuthology.orchestra.run.smithi167:> uname -r 2023-10-05T19:19:37.612 INFO:teuthology.orchestra.run.smithi167.stdout:5.4.0-164-generic 2023-10-05T19:19:37.613 DEBUG:teuthology.task.kernel:current kernel version is 5.4.0-164-generic vs 5.4.0-164-generic 2023-10-05T19:19:37.613 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2023-10-05T19:19:37.613 DEBUG:teuthology.task.kernel:Distro of this test job: ubuntu 2023-10-05T19:19:38.614 INFO:teuthology.task.kernel:Enabling kdb on mgr.x... 2023-10-05T19:19:38.614 DEBUG:teuthology.orchestra.run.smithi167:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2023-10-05T19:19:38.708 INFO:teuthology.orchestra.run.smithi167.stdout:ttyS1 2023-10-05T19:19:38.730 DEBUG:teuthology.parallel:result is None 2023-10-05T19:19:38.731 INFO:teuthology.run_tasks:Running task internal.base... 2023-10-05T19:19:38.738 INFO:teuthology.task.internal:Creating test directory... 2023-10-05T19:19:38.738 DEBUG:teuthology.orchestra.run.smithi118:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2023-10-05T19:19:38.744 DEBUG:teuthology.orchestra.run.smithi167:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2023-10-05T19:19:38.754 INFO:teuthology.run_tasks:Running task internal.archive_upload... 2023-10-05T19:19:38.777 INFO:teuthology.run_tasks:Running task internal.archive... 2023-10-05T19:19:38.815 INFO:teuthology.task.internal:Creating archive directory... 2023-10-05T19:19:38.815 DEBUG:teuthology.orchestra.run.smithi118:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2023-10-05T19:19:38.818 DEBUG:teuthology.orchestra.run.smithi167:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2023-10-05T19:19:38.837 INFO:teuthology.run_tasks:Running task internal.coredump... 2023-10-05T19:19:38.885 INFO:teuthology.task.internal:Enabling coredump saving... 2023-10-05T19:19:38.885 DEBUG:teuthology.orchestra.run.smithi118:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2023-10-05T19:19:38.888 DEBUG:teuthology.orchestra.run.smithi167:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2023-10-05T19:19:38.905 INFO:teuthology.orchestra.run.smithi167.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2023-10-05T19:19:38.911 INFO:teuthology.orchestra.run.smithi118.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2023-10-05T19:19:38.915 INFO:teuthology.orchestra.run.smithi167.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2023-10-05T19:19:38.922 INFO:teuthology.orchestra.run.smithi118.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2023-10-05T19:19:38.924 INFO:teuthology.run_tasks:Running task internal.sudo... 2023-10-05T19:19:38.932 INFO:teuthology.task.internal:Configuring sudo... 2023-10-05T19:19:38.933 DEBUG:teuthology.orchestra.run.smithi118:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2023-10-05T19:19:38.935 DEBUG:teuthology.orchestra.run.smithi167:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2023-10-05T19:19:38.949 INFO:teuthology.run_tasks:Running task internal.syslog... 2023-10-05T19:19:38.977 INFO:teuthology.task.internal.syslog:Starting syslog monitoring... 2023-10-05T19:19:38.978 DEBUG:teuthology.orchestra.run.smithi118:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2023-10-05T19:19:38.993 DEBUG:teuthology.orchestra.run.smithi167:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2023-10-05T19:19:39.000 DEBUG:teuthology.orchestra.run.smithi118:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2023-10-05T19:19:39.006 DEBUG:teuthology.orchestra.run.smithi118:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2023-10-05T19:19:39.012 DEBUG:teuthology.orchestra.run.smithi118:> set -ex 2023-10-05T19:19:39.012 DEBUG:teuthology.orchestra.run.smithi118:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2023-10-05T19:19:39.079 DEBUG:teuthology.orchestra.run.smithi167:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2023-10-05T19:19:39.086 DEBUG:teuthology.orchestra.run.smithi167:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2023-10-05T19:19:39.140 DEBUG:teuthology.orchestra.run.smithi167:> set -ex 2023-10-05T19:19:39.140 DEBUG:teuthology.orchestra.run.smithi167:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2023-10-05T19:19:39.182 DEBUG:teuthology.orchestra.run.smithi118:> sudo service rsyslog restart 2023-10-05T19:19:39.185 DEBUG:teuthology.orchestra.run.smithi167:> sudo service rsyslog restart 2023-10-05T19:19:39.235 INFO:teuthology.run_tasks:Running task internal.timer... 2023-10-05T19:19:39.243 INFO:teuthology.task.internal:Starting timer... 2023-10-05T19:19:39.243 INFO:teuthology.run_tasks:Running task pcp... 2023-10-05T19:19:39.262 INFO:teuthology.run_tasks:Running task selinux... 2023-10-05T19:19:39.294 DEBUG:teuthology.task.selinux:Excluding smithi118: OS 'ubuntu' does not support SELinux 2023-10-05T19:19:39.295 DEBUG:teuthology.task.selinux:Excluding smithi167: OS 'ubuntu' does not support SELinux 2023-10-05T19:19:39.295 DEBUG:teuthology.task.selinux:Getting current SELinux state 2023-10-05T19:19:39.295 DEBUG:teuthology.task.selinux:Existing SELinux modes: {} 2023-10-05T19:19:39.295 INFO:teuthology.task.selinux:Putting SELinux into permissive mode 2023-10-05T19:19:39.295 INFO:teuthology.run_tasks:Running task ansible.cephlab... 2023-10-05T19:19:39.336 DEBUG:teuthology.repo_utils:Setting repo remote to https://git.ceph.com/ceph-cm-ansible.git 2023-10-05T19:19:39.342 INFO:teuthology.repo_utils:Fetching main from origin 2023-10-05T19:19:39.447 INFO:teuthology.repo_utils:Resetting repo at /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main to origin/main 2023-10-05T19:19:39.462 INFO:teuthology.task.ansible:Playbook: [{'import_playbook': 'ansible_managed.yml'}, {'import_playbook': 'teuthology.yml'}, {'hosts': 'testnodes', 'tasks': [{'set_fact': {'ran_from_cephlab_playbook': True}}]}, {'import_playbook': 'testnodes.yml'}, {'import_playbook': 'container-host.yml'}, {'import_playbook': 'cobbler.yml'}, {'import_playbook': 'paddles.yml'}, {'import_playbook': 'pulpito.yml'}, {'hosts': 'testnodes', 'become': True, 'tasks': [{'name': 'Touch /ceph-qa-ready', 'file': {'path': '/ceph-qa-ready', 'state': 'touch'}, 'when': 'ran_from_cephlab_playbook|bool'}]}] 2023-10-05T19:19:39.463 DEBUG:teuthology.task.ansible:Running ansible-playbook -v --extra-vars '{"ansible_ssh_user": "ubuntu"}' -i /etc/ansible/hosts --limit smithi118.front.sepia.ceph.com,smithi167.front.sepia.ceph.com /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main/cephlab.yml 2023-10-05T19:25:15.018 DEBUG:teuthology.task.ansible:Reconnecting to [Remote(name='ubuntu@smithi118.front.sepia.ceph.com'), Remote(name='ubuntu@smithi167.front.sepia.ceph.com')] 2023-10-05T19:25:15.019 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi118.front.sepia.ceph.com' 2023-10-05T19:25:15.020 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi118.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-10-05T19:25:15.068 DEBUG:teuthology.orchestra.run.smithi118:> true 2023-10-05T19:25:15.188 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi118.front.sepia.ceph.com' 2023-10-05T19:25:15.188 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi167.front.sepia.ceph.com' 2023-10-05T19:25:15.188 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi167.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-10-05T19:25:15.233 DEBUG:teuthology.orchestra.run.smithi167:> true 2023-10-05T19:25:15.352 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi167.front.sepia.ceph.com' 2023-10-05T19:25:15.352 INFO:teuthology.run_tasks:Running task clock... 2023-10-05T19:25:15.363 INFO:teuthology.task.clock:Syncing clocks and checking initial clock skew... 2023-10-05T19:25:15.363 INFO:teuthology.orchestra.run:Running command with timeout 360 2023-10-05T19:25:15.364 DEBUG:teuthology.orchestra.run.smithi118:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2023-10-05T19:25:15.366 INFO:teuthology.orchestra.run:Running command with timeout 360 2023-10-05T19:25:15.366 DEBUG:teuthology.orchestra.run.smithi167:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2023-10-05T19:25:15.400 INFO:teuthology.orchestra.run.smithi118.stdout: 5 Oct 19:25:15 ntpd[22869]: ntpd 4.2.8p12@1.3728-o (1): Starting 2023-10-05T19:25:15.400 INFO:teuthology.orchestra.run.smithi118.stdout: 5 Oct 19:25:15 ntpd[22869]: Command line: ntpd -gq 2023-10-05T19:25:15.401 INFO:teuthology.orchestra.run.smithi118.stdout: 5 Oct 19:25:15 ntpd[22869]: proto: precision = 0.087 usec (-23) 2023-10-05T19:25:15.401 INFO:teuthology.orchestra.run.smithi118.stderr:restrict 0.0.0.0: KOD does nothing without LIMITED. 2023-10-05T19:25:15.402 INFO:teuthology.orchestra.run.smithi118.stdout: 5 Oct 19:25:15 ntpd[22869]: restrict 0.0.0.0: KOD does nothing without LIMITED. 2023-10-05T19:25:15.402 INFO:teuthology.orchestra.run.smithi118.stderr:restrict ::: KOD does nothing without LIMITED. 2023-10-05T19:25:15.403 INFO:teuthology.orchestra.run.smithi118.stdout: 5 Oct 19:25:15 ntpd[22869]: restrict ::: KOD does nothing without LIMITED. 2023-10-05T19:25:15.403 INFO:teuthology.orchestra.run.smithi118.stdout: 5 Oct 19:25:15 ntpd[22869]: Listen and drop on 0 v6wildcard [::]:123 2023-10-05T19:25:15.403 INFO:teuthology.orchestra.run.smithi118.stdout: 5 Oct 19:25:15 ntpd[22869]: Listen and drop on 1 v4wildcard 0.0.0.0:123 2023-10-05T19:25:15.403 INFO:teuthology.orchestra.run.smithi118.stdout: 5 Oct 19:25:15 ntpd[22869]: Listen normally on 2 lo 127.0.0.1:123 2023-10-05T19:25:15.403 INFO:teuthology.orchestra.run.smithi118.stdout: 5 Oct 19:25:15 ntpd[22869]: Listen normally on 3 enp3s0f1 172.21.15.118:123 2023-10-05T19:25:15.403 INFO:teuthology.orchestra.run.smithi118.stdout: 5 Oct 19:25:15 ntpd[22869]: Listen normally on 4 lo [::1]:123 2023-10-05T19:25:15.403 INFO:teuthology.orchestra.run.smithi118.stdout: 5 Oct 19:25:15 ntpd[22869]: Listen normally on 5 enp3s0f1 [fe80::ec4:7aff:fe8f:d725%5]:123 2023-10-05T19:25:15.404 INFO:teuthology.orchestra.run.smithi118.stdout: 5 Oct 19:25:15 ntpd[22869]: Listening on routing socket on fd #22 for interface updates 2023-10-05T19:25:15.404 INFO:teuthology.orchestra.run.smithi167.stdout: 5 Oct 19:25:15 ntpd[22870]: ntpd 4.2.8p12@1.3728-o (1): Starting 2023-10-05T19:25:15.404 INFO:teuthology.orchestra.run.smithi167.stdout: 5 Oct 19:25:15 ntpd[22870]: Command line: ntpd -gq 2023-10-05T19:25:15.404 INFO:teuthology.orchestra.run.smithi167.stdout: 5 Oct 19:25:15 ntpd[22870]: proto: precision = 0.080 usec (-23) 2023-10-05T19:25:15.405 INFO:teuthology.orchestra.run.smithi167.stderr:restrict 0.0.0.0: KOD does nothing without LIMITED. 2023-10-05T19:25:15.405 INFO:teuthology.orchestra.run.smithi167.stdout: 5 Oct 19:25:15 ntpd[22870]: restrict 0.0.0.0: KOD does nothing without LIMITED. 2023-10-05T19:25:15.405 INFO:teuthology.orchestra.run.smithi167.stderr:restrict ::: KOD does nothing without LIMITED. 2023-10-05T19:25:15.406 INFO:teuthology.orchestra.run.smithi167.stdout: 5 Oct 19:25:15 ntpd[22870]: restrict ::: KOD does nothing without LIMITED. 2023-10-05T19:25:15.406 INFO:teuthology.orchestra.run.smithi167.stdout: 5 Oct 19:25:15 ntpd[22870]: Listen and drop on 0 v6wildcard [::]:123 2023-10-05T19:25:15.406 INFO:teuthology.orchestra.run.smithi167.stdout: 5 Oct 19:25:15 ntpd[22870]: Listen and drop on 1 v4wildcard 0.0.0.0:123 2023-10-05T19:25:15.406 INFO:teuthology.orchestra.run.smithi167.stdout: 5 Oct 19:25:15 ntpd[22870]: Listen normally on 2 lo 127.0.0.1:123 2023-10-05T19:25:15.406 INFO:teuthology.orchestra.run.smithi167.stdout: 5 Oct 19:25:15 ntpd[22870]: Listen normally on 3 enp3s0f1 172.21.15.167:123 2023-10-05T19:25:15.407 INFO:teuthology.orchestra.run.smithi167.stdout: 5 Oct 19:25:15 ntpd[22870]: Listen normally on 4 lo [::1]:123 2023-10-05T19:25:15.407 INFO:teuthology.orchestra.run.smithi167.stdout: 5 Oct 19:25:15 ntpd[22870]: Listen normally on 5 enp3s0f1 [fe80::ec4:7aff:fe88:7309%5]:123 2023-10-05T19:25:15.407 INFO:teuthology.orchestra.run.smithi167.stdout: 5 Oct 19:25:15 ntpd[22870]: Listening on routing socket on fd #22 for interface updates 2023-10-05T19:25:16.401 INFO:teuthology.orchestra.run.smithi118.stderr: 5 Oct 19:25:16 ntpd[22869]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-10-05T19:25:16.401 INFO:teuthology.orchestra.run.smithi118.stderr: 5 Oct 19:25:16 ntpd[22869]: can't open /var/log/ntpstats/rawstats.20231005: Permission denied 2023-10-05T19:25:16.404 INFO:teuthology.orchestra.run.smithi167.stderr: 5 Oct 19:25:16 ntpd[22870]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-10-05T19:25:16.404 INFO:teuthology.orchestra.run.smithi167.stderr: 5 Oct 19:25:16 ntpd[22870]: can't open /var/log/ntpstats/rawstats.20231005: Permission denied 2023-10-05T19:25:16.404 INFO:teuthology.orchestra.run.smithi167.stderr: 5 Oct 19:25:16 ntpd[22870]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-10-05T19:25:16.404 INFO:teuthology.orchestra.run.smithi167.stderr: 5 Oct 19:25:16 ntpd[22870]: can't open /var/log/ntpstats/rawstats.20231005: Permission denied 2023-10-05T19:25:16.405 INFO:teuthology.orchestra.run.smithi167.stderr: 5 Oct 19:25:16 ntpd[22870]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-10-05T19:25:16.405 INFO:teuthology.orchestra.run.smithi167.stderr: 5 Oct 19:25:16 ntpd[22870]: can't open /var/log/ntpstats/rawstats.20231005: Permission denied 2023-10-05T19:25:16.405 INFO:teuthology.orchestra.run.smithi167.stderr: 5 Oct 19:25:16 ntpd[22870]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-10-05T19:25:16.405 INFO:teuthology.orchestra.run.smithi167.stderr: 5 Oct 19:25:16 ntpd[22870]: can't open /var/log/ntpstats/rawstats.20231005: Permission denied 2023-10-05T19:25:16.405 INFO:teuthology.orchestra.run.smithi167.stderr: 5 Oct 19:25:16 ntpd[22870]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2023-10-05T19:25:16.405 INFO:teuthology.orchestra.run.smithi167.stderr: 5 Oct 19:25:16 ntpd[22870]: can't open /var/log/ntpstats/peerstats.20231005: Permission denied 2023-10-05T19:25:16.405 INFO:teuthology.orchestra.run.smithi167.stderr: 5 Oct 19:25:16 ntpd[22870]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-10-05T19:25:16.405 INFO:teuthology.orchestra.run.smithi167.stderr: 5 Oct 19:25:16 ntpd[22870]: can't open /var/log/ntpstats/rawstats.20231005: Permission denied 2023-10-05T19:25:16.405 INFO:teuthology.orchestra.run.smithi167.stderr: 5 Oct 19:25:16 ntpd[22870]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2023-10-05T19:25:16.405 INFO:teuthology.orchestra.run.smithi167.stderr: 5 Oct 19:25:16 ntpd[22870]: can't open /var/log/ntpstats/peerstats.20231005: Permission denied 2023-10-05T19:25:17.401 INFO:teuthology.orchestra.run.smithi118.stderr: 5 Oct 19:25:17 ntpd[22869]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-10-05T19:25:17.401 INFO:teuthology.orchestra.run.smithi118.stderr: 5 Oct 19:25:17 ntpd[22869]: can't open /var/log/ntpstats/rawstats.20231005: Permission denied 2023-10-05T19:25:17.401 INFO:teuthology.orchestra.run.smithi118.stderr: 5 Oct 19:25:17 ntpd[22869]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-10-05T19:25:17.401 INFO:teuthology.orchestra.run.smithi118.stderr: 5 Oct 19:25:17 ntpd[22869]: can't open /var/log/ntpstats/rawstats.20231005: Permission denied 2023-10-05T19:25:17.401 INFO:teuthology.orchestra.run.smithi118.stderr: 5 Oct 19:25:17 ntpd[22869]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2023-10-05T19:25:17.401 INFO:teuthology.orchestra.run.smithi118.stderr: 5 Oct 19:25:17 ntpd[22869]: can't open /var/log/ntpstats/peerstats.20231005: Permission denied 2023-10-05T19:25:18.404 INFO:teuthology.orchestra.run.smithi167.stderr: 5 Oct 19:25:18 ntpd[22870]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-10-05T19:25:18.404 INFO:teuthology.orchestra.run.smithi167.stderr: 5 Oct 19:25:18 ntpd[22870]: can't open /var/log/ntpstats/rawstats.20231005: Permission denied 2023-10-05T19:25:18.404 INFO:teuthology.orchestra.run.smithi167.stderr: 5 Oct 19:25:18 ntpd[22870]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-10-05T19:25:18.404 INFO:teuthology.orchestra.run.smithi167.stderr: 5 Oct 19:25:18 ntpd[22870]: can't open /var/log/ntpstats/rawstats.20231005: Permission denied 2023-10-05T19:25:18.404 INFO:teuthology.orchestra.run.smithi167.stderr: 5 Oct 19:25:18 ntpd[22870]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-10-05T19:25:18.404 INFO:teuthology.orchestra.run.smithi167.stderr: 5 Oct 19:25:18 ntpd[22870]: can't open /var/log/ntpstats/rawstats.20231005: Permission denied 2023-10-05T19:25:18.404 INFO:teuthology.orchestra.run.smithi167.stderr: 5 Oct 19:25:18 ntpd[22870]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2023-10-05T19:25:18.405 INFO:teuthology.orchestra.run.smithi167.stderr: 5 Oct 19:25:18 ntpd[22870]: can't open /var/log/ntpstats/peerstats.20231005: Permission denied 2023-10-05T19:25:18.405 INFO:teuthology.orchestra.run.smithi167.stderr: 5 Oct 19:25:18 ntpd[22870]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-10-05T19:25:18.405 INFO:teuthology.orchestra.run.smithi167.stderr: 5 Oct 19:25:18 ntpd[22870]: can't open /var/log/ntpstats/rawstats.20231005: Permission denied 2023-10-05T19:25:18.405 INFO:teuthology.orchestra.run.smithi167.stderr: 5 Oct 19:25:18 ntpd[22870]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2023-10-05T19:25:18.405 INFO:teuthology.orchestra.run.smithi167.stderr: 5 Oct 19:25:18 ntpd[22870]: can't open /var/log/ntpstats/peerstats.20231005: Permission denied 2023-10-05T19:25:19.401 INFO:teuthology.orchestra.run.smithi118.stderr: 5 Oct 19:25:19 ntpd[22869]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-10-05T19:25:19.401 INFO:teuthology.orchestra.run.smithi118.stderr: 5 Oct 19:25:19 ntpd[22869]: can't open /var/log/ntpstats/rawstats.20231005: Permission denied 2023-10-05T19:25:19.401 INFO:teuthology.orchestra.run.smithi118.stderr: 5 Oct 19:25:19 ntpd[22869]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-10-05T19:25:19.401 INFO:teuthology.orchestra.run.smithi118.stderr: 5 Oct 19:25:19 ntpd[22869]: can't open /var/log/ntpstats/rawstats.20231005: Permission denied 2023-10-05T19:25:19.401 INFO:teuthology.orchestra.run.smithi118.stderr: 5 Oct 19:25:19 ntpd[22869]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2023-10-05T19:25:19.402 INFO:teuthology.orchestra.run.smithi118.stderr: 5 Oct 19:25:19 ntpd[22869]: can't open /var/log/ntpstats/peerstats.20231005: Permission denied 2023-10-05T19:25:19.403 INFO:teuthology.orchestra.run.smithi167.stderr: 5 Oct 19:25:19 ntpd[22870]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-10-05T19:25:19.403 INFO:teuthology.orchestra.run.smithi167.stderr: 5 Oct 19:25:19 ntpd[22870]: can't open /var/log/ntpstats/rawstats.20231005: Permission denied 2023-10-05T19:25:19.403 INFO:teuthology.orchestra.run.smithi167.stderr: 5 Oct 19:25:19 ntpd[22870]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-10-05T19:25:19.404 INFO:teuthology.orchestra.run.smithi167.stderr: 5 Oct 19:25:19 ntpd[22870]: can't open /var/log/ntpstats/rawstats.20231005: Permission denied 2023-10-05T19:25:19.404 INFO:teuthology.orchestra.run.smithi167.stderr: 5 Oct 19:25:19 ntpd[22870]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2023-10-05T19:25:19.404 INFO:teuthology.orchestra.run.smithi167.stderr: 5 Oct 19:25:19 ntpd[22870]: can't open /var/log/ntpstats/peerstats.20231005: Permission denied 2023-10-05T19:25:20.402 INFO:teuthology.orchestra.run.smithi118.stderr: 5 Oct 19:25:20 ntpd[22869]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-10-05T19:25:20.422 INFO:teuthology.orchestra.run.smithi118.stderr: 5 Oct 19:25:20 ntpd[22869]: can't open /var/log/ntpstats/rawstats.20231005: Permission denied 2023-10-05T19:25:20.422 INFO:teuthology.orchestra.run.smithi118.stderr: 5 Oct 19:25:20 ntpd[22869]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-10-05T19:25:20.422 INFO:teuthology.orchestra.run.smithi118.stderr: 5 Oct 19:25:20 ntpd[22869]: can't open /var/log/ntpstats/rawstats.20231005: Permission denied 2023-10-05T19:25:20.422 INFO:teuthology.orchestra.run.smithi118.stderr: 5 Oct 19:25:20 ntpd[22869]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-10-05T19:25:20.423 INFO:teuthology.orchestra.run.smithi118.stderr: 5 Oct 19:25:20 ntpd[22869]: can't open /var/log/ntpstats/rawstats.20231005: Permission denied 2023-10-05T19:25:20.423 INFO:teuthology.orchestra.run.smithi118.stderr: 5 Oct 19:25:20 ntpd[22869]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2023-10-05T19:25:20.423 INFO:teuthology.orchestra.run.smithi118.stderr: 5 Oct 19:25:20 ntpd[22869]: can't open /var/log/ntpstats/peerstats.20231005: Permission denied 2023-10-05T19:25:20.423 INFO:teuthology.orchestra.run.smithi118.stderr: 5 Oct 19:25:20 ntpd[22869]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-10-05T19:25:20.423 INFO:teuthology.orchestra.run.smithi118.stderr: 5 Oct 19:25:20 ntpd[22869]: can't open /var/log/ntpstats/rawstats.20231005: Permission denied 2023-10-05T19:25:20.423 INFO:teuthology.orchestra.run.smithi118.stderr: 5 Oct 19:25:20 ntpd[22869]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2023-10-05T19:25:20.423 INFO:teuthology.orchestra.run.smithi118.stderr: 5 Oct 19:25:20 ntpd[22869]: can't open /var/log/ntpstats/peerstats.20231005: Permission denied 2023-10-05T19:25:20.424 INFO:teuthology.orchestra.run.smithi167.stderr: 5 Oct 19:25:20 ntpd[22870]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-10-05T19:25:20.424 INFO:teuthology.orchestra.run.smithi167.stderr: 5 Oct 19:25:20 ntpd[22870]: can't open /var/log/ntpstats/rawstats.20231005: Permission denied 2023-10-05T19:25:20.425 INFO:teuthology.orchestra.run.smithi167.stderr: 5 Oct 19:25:20 ntpd[22870]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-10-05T19:25:20.425 INFO:teuthology.orchestra.run.smithi167.stderr: 5 Oct 19:25:20 ntpd[22870]: can't open /var/log/ntpstats/rawstats.20231005: Permission denied 2023-10-05T19:25:20.425 INFO:teuthology.orchestra.run.smithi167.stderr: 5 Oct 19:25:20 ntpd[22870]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-10-05T19:25:20.425 INFO:teuthology.orchestra.run.smithi167.stderr: 5 Oct 19:25:20 ntpd[22870]: can't open /var/log/ntpstats/rawstats.20231005: Permission denied 2023-10-05T19:25:20.425 INFO:teuthology.orchestra.run.smithi167.stderr: 5 Oct 19:25:20 ntpd[22870]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2023-10-05T19:25:20.425 INFO:teuthology.orchestra.run.smithi167.stderr: 5 Oct 19:25:20 ntpd[22870]: can't open /var/log/ntpstats/peerstats.20231005: Permission denied 2023-10-05T19:25:20.425 INFO:teuthology.orchestra.run.smithi167.stderr: 5 Oct 19:25:20 ntpd[22870]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-10-05T19:25:20.425 INFO:teuthology.orchestra.run.smithi167.stderr: 5 Oct 19:25:20 ntpd[22870]: can't open /var/log/ntpstats/rawstats.20231005: Permission denied 2023-10-05T19:25:20.425 INFO:teuthology.orchestra.run.smithi167.stderr: 5 Oct 19:25:20 ntpd[22870]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2023-10-05T19:25:20.425 INFO:teuthology.orchestra.run.smithi167.stderr: 5 Oct 19:25:20 ntpd[22870]: can't open /var/log/ntpstats/peerstats.20231005: Permission denied 2023-10-05T19:25:21.401 INFO:teuthology.orchestra.run.smithi118.stderr: 5 Oct 19:25:21 ntpd[22869]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-10-05T19:25:21.401 INFO:teuthology.orchestra.run.smithi118.stderr: 5 Oct 19:25:21 ntpd[22869]: can't open /var/log/ntpstats/rawstats.20231005: Permission denied 2023-10-05T19:25:21.401 INFO:teuthology.orchestra.run.smithi118.stderr: 5 Oct 19:25:21 ntpd[22869]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-10-05T19:25:21.401 INFO:teuthology.orchestra.run.smithi118.stderr: 5 Oct 19:25:21 ntpd[22869]: can't open /var/log/ntpstats/rawstats.20231005: Permission denied 2023-10-05T19:25:21.402 INFO:teuthology.orchestra.run.smithi118.stderr: 5 Oct 19:25:21 ntpd[22869]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2023-10-05T19:25:21.402 INFO:teuthology.orchestra.run.smithi118.stderr: 5 Oct 19:25:21 ntpd[22869]: can't open /var/log/ntpstats/peerstats.20231005: Permission denied 2023-10-05T19:25:21.403 INFO:teuthology.orchestra.run.smithi167.stderr: 5 Oct 19:25:21 ntpd[22870]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-10-05T19:25:21.403 INFO:teuthology.orchestra.run.smithi167.stderr: 5 Oct 19:25:21 ntpd[22870]: can't open /var/log/ntpstats/rawstats.20231005: Permission denied 2023-10-05T19:25:21.403 INFO:teuthology.orchestra.run.smithi167.stderr: 5 Oct 19:25:21 ntpd[22870]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-10-05T19:25:21.404 INFO:teuthology.orchestra.run.smithi167.stderr: 5 Oct 19:25:21 ntpd[22870]: can't open /var/log/ntpstats/rawstats.20231005: Permission denied 2023-10-05T19:25:21.404 INFO:teuthology.orchestra.run.smithi167.stderr: 5 Oct 19:25:21 ntpd[22870]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2023-10-05T19:25:21.404 INFO:teuthology.orchestra.run.smithi167.stderr: 5 Oct 19:25:21 ntpd[22870]: can't open /var/log/ntpstats/peerstats.20231005: Permission denied 2023-10-05T19:25:22.402 INFO:teuthology.orchestra.run.smithi118.stderr: 5 Oct 19:25:22 ntpd[22869]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-10-05T19:25:22.402 INFO:teuthology.orchestra.run.smithi118.stderr: 5 Oct 19:25:22 ntpd[22869]: can't open /var/log/ntpstats/rawstats.20231005: Permission denied 2023-10-05T19:25:22.402 INFO:teuthology.orchestra.run.smithi118.stderr: 5 Oct 19:25:22 ntpd[22869]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-10-05T19:25:22.402 INFO:teuthology.orchestra.run.smithi118.stderr: 5 Oct 19:25:22 ntpd[22869]: can't open /var/log/ntpstats/rawstats.20231005: Permission denied 2023-10-05T19:25:22.402 INFO:teuthology.orchestra.run.smithi118.stderr: 5 Oct 19:25:22 ntpd[22869]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-10-05T19:25:22.402 INFO:teuthology.orchestra.run.smithi118.stderr: 5 Oct 19:25:22 ntpd[22869]: can't open /var/log/ntpstats/rawstats.20231005: Permission denied 2023-10-05T19:25:22.402 INFO:teuthology.orchestra.run.smithi118.stderr: 5 Oct 19:25:22 ntpd[22869]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2023-10-05T19:25:22.403 INFO:teuthology.orchestra.run.smithi118.stderr: 5 Oct 19:25:22 ntpd[22869]: can't open /var/log/ntpstats/peerstats.20231005: Permission denied 2023-10-05T19:25:22.403 INFO:teuthology.orchestra.run.smithi118.stderr: 5 Oct 19:25:22 ntpd[22869]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-10-05T19:25:22.403 INFO:teuthology.orchestra.run.smithi118.stderr: 5 Oct 19:25:22 ntpd[22869]: can't open /var/log/ntpstats/rawstats.20231005: Permission denied 2023-10-05T19:25:22.403 INFO:teuthology.orchestra.run.smithi118.stderr: 5 Oct 19:25:22 ntpd[22869]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2023-10-05T19:25:22.403 INFO:teuthology.orchestra.run.smithi118.stderr: 5 Oct 19:25:22 ntpd[22869]: can't open /var/log/ntpstats/peerstats.20231005: Permission denied 2023-10-05T19:25:22.404 INFO:teuthology.orchestra.run.smithi167.stderr: 5 Oct 19:25:22 ntpd[22870]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-10-05T19:25:22.404 INFO:teuthology.orchestra.run.smithi167.stderr: 5 Oct 19:25:22 ntpd[22870]: can't open /var/log/ntpstats/rawstats.20231005: Permission denied 2023-10-05T19:25:22.404 INFO:teuthology.orchestra.run.smithi167.stderr: 5 Oct 19:25:22 ntpd[22870]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-10-05T19:25:22.405 INFO:teuthology.orchestra.run.smithi167.stderr: 5 Oct 19:25:22 ntpd[22870]: can't open /var/log/ntpstats/rawstats.20231005: Permission denied 2023-10-05T19:25:22.405 INFO:teuthology.orchestra.run.smithi167.stderr: 5 Oct 19:25:22 ntpd[22870]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-10-05T19:25:22.405 INFO:teuthology.orchestra.run.smithi167.stderr: 5 Oct 19:25:22 ntpd[22870]: can't open /var/log/ntpstats/rawstats.20231005: Permission denied 2023-10-05T19:25:22.405 INFO:teuthology.orchestra.run.smithi167.stderr: 5 Oct 19:25:22 ntpd[22870]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2023-10-05T19:25:22.405 INFO:teuthology.orchestra.run.smithi167.stderr: 5 Oct 19:25:22 ntpd[22870]: can't open /var/log/ntpstats/peerstats.20231005: Permission denied 2023-10-05T19:25:22.405 INFO:teuthology.orchestra.run.smithi167.stdout: 5 Oct 19:25:22 ntpd[22870]: ntpd: time slew +0.000930 s 2023-10-05T19:25:22.405 INFO:teuthology.orchestra.run.smithi167.stdout:ntpd: time slew +0.000930s 2023-10-05T19:25:22.406 INFO:teuthology.orchestra.run.smithi167.stderr: 5 Oct 19:25:22 ntpd[22870]: couldn't unlink /var/log/ntpstats/loopstats: Permission denied 2023-10-05T19:25:22.406 INFO:teuthology.orchestra.run.smithi167.stderr: 5 Oct 19:25:22 ntpd[22870]: can't open /var/log/ntpstats/loopstats.20231005: Permission denied 2023-10-05T19:25:22.478 INFO:teuthology.orchestra.run.smithi167.stdout: remote refid st t when poll reach delay offset jitter 2023-10-05T19:25:22.478 INFO:teuthology.orchestra.run.smithi167.stdout:============================================================================== 2023-10-05T19:25:22.478 INFO:teuthology.orchestra.run.smithi167.stdout: hv01.front.sepi .INIT. 16 u - 64 0 0.000 0.000 0.000 2023-10-05T19:25:22.478 INFO:teuthology.orchestra.run.smithi167.stdout: hv02.front.sepi .INIT. 16 u - 64 0 0.000 0.000 0.000 2023-10-05T19:25:22.478 INFO:teuthology.orchestra.run.smithi167.stdout: hv03.front.sepi .INIT. 16 u - 64 0 0.000 0.000 0.000 2023-10-05T19:25:22.479 INFO:teuthology.orchestra.run.smithi167.stdout: hv04.front.sepi .INIT. 16 u - 64 0 0.000 0.000 0.000 2023-10-05T19:25:23.401 INFO:teuthology.orchestra.run.smithi118.stderr: 5 Oct 19:25:23 ntpd[22869]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-10-05T19:25:23.401 INFO:teuthology.orchestra.run.smithi118.stderr: 5 Oct 19:25:23 ntpd[22869]: can't open /var/log/ntpstats/rawstats.20231005: Permission denied 2023-10-05T19:25:23.401 INFO:teuthology.orchestra.run.smithi118.stderr: 5 Oct 19:25:23 ntpd[22869]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-10-05T19:25:23.401 INFO:teuthology.orchestra.run.smithi118.stderr: 5 Oct 19:25:23 ntpd[22869]: can't open /var/log/ntpstats/rawstats.20231005: Permission denied 2023-10-05T19:25:23.401 INFO:teuthology.orchestra.run.smithi118.stderr: 5 Oct 19:25:23 ntpd[22869]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2023-10-05T19:25:23.401 INFO:teuthology.orchestra.run.smithi118.stderr: 5 Oct 19:25:23 ntpd[22869]: can't open /var/log/ntpstats/peerstats.20231005: Permission denied 2023-10-05T19:25:23.402 INFO:teuthology.orchestra.run.smithi118.stdout: 5 Oct 19:25:23 ntpd[22869]: ntpd: time slew +0.000242 s 2023-10-05T19:25:23.402 INFO:teuthology.orchestra.run.smithi118.stdout:ntpd: time slew +0.000242s 2023-10-05T19:25:23.402 INFO:teuthology.orchestra.run.smithi118.stderr: 5 Oct 19:25:23 ntpd[22869]: couldn't unlink /var/log/ntpstats/loopstats: Permission denied 2023-10-05T19:25:23.402 INFO:teuthology.orchestra.run.smithi118.stderr: 5 Oct 19:25:23 ntpd[22869]: can't open /var/log/ntpstats/loopstats.20231005: Permission denied 2023-10-05T19:25:23.473 INFO:teuthology.orchestra.run.smithi118.stdout: remote refid st t when poll reach delay offset jitter 2023-10-05T19:25:23.473 INFO:teuthology.orchestra.run.smithi118.stdout:============================================================================== 2023-10-05T19:25:23.473 INFO:teuthology.orchestra.run.smithi118.stdout: hv01.front.sepi .INIT. 16 u - 64 0 0.000 0.000 0.000 2023-10-05T19:25:23.474 INFO:teuthology.orchestra.run.smithi118.stdout: hv02.front.sepi .INIT. 16 u - 64 0 0.000 0.000 0.000 2023-10-05T19:25:23.474 INFO:teuthology.orchestra.run.smithi118.stdout: hv03.front.sepi .INIT. 16 u - 64 0 0.000 0.000 0.000 2023-10-05T19:25:23.474 INFO:teuthology.orchestra.run.smithi118.stdout: hv04.front.sepi .INIT. 16 u - 64 0 0.000 0.000 0.000 2023-10-05T19:25:23.475 INFO:teuthology.run_tasks:Running task install... 2023-10-05T19:25:23.485 DEBUG:teuthology.task.install:project ceph 2023-10-05T19:25:23.485 DEBUG:teuthology.task.install:INSTALL overrides: {'ceph': {'flavor': 'default', 'sha1': '487ffcb5ddb60d29c5688d1a457e812b1b697b90'}} 2023-10-05T19:25:23.485 DEBUG:teuthology.task.install:config {'flavor': 'default', 'sha1': '487ffcb5ddb60d29c5688d1a457e812b1b697b90'} 2023-10-05T19:25:23.485 INFO:teuthology.task.install:Using flavor: default 2023-10-05T19:25:23.491 DEBUG:teuthology.task.install:Package list is: {'deb': ['ceph', 'cephadm', 'ceph-mds', 'ceph-mgr', 'ceph-common', 'ceph-fuse', 'ceph-test', 'ceph-volume', 'radosgw', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'libcephfs2', 'libcephfs-dev', 'librados2', 'librbd1', 'rbd-fuse'], 'rpm': ['ceph-radosgw', 'ceph-test', 'ceph', 'ceph-base', 'cephadm', 'ceph-immutable-object-cache', 'ceph-mgr', 'ceph-mgr-dashboard', 'ceph-mgr-diskprediction-local', 'ceph-mgr-rook', 'ceph-mgr-cephadm', 'ceph-fuse', 'ceph-volume', 'librados-devel', 'libcephfs2', 'libcephfs-devel', 'librados2', 'librbd1', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'rbd-fuse', 'rbd-mirror', 'rbd-nbd']} 2023-10-05T19:25:23.491 INFO:teuthology.task.install:extra packages: [] 2023-10-05T19:25:23.491 DEBUG:teuthology.orchestra.run.smithi118:> sudo apt-key list | grep Ceph 2023-10-05T19:25:23.491 DEBUG:teuthology.orchestra.run.smithi167:> sudo apt-key list | grep Ceph 2023-10-05T19:25:23.588 INFO:teuthology.orchestra.run.smithi167.stderr:Warning: apt-key output should not be parsed (stdout is not a terminal) 2023-10-05T19:25:23.615 INFO:teuthology.orchestra.run.smithi118.stderr:Warning: apt-key output should not be parsed (stdout is not a terminal) 2023-10-05T19:25:23.651 INFO:teuthology.orchestra.run.smithi167.stdout:uid [ unknown] Ceph automated package build (Ceph automated package build) 2023-10-05T19:25:23.651 INFO:teuthology.orchestra.run.smithi167.stdout:uid [ unknown] Ceph.com (release key) 2023-10-05T19:25:23.652 INFO:teuthology.task.install.deb:Installing packages: ceph, cephadm, ceph-mds, ceph-mgr, ceph-common, ceph-fuse, ceph-test, ceph-volume, radosgw, python3-rados, python3-rgw, python3-cephfs, python3-rbd, libcephfs2, libcephfs-dev, librados2, librbd1, rbd-fuse on remote deb x86_64 2023-10-05T19:25:23.652 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=ubuntu%2F20.04%2Fx86_64&sha1=487ffcb5ddb60d29c5688d1a457e812b1b697b90 2023-10-05T19:25:23.677 INFO:teuthology.orchestra.run.smithi118.stdout:uid [ unknown] Ceph automated package build (Ceph automated package build) 2023-10-05T19:25:23.677 INFO:teuthology.orchestra.run.smithi118.stdout:uid [ unknown] Ceph.com (release key) 2023-10-05T19:25:23.678 INFO:teuthology.task.install.deb:Installing packages: ceph, cephadm, ceph-mds, ceph-mgr, ceph-common, ceph-fuse, ceph-test, ceph-volume, radosgw, python3-rados, python3-rgw, python3-cephfs, python3-rbd, libcephfs2, libcephfs-dev, librados2, librbd1, rbd-fuse on remote deb x86_64 2023-10-05T19:25:23.678 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=ubuntu%2F20.04%2Fx86_64&sha1=487ffcb5ddb60d29c5688d1a457e812b1b697b90 2023-10-05T19:25:23.880 INFO:teuthology.task.install.deb:Pulling from https://3.chacra.ceph.com/r/ceph/wip-yuri2-testing-2023-10-04-1333-quincy/487ffcb5ddb60d29c5688d1a457e812b1b697b90/ubuntu/focal/flavors/default/ 2023-10-05T19:25:23.880 INFO:teuthology.task.install.deb:Package version is 17.2.6-1278-g487ffcb5-1focal 2023-10-05T19:25:23.893 INFO:teuthology.task.install.deb:Pulling from https://3.chacra.ceph.com/r/ceph/wip-yuri2-testing-2023-10-04-1333-quincy/487ffcb5ddb60d29c5688d1a457e812b1b697b90/ubuntu/focal/flavors/default/ 2023-10-05T19:25:23.893 INFO:teuthology.task.install.deb:Package version is 17.2.6-1278-g487ffcb5-1focal 2023-10-05T19:25:23.987 DEBUG:teuthology.orchestra.run.smithi167:> set -ex 2023-10-05T19:25:23.987 DEBUG:teuthology.orchestra.run.smithi167:> sudo dd of=/etc/apt/sources.list.d/ceph.list 2023-10-05T19:25:23.997 DEBUG:teuthology.orchestra.run.smithi118:> set -ex 2023-10-05T19:25:23.997 DEBUG:teuthology.orchestra.run.smithi118:> sudo dd of=/etc/apt/sources.list.d/ceph.list 2023-10-05T19:25:24.003 DEBUG:teuthology.orchestra.run.smithi167:> sudo apt-get update 2023-10-05T19:25:24.013 DEBUG:teuthology.orchestra.run.smithi118:> sudo apt-get update 2023-10-05T19:25:24.169 INFO:teuthology.orchestra.run.smithi167.stdout:Hit:1 http://archive.ubuntu.com/ubuntu focal InRelease 2023-10-05T19:25:24.177 INFO:teuthology.orchestra.run.smithi167.stdout:Hit:2 http://archive.ubuntu.com/ubuntu focal-updates InRelease 2023-10-05T19:25:24.197 INFO:teuthology.orchestra.run.smithi167.stdout:Hit:3 http://archive.ubuntu.com/ubuntu focal-backports InRelease 2023-10-05T19:25:24.202 INFO:teuthology.orchestra.run.smithi118.stdout:Hit:1 http://security.ubuntu.com/ubuntu focal-security InRelease 2023-10-05T19:25:24.259 INFO:teuthology.orchestra.run.smithi118.stdout:Ign:2 https://3.chacra.ceph.com/r/ceph/wip-yuri2-testing-2023-10-04-1333-quincy/487ffcb5ddb60d29c5688d1a457e812b1b697b90/ubuntu/focal/flavors/default focal InRelease 2023-10-05T19:25:24.263 INFO:teuthology.orchestra.run.smithi167.stdout:Ign:4 https://3.chacra.ceph.com/r/ceph/wip-yuri2-testing-2023-10-04-1333-quincy/487ffcb5ddb60d29c5688d1a457e812b1b697b90/ubuntu/focal/flavors/default focal InRelease 2023-10-05T19:25:24.281 INFO:teuthology.orchestra.run.smithi118.stdout:Get:3 https://3.chacra.ceph.com/r/ceph/wip-yuri2-testing-2023-10-04-1333-quincy/487ffcb5ddb60d29c5688d1a457e812b1b697b90/ubuntu/focal/flavors/default focal Release [7,662 B] 2023-10-05T19:25:24.285 INFO:teuthology.orchestra.run.smithi167.stdout:Hit:5 http://security.ubuntu.com/ubuntu focal-security InRelease 2023-10-05T19:25:24.300 INFO:teuthology.orchestra.run.smithi167.stdout:Get:6 https://3.chacra.ceph.com/r/ceph/wip-yuri2-testing-2023-10-04-1333-quincy/487ffcb5ddb60d29c5688d1a457e812b1b697b90/ubuntu/focal/flavors/default focal Release [7,662 B] 2023-10-05T19:25:24.304 INFO:teuthology.orchestra.run.smithi118.stdout:Ign:4 https://3.chacra.ceph.com/r/ceph/wip-yuri2-testing-2023-10-04-1333-quincy/487ffcb5ddb60d29c5688d1a457e812b1b697b90/ubuntu/focal/flavors/default focal Release.gpg 2023-10-05T19:25:24.325 INFO:teuthology.orchestra.run.smithi118.stdout:Get:5 https://3.chacra.ceph.com/r/ceph/wip-yuri2-testing-2023-10-04-1333-quincy/487ffcb5ddb60d29c5688d1a457e812b1b697b90/ubuntu/focal/flavors/default focal/main i386 Packages [3,372 B] 2023-10-05T19:25:24.337 INFO:teuthology.orchestra.run.smithi167.stdout:Ign:7 https://3.chacra.ceph.com/r/ceph/wip-yuri2-testing-2023-10-04-1333-quincy/487ffcb5ddb60d29c5688d1a457e812b1b697b90/ubuntu/focal/flavors/default focal Release.gpg 2023-10-05T19:25:24.370 INFO:teuthology.orchestra.run.smithi118.stdout:Get:6 https://3.chacra.ceph.com/r/ceph/wip-yuri2-testing-2023-10-04-1333-quincy/487ffcb5ddb60d29c5688d1a457e812b1b697b90/ubuntu/focal/flavors/default focal/main amd64 Packages [17.4 kB] 2023-10-05T19:25:24.374 INFO:teuthology.orchestra.run.smithi167.stdout:Get:8 https://3.chacra.ceph.com/r/ceph/wip-yuri2-testing-2023-10-04-1333-quincy/487ffcb5ddb60d29c5688d1a457e812b1b697b90/ubuntu/focal/flavors/default focal/main i386 Packages [3,372 B] 2023-10-05T19:25:24.427 INFO:teuthology.orchestra.run.smithi167.stdout:Get:9 https://3.chacra.ceph.com/r/ceph/wip-yuri2-testing-2023-10-04-1333-quincy/487ffcb5ddb60d29c5688d1a457e812b1b697b90/ubuntu/focal/flavors/default focal/main amd64 Packages [17.4 kB] 2023-10-05T19:25:24.698 INFO:teuthology.orchestra.run.smithi118.stdout:Hit:7 http://archive.ubuntu.com/ubuntu focal InRelease 2023-10-05T19:25:24.718 INFO:teuthology.orchestra.run.smithi118.stdout:Hit:8 http://archive.ubuntu.com/ubuntu focal-updates InRelease 2023-10-05T19:25:24.737 INFO:teuthology.orchestra.run.smithi118.stdout:Hit:9 http://archive.ubuntu.com/ubuntu focal-backports InRelease 2023-10-05T19:25:24.818 INFO:teuthology.orchestra.run.smithi167.stdout:Fetched 28.4 kB in 1s (43.8 kB/s) 2023-10-05T19:25:25.230 INFO:teuthology.orchestra.run.smithi118.stdout:Fetched 28.4 kB in 1s (28.3 kB/s) 2023-10-05T19:25:25.846 INFO:teuthology.orchestra.run.smithi167.stdout:Reading package lists... 2023-10-05T19:25:25.872 DEBUG:teuthology.orchestra.run.smithi167:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y --force-yes -o Dpkg::Options::="--force-confdef" -o Dpkg::Options::="--force-confold" install ceph=17.2.6-1278-g487ffcb5-1focal cephadm=17.2.6-1278-g487ffcb5-1focal ceph-mds=17.2.6-1278-g487ffcb5-1focal ceph-mgr=17.2.6-1278-g487ffcb5-1focal ceph-common=17.2.6-1278-g487ffcb5-1focal ceph-fuse=17.2.6-1278-g487ffcb5-1focal ceph-test=17.2.6-1278-g487ffcb5-1focal ceph-volume=17.2.6-1278-g487ffcb5-1focal radosgw=17.2.6-1278-g487ffcb5-1focal python3-rados=17.2.6-1278-g487ffcb5-1focal python3-rgw=17.2.6-1278-g487ffcb5-1focal python3-cephfs=17.2.6-1278-g487ffcb5-1focal python3-rbd=17.2.6-1278-g487ffcb5-1focal libcephfs2=17.2.6-1278-g487ffcb5-1focal libcephfs-dev=17.2.6-1278-g487ffcb5-1focal librados2=17.2.6-1278-g487ffcb5-1focal librbd1=17.2.6-1278-g487ffcb5-1focal rbd-fuse=17.2.6-1278-g487ffcb5-1focal 2023-10-05T19:25:25.940 INFO:teuthology.orchestra.run.smithi167.stdout:Reading package lists... 2023-10-05T19:25:26.135 INFO:teuthology.orchestra.run.smithi167.stdout:Building dependency tree... 2023-10-05T19:25:26.136 INFO:teuthology.orchestra.run.smithi167.stdout:Reading state information... 2023-10-05T19:25:26.260 INFO:teuthology.orchestra.run.smithi118.stdout:Reading package lists... 2023-10-05T19:25:26.285 DEBUG:teuthology.orchestra.run.smithi118:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y --force-yes -o Dpkg::Options::="--force-confdef" -o Dpkg::Options::="--force-confold" install ceph=17.2.6-1278-g487ffcb5-1focal cephadm=17.2.6-1278-g487ffcb5-1focal ceph-mds=17.2.6-1278-g487ffcb5-1focal ceph-mgr=17.2.6-1278-g487ffcb5-1focal ceph-common=17.2.6-1278-g487ffcb5-1focal ceph-fuse=17.2.6-1278-g487ffcb5-1focal ceph-test=17.2.6-1278-g487ffcb5-1focal ceph-volume=17.2.6-1278-g487ffcb5-1focal radosgw=17.2.6-1278-g487ffcb5-1focal python3-rados=17.2.6-1278-g487ffcb5-1focal python3-rgw=17.2.6-1278-g487ffcb5-1focal python3-cephfs=17.2.6-1278-g487ffcb5-1focal python3-rbd=17.2.6-1278-g487ffcb5-1focal libcephfs2=17.2.6-1278-g487ffcb5-1focal libcephfs-dev=17.2.6-1278-g487ffcb5-1focal librados2=17.2.6-1278-g487ffcb5-1focal librbd1=17.2.6-1278-g487ffcb5-1focal rbd-fuse=17.2.6-1278-g487ffcb5-1focal 2023-10-05T19:25:26.306 INFO:teuthology.orchestra.run.smithi167.stdout:The following packages were automatically installed and are no longer required: 2023-10-05T19:25:26.307 INFO:teuthology.orchestra.run.smithi167.stdout: libboost-iostreams1.71.0 libboost-thread1.71.0 2023-10-05T19:25:26.307 INFO:teuthology.orchestra.run.smithi167.stdout:Use 'sudo apt autoremove' to remove them. 2023-10-05T19:25:26.307 INFO:teuthology.orchestra.run.smithi167.stdout:The following additional packages will be installed: 2023-10-05T19:25:26.307 INFO:teuthology.orchestra.run.smithi167.stdout: ceph-base ceph-mgr-cephadm ceph-mgr-dashboard ceph-mgr-diskprediction-local 2023-10-05T19:25:26.309 INFO:teuthology.orchestra.run.smithi167.stdout: ceph-mgr-k8sevents ceph-mgr-modules-core ceph-mon ceph-osd fonts-mathjax jq 2023-10-05T19:25:26.309 INFO:teuthology.orchestra.run.smithi167.stdout: libdouble-conversion3 libjq1 libjs-mathjax libjs-sphinxdoc libjs-underscore 2023-10-05T19:25:26.309 INFO:teuthology.orchestra.run.smithi167.stdout: liblbfgsb0 liblttng-ust0 liblua5.3-0 liboath0 libonig5 libpcre2-16-0 2023-10-05T19:25:26.309 INFO:teuthology.orchestra.run.smithi167.stdout: libqt5core5a libqt5dbus5 libqt5network5 librabbitmq4 libradosstriper1 2023-10-05T19:25:26.309 INFO:teuthology.orchestra.run.smithi167.stdout: librdkafka1 librgw2 libsqlite3-mod-ceph libthrift-0.13.0 nvme-cli 2023-10-05T19:25:26.309 INFO:teuthology.orchestra.run.smithi167.stdout: python-natsort-doc python-pastedeploy-tpl python3-atomicwrites python3-attr 2023-10-05T19:25:26.309 INFO:teuthology.orchestra.run.smithi167.stdout: python3-bcrypt python3-blinker python3-bs4 python3-cachetools 2023-10-05T19:25:26.310 INFO:teuthology.orchestra.run.smithi167.stdout: python3-ceph-argparse python3-ceph-common python3-cherrypy3 2023-10-05T19:25:26.310 INFO:teuthology.orchestra.run.smithi167.stdout: python3-cryptography python3-decorator python3-google-auth python3-html5lib 2023-10-05T19:25:26.310 INFO:teuthology.orchestra.run.smithi167.stdout: python3-jinja2 python3-joblib python3-jwt python3-kubernetes 2023-10-05T19:25:26.310 INFO:teuthology.orchestra.run.smithi167.stdout: python3-logutils python3-lxml python3-mako python3-markupsafe 2023-10-05T19:25:26.310 INFO:teuthology.orchestra.run.smithi167.stdout: python3-natsort python3-oauthlib python3-openssl python3-packaging 2023-10-05T19:25:26.310 INFO:teuthology.orchestra.run.smithi167.stdout: python3-paste python3-pastedeploy python3-pastescript python3-pecan 2023-10-05T19:25:26.310 INFO:teuthology.orchestra.run.smithi167.stdout: python3-pluggy python3-prettytable python3-psutil python3-py python3-pyasn1 2023-10-05T19:25:26.310 INFO:teuthology.orchestra.run.smithi167.stdout: python3-pyasn1-modules python3-pyinotify python3-pytest python3-repoze.lru 2023-10-05T19:25:26.310 INFO:teuthology.orchestra.run.smithi167.stdout: python3-requests-oauthlib python3-routes python3-rsa python3-scipy 2023-10-05T19:25:26.310 INFO:teuthology.orchestra.run.smithi167.stdout: python3-simplegeneric python3-simplejson python3-singledispatch 2023-10-05T19:25:26.311 INFO:teuthology.orchestra.run.smithi167.stdout: python3-sklearn python3-sklearn-lib python3-soupsieve python3-tempita 2023-10-05T19:25:26.311 INFO:teuthology.orchestra.run.smithi167.stdout: python3-waitress python3-wcwidth python3-webencodings python3-webob 2023-10-05T19:25:26.311 INFO:teuthology.orchestra.run.smithi167.stdout: python3-websocket python3-webtest python3-werkzeug qttranslations5-l10n 2023-10-05T19:25:26.311 INFO:teuthology.orchestra.run.smithi167.stdout: socat xmlstarlet 2023-10-05T19:25:26.311 INFO:teuthology.orchestra.run.smithi167.stdout:Suggested packages: 2023-10-05T19:25:26.311 INFO:teuthology.orchestra.run.smithi167.stdout: python3-influxdb fonts-mathjax-extras fonts-stix libjs-mathjax-doc 2023-10-05T19:25:26.311 INFO:teuthology.orchestra.run.smithi167.stdout: python-attr-doc python-blinker-doc python-cryptography-doc 2023-10-05T19:25:26.312 INFO:teuthology.orchestra.run.smithi167.stdout: python3-cryptography-vectors python3-genshi python-jinja2-doc python3-crypto 2023-10-05T19:25:26.312 INFO:teuthology.orchestra.run.smithi167.stdout: python3-lxml-dbg python-lxml-doc python3-beaker python-mako-doc 2023-10-05T19:25:26.312 INFO:teuthology.orchestra.run.smithi167.stdout: python-openssl-doc python3-openssl-dbg httpd-wsgi libapache2-mod-python 2023-10-05T19:25:26.312 INFO:teuthology.orchestra.run.smithi167.stdout: libapache2-mod-scgi libjs-mochikit python-pecan-doc python-psutil-doc 2023-10-05T19:25:26.312 INFO:teuthology.orchestra.run.smithi167.stdout: subversion python-pyinotify-doc python-scipy-doc python3-dap 2023-10-05T19:25:26.312 INFO:teuthology.orchestra.run.smithi167.stdout: python-sklearn-doc ipython3 python-waitress-doc python-webob-doc 2023-10-05T19:25:26.312 INFO:teuthology.orchestra.run.smithi167.stdout: python-webtest-doc python-werkzeug-doc python3-termcolor python3-watchdog 2023-10-05T19:25:26.312 INFO:teuthology.orchestra.run.smithi167.stdout:Recommended packages: 2023-10-05T19:25:26.312 INFO:teuthology.orchestra.run.smithi167.stdout: btrfs-tools 2023-10-05T19:25:26.359 INFO:teuthology.orchestra.run.smithi118.stdout:Reading package lists... 2023-10-05T19:25:26.422 INFO:teuthology.orchestra.run.smithi167.stdout:The following NEW packages will be installed: 2023-10-05T19:25:26.422 INFO:teuthology.orchestra.run.smithi167.stdout: ceph ceph-base ceph-common ceph-fuse ceph-mds ceph-mgr ceph-mgr-cephadm 2023-10-05T19:25:26.422 INFO:teuthology.orchestra.run.smithi167.stdout: ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-k8sevents 2023-10-05T19:25:26.422 INFO:teuthology.orchestra.run.smithi167.stdout: ceph-mgr-modules-core ceph-mon ceph-osd ceph-test ceph-volume cephadm 2023-10-05T19:25:26.424 INFO:teuthology.orchestra.run.smithi167.stdout: fonts-mathjax jq libcephfs-dev libcephfs2 libdouble-conversion3 libjq1 2023-10-05T19:25:26.424 INFO:teuthology.orchestra.run.smithi167.stdout: libjs-mathjax libjs-sphinxdoc libjs-underscore liblbfgsb0 liblttng-ust0 2023-10-05T19:25:26.424 INFO:teuthology.orchestra.run.smithi167.stdout: liblua5.3-0 liboath0 libonig5 libpcre2-16-0 libqt5core5a libqt5dbus5 2023-10-05T19:25:26.424 INFO:teuthology.orchestra.run.smithi167.stdout: libqt5network5 librabbitmq4 libradosstriper1 librdkafka1 librgw2 2023-10-05T19:25:26.424 INFO:teuthology.orchestra.run.smithi167.stdout: libsqlite3-mod-ceph libthrift-0.13.0 nvme-cli python-natsort-doc 2023-10-05T19:25:26.424 INFO:teuthology.orchestra.run.smithi167.stdout: python-pastedeploy-tpl python3-atomicwrites python3-attr python3-bcrypt 2023-10-05T19:25:26.425 INFO:teuthology.orchestra.run.smithi167.stdout: python3-blinker python3-bs4 python3-cachetools python3-ceph-argparse 2023-10-05T19:25:26.425 INFO:teuthology.orchestra.run.smithi167.stdout: python3-ceph-common python3-cephfs python3-cherrypy3 python3-cryptography 2023-10-05T19:25:26.425 INFO:teuthology.orchestra.run.smithi167.stdout: python3-decorator python3-google-auth python3-html5lib python3-jinja2 2023-10-05T19:25:26.425 INFO:teuthology.orchestra.run.smithi167.stdout: python3-joblib python3-jwt python3-kubernetes python3-logutils python3-lxml 2023-10-05T19:25:26.425 INFO:teuthology.orchestra.run.smithi167.stdout: python3-mako python3-markupsafe python3-natsort python3-oauthlib 2023-10-05T19:25:26.425 INFO:teuthology.orchestra.run.smithi167.stdout: python3-openssl python3-packaging python3-paste python3-pastedeploy 2023-10-05T19:25:26.425 INFO:teuthology.orchestra.run.smithi167.stdout: python3-pastescript python3-pecan python3-pluggy python3-prettytable 2023-10-05T19:25:26.425 INFO:teuthology.orchestra.run.smithi167.stdout: python3-psutil python3-py python3-pyasn1 python3-pyasn1-modules 2023-10-05T19:25:26.425 INFO:teuthology.orchestra.run.smithi167.stdout: python3-pyinotify python3-pytest python3-rados python3-rbd 2023-10-05T19:25:26.425 INFO:teuthology.orchestra.run.smithi167.stdout: python3-repoze.lru python3-requests-oauthlib python3-rgw python3-routes 2023-10-05T19:25:26.425 INFO:teuthology.orchestra.run.smithi167.stdout: python3-rsa python3-scipy python3-simplegeneric python3-simplejson 2023-10-05T19:25:26.425 INFO:teuthology.orchestra.run.smithi167.stdout: python3-singledispatch python3-sklearn python3-sklearn-lib python3-soupsieve 2023-10-05T19:25:26.426 INFO:teuthology.orchestra.run.smithi167.stdout: python3-tempita python3-waitress python3-wcwidth python3-webencodings 2023-10-05T19:25:26.426 INFO:teuthology.orchestra.run.smithi167.stdout: python3-webob python3-websocket python3-webtest python3-werkzeug 2023-10-05T19:25:26.426 INFO:teuthology.orchestra.run.smithi167.stdout: qttranslations5-l10n radosgw rbd-fuse socat xmlstarlet 2023-10-05T19:25:26.426 INFO:teuthology.orchestra.run.smithi167.stdout:The following packages will be upgraded: 2023-10-05T19:25:26.426 INFO:teuthology.orchestra.run.smithi167.stdout: librados2 librbd1 2023-10-05T19:25:26.560 INFO:teuthology.orchestra.run.smithi118.stdout:Building dependency tree... 2023-10-05T19:25:26.560 INFO:teuthology.orchestra.run.smithi118.stdout:Reading state information... 2023-10-05T19:25:26.650 INFO:teuthology.orchestra.run.smithi167.stdout:2 upgraded, 108 newly installed, 0 to remove and 253 not upgraded. 2023-10-05T19:25:26.650 INFO:teuthology.orchestra.run.smithi167.stdout:Need to get 210 MB of archives. 2023-10-05T19:25:26.650 INFO:teuthology.orchestra.run.smithi167.stdout:After this operation, 980 MB of additional disk space will be used. 2023-10-05T19:25:26.650 INFO:teuthology.orchestra.run.smithi167.stdout:Get:1 https://3.chacra.ceph.com/r/ceph/wip-yuri2-testing-2023-10-04-1333-quincy/487ffcb5ddb60d29c5688d1a457e812b1b697b90/ubuntu/focal/flavors/default focal/main amd64 librbd1 amd64 17.2.6-1278-g487ffcb5-1focal [3,501 kB] 2023-10-05T19:25:26.738 INFO:teuthology.orchestra.run.smithi118.stdout:The following packages were automatically installed and are no longer required: 2023-10-05T19:25:26.739 INFO:teuthology.orchestra.run.smithi118.stdout: libboost-iostreams1.71.0 libboost-thread1.71.0 2023-10-05T19:25:26.740 INFO:teuthology.orchestra.run.smithi118.stdout:Use 'sudo apt autoremove' to remove them. 2023-10-05T19:25:26.740 INFO:teuthology.orchestra.run.smithi118.stdout:The following additional packages will be installed: 2023-10-05T19:25:26.740 INFO:teuthology.orchestra.run.smithi118.stdout: ceph-base ceph-mgr-cephadm ceph-mgr-dashboard ceph-mgr-diskprediction-local 2023-10-05T19:25:26.741 INFO:teuthology.orchestra.run.smithi118.stdout: ceph-mgr-k8sevents ceph-mgr-modules-core ceph-mon ceph-osd fonts-mathjax jq 2023-10-05T19:25:26.741 INFO:teuthology.orchestra.run.smithi118.stdout: libdouble-conversion3 libjq1 libjs-mathjax libjs-sphinxdoc libjs-underscore 2023-10-05T19:25:26.741 INFO:teuthology.orchestra.run.smithi118.stdout: liblbfgsb0 liblttng-ust0 liblua5.3-0 liboath0 libonig5 libpcre2-16-0 2023-10-05T19:25:26.742 INFO:teuthology.orchestra.run.smithi118.stdout: libqt5core5a libqt5dbus5 libqt5network5 librabbitmq4 libradosstriper1 2023-10-05T19:25:26.742 INFO:teuthology.orchestra.run.smithi118.stdout: librdkafka1 librgw2 libsqlite3-mod-ceph libthrift-0.13.0 nvme-cli 2023-10-05T19:25:26.742 INFO:teuthology.orchestra.run.smithi118.stdout: python-natsort-doc python-pastedeploy-tpl python3-atomicwrites python3-attr 2023-10-05T19:25:26.742 INFO:teuthology.orchestra.run.smithi118.stdout: python3-bcrypt python3-blinker python3-bs4 python3-cachetools 2023-10-05T19:25:26.742 INFO:teuthology.orchestra.run.smithi118.stdout: python3-ceph-argparse python3-ceph-common python3-cherrypy3 2023-10-05T19:25:26.742 INFO:teuthology.orchestra.run.smithi118.stdout: python3-cryptography python3-decorator python3-google-auth python3-html5lib 2023-10-05T19:25:26.742 INFO:teuthology.orchestra.run.smithi118.stdout: python3-jinja2 python3-joblib python3-jwt python3-kubernetes 2023-10-05T19:25:26.742 INFO:teuthology.orchestra.run.smithi118.stdout: python3-logutils python3-lxml python3-mako python3-markupsafe 2023-10-05T19:25:26.742 INFO:teuthology.orchestra.run.smithi118.stdout: python3-natsort python3-oauthlib python3-openssl python3-packaging 2023-10-05T19:25:26.742 INFO:teuthology.orchestra.run.smithi118.stdout: python3-paste python3-pastedeploy python3-pastescript python3-pecan 2023-10-05T19:25:26.742 INFO:teuthology.orchestra.run.smithi118.stdout: python3-pluggy python3-prettytable python3-psutil python3-py python3-pyasn1 2023-10-05T19:25:26.742 INFO:teuthology.orchestra.run.smithi118.stdout: python3-pyasn1-modules python3-pyinotify python3-pytest python3-repoze.lru 2023-10-05T19:25:26.743 INFO:teuthology.orchestra.run.smithi118.stdout: python3-requests-oauthlib python3-routes python3-rsa python3-scipy 2023-10-05T19:25:26.743 INFO:teuthology.orchestra.run.smithi118.stdout: python3-simplegeneric python3-simplejson python3-singledispatch 2023-10-05T19:25:26.743 INFO:teuthology.orchestra.run.smithi118.stdout: python3-sklearn python3-sklearn-lib python3-soupsieve python3-tempita 2023-10-05T19:25:26.743 INFO:teuthology.orchestra.run.smithi118.stdout: python3-waitress python3-wcwidth python3-webencodings python3-webob 2023-10-05T19:25:26.743 INFO:teuthology.orchestra.run.smithi118.stdout: python3-websocket python3-webtest python3-werkzeug qttranslations5-l10n 2023-10-05T19:25:26.743 INFO:teuthology.orchestra.run.smithi118.stdout: socat xmlstarlet 2023-10-05T19:25:26.743 INFO:teuthology.orchestra.run.smithi118.stdout:Suggested packages: 2023-10-05T19:25:26.744 INFO:teuthology.orchestra.run.smithi118.stdout: python3-influxdb fonts-mathjax-extras fonts-stix libjs-mathjax-doc 2023-10-05T19:25:26.744 INFO:teuthology.orchestra.run.smithi118.stdout: python-attr-doc python-blinker-doc python-cryptography-doc 2023-10-05T19:25:26.744 INFO:teuthology.orchestra.run.smithi118.stdout: python3-cryptography-vectors python3-genshi python-jinja2-doc python3-crypto 2023-10-05T19:25:26.744 INFO:teuthology.orchestra.run.smithi118.stdout: python3-lxml-dbg python-lxml-doc python3-beaker python-mako-doc 2023-10-05T19:25:26.744 INFO:teuthology.orchestra.run.smithi118.stdout: python-openssl-doc python3-openssl-dbg httpd-wsgi libapache2-mod-python 2023-10-05T19:25:26.744 INFO:teuthology.orchestra.run.smithi118.stdout: libapache2-mod-scgi libjs-mochikit python-pecan-doc python-psutil-doc 2023-10-05T19:25:26.744 INFO:teuthology.orchestra.run.smithi118.stdout: subversion python-pyinotify-doc python-scipy-doc python3-dap 2023-10-05T19:25:26.744 INFO:teuthology.orchestra.run.smithi118.stdout: python-sklearn-doc ipython3 python-waitress-doc python-webob-doc 2023-10-05T19:25:26.744 INFO:teuthology.orchestra.run.smithi118.stdout: python-webtest-doc python-werkzeug-doc python3-termcolor python3-watchdog 2023-10-05T19:25:26.744 INFO:teuthology.orchestra.run.smithi118.stdout:Recommended packages: 2023-10-05T19:25:26.744 INFO:teuthology.orchestra.run.smithi118.stdout: btrfs-tools 2023-10-05T19:25:26.858 INFO:teuthology.orchestra.run.smithi118.stdout:The following NEW packages will be installed: 2023-10-05T19:25:26.858 INFO:teuthology.orchestra.run.smithi118.stdout: ceph ceph-base ceph-common ceph-fuse ceph-mds ceph-mgr ceph-mgr-cephadm 2023-10-05T19:25:26.858 INFO:teuthology.orchestra.run.smithi118.stdout: ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-k8sevents 2023-10-05T19:25:26.858 INFO:teuthology.orchestra.run.smithi118.stdout: ceph-mgr-modules-core ceph-mon ceph-osd ceph-test ceph-volume cephadm 2023-10-05T19:25:26.858 INFO:teuthology.orchestra.run.smithi118.stdout: fonts-mathjax jq libcephfs-dev libcephfs2 libdouble-conversion3 libjq1 2023-10-05T19:25:26.858 INFO:teuthology.orchestra.run.smithi118.stdout: libjs-mathjax libjs-sphinxdoc libjs-underscore liblbfgsb0 liblttng-ust0 2023-10-05T19:25:26.858 INFO:teuthology.orchestra.run.smithi118.stdout: liblua5.3-0 liboath0 libonig5 libpcre2-16-0 libqt5core5a libqt5dbus5 2023-10-05T19:25:26.858 INFO:teuthology.orchestra.run.smithi118.stdout: libqt5network5 librabbitmq4 libradosstriper1 librdkafka1 librgw2 2023-10-05T19:25:26.860 INFO:teuthology.orchestra.run.smithi118.stdout: libsqlite3-mod-ceph libthrift-0.13.0 nvme-cli python-natsort-doc 2023-10-05T19:25:26.860 INFO:teuthology.orchestra.run.smithi118.stdout: python-pastedeploy-tpl python3-atomicwrites python3-attr python3-bcrypt 2023-10-05T19:25:26.860 INFO:teuthology.orchestra.run.smithi118.stdout: python3-blinker python3-bs4 python3-cachetools python3-ceph-argparse 2023-10-05T19:25:26.860 INFO:teuthology.orchestra.run.smithi118.stdout: python3-ceph-common python3-cephfs python3-cherrypy3 python3-cryptography 2023-10-05T19:25:26.860 INFO:teuthology.orchestra.run.smithi118.stdout: python3-decorator python3-google-auth python3-html5lib python3-jinja2 2023-10-05T19:25:26.860 INFO:teuthology.orchestra.run.smithi118.stdout: python3-joblib python3-jwt python3-kubernetes python3-logutils python3-lxml 2023-10-05T19:25:26.861 INFO:teuthology.orchestra.run.smithi118.stdout: python3-mako python3-markupsafe python3-natsort python3-oauthlib 2023-10-05T19:25:26.861 INFO:teuthology.orchestra.run.smithi118.stdout: python3-openssl python3-packaging python3-paste python3-pastedeploy 2023-10-05T19:25:26.861 INFO:teuthology.orchestra.run.smithi118.stdout: python3-pastescript python3-pecan python3-pluggy python3-prettytable 2023-10-05T19:25:26.861 INFO:teuthology.orchestra.run.smithi118.stdout: python3-psutil python3-py python3-pyasn1 python3-pyasn1-modules 2023-10-05T19:25:26.861 INFO:teuthology.orchestra.run.smithi118.stdout: python3-pyinotify python3-pytest python3-rados python3-rbd 2023-10-05T19:25:26.861 INFO:teuthology.orchestra.run.smithi118.stdout: python3-repoze.lru python3-requests-oauthlib python3-rgw python3-routes 2023-10-05T19:25:26.861 INFO:teuthology.orchestra.run.smithi118.stdout: python3-rsa python3-scipy python3-simplegeneric python3-simplejson 2023-10-05T19:25:26.861 INFO:teuthology.orchestra.run.smithi118.stdout: python3-singledispatch python3-sklearn python3-sklearn-lib python3-soupsieve 2023-10-05T19:25:26.861 INFO:teuthology.orchestra.run.smithi118.stdout: python3-tempita python3-waitress python3-wcwidth python3-webencodings 2023-10-05T19:25:26.861 INFO:teuthology.orchestra.run.smithi118.stdout: python3-webob python3-websocket python3-webtest python3-werkzeug 2023-10-05T19:25:26.861 INFO:teuthology.orchestra.run.smithi118.stdout: qttranslations5-l10n radosgw rbd-fuse socat xmlstarlet 2023-10-05T19:25:26.861 INFO:teuthology.orchestra.run.smithi118.stdout:The following packages will be upgraded: 2023-10-05T19:25:26.861 INFO:teuthology.orchestra.run.smithi118.stdout: librados2 librbd1 2023-10-05T19:25:26.924 INFO:teuthology.orchestra.run.smithi118.stdout:2 upgraded, 108 newly installed, 0 to remove and 253 not upgraded. 2023-10-05T19:25:26.924 INFO:teuthology.orchestra.run.smithi118.stdout:Need to get 210 MB of archives. 2023-10-05T19:25:26.924 INFO:teuthology.orchestra.run.smithi118.stdout:After this operation, 980 MB of additional disk space will be used. 2023-10-05T19:25:26.924 INFO:teuthology.orchestra.run.smithi118.stdout:Get:1 http://archive.ubuntu.com/ubuntu focal/main amd64 liblttng-ust0 amd64 2.11.0-1 [160 kB] 2023-10-05T19:25:27.007 INFO:teuthology.orchestra.run.smithi118.stdout:Get:2 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-prettytable all 0.7.2-5 [20.1 kB] 2023-10-05T19:25:27.008 INFO:teuthology.orchestra.run.smithi118.stdout:Get:3 http://archive.ubuntu.com/ubuntu focal/main amd64 liblua5.3-0 amd64 5.3.3-1.1ubuntu2 [116 kB] 2023-10-05T19:25:27.017 INFO:teuthology.orchestra.run.smithi118.stdout:Get:4 http://archive.ubuntu.com/ubuntu focal/main amd64 librabbitmq4 amd64 0.10.0-1 [35.8 kB] 2023-10-05T19:25:27.020 INFO:teuthology.orchestra.run.smithi118.stdout:Get:5 http://archive.ubuntu.com/ubuntu focal/universe amd64 librdkafka1 amd64 1.2.1-1ubuntu1 [410 kB] 2023-10-05T19:25:27.047 INFO:teuthology.orchestra.run.smithi118.stdout:Get:6 http://archive.ubuntu.com/ubuntu focal/universe amd64 libdouble-conversion3 amd64 3.1.5-4ubuntu1 [37.9 kB] 2023-10-05T19:25:27.048 INFO:teuthology.orchestra.run.smithi118.stdout:Get:7 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 libpcre2-16-0 amd64 10.34-7ubuntu0.1 [181 kB] 2023-10-05T19:25:27.054 INFO:teuthology.orchestra.run.smithi118.stdout:Get:8 http://archive.ubuntu.com/ubuntu focal-updates/universe amd64 libqt5core5a amd64 5.12.8+dfsg-0ubuntu2.1 [2,006 kB] 2023-10-05T19:25:27.085 INFO:teuthology.orchestra.run.smithi118.stdout:Get:9 http://archive.ubuntu.com/ubuntu focal-updates/universe amd64 libqt5dbus5 amd64 5.12.8+dfsg-0ubuntu2.1 [208 kB] 2023-10-05T19:25:27.086 INFO:teuthology.orchestra.run.smithi118.stdout:Get:10 http://archive.ubuntu.com/ubuntu focal-updates/universe amd64 libqt5network5 amd64 5.12.8+dfsg-0ubuntu2.1 [673 kB] 2023-10-05T19:25:27.088 INFO:teuthology.orchestra.run.smithi118.stdout:Get:11 https://3.chacra.ceph.com/r/ceph/wip-yuri2-testing-2023-10-04-1333-quincy/487ffcb5ddb60d29c5688d1a457e812b1b697b90/ubuntu/focal/flavors/default focal/main amd64 librbd1 amd64 17.2.6-1278-g487ffcb5-1focal [3,501 kB] 2023-10-05T19:25:27.092 INFO:teuthology.orchestra.run.smithi118.stdout:Get:12 http://archive.ubuntu.com/ubuntu focal/universe amd64 libthrift-0.13.0 amd64 0.13.0-2build2 [249 kB] 2023-10-05T19:25:27.094 INFO:teuthology.orchestra.run.smithi118.stdout:Get:13 http://archive.ubuntu.com/ubuntu focal/main amd64 liboath0 amd64 2.6.1-1.3 [44.7 kB] 2023-10-05T19:25:27.095 INFO:teuthology.orchestra.run.smithi118.stdout:Get:14 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 python3-cryptography amd64 2.8-3ubuntu0.1 [211 kB] 2023-10-05T19:25:27.097 INFO:teuthology.orchestra.run.smithi118.stdout:Get:15 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-openssl all 19.0.0-1build1 [43.3 kB] 2023-10-05T19:25:27.097 INFO:teuthology.orchestra.run.smithi118.stdout:Get:16 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-bcrypt amd64 3.1.7-2ubuntu1 [30.4 kB] 2023-10-05T19:25:27.097 INFO:teuthology.orchestra.run.smithi118.stdout:Get:17 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-cherrypy3 all 8.9.1-6 [160 kB] 2023-10-05T19:25:27.099 INFO:teuthology.orchestra.run.smithi118.stdout:Get:18 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-natsort all 7.0.1-1 [29.2 kB] 2023-10-05T19:25:27.102 INFO:teuthology.orchestra.run.smithi118.stdout:Get:19 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 python3-jwt all 1.7.1-2ubuntu2.1 [18.0 kB] 2023-10-05T19:25:27.116 INFO:teuthology.orchestra.run.smithi118.stdout:Get:20 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-logutils all 0.3.3-7 [17.3 kB] 2023-10-05T19:25:27.117 INFO:teuthology.orchestra.run.smithi118.stdout:Get:21 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-markupsafe amd64 1.1.0-1build2 [13.9 kB] 2023-10-05T19:25:27.117 INFO:teuthology.orchestra.run.smithi118.stdout:Get:22 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 python3-mako all 1.1.0+ds1-1ubuntu2.1 [59.2 kB] 2023-10-05T19:25:27.117 INFO:teuthology.orchestra.run.smithi167.stdout:Get:2 http://archive.ubuntu.com/ubuntu focal/main amd64 liblttng-ust0 amd64 2.11.0-1 [160 kB] 2023-10-05T19:25:27.118 INFO:teuthology.orchestra.run.smithi118.stdout:Get:23 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-simplegeneric all 0.8.1-3 [11.3 kB] 2023-10-05T19:25:27.118 INFO:teuthology.orchestra.run.smithi118.stdout:Get:24 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-singledispatch all 3.4.0.3-2build1 [7,128 B] 2023-10-05T19:25:27.119 INFO:teuthology.orchestra.run.smithi118.stdout:Get:25 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-webob all 1:1.8.5-2 [84.4 kB] 2023-10-05T19:25:27.120 INFO:teuthology.orchestra.run.smithi118.stdout:Get:26 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-soupsieve all 1.9.5+dfsg-1 [29.1 kB] 2023-10-05T19:25:27.120 INFO:teuthology.orchestra.run.smithi118.stdout:Get:27 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-bs4 all 4.8.2-1 [83.0 kB] 2023-10-05T19:25:27.122 INFO:teuthology.orchestra.run.smithi118.stdout:Get:28 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 python3-waitress all 1.4.1-1ubuntu0.1 [81.4 kB] 2023-10-05T19:25:27.136 INFO:teuthology.orchestra.run.smithi118.stdout:Get:29 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-tempita all 0.5.2-6 [14.8 kB] 2023-10-05T19:25:27.136 INFO:teuthology.orchestra.run.smithi118.stdout:Get:30 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-paste all 3.4.0+dfsg1-1 [456 kB] 2023-10-05T19:25:27.143 INFO:teuthology.orchestra.run.smithi118.stdout:Get:31 http://archive.ubuntu.com/ubuntu focal/main amd64 python-pastedeploy-tpl all 2.1.0-1 [5,016 B] 2023-10-05T19:25:27.143 INFO:teuthology.orchestra.run.smithi118.stdout:Get:32 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-pastedeploy all 2.1.0-1 [26.5 kB] 2023-10-05T19:25:27.152 INFO:teuthology.orchestra.run.smithi118.stdout:Get:33 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-webtest all 2.0.32-2 [28.2 kB] 2023-10-05T19:25:27.153 INFO:teuthology.orchestra.run.smithi118.stdout:Get:34 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-pecan all 1.3.3-3ubuntu1 [85.8 kB] 2023-10-05T19:25:27.154 INFO:teuthology.orchestra.run.smithi118.stdout:Get:35 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 python3-werkzeug all 0.16.1+dfsg1-2ubuntu0.1 [183 kB] 2023-10-05T19:25:27.156 INFO:teuthology.orchestra.run.smithi118.stdout:Get:36 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-jinja2 all 2.10.1-2 [95.5 kB] 2023-10-05T19:25:27.157 INFO:teuthology.orchestra.run.smithi118.stdout:Get:37 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-repoze.lru all 0.7-2 [12.1 kB] 2023-10-05T19:25:27.157 INFO:teuthology.orchestra.run.smithi118.stdout:Get:38 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 libjs-underscore all 1.9.1~dfsg-1ubuntu0.20.04.1 [99.5 kB] 2023-10-05T19:25:27.159 INFO:teuthology.orchestra.run.smithi167.stdout:Get:3 https://3.chacra.ceph.com/r/ceph/wip-yuri2-testing-2023-10-04-1333-quincy/487ffcb5ddb60d29c5688d1a457e812b1b697b90/ubuntu/focal/flavors/default focal/main amd64 librados2 amd64 17.2.6-1278-g487ffcb5-1focal [3,555 kB] 2023-10-05T19:25:27.169 INFO:teuthology.orchestra.run.smithi118.stdout:Get:39 http://archive.ubuntu.com/ubuntu focal/main amd64 libjs-sphinxdoc all 1.8.5-7ubuntu3 [97.1 kB] 2023-10-05T19:25:27.171 INFO:teuthology.orchestra.run.smithi118.stdout:Get:40 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-routes all 2.4.1-2 [88.1 kB] 2023-10-05T19:25:27.173 INFO:teuthology.orchestra.run.smithi118.stdout:Get:41 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-decorator all 4.4.2-0ubuntu1 [10.3 kB] 2023-10-05T19:25:27.173 INFO:teuthology.orchestra.run.smithi118.stdout:Get:42 http://archive.ubuntu.com/ubuntu focal/universe amd64 liblbfgsb0 amd64 3.0+dfsg.3-7build1 [29.5 kB] 2023-10-05T19:25:27.186 INFO:teuthology.orchestra.run.smithi118.stdout:Get:43 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-scipy amd64 1.3.3-3build1 [11.1 MB] 2023-10-05T19:25:27.268 INFO:teuthology.orchestra.run.smithi118.stdout:Get:44 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-sklearn-lib amd64 0.22.2.post1+dfsg-5 [2,358 kB] 2023-10-05T19:25:27.284 INFO:teuthology.orchestra.run.smithi118.stdout:Get:45 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-joblib all 0.14.0-3 [192 kB] 2023-10-05T19:25:27.347 INFO:teuthology.orchestra.run.smithi118.stdout:Get:46 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-sklearn all 0.22.2.post1+dfsg-5 [1,773 kB] 2023-10-05T19:25:27.365 INFO:teuthology.orchestra.run.smithi118.stdout:Get:47 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-cachetools all 4.0.0-1 [9,820 B] 2023-10-05T19:25:27.366 INFO:teuthology.orchestra.run.smithi118.stdout:Get:48 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-pyasn1 all 0.4.2-3build1 [46.7 kB] 2023-10-05T19:25:27.366 INFO:teuthology.orchestra.run.smithi118.stdout:Get:49 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-pyasn1-modules all 0.2.1-0.2build1 [32.9 kB] 2023-10-05T19:25:27.366 INFO:teuthology.orchestra.run.smithi118.stdout:Get:50 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-rsa all 4.0-3 [24.7 kB] 2023-10-05T19:25:27.367 INFO:teuthology.orchestra.run.smithi118.stdout:Get:51 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-google-auth all 1.5.1-2 [35.6 kB] 2023-10-05T19:25:27.367 INFO:teuthology.orchestra.run.smithi118.stdout:Get:52 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-blinker all 1.4+dfsg1-0.3ubuntu1 [13.2 kB] 2023-10-05T19:25:27.367 INFO:teuthology.orchestra.run.smithi118.stdout:Get:53 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-oauthlib all 3.1.0-1ubuntu2 [84.8 kB] 2023-10-05T19:25:27.368 INFO:teuthology.orchestra.run.smithi118.stdout:Get:54 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-requests-oauthlib all 1.0.0-1.1build1 [16.8 kB] 2023-10-05T19:25:27.368 INFO:teuthology.orchestra.run.smithi118.stdout:Get:55 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-websocket all 0.53.0-2ubuntu1 [32.3 kB] 2023-10-05T19:25:27.383 INFO:teuthology.orchestra.run.smithi118.stdout:Get:56 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-kubernetes all 7.0.0~a1-2 [271 kB] 2023-10-05T19:25:27.385 INFO:teuthology.orchestra.run.smithi118.stdout:Get:57 http://archive.ubuntu.com/ubuntu focal/universe amd64 libonig5 amd64 6.9.4-1 [142 kB] 2023-10-05T19:25:27.387 INFO:teuthology.orchestra.run.smithi118.stdout:Get:58 http://archive.ubuntu.com/ubuntu focal-updates/universe amd64 libjq1 amd64 1.6-1ubuntu0.20.04.1 [121 kB] 2023-10-05T19:25:27.400 INFO:teuthology.orchestra.run.smithi118.stdout:Get:59 http://archive.ubuntu.com/ubuntu focal-updates/universe amd64 jq amd64 1.6-1ubuntu0.20.04.1 [50.2 kB] 2023-10-05T19:25:27.401 INFO:teuthology.orchestra.run.smithi118.stdout:Get:60 http://archive.ubuntu.com/ubuntu focal/main amd64 socat amd64 1.7.3.3-2 [323 kB] 2023-10-05T19:25:27.405 INFO:teuthology.orchestra.run.smithi118.stdout:Get:61 http://archive.ubuntu.com/ubuntu focal/universe amd64 xmlstarlet amd64 1.6.1-2build1 [267 kB] 2023-10-05T19:25:27.408 INFO:teuthology.orchestra.run.smithi118.stdout:Get:62 http://archive.ubuntu.com/ubuntu focal/universe amd64 fonts-mathjax all 2.7.4+dfsg-1 [2,208 kB] 2023-10-05T19:25:27.416 INFO:teuthology.orchestra.run.smithi167.stdout:Get:4 https://3.chacra.ceph.com/r/ceph/wip-yuri2-testing-2023-10-04-1333-quincy/487ffcb5ddb60d29c5688d1a457e812b1b697b90/ubuntu/focal/flavors/default focal/main amd64 libcephfs2 amd64 17.2.6-1278-g487ffcb5-1focal [760 kB] 2023-10-05T19:25:27.424 INFO:teuthology.orchestra.run.smithi118.stdout:Get:63 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 nvme-cli amd64 1.9-1ubuntu0.1 [283 kB] 2023-10-05T19:25:27.427 INFO:teuthology.orchestra.run.smithi118.stdout:Get:64 http://archive.ubuntu.com/ubuntu focal/universe amd64 libjs-mathjax all 2.7.4+dfsg-1 [5,654 kB] 2023-10-05T19:25:27.456 INFO:teuthology.orchestra.run.smithi167.stdout:Get:5 https://3.chacra.ceph.com/r/ceph/wip-yuri2-testing-2023-10-04-1333-quincy/487ffcb5ddb60d29c5688d1a457e812b1b697b90/ubuntu/focal/flavors/default focal/main amd64 python3-rados amd64 17.2.6-1278-g487ffcb5-1focal [412 kB] 2023-10-05T19:25:27.463 INFO:teuthology.orchestra.run.smithi118.stdout:Get:65 http://archive.ubuntu.com/ubuntu focal/universe amd64 python-natsort-doc all 7.0.1-1 [103 kB] 2023-10-05T19:25:27.464 INFO:teuthology.orchestra.run.smithi118.stdout:Get:66 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-atomicwrites all 1.1.5-2build1 [7,016 B] 2023-10-05T19:25:27.464 INFO:teuthology.orchestra.run.smithi118.stdout:Get:67 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-attr all 19.3.0-2 [33.9 kB] 2023-10-05T19:25:27.465 INFO:teuthology.orchestra.run.smithi118.stdout:Get:68 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-webencodings all 0.5.1-1ubuntu1 [11.0 kB] 2023-10-05T19:25:27.465 INFO:teuthology.orchestra.run.smithi118.stdout:Get:69 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-html5lib all 1.0.1-2 [84.3 kB] 2023-10-05T19:25:27.466 INFO:teuthology.orchestra.run.smithi118.stdout:Get:70 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 python3-lxml amd64 4.5.0-1ubuntu0.5 [1,384 kB] 2023-10-05T19:25:27.477 INFO:teuthology.orchestra.run.smithi118.stdout:Get:71 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-packaging all 20.3-1 [26.8 kB] 2023-10-05T19:25:27.477 INFO:teuthology.orchestra.run.smithi118.stdout:Get:72 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-pastescript all 2.0.2-4 [54.6 kB] 2023-10-05T19:25:27.478 INFO:teuthology.orchestra.run.smithi118.stdout:Get:73 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-pluggy all 0.13.0-2 [18.4 kB] 2023-10-05T19:25:27.480 INFO:teuthology.orchestra.run.smithi118.stdout:Get:74 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-psutil amd64 5.5.1-1ubuntu4 [141 kB] 2023-10-05T19:25:27.487 INFO:teuthology.orchestra.run.smithi167.stdout:Get:6 https://3.chacra.ceph.com/r/ceph/wip-yuri2-testing-2023-10-04-1333-quincy/487ffcb5ddb60d29c5688d1a457e812b1b697b90/ubuntu/focal/flavors/default focal/main amd64 python3-ceph-argparse all 17.2.6-1278-g487ffcb5-1focal [22.5 kB] 2023-10-05T19:25:27.488 INFO:teuthology.orchestra.run.smithi167.stdout:Get:7 https://3.chacra.ceph.com/r/ceph/wip-yuri2-testing-2023-10-04-1333-quincy/487ffcb5ddb60d29c5688d1a457e812b1b697b90/ubuntu/focal/flavors/default focal/main amd64 python3-cephfs amd64 17.2.6-1278-g487ffcb5-1focal [200 kB] 2023-10-05T19:25:27.492 INFO:teuthology.orchestra.run.smithi167.stdout:Get:8 https://3.chacra.ceph.com/r/ceph/wip-yuri2-testing-2023-10-04-1333-quincy/487ffcb5ddb60d29c5688d1a457e812b1b697b90/ubuntu/focal/flavors/default focal/main amd64 python3-ceph-common all 17.2.6-1278-g487ffcb5-1focal [47.2 kB] 2023-10-05T19:25:27.496 INFO:teuthology.orchestra.run.smithi167.stdout:Get:9 https://3.chacra.ceph.com/r/ceph/wip-yuri2-testing-2023-10-04-1333-quincy/487ffcb5ddb60d29c5688d1a457e812b1b697b90/ubuntu/focal/flavors/default focal/main amd64 python3-rbd amd64 17.2.6-1278-g487ffcb5-1focal [386 kB] 2023-10-05T19:25:27.497 INFO:teuthology.orchestra.run.smithi118.stdout:Get:75 http://archive.ubuntu.com/ubuntu focal-updates/universe amd64 python3-py all 1.8.1-1ubuntu0.1 [65.6 kB] 2023-10-05T19:25:27.498 INFO:teuthology.orchestra.run.smithi118.stdout:Get:76 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-pyinotify all 0.9.6-1.2ubuntu1 [24.8 kB] 2023-10-05T19:25:27.499 INFO:teuthology.orchestra.run.smithi118.stdout:Get:77 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-wcwidth all 0.1.8+dfsg1-3 [17.4 kB] 2023-10-05T19:25:27.499 INFO:teuthology.orchestra.run.smithi118.stdout:Get:78 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-pytest all 4.6.9-1 [178 kB] 2023-10-05T19:25:27.501 INFO:teuthology.orchestra.run.smithi118.stdout:Get:79 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-simplejson amd64 3.16.0-2ubuntu2 [50.8 kB] 2023-10-05T19:25:27.502 INFO:teuthology.orchestra.run.smithi118.stdout:Get:80 http://archive.ubuntu.com/ubuntu focal/universe amd64 qttranslations5-l10n all 5.12.8-0ubuntu1 [1,486 kB] 2023-10-05T19:25:27.513 INFO:teuthology.orchestra.run.smithi167.stdout:Get:10 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-prettytable all 0.7.2-5 [20.1 kB] 2023-10-05T19:25:27.520 INFO:teuthology.orchestra.run.smithi167.stdout:Get:11 http://archive.ubuntu.com/ubuntu focal/main amd64 liblua5.3-0 amd64 5.3.3-1.1ubuntu2 [116 kB] 2023-10-05T19:25:27.524 INFO:teuthology.orchestra.run.smithi167.stdout:Get:12 https://3.chacra.ceph.com/r/ceph/wip-yuri2-testing-2023-10-04-1333-quincy/487ffcb5ddb60d29c5688d1a457e812b1b697b90/ubuntu/focal/flavors/default focal/main amd64 librgw2 amd64 17.2.6-1278-g487ffcb5-1focal [5,577 kB] 2023-10-05T19:25:27.564 INFO:teuthology.orchestra.run.smithi167.stdout:Get:13 http://archive.ubuntu.com/ubuntu focal/main amd64 librabbitmq4 amd64 0.10.0-1 [35.8 kB] 2023-10-05T19:25:27.578 INFO:teuthology.orchestra.run.smithi167.stdout:Get:14 http://archive.ubuntu.com/ubuntu focal/universe amd64 librdkafka1 amd64 1.2.1-1ubuntu1 [410 kB] 2023-10-05T19:25:27.599 INFO:teuthology.orchestra.run.smithi118.stdout:Get:81 https://3.chacra.ceph.com/r/ceph/wip-yuri2-testing-2023-10-04-1333-quincy/487ffcb5ddb60d29c5688d1a457e812b1b697b90/ubuntu/focal/flavors/default focal/main amd64 librados2 amd64 17.2.6-1278-g487ffcb5-1focal [3,555 kB] 2023-10-05T19:25:27.704 INFO:teuthology.orchestra.run.smithi167.stdout:Get:15 http://archive.ubuntu.com/ubuntu focal/universe amd64 libdouble-conversion3 amd64 3.1.5-4ubuntu1 [37.9 kB] 2023-10-05T19:25:27.709 INFO:teuthology.orchestra.run.smithi167.stdout:Get:16 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 libpcre2-16-0 amd64 10.34-7ubuntu0.1 [181 kB] 2023-10-05T19:25:27.739 INFO:teuthology.orchestra.run.smithi167.stdout:Get:17 http://archive.ubuntu.com/ubuntu focal-updates/universe amd64 libqt5core5a amd64 5.12.8+dfsg-0ubuntu2.1 [2,006 kB] 2023-10-05T19:25:27.817 INFO:teuthology.orchestra.run.smithi167.stdout:Get:18 https://3.chacra.ceph.com/r/ceph/wip-yuri2-testing-2023-10-04-1333-quincy/487ffcb5ddb60d29c5688d1a457e812b1b697b90/ubuntu/focal/flavors/default focal/main amd64 python3-rgw amd64 17.2.6-1278-g487ffcb5-1focal [116 kB] 2023-10-05T19:25:27.822 INFO:teuthology.orchestra.run.smithi167.stdout:Get:19 https://3.chacra.ceph.com/r/ceph/wip-yuri2-testing-2023-10-04-1333-quincy/487ffcb5ddb60d29c5688d1a457e812b1b697b90/ubuntu/focal/flavors/default focal/main amd64 libradosstriper1 amd64 17.2.6-1278-g487ffcb5-1focal [479 kB] 2023-10-05T19:25:27.837 INFO:teuthology.orchestra.run.smithi167.stdout:Get:20 https://3.chacra.ceph.com/r/ceph/wip-yuri2-testing-2023-10-04-1333-quincy/487ffcb5ddb60d29c5688d1a457e812b1b697b90/ubuntu/focal/flavors/default focal/main amd64 ceph-common amd64 17.2.6-1278-g487ffcb5-1focal [27.7 MB] 2023-10-05T19:25:27.880 INFO:teuthology.orchestra.run.smithi167.stdout:Get:21 http://archive.ubuntu.com/ubuntu focal-updates/universe amd64 libqt5dbus5 amd64 5.12.8+dfsg-0ubuntu2.1 [208 kB] 2023-10-05T19:25:27.888 INFO:teuthology.orchestra.run.smithi167.stdout:Get:22 http://archive.ubuntu.com/ubuntu focal-updates/universe amd64 libqt5network5 amd64 5.12.8+dfsg-0ubuntu2.1 [673 kB] 2023-10-05T19:25:27.908 INFO:teuthology.orchestra.run.smithi118.stdout:Get:82 https://3.chacra.ceph.com/r/ceph/wip-yuri2-testing-2023-10-04-1333-quincy/487ffcb5ddb60d29c5688d1a457e812b1b697b90/ubuntu/focal/flavors/default focal/main amd64 libcephfs2 amd64 17.2.6-1278-g487ffcb5-1focal [760 kB] 2023-10-05T19:25:27.911 INFO:teuthology.orchestra.run.smithi167.stdout:Get:23 http://archive.ubuntu.com/ubuntu focal/universe amd64 libthrift-0.13.0 amd64 0.13.0-2build2 [249 kB] 2023-10-05T19:25:27.967 INFO:teuthology.orchestra.run.smithi118.stdout:Get:83 https://3.chacra.ceph.com/r/ceph/wip-yuri2-testing-2023-10-04-1333-quincy/487ffcb5ddb60d29c5688d1a457e812b1b697b90/ubuntu/focal/flavors/default focal/main amd64 python3-rados amd64 17.2.6-1278-g487ffcb5-1focal [412 kB] 2023-10-05T19:25:28.016 INFO:teuthology.orchestra.run.smithi118.stdout:Get:84 https://3.chacra.ceph.com/r/ceph/wip-yuri2-testing-2023-10-04-1333-quincy/487ffcb5ddb60d29c5688d1a457e812b1b697b90/ubuntu/focal/flavors/default focal/main amd64 python3-ceph-argparse all 17.2.6-1278-g487ffcb5-1focal [22.5 kB] 2023-10-05T19:25:28.017 INFO:teuthology.orchestra.run.smithi118.stdout:Get:85 https://3.chacra.ceph.com/r/ceph/wip-yuri2-testing-2023-10-04-1333-quincy/487ffcb5ddb60d29c5688d1a457e812b1b697b90/ubuntu/focal/flavors/default focal/main amd64 python3-cephfs amd64 17.2.6-1278-g487ffcb5-1focal [200 kB] 2023-10-05T19:25:28.023 INFO:teuthology.orchestra.run.smithi118.stdout:Get:86 https://3.chacra.ceph.com/r/ceph/wip-yuri2-testing-2023-10-04-1333-quincy/487ffcb5ddb60d29c5688d1a457e812b1b697b90/ubuntu/focal/flavors/default focal/main amd64 python3-ceph-common all 17.2.6-1278-g487ffcb5-1focal [47.2 kB] 2023-10-05T19:25:28.025 INFO:teuthology.orchestra.run.smithi118.stdout:Get:87 https://3.chacra.ceph.com/r/ceph/wip-yuri2-testing-2023-10-04-1333-quincy/487ffcb5ddb60d29c5688d1a457e812b1b697b90/ubuntu/focal/flavors/default focal/main amd64 python3-rbd amd64 17.2.6-1278-g487ffcb5-1focal [386 kB] 2023-10-05T19:25:28.076 INFO:teuthology.orchestra.run.smithi118.stdout:Get:88 https://3.chacra.ceph.com/r/ceph/wip-yuri2-testing-2023-10-04-1333-quincy/487ffcb5ddb60d29c5688d1a457e812b1b697b90/ubuntu/focal/flavors/default focal/main amd64 librgw2 amd64 17.2.6-1278-g487ffcb5-1focal [5,577 kB] 2023-10-05T19:25:28.082 INFO:teuthology.orchestra.run.smithi167.stdout:Get:24 http://archive.ubuntu.com/ubuntu focal/main amd64 liboath0 amd64 2.6.1-1.3 [44.7 kB] 2023-10-05T19:25:28.331 INFO:teuthology.orchestra.run.smithi167.stdout:Get:25 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 python3-cryptography amd64 2.8-3ubuntu0.1 [211 kB] 2023-10-05T19:25:28.526 INFO:teuthology.orchestra.run.smithi167.stdout:Get:26 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-openssl all 19.0.0-1build1 [43.3 kB] 2023-10-05T19:25:28.547 INFO:teuthology.orchestra.run.smithi167.stdout:Get:27 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-bcrypt amd64 3.1.7-2ubuntu1 [30.4 kB] 2023-10-05T19:25:28.561 INFO:teuthology.orchestra.run.smithi167.stdout:Get:28 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-cherrypy3 all 8.9.1-6 [160 kB] 2023-10-05T19:25:28.583 INFO:teuthology.orchestra.run.smithi118.stdout:Get:89 https://3.chacra.ceph.com/r/ceph/wip-yuri2-testing-2023-10-04-1333-quincy/487ffcb5ddb60d29c5688d1a457e812b1b697b90/ubuntu/focal/flavors/default focal/main amd64 python3-rgw amd64 17.2.6-1278-g487ffcb5-1focal [116 kB] 2023-10-05T19:25:28.586 INFO:teuthology.orchestra.run.smithi118.stdout:Get:90 https://3.chacra.ceph.com/r/ceph/wip-yuri2-testing-2023-10-04-1333-quincy/487ffcb5ddb60d29c5688d1a457e812b1b697b90/ubuntu/focal/flavors/default focal/main amd64 libradosstriper1 amd64 17.2.6-1278-g487ffcb5-1focal [479 kB] 2023-10-05T19:25:28.613 INFO:teuthology.orchestra.run.smithi167.stdout:Get:29 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-natsort all 7.0.1-1 [29.2 kB] 2023-10-05T19:25:28.620 INFO:teuthology.orchestra.run.smithi167.stdout:Get:30 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 python3-jwt all 1.7.1-2ubuntu2.1 [18.0 kB] 2023-10-05T19:25:28.624 INFO:teuthology.orchestra.run.smithi167.stdout:Get:31 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-logutils all 0.3.3-7 [17.3 kB] 2023-10-05T19:25:28.628 INFO:teuthology.orchestra.run.smithi167.stdout:Get:32 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-markupsafe amd64 1.1.0-1build2 [13.9 kB] 2023-10-05T19:25:28.630 INFO:teuthology.orchestra.run.smithi167.stdout:Get:33 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 python3-mako all 1.1.0+ds1-1ubuntu2.1 [59.2 kB] 2023-10-05T19:25:28.642 INFO:teuthology.orchestra.run.smithi167.stdout:Get:34 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-simplegeneric all 0.8.1-3 [11.3 kB] 2023-10-05T19:25:28.657 INFO:teuthology.orchestra.run.smithi118.stdout:Get:91 https://3.chacra.ceph.com/r/ceph/wip-yuri2-testing-2023-10-04-1333-quincy/487ffcb5ddb60d29c5688d1a457e812b1b697b90/ubuntu/focal/flavors/default focal/main amd64 ceph-common amd64 17.2.6-1278-g487ffcb5-1focal [27.7 MB] 2023-10-05T19:25:28.808 INFO:teuthology.orchestra.run.smithi167.stdout:Get:35 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-singledispatch all 3.4.0.3-2build1 [7,128 B] 2023-10-05T19:25:28.892 INFO:teuthology.orchestra.run.smithi167.stdout:Get:36 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-webob all 1:1.8.5-2 [84.4 kB] 2023-10-05T19:25:29.077 INFO:teuthology.orchestra.run.smithi167.stdout:Get:37 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-soupsieve all 1.9.5+dfsg-1 [29.1 kB] 2023-10-05T19:25:29.111 INFO:teuthology.orchestra.run.smithi167.stdout:Get:38 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-bs4 all 4.8.2-1 [83.0 kB] 2023-10-05T19:25:29.176 INFO:teuthology.orchestra.run.smithi167.stdout:Get:39 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 python3-waitress all 1.4.1-1ubuntu0.1 [81.4 kB] 2023-10-05T19:25:29.216 INFO:teuthology.orchestra.run.smithi167.stdout:Get:40 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-tempita all 0.5.2-6 [14.8 kB] 2023-10-05T19:25:29.226 INFO:teuthology.orchestra.run.smithi167.stdout:Get:41 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-paste all 3.4.0+dfsg1-1 [456 kB] 2023-10-05T19:25:29.360 INFO:teuthology.orchestra.run.smithi167.stdout:Get:42 http://archive.ubuntu.com/ubuntu focal/main amd64 python-pastedeploy-tpl all 2.1.0-1 [5,016 B] 2023-10-05T19:25:29.361 INFO:teuthology.orchestra.run.smithi167.stdout:Get:43 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-pastedeploy all 2.1.0-1 [26.5 kB] 2023-10-05T19:25:29.362 INFO:teuthology.orchestra.run.smithi167.stdout:Get:44 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-webtest all 2.0.32-2 [28.2 kB] 2023-10-05T19:25:29.395 INFO:teuthology.orchestra.run.smithi167.stdout:Get:45 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-pecan all 1.3.3-3ubuntu1 [85.8 kB] 2023-10-05T19:25:29.395 INFO:teuthology.orchestra.run.smithi167.stdout:Get:46 https://3.chacra.ceph.com/r/ceph/wip-yuri2-testing-2023-10-04-1333-quincy/487ffcb5ddb60d29c5688d1a457e812b1b697b90/ubuntu/focal/flavors/default focal/main amd64 ceph-base amd64 17.2.6-1278-g487ffcb5-1focal [6,450 kB] 2023-10-05T19:25:29.561 INFO:teuthology.orchestra.run.smithi167.stdout:Get:47 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 python3-werkzeug all 0.16.1+dfsg1-2ubuntu0.1 [183 kB] 2023-10-05T19:25:29.684 INFO:teuthology.orchestra.run.smithi167.stdout:Get:48 https://3.chacra.ceph.com/r/ceph/wip-yuri2-testing-2023-10-04-1333-quincy/487ffcb5ddb60d29c5688d1a457e812b1b697b90/ubuntu/focal/flavors/default focal/main amd64 ceph-mgr-modules-core all 17.2.6-1278-g487ffcb5-1focal [221 kB] 2023-10-05T19:25:29.691 INFO:teuthology.orchestra.run.smithi167.stdout:Get:49 https://3.chacra.ceph.com/r/ceph/wip-yuri2-testing-2023-10-04-1333-quincy/487ffcb5ddb60d29c5688d1a457e812b1b697b90/ubuntu/focal/flavors/default focal/main amd64 libsqlite3-mod-ceph amd64 17.2.6-1278-g487ffcb5-1focal [152 kB] 2023-10-05T19:25:29.696 INFO:teuthology.orchestra.run.smithi167.stdout:Get:50 https://3.chacra.ceph.com/r/ceph/wip-yuri2-testing-2023-10-04-1333-quincy/487ffcb5ddb60d29c5688d1a457e812b1b697b90/ubuntu/focal/flavors/default focal/main amd64 ceph-mgr amd64 17.2.6-1278-g487ffcb5-1focal [1,591 kB] 2023-10-05T19:25:29.784 INFO:teuthology.orchestra.run.smithi167.stdout:Get:51 https://3.chacra.ceph.com/r/ceph/wip-yuri2-testing-2023-10-04-1333-quincy/487ffcb5ddb60d29c5688d1a457e812b1b697b90/ubuntu/focal/flavors/default focal/main amd64 ceph-mon amd64 17.2.6-1278-g487ffcb5-1focal [7,845 kB] 2023-10-05T19:25:29.978 INFO:teuthology.orchestra.run.smithi167.stdout:Get:52 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-jinja2 all 2.10.1-2 [95.5 kB] 2023-10-05T19:25:30.010 INFO:teuthology.orchestra.run.smithi167.stdout:Get:53 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-repoze.lru all 0.7-2 [12.1 kB] 2023-10-05T19:25:30.014 INFO:teuthology.orchestra.run.smithi167.stdout:Get:54 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 libjs-underscore all 1.9.1~dfsg-1ubuntu0.20.04.1 [99.5 kB] 2023-10-05T19:25:30.063 INFO:teuthology.orchestra.run.smithi167.stdout:Get:55 http://archive.ubuntu.com/ubuntu focal/main amd64 libjs-sphinxdoc all 1.8.5-7ubuntu3 [97.1 kB] 2023-10-05T19:25:30.095 INFO:teuthology.orchestra.run.smithi167.stdout:Get:56 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-routes all 2.4.1-2 [88.1 kB] 2023-10-05T19:25:30.123 INFO:teuthology.orchestra.run.smithi167.stdout:Get:57 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-decorator all 4.4.2-0ubuntu1 [10.3 kB] 2023-10-05T19:25:30.126 INFO:teuthology.orchestra.run.smithi167.stdout:Get:58 http://archive.ubuntu.com/ubuntu focal/universe amd64 liblbfgsb0 amd64 3.0+dfsg.3-7build1 [29.5 kB] 2023-10-05T19:25:30.134 INFO:teuthology.orchestra.run.smithi167.stdout:Get:59 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-scipy amd64 1.3.3-3build1 [11.1 MB] 2023-10-05T19:25:30.234 INFO:teuthology.orchestra.run.smithi167.stdout:Get:60 https://3.chacra.ceph.com/r/ceph/wip-yuri2-testing-2023-10-04-1333-quincy/487ffcb5ddb60d29c5688d1a457e812b1b697b90/ubuntu/focal/flavors/default focal/main amd64 ceph-osd amd64 17.2.6-1278-g487ffcb5-1focal [28.7 MB] 2023-10-05T19:25:30.611 INFO:teuthology.orchestra.run.smithi167.stdout:Get:61 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-sklearn-lib amd64 0.22.2.post1+dfsg-5 [2,358 kB] 2023-10-05T19:25:30.648 INFO:teuthology.orchestra.run.smithi167.stdout:Get:62 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-joblib all 0.14.0-3 [192 kB] 2023-10-05T19:25:30.814 INFO:teuthology.orchestra.run.smithi167.stdout:Get:63 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-sklearn all 0.22.2.post1+dfsg-5 [1,773 kB] 2023-10-05T19:25:30.969 INFO:teuthology.orchestra.run.smithi118.stdout:Get:92 https://3.chacra.ceph.com/r/ceph/wip-yuri2-testing-2023-10-04-1333-quincy/487ffcb5ddb60d29c5688d1a457e812b1b697b90/ubuntu/focal/flavors/default focal/main amd64 ceph-base amd64 17.2.6-1278-g487ffcb5-1focal [6,450 kB] 2023-10-05T19:25:31.429 INFO:teuthology.orchestra.run.smithi118.stdout:Get:93 https://3.chacra.ceph.com/r/ceph/wip-yuri2-testing-2023-10-04-1333-quincy/487ffcb5ddb60d29c5688d1a457e812b1b697b90/ubuntu/focal/flavors/default focal/main amd64 ceph-mgr-modules-core all 17.2.6-1278-g487ffcb5-1focal [221 kB] 2023-10-05T19:25:31.435 INFO:teuthology.orchestra.run.smithi118.stdout:Get:94 https://3.chacra.ceph.com/r/ceph/wip-yuri2-testing-2023-10-04-1333-quincy/487ffcb5ddb60d29c5688d1a457e812b1b697b90/ubuntu/focal/flavors/default focal/main amd64 libsqlite3-mod-ceph amd64 17.2.6-1278-g487ffcb5-1focal [152 kB] 2023-10-05T19:25:31.444 INFO:teuthology.orchestra.run.smithi118.stdout:Get:95 https://3.chacra.ceph.com/r/ceph/wip-yuri2-testing-2023-10-04-1333-quincy/487ffcb5ddb60d29c5688d1a457e812b1b697b90/ubuntu/focal/flavors/default focal/main amd64 ceph-mgr amd64 17.2.6-1278-g487ffcb5-1focal [1,591 kB] 2023-10-05T19:25:31.499 INFO:teuthology.orchestra.run.smithi167.stdout:Get:64 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-cachetools all 4.0.0-1 [9,820 B] 2023-10-05T19:25:31.499 INFO:teuthology.orchestra.run.smithi167.stdout:Get:65 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-pyasn1 all 0.4.2-3build1 [46.7 kB] 2023-10-05T19:25:31.501 INFO:teuthology.orchestra.run.smithi167.stdout:Get:66 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-pyasn1-modules all 0.2.1-0.2build1 [32.9 kB] 2023-10-05T19:25:31.502 INFO:teuthology.orchestra.run.smithi167.stdout:Get:67 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-rsa all 4.0-3 [24.7 kB] 2023-10-05T19:25:31.503 INFO:teuthology.orchestra.run.smithi167.stdout:Get:68 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-google-auth all 1.5.1-2 [35.6 kB] 2023-10-05T19:25:31.504 INFO:teuthology.orchestra.run.smithi167.stdout:Get:69 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-blinker all 1.4+dfsg1-0.3ubuntu1 [13.2 kB] 2023-10-05T19:25:31.505 INFO:teuthology.orchestra.run.smithi167.stdout:Get:70 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-oauthlib all 3.1.0-1ubuntu2 [84.8 kB] 2023-10-05T19:25:31.507 INFO:teuthology.orchestra.run.smithi167.stdout:Get:71 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-requests-oauthlib all 1.0.0-1.1build1 [16.8 kB] 2023-10-05T19:25:31.508 INFO:teuthology.orchestra.run.smithi167.stdout:Get:72 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-websocket all 0.53.0-2ubuntu1 [32.3 kB] 2023-10-05T19:25:31.580 INFO:teuthology.orchestra.run.smithi167.stdout:Get:73 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-kubernetes all 7.0.0~a1-2 [271 kB] 2023-10-05T19:25:31.591 INFO:teuthology.orchestra.run.smithi118.stdout:Get:96 https://3.chacra.ceph.com/r/ceph/wip-yuri2-testing-2023-10-04-1333-quincy/487ffcb5ddb60d29c5688d1a457e812b1b697b90/ubuntu/focal/flavors/default focal/main amd64 ceph-mon amd64 17.2.6-1278-g487ffcb5-1focal [7,845 kB] 2023-10-05T19:25:31.752 INFO:teuthology.orchestra.run.smithi167.stdout:Get:74 http://archive.ubuntu.com/ubuntu focal/universe amd64 libonig5 amd64 6.9.4-1 [142 kB] 2023-10-05T19:25:31.893 INFO:teuthology.orchestra.run.smithi167.stdout:Get:75 https://3.chacra.ceph.com/r/ceph/wip-yuri2-testing-2023-10-04-1333-quincy/487ffcb5ddb60d29c5688d1a457e812b1b697b90/ubuntu/focal/flavors/default focal/main amd64 ceph amd64 17.2.6-1278-g487ffcb5-1focal [3,928 B] 2023-10-05T19:25:31.980 INFO:teuthology.orchestra.run.smithi167.stdout:Get:76 https://3.chacra.ceph.com/r/ceph/wip-yuri2-testing-2023-10-04-1333-quincy/487ffcb5ddb60d29c5688d1a457e812b1b697b90/ubuntu/focal/flavors/default focal/main amd64 ceph-fuse amd64 17.2.6-1278-g487ffcb5-1focal [869 kB] 2023-10-05T19:25:31.995 INFO:teuthology.orchestra.run.smithi167.stdout:Get:77 https://3.chacra.ceph.com/r/ceph/wip-yuri2-testing-2023-10-04-1333-quincy/487ffcb5ddb60d29c5688d1a457e812b1b697b90/ubuntu/focal/flavors/default focal/main amd64 ceph-mds amd64 17.2.6-1278-g487ffcb5-1focal [2,406 kB] 2023-10-05T19:25:32.012 INFO:teuthology.orchestra.run.smithi167.stdout:Get:78 https://3.chacra.ceph.com/r/ceph/wip-yuri2-testing-2023-10-04-1333-quincy/487ffcb5ddb60d29c5688d1a457e812b1b697b90/ubuntu/focal/flavors/default focal/main amd64 cephadm amd64 17.2.6-1278-g487ffcb5-1focal [79.1 kB] 2023-10-05T19:25:32.013 INFO:teuthology.orchestra.run.smithi167.stdout:Get:79 https://3.chacra.ceph.com/r/ceph/wip-yuri2-testing-2023-10-04-1333-quincy/487ffcb5ddb60d29c5688d1a457e812b1b697b90/ubuntu/focal/flavors/default focal/main amd64 ceph-mgr-cephadm all 17.2.6-1278-g487ffcb5-1focal [117 kB] 2023-10-05T19:25:32.014 INFO:teuthology.orchestra.run.smithi167.stdout:Get:80 https://3.chacra.ceph.com/r/ceph/wip-yuri2-testing-2023-10-04-1333-quincy/487ffcb5ddb60d29c5688d1a457e812b1b697b90/ubuntu/focal/flavors/default focal/main amd64 ceph-mgr-dashboard all 17.2.6-1278-g487ffcb5-1focal [1,760 kB] 2023-10-05T19:25:32.091 INFO:teuthology.orchestra.run.smithi167.stdout:Get:81 https://3.chacra.ceph.com/r/ceph/wip-yuri2-testing-2023-10-04-1333-quincy/487ffcb5ddb60d29c5688d1a457e812b1b697b90/ubuntu/focal/flavors/default focal/main amd64 ceph-mgr-diskprediction-local all 17.2.6-1278-g487ffcb5-1focal [7,879 kB] 2023-10-05T19:25:32.133 INFO:teuthology.orchestra.run.smithi167.stdout:Get:82 http://archive.ubuntu.com/ubuntu focal-updates/universe amd64 libjq1 amd64 1.6-1ubuntu0.20.04.1 [121 kB] 2023-10-05T19:25:32.183 INFO:teuthology.orchestra.run.smithi167.stdout:Get:83 http://archive.ubuntu.com/ubuntu focal-updates/universe amd64 jq amd64 1.6-1ubuntu0.20.04.1 [50.2 kB] 2023-10-05T19:25:32.186 INFO:teuthology.orchestra.run.smithi118.stdout:Get:97 https://3.chacra.ceph.com/r/ceph/wip-yuri2-testing-2023-10-04-1333-quincy/487ffcb5ddb60d29c5688d1a457e812b1b697b90/ubuntu/focal/flavors/default focal/main amd64 ceph-osd amd64 17.2.6-1278-g487ffcb5-1focal [28.7 MB] 2023-10-05T19:25:32.219 INFO:teuthology.orchestra.run.smithi167.stdout:Get:84 http://archive.ubuntu.com/ubuntu focal/main amd64 socat amd64 1.7.3.3-2 [323 kB] 2023-10-05T19:25:32.319 INFO:teuthology.orchestra.run.smithi167.stdout:Get:85 http://archive.ubuntu.com/ubuntu focal/universe amd64 xmlstarlet amd64 1.6.1-2build1 [267 kB] 2023-10-05T19:25:32.367 INFO:teuthology.orchestra.run.smithi167.stdout:Get:86 http://archive.ubuntu.com/ubuntu focal/universe amd64 fonts-mathjax all 2.7.4+dfsg-1 [2,208 kB] 2023-10-05T19:25:32.516 INFO:teuthology.orchestra.run.smithi167.stdout:Get:87 https://3.chacra.ceph.com/r/ceph/wip-yuri2-testing-2023-10-04-1333-quincy/487ffcb5ddb60d29c5688d1a457e812b1b697b90/ubuntu/focal/flavors/default focal/main amd64 ceph-mgr-k8sevents all 17.2.6-1278-g487ffcb5-1focal [4,044 B] 2023-10-05T19:25:32.517 INFO:teuthology.orchestra.run.smithi167.stdout:Get:88 https://3.chacra.ceph.com/r/ceph/wip-yuri2-testing-2023-10-04-1333-quincy/487ffcb5ddb60d29c5688d1a457e812b1b697b90/ubuntu/focal/flavors/default focal/main amd64 ceph-test amd64 17.2.6-1278-g487ffcb5-1focal [56.3 MB] 2023-10-05T19:25:32.521 INFO:teuthology.orchestra.run.smithi167.stdout:Get:89 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 nvme-cli amd64 1.9-1ubuntu0.1 [283 kB] 2023-10-05T19:25:32.530 INFO:teuthology.orchestra.run.smithi167.stdout:Get:90 http://archive.ubuntu.com/ubuntu focal/universe amd64 libjs-mathjax all 2.7.4+dfsg-1 [5,654 kB] 2023-10-05T19:25:32.679 INFO:teuthology.orchestra.run.smithi167.stdout:Get:91 http://archive.ubuntu.com/ubuntu focal/universe amd64 python-natsort-doc all 7.0.1-1 [103 kB] 2023-10-05T19:25:32.680 INFO:teuthology.orchestra.run.smithi167.stdout:Get:92 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-atomicwrites all 1.1.5-2build1 [7,016 B] 2023-10-05T19:25:32.680 INFO:teuthology.orchestra.run.smithi167.stdout:Get:93 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-attr all 19.3.0-2 [33.9 kB] 2023-10-05T19:25:32.845 INFO:teuthology.orchestra.run.smithi167.stdout:Get:94 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-webencodings all 0.5.1-1ubuntu1 [11.0 kB] 2023-10-05T19:25:32.928 INFO:teuthology.orchestra.run.smithi167.stdout:Get:95 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-html5lib all 1.0.1-2 [84.3 kB] 2023-10-05T19:25:33.110 INFO:teuthology.orchestra.run.smithi167.stdout:Get:96 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 python3-lxml amd64 4.5.0-1ubuntu0.5 [1,384 kB] 2023-10-05T19:25:33.440 INFO:teuthology.orchestra.run.smithi167.stdout:Get:97 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-packaging all 20.3-1 [26.8 kB] 2023-10-05T19:25:33.441 INFO:teuthology.orchestra.run.smithi167.stdout:Get:98 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-pastescript all 2.0.2-4 [54.6 kB] 2023-10-05T19:25:33.446 INFO:teuthology.orchestra.run.smithi167.stdout:Get:99 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-pluggy all 0.13.0-2 [18.4 kB] 2023-10-05T19:25:33.447 INFO:teuthology.orchestra.run.smithi167.stdout:Get:100 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-psutil amd64 5.5.1-1ubuntu4 [141 kB] 2023-10-05T19:25:33.457 INFO:teuthology.orchestra.run.smithi167.stdout:Get:101 http://archive.ubuntu.com/ubuntu focal-updates/universe amd64 python3-py all 1.8.1-1ubuntu0.1 [65.6 kB] 2023-10-05T19:25:33.461 INFO:teuthology.orchestra.run.smithi167.stdout:Get:102 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-pyinotify all 0.9.6-1.2ubuntu1 [24.8 kB] 2023-10-05T19:25:33.463 INFO:teuthology.orchestra.run.smithi167.stdout:Get:103 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-wcwidth all 0.1.8+dfsg1-3 [17.4 kB] 2023-10-05T19:25:33.464 INFO:teuthology.orchestra.run.smithi167.stdout:Get:104 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-pytest all 4.6.9-1 [178 kB] 2023-10-05T19:25:33.639 INFO:teuthology.orchestra.run.smithi167.stdout:Get:105 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-simplejson amd64 3.16.0-2ubuntu2 [50.8 kB] 2023-10-05T19:25:33.894 INFO:teuthology.orchestra.run.smithi167.stdout:Get:106 http://archive.ubuntu.com/ubuntu focal/universe amd64 qttranslations5-l10n all 5.12.8-0ubuntu1 [1,486 kB] 2023-10-05T19:25:34.729 INFO:teuthology.orchestra.run.smithi118.stdout:Get:98 https://3.chacra.ceph.com/r/ceph/wip-yuri2-testing-2023-10-04-1333-quincy/487ffcb5ddb60d29c5688d1a457e812b1b697b90/ubuntu/focal/flavors/default focal/main amd64 ceph amd64 17.2.6-1278-g487ffcb5-1focal [3,928 B] 2023-10-05T19:25:34.730 INFO:teuthology.orchestra.run.smithi118.stdout:Get:99 https://3.chacra.ceph.com/r/ceph/wip-yuri2-testing-2023-10-04-1333-quincy/487ffcb5ddb60d29c5688d1a457e812b1b697b90/ubuntu/focal/flavors/default focal/main amd64 ceph-fuse amd64 17.2.6-1278-g487ffcb5-1focal [869 kB] 2023-10-05T19:25:34.745 INFO:teuthology.orchestra.run.smithi118.stdout:Get:100 https://3.chacra.ceph.com/r/ceph/wip-yuri2-testing-2023-10-04-1333-quincy/487ffcb5ddb60d29c5688d1a457e812b1b697b90/ubuntu/focal/flavors/default focal/main amd64 ceph-mds amd64 17.2.6-1278-g487ffcb5-1focal [2,406 kB] 2023-10-05T19:25:34.882 INFO:teuthology.orchestra.run.smithi118.stdout:Get:101 https://3.chacra.ceph.com/r/ceph/wip-yuri2-testing-2023-10-04-1333-quincy/487ffcb5ddb60d29c5688d1a457e812b1b697b90/ubuntu/focal/flavors/default focal/main amd64 cephadm amd64 17.2.6-1278-g487ffcb5-1focal [79.1 kB] 2023-10-05T19:25:34.884 INFO:teuthology.orchestra.run.smithi118.stdout:Get:102 https://3.chacra.ceph.com/r/ceph/wip-yuri2-testing-2023-10-04-1333-quincy/487ffcb5ddb60d29c5688d1a457e812b1b697b90/ubuntu/focal/flavors/default focal/main amd64 ceph-mgr-cephadm all 17.2.6-1278-g487ffcb5-1focal [117 kB] 2023-10-05T19:25:34.885 INFO:teuthology.orchestra.run.smithi118.stdout:Get:103 https://3.chacra.ceph.com/r/ceph/wip-yuri2-testing-2023-10-04-1333-quincy/487ffcb5ddb60d29c5688d1a457e812b1b697b90/ubuntu/focal/flavors/default focal/main amd64 ceph-mgr-dashboard all 17.2.6-1278-g487ffcb5-1focal [1,760 kB] 2023-10-05T19:25:35.025 INFO:teuthology.orchestra.run.smithi118.stdout:Get:104 https://3.chacra.ceph.com/r/ceph/wip-yuri2-testing-2023-10-04-1333-quincy/487ffcb5ddb60d29c5688d1a457e812b1b697b90/ubuntu/focal/flavors/default focal/main amd64 ceph-mgr-diskprediction-local all 17.2.6-1278-g487ffcb5-1focal [7,879 kB] 2023-10-05T19:25:35.590 INFO:teuthology.orchestra.run.smithi118.stdout:Get:105 https://3.chacra.ceph.com/r/ceph/wip-yuri2-testing-2023-10-04-1333-quincy/487ffcb5ddb60d29c5688d1a457e812b1b697b90/ubuntu/focal/flavors/default focal/main amd64 ceph-mgr-k8sevents all 17.2.6-1278-g487ffcb5-1focal [4,044 B] 2023-10-05T19:25:35.591 INFO:teuthology.orchestra.run.smithi118.stdout:Get:106 https://3.chacra.ceph.com/r/ceph/wip-yuri2-testing-2023-10-04-1333-quincy/487ffcb5ddb60d29c5688d1a457e812b1b697b90/ubuntu/focal/flavors/default focal/main amd64 ceph-test amd64 17.2.6-1278-g487ffcb5-1focal [56.3 MB] 2023-10-05T19:25:35.860 INFO:teuthology.orchestra.run.smithi167.stdout:Get:107 https://3.chacra.ceph.com/r/ceph/wip-yuri2-testing-2023-10-04-1333-quincy/487ffcb5ddb60d29c5688d1a457e812b1b697b90/ubuntu/focal/flavors/default focal/main amd64 ceph-volume all 17.2.6-1278-g487ffcb5-1focal [229 kB] 2023-10-05T19:25:35.864 INFO:teuthology.orchestra.run.smithi167.stdout:Get:108 https://3.chacra.ceph.com/r/ceph/wip-yuri2-testing-2023-10-04-1333-quincy/487ffcb5ddb60d29c5688d1a457e812b1b697b90/ubuntu/focal/flavors/default focal/main amd64 libcephfs-dev amd64 17.2.6-1278-g487ffcb5-1focal [21.9 kB] 2023-10-05T19:25:35.865 INFO:teuthology.orchestra.run.smithi167.stdout:Get:109 https://3.chacra.ceph.com/r/ceph/wip-yuri2-testing-2023-10-04-1333-quincy/487ffcb5ddb60d29c5688d1a457e812b1b697b90/ubuntu/focal/flavors/default focal/main amd64 radosgw amd64 17.2.6-1278-g487ffcb5-1focal [17.1 MB] 2023-10-05T19:25:36.765 INFO:teuthology.orchestra.run.smithi167.stdout:Get:110 https://3.chacra.ceph.com/r/ceph/wip-yuri2-testing-2023-10-04-1333-quincy/487ffcb5ddb60d29c5688d1a457e812b1b697b90/ubuntu/focal/flavors/default focal/main amd64 rbd-fuse amd64 17.2.6-1278-g487ffcb5-1focal [79.5 kB] 2023-10-05T19:25:37.393 INFO:teuthology.orchestra.run.smithi167.stdout:Fetched 210 MB in 10s (20.3 MB/s) 2023-10-05T19:25:37.548 INFO:teuthology.orchestra.run.smithi167.stdout:Selecting previously unselected package liblttng-ust0:amd64. 2023-10-05T19:25:37.604 INFO:teuthology.orchestra.run.smithi167.stdout:(Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 141736 files and directories currently installed.) 2023-10-05T19:25:37.608 INFO:teuthology.orchestra.run.smithi167.stdout:Preparing to unpack .../000-liblttng-ust0_2.11.0-1_amd64.deb ... 2023-10-05T19:25:37.681 INFO:teuthology.orchestra.run.smithi167.stdout:Unpacking liblttng-ust0:amd64 (2.11.0-1) ... 2023-10-05T19:25:38.002 INFO:teuthology.orchestra.run.smithi167.stdout:Preparing to unpack .../001-librbd1_17.2.6-1278-g487ffcb5-1focal_amd64.deb ... 2023-10-05T19:25:38.119 INFO:teuthology.orchestra.run.smithi167.stdout:Unpacking librbd1 (17.2.6-1278-g487ffcb5-1focal) over (15.2.17-0ubuntu0.20.04.4) ... 2023-10-05T19:25:38.274 INFO:teuthology.orchestra.run.smithi118.stdout:Get:107 https://3.chacra.ceph.com/r/ceph/wip-yuri2-testing-2023-10-04-1333-quincy/487ffcb5ddb60d29c5688d1a457e812b1b697b90/ubuntu/focal/flavors/default focal/main amd64 ceph-volume all 17.2.6-1278-g487ffcb5-1focal [229 kB] 2023-10-05T19:25:38.278 INFO:teuthology.orchestra.run.smithi118.stdout:Get:108 https://3.chacra.ceph.com/r/ceph/wip-yuri2-testing-2023-10-04-1333-quincy/487ffcb5ddb60d29c5688d1a457e812b1b697b90/ubuntu/focal/flavors/default focal/main amd64 libcephfs-dev amd64 17.2.6-1278-g487ffcb5-1focal [21.9 kB] 2023-10-05T19:25:38.349 INFO:teuthology.orchestra.run.smithi118.stdout:Get:109 https://3.chacra.ceph.com/r/ceph/wip-yuri2-testing-2023-10-04-1333-quincy/487ffcb5ddb60d29c5688d1a457e812b1b697b90/ubuntu/focal/flavors/default focal/main amd64 radosgw amd64 17.2.6-1278-g487ffcb5-1focal [17.1 MB] 2023-10-05T19:25:38.830 INFO:teuthology.orchestra.run.smithi167.stdout:Preparing to unpack .../002-librados2_17.2.6-1278-g487ffcb5-1focal_amd64.deb ... 2023-10-05T19:25:38.858 INFO:teuthology.orchestra.run.smithi118.stdout:Get:110 https://3.chacra.ceph.com/r/ceph/wip-yuri2-testing-2023-10-04-1333-quincy/487ffcb5ddb60d29c5688d1a457e812b1b697b90/ubuntu/focal/flavors/default focal/main amd64 rbd-fuse amd64 17.2.6-1278-g487ffcb5-1focal [79.5 kB] 2023-10-05T19:25:38.957 INFO:teuthology.orchestra.run.smithi167.stdout:Unpacking librados2 (17.2.6-1278-g487ffcb5-1focal) over (15.2.17-0ubuntu0.20.04.4) ... 2023-10-05T19:25:39.479 INFO:teuthology.orchestra.run.smithi118.stdout:Fetched 210 MB in 12s (17.5 MB/s) 2023-10-05T19:25:39.604 INFO:teuthology.orchestra.run.smithi118.stdout:Selecting previously unselected package liblttng-ust0:amd64. 2023-10-05T19:25:39.659 INFO:teuthology.orchestra.run.smithi118.stdout:(Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 141736 files and directories currently installed.) 2023-10-05T19:25:39.664 INFO:teuthology.orchestra.run.smithi118.stdout:Preparing to unpack .../000-liblttng-ust0_2.11.0-1_amd64.deb ... 2023-10-05T19:25:39.669 INFO:teuthology.orchestra.run.smithi167.stdout:Selecting previously unselected package libcephfs2. 2023-10-05T19:25:39.685 INFO:teuthology.orchestra.run.smithi167.stdout:Preparing to unpack .../003-libcephfs2_17.2.6-1278-g487ffcb5-1focal_amd64.deb ... 2023-10-05T19:25:39.727 INFO:teuthology.orchestra.run.smithi167.stdout:Unpacking libcephfs2 (17.2.6-1278-g487ffcb5-1focal) ... 2023-10-05T19:25:39.735 INFO:teuthology.orchestra.run.smithi118.stdout:Unpacking liblttng-ust0:amd64 (2.11.0-1) ... 2023-10-05T19:25:40.063 INFO:teuthology.orchestra.run.smithi167.stdout:Selecting previously unselected package python3-rados. 2023-10-05T19:25:40.082 INFO:teuthology.orchestra.run.smithi167.stdout:Preparing to unpack .../004-python3-rados_17.2.6-1278-g487ffcb5-1focal_amd64.deb ... 2023-10-05T19:25:40.107 INFO:teuthology.orchestra.run.smithi118.stdout:Preparing to unpack .../001-librbd1_17.2.6-1278-g487ffcb5-1focal_amd64.deb ... 2023-10-05T19:25:40.113 INFO:teuthology.orchestra.run.smithi167.stdout:Unpacking python3-rados (17.2.6-1278-g487ffcb5-1focal) ... 2023-10-05T19:25:40.232 INFO:teuthology.orchestra.run.smithi118.stdout:Unpacking librbd1 (17.2.6-1278-g487ffcb5-1focal) over (15.2.17-0ubuntu0.20.04.4) ... 2023-10-05T19:25:40.483 INFO:teuthology.orchestra.run.smithi167.stdout:Selecting previously unselected package python3-ceph-argparse. 2023-10-05T19:25:40.500 INFO:teuthology.orchestra.run.smithi167.stdout:Preparing to unpack .../005-python3-ceph-argparse_17.2.6-1278-g487ffcb5-1focal_all.deb ... 2023-10-05T19:25:40.541 INFO:teuthology.orchestra.run.smithi167.stdout:Unpacking python3-ceph-argparse (17.2.6-1278-g487ffcb5-1focal) ... 2023-10-05T19:25:40.777 INFO:teuthology.orchestra.run.smithi167.stdout:Selecting previously unselected package python3-cephfs. 2023-10-05T19:25:40.793 INFO:teuthology.orchestra.run.smithi167.stdout:Preparing to unpack .../006-python3-cephfs_17.2.6-1278-g487ffcb5-1focal_amd64.deb ... 2023-10-05T19:25:40.835 INFO:teuthology.orchestra.run.smithi167.stdout:Unpacking python3-cephfs (17.2.6-1278-g487ffcb5-1focal) ... 2023-10-05T19:25:40.927 INFO:teuthology.orchestra.run.smithi118.stdout:Preparing to unpack .../002-librados2_17.2.6-1278-g487ffcb5-1focal_amd64.deb ... 2023-10-05T19:25:41.028 INFO:teuthology.orchestra.run.smithi118.stdout:Unpacking librados2 (17.2.6-1278-g487ffcb5-1focal) over (15.2.17-0ubuntu0.20.04.4) ... 2023-10-05T19:25:41.171 INFO:teuthology.orchestra.run.smithi167.stdout:Selecting previously unselected package python3-ceph-common. 2023-10-05T19:25:41.190 INFO:teuthology.orchestra.run.smithi167.stdout:Preparing to unpack .../007-python3-ceph-common_17.2.6-1278-g487ffcb5-1focal_all.deb ... 2023-10-05T19:25:41.229 INFO:teuthology.orchestra.run.smithi167.stdout:Unpacking python3-ceph-common (17.2.6-1278-g487ffcb5-1focal) ... 2023-10-05T19:25:41.516 INFO:teuthology.orchestra.run.smithi167.stdout:Selecting previously unselected package python3-prettytable. 2023-10-05T19:25:41.532 INFO:teuthology.orchestra.run.smithi167.stdout:Preparing to unpack .../008-python3-prettytable_0.7.2-5_all.deb ... 2023-10-05T19:25:41.566 INFO:teuthology.orchestra.run.smithi167.stdout:Unpacking python3-prettytable (0.7.2-5) ... 2023-10-05T19:25:41.716 INFO:teuthology.orchestra.run.smithi118.stdout:Selecting previously unselected package libcephfs2. 2023-10-05T19:25:41.734 INFO:teuthology.orchestra.run.smithi118.stdout:Preparing to unpack .../003-libcephfs2_17.2.6-1278-g487ffcb5-1focal_amd64.deb ... 2023-10-05T19:25:41.774 INFO:teuthology.orchestra.run.smithi118.stdout:Unpacking libcephfs2 (17.2.6-1278-g487ffcb5-1focal) ... 2023-10-05T19:25:41.877 INFO:teuthology.orchestra.run.smithi167.stdout:Selecting previously unselected package python3-rbd. 2023-10-05T19:25:41.895 INFO:teuthology.orchestra.run.smithi167.stdout:Preparing to unpack .../009-python3-rbd_17.2.6-1278-g487ffcb5-1focal_amd64.deb ... 2023-10-05T19:25:41.935 INFO:teuthology.orchestra.run.smithi167.stdout:Unpacking python3-rbd (17.2.6-1278-g487ffcb5-1focal) ... 2023-10-05T19:25:42.127 INFO:teuthology.orchestra.run.smithi118.stdout:Selecting previously unselected package python3-rados. 2023-10-05T19:25:42.143 INFO:teuthology.orchestra.run.smithi118.stdout:Preparing to unpack .../004-python3-rados_17.2.6-1278-g487ffcb5-1focal_amd64.deb ... 2023-10-05T19:25:42.177 INFO:teuthology.orchestra.run.smithi118.stdout:Unpacking python3-rados (17.2.6-1278-g487ffcb5-1focal) ... 2023-10-05T19:25:42.288 INFO:teuthology.orchestra.run.smithi167.stdout:Selecting previously unselected package liblua5.3-0:amd64. 2023-10-05T19:25:42.304 INFO:teuthology.orchestra.run.smithi167.stdout:Preparing to unpack .../010-liblua5.3-0_5.3.3-1.1ubuntu2_amd64.deb ... 2023-10-05T19:25:42.346 INFO:teuthology.orchestra.run.smithi167.stdout:Unpacking liblua5.3-0:amd64 (5.3.3-1.1ubuntu2) ... 2023-10-05T19:25:42.472 INFO:teuthology.orchestra.run.smithi118.stdout:Selecting previously unselected package python3-ceph-argparse. 2023-10-05T19:25:42.489 INFO:teuthology.orchestra.run.smithi118.stdout:Preparing to unpack .../005-python3-ceph-argparse_17.2.6-1278-g487ffcb5-1focal_all.deb ... 2023-10-05T19:25:42.530 INFO:teuthology.orchestra.run.smithi118.stdout:Unpacking python3-ceph-argparse (17.2.6-1278-g487ffcb5-1focal) ... 2023-10-05T19:25:42.667 INFO:teuthology.orchestra.run.smithi167.stdout:Selecting previously unselected package librabbitmq4:amd64. 2023-10-05T19:25:42.688 INFO:teuthology.orchestra.run.smithi167.stdout:Preparing to unpack .../011-librabbitmq4_0.10.0-1_amd64.deb ... 2023-10-05T19:25:42.716 INFO:teuthology.orchestra.run.smithi167.stdout:Unpacking librabbitmq4:amd64 (0.10.0-1) ... 2023-10-05T19:25:42.782 INFO:teuthology.orchestra.run.smithi118.stdout:Selecting previously unselected package python3-cephfs. 2023-10-05T19:25:42.801 INFO:teuthology.orchestra.run.smithi118.stdout:Preparing to unpack .../006-python3-cephfs_17.2.6-1278-g487ffcb5-1focal_amd64.deb ... 2023-10-05T19:25:42.857 INFO:teuthology.orchestra.run.smithi118.stdout:Unpacking python3-cephfs (17.2.6-1278-g487ffcb5-1focal) ... 2023-10-05T19:25:43.067 INFO:teuthology.orchestra.run.smithi167.stdout:Selecting previously unselected package librdkafka1:amd64. 2023-10-05T19:25:43.086 INFO:teuthology.orchestra.run.smithi167.stdout:Preparing to unpack .../012-librdkafka1_1.2.1-1ubuntu1_amd64.deb ... 2023-10-05T19:25:43.117 INFO:teuthology.orchestra.run.smithi167.stdout:Unpacking librdkafka1:amd64 (1.2.1-1ubuntu1) ... 2023-10-05T19:25:43.194 INFO:teuthology.orchestra.run.smithi118.stdout:Selecting previously unselected package python3-ceph-common. 2023-10-05T19:25:43.211 INFO:teuthology.orchestra.run.smithi118.stdout:Preparing to unpack .../007-python3-ceph-common_17.2.6-1278-g487ffcb5-1focal_all.deb ... 2023-10-05T19:25:43.252 INFO:teuthology.orchestra.run.smithi118.stdout:Unpacking python3-ceph-common (17.2.6-1278-g487ffcb5-1focal) ... 2023-10-05T19:25:43.458 INFO:teuthology.orchestra.run.smithi167.stdout:Selecting previously unselected package libdouble-conversion3:amd64. 2023-10-05T19:25:43.477 INFO:teuthology.orchestra.run.smithi167.stdout:Preparing to unpack .../013-libdouble-conversion3_3.1.5-4ubuntu1_amd64.deb ... 2023-10-05T19:25:43.508 INFO:teuthology.orchestra.run.smithi167.stdout:Unpacking libdouble-conversion3:amd64 (3.1.5-4ubuntu1) ... 2023-10-05T19:25:43.572 INFO:teuthology.orchestra.run.smithi118.stdout:Selecting previously unselected package python3-prettytable. 2023-10-05T19:25:43.590 INFO:teuthology.orchestra.run.smithi118.stdout:Preparing to unpack .../008-python3-prettytable_0.7.2-5_all.deb ... 2023-10-05T19:25:43.630 INFO:teuthology.orchestra.run.smithi118.stdout:Unpacking python3-prettytable (0.7.2-5) ... 2023-10-05T19:25:44.052 INFO:teuthology.orchestra.run.smithi118.stdout:Selecting previously unselected package python3-rbd. 2023-10-05T19:25:44.068 INFO:teuthology.orchestra.run.smithi118.stdout:Preparing to unpack .../009-python3-rbd_17.2.6-1278-g487ffcb5-1focal_amd64.deb ... 2023-10-05T19:25:44.093 INFO:teuthology.orchestra.run.smithi167.stdout:Selecting previously unselected package libpcre2-16-0:amd64. 2023-10-05T19:25:44.109 INFO:teuthology.orchestra.run.smithi167.stdout:Preparing to unpack .../014-libpcre2-16-0_10.34-7ubuntu0.1_amd64.deb ... 2023-10-05T19:25:44.110 INFO:teuthology.orchestra.run.smithi118.stdout:Unpacking python3-rbd (17.2.6-1278-g487ffcb5-1focal) ... 2023-10-05T19:25:44.151 INFO:teuthology.orchestra.run.smithi167.stdout:Unpacking libpcre2-16-0:amd64 (10.34-7ubuntu0.1) ... 2023-10-05T19:25:44.514 INFO:teuthology.orchestra.run.smithi118.stdout:Selecting previously unselected package liblua5.3-0:amd64. 2023-10-05T19:25:44.531 INFO:teuthology.orchestra.run.smithi118.stdout:Preparing to unpack .../010-liblua5.3-0_5.3.3-1.1ubuntu2_amd64.deb ... 2023-10-05T19:25:44.564 INFO:teuthology.orchestra.run.smithi118.stdout:Unpacking liblua5.3-0:amd64 (5.3.3-1.1ubuntu2) ... 2023-10-05T19:25:44.638 INFO:teuthology.orchestra.run.smithi167.stdout:Selecting previously unselected package libqt5core5a:amd64. 2023-10-05T19:25:44.658 INFO:teuthology.orchestra.run.smithi167.stdout:Preparing to unpack .../015-libqt5core5a_5.12.8+dfsg-0ubuntu2.1_amd64.deb ... 2023-10-05T19:25:44.706 INFO:teuthology.orchestra.run.smithi167.stdout:Unpacking libqt5core5a:amd64 (5.12.8+dfsg-0ubuntu2.1) ... 2023-10-05T19:25:44.910 INFO:teuthology.orchestra.run.smithi118.stdout:Selecting previously unselected package librabbitmq4:amd64. 2023-10-05T19:25:44.927 INFO:teuthology.orchestra.run.smithi118.stdout:Preparing to unpack .../011-librabbitmq4_0.10.0-1_amd64.deb ... 2023-10-05T19:25:44.970 INFO:teuthology.orchestra.run.smithi118.stdout:Unpacking librabbitmq4:amd64 (0.10.0-1) ... 2023-10-05T19:25:45.183 INFO:teuthology.orchestra.run.smithi167.stdout:Selecting previously unselected package libqt5dbus5:amd64. 2023-10-05T19:25:45.200 INFO:teuthology.orchestra.run.smithi167.stdout:Preparing to unpack .../016-libqt5dbus5_5.12.8+dfsg-0ubuntu2.1_amd64.deb ... 2023-10-05T19:25:45.242 INFO:teuthology.orchestra.run.smithi167.stdout:Unpacking libqt5dbus5:amd64 (5.12.8+dfsg-0ubuntu2.1) ... 2023-10-05T19:25:45.327 INFO:teuthology.orchestra.run.smithi118.stdout:Selecting previously unselected package librdkafka1:amd64. 2023-10-05T19:25:45.344 INFO:teuthology.orchestra.run.smithi118.stdout:Preparing to unpack .../012-librdkafka1_1.2.1-1ubuntu1_amd64.deb ... 2023-10-05T19:25:45.385 INFO:teuthology.orchestra.run.smithi118.stdout:Unpacking librdkafka1:amd64 (1.2.1-1ubuntu1) ... 2023-10-05T19:25:45.611 INFO:teuthology.orchestra.run.smithi167.stdout:Selecting previously unselected package libqt5network5:amd64. 2023-10-05T19:25:45.629 INFO:teuthology.orchestra.run.smithi167.stdout:Preparing to unpack .../017-libqt5network5_5.12.8+dfsg-0ubuntu2.1_amd64.deb ... 2023-10-05T19:25:45.669 INFO:teuthology.orchestra.run.smithi167.stdout:Unpacking libqt5network5:amd64 (5.12.8+dfsg-0ubuntu2.1) ... 2023-10-05T19:25:45.789 INFO:teuthology.orchestra.run.smithi118.stdout:Selecting previously unselected package libdouble-conversion3:amd64. 2023-10-05T19:25:45.806 INFO:teuthology.orchestra.run.smithi118.stdout:Preparing to unpack .../013-libdouble-conversion3_3.1.5-4ubuntu1_amd64.deb ... 2023-10-05T19:25:45.847 INFO:teuthology.orchestra.run.smithi118.stdout:Unpacking libdouble-conversion3:amd64 (3.1.5-4ubuntu1) ... 2023-10-05T19:25:46.006 INFO:teuthology.orchestra.run.smithi167.stdout:Selecting previously unselected package libthrift-0.13.0. 2023-10-05T19:25:46.022 INFO:teuthology.orchestra.run.smithi167.stdout:Preparing to unpack .../018-libthrift-0.13.0_0.13.0-2build2_amd64.deb ... 2023-10-05T19:25:46.056 INFO:teuthology.orchestra.run.smithi167.stdout:Unpacking libthrift-0.13.0 (0.13.0-2build2) ... 2023-10-05T19:25:46.151 INFO:teuthology.orchestra.run.smithi118.stdout:Selecting previously unselected package libpcre2-16-0:amd64. 2023-10-05T19:25:46.169 INFO:teuthology.orchestra.run.smithi118.stdout:Preparing to unpack .../014-libpcre2-16-0_10.34-7ubuntu0.1_amd64.deb ... 2023-10-05T19:25:46.209 INFO:teuthology.orchestra.run.smithi118.stdout:Unpacking libpcre2-16-0:amd64 (10.34-7ubuntu0.1) ... 2023-10-05T19:25:46.418 INFO:teuthology.orchestra.run.smithi167.stdout:Selecting previously unselected package librgw2. 2023-10-05T19:25:46.435 INFO:teuthology.orchestra.run.smithi167.stdout:Preparing to unpack .../019-librgw2_17.2.6-1278-g487ffcb5-1focal_amd64.deb ... 2023-10-05T19:25:46.476 INFO:teuthology.orchestra.run.smithi167.stdout:Unpacking librgw2 (17.2.6-1278-g487ffcb5-1focal) ... 2023-10-05T19:25:46.704 INFO:teuthology.orchestra.run.smithi118.stdout:Selecting previously unselected package libqt5core5a:amd64. 2023-10-05T19:25:46.720 INFO:teuthology.orchestra.run.smithi118.stdout:Preparing to unpack .../015-libqt5core5a_5.12.8+dfsg-0ubuntu2.1_amd64.deb ... 2023-10-05T19:25:46.773 INFO:teuthology.orchestra.run.smithi118.stdout:Unpacking libqt5core5a:amd64 (5.12.8+dfsg-0ubuntu2.1) ... 2023-10-05T19:25:47.279 INFO:teuthology.orchestra.run.smithi167.stdout:Selecting previously unselected package python3-rgw. 2023-10-05T19:25:47.294 INFO:teuthology.orchestra.run.smithi167.stdout:Preparing to unpack .../020-python3-rgw_17.2.6-1278-g487ffcb5-1focal_amd64.deb ... 2023-10-05T19:25:47.299 INFO:teuthology.orchestra.run.smithi118.stdout:Selecting previously unselected package libqt5dbus5:amd64. 2023-10-05T19:25:47.318 INFO:teuthology.orchestra.run.smithi118.stdout:Preparing to unpack .../016-libqt5dbus5_5.12.8+dfsg-0ubuntu2.1_amd64.deb ... 2023-10-05T19:25:47.337 INFO:teuthology.orchestra.run.smithi167.stdout:Unpacking python3-rgw (17.2.6-1278-g487ffcb5-1focal) ... 2023-10-05T19:25:47.357 INFO:teuthology.orchestra.run.smithi118.stdout:Unpacking libqt5dbus5:amd64 (5.12.8+dfsg-0ubuntu2.1) ... 2023-10-05T19:25:47.673 INFO:teuthology.orchestra.run.smithi167.stdout:Selecting previously unselected package liboath0. 2023-10-05T19:25:47.689 INFO:teuthology.orchestra.run.smithi167.stdout:Preparing to unpack .../021-liboath0_2.6.1-1.3_amd64.deb ... 2023-10-05T19:25:47.723 INFO:teuthology.orchestra.run.smithi167.stdout:Unpacking liboath0 (2.6.1-1.3) ... 2023-10-05T19:25:47.753 INFO:teuthology.orchestra.run.smithi118.stdout:Selecting previously unselected package libqt5network5:amd64. 2023-10-05T19:25:47.769 INFO:teuthology.orchestra.run.smithi118.stdout:Preparing to unpack .../017-libqt5network5_5.12.8+dfsg-0ubuntu2.1_amd64.deb ... 2023-10-05T19:25:47.811 INFO:teuthology.orchestra.run.smithi118.stdout:Unpacking libqt5network5:amd64 (5.12.8+dfsg-0ubuntu2.1) ... 2023-10-05T19:25:48.026 INFO:teuthology.orchestra.run.smithi167.stdout:Selecting previously unselected package libradosstriper1. 2023-10-05T19:25:48.042 INFO:teuthology.orchestra.run.smithi167.stdout:Preparing to unpack .../022-libradosstriper1_17.2.6-1278-g487ffcb5-1focal_amd64.deb ... 2023-10-05T19:25:48.085 INFO:teuthology.orchestra.run.smithi167.stdout:Unpacking libradosstriper1 (17.2.6-1278-g487ffcb5-1focal) ... 2023-10-05T19:25:48.223 INFO:teuthology.orchestra.run.smithi118.stdout:Selecting previously unselected package libthrift-0.13.0. 2023-10-05T19:25:48.240 INFO:teuthology.orchestra.run.smithi118.stdout:Preparing to unpack .../018-libthrift-0.13.0_0.13.0-2build2_amd64.deb ... 2023-10-05T19:25:48.281 INFO:teuthology.orchestra.run.smithi118.stdout:Unpacking libthrift-0.13.0 (0.13.0-2build2) ... 2023-10-05T19:25:48.538 INFO:teuthology.orchestra.run.smithi167.stdout:Selecting previously unselected package ceph-common. 2023-10-05T19:25:48.558 INFO:teuthology.orchestra.run.smithi167.stdout:Preparing to unpack .../023-ceph-common_17.2.6-1278-g487ffcb5-1focal_amd64.deb ... 2023-10-05T19:25:48.596 INFO:teuthology.orchestra.run.smithi167.stdout:Unpacking ceph-common (17.2.6-1278-g487ffcb5-1focal) ... 2023-10-05T19:25:48.618 INFO:teuthology.orchestra.run.smithi118.stdout:Selecting previously unselected package librgw2. 2023-10-05T19:25:48.635 INFO:teuthology.orchestra.run.smithi118.stdout:Preparing to unpack .../019-librgw2_17.2.6-1278-g487ffcb5-1focal_amd64.deb ... 2023-10-05T19:25:48.676 INFO:teuthology.orchestra.run.smithi118.stdout:Unpacking librgw2 (17.2.6-1278-g487ffcb5-1focal) ... 2023-10-05T19:25:49.471 INFO:teuthology.orchestra.run.smithi118.stdout:Selecting previously unselected package python3-rgw. 2023-10-05T19:25:49.490 INFO:teuthology.orchestra.run.smithi118.stdout:Preparing to unpack .../020-python3-rgw_17.2.6-1278-g487ffcb5-1focal_amd64.deb ... 2023-10-05T19:25:49.529 INFO:teuthology.orchestra.run.smithi118.stdout:Unpacking python3-rgw (17.2.6-1278-g487ffcb5-1focal) ... 2023-10-05T19:25:49.882 INFO:teuthology.orchestra.run.smithi118.stdout:Selecting previously unselected package liboath0. 2023-10-05T19:25:49.899 INFO:teuthology.orchestra.run.smithi118.stdout:Preparing to unpack .../021-liboath0_2.6.1-1.3_amd64.deb ... 2023-10-05T19:25:49.941 INFO:teuthology.orchestra.run.smithi118.stdout:Unpacking liboath0 (2.6.1-1.3) ... 2023-10-05T19:25:50.277 INFO:teuthology.orchestra.run.smithi118.stdout:Selecting previously unselected package libradosstriper1. 2023-10-05T19:25:50.294 INFO:teuthology.orchestra.run.smithi118.stdout:Preparing to unpack .../022-libradosstriper1_17.2.6-1278-g487ffcb5-1focal_amd64.deb ... 2023-10-05T19:25:50.335 INFO:teuthology.orchestra.run.smithi118.stdout:Unpacking libradosstriper1 (17.2.6-1278-g487ffcb5-1focal) ... 2023-10-05T19:25:50.739 INFO:teuthology.orchestra.run.smithi118.stdout:Selecting previously unselected package ceph-common. 2023-10-05T19:25:50.755 INFO:teuthology.orchestra.run.smithi118.stdout:Preparing to unpack .../023-ceph-common_17.2.6-1278-g487ffcb5-1focal_amd64.deb ... 2023-10-05T19:25:50.788 INFO:teuthology.orchestra.run.smithi118.stdout:Unpacking ceph-common (17.2.6-1278-g487ffcb5-1focal) ... 2023-10-05T19:25:51.377 INFO:teuthology.orchestra.run.smithi167.stdout:Selecting previously unselected package ceph-base. 2023-10-05T19:25:51.393 INFO:teuthology.orchestra.run.smithi167.stdout:Preparing to unpack .../024-ceph-base_17.2.6-1278-g487ffcb5-1focal_amd64.deb ... 2023-10-05T19:25:51.446 INFO:teuthology.orchestra.run.smithi167.stdout:Unpacking ceph-base (17.2.6-1278-g487ffcb5-1focal) ... 2023-10-05T19:25:52.408 INFO:teuthology.orchestra.run.smithi167.stdout:Selecting previously unselected package python3-cryptography. 2023-10-05T19:25:52.427 INFO:teuthology.orchestra.run.smithi167.stdout:Preparing to unpack .../025-python3-cryptography_2.8-3ubuntu0.1_amd64.deb ... 2023-10-05T19:25:52.466 INFO:teuthology.orchestra.run.smithi167.stdout:Unpacking python3-cryptography (2.8-3ubuntu0.1) ... 2023-10-05T19:25:52.862 INFO:teuthology.orchestra.run.smithi167.stdout:Selecting previously unselected package python3-openssl. 2023-10-05T19:25:52.878 INFO:teuthology.orchestra.run.smithi167.stdout:Preparing to unpack .../026-python3-openssl_19.0.0-1build1_all.deb ... 2023-10-05T19:25:52.936 INFO:teuthology.orchestra.run.smithi167.stdout:Unpacking python3-openssl (19.0.0-1build1) ... 2023-10-05T19:25:53.189 INFO:teuthology.orchestra.run.smithi167.stdout:Selecting previously unselected package ceph-mgr-modules-core. 2023-10-05T19:25:53.208 INFO:teuthology.orchestra.run.smithi167.stdout:Preparing to unpack .../027-ceph-mgr-modules-core_17.2.6-1278-g487ffcb5-1focal_all.deb ... 2023-10-05T19:25:53.247 INFO:teuthology.orchestra.run.smithi167.stdout:Unpacking ceph-mgr-modules-core (17.2.6-1278-g487ffcb5-1focal) ... 2023-10-05T19:25:53.529 INFO:teuthology.orchestra.run.smithi118.stdout:Selecting previously unselected package ceph-base. 2023-10-05T19:25:53.549 INFO:teuthology.orchestra.run.smithi118.stdout:Preparing to unpack .../024-ceph-base_17.2.6-1278-g487ffcb5-1focal_amd64.deb ... 2023-10-05T19:25:53.598 INFO:teuthology.orchestra.run.smithi118.stdout:Unpacking ceph-base (17.2.6-1278-g487ffcb5-1focal) ... 2023-10-05T19:25:53.602 INFO:teuthology.orchestra.run.smithi167.stdout:Selecting previously unselected package python3-bcrypt. 2023-10-05T19:25:53.623 INFO:teuthology.orchestra.run.smithi167.stdout:Preparing to unpack .../028-python3-bcrypt_3.1.7-2ubuntu1_amd64.deb ... 2023-10-05T19:25:53.661 INFO:teuthology.orchestra.run.smithi167.stdout:Unpacking python3-bcrypt (3.1.7-2ubuntu1) ... 2023-10-05T19:25:53.997 INFO:teuthology.orchestra.run.smithi167.stdout:Selecting previously unselected package python3-cherrypy3. 2023-10-05T19:25:54.013 INFO:teuthology.orchestra.run.smithi167.stdout:Preparing to unpack .../029-python3-cherrypy3_8.9.1-6_all.deb ... 2023-10-05T19:25:54.071 INFO:teuthology.orchestra.run.smithi167.stdout:Unpacking python3-cherrypy3 (8.9.1-6) ... 2023-10-05T19:25:54.417 INFO:teuthology.orchestra.run.smithi167.stdout:Selecting previously unselected package python3-natsort. 2023-10-05T19:25:54.433 INFO:teuthology.orchestra.run.smithi167.stdout:Preparing to unpack .../030-python3-natsort_7.0.1-1_all.deb ... 2023-10-05T19:25:54.476 INFO:teuthology.orchestra.run.smithi167.stdout:Unpacking python3-natsort (7.0.1-1) ... 2023-10-05T19:25:54.659 INFO:teuthology.orchestra.run.smithi118.stdout:Selecting previously unselected package python3-cryptography. 2023-10-05T19:25:54.675 INFO:teuthology.orchestra.run.smithi118.stdout:Preparing to unpack .../025-python3-cryptography_2.8-3ubuntu0.1_amd64.deb ... 2023-10-05T19:25:54.717 INFO:teuthology.orchestra.run.smithi118.stdout:Unpacking python3-cryptography (2.8-3ubuntu0.1) ... 2023-10-05T19:25:54.795 INFO:teuthology.orchestra.run.smithi167.stdout:Selecting previously unselected package python3-jwt. 2023-10-05T19:25:54.815 INFO:teuthology.orchestra.run.smithi167.stdout:Preparing to unpack .../031-python3-jwt_1.7.1-2ubuntu2.1_all.deb ... 2023-10-05T19:25:54.853 INFO:teuthology.orchestra.run.smithi167.stdout:Unpacking python3-jwt (1.7.1-2ubuntu2.1) ... 2023-10-05T19:25:55.105 INFO:teuthology.orchestra.run.smithi118.stdout:Selecting previously unselected package python3-openssl. 2023-10-05T19:25:55.122 INFO:teuthology.orchestra.run.smithi118.stdout:Preparing to unpack .../026-python3-openssl_19.0.0-1build1_all.deb ... 2023-10-05T19:25:55.171 INFO:teuthology.orchestra.run.smithi118.stdout:Unpacking python3-openssl (19.0.0-1build1) ... 2023-10-05T19:25:55.174 INFO:teuthology.orchestra.run.smithi167.stdout:Selecting previously unselected package python3-logutils. 2023-10-05T19:25:55.190 INFO:teuthology.orchestra.run.smithi167.stdout:Preparing to unpack .../032-python3-logutils_0.3.3-7_all.deb ... 2023-10-05T19:25:55.232 INFO:teuthology.orchestra.run.smithi167.stdout:Unpacking python3-logutils (0.3.3-7) ... 2023-10-05T19:25:55.466 INFO:teuthology.orchestra.run.smithi118.stdout:Selecting previously unselected package ceph-mgr-modules-core. 2023-10-05T19:25:55.483 INFO:teuthology.orchestra.run.smithi118.stdout:Preparing to unpack .../027-ceph-mgr-modules-core_17.2.6-1278-g487ffcb5-1focal_all.deb ... 2023-10-05T19:25:55.524 INFO:teuthology.orchestra.run.smithi118.stdout:Unpacking ceph-mgr-modules-core (17.2.6-1278-g487ffcb5-1focal) ... 2023-10-05T19:25:55.560 INFO:teuthology.orchestra.run.smithi167.stdout:Selecting previously unselected package python3-markupsafe. 2023-10-05T19:25:55.577 INFO:teuthology.orchestra.run.smithi167.stdout:Preparing to unpack .../033-python3-markupsafe_1.1.0-1build2_amd64.deb ... 2023-10-05T19:25:55.619 INFO:teuthology.orchestra.run.smithi167.stdout:Unpacking python3-markupsafe (1.1.0-1build2) ... 2023-10-05T19:25:55.888 INFO:teuthology.orchestra.run.smithi118.stdout:Selecting previously unselected package python3-bcrypt. 2023-10-05T19:25:55.903 INFO:teuthology.orchestra.run.smithi118.stdout:Preparing to unpack .../028-python3-bcrypt_3.1.7-2ubuntu1_amd64.deb ... 2023-10-05T19:25:55.922 INFO:teuthology.orchestra.run.smithi167.stdout:Selecting previously unselected package python3-mako. 2023-10-05T19:25:55.940 INFO:teuthology.orchestra.run.smithi167.stdout:Preparing to unpack .../034-python3-mako_1.1.0+ds1-1ubuntu2.1_all.deb ... 2023-10-05T19:25:55.945 INFO:teuthology.orchestra.run.smithi118.stdout:Unpacking python3-bcrypt (3.1.7-2ubuntu1) ... 2023-10-05T19:25:55.980 INFO:teuthology.orchestra.run.smithi167.stdout:Unpacking python3-mako (1.1.0+ds1-1ubuntu2.1) ... 2023-10-05T19:25:56.274 INFO:teuthology.orchestra.run.smithi118.stdout:Selecting previously unselected package python3-cherrypy3. 2023-10-05T19:25:56.294 INFO:teuthology.orchestra.run.smithi118.stdout:Preparing to unpack .../029-python3-cherrypy3_8.9.1-6_all.deb ... 2023-10-05T19:25:56.333 INFO:teuthology.orchestra.run.smithi118.stdout:Unpacking python3-cherrypy3 (8.9.1-6) ... 2023-10-05T19:25:56.350 INFO:teuthology.orchestra.run.smithi167.stdout:Selecting previously unselected package python3-simplegeneric. 2023-10-05T19:25:56.366 INFO:teuthology.orchestra.run.smithi167.stdout:Preparing to unpack .../035-python3-simplegeneric_0.8.1-3_all.deb ... 2023-10-05T19:25:56.408 INFO:teuthology.orchestra.run.smithi167.stdout:Unpacking python3-simplegeneric (0.8.1-3) ... 2023-10-05T19:25:56.670 INFO:teuthology.orchestra.run.smithi118.stdout:Selecting previously unselected package python3-natsort. 2023-10-05T19:25:56.686 INFO:teuthology.orchestra.run.smithi167.stdout:Selecting previously unselected package python3-singledispatch. 2023-10-05T19:25:56.686 INFO:teuthology.orchestra.run.smithi118.stdout:Preparing to unpack .../030-python3-natsort_7.0.1-1_all.deb ... 2023-10-05T19:25:56.702 INFO:teuthology.orchestra.run.smithi167.stdout:Preparing to unpack .../036-python3-singledispatch_3.4.0.3-2build1_all.deb ... 2023-10-05T19:25:56.727 INFO:teuthology.orchestra.run.smithi118.stdout:Unpacking python3-natsort (7.0.1-1) ... 2023-10-05T19:25:56.736 INFO:teuthology.orchestra.run.smithi167.stdout:Unpacking python3-singledispatch (3.4.0.3-2build1) ... 2023-10-05T19:25:57.006 INFO:teuthology.orchestra.run.smithi167.stdout:Selecting previously unselected package python3-webob. 2023-10-05T19:25:57.024 INFO:teuthology.orchestra.run.smithi167.stdout:Preparing to unpack .../037-python3-webob_1%3a1.8.5-2_all.deb ... 2023-10-05T19:25:57.073 INFO:teuthology.orchestra.run.smithi118.stdout:Selecting previously unselected package python3-jwt. 2023-10-05T19:25:57.080 INFO:teuthology.orchestra.run.smithi167.stdout:Unpacking python3-webob (1:1.8.5-2) ... 2023-10-05T19:25:57.089 INFO:teuthology.orchestra.run.smithi118.stdout:Preparing to unpack .../031-python3-jwt_1.7.1-2ubuntu2.1_all.deb ... 2023-10-05T19:25:57.131 INFO:teuthology.orchestra.run.smithi118.stdout:Unpacking python3-jwt (1.7.1-2ubuntu2.1) ... 2023-10-05T19:25:57.377 INFO:teuthology.orchestra.run.smithi167.stdout:Selecting previously unselected package python3-soupsieve. 2023-10-05T19:25:57.393 INFO:teuthology.orchestra.run.smithi167.stdout:Preparing to unpack .../038-python3-soupsieve_1.9.5+dfsg-1_all.deb ... 2023-10-05T19:25:57.426 INFO:teuthology.orchestra.run.smithi118.stdout:Selecting previously unselected package python3-logutils. 2023-10-05T19:25:57.435 INFO:teuthology.orchestra.run.smithi167.stdout:Unpacking python3-soupsieve (1.9.5+dfsg-1) ... 2023-10-05T19:25:57.442 INFO:teuthology.orchestra.run.smithi118.stdout:Preparing to unpack .../032-python3-logutils_0.3.3-7_all.deb ... 2023-10-05T19:25:57.484 INFO:teuthology.orchestra.run.smithi118.stdout:Unpacking python3-logutils (0.3.3-7) ... 2023-10-05T19:25:57.738 INFO:teuthology.orchestra.run.smithi167.stdout:Selecting previously unselected package python3-bs4. 2023-10-05T19:25:57.754 INFO:teuthology.orchestra.run.smithi167.stdout:Preparing to unpack .../039-python3-bs4_4.8.2-1_all.deb ... 2023-10-05T19:25:57.796 INFO:teuthology.orchestra.run.smithi167.stdout:Unpacking python3-bs4 (4.8.2-1) ... 2023-10-05T19:25:57.813 INFO:teuthology.orchestra.run.smithi118.stdout:Selecting previously unselected package python3-markupsafe. 2023-10-05T19:25:57.829 INFO:teuthology.orchestra.run.smithi118.stdout:Preparing to unpack .../033-python3-markupsafe_1.1.0-1build2_amd64.deb ... 2023-10-05T19:25:57.871 INFO:teuthology.orchestra.run.smithi118.stdout:Unpacking python3-markupsafe (1.1.0-1build2) ... 2023-10-05T19:25:58.150 INFO:teuthology.orchestra.run.smithi167.stdout:Selecting previously unselected package python3-waitress. 2023-10-05T19:25:58.167 INFO:teuthology.orchestra.run.smithi167.stdout:Preparing to unpack .../040-python3-waitress_1.4.1-1ubuntu0.1_all.deb ... 2023-10-05T19:25:58.174 INFO:teuthology.orchestra.run.smithi118.stdout:Selecting previously unselected package python3-mako. 2023-10-05T19:25:58.191 INFO:teuthology.orchestra.run.smithi118.stdout:Preparing to unpack .../034-python3-mako_1.1.0+ds1-1ubuntu2.1_all.deb ... 2023-10-05T19:25:58.211 INFO:teuthology.orchestra.run.smithi167.stdout:Unpacking python3-waitress (1.4.1-1ubuntu0.1) ... 2023-10-05T19:25:58.232 INFO:teuthology.orchestra.run.smithi118.stdout:Unpacking python3-mako (1.1.0+ds1-1ubuntu2.1) ... 2023-10-05T19:25:58.503 INFO:teuthology.orchestra.run.smithi167.stdout:Selecting previously unselected package python3-tempita. 2023-10-05T19:25:58.522 INFO:teuthology.orchestra.run.smithi167.stdout:Preparing to unpack .../041-python3-tempita_0.5.2-6_all.deb ... 2023-10-05T19:25:58.561 INFO:teuthology.orchestra.run.smithi167.stdout:Unpacking python3-tempita (0.5.2-6) ... 2023-10-05T19:25:58.586 INFO:teuthology.orchestra.run.smithi118.stdout:Selecting previously unselected package python3-simplegeneric. 2023-10-05T19:25:58.603 INFO:teuthology.orchestra.run.smithi118.stdout:Preparing to unpack .../035-python3-simplegeneric_0.8.1-3_all.deb ... 2023-10-05T19:25:58.644 INFO:teuthology.orchestra.run.smithi118.stdout:Unpacking python3-simplegeneric (0.8.1-3) ... 2023-10-05T19:25:58.864 INFO:teuthology.orchestra.run.smithi167.stdout:Selecting previously unselected package python3-paste. 2023-10-05T19:25:58.883 INFO:teuthology.orchestra.run.smithi167.stdout:Preparing to unpack .../042-python3-paste_3.4.0+dfsg1-1_all.deb ... 2023-10-05T19:25:58.922 INFO:teuthology.orchestra.run.smithi167.stdout:Unpacking python3-paste (3.4.0+dfsg1-1) ... 2023-10-05T19:25:58.939 INFO:teuthology.orchestra.run.smithi118.stdout:Selecting previously unselected package python3-singledispatch. 2023-10-05T19:25:58.955 INFO:teuthology.orchestra.run.smithi118.stdout:Preparing to unpack .../036-python3-singledispatch_3.4.0.3-2build1_all.deb ... 2023-10-05T19:25:58.997 INFO:teuthology.orchestra.run.smithi118.stdout:Unpacking python3-singledispatch (3.4.0.3-2build1) ... 2023-10-05T19:25:59.333 INFO:teuthology.orchestra.run.smithi118.stdout:Selecting previously unselected package python3-webob. 2023-10-05T19:25:59.353 INFO:teuthology.orchestra.run.smithi118.stdout:Preparing to unpack .../037-python3-webob_1%3a1.8.5-2_all.deb ... 2023-10-05T19:25:59.935 INFO:teuthology.orchestra.run.smithi167.stdout:Selecting previously unselected package python-pastedeploy-tpl. 2023-10-05T19:25:59.954 INFO:teuthology.orchestra.run.smithi167.stdout:Preparing to unpack .../043-python-pastedeploy-tpl_2.1.0-1_all.deb ... 2023-10-05T19:25:59.967 INFO:teuthology.orchestra.run.smithi118.stdout:Unpacking python3-webob (1:1.8.5-2) ... 2023-10-05T19:25:59.992 INFO:teuthology.orchestra.run.smithi167.stdout:Unpacking python-pastedeploy-tpl (2.1.0-1) ... 2023-10-05T19:26:00.279 INFO:teuthology.orchestra.run.smithi167.stdout:Selecting previously unselected package python3-pastedeploy. 2023-10-05T19:26:00.299 INFO:teuthology.orchestra.run.smithi167.stdout:Preparing to unpack .../044-python3-pastedeploy_2.1.0-1_all.deb ... 2023-10-05T19:26:00.337 INFO:teuthology.orchestra.run.smithi167.stdout:Unpacking python3-pastedeploy (2.1.0-1) ... 2023-10-05T19:26:00.445 INFO:teuthology.orchestra.run.smithi118.stdout:Selecting previously unselected package python3-soupsieve. 2023-10-05T19:26:00.461 INFO:teuthology.orchestra.run.smithi118.stdout:Preparing to unpack .../038-python3-soupsieve_1.9.5+dfsg-1_all.deb ... 2023-10-05T19:26:00.503 INFO:teuthology.orchestra.run.smithi118.stdout:Unpacking python3-soupsieve (1.9.5+dfsg-1) ... 2023-10-05T19:26:00.632 INFO:teuthology.orchestra.run.smithi167.stdout:Selecting previously unselected package python3-webtest. 2023-10-05T19:26:00.648 INFO:teuthology.orchestra.run.smithi167.stdout:Preparing to unpack .../045-python3-webtest_2.0.32-2_all.deb ... 2023-10-05T19:26:00.682 INFO:teuthology.orchestra.run.smithi167.stdout:Unpacking python3-webtest (2.0.32-2) ... 2023-10-05T19:26:00.840 INFO:teuthology.orchestra.run.smithi118.stdout:Selecting previously unselected package python3-bs4. 2023-10-05T19:26:00.856 INFO:teuthology.orchestra.run.smithi118.stdout:Preparing to unpack .../039-python3-bs4_4.8.2-1_all.deb ... 2023-10-05T19:26:00.890 INFO:teuthology.orchestra.run.smithi118.stdout:Unpacking python3-bs4 (4.8.2-1) ... 2023-10-05T19:26:00.943 INFO:teuthology.orchestra.run.smithi167.stdout:Selecting previously unselected package python3-pecan. 2023-10-05T19:26:00.959 INFO:teuthology.orchestra.run.smithi167.stdout:Preparing to unpack .../046-python3-pecan_1.3.3-3ubuntu1_all.deb ... 2023-10-05T19:26:00.993 INFO:teuthology.orchestra.run.smithi167.stdout:Unpacking python3-pecan (1.3.3-3ubuntu1) ... 2023-10-05T19:26:01.243 INFO:teuthology.orchestra.run.smithi118.stdout:Selecting previously unselected package python3-waitress. 2023-10-05T19:26:01.262 INFO:teuthology.orchestra.run.smithi118.stdout:Preparing to unpack .../040-python3-waitress_1.4.1-1ubuntu0.1_all.deb ... 2023-10-05T19:26:01.304 INFO:teuthology.orchestra.run.smithi118.stdout:Unpacking python3-waitress (1.4.1-1ubuntu0.1) ... 2023-10-05T19:26:01.373 INFO:teuthology.orchestra.run.smithi167.stdout:Selecting previously unselected package python3-werkzeug. 2023-10-05T19:26:01.389 INFO:teuthology.orchestra.run.smithi167.stdout:Preparing to unpack .../047-python3-werkzeug_0.16.1+dfsg1-2ubuntu0.1_all.deb ... 2023-10-05T19:26:01.431 INFO:teuthology.orchestra.run.smithi167.stdout:Unpacking python3-werkzeug (0.16.1+dfsg1-2ubuntu0.1) ... 2023-10-05T19:26:01.630 INFO:teuthology.orchestra.run.smithi118.stdout:Selecting previously unselected package python3-tempita. 2023-10-05T19:26:01.649 INFO:teuthology.orchestra.run.smithi118.stdout:Preparing to unpack .../041-python3-tempita_0.5.2-6_all.deb ... 2023-10-05T19:26:01.688 INFO:teuthology.orchestra.run.smithi118.stdout:Unpacking python3-tempita (0.5.2-6) ... 2023-10-05T19:26:01.785 INFO:teuthology.orchestra.run.smithi167.stdout:Selecting previously unselected package libsqlite3-mod-ceph. 2023-10-05T19:26:01.801 INFO:teuthology.orchestra.run.smithi167.stdout:Preparing to unpack .../048-libsqlite3-mod-ceph_17.2.6-1278-g487ffcb5-1focal_amd64.deb ... 2023-10-05T19:26:01.843 INFO:teuthology.orchestra.run.smithi167.stdout:Unpacking libsqlite3-mod-ceph (17.2.6-1278-g487ffcb5-1focal) ... 2023-10-05T19:26:01.974 INFO:teuthology.orchestra.run.smithi118.stdout:Selecting previously unselected package python3-paste. 2023-10-05T19:26:01.993 INFO:teuthology.orchestra.run.smithi118.stdout:Preparing to unpack .../042-python3-paste_3.4.0+dfsg1-1_all.deb ... 2023-10-05T19:26:02.032 INFO:teuthology.orchestra.run.smithi118.stdout:Unpacking python3-paste (3.4.0+dfsg1-1) ... 2023-10-05T19:26:02.197 INFO:teuthology.orchestra.run.smithi167.stdout:Selecting previously unselected package ceph-mgr. 2023-10-05T19:26:02.213 INFO:teuthology.orchestra.run.smithi167.stdout:Preparing to unpack .../049-ceph-mgr_17.2.6-1278-g487ffcb5-1focal_amd64.deb ... 2023-10-05T19:26:02.254 INFO:teuthology.orchestra.run.smithi167.stdout:Unpacking ceph-mgr (17.2.6-1278-g487ffcb5-1focal) ... 2023-10-05T19:26:02.361 INFO:teuthology.orchestra.run.smithi118.stdout:Selecting previously unselected package python-pastedeploy-tpl. 2023-10-05T19:26:02.377 INFO:teuthology.orchestra.run.smithi118.stdout:Preparing to unpack .../043-python-pastedeploy-tpl_2.1.0-1_all.deb ... 2023-10-05T19:26:02.421 INFO:teuthology.orchestra.run.smithi118.stdout:Unpacking python-pastedeploy-tpl (2.1.0-1) ... 2023-10-05T19:26:02.683 INFO:teuthology.orchestra.run.smithi167.stdout:Selecting previously unselected package ceph-mon. 2023-10-05T19:26:02.700 INFO:teuthology.orchestra.run.smithi167.stdout:Preparing to unpack .../050-ceph-mon_17.2.6-1278-g487ffcb5-1focal_amd64.deb ... 2023-10-05T19:26:02.731 INFO:teuthology.orchestra.run.smithi118.stdout:Selecting previously unselected package python3-pastedeploy. 2023-10-05T19:26:02.732 INFO:teuthology.orchestra.run.smithi167.stdout:Unpacking ceph-mon (17.2.6-1278-g487ffcb5-1focal) ... 2023-10-05T19:26:02.748 INFO:teuthology.orchestra.run.smithi118.stdout:Preparing to unpack .../044-python3-pastedeploy_2.1.0-1_all.deb ... 2023-10-05T19:26:02.789 INFO:teuthology.orchestra.run.smithi118.stdout:Unpacking python3-pastedeploy (2.1.0-1) ... 2023-10-05T19:26:03.109 INFO:teuthology.orchestra.run.smithi118.stdout:Selecting previously unselected package python3-webtest. 2023-10-05T19:26:03.129 INFO:teuthology.orchestra.run.smithi118.stdout:Preparing to unpack .../045-python3-webtest_2.0.32-2_all.deb ... 2023-10-05T19:26:03.167 INFO:teuthology.orchestra.run.smithi118.stdout:Unpacking python3-webtest (2.0.32-2) ... 2023-10-05T19:26:03.470 INFO:teuthology.orchestra.run.smithi118.stdout:Selecting previously unselected package python3-pecan. 2023-10-05T19:26:03.486 INFO:teuthology.orchestra.run.smithi118.stdout:Preparing to unpack .../046-python3-pecan_1.3.3-3ubuntu1_all.deb ... 2023-10-05T19:26:03.528 INFO:teuthology.orchestra.run.smithi118.stdout:Unpacking python3-pecan (1.3.3-3ubuntu1) ... 2023-10-05T19:26:03.878 INFO:teuthology.orchestra.run.smithi167.stdout:Selecting previously unselected package ceph-osd. 2023-10-05T19:26:03.894 INFO:teuthology.orchestra.run.smithi167.stdout:Preparing to unpack .../051-ceph-osd_17.2.6-1278-g487ffcb5-1focal_amd64.deb ... 2023-10-05T19:26:03.925 INFO:teuthology.orchestra.run.smithi118.stdout:Selecting previously unselected package python3-werkzeug. 2023-10-05T19:26:03.936 INFO:teuthology.orchestra.run.smithi167.stdout:Unpacking ceph-osd (17.2.6-1278-g487ffcb5-1focal) ... 2023-10-05T19:26:03.943 INFO:teuthology.orchestra.run.smithi118.stdout:Preparing to unpack .../047-python3-werkzeug_0.16.1+dfsg1-2ubuntu0.1_all.deb ... 2023-10-05T19:26:03.984 INFO:teuthology.orchestra.run.smithi118.stdout:Unpacking python3-werkzeug (0.16.1+dfsg1-2ubuntu0.1) ... 2023-10-05T19:26:04.329 INFO:teuthology.orchestra.run.smithi118.stdout:Selecting previously unselected package libsqlite3-mod-ceph. 2023-10-05T19:26:04.347 INFO:teuthology.orchestra.run.smithi118.stdout:Preparing to unpack .../048-libsqlite3-mod-ceph_17.2.6-1278-g487ffcb5-1focal_amd64.deb ... 2023-10-05T19:26:04.387 INFO:teuthology.orchestra.run.smithi118.stdout:Unpacking libsqlite3-mod-ceph (17.2.6-1278-g487ffcb5-1focal) ... 2023-10-05T19:26:04.734 INFO:teuthology.orchestra.run.smithi118.stdout:Selecting previously unselected package ceph-mgr. 2023-10-05T19:26:04.751 INFO:teuthology.orchestra.run.smithi118.stdout:Preparing to unpack .../049-ceph-mgr_17.2.6-1278-g487ffcb5-1focal_amd64.deb ... 2023-10-05T19:26:04.792 INFO:teuthology.orchestra.run.smithi118.stdout:Unpacking ceph-mgr (17.2.6-1278-g487ffcb5-1focal) ... 2023-10-05T19:26:05.354 INFO:teuthology.orchestra.run.smithi118.stdout:Selecting previously unselected package ceph-mon. 2023-10-05T19:26:05.371 INFO:teuthology.orchestra.run.smithi118.stdout:Preparing to unpack .../050-ceph-mon_17.2.6-1278-g487ffcb5-1focal_amd64.deb ... 2023-10-05T19:26:05.412 INFO:teuthology.orchestra.run.smithi118.stdout:Unpacking ceph-mon (17.2.6-1278-g487ffcb5-1focal) ... 2023-10-05T19:26:06.541 INFO:teuthology.orchestra.run.smithi118.stdout:Selecting previously unselected package ceph-osd. 2023-10-05T19:26:06.557 INFO:teuthology.orchestra.run.smithi118.stdout:Preparing to unpack .../051-ceph-osd_17.2.6-1278-g487ffcb5-1focal_amd64.deb ... 2023-10-05T19:26:06.632 INFO:teuthology.orchestra.run.smithi118.stdout:Unpacking ceph-osd (17.2.6-1278-g487ffcb5-1focal) ... 2023-10-05T19:26:06.673 INFO:teuthology.orchestra.run.smithi167.stdout:Selecting previously unselected package ceph. 2023-10-05T19:26:06.690 INFO:teuthology.orchestra.run.smithi167.stdout:Preparing to unpack .../052-ceph_17.2.6-1278-g487ffcb5-1focal_amd64.deb ... 2023-10-05T19:26:06.723 INFO:teuthology.orchestra.run.smithi167.stdout:Unpacking ceph (17.2.6-1278-g487ffcb5-1focal) ... 2023-10-05T19:26:07.034 INFO:teuthology.orchestra.run.smithi167.stdout:Selecting previously unselected package ceph-fuse. 2023-10-05T19:26:07.052 INFO:teuthology.orchestra.run.smithi167.stdout:Preparing to unpack .../053-ceph-fuse_17.2.6-1278-g487ffcb5-1focal_amd64.deb ... 2023-10-05T19:26:07.108 INFO:teuthology.orchestra.run.smithi167.stdout:Unpacking ceph-fuse (17.2.6-1278-g487ffcb5-1focal) ... 2023-10-05T19:26:07.512 INFO:teuthology.orchestra.run.smithi167.stdout:Selecting previously unselected package ceph-mds. 2023-10-05T19:26:07.531 INFO:teuthology.orchestra.run.smithi167.stdout:Preparing to unpack .../054-ceph-mds_17.2.6-1278-g487ffcb5-1focal_amd64.deb ... 2023-10-05T19:26:07.570 INFO:teuthology.orchestra.run.smithi167.stdout:Unpacking ceph-mds (17.2.6-1278-g487ffcb5-1focal) ... 2023-10-05T19:26:08.073 INFO:teuthology.orchestra.run.smithi167.stdout:Selecting previously unselected package cephadm. 2023-10-05T19:26:08.089 INFO:teuthology.orchestra.run.smithi167.stdout:Preparing to unpack .../055-cephadm_17.2.6-1278-g487ffcb5-1focal_amd64.deb ... 2023-10-05T19:26:08.123 INFO:teuthology.orchestra.run.smithi167.stdout:Unpacking cephadm (17.2.6-1278-g487ffcb5-1focal) ... 2023-10-05T19:26:08.434 INFO:teuthology.orchestra.run.smithi167.stdout:Selecting previously unselected package python3-jinja2. 2023-10-05T19:26:08.450 INFO:teuthology.orchestra.run.smithi167.stdout:Preparing to unpack .../056-python3-jinja2_2.10.1-2_all.deb ... 2023-10-05T19:26:08.493 INFO:teuthology.orchestra.run.smithi167.stdout:Unpacking python3-jinja2 (2.10.1-2) ... 2023-10-05T19:26:08.846 INFO:teuthology.orchestra.run.smithi167.stdout:Selecting previously unselected package ceph-mgr-cephadm. 2023-10-05T19:26:08.862 INFO:teuthology.orchestra.run.smithi167.stdout:Preparing to unpack .../057-ceph-mgr-cephadm_17.2.6-1278-g487ffcb5-1focal_all.deb ... 2023-10-05T19:26:08.904 INFO:teuthology.orchestra.run.smithi167.stdout:Unpacking ceph-mgr-cephadm (17.2.6-1278-g487ffcb5-1focal) ... 2023-10-05T19:26:09.241 INFO:teuthology.orchestra.run.smithi167.stdout:Selecting previously unselected package python3-repoze.lru. 2023-10-05T19:26:09.257 INFO:teuthology.orchestra.run.smithi167.stdout:Preparing to unpack .../058-python3-repoze.lru_0.7-2_all.deb ... 2023-10-05T19:26:09.299 INFO:teuthology.orchestra.run.smithi167.stdout:Unpacking python3-repoze.lru (0.7-2) ... 2023-10-05T19:26:09.536 INFO:teuthology.orchestra.run.smithi118.stdout:Selecting previously unselected package ceph. 2023-10-05T19:26:09.552 INFO:teuthology.orchestra.run.smithi118.stdout:Preparing to unpack .../052-ceph_17.2.6-1278-g487ffcb5-1focal_amd64.deb ... 2023-10-05T19:26:09.570 INFO:teuthology.orchestra.run.smithi167.stdout:Selecting previously unselected package libjs-underscore. 2023-10-05T19:26:09.589 INFO:teuthology.orchestra.run.smithi167.stdout:Preparing to unpack .../059-libjs-underscore_1.9.1~dfsg-1ubuntu0.20.04.1_all.deb ... 2023-10-05T19:26:09.593 INFO:teuthology.orchestra.run.smithi118.stdout:Unpacking ceph (17.2.6-1278-g487ffcb5-1focal) ... 2023-10-05T19:26:09.635 INFO:teuthology.orchestra.run.smithi167.stdout:Unpacking libjs-underscore (1.9.1~dfsg-1ubuntu0.20.04.1) ... 2023-10-05T19:26:09.913 INFO:teuthology.orchestra.run.smithi167.stdout:Selecting previously unselected package libjs-sphinxdoc. 2023-10-05T19:26:09.929 INFO:teuthology.orchestra.run.smithi167.stdout:Preparing to unpack .../060-libjs-sphinxdoc_1.8.5-7ubuntu3_all.deb ... 2023-10-05T19:26:09.938 INFO:teuthology.orchestra.run.smithi118.stdout:Selecting previously unselected package ceph-fuse. 2023-10-05T19:26:09.955 INFO:teuthology.orchestra.run.smithi118.stdout:Preparing to unpack .../053-ceph-fuse_17.2.6-1278-g487ffcb5-1focal_amd64.deb ... 2023-10-05T19:26:09.970 INFO:teuthology.orchestra.run.smithi167.stdout:Unpacking libjs-sphinxdoc (1.8.5-7ubuntu3) ... 2023-10-05T19:26:09.996 INFO:teuthology.orchestra.run.smithi118.stdout:Unpacking ceph-fuse (17.2.6-1278-g487ffcb5-1focal) ... 2023-10-05T19:26:10.357 INFO:teuthology.orchestra.run.smithi167.stdout:Selecting previously unselected package python3-routes. 2023-10-05T19:26:10.373 INFO:teuthology.orchestra.run.smithi167.stdout:Preparing to unpack .../061-python3-routes_2.4.1-2_all.deb ... 2023-10-05T19:26:10.432 INFO:teuthology.orchestra.run.smithi167.stdout:Unpacking python3-routes (2.4.1-2) ... 2023-10-05T19:26:10.559 INFO:teuthology.orchestra.run.smithi118.stdout:Selecting previously unselected package ceph-mds. 2023-10-05T19:26:10.575 INFO:teuthology.orchestra.run.smithi118.stdout:Preparing to unpack .../054-ceph-mds_17.2.6-1278-g487ffcb5-1focal_amd64.deb ... 2023-10-05T19:26:10.616 INFO:teuthology.orchestra.run.smithi118.stdout:Unpacking ceph-mds (17.2.6-1278-g487ffcb5-1focal) ... 2023-10-05T19:26:10.743 INFO:teuthology.orchestra.run.smithi167.stdout:Selecting previously unselected package ceph-mgr-dashboard. 2023-10-05T19:26:10.763 INFO:teuthology.orchestra.run.smithi167.stdout:Preparing to unpack .../062-ceph-mgr-dashboard_17.2.6-1278-g487ffcb5-1focal_all.deb ... 2023-10-05T19:26:10.801 INFO:teuthology.orchestra.run.smithi167.stdout:Unpacking ceph-mgr-dashboard (17.2.6-1278-g487ffcb5-1focal) ... 2023-10-05T19:26:11.136 INFO:teuthology.orchestra.run.smithi118.stdout:Selecting previously unselected package cephadm. 2023-10-05T19:26:11.152 INFO:teuthology.orchestra.run.smithi118.stdout:Preparing to unpack .../055-cephadm_17.2.6-1278-g487ffcb5-1focal_amd64.deb ... 2023-10-05T19:26:11.194 INFO:teuthology.orchestra.run.smithi118.stdout:Unpacking cephadm (17.2.6-1278-g487ffcb5-1focal) ... 2023-10-05T19:26:11.322 INFO:teuthology.orchestra.run.smithi167.stdout:Selecting previously unselected package python3-decorator. 2023-10-05T19:26:11.338 INFO:teuthology.orchestra.run.smithi167.stdout:Preparing to unpack .../063-python3-decorator_4.4.2-0ubuntu1_all.deb ... 2023-10-05T19:26:11.380 INFO:teuthology.orchestra.run.smithi167.stdout:Unpacking python3-decorator (4.4.2-0ubuntu1) ... 2023-10-05T19:26:11.481 INFO:teuthology.orchestra.run.smithi118.stdout:Selecting previously unselected package python3-jinja2. 2023-10-05T19:26:11.497 INFO:teuthology.orchestra.run.smithi118.stdout:Preparing to unpack .../056-python3-jinja2_2.10.1-2_all.deb ... 2023-10-05T19:26:11.539 INFO:teuthology.orchestra.run.smithi118.stdout:Unpacking python3-jinja2 (2.10.1-2) ... 2023-10-05T19:26:11.683 INFO:teuthology.orchestra.run.smithi167.stdout:Selecting previously unselected package liblbfgsb0:amd64. 2023-10-05T19:26:11.700 INFO:teuthology.orchestra.run.smithi167.stdout:Preparing to unpack .../064-liblbfgsb0_3.0+dfsg.3-7build1_amd64.deb ... 2023-10-05T19:26:11.741 INFO:teuthology.orchestra.run.smithi167.stdout:Unpacking liblbfgsb0:amd64 (3.0+dfsg.3-7build1) ... 2023-10-05T19:26:11.859 INFO:teuthology.orchestra.run.smithi118.stdout:Selecting previously unselected package ceph-mgr-cephadm. 2023-10-05T19:26:11.878 INFO:teuthology.orchestra.run.smithi118.stdout:Preparing to unpack .../057-ceph-mgr-cephadm_17.2.6-1278-g487ffcb5-1focal_all.deb ... 2023-10-05T19:26:11.917 INFO:teuthology.orchestra.run.smithi118.stdout:Unpacking ceph-mgr-cephadm (17.2.6-1278-g487ffcb5-1focal) ... 2023-10-05T19:26:12.077 INFO:teuthology.orchestra.run.smithi167.stdout:Selecting previously unselected package python3-scipy. 2023-10-05T19:26:12.093 INFO:teuthology.orchestra.run.smithi167.stdout:Preparing to unpack .../065-python3-scipy_1.3.3-3build1_amd64.deb ... 2023-10-05T19:26:12.144 INFO:teuthology.orchestra.run.smithi167.stdout:Unpacking python3-scipy (1.3.3-3build1) ... 2023-10-05T19:26:12.237 INFO:teuthology.orchestra.run.smithi118.stdout:Selecting previously unselected package python3-repoze.lru. 2023-10-05T19:26:12.255 INFO:teuthology.orchestra.run.smithi118.stdout:Preparing to unpack .../058-python3-repoze.lru_0.7-2_all.deb ... 2023-10-05T19:26:12.295 INFO:teuthology.orchestra.run.smithi118.stdout:Unpacking python3-repoze.lru (0.7-2) ... 2023-10-05T19:26:12.548 INFO:teuthology.orchestra.run.smithi118.stdout:Selecting previously unselected package libjs-underscore. 2023-10-05T19:26:12.565 INFO:teuthology.orchestra.run.smithi118.stdout:Preparing to unpack .../059-libjs-underscore_1.9.1~dfsg-1ubuntu0.20.04.1_all.deb ... 2023-10-05T19:26:12.614 INFO:teuthology.orchestra.run.smithi118.stdout:Unpacking libjs-underscore (1.9.1~dfsg-1ubuntu0.20.04.1) ... 2023-10-05T19:26:12.892 INFO:teuthology.orchestra.run.smithi118.stdout:Selecting previously unselected package libjs-sphinxdoc. 2023-10-05T19:26:12.908 INFO:teuthology.orchestra.run.smithi118.stdout:Preparing to unpack .../060-libjs-sphinxdoc_1.8.5-7ubuntu3_all.deb ... 2023-10-05T19:26:12.950 INFO:teuthology.orchestra.run.smithi118.stdout:Unpacking libjs-sphinxdoc (1.8.5-7ubuntu3) ... 2023-10-05T19:26:13.270 INFO:teuthology.orchestra.run.smithi118.stdout:Selecting previously unselected package python3-routes. 2023-10-05T19:26:13.287 INFO:teuthology.orchestra.run.smithi118.stdout:Preparing to unpack .../061-python3-routes_2.4.1-2_all.deb ... 2023-10-05T19:26:13.353 INFO:teuthology.orchestra.run.smithi118.stdout:Unpacking python3-routes (2.4.1-2) ... 2023-10-05T19:26:13.486 INFO:teuthology.orchestra.run.smithi167.stdout:Selecting previously unselected package python3-sklearn-lib. 2023-10-05T19:26:13.502 INFO:teuthology.orchestra.run.smithi167.stdout:Preparing to unpack .../066-python3-sklearn-lib_0.22.2.post1+dfsg-5_amd64.deb ... 2023-10-05T19:26:13.544 INFO:teuthology.orchestra.run.smithi167.stdout:Unpacking python3-sklearn-lib (0.22.2.post1+dfsg-5) ... 2023-10-05T19:26:13.681 INFO:teuthology.orchestra.run.smithi118.stdout:Selecting previously unselected package ceph-mgr-dashboard. 2023-10-05T19:26:13.697 INFO:teuthology.orchestra.run.smithi118.stdout:Preparing to unpack .../062-ceph-mgr-dashboard_17.2.6-1278-g487ffcb5-1focal_all.deb ... 2023-10-05T19:26:13.739 INFO:teuthology.orchestra.run.smithi118.stdout:Unpacking ceph-mgr-dashboard (17.2.6-1278-g487ffcb5-1focal) ... 2023-10-05T19:26:14.107 INFO:teuthology.orchestra.run.smithi167.stdout:Selecting previously unselected package python3-joblib. 2023-10-05T19:26:14.124 INFO:teuthology.orchestra.run.smithi167.stdout:Preparing to unpack .../067-python3-joblib_0.14.0-3_all.deb ... 2023-10-05T19:26:14.164 INFO:teuthology.orchestra.run.smithi167.stdout:Unpacking python3-joblib (0.14.0-3) ... 2023-10-05T19:26:14.243 INFO:teuthology.orchestra.run.smithi118.stdout:Selecting previously unselected package python3-decorator. 2023-10-05T19:26:14.259 INFO:teuthology.orchestra.run.smithi118.stdout:Preparing to unpack .../063-python3-decorator_4.4.2-0ubuntu1_all.deb ... 2023-10-05T19:26:14.301 INFO:teuthology.orchestra.run.smithi118.stdout:Unpacking python3-decorator (4.4.2-0ubuntu1) ... 2023-10-05T19:26:14.577 INFO:teuthology.orchestra.run.smithi167.stdout:Selecting previously unselected package python3-sklearn. 2023-10-05T19:26:14.596 INFO:teuthology.orchestra.run.smithi118.stdout:Selecting previously unselected package liblbfgsb0:amd64. 2023-10-05T19:26:14.596 INFO:teuthology.orchestra.run.smithi167.stdout:Preparing to unpack .../068-python3-sklearn_0.22.2.post1+dfsg-5_all.deb ... 2023-10-05T19:26:14.612 INFO:teuthology.orchestra.run.smithi118.stdout:Preparing to unpack .../064-liblbfgsb0_3.0+dfsg.3-7build1_amd64.deb ... 2023-10-05T19:26:14.634 INFO:teuthology.orchestra.run.smithi167.stdout:Unpacking python3-sklearn (0.22.2.post1+dfsg-5) ... 2023-10-05T19:26:14.653 INFO:teuthology.orchestra.run.smithi118.stdout:Unpacking liblbfgsb0:amd64 (3.0+dfsg.3-7build1) ... 2023-10-05T19:26:15.024 INFO:teuthology.orchestra.run.smithi118.stdout:Selecting previously unselected package python3-scipy. 2023-10-05T19:26:15.040 INFO:teuthology.orchestra.run.smithi118.stdout:Preparing to unpack .../065-python3-scipy_1.3.3-3build1_amd64.deb ... 2023-10-05T19:26:15.082 INFO:teuthology.orchestra.run.smithi118.stdout:Unpacking python3-scipy (1.3.3-3build1) ... 2023-10-05T19:26:15.294 INFO:teuthology.orchestra.run.smithi167.stdout:Selecting previously unselected package ceph-mgr-diskprediction-local. 2023-10-05T19:26:15.313 INFO:teuthology.orchestra.run.smithi167.stdout:Preparing to unpack .../069-ceph-mgr-diskprediction-local_17.2.6-1278-g487ffcb5-1focal_all.deb ... 2023-10-05T19:26:15.687 INFO:teuthology.orchestra.run.smithi167.stdout:Unpacking ceph-mgr-diskprediction-local (17.2.6-1278-g487ffcb5-1focal) ... 2023-10-05T19:26:16.708 INFO:teuthology.orchestra.run.smithi118.stdout:Selecting previously unselected package python3-sklearn-lib. 2023-10-05T19:26:16.725 INFO:teuthology.orchestra.run.smithi118.stdout:Preparing to unpack .../066-python3-sklearn-lib_0.22.2.post1+dfsg-5_amd64.deb ... 2023-10-05T19:26:16.766 INFO:teuthology.orchestra.run.smithi118.stdout:Unpacking python3-sklearn-lib (0.22.2.post1+dfsg-5) ... 2023-10-05T19:26:16.830 INFO:teuthology.orchestra.run.smithi167.stdout:Selecting previously unselected package python3-cachetools. 2023-10-05T19:26:16.850 INFO:teuthology.orchestra.run.smithi167.stdout:Preparing to unpack .../070-python3-cachetools_4.0.0-1_all.deb ... 2023-10-05T19:26:16.888 INFO:teuthology.orchestra.run.smithi167.stdout:Unpacking python3-cachetools (4.0.0-1) ... 2023-10-05T19:26:17.199 INFO:teuthology.orchestra.run.smithi167.stdout:Selecting previously unselected package python3-pyasn1. 2023-10-05T19:26:17.215 INFO:teuthology.orchestra.run.smithi167.stdout:Preparing to unpack .../071-python3-pyasn1_0.4.2-3build1_all.deb ... 2023-10-05T19:26:17.257 INFO:teuthology.orchestra.run.smithi167.stdout:Unpacking python3-pyasn1 (0.4.2-3build1) ... 2023-10-05T19:26:17.295 INFO:teuthology.orchestra.run.smithi118.stdout:Selecting previously unselected package python3-joblib. 2023-10-05T19:26:17.314 INFO:teuthology.orchestra.run.smithi118.stdout:Preparing to unpack .../067-python3-joblib_0.14.0-3_all.deb ... 2023-10-05T19:26:17.344 INFO:teuthology.orchestra.run.smithi118.stdout:Unpacking python3-joblib (0.14.0-3) ... 2023-10-05T19:26:17.620 INFO:teuthology.orchestra.run.smithi167.stdout:Selecting previously unselected package python3-pyasn1-modules. 2023-10-05T19:26:17.636 INFO:teuthology.orchestra.run.smithi167.stdout:Preparing to unpack .../072-python3-pyasn1-modules_0.2.1-0.2build1_all.deb ... 2023-10-05T19:26:17.677 INFO:teuthology.orchestra.run.smithi167.stdout:Unpacking python3-pyasn1-modules (0.2.1-0.2build1) ... 2023-10-05T19:26:17.706 INFO:teuthology.orchestra.run.smithi118.stdout:Selecting previously unselected package python3-sklearn. 2023-10-05T19:26:17.723 INFO:teuthology.orchestra.run.smithi118.stdout:Preparing to unpack .../068-python3-sklearn_0.22.2.post1+dfsg-5_all.deb ... 2023-10-05T19:26:17.764 INFO:teuthology.orchestra.run.smithi118.stdout:Unpacking python3-sklearn (0.22.2.post1+dfsg-5) ... 2023-10-05T19:26:17.989 INFO:teuthology.orchestra.run.smithi167.stdout:Selecting previously unselected package python3-rsa. 2023-10-05T19:26:18.008 INFO:teuthology.orchestra.run.smithi167.stdout:Preparing to unpack .../073-python3-rsa_4.0-3_all.deb ... 2023-10-05T19:26:18.047 INFO:teuthology.orchestra.run.smithi167.stdout:Unpacking python3-rsa (4.0-3) ... 2023-10-05T19:26:18.367 INFO:teuthology.orchestra.run.smithi167.stdout:Selecting previously unselected package python3-google-auth. 2023-10-05T19:26:18.387 INFO:teuthology.orchestra.run.smithi167.stdout:Preparing to unpack .../074-python3-google-auth_1.5.1-2_all.deb ... 2023-10-05T19:26:18.425 INFO:teuthology.orchestra.run.smithi167.stdout:Unpacking python3-google-auth (1.5.1-2) ... 2023-10-05T19:26:18.457 INFO:teuthology.orchestra.run.smithi118.stdout:Selecting previously unselected package ceph-mgr-diskprediction-local. 2023-10-05T19:26:18.473 INFO:teuthology.orchestra.run.smithi118.stdout:Preparing to unpack .../069-ceph-mgr-diskprediction-local_17.2.6-1278-g487ffcb5-1focal_all.deb ... 2023-10-05T19:26:18.514 INFO:teuthology.orchestra.run.smithi118.stdout:Unpacking ceph-mgr-diskprediction-local (17.2.6-1278-g487ffcb5-1focal) ... 2023-10-05T19:26:18.746 INFO:teuthology.orchestra.run.smithi167.stdout:Selecting previously unselected package python3-blinker. 2023-10-05T19:26:18.761 INFO:teuthology.orchestra.run.smithi167.stdout:Preparing to unpack .../075-python3-blinker_1.4+dfsg1-0.3ubuntu1_all.deb ... 2023-10-05T19:26:18.803 INFO:teuthology.orchestra.run.smithi167.stdout:Unpacking python3-blinker (1.4+dfsg1-0.3ubuntu1) ... 2023-10-05T19:26:19.123 INFO:teuthology.orchestra.run.smithi167.stdout:Selecting previously unselected package python3-oauthlib. 2023-10-05T19:26:19.140 INFO:teuthology.orchestra.run.smithi167.stdout:Preparing to unpack .../076-python3-oauthlib_3.1.0-1ubuntu2_all.deb ... 2023-10-05T19:26:19.181 INFO:teuthology.orchestra.run.smithi167.stdout:Unpacking python3-oauthlib (3.1.0-1ubuntu2) ... 2023-10-05T19:26:19.534 INFO:teuthology.orchestra.run.smithi167.stdout:Selecting previously unselected package python3-requests-oauthlib. 2023-10-05T19:26:19.550 INFO:teuthology.orchestra.run.smithi167.stdout:Preparing to unpack .../077-python3-requests-oauthlib_1.0.0-1.1build1_all.deb ... 2023-10-05T19:26:19.592 INFO:teuthology.orchestra.run.smithi167.stdout:Unpacking python3-requests-oauthlib (1.0.0-1.1build1) ... 2023-10-05T19:26:19.676 INFO:teuthology.orchestra.run.smithi118.stdout:Selecting previously unselected package python3-cachetools. 2023-10-05T19:26:19.693 INFO:teuthology.orchestra.run.smithi118.stdout:Preparing to unpack .../070-python3-cachetools_4.0.0-1_all.deb ... 2023-10-05T19:26:19.734 INFO:teuthology.orchestra.run.smithi118.stdout:Unpacking python3-cachetools (4.0.0-1) ... 2023-10-05T19:26:19.912 INFO:teuthology.orchestra.run.smithi167.stdout:Selecting previously unselected package python3-websocket. 2023-10-05T19:26:19.932 INFO:teuthology.orchestra.run.smithi167.stdout:Preparing to unpack .../078-python3-websocket_0.53.0-2ubuntu1_all.deb ... 2023-10-05T19:26:19.970 INFO:teuthology.orchestra.run.smithi167.stdout:Unpacking python3-websocket (0.53.0-2ubuntu1) ... 2023-10-05T19:26:20.045 INFO:teuthology.orchestra.run.smithi118.stdout:Selecting previously unselected package python3-pyasn1. 2023-10-05T19:26:20.061 INFO:teuthology.orchestra.run.smithi118.stdout:Preparing to unpack .../071-python3-pyasn1_0.4.2-3build1_all.deb ... 2023-10-05T19:26:20.128 INFO:teuthology.orchestra.run.smithi118.stdout:Unpacking python3-pyasn1 (0.4.2-3build1) ... 2023-10-05T19:26:20.282 INFO:teuthology.orchestra.run.smithi167.stdout:Selecting previously unselected package python3-kubernetes. 2023-10-05T19:26:20.300 INFO:teuthology.orchestra.run.smithi167.stdout:Preparing to unpack .../079-python3-kubernetes_7.0.0~a1-2_all.deb ... 2023-10-05T19:26:20.340 INFO:teuthology.orchestra.run.smithi167.stdout:Unpacking python3-kubernetes (7.0.0~a1-2) ... 2023-10-05T19:26:20.457 INFO:teuthology.orchestra.run.smithi118.stdout:Selecting previously unselected package python3-pyasn1-modules. 2023-10-05T19:26:20.475 INFO:teuthology.orchestra.run.smithi118.stdout:Preparing to unpack .../072-python3-pyasn1-modules_0.2.1-0.2build1_all.deb ... 2023-10-05T19:26:20.515 INFO:teuthology.orchestra.run.smithi118.stdout:Unpacking python3-pyasn1-modules (0.2.1-0.2build1) ... 2023-10-05T19:26:20.860 INFO:teuthology.orchestra.run.smithi118.stdout:Selecting previously unselected package python3-rsa. 2023-10-05T19:26:20.877 INFO:teuthology.orchestra.run.smithi118.stdout:Preparing to unpack .../073-python3-rsa_4.0-3_all.deb ... 2023-10-05T19:26:20.918 INFO:teuthology.orchestra.run.smithi118.stdout:Unpacking python3-rsa (4.0-3) ... 2023-10-05T19:26:20.929 INFO:teuthology.orchestra.run.smithi167.stdout:Selecting previously unselected package ceph-mgr-k8sevents. 2023-10-05T19:26:20.948 INFO:teuthology.orchestra.run.smithi167.stdout:Preparing to unpack .../080-ceph-mgr-k8sevents_17.2.6-1278-g487ffcb5-1focal_all.deb ... 2023-10-05T19:26:20.987 INFO:teuthology.orchestra.run.smithi167.stdout:Unpacking ceph-mgr-k8sevents (17.2.6-1278-g487ffcb5-1focal) ... 2023-10-05T19:26:21.229 INFO:teuthology.orchestra.run.smithi118.stdout:Selecting previously unselected package python3-google-auth. 2023-10-05T19:26:21.246 INFO:teuthology.orchestra.run.smithi118.stdout:Preparing to unpack .../074-python3-google-auth_1.5.1-2_all.deb ... 2023-10-05T19:26:21.279 INFO:teuthology.orchestra.run.smithi118.stdout:Unpacking python3-google-auth (1.5.1-2) ... 2023-10-05T19:26:21.384 INFO:teuthology.orchestra.run.smithi167.stdout:Selecting previously unselected package libonig5:amd64. 2023-10-05T19:26:21.401 INFO:teuthology.orchestra.run.smithi167.stdout:Preparing to unpack .../081-libonig5_6.9.4-1_amd64.deb ... 2023-10-05T19:26:21.441 INFO:teuthology.orchestra.run.smithi167.stdout:Unpacking libonig5:amd64 (6.9.4-1) ... 2023-10-05T19:26:21.633 INFO:teuthology.orchestra.run.smithi118.stdout:Selecting previously unselected package python3-blinker. 2023-10-05T19:26:21.649 INFO:teuthology.orchestra.run.smithi118.stdout:Preparing to unpack .../075-python3-blinker_1.4+dfsg1-0.3ubuntu1_all.deb ... 2023-10-05T19:26:21.699 INFO:teuthology.orchestra.run.smithi118.stdout:Unpacking python3-blinker (1.4+dfsg1-0.3ubuntu1) ... 2023-10-05T19:26:21.828 INFO:teuthology.orchestra.run.smithi167.stdout:Selecting previously unselected package libjq1:amd64. 2023-10-05T19:26:21.845 INFO:teuthology.orchestra.run.smithi167.stdout:Preparing to unpack .../082-libjq1_1.6-1ubuntu0.20.04.1_amd64.deb ... 2023-10-05T19:26:21.887 INFO:teuthology.orchestra.run.smithi167.stdout:Unpacking libjq1:amd64 (1.6-1ubuntu0.20.04.1) ... 2023-10-05T19:26:21.952 INFO:teuthology.orchestra.run.smithi118.stdout:Selecting previously unselected package python3-oauthlib. 2023-10-05T19:26:21.970 INFO:teuthology.orchestra.run.smithi118.stdout:Preparing to unpack .../076-python3-oauthlib_3.1.0-1ubuntu2_all.deb ... 2023-10-05T19:26:22.002 INFO:teuthology.orchestra.run.smithi118.stdout:Unpacking python3-oauthlib (3.1.0-1ubuntu2) ... 2023-10-05T19:26:22.156 INFO:teuthology.orchestra.run.smithi167.stdout:Selecting previously unselected package jq. 2023-10-05T19:26:22.173 INFO:teuthology.orchestra.run.smithi167.stdout:Preparing to unpack .../083-jq_1.6-1ubuntu0.20.04.1_amd64.deb ... 2023-10-05T19:26:22.214 INFO:teuthology.orchestra.run.smithi167.stdout:Unpacking jq (1.6-1ubuntu0.20.04.1) ... 2023-10-05T19:26:22.280 INFO:teuthology.orchestra.run.smithi118.stdout:Selecting previously unselected package python3-requests-oauthlib. 2023-10-05T19:26:22.299 INFO:teuthology.orchestra.run.smithi118.stdout:Preparing to unpack .../077-python3-requests-oauthlib_1.0.0-1.1build1_all.deb ... 2023-10-05T19:26:22.338 INFO:teuthology.orchestra.run.smithi118.stdout:Unpacking python3-requests-oauthlib (1.0.0-1.1build1) ... 2023-10-05T19:26:22.492 INFO:teuthology.orchestra.run.smithi167.stdout:Selecting previously unselected package socat. 2023-10-05T19:26:22.509 INFO:teuthology.orchestra.run.smithi167.stdout:Preparing to unpack .../084-socat_1.7.3.3-2_amd64.deb ... 2023-10-05T19:26:22.550 INFO:teuthology.orchestra.run.smithi167.stdout:Unpacking socat (1.7.3.3-2) ... 2023-10-05T19:26:22.625 INFO:teuthology.orchestra.run.smithi118.stdout:Selecting previously unselected package python3-websocket. 2023-10-05T19:26:22.645 INFO:teuthology.orchestra.run.smithi118.stdout:Preparing to unpack .../078-python3-websocket_0.53.0-2ubuntu1_all.deb ... 2023-10-05T19:26:22.683 INFO:teuthology.orchestra.run.smithi118.stdout:Unpacking python3-websocket (0.53.0-2ubuntu1) ... 2023-10-05T19:26:22.863 INFO:teuthology.orchestra.run.smithi167.stdout:Selecting previously unselected package xmlstarlet. 2023-10-05T19:26:22.879 INFO:teuthology.orchestra.run.smithi167.stdout:Preparing to unpack .../085-xmlstarlet_1.6.1-2build1_amd64.deb ... 2023-10-05T19:26:22.921 INFO:teuthology.orchestra.run.smithi167.stdout:Unpacking xmlstarlet (1.6.1-2build1) ... 2023-10-05T19:26:22.995 INFO:teuthology.orchestra.run.smithi118.stdout:Selecting previously unselected package python3-kubernetes. 2023-10-05T19:26:23.012 INFO:teuthology.orchestra.run.smithi118.stdout:Preparing to unpack .../079-python3-kubernetes_7.0.0~a1-2_all.deb ... 2023-10-05T19:26:23.078 INFO:teuthology.orchestra.run.smithi118.stdout:Unpacking python3-kubernetes (7.0.0~a1-2) ... 2023-10-05T19:26:23.284 INFO:teuthology.orchestra.run.smithi167.stdout:Selecting previously unselected package ceph-test. 2023-10-05T19:26:23.303 INFO:teuthology.orchestra.run.smithi167.stdout:Preparing to unpack .../086-ceph-test_17.2.6-1278-g487ffcb5-1focal_amd64.deb ... 2023-10-05T19:26:23.342 INFO:teuthology.orchestra.run.smithi167.stdout:Unpacking ceph-test (17.2.6-1278-g487ffcb5-1focal) ... 2023-10-05T19:26:23.641 INFO:teuthology.orchestra.run.smithi118.stdout:Selecting previously unselected package ceph-mgr-k8sevents. 2023-10-05T19:26:23.659 INFO:teuthology.orchestra.run.smithi118.stdout:Preparing to unpack .../080-ceph-mgr-k8sevents_17.2.6-1278-g487ffcb5-1focal_all.deb ... 2023-10-05T19:26:23.709 INFO:teuthology.orchestra.run.smithi118.stdout:Unpacking ceph-mgr-k8sevents (17.2.6-1278-g487ffcb5-1focal) ... 2023-10-05T19:26:24.138 INFO:teuthology.orchestra.run.smithi118.stdout:Selecting previously unselected package libonig5:amd64. 2023-10-05T19:26:24.155 INFO:teuthology.orchestra.run.smithi118.stdout:Preparing to unpack .../081-libonig5_6.9.4-1_amd64.deb ... 2023-10-05T19:26:24.196 INFO:teuthology.orchestra.run.smithi118.stdout:Unpacking libonig5:amd64 (6.9.4-1) ... 2023-10-05T19:26:24.583 INFO:teuthology.orchestra.run.smithi118.stdout:Selecting previously unselected package libjq1:amd64. 2023-10-05T19:26:24.600 INFO:teuthology.orchestra.run.smithi118.stdout:Preparing to unpack .../082-libjq1_1.6-1ubuntu0.20.04.1_amd64.deb ... 2023-10-05T19:26:24.641 INFO:teuthology.orchestra.run.smithi118.stdout:Unpacking libjq1:amd64 (1.6-1ubuntu0.20.04.1) ... 2023-10-05T19:26:24.919 INFO:teuthology.orchestra.run.smithi118.stdout:Selecting previously unselected package jq. 2023-10-05T19:26:24.936 INFO:teuthology.orchestra.run.smithi118.stdout:Preparing to unpack .../083-jq_1.6-1ubuntu0.20.04.1_amd64.deb ... 2023-10-05T19:26:24.977 INFO:teuthology.orchestra.run.smithi118.stdout:Unpacking jq (1.6-1ubuntu0.20.04.1) ... 2023-10-05T19:26:25.255 INFO:teuthology.orchestra.run.smithi118.stdout:Selecting previously unselected package socat. 2023-10-05T19:26:25.272 INFO:teuthology.orchestra.run.smithi118.stdout:Preparing to unpack .../084-socat_1.7.3.3-2_amd64.deb ... 2023-10-05T19:26:25.321 INFO:teuthology.orchestra.run.smithi118.stdout:Unpacking socat (1.7.3.3-2) ... 2023-10-05T19:26:25.634 INFO:teuthology.orchestra.run.smithi118.stdout:Selecting previously unselected package xmlstarlet. 2023-10-05T19:26:25.651 INFO:teuthology.orchestra.run.smithi118.stdout:Preparing to unpack .../085-xmlstarlet_1.6.1-2build1_amd64.deb ... 2023-10-05T19:26:25.692 INFO:teuthology.orchestra.run.smithi118.stdout:Unpacking xmlstarlet (1.6.1-2build1) ... 2023-10-05T19:26:26.080 INFO:teuthology.orchestra.run.smithi118.stdout:Selecting previously unselected package ceph-test. 2023-10-05T19:26:26.099 INFO:teuthology.orchestra.run.smithi118.stdout:Preparing to unpack .../086-ceph-test_17.2.6-1278-g487ffcb5-1focal_amd64.deb ... 2023-10-05T19:26:26.138 INFO:teuthology.orchestra.run.smithi118.stdout:Unpacking ceph-test (17.2.6-1278-g487ffcb5-1focal) ... 2023-10-05T19:26:28.372 INFO:teuthology.orchestra.run.smithi167.stdout:Selecting previously unselected package ceph-volume. 2023-10-05T19:26:28.389 INFO:teuthology.orchestra.run.smithi167.stdout:Preparing to unpack .../087-ceph-volume_17.2.6-1278-g487ffcb5-1focal_all.deb ... 2023-10-05T19:26:28.429 INFO:teuthology.orchestra.run.smithi167.stdout:Unpacking ceph-volume (17.2.6-1278-g487ffcb5-1focal) ... 2023-10-05T19:26:28.767 INFO:teuthology.orchestra.run.smithi167.stdout:Selecting previously unselected package fonts-mathjax. 2023-10-05T19:26:28.785 INFO:teuthology.orchestra.run.smithi167.stdout:Preparing to unpack .../088-fonts-mathjax_2.7.4+dfsg-1_all.deb ... 2023-10-05T19:26:28.825 INFO:teuthology.orchestra.run.smithi167.stdout:Unpacking fonts-mathjax (2.7.4+dfsg-1) ... 2023-10-05T19:26:29.363 INFO:teuthology.orchestra.run.smithi167.stdout:Selecting previously unselected package libcephfs-dev. 2023-10-05T19:26:29.383 INFO:teuthology.orchestra.run.smithi167.stdout:Preparing to unpack .../089-libcephfs-dev_17.2.6-1278-g487ffcb5-1focal_amd64.deb ... 2023-10-05T19:26:29.421 INFO:teuthology.orchestra.run.smithi167.stdout:Unpacking libcephfs-dev (17.2.6-1278-g487ffcb5-1focal) ... 2023-10-05T19:26:29.808 INFO:teuthology.orchestra.run.smithi167.stdout:Selecting previously unselected package nvme-cli. 2023-10-05T19:26:29.825 INFO:teuthology.orchestra.run.smithi167.stdout:Preparing to unpack .../090-nvme-cli_1.9-1ubuntu0.1_amd64.deb ... 2023-10-05T19:26:29.907 INFO:teuthology.orchestra.run.smithi167.stdout:Unpacking nvme-cli (1.9-1ubuntu0.1) ... 2023-10-05T19:26:30.222 INFO:teuthology.orchestra.run.smithi167.stdout:Selecting previously unselected package libjs-mathjax. 2023-10-05T19:26:30.239 INFO:teuthology.orchestra.run.smithi167.stdout:Preparing to unpack .../091-libjs-mathjax_2.7.4+dfsg-1_all.deb ... 2023-10-05T19:26:30.279 INFO:teuthology.orchestra.run.smithi167.stdout:Unpacking libjs-mathjax (2.7.4+dfsg-1) ... 2023-10-05T19:26:31.251 INFO:teuthology.orchestra.run.smithi118.stdout:Selecting previously unselected package ceph-volume. 2023-10-05T19:26:31.268 INFO:teuthology.orchestra.run.smithi118.stdout:Preparing to unpack .../087-ceph-volume_17.2.6-1278-g487ffcb5-1focal_all.deb ... 2023-10-05T19:26:31.334 INFO:teuthology.orchestra.run.smithi118.stdout:Unpacking ceph-volume (17.2.6-1278-g487ffcb5-1focal) ... 2023-10-05T19:26:31.647 INFO:teuthology.orchestra.run.smithi118.stdout:Selecting previously unselected package fonts-mathjax. 2023-10-05T19:26:31.664 INFO:teuthology.orchestra.run.smithi118.stdout:Preparing to unpack .../088-fonts-mathjax_2.7.4+dfsg-1_all.deb ... 2023-10-05T19:26:31.705 INFO:teuthology.orchestra.run.smithi118.stdout:Unpacking fonts-mathjax (2.7.4+dfsg-1) ... 2023-10-05T19:26:31.950 INFO:teuthology.orchestra.run.smithi167.stdout:Selecting previously unselected package python-natsort-doc. 2023-10-05T19:26:31.971 INFO:teuthology.orchestra.run.smithi167.stdout:Preparing to unpack .../092-python-natsort-doc_7.0.1-1_all.deb ... 2023-10-05T19:26:32.008 INFO:teuthology.orchestra.run.smithi167.stdout:Unpacking python-natsort-doc (7.0.1-1) ... 2023-10-05T19:26:32.251 INFO:teuthology.orchestra.run.smithi118.stdout:Selecting previously unselected package libcephfs-dev. 2023-10-05T19:26:32.270 INFO:teuthology.orchestra.run.smithi118.stdout:Preparing to unpack .../089-libcephfs-dev_17.2.6-1278-g487ffcb5-1focal_amd64.deb ... 2023-10-05T19:26:32.309 INFO:teuthology.orchestra.run.smithi118.stdout:Unpacking libcephfs-dev (17.2.6-1278-g487ffcb5-1focal) ... 2023-10-05T19:26:32.311 INFO:teuthology.orchestra.run.smithi167.stdout:Selecting previously unselected package python3-atomicwrites. 2023-10-05T19:26:32.329 INFO:teuthology.orchestra.run.smithi167.stdout:Preparing to unpack .../093-python3-atomicwrites_1.1.5-2build1_all.deb ... 2023-10-05T19:26:32.369 INFO:teuthology.orchestra.run.smithi167.stdout:Unpacking python3-atomicwrites (1.1.5-2build1) ... 2023-10-05T19:26:32.705 INFO:teuthology.orchestra.run.smithi118.stdout:Selecting previously unselected package nvme-cli. 2023-10-05T19:26:32.705 INFO:teuthology.orchestra.run.smithi167.stdout:Selecting previously unselected package python3-attr. 2023-10-05T19:26:32.721 INFO:teuthology.orchestra.run.smithi118.stdout:Preparing to unpack .../090-nvme-cli_1.9-1ubuntu0.1_amd64.deb ... 2023-10-05T19:26:32.722 INFO:teuthology.orchestra.run.smithi167.stdout:Preparing to unpack .../094-python3-attr_19.3.0-2_all.deb ... 2023-10-05T19:26:32.763 INFO:teuthology.orchestra.run.smithi167.stdout:Unpacking python3-attr (19.3.0-2) ... 2023-10-05T19:26:32.805 INFO:teuthology.orchestra.run.smithi118.stdout:Unpacking nvme-cli (1.9-1ubuntu0.1) ... 2023-10-05T19:26:33.100 INFO:teuthology.orchestra.run.smithi167.stdout:Selecting previously unselected package python3-webencodings. 2023-10-05T19:26:33.117 INFO:teuthology.orchestra.run.smithi167.stdout:Preparing to unpack .../095-python3-webencodings_0.5.1-1ubuntu1_all.deb ... 2023-10-05T19:26:33.126 INFO:teuthology.orchestra.run.smithi118.stdout:Selecting previously unselected package libjs-mathjax. 2023-10-05T19:26:33.143 INFO:teuthology.orchestra.run.smithi118.stdout:Preparing to unpack .../091-libjs-mathjax_2.7.4+dfsg-1_all.deb ... 2023-10-05T19:26:33.166 INFO:teuthology.orchestra.run.smithi167.stdout:Unpacking python3-webencodings (0.5.1-1ubuntu1) ... 2023-10-05T19:26:33.184 INFO:teuthology.orchestra.run.smithi118.stdout:Unpacking libjs-mathjax (2.7.4+dfsg-1) ... 2023-10-05T19:26:33.494 INFO:teuthology.orchestra.run.smithi167.stdout:Selecting previously unselected package python3-html5lib. 2023-10-05T19:26:33.512 INFO:teuthology.orchestra.run.smithi167.stdout:Preparing to unpack .../096-python3-html5lib_1.0.1-2_all.deb ... 2023-10-05T19:26:33.552 INFO:teuthology.orchestra.run.smithi167.stdout:Unpacking python3-html5lib (1.0.1-2) ... 2023-10-05T19:26:33.897 INFO:teuthology.orchestra.run.smithi167.stdout:Selecting previously unselected package python3-lxml:amd64. 2023-10-05T19:26:33.915 INFO:teuthology.orchestra.run.smithi167.stdout:Preparing to unpack .../097-python3-lxml_4.5.0-1ubuntu0.5_amd64.deb ... 2023-10-05T19:26:33.946 INFO:teuthology.orchestra.run.smithi167.stdout:Unpacking python3-lxml:amd64 (4.5.0-1ubuntu0.5) ... 2023-10-05T19:26:34.425 INFO:teuthology.orchestra.run.smithi167.stdout:Selecting previously unselected package python3-packaging. 2023-10-05T19:26:34.442 INFO:teuthology.orchestra.run.smithi167.stdout:Preparing to unpack .../098-python3-packaging_20.3-1_all.deb ... 2023-10-05T19:26:34.483 INFO:teuthology.orchestra.run.smithi167.stdout:Unpacking python3-packaging (20.3-1) ... 2023-10-05T19:26:34.786 INFO:teuthology.orchestra.run.smithi167.stdout:Selecting previously unselected package python3-pastescript. 2023-10-05T19:26:34.805 INFO:teuthology.orchestra.run.smithi167.stdout:Preparing to unpack .../099-python3-pastescript_2.0.2-4_all.deb ... 2023-10-05T19:26:34.844 INFO:teuthology.orchestra.run.smithi167.stdout:Unpacking python3-pastescript (2.0.2-4) ... 2023-10-05T19:26:34.906 INFO:teuthology.orchestra.run.smithi118.stdout:Selecting previously unselected package python-natsort-doc. 2023-10-05T19:26:34.925 INFO:teuthology.orchestra.run.smithi118.stdout:Preparing to unpack .../092-python-natsort-doc_7.0.1-1_all.deb ... 2023-10-05T19:26:34.989 INFO:teuthology.orchestra.run.smithi118.stdout:Unpacking python-natsort-doc (7.0.1-1) ... 2023-10-05T19:26:35.206 INFO:teuthology.orchestra.run.smithi167.stdout:Selecting previously unselected package python3-pluggy. 2023-10-05T19:26:35.222 INFO:teuthology.orchestra.run.smithi167.stdout:Preparing to unpack .../100-python3-pluggy_0.13.0-2_all.deb ... 2023-10-05T19:26:35.264 INFO:teuthology.orchestra.run.smithi167.stdout:Unpacking python3-pluggy (0.13.0-2) ... 2023-10-05T19:26:35.326 INFO:teuthology.orchestra.run.smithi118.stdout:Selecting previously unselected package python3-atomicwrites. 2023-10-05T19:26:35.344 INFO:teuthology.orchestra.run.smithi118.stdout:Preparing to unpack .../093-python3-atomicwrites_1.1.5-2build1_all.deb ... 2023-10-05T19:26:35.384 INFO:teuthology.orchestra.run.smithi118.stdout:Unpacking python3-atomicwrites (1.1.5-2build1) ... 2023-10-05T19:26:35.584 INFO:teuthology.orchestra.run.smithi167.stdout:Selecting previously unselected package python3-psutil. 2023-10-05T19:26:35.600 INFO:teuthology.orchestra.run.smithi167.stdout:Preparing to unpack .../101-python3-psutil_5.5.1-1ubuntu4_amd64.deb ... 2023-10-05T19:26:35.642 INFO:teuthology.orchestra.run.smithi167.stdout:Unpacking python3-psutil (5.5.1-1ubuntu4) ... 2023-10-05T19:26:35.670 INFO:teuthology.orchestra.run.smithi118.stdout:Selecting previously unselected package python3-attr. 2023-10-05T19:26:35.690 INFO:teuthology.orchestra.run.smithi118.stdout:Preparing to unpack .../094-python3-attr_19.3.0-2_all.deb ... 2023-10-05T19:26:35.753 INFO:teuthology.orchestra.run.smithi118.stdout:Unpacking python3-attr (19.3.0-2) ... 2023-10-05T19:26:35.995 INFO:teuthology.orchestra.run.smithi167.stdout:Selecting previously unselected package python3-py. 2023-10-05T19:26:36.011 INFO:teuthology.orchestra.run.smithi167.stdout:Preparing to unpack .../102-python3-py_1.8.1-1ubuntu0.1_all.deb ... 2023-10-05T19:26:36.053 INFO:teuthology.orchestra.run.smithi167.stdout:Unpacking python3-py (1.8.1-1ubuntu0.1) ... 2023-10-05T19:26:36.098 INFO:teuthology.orchestra.run.smithi118.stdout:Selecting previously unselected package python3-webencodings. 2023-10-05T19:26:36.116 INFO:teuthology.orchestra.run.smithi118.stdout:Preparing to unpack .../095-python3-webencodings_0.5.1-1ubuntu1_all.deb ... 2023-10-05T19:26:36.156 INFO:teuthology.orchestra.run.smithi118.stdout:Unpacking python3-webencodings (0.5.1-1ubuntu1) ... 2023-10-05T19:26:36.381 INFO:teuthology.orchestra.run.smithi167.stdout:Selecting previously unselected package python3-pyinotify. 2023-10-05T19:26:36.398 INFO:teuthology.orchestra.run.smithi167.stdout:Preparing to unpack .../103-python3-pyinotify_0.9.6-1.2ubuntu1_all.deb ... 2023-10-05T19:26:36.439 INFO:teuthology.orchestra.run.smithi167.stdout:Unpacking python3-pyinotify (0.9.6-1.2ubuntu1) ... 2023-10-05T19:26:36.484 INFO:teuthology.orchestra.run.smithi118.stdout:Selecting previously unselected package python3-html5lib. 2023-10-05T19:26:36.502 INFO:teuthology.orchestra.run.smithi118.stdout:Preparing to unpack .../096-python3-html5lib_1.0.1-2_all.deb ... 2023-10-05T19:26:36.542 INFO:teuthology.orchestra.run.smithi118.stdout:Unpacking python3-html5lib (1.0.1-2) ... 2023-10-05T19:26:36.758 INFO:teuthology.orchestra.run.smithi167.stdout:Selecting previously unselected package python3-wcwidth. 2023-10-05T19:26:36.775 INFO:teuthology.orchestra.run.smithi167.stdout:Preparing to unpack .../104-python3-wcwidth_0.1.8+dfsg1-3_all.deb ... 2023-10-05T19:26:36.825 INFO:teuthology.orchestra.run.smithi167.stdout:Unpacking python3-wcwidth (0.1.8+dfsg1-3) ... 2023-10-05T19:26:36.870 INFO:teuthology.orchestra.run.smithi118.stdout:Selecting previously unselected package python3-lxml:amd64. 2023-10-05T19:26:36.890 INFO:teuthology.orchestra.run.smithi118.stdout:Preparing to unpack .../097-python3-lxml_4.5.0-1ubuntu0.5_amd64.deb ... 2023-10-05T19:26:36.928 INFO:teuthology.orchestra.run.smithi118.stdout:Unpacking python3-lxml:amd64 (4.5.0-1ubuntu0.5) ... 2023-10-05T19:26:37.111 INFO:teuthology.orchestra.run.smithi167.stdout:Selecting previously unselected package python3-pytest. 2023-10-05T19:26:37.131 INFO:teuthology.orchestra.run.smithi167.stdout:Preparing to unpack .../105-python3-pytest_4.6.9-1_all.deb ... 2023-10-05T19:26:37.169 INFO:teuthology.orchestra.run.smithi167.stdout:Unpacking python3-pytest (4.6.9-1) ... 2023-10-05T19:26:37.424 INFO:teuthology.orchestra.run.smithi118.stdout:Selecting previously unselected package python3-packaging. 2023-10-05T19:26:37.441 INFO:teuthology.orchestra.run.smithi118.stdout:Preparing to unpack .../098-python3-packaging_20.3-1_all.deb ... 2023-10-05T19:26:37.481 INFO:teuthology.orchestra.run.smithi118.stdout:Unpacking python3-packaging (20.3-1) ... 2023-10-05T19:26:37.532 INFO:teuthology.orchestra.run.smithi167.stdout:Selecting previously unselected package python3-simplejson. 2023-10-05T19:26:37.549 INFO:teuthology.orchestra.run.smithi167.stdout:Preparing to unpack .../106-python3-simplejson_3.16.0-2ubuntu2_amd64.deb ... 2023-10-05T19:26:37.590 INFO:teuthology.orchestra.run.smithi167.stdout:Unpacking python3-simplejson (3.16.0-2ubuntu2) ... 2023-10-05T19:26:37.784 INFO:teuthology.orchestra.run.smithi118.stdout:Selecting previously unselected package python3-pastescript. 2023-10-05T19:26:37.801 INFO:teuthology.orchestra.run.smithi118.stdout:Preparing to unpack .../099-python3-pastescript_2.0.2-4_all.deb ... 2023-10-05T19:26:37.842 INFO:teuthology.orchestra.run.smithi118.stdout:Unpacking python3-pastescript (2.0.2-4) ... 2023-10-05T19:26:37.876 INFO:teuthology.orchestra.run.smithi167.stdout:Selecting previously unselected package qttranslations5-l10n. 2023-10-05T19:26:37.893 INFO:teuthology.orchestra.run.smithi167.stdout:Preparing to unpack .../107-qttranslations5-l10n_5.12.8-0ubuntu1_all.deb ... 2023-10-05T19:26:37.934 INFO:teuthology.orchestra.run.smithi167.stdout:Unpacking qttranslations5-l10n (5.12.8-0ubuntu1) ... 2023-10-05T19:26:38.188 INFO:teuthology.orchestra.run.smithi118.stdout:Selecting previously unselected package python3-pluggy. 2023-10-05T19:26:38.204 INFO:teuthology.orchestra.run.smithi118.stdout:Preparing to unpack .../100-python3-pluggy_0.13.0-2_all.deb ... 2023-10-05T19:26:38.246 INFO:teuthology.orchestra.run.smithi118.stdout:Unpacking python3-pluggy (0.13.0-2) ... 2023-10-05T19:26:38.558 INFO:teuthology.orchestra.run.smithi118.stdout:Selecting previously unselected package python3-psutil. 2023-10-05T19:26:38.574 INFO:teuthology.orchestra.run.smithi118.stdout:Preparing to unpack .../101-python3-psutil_5.5.1-1ubuntu4_amd64.deb ... 2023-10-05T19:26:38.616 INFO:teuthology.orchestra.run.smithi118.stdout:Unpacking python3-psutil (5.5.1-1ubuntu4) ... 2023-10-05T19:26:38.697 INFO:teuthology.orchestra.run.smithi167.stdout:Selecting previously unselected package radosgw. 2023-10-05T19:26:38.714 INFO:teuthology.orchestra.run.smithi167.stdout:Preparing to unpack .../108-radosgw_17.2.6-1278-g487ffcb5-1focal_amd64.deb ... 2023-10-05T19:26:38.754 INFO:teuthology.orchestra.run.smithi167.stdout:Unpacking radosgw (17.2.6-1278-g487ffcb5-1focal) ... 2023-10-05T19:26:38.953 INFO:teuthology.orchestra.run.smithi118.stdout:Selecting previously unselected package python3-py. 2023-10-05T19:26:38.970 INFO:teuthology.orchestra.run.smithi118.stdout:Preparing to unpack .../102-python3-py_1.8.1-1ubuntu0.1_all.deb ... 2023-10-05T19:26:39.011 INFO:teuthology.orchestra.run.smithi118.stdout:Unpacking python3-py (1.8.1-1ubuntu0.1) ... 2023-10-05T19:26:39.331 INFO:teuthology.orchestra.run.smithi118.stdout:Selecting previously unselected package python3-pyinotify. 2023-10-05T19:26:39.348 INFO:teuthology.orchestra.run.smithi118.stdout:Preparing to unpack .../103-python3-pyinotify_0.9.6-1.2ubuntu1_all.deb ... 2023-10-05T19:26:39.414 INFO:teuthology.orchestra.run.smithi118.stdout:Unpacking python3-pyinotify (0.9.6-1.2ubuntu1) ... 2023-10-05T19:26:39.725 INFO:teuthology.orchestra.run.smithi118.stdout:Selecting previously unselected package python3-wcwidth. 2023-10-05T19:26:39.742 INFO:teuthology.orchestra.run.smithi118.stdout:Preparing to unpack .../104-python3-wcwidth_0.1.8+dfsg1-3_all.deb ... 2023-10-05T19:26:39.783 INFO:teuthology.orchestra.run.smithi118.stdout:Unpacking python3-wcwidth (0.1.8+dfsg1-3) ... 2023-10-05T19:26:40.070 INFO:teuthology.orchestra.run.smithi118.stdout:Selecting previously unselected package python3-pytest. 2023-10-05T19:26:40.090 INFO:teuthology.orchestra.run.smithi118.stdout:Preparing to unpack .../105-python3-pytest_4.6.9-1_all.deb ... 2023-10-05T19:26:40.145 INFO:teuthology.orchestra.run.smithi118.stdout:Unpacking python3-pytest (4.6.9-1) ... 2023-10-05T19:26:40.459 INFO:teuthology.orchestra.run.smithi167.stdout:Selecting previously unselected package rbd-fuse. 2023-10-05T19:26:40.476 INFO:teuthology.orchestra.run.smithi167.stdout:Preparing to unpack .../109-rbd-fuse_17.2.6-1278-g487ffcb5-1focal_amd64.deb ... 2023-10-05T19:26:40.516 INFO:teuthology.orchestra.run.smithi118.stdout:Selecting previously unselected package python3-simplejson. 2023-10-05T19:26:40.517 INFO:teuthology.orchestra.run.smithi167.stdout:Unpacking rbd-fuse (17.2.6-1278-g487ffcb5-1focal) ... 2023-10-05T19:26:40.533 INFO:teuthology.orchestra.run.smithi118.stdout:Preparing to unpack .../106-python3-simplejson_3.16.0-2ubuntu2_amd64.deb ... 2023-10-05T19:26:40.574 INFO:teuthology.orchestra.run.smithi118.stdout:Unpacking python3-simplejson (3.16.0-2ubuntu2) ... 2023-10-05T19:26:40.852 INFO:teuthology.orchestra.run.smithi167.stdout:Setting up python3-attr (19.3.0-2) ... 2023-10-05T19:26:40.885 INFO:teuthology.orchestra.run.smithi118.stdout:Selecting previously unselected package qttranslations5-l10n. 2023-10-05T19:26:40.902 INFO:teuthology.orchestra.run.smithi118.stdout:Preparing to unpack .../107-qttranslations5-l10n_5.12.8-0ubuntu1_all.deb ... 2023-10-05T19:26:40.943 INFO:teuthology.orchestra.run.smithi118.stdout:Unpacking qttranslations5-l10n (5.12.8-0ubuntu1) ... 2023-10-05T19:26:41.155 INFO:teuthology.orchestra.run.smithi167.stdout:Setting up librabbitmq4:amd64 (0.10.0-1) ... 2023-10-05T19:26:41.298 INFO:teuthology.orchestra.run.smithi167.stdout:Setting up libdouble-conversion3:amd64 (3.1.5-4ubuntu1) ... 2023-10-05T19:26:41.453 INFO:teuthology.orchestra.run.smithi167.stdout:Setting up nvme-cli (1.9-1ubuntu0.1) ... 2023-10-05T19:26:41.679 INFO:teuthology.orchestra.run.smithi167.stdout:Setting up cephadm (17.2.6-1278-g487ffcb5-1focal) ... 2023-10-05T19:26:41.727 INFO:teuthology.orchestra.run.smithi118.stdout:Selecting previously unselected package radosgw. 2023-10-05T19:26:41.745 INFO:teuthology.orchestra.run.smithi118.stdout:Preparing to unpack .../108-radosgw_17.2.6-1278-g487ffcb5-1focal_amd64.deb ... 2023-10-05T19:26:41.784 INFO:teuthology.orchestra.run.smithi118.stdout:Unpacking radosgw (17.2.6-1278-g487ffcb5-1focal) ... 2023-10-05T19:26:42.308 INFO:teuthology.orchestra.run.smithi167.stdout:Adding system user cephadm....done 2023-10-05T19:26:42.494 INFO:teuthology.orchestra.run.smithi167.stdout:Setting up fonts-mathjax (2.7.4+dfsg-1) ... 2023-10-05T19:26:42.620 INFO:teuthology.orchestra.run.smithi167.stdout:Setting up python3-waitress (1.4.1-1ubuntu0.1) ... 2023-10-05T19:26:42.980 INFO:teuthology.orchestra.run.smithi167.stdout:Setting up python3-repoze.lru (0.7-2) ... 2023-10-05T19:26:43.296 INFO:teuthology.orchestra.run.smithi167.stdout:Setting up libjs-mathjax (2.7.4+dfsg-1) ... 2023-10-05T19:26:43.416 INFO:teuthology.orchestra.run.smithi167.stdout:Setting up liboath0 (2.6.1-1.3) ... 2023-10-05T19:26:43.521 INFO:teuthology.orchestra.run.smithi118.stdout:Selecting previously unselected package rbd-fuse. 2023-10-05T19:26:43.538 INFO:teuthology.orchestra.run.smithi118.stdout:Preparing to unpack .../109-rbd-fuse_17.2.6-1278-g487ffcb5-1focal_amd64.deb ... 2023-10-05T19:26:43.543 INFO:teuthology.orchestra.run.smithi167.stdout:Setting up python3-py (1.8.1-1ubuntu0.1) ... 2023-10-05T19:26:43.579 INFO:teuthology.orchestra.run.smithi118.stdout:Unpacking rbd-fuse (17.2.6-1278-g487ffcb5-1focal) ... 2023-10-05T19:26:43.878 INFO:teuthology.orchestra.run.smithi167.stdout:Setting up python3-joblib (0.14.0-3) ... 2023-10-05T19:26:43.904 INFO:teuthology.orchestra.run.smithi118.stdout:Setting up python3-attr (19.3.0-2) ... 2023-10-05T19:26:44.185 INFO:teuthology.orchestra.run.smithi118.stdout:Setting up librabbitmq4:amd64 (0.10.0-1) ... 2023-10-05T19:26:44.311 INFO:teuthology.orchestra.run.smithi118.stdout:Setting up libdouble-conversion3:amd64 (3.1.5-4ubuntu1) ... 2023-10-05T19:26:44.312 INFO:teuthology.orchestra.run.smithi167.stdout:Setting up python3-jwt (1.7.1-2ubuntu2.1) ... 2023-10-05T19:26:44.437 INFO:teuthology.orchestra.run.smithi118.stdout:Setting up nvme-cli (1.9-1ubuntu0.1) ... 2023-10-05T19:26:44.597 INFO:teuthology.orchestra.run.smithi167.stdout:Setting up python3-cachetools (4.0.0-1) ... 2023-10-05T19:26:44.672 INFO:teuthology.orchestra.run.smithi118.stdout:Setting up cephadm (17.2.6-1278-g487ffcb5-1focal) ... 2023-10-05T19:26:44.894 INFO:teuthology.orchestra.run.smithi167.stdout:Setting up python3-bcrypt (3.1.7-2ubuntu1) ... 2023-10-05T19:26:45.220 INFO:teuthology.orchestra.run.smithi167.stdout:Setting up python3-pyinotify (0.9.6-1.2ubuntu1) ... 2023-10-05T19:26:45.251 INFO:teuthology.orchestra.run.smithi118.stdout:Adding system user cephadm....done 2023-10-05T19:26:45.421 INFO:teuthology.orchestra.run.smithi118.stdout:Setting up fonts-mathjax (2.7.4+dfsg-1) ... 2023-10-05T19:26:45.529 INFO:teuthology.orchestra.run.smithi167.stdout:Setting up python3-ceph-argparse (17.2.6-1278-g487ffcb5-1focal) ... 2023-10-05T19:26:45.543 INFO:teuthology.orchestra.run.smithi118.stdout:Setting up python3-waitress (1.4.1-1ubuntu0.1) ... 2023-10-05T19:26:45.830 INFO:teuthology.orchestra.run.smithi167.stdout:Setting up python3-sklearn-lib (0.22.2.post1+dfsg-5) ... 2023-10-05T19:26:45.895 INFO:teuthology.orchestra.run.smithi118.stdout:Setting up python3-repoze.lru (0.7-2) ... 2023-10-05T19:26:45.957 INFO:teuthology.orchestra.run.smithi167.stdout:Setting up python3-markupsafe (1.1.0-1build2) ... 2023-10-05T19:26:46.188 INFO:teuthology.orchestra.run.smithi118.stdout:Setting up libjs-mathjax (2.7.4+dfsg-1) ... 2023-10-05T19:26:46.249 INFO:teuthology.orchestra.run.smithi167.stdout:Setting up python3-webencodings (0.5.1-1ubuntu1) ... 2023-10-05T19:26:46.314 INFO:teuthology.orchestra.run.smithi118.stdout:Setting up liboath0 (2.6.1-1.3) ... 2023-10-05T19:26:46.432 INFO:teuthology.orchestra.run.smithi118.stdout:Setting up python3-py (1.8.1-1ubuntu0.1) ... 2023-10-05T19:26:46.935 INFO:teuthology.orchestra.run.smithi167.stdout:Setting up libpcre2-16-0:amd64 (10.34-7ubuntu0.1) ... 2023-10-05T19:26:47.036 INFO:teuthology.orchestra.run.smithi118.stdout:Setting up python3-joblib (0.14.0-3) ... 2023-10-05T19:26:47.060 INFO:teuthology.orchestra.run.smithi167.stdout:Setting up python3-psutil (5.5.1-1ubuntu4) ... 2023-10-05T19:26:47.438 INFO:teuthology.orchestra.run.smithi118.stdout:Setting up python3-jwt (1.7.1-2ubuntu2.1) ... 2023-10-05T19:26:47.469 INFO:teuthology.orchestra.run.smithi167.stdout:Setting up python3-natsort (7.0.1-1) ... 2023-10-05T19:26:47.765 INFO:teuthology.orchestra.run.smithi118.stdout:Setting up python3-cachetools (4.0.0-1) ... 2023-10-05T19:26:47.772 INFO:teuthology.orchestra.run.smithi167.stdout:Setting up python3-atomicwrites (1.1.5-2build1) ... 2023-10-05T19:26:48.066 INFO:teuthology.orchestra.run.smithi118.stdout:Setting up python3-bcrypt (3.1.7-2ubuntu1) ... 2023-10-05T19:26:48.073 INFO:teuthology.orchestra.run.smithi167.stdout:Setting up python3-simplejson (3.16.0-2ubuntu2) ... 2023-10-05T19:26:48.350 INFO:teuthology.orchestra.run.smithi118.stdout:Setting up python3-pyinotify (0.9.6-1.2ubuntu1) ... 2023-10-05T19:26:48.425 INFO:teuthology.orchestra.run.smithi167.stdout:Setting up python3-decorator (4.4.2-0ubuntu1) ... 2023-10-05T19:26:48.651 INFO:teuthology.orchestra.run.smithi118.stdout:Setting up python3-ceph-argparse (17.2.6-1278-g487ffcb5-1focal) ... 2023-10-05T19:26:48.709 INFO:teuthology.orchestra.run.smithi167.stdout:Setting up python3-jinja2 (2.10.1-2) ... 2023-10-05T19:26:48.952 INFO:teuthology.orchestra.run.smithi118.stdout:Setting up python3-sklearn-lib (0.22.2.post1+dfsg-5) ... 2023-10-05T19:26:49.052 INFO:teuthology.orchestra.run.smithi167.stdout:Setting up python3-tempita (0.5.2-6) ... 2023-10-05T19:26:49.095 INFO:teuthology.orchestra.run.smithi118.stdout:Setting up python3-markupsafe (1.1.0-1build2) ... 2023-10-05T19:26:49.378 INFO:teuthology.orchestra.run.smithi167.stdout:Setting up python3-packaging (20.3-1) ... 2023-10-05T19:26:49.379 INFO:teuthology.orchestra.run.smithi118.stdout:Setting up python3-webencodings (0.5.1-1ubuntu1) ... 2023-10-05T19:26:49.672 INFO:teuthology.orchestra.run.smithi118.stdout:Setting up libpcre2-16-0:amd64 (10.34-7ubuntu0.1) ... 2023-10-05T19:26:49.679 INFO:teuthology.orchestra.run.smithi167.stdout:Setting up python-pastedeploy-tpl (2.1.0-1) ... 2023-10-05T19:26:49.805 INFO:teuthology.orchestra.run.smithi167.stdout:Setting up qttranslations5-l10n (5.12.8-0ubuntu1) ... 2023-10-05T19:26:49.823 INFO:teuthology.orchestra.run.smithi118.stdout:Setting up python3-psutil (5.5.1-1ubuntu4) ... 2023-10-05T19:26:49.951 INFO:teuthology.orchestra.run.smithi167.stdout:Setting up python3-wcwidth (0.1.8+dfsg1-3) ... 2023-10-05T19:26:50.200 INFO:teuthology.orchestra.run.smithi118.stdout:Setting up python3-natsort (7.0.1-1) ... 2023-10-05T19:26:50.249 INFO:teuthology.orchestra.run.smithi167.stdout:Setting up python3-paste (3.4.0+dfsg1-1) ... 2023-10-05T19:26:50.509 INFO:teuthology.orchestra.run.smithi118.stdout:Setting up python3-atomicwrites (1.1.5-2build1) ... 2023-10-05T19:26:50.642 INFO:teuthology.orchestra.run.smithi167.stdout:Setting up python3-werkzeug (0.16.1+dfsg1-2ubuntu0.1) ... 2023-10-05T19:26:50.793 INFO:teuthology.orchestra.run.smithi118.stdout:Setting up python3-simplejson (3.16.0-2ubuntu2) ... 2023-10-05T19:26:51.019 INFO:teuthology.orchestra.run.smithi167.stdout:Setting up python3-cryptography (2.8-3ubuntu0.1) ... 2023-10-05T19:26:51.145 INFO:teuthology.orchestra.run.smithi118.stdout:Setting up python3-decorator (4.4.2-0ubuntu1) ... 2023-10-05T19:26:51.379 INFO:teuthology.orchestra.run.smithi167.stdout:Setting up socat (1.7.3.3-2) ... 2023-10-05T19:26:51.446 INFO:teuthology.orchestra.run.smithi118.stdout:Setting up python3-jinja2 (2.10.1-2) ... 2023-10-05T19:26:51.505 INFO:teuthology.orchestra.run.smithi167.stdout:Setting up python3-ceph-common (17.2.6-1278-g487ffcb5-1focal) ... 2023-10-05T19:26:51.789 INFO:teuthology.orchestra.run.smithi118.stdout:Setting up python3-tempita (0.5.2-6) ... 2023-10-05T19:26:51.823 INFO:teuthology.orchestra.run.smithi167.stdout:Setting up libqt5core5a:amd64 (5.12.8+dfsg-0ubuntu2.1) ... 2023-10-05T19:26:51.957 INFO:teuthology.orchestra.run.smithi167.stdout:Setting up python3-html5lib (1.0.1-2) ... 2023-10-05T19:26:52.082 INFO:teuthology.orchestra.run.smithi118.stdout:Setting up python3-packaging (20.3-1) ... 2023-10-05T19:26:52.308 INFO:teuthology.orchestra.run.smithi167.stdout:Setting up librdkafka1:amd64 (1.2.1-1ubuntu1) ... 2023-10-05T19:26:52.383 INFO:teuthology.orchestra.run.smithi118.stdout:Setting up python-pastedeploy-tpl (2.1.0-1) ... 2023-10-05T19:26:52.468 INFO:teuthology.orchestra.run.smithi167.stdout:Setting up xmlstarlet (1.6.1-2build1) ... 2023-10-05T19:26:52.509 INFO:teuthology.orchestra.run.smithi118.stdout:Setting up qttranslations5-l10n (5.12.8-0ubuntu1) ... 2023-10-05T19:26:52.594 INFO:teuthology.orchestra.run.smithi167.stdout:Setting up python3-pluggy (0.13.0-2) ... 2023-10-05T19:26:52.635 INFO:teuthology.orchestra.run.smithi118.stdout:Setting up python3-wcwidth (0.1.8+dfsg1-3) ... 2023-10-05T19:26:52.887 INFO:teuthology.orchestra.run.smithi167.stdout:Setting up python3-lxml:amd64 (4.5.0-1ubuntu0.5) ... 2023-10-05T19:26:52.952 INFO:teuthology.orchestra.run.smithi118.stdout:Setting up python3-paste (3.4.0+dfsg1-1) ... 2023-10-05T19:26:53.196 INFO:teuthology.orchestra.run.smithi167.stdout:Setting up libqt5dbus5:amd64 (5.12.8+dfsg-0ubuntu2.1) ... 2023-10-05T19:26:53.346 INFO:teuthology.orchestra.run.smithi118.stdout:Setting up python3-werkzeug (0.16.1+dfsg1-2ubuntu0.1) ... 2023-10-05T19:26:53.347 INFO:teuthology.orchestra.run.smithi167.stdout:Setting up liblua5.3-0:amd64 (5.3.3-1.1ubuntu2) ... 2023-10-05T19:26:53.465 INFO:teuthology.orchestra.run.smithi167.stdout:Setting up python3-pyasn1 (0.4.2-3build1) ... 2023-10-05T19:26:53.731 INFO:teuthology.orchestra.run.smithi118.stdout:Setting up python3-cryptography (2.8-3ubuntu0.1) ... 2023-10-05T19:26:53.801 INFO:teuthology.orchestra.run.smithi167.stdout:Setting up python3-singledispatch (3.4.0.3-2build1) ... 2023-10-05T19:26:54.078 INFO:teuthology.orchestra.run.smithi167.stdout:Setting up python3-logutils (0.3.3-7) ... 2023-10-05T19:26:54.092 INFO:teuthology.orchestra.run.smithi118.stdout:Setting up socat (1.7.3.3-2) ... 2023-10-05T19:26:54.243 INFO:teuthology.orchestra.run.smithi118.stdout:Setting up python3-ceph-common (17.2.6-1278-g487ffcb5-1focal) ... 2023-10-05T19:26:54.415 INFO:teuthology.orchestra.run.smithi167.stdout:Setting up liblttng-ust0:amd64 (2.11.0-1) ... 2023-10-05T19:26:54.540 INFO:teuthology.orchestra.run.smithi167.stdout:Setting up python3-simplegeneric (0.8.1-3) ... 2023-10-05T19:26:54.563 INFO:teuthology.orchestra.run.smithi118.stdout:Setting up libqt5core5a:amd64 (5.12.8+dfsg-0ubuntu2.1) ... 2023-10-05T19:26:54.698 INFO:teuthology.orchestra.run.smithi118.stdout:Setting up python3-html5lib (1.0.1-2) ... 2023-10-05T19:26:54.832 INFO:teuthology.orchestra.run.smithi167.stdout:Setting up python3-prettytable (0.7.2-5) ... 2023-10-05T19:26:55.041 INFO:teuthology.orchestra.run.smithi118.stdout:Setting up librdkafka1:amd64 (1.2.1-1ubuntu1) ... 2023-10-05T19:26:55.167 INFO:teuthology.orchestra.run.smithi118.stdout:Setting up xmlstarlet (1.6.1-2build1) ... 2023-10-05T19:26:55.175 INFO:teuthology.orchestra.run.smithi167.stdout:Setting up python3-soupsieve (1.9.5+dfsg-1) ... 2023-10-05T19:26:55.293 INFO:teuthology.orchestra.run.smithi118.stdout:Setting up python3-pluggy (0.13.0-2) ... 2023-10-05T19:26:55.484 INFO:teuthology.orchestra.run.smithi167.stdout:Setting up python3-websocket (0.53.0-2ubuntu1) ... 2023-10-05T19:26:55.586 INFO:teuthology.orchestra.run.smithi118.stdout:Setting up python3-lxml:amd64 (4.5.0-1ubuntu0.5) ... 2023-10-05T19:26:55.920 INFO:teuthology.orchestra.run.smithi167.stdout:update-alternatives: using /usr/bin/python3-wsdump to provide /usr/bin/wsdump (wsdump) in auto mode 2023-10-05T19:26:55.921 INFO:teuthology.orchestra.run.smithi118.stdout:Setting up libqt5dbus5:amd64 (5.12.8+dfsg-0ubuntu2.1) ... 2023-10-05T19:26:56.050 INFO:teuthology.orchestra.run.smithi118.stdout:Setting up liblua5.3-0:amd64 (5.3.3-1.1ubuntu2) ... 2023-10-05T19:26:56.176 INFO:teuthology.orchestra.run.smithi118.stdout:Setting up python3-pyasn1 (0.4.2-3build1) ... 2023-10-05T19:26:56.190 INFO:teuthology.orchestra.run.smithi167.stdout:Setting up libonig5:amd64 (6.9.4-1) ... 2023-10-05T19:26:56.316 INFO:teuthology.orchestra.run.smithi167.stdout:Setting up python3-mako (1.1.0+ds1-1ubuntu2.1) ... 2023-10-05T19:26:56.511 INFO:teuthology.orchestra.run.smithi118.stdout:Setting up python3-singledispatch (3.4.0.3-2build1) ... 2023-10-05T19:26:56.650 INFO:teuthology.orchestra.run.smithi167.stdout:Setting up python3-cherrypy3 (8.9.1-6) ... 2023-10-05T19:26:56.787 INFO:teuthology.orchestra.run.smithi118.stdout:Setting up python3-logutils (0.3.3-7) ... 2023-10-05T19:26:57.019 INFO:teuthology.orchestra.run.smithi167.stdout:Setting up libjs-underscore (1.9.1~dfsg-1ubuntu0.20.04.1) ... 2023-10-05T19:26:57.063 INFO:teuthology.orchestra.run.smithi118.stdout:Setting up liblttng-ust0:amd64 (2.11.0-1) ... 2023-10-05T19:26:57.145 INFO:teuthology.orchestra.run.smithi167.stdout:Setting up liblbfgsb0:amd64 (3.0+dfsg.3-7build1) ... 2023-10-05T19:26:57.214 INFO:teuthology.orchestra.run.smithi118.stdout:Setting up python3-simplegeneric (0.8.1-3) ... 2023-10-05T19:26:57.271 INFO:teuthology.orchestra.run.smithi167.stdout:Setting up python3-webob (1:1.8.5-2) ... 2023-10-05T19:26:57.490 INFO:teuthology.orchestra.run.smithi118.stdout:Setting up python3-prettytable (0.7.2-5) ... 2023-10-05T19:26:57.605 INFO:teuthology.orchestra.run.smithi167.stdout:Setting up python3-blinker (1.4+dfsg1-0.3ubuntu1) ... 2023-10-05T19:26:57.767 INFO:teuthology.orchestra.run.smithi118.stdout:Setting up python3-soupsieve (1.9.5+dfsg-1) ... 2023-10-05T19:26:57.892 INFO:teuthology.orchestra.run.smithi167.stdout:Setting up librados2 (17.2.6-1278-g487ffcb5-1focal) ... 2023-10-05T19:26:58.026 INFO:teuthology.orchestra.run.smithi167.stdout:Setting up python3-scipy (1.3.3-3build1) ... 2023-10-05T19:26:58.087 INFO:teuthology.orchestra.run.smithi118.stdout:Setting up python3-websocket (0.53.0-2ubuntu1) ... 2023-10-05T19:26:58.445 INFO:teuthology.orchestra.run.smithi118.stdout:update-alternatives: using /usr/bin/python3-wsdump to provide /usr/bin/wsdump (wsdump) in auto mode 2023-10-05T19:26:58.689 INFO:teuthology.orchestra.run.smithi118.stdout:Setting up libonig5:amd64 (6.9.4-1) ... 2023-10-05T19:26:58.816 INFO:teuthology.orchestra.run.smithi118.stdout:Setting up python3-mako (1.1.0+ds1-1ubuntu2.1) ... 2023-10-05T19:26:59.133 INFO:teuthology.orchestra.run.smithi118.stdout:Setting up python3-cherrypy3 (8.9.1-6) ... 2023-10-05T19:26:59.498 INFO:teuthology.orchestra.run.smithi167.stdout:Setting up libsqlite3-mod-ceph (17.2.6-1278-g487ffcb5-1focal) ... 2023-10-05T19:26:59.501 INFO:teuthology.orchestra.run.smithi118.stdout:Setting up libjs-underscore (1.9.1~dfsg-1ubuntu0.20.04.1) ... 2023-10-05T19:26:59.624 INFO:teuthology.orchestra.run.smithi167.stdout:Setting up libjq1:amd64 (1.6-1ubuntu0.20.04.1) ... 2023-10-05T19:26:59.627 INFO:teuthology.orchestra.run.smithi118.stdout:Setting up liblbfgsb0:amd64 (3.0+dfsg.3-7build1) ... 2023-10-05T19:26:59.750 INFO:teuthology.orchestra.run.smithi167.stdout:Setting up python3-pytest (4.6.9-1) ... 2023-10-05T19:26:59.753 INFO:teuthology.orchestra.run.smithi118.stdout:Setting up python3-webob (1:1.8.5-2) ... 2023-10-05T19:27:00.071 INFO:teuthology.orchestra.run.smithi118.stdout:Setting up python3-blinker (1.4+dfsg1-0.3ubuntu1) ... 2023-10-05T19:27:00.135 INFO:teuthology.orchestra.run.smithi167.stdout:Setting up python3-openssl (19.0.0-1build1) ... 2023-10-05T19:27:00.364 INFO:teuthology.orchestra.run.smithi118.stdout:Setting up librados2 (17.2.6-1278-g487ffcb5-1focal) ... 2023-10-05T19:27:00.436 INFO:teuthology.orchestra.run.smithi167.stdout:Setting up python3-pastedeploy (2.1.0-1) ... 2023-10-05T19:27:00.490 INFO:teuthology.orchestra.run.smithi118.stdout:Setting up python3-scipy (1.3.3-3build1) ... 2023-10-05T19:27:00.712 INFO:teuthology.orchestra.run.smithi167.stdout:Setting up libqt5network5:amd64 (5.12.8+dfsg-0ubuntu2.1) ... 2023-10-05T19:27:00.863 INFO:teuthology.orchestra.run.smithi167.stdout:Setting up python3-bs4 (4.8.2-1) ... 2023-10-05T19:27:01.215 INFO:teuthology.orchestra.run.smithi167.stdout:Setting up libcephfs2 (17.2.6-1278-g487ffcb5-1focal) ... 2023-10-05T19:27:01.332 INFO:teuthology.orchestra.run.smithi167.stdout:Setting up python3-oauthlib (3.1.0-1ubuntu2) ... 2023-10-05T19:27:01.684 INFO:teuthology.orchestra.run.smithi167.stdout:Setting up libradosstriper1 (17.2.6-1278-g487ffcb5-1focal) ... 2023-10-05T19:27:01.810 INFO:teuthology.orchestra.run.smithi167.stdout:Setting up python3-pyasn1-modules (0.2.1-0.2build1) ... 2023-10-05T19:27:01.938 INFO:teuthology.orchestra.run.smithi118.stdout:Setting up libsqlite3-mod-ceph (17.2.6-1278-g487ffcb5-1focal) ... 2023-10-05T19:27:02.089 INFO:teuthology.orchestra.run.smithi118.stdout:Setting up libjq1:amd64 (1.6-1ubuntu0.20.04.1) ... 2023-10-05T19:27:02.153 INFO:teuthology.orchestra.run.smithi167.stdout:Setting up python3-sklearn (0.22.2.post1+dfsg-5) ... 2023-10-05T19:27:02.258 INFO:teuthology.orchestra.run.smithi118.stdout:Setting up python3-pytest (4.6.9-1) ... 2023-10-05T19:27:02.642 INFO:teuthology.orchestra.run.smithi118.stdout:Setting up python3-openssl (19.0.0-1build1) ... 2023-10-05T19:27:02.943 INFO:teuthology.orchestra.run.smithi118.stdout:Setting up python3-pastedeploy (2.1.0-1) ... 2023-10-05T19:27:03.175 INFO:teuthology.orchestra.run.smithi167.stdout:Setting up librbd1 (17.2.6-1278-g487ffcb5-1focal) ... 2023-10-05T19:27:03.269 INFO:teuthology.orchestra.run.smithi118.stdout:Setting up libqt5network5:amd64 (5.12.8+dfsg-0ubuntu2.1) ... 2023-10-05T19:27:03.343 INFO:teuthology.orchestra.run.smithi167.stdout:Setting up libjs-sphinxdoc (1.8.5-7ubuntu3) ... 2023-10-05T19:27:03.395 INFO:teuthology.orchestra.run.smithi118.stdout:Setting up python3-bs4 (4.8.2-1) ... 2023-10-05T19:27:03.469 INFO:teuthology.orchestra.run.smithi167.stdout:Setting up python-natsort-doc (7.0.1-1) ... 2023-10-05T19:27:03.595 INFO:teuthology.orchestra.run.smithi167.stdout:Setting up jq (1.6-1ubuntu0.20.04.1) ... 2023-10-05T19:27:03.721 INFO:teuthology.orchestra.run.smithi167.stdout:Setting up python3-rsa (4.0-3) ... 2023-10-05T19:27:03.746 INFO:teuthology.orchestra.run.smithi118.stdout:Setting up libcephfs2 (17.2.6-1278-g487ffcb5-1focal) ... 2023-10-05T19:27:03.872 INFO:teuthology.orchestra.run.smithi118.stdout:Setting up python3-oauthlib (3.1.0-1ubuntu2) ... 2023-10-05T19:27:04.014 INFO:teuthology.orchestra.run.smithi167.stdout:Setting up libthrift-0.13.0 (0.13.0-2build2) ... 2023-10-05T19:27:04.165 INFO:teuthology.orchestra.run.smithi167.stdout:Setting up python3-webtest (2.0.32-2) ... 2023-10-05T19:27:04.199 INFO:teuthology.orchestra.run.smithi118.stdout:Setting up libradosstriper1 (17.2.6-1278-g487ffcb5-1focal) ... 2023-10-05T19:27:04.325 INFO:teuthology.orchestra.run.smithi118.stdout:Setting up python3-pyasn1-modules (0.2.1-0.2build1) ... 2023-10-05T19:27:04.466 INFO:teuthology.orchestra.run.smithi167.stdout:Setting up ceph-mgr-modules-core (17.2.6-1278-g487ffcb5-1focal) ... 2023-10-05T19:27:04.592 INFO:teuthology.orchestra.run.smithi167.stdout:Setting up ceph-fuse (17.2.6-1278-g487ffcb5-1focal) ... 2023-10-05T19:27:04.652 INFO:teuthology.orchestra.run.smithi118.stdout:Setting up python3-sklearn (0.22.2.post1+dfsg-5) ... 2023-10-05T19:27:04.818 INFO:teuthology.orchestra.run.smithi167.stdout:Created symlink /etc/systemd/system/remote-fs.target.wants/ceph-fuse.target -> /lib/systemd/system/ceph-fuse.target. 2023-10-05T19:27:04.818 INFO:teuthology.orchestra.run.smithi167.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-fuse.target -> /lib/systemd/system/ceph-fuse.target. 2023-10-05T19:27:05.268 INFO:teuthology.orchestra.run.smithi167.stdout:Setting up libcephfs-dev (17.2.6-1278-g487ffcb5-1focal) ... 2023-10-05T19:27:05.480 INFO:teuthology.orchestra.run.smithi167.stdout:Setting up python3-requests-oauthlib (1.0.0-1.1build1) ... 2023-10-05T19:27:05.699 INFO:teuthology.orchestra.run.smithi118.stdout:Setting up librbd1 (17.2.6-1278-g487ffcb5-1focal) ... 2023-10-05T19:27:05.764 INFO:teuthology.orchestra.run.smithi167.stdout:Setting up python3-pastescript (2.0.2-4) ... 2023-10-05T19:27:05.825 INFO:teuthology.orchestra.run.smithi118.stdout:Setting up libjs-sphinxdoc (1.8.5-7ubuntu3) ... 2023-10-05T19:27:05.976 INFO:teuthology.orchestra.run.smithi118.stdout:Setting up python-natsort-doc (7.0.1-1) ... 2023-10-05T19:27:06.099 INFO:teuthology.orchestra.run.smithi167.stdout:Setting up python3-pecan (1.3.3-3ubuntu1) ... 2023-10-05T19:27:06.102 INFO:teuthology.orchestra.run.smithi118.stdout:Setting up jq (1.6-1ubuntu0.20.04.1) ... 2023-10-05T19:27:06.228 INFO:teuthology.orchestra.run.smithi118.stdout:Setting up python3-rsa (4.0-3) ... 2023-10-05T19:27:06.444 INFO:teuthology.orchestra.run.smithi167.stdout:Setting up python3-rados (17.2.6-1278-g487ffcb5-1focal) ... 2023-10-05T19:27:06.537 INFO:teuthology.orchestra.run.smithi118.stdout:Setting up libthrift-0.13.0 (0.13.0-2build2) ... 2023-10-05T19:27:06.569 INFO:teuthology.orchestra.run.smithi167.stdout:Setting up librgw2 (17.2.6-1278-g487ffcb5-1focal) ... 2023-10-05T19:27:06.697 INFO:teuthology.orchestra.run.smithi118.stdout:Setting up python3-webtest (2.0.32-2) ... 2023-10-05T19:27:06.721 INFO:teuthology.orchestra.run.smithi167.stdout:Setting up python3-rbd (17.2.6-1278-g487ffcb5-1focal) ... 2023-10-05T19:27:06.847 INFO:teuthology.orchestra.run.smithi167.stdout:Setting up rbd-fuse (17.2.6-1278-g487ffcb5-1focal) ... 2023-10-05T19:27:06.973 INFO:teuthology.orchestra.run.smithi167.stdout:Setting up python3-routes (2.4.1-2) ... 2023-10-05T19:27:07.006 INFO:teuthology.orchestra.run.smithi118.stdout:Setting up ceph-mgr-modules-core (17.2.6-1278-g487ffcb5-1focal) ... 2023-10-05T19:27:07.132 INFO:teuthology.orchestra.run.smithi118.stdout:Setting up ceph-fuse (17.2.6-1278-g487ffcb5-1focal) ... 2023-10-05T19:27:07.274 INFO:teuthology.orchestra.run.smithi167.stdout:Setting up python3-google-auth (1.5.1-2) ... 2023-10-05T19:27:07.353 INFO:teuthology.orchestra.run.smithi118.stdout:Created symlink /etc/systemd/system/remote-fs.target.wants/ceph-fuse.target -> /lib/systemd/system/ceph-fuse.target. 2023-10-05T19:27:07.353 INFO:teuthology.orchestra.run.smithi118.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-fuse.target -> /lib/systemd/system/ceph-fuse.target. 2023-10-05T19:27:07.567 INFO:teuthology.orchestra.run.smithi167.stdout:Setting up python3-rgw (17.2.6-1278-g487ffcb5-1focal) ... 2023-10-05T19:27:07.693 INFO:teuthology.orchestra.run.smithi167.stdout:Setting up python3-cephfs (17.2.6-1278-g487ffcb5-1focal) ... 2023-10-05T19:27:07.775 INFO:teuthology.orchestra.run.smithi118.stdout:Setting up libcephfs-dev (17.2.6-1278-g487ffcb5-1focal) ... 2023-10-05T19:27:07.819 INFO:teuthology.orchestra.run.smithi167.stdout:Setting up python3-kubernetes (7.0.0~a1-2) ... 2023-10-05T19:27:07.978 INFO:teuthology.orchestra.run.smithi118.stdout:Setting up python3-requests-oauthlib (1.0.0-1.1build1) ... 2023-10-05T19:27:08.272 INFO:teuthology.orchestra.run.smithi118.stdout:Setting up python3-pastescript (2.0.2-4) ... 2023-10-05T19:27:08.590 INFO:teuthology.orchestra.run.smithi118.stdout:Setting up python3-pecan (1.3.3-3ubuntu1) ... 2023-10-05T19:27:08.722 INFO:teuthology.orchestra.run.smithi167.stdout:Setting up ceph-common (17.2.6-1278-g487ffcb5-1focal) ... 2023-10-05T19:27:08.951 INFO:teuthology.orchestra.run.smithi118.stdout:Setting up python3-rados (17.2.6-1278-g487ffcb5-1focal) ... 2023-10-05T19:27:09.069 INFO:teuthology.orchestra.run.smithi118.stdout:Setting up librgw2 (17.2.6-1278-g487ffcb5-1focal) ... 2023-10-05T19:27:09.195 INFO:teuthology.orchestra.run.smithi118.stdout:Setting up python3-rbd (17.2.6-1278-g487ffcb5-1focal) ... 2023-10-05T19:27:09.285 INFO:teuthology.orchestra.run.smithi167.stdout:Adding group ceph....done 2023-10-05T19:27:09.329 INFO:teuthology.orchestra.run.smithi118.stdout:Setting up rbd-fuse (17.2.6-1278-g487ffcb5-1focal) ... 2023-10-05T19:27:09.455 INFO:teuthology.orchestra.run.smithi118.stdout:Setting up python3-routes (2.4.1-2) ... 2023-10-05T19:27:09.612 INFO:teuthology.orchestra.run.smithi167.stdout:Adding system user ceph....done 2023-10-05T19:27:09.765 INFO:teuthology.orchestra.run.smithi118.stdout:Setting up python3-google-auth (1.5.1-2) ... 2023-10-05T19:27:09.805 INFO:teuthology.orchestra.run.smithi167.stdout:Setting system user ceph properties....done 2023-10-05T19:27:09.825 INFO:teuthology.orchestra.run.smithi167.stdout:chown: cannot access '/var/log/ceph/*.log*': No such file or directory 2023-10-05T19:27:09.977 INFO:teuthology.orchestra.run.smithi167.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target -> /lib/systemd/system/ceph.target. 2023-10-05T19:27:10.083 INFO:teuthology.orchestra.run.smithi118.stdout:Setting up python3-rgw (17.2.6-1278-g487ffcb5-1focal) ... 2023-10-05T19:27:10.209 INFO:teuthology.orchestra.run.smithi118.stdout:Setting up python3-cephfs (17.2.6-1278-g487ffcb5-1focal) ... 2023-10-05T19:27:10.287 INFO:teuthology.orchestra.run.smithi167.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/rbdmap.service -> /lib/systemd/system/rbdmap.service. 2023-10-05T19:27:10.360 INFO:teuthology.orchestra.run.smithi118.stdout:Setting up python3-kubernetes (7.0.0~a1-2) ... 2023-10-05T19:27:10.743 INFO:teuthology.orchestra.run.smithi167.stdout:Setting up ceph-test (17.2.6-1278-g487ffcb5-1focal) ... 2023-10-05T19:27:10.902 INFO:teuthology.orchestra.run.smithi167.stdout:Setting up radosgw (17.2.6-1278-g487ffcb5-1focal) ... 2023-10-05T19:27:11.186 INFO:teuthology.orchestra.run.smithi167.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target -> /lib/systemd/system/ceph-radosgw.target. 2023-10-05T19:27:11.187 INFO:teuthology.orchestra.run.smithi167.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target -> /lib/systemd/system/ceph-radosgw.target. 2023-10-05T19:27:11.272 INFO:teuthology.orchestra.run.smithi118.stdout:Setting up ceph-common (17.2.6-1278-g487ffcb5-1focal) ... 2023-10-05T19:27:11.838 INFO:teuthology.orchestra.run.smithi167.stdout:Setting up ceph-base (17.2.6-1278-g487ffcb5-1focal) ... 2023-10-05T19:27:11.842 INFO:teuthology.orchestra.run.smithi118.stdout:Adding group ceph....done 2023-10-05T19:27:12.202 INFO:teuthology.orchestra.run.smithi118.stdout:Adding system user ceph....done 2023-10-05T19:27:12.216 INFO:teuthology.orchestra.run.smithi167.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service -> /lib/systemd/system/ceph-crash.service. 2023-10-05T19:27:12.381 INFO:teuthology.orchestra.run.smithi118.stdout:Setting system user ceph properties....done 2023-10-05T19:27:12.407 INFO:teuthology.orchestra.run.smithi118.stdout:chown: cannot access '/var/log/ceph/*.log*': No such file or directory 2023-10-05T19:27:12.556 INFO:teuthology.orchestra.run.smithi118.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target -> /lib/systemd/system/ceph.target. 2023-10-05T19:27:12.665 INFO:teuthology.orchestra.run.smithi167.stdout:Setting up ceph-mds (17.2.6-1278-g487ffcb5-1focal) ... 2023-10-05T19:27:12.877 INFO:teuthology.orchestra.run.smithi118.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/rbdmap.service -> /lib/systemd/system/rbdmap.service. 2023-10-05T19:27:12.914 INFO:teuthology.orchestra.run.smithi167.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target -> /lib/systemd/system/ceph-mds.target. 2023-10-05T19:27:12.914 INFO:teuthology.orchestra.run.smithi167.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target -> /lib/systemd/system/ceph-mds.target. 2023-10-05T19:27:13.311 INFO:teuthology.orchestra.run.smithi118.stdout:Setting up ceph-test (17.2.6-1278-g487ffcb5-1focal) ... 2023-10-05T19:27:13.437 INFO:teuthology.orchestra.run.smithi118.stdout:Setting up radosgw (17.2.6-1278-g487ffcb5-1focal) ... 2023-10-05T19:27:13.459 INFO:teuthology.orchestra.run.smithi167.stdout:Setting up ceph-mgr (17.2.6-1278-g487ffcb5-1focal) ... 2023-10-05T19:27:13.690 INFO:teuthology.orchestra.run.smithi167.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target -> /lib/systemd/system/ceph-mgr.target. 2023-10-05T19:27:13.690 INFO:teuthology.orchestra.run.smithi167.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target -> /lib/systemd/system/ceph-mgr.target. 2023-10-05T19:27:13.713 INFO:teuthology.orchestra.run.smithi118.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target -> /lib/systemd/system/ceph-radosgw.target. 2023-10-05T19:27:13.713 INFO:teuthology.orchestra.run.smithi118.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target -> /lib/systemd/system/ceph-radosgw.target. 2023-10-05T19:27:14.138 INFO:teuthology.orchestra.run.smithi167.stdout:Setting up ceph-osd (17.2.6-1278-g487ffcb5-1focal) ... 2023-10-05T19:27:14.372 INFO:teuthology.orchestra.run.smithi118.stdout:Setting up ceph-base (17.2.6-1278-g487ffcb5-1focal) ... 2023-10-05T19:27:14.669 INFO:teuthology.orchestra.run.smithi167.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target -> /lib/systemd/system/ceph-osd.target. 2023-10-05T19:27:14.669 INFO:teuthology.orchestra.run.smithi167.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target -> /lib/systemd/system/ceph-osd.target. 2023-10-05T19:27:14.734 INFO:teuthology.orchestra.run.smithi118.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service -> /lib/systemd/system/ceph-crash.service. 2023-10-05T19:27:15.157 INFO:teuthology.orchestra.run.smithi167.stdout:Setting up ceph-mgr-k8sevents (17.2.6-1278-g487ffcb5-1focal) ... 2023-10-05T19:27:15.174 INFO:teuthology.orchestra.run.smithi118.stdout:Setting up ceph-mds (17.2.6-1278-g487ffcb5-1focal) ... 2023-10-05T19:27:15.283 INFO:teuthology.orchestra.run.smithi167.stdout:Setting up ceph-mgr-diskprediction-local (17.2.6-1278-g487ffcb5-1focal) ... 2023-10-05T19:27:15.407 INFO:teuthology.orchestra.run.smithi118.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target -> /lib/systemd/system/ceph-mds.target. 2023-10-05T19:27:15.407 INFO:teuthology.orchestra.run.smithi118.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target -> /lib/systemd/system/ceph-mds.target. 2023-10-05T19:27:15.425 INFO:teuthology.orchestra.run.smithi167.stdout:Setting up ceph-mon (17.2.6-1278-g487ffcb5-1focal) ... 2023-10-05T19:27:15.650 INFO:teuthology.orchestra.run.smithi167.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target -> /lib/systemd/system/ceph-mon.target. 2023-10-05T19:27:15.650 INFO:teuthology.orchestra.run.smithi167.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target -> /lib/systemd/system/ceph-mon.target. 2023-10-05T19:27:15.834 INFO:teuthology.orchestra.run.smithi118.stdout:Setting up ceph-mgr (17.2.6-1278-g487ffcb5-1focal) ... 2023-10-05T19:27:16.062 INFO:teuthology.orchestra.run.smithi118.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target -> /lib/systemd/system/ceph-mgr.target. 2023-10-05T19:27:16.062 INFO:teuthology.orchestra.run.smithi118.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target -> /lib/systemd/system/ceph-mgr.target. 2023-10-05T19:27:16.110 INFO:teuthology.orchestra.run.smithi167.stdout:Setting up ceph-mgr-cephadm (17.2.6-1278-g487ffcb5-1focal) ... 2023-10-05T19:27:16.262 INFO:teuthology.orchestra.run.smithi167.stdout:Setting up ceph (17.2.6-1278-g487ffcb5-1focal) ... 2023-10-05T19:27:16.388 INFO:teuthology.orchestra.run.smithi167.stdout:Setting up ceph-mgr-dashboard (17.2.6-1278-g487ffcb5-1focal) ... 2023-10-05T19:27:16.510 INFO:teuthology.orchestra.run.smithi118.stdout:Setting up ceph-osd (17.2.6-1278-g487ffcb5-1focal) ... 2023-10-05T19:27:16.522 INFO:teuthology.orchestra.run.smithi167.stdout:Setting up ceph-volume (17.2.6-1278-g487ffcb5-1focal) ... 2023-10-05T19:27:16.632 INFO:teuthology.orchestra.run.smithi167.stdout:Processing triggers for libc-bin (2.31-0ubuntu9.9) ... 2023-10-05T19:27:16.817 INFO:teuthology.orchestra.run.smithi167.stdout:Processing triggers for systemd (245.4-4ubuntu3.17) ... 2023-10-05T19:27:16.991 INFO:teuthology.orchestra.run.smithi118.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target -> /lib/systemd/system/ceph-osd.target. 2023-10-05T19:27:16.991 INFO:teuthology.orchestra.run.smithi118.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target -> /lib/systemd/system/ceph-osd.target. 2023-10-05T19:27:17.119 INFO:teuthology.orchestra.run.smithi167.stdout:Processing triggers for man-db (2.9.1-1) ... 2023-10-05T19:27:17.412 INFO:teuthology.orchestra.run.smithi118.stdout:Setting up ceph-mgr-k8sevents (17.2.6-1278-g487ffcb5-1focal) ... 2023-10-05T19:27:18.006 INFO:teuthology.orchestra.run.smithi118.stdout:Setting up ceph-mgr-diskprediction-local (17.2.6-1278-g487ffcb5-1focal) ... 2023-10-05T19:27:18.182 INFO:teuthology.orchestra.run.smithi118.stdout:Setting up ceph-mon (17.2.6-1278-g487ffcb5-1focal) ... 2023-10-05T19:27:18.413 INFO:teuthology.orchestra.run.smithi118.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target -> /lib/systemd/system/ceph-mon.target. 2023-10-05T19:27:18.413 INFO:teuthology.orchestra.run.smithi118.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target -> /lib/systemd/system/ceph-mon.target. 2023-10-05T19:27:18.860 INFO:teuthology.orchestra.run.smithi118.stdout:Setting up ceph-mgr-cephadm (17.2.6-1278-g487ffcb5-1focal) ... 2023-10-05T19:27:19.003 INFO:teuthology.orchestra.run.smithi118.stdout:Setting up ceph (17.2.6-1278-g487ffcb5-1focal) ... 2023-10-05T19:27:19.129 INFO:teuthology.orchestra.run.smithi118.stdout:Setting up ceph-mgr-dashboard (17.2.6-1278-g487ffcb5-1focal) ... 2023-10-05T19:27:19.299 INFO:teuthology.orchestra.run.smithi118.stdout:Setting up ceph-volume (17.2.6-1278-g487ffcb5-1focal) ... 2023-10-05T19:27:19.426 INFO:teuthology.orchestra.run.smithi118.stdout:Processing triggers for libc-bin (2.31-0ubuntu9.9) ... 2023-10-05T19:27:19.570 INFO:teuthology.orchestra.run.smithi118.stdout:Processing triggers for systemd (245.4-4ubuntu3.17) ... 2023-10-05T19:27:19.854 INFO:teuthology.orchestra.run.smithi118.stdout:Processing triggers for man-db (2.9.1-1) ... 2023-10-05T19:27:19.899 INFO:teuthology.orchestra.run.smithi167.stdout:Processing triggers for fontconfig (2.13.1-2ubuntu3) ... 2023-10-05T19:27:20.507 INFO:teuthology.orchestra.run.smithi167.stderr:W: --force-yes is deprecated, use one of the options starting with --allow instead. 2023-10-05T19:27:20.510 DEBUG:teuthology.parallel:result is None 2023-10-05T19:27:22.732 INFO:teuthology.orchestra.run.smithi118.stdout:Processing triggers for fontconfig (2.13.1-2ubuntu3) ... 2023-10-05T19:27:23.555 INFO:teuthology.orchestra.run.smithi118.stderr:W: --force-yes is deprecated, use one of the options starting with --allow instead. 2023-10-05T19:27:23.558 DEBUG:teuthology.parallel:result is None 2023-10-05T19:27:23.558 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=ubuntu%2F20.04%2Fx86_64&sha1=487ffcb5ddb60d29c5688d1a457e812b1b697b90 2023-10-05T19:27:23.778 DEBUG:teuthology.orchestra.run.smithi118:> dpkg-query -W -f '${Version}' ceph 2023-10-05T19:27:23.801 INFO:teuthology.orchestra.run.smithi118.stdout:17.2.6-1278-g487ffcb5-1focal 2023-10-05T19:27:23.802 INFO:teuthology.packaging:The installed version of ceph is 17.2.6-1278-g487ffcb5-1focal 2023-10-05T19:27:23.802 INFO:teuthology.task.install:The correct ceph version 17.2.6-1278-g487ffcb5-1focal is installed. 2023-10-05T19:27:23.802 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=ubuntu%2F20.04%2Fx86_64&sha1=487ffcb5ddb60d29c5688d1a457e812b1b697b90 2023-10-05T19:27:24.016 DEBUG:teuthology.orchestra.run.smithi167:> dpkg-query -W -f '${Version}' ceph 2023-10-05T19:27:24.039 INFO:teuthology.orchestra.run.smithi167.stdout:17.2.6-1278-g487ffcb5-1focal 2023-10-05T19:27:24.039 INFO:teuthology.packaging:The installed version of ceph is 17.2.6-1278-g487ffcb5-1focal 2023-10-05T19:27:24.040 INFO:teuthology.task.install:The correct ceph version 17.2.6-1278-g487ffcb5-1focal is installed. 2023-10-05T19:27:24.040 INFO:teuthology.task.install.util:Shipping valgrind.supp... 2023-10-05T19:27:24.040 DEBUG:teuthology.orchestra.run.smithi118:> set -ex 2023-10-05T19:27:24.040 DEBUG:teuthology.orchestra.run.smithi118:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2023-10-05T19:27:24.057 DEBUG:teuthology.orchestra.run.smithi167:> set -ex 2023-10-05T19:27:24.057 DEBUG:teuthology.orchestra.run.smithi167:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2023-10-05T19:27:24.069 INFO:teuthology.task.install.util:Shipping 'daemon-helper'... 2023-10-05T19:27:24.070 DEBUG:teuthology.orchestra.run.smithi118:> set -ex 2023-10-05T19:27:24.070 DEBUG:teuthology.orchestra.run.smithi118:> sudo dd of=/usr/bin/daemon-helper 2023-10-05T19:27:24.115 DEBUG:teuthology.orchestra.run.smithi118:> sudo chmod a=rx -- /usr/bin/daemon-helper 2023-10-05T19:27:24.124 DEBUG:teuthology.orchestra.run.smithi167:> set -ex 2023-10-05T19:27:24.124 DEBUG:teuthology.orchestra.run.smithi167:> sudo dd of=/usr/bin/daemon-helper 2023-10-05T19:27:24.140 DEBUG:teuthology.orchestra.run.smithi167:> sudo chmod a=rx -- /usr/bin/daemon-helper 2023-10-05T19:27:24.149 INFO:teuthology.task.install.util:Shipping 'adjust-ulimits'... 2023-10-05T19:27:24.149 DEBUG:teuthology.orchestra.run.smithi118:> set -ex 2023-10-05T19:27:24.149 DEBUG:teuthology.orchestra.run.smithi118:> sudo dd of=/usr/bin/adjust-ulimits 2023-10-05T19:27:24.163 DEBUG:teuthology.orchestra.run.smithi118:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2023-10-05T19:27:24.218 DEBUG:teuthology.orchestra.run.smithi167:> set -ex 2023-10-05T19:27:24.219 DEBUG:teuthology.orchestra.run.smithi167:> sudo dd of=/usr/bin/adjust-ulimits 2023-10-05T19:27:24.234 DEBUG:teuthology.orchestra.run.smithi167:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2023-10-05T19:27:24.292 INFO:teuthology.task.install.util:Shipping 'stdin-killer'... 2023-10-05T19:27:24.293 DEBUG:teuthology.orchestra.run.smithi118:> set -ex 2023-10-05T19:27:24.293 DEBUG:teuthology.orchestra.run.smithi118:> sudo dd of=/usr/bin/stdin-killer 2023-10-05T19:27:24.308 DEBUG:teuthology.orchestra.run.smithi118:> sudo chmod a=rx -- /usr/bin/stdin-killer 2023-10-05T19:27:24.323 DEBUG:teuthology.orchestra.run.smithi167:> set -ex 2023-10-05T19:27:24.323 DEBUG:teuthology.orchestra.run.smithi167:> sudo dd of=/usr/bin/stdin-killer 2023-10-05T19:27:24.338 DEBUG:teuthology.orchestra.run.smithi167:> sudo chmod a=rx -- /usr/bin/stdin-killer 2023-10-05T19:27:24.396 INFO:teuthology.run_tasks:Running task ceph... 2023-10-05T19:27:24.464 INFO:tasks.ceph:Making ceph log dir writeable by non-root... 2023-10-05T19:27:24.464 DEBUG:teuthology.orchestra.run.smithi118:> sudo chmod 777 /var/log/ceph 2023-10-05T19:27:24.467 DEBUG:teuthology.orchestra.run.smithi167:> sudo chmod 777 /var/log/ceph 2023-10-05T19:27:24.482 INFO:tasks.ceph:Disabling ceph logrotate... 2023-10-05T19:27:24.482 DEBUG:teuthology.orchestra.run.smithi118:> sudo rm -f -- /etc/logrotate.d/ceph 2023-10-05T19:27:24.526 DEBUG:teuthology.orchestra.run.smithi167:> sudo rm -f -- /etc/logrotate.d/ceph 2023-10-05T19:27:24.541 INFO:tasks.ceph:Creating extra log directories... 2023-10-05T19:27:24.541 DEBUG:teuthology.orchestra.run.smithi118:> sudo install -d -m0777 -- /var/log/ceph/valgrind /var/log/ceph/profiling-logger 2023-10-05T19:27:24.543 DEBUG:teuthology.orchestra.run.smithi167:> sudo install -d -m0777 -- /var/log/ceph/valgrind /var/log/ceph/profiling-logger 2023-10-05T19:27:24.563 INFO:tasks.ceph:Creating ceph cluster ceph... 2023-10-05T19:27:24.563 INFO:tasks.ceph:config {'conf': {'client': {'debug ms': 1, 'rgw gc obj min wait': 15, 'debug rgw': 20, 'setgroup': 'ceph', 'setuser': 'ceph'}, 'osd': {'debug ms': 1, 'debug objclass': 20, 'bdev async discard': True, 'bdev enable discard': True, 'bluestore allocator': 'bitmap', 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': '1/20', 'debug bluestore': '1/20', 'debug osd': 20, 'debug rocksdb': '4/10', 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore'}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20}}, 'fs': 'xfs', 'mkfs_options': None, 'mount_options': None, 'skip_mgr_daemons': False, 'log_ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(PG_AVAILABILITY\\)', '\\(PG_DEGRADED\\)'], 'cpu_profile': set(), 'cluster': 'ceph', 'mon_bind_msgr2': True, 'mon_bind_addrvec': True} 2023-10-05T19:27:24.563 INFO:tasks.ceph:ctx.config {'archive_path': '/home/teuthworker/archive/yuriw-2023-10-05_14:30:58-rgw-wip-yuri2-testing-2023-10-04-1333-quincy-distro-default-smithi/7411744', 'branch': 'wip-yuri2-testing-2023-10-04-1333-quincy', 'description': 'rgw/singleton/{all/radosgw-admin frontend/beast ignore-pg-availability objectstore/bluestore-bitmap overrides rgw_pool_type/ec-profile supported-random-distro$/{ubuntu_latest}}', 'email': 'yweinste@redhat.com', 'first_in_suite': False, 'job_id': '7411744', 'kernel': {'kdb': True, 'sha1': 'distro'}, 'ktype': 'distro', 'last_in_suite': False, 'machine_type': 'smithi', 'name': 'yuriw-2023-10-05_14:30:58-rgw-wip-yuri2-testing-2023-10-04-1333-quincy-distro-default-smithi', 'no_nested_subset': False, 'nuke-on-error': True, 'openstack': [{'volumes': {'count': 3, 'size': 10}}], 'os_type': 'ubuntu', 'os_version': '20.04', 'overrides': {'admin_socket': {'branch': 'wip-yuri2-testing-2023-10-04-1333-quincy'}, 'ceph': {'conf': {'client': {'debug rgw': 20, 'setgroup': 'ceph', 'setuser': 'ceph'}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20}, 'osd': {'bdev async discard': True, 'bdev enable discard': True, 'bluestore allocator': 'bitmap', 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': '1/20', 'debug bluestore': '1/20', 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': '4/10', 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore'}}, 'flavor': 'default', 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(PG_AVAILABILITY\\)', '\\(PG_DEGRADED\\)'], 'sha1': '487ffcb5ddb60d29c5688d1a457e812b1b697b90', 'wait-for-scrub': False}, 'ceph-deploy': {'bluestore': True, 'conf': {'client': {'log file': '/var/log/ceph/ceph-$name.$pid.log'}, 'mon': {}, 'osd': {'bdev async discard': True, 'bdev enable discard': True, 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': '1/20', 'debug bluestore': '1/20', 'debug rocksdb': '4/10', 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore'}}, 'fs': 'xfs'}, 'install': {'ceph': {'flavor': 'default', 'sha1': '487ffcb5ddb60d29c5688d1a457e812b1b697b90'}}, 'rgw': {'ec-data-pool': True, 'erasure_code_profile': {'crush-failure-domain': 'osd', 'k': 3, 'm': 1, 'name': 'testprofile'}, 'frontend': 'beast'}, 's3tests': {'slow_backend': True}, 'thrashosds': {'bdev_inject_crash': 2, 'bdev_inject_crash_probability': 0.5}, 'workunit': {'branch': 'wip-yuri2-testing-2023-10-04-1333-quincy', 'sha1': '487ffcb5ddb60d29c5688d1a457e812b1b697b90'}}, 'owner': 'scheduled_yuriw@teuthology', 'priority': 75, 'repo': 'https://github.com/ceph/ceph-ci.git', 'roles': [['mon.a', 'osd.0'], ['mgr.x', 'client.0', 'osd.1', 'osd.2', 'osd.3']], 'seed': 880, 'sha1': '487ffcb5ddb60d29c5688d1a457e812b1b697b90', 'sleep_before_teardown': 0, 'subset': '111/12000', 'suite': 'rgw', 'suite_branch': 'wip-yuri2-testing-2023-10-04-1333-quincy', 'suite_path': '/home/teuthworker/src/github.com_ceph_ceph-c_487ffcb5ddb60d29c5688d1a457e812b1b697b90/qa', 'suite_relpath': 'qa', 'suite_repo': 'https://github.com/ceph/ceph-ci.git', 'suite_sha1': '487ffcb5ddb60d29c5688d1a457e812b1b697b90', 'targets': {'smithi118.front.sepia.ceph.com': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBFyT/r7xuKfmg50w0OdQebr4A/ptJo1Zk6DyZNZNZM/q5rW+sznh+wvHqeT3e4YtCdZLaXyvMms8o8u8H5nXOYM=', 'smithi167.front.sepia.ceph.com': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBFyT/r7xuKfmg50w0OdQebr4A/ptJo1Zk6DyZNZNZM/q5rW+sznh+wvHqeT3e4YtCdZLaXyvMms8o8u8H5nXOYM='}, 'tasks': [{'internal.check_packages': None}, {'internal.buildpackages_prep': None}, {'internal.save_config': None}, {'internal.check_lock': None}, {'internal.add_remotes': None}, {'console_log': None}, {'internal.connect': None}, {'internal.push_inventory': None}, {'internal.serialize_remote_roles': None}, {'internal.check_conflict': None}, {'internal.check_ceph_data': None}, {'internal.vm_setup': None}, {'kernel': {'kdb': True, 'sha1': 'distro'}}, {'internal.base': None}, {'internal.archive_upload': None}, {'internal.archive': None}, {'internal.coredump': None}, {'internal.sudo': None}, {'internal.syslog': None}, {'internal.timer': None}, {'pcp': None}, {'selinux': None}, {'ansible.cephlab': None}, {'clock': None}, {'install': None}, {'ceph': {'conf': {'client': {'debug ms': 1, 'rgw gc obj min wait': 15, 'debug rgw': 20, 'setgroup': 'ceph', 'setuser': 'ceph'}, 'osd': {'debug ms': 1, 'debug objclass': 20, 'bdev async discard': True, 'bdev enable discard': True, 'bluestore allocator': 'bitmap', 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': '1/20', 'debug bluestore': '1/20', 'debug osd': 20, 'debug rocksdb': '4/10', 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore'}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20}}, 'flavor': 'default', 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(PG_AVAILABILITY\\)', '\\(PG_DEGRADED\\)'], 'sha1': '487ffcb5ddb60d29c5688d1a457e812b1b697b90', 'wait-for-scrub': False, 'cluster': 'ceph'}}, {'rgw': {'client.0': None}}, {'radosgw-admin': None}, {'radosgw-admin-rest': None}], 'teuthology': {'fragments_dropped': [], 'meta': {}, 'postmerge': []}, 'teuthology_branch': 'main', 'teuthology_sha1': '54e62bcbac4e53d9685e08328b790d3b20d71cae', 'timestamp': '2023-10-05_14:30:58', 'tube': 'smithi', 'user': 'yuriw', 'verbose': True, 'worker_log': '/home/teuthworker/archive/worker_logs/dispatcher.smithi.2830'} 2023-10-05T19:27:24.564 DEBUG:teuthology.orchestra.run.smithi118:> install -d -m0755 -- /home/ubuntu/cephtest/ceph.data 2023-10-05T19:27:24.598 DEBUG:teuthology.orchestra.run.smithi167:> install -d -m0755 -- /home/ubuntu/cephtest/ceph.data 2023-10-05T19:27:24.605 DEBUG:teuthology.orchestra.run.smithi118:> sudo install -d -m0777 -- /var/run/ceph 2023-10-05T19:27:24.607 DEBUG:teuthology.orchestra.run.smithi167:> sudo install -d -m0777 -- /var/run/ceph 2023-10-05T19:27:24.661 DEBUG:teuthology.orchestra.run.smithi118:> set -ex 2023-10-05T19:27:24.661 DEBUG:teuthology.orchestra.run.smithi118:> dd if=/scratch_devs of=/dev/stdout 2023-10-05T19:27:24.669 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2023-10-05T19:27:24.669 DEBUG:teuthology.orchestra.run.smithi118:> stat /dev/vg_nvme/lv_1 2023-10-05T19:27:24.677 INFO:teuthology.orchestra.run.smithi118.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2023-10-05T19:27:24.677 INFO:teuthology.orchestra.run.smithi118.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-10-05T19:27:24.677 INFO:teuthology.orchestra.run.smithi118.stdout:Device: 6h/6d Inode: 361 Links: 1 2023-10-05T19:27:24.678 INFO:teuthology.orchestra.run.smithi118.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-10-05T19:27:24.678 INFO:teuthology.orchestra.run.smithi118.stdout:Access: 2023-10-05 19:23:46.068745983 +0000 2023-10-05T19:27:24.678 INFO:teuthology.orchestra.run.smithi118.stdout:Modify: 2023-10-05 19:23:45.772752808 +0000 2023-10-05T19:27:24.678 INFO:teuthology.orchestra.run.smithi118.stdout:Change: 2023-10-05 19:23:45.772752808 +0000 2023-10-05T19:27:24.678 INFO:teuthology.orchestra.run.smithi118.stdout: Birth: - 2023-10-05T19:27:24.679 DEBUG:teuthology.orchestra.run.smithi118:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2023-10-05T19:27:24.734 INFO:teuthology.orchestra.run.smithi118.stderr:1+0 records in 2023-10-05T19:27:24.735 INFO:teuthology.orchestra.run.smithi118.stderr:1+0 records out 2023-10-05T19:27:24.735 INFO:teuthology.orchestra.run.smithi118.stderr:512 bytes copied, 0.000317427 s, 1.6 MB/s 2023-10-05T19:27:24.736 DEBUG:teuthology.orchestra.run.smithi118:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2023-10-05T19:27:24.744 DEBUG:teuthology.orchestra.run.smithi118:> stat /dev/vg_nvme/lv_2 2023-10-05T19:27:24.749 INFO:teuthology.orchestra.run.smithi118.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2023-10-05T19:27:24.750 INFO:teuthology.orchestra.run.smithi118.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-10-05T19:27:24.750 INFO:teuthology.orchestra.run.smithi118.stdout:Device: 6h/6d Inode: 366 Links: 1 2023-10-05T19:27:24.750 INFO:teuthology.orchestra.run.smithi118.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-10-05T19:27:24.750 INFO:teuthology.orchestra.run.smithi118.stdout:Access: 2023-10-05 19:23:46.532735282 +0000 2023-10-05T19:27:24.750 INFO:teuthology.orchestra.run.smithi118.stdout:Modify: 2023-10-05 19:23:46.260741555 +0000 2023-10-05T19:27:24.750 INFO:teuthology.orchestra.run.smithi118.stdout:Change: 2023-10-05 19:23:46.260741555 +0000 2023-10-05T19:27:24.750 INFO:teuthology.orchestra.run.smithi118.stdout: Birth: - 2023-10-05T19:27:24.750 DEBUG:teuthology.orchestra.run.smithi118:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2023-10-05T19:27:24.806 INFO:teuthology.orchestra.run.smithi118.stderr:1+0 records in 2023-10-05T19:27:24.806 INFO:teuthology.orchestra.run.smithi118.stderr:1+0 records out 2023-10-05T19:27:24.806 INFO:teuthology.orchestra.run.smithi118.stderr:512 bytes copied, 0.000290177 s, 1.8 MB/s 2023-10-05T19:27:24.807 DEBUG:teuthology.orchestra.run.smithi118:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2023-10-05T19:27:24.814 DEBUG:teuthology.orchestra.run.smithi118:> stat /dev/vg_nvme/lv_3 2023-10-05T19:27:24.819 INFO:teuthology.orchestra.run.smithi118.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2023-10-05T19:27:24.819 INFO:teuthology.orchestra.run.smithi118.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-10-05T19:27:24.819 INFO:teuthology.orchestra.run.smithi118.stdout:Device: 6h/6d Inode: 371 Links: 1 2023-10-05T19:27:24.819 INFO:teuthology.orchestra.run.smithi118.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-10-05T19:27:24.819 INFO:teuthology.orchestra.run.smithi118.stdout:Access: 2023-10-05 19:23:46.996724581 +0000 2023-10-05T19:27:24.819 INFO:teuthology.orchestra.run.smithi118.stdout:Modify: 2023-10-05 19:23:46.736730578 +0000 2023-10-05T19:27:24.819 INFO:teuthology.orchestra.run.smithi118.stdout:Change: 2023-10-05 19:23:46.736730578 +0000 2023-10-05T19:27:24.819 INFO:teuthology.orchestra.run.smithi118.stdout: Birth: - 2023-10-05T19:27:24.819 DEBUG:teuthology.orchestra.run.smithi118:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2023-10-05T19:27:24.830 INFO:teuthology.orchestra.run.smithi118.stderr:1+0 records in 2023-10-05T19:27:24.830 INFO:teuthology.orchestra.run.smithi118.stderr:1+0 records out 2023-10-05T19:27:24.830 INFO:teuthology.orchestra.run.smithi118.stderr:512 bytes copied, 0.00028848 s, 1.8 MB/s 2023-10-05T19:27:24.831 DEBUG:teuthology.orchestra.run.smithi118:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2023-10-05T19:27:24.880 DEBUG:teuthology.orchestra.run.smithi118:> stat /dev/vg_nvme/lv_4 2023-10-05T19:27:24.887 INFO:teuthology.orchestra.run.smithi118.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2023-10-05T19:27:24.887 INFO:teuthology.orchestra.run.smithi118.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-10-05T19:27:24.887 INFO:teuthology.orchestra.run.smithi118.stdout:Device: 6h/6d Inode: 378 Links: 1 2023-10-05T19:27:24.887 INFO:teuthology.orchestra.run.smithi118.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-10-05T19:27:24.887 INFO:teuthology.orchestra.run.smithi118.stdout:Access: 2023-10-05 19:23:47.220719416 +0000 2023-10-05T19:27:24.888 INFO:teuthology.orchestra.run.smithi118.stdout:Modify: 2023-10-05 19:23:47.216719508 +0000 2023-10-05T19:27:24.888 INFO:teuthology.orchestra.run.smithi118.stdout:Change: 2023-10-05 19:23:47.216719508 +0000 2023-10-05T19:27:24.888 INFO:teuthology.orchestra.run.smithi118.stdout: Birth: - 2023-10-05T19:27:24.888 DEBUG:teuthology.orchestra.run.smithi118:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2023-10-05T19:27:24.899 INFO:teuthology.orchestra.run.smithi118.stderr:1+0 records in 2023-10-05T19:27:24.899 INFO:teuthology.orchestra.run.smithi118.stderr:1+0 records out 2023-10-05T19:27:24.899 INFO:teuthology.orchestra.run.smithi118.stderr:512 bytes copied, 0.000292443 s, 1.8 MB/s 2023-10-05T19:27:24.901 DEBUG:teuthology.orchestra.run.smithi118:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2023-10-05T19:27:24.908 INFO:tasks.ceph:osd dev map: {'osd.0': '/dev/vg_nvme/lv_1'} 2023-10-05T19:27:24.908 DEBUG:teuthology.orchestra.run.smithi167:> set -ex 2023-10-05T19:27:24.908 DEBUG:teuthology.orchestra.run.smithi167:> dd if=/scratch_devs of=/dev/stdout 2023-10-05T19:27:24.915 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2023-10-05T19:27:24.915 DEBUG:teuthology.orchestra.run.smithi167:> stat /dev/vg_nvme/lv_1 2023-10-05T19:27:24.923 INFO:teuthology.orchestra.run.smithi167.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2023-10-05T19:27:24.923 INFO:teuthology.orchestra.run.smithi167.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-10-05T19:27:24.923 INFO:teuthology.orchestra.run.smithi167.stdout:Device: 6h/6d Inode: 361 Links: 1 2023-10-05T19:27:24.924 INFO:teuthology.orchestra.run.smithi167.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-10-05T19:27:24.924 INFO:teuthology.orchestra.run.smithi167.stdout:Access: 2023-10-05 19:23:45.227933604 +0000 2023-10-05T19:27:24.924 INFO:teuthology.orchestra.run.smithi167.stdout:Modify: 2023-10-05 19:23:44.955939876 +0000 2023-10-05T19:27:24.924 INFO:teuthology.orchestra.run.smithi167.stdout:Change: 2023-10-05 19:23:44.955939876 +0000 2023-10-05T19:27:24.924 INFO:teuthology.orchestra.run.smithi167.stdout: Birth: - 2023-10-05T19:27:24.924 DEBUG:teuthology.orchestra.run.smithi167:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2023-10-05T19:27:24.936 INFO:teuthology.orchestra.run.smithi167.stderr:1+0 records in 2023-10-05T19:27:24.936 INFO:teuthology.orchestra.run.smithi167.stderr:1+0 records out 2023-10-05T19:27:24.937 INFO:teuthology.orchestra.run.smithi167.stderr:512 bytes copied, 0.000395345 s, 1.3 MB/s 2023-10-05T19:27:24.937 DEBUG:teuthology.orchestra.run.smithi167:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2023-10-05T19:27:24.990 DEBUG:teuthology.orchestra.run.smithi167:> stat /dev/vg_nvme/lv_2 2023-10-05T19:27:24.997 INFO:teuthology.orchestra.run.smithi167.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2023-10-05T19:27:24.997 INFO:teuthology.orchestra.run.smithi167.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-10-05T19:27:24.997 INFO:teuthology.orchestra.run.smithi167.stdout:Device: 6h/6d Inode: 364 Links: 1 2023-10-05T19:27:24.997 INFO:teuthology.orchestra.run.smithi167.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-10-05T19:27:24.997 INFO:teuthology.orchestra.run.smithi167.stdout:Access: 2023-10-05 19:23:45.731921981 +0000 2023-10-05T19:27:24.997 INFO:teuthology.orchestra.run.smithi167.stdout:Modify: 2023-10-05 19:23:45.451928439 +0000 2023-10-05T19:27:24.998 INFO:teuthology.orchestra.run.smithi167.stdout:Change: 2023-10-05 19:23:45.451928439 +0000 2023-10-05T19:27:24.998 INFO:teuthology.orchestra.run.smithi167.stdout: Birth: - 2023-10-05T19:27:24.998 DEBUG:teuthology.orchestra.run.smithi167:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2023-10-05T19:27:25.009 INFO:teuthology.orchestra.run.smithi167.stderr:1+0 records in 2023-10-05T19:27:25.009 INFO:teuthology.orchestra.run.smithi167.stderr:1+0 records out 2023-10-05T19:27:25.009 INFO:teuthology.orchestra.run.smithi167.stderr:512 bytes copied, 0.000367166 s, 1.4 MB/s 2023-10-05T19:27:25.010 DEBUG:teuthology.orchestra.run.smithi167:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2023-10-05T19:27:25.062 DEBUG:teuthology.orchestra.run.smithi167:> stat /dev/vg_nvme/lv_3 2023-10-05T19:27:25.069 INFO:teuthology.orchestra.run.smithi167.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2023-10-05T19:27:25.070 INFO:teuthology.orchestra.run.smithi167.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-10-05T19:27:25.070 INFO:teuthology.orchestra.run.smithi167.stdout:Device: 6h/6d Inode: 373 Links: 1 2023-10-05T19:27:25.070 INFO:teuthology.orchestra.run.smithi167.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-10-05T19:27:25.070 INFO:teuthology.orchestra.run.smithi167.stdout:Access: 2023-10-05 19:23:46.231910452 +0000 2023-10-05T19:27:25.070 INFO:teuthology.orchestra.run.smithi167.stdout:Modify: 2023-10-05 19:23:45.971916447 +0000 2023-10-05T19:27:25.070 INFO:teuthology.orchestra.run.smithi167.stdout:Change: 2023-10-05 19:23:45.971916447 +0000 2023-10-05T19:27:25.070 INFO:teuthology.orchestra.run.smithi167.stdout: Birth: - 2023-10-05T19:27:25.070 DEBUG:teuthology.orchestra.run.smithi167:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2023-10-05T19:27:25.084 INFO:teuthology.orchestra.run.smithi167.stderr:1+0 records in 2023-10-05T19:27:25.084 INFO:teuthology.orchestra.run.smithi167.stderr:1+0 records out 2023-10-05T19:27:25.084 INFO:teuthology.orchestra.run.smithi167.stderr:512 bytes copied, 0.000338842 s, 1.5 MB/s 2023-10-05T19:27:25.085 DEBUG:teuthology.orchestra.run.smithi167:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2023-10-05T19:27:25.093 DEBUG:teuthology.orchestra.run.smithi167:> stat /dev/vg_nvme/lv_4 2023-10-05T19:27:25.140 INFO:teuthology.orchestra.run.smithi167.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2023-10-05T19:27:25.140 INFO:teuthology.orchestra.run.smithi167.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-10-05T19:27:25.140 INFO:teuthology.orchestra.run.smithi167.stdout:Device: 6h/6d Inode: 378 Links: 1 2023-10-05T19:27:25.141 INFO:teuthology.orchestra.run.smithi167.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-10-05T19:27:25.141 INFO:teuthology.orchestra.run.smithi167.stdout:Access: 2023-10-05 19:23:46.475904825 +0000 2023-10-05T19:27:25.141 INFO:teuthology.orchestra.run.smithi167.stdout:Modify: 2023-10-05 19:23:46.471904916 +0000 2023-10-05T19:27:25.141 INFO:teuthology.orchestra.run.smithi167.stdout:Change: 2023-10-05 19:23:46.471904916 +0000 2023-10-05T19:27:25.141 INFO:teuthology.orchestra.run.smithi167.stdout: Birth: - 2023-10-05T19:27:25.141 DEBUG:teuthology.orchestra.run.smithi167:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2023-10-05T19:27:25.152 INFO:teuthology.orchestra.run.smithi167.stderr:1+0 records in 2023-10-05T19:27:25.153 INFO:teuthology.orchestra.run.smithi167.stderr:1+0 records out 2023-10-05T19:27:25.153 INFO:teuthology.orchestra.run.smithi167.stderr:512 bytes copied, 0.000423004 s, 1.2 MB/s 2023-10-05T19:27:25.154 DEBUG:teuthology.orchestra.run.smithi167:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2023-10-05T19:27:25.163 INFO:tasks.ceph:osd dev map: {'osd.1': '/dev/vg_nvme/lv_1', 'osd.2': '/dev/vg_nvme/lv_2', 'osd.3': '/dev/vg_nvme/lv_3'} 2023-10-05T19:27:25.163 INFO:tasks.ceph:remote_to_roles_to_devs: {Remote(name='ubuntu@smithi118.front.sepia.ceph.com'): {'osd.0': '/dev/vg_nvme/lv_1'}, Remote(name='ubuntu@smithi167.front.sepia.ceph.com'): {'osd.1': '/dev/vg_nvme/lv_1', 'osd.2': '/dev/vg_nvme/lv_2', 'osd.3': '/dev/vg_nvme/lv_3'}} 2023-10-05T19:27:25.163 INFO:tasks.ceph:Generating config... 2023-10-05T19:27:25.164 INFO:tasks.ceph:[client] debug ms = 1 2023-10-05T19:27:25.164 INFO:tasks.ceph:[client] rgw gc obj min wait = 15 2023-10-05T19:27:25.164 INFO:tasks.ceph:[client] debug rgw = 20 2023-10-05T19:27:25.164 INFO:tasks.ceph:[client] setgroup = ceph 2023-10-05T19:27:25.165 INFO:tasks.ceph:[client] setuser = ceph 2023-10-05T19:27:25.165 INFO:tasks.ceph:[osd] debug ms = 1 2023-10-05T19:27:25.165 INFO:tasks.ceph:[osd] debug objclass = 20 2023-10-05T19:27:25.165 INFO:tasks.ceph:[osd] bdev async discard = True 2023-10-05T19:27:25.165 INFO:tasks.ceph:[osd] bdev enable discard = True 2023-10-05T19:27:25.165 INFO:tasks.ceph:[osd] bluestore allocator = bitmap 2023-10-05T19:27:25.165 INFO:tasks.ceph:[osd] bluestore block size = 96636764160 2023-10-05T19:27:25.165 INFO:tasks.ceph:[osd] bluestore fsck on mount = True 2023-10-05T19:27:25.165 INFO:tasks.ceph:[osd] debug bluefs = 1/20 2023-10-05T19:27:25.165 INFO:tasks.ceph:[osd] debug bluestore = 1/20 2023-10-05T19:27:25.165 INFO:tasks.ceph:[osd] debug osd = 20 2023-10-05T19:27:25.165 INFO:tasks.ceph:[osd] debug rocksdb = 4/10 2023-10-05T19:27:25.166 INFO:tasks.ceph:[osd] mon osd backfillfull_ratio = 0.85 2023-10-05T19:27:25.166 INFO:tasks.ceph:[osd] mon osd full ratio = 0.9 2023-10-05T19:27:25.166 INFO:tasks.ceph:[osd] mon osd nearfull ratio = 0.8 2023-10-05T19:27:25.166 INFO:tasks.ceph:[osd] osd failsafe full ratio = 0.95 2023-10-05T19:27:25.166 INFO:tasks.ceph:[osd] osd objectstore = bluestore 2023-10-05T19:27:25.166 INFO:tasks.ceph:[mgr] debug mgr = 20 2023-10-05T19:27:25.166 INFO:tasks.ceph:[mgr] debug ms = 1 2023-10-05T19:27:25.166 INFO:tasks.ceph:[mon] debug mon = 20 2023-10-05T19:27:25.166 INFO:tasks.ceph:[mon] debug ms = 1 2023-10-05T19:27:25.166 INFO:tasks.ceph:[mon] debug paxos = 20 2023-10-05T19:27:25.166 INFO:tasks.ceph:Setting up mon.a... 2023-10-05T19:27:25.167 DEBUG:teuthology.orchestra.run.smithi118:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring /etc/ceph/ceph.keyring 2023-10-05T19:27:25.204 INFO:teuthology.orchestra.run.smithi118.stdout:creating /etc/ceph/ceph.keyring 2023-10-05T19:27:25.206 DEBUG:teuthology.orchestra.run.smithi118:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=mon. /etc/ceph/ceph.keyring 2023-10-05T19:27:25.239 DEBUG:teuthology.orchestra.run.smithi118:> sudo chmod 0644 /etc/ceph/ceph.keyring 2023-10-05T19:27:25.250 DEBUG:tasks.ceph:Ceph mon addresses: [('mon.a', '172.21.15.118')] 2023-10-05T19:27:25.250 DEBUG:tasks.ceph:writing out conf {'global': {'chdir': '', 'pid file': '/var/run/ceph/$cluster-$name.pid', 'auth supported': 'cephx', 'filestore xattr use omap': 'true', 'mon clock drift allowed': '1.000', 'osd crush chooseleaf type': '0', 'auth debug': 'true', 'ms die on old message': 'true', 'ms die on bug': 'true', 'mon max pg per osd': '10000', 'mon pg warn max object skew': '0', 'osd_pool_default_pg_autoscale_mode': 'off', 'osd pool default size': '2', 'mon osd allow primary affinity': 'true', 'mon osd allow pg remap': 'true', 'mon warn on legacy crush tunables': 'false', 'mon warn on crush straw calc version zero': 'false', 'mon warn on no sortbitwise': 'false', 'mon warn on osd down out interval zero': 'false', 'mon warn on too few osds': 'false', 'mon_warn_on_pool_pg_num_not_power_of_two': 'false', 'mon_warn_on_pool_no_redundancy': 'false', 'mon_allow_pool_size_one': 'true', 'osd pool default erasure code profile': 'plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd', 'osd default data pool replay window': '5', 'mon allow pool delete': 'true', 'mon cluster log file level': 'debug', 'debug asserts on shutdown': 'true', 'mon health detail to clog': 'false', 'mon host': '172.21.15.118'}, 'osd': {'osd journal size': '100', 'osd scrub load threshold': '5.0', 'osd scrub max interval': '600', 'osd mclock profile': 'high_recovery_ops', 'osd recover clone overlap': 'true', 'osd recovery max chunk': '1048576', 'osd debug shutdown': 'true', 'osd debug op order': 'true', 'osd debug verify stray on activate': 'true', 'osd open classes on start': 'true', 'osd debug pg log writeout': 'true', 'osd deep scrub update digest min age': '30', 'osd map max advance': '10', 'journal zero on create': 'true', 'filestore ondisk finisher threads': '3', 'filestore apply finisher threads': '3', 'bdev debug aio': 'true', 'osd debug misdirected ops': 'true', 'debug ms': 1, 'debug objclass': 20, 'bdev async discard': True, 'bdev enable discard': True, 'bluestore allocator': 'bitmap', 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': '1/20', 'debug bluestore': '1/20', 'debug osd': 20, 'debug rocksdb': '4/10', 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore'}, 'mgr': {'debug ms': 1, 'debug mgr': 20, 'debug mon': '20', 'debug auth': '20', 'mon reweight min pgs per osd': '4', 'mon reweight min bytes per osd': '10', 'mgr/telemetry/nag': 'false'}, 'mon': {'debug ms': 1, 'debug mon': 20, 'debug paxos': 20, 'debug auth': '20', 'mon data avail warn': '5', 'mon mgr mkfs grace': '240', 'mon reweight min pgs per osd': '4', 'mon osd reporter subtree level': 'osd', 'mon osd prime pg temp': 'true', 'mon reweight min bytes per osd': '10', 'auth mon ticket ttl': '660', 'auth service ticket ttl': '240', 'mon_warn_on_insecure_global_id_reclaim': 'false', 'mon_warn_on_insecure_global_id_reclaim_allowed': 'false', 'mon_down_mkfs_grace': '2m', 'mon_warn_on_filestore_osds': 'false'}, 'client': {'rgw cache enabled': 'true', 'rgw enable ops log': 'true', 'rgw enable usage log': 'true', 'log file': '/var/log/ceph/$cluster-$name.$pid.log', 'admin socket': '/var/run/ceph/$cluster-$name.$pid.asok', 'debug ms': 1, 'rgw gc obj min wait': 15, 'debug rgw': 20, 'setgroup': 'ceph', 'setuser': 'ceph'}, 'mon.a': {}} 2023-10-05T19:27:25.251 DEBUG:teuthology.orchestra.run.smithi118:> set -ex 2023-10-05T19:27:25.251 DEBUG:teuthology.orchestra.run.smithi118:> dd of=/home/ubuntu/cephtest/ceph.tmp.conf 2023-10-05T19:27:25.298 DEBUG:teuthology.orchestra.run.smithi118:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage monmaptool -c /home/ubuntu/cephtest/ceph.tmp.conf --create --clobber --enable-all-features --add a 172.21.15.118 --print /home/ubuntu/cephtest/ceph.monmap 2023-10-05T19:27:25.327 INFO:teuthology.orchestra.run.smithi118.stderr:ignoring --setuser ceph since I am not root 2023-10-05T19:27:25.328 INFO:teuthology.orchestra.run.smithi118.stderr:ignoring --setgroup ceph since I am not root 2023-10-05T19:27:25.328 INFO:teuthology.orchestra.run.smithi118.stdout:monmaptool: monmap file /home/ubuntu/cephtest/ceph.monmap 2023-10-05T19:27:25.328 INFO:teuthology.orchestra.run.smithi118.stdout:monmaptool: generated fsid 8c4bd0d1-cbcb-4713-96b7-dcb14bcaf584 2023-10-05T19:27:25.328 INFO:teuthology.orchestra.run.smithi118.stdout:setting min_mon_release = octopus 2023-10-05T19:27:25.328 INFO:teuthology.orchestra.run.smithi118.stdout:epoch 0 2023-10-05T19:27:25.329 INFO:teuthology.orchestra.run.smithi118.stdout:fsid 8c4bd0d1-cbcb-4713-96b7-dcb14bcaf584 2023-10-05T19:27:25.329 INFO:teuthology.orchestra.run.smithi118.stdout:last_changed 2023-10-05T19:27:25.326402+0000 2023-10-05T19:27:25.329 INFO:teuthology.orchestra.run.smithi118.stdout:created 2023-10-05T19:27:25.326402+0000 2023-10-05T19:27:25.329 INFO:teuthology.orchestra.run.smithi118.stdout:min_mon_release 15 (octopus) 2023-10-05T19:27:25.329 INFO:teuthology.orchestra.run.smithi118.stdout:election_strategy: 1 2023-10-05T19:27:25.329 INFO:teuthology.orchestra.run.smithi118.stdout:0: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.a 2023-10-05T19:27:25.329 INFO:teuthology.orchestra.run.smithi118.stdout:monmaptool: writing epoch 0 to /home/ubuntu/cephtest/ceph.monmap (1 monitors) 2023-10-05T19:27:25.330 DEBUG:teuthology.orchestra.run.smithi118:> rm -- /home/ubuntu/cephtest/ceph.tmp.conf 2023-10-05T19:27:25.334 INFO:tasks.ceph:Writing /etc/ceph/ceph.conf for FSID 8c4bd0d1-cbcb-4713-96b7-dcb14bcaf584... 2023-10-05T19:27:25.335 DEBUG:teuthology.orchestra.run.smithi118:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2023-10-05T19:27:25.337 DEBUG:teuthology.orchestra.run.smithi167:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2023-10-05T19:27:25.364 INFO:teuthology.orchestra.run.smithi118.stdout:[global] 2023-10-05T19:27:25.364 INFO:teuthology.orchestra.run.smithi118.stdout: chdir = "" 2023-10-05T19:27:25.364 INFO:teuthology.orchestra.run.smithi118.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2023-10-05T19:27:25.365 INFO:teuthology.orchestra.run.smithi118.stdout: auth supported = cephx 2023-10-05T19:27:25.365 INFO:teuthology.orchestra.run.smithi118.stdout: 2023-10-05T19:27:25.365 INFO:teuthology.orchestra.run.smithi118.stdout: filestore xattr use omap = true 2023-10-05T19:27:25.365 INFO:teuthology.orchestra.run.smithi118.stdout: 2023-10-05T19:27:25.365 INFO:teuthology.orchestra.run.smithi118.stdout: mon clock drift allowed = 1.000 2023-10-05T19:27:25.365 INFO:teuthology.orchestra.run.smithi118.stdout: 2023-10-05T19:27:25.365 INFO:teuthology.orchestra.run.smithi118.stdout: osd crush chooseleaf type = 0 2023-10-05T19:27:25.365 INFO:teuthology.orchestra.run.smithi118.stdout: auth debug = true 2023-10-05T19:27:25.365 INFO:teuthology.orchestra.run.smithi118.stdout: 2023-10-05T19:27:25.365 INFO:teuthology.orchestra.run.smithi118.stdout: ms die on old message = true 2023-10-05T19:27:25.365 INFO:teuthology.orchestra.run.smithi118.stdout: ms die on bug = true 2023-10-05T19:27:25.365 INFO:teuthology.orchestra.run.smithi118.stdout: 2023-10-05T19:27:25.366 INFO:teuthology.orchestra.run.smithi118.stdout: mon max pg per osd = 10000 # >= luminous 2023-10-05T19:27:25.366 INFO:teuthology.orchestra.run.smithi118.stdout: mon pg warn max object skew = 0 2023-10-05T19:27:25.366 INFO:teuthology.orchestra.run.smithi118.stdout: 2023-10-05T19:27:25.366 INFO:teuthology.orchestra.run.smithi118.stdout: # disable pg_autoscaler by default for new pools 2023-10-05T19:27:25.366 INFO:teuthology.orchestra.run.smithi118.stdout: osd_pool_default_pg_autoscale_mode = off 2023-10-05T19:27:25.366 INFO:teuthology.orchestra.run.smithi118.stdout: 2023-10-05T19:27:25.366 INFO:teuthology.orchestra.run.smithi118.stdout: osd pool default size = 2 2023-10-05T19:27:25.366 INFO:teuthology.orchestra.run.smithi118.stdout: 2023-10-05T19:27:25.366 INFO:teuthology.orchestra.run.smithi118.stdout: mon osd allow primary affinity = true 2023-10-05T19:27:25.366 INFO:teuthology.orchestra.run.smithi118.stdout: mon osd allow pg remap = true 2023-10-05T19:27:25.366 INFO:teuthology.orchestra.run.smithi118.stdout: mon warn on legacy crush tunables = false 2023-10-05T19:27:25.366 INFO:teuthology.orchestra.run.smithi118.stdout: mon warn on crush straw calc version zero = false 2023-10-05T19:27:25.367 INFO:teuthology.orchestra.run.smithi118.stdout: mon warn on no sortbitwise = false 2023-10-05T19:27:25.367 INFO:teuthology.orchestra.run.smithi118.stdout: mon warn on osd down out interval zero = false 2023-10-05T19:27:25.367 INFO:teuthology.orchestra.run.smithi118.stdout: mon warn on too few osds = false 2023-10-05T19:27:25.367 INFO:teuthology.orchestra.run.smithi118.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2023-10-05T19:27:25.367 INFO:teuthology.orchestra.run.smithi118.stdout: mon_warn_on_pool_no_redundancy = false 2023-10-05T19:27:25.367 INFO:teuthology.orchestra.run.smithi118.stdout: mon_allow_pool_size_one = true 2023-10-05T19:27:25.367 INFO:teuthology.orchestra.run.smithi118.stdout: 2023-10-05T19:27:25.367 INFO:teuthology.orchestra.run.smithi118.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2023-10-05T19:27:25.367 INFO:teuthology.orchestra.run.smithi118.stdout: 2023-10-05T19:27:25.367 INFO:teuthology.orchestra.run.smithi118.stdout: osd default data pool replay window = 5 2023-10-05T19:27:25.367 INFO:teuthology.orchestra.run.smithi118.stdout: 2023-10-05T19:27:25.367 INFO:teuthology.orchestra.run.smithi118.stdout: mon allow pool delete = true 2023-10-05T19:27:25.367 INFO:teuthology.orchestra.run.smithi118.stdout: 2023-10-05T19:27:25.368 INFO:teuthology.orchestra.run.smithi118.stdout: mon cluster log file level = debug 2023-10-05T19:27:25.368 INFO:teuthology.orchestra.run.smithi118.stdout: debug asserts on shutdown = true 2023-10-05T19:27:25.368 INFO:teuthology.orchestra.run.smithi118.stdout: mon health detail to clog = false 2023-10-05T19:27:25.368 INFO:teuthology.orchestra.run.smithi118.stdout: mon host = 172.21.15.118 2023-10-05T19:27:25.368 INFO:teuthology.orchestra.run.smithi118.stdout: fsid = 8c4bd0d1-cbcb-4713-96b7-dcb14bcaf584 2023-10-05T19:27:25.368 INFO:teuthology.orchestra.run.smithi118.stdout: 2023-10-05T19:27:25.368 INFO:teuthology.orchestra.run.smithi118.stdout:[osd] 2023-10-05T19:27:25.368 INFO:teuthology.orchestra.run.smithi118.stdout: osd journal size = 100 2023-10-05T19:27:25.368 INFO:teuthology.orchestra.run.smithi118.stdout: 2023-10-05T19:27:25.368 INFO:teuthology.orchestra.run.smithi118.stdout: osd scrub load threshold = 5.0 2023-10-05T19:27:25.368 INFO:teuthology.orchestra.run.smithi118.stdout: osd scrub max interval = 600 2023-10-05T19:27:25.368 INFO:teuthology.orchestra.run.smithi118.stdout: osd mclock profile = high_recovery_ops 2023-10-05T19:27:25.369 INFO:teuthology.orchestra.run.smithi118.stdout: 2023-10-05T19:27:25.369 INFO:teuthology.orchestra.run.smithi118.stdout: osd recover clone overlap = true 2023-10-05T19:27:25.369 INFO:teuthology.orchestra.run.smithi118.stdout: osd recovery max chunk = 1048576 2023-10-05T19:27:25.369 INFO:teuthology.orchestra.run.smithi118.stdout: 2023-10-05T19:27:25.369 INFO:teuthology.orchestra.run.smithi118.stdout: osd debug shutdown = true 2023-10-05T19:27:25.369 INFO:teuthology.orchestra.run.smithi118.stdout: osd debug op order = true 2023-10-05T19:27:25.369 INFO:teuthology.orchestra.run.smithi118.stdout: osd debug verify stray on activate = true 2023-10-05T19:27:25.369 INFO:teuthology.orchestra.run.smithi118.stdout: 2023-10-05T19:27:25.369 INFO:teuthology.orchestra.run.smithi118.stdout: osd open classes on start = true 2023-10-05T19:27:25.369 INFO:teuthology.orchestra.run.smithi118.stdout: osd debug pg log writeout = true 2023-10-05T19:27:25.369 INFO:teuthology.orchestra.run.smithi118.stdout: 2023-10-05T19:27:25.369 INFO:teuthology.orchestra.run.smithi118.stdout: osd deep scrub update digest min age = 30 2023-10-05T19:27:25.370 INFO:teuthology.orchestra.run.smithi118.stdout: 2023-10-05T19:27:25.370 INFO:teuthology.orchestra.run.smithi118.stdout: osd map max advance = 10 2023-10-05T19:27:25.370 INFO:teuthology.orchestra.run.smithi118.stdout: 2023-10-05T19:27:25.370 INFO:teuthology.orchestra.run.smithi118.stdout: journal zero on create = true 2023-10-05T19:27:25.370 INFO:teuthology.orchestra.run.smithi118.stdout: 2023-10-05T19:27:25.370 INFO:teuthology.orchestra.run.smithi118.stdout: filestore ondisk finisher threads = 3 2023-10-05T19:27:25.370 INFO:teuthology.orchestra.run.smithi118.stdout: filestore apply finisher threads = 3 2023-10-05T19:27:25.370 INFO:teuthology.orchestra.run.smithi118.stdout: 2023-10-05T19:27:25.370 INFO:teuthology.orchestra.run.smithi118.stdout: bdev debug aio = true 2023-10-05T19:27:25.370 INFO:teuthology.orchestra.run.smithi118.stdout: osd debug misdirected ops = true 2023-10-05T19:27:25.370 INFO:teuthology.orchestra.run.smithi118.stdout: debug ms = 1 2023-10-05T19:27:25.371 INFO:teuthology.orchestra.run.smithi118.stdout: debug objclass = 20 2023-10-05T19:27:25.371 INFO:teuthology.orchestra.run.smithi118.stdout: bdev async discard = True 2023-10-05T19:27:25.371 INFO:teuthology.orchestra.run.smithi118.stdout: bdev enable discard = True 2023-10-05T19:27:25.371 INFO:teuthology.orchestra.run.smithi118.stdout: bluestore allocator = bitmap 2023-10-05T19:27:25.371 INFO:teuthology.orchestra.run.smithi118.stdout: bluestore block size = 96636764160 2023-10-05T19:27:25.371 INFO:teuthology.orchestra.run.smithi118.stdout: bluestore fsck on mount = True 2023-10-05T19:27:25.371 INFO:teuthology.orchestra.run.smithi118.stdout: debug bluefs = 1/20 2023-10-05T19:27:25.371 INFO:teuthology.orchestra.run.smithi118.stdout: debug bluestore = 1/20 2023-10-05T19:27:25.371 INFO:teuthology.orchestra.run.smithi118.stdout: debug osd = 20 2023-10-05T19:27:25.371 INFO:teuthology.orchestra.run.smithi118.stdout: debug rocksdb = 4/10 2023-10-05T19:27:25.371 INFO:teuthology.orchestra.run.smithi118.stdout: mon osd backfillfull_ratio = 0.85 2023-10-05T19:27:25.371 INFO:teuthology.orchestra.run.smithi118.stdout: mon osd full ratio = 0.9 2023-10-05T19:27:25.372 INFO:teuthology.orchestra.run.smithi118.stdout: mon osd nearfull ratio = 0.8 2023-10-05T19:27:25.372 INFO:teuthology.orchestra.run.smithi118.stdout: osd failsafe full ratio = 0.95 2023-10-05T19:27:25.372 INFO:teuthology.orchestra.run.smithi118.stdout: osd objectstore = bluestore 2023-10-05T19:27:25.372 INFO:teuthology.orchestra.run.smithi118.stdout: 2023-10-05T19:27:25.372 INFO:teuthology.orchestra.run.smithi118.stdout:[mgr] 2023-10-05T19:27:25.372 INFO:teuthology.orchestra.run.smithi118.stdout: debug ms = 1 2023-10-05T19:27:25.372 INFO:teuthology.orchestra.run.smithi118.stdout: debug mgr = 20 2023-10-05T19:27:25.372 INFO:teuthology.orchestra.run.smithi118.stdout: debug mon = 20 2023-10-05T19:27:25.372 INFO:teuthology.orchestra.run.smithi118.stdout: debug auth = 20 2023-10-05T19:27:25.372 INFO:teuthology.orchestra.run.smithi118.stdout: mon reweight min pgs per osd = 4 2023-10-05T19:27:25.372 INFO:teuthology.orchestra.run.smithi118.stdout: mon reweight min bytes per osd = 10 2023-10-05T19:27:25.372 INFO:teuthology.orchestra.run.smithi118.stdout: mgr/telemetry/nag = false 2023-10-05T19:27:25.373 INFO:teuthology.orchestra.run.smithi118.stdout: 2023-10-05T19:27:25.373 INFO:teuthology.orchestra.run.smithi118.stdout:[mon] 2023-10-05T19:27:25.373 INFO:teuthology.orchestra.run.smithi118.stdout: debug ms = 1 2023-10-05T19:27:25.373 INFO:teuthology.orchestra.run.smithi118.stdout: debug mon = 20 2023-10-05T19:27:25.373 INFO:teuthology.orchestra.run.smithi118.stdout: debug paxos = 20 2023-10-05T19:27:25.373 INFO:teuthology.orchestra.run.smithi118.stdout: debug auth = 20 2023-10-05T19:27:25.373 INFO:teuthology.orchestra.run.smithi118.stdout: mon data avail warn = 5 2023-10-05T19:27:25.373 INFO:teuthology.orchestra.run.smithi118.stdout: mon mgr mkfs grace = 240 2023-10-05T19:27:25.373 INFO:teuthology.orchestra.run.smithi118.stdout: mon reweight min pgs per osd = 4 2023-10-05T19:27:25.373 INFO:teuthology.orchestra.run.smithi118.stdout: mon osd reporter subtree level = osd 2023-10-05T19:27:25.373 INFO:teuthology.orchestra.run.smithi118.stdout: mon osd prime pg temp = true 2023-10-05T19:27:25.373 INFO:teuthology.orchestra.run.smithi118.stdout: mon reweight min bytes per osd = 10 2023-10-05T19:27:25.373 INFO:teuthology.orchestra.run.smithi118.stdout: 2023-10-05T19:27:25.374 INFO:teuthology.orchestra.run.smithi118.stdout: # rotate auth tickets quickly to exercise renewal paths 2023-10-05T19:27:25.374 INFO:teuthology.orchestra.run.smithi118.stdout: auth mon ticket ttl = 660 # 11m 2023-10-05T19:27:25.374 INFO:teuthology.orchestra.run.smithi118.stdout: auth service ticket ttl = 240 # 4m 2023-10-05T19:27:25.374 INFO:teuthology.orchestra.run.smithi118.stdout: 2023-10-05T19:27:25.374 INFO:teuthology.orchestra.run.smithi118.stdout: # don't complain about insecure global_id in the test suite 2023-10-05T19:27:25.374 INFO:teuthology.orchestra.run.smithi118.stdout: mon_warn_on_insecure_global_id_reclaim = false 2023-10-05T19:27:25.374 INFO:teuthology.orchestra.run.smithi118.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2023-10-05T19:27:25.374 INFO:teuthology.orchestra.run.smithi118.stdout: 2023-10-05T19:27:25.374 INFO:teuthology.orchestra.run.smithi118.stdout: # 1m isn't quite enough 2023-10-05T19:27:25.374 INFO:teuthology.orchestra.run.smithi118.stdout: mon_down_mkfs_grace = 2m 2023-10-05T19:27:25.374 INFO:teuthology.orchestra.run.smithi118.stdout: 2023-10-05T19:27:25.375 INFO:teuthology.orchestra.run.smithi118.stdout: mon_warn_on_filestore_osds = false 2023-10-05T19:27:25.375 INFO:teuthology.orchestra.run.smithi118.stdout: 2023-10-05T19:27:25.375 INFO:teuthology.orchestra.run.smithi118.stdout:[client] 2023-10-05T19:27:25.375 INFO:teuthology.orchestra.run.smithi118.stdout: rgw cache enabled = true 2023-10-05T19:27:25.375 INFO:teuthology.orchestra.run.smithi118.stdout: rgw enable ops log = true 2023-10-05T19:27:25.375 INFO:teuthology.orchestra.run.smithi118.stdout: rgw enable usage log = true 2023-10-05T19:27:25.375 INFO:teuthology.orchestra.run.smithi118.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2023-10-05T19:27:25.375 INFO:teuthology.orchestra.run.smithi118.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2023-10-05T19:27:25.375 INFO:teuthology.orchestra.run.smithi118.stdout: debug ms = 1 2023-10-05T19:27:25.375 INFO:teuthology.orchestra.run.smithi118.stdout: rgw gc obj min wait = 15 2023-10-05T19:27:25.375 INFO:teuthology.orchestra.run.smithi118.stdout: debug rgw = 20 2023-10-05T19:27:25.375 INFO:teuthology.orchestra.run.smithi118.stdout: setgroup = ceph 2023-10-05T19:27:25.375 INFO:teuthology.orchestra.run.smithi118.stdout: setuser = ceph 2023-10-05T19:27:25.376 INFO:teuthology.orchestra.run.smithi118.stdout:[mon.a] 2023-10-05T19:27:25.376 INFO:teuthology.orchestra.run.smithi167.stdout:[global] 2023-10-05T19:27:25.376 INFO:teuthology.orchestra.run.smithi167.stdout: chdir = "" 2023-10-05T19:27:25.376 INFO:teuthology.orchestra.run.smithi167.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2023-10-05T19:27:25.376 INFO:teuthology.orchestra.run.smithi167.stdout: auth supported = cephx 2023-10-05T19:27:25.377 INFO:teuthology.orchestra.run.smithi167.stdout: 2023-10-05T19:27:25.377 INFO:teuthology.orchestra.run.smithi167.stdout: filestore xattr use omap = true 2023-10-05T19:27:25.377 INFO:teuthology.orchestra.run.smithi167.stdout: 2023-10-05T19:27:25.377 INFO:teuthology.orchestra.run.smithi167.stdout: mon clock drift allowed = 1.000 2023-10-05T19:27:25.377 INFO:teuthology.orchestra.run.smithi167.stdout: 2023-10-05T19:27:25.377 INFO:teuthology.orchestra.run.smithi167.stdout: osd crush chooseleaf type = 0 2023-10-05T19:27:25.377 INFO:teuthology.orchestra.run.smithi167.stdout: auth debug = true 2023-10-05T19:27:25.377 INFO:teuthology.orchestra.run.smithi167.stdout: 2023-10-05T19:27:25.377 INFO:teuthology.orchestra.run.smithi167.stdout: ms die on old message = true 2023-10-05T19:27:25.377 INFO:teuthology.orchestra.run.smithi167.stdout: ms die on bug = true 2023-10-05T19:27:25.377 INFO:teuthology.orchestra.run.smithi167.stdout: 2023-10-05T19:27:25.377 INFO:teuthology.orchestra.run.smithi167.stdout: mon max pg per osd = 10000 # >= luminous 2023-10-05T19:27:25.378 INFO:teuthology.orchestra.run.smithi167.stdout: mon pg warn max object skew = 0 2023-10-05T19:27:25.378 INFO:teuthology.orchestra.run.smithi167.stdout: 2023-10-05T19:27:25.378 INFO:teuthology.orchestra.run.smithi167.stdout: # disable pg_autoscaler by default for new pools 2023-10-05T19:27:25.378 INFO:teuthology.orchestra.run.smithi167.stdout: osd_pool_default_pg_autoscale_mode = off 2023-10-05T19:27:25.378 INFO:teuthology.orchestra.run.smithi167.stdout: 2023-10-05T19:27:25.378 INFO:teuthology.orchestra.run.smithi167.stdout: osd pool default size = 2 2023-10-05T19:27:25.378 INFO:teuthology.orchestra.run.smithi167.stdout: 2023-10-05T19:27:25.378 INFO:teuthology.orchestra.run.smithi167.stdout: mon osd allow primary affinity = true 2023-10-05T19:27:25.378 INFO:teuthology.orchestra.run.smithi167.stdout: mon osd allow pg remap = true 2023-10-05T19:27:25.378 INFO:teuthology.orchestra.run.smithi167.stdout: mon warn on legacy crush tunables = false 2023-10-05T19:27:25.378 INFO:teuthology.orchestra.run.smithi167.stdout: mon warn on crush straw calc version zero = false 2023-10-05T19:27:25.378 INFO:teuthology.orchestra.run.smithi167.stdout: mon warn on no sortbitwise = false 2023-10-05T19:27:25.379 INFO:teuthology.orchestra.run.smithi167.stdout: mon warn on osd down out interval zero = false 2023-10-05T19:27:25.379 INFO:teuthology.orchestra.run.smithi167.stdout: mon warn on too few osds = false 2023-10-05T19:27:25.379 INFO:teuthology.orchestra.run.smithi167.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2023-10-05T19:27:25.379 INFO:teuthology.orchestra.run.smithi167.stdout: mon_warn_on_pool_no_redundancy = false 2023-10-05T19:27:25.379 INFO:teuthology.orchestra.run.smithi167.stdout: mon_allow_pool_size_one = true 2023-10-05T19:27:25.379 INFO:teuthology.orchestra.run.smithi167.stdout: 2023-10-05T19:27:25.379 INFO:teuthology.orchestra.run.smithi167.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2023-10-05T19:27:25.379 INFO:teuthology.orchestra.run.smithi167.stdout: 2023-10-05T19:27:25.379 INFO:teuthology.orchestra.run.smithi167.stdout: osd default data pool replay window = 5 2023-10-05T19:27:25.379 INFO:teuthology.orchestra.run.smithi167.stdout: 2023-10-05T19:27:25.379 INFO:teuthology.orchestra.run.smithi167.stdout: mon allow pool delete = true 2023-10-05T19:27:25.379 INFO:teuthology.orchestra.run.smithi167.stdout: 2023-10-05T19:27:25.380 INFO:teuthology.orchestra.run.smithi167.stdout: mon cluster log file level = debug 2023-10-05T19:27:25.380 INFO:teuthology.orchestra.run.smithi167.stdout: debug asserts on shutdown = true 2023-10-05T19:27:25.380 INFO:teuthology.orchestra.run.smithi167.stdout: mon health detail to clog = false 2023-10-05T19:27:25.380 INFO:teuthology.orchestra.run.smithi167.stdout: mon host = 172.21.15.118 2023-10-05T19:27:25.380 INFO:teuthology.orchestra.run.smithi167.stdout: fsid = 8c4bd0d1-cbcb-4713-96b7-dcb14bcaf584 2023-10-05T19:27:25.380 INFO:teuthology.orchestra.run.smithi167.stdout: 2023-10-05T19:27:25.380 INFO:teuthology.orchestra.run.smithi167.stdout:[osd] 2023-10-05T19:27:25.380 INFO:teuthology.orchestra.run.smithi167.stdout: osd journal size = 100 2023-10-05T19:27:25.380 INFO:teuthology.orchestra.run.smithi167.stdout: 2023-10-05T19:27:25.380 INFO:teuthology.orchestra.run.smithi167.stdout: osd scrub load threshold = 5.0 2023-10-05T19:27:25.380 INFO:teuthology.orchestra.run.smithi167.stdout: osd scrub max interval = 600 2023-10-05T19:27:25.380 INFO:teuthology.orchestra.run.smithi167.stdout: osd mclock profile = high_recovery_ops 2023-10-05T19:27:25.380 INFO:teuthology.orchestra.run.smithi167.stdout: 2023-10-05T19:27:25.381 INFO:teuthology.orchestra.run.smithi167.stdout: osd recover clone overlap = true 2023-10-05T19:27:25.381 INFO:teuthology.orchestra.run.smithi167.stdout: osd recovery max chunk = 1048576 2023-10-05T19:27:25.381 INFO:teuthology.orchestra.run.smithi167.stdout: 2023-10-05T19:27:25.381 INFO:teuthology.orchestra.run.smithi167.stdout: osd debug shutdown = true 2023-10-05T19:27:25.381 INFO:teuthology.orchestra.run.smithi167.stdout: osd debug op order = true 2023-10-05T19:27:25.381 INFO:teuthology.orchestra.run.smithi167.stdout: osd debug verify stray on activate = true 2023-10-05T19:27:25.381 INFO:teuthology.orchestra.run.smithi167.stdout: 2023-10-05T19:27:25.381 INFO:teuthology.orchestra.run.smithi167.stdout: osd open classes on start = true 2023-10-05T19:27:25.381 INFO:teuthology.orchestra.run.smithi167.stdout: osd debug pg log writeout = true 2023-10-05T19:27:25.381 INFO:teuthology.orchestra.run.smithi167.stdout: 2023-10-05T19:27:25.381 INFO:teuthology.orchestra.run.smithi167.stdout: osd deep scrub update digest min age = 30 2023-10-05T19:27:25.381 INFO:teuthology.orchestra.run.smithi167.stdout: 2023-10-05T19:27:25.381 INFO:teuthology.orchestra.run.smithi167.stdout: osd map max advance = 10 2023-10-05T19:27:25.382 INFO:teuthology.orchestra.run.smithi167.stdout: 2023-10-05T19:27:25.382 INFO:teuthology.orchestra.run.smithi167.stdout: journal zero on create = true 2023-10-05T19:27:25.382 INFO:teuthology.orchestra.run.smithi167.stdout: 2023-10-05T19:27:25.382 INFO:teuthology.orchestra.run.smithi167.stdout: filestore ondisk finisher threads = 3 2023-10-05T19:27:25.382 INFO:teuthology.orchestra.run.smithi167.stdout: filestore apply finisher threads = 3 2023-10-05T19:27:25.382 INFO:teuthology.orchestra.run.smithi167.stdout: 2023-10-05T19:27:25.382 INFO:teuthology.orchestra.run.smithi167.stdout: bdev debug aio = true 2023-10-05T19:27:25.382 INFO:teuthology.orchestra.run.smithi167.stdout: osd debug misdirected ops = true 2023-10-05T19:27:25.382 INFO:teuthology.orchestra.run.smithi167.stdout: debug ms = 1 2023-10-05T19:27:25.382 INFO:teuthology.orchestra.run.smithi167.stdout: debug objclass = 20 2023-10-05T19:27:25.382 INFO:teuthology.orchestra.run.smithi167.stdout: bdev async discard = True 2023-10-05T19:27:25.382 INFO:teuthology.orchestra.run.smithi167.stdout: bdev enable discard = True 2023-10-05T19:27:25.383 INFO:teuthology.orchestra.run.smithi167.stdout: bluestore allocator = bitmap 2023-10-05T19:27:25.383 INFO:teuthology.orchestra.run.smithi167.stdout: bluestore block size = 96636764160 2023-10-05T19:27:25.383 INFO:teuthology.orchestra.run.smithi167.stdout: bluestore fsck on mount = True 2023-10-05T19:27:25.383 INFO:teuthology.orchestra.run.smithi167.stdout: debug bluefs = 1/20 2023-10-05T19:27:25.383 INFO:teuthology.orchestra.run.smithi167.stdout: debug bluestore = 1/20 2023-10-05T19:27:25.383 INFO:teuthology.orchestra.run.smithi167.stdout: debug osd = 20 2023-10-05T19:27:25.383 INFO:teuthology.orchestra.run.smithi167.stdout: debug rocksdb = 4/10 2023-10-05T19:27:25.383 INFO:teuthology.orchestra.run.smithi167.stdout: mon osd backfillfull_ratio = 0.85 2023-10-05T19:27:25.383 INFO:teuthology.orchestra.run.smithi167.stdout: mon osd full ratio = 0.9 2023-10-05T19:27:25.383 INFO:teuthology.orchestra.run.smithi167.stdout: mon osd nearfull ratio = 0.8 2023-10-05T19:27:25.383 INFO:teuthology.orchestra.run.smithi167.stdout: osd failsafe full ratio = 0.95 2023-10-05T19:27:25.383 INFO:teuthology.orchestra.run.smithi167.stdout: osd objectstore = bluestore 2023-10-05T19:27:25.383 INFO:teuthology.orchestra.run.smithi167.stdout: 2023-10-05T19:27:25.384 INFO:teuthology.orchestra.run.smithi167.stdout:[mgr] 2023-10-05T19:27:25.384 INFO:teuthology.orchestra.run.smithi167.stdout: debug ms = 1 2023-10-05T19:27:25.384 INFO:teuthology.orchestra.run.smithi167.stdout: debug mgr = 20 2023-10-05T19:27:25.384 INFO:teuthology.orchestra.run.smithi167.stdout: debug mon = 20 2023-10-05T19:27:25.384 INFO:teuthology.orchestra.run.smithi167.stdout: debug auth = 20 2023-10-05T19:27:25.384 INFO:teuthology.orchestra.run.smithi167.stdout: mon reweight min pgs per osd = 4 2023-10-05T19:27:25.384 INFO:teuthology.orchestra.run.smithi167.stdout: mon reweight min bytes per osd = 10 2023-10-05T19:27:25.384 INFO:teuthology.orchestra.run.smithi167.stdout: mgr/telemetry/nag = false 2023-10-05T19:27:25.384 INFO:teuthology.orchestra.run.smithi167.stdout: 2023-10-05T19:27:25.384 INFO:teuthology.orchestra.run.smithi167.stdout:[mon] 2023-10-05T19:27:25.384 INFO:teuthology.orchestra.run.smithi167.stdout: debug ms = 1 2023-10-05T19:27:25.384 INFO:teuthology.orchestra.run.smithi167.stdout: debug mon = 20 2023-10-05T19:27:25.384 INFO:teuthology.orchestra.run.smithi167.stdout: debug paxos = 20 2023-10-05T19:27:25.385 INFO:teuthology.orchestra.run.smithi167.stdout: debug auth = 20 2023-10-05T19:27:25.385 INFO:teuthology.orchestra.run.smithi167.stdout: mon data avail warn = 5 2023-10-05T19:27:25.385 INFO:teuthology.orchestra.run.smithi167.stdout: mon mgr mkfs grace = 240 2023-10-05T19:27:25.385 INFO:teuthology.orchestra.run.smithi167.stdout: mon reweight min pgs per osd = 4 2023-10-05T19:27:25.385 INFO:teuthology.orchestra.run.smithi167.stdout: mon osd reporter subtree level = osd 2023-10-05T19:27:25.385 INFO:teuthology.orchestra.run.smithi167.stdout: mon osd prime pg temp = true 2023-10-05T19:27:25.385 INFO:teuthology.orchestra.run.smithi167.stdout: mon reweight min bytes per osd = 10 2023-10-05T19:27:25.385 INFO:teuthology.orchestra.run.smithi167.stdout: 2023-10-05T19:27:25.385 INFO:teuthology.orchestra.run.smithi167.stdout: # rotate auth tickets quickly to exercise renewal paths 2023-10-05T19:27:25.385 INFO:teuthology.orchestra.run.smithi167.stdout: auth mon ticket ttl = 660 # 11m 2023-10-05T19:27:25.385 INFO:teuthology.orchestra.run.smithi167.stdout: auth service ticket ttl = 240 # 4m 2023-10-05T19:27:25.386 INFO:teuthology.orchestra.run.smithi167.stdout: 2023-10-05T19:27:25.386 INFO:teuthology.orchestra.run.smithi167.stdout: # don't complain about insecure global_id in the test suite 2023-10-05T19:27:25.386 INFO:teuthology.orchestra.run.smithi167.stdout: mon_warn_on_insecure_global_id_reclaim = false 2023-10-05T19:27:25.386 INFO:teuthology.orchestra.run.smithi167.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2023-10-05T19:27:25.386 INFO:teuthology.orchestra.run.smithi167.stdout: 2023-10-05T19:27:25.386 INFO:teuthology.orchestra.run.smithi167.stdout: # 1m isn't quite enough 2023-10-05T19:27:25.386 INFO:teuthology.orchestra.run.smithi167.stdout: mon_down_mkfs_grace = 2m 2023-10-05T19:27:25.386 INFO:teuthology.orchestra.run.smithi167.stdout: 2023-10-05T19:27:25.386 INFO:teuthology.orchestra.run.smithi167.stdout: mon_warn_on_filestore_osds = false 2023-10-05T19:27:25.386 INFO:teuthology.orchestra.run.smithi167.stdout: 2023-10-05T19:27:25.386 INFO:teuthology.orchestra.run.smithi167.stdout:[client] 2023-10-05T19:27:25.386 INFO:teuthology.orchestra.run.smithi167.stdout: rgw cache enabled = true 2023-10-05T19:27:25.387 INFO:teuthology.orchestra.run.smithi167.stdout: rgw enable ops log = true 2023-10-05T19:27:25.387 INFO:teuthology.orchestra.run.smithi167.stdout: rgw enable usage log = true 2023-10-05T19:27:25.387 INFO:teuthology.orchestra.run.smithi167.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2023-10-05T19:27:25.387 INFO:teuthology.orchestra.run.smithi167.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2023-10-05T19:27:25.387 INFO:teuthology.orchestra.run.smithi167.stdout: debug ms = 1 2023-10-05T19:27:25.387 INFO:teuthology.orchestra.run.smithi167.stdout: rgw gc obj min wait = 15 2023-10-05T19:27:25.387 INFO:teuthology.orchestra.run.smithi167.stdout: debug rgw = 20 2023-10-05T19:27:25.387 INFO:teuthology.orchestra.run.smithi167.stdout: setgroup = ceph 2023-10-05T19:27:25.387 INFO:teuthology.orchestra.run.smithi167.stdout: setuser = ceph 2023-10-05T19:27:25.387 INFO:teuthology.orchestra.run.smithi167.stdout:[mon.a] 2023-10-05T19:27:25.388 INFO:tasks.ceph:Creating admin key on mon.a... 2023-10-05T19:27:25.389 DEBUG:teuthology.orchestra.run.smithi118:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=client.admin --cap mon 'allow *' --cap osd 'allow *' --cap mds 'allow *' --cap mgr 'allow *' /etc/ceph/ceph.keyring 2023-10-05T19:27:25.480 INFO:tasks.ceph:Copying monmap to all nodes... 2023-10-05T19:27:25.480 DEBUG:teuthology.orchestra.run.smithi118:> set -ex 2023-10-05T19:27:25.480 DEBUG:teuthology.orchestra.run.smithi118:> dd if=/etc/ceph/ceph.keyring of=/dev/stdout 2023-10-05T19:27:25.487 DEBUG:teuthology.orchestra.run.smithi118:> set -ex 2023-10-05T19:27:25.487 DEBUG:teuthology.orchestra.run.smithi118:> dd if=/home/ubuntu/cephtest/ceph.monmap of=/dev/stdout 2023-10-05T19:27:25.494 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi118.front.sepia.ceph.com 2023-10-05T19:27:25.494 DEBUG:teuthology.orchestra.run.smithi118:> set -ex 2023-10-05T19:27:25.494 DEBUG:teuthology.orchestra.run.smithi118:> sudo dd of=/etc/ceph/ceph.keyring 2023-10-05T19:27:25.494 DEBUG:teuthology.orchestra.run.smithi118:> sudo chmod 0644 /etc/ceph/ceph.keyring 2023-10-05T19:27:25.518 DEBUG:teuthology.orchestra.run.smithi118:> set -ex 2023-10-05T19:27:25.518 DEBUG:teuthology.orchestra.run.smithi118:> dd of=/home/ubuntu/cephtest/ceph.monmap 2023-10-05T19:27:25.525 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi167.front.sepia.ceph.com 2023-10-05T19:27:25.525 DEBUG:teuthology.orchestra.run.smithi167:> set -ex 2023-10-05T19:27:25.525 DEBUG:teuthology.orchestra.run.smithi167:> sudo dd of=/etc/ceph/ceph.keyring 2023-10-05T19:27:25.525 DEBUG:teuthology.orchestra.run.smithi167:> sudo chmod 0644 /etc/ceph/ceph.keyring 2023-10-05T19:27:25.551 DEBUG:teuthology.orchestra.run.smithi167:> set -ex 2023-10-05T19:27:25.551 DEBUG:teuthology.orchestra.run.smithi167:> dd of=/home/ubuntu/cephtest/ceph.monmap 2023-10-05T19:27:25.557 INFO:tasks.ceph:Setting up mon nodes... 2023-10-05T19:27:25.557 INFO:tasks.ceph:Setting up mgr nodes... 2023-10-05T19:27:25.558 DEBUG:teuthology.orchestra.run.smithi167:> sudo mkdir -p /var/lib/ceph/mgr/ceph-x && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mgr.x /var/lib/ceph/mgr/ceph-x/keyring 2023-10-05T19:27:25.596 INFO:teuthology.orchestra.run.smithi167.stdout:creating /var/lib/ceph/mgr/ceph-x/keyring 2023-10-05T19:27:25.598 INFO:tasks.ceph:Setting up mds nodes... 2023-10-05T19:27:25.599 INFO:tasks.ceph_client:Setting up client nodes... 2023-10-05T19:27:25.599 DEBUG:teuthology.orchestra.run.smithi167:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.0 /etc/ceph/ceph.client.0.keyring && sudo chmod 0644 /etc/ceph/ceph.client.0.keyring 2023-10-05T19:27:25.678 INFO:teuthology.orchestra.run.smithi167.stdout:creating /etc/ceph/ceph.client.0.keyring 2023-10-05T19:27:25.691 INFO:tasks.ceph:Running mkfs on osd nodes... 2023-10-05T19:27:25.691 INFO:tasks.ceph:ctx.disk_config.remote_to_roles_to_dev: {Remote(name='ubuntu@smithi118.front.sepia.ceph.com'): {'osd.0': '/dev/vg_nvme/lv_1'}, Remote(name='ubuntu@smithi167.front.sepia.ceph.com'): {'osd.1': '/dev/vg_nvme/lv_1', 'osd.2': '/dev/vg_nvme/lv_2', 'osd.3': '/dev/vg_nvme/lv_3'}} 2023-10-05T19:27:25.691 DEBUG:teuthology.orchestra.run.smithi118:> sudo mkdir -p /var/lib/ceph/osd/ceph-0 2023-10-05T19:27:25.708 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1'} 2023-10-05T19:27:25.708 INFO:tasks.ceph:role: osd.0 2023-10-05T19:27:25.708 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_1 on ubuntu@smithi118.front.sepia.ceph.com 2023-10-05T19:27:25.708 DEBUG:teuthology.orchestra.run.smithi118:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_1 2023-10-05T19:27:25.796 INFO:teuthology.orchestra.run.smithi118.stdout:meta-data=/dev/vg_nvme/lv_1 isize=2048 agcount=4, agsize=5859072 blks 2023-10-05T19:27:25.796 INFO:teuthology.orchestra.run.smithi118.stdout: = sectsz=512 attr=2, projid32bit=1 2023-10-05T19:27:25.796 INFO:teuthology.orchestra.run.smithi118.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2023-10-05T19:27:25.797 INFO:teuthology.orchestra.run.smithi118.stdout: = reflink=1 2023-10-05T19:27:25.797 INFO:teuthology.orchestra.run.smithi118.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2023-10-05T19:27:25.797 INFO:teuthology.orchestra.run.smithi118.stdout: = sunit=0 swidth=0 blks 2023-10-05T19:27:25.797 INFO:teuthology.orchestra.run.smithi118.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2023-10-05T19:27:25.797 INFO:teuthology.orchestra.run.smithi118.stdout:log =internal log bsize=4096 blocks=11443, version=2 2023-10-05T19:27:25.797 INFO:teuthology.orchestra.run.smithi118.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2023-10-05T19:27:25.797 INFO:teuthology.orchestra.run.smithi118.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2023-10-05T19:27:25.798 INFO:tasks.ceph:mount /dev/vg_nvme/lv_1 on ubuntu@smithi118.front.sepia.ceph.com -o noatime 2023-10-05T19:27:25.798 DEBUG:teuthology.orchestra.run.smithi118:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_1 /var/lib/ceph/osd/ceph-0 2023-10-05T19:27:25.817 DEBUG:teuthology.orchestra.run.smithi118:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-0 2023-10-05T19:27:25.825 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-10-05T19:27:25.825 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: /sbin/restorecon: command not found 2023-10-05T19:27:25.825 DEBUG:teuthology.orchestra.run.smithi118:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 0 --monmap /home/ubuntu/cephtest/ceph.monmap 2023-10-05T19:27:25.894 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-05T19:27:25.887+0000 7fe4e0ab83c0 -1 auth: error reading file: /var/lib/ceph/osd/ceph-0/keyring: can't open /var/lib/ceph/osd/ceph-0/keyring: (2) No such file or directory 2023-10-05T19:27:25.894 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-05T19:27:25.887+0000 7fe4e0ab83c0 -1 created new key in keyring /var/lib/ceph/osd/ceph-0/keyring 2023-10-05T19:27:25.894 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-05T19:27:25.887+0000 7fe4e0ab83c0 -1 bluestore(/var/lib/ceph/osd/ceph-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-0/block: (2) No such file or directory 2023-10-05T19:27:25.894 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-05T19:27:25.887+0000 7fe4e0ab83c0 -1 bluestore(/var/lib/ceph/osd/ceph-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-0/block: (2) No such file or directory 2023-10-05T19:27:25.894 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-05T19:27:25.887+0000 7fe4e0ab83c0 -1 bluestore(/var/lib/ceph/osd/ceph-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-0/block: (2) No such file or directory 2023-10-05T19:27:25.895 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-05T19:27:25.887+0000 7fe4e0ab83c0 -1 bluestore(/var/lib/ceph/osd/ceph-0) _read_fsid unparsable uuid 2023-10-05T19:27:26.461 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-05T19:27:26.451+0000 7fe4e0ab83c0 -1 bluestore::NCB::__restore_allocator::Failed open_for_read with error-code -2 2023-10-05T19:27:27.237 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-05T19:27:27.231+0000 7fe4e0ab83c0 -1 bluestore::NCB::__restore_allocator::Failed open_for_read with error-code -2 2023-10-05T19:27:28.020 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-05T19:27:28.011+0000 7fe4e0ab83c0 -1 bluestore::NCB::__restore_allocator::Failed open_for_read with error-code -2 2023-10-05T19:27:28.894 DEBUG:teuthology.orchestra.run.smithi118:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-0 2023-10-05T19:27:28.907 DEBUG:teuthology.orchestra.run.smithi167:> sudo mkdir -p /var/lib/ceph/osd/ceph-1 2023-10-05T19:27:28.924 INFO:tasks.ceph:roles_to_devs: {'osd.1': '/dev/vg_nvme/lv_1', 'osd.2': '/dev/vg_nvme/lv_2', 'osd.3': '/dev/vg_nvme/lv_3'} 2023-10-05T19:27:28.924 INFO:tasks.ceph:role: osd.1 2023-10-05T19:27:28.924 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_1 on ubuntu@smithi167.front.sepia.ceph.com 2023-10-05T19:27:28.924 DEBUG:teuthology.orchestra.run.smithi167:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_1 2023-10-05T19:27:29.457 INFO:teuthology.orchestra.run.smithi167.stdout:meta-data=/dev/vg_nvme/lv_1 isize=2048 agcount=4, agsize=5859072 blks 2023-10-05T19:27:29.457 INFO:teuthology.orchestra.run.smithi167.stdout: = sectsz=512 attr=2, projid32bit=1 2023-10-05T19:27:29.457 INFO:teuthology.orchestra.run.smithi167.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2023-10-05T19:27:29.457 INFO:teuthology.orchestra.run.smithi167.stdout: = reflink=1 2023-10-05T19:27:29.458 INFO:teuthology.orchestra.run.smithi167.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2023-10-05T19:27:29.458 INFO:teuthology.orchestra.run.smithi167.stdout: = sunit=0 swidth=0 blks 2023-10-05T19:27:29.458 INFO:teuthology.orchestra.run.smithi167.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2023-10-05T19:27:29.458 INFO:teuthology.orchestra.run.smithi167.stdout:log =internal log bsize=4096 blocks=11443, version=2 2023-10-05T19:27:29.458 INFO:teuthology.orchestra.run.smithi167.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2023-10-05T19:27:29.458 INFO:teuthology.orchestra.run.smithi167.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2023-10-05T19:27:29.459 INFO:tasks.ceph:mount /dev/vg_nvme/lv_1 on ubuntu@smithi167.front.sepia.ceph.com -o noatime 2023-10-05T19:27:29.459 DEBUG:teuthology.orchestra.run.smithi167:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_1 /var/lib/ceph/osd/ceph-1 2023-10-05T19:27:29.562 DEBUG:teuthology.orchestra.run.smithi167:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-1 2023-10-05T19:27:29.574 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-10-05T19:27:29.575 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: /sbin/restorecon: command not found 2023-10-05T19:27:29.575 DEBUG:teuthology.orchestra.run.smithi167:> sudo mkdir -p /var/lib/ceph/osd/ceph-2 2023-10-05T19:27:29.585 INFO:tasks.ceph:roles_to_devs: {'osd.1': '/dev/vg_nvme/lv_1', 'osd.2': '/dev/vg_nvme/lv_2', 'osd.3': '/dev/vg_nvme/lv_3'} 2023-10-05T19:27:29.585 INFO:tasks.ceph:role: osd.2 2023-10-05T19:27:29.585 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_2 on ubuntu@smithi167.front.sepia.ceph.com 2023-10-05T19:27:29.585 DEBUG:teuthology.orchestra.run.smithi167:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_2 2023-10-05T19:27:30.134 INFO:teuthology.orchestra.run.smithi167.stdout:meta-data=/dev/vg_nvme/lv_2 isize=2048 agcount=4, agsize=5859072 blks 2023-10-05T19:27:30.134 INFO:teuthology.orchestra.run.smithi167.stdout: = sectsz=512 attr=2, projid32bit=1 2023-10-05T19:27:30.135 INFO:teuthology.orchestra.run.smithi167.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2023-10-05T19:27:30.135 INFO:teuthology.orchestra.run.smithi167.stdout: = reflink=1 2023-10-05T19:27:30.135 INFO:teuthology.orchestra.run.smithi167.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2023-10-05T19:27:30.135 INFO:teuthology.orchestra.run.smithi167.stdout: = sunit=0 swidth=0 blks 2023-10-05T19:27:30.135 INFO:teuthology.orchestra.run.smithi167.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2023-10-05T19:27:30.135 INFO:teuthology.orchestra.run.smithi167.stdout:log =internal log bsize=4096 blocks=11443, version=2 2023-10-05T19:27:30.135 INFO:teuthology.orchestra.run.smithi167.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2023-10-05T19:27:30.135 INFO:teuthology.orchestra.run.smithi167.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2023-10-05T19:27:30.136 INFO:tasks.ceph:mount /dev/vg_nvme/lv_2 on ubuntu@smithi167.front.sepia.ceph.com -o noatime 2023-10-05T19:27:30.136 DEBUG:teuthology.orchestra.run.smithi167:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_2 /var/lib/ceph/osd/ceph-2 2023-10-05T19:27:30.175 DEBUG:teuthology.orchestra.run.smithi167:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-2 2023-10-05T19:27:30.229 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: /sbin/restorecon: command not found 2023-10-05T19:27:30.230 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-10-05T19:27:30.230 DEBUG:teuthology.orchestra.run.smithi167:> sudo mkdir -p /var/lib/ceph/osd/ceph-3 2023-10-05T19:27:30.242 INFO:tasks.ceph:roles_to_devs: {'osd.1': '/dev/vg_nvme/lv_1', 'osd.2': '/dev/vg_nvme/lv_2', 'osd.3': '/dev/vg_nvme/lv_3'} 2023-10-05T19:27:30.242 INFO:tasks.ceph:role: osd.3 2023-10-05T19:27:30.243 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_3 on ubuntu@smithi167.front.sepia.ceph.com 2023-10-05T19:27:30.243 DEBUG:teuthology.orchestra.run.smithi167:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_3 2023-10-05T19:27:30.712 INFO:teuthology.orchestra.run.smithi167.stdout:meta-data=/dev/vg_nvme/lv_3 isize=2048 agcount=4, agsize=5859072 blks 2023-10-05T19:27:30.712 INFO:teuthology.orchestra.run.smithi167.stdout: = sectsz=512 attr=2, projid32bit=1 2023-10-05T19:27:30.712 INFO:teuthology.orchestra.run.smithi167.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2023-10-05T19:27:30.712 INFO:teuthology.orchestra.run.smithi167.stdout: = reflink=1 2023-10-05T19:27:30.712 INFO:teuthology.orchestra.run.smithi167.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2023-10-05T19:27:30.712 INFO:teuthology.orchestra.run.smithi167.stdout: = sunit=0 swidth=0 blks 2023-10-05T19:27:30.712 INFO:teuthology.orchestra.run.smithi167.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2023-10-05T19:27:30.712 INFO:teuthology.orchestra.run.smithi167.stdout:log =internal log bsize=4096 blocks=11443, version=2 2023-10-05T19:27:30.713 INFO:teuthology.orchestra.run.smithi167.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2023-10-05T19:27:30.713 INFO:teuthology.orchestra.run.smithi167.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2023-10-05T19:27:30.714 INFO:tasks.ceph:mount /dev/vg_nvme/lv_3 on ubuntu@smithi167.front.sepia.ceph.com -o noatime 2023-10-05T19:27:30.714 DEBUG:teuthology.orchestra.run.smithi167:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_3 /var/lib/ceph/osd/ceph-3 2023-10-05T19:27:30.778 DEBUG:teuthology.orchestra.run.smithi167:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-3 2023-10-05T19:27:30.788 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-10-05T19:27:30.788 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: /sbin/restorecon: command not found 2023-10-05T19:27:30.788 DEBUG:teuthology.orchestra.run.smithi167:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 1 --monmap /home/ubuntu/cephtest/ceph.monmap 2023-10-05T19:27:30.875 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:27:30.867+0000 7f7c5f9953c0 -1 auth: error reading file: /var/lib/ceph/osd/ceph-1/keyring: can't open /var/lib/ceph/osd/ceph-1/keyring: (2) No such file or directory 2023-10-05T19:27:30.879 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:27:30.871+0000 7f7c5f9953c0 -1 created new key in keyring /var/lib/ceph/osd/ceph-1/keyring 2023-10-05T19:27:30.879 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:27:30.871+0000 7f7c5f9953c0 -1 bluestore(/var/lib/ceph/osd/ceph-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-1/block: (2) No such file or directory 2023-10-05T19:27:30.879 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:27:30.871+0000 7f7c5f9953c0 -1 bluestore(/var/lib/ceph/osd/ceph-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-1/block: (2) No such file or directory 2023-10-05T19:27:30.879 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:27:30.871+0000 7f7c5f9953c0 -1 bluestore(/var/lib/ceph/osd/ceph-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-1/block: (2) No such file or directory 2023-10-05T19:27:30.881 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:27:30.871+0000 7f7c5f9953c0 -1 bluestore(/var/lib/ceph/osd/ceph-1) _read_fsid unparsable uuid 2023-10-05T19:27:31.458 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:27:31.451+0000 7f7c5f9953c0 -1 bluestore::NCB::__restore_allocator::Failed open_for_read with error-code -2 2023-10-05T19:27:32.237 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:27:32.227+0000 7f7c5f9953c0 -1 bluestore::NCB::__restore_allocator::Failed open_for_read with error-code -2 2023-10-05T19:27:33.013 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:27:33.003+0000 7f7c5f9953c0 -1 bluestore::NCB::__restore_allocator::Failed open_for_read with error-code -2 2023-10-05T19:27:33.871 DEBUG:teuthology.orchestra.run.smithi167:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-1 2023-10-05T19:27:33.886 DEBUG:teuthology.orchestra.run.smithi167:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 2 --monmap /home/ubuntu/cephtest/ceph.monmap 2023-10-05T19:27:33.926 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:27:33.915+0000 7efe531183c0 -1 auth: error reading file: /var/lib/ceph/osd/ceph-2/keyring: can't open /var/lib/ceph/osd/ceph-2/keyring: (2) No such file or directory 2023-10-05T19:27:33.926 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:27:33.919+0000 7efe531183c0 -1 created new key in keyring /var/lib/ceph/osd/ceph-2/keyring 2023-10-05T19:27:33.926 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:27:33.919+0000 7efe531183c0 -1 bluestore(/var/lib/ceph/osd/ceph-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-2/block: (2) No such file or directory 2023-10-05T19:27:33.926 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:27:33.919+0000 7efe531183c0 -1 bluestore(/var/lib/ceph/osd/ceph-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-2/block: (2) No such file or directory 2023-10-05T19:27:33.926 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:27:33.919+0000 7efe531183c0 -1 bluestore(/var/lib/ceph/osd/ceph-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-2/block: (2) No such file or directory 2023-10-05T19:27:33.927 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:27:33.919+0000 7efe531183c0 -1 bluestore(/var/lib/ceph/osd/ceph-2) _read_fsid unparsable uuid 2023-10-05T19:27:34.500 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:27:34.491+0000 7efe531183c0 -1 bluestore::NCB::__restore_allocator::Failed open_for_read with error-code -2 2023-10-05T19:27:35.287 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:27:35.279+0000 7efe531183c0 -1 bluestore::NCB::__restore_allocator::Failed open_for_read with error-code -2 2023-10-05T19:27:36.062 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:27:36.055+0000 7efe531183c0 -1 bluestore::NCB::__restore_allocator::Failed open_for_read with error-code -2 2023-10-05T19:27:36.919 DEBUG:teuthology.orchestra.run.smithi167:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-2 2023-10-05T19:27:36.932 DEBUG:teuthology.orchestra.run.smithi167:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 3 --monmap /home/ubuntu/cephtest/ceph.monmap 2023-10-05T19:27:37.014 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:27:37.007+0000 7f38243323c0 -1 auth: error reading file: /var/lib/ceph/osd/ceph-3/keyring: can't open /var/lib/ceph/osd/ceph-3/keyring: (2) No such file or directory 2023-10-05T19:27:37.014 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:27:37.007+0000 7f38243323c0 -1 created new key in keyring /var/lib/ceph/osd/ceph-3/keyring 2023-10-05T19:27:37.014 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:27:37.007+0000 7f38243323c0 -1 bluestore(/var/lib/ceph/osd/ceph-3/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-3/block: (2) No such file or directory 2023-10-05T19:27:37.015 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:27:37.007+0000 7f38243323c0 -1 bluestore(/var/lib/ceph/osd/ceph-3/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-3/block: (2) No such file or directory 2023-10-05T19:27:37.015 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:27:37.007+0000 7f38243323c0 -1 bluestore(/var/lib/ceph/osd/ceph-3/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-3/block: (2) No such file or directory 2023-10-05T19:27:37.015 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:27:37.007+0000 7f38243323c0 -1 bluestore(/var/lib/ceph/osd/ceph-3) _read_fsid unparsable uuid 2023-10-05T19:27:37.590 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:27:37.583+0000 7f38243323c0 -1 bluestore::NCB::__restore_allocator::Failed open_for_read with error-code -2 2023-10-05T19:27:38.375 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:27:38.367+0000 7f38243323c0 -1 bluestore::NCB::__restore_allocator::Failed open_for_read with error-code -2 2023-10-05T19:27:39.153 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:27:39.143+0000 7f38243323c0 -1 bluestore::NCB::__restore_allocator::Failed open_for_read with error-code -2 2023-10-05T19:27:40.014 DEBUG:teuthology.orchestra.run.smithi167:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-3 2023-10-05T19:27:40.028 INFO:tasks.ceph:Reading keys from all nodes... 2023-10-05T19:27:40.028 DEBUG:teuthology.orchestra.run.smithi118:> set -ex 2023-10-05T19:27:40.028 DEBUG:teuthology.orchestra.run.smithi118:> sudo dd if=/var/lib/ceph/osd/ceph-0/keyring of=/dev/stdout 2023-10-05T19:27:40.044 DEBUG:teuthology.orchestra.run.smithi167:> set -ex 2023-10-05T19:27:40.044 DEBUG:teuthology.orchestra.run.smithi167:> sudo dd if=/var/lib/ceph/mgr/ceph-x/keyring of=/dev/stdout 2023-10-05T19:27:40.059 DEBUG:teuthology.orchestra.run.smithi167:> set -ex 2023-10-05T19:27:40.059 DEBUG:teuthology.orchestra.run.smithi167:> sudo dd if=/var/lib/ceph/osd/ceph-1/keyring of=/dev/stdout 2023-10-05T19:27:40.115 DEBUG:teuthology.orchestra.run.smithi167:> set -ex 2023-10-05T19:27:40.115 DEBUG:teuthology.orchestra.run.smithi167:> sudo dd if=/var/lib/ceph/osd/ceph-2/keyring of=/dev/stdout 2023-10-05T19:27:40.128 DEBUG:teuthology.orchestra.run.smithi167:> set -ex 2023-10-05T19:27:40.129 DEBUG:teuthology.orchestra.run.smithi167:> sudo dd if=/var/lib/ceph/osd/ceph-3/keyring of=/dev/stdout 2023-10-05T19:27:40.141 DEBUG:teuthology.orchestra.run.smithi167:> set -ex 2023-10-05T19:27:40.141 DEBUG:teuthology.orchestra.run.smithi167:> dd if=/etc/ceph/ceph.client.0.keyring of=/dev/stdout 2023-10-05T19:27:40.145 INFO:tasks.ceph:Adding keys to all mons... 2023-10-05T19:27:40.146 DEBUG:teuthology.orchestra.run.smithi118:> sudo tee -a /etc/ceph/ceph.keyring 2023-10-05T19:27:40.160 INFO:teuthology.orchestra.run.smithi118.stdout:[osd.0] 2023-10-05T19:27:40.160 INFO:teuthology.orchestra.run.smithi118.stdout: key = AQAdDh9lCM4mNRAAZz8/YwgpMBFfIfdmtnfYtA== 2023-10-05T19:27:40.161 INFO:teuthology.orchestra.run.smithi118.stdout:[mgr.x] 2023-10-05T19:27:40.161 INFO:teuthology.orchestra.run.smithi118.stdout: key = AQAdDh9lVpljIxAAq1RqW2c27G6r3JQtEN+RYw== 2023-10-05T19:27:40.161 INFO:teuthology.orchestra.run.smithi118.stdout:[osd.1] 2023-10-05T19:27:40.161 INFO:teuthology.orchestra.run.smithi118.stdout: key = AQAiDh9lRFkNNBAAu/vSXoK5Zjevn0QAyl374g== 2023-10-05T19:27:40.161 INFO:teuthology.orchestra.run.smithi118.stdout:[osd.2] 2023-10-05T19:27:40.161 INFO:teuthology.orchestra.run.smithi118.stdout: key = AQAlDh9lqjoHNxAAsPknk8d6G2IWD3dprgHdRg== 2023-10-05T19:27:40.161 INFO:teuthology.orchestra.run.smithi118.stdout:[osd.3] 2023-10-05T19:27:40.161 INFO:teuthology.orchestra.run.smithi118.stdout: key = AQApDh9lh0KyABAAud3/xkJps5Z3Rm5TD53lLg== 2023-10-05T19:27:40.161 INFO:teuthology.orchestra.run.smithi118.stdout:[client.0] 2023-10-05T19:27:40.161 INFO:teuthology.orchestra.run.smithi118.stdout: key = AQAdDh9l3a5KKBAAHBUQNBMsL1Ovsm6NBR+XZA== 2023-10-05T19:27:40.162 DEBUG:teuthology.orchestra.run.smithi118:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.0 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-10-05T19:27:40.250 DEBUG:teuthology.orchestra.run.smithi118:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.x --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2023-10-05T19:27:40.293 DEBUG:teuthology.orchestra.run.smithi118:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.1 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-10-05T19:27:40.337 DEBUG:teuthology.orchestra.run.smithi118:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.2 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-10-05T19:27:40.374 DEBUG:teuthology.orchestra.run.smithi118:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.3 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-10-05T19:27:40.415 DEBUG:teuthology.orchestra.run.smithi118:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.0 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2023-10-05T19:27:40.458 INFO:tasks.ceph:Running mkfs on mon nodes... 2023-10-05T19:27:40.459 DEBUG:teuthology.orchestra.run.smithi118:> sudo mkdir -p /var/lib/ceph/mon/ceph-a 2023-10-05T19:27:40.515 DEBUG:teuthology.orchestra.run.smithi118:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i a --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2023-10-05T19:27:40.568 DEBUG:teuthology.orchestra.run.smithi118:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-a 2023-10-05T19:27:40.580 DEBUG:teuthology.orchestra.run.smithi118:> rm -- /home/ubuntu/cephtest/ceph.monmap 2023-10-05T19:27:40.630 INFO:tasks.ceph:Starting mon daemons in cluster ceph... 2023-10-05T19:27:40.630 INFO:tasks.ceph.mon.a:Restarting daemon 2023-10-05T19:27:40.630 DEBUG:teuthology.orchestra.run.smithi118:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i a 2023-10-05T19:27:40.632 INFO:tasks.ceph.mon.a:Started 2023-10-05T19:27:40.633 INFO:tasks.ceph:Starting mgr daemons in cluster ceph... 2023-10-05T19:27:40.634 INFO:tasks.ceph.mgr.x:Restarting daemon 2023-10-05T19:27:40.634 DEBUG:teuthology.orchestra.run.smithi167:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mgr -f --cluster ceph -i x 2023-10-05T19:27:40.636 INFO:tasks.ceph.mgr.x:Started 2023-10-05T19:27:40.636 INFO:tasks.ceph:Setting crush tunables to default 2023-10-05T19:27:40.636 DEBUG:teuthology.orchestra.run.smithi118:> sudo ceph --cluster ceph osd crush tunables default 2023-10-05T19:27:40.734 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-05T19:27:40.727+0000 7f7c378b5700 1 Processor -- start 2023-10-05T19:27:40.734 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-05T19:27:40.727+0000 7f7c378b5700 1 -- start start 2023-10-05T19:27:40.735 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-05T19:27:40.727+0000 7f7c378b5700 1 --2- >> v2:172.21.15.118:3300/0 conn(0x7f7c3006a760 0x7f7c3006ab30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-05T19:27:40.736 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-05T19:27:40.727+0000 7f7c378b5700 1 -- --> v1:172.21.15.118:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7c30096b40 con 0x7f7c3006b070 2023-10-05T19:27:40.736 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-05T19:27:40.727+0000 7f7c378b5700 1 -- --> v2:172.21.15.118:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7c3006ed00 con 0x7f7c3006a760 2023-10-05T19:27:40.736 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-05T19:27:40.727+0000 7f7c35651700 1 --2- >> v2:172.21.15.118:3300/0 conn(0x7f7c3006a760 0x7f7c3006ab30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-05T19:27:40.736 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-05T19:27:40.727+0000 7f7c34e50700 1 --1- >> v1:172.21.15.118:6789/0 conn(0x7f7c3006b070 0x7f7c3011fcf0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.118:6789/0 says I am v1:172.21.15.118:43100/0 (socket says 172.21.15.118:43100) 2023-10-05T19:27:40.736 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-05T19:27:40.727+0000 7f7c34e50700 1 -- 172.21.15.118:0/1809912960 learned_addr learned my addr 172.21.15.118:0/1809912960 (peer_addr_for_me v1:172.21.15.118:0/0) 2023-10-05T19:27:40.736 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-05T19:27:40.727+0000 7f7c35651700 1 -- 172.21.15.118:0/1809912960 >> v2:172.21.15.118:3300/0 conn(0x7f7c3006a760 msgr2=0x7f7c3006ab30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2023-10-05T19:27:40.736 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-05T19:27:40.727+0000 7f7c35651700 1 -- 172.21.15.118:0/1809912960 >> v2:172.21.15.118:3300/0 conn(0x7f7c3006a760 msgr2=0x7f7c3006ab30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2023-10-05T19:27:40.736 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-05T19:27:40.727+0000 7f7c35651700 1 --2- 172.21.15.118:0/1809912960 >> v2:172.21.15.118:3300/0 conn(0x7f7c3006a760 0x7f7c3006ab30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2023-10-05T19:27:40.736 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-05T19:27:40.727+0000 7f7c35651700 1 --2- 172.21.15.118:0/1809912960 >> v2:172.21.15.118:3300/0 conn(0x7f7c3006a760 0x7f7c3006ab30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-05T19:27:40.776 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-05T19:27:40.767+0000 7f7c27fff700 1 -- 172.21.15.118:0/1809912960 <== mon.0 v1:172.21.15.118:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2081413806 0 0) 0x7f7c30096b40 con 0x7f7c3006b070 2023-10-05T19:27:40.777 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-05T19:27:40.767+0000 7f7c27fff700 1 -- 172.21.15.118:0/1809912960 --> v1:172.21.15.118:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7c1c003570 con 0x7f7c3006b070 2023-10-05T19:27:40.777 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-05T19:27:40.767+0000 7f7c27fff700 1 -- 172.21.15.118:0/1809912960 <== mon.0 v1:172.21.15.118:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1467887936 0 0) 0x7f7c1c003570 con 0x7f7c3006b070 2023-10-05T19:27:40.777 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-05T19:27:40.767+0000 7f7c27fff700 1 -- 172.21.15.118:0/1809912960 >> v2:172.21.15.118:3300/0 conn(0x7f7c3006a760 msgr2=0x7f7c3006ab30 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-10-05T19:27:40.777 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-05T19:27:40.767+0000 7f7c27fff700 1 --2- 172.21.15.118:0/1809912960 >> v2:172.21.15.118:3300/0 conn(0x7f7c3006a760 0x7f7c3006ab30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-05T19:27:40.777 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-05T19:27:40.767+0000 7f7c27fff700 1 -- 172.21.15.118:0/1809912960 --> v1:172.21.15.118:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7c3006c990 con 0x7f7c3006b070 2023-10-05T19:27:40.777 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-05T19:27:40.771+0000 7f7c27fff700 1 -- 172.21.15.118:0/1809912960 <== mon.0 v1:172.21.15.118:6789/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (unknown 2681859047 0 0) 0x7f7c2c0015b0 con 0x7f7c3006b070 2023-10-05T19:27:40.777 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-05T19:27:40.771+0000 7f7c27fff700 1 -- 172.21.15.118:0/1809912960 >> v1:172.21.15.118:6789/0 conn(0x7f7c3006b070 legacy=0x7f7c3011fcf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-05T19:27:40.777 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-05T19:27:40.771+0000 7f7c27fff700 1 --2- 172.21.15.118:0/1809912960 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f7c1c003a90 0x7f7c1c003e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-05T19:27:40.777 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-05T19:27:40.771+0000 7f7c27fff700 1 -- 172.21.15.118:0/1809912960 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7c3006ed00 con 0x7f7c1c003a90 2023-10-05T19:27:40.777 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-05T19:27:40.771+0000 7f7c35651700 1 --2- 172.21.15.118:0/1809912960 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f7c1c003a90 0x7f7c1c003e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-05T19:27:40.778 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-05T19:27:40.771+0000 7f7c35651700 1 -- 172.21.15.118:0/1809912960 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7c1c003570 con 0x7f7c1c003a90 2023-10-05T19:27:40.778 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-05T19:27:40.771+0000 7f7c35651700 1 --2- 172.21.15.118:0/1809912960 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f7c1c003a90 0x7f7c1c003e80 secure :-1 s=READY pgs=3 cs=0 l=1 rev1=1 crypto rx=0x7f7c2000ba40 tx=0x7f7c20008040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=41d31b480dfd04ee server_cookie=0 in_seq=0 out_seq=0 2023-10-05T19:27:40.779 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-05T19:27:40.771+0000 7f7c27fff700 1 -- 172.21.15.118:0/1809912960 <== mon.0 v2:172.21.15.118:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f7c20018070 con 0x7f7c1c003a90 2023-10-05T19:27:40.779 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-05T19:27:40.771+0000 7f7c27fff700 1 -- 172.21.15.118:0/1809912960 <== mon.0 v2:172.21.15.118:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7c2000d5d0 con 0x7f7c1c003a90 2023-10-05T19:27:40.779 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-05T19:27:40.771+0000 7f7c27fff700 1 -- 172.21.15.118:0/1809912960 <== mon.0 v2:172.21.15.118:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f7c2000b4b0 con 0x7f7c1c003a90 2023-10-05T19:27:40.780 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-05T19:27:40.771+0000 7f7c378b5700 1 -- 172.21.15.118:0/1809912960 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f7c1c003a90 msgr2=0x7f7c1c003e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-05T19:27:40.780 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-05T19:27:40.771+0000 7f7c378b5700 1 --2- 172.21.15.118:0/1809912960 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f7c1c003a90 0x7f7c1c003e80 secure :-1 s=READY pgs=3 cs=0 l=1 rev1=1 crypto rx=0x7f7c2000ba40 tx=0x7f7c20008040 comp rx=0 tx=0).stop 2023-10-05T19:27:40.780 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-05T19:27:40.771+0000 7f7c378b5700 1 -- 172.21.15.118:0/1809912960 shutdown_connections 2023-10-05T19:27:40.780 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-05T19:27:40.771+0000 7f7c378b5700 1 --2- 172.21.15.118:0/1809912960 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f7c1c003a90 0x7f7c1c003e80 unknown :-1 s=CLOSED pgs=3 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-05T19:27:40.780 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-05T19:27:40.771+0000 7f7c378b5700 1 --2- 172.21.15.118:0/1809912960 >> v2:172.21.15.118:3300/0 conn(0x7f7c3006a760 0x7f7c3006ab30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-05T19:27:40.780 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-05T19:27:40.771+0000 7f7c378b5700 1 -- 172.21.15.118:0/1809912960 >> 172.21.15.118:0/1809912960 conn(0x7f7c3009a0a0 msgr2=0x7f7c300949f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-05T19:27:40.780 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-05T19:27:40.771+0000 7f7c378b5700 1 -- 172.21.15.118:0/1809912960 shutdown_connections 2023-10-05T19:27:40.780 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-05T19:27:40.771+0000 7f7c378b5700 1 -- 172.21.15.118:0/1809912960 wait complete. 2023-10-05T19:27:40.781 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-05T19:27:40.775+0000 7f7c378b5700 1 Processor -- start 2023-10-05T19:27:40.781 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-05T19:27:40.775+0000 7f7c378b5700 1 -- start start 2023-10-05T19:27:40.782 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-05T19:27:40.775+0000 7f7c378b5700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f7c1c003a90 0x7f7c30116a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-05T19:27:40.782 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-05T19:27:40.775+0000 7f7c378b5700 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7c30118470 con 0x7f7c1c003a90 2023-10-05T19:27:40.782 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-05T19:27:40.775+0000 7f7c35651700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f7c1c003a90 0x7f7c30116a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-05T19:27:40.782 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-05T19:27:40.775+0000 7f7c35651700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f7c1c003a90 0x7f7c30116a80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.118:3300/0 says I am v2:172.21.15.118:56504/0 (socket says 172.21.15.118:56504) 2023-10-05T19:27:40.783 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-05T19:27:40.775+0000 7f7c35651700 1 -- 172.21.15.118:0/2140029799 learned_addr learned my addr 172.21.15.118:0/2140029799 (peer_addr_for_me v2:172.21.15.118:0/0) 2023-10-05T19:27:40.783 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-05T19:27:40.775+0000 7f7c35651700 1 -- 172.21.15.118:0/2140029799 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7c30118dd0 con 0x7f7c1c003a90 2023-10-05T19:27:40.784 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-05T19:27:40.775+0000 7f7c35651700 1 --2- 172.21.15.118:0/2140029799 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f7c1c003a90 0x7f7c30116a80 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0x7f7c20015000 tx=0x7f7c2001a000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-05T19:27:40.784 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-05T19:27:40.775+0000 7f7c257fa700 1 -- 172.21.15.118:0/2140029799 <== mon.0 v2:172.21.15.118:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f7c200147e0 con 0x7f7c1c003a90 2023-10-05T19:27:40.784 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-05T19:27:40.775+0000 7f7c257fa700 1 -- 172.21.15.118:0/2140029799 <== mon.0 v2:172.21.15.118:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7c20014960 con 0x7f7c1c003a90 2023-10-05T19:27:40.784 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-05T19:27:40.775+0000 7f7c378b5700 1 -- 172.21.15.118:0/2140029799 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7c30120e20 con 0x7f7c1c003a90 2023-10-05T19:27:40.784 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-05T19:27:40.775+0000 7f7c257fa700 1 -- 172.21.15.118:0/2140029799 <== mon.0 v2:172.21.15.118:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f7c20014c50 con 0x7f7c1c003a90 2023-10-05T19:27:40.784 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-05T19:27:40.775+0000 7f7c378b5700 1 -- 172.21.15.118:0/2140029799 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7c30116fc0 con 0x7f7c1c003a90 2023-10-05T19:27:40.785 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-05T19:27:40.779+0000 7f7c257fa700 1 -- 172.21.15.118:0/2140029799 <== mon.0 v2:172.21.15.118:3300/0 4 ==== mgrmap(e 1) v1 ==== 521+0+0 (secure 0 0 0) 0x7f7c200286e0 con 0x7f7c1c003a90 2023-10-05T19:27:40.785 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-05T19:27:40.779+0000 7f7c257fa700 1 -- 172.21.15.118:0/2140029799 <== mon.0 v2:172.21.15.118:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 721+0+0 (secure 0 0 0) 0x7f7c2001f780 con 0x7f7c1c003a90 2023-10-05T19:27:40.785 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-05T19:27:40.779+0000 7f7c378b5700 1 -- 172.21.15.118:0/2140029799 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7c301238e0 con 0x7f7c1c003a90 2023-10-05T19:27:40.791 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-05T19:27:40.783+0000 7f7c257fa700 1 -- 172.21.15.118:0/2140029799 <== mon.0 v2:172.21.15.118:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+71325 (secure 0 0 0) 0x7f7c200476b0 con 0x7f7c1c003a90 2023-10-05T19:27:40.865 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-05T19:27:40.859+0000 7f7c378b5700 1 -- 172.21.15.118:0/2140029799 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_command({"prefix": "osd crush tunables", "profile": "default"} v 0) v1 -- 0x7f7c3005ecb0 con 0x7f7c1c003a90 2023-10-05T19:27:40.870 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-05T19:27:40.863+0000 7f7c257fa700 1 -- 172.21.15.118:0/2140029799 <== mon.0 v2:172.21.15.118:3300/0 7 ==== mon_command_ack([{"prefix": "osd crush tunables", "profile": "default"}]=0 adjusted tunables profile to default v2) v1 ==== 124+0+0 (secure 0 0 0) 0x7f7c20036020 con 0x7f7c1c003a90 2023-10-05T19:27:40.872 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-05T19:27:40.863+0000 7f7c378b5700 1 -- 172.21.15.118:0/2140029799 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f7c1c003a90 msgr2=0x7f7c30116a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-05T19:27:40.872 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-05T19:27:40.863+0000 7f7c378b5700 1 --2- 172.21.15.118:0/2140029799 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f7c1c003a90 0x7f7c30116a80 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0x7f7c20015000 tx=0x7f7c2001a000 comp rx=0 tx=0).stop 2023-10-05T19:27:40.872 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-05T19:27:40.863+0000 7f7c378b5700 1 -- 172.21.15.118:0/2140029799 shutdown_connections 2023-10-05T19:27:40.872 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-05T19:27:40.863+0000 7f7c378b5700 1 --2- 172.21.15.118:0/2140029799 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f7c1c003a90 0x7f7c30116a80 unknown :-1 s=CLOSED pgs=5 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-05T19:27:40.872 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-05T19:27:40.863+0000 7f7c378b5700 1 -- 172.21.15.118:0/2140029799 >> 172.21.15.118:0/2140029799 conn(0x7f7c3009a0a0 msgr2=0x7f7c3010fbf0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-05T19:27:40.872 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-05T19:27:40.863+0000 7f7c378b5700 1 -- 172.21.15.118:0/2140029799 shutdown_connections 2023-10-05T19:27:40.872 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-05T19:27:40.863+0000 7f7c378b5700 1 -- 172.21.15.118:0/2140029799 wait complete. 2023-10-05T19:27:40.873 INFO:teuthology.orchestra.run.smithi118.stderr:adjusted tunables profile to default 2023-10-05T19:27:40.884 INFO:tasks.ceph:Starting osd daemons in cluster ceph... 2023-10-05T19:27:40.885 DEBUG:teuthology.orchestra.run.smithi118:> set -ex 2023-10-05T19:27:40.885 DEBUG:teuthology.orchestra.run.smithi118:> sudo dd if=/var/lib/ceph/osd/ceph-0/fsid of=/dev/stdout 2023-10-05T19:27:40.900 DEBUG:teuthology.orchestra.run.smithi167:> set -ex 2023-10-05T19:27:40.900 DEBUG:teuthology.orchestra.run.smithi167:> sudo dd if=/var/lib/ceph/osd/ceph-1/fsid of=/dev/stdout 2023-10-05T19:27:40.916 DEBUG:teuthology.orchestra.run.smithi167:> set -ex 2023-10-05T19:27:40.916 DEBUG:teuthology.orchestra.run.smithi167:> sudo dd if=/var/lib/ceph/osd/ceph-2/fsid of=/dev/stdout 2023-10-05T19:27:40.925 DEBUG:teuthology.orchestra.run.smithi167:> set -ex 2023-10-05T19:27:40.925 DEBUG:teuthology.orchestra.run.smithi167:> sudo dd if=/var/lib/ceph/osd/ceph-3/fsid of=/dev/stdout 2023-10-05T19:27:40.935 DEBUG:teuthology.orchestra.run.smithi167:> sudo ceph --cluster ceph osd new 61ba9225-834e-4288-b391-7ffa540f7334 0 2023-10-05T19:27:40.994 INFO:tasks.ceph.mgr.x.smithi167.stderr:2023-10-05T19:27:40.991+0000 7fb91b495000 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2023-10-05T19:27:41.084 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:27:41.079+0000 7fb7360d7700 1 Processor -- start 2023-10-05T19:27:41.085 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:27:41.079+0000 7fb7360d7700 1 -- start start 2023-10-05T19:27:41.085 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:27:41.079+0000 7fb7360d7700 1 --2- >> v2:172.21.15.118:3300/0 conn(0x7fb73010b900 0x7fb7300678b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-05T19:27:41.085 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:27:41.079+0000 7fb7360d7700 1 -- --> v1:172.21.15.118:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb730068270 con 0x7fb730067df0 2023-10-05T19:27:41.085 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:27:41.079+0000 7fb7360d7700 1 -- --> v2:172.21.15.118:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb73004c3e0 con 0x7fb73010b900 2023-10-05T19:27:41.085 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:27:41.079+0000 7fb72f7fe700 1 --2- >> v2:172.21.15.118:3300/0 conn(0x7fb73010b900 0x7fb7300678b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-05T19:27:41.085 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:27:41.079+0000 7fb72effd700 1 --1- >> v1:172.21.15.118:6789/0 conn(0x7fb730067df0 0x7fb7301173b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.118:6789/0 says I am v1:172.21.15.167:54506/0 (socket says 172.21.15.167:54506) 2023-10-05T19:27:41.085 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:27:41.079+0000 7fb72f7fe700 1 --2- >> v2:172.21.15.118:3300/0 conn(0x7fb73010b900 0x7fb7300678b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.118:3300/0 says I am v2:172.21.15.167:41216/0 (socket says 172.21.15.167:41216) 2023-10-05T19:27:41.085 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:27:41.079+0000 7fb72effd700 1 -- 172.21.15.167:0/3471361562 learned_addr learned my addr 172.21.15.167:0/3471361562 (peer_addr_for_me v1:172.21.15.167:0/0) 2023-10-05T19:27:41.087 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:27:41.079+0000 7fb72f7fe700 1 -- 172.21.15.167:0/3471361562 >> v1:172.21.15.118:6789/0 conn(0x7fb730067df0 legacy=0x7fb7301173b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-05T19:27:41.087 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:27:41.079+0000 7fb72f7fe700 1 -- 172.21.15.167:0/3471361562 --> v2:172.21.15.118:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb730068400 con 0x7fb73010b900 2023-10-05T19:27:41.087 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:27:41.079+0000 7fb72f7fe700 1 --2- 172.21.15.167:0/3471361562 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb73010b900 0x7fb7300678b0 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7fb720009860 tx=0x7fb7200091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=80f44b3c491fc329 server_cookie=0 in_seq=0 out_seq=0 2023-10-05T19:27:41.087 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:27:41.083+0000 7fb72e7fc700 1 -- 172.21.15.167:0/3471361562 <== mon.0 v2:172.21.15.118:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fb72000b680 con 0x7fb73010b900 2023-10-05T19:27:41.087 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:27:41.083+0000 7fb72e7fc700 1 -- 172.21.15.167:0/3471361562 <== mon.0 v2:172.21.15.118:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb72000b7e0 con 0x7fb73010b900 2023-10-05T19:27:41.087 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:27:41.083+0000 7fb72e7fc700 1 -- 172.21.15.167:0/3471361562 <== mon.0 v2:172.21.15.118:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fb72000b990 con 0x7fb73010b900 2023-10-05T19:27:41.087 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:27:41.083+0000 7fb7360d7700 1 -- 172.21.15.167:0/3471361562 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb73010b900 msgr2=0x7fb7300678b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-05T19:27:41.088 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:27:41.083+0000 7fb7360d7700 1 --2- 172.21.15.167:0/3471361562 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb73010b900 0x7fb7300678b0 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7fb720009860 tx=0x7fb7200091a0 comp rx=0 tx=0).stop 2023-10-05T19:27:41.088 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:27:41.083+0000 7fb7360d7700 1 -- 172.21.15.167:0/3471361562 shutdown_connections 2023-10-05T19:27:41.088 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:27:41.083+0000 7fb7360d7700 1 --2- 172.21.15.167:0/3471361562 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb73010b900 0x7fb7300678b0 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-05T19:27:41.088 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:27:41.083+0000 7fb7360d7700 1 -- 172.21.15.167:0/3471361562 >> 172.21.15.167:0/3471361562 conn(0x7fb73006b160 msgr2=0x7fb73006d570 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-05T19:27:41.088 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:27:41.083+0000 7fb7360d7700 1 -- 172.21.15.167:0/3471361562 shutdown_connections 2023-10-05T19:27:41.088 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:27:41.083+0000 7fb7360d7700 1 -- 172.21.15.167:0/3471361562 wait complete. 2023-10-05T19:27:41.090 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:27:41.083+0000 7fb7360d7700 1 Processor -- start 2023-10-05T19:27:41.090 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:27:41.083+0000 7fb7360d7700 1 -- start start 2023-10-05T19:27:41.090 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:27:41.083+0000 7fb7360d7700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb730067df0 0x7fb730121a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-05T19:27:41.090 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:27:41.083+0000 7fb7360d7700 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb73006da10 con 0x7fb730067df0 2023-10-05T19:27:41.090 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:27:41.083+0000 7fb72f7fe700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb730067df0 0x7fb730121a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-05T19:27:41.090 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:27:41.083+0000 7fb72f7fe700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb730067df0 0x7fb730121a70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.118:3300/0 says I am v2:172.21.15.167:41218/0 (socket says 172.21.15.167:41218) 2023-10-05T19:27:41.091 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:27:41.083+0000 7fb72f7fe700 1 -- 172.21.15.167:0/523229766 learned_addr learned my addr 172.21.15.167:0/523229766 (peer_addr_for_me v2:172.21.15.167:0/0) 2023-10-05T19:27:41.091 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:27:41.083+0000 7fb72f7fe700 1 -- 172.21.15.167:0/523229766 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb720009420 con 0x7fb730067df0 2023-10-05T19:27:41.091 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:27:41.083+0000 7fb72f7fe700 1 --2- 172.21.15.167:0/523229766 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb730067df0 0x7fb730121a70 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7fb720000c00 tx=0x7fb720016000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-05T19:27:41.091 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:27:41.083+0000 7fb7177fe700 1 -- 172.21.15.167:0/523229766 <== mon.0 v2:172.21.15.118:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fb720025470 con 0x7fb730067df0 2023-10-05T19:27:41.091 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:27:41.083+0000 7fb7177fe700 1 -- 172.21.15.167:0/523229766 <== mon.0 v2:172.21.15.118:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb7200255d0 con 0x7fb730067df0 2023-10-05T19:27:41.091 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:27:41.083+0000 7fb7177fe700 1 -- 172.21.15.167:0/523229766 <== mon.0 v2:172.21.15.118:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fb720025880 con 0x7fb730067df0 2023-10-05T19:27:41.091 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:27:41.083+0000 7fb7360d7700 1 -- 172.21.15.167:0/523229766 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb730122100 con 0x7fb730067df0 2023-10-05T19:27:41.091 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:27:41.087+0000 7fb7360d7700 1 -- 172.21.15.167:0/523229766 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb7301224f0 con 0x7fb730067df0 2023-10-05T19:27:41.091 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:27:41.087+0000 7fb7177fe700 1 -- 172.21.15.167:0/523229766 <== mon.0 v2:172.21.15.118:3300/0 4 ==== mgrmap(e 1) v1 ==== 521+0+0 (secure 0 0 0) 0x7fb720022070 con 0x7fb730067df0 2023-10-05T19:27:41.091 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:27:41.087+0000 7fb7177fe700 1 -- 172.21.15.167:0/523229766 <== mon.0 v2:172.21.15.118:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 721+0+0 (secure 0 0 0) 0x7fb72001b680 con 0x7fb730067df0 2023-10-05T19:27:41.092 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:27:41.087+0000 7fb7360d7700 1 -- 172.21.15.167:0/523229766 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb73005ecb0 con 0x7fb730067df0 2023-10-05T19:27:41.096 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:27:41.091+0000 7fb7177fe700 1 -- 172.21.15.167:0/523229766 <== mon.0 v2:172.21.15.118:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+71325 (secure 0 0 0) 0x7fb7200446b0 con 0x7fb730067df0 2023-10-05T19:27:41.171 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:27:41.167+0000 7fb7360d7700 1 -- 172.21.15.167:0/523229766 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_command({"prefix": "osd new", "uuid": "61ba9225-834e-4288-b391-7ffa540f7334", "id": 0} v 0) v1 -- 0x7fb73010b900 con 0x7fb730067df0 2023-10-05T19:27:41.176 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:27:41.171+0000 7fb7177fe700 1 -- 172.21.15.167:0/523229766 <== mon.0 v2:172.21.15.118:3300/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "61ba9225-834e-4288-b391-7ffa540f7334", "id": 0}]=0 v3) v1 ==== 112+0+2 (secure 0 0 0) 0x7fb720033070 con 0x7fb730067df0 2023-10-05T19:27:41.176 INFO:teuthology.orchestra.run.smithi167.stdout:0 2023-10-05T19:27:41.177 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:27:41.171+0000 7fb7360d7700 1 -- 172.21.15.167:0/523229766 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb730067df0 msgr2=0x7fb730121a70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-05T19:27:41.177 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:27:41.171+0000 7fb7360d7700 1 --2- 172.21.15.167:0/523229766 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb730067df0 0x7fb730121a70 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7fb720000c00 tx=0x7fb720016000 comp rx=0 tx=0).stop 2023-10-05T19:27:41.177 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:27:41.171+0000 7fb7360d7700 1 -- 172.21.15.167:0/523229766 shutdown_connections 2023-10-05T19:27:41.178 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:27:41.171+0000 7fb7360d7700 1 --2- 172.21.15.167:0/523229766 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb730067df0 0x7fb730121a70 unknown :-1 s=CLOSED pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-05T19:27:41.178 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:27:41.171+0000 7fb7360d7700 1 -- 172.21.15.167:0/523229766 >> 172.21.15.167:0/523229766 conn(0x7fb73006b160 msgr2=0x7fb730117ae0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-05T19:27:41.178 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:27:41.171+0000 7fb7360d7700 1 -- 172.21.15.167:0/523229766 shutdown_connections 2023-10-05T19:27:41.178 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:27:41.171+0000 7fb7360d7700 1 -- 172.21.15.167:0/523229766 wait complete. 2023-10-05T19:27:41.189 DEBUG:teuthology.orchestra.run.smithi167:> sudo ceph --cluster ceph osd new 15a3631e-9f20-4b33-9ee3-39eb73e13709 1 2023-10-05T19:27:41.298 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:27:41.291+0000 7f522ec6a700 1 Processor -- start 2023-10-05T19:27:41.298 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:27:41.291+0000 7f522ec6a700 1 -- start start 2023-10-05T19:27:41.299 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:27:41.295+0000 7f522ec6a700 1 --2- >> v2:172.21.15.118:3300/0 conn(0x7f5228112570 0x7f5228114950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-05T19:27:41.299 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:27:41.295+0000 7f522ec6a700 1 -- --> v1:172.21.15.118:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5228114e90 con 0x7f522810f8b0 2023-10-05T19:27:41.299 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:27:41.295+0000 7f522ec6a700 1 -- --> v2:172.21.15.118:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f522804c3e0 con 0x7f5228112570 2023-10-05T19:27:41.299 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:27:41.295+0000 7f522d467700 1 --2- >> v2:172.21.15.118:3300/0 conn(0x7f5228112570 0x7f5228114950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-05T19:27:41.299 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:27:41.295+0000 7f522d467700 1 --2- >> v2:172.21.15.118:3300/0 conn(0x7f5228112570 0x7f5228114950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.118:3300/0 says I am v2:172.21.15.167:41224/0 (socket says 172.21.15.167:41224) 2023-10-05T19:27:41.299 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:27:41.295+0000 7f522d467700 1 -- 172.21.15.167:0/2253579221 learned_addr learned my addr 172.21.15.167:0/2253579221 (peer_addr_for_me v2:172.21.15.167:0/0) 2023-10-05T19:27:41.300 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:27:41.295+0000 7f522d467700 1 -- 172.21.15.167:0/2253579221 >> v1:172.21.15.118:6789/0 conn(0x7f522810f8b0 legacy=0x7f522810fc80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-05T19:27:41.300 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:27:41.295+0000 7f522d467700 1 -- 172.21.15.167:0/2253579221 --> v2:172.21.15.118:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5228116030 con 0x7f5228112570 2023-10-05T19:27:41.301 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:27:41.295+0000 7f522d467700 1 --2- 172.21.15.167:0/2253579221 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f5228112570 0x7f5228114950 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7f521800bbd0 tx=0x7f521800b4f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=18cade7a7a7d59fe server_cookie=0 in_seq=0 out_seq=0 2023-10-05T19:27:41.301 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:27:41.295+0000 7f522cc66700 1 -- 172.21.15.167:0/2253579221 <== mon.0 v2:172.21.15.118:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f521800dc50 con 0x7f5228112570 2023-10-05T19:27:41.301 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:27:41.295+0000 7f522cc66700 1 -- 172.21.15.167:0/2253579221 <== mon.0 v2:172.21.15.118:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f521800ddb0 con 0x7f5228112570 2023-10-05T19:27:41.301 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:27:41.295+0000 7f522cc66700 1 -- 172.21.15.167:0/2253579221 <== mon.0 v2:172.21.15.118:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f52180105d0 con 0x7f5228112570 2023-10-05T19:27:41.301 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:27:41.295+0000 7f522ec6a700 1 -- 172.21.15.167:0/2253579221 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f5228112570 msgr2=0x7f5228114950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-05T19:27:41.301 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:27:41.295+0000 7f522ec6a700 1 --2- 172.21.15.167:0/2253579221 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f5228112570 0x7f5228114950 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7f521800bbd0 tx=0x7f521800b4f0 comp rx=0 tx=0).stop 2023-10-05T19:27:41.301 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:27:41.295+0000 7f522ec6a700 1 -- 172.21.15.167:0/2253579221 shutdown_connections 2023-10-05T19:27:41.301 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:27:41.295+0000 7f522ec6a700 1 --2- 172.21.15.167:0/2253579221 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f5228112570 0x7f5228114950 unknown :-1 s=CLOSED pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-05T19:27:41.301 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:27:41.295+0000 7f522ec6a700 1 -- 172.21.15.167:0/2253579221 >> 172.21.15.167:0/2253579221 conn(0x7f522806afc0 msgr2=0x7f522806d3d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-05T19:27:41.301 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:27:41.295+0000 7f522ec6a700 1 -- 172.21.15.167:0/2253579221 shutdown_connections 2023-10-05T19:27:41.301 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:27:41.295+0000 7f522ec6a700 1 -- 172.21.15.167:0/2253579221 wait complete. 2023-10-05T19:27:41.303 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:27:41.295+0000 7f522ec6a700 1 Processor -- start 2023-10-05T19:27:41.303 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:27:41.295+0000 7f522ec6a700 1 -- start start 2023-10-05T19:27:41.303 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:27:41.299+0000 7f522ec6a700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f522810f8b0 0x7f522811d4b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-05T19:27:41.304 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:27:41.299+0000 7f522ec6a700 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5228110d90 con 0x7f522810f8b0 2023-10-05T19:27:41.304 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:27:41.299+0000 7f522dc68700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f522810f8b0 0x7f522811d4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-05T19:27:41.304 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:27:41.299+0000 7f522dc68700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f522810f8b0 0x7f522811d4b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.118:3300/0 says I am v2:172.21.15.167:41240/0 (socket says 172.21.15.167:41240) 2023-10-05T19:27:41.304 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:27:41.299+0000 7f522dc68700 1 -- 172.21.15.167:0/2333240423 learned_addr learned my addr 172.21.15.167:0/2333240423 (peer_addr_for_me v2:172.21.15.167:0/0) 2023-10-05T19:27:41.304 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:27:41.299+0000 7f522dc68700 1 -- 172.21.15.167:0/2333240423 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f521800b790 con 0x7f522810f8b0 2023-10-05T19:27:41.304 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:27:41.299+0000 7f522dc68700 1 --2- 172.21.15.167:0/2333240423 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f522810f8b0 0x7f522811d4b0 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7f522400c690 tx=0x7f5224007f40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-05T19:27:41.304 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:27:41.299+0000 7f521dffb700 1 -- 172.21.15.167:0/2333240423 <== mon.0 v2:172.21.15.118:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f52240104f0 con 0x7f522810f8b0 2023-10-05T19:27:41.304 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:27:41.299+0000 7f521dffb700 1 -- 172.21.15.167:0/2333240423 <== mon.0 v2:172.21.15.118:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5224010670 con 0x7f522810f8b0 2023-10-05T19:27:41.304 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:27:41.299+0000 7f522ec6a700 1 -- 172.21.15.167:0/2333240423 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f522811db40 con 0x7f522810f8b0 2023-10-05T19:27:41.304 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:27:41.299+0000 7f521dffb700 1 -- 172.21.15.167:0/2333240423 <== mon.0 v2:172.21.15.118:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f5224010960 con 0x7f522810f8b0 2023-10-05T19:27:41.305 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:27:41.299+0000 7f522ec6a700 1 -- 172.21.15.167:0/2333240423 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f522811df90 con 0x7f522810f8b0 2023-10-05T19:27:41.305 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:27:41.299+0000 7f521dffb700 1 -- 172.21.15.167:0/2333240423 <== mon.0 v2:172.21.15.118:3300/0 4 ==== mgrmap(e 1) v1 ==== 521+0+0 (secure 0 0 0) 0x7f5224010d40 con 0x7f522810f8b0 2023-10-05T19:27:41.305 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:27:41.299+0000 7f521dffb700 1 -- 172.21.15.167:0/2333240423 <== mon.0 v2:172.21.15.118:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 836+0+0 (secure 0 0 0) 0x7f52240127e0 con 0x7f522810f8b0 2023-10-05T19:27:41.305 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:27:41.299+0000 7f522ec6a700 1 -- 172.21.15.167:0/2333240423 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f522805ecb0 con 0x7f522810f8b0 2023-10-05T19:27:41.310 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:27:41.307+0000 7f521dffb700 1 -- 172.21.15.167:0/2333240423 <== mon.0 v2:172.21.15.118:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+71325 (secure 0 0 0) 0x7f5224017070 con 0x7f522810f8b0 2023-10-05T19:27:41.389 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:27:41.383+0000 7f522ec6a700 1 -- 172.21.15.167:0/2333240423 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_command({"prefix": "osd new", "uuid": "15a3631e-9f20-4b33-9ee3-39eb73e13709", "id": 1} v 0) v1 -- 0x7f5228112570 con 0x7f522810f8b0 2023-10-05T19:27:41.393 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:27:41.387+0000 7f521dffb700 1 -- 172.21.15.167:0/2333240423 <== mon.0 v2:172.21.15.118:3300/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "15a3631e-9f20-4b33-9ee3-39eb73e13709", "id": 1}]=0 v4) v1 ==== 112+0+2 (secure 0 0 0) 0x7f522401c750 con 0x7f522810f8b0 2023-10-05T19:27:41.393 INFO:teuthology.orchestra.run.smithi167.stdout:1 2023-10-05T19:27:41.395 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:27:41.391+0000 7f522ec6a700 1 -- 172.21.15.167:0/2333240423 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f522810f8b0 msgr2=0x7f522811d4b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-05T19:27:41.395 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:27:41.391+0000 7f522ec6a700 1 --2- 172.21.15.167:0/2333240423 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f522810f8b0 0x7f522811d4b0 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7f522400c690 tx=0x7f5224007f40 comp rx=0 tx=0).stop 2023-10-05T19:27:41.395 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:27:41.391+0000 7f522ec6a700 1 -- 172.21.15.167:0/2333240423 shutdown_connections 2023-10-05T19:27:41.395 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:27:41.391+0000 7f522ec6a700 1 --2- 172.21.15.167:0/2333240423 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f522810f8b0 0x7f522811d4b0 unknown :-1 s=CLOSED pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-05T19:27:41.395 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:27:41.391+0000 7f522ec6a700 1 -- 172.21.15.167:0/2333240423 >> 172.21.15.167:0/2333240423 conn(0x7f522806afc0 msgr2=0x7f522806b390 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-05T19:27:41.395 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:27:41.391+0000 7f522ec6a700 1 -- 172.21.15.167:0/2333240423 shutdown_connections 2023-10-05T19:27:41.395 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:27:41.391+0000 7f522ec6a700 1 -- 172.21.15.167:0/2333240423 wait complete. 2023-10-05T19:27:41.406 DEBUG:teuthology.orchestra.run.smithi167:> sudo ceph --cluster ceph osd new 5712158a-931c-4ce7-adf0-5c9fbb59d521 2 2023-10-05T19:27:41.514 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:27:41.507+0000 7f5248932700 1 Processor -- start 2023-10-05T19:27:41.514 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:27:41.507+0000 7f5248932700 1 -- start start 2023-10-05T19:27:41.514 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:27:41.507+0000 7f5248932700 1 --2- >> v2:172.21.15.118:3300/0 conn(0x7f524410b740 0x7f52440678b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-05T19:27:41.514 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:27:41.507+0000 7f5248932700 1 -- --> v1:172.21.15.118:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f524410bc90 con 0x7f5244067df0 2023-10-05T19:27:41.514 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:27:41.507+0000 7f5248932700 1 -- --> v2:172.21.15.118:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f524404c3e0 con 0x7f524410b740 2023-10-05T19:27:41.514 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:27:41.511+0000 7f5242d9d700 1 --2- >> v2:172.21.15.118:3300/0 conn(0x7f524410b740 0x7f52440678b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-05T19:27:41.515 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:27:41.511+0000 7f5242d9d700 1 --2- >> v2:172.21.15.118:3300/0 conn(0x7f524410b740 0x7f52440678b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.118:3300/0 says I am v2:172.21.15.167:41246/0 (socket says 172.21.15.167:41246) 2023-10-05T19:27:41.515 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:27:41.511+0000 7f5242d9d700 1 -- 172.21.15.167:0/850198328 learned_addr learned my addr 172.21.15.167:0/850198328 (peer_addr_for_me v2:172.21.15.167:0/0) 2023-10-05T19:27:41.515 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:27:41.511+0000 7f5242d9d700 1 -- 172.21.15.167:0/850198328 >> v1:172.21.15.118:6789/0 conn(0x7f5244067df0 legacy=0x7f52441173b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-05T19:27:41.516 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:27:41.511+0000 7f5242d9d700 1 -- 172.21.15.167:0/850198328 --> v2:172.21.15.118:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5244068290 con 0x7f524410b740 2023-10-05T19:27:41.516 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:27:41.511+0000 7f5242d9d700 1 --2- 172.21.15.167:0/850198328 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f524410b740 0x7f52440678b0 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7f5234009930 tx=0x7f5234009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=606555e3b83fe962 server_cookie=0 in_seq=0 out_seq=0 2023-10-05T19:27:41.516 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:27:41.511+0000 7f5241d9b700 1 -- 172.21.15.167:0/850198328 <== mon.0 v2:172.21.15.118:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f523400b860 con 0x7f524410b740 2023-10-05T19:27:41.516 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:27:41.511+0000 7f5241d9b700 1 -- 172.21.15.167:0/850198328 <== mon.0 v2:172.21.15.118:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f523400b9c0 con 0x7f524410b740 2023-10-05T19:27:41.516 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:27:41.511+0000 7f5241d9b700 1 -- 172.21.15.167:0/850198328 <== mon.0 v2:172.21.15.118:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f523400bb50 con 0x7f524410b740 2023-10-05T19:27:41.517 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:27:41.511+0000 7f5248932700 1 -- 172.21.15.167:0/850198328 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f524410b740 msgr2=0x7f52440678b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-05T19:27:41.517 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:27:41.511+0000 7f5248932700 1 --2- 172.21.15.167:0/850198328 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f524410b740 0x7f52440678b0 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7f5234009930 tx=0x7f5234009250 comp rx=0 tx=0).stop 2023-10-05T19:27:41.517 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:27:41.511+0000 7f5248932700 1 -- 172.21.15.167:0/850198328 shutdown_connections 2023-10-05T19:27:41.517 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:27:41.511+0000 7f5248932700 1 --2- 172.21.15.167:0/850198328 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f524410b740 0x7f52440678b0 unknown :-1 s=CLOSED pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-05T19:27:41.518 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:27:41.511+0000 7f5248932700 1 -- 172.21.15.167:0/850198328 >> 172.21.15.167:0/850198328 conn(0x7f524406afc0 msgr2=0x7f524406d3d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-05T19:27:41.518 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:27:41.511+0000 7f5248932700 1 -- 172.21.15.167:0/850198328 shutdown_connections 2023-10-05T19:27:41.518 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:27:41.511+0000 7f5248932700 1 -- 172.21.15.167:0/850198328 wait complete. 2023-10-05T19:27:41.518 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:27:41.511+0000 7f5248932700 1 Processor -- start 2023-10-05T19:27:41.518 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:27:41.511+0000 7f5248932700 1 -- start start 2023-10-05T19:27:41.518 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:27:41.511+0000 7f5248932700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f5244067df0 0x7f524411bc90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-05T19:27:41.518 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:27:41.511+0000 7f5248932700 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f524406d860 con 0x7f5244067df0 2023-10-05T19:27:41.520 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:27:41.515+0000 7f5242d9d700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f5244067df0 0x7f524411bc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-05T19:27:41.520 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:27:41.515+0000 7f5242d9d700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f5244067df0 0x7f524411bc90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.118:3300/0 says I am v2:172.21.15.167:41248/0 (socket says 172.21.15.167:41248) 2023-10-05T19:27:41.520 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:27:41.515+0000 7f5242d9d700 1 -- 172.21.15.167:0/577418171 learned_addr learned my addr 172.21.15.167:0/577418171 (peer_addr_for_me v2:172.21.15.167:0/0) 2023-10-05T19:27:41.520 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:27:41.515+0000 7f5242d9d700 1 -- 172.21.15.167:0/577418171 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f52340094f0 con 0x7f5244067df0 2023-10-05T19:27:41.520 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:27:41.515+0000 7f5242d9d700 1 --2- 172.21.15.167:0/577418171 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f5244067df0 0x7f524411bc90 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7f52340094c0 tx=0x7f5234016000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-05T19:27:41.520 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:27:41.515+0000 7f522affd700 1 -- 172.21.15.167:0/577418171 <== mon.0 v2:172.21.15.118:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f523400bd40 con 0x7f5244067df0 2023-10-05T19:27:41.520 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:27:41.515+0000 7f522affd700 1 -- 172.21.15.167:0/577418171 <== mon.0 v2:172.21.15.118:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5234020070 con 0x7f5244067df0 2023-10-05T19:27:41.520 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:27:41.515+0000 7f522affd700 1 -- 172.21.15.167:0/577418171 <== mon.0 v2:172.21.15.118:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f523401b450 con 0x7f5244067df0 2023-10-05T19:27:41.520 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:27:41.515+0000 7f5248932700 1 -- 172.21.15.167:0/577418171 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f524411c1d0 con 0x7f5244067df0 2023-10-05T19:27:41.520 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:27:41.515+0000 7f5248932700 1 -- 172.21.15.167:0/577418171 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f524411c6a0 con 0x7f5244067df0 2023-10-05T19:27:41.521 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:27:41.515+0000 7f522affd700 1 -- 172.21.15.167:0/577418171 <== mon.0 v2:172.21.15.118:3300/0 4 ==== mgrmap(e 1) v1 ==== 521+0+0 (secure 0 0 0) 0x7f5234022030 con 0x7f5244067df0 2023-10-05T19:27:41.521 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:27:41.515+0000 7f522affd700 1 -- 172.21.15.167:0/577418171 <== mon.0 v2:172.21.15.118:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 951+0+0 (secure 0 0 0) 0x7f523401c880 con 0x7f5244067df0 2023-10-05T19:27:41.521 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:27:41.515+0000 7f5248932700 1 -- 172.21.15.167:0/577418171 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f524411b8a0 con 0x7f5244067df0 2023-10-05T19:27:41.526 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:27:41.523+0000 7f522affd700 1 -- 172.21.15.167:0/577418171 <== mon.0 v2:172.21.15.118:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+71325 (secure 0 0 0) 0x7f523401c450 con 0x7f5244067df0 2023-10-05T19:27:41.605 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:27:41.599+0000 7f5248932700 1 -- 172.21.15.167:0/577418171 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_command({"prefix": "osd new", "uuid": "5712158a-931c-4ce7-adf0-5c9fbb59d521", "id": 2} v 0) v1 -- 0x7f52440685a0 con 0x7f5244067df0 2023-10-05T19:27:41.610 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:27:41.607+0000 7f522affd700 1 -- 172.21.15.167:0/577418171 <== mon.0 v2:172.21.15.118:3300/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "5712158a-931c-4ce7-adf0-5c9fbb59d521", "id": 2}]=0 v5) v1 ==== 112+0+2 (secure 0 0 0) 0x7f523401c670 con 0x7f5244067df0 2023-10-05T19:27:41.610 INFO:teuthology.orchestra.run.smithi167.stdout:2 2023-10-05T19:27:41.611 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:27:41.607+0000 7f5248932700 1 -- 172.21.15.167:0/577418171 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f5244067df0 msgr2=0x7f524411bc90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-05T19:27:41.612 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:27:41.607+0000 7f5248932700 1 --2- 172.21.15.167:0/577418171 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f5244067df0 0x7f524411bc90 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7f52340094c0 tx=0x7f5234016000 comp rx=0 tx=0).stop 2023-10-05T19:27:41.612 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:27:41.607+0000 7f5248932700 1 -- 172.21.15.167:0/577418171 shutdown_connections 2023-10-05T19:27:41.612 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:27:41.607+0000 7f5248932700 1 --2- 172.21.15.167:0/577418171 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f5244067df0 0x7f524411bc90 unknown :-1 s=CLOSED pgs=15 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-05T19:27:41.612 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:27:41.607+0000 7f5248932700 1 -- 172.21.15.167:0/577418171 >> 172.21.15.167:0/577418171 conn(0x7f524406afc0 msgr2=0x7f5244117ad0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-05T19:27:41.612 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:27:41.607+0000 7f5248932700 1 -- 172.21.15.167:0/577418171 shutdown_connections 2023-10-05T19:27:41.612 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:27:41.607+0000 7f5248932700 1 -- 172.21.15.167:0/577418171 wait complete. 2023-10-05T19:27:41.623 DEBUG:teuthology.orchestra.run.smithi167:> sudo ceph --cluster ceph osd new f838ed4e-05a7-49ef-ad9d-8f7dd7be30f1 3 2023-10-05T19:27:41.627 INFO:tasks.ceph.mgr.x.smithi167.stderr:2023-10-05T19:27:41.623+0000 7fb91b495000 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2023-10-05T19:27:41.685 INFO:tasks.ceph.mgr.x.smithi167.stderr:2023-10-05T19:27:41.679+0000 7fb91b495000 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2023-10-05T19:27:41.728 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:27:41.723+0000 7f61b095c700 1 Processor -- start 2023-10-05T19:27:41.728 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:27:41.723+0000 7f61b095c700 1 -- start start 2023-10-05T19:27:41.729 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:27:41.723+0000 7f61b095c700 1 --2- >> v2:172.21.15.118:3300/0 conn(0x7f61ac06efb0 0x7f61ac1160d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-05T19:27:41.729 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:27:41.723+0000 7f61b095c700 1 -- --> v1:172.21.15.118:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f61ac10ac90 con 0x7f61ac10a740 2023-10-05T19:27:41.729 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:27:41.723+0000 7f61b095c700 1 -- --> v2:172.21.15.118:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f61ac04c3e0 con 0x7f61ac06efb0 2023-10-05T19:27:41.729 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:27:41.723+0000 7f61aa59c700 1 --2- >> v2:172.21.15.118:3300/0 conn(0x7f61ac06efb0 0x7f61ac1160d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-05T19:27:41.729 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:27:41.723+0000 7f61aa59c700 1 --2- >> v2:172.21.15.118:3300/0 conn(0x7f61ac06efb0 0x7f61ac1160d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.118:3300/0 says I am v2:172.21.15.167:41252/0 (socket says 172.21.15.167:41252) 2023-10-05T19:27:41.729 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:27:41.723+0000 7f61aa59c700 1 -- 172.21.15.167:0/3648374812 learned_addr learned my addr 172.21.15.167:0/3648374812 (peer_addr_for_me v2:172.21.15.167:0/0) 2023-10-05T19:27:41.729 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:27:41.723+0000 7f61aad9d700 1 --1- 172.21.15.167:0/3648374812 >> v1:172.21.15.118:6789/0 conn(0x7f61ac10a740 0x7f61ac06e8a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.118:6789/0 says I am v1:172.21.15.167:54542/0 (socket says 172.21.15.167:54542) 2023-10-05T19:27:41.731 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:27:41.723+0000 7f61aa59c700 1 -- 172.21.15.167:0/3648374812 >> v1:172.21.15.118:6789/0 conn(0x7f61ac10a740 legacy=0x7f61ac06e8a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-05T19:27:41.731 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:27:41.723+0000 7f61aa59c700 1 -- 172.21.15.167:0/3648374812 --> v2:172.21.15.118:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f61ac06f450 con 0x7f61ac06efb0 2023-10-05T19:27:41.731 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:27:41.723+0000 7f61aa59c700 1 --2- 172.21.15.167:0/3648374812 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f61ac06efb0 0x7f61ac1160d0 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7f61a0009930 tx=0x7f61a0009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=52a62fc4ee7cf952 server_cookie=0 in_seq=0 out_seq=0 2023-10-05T19:27:41.731 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:27:41.727+0000 7f61a9d9b700 1 -- 172.21.15.167:0/3648374812 <== mon.0 v2:172.21.15.118:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f61a000b860 con 0x7f61ac06efb0 2023-10-05T19:27:41.731 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:27:41.727+0000 7f61a9d9b700 1 -- 172.21.15.167:0/3648374812 <== mon.0 v2:172.21.15.118:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f61a000b9c0 con 0x7f61ac06efb0 2023-10-05T19:27:41.731 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:27:41.727+0000 7f61a9d9b700 1 -- 172.21.15.167:0/3648374812 <== mon.0 v2:172.21.15.118:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f61a000bb50 con 0x7f61ac06efb0 2023-10-05T19:27:41.732 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:27:41.727+0000 7f61b095c700 1 -- 172.21.15.167:0/3648374812 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f61ac06efb0 msgr2=0x7f61ac1160d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-05T19:27:41.732 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:27:41.727+0000 7f61b095c700 1 --2- 172.21.15.167:0/3648374812 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f61ac06efb0 0x7f61ac1160d0 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7f61a0009930 tx=0x7f61a0009250 comp rx=0 tx=0).stop 2023-10-05T19:27:41.732 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:27:41.727+0000 7f61b095c700 1 -- 172.21.15.167:0/3648374812 shutdown_connections 2023-10-05T19:27:41.732 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:27:41.727+0000 7f61b095c700 1 --2- 172.21.15.167:0/3648374812 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f61ac06efb0 0x7f61ac1160d0 unknown :-1 s=CLOSED pgs=17 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-05T19:27:41.732 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:27:41.727+0000 7f61b095c700 1 -- 172.21.15.167:0/3648374812 >> 172.21.15.167:0/3648374812 conn(0x7f61ac06a740 msgr2=0x7f61ac06cb50 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-05T19:27:41.732 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:27:41.727+0000 7f61b095c700 1 -- 172.21.15.167:0/3648374812 shutdown_connections 2023-10-05T19:27:41.732 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:27:41.727+0000 7f61b095c700 1 -- 172.21.15.167:0/3648374812 wait complete. 2023-10-05T19:27:41.732 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:27:41.727+0000 7f61b095c700 1 Processor -- start 2023-10-05T19:27:41.732 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:27:41.727+0000 7f61b095c700 1 -- start start 2023-10-05T19:27:41.732 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:27:41.727+0000 7f61b095c700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f61ac06efb0 0x7f61ac10af20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-05T19:27:41.732 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:27:41.727+0000 7f61b095c700 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f61ac06de20 con 0x7f61ac06efb0 2023-10-05T19:27:41.734 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:27:41.727+0000 7f61aad9d700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f61ac06efb0 0x7f61ac10af20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-05T19:27:41.734 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:27:41.727+0000 7f61aad9d700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f61ac06efb0 0x7f61ac10af20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.118:3300/0 says I am v2:172.21.15.167:41258/0 (socket says 172.21.15.167:41258) 2023-10-05T19:27:41.734 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:27:41.727+0000 7f61aad9d700 1 -- 172.21.15.167:0/1618022753 learned_addr learned my addr 172.21.15.167:0/1618022753 (peer_addr_for_me v2:172.21.15.167:0/0) 2023-10-05T19:27:41.734 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:27:41.727+0000 7f61aad9d700 1 -- 172.21.15.167:0/1618022753 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f61a00094f0 con 0x7f61ac06efb0 2023-10-05T19:27:41.734 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:27:41.727+0000 7f61aad9d700 1 --2- 172.21.15.167:0/1618022753 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f61ac06efb0 0x7f61ac10af20 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7f619c00c690 tx=0x7f619c007f40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-05T19:27:41.734 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:27:41.727+0000 7f6192ffd700 1 -- 172.21.15.167:0/1618022753 <== mon.0 v2:172.21.15.118:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f619c005b90 con 0x7f61ac06efb0 2023-10-05T19:27:41.735 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:27:41.727+0000 7f6192ffd700 1 -- 172.21.15.167:0/1618022753 <== mon.0 v2:172.21.15.118:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f619c005cf0 con 0x7f61ac06efb0 2023-10-05T19:27:41.735 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:27:41.727+0000 7f6192ffd700 1 -- 172.21.15.167:0/1618022753 <== mon.0 v2:172.21.15.118:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f619c0070d0 con 0x7f61ac06efb0 2023-10-05T19:27:41.735 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:27:41.727+0000 7f61b095c700 1 -- 172.21.15.167:0/1618022753 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f61ac10b460 con 0x7f61ac06efb0 2023-10-05T19:27:41.735 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:27:41.727+0000 7f61b095c700 1 -- 172.21.15.167:0/1618022753 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f61ac10b930 con 0x7f61ac06efb0 2023-10-05T19:27:41.735 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:27:41.727+0000 7f6192ffd700 1 -- 172.21.15.167:0/1618022753 <== mon.0 v2:172.21.15.118:3300/0 4 ==== mgrmap(e 1) v1 ==== 521+0+0 (secure 0 0 0) 0x7f619c007500 con 0x7f61ac06efb0 2023-10-05T19:27:41.735 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:27:41.727+0000 7f6192ffd700 1 -- 172.21.15.167:0/1618022753 <== mon.0 v2:172.21.15.118:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1066+0+0 (secure 0 0 0) 0x7f619c0128a0 con 0x7f61ac06efb0 2023-10-05T19:27:41.735 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:27:41.731+0000 7f61b095c700 1 -- 172.21.15.167:0/1618022753 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f61ac05ecb0 con 0x7f61ac06efb0 2023-10-05T19:27:41.739 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:27:41.735+0000 7f6192ffd700 1 -- 172.21.15.167:0/1618022753 <== mon.0 v2:172.21.15.118:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+71325 (secure 0 0 0) 0x7f619c017070 con 0x7f61ac06efb0 2023-10-05T19:27:41.800 INFO:tasks.ceph.mgr.x.smithi167.stderr:2023-10-05T19:27:41.795+0000 7fb91b495000 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2023-10-05T19:27:41.812 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:27:41.807+0000 7f61b095c700 1 -- 172.21.15.167:0/1618022753 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_command({"prefix": "osd new", "uuid": "f838ed4e-05a7-49ef-ad9d-8f7dd7be30f1", "id": 3} v 0) v1 -- 0x7f61ac10a740 con 0x7f61ac06efb0 2023-10-05T19:27:41.818 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:27:41.815+0000 7f6192ffd700 1 -- 172.21.15.167:0/1618022753 <== mon.0 v2:172.21.15.118:3300/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "f838ed4e-05a7-49ef-ad9d-8f7dd7be30f1", "id": 3}]=0 v6) v1 ==== 112+0+2 (secure 0 0 0) 0x7f619c0124e0 con 0x7f61ac06efb0 2023-10-05T19:27:41.818 INFO:teuthology.orchestra.run.smithi167.stdout:3 2023-10-05T19:27:41.819 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:27:41.815+0000 7f61b095c700 1 -- 172.21.15.167:0/1618022753 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f61ac06efb0 msgr2=0x7f61ac10af20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-05T19:27:41.819 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:27:41.815+0000 7f61b095c700 1 --2- 172.21.15.167:0/1618022753 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f61ac06efb0 0x7f61ac10af20 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7f619c00c690 tx=0x7f619c007f40 comp rx=0 tx=0).stop 2023-10-05T19:27:41.820 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:27:41.815+0000 7f61b095c700 1 -- 172.21.15.167:0/1618022753 shutdown_connections 2023-10-05T19:27:41.820 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:27:41.815+0000 7f61b095c700 1 --2- 172.21.15.167:0/1618022753 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f61ac06efb0 0x7f61ac10af20 unknown :-1 s=CLOSED pgs=18 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-05T19:27:41.820 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:27:41.815+0000 7f61b095c700 1 -- 172.21.15.167:0/1618022753 >> 172.21.15.167:0/1618022753 conn(0x7f61ac06a740 msgr2=0x7f61ac115580 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-05T19:27:41.820 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:27:41.815+0000 7f61b095c700 1 -- 172.21.15.167:0/1618022753 shutdown_connections 2023-10-05T19:27:41.820 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:27:41.815+0000 7f61b095c700 1 -- 172.21.15.167:0/1618022753 wait complete. 2023-10-05T19:27:41.832 INFO:tasks.ceph.osd.0:Restarting daemon 2023-10-05T19:27:41.832 DEBUG:teuthology.orchestra.run.smithi118:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 0 2023-10-05T19:27:41.834 INFO:tasks.ceph.osd.0:Started 2023-10-05T19:27:41.835 INFO:tasks.ceph.osd.1:Restarting daemon 2023-10-05T19:27:41.835 DEBUG:teuthology.orchestra.run.smithi167:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 1 2023-10-05T19:27:41.875 INFO:tasks.ceph.osd.1:Started 2023-10-05T19:27:41.875 INFO:tasks.ceph.osd.2:Restarting daemon 2023-10-05T19:27:41.875 DEBUG:teuthology.orchestra.run.smithi167:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 2 2023-10-05T19:27:41.877 INFO:tasks.ceph.osd.2:Started 2023-10-05T19:27:41.878 INFO:tasks.ceph.osd.3:Restarting daemon 2023-10-05T19:27:41.878 DEBUG:teuthology.orchestra.run.smithi167:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 3 2023-10-05T19:27:41.880 INFO:tasks.ceph.osd.3:Started 2023-10-05T19:27:41.880 DEBUG:teuthology.orchestra.run.smithi118:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-05T19:27:41.989 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-05T19:27:41.983+0000 7fc9c359e700 1 Processor -- start 2023-10-05T19:27:41.989 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-05T19:27:41.983+0000 7fc9c359e700 1 -- start start 2023-10-05T19:27:41.989 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-05T19:27:41.983+0000 7fc9c359e700 1 --2- >> v2:172.21.15.118:3300/0 conn(0x7fc9c4095390 0x7fc9c4096b00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-05T19:27:41.989 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-05T19:27:41.983+0000 7fc9c359e700 1 -- --> v1:172.21.15.118:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc9c40961c0 con 0x7fc9c4097040 2023-10-05T19:27:41.990 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-05T19:27:41.983+0000 7fc9c359e700 1 -- --> v2:172.21.15.118:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc9c4096470 con 0x7fc9c4095390 2023-10-05T19:27:41.990 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-05T19:27:41.983+0000 7fc9c259c700 1 --2- >> v2:172.21.15.118:3300/0 conn(0x7fc9c4095390 0x7fc9c4096b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-05T19:27:41.990 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-05T19:27:41.983+0000 7fc9c259c700 1 --2- >> v2:172.21.15.118:3300/0 conn(0x7fc9c4095390 0x7fc9c4096b00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.118:3300/0 says I am v2:172.21.15.118:56524/0 (socket says 172.21.15.118:56524) 2023-10-05T19:27:41.990 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-05T19:27:41.983+0000 7fc9c259c700 1 -- 172.21.15.118:0/2663662165 learned_addr learned my addr 172.21.15.118:0/2663662165 (peer_addr_for_me v2:172.21.15.118:0/0) 2023-10-05T19:27:41.990 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-05T19:27:41.983+0000 7fc9c1d9b700 1 --1- 172.21.15.118:0/2663662165 >> v1:172.21.15.118:6789/0 conn(0x7fc9c4097040 0x7fc9c4126fe0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.118:6789/0 says I am v1:172.21.15.118:43114/0 (socket says 172.21.15.118:43114) 2023-10-05T19:27:41.991 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-05T19:27:41.983+0000 7fc9c259c700 1 -- 172.21.15.118:0/2663662165 >> v1:172.21.15.118:6789/0 conn(0x7fc9c4097040 legacy=0x7fc9c4126fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-05T19:27:41.991 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-05T19:27:41.983+0000 7fc9c259c700 1 -- 172.21.15.118:0/2663662165 --> v2:172.21.15.118:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc9c4096780 con 0x7fc9c4095390 2023-10-05T19:27:41.992 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-05T19:27:41.983+0000 7fc9c259c700 1 --2- 172.21.15.118:0/2663662165 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fc9c4095390 0x7fc9c4096b00 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7fc9b4009930 tx=0x7fc9b4009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7387acc1d2a0f4d4 server_cookie=0 in_seq=0 out_seq=0 2023-10-05T19:27:41.992 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-05T19:27:41.983+0000 7fc9c159a700 1 -- 172.21.15.118:0/2663662165 <== mon.0 v2:172.21.15.118:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fc9b400b860 con 0x7fc9c4095390 2023-10-05T19:27:41.992 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-05T19:27:41.983+0000 7fc9c159a700 1 -- 172.21.15.118:0/2663662165 <== mon.0 v2:172.21.15.118:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc9b400b9c0 con 0x7fc9c4095390 2023-10-05T19:27:41.992 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-05T19:27:41.983+0000 7fc9c159a700 1 -- 172.21.15.118:0/2663662165 <== mon.0 v2:172.21.15.118:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fc9b400bc90 con 0x7fc9c4095390 2023-10-05T19:27:41.992 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-05T19:27:41.983+0000 7fc9c359e700 1 -- 172.21.15.118:0/2663662165 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fc9c4095390 msgr2=0x7fc9c4096b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-05T19:27:41.992 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-05T19:27:41.983+0000 7fc9c359e700 1 --2- 172.21.15.118:0/2663662165 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fc9c4095390 0x7fc9c4096b00 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7fc9b4009930 tx=0x7fc9b4009250 comp rx=0 tx=0).stop 2023-10-05T19:27:41.992 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-05T19:27:41.983+0000 7fc9c359e700 1 -- 172.21.15.118:0/2663662165 shutdown_connections 2023-10-05T19:27:41.992 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-05T19:27:41.983+0000 7fc9c359e700 1 --2- 172.21.15.118:0/2663662165 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fc9c4095390 0x7fc9c4096b00 unknown :-1 s=CLOSED pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-05T19:27:41.992 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-05T19:27:41.983+0000 7fc9c359e700 1 -- 172.21.15.118:0/2663662165 >> 172.21.15.118:0/2663662165 conn(0x7fc9c409a740 msgr2=0x7fc9c409ab40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-05T19:27:41.992 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-05T19:27:41.983+0000 7fc9c359e700 1 -- 172.21.15.118:0/2663662165 shutdown_connections 2023-10-05T19:27:41.992 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-05T19:27:41.983+0000 7fc9c359e700 1 -- 172.21.15.118:0/2663662165 wait complete. 2023-10-05T19:27:41.994 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-05T19:27:41.987+0000 7fc9c359e700 1 Processor -- start 2023-10-05T19:27:41.994 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-05T19:27:41.987+0000 7fc9c359e700 1 -- start start 2023-10-05T19:27:41.994 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-05T19:27:41.987+0000 7fc9c359e700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fc9c4095390 0x7fc9c4138470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-05T19:27:41.994 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-05T19:27:41.987+0000 7fc9c359e700 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc9c411ce90 con 0x7fc9c4095390 2023-10-05T19:27:41.995 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-05T19:27:41.987+0000 7fc9c259c700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fc9c4095390 0x7fc9c4138470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-05T19:27:41.995 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-05T19:27:41.987+0000 7fc9c259c700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fc9c4095390 0x7fc9c4138470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.118:3300/0 says I am v2:172.21.15.118:56534/0 (socket says 172.21.15.118:56534) 2023-10-05T19:27:41.995 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-05T19:27:41.987+0000 7fc9c259c700 1 -- 172.21.15.118:0/2546761113 learned_addr learned my addr 172.21.15.118:0/2546761113 (peer_addr_for_me v2:172.21.15.118:0/0) 2023-10-05T19:27:41.995 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-05T19:27:41.987+0000 7fc9c259c700 1 -- 172.21.15.118:0/2546761113 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc9b40094f0 con 0x7fc9c4095390 2023-10-05T19:27:41.995 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-05T19:27:41.987+0000 7fc9c259c700 1 --2- 172.21.15.118:0/2546761113 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fc9c4095390 0x7fc9c4138470 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7fc9b40094c0 tx=0x7fc9b4016000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-05T19:27:41.995 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-05T19:27:41.987+0000 7fc9b27fc700 1 -- 172.21.15.118:0/2546761113 <== mon.0 v2:172.21.15.118:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fc9b400bb30 con 0x7fc9c4095390 2023-10-05T19:27:41.995 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-05T19:27:41.987+0000 7fc9b27fc700 1 -- 172.21.15.118:0/2546761113 <== mon.0 v2:172.21.15.118:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc9b4020070 con 0x7fc9c4095390 2023-10-05T19:27:41.995 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-05T19:27:41.987+0000 7fc9b27fc700 1 -- 172.21.15.118:0/2546761113 <== mon.0 v2:172.21.15.118:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fc9b401b450 con 0x7fc9c4095390 2023-10-05T19:27:41.995 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-05T19:27:41.987+0000 7fc9c359e700 1 -- 172.21.15.118:0/2546761113 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc9c411be00 con 0x7fc9c4095390 2023-10-05T19:27:41.995 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-05T19:27:41.987+0000 7fc9c359e700 1 -- 172.21.15.118:0/2546761113 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc9c406b5a0 con 0x7fc9c4095390 2023-10-05T19:27:41.995 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-05T19:27:41.987+0000 7fc9b27fc700 1 -- 172.21.15.118:0/2546761113 <== mon.0 v2:172.21.15.118:3300/0 4 ==== mgrmap(e 1) v1 ==== 521+0+0 (secure 0 0 0) 0x7fc9b4022030 con 0x7fc9c4095390 2023-10-05T19:27:41.995 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-05T19:27:41.987+0000 7fc9b27fc700 1 -- 172.21.15.118:0/2546761113 <== mon.0 v2:172.21.15.118:3300/0 5 ==== osd_map(6..6 src has 1..6) v4 ==== 1181+0+0 (secure 0 0 0) 0x7fc9b401c960 con 0x7fc9c4095390 2023-10-05T19:27:41.996 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-05T19:27:41.987+0000 7fc9c359e700 1 -- 172.21.15.118:0/2546761113 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc9c405ecb0 con 0x7fc9c4095390 2023-10-05T19:27:41.999 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-05T19:27:41.991+0000 7fc9b27fc700 1 -- 172.21.15.118:0/2546761113 <== mon.0 v2:172.21.15.118:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+71325 (secure 0 0 0) 0x7fc9b401cc00 con 0x7fc9c4095390 2023-10-05T19:27:42.049 INFO:tasks.ceph.mgr.x.smithi167.stderr:2023-10-05T19:27:42.043+0000 7fb91b495000 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2023-10-05T19:27:42.063 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-05T19:27:42.055+0000 7fc9c359e700 1 -- 172.21.15.118:0/2546761113 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fc9c4095b00 con 0x7fc9c4095390 2023-10-05T19:27:42.064 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-05T19:27:42.055+0000 7fc9b27fc700 1 -- 172.21.15.118:0/2546761113 <== mon.0 v2:172.21.15.118:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v6) v1 ==== 74+0+4108 (secure 0 0 0) 0x7fc9b401cda0 con 0x7fc9c4095390 2023-10-05T19:27:42.064 INFO:teuthology.orchestra.run.smithi118.stdout: 2023-10-05T19:27:42.064 INFO:teuthology.orchestra.run.smithi118.stdout:{"epoch":6,"fsid":"8c4bd0d1-cbcb-4713-96b7-dcb14bcaf584","created":"2023-10-05T19:27:40.755769+0000","modified":"2023-10-05T19:27:41.811702+0000","last_up_change":"0.000000","last_in_change":"2023-10-05T19:27:41.811702+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[],"osds":[{"osd":0,"uuid":"61ba9225-834e-4288-b391-7ffa540f7334","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"15a3631e-9f20-4b33-9ee3-39eb73e13709","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"5712158a-931c-4ce7-adf0-5c9fbb59d521","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"f838ed4e-05a7-49ef-ad9d-8f7dd7be30f1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-05T19:27:42.066 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-05T19:27:42.059+0000 7fc9c359e700 1 -- 172.21.15.118:0/2546761113 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fc9c4095390 msgr2=0x7fc9c4138470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-05T19:27:42.066 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-05T19:27:42.059+0000 7fc9c359e700 1 --2- 172.21.15.118:0/2546761113 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fc9c4095390 0x7fc9c4138470 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7fc9b40094c0 tx=0x7fc9b4016000 comp rx=0 tx=0).stop 2023-10-05T19:27:42.067 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-05T19:27:42.059+0000 7fc9c359e700 1 -- 172.21.15.118:0/2546761113 shutdown_connections 2023-10-05T19:27:42.067 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-05T19:27:42.059+0000 7fc9c359e700 1 --2- 172.21.15.118:0/2546761113 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fc9c4095390 0x7fc9c4138470 unknown :-1 s=CLOSED pgs=29 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-05T19:27:42.067 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-05T19:27:42.059+0000 7fc9c359e700 1 -- 172.21.15.118:0/2546761113 >> 172.21.15.118:0/2546761113 conn(0x7fc9c409a740 msgr2=0x7fc9c4110c00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-05T19:27:42.067 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-05T19:27:42.059+0000 7fc9c359e700 1 -- 172.21.15.118:0/2546761113 shutdown_connections 2023-10-05T19:27:42.067 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-05T19:27:42.059+0000 7fc9c359e700 1 -- 172.21.15.118:0/2546761113 wait complete. 2023-10-05T19:27:42.077 INFO:tasks.ceph.ceph_manager.ceph:[] 2023-10-05T19:27:42.077 INFO:tasks.ceph:Waiting for OSDs to come up 2023-10-05T19:27:42.101 INFO:tasks.ceph.mgr.x.smithi167.stderr:2023-10-05T19:27:42.095+0000 7fb91b495000 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2023-10-05T19:27:42.290 INFO:tasks.ceph.mgr.x.smithi167.stderr:2023-10-05T19:27:42.287+0000 7fb91b495000 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2023-10-05T19:27:42.352 INFO:tasks.ceph.mgr.x.smithi167.stderr:2023-10-05T19:27:42.347+0000 7fb91b495000 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2023-10-05T19:27:42.412 INFO:tasks.ceph.mgr.x.smithi167.stderr:2023-10-05T19:27:42.407+0000 7fb91b495000 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2023-10-05T19:27:42.458 INFO:tasks.ceph.osd.0.smithi118.stderr:2023-10-05T19:27:42.451+0000 7f6eef56f3c0 -1 Falling back to public interface 2023-10-05T19:27:42.479 DEBUG:teuthology.orchestra.run.smithi118:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2023-10-05T19:27:42.504 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-05T19:27:42.499+0000 7fbd287073c0 -1 Falling back to public interface 2023-10-05T19:27:42.506 INFO:tasks.ceph.osd.3.smithi167.stderr:2023-10-05T19:27:42.503+0000 7f6c015ca3c0 -1 Falling back to public interface 2023-10-05T19:27:42.518 INFO:tasks.ceph.osd.2.smithi167.stderr:2023-10-05T19:27:42.515+0000 7f8d55fc83c0 -1 Falling back to public interface 2023-10-05T19:27:42.580 INFO:teuthology.misc.health.smithi118.stderr:2023-10-05T19:27:42.571+0000 7f44bf59e700 1 Processor -- start 2023-10-05T19:27:42.581 INFO:teuthology.misc.health.smithi118.stderr:2023-10-05T19:27:42.571+0000 7f44bf59e700 1 -- start start 2023-10-05T19:27:42.581 INFO:teuthology.misc.health.smithi118.stderr:2023-10-05T19:27:42.575+0000 7f44bf59e700 1 --2- >> v2:172.21.15.118:3300/0 conn(0x7f44c0091280 0x7f44c012b590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-05T19:27:42.581 INFO:teuthology.misc.health.smithi118.stderr:2023-10-05T19:27:42.575+0000 7f44bf59e700 1 -- --> v1:172.21.15.118:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f44c0091ae0 con 0x7f44c00907a0 2023-10-05T19:27:42.581 INFO:teuthology.misc.health.smithi118.stderr:2023-10-05T19:27:42.575+0000 7f44bf59e700 1 -- --> v2:172.21.15.118:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f44c0091f90 con 0x7f44c0091280 2023-10-05T19:27:42.581 INFO:teuthology.misc.health.smithi118.stderr:2023-10-05T19:27:42.575+0000 7f44be59c700 1 --1- >> v1:172.21.15.118:6789/0 conn(0x7f44c00907a0 0x7f44c0090b70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.118:6789/0 says I am v1:172.21.15.118:43120/0 (socket says 172.21.15.118:43120) 2023-10-05T19:27:42.581 INFO:teuthology.misc.health.smithi118.stderr:2023-10-05T19:27:42.575+0000 7f44be59c700 1 -- 172.21.15.118:0/2626045551 learned_addr learned my addr 172.21.15.118:0/2626045551 (peer_addr_for_me v1:172.21.15.118:0/0) 2023-10-05T19:27:42.581 INFO:teuthology.misc.health.smithi118.stderr:2023-10-05T19:27:42.575+0000 7f44bdd9b700 1 --2- 172.21.15.118:0/2626045551 >> v2:172.21.15.118:3300/0 conn(0x7f44c0091280 0x7f44c012b590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-05T19:27:42.583 INFO:teuthology.misc.health.smithi118.stderr:2023-10-05T19:27:42.575+0000 7f44bd59a700 1 -- 172.21.15.118:0/2626045551 <== mon.0 v1:172.21.15.118:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4278408971 0 0) 0x7f44c0091ae0 con 0x7f44c00907a0 2023-10-05T19:27:42.583 INFO:teuthology.misc.health.smithi118.stderr:2023-10-05T19:27:42.575+0000 7f44bd59a700 1 -- 172.21.15.118:0/2626045551 --> v1:172.21.15.118:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f44a8003570 con 0x7f44c00907a0 2023-10-05T19:27:42.583 INFO:teuthology.misc.health.smithi118.stderr:2023-10-05T19:27:42.575+0000 7f44bdd9b700 1 -- 172.21.15.118:0/2626045551 >> v1:172.21.15.118:6789/0 conn(0x7f44c00907a0 legacy=0x7f44c0090b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-05T19:27:42.583 INFO:teuthology.misc.health.smithi118.stderr:2023-10-05T19:27:42.575+0000 7f44bdd9b700 1 -- 172.21.15.118:0/2626045551 --> v2:172.21.15.118:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f44c0069940 con 0x7f44c0091280 2023-10-05T19:27:42.583 INFO:teuthology.misc.health.smithi118.stderr:2023-10-05T19:27:42.575+0000 7f44bdd9b700 1 --2- 172.21.15.118:0/2626045551 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f44c0091280 0x7f44c012b590 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7f44b4009860 tx=0x7f44b40091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=11bdb7f8b5fbab2e server_cookie=0 in_seq=0 out_seq=0 2023-10-05T19:27:42.583 INFO:teuthology.misc.health.smithi118.stderr:2023-10-05T19:27:42.575+0000 7f44bd59a700 1 -- 172.21.15.118:0/2626045551 <== mon.0 v2:172.21.15.118:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f44b4011630 con 0x7f44c0091280 2023-10-05T19:27:42.583 INFO:teuthology.misc.health.smithi118.stderr:2023-10-05T19:27:42.575+0000 7f44bd59a700 1 -- 172.21.15.118:0/2626045551 <== mon.0 v2:172.21.15.118:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f44b4011790 con 0x7f44c0091280 2023-10-05T19:27:42.583 INFO:teuthology.misc.health.smithi118.stderr:2023-10-05T19:27:42.575+0000 7f44bd59a700 1 -- 172.21.15.118:0/2626045551 <== mon.0 v2:172.21.15.118:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f44b4011a40 con 0x7f44c0091280 2023-10-05T19:27:42.583 INFO:teuthology.misc.health.smithi118.stderr:2023-10-05T19:27:42.575+0000 7f44bf59e700 1 -- 172.21.15.118:0/2626045551 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f44c0091280 msgr2=0x7f44c012b590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-05T19:27:42.583 INFO:teuthology.misc.health.smithi118.stderr:2023-10-05T19:27:42.575+0000 7f44bf59e700 1 --2- 172.21.15.118:0/2626045551 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f44c0091280 0x7f44c012b590 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7f44b4009860 tx=0x7f44b40091a0 comp rx=0 tx=0).stop 2023-10-05T19:27:42.584 INFO:teuthology.misc.health.smithi118.stderr:2023-10-05T19:27:42.575+0000 7f44bf59e700 1 -- 172.21.15.118:0/2626045551 shutdown_connections 2023-10-05T19:27:42.584 INFO:teuthology.misc.health.smithi118.stderr:2023-10-05T19:27:42.575+0000 7f44bf59e700 1 --2- 172.21.15.118:0/2626045551 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f44c0091280 0x7f44c012b590 unknown :-1 s=CLOSED pgs=31 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-05T19:27:42.584 INFO:teuthology.misc.health.smithi118.stderr:2023-10-05T19:27:42.575+0000 7f44bf59e700 1 -- 172.21.15.118:0/2626045551 >> 172.21.15.118:0/2626045551 conn(0x7f44c0094ca0 msgr2=0x7f44c00950a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-05T19:27:42.584 INFO:teuthology.misc.health.smithi118.stderr:2023-10-05T19:27:42.575+0000 7f44bf59e700 1 -- 172.21.15.118:0/2626045551 shutdown_connections 2023-10-05T19:27:42.584 INFO:teuthology.misc.health.smithi118.stderr:2023-10-05T19:27:42.575+0000 7f44bf59e700 1 -- 172.21.15.118:0/2626045551 wait complete. 2023-10-05T19:27:42.585 INFO:teuthology.misc.health.smithi118.stderr:2023-10-05T19:27:42.575+0000 7f44bf59e700 1 Processor -- start 2023-10-05T19:27:42.585 INFO:teuthology.misc.health.smithi118.stderr:2023-10-05T19:27:42.575+0000 7f44bf59e700 1 -- start start 2023-10-05T19:27:42.585 INFO:teuthology.misc.health.smithi118.stderr:2023-10-05T19:27:42.579+0000 7f44bf59e700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f44c00907a0 0x7f44c008b620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-05T19:27:42.585 INFO:teuthology.misc.health.smithi118.stderr:2023-10-05T19:27:42.579+0000 7f44bf59e700 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f44c006e340 con 0x7f44c00907a0 2023-10-05T19:27:42.585 INFO:teuthology.misc.health.smithi118.stderr:2023-10-05T19:27:42.579+0000 7f44be59c700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f44c00907a0 0x7f44c008b620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-05T19:27:42.585 INFO:teuthology.misc.health.smithi118.stderr:2023-10-05T19:27:42.579+0000 7f44be59c700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f44c00907a0 0x7f44c008b620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.118:3300/0 says I am v2:172.21.15.118:56548/0 (socket says 172.21.15.118:56548) 2023-10-05T19:27:42.586 INFO:teuthology.misc.health.smithi118.stderr:2023-10-05T19:27:42.579+0000 7f44be59c700 1 -- 172.21.15.118:0/849056191 learned_addr learned my addr 172.21.15.118:0/849056191 (peer_addr_for_me v2:172.21.15.118:0/0) 2023-10-05T19:27:42.586 INFO:teuthology.misc.health.smithi118.stderr:2023-10-05T19:27:42.579+0000 7f44be59c700 1 -- 172.21.15.118:0/849056191 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f44b4009420 con 0x7f44c00907a0 2023-10-05T19:27:42.586 INFO:teuthology.misc.health.smithi118.stderr:2023-10-05T19:27:42.579+0000 7f44be59c700 1 --2- 172.21.15.118:0/849056191 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f44c00907a0 0x7f44c008b620 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7f44b800cc00 tx=0x7f44b8001b00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-05T19:27:42.587 INFO:teuthology.misc.health.smithi118.stderr:2023-10-05T19:27:42.579+0000 7f44ae7fc700 1 -- 172.21.15.118:0/849056191 <== mon.0 v2:172.21.15.118:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f44b800a980 con 0x7f44c00907a0 2023-10-05T19:27:42.587 INFO:teuthology.misc.health.smithi118.stderr:2023-10-05T19:27:42.579+0000 7f44ae7fc700 1 -- 172.21.15.118:0/849056191 <== mon.0 v2:172.21.15.118:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f44b800aae0 con 0x7f44c00907a0 2023-10-05T19:27:42.587 INFO:teuthology.misc.health.smithi118.stderr:2023-10-05T19:27:42.579+0000 7f44bf59e700 1 -- 172.21.15.118:0/849056191 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f44c00900d0 con 0x7f44c00907a0 2023-10-05T19:27:42.587 INFO:teuthology.misc.health.smithi118.stderr:2023-10-05T19:27:42.579+0000 7f44ae7fc700 1 -- 172.21.15.118:0/849056191 <== mon.0 v2:172.21.15.118:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f44b800adb0 con 0x7f44c00907a0 2023-10-05T19:27:42.587 INFO:teuthology.misc.health.smithi118.stderr:2023-10-05T19:27:42.579+0000 7f44bf59e700 1 -- 172.21.15.118:0/849056191 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f44c008e150 con 0x7f44c00907a0 2023-10-05T19:27:42.587 INFO:teuthology.misc.health.smithi118.stderr:2023-10-05T19:27:42.579+0000 7f44ae7fc700 1 -- 172.21.15.118:0/849056191 <== mon.0 v2:172.21.15.118:3300/0 4 ==== mgrmap(e 1) v1 ==== 521+0+0 (secure 0 0 0) 0x7f44b80070f0 con 0x7f44c00907a0 2023-10-05T19:27:42.587 INFO:teuthology.misc.health.smithi118.stderr:2023-10-05T19:27:42.579+0000 7f44ae7fc700 1 -- 172.21.15.118:0/849056191 <== mon.0 v2:172.21.15.118:3300/0 5 ==== osd_map(6..6 src has 1..6) v4 ==== 1181+0+0 (secure 0 0 0) 0x7f44b801b910 con 0x7f44c00907a0 2023-10-05T19:27:42.587 INFO:teuthology.misc.health.smithi118.stderr:2023-10-05T19:27:42.579+0000 7f44bf59e700 1 -- 172.21.15.118:0/849056191 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f44c008d2f0 con 0x7f44c00907a0 2023-10-05T19:27:42.590 INFO:teuthology.misc.health.smithi118.stderr:2023-10-05T19:27:42.583+0000 7f44ae7fc700 1 -- 172.21.15.118:0/849056191 <== mon.0 v2:172.21.15.118:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+71325 (secure 0 0 0) 0x7f44b8007530 con 0x7f44c00907a0 2023-10-05T19:27:42.658 INFO:teuthology.misc.health.smithi118.stderr:2023-10-05T19:27:42.651+0000 7f44bf59e700 1 -- 172.21.15.118:0/849056191 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f44c005ecb0 con 0x7f44c00907a0 2023-10-05T19:27:42.659 INFO:teuthology.misc.health.smithi118.stderr:2023-10-05T19:27:42.651+0000 7f44ae7fc700 1 -- 172.21.15.118:0/849056191 <== mon.0 v2:172.21.15.118:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v6) v1 ==== 74+0+4108 (secure 0 0 0) 0x7f44b8007790 con 0x7f44c00907a0 2023-10-05T19:27:42.659 INFO:teuthology.misc.health.smithi118.stdout: 2023-10-05T19:27:42.659 INFO:teuthology.misc.health.smithi118.stdout:{"epoch":6,"fsid":"8c4bd0d1-cbcb-4713-96b7-dcb14bcaf584","created":"2023-10-05T19:27:40.755769+0000","modified":"2023-10-05T19:27:41.811702+0000","last_up_change":"0.000000","last_in_change":"2023-10-05T19:27:41.811702+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[],"osds":[{"osd":0,"uuid":"61ba9225-834e-4288-b391-7ffa540f7334","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"15a3631e-9f20-4b33-9ee3-39eb73e13709","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"5712158a-931c-4ce7-adf0-5c9fbb59d521","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"f838ed4e-05a7-49ef-ad9d-8f7dd7be30f1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-05T19:27:42.661 INFO:teuthology.misc.health.smithi118.stderr:2023-10-05T19:27:42.655+0000 7f44bf59e700 1 -- 172.21.15.118:0/849056191 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f44c00907a0 msgr2=0x7f44c008b620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-05T19:27:42.661 INFO:teuthology.misc.health.smithi118.stderr:2023-10-05T19:27:42.655+0000 7f44bf59e700 1 --2- 172.21.15.118:0/849056191 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f44c00907a0 0x7f44c008b620 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7f44b800cc00 tx=0x7f44b8001b00 comp rx=0 tx=0).stop 2023-10-05T19:27:42.661 INFO:teuthology.misc.health.smithi118.stderr:2023-10-05T19:27:42.655+0000 7f44bf59e700 1 -- 172.21.15.118:0/849056191 shutdown_connections 2023-10-05T19:27:42.661 INFO:teuthology.misc.health.smithi118.stderr:2023-10-05T19:27:42.655+0000 7f44bf59e700 1 --2- 172.21.15.118:0/849056191 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f44c00907a0 0x7f44c008b620 unknown :-1 s=CLOSED pgs=32 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-05T19:27:42.661 INFO:teuthology.misc.health.smithi118.stderr:2023-10-05T19:27:42.655+0000 7f44bf59e700 1 -- 172.21.15.118:0/849056191 >> 172.21.15.118:0/849056191 conn(0x7f44c0094ca0 msgr2=0x7f44c006cf90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-05T19:27:42.661 INFO:teuthology.misc.health.smithi118.stderr:2023-10-05T19:27:42.655+0000 7f44bf59e700 1 -- 172.21.15.118:0/849056191 shutdown_connections 2023-10-05T19:27:42.661 INFO:teuthology.misc.health.smithi118.stderr:2023-10-05T19:27:42.655+0000 7f44bf59e700 1 -- 172.21.15.118:0/849056191 wait complete. 2023-10-05T19:27:42.676 DEBUG:teuthology.misc:0 of 4 OSDs are up 2023-10-05T19:27:42.818 INFO:tasks.ceph.mgr.x.smithi167.stderr:2023-10-05T19:27:42.811+0000 7fb91b495000 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2023-10-05T19:27:42.930 INFO:tasks.ceph.mgr.x.smithi167.stderr:2023-10-05T19:27:42.923+0000 7fb91b495000 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2023-10-05T19:27:43.220 INFO:tasks.ceph.mgr.x.smithi167.stderr:2023-10-05T19:27:43.215+0000 7fb91b495000 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2023-10-05T19:27:43.382 INFO:tasks.ceph.mgr.x.smithi167.stderr:2023-10-05T19:27:43.379+0000 7fb91b495000 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2023-10-05T19:27:43.448 INFO:tasks.ceph.mgr.x.smithi167.stderr:2023-10-05T19:27:43.443+0000 7fb91b495000 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2023-10-05T19:27:43.500 INFO:tasks.ceph.mgr.x.smithi167.stderr:2023-10-05T19:27:43.495+0000 7fb91b495000 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2023-10-05T19:27:43.559 INFO:tasks.ceph.mgr.x.smithi167.stderr:2023-10-05T19:27:43.555+0000 7fb91b495000 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2023-10-05T19:27:43.832 INFO:tasks.ceph.mgr.x.smithi167.stderr:2023-10-05T19:27:43.827+0000 7fb91b495000 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2023-10-05T19:27:43.887 INFO:tasks.ceph.mgr.x.smithi167.stderr:2023-10-05T19:27:43.883+0000 7fb91b495000 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2023-10-05T19:27:43.952 INFO:tasks.ceph.mgr.x.smithi167.stderr:2023-10-05T19:27:43.947+0000 7fb91b495000 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2023-10-05T19:27:44.111 INFO:tasks.ceph.mgr.x.smithi167.stderr:2023-10-05T19:27:44.107+0000 7fb91b495000 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2023-10-05T19:27:44.177 INFO:tasks.ceph.mgr.x.smithi167.stderr:2023-10-05T19:27:44.171+0000 7fb91b495000 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2023-10-05T19:27:44.882 INFO:tasks.ceph.mgr.x.smithi167.stderr:2023-10-05T19:27:44.879+0000 7fb91b495000 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2023-10-05T19:27:45.918 INFO:tasks.ceph.mgr.x.smithi167.stderr:2023-10-05T19:27:45.915+0000 7fb90458d700 -1 mgr.server handle_report got status from non-daemon mon.a 2023-10-05T19:27:46.158 INFO:tasks.ceph.osd.0.smithi118.stderr:2023-10-05T19:27:46.151+0000 7f6eef56f3c0 -1 osd.0 0 log_to_monitors true 2023-10-05T19:27:46.327 INFO:tasks.ceph.osd.3.smithi167.stderr:2023-10-05T19:27:46.323+0000 7f6c015ca3c0 -1 osd.3 0 log_to_monitors true 2023-10-05T19:27:46.347 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-05T19:27:46.343+0000 7fbd287073c0 -1 osd.1 0 log_to_monitors true 2023-10-05T19:27:46.410 INFO:tasks.ceph.osd.2.smithi167.stderr:2023-10-05T19:27:46.407+0000 7f8d55fc83c0 -1 osd.2 0 log_to_monitors true 2023-10-05T19:27:49.079 DEBUG:teuthology.orchestra.run.smithi118:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2023-10-05T19:27:49.197 INFO:tasks.ceph.osd.0.smithi118.stderr:2023-10-05T19:27:49.191+0000 7f6eec534700 -1 osd.0 0 waiting for initial osdmap 2023-10-05T19:27:49.203 INFO:tasks.ceph.osd.0.smithi118.stderr:2023-10-05T19:27:49.195+0000 7f6ee3ab3700 -1 osd.0 8 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2023-10-05T19:27:49.207 INFO:teuthology.misc.health.smithi118.stderr:2023-10-05T19:27:49.199+0000 7f9720a35700 1 Processor -- start 2023-10-05T19:27:49.207 INFO:teuthology.misc.health.smithi118.stderr:2023-10-05T19:27:49.199+0000 7f9720a35700 1 -- start start 2023-10-05T19:27:49.207 INFO:teuthology.misc.health.smithi118.stderr:2023-10-05T19:27:49.199+0000 7f9720a35700 1 --2- >> v2:172.21.15.118:3300/0 conn(0x7f971c0912f0 0x7f971c12b6c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-05T19:27:49.208 INFO:teuthology.misc.health.smithi118.stderr:2023-10-05T19:27:49.199+0000 7f9720a35700 1 -- --> v1:172.21.15.118:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f971c096ca0 con 0x7f971c090810 2023-10-05T19:27:49.208 INFO:teuthology.misc.health.smithi118.stderr:2023-10-05T19:27:49.199+0000 7f9720a35700 1 -- --> v2:172.21.15.118:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f971c092a70 con 0x7f971c0912f0 2023-10-05T19:27:49.208 INFO:teuthology.misc.health.smithi118.stderr:2023-10-05T19:27:49.199+0000 7f971affd700 1 --2- >> v2:172.21.15.118:3300/0 conn(0x7f971c0912f0 0x7f971c12b6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-05T19:27:49.208 INFO:teuthology.misc.health.smithi118.stderr:2023-10-05T19:27:49.199+0000 7f971b7fe700 1 --1- >> v1:172.21.15.118:6789/0 conn(0x7f971c090810 0x7f971c090be0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.118:6789/0 says I am v1:172.21.15.118:34158/0 (socket says 172.21.15.118:34158) 2023-10-05T19:27:49.208 INFO:teuthology.misc.health.smithi118.stderr:2023-10-05T19:27:49.199+0000 7f971b7fe700 1 -- 172.21.15.118:0/2627211573 learned_addr learned my addr 172.21.15.118:0/2627211573 (peer_addr_for_me v1:172.21.15.118:0/0) 2023-10-05T19:27:49.208 INFO:teuthology.misc.health.smithi118.stderr:2023-10-05T19:27:49.199+0000 7f971affd700 1 -- 172.21.15.118:0/2627211573 >> v1:172.21.15.118:6789/0 conn(0x7f971c090810 legacy=0x7f971c090be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-05T19:27:49.208 INFO:teuthology.misc.health.smithi118.stderr:2023-10-05T19:27:49.199+0000 7f971a7fc700 1 -- 172.21.15.118:0/2627211573 <== mon.0 v1:172.21.15.118:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3551688439 0 0) 0x7f971c096ca0 con 0x7f971c090810 2023-10-05T19:27:49.208 INFO:teuthology.misc.health.smithi118.stderr:2023-10-05T19:27:49.199+0000 7f971affd700 1 -- 172.21.15.118:0/2627211573 --> v2:172.21.15.118:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f971c091ad0 con 0x7f971c0912f0 2023-10-05T19:27:49.208 INFO:teuthology.misc.health.smithi118.stderr:2023-10-05T19:27:49.199+0000 7f971affd700 1 --2- 172.21.15.118:0/2627211573 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f971c0912f0 0x7f971c12b6c0 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f970c009860 tx=0x7f970c0091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d5ed815edfde6e63 server_cookie=0 in_seq=0 out_seq=0 2023-10-05T19:27:49.208 INFO:teuthology.misc.health.smithi118.stderr:2023-10-05T19:27:49.199+0000 7f971a7fc700 1 -- 172.21.15.118:0/2627211573 <== mon.0 v2:172.21.15.118:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f970c00b680 con 0x7f971c0912f0 2023-10-05T19:27:49.208 INFO:teuthology.misc.health.smithi118.stderr:2023-10-05T19:27:49.199+0000 7f971a7fc700 1 -- 172.21.15.118:0/2627211573 <== mon.0 v2:172.21.15.118:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f970c00b7e0 con 0x7f971c0912f0 2023-10-05T19:27:49.208 INFO:teuthology.misc.health.smithi118.stderr:2023-10-05T19:27:49.199+0000 7f971a7fc700 1 -- 172.21.15.118:0/2627211573 <== mon.0 v2:172.21.15.118:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f970c00b990 con 0x7f971c0912f0 2023-10-05T19:27:49.210 INFO:teuthology.misc.health.smithi118.stderr:2023-10-05T19:27:49.199+0000 7f9720a35700 1 -- 172.21.15.118:0/2627211573 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f971c0912f0 msgr2=0x7f971c12b6c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-05T19:27:49.211 INFO:teuthology.misc.health.smithi118.stderr:2023-10-05T19:27:49.199+0000 7f9720a35700 1 --2- 172.21.15.118:0/2627211573 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f971c0912f0 0x7f971c12b6c0 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f970c009860 tx=0x7f970c0091a0 comp rx=0 tx=0).stop 2023-10-05T19:27:49.211 INFO:teuthology.misc.health.smithi118.stderr:2023-10-05T19:27:49.199+0000 7f9720a35700 1 -- 172.21.15.118:0/2627211573 shutdown_connections 2023-10-05T19:27:49.211 INFO:teuthology.misc.health.smithi118.stderr:2023-10-05T19:27:49.199+0000 7f9720a35700 1 --2- 172.21.15.118:0/2627211573 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f971c0912f0 0x7f971c12b6c0 unknown :-1 s=CLOSED pgs=45 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-05T19:27:49.211 INFO:teuthology.misc.health.smithi118.stderr:2023-10-05T19:27:49.199+0000 7f9720a35700 1 -- 172.21.15.118:0/2627211573 >> 172.21.15.118:0/2627211573 conn(0x7f971c096190 msgr2=0x7f971c096590 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-05T19:27:49.211 INFO:teuthology.misc.health.smithi118.stderr:2023-10-05T19:27:49.199+0000 7f9720a35700 1 -- 172.21.15.118:0/2627211573 shutdown_connections 2023-10-05T19:27:49.211 INFO:teuthology.misc.health.smithi118.stderr:2023-10-05T19:27:49.203+0000 7f9720a35700 1 -- 172.21.15.118:0/2627211573 wait complete. 2023-10-05T19:27:49.211 INFO:teuthology.misc.health.smithi118.stderr:2023-10-05T19:27:49.203+0000 7f9720a35700 1 Processor -- start 2023-10-05T19:27:49.211 INFO:teuthology.misc.health.smithi118.stderr:2023-10-05T19:27:49.203+0000 7f9720a35700 1 -- start start 2023-10-05T19:27:49.211 INFO:teuthology.misc.health.smithi118.stderr:2023-10-05T19:27:49.203+0000 7f9720a35700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f971c090810 0x7f971c08c920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-05T19:27:49.211 INFO:teuthology.misc.health.smithi118.stderr:2023-10-05T19:27:49.203+0000 7f9720a35700 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f971c08ac50 con 0x7f971c090810 2023-10-05T19:27:49.211 INFO:teuthology.misc.health.smithi118.stderr:2023-10-05T19:27:49.203+0000 7f971b7fe700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f971c090810 0x7f971c08c920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-05T19:27:49.211 INFO:teuthology.misc.health.smithi118.stderr:2023-10-05T19:27:49.203+0000 7f971b7fe700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f971c090810 0x7f971c08c920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.118:3300/0 says I am v2:172.21.15.118:50112/0 (socket says 172.21.15.118:50112) 2023-10-05T19:27:49.212 INFO:teuthology.misc.health.smithi118.stderr:2023-10-05T19:27:49.203+0000 7f971b7fe700 1 -- 172.21.15.118:0/3344819870 learned_addr learned my addr 172.21.15.118:0/3344819870 (peer_addr_for_me v2:172.21.15.118:0/0) 2023-10-05T19:27:49.212 INFO:teuthology.misc.health.smithi118.stderr:2023-10-05T19:27:49.203+0000 7f971b7fe700 1 -- 172.21.15.118:0/3344819870 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f970c009420 con 0x7f971c090810 2023-10-05T19:27:49.212 INFO:teuthology.misc.health.smithi118.stderr:2023-10-05T19:27:49.203+0000 7f971b7fe700 1 --2- 172.21.15.118:0/3344819870 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f971c090810 0x7f971c08c920 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f971000c710 tx=0x7f9710007d90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-05T19:27:49.212 INFO:teuthology.misc.health.smithi118.stderr:2023-10-05T19:27:49.203+0000 7f97037fe700 1 -- 172.21.15.118:0/3344819870 <== mon.0 v2:172.21.15.118:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f9710005d90 con 0x7f971c090810 2023-10-05T19:27:49.212 INFO:teuthology.misc.health.smithi118.stderr:2023-10-05T19:27:49.203+0000 7f97037fe700 1 -- 172.21.15.118:0/3344819870 <== mon.0 v2:172.21.15.118:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f97100070d0 con 0x7f971c090810 2023-10-05T19:27:49.212 INFO:teuthology.misc.health.smithi118.stderr:2023-10-05T19:27:49.203+0000 7f9720a35700 1 -- 172.21.15.118:0/3344819870 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f971c089970 con 0x7f971c090810 2023-10-05T19:27:49.212 INFO:teuthology.misc.health.smithi118.stderr:2023-10-05T19:27:49.203+0000 7f97037fe700 1 -- 172.21.15.118:0/3344819870 <== mon.0 v2:172.21.15.118:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f97100073c0 con 0x7f971c090810 2023-10-05T19:27:49.212 INFO:teuthology.misc.health.smithi118.stderr:2023-10-05T19:27:49.203+0000 7f9720a35700 1 -- 172.21.15.118:0/3344819870 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f971c08ce60 con 0x7f971c090810 2023-10-05T19:27:49.213 INFO:teuthology.misc.health.smithi118.stderr:2023-10-05T19:27:49.203+0000 7f9720a35700 1 -- 172.21.15.118:0/3344819870 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f971c08a810 con 0x7f971c090810 2023-10-05T19:27:49.213 INFO:teuthology.misc.health.smithi118.stderr:2023-10-05T19:27:49.203+0000 7f97037fe700 1 -- 172.21.15.118:0/3344819870 <== mon.0 v2:172.21.15.118:3300/0 4 ==== mgrmap(e 4) v1 ==== 40273+0+0 (secure 0 0 0) 0x7f9710007520 con 0x7f971c090810 2023-10-05T19:27:49.213 INFO:teuthology.misc.health.smithi118.stderr:2023-10-05T19:27:49.203+0000 7f97037fe700 1 --2- 172.21.15.118:0/3344819870 >> [v2:172.21.15.167:6824/29889,v1:172.21.15.167:6825/29889] conn(0x7f9708032350 0x7f9708034800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-05T19:27:49.213 INFO:teuthology.misc.health.smithi118.stderr:2023-10-05T19:27:49.203+0000 7f97037fe700 1 -- 172.21.15.118:0/3344819870 <== mon.0 v2:172.21.15.118:3300/0 5 ==== osd_map(8..8 src has 1..8) v4 ==== 1619+0+0 (secure 0 0 0) 0x7f9710042780 con 0x7f971c090810 2023-10-05T19:27:49.213 INFO:teuthology.misc.health.smithi118.stderr:2023-10-05T19:27:49.203+0000 7f971affd700 1 --2- 172.21.15.118:0/3344819870 >> [v2:172.21.15.167:6824/29889,v1:172.21.15.167:6825/29889] conn(0x7f9708032350 0x7f9708034800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-05T19:27:49.214 INFO:teuthology.misc.health.smithi118.stderr:2023-10-05T19:27:49.207+0000 7f971affd700 1 --2- 172.21.15.118:0/3344819870 >> [v2:172.21.15.167:6824/29889,v1:172.21.15.167:6825/29889] conn(0x7f9708032350 0x7f9708034800 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7f970c0030c0 tx=0x7f970c016000 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-05T19:27:49.217 INFO:teuthology.misc.health.smithi118.stderr:2023-10-05T19:27:49.211+0000 7f97037fe700 1 -- 172.21.15.118:0/3344819870 <== mon.0 v2:172.21.15.118:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166807 (secure 0 0 0) 0x7f971c08a810 con 0x7f971c090810 2023-10-05T19:27:49.376 INFO:teuthology.misc.health.smithi118.stderr:2023-10-05T19:27:49.367+0000 7f9720a35700 1 -- 172.21.15.118:0/3344819870 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f971c08a810 con 0x7f971c090810 2023-10-05T19:27:49.377 INFO:teuthology.misc.health.smithi118.stderr:2023-10-05T19:27:49.371+0000 7f97037fe700 1 -- 172.21.15.118:0/3344819870 <== mon.0 v2:172.21.15.118:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v8) v1 ==== 74+0+4108 (secure 0 0 0) 0x7f971c08a810 con 0x7f971c090810 2023-10-05T19:27:49.377 INFO:teuthology.misc.health.smithi118.stdout: 2023-10-05T19:27:49.377 INFO:teuthology.misc.health.smithi118.stdout:{"epoch":8,"fsid":"8c4bd0d1-cbcb-4713-96b7-dcb14bcaf584","created":"2023-10-05T19:27:40.755769+0000","modified":"2023-10-05T19:27:47.970219+0000","last_up_change":"0.000000","last_in_change":"2023-10-05T19:27:41.811702+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[],"osds":[{"osd":0,"uuid":"61ba9225-834e-4288-b391-7ffa540f7334","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"15a3631e-9f20-4b33-9ee3-39eb73e13709","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"5712158a-931c-4ce7-adf0-5c9fbb59d521","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"f838ed4e-05a7-49ef-ad9d-8f7dd7be30f1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-05T19:27:49.379 INFO:teuthology.misc.health.smithi118.stderr:2023-10-05T19:27:49.371+0000 7f9720a35700 1 -- 172.21.15.118:0/3344819870 >> [v2:172.21.15.167:6824/29889,v1:172.21.15.167:6825/29889] conn(0x7f9708032350 msgr2=0x7f9708034800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-05T19:27:49.379 INFO:teuthology.misc.health.smithi118.stderr:2023-10-05T19:27:49.371+0000 7f9720a35700 1 --2- 172.21.15.118:0/3344819870 >> [v2:172.21.15.167:6824/29889,v1:172.21.15.167:6825/29889] conn(0x7f9708032350 0x7f9708034800 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7f970c0030c0 tx=0x7f970c016000 comp rx=0 tx=0).stop 2023-10-05T19:27:49.379 INFO:teuthology.misc.health.smithi118.stderr:2023-10-05T19:27:49.371+0000 7f9720a35700 1 -- 172.21.15.118:0/3344819870 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f971c090810 msgr2=0x7f971c08c920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-05T19:27:49.379 INFO:teuthology.misc.health.smithi118.stderr:2023-10-05T19:27:49.371+0000 7f9720a35700 1 --2- 172.21.15.118:0/3344819870 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f971c090810 0x7f971c08c920 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f971000c710 tx=0x7f9710007d90 comp rx=0 tx=0).stop 2023-10-05T19:27:49.379 INFO:teuthology.misc.health.smithi118.stderr:2023-10-05T19:27:49.371+0000 7f9720a35700 1 -- 172.21.15.118:0/3344819870 shutdown_connections 2023-10-05T19:27:49.379 INFO:teuthology.misc.health.smithi118.stderr:2023-10-05T19:27:49.371+0000 7f9720a35700 1 --2- 172.21.15.118:0/3344819870 >> [v2:172.21.15.167:6824/29889,v1:172.21.15.167:6825/29889] conn(0x7f9708032350 0x7f9708034800 unknown :-1 s=CLOSED pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-05T19:27:49.380 INFO:teuthology.misc.health.smithi118.stderr:2023-10-05T19:27:49.371+0000 7f9720a35700 1 --2- 172.21.15.118:0/3344819870 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f971c090810 0x7f971c08c920 unknown :-1 s=CLOSED pgs=46 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-05T19:27:49.380 INFO:teuthology.misc.health.smithi118.stderr:2023-10-05T19:27:49.371+0000 7f9720a35700 1 -- 172.21.15.118:0/3344819870 >> 172.21.15.118:0/3344819870 conn(0x7f971c096190 msgr2=0x7f971c097df0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-05T19:27:49.380 INFO:teuthology.misc.health.smithi118.stderr:2023-10-05T19:27:49.371+0000 7f9720a35700 1 -- 172.21.15.118:0/3344819870 shutdown_connections 2023-10-05T19:27:49.380 INFO:teuthology.misc.health.smithi118.stderr:2023-10-05T19:27:49.371+0000 7f9720a35700 1 -- 172.21.15.118:0/3344819870 wait complete. 2023-10-05T19:27:49.389 DEBUG:teuthology.misc:0 of 4 OSDs are up 2023-10-05T19:27:49.901 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-05T19:27:49.895+0000 7fbd256cc700 -1 osd.1 0 waiting for initial osdmap 2023-10-05T19:27:49.909 INFO:tasks.ceph.osd.1.smithi167.stderr:2023-10-05T19:27:49.903+0000 7fbd1cc4b700 -1 osd.1 8 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2023-10-05T19:27:49.927 INFO:tasks.ceph.osd.3.smithi167.stderr:2023-10-05T19:27:49.923+0000 7f6bfcd6a700 -1 osd.3 0 waiting for initial osdmap 2023-10-05T19:27:49.935 INFO:tasks.ceph.osd.2.smithi167.stderr:2023-10-05T19:27:49.931+0000 7f8d51768700 -1 osd.2 0 waiting for initial osdmap 2023-10-05T19:27:49.936 INFO:tasks.ceph.osd.3.smithi167.stderr:2023-10-05T19:27:49.931+0000 7f6bf5b0e700 -1 osd.3 8 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2023-10-05T19:27:49.941 INFO:tasks.ceph.osd.2.smithi167.stderr:2023-10-05T19:27:49.935+0000 7f8d4a50c700 -1 osd.2 8 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2023-10-05T19:27:55.793 DEBUG:teuthology.orchestra.run.smithi118:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2023-10-05T19:27:55.917 INFO:teuthology.misc.health.smithi118.stderr:2023-10-05T19:27:55.911+0000 7fb4cf741700 1 Processor -- start 2023-10-05T19:27:55.917 INFO:teuthology.misc.health.smithi118.stderr:2023-10-05T19:27:55.911+0000 7fb4cf741700 1 -- start start 2023-10-05T19:27:55.918 INFO:teuthology.misc.health.smithi118.stderr:2023-10-05T19:27:55.911+0000 7fb4cf741700 1 --2- >> v2:172.21.15.118:3300/0 conn(0x7fb4c808a410 0x7fb4c812b730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-05T19:27:55.918 INFO:teuthology.misc.health.smithi118.stderr:2023-10-05T19:27:55.911+0000 7fb4cf741700 1 -- --> v1:172.21.15.118:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb4c806a630 con 0x7fb4c8089930 2023-10-05T19:27:55.918 INFO:teuthology.misc.health.smithi118.stderr:2023-10-05T19:27:55.911+0000 7fb4cf741700 1 -- --> v2:172.21.15.118:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb4c808ac20 con 0x7fb4c808a410 2023-10-05T19:27:55.918 INFO:teuthology.misc.health.smithi118.stderr:2023-10-05T19:27:55.911+0000 7fb4cccdc700 1 --2- >> v2:172.21.15.118:3300/0 conn(0x7fb4c808a410 0x7fb4c812b730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-05T19:27:55.918 INFO:teuthology.misc.health.smithi118.stderr:2023-10-05T19:27:55.911+0000 7fb4cccdc700 1 --2- >> v2:172.21.15.118:3300/0 conn(0x7fb4c808a410 0x7fb4c812b730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.118:3300/0 says I am v2:172.21.15.118:50122/0 (socket says 172.21.15.118:50122) 2023-10-05T19:27:55.918 INFO:teuthology.misc.health.smithi118.stderr:2023-10-05T19:27:55.911+0000 7fb4cccdc700 1 -- 172.21.15.118:0/1547526161 learned_addr learned my addr 172.21.15.118:0/1547526161 (peer_addr_for_me v2:172.21.15.118:0/0) 2023-10-05T19:27:55.918 INFO:teuthology.misc.health.smithi118.stderr:2023-10-05T19:27:55.911+0000 7fb4cccdc700 1 -- 172.21.15.118:0/1547526161 >> v1:172.21.15.118:6789/0 conn(0x7fb4c8089930 legacy=0x7fb4c8089d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-05T19:27:55.918 INFO:teuthology.misc.health.smithi118.stderr:2023-10-05T19:27:55.911+0000 7fb4cccdc700 1 -- 172.21.15.118:0/1547526161 --> v2:172.21.15.118:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb4c806a300 con 0x7fb4c808a410 2023-10-05T19:27:55.919 INFO:teuthology.misc.health.smithi118.stderr:2023-10-05T19:27:55.911+0000 7fb4cccdc700 1 --2- 172.21.15.118:0/1547526161 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb4c808a410 0x7fb4c812b730 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7fb4b8009930 tx=0x7fb4b8009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=33e9aef616152da8 server_cookie=0 in_seq=0 out_seq=0 2023-10-05T19:27:55.920 INFO:teuthology.misc.health.smithi118.stderr:2023-10-05T19:27:55.911+0000 7fb4bffff700 1 -- 172.21.15.118:0/1547526161 <== mon.0 v2:172.21.15.118:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fb4b800b860 con 0x7fb4c808a410 2023-10-05T19:27:55.920 INFO:teuthology.misc.health.smithi118.stderr:2023-10-05T19:27:55.911+0000 7fb4bffff700 1 -- 172.21.15.118:0/1547526161 <== mon.0 v2:172.21.15.118:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb4b800b9c0 con 0x7fb4c808a410 2023-10-05T19:27:55.920 INFO:teuthology.misc.health.smithi118.stderr:2023-10-05T19:27:55.911+0000 7fb4bffff700 1 -- 172.21.15.118:0/1547526161 <== mon.0 v2:172.21.15.118:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fb4b800bb50 con 0x7fb4c808a410 2023-10-05T19:27:55.920 INFO:teuthology.misc.health.smithi118.stderr:2023-10-05T19:27:55.911+0000 7fb4cf741700 1 -- 172.21.15.118:0/1547526161 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb4c808a410 msgr2=0x7fb4c812b730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-05T19:27:55.920 INFO:teuthology.misc.health.smithi118.stderr:2023-10-05T19:27:55.911+0000 7fb4cf741700 1 --2- 172.21.15.118:0/1547526161 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb4c808a410 0x7fb4c812b730 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7fb4b8009930 tx=0x7fb4b8009250 comp rx=0 tx=0).stop 2023-10-05T19:27:55.920 INFO:teuthology.misc.health.smithi118.stderr:2023-10-05T19:27:55.911+0000 7fb4cf741700 1 -- 172.21.15.118:0/1547526161 shutdown_connections 2023-10-05T19:27:55.920 INFO:teuthology.misc.health.smithi118.stderr:2023-10-05T19:27:55.911+0000 7fb4cf741700 1 --2- 172.21.15.118:0/1547526161 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb4c808a410 0x7fb4c812b730 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-05T19:27:55.920 INFO:teuthology.misc.health.smithi118.stderr:2023-10-05T19:27:55.911+0000 7fb4cf741700 1 -- 172.21.15.118:0/1547526161 >> 172.21.15.118:0/1547526161 conn(0x7fb4c80964a0 msgr2=0x7fb4c80968a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-05T19:27:55.920 INFO:teuthology.misc.health.smithi118.stderr:2023-10-05T19:27:55.911+0000 7fb4cf741700 1 -- 172.21.15.118:0/1547526161 shutdown_connections 2023-10-05T19:27:55.920 INFO:teuthology.misc.health.smithi118.stderr:2023-10-05T19:27:55.911+0000 7fb4cf741700 1 -- 172.21.15.118:0/1547526161 wait complete. 2023-10-05T19:27:55.921 INFO:teuthology.misc.health.smithi118.stderr:2023-10-05T19:27:55.915+0000 7fb4cf741700 1 Processor -- start 2023-10-05T19:27:55.921 INFO:teuthology.misc.health.smithi118.stderr:2023-10-05T19:27:55.915+0000 7fb4cf741700 1 -- start start 2023-10-05T19:27:55.923 INFO:teuthology.misc.health.smithi118.stderr:2023-10-05T19:27:55.915+0000 7fb4cf741700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb4c8089930 0x7fb4c80947d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-05T19:27:55.923 INFO:teuthology.misc.health.smithi118.stderr:2023-10-05T19:27:55.915+0000 7fb4cf741700 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb4c808a800 con 0x7fb4c8089930 2023-10-05T19:27:55.923 INFO:teuthology.misc.health.smithi118.stderr:2023-10-05T19:27:55.915+0000 7fb4cd4dd700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb4c8089930 0x7fb4c80947d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-05T19:27:55.923 INFO:teuthology.misc.health.smithi118.stderr:2023-10-05T19:27:55.915+0000 7fb4cd4dd700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb4c8089930 0x7fb4c80947d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.118:3300/0 says I am v2:172.21.15.118:50128/0 (socket says 172.21.15.118:50128) 2023-10-05T19:27:55.923 INFO:teuthology.misc.health.smithi118.stderr:2023-10-05T19:27:55.915+0000 7fb4cd4dd700 1 -- 172.21.15.118:0/2096882974 learned_addr learned my addr 172.21.15.118:0/2096882974 (peer_addr_for_me v2:172.21.15.118:0/0) 2023-10-05T19:27:55.923 INFO:teuthology.misc.health.smithi118.stderr:2023-10-05T19:27:55.915+0000 7fb4cd4dd700 1 -- 172.21.15.118:0/2096882974 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb4b80094f0 con 0x7fb4c8089930 2023-10-05T19:27:55.924 INFO:teuthology.misc.health.smithi118.stderr:2023-10-05T19:27:55.915+0000 7fb4cd4dd700 1 --2- 172.21.15.118:0/2096882974 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb4c8089930 0x7fb4c80947d0 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7fb4c400ca40 tx=0x7fb4c4001770 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-05T19:27:55.924 INFO:teuthology.misc.health.smithi118.stderr:2023-10-05T19:27:55.915+0000 7fb4bd7fa700 1 -- 172.21.15.118:0/2096882974 <== mon.0 v2:172.21.15.118:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fb4c4004120 con 0x7fb4c8089930 2023-10-05T19:27:55.924 INFO:teuthology.misc.health.smithi118.stderr:2023-10-05T19:27:55.915+0000 7fb4bd7fa700 1 -- 172.21.15.118:0/2096882974 <== mon.0 v2:172.21.15.118:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb4c400a7e0 con 0x7fb4c8089930 2023-10-05T19:27:55.925 INFO:teuthology.misc.health.smithi118.stderr:2023-10-05T19:27:55.919+0000 7fb4cf741700 1 -- 172.21.15.118:0/2096882974 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb4c806e170 con 0x7fb4c8089930 2023-10-05T19:27:55.925 INFO:teuthology.misc.health.smithi118.stderr:2023-10-05T19:27:55.919+0000 7fb4bd7fa700 1 -- 172.21.15.118:0/2096882974 <== mon.0 v2:172.21.15.118:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fb4c400aad0 con 0x7fb4c8089930 2023-10-05T19:27:55.925 INFO:teuthology.misc.health.smithi118.stderr:2023-10-05T19:27:55.919+0000 7fb4cf741700 1 -- 172.21.15.118:0/2096882974 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb4c811e680 con 0x7fb4c8089930 2023-10-05T19:27:55.925 INFO:teuthology.misc.health.smithi118.stderr:2023-10-05T19:27:55.919+0000 7fb4bd7fa700 1 -- 172.21.15.118:0/2096882974 <== mon.0 v2:172.21.15.118:3300/0 4 ==== mgrmap(e 5) v1 ==== 40329+0+0 (secure 0 0 0) 0x7fb4c400acc0 con 0x7fb4c8089930 2023-10-05T19:27:55.925 INFO:teuthology.misc.health.smithi118.stderr:2023-10-05T19:27:55.919+0000 7fb4cf741700 1 -- 172.21.15.118:0/2096882974 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb4c805ecb0 con 0x7fb4c8089930 2023-10-05T19:27:55.926 INFO:teuthology.misc.health.smithi118.stderr:2023-10-05T19:27:55.919+0000 7fb4bd7fa700 1 --2- 172.21.15.118:0/2096882974 >> [v2:172.21.15.167:6824/29889,v1:172.21.15.167:6825/29889] conn(0x7fb4ac032390 0x7fb4ac034840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-05T19:27:55.926 INFO:teuthology.misc.health.smithi118.stderr:2023-10-05T19:27:55.919+0000 7fb4bd7fa700 1 -- 172.21.15.118:0/2096882974 <== mon.0 v2:172.21.15.118:3300/0 5 ==== osd_map(12..12 src has 1..12) v4 ==== 3125+0+0 (secure 0 0 0) 0x7fb4c40431c0 con 0x7fb4c8089930 2023-10-05T19:27:55.926 INFO:teuthology.misc.health.smithi118.stderr:2023-10-05T19:27:55.919+0000 7fb4cccdc700 1 --2- 172.21.15.118:0/2096882974 >> [v2:172.21.15.167:6824/29889,v1:172.21.15.167:6825/29889] conn(0x7fb4ac032390 0x7fb4ac034840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-05T19:27:55.927 INFO:teuthology.misc.health.smithi118.stderr:2023-10-05T19:27:55.919+0000 7fb4cccdc700 1 --2- 172.21.15.118:0/2096882974 >> [v2:172.21.15.167:6824/29889,v1:172.21.15.167:6825/29889] conn(0x7fb4ac032390 0x7fb4ac034840 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7fb4b8009930 tx=0x7fb4b8016000 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-05T19:27:55.934 INFO:teuthology.misc.health.smithi118.stderr:2023-10-05T19:27:55.927+0000 7fb4bd7fa700 1 -- 172.21.15.118:0/2096882974 <== mon.0 v2:172.21.15.118:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166807 (secure 0 0 0) 0x7fb4c400fe20 con 0x7fb4c8089930 2023-10-05T19:27:56.096 INFO:teuthology.misc.health.smithi118.stderr:2023-10-05T19:27:56.091+0000 7fb4cf741700 1 -- 172.21.15.118:0/2096882974 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fb4c80695e0 con 0x7fb4c8089930 2023-10-05T19:27:56.098 INFO:teuthology.misc.health.smithi118.stderr:2023-10-05T19:27:56.091+0000 7fb4bd7fa700 1 -- 172.21.15.118:0/2096882974 <== mon.0 v2:172.21.15.118:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v12) v1 ==== 74+0+7471 (secure 0 0 0) 0x7fb4c4017020 con 0x7fb4c8089930 2023-10-05T19:27:56.098 INFO:teuthology.misc.health.smithi118.stdout: 2023-10-05T19:27:56.098 INFO:teuthology.misc.health.smithi118.stdout:{"epoch":12,"fsid":"8c4bd0d1-cbcb-4713-96b7-dcb14bcaf584","created":"2023-10-05T19:27:40.755769+0000","modified":"2023-10-05T19:27:53.201158+0000","last_up_change":"2023-10-05T19:27:50.196733+0000","last_in_change":"2023-10-05T19:27:41.811702+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-05T19:27:50.980612+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"12","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}}],"osds":[{"osd":0,"uuid":"61ba9225-834e-4288-b391-7ffa540f7334","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6800","nonce":27690},{"type":"v1","addr":"172.21.15.118:6801","nonce":27690}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6802","nonce":27690},{"type":"v1","addr":"172.21.15.118:6803","nonce":27690}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6806","nonce":27690},{"type":"v1","addr":"172.21.15.118:6807","nonce":27690}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6804","nonce":27690},{"type":"v1","addr":"172.21.15.118:6805","nonce":27690}]},"public_addr":"172.21.15.118:6801/27690","cluster_addr":"172.21.15.118:6803/27690","heartbeat_back_addr":"172.21.15.118:6807/27690","heartbeat_front_addr":"172.21.15.118:6805/27690","state":["exists","up"]},{"osd":1,"uuid":"15a3631e-9f20-4b33-9ee3-39eb73e13709","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":10,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6800","nonce":30035},{"type":"v1","addr":"172.21.15.167:6801","nonce":30035}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6802","nonce":30035},{"type":"v1","addr":"172.21.15.167:6803","nonce":30035}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6806","nonce":30035},{"type":"v1","addr":"172.21.15.167:6807","nonce":30035}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6804","nonce":30035},{"type":"v1","addr":"172.21.15.167:6805","nonce":30035}]},"public_addr":"172.21.15.167:6801/30035","cluster_addr":"172.21.15.167:6803/30035","heartbeat_back_addr":"172.21.15.167:6807/30035","heartbeat_front_addr":"172.21.15.167:6805/30035","state":["exists","up"]},{"osd":2,"uuid":"5712158a-931c-4ce7-adf0-5c9fbb59d521","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6816","nonce":30037},{"type":"v1","addr":"172.21.15.167:6817","nonce":30037}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6818","nonce":30037},{"type":"v1","addr":"172.21.15.167:6819","nonce":30037}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6822","nonce":30037},{"type":"v1","addr":"172.21.15.167:6823","nonce":30037}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6820","nonce":30037},{"type":"v1","addr":"172.21.15.167:6821","nonce":30037}]},"public_addr":"172.21.15.167:6817/30037","cluster_addr":"172.21.15.167:6819/30037","heartbeat_back_addr":"172.21.15.167:6823/30037","heartbeat_front_addr":"172.21.15.167:6821/30037","state":["exists","up"]},{"osd":3,"uuid":"f838ed4e-05a7-49ef-ad9d-8f7dd7be30f1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6808","nonce":30036},{"type":"v1","addr":"172.21.15.167:6809","nonce":30036}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6810","nonce":30036},{"type":"v1","addr":"172.21.15.167:6811","nonce":30036}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6814","nonce":30036},{"type":"v1","addr":"172.21.15.167:6815","nonce":30036}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6812","nonce":30036},{"type":"v1","addr":"172.21.15.167:6813","nonce":30036}]},"public_addr":"172.21.15.167:6809/30036","cluster_addr":"172.21.15.167:6811/30036","heartbeat_back_addr":"172.21.15.167:6815/30036","heartbeat_front_addr":"172.21.15.167:6813/30036","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-05T19:27:47.190191+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-05T19:27:47.302630+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-05T19:27:47.374216+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-05T19:27:47.350455+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-05T19:27:56.100 INFO:teuthology.misc.health.smithi118.stderr:2023-10-05T19:27:56.091+0000 7fb4cf741700 1 -- 172.21.15.118:0/2096882974 >> [v2:172.21.15.167:6824/29889,v1:172.21.15.167:6825/29889] conn(0x7fb4ac032390 msgr2=0x7fb4ac034840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-05T19:27:56.100 INFO:teuthology.misc.health.smithi118.stderr:2023-10-05T19:27:56.091+0000 7fb4cf741700 1 --2- 172.21.15.118:0/2096882974 >> [v2:172.21.15.167:6824/29889,v1:172.21.15.167:6825/29889] conn(0x7fb4ac032390 0x7fb4ac034840 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7fb4b8009930 tx=0x7fb4b8016000 comp rx=0 tx=0).stop 2023-10-05T19:27:56.100 INFO:teuthology.misc.health.smithi118.stderr:2023-10-05T19:27:56.091+0000 7fb4cf741700 1 -- 172.21.15.118:0/2096882974 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb4c8089930 msgr2=0x7fb4c80947d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-05T19:27:56.100 INFO:teuthology.misc.health.smithi118.stderr:2023-10-05T19:27:56.091+0000 7fb4cf741700 1 --2- 172.21.15.118:0/2096882974 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb4c8089930 0x7fb4c80947d0 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7fb4c400ca40 tx=0x7fb4c4001770 comp rx=0 tx=0).stop 2023-10-05T19:27:56.100 INFO:teuthology.misc.health.smithi118.stderr:2023-10-05T19:27:56.091+0000 7fb4cf741700 1 -- 172.21.15.118:0/2096882974 shutdown_connections 2023-10-05T19:27:56.100 INFO:teuthology.misc.health.smithi118.stderr:2023-10-05T19:27:56.091+0000 7fb4cf741700 1 --2- 172.21.15.118:0/2096882974 >> [v2:172.21.15.167:6824/29889,v1:172.21.15.167:6825/29889] conn(0x7fb4ac032390 0x7fb4ac034840 unknown :-1 s=CLOSED pgs=24 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-05T19:27:56.100 INFO:teuthology.misc.health.smithi118.stderr:2023-10-05T19:27:56.091+0000 7fb4cf741700 1 --2- 172.21.15.118:0/2096882974 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb4c8089930 0x7fb4c80947d0 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-05T19:27:56.101 INFO:teuthology.misc.health.smithi118.stderr:2023-10-05T19:27:56.091+0000 7fb4cf741700 1 -- 172.21.15.118:0/2096882974 >> 172.21.15.118:0/2096882974 conn(0x7fb4c80964a0 msgr2=0x7fb4c8086d30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-05T19:27:56.101 INFO:teuthology.misc.health.smithi118.stderr:2023-10-05T19:27:56.091+0000 7fb4cf741700 1 -- 172.21.15.118:0/2096882974 shutdown_connections 2023-10-05T19:27:56.101 INFO:teuthology.misc.health.smithi118.stderr:2023-10-05T19:27:56.095+0000 7fb4cf741700 1 -- 172.21.15.118:0/2096882974 wait complete. 2023-10-05T19:27:56.110 DEBUG:teuthology.misc:4 of 4 OSDs are up 2023-10-05T19:27:56.110 INFO:tasks.ceph:Creating RBD pool 2023-10-05T19:27:56.110 DEBUG:teuthology.orchestra.run.smithi118:> sudo ceph --cluster ceph osd pool create rbd 8 2023-10-05T19:27:56.214 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-05T19:27:56.207+0000 7f7bd253c700 1 Processor -- start 2023-10-05T19:27:56.214 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-05T19:27:56.207+0000 7f7bd253c700 1 -- start start 2023-10-05T19:27:56.215 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-05T19:27:56.207+0000 7f7bd253c700 1 --2- >> v2:172.21.15.118:3300/0 conn(0x7f7bcc093b00 0x7f7bcc093ed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-05T19:27:56.215 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-05T19:27:56.207+0000 7f7bd253c700 1 -- --> v1:172.21.15.118:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7bcc096080 con 0x7f7bcc094410 2023-10-05T19:27:56.215 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-05T19:27:56.207+0000 7f7bd253c700 1 -- --> v2:172.21.15.118:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7bcc068d50 con 0x7f7bcc093b00 2023-10-05T19:27:56.215 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-05T19:27:56.207+0000 7f7bd0d39700 1 --1- >> v1:172.21.15.118:6789/0 conn(0x7f7bcc094410 0x7f7bcc129770 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.118:6789/0 says I am v1:172.21.15.118:34174/0 (socket says 172.21.15.118:34174) 2023-10-05T19:27:56.215 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-05T19:27:56.207+0000 7f7bd153a700 1 --2- >> v2:172.21.15.118:3300/0 conn(0x7f7bcc093b00 0x7f7bcc093ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-05T19:27:56.216 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-05T19:27:56.207+0000 7f7bd0d39700 1 -- 172.21.15.118:0/377039622 learned_addr learned my addr 172.21.15.118:0/377039622 (peer_addr_for_me v1:172.21.15.118:0/0) 2023-10-05T19:27:56.216 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-05T19:27:56.207+0000 7f7bc3fff700 1 -- 172.21.15.118:0/377039622 <== mon.0 v1:172.21.15.118:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 569070308 0 0) 0x7f7bcc096080 con 0x7f7bcc094410 2023-10-05T19:27:56.217 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-05T19:27:56.207+0000 7f7bc3fff700 1 -- 172.21.15.118:0/377039622 --> v1:172.21.15.118:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7bb8003570 con 0x7f7bcc094410 2023-10-05T19:27:56.217 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-05T19:27:56.207+0000 7f7bd153a700 1 -- 172.21.15.118:0/377039622 >> v1:172.21.15.118:6789/0 conn(0x7f7bcc094410 legacy=0x7f7bcc129770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-05T19:27:56.217 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-05T19:27:56.207+0000 7f7bd153a700 1 -- 172.21.15.118:0/377039622 --> v2:172.21.15.118:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7bcc095060 con 0x7f7bcc093b00 2023-10-05T19:27:56.217 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-05T19:27:56.207+0000 7f7bd153a700 1 --2- 172.21.15.118:0/377039622 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f7bcc093b00 0x7f7bcc093ed0 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f7bc8009860 tx=0x7f7bc80091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a88c0f6964622b13 server_cookie=0 in_seq=0 out_seq=0 2023-10-05T19:27:56.217 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-05T19:27:56.207+0000 7f7bc3fff700 1 -- 172.21.15.118:0/377039622 <== mon.0 v2:172.21.15.118:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f7bc8011630 con 0x7f7bcc093b00 2023-10-05T19:27:56.217 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-05T19:27:56.211+0000 7f7bc3fff700 1 -- 172.21.15.118:0/377039622 <== mon.0 v2:172.21.15.118:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7bc8011790 con 0x7f7bcc093b00 2023-10-05T19:27:56.217 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-05T19:27:56.211+0000 7f7bc3fff700 1 -- 172.21.15.118:0/377039622 <== mon.0 v2:172.21.15.118:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f7bc8011a60 con 0x7f7bcc093b00 2023-10-05T19:27:56.218 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-05T19:27:56.211+0000 7f7bd253c700 1 -- 172.21.15.118:0/377039622 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f7bcc093b00 msgr2=0x7f7bcc093ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-05T19:27:56.218 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-05T19:27:56.211+0000 7f7bd253c700 1 --2- 172.21.15.118:0/377039622 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f7bcc093b00 0x7f7bcc093ed0 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f7bc8009860 tx=0x7f7bc80091a0 comp rx=0 tx=0).stop 2023-10-05T19:27:56.218 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-05T19:27:56.211+0000 7f7bd253c700 1 -- 172.21.15.118:0/377039622 shutdown_connections 2023-10-05T19:27:56.218 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-05T19:27:56.211+0000 7f7bd253c700 1 --2- 172.21.15.118:0/377039622 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f7bcc093b00 0x7f7bcc093ed0 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-05T19:27:56.218 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-05T19:27:56.211+0000 7f7bd253c700 1 -- 172.21.15.118:0/377039622 >> 172.21.15.118:0/377039622 conn(0x7f7bcc099110 msgr2=0x7f7bcc099510 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-05T19:27:56.219 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-05T19:27:56.211+0000 7f7bd253c700 1 -- 172.21.15.118:0/377039622 shutdown_connections 2023-10-05T19:27:56.219 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-05T19:27:56.211+0000 7f7bd253c700 1 -- 172.21.15.118:0/377039622 wait complete. 2023-10-05T19:27:56.219 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-05T19:27:56.211+0000 7f7bd253c700 1 Processor -- start 2023-10-05T19:27:56.219 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-05T19:27:56.211+0000 7f7bd253c700 1 -- start start 2023-10-05T19:27:56.220 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-05T19:27:56.211+0000 7f7bd253c700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f7bcc093b00 0x7f7bcc08c080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-05T19:27:56.220 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-05T19:27:56.211+0000 7f7bd253c700 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7bcc08f990 con 0x7f7bcc093b00 2023-10-05T19:27:56.220 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-05T19:27:56.211+0000 7f7bd153a700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f7bcc093b00 0x7f7bcc08c080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-05T19:27:56.220 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-05T19:27:56.211+0000 7f7bd153a700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f7bcc093b00 0x7f7bcc08c080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.118:3300/0 says I am v2:172.21.15.118:50152/0 (socket says 172.21.15.118:50152) 2023-10-05T19:27:56.221 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-05T19:27:56.211+0000 7f7bd153a700 1 -- 172.21.15.118:0/1085661039 learned_addr learned my addr 172.21.15.118:0/1085661039 (peer_addr_for_me v2:172.21.15.118:0/0) 2023-10-05T19:27:56.221 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-05T19:27:56.211+0000 7f7bd153a700 1 -- 172.21.15.118:0/1085661039 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7bc8009420 con 0x7f7bcc093b00 2023-10-05T19:27:56.221 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-05T19:27:56.211+0000 7f7bd153a700 1 --2- 172.21.15.118:0/1085661039 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f7bcc093b00 0x7f7bcc08c080 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f7bc8003080 tx=0x7f7bc8016000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-05T19:27:56.221 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-05T19:27:56.211+0000 7f7bc17fa700 1 -- 172.21.15.118:0/1085661039 <== mon.0 v2:172.21.15.118:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f7bc8011a80 con 0x7f7bcc093b00 2023-10-05T19:27:56.221 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-05T19:27:56.211+0000 7f7bc17fa700 1 -- 172.21.15.118:0/1085661039 <== mon.0 v2:172.21.15.118:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7bc801f070 con 0x7f7bcc093b00 2023-10-05T19:27:56.221 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-05T19:27:56.211+0000 7f7bd253c700 1 -- 172.21.15.118:0/1085661039 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7bcc092eb0 con 0x7f7bcc093b00 2023-10-05T19:27:56.221 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-05T19:27:56.211+0000 7f7bc17fa700 1 -- 172.21.15.118:0/1085661039 <== mon.0 v2:172.21.15.118:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f7bc8011e70 con 0x7f7bcc093b00 2023-10-05T19:27:56.221 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-05T19:27:56.211+0000 7f7bd253c700 1 -- 172.21.15.118:0/1085661039 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7bcc08f680 con 0x7f7bcc093b00 2023-10-05T19:27:56.222 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-05T19:27:56.215+0000 7f7bc17fa700 1 -- 172.21.15.118:0/1085661039 <== mon.0 v2:172.21.15.118:3300/0 4 ==== mgrmap(e 5) v1 ==== 40329+0+0 (secure 0 0 0) 0x7f7bc8026480 con 0x7f7bcc093b00 2023-10-05T19:27:56.222 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-05T19:27:56.215+0000 7f7bd253c700 1 -- 172.21.15.118:0/1085661039 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7bcc08ea30 con 0x7f7bcc093b00 2023-10-05T19:27:56.222 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-05T19:27:56.215+0000 7f7bc17fa700 1 --2- 172.21.15.118:0/1085661039 >> [v2:172.21.15.167:6824/29889,v1:172.21.15.167:6825/29889] conn(0x7f7bb00323e0 0x7f7bb0034890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-05T19:27:56.222 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-05T19:27:56.215+0000 7f7bc17fa700 1 -- 172.21.15.118:0/1085661039 <== mon.0 v2:172.21.15.118:3300/0 5 ==== osd_map(12..12 src has 1..12) v4 ==== 3125+0+0 (secure 0 0 0) 0x7f7bc8021070 con 0x7f7bcc093b00 2023-10-05T19:27:56.222 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-05T19:27:56.215+0000 7f7bd0d39700 1 --2- 172.21.15.118:0/1085661039 >> [v2:172.21.15.167:6824/29889,v1:172.21.15.167:6825/29889] conn(0x7f7bb00323e0 0x7f7bb0034890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-05T19:27:56.223 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-05T19:27:56.215+0000 7f7bd0d39700 1 --2- 172.21.15.118:0/1085661039 >> [v2:172.21.15.167:6824/29889,v1:172.21.15.167:6825/29889] conn(0x7f7bb00323e0 0x7f7bb0034890 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7f7bbc0058d0 tx=0x7f7bbc0079d0 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-05T19:27:56.226 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-05T19:27:56.219+0000 7f7bc17fa700 1 -- 172.21.15.118:0/1085661039 <== mon.0 v2:172.21.15.118:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166807 (secure 0 0 0) 0x7f7bc80259d0 con 0x7f7bcc093b00 2023-10-05T19:27:56.388 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-05T19:27:56.379+0000 7f7bd253c700 1 -- 172.21.15.118:0/1085661039 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f7bcc05ecb0 con 0x7f7bcc093b00 2023-10-05T19:27:56.746 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-05T19:27:56.739+0000 7f7bc17fa700 1 -- 172.21.15.118:0/1085661039 <== mon.0 v2:172.21.15.118:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}]=0 pool 'rbd' created v13) v1 ==== 109+0+0 (secure 0 0 0) 0x7f7bc800b3f0 con 0x7f7bcc093b00 2023-10-05T19:27:56.750 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-05T19:27:56.743+0000 7f7bd253c700 1 -- 172.21.15.118:0/1085661039 >> [v2:172.21.15.167:6824/29889,v1:172.21.15.167:6825/29889] conn(0x7f7bb00323e0 msgr2=0x7f7bb0034890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-05T19:27:56.750 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-05T19:27:56.743+0000 7f7bd253c700 1 --2- 172.21.15.118:0/1085661039 >> [v2:172.21.15.167:6824/29889,v1:172.21.15.167:6825/29889] conn(0x7f7bb00323e0 0x7f7bb0034890 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7f7bbc0058d0 tx=0x7f7bbc0079d0 comp rx=0 tx=0).stop 2023-10-05T19:27:56.750 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-05T19:27:56.743+0000 7f7bd253c700 1 -- 172.21.15.118:0/1085661039 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f7bcc093b00 msgr2=0x7f7bcc08c080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-05T19:27:56.750 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-05T19:27:56.743+0000 7f7bd253c700 1 --2- 172.21.15.118:0/1085661039 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f7bcc093b00 0x7f7bcc08c080 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f7bc8003080 tx=0x7f7bc8016000 comp rx=0 tx=0).stop 2023-10-05T19:27:56.750 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-05T19:27:56.743+0000 7f7bd253c700 1 -- 172.21.15.118:0/1085661039 shutdown_connections 2023-10-05T19:27:56.750 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-05T19:27:56.743+0000 7f7bd253c700 1 --2- 172.21.15.118:0/1085661039 >> [v2:172.21.15.167:6824/29889,v1:172.21.15.167:6825/29889] conn(0x7f7bb00323e0 0x7f7bb0034890 unknown :-1 s=CLOSED pgs=25 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-05T19:27:56.750 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-05T19:27:56.743+0000 7f7bd253c700 1 --2- 172.21.15.118:0/1085661039 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f7bcc093b00 0x7f7bcc08c080 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-05T19:27:56.750 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-05T19:27:56.743+0000 7f7bd253c700 1 -- 172.21.15.118:0/1085661039 >> 172.21.15.118:0/1085661039 conn(0x7f7bcc099110 msgr2=0x7f7bcc091c60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-05T19:27:56.750 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-05T19:27:56.743+0000 7f7bd253c700 1 -- 172.21.15.118:0/1085661039 shutdown_connections 2023-10-05T19:27:56.750 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-05T19:27:56.743+0000 7f7bd253c700 1 -- 172.21.15.118:0/1085661039 wait complete. 2023-10-05T19:27:56.751 INFO:teuthology.orchestra.run.smithi118.stderr:pool 'rbd' created 2023-10-05T19:27:56.766 DEBUG:teuthology.orchestra.run.smithi118:> sudo ceph --cluster ceph osd pool application enable rbd rbd --yes-i-really-mean-it 2023-10-05T19:27:56.880 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-05T19:27:56.871+0000 7f8a3895e700 1 Processor -- start 2023-10-05T19:27:56.880 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-05T19:27:56.871+0000 7f8a3895e700 1 -- start start 2023-10-05T19:27:56.881 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-05T19:27:56.875+0000 7f8a3895e700 1 --2- >> v2:172.21.15.118:3300/0 conn(0x7f8a34094220 0x7f8a3412e9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-05T19:27:56.881 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-05T19:27:56.875+0000 7f8a3895e700 1 -- --> v1:172.21.15.118:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8a34095f10 con 0x7f8a34098030 2023-10-05T19:27:56.881 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-05T19:27:56.875+0000 7f8a3895e700 1 -- --> v2:172.21.15.118:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8a340953a0 con 0x7f8a34094220 2023-10-05T19:27:56.881 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-05T19:27:56.875+0000 7f8a3259c700 1 --2- >> v2:172.21.15.118:3300/0 conn(0x7f8a34094220 0x7f8a3412e9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-05T19:27:56.881 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-05T19:27:56.875+0000 7f8a32d9d700 1 --1- >> v1:172.21.15.118:6789/0 conn(0x7f8a34098030 0x7f8a34093b10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.118:6789/0 says I am v1:172.21.15.118:52016/0 (socket says 172.21.15.118:52016) 2023-10-05T19:27:56.882 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-05T19:27:56.875+0000 7f8a3259c700 1 --2- >> v2:172.21.15.118:3300/0 conn(0x7f8a34094220 0x7f8a3412e9c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.118:3300/0 says I am v2:172.21.15.118:55660/0 (socket says 172.21.15.118:55660) 2023-10-05T19:27:56.882 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-05T19:27:56.875+0000 7f8a32d9d700 1 -- 172.21.15.118:0/2993493378 learned_addr learned my addr 172.21.15.118:0/2993493378 (peer_addr_for_me v1:172.21.15.118:0/0) 2023-10-05T19:27:56.882 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-05T19:27:56.875+0000 7f8a3259c700 1 -- 172.21.15.118:0/2993493378 >> v1:172.21.15.118:6789/0 conn(0x7f8a34098030 legacy=0x7f8a34093b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-05T19:27:56.882 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-05T19:27:56.875+0000 7f8a31d9b700 1 -- 172.21.15.118:0/2993493378 <== mon.0 v1:172.21.15.118:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2859814636 0 0) 0x7f8a34095f10 con 0x7f8a34098030 2023-10-05T19:27:56.882 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-05T19:27:56.875+0000 7f8a3259c700 1 -- 172.21.15.118:0/2993493378 --> v2:172.21.15.118:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8a34098ee0 con 0x7f8a34094220 2023-10-05T19:27:56.883 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-05T19:27:56.875+0000 7f8a3259c700 1 --2- 172.21.15.118:0/2993493378 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f8a34094220 0x7f8a3412e9c0 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f8a24009860 tx=0x7f8a240091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e961de218af91bb6 server_cookie=0 in_seq=0 out_seq=0 2023-10-05T19:27:56.883 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-05T19:27:56.875+0000 7f8a31d9b700 1 -- 172.21.15.118:0/2993493378 <== mon.0 v2:172.21.15.118:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f8a2400b680 con 0x7f8a34094220 2023-10-05T19:27:56.883 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-05T19:27:56.875+0000 7f8a31d9b700 1 -- 172.21.15.118:0/2993493378 <== mon.0 v2:172.21.15.118:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8a2400b7e0 con 0x7f8a34094220 2023-10-05T19:27:56.883 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-05T19:27:56.875+0000 7f8a31d9b700 1 -- 172.21.15.118:0/2993493378 <== mon.0 v2:172.21.15.118:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f8a2400b990 con 0x7f8a34094220 2023-10-05T19:27:56.883 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-05T19:27:56.875+0000 7f8a3895e700 1 -- 172.21.15.118:0/2993493378 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f8a34094220 msgr2=0x7f8a3412e9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-05T19:27:56.883 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-05T19:27:56.875+0000 7f8a3895e700 1 --2- 172.21.15.118:0/2993493378 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f8a34094220 0x7f8a3412e9c0 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f8a24009860 tx=0x7f8a240091a0 comp rx=0 tx=0).stop 2023-10-05T19:27:56.884 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-05T19:27:56.875+0000 7f8a3895e700 1 -- 172.21.15.118:0/2993493378 shutdown_connections 2023-10-05T19:27:56.884 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-05T19:27:56.875+0000 7f8a3895e700 1 --2- 172.21.15.118:0/2993493378 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f8a34094220 0x7f8a3412e9c0 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-05T19:27:56.884 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-05T19:27:56.875+0000 7f8a3895e700 1 -- 172.21.15.118:0/2993493378 >> 172.21.15.118:0/2993493378 conn(0x7f8a34099120 msgr2=0x7f8a34099520 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-05T19:27:56.884 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-05T19:27:56.875+0000 7f8a3895e700 1 -- 172.21.15.118:0/2993493378 shutdown_connections 2023-10-05T19:27:56.884 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-05T19:27:56.875+0000 7f8a3895e700 1 -- 172.21.15.118:0/2993493378 wait complete. 2023-10-05T19:27:56.884 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-05T19:27:56.879+0000 7f8a3895e700 1 Processor -- start 2023-10-05T19:27:56.884 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-05T19:27:56.879+0000 7f8a3895e700 1 -- start start 2023-10-05T19:27:56.885 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-05T19:27:56.879+0000 7f8a3895e700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f8a34094220 0x7f8a34138240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-05T19:27:56.885 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-05T19:27:56.879+0000 7f8a3895e700 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8a34125f00 con 0x7f8a34094220 2023-10-05T19:27:56.885 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-05T19:27:56.879+0000 7f8a32d9d700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f8a34094220 0x7f8a34138240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-05T19:27:56.885 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-05T19:27:56.879+0000 7f8a32d9d700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f8a34094220 0x7f8a34138240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.118:3300/0 says I am v2:172.21.15.118:55676/0 (socket says 172.21.15.118:55676) 2023-10-05T19:27:56.886 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-05T19:27:56.879+0000 7f8a32d9d700 1 -- 172.21.15.118:0/3900550082 learned_addr learned my addr 172.21.15.118:0/3900550082 (peer_addr_for_me v2:172.21.15.118:0/0) 2023-10-05T19:27:56.886 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-05T19:27:56.879+0000 7f8a32d9d700 1 -- 172.21.15.118:0/3900550082 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8a24009420 con 0x7f8a34094220 2023-10-05T19:27:56.886 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-05T19:27:56.879+0000 7f8a32d9d700 1 --2- 172.21.15.118:0/3900550082 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f8a34094220 0x7f8a34138240 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f8a2800c6c0 tx=0x7f8a28007c40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-05T19:27:56.887 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-05T19:27:56.879+0000 7f8a1affd700 1 -- 172.21.15.118:0/3900550082 <== mon.0 v2:172.21.15.118:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f8a28007320 con 0x7f8a34094220 2023-10-05T19:27:56.887 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-05T19:27:56.879+0000 7f8a1affd700 1 -- 172.21.15.118:0/3900550082 <== mon.0 v2:172.21.15.118:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8a280074a0 con 0x7f8a34094220 2023-10-05T19:27:56.887 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-05T19:27:56.879+0000 7f8a3895e700 1 -- 172.21.15.118:0/3900550082 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8a34125b80 con 0x7f8a34094220 2023-10-05T19:27:56.887 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-05T19:27:56.879+0000 7f8a1affd700 1 -- 172.21.15.118:0/3900550082 <== mon.0 v2:172.21.15.118:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f8a280057f0 con 0x7f8a34094220 2023-10-05T19:27:56.887 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-05T19:27:56.879+0000 7f8a3895e700 1 -- 172.21.15.118:0/3900550082 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8a341254b0 con 0x7f8a34094220 2023-10-05T19:27:56.888 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-05T19:27:56.879+0000 7f8a3895e700 1 -- 172.21.15.118:0/3900550082 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8a341259e0 con 0x7f8a34094220 2023-10-05T19:27:56.888 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-05T19:27:56.883+0000 7f8a1affd700 1 -- 172.21.15.118:0/3900550082 <== mon.0 v2:172.21.15.118:3300/0 4 ==== mgrmap(e 5) v1 ==== 40329+0+0 (secure 0 0 0) 0x7f8a28005950 con 0x7f8a34094220 2023-10-05T19:27:56.889 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-05T19:27:56.883+0000 7f8a1affd700 1 --2- 172.21.15.118:0/3900550082 >> [v2:172.21.15.167:6824/29889,v1:172.21.15.167:6825/29889] conn(0x7f8a200323e0 0x7f8a20034890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-05T19:27:56.889 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-05T19:27:56.883+0000 7f8a1affd700 1 -- 172.21.15.118:0/3900550082 <== mon.0 v2:172.21.15.118:3300/0 5 ==== osd_map(13..13 src has 1..13) v4 ==== 3467+0+0 (secure 0 0 0) 0x7f8a28001e40 con 0x7f8a34094220 2023-10-05T19:27:56.889 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-05T19:27:56.883+0000 7f8a3259c700 1 --2- 172.21.15.118:0/3900550082 >> [v2:172.21.15.167:6824/29889,v1:172.21.15.167:6825/29889] conn(0x7f8a200323e0 0x7f8a20034890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-05T19:27:56.890 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-05T19:27:56.883+0000 7f8a3259c700 1 --2- 172.21.15.118:0/3900550082 >> [v2:172.21.15.167:6824/29889,v1:172.21.15.167:6825/29889] conn(0x7f8a200323e0 0x7f8a20034890 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f8a240030c0 tx=0x7f8a24016000 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-05T19:27:56.896 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-05T19:27:56.891+0000 7f8a1affd700 1 -- 172.21.15.118:0/3900550082 <== mon.0 v2:172.21.15.118:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166807 (secure 0 0 0) 0x7f8a341259e0 con 0x7f8a34094220 2023-10-05T19:27:57.060 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-05T19:27:57.051+0000 7f8a3895e700 1 -- 172.21.15.118:0/3900550082 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_command({"prefix": "osd pool application enable", "pool": "rbd", "app": "rbd", "yes_i_really_mean_it": true} v 0) v1 -- 0x7f8a341259e0 con 0x7f8a34094220 2023-10-05T19:27:57.754 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-05T19:27:57.747+0000 7f8a1affd700 1 -- 172.21.15.118:0/3900550082 <== mon.0 v2:172.21.15.118:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool application enable", "pool": "rbd", "app": "rbd", "yes_i_really_mean_it": true}]=0 enabled application 'rbd' on pool 'rbd' v14) v1 ==== 173+0+0 (secure 0 0 0) 0x7f8a341259e0 con 0x7f8a34094220 2023-10-05T19:27:57.759 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-05T19:27:57.751+0000 7f8a3895e700 1 -- 172.21.15.118:0/3900550082 >> [v2:172.21.15.167:6824/29889,v1:172.21.15.167:6825/29889] conn(0x7f8a200323e0 msgr2=0x7f8a20034890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-05T19:27:57.759 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-05T19:27:57.751+0000 7f8a3895e700 1 --2- 172.21.15.118:0/3900550082 >> [v2:172.21.15.167:6824/29889,v1:172.21.15.167:6825/29889] conn(0x7f8a200323e0 0x7f8a20034890 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f8a240030c0 tx=0x7f8a24016000 comp rx=0 tx=0).stop 2023-10-05T19:27:57.760 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-05T19:27:57.751+0000 7f8a3895e700 1 -- 172.21.15.118:0/3900550082 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f8a34094220 msgr2=0x7f8a34138240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-05T19:27:57.760 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-05T19:27:57.751+0000 7f8a3895e700 1 --2- 172.21.15.118:0/3900550082 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f8a34094220 0x7f8a34138240 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f8a2800c6c0 tx=0x7f8a28007c40 comp rx=0 tx=0).stop 2023-10-05T19:27:57.760 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-05T19:27:57.751+0000 7f8a3895e700 1 -- 172.21.15.118:0/3900550082 shutdown_connections 2023-10-05T19:27:57.760 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-05T19:27:57.751+0000 7f8a3895e700 1 --2- 172.21.15.118:0/3900550082 >> [v2:172.21.15.167:6824/29889,v1:172.21.15.167:6825/29889] conn(0x7f8a200323e0 0x7f8a20034890 unknown :-1 s=CLOSED pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-05T19:27:57.760 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-05T19:27:57.751+0000 7f8a3895e700 1 --2- 172.21.15.118:0/3900550082 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f8a34094220 0x7f8a34138240 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-05T19:27:57.760 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-05T19:27:57.751+0000 7f8a3895e700 1 -- 172.21.15.118:0/3900550082 >> 172.21.15.118:0/3900550082 conn(0x7f8a34099120 msgr2=0x7f8a3409b3c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-05T19:27:57.760 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-05T19:27:57.751+0000 7f8a3895e700 1 -- 172.21.15.118:0/3900550082 shutdown_connections 2023-10-05T19:27:57.760 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-05T19:27:57.751+0000 7f8a3895e700 1 -- 172.21.15.118:0/3900550082 wait complete. 2023-10-05T19:27:57.761 INFO:teuthology.orchestra.run.smithi118.stderr:enabled application 'rbd' on pool 'rbd' 2023-10-05T19:27:57.774 INFO:tasks.ceph:Starting mds daemons in cluster ceph... 2023-10-05T19:27:57.775 INFO:tasks.ceph:Waiting until ceph daemons up and pgs clean... 2023-10-05T19:27:57.775 INFO:tasks.ceph.ceph_manager.ceph:waiting for mgr available 2023-10-05T19:27:57.775 DEBUG:teuthology.orchestra.run.smithi118:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mgr dump --format=json 2023-10-05T19:27:57.775 INFO:tasks.daemonwatchdog.daemon_watchdog:watchdog starting 2023-10-05T19:27:57.877 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-05T19:27:57.871+0000 7fdd85266700 1 Processor -- start 2023-10-05T19:27:57.877 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-05T19:27:57.871+0000 7fdd85266700 1 -- start start 2023-10-05T19:27:57.879 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-05T19:27:57.871+0000 7fdd85266700 1 --2- >> v2:172.21.15.118:3300/0 conn(0x7fdd80097710 0x7fdd80069b20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-05T19:27:57.879 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-05T19:27:57.871+0000 7fdd85266700 1 -- --> v1:172.21.15.118:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdd8006b610 con 0x7fdd8006a060 2023-10-05T19:27:57.879 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-05T19:27:57.871+0000 7fdd85266700 1 -- --> v2:172.21.15.118:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdd8006b0f0 con 0x7fdd80097710 2023-10-05T19:27:57.879 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-05T19:27:57.871+0000 7fdd7f7fe700 1 --1- >> v1:172.21.15.118:6789/0 conn(0x7fdd8006a060 0x7fdd8012d9b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.118:6789/0 says I am v1:172.21.15.118:52026/0 (socket says 172.21.15.118:52026) 2023-10-05T19:27:57.879 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-05T19:27:57.871+0000 7fdd7f7fe700 1 -- 172.21.15.118:0/1468234000 learned_addr learned my addr 172.21.15.118:0/1468234000 (peer_addr_for_me v1:172.21.15.118:0/0) 2023-10-05T19:27:57.879 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-05T19:27:57.871+0000 7fdd7ffff700 1 --2- >> v2:172.21.15.118:3300/0 conn(0x7fdd80097710 0x7fdd80069b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-05T19:27:57.879 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-05T19:27:57.871+0000 7fdd7effd700 1 -- 172.21.15.118:0/1468234000 <== mon.0 v1:172.21.15.118:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1291297523 0 0) 0x7fdd8006b610 con 0x7fdd8006a060 2023-10-05T19:27:57.879 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-05T19:27:57.871+0000 7fdd7effd700 1 -- 172.21.15.118:0/1468234000 --> v1:172.21.15.118:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdd68003570 con 0x7fdd8006a060 2023-10-05T19:27:57.879 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-05T19:27:57.871+0000 7fdd7ffff700 1 -- 172.21.15.118:0/1468234000 >> v1:172.21.15.118:6789/0 conn(0x7fdd8006a060 legacy=0x7fdd8012d9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-05T19:27:57.880 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-05T19:27:57.871+0000 7fdd7ffff700 1 -- 172.21.15.118:0/1468234000 --> v2:172.21.15.118:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdd8006b300 con 0x7fdd80097710 2023-10-05T19:27:57.881 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-05T19:27:57.871+0000 7fdd7ffff700 1 --2- 172.21.15.118:0/1468234000 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fdd80097710 0x7fdd80069b20 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7fdd74009860 tx=0x7fdd740091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=100614b9be5d3c35 server_cookie=0 in_seq=0 out_seq=0 2023-10-05T19:27:57.881 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-05T19:27:57.871+0000 7fdd7effd700 1 -- 172.21.15.118:0/1468234000 <== mon.0 v2:172.21.15.118:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fdd74011630 con 0x7fdd80097710 2023-10-05T19:27:57.881 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-05T19:27:57.871+0000 7fdd7effd700 1 -- 172.21.15.118:0/1468234000 <== mon.0 v2:172.21.15.118:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdd74011790 con 0x7fdd80097710 2023-10-05T19:27:57.881 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-05T19:27:57.871+0000 7fdd7effd700 1 -- 172.21.15.118:0/1468234000 <== mon.0 v2:172.21.15.118:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fdd74011900 con 0x7fdd80097710 2023-10-05T19:27:57.881 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-05T19:27:57.871+0000 7fdd85266700 1 -- 172.21.15.118:0/1468234000 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fdd80097710 msgr2=0x7fdd80069b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-05T19:27:57.882 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-05T19:27:57.871+0000 7fdd85266700 1 --2- 172.21.15.118:0/1468234000 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fdd80097710 0x7fdd80069b20 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7fdd74009860 tx=0x7fdd740091a0 comp rx=0 tx=0).stop 2023-10-05T19:27:57.882 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-05T19:27:57.875+0000 7fdd85266700 1 -- 172.21.15.118:0/1468234000 shutdown_connections 2023-10-05T19:27:57.882 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-05T19:27:57.875+0000 7fdd85266700 1 --2- 172.21.15.118:0/1468234000 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fdd80097710 0x7fdd80069b20 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-05T19:27:57.882 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-05T19:27:57.875+0000 7fdd85266700 1 -- 172.21.15.118:0/1468234000 >> 172.21.15.118:0/1468234000 conn(0x7fdd800952f0 msgr2=0x7fdd800956f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-05T19:27:57.882 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-05T19:27:57.875+0000 7fdd85266700 1 -- 172.21.15.118:0/1468234000 shutdown_connections 2023-10-05T19:27:57.882 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-05T19:27:57.875+0000 7fdd85266700 1 -- 172.21.15.118:0/1468234000 wait complete. 2023-10-05T19:27:57.883 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-05T19:27:57.875+0000 7fdd85266700 1 Processor -- start 2023-10-05T19:27:57.884 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-05T19:27:57.875+0000 7fdd85266700 1 -- start start 2023-10-05T19:27:57.884 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-05T19:27:57.875+0000 7fdd85266700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fdd8006a060 0x7fdd8011ddc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-05T19:27:57.884 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-05T19:27:57.875+0000 7fdd85266700 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdd80135be0 con 0x7fdd8006a060 2023-10-05T19:27:57.884 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-05T19:27:57.875+0000 7fdd7ffff700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fdd8006a060 0x7fdd8011ddc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-05T19:27:57.884 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-05T19:27:57.875+0000 7fdd7ffff700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fdd8006a060 0x7fdd8011ddc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.118:3300/0 says I am v2:172.21.15.118:55696/0 (socket says 172.21.15.118:55696) 2023-10-05T19:27:57.884 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-05T19:27:57.875+0000 7fdd7ffff700 1 -- 172.21.15.118:0/2349498471 learned_addr learned my addr 172.21.15.118:0/2349498471 (peer_addr_for_me v2:172.21.15.118:0/0) 2023-10-05T19:27:57.884 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-05T19:27:57.875+0000 7fdd7ffff700 1 -- 172.21.15.118:0/2349498471 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdd74009420 con 0x7fdd8006a060 2023-10-05T19:27:57.884 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-05T19:27:57.875+0000 7fdd7ffff700 1 --2- 172.21.15.118:0/2349498471 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fdd8006a060 0x7fdd8011ddc0 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7fdd740093f0 tx=0x7fdd74016000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-05T19:27:57.884 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-05T19:27:57.875+0000 7fdd67fff700 1 -- 172.21.15.118:0/2349498471 <== mon.0 v2:172.21.15.118:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fdd7401ae70 con 0x7fdd8006a060 2023-10-05T19:27:57.884 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-05T19:27:57.875+0000 7fdd67fff700 1 -- 172.21.15.118:0/2349498471 <== mon.0 v2:172.21.15.118:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdd7401f070 con 0x7fdd8006a060 2023-10-05T19:27:57.885 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-05T19:27:57.875+0000 7fdd85266700 1 -- 172.21.15.118:0/2349498471 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdd8011f3b0 con 0x7fdd8006a060 2023-10-05T19:27:57.885 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-05T19:27:57.875+0000 7fdd67fff700 1 -- 172.21.15.118:0/2349498471 <== mon.0 v2:172.21.15.118:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fdd74011900 con 0x7fdd8006a060 2023-10-05T19:27:57.885 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-05T19:27:57.875+0000 7fdd85266700 1 -- 172.21.15.118:0/2349498471 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdd8011d970 con 0x7fdd8006a060 2023-10-05T19:27:57.886 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-05T19:27:57.879+0000 7fdd67fff700 1 -- 172.21.15.118:0/2349498471 <== mon.0 v2:172.21.15.118:3300/0 4 ==== mgrmap(e 5) v1 ==== 40329+0+0 (secure 0 0 0) 0x7fdd74026480 con 0x7fdd8006a060 2023-10-05T19:27:57.886 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-05T19:27:57.879+0000 7fdd67fff700 1 --2- 172.21.15.118:0/2349498471 >> [v2:172.21.15.167:6824/29889,v1:172.21.15.167:6825/29889] conn(0x7fdd6c0323d0 0x7fdd6c034880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-05T19:27:57.886 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-05T19:27:57.879+0000 7fdd67fff700 1 -- 172.21.15.118:0/2349498471 <== mon.0 v2:172.21.15.118:3300/0 5 ==== osd_map(14..14 src has 1..14) v4 ==== 3478+0+0 (secure 0 0 0) 0x7fdd74021070 con 0x7fdd8006a060 2023-10-05T19:27:57.886 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-05T19:27:57.879+0000 7fdd85266700 1 -- 172.21.15.118:0/2349498471 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdd8005ecb0 con 0x7fdd8006a060 2023-10-05T19:27:57.886 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-05T19:27:57.879+0000 7fdd7f7fe700 1 --2- 172.21.15.118:0/2349498471 >> [v2:172.21.15.167:6824/29889,v1:172.21.15.167:6825/29889] conn(0x7fdd6c0323d0 0x7fdd6c034880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-05T19:27:57.887 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-05T19:27:57.879+0000 7fdd7f7fe700 1 --2- 172.21.15.118:0/2349498471 >> [v2:172.21.15.167:6824/29889,v1:172.21.15.167:6825/29889] conn(0x7fdd6c0323d0 0x7fdd6c034880 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7fdd70005c50 tx=0x7fdd700016e0 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-05T19:27:57.891 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-05T19:27:57.883+0000 7fdd67fff700 1 -- 172.21.15.118:0/2349498471 <== mon.0 v2:172.21.15.118:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166807 (secure 0 0 0) 0x7fdd74010970 con 0x7fdd8006a060 2023-10-05T19:27:58.087 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-05T19:27:58.079+0000 7fdd85266700 1 -- 172.21.15.118:0/2349498471 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_command({"prefix": "mgr dump", "format": "json"} v 0) v1 -- 0x7fdd8006e870 con 0x7fdd8006a060 2023-10-05T19:27:58.091 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-05T19:27:58.083+0000 7fdd67fff700 1 -- 172.21.15.118:0/2349498471 <== mon.0 v2:172.21.15.118:3300/0 7 ==== mon_command_ack([{"prefix": "mgr dump", "format": "json"}]=0 v5) v1 ==== 74+0+77919 (secure 0 0 0) 0x7fdd7400b3f0 con 0x7fdd8006a060 2023-10-05T19:27:58.092 INFO:teuthology.orchestra.run.smithi118.stdout: 2023-10-05T19:27:58.093 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-05T19:27:58.087+0000 7fdd85266700 1 -- 172.21.15.118:0/2349498471 >> [v2:172.21.15.167:6824/29889,v1:172.21.15.167:6825/29889] conn(0x7fdd6c0323d0 msgr2=0x7fdd6c034880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-05T19:27:58.094 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-05T19:27:58.087+0000 7fdd85266700 1 --2- 172.21.15.118:0/2349498471 >> [v2:172.21.15.167:6824/29889,v1:172.21.15.167:6825/29889] conn(0x7fdd6c0323d0 0x7fdd6c034880 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7fdd70005c50 tx=0x7fdd700016e0 comp rx=0 tx=0).stop 2023-10-05T19:27:58.094 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-05T19:27:58.087+0000 7fdd85266700 1 -- 172.21.15.118:0/2349498471 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fdd8006a060 msgr2=0x7fdd8011ddc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-05T19:27:58.094 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-05T19:27:58.087+0000 7fdd85266700 1 --2- 172.21.15.118:0/2349498471 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fdd8006a060 0x7fdd8011ddc0 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7fdd740093f0 tx=0x7fdd74016000 comp rx=0 tx=0).stop 2023-10-05T19:27:58.094 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-05T19:27:58.087+0000 7fdd85266700 1 -- 172.21.15.118:0/2349498471 shutdown_connections 2023-10-05T19:27:58.094 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-05T19:27:58.087+0000 7fdd85266700 1 --2- 172.21.15.118:0/2349498471 >> [v2:172.21.15.167:6824/29889,v1:172.21.15.167:6825/29889] conn(0x7fdd6c0323d0 0x7fdd6c034880 unknown :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-05T19:27:58.094 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-05T19:27:58.087+0000 7fdd85266700 1 --2- 172.21.15.118:0/2349498471 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fdd8006a060 0x7fdd8011ddc0 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-05T19:27:58.094 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-05T19:27:58.087+0000 7fdd85266700 1 -- 172.21.15.118:0/2349498471 >> 172.21.15.118:0/2349498471 conn(0x7fdd800952f0 msgr2=0x7fdd8006d500 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-05T19:27:58.094 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-05T19:27:58.087+0000 7fdd85266700 1 -- 172.21.15.118:0/2349498471 shutdown_connections 2023-10-05T19:27:58.094 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-05T19:27:58.087+0000 7fdd85266700 1 -- 172.21.15.118:0/2349498471 wait complete. 2023-10-05T19:27:58.105 INFO:teuthology.orchestra.run.smithi118.stdout:{"epoch":5,"active_gid":4100,"active_name":"x","active_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6824","nonce":29889},{"type":"v1","addr":"172.21.15.167:6825","nonce":29889}]},"active_addr":"172.21.15.167:6825/29889","active_change":"2023-10-05T19:27:44.901914+0000","active_mgr_features":4540138320759226367,"available":true,"standbys":[],"modules":["iostat","nfs","restful"],"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"7","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 or 7 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2400","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"7","min":"0","max":"7","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 or 7 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","upmap"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":false,"error_string":"loading asyncssh library:No module named 'asyncssh'","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/ceph-grafana:9.4.7","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.1.5","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"secs","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"noautoscale":{"name":"noautoscale","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"global autoscale flag","long_desc":"Option to turn on/off the autoscaler for all pools","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}],"services":{},"always_on_modules":{"octopus":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"pacific":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"quincy":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"]},"last_failure_osd_epoch":0,"active_clients":[{"name":"devicehealth","addrvec":[{"type":"v2","addr":"172.21.15.167:0","nonce":4076876152}]},{"name":"libcephsqlite","addrvec":[{"type":"v2","addr":"172.21.15.167:0","nonce":3041943082}]},{"name":"rbd_support","addrvec":[{"type":"v2","addr":"172.21.15.167:0","nonce":3566012142}]},{"name":"volumes","addrvec":[{"type":"v2","addr":"172.21.15.167:0","nonce":370599494}]}]} 2023-10-05T19:27:58.108 INFO:tasks.ceph.ceph_manager.ceph:mgr available! 2023-10-05T19:27:58.108 INFO:tasks.ceph.ceph_manager.ceph:waiting for all up 2023-10-05T19:27:58.108 DEBUG:teuthology.orchestra.run.smithi118:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-05T19:27:58.214 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-05T19:27:58.207+0000 7f3668e77700 1 Processor -- start 2023-10-05T19:27:58.214 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-05T19:27:58.207+0000 7f3668e77700 1 -- start start 2023-10-05T19:27:58.214 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-05T19:27:58.207+0000 7f3668e77700 1 --2- >> v2:172.21.15.118:3300/0 conn(0x7f3664095ff0 0x7f36640963c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-05T19:27:58.214 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-05T19:27:58.207+0000 7f3668e77700 1 -- --> v1:172.21.15.118:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3664092760 con 0x7f36640905f0 2023-10-05T19:27:58.214 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-05T19:27:58.207+0000 7f3668e77700 1 -- --> v2:172.21.15.118:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f36640918a0 con 0x7f3664095ff0 2023-10-05T19:27:58.215 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-05T19:27:58.207+0000 7f36637fe700 1 --2- >> v2:172.21.15.118:3300/0 conn(0x7f3664095ff0 0x7f36640963c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-05T19:27:58.215 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-05T19:27:58.207+0000 7f36637fe700 1 --2- >> v2:172.21.15.118:3300/0 conn(0x7f3664095ff0 0x7f36640963c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.118:3300/0 says I am v2:172.21.15.118:55710/0 (socket says 172.21.15.118:55710) 2023-10-05T19:27:58.215 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-05T19:27:58.207+0000 7f36637fe700 1 -- 172.21.15.118:0/1289109614 learned_addr learned my addr 172.21.15.118:0/1289109614 (peer_addr_for_me v2:172.21.15.118:0/0) 2023-10-05T19:27:58.215 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-05T19:27:58.207+0000 7f3662ffd700 1 --1- 172.21.15.118:0/1289109614 >> v1:172.21.15.118:6789/0 conn(0x7f36640905f0 0x7f36640909c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.118:6789/0 says I am v1:172.21.15.118:52040/0 (socket says 172.21.15.118:52040) 2023-10-05T19:27:58.215 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-05T19:27:58.207+0000 7f36637fe700 1 -- 172.21.15.118:0/1289109614 >> v1:172.21.15.118:6789/0 conn(0x7f36640905f0 legacy=0x7f36640909c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-05T19:27:58.215 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-05T19:27:58.207+0000 7f36637fe700 1 -- 172.21.15.118:0/1289109614 --> v2:172.21.15.118:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3664093ba0 con 0x7f3664095ff0 2023-10-05T19:27:58.215 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-05T19:27:58.207+0000 7f36637fe700 1 --2- 172.21.15.118:0/1289109614 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f3664095ff0 0x7f36640963c0 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f365c009ea0 tx=0x7f365c009840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=17358b001b9b8775 server_cookie=0 in_seq=0 out_seq=0 2023-10-05T19:27:58.216 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-05T19:27:58.207+0000 7f36627fc700 1 -- 172.21.15.118:0/1289109614 <== mon.0 v2:172.21.15.118:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f365c00e070 con 0x7f3664095ff0 2023-10-05T19:27:58.217 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-05T19:27:58.207+0000 7f36627fc700 1 -- 172.21.15.118:0/1289109614 <== mon.0 v2:172.21.15.118:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f365c00ba00 con 0x7f3664095ff0 2023-10-05T19:27:58.217 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-05T19:27:58.207+0000 7f36627fc700 1 -- 172.21.15.118:0/1289109614 <== mon.0 v2:172.21.15.118:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f365c0092e0 con 0x7f3664095ff0 2023-10-05T19:27:58.217 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-05T19:27:58.207+0000 7f3668e77700 1 -- 172.21.15.118:0/1289109614 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f3664095ff0 msgr2=0x7f36640963c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-05T19:27:58.217 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-05T19:27:58.207+0000 7f3668e77700 1 --2- 172.21.15.118:0/1289109614 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f3664095ff0 0x7f36640963c0 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f365c009ea0 tx=0x7f365c009840 comp rx=0 tx=0).stop 2023-10-05T19:27:58.217 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-05T19:27:58.207+0000 7f3668e77700 1 -- 172.21.15.118:0/1289109614 shutdown_connections 2023-10-05T19:27:58.217 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-05T19:27:58.207+0000 7f3668e77700 1 --2- 172.21.15.118:0/1289109614 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f3664095ff0 0x7f36640963c0 unknown :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-05T19:27:58.217 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-05T19:27:58.207+0000 7f3668e77700 1 -- 172.21.15.118:0/1289109614 >> 172.21.15.118:0/1289109614 conn(0x7f366409c5d0 msgr2=0x7f366409c9d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-05T19:27:58.217 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-05T19:27:58.207+0000 7f3668e77700 1 -- 172.21.15.118:0/1289109614 shutdown_connections 2023-10-05T19:27:58.217 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-05T19:27:58.207+0000 7f3668e77700 1 -- 172.21.15.118:0/1289109614 wait complete. 2023-10-05T19:27:58.217 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-05T19:27:58.211+0000 7f3668e77700 1 Processor -- start 2023-10-05T19:27:58.217 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-05T19:27:58.211+0000 7f3668e77700 1 -- start start 2023-10-05T19:27:58.219 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-05T19:27:58.211+0000 7f3668e77700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f36640905f0 0x7f366408c0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-05T19:27:58.219 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-05T19:27:58.211+0000 7f3668e77700 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f366408f7f0 con 0x7f36640905f0 2023-10-05T19:27:58.219 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-05T19:27:58.211+0000 7f36637fe700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f36640905f0 0x7f366408c0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-05T19:27:58.219 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-05T19:27:58.211+0000 7f36637fe700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f36640905f0 0x7f366408c0c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.118:3300/0 says I am v2:172.21.15.118:55714/0 (socket says 172.21.15.118:55714) 2023-10-05T19:27:58.219 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-05T19:27:58.211+0000 7f36637fe700 1 -- 172.21.15.118:0/1497983639 learned_addr learned my addr 172.21.15.118:0/1497983639 (peer_addr_for_me v2:172.21.15.118:0/0) 2023-10-05T19:27:58.219 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-05T19:27:58.211+0000 7f36637fe700 1 -- 172.21.15.118:0/1497983639 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f365c009ae0 con 0x7f36640905f0 2023-10-05T19:27:58.219 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-05T19:27:58.211+0000 7f36637fe700 1 --2- 172.21.15.118:0/1497983639 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f36640905f0 0x7f366408c0c0 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f365c009540 tx=0x7f365c017000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-05T19:27:58.219 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-05T19:27:58.211+0000 7f364b7fe700 1 -- 172.21.15.118:0/1497983639 <== mon.0 v2:172.21.15.118:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f365c007170 con 0x7f36640905f0 2023-10-05T19:27:58.220 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-05T19:27:58.211+0000 7f364b7fe700 1 -- 172.21.15.118:0/1497983639 <== mon.0 v2:172.21.15.118:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f365c012b90 con 0x7f36640905f0 2023-10-05T19:27:58.220 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-05T19:27:58.211+0000 7f3668e77700 1 -- 172.21.15.118:0/1497983639 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f366408da60 con 0x7f36640905f0 2023-10-05T19:27:58.220 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-05T19:27:58.211+0000 7f364b7fe700 1 -- 172.21.15.118:0/1497983639 <== mon.0 v2:172.21.15.118:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f365c012d20 con 0x7f36640905f0 2023-10-05T19:27:58.220 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-05T19:27:58.211+0000 7f3668e77700 1 -- 172.21.15.118:0/1497983639 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f366408e310 con 0x7f36640905f0 2023-10-05T19:27:58.221 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-05T19:27:58.211+0000 7f364b7fe700 1 -- 172.21.15.118:0/1497983639 <== mon.0 v2:172.21.15.118:3300/0 4 ==== mgrmap(e 5) v1 ==== 40329+0+0 (secure 0 0 0) 0x7f365c01d420 con 0x7f36640905f0 2023-10-05T19:27:58.221 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-05T19:27:58.215+0000 7f3668e77700 1 -- 172.21.15.118:0/1497983639 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f366405ecb0 con 0x7f36640905f0 2023-10-05T19:27:58.221 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-05T19:27:58.215+0000 7f364b7fe700 1 --2- 172.21.15.118:0/1497983639 >> [v2:172.21.15.167:6824/29889,v1:172.21.15.167:6825/29889] conn(0x7f36500323e0 0x7f3650034890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-05T19:27:58.221 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-05T19:27:58.215+0000 7f364b7fe700 1 -- 172.21.15.118:0/1497983639 <== mon.0 v2:172.21.15.118:3300/0 5 ==== osd_map(14..14 src has 1..14) v4 ==== 3478+0+0 (secure 0 0 0) 0x7f365c045310 con 0x7f36640905f0 2023-10-05T19:27:58.221 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-05T19:27:58.215+0000 7f3662ffd700 1 --2- 172.21.15.118:0/1497983639 >> [v2:172.21.15.167:6824/29889,v1:172.21.15.167:6825/29889] conn(0x7f36500323e0 0x7f3650034890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-05T19:27:58.221 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-05T19:27:58.215+0000 7f3662ffd700 1 --2- 172.21.15.118:0/1497983639 >> [v2:172.21.15.167:6824/29889,v1:172.21.15.167:6825/29889] conn(0x7f36500323e0 0x7f3650034890 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f3658001650 tx=0x7f3658007a20 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-05T19:27:58.227 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-05T19:27:58.219+0000 7f364b7fe700 1 -- 172.21.15.118:0/1497983639 <== mon.0 v2:172.21.15.118:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166807 (secure 0 0 0) 0x7f365c02dce0 con 0x7f36640905f0 2023-10-05T19:27:58.385 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-05T19:27:58.379+0000 7f3668e77700 1 -- 172.21.15.118:0/1497983639 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f3664091b10 con 0x7f36640905f0 2023-10-05T19:27:58.385 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-05T19:27:58.379+0000 7f364b7fe700 1 -- 172.21.15.118:0/1497983639 <== mon.0 v2:172.21.15.118:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v14) v1 ==== 74+0+9003 (secure 0 0 0) 0x7f365c01ce60 con 0x7f36640905f0 2023-10-05T19:27:58.386 INFO:teuthology.orchestra.run.smithi118.stdout: 2023-10-05T19:27:58.386 INFO:teuthology.orchestra.run.smithi118.stdout:{"epoch":14,"fsid":"8c4bd0d1-cbcb-4713-96b7-dcb14bcaf584","created":"2023-10-05T19:27:40.755769+0000","modified":"2023-10-05T19:27:57.740909+0000","last_up_change":"2023-10-05T19:27:50.196733+0000","last_in_change":"2023-10-05T19:27:41.811702+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-05T19:27:50.980612+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"12","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}},{"pool":2,"pool_name":"rbd","create_time":"2023-10-05T19:27:56.423831+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"14","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"61ba9225-834e-4288-b391-7ffa540f7334","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":13,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6800","nonce":27690},{"type":"v1","addr":"172.21.15.118:6801","nonce":27690}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6802","nonce":27690},{"type":"v1","addr":"172.21.15.118:6803","nonce":27690}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6806","nonce":27690},{"type":"v1","addr":"172.21.15.118:6807","nonce":27690}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6804","nonce":27690},{"type":"v1","addr":"172.21.15.118:6805","nonce":27690}]},"public_addr":"172.21.15.118:6801/27690","cluster_addr":"172.21.15.118:6803/27690","heartbeat_back_addr":"172.21.15.118:6807/27690","heartbeat_front_addr":"172.21.15.118:6805/27690","state":["exists","up"]},{"osd":1,"uuid":"15a3631e-9f20-4b33-9ee3-39eb73e13709","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":10,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6800","nonce":30035},{"type":"v1","addr":"172.21.15.167:6801","nonce":30035}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6802","nonce":30035},{"type":"v1","addr":"172.21.15.167:6803","nonce":30035}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6806","nonce":30035},{"type":"v1","addr":"172.21.15.167:6807","nonce":30035}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6804","nonce":30035},{"type":"v1","addr":"172.21.15.167:6805","nonce":30035}]},"public_addr":"172.21.15.167:6801/30035","cluster_addr":"172.21.15.167:6803/30035","heartbeat_back_addr":"172.21.15.167:6807/30035","heartbeat_front_addr":"172.21.15.167:6805/30035","state":["exists","up"]},{"osd":2,"uuid":"5712158a-931c-4ce7-adf0-5c9fbb59d521","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":13,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6816","nonce":30037},{"type":"v1","addr":"172.21.15.167:6817","nonce":30037}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6818","nonce":30037},{"type":"v1","addr":"172.21.15.167:6819","nonce":30037}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6822","nonce":30037},{"type":"v1","addr":"172.21.15.167:6823","nonce":30037}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6820","nonce":30037},{"type":"v1","addr":"172.21.15.167:6821","nonce":30037}]},"public_addr":"172.21.15.167:6817/30037","cluster_addr":"172.21.15.167:6819/30037","heartbeat_back_addr":"172.21.15.167:6823/30037","heartbeat_front_addr":"172.21.15.167:6821/30037","state":["exists","up"]},{"osd":3,"uuid":"f838ed4e-05a7-49ef-ad9d-8f7dd7be30f1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":13,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6808","nonce":30036},{"type":"v1","addr":"172.21.15.167:6809","nonce":30036}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6810","nonce":30036},{"type":"v1","addr":"172.21.15.167:6811","nonce":30036}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6814","nonce":30036},{"type":"v1","addr":"172.21.15.167:6815","nonce":30036}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6812","nonce":30036},{"type":"v1","addr":"172.21.15.167:6813","nonce":30036}]},"public_addr":"172.21.15.167:6809/30036","cluster_addr":"172.21.15.167:6811/30036","heartbeat_back_addr":"172.21.15.167:6815/30036","heartbeat_front_addr":"172.21.15.167:6813/30036","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-05T19:27:47.190191+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-05T19:27:47.302630+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-05T19:27:47.374216+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-05T19:27:47.350455+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-05T19:27:58.388 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-05T19:27:58.379+0000 7f3668e77700 1 -- 172.21.15.118:0/1497983639 >> [v2:172.21.15.167:6824/29889,v1:172.21.15.167:6825/29889] conn(0x7f36500323e0 msgr2=0x7f3650034890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-05T19:27:58.388 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-05T19:27:58.379+0000 7f3668e77700 1 --2- 172.21.15.118:0/1497983639 >> [v2:172.21.15.167:6824/29889,v1:172.21.15.167:6825/29889] conn(0x7f36500323e0 0x7f3650034890 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f3658001650 tx=0x7f3658007a20 comp rx=0 tx=0).stop 2023-10-05T19:27:58.388 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-05T19:27:58.379+0000 7f3668e77700 1 -- 172.21.15.118:0/1497983639 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f36640905f0 msgr2=0x7f366408c0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-05T19:27:58.388 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-05T19:27:58.379+0000 7f3668e77700 1 --2- 172.21.15.118:0/1497983639 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f36640905f0 0x7f366408c0c0 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f365c009540 tx=0x7f365c017000 comp rx=0 tx=0).stop 2023-10-05T19:27:58.388 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-05T19:27:58.379+0000 7f3668e77700 1 -- 172.21.15.118:0/1497983639 shutdown_connections 2023-10-05T19:27:58.388 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-05T19:27:58.379+0000 7f3668e77700 1 --2- 172.21.15.118:0/1497983639 >> [v2:172.21.15.167:6824/29889,v1:172.21.15.167:6825/29889] conn(0x7f36500323e0 0x7f3650034890 unknown :-1 s=CLOSED pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-05T19:27:58.388 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-05T19:27:58.379+0000 7f3668e77700 1 --2- 172.21.15.118:0/1497983639 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f36640905f0 0x7f366408c0c0 unknown :-1 s=CLOSED pgs=64 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-05T19:27:58.388 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-05T19:27:58.379+0000 7f3668e77700 1 -- 172.21.15.118:0/1497983639 >> 172.21.15.118:0/1497983639 conn(0x7f366409c5d0 msgr2=0x7f3664094930 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-05T19:27:58.388 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-05T19:27:58.379+0000 7f3668e77700 1 -- 172.21.15.118:0/1497983639 shutdown_connections 2023-10-05T19:27:58.388 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-05T19:27:58.379+0000 7f3668e77700 1 -- 172.21.15.118:0/1497983639 wait complete. 2023-10-05T19:27:58.399 INFO:tasks.ceph.ceph_manager.ceph:all up! 2023-10-05T19:27:58.399 DEBUG:teuthology.orchestra.run.smithi118:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-05T19:27:58.502 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-05T19:27:58.495+0000 7efed29f3700 1 Processor -- start 2023-10-05T19:27:58.503 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-05T19:27:58.495+0000 7efed29f3700 1 -- start start 2023-10-05T19:27:58.503 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-05T19:27:58.495+0000 7efed29f3700 1 --2- >> v2:172.21.15.118:3300/0 conn(0x7efecc12e280 0x7efecc132db0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-05T19:27:58.503 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-05T19:27:58.495+0000 7efed29f3700 1 -- --> v1:172.21.15.118:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efecc091240 con 0x7efecc093bb0 2023-10-05T19:27:58.503 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-05T19:27:58.495+0000 7efed29f3700 1 -- --> v2:172.21.15.118:3300/0 -- mon_getmap magic: 0 v1 -- 0x7efecc091800 con 0x7efecc12e280 2023-10-05T19:27:58.503 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-05T19:27:58.495+0000 7efed19f1700 1 --1- >> v1:172.21.15.118:6789/0 conn(0x7efecc093bb0 0x7efecc12db70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.118:6789/0 says I am v1:172.21.15.118:52050/0 (socket says 172.21.15.118:52050) 2023-10-05T19:27:58.503 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-05T19:27:58.495+0000 7efed19f1700 1 -- 172.21.15.118:0/1651759265 learned_addr learned my addr 172.21.15.118:0/1651759265 (peer_addr_for_me v1:172.21.15.118:0/0) 2023-10-05T19:27:58.504 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-05T19:27:58.495+0000 7efed11f0700 1 --2- >> v2:172.21.15.118:3300/0 conn(0x7efecc12e280 0x7efecc132db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-05T19:27:58.505 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-05T19:27:58.495+0000 7efed09ef700 1 -- 172.21.15.118:0/1651759265 <== mon.0 v1:172.21.15.118:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2809728048 0 0) 0x7efecc091240 con 0x7efecc093bb0 2023-10-05T19:27:58.506 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-05T19:27:58.495+0000 7efed09ef700 1 -- 172.21.15.118:0/1651759265 --> v1:172.21.15.118:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7efeb8003570 con 0x7efecc093bb0 2023-10-05T19:27:58.506 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-05T19:27:58.495+0000 7efed11f0700 1 -- 172.21.15.118:0/1651759265 >> v1:172.21.15.118:6789/0 conn(0x7efecc093bb0 legacy=0x7efecc12db70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-05T19:27:58.506 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-05T19:27:58.495+0000 7efed11f0700 1 -- 172.21.15.118:0/1651759265 --> v2:172.21.15.118:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efecc0947e0 con 0x7efecc12e280 2023-10-05T19:27:58.506 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-05T19:27:58.495+0000 7efed11f0700 1 --2- 172.21.15.118:0/1651759265 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7efecc12e280 0x7efecc132db0 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7efebc009860 tx=0x7efebc0091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5f241692f7feb1a server_cookie=0 in_seq=0 out_seq=0 2023-10-05T19:27:58.506 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-05T19:27:58.495+0000 7efed09ef700 1 -- 172.21.15.118:0/1651759265 <== mon.0 v2:172.21.15.118:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7efebc011630 con 0x7efecc12e280 2023-10-05T19:27:58.506 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-05T19:27:58.495+0000 7efed09ef700 1 -- 172.21.15.118:0/1651759265 <== mon.0 v2:172.21.15.118:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efebc011790 con 0x7efecc12e280 2023-10-05T19:27:58.506 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-05T19:27:58.495+0000 7efed09ef700 1 -- 172.21.15.118:0/1651759265 <== mon.0 v2:172.21.15.118:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7efebc011920 con 0x7efecc12e280 2023-10-05T19:27:58.506 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-05T19:27:58.499+0000 7efed29f3700 1 -- 172.21.15.118:0/1651759265 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7efecc12e280 msgr2=0x7efecc132db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-05T19:27:58.506 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-05T19:27:58.499+0000 7efed29f3700 1 --2- 172.21.15.118:0/1651759265 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7efecc12e280 0x7efecc132db0 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7efebc009860 tx=0x7efebc0091a0 comp rx=0 tx=0).stop 2023-10-05T19:27:58.507 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-05T19:27:58.499+0000 7efed29f3700 1 -- 172.21.15.118:0/1651759265 shutdown_connections 2023-10-05T19:27:58.507 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-05T19:27:58.499+0000 7efed29f3700 1 --2- 172.21.15.118:0/1651759265 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7efecc12e280 0x7efecc132db0 unknown :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-05T19:27:58.507 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-05T19:27:58.499+0000 7efed29f3700 1 -- 172.21.15.118:0/1651759265 >> 172.21.15.118:0/1651759265 conn(0x7efecc0981b0 msgr2=0x7efecc0985b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-05T19:27:58.507 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-05T19:27:58.499+0000 7efed29f3700 1 -- 172.21.15.118:0/1651759265 shutdown_connections 2023-10-05T19:27:58.507 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-05T19:27:58.499+0000 7efed29f3700 1 -- 172.21.15.118:0/1651759265 wait complete. 2023-10-05T19:27:58.508 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-05T19:27:58.499+0000 7efed29f3700 1 Processor -- start 2023-10-05T19:27:58.508 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-05T19:27:58.499+0000 7efed29f3700 1 -- start start 2023-10-05T19:27:58.508 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-05T19:27:58.499+0000 7efed29f3700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7efecc093bb0 0x7efecc129650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-05T19:27:58.509 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-05T19:27:58.499+0000 7efed29f3700 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efecc12d070 con 0x7efecc093bb0 2023-10-05T19:27:58.509 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-05T19:27:58.499+0000 7efed19f1700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7efecc093bb0 0x7efecc129650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-05T19:27:58.509 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-05T19:27:58.499+0000 7efed19f1700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7efecc093bb0 0x7efecc129650 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.118:3300/0 says I am v2:172.21.15.118:55732/0 (socket says 172.21.15.118:55732) 2023-10-05T19:27:58.509 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-05T19:27:58.499+0000 7efed19f1700 1 -- 172.21.15.118:0/54253624 learned_addr learned my addr 172.21.15.118:0/54253624 (peer_addr_for_me v2:172.21.15.118:0/0) 2023-10-05T19:27:58.509 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-05T19:27:58.503+0000 7efed19f1700 1 -- 172.21.15.118:0/54253624 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efebc009420 con 0x7efecc093bb0 2023-10-05T19:27:58.510 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-05T19:27:58.503+0000 7efed19f1700 1 --2- 172.21.15.118:0/54253624 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7efecc093bb0 0x7efecc129650 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7efec800c490 tx=0x7efec8007e00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-05T19:27:58.511 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-05T19:27:58.503+0000 7efec1ffb700 1 -- 172.21.15.118:0/54253624 <== mon.0 v2:172.21.15.118:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7efec8005b30 con 0x7efecc093bb0 2023-10-05T19:27:58.511 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-05T19:27:58.503+0000 7efec1ffb700 1 -- 172.21.15.118:0/54253624 <== mon.0 v2:172.21.15.118:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efec8005c90 con 0x7efecc093bb0 2023-10-05T19:27:58.511 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-05T19:27:58.503+0000 7efed29f3700 1 -- 172.21.15.118:0/54253624 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efecc12b9a0 con 0x7efecc093bb0 2023-10-05T19:27:58.511 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-05T19:27:58.503+0000 7efec1ffb700 1 -- 172.21.15.118:0/54253624 <== mon.0 v2:172.21.15.118:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7efec80070d0 con 0x7efecc093bb0 2023-10-05T19:27:58.511 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-05T19:27:58.503+0000 7efed29f3700 1 -- 172.21.15.118:0/54253624 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efecc12c140 con 0x7efecc093bb0 2023-10-05T19:27:58.511 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-05T19:27:58.503+0000 7efec1ffb700 1 -- 172.21.15.118:0/54253624 <== mon.0 v2:172.21.15.118:3300/0 4 ==== mgrmap(e 5) v1 ==== 40329+0+0 (secure 0 0 0) 0x7efec8007300 con 0x7efecc093bb0 2023-10-05T19:27:58.511 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-05T19:27:58.503+0000 7efec1ffb700 1 --2- 172.21.15.118:0/54253624 >> [v2:172.21.15.167:6824/29889,v1:172.21.15.167:6825/29889] conn(0x7efeb0032390 0x7efeb0034840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-05T19:27:58.512 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-05T19:27:58.503+0000 7efec1ffb700 1 -- 172.21.15.118:0/54253624 <== mon.0 v2:172.21.15.118:3300/0 5 ==== osd_map(14..14 src has 1..14) v4 ==== 3478+0+0 (secure 0 0 0) 0x7efec8042fb0 con 0x7efecc093bb0 2023-10-05T19:27:58.512 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-05T19:27:58.503+0000 7efed29f3700 1 -- 172.21.15.118:0/54253624 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efecc05ecb0 con 0x7efecc093bb0 2023-10-05T19:27:58.512 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-05T19:27:58.503+0000 7efed11f0700 1 --2- 172.21.15.118:0/54253624 >> [v2:172.21.15.167:6824/29889,v1:172.21.15.167:6825/29889] conn(0x7efeb0032390 0x7efeb0034840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-05T19:27:58.512 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-05T19:27:58.503+0000 7efed11f0700 1 --2- 172.21.15.118:0/54253624 >> [v2:172.21.15.167:6824/29889,v1:172.21.15.167:6825/29889] conn(0x7efeb0032390 0x7efeb0034840 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7efebc0091e0 tx=0x7efebc016000 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-05T19:27:58.520 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-05T19:27:58.515+0000 7efec1ffb700 1 -- 172.21.15.118:0/54253624 <== mon.0 v2:172.21.15.118:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166807 (secure 0 0 0) 0x7efec8017070 con 0x7efecc093bb0 2023-10-05T19:27:58.685 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-05T19:27:58.679+0000 7efed29f3700 1 -- 172.21.15.118:0/54253624 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7efecc04b5c0 con 0x7efecc093bb0 2023-10-05T19:27:58.686 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-05T19:27:58.679+0000 7efec1ffb700 1 -- 172.21.15.118:0/54253624 <== mon.0 v2:172.21.15.118:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v14) v1 ==== 74+0+9003 (secure 0 0 0) 0x7efec800fac0 con 0x7efecc093bb0 2023-10-05T19:27:58.686 INFO:teuthology.orchestra.run.smithi118.stdout: 2023-10-05T19:27:58.686 INFO:teuthology.orchestra.run.smithi118.stdout:{"epoch":14,"fsid":"8c4bd0d1-cbcb-4713-96b7-dcb14bcaf584","created":"2023-10-05T19:27:40.755769+0000","modified":"2023-10-05T19:27:57.740909+0000","last_up_change":"2023-10-05T19:27:50.196733+0000","last_in_change":"2023-10-05T19:27:41.811702+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-05T19:27:50.980612+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"12","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}},{"pool":2,"pool_name":"rbd","create_time":"2023-10-05T19:27:56.423831+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"14","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"61ba9225-834e-4288-b391-7ffa540f7334","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":13,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6800","nonce":27690},{"type":"v1","addr":"172.21.15.118:6801","nonce":27690}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6802","nonce":27690},{"type":"v1","addr":"172.21.15.118:6803","nonce":27690}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6806","nonce":27690},{"type":"v1","addr":"172.21.15.118:6807","nonce":27690}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6804","nonce":27690},{"type":"v1","addr":"172.21.15.118:6805","nonce":27690}]},"public_addr":"172.21.15.118:6801/27690","cluster_addr":"172.21.15.118:6803/27690","heartbeat_back_addr":"172.21.15.118:6807/27690","heartbeat_front_addr":"172.21.15.118:6805/27690","state":["exists","up"]},{"osd":1,"uuid":"15a3631e-9f20-4b33-9ee3-39eb73e13709","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":10,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6800","nonce":30035},{"type":"v1","addr":"172.21.15.167:6801","nonce":30035}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6802","nonce":30035},{"type":"v1","addr":"172.21.15.167:6803","nonce":30035}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6806","nonce":30035},{"type":"v1","addr":"172.21.15.167:6807","nonce":30035}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6804","nonce":30035},{"type":"v1","addr":"172.21.15.167:6805","nonce":30035}]},"public_addr":"172.21.15.167:6801/30035","cluster_addr":"172.21.15.167:6803/30035","heartbeat_back_addr":"172.21.15.167:6807/30035","heartbeat_front_addr":"172.21.15.167:6805/30035","state":["exists","up"]},{"osd":2,"uuid":"5712158a-931c-4ce7-adf0-5c9fbb59d521","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":13,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6816","nonce":30037},{"type":"v1","addr":"172.21.15.167:6817","nonce":30037}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6818","nonce":30037},{"type":"v1","addr":"172.21.15.167:6819","nonce":30037}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6822","nonce":30037},{"type":"v1","addr":"172.21.15.167:6823","nonce":30037}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6820","nonce":30037},{"type":"v1","addr":"172.21.15.167:6821","nonce":30037}]},"public_addr":"172.21.15.167:6817/30037","cluster_addr":"172.21.15.167:6819/30037","heartbeat_back_addr":"172.21.15.167:6823/30037","heartbeat_front_addr":"172.21.15.167:6821/30037","state":["exists","up"]},{"osd":3,"uuid":"f838ed4e-05a7-49ef-ad9d-8f7dd7be30f1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":13,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6808","nonce":30036},{"type":"v1","addr":"172.21.15.167:6809","nonce":30036}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6810","nonce":30036},{"type":"v1","addr":"172.21.15.167:6811","nonce":30036}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6814","nonce":30036},{"type":"v1","addr":"172.21.15.167:6815","nonce":30036}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6812","nonce":30036},{"type":"v1","addr":"172.21.15.167:6813","nonce":30036}]},"public_addr":"172.21.15.167:6809/30036","cluster_addr":"172.21.15.167:6811/30036","heartbeat_back_addr":"172.21.15.167:6815/30036","heartbeat_front_addr":"172.21.15.167:6813/30036","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-05T19:27:47.190191+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-05T19:27:47.302630+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-05T19:27:47.374216+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-05T19:27:47.350455+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-05T19:27:58.689 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-05T19:27:58.683+0000 7efed29f3700 1 -- 172.21.15.118:0/54253624 >> [v2:172.21.15.167:6824/29889,v1:172.21.15.167:6825/29889] conn(0x7efeb0032390 msgr2=0x7efeb0034840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-05T19:27:58.689 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-05T19:27:58.683+0000 7efed29f3700 1 --2- 172.21.15.118:0/54253624 >> [v2:172.21.15.167:6824/29889,v1:172.21.15.167:6825/29889] conn(0x7efeb0032390 0x7efeb0034840 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7efebc0091e0 tx=0x7efebc016000 comp rx=0 tx=0).stop 2023-10-05T19:27:58.689 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-05T19:27:58.683+0000 7efed29f3700 1 -- 172.21.15.118:0/54253624 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7efecc093bb0 msgr2=0x7efecc129650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-05T19:27:58.689 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-05T19:27:58.683+0000 7efed29f3700 1 --2- 172.21.15.118:0/54253624 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7efecc093bb0 0x7efecc129650 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7efec800c490 tx=0x7efec8007e00 comp rx=0 tx=0).stop 2023-10-05T19:27:58.689 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-05T19:27:58.683+0000 7efed29f3700 1 -- 172.21.15.118:0/54253624 shutdown_connections 2023-10-05T19:27:58.689 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-05T19:27:58.683+0000 7efed29f3700 1 --2- 172.21.15.118:0/54253624 >> [v2:172.21.15.167:6824/29889,v1:172.21.15.167:6825/29889] conn(0x7efeb0032390 0x7efeb0034840 unknown :-1 s=CLOSED pgs=29 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-05T19:27:58.689 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-05T19:27:58.683+0000 7efed29f3700 1 --2- 172.21.15.118:0/54253624 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7efecc093bb0 0x7efecc129650 unknown :-1 s=CLOSED pgs=67 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-05T19:27:58.689 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-05T19:27:58.683+0000 7efed29f3700 1 -- 172.21.15.118:0/54253624 >> 172.21.15.118:0/54253624 conn(0x7efecc0981b0 msgr2=0x7efecc06c2e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-05T19:27:58.689 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-05T19:27:58.683+0000 7efed29f3700 1 -- 172.21.15.118:0/54253624 shutdown_connections 2023-10-05T19:27:58.689 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-05T19:27:58.683+0000 7efed29f3700 1 -- 172.21.15.118:0/54253624 wait complete. 2023-10-05T19:27:58.701 DEBUG:teuthology.orchestra.run.smithi118:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 flush_pg_stats 2023-10-05T19:27:58.701 DEBUG:teuthology.orchestra.run.smithi118:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 flush_pg_stats 2023-10-05T19:27:58.701 DEBUG:teuthology.orchestra.run.smithi118:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 flush_pg_stats 2023-10-05T19:27:58.702 DEBUG:teuthology.orchestra.run.smithi118:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.3 flush_pg_stats 2023-10-05T19:27:58.796 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-05T19:27:58.787+0000 7fd575b76700 1 Processor -- start 2023-10-05T19:27:58.796 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-05T19:27:58.791+0000 7fd575b76700 1 -- start start 2023-10-05T19:27:58.797 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-05T19:27:58.791+0000 7fd575b76700 1 --2- >> v2:172.21.15.118:3300/0 conn(0x7fd57005dac0 0x7fd570123e60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-05T19:27:58.797 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-05T19:27:58.791+0000 7fd575b76700 1 -- --> v1:172.21.15.118:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd570095330 con 0x7fd570094e20 2023-10-05T19:27:58.797 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-05T19:27:58.791+0000 7fd575b76700 1 -- --> v2:172.21.15.118:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd57006da80 con 0x7fd57005dac0 2023-10-05T19:27:58.797 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-05T19:27:58.791+0000 7fd574b74700 1 --2- >> v2:172.21.15.118:3300/0 conn(0x7fd57005dac0 0x7fd570123e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-05T19:27:58.797 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-05T19:27:58.791+0000 7fd56ffff700 1 --1- >> v1:172.21.15.118:6789/0 conn(0x7fd570094e20 0x7fd5701243a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.118:6789/0 says I am v1:172.21.15.118:52058/0 (socket says 172.21.15.118:52058) 2023-10-05T19:27:58.798 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-05T19:27:58.791+0000 7fd56ffff700 1 -- 172.21.15.118:0/2138789138 learned_addr learned my addr 172.21.15.118:0/2138789138 (peer_addr_for_me v1:172.21.15.118:0/0) 2023-10-05T19:27:58.798 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-05T19:27:58.791+0000 7fd574b74700 1 -- 172.21.15.118:0/2138789138 >> v1:172.21.15.118:6789/0 conn(0x7fd570094e20 legacy=0x7fd5701243a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-05T19:27:58.798 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-05T19:27:58.791+0000 7fd574b74700 1 -- 172.21.15.118:0/2138789138 --> v2:172.21.15.118:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd570124ab0 con 0x7fd57005dac0 2023-10-05T19:27:58.799 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-05T19:27:58.791+0000 7fd574b74700 1 --2- 172.21.15.118:0/2138789138 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fd57005dac0 0x7fd570123e60 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7fd564009860 tx=0x7fd5640091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5605a8a8c2f29ab0 server_cookie=0 in_seq=0 out_seq=0 2023-10-05T19:27:58.799 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-05T19:27:58.791+0000 7fd56f7fe700 1 -- 172.21.15.118:0/2138789138 <== mon.0 v2:172.21.15.118:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fd564011630 con 0x7fd57005dac0 2023-10-05T19:27:58.799 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-05T19:27:58.791+0000 7fd56f7fe700 1 -- 172.21.15.118:0/2138789138 <== mon.0 v2:172.21.15.118:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd564011790 con 0x7fd57005dac0 2023-10-05T19:27:58.799 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-05T19:27:58.791+0000 7fd56f7fe700 1 -- 172.21.15.118:0/2138789138 <== mon.0 v2:172.21.15.118:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fd564011940 con 0x7fd57005dac0 2023-10-05T19:27:58.799 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-05T19:27:58.791+0000 7fd575b76700 1 -- 172.21.15.118:0/2138789138 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fd57005dac0 msgr2=0x7fd570123e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-05T19:27:58.800 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-05T19:27:58.791+0000 7fd575b76700 1 --2- 172.21.15.118:0/2138789138 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fd57005dac0 0x7fd570123e60 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7fd564009860 tx=0x7fd5640091a0 comp rx=0 tx=0).stop 2023-10-05T19:27:58.800 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-05T19:27:58.791+0000 7fd575b76700 1 -- 172.21.15.118:0/2138789138 shutdown_connections 2023-10-05T19:27:58.800 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-05T19:27:58.791+0000 7fd575b76700 1 --2- 172.21.15.118:0/2138789138 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fd57005dac0 0x7fd570123e60 unknown :-1 s=CLOSED pgs=69 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-05T19:27:58.800 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-05T19:27:58.791+0000 7fd575b76700 1 -- 172.21.15.118:0/2138789138 >> 172.21.15.118:0/2138789138 conn(0x7fd570095970 msgr2=0x7fd570095d70 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-05T19:27:58.800 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-05T19:27:58.791+0000 7fd575b76700 1 -- 172.21.15.118:0/2138789138 shutdown_connections 2023-10-05T19:27:58.800 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-05T19:27:58.791+0000 7fd575b76700 1 -- 172.21.15.118:0/2138789138 wait complete. 2023-10-05T19:27:58.801 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-05T19:27:58.791+0000 7fd575b76700 1 Processor -- start 2023-10-05T19:27:58.801 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-05T19:27:58.791+0000 7fd575b76700 1 -- start start 2023-10-05T19:27:58.802 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-05T19:27:58.795+0000 7fd575b76700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fd57005dac0 0x7fd57011f770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-05T19:27:58.802 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-05T19:27:58.795+0000 7fd575b76700 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd570097330 con 0x7fd57005dac0 2023-10-05T19:27:58.802 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-05T19:27:58.795+0000 7fd574b74700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fd57005dac0 0x7fd57011f770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-05T19:27:58.802 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-05T19:27:58.795+0000 7fd574b74700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fd57005dac0 0x7fd57011f770 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.118:3300/0 says I am v2:172.21.15.118:55740/0 (socket says 172.21.15.118:55740) 2023-10-05T19:27:58.802 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-05T19:27:58.795+0000 7fd574b74700 1 -- 172.21.15.118:0/2560005492 learned_addr learned my addr 172.21.15.118:0/2560005492 (peer_addr_for_me v2:172.21.15.118:0/0) 2023-10-05T19:27:58.802 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-05T19:27:58.795+0000 7fd574b74700 1 -- 172.21.15.118:0/2560005492 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd564009420 con 0x7fd57005dac0 2023-10-05T19:27:58.802 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-05T19:27:58.795+0000 7fd574b74700 1 --2- 172.21.15.118:0/2560005492 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fd57005dac0 0x7fd57011f770 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7fd564000c00 tx=0x7fd564016000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-05T19:27:58.802 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-05T19:27:58.795+0000 7fd56cff9700 1 -- 172.21.15.118:0/2560005492 <== mon.0 v2:172.21.15.118:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fd56401a660 con 0x7fd57005dac0 2023-10-05T19:27:58.802 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-05T19:27:58.795+0000 7fd56cff9700 1 -- 172.21.15.118:0/2560005492 <== mon.0 v2:172.21.15.118:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd56401a7c0 con 0x7fd57005dac0 2023-10-05T19:27:58.802 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-05T19:27:58.795+0000 7fd575b76700 1 -- 172.21.15.118:0/2560005492 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd570122f00 con 0x7fd57005dac0 2023-10-05T19:27:58.803 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-05T19:27:58.795+0000 7fd56cff9700 1 -- 172.21.15.118:0/2560005492 <== mon.0 v2:172.21.15.118:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fd56401aab0 con 0x7fd57005dac0 2023-10-05T19:27:58.803 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-05T19:27:58.795+0000 7fd575b76700 1 -- 172.21.15.118:0/2560005492 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd570121170 con 0x7fd57005dac0 2023-10-05T19:27:58.803 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-05T19:27:58.795+0000 7fd56cff9700 1 -- 172.21.15.118:0/2560005492 <== mon.0 v2:172.21.15.118:3300/0 4 ==== mgrmap(e 5) v1 ==== 40329+0+0 (secure 0 0 0) 0x7fd56401ac60 con 0x7fd57005dac0 2023-10-05T19:27:58.803 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-05T19:27:58.795+0000 7fd56cff9700 1 --2- 172.21.15.118:0/2560005492 >> [v2:172.21.15.167:6824/29889,v1:172.21.15.167:6825/29889] conn(0x7fd5540323e0 0x7fd554034890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-05T19:27:58.804 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-05T19:27:58.795+0000 7fd56cff9700 1 -- 172.21.15.118:0/2560005492 <== mon.0 v2:172.21.15.118:3300/0 5 ==== osd_map(15..15 src has 1..15) v4 ==== 3478+0+0 (secure 0 0 0) 0x7fd564045d00 con 0x7fd57005dac0 2023-10-05T19:27:58.804 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-05T19:27:58.795+0000 7fd56ffff700 1 --2- 172.21.15.118:0/2560005492 >> [v2:172.21.15.167:6824/29889,v1:172.21.15.167:6825/29889] conn(0x7fd5540323e0 0x7fd554034890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-05T19:27:58.804 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-05T19:27:58.795+0000 7fd56ffff700 1 --2- 172.21.15.118:0/2560005492 >> [v2:172.21.15.167:6824/29889,v1:172.21.15.167:6825/29889] conn(0x7fd5540323e0 0x7fd554034890 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7fd55c007fd0 tx=0x7fd55c007b20 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-05T19:27:58.804 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-05T19:27:58.799+0000 7fd575b76700 1 --2- 172.21.15.118:0/2560005492 >> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] conn(0x7fd570094e20 0x7fd57011fcb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-05T19:27:58.804 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-05T19:27:58.799+0000 7fd575b76700 1 -- 172.21.15.118:0/2560005492 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fd57005ecb0 con 0x7fd570094e20 2023-10-05T19:27:58.805 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-05T19:27:58.799+0000 7fd575375700 1 --2- 172.21.15.118:0/2560005492 >> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] conn(0x7fd570094e20 0x7fd57011fcb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-05T19:27:58.805 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-05T19:27:58.799+0000 7fd575375700 1 --2- 172.21.15.118:0/2560005492 >> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] conn(0x7fd570094e20 0x7fd57011fcb0 crc :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-05T19:27:58.807 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-05T19:27:58.799+0000 7fd56cff9700 1 -- 172.21.15.118:0/2560005492 <== osd.2 v2:172.21.15.167:6816/30037 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+23999 (crc 0 0 0) 0x7fd57005ecb0 con 0x7fd570094e20 2023-10-05T19:27:58.810 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-05T19:27:58.803+0000 7f980e668700 1 Processor -- start 2023-10-05T19:27:58.810 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-05T19:27:58.803+0000 7f980e668700 1 -- start start 2023-10-05T19:27:58.811 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-05T19:27:58.803+0000 7f980e668700 1 --2- >> v2:172.21.15.118:3300/0 conn(0x7f980806b130 0x7f980812e7a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-05T19:27:58.811 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-05T19:27:58.803+0000 7f980e668700 1 -- --> v1:172.21.15.118:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f980806a660 con 0x7f9808095650 2023-10-05T19:27:58.811 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-05T19:27:58.803+0000 7f980e668700 1 -- --> v2:172.21.15.118:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f980806d020 con 0x7f980806b130 2023-10-05T19:27:58.812 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-05T19:27:58.803+0000 7f98077fe700 1 --2- >> v2:172.21.15.118:3300/0 conn(0x7f980806b130 0x7f980812e7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-05T19:27:58.812 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-05T19:27:58.803+0000 7f9807fff700 1 --1- >> v1:172.21.15.118:6789/0 conn(0x7f9808095650 0x7f980806aa20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.118:6789/0 says I am v1:172.21.15.118:52074/0 (socket says 172.21.15.118:52074) 2023-10-05T19:27:58.812 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-05T19:27:58.803+0000 7f9807fff700 1 -- 172.21.15.118:0/3041201356 learned_addr learned my addr 172.21.15.118:0/3041201356 (peer_addr_for_me v1:172.21.15.118:0/0) 2023-10-05T19:27:58.812 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-05T19:27:58.803+0000 7f9806ffd700 1 -- 172.21.15.118:0/3041201356 <== mon.0 v1:172.21.15.118:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3763673466 0 0) 0x7f980806a660 con 0x7f9808095650 2023-10-05T19:27:58.812 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-05T19:27:58.803+0000 7f9806ffd700 1 -- 172.21.15.118:0/3041201356 --> v1:172.21.15.118:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f97f0003570 con 0x7f9808095650 2023-10-05T19:27:58.812 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-05T19:27:58.803+0000 7f98077fe700 1 -- 172.21.15.118:0/3041201356 >> v1:172.21.15.118:6789/0 conn(0x7f9808095650 legacy=0x7f980806aa20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-05T19:27:58.813 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-05T19:27:58.803+0000 7f98077fe700 1 -- 172.21.15.118:0/3041201356 --> v2:172.21.15.118:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f980806b570 con 0x7f980806b130 2023-10-05T19:27:58.813 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-05T19:27:58.803+0000 7f98077fe700 1 --2- 172.21.15.118:0/3041201356 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f980806b130 0x7f980812e7a0 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f97fc009860 tx=0x7f97fc0091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3c7accadd23cf5a2 server_cookie=0 in_seq=0 out_seq=0 2023-10-05T19:27:58.813 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-05T19:27:58.803+0000 7f9806ffd700 1 -- 172.21.15.118:0/3041201356 <== mon.0 v2:172.21.15.118:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f97fc011630 con 0x7f980806b130 2023-10-05T19:27:58.813 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-05T19:27:58.803+0000 7f9806ffd700 1 -- 172.21.15.118:0/3041201356 <== mon.0 v2:172.21.15.118:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f97fc011790 con 0x7f980806b130 2023-10-05T19:27:58.813 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-05T19:27:58.803+0000 7f9806ffd700 1 -- 172.21.15.118:0/3041201356 <== mon.0 v2:172.21.15.118:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f97fc011630 con 0x7f980806b130 2023-10-05T19:27:58.814 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-05T19:27:58.807+0000 7f980e668700 1 -- 172.21.15.118:0/3041201356 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f980806b130 msgr2=0x7f980812e7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-05T19:27:58.815 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-05T19:27:58.807+0000 7f980e668700 1 --2- 172.21.15.118:0/3041201356 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f980806b130 0x7f980812e7a0 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f97fc009860 tx=0x7f97fc0091a0 comp rx=0 tx=0).stop 2023-10-05T19:27:58.815 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-05T19:27:58.807+0000 7f980e668700 1 -- 172.21.15.118:0/3041201356 shutdown_connections 2023-10-05T19:27:58.815 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-05T19:27:58.807+0000 7f980e668700 1 --2- 172.21.15.118:0/3041201356 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f980806b130 0x7f980812e7a0 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-05T19:27:58.815 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-05T19:27:58.807+0000 7f980e668700 1 -- 172.21.15.118:0/3041201356 >> 172.21.15.118:0/3041201356 conn(0x7f9808098f60 msgr2=0x7f98080949f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-05T19:27:58.815 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-05T19:27:58.807+0000 7f980e668700 1 -- 172.21.15.118:0/3041201356 shutdown_connections 2023-10-05T19:27:58.815 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-05T19:27:58.807+0000 7f980e668700 1 -- 172.21.15.118:0/3041201356 wait complete. 2023-10-05T19:27:58.815 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-05T19:27:58.807+0000 7f980e668700 1 Processor -- start 2023-10-05T19:27:58.815 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-05T19:27:58.807+0000 7f980e668700 1 -- start start 2023-10-05T19:27:58.816 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-05T19:27:58.807+0000 7f779e67a700 1 Processor -- start 2023-10-05T19:27:58.816 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-05T19:27:58.807+0000 7f779e67a700 1 -- start start 2023-10-05T19:27:58.816 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-05T19:27:58.807+0000 7f980e668700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f980806b130 0x7f9808119a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-05T19:27:58.817 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-05T19:27:58.807+0000 7f980e668700 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9808129f70 con 0x7f980806b130 2023-10-05T19:27:58.817 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-05T19:27:58.807+0000 7f779e67a700 1 --2- >> v2:172.21.15.118:3300/0 conn(0x7f7798095ff0 0x7f77980963c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-05T19:27:58.817 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-05T19:27:58.807+0000 7f9807fff700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f980806b130 0x7f9808119a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-05T19:27:58.817 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-05T19:27:58.807+0000 7f9807fff700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f980806b130 0x7f9808119a60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.118:3300/0 says I am v2:172.21.15.118:55748/0 (socket says 172.21.15.118:55748) 2023-10-05T19:27:58.817 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-05T19:27:58.807+0000 7f9807fff700 1 -- 172.21.15.118:0/2418650951 learned_addr learned my addr 172.21.15.118:0/2418650951 (peer_addr_for_me v2:172.21.15.118:0/0) 2023-10-05T19:27:58.818 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-05T19:27:58.807+0000 7f779e67a700 1 -- --> v1:172.21.15.118:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7798092760 con 0x7f77980905f0 2023-10-05T19:27:58.818 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-05T19:27:58.807+0000 7f779e67a700 1 -- --> v2:172.21.15.118:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f77980918a0 con 0x7f7798095ff0 2023-10-05T19:27:58.818 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-05T19:27:58.807+0000 7f779d678700 1 --2- >> v2:172.21.15.118:3300/0 conn(0x7f7798095ff0 0x7f77980963c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-05T19:27:58.818 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-05T19:27:58.807+0000 7f779d678700 1 --2- >> v2:172.21.15.118:3300/0 conn(0x7f7798095ff0 0x7f77980963c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.118:3300/0 says I am v2:172.21.15.118:55752/0 (socket says 172.21.15.118:55752) 2023-10-05T19:27:58.818 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-05T19:27:58.807+0000 7f779d678700 1 -- 172.21.15.118:0/2404026088 learned_addr learned my addr 172.21.15.118:0/2404026088 (peer_addr_for_me v2:172.21.15.118:0/0) 2023-10-05T19:27:58.818 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-05T19:27:58.807+0000 7f779ce77700 1 --1- >> v1:172.21.15.118:6789/0 conn(0x7f77980905f0 0x7f77980909c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.118:6789/0 says I am v1:172.21.15.118:52082/0 (socket says 172.21.15.118:52082) 2023-10-05T19:27:58.819 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-05T19:27:58.807+0000 7f9807fff700 1 -- 172.21.15.118:0/2418650951 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f97fc009420 con 0x7f980806b130 2023-10-05T19:27:58.819 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-05T19:27:58.807+0000 7f779d678700 1 -- 172.21.15.118:0/2404026088 >> v1:172.21.15.118:6789/0 conn(0x7f77980905f0 legacy=0x7f77980909c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-05T19:27:58.819 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-05T19:27:58.807+0000 7f779d678700 1 -- 172.21.15.118:0/2404026088 --> v2:172.21.15.118:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7798093ba0 con 0x7f7798095ff0 2023-10-05T19:27:58.819 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-05T19:27:58.807+0000 7f9807fff700 1 --2- 172.21.15.118:0/2418650951 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f980806b130 0x7f9808119a60 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f980000cc00 tx=0x7f9800001b00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-05T19:27:58.820 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-05T19:27:58.807+0000 7f97effff700 1 -- 172.21.15.118:0/2418650951 <== mon.0 v2:172.21.15.118:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f980000a980 con 0x7f980806b130 2023-10-05T19:27:58.820 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-05T19:27:58.807+0000 7f779d678700 1 --2- 172.21.15.118:0/2404026088 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f7798095ff0 0x7f77980963c0 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f7790009ea0 tx=0x7f7790009840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=79bf63e721741c8c server_cookie=0 in_seq=0 out_seq=0 2023-10-05T19:27:58.820 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-05T19:27:58.807+0000 7f97effff700 1 -- 172.21.15.118:0/2418650951 <== mon.0 v2:172.21.15.118:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f980000aae0 con 0x7f980806b130 2023-10-05T19:27:58.821 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-05T19:27:58.807+0000 7f980e668700 1 -- 172.21.15.118:0/2418650951 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f980811c5e0 con 0x7f980806b130 2023-10-05T19:27:58.821 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-05T19:27:58.807+0000 7f97effff700 1 -- 172.21.15.118:0/2418650951 <== mon.0 v2:172.21.15.118:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f980000adb0 con 0x7f980806b130 2023-10-05T19:27:58.821 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-05T19:27:58.807+0000 7f980e668700 1 -- 172.21.15.118:0/2418650951 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f980811cd50 con 0x7f980806b130 2023-10-05T19:27:58.821 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-05T19:27:58.811+0000 7f778ffff700 1 -- 172.21.15.118:0/2404026088 <== mon.0 v2:172.21.15.118:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f779000e070 con 0x7f7798095ff0 2023-10-05T19:27:58.821 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-05T19:27:58.811+0000 7f778ffff700 1 -- 172.21.15.118:0/2404026088 <== mon.0 v2:172.21.15.118:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f779000b900 con 0x7f7798095ff0 2023-10-05T19:27:58.822 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-05T19:27:58.811+0000 7f778ffff700 1 -- 172.21.15.118:0/2404026088 <== mon.0 v2:172.21.15.118:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f77900092e0 con 0x7f7798095ff0 2023-10-05T19:27:58.822 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-05T19:27:58.811+0000 7f97effff700 1 -- 172.21.15.118:0/2418650951 <== mon.0 v2:172.21.15.118:3300/0 4 ==== mgrmap(e 5) v1 ==== 40329+0+0 (secure 0 0 0) 0x7f98000070f0 con 0x7f980806b130 2023-10-05T19:27:58.822 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-05T19:27:58.811+0000 7f779e67a700 1 -- 172.21.15.118:0/2404026088 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f7798095ff0 msgr2=0x7f77980963c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-05T19:27:58.823 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-05T19:27:58.811+0000 7f779e67a700 1 --2- 172.21.15.118:0/2404026088 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f7798095ff0 0x7f77980963c0 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f7790009ea0 tx=0x7f7790009840 comp rx=0 tx=0).stop 2023-10-05T19:27:58.823 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-05T19:27:58.811+0000 7f779e67a700 1 -- 172.21.15.118:0/2404026088 shutdown_connections 2023-10-05T19:27:58.823 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-05T19:27:58.811+0000 7f779e67a700 1 --2- 172.21.15.118:0/2404026088 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f7798095ff0 0x7f77980963c0 unknown :-1 s=CLOSED pgs=75 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-05T19:27:58.823 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-05T19:27:58.811+0000 7f779e67a700 1 -- 172.21.15.118:0/2404026088 >> 172.21.15.118:0/2404026088 conn(0x7f779809c5d0 msgr2=0x7f779809c9d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-05T19:27:58.823 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-05T19:27:58.811+0000 7f779e67a700 1 -- 172.21.15.118:0/2404026088 shutdown_connections 2023-10-05T19:27:58.823 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-05T19:27:58.811+0000 7f97effff700 1 --2- 172.21.15.118:0/2418650951 >> [v2:172.21.15.167:6824/29889,v1:172.21.15.167:6825/29889] conn(0x7f97f40323e0 0x7f97f4034890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-05T19:27:58.824 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-05T19:27:58.811+0000 7f779e67a700 1 -- 172.21.15.118:0/2404026088 wait complete. 2023-10-05T19:27:58.824 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-05T19:27:58.811+0000 7f97effff700 1 -- 172.21.15.118:0/2418650951 <== mon.0 v2:172.21.15.118:3300/0 5 ==== osd_map(15..15 src has 1..15) v4 ==== 3478+0+0 (secure 0 0 0) 0x7f9800043e20 con 0x7f980806b130 2023-10-05T19:27:58.824 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-05T19:27:58.811+0000 7f98077fe700 1 --2- 172.21.15.118:0/2418650951 >> [v2:172.21.15.167:6824/29889,v1:172.21.15.167:6825/29889] conn(0x7f97f40323e0 0x7f97f4034890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-05T19:27:58.824 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-05T19:27:58.811+0000 7f779e67a700 1 Processor -- start 2023-10-05T19:27:58.825 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-05T19:27:58.811+0000 7f98077fe700 1 --2- 172.21.15.118:0/2418650951 >> [v2:172.21.15.167:6824/29889,v1:172.21.15.167:6825/29889] conn(0x7f97f40323e0 0x7f97f4034890 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7f97fc0091e0 tx=0x7f97fc016000 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-05T19:27:58.826 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-05T19:27:58.811+0000 7f779e67a700 1 -- start start 2023-10-05T19:27:58.826 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-05T19:27:58.811+0000 7f779e67a700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f77980905f0 0x7f779808c050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-05T19:27:58.826 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-05T19:27:58.811+0000 7f779e67a700 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7798095c10 con 0x7f77980905f0 2023-10-05T19:27:58.826 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-05T19:27:58.811+0000 7f779d678700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f77980905f0 0x7f779808c050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-05T19:27:58.826 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-05T19:27:58.811+0000 7f779d678700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f77980905f0 0x7f779808c050 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.118:3300/0 says I am v2:172.21.15.118:55762/0 (socket says 172.21.15.118:55762) 2023-10-05T19:27:58.827 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-05T19:27:58.811+0000 7f779d678700 1 -- 172.21.15.118:0/803032172 learned_addr learned my addr 172.21.15.118:0/803032172 (peer_addr_for_me v2:172.21.15.118:0/0) 2023-10-05T19:27:58.827 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-05T19:27:58.811+0000 7f779d678700 1 -- 172.21.15.118:0/803032172 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7790009ae0 con 0x7f77980905f0 2023-10-05T19:27:58.827 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-05T19:27:58.811+0000 7f779d678700 1 --2- 172.21.15.118:0/803032172 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f77980905f0 0x7f779808c050 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f7790000f80 tx=0x7f7790017000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-05T19:27:58.827 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-05T19:27:58.811+0000 7f778d7fa700 1 -- 172.21.15.118:0/803032172 <== mon.0 v2:172.21.15.118:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f77900127c0 con 0x7f77980905f0 2023-10-05T19:27:58.827 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-05T19:27:58.811+0000 7f778d7fa700 1 -- 172.21.15.118:0/803032172 <== mon.0 v2:172.21.15.118:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7790012920 con 0x7f77980905f0 2023-10-05T19:27:58.827 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-05T19:27:58.811+0000 7f778d7fa700 1 -- 172.21.15.118:0/803032172 <== mon.0 v2:172.21.15.118:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f7790012c10 con 0x7f77980905f0 2023-10-05T19:27:58.827 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-05T19:27:58.811+0000 7f779e67a700 1 -- 172.21.15.118:0/803032172 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f779808f760 con 0x7f77980905f0 2023-10-05T19:27:58.827 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-05T19:27:58.811+0000 7f779e67a700 1 -- 172.21.15.118:0/803032172 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f779808fa00 con 0x7f77980905f0 2023-10-05T19:27:58.828 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-05T19:27:58.815+0000 7f980e668700 1 --2- 172.21.15.118:0/2418650951 >> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] conn(0x7f980811bc70 0x7f980811a4b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-05T19:27:58.828 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-05T19:27:58.815+0000 7f980e668700 1 -- 172.21.15.118:0/2418650951 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f980806c910 con 0x7f980811bc70 2023-10-05T19:27:58.828 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-05T19:27:58.815+0000 7f778d7fa700 1 -- 172.21.15.118:0/803032172 <== mon.0 v2:172.21.15.118:3300/0 4 ==== mgrmap(e 5) v1 ==== 40329+0+0 (secure 0 0 0) 0x7f7790028460 con 0x7f77980905f0 2023-10-05T19:27:58.828 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-05T19:27:58.815+0000 7f980cc05700 1 --2- 172.21.15.118:0/2418650951 >> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] conn(0x7f980811bc70 0x7f980811a4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-05T19:27:58.829 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-05T19:27:58.815+0000 7f778d7fa700 1 --2- 172.21.15.118:0/803032172 >> [v2:172.21.15.167:6824/29889,v1:172.21.15.167:6825/29889] conn(0x7f777c032350 0x7f777c034800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-05T19:27:58.829 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-05T19:27:58.815+0000 7f778d7fa700 1 -- 172.21.15.118:0/803032172 <== mon.0 v2:172.21.15.118:3300/0 5 ==== osd_map(15..15 src has 1..15) v4 ==== 3478+0+0 (secure 0 0 0) 0x7f77900453d0 con 0x7f77980905f0 2023-10-05T19:27:58.829 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-05T19:27:58.815+0000 7f779ce77700 1 --2- 172.21.15.118:0/803032172 >> [v2:172.21.15.167:6824/29889,v1:172.21.15.167:6825/29889] conn(0x7f777c032350 0x7f777c034800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-05T19:27:58.829 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-05T19:27:58.815+0000 7f980cc05700 1 --2- 172.21.15.118:0/2418650951 >> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] conn(0x7f980811bc70 0x7f980811a4b0 crc :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-05T19:27:58.830 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-05T19:27:58.815+0000 7f779ce77700 1 --2- 172.21.15.118:0/803032172 >> [v2:172.21.15.167:6824/29889,v1:172.21.15.167:6825/29889] conn(0x7f777c032350 0x7f777c034800 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7f7794001020 tx=0x7f77940079b0 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-05T19:27:58.830 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-05T19:27:58.815+0000 7f97effff700 1 -- 172.21.15.118:0/2418650951 <== osd.1 v2:172.21.15.167:6800/30035 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+23999 (crc 0 0 0) 0x7f980806c910 con 0x7f980811bc70 2023-10-05T19:27:58.830 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-05T19:27:58.815+0000 7fd575b76700 1 -- 172.21.15.118:0/2560005492 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7fd57006d1d0 con 0x7fd570094e20 2023-10-05T19:27:58.831 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-05T19:27:58.815+0000 7f8c20136700 1 Processor -- start 2023-10-05T19:27:58.831 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-05T19:27:58.815+0000 7f8c20136700 1 -- start start 2023-10-05T19:27:58.831 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-05T19:27:58.815+0000 7f8c20136700 1 --2- >> v2:172.21.15.118:3300/0 conn(0x7f8c18088a70 0x7f8c18082020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-05T19:27:58.831 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-05T19:27:58.815+0000 7f8c20136700 1 -- --> v1:172.21.15.118:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8c1808abd0 con 0x7f8c18087f90 2023-10-05T19:27:58.831 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-05T19:27:58.815+0000 7f8c20136700 1 -- --> v2:172.21.15.118:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8c180892f0 con 0x7f8c18088a70 2023-10-05T19:27:58.831 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-05T19:27:58.815+0000 7f8c1d6d1700 1 --2- >> v2:172.21.15.118:3300/0 conn(0x7f8c18088a70 0x7f8c18082020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-05T19:27:58.831 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-05T19:27:58.815+0000 7f8c1ded2700 1 --1- >> v1:172.21.15.118:6789/0 conn(0x7f8c18087f90 0x7f8c18088360 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.118:6789/0 says I am v1:172.21.15.118:52086/0 (socket says 172.21.15.118:52086) 2023-10-05T19:27:58.831 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-05T19:27:58.815+0000 7f8c1ded2700 1 -- 172.21.15.118:0/1043746007 learned_addr learned my addr 172.21.15.118:0/1043746007 (peer_addr_for_me v1:172.21.15.118:0/0) 2023-10-05T19:27:58.832 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-05T19:27:58.815+0000 7f8c1ced0700 1 -- 172.21.15.118:0/1043746007 <== mon.0 v1:172.21.15.118:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 95451723 0 0) 0x7f8c1808abd0 con 0x7f8c18087f90 2023-10-05T19:27:58.832 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-05T19:27:58.815+0000 7f8c1d6d1700 1 -- 172.21.15.118:0/1043746007 >> v1:172.21.15.118:6789/0 conn(0x7f8c18087f90 legacy=0x7f8c18088360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-05T19:27:58.832 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-05T19:27:58.815+0000 7f8c1d6d1700 1 -- 172.21.15.118:0/1043746007 --> v2:172.21.15.118:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8c1806f4e0 con 0x7f8c18088a70 2023-10-05T19:27:58.832 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-05T19:27:58.815+0000 7f779e67a700 1 --2- 172.21.15.118:0/803032172 >> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] conn(0x7f7798095ff0 0x7f779808c590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-05T19:27:58.832 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-05T19:27:58.815+0000 7f779e67a700 1 -- 172.21.15.118:0/803032172 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f779808f570 con 0x7f7798095ff0 2023-10-05T19:27:58.832 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-05T19:27:58.819+0000 7f779de79700 1 --2- 172.21.15.118:0/803032172 >> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] conn(0x7f7798095ff0 0x7f779808c590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-05T19:27:58.833 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-05T19:27:58.819+0000 7f8c1d6d1700 1 --2- 172.21.15.118:0/1043746007 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f8c18088a70 0x7f8c18082020 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f8c14006d90 tx=0x7f8c140091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=670ae8245df33858 server_cookie=0 in_seq=0 out_seq=0 2023-10-05T19:27:58.833 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-05T19:27:58.819+0000 7fd56cff9700 1 -- 172.21.15.118:0/2560005492 <== osd.2 v2:172.21.15.167:6816/30037 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7fd57006d1d0 con 0x7fd570094e20 2023-10-05T19:27:58.833 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-05T19:27:58.819+0000 7f8c1ced0700 1 -- 172.21.15.118:0/1043746007 <== mon.0 v2:172.21.15.118:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f8c1400f750 con 0x7f8c18088a70 2023-10-05T19:27:58.833 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-05T19:27:58.819+0000 7f8c1ced0700 1 -- 172.21.15.118:0/1043746007 <== mon.0 v2:172.21.15.118:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8c1400f8b0 con 0x7f8c18088a70 2023-10-05T19:27:58.834 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-05T19:27:58.819+0000 7f8c1ced0700 1 -- 172.21.15.118:0/1043746007 <== mon.0 v2:172.21.15.118:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f8c1400fb80 con 0x7f8c18088a70 2023-10-05T19:27:58.834 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-05T19:27:58.819+0000 7f779de79700 1 --2- 172.21.15.118:0/803032172 >> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] conn(0x7f7798095ff0 0x7f779808c590 crc :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-05T19:27:58.835 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-05T19:27:58.819+0000 7fd575b76700 1 -- 172.21.15.118:0/2560005492 >> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] conn(0x7fd570094e20 msgr2=0x7fd57011fcb0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-05T19:27:58.835 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-05T19:27:58.819+0000 7fd575b76700 1 --2- 172.21.15.118:0/2560005492 >> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] conn(0x7fd570094e20 0x7fd57011fcb0 crc :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-05T19:27:58.835 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-05T19:27:58.819+0000 7fd575b76700 1 -- 172.21.15.118:0/2560005492 >> [v2:172.21.15.167:6824/29889,v1:172.21.15.167:6825/29889] conn(0x7fd5540323e0 msgr2=0x7fd554034890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-05T19:27:58.836 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-05T19:27:58.819+0000 7fd575b76700 1 --2- 172.21.15.118:0/2560005492 >> [v2:172.21.15.167:6824/29889,v1:172.21.15.167:6825/29889] conn(0x7fd5540323e0 0x7fd554034890 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7fd55c007fd0 tx=0x7fd55c007b20 comp rx=0 tx=0).stop 2023-10-05T19:27:58.836 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-05T19:27:58.819+0000 7fd575b76700 1 -- 172.21.15.118:0/2560005492 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fd57005dac0 msgr2=0x7fd57011f770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-05T19:27:58.836 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-05T19:27:58.819+0000 7fd575b76700 1 --2- 172.21.15.118:0/2560005492 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fd57005dac0 0x7fd57011f770 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7fd564000c00 tx=0x7fd564016000 comp rx=0 tx=0).stop 2023-10-05T19:27:58.836 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-05T19:27:58.819+0000 7fd575b76700 1 -- 172.21.15.118:0/2560005492 shutdown_connections 2023-10-05T19:27:58.836 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-05T19:27:58.819+0000 7fd575b76700 1 --2- 172.21.15.118:0/2560005492 >> [v2:172.21.15.167:6824/29889,v1:172.21.15.167:6825/29889] conn(0x7fd5540323e0 0x7fd554034890 unknown :-1 s=CLOSED pgs=30 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-05T19:27:58.836 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-05T19:27:58.819+0000 7fd575b76700 1 --2- 172.21.15.118:0/2560005492 >> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] conn(0x7fd570094e20 0x7fd57011fcb0 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-05T19:27:58.836 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-05T19:27:58.819+0000 7fd575b76700 1 --2- 172.21.15.118:0/2560005492 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fd57005dac0 0x7fd57011f770 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-05T19:27:58.837 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-05T19:27:58.819+0000 7fd575b76700 1 -- 172.21.15.118:0/2560005492 >> 172.21.15.118:0/2560005492 conn(0x7fd570095970 msgr2=0x7fd570126f80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-05T19:27:58.837 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-05T19:27:58.819+0000 7fd575b76700 1 -- 172.21.15.118:0/2560005492 shutdown_connections 2023-10-05T19:27:58.837 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-05T19:27:58.819+0000 7fd575b76700 1 -- 172.21.15.118:0/2560005492 wait complete. 2023-10-05T19:27:58.837 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-05T19:27:58.819+0000 7f8c20136700 1 -- 172.21.15.118:0/1043746007 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f8c18088a70 msgr2=0x7f8c18082020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-05T19:27:58.838 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-05T19:27:58.819+0000 7f8c20136700 1 --2- 172.21.15.118:0/1043746007 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f8c18088a70 0x7f8c18082020 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f8c14006d90 tx=0x7f8c140091a0 comp rx=0 tx=0).stop 2023-10-05T19:27:58.838 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-05T19:27:58.819+0000 7f8c20136700 1 -- 172.21.15.118:0/1043746007 shutdown_connections 2023-10-05T19:27:58.838 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-05T19:27:58.819+0000 7f8c20136700 1 --2- 172.21.15.118:0/1043746007 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f8c18088a70 0x7f8c18082020 unknown :-1 s=CLOSED pgs=78 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-05T19:27:58.838 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-05T19:27:58.819+0000 7f8c20136700 1 -- 172.21.15.118:0/1043746007 >> 172.21.15.118:0/1043746007 conn(0x7f8c180964a0 msgr2=0x7f8c180968a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-05T19:27:58.839 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-05T19:27:58.819+0000 7f778d7fa700 1 -- 172.21.15.118:0/803032172 <== osd.0 v2:172.21.15.118:6800/27690 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+23999 (crc 0 0 0) 0x7f7788009900 con 0x7f7798095ff0 2023-10-05T19:27:58.841 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-05T19:27:58.819+0000 7f8c20136700 1 -- 172.21.15.118:0/1043746007 shutdown_connections 2023-10-05T19:27:58.841 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-05T19:27:58.819+0000 7f8c20136700 1 -- 172.21.15.118:0/1043746007 wait complete. 2023-10-05T19:27:58.841 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-05T19:27:58.819+0000 7f8c20136700 1 Processor -- start 2023-10-05T19:27:58.841 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-05T19:27:58.819+0000 7f8c20136700 1 -- start start 2023-10-05T19:27:58.841 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-05T19:27:58.819+0000 7f8c20136700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f8c18087f90 0x7f8c18093820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-05T19:27:58.841 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-05T19:27:58.819+0000 7f8c20136700 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8c1806dce0 con 0x7f8c18087f90 2023-10-05T19:27:58.841 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-05T19:27:58.819+0000 7f8c1ded2700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f8c18087f90 0x7f8c18093820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-05T19:27:58.842 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-05T19:27:58.819+0000 7f8c1ded2700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f8c18087f90 0x7f8c18093820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.118:3300/0 says I am v2:172.21.15.118:55774/0 (socket says 172.21.15.118:55774) 2023-10-05T19:27:58.842 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-05T19:27:58.819+0000 7f8c1ded2700 1 -- 172.21.15.118:0/806127732 learned_addr learned my addr 172.21.15.118:0/806127732 (peer_addr_for_me v2:172.21.15.118:0/0) 2023-10-05T19:27:58.842 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-05T19:27:58.819+0000 7f8c1ded2700 1 -- 172.21.15.118:0/806127732 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8c14009420 con 0x7f8c18087f90 2023-10-05T19:27:58.842 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-05T19:27:58.819+0000 7f8c1ded2700 1 --2- 172.21.15.118:0/806127732 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f8c18087f90 0x7f8c18093820 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f8c0800c710 tx=0x7f8c08007d90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-05T19:27:58.842 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-05T19:27:58.819+0000 7f8c0dffb700 1 -- 172.21.15.118:0/806127732 <== mon.0 v2:172.21.15.118:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f8c08005c50 con 0x7f8c18087f90 2023-10-05T19:27:58.842 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-05T19:27:58.819+0000 7f8c0dffb700 1 -- 172.21.15.118:0/806127732 <== mon.0 v2:172.21.15.118:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8c08005db0 con 0x7f8c18087f90 2023-10-05T19:27:58.842 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-05T19:27:58.819+0000 7f8c20136700 1 -- 172.21.15.118:0/806127732 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8c18079e70 con 0x7f8c18087f90 2023-10-05T19:27:58.842 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-05T19:27:58.819+0000 7f8c0dffb700 1 -- 172.21.15.118:0/806127732 <== mon.0 v2:172.21.15.118:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f8c08007250 con 0x7f8c18087f90 2023-10-05T19:27:58.842 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-05T19:27:58.819+0000 7f8c20136700 1 -- 172.21.15.118:0/806127732 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8c18095070 con 0x7f8c18087f90 2023-10-05T19:27:58.842 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-05T19:27:58.823+0000 7f8c20136700 1 -- 172.21.15.118:0/806127732 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f8c1807a320 con 0x7f8c18087f90 2023-10-05T19:27:58.842 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-05T19:27:58.823+0000 7f8c0dffb700 1 -- 172.21.15.118:0/806127732 <== mon.0 v2:172.21.15.118:3300/0 4 ==== mgrmap(e 5) v1 ==== 40329+0+0 (secure 0 0 0) 0x7f8c080073b0 con 0x7f8c18087f90 2023-10-05T19:27:58.843 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-05T19:27:58.823+0000 7f8c0dffb700 1 --2- 172.21.15.118:0/806127732 >> [v2:172.21.15.167:6824/29889,v1:172.21.15.167:6825/29889] conn(0x7f8bfc0323e0 0x7f8bfc034890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-05T19:27:58.843 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-05T19:27:58.823+0000 7f8c0dffb700 1 -- 172.21.15.118:0/806127732 <== mon.0 v2:172.21.15.118:3300/0 5 ==== osd_map(15..15 src has 1..15) v4 ==== 3478+0+0 (secure 0 0 0) 0x7f8c08042ec0 con 0x7f8c18087f90 2023-10-05T19:27:58.843 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-05T19:27:58.823+0000 7f8c0dffb700 1 --2- 172.21.15.118:0/806127732 >> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] conn(0x7f8bfc037af0 0x7f8bfc039f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-05T19:27:58.843 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-05T19:27:58.823+0000 7f8c0dffb700 1 -- 172.21.15.118:0/806127732 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f8bfc03a5f0 con 0x7f8bfc037af0 2023-10-05T19:27:58.843 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-05T19:27:58.823+0000 7f8c0dffb700 1 -- 172.21.15.118:0/806127732 <== mon.0 v2:172.21.15.118:3300/0 6 ==== mon_get_version_reply(handle=1 version=15) v2 ==== 24+0+0 (secure 0 0 0) 0x7f8c08045530 con 0x7f8c18087f90 2023-10-05T19:27:58.843 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-05T19:27:58.823+0000 7f8c1d6d1700 1 --2- 172.21.15.118:0/806127732 >> [v2:172.21.15.167:6824/29889,v1:172.21.15.167:6825/29889] conn(0x7f8bfc0323e0 0x7f8bfc034890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-05T19:27:58.843 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-05T19:27:58.823+0000 7f8c1e6d3700 1 --2- 172.21.15.118:0/806127732 >> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] conn(0x7f8bfc037af0 0x7f8bfc039f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-05T19:27:58.843 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-05T19:27:58.823+0000 7f8c1d6d1700 1 --2- 172.21.15.118:0/806127732 >> [v2:172.21.15.167:6824/29889,v1:172.21.15.167:6825/29889] conn(0x7f8bfc0323e0 0x7f8bfc034890 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7f8c140091e0 tx=0x7f8c14019000 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-05T19:27:58.843 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-05T19:27:58.823+0000 7f8c1e6d3700 1 --2- 172.21.15.118:0/806127732 >> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] conn(0x7f8bfc037af0 0x7f8bfc039f20 crc :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-05T19:27:58.843 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-05T19:27:58.827+0000 7f8c0dffb700 1 -- 172.21.15.118:0/806127732 <== osd.3 v2:172.21.15.167:6808/30036 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+23999 (crc 0 0 0) 0x7f8bfc03a5f0 con 0x7f8bfc037af0 2023-10-05T19:27:58.844 INFO:teuthology.orchestra.run.smithi118.stdout:38654705670 2023-10-05T19:27:58.844 DEBUG:teuthology.orchestra.run.smithi118:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2023-10-05T19:27:58.847 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-05T19:27:58.839+0000 7f8c20136700 1 -- 172.21.15.118:0/806127732 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f8c1805ecb0 con 0x7f8bfc037af0 2023-10-05T19:27:58.848 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-05T19:27:58.843+0000 7f8c0dffb700 1 -- 172.21.15.118:0/806127732 <== osd.3 v2:172.21.15.167:6808/30036 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7f8c1805ecb0 con 0x7f8bfc037af0 2023-10-05T19:27:58.849 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-05T19:27:58.843+0000 7f8c20136700 1 -- 172.21.15.118:0/806127732 >> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] conn(0x7f8bfc037af0 msgr2=0x7f8bfc039f20 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-05T19:27:58.849 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-05T19:27:58.843+0000 7f8c20136700 1 --2- 172.21.15.118:0/806127732 >> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] conn(0x7f8bfc037af0 0x7f8bfc039f20 crc :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-05T19:27:58.849 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-05T19:27:58.843+0000 7f8c20136700 1 -- 172.21.15.118:0/806127732 >> [v2:172.21.15.167:6824/29889,v1:172.21.15.167:6825/29889] conn(0x7f8bfc0323e0 msgr2=0x7f8bfc034890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-05T19:27:58.849 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-05T19:27:58.843+0000 7f8c20136700 1 --2- 172.21.15.118:0/806127732 >> [v2:172.21.15.167:6824/29889,v1:172.21.15.167:6825/29889] conn(0x7f8bfc0323e0 0x7f8bfc034890 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7f8c140091e0 tx=0x7f8c14019000 comp rx=0 tx=0).stop 2023-10-05T19:27:58.849 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-05T19:27:58.843+0000 7f8c20136700 1 -- 172.21.15.118:0/806127732 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f8c18087f90 msgr2=0x7f8c18093820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-05T19:27:58.850 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-05T19:27:58.843+0000 7f8c20136700 1 --2- 172.21.15.118:0/806127732 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f8c18087f90 0x7f8c18093820 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f8c0800c710 tx=0x7f8c08007d90 comp rx=0 tx=0).stop 2023-10-05T19:27:58.850 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-05T19:27:58.843+0000 7f8c20136700 1 -- 172.21.15.118:0/806127732 shutdown_connections 2023-10-05T19:27:58.850 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-05T19:27:58.843+0000 7f8c20136700 1 --2- 172.21.15.118:0/806127732 >> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] conn(0x7f8bfc037af0 0x7f8bfc039f20 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-05T19:27:58.850 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-05T19:27:58.843+0000 7f8c20136700 1 --2- 172.21.15.118:0/806127732 >> [v2:172.21.15.167:6824/29889,v1:172.21.15.167:6825/29889] conn(0x7f8bfc0323e0 0x7f8bfc034890 unknown :-1 s=CLOSED pgs=33 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-05T19:27:58.850 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-05T19:27:58.843+0000 7f8c20136700 1 --2- 172.21.15.118:0/806127732 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f8c18087f90 0x7f8c18093820 unknown :-1 s=CLOSED pgs=79 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-05T19:27:58.850 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-05T19:27:58.843+0000 7f8c20136700 1 -- 172.21.15.118:0/806127732 >> 172.21.15.118:0/806127732 conn(0x7f8c180964a0 msgr2=0x7f8c1806af90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-05T19:27:58.850 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-05T19:27:58.843+0000 7f8c20136700 1 -- 172.21.15.118:0/806127732 shutdown_connections 2023-10-05T19:27:58.850 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-05T19:27:58.843+0000 7f8c20136700 1 -- 172.21.15.118:0/806127732 wait complete. 2023-10-05T19:27:58.852 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-05T19:27:58.847+0000 7f980e668700 1 -- 172.21.15.118:0/2418650951 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f980804b5c0 con 0x7f980811bc70 2023-10-05T19:27:58.853 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-05T19:27:58.847+0000 7f97effff700 1 -- 172.21.15.118:0/2418650951 <== osd.1 v2:172.21.15.167:6800/30035 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7f980804b5c0 con 0x7f980811bc70 2023-10-05T19:27:58.853 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-05T19:27:58.847+0000 7f980e668700 1 -- 172.21.15.118:0/2418650951 >> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] conn(0x7f980811bc70 msgr2=0x7f980811a4b0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-05T19:27:58.853 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-05T19:27:58.847+0000 7f980e668700 1 --2- 172.21.15.118:0/2418650951 >> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] conn(0x7f980811bc70 0x7f980811a4b0 crc :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-05T19:27:58.853 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-05T19:27:58.847+0000 7f779e67a700 1 -- 172.21.15.118:0/803032172 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f7798091b10 con 0x7f7798095ff0 2023-10-05T19:27:58.854 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-05T19:27:58.847+0000 7f980e668700 1 -- 172.21.15.118:0/2418650951 >> [v2:172.21.15.167:6824/29889,v1:172.21.15.167:6825/29889] conn(0x7f97f40323e0 msgr2=0x7f97f4034890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-05T19:27:58.854 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-05T19:27:58.847+0000 7f980e668700 1 --2- 172.21.15.118:0/2418650951 >> [v2:172.21.15.167:6824/29889,v1:172.21.15.167:6825/29889] conn(0x7f97f40323e0 0x7f97f4034890 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7f97fc0091e0 tx=0x7f97fc016000 comp rx=0 tx=0).stop 2023-10-05T19:27:58.854 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-05T19:27:58.847+0000 7f980e668700 1 -- 172.21.15.118:0/2418650951 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f980806b130 msgr2=0x7f9808119a60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-05T19:27:58.854 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-05T19:27:58.847+0000 7f980e668700 1 --2- 172.21.15.118:0/2418650951 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f980806b130 0x7f9808119a60 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f980000cc00 tx=0x7f9800001b00 comp rx=0 tx=0).stop 2023-10-05T19:27:58.855 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-05T19:27:58.847+0000 7f980e668700 1 -- 172.21.15.118:0/2418650951 shutdown_connections 2023-10-05T19:27:58.855 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-05T19:27:58.847+0000 7f980e668700 1 --2- 172.21.15.118:0/2418650951 >> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] conn(0x7f980811bc70 0x7f980811a4b0 unknown :-1 s=CLOSED pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-05T19:27:58.855 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-05T19:27:58.847+0000 7f980e668700 1 --2- 172.21.15.118:0/2418650951 >> [v2:172.21.15.167:6824/29889,v1:172.21.15.167:6825/29889] conn(0x7f97f40323e0 0x7f97f4034890 unknown :-1 s=CLOSED pgs=31 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-05T19:27:58.855 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-05T19:27:58.847+0000 7f980e668700 1 --2- 172.21.15.118:0/2418650951 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f980806b130 0x7f9808119a60 unknown :-1 s=CLOSED pgs=74 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-05T19:27:58.855 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-05T19:27:58.847+0000 7f980e668700 1 -- 172.21.15.118:0/2418650951 >> 172.21.15.118:0/2418650951 conn(0x7f9808098f60 msgr2=0x7f9808112f00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-05T19:27:58.855 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-05T19:27:58.847+0000 7f778d7fa700 1 -- 172.21.15.118:0/803032172 <== osd.0 v2:172.21.15.118:6800/27690 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7f7798091b10 con 0x7f7798095ff0 2023-10-05T19:27:58.856 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-05T19:27:58.847+0000 7f980e668700 1 -- 172.21.15.118:0/2418650951 shutdown_connections 2023-10-05T19:27:58.856 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-05T19:27:58.847+0000 7f980e668700 1 -- 172.21.15.118:0/2418650951 wait complete. 2023-10-05T19:27:58.856 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-05T19:27:58.847+0000 7f779e67a700 1 -- 172.21.15.118:0/803032172 >> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] conn(0x7f7798095ff0 msgr2=0x7f779808c590 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-05T19:27:58.857 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-05T19:27:58.847+0000 7f779e67a700 1 --2- 172.21.15.118:0/803032172 >> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] conn(0x7f7798095ff0 0x7f779808c590 crc :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-05T19:27:58.857 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-05T19:27:58.847+0000 7f779e67a700 1 -- 172.21.15.118:0/803032172 >> [v2:172.21.15.167:6824/29889,v1:172.21.15.167:6825/29889] conn(0x7f777c032350 msgr2=0x7f777c034800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-05T19:27:58.857 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-05T19:27:58.847+0000 7f779e67a700 1 --2- 172.21.15.118:0/803032172 >> [v2:172.21.15.167:6824/29889,v1:172.21.15.167:6825/29889] conn(0x7f777c032350 0x7f777c034800 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7f7794001020 tx=0x7f77940079b0 comp rx=0 tx=0).stop 2023-10-05T19:27:58.857 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-05T19:27:58.847+0000 7f779e67a700 1 -- 172.21.15.118:0/803032172 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f77980905f0 msgr2=0x7f779808c050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-05T19:27:58.857 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-05T19:27:58.847+0000 7f779e67a700 1 --2- 172.21.15.118:0/803032172 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f77980905f0 0x7f779808c050 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f7790000f80 tx=0x7f7790017000 comp rx=0 tx=0).stop 2023-10-05T19:27:58.857 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-05T19:27:58.847+0000 7f779e67a700 1 -- 172.21.15.118:0/803032172 shutdown_connections 2023-10-05T19:27:58.857 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-05T19:27:58.847+0000 7f779e67a700 1 --2- 172.21.15.118:0/803032172 >> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] conn(0x7f7798095ff0 0x7f779808c590 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-05T19:27:58.857 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-05T19:27:58.847+0000 7f779e67a700 1 --2- 172.21.15.118:0/803032172 >> [v2:172.21.15.167:6824/29889,v1:172.21.15.167:6825/29889] conn(0x7f777c032350 0x7f777c034800 unknown :-1 s=CLOSED pgs=32 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-05T19:27:58.857 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-05T19:27:58.847+0000 7f779e67a700 1 --2- 172.21.15.118:0/803032172 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f77980905f0 0x7f779808c050 secure :-1 s=CLOSED pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f7790000f80 tx=0x7f7790017000 comp rx=0 tx=0).stop 2023-10-05T19:27:58.858 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-05T19:27:58.847+0000 7f779e67a700 1 -- 172.21.15.118:0/803032172 >> 172.21.15.118:0/803032172 conn(0x7f779809c5d0 msgr2=0x7f77980949e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-05T19:27:58.858 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-05T19:27:58.847+0000 7f779e67a700 1 -- 172.21.15.118:0/803032172 shutdown_connections 2023-10-05T19:27:58.858 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-05T19:27:58.851+0000 7f779e67a700 1 -- 172.21.15.118:0/803032172 wait complete. 2023-10-05T19:27:58.860 INFO:teuthology.orchestra.run.smithi118.stdout:38654705668 2023-10-05T19:27:58.860 DEBUG:teuthology.orchestra.run.smithi118:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3 2023-10-05T19:27:58.865 INFO:teuthology.orchestra.run.smithi118.stdout:38654705669 2023-10-05T19:27:58.865 DEBUG:teuthology.orchestra.run.smithi118:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2023-10-05T19:27:58.872 INFO:teuthology.orchestra.run.smithi118.stdout:38654705668 2023-10-05T19:27:58.872 DEBUG:teuthology.orchestra.run.smithi118:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2023-10-05T19:27:58.940 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-05T19:27:58.931+0000 7f16f321d700 1 Processor -- start 2023-10-05T19:27:58.940 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-05T19:27:58.931+0000 7f16f321d700 1 -- start start 2023-10-05T19:27:58.941 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-05T19:27:58.935+0000 7f16f321d700 1 --2- >> v2:172.21.15.118:3300/0 conn(0x7f16ec12c7a0 0x7f16ec12eb80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-05T19:27:58.941 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-05T19:27:58.935+0000 7f16f321d700 1 -- --> v1:172.21.15.118:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f16ec092760 con 0x7f16ec095ff0 2023-10-05T19:27:58.941 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-05T19:27:58.935+0000 7f16f321d700 1 -- --> v2:172.21.15.118:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f16ec096ad0 con 0x7f16ec12c7a0 2023-10-05T19:27:58.941 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-05T19:27:58.935+0000 7f16f1a1a700 1 --2- >> v2:172.21.15.118:3300/0 conn(0x7f16ec12c7a0 0x7f16ec12eb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-05T19:27:58.941 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-05T19:27:58.935+0000 7f16f221b700 1 --1- >> v1:172.21.15.118:6789/0 conn(0x7f16ec095ff0 0x7f16ec0963c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.118:6789/0 says I am v1:172.21.15.118:52092/0 (socket says 172.21.15.118:52092) 2023-10-05T19:27:58.941 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-05T19:27:58.935+0000 7f16f221b700 1 -- 172.21.15.118:0/3132835454 learned_addr learned my addr 172.21.15.118:0/3132835454 (peer_addr_for_me v1:172.21.15.118:0/0) 2023-10-05T19:27:58.942 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-05T19:27:58.935+0000 7f16f1219700 1 -- 172.21.15.118:0/3132835454 <== mon.0 v1:172.21.15.118:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2975897190 0 0) 0x7f16ec092760 con 0x7f16ec095ff0 2023-10-05T19:27:58.943 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-05T19:27:58.935+0000 7f16f1219700 1 -- 172.21.15.118:0/3132835454 --> v1:172.21.15.118:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f16d8003570 con 0x7f16ec095ff0 2023-10-05T19:27:58.943 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-05T19:27:58.935+0000 7f16f1a1a700 1 -- 172.21.15.118:0/3132835454 >> v1:172.21.15.118:6789/0 conn(0x7f16ec095ff0 legacy=0x7f16ec0963c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-05T19:27:58.943 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-05T19:27:58.935+0000 7f16f1a1a700 1 -- 172.21.15.118:0/3132835454 --> v2:172.21.15.118:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f16ec0918a0 con 0x7f16ec12c7a0 2023-10-05T19:27:58.943 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-05T19:27:58.935+0000 7f16f1a1a700 1 --2- 172.21.15.118:0/3132835454 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f16ec12c7a0 0x7f16ec12eb80 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f16dc009850 tx=0x7f16dc009120 comp rx=0 tx=0).ready entity=mon.0 client_cookie=465b67329c3bfadd server_cookie=0 in_seq=0 out_seq=0 2023-10-05T19:27:58.943 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-05T19:27:58.935+0000 7f16f1219700 1 -- 172.21.15.118:0/3132835454 <== mon.0 v2:172.21.15.118:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f16dc005c10 con 0x7f16ec12c7a0 2023-10-05T19:27:58.943 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-05T19:27:58.935+0000 7f16f1219700 1 -- 172.21.15.118:0/3132835454 <== mon.0 v2:172.21.15.118:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f16dc005d70 con 0x7f16ec12c7a0 2023-10-05T19:27:58.943 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-05T19:27:58.935+0000 7f16f1219700 1 -- 172.21.15.118:0/3132835454 <== mon.0 v2:172.21.15.118:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f16dc006190 con 0x7f16ec12c7a0 2023-10-05T19:27:58.943 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-05T19:27:58.935+0000 7f16f321d700 1 -- 172.21.15.118:0/3132835454 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f16ec12c7a0 msgr2=0x7f16ec12eb80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-05T19:27:58.943 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-05T19:27:58.935+0000 7f16f321d700 1 --2- 172.21.15.118:0/3132835454 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f16ec12c7a0 0x7f16ec12eb80 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f16dc009850 tx=0x7f16dc009120 comp rx=0 tx=0).stop 2023-10-05T19:27:58.943 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-05T19:27:58.935+0000 7f16f321d700 1 -- 172.21.15.118:0/3132835454 shutdown_connections 2023-10-05T19:27:58.944 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-05T19:27:58.935+0000 7f16f321d700 1 --2- 172.21.15.118:0/3132835454 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f16ec12c7a0 0x7f16ec12eb80 unknown :-1 s=CLOSED pgs=81 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-05T19:27:58.944 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-05T19:27:58.935+0000 7f16f321d700 1 -- 172.21.15.118:0/3132835454 >> 172.21.15.118:0/3132835454 conn(0x7f16ec09c5d0 msgr2=0x7f16ec09c9d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-05T19:27:58.944 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-05T19:27:58.935+0000 7f16f321d700 1 -- 172.21.15.118:0/3132835454 shutdown_connections 2023-10-05T19:27:58.944 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-05T19:27:58.935+0000 7f16f321d700 1 -- 172.21.15.118:0/3132835454 wait complete. 2023-10-05T19:27:58.945 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-05T19:27:58.935+0000 7f16f321d700 1 Processor -- start 2023-10-05T19:27:58.945 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-05T19:27:58.935+0000 7f16f321d700 1 -- start start 2023-10-05T19:27:58.945 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-05T19:27:58.935+0000 7f16f321d700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f16ec095ff0 0x7f16ec12b7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-05T19:27:58.946 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-05T19:27:58.935+0000 7f16f321d700 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f16ec08d2d0 con 0x7f16ec095ff0 2023-10-05T19:27:58.946 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-05T19:27:58.935+0000 7f16f221b700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f16ec095ff0 0x7f16ec12b7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-05T19:27:58.946 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-05T19:27:58.935+0000 7f16f221b700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f16ec095ff0 0x7f16ec12b7a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.118:3300/0 says I am v2:172.21.15.118:55784/0 (socket says 172.21.15.118:55784) 2023-10-05T19:27:58.946 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-05T19:27:58.935+0000 7f16f221b700 1 -- 172.21.15.118:0/2201640378 learned_addr learned my addr 172.21.15.118:0/2201640378 (peer_addr_for_me v2:172.21.15.118:0/0) 2023-10-05T19:27:58.946 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-05T19:27:58.939+0000 7f16f221b700 1 -- 172.21.15.118:0/2201640378 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f16dc0093c0 con 0x7f16ec095ff0 2023-10-05T19:27:58.946 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-05T19:27:58.939+0000 7f16f221b700 1 --2- 172.21.15.118:0/2201640378 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f16ec095ff0 0x7f16ec12b7a0 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f16e800c6c0 tx=0x7f16e8007e00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-05T19:27:58.946 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-05T19:27:58.939+0000 7f16e27fc700 1 -- 172.21.15.118:0/2201640378 <== mon.0 v2:172.21.15.118:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f16e8005b30 con 0x7f16ec095ff0 2023-10-05T19:27:58.946 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-05T19:27:58.939+0000 7f16e27fc700 1 -- 172.21.15.118:0/2201640378 <== mon.0 v2:172.21.15.118:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f16e8005c90 con 0x7f16ec095ff0 2023-10-05T19:27:58.946 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-05T19:27:58.939+0000 7f16f321d700 1 -- 172.21.15.118:0/2201640378 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f16ec08ec50 con 0x7f16ec095ff0 2023-10-05T19:27:58.946 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-05T19:27:58.939+0000 7f16e27fc700 1 -- 172.21.15.118:0/2201640378 <== mon.0 v2:172.21.15.118:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f16e80070d0 con 0x7f16ec095ff0 2023-10-05T19:27:58.946 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-05T19:27:58.939+0000 7f16f321d700 1 -- 172.21.15.118:0/2201640378 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f16ec08ff60 con 0x7f16ec095ff0 2023-10-05T19:27:58.947 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-05T19:27:58.939+0000 7f16e27fc700 1 -- 172.21.15.118:0/2201640378 <== mon.0 v2:172.21.15.118:3300/0 4 ==== mgrmap(e 5) v1 ==== 40329+0+0 (secure 0 0 0) 0x7f16e8007230 con 0x7f16ec095ff0 2023-10-05T19:27:58.947 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-05T19:27:58.939+0000 7f16f321d700 1 -- 172.21.15.118:0/2201640378 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f16ec05ecb0 con 0x7f16ec095ff0 2023-10-05T19:27:58.947 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-05T19:27:58.939+0000 7f16e27fc700 1 --2- 172.21.15.118:0/2201640378 >> [v2:172.21.15.167:6824/29889,v1:172.21.15.167:6825/29889] conn(0x7f16d00323d0 0x7f16d0034880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-05T19:27:58.948 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-05T19:27:58.939+0000 7f16e27fc700 1 -- 172.21.15.118:0/2201640378 <== mon.0 v2:172.21.15.118:3300/0 5 ==== osd_map(15..15 src has 1..15) v4 ==== 3478+0+0 (secure 0 0 0) 0x7f16e8042f30 con 0x7f16ec095ff0 2023-10-05T19:27:58.948 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-05T19:27:58.939+0000 7f16f1a1a700 1 --2- 172.21.15.118:0/2201640378 >> [v2:172.21.15.167:6824/29889,v1:172.21.15.167:6825/29889] conn(0x7f16d00323d0 0x7f16d0034880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-05T19:27:58.948 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-05T19:27:58.939+0000 7f16f1a1a700 1 --2- 172.21.15.118:0/2201640378 >> [v2:172.21.15.167:6824/29889,v1:172.21.15.167:6825/29889] conn(0x7f16d00323d0 0x7f16d0034880 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7f16dc003080 tx=0x7f16dc015000 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-05T19:27:58.952 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-05T19:27:58.943+0000 7fbb3fff6700 1 Processor -- start 2023-10-05T19:27:58.952 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-05T19:27:58.943+0000 7fbb3fff6700 1 -- start start 2023-10-05T19:27:58.952 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-05T19:27:58.943+0000 7f16e27fc700 1 -- 172.21.15.118:0/2201640378 <== mon.0 v2:172.21.15.118:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166807 (secure 0 0 0) 0x7f16e8072bb0 con 0x7f16ec095ff0 2023-10-05T19:27:58.953 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-05T19:27:58.947+0000 7fbb3fff6700 1 --2- >> v2:172.21.15.118:3300/0 conn(0x7fbb381260f0 0x7fbb381264c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-05T19:27:58.953 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-05T19:27:58.947+0000 7fbb3fff6700 1 -- --> v1:172.21.15.118:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbb3806e5e0 con 0x7fbb38126a00 2023-10-05T19:27:58.953 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-05T19:27:58.947+0000 7fbb3fff6700 1 -- --> v2:172.21.15.118:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbb3806e810 con 0x7fbb381260f0 2023-10-05T19:27:58.953 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-05T19:27:58.947+0000 7fbb3dd92700 1 --2- >> v2:172.21.15.118:3300/0 conn(0x7fbb381260f0 0x7fbb381264c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-05T19:27:58.954 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-05T19:27:58.947+0000 7fbb3dd92700 1 --2- >> v2:172.21.15.118:3300/0 conn(0x7fbb381260f0 0x7fbb381264c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.118:3300/0 says I am v2:172.21.15.118:55798/0 (socket says 172.21.15.118:55798) 2023-10-05T19:27:58.954 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-05T19:27:58.947+0000 7fbb3dd92700 1 -- 172.21.15.118:0/880303701 learned_addr learned my addr 172.21.15.118:0/880303701 (peer_addr_for_me v2:172.21.15.118:0/0) 2023-10-05T19:27:58.954 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-05T19:27:58.947+0000 7fbb3d591700 1 --1- 172.21.15.118:0/880303701 >> v1:172.21.15.118:6789/0 conn(0x7fbb38126a00 0x7fbb3811fc00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.118:6789/0 says I am v1:172.21.15.118:52108/0 (socket says 172.21.15.118:52108) 2023-10-05T19:27:58.954 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-05T19:27:58.947+0000 7fbb3dd92700 1 -- 172.21.15.118:0/880303701 >> v1:172.21.15.118:6789/0 conn(0x7fbb38126a00 legacy=0x7fbb3811fc00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-05T19:27:58.954 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-05T19:27:58.947+0000 7fbb3dd92700 1 -- 172.21.15.118:0/880303701 --> v2:172.21.15.118:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbb3806d420 con 0x7fbb381260f0 2023-10-05T19:27:58.955 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-05T19:27:58.947+0000 7fbb3dd92700 1 --2- 172.21.15.118:0/880303701 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fbb381260f0 0x7fbb381264c0 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7fbb34009930 tx=0x7fbb34009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2dd9e241bfc39dc3 server_cookie=0 in_seq=0 out_seq=0 2023-10-05T19:27:58.955 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-05T19:27:58.947+0000 7fbb3cd90700 1 -- 172.21.15.118:0/880303701 <== mon.0 v2:172.21.15.118:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fbb3400b860 con 0x7fbb381260f0 2023-10-05T19:27:58.955 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-05T19:27:58.947+0000 7fbb3cd90700 1 -- 172.21.15.118:0/880303701 <== mon.0 v2:172.21.15.118:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbb3400b9c0 con 0x7fbb381260f0 2023-10-05T19:27:58.955 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-05T19:27:58.947+0000 7fbb3cd90700 1 -- 172.21.15.118:0/880303701 <== mon.0 v2:172.21.15.118:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fbb3400bb50 con 0x7fbb381260f0 2023-10-05T19:27:58.955 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-05T19:27:58.947+0000 7fbb3fff6700 1 -- 172.21.15.118:0/880303701 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fbb381260f0 msgr2=0x7fbb381264c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-05T19:27:58.955 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-05T19:27:58.947+0000 7fbb3fff6700 1 --2- 172.21.15.118:0/880303701 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fbb381260f0 0x7fbb381264c0 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7fbb34009930 tx=0x7fbb34009250 comp rx=0 tx=0).stop 2023-10-05T19:27:58.956 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-05T19:27:58.947+0000 7fbb3fff6700 1 -- 172.21.15.118:0/880303701 shutdown_connections 2023-10-05T19:27:58.956 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-05T19:27:58.947+0000 7fbb3fff6700 1 --2- 172.21.15.118:0/880303701 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fbb381260f0 0x7fbb381264c0 unknown :-1 s=CLOSED pgs=84 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-05T19:27:58.956 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-05T19:27:58.947+0000 7fbb3fff6700 1 -- 172.21.15.118:0/880303701 >> 172.21.15.118:0/880303701 conn(0x7fbb38098f60 msgr2=0x7fbb380949f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-05T19:27:58.956 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-05T19:27:58.947+0000 7fbb3fff6700 1 -- 172.21.15.118:0/880303701 shutdown_connections 2023-10-05T19:27:58.956 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-05T19:27:58.947+0000 7fbb3fff6700 1 -- 172.21.15.118:0/880303701 wait complete. 2023-10-05T19:27:58.957 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-05T19:27:58.947+0000 7fbb3fff6700 1 Processor -- start 2023-10-05T19:27:58.958 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-05T19:27:58.947+0000 7fbb3fff6700 1 -- start start 2023-10-05T19:27:58.958 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-05T19:27:58.951+0000 7fbb3fff6700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fbb381260f0 0x7fbb38116790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-05T19:27:58.958 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-05T19:27:58.951+0000 7fbb3fff6700 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbb38118470 con 0x7fbb381260f0 2023-10-05T19:27:58.958 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-05T19:27:58.951+0000 7fbb3dd92700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fbb381260f0 0x7fbb38116790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-05T19:27:58.958 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-05T19:27:58.951+0000 7fbb3dd92700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fbb381260f0 0x7fbb38116790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.118:3300/0 says I am v2:172.21.15.118:55812/0 (socket says 172.21.15.118:55812) 2023-10-05T19:27:58.958 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-05T19:27:58.951+0000 7fbb3dd92700 1 -- 172.21.15.118:0/1876835572 learned_addr learned my addr 172.21.15.118:0/1876835572 (peer_addr_for_me v2:172.21.15.118:0/0) 2023-10-05T19:27:58.958 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-05T19:27:58.951+0000 7fbb3dd92700 1 -- 172.21.15.118:0/1876835572 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbb340094f0 con 0x7fbb381260f0 2023-10-05T19:27:58.958 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-05T19:27:58.951+0000 7fbb3dd92700 1 --2- 172.21.15.118:0/1876835572 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fbb381260f0 0x7fbb38116790 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7fbb3806e950 tx=0x7fbb34016000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-05T19:27:58.958 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-05T19:27:58.951+0000 7fbb2dffb700 1 -- 172.21.15.118:0/1876835572 <== mon.0 v2:172.21.15.118:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fbb3400bd50 con 0x7fbb381260f0 2023-10-05T19:27:58.959 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-05T19:27:58.951+0000 7fbb2dffb700 1 -- 172.21.15.118:0/1876835572 <== mon.0 v2:172.21.15.118:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbb3401f070 con 0x7fbb381260f0 2023-10-05T19:27:58.959 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-05T19:27:58.951+0000 7fbb3fff6700 1 -- 172.21.15.118:0/1876835572 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbb38123810 con 0x7fbb381260f0 2023-10-05T19:27:58.959 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-05T19:27:58.951+0000 7fbb2dffb700 1 -- 172.21.15.118:0/1876835572 <== mon.0 v2:172.21.15.118:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fbb3401a440 con 0x7fbb381260f0 2023-10-05T19:27:58.959 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-05T19:27:58.951+0000 7fbb3fff6700 1 -- 172.21.15.118:0/1876835572 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbb38117cd0 con 0x7fbb381260f0 2023-10-05T19:27:58.960 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-05T19:27:58.951+0000 7fbb3fff6700 1 -- 172.21.15.118:0/1876835572 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbb381185c0 con 0x7fbb381260f0 2023-10-05T19:27:58.960 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-05T19:27:58.951+0000 7fbb2dffb700 1 -- 172.21.15.118:0/1876835572 <== mon.0 v2:172.21.15.118:3300/0 4 ==== mgrmap(e 5) v1 ==== 40329+0+0 (secure 0 0 0) 0x7fbb34021030 con 0x7fbb381260f0 2023-10-05T19:27:58.960 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-05T19:27:58.951+0000 7fbb2dffb700 1 --2- 172.21.15.118:0/1876835572 >> [v2:172.21.15.167:6824/29889,v1:172.21.15.167:6825/29889] conn(0x7fbb1c0323e0 0x7fbb1c034890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-05T19:27:58.960 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-05T19:27:58.951+0000 7fbb2dffb700 1 -- 172.21.15.118:0/1876835572 <== mon.0 v2:172.21.15.118:3300/0 5 ==== osd_map(15..15 src has 1..15) v4 ==== 3478+0+0 (secure 0 0 0) 0x7fbb34045580 con 0x7fbb381260f0 2023-10-05T19:27:58.960 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-05T19:27:58.951+0000 7fbb3d591700 1 --2- 172.21.15.118:0/1876835572 >> [v2:172.21.15.167:6824/29889,v1:172.21.15.167:6825/29889] conn(0x7fbb1c0323e0 0x7fbb1c034890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-05T19:27:58.960 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-05T19:27:58.955+0000 7fbb3d591700 1 --2- 172.21.15.118:0/1876835572 >> [v2:172.21.15.167:6824/29889,v1:172.21.15.167:6825/29889] conn(0x7fbb1c0323e0 0x7fbb1c034890 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7fbb280016c0 tx=0x7fbb28007af0 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-05T19:27:58.965 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-05T19:27:58.959+0000 7f71ed42c700 1 Processor -- start 2023-10-05T19:27:58.965 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-05T19:27:58.959+0000 7f71ed42c700 1 -- start start 2023-10-05T19:27:58.966 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-05T19:27:58.959+0000 7f71ed42c700 1 --2- >> v2:172.21.15.118:3300/0 conn(0x7f71e80959a0 0x7f71e8127bc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-05T19:27:58.966 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-05T19:27:58.959+0000 7f71ed42c700 1 -- --> v1:172.21.15.118:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f71e806e030 con 0x7f71e81270e0 2023-10-05T19:27:58.966 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-05T19:27:58.959+0000 7f71ed42c700 1 -- --> v2:172.21.15.118:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f71e8095470 con 0x7f71e80959a0 2023-10-05T19:27:58.966 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-05T19:27:58.959+0000 7f71e77fe700 1 --2- >> v2:172.21.15.118:3300/0 conn(0x7f71e80959a0 0x7f71e8127bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-05T19:27:58.966 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-05T19:27:58.959+0000 7f71e7fff700 1 --1- >> v1:172.21.15.118:6789/0 conn(0x7f71e81270e0 0x7f71e81274b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.118:6789/0 says I am v1:172.21.15.118:52116/0 (socket says 172.21.15.118:52116) 2023-10-05T19:27:58.966 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-05T19:27:58.959+0000 7f71e7fff700 1 -- 172.21.15.118:0/2654214804 learned_addr learned my addr 172.21.15.118:0/2654214804 (peer_addr_for_me v1:172.21.15.118:0/0) 2023-10-05T19:27:58.967 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-05T19:27:58.959+0000 7f71e77fe700 1 -- 172.21.15.118:0/2654214804 >> v1:172.21.15.118:6789/0 conn(0x7f71e81270e0 legacy=0x7f71e81274b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-05T19:27:58.967 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-05T19:27:58.959+0000 7f71e77fe700 1 -- 172.21.15.118:0/2654214804 --> v2:172.21.15.118:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f71e8096110 con 0x7f71e80959a0 2023-10-05T19:27:58.967 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-05T19:27:58.959+0000 7f71e77fe700 1 --2- 172.21.15.118:0/2654214804 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f71e80959a0 0x7f71e8127bc0 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f71dc009860 tx=0x7f71dc0091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=203218f2f14240d server_cookie=0 in_seq=0 out_seq=0 2023-10-05T19:27:58.967 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-05T19:27:58.959+0000 7f71e6ffd700 1 -- 172.21.15.118:0/2654214804 <== mon.0 v2:172.21.15.118:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f71dc00b680 con 0x7f71e80959a0 2023-10-05T19:27:58.968 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-05T19:27:58.959+0000 7f71e6ffd700 1 -- 172.21.15.118:0/2654214804 <== mon.0 v2:172.21.15.118:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f71dc00b7e0 con 0x7f71e80959a0 2023-10-05T19:27:58.968 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-05T19:27:58.959+0000 7f71e6ffd700 1 -- 172.21.15.118:0/2654214804 <== mon.0 v2:172.21.15.118:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f71dc00b990 con 0x7f71e80959a0 2023-10-05T19:27:58.968 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-05T19:27:58.959+0000 7fbb2dffb700 1 -- 172.21.15.118:0/1876835572 <== mon.0 v2:172.21.15.118:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166807 (secure 0 0 0) 0x7fbb34010740 con 0x7fbb381260f0 2023-10-05T19:27:58.968 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-05T19:27:58.963+0000 7f71ed42c700 1 -- 172.21.15.118:0/2654214804 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f71e80959a0 msgr2=0x7f71e8127bc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-05T19:27:58.969 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-05T19:27:58.963+0000 7f71ed42c700 1 --2- 172.21.15.118:0/2654214804 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f71e80959a0 0x7f71e8127bc0 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f71dc009860 tx=0x7f71dc0091a0 comp rx=0 tx=0).stop 2023-10-05T19:27:58.969 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-05T19:27:58.963+0000 7f71ed42c700 1 -- 172.21.15.118:0/2654214804 shutdown_connections 2023-10-05T19:27:58.969 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-05T19:27:58.963+0000 7f71ed42c700 1 --2- 172.21.15.118:0/2654214804 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f71e80959a0 0x7f71e8127bc0 unknown :-1 s=CLOSED pgs=87 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-05T19:27:58.969 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-05T19:27:58.963+0000 7f71ed42c700 1 -- 172.21.15.118:0/2654214804 >> 172.21.15.118:0/2654214804 conn(0x7f71e809a530 msgr2=0x7f71e809a930 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-05T19:27:58.972 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-05T19:27:58.967+0000 7f71ed42c700 1 -- 172.21.15.118:0/2654214804 shutdown_connections 2023-10-05T19:27:58.972 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-05T19:27:58.967+0000 7f71ed42c700 1 -- 172.21.15.118:0/2654214804 wait complete. 2023-10-05T19:27:58.973 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-05T19:27:58.967+0000 7f71ed42c700 1 Processor -- start 2023-10-05T19:27:58.973 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-05T19:27:58.967+0000 7f71ed42c700 1 -- start start 2023-10-05T19:27:58.974 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-05T19:27:58.967+0000 7f71ed42c700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f71e80959a0 0x7f71e8113130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-05T19:27:58.974 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-05T19:27:58.967+0000 7f71ed42c700 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f71e81156b0 con 0x7f71e80959a0 2023-10-05T19:27:58.974 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-05T19:27:58.967+0000 7f71e7fff700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f71e80959a0 0x7f71e8113130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-05T19:27:58.974 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-05T19:27:58.967+0000 7f71e7fff700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f71e80959a0 0x7f71e8113130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.118:3300/0 says I am v2:172.21.15.118:55832/0 (socket says 172.21.15.118:55832) 2023-10-05T19:27:58.974 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-05T19:27:58.967+0000 7f71e7fff700 1 -- 172.21.15.118:0/1230019494 learned_addr learned my addr 172.21.15.118:0/1230019494 (peer_addr_for_me v2:172.21.15.118:0/0) 2023-10-05T19:27:58.975 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-05T19:27:58.967+0000 7f71e7fff700 1 -- 172.21.15.118:0/1230019494 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f71dc009420 con 0x7f71e80959a0 2023-10-05T19:27:58.975 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-05T19:27:58.967+0000 7f71e7fff700 1 --2- 172.21.15.118:0/1230019494 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f71e80959a0 0x7f71e8113130 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f71e000cbb0 tx=0x7f71e0001ac0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-05T19:27:58.975 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-05T19:27:58.967+0000 7f71cffff700 1 -- 172.21.15.118:0/1230019494 <== mon.0 v2:172.21.15.118:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f71e000a9c0 con 0x7f71e80959a0 2023-10-05T19:27:58.975 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-05T19:27:58.967+0000 7f71cffff700 1 -- 172.21.15.118:0/1230019494 <== mon.0 v2:172.21.15.118:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f71e000ab20 con 0x7f71e80959a0 2023-10-05T19:27:58.975 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-05T19:27:58.967+0000 7f71ed42c700 1 -- 172.21.15.118:0/1230019494 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f71e81166f0 con 0x7f71e80959a0 2023-10-05T19:27:58.976 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-05T19:27:58.971+0000 7f71cffff700 1 -- 172.21.15.118:0/1230019494 <== mon.0 v2:172.21.15.118:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f71e000add0 con 0x7f71e80959a0 2023-10-05T19:27:58.976 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-05T19:27:58.971+0000 7f71ed42c700 1 -- 172.21.15.118:0/1230019494 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f71e8115e50 con 0x7f71e80959a0 2023-10-05T19:27:58.977 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-05T19:27:58.971+0000 7f71cffff700 1 -- 172.21.15.118:0/1230019494 <== mon.0 v2:172.21.15.118:3300/0 4 ==== mgrmap(e 5) v1 ==== 40329+0+0 (secure 0 0 0) 0x7f71e00070f0 con 0x7f71e80959a0 2023-10-05T19:27:58.977 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-05T19:27:58.971+0000 7f71cffff700 1 --2- 172.21.15.118:0/1230019494 >> [v2:172.21.15.167:6824/29889,v1:172.21.15.167:6825/29889] conn(0x7f71d4032430 0x7f71d40348e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-05T19:27:58.977 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-05T19:27:58.971+0000 7f71cffff700 1 -- 172.21.15.118:0/1230019494 <== mon.0 v2:172.21.15.118:3300/0 5 ==== osd_map(15..15 src has 1..15) v4 ==== 3478+0+0 (secure 0 0 0) 0x7f71e0047120 con 0x7f71e80959a0 2023-10-05T19:27:58.977 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-05T19:27:58.971+0000 7f71e77fe700 1 --2- 172.21.15.118:0/1230019494 >> [v2:172.21.15.167:6824/29889,v1:172.21.15.167:6825/29889] conn(0x7f71d4032430 0x7f71d40348e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-05T19:27:58.978 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-05T19:27:58.971+0000 7f71e77fe700 1 --2- 172.21.15.118:0/1230019494 >> [v2:172.21.15.167:6824/29889,v1:172.21.15.167:6825/29889] conn(0x7f71d4032430 0x7f71d40348e0 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f71dc0030c0 tx=0x7f71dc016000 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-05T19:27:58.979 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-05T19:27:58.971+0000 7f71ed42c700 1 -- 172.21.15.118:0/1230019494 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f71e8125950 con 0x7f71e80959a0 2023-10-05T19:27:58.979 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-05T19:27:58.971+0000 7efd37a77700 1 Processor -- start 2023-10-05T19:27:58.980 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-05T19:27:58.971+0000 7efd37a77700 1 -- start start 2023-10-05T19:27:58.981 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-05T19:27:58.975+0000 7efd37a77700 1 --2- >> v2:172.21.15.118:3300/0 conn(0x7efd30095650 0x7efd30125c60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-05T19:27:58.982 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-05T19:27:58.975+0000 7efd37a77700 1 -- --> v1:172.21.15.118:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efd3006de20 con 0x7efd301261a0 2023-10-05T19:27:58.982 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-05T19:27:58.975+0000 7efd37a77700 1 -- --> v2:172.21.15.118:3300/0 -- mon_getmap magic: 0 v1 -- 0x7efd3006cef0 con 0x7efd30095650 2023-10-05T19:27:58.982 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-05T19:27:58.975+0000 7efd35813700 1 --2- >> v2:172.21.15.118:3300/0 conn(0x7efd30095650 0x7efd30125c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-05T19:27:58.982 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-05T19:27:58.975+0000 7efd35012700 1 --1- >> v1:172.21.15.118:6789/0 conn(0x7efd301261a0 0x7efd3006aef0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.118:6789/0 says I am v1:172.21.15.118:52118/0 (socket says 172.21.15.118:52118) 2023-10-05T19:27:58.982 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-05T19:27:58.975+0000 7efd35012700 1 -- 172.21.15.118:0/2281221655 learned_addr learned my addr 172.21.15.118:0/2281221655 (peer_addr_for_me v1:172.21.15.118:0/0) 2023-10-05T19:27:58.982 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-05T19:27:58.975+0000 7efd35813700 1 -- 172.21.15.118:0/2281221655 >> v1:172.21.15.118:6789/0 conn(0x7efd301261a0 legacy=0x7efd3006aef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-05T19:27:58.982 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-05T19:27:58.975+0000 7efd35813700 1 -- 172.21.15.118:0/2281221655 --> v2:172.21.15.118:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efd3006db10 con 0x7efd30095650 2023-10-05T19:27:58.982 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-05T19:27:58.975+0000 7efd35813700 1 --2- 172.21.15.118:0/2281221655 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7efd30095650 0x7efd30125c60 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7efd2000bb10 tx=0x7efd2000b430 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2027fabd40ddd5ea server_cookie=0 in_seq=0 out_seq=0 2023-10-05T19:27:58.987 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-05T19:27:58.979+0000 7efd34811700 1 -- 172.21.15.118:0/2281221655 <== mon.0 v2:172.21.15.118:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7efd2000db50 con 0x7efd30095650 2023-10-05T19:27:58.988 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-05T19:27:58.979+0000 7efd34811700 1 -- 172.21.15.118:0/2281221655 <== mon.0 v2:172.21.15.118:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efd2000dcb0 con 0x7efd30095650 2023-10-05T19:27:58.988 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-05T19:27:58.979+0000 7efd34811700 1 -- 172.21.15.118:0/2281221655 <== mon.0 v2:172.21.15.118:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7efd20017410 con 0x7efd30095650 2023-10-05T19:27:58.988 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-05T19:27:58.979+0000 7f71cffff700 1 -- 172.21.15.118:0/1230019494 <== mon.0 v2:172.21.15.118:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166807 (secure 0 0 0) 0x7f71e8125950 con 0x7f71e80959a0 2023-10-05T19:27:58.989 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-05T19:27:58.983+0000 7efd37a77700 1 -- 172.21.15.118:0/2281221655 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7efd30095650 msgr2=0x7efd30125c60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-05T19:27:58.989 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-05T19:27:58.983+0000 7efd37a77700 1 --2- 172.21.15.118:0/2281221655 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7efd30095650 0x7efd30125c60 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7efd2000bb10 tx=0x7efd2000b430 comp rx=0 tx=0).stop 2023-10-05T19:27:58.989 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-05T19:27:58.983+0000 7efd37a77700 1 -- 172.21.15.118:0/2281221655 shutdown_connections 2023-10-05T19:27:58.989 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-05T19:27:58.983+0000 7efd37a77700 1 --2- 172.21.15.118:0/2281221655 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7efd30095650 0x7efd30125c60 unknown :-1 s=CLOSED pgs=90 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-05T19:27:58.989 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-05T19:27:58.983+0000 7efd37a77700 1 -- 172.21.15.118:0/2281221655 >> 172.21.15.118:0/2281221655 conn(0x7efd30098f60 msgr2=0x7efd300949f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-05T19:27:58.989 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-05T19:27:58.983+0000 7efd37a77700 1 -- 172.21.15.118:0/2281221655 shutdown_connections 2023-10-05T19:27:58.989 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-05T19:27:58.983+0000 7efd37a77700 1 -- 172.21.15.118:0/2281221655 wait complete. 2023-10-05T19:27:58.990 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-05T19:27:58.983+0000 7efd37a77700 1 Processor -- start 2023-10-05T19:27:58.990 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-05T19:27:58.983+0000 7efd37a77700 1 -- start start 2023-10-05T19:27:58.991 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-05T19:27:58.983+0000 7efd37a77700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7efd30095650 0x7efd30116d70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-05T19:27:58.991 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-05T19:27:58.983+0000 7efd37a77700 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efd3011a640 con 0x7efd30095650 2023-10-05T19:27:58.991 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-05T19:27:58.983+0000 7efd35813700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7efd30095650 0x7efd30116d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-05T19:27:58.991 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-05T19:27:58.983+0000 7efd35813700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7efd30095650 0x7efd30116d70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.118:3300/0 says I am v2:172.21.15.118:55848/0 (socket says 172.21.15.118:55848) 2023-10-05T19:27:58.991 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-05T19:27:58.983+0000 7efd35813700 1 -- 172.21.15.118:0/2819438557 learned_addr learned my addr 172.21.15.118:0/2819438557 (peer_addr_for_me v2:172.21.15.118:0/0) 2023-10-05T19:27:58.992 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-05T19:27:58.983+0000 7efd35813700 1 -- 172.21.15.118:0/2819438557 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efd2000b6d0 con 0x7efd30095650 2023-10-05T19:27:58.992 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-05T19:27:58.983+0000 7efd35813700 1 --2- 172.21.15.118:0/2819438557 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7efd30095650 0x7efd30116d70 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7efd2000b6a0 tx=0x7efd20019000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-05T19:27:58.992 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-05T19:27:58.983+0000 7efd25ffb700 1 -- 172.21.15.118:0/2819438557 <== mon.0 v2:172.21.15.118:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7efd200218c0 con 0x7efd30095650 2023-10-05T19:27:58.992 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-05T19:27:58.983+0000 7efd25ffb700 1 -- 172.21.15.118:0/2819438557 <== mon.0 v2:172.21.15.118:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efd20021a20 con 0x7efd30095650 2023-10-05T19:27:58.992 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-05T19:27:58.983+0000 7efd25ffb700 1 -- 172.21.15.118:0/2819438557 <== mon.0 v2:172.21.15.118:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7efd20021cd0 con 0x7efd30095650 2023-10-05T19:27:58.992 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-05T19:27:58.983+0000 7efd37a77700 1 -- 172.21.15.118:0/2819438557 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efd30119360 con 0x7efd30095650 2023-10-05T19:27:58.993 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-05T19:27:58.983+0000 7efd37a77700 1 -- 172.21.15.118:0/2819438557 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efd30119a90 con 0x7efd30095650 2023-10-05T19:27:58.994 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-05T19:27:58.987+0000 7efd25ffb700 1 -- 172.21.15.118:0/2819438557 <== mon.0 v2:172.21.15.118:3300/0 4 ==== mgrmap(e 5) v1 ==== 40329+0+0 (secure 0 0 0) 0x7efd20015070 con 0x7efd30095650 2023-10-05T19:27:58.994 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-05T19:27:58.987+0000 7efd25ffb700 1 --2- 172.21.15.118:0/2819438557 >> [v2:172.21.15.167:6824/29889,v1:172.21.15.167:6825/29889] conn(0x7efd140323e0 0x7efd14034890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-05T19:27:58.994 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-05T19:27:58.987+0000 7efd25ffb700 1 -- 172.21.15.118:0/2819438557 <== mon.0 v2:172.21.15.118:3300/0 5 ==== osd_map(15..15 src has 1..15) v4 ==== 3478+0+0 (secure 0 0 0) 0x7efd20044ff0 con 0x7efd30095650 2023-10-05T19:27:58.995 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-05T19:27:58.987+0000 7efd35012700 1 --2- 172.21.15.118:0/2819438557 >> [v2:172.21.15.167:6824/29889,v1:172.21.15.167:6825/29889] conn(0x7efd140323e0 0x7efd14034890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-05T19:27:58.995 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-05T19:27:58.987+0000 7efd35012700 1 --2- 172.21.15.118:0/2819438557 >> [v2:172.21.15.167:6824/29889,v1:172.21.15.167:6825/29889] conn(0x7efd140323e0 0x7efd14034890 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7efd2c0016a0 tx=0x7efd2c007b30 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-05T19:27:58.996 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-05T19:27:58.991+0000 7efd37a77700 1 -- 172.21.15.118:0/2819438557 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efd3005ecb0 con 0x7efd30095650 2023-10-05T19:27:59.003 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-05T19:27:58.995+0000 7efd25ffb700 1 -- 172.21.15.118:0/2819438557 <== mon.0 v2:172.21.15.118:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166807 (secure 0 0 0) 0x7efd20026030 con 0x7efd30095650 2023-10-05T19:27:59.118 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-05T19:27:59.111+0000 7f16f321d700 1 -- 172.21.15.118:0/2201640378 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 2} v 0) v1 -- 0x7f16ec0934e0 con 0x7f16ec095ff0 2023-10-05T19:27:59.118 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-05T19:27:59.111+0000 7f16e27fc700 1 -- 172.21.15.118:0/2201640378 <== mon.0 v2:172.21.15.118:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 2}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f16e8017070 con 0x7f16ec095ff0 2023-10-05T19:27:59.119 INFO:teuthology.orchestra.run.smithi118.stdout:38654705669 2023-10-05T19:27:59.121 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-05T19:27:59.115+0000 7f16f321d700 1 -- 172.21.15.118:0/2201640378 >> [v2:172.21.15.167:6824/29889,v1:172.21.15.167:6825/29889] conn(0x7f16d00323d0 msgr2=0x7f16d0034880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-05T19:27:59.121 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-05T19:27:59.115+0000 7f16f321d700 1 --2- 172.21.15.118:0/2201640378 >> [v2:172.21.15.167:6824/29889,v1:172.21.15.167:6825/29889] conn(0x7f16d00323d0 0x7f16d0034880 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7f16dc003080 tx=0x7f16dc015000 comp rx=0 tx=0).stop 2023-10-05T19:27:59.121 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-05T19:27:59.115+0000 7f16f321d700 1 -- 172.21.15.118:0/2201640378 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f16ec095ff0 msgr2=0x7f16ec12b7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-05T19:27:59.121 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-05T19:27:59.115+0000 7f16f321d700 1 --2- 172.21.15.118:0/2201640378 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f16ec095ff0 0x7f16ec12b7a0 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f16e800c6c0 tx=0x7f16e8007e00 comp rx=0 tx=0).stop 2023-10-05T19:27:59.121 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-05T19:27:59.115+0000 7f16f321d700 1 -- 172.21.15.118:0/2201640378 shutdown_connections 2023-10-05T19:27:59.121 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-05T19:27:59.115+0000 7f16f321d700 1 --2- 172.21.15.118:0/2201640378 >> [v2:172.21.15.167:6824/29889,v1:172.21.15.167:6825/29889] conn(0x7f16d00323d0 0x7f16d0034880 unknown :-1 s=CLOSED pgs=34 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-05T19:27:59.121 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-05T19:27:59.115+0000 7f16f321d700 1 --2- 172.21.15.118:0/2201640378 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f16ec095ff0 0x7f16ec12b7a0 unknown :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-05T19:27:59.121 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-05T19:27:59.115+0000 7f16f321d700 1 -- 172.21.15.118:0/2201640378 >> 172.21.15.118:0/2201640378 conn(0x7f16ec09c5d0 msgr2=0x7f16ec09af70 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-05T19:27:59.122 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-05T19:27:59.115+0000 7f16f321d700 1 -- 172.21.15.118:0/2201640378 shutdown_connections 2023-10-05T19:27:59.122 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-05T19:27:59.115+0000 7f16f321d700 1 -- 172.21.15.118:0/2201640378 wait complete. 2023-10-05T19:27:59.129 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-05T19:27:59.123+0000 7fbb3fff6700 1 -- 172.21.15.118:0/1876835572 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 3} v 0) v1 -- 0x7fbb3805ecb0 con 0x7fbb381260f0 2023-10-05T19:27:59.129 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-05T19:27:59.123+0000 7fbb2dffb700 1 -- 172.21.15.118:0/1876835572 <== mon.0 v2:172.21.15.118:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 3}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7fbb340193c0 con 0x7fbb381260f0 2023-10-05T19:27:59.130 INFO:teuthology.orchestra.run.smithi118.stdout:38654705667 2023-10-05T19:27:59.131 INFO:tasks.ceph.ceph_manager.ceph:need seq 38654705670 got 38654705669 for osd.2 2023-10-05T19:27:59.132 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-05T19:27:59.123+0000 7fbb3fff6700 1 -- 172.21.15.118:0/1876835572 >> [v2:172.21.15.167:6824/29889,v1:172.21.15.167:6825/29889] conn(0x7fbb1c0323e0 msgr2=0x7fbb1c034890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-05T19:27:59.132 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-05T19:27:59.123+0000 7fbb3fff6700 1 --2- 172.21.15.118:0/1876835572 >> [v2:172.21.15.167:6824/29889,v1:172.21.15.167:6825/29889] conn(0x7fbb1c0323e0 0x7fbb1c034890 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7fbb280016c0 tx=0x7fbb28007af0 comp rx=0 tx=0).stop 2023-10-05T19:27:59.132 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-05T19:27:59.123+0000 7fbb3fff6700 1 -- 172.21.15.118:0/1876835572 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fbb381260f0 msgr2=0x7fbb38116790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-05T19:27:59.132 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-05T19:27:59.123+0000 7fbb3fff6700 1 --2- 172.21.15.118:0/1876835572 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fbb381260f0 0x7fbb38116790 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7fbb3806e950 tx=0x7fbb34016000 comp rx=0 tx=0).stop 2023-10-05T19:27:59.133 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-05T19:27:59.123+0000 7fbb3fff6700 1 -- 172.21.15.118:0/1876835572 shutdown_connections 2023-10-05T19:27:59.133 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-05T19:27:59.123+0000 7fbb3fff6700 1 --2- 172.21.15.118:0/1876835572 >> [v2:172.21.15.167:6824/29889,v1:172.21.15.167:6825/29889] conn(0x7fbb1c0323e0 0x7fbb1c034890 unknown :-1 s=CLOSED pgs=35 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-05T19:27:59.133 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-05T19:27:59.123+0000 7fbb3fff6700 1 --2- 172.21.15.118:0/1876835572 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fbb381260f0 0x7fbb38116790 unknown :-1 s=CLOSED pgs=85 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-05T19:27:59.133 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-05T19:27:59.123+0000 7fbb3fff6700 1 -- 172.21.15.118:0/1876835572 >> 172.21.15.118:0/1876835572 conn(0x7fbb38098f60 msgr2=0x7fbb38096ce0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-05T19:27:59.133 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-05T19:27:59.123+0000 7fbb3fff6700 1 -- 172.21.15.118:0/1876835572 shutdown_connections 2023-10-05T19:27:59.133 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-05T19:27:59.123+0000 7fbb3fff6700 1 -- 172.21.15.118:0/1876835572 wait complete. 2023-10-05T19:27:59.143 INFO:tasks.ceph.ceph_manager.ceph:need seq 38654705668 got 38654705667 for osd.3 2023-10-05T19:27:59.149 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-05T19:27:59.143+0000 7f71ed42c700 1 -- 172.21.15.118:0/1230019494 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 1} v 0) v1 -- 0x7f71e8125950 con 0x7f71e80959a0 2023-10-05T19:27:59.149 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-05T19:27:59.143+0000 7f71cffff700 1 -- 172.21.15.118:0/1230019494 <== mon.0 v2:172.21.15.118:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 1}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f71e8125950 con 0x7f71e80959a0 2023-10-05T19:27:59.150 INFO:teuthology.orchestra.run.smithi118.stdout:38654705668 2023-10-05T19:27:59.152 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-05T19:27:59.143+0000 7f71ed42c700 1 -- 172.21.15.118:0/1230019494 >> [v2:172.21.15.167:6824/29889,v1:172.21.15.167:6825/29889] conn(0x7f71d4032430 msgr2=0x7f71d40348e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-05T19:27:59.152 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-05T19:27:59.143+0000 7f71ed42c700 1 --2- 172.21.15.118:0/1230019494 >> [v2:172.21.15.167:6824/29889,v1:172.21.15.167:6825/29889] conn(0x7f71d4032430 0x7f71d40348e0 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f71dc0030c0 tx=0x7f71dc016000 comp rx=0 tx=0).stop 2023-10-05T19:27:59.152 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-05T19:27:59.143+0000 7f71ed42c700 1 -- 172.21.15.118:0/1230019494 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f71e80959a0 msgr2=0x7f71e8113130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-05T19:27:59.152 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-05T19:27:59.143+0000 7f71ed42c700 1 --2- 172.21.15.118:0/1230019494 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f71e80959a0 0x7f71e8113130 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f71e000cbb0 tx=0x7f71e0001ac0 comp rx=0 tx=0).stop 2023-10-05T19:27:59.152 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-05T19:27:59.143+0000 7f71ed42c700 1 -- 172.21.15.118:0/1230019494 shutdown_connections 2023-10-05T19:27:59.153 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-05T19:27:59.143+0000 7f71ed42c700 1 --2- 172.21.15.118:0/1230019494 >> [v2:172.21.15.167:6824/29889,v1:172.21.15.167:6825/29889] conn(0x7f71d4032430 0x7f71d40348e0 unknown :-1 s=CLOSED pgs=36 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-05T19:27:59.153 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-05T19:27:59.143+0000 7f71ed42c700 1 --2- 172.21.15.118:0/1230019494 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f71e80959a0 0x7f71e8113130 unknown :-1 s=CLOSED pgs=88 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-05T19:27:59.153 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-05T19:27:59.143+0000 7f71ed42c700 1 -- 172.21.15.118:0/1230019494 >> 172.21.15.118:0/1230019494 conn(0x7f71e809a530 msgr2=0x7f71e8099830 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-05T19:27:59.153 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-05T19:27:59.143+0000 7f71ed42c700 1 -- 172.21.15.118:0/1230019494 shutdown_connections 2023-10-05T19:27:59.153 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-05T19:27:59.143+0000 7f71ed42c700 1 -- 172.21.15.118:0/1230019494 wait complete. 2023-10-05T19:27:59.161 INFO:tasks.ceph.ceph_manager.ceph:need seq 38654705669 got 38654705668 for osd.1 2023-10-05T19:27:59.165 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-05T19:27:59.159+0000 7efd37a77700 1 -- 172.21.15.118:0/2819438557 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 0} v 0) v1 -- 0x7efd3004b5c0 con 0x7efd30095650 2023-10-05T19:27:59.166 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-05T19:27:59.159+0000 7efd25ffb700 1 -- 172.21.15.118:0/2819438557 <== mon.0 v2:172.21.15.118:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 0}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7efd20026020 con 0x7efd30095650 2023-10-05T19:27:59.166 INFO:teuthology.orchestra.run.smithi118.stdout:38654705667 2023-10-05T19:27:59.168 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-05T19:27:59.163+0000 7efd37a77700 1 -- 172.21.15.118:0/2819438557 >> [v2:172.21.15.167:6824/29889,v1:172.21.15.167:6825/29889] conn(0x7efd140323e0 msgr2=0x7efd14034890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-05T19:27:59.169 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-05T19:27:59.163+0000 7efd37a77700 1 --2- 172.21.15.118:0/2819438557 >> [v2:172.21.15.167:6824/29889,v1:172.21.15.167:6825/29889] conn(0x7efd140323e0 0x7efd14034890 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7efd2c0016a0 tx=0x7efd2c007b30 comp rx=0 tx=0).stop 2023-10-05T19:27:59.169 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-05T19:27:59.163+0000 7efd37a77700 1 -- 172.21.15.118:0/2819438557 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7efd30095650 msgr2=0x7efd30116d70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-05T19:27:59.169 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-05T19:27:59.163+0000 7efd37a77700 1 --2- 172.21.15.118:0/2819438557 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7efd30095650 0x7efd30116d70 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7efd2000b6a0 tx=0x7efd20019000 comp rx=0 tx=0).stop 2023-10-05T19:27:59.169 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-05T19:27:59.163+0000 7efd37a77700 1 -- 172.21.15.118:0/2819438557 shutdown_connections 2023-10-05T19:27:59.169 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-05T19:27:59.163+0000 7efd37a77700 1 --2- 172.21.15.118:0/2819438557 >> [v2:172.21.15.167:6824/29889,v1:172.21.15.167:6825/29889] conn(0x7efd140323e0 0x7efd14034890 secure :-1 s=CLOSED pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7efd2c0016a0 tx=0x7efd2c007b30 comp rx=0 tx=0).stop 2023-10-05T19:27:59.169 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-05T19:27:59.163+0000 7efd37a77700 1 --2- 172.21.15.118:0/2819438557 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7efd30095650 0x7efd30116d70 unknown :-1 s=CLOSED pgs=91 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-05T19:27:59.169 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-05T19:27:59.163+0000 7efd37a77700 1 -- 172.21.15.118:0/2819438557 >> 172.21.15.118:0/2819438557 conn(0x7efd30098f60 msgr2=0x7efd30069650 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-05T19:27:59.169 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-05T19:27:59.163+0000 7efd37a77700 1 -- 172.21.15.118:0/2819438557 shutdown_connections 2023-10-05T19:27:59.169 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-05T19:27:59.163+0000 7efd37a77700 1 -- 172.21.15.118:0/2819438557 wait complete. 2023-10-05T19:27:59.178 INFO:tasks.ceph.ceph_manager.ceph:need seq 38654705668 got 38654705667 for osd.0 2023-10-05T19:28:00.131 DEBUG:teuthology.orchestra.run.smithi118:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2023-10-05T19:28:00.143 DEBUG:teuthology.orchestra.run.smithi118:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3 2023-10-05T19:28:00.162 DEBUG:teuthology.orchestra.run.smithi118:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2023-10-05T19:28:00.179 DEBUG:teuthology.orchestra.run.smithi118:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2023-10-05T19:28:00.250 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-05T19:28:00.243+0000 7f65f89dc700 1 Processor -- start 2023-10-05T19:28:00.250 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-05T19:28:00.243+0000 7f65f89dc700 1 -- start start 2023-10-05T19:28:00.251 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-05T19:28:00.243+0000 7f65f89dc700 1 --2- >> v2:172.21.15.118:3300/0 conn(0x7f65f4095ff0 0x7f65f40963c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-05T19:28:00.251 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-05T19:28:00.243+0000 7f65f89dc700 1 -- --> v1:172.21.15.118:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f65f4092760 con 0x7f65f40905f0 2023-10-05T19:28:00.251 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-05T19:28:00.243+0000 7f65f89dc700 1 -- --> v2:172.21.15.118:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f65f40918a0 con 0x7f65f4095ff0 2023-10-05T19:28:00.252 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-05T19:28:00.243+0000 7f65f37fe700 1 --2- >> v2:172.21.15.118:3300/0 conn(0x7f65f4095ff0 0x7f65f40963c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-05T19:28:00.252 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-05T19:28:00.243+0000 7f65ebfff700 1 --1- >> v1:172.21.15.118:6789/0 conn(0x7f65f40905f0 0x7f65f40909c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.118:6789/0 says I am v1:172.21.15.118:52126/0 (socket says 172.21.15.118:52126) 2023-10-05T19:28:00.252 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-05T19:28:00.243+0000 7f65ebfff700 1 -- 172.21.15.118:0/3702452172 learned_addr learned my addr 172.21.15.118:0/3702452172 (peer_addr_for_me v1:172.21.15.118:0/0) 2023-10-05T19:28:00.252 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-05T19:28:00.243+0000 7f65f37fe700 1 -- 172.21.15.118:0/3702452172 >> v1:172.21.15.118:6789/0 conn(0x7f65f40905f0 legacy=0x7f65f40909c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-05T19:28:00.252 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-05T19:28:00.243+0000 7f65f2ffd700 1 -- 172.21.15.118:0/3702452172 <== mon.0 v1:172.21.15.118:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1779679678 0 0) 0x7f65f4092760 con 0x7f65f40905f0 2023-10-05T19:28:00.252 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-05T19:28:00.243+0000 7f65f37fe700 1 -- 172.21.15.118:0/3702452172 --> v2:172.21.15.118:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f65f4093ba0 con 0x7f65f4095ff0 2023-10-05T19:28:00.252 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-05T19:28:00.243+0000 7f65f37fe700 1 --2- 172.21.15.118:0/3702452172 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f65f4095ff0 0x7f65f40963c0 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f65dc009860 tx=0x7f65dc0091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7484bca49ae874d0 server_cookie=0 in_seq=0 out_seq=0 2023-10-05T19:28:00.252 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-05T19:28:00.243+0000 7f65f2ffd700 1 -- 172.21.15.118:0/3702452172 <== mon.0 v2:172.21.15.118:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f65dc00b680 con 0x7f65f4095ff0 2023-10-05T19:28:00.252 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-05T19:28:00.243+0000 7f65f2ffd700 1 -- 172.21.15.118:0/3702452172 <== mon.0 v2:172.21.15.118:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f65dc00b7e0 con 0x7f65f4095ff0 2023-10-05T19:28:00.252 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-05T19:28:00.243+0000 7f65f2ffd700 1 -- 172.21.15.118:0/3702452172 <== mon.0 v2:172.21.15.118:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f65dc00b990 con 0x7f65f4095ff0 2023-10-05T19:28:00.253 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-05T19:28:00.247+0000 7f65f89dc700 1 -- 172.21.15.118:0/3702452172 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f65f4095ff0 msgr2=0x7f65f40963c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-05T19:28:00.253 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-05T19:28:00.247+0000 7f65f89dc700 1 --2- 172.21.15.118:0/3702452172 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f65f4095ff0 0x7f65f40963c0 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f65dc009860 tx=0x7f65dc0091a0 comp rx=0 tx=0).stop 2023-10-05T19:28:00.254 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-05T19:28:00.247+0000 7f65f89dc700 1 -- 172.21.15.118:0/3702452172 shutdown_connections 2023-10-05T19:28:00.254 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-05T19:28:00.247+0000 7f65f89dc700 1 --2- 172.21.15.118:0/3702452172 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f65f4095ff0 0x7f65f40963c0 unknown :-1 s=CLOSED pgs=93 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-05T19:28:00.254 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-05T19:28:00.247+0000 7f65f89dc700 1 -- 172.21.15.118:0/3702452172 >> 172.21.15.118:0/3702452172 conn(0x7f65f409c5d0 msgr2=0x7f65f409c9d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-05T19:28:00.254 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-05T19:28:00.247+0000 7f65f89dc700 1 -- 172.21.15.118:0/3702452172 shutdown_connections 2023-10-05T19:28:00.254 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-05T19:28:00.247+0000 7f65f89dc700 1 -- 172.21.15.118:0/3702452172 wait complete. 2023-10-05T19:28:00.254 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-05T19:28:00.247+0000 7f65f89dc700 1 Processor -- start 2023-10-05T19:28:00.254 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-05T19:28:00.247+0000 7f65f89dc700 1 -- start start 2023-10-05T19:28:00.255 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-05T19:28:00.247+0000 7f65f89dc700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f65f40905f0 0x7f65f408bf20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-05T19:28:00.255 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-05T19:28:00.247+0000 7f65f89dc700 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f65f40903a0 con 0x7f65f40905f0 2023-10-05T19:28:00.255 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-05T19:28:00.247+0000 7f65f37fe700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f65f40905f0 0x7f65f408bf20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-05T19:28:00.255 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-05T19:28:00.247+0000 7f65f37fe700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f65f40905f0 0x7f65f408bf20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.118:3300/0 says I am v2:172.21.15.118:55872/0 (socket says 172.21.15.118:55872) 2023-10-05T19:28:00.256 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-05T19:28:00.247+0000 7f65f37fe700 1 -- 172.21.15.118:0/2675926607 learned_addr learned my addr 172.21.15.118:0/2675926607 (peer_addr_for_me v2:172.21.15.118:0/0) 2023-10-05T19:28:00.256 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-05T19:28:00.247+0000 7f65f37fe700 1 -- 172.21.15.118:0/2675926607 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f65dc009420 con 0x7f65f40905f0 2023-10-05T19:28:00.256 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-05T19:28:00.247+0000 7f65f37fe700 1 --2- 172.21.15.118:0/2675926607 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f65f40905f0 0x7f65f408bf20 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7f65dc000c00 tx=0x7f65dc016000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-05T19:28:00.256 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-05T19:28:00.247+0000 7f65eb7fe700 1 -- 172.21.15.118:0/2675926607 <== mon.0 v2:172.21.15.118:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f65dc011e70 con 0x7f65f40905f0 2023-10-05T19:28:00.256 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-05T19:28:00.247+0000 7f65eb7fe700 1 -- 172.21.15.118:0/2675926607 <== mon.0 v2:172.21.15.118:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f65dc020070 con 0x7f65f40905f0 2023-10-05T19:28:00.256 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-05T19:28:00.247+0000 7f65eb7fe700 1 -- 172.21.15.118:0/2675926607 <== mon.0 v2:172.21.15.118:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f65dc01b5b0 con 0x7f65f40905f0 2023-10-05T19:28:00.256 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-05T19:28:00.247+0000 7f65f89dc700 1 -- 172.21.15.118:0/2675926607 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f65f408d710 con 0x7f65f40905f0 2023-10-05T19:28:00.256 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-05T19:28:00.247+0000 7f65f89dc700 1 -- 172.21.15.118:0/2675926607 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f65f408e920 con 0x7f65f40905f0 2023-10-05T19:28:00.257 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-05T19:28:00.251+0000 7f65eb7fe700 1 -- 172.21.15.118:0/2675926607 <== mon.0 v2:172.21.15.118:3300/0 4 ==== mgrmap(e 5) v1 ==== 40329+0+0 (secure 0 0 0) 0x7f65dc022030 con 0x7f65f40905f0 2023-10-05T19:28:00.257 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-05T19:28:00.251+0000 7f65eb7fe700 1 --2- 172.21.15.118:0/2675926607 >> [v2:172.21.15.167:6824/29889,v1:172.21.15.167:6825/29889] conn(0x7f65d40323e0 0x7f65d4034890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-05T19:28:00.257 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-05T19:28:00.251+0000 7f65eb7fe700 1 -- 172.21.15.118:0/2675926607 <== mon.0 v2:172.21.15.118:3300/0 5 ==== osd_map(15..15 src has 1..15) v4 ==== 3478+0+0 (secure 0 0 0) 0x7f65dc0459f0 con 0x7f65f40905f0 2023-10-05T19:28:00.257 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-05T19:28:00.251+0000 7f65f89dc700 1 -- 172.21.15.118:0/2675926607 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f65f405ecb0 con 0x7f65f40905f0 2023-10-05T19:28:00.257 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-05T19:28:00.251+0000 7f65ebfff700 1 --2- 172.21.15.118:0/2675926607 >> [v2:172.21.15.167:6824/29889,v1:172.21.15.167:6825/29889] conn(0x7f65d40323e0 0x7f65d4034890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-05T19:28:00.258 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-05T19:28:00.251+0000 7f65ebfff700 1 --2- 172.21.15.118:0/2675926607 >> [v2:172.21.15.167:6824/29889,v1:172.21.15.167:6825/29889] conn(0x7f65d40323e0 0x7f65d4034890 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f65e4000c00 tx=0x7f65e4007810 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-05T19:28:00.261 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-05T19:28:00.255+0000 7fbd3084a700 1 Processor -- start 2023-10-05T19:28:00.261 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-05T19:28:00.255+0000 7fbd3084a700 1 -- start start 2023-10-05T19:28:00.262 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-05T19:28:00.255+0000 7f65eb7fe700 1 -- 172.21.15.118:0/2675926607 <== mon.0 v2:172.21.15.118:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166807 (secure 0 0 0) 0x7f65dc0249c0 con 0x7f65f40905f0 2023-10-05T19:28:00.263 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-05T19:28:00.255+0000 7fbd3084a700 1 --2- >> v2:172.21.15.118:3300/0 conn(0x7fbd2c06b280 0x7fbd2c06b650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-05T19:28:00.263 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-05T19:28:00.255+0000 7fbd3084a700 1 -- --> v1:172.21.15.118:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbd2c06b060 con 0x7fbd2c094e60 2023-10-05T19:28:00.263 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-05T19:28:00.255+0000 7fbd3084a700 1 -- --> v2:172.21.15.118:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbd2c094b40 con 0x7fbd2c06b280 2023-10-05T19:28:00.263 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-05T19:28:00.255+0000 7fbd2ad9d700 1 --2- >> v2:172.21.15.118:3300/0 conn(0x7fbd2c06b280 0x7fbd2c06b650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-05T19:28:00.263 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-05T19:28:00.255+0000 7fbd2a59c700 1 --1- >> v1:172.21.15.118:6789/0 conn(0x7fbd2c094e60 0x7fbd2c126010 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.118:6789/0 says I am v1:172.21.15.118:52140/0 (socket says 172.21.15.118:52140) 2023-10-05T19:28:00.263 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-05T19:28:00.255+0000 7fbd2a59c700 1 -- 172.21.15.118:0/2738049794 learned_addr learned my addr 172.21.15.118:0/2738049794 (peer_addr_for_me v1:172.21.15.118:0/0) 2023-10-05T19:28:00.263 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-05T19:28:00.255+0000 7fbd2ad9d700 1 -- 172.21.15.118:0/2738049794 >> v1:172.21.15.118:6789/0 conn(0x7fbd2c094e60 legacy=0x7fbd2c126010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-05T19:28:00.263 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-05T19:28:00.255+0000 7fbd29d9b700 1 -- 172.21.15.118:0/2738049794 <== mon.0 v1:172.21.15.118:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 164270542 0 0) 0x7fbd2c06b060 con 0x7fbd2c094e60 2023-10-05T19:28:00.263 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-05T19:28:00.255+0000 7fbd2ad9d700 1 -- 172.21.15.118:0/2738049794 --> v2:172.21.15.118:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbd2c0953b0 con 0x7fbd2c06b280 2023-10-05T19:28:00.263 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-05T19:28:00.255+0000 7fbd2ad9d700 1 --2- 172.21.15.118:0/2738049794 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fbd2c06b280 0x7fbd2c06b650 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7fbd20009860 tx=0x7fbd200091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6021fa1bacd0094d server_cookie=0 in_seq=0 out_seq=0 2023-10-05T19:28:00.264 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-05T19:28:00.255+0000 7fbd29d9b700 1 -- 172.21.15.118:0/2738049794 <== mon.0 v2:172.21.15.118:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fbd2000f6b0 con 0x7fbd2c06b280 2023-10-05T19:28:00.264 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-05T19:28:00.255+0000 7fbd29d9b700 1 -- 172.21.15.118:0/2738049794 <== mon.0 v2:172.21.15.118:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbd2000f810 con 0x7fbd2c06b280 2023-10-05T19:28:00.264 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-05T19:28:00.255+0000 7fbd29d9b700 1 -- 172.21.15.118:0/2738049794 <== mon.0 v2:172.21.15.118:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fbd2000fb00 con 0x7fbd2c06b280 2023-10-05T19:28:00.265 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-05T19:28:00.259+0000 7fbd3084a700 1 -- 172.21.15.118:0/2738049794 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fbd2c06b280 msgr2=0x7fbd2c06b650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-05T19:28:00.265 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-05T19:28:00.259+0000 7fbd3084a700 1 --2- 172.21.15.118:0/2738049794 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fbd2c06b280 0x7fbd2c06b650 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7fbd20009860 tx=0x7fbd200091a0 comp rx=0 tx=0).stop 2023-10-05T19:28:00.265 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-05T19:28:00.259+0000 7fbd3084a700 1 -- 172.21.15.118:0/2738049794 shutdown_connections 2023-10-05T19:28:00.265 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-05T19:28:00.259+0000 7fbd3084a700 1 --2- 172.21.15.118:0/2738049794 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fbd2c06b280 0x7fbd2c06b650 unknown :-1 s=CLOSED pgs=96 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-05T19:28:00.265 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-05T19:28:00.259+0000 7fbd3084a700 1 -- 172.21.15.118:0/2738049794 >> 172.21.15.118:0/2738049794 conn(0x7fbd2c095970 msgr2=0x7fbd2c095d70 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-05T19:28:00.265 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-05T19:28:00.259+0000 7fbd3084a700 1 -- 172.21.15.118:0/2738049794 shutdown_connections 2023-10-05T19:28:00.265 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-05T19:28:00.259+0000 7fbd3084a700 1 -- 172.21.15.118:0/2738049794 wait complete. 2023-10-05T19:28:00.266 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-05T19:28:00.259+0000 7fbd3084a700 1 Processor -- start 2023-10-05T19:28:00.267 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-05T19:28:00.259+0000 7fbd3084a700 1 -- start start 2023-10-05T19:28:00.267 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-05T19:28:00.259+0000 7fbd3084a700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fbd2c06b280 0x7fbd2c087b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-05T19:28:00.267 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-05T19:28:00.259+0000 7fbd3084a700 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbd2c08af00 con 0x7fbd2c06b280 2023-10-05T19:28:00.268 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-05T19:28:00.259+0000 7fbd2ad9d700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fbd2c06b280 0x7fbd2c087b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-05T19:28:00.268 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-05T19:28:00.259+0000 7fbd2ad9d700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fbd2c06b280 0x7fbd2c087b70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.118:3300/0 says I am v2:172.21.15.118:55898/0 (socket says 172.21.15.118:55898) 2023-10-05T19:28:00.268 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-05T19:28:00.259+0000 7fbd2ad9d700 1 -- 172.21.15.118:0/1105091630 learned_addr learned my addr 172.21.15.118:0/1105091630 (peer_addr_for_me v2:172.21.15.118:0/0) 2023-10-05T19:28:00.268 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-05T19:28:00.263+0000 7fbd2ad9d700 1 -- 172.21.15.118:0/1105091630 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbd20009420 con 0x7fbd2c06b280 2023-10-05T19:28:00.269 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-05T19:28:00.263+0000 7fbd2ad9d700 1 --2- 172.21.15.118:0/1105091630 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fbd2c06b280 0x7fbd2c087b70 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7fbd200091e0 tx=0x7fbd20018000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-05T19:28:00.269 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-05T19:28:00.263+0000 7fbd12ffd700 1 -- 172.21.15.118:0/1105091630 <== mon.0 v2:172.21.15.118:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fbd2001f070 con 0x7fbd2c06b280 2023-10-05T19:28:00.269 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-05T19:28:00.263+0000 7fbd12ffd700 1 -- 172.21.15.118:0/1105091630 <== mon.0 v2:172.21.15.118:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbd20010e20 con 0x7fbd2c06b280 2023-10-05T19:28:00.269 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-05T19:28:00.263+0000 7fbd3084a700 1 -- 172.21.15.118:0/1105091630 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbd2c098360 con 0x7fbd2c06b280 2023-10-05T19:28:00.270 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-05T19:28:00.263+0000 7fbd12ffd700 1 -- 172.21.15.118:0/1105091630 <== mon.0 v2:172.21.15.118:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fbd200187b0 con 0x7fbd2c06b280 2023-10-05T19:28:00.270 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-05T19:28:00.263+0000 7fbd3084a700 1 -- 172.21.15.118:0/1105091630 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbd2c089de0 con 0x7fbd2c06b280 2023-10-05T19:28:00.270 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-05T19:28:00.263+0000 7fbd12ffd700 1 -- 172.21.15.118:0/1105091630 <== mon.0 v2:172.21.15.118:3300/0 4 ==== mgrmap(e 5) v1 ==== 40329+0+0 (secure 0 0 0) 0x7fbd20018910 con 0x7fbd2c06b280 2023-10-05T19:28:00.270 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-05T19:28:00.263+0000 7fbd3084a700 1 -- 172.21.15.118:0/1105091630 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbd2c092fb0 con 0x7fbd2c06b280 2023-10-05T19:28:00.270 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-05T19:28:00.263+0000 7fbd12ffd700 1 --2- 172.21.15.118:0/1105091630 >> [v2:172.21.15.167:6824/29889,v1:172.21.15.167:6825/29889] conn(0x7fbd180323e0 0x7fbd18034890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-05T19:28:00.270 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-05T19:28:00.263+0000 7fbd12ffd700 1 -- 172.21.15.118:0/1105091630 <== mon.0 v2:172.21.15.118:3300/0 5 ==== osd_map(15..15 src has 1..15) v4 ==== 3478+0+0 (secure 0 0 0) 0x7fbd20014070 con 0x7fbd2c06b280 2023-10-05T19:28:00.271 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-05T19:28:00.263+0000 7fbd2a59c700 1 --2- 172.21.15.118:0/1105091630 >> [v2:172.21.15.167:6824/29889,v1:172.21.15.167:6825/29889] conn(0x7fbd180323e0 0x7fbd18034890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-05T19:28:00.271 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-05T19:28:00.263+0000 7fbd2a59c700 1 --2- 172.21.15.118:0/1105091630 >> [v2:172.21.15.167:6824/29889,v1:172.21.15.167:6825/29889] conn(0x7fbd180323e0 0x7fbd18034890 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7fbd1c0011e0 tx=0x7fbd1c0079d0 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-05T19:28:00.278 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-05T19:28:00.271+0000 7fbd12ffd700 1 -- 172.21.15.118:0/1105091630 <== mon.0 v2:172.21.15.118:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166807 (secure 0 0 0) 0x7fbd20024980 con 0x7fbd2c06b280 2023-10-05T19:28:00.284 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-05T19:28:00.279+0000 7f5e19d73700 1 Processor -- start 2023-10-05T19:28:00.285 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-05T19:28:00.279+0000 7f5e19d73700 1 -- start start 2023-10-05T19:28:00.287 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-05T19:28:00.279+0000 7f5e19d73700 1 --2- >> v2:172.21.15.118:3300/0 conn(0x7f5e1406d730 0x7f5e1412e690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-05T19:28:00.288 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-05T19:28:00.279+0000 7f5e19d73700 1 -- --> v1:172.21.15.118:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5e1406b850 con 0x7f5e1406cc50 2023-10-05T19:28:00.288 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-05T19:28:00.279+0000 7f5e19d73700 1 -- --> v2:172.21.15.118:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5e1406db60 con 0x7f5e1406d730 2023-10-05T19:28:00.288 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-05T19:28:00.279+0000 7f5e13fff700 1 --2- >> v2:172.21.15.118:3300/0 conn(0x7f5e1406d730 0x7f5e1412e690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-05T19:28:00.288 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-05T19:28:00.279+0000 7f5e18d71700 1 --1- >> v1:172.21.15.118:6789/0 conn(0x7f5e1406cc50 0x7f5e1406d020 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.118:6789/0 says I am v1:172.21.15.118:52154/0 (socket says 172.21.15.118:52154) 2023-10-05T19:28:00.288 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-05T19:28:00.279+0000 7f5e18d71700 1 -- 172.21.15.118:0/3178472746 learned_addr learned my addr 172.21.15.118:0/3178472746 (peer_addr_for_me v1:172.21.15.118:0/0) 2023-10-05T19:28:00.288 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-05T19:28:00.279+0000 7f5e13fff700 1 -- 172.21.15.118:0/3178472746 >> v1:172.21.15.118:6789/0 conn(0x7f5e1406cc50 legacy=0x7f5e1406d020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-05T19:28:00.289 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-05T19:28:00.279+0000 7f5e13fff700 1 -- 172.21.15.118:0/3178472746 --> v2:172.21.15.118:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5e1406a670 con 0x7f5e1406d730 2023-10-05T19:28:00.289 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-05T19:28:00.279+0000 7f5e13fff700 1 --2- 172.21.15.118:0/3178472746 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f5e1406d730 0x7f5e1412e690 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7f5e04009860 tx=0x7f5e040091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4b465ffcc577484c server_cookie=0 in_seq=0 out_seq=0 2023-10-05T19:28:00.289 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-05T19:28:00.279+0000 7f5e137fe700 1 -- 172.21.15.118:0/3178472746 <== mon.0 v2:172.21.15.118:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f5e0400b680 con 0x7f5e1406d730 2023-10-05T19:28:00.289 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-05T19:28:00.279+0000 7f5e137fe700 1 -- 172.21.15.118:0/3178472746 <== mon.0 v2:172.21.15.118:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5e0400b7e0 con 0x7f5e1406d730 2023-10-05T19:28:00.289 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-05T19:28:00.279+0000 7f5e137fe700 1 -- 172.21.15.118:0/3178472746 <== mon.0 v2:172.21.15.118:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f5e0400b990 con 0x7f5e1406d730 2023-10-05T19:28:00.290 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-05T19:28:00.279+0000 7f5e19d73700 1 -- 172.21.15.118:0/3178472746 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f5e1406d730 msgr2=0x7f5e1412e690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-05T19:28:00.290 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-05T19:28:00.279+0000 7f5e19d73700 1 --2- 172.21.15.118:0/3178472746 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f5e1406d730 0x7f5e1412e690 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7f5e04009860 tx=0x7f5e040091a0 comp rx=0 tx=0).stop 2023-10-05T19:28:00.290 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-05T19:28:00.279+0000 7f5e19d73700 1 -- 172.21.15.118:0/3178472746 shutdown_connections 2023-10-05T19:28:00.290 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-05T19:28:00.279+0000 7f5e19d73700 1 --2- 172.21.15.118:0/3178472746 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f5e1406d730 0x7f5e1412e690 unknown :-1 s=CLOSED pgs=99 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-05T19:28:00.290 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-05T19:28:00.279+0000 7f5e19d73700 1 -- 172.21.15.118:0/3178472746 >> 172.21.15.118:0/3178472746 conn(0x7f5e1409a530 msgr2=0x7f5e1409a930 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-05T19:28:00.291 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-05T19:28:00.279+0000 7f5e19d73700 1 -- 172.21.15.118:0/3178472746 shutdown_connections 2023-10-05T19:28:00.291 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-05T19:28:00.279+0000 7f5e19d73700 1 -- 172.21.15.118:0/3178472746 wait complete. 2023-10-05T19:28:00.293 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-05T19:28:00.283+0000 7f5e19d73700 1 Processor -- start 2023-10-05T19:28:00.293 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-05T19:28:00.283+0000 7f5e19d73700 1 -- start start 2023-10-05T19:28:00.293 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-05T19:28:00.283+0000 7f5e19d73700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f5e1406cc50 0x7f5e14138770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-05T19:28:00.293 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-05T19:28:00.283+0000 7f5e19d73700 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5e140954a0 con 0x7f5e1406cc50 2023-10-05T19:28:00.293 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-05T19:28:00.283+0000 7f5e18d71700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f5e1406cc50 0x7f5e14138770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-05T19:28:00.294 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-05T19:28:00.283+0000 7f5e18d71700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f5e1406cc50 0x7f5e14138770 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.118:3300/0 says I am v2:172.21.15.118:55922/0 (socket says 172.21.15.118:55922) 2023-10-05T19:28:00.294 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-05T19:28:00.283+0000 7f5e18d71700 1 -- 172.21.15.118:0/2611021921 learned_addr learned my addr 172.21.15.118:0/2611021921 (peer_addr_for_me v2:172.21.15.118:0/0) 2023-10-05T19:28:00.294 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-05T19:28:00.283+0000 7f5e18d71700 1 -- 172.21.15.118:0/2611021921 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5e04009420 con 0x7f5e1406cc50 2023-10-05T19:28:00.294 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-05T19:28:00.283+0000 7f5e18d71700 1 --2- 172.21.15.118:0/2611021921 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f5e1406cc50 0x7f5e14138770 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7f5e0c00cc30 tx=0x7f5e0c001ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-05T19:28:00.294 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-05T19:28:00.283+0000 7f5e10ff9700 1 -- 172.21.15.118:0/2611021921 <== mon.0 v2:172.21.15.118:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f5e0c004c10 con 0x7f5e1406cc50 2023-10-05T19:28:00.294 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-05T19:28:00.283+0000 7f5e10ff9700 1 -- 172.21.15.118:0/2611021921 <== mon.0 v2:172.21.15.118:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5e0c004d70 con 0x7f5e1406cc50 2023-10-05T19:28:00.294 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-05T19:28:00.283+0000 7f5e19d73700 1 -- 172.21.15.118:0/2611021921 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5e14120160 con 0x7f5e1406cc50 2023-10-05T19:28:00.294 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-05T19:28:00.283+0000 7f5e10ff9700 1 -- 172.21.15.118:0/2611021921 <== mon.0 v2:172.21.15.118:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f5e0c00aaa0 con 0x7f5e1406cc50 2023-10-05T19:28:00.294 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-05T19:28:00.283+0000 7f5e19d73700 1 -- 172.21.15.118:0/2611021921 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5e1411e6b0 con 0x7f5e1406cc50 2023-10-05T19:28:00.294 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-05T19:28:00.283+0000 7f5e10ff9700 1 -- 172.21.15.118:0/2611021921 <== mon.0 v2:172.21.15.118:3300/0 4 ==== mgrmap(e 5) v1 ==== 40329+0+0 (secure 0 0 0) 0x7f5e0c00ac00 con 0x7f5e1406cc50 2023-10-05T19:28:00.294 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-05T19:28:00.283+0000 7f5e10ff9700 1 --2- 172.21.15.118:0/2611021921 >> [v2:172.21.15.167:6824/29889,v1:172.21.15.167:6825/29889] conn(0x7f5e00032430 0x7f5e000348e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-05T19:28:00.294 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-05T19:28:00.283+0000 7f5e10ff9700 1 -- 172.21.15.118:0/2611021921 <== mon.0 v2:172.21.15.118:3300/0 5 ==== osd_map(15..15 src has 1..15) v4 ==== 3478+0+0 (secure 0 0 0) 0x7f5e0c043760 con 0x7f5e1406cc50 2023-10-05T19:28:00.295 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-05T19:28:00.287+0000 7f5e13fff700 1 --2- 172.21.15.118:0/2611021921 >> [v2:172.21.15.167:6824/29889,v1:172.21.15.167:6825/29889] conn(0x7f5e00032430 0x7f5e000348e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-05T19:28:00.295 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-05T19:28:00.287+0000 7f5e19d73700 1 -- 172.21.15.118:0/2611021921 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5e1405ecb0 con 0x7f5e1406cc50 2023-10-05T19:28:00.295 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-05T19:28:00.287+0000 7f5e13fff700 1 --2- 172.21.15.118:0/2611021921 >> [v2:172.21.15.167:6824/29889,v1:172.21.15.167:6825/29889] conn(0x7f5e00032430 0x7f5e000348e0 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f5e040091e0 tx=0x7f5e04016000 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-05T19:28:00.301 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-05T19:28:00.291+0000 7f5e10ff9700 1 -- 172.21.15.118:0/2611021921 <== mon.0 v2:172.21.15.118:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166807 (secure 0 0 0) 0x7f5e0c018070 con 0x7f5e1406cc50 2023-10-05T19:28:00.322 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-05T19:28:00.315+0000 7f7ffb3a8700 1 Processor -- start 2023-10-05T19:28:00.322 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-05T19:28:00.315+0000 7f7ffb3a8700 1 -- start start 2023-10-05T19:28:00.323 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-05T19:28:00.315+0000 7f7ffb3a8700 1 --2- >> v2:172.21.15.118:3300/0 conn(0x7f7ff4126350 0x7f7ff4126720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-05T19:28:00.323 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-05T19:28:00.315+0000 7f7ffb3a8700 1 -- --> v1:172.21.15.118:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7ff406c170 con 0x7f7ff4126c60 2023-10-05T19:28:00.323 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-05T19:28:00.315+0000 7f7ffb3a8700 1 -- --> v2:172.21.15.118:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7ff406e0e0 con 0x7f7ff4126350 2023-10-05T19:28:00.323 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-05T19:28:00.315+0000 7f7ff9144700 1 --2- >> v2:172.21.15.118:3300/0 conn(0x7f7ff4126350 0x7f7ff4126720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-05T19:28:00.323 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-05T19:28:00.315+0000 7f7ff8943700 1 --1- >> v1:172.21.15.118:6789/0 conn(0x7f7ff4126c60 0x7f7ff406a660 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.118:6789/0 says I am v1:172.21.15.118:52162/0 (socket says 172.21.15.118:52162) 2023-10-05T19:28:00.323 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-05T19:28:00.315+0000 7f7ff9144700 1 --2- >> v2:172.21.15.118:3300/0 conn(0x7f7ff4126350 0x7f7ff4126720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.118:3300/0 says I am v2:172.21.15.118:55936/0 (socket says 172.21.15.118:55936) 2023-10-05T19:28:00.323 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-05T19:28:00.315+0000 7f7ff8943700 1 -- 172.21.15.118:0/3621364368 learned_addr learned my addr 172.21.15.118:0/3621364368 (peer_addr_for_me v1:172.21.15.118:0/0) 2023-10-05T19:28:00.324 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-05T19:28:00.315+0000 7f7ff9144700 1 -- 172.21.15.118:0/3621364368 >> v1:172.21.15.118:6789/0 conn(0x7f7ff4126c60 legacy=0x7f7ff406a660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-05T19:28:00.324 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-05T19:28:00.315+0000 7f7ff9144700 1 -- 172.21.15.118:0/3621364368 --> v2:172.21.15.118:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7ff4094f30 con 0x7f7ff4126350 2023-10-05T19:28:00.325 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-05T19:28:00.315+0000 7f7ff9144700 1 --2- 172.21.15.118:0/3621364368 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f7ff4126350 0x7f7ff4126720 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7f7fe4009860 tx=0x7f7fe40091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1e69f23a81945542 server_cookie=0 in_seq=0 out_seq=0 2023-10-05T19:28:00.325 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-05T19:28:00.315+0000 7f7febfff700 1 -- 172.21.15.118:0/3621364368 <== mon.0 v2:172.21.15.118:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f7fe400b680 con 0x7f7ff4126350 2023-10-05T19:28:00.325 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-05T19:28:00.315+0000 7f7febfff700 1 -- 172.21.15.118:0/3621364368 <== mon.0 v2:172.21.15.118:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7fe400b7e0 con 0x7f7ff4126350 2023-10-05T19:28:00.325 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-05T19:28:00.315+0000 7f7febfff700 1 -- 172.21.15.118:0/3621364368 <== mon.0 v2:172.21.15.118:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f7fe400b990 con 0x7f7ff4126350 2023-10-05T19:28:00.325 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-05T19:28:00.319+0000 7f7ffb3a8700 1 -- 172.21.15.118:0/3621364368 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f7ff4126350 msgr2=0x7f7ff4126720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-05T19:28:00.325 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-05T19:28:00.319+0000 7f7ffb3a8700 1 --2- 172.21.15.118:0/3621364368 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f7ff4126350 0x7f7ff4126720 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7f7fe4009860 tx=0x7f7fe40091a0 comp rx=0 tx=0).stop 2023-10-05T19:28:00.325 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-05T19:28:00.319+0000 7f7ffb3a8700 1 -- 172.21.15.118:0/3621364368 shutdown_connections 2023-10-05T19:28:00.325 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-05T19:28:00.319+0000 7f7ffb3a8700 1 --2- 172.21.15.118:0/3621364368 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f7ff4126350 0x7f7ff4126720 unknown :-1 s=CLOSED pgs=102 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-05T19:28:00.325 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-05T19:28:00.319+0000 7f7ffb3a8700 1 -- 172.21.15.118:0/3621364368 >> 172.21.15.118:0/3621364368 conn(0x7f7ff40997a0 msgr2=0x7f7ff4099ba0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-05T19:28:00.326 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-05T19:28:00.319+0000 7f7ffb3a8700 1 -- 172.21.15.118:0/3621364368 shutdown_connections 2023-10-05T19:28:00.326 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-05T19:28:00.319+0000 7f7ffb3a8700 1 -- 172.21.15.118:0/3621364368 wait complete. 2023-10-05T19:28:00.327 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-05T19:28:00.319+0000 7f7ffb3a8700 1 Processor -- start 2023-10-05T19:28:00.327 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-05T19:28:00.319+0000 7f7ffb3a8700 1 -- start start 2023-10-05T19:28:00.327 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-05T19:28:00.319+0000 7f7ffb3a8700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f7ff4126350 0x7f7ff4116d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-05T19:28:00.328 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-05T19:28:00.319+0000 7f7ffb3a8700 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7ff4117da0 con 0x7f7ff4126350 2023-10-05T19:28:00.328 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-05T19:28:00.319+0000 7f7ff9144700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f7ff4126350 0x7f7ff4116d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-05T19:28:00.328 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-05T19:28:00.319+0000 7f7ff9144700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f7ff4126350 0x7f7ff4116d50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.118:3300/0 says I am v2:172.21.15.118:55940/0 (socket says 172.21.15.118:55940) 2023-10-05T19:28:00.328 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-05T19:28:00.319+0000 7f7ff9144700 1 -- 172.21.15.118:0/2245906245 learned_addr learned my addr 172.21.15.118:0/2245906245 (peer_addr_for_me v2:172.21.15.118:0/0) 2023-10-05T19:28:00.328 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-05T19:28:00.319+0000 7f7ff9144700 1 -- 172.21.15.118:0/2245906245 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7fe4009420 con 0x7f7ff4126350 2023-10-05T19:28:00.328 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-05T19:28:00.319+0000 7f7ff9144700 1 --2- 172.21.15.118:0/2245906245 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f7ff4126350 0x7f7ff4116d50 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7f7fe40091e0 tx=0x7f7fe4016000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-05T19:28:00.328 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-05T19:28:00.319+0000 7f7fe97fa700 1 -- 172.21.15.118:0/2245906245 <== mon.0 v2:172.21.15.118:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f7fe4011cf0 con 0x7f7ff4126350 2023-10-05T19:28:00.328 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-05T19:28:00.319+0000 7f7fe97fa700 1 -- 172.21.15.118:0/2245906245 <== mon.0 v2:172.21.15.118:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7fe401f070 con 0x7f7ff4126350 2023-10-05T19:28:00.328 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-05T19:28:00.319+0000 7f7ffb3a8700 1 -- 172.21.15.118:0/2245906245 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7ff4069460 con 0x7f7ff4126350 2023-10-05T19:28:00.328 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-05T19:28:00.319+0000 7f7fe97fa700 1 -- 172.21.15.118:0/2245906245 <== mon.0 v2:172.21.15.118:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f7fe401a440 con 0x7f7ff4126350 2023-10-05T19:28:00.328 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-05T19:28:00.319+0000 7f7ffb3a8700 1 -- 172.21.15.118:0/2245906245 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7ff4118540 con 0x7f7ff4126350 2023-10-05T19:28:00.329 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-05T19:28:00.319+0000 7f7ffb3a8700 1 -- 172.21.15.118:0/2245906245 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7ff405ecb0 con 0x7f7ff4126350 2023-10-05T19:28:00.329 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-05T19:28:00.319+0000 7f7fe97fa700 1 -- 172.21.15.118:0/2245906245 <== mon.0 v2:172.21.15.118:3300/0 4 ==== mgrmap(e 5) v1 ==== 40329+0+0 (secure 0 0 0) 0x7f7fe4021030 con 0x7f7ff4126350 2023-10-05T19:28:00.329 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-05T19:28:00.323+0000 7f7fe97fa700 1 --2- 172.21.15.118:0/2245906245 >> [v2:172.21.15.167:6824/29889,v1:172.21.15.167:6825/29889] conn(0x7f7fd80323e0 0x7f7fd8034890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-05T19:28:00.330 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-05T19:28:00.323+0000 7f7fe97fa700 1 -- 172.21.15.118:0/2245906245 <== mon.0 v2:172.21.15.118:3300/0 5 ==== osd_map(15..15 src has 1..15) v4 ==== 3478+0+0 (secure 0 0 0) 0x7f7fe40455e0 con 0x7f7ff4126350 2023-10-05T19:28:00.330 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-05T19:28:00.323+0000 7f7ff8943700 1 --2- 172.21.15.118:0/2245906245 >> [v2:172.21.15.167:6824/29889,v1:172.21.15.167:6825/29889] conn(0x7f7fd80323e0 0x7f7fd8034890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-05T19:28:00.330 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-05T19:28:00.323+0000 7f7ff8943700 1 --2- 172.21.15.118:0/2245906245 >> [v2:172.21.15.167:6824/29889,v1:172.21.15.167:6825/29889] conn(0x7f7fd80323e0 0x7f7fd8034890 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f7ff0007fd0 tx=0x7f7ff0007b00 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-05T19:28:00.334 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-05T19:28:00.327+0000 7f7fe97fa700 1 -- 172.21.15.118:0/2245906245 <== mon.0 v2:172.21.15.118:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166807 (secure 0 0 0) 0x7f7fe402c980 con 0x7f7ff4126350 2023-10-05T19:28:00.424 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-05T19:28:00.415+0000 7f65f89dc700 1 -- 172.21.15.118:0/2675926607 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 3} v 0) v1 -- 0x7f65f408f090 con 0x7f65f40905f0 2023-10-05T19:28:00.424 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-05T19:28:00.419+0000 7f65eb7fe700 1 -- 172.21.15.118:0/2675926607 <== mon.0 v2:172.21.15.118:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 3}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f65dc0193f0 con 0x7f65f40905f0 2023-10-05T19:28:00.424 INFO:teuthology.orchestra.run.smithi118.stdout:38654705668 2023-10-05T19:28:00.427 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-05T19:28:00.419+0000 7f65f89dc700 1 -- 172.21.15.118:0/2675926607 >> [v2:172.21.15.167:6824/29889,v1:172.21.15.167:6825/29889] conn(0x7f65d40323e0 msgr2=0x7f65d4034890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-05T19:28:00.427 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-05T19:28:00.419+0000 7f65f89dc700 1 --2- 172.21.15.118:0/2675926607 >> [v2:172.21.15.167:6824/29889,v1:172.21.15.167:6825/29889] conn(0x7f65d40323e0 0x7f65d4034890 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f65e4000c00 tx=0x7f65e4007810 comp rx=0 tx=0).stop 2023-10-05T19:28:00.428 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-05T19:28:00.419+0000 7f65f89dc700 1 -- 172.21.15.118:0/2675926607 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f65f40905f0 msgr2=0x7f65f408bf20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-05T19:28:00.428 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-05T19:28:00.419+0000 7f65f89dc700 1 --2- 172.21.15.118:0/2675926607 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f65f40905f0 0x7f65f408bf20 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7f65dc000c00 tx=0x7f65dc016000 comp rx=0 tx=0).stop 2023-10-05T19:28:00.428 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-05T19:28:00.419+0000 7f65f89dc700 1 -- 172.21.15.118:0/2675926607 shutdown_connections 2023-10-05T19:28:00.428 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-05T19:28:00.419+0000 7f65f89dc700 1 --2- 172.21.15.118:0/2675926607 >> [v2:172.21.15.167:6824/29889,v1:172.21.15.167:6825/29889] conn(0x7f65d40323e0 0x7f65d4034890 unknown :-1 s=CLOSED pgs=38 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-05T19:28:00.428 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-05T19:28:00.419+0000 7f65f89dc700 1 --2- 172.21.15.118:0/2675926607 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f65f40905f0 0x7f65f408bf20 unknown :-1 s=CLOSED pgs=94 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-05T19:28:00.429 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-05T19:28:00.419+0000 7f65f89dc700 1 -- 172.21.15.118:0/2675926607 >> 172.21.15.118:0/2675926607 conn(0x7f65f409c5d0 msgr2=0x7f65f409a4d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-05T19:28:00.429 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-05T19:28:00.419+0000 7f65f89dc700 1 -- 172.21.15.118:0/2675926607 shutdown_connections 2023-10-05T19:28:00.429 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-05T19:28:00.419+0000 7f65f89dc700 1 -- 172.21.15.118:0/2675926607 wait complete. 2023-10-05T19:28:00.437 INFO:tasks.ceph.ceph_manager.ceph:need seq 38654705668 got 38654705668 for osd.3 2023-10-05T19:28:00.437 DEBUG:teuthology.parallel:result is None 2023-10-05T19:28:00.440 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-05T19:28:00.431+0000 7fbd3084a700 1 -- 172.21.15.118:0/1105091630 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 2} v 0) v1 -- 0x7fbd2c05ecb0 con 0x7fbd2c06b280 2023-10-05T19:28:00.440 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-05T19:28:00.435+0000 7fbd12ffd700 1 -- 172.21.15.118:0/1105091630 <== mon.0 v2:172.21.15.118:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 2}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7fbd2000ee80 con 0x7fbd2c06b280 2023-10-05T19:28:00.440 INFO:teuthology.orchestra.run.smithi118.stdout:38654705670 2023-10-05T19:28:00.442 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-05T19:28:00.435+0000 7fbd3084a700 1 -- 172.21.15.118:0/1105091630 >> [v2:172.21.15.167:6824/29889,v1:172.21.15.167:6825/29889] conn(0x7fbd180323e0 msgr2=0x7fbd18034890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-05T19:28:00.443 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-05T19:28:00.435+0000 7fbd3084a700 1 --2- 172.21.15.118:0/1105091630 >> [v2:172.21.15.167:6824/29889,v1:172.21.15.167:6825/29889] conn(0x7fbd180323e0 0x7fbd18034890 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7fbd1c0011e0 tx=0x7fbd1c0079d0 comp rx=0 tx=0).stop 2023-10-05T19:28:00.443 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-05T19:28:00.435+0000 7fbd3084a700 1 -- 172.21.15.118:0/1105091630 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fbd2c06b280 msgr2=0x7fbd2c087b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-05T19:28:00.443 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-05T19:28:00.435+0000 7fbd3084a700 1 --2- 172.21.15.118:0/1105091630 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fbd2c06b280 0x7fbd2c087b70 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7fbd200091e0 tx=0x7fbd20018000 comp rx=0 tx=0).stop 2023-10-05T19:28:00.443 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-05T19:28:00.435+0000 7fbd3084a700 1 -- 172.21.15.118:0/1105091630 shutdown_connections 2023-10-05T19:28:00.443 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-05T19:28:00.435+0000 7fbd3084a700 1 --2- 172.21.15.118:0/1105091630 >> [v2:172.21.15.167:6824/29889,v1:172.21.15.167:6825/29889] conn(0x7fbd180323e0 0x7fbd18034890 unknown :-1 s=CLOSED pgs=39 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-05T19:28:00.443 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-05T19:28:00.435+0000 7fbd3084a700 1 --2- 172.21.15.118:0/1105091630 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fbd2c06b280 0x7fbd2c087b70 unknown :-1 s=CLOSED pgs=97 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-05T19:28:00.443 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-05T19:28:00.435+0000 7fbd3084a700 1 -- 172.21.15.118:0/1105091630 >> 172.21.15.118:0/1105091630 conn(0x7fbd2c095970 msgr2=0x7fbd2c097170 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-05T19:28:00.443 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-05T19:28:00.435+0000 7fbd3084a700 1 -- 172.21.15.118:0/1105091630 shutdown_connections 2023-10-05T19:28:00.444 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-05T19:28:00.435+0000 7fbd3084a700 1 -- 172.21.15.118:0/1105091630 wait complete. 2023-10-05T19:28:00.452 INFO:tasks.ceph.ceph_manager.ceph:need seq 38654705670 got 38654705670 for osd.2 2023-10-05T19:28:00.453 DEBUG:teuthology.parallel:result is None 2023-10-05T19:28:00.467 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-05T19:28:00.459+0000 7f5e19d73700 1 -- 172.21.15.118:0/2611021921 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 1} v 0) v1 -- 0x7f5e1406ab90 con 0x7f5e1406cc50 2023-10-05T19:28:00.468 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-05T19:28:00.459+0000 7f5e10ff9700 1 -- 172.21.15.118:0/2611021921 <== mon.0 v2:172.21.15.118:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 1}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f5e0c01b7d0 con 0x7f5e1406cc50 2023-10-05T19:28:00.468 INFO:teuthology.orchestra.run.smithi118.stdout:38654705669 2023-10-05T19:28:00.470 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-05T19:28:00.463+0000 7f5e19d73700 1 -- 172.21.15.118:0/2611021921 >> [v2:172.21.15.167:6824/29889,v1:172.21.15.167:6825/29889] conn(0x7f5e00032430 msgr2=0x7f5e000348e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-05T19:28:00.470 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-05T19:28:00.463+0000 7f5e19d73700 1 --2- 172.21.15.118:0/2611021921 >> [v2:172.21.15.167:6824/29889,v1:172.21.15.167:6825/29889] conn(0x7f5e00032430 0x7f5e000348e0 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f5e040091e0 tx=0x7f5e04016000 comp rx=0 tx=0).stop 2023-10-05T19:28:00.470 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-05T19:28:00.463+0000 7f5e19d73700 1 -- 172.21.15.118:0/2611021921 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f5e1406cc50 msgr2=0x7f5e14138770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-05T19:28:00.470 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-05T19:28:00.463+0000 7f5e19d73700 1 --2- 172.21.15.118:0/2611021921 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f5e1406cc50 0x7f5e14138770 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7f5e0c00cc30 tx=0x7f5e0c001ad0 comp rx=0 tx=0).stop 2023-10-05T19:28:00.470 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-05T19:28:00.463+0000 7f5e19d73700 1 -- 172.21.15.118:0/2611021921 shutdown_connections 2023-10-05T19:28:00.470 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-05T19:28:00.463+0000 7f5e19d73700 1 --2- 172.21.15.118:0/2611021921 >> [v2:172.21.15.167:6824/29889,v1:172.21.15.167:6825/29889] conn(0x7f5e00032430 0x7f5e000348e0 unknown :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-05T19:28:00.471 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-05T19:28:00.463+0000 7f5e19d73700 1 --2- 172.21.15.118:0/2611021921 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f5e1406cc50 0x7f5e14138770 unknown :-1 s=CLOSED pgs=100 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-05T19:28:00.471 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-05T19:28:00.463+0000 7f5e19d73700 1 -- 172.21.15.118:0/2611021921 >> 172.21.15.118:0/2611021921 conn(0x7f5e1409a530 msgr2=0x7f5e14096500 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-05T19:28:00.471 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-05T19:28:00.463+0000 7f5e19d73700 1 -- 172.21.15.118:0/2611021921 shutdown_connections 2023-10-05T19:28:00.471 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-05T19:28:00.463+0000 7f5e19d73700 1 -- 172.21.15.118:0/2611021921 wait complete. 2023-10-05T19:28:00.481 INFO:tasks.ceph.ceph_manager.ceph:need seq 38654705669 got 38654705669 for osd.1 2023-10-05T19:28:00.481 DEBUG:teuthology.parallel:result is None 2023-10-05T19:28:00.492 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-05T19:28:00.487+0000 7f7ffb3a8700 1 -- 172.21.15.118:0/2245906245 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 0} v 0) v1 -- 0x7f7ff406c8e0 con 0x7f7ff4126350 2023-10-05T19:28:00.493 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-05T19:28:00.487+0000 7f7fe97fa700 1 -- 172.21.15.118:0/2245906245 <== mon.0 v2:172.21.15.118:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 0}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f7fe40193c0 con 0x7f7ff4126350 2023-10-05T19:28:00.493 INFO:teuthology.orchestra.run.smithi118.stdout:38654705668 2023-10-05T19:28:00.496 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-05T19:28:00.487+0000 7f7ffb3a8700 1 -- 172.21.15.118:0/2245906245 >> [v2:172.21.15.167:6824/29889,v1:172.21.15.167:6825/29889] conn(0x7f7fd80323e0 msgr2=0x7f7fd8034890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-05T19:28:00.496 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-05T19:28:00.487+0000 7f7ffb3a8700 1 --2- 172.21.15.118:0/2245906245 >> [v2:172.21.15.167:6824/29889,v1:172.21.15.167:6825/29889] conn(0x7f7fd80323e0 0x7f7fd8034890 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f7ff0007fd0 tx=0x7f7ff0007b00 comp rx=0 tx=0).stop 2023-10-05T19:28:00.496 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-05T19:28:00.487+0000 7f7ffb3a8700 1 -- 172.21.15.118:0/2245906245 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f7ff4126350 msgr2=0x7f7ff4116d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-05T19:28:00.496 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-05T19:28:00.487+0000 7f7ffb3a8700 1 --2- 172.21.15.118:0/2245906245 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f7ff4126350 0x7f7ff4116d50 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7f7fe40091e0 tx=0x7f7fe4016000 comp rx=0 tx=0).stop 2023-10-05T19:28:00.496 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-05T19:28:00.487+0000 7f7ffb3a8700 1 -- 172.21.15.118:0/2245906245 shutdown_connections 2023-10-05T19:28:00.496 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-05T19:28:00.487+0000 7f7ffb3a8700 1 --2- 172.21.15.118:0/2245906245 >> [v2:172.21.15.167:6824/29889,v1:172.21.15.167:6825/29889] conn(0x7f7fd80323e0 0x7f7fd8034890 unknown :-1 s=CLOSED pgs=41 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-05T19:28:00.497 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-05T19:28:00.487+0000 7f7ffb3a8700 1 --2- 172.21.15.118:0/2245906245 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f7ff4126350 0x7f7ff4116d50 unknown :-1 s=CLOSED pgs=103 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-05T19:28:00.497 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-05T19:28:00.487+0000 7f7ffb3a8700 1 -- 172.21.15.118:0/2245906245 >> 172.21.15.118:0/2245906245 conn(0x7f7ff40997a0 msgr2=0x7f7ff410fc60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-05T19:28:00.497 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-05T19:28:00.487+0000 7f7ffb3a8700 1 -- 172.21.15.118:0/2245906245 shutdown_connections 2023-10-05T19:28:00.497 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-05T19:28:00.487+0000 7f7ffb3a8700 1 -- 172.21.15.118:0/2245906245 wait complete. 2023-10-05T19:28:00.507 INFO:tasks.ceph.ceph_manager.ceph:need seq 38654705668 got 38654705668 for osd.0 2023-10-05T19:28:00.507 DEBUG:teuthology.parallel:result is None 2023-10-05T19:28:00.507 INFO:tasks.ceph.ceph_manager.ceph:waiting for clean 2023-10-05T19:28:00.507 DEBUG:teuthology.orchestra.run.smithi118:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-05T19:28:00.608 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-05T19:28:00.599+0000 7fefc9af7700 1 Processor -- start 2023-10-05T19:28:00.608 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-05T19:28:00.599+0000 7fefc9af7700 1 -- start start 2023-10-05T19:28:00.608 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-05T19:28:00.599+0000 7fefc9af7700 1 --2- >> v2:172.21.15.118:3300/0 conn(0x7fefc4093bb0 0x7fefc412db70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-05T19:28:00.608 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-05T19:28:00.599+0000 7fefc9af7700 1 -- --> v1:172.21.15.118:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fefc4091240 con 0x7fefc412e0b0 2023-10-05T19:28:00.608 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-05T19:28:00.599+0000 7fefc9af7700 1 -- --> v2:172.21.15.118:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fefc4091800 con 0x7fefc4093bb0 2023-10-05T19:28:00.608 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-05T19:28:00.599+0000 7fefbbfff700 1 --1- >> v1:172.21.15.118:6789/0 conn(0x7fefc412e0b0 0x7fefc408f690 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.118:6789/0 says I am v1:172.21.15.118:52170/0 (socket says 172.21.15.118:52170) 2023-10-05T19:28:00.609 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-05T19:28:00.599+0000 7fefbbfff700 1 -- 172.21.15.118:0/2646618262 learned_addr learned my addr 172.21.15.118:0/2646618262 (peer_addr_for_me v1:172.21.15.118:0/0) 2023-10-05T19:28:00.609 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-05T19:28:00.599+0000 7fefc8af5700 1 --2- 172.21.15.118:0/2646618262 >> v2:172.21.15.118:3300/0 conn(0x7fefc4093bb0 0x7fefc412db70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-05T19:28:00.610 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-05T19:28:00.603+0000 7fefbb7fe700 1 -- 172.21.15.118:0/2646618262 <== mon.0 v1:172.21.15.118:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1197697160 0 0) 0x7fefc4091240 con 0x7fefc412e0b0 2023-10-05T19:28:00.610 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-05T19:28:00.603+0000 7fefbb7fe700 1 -- 172.21.15.118:0/2646618262 --> v1:172.21.15.118:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fefac003570 con 0x7fefc412e0b0 2023-10-05T19:28:00.611 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-05T19:28:00.603+0000 7fefbb7fe700 1 -- 172.21.15.118:0/2646618262 <== mon.0 v1:172.21.15.118:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1086560430 0 0) 0x7fefac003570 con 0x7fefc412e0b0 2023-10-05T19:28:00.611 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-05T19:28:00.603+0000 7fefc8af5700 1 -- 172.21.15.118:0/2646618262 >> v1:172.21.15.118:6789/0 conn(0x7fefc412e0b0 legacy=0x7fefc408f690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-05T19:28:00.611 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-05T19:28:00.603+0000 7fefc8af5700 1 -- 172.21.15.118:0/2646618262 --> v2:172.21.15.118:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fefc40947e0 con 0x7fefc4093bb0 2023-10-05T19:28:00.611 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-05T19:28:00.603+0000 7fefc8af5700 1 --2- 172.21.15.118:0/2646618262 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fefc4093bb0 0x7fefc412db70 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7fefc0009860 tx=0x7fefc00091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ef0b56cbe6b5424e server_cookie=0 in_seq=0 out_seq=0 2023-10-05T19:28:00.611 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-05T19:28:00.603+0000 7fefbb7fe700 1 -- 172.21.15.118:0/2646618262 <== mon.0 v2:172.21.15.118:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fefb40015b0 con 0x7fefc4093bb0 2023-10-05T19:28:00.611 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-05T19:28:00.603+0000 7fefbb7fe700 1 -- 172.21.15.118:0/2646618262 <== mon.0 v2:172.21.15.118:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fefc0011630 con 0x7fefc4093bb0 2023-10-05T19:28:00.611 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-05T19:28:00.603+0000 7fefbb7fe700 1 -- 172.21.15.118:0/2646618262 <== mon.0 v2:172.21.15.118:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fefc00117c0 con 0x7fefc4093bb0 2023-10-05T19:28:00.611 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-05T19:28:00.603+0000 7fefc9af7700 1 -- 172.21.15.118:0/2646618262 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fefc4093bb0 msgr2=0x7fefc412db70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-05T19:28:00.611 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-05T19:28:00.603+0000 7fefc9af7700 1 --2- 172.21.15.118:0/2646618262 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fefc4093bb0 0x7fefc412db70 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7fefc0009860 tx=0x7fefc00091a0 comp rx=0 tx=0).stop 2023-10-05T19:28:00.612 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-05T19:28:00.603+0000 7fefc9af7700 1 -- 172.21.15.118:0/2646618262 shutdown_connections 2023-10-05T19:28:00.612 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-05T19:28:00.603+0000 7fefc9af7700 1 --2- 172.21.15.118:0/2646618262 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fefc4093bb0 0x7fefc412db70 unknown :-1 s=CLOSED pgs=105 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-05T19:28:00.612 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-05T19:28:00.603+0000 7fefc9af7700 1 -- 172.21.15.118:0/2646618262 >> 172.21.15.118:0/2646618262 conn(0x7fefc40981b0 msgr2=0x7fefc40985b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-05T19:28:00.612 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-05T19:28:00.603+0000 7fefc9af7700 1 -- 172.21.15.118:0/2646618262 shutdown_connections 2023-10-05T19:28:00.612 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-05T19:28:00.603+0000 7fefc9af7700 1 -- 172.21.15.118:0/2646618262 wait complete. 2023-10-05T19:28:00.614 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-05T19:28:00.603+0000 7fefc9af7700 1 Processor -- start 2023-10-05T19:28:00.614 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-05T19:28:00.603+0000 7fefc9af7700 1 -- start start 2023-10-05T19:28:00.614 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-05T19:28:00.603+0000 7fefc9af7700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fefc4093bb0 0x7fefc413c7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-05T19:28:00.614 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-05T19:28:00.603+0000 7fefc9af7700 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fefc406f230 con 0x7fefc4093bb0 2023-10-05T19:28:00.614 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-05T19:28:00.603+0000 7fefc8af5700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fefc4093bb0 0x7fefc413c7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-05T19:28:00.614 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-05T19:28:00.603+0000 7fefc8af5700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fefc4093bb0 0x7fefc413c7d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.118:3300/0 says I am v2:172.21.15.118:55970/0 (socket says 172.21.15.118:55970) 2023-10-05T19:28:00.614 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-05T19:28:00.603+0000 7fefc8af5700 1 -- 172.21.15.118:0/636796791 learned_addr learned my addr 172.21.15.118:0/636796791 (peer_addr_for_me v2:172.21.15.118:0/0) 2023-10-05T19:28:00.614 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-05T19:28:00.607+0000 7fefc8af5700 1 -- 172.21.15.118:0/636796791 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fefc0009420 con 0x7fefc4093bb0 2023-10-05T19:28:00.615 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-05T19:28:00.607+0000 7fefc8af5700 1 --2- 172.21.15.118:0/636796791 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fefc4093bb0 0x7fefc413c7d0 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7fefc00030c0 tx=0x7fefc0016000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-05T19:28:00.615 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-05T19:28:00.607+0000 7fefb8ff9700 1 -- 172.21.15.118:0/636796791 <== mon.0 v2:172.21.15.118:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fefc001cdc0 con 0x7fefc4093bb0 2023-10-05T19:28:00.615 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-05T19:28:00.607+0000 7fefb8ff9700 1 -- 172.21.15.118:0/636796791 <== mon.0 v2:172.21.15.118:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fefc001a450 con 0x7fefc4093bb0 2023-10-05T19:28:00.615 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-05T19:28:00.607+0000 7fefc9af7700 1 -- 172.21.15.118:0/636796791 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fefc412d070 con 0x7fefc4093bb0 2023-10-05T19:28:00.615 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-05T19:28:00.607+0000 7fefb8ff9700 1 -- 172.21.15.118:0/636796791 <== mon.0 v2:172.21.15.118:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fefc001a600 con 0x7fefc4093bb0 2023-10-05T19:28:00.615 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-05T19:28:00.607+0000 7fefc9af7700 1 -- 172.21.15.118:0/636796791 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fefc412d310 con 0x7fefc4093bb0 2023-10-05T19:28:00.616 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-05T19:28:00.607+0000 7fefb8ff9700 1 -- 172.21.15.118:0/636796791 <== mon.0 v2:172.21.15.118:3300/0 4 ==== mgrmap(e 5) v1 ==== 40329+0+0 (secure 0 0 0) 0x7fefc001a830 con 0x7fefc4093bb0 2023-10-05T19:28:00.616 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-05T19:28:00.607+0000 7fefc9af7700 1 -- 172.21.15.118:0/636796791 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fefc405ecb0 con 0x7fefc4093bb0 2023-10-05T19:28:00.616 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-05T19:28:00.607+0000 7fefb8ff9700 1 --2- 172.21.15.118:0/636796791 >> [v2:172.21.15.167:6824/29889,v1:172.21.15.167:6825/29889] conn(0x7fefb00323e0 0x7fefb0034890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-05T19:28:00.616 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-05T19:28:00.607+0000 7fefb8ff9700 1 -- 172.21.15.118:0/636796791 <== mon.0 v2:172.21.15.118:3300/0 5 ==== osd_map(15..15 src has 1..15) v4 ==== 3478+0+0 (secure 0 0 0) 0x7fefc0045780 con 0x7fefc4093bb0 2023-10-05T19:28:00.616 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-05T19:28:00.611+0000 7fefbbfff700 1 --2- 172.21.15.118:0/636796791 >> [v2:172.21.15.167:6824/29889,v1:172.21.15.167:6825/29889] conn(0x7fefb00323e0 0x7fefb0034890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-05T19:28:00.617 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-05T19:28:00.611+0000 7fefbbfff700 1 --2- 172.21.15.118:0/636796791 >> [v2:172.21.15.167:6824/29889,v1:172.21.15.167:6825/29889] conn(0x7fefb00323e0 0x7fefb0034890 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7fefb4007a60 tx=0x7fefb40010d0 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-05T19:28:00.621 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-05T19:28:00.615+0000 7fefb8ff9700 1 -- 172.21.15.118:0/636796791 <== mon.0 v2:172.21.15.118:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166807 (secure 0 0 0) 0x7fefc001aae0 con 0x7fefc4093bb0 2023-10-05T19:28:00.776 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-05T19:28:00.771+0000 7fefc9af7700 1 -- 172.21.15.118:0/636796791 --> [v2:172.21.15.167:6824/29889,v1:172.21.15.167:6825/29889] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7fefc404b5c0 con 0x7fefb00323e0 2023-10-05T19:28:00.779 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-05T19:28:00.771+0000 7fefb8ff9700 1 -- 172.21.15.118:0/636796791 <== mgr.4100 v2:172.21.15.167:6824/29889 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+34676 (secure 0 0 0) 0x7fefc404b5c0 con 0x7fefb00323e0 2023-10-05T19:28:00.780 INFO:teuthology.orchestra.run.smithi118.stdout: 2023-10-05T19:28:00.782 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-05T19:28:00.775+0000 7fefc9af7700 1 -- 172.21.15.118:0/636796791 >> [v2:172.21.15.167:6824/29889,v1:172.21.15.167:6825/29889] conn(0x7fefb00323e0 msgr2=0x7fefb0034890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-05T19:28:00.783 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-05T19:28:00.775+0000 7fefc9af7700 1 --2- 172.21.15.118:0/636796791 >> [v2:172.21.15.167:6824/29889,v1:172.21.15.167:6825/29889] conn(0x7fefb00323e0 0x7fefb0034890 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7fefb4007a60 tx=0x7fefb40010d0 comp rx=0 tx=0).stop 2023-10-05T19:28:00.783 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-05T19:28:00.775+0000 7fefc9af7700 1 -- 172.21.15.118:0/636796791 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fefc4093bb0 msgr2=0x7fefc413c7d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-05T19:28:00.783 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-05T19:28:00.775+0000 7fefc9af7700 1 --2- 172.21.15.118:0/636796791 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fefc4093bb0 0x7fefc413c7d0 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7fefc00030c0 tx=0x7fefc0016000 comp rx=0 tx=0).stop 2023-10-05T19:28:00.783 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-05T19:28:00.775+0000 7fefc9af7700 1 -- 172.21.15.118:0/636796791 shutdown_connections 2023-10-05T19:28:00.783 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-05T19:28:00.775+0000 7fefc9af7700 1 --2- 172.21.15.118:0/636796791 >> [v2:172.21.15.167:6824/29889,v1:172.21.15.167:6825/29889] conn(0x7fefb00323e0 0x7fefb0034890 unknown :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-05T19:28:00.783 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-05T19:28:00.775+0000 7fefc9af7700 1 --2- 172.21.15.118:0/636796791 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fefc4093bb0 0x7fefc413c7d0 unknown :-1 s=CLOSED pgs=106 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-05T19:28:00.783 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-05T19:28:00.775+0000 7fefc9af7700 1 -- 172.21.15.118:0/636796791 >> 172.21.15.118:0/636796791 conn(0x7fefc40981b0 msgr2=0x7fefc406bef0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-05T19:28:00.783 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-05T19:28:00.775+0000 7fefc9af7700 1 -- 172.21.15.118:0/636796791 shutdown_connections 2023-10-05T19:28:00.783 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-05T19:28:00.775+0000 7fefc9af7700 1 -- 172.21.15.118:0/636796791 wait complete. 2023-10-05T19:28:00.784 INFO:teuthology.orchestra.run.smithi118.stderr:dumped all 2023-10-05T19:28:00.794 INFO:teuthology.orchestra.run.smithi118.stdout:{"pg_ready":true,"pg_map":{"version":16,"stamp":"2023-10-05T19:27:58.914046+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":14,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":1194888,"kb_used_data":1640,"kb_used_omap":0,"kb_used_meta":1193216,"kb_avail":376292472,"statfs":{"total":386547056640,"available":385323491328,"internally_reserved":0,"allocated":1679360,"data_stored":1327976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":1221853184},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"2.173493"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":8,"reported_epoch":14,"state":"active+clean","last_fresh":"2023-10-05T19:27:58.035963+0000","last_change":"2023-10-05T19:27:58.035963+0000","last_active":"2023-10-05T19:27:58.035963+0000","last_peered":"2023-10-05T19:27:58.035963+0000","last_clean":"2023-10-05T19:27:58.035963+0000","last_became_active":"2023-10-05T19:27:58.034862+0000","last_became_peered":"2023-10-05T19:27:58.034862+0000","last_unstale":"2023-10-05T19:27:58.035963+0000","last_undegraded":"2023-10-05T19:27:58.035963+0000","last_fullsized":"2023-10-05T19:27:58.035963+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-05T19:27:56.732981+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-05T19:27:56.732981+0000","last_clean_scrub_stamp":"2023-10-05T19:27:56.732981+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-06T22:43:15.989376+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":9,"reported_epoch":14,"state":"active+clean","last_fresh":"2023-10-05T19:27:57.763364+0000","last_change":"2023-10-05T19:27:57.762520+0000","last_active":"2023-10-05T19:27:57.763364+0000","last_peered":"2023-10-05T19:27:57.763364+0000","last_clean":"2023-10-05T19:27:57.763364+0000","last_became_active":"2023-10-05T19:27:57.762202+0000","last_became_peered":"2023-10-05T19:27:57.762202+0000","last_unstale":"2023-10-05T19:27:57.763364+0000","last_undegraded":"2023-10-05T19:27:57.763364+0000","last_fullsized":"2023-10-05T19:27:57.763364+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-05T19:27:56.732981+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-05T19:27:56.732981+0000","last_clean_scrub_stamp":"2023-10-05T19:27:56.732981+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-06T22:08:09.710054+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":8,"reported_epoch":14,"state":"active+clean","last_fresh":"2023-10-05T19:27:57.759577+0000","last_change":"2023-10-05T19:27:57.759577+0000","last_active":"2023-10-05T19:27:57.759577+0000","last_peered":"2023-10-05T19:27:57.759577+0000","last_clean":"2023-10-05T19:27:57.759577+0000","last_became_active":"2023-10-05T19:27:57.758981+0000","last_became_peered":"2023-10-05T19:27:57.758981+0000","last_unstale":"2023-10-05T19:27:57.759577+0000","last_undegraded":"2023-10-05T19:27:57.759577+0000","last_fullsized":"2023-10-05T19:27:57.759577+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-05T19:27:56.732981+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-05T19:27:56.732981+0000","last_clean_scrub_stamp":"2023-10-05T19:27:56.732981+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-06T22:41:16.969181+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":9,"reported_epoch":14,"state":"active+clean","last_fresh":"2023-10-05T19:27:57.762620+0000","last_change":"2023-10-05T19:27:57.761710+0000","last_active":"2023-10-05T19:27:57.762620+0000","last_peered":"2023-10-05T19:27:57.762620+0000","last_clean":"2023-10-05T19:27:57.762620+0000","last_became_active":"2023-10-05T19:27:57.761424+0000","last_became_peered":"2023-10-05T19:27:57.761424+0000","last_unstale":"2023-10-05T19:27:57.762620+0000","last_undegraded":"2023-10-05T19:27:57.762620+0000","last_fullsized":"2023-10-05T19:27:57.762620+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-05T19:27:56.732981+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-05T19:27:56.732981+0000","last_clean_scrub_stamp":"2023-10-05T19:27:56.732981+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T02:33:00.026989+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":8,"reported_epoch":14,"state":"active+clean","last_fresh":"2023-10-05T19:27:57.759916+0000","last_change":"2023-10-05T19:27:57.759916+0000","last_active":"2023-10-05T19:27:57.759916+0000","last_peered":"2023-10-05T19:27:57.759916+0000","last_clean":"2023-10-05T19:27:57.759916+0000","last_became_active":"2023-10-05T19:27:57.759323+0000","last_became_peered":"2023-10-05T19:27:57.759323+0000","last_unstale":"2023-10-05T19:27:57.759916+0000","last_undegraded":"2023-10-05T19:27:57.759916+0000","last_fullsized":"2023-10-05T19:27:57.759916+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-05T19:27:56.732981+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-05T19:27:56.732981+0000","last_clean_scrub_stamp":"2023-10-05T19:27:56.732981+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T00:41:14.340083+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":9,"reported_epoch":14,"state":"active+clean","last_fresh":"2023-10-05T19:27:57.763870+0000","last_change":"2023-10-05T19:27:57.760314+0000","last_active":"2023-10-05T19:27:57.763870+0000","last_peered":"2023-10-05T19:27:57.763870+0000","last_clean":"2023-10-05T19:27:57.763870+0000","last_became_active":"2023-10-05T19:27:57.759809+0000","last_became_peered":"2023-10-05T19:27:57.759809+0000","last_unstale":"2023-10-05T19:27:57.763870+0000","last_undegraded":"2023-10-05T19:27:57.763870+0000","last_fullsized":"2023-10-05T19:27:57.763870+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-05T19:27:56.732981+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-05T19:27:56.732981+0000","last_clean_scrub_stamp":"2023-10-05T19:27:56.732981+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T04:11:37.673289+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":8,"reported_epoch":14,"state":"active+clean","last_fresh":"2023-10-05T19:27:57.759697+0000","last_change":"2023-10-05T19:27:57.759697+0000","last_active":"2023-10-05T19:27:57.759697+0000","last_peered":"2023-10-05T19:27:57.759697+0000","last_clean":"2023-10-05T19:27:57.759697+0000","last_became_active":"2023-10-05T19:27:57.759232+0000","last_became_peered":"2023-10-05T19:27:57.759232+0000","last_unstale":"2023-10-05T19:27:57.759697+0000","last_undegraded":"2023-10-05T19:27:57.759697+0000","last_fullsized":"2023-10-05T19:27:57.759697+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-05T19:27:56.732981+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-05T19:27:56.732981+0000","last_clean_scrub_stamp":"2023-10-05T19:27:56.732981+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T03:39:17.485984+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":8,"reported_epoch":14,"state":"active+clean","last_fresh":"2023-10-05T19:27:57.759809+0000","last_change":"2023-10-05T19:27:57.759809+0000","last_active":"2023-10-05T19:27:57.759809+0000","last_peered":"2023-10-05T19:27:57.759809+0000","last_clean":"2023-10-05T19:27:57.759809+0000","last_became_active":"2023-10-05T19:27:57.759347+0000","last_became_peered":"2023-10-05T19:27:57.759347+0000","last_unstale":"2023-10-05T19:27:57.759809+0000","last_undegraded":"2023-10-05T19:27:57.759809+0000","last_fullsized":"2023-10-05T19:27:57.759809+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-05T19:27:56.732981+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-05T19:27:56.732981+0000","last_clean_scrub_stamp":"2023-10-05T19:27:56.732981+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T01:49:38.110242+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'117","reported_seq":161,"reported_epoch":13,"state":"active+clean","last_fresh":"2023-10-05T19:27:56.743894+0000","last_change":"2023-10-05T19:27:52.467966+0000","last_active":"2023-10-05T19:27:56.743894+0000","last_peered":"2023-10-05T19:27:56.743894+0000","last_clean":"2023-10-05T19:27:56.743894+0000","last_became_active":"2023-10-05T19:27:52.465477+0000","last_became_peered":"2023-10-05T19:27:52.465477+0000","last_unstale":"2023-10-05T19:27:56.743894+0000","last_undegraded":"2023-10-05T19:27:56.743894+0000","last_fullsized":"2023-10-05T19:27:56.743894+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-05T19:27:51.198374+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-05T19:27:51.198374+0000","last_clean_scrub_stamp":"2023-10-05T19:27:51.198374+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T03:59:55.896733+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":4},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705668,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297408,"kb_used_data":120,"kb_used_omap":0,"kb_used_meta":297280,"kb_avail":94074432,"statfs":{"total":96636764160,"available":96332218368,"internally_reserved":0,"allocated":122880,"data_stored":36810,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304414720},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.70799999999999996}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.56599999999999995}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.63600000000000001}]}]},{"osd":2,"up_from":9,"seq":38654705670,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297988,"kb_used_data":700,"kb_used_omap":0,"kb_used_meta":297280,"kb_avail":94073852,"statfs":{"total":96636764160,"available":96331624448,"internally_reserved":0,"allocated":716800,"data_stored":627178,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304414720},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.63300000000000001}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.50700000000000001}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.58799999999999997}]}]},{"osd":1,"up_from":9,"seq":38654705669,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297988,"kb_used_data":700,"kb_used_omap":0,"kb_used_meta":297280,"kb_avail":94073852,"statfs":{"total":96636764160,"available":96331624448,"internally_reserved":0,"allocated":716800,"data_stored":627178,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304414720},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.80900000000000005}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.73599999999999999}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.89100000000000001}]}]},{"osd":0,"up_from":9,"seq":38654705668,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":301504,"kb_used_data":120,"kb_used_omap":0,"kb_used_meta":301376,"kb_avail":94070336,"statfs":{"total":96636764160,"available":96328024064,"internally_reserved":0,"allocated":122880,"data_stored":36810,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":308609024},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.77200000000000002}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.64000000000000001}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.59999999999999998}]}]}],"pool_statfs":[{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-10-05T19:28:00.795 DEBUG:teuthology.orchestra.run.smithi118:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-05T19:28:00.897 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-05T19:28:00.891+0000 7fb8b2ec3700 1 Processor -- start 2023-10-05T19:28:00.897 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-05T19:28:00.891+0000 7fb8b2ec3700 1 -- start start 2023-10-05T19:28:00.898 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-05T19:28:00.891+0000 7fb8b2ec3700 1 --2- >> v2:172.21.15.118:3300/0 conn(0x7fb8ac12c7a0 0x7fb8ac12eb80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-05T19:28:00.898 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-05T19:28:00.891+0000 7fb8b2ec3700 1 -- --> v1:172.21.15.118:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb8ac092760 con 0x7fb8ac095ff0 2023-10-05T19:28:00.898 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-05T19:28:00.891+0000 7fb8b2ec3700 1 -- --> v2:172.21.15.118:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb8ac096ad0 con 0x7fb8ac12c7a0 2023-10-05T19:28:00.898 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-05T19:28:00.891+0000 7fb8b1ec1700 1 --1- >> v1:172.21.15.118:6789/0 conn(0x7fb8ac095ff0 0x7fb8ac0963c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.118:6789/0 says I am v1:172.21.15.118:52172/0 (socket says 172.21.15.118:52172) 2023-10-05T19:28:00.898 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-05T19:28:00.891+0000 7fb8b16c0700 1 --2- >> v2:172.21.15.118:3300/0 conn(0x7fb8ac12c7a0 0x7fb8ac12eb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-05T19:28:00.898 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-05T19:28:00.891+0000 7fb8b1ec1700 1 -- 172.21.15.118:0/3275623223 learned_addr learned my addr 172.21.15.118:0/3275623223 (peer_addr_for_me v1:172.21.15.118:0/0) 2023-10-05T19:28:00.899 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-05T19:28:00.891+0000 7fb8b16c0700 1 -- 172.21.15.118:0/3275623223 >> v1:172.21.15.118:6789/0 conn(0x7fb8ac095ff0 legacy=0x7fb8ac0963c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-05T19:28:00.899 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-05T19:28:00.891+0000 7fb8b16c0700 1 -- 172.21.15.118:0/3275623223 --> v2:172.21.15.118:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb8ac0918a0 con 0x7fb8ac12c7a0 2023-10-05T19:28:00.900 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-05T19:28:00.891+0000 7fb8b0ebf700 1 -- 172.21.15.118:0/3275623223 <== mon.0 v1:172.21.15.118:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3601696437 0 0) 0x7fb8ac092760 con 0x7fb8ac095ff0 2023-10-05T19:28:00.900 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-05T19:28:00.891+0000 7fb8b16c0700 1 --2- 172.21.15.118:0/3275623223 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb8ac12c7a0 0x7fb8ac12eb80 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7fb8a0009850 tx=0x7fb8a0009120 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e1ea47e63087345a server_cookie=0 in_seq=0 out_seq=0 2023-10-05T19:28:00.900 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-05T19:28:00.891+0000 7fb8b0ebf700 1 -- 172.21.15.118:0/3275623223 <== mon.0 v2:172.21.15.118:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fb8a0005c10 con 0x7fb8ac12c7a0 2023-10-05T19:28:00.900 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-05T19:28:00.891+0000 7fb8b0ebf700 1 -- 172.21.15.118:0/3275623223 <== mon.0 v2:172.21.15.118:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb8a0005d70 con 0x7fb8ac12c7a0 2023-10-05T19:28:00.900 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-05T19:28:00.891+0000 7fb8b0ebf700 1 -- 172.21.15.118:0/3275623223 <== mon.0 v2:172.21.15.118:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fb8a0006190 con 0x7fb8ac12c7a0 2023-10-05T19:28:00.901 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-05T19:28:00.891+0000 7fb8b2ec3700 1 -- 172.21.15.118:0/3275623223 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb8ac12c7a0 msgr2=0x7fb8ac12eb80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-05T19:28:00.901 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-05T19:28:00.891+0000 7fb8b2ec3700 1 --2- 172.21.15.118:0/3275623223 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb8ac12c7a0 0x7fb8ac12eb80 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7fb8a0009850 tx=0x7fb8a0009120 comp rx=0 tx=0).stop 2023-10-05T19:28:00.901 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-05T19:28:00.891+0000 7fb8b2ec3700 1 -- 172.21.15.118:0/3275623223 shutdown_connections 2023-10-05T19:28:00.901 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-05T19:28:00.891+0000 7fb8b2ec3700 1 --2- 172.21.15.118:0/3275623223 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb8ac12c7a0 0x7fb8ac12eb80 unknown :-1 s=CLOSED pgs=108 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-05T19:28:00.901 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-05T19:28:00.895+0000 7fb8b2ec3700 1 -- 172.21.15.118:0/3275623223 >> 172.21.15.118:0/3275623223 conn(0x7fb8ac09c5d0 msgr2=0x7fb8ac09c9d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-05T19:28:00.901 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-05T19:28:00.895+0000 7fb8b2ec3700 1 -- 172.21.15.118:0/3275623223 shutdown_connections 2023-10-05T19:28:00.901 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-05T19:28:00.895+0000 7fb8b2ec3700 1 -- 172.21.15.118:0/3275623223 wait complete. 2023-10-05T19:28:00.902 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-05T19:28:00.895+0000 7fb8b2ec3700 1 Processor -- start 2023-10-05T19:28:00.902 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-05T19:28:00.895+0000 7fb8b2ec3700 1 -- start start 2023-10-05T19:28:00.903 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-05T19:28:00.895+0000 7fb8b2ec3700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb8ac095ff0 0x7fb8ac12b7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-05T19:28:00.904 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-05T19:28:00.895+0000 7fb8b2ec3700 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb8ac08d2d0 con 0x7fb8ac095ff0 2023-10-05T19:28:00.904 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-05T19:28:00.895+0000 7fb8b1ec1700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb8ac095ff0 0x7fb8ac12b7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-05T19:28:00.904 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-05T19:28:00.895+0000 7fb8b1ec1700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb8ac095ff0 0x7fb8ac12b7a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.118:3300/0 says I am v2:172.21.15.118:55990/0 (socket says 172.21.15.118:55990) 2023-10-05T19:28:00.904 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-05T19:28:00.895+0000 7fb8b1ec1700 1 -- 172.21.15.118:0/3682425553 learned_addr learned my addr 172.21.15.118:0/3682425553 (peer_addr_for_me v2:172.21.15.118:0/0) 2023-10-05T19:28:00.904 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-05T19:28:00.895+0000 7fb8b1ec1700 1 -- 172.21.15.118:0/3682425553 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb8a00093c0 con 0x7fb8ac095ff0 2023-10-05T19:28:00.905 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-05T19:28:00.895+0000 7fb8b1ec1700 1 --2- 172.21.15.118:0/3682425553 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb8ac095ff0 0x7fb8ac12b7a0 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7fb8a800c710 tx=0x7fb8a8007d90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-05T19:28:00.905 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-05T19:28:00.899+0000 7fb89dffb700 1 -- 172.21.15.118:0/3682425553 <== mon.0 v2:172.21.15.118:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fb8a8005c50 con 0x7fb8ac095ff0 2023-10-05T19:28:00.905 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-05T19:28:00.899+0000 7fb89dffb700 1 -- 172.21.15.118:0/3682425553 <== mon.0 v2:172.21.15.118:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb8a8005db0 con 0x7fb8ac095ff0 2023-10-05T19:28:00.905 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-05T19:28:00.899+0000 7fb8b2ec3700 1 -- 172.21.15.118:0/3682425553 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb8ac08f3a0 con 0x7fb8ac095ff0 2023-10-05T19:28:00.905 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-05T19:28:00.899+0000 7fb89dffb700 1 -- 172.21.15.118:0/3682425553 <== mon.0 v2:172.21.15.118:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fb8a8007250 con 0x7fb8ac095ff0 2023-10-05T19:28:00.905 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-05T19:28:00.899+0000 7fb8b2ec3700 1 -- 172.21.15.118:0/3682425553 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb8ac08ec50 con 0x7fb8ac095ff0 2023-10-05T19:28:00.907 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-05T19:28:00.899+0000 7fb89dffb700 1 -- 172.21.15.118:0/3682425553 <== mon.0 v2:172.21.15.118:3300/0 4 ==== mgrmap(e 5) v1 ==== 40329+0+0 (secure 0 0 0) 0x7fb8a80074c0 con 0x7fb8ac095ff0 2023-10-05T19:28:00.907 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-05T19:28:00.899+0000 7fb89dffb700 1 --2- 172.21.15.118:0/3682425553 >> [v2:172.21.15.167:6824/29889,v1:172.21.15.167:6825/29889] conn(0x7fb890032390 0x7fb890034840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-05T19:28:00.907 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-05T19:28:00.899+0000 7fb8b2ec3700 1 -- 172.21.15.118:0/3682425553 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb8ac08eff0 con 0x7fb8ac095ff0 2023-10-05T19:28:00.907 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-05T19:28:00.899+0000 7fb89dffb700 1 -- 172.21.15.118:0/3682425553 <== mon.0 v2:172.21.15.118:3300/0 5 ==== osd_map(15..15 src has 1..15) v4 ==== 3478+0+0 (secure 0 0 0) 0x7fb8a8042f90 con 0x7fb8ac095ff0 2023-10-05T19:28:00.907 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-05T19:28:00.899+0000 7fb8b16c0700 1 --2- 172.21.15.118:0/3682425553 >> [v2:172.21.15.167:6824/29889,v1:172.21.15.167:6825/29889] conn(0x7fb890032390 0x7fb890034840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-05T19:28:00.908 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-05T19:28:00.899+0000 7fb8b16c0700 1 --2- 172.21.15.118:0/3682425553 >> [v2:172.21.15.167:6824/29889,v1:172.21.15.167:6825/29889] conn(0x7fb890032390 0x7fb890034840 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7fb8a0009160 tx=0x7fb8a0015000 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-05T19:28:00.915 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-05T19:28:00.907+0000 7fb89dffb700 1 -- 172.21.15.118:0/3682425553 <== mon.0 v2:172.21.15.118:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166807 (secure 0 0 0) 0x7fb8a80192c0 con 0x7fb8ac095ff0 2023-10-05T19:28:01.071 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-05T19:28:01.063+0000 7fb8b2ec3700 1 -- 172.21.15.118:0/3682425553 --> [v2:172.21.15.167:6824/29889,v1:172.21.15.167:6825/29889] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7fb8ac05ecb0 con 0x7fb890032390 2023-10-05T19:28:01.075 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-05T19:28:01.071+0000 7fb89dffb700 1 -- 172.21.15.118:0/3682425553 <== mgr.4100 v2:172.21.15.167:6824/29889 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+34676 (secure 0 0 0) 0x7fb8ac05ecb0 con 0x7fb890032390 2023-10-05T19:28:01.075 INFO:teuthology.orchestra.run.smithi118.stdout: 2023-10-05T19:28:01.079 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-05T19:28:01.071+0000 7fb8b2ec3700 1 -- 172.21.15.118:0/3682425553 >> [v2:172.21.15.167:6824/29889,v1:172.21.15.167:6825/29889] conn(0x7fb890032390 msgr2=0x7fb890034840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-05T19:28:01.079 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-05T19:28:01.071+0000 7fb8b2ec3700 1 --2- 172.21.15.118:0/3682425553 >> [v2:172.21.15.167:6824/29889,v1:172.21.15.167:6825/29889] conn(0x7fb890032390 0x7fb890034840 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7fb8a0009160 tx=0x7fb8a0015000 comp rx=0 tx=0).stop 2023-10-05T19:28:01.079 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-05T19:28:01.071+0000 7fb8b2ec3700 1 -- 172.21.15.118:0/3682425553 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb8ac095ff0 msgr2=0x7fb8ac12b7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-05T19:28:01.079 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-05T19:28:01.071+0000 7fb8b2ec3700 1 --2- 172.21.15.118:0/3682425553 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb8ac095ff0 0x7fb8ac12b7a0 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7fb8a800c710 tx=0x7fb8a8007d90 comp rx=0 tx=0).stop 2023-10-05T19:28:01.079 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-05T19:28:01.071+0000 7fb8b2ec3700 1 -- 172.21.15.118:0/3682425553 shutdown_connections 2023-10-05T19:28:01.079 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-05T19:28:01.071+0000 7fb8b2ec3700 1 --2- 172.21.15.118:0/3682425553 >> [v2:172.21.15.167:6824/29889,v1:172.21.15.167:6825/29889] conn(0x7fb890032390 0x7fb890034840 unknown :-1 s=CLOSED pgs=43 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-05T19:28:01.079 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-05T19:28:01.071+0000 7fb8b2ec3700 1 --2- 172.21.15.118:0/3682425553 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb8ac095ff0 0x7fb8ac12b7a0 unknown :-1 s=CLOSED pgs=109 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-05T19:28:01.079 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-05T19:28:01.071+0000 7fb8b2ec3700 1 -- 172.21.15.118:0/3682425553 >> 172.21.15.118:0/3682425553 conn(0x7fb8ac09c5d0 msgr2=0x7fb8ac09ad60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-05T19:28:01.080 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-05T19:28:01.071+0000 7fb8b2ec3700 1 -- 172.21.15.118:0/3682425553 shutdown_connections 2023-10-05T19:28:01.080 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-05T19:28:01.071+0000 7fb8b2ec3700 1 -- 172.21.15.118:0/3682425553 wait complete. 2023-10-05T19:28:01.080 INFO:teuthology.orchestra.run.smithi118.stderr:dumped all 2023-10-05T19:28:01.094 INFO:teuthology.orchestra.run.smithi118.stdout:{"pg_ready":true,"pg_map":{"version":17,"stamp":"2023-10-05T19:28:00.914592+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":14,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":1194888,"kb_used_data":1640,"kb_used_omap":0,"kb_used_meta":1193216,"kb_avail":376292472,"statfs":{"total":386547056640,"available":385323491328,"internally_reserved":0,"allocated":1679360,"data_stored":1327976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":1221853184},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.174039"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":8,"reported_epoch":14,"state":"active+clean","last_fresh":"2023-10-05T19:27:58.035963+0000","last_change":"2023-10-05T19:27:58.035963+0000","last_active":"2023-10-05T19:27:58.035963+0000","last_peered":"2023-10-05T19:27:58.035963+0000","last_clean":"2023-10-05T19:27:58.035963+0000","last_became_active":"2023-10-05T19:27:58.034862+0000","last_became_peered":"2023-10-05T19:27:58.034862+0000","last_unstale":"2023-10-05T19:27:58.035963+0000","last_undegraded":"2023-10-05T19:27:58.035963+0000","last_fullsized":"2023-10-05T19:27:58.035963+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-05T19:27:56.732981+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-05T19:27:56.732981+0000","last_clean_scrub_stamp":"2023-10-05T19:27:56.732981+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-06T22:43:15.989376+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":9,"reported_epoch":14,"state":"active+clean","last_fresh":"2023-10-05T19:27:57.763364+0000","last_change":"2023-10-05T19:27:57.762520+0000","last_active":"2023-10-05T19:27:57.763364+0000","last_peered":"2023-10-05T19:27:57.763364+0000","last_clean":"2023-10-05T19:27:57.763364+0000","last_became_active":"2023-10-05T19:27:57.762202+0000","last_became_peered":"2023-10-05T19:27:57.762202+0000","last_unstale":"2023-10-05T19:27:57.763364+0000","last_undegraded":"2023-10-05T19:27:57.763364+0000","last_fullsized":"2023-10-05T19:27:57.763364+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-05T19:27:56.732981+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-05T19:27:56.732981+0000","last_clean_scrub_stamp":"2023-10-05T19:27:56.732981+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-06T22:08:09.710054+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":8,"reported_epoch":14,"state":"active+clean","last_fresh":"2023-10-05T19:27:57.759577+0000","last_change":"2023-10-05T19:27:57.759577+0000","last_active":"2023-10-05T19:27:57.759577+0000","last_peered":"2023-10-05T19:27:57.759577+0000","last_clean":"2023-10-05T19:27:57.759577+0000","last_became_active":"2023-10-05T19:27:57.758981+0000","last_became_peered":"2023-10-05T19:27:57.758981+0000","last_unstale":"2023-10-05T19:27:57.759577+0000","last_undegraded":"2023-10-05T19:27:57.759577+0000","last_fullsized":"2023-10-05T19:27:57.759577+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-05T19:27:56.732981+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-05T19:27:56.732981+0000","last_clean_scrub_stamp":"2023-10-05T19:27:56.732981+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-06T22:41:16.969181+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":9,"reported_epoch":14,"state":"active+clean","last_fresh":"2023-10-05T19:27:57.762620+0000","last_change":"2023-10-05T19:27:57.761710+0000","last_active":"2023-10-05T19:27:57.762620+0000","last_peered":"2023-10-05T19:27:57.762620+0000","last_clean":"2023-10-05T19:27:57.762620+0000","last_became_active":"2023-10-05T19:27:57.761424+0000","last_became_peered":"2023-10-05T19:27:57.761424+0000","last_unstale":"2023-10-05T19:27:57.762620+0000","last_undegraded":"2023-10-05T19:27:57.762620+0000","last_fullsized":"2023-10-05T19:27:57.762620+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-05T19:27:56.732981+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-05T19:27:56.732981+0000","last_clean_scrub_stamp":"2023-10-05T19:27:56.732981+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T02:33:00.026989+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":8,"reported_epoch":14,"state":"active+clean","last_fresh":"2023-10-05T19:27:57.759916+0000","last_change":"2023-10-05T19:27:57.759916+0000","last_active":"2023-10-05T19:27:57.759916+0000","last_peered":"2023-10-05T19:27:57.759916+0000","last_clean":"2023-10-05T19:27:57.759916+0000","last_became_active":"2023-10-05T19:27:57.759323+0000","last_became_peered":"2023-10-05T19:27:57.759323+0000","last_unstale":"2023-10-05T19:27:57.759916+0000","last_undegraded":"2023-10-05T19:27:57.759916+0000","last_fullsized":"2023-10-05T19:27:57.759916+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-05T19:27:56.732981+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-05T19:27:56.732981+0000","last_clean_scrub_stamp":"2023-10-05T19:27:56.732981+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T00:41:14.340083+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":9,"reported_epoch":14,"state":"active+clean","last_fresh":"2023-10-05T19:27:57.763870+0000","last_change":"2023-10-05T19:27:57.760314+0000","last_active":"2023-10-05T19:27:57.763870+0000","last_peered":"2023-10-05T19:27:57.763870+0000","last_clean":"2023-10-05T19:27:57.763870+0000","last_became_active":"2023-10-05T19:27:57.759809+0000","last_became_peered":"2023-10-05T19:27:57.759809+0000","last_unstale":"2023-10-05T19:27:57.763870+0000","last_undegraded":"2023-10-05T19:27:57.763870+0000","last_fullsized":"2023-10-05T19:27:57.763870+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-05T19:27:56.732981+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-05T19:27:56.732981+0000","last_clean_scrub_stamp":"2023-10-05T19:27:56.732981+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T04:11:37.673289+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":8,"reported_epoch":14,"state":"active+clean","last_fresh":"2023-10-05T19:27:57.759697+0000","last_change":"2023-10-05T19:27:57.759697+0000","last_active":"2023-10-05T19:27:57.759697+0000","last_peered":"2023-10-05T19:27:57.759697+0000","last_clean":"2023-10-05T19:27:57.759697+0000","last_became_active":"2023-10-05T19:27:57.759232+0000","last_became_peered":"2023-10-05T19:27:57.759232+0000","last_unstale":"2023-10-05T19:27:57.759697+0000","last_undegraded":"2023-10-05T19:27:57.759697+0000","last_fullsized":"2023-10-05T19:27:57.759697+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-05T19:27:56.732981+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-05T19:27:56.732981+0000","last_clean_scrub_stamp":"2023-10-05T19:27:56.732981+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T03:39:17.485984+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":8,"reported_epoch":14,"state":"active+clean","last_fresh":"2023-10-05T19:27:57.759809+0000","last_change":"2023-10-05T19:27:57.759809+0000","last_active":"2023-10-05T19:27:57.759809+0000","last_peered":"2023-10-05T19:27:57.759809+0000","last_clean":"2023-10-05T19:27:57.759809+0000","last_became_active":"2023-10-05T19:27:57.759347+0000","last_became_peered":"2023-10-05T19:27:57.759347+0000","last_unstale":"2023-10-05T19:27:57.759809+0000","last_undegraded":"2023-10-05T19:27:57.759809+0000","last_fullsized":"2023-10-05T19:27:57.759809+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-05T19:27:56.732981+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-05T19:27:56.732981+0000","last_clean_scrub_stamp":"2023-10-05T19:27:56.732981+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T01:49:38.110242+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'117","reported_seq":161,"reported_epoch":13,"state":"active+clean","last_fresh":"2023-10-05T19:27:56.743894+0000","last_change":"2023-10-05T19:27:52.467966+0000","last_active":"2023-10-05T19:27:56.743894+0000","last_peered":"2023-10-05T19:27:56.743894+0000","last_clean":"2023-10-05T19:27:56.743894+0000","last_became_active":"2023-10-05T19:27:52.465477+0000","last_became_peered":"2023-10-05T19:27:52.465477+0000","last_unstale":"2023-10-05T19:27:56.743894+0000","last_undegraded":"2023-10-05T19:27:56.743894+0000","last_fullsized":"2023-10-05T19:27:56.743894+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-05T19:27:51.198374+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-05T19:27:51.198374+0000","last_clean_scrub_stamp":"2023-10-05T19:27:51.198374+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T03:59:55.896733+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":4},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705668,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297408,"kb_used_data":120,"kb_used_omap":0,"kb_used_meta":297280,"kb_avail":94074432,"statfs":{"total":96636764160,"available":96332218368,"internally_reserved":0,"allocated":122880,"data_stored":36810,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304414720},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.70799999999999996}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.56599999999999995}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.63600000000000001}]}]},{"osd":2,"up_from":9,"seq":38654705670,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297988,"kb_used_data":700,"kb_used_omap":0,"kb_used_meta":297280,"kb_avail":94073852,"statfs":{"total":96636764160,"available":96331624448,"internally_reserved":0,"allocated":716800,"data_stored":627178,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304414720},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.63300000000000001}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.50700000000000001}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.58799999999999997}]}]},{"osd":1,"up_from":9,"seq":38654705669,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297988,"kb_used_data":700,"kb_used_omap":0,"kb_used_meta":297280,"kb_avail":94073852,"statfs":{"total":96636764160,"available":96331624448,"internally_reserved":0,"allocated":716800,"data_stored":627178,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304414720},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.80900000000000005}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.73599999999999999}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.89100000000000001}]}]},{"osd":0,"up_from":9,"seq":38654705668,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":301504,"kb_used_data":120,"kb_used_omap":0,"kb_used_meta":301376,"kb_avail":94070336,"statfs":{"total":96636764160,"available":96328024064,"internally_reserved":0,"allocated":122880,"data_stored":36810,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":308609024},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.77200000000000002}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.64000000000000001}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.59999999999999998}]}]}],"pool_statfs":[{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-10-05T19:28:01.095 INFO:tasks.ceph.ceph_manager.ceph:clean! 2023-10-05T19:28:01.095 INFO:tasks.ceph:Waiting until ceph cluster ceph is healthy... 2023-10-05T19:28:01.095 INFO:tasks.ceph.ceph_manager.ceph:wait_until_healthy 2023-10-05T19:28:01.095 DEBUG:teuthology.orchestra.run.smithi118:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-10-05T19:28:01.198 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-05T19:28:01.191+0000 7f1d2d539700 1 Processor -- start 2023-10-05T19:28:01.198 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-05T19:28:01.191+0000 7f1d2d539700 1 -- start start 2023-10-05T19:28:01.199 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-05T19:28:01.191+0000 7f1d2d539700 1 --2- >> v2:172.21.15.118:3300/0 conn(0x7f1d2806d730 0x7f1d2812e690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-05T19:28:01.199 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-05T19:28:01.191+0000 7f1d2d539700 1 -- --> v1:172.21.15.118:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1d2806b850 con 0x7f1d2806cc50 2023-10-05T19:28:01.199 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-05T19:28:01.191+0000 7f1d2d539700 1 -- --> v2:172.21.15.118:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1d2806db60 con 0x7f1d2806d730 2023-10-05T19:28:01.199 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-05T19:28:01.191+0000 7f1d27fff700 1 --1- >> v1:172.21.15.118:6789/0 conn(0x7f1d2806cc50 0x7f1d2806d020 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.118:6789/0 says I am v1:172.21.15.118:52188/0 (socket says 172.21.15.118:52188) 2023-10-05T19:28:01.199 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-05T19:28:01.191+0000 7f1d27fff700 1 -- 172.21.15.118:0/1707161439 learned_addr learned my addr 172.21.15.118:0/1707161439 (peer_addr_for_me v1:172.21.15.118:0/0) 2023-10-05T19:28:01.199 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-05T19:28:01.191+0000 7f1d277fe700 1 --2- 172.21.15.118:0/1707161439 >> v2:172.21.15.118:3300/0 conn(0x7f1d2806d730 0x7f1d2812e690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-05T19:28:01.199 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-05T19:28:01.191+0000 7f1d26ffd700 1 -- 172.21.15.118:0/1707161439 <== mon.0 v1:172.21.15.118:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1621775654 0 0) 0x7f1d2806b850 con 0x7f1d2806cc50 2023-10-05T19:28:01.199 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-05T19:28:01.191+0000 7f1d26ffd700 1 -- 172.21.15.118:0/1707161439 --> v1:172.21.15.118:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1d10003570 con 0x7f1d2806cc50 2023-10-05T19:28:01.201 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-05T19:28:01.191+0000 7f1d26ffd700 1 -- 172.21.15.118:0/1707161439 <== mon.0 v1:172.21.15.118:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3055963895 0 0) 0x7f1d10003570 con 0x7f1d2806cc50 2023-10-05T19:28:01.201 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-05T19:28:01.191+0000 7f1d26ffd700 1 -- 172.21.15.118:0/1707161439 >> v2:172.21.15.118:3300/0 conn(0x7f1d2806d730 msgr2=0x7f1d2812e690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-05T19:28:01.201 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-05T19:28:01.191+0000 7f1d26ffd700 1 --2- 172.21.15.118:0/1707161439 >> v2:172.21.15.118:3300/0 conn(0x7f1d2806d730 0x7f1d2812e690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-05T19:28:01.201 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-05T19:28:01.191+0000 7f1d26ffd700 1 -- 172.21.15.118:0/1707161439 --> v1:172.21.15.118:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1d2806a670 con 0x7f1d2806cc50 2023-10-05T19:28:01.201 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-05T19:28:01.191+0000 7f1d26ffd700 1 -- 172.21.15.118:0/1707161439 <== mon.0 v1:172.21.15.118:6789/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (unknown 2681859047 0 0) 0x7f1d20001930 con 0x7f1d2806cc50 2023-10-05T19:28:01.201 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-05T19:28:01.191+0000 7f1d26ffd700 1 -- 172.21.15.118:0/1707161439 >> v1:172.21.15.118:6789/0 conn(0x7f1d2806cc50 legacy=0x7f1d2806d020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-05T19:28:01.201 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-05T19:28:01.191+0000 7f1d26ffd700 1 --2- 172.21.15.118:0/1707161439 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f1d10003a60 0x7f1d10003e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-05T19:28:01.201 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-05T19:28:01.191+0000 7f1d26ffd700 1 -- 172.21.15.118:0/1707161439 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1d2806db60 con 0x7f1d10003a60 2023-10-05T19:28:01.202 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-05T19:28:01.191+0000 7f1d277fe700 1 --2- 172.21.15.118:0/1707161439 >> v2:172.21.15.118:3300/0 conn(0x7f1d2806d730 0x7f1d2812e690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-05T19:28:01.202 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-05T19:28:01.191+0000 7f1d27fff700 1 --2- 172.21.15.118:0/1707161439 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f1d10003a60 0x7f1d10003e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-05T19:28:01.202 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-05T19:28:01.191+0000 7f1d27fff700 1 -- 172.21.15.118:0/1707161439 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1d10003570 con 0x7f1d10003a60 2023-10-05T19:28:01.202 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-05T19:28:01.191+0000 7f1d27fff700 1 --2- 172.21.15.118:0/1707161439 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f1d10003a60 0x7f1d10003e50 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7f1d2000a7c0 tx=0x7f1d20002160 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bf280bd7bfa5c5b5 server_cookie=0 in_seq=0 out_seq=0 2023-10-05T19:28:01.202 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-05T19:28:01.195+0000 7f1d26ffd700 1 -- 172.21.15.118:0/1707161439 <== mon.0 v2:172.21.15.118:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f1d200070f0 con 0x7f1d10003a60 2023-10-05T19:28:01.202 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-05T19:28:01.195+0000 7f1d26ffd700 1 -- 172.21.15.118:0/1707161439 <== mon.0 v2:172.21.15.118:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1d20007270 con 0x7f1d10003a60 2023-10-05T19:28:01.202 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-05T19:28:01.195+0000 7f1d26ffd700 1 -- 172.21.15.118:0/1707161439 <== mon.0 v2:172.21.15.118:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f1d20007420 con 0x7f1d10003a60 2023-10-05T19:28:01.203 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-05T19:28:01.195+0000 7f1d2d539700 1 -- 172.21.15.118:0/1707161439 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f1d10003a60 msgr2=0x7f1d10003e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-05T19:28:01.204 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-05T19:28:01.195+0000 7f1d2d539700 1 --2- 172.21.15.118:0/1707161439 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f1d10003a60 0x7f1d10003e50 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7f1d2000a7c0 tx=0x7f1d20002160 comp rx=0 tx=0).stop 2023-10-05T19:28:01.204 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-05T19:28:01.195+0000 7f1d2d539700 1 -- 172.21.15.118:0/1707161439 shutdown_connections 2023-10-05T19:28:01.204 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-05T19:28:01.195+0000 7f1d2d539700 1 --2- 172.21.15.118:0/1707161439 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f1d10003a60 0x7f1d10003e50 unknown :-1 s=CLOSED pgs=111 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-05T19:28:01.204 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-05T19:28:01.195+0000 7f1d2d539700 1 --2- 172.21.15.118:0/1707161439 >> v2:172.21.15.118:3300/0 conn(0x7f1d2806d730 0x7f1d2812e690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-05T19:28:01.204 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-05T19:28:01.195+0000 7f1d2d539700 1 -- 172.21.15.118:0/1707161439 >> 172.21.15.118:0/1707161439 conn(0x7f1d2809a530 msgr2=0x7f1d2809a930 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-05T19:28:01.204 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-05T19:28:01.195+0000 7f1d2d539700 1 -- 172.21.15.118:0/1707161439 shutdown_connections 2023-10-05T19:28:01.204 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-05T19:28:01.195+0000 7f1d2d539700 1 -- 172.21.15.118:0/1707161439 wait complete. 2023-10-05T19:28:01.204 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-05T19:28:01.195+0000 7f1d2d539700 1 Processor -- start 2023-10-05T19:28:01.204 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-05T19:28:01.195+0000 7f1d2d539700 1 -- start start 2023-10-05T19:28:01.206 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-05T19:28:01.199+0000 7f1d2d539700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f1d10003a60 0x7f1d2811c700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-05T19:28:01.206 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-05T19:28:01.199+0000 7f1d2d539700 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1d28129fa0 con 0x7f1d10003a60 2023-10-05T19:28:01.206 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-05T19:28:01.199+0000 7f1d27fff700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f1d10003a60 0x7f1d2811c700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-05T19:28:01.206 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-05T19:28:01.199+0000 7f1d27fff700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f1d10003a60 0x7f1d2811c700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.118:3300/0 says I am v2:172.21.15.118:56032/0 (socket says 172.21.15.118:56032) 2023-10-05T19:28:01.206 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-05T19:28:01.199+0000 7f1d27fff700 1 -- 172.21.15.118:0/1381521167 learned_addr learned my addr 172.21.15.118:0/1381521167 (peer_addr_for_me v2:172.21.15.118:0/0) 2023-10-05T19:28:01.206 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-05T19:28:01.199+0000 7f1d27fff700 1 -- 172.21.15.118:0/1381521167 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1d1c009420 con 0x7f1d10003a60 2023-10-05T19:28:01.206 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-05T19:28:01.199+0000 7f1d27fff700 1 --2- 172.21.15.118:0/1381521167 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f1d10003a60 0x7f1d2811c700 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7f1d2000a380 tx=0x7f1d20013000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-05T19:28:01.206 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-05T19:28:01.199+0000 7f1d0ffff700 1 -- 172.21.15.118:0/1381521167 <== mon.0 v2:172.21.15.118:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f1d20022030 con 0x7f1d10003a60 2023-10-05T19:28:01.207 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-05T19:28:01.199+0000 7f1d0ffff700 1 -- 172.21.15.118:0/1381521167 <== mon.0 v2:172.21.15.118:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1d20007b40 con 0x7f1d10003a60 2023-10-05T19:28:01.207 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-05T19:28:01.199+0000 7f1d2d539700 1 -- 172.21.15.118:0/1381521167 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1d2811fda0 con 0x7f1d10003a60 2023-10-05T19:28:01.207 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-05T19:28:01.199+0000 7f1d0ffff700 1 -- 172.21.15.118:0/1381521167 <== mon.0 v2:172.21.15.118:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f1d200073e0 con 0x7f1d10003a60 2023-10-05T19:28:01.207 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-05T19:28:01.199+0000 7f1d2d539700 1 -- 172.21.15.118:0/1381521167 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1d2811f420 con 0x7f1d10003a60 2023-10-05T19:28:01.208 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-05T19:28:01.199+0000 7f1d2d539700 1 -- 172.21.15.118:0/1381521167 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1d2805ecb0 con 0x7f1d10003a60 2023-10-05T19:28:01.208 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-05T19:28:01.199+0000 7f1d0ffff700 1 -- 172.21.15.118:0/1381521167 <== mon.0 v2:172.21.15.118:3300/0 4 ==== mgrmap(e 5) v1 ==== 40329+0+0 (secure 0 0 0) 0x7f1d20025070 con 0x7f1d10003a60 2023-10-05T19:28:01.208 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-05T19:28:01.199+0000 7f1d0ffff700 1 --2- 172.21.15.118:0/1381521167 >> [v2:172.21.15.167:6824/29889,v1:172.21.15.167:6825/29889] conn(0x7f1d14032380 0x7f1d14034830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-05T19:28:01.208 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-05T19:28:01.203+0000 7f1d0ffff700 1 -- 172.21.15.118:0/1381521167 <== mon.0 v2:172.21.15.118:3300/0 5 ==== osd_map(15..15 src has 1..15) v4 ==== 3478+0+0 (secure 0 0 0) 0x7f1d20023070 con 0x7f1d10003a60 2023-10-05T19:28:01.209 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-05T19:28:01.203+0000 7f1d277fe700 1 --2- 172.21.15.118:0/1381521167 >> [v2:172.21.15.167:6824/29889,v1:172.21.15.167:6825/29889] conn(0x7f1d14032380 0x7f1d14034830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-05T19:28:01.209 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-05T19:28:01.203+0000 7f1d277fe700 1 --2- 172.21.15.118:0/1381521167 >> [v2:172.21.15.167:6824/29889,v1:172.21.15.167:6825/29889] conn(0x7f1d14032380 0x7f1d14034830 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f1d1c009c80 tx=0x7f1d1c0091a0 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-05T19:28:01.213 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-05T19:28:01.207+0000 7f1d0ffff700 1 -- 172.21.15.118:0/1381521167 <== mon.0 v2:172.21.15.118:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166807 (secure 0 0 0) 0x7f1d20007070 con 0x7f1d10003a60 2023-10-05T19:28:01.406 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-05T19:28:01.399+0000 7f1d2d539700 1 -- 172.21.15.118:0/1381521167 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7f1d2811fb90 con 0x7f1d10003a60 2023-10-05T19:28:01.406 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-05T19:28:01.399+0000 7f1d0ffff700 1 -- 172.21.15.118:0/1381521167 <== mon.0 v2:172.21.15.118:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+46 (secure 0 0 0) 0x7f1d2001b420 con 0x7f1d10003a60 2023-10-05T19:28:01.407 INFO:teuthology.orchestra.run.smithi118.stdout: 2023-10-05T19:28:01.407 INFO:teuthology.orchestra.run.smithi118.stdout:{"status":"HEALTH_OK","checks":{},"mutes":[]} 2023-10-05T19:28:01.409 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-05T19:28:01.403+0000 7f1d2d539700 1 -- 172.21.15.118:0/1381521167 >> [v2:172.21.15.167:6824/29889,v1:172.21.15.167:6825/29889] conn(0x7f1d14032380 msgr2=0x7f1d14034830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-05T19:28:01.409 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-05T19:28:01.403+0000 7f1d2d539700 1 --2- 172.21.15.118:0/1381521167 >> [v2:172.21.15.167:6824/29889,v1:172.21.15.167:6825/29889] conn(0x7f1d14032380 0x7f1d14034830 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f1d1c009c80 tx=0x7f1d1c0091a0 comp rx=0 tx=0).stop 2023-10-05T19:28:01.409 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-05T19:28:01.403+0000 7f1d2d539700 1 -- 172.21.15.118:0/1381521167 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f1d10003a60 msgr2=0x7f1d2811c700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-05T19:28:01.409 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-05T19:28:01.403+0000 7f1d2d539700 1 --2- 172.21.15.118:0/1381521167 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f1d10003a60 0x7f1d2811c700 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7f1d2000a380 tx=0x7f1d20013000 comp rx=0 tx=0).stop 2023-10-05T19:28:01.409 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-05T19:28:01.403+0000 7f1d2d539700 1 -- 172.21.15.118:0/1381521167 shutdown_connections 2023-10-05T19:28:01.410 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-05T19:28:01.403+0000 7f1d2d539700 1 --2- 172.21.15.118:0/1381521167 >> [v2:172.21.15.167:6824/29889,v1:172.21.15.167:6825/29889] conn(0x7f1d14032380 0x7f1d14034830 unknown :-1 s=CLOSED pgs=44 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-05T19:28:01.410 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-05T19:28:01.403+0000 7f1d2d539700 1 --2- 172.21.15.118:0/1381521167 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f1d10003a60 0x7f1d2811c700 unknown :-1 s=CLOSED pgs=112 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-05T19:28:01.410 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-05T19:28:01.403+0000 7f1d2d539700 1 -- 172.21.15.118:0/1381521167 >> 172.21.15.118:0/1381521167 conn(0x7f1d2809a530 msgr2=0x7f1d28096130 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-05T19:28:01.410 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-05T19:28:01.403+0000 7f1d2d539700 1 -- 172.21.15.118:0/1381521167 shutdown_connections 2023-10-05T19:28:01.410 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-05T19:28:01.403+0000 7f1d2d539700 1 -- 172.21.15.118:0/1381521167 wait complete. 2023-10-05T19:28:01.420 INFO:tasks.ceph.ceph_manager.ceph:wait_until_healthy done 2023-10-05T19:28:01.420 INFO:teuthology.run_tasks:Running task rgw... 2023-10-05T19:28:01.437 DEBUG:tasks.rgw:config is {'client.0': None} 2023-10-05T19:28:01.437 DEBUG:tasks.rgw:client list is dict_keys(['client.0']) 2023-10-05T19:28:01.437 INFO:tasks.rgw:Creating data pools 2023-10-05T19:28:01.437 DEBUG:tasks.rgw:Obtaining remote for client client.0 2023-10-05T19:28:01.438 DEBUG:teuthology.orchestra.run.smithi167:> sudo ceph osd erasure-code-profile set client.0 crush-failure-domain=osd k=3 m=1 name=testprofile --cluster ceph 2023-10-05T19:28:01.556 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:01.550+0000 7f43e3fff700 1 Processor -- start 2023-10-05T19:28:01.556 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:01.550+0000 7f43e3fff700 1 -- start start 2023-10-05T19:28:01.556 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:01.550+0000 7f43e3fff700 1 --2- >> v2:172.21.15.118:3300/0 conn(0x7f43e410cba0 0x7f43e406ed50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-05T19:28:01.556 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:01.550+0000 7f43e3fff700 1 -- --> v1:172.21.15.118:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f43e406f320 con 0x7f43e410c6e0 2023-10-05T19:28:01.556 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:01.550+0000 7f43e3fff700 1 -- --> v2:172.21.15.118:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f43e40631f0 con 0x7f43e410cba0 2023-10-05T19:28:01.557 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:01.550+0000 7f43e27fc700 1 --2- >> v2:172.21.15.118:3300/0 conn(0x7f43e410cba0 0x7f43e406ed50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-05T19:28:01.557 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:01.550+0000 7f43e27fc700 1 --2- >> v2:172.21.15.118:3300/0 conn(0x7f43e410cba0 0x7f43e406ed50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.118:3300/0 says I am v2:172.21.15.167:55978/0 (socket says 172.21.15.167:55978) 2023-10-05T19:28:01.558 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:01.550+0000 7f43e27fc700 1 -- 172.21.15.167:0/1424641720 learned_addr learned my addr 172.21.15.167:0/1424641720 (peer_addr_for_me v2:172.21.15.167:0/0) 2023-10-05T19:28:01.558 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:01.550+0000 7f43e2ffd700 1 --1- >> v1:172.21.15.118:6789/0 conn(0x7f43e410c6e0 0x7f43e406e640 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.118:6789/0 says I am v1:172.21.15.167:50492/0 (socket says 172.21.15.167:50492) 2023-10-05T19:28:01.558 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:01.550+0000 7f43e27fc700 1 -- 172.21.15.167:0/1424641720 >> v1:172.21.15.118:6789/0 conn(0x7f43e410c6e0 legacy=0x7f43e406e640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-05T19:28:01.558 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:01.550+0000 7f43e27fc700 1 -- 172.21.15.167:0/1424641720 --> v2:172.21.15.118:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f43e406f4b0 con 0x7f43e410cba0 2023-10-05T19:28:01.558 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:01.550+0000 7f43e27fc700 1 --2- 172.21.15.167:0/1424641720 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f43e410cba0 0x7f43e406ed50 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7f43d4009930 tx=0x7f43d4009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9b8fabf67c8943dd server_cookie=0 in_seq=0 out_seq=0 2023-10-05T19:28:01.559 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:01.554+0000 7f43e1ffb700 1 -- 172.21.15.167:0/1424641720 <== mon.0 v2:172.21.15.118:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f43d400b860 con 0x7f43e410cba0 2023-10-05T19:28:01.560 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:01.554+0000 7f43e1ffb700 1 -- 172.21.15.167:0/1424641720 <== mon.0 v2:172.21.15.118:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f43d400b9c0 con 0x7f43e410cba0 2023-10-05T19:28:01.560 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:01.554+0000 7f43e1ffb700 1 -- 172.21.15.167:0/1424641720 <== mon.0 v2:172.21.15.118:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f43d400bb50 con 0x7f43e410cba0 2023-10-05T19:28:01.560 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:01.554+0000 7f43e3fff700 1 -- 172.21.15.167:0/1424641720 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f43e410cba0 msgr2=0x7f43e406ed50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-05T19:28:01.560 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:01.554+0000 7f43e3fff700 1 --2- 172.21.15.167:0/1424641720 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f43e410cba0 0x7f43e406ed50 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7f43d4009930 tx=0x7f43d4009250 comp rx=0 tx=0).stop 2023-10-05T19:28:01.560 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:01.554+0000 7f43e3fff700 1 -- 172.21.15.167:0/1424641720 shutdown_connections 2023-10-05T19:28:01.560 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:01.554+0000 7f43e3fff700 1 --2- 172.21.15.167:0/1424641720 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f43e410cba0 0x7f43e406ed50 unknown :-1 s=CLOSED pgs=114 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-05T19:28:01.560 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:01.554+0000 7f43e3fff700 1 -- 172.21.15.167:0/1424641720 >> 172.21.15.167:0/1424641720 conn(0x7f43e406a590 msgr2=0x7f43e406c9a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-05T19:28:01.560 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:01.554+0000 7f43e3fff700 1 -- 172.21.15.167:0/1424641720 shutdown_connections 2023-10-05T19:28:01.560 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:01.554+0000 7f43e3fff700 1 -- 172.21.15.167:0/1424641720 wait complete. 2023-10-05T19:28:01.560 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:01.554+0000 7f43e3fff700 1 Processor -- start 2023-10-05T19:28:01.561 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:01.554+0000 7f43e3fff700 1 -- start start 2023-10-05T19:28:01.562 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:01.554+0000 7f43e3fff700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f43e410c6e0 0x7f43e411bce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-05T19:28:01.562 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:01.554+0000 7f43e3fff700 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f43e406ad90 con 0x7f43e410c6e0 2023-10-05T19:28:01.562 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:01.554+0000 7f43e2ffd700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f43e410c6e0 0x7f43e411bce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-05T19:28:01.562 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:01.554+0000 7f43e2ffd700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f43e410c6e0 0x7f43e411bce0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.118:3300/0 says I am v2:172.21.15.167:55984/0 (socket says 172.21.15.167:55984) 2023-10-05T19:28:01.562 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:01.554+0000 7f43e2ffd700 1 -- 172.21.15.167:0/3449033763 learned_addr learned my addr 172.21.15.167:0/3449033763 (peer_addr_for_me v2:172.21.15.167:0/0) 2023-10-05T19:28:01.563 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:01.554+0000 7f43e2ffd700 1 -- 172.21.15.167:0/3449033763 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f43d40094f0 con 0x7f43e410c6e0 2023-10-05T19:28:01.563 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:01.554+0000 7f43e2ffd700 1 --2- 172.21.15.167:0/3449033763 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f43e410c6e0 0x7f43e411bce0 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7f43cc00c6c0 tx=0x7f43cc007f30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-05T19:28:01.563 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:01.554+0000 7f43daffd700 1 -- 172.21.15.167:0/3449033763 <== mon.0 v2:172.21.15.118:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f43cc005a70 con 0x7f43e410c6e0 2023-10-05T19:28:01.563 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:01.554+0000 7f43e3fff700 1 -- 172.21.15.167:0/3449033763 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f43e411c220 con 0x7f43e410c6e0 2023-10-05T19:28:01.563 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:01.554+0000 7f43daffd700 1 -- 172.21.15.167:0/3449033763 <== mon.0 v2:172.21.15.118:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f43cc005bd0 con 0x7f43e410c6e0 2023-10-05T19:28:01.563 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:01.554+0000 7f43e3fff700 1 -- 172.21.15.167:0/3449033763 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f43e411c670 con 0x7f43e410c6e0 2023-10-05T19:28:01.563 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:01.554+0000 7f43daffd700 1 -- 172.21.15.167:0/3449033763 <== mon.0 v2:172.21.15.118:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f43cc005e80 con 0x7f43e410c6e0 2023-10-05T19:28:01.564 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:01.558+0000 7f43daffd700 1 -- 172.21.15.167:0/3449033763 <== mon.0 v2:172.21.15.118:3300/0 4 ==== mgrmap(e 5) v1 ==== 40329+0+0 (secure 0 0 0) 0x7f43cc0070d0 con 0x7f43e410c6e0 2023-10-05T19:28:01.564 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:01.558+0000 7f43daffd700 1 --2- 172.21.15.167:0/3449033763 >> [v2:172.21.15.167:6824/29889,v1:172.21.15.167:6825/29889] conn(0x7f43c4032390 0x7f43c4034840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-05T19:28:01.564 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:01.558+0000 7f43daffd700 1 -- 172.21.15.167:0/3449033763 <== mon.0 v2:172.21.15.118:3300/0 5 ==== osd_map(15..15 src has 1..15) v4 ==== 3478+0+0 (secure 0 0 0) 0x7f43cc042ee0 con 0x7f43e410c6e0 2023-10-05T19:28:01.564 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:01.558+0000 7f43e3fff700 1 -- 172.21.15.167:0/3449033763 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f43e411b8f0 con 0x7f43e410c6e0 2023-10-05T19:28:01.564 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:01.558+0000 7f43e27fc700 1 --2- 172.21.15.167:0/3449033763 >> [v2:172.21.15.167:6824/29889,v1:172.21.15.167:6825/29889] conn(0x7f43c4032390 0x7f43c4034840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-05T19:28:01.564 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:01.558+0000 7f43e27fc700 1 --2- 172.21.15.167:0/3449033763 >> [v2:172.21.15.167:6824/29889,v1:172.21.15.167:6825/29889] conn(0x7f43c4032390 0x7f43c4034840 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f43d4009930 tx=0x7f43d4016000 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-05T19:28:01.572 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:01.566+0000 7f43daffd700 1 -- 172.21.15.167:0/3449033763 <== mon.0 v2:172.21.15.118:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166807 (secure 0 0 0) 0x7f43cc017020 con 0x7f43e410c6e0 2023-10-05T19:28:01.745 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:01.738+0000 7f43e3fff700 1 -- 172.21.15.167:0/3449033763 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_command({"prefix": "osd erasure-code-profile set", "name": "client.0", "profile": ["crush-failure-domain=osd", "k=3", "m=1", "name=testprofile"]} v 0) v1 -- 0x7f43e405ecb0 con 0x7f43e410c6e0 2023-10-05T19:28:01.772 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:01.766+0000 7f43daffd700 1 -- 172.21.15.167:0/3449033763 <== mon.0 v2:172.21.15.118:3300/0 7 ==== mon_command_ack([{"prefix": "osd erasure-code-profile set", "name": "client.0", "profile": ["crush-failure-domain=osd", "k=3", "m=1", "name=testprofile"]}]=0 v16) v1 ==== 171+0+0 (secure 0 0 0) 0x7f43cc024070 con 0x7f43e410c6e0 2023-10-05T19:28:01.774 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:01.770+0000 7f43e3fff700 1 -- 172.21.15.167:0/3449033763 >> [v2:172.21.15.167:6824/29889,v1:172.21.15.167:6825/29889] conn(0x7f43c4032390 msgr2=0x7f43c4034840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-05T19:28:01.775 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:01.770+0000 7f43e3fff700 1 --2- 172.21.15.167:0/3449033763 >> [v2:172.21.15.167:6824/29889,v1:172.21.15.167:6825/29889] conn(0x7f43c4032390 0x7f43c4034840 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f43d4009930 tx=0x7f43d4016000 comp rx=0 tx=0).stop 2023-10-05T19:28:01.775 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:01.770+0000 7f43e3fff700 1 -- 172.21.15.167:0/3449033763 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f43e410c6e0 msgr2=0x7f43e411bce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-05T19:28:01.775 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:01.770+0000 7f43e3fff700 1 --2- 172.21.15.167:0/3449033763 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f43e410c6e0 0x7f43e411bce0 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7f43cc00c6c0 tx=0x7f43cc007f30 comp rx=0 tx=0).stop 2023-10-05T19:28:01.775 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:01.770+0000 7f43e3fff700 1 -- 172.21.15.167:0/3449033763 shutdown_connections 2023-10-05T19:28:01.775 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:01.770+0000 7f43e3fff700 1 --2- 172.21.15.167:0/3449033763 >> [v2:172.21.15.167:6824/29889,v1:172.21.15.167:6825/29889] conn(0x7f43c4032390 0x7f43c4034840 unknown :-1 s=CLOSED pgs=45 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-05T19:28:01.775 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:01.770+0000 7f43e3fff700 1 --2- 172.21.15.167:0/3449033763 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f43e410c6e0 0x7f43e411bce0 unknown :-1 s=CLOSED pgs=115 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-05T19:28:01.775 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:01.770+0000 7f43e3fff700 1 -- 172.21.15.167:0/3449033763 >> 172.21.15.167:0/3449033763 conn(0x7f43e406a590 msgr2=0x7f43e406d920 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-05T19:28:01.775 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:01.770+0000 7f43e3fff700 1 -- 172.21.15.167:0/3449033763 shutdown_connections 2023-10-05T19:28:01.775 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:01.770+0000 7f43e3fff700 1 -- 172.21.15.167:0/3449033763 wait complete. 2023-10-05T19:28:01.788 DEBUG:teuthology.orchestra.run.smithi167:> sudo ceph osd pool create default.rgw.buckets.data 64 64 erasure client.0 --cluster ceph 2023-10-05T19:28:01.897 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:01.890+0000 7f1010d7e700 1 Processor -- start 2023-10-05T19:28:01.897 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:01.890+0000 7f1010d7e700 1 -- start start 2023-10-05T19:28:01.898 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:01.890+0000 7f1010d7e700 1 --2- >> v2:172.21.15.118:3300/0 conn(0x7f100c115800 0x7f100c117be0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-05T19:28:01.898 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:01.890+0000 7f1010d7e700 1 -- --> v1:172.21.15.118:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f100c118120 con 0x7f100c112aa0 2023-10-05T19:28:01.898 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:01.890+0000 7f1010d7e700 1 -- --> v2:172.21.15.118:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f100c04c3e0 con 0x7f100c115800 2023-10-05T19:28:01.898 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:01.894+0000 7f1002ffd700 1 --2- >> v2:172.21.15.118:3300/0 conn(0x7f100c115800 0x7f100c117be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-05T19:28:01.898 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:01.894+0000 7f1002ffd700 1 --2- >> v2:172.21.15.118:3300/0 conn(0x7f100c115800 0x7f100c117be0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.118:3300/0 says I am v2:172.21.15.167:55992/0 (socket says 172.21.15.167:55992) 2023-10-05T19:28:01.898 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:01.894+0000 7f1002ffd700 1 -- 172.21.15.167:0/1282134332 learned_addr learned my addr 172.21.15.167:0/1282134332 (peer_addr_for_me v2:172.21.15.167:0/0) 2023-10-05T19:28:01.898 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:01.894+0000 7f100b7fe700 1 --1- >> v1:172.21.15.118:6789/0 conn(0x7f100c112aa0 0x7f100c112e70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.118:6789/0 says I am v1:172.21.15.167:50506/0 (socket says 172.21.15.167:50506) 2023-10-05T19:28:01.899 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:01.894+0000 7f1002ffd700 1 -- 172.21.15.167:0/1282134332 >> v1:172.21.15.118:6789/0 conn(0x7f100c112aa0 legacy=0x7f100c112e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-05T19:28:01.899 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:01.894+0000 7f1002ffd700 1 -- 172.21.15.167:0/1282134332 --> v2:172.21.15.118:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f100c1192c0 con 0x7f100c115800 2023-10-05T19:28:01.899 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:01.894+0000 7f1002ffd700 1 --2- 172.21.15.167:0/1282134332 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f100c115800 0x7f100c117be0 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7f0ff4009930 tx=0x7f0ff4009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ec419a4d5cf00761 server_cookie=0 in_seq=0 out_seq=0 2023-10-05T19:28:01.899 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:01.894+0000 7f100affd700 1 -- 172.21.15.167:0/1282134332 <== mon.0 v2:172.21.15.118:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f0ff400b860 con 0x7f100c115800 2023-10-05T19:28:01.899 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:01.894+0000 7f100affd700 1 -- 172.21.15.167:0/1282134332 <== mon.0 v2:172.21.15.118:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0ff400b9c0 con 0x7f100c115800 2023-10-05T19:28:01.900 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:01.894+0000 7f100affd700 1 -- 172.21.15.167:0/1282134332 <== mon.0 v2:172.21.15.118:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f0ff400bb50 con 0x7f100c115800 2023-10-05T19:28:01.901 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:01.894+0000 7f1010d7e700 1 -- 172.21.15.167:0/1282134332 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f100c115800 msgr2=0x7f100c117be0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-05T19:28:01.901 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:01.894+0000 7f1010d7e700 1 --2- 172.21.15.167:0/1282134332 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f100c115800 0x7f100c117be0 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7f0ff4009930 tx=0x7f0ff4009250 comp rx=0 tx=0).stop 2023-10-05T19:28:01.901 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:01.894+0000 7f1010d7e700 1 -- 172.21.15.167:0/1282134332 shutdown_connections 2023-10-05T19:28:01.901 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:01.894+0000 7f1010d7e700 1 --2- 172.21.15.167:0/1282134332 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f100c115800 0x7f100c117be0 unknown :-1 s=CLOSED pgs=117 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-05T19:28:01.901 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:01.894+0000 7f1010d7e700 1 -- 172.21.15.167:0/1282134332 >> 172.21.15.167:0/1282134332 conn(0x7f100c0695c0 msgr2=0x7f100c0699c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-05T19:28:01.901 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:01.894+0000 7f1010d7e700 1 -- 172.21.15.167:0/1282134332 shutdown_connections 2023-10-05T19:28:01.901 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:01.894+0000 7f1010d7e700 1 -- 172.21.15.167:0/1282134332 wait complete. 2023-10-05T19:28:01.902 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:01.898+0000 7f1010d7e700 1 Processor -- start 2023-10-05T19:28:01.902 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:01.898+0000 7f1010d7e700 1 -- start start 2023-10-05T19:28:01.902 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:01.898+0000 7f1010d7e700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f100c112aa0 0x7f100c11bcf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-05T19:28:01.902 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:01.898+0000 7f1010d7e700 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f100c115e90 con 0x7f100c112aa0 2023-10-05T19:28:01.903 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:01.898+0000 7f100b7fe700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f100c112aa0 0x7f100c11bcf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-05T19:28:01.903 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:01.898+0000 7f100b7fe700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f100c112aa0 0x7f100c11bcf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.118:3300/0 says I am v2:172.21.15.167:56006/0 (socket says 172.21.15.167:56006) 2023-10-05T19:28:01.903 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:01.898+0000 7f100b7fe700 1 -- 172.21.15.167:0/3734337590 learned_addr learned my addr 172.21.15.167:0/3734337590 (peer_addr_for_me v2:172.21.15.167:0/0) 2023-10-05T19:28:01.904 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:01.898+0000 7f100b7fe700 1 -- 172.21.15.167:0/3734337590 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0ff40094f0 con 0x7f100c112aa0 2023-10-05T19:28:01.904 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:01.898+0000 7f100b7fe700 1 --2- 172.21.15.167:0/3734337590 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f100c112aa0 0x7f100c11bcf0 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7f0ffc00c690 tx=0x7f0ffc007f40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-05T19:28:01.904 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:01.898+0000 7f1003fff700 1 -- 172.21.15.167:0/3734337590 <== mon.0 v2:172.21.15.118:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f0ffc005b90 con 0x7f100c112aa0 2023-10-05T19:28:01.904 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:01.898+0000 7f1003fff700 1 -- 172.21.15.167:0/3734337590 <== mon.0 v2:172.21.15.118:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0ffc005cf0 con 0x7f100c112aa0 2023-10-05T19:28:01.905 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:01.898+0000 7f1010d7e700 1 -- 172.21.15.167:0/3734337590 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f100c11c230 con 0x7f100c112aa0 2023-10-05T19:28:01.905 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:01.898+0000 7f1003fff700 1 -- 172.21.15.167:0/3734337590 <== mon.0 v2:172.21.15.118:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f0ffc0070d0 con 0x7f100c112aa0 2023-10-05T19:28:01.905 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:01.898+0000 7f1010d7e700 1 -- 172.21.15.167:0/3734337590 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f100c11c680 con 0x7f100c112aa0 2023-10-05T19:28:01.905 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:01.898+0000 7f1003fff700 1 -- 172.21.15.167:0/3734337590 <== mon.0 v2:172.21.15.118:3300/0 4 ==== mgrmap(e 5) v1 ==== 40329+0+0 (secure 0 0 0) 0x7f0ffc007280 con 0x7f100c112aa0 2023-10-05T19:28:01.905 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:01.898+0000 7f1010d7e700 1 -- 172.21.15.167:0/3734337590 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f100c11b900 con 0x7f100c112aa0 2023-10-05T19:28:01.906 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:01.902+0000 7f1003fff700 1 --2- 172.21.15.167:0/3734337590 >> [v2:172.21.15.167:6824/29889,v1:172.21.15.167:6825/29889] conn(0x7f0ff0032390 0x7f0ff0034840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-05T19:28:01.906 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:01.902+0000 7f1003fff700 1 -- 172.21.15.167:0/3734337590 <== mon.0 v2:172.21.15.118:3300/0 5 ==== osd_map(16..16 src has 1..16) v4 ==== 3721+0+0 (secure 0 0 0) 0x7f0ffc043130 con 0x7f100c112aa0 2023-10-05T19:28:01.906 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:01.902+0000 7f1002ffd700 1 --2- 172.21.15.167:0/3734337590 >> [v2:172.21.15.167:6824/29889,v1:172.21.15.167:6825/29889] conn(0x7f0ff0032390 0x7f0ff0034840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-05T19:28:01.907 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:01.902+0000 7f1002ffd700 1 --2- 172.21.15.167:0/3734337590 >> [v2:172.21.15.167:6824/29889,v1:172.21.15.167:6825/29889] conn(0x7f0ff0032390 0x7f0ff0034840 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f0ff4009930 tx=0x7f0ff4016000 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-05T19:28:01.915 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:01.910+0000 7f1003fff700 1 -- 172.21.15.167:0/3734337590 <== mon.0 v2:172.21.15.118:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166807 (secure 0 0 0) 0x7f0ffc00fc70 con 0x7f100c112aa0 2023-10-05T19:28:02.079 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:02.074+0000 7f1010d7e700 1 -- 172.21.15.167:0/3734337590 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "default.rgw.buckets.data", "pg_num": 64, "pgp_num": 64, "pool_type": "erasure", "erasure_code_profile": "client.0"} v 0) v1 -- 0x7f100c115850 con 0x7f100c112aa0 2023-10-05T19:28:03.775 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:03.770+0000 7f1003fff700 1 -- 172.21.15.167:0/3734337590 <== mon.0 v2:172.21.15.118:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "default.rgw.buckets.data", "pg_num": 64, "pgp_num": 64, "pool_type": "erasure", "erasure_code_profile": "client.0"}]=0 pool 'default.rgw.buckets.data' created v18) v1 ==== 227+0+0 (secure 0 0 0) 0x7f0ffc017020 con 0x7f100c112aa0 2023-10-05T19:28:03.781 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:03.774+0000 7f1010d7e700 1 -- 172.21.15.167:0/3734337590 >> [v2:172.21.15.167:6824/29889,v1:172.21.15.167:6825/29889] conn(0x7f0ff0032390 msgr2=0x7f0ff0034840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-05T19:28:03.782 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:03.774+0000 7f1010d7e700 1 --2- 172.21.15.167:0/3734337590 >> [v2:172.21.15.167:6824/29889,v1:172.21.15.167:6825/29889] conn(0x7f0ff0032390 0x7f0ff0034840 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f0ff4009930 tx=0x7f0ff4016000 comp rx=0 tx=0).stop 2023-10-05T19:28:03.782 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:03.774+0000 7f1010d7e700 1 -- 172.21.15.167:0/3734337590 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f100c112aa0 msgr2=0x7f100c11bcf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-05T19:28:03.782 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:03.774+0000 7f1010d7e700 1 --2- 172.21.15.167:0/3734337590 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f100c112aa0 0x7f100c11bcf0 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7f0ffc00c690 tx=0x7f0ffc007f40 comp rx=0 tx=0).stop 2023-10-05T19:28:03.782 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:03.774+0000 7f1010d7e700 1 -- 172.21.15.167:0/3734337590 shutdown_connections 2023-10-05T19:28:03.782 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:03.774+0000 7f1010d7e700 1 --2- 172.21.15.167:0/3734337590 >> [v2:172.21.15.167:6824/29889,v1:172.21.15.167:6825/29889] conn(0x7f0ff0032390 0x7f0ff0034840 secure :-1 s=CLOSED pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f0ff4009930 tx=0x7f0ff4016000 comp rx=0 tx=0).stop 2023-10-05T19:28:03.783 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:03.774+0000 7f1010d7e700 1 --2- 172.21.15.167:0/3734337590 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f100c112aa0 0x7f100c11bcf0 unknown :-1 s=CLOSED pgs=118 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-05T19:28:03.783 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:03.774+0000 7f1010d7e700 1 -- 172.21.15.167:0/3734337590 >> 172.21.15.167:0/3734337590 conn(0x7f100c0695c0 msgr2=0x7f100c116860 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-05T19:28:03.784 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:03.778+0000 7f1010d7e700 1 -- 172.21.15.167:0/3734337590 shutdown_connections 2023-10-05T19:28:03.784 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:03.778+0000 7f1010d7e700 1 -- 172.21.15.167:0/3734337590 wait complete. 2023-10-05T19:28:03.788 INFO:teuthology.orchestra.run.smithi167.stderr:pool 'default.rgw.buckets.data' created 2023-10-05T19:28:03.807 DEBUG:teuthology.orchestra.run.smithi167:> sudo ceph osd pool application enable default.rgw.buckets.data rgw --cluster ceph 2023-10-05T19:28:03.906 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:03.898+0000 7f2a9ac5d700 1 Processor -- start 2023-10-05T19:28:03.906 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:03.898+0000 7f2a9ac5d700 1 -- start start 2023-10-05T19:28:03.906 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:03.898+0000 7f2a9ac5d700 1 --2- >> v2:172.21.15.118:3300/0 conn(0x7f2a9410cf90 0x7f2a9410d360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-05T19:28:03.907 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:03.898+0000 7f2a9ac5d700 1 -- --> v1:172.21.15.118:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2a94118260 con 0x7f2a9410d990 2023-10-05T19:28:03.907 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:03.898+0000 7f2a9ac5d700 1 -- --> v2:172.21.15.118:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2a9404bb20 con 0x7f2a9410cf90 2023-10-05T19:28:03.907 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:03.902+0000 7f2a989f9700 1 --2- >> v2:172.21.15.118:3300/0 conn(0x7f2a9410cf90 0x7f2a9410d360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-05T19:28:03.907 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:03.902+0000 7f2a989f9700 1 --2- >> v2:172.21.15.118:3300/0 conn(0x7f2a9410cf90 0x7f2a9410d360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.118:3300/0 says I am v2:172.21.15.167:56010/0 (socket says 172.21.15.167:56010) 2023-10-05T19:28:03.907 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:03.902+0000 7f2a93fff700 1 --1- >> v1:172.21.15.118:6789/0 conn(0x7f2a9410d990 0x7f2a94116a50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.118:6789/0 says I am v1:172.21.15.167:50508/0 (socket says 172.21.15.167:50508) 2023-10-05T19:28:03.908 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:03.902+0000 7f2a989f9700 1 -- 172.21.15.167:0/502805233 learned_addr learned my addr 172.21.15.167:0/502805233 (peer_addr_for_me v2:172.21.15.167:0/0) 2023-10-05T19:28:03.910 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:03.902+0000 7f2a989f9700 1 -- 172.21.15.167:0/502805233 >> v1:172.21.15.118:6789/0 conn(0x7f2a9410d990 legacy=0x7f2a94116a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-05T19:28:03.910 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:03.902+0000 7f2a989f9700 1 -- 172.21.15.167:0/502805233 --> v2:172.21.15.118:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2a94119430 con 0x7f2a9410cf90 2023-10-05T19:28:03.910 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:03.902+0000 7f2a989f9700 1 --2- 172.21.15.167:0/502805233 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f2a9410cf90 0x7f2a9410d360 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7f2a7c009930 tx=0x7f2a7c009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f2b00949bba75877 server_cookie=0 in_seq=0 out_seq=0 2023-10-05T19:28:03.910 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:03.902+0000 7f2a937fe700 1 -- 172.21.15.167:0/502805233 <== mon.0 v2:172.21.15.118:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f2a7c00b860 con 0x7f2a9410cf90 2023-10-05T19:28:03.910 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:03.902+0000 7f2a937fe700 1 -- 172.21.15.167:0/502805233 <== mon.0 v2:172.21.15.118:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2a7c00b9c0 con 0x7f2a9410cf90 2023-10-05T19:28:03.911 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:03.902+0000 7f2a937fe700 1 -- 172.21.15.167:0/502805233 <== mon.0 v2:172.21.15.118:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f2a7c00bb50 con 0x7f2a9410cf90 2023-10-05T19:28:03.911 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:03.902+0000 7f2a9ac5d700 1 -- 172.21.15.167:0/502805233 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f2a9410cf90 msgr2=0x7f2a9410d360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-05T19:28:03.911 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:03.902+0000 7f2a9ac5d700 1 --2- 172.21.15.167:0/502805233 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f2a9410cf90 0x7f2a9410d360 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7f2a7c009930 tx=0x7f2a7c009250 comp rx=0 tx=0).stop 2023-10-05T19:28:03.911 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:03.902+0000 7f2a9ac5d700 1 -- 172.21.15.167:0/502805233 shutdown_connections 2023-10-05T19:28:03.911 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:03.902+0000 7f2a9ac5d700 1 --2- 172.21.15.167:0/502805233 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f2a9410cf90 0x7f2a9410d360 unknown :-1 s=CLOSED pgs=120 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-05T19:28:03.911 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:03.902+0000 7f2a9ac5d700 1 -- 172.21.15.167:0/502805233 >> 172.21.15.167:0/502805233 conn(0x7f2a94069780 msgr2=0x7f2a94069b80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-05T19:28:03.911 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:03.902+0000 7f2a9ac5d700 1 -- 172.21.15.167:0/502805233 shutdown_connections 2023-10-05T19:28:03.911 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:03.902+0000 7f2a9ac5d700 1 -- 172.21.15.167:0/502805233 wait complete. 2023-10-05T19:28:03.911 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:03.902+0000 7f2a9ac5d700 1 Processor -- start 2023-10-05T19:28:03.911 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:03.902+0000 7f2a9ac5d700 1 -- start start 2023-10-05T19:28:03.911 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:03.906+0000 7f2a9ac5d700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f2a9410cf90 0x7f2a9411be80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-05T19:28:03.911 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:03.906+0000 7f2a9ac5d700 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2a9410c160 con 0x7f2a9410cf90 2023-10-05T19:28:03.912 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:03.906+0000 7f2a989f9700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f2a9410cf90 0x7f2a9411be80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-05T19:28:03.912 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:03.906+0000 7f2a989f9700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f2a9410cf90 0x7f2a9411be80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.118:3300/0 says I am v2:172.21.15.167:56018/0 (socket says 172.21.15.167:56018) 2023-10-05T19:28:03.912 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:03.906+0000 7f2a989f9700 1 -- 172.21.15.167:0/3277619547 learned_addr learned my addr 172.21.15.167:0/3277619547 (peer_addr_for_me v2:172.21.15.167:0/0) 2023-10-05T19:28:03.913 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:03.906+0000 7f2a989f9700 1 -- 172.21.15.167:0/3277619547 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2a7c0094f0 con 0x7f2a9410cf90 2023-10-05T19:28:03.913 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:03.906+0000 7f2a989f9700 1 --2- 172.21.15.167:0/3277619547 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f2a9410cf90 0x7f2a9411be80 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7f2a7c009fd0 tx=0x7f2a7c016000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-05T19:28:03.913 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:03.906+0000 7f2a90ff9700 1 -- 172.21.15.167:0/3277619547 <== mon.0 v2:172.21.15.118:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f2a7c00bd40 con 0x7f2a9410cf90 2023-10-05T19:28:03.913 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:03.906+0000 7f2a9ac5d700 1 -- 172.21.15.167:0/3277619547 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2a9411c3c0 con 0x7f2a9410cf90 2023-10-05T19:28:03.913 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:03.906+0000 7f2a90ff9700 1 -- 172.21.15.167:0/3277619547 <== mon.0 v2:172.21.15.118:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2a7c020070 con 0x7f2a9410cf90 2023-10-05T19:28:03.913 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:03.906+0000 7f2a90ff9700 1 -- 172.21.15.167:0/3277619547 <== mon.0 v2:172.21.15.118:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f2a7c01b450 con 0x7f2a9410cf90 2023-10-05T19:28:03.913 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:03.906+0000 7f2a9ac5d700 1 -- 172.21.15.167:0/3277619547 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2a9411c810 con 0x7f2a9410cf90 2023-10-05T19:28:03.913 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:03.906+0000 7f2a9ac5d700 1 -- 172.21.15.167:0/3277619547 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2a9411ba50 con 0x7f2a9410cf90 2023-10-05T19:28:03.914 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:03.906+0000 7f2a90ff9700 1 -- 172.21.15.167:0/3277619547 <== mon.0 v2:172.21.15.118:3300/0 4 ==== mgrmap(e 5) v1 ==== 40329+0+0 (secure 0 0 0) 0x7f2a7c022030 con 0x7f2a9410cf90 2023-10-05T19:28:03.914 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:03.906+0000 7f2a90ff9700 1 --2- 172.21.15.167:0/3277619547 >> [v2:172.21.15.167:6824/29889,v1:172.21.15.167:6825/29889] conn(0x7f2a74032380 0x7f2a74034830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-05T19:28:03.914 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:03.906+0000 7f2a90ff9700 1 -- 172.21.15.167:0/3277619547 <== mon.0 v2:172.21.15.118:3300/0 5 ==== osd_map(18..18 src has 1..18) v4 ==== 4196+0+0 (secure 0 0 0) 0x7f2a7c045ca0 con 0x7f2a9410cf90 2023-10-05T19:28:03.914 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:03.906+0000 7f2a93fff700 1 --2- 172.21.15.167:0/3277619547 >> [v2:172.21.15.167:6824/29889,v1:172.21.15.167:6825/29889] conn(0x7f2a74032380 0x7f2a74034830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-05T19:28:03.915 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:03.910+0000 7f2a93fff700 1 --2- 172.21.15.167:0/3277619547 >> [v2:172.21.15.167:6824/29889,v1:172.21.15.167:6825/29889] conn(0x7f2a74032380 0x7f2a74034830 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f2a94064790 tx=0x7f2a84007b30 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-05T19:28:03.926 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:03.918+0000 7f2a90ff9700 1 -- 172.21.15.167:0/3277619547 <== mon.0 v2:172.21.15.118:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166807 (secure 0 0 0) 0x7f2a7c04a070 con 0x7f2a9410cf90 2023-10-05T19:28:04.087 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:04.082+0000 7f2a9ac5d700 1 -- 172.21.15.167:0/3277619547 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_command({"prefix": "osd pool application enable", "pool": "default.rgw.buckets.data", "app": "rgw"} v 0) v1 -- 0x7f2a9405ecb0 con 0x7f2a9410cf90 2023-10-05T19:28:04.778 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:04.770+0000 7f2a90ff9700 1 -- 172.21.15.167:0/3277619547 <== mon.0 v2:172.21.15.118:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool application enable", "pool": "default.rgw.buckets.data", "app": "rgw"}]=0 enabled application 'rgw' on pool 'default.rgw.buckets.data' v19) v1 ==== 185+0+0 (secure 0 0 0) 0x7f2a7c010740 con 0x7f2a9410cf90 2023-10-05T19:28:04.795 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:04.790+0000 7f2a9ac5d700 1 -- 172.21.15.167:0/3277619547 >> [v2:172.21.15.167:6824/29889,v1:172.21.15.167:6825/29889] conn(0x7f2a74032380 msgr2=0x7f2a74034830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-05T19:28:04.796 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:04.790+0000 7f2a9ac5d700 1 --2- 172.21.15.167:0/3277619547 >> [v2:172.21.15.167:6824/29889,v1:172.21.15.167:6825/29889] conn(0x7f2a74032380 0x7f2a74034830 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f2a94064790 tx=0x7f2a84007b30 comp rx=0 tx=0).stop 2023-10-05T19:28:04.796 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:04.790+0000 7f2a9ac5d700 1 -- 172.21.15.167:0/3277619547 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f2a9410cf90 msgr2=0x7f2a9411be80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-05T19:28:04.796 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:04.790+0000 7f2a9ac5d700 1 --2- 172.21.15.167:0/3277619547 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f2a9410cf90 0x7f2a9411be80 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7f2a7c009fd0 tx=0x7f2a7c016000 comp rx=0 tx=0).stop 2023-10-05T19:28:04.796 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:04.790+0000 7f2a9ac5d700 1 -- 172.21.15.167:0/3277619547 shutdown_connections 2023-10-05T19:28:04.796 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:04.790+0000 7f2a9ac5d700 1 --2- 172.21.15.167:0/3277619547 >> [v2:172.21.15.167:6824/29889,v1:172.21.15.167:6825/29889] conn(0x7f2a74032380 0x7f2a74034830 unknown :-1 s=CLOSED pgs=47 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-05T19:28:04.796 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:04.790+0000 7f2a9ac5d700 1 --2- 172.21.15.167:0/3277619547 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f2a9410cf90 0x7f2a9411be80 unknown :-1 s=CLOSED pgs=121 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-05T19:28:04.796 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:04.790+0000 7f2a9ac5d700 1 -- 172.21.15.167:0/3277619547 >> 172.21.15.167:0/3277619547 conn(0x7f2a94069780 msgr2=0x7f2a94111610 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-05T19:28:04.797 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:04.790+0000 7f2a9ac5d700 1 -- 172.21.15.167:0/3277619547 shutdown_connections 2023-10-05T19:28:04.797 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:04.790+0000 7f2a9ac5d700 1 -- 172.21.15.167:0/3277619547 wait complete. 2023-10-05T19:28:04.804 INFO:teuthology.orchestra.run.smithi167.stderr:enabled application 'rgw' on pool 'default.rgw.buckets.data' 2023-10-05T19:28:04.838 DEBUG:teuthology.orchestra.run.smithi167:> sudo ceph osd pool create default.rgw.buckets.index 64 64 --cluster ceph 2023-10-05T19:28:04.941 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:04.934+0000 7fdeba400700 1 Processor -- start 2023-10-05T19:28:04.941 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:04.934+0000 7fdeba400700 1 -- start start 2023-10-05T19:28:04.941 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:04.934+0000 7fdeba400700 1 --2- >> v2:172.21.15.118:3300/0 conn(0x7fdeb41158f0 0x7fdeb4117cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-05T19:28:04.942 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:04.934+0000 7fdeba400700 1 -- --> v1:172.21.15.118:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdeb4118210 con 0x7fdeb4112bc0 2023-10-05T19:28:04.942 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:04.934+0000 7fdeba400700 1 -- --> v2:172.21.15.118:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdeb40631f0 con 0x7fdeb41158f0 2023-10-05T19:28:04.942 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:04.934+0000 7fdeb8bfd700 1 --2- >> v2:172.21.15.118:3300/0 conn(0x7fdeb41158f0 0x7fdeb4117cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-05T19:28:04.942 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:04.934+0000 7fdeb93fe700 1 --1- >> v1:172.21.15.118:6789/0 conn(0x7fdeb4112bc0 0x7fdeb4112f90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.118:6789/0 says I am v1:172.21.15.167:34212/0 (socket says 172.21.15.167:34212) 2023-10-05T19:28:04.942 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:04.934+0000 7fdeb8bfd700 1 --2- >> v2:172.21.15.118:3300/0 conn(0x7fdeb41158f0 0x7fdeb4117cd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.118:3300/0 says I am v2:172.21.15.167:60680/0 (socket says 172.21.15.167:60680) 2023-10-05T19:28:04.942 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:04.934+0000 7fdeb93fe700 1 -- 172.21.15.167:0/1956636465 learned_addr learned my addr 172.21.15.167:0/1956636465 (peer_addr_for_me v1:172.21.15.167:0/0) 2023-10-05T19:28:04.943 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:04.934+0000 7fdeb8bfd700 1 -- 172.21.15.167:0/1956636465 >> v1:172.21.15.118:6789/0 conn(0x7fdeb4112bc0 legacy=0x7fdeb4112f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-05T19:28:04.943 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:04.934+0000 7fdeb8bfd700 1 -- 172.21.15.167:0/1956636465 --> v2:172.21.15.118:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdeb41193b0 con 0x7fdeb41158f0 2023-10-05T19:28:04.944 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:04.938+0000 7fdeb8bfd700 1 --2- 172.21.15.167:0/1956636465 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fdeb41158f0 0x7fdeb4117cd0 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7fdea4009860 tx=0x7fdea40091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=348ac16631141453 server_cookie=0 in_seq=0 out_seq=0 2023-10-05T19:28:04.944 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:04.938+0000 7fdeabfff700 1 -- 172.21.15.167:0/1956636465 <== mon.0 v2:172.21.15.118:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fdea400b680 con 0x7fdeb41158f0 2023-10-05T19:28:04.944 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:04.938+0000 7fdeabfff700 1 -- 172.21.15.167:0/1956636465 <== mon.0 v2:172.21.15.118:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdea400b7e0 con 0x7fdeb41158f0 2023-10-05T19:28:04.944 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:04.938+0000 7fdeabfff700 1 -- 172.21.15.167:0/1956636465 <== mon.0 v2:172.21.15.118:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fdea400b990 con 0x7fdeb41158f0 2023-10-05T19:28:04.944 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:04.938+0000 7fdeba400700 1 -- 172.21.15.167:0/1956636465 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fdeb41158f0 msgr2=0x7fdeb4117cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-05T19:28:04.944 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:04.938+0000 7fdeba400700 1 --2- 172.21.15.167:0/1956636465 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fdeb41158f0 0x7fdeb4117cd0 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7fdea4009860 tx=0x7fdea40091a0 comp rx=0 tx=0).stop 2023-10-05T19:28:04.944 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:04.938+0000 7fdeba400700 1 -- 172.21.15.167:0/1956636465 shutdown_connections 2023-10-05T19:28:04.944 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:04.938+0000 7fdeba400700 1 --2- 172.21.15.167:0/1956636465 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fdeb41158f0 0x7fdeb4117cd0 unknown :-1 s=CLOSED pgs=123 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-05T19:28:04.945 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:04.938+0000 7fdeba400700 1 -- 172.21.15.167:0/1956636465 >> 172.21.15.167:0/1956636465 conn(0x7fdeb40695f0 msgr2=0x7fdeb40699f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-05T19:28:04.945 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:04.938+0000 7fdeba400700 1 -- 172.21.15.167:0/1956636465 shutdown_connections 2023-10-05T19:28:04.945 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:04.938+0000 7fdeba400700 1 -- 172.21.15.167:0/1956636465 wait complete. 2023-10-05T19:28:04.946 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:04.938+0000 7fdeba400700 1 Processor -- start 2023-10-05T19:28:04.946 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:04.938+0000 7fdeba400700 1 -- start start 2023-10-05T19:28:04.946 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:04.938+0000 7fdeba400700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fdeb4112bc0 0x7fdeb411bcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-05T19:28:04.947 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:04.938+0000 7fdeba400700 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdeb4115ef0 con 0x7fdeb4112bc0 2023-10-05T19:28:04.947 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:04.938+0000 7fdeb93fe700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fdeb4112bc0 0x7fdeb411bcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-05T19:28:04.947 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:04.938+0000 7fdeb93fe700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fdeb4112bc0 0x7fdeb411bcd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.118:3300/0 says I am v2:172.21.15.167:60684/0 (socket says 172.21.15.167:60684) 2023-10-05T19:28:04.947 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:04.938+0000 7fdeb93fe700 1 -- 172.21.15.167:0/2502817027 learned_addr learned my addr 172.21.15.167:0/2502817027 (peer_addr_for_me v2:172.21.15.167:0/0) 2023-10-05T19:28:04.947 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:04.938+0000 7fdeb93fe700 1 -- 172.21.15.167:0/2502817027 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdea4009420 con 0x7fdeb4112bc0 2023-10-05T19:28:04.947 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:04.942+0000 7fdeb93fe700 1 --2- 172.21.15.167:0/2502817027 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fdeb4112bc0 0x7fdeb411bcd0 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7fdeac00cc30 tx=0x7fdeac001ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-05T19:28:04.947 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:04.942+0000 7fdea97fa700 1 -- 172.21.15.167:0/2502817027 <== mon.0 v2:172.21.15.118:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fdeac004c10 con 0x7fdeb4112bc0 2023-10-05T19:28:04.947 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:04.942+0000 7fdea97fa700 1 -- 172.21.15.167:0/2502817027 <== mon.0 v2:172.21.15.118:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdeac004d70 con 0x7fdeb4112bc0 2023-10-05T19:28:04.947 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:04.942+0000 7fdeba400700 1 -- 172.21.15.167:0/2502817027 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdeb411c210 con 0x7fdeb4112bc0 2023-10-05T19:28:04.947 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:04.942+0000 7fdea97fa700 1 -- 172.21.15.167:0/2502817027 <== mon.0 v2:172.21.15.118:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fdeac00aaa0 con 0x7fdeb4112bc0 2023-10-05T19:28:04.947 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:04.942+0000 7fdeba400700 1 -- 172.21.15.167:0/2502817027 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdeb411c6e0 con 0x7fdeb4112bc0 2023-10-05T19:28:04.948 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:04.942+0000 7fdea97fa700 1 -- 172.21.15.167:0/2502817027 <== mon.0 v2:172.21.15.118:3300/0 4 ==== mgrmap(e 5) v1 ==== 40329+0+0 (secure 0 0 0) 0x7fdeac00ac00 con 0x7fdeb4112bc0 2023-10-05T19:28:04.948 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:04.942+0000 7fdea97fa700 1 --2- 172.21.15.167:0/2502817027 >> [v2:172.21.15.167:6824/29889,v1:172.21.15.167:6825/29889] conn(0x7fde980323e0 0x7fde98034890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-05T19:28:04.949 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:04.942+0000 7fdea97fa700 1 -- 172.21.15.167:0/2502817027 <== mon.0 v2:172.21.15.118:3300/0 5 ==== osd_map(19..19 src has 1..19) v4 ==== 4207+0+0 (secure 0 0 0) 0x7fdeac043bc0 con 0x7fdeb4112bc0 2023-10-05T19:28:04.949 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:04.942+0000 7fdeb8bfd700 1 --2- 172.21.15.167:0/2502817027 >> [v2:172.21.15.167:6824/29889,v1:172.21.15.167:6825/29889] conn(0x7fde980323e0 0x7fde98034890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-05T19:28:04.949 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:04.942+0000 7fdeba400700 1 -- 172.21.15.167:0/2502817027 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdeb411b8e0 con 0x7fdeb4112bc0 2023-10-05T19:28:04.949 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:04.942+0000 7fdeb8bfd700 1 --2- 172.21.15.167:0/2502817027 >> [v2:172.21.15.167:6824/29889,v1:172.21.15.167:6825/29889] conn(0x7fde980323e0 0x7fde98034890 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7fdea4009860 tx=0x7fdea4016000 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-05T19:28:04.959 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:04.954+0000 7fdea97fa700 1 -- 172.21.15.167:0/2502817027 <== mon.0 v2:172.21.15.118:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166807 (secure 0 0 0) 0x7fdeac018070 con 0x7fdeb4112bc0 2023-10-05T19:28:05.119 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:05.114+0000 7fdeba400700 1 -- 172.21.15.167:0/2502817027 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "default.rgw.buckets.index", "pg_num": 64, "pgp_num": 64} v 0) v1 -- 0x7fdeb405ecb0 con 0x7fdeb4112bc0 2023-10-05T19:28:05.780 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:05.774+0000 7fdea97fa700 1 -- 172.21.15.167:0/2502817027 <== mon.0 v2:172.21.15.118:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "default.rgw.buckets.index", "pg_num": 64, "pgp_num": 64}]=0 pool 'default.rgw.buckets.index' created v20) v1 ==== 169+0+0 (secure 0 0 0) 0x7fdeac010300 con 0x7fdeb4112bc0 2023-10-05T19:28:05.787 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:05.782+0000 7fdeba400700 1 -- 172.21.15.167:0/2502817027 >> [v2:172.21.15.167:6824/29889,v1:172.21.15.167:6825/29889] conn(0x7fde980323e0 msgr2=0x7fde98034890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-05T19:28:05.787 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:05.782+0000 7fdeba400700 1 --2- 172.21.15.167:0/2502817027 >> [v2:172.21.15.167:6824/29889,v1:172.21.15.167:6825/29889] conn(0x7fde980323e0 0x7fde98034890 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7fdea4009860 tx=0x7fdea4016000 comp rx=0 tx=0).stop 2023-10-05T19:28:05.787 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:05.782+0000 7fdeba400700 1 -- 172.21.15.167:0/2502817027 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fdeb4112bc0 msgr2=0x7fdeb411bcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-05T19:28:05.788 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:05.782+0000 7fdeba400700 1 --2- 172.21.15.167:0/2502817027 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fdeb4112bc0 0x7fdeb411bcd0 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7fdeac00cc30 tx=0x7fdeac001ad0 comp rx=0 tx=0).stop 2023-10-05T19:28:05.788 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:05.782+0000 7fdeba400700 1 -- 172.21.15.167:0/2502817027 shutdown_connections 2023-10-05T19:28:05.788 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:05.782+0000 7fdeba400700 1 --2- 172.21.15.167:0/2502817027 >> [v2:172.21.15.167:6824/29889,v1:172.21.15.167:6825/29889] conn(0x7fde980323e0 0x7fde98034890 unknown :-1 s=CLOSED pgs=48 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-05T19:28:05.788 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:05.782+0000 7fdeba400700 1 --2- 172.21.15.167:0/2502817027 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fdeb4112bc0 0x7fdeb411bcd0 unknown :-1 s=CLOSED pgs=124 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-05T19:28:05.788 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:05.782+0000 7fdeba400700 1 -- 172.21.15.167:0/2502817027 >> 172.21.15.167:0/2502817027 conn(0x7fdeb40695f0 msgr2=0x7fdeb41168c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-05T19:28:05.788 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:05.782+0000 7fdeba400700 1 -- 172.21.15.167:0/2502817027 shutdown_connections 2023-10-05T19:28:05.788 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:05.782+0000 7fdeba400700 1 -- 172.21.15.167:0/2502817027 wait complete. 2023-10-05T19:28:05.800 INFO:teuthology.orchestra.run.smithi167.stderr:pool 'default.rgw.buckets.index' created 2023-10-05T19:28:05.812 DEBUG:teuthology.orchestra.run.smithi167:> sudo ceph osd pool application enable default.rgw.buckets.index rgw --cluster ceph 2023-10-05T19:28:05.917 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:05.910+0000 7f52949e5700 1 Processor -- start 2023-10-05T19:28:05.917 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:05.910+0000 7f52949e5700 1 -- start start 2023-10-05T19:28:05.917 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:05.910+0000 7f52949e5700 1 --2- >> v2:172.21.15.118:3300/0 conn(0x7f5290112bc0 0x7f5290112f90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-05T19:28:05.917 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:05.910+0000 7f52949e5700 1 -- --> v1:172.21.15.118:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f529010d5c0 con 0x7f52901134d0 2023-10-05T19:28:05.917 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:05.910+0000 7f52949e5700 1 -- --> v2:172.21.15.118:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f52900631f0 con 0x7f5290112bc0 2023-10-05T19:28:05.918 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:05.910+0000 7f528f7fe700 1 --2- >> v2:172.21.15.118:3300/0 conn(0x7f5290112bc0 0x7f5290112f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-05T19:28:05.918 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:05.910+0000 7f528f7fe700 1 --2- >> v2:172.21.15.118:3300/0 conn(0x7f5290112bc0 0x7f5290112f90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.118:3300/0 says I am v2:172.21.15.167:60692/0 (socket says 172.21.15.167:60692) 2023-10-05T19:28:05.918 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:05.910+0000 7f528f7fe700 1 -- 172.21.15.167:0/1854287712 learned_addr learned my addr 172.21.15.167:0/1854287712 (peer_addr_for_me v2:172.21.15.167:0/0) 2023-10-05T19:28:05.918 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:05.910+0000 7f528effd700 1 --1- >> v1:172.21.15.118:6789/0 conn(0x7f52901134d0 0x7f529010cd60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.118:6789/0 says I am v1:172.21.15.167:34226/0 (socket says 172.21.15.167:34226) 2023-10-05T19:28:05.918 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:05.914+0000 7f528f7fe700 1 -- 172.21.15.167:0/1854287712 >> v1:172.21.15.118:6789/0 conn(0x7f52901134d0 legacy=0x7f529010cd60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-05T19:28:05.919 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:05.914+0000 7f528f7fe700 1 -- 172.21.15.167:0/1854287712 --> v2:172.21.15.118:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f529010d7a0 con 0x7f5290112bc0 2023-10-05T19:28:05.919 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:05.914+0000 7f528f7fe700 1 --2- 172.21.15.167:0/1854287712 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f5290112bc0 0x7f5290112f90 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7f5280009930 tx=0x7f5280009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=28a3e2219c172f2b server_cookie=0 in_seq=0 out_seq=0 2023-10-05T19:28:05.920 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:05.914+0000 7f528e7fc700 1 -- 172.21.15.167:0/1854287712 <== mon.0 v2:172.21.15.118:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f528000b860 con 0x7f5290112bc0 2023-10-05T19:28:05.920 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:05.914+0000 7f528e7fc700 1 -- 172.21.15.167:0/1854287712 <== mon.0 v2:172.21.15.118:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f528000b9c0 con 0x7f5290112bc0 2023-10-05T19:28:05.920 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:05.914+0000 7f528e7fc700 1 -- 172.21.15.167:0/1854287712 <== mon.0 v2:172.21.15.118:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f528000bb50 con 0x7f5290112bc0 2023-10-05T19:28:05.920 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:05.914+0000 7f52949e5700 1 -- 172.21.15.167:0/1854287712 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f5290112bc0 msgr2=0x7f5290112f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-05T19:28:05.920 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:05.914+0000 7f52949e5700 1 --2- 172.21.15.167:0/1854287712 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f5290112bc0 0x7f5290112f90 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7f5280009930 tx=0x7f5280009250 comp rx=0 tx=0).stop 2023-10-05T19:28:05.920 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:05.914+0000 7f52949e5700 1 -- 172.21.15.167:0/1854287712 shutdown_connections 2023-10-05T19:28:05.920 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:05.914+0000 7f52949e5700 1 --2- 172.21.15.167:0/1854287712 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f5290112bc0 0x7f5290112f90 unknown :-1 s=CLOSED pgs=126 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-05T19:28:05.920 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:05.914+0000 7f52949e5700 1 -- 172.21.15.167:0/1854287712 >> 172.21.15.167:0/1854287712 conn(0x7f52900695f0 msgr2=0x7f52900699f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-05T19:28:05.920 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:05.914+0000 7f52949e5700 1 -- 172.21.15.167:0/1854287712 shutdown_connections 2023-10-05T19:28:05.920 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:05.914+0000 7f52949e5700 1 -- 172.21.15.167:0/1854287712 wait complete. 2023-10-05T19:28:05.921 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:05.914+0000 7f52949e5700 1 Processor -- start 2023-10-05T19:28:05.921 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:05.914+0000 7f52949e5700 1 -- start start 2023-10-05T19:28:05.921 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:05.914+0000 7f52949e5700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f5290112bc0 0x7f529011bc80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-05T19:28:05.922 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:05.914+0000 7f52949e5700 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f529010cc80 con 0x7f5290112bc0 2023-10-05T19:28:05.922 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:05.914+0000 7f528f7fe700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f5290112bc0 0x7f529011bc80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-05T19:28:05.922 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:05.914+0000 7f528f7fe700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f5290112bc0 0x7f529011bc80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.118:3300/0 says I am v2:172.21.15.167:60696/0 (socket says 172.21.15.167:60696) 2023-10-05T19:28:05.922 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:05.914+0000 7f528f7fe700 1 -- 172.21.15.167:0/1099506815 learned_addr learned my addr 172.21.15.167:0/1099506815 (peer_addr_for_me v2:172.21.15.167:0/0) 2023-10-05T19:28:05.923 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:05.918+0000 7f528f7fe700 1 -- 172.21.15.167:0/1099506815 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f52800094f0 con 0x7f5290112bc0 2023-10-05T19:28:05.923 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:05.918+0000 7f528f7fe700 1 --2- 172.21.15.167:0/1099506815 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f5290112bc0 0x7f529011bc80 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7f5280009fd0 tx=0x7f5280016000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-05T19:28:05.923 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:05.918+0000 7f52877fe700 1 -- 172.21.15.167:0/1099506815 <== mon.0 v2:172.21.15.118:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f528000bd40 con 0x7f5290112bc0 2023-10-05T19:28:05.923 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:05.918+0000 7f52877fe700 1 -- 172.21.15.167:0/1099506815 <== mon.0 v2:172.21.15.118:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5280020070 con 0x7f5290112bc0 2023-10-05T19:28:05.923 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:05.918+0000 7f52949e5700 1 -- 172.21.15.167:0/1099506815 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f529011c310 con 0x7f5290112bc0 2023-10-05T19:28:05.923 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:05.918+0000 7f52949e5700 1 -- 172.21.15.167:0/1099506815 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f529011c700 con 0x7f5290112bc0 2023-10-05T19:28:05.923 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:05.918+0000 7f52877fe700 1 -- 172.21.15.167:0/1099506815 <== mon.0 v2:172.21.15.118:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f528001b450 con 0x7f5290112bc0 2023-10-05T19:28:05.924 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:05.918+0000 7f52949e5700 1 -- 172.21.15.167:0/1099506815 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f52740050b0 con 0x7f5290112bc0 2023-10-05T19:28:05.924 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:05.918+0000 7f52877fe700 1 -- 172.21.15.167:0/1099506815 <== mon.0 v2:172.21.15.118:3300/0 4 ==== mgrmap(e 5) v1 ==== 40329+0+0 (secure 0 0 0) 0x7f5280022030 con 0x7f5290112bc0 2023-10-05T19:28:05.924 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:05.918+0000 7f52877fe700 1 --2- 172.21.15.167:0/1099506815 >> [v2:172.21.15.167:6824/29889,v1:172.21.15.167:6825/29889] conn(0x7f52700323d0 0x7f5270034880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-05T19:28:05.925 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:05.918+0000 7f52877fe700 1 -- 172.21.15.167:0/1099506815 <== mon.0 v2:172.21.15.118:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 4571+0+0 (secure 0 0 0) 0x7f5280045cf0 con 0x7f5290112bc0 2023-10-05T19:28:05.925 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:05.918+0000 7f528effd700 1 --2- 172.21.15.167:0/1099506815 >> [v2:172.21.15.167:6824/29889,v1:172.21.15.167:6825/29889] conn(0x7f52700323d0 0x7f5270034880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-05T19:28:05.925 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:05.918+0000 7f528effd700 1 --2- 172.21.15.167:0/1099506815 >> [v2:172.21.15.167:6824/29889,v1:172.21.15.167:6825/29889] conn(0x7f52700323d0 0x7f5270034880 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f529010e4d0 tx=0x7f5278007af0 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-05T19:28:05.934 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:05.930+0000 7f52877fe700 1 -- 172.21.15.167:0/1099506815 <== mon.0 v2:172.21.15.118:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166807 (secure 0 0 0) 0x7f5280006d70 con 0x7f5290112bc0 2023-10-05T19:28:06.100 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:06.094+0000 7f52949e5700 1 -- 172.21.15.167:0/1099506815 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_command({"prefix": "osd pool application enable", "pool": "default.rgw.buckets.index", "app": "rgw"} v 0) v1 -- 0x7f52740058e0 con 0x7f5290112bc0 2023-10-05T19:28:06.782 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:06.778+0000 7f52877fe700 1 -- 172.21.15.167:0/1099506815 <== mon.0 v2:172.21.15.118:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool application enable", "pool": "default.rgw.buckets.index", "app": "rgw"}]=0 enabled application 'rgw' on pool 'default.rgw.buckets.index' v21) v1 ==== 187+0+0 (secure 0 0 0) 0x7f528001a3d0 con 0x7f5290112bc0 2023-10-05T19:28:06.795 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:06.790+0000 7f52949e5700 1 -- 172.21.15.167:0/1099506815 >> [v2:172.21.15.167:6824/29889,v1:172.21.15.167:6825/29889] conn(0x7f52700323d0 msgr2=0x7f5270034880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-05T19:28:06.795 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:06.790+0000 7f52949e5700 1 --2- 172.21.15.167:0/1099506815 >> [v2:172.21.15.167:6824/29889,v1:172.21.15.167:6825/29889] conn(0x7f52700323d0 0x7f5270034880 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f529010e4d0 tx=0x7f5278007af0 comp rx=0 tx=0).stop 2023-10-05T19:28:06.796 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:06.790+0000 7f52949e5700 1 -- 172.21.15.167:0/1099506815 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f5290112bc0 msgr2=0x7f529011bc80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-05T19:28:06.796 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:06.790+0000 7f52949e5700 1 --2- 172.21.15.167:0/1099506815 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f5290112bc0 0x7f529011bc80 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7f5280009fd0 tx=0x7f5280016000 comp rx=0 tx=0).stop 2023-10-05T19:28:06.796 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:06.790+0000 7f52949e5700 1 -- 172.21.15.167:0/1099506815 shutdown_connections 2023-10-05T19:28:06.797 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:06.790+0000 7f52949e5700 1 --2- 172.21.15.167:0/1099506815 >> [v2:172.21.15.167:6824/29889,v1:172.21.15.167:6825/29889] conn(0x7f52700323d0 0x7f5270034880 unknown :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-05T19:28:06.797 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:06.790+0000 7f52949e5700 1 --2- 172.21.15.167:0/1099506815 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f5290112bc0 0x7f529011bc80 unknown :-1 s=CLOSED pgs=127 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-05T19:28:06.797 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:06.790+0000 7f52949e5700 1 -- 172.21.15.167:0/1099506815 >> 172.21.15.167:0/1099506815 conn(0x7f52900695f0 msgr2=0x7f5290117930 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-05T19:28:06.798 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:06.790+0000 7f52949e5700 1 -- 172.21.15.167:0/1099506815 shutdown_connections 2023-10-05T19:28:06.798 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:06.790+0000 7f52949e5700 1 -- 172.21.15.167:0/1099506815 wait complete. 2023-10-05T19:28:06.803 INFO:teuthology.orchestra.run.smithi167.stderr:enabled application 'rgw' on pool 'default.rgw.buckets.index' 2023-10-05T19:28:06.819 DEBUG:tasks.rgw:Pools created 2023-10-05T19:28:06.819 INFO:tasks.rgw:Starting rgw... 2023-10-05T19:28:06.819 INFO:tasks.rgw:rgw client.0 config is {} 2023-10-05T19:28:06.819 INFO:tasks.rgw:Using beast as radosgw frontend 2023-10-05T19:28:06.819 INFO:tasks.rgw.client.0:Restarting daemon 2023-10-05T19:28:06.820 DEBUG:teuthology.orchestra.run.smithi167:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term radosgw --rgw-frontends 'beast port=80' -n client.0 --cluster ceph -k /etc/ceph/ceph.client.0.keyring --log-file /var/log/ceph/rgw.ceph.client.0.log --rgw_ops_log_socket_path /home/ubuntu/cephtest/rgw.opslog.ceph.client.0.sock --foreground | sudo tee /var/log/ceph/rgw.ceph.client.0.stdout 2>&1 2023-10-05T19:28:06.822 INFO:tasks.rgw.client.0:Started 2023-10-05T19:28:06.822 INFO:tasks.rgw:Polling client.0 until it starts accepting connections on http://smithi167.front.sepia.ceph.com:80/ 2023-10-05T19:28:06.822 DEBUG:teuthology.orchestra.run.smithi167:> curl http://smithi167.front.sepia.ceph.com:80/ 2023-10-05T19:28:06.860 INFO:teuthology.orchestra.run.smithi167.stderr: % Total % Received % Xferd Average Speed Time Time Time Current 2023-10-05T19:28:06.860 INFO:teuthology.orchestra.run.smithi167.stderr: Dload Upload Total Spent Left Speed 2023-10-05T19:28:06.861 DEBUG:teuthology.orchestra.run:got remote process result: 7 2023-10-05T19:28:06.862 INFO:teuthology.orchestra.run.smithi167.stderr: 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 2023-10-05T19:28:06.862 INFO:teuthology.orchestra.run.smithi167.stderr:curl: (7) Failed to connect to smithi167.front.sepia.ceph.com port 80: Connection refused 2023-10-05T19:28:07.863 DEBUG:teuthology.orchestra.run.smithi167:> curl http://smithi167.front.sepia.ceph.com:80/ 2023-10-05T19:28:07.877 INFO:teuthology.orchestra.run.smithi167.stderr: % Total % Received % Xferd Average Speed Time Time Time Current 2023-10-05T19:28:07.877 INFO:teuthology.orchestra.run.smithi167.stderr: Dload Upload Total Spent Left Speed 2023-10-05T19:28:07.877 INFO:teuthology.orchestra.run.smithi167.stderr: 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 2023-10-05T19:28:07.877 INFO:teuthology.orchestra.run.smithi167.stderr:curl: (7) Failed to connect to smithi167.front.sepia.ceph.com port 80: Connection refused 2023-10-05T19:28:07.878 DEBUG:teuthology.orchestra.run:got remote process result: 7 2023-10-05T19:28:09.879 DEBUG:teuthology.orchestra.run.smithi167:> curl http://smithi167.front.sepia.ceph.com:80/ 2023-10-05T19:28:09.894 INFO:teuthology.orchestra.run.smithi167.stderr: % Total % Received % Xferd Average Speed Time Time Time Current 2023-10-05T19:28:09.894 INFO:teuthology.orchestra.run.smithi167.stderr: Dload Upload Total Spent Left Speed 2023-10-05T19:28:09.895 DEBUG:teuthology.orchestra.run:got remote process result: 7 2023-10-05T19:28:09.895 INFO:teuthology.orchestra.run.smithi167.stderr: 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 2023-10-05T19:28:09.895 INFO:teuthology.orchestra.run.smithi167.stderr:curl: (7) Failed to connect to smithi167.front.sepia.ceph.com port 80: Connection refused 2023-10-05T19:28:13.897 DEBUG:teuthology.orchestra.run.smithi167:> curl http://smithi167.front.sepia.ceph.com:80/ 2023-10-05T19:28:13.911 INFO:teuthology.orchestra.run.smithi167.stderr: % Total % Received % Xferd Average Speed Time Time Time Current 2023-10-05T19:28:13.911 INFO:teuthology.orchestra.run.smithi167.stderr: Dload Upload Total Spent Left Speed 2023-10-05T19:28:13.912 INFO:teuthology.orchestra.run.smithi167.stderr: 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 2023-10-05T19:28:13.912 INFO:teuthology.orchestra.run.smithi167.stderr:curl: (7) Failed to connect to smithi167.front.sepia.ceph.com port 80: Connection refused 2023-10-05T19:28:13.912 DEBUG:teuthology.orchestra.run:got remote process result: 7 2023-10-05T19:28:21.914 DEBUG:teuthology.orchestra.run.smithi167:> curl http://smithi167.front.sepia.ceph.com:80/ 2023-10-05T19:28:21.928 INFO:teuthology.orchestra.run.smithi167.stderr: % Total % Received % Xferd Average Speed Time Time Time Current 2023-10-05T19:28:21.928 INFO:teuthology.orchestra.run.smithi167.stderr: Dload Upload Total Spent Left Speed 2023-10-05T19:28:21.949 INFO:teuthology.orchestra.run.smithi167.stderr: 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 100 214 0 214 0 0 10190 0 --:--:-- --:--:-- --:--:-- 10190 2023-10-05T19:28:21.951 INFO:teuthology.orchestra.run.smithi167.stdout:anonymous 2023-10-05T19:28:21.951 INFO:teuthology.run_tasks:Running task radosgw-admin... 2023-10-05T19:28:23.493 DEBUG:tasks.radosgw_admin:config is: {'client.0': None} 2023-10-05T19:28:23.494 INFO:tasks.util.rgw:rgwadmin: client.0 : ['bucket', 'list'] 2023-10-05T19:28:23.494 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'ceph', 'bucket', 'list'] 2023-10-05T19:28:23.494 DEBUG:teuthology.orchestra.run.smithi167:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster ceph bucket list 2023-10-05T19:28:23.533 INFO:teuthology.orchestra.run.smithi167.stderr:ignoring --setuser ceph since I am not root 2023-10-05T19:28:23.533 INFO:teuthology.orchestra.run.smithi167.stderr:ignoring --setgroup ceph since I am not root 2023-10-05T19:28:23.534 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:23.526+0000 7fed20059e40 1 Processor -- start 2023-10-05T19:28:23.534 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:23.526+0000 7fed20059e40 1 -- start start 2023-10-05T19:28:23.534 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:23.530+0000 7fed20059e40 1 --2- >> v2:172.21.15.118:3300/0 conn(0x560ecbc83ff0 0x560ecbc843c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-05T19:28:23.534 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:23.530+0000 7fed20059e40 1 -- --> v1:172.21.15.118:6789/0 -- auth(proto 0 26 bytes epoch 0) v1 -- 0x560ecbbcec00 con 0x560ecbc84900 2023-10-05T19:28:23.534 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:23.530+0000 7fed20059e40 1 -- --> v2:172.21.15.118:3300/0 -- mon_getmap magic: 0 v1 -- 0x560ecbb8bee0 con 0x560ecbc83ff0 2023-10-05T19:28:23.535 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:23.530+0000 7fed1db8e700 1 --2- >> v2:172.21.15.118:3300/0 conn(0x560ecbc83ff0 0x560ecbc843c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-05T19:28:23.535 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:23.530+0000 7fed1d38d700 1 --1- >> v1:172.21.15.118:6789/0 conn(0x560ecbc84900 0x560ecbc853a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.118:6789/0 says I am v1:172.21.15.167:37558/0 (socket says 172.21.15.167:37558) 2023-10-05T19:28:23.535 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:23.530+0000 7fed1db8e700 1 --2- >> v2:172.21.15.118:3300/0 conn(0x560ecbc83ff0 0x560ecbc843c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.118:3300/0 says I am v2:172.21.15.167:50556/0 (socket says 172.21.15.167:50556) 2023-10-05T19:28:23.535 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:23.530+0000 7fed1d38d700 1 -- 172.21.15.167:0/744583879 learned_addr learned my addr 172.21.15.167:0/744583879 (peer_addr_for_me v1:172.21.15.167:0/0) 2023-10-05T19:28:23.536 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:23.530+0000 7fed1db8e700 1 -- 172.21.15.167:0/744583879 >> v1:172.21.15.118:6789/0 conn(0x560ecbc84900 legacy=0x560ecbc853a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-05T19:28:23.536 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:23.530+0000 7fed1cb8c700 1 -- 172.21.15.167:0/744583879 <== mon.0 v1:172.21.15.118:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 98923932 0 0) 0x560ecbbcec00 con 0x560ecbc84900 2023-10-05T19:28:23.536 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:23.530+0000 7fed1db8e700 1 -- 172.21.15.167:0/744583879 --> v2:172.21.15.118:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x560ecbc85ab0 con 0x560ecbc83ff0 2023-10-05T19:28:23.536 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:23.530+0000 7fed1db8e700 1 --2- 172.21.15.167:0/744583879 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x560ecbc83ff0 0x560ecbc843c0 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7fed1000a520 tx=0x7fed10005d40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ea065e1267fe6b09 server_cookie=0 in_seq=0 out_seq=0 2023-10-05T19:28:23.536 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:23.530+0000 7fed1cb8c700 1 -- 172.21.15.167:0/744583879 <== mon.0 v2:172.21.15.118:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fed1000f590 con 0x560ecbc83ff0 2023-10-05T19:28:23.536 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:23.530+0000 7fed1cb8c700 1 -- 172.21.15.167:0/744583879 <== mon.0 v2:172.21.15.118:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fed1000f6f0 con 0x560ecbc83ff0 2023-10-05T19:28:23.537 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:23.530+0000 7fed1cb8c700 1 -- 172.21.15.167:0/744583879 <== mon.0 v2:172.21.15.118:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fed1000f880 con 0x560ecbc83ff0 2023-10-05T19:28:23.537 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:23.530+0000 7fed20059e40 1 -- 172.21.15.167:0/744583879 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x560ecbc83ff0 msgr2=0x560ecbc843c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-05T19:28:23.537 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:23.530+0000 7fed20059e40 1 --2- 172.21.15.167:0/744583879 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x560ecbc83ff0 0x560ecbc843c0 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7fed1000a520 tx=0x7fed10005d40 comp rx=0 tx=0).stop 2023-10-05T19:28:23.537 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:23.530+0000 7fed20059e40 1 -- 172.21.15.167:0/744583879 shutdown_connections 2023-10-05T19:28:23.538 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:23.530+0000 7fed20059e40 1 --2- 172.21.15.167:0/744583879 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x560ecbc83ff0 0x560ecbc843c0 unknown :-1 s=CLOSED pgs=135 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-05T19:28:23.538 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:23.530+0000 7fed20059e40 1 -- 172.21.15.167:0/744583879 >> 172.21.15.167:0/744583879 conn(0x560ecbbf7840 msgr2=0x560ecbc824e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-05T19:28:23.538 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:23.530+0000 7fed20059e40 1 -- 172.21.15.167:0/744583879 shutdown_connections 2023-10-05T19:28:23.538 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:23.530+0000 7fed20059e40 1 -- 172.21.15.167:0/744583879 wait complete. 2023-10-05T19:28:23.539 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:23.530+0000 7fed20059e40 1 Processor -- start 2023-10-05T19:28:23.539 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:23.534+0000 7fed20059e40 1 -- start start 2023-10-05T19:28:23.539 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:23.534+0000 7fed20059e40 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x560ecbc83ff0 0x560ecbca2e60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-05T19:28:23.539 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:23.534+0000 7fed20059e40 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x560ecbb4fe30 con 0x560ecbc83ff0 2023-10-05T19:28:23.539 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:23.534+0000 7fed1db8e700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x560ecbc83ff0 0x560ecbca2e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-05T19:28:23.539 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:23.534+0000 7fed1db8e700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x560ecbc83ff0 0x560ecbca2e60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.118:3300/0 says I am v2:172.21.15.167:50566/0 (socket says 172.21.15.167:50566) 2023-10-05T19:28:23.539 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:23.534+0000 7fed1db8e700 1 -- 172.21.15.167:0/4279021675 learned_addr learned my addr 172.21.15.167:0/4279021675 (peer_addr_for_me v2:172.21.15.167:0/0) 2023-10-05T19:28:23.540 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:23.534+0000 7fed1db8e700 1 -- 172.21.15.167:0/4279021675 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fed1000a0e0 con 0x560ecbc83ff0 2023-10-05T19:28:23.541 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:23.534+0000 7fed1db8e700 1 --2- 172.21.15.167:0/4279021675 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x560ecbc83ff0 0x560ecbca2e60 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7fed10005b20 tx=0x7fed10014000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8e7dfa23a76d59d server_cookie=0 in_seq=0 out_seq=0 2023-10-05T19:28:23.541 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:23.534+0000 7fed177fe700 1 -- 172.21.15.167:0/4279021675 <== mon.0 v2:172.21.15.118:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fed1000f860 con 0x560ecbc83ff0 2023-10-05T19:28:23.541 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:23.534+0000 7fed177fe700 1 -- 172.21.15.167:0/4279021675 <== mon.0 v2:172.21.15.118:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fed1000f9c0 con 0x560ecbc83ff0 2023-10-05T19:28:23.541 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:23.534+0000 7fed177fe700 1 -- 172.21.15.167:0/4279021675 <== mon.0 v2:172.21.15.118:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fed1000fb70 con 0x560ecbc83ff0 2023-10-05T19:28:23.541 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:23.534+0000 7fed20059e40 1 -- 172.21.15.167:0/4279021675 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x560ecbc83ff0 msgr2=0x560ecbca2e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-05T19:28:23.541 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:23.534+0000 7fed20059e40 1 --2- 172.21.15.167:0/4279021675 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x560ecbc83ff0 0x560ecbca2e60 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7fed10005b20 tx=0x7fed10014000 comp rx=0 tx=0).stop 2023-10-05T19:28:23.541 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:23.534+0000 7fed20059e40 1 -- 172.21.15.167:0/4279021675 shutdown_connections 2023-10-05T19:28:23.541 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:23.534+0000 7fed20059e40 1 --2- 172.21.15.167:0/4279021675 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x560ecbc83ff0 0x560ecbca2e60 unknown :-1 s=CLOSED pgs=136 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-05T19:28:23.542 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:23.534+0000 7fed20059e40 1 -- 172.21.15.167:0/4279021675 >> 172.21.15.167:0/4279021675 conn(0x560ecbbf7840 msgr2=0x560ecbc9ea20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-05T19:28:23.542 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:23.534+0000 7fed20059e40 1 -- 172.21.15.167:0/4279021675 shutdown_connections 2023-10-05T19:28:23.542 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:23.534+0000 7fed20059e40 1 -- 172.21.15.167:0/4279021675 wait complete. 2023-10-05T19:28:23.542 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:23.534+0000 7fed20059e40 1 Processor -- start 2023-10-05T19:28:23.542 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:23.534+0000 7fed20059e40 1 -- start start 2023-10-05T19:28:23.543 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:23.534+0000 7fed20059e40 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x560ecbc83ff0 0x560ecbca9b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-05T19:28:23.543 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:23.534+0000 7fed20059e40 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x560ecbb22ed0 con 0x560ecbc83ff0 2023-10-05T19:28:23.543 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:23.534+0000 7fed1db8e700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x560ecbc83ff0 0x560ecbca9b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-05T19:28:23.543 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:23.538+0000 7fed1db8e700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x560ecbc83ff0 0x560ecbca9b60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.118:3300/0 says I am v2:172.21.15.167:50574/0 (socket says 172.21.15.167:50574) 2023-10-05T19:28:23.543 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:23.538+0000 7fed1db8e700 1 -- 172.21.15.167:0/704848384 learned_addr learned my addr 172.21.15.167:0/704848384 (peer_addr_for_me v2:172.21.15.167:0/0) 2023-10-05T19:28:23.543 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:23.538+0000 7fed1db8e700 1 -- 172.21.15.167:0/704848384 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x560ecbc85ab0 con 0x560ecbc83ff0 2023-10-05T19:28:23.543 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:23.538+0000 7fed1db8e700 1 --2- 172.21.15.167:0/704848384 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x560ecbc83ff0 0x560ecbca9b60 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x560ecbc9f540 tx=0x7fed1001d000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-05T19:28:23.544 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:23.538+0000 7fed15ffb700 1 -- 172.21.15.167:0/704848384 <== mon.0 v2:172.21.15.118:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fed1002f430 con 0x560ecbc83ff0 2023-10-05T19:28:23.544 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:23.538+0000 7fed20059e40 1 -- 172.21.15.167:0/704848384 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x560ecbcaa0a0 con 0x560ecbc83ff0 2023-10-05T19:28:23.544 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:23.538+0000 7fed15ffb700 1 -- 172.21.15.167:0/704848384 <== mon.0 v2:172.21.15.118:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fed1002f5b0 con 0x560ecbc83ff0 2023-10-05T19:28:23.544 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:23.538+0000 7fed15ffb700 1 -- 172.21.15.167:0/704848384 <== mon.0 v2:172.21.15.118:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fed1002f760 con 0x560ecbc83ff0 2023-10-05T19:28:23.544 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:23.538+0000 7fed20059e40 1 -- 172.21.15.167:0/704848384 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x560ecbcaa540 con 0x560ecbc83ff0 2023-10-05T19:28:23.545 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:23.538+0000 7fed20059e40 20 rados->read ofs=0 len=0 2023-10-05T19:28:23.545 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:23.538+0000 7fed15ffb700 1 -- 172.21.15.167:0/704848384 <== mon.0 v2:172.21.15.118:3300/0 4 ==== mgrmap(e 5) v1 ==== 40329+0+0 (secure 0 0 0) 0x7fed10010420 con 0x560ecbc83ff0 2023-10-05T19:28:23.546 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:23.538+0000 7fed15ffb700 1 --2- 172.21.15.167:0/704848384 >> [v2:172.21.15.167:6824/29889,v1:172.21.15.167:6825/29889] conn(0x7fed00030ca0 0x7fed00033150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-05T19:28:23.546 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:23.538+0000 7fed15ffb700 1 -- 172.21.15.167:0/704848384 <== mon.0 v2:172.21.15.118:3300/0 5 ==== osd_map(30..30 src has 1..30) v4 ==== 6057+0+0 (secure 0 0 0) 0x7fed1002b070 con 0x560ecbc83ff0 2023-10-05T19:28:23.546 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:23.542+0000 7fed20059e40 1 --2- 172.21.15.167:0/704848384 >> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] conn(0x560ecbc84900 0x560ecbcb5600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-05T19:28:23.546 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:23.542+0000 7fed20059e40 1 -- 172.21.15.167:0/704848384 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:1 5.12 5:49953fa1:::default.realm:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x560ecbcb5c70 con 0x560ecbc84900 2023-10-05T19:28:23.546 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:23.542+0000 7fed1d38d700 1 --2- 172.21.15.167:0/704848384 >> [v2:172.21.15.167:6824/29889,v1:172.21.15.167:6825/29889] conn(0x7fed00030ca0 0x7fed00033150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-05T19:28:23.547 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:23.542+0000 7fed1e38f700 1 --2- 172.21.15.167:0/704848384 >> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] conn(0x560ecbc84900 0x560ecbcb5600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-05T19:28:23.547 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:23.542+0000 7fed1d38d700 1 --2- 172.21.15.167:0/704848384 >> [v2:172.21.15.167:6824/29889,v1:172.21.15.167:6825/29889] conn(0x7fed00030ca0 0x7fed00033150 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7fed080011e0 tx=0x7fed080013a0 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-05T19:28:23.547 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:23.542+0000 7fed1e38f700 1 --2- 172.21.15.167:0/704848384 >> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] conn(0x560ecbc84900 0x560ecbcb5600 crc :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-05T19:28:23.548 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:23.542+0000 7fed1e38f700 1 -- 172.21.15.167:0/704848384 <== osd.1 v2:172.21.15.167:6800/30035 1 ==== osd_op_reply(1 default.realm [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 ==== 157+0+0 (crc 0 0 0) 0x7fed1800a500 con 0x560ecbc84900 2023-10-05T19:28:23.548 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:23.542+0000 7fed20059e40 20 rados_obj.operate() r=-2 bl.length=0 2023-10-05T19:28:23.548 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:23.542+0000 7fed20059e40 20 realm 2023-10-05T19:28:23.548 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:23.542+0000 7fed20059e40 20 rados->read ofs=0 len=0 2023-10-05T19:28:23.549 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:23.542+0000 7fed20059e40 1 -- 172.21.15.167:0/704848384 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:2 5.12 5:49953fa1:::default.realm:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x560ecbcb2c60 con 0x560ecbc84900 2023-10-05T19:28:23.550 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:23.542+0000 7fed1e38f700 1 -- 172.21.15.167:0/704848384 <== osd.1 v2:172.21.15.167:6800/30035 2 ==== osd_op_reply(2 default.realm [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 ==== 157+0+0 (crc 0 0 0) 0x7fed1800a500 con 0x560ecbc84900 2023-10-05T19:28:23.550 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:23.542+0000 7fed20059e40 20 rados_obj.operate() r=-2 bl.length=0 2023-10-05T19:28:23.550 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:23.542+0000 7fed20059e40 4 RGWPeriod::init failed to init realm id : (2) No such file or directory 2023-10-05T19:28:23.550 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:23.542+0000 7fed20059e40 20 rados->read ofs=0 len=0 2023-10-05T19:28:23.550 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:23.542+0000 7fed20059e40 1 -- 172.21.15.167:0/704848384 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:3 5.12 5:49953fa1:::default.realm:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x560ecbcb6750 con 0x560ecbc84900 2023-10-05T19:28:23.550 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:23.542+0000 7fed1e38f700 1 -- 172.21.15.167:0/704848384 <== osd.1 v2:172.21.15.167:6800/30035 3 ==== osd_op_reply(3 default.realm [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 ==== 157+0+0 (crc 0 0 0) 0x7fed1800a500 con 0x560ecbc84900 2023-10-05T19:28:23.550 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:23.542+0000 7fed20059e40 20 rados_obj.operate() r=-2 bl.length=0 2023-10-05T19:28:23.550 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:23.542+0000 7fed20059e40 20 rados->read ofs=0 len=0 2023-10-05T19:28:23.551 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:23.542+0000 7fed20059e40 1 -- 172.21.15.167:0/704848384 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:4 5.f 5:f43b6ece:::zone_names.default:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x560ecbcb6330 con 0x560ecbc84900 2023-10-05T19:28:23.551 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:23.542+0000 7fed1e38f700 1 -- 172.21.15.167:0/704848384 <== osd.1 v2:172.21.15.167:6800/30035 4 ==== osd_op_reply(4 zone_names.default [read 0~46 out=46b] v0'0 uv1 ondisk = 0) v8 ==== 162+0+46 (crc 0 0 0) 0x7fed1800a500 con 0x560ecbc84900 2023-10-05T19:28:23.551 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:23.542+0000 7fed20059e40 20 rados_obj.operate() r=0 bl.length=46 2023-10-05T19:28:23.551 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:23.542+0000 7fed20059e40 20 rados->read ofs=0 len=0 2023-10-05T19:28:23.551 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:23.542+0000 7fed20059e40 1 -- 172.21.15.167:0/704848384 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:5 5.1d 5:b8374eb4:::zone_info.9a9df965-6612-45d4-a45f-6e3c889bacc2:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x560ecbcb71c0 con 0x560ecbc84900 2023-10-05T19:28:23.552 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:23.546+0000 7fed1e38f700 1 -- 172.21.15.167:0/704848384 <== osd.1 v2:172.21.15.167:6800/30035 5 ==== osd_op_reply(5 zone_info.9a9df965-6612-45d4-a45f-6e3c889bacc2 [read 0~874 out=874b] v0'0 uv1 ondisk = 0) v8 ==== 190+0+874 (crc 0 0 0) 0x7fed1800a500 con 0x560ecbc84900 2023-10-05T19:28:23.552 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:23.546+0000 7fed20059e40 20 rados_obj.operate() r=0 bl.length=874 2023-10-05T19:28:23.552 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:23.546+0000 7fed20059e40 20 searching for the correct realm 2023-10-05T19:28:23.552 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:23.546+0000 7fed20059e40 1 --2- 172.21.15.167:0/704848384 >> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] conn(0x560ecbcb9010 0x560ecbcbb3f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-05T19:28:23.552 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:23.546+0000 7fed20059e40 1 -- 172.21.15.167:0/704848384 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:6 5.0 5:00000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x560ecbcbb930 con 0x560ecbcb9010 2023-10-05T19:28:23.552 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:23.546+0000 7fed1db8e700 1 --2- 172.21.15.167:0/704848384 >> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] conn(0x560ecbcb9010 0x560ecbcbb3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-05T19:28:23.552 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:23.546+0000 7fed1db8e700 1 --2- 172.21.15.167:0/704848384 >> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] conn(0x560ecbcb9010 0x560ecbcbb3f0 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-05T19:28:23.553 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:23.546+0000 7fed1db8e700 1 -- 172.21.15.167:0/704848384 <== osd.3 v2:172.21.15.167:6808/30036 1 ==== osd_op_reply(6 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fed10024400 con 0x560ecbcb9010 2023-10-05T19:28:23.553 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:23.546+0000 7fed1db8e700 1 --2- 172.21.15.167:0/704848384 >> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] conn(0x7fed1001bbe0 0x7fed100383b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-05T19:28:23.553 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:23.546+0000 7fed1db8e700 1 -- 172.21.15.167:0/704848384 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:7 5.10 5:08000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x560ecbcbb930 con 0x7fed1001bbe0 2023-10-05T19:28:23.554 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:23.546+0000 7fed1d38d700 1 --2- 172.21.15.167:0/704848384 >> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] conn(0x7fed1001bbe0 0x7fed100383b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-05T19:28:23.554 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:23.546+0000 7fed1d38d700 1 --2- 172.21.15.167:0/704848384 >> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] conn(0x7fed1001bbe0 0x7fed100383b0 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-05T19:28:23.554 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:23.550+0000 7fed1d38d700 1 -- 172.21.15.167:0/704848384 <== osd.2 v2:172.21.15.167:6816/30037 1 ==== osd_op_reply(7 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fed08006070 con 0x7fed1001bbe0 2023-10-05T19:28:23.554 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:23.550+0000 7fed1d38d700 1 --2- 172.21.15.167:0/704848384 >> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] conn(0x7fed0800d8e0 0x7fed0800c360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-05T19:28:23.554 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:23.550+0000 7fed1d38d700 1 -- 172.21.15.167:0/704848384 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:8 5.8 5:10000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x560ecbcbb930 con 0x7fed0800d8e0 2023-10-05T19:28:23.555 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:23.550+0000 7fed1e38f700 1 --2- 172.21.15.167:0/704848384 >> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] conn(0x7fed0800d8e0 0x7fed0800c360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-05T19:28:23.555 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:23.550+0000 7fed1e38f700 1 --2- 172.21.15.167:0/704848384 >> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] conn(0x7fed0800d8e0 0x7fed0800c360 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-05T19:28:23.556 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:23.550+0000 7fed1e38f700 1 -- 172.21.15.167:0/704848384 <== osd.0 v2:172.21.15.118:6800/27690 1 ==== osd_op_reply(8 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fed18014040 con 0x7fed0800d8e0 2023-10-05T19:28:23.557 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:23.550+0000 7fed1e38f700 1 -- 172.21.15.167:0/704848384 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:9 5.18 5:18000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x560ecbcbb930 con 0x560ecbc84900 2023-10-05T19:28:23.557 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:23.550+0000 7fed1e38f700 1 -- 172.21.15.167:0/704848384 <== osd.1 v2:172.21.15.167:6800/30035 6 ==== osd_op_reply(9 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fed18014040 con 0x560ecbc84900 2023-10-05T19:28:23.557 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:23.550+0000 7fed1e38f700 1 -- 172.21.15.167:0/704848384 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:10 5.4 5:20000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x560ecbcbb930 con 0x560ecbc84900 2023-10-05T19:28:23.558 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:23.554+0000 7fed1e38f700 1 -- 172.21.15.167:0/704848384 <== osd.1 v2:172.21.15.167:6800/30035 7 ==== osd_op_reply(10 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fed18014040 con 0x560ecbc84900 2023-10-05T19:28:23.558 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:23.554+0000 7fed1e38f700 1 -- 172.21.15.167:0/704848384 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:11 5.14 5:28000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x560ecbcbb930 con 0x7fed0800d8e0 2023-10-05T19:28:23.559 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:23.554+0000 7fed1e38f700 1 -- 172.21.15.167:0/704848384 <== osd.0 v2:172.21.15.118:6800/27690 2 ==== osd_op_reply(11 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fed18014040 con 0x7fed0800d8e0 2023-10-05T19:28:23.559 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:23.554+0000 7fed1e38f700 1 -- 172.21.15.167:0/704848384 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:12 5.c 5:30000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x560ecbcbb930 con 0x7fed0800d8e0 2023-10-05T19:28:23.560 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:23.554+0000 7fed1e38f700 1 -- 172.21.15.167:0/704848384 <== osd.0 v2:172.21.15.118:6800/27690 3 ==== osd_op_reply(12 [pgnls start_epoch 30 out=112b] v23'1 uv1 ondisk = 1) v8 ==== 144+0+112 (crc 0 0 0) 0x7fed18014040 con 0x7fed0800d8e0 2023-10-05T19:28:23.560 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:23.554+0000 7fed1e38f700 1 -- 172.21.15.167:0/704848384 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:13 5.1c 5:38000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x560ecbcbb930 con 0x7fed1001bbe0 2023-10-05T19:28:23.560 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:23.554+0000 7fed1d38d700 1 -- 172.21.15.167:0/704848384 <== osd.2 v2:172.21.15.167:6816/30037 2 ==== osd_op_reply(13 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fed08006070 con 0x7fed1001bbe0 2023-10-05T19:28:23.560 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:23.554+0000 7fed1d38d700 1 -- 172.21.15.167:0/704848384 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:14 5.2 5:40000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x560ecbcbb930 con 0x560ecbcb9010 2023-10-05T19:28:23.561 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:23.554+0000 7fed1db8e700 1 -- 172.21.15.167:0/704848384 <== osd.3 v2:172.21.15.167:6808/30036 2 ==== osd_op_reply(14 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fed10024400 con 0x560ecbcb9010 2023-10-05T19:28:23.561 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:23.554+0000 7fed1db8e700 1 -- 172.21.15.167:0/704848384 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:15 5.12 5:48000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x560ecbcbb930 con 0x560ecbc84900 2023-10-05T19:28:23.562 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:23.558+0000 7fed1e38f700 1 -- 172.21.15.167:0/704848384 <== osd.1 v2:172.21.15.167:6800/30035 8 ==== osd_op_reply(15 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fed18014040 con 0x560ecbc84900 2023-10-05T19:28:23.562 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:23.558+0000 7fed1e38f700 1 -- 172.21.15.167:0/704848384 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:16 5.a 5:50000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x560ecbcbb930 con 0x7fed0800d8e0 2023-10-05T19:28:23.563 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:23.558+0000 7fed1e38f700 1 -- 172.21.15.167:0/704848384 <== osd.0 v2:172.21.15.118:6800/27690 4 ==== osd_op_reply(16 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fed18022040 con 0x7fed0800d8e0 2023-10-05T19:28:23.563 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:23.558+0000 7fed1e38f700 1 -- 172.21.15.167:0/704848384 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:17 5.1a 5:58000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x560ecbcbb930 con 0x560ecbc84900 2023-10-05T19:28:23.563 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:23.558+0000 7fed1e38f700 1 -- 172.21.15.167:0/704848384 <== osd.1 v2:172.21.15.167:6800/30035 9 ==== osd_op_reply(17 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fed18022040 con 0x560ecbc84900 2023-10-05T19:28:23.563 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:23.558+0000 7fed1e38f700 1 -- 172.21.15.167:0/704848384 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:18 5.6 5:60000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x560ecbcbb930 con 0x7fed1001bbe0 2023-10-05T19:28:23.564 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:23.558+0000 7fed1d38d700 1 -- 172.21.15.167:0/704848384 <== osd.2 v2:172.21.15.167:6816/30037 3 ==== osd_op_reply(18 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fed08006070 con 0x7fed1001bbe0 2023-10-05T19:28:23.564 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:23.558+0000 7fed1d38d700 1 -- 172.21.15.167:0/704848384 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:19 5.16 5:68000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x560ecbcbb930 con 0x560ecbcb9010 2023-10-05T19:28:23.565 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:23.558+0000 7fed1db8e700 1 -- 172.21.15.167:0/704848384 <== osd.3 v2:172.21.15.167:6808/30036 3 ==== osd_op_reply(19 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fed10024400 con 0x560ecbcb9010 2023-10-05T19:28:23.565 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:23.558+0000 7fed1db8e700 1 -- 172.21.15.167:0/704848384 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:20 5.e 5:70000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x560ecbcbb930 con 0x560ecbcb9010 2023-10-05T19:28:23.565 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:23.558+0000 7fed1db8e700 1 -- 172.21.15.167:0/704848384 <== osd.3 v2:172.21.15.167:6808/30036 4 ==== osd_op_reply(20 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fed10024400 con 0x560ecbcb9010 2023-10-05T19:28:23.566 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:23.558+0000 7fed1db8e700 1 -- 172.21.15.167:0/704848384 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:21 5.1e 5:78000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x560ecbcbb930 con 0x7fed0800d8e0 2023-10-05T19:28:23.567 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:23.562+0000 7fed1e38f700 1 -- 172.21.15.167:0/704848384 <== osd.0 v2:172.21.15.118:6800/27690 5 ==== osd_op_reply(21 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fed18022040 con 0x7fed0800d8e0 2023-10-05T19:28:23.567 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:23.562+0000 7fed1e38f700 1 -- 172.21.15.167:0/704848384 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:22 5.1 5:80000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x560ecbcbb930 con 0x560ecbcb9010 2023-10-05T19:28:23.567 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:23.562+0000 7fed1db8e700 1 -- 172.21.15.167:0/704848384 <== osd.3 v2:172.21.15.167:6808/30036 5 ==== osd_op_reply(22 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fed10024400 con 0x560ecbcb9010 2023-10-05T19:28:23.567 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:23.562+0000 7fed1db8e700 1 -- 172.21.15.167:0/704848384 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:23 5.11 5:88000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x560ecbcbb930 con 0x560ecbcb9010 2023-10-05T19:28:23.568 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:23.562+0000 7fed1db8e700 1 -- 172.21.15.167:0/704848384 <== osd.3 v2:172.21.15.167:6808/30036 6 ==== osd_op_reply(23 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fed10024400 con 0x560ecbcb9010 2023-10-05T19:28:23.568 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:23.562+0000 7fed1db8e700 1 -- 172.21.15.167:0/704848384 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:24 5.9 5:90000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x560ecbcbb930 con 0x560ecbc84900 2023-10-05T19:28:23.569 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:23.562+0000 7fed1e38f700 1 -- 172.21.15.167:0/704848384 <== osd.1 v2:172.21.15.167:6800/30035 10 ==== osd_op_reply(24 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fed18022040 con 0x560ecbc84900 2023-10-05T19:28:23.569 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:23.562+0000 7fed1e38f700 1 -- 172.21.15.167:0/704848384 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:25 5.19 5:98000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x560ecbcbb930 con 0x560ecbc84900 2023-10-05T19:28:23.569 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:23.562+0000 7fed1e38f700 1 -- 172.21.15.167:0/704848384 <== osd.1 v2:172.21.15.167:6800/30035 11 ==== osd_op_reply(25 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fed18022040 con 0x560ecbc84900 2023-10-05T19:28:23.569 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:23.562+0000 7fed1e38f700 1 -- 172.21.15.167:0/704848384 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:26 5.5 5:a0000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x560ecbcbb930 con 0x7fed0800d8e0 2023-10-05T19:28:23.570 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:23.566+0000 7fed1e38f700 1 -- 172.21.15.167:0/704848384 <== osd.0 v2:172.21.15.118:6800/27690 6 ==== osd_op_reply(26 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fed18022040 con 0x7fed0800d8e0 2023-10-05T19:28:23.571 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:23.566+0000 7fed1e38f700 1 -- 172.21.15.167:0/704848384 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:27 5.15 5:a8000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x560ecbcbb930 con 0x560ecbcb9010 2023-10-05T19:28:23.571 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:23.566+0000 7fed1db8e700 1 -- 172.21.15.167:0/704848384 <== osd.3 v2:172.21.15.167:6808/30036 7 ==== osd_op_reply(27 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fed10024400 con 0x560ecbcb9010 2023-10-05T19:28:23.571 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:23.566+0000 7fed1db8e700 1 -- 172.21.15.167:0/704848384 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:28 5.d 5:b0000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x560ecbcbb930 con 0x7fed1001bbe0 2023-10-05T19:28:23.572 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:23.566+0000 7fed1d38d700 1 -- 172.21.15.167:0/704848384 <== osd.2 v2:172.21.15.167:6816/30037 4 ==== osd_op_reply(28 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fed08006070 con 0x7fed1001bbe0 2023-10-05T19:28:23.572 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:23.566+0000 7fed1d38d700 1 -- 172.21.15.167:0/704848384 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:29 5.1d 5:b8000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x560ecbcbb930 con 0x560ecbc84900 2023-10-05T19:28:23.573 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:23.566+0000 7fed1e38f700 1 -- 172.21.15.167:0/704848384 <== osd.1 v2:172.21.15.167:6800/30035 12 ==== osd_op_reply(29 [pgnls start_epoch 30 out=107b] v23'1 uv1 ondisk = 1) v8 ==== 144+0+107 (crc 0 0 0) 0x7fed18022040 con 0x560ecbc84900 2023-10-05T19:28:23.573 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:23.566+0000 7fed1e38f700 1 -- 172.21.15.167:0/704848384 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:30 5.3 5:c0000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x560ecbcbb930 con 0x7fed0800d8e0 2023-10-05T19:28:23.574 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:23.570+0000 7fed1e38f700 1 -- 172.21.15.167:0/704848384 <== osd.0 v2:172.21.15.118:6800/27690 7 ==== osd_op_reply(30 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fed18022040 con 0x7fed0800d8e0 2023-10-05T19:28:23.574 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:23.570+0000 7fed1e38f700 1 -- 172.21.15.167:0/704848384 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:31 5.13 5:c8000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x560ecbcbb930 con 0x560ecbcb9010 2023-10-05T19:28:23.575 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:23.570+0000 7fed1db8e700 1 -- 172.21.15.167:0/704848384 <== osd.3 v2:172.21.15.167:6808/30036 8 ==== osd_op_reply(31 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fed1004f040 con 0x560ecbcb9010 2023-10-05T19:28:23.575 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:23.570+0000 7fed1db8e700 1 -- 172.21.15.167:0/704848384 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:32 5.b 5:d0000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x560ecbcbb930 con 0x7fed0800d8e0 2023-10-05T19:28:23.576 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:23.570+0000 7fed1e38f700 1 -- 172.21.15.167:0/704848384 <== osd.0 v2:172.21.15.118:6800/27690 8 ==== osd_op_reply(32 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fed18022040 con 0x7fed0800d8e0 2023-10-05T19:28:23.576 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:23.570+0000 7fed1e38f700 1 -- 172.21.15.167:0/704848384 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:33 5.1b 5:d8000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x560ecbcbb930 con 0x7fed1001bbe0 2023-10-05T19:28:23.576 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:23.570+0000 7fed1d38d700 1 -- 172.21.15.167:0/704848384 <== osd.2 v2:172.21.15.167:6816/30037 5 ==== osd_op_reply(33 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fed08006070 con 0x7fed1001bbe0 2023-10-05T19:28:23.577 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:23.570+0000 7fed1d38d700 1 -- 172.21.15.167:0/704848384 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:34 5.7 5:e0000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x560ecbcbb930 con 0x560ecbcb9010 2023-10-05T19:28:23.577 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:23.570+0000 7fed1db8e700 1 -- 172.21.15.167:0/704848384 <== osd.3 v2:172.21.15.167:6808/30036 9 ==== osd_op_reply(34 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fed1004f040 con 0x560ecbcb9010 2023-10-05T19:28:23.577 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:23.570+0000 7fed1db8e700 1 -- 172.21.15.167:0/704848384 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:35 5.17 5:e8000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x560ecbcbb930 con 0x7fed0800d8e0 2023-10-05T19:28:23.578 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:23.574+0000 7fed1e38f700 1 -- 172.21.15.167:0/704848384 <== osd.0 v2:172.21.15.118:6800/27690 9 ==== osd_op_reply(35 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fed18022040 con 0x7fed0800d8e0 2023-10-05T19:28:23.579 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:23.574+0000 7fed1e38f700 1 -- 172.21.15.167:0/704848384 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:36 5.f 5:f0000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x560ecbcbb930 con 0x560ecbc84900 2023-10-05T19:28:23.579 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:23.574+0000 7fed1e38f700 1 -- 172.21.15.167:0/704848384 <== osd.1 v2:172.21.15.167:6800/30035 13 ==== osd_op_reply(36 [pgnls start_epoch 30 out=115b] v23'2 uv2 ondisk = 1) v8 ==== 144+0+115 (crc 0 0 0) 0x7fed1802f080 con 0x560ecbc84900 2023-10-05T19:28:23.579 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:23.574+0000 7fed1e38f700 1 -- 172.21.15.167:0/704848384 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:37 5.1f 5:f8000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x560ecbcbb930 con 0x7fed1001bbe0 2023-10-05T19:28:23.580 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:23.574+0000 7fed1d38d700 1 -- 172.21.15.167:0/704848384 <== osd.2 v2:172.21.15.167:6816/30037 6 ==== osd_op_reply(37 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fed08006070 con 0x7fed1001bbe0 2023-10-05T19:28:23.581 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:23.574+0000 7fed20059e40 20 RGWRados::pool_iterate: got zonegroup_info.4f1f719d-9648-417b-b10c-e5d939825419 2023-10-05T19:28:23.581 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:23.574+0000 7fed20059e40 20 RGWRados::pool_iterate: got zone_info.9a9df965-6612-45d4-a45f-6e3c889bacc2 2023-10-05T19:28:23.581 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:23.574+0000 7fed20059e40 20 RGWRados::pool_iterate: got zone_names.default 2023-10-05T19:28:23.581 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:23.574+0000 7fed20059e40 20 RGWRados::pool_iterate: got zonegroups_names.default 2023-10-05T19:28:23.581 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:23.574+0000 7fed20059e40 20 rados->read ofs=0 len=0 2023-10-05T19:28:23.581 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:23.574+0000 7fed20059e40 1 -- 172.21.15.167:0/704848384 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:38 5.12 5:49953fa1:::default.realm:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x560ecbcb7dc0 con 0x560ecbc84900 2023-10-05T19:28:23.582 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:23.574+0000 7fed1e38f700 1 -- 172.21.15.167:0/704848384 <== osd.1 v2:172.21.15.167:6800/30035 14 ==== osd_op_reply(38 default.realm [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 ==== 157+0+0 (crc 0 0 0) 0x7fed1802f080 con 0x560ecbc84900 2023-10-05T19:28:23.582 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:23.574+0000 7fed20059e40 20 rados_obj.operate() r=-2 bl.length=0 2023-10-05T19:28:23.582 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:23.574+0000 7fed20059e40 20 rados->read ofs=0 len=0 2023-10-05T19:28:23.582 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:23.574+0000 7fed20059e40 1 -- 172.21.15.167:0/704848384 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:39 5.f 5:f4c53578:::zonegroups_names.default:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x560ecbcb8740 con 0x560ecbc84900 2023-10-05T19:28:23.582 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:23.574+0000 7fed1e38f700 1 -- 172.21.15.167:0/704848384 <== osd.1 v2:172.21.15.167:6800/30035 15 ==== osd_op_reply(39 zonegroups_names.default [read 0~46 out=46b] v0'0 uv2 ondisk = 0) v8 ==== 168+0+46 (crc 0 0 0) 0x7fed1802f080 con 0x560ecbc84900 2023-10-05T19:28:23.582 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:23.574+0000 7fed20059e40 20 rados_obj.operate() r=0 bl.length=46 2023-10-05T19:28:23.582 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:23.578+0000 7fed20059e40 20 rados->read ofs=0 len=0 2023-10-05T19:28:23.583 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:23.578+0000 7fed20059e40 1 -- 172.21.15.167:0/704848384 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:40 5.c 5:30d299a5:::zonegroup_info.4f1f719d-9648-417b-b10c-e5d939825419:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x560ecbcb79b0 con 0x7fed0800d8e0 2023-10-05T19:28:23.583 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:23.578+0000 7fed1e38f700 1 -- 172.21.15.167:0/704848384 <== osd.0 v2:172.21.15.118:6800/27690 10 ==== osd_op_reply(40 zonegroup_info.4f1f719d-9648-417b-b10c-e5d939825419 [read 0~362 out=362b] v0'0 uv1 ondisk = 0) v8 ==== 195+0+362 (crc 0 0 0) 0x7fed1802f080 con 0x7fed0800d8e0 2023-10-05T19:28:23.583 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:23.578+0000 7fed20059e40 20 rados_obj.operate() r=0 bl.length=362 2023-10-05T19:28:23.583 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:23.578+0000 7fed20059e40 20 rados->read ofs=0 len=0 2023-10-05T19:28:23.583 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:23.578+0000 7fed20059e40 1 -- 172.21.15.167:0/704848384 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:41 5.13 5:cbbf6c60:::converted:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x560ecbcbc1d0 con 0x560ecbcb9010 2023-10-05T19:28:23.584 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:23.578+0000 7fed1db8e700 1 -- 172.21.15.167:0/704848384 <== osd.3 v2:172.21.15.167:6808/30036 10 ==== osd_op_reply(41 converted [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 ==== 153+0+0 (crc 0 0 0) 0x7fed1004f040 con 0x560ecbcb9010 2023-10-05T19:28:23.584 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:23.578+0000 7fed20059e40 20 rados_obj.operate() r=-2 bl.length=0 2023-10-05T19:28:23.584 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:23.578+0000 7fed20059e40 20 rados->read ofs=0 len=0 2023-10-05T19:28:23.585 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:23.578+0000 7fed20059e40 1 -- 172.21.15.167:0/704848384 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:42 5.12 5:49953fa1:::default.realm:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x560ecbcbc580 con 0x560ecbc84900 2023-10-05T19:28:23.585 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:23.578+0000 7fed1e38f700 1 -- 172.21.15.167:0/704848384 <== osd.1 v2:172.21.15.167:6800/30035 16 ==== osd_op_reply(42 default.realm [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 ==== 157+0+0 (crc 0 0 0) 0x7fed1802f080 con 0x560ecbc84900 2023-10-05T19:28:23.585 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:23.578+0000 7fed20059e40 20 rados_obj.operate() r=-2 bl.length=0 2023-10-05T19:28:23.585 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:23.578+0000 7fed20059e40 20 rados->read ofs=0 len=0 2023-10-05T19:28:23.585 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:23.578+0000 7fed20059e40 1 -- 172.21.15.167:0/704848384 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:43 5.f 5:f4c53578:::zonegroups_names.default:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x560ecbcbc580 con 0x560ecbc84900 2023-10-05T19:28:23.585 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:23.578+0000 7fed1e38f700 1 -- 172.21.15.167:0/704848384 <== osd.1 v2:172.21.15.167:6800/30035 17 ==== osd_op_reply(43 zonegroups_names.default [read 0~46 out=46b] v0'0 uv2 ondisk = 0) v8 ==== 168+0+46 (crc 0 0 0) 0x7fed1802f080 con 0x560ecbc84900 2023-10-05T19:28:23.585 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:23.578+0000 7fed20059e40 20 rados_obj.operate() r=0 bl.length=46 2023-10-05T19:28:23.586 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:23.578+0000 7fed20059e40 1 -- 172.21.15.167:0/704848384 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:44 5.0 5:00000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x560ecbcbd590 con 0x560ecbcb9010 2023-10-05T19:28:23.588 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:23.582+0000 7fed1db8e700 1 -- 172.21.15.167:0/704848384 <== osd.3 v2:172.21.15.167:6808/30036 11 ==== osd_op_reply(44 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fed1004f040 con 0x560ecbcb9010 2023-10-05T19:28:23.588 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:23.582+0000 7fed1db8e700 1 -- 172.21.15.167:0/704848384 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:45 5.10 5:08000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x560ecbcbd590 con 0x7fed1001bbe0 2023-10-05T19:28:23.589 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:23.582+0000 7fed1d38d700 1 -- 172.21.15.167:0/704848384 <== osd.2 v2:172.21.15.167:6816/30037 7 ==== osd_op_reply(45 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fed08006070 con 0x7fed1001bbe0 2023-10-05T19:28:23.589 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:23.582+0000 7fed1d38d700 1 -- 172.21.15.167:0/704848384 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:46 5.8 5:10000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x560ecbcbd590 con 0x7fed0800d8e0 2023-10-05T19:28:23.590 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:23.582+0000 7fed1e38f700 1 -- 172.21.15.167:0/704848384 <== osd.0 v2:172.21.15.118:6800/27690 11 ==== osd_op_reply(46 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fed1802f080 con 0x7fed0800d8e0 2023-10-05T19:28:23.590 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:23.582+0000 7fed1e38f700 1 -- 172.21.15.167:0/704848384 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:47 5.18 5:18000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x560ecbcb79b0 con 0x560ecbc84900 2023-10-05T19:28:23.591 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:23.586+0000 7fed1e38f700 1 -- 172.21.15.167:0/704848384 <== osd.1 v2:172.21.15.167:6800/30035 18 ==== osd_op_reply(47 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fed1802f080 con 0x560ecbc84900 2023-10-05T19:28:23.591 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:23.586+0000 7fed1e38f700 1 -- 172.21.15.167:0/704848384 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:48 5.4 5:20000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x560ecbcb79b0 con 0x560ecbc84900 2023-10-05T19:28:23.591 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:23.586+0000 7fed1e38f700 1 -- 172.21.15.167:0/704848384 <== osd.1 v2:172.21.15.167:6800/30035 19 ==== osd_op_reply(48 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fed1802f080 con 0x560ecbc84900 2023-10-05T19:28:23.591 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:23.586+0000 7fed1e38f700 1 -- 172.21.15.167:0/704848384 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:49 5.14 5:28000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x560ecbcb79b0 con 0x7fed0800d8e0 2023-10-05T19:28:23.592 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:23.586+0000 7fed1e38f700 1 -- 172.21.15.167:0/704848384 <== osd.0 v2:172.21.15.118:6800/27690 12 ==== osd_op_reply(49 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fed1802f080 con 0x7fed0800d8e0 2023-10-05T19:28:23.592 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:23.586+0000 7fed1e38f700 1 -- 172.21.15.167:0/704848384 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:50 5.c 5:30000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x560ecbcb79b0 con 0x7fed0800d8e0 2023-10-05T19:28:23.593 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:23.586+0000 7fed1e38f700 1 -- 172.21.15.167:0/704848384 <== osd.0 v2:172.21.15.118:6800/27690 13 ==== osd_op_reply(50 [pgnls start_epoch 30 out=112b] v23'1 uv1 ondisk = 1) v8 ==== 144+0+112 (crc 0 0 0) 0x7fed1802f080 con 0x7fed0800d8e0 2023-10-05T19:28:23.593 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:23.586+0000 7fed1e38f700 1 -- 172.21.15.167:0/704848384 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:51 5.1c 5:38000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x560ecbcb79b0 con 0x7fed1001bbe0 2023-10-05T19:28:23.594 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:23.586+0000 7fed1d38d700 1 -- 172.21.15.167:0/704848384 <== osd.2 v2:172.21.15.167:6816/30037 8 ==== osd_op_reply(51 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fed08006070 con 0x7fed1001bbe0 2023-10-05T19:28:23.594 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:23.586+0000 7fed1d38d700 1 -- 172.21.15.167:0/704848384 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:52 5.2 5:40000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x560ecbcb79b0 con 0x560ecbcb9010 2023-10-05T19:28:23.594 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:23.590+0000 7fed1db8e700 1 -- 172.21.15.167:0/704848384 <== osd.3 v2:172.21.15.167:6808/30036 12 ==== osd_op_reply(52 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fed1004f040 con 0x560ecbcb9010 2023-10-05T19:28:23.594 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:23.590+0000 7fed1db8e700 1 -- 172.21.15.167:0/704848384 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:53 5.12 5:48000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x560ecbcb79b0 con 0x560ecbc84900 2023-10-05T19:28:23.595 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:23.590+0000 7fed1e38f700 1 -- 172.21.15.167:0/704848384 <== osd.1 v2:172.21.15.167:6800/30035 20 ==== osd_op_reply(53 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fed1802f080 con 0x560ecbc84900 2023-10-05T19:28:23.595 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:23.590+0000 7fed1e38f700 1 -- 172.21.15.167:0/704848384 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:54 5.a 5:50000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x560ecbcb79b0 con 0x7fed0800d8e0 2023-10-05T19:28:23.596 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:23.590+0000 7fed1e38f700 1 -- 172.21.15.167:0/704848384 <== osd.0 v2:172.21.15.118:6800/27690 14 ==== osd_op_reply(54 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fed1802f080 con 0x7fed0800d8e0 2023-10-05T19:28:23.596 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:23.590+0000 7fed1e38f700 1 -- 172.21.15.167:0/704848384 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:55 5.1a 5:58000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x560ecbcb79b0 con 0x560ecbc84900 2023-10-05T19:28:23.597 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:23.590+0000 7fed1e38f700 1 -- 172.21.15.167:0/704848384 <== osd.1 v2:172.21.15.167:6800/30035 21 ==== osd_op_reply(55 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fed1802f080 con 0x560ecbc84900 2023-10-05T19:28:23.597 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:23.590+0000 7fed1e38f700 1 -- 172.21.15.167:0/704848384 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:56 5.6 5:60000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x560ecbcb79b0 con 0x7fed1001bbe0 2023-10-05T19:28:23.597 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:23.590+0000 7fed1d38d700 1 -- 172.21.15.167:0/704848384 <== osd.2 v2:172.21.15.167:6816/30037 9 ==== osd_op_reply(56 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fed08006070 con 0x7fed1001bbe0 2023-10-05T19:28:23.598 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:23.590+0000 7fed1d38d700 1 -- 172.21.15.167:0/704848384 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:57 5.16 5:68000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x560ecbcb79b0 con 0x560ecbcb9010 2023-10-05T19:28:23.598 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:23.594+0000 7fed1db8e700 1 -- 172.21.15.167:0/704848384 <== osd.3 v2:172.21.15.167:6808/30036 13 ==== osd_op_reply(57 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fed1004f040 con 0x560ecbcb9010 2023-10-05T19:28:23.598 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:23.594+0000 7fed1db8e700 1 -- 172.21.15.167:0/704848384 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:58 5.e 5:70000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x560ecbcb79b0 con 0x560ecbcb9010 2023-10-05T19:28:23.599 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:23.594+0000 7fed1db8e700 1 -- 172.21.15.167:0/704848384 <== osd.3 v2:172.21.15.167:6808/30036 14 ==== osd_op_reply(58 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fed1004f040 con 0x560ecbcb9010 2023-10-05T19:28:23.599 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:23.594+0000 7fed1db8e700 1 -- 172.21.15.167:0/704848384 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:59 5.1e 5:78000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x560ecbcb79b0 con 0x7fed0800d8e0 2023-10-05T19:28:23.601 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:23.594+0000 7fed1e38f700 1 -- 172.21.15.167:0/704848384 <== osd.0 v2:172.21.15.118:6800/27690 15 ==== osd_op_reply(59 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fed1802f080 con 0x7fed0800d8e0 2023-10-05T19:28:23.601 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:23.594+0000 7fed1e38f700 1 -- 172.21.15.167:0/704848384 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:60 5.1 5:80000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x560ecbcb79b0 con 0x560ecbcb9010 2023-10-05T19:28:23.601 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:23.594+0000 7fed1db8e700 1 -- 172.21.15.167:0/704848384 <== osd.3 v2:172.21.15.167:6808/30036 15 ==== osd_op_reply(60 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fed1004f040 con 0x560ecbcb9010 2023-10-05T19:28:23.601 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:23.594+0000 7fed1db8e700 1 -- 172.21.15.167:0/704848384 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:61 5.11 5:88000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x560ecbcb79b0 con 0x560ecbcb9010 2023-10-05T19:28:23.602 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:23.598+0000 7fed1db8e700 1 -- 172.21.15.167:0/704848384 <== osd.3 v2:172.21.15.167:6808/30036 16 ==== osd_op_reply(61 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fed1004f040 con 0x560ecbcb9010 2023-10-05T19:28:23.602 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:23.598+0000 7fed1db8e700 1 -- 172.21.15.167:0/704848384 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:62 5.9 5:90000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x560ecbcb79b0 con 0x560ecbc84900 2023-10-05T19:28:23.603 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:23.598+0000 7fed1e38f700 1 -- 172.21.15.167:0/704848384 <== osd.1 v2:172.21.15.167:6800/30035 22 ==== osd_op_reply(62 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fed1802f080 con 0x560ecbc84900 2023-10-05T19:28:23.603 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:23.598+0000 7fed1e38f700 1 -- 172.21.15.167:0/704848384 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:63 5.19 5:98000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x560ecbcb79b0 con 0x560ecbc84900 2023-10-05T19:28:23.604 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:23.598+0000 7fed1e38f700 1 -- 172.21.15.167:0/704848384 <== osd.1 v2:172.21.15.167:6800/30035 23 ==== osd_op_reply(63 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fed1802f080 con 0x560ecbc84900 2023-10-05T19:28:23.604 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:23.598+0000 7fed1e38f700 1 -- 172.21.15.167:0/704848384 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:64 5.5 5:a0000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x560ecbcb79b0 con 0x7fed0800d8e0 2023-10-05T19:28:23.605 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:23.598+0000 7fed1e38f700 1 -- 172.21.15.167:0/704848384 <== osd.0 v2:172.21.15.118:6800/27690 16 ==== osd_op_reply(64 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fed1802f080 con 0x7fed0800d8e0 2023-10-05T19:28:23.605 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:23.598+0000 7fed1e38f700 1 -- 172.21.15.167:0/704848384 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:65 5.15 5:a8000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x560ecbcb79b0 con 0x560ecbcb9010 2023-10-05T19:28:23.606 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:23.598+0000 7fed1db8e700 1 -- 172.21.15.167:0/704848384 <== osd.3 v2:172.21.15.167:6808/30036 17 ==== osd_op_reply(65 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fed1004f040 con 0x560ecbcb9010 2023-10-05T19:28:23.606 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:23.598+0000 7fed1db8e700 1 -- 172.21.15.167:0/704848384 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:66 5.d 5:b0000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x560ecbcb79b0 con 0x7fed1001bbe0 2023-10-05T19:28:23.606 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:23.602+0000 7fed1d38d700 1 -- 172.21.15.167:0/704848384 <== osd.2 v2:172.21.15.167:6816/30037 10 ==== osd_op_reply(66 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fed08006070 con 0x7fed1001bbe0 2023-10-05T19:28:23.607 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:23.602+0000 7fed1d38d700 1 -- 172.21.15.167:0/704848384 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:67 5.1d 5:b8000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x560ecbcb79b0 con 0x560ecbc84900 2023-10-05T19:28:23.607 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:23.602+0000 7fed1e38f700 1 -- 172.21.15.167:0/704848384 <== osd.1 v2:172.21.15.167:6800/30035 24 ==== osd_op_reply(67 [pgnls start_epoch 30 out=107b] v23'1 uv1 ondisk = 1) v8 ==== 144+0+107 (crc 0 0 0) 0x7fed1802f080 con 0x560ecbc84900 2023-10-05T19:28:23.607 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:23.602+0000 7fed1e38f700 1 -- 172.21.15.167:0/704848384 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:68 5.3 5:c0000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x560ecbcb79b0 con 0x7fed0800d8e0 2023-10-05T19:28:23.609 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:23.602+0000 7fed1e38f700 1 -- 172.21.15.167:0/704848384 <== osd.0 v2:172.21.15.118:6800/27690 17 ==== osd_op_reply(68 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fed1802f080 con 0x7fed0800d8e0 2023-10-05T19:28:23.609 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:23.602+0000 7fed1e38f700 1 -- 172.21.15.167:0/704848384 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:69 5.13 5:c8000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x560ecbcb79b0 con 0x560ecbcb9010 2023-10-05T19:28:23.609 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:23.602+0000 7fed1db8e700 1 -- 172.21.15.167:0/704848384 <== osd.3 v2:172.21.15.167:6808/30036 18 ==== osd_op_reply(69 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fed1004f040 con 0x560ecbcb9010 2023-10-05T19:28:23.609 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:23.602+0000 7fed1db8e700 1 -- 172.21.15.167:0/704848384 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:70 5.b 5:d0000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x560ecbcb79b0 con 0x7fed0800d8e0 2023-10-05T19:28:23.610 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:23.606+0000 7fed1e38f700 1 -- 172.21.15.167:0/704848384 <== osd.0 v2:172.21.15.118:6800/27690 18 ==== osd_op_reply(70 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fed1802f080 con 0x7fed0800d8e0 2023-10-05T19:28:23.610 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:23.606+0000 7fed1e38f700 1 -- 172.21.15.167:0/704848384 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:71 5.1b 5:d8000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x560ecbcb79b0 con 0x7fed1001bbe0 2023-10-05T19:28:23.611 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:23.606+0000 7fed1d38d700 1 -- 172.21.15.167:0/704848384 <== osd.2 v2:172.21.15.167:6816/30037 11 ==== osd_op_reply(71 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fed08006070 con 0x7fed1001bbe0 2023-10-05T19:28:23.611 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:23.606+0000 7fed1d38d700 1 -- 172.21.15.167:0/704848384 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:72 5.7 5:e0000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x560ecbcb79b0 con 0x560ecbcb9010 2023-10-05T19:28:23.612 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:23.606+0000 7fed1db8e700 1 -- 172.21.15.167:0/704848384 <== osd.3 v2:172.21.15.167:6808/30036 19 ==== osd_op_reply(72 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fed1004f040 con 0x560ecbcb9010 2023-10-05T19:28:23.612 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:23.606+0000 7fed1db8e700 1 -- 172.21.15.167:0/704848384 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:73 5.17 5:e8000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x560ecbcb79b0 con 0x7fed0800d8e0 2023-10-05T19:28:23.613 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:23.606+0000 7fed1e38f700 1 -- 172.21.15.167:0/704848384 <== osd.0 v2:172.21.15.118:6800/27690 19 ==== osd_op_reply(73 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fed1802f080 con 0x7fed0800d8e0 2023-10-05T19:28:23.613 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:23.606+0000 7fed1e38f700 1 -- 172.21.15.167:0/704848384 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:74 5.f 5:f0000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x560ecbcb79b0 con 0x560ecbc84900 2023-10-05T19:28:23.614 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:23.610+0000 7fed1e38f700 1 -- 172.21.15.167:0/704848384 <== osd.1 v2:172.21.15.167:6800/30035 25 ==== osd_op_reply(74 [pgnls start_epoch 30 out=115b] v23'2 uv2 ondisk = 1) v8 ==== 144+0+115 (crc 0 0 0) 0x7fed1802f080 con 0x560ecbc84900 2023-10-05T19:28:23.614 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:23.610+0000 7fed1e38f700 1 -- 172.21.15.167:0/704848384 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:75 5.1f 5:f8000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x560ecbcb79b0 con 0x7fed1001bbe0 2023-10-05T19:28:23.615 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:23.610+0000 7fed1d38d700 1 -- 172.21.15.167:0/704848384 <== osd.2 v2:172.21.15.167:6816/30037 12 ==== osd_op_reply(75 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fed08006070 con 0x7fed1001bbe0 2023-10-05T19:28:23.615 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:23.610+0000 7fed20059e40 20 RGWRados::pool_iterate: got zonegroup_info.4f1f719d-9648-417b-b10c-e5d939825419 2023-10-05T19:28:23.615 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:23.610+0000 7fed20059e40 20 RGWRados::pool_iterate: got zone_info.9a9df965-6612-45d4-a45f-6e3c889bacc2 2023-10-05T19:28:23.615 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:23.610+0000 7fed20059e40 20 RGWRados::pool_iterate: got zone_names.default 2023-10-05T19:28:23.615 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:23.610+0000 7fed20059e40 20 RGWRados::pool_iterate: got zonegroups_names.default 2023-10-05T19:28:23.615 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:23.610+0000 7fed20059e40 20 rados->read ofs=0 len=0 2023-10-05T19:28:23.616 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:23.610+0000 7fed20059e40 1 -- 172.21.15.167:0/704848384 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:76 5.f 5:f43b6ece:::zone_names.default:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x560ecbcbc580 con 0x560ecbc84900 2023-10-05T19:28:23.616 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:23.610+0000 7fed1e38f700 1 -- 172.21.15.167:0/704848384 <== osd.1 v2:172.21.15.167:6800/30035 26 ==== osd_op_reply(76 zone_names.default [read 0~46 out=46b] v0'0 uv1 ondisk = 0) v8 ==== 162+0+46 (crc 0 0 0) 0x7fed1802f080 con 0x560ecbc84900 2023-10-05T19:28:23.617 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:23.610+0000 7fed20059e40 20 rados_obj.operate() r=0 bl.length=46 2023-10-05T19:28:23.617 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:23.610+0000 7fed20059e40 20 rados->read ofs=0 len=0 2023-10-05T19:28:23.617 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:23.610+0000 7fed20059e40 1 -- 172.21.15.167:0/704848384 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:77 5.1d 5:b8374eb4:::zone_info.9a9df965-6612-45d4-a45f-6e3c889bacc2:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x560ecbcbc930 con 0x560ecbc84900 2023-10-05T19:28:23.617 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:23.610+0000 7fed1e38f700 1 -- 172.21.15.167:0/704848384 <== osd.1 v2:172.21.15.167:6800/30035 27 ==== osd_op_reply(77 zone_info.9a9df965-6612-45d4-a45f-6e3c889bacc2 [read 0~874 out=874b] v0'0 uv1 ondisk = 0) v8 ==== 190+0+874 (crc 0 0 0) 0x7fed1802f080 con 0x560ecbc84900 2023-10-05T19:28:23.617 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:23.610+0000 7fed20059e40 20 rados_obj.operate() r=0 bl.length=874 2023-10-05T19:28:23.617 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:23.610+0000 7fed20059e40 20 rados->read ofs=0 len=0 2023-10-05T19:28:23.617 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:23.610+0000 7fed20059e40 1 -- 172.21.15.167:0/704848384 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:78 5.f 5:f4c53578:::zonegroups_names.default:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x560ecbcbce20 con 0x560ecbc84900 2023-10-05T19:28:23.618 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:23.610+0000 7fed1e38f700 1 -- 172.21.15.167:0/704848384 <== osd.1 v2:172.21.15.167:6800/30035 28 ==== osd_op_reply(78 zonegroups_names.default [read 0~46 out=46b] v0'0 uv2 ondisk = 0) v8 ==== 168+0+46 (crc 0 0 0) 0x7fed1802f080 con 0x560ecbc84900 2023-10-05T19:28:23.618 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:23.610+0000 7fed20059e40 20 rados_obj.operate() r=0 bl.length=46 2023-10-05T19:28:23.618 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:23.610+0000 7fed20059e40 20 rados->read ofs=0 len=0 2023-10-05T19:28:23.618 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:23.610+0000 7fed20059e40 1 -- 172.21.15.167:0/704848384 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:79 5.c 5:30d299a5:::zonegroup_info.4f1f719d-9648-417b-b10c-e5d939825419:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x560ecbcbce20 con 0x7fed0800d8e0 2023-10-05T19:28:23.618 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:23.614+0000 7fed1e38f700 1 -- 172.21.15.167:0/704848384 <== osd.0 v2:172.21.15.118:6800/27690 20 ==== osd_op_reply(79 zonegroup_info.4f1f719d-9648-417b-b10c-e5d939825419 [read 0~362 out=362b] v0'0 uv1 ondisk = 0) v8 ==== 195+0+362 (crc 0 0 0) 0x7fed1802f080 con 0x7fed0800d8e0 2023-10-05T19:28:23.619 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:23.614+0000 7fed20059e40 20 rados_obj.operate() r=0 bl.length=362 2023-10-05T19:28:23.619 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:23.614+0000 7fed20059e40 20 rados->read ofs=0 len=0 2023-10-05T19:28:23.619 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:23.614+0000 7fed20059e40 1 -- 172.21.15.167:0/704848384 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:80 5.3 5:c0ad8cbd:::region_map:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x560ecbcbce20 con 0x7fed0800d8e0 2023-10-05T19:28:23.619 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:23.614+0000 7fed1e38f700 1 -- 172.21.15.167:0/704848384 <== osd.0 v2:172.21.15.118:6800/27690 21 ==== osd_op_reply(80 region_map [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 ==== 154+0+0 (crc 0 0 0) 0x7fed18050040 con 0x7fed0800d8e0 2023-10-05T19:28:23.620 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:23.614+0000 7fed20059e40 20 rados_obj.operate() r=-2 bl.length=0 2023-10-05T19:28:23.620 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:23.614+0000 7fed20059e40 20 zone default found 2023-10-05T19:28:23.620 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:23.614+0000 7fed20059e40 4 Realm: () 2023-10-05T19:28:23.620 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:23.614+0000 7fed20059e40 4 ZoneGroup: default (4f1f719d-9648-417b-b10c-e5d939825419) 2023-10-05T19:28:23.620 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:23.614+0000 7fed20059e40 4 Zone: default (9a9df965-6612-45d4-a45f-6e3c889bacc2) 2023-10-05T19:28:23.620 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:23.614+0000 7fed20059e40 10 cannot find current period zonegroup using local zonegroup configuration 2023-10-05T19:28:23.620 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:23.614+0000 7fed20059e40 20 zonegroup default 2023-10-05T19:28:23.620 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:23.614+0000 7fed20059e40 20 rados->read ofs=0 len=0 2023-10-05T19:28:23.620 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:23.614+0000 7fed20059e40 1 -- 172.21.15.167:0/704848384 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:81 5.3 5:c52100b6:::period_config.default:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x560ecbcbce20 con 0x7fed0800d8e0 2023-10-05T19:28:23.620 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:23.614+0000 7fed1e38f700 1 -- 172.21.15.167:0/704848384 <== osd.0 v2:172.21.15.118:6800/27690 22 ==== osd_op_reply(81 period_config.default [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 ==== 165+0+0 (crc 0 0 0) 0x7fed18050040 con 0x7fed0800d8e0 2023-10-05T19:28:23.620 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:23.614+0000 7fed20059e40 20 rados_obj.operate() r=-2 bl.length=0 2023-10-05T19:28:23.621 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:23.614+0000 7fed20059e40 20 rados->read ofs=0 len=0 2023-10-05T19:28:23.621 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:23.614+0000 7fed20059e40 1 -- 172.21.15.167:0/704848384 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:82 6.2 6:4347d321:::bucket.sync-source-hints.:head [call version.read in=11b,read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x560ecbcbe7a0 con 0x560ecbcb9010 2023-10-05T19:28:23.622 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:23.614+0000 7fed1db8e700 1 -- 172.21.15.167:0/704848384 <== osd.3 v2:172.21.15.167:6808/30036 20 ==== osd_op_reply(82 bucket.sync-source-hints. [call,read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 ==== 211+0+0 (crc 0 0 0) 0x7fed1004f040 con 0x560ecbcb9010 2023-10-05T19:28:23.622 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:23.614+0000 7fed20059e40 20 rados_obj.operate() r=-2 bl.length=0 2023-10-05T19:28:23.622 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:23.614+0000 7fed20059e40 20 rados->read ofs=0 len=0 2023-10-05T19:28:23.622 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:23.614+0000 7fed20059e40 1 -- 172.21.15.167:0/704848384 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:83 6.b 6:d467b91b:::bucket.sync-target-hints.:head [call version.read in=11b,read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x560ecbcbeb50 con 0x560ecbcb9010 2023-10-05T19:28:23.622 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:23.614+0000 7fed1db8e700 1 -- 172.21.15.167:0/704848384 <== osd.3 v2:172.21.15.167:6808/30036 21 ==== osd_op_reply(83 bucket.sync-target-hints. [call,read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 ==== 211+0+0 (crc 0 0 0) 0x7fed1004f040 con 0x560ecbcb9010 2023-10-05T19:28:23.622 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:23.614+0000 7fed20059e40 20 rados_obj.operate() r=-2 bl.length=0 2023-10-05T19:28:23.622 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:23.614+0000 7fed20059e40 20 started sync module instance, tier type = 2023-10-05T19:28:23.622 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:23.614+0000 7fed20059e40 20 started zone id=9a9df965-6612-45d4-a45f-6e3c889bacc2 (name=default) with tier type = 2023-10-05T19:28:23.622 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:23.614+0000 7fed20059e40 1 -- 172.21.15.167:0/704848384 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:84 7.1f 7:f95f44c2:::notify.0:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x560ecbcbde60 con 0x7fed1001bbe0 2023-10-05T19:28:23.624 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:23.618+0000 7fed1d38d700 1 -- 172.21.15.167:0/704848384 <== osd.2 v2:172.21.15.167:6816/30037 13 ==== osd_op_reply(84 notify.0 [create] v30'3 uv3 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7fed08006070 con 0x7fed1001bbe0 2023-10-05T19:28:23.624 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:23.618+0000 7fed20059e40 1 -- 172.21.15.167:0/704848384 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:85 7.1f 7:f95f44c2:::notify.0:head [watch watch cookie 94621548667488] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x560ecbcbda90 con 0x7fed1001bbe0 2023-10-05T19:28:23.624 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:23.618+0000 7fed20059e40 1 -- 172.21.15.167:0/704848384 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:86 7.0 7:05bf5b68:::notify.1:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x560ecbcbf940 con 0x560ecbc84900 2023-10-05T19:28:23.627 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:23.622+0000 7fed1d38d700 1 -- 172.21.15.167:0/704848384 <== osd.2 v2:172.21.15.167:6816/30037 14 ==== osd_op_reply(85 notify.0 [watch watch cookie 94621548667488] v30'4 uv3 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7fed08006070 con 0x7fed1001bbe0 2023-10-05T19:28:23.628 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:23.622+0000 7fed1e38f700 1 -- 172.21.15.167:0/704848384 <== osd.1 v2:172.21.15.167:6800/30035 29 ==== osd_op_reply(86 notify.1 [create] v30'3 uv3 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7fed18050040 con 0x560ecbc84900 2023-10-05T19:28:23.628 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:23.622+0000 7fed20059e40 1 -- 172.21.15.167:0/704848384 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:87 7.0 7:05bf5b68:::notify.1:head [watch watch cookie 94621548672080] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x560ecbcbf8a0 con 0x560ecbc84900 2023-10-05T19:28:23.628 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:23.622+0000 7fed20059e40 1 -- 172.21.15.167:0/704848384 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:88 7.15 7:a93a5511:::notify.2:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x560ecbcc0720 con 0x560ecbcb9010 2023-10-05T19:28:23.631 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:23.626+0000 7fed1e38f700 1 -- 172.21.15.167:0/704848384 <== osd.1 v2:172.21.15.167:6800/30035 30 ==== osd_op_reply(87 notify.1 [watch watch cookie 94621548672080] v30'4 uv3 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7fed18050040 con 0x560ecbc84900 2023-10-05T19:28:23.631 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:23.626+0000 7fed1db8e700 1 -- 172.21.15.167:0/704848384 <== osd.3 v2:172.21.15.167:6808/30036 22 ==== osd_op_reply(88 notify.2 [create] v30'3 uv3 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7fed1004f040 con 0x560ecbcb9010 2023-10-05T19:28:23.631 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:23.626+0000 7fed20059e40 1 -- 172.21.15.167:0/704848384 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:89 7.15 7:a93a5511:::notify.2:head [watch watch cookie 94621548674368] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x560ecbcbf8a0 con 0x560ecbcb9010 2023-10-05T19:28:23.632 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:23.626+0000 7fed20059e40 1 -- 172.21.15.167:0/704848384 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:90 7.e 7:7759931f:::notify.3:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x560ecbcc11f0 con 0x7fed1001bbe0 2023-10-05T19:28:23.634 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:23.626+0000 7fed1d38d700 1 -- 172.21.15.167:0/704848384 <== osd.2 v2:172.21.15.167:6816/30037 15 ==== osd_op_reply(90 notify.3 [create] v30'3 uv3 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7fed08006070 con 0x7fed1001bbe0 2023-10-05T19:28:23.634 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:23.626+0000 7fed20059e40 1 -- 172.21.15.167:0/704848384 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:91 7.e 7:7759931f:::notify.3:head [watch watch cookie 94621548680688] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x560ecbcc1f30 con 0x7fed1001bbe0 2023-10-05T19:28:23.634 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:23.626+0000 7fed1db8e700 1 -- 172.21.15.167:0/704848384 <== osd.3 v2:172.21.15.167:6808/30036 23 ==== osd_op_reply(89 notify.2 [watch watch cookie 94621548674368] v30'4 uv3 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7fed1004f040 con 0x560ecbcb9010 2023-10-05T19:28:23.634 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:23.626+0000 7fed20059e40 1 -- 172.21.15.167:0/704848384 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:92 7.d 7:b4812045:::notify.4:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x560ecbcc2810 con 0x560ecbc84900 2023-10-05T19:28:23.636 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:23.630+0000 7fed1e38f700 1 -- 172.21.15.167:0/704848384 <== osd.1 v2:172.21.15.167:6800/30035 31 ==== osd_op_reply(92 notify.4 [create] v30'3 uv3 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7fed18050040 con 0x560ecbc84900 2023-10-05T19:28:23.636 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:23.630+0000 7fed20059e40 1 -- 172.21.15.167:0/704848384 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:93 7.d 7:b4812045:::notify.4:head [watch watch cookie 94621548675472] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x560ecbcc2a00 con 0x560ecbc84900 2023-10-05T19:28:23.637 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:23.630+0000 7fed1d38d700 1 -- 172.21.15.167:0/704848384 <== osd.2 v2:172.21.15.167:6816/30037 16 ==== osd_op_reply(91 notify.3 [watch watch cookie 94621548680688] v30'4 uv3 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7fed08006070 con 0x7fed1001bbe0 2023-10-05T19:28:23.637 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:23.630+0000 7fed20059e40 1 -- 172.21.15.167:0/704848384 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:94 7.3 7:c609908c:::notify.5:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x560ecbcc3550 con 0x7fed0800d8e0 2023-10-05T19:28:23.641 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:23.634+0000 7fed1e38f700 1 -- 172.21.15.167:0/704848384 <== osd.0 v2:172.21.15.118:6800/27690 23 ==== osd_op_reply(94 notify.5 [create] v30'3 uv3 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7fed18050040 con 0x7fed0800d8e0 2023-10-05T19:28:23.641 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:23.634+0000 7fed20059e40 1 -- 172.21.15.167:0/704848384 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:95 7.3 7:c609908c:::notify.5:head [watch watch cookie 94621548681792] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x560ecbcc3550 con 0x7fed0800d8e0 2023-10-05T19:28:23.641 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:23.634+0000 7fed20059e40 1 -- 172.21.15.167:0/704848384 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:96 7.14 7:2b04a3e9:::notify.6:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x560ecbcc3fb0 con 0x7fed0800d8e0 2023-10-05T19:28:23.641 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:23.634+0000 7fed1e38f700 1 -- 172.21.15.167:0/704848384 <== osd.1 v2:172.21.15.167:6800/30035 32 ==== osd_op_reply(93 notify.4 [watch watch cookie 94621548675472] v30'4 uv3 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7fed18050040 con 0x560ecbc84900 2023-10-05T19:28:23.642 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:23.638+0000 7fed1e38f700 1 -- 172.21.15.167:0/704848384 <== osd.0 v2:172.21.15.118:6800/27690 24 ==== osd_op_reply(96 notify.6 [create] v30'3 uv3 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7fed18050040 con 0x7fed0800d8e0 2023-10-05T19:28:23.642 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:23.638+0000 7fed20059e40 1 -- 172.21.15.167:0/704848384 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:97 7.14 7:2b04a3e9:::notify.6:head [watch watch cookie 94621548684560] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x560ecbcc3fb0 con 0x7fed0800d8e0 2023-10-05T19:28:23.642 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:23.638+0000 7fed20059e40 1 -- 172.21.15.167:0/704848384 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:98 7.9 7:93e5b521:::notify.7:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x560ecbcc49c0 con 0x7fed0800d8e0 2023-10-05T19:28:23.643 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:23.638+0000 7fed1e38f700 1 -- 172.21.15.167:0/704848384 <== osd.0 v2:172.21.15.118:6800/27690 25 ==== osd_op_reply(95 notify.5 [watch watch cookie 94621548681792] v30'4 uv3 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7fed18050040 con 0x7fed0800d8e0 2023-10-05T19:28:23.645 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:23.638+0000 7fed1e38f700 1 -- 172.21.15.167:0/704848384 <== osd.0 v2:172.21.15.118:6800/27690 26 ==== osd_op_reply(98 notify.7 [create] v30'3 uv3 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7fed18050040 con 0x7fed0800d8e0 2023-10-05T19:28:23.645 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:23.638+0000 7fed1e38f700 1 -- 172.21.15.167:0/704848384 <== osd.0 v2:172.21.15.118:6800/27690 27 ==== osd_op_reply(97 notify.6 [watch watch cookie 94621548684560] v30'4 uv3 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7fed18050040 con 0x7fed0800d8e0 2023-10-05T19:28:23.645 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:23.638+0000 7fed20059e40 1 -- 172.21.15.167:0/704848384 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:99 7.9 7:93e5b521:::notify.7:head [watch watch cookie 94621548687456] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x560ecbcc49c0 con 0x7fed0800d8e0 2023-10-05T19:28:23.645 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:23.638+0000 7fed20059e40 20 add_watcher() i=0 2023-10-05T19:28:23.645 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:23.638+0000 7fed20059e40 20 add_watcher() i=1 2023-10-05T19:28:23.645 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:23.638+0000 7fed20059e40 20 add_watcher() i=2 2023-10-05T19:28:23.645 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:23.638+0000 7fed20059e40 20 add_watcher() i=3 2023-10-05T19:28:23.646 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:23.638+0000 7fed20059e40 20 add_watcher() i=4 2023-10-05T19:28:23.646 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:23.638+0000 7fed20059e40 20 add_watcher() i=5 2023-10-05T19:28:23.646 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:23.638+0000 7fed20059e40 20 add_watcher() i=6 2023-10-05T19:28:23.647 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:23.642+0000 7fed1e38f700 1 -- 172.21.15.167:0/704848384 <== osd.0 v2:172.21.15.118:6800/27690 28 ==== osd_op_reply(99 notify.7 [watch watch cookie 94621548687456] v30'4 uv3 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7fed18050040 con 0x7fed0800d8e0 2023-10-05T19:28:23.647 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:23.642+0000 7fed20059e40 20 add_watcher() i=7 2023-10-05T19:28:23.648 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:23.642+0000 7fed20059e40 2 all 8 watchers are set, enabling cache 2023-10-05T19:28:23.648 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:23.642+0000 7fed20059e40 1 -- 172.21.15.167:0/704848384 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:100 6.4 6:22d26bf9:::data_loggenerations_metadata:head [call version.check_conds in=74b,call version.read in=11b,read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x560ecbcc75a0 con 0x560ecbc84900 2023-10-05T19:28:23.648 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:23.642+0000 7fed1e38f700 1 -- 172.21.15.167:0/704848384 <== osd.1 v2:172.21.15.167:6800/30035 33 ==== osd_op_reply(100 data_loggenerations_metadata [call,call out=48b,read 0~28 out=28b] v0'0 uv1 ondisk = 0) v8 ==== 256+0+76 (crc 0 0 0) 0x7fed18050040 con 0x560ecbc84900 2023-10-05T19:28:23.648 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:23.642+0000 7fed20059e40 1 -- 172.21.15.167:0/704848384 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:101 6.4 6:22d26bf9:::data_loggenerations_metadata:head [watch watch cookie 94621548690112] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x560ecbcc49c0 con 0x560ecbc84900 2023-10-05T19:28:23.650 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:23.646+0000 7fed1e38f700 1 -- 172.21.15.167:0/704848384 <== osd.1 v2:172.21.15.167:6800/30035 34 ==== osd_op_reply(101 data_loggenerations_metadata [watch watch cookie 94621548690112] v30'24 uv1 ondisk = 0) v8 ==== 172+0+0 (crc 0 0 0) 0x7fed18050040 con 0x560ecbc84900 2023-10-05T19:28:23.651 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:23.646+0000 7feceefe5700 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2023-10-05T19:28:23.651 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:23.646+0000 7fed20059e40 20 check_secure_mon_conn(): auth registy supported: methods=[2] modes=[2,1] 2023-10-05T19:28:23.651 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:23.646+0000 7fed20059e40 20 check_secure_mon_conn(): mode 1 is insecure 2023-10-05T19:28:23.651 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:23.646+0000 7fed20059e40 1 Processor -- start 2023-10-05T19:28:23.652 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:23.646+0000 7fed20059e40 1 -- start start 2023-10-05T19:28:23.652 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:23.646+0000 7fed20059e40 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x560ecbcd1a10 0x560ecbcd1de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-05T19:28:23.652 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:23.646+0000 7fed20059e40 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x560ecbcd2320 con 0x560ecbcd1a10 2023-10-05T19:28:23.653 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:23.646+0000 7fed1d38d700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x560ecbcd1a10 0x560ecbcd1de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-05T19:28:23.653 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:23.646+0000 7fed1d38d700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x560ecbcd1a10 0x560ecbcd1de0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.118:3300/0 says I am v2:172.21.15.167:50582/0 (socket says 172.21.15.167:50582) 2023-10-05T19:28:23.653 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:23.646+0000 7fed1d38d700 1 -- 172.21.15.167:0/1578218563 learned_addr learned my addr 172.21.15.167:0/1578218563 (peer_addr_for_me v2:172.21.15.167:0/0) 2023-10-05T19:28:23.653 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:23.646+0000 7fed1d38d700 1 -- 172.21.15.167:0/1578218563 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x560ecbcd2460 con 0x560ecbcd1a10 2023-10-05T19:28:23.653 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:23.646+0000 7fed1d38d700 1 --2- 172.21.15.167:0/1578218563 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x560ecbcd1a10 0x560ecbcd1de0 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x560ecbcb6d60 tx=0x7fed08006000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5c773204390e283f server_cookie=0 in_seq=0 out_seq=0 2023-10-05T19:28:23.654 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:23.646+0000 7fecee7e4700 1 -- 172.21.15.167:0/1578218563 <== mon.0 v2:172.21.15.118:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fed08023040 con 0x560ecbcd1a10 2023-10-05T19:28:23.654 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:23.646+0000 7fecee7e4700 1 -- 172.21.15.167:0/1578218563 <== mon.0 v2:172.21.15.118:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fed08024040 con 0x560ecbcd1a10 2023-10-05T19:28:23.654 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:23.646+0000 7fecee7e4700 1 -- 172.21.15.167:0/1578218563 <== mon.0 v2:172.21.15.118:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fed08013040 con 0x560ecbcd1a10 2023-10-05T19:28:23.655 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:23.650+0000 7fed20059e40 1 -- 172.21.15.167:0/1578218563 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x560ecbcd1a10 msgr2=0x560ecbcd1de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-05T19:28:23.655 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:23.650+0000 7fed20059e40 1 --2- 172.21.15.167:0/1578218563 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x560ecbcd1a10 0x560ecbcd1de0 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x560ecbcb6d60 tx=0x7fed08006000 comp rx=0 tx=0).stop 2023-10-05T19:28:23.655 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:23.650+0000 7fed20059e40 1 -- 172.21.15.167:0/1578218563 shutdown_connections 2023-10-05T19:28:23.655 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:23.650+0000 7fed20059e40 1 --2- 172.21.15.167:0/1578218563 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x560ecbcd1a10 0x560ecbcd1de0 unknown :-1 s=CLOSED pgs=138 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-05T19:28:23.655 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:23.650+0000 7fed20059e40 1 -- 172.21.15.167:0/1578218563 >> 172.21.15.167:0/1578218563 conn(0x560ecbcccdd0 msgr2=0x560ecbccaf80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-05T19:28:23.656 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:23.650+0000 7fed20059e40 1 -- 172.21.15.167:0/1578218563 shutdown_connections 2023-10-05T19:28:23.656 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:23.650+0000 7fed20059e40 1 -- 172.21.15.167:0/1578218563 wait complete. 2023-10-05T19:28:23.657 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:23.650+0000 7fed20059e40 1 Processor -- start 2023-10-05T19:28:23.657 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:23.650+0000 7fed20059e40 1 -- start start 2023-10-05T19:28:23.657 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:23.650+0000 7fed20059e40 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x560ecbcd1a10 0x560ecbcd7540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-05T19:28:23.657 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:23.650+0000 7fed20059e40 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x560ecbcd8540 con 0x560ecbcd1a10 2023-10-05T19:28:23.657 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:23.650+0000 7fed1d38d700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x560ecbcd1a10 0x560ecbcd7540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-05T19:28:23.657 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:23.650+0000 7fed1d38d700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x560ecbcd1a10 0x560ecbcd7540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.118:3300/0 says I am v2:172.21.15.167:50598/0 (socket says 172.21.15.167:50598) 2023-10-05T19:28:23.657 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:23.650+0000 7fed1d38d700 1 -- 172.21.15.167:0/3974143945 learned_addr learned my addr 172.21.15.167:0/3974143945 (peer_addr_for_me v2:172.21.15.167:0/0) 2023-10-05T19:28:23.657 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:23.650+0000 7fed1d38d700 1 -- 172.21.15.167:0/3974143945 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fed0800ece0 con 0x560ecbcd1a10 2023-10-05T19:28:23.658 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:23.650+0000 7fed1d38d700 1 --2- 172.21.15.167:0/3974143945 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x560ecbcd1a10 0x560ecbcd7540 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x560ecbcb6b00 tx=0x7fed08030000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-05T19:28:23.658 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:23.650+0000 7fececfe1700 1 -- 172.21.15.167:0/3974143945 <== mon.0 v2:172.21.15.118:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fed08023040 con 0x560ecbcd1a10 2023-10-05T19:28:23.659 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:23.650+0000 7fececfe1700 1 -- 172.21.15.167:0/3974143945 <== mon.0 v2:172.21.15.118:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fed0801e4a0 con 0x560ecbcd1a10 2023-10-05T19:28:23.659 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:23.650+0000 7fed20059e40 1 -- 172.21.15.167:0/3974143945 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x560ecbcd7b40 con 0x560ecbcd1a10 2023-10-05T19:28:23.659 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:23.650+0000 7fececfe1700 1 -- 172.21.15.167:0/3974143945 <== mon.0 v2:172.21.15.118:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fed0801e670 con 0x560ecbcd1a10 2023-10-05T19:28:23.659 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:23.650+0000 7fed20059e40 1 -- 172.21.15.167:0/3974143945 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x560ecbcd7f80 con 0x560ecbcd1a10 2023-10-05T19:28:23.660 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:23.654+0000 7fececfe1700 1 -- 172.21.15.167:0/3974143945 <== mon.0 v2:172.21.15.118:3300/0 4 ==== mgrmap(e 5) v1 ==== 40329+0+0 (secure 0 0 0) 0x7fed0801e7d0 con 0x560ecbcd1a10 2023-10-05T19:28:23.660 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:23.654+0000 7fececfe1700 1 --2- 172.21.15.167:0/3974143945 >> [v2:172.21.15.167:6824/29889,v1:172.21.15.167:6825/29889] conn(0x7fec64030ca0 0x7fec64033150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-05T19:28:23.660 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:23.654+0000 7fececfe1700 1 -- 172.21.15.167:0/3974143945 <== mon.0 v2:172.21.15.118:3300/0 5 ==== osd_map(30..30 src has 1..30) v4 ==== 6057+0+0 (secure 0 0 0) 0x7fed0804fc60 con 0x560ecbcd1a10 2023-10-05T19:28:23.660 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:23.654+0000 7fed1e38f700 1 --2- 172.21.15.167:0/3974143945 >> [v2:172.21.15.167:6824/29889,v1:172.21.15.167:6825/29889] conn(0x7fec64030ca0 0x7fec64033150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-05T19:28:23.660 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:23.654+0000 7fed20059e40 5 note: GC not initialized 2023-10-05T19:28:23.661 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:23.654+0000 7fed1e38f700 1 --2- 172.21.15.167:0/3974143945 >> [v2:172.21.15.167:6824/29889,v1:172.21.15.167:6825/29889] conn(0x7fec64030ca0 0x7fec64033150 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x560ecbcc3f50 tx=0x7fed18000f90 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-05T19:28:23.662 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:23.658+0000 7fec5dffb700 20 reqs_thread_entry: start 2023-10-05T19:28:23.663 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:23.658+0000 7fec577fe700 20 reqs_thread_entry: start 2023-10-05T19:28:23.663 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:23.658+0000 7fed20059e40 20 init_complete bucket index max shards: 11 2023-10-05T19:28:23.663 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:23.658+0000 7fed20059e40 10 rgw notify: Started notification manager with: 1 workers 2023-10-05T19:28:23.663 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:23.658+0000 7fec55ffb700 20 rgw notify: INFO: next queues processing will happen at: Thu Oct 5 19:28:53 2023 2023-10-05T19:28:23.664 INFO:teuthology.orchestra.run.smithi167.stderr: 2023-10-05T19:28:23.664 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:23.658+0000 7fec557fa700 20 reqs_thread_entry: start 2023-10-05T19:28:23.664 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:23.658+0000 7fed20059e40 1 -- 172.21.15.167:0/704848384 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:102 8.0 8:00000000:root:::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x560ecbd3f7a0 con 0x560ecbcb9010 2023-10-05T19:28:23.666 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:23.662+0000 7fed1db8e700 1 -- 172.21.15.167:0/704848384 <== osd.3 v2:172.21.15.167:6808/30036 24 ==== osd_op_reply(102 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fed1004f040 con 0x560ecbcb9010 2023-10-05T19:28:23.666 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:23.662+0000 7fed1db8e700 1 -- 172.21.15.167:0/704848384 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:103 8.10 8:08000000:root:::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x560ecbd3f7a0 con 0x560ecbcb9010 2023-10-05T19:28:23.666 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:23.662+0000 7fed1db8e700 1 -- 172.21.15.167:0/704848384 <== osd.3 v2:172.21.15.167:6808/30036 25 ==== osd_op_reply(103 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fed1004f040 con 0x560ecbcb9010 2023-10-05T19:28:23.667 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:23.662+0000 7fed1db8e700 1 -- 172.21.15.167:0/704848384 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:104 8.8 8:10000000:root:::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x560ecbd3f7a0 con 0x560ecbc84900 2023-10-05T19:28:23.667 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:23.662+0000 7fed1e38f700 1 -- 172.21.15.167:0/704848384 <== osd.1 v2:172.21.15.167:6800/30035 35 ==== osd_op_reply(104 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fed18050040 con 0x560ecbc84900 2023-10-05T19:28:23.667 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:23.662+0000 7fed1e38f700 1 -- 172.21.15.167:0/704848384 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:105 8.18 8:18000000:root:::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x560ecbcbc580 con 0x560ecbc84900 2023-10-05T19:28:23.668 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:23.662+0000 7fed1e38f700 1 -- 172.21.15.167:0/704848384 <== osd.1 v2:172.21.15.167:6800/30035 36 ==== osd_op_reply(105 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fed18050040 con 0x560ecbc84900 2023-10-05T19:28:23.668 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:23.662+0000 7fed1e38f700 1 -- 172.21.15.167:0/704848384 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:106 8.4 8:20000000:root:::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x560ecbcbc580 con 0x7fed1001bbe0 2023-10-05T19:28:23.669 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:23.662+0000 7fed1d38d700 1 -- 172.21.15.167:0/704848384 <== osd.2 v2:172.21.15.167:6816/30037 17 ==== osd_op_reply(106 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fed08006070 con 0x7fed1001bbe0 2023-10-05T19:28:23.669 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:23.662+0000 7fed1d38d700 1 -- 172.21.15.167:0/704848384 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:107 8.14 8:28000000:root:::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x560ecbcbc580 con 0x560ecbcb9010 2023-10-05T19:28:23.669 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:23.662+0000 7fed1db8e700 1 -- 172.21.15.167:0/704848384 <== osd.3 v2:172.21.15.167:6808/30036 26 ==== osd_op_reply(107 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fed1004f040 con 0x560ecbcb9010 2023-10-05T19:28:23.670 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:23.662+0000 7fed1db8e700 1 -- 172.21.15.167:0/704848384 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:108 8.c 8:30000000:root:::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x560ecbcbc580 con 0x560ecbc84900 2023-10-05T19:28:23.670 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:23.666+0000 7fed1e38f700 1 -- 172.21.15.167:0/704848384 <== osd.1 v2:172.21.15.167:6800/30035 37 ==== osd_op_reply(108 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fed18050040 con 0x560ecbc84900 2023-10-05T19:28:23.670 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:23.666+0000 7fed1e38f700 1 -- 172.21.15.167:0/704848384 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:109 8.1c 8:38000000:root:::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x560ecbcbc580 con 0x560ecbcb9010 2023-10-05T19:28:23.671 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:23.666+0000 7fed1db8e700 1 -- 172.21.15.167:0/704848384 <== osd.3 v2:172.21.15.167:6808/30036 27 ==== osd_op_reply(109 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fed1004f040 con 0x560ecbcb9010 2023-10-05T19:28:23.671 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:23.666+0000 7fed1db8e700 1 -- 172.21.15.167:0/704848384 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:110 8.2 8:40000000:root:::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x560ecbcbc580 con 0x7fed1001bbe0 2023-10-05T19:28:23.672 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:23.666+0000 7fed1d38d700 1 -- 172.21.15.167:0/704848384 <== osd.2 v2:172.21.15.167:6816/30037 18 ==== osd_op_reply(110 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fed08006070 con 0x7fed1001bbe0 2023-10-05T19:28:23.672 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:23.666+0000 7fed1d38d700 1 -- 172.21.15.167:0/704848384 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:111 8.12 8:48000000:root:::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x560ecbcbc580 con 0x560ecbcb9010 2023-10-05T19:28:23.672 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:23.666+0000 7fed1db8e700 1 -- 172.21.15.167:0/704848384 <== osd.3 v2:172.21.15.167:6808/30036 28 ==== osd_op_reply(111 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fed1005b040 con 0x560ecbcb9010 2023-10-05T19:28:23.672 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:23.666+0000 7fed1db8e700 1 -- 172.21.15.167:0/704848384 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:112 8.a 8:50000000:root:::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x560ecbcbc580 con 0x560ecbc84900 2023-10-05T19:28:23.673 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:23.666+0000 7fed1e38f700 1 -- 172.21.15.167:0/704848384 <== osd.1 v2:172.21.15.167:6800/30035 38 ==== osd_op_reply(112 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fed18050040 con 0x560ecbc84900 2023-10-05T19:28:23.673 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:23.666+0000 7fed1e38f700 1 -- 172.21.15.167:0/704848384 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:113 8.1a 8:58000000:root:::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x560ecbcbc580 con 0x7fed0800d8e0 2023-10-05T19:28:23.674 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:23.670+0000 7fed1e38f700 1 -- 172.21.15.167:0/704848384 <== osd.0 v2:172.21.15.118:6800/27690 29 ==== osd_op_reply(113 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fed18050040 con 0x7fed0800d8e0 2023-10-05T19:28:23.674 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:23.670+0000 7fed1e38f700 1 -- 172.21.15.167:0/704848384 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:114 8.6 8:60000000:root:::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x560ecbcbc580 con 0x7fed0800d8e0 2023-10-05T19:28:23.675 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:23.670+0000 7fed1e38f700 1 -- 172.21.15.167:0/704848384 <== osd.0 v2:172.21.15.118:6800/27690 30 ==== osd_op_reply(114 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fed18050040 con 0x7fed0800d8e0 2023-10-05T19:28:23.675 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:23.670+0000 7fed1e38f700 1 -- 172.21.15.167:0/704848384 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:115 8.16 8:68000000:root:::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x560ecbcbc580 con 0x560ecbcb9010 2023-10-05T19:28:23.676 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:23.670+0000 7fed1db8e700 1 -- 172.21.15.167:0/704848384 <== osd.3 v2:172.21.15.167:6808/30036 29 ==== osd_op_reply(115 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fed1005b040 con 0x560ecbcb9010 2023-10-05T19:28:23.676 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:23.670+0000 7fed1db8e700 1 -- 172.21.15.167:0/704848384 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:116 8.e 8:70000000:root:::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x560ecbcbc580 con 0x7fed0800d8e0 2023-10-05T19:28:23.677 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:23.670+0000 7fed1e38f700 1 -- 172.21.15.167:0/704848384 <== osd.0 v2:172.21.15.118:6800/27690 31 ==== osd_op_reply(116 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fed18050040 con 0x7fed0800d8e0 2023-10-05T19:28:23.677 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:23.670+0000 7fed1e38f700 1 -- 172.21.15.167:0/704848384 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:117 8.1e 8:78000000:root:::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x560ecbcbc580 con 0x7fed0800d8e0 2023-10-05T19:28:23.678 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:23.670+0000 7fed1e38f700 1 -- 172.21.15.167:0/704848384 <== osd.0 v2:172.21.15.118:6800/27690 32 ==== osd_op_reply(117 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fed18050040 con 0x7fed0800d8e0 2023-10-05T19:28:23.678 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:23.674+0000 7fed1e38f700 1 -- 172.21.15.167:0/704848384 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:118 8.1 8:80000000:root:::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x560ecbcbc580 con 0x7fed0800d8e0 2023-10-05T19:28:23.679 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:23.674+0000 7fed1e38f700 1 -- 172.21.15.167:0/704848384 <== osd.0 v2:172.21.15.118:6800/27690 33 ==== osd_op_reply(118 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fed18050040 con 0x7fed0800d8e0 2023-10-05T19:28:23.679 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:23.674+0000 7fed1e38f700 1 -- 172.21.15.167:0/704848384 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:119 8.11 8:88000000:root:::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x560ecbcbc580 con 0x560ecbcb9010 2023-10-05T19:28:23.680 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:23.674+0000 7fed1db8e700 1 -- 172.21.15.167:0/704848384 <== osd.3 v2:172.21.15.167:6808/30036 30 ==== osd_op_reply(119 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fed1005b040 con 0x560ecbcb9010 2023-10-05T19:28:23.680 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:23.674+0000 7fed1db8e700 1 -- 172.21.15.167:0/704848384 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:120 8.9 8:90000000:root:::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x560ecbcbc580 con 0x7fed1001bbe0 2023-10-05T19:28:23.680 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:23.674+0000 7fed1d38d700 1 -- 172.21.15.167:0/704848384 <== osd.2 v2:172.21.15.167:6816/30037 19 ==== osd_op_reply(120 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fed08006070 con 0x7fed1001bbe0 2023-10-05T19:28:23.681 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:23.674+0000 7fed1d38d700 1 -- 172.21.15.167:0/704848384 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:121 8.19 8:98000000:root:::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x560ecbcbc580 con 0x560ecbc84900 2023-10-05T19:28:23.681 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:23.674+0000 7fed1e38f700 1 -- 172.21.15.167:0/704848384 <== osd.1 v2:172.21.15.167:6800/30035 39 ==== osd_op_reply(121 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fed18050040 con 0x560ecbc84900 2023-10-05T19:28:23.681 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:23.674+0000 7fed1e38f700 1 -- 172.21.15.167:0/704848384 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:122 8.5 8:a0000000:root:::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x560ecbcbc580 con 0x560ecbc84900 2023-10-05T19:28:23.682 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:23.674+0000 7fed1e38f700 1 -- 172.21.15.167:0/704848384 <== osd.1 v2:172.21.15.167:6800/30035 40 ==== osd_op_reply(122 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fed18050040 con 0x560ecbc84900 2023-10-05T19:28:23.682 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:23.674+0000 7fed1e38f700 1 -- 172.21.15.167:0/704848384 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:123 8.15 8:a8000000:root:::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x560ecbcbc580 con 0x560ecbcb9010 2023-10-05T19:28:23.682 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:23.678+0000 7fed1db8e700 1 -- 172.21.15.167:0/704848384 <== osd.3 v2:172.21.15.167:6808/30036 31 ==== osd_op_reply(123 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fed1005b040 con 0x560ecbcb9010 2023-10-05T19:28:23.683 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:23.678+0000 7fed1db8e700 1 -- 172.21.15.167:0/704848384 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:124 8.d 8:b0000000:root:::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x560ecbcbc580 con 0x7fed1001bbe0 2023-10-05T19:28:23.683 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:23.678+0000 7fed1d38d700 1 -- 172.21.15.167:0/704848384 <== osd.2 v2:172.21.15.167:6816/30037 20 ==== osd_op_reply(124 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fed08006070 con 0x7fed1001bbe0 2023-10-05T19:28:23.683 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:23.678+0000 7fed1d38d700 1 -- 172.21.15.167:0/704848384 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:125 8.1d 8:b8000000:root:::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x560ecbcbc580 con 0x7fed0800d8e0 2023-10-05T19:28:23.684 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:23.678+0000 7fed1e38f700 1 -- 172.21.15.167:0/704848384 <== osd.0 v2:172.21.15.118:6800/27690 34 ==== osd_op_reply(125 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fed18050040 con 0x7fed0800d8e0 2023-10-05T19:28:23.684 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:23.678+0000 7fed1e38f700 1 -- 172.21.15.167:0/704848384 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:126 8.3 8:c0000000:root:::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x560ecbcbc580 con 0x560ecbc84900 2023-10-05T19:28:23.685 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:23.678+0000 7fed1e38f700 1 -- 172.21.15.167:0/704848384 <== osd.1 v2:172.21.15.167:6800/30035 41 ==== osd_op_reply(126 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fed18050040 con 0x560ecbc84900 2023-10-05T19:28:23.685 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:23.678+0000 7fed1e38f700 1 -- 172.21.15.167:0/704848384 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:127 8.13 8:c8000000:root:::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x560ecbcbc580 con 0x7fed0800d8e0 2023-10-05T19:28:23.687 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:23.682+0000 7fed1e38f700 1 -- 172.21.15.167:0/704848384 <== osd.0 v2:172.21.15.118:6800/27690 35 ==== osd_op_reply(127 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fed18050040 con 0x7fed0800d8e0 2023-10-05T19:28:23.687 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:23.682+0000 7fed1e38f700 1 -- 172.21.15.167:0/704848384 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:128 8.b 8:d0000000:root:::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x560ecbcbc580 con 0x560ecbcb9010 2023-10-05T19:28:23.687 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:23.682+0000 7fed1db8e700 1 -- 172.21.15.167:0/704848384 <== osd.3 v2:172.21.15.167:6808/30036 32 ==== osd_op_reply(128 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fed1005b040 con 0x560ecbcb9010 2023-10-05T19:28:23.687 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:23.682+0000 7fed1db8e700 1 -- 172.21.15.167:0/704848384 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:129 8.1b 8:d8000000:root:::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x560ecbcbc580 con 0x7fed1001bbe0 2023-10-05T19:28:23.688 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:23.682+0000 7fed1d38d700 1 -- 172.21.15.167:0/704848384 <== osd.2 v2:172.21.15.167:6816/30037 21 ==== osd_op_reply(129 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fed08006070 con 0x7fed1001bbe0 2023-10-05T19:28:23.688 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:23.682+0000 7fed1d38d700 1 -- 172.21.15.167:0/704848384 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:130 8.7 8:e0000000:root:::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x560ecbcbc580 con 0x7fed0800d8e0 2023-10-05T19:28:23.689 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:23.682+0000 7fed1e38f700 1 -- 172.21.15.167:0/704848384 <== osd.0 v2:172.21.15.118:6800/27690 36 ==== osd_op_reply(130 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fed18050040 con 0x7fed0800d8e0 2023-10-05T19:28:23.689 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:23.682+0000 7fed1e38f700 1 -- 172.21.15.167:0/704848384 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:131 8.17 8:e8000000:root:::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x560ecbcbc580 con 0x560ecbc84900 2023-10-05T19:28:23.690 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:23.682+0000 7fed1e38f700 1 -- 172.21.15.167:0/704848384 <== osd.1 v2:172.21.15.167:6800/30035 42 ==== osd_op_reply(131 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fed18050040 con 0x560ecbc84900 2023-10-05T19:28:23.690 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:23.686+0000 7fed1e38f700 1 -- 172.21.15.167:0/704848384 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:132 8.f 8:f0000000:root:::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x560ecbcbc580 con 0x560ecbcb9010 2023-10-05T19:28:23.690 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:23.686+0000 7fed1db8e700 1 -- 172.21.15.167:0/704848384 <== osd.3 v2:172.21.15.167:6808/30036 33 ==== osd_op_reply(132 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fed1005b040 con 0x560ecbcb9010 2023-10-05T19:28:23.691 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:23.686+0000 7fed1db8e700 1 -- 172.21.15.167:0/704848384 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:133 8.1f 8:f8000000:root:::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x560ecbcbc580 con 0x7fed1001bbe0 2023-10-05T19:28:23.691 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:23.686+0000 7fed1d38d700 1 -- 172.21.15.167:0/704848384 <== osd.2 v2:172.21.15.167:6816/30037 22 ==== osd_op_reply(133 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fed08006070 con 0x7fed1001bbe0 2023-10-05T19:28:23.692 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:23.686+0000 7fed20059e40 1 -- 172.21.15.167:0/704848384 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:134 7.1f 7:f95f44c2:::notify.0:head [watch unwatch cookie 94621548667488] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x560ecbcfe100 con 0x7fed1001bbe0 2023-10-05T19:28:23.695 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:23.690+0000 7fed1d38d700 1 -- 172.21.15.167:0/704848384 <== osd.2 v2:172.21.15.167:6816/30037 23 ==== osd_op_reply(134 notify.0 [watch unwatch cookie 94621548667488] v30'5 uv3 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7fed08006070 con 0x7fed1001bbe0 2023-10-05T19:28:23.695 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:23.690+0000 7fed20059e40 20 remove_watcher() i=0 2023-10-05T19:28:23.695 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:23.690+0000 7fed20059e40 2 removed watcher, disabling cache 2023-10-05T19:28:23.695 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:23.690+0000 7fed20059e40 1 -- 172.21.15.167:0/704848384 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:135 7.0 7:05bf5b68:::notify.1:head [watch unwatch cookie 94621548672080] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x560ecbcbe260 con 0x560ecbc84900 2023-10-05T19:28:23.698 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:23.690+0000 7fed1e38f700 1 -- 172.21.15.167:0/704848384 <== osd.1 v2:172.21.15.167:6800/30035 43 ==== osd_op_reply(135 notify.1 [watch unwatch cookie 94621548672080] v30'5 uv3 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7fed18050040 con 0x560ecbc84900 2023-10-05T19:28:23.698 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:23.690+0000 7fed20059e40 20 remove_watcher() i=1 2023-10-05T19:28:23.698 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:23.690+0000 7fed20059e40 1 -- 172.21.15.167:0/704848384 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:136 7.15 7:a93a5511:::notify.2:head [watch unwatch cookie 94621548674368] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x560ecbcbf450 con 0x560ecbcb9010 2023-10-05T19:28:23.701 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:23.694+0000 7fed1db8e700 1 -- 172.21.15.167:0/704848384 <== osd.3 v2:172.21.15.167:6808/30036 34 ==== osd_op_reply(136 notify.2 [watch unwatch cookie 94621548674368] v30'5 uv3 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7fed1005b040 con 0x560ecbcb9010 2023-10-05T19:28:23.701 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:23.694+0000 7fed20059e40 20 remove_watcher() i=2 2023-10-05T19:28:23.701 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:23.694+0000 7fed20059e40 1 -- 172.21.15.167:0/704848384 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:137 7.e 7:7759931f:::notify.3:head [watch unwatch cookie 94621548680688] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x560ecbcbfd40 con 0x7fed1001bbe0 2023-10-05T19:28:23.703 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:23.698+0000 7fed1d38d700 1 -- 172.21.15.167:0/704848384 <== osd.2 v2:172.21.15.167:6816/30037 24 ==== osd_op_reply(137 notify.3 [watch unwatch cookie 94621548680688] v30'5 uv3 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7fed08006070 con 0x7fed1001bbe0 2023-10-05T19:28:23.703 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:23.698+0000 7fed20059e40 20 remove_watcher() i=3 2023-10-05T19:28:23.703 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:23.698+0000 7fed20059e40 1 -- 172.21.15.167:0/704848384 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:138 7.d 7:b4812045:::notify.4:head [watch unwatch cookie 94621548675472] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x560ecbcbfd40 con 0x560ecbc84900 2023-10-05T19:28:23.706 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:23.702+0000 7fed1e38f700 1 -- 172.21.15.167:0/704848384 <== osd.1 v2:172.21.15.167:6800/30035 44 ==== osd_op_reply(138 notify.4 [watch unwatch cookie 94621548675472] v30'5 uv3 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7fed18050040 con 0x560ecbc84900 2023-10-05T19:28:23.707 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:23.702+0000 7fed20059e40 20 remove_watcher() i=4 2023-10-05T19:28:23.707 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:23.702+0000 7fed20059e40 1 -- 172.21.15.167:0/704848384 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:139 7.3 7:c609908c:::notify.5:head [watch unwatch cookie 94621548681792] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x560ecbcc0230 con 0x7fed0800d8e0 2023-10-05T19:28:23.710 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:23.702+0000 7fed1e38f700 1 -- 172.21.15.167:0/704848384 <== osd.0 v2:172.21.15.118:6800/27690 37 ==== osd_op_reply(139 notify.5 [watch unwatch cookie 94621548681792] v30'5 uv3 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7fed18050040 con 0x7fed0800d8e0 2023-10-05T19:28:23.710 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:23.702+0000 7fed20059e40 20 remove_watcher() i=5 2023-10-05T19:28:23.710 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:23.702+0000 7fed20059e40 1 -- 172.21.15.167:0/704848384 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:140 7.14 7:2b04a3e9:::notify.6:head [watch unwatch cookie 94621548684560] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x560ecbcc0230 con 0x7fed0800d8e0 2023-10-05T19:28:23.712 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:23.706+0000 7fed1e38f700 1 -- 172.21.15.167:0/704848384 <== osd.0 v2:172.21.15.118:6800/27690 38 ==== osd_op_reply(140 notify.6 [watch unwatch cookie 94621548684560] v30'5 uv3 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7fed18050040 con 0x7fed0800d8e0 2023-10-05T19:28:23.712 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:23.706+0000 7fed20059e40 20 remove_watcher() i=6 2023-10-05T19:28:23.713 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:23.706+0000 7fed20059e40 1 -- 172.21.15.167:0/704848384 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:141 7.9 7:93e5b521:::notify.7:head [watch unwatch cookie 94621548687456] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x560ecbcc1ae0 con 0x7fed0800d8e0 2023-10-05T19:28:23.715 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:23.710+0000 7fed1e38f700 1 -- 172.21.15.167:0/704848384 <== osd.0 v2:172.21.15.118:6800/27690 39 ==== osd_op_reply(141 notify.7 [watch unwatch cookie 94621548687456] v30'5 uv3 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7fed18050040 con 0x7fed0800d8e0 2023-10-05T19:28:23.715 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:23.710+0000 7fed20059e40 20 remove_watcher() i=7 2023-10-05T19:28:23.717 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:23.710+0000 7feceefe5700 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2023-10-05T19:28:23.717 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:23.710+0000 7fed20059e40 1 -- 172.21.15.167:0/704848384 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:142 6.4 6:22d26bf9:::data_loggenerations_metadata:head [watch unwatch cookie 94621548690112] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x560ecbcc1ae0 con 0x560ecbc84900 2023-10-05T19:28:23.721 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:23.714+0000 7fed1e38f700 1 -- 172.21.15.167:0/704848384 <== osd.1 v2:172.21.15.167:6800/30035 45 ==== osd_op_reply(142 data_loggenerations_metadata [watch unwatch cookie 94621548690112] v30'25 uv1 ondisk = 0) v8 ==== 172+0+0 (crc 0 0 0) 0x7fed18050040 con 0x560ecbc84900 2023-10-05T19:28:23.721 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:23.714+0000 7fed20059e40 1 -- 172.21.15.167:0/704848384 >> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] conn(0x7fed0800d8e0 msgr2=0x7fed0800c360 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-05T19:28:23.721 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:23.714+0000 7fed20059e40 1 --2- 172.21.15.167:0/704848384 >> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] conn(0x7fed0800d8e0 0x7fed0800c360 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-05T19:28:23.721 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:23.714+0000 7fed20059e40 1 -- 172.21.15.167:0/704848384 >> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] conn(0x560ecbc84900 msgr2=0x560ecbcb5600 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-05T19:28:23.721 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:23.714+0000 7fed20059e40 1 --2- 172.21.15.167:0/704848384 >> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] conn(0x560ecbc84900 0x560ecbcb5600 crc :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-05T19:28:23.721 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:23.714+0000 7fed20059e40 1 -- 172.21.15.167:0/704848384 >> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] conn(0x7fed1001bbe0 msgr2=0x7fed100383b0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-05T19:28:23.721 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:23.714+0000 7fed20059e40 1 --2- 172.21.15.167:0/704848384 >> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] conn(0x7fed1001bbe0 0x7fed100383b0 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-05T19:28:23.721 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:23.714+0000 7fed20059e40 1 -- 172.21.15.167:0/704848384 >> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] conn(0x560ecbcb9010 msgr2=0x560ecbcbb3f0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-05T19:28:23.722 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:23.714+0000 7fed20059e40 1 --2- 172.21.15.167:0/704848384 >> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] conn(0x560ecbcb9010 0x560ecbcbb3f0 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-05T19:28:23.722 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:23.714+0000 7fed20059e40 1 -- 172.21.15.167:0/704848384 >> [v2:172.21.15.167:6824/29889,v1:172.21.15.167:6825/29889] conn(0x7fed00030ca0 msgr2=0x7fed00033150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-05T19:28:23.722 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:23.714+0000 7fed20059e40 1 --2- 172.21.15.167:0/704848384 >> [v2:172.21.15.167:6824/29889,v1:172.21.15.167:6825/29889] conn(0x7fed00030ca0 0x7fed00033150 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7fed080011e0 tx=0x7fed080013a0 comp rx=0 tx=0).stop 2023-10-05T19:28:23.722 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:23.714+0000 7fed20059e40 1 -- 172.21.15.167:0/704848384 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x560ecbc83ff0 msgr2=0x560ecbca9b60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-05T19:28:23.722 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:23.714+0000 7fed1e38f700 1 -- 172.21.15.167:0/704848384 reap_dead start 2023-10-05T19:28:23.722 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:23.714+0000 7fed20059e40 1 --2- 172.21.15.167:0/704848384 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x560ecbc83ff0 0x560ecbca9b60 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x560ecbc9f540 tx=0x7fed1001d000 comp rx=0 tx=0).stop 2023-10-05T19:28:23.722 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:23.714+0000 7fed20059e40 1 -- 172.21.15.167:0/704848384 shutdown_connections 2023-10-05T19:28:23.722 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:23.714+0000 7fed20059e40 1 --2- 172.21.15.167:0/704848384 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x560ecbc83ff0 0x560ecbca9b60 unknown :-1 s=CLOSED pgs=137 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-05T19:28:23.722 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:23.714+0000 7fed20059e40 1 -- 172.21.15.167:0/704848384 >> 172.21.15.167:0/704848384 conn(0x560ecbbf7840 msgr2=0x560ecbc9d4a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-05T19:28:23.722 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:23.714+0000 7fed20059e40 1 -- 172.21.15.167:0/704848384 shutdown_connections 2023-10-05T19:28:23.722 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:23.714+0000 7fed20059e40 1 -- 172.21.15.167:0/704848384 wait complete. 2023-10-05T19:28:23.723 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:23.714+0000 7fed20059e40 1 -- 172.21.15.167:0/3974143945 >> [v2:172.21.15.167:6824/29889,v1:172.21.15.167:6825/29889] conn(0x7fec64030ca0 msgr2=0x7fec64033150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-05T19:28:23.724 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:23.714+0000 7fed20059e40 1 --2- 172.21.15.167:0/3974143945 >> [v2:172.21.15.167:6824/29889,v1:172.21.15.167:6825/29889] conn(0x7fec64030ca0 0x7fec64033150 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x560ecbcc3f50 tx=0x7fed18000f90 comp rx=0 tx=0).stop 2023-10-05T19:28:23.724 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:23.718+0000 7fed20059e40 1 -- 172.21.15.167:0/3974143945 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x560ecbcd1a10 msgr2=0x560ecbcd7540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-05T19:28:23.724 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:23.718+0000 7fed20059e40 1 --2- 172.21.15.167:0/3974143945 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x560ecbcd1a10 0x560ecbcd7540 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x560ecbcb6b00 tx=0x7fed08030000 comp rx=0 tx=0).stop 2023-10-05T19:28:23.724 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:23.718+0000 7fed20059e40 1 -- 172.21.15.167:0/3974143945 shutdown_connections 2023-10-05T19:28:23.724 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:23.718+0000 7fed20059e40 1 --2- 172.21.15.167:0/3974143945 >> [v2:172.21.15.167:6824/29889,v1:172.21.15.167:6825/29889] conn(0x7fec64030ca0 0x7fec64033150 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-05T19:28:23.724 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:23.718+0000 7fed20059e40 1 --2- 172.21.15.167:0/3974143945 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x560ecbcd1a10 0x560ecbcd7540 unknown :-1 s=CLOSED pgs=139 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-05T19:28:23.724 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:23.718+0000 7fed20059e40 1 -- 172.21.15.167:0/3974143945 >> 172.21.15.167:0/3974143945 conn(0x560ecbcccdd0 msgr2=0x560ecbccaed0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-05T19:28:23.724 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:23.718+0000 7fed20059e40 1 -- 172.21.15.167:0/3974143945 shutdown_connections 2023-10-05T19:28:23.724 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:23.718+0000 7fed20059e40 1 -- 172.21.15.167:0/3974143945 wait complete. 2023-10-05T19:28:23.725 INFO:teuthology.orchestra.run.smithi167.stdout:[] 2023-10-05T19:28:23.728 DEBUG:tasks.util.rgw: json result: [] 2023-10-05T19:28:23.728 INFO:tasks.util.rgw:rgwadmin: client.0 : ['user', 'rm', '--uid', 'bud'] 2023-10-05T19:28:23.728 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'ceph', 'user', 'rm', '--uid', 'bud'] 2023-10-05T19:28:23.729 DEBUG:teuthology.orchestra.run.smithi167:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster ceph user rm --uid bud 2023-10-05T19:28:23.768 INFO:teuthology.orchestra.run.smithi167.stderr:ignoring --setuser ceph since I am not root 2023-10-05T19:28:23.768 INFO:teuthology.orchestra.run.smithi167.stderr:ignoring --setgroup ceph since I am not root 2023-10-05T19:28:23.769 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:23.762+0000 7f21fd308e40 1 Processor -- start 2023-10-05T19:28:23.769 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:23.762+0000 7f21fd308e40 1 -- start start 2023-10-05T19:28:23.769 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:23.762+0000 7f21fd308e40 1 --2- >> v2:172.21.15.118:3300/0 conn(0x556c95672600 0x556c956749e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-05T19:28:23.769 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:23.762+0000 7f21fd308e40 1 -- --> v1:172.21.15.118:6789/0 -- auth(proto 0 26 bytes epoch 0) v1 -- 0x556c955b8bd0 con 0x556c9566e020 2023-10-05T19:28:23.769 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:23.762+0000 7f21fd308e40 1 -- --> v2:172.21.15.118:3300/0 -- mon_getmap magic: 0 v1 -- 0x556c95575ee0 con 0x556c95672600 2023-10-05T19:28:23.770 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:23.762+0000 7f21fa63c700 1 --2- >> v2:172.21.15.118:3300/0 conn(0x556c95672600 0x556c956749e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-05T19:28:23.770 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:23.762+0000 7f21fa63c700 1 --2- >> v2:172.21.15.118:3300/0 conn(0x556c95672600 0x556c956749e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.118:3300/0 says I am v2:172.21.15.167:50612/0 (socket says 172.21.15.167:50612) 2023-10-05T19:28:23.770 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:23.762+0000 7f21fa63c700 1 -- 172.21.15.167:0/4223348112 learned_addr learned my addr 172.21.15.167:0/4223348112 (peer_addr_for_me v2:172.21.15.167:0/0) 2023-10-05T19:28:23.770 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:23.762+0000 7f21fae3d700 1 --1- 172.21.15.167:0/4223348112 >> v1:172.21.15.118:6789/0 conn(0x556c9566e020 0x556c9566e3f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.118:6789/0 says I am v1:172.21.15.167:37560/0 (socket says 172.21.15.167:37560) 2023-10-05T19:28:23.771 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:23.766+0000 7f21fa63c700 1 -- 172.21.15.167:0/4223348112 >> v1:172.21.15.118:6789/0 conn(0x556c9566e020 legacy=0x556c9566e3f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-05T19:28:23.771 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:23.766+0000 7f21fa63c700 1 -- 172.21.15.167:0/4223348112 --> v2:172.21.15.118:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x556c9566ec30 con 0x556c95672600 2023-10-05T19:28:23.771 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:23.766+0000 7f21fa63c700 1 --2- 172.21.15.167:0/4223348112 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x556c95672600 0x556c956749e0 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7f21e400a5c0 tx=0x7f21e4005e00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d4a8a321443cb6cf server_cookie=0 in_seq=0 out_seq=0 2023-10-05T19:28:23.771 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:23.766+0000 7f21f9e3b700 1 -- 172.21.15.167:0/4223348112 <== mon.0 v2:172.21.15.118:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f21e400e510 con 0x556c95672600 2023-10-05T19:28:23.771 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:23.766+0000 7f21f9e3b700 1 -- 172.21.15.167:0/4223348112 <== mon.0 v2:172.21.15.118:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21e400e670 con 0x556c95672600 2023-10-05T19:28:23.771 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:23.766+0000 7f21f9e3b700 1 -- 172.21.15.167:0/4223348112 <== mon.0 v2:172.21.15.118:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f21e400e920 con 0x556c95672600 2023-10-05T19:28:23.772 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:23.766+0000 7f21fd308e40 1 -- 172.21.15.167:0/4223348112 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x556c95672600 msgr2=0x556c956749e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-05T19:28:23.772 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:23.766+0000 7f21fd308e40 1 --2- 172.21.15.167:0/4223348112 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x556c95672600 0x556c956749e0 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7f21e400a5c0 tx=0x7f21e4005e00 comp rx=0 tx=0).stop 2023-10-05T19:28:23.772 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:23.766+0000 7f21fd308e40 1 -- 172.21.15.167:0/4223348112 shutdown_connections 2023-10-05T19:28:23.773 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:23.766+0000 7f21fd308e40 1 --2- 172.21.15.167:0/4223348112 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x556c95672600 0x556c956749e0 unknown :-1 s=CLOSED pgs=141 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-05T19:28:23.773 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:23.766+0000 7f21fd308e40 1 -- 172.21.15.167:0/4223348112 >> 172.21.15.167:0/4223348112 conn(0x556c955e1870 msgr2=0x556c9566c510 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-05T19:28:23.773 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:23.766+0000 7f21fd308e40 1 -- 172.21.15.167:0/4223348112 shutdown_connections 2023-10-05T19:28:23.773 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:23.766+0000 7f21fd308e40 1 -- 172.21.15.167:0/4223348112 wait complete. 2023-10-05T19:28:23.773 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:23.766+0000 7f21fd308e40 1 Processor -- start 2023-10-05T19:28:23.773 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:23.766+0000 7f21fd308e40 1 -- start start 2023-10-05T19:28:23.774 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:23.766+0000 7f21fd308e40 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x556c9566e020 0x556c9568d100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-05T19:28:23.774 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:23.766+0000 7f21fd308e40 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x556c95539e30 con 0x556c9566e020 2023-10-05T19:28:23.774 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:23.766+0000 7f21fae3d700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x556c9566e020 0x556c9568d100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-05T19:28:23.774 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:23.766+0000 7f21fae3d700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x556c9566e020 0x556c9568d100 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.118:3300/0 says I am v2:172.21.15.167:50626/0 (socket says 172.21.15.167:50626) 2023-10-05T19:28:23.774 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:23.766+0000 7f21fae3d700 1 -- 172.21.15.167:0/1139819866 learned_addr learned my addr 172.21.15.167:0/1139819866 (peer_addr_for_me v2:172.21.15.167:0/0) 2023-10-05T19:28:23.774 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:23.770+0000 7f21fae3d700 1 -- 172.21.15.167:0/1139819866 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f21e400a180 con 0x556c9566e020 2023-10-05T19:28:23.775 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:23.770+0000 7f21fae3d700 1 --2- 172.21.15.167:0/1139819866 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x556c9566e020 0x556c9568d100 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7f21f4001cb0 tx=0x7f21f4001860 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b3906146c98638e7 server_cookie=0 in_seq=0 out_seq=0 2023-10-05T19:28:23.775 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:23.770+0000 7f21f8e39700 1 -- 172.21.15.167:0/1139819866 <== mon.0 v2:172.21.15.118:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f21f4014070 con 0x556c9566e020 2023-10-05T19:28:23.775 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:23.770+0000 7f21f8e39700 1 -- 172.21.15.167:0/1139819866 <== mon.0 v2:172.21.15.118:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21f400bd40 con 0x556c9566e020 2023-10-05T19:28:23.775 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:23.770+0000 7f21f8e39700 1 -- 172.21.15.167:0/1139819866 <== mon.0 v2:172.21.15.118:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f21f400f450 con 0x556c9566e020 2023-10-05T19:28:23.775 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:23.770+0000 7f21fd308e40 1 -- 172.21.15.167:0/1139819866 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x556c9566e020 msgr2=0x556c9568d100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-05T19:28:23.775 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:23.770+0000 7f21fd308e40 1 --2- 172.21.15.167:0/1139819866 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x556c9566e020 0x556c9568d100 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7f21f4001cb0 tx=0x7f21f4001860 comp rx=0 tx=0).stop 2023-10-05T19:28:23.777 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:23.770+0000 7f21fd308e40 1 -- 172.21.15.167:0/1139819866 shutdown_connections 2023-10-05T19:28:23.777 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:23.770+0000 7f21fd308e40 1 --2- 172.21.15.167:0/1139819866 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x556c9566e020 0x556c9568d100 unknown :-1 s=CLOSED pgs=142 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-05T19:28:23.777 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:23.770+0000 7f21fd308e40 1 -- 172.21.15.167:0/1139819866 >> 172.21.15.167:0/1139819866 conn(0x556c955e1870 msgr2=0x556c95688cc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-05T19:28:23.777 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:23.770+0000 7f21fd308e40 1 -- 172.21.15.167:0/1139819866 shutdown_connections 2023-10-05T19:28:23.777 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:23.770+0000 7f21fd308e40 1 -- 172.21.15.167:0/1139819866 wait complete. 2023-10-05T19:28:23.777 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:23.770+0000 7f21fd308e40 1 Processor -- start 2023-10-05T19:28:23.777 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:23.770+0000 7f21fd308e40 1 -- start start 2023-10-05T19:28:23.777 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:23.770+0000 7f21fd308e40 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x556c9566e020 0x556c95693ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-05T19:28:23.777 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:23.770+0000 7f21fd308e40 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x556c9550ced0 con 0x556c9566e020 2023-10-05T19:28:23.777 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:23.770+0000 7f21fae3d700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x556c9566e020 0x556c95693ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-05T19:28:23.777 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:23.770+0000 7f21fae3d700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x556c9566e020 0x556c95693ed0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.118:3300/0 says I am v2:172.21.15.167:50630/0 (socket says 172.21.15.167:50630) 2023-10-05T19:28:23.778 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:23.770+0000 7f21fae3d700 1 -- 172.21.15.167:0/3301061136 learned_addr learned my addr 172.21.15.167:0/3301061136 (peer_addr_for_me v2:172.21.15.167:0/0) 2023-10-05T19:28:23.778 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:23.770+0000 7f21fae3d700 1 -- 172.21.15.167:0/3301061136 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x556c955b8bd0 con 0x556c9566e020 2023-10-05T19:28:23.779 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:23.770+0000 7f21fae3d700 1 --2- 172.21.15.167:0/3301061136 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x556c9566e020 0x556c95693ed0 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7f21f4000c00 tx=0x7f21f4016000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-05T19:28:23.779 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:23.774+0000 7f21f2ffd700 1 -- 172.21.15.167:0/3301061136 <== mon.0 v2:172.21.15.118:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f21f400beb0 con 0x556c9566e020 2023-10-05T19:28:23.779 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:23.774+0000 7f21f2ffd700 1 -- 172.21.15.167:0/3301061136 <== mon.0 v2:172.21.15.118:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21f400f450 con 0x556c9566e020 2023-10-05T19:28:23.779 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:23.774+0000 7f21fd308e40 1 -- 172.21.15.167:0/3301061136 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x556c95694410 con 0x556c9566e020 2023-10-05T19:28:23.779 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:23.774+0000 7f21f2ffd700 1 -- 172.21.15.167:0/3301061136 <== mon.0 v2:172.21.15.118:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f21f401cc80 con 0x556c9566e020 2023-10-05T19:28:23.779 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:23.774+0000 7f21fd308e40 1 -- 172.21.15.167:0/3301061136 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x556c95694850 con 0x556c9566e020 2023-10-05T19:28:23.779 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:23.774+0000 7f21fd308e40 20 rados->read ofs=0 len=0 2023-10-05T19:28:23.780 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:23.774+0000 7f21f2ffd700 1 -- 172.21.15.167:0/3301061136 <== mon.0 v2:172.21.15.118:3300/0 4 ==== mgrmap(e 5) v1 ==== 40329+0+0 (secure 0 0 0) 0x7f21f4023020 con 0x556c9566e020 2023-10-05T19:28:23.781 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:23.774+0000 7f21f2ffd700 1 --2- 172.21.15.167:0/3301061136 >> [v2:172.21.15.167:6824/29889,v1:172.21.15.167:6825/29889] conn(0x7f21dc030c50 0x7f21dc033100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-05T19:28:23.781 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:23.774+0000 7f21f2ffd700 1 -- 172.21.15.167:0/3301061136 <== mon.0 v2:172.21.15.118:3300/0 5 ==== osd_map(30..30 src has 1..30) v4 ==== 6057+0+0 (secure 0 0 0) 0x7f21f4008030 con 0x556c9566e020 2023-10-05T19:28:23.781 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:23.774+0000 7f21fa63c700 1 --2- 172.21.15.167:0/3301061136 >> [v2:172.21.15.167:6824/29889,v1:172.21.15.167:6825/29889] conn(0x7f21dc030c50 0x7f21dc033100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-05T19:28:23.781 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:23.774+0000 7f21fd308e40 1 --2- 172.21.15.167:0/3301061136 >> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] conn(0x556c95672600 0x556c9569f9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-05T19:28:23.782 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:23.774+0000 7f21fd308e40 1 -- 172.21.15.167:0/3301061136 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:1 5.12 5:49953fa1:::default.realm:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x556c956a00c0 con 0x556c95672600 2023-10-05T19:28:23.782 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:23.774+0000 7f21fb63e700 1 --2- 172.21.15.167:0/3301061136 >> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] conn(0x556c95672600 0x556c9569f9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-05T19:28:23.782 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:23.778+0000 7f21fa63c700 1 --2- 172.21.15.167:0/3301061136 >> [v2:172.21.15.167:6824/29889,v1:172.21.15.167:6825/29889] conn(0x7f21dc030c50 0x7f21dc033100 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f21e4005e40 tx=0x7f21e4013000 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-05T19:28:23.782 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:23.778+0000 7f21fb63e700 1 --2- 172.21.15.167:0/3301061136 >> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] conn(0x556c95672600 0x556c9569f9c0 crc :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-05T19:28:23.783 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:23.778+0000 7f21fb63e700 1 -- 172.21.15.167:0/3301061136 <== osd.1 v2:172.21.15.167:6800/30035 1 ==== osd_op_reply(1 default.realm [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 ==== 157+0+0 (crc 0 0 0) 0x7f21ec00a500 con 0x556c95672600 2023-10-05T19:28:23.783 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:23.778+0000 7f21fd308e40 20 rados_obj.operate() r=-2 bl.length=0 2023-10-05T19:28:23.783 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:23.778+0000 7f21fd308e40 20 realm 2023-10-05T19:28:23.783 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:23.778+0000 7f21fd308e40 20 rados->read ofs=0 len=0 2023-10-05T19:28:23.783 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:23.778+0000 7f21fd308e40 1 -- 172.21.15.167:0/3301061136 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:2 5.12 5:49953fa1:::default.realm:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x556c9569d020 con 0x556c95672600 2023-10-05T19:28:23.784 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:23.778+0000 7f21fb63e700 1 -- 172.21.15.167:0/3301061136 <== osd.1 v2:172.21.15.167:6800/30035 2 ==== osd_op_reply(2 default.realm [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 ==== 157+0+0 (crc 0 0 0) 0x7f21ec00a500 con 0x556c95672600 2023-10-05T19:28:23.785 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:23.778+0000 7f21fd308e40 20 rados_obj.operate() r=-2 bl.length=0 2023-10-05T19:28:23.785 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:23.778+0000 7f21fd308e40 4 RGWPeriod::init failed to init realm id : (2) No such file or directory 2023-10-05T19:28:23.785 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:23.778+0000 7f21fd308e40 20 rados->read ofs=0 len=0 2023-10-05T19:28:23.785 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:23.778+0000 7f21fd308e40 1 -- 172.21.15.167:0/3301061136 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:3 5.12 5:49953fa1:::default.realm:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x556c956a0ba0 con 0x556c95672600 2023-10-05T19:28:23.785 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:23.778+0000 7f21fb63e700 1 -- 172.21.15.167:0/3301061136 <== osd.1 v2:172.21.15.167:6800/30035 3 ==== osd_op_reply(3 default.realm [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 ==== 157+0+0 (crc 0 0 0) 0x7f21ec00a500 con 0x556c95672600 2023-10-05T19:28:23.785 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:23.778+0000 7f21fd308e40 20 rados_obj.operate() r=-2 bl.length=0 2023-10-05T19:28:23.785 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:23.778+0000 7f21fd308e40 20 rados->read ofs=0 len=0 2023-10-05T19:28:23.785 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:23.778+0000 7f21fd308e40 1 -- 172.21.15.167:0/3301061136 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:4 5.f 5:f43b6ece:::zone_names.default:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x556c956a07b0 con 0x556c95672600 2023-10-05T19:28:23.785 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:23.778+0000 7f21fb63e700 1 -- 172.21.15.167:0/3301061136 <== osd.1 v2:172.21.15.167:6800/30035 4 ==== osd_op_reply(4 zone_names.default [read 0~46 out=46b] v0'0 uv1 ondisk = 0) v8 ==== 162+0+46 (crc 0 0 0) 0x7f21ec00a500 con 0x556c95672600 2023-10-05T19:28:23.785 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:23.778+0000 7f21fd308e40 20 rados_obj.operate() r=0 bl.length=46 2023-10-05T19:28:23.785 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:23.778+0000 7f21fd308e40 20 rados->read ofs=0 len=0 2023-10-05T19:28:23.786 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:23.778+0000 7f21fd308e40 1 -- 172.21.15.167:0/3301061136 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:5 5.1d 5:b8374eb4:::zone_info.9a9df965-6612-45d4-a45f-6e3c889bacc2:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x556c956a1a30 con 0x556c95672600 2023-10-05T19:28:23.786 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:23.778+0000 7f21fb63e700 1 -- 172.21.15.167:0/3301061136 <== osd.1 v2:172.21.15.167:6800/30035 5 ==== osd_op_reply(5 zone_info.9a9df965-6612-45d4-a45f-6e3c889bacc2 [read 0~874 out=874b] v0'0 uv1 ondisk = 0) v8 ==== 190+0+874 (crc 0 0 0) 0x7f21ec00a500 con 0x556c95672600 2023-10-05T19:28:23.787 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:23.778+0000 7f21fd308e40 20 rados_obj.operate() r=0 bl.length=874 2023-10-05T19:28:23.787 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:23.778+0000 7f21fd308e40 20 searching for the correct realm 2023-10-05T19:28:23.787 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:23.778+0000 7f21fd308e40 1 --2- 172.21.15.167:0/3301061136 >> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] conn(0x556c956a37f0 0x556c956a5bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-05T19:28:23.787 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:23.778+0000 7f21fd308e40 1 -- 172.21.15.167:0/3301061136 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:6 5.0 5:00000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x556c956a6110 con 0x556c956a37f0 2023-10-05T19:28:23.787 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:23.778+0000 7f21fae3d700 1 --2- 172.21.15.167:0/3301061136 >> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] conn(0x556c956a37f0 0x556c956a5bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-05T19:28:23.787 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:23.782+0000 7f21fae3d700 1 --2- 172.21.15.167:0/3301061136 >> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] conn(0x556c956a37f0 0x556c956a5bd0 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-05T19:28:23.788 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:23.782+0000 7f21fae3d700 1 -- 172.21.15.167:0/3301061136 <== osd.3 v2:172.21.15.167:6808/30036 1 ==== osd_op_reply(6 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f21f4016070 con 0x556c956a37f0 2023-10-05T19:28:23.788 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:23.782+0000 7f21fae3d700 1 --2- 172.21.15.167:0/3301061136 >> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] conn(0x7f21f401a070 0x7f21f4042cb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-05T19:28:23.788 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:23.782+0000 7f21fae3d700 1 -- 172.21.15.167:0/3301061136 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:7 5.10 5:08000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x556c956a6110 con 0x7f21f401a070 2023-10-05T19:28:23.789 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:23.782+0000 7f21fa63c700 1 --2- 172.21.15.167:0/3301061136 >> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] conn(0x7f21f401a070 0x7f21f4042cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-05T19:28:23.789 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:23.782+0000 7f21fa63c700 1 --2- 172.21.15.167:0/3301061136 >> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] conn(0x7f21f401a070 0x7f21f4042cb0 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-05T19:28:23.790 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:23.782+0000 7f21fa63c700 1 -- 172.21.15.167:0/3301061136 <== osd.2 v2:172.21.15.167:6816/30037 1 ==== osd_op_reply(7 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f21e4017050 con 0x7f21f401a070 2023-10-05T19:28:23.790 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:23.786+0000 7f21fa63c700 1 --2- 172.21.15.167:0/3301061136 >> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] conn(0x7f21e4010070 0x7f21e401c430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-05T19:28:23.790 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:23.786+0000 7f21fa63c700 1 -- 172.21.15.167:0/3301061136 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:8 5.8 5:10000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x556c956a6110 con 0x7f21e4010070 2023-10-05T19:28:23.790 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:23.786+0000 7f21fb63e700 1 --2- 172.21.15.167:0/3301061136 >> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] conn(0x7f21e4010070 0x7f21e401c430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-05T19:28:23.791 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:23.786+0000 7f21fb63e700 1 --2- 172.21.15.167:0/3301061136 >> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] conn(0x7f21e4010070 0x7f21e401c430 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-05T19:28:23.792 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:23.786+0000 7f21fb63e700 1 -- 172.21.15.167:0/3301061136 <== osd.0 v2:172.21.15.118:6800/27690 1 ==== osd_op_reply(8 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f21ec014040 con 0x7f21e4010070 2023-10-05T19:28:23.792 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:23.786+0000 7f21fb63e700 1 -- 172.21.15.167:0/3301061136 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:9 5.18 5:18000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x556c956a6110 con 0x556c95672600 2023-10-05T19:28:23.792 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:23.786+0000 7f21fb63e700 1 -- 172.21.15.167:0/3301061136 <== osd.1 v2:172.21.15.167:6800/30035 6 ==== osd_op_reply(9 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f21ec014040 con 0x556c95672600 2023-10-05T19:28:23.792 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:23.786+0000 7f21fb63e700 1 -- 172.21.15.167:0/3301061136 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:10 5.4 5:20000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x556c956a6110 con 0x556c95672600 2023-10-05T19:28:23.793 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:23.786+0000 7f21fb63e700 1 -- 172.21.15.167:0/3301061136 <== osd.1 v2:172.21.15.167:6800/30035 7 ==== osd_op_reply(10 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f21ec014040 con 0x556c95672600 2023-10-05T19:28:23.793 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:23.786+0000 7f21fb63e700 1 -- 172.21.15.167:0/3301061136 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:11 5.14 5:28000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x556c956a6110 con 0x7f21e4010070 2023-10-05T19:28:23.794 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:23.790+0000 7f21fb63e700 1 -- 172.21.15.167:0/3301061136 <== osd.0 v2:172.21.15.118:6800/27690 2 ==== osd_op_reply(11 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f21ec014040 con 0x7f21e4010070 2023-10-05T19:28:23.794 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:23.790+0000 7f21fb63e700 1 -- 172.21.15.167:0/3301061136 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:12 5.c 5:30000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x556c956a6110 con 0x7f21e4010070 2023-10-05T19:28:23.795 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:23.790+0000 7f21fb63e700 1 -- 172.21.15.167:0/3301061136 <== osd.0 v2:172.21.15.118:6800/27690 3 ==== osd_op_reply(12 [pgnls start_epoch 30 out=112b] v23'1 uv1 ondisk = 1) v8 ==== 144+0+112 (crc 0 0 0) 0x7f21ec014040 con 0x7f21e4010070 2023-10-05T19:28:23.795 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:23.790+0000 7f21fb63e700 1 -- 172.21.15.167:0/3301061136 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:13 5.1c 5:38000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x556c956a6110 con 0x7f21f401a070 2023-10-05T19:28:23.796 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:23.790+0000 7f21fa63c700 1 -- 172.21.15.167:0/3301061136 <== osd.2 v2:172.21.15.167:6816/30037 2 ==== osd_op_reply(13 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f21e4017050 con 0x7f21f401a070 2023-10-05T19:28:23.796 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:23.790+0000 7f21fa63c700 1 -- 172.21.15.167:0/3301061136 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:14 5.2 5:40000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x556c956a6110 con 0x556c956a37f0 2023-10-05T19:28:23.796 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:23.790+0000 7f21fae3d700 1 -- 172.21.15.167:0/3301061136 <== osd.3 v2:172.21.15.167:6808/30036 2 ==== osd_op_reply(14 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f21f4016070 con 0x556c956a37f0 2023-10-05T19:28:23.796 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:23.790+0000 7f21fae3d700 1 -- 172.21.15.167:0/3301061136 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:15 5.12 5:48000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x556c956a6110 con 0x556c95672600 2023-10-05T19:28:23.797 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:23.790+0000 7f21fb63e700 1 -- 172.21.15.167:0/3301061136 <== osd.1 v2:172.21.15.167:6800/30035 8 ==== osd_op_reply(15 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f21ec014040 con 0x556c95672600 2023-10-05T19:28:23.797 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:23.790+0000 7f21fb63e700 1 -- 172.21.15.167:0/3301061136 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:16 5.a 5:50000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x556c956a6110 con 0x7f21e4010070 2023-10-05T19:28:23.798 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:23.794+0000 7f21fb63e700 1 -- 172.21.15.167:0/3301061136 <== osd.0 v2:172.21.15.118:6800/27690 4 ==== osd_op_reply(16 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f21ec022040 con 0x7f21e4010070 2023-10-05T19:28:23.798 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:23.794+0000 7f21fb63e700 1 -- 172.21.15.167:0/3301061136 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:17 5.1a 5:58000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x556c956a6110 con 0x556c95672600 2023-10-05T19:28:23.799 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:23.794+0000 7f21fb63e700 1 -- 172.21.15.167:0/3301061136 <== osd.1 v2:172.21.15.167:6800/30035 9 ==== osd_op_reply(17 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f21ec022040 con 0x556c95672600 2023-10-05T19:28:23.799 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:23.794+0000 7f21fb63e700 1 -- 172.21.15.167:0/3301061136 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:18 5.6 5:60000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x556c956a6110 con 0x7f21f401a070 2023-10-05T19:28:23.800 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:23.794+0000 7f21fa63c700 1 -- 172.21.15.167:0/3301061136 <== osd.2 v2:172.21.15.167:6816/30037 3 ==== osd_op_reply(18 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f21e4017050 con 0x7f21f401a070 2023-10-05T19:28:23.800 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:23.794+0000 7f21fa63c700 1 -- 172.21.15.167:0/3301061136 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:19 5.16 5:68000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x556c956a6110 con 0x556c956a37f0 2023-10-05T19:28:23.801 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:23.794+0000 7f21fae3d700 1 -- 172.21.15.167:0/3301061136 <== osd.3 v2:172.21.15.167:6808/30036 3 ==== osd_op_reply(19 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f21f4016070 con 0x556c956a37f0 2023-10-05T19:28:23.801 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:23.794+0000 7f21fae3d700 1 -- 172.21.15.167:0/3301061136 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:20 5.e 5:70000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x556c956a6110 con 0x556c956a37f0 2023-10-05T19:28:23.801 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:23.794+0000 7f21fae3d700 1 -- 172.21.15.167:0/3301061136 <== osd.3 v2:172.21.15.167:6808/30036 4 ==== osd_op_reply(20 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f21f4016070 con 0x556c956a37f0 2023-10-05T19:28:23.802 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:23.794+0000 7f21fae3d700 1 -- 172.21.15.167:0/3301061136 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:21 5.1e 5:78000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x556c956a6110 con 0x7f21e4010070 2023-10-05T19:28:23.803 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:23.798+0000 7f21fb63e700 1 -- 172.21.15.167:0/3301061136 <== osd.0 v2:172.21.15.118:6800/27690 5 ==== osd_op_reply(21 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f21ec022040 con 0x7f21e4010070 2023-10-05T19:28:23.803 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:23.798+0000 7f21fb63e700 1 -- 172.21.15.167:0/3301061136 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:22 5.1 5:80000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x556c956a6110 con 0x556c956a37f0 2023-10-05T19:28:23.803 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:23.798+0000 7f21fae3d700 1 -- 172.21.15.167:0/3301061136 <== osd.3 v2:172.21.15.167:6808/30036 5 ==== osd_op_reply(22 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f21f4039040 con 0x556c956a37f0 2023-10-05T19:28:23.804 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:23.798+0000 7f21fae3d700 1 -- 172.21.15.167:0/3301061136 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:23 5.11 5:88000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x556c956a6110 con 0x556c956a37f0 2023-10-05T19:28:23.804 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:23.798+0000 7f21fae3d700 1 -- 172.21.15.167:0/3301061136 <== osd.3 v2:172.21.15.167:6808/30036 6 ==== osd_op_reply(23 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f21f4039040 con 0x556c956a37f0 2023-10-05T19:28:23.804 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:23.798+0000 7f21fae3d700 1 -- 172.21.15.167:0/3301061136 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:24 5.9 5:90000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x556c956a6110 con 0x556c95672600 2023-10-05T19:28:23.805 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:23.798+0000 7f21fb63e700 1 -- 172.21.15.167:0/3301061136 <== osd.1 v2:172.21.15.167:6800/30035 10 ==== osd_op_reply(24 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f21ec022040 con 0x556c95672600 2023-10-05T19:28:23.805 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:23.798+0000 7f21fb63e700 1 -- 172.21.15.167:0/3301061136 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:25 5.19 5:98000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x556c956a6110 con 0x556c95672600 2023-10-05T19:28:23.806 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:23.798+0000 7f21fb63e700 1 -- 172.21.15.167:0/3301061136 <== osd.1 v2:172.21.15.167:6800/30035 11 ==== osd_op_reply(25 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f21ec022040 con 0x556c95672600 2023-10-05T19:28:23.806 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:23.802+0000 7f21fb63e700 1 -- 172.21.15.167:0/3301061136 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:26 5.5 5:a0000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x556c956a6110 con 0x7f21e4010070 2023-10-05T19:28:23.807 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:23.802+0000 7f21fb63e700 1 -- 172.21.15.167:0/3301061136 <== osd.0 v2:172.21.15.118:6800/27690 6 ==== osd_op_reply(26 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f21ec022040 con 0x7f21e4010070 2023-10-05T19:28:23.807 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:23.802+0000 7f21fb63e700 1 -- 172.21.15.167:0/3301061136 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:27 5.15 5:a8000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x556c956a6110 con 0x556c956a37f0 2023-10-05T19:28:23.808 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:23.802+0000 7f21fae3d700 1 -- 172.21.15.167:0/3301061136 <== osd.3 v2:172.21.15.167:6808/30036 7 ==== osd_op_reply(27 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f21f4039040 con 0x556c956a37f0 2023-10-05T19:28:23.808 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:23.802+0000 7f21fae3d700 1 -- 172.21.15.167:0/3301061136 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:28 5.d 5:b0000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x556c956a6110 con 0x7f21f401a070 2023-10-05T19:28:23.808 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:23.802+0000 7f21fa63c700 1 -- 172.21.15.167:0/3301061136 <== osd.2 v2:172.21.15.167:6816/30037 4 ==== osd_op_reply(28 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f21e4017050 con 0x7f21f401a070 2023-10-05T19:28:23.808 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:23.802+0000 7f21fa63c700 1 -- 172.21.15.167:0/3301061136 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:29 5.1d 5:b8000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x556c956a6110 con 0x556c95672600 2023-10-05T19:28:23.809 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:23.802+0000 7f21fb63e700 1 -- 172.21.15.167:0/3301061136 <== osd.1 v2:172.21.15.167:6800/30035 12 ==== osd_op_reply(29 [pgnls start_epoch 30 out=107b] v23'1 uv1 ondisk = 1) v8 ==== 144+0+107 (crc 0 0 0) 0x7f21ec022040 con 0x556c95672600 2023-10-05T19:28:23.809 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:23.802+0000 7f21fb63e700 1 -- 172.21.15.167:0/3301061136 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:30 5.3 5:c0000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x556c956a6110 con 0x7f21e4010070 2023-10-05T19:28:23.810 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:23.806+0000 7f21fb63e700 1 -- 172.21.15.167:0/3301061136 <== osd.0 v2:172.21.15.118:6800/27690 7 ==== osd_op_reply(30 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f21ec022040 con 0x7f21e4010070 2023-10-05T19:28:23.811 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:23.806+0000 7f21fb63e700 1 -- 172.21.15.167:0/3301061136 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:31 5.13 5:c8000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x556c956a6110 con 0x556c956a37f0 2023-10-05T19:28:23.811 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:23.806+0000 7f21fae3d700 1 -- 172.21.15.167:0/3301061136 <== osd.3 v2:172.21.15.167:6808/30036 8 ==== osd_op_reply(31 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f21f4039040 con 0x556c956a37f0 2023-10-05T19:28:23.812 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:23.806+0000 7f21fae3d700 1 -- 172.21.15.167:0/3301061136 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:32 5.b 5:d0000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x556c956a6110 con 0x7f21e4010070 2023-10-05T19:28:23.812 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:23.806+0000 7f21fb63e700 1 -- 172.21.15.167:0/3301061136 <== osd.0 v2:172.21.15.118:6800/27690 8 ==== osd_op_reply(32 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f21ec022040 con 0x7f21e4010070 2023-10-05T19:28:23.813 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:23.806+0000 7f21fb63e700 1 -- 172.21.15.167:0/3301061136 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:33 5.1b 5:d8000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x556c956a6110 con 0x7f21f401a070 2023-10-05T19:28:23.813 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:23.806+0000 7f21fa63c700 1 -- 172.21.15.167:0/3301061136 <== osd.2 v2:172.21.15.167:6816/30037 5 ==== osd_op_reply(33 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f21e4017050 con 0x7f21f401a070 2023-10-05T19:28:23.813 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:23.806+0000 7f21fa63c700 1 -- 172.21.15.167:0/3301061136 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:34 5.7 5:e0000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x556c956a6110 con 0x556c956a37f0 2023-10-05T19:28:23.814 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:23.810+0000 7f21fae3d700 1 -- 172.21.15.167:0/3301061136 <== osd.3 v2:172.21.15.167:6808/30036 9 ==== osd_op_reply(34 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f21f4039040 con 0x556c956a37f0 2023-10-05T19:28:23.814 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:23.810+0000 7f21fae3d700 1 -- 172.21.15.167:0/3301061136 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:35 5.17 5:e8000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x556c956a6110 con 0x7f21e4010070 2023-10-05T19:28:23.815 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:23.810+0000 7f21fb63e700 1 -- 172.21.15.167:0/3301061136 <== osd.0 v2:172.21.15.118:6800/27690 9 ==== osd_op_reply(35 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f21ec022040 con 0x7f21e4010070 2023-10-05T19:28:23.815 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:23.810+0000 7f21fb63e700 1 -- 172.21.15.167:0/3301061136 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:36 5.f 5:f0000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x556c956a6110 con 0x556c95672600 2023-10-05T19:28:23.816 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:23.810+0000 7f21fb63e700 1 -- 172.21.15.167:0/3301061136 <== osd.1 v2:172.21.15.167:6800/30035 13 ==== osd_op_reply(36 [pgnls start_epoch 30 out=115b] v23'2 uv2 ondisk = 1) v8 ==== 144+0+115 (crc 0 0 0) 0x7f21ec02f080 con 0x556c95672600 2023-10-05T19:28:23.816 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:23.810+0000 7f21fb63e700 1 -- 172.21.15.167:0/3301061136 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:37 5.1f 5:f8000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x556c956a6110 con 0x7f21f401a070 2023-10-05T19:28:23.817 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:23.810+0000 7f21fa63c700 1 -- 172.21.15.167:0/3301061136 <== osd.2 v2:172.21.15.167:6816/30037 6 ==== osd_op_reply(37 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f21e4017050 con 0x7f21f401a070 2023-10-05T19:28:23.817 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:23.810+0000 7f21fd308e40 20 RGWRados::pool_iterate: got zonegroup_info.4f1f719d-9648-417b-b10c-e5d939825419 2023-10-05T19:28:23.818 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:23.810+0000 7f21fd308e40 20 RGWRados::pool_iterate: got zone_info.9a9df965-6612-45d4-a45f-6e3c889bacc2 2023-10-05T19:28:23.818 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:23.810+0000 7f21fd308e40 20 RGWRados::pool_iterate: got zone_names.default 2023-10-05T19:28:23.818 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:23.810+0000 7f21fd308e40 20 RGWRados::pool_iterate: got zonegroups_names.default 2023-10-05T19:28:23.818 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:23.810+0000 7f21fd308e40 20 rados->read ofs=0 len=0 2023-10-05T19:28:23.818 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:23.810+0000 7f21fd308e40 1 -- 172.21.15.167:0/3301061136 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:38 5.12 5:49953fa1:::default.realm:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x556c956a25a0 con 0x556c95672600 2023-10-05T19:28:23.819 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:23.814+0000 7f21fb63e700 1 -- 172.21.15.167:0/3301061136 <== osd.1 v2:172.21.15.167:6800/30035 14 ==== osd_op_reply(38 default.realm [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 ==== 157+0+0 (crc 0 0 0) 0x7f21ec02f080 con 0x556c95672600 2023-10-05T19:28:23.819 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:23.814+0000 7f21fd308e40 20 rados_obj.operate() r=-2 bl.length=0 2023-10-05T19:28:23.819 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:23.814+0000 7f21fd308e40 20 rados->read ofs=0 len=0 2023-10-05T19:28:23.819 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:23.814+0000 7f21fd308e40 1 -- 172.21.15.167:0/3301061136 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:39 5.f 5:f4c53578:::zonegroups_names.default:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x556c956a2e90 con 0x556c95672600 2023-10-05T19:28:23.819 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:23.814+0000 7f21fb63e700 1 -- 172.21.15.167:0/3301061136 <== osd.1 v2:172.21.15.167:6800/30035 15 ==== osd_op_reply(39 zonegroups_names.default [read 0~46 out=46b] v0'0 uv2 ondisk = 0) v8 ==== 168+0+46 (crc 0 0 0) 0x7f21ec02f080 con 0x556c95672600 2023-10-05T19:28:23.819 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:23.814+0000 7f21fd308e40 20 rados_obj.operate() r=0 bl.length=46 2023-10-05T19:28:23.819 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:23.814+0000 7f21fd308e40 20 rados->read ofs=0 len=0 2023-10-05T19:28:23.820 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:23.814+0000 7f21fd308e40 1 -- 172.21.15.167:0/3301061136 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:40 5.c 5:30d299a5:::zonegroup_info.4f1f719d-9648-417b-b10c-e5d939825419:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x556c956a6550 con 0x7f21e4010070 2023-10-05T19:28:23.820 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:23.814+0000 7f21fb63e700 1 -- 172.21.15.167:0/3301061136 <== osd.0 v2:172.21.15.118:6800/27690 10 ==== osd_op_reply(40 zonegroup_info.4f1f719d-9648-417b-b10c-e5d939825419 [read 0~362 out=362b] v0'0 uv1 ondisk = 0) v8 ==== 195+0+362 (crc 0 0 0) 0x7f21ec02f080 con 0x7f21e4010070 2023-10-05T19:28:23.820 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:23.814+0000 7f21fd308e40 20 rados_obj.operate() r=0 bl.length=362 2023-10-05T19:28:23.820 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:23.814+0000 7f21fd308e40 20 rados->read ofs=0 len=0 2023-10-05T19:28:23.820 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:23.814+0000 7f21fd308e40 1 -- 172.21.15.167:0/3301061136 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:41 5.13 5:cbbf6c60:::converted:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x556c956a6550 con 0x556c956a37f0 2023-10-05T19:28:23.821 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:23.814+0000 7f21fae3d700 1 -- 172.21.15.167:0/3301061136 <== osd.3 v2:172.21.15.167:6808/30036 10 ==== osd_op_reply(41 converted [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 ==== 153+0+0 (crc 0 0 0) 0x7f21f4039040 con 0x556c956a37f0 2023-10-05T19:28:23.822 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:23.814+0000 7f21fd308e40 20 rados_obj.operate() r=-2 bl.length=0 2023-10-05T19:28:23.822 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:23.814+0000 7f21fd308e40 20 rados->read ofs=0 len=0 2023-10-05T19:28:23.822 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:23.814+0000 7f21fd308e40 1 -- 172.21.15.167:0/3301061136 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:42 5.12 5:49953fa1:::default.realm:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x556c956a6900 con 0x556c95672600 2023-10-05T19:28:23.822 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:23.814+0000 7f21fb63e700 1 -- 172.21.15.167:0/3301061136 <== osd.1 v2:172.21.15.167:6800/30035 16 ==== osd_op_reply(42 default.realm [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 ==== 157+0+0 (crc 0 0 0) 0x7f21ec02f080 con 0x556c95672600 2023-10-05T19:28:23.822 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:23.814+0000 7f21fd308e40 20 rados_obj.operate() r=-2 bl.length=0 2023-10-05T19:28:23.822 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:23.814+0000 7f21fd308e40 20 rados->read ofs=0 len=0 2023-10-05T19:28:23.822 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:23.814+0000 7f21fd308e40 1 -- 172.21.15.167:0/3301061136 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:43 5.f 5:f4c53578:::zonegroups_names.default:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x556c956a2ac0 con 0x556c95672600 2023-10-05T19:28:23.823 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:23.818+0000 7f21fb63e700 1 -- 172.21.15.167:0/3301061136 <== osd.1 v2:172.21.15.167:6800/30035 17 ==== osd_op_reply(43 zonegroups_names.default [read 0~46 out=46b] v0'0 uv2 ondisk = 0) v8 ==== 168+0+46 (crc 0 0 0) 0x7f21ec02f080 con 0x556c95672600 2023-10-05T19:28:23.823 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:23.818+0000 7f21fd308e40 20 rados_obj.operate() r=0 bl.length=46 2023-10-05T19:28:23.823 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:23.818+0000 7f21fd308e40 1 -- 172.21.15.167:0/3301061136 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:44 5.0 5:00000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x556c956a2a90 con 0x556c956a37f0 2023-10-05T19:28:23.825 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:23.818+0000 7f21fae3d700 1 -- 172.21.15.167:0/3301061136 <== osd.3 v2:172.21.15.167:6808/30036 11 ==== osd_op_reply(44 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f21f4039040 con 0x556c956a37f0 2023-10-05T19:28:23.825 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:23.818+0000 7f21fae3d700 1 -- 172.21.15.167:0/3301061136 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:45 5.10 5:08000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x556c956a2a90 con 0x7f21f401a070 2023-10-05T19:28:23.825 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:23.818+0000 7f21fa63c700 1 -- 172.21.15.167:0/3301061136 <== osd.2 v2:172.21.15.167:6816/30037 7 ==== osd_op_reply(45 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f21e4017050 con 0x7f21f401a070 2023-10-05T19:28:23.825 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:23.818+0000 7f21fa63c700 1 -- 172.21.15.167:0/3301061136 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:46 5.8 5:10000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x556c956a2a90 con 0x7f21e4010070 2023-10-05T19:28:23.826 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:23.822+0000 7f21fb63e700 1 -- 172.21.15.167:0/3301061136 <== osd.0 v2:172.21.15.118:6800/27690 11 ==== osd_op_reply(46 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f21ec02f080 con 0x7f21e4010070 2023-10-05T19:28:23.827 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:23.822+0000 7f21fb63e700 1 -- 172.21.15.167:0/3301061136 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:47 5.18 5:18000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x556c956a2e90 con 0x556c95672600 2023-10-05T19:28:23.827 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:23.822+0000 7f21fb63e700 1 -- 172.21.15.167:0/3301061136 <== osd.1 v2:172.21.15.167:6800/30035 18 ==== osd_op_reply(47 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f21ec02f080 con 0x556c95672600 2023-10-05T19:28:23.827 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:23.822+0000 7f21fb63e700 1 -- 172.21.15.167:0/3301061136 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:48 5.4 5:20000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x556c956a2e90 con 0x556c95672600 2023-10-05T19:28:23.828 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:23.822+0000 7f21fb63e700 1 -- 172.21.15.167:0/3301061136 <== osd.1 v2:172.21.15.167:6800/30035 19 ==== osd_op_reply(48 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f21ec02f080 con 0x556c95672600 2023-10-05T19:28:23.828 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:23.822+0000 7f21fb63e700 1 -- 172.21.15.167:0/3301061136 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:49 5.14 5:28000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x556c956a2e90 con 0x7f21e4010070 2023-10-05T19:28:23.829 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:23.822+0000 7f21fb63e700 1 -- 172.21.15.167:0/3301061136 <== osd.0 v2:172.21.15.118:6800/27690 12 ==== osd_op_reply(49 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f21ec02f080 con 0x7f21e4010070 2023-10-05T19:28:23.829 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:23.822+0000 7f21fb63e700 1 -- 172.21.15.167:0/3301061136 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:50 5.c 5:30000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x556c956a2e90 con 0x7f21e4010070 2023-10-05T19:28:23.830 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:23.826+0000 7f21fb63e700 1 -- 172.21.15.167:0/3301061136 <== osd.0 v2:172.21.15.118:6800/27690 13 ==== osd_op_reply(50 [pgnls start_epoch 30 out=112b] v23'1 uv1 ondisk = 1) v8 ==== 144+0+112 (crc 0 0 0) 0x7f21ec02f080 con 0x7f21e4010070 2023-10-05T19:28:23.830 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:23.826+0000 7f21fb63e700 1 -- 172.21.15.167:0/3301061136 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:51 5.1c 5:38000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x556c956a2e90 con 0x7f21f401a070 2023-10-05T19:28:23.831 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:23.826+0000 7f21fa63c700 1 -- 172.21.15.167:0/3301061136 <== osd.2 v2:172.21.15.167:6816/30037 8 ==== osd_op_reply(51 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f21e4017050 con 0x7f21f401a070 2023-10-05T19:28:23.831 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:23.826+0000 7f21fa63c700 1 -- 172.21.15.167:0/3301061136 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:52 5.2 5:40000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x556c956a2e90 con 0x556c956a37f0 2023-10-05T19:28:23.832 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:23.826+0000 7f21fae3d700 1 -- 172.21.15.167:0/3301061136 <== osd.3 v2:172.21.15.167:6808/30036 12 ==== osd_op_reply(52 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f21f4039040 con 0x556c956a37f0 2023-10-05T19:28:23.832 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:23.826+0000 7f21fae3d700 1 -- 172.21.15.167:0/3301061136 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:53 5.12 5:48000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x556c956a2e90 con 0x556c95672600 2023-10-05T19:28:23.832 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:23.826+0000 7f21fb63e700 1 -- 172.21.15.167:0/3301061136 <== osd.1 v2:172.21.15.167:6800/30035 20 ==== osd_op_reply(53 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f21ec02f080 con 0x556c95672600 2023-10-05T19:28:23.832 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:23.826+0000 7f21fb63e700 1 -- 172.21.15.167:0/3301061136 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:54 5.a 5:50000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x556c956a2e90 con 0x7f21e4010070 2023-10-05T19:28:23.833 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:23.826+0000 7f21fb63e700 1 -- 172.21.15.167:0/3301061136 <== osd.0 v2:172.21.15.118:6800/27690 14 ==== osd_op_reply(54 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f21ec02f080 con 0x7f21e4010070 2023-10-05T19:28:23.833 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:23.826+0000 7f21fb63e700 1 -- 172.21.15.167:0/3301061136 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:55 5.1a 5:58000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x556c956a2e90 con 0x556c95672600 2023-10-05T19:28:23.834 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:23.830+0000 7f21fb63e700 1 -- 172.21.15.167:0/3301061136 <== osd.1 v2:172.21.15.167:6800/30035 21 ==== osd_op_reply(55 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f21ec02f080 con 0x556c95672600 2023-10-05T19:28:23.834 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:23.830+0000 7f21fb63e700 1 -- 172.21.15.167:0/3301061136 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:56 5.6 5:60000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x556c956a2e90 con 0x7f21f401a070 2023-10-05T19:28:23.835 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:23.830+0000 7f21fa63c700 1 -- 172.21.15.167:0/3301061136 <== osd.2 v2:172.21.15.167:6816/30037 9 ==== osd_op_reply(56 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f21e4017050 con 0x7f21f401a070 2023-10-05T19:28:23.835 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:23.830+0000 7f21fa63c700 1 -- 172.21.15.167:0/3301061136 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:57 5.16 5:68000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x556c956a2e90 con 0x556c956a37f0 2023-10-05T19:28:23.836 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:23.830+0000 7f21fae3d700 1 -- 172.21.15.167:0/3301061136 <== osd.3 v2:172.21.15.167:6808/30036 13 ==== osd_op_reply(57 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f21f4039040 con 0x556c956a37f0 2023-10-05T19:28:23.836 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:23.830+0000 7f21fae3d700 1 -- 172.21.15.167:0/3301061136 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:58 5.e 5:70000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x556c956a2e90 con 0x556c956a37f0 2023-10-05T19:28:23.836 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:23.830+0000 7f21fae3d700 1 -- 172.21.15.167:0/3301061136 <== osd.3 v2:172.21.15.167:6808/30036 14 ==== osd_op_reply(58 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f21f4039040 con 0x556c956a37f0 2023-10-05T19:28:23.837 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:23.830+0000 7f21fae3d700 1 -- 172.21.15.167:0/3301061136 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:59 5.1e 5:78000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x556c956a2e90 con 0x7f21e4010070 2023-10-05T19:28:23.837 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:23.830+0000 7f21fb63e700 1 -- 172.21.15.167:0/3301061136 <== osd.0 v2:172.21.15.118:6800/27690 15 ==== osd_op_reply(59 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f21ec02f080 con 0x7f21e4010070 2023-10-05T19:28:23.838 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:23.830+0000 7f21fb63e700 1 -- 172.21.15.167:0/3301061136 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:60 5.1 5:80000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x556c956a2e90 con 0x556c956a37f0 2023-10-05T19:28:23.838 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:23.834+0000 7f21fae3d700 1 -- 172.21.15.167:0/3301061136 <== osd.3 v2:172.21.15.167:6808/30036 15 ==== osd_op_reply(60 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f21f4039040 con 0x556c956a37f0 2023-10-05T19:28:23.838 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:23.834+0000 7f21fae3d700 1 -- 172.21.15.167:0/3301061136 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:61 5.11 5:88000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x556c956a2e90 con 0x556c956a37f0 2023-10-05T19:28:23.839 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:23.834+0000 7f21fae3d700 1 -- 172.21.15.167:0/3301061136 <== osd.3 v2:172.21.15.167:6808/30036 16 ==== osd_op_reply(61 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f21f4039040 con 0x556c956a37f0 2023-10-05T19:28:23.839 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:23.834+0000 7f21fae3d700 1 -- 172.21.15.167:0/3301061136 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:62 5.9 5:90000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x556c956a2e90 con 0x556c95672600 2023-10-05T19:28:23.840 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:23.834+0000 7f21fb63e700 1 -- 172.21.15.167:0/3301061136 <== osd.1 v2:172.21.15.167:6800/30035 22 ==== osd_op_reply(62 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f21ec02f080 con 0x556c95672600 2023-10-05T19:28:23.840 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:23.834+0000 7f21fb63e700 1 -- 172.21.15.167:0/3301061136 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:63 5.19 5:98000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x556c956a2e90 con 0x556c95672600 2023-10-05T19:28:23.841 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:23.834+0000 7f21fb63e700 1 -- 172.21.15.167:0/3301061136 <== osd.1 v2:172.21.15.167:6800/30035 23 ==== osd_op_reply(63 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f21ec02f080 con 0x556c95672600 2023-10-05T19:28:23.841 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:23.834+0000 7f21fb63e700 1 -- 172.21.15.167:0/3301061136 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:64 5.5 5:a0000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x556c956a2e90 con 0x7f21e4010070 2023-10-05T19:28:23.842 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:23.838+0000 7f21fb63e700 1 -- 172.21.15.167:0/3301061136 <== osd.0 v2:172.21.15.118:6800/27690 16 ==== osd_op_reply(64 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f21ec02f080 con 0x7f21e4010070 2023-10-05T19:28:23.842 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:23.838+0000 7f21fb63e700 1 -- 172.21.15.167:0/3301061136 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:65 5.15 5:a8000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x556c956a2e90 con 0x556c956a37f0 2023-10-05T19:28:23.843 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:23.838+0000 7f21fae3d700 1 -- 172.21.15.167:0/3301061136 <== osd.3 v2:172.21.15.167:6808/30036 17 ==== osd_op_reply(65 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f21f4039040 con 0x556c956a37f0 2023-10-05T19:28:23.843 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:23.838+0000 7f21fae3d700 1 -- 172.21.15.167:0/3301061136 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:66 5.d 5:b0000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x556c956a2e90 con 0x7f21f401a070 2023-10-05T19:28:23.846 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:23.838+0000 7f21fa63c700 1 -- 172.21.15.167:0/3301061136 <== osd.2 v2:172.21.15.167:6816/30037 10 ==== osd_op_reply(66 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f21e4017050 con 0x7f21f401a070 2023-10-05T19:28:23.846 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:23.838+0000 7f21fa63c700 1 -- 172.21.15.167:0/3301061136 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:67 5.1d 5:b8000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x556c956a2e90 con 0x556c95672600 2023-10-05T19:28:23.846 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:23.838+0000 7f21fb63e700 1 -- 172.21.15.167:0/3301061136 <== osd.1 v2:172.21.15.167:6800/30035 24 ==== osd_op_reply(67 [pgnls start_epoch 30 out=107b] v23'1 uv1 ondisk = 1) v8 ==== 144+0+107 (crc 0 0 0) 0x7f21ec02f080 con 0x556c95672600 2023-10-05T19:28:23.846 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:23.838+0000 7f21fb63e700 1 -- 172.21.15.167:0/3301061136 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:68 5.3 5:c0000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x556c956a2e90 con 0x7f21e4010070 2023-10-05T19:28:23.846 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:23.838+0000 7f21fb63e700 1 -- 172.21.15.167:0/3301061136 <== osd.0 v2:172.21.15.118:6800/27690 17 ==== osd_op_reply(68 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f21ec02f080 con 0x7f21e4010070 2023-10-05T19:28:23.846 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:23.838+0000 7f21fb63e700 1 -- 172.21.15.167:0/3301061136 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:69 5.13 5:c8000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x556c956a2e90 con 0x556c956a37f0 2023-10-05T19:28:23.847 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:23.842+0000 7f21fae3d700 1 -- 172.21.15.167:0/3301061136 <== osd.3 v2:172.21.15.167:6808/30036 18 ==== osd_op_reply(69 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f21f4039040 con 0x556c956a37f0 2023-10-05T19:28:23.847 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:23.842+0000 7f21fae3d700 1 -- 172.21.15.167:0/3301061136 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:70 5.b 5:d0000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x556c956a2e90 con 0x7f21e4010070 2023-10-05T19:28:23.847 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:23.842+0000 7f21fb63e700 1 -- 172.21.15.167:0/3301061136 <== osd.0 v2:172.21.15.118:6800/27690 18 ==== osd_op_reply(70 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f21ec02f080 con 0x7f21e4010070 2023-10-05T19:28:23.848 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:23.842+0000 7f21fb63e700 1 -- 172.21.15.167:0/3301061136 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:71 5.1b 5:d8000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x556c956a2e90 con 0x7f21f401a070 2023-10-05T19:28:23.848 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:23.842+0000 7f21fa63c700 1 -- 172.21.15.167:0/3301061136 <== osd.2 v2:172.21.15.167:6816/30037 11 ==== osd_op_reply(71 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f21e4017050 con 0x7f21f401a070 2023-10-05T19:28:23.849 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:23.842+0000 7f21fa63c700 1 -- 172.21.15.167:0/3301061136 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:72 5.7 5:e0000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x556c956a2e90 con 0x556c956a37f0 2023-10-05T19:28:23.849 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:23.842+0000 7f21fae3d700 1 -- 172.21.15.167:0/3301061136 <== osd.3 v2:172.21.15.167:6808/30036 19 ==== osd_op_reply(72 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f21f4039040 con 0x556c956a37f0 2023-10-05T19:28:23.849 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:23.842+0000 7f21fae3d700 1 -- 172.21.15.167:0/3301061136 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:73 5.17 5:e8000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x556c956a2e90 con 0x7f21e4010070 2023-10-05T19:28:23.850 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:23.846+0000 7f21fb63e700 1 -- 172.21.15.167:0/3301061136 <== osd.0 v2:172.21.15.118:6800/27690 19 ==== osd_op_reply(73 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f21ec02f080 con 0x7f21e4010070 2023-10-05T19:28:23.850 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:23.846+0000 7f21fb63e700 1 -- 172.21.15.167:0/3301061136 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:74 5.f 5:f0000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x556c956a2e90 con 0x556c95672600 2023-10-05T19:28:23.851 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:23.846+0000 7f21fb63e700 1 -- 172.21.15.167:0/3301061136 <== osd.1 v2:172.21.15.167:6800/30035 25 ==== osd_op_reply(74 [pgnls start_epoch 30 out=115b] v23'2 uv2 ondisk = 1) v8 ==== 144+0+115 (crc 0 0 0) 0x7f21ec02f080 con 0x556c95672600 2023-10-05T19:28:23.851 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:23.846+0000 7f21fb63e700 1 -- 172.21.15.167:0/3301061136 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:75 5.1f 5:f8000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x556c956a2e90 con 0x7f21f401a070 2023-10-05T19:28:23.852 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:23.846+0000 7f21fa63c700 1 -- 172.21.15.167:0/3301061136 <== osd.2 v2:172.21.15.167:6816/30037 12 ==== osd_op_reply(75 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f21e4017050 con 0x7f21f401a070 2023-10-05T19:28:23.852 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:23.846+0000 7f21fd308e40 20 RGWRados::pool_iterate: got zonegroup_info.4f1f719d-9648-417b-b10c-e5d939825419 2023-10-05T19:28:23.852 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:23.846+0000 7f21fd308e40 20 RGWRados::pool_iterate: got zone_info.9a9df965-6612-45d4-a45f-6e3c889bacc2 2023-10-05T19:28:23.852 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:23.846+0000 7f21fd308e40 20 RGWRados::pool_iterate: got zone_names.default 2023-10-05T19:28:23.853 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:23.846+0000 7f21fd308e40 20 RGWRados::pool_iterate: got zonegroups_names.default 2023-10-05T19:28:23.853 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:23.846+0000 7f21fd308e40 20 rados->read ofs=0 len=0 2023-10-05T19:28:23.853 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:23.846+0000 7f21fd308e40 1 -- 172.21.15.167:0/3301061136 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:76 5.f 5:f43b6ece:::zone_names.default:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x556c956a2a90 con 0x556c95672600 2023-10-05T19:28:23.854 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:23.846+0000 7f21fb63e700 1 -- 172.21.15.167:0/3301061136 <== osd.1 v2:172.21.15.167:6800/30035 26 ==== osd_op_reply(76 zone_names.default [read 0~46 out=46b] v0'0 uv1 ondisk = 0) v8 ==== 162+0+46 (crc 0 0 0) 0x7f21ec02f080 con 0x556c95672600 2023-10-05T19:28:23.854 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:23.846+0000 7f21fd308e40 20 rados_obj.operate() r=0 bl.length=46 2023-10-05T19:28:23.854 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:23.846+0000 7f21fd308e40 20 rados->read ofs=0 len=0 2023-10-05T19:28:23.854 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:23.846+0000 7f21fd308e40 1 -- 172.21.15.167:0/3301061136 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:77 5.1d 5:b8374eb4:::zone_info.9a9df965-6612-45d4-a45f-6e3c889bacc2:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x556c956a6e70 con 0x556c95672600 2023-10-05T19:28:23.854 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:23.846+0000 7f21fb63e700 1 -- 172.21.15.167:0/3301061136 <== osd.1 v2:172.21.15.167:6800/30035 27 ==== osd_op_reply(77 zone_info.9a9df965-6612-45d4-a45f-6e3c889bacc2 [read 0~874 out=874b] v0'0 uv1 ondisk = 0) v8 ==== 190+0+874 (crc 0 0 0) 0x7f21ec02f080 con 0x556c95672600 2023-10-05T19:28:23.854 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:23.846+0000 7f21fd308e40 20 rados_obj.operate() r=0 bl.length=874 2023-10-05T19:28:23.854 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:23.846+0000 7f21fd308e40 20 rados->read ofs=0 len=0 2023-10-05T19:28:23.854 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:23.846+0000 7f21fd308e40 1 -- 172.21.15.167:0/3301061136 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:78 5.f 5:f4c53578:::zonegroups_names.default:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x556c956a7360 con 0x556c95672600 2023-10-05T19:28:23.855 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:23.850+0000 7f21fb63e700 1 -- 172.21.15.167:0/3301061136 <== osd.1 v2:172.21.15.167:6800/30035 28 ==== osd_op_reply(78 zonegroups_names.default [read 0~46 out=46b] v0'0 uv2 ondisk = 0) v8 ==== 168+0+46 (crc 0 0 0) 0x7f21ec02f080 con 0x556c95672600 2023-10-05T19:28:23.855 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:23.850+0000 7f21fd308e40 20 rados_obj.operate() r=0 bl.length=46 2023-10-05T19:28:23.855 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:23.850+0000 7f21fd308e40 20 rados->read ofs=0 len=0 2023-10-05T19:28:23.855 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:23.850+0000 7f21fd308e40 1 -- 172.21.15.167:0/3301061136 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:79 5.c 5:30d299a5:::zonegroup_info.4f1f719d-9648-417b-b10c-e5d939825419:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x556c956a7360 con 0x7f21e4010070 2023-10-05T19:28:23.856 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:23.850+0000 7f21fb63e700 1 -- 172.21.15.167:0/3301061136 <== osd.0 v2:172.21.15.118:6800/27690 20 ==== osd_op_reply(79 zonegroup_info.4f1f719d-9648-417b-b10c-e5d939825419 [read 0~362 out=362b] v0'0 uv1 ondisk = 0) v8 ==== 195+0+362 (crc 0 0 0) 0x7f21ec02f080 con 0x7f21e4010070 2023-10-05T19:28:23.856 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:23.850+0000 7f21fd308e40 20 rados_obj.operate() r=0 bl.length=362 2023-10-05T19:28:23.856 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:23.850+0000 7f21fd308e40 20 rados->read ofs=0 len=0 2023-10-05T19:28:23.856 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:23.850+0000 7f21fd308e40 1 -- 172.21.15.167:0/3301061136 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:80 5.3 5:c0ad8cbd:::region_map:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x556c956a7360 con 0x7f21e4010070 2023-10-05T19:28:23.857 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:23.850+0000 7f21fb63e700 1 -- 172.21.15.167:0/3301061136 <== osd.0 v2:172.21.15.118:6800/27690 21 ==== osd_op_reply(80 region_map [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 ==== 154+0+0 (crc 0 0 0) 0x7f21ec02f080 con 0x7f21e4010070 2023-10-05T19:28:23.857 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:23.850+0000 7f21fd308e40 20 rados_obj.operate() r=-2 bl.length=0 2023-10-05T19:28:23.857 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:23.850+0000 7f21fd308e40 20 zone default found 2023-10-05T19:28:23.857 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:23.850+0000 7f21fd308e40 4 Realm: () 2023-10-05T19:28:23.857 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:23.850+0000 7f21fd308e40 4 ZoneGroup: default (4f1f719d-9648-417b-b10c-e5d939825419) 2023-10-05T19:28:23.857 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:23.850+0000 7f21fd308e40 4 Zone: default (9a9df965-6612-45d4-a45f-6e3c889bacc2) 2023-10-05T19:28:23.857 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:23.850+0000 7f21fd308e40 10 cannot find current period zonegroup using local zonegroup configuration 2023-10-05T19:28:23.857 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:23.850+0000 7f21fd308e40 20 zonegroup default 2023-10-05T19:28:23.858 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:23.850+0000 7f21fd308e40 20 rados->read ofs=0 len=0 2023-10-05T19:28:23.858 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:23.850+0000 7f21fd308e40 1 -- 172.21.15.167:0/3301061136 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:81 5.3 5:c52100b6:::period_config.default:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x556c956a7360 con 0x7f21e4010070 2023-10-05T19:28:23.859 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:23.850+0000 7f21fb63e700 1 -- 172.21.15.167:0/3301061136 <== osd.0 v2:172.21.15.118:6800/27690 22 ==== osd_op_reply(81 period_config.default [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 ==== 165+0+0 (crc 0 0 0) 0x7f21ec050040 con 0x7f21e4010070 2023-10-05T19:28:23.859 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:23.850+0000 7f21fd308e40 20 rados_obj.operate() r=-2 bl.length=0 2023-10-05T19:28:23.859 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:23.850+0000 7f21fd308e40 20 rados->read ofs=0 len=0 2023-10-05T19:28:23.859 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:23.850+0000 7f21fd308e40 1 -- 172.21.15.167:0/3301061136 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:82 6.2 6:4347d321:::bucket.sync-source-hints.:head [call version.read in=11b,read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x556c956a8ce0 con 0x556c956a37f0 2023-10-05T19:28:23.859 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:23.854+0000 7f21fae3d700 1 -- 172.21.15.167:0/3301061136 <== osd.3 v2:172.21.15.167:6808/30036 20 ==== osd_op_reply(82 bucket.sync-source-hints. [call,read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 ==== 211+0+0 (crc 0 0 0) 0x7f21f4039040 con 0x556c956a37f0 2023-10-05T19:28:23.859 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:23.854+0000 7f21fd308e40 20 rados_obj.operate() r=-2 bl.length=0 2023-10-05T19:28:23.859 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:23.854+0000 7f21fd308e40 20 rados->read ofs=0 len=0 2023-10-05T19:28:23.859 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:23.854+0000 7f21fd308e40 1 -- 172.21.15.167:0/3301061136 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:83 6.b 6:d467b91b:::bucket.sync-target-hints.:head [call version.read in=11b,read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x556c956a9090 con 0x556c956a37f0 2023-10-05T19:28:23.860 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:23.854+0000 7f21fae3d700 1 -- 172.21.15.167:0/3301061136 <== osd.3 v2:172.21.15.167:6808/30036 21 ==== osd_op_reply(83 bucket.sync-target-hints. [call,read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 ==== 211+0+0 (crc 0 0 0) 0x7f21f4039040 con 0x556c956a37f0 2023-10-05T19:28:23.860 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:23.854+0000 7f21fd308e40 20 rados_obj.operate() r=-2 bl.length=0 2023-10-05T19:28:23.860 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:23.854+0000 7f21fd308e40 20 started sync module instance, tier type = 2023-10-05T19:28:23.860 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:23.854+0000 7f21fd308e40 20 started zone id=9a9df965-6612-45d4-a45f-6e3c889bacc2 (name=default) with tier type = 2023-10-05T19:28:23.860 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:23.854+0000 7f21fd308e40 1 -- 172.21.15.167:0/3301061136 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:84 7.1f 7:f95f44c2:::notify.0:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x556c956a83a0 con 0x7f21f401a070 2023-10-05T19:28:23.862 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:23.858+0000 7f21fa63c700 1 -- 172.21.15.167:0/3301061136 <== osd.2 v2:172.21.15.167:6816/30037 13 ==== osd_op_reply(84 notify.0 [create] v30'6 uv6 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f21e4017050 con 0x7f21f401a070 2023-10-05T19:28:23.863 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:23.858+0000 7f21fd308e40 1 -- 172.21.15.167:0/3301061136 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:85 7.1f 7:f95f44c2:::notify.0:head [watch watch cookie 93924851615568] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x556c956a7fd0 con 0x7f21f401a070 2023-10-05T19:28:23.863 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:23.858+0000 7f21fd308e40 1 -- 172.21.15.167:0/3301061136 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:86 7.0 7:05bf5b68:::notify.1:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x556c956a9e20 con 0x556c95672600 2023-10-05T19:28:23.865 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:23.858+0000 7f21fa63c700 1 -- 172.21.15.167:0/3301061136 <== osd.2 v2:172.21.15.167:6816/30037 14 ==== osd_op_reply(85 notify.0 [watch watch cookie 93924851615568] v30'7 uv6 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f21e4017050 con 0x7f21f401a070 2023-10-05T19:28:23.866 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:23.862+0000 7f21fb63e700 1 -- 172.21.15.167:0/3301061136 <== osd.1 v2:172.21.15.167:6800/30035 29 ==== osd_op_reply(86 notify.1 [create] v30'6 uv6 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f21ec050040 con 0x556c95672600 2023-10-05T19:28:23.866 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:23.862+0000 7f21fd308e40 1 -- 172.21.15.167:0/3301061136 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:87 7.0 7:05bf5b68:::notify.1:head [watch watch cookie 93924851619168] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x556c956aa220 con 0x556c95672600 2023-10-05T19:28:23.866 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:23.862+0000 7f21fd308e40 1 -- 172.21.15.167:0/3301061136 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:88 7.15 7:a93a5511:::notify.2:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x556c956aaac0 con 0x556c956a37f0 2023-10-05T19:28:23.869 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:23.862+0000 7f21fb63e700 1 -- 172.21.15.167:0/3301061136 <== osd.1 v2:172.21.15.167:6800/30035 30 ==== osd_op_reply(87 notify.1 [watch watch cookie 93924851619168] v30'7 uv6 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f21ec050040 con 0x556c95672600 2023-10-05T19:28:23.870 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:23.866+0000 7f21fae3d700 1 -- 172.21.15.167:0/3301061136 <== osd.3 v2:172.21.15.167:6808/30036 22 ==== osd_op_reply(88 notify.2 [create] v30'6 uv6 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f21f4039040 con 0x556c956a37f0 2023-10-05T19:28:23.870 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:23.866+0000 7f21fd308e40 1 -- 172.21.15.167:0/3301061136 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:89 7.15 7:a93a5511:::notify.2:head [watch watch cookie 93924851620272] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x556c956aa710 con 0x556c956a37f0 2023-10-05T19:28:23.870 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:23.866+0000 7f21fd308e40 1 -- 172.21.15.167:0/3301061136 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:90 7.e 7:7759931f:::notify.3:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x556c956ab570 con 0x7f21f401a070 2023-10-05T19:28:23.873 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:23.866+0000 7f21fa63c700 1 -- 172.21.15.167:0/3301061136 <== osd.2 v2:172.21.15.167:6816/30037 15 ==== osd_op_reply(90 notify.3 [create] v30'6 uv6 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f21e4017050 con 0x7f21f401a070 2023-10-05T19:28:23.873 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:23.866+0000 7f21fd308e40 1 -- 172.21.15.167:0/3301061136 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:91 7.e 7:7759931f:::notify.3:head [watch watch cookie 93924851628400] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x556c956ac2b0 con 0x7f21f401a070 2023-10-05T19:28:23.873 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:23.866+0000 7f21fd308e40 1 -- 172.21.15.167:0/3301061136 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:92 7.d 7:b4812045:::notify.4:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x556c956acb90 con 0x556c95672600 2023-10-05T19:28:23.873 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:23.866+0000 7f21fae3d700 1 -- 172.21.15.167:0/3301061136 <== osd.3 v2:172.21.15.167:6808/30036 23 ==== osd_op_reply(89 notify.2 [watch watch cookie 93924851620272] v30'7 uv6 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f21f4039040 con 0x556c956a37f0 2023-10-05T19:28:23.876 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:23.870+0000 7f21fb63e700 1 -- 172.21.15.167:0/3301061136 <== osd.1 v2:172.21.15.167:6800/30035 31 ==== osd_op_reply(92 notify.4 [create] v30'6 uv6 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f21ec050040 con 0x556c95672600 2023-10-05T19:28:23.876 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:23.870+0000 7f21fd308e40 1 -- 172.21.15.167:0/3301061136 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:93 7.d 7:b4812045:::notify.4:head [watch watch cookie 93924851631712] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x556c956acf90 con 0x556c95672600 2023-10-05T19:28:23.876 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:23.870+0000 7f21fd308e40 1 -- 172.21.15.167:0/3301061136 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:94 7.3 7:c609908c:::notify.5:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x556c956ad9a0 con 0x7f21e4010070 2023-10-05T19:28:23.876 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:23.870+0000 7f21fa63c700 1 -- 172.21.15.167:0/3301061136 <== osd.2 v2:172.21.15.167:6816/30037 16 ==== osd_op_reply(91 notify.3 [watch watch cookie 93924851628400] v30'7 uv6 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f21e4017050 con 0x7f21f401a070 2023-10-05T19:28:23.878 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:23.874+0000 7f21fb63e700 1 -- 172.21.15.167:0/3301061136 <== osd.1 v2:172.21.15.167:6800/30035 32 ==== osd_op_reply(93 notify.4 [watch watch cookie 93924851631712] v30'7 uv6 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f21ec050040 con 0x556c95672600 2023-10-05T19:28:23.879 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:23.874+0000 7f21fb63e700 1 -- 172.21.15.167:0/3301061136 <== osd.0 v2:172.21.15.118:6800/27690 23 ==== osd_op_reply(94 notify.5 [create] v30'6 uv6 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f21ec050040 con 0x7f21e4010070 2023-10-05T19:28:23.879 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:23.874+0000 7f21fd308e40 1 -- 172.21.15.167:0/3301061136 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:95 7.3 7:c609908c:::notify.5:head [watch watch cookie 93924851629504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x556c956ad4b0 con 0x7f21e4010070 2023-10-05T19:28:23.879 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:23.874+0000 7f21fd308e40 1 -- 172.21.15.167:0/3301061136 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:96 7.14 7:2b04a3e9:::notify.6:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x556c956add50 con 0x7f21e4010070 2023-10-05T19:28:23.882 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:23.878+0000 7f21fb63e700 1 -- 172.21.15.167:0/3301061136 <== osd.0 v2:172.21.15.118:6800/27690 24 ==== osd_op_reply(96 notify.6 [create] v30'6 uv6 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f21ec050040 con 0x7f21e4010070 2023-10-05T19:28:23.882 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:23.878+0000 7f21fd308e40 1 -- 172.21.15.167:0/3301061136 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:97 7.14 7:2b04a3e9:::notify.6:head [watch watch cookie 93924851635936] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x556c956ae0a0 con 0x7f21e4010070 2023-10-05T19:28:23.882 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:23.878+0000 7f21fd308e40 1 -- 172.21.15.167:0/3301061136 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:98 7.9 7:93e5b521:::notify.7:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x556c956aec30 con 0x7f21e4010070 2023-10-05T19:28:23.882 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:23.878+0000 7f21fb63e700 1 -- 172.21.15.167:0/3301061136 <== osd.0 v2:172.21.15.118:6800/27690 25 ==== osd_op_reply(95 notify.5 [watch watch cookie 93924851629504] v30'7 uv6 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f21ec050040 con 0x7f21e4010070 2023-10-05T19:28:23.885 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:23.878+0000 7f21fb63e700 1 -- 172.21.15.167:0/3301061136 <== osd.0 v2:172.21.15.118:6800/27690 26 ==== osd_op_reply(98 notify.7 [create] v30'6 uv6 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f21ec050040 con 0x7f21e4010070 2023-10-05T19:28:23.885 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:23.878+0000 7f21fd308e40 1 -- 172.21.15.167:0/3301061136 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:99 7.9 7:93e5b521:::notify.7:head [watch watch cookie 93924851622320] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x556c956aebf0 con 0x7f21e4010070 2023-10-05T19:28:23.885 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:23.878+0000 7f21fd308e40 20 add_watcher() i=0 2023-10-05T19:28:23.885 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:23.878+0000 7f21fd308e40 20 add_watcher() i=1 2023-10-05T19:28:23.886 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:23.878+0000 7f21fd308e40 20 add_watcher() i=2 2023-10-05T19:28:23.886 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:23.878+0000 7f21fd308e40 20 add_watcher() i=3 2023-10-05T19:28:23.886 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:23.878+0000 7f21fd308e40 20 add_watcher() i=4 2023-10-05T19:28:23.886 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:23.878+0000 7f21fd308e40 20 add_watcher() i=5 2023-10-05T19:28:23.886 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:23.878+0000 7f21fb63e700 1 -- 172.21.15.167:0/3301061136 <== osd.0 v2:172.21.15.118:6800/27690 27 ==== osd_op_reply(97 notify.6 [watch watch cookie 93924851635936] v30'7 uv6 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f21ec050040 con 0x7f21e4010070 2023-10-05T19:28:23.886 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:23.878+0000 7f21fd308e40 20 add_watcher() i=6 2023-10-05T19:28:23.887 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:23.882+0000 7f21fb63e700 1 -- 172.21.15.167:0/3301061136 <== osd.0 v2:172.21.15.118:6800/27690 28 ==== osd_op_reply(99 notify.7 [watch watch cookie 93924851622320] v30'7 uv6 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f21ec050040 con 0x7f21e4010070 2023-10-05T19:28:23.887 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:23.882+0000 7f21fd308e40 20 add_watcher() i=7 2023-10-05T19:28:23.887 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:23.882+0000 7f21fd308e40 2 all 8 watchers are set, enabling cache 2023-10-05T19:28:23.887 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:23.882+0000 7f21fd308e40 1 -- 172.21.15.167:0/3301061136 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:100 6.4 6:22d26bf9:::data_loggenerations_metadata:head [call version.check_conds in=74b,call version.read in=11b,read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x556c956adb30 con 0x556c95672600 2023-10-05T19:28:23.888 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:23.882+0000 7f21fb63e700 1 -- 172.21.15.167:0/3301061136 <== osd.1 v2:172.21.15.167:6800/30035 33 ==== osd_op_reply(100 data_loggenerations_metadata [call,call out=48b,read 0~28 out=28b] v0'0 uv1 ondisk = 0) v8 ==== 256+0+76 (crc 0 0 0) 0x7f21ec050040 con 0x556c95672600 2023-10-05T19:28:23.888 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:23.882+0000 7f21fd308e40 1 -- 172.21.15.167:0/3301061136 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:101 6.4 6:22d26bf9:::data_loggenerations_metadata:head [watch watch cookie 93924851640064] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x556c956aefe0 con 0x556c95672600 2023-10-05T19:28:23.890 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:23.886+0000 7f21fb63e700 1 -- 172.21.15.167:0/3301061136 <== osd.1 v2:172.21.15.167:6800/30035 34 ==== osd_op_reply(101 data_loggenerations_metadata [watch watch cookie 93924851640064] v30'26 uv1 ondisk = 0) v8 ==== 172+0+0 (crc 0 0 0) 0x7f21ec050040 con 0x556c95672600 2023-10-05T19:28:23.890 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:23.886+0000 7f2147fff700 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2023-10-05T19:28:23.890 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:23.886+0000 7f21fd308e40 20 check_secure_mon_conn(): auth registy supported: methods=[2] modes=[2,1] 2023-10-05T19:28:23.890 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:23.886+0000 7f21fd308e40 20 check_secure_mon_conn(): mode 1 is insecure 2023-10-05T19:28:23.891 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:23.886+0000 7f21fd308e40 1 Processor -- start 2023-10-05T19:28:23.891 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:23.886+0000 7f21fd308e40 1 -- start start 2023-10-05T19:28:23.891 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:23.886+0000 7f21fd308e40 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x556c956bc110 0x556c956bc4e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-05T19:28:23.891 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:23.886+0000 7f21fd308e40 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x556c956bca20 con 0x556c956bc110 2023-10-05T19:28:23.892 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:23.886+0000 7f21fa63c700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x556c956bc110 0x556c956bc4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-05T19:28:23.892 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:23.886+0000 7f21fa63c700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x556c956bc110 0x556c956bc4e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.118:3300/0 says I am v2:172.21.15.167:50642/0 (socket says 172.21.15.167:50642) 2023-10-05T19:28:23.892 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:23.886+0000 7f21fa63c700 1 -- 172.21.15.167:0/2642180365 learned_addr learned my addr 172.21.15.167:0/2642180365 (peer_addr_for_me v2:172.21.15.167:0/0) 2023-10-05T19:28:23.892 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:23.886+0000 7f21fa63c700 1 -- 172.21.15.167:0/2642180365 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x556c956bcb60 con 0x556c956bc110 2023-10-05T19:28:23.893 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:23.886+0000 7f21fa63c700 1 --2- 172.21.15.167:0/2642180365 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x556c956bc110 0x556c956bc4e0 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7f21e4003000 tx=0x7f21e400f000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f62586205a7f3584 server_cookie=0 in_seq=0 out_seq=0 2023-10-05T19:28:23.893 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:23.886+0000 7f21477fe700 1 -- 172.21.15.167:0/2642180365 <== mon.0 v2:172.21.15.118:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f21e401c9f0 con 0x556c956bc110 2023-10-05T19:28:23.893 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:23.886+0000 7f21477fe700 1 -- 172.21.15.167:0/2642180365 <== mon.0 v2:172.21.15.118:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21e401cb50 con 0x556c956bc110 2023-10-05T19:28:23.893 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:23.886+0000 7f21477fe700 1 -- 172.21.15.167:0/2642180365 <== mon.0 v2:172.21.15.118:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f21e401ce60 con 0x556c956bc110 2023-10-05T19:28:23.894 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:23.890+0000 7f21fd308e40 1 -- 172.21.15.167:0/2642180365 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x556c956bc110 msgr2=0x556c956bc4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-05T19:28:23.894 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:23.890+0000 7f21fd308e40 1 --2- 172.21.15.167:0/2642180365 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x556c956bc110 0x556c956bc4e0 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7f21e4003000 tx=0x7f21e400f000 comp rx=0 tx=0).stop 2023-10-05T19:28:23.894 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:23.890+0000 7f21fd308e40 1 -- 172.21.15.167:0/2642180365 shutdown_connections 2023-10-05T19:28:23.894 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:23.890+0000 7f21fd308e40 1 --2- 172.21.15.167:0/2642180365 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x556c956bc110 0x556c956bc4e0 unknown :-1 s=CLOSED pgs=144 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-05T19:28:23.894 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:23.890+0000 7f21fd308e40 1 -- 172.21.15.167:0/2642180365 >> 172.21.15.167:0/2642180365 conn(0x556c956b74d0 msgr2=0x556c956b5620 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-05T19:28:23.895 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:23.890+0000 7f21fd308e40 1 -- 172.21.15.167:0/2642180365 shutdown_connections 2023-10-05T19:28:23.895 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:23.890+0000 7f21fd308e40 1 -- 172.21.15.167:0/2642180365 wait complete. 2023-10-05T19:28:23.896 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:23.890+0000 7f21fd308e40 1 Processor -- start 2023-10-05T19:28:23.896 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:23.890+0000 7f21fd308e40 1 -- start start 2023-10-05T19:28:23.896 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:23.890+0000 7f21fd308e40 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x556c956bc110 0x556c956c1c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-05T19:28:23.896 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:23.890+0000 7f21fd308e40 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x556c956c2c40 con 0x556c956bc110 2023-10-05T19:28:23.896 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:23.890+0000 7f21fa63c700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x556c956bc110 0x556c956c1c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-05T19:28:23.896 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:23.890+0000 7f21fa63c700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x556c956bc110 0x556c956c1c40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.118:3300/0 says I am v2:172.21.15.167:50650/0 (socket says 172.21.15.167:50650) 2023-10-05T19:28:23.896 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:23.890+0000 7f21fa63c700 1 -- 172.21.15.167:0/3182581180 learned_addr learned my addr 172.21.15.167:0/3182581180 (peer_addr_for_me v2:172.21.15.167:0/0) 2023-10-05T19:28:23.896 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:23.890+0000 7f21fa63c700 1 -- 172.21.15.167:0/3182581180 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x556c9566ec30 con 0x556c956bc110 2023-10-05T19:28:23.897 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:23.890+0000 7f21fa63c700 1 --2- 172.21.15.167:0/3182581180 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x556c956bc110 0x556c956c1c40 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7f21e400f040 tx=0x7f21e402d030 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-05T19:28:23.897 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:23.890+0000 7f2145ffb700 1 -- 172.21.15.167:0/3182581180 <== mon.0 v2:172.21.15.118:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f21e400ee70 con 0x556c956bc110 2023-10-05T19:28:23.898 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:23.890+0000 7f2145ffb700 1 -- 172.21.15.167:0/3182581180 <== mon.0 v2:172.21.15.118:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21e402bbb0 con 0x556c956bc110 2023-10-05T19:28:23.898 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:23.890+0000 7f21fd308e40 1 -- 172.21.15.167:0/3182581180 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x556c956c2240 con 0x556c956bc110 2023-10-05T19:28:23.898 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:23.890+0000 7f2145ffb700 1 -- 172.21.15.167:0/3182581180 <== mon.0 v2:172.21.15.118:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f21e4042450 con 0x556c956bc110 2023-10-05T19:28:23.898 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:23.890+0000 7f21fd308e40 1 -- 172.21.15.167:0/3182581180 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x556c956c2620 con 0x556c956bc110 2023-10-05T19:28:23.899 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:23.894+0000 7f2145ffb700 1 -- 172.21.15.167:0/3182581180 <== mon.0 v2:172.21.15.118:3300/0 4 ==== mgrmap(e 5) v1 ==== 40329+0+0 (secure 0 0 0) 0x7f21e4036400 con 0x556c956bc110 2023-10-05T19:28:23.899 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:23.894+0000 7f2145ffb700 1 --2- 172.21.15.167:0/3182581180 >> [v2:172.21.15.167:6824/29889,v1:172.21.15.167:6825/29889] conn(0x7f2138030bb0 0x7f2138033060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-05T19:28:23.899 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:23.894+0000 7f2145ffb700 1 -- 172.21.15.167:0/3182581180 <== mon.0 v2:172.21.15.118:3300/0 5 ==== osd_map(30..30 src has 1..30) v4 ==== 6057+0+0 (secure 0 0 0) 0x7f21e4043dd0 con 0x556c956bc110 2023-10-05T19:28:23.899 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:23.894+0000 7f21fb63e700 1 --2- 172.21.15.167:0/3182581180 >> [v2:172.21.15.167:6824/29889,v1:172.21.15.167:6825/29889] conn(0x7f2138030bb0 0x7f2138033060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-05T19:28:23.899 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:23.894+0000 7f21fd308e40 1 --2- 172.21.15.167:0/3182581180 >> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] conn(0x556c956cc860 0x556c956cec70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-05T19:28:23.900 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:23.894+0000 7f21fd308e40 1 -- 172.21.15.167:0/3182581180 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:1 6.19 6:9a6dd8c0:gc::gc.0:head [create,call version.check_conds in=74b,call rgw_gc.rgw_gc_queue_init in=45b,call version.set in=34b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x556c956cf310 con 0x556c956cc860 2023-10-05T19:28:23.900 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:23.894+0000 7f21fae3d700 1 --2- 172.21.15.167:0/3182581180 >> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] conn(0x556c956cc860 0x556c956cec70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-05T19:28:23.900 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:23.894+0000 7f21fb63e700 1 --2- 172.21.15.167:0/3182581180 >> [v2:172.21.15.167:6824/29889,v1:172.21.15.167:6825/29889] conn(0x7f2138030bb0 0x7f2138033060 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x556c956ae020 tx=0x7f21ec000f90 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-05T19:28:23.900 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:23.894+0000 7f21fae3d700 1 --2- 172.21.15.167:0/3182581180 >> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] conn(0x556c956cc860 0x556c956cec70 crc :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-05T19:28:23.903 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:23.898+0000 7f21fae3d700 1 -- 172.21.15.167:0/3182581180 <== osd.1 v2:172.21.15.167:6800/30035 1 ==== osd_op_reply(1 gc.0 [create,call,call,call] v30'16 uv3 ondisk = -125 ((125) Operation canceled)) v8 ==== 274+0+0 (crc 0 0 0) 0x7f21f4039040 con 0x556c956cc860 2023-10-05T19:28:23.903 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:23.898+0000 7f21fd308e40 1 --2- 172.21.15.167:0/3182581180 >> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] conn(0x556c956c8470 0x556c956d2980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-05T19:28:23.903 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:23.898+0000 7f21fd308e40 1 -- 172.21.15.167:0/3182581180 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:2 6.18 6:1e85421f:gc::gc.1:head [create,call version.check_conds in=74b,call rgw_gc.rgw_gc_queue_init in=45b,call version.set in=34b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x556c956d2ec0 con 0x556c956c8470 2023-10-05T19:28:23.904 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:23.898+0000 7f21fa63c700 1 --2- 172.21.15.167:0/3182581180 >> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] conn(0x556c956c8470 0x556c956d2980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-05T19:28:23.904 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:23.898+0000 7f21fa63c700 1 --2- 172.21.15.167:0/3182581180 >> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] conn(0x556c956c8470 0x556c956d2980 crc :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-05T19:28:23.907 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:23.902+0000 7f21fa63c700 1 -- 172.21.15.167:0/3182581180 <== osd.0 v2:172.21.15.118:6800/27690 1 ==== osd_op_reply(2 gc.1 [create,call,call,call] v30'10 uv5 ondisk = -125 ((125) Operation canceled)) v8 ==== 274+0+0 (crc 0 0 0) 0x7f21e403d040 con 0x556c956c8470 2023-10-05T19:28:23.907 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:23.902+0000 7f21fd308e40 1 --2- 172.21.15.167:0/3182581180 >> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] conn(0x556c956d3f10 0x556c956d62f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-05T19:28:23.907 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:23.902+0000 7f21fd308e40 1 -- 172.21.15.167:0/3182581180 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:3 6.b 6:d242335b:gc::gc.2:head [create,call version.check_conds in=74b,call rgw_gc.rgw_gc_queue_init in=45b,call version.set in=34b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x556c956d69f0 con 0x556c956d3f10 2023-10-05T19:28:23.907 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:23.902+0000 7f21fb63e700 1 --2- 172.21.15.167:0/3182581180 >> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] conn(0x556c956d3f10 0x556c956d62f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-05T19:28:23.908 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:23.902+0000 7f21fb63e700 1 --2- 172.21.15.167:0/3182581180 >> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] conn(0x556c956d3f10 0x556c956d62f0 crc :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-05T19:28:23.910 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:23.906+0000 7f21fb63e700 1 -- 172.21.15.167:0/3182581180 <== osd.3 v2:172.21.15.167:6808/30036 1 ==== osd_op_reply(3 gc.2 [create,call,call,call] v30'10 uv3 ondisk = -125 ((125) Operation canceled)) v8 ==== 274+0+0 (crc 0 0 0) 0x7f21ec050040 con 0x556c956d3f10 2023-10-05T19:28:23.910 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:23.906+0000 7f21fd308e40 1 -- 172.21.15.167:0/3182581180 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:4 6.4 6:2166d8e1:gc::gc.3:head [create,call version.check_conds in=74b,call rgw_gc.rgw_gc_queue_init in=45b,call version.set in=34b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x556c956d7040 con 0x556c956cc860 2023-10-05T19:28:23.913 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:23.906+0000 7f21fae3d700 1 -- 172.21.15.167:0/3182581180 <== osd.1 v2:172.21.15.167:6800/30035 2 ==== osd_op_reply(4 gc.3 [create,call,call,call] v30'27 uv7 ondisk = -125 ((125) Operation canceled)) v8 ==== 274+0+0 (crc 0 0 0) 0x7f21f4039040 con 0x556c956cc860 2023-10-05T19:28:23.913 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:23.906+0000 7f21fd308e40 1 --2- 172.21.15.167:0/3182581180 >> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] conn(0x556c956d7ce0 0x556c956da0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-05T19:28:23.913 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:23.906+0000 7f21fd308e40 1 -- 172.21.15.167:0/3182581180 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:5 6.3 6:c60da066:gc::gc.4:head [create,call version.check_conds in=74b,call rgw_gc.rgw_gc_queue_init in=45b,call version.set in=34b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x556c956da720 con 0x556c956d7ce0 2023-10-05T19:28:23.913 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:23.906+0000 7f21fae3d700 1 --2- 172.21.15.167:0/3182581180 >> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] conn(0x556c956d7ce0 0x556c956da0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-05T19:28:23.914 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:23.906+0000 7f21fae3d700 1 --2- 172.21.15.167:0/3182581180 >> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] conn(0x556c956d7ce0 0x556c956da0c0 crc :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-05T19:28:23.916 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:23.910+0000 7f21fae3d700 1 -- 172.21.15.167:0/3182581180 <== osd.2 v2:172.21.15.167:6816/30037 1 ==== osd_op_reply(5 gc.4 [create,call,call,call] v30'20 uv7 ondisk = -125 ((125) Operation canceled)) v8 ==== 274+0+0 (crc 0 0 0) 0x7f21f4039040 con 0x556c956d7ce0 2023-10-05T19:28:23.916 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:23.910+0000 7f21fd308e40 1 -- 172.21.15.167:0/3182581180 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:6 6.9 6:954be8d2:gc::gc.5:head [create,call version.check_conds in=74b,call rgw_gc.rgw_gc_queue_init in=45b,call version.set in=34b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x556c956dad70 con 0x556c956d3f10 2023-10-05T19:28:23.918 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:23.914+0000 7f21fb63e700 1 -- 172.21.15.167:0/3182581180 <== osd.3 v2:172.21.15.167:6808/30036 2 ==== osd_op_reply(6 gc.5 [create,call,call,call] v30'13 uv4 ondisk = -125 ((125) Operation canceled)) v8 ==== 274+0+0 (crc 0 0 0) 0x7f21ec050040 con 0x556c956d3f10 2023-10-05T19:28:23.919 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:23.914+0000 7f21fd308e40 1 -- 172.21.15.167:0/3182581180 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:7 6.e 6:748722ff:gc::gc.6:head [create,call version.check_conds in=74b,call rgw_gc.rgw_gc_queue_init in=45b,call version.set in=34b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x556c956db390 con 0x556c956cc860 2023-10-05T19:28:23.921 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:23.914+0000 7f21fae3d700 1 -- 172.21.15.167:0/3182581180 <== osd.1 v2:172.21.15.167:6800/30035 3 ==== osd_op_reply(7 gc.6 [create,call,call,call] v30'12 uv5 ondisk = -125 ((125) Operation canceled)) v8 ==== 274+0+0 (crc 0 0 0) 0x7f21f4039040 con 0x556c956cc860 2023-10-05T19:28:23.921 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:23.914+0000 7f21fd308e40 1 -- 172.21.15.167:0/3182581180 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:8 6.1d 6:beaa4be4:gc::gc.7:head [create,call version.check_conds in=74b,call rgw_gc.rgw_gc_queue_init in=45b,call version.set in=34b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x556c956db390 con 0x556c956c8470 2023-10-05T19:28:23.924 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:23.918+0000 7f21fa63c700 1 -- 172.21.15.167:0/3182581180 <== osd.0 v2:172.21.15.118:6800/27690 2 ==== osd_op_reply(8 gc.7 [create,call,call,call] v30'13 uv4 ondisk = -125 ((125) Operation canceled)) v8 ==== 274+0+0 (crc 0 0 0) 0x7f21e403d040 con 0x556c956c8470 2023-10-05T19:28:23.924 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:23.918+0000 7f21fd308e40 1 -- 172.21.15.167:0/3182581180 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:9 6.12 6:4f78e9a2:gc::gc.8:head [create,call version.check_conds in=74b,call rgw_gc.rgw_gc_queue_init in=45b,call version.set in=34b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x556c956db9b0 con 0x556c956d7ce0 2023-10-05T19:28:23.927 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:23.922+0000 7f21fae3d700 1 -- 172.21.15.167:0/3182581180 <== osd.2 v2:172.21.15.167:6816/30037 2 ==== osd_op_reply(9 gc.8 [create,call,call,call] v30'10 uv3 ondisk = -125 ((125) Operation canceled)) v8 ==== 274+0+0 (crc 0 0 0) 0x7f21f4039040 con 0x556c956d7ce0 2023-10-05T19:28:23.927 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:23.922+0000 7f21fd308e40 1 -- 172.21.15.167:0/3182581180 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:10 6.1c 6:3e426232:gc::gc.9:head [create,call version.check_conds in=74b,call rgw_gc.rgw_gc_queue_init in=45b,call version.set in=34b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x556c956db9b0 con 0x556c956d3f10 2023-10-05T19:28:23.929 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:23.922+0000 7f21fb63e700 1 -- 172.21.15.167:0/3182581180 <== osd.3 v2:172.21.15.167:6808/30036 3 ==== osd_op_reply(10 gc.9 [create,call,call,call] v30'16 uv3 ondisk = -125 ((125) Operation canceled)) v8 ==== 274+0+0 (crc 0 0 0) 0x7f21ec050040 con 0x556c956d3f10 2023-10-05T19:28:23.930 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:23.922+0000 7f21fd308e40 1 -- 172.21.15.167:0/3182581180 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:11 6.3 6:c31f41d7:gc::gc.10:head [create,call version.check_conds in=74b,call rgw_gc.rgw_gc_queue_init in=45b,call version.set in=34b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x556c956dbea0 con 0x556c956d7ce0 2023-10-05T19:28:23.932 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:23.926+0000 7f21fae3d700 1 -- 172.21.15.167:0/3182581180 <== osd.2 v2:172.21.15.167:6816/30037 3 ==== osd_op_reply(11 gc.10 [create,call,call,call] v30'21 uv11 ondisk = -125 ((125) Operation canceled)) v8 ==== 275+0+0 (crc 0 0 0) 0x7f21f4039040 con 0x556c956d7ce0 2023-10-05T19:28:23.932 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:23.926+0000 7f21fd308e40 1 -- 172.21.15.167:0/3182581180 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:12 6.13 6:c8333aab:gc::gc.11:head [create,call version.check_conds in=74b,call rgw_gc.rgw_gc_queue_init in=45b,call version.set in=34b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x556c956dbea0 con 0x556c956c8470 2023-10-05T19:28:23.935 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:23.930+0000 7f21fa63c700 1 -- 172.21.15.167:0/3182581180 <== osd.0 v2:172.21.15.118:6800/27690 3 ==== osd_op_reply(12 gc.11 [create,call,call,call] v30'12 uv5 ondisk = -125 ((125) Operation canceled)) v8 ==== 275+0+0 (crc 0 0 0) 0x7f21e403d040 con 0x556c956c8470 2023-10-05T19:28:23.935 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:23.930+0000 7f21fd308e40 1 -- 172.21.15.167:0/3182581180 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:13 6.15 6:ab569f81:gc::gc.12:head [create,call version.check_conds in=74b,call rgw_gc.rgw_gc_queue_init in=45b,call version.set in=34b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x556c956dc4c0 con 0x556c956d7ce0 2023-10-05T19:28:23.938 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:23.934+0000 7f21fae3d700 1 -- 172.21.15.167:0/3182581180 <== osd.2 v2:172.21.15.167:6816/30037 4 ==== osd_op_reply(13 gc.12 [create,call,call,call] v30'10 uv3 ondisk = -125 ((125) Operation canceled)) v8 ==== 275+0+0 (crc 0 0 0) 0x7f21f4039040 con 0x556c956d7ce0 2023-10-05T19:28:23.938 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:23.934+0000 7f21fd308e40 1 -- 172.21.15.167:0/3182581180 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:14 6.0 6:0667bb13:gc::gc.13:head [create,call version.check_conds in=74b,call rgw_gc.rgw_gc_queue_init in=45b,call version.set in=34b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x556c956dc4c0 con 0x556c956c8470 2023-10-05T19:28:23.941 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:23.934+0000 7f21fa63c700 1 -- 172.21.15.167:0/3182581180 <== osd.0 v2:172.21.15.118:6800/27690 4 ==== osd_op_reply(14 gc.13 [create,call,call,call] v30'19 uv6 ondisk = -125 ((125) Operation canceled)) v8 ==== 275+0+0 (crc 0 0 0) 0x7f21e403d040 con 0x556c956c8470 2023-10-05T19:28:23.941 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:23.934+0000 7f21fd308e40 1 -- 172.21.15.167:0/3182581180 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:15 6.4 6:227f63b5:gc::gc.14:head [create,call version.check_conds in=74b,call rgw_gc.rgw_gc_queue_init in=45b,call version.set in=34b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x556c956dc9b0 con 0x556c956cc860 2023-10-05T19:28:23.944 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:23.938+0000 7f21fae3d700 1 -- 172.21.15.167:0/3182581180 <== osd.1 v2:172.21.15.167:6800/30035 4 ==== osd_op_reply(15 gc.14 [create,call,call,call] v30'28 uv13 ondisk = -125 ((125) Operation canceled)) v8 ==== 275+0+0 (crc 0 0 0) 0x7f21f4039040 con 0x556c956cc860 2023-10-05T19:28:23.944 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:23.938+0000 7f21fd308e40 1 -- 172.21.15.167:0/3182581180 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:16 6.7 6:e07a6548:gc::gc.15:head [create,call version.check_conds in=74b,call rgw_gc.rgw_gc_queue_init in=45b,call version.set in=34b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x556c956dc9b0 con 0x556c956d3f10 2023-10-05T19:28:23.947 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:23.942+0000 7f21fb63e700 1 -- 172.21.15.167:0/3182581180 <== osd.3 v2:172.21.15.167:6808/30036 4 ==== osd_op_reply(16 gc.15 [create,call,call,call] v30'14 uv7 ondisk = -125 ((125) Operation canceled)) v8 ==== 275+0+0 (crc 0 0 0) 0x7f21ec050040 con 0x556c956d3f10 2023-10-05T19:28:23.947 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:23.942+0000 7f21fd308e40 1 -- 172.21.15.167:0/3182581180 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:17 6.3 6:c623b173:gc::gc.16:head [create,call version.check_conds in=74b,call rgw_gc.rgw_gc_queue_init in=45b,call version.set in=34b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x556c956dc9b0 con 0x556c956d7ce0 2023-10-05T19:28:23.950 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:23.946+0000 7f21fae3d700 1 -- 172.21.15.167:0/3182581180 <== osd.2 v2:172.21.15.167:6816/30037 5 ==== osd_op_reply(17 gc.16 [create,call,call,call] v30'22 uv13 ondisk = -125 ((125) Operation canceled)) v8 ==== 275+0+0 (crc 0 0 0) 0x7f21f4039040 con 0x556c956d7ce0 2023-10-05T19:28:23.950 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:23.946+0000 7f21fd308e40 1 -- 172.21.15.167:0/3182581180 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:18 6.16 6:6849fd06:gc::gc.17:head [create,call version.check_conds in=74b,call rgw_gc.rgw_gc_queue_init in=45b,call version.set in=34b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x556c956dc9b0 con 0x556c956d3f10 2023-10-05T19:28:23.952 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:23.946+0000 7f21fb63e700 1 -- 172.21.15.167:0/3182581180 <== osd.3 v2:172.21.15.167:6808/30036 5 ==== osd_op_reply(18 gc.17 [create,call,call,call] v30'10 uv7 ondisk = -125 ((125) Operation canceled)) v8 ==== 275+0+0 (crc 0 0 0) 0x7f21ec050040 con 0x556c956d3f10 2023-10-05T19:28:23.952 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:23.946+0000 7f21fd308e40 1 -- 172.21.15.167:0/3182581180 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:19 6.d 6:b019ea83:gc::gc.18:head [create,call version.check_conds in=74b,call rgw_gc.rgw_gc_queue_init in=45b,call version.set in=34b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x556c956dc9b0 con 0x556c956cc860 2023-10-05T19:28:23.955 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:23.950+0000 7f21fae3d700 1 -- 172.21.15.167:0/3182581180 <== osd.1 v2:172.21.15.167:6800/30035 5 ==== osd_op_reply(19 gc.18 [create,call,call,call] v30'18 uv5 ondisk = -125 ((125) Operation canceled)) v8 ==== 275+0+0 (crc 0 0 0) 0x7f21f4039040 con 0x556c956cc860 2023-10-05T19:28:23.955 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:23.950+0000 7f21fd308e40 1 -- 172.21.15.167:0/3182581180 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:20 6.4 6:24c8e1ae:gc::gc.19:head [create,call version.check_conds in=74b,call rgw_gc.rgw_gc_queue_init in=45b,call version.set in=34b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x556c956dca40 con 0x556c956cc860 2023-10-05T19:28:23.958 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:23.950+0000 7f21fae3d700 1 -- 172.21.15.167:0/3182581180 <== osd.1 v2:172.21.15.167:6800/30035 6 ==== osd_op_reply(20 gc.19 [create,call,call,call] v30'29 uv15 ondisk = -125 ((125) Operation canceled)) v8 ==== 275+0+0 (crc 0 0 0) 0x7f21f4039040 con 0x556c956cc860 2023-10-05T19:28:23.958 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:23.950+0000 7f21fd308e40 1 -- 172.21.15.167:0/3182581180 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:21 6.7 6:e622f76a:gc::gc.20:head [create,call version.check_conds in=74b,call rgw_gc.rgw_gc_queue_init in=45b,call version.set in=34b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x556c956dca40 con 0x556c956d3f10 2023-10-05T19:28:23.961 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:23.954+0000 7f21fb63e700 1 -- 172.21.15.167:0/3182581180 <== osd.3 v2:172.21.15.167:6808/30036 6 ==== osd_op_reply(21 gc.20 [create,call,call,call] v30'15 uv9 ondisk = -125 ((125) Operation canceled)) v8 ==== 275+0+0 (crc 0 0 0) 0x7f21ec050040 con 0x556c956d3f10 2023-10-05T19:28:23.961 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:23.954+0000 7f21fd308e40 1 -- 172.21.15.167:0/3182581180 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:22 6.a 6:5127e642:gc::gc.21:head [create,call version.check_conds in=74b,call rgw_gc.rgw_gc_queue_init in=45b,call version.set in=34b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x556c956dca40 con 0x556c956d7ce0 2023-10-05T19:28:23.963 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:23.958+0000 7f21fae3d700 1 -- 172.21.15.167:0/3182581180 <== osd.2 v2:172.21.15.167:6816/30037 6 ==== osd_op_reply(22 gc.21 [create,call,call,call] v30'20 uv7 ondisk = -125 ((125) Operation canceled)) v8 ==== 275+0+0 (crc 0 0 0) 0x7f21f4039040 con 0x556c956d7ce0 2023-10-05T19:28:23.963 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:23.958+0000 7f21fd308e40 1 -- 172.21.15.167:0/3182581180 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:23 6.11 6:8eb99c53:gc::gc.22:head [create,call version.check_conds in=74b,call rgw_gc.rgw_gc_queue_init in=45b,call version.set in=34b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x556c956dca40 con 0x556c956d3f10 2023-10-05T19:28:23.965 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:23.958+0000 7f21fb63e700 1 -- 172.21.15.167:0/3182581180 <== osd.3 v2:172.21.15.167:6808/30036 7 ==== osd_op_reply(23 gc.22 [create,call,call,call] v30'12 uv3 ondisk = -125 ((125) Operation canceled)) v8 ==== 275+0+0 (crc 0 0 0) 0x7f21ec050040 con 0x556c956d3f10 2023-10-05T19:28:23.966 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:23.958+0000 7f21fd308e40 1 -- 172.21.15.167:0/3182581180 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:24 6.8 6:17d7e8ed:gc::gc.23:head [create,call version.check_conds in=74b,call rgw_gc.rgw_gc_queue_init in=45b,call version.set in=34b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x556c956dc9b0 con 0x556c956d3f10 2023-10-05T19:28:23.968 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:23.962+0000 7f21fb63e700 1 -- 172.21.15.167:0/3182581180 <== osd.3 v2:172.21.15.167:6808/30036 8 ==== osd_op_reply(24 gc.23 [create,call,call,call] v30'17 uv5 ondisk = -125 ((125) Operation canceled)) v8 ==== 275+0+0 (crc 0 0 0) 0x7f21ec050040 con 0x556c956d3f10 2023-10-05T19:28:23.968 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:23.962+0000 7f21fd308e40 1 -- 172.21.15.167:0/3182581180 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:25 6.1b 6:db9d130d:gc::gc.24:head [create,call version.check_conds in=74b,call rgw_gc.rgw_gc_queue_init in=45b,call version.set in=34b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x556c956dca40 con 0x556c956d3f10 2023-10-05T19:28:23.970 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:23.966+0000 7f21fb63e700 1 -- 172.21.15.167:0/3182581180 <== osd.3 v2:172.21.15.167:6808/30036 9 ==== osd_op_reply(25 gc.24 [create,call,call,call] v30'6 uv3 ondisk = -125 ((125) Operation canceled)) v8 ==== 275+0+0 (crc 0 0 0) 0x7f21ec050040 con 0x556c956d3f10 2023-10-05T19:28:23.970 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:23.966+0000 7f21fd308e40 1 -- 172.21.15.167:0/3182581180 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:26 6.9 6:921ecdb1:gc::gc.25:head [create,call version.check_conds in=74b,call rgw_gc.rgw_gc_queue_init in=45b,call version.set in=34b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x556c956dca40 con 0x556c956d3f10 2023-10-05T19:28:23.973 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:23.966+0000 7f21fb63e700 1 -- 172.21.15.167:0/3182581180 <== osd.3 v2:172.21.15.167:6808/30036 10 ==== osd_op_reply(26 gc.25 [create,call,call,call] v30'14 uv10 ondisk = -125 ((125) Operation canceled)) v8 ==== 275+0+0 (crc 0 0 0) 0x7f21ec050040 con 0x556c956d3f10 2023-10-05T19:28:23.973 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:23.966+0000 7f21fd308e40 1 -- 172.21.15.167:0/3182581180 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:27 6.1f 6:fde4dd55:gc::gc.26:head [create,call version.check_conds in=74b,call rgw_gc.rgw_gc_queue_init in=45b,call version.set in=34b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x556c956dca40 con 0x556c956c8470 2023-10-05T19:28:23.976 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:23.970+0000 7f21fa63c700 1 -- 172.21.15.167:0/3182581180 <== osd.0 v2:172.21.15.118:6800/27690 5 ==== osd_op_reply(27 gc.26 [create,call,call,call] v30'12 uv5 ondisk = -125 ((125) Operation canceled)) v8 ==== 275+0+0 (crc 0 0 0) 0x7f21e403d040 con 0x556c956c8470 2023-10-05T19:28:23.976 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:23.970+0000 7f21fd308e40 1 -- 172.21.15.167:0/3182581180 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:28 6.1a 6:5909e671:gc::gc.27:head [create,call version.check_conds in=74b,call rgw_gc.rgw_gc_queue_init in=45b,call version.set in=34b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x556c956dca40 con 0x556c956d3f10 2023-10-05T19:28:23.978 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:23.974+0000 7f21fb63e700 1 -- 172.21.15.167:0/3182581180 <== osd.3 v2:172.21.15.167:6808/30036 11 ==== osd_op_reply(28 gc.27 [create,call,call,call] v30'10 uv7 ondisk = -125 ((125) Operation canceled)) v8 ==== 275+0+0 (crc 0 0 0) 0x7f21ec050040 con 0x556c956d3f10 2023-10-05T19:28:23.979 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:23.974+0000 7f21fd308e40 1 -- 172.21.15.167:0/3182581180 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:29 6.1d 6:b88e3271:gc::gc.28:head [create,call version.check_conds in=74b,call rgw_gc.rgw_gc_queue_init in=45b,call version.set in=34b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x556c956dc9b0 con 0x556c956c8470 2023-10-05T19:28:23.982 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:23.974+0000 7f21fa63c700 1 -- 172.21.15.167:0/3182581180 <== osd.0 v2:172.21.15.118:6800/27690 6 ==== osd_op_reply(29 gc.28 [create,call,call,call] v30'14 uv8 ondisk = -125 ((125) Operation canceled)) v8 ==== 275+0+0 (crc 0 0 0) 0x7f21e403d040 con 0x556c956c8470 2023-10-05T19:28:23.982 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:23.974+0000 7f21fd308e40 1 -- 172.21.15.167:0/3182581180 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:30 6.7 6:e47c9322:gc::gc.29:head [create,call version.check_conds in=74b,call rgw_gc.rgw_gc_queue_init in=45b,call version.set in=34b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x556c956dca40 con 0x556c956d3f10 2023-10-05T19:28:23.985 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:23.978+0000 7f21fb63e700 1 -- 172.21.15.167:0/3182581180 <== osd.3 v2:172.21.15.167:6808/30036 12 ==== osd_op_reply(30 gc.29 [create,call,call,call] v30'16 uv11 ondisk = -125 ((125) Operation canceled)) v8 ==== 275+0+0 (crc 0 0 0) 0x7f21ec050040 con 0x556c956d3f10 2023-10-05T19:28:23.985 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:23.978+0000 7f21fd308e40 1 -- 172.21.15.167:0/3182581180 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:31 6.0 6:013d6c33:gc::gc.30:head [create,call version.check_conds in=74b,call rgw_gc.rgw_gc_queue_init in=45b,call version.set in=34b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x556c956dca40 con 0x556c956c8470 2023-10-05T19:28:23.987 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:23.982+0000 7f21fa63c700 1 -- 172.21.15.167:0/3182581180 <== osd.0 v2:172.21.15.118:6800/27690 7 ==== osd_op_reply(31 gc.30 [create,call,call,call] v30'20 uv12 ondisk = -125 ((125) Operation canceled)) v8 ==== 275+0+0 (crc 0 0 0) 0x7f21e403d040 con 0x556c956c8470 2023-10-05T19:28:23.988 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:23.982+0000 7f21fd308e40 1 -- 172.21.15.167:0/3182581180 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:32 6.f 6:f24cb4a8:gc::gc.31:head [create,call version.check_conds in=74b,call rgw_gc.rgw_gc_queue_init in=45b,call version.set in=34b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x556c956dc9b0 con 0x556c956d7ce0 2023-10-05T19:28:23.990 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:23.986+0000 7f21fae3d700 1 -- 172.21.15.167:0/3182581180 <== osd.2 v2:172.21.15.167:6816/30037 7 ==== osd_op_reply(32 gc.31 [create,call,call,call] v30'16 uv5 ondisk = -125 ((125) Operation canceled)) v8 ==== 275+0+0 (crc 0 0 0) 0x7f21f4039040 con 0x556c956d7ce0 2023-10-05T19:28:23.993 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:23.986+0000 7f213effd700 20 reqs_thread_entry: start 2023-10-05T19:28:23.994 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:23.986+0000 7f213cff9700 20 reqs_thread_entry: start 2023-10-05T19:28:23.994 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:23.986+0000 7f21fd308e40 20 init_complete bucket index max shards: 11 2023-10-05T19:28:23.995 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:23.990+0000 7f21fd308e40 10 rgw notify: Started notification manager with: 1 workers 2023-10-05T19:28:23.995 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:23.990+0000 7f212affd700 20 rgw notify: INFO: next queues processing will happen at: Thu Oct 5 19:28:54 2023 2023-10-05T19:28:23.995 INFO:teuthology.orchestra.run.smithi167.stderr: 2023-10-05T19:28:23.995 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:23.990+0000 7f212a7fc700 20 reqs_thread_entry: start 2023-10-05T19:28:23.995 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:23.990+0000 7f21fd308e40 20 get_system_obj_state: rctx=0x7ffeb9034f18 obj=default.rgw.meta:users.uid:bud state=0x556c955450b0 s->prefetch_data=0 2023-10-05T19:28:23.995 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:23.990+0000 7f21fd308e40 10 cache get: name=default.rgw.meta+users.uid+bud : miss 2023-10-05T19:28:23.995 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:23.990+0000 7f21fd308e40 1 -- 172.21.15.167:0/3301061136 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:102 8.4 8:267425a8:users.uid::bud:head [call version.read in=11b,getxattrs,stat] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x556c9573b450 con 0x7f21f401a070 2023-10-05T19:28:23.996 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:23.990+0000 7f21fa63c700 1 -- 172.21.15.167:0/3301061136 <== osd.2 v2:172.21.15.167:6816/30037 17 ==== osd_op_reply(102 bud [call,getxattrs,stat] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 ==== 231+0+0 (crc 0 0 0) 0x7f21e4050040 con 0x7f21f401a070 2023-10-05T19:28:23.996 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:23.990+0000 7f21fd308e40 10 cache put: name=default.rgw.meta+users.uid+bud info.flags=0x0 2023-10-05T19:28:23.996 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:23.990+0000 7f21fd308e40 10 adding default.rgw.meta+users.uid+bud to cache LRU end 2023-10-05T19:28:23.996 INFO:teuthology.orchestra.run.smithi167.stderr:could not remove user: unable to remove user, user does not exist 2023-10-05T19:28:23.996 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:23.990+0000 7f21fd308e40 1 -- 172.21.15.167:0/3301061136 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:103 7.1f 7:f95f44c2:::notify.0:head [watch unwatch cookie 93924851615568] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x556c9573b450 con 0x7f21f401a070 2023-10-05T19:28:23.998 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:23.994+0000 7f21fa63c700 1 -- 172.21.15.167:0/3301061136 <== osd.2 v2:172.21.15.167:6816/30037 18 ==== osd_op_reply(103 notify.0 [watch unwatch cookie 93924851615568] v30'8 uv6 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f21e4050040 con 0x7f21f401a070 2023-10-05T19:28:23.999 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:23.994+0000 7f21fd308e40 20 remove_watcher() i=0 2023-10-05T19:28:23.999 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:23.994+0000 7f21fd308e40 2 removed watcher, disabling cache 2023-10-05T19:28:23.999 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:23.994+0000 7f21fd308e40 1 -- 172.21.15.167:0/3301061136 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:104 7.0 7:05bf5b68:::notify.1:head [watch unwatch cookie 93924851619168] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x556c9573b450 con 0x556c95672600 2023-10-05T19:28:24.001 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:23.994+0000 7f21fb63e700 1 -- 172.21.15.167:0/3301061136 <== osd.1 v2:172.21.15.167:6800/30035 35 ==== osd_op_reply(104 notify.1 [watch unwatch cookie 93924851619168] v30'8 uv6 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f21ec050040 con 0x556c95672600 2023-10-05T19:28:24.002 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:23.994+0000 7f21fd308e40 20 remove_watcher() i=1 2023-10-05T19:28:24.002 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:23.994+0000 7f21fd308e40 1 -- 172.21.15.167:0/3301061136 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:105 7.15 7:a93a5511:::notify.2:head [watch unwatch cookie 93924851620272] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x556c9573b450 con 0x556c956a37f0 2023-10-05T19:28:24.005 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:23.998+0000 7f21fae3d700 1 -- 172.21.15.167:0/3301061136 <== osd.3 v2:172.21.15.167:6808/30036 24 ==== osd_op_reply(105 notify.2 [watch unwatch cookie 93924851620272] v30'8 uv6 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f21f4039040 con 0x556c956a37f0 2023-10-05T19:28:24.005 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:23.998+0000 7f21fd308e40 20 remove_watcher() i=2 2023-10-05T19:28:24.005 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:23.998+0000 7f21fd308e40 1 -- 172.21.15.167:0/3301061136 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:106 7.e 7:7759931f:::notify.3:head [watch unwatch cookie 93924851628400] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x556c9573b450 con 0x7f21f401a070 2023-10-05T19:28:24.007 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.002+0000 7f21fa63c700 1 -- 172.21.15.167:0/3301061136 <== osd.2 v2:172.21.15.167:6816/30037 19 ==== osd_op_reply(106 notify.3 [watch unwatch cookie 93924851628400] v30'8 uv6 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f21e4050040 con 0x7f21f401a070 2023-10-05T19:28:24.007 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.002+0000 7f21fd308e40 20 remove_watcher() i=3 2023-10-05T19:28:24.007 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.002+0000 7f21fd308e40 1 -- 172.21.15.167:0/3301061136 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:107 7.d 7:b4812045:::notify.4:head [watch unwatch cookie 93924851631712] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x556c9573b450 con 0x556c95672600 2023-10-05T19:28:24.010 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.006+0000 7f21fb63e700 1 -- 172.21.15.167:0/3301061136 <== osd.1 v2:172.21.15.167:6800/30035 36 ==== osd_op_reply(107 notify.4 [watch unwatch cookie 93924851631712] v30'8 uv6 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f21ec050040 con 0x556c95672600 2023-10-05T19:28:24.010 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.006+0000 7f21fd308e40 20 remove_watcher() i=4 2023-10-05T19:28:24.010 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.006+0000 7f21fd308e40 1 -- 172.21.15.167:0/3301061136 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:108 7.3 7:c609908c:::notify.5:head [watch unwatch cookie 93924851629504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x556c9573b450 con 0x7f21e4010070 2023-10-05T19:28:24.013 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.006+0000 7f21fb63e700 1 -- 172.21.15.167:0/3301061136 <== osd.0 v2:172.21.15.118:6800/27690 29 ==== osd_op_reply(108 notify.5 [watch unwatch cookie 93924851629504] v30'8 uv6 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f21ec050040 con 0x7f21e4010070 2023-10-05T19:28:24.013 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.006+0000 7f21fd308e40 20 remove_watcher() i=5 2023-10-05T19:28:24.013 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.006+0000 7f21fd308e40 1 -- 172.21.15.167:0/3301061136 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:109 7.14 7:2b04a3e9:::notify.6:head [watch unwatch cookie 93924851635936] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x556c956abe60 con 0x7f21e4010070 2023-10-05T19:28:24.016 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.010+0000 7f21fb63e700 1 -- 172.21.15.167:0/3301061136 <== osd.0 v2:172.21.15.118:6800/27690 30 ==== osd_op_reply(109 notify.6 [watch unwatch cookie 93924851635936] v30'8 uv6 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f21ec050040 con 0x7f21e4010070 2023-10-05T19:28:24.016 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.010+0000 7f21fd308e40 20 remove_watcher() i=6 2023-10-05T19:28:24.016 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.010+0000 7f21fd308e40 1 -- 172.21.15.167:0/3301061136 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:110 7.9 7:93e5b521:::notify.7:head [watch unwatch cookie 93924851622320] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x556c9573b450 con 0x7f21e4010070 2023-10-05T19:28:24.019 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.014+0000 7f21fb63e700 1 -- 172.21.15.167:0/3301061136 <== osd.0 v2:172.21.15.118:6800/27690 31 ==== osd_op_reply(110 notify.7 [watch unwatch cookie 93924851622320] v30'8 uv6 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f21ec050040 con 0x7f21e4010070 2023-10-05T19:28:24.019 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.014+0000 7f21fd308e40 20 remove_watcher() i=7 2023-10-05T19:28:24.021 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.014+0000 7f2147fff700 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2023-10-05T19:28:24.021 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.014+0000 7f21fd308e40 1 -- 172.21.15.167:0/3301061136 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:111 6.4 6:22d26bf9:::data_loggenerations_metadata:head [watch unwatch cookie 93924851640064] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x556c956abe60 con 0x556c95672600 2023-10-05T19:28:24.024 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.018+0000 7f21fb63e700 1 -- 172.21.15.167:0/3301061136 <== osd.1 v2:172.21.15.167:6800/30035 37 ==== osd_op_reply(111 data_loggenerations_metadata [watch unwatch cookie 93924851640064] v30'30 uv1 ondisk = 0) v8 ==== 172+0+0 (crc 0 0 0) 0x7f21ec050040 con 0x556c95672600 2023-10-05T19:28:24.025 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.018+0000 7f21fd308e40 1 -- 172.21.15.167:0/3301061136 >> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] conn(0x7f21e4010070 msgr2=0x7f21e401c430 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-05T19:28:24.025 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.018+0000 7f21fd308e40 1 --2- 172.21.15.167:0/3301061136 >> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] conn(0x7f21e4010070 0x7f21e401c430 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-05T19:28:24.025 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.018+0000 7f21fd308e40 1 -- 172.21.15.167:0/3301061136 >> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] conn(0x556c95672600 msgr2=0x556c9569f9c0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-05T19:28:24.025 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.018+0000 7f21fd308e40 1 --2- 172.21.15.167:0/3301061136 >> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] conn(0x556c95672600 0x556c9569f9c0 crc :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-05T19:28:24.025 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.018+0000 7f21fd308e40 1 -- 172.21.15.167:0/3301061136 >> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] conn(0x7f21f401a070 msgr2=0x7f21f4042cb0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-05T19:28:24.025 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.018+0000 7f21fd308e40 1 --2- 172.21.15.167:0/3301061136 >> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] conn(0x7f21f401a070 0x7f21f4042cb0 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-05T19:28:24.025 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.018+0000 7f21fd308e40 1 -- 172.21.15.167:0/3301061136 >> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] conn(0x556c956a37f0 msgr2=0x556c956a5bd0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-05T19:28:24.025 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.018+0000 7f21fd308e40 1 --2- 172.21.15.167:0/3301061136 >> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] conn(0x556c956a37f0 0x556c956a5bd0 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-05T19:28:24.025 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.018+0000 7f21fd308e40 1 -- 172.21.15.167:0/3301061136 >> [v2:172.21.15.167:6824/29889,v1:172.21.15.167:6825/29889] conn(0x7f21dc030c50 msgr2=0x7f21dc033100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-05T19:28:24.025 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.018+0000 7f21fd308e40 1 --2- 172.21.15.167:0/3301061136 >> [v2:172.21.15.167:6824/29889,v1:172.21.15.167:6825/29889] conn(0x7f21dc030c50 0x7f21dc033100 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f21e4005e40 tx=0x7f21e4013000 comp rx=0 tx=0).stop 2023-10-05T19:28:24.025 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.018+0000 7f21fd308e40 1 -- 172.21.15.167:0/3301061136 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x556c9566e020 msgr2=0x556c95693ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-05T19:28:24.026 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.018+0000 7f21fb63e700 1 -- 172.21.15.167:0/3301061136 reap_dead start 2023-10-05T19:28:24.026 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.018+0000 7f21fd308e40 1 --2- 172.21.15.167:0/3301061136 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x556c9566e020 0x556c95693ed0 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7f21f4000c00 tx=0x7f21f4016000 comp rx=0 tx=0).stop 2023-10-05T19:28:24.026 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.018+0000 7f21fd308e40 1 -- 172.21.15.167:0/3301061136 shutdown_connections 2023-10-05T19:28:24.026 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.018+0000 7f21fd308e40 1 --2- 172.21.15.167:0/3301061136 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x556c9566e020 0x556c95693ed0 unknown :-1 s=CLOSED pgs=143 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-05T19:28:24.026 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.018+0000 7f21fd308e40 1 -- 172.21.15.167:0/3301061136 >> 172.21.15.167:0/3301061136 conn(0x556c955e1870 msgr2=0x556c95687740 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-05T19:28:24.026 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.018+0000 7f21fd308e40 1 -- 172.21.15.167:0/3301061136 shutdown_connections 2023-10-05T19:28:24.026 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.018+0000 7f21fd308e40 1 -- 172.21.15.167:0/3301061136 wait complete. 2023-10-05T19:28:24.027 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.018+0000 7f21fd308e40 1 -- 172.21.15.167:0/3182581180 >> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] conn(0x556c956c8470 msgr2=0x556c956d2980 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-05T19:28:24.027 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.018+0000 7f21fd308e40 1 --2- 172.21.15.167:0/3182581180 >> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] conn(0x556c956c8470 0x556c956d2980 crc :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-05T19:28:24.027 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.018+0000 7f21fd308e40 1 -- 172.21.15.167:0/3182581180 >> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] conn(0x556c956cc860 msgr2=0x556c956cec70 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-05T19:28:24.028 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.018+0000 7f21fd308e40 1 --2- 172.21.15.167:0/3182581180 >> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] conn(0x556c956cc860 0x556c956cec70 crc :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-05T19:28:24.028 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.018+0000 7f21fd308e40 1 -- 172.21.15.167:0/3182581180 >> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] conn(0x556c956d7ce0 msgr2=0x556c956da0c0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-05T19:28:24.028 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.018+0000 7f21fd308e40 1 --2- 172.21.15.167:0/3182581180 >> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] conn(0x556c956d7ce0 0x556c956da0c0 crc :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-05T19:28:24.028 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.018+0000 7f21fd308e40 1 -- 172.21.15.167:0/3182581180 >> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] conn(0x556c956d3f10 msgr2=0x556c956d62f0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-05T19:28:24.028 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.018+0000 7f21fd308e40 1 --2- 172.21.15.167:0/3182581180 >> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] conn(0x556c956d3f10 0x556c956d62f0 crc :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-05T19:28:24.028 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.018+0000 7f21fd308e40 1 -- 172.21.15.167:0/3182581180 >> [v2:172.21.15.167:6824/29889,v1:172.21.15.167:6825/29889] conn(0x7f2138030bb0 msgr2=0x7f2138033060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-05T19:28:24.028 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.018+0000 7f21fd308e40 1 --2- 172.21.15.167:0/3182581180 >> [v2:172.21.15.167:6824/29889,v1:172.21.15.167:6825/29889] conn(0x7f2138030bb0 0x7f2138033060 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x556c956ae020 tx=0x7f21ec000f90 comp rx=0 tx=0).stop 2023-10-05T19:28:24.028 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.018+0000 7f21fd308e40 1 -- 172.21.15.167:0/3182581180 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x556c956bc110 msgr2=0x556c956c1c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-05T19:28:24.028 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.018+0000 7f21fd308e40 1 --2- 172.21.15.167:0/3182581180 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x556c956bc110 0x556c956c1c40 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7f21e400f040 tx=0x7f21e402d030 comp rx=0 tx=0).stop 2023-10-05T19:28:24.028 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.018+0000 7f21fae3d700 1 -- 172.21.15.167:0/3182581180 reap_dead start 2023-10-05T19:28:24.028 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.018+0000 7f21fd308e40 1 -- 172.21.15.167:0/3182581180 shutdown_connections 2023-10-05T19:28:24.029 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.018+0000 7f21fd308e40 1 --2- 172.21.15.167:0/3182581180 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x556c956bc110 0x556c956c1c40 unknown :-1 s=CLOSED pgs=145 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-05T19:28:24.029 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.022+0000 7f21fd308e40 1 -- 172.21.15.167:0/3182581180 >> 172.21.15.167:0/3182581180 conn(0x556c956b74d0 msgr2=0x556c956b5570 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-05T19:28:24.029 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.022+0000 7f21fd308e40 1 -- 172.21.15.167:0/3182581180 shutdown_connections 2023-10-05T19:28:24.029 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.022+0000 7f21fd308e40 1 -- 172.21.15.167:0/3182581180 wait complete. 2023-10-05T19:28:24.031 DEBUG:teuthology.orchestra.run:got remote process result: 2 2023-10-05T19:28:24.032 INFO:tasks.util.rgw:rgwadmin: client.0 : ['user', 'rm', '--uid', 'bar'] 2023-10-05T19:28:24.032 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'ceph', 'user', 'rm', '--uid', 'bar'] 2023-10-05T19:28:24.032 DEBUG:teuthology.orchestra.run.smithi167:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster ceph user rm --uid bar 2023-10-05T19:28:24.073 INFO:teuthology.orchestra.run.smithi167.stderr:ignoring --setuser ceph since I am not root 2023-10-05T19:28:24.073 INFO:teuthology.orchestra.run.smithi167.stderr:ignoring --setgroup ceph since I am not root 2023-10-05T19:28:24.074 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.066+0000 7ff9ea92de40 1 Processor -- start 2023-10-05T19:28:24.074 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.066+0000 7ff9ea92de40 1 -- start start 2023-10-05T19:28:24.074 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.066+0000 7ff9ea92de40 1 --2- >> v2:172.21.15.118:3300/0 conn(0x55dfdac8d910 0x55dfdad21f60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-05T19:28:24.074 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.066+0000 7ff9ea92de40 1 -- --> v1:172.21.15.118:6789/0 -- auth(proto 0 26 bytes epoch 0) v1 -- 0x55dfdac65bd0 con 0x55dfdac8ec20 2023-10-05T19:28:24.074 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.066+0000 7ff9ea92de40 1 -- --> v2:172.21.15.118:3300/0 -- mon_getmap magic: 0 v1 -- 0x55dfdac22ee0 con 0x55dfdac8d910 2023-10-05T19:28:24.074 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.070+0000 7ff9e7c61700 1 --2- >> v2:172.21.15.118:3300/0 conn(0x55dfdac8d910 0x55dfdad21f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-05T19:28:24.074 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.070+0000 7ff9e8462700 1 --1- >> v1:172.21.15.118:6789/0 conn(0x55dfdac8ec20 0x55dfdac8d200 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.118:6789/0 says I am v1:172.21.15.167:37562/0 (socket says 172.21.15.167:37562) 2023-10-05T19:28:24.075 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.070+0000 7ff9e8462700 1 -- 172.21.15.167:0/1011025208 learned_addr learned my addr 172.21.15.167:0/1011025208 (peer_addr_for_me v1:172.21.15.167:0/0) 2023-10-05T19:28:24.075 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.070+0000 7ff9e7460700 1 -- 172.21.15.167:0/1011025208 <== mon.0 v1:172.21.15.118:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3911221054 0 0) 0x55dfdac65bd0 con 0x55dfdac8ec20 2023-10-05T19:28:24.076 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.070+0000 7ff9e7460700 1 -- 172.21.15.167:0/1011025208 --> v1:172.21.15.118:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff9d0001e10 con 0x55dfdac8ec20 2023-10-05T19:28:24.076 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.070+0000 7ff9e7c61700 1 -- 172.21.15.167:0/1011025208 >> v1:172.21.15.118:6789/0 conn(0x55dfdac8ec20 legacy=0x55dfdac8d200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-05T19:28:24.076 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.070+0000 7ff9e7c61700 1 -- 172.21.15.167:0/1011025208 --> v2:172.21.15.118:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55dfdac8f0b0 con 0x55dfdac8d910 2023-10-05T19:28:24.076 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.070+0000 7ff9e7c61700 1 --2- 172.21.15.167:0/1011025208 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55dfdac8d910 0x55dfdad21f60 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7ff9dc00a5d0 tx=0x7ff9dc005a50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=383234316f3927c0 server_cookie=0 in_seq=0 out_seq=0 2023-10-05T19:28:24.076 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.070+0000 7ff9e7460700 1 -- 172.21.15.167:0/1011025208 <== mon.0 v2:172.21.15.118:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7ff9dc007610 con 0x55dfdac8d910 2023-10-05T19:28:24.076 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.070+0000 7ff9e7460700 1 -- 172.21.15.167:0/1011025208 <== mon.0 v2:172.21.15.118:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff9dc007770 con 0x55dfdac8d910 2023-10-05T19:28:24.076 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.070+0000 7ff9e7460700 1 -- 172.21.15.167:0/1011025208 <== mon.0 v2:172.21.15.118:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7ff9dc007a40 con 0x55dfdac8d910 2023-10-05T19:28:24.077 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.070+0000 7ff9ea92de40 1 -- 172.21.15.167:0/1011025208 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55dfdac8d910 msgr2=0x55dfdad21f60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-05T19:28:24.077 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.070+0000 7ff9ea92de40 1 --2- 172.21.15.167:0/1011025208 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55dfdac8d910 0x55dfdad21f60 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7ff9dc00a5d0 tx=0x7ff9dc005a50 comp rx=0 tx=0).stop 2023-10-05T19:28:24.077 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.070+0000 7ff9ea92de40 1 -- 172.21.15.167:0/1011025208 shutdown_connections 2023-10-05T19:28:24.078 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.070+0000 7ff9ea92de40 1 --2- 172.21.15.167:0/1011025208 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55dfdac8d910 0x55dfdad21f60 unknown :-1 s=CLOSED pgs=147 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-05T19:28:24.078 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.070+0000 7ff9ea92de40 1 -- 172.21.15.167:0/1011025208 >> 172.21.15.167:0/1011025208 conn(0x55dfdad14850 msgr2=0x55dfdad16c60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-05T19:28:24.078 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.070+0000 7ff9ea92de40 1 -- 172.21.15.167:0/1011025208 shutdown_connections 2023-10-05T19:28:24.078 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.070+0000 7ff9ea92de40 1 -- 172.21.15.167:0/1011025208 wait complete. 2023-10-05T19:28:24.078 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.070+0000 7ff9ea92de40 1 Processor -- start 2023-10-05T19:28:24.078 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.070+0000 7ff9ea92de40 1 -- start start 2023-10-05T19:28:24.078 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.070+0000 7ff9ea92de40 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55dfdac8d910 0x55dfdad39f10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-05T19:28:24.078 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.070+0000 7ff9ea92de40 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x55dfdabe6e30 con 0x55dfdac8d910 2023-10-05T19:28:24.079 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.074+0000 7ff9e8462700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55dfdac8d910 0x55dfdad39f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-05T19:28:24.079 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.074+0000 7ff9e8462700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55dfdac8d910 0x55dfdad39f10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.118:3300/0 says I am v2:172.21.15.167:50668/0 (socket says 172.21.15.167:50668) 2023-10-05T19:28:24.079 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.074+0000 7ff9e8462700 1 -- 172.21.15.167:0/2754223304 learned_addr learned my addr 172.21.15.167:0/2754223304 (peer_addr_for_me v2:172.21.15.167:0/0) 2023-10-05T19:28:24.079 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.074+0000 7ff9e8462700 1 -- 172.21.15.167:0/2754223304 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff9dc005cf0 con 0x55dfdac8d910 2023-10-05T19:28:24.080 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.074+0000 7ff9e8462700 1 --2- 172.21.15.167:0/2754223304 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55dfdac8d910 0x55dfdad39f10 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7ff9e0006e70 tx=0x7ff9e0006a20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f2a297f756c8e90 server_cookie=0 in_seq=0 out_seq=0 2023-10-05T19:28:24.080 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.074+0000 7ff9e645e700 1 -- 172.21.15.167:0/2754223304 <== mon.0 v2:172.21.15.118:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7ff9e0006110 con 0x55dfdac8d910 2023-10-05T19:28:24.080 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.074+0000 7ff9e645e700 1 -- 172.21.15.167:0/2754223304 <== mon.0 v2:172.21.15.118:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff9e0006290 con 0x55dfdac8d910 2023-10-05T19:28:24.080 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.074+0000 7ff9e645e700 1 -- 172.21.15.167:0/2754223304 <== mon.0 v2:172.21.15.118:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7ff9e0004520 con 0x55dfdac8d910 2023-10-05T19:28:24.081 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.074+0000 7ff9ea92de40 1 -- 172.21.15.167:0/2754223304 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55dfdac8d910 msgr2=0x55dfdad39f10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-05T19:28:24.081 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.074+0000 7ff9ea92de40 1 --2- 172.21.15.167:0/2754223304 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55dfdac8d910 0x55dfdad39f10 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7ff9e0006e70 tx=0x7ff9e0006a20 comp rx=0 tx=0).stop 2023-10-05T19:28:24.081 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.074+0000 7ff9ea92de40 1 -- 172.21.15.167:0/2754223304 shutdown_connections 2023-10-05T19:28:24.081 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.074+0000 7ff9ea92de40 1 --2- 172.21.15.167:0/2754223304 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55dfdac8d910 0x55dfdad39f10 unknown :-1 s=CLOSED pgs=148 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-05T19:28:24.081 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.074+0000 7ff9ea92de40 1 -- 172.21.15.167:0/2754223304 >> 172.21.15.167:0/2754223304 conn(0x55dfdad14850 msgr2=0x55dfdad35ad0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-05T19:28:24.081 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.074+0000 7ff9ea92de40 1 -- 172.21.15.167:0/2754223304 shutdown_connections 2023-10-05T19:28:24.081 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.074+0000 7ff9ea92de40 1 -- 172.21.15.167:0/2754223304 wait complete. 2023-10-05T19:28:24.082 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.074+0000 7ff9ea92de40 1 Processor -- start 2023-10-05T19:28:24.082 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.074+0000 7ff9ea92de40 1 -- start start 2023-10-05T19:28:24.082 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.074+0000 7ff9ea92de40 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55dfdac8d910 0x55dfdad40cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-05T19:28:24.082 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.078+0000 7ff9ea92de40 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x55dfdabb9ed0 con 0x55dfdac8d910 2023-10-05T19:28:24.083 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.078+0000 7ff9e8462700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55dfdac8d910 0x55dfdad40cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-05T19:28:24.083 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.078+0000 7ff9e8462700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55dfdac8d910 0x55dfdad40cf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.118:3300/0 says I am v2:172.21.15.167:50672/0 (socket says 172.21.15.167:50672) 2023-10-05T19:28:24.083 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.078+0000 7ff9e8462700 1 -- 172.21.15.167:0/3611872326 learned_addr learned my addr 172.21.15.167:0/3611872326 (peer_addr_for_me v2:172.21.15.167:0/0) 2023-10-05T19:28:24.083 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.078+0000 7ff9e8462700 1 -- 172.21.15.167:0/3611872326 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff9d0001e10 con 0x55dfdac8d910 2023-10-05T19:28:24.083 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.078+0000 7ff9e8462700 1 --2- 172.21.15.167:0/3611872326 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55dfdac8d910 0x55dfdad40cf0 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x55dfdad365f0 tx=0x7ff9e0014000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-05T19:28:24.084 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.078+0000 7ff9e4c5b700 1 -- 172.21.15.167:0/3611872326 <== mon.0 v2:172.21.15.118:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7ff9e0013540 con 0x55dfdac8d910 2023-10-05T19:28:24.084 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.078+0000 7ff9e4c5b700 1 -- 172.21.15.167:0/3611872326 <== mon.0 v2:172.21.15.118:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff9e00136a0 con 0x55dfdac8d910 2023-10-05T19:28:24.084 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.078+0000 7ff9ea92de40 1 -- 172.21.15.167:0/3611872326 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x55dfdad41230 con 0x55dfdac8d910 2023-10-05T19:28:24.084 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.078+0000 7ff9e4c5b700 1 -- 172.21.15.167:0/3611872326 <== mon.0 v2:172.21.15.118:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7ff9e0013990 con 0x55dfdac8d910 2023-10-05T19:28:24.085 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.078+0000 7ff9ea92de40 1 -- 172.21.15.167:0/3611872326 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55dfdad41670 con 0x55dfdac8d910 2023-10-05T19:28:24.086 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.078+0000 7ff9e4c5b700 1 -- 172.21.15.167:0/3611872326 <== mon.0 v2:172.21.15.118:3300/0 4 ==== mgrmap(e 5) v1 ==== 40329+0+0 (secure 0 0 0) 0x7ff9e001a070 con 0x55dfdac8d910 2023-10-05T19:28:24.086 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.082+0000 7ff9ea92de40 20 rados->read ofs=0 len=0 2023-10-05T19:28:24.086 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.082+0000 7ff9e4c5b700 1 --2- 172.21.15.167:0/3611872326 >> [v2:172.21.15.167:6824/29889,v1:172.21.15.167:6825/29889] conn(0x7ff9d4030c00 0x7ff9d40330b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-05T19:28:24.086 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.082+0000 7ff9e4c5b700 1 -- 172.21.15.167:0/3611872326 <== mon.0 v2:172.21.15.118:3300/0 5 ==== osd_map(30..30 src has 1..30) v4 ==== 6057+0+0 (secure 0 0 0) 0x7ff9e0044da0 con 0x55dfdac8d910 2023-10-05T19:28:24.087 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.082+0000 7ff9ea92de40 1 --2- 172.21.15.167:0/3611872326 >> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] conn(0x55dfdac8ec20 0x55dfdad4c7e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-05T19:28:24.087 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.082+0000 7ff9e7c61700 1 --2- 172.21.15.167:0/3611872326 >> [v2:172.21.15.167:6824/29889,v1:172.21.15.167:6825/29889] conn(0x7ff9d4030c00 0x7ff9d40330b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-05T19:28:24.087 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.082+0000 7ff9ea92de40 1 -- 172.21.15.167:0/3611872326 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:1 5.12 5:49953fa1:::default.realm:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x55dfdad4cee0 con 0x55dfdac8ec20 2023-10-05T19:28:24.087 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.082+0000 7ff9e8c63700 1 --2- 172.21.15.167:0/3611872326 >> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] conn(0x55dfdac8ec20 0x55dfdad4c7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-05T19:28:24.088 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.082+0000 7ff9e7c61700 1 --2- 172.21.15.167:0/3611872326 >> [v2:172.21.15.167:6824/29889,v1:172.21.15.167:6825/29889] conn(0x7ff9d4030c00 0x7ff9d40330b0 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7ff9dc00afd0 tx=0x7ff9dc014000 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-05T19:28:24.088 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.082+0000 7ff9e8c63700 1 --2- 172.21.15.167:0/3611872326 >> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] conn(0x55dfdac8ec20 0x55dfdad4c7e0 crc :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-05T19:28:24.089 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.082+0000 7ff9e8c63700 1 -- 172.21.15.167:0/3611872326 <== osd.1 v2:172.21.15.167:6800/30035 1 ==== osd_op_reply(1 default.realm [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 ==== 157+0+0 (crc 0 0 0) 0x7ff9d800a500 con 0x55dfdac8ec20 2023-10-05T19:28:24.089 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.082+0000 7ff9ea92de40 20 rados_obj.operate() r=-2 bl.length=0 2023-10-05T19:28:24.089 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.082+0000 7ff9ea92de40 20 realm 2023-10-05T19:28:24.089 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.082+0000 7ff9ea92de40 20 rados->read ofs=0 len=0 2023-10-05T19:28:24.089 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.082+0000 7ff9ea92de40 1 -- 172.21.15.167:0/3611872326 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:2 5.12 5:49953fa1:::default.realm:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x55dfdad49e40 con 0x55dfdac8ec20 2023-10-05T19:28:24.089 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.082+0000 7ff9e8c63700 1 -- 172.21.15.167:0/3611872326 <== osd.1 v2:172.21.15.167:6800/30035 2 ==== osd_op_reply(2 default.realm [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 ==== 157+0+0 (crc 0 0 0) 0x7ff9d800a500 con 0x55dfdac8ec20 2023-10-05T19:28:24.089 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.082+0000 7ff9ea92de40 20 rados_obj.operate() r=-2 bl.length=0 2023-10-05T19:28:24.089 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.082+0000 7ff9ea92de40 4 RGWPeriod::init failed to init realm id : (2) No such file or directory 2023-10-05T19:28:24.089 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.082+0000 7ff9ea92de40 20 rados->read ofs=0 len=0 2023-10-05T19:28:24.089 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.082+0000 7ff9ea92de40 1 -- 172.21.15.167:0/3611872326 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:3 5.12 5:49953fa1:::default.realm:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x55dfdad4d9c0 con 0x55dfdac8ec20 2023-10-05T19:28:24.091 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.082+0000 7ff9e8c63700 1 -- 172.21.15.167:0/3611872326 <== osd.1 v2:172.21.15.167:6800/30035 3 ==== osd_op_reply(3 default.realm [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 ==== 157+0+0 (crc 0 0 0) 0x7ff9d800a500 con 0x55dfdac8ec20 2023-10-05T19:28:24.091 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.082+0000 7ff9ea92de40 20 rados_obj.operate() r=-2 bl.length=0 2023-10-05T19:28:24.091 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.082+0000 7ff9ea92de40 20 rados->read ofs=0 len=0 2023-10-05T19:28:24.091 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.082+0000 7ff9ea92de40 1 -- 172.21.15.167:0/3611872326 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:4 5.f 5:f43b6ece:::zone_names.default:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x55dfdad4d5d0 con 0x55dfdac8ec20 2023-10-05T19:28:24.091 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.082+0000 7ff9e8c63700 1 -- 172.21.15.167:0/3611872326 <== osd.1 v2:172.21.15.167:6800/30035 4 ==== osd_op_reply(4 zone_names.default [read 0~46 out=46b] v0'0 uv1 ondisk = 0) v8 ==== 162+0+46 (crc 0 0 0) 0x7ff9d800a500 con 0x55dfdac8ec20 2023-10-05T19:28:24.091 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.082+0000 7ff9ea92de40 20 rados_obj.operate() r=0 bl.length=46 2023-10-05T19:28:24.091 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.082+0000 7ff9ea92de40 20 rados->read ofs=0 len=0 2023-10-05T19:28:24.092 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.082+0000 7ff9ea92de40 1 -- 172.21.15.167:0/3611872326 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:5 5.1d 5:b8374eb4:::zone_info.9a9df965-6612-45d4-a45f-6e3c889bacc2:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x55dfdad4e850 con 0x55dfdac8ec20 2023-10-05T19:28:24.092 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.086+0000 7ff9e8c63700 1 -- 172.21.15.167:0/3611872326 <== osd.1 v2:172.21.15.167:6800/30035 5 ==== osd_op_reply(5 zone_info.9a9df965-6612-45d4-a45f-6e3c889bacc2 [read 0~874 out=874b] v0'0 uv1 ondisk = 0) v8 ==== 190+0+874 (crc 0 0 0) 0x7ff9d800a500 con 0x55dfdac8ec20 2023-10-05T19:28:24.092 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.086+0000 7ff9ea92de40 20 rados_obj.operate() r=0 bl.length=874 2023-10-05T19:28:24.092 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.086+0000 7ff9ea92de40 20 searching for the correct realm 2023-10-05T19:28:24.092 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.086+0000 7ff9ea92de40 1 --2- 172.21.15.167:0/3611872326 >> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] conn(0x55dfdad50610 0x55dfdad529f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-05T19:28:24.092 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.086+0000 7ff9ea92de40 1 -- 172.21.15.167:0/3611872326 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:6 5.0 5:00000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x55dfdad52f30 con 0x55dfdad50610 2023-10-05T19:28:24.092 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.086+0000 7ff9e8462700 1 --2- 172.21.15.167:0/3611872326 >> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] conn(0x55dfdad50610 0x55dfdad529f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-05T19:28:24.092 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.086+0000 7ff9e8462700 1 --2- 172.21.15.167:0/3611872326 >> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] conn(0x55dfdad50610 0x55dfdad529f0 crc :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-05T19:28:24.093 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.086+0000 7ff9e8462700 1 -- 172.21.15.167:0/3611872326 <== osd.3 v2:172.21.15.167:6808/30036 1 ==== osd_op_reply(6 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7ff9e0025020 con 0x55dfdad50610 2023-10-05T19:28:24.093 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.086+0000 7ff9e8462700 1 --2- 172.21.15.167:0/3611872326 >> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] conn(0x7ff9e001fc20 0x7ff9e00080e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-05T19:28:24.093 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.086+0000 7ff9e8462700 1 -- 172.21.15.167:0/3611872326 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:7 5.10 5:08000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x55dfdad52f30 con 0x7ff9e001fc20 2023-10-05T19:28:24.093 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.086+0000 7ff9e7c61700 1 --2- 172.21.15.167:0/3611872326 >> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] conn(0x7ff9e001fc20 0x7ff9e00080e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-05T19:28:24.094 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.090+0000 7ff9e7c61700 1 --2- 172.21.15.167:0/3611872326 >> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] conn(0x7ff9e001fc20 0x7ff9e00080e0 crc :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-05T19:28:24.094 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.090+0000 7ff9e7c61700 1 -- 172.21.15.167:0/3611872326 <== osd.2 v2:172.21.15.167:6816/30037 1 ==== osd_op_reply(7 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7ff9dc007a50 con 0x7ff9e001fc20 2023-10-05T19:28:24.095 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.090+0000 7ff9e7c61700 1 --2- 172.21.15.167:0/3611872326 >> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] conn(0x7ff9dc007030 0x7ff9dc01d430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-05T19:28:24.095 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.090+0000 7ff9e7c61700 1 -- 172.21.15.167:0/3611872326 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:8 5.8 5:10000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x55dfdad52f30 con 0x7ff9dc007030 2023-10-05T19:28:24.095 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.090+0000 7ff9e8c63700 1 --2- 172.21.15.167:0/3611872326 >> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] conn(0x7ff9dc007030 0x7ff9dc01d430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-05T19:28:24.095 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.090+0000 7ff9e8c63700 1 --2- 172.21.15.167:0/3611872326 >> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] conn(0x7ff9dc007030 0x7ff9dc01d430 crc :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-05T19:28:24.096 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.090+0000 7ff9e8c63700 1 -- 172.21.15.167:0/3611872326 <== osd.0 v2:172.21.15.118:6800/27690 1 ==== osd_op_reply(8 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7ff9d8014040 con 0x7ff9dc007030 2023-10-05T19:28:24.097 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.090+0000 7ff9e8c63700 1 -- 172.21.15.167:0/3611872326 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:9 5.18 5:18000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x55dfdad52f30 con 0x55dfdac8ec20 2023-10-05T19:28:24.097 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.090+0000 7ff9e8c63700 1 -- 172.21.15.167:0/3611872326 <== osd.1 v2:172.21.15.167:6800/30035 6 ==== osd_op_reply(9 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7ff9d8014040 con 0x55dfdac8ec20 2023-10-05T19:28:24.097 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.090+0000 7ff9e8c63700 1 -- 172.21.15.167:0/3611872326 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:10 5.4 5:20000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x55dfdad52f30 con 0x55dfdac8ec20 2023-10-05T19:28:24.098 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.094+0000 7ff9e8c63700 1 -- 172.21.15.167:0/3611872326 <== osd.1 v2:172.21.15.167:6800/30035 7 ==== osd_op_reply(10 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7ff9d8014040 con 0x55dfdac8ec20 2023-10-05T19:28:24.098 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.094+0000 7ff9e8c63700 1 -- 172.21.15.167:0/3611872326 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:11 5.14 5:28000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x55dfdad52f30 con 0x7ff9dc007030 2023-10-05T19:28:24.099 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.094+0000 7ff9e8c63700 1 -- 172.21.15.167:0/3611872326 <== osd.0 v2:172.21.15.118:6800/27690 2 ==== osd_op_reply(11 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7ff9d8014040 con 0x7ff9dc007030 2023-10-05T19:28:24.099 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.094+0000 7ff9e8c63700 1 -- 172.21.15.167:0/3611872326 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:12 5.c 5:30000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x55dfdad52f30 con 0x7ff9dc007030 2023-10-05T19:28:24.100 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.094+0000 7ff9e8c63700 1 -- 172.21.15.167:0/3611872326 <== osd.0 v2:172.21.15.118:6800/27690 3 ==== osd_op_reply(12 [pgnls start_epoch 30 out=112b] v23'1 uv1 ondisk = 1) v8 ==== 144+0+112 (crc 0 0 0) 0x7ff9d8014040 con 0x7ff9dc007030 2023-10-05T19:28:24.100 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.094+0000 7ff9e8c63700 1 -- 172.21.15.167:0/3611872326 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:13 5.1c 5:38000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x55dfdad52f30 con 0x7ff9e001fc20 2023-10-05T19:28:24.101 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.094+0000 7ff9e7c61700 1 -- 172.21.15.167:0/3611872326 <== osd.2 v2:172.21.15.167:6816/30037 2 ==== osd_op_reply(13 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7ff9dc029040 con 0x7ff9e001fc20 2023-10-05T19:28:24.101 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.094+0000 7ff9e7c61700 1 -- 172.21.15.167:0/3611872326 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:14 5.2 5:40000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x55dfdad52f30 con 0x55dfdad50610 2023-10-05T19:28:24.101 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.094+0000 7ff9e8462700 1 -- 172.21.15.167:0/3611872326 <== osd.3 v2:172.21.15.167:6808/30036 2 ==== osd_op_reply(14 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7ff9e0025020 con 0x55dfdad50610 2023-10-05T19:28:24.101 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.094+0000 7ff9e8462700 1 -- 172.21.15.167:0/3611872326 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:15 5.12 5:48000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x55dfdad52f30 con 0x55dfdac8ec20 2023-10-05T19:28:24.102 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.098+0000 7ff9e8c63700 1 -- 172.21.15.167:0/3611872326 <== osd.1 v2:172.21.15.167:6800/30035 8 ==== osd_op_reply(15 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7ff9d8014040 con 0x55dfdac8ec20 2023-10-05T19:28:24.102 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.098+0000 7ff9e8c63700 1 -- 172.21.15.167:0/3611872326 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:16 5.a 5:50000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x55dfdad52f30 con 0x7ff9dc007030 2023-10-05T19:28:24.103 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.098+0000 7ff9e8c63700 1 -- 172.21.15.167:0/3611872326 <== osd.0 v2:172.21.15.118:6800/27690 4 ==== osd_op_reply(16 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7ff9d8022040 con 0x7ff9dc007030 2023-10-05T19:28:24.103 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.098+0000 7ff9e8c63700 1 -- 172.21.15.167:0/3611872326 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:17 5.1a 5:58000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x55dfdad52f30 con 0x55dfdac8ec20 2023-10-05T19:28:24.104 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.098+0000 7ff9e8c63700 1 -- 172.21.15.167:0/3611872326 <== osd.1 v2:172.21.15.167:6800/30035 9 ==== osd_op_reply(17 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7ff9d8022040 con 0x55dfdac8ec20 2023-10-05T19:28:24.104 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.098+0000 7ff9e8c63700 1 -- 172.21.15.167:0/3611872326 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:18 5.6 5:60000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x55dfdad52f30 con 0x7ff9e001fc20 2023-10-05T19:28:24.105 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.098+0000 7ff9e7c61700 1 -- 172.21.15.167:0/3611872326 <== osd.2 v2:172.21.15.167:6816/30037 3 ==== osd_op_reply(18 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7ff9dc029040 con 0x7ff9e001fc20 2023-10-05T19:28:24.105 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.098+0000 7ff9e7c61700 1 -- 172.21.15.167:0/3611872326 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:19 5.16 5:68000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x55dfdad52f30 con 0x55dfdad50610 2023-10-05T19:28:24.106 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.098+0000 7ff9e8462700 1 -- 172.21.15.167:0/3611872326 <== osd.3 v2:172.21.15.167:6808/30036 3 ==== osd_op_reply(19 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7ff9e0025020 con 0x55dfdad50610 2023-10-05T19:28:24.106 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.098+0000 7ff9e8462700 1 -- 172.21.15.167:0/3611872326 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:20 5.e 5:70000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x55dfdad52f30 con 0x55dfdad50610 2023-10-05T19:28:24.106 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.102+0000 7ff9e8462700 1 -- 172.21.15.167:0/3611872326 <== osd.3 v2:172.21.15.167:6808/30036 4 ==== osd_op_reply(20 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7ff9e0025020 con 0x55dfdad50610 2023-10-05T19:28:24.106 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.102+0000 7ff9e8462700 1 -- 172.21.15.167:0/3611872326 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:21 5.1e 5:78000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x55dfdad52f30 con 0x7ff9dc007030 2023-10-05T19:28:24.107 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.102+0000 7ff9e8c63700 1 -- 172.21.15.167:0/3611872326 <== osd.0 v2:172.21.15.118:6800/27690 5 ==== osd_op_reply(21 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7ff9d8022040 con 0x7ff9dc007030 2023-10-05T19:28:24.107 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.102+0000 7ff9e8c63700 1 -- 172.21.15.167:0/3611872326 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:22 5.1 5:80000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x55dfdad52f30 con 0x55dfdad50610 2023-10-05T19:28:24.108 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.102+0000 7ff9e8462700 1 -- 172.21.15.167:0/3611872326 <== osd.3 v2:172.21.15.167:6808/30036 5 ==== osd_op_reply(22 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7ff9e0025020 con 0x55dfdad50610 2023-10-05T19:28:24.108 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.102+0000 7ff9e8462700 1 -- 172.21.15.167:0/3611872326 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:23 5.11 5:88000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x55dfdad52f30 con 0x55dfdad50610 2023-10-05T19:28:24.109 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.102+0000 7ff9e8462700 1 -- 172.21.15.167:0/3611872326 <== osd.3 v2:172.21.15.167:6808/30036 6 ==== osd_op_reply(23 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7ff9e0025020 con 0x55dfdad50610 2023-10-05T19:28:24.109 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.102+0000 7ff9e8462700 1 -- 172.21.15.167:0/3611872326 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:24 5.9 5:90000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x55dfdad52f30 con 0x55dfdac8ec20 2023-10-05T19:28:24.109 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.102+0000 7ff9e8c63700 1 -- 172.21.15.167:0/3611872326 <== osd.1 v2:172.21.15.167:6800/30035 10 ==== osd_op_reply(24 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7ff9d8022040 con 0x55dfdac8ec20 2023-10-05T19:28:24.110 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.102+0000 7ff9e8c63700 1 -- 172.21.15.167:0/3611872326 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:25 5.19 5:98000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x55dfdad52f30 con 0x55dfdac8ec20 2023-10-05T19:28:24.110 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.106+0000 7ff9e8c63700 1 -- 172.21.15.167:0/3611872326 <== osd.1 v2:172.21.15.167:6800/30035 11 ==== osd_op_reply(25 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7ff9d8022040 con 0x55dfdac8ec20 2023-10-05T19:28:24.110 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.106+0000 7ff9e8c63700 1 -- 172.21.15.167:0/3611872326 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:26 5.5 5:a0000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x55dfdad52f30 con 0x7ff9dc007030 2023-10-05T19:28:24.111 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.106+0000 7ff9e8c63700 1 -- 172.21.15.167:0/3611872326 <== osd.0 v2:172.21.15.118:6800/27690 6 ==== osd_op_reply(26 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7ff9d8022040 con 0x7ff9dc007030 2023-10-05T19:28:24.112 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.106+0000 7ff9e8c63700 1 -- 172.21.15.167:0/3611872326 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:27 5.15 5:a8000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x55dfdad52f30 con 0x55dfdad50610 2023-10-05T19:28:24.112 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.106+0000 7ff9e8462700 1 -- 172.21.15.167:0/3611872326 <== osd.3 v2:172.21.15.167:6808/30036 7 ==== osd_op_reply(27 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7ff9e0048040 con 0x55dfdad50610 2023-10-05T19:28:24.112 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.106+0000 7ff9e8462700 1 -- 172.21.15.167:0/3611872326 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:28 5.d 5:b0000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x55dfdad52f30 con 0x7ff9e001fc20 2023-10-05T19:28:24.113 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.106+0000 7ff9e7c61700 1 -- 172.21.15.167:0/3611872326 <== osd.2 v2:172.21.15.167:6816/30037 4 ==== osd_op_reply(28 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7ff9dc029040 con 0x7ff9e001fc20 2023-10-05T19:28:24.113 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.106+0000 7ff9e7c61700 1 -- 172.21.15.167:0/3611872326 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:29 5.1d 5:b8000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x55dfdad52f30 con 0x55dfdac8ec20 2023-10-05T19:28:24.114 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.110+0000 7ff9e8c63700 1 -- 172.21.15.167:0/3611872326 <== osd.1 v2:172.21.15.167:6800/30035 12 ==== osd_op_reply(29 [pgnls start_epoch 30 out=107b] v23'1 uv1 ondisk = 1) v8 ==== 144+0+107 (crc 0 0 0) 0x7ff9d8022040 con 0x55dfdac8ec20 2023-10-05T19:28:24.114 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.110+0000 7ff9e8c63700 1 -- 172.21.15.167:0/3611872326 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:30 5.3 5:c0000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x55dfdad52f30 con 0x7ff9dc007030 2023-10-05T19:28:24.115 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.110+0000 7ff9e8c63700 1 -- 172.21.15.167:0/3611872326 <== osd.0 v2:172.21.15.118:6800/27690 7 ==== osd_op_reply(30 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7ff9d8022040 con 0x7ff9dc007030 2023-10-05T19:28:24.115 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.110+0000 7ff9e8c63700 1 -- 172.21.15.167:0/3611872326 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:31 5.13 5:c8000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x55dfdad52f30 con 0x55dfdad50610 2023-10-05T19:28:24.116 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.110+0000 7ff9e8462700 1 -- 172.21.15.167:0/3611872326 <== osd.3 v2:172.21.15.167:6808/30036 8 ==== osd_op_reply(31 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7ff9e0048040 con 0x55dfdad50610 2023-10-05T19:28:24.116 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.110+0000 7ff9e8462700 1 -- 172.21.15.167:0/3611872326 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:32 5.b 5:d0000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x55dfdad52f30 con 0x7ff9dc007030 2023-10-05T19:28:24.117 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.110+0000 7ff9e8c63700 1 -- 172.21.15.167:0/3611872326 <== osd.0 v2:172.21.15.118:6800/27690 8 ==== osd_op_reply(32 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7ff9d8022040 con 0x7ff9dc007030 2023-10-05T19:28:24.117 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.110+0000 7ff9e8c63700 1 -- 172.21.15.167:0/3611872326 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:33 5.1b 5:d8000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x55dfdad52f30 con 0x7ff9e001fc20 2023-10-05T19:28:24.118 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.114+0000 7ff9e7c61700 1 -- 172.21.15.167:0/3611872326 <== osd.2 v2:172.21.15.167:6816/30037 5 ==== osd_op_reply(33 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7ff9dc029040 con 0x7ff9e001fc20 2023-10-05T19:28:24.118 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.114+0000 7ff9e7c61700 1 -- 172.21.15.167:0/3611872326 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:34 5.7 5:e0000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x55dfdad52f30 con 0x55dfdad50610 2023-10-05T19:28:24.119 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.114+0000 7ff9e8462700 1 -- 172.21.15.167:0/3611872326 <== osd.3 v2:172.21.15.167:6808/30036 9 ==== osd_op_reply(34 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7ff9e0048040 con 0x55dfdad50610 2023-10-05T19:28:24.119 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.114+0000 7ff9e8462700 1 -- 172.21.15.167:0/3611872326 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:35 5.17 5:e8000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x55dfdad52f30 con 0x7ff9dc007030 2023-10-05T19:28:24.120 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.114+0000 7ff9e8c63700 1 -- 172.21.15.167:0/3611872326 <== osd.0 v2:172.21.15.118:6800/27690 9 ==== osd_op_reply(35 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7ff9d8022040 con 0x7ff9dc007030 2023-10-05T19:28:24.120 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.114+0000 7ff9e8c63700 1 -- 172.21.15.167:0/3611872326 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:36 5.f 5:f0000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x55dfdad52f30 con 0x55dfdac8ec20 2023-10-05T19:28:24.121 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.114+0000 7ff9e8c63700 1 -- 172.21.15.167:0/3611872326 <== osd.1 v2:172.21.15.167:6800/30035 13 ==== osd_op_reply(36 [pgnls start_epoch 30 out=115b] v23'2 uv2 ondisk = 1) v8 ==== 144+0+115 (crc 0 0 0) 0x7ff9d802f080 con 0x55dfdac8ec20 2023-10-05T19:28:24.121 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.114+0000 7ff9e8c63700 1 -- 172.21.15.167:0/3611872326 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:37 5.1f 5:f8000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x55dfdad52f30 con 0x7ff9e001fc20 2023-10-05T19:28:24.122 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.114+0000 7ff9e7c61700 1 -- 172.21.15.167:0/3611872326 <== osd.2 v2:172.21.15.167:6816/30037 6 ==== osd_op_reply(37 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7ff9dc029040 con 0x7ff9e001fc20 2023-10-05T19:28:24.122 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.118+0000 7ff9ea92de40 20 RGWRados::pool_iterate: got zonegroup_info.4f1f719d-9648-417b-b10c-e5d939825419 2023-10-05T19:28:24.122 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.118+0000 7ff9ea92de40 20 RGWRados::pool_iterate: got zone_info.9a9df965-6612-45d4-a45f-6e3c889bacc2 2023-10-05T19:28:24.122 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.118+0000 7ff9ea92de40 20 RGWRados::pool_iterate: got zone_names.default 2023-10-05T19:28:24.122 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.118+0000 7ff9ea92de40 20 RGWRados::pool_iterate: got zonegroups_names.default 2023-10-05T19:28:24.123 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.118+0000 7ff9ea92de40 20 rados->read ofs=0 len=0 2023-10-05T19:28:24.123 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.118+0000 7ff9ea92de40 1 -- 172.21.15.167:0/3611872326 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:38 5.12 5:49953fa1:::default.realm:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x55dfdad4f360 con 0x55dfdac8ec20 2023-10-05T19:28:24.124 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.118+0000 7ff9e8c63700 1 -- 172.21.15.167:0/3611872326 <== osd.1 v2:172.21.15.167:6800/30035 14 ==== osd_op_reply(38 default.realm [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 ==== 157+0+0 (crc 0 0 0) 0x7ff9d802f080 con 0x55dfdac8ec20 2023-10-05T19:28:24.124 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.118+0000 7ff9ea92de40 20 rados_obj.operate() r=-2 bl.length=0 2023-10-05T19:28:24.124 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.118+0000 7ff9ea92de40 20 rados->read ofs=0 len=0 2023-10-05T19:28:24.124 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.118+0000 7ff9ea92de40 1 -- 172.21.15.167:0/3611872326 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:39 5.f 5:f4c53578:::zonegroups_names.default:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x55dfdad4fc50 con 0x55dfdac8ec20 2023-10-05T19:28:24.124 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.118+0000 7ff9e8c63700 1 -- 172.21.15.167:0/3611872326 <== osd.1 v2:172.21.15.167:6800/30035 15 ==== osd_op_reply(39 zonegroups_names.default [read 0~46 out=46b] v0'0 uv2 ondisk = 0) v8 ==== 168+0+46 (crc 0 0 0) 0x7ff9d802f080 con 0x55dfdac8ec20 2023-10-05T19:28:24.124 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.118+0000 7ff9ea92de40 20 rados_obj.operate() r=0 bl.length=46 2023-10-05T19:28:24.124 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.118+0000 7ff9ea92de40 20 rados->read ofs=0 len=0 2023-10-05T19:28:24.124 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.118+0000 7ff9ea92de40 1 -- 172.21.15.167:0/3611872326 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:40 5.c 5:30d299a5:::zonegroup_info.4f1f719d-9648-417b-b10c-e5d939825419:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x55dfdad53370 con 0x7ff9dc007030 2023-10-05T19:28:24.125 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.118+0000 7ff9e8c63700 1 -- 172.21.15.167:0/3611872326 <== osd.0 v2:172.21.15.118:6800/27690 10 ==== osd_op_reply(40 zonegroup_info.4f1f719d-9648-417b-b10c-e5d939825419 [read 0~362 out=362b] v0'0 uv1 ondisk = 0) v8 ==== 195+0+362 (crc 0 0 0) 0x7ff9d802f080 con 0x7ff9dc007030 2023-10-05T19:28:24.126 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.118+0000 7ff9ea92de40 20 rados_obj.operate() r=0 bl.length=362 2023-10-05T19:28:24.126 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.118+0000 7ff9ea92de40 20 rados->read ofs=0 len=0 2023-10-05T19:28:24.126 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.118+0000 7ff9ea92de40 1 -- 172.21.15.167:0/3611872326 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:41 5.13 5:cbbf6c60:::converted:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x55dfdad53370 con 0x55dfdad50610 2023-10-05T19:28:24.126 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.118+0000 7ff9e8462700 1 -- 172.21.15.167:0/3611872326 <== osd.3 v2:172.21.15.167:6808/30036 10 ==== osd_op_reply(41 converted [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 ==== 153+0+0 (crc 0 0 0) 0x7ff9e0048040 con 0x55dfdad50610 2023-10-05T19:28:24.126 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.118+0000 7ff9ea92de40 20 rados_obj.operate() r=-2 bl.length=0 2023-10-05T19:28:24.126 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.118+0000 7ff9ea92de40 20 rados->read ofs=0 len=0 2023-10-05T19:28:24.126 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.118+0000 7ff9ea92de40 1 -- 172.21.15.167:0/3611872326 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:42 5.12 5:49953fa1:::default.realm:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x55dfdad53720 con 0x55dfdac8ec20 2023-10-05T19:28:24.127 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.122+0000 7ff9e8c63700 1 -- 172.21.15.167:0/3611872326 <== osd.1 v2:172.21.15.167:6800/30035 16 ==== osd_op_reply(42 default.realm [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 ==== 157+0+0 (crc 0 0 0) 0x7ff9d802f080 con 0x55dfdac8ec20 2023-10-05T19:28:24.127 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.122+0000 7ff9ea92de40 20 rados_obj.operate() r=-2 bl.length=0 2023-10-05T19:28:24.127 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.122+0000 7ff9ea92de40 20 rados->read ofs=0 len=0 2023-10-05T19:28:24.128 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.122+0000 7ff9ea92de40 1 -- 172.21.15.167:0/3611872326 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:43 5.f 5:f4c53578:::zonegroups_names.default:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x55dfdad4f880 con 0x55dfdac8ec20 2023-10-05T19:28:24.128 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.122+0000 7ff9e8c63700 1 -- 172.21.15.167:0/3611872326 <== osd.1 v2:172.21.15.167:6800/30035 17 ==== osd_op_reply(43 zonegroups_names.default [read 0~46 out=46b] v0'0 uv2 ondisk = 0) v8 ==== 168+0+46 (crc 0 0 0) 0x7ff9d802f080 con 0x55dfdac8ec20 2023-10-05T19:28:24.128 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.122+0000 7ff9ea92de40 20 rados_obj.operate() r=0 bl.length=46 2023-10-05T19:28:24.128 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.122+0000 7ff9ea92de40 1 -- 172.21.15.167:0/3611872326 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:44 5.0 5:00000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x55dfdad4f850 con 0x55dfdad50610 2023-10-05T19:28:24.129 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.122+0000 7ff9e8462700 1 -- 172.21.15.167:0/3611872326 <== osd.3 v2:172.21.15.167:6808/30036 11 ==== osd_op_reply(44 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7ff9e0048040 con 0x55dfdad50610 2023-10-05T19:28:24.130 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.122+0000 7ff9e8462700 1 -- 172.21.15.167:0/3611872326 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:45 5.10 5:08000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x55dfdad4f850 con 0x7ff9e001fc20 2023-10-05T19:28:24.130 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.126+0000 7ff9e7c61700 1 -- 172.21.15.167:0/3611872326 <== osd.2 v2:172.21.15.167:6816/30037 7 ==== osd_op_reply(45 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7ff9dc029040 con 0x7ff9e001fc20 2023-10-05T19:28:24.130 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.126+0000 7ff9e7c61700 1 -- 172.21.15.167:0/3611872326 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:46 5.8 5:10000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x55dfdad4f850 con 0x7ff9dc007030 2023-10-05T19:28:24.131 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.126+0000 7ff9e8c63700 1 -- 172.21.15.167:0/3611872326 <== osd.0 v2:172.21.15.118:6800/27690 11 ==== osd_op_reply(46 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7ff9d802f080 con 0x7ff9dc007030 2023-10-05T19:28:24.132 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.126+0000 7ff9e8c63700 1 -- 172.21.15.167:0/3611872326 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:47 5.18 5:18000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x55dfdad4fc50 con 0x55dfdac8ec20 2023-10-05T19:28:24.132 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.126+0000 7ff9e8c63700 1 -- 172.21.15.167:0/3611872326 <== osd.1 v2:172.21.15.167:6800/30035 18 ==== osd_op_reply(47 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7ff9d802f080 con 0x55dfdac8ec20 2023-10-05T19:28:24.132 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.126+0000 7ff9e8c63700 1 -- 172.21.15.167:0/3611872326 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:48 5.4 5:20000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x55dfdad4fc50 con 0x55dfdac8ec20 2023-10-05T19:28:24.133 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.126+0000 7ff9e8c63700 1 -- 172.21.15.167:0/3611872326 <== osd.1 v2:172.21.15.167:6800/30035 19 ==== osd_op_reply(48 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7ff9d802f080 con 0x55dfdac8ec20 2023-10-05T19:28:24.133 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.126+0000 7ff9e8c63700 1 -- 172.21.15.167:0/3611872326 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:49 5.14 5:28000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x55dfdad4fc50 con 0x7ff9dc007030 2023-10-05T19:28:24.134 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.130+0000 7ff9e8c63700 1 -- 172.21.15.167:0/3611872326 <== osd.0 v2:172.21.15.118:6800/27690 12 ==== osd_op_reply(49 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7ff9d802f080 con 0x7ff9dc007030 2023-10-05T19:28:24.134 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.130+0000 7ff9e8c63700 1 -- 172.21.15.167:0/3611872326 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:50 5.c 5:30000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x55dfdad4fc50 con 0x7ff9dc007030 2023-10-05T19:28:24.135 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.130+0000 7ff9e8c63700 1 -- 172.21.15.167:0/3611872326 <== osd.0 v2:172.21.15.118:6800/27690 13 ==== osd_op_reply(50 [pgnls start_epoch 30 out=112b] v23'1 uv1 ondisk = 1) v8 ==== 144+0+112 (crc 0 0 0) 0x7ff9d802f080 con 0x7ff9dc007030 2023-10-05T19:28:24.135 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.130+0000 7ff9e8c63700 1 -- 172.21.15.167:0/3611872326 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:51 5.1c 5:38000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x55dfdad4fc50 con 0x7ff9e001fc20 2023-10-05T19:28:24.136 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.130+0000 7ff9e7c61700 1 -- 172.21.15.167:0/3611872326 <== osd.2 v2:172.21.15.167:6816/30037 8 ==== osd_op_reply(51 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7ff9dc029040 con 0x7ff9e001fc20 2023-10-05T19:28:24.136 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.130+0000 7ff9e7c61700 1 -- 172.21.15.167:0/3611872326 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:52 5.2 5:40000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x55dfdad4fc50 con 0x55dfdad50610 2023-10-05T19:28:24.137 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.130+0000 7ff9e8462700 1 -- 172.21.15.167:0/3611872326 <== osd.3 v2:172.21.15.167:6808/30036 12 ==== osd_op_reply(52 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7ff9e0048040 con 0x55dfdad50610 2023-10-05T19:28:24.137 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.130+0000 7ff9e8462700 1 -- 172.21.15.167:0/3611872326 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:53 5.12 5:48000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x55dfdad4fc50 con 0x55dfdac8ec20 2023-10-05T19:28:24.138 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.134+0000 7ff9e8c63700 1 -- 172.21.15.167:0/3611872326 <== osd.1 v2:172.21.15.167:6800/30035 20 ==== osd_op_reply(53 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7ff9d802f080 con 0x55dfdac8ec20 2023-10-05T19:28:24.138 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.134+0000 7ff9e8c63700 1 -- 172.21.15.167:0/3611872326 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:54 5.a 5:50000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x55dfdad4fc50 con 0x7ff9dc007030 2023-10-05T19:28:24.139 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.134+0000 7ff9e8c63700 1 -- 172.21.15.167:0/3611872326 <== osd.0 v2:172.21.15.118:6800/27690 14 ==== osd_op_reply(54 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7ff9d802f080 con 0x7ff9dc007030 2023-10-05T19:28:24.139 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.134+0000 7ff9e8c63700 1 -- 172.21.15.167:0/3611872326 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:55 5.1a 5:58000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x55dfdad4fc50 con 0x55dfdac8ec20 2023-10-05T19:28:24.140 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.134+0000 7ff9e8c63700 1 -- 172.21.15.167:0/3611872326 <== osd.1 v2:172.21.15.167:6800/30035 21 ==== osd_op_reply(55 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7ff9d802f080 con 0x55dfdac8ec20 2023-10-05T19:28:24.140 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.134+0000 7ff9e8c63700 1 -- 172.21.15.167:0/3611872326 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:56 5.6 5:60000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x55dfdad4fc50 con 0x7ff9e001fc20 2023-10-05T19:28:24.141 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.134+0000 7ff9e7c61700 1 -- 172.21.15.167:0/3611872326 <== osd.2 v2:172.21.15.167:6816/30037 9 ==== osd_op_reply(56 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7ff9dc029040 con 0x7ff9e001fc20 2023-10-05T19:28:24.141 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.134+0000 7ff9e7c61700 1 -- 172.21.15.167:0/3611872326 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:57 5.16 5:68000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x55dfdad4fc50 con 0x55dfdad50610 2023-10-05T19:28:24.142 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.134+0000 7ff9e8462700 1 -- 172.21.15.167:0/3611872326 <== osd.3 v2:172.21.15.167:6808/30036 13 ==== osd_op_reply(57 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7ff9e0048040 con 0x55dfdad50610 2023-10-05T19:28:24.142 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.134+0000 7ff9e8462700 1 -- 172.21.15.167:0/3611872326 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:58 5.e 5:70000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x55dfdad4fc50 con 0x55dfdad50610 2023-10-05T19:28:24.142 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.138+0000 7ff9e8462700 1 -- 172.21.15.167:0/3611872326 <== osd.3 v2:172.21.15.167:6808/30036 14 ==== osd_op_reply(58 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7ff9e0048040 con 0x55dfdad50610 2023-10-05T19:28:24.142 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.138+0000 7ff9e8462700 1 -- 172.21.15.167:0/3611872326 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:59 5.1e 5:78000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x55dfdad4fc50 con 0x7ff9dc007030 2023-10-05T19:28:24.143 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.138+0000 7ff9e8c63700 1 -- 172.21.15.167:0/3611872326 <== osd.0 v2:172.21.15.118:6800/27690 15 ==== osd_op_reply(59 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7ff9d802f080 con 0x7ff9dc007030 2023-10-05T19:28:24.143 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.138+0000 7ff9e8c63700 1 -- 172.21.15.167:0/3611872326 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:60 5.1 5:80000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x55dfdad4fc50 con 0x55dfdad50610 2023-10-05T19:28:24.144 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.138+0000 7ff9e8462700 1 -- 172.21.15.167:0/3611872326 <== osd.3 v2:172.21.15.167:6808/30036 15 ==== osd_op_reply(60 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7ff9e0048040 con 0x55dfdad50610 2023-10-05T19:28:24.144 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.138+0000 7ff9e8462700 1 -- 172.21.15.167:0/3611872326 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:61 5.11 5:88000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x55dfdad4fc50 con 0x55dfdad50610 2023-10-05T19:28:24.145 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.138+0000 7ff9e8462700 1 -- 172.21.15.167:0/3611872326 <== osd.3 v2:172.21.15.167:6808/30036 16 ==== osd_op_reply(61 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7ff9e0048040 con 0x55dfdad50610 2023-10-05T19:28:24.145 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.138+0000 7ff9e8462700 1 -- 172.21.15.167:0/3611872326 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:62 5.9 5:90000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x55dfdad4fc50 con 0x55dfdac8ec20 2023-10-05T19:28:24.145 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.138+0000 7ff9e8c63700 1 -- 172.21.15.167:0/3611872326 <== osd.1 v2:172.21.15.167:6800/30035 22 ==== osd_op_reply(62 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7ff9d802f080 con 0x55dfdac8ec20 2023-10-05T19:28:24.146 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.138+0000 7ff9e8c63700 1 -- 172.21.15.167:0/3611872326 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:63 5.19 5:98000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x55dfdad4fc50 con 0x55dfdac8ec20 2023-10-05T19:28:24.146 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.142+0000 7ff9e8c63700 1 -- 172.21.15.167:0/3611872326 <== osd.1 v2:172.21.15.167:6800/30035 23 ==== osd_op_reply(63 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7ff9d802f080 con 0x55dfdac8ec20 2023-10-05T19:28:24.146 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.142+0000 7ff9e8c63700 1 -- 172.21.15.167:0/3611872326 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:64 5.5 5:a0000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x55dfdad4fc50 con 0x7ff9dc007030 2023-10-05T19:28:24.147 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.142+0000 7ff9e8c63700 1 -- 172.21.15.167:0/3611872326 <== osd.0 v2:172.21.15.118:6800/27690 16 ==== osd_op_reply(64 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7ff9d802f080 con 0x7ff9dc007030 2023-10-05T19:28:24.147 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.142+0000 7ff9e8c63700 1 -- 172.21.15.167:0/3611872326 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:65 5.15 5:a8000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x55dfdad4fc50 con 0x55dfdad50610 2023-10-05T19:28:24.148 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.142+0000 7ff9e8462700 1 -- 172.21.15.167:0/3611872326 <== osd.3 v2:172.21.15.167:6808/30036 17 ==== osd_op_reply(65 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7ff9e0048040 con 0x55dfdad50610 2023-10-05T19:28:24.148 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.142+0000 7ff9e8462700 1 -- 172.21.15.167:0/3611872326 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:66 5.d 5:b0000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x55dfdad4fc50 con 0x7ff9e001fc20 2023-10-05T19:28:24.149 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.142+0000 7ff9e7c61700 1 -- 172.21.15.167:0/3611872326 <== osd.2 v2:172.21.15.167:6816/30037 10 ==== osd_op_reply(66 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7ff9dc029040 con 0x7ff9e001fc20 2023-10-05T19:28:24.149 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.142+0000 7ff9e7c61700 1 -- 172.21.15.167:0/3611872326 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:67 5.1d 5:b8000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x55dfdad4fc50 con 0x55dfdac8ec20 2023-10-05T19:28:24.150 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.142+0000 7ff9e8c63700 1 -- 172.21.15.167:0/3611872326 <== osd.1 v2:172.21.15.167:6800/30035 24 ==== osd_op_reply(67 [pgnls start_epoch 30 out=107b] v23'1 uv1 ondisk = 1) v8 ==== 144+0+107 (crc 0 0 0) 0x7ff9d802f080 con 0x55dfdac8ec20 2023-10-05T19:28:24.150 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.142+0000 7ff9e8c63700 1 -- 172.21.15.167:0/3611872326 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:68 5.3 5:c0000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x55dfdad4fc50 con 0x7ff9dc007030 2023-10-05T19:28:24.151 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.146+0000 7ff9e8c63700 1 -- 172.21.15.167:0/3611872326 <== osd.0 v2:172.21.15.118:6800/27690 17 ==== osd_op_reply(68 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7ff9d802f080 con 0x7ff9dc007030 2023-10-05T19:28:24.151 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.146+0000 7ff9e8c63700 1 -- 172.21.15.167:0/3611872326 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:69 5.13 5:c8000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x55dfdad4fc50 con 0x55dfdad50610 2023-10-05T19:28:24.152 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.146+0000 7ff9e8462700 1 -- 172.21.15.167:0/3611872326 <== osd.3 v2:172.21.15.167:6808/30036 18 ==== osd_op_reply(69 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7ff9e0048040 con 0x55dfdad50610 2023-10-05T19:28:24.152 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.146+0000 7ff9e8462700 1 -- 172.21.15.167:0/3611872326 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:70 5.b 5:d0000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x55dfdad4fc50 con 0x7ff9dc007030 2023-10-05T19:28:24.153 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.146+0000 7ff9e8c63700 1 -- 172.21.15.167:0/3611872326 <== osd.0 v2:172.21.15.118:6800/27690 18 ==== osd_op_reply(70 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7ff9d802f080 con 0x7ff9dc007030 2023-10-05T19:28:24.153 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.146+0000 7ff9e8c63700 1 -- 172.21.15.167:0/3611872326 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:71 5.1b 5:d8000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x55dfdad4fc50 con 0x7ff9e001fc20 2023-10-05T19:28:24.154 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.150+0000 7ff9e7c61700 1 -- 172.21.15.167:0/3611872326 <== osd.2 v2:172.21.15.167:6816/30037 11 ==== osd_op_reply(71 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7ff9dc029040 con 0x7ff9e001fc20 2023-10-05T19:28:24.154 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.150+0000 7ff9e7c61700 1 -- 172.21.15.167:0/3611872326 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:72 5.7 5:e0000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x55dfdad4fc50 con 0x55dfdad50610 2023-10-05T19:28:24.154 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.150+0000 7ff9e8462700 1 -- 172.21.15.167:0/3611872326 <== osd.3 v2:172.21.15.167:6808/30036 19 ==== osd_op_reply(72 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7ff9e0048040 con 0x55dfdad50610 2023-10-05T19:28:24.155 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.150+0000 7ff9e8462700 1 -- 172.21.15.167:0/3611872326 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:73 5.17 5:e8000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x55dfdad4fc50 con 0x7ff9dc007030 2023-10-05T19:28:24.155 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.150+0000 7ff9e8c63700 1 -- 172.21.15.167:0/3611872326 <== osd.0 v2:172.21.15.118:6800/27690 19 ==== osd_op_reply(73 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7ff9d802f080 con 0x7ff9dc007030 2023-10-05T19:28:24.156 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.150+0000 7ff9e8c63700 1 -- 172.21.15.167:0/3611872326 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:74 5.f 5:f0000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x55dfdad4fc50 con 0x55dfdac8ec20 2023-10-05T19:28:24.156 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.150+0000 7ff9e8c63700 1 -- 172.21.15.167:0/3611872326 <== osd.1 v2:172.21.15.167:6800/30035 25 ==== osd_op_reply(74 [pgnls start_epoch 30 out=115b] v23'2 uv2 ondisk = 1) v8 ==== 144+0+115 (crc 0 0 0) 0x7ff9d802f080 con 0x55dfdac8ec20 2023-10-05T19:28:24.157 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.150+0000 7ff9e8c63700 1 -- 172.21.15.167:0/3611872326 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:75 5.1f 5:f8000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x55dfdad4fc50 con 0x7ff9e001fc20 2023-10-05T19:28:24.158 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.150+0000 7ff9e7c61700 1 -- 172.21.15.167:0/3611872326 <== osd.2 v2:172.21.15.167:6816/30037 12 ==== osd_op_reply(75 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7ff9dc029040 con 0x7ff9e001fc20 2023-10-05T19:28:24.158 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.150+0000 7ff9ea92de40 20 RGWRados::pool_iterate: got zonegroup_info.4f1f719d-9648-417b-b10c-e5d939825419 2023-10-05T19:28:24.158 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.150+0000 7ff9ea92de40 20 RGWRados::pool_iterate: got zone_info.9a9df965-6612-45d4-a45f-6e3c889bacc2 2023-10-05T19:28:24.158 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.150+0000 7ff9ea92de40 20 RGWRados::pool_iterate: got zone_names.default 2023-10-05T19:28:24.158 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.150+0000 7ff9ea92de40 20 RGWRados::pool_iterate: got zonegroups_names.default 2023-10-05T19:28:24.158 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.150+0000 7ff9ea92de40 20 rados->read ofs=0 len=0 2023-10-05T19:28:24.158 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.150+0000 7ff9ea92de40 1 -- 172.21.15.167:0/3611872326 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:76 5.f 5:f43b6ece:::zone_names.default:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x55dfdad4f850 con 0x55dfdac8ec20 2023-10-05T19:28:24.159 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.154+0000 7ff9e8c63700 1 -- 172.21.15.167:0/3611872326 <== osd.1 v2:172.21.15.167:6800/30035 26 ==== osd_op_reply(76 zone_names.default [read 0~46 out=46b] v0'0 uv1 ondisk = 0) v8 ==== 162+0+46 (crc 0 0 0) 0x7ff9d802f080 con 0x55dfdac8ec20 2023-10-05T19:28:24.159 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.154+0000 7ff9ea92de40 20 rados_obj.operate() r=0 bl.length=46 2023-10-05T19:28:24.160 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.154+0000 7ff9ea92de40 20 rados->read ofs=0 len=0 2023-10-05T19:28:24.160 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.154+0000 7ff9ea92de40 1 -- 172.21.15.167:0/3611872326 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:77 5.1d 5:b8374eb4:::zone_info.9a9df965-6612-45d4-a45f-6e3c889bacc2:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x55dfdad53c90 con 0x55dfdac8ec20 2023-10-05T19:28:24.160 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.154+0000 7ff9e8c63700 1 -- 172.21.15.167:0/3611872326 <== osd.1 v2:172.21.15.167:6800/30035 27 ==== osd_op_reply(77 zone_info.9a9df965-6612-45d4-a45f-6e3c889bacc2 [read 0~874 out=874b] v0'0 uv1 ondisk = 0) v8 ==== 190+0+874 (crc 0 0 0) 0x7ff9d802f080 con 0x55dfdac8ec20 2023-10-05T19:28:24.160 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.154+0000 7ff9ea92de40 20 rados_obj.operate() r=0 bl.length=874 2023-10-05T19:28:24.160 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.154+0000 7ff9ea92de40 20 rados->read ofs=0 len=0 2023-10-05T19:28:24.160 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.154+0000 7ff9ea92de40 1 -- 172.21.15.167:0/3611872326 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:78 5.f 5:f4c53578:::zonegroups_names.default:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x55dfdad54180 con 0x55dfdac8ec20 2023-10-05T19:28:24.161 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.154+0000 7ff9e8c63700 1 -- 172.21.15.167:0/3611872326 <== osd.1 v2:172.21.15.167:6800/30035 28 ==== osd_op_reply(78 zonegroups_names.default [read 0~46 out=46b] v0'0 uv2 ondisk = 0) v8 ==== 168+0+46 (crc 0 0 0) 0x7ff9d802f080 con 0x55dfdac8ec20 2023-10-05T19:28:24.161 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.154+0000 7ff9ea92de40 20 rados_obj.operate() r=0 bl.length=46 2023-10-05T19:28:24.161 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.154+0000 7ff9ea92de40 20 rados->read ofs=0 len=0 2023-10-05T19:28:24.161 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.154+0000 7ff9ea92de40 1 -- 172.21.15.167:0/3611872326 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:79 5.c 5:30d299a5:::zonegroup_info.4f1f719d-9648-417b-b10c-e5d939825419:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x55dfdad54180 con 0x7ff9dc007030 2023-10-05T19:28:24.161 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.154+0000 7ff9e8c63700 1 -- 172.21.15.167:0/3611872326 <== osd.0 v2:172.21.15.118:6800/27690 20 ==== osd_op_reply(79 zonegroup_info.4f1f719d-9648-417b-b10c-e5d939825419 [read 0~362 out=362b] v0'0 uv1 ondisk = 0) v8 ==== 195+0+362 (crc 0 0 0) 0x7ff9d802f080 con 0x7ff9dc007030 2023-10-05T19:28:24.162 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.154+0000 7ff9ea92de40 20 rados_obj.operate() r=0 bl.length=362 2023-10-05T19:28:24.162 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.154+0000 7ff9ea92de40 20 rados->read ofs=0 len=0 2023-10-05T19:28:24.162 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.154+0000 7ff9ea92de40 1 -- 172.21.15.167:0/3611872326 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:80 5.3 5:c0ad8cbd:::region_map:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x55dfdad54280 con 0x7ff9dc007030 2023-10-05T19:28:24.162 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.158+0000 7ff9e8c63700 1 -- 172.21.15.167:0/3611872326 <== osd.0 v2:172.21.15.118:6800/27690 21 ==== osd_op_reply(80 region_map [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 ==== 154+0+0 (crc 0 0 0) 0x7ff9d8050040 con 0x7ff9dc007030 2023-10-05T19:28:24.162 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.158+0000 7ff9ea92de40 20 rados_obj.operate() r=-2 bl.length=0 2023-10-05T19:28:24.163 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.158+0000 7ff9ea92de40 20 zone default found 2023-10-05T19:28:24.163 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.158+0000 7ff9ea92de40 4 Realm: () 2023-10-05T19:28:24.163 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.158+0000 7ff9ea92de40 4 ZoneGroup: default (4f1f719d-9648-417b-b10c-e5d939825419) 2023-10-05T19:28:24.163 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.158+0000 7ff9ea92de40 4 Zone: default (9a9df965-6612-45d4-a45f-6e3c889bacc2) 2023-10-05T19:28:24.163 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.158+0000 7ff9ea92de40 10 cannot find current period zonegroup using local zonegroup configuration 2023-10-05T19:28:24.163 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.158+0000 7ff9ea92de40 20 zonegroup default 2023-10-05T19:28:24.163 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.158+0000 7ff9ea92de40 20 rados->read ofs=0 len=0 2023-10-05T19:28:24.163 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.158+0000 7ff9ea92de40 1 -- 172.21.15.167:0/3611872326 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:81 5.3 5:c52100b6:::period_config.default:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x55dfdad54280 con 0x7ff9dc007030 2023-10-05T19:28:24.164 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.158+0000 7ff9e8c63700 1 -- 172.21.15.167:0/3611872326 <== osd.0 v2:172.21.15.118:6800/27690 22 ==== osd_op_reply(81 period_config.default [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 ==== 165+0+0 (crc 0 0 0) 0x7ff9d8050040 con 0x7ff9dc007030 2023-10-05T19:28:24.164 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.158+0000 7ff9ea92de40 20 rados_obj.operate() r=-2 bl.length=0 2023-10-05T19:28:24.165 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.158+0000 7ff9ea92de40 20 rados->read ofs=0 len=0 2023-10-05T19:28:24.165 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.158+0000 7ff9ea92de40 1 -- 172.21.15.167:0/3611872326 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:82 6.2 6:4347d321:::bucket.sync-source-hints.:head [call version.read in=11b,read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x55dfdad55c00 con 0x55dfdad50610 2023-10-05T19:28:24.165 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.158+0000 7ff9e8462700 1 -- 172.21.15.167:0/3611872326 <== osd.3 v2:172.21.15.167:6808/30036 20 ==== osd_op_reply(82 bucket.sync-source-hints. [call,read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 ==== 211+0+0 (crc 0 0 0) 0x7ff9e0048040 con 0x55dfdad50610 2023-10-05T19:28:24.165 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.158+0000 7ff9ea92de40 20 rados_obj.operate() r=-2 bl.length=0 2023-10-05T19:28:24.165 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.158+0000 7ff9ea92de40 20 rados->read ofs=0 len=0 2023-10-05T19:28:24.165 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.158+0000 7ff9ea92de40 1 -- 172.21.15.167:0/3611872326 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:83 6.b 6:d467b91b:::bucket.sync-target-hints.:head [call version.read in=11b,read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x55dfdad55fb0 con 0x55dfdad50610 2023-10-05T19:28:24.166 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.158+0000 7ff9e8462700 1 -- 172.21.15.167:0/3611872326 <== osd.3 v2:172.21.15.167:6808/30036 21 ==== osd_op_reply(83 bucket.sync-target-hints. [call,read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 ==== 211+0+0 (crc 0 0 0) 0x7ff9e0048040 con 0x55dfdad50610 2023-10-05T19:28:24.166 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.158+0000 7ff9ea92de40 20 rados_obj.operate() r=-2 bl.length=0 2023-10-05T19:28:24.166 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.158+0000 7ff9ea92de40 20 started sync module instance, tier type = 2023-10-05T19:28:24.166 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.158+0000 7ff9ea92de40 20 started zone id=9a9df965-6612-45d4-a45f-6e3c889bacc2 (name=default) with tier type = 2023-10-05T19:28:24.166 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.158+0000 7ff9ea92de40 1 -- 172.21.15.167:0/3611872326 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:84 7.1f 7:f95f44c2:::notify.0:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x55dfdad552c0 con 0x7ff9e001fc20 2023-10-05T19:28:24.168 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.162+0000 7ff9e7c61700 1 -- 172.21.15.167:0/3611872326 <== osd.2 v2:172.21.15.167:6816/30037 13 ==== osd_op_reply(84 notify.0 [create] v30'9 uv9 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7ff9dc029040 con 0x7ff9e001fc20 2023-10-05T19:28:24.168 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.162+0000 7ff9ea92de40 1 -- 172.21.15.167:0/3611872326 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:85 7.1f 7:f95f44c2:::notify.0:head [watch watch cookie 94419937482352] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x55dfdad54ef0 con 0x7ff9e001fc20 2023-10-05T19:28:24.168 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.162+0000 7ff9ea92de40 1 -- 172.21.15.167:0/3611872326 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:86 7.0 7:05bf5b68:::notify.1:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x55dfdad56d40 con 0x55dfdac8ec20 2023-10-05T19:28:24.171 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.166+0000 7ff9e7c61700 1 -- 172.21.15.167:0/3611872326 <== osd.2 v2:172.21.15.167:6816/30037 14 ==== osd_op_reply(85 notify.0 [watch watch cookie 94419937482352] v30'10 uv9 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7ff9dc029040 con 0x7ff9e001fc20 2023-10-05T19:28:24.171 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.166+0000 7ff9e8c63700 1 -- 172.21.15.167:0/3611872326 <== osd.1 v2:172.21.15.167:6800/30035 29 ==== osd_op_reply(86 notify.1 [create] v30'9 uv9 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7ff9d8050040 con 0x55dfdac8ec20 2023-10-05T19:28:24.171 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.166+0000 7ff9ea92de40 1 -- 172.21.15.167:0/3611872326 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:87 7.0 7:05bf5b68:::notify.1:head [watch watch cookie 94419937485952] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x55dfdad56dc0 con 0x55dfdac8ec20 2023-10-05T19:28:24.171 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.166+0000 7ff9ea92de40 1 -- 172.21.15.167:0/3611872326 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:88 7.15 7:a93a5511:::notify.2:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x55dfdad57770 con 0x55dfdad50610 2023-10-05T19:28:24.174 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.170+0000 7ff9e8c63700 1 -- 172.21.15.167:0/3611872326 <== osd.1 v2:172.21.15.167:6800/30035 30 ==== osd_op_reply(87 notify.1 [watch watch cookie 94419937485952] v30'10 uv9 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7ff9d8050040 con 0x55dfdac8ec20 2023-10-05T19:28:24.175 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.170+0000 7ff9e8462700 1 -- 172.21.15.167:0/3611872326 <== osd.3 v2:172.21.15.167:6808/30036 22 ==== osd_op_reply(88 notify.2 [create] v30'9 uv9 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7ff9e0048040 con 0x55dfdad50610 2023-10-05T19:28:24.175 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.170+0000 7ff9ea92de40 1 -- 172.21.15.167:0/3611872326 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:89 7.15 7:a93a5511:::notify.2:head [watch watch cookie 94419937487056] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x55dfdad56d20 con 0x55dfdad50610 2023-10-05T19:28:24.175 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.170+0000 7ff9ea92de40 1 -- 172.21.15.167:0/3611872326 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:90 7.e 7:7759931f:::notify.3:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x55dfdad58550 con 0x7ff9e001fc20 2023-10-05T19:28:24.177 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.170+0000 7ff9e7c61700 1 -- 172.21.15.167:0/3611872326 <== osd.2 v2:172.21.15.167:6816/30037 15 ==== osd_op_reply(90 notify.3 [create] v30'9 uv9 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7ff9dc029040 con 0x7ff9e001fc20 2023-10-05T19:28:24.177 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.170+0000 7ff9ea92de40 1 -- 172.21.15.167:0/3611872326 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:91 7.e 7:7759931f:::notify.3:head [watch watch cookie 94419937495296] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x55dfdad59240 con 0x7ff9e001fc20 2023-10-05T19:28:24.178 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.170+0000 7ff9ea92de40 1 -- 172.21.15.167:0/3611872326 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:92 7.d 7:b4812045:::notify.4:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x55dfdad59ae0 con 0x55dfdac8ec20 2023-10-05T19:28:24.178 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.174+0000 7ff9e8462700 1 -- 172.21.15.167:0/3611872326 <== osd.3 v2:172.21.15.167:6808/30036 23 ==== osd_op_reply(89 notify.2 [watch watch cookie 94419937487056] v30'10 uv9 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7ff9e0048040 con 0x55dfdad50610 2023-10-05T19:28:24.180 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.174+0000 7ff9e7c61700 1 -- 172.21.15.167:0/3611872326 <== osd.2 v2:172.21.15.167:6816/30037 16 ==== osd_op_reply(91 notify.3 [watch watch cookie 94419937495296] v30'10 uv9 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7ff9dc029040 con 0x7ff9e001fc20 2023-10-05T19:28:24.180 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.174+0000 7ff9e8c63700 1 -- 172.21.15.167:0/3611872326 <== osd.1 v2:172.21.15.167:6800/30035 31 ==== osd_op_reply(92 notify.4 [create] v30'9 uv9 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7ff9d8050040 con 0x55dfdac8ec20 2023-10-05T19:28:24.181 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.174+0000 7ff9ea92de40 1 -- 172.21.15.167:0/3611872326 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:93 7.d 7:b4812045:::notify.4:head [watch watch cookie 94419937491824] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x55dfdad59ae0 con 0x55dfdac8ec20 2023-10-05T19:28:24.181 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.174+0000 7ff9ea92de40 1 -- 172.21.15.167:0/3611872326 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:94 7.3 7:c609908c:::notify.5:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x55dfdad5a3d0 con 0x7ff9dc007030 2023-10-05T19:28:24.183 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.178+0000 7ff9e8c63700 1 -- 172.21.15.167:0/3611872326 <== osd.0 v2:172.21.15.118:6800/27690 23 ==== osd_op_reply(94 notify.5 [create] v30'9 uv9 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7ff9d8050040 con 0x7ff9dc007030 2023-10-05T19:28:24.183 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.178+0000 7ff9ea92de40 1 -- 172.21.15.167:0/3611872326 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:95 7.3 7:c609908c:::notify.5:head [watch watch cookie 94419937500432] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x55dfdad5a690 con 0x7ff9dc007030 2023-10-05T19:28:24.183 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.178+0000 7ff9ea92de40 1 -- 172.21.15.167:0/3611872326 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:96 7.14 7:2b04a3e9:::notify.6:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x55dfdad5b220 con 0x7ff9dc007030 2023-10-05T19:28:24.183 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.178+0000 7ff9e8c63700 1 -- 172.21.15.167:0/3611872326 <== osd.1 v2:172.21.15.167:6800/30035 32 ==== osd_op_reply(93 notify.4 [watch watch cookie 94419937491824] v30'10 uv9 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7ff9d8050040 con 0x55dfdac8ec20 2023-10-05T19:28:24.185 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.178+0000 7ff9e8c63700 1 -- 172.21.15.167:0/3611872326 <== osd.0 v2:172.21.15.118:6800/27690 24 ==== osd_op_reply(95 notify.5 [watch watch cookie 94419937500432] v30'10 uv9 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7ff9d8050040 con 0x7ff9dc007030 2023-10-05T19:28:24.185 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.178+0000 7ff9e8c63700 1 -- 172.21.15.167:0/3611872326 <== osd.0 v2:172.21.15.118:6800/27690 25 ==== osd_op_reply(96 notify.6 [create] v30'9 uv9 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7ff9d8050040 con 0x7ff9dc007030 2023-10-05T19:28:24.185 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.178+0000 7ff9ea92de40 1 -- 172.21.15.167:0/3611872326 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:97 7.14 7:2b04a3e9:::notify.6:head [watch watch cookie 94419937498608] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x55dfdad5acf0 con 0x7ff9dc007030 2023-10-05T19:28:24.186 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.178+0000 7ff9ea92de40 1 -- 172.21.15.167:0/3611872326 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:98 7.9 7:93e5b521:::notify.7:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x55dfdad5bb10 con 0x7ff9dc007030 2023-10-05T19:28:24.188 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.182+0000 7ff9e8c63700 1 -- 172.21.15.167:0/3611872326 <== osd.0 v2:172.21.15.118:6800/27690 26 ==== osd_op_reply(98 notify.7 [create] v30'9 uv9 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7ff9d8050040 con 0x7ff9dc007030 2023-10-05T19:28:24.188 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.182+0000 7ff9ea92de40 1 -- 172.21.15.167:0/3611872326 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:99 7.9 7:93e5b521:::notify.7:head [watch watch cookie 94419937506848] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x55dfdad5acf0 con 0x7ff9dc007030 2023-10-05T19:28:24.188 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.182+0000 7ff9ea92de40 20 add_watcher() i=0 2023-10-05T19:28:24.188 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.182+0000 7ff9ea92de40 20 add_watcher() i=1 2023-10-05T19:28:24.189 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.182+0000 7ff9ea92de40 20 add_watcher() i=2 2023-10-05T19:28:24.189 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.182+0000 7ff9ea92de40 20 add_watcher() i=3 2023-10-05T19:28:24.189 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.182+0000 7ff9ea92de40 20 add_watcher() i=4 2023-10-05T19:28:24.189 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.182+0000 7ff9ea92de40 20 add_watcher() i=5 2023-10-05T19:28:24.189 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.182+0000 7ff9e8c63700 1 -- 172.21.15.167:0/3611872326 <== osd.0 v2:172.21.15.118:6800/27690 27 ==== osd_op_reply(97 notify.6 [watch watch cookie 94419937498608] v30'10 uv9 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7ff9d8050040 con 0x7ff9dc007030 2023-10-05T19:28:24.189 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.182+0000 7ff9ea92de40 20 add_watcher() i=6 2023-10-05T19:28:24.190 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.186+0000 7ff9e8c63700 1 -- 172.21.15.167:0/3611872326 <== osd.0 v2:172.21.15.118:6800/27690 28 ==== osd_op_reply(99 notify.7 [watch watch cookie 94419937506848] v30'10 uv9 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7ff9d8050040 con 0x7ff9dc007030 2023-10-05T19:28:24.190 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.186+0000 7ff9ea92de40 20 add_watcher() i=7 2023-10-05T19:28:24.190 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.186+0000 7ff9ea92de40 2 all 8 watchers are set, enabling cache 2023-10-05T19:28:24.190 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.186+0000 7ff9ea92de40 1 -- 172.21.15.167:0/3611872326 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:100 6.4 6:22d26bf9:::data_loggenerations_metadata:head [call version.check_conds in=74b,call version.read in=11b,read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x55dfdad5eaa0 con 0x55dfdac8ec20 2023-10-05T19:28:24.191 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.186+0000 7ff9e8c63700 1 -- 172.21.15.167:0/3611872326 <== osd.1 v2:172.21.15.167:6800/30035 33 ==== osd_op_reply(100 data_loggenerations_metadata [call,call out=48b,read 0~28 out=28b] v0'0 uv1 ondisk = 0) v8 ==== 256+0+76 (crc 0 0 0) 0x7ff9d8050040 con 0x55dfdac8ec20 2023-10-05T19:28:24.191 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.186+0000 7ff9ea92de40 1 -- 172.21.15.167:0/3611872326 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:101 6.4 6:22d26bf9:::data_loggenerations_metadata:head [watch watch cookie 94419937501536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x55dfdad5bf60 con 0x55dfdac8ec20 2023-10-05T19:28:24.193 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.186+0000 7ff9e8c63700 1 -- 172.21.15.167:0/3611872326 <== osd.1 v2:172.21.15.167:6800/30035 34 ==== osd_op_reply(101 data_loggenerations_metadata [watch watch cookie 94419937501536] v30'31 uv1 ondisk = 0) v8 ==== 172+0+0 (crc 0 0 0) 0x7ff9d8050040 con 0x55dfdac8ec20 2023-10-05T19:28:24.193 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.186+0000 7ff9b9fe3700 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2023-10-05T19:28:24.193 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.186+0000 7ff9ea92de40 20 check_secure_mon_conn(): auth registy supported: methods=[2] modes=[2,1] 2023-10-05T19:28:24.194 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.186+0000 7ff9ea92de40 20 check_secure_mon_conn(): mode 1 is insecure 2023-10-05T19:28:24.195 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.186+0000 7ff9ea92de40 1 Processor -- start 2023-10-05T19:28:24.195 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.190+0000 7ff9ea92de40 1 -- start start 2023-10-05T19:28:24.195 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.190+0000 7ff9ea92de40 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55dfdad690d0 0x55dfdad694a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-05T19:28:24.195 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.190+0000 7ff9ea92de40 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x55dfdad699e0 con 0x55dfdad690d0 2023-10-05T19:28:24.195 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.190+0000 7ff9e7c61700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55dfdad690d0 0x55dfdad694a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-05T19:28:24.195 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.190+0000 7ff9e7c61700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55dfdad690d0 0x55dfdad694a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.118:3300/0 says I am v2:172.21.15.167:50684/0 (socket says 172.21.15.167:50684) 2023-10-05T19:28:24.195 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.190+0000 7ff9e7c61700 1 -- 172.21.15.167:0/2846096232 learned_addr learned my addr 172.21.15.167:0/2846096232 (peer_addr_for_me v2:172.21.15.167:0/0) 2023-10-05T19:28:24.196 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.190+0000 7ff9e7c61700 1 -- 172.21.15.167:0/2846096232 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55dfdad69b20 con 0x55dfdad690d0 2023-10-05T19:28:24.196 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.190+0000 7ff9e7c61700 1 --2- 172.21.15.167:0/2846096232 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55dfdad690d0 0x55dfdad694a0 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7ff9dc002000 tx=0x7ff9dc011000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f328c63fa73f9869 server_cookie=0 in_seq=0 out_seq=0 2023-10-05T19:28:24.197 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.190+0000 7ff9b97e2700 1 -- 172.21.15.167:0/2846096232 <== mon.0 v2:172.21.15.118:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7ff9dc025040 con 0x55dfdad690d0 2023-10-05T19:28:24.197 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.190+0000 7ff9b97e2700 1 -- 172.21.15.167:0/2846096232 <== mon.0 v2:172.21.15.118:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff9dc039450 con 0x55dfdad690d0 2023-10-05T19:28:24.197 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.190+0000 7ff9b97e2700 1 -- 172.21.15.167:0/2846096232 <== mon.0 v2:172.21.15.118:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7ff9dc039720 con 0x55dfdad690d0 2023-10-05T19:28:24.198 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.190+0000 7ff9ea92de40 1 -- 172.21.15.167:0/2846096232 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55dfdad690d0 msgr2=0x55dfdad694a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-05T19:28:24.198 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.190+0000 7ff9ea92de40 1 --2- 172.21.15.167:0/2846096232 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55dfdad690d0 0x55dfdad694a0 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7ff9dc002000 tx=0x7ff9dc011000 comp rx=0 tx=0).stop 2023-10-05T19:28:24.198 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.190+0000 7ff9ea92de40 1 -- 172.21.15.167:0/2846096232 shutdown_connections 2023-10-05T19:28:24.198 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.190+0000 7ff9ea92de40 1 --2- 172.21.15.167:0/2846096232 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55dfdad690d0 0x55dfdad694a0 unknown :-1 s=CLOSED pgs=150 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-05T19:28:24.198 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.190+0000 7ff9ea92de40 1 -- 172.21.15.167:0/2846096232 >> 172.21.15.167:0/2846096232 conn(0x55dfdad64490 msgr2=0x55dfdad625e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-05T19:28:24.198 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.190+0000 7ff9ea92de40 1 -- 172.21.15.167:0/2846096232 shutdown_connections 2023-10-05T19:28:24.198 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.190+0000 7ff9ea92de40 1 -- 172.21.15.167:0/2846096232 wait complete. 2023-10-05T19:28:24.199 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.194+0000 7ff9ea92de40 1 Processor -- start 2023-10-05T19:28:24.199 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.194+0000 7ff9ea92de40 1 -- start start 2023-10-05T19:28:24.200 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.194+0000 7ff9ea92de40 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55dfdad690d0 0x55dfdad6eae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-05T19:28:24.200 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.194+0000 7ff9ea92de40 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x55dfdad6fae0 con 0x55dfdad690d0 2023-10-05T19:28:24.200 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.194+0000 7ff9e7c61700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55dfdad690d0 0x55dfdad6eae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-05T19:28:24.200 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.194+0000 7ff9e7c61700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55dfdad690d0 0x55dfdad6eae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.118:3300/0 says I am v2:172.21.15.167:50698/0 (socket says 172.21.15.167:50698) 2023-10-05T19:28:24.200 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.194+0000 7ff9e7c61700 1 -- 172.21.15.167:0/191540054 learned_addr learned my addr 172.21.15.167:0/191540054 (peer_addr_for_me v2:172.21.15.167:0/0) 2023-10-05T19:28:24.200 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.194+0000 7ff9e7c61700 1 -- 172.21.15.167:0/191540054 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55dfdac8f0b0 con 0x55dfdad690d0 2023-10-05T19:28:24.201 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.194+0000 7ff9e7c61700 1 --2- 172.21.15.167:0/191540054 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55dfdad690d0 0x55dfdad6eae0 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7ff9dc007fd0 tx=0x7ff9dc039000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-05T19:28:24.201 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.194+0000 7ff9337fe700 1 -- 172.21.15.167:0/191540054 <== mon.0 v2:172.21.15.118:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7ff9dc025070 con 0x55dfdad690d0 2023-10-05T19:28:24.201 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.194+0000 7ff9337fe700 1 -- 172.21.15.167:0/191540054 <== mon.0 v2:172.21.15.118:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff9dc02d8b0 con 0x55dfdad690d0 2023-10-05T19:28:24.201 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.194+0000 7ff9ea92de40 1 -- 172.21.15.167:0/191540054 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x55dfdad6f0e0 con 0x55dfdad690d0 2023-10-05T19:28:24.202 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.194+0000 7ff9337fe700 1 -- 172.21.15.167:0/191540054 <== mon.0 v2:172.21.15.118:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7ff9dc02db60 con 0x55dfdad690d0 2023-10-05T19:28:24.202 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.194+0000 7ff9ea92de40 1 -- 172.21.15.167:0/191540054 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55dfdad6f580 con 0x55dfdad690d0 2023-10-05T19:28:24.203 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.198+0000 7ff9337fe700 1 -- 172.21.15.167:0/191540054 <== mon.0 v2:172.21.15.118:3300/0 4 ==== mgrmap(e 5) v1 ==== 40329+0+0 (secure 0 0 0) 0x7ff9dc02dcc0 con 0x55dfdad690d0 2023-10-05T19:28:24.203 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.198+0000 7ff9337fe700 1 --2- 172.21.15.167:0/191540054 >> [v2:172.21.15.167:6824/29889,v1:172.21.15.167:6825/29889] conn(0x7ff928030c00 0x7ff9280330b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-05T19:28:24.203 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.198+0000 7ff9337fe700 1 -- 172.21.15.167:0/191540054 <== mon.0 v2:172.21.15.118:3300/0 5 ==== osd_map(30..30 src has 1..30) v4 ==== 6057+0+0 (secure 0 0 0) 0x7ff9dc032030 con 0x55dfdad690d0 2023-10-05T19:28:24.203 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.198+0000 7ff9e8c63700 1 --2- 172.21.15.167:0/191540054 >> [v2:172.21.15.167:6824/29889,v1:172.21.15.167:6825/29889] conn(0x7ff928030c00 0x7ff9280330b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-05T19:28:24.204 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.198+0000 7ff9ea92de40 1 --2- 172.21.15.167:0/191540054 >> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] conn(0x55dfdad796d0 0x55dfdad7bab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-05T19:28:24.204 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.198+0000 7ff9ea92de40 1 -- 172.21.15.167:0/191540054 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:1 6.19 6:9a6dd8c0:gc::gc.0:head [create,call version.check_conds in=74b,call rgw_gc.rgw_gc_queue_init in=45b,call version.set in=34b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x55dfdad7c100 con 0x55dfdad796d0 2023-10-05T19:28:24.204 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.198+0000 7ff9e8462700 1 --2- 172.21.15.167:0/191540054 >> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] conn(0x55dfdad796d0 0x55dfdad7bab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-05T19:28:24.204 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.198+0000 7ff9e8c63700 1 --2- 172.21.15.167:0/191540054 >> [v2:172.21.15.167:6824/29889,v1:172.21.15.167:6825/29889] conn(0x7ff928030c00 0x7ff9280330b0 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x55dfdad5a620 tx=0x7ff9d8000f90 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-05T19:28:24.204 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.198+0000 7ff9e8462700 1 --2- 172.21.15.167:0/191540054 >> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] conn(0x55dfdad796d0 0x55dfdad7bab0 crc :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-05T19:28:24.207 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.202+0000 7ff9e8462700 1 -- 172.21.15.167:0/191540054 <== osd.1 v2:172.21.15.167:6800/30035 1 ==== osd_op_reply(1 gc.0 [create,call,call,call] v30'17 uv3 ondisk = -125 ((125) Operation canceled)) v8 ==== 274+0+0 (crc 0 0 0) 0x7ff9e0048040 con 0x55dfdad796d0 2023-10-05T19:28:24.207 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.202+0000 7ff9ea92de40 1 --2- 172.21.15.167:0/191540054 >> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] conn(0x55dfdad75310 0x55dfdad7f770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-05T19:28:24.207 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.202+0000 7ff9ea92de40 1 -- 172.21.15.167:0/191540054 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:2 6.18 6:1e85421f:gc::gc.1:head [create,call version.check_conds in=74b,call rgw_gc.rgw_gc_queue_init in=45b,call version.set in=34b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x55dfdad7fcf0 con 0x55dfdad75310 2023-10-05T19:28:24.207 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.202+0000 7ff9e7c61700 1 --2- 172.21.15.167:0/191540054 >> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] conn(0x55dfdad75310 0x55dfdad7f770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-05T19:28:24.208 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.202+0000 7ff9e7c61700 1 --2- 172.21.15.167:0/191540054 >> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] conn(0x55dfdad75310 0x55dfdad7f770 crc :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-05T19:28:24.211 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.206+0000 7ff9e7c61700 1 -- 172.21.15.167:0/191540054 <== osd.0 v2:172.21.15.118:6800/27690 1 ==== osd_op_reply(2 gc.1 [create,call,call,call] v30'11 uv5 ondisk = -125 ((125) Operation canceled)) v8 ==== 274+0+0 (crc 0 0 0) 0x7ff9dc048070 con 0x55dfdad75310 2023-10-05T19:28:24.211 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.206+0000 7ff9ea92de40 1 --2- 172.21.15.167:0/191540054 >> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] conn(0x55dfdad809c0 0x55dfdad82da0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-05T19:28:24.211 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.206+0000 7ff9ea92de40 1 -- 172.21.15.167:0/191540054 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:3 6.b 6:d242335b:gc::gc.2:head [create,call version.check_conds in=74b,call rgw_gc.rgw_gc_queue_init in=45b,call version.set in=34b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x55dfdad834a0 con 0x55dfdad809c0 2023-10-05T19:28:24.211 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.206+0000 7ff9e8c63700 1 --2- 172.21.15.167:0/191540054 >> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] conn(0x55dfdad809c0 0x55dfdad82da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-05T19:28:24.212 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.206+0000 7ff9e8c63700 1 --2- 172.21.15.167:0/191540054 >> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] conn(0x55dfdad809c0 0x55dfdad82da0 crc :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-05T19:28:24.214 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.210+0000 7ff9e8c63700 1 -- 172.21.15.167:0/191540054 <== osd.3 v2:172.21.15.167:6808/30036 1 ==== osd_op_reply(3 gc.2 [create,call,call,call] v30'11 uv3 ondisk = -125 ((125) Operation canceled)) v8 ==== 274+0+0 (crc 0 0 0) 0x7ff9d8050040 con 0x55dfdad809c0 2023-10-05T19:28:24.214 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.210+0000 7ff9ea92de40 1 -- 172.21.15.167:0/191540054 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:4 6.4 6:2166d8e1:gc::gc.3:head [create,call version.check_conds in=74b,call rgw_gc.rgw_gc_queue_init in=45b,call version.set in=34b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x55dfdad83ac0 con 0x55dfdad796d0 2023-10-05T19:28:24.217 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.210+0000 7ff9e8462700 1 -- 172.21.15.167:0/191540054 <== osd.1 v2:172.21.15.167:6800/30035 2 ==== osd_op_reply(4 gc.3 [create,call,call,call] v30'32 uv7 ondisk = -125 ((125) Operation canceled)) v8 ==== 274+0+0 (crc 0 0 0) 0x7ff9e0048040 con 0x55dfdad796d0 2023-10-05T19:28:24.217 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.210+0000 7ff9ea92de40 1 --2- 172.21.15.167:0/191540054 >> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] conn(0x55dfdad84ae0 0x55dfdad86ec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-05T19:28:24.217 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.210+0000 7ff9ea92de40 1 -- 172.21.15.167:0/191540054 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:5 6.3 6:c60da066:gc::gc.4:head [create,call version.check_conds in=74b,call rgw_gc.rgw_gc_queue_init in=45b,call version.set in=34b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x55dfdad87570 con 0x55dfdad84ae0 2023-10-05T19:28:24.217 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.210+0000 7ff9e8462700 1 --2- 172.21.15.167:0/191540054 >> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] conn(0x55dfdad84ae0 0x55dfdad86ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-05T19:28:24.218 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.214+0000 7ff9e8462700 1 --2- 172.21.15.167:0/191540054 >> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] conn(0x55dfdad84ae0 0x55dfdad86ec0 crc :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-05T19:28:24.220 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.214+0000 7ff9e8462700 1 -- 172.21.15.167:0/191540054 <== osd.2 v2:172.21.15.167:6816/30037 1 ==== osd_op_reply(5 gc.4 [create,call,call,call] v30'23 uv7 ondisk = -125 ((125) Operation canceled)) v8 ==== 274+0+0 (crc 0 0 0) 0x7ff9e0048040 con 0x55dfdad84ae0 2023-10-05T19:28:24.221 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.214+0000 7ff9ea92de40 1 -- 172.21.15.167:0/191540054 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:6 6.9 6:954be8d2:gc::gc.5:head [create,call version.check_conds in=74b,call rgw_gc.rgw_gc_queue_init in=45b,call version.set in=34b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x55dfdad87bc0 con 0x55dfdad809c0 2023-10-05T19:28:24.223 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.218+0000 7ff9e8c63700 1 -- 172.21.15.167:0/191540054 <== osd.3 v2:172.21.15.167:6808/30036 2 ==== osd_op_reply(6 gc.5 [create,call,call,call] v30'15 uv4 ondisk = -125 ((125) Operation canceled)) v8 ==== 274+0+0 (crc 0 0 0) 0x7ff9d8050040 con 0x55dfdad809c0 2023-10-05T19:28:24.223 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.218+0000 7ff9ea92de40 1 -- 172.21.15.167:0/191540054 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:7 6.e 6:748722ff:gc::gc.6:head [create,call version.check_conds in=74b,call rgw_gc.rgw_gc_queue_init in=45b,call version.set in=34b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x55dfdad881e0 con 0x55dfdad796d0 2023-10-05T19:28:24.226 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.222+0000 7ff9e8462700 1 -- 172.21.15.167:0/191540054 <== osd.1 v2:172.21.15.167:6800/30035 3 ==== osd_op_reply(7 gc.6 [create,call,call,call] v30'13 uv5 ondisk = -125 ((125) Operation canceled)) v8 ==== 274+0+0 (crc 0 0 0) 0x7ff9e0048040 con 0x55dfdad796d0 2023-10-05T19:28:24.226 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.222+0000 7ff9ea92de40 1 -- 172.21.15.167:0/191540054 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:8 6.1d 6:beaa4be4:gc::gc.7:head [create,call version.check_conds in=74b,call rgw_gc.rgw_gc_queue_init in=45b,call version.set in=34b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x55dfdad886d0 con 0x55dfdad75310 2023-10-05T19:28:24.229 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.222+0000 7ff9e7c61700 1 -- 172.21.15.167:0/191540054 <== osd.0 v2:172.21.15.118:6800/27690 2 ==== osd_op_reply(8 gc.7 [create,call,call,call] v30'15 uv4 ondisk = -125 ((125) Operation canceled)) v8 ==== 274+0+0 (crc 0 0 0) 0x7ff9dc048070 con 0x55dfdad75310 2023-10-05T19:28:24.229 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.222+0000 7ff9ea92de40 1 -- 172.21.15.167:0/191540054 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:9 6.12 6:4f78e9a2:gc::gc.8:head [create,call version.check_conds in=74b,call rgw_gc.rgw_gc_queue_init in=45b,call version.set in=34b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x55dfdad78220 con 0x55dfdad84ae0 2023-10-05T19:28:24.231 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.226+0000 7ff9e8462700 1 -- 172.21.15.167:0/191540054 <== osd.2 v2:172.21.15.167:6816/30037 2 ==== osd_op_reply(9 gc.8 [create,call,call,call] v30'11 uv3 ondisk = -125 ((125) Operation canceled)) v8 ==== 274+0+0 (crc 0 0 0) 0x7ff9e0048040 con 0x55dfdad84ae0 2023-10-05T19:28:24.231 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.226+0000 7ff9ea92de40 1 -- 172.21.15.167:0/191540054 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:10 6.1c 6:3e426232:gc::gc.9:head [create,call version.check_conds in=74b,call rgw_gc.rgw_gc_queue_init in=45b,call version.set in=34b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x55dfdad78220 con 0x55dfdad809c0 2023-10-05T19:28:24.234 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.226+0000 7ff9e8c63700 1 -- 172.21.15.167:0/191540054 <== osd.3 v2:172.21.15.167:6808/30036 3 ==== osd_op_reply(10 gc.9 [create,call,call,call] v30'17 uv3 ondisk = -125 ((125) Operation canceled)) v8 ==== 274+0+0 (crc 0 0 0) 0x7ff9d8050040 con 0x55dfdad809c0 2023-10-05T19:28:24.234 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.226+0000 7ff9ea92de40 1 -- 172.21.15.167:0/191540054 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:11 6.3 6:c31f41d7:gc::gc.10:head [create,call version.check_conds in=74b,call rgw_gc.rgw_gc_queue_init in=45b,call version.set in=34b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x55dfdad78490 con 0x55dfdad84ae0 2023-10-05T19:28:24.236 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.230+0000 7ff9e8462700 1 -- 172.21.15.167:0/191540054 <== osd.2 v2:172.21.15.167:6816/30037 3 ==== osd_op_reply(11 gc.10 [create,call,call,call] v30'24 uv11 ondisk = -125 ((125) Operation canceled)) v8 ==== 275+0+0 (crc 0 0 0) 0x7ff9e0048040 con 0x55dfdad84ae0 2023-10-05T19:28:24.236 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.230+0000 7ff9ea92de40 1 -- 172.21.15.167:0/191540054 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:12 6.13 6:c8333aab:gc::gc.11:head [create,call version.check_conds in=74b,call rgw_gc.rgw_gc_queue_init in=45b,call version.set in=34b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x55dfdad78490 con 0x55dfdad75310 2023-10-05T19:28:24.239 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.234+0000 7ff9e7c61700 1 -- 172.21.15.167:0/191540054 <== osd.0 v2:172.21.15.118:6800/27690 3 ==== osd_op_reply(12 gc.11 [create,call,call,call] v30'13 uv5 ondisk = -125 ((125) Operation canceled)) v8 ==== 275+0+0 (crc 0 0 0) 0x7ff9dc048070 con 0x55dfdad75310 2023-10-05T19:28:24.239 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.234+0000 7ff9ea92de40 1 -- 172.21.15.167:0/191540054 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:13 6.15 6:ab569f81:gc::gc.12:head [create,call version.check_conds in=74b,call rgw_gc.rgw_gc_queue_init in=45b,call version.set in=34b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x55dfdad89460 con 0x55dfdad84ae0 2023-10-05T19:28:24.242 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.234+0000 7ff9e8462700 1 -- 172.21.15.167:0/191540054 <== osd.2 v2:172.21.15.167:6816/30037 4 ==== osd_op_reply(13 gc.12 [create,call,call,call] v30'11 uv3 ondisk = -125 ((125) Operation canceled)) v8 ==== 275+0+0 (crc 0 0 0) 0x7ff9e0048040 con 0x55dfdad84ae0 2023-10-05T19:28:24.242 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.234+0000 7ff9ea92de40 1 -- 172.21.15.167:0/191540054 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:14 6.0 6:0667bb13:gc::gc.13:head [create,call version.check_conds in=74b,call rgw_gc.rgw_gc_queue_init in=45b,call version.set in=34b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x55dfdad89460 con 0x55dfdad75310 2023-10-05T19:28:24.245 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.238+0000 7ff9e7c61700 1 -- 172.21.15.167:0/191540054 <== osd.0 v2:172.21.15.118:6800/27690 4 ==== osd_op_reply(14 gc.13 [create,call,call,call] v30'21 uv6 ondisk = -125 ((125) Operation canceled)) v8 ==== 275+0+0 (crc 0 0 0) 0x7ff9dc048070 con 0x55dfdad75310 2023-10-05T19:28:24.245 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.238+0000 7ff9ea92de40 1 -- 172.21.15.167:0/191540054 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:15 6.4 6:227f63b5:gc::gc.14:head [create,call version.check_conds in=74b,call rgw_gc.rgw_gc_queue_init in=45b,call version.set in=34b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x55dfdad89a80 con 0x55dfdad796d0 2023-10-05T19:28:24.247 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.242+0000 7ff9e8462700 1 -- 172.21.15.167:0/191540054 <== osd.1 v2:172.21.15.167:6800/30035 4 ==== osd_op_reply(15 gc.14 [create,call,call,call] v30'33 uv13 ondisk = -125 ((125) Operation canceled)) v8 ==== 275+0+0 (crc 0 0 0) 0x7ff9e0048040 con 0x55dfdad796d0 2023-10-05T19:28:24.248 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.242+0000 7ff9ea92de40 1 -- 172.21.15.167:0/191540054 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:16 6.7 6:e07a6548:gc::gc.15:head [create,call version.check_conds in=74b,call rgw_gc.rgw_gc_queue_init in=45b,call version.set in=34b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x55dfdad89a80 con 0x55dfdad809c0 2023-10-05T19:28:24.250 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.246+0000 7ff9e8c63700 1 -- 172.21.15.167:0/191540054 <== osd.3 v2:172.21.15.167:6808/30036 4 ==== osd_op_reply(16 gc.15 [create,call,call,call] v30'17 uv7 ondisk = -125 ((125) Operation canceled)) v8 ==== 275+0+0 (crc 0 0 0) 0x7ff9d8050040 con 0x55dfdad809c0 2023-10-05T19:28:24.251 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.246+0000 7ff9ea92de40 1 -- 172.21.15.167:0/191540054 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:17 6.3 6:c623b173:gc::gc.16:head [create,call version.check_conds in=74b,call rgw_gc.rgw_gc_queue_init in=45b,call version.set in=34b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x55dfdad89a80 con 0x55dfdad84ae0 2023-10-05T19:28:24.253 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.246+0000 7ff9e8462700 1 -- 172.21.15.167:0/191540054 <== osd.2 v2:172.21.15.167:6816/30037 5 ==== osd_op_reply(17 gc.16 [create,call,call,call] v30'25 uv13 ondisk = -125 ((125) Operation canceled)) v8 ==== 275+0+0 (crc 0 0 0) 0x7ff9e0048040 con 0x55dfdad84ae0 2023-10-05T19:28:24.253 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.246+0000 7ff9ea92de40 1 -- 172.21.15.167:0/191540054 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:18 6.16 6:6849fd06:gc::gc.17:head [create,call version.check_conds in=74b,call rgw_gc.rgw_gc_queue_init in=45b,call version.set in=34b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x55dfdad89a80 con 0x55dfdad809c0 2023-10-05T19:28:24.256 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.250+0000 7ff9e8c63700 1 -- 172.21.15.167:0/191540054 <== osd.3 v2:172.21.15.167:6808/30036 5 ==== osd_op_reply(18 gc.17 [create,call,call,call] v30'11 uv7 ondisk = -125 ((125) Operation canceled)) v8 ==== 275+0+0 (crc 0 0 0) 0x7ff9d8050040 con 0x55dfdad809c0 2023-10-05T19:28:24.256 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.250+0000 7ff9ea92de40 1 -- 172.21.15.167:0/191540054 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:19 6.d 6:b019ea83:gc::gc.18:head [create,call version.check_conds in=74b,call rgw_gc.rgw_gc_queue_init in=45b,call version.set in=34b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x55dfdad89a80 con 0x55dfdad796d0 2023-10-05T19:28:24.261 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.254+0000 7ff9e8462700 1 -- 172.21.15.167:0/191540054 <== osd.1 v2:172.21.15.167:6800/30035 5 ==== osd_op_reply(19 gc.18 [create,call,call,call] v30'19 uv5 ondisk = -125 ((125) Operation canceled)) v8 ==== 275+0+0 (crc 0 0 0) 0x7ff9e0048040 con 0x55dfdad796d0 2023-10-05T19:28:24.261 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.254+0000 7ff9ea92de40 1 -- 172.21.15.167:0/191540054 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:20 6.4 6:24c8e1ae:gc::gc.19:head [create,call version.check_conds in=74b,call rgw_gc.rgw_gc_queue_init in=45b,call version.set in=34b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x55dfdad89a80 con 0x55dfdad796d0 2023-10-05T19:28:24.262 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.254+0000 7ff9e8462700 1 -- 172.21.15.167:0/191540054 <== osd.1 v2:172.21.15.167:6800/30035 6 ==== osd_op_reply(20 gc.19 [create,call,call,call] v30'34 uv15 ondisk = -125 ((125) Operation canceled)) v8 ==== 275+0+0 (crc 0 0 0) 0x7ff9e0048040 con 0x55dfdad796d0 2023-10-05T19:28:24.262 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.254+0000 7ff9ea92de40 1 -- 172.21.15.167:0/191540054 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:21 6.7 6:e622f76a:gc::gc.20:head [create,call version.check_conds in=74b,call rgw_gc.rgw_gc_queue_init in=45b,call version.set in=34b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x55dfdad89a80 con 0x55dfdad809c0 2023-10-05T19:28:24.265 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.258+0000 7ff9e8c63700 1 -- 172.21.15.167:0/191540054 <== osd.3 v2:172.21.15.167:6808/30036 6 ==== osd_op_reply(21 gc.20 [create,call,call,call] v30'18 uv9 ondisk = -125 ((125) Operation canceled)) v8 ==== 275+0+0 (crc 0 0 0) 0x7ff9d8050040 con 0x55dfdad809c0 2023-10-05T19:28:24.265 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.258+0000 7ff9ea92de40 1 -- 172.21.15.167:0/191540054 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:22 6.a 6:5127e642:gc::gc.21:head [create,call version.check_conds in=74b,call rgw_gc.rgw_gc_queue_init in=45b,call version.set in=34b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x55dfdad89a80 con 0x55dfdad84ae0 2023-10-05T19:28:24.267 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.262+0000 7ff9e8462700 1 -- 172.21.15.167:0/191540054 <== osd.2 v2:172.21.15.167:6816/30037 6 ==== osd_op_reply(22 gc.21 [create,call,call,call] v30'21 uv7 ondisk = -125 ((125) Operation canceled)) v8 ==== 275+0+0 (crc 0 0 0) 0x7ff9e0048040 con 0x55dfdad84ae0 2023-10-05T19:28:24.267 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.262+0000 7ff9ea92de40 1 -- 172.21.15.167:0/191540054 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:23 6.11 6:8eb99c53:gc::gc.22:head [create,call version.check_conds in=74b,call rgw_gc.rgw_gc_queue_init in=45b,call version.set in=34b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x55dfdad89a80 con 0x55dfdad809c0 2023-10-05T19:28:24.270 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.262+0000 7ff9e8c63700 1 -- 172.21.15.167:0/191540054 <== osd.3 v2:172.21.15.167:6808/30036 7 ==== osd_op_reply(23 gc.22 [create,call,call,call] v30'13 uv3 ondisk = -125 ((125) Operation canceled)) v8 ==== 275+0+0 (crc 0 0 0) 0x7ff9d8050040 con 0x55dfdad809c0 2023-10-05T19:28:24.270 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.262+0000 7ff9ea92de40 1 -- 172.21.15.167:0/191540054 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:24 6.8 6:17d7e8ed:gc::gc.23:head [create,call version.check_conds in=74b,call rgw_gc.rgw_gc_queue_init in=45b,call version.set in=34b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x55dfdad89cf0 con 0x55dfdad809c0 2023-10-05T19:28:24.272 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.266+0000 7ff9e8c63700 1 -- 172.21.15.167:0/191540054 <== osd.3 v2:172.21.15.167:6808/30036 8 ==== osd_op_reply(24 gc.23 [create,call,call,call] v30'18 uv5 ondisk = -125 ((125) Operation canceled)) v8 ==== 275+0+0 (crc 0 0 0) 0x7ff9d8050040 con 0x55dfdad809c0 2023-10-05T19:28:24.272 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.266+0000 7ff9ea92de40 1 -- 172.21.15.167:0/191540054 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:25 6.1b 6:db9d130d:gc::gc.24:head [create,call version.check_conds in=74b,call rgw_gc.rgw_gc_queue_init in=45b,call version.set in=34b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x55dfdad89cf0 con 0x55dfdad809c0 2023-10-05T19:28:24.274 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.270+0000 7ff9e8c63700 1 -- 172.21.15.167:0/191540054 <== osd.3 v2:172.21.15.167:6808/30036 9 ==== osd_op_reply(25 gc.24 [create,call,call,call] v30'7 uv3 ondisk = -125 ((125) Operation canceled)) v8 ==== 275+0+0 (crc 0 0 0) 0x7ff9d8050040 con 0x55dfdad809c0 2023-10-05T19:28:24.275 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.270+0000 7ff9ea92de40 1 -- 172.21.15.167:0/191540054 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:26 6.9 6:921ecdb1:gc::gc.25:head [create,call version.check_conds in=74b,call rgw_gc.rgw_gc_queue_init in=45b,call version.set in=34b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x55dfdad8a0f0 con 0x55dfdad809c0 2023-10-05T19:28:24.276 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.270+0000 7ff9e8c63700 1 -- 172.21.15.167:0/191540054 <== osd.3 v2:172.21.15.167:6808/30036 10 ==== osd_op_reply(26 gc.25 [create,call,call,call] v30'16 uv10 ondisk = -125 ((125) Operation canceled)) v8 ==== 275+0+0 (crc 0 0 0) 0x7ff9d8050040 con 0x55dfdad809c0 2023-10-05T19:28:24.276 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.270+0000 7ff9ea92de40 1 -- 172.21.15.167:0/191540054 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:27 6.1f 6:fde4dd55:gc::gc.26:head [create,call version.check_conds in=74b,call rgw_gc.rgw_gc_queue_init in=45b,call version.set in=34b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x55dfdad8a0f0 con 0x55dfdad75310 2023-10-05T19:28:24.279 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.274+0000 7ff9e7c61700 1 -- 172.21.15.167:0/191540054 <== osd.0 v2:172.21.15.118:6800/27690 5 ==== osd_op_reply(27 gc.26 [create,call,call,call] v30'13 uv5 ondisk = -125 ((125) Operation canceled)) v8 ==== 275+0+0 (crc 0 0 0) 0x7ff9dc048070 con 0x55dfdad75310 2023-10-05T19:28:24.279 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.274+0000 7ff9ea92de40 1 -- 172.21.15.167:0/191540054 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:28 6.1a 6:5909e671:gc::gc.27:head [create,call version.check_conds in=74b,call rgw_gc.rgw_gc_queue_init in=45b,call version.set in=34b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x55dfdad8a0f0 con 0x55dfdad809c0 2023-10-05T19:28:24.281 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.274+0000 7ff9e8c63700 1 -- 172.21.15.167:0/191540054 <== osd.3 v2:172.21.15.167:6808/30036 11 ==== osd_op_reply(28 gc.27 [create,call,call,call] v30'11 uv7 ondisk = -125 ((125) Operation canceled)) v8 ==== 275+0+0 (crc 0 0 0) 0x7ff9d8050040 con 0x55dfdad809c0 2023-10-05T19:28:24.281 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.274+0000 7ff9ea92de40 1 -- 172.21.15.167:0/191540054 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:29 6.1d 6:b88e3271:gc::gc.28:head [create,call version.check_conds in=74b,call rgw_gc.rgw_gc_queue_init in=45b,call version.set in=34b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x55dfdad8a0f0 con 0x55dfdad75310 2023-10-05T19:28:24.284 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.278+0000 7ff9e7c61700 1 -- 172.21.15.167:0/191540054 <== osd.0 v2:172.21.15.118:6800/27690 6 ==== osd_op_reply(29 gc.28 [create,call,call,call] v30'16 uv8 ondisk = -125 ((125) Operation canceled)) v8 ==== 275+0+0 (crc 0 0 0) 0x7ff9dc048070 con 0x55dfdad75310 2023-10-05T19:28:24.284 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.278+0000 7ff9ea92de40 1 -- 172.21.15.167:0/191540054 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:30 6.7 6:e47c9322:gc::gc.29:head [create,call version.check_conds in=74b,call rgw_gc.rgw_gc_queue_init in=45b,call version.set in=34b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x55dfdad8a0f0 con 0x55dfdad809c0 2023-10-05T19:28:24.287 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.282+0000 7ff9e8c63700 1 -- 172.21.15.167:0/191540054 <== osd.3 v2:172.21.15.167:6808/30036 12 ==== osd_op_reply(30 gc.29 [create,call,call,call] v30'19 uv11 ondisk = -125 ((125) Operation canceled)) v8 ==== 275+0+0 (crc 0 0 0) 0x7ff9d8050040 con 0x55dfdad809c0 2023-10-05T19:28:24.287 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.282+0000 7ff9ea92de40 1 -- 172.21.15.167:0/191540054 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:31 6.0 6:013d6c33:gc::gc.30:head [create,call version.check_conds in=74b,call rgw_gc.rgw_gc_queue_init in=45b,call version.set in=34b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x55dfdad8a0f0 con 0x55dfdad75310 2023-10-05T19:28:24.290 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.286+0000 7ff9e7c61700 1 -- 172.21.15.167:0/191540054 <== osd.0 v2:172.21.15.118:6800/27690 7 ==== osd_op_reply(31 gc.30 [create,call,call,call] v30'22 uv12 ondisk = -125 ((125) Operation canceled)) v8 ==== 275+0+0 (crc 0 0 0) 0x7ff9dc048070 con 0x55dfdad75310 2023-10-05T19:28:24.290 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.286+0000 7ff9ea92de40 1 -- 172.21.15.167:0/191540054 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:32 6.f 6:f24cb4a8:gc::gc.31:head [create,call version.check_conds in=74b,call rgw_gc.rgw_gc_queue_init in=45b,call version.set in=34b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x55dfdad8a0f0 con 0x55dfdad84ae0 2023-10-05T19:28:24.295 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.286+0000 7ff9e8462700 1 -- 172.21.15.167:0/191540054 <== osd.2 v2:172.21.15.167:6816/30037 7 ==== osd_op_reply(32 gc.31 [create,call,call,call] v30'17 uv5 ondisk = -125 ((125) Operation canceled)) v8 ==== 275+0+0 (crc 0 0 0) 0x7ff9e0048040 con 0x55dfdad84ae0 2023-10-05T19:28:24.296 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.290+0000 7ff930ff9700 20 reqs_thread_entry: start 2023-10-05T19:28:24.296 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.290+0000 7ff9267fc700 20 reqs_thread_entry: start 2023-10-05T19:28:24.296 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.290+0000 7ff9ea92de40 20 init_complete bucket index max shards: 11 2023-10-05T19:28:24.296 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.290+0000 7ff9ea92de40 10 rgw notify: Started notification manager with: 1 workers 2023-10-05T19:28:24.297 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.290+0000 7ff924ff9700 20 rgw notify: INFO: next queues processing will happen at: Thu Oct 5 19:28:54 2023 2023-10-05T19:28:24.297 INFO:teuthology.orchestra.run.smithi167.stderr: 2023-10-05T19:28:24.297 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.290+0000 7ff90bfff700 20 reqs_thread_entry: start 2023-10-05T19:28:24.297 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.290+0000 7ff9ea92de40 20 get_system_obj_state: rctx=0x7ffedcaebd38 obj=default.rgw.meta:users.uid:bar state=0x55dfdabf20b0 s->prefetch_data=0 2023-10-05T19:28:24.297 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.290+0000 7ff9ea92de40 10 cache get: name=default.rgw.meta+users.uid+bar : miss 2023-10-05T19:28:24.297 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.290+0000 7ff9ea92de40 1 -- 172.21.15.167:0/3611872326 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:102 8.e 8:718f6a54:users.uid::bar:head [call version.read in=11b,getxattrs,stat] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x55dfdade8790 con 0x7ff9dc007030 2023-10-05T19:28:24.298 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.290+0000 7ff9e8c63700 1 -- 172.21.15.167:0/3611872326 <== osd.0 v2:172.21.15.118:6800/27690 29 ==== osd_op_reply(102 bar [call,getxattrs,stat] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 ==== 231+0+0 (crc 0 0 0) 0x7ff9d8050040 con 0x7ff9dc007030 2023-10-05T19:28:24.298 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.290+0000 7ff9ea92de40 10 cache put: name=default.rgw.meta+users.uid+bar info.flags=0x0 2023-10-05T19:28:24.298 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.290+0000 7ff9ea92de40 10 adding default.rgw.meta+users.uid+bar to cache LRU end 2023-10-05T19:28:24.298 INFO:teuthology.orchestra.run.smithi167.stderr:could not remove user: unable to remove user, user does not exist 2023-10-05T19:28:24.298 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.294+0000 7ff9ea92de40 1 -- 172.21.15.167:0/3611872326 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:103 7.1f 7:f95f44c2:::notify.0:head [watch unwatch cookie 94419937482352] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x55dfdade8790 con 0x7ff9e001fc20 2023-10-05T19:28:24.301 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.294+0000 7ff9e7c61700 1 -- 172.21.15.167:0/3611872326 <== osd.2 v2:172.21.15.167:6816/30037 17 ==== osd_op_reply(103 notify.0 [watch unwatch cookie 94419937482352] v30'11 uv9 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7ff9dc048070 con 0x7ff9e001fc20 2023-10-05T19:28:24.301 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.294+0000 7ff9ea92de40 20 remove_watcher() i=0 2023-10-05T19:28:24.301 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.294+0000 7ff9ea92de40 2 removed watcher, disabling cache 2023-10-05T19:28:24.301 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.294+0000 7ff9ea92de40 1 -- 172.21.15.167:0/3611872326 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:104 7.0 7:05bf5b68:::notify.1:head [watch unwatch cookie 94419937485952] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x55dfdade8790 con 0x55dfdac8ec20 2023-10-05T19:28:24.303 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.298+0000 7ff9e8c63700 1 -- 172.21.15.167:0/3611872326 <== osd.1 v2:172.21.15.167:6800/30035 35 ==== osd_op_reply(104 notify.1 [watch unwatch cookie 94419937485952] v30'11 uv9 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7ff9d8050040 con 0x55dfdac8ec20 2023-10-05T19:28:24.304 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.298+0000 7ff9ea92de40 20 remove_watcher() i=1 2023-10-05T19:28:24.304 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.298+0000 7ff9ea92de40 1 -- 172.21.15.167:0/3611872326 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:105 7.15 7:a93a5511:::notify.2:head [watch unwatch cookie 94419937487056] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x55dfdade8790 con 0x55dfdad50610 2023-10-05T19:28:24.307 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.302+0000 7ff9e8462700 1 -- 172.21.15.167:0/3611872326 <== osd.3 v2:172.21.15.167:6808/30036 24 ==== osd_op_reply(105 notify.2 [watch unwatch cookie 94419937487056] v30'11 uv9 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7ff9e0048040 con 0x55dfdad50610 2023-10-05T19:28:24.307 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.302+0000 7ff9ea92de40 20 remove_watcher() i=2 2023-10-05T19:28:24.307 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.302+0000 7ff9ea92de40 1 -- 172.21.15.167:0/3611872326 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:106 7.e 7:7759931f:::notify.3:head [watch unwatch cookie 94419937495296] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x55dfdade8790 con 0x7ff9e001fc20 2023-10-05T19:28:24.309 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.302+0000 7ff9e7c61700 1 -- 172.21.15.167:0/3611872326 <== osd.2 v2:172.21.15.167:6816/30037 18 ==== osd_op_reply(106 notify.3 [watch unwatch cookie 94419937495296] v30'11 uv9 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7ff9dc048070 con 0x7ff9e001fc20 2023-10-05T19:28:24.309 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.302+0000 7ff9ea92de40 20 remove_watcher() i=3 2023-10-05T19:28:24.309 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.302+0000 7ff9ea92de40 1 -- 172.21.15.167:0/3611872326 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:107 7.d 7:b4812045:::notify.4:head [watch unwatch cookie 94419937491824] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x55dfdade8790 con 0x55dfdac8ec20 2023-10-05T19:28:24.312 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.306+0000 7ff9e8c63700 1 -- 172.21.15.167:0/3611872326 <== osd.1 v2:172.21.15.167:6800/30035 36 ==== osd_op_reply(107 notify.4 [watch unwatch cookie 94419937491824] v30'11 uv9 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7ff9d8050040 con 0x55dfdac8ec20 2023-10-05T19:28:24.312 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.306+0000 7ff9ea92de40 20 remove_watcher() i=4 2023-10-05T19:28:24.312 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.306+0000 7ff9ea92de40 1 -- 172.21.15.167:0/3611872326 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:108 7.3 7:c609908c:::notify.5:head [watch unwatch cookie 94419937500432] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x55dfdade8790 con 0x7ff9dc007030 2023-10-05T19:28:24.314 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.310+0000 7ff9e8c63700 1 -- 172.21.15.167:0/3611872326 <== osd.0 v2:172.21.15.118:6800/27690 30 ==== osd_op_reply(108 notify.5 [watch unwatch cookie 94419937500432] v30'11 uv9 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7ff9d8050040 con 0x7ff9dc007030 2023-10-05T19:28:24.314 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.310+0000 7ff9ea92de40 20 remove_watcher() i=5 2023-10-05T19:28:24.315 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.310+0000 7ff9ea92de40 1 -- 172.21.15.167:0/3611872326 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:109 7.14 7:2b04a3e9:::notify.6:head [watch unwatch cookie 94419937498608] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x55dfdade8790 con 0x7ff9dc007030 2023-10-05T19:28:24.317 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.310+0000 7ff9e8c63700 1 -- 172.21.15.167:0/3611872326 <== osd.0 v2:172.21.15.118:6800/27690 31 ==== osd_op_reply(109 notify.6 [watch unwatch cookie 94419937498608] v30'11 uv9 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7ff9d8050040 con 0x7ff9dc007030 2023-10-05T19:28:24.317 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.310+0000 7ff9ea92de40 20 remove_watcher() i=6 2023-10-05T19:28:24.317 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.310+0000 7ff9ea92de40 1 -- 172.21.15.167:0/3611872326 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:110 7.9 7:93e5b521:::notify.7:head [watch unwatch cookie 94419937506848] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x55dfdade8790 con 0x7ff9dc007030 2023-10-05T19:28:24.320 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.314+0000 7ff9e8c63700 1 -- 172.21.15.167:0/3611872326 <== osd.0 v2:172.21.15.118:6800/27690 32 ==== osd_op_reply(110 notify.7 [watch unwatch cookie 94419937506848] v30'11 uv9 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7ff9d8050040 con 0x7ff9dc007030 2023-10-05T19:28:24.320 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.314+0000 7ff9ea92de40 20 remove_watcher() i=7 2023-10-05T19:28:24.322 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.318+0000 7ff9b9fe3700 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2023-10-05T19:28:24.322 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.318+0000 7ff9ea92de40 1 -- 172.21.15.167:0/3611872326 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:111 6.4 6:22d26bf9:::data_loggenerations_metadata:head [watch unwatch cookie 94419937501536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x55dfdade8790 con 0x55dfdac8ec20 2023-10-05T19:28:24.326 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.318+0000 7ff9e8c63700 1 -- 172.21.15.167:0/3611872326 <== osd.1 v2:172.21.15.167:6800/30035 37 ==== osd_op_reply(111 data_loggenerations_metadata [watch unwatch cookie 94419937501536] v30'35 uv1 ondisk = 0) v8 ==== 172+0+0 (crc 0 0 0) 0x7ff9d8050040 con 0x55dfdac8ec20 2023-10-05T19:28:24.326 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.318+0000 7ff9ea92de40 1 -- 172.21.15.167:0/3611872326 >> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] conn(0x7ff9dc007030 msgr2=0x7ff9dc01d430 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-05T19:28:24.326 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.318+0000 7ff9ea92de40 1 --2- 172.21.15.167:0/3611872326 >> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] conn(0x7ff9dc007030 0x7ff9dc01d430 crc :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-05T19:28:24.326 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.318+0000 7ff9ea92de40 1 -- 172.21.15.167:0/3611872326 >> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] conn(0x55dfdac8ec20 msgr2=0x55dfdad4c7e0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-05T19:28:24.326 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.318+0000 7ff9ea92de40 1 --2- 172.21.15.167:0/3611872326 >> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] conn(0x55dfdac8ec20 0x55dfdad4c7e0 crc :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-05T19:28:24.326 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.318+0000 7ff9ea92de40 1 -- 172.21.15.167:0/3611872326 >> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] conn(0x7ff9e001fc20 msgr2=0x7ff9e00080e0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-05T19:28:24.326 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.318+0000 7ff9ea92de40 1 --2- 172.21.15.167:0/3611872326 >> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] conn(0x7ff9e001fc20 0x7ff9e00080e0 crc :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-05T19:28:24.326 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.318+0000 7ff9ea92de40 1 -- 172.21.15.167:0/3611872326 >> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] conn(0x55dfdad50610 msgr2=0x55dfdad529f0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-05T19:28:24.326 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.318+0000 7ff9ea92de40 1 --2- 172.21.15.167:0/3611872326 >> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] conn(0x55dfdad50610 0x55dfdad529f0 crc :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-05T19:28:24.326 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.318+0000 7ff9ea92de40 1 -- 172.21.15.167:0/3611872326 >> [v2:172.21.15.167:6824/29889,v1:172.21.15.167:6825/29889] conn(0x7ff9d4030c00 msgr2=0x7ff9d40330b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-05T19:28:24.327 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.318+0000 7ff9ea92de40 1 --2- 172.21.15.167:0/3611872326 >> [v2:172.21.15.167:6824/29889,v1:172.21.15.167:6825/29889] conn(0x7ff9d4030c00 0x7ff9d40330b0 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7ff9dc00afd0 tx=0x7ff9dc014000 comp rx=0 tx=0).stop 2023-10-05T19:28:24.327 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.318+0000 7ff9ea92de40 1 -- 172.21.15.167:0/3611872326 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55dfdac8d910 msgr2=0x55dfdad40cf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-05T19:28:24.327 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.318+0000 7ff9ea92de40 1 --2- 172.21.15.167:0/3611872326 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55dfdac8d910 0x55dfdad40cf0 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x55dfdad365f0 tx=0x7ff9e0014000 comp rx=0 tx=0).stop 2023-10-05T19:28:24.327 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.318+0000 7ff9e8c63700 1 -- 172.21.15.167:0/3611872326 reap_dead start 2023-10-05T19:28:24.327 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.318+0000 7ff9ea92de40 1 -- 172.21.15.167:0/3611872326 shutdown_connections 2023-10-05T19:28:24.327 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.318+0000 7ff9ea92de40 1 -- 172.21.15.167:0/3611872326 >> 172.21.15.167:0/3611872326 conn(0x55dfdad14850 msgr2=0x55dfdad34550 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-05T19:28:24.327 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.318+0000 7ff9ea92de40 1 -- 172.21.15.167:0/3611872326 shutdown_connections 2023-10-05T19:28:24.327 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.318+0000 7ff9ea92de40 1 -- 172.21.15.167:0/3611872326 wait complete. 2023-10-05T19:28:24.328 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.322+0000 7ff9ea92de40 1 -- 172.21.15.167:0/191540054 >> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] conn(0x55dfdad75310 msgr2=0x55dfdad7f770 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-05T19:28:24.329 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.322+0000 7ff9ea92de40 1 --2- 172.21.15.167:0/191540054 >> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] conn(0x55dfdad75310 0x55dfdad7f770 crc :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-05T19:28:24.329 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.322+0000 7ff9ea92de40 1 -- 172.21.15.167:0/191540054 >> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] conn(0x55dfdad796d0 msgr2=0x55dfdad7bab0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-05T19:28:24.329 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.322+0000 7ff9ea92de40 1 --2- 172.21.15.167:0/191540054 >> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] conn(0x55dfdad796d0 0x55dfdad7bab0 crc :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-05T19:28:24.329 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.322+0000 7ff9ea92de40 1 -- 172.21.15.167:0/191540054 >> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] conn(0x55dfdad84ae0 msgr2=0x55dfdad86ec0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-05T19:28:24.329 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.322+0000 7ff9ea92de40 1 --2- 172.21.15.167:0/191540054 >> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] conn(0x55dfdad84ae0 0x55dfdad86ec0 crc :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-05T19:28:24.329 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.322+0000 7ff9ea92de40 1 -- 172.21.15.167:0/191540054 >> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] conn(0x55dfdad809c0 msgr2=0x55dfdad82da0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-05T19:28:24.329 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.322+0000 7ff9ea92de40 1 --2- 172.21.15.167:0/191540054 >> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] conn(0x55dfdad809c0 0x55dfdad82da0 crc :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-05T19:28:24.329 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.322+0000 7ff9ea92de40 1 -- 172.21.15.167:0/191540054 >> [v2:172.21.15.167:6824/29889,v1:172.21.15.167:6825/29889] conn(0x7ff928030c00 msgr2=0x7ff9280330b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-05T19:28:24.329 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.322+0000 7ff9ea92de40 1 --2- 172.21.15.167:0/191540054 >> [v2:172.21.15.167:6824/29889,v1:172.21.15.167:6825/29889] conn(0x7ff928030c00 0x7ff9280330b0 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x55dfdad5a620 tx=0x7ff9d8000f90 comp rx=0 tx=0).stop 2023-10-05T19:28:24.329 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.322+0000 7ff9ea92de40 1 -- 172.21.15.167:0/191540054 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55dfdad690d0 msgr2=0x55dfdad6eae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-05T19:28:24.329 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.322+0000 7ff9e8462700 1 -- 172.21.15.167:0/191540054 reap_dead start 2023-10-05T19:28:24.330 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.322+0000 7ff9ea92de40 1 --2- 172.21.15.167:0/191540054 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55dfdad690d0 0x55dfdad6eae0 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7ff9dc007fd0 tx=0x7ff9dc039000 comp rx=0 tx=0).stop 2023-10-05T19:28:24.330 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.322+0000 7ff9ea92de40 1 -- 172.21.15.167:0/191540054 shutdown_connections 2023-10-05T19:28:24.330 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.322+0000 7ff9ea92de40 1 --2- 172.21.15.167:0/191540054 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55dfdad690d0 0x55dfdad6eae0 unknown :-1 s=CLOSED pgs=151 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-05T19:28:24.330 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.322+0000 7ff9ea92de40 1 -- 172.21.15.167:0/191540054 >> 172.21.15.167:0/191540054 conn(0x55dfdad64490 msgr2=0x55dfdad62530 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-05T19:28:24.330 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.322+0000 7ff9ea92de40 1 -- 172.21.15.167:0/191540054 shutdown_connections 2023-10-05T19:28:24.330 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.322+0000 7ff9ea92de40 1 -- 172.21.15.167:0/191540054 wait complete. 2023-10-05T19:28:24.334 DEBUG:teuthology.orchestra.run:got remote process result: 2 2023-10-05T19:28:24.334 INFO:tasks.util.rgw:rgwadmin: client.0 : ['user', 'rm', '--uid', 'fud'] 2023-10-05T19:28:24.334 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'ceph', 'user', 'rm', '--uid', 'fud'] 2023-10-05T19:28:24.335 DEBUG:teuthology.orchestra.run.smithi167:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster ceph user rm --uid fud 2023-10-05T19:28:24.374 INFO:teuthology.orchestra.run.smithi167.stderr:ignoring --setuser ceph since I am not root 2023-10-05T19:28:24.374 INFO:teuthology.orchestra.run.smithi167.stderr:ignoring --setgroup ceph since I am not root 2023-10-05T19:28:24.375 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.370+0000 7fa26411be40 1 Processor -- start 2023-10-05T19:28:24.375 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.370+0000 7fa26411be40 1 -- start start 2023-10-05T19:28:24.376 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.370+0000 7fa26411be40 1 --2- >> v2:172.21.15.118:3300/0 conn(0x55841eb3c410 0x55841eb3afc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-05T19:28:24.376 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.370+0000 7fa26411be40 1 -- --> v1:172.21.15.118:6789/0 -- auth(proto 0 26 bytes epoch 0) v1 -- 0x55841ea88bd0 con 0x55841eb3c7e0 2023-10-05T19:28:24.376 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.370+0000 7fa26411be40 1 -- --> v2:172.21.15.118:3300/0 -- mon_getmap magic: 0 v1 -- 0x55841ea45ee0 con 0x55841eb3c410 2023-10-05T19:28:24.376 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.370+0000 7fa261c50700 1 --2- >> v2:172.21.15.118:3300/0 conn(0x55841eb3c410 0x55841eb3afc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-05T19:28:24.376 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.370+0000 7fa261c50700 1 --2- >> v2:172.21.15.118:3300/0 conn(0x55841eb3c410 0x55841eb3afc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.118:3300/0 says I am v2:172.21.15.167:56922/0 (socket says 172.21.15.167:56922) 2023-10-05T19:28:24.377 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.370+0000 7fa261c50700 1 -- 172.21.15.167:0/1384687092 learned_addr learned my addr 172.21.15.167:0/1384687092 (peer_addr_for_me v2:172.21.15.167:0/0) 2023-10-05T19:28:24.377 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.370+0000 7fa261c50700 1 -- 172.21.15.167:0/1384687092 >> v1:172.21.15.118:6789/0 conn(0x55841eb3c7e0 legacy=0x55841eb3b500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-05T19:28:24.377 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.370+0000 7fa261c50700 1 -- 172.21.15.167:0/1384687092 --> v2:172.21.15.118:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55841eb3bcd0 con 0x55841eb3c410 2023-10-05T19:28:24.377 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.370+0000 7fa261c50700 1 --2- 172.21.15.167:0/1384687092 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55841eb3c410 0x55841eb3afc0 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7fa25c00a5c0 tx=0x7fa25c005e00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c2159fedab1a1ab9 server_cookie=0 in_seq=0 out_seq=0 2023-10-05T19:28:24.377 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.370+0000 7fa260c4e700 1 -- 172.21.15.167:0/1384687092 <== mon.0 v2:172.21.15.118:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fa25c00e510 con 0x55841eb3c410 2023-10-05T19:28:24.378 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.370+0000 7fa260c4e700 1 -- 172.21.15.167:0/1384687092 <== mon.0 v2:172.21.15.118:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa25c00e670 con 0x55841eb3c410 2023-10-05T19:28:24.378 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.370+0000 7fa260c4e700 1 -- 172.21.15.167:0/1384687092 <== mon.0 v2:172.21.15.118:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fa25c00e920 con 0x55841eb3c410 2023-10-05T19:28:24.379 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.370+0000 7fa26411be40 1 -- 172.21.15.167:0/1384687092 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55841eb3c410 msgr2=0x55841eb3afc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-05T19:28:24.379 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.370+0000 7fa26411be40 1 --2- 172.21.15.167:0/1384687092 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55841eb3c410 0x55841eb3afc0 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7fa25c00a5c0 tx=0x7fa25c005e00 comp rx=0 tx=0).stop 2023-10-05T19:28:24.379 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.374+0000 7fa26411be40 1 -- 172.21.15.167:0/1384687092 shutdown_connections 2023-10-05T19:28:24.379 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.374+0000 7fa26411be40 1 --2- 172.21.15.167:0/1384687092 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55841eb3c410 0x55841eb3afc0 unknown :-1 s=CLOSED pgs=153 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-05T19:28:24.379 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.374+0000 7fa26411be40 1 -- 172.21.15.167:0/1384687092 >> 172.21.15.167:0/1384687092 conn(0x55841eb36720 msgr2=0x55841eb38b30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-05T19:28:24.379 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.374+0000 7fa26411be40 1 -- 172.21.15.167:0/1384687092 shutdown_connections 2023-10-05T19:28:24.380 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.374+0000 7fa26411be40 1 -- 172.21.15.167:0/1384687092 wait complete. 2023-10-05T19:28:24.380 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.374+0000 7fa26411be40 1 Processor -- start 2023-10-05T19:28:24.380 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.374+0000 7fa26411be40 1 -- start start 2023-10-05T19:28:24.380 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.374+0000 7fa26411be40 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55841eb3c410 0x55841eb5d150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-05T19:28:24.380 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.374+0000 7fa26411be40 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x55841ea09e30 con 0x55841eb3c410 2023-10-05T19:28:24.382 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.374+0000 7fa261c50700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55841eb3c410 0x55841eb5d150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-05T19:28:24.382 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.374+0000 7fa261c50700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55841eb3c410 0x55841eb5d150 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.118:3300/0 says I am v2:172.21.15.167:56932/0 (socket says 172.21.15.167:56932) 2023-10-05T19:28:24.382 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.374+0000 7fa261c50700 1 -- 172.21.15.167:0/2052744808 learned_addr learned my addr 172.21.15.167:0/2052744808 (peer_addr_for_me v2:172.21.15.167:0/0) 2023-10-05T19:28:24.382 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.374+0000 7fa261c50700 1 -- 172.21.15.167:0/2052744808 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa25c00a180 con 0x55841eb3c410 2023-10-05T19:28:24.382 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.374+0000 7fa261c50700 1 --2- 172.21.15.167:0/2052744808 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55841eb3c410 0x55841eb5d150 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x55841eb3bca0 tx=0x7fa25c013000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fb479629448c7862 server_cookie=0 in_seq=0 out_seq=0 2023-10-05T19:28:24.382 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.374+0000 7fa2537fe700 1 -- 172.21.15.167:0/2052744808 <== mon.0 v2:172.21.15.118:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fa25c0032f0 con 0x55841eb3c410 2023-10-05T19:28:24.382 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.374+0000 7fa2537fe700 1 -- 172.21.15.167:0/2052744808 <== mon.0 v2:172.21.15.118:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa25c00e7e0 con 0x55841eb3c410 2023-10-05T19:28:24.382 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.374+0000 7fa2537fe700 1 -- 172.21.15.167:0/2052744808 <== mon.0 v2:172.21.15.118:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fa25c00ead0 con 0x55841eb3c410 2023-10-05T19:28:24.382 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.374+0000 7fa26411be40 1 -- 172.21.15.167:0/2052744808 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55841eb3c410 msgr2=0x55841eb5d150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-05T19:28:24.382 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.374+0000 7fa26411be40 1 --2- 172.21.15.167:0/2052744808 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55841eb3c410 0x55841eb5d150 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x55841eb3bca0 tx=0x7fa25c013000 comp rx=0 tx=0).stop 2023-10-05T19:28:24.383 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.374+0000 7fa26411be40 1 -- 172.21.15.167:0/2052744808 shutdown_connections 2023-10-05T19:28:24.383 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.374+0000 7fa26411be40 1 --2- 172.21.15.167:0/2052744808 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55841eb3c410 0x55841eb5d150 unknown :-1 s=CLOSED pgs=154 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-05T19:28:24.383 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.374+0000 7fa26411be40 1 -- 172.21.15.167:0/2052744808 >> 172.21.15.167:0/2052744808 conn(0x55841eb36720 msgr2=0x55841eb58d10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-05T19:28:24.383 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.374+0000 7fa26411be40 1 -- 172.21.15.167:0/2052744808 shutdown_connections 2023-10-05T19:28:24.383 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.374+0000 7fa26411be40 1 -- 172.21.15.167:0/2052744808 wait complete. 2023-10-05T19:28:24.383 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.374+0000 7fa26411be40 1 Processor -- start 2023-10-05T19:28:24.383 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.374+0000 7fa26411be40 1 -- start start 2023-10-05T19:28:24.383 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.374+0000 7fa26411be40 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55841eb3c410 0x55841eb63d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-05T19:28:24.383 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.374+0000 7fa26411be40 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x55841e9dced0 con 0x55841eb3c410 2023-10-05T19:28:24.384 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.378+0000 7fa261c50700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55841eb3c410 0x55841eb63d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-05T19:28:24.385 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.378+0000 7fa261c50700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55841eb3c410 0x55841eb63d80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.118:3300/0 says I am v2:172.21.15.167:56938/0 (socket says 172.21.15.167:56938) 2023-10-05T19:28:24.385 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.378+0000 7fa261c50700 1 -- 172.21.15.167:0/3175752868 learned_addr learned my addr 172.21.15.167:0/3175752868 (peer_addr_for_me v2:172.21.15.167:0/0) 2023-10-05T19:28:24.385 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.378+0000 7fa261c50700 1 -- 172.21.15.167:0/3175752868 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55841eb3bcd0 con 0x55841eb3c410 2023-10-05T19:28:24.385 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.378+0000 7fa261c50700 1 --2- 172.21.15.167:0/3175752868 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55841eb3c410 0x55841eb63d80 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7fa25c005860 tx=0x7fa25c01c000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-05T19:28:24.385 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.378+0000 7fa251ffb700 1 -- 172.21.15.167:0/3175752868 <== mon.0 v2:172.21.15.118:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fa25c0109f0 con 0x55841eb3c410 2023-10-05T19:28:24.385 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.378+0000 7fa251ffb700 1 -- 172.21.15.167:0/3175752868 <== mon.0 v2:172.21.15.118:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa25c010b70 con 0x55841eb3c410 2023-10-05T19:28:24.385 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.378+0000 7fa26411be40 1 -- 172.21.15.167:0/3175752868 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x55841eb642c0 con 0x55841eb3c410 2023-10-05T19:28:24.385 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.378+0000 7fa251ffb700 1 -- 172.21.15.167:0/3175752868 <== mon.0 v2:172.21.15.118:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fa25c010e60 con 0x55841eb3c410 2023-10-05T19:28:24.385 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.378+0000 7fa26411be40 1 -- 172.21.15.167:0/3175752868 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55841eb64700 con 0x55841eb3c410 2023-10-05T19:28:24.385 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.378+0000 7fa251ffb700 1 -- 172.21.15.167:0/3175752868 <== mon.0 v2:172.21.15.118:3300/0 4 ==== mgrmap(e 5) v1 ==== 40329+0+0 (secure 0 0 0) 0x7fa25c01c430 con 0x55841eb3c410 2023-10-05T19:28:24.386 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.378+0000 7fa26411be40 20 rados->read ofs=0 len=0 2023-10-05T19:28:24.386 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.378+0000 7fa251ffb700 1 --2- 172.21.15.167:0/3175752868 >> [v2:172.21.15.167:6824/29889,v1:172.21.15.167:6825/29889] conn(0x7fa244030ca0 0x7fa244033150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-05T19:28:24.387 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.378+0000 7fa251ffb700 1 -- 172.21.15.167:0/3175752868 <== mon.0 v2:172.21.15.118:3300/0 5 ==== osd_map(30..30 src has 1..30) v4 ==== 6057+0+0 (secure 0 0 0) 0x7fa25c046220 con 0x55841eb3c410 2023-10-05T19:28:24.387 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.378+0000 7fa26144f700 1 --2- 172.21.15.167:0/3175752868 >> [v2:172.21.15.167:6824/29889,v1:172.21.15.167:6825/29889] conn(0x7fa244030ca0 0x7fa244033150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-05T19:28:24.387 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.382+0000 7fa26411be40 1 --2- 172.21.15.167:0/3175752868 >> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] conn(0x55841eb3c7e0 0x55841eb6f810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-05T19:28:24.387 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.382+0000 7fa26411be40 1 -- 172.21.15.167:0/3175752868 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:1 5.12 5:49953fa1:::default.realm:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x55841eb6fec0 con 0x55841eb3c7e0 2023-10-05T19:28:24.387 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.382+0000 7fa262451700 1 --2- 172.21.15.167:0/3175752868 >> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] conn(0x55841eb3c7e0 0x55841eb6f810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-05T19:28:24.387 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.382+0000 7fa26144f700 1 --2- 172.21.15.167:0/3175752868 >> [v2:172.21.15.167:6824/29889,v1:172.21.15.167:6825/29889] conn(0x7fa244030ca0 0x7fa244033150 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7fa25800b2b0 tx=0x7fa258001490 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-05T19:28:24.388 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.382+0000 7fa262451700 1 --2- 172.21.15.167:0/3175752868 >> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] conn(0x55841eb3c7e0 0x55841eb6f810 crc :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-05T19:28:24.389 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.382+0000 7fa262451700 1 -- 172.21.15.167:0/3175752868 <== osd.1 v2:172.21.15.167:6800/30035 1 ==== osd_op_reply(1 default.realm [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 ==== 157+0+0 (crc 0 0 0) 0x7fa25400a500 con 0x55841eb3c7e0 2023-10-05T19:28:24.389 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.382+0000 7fa26411be40 20 rados_obj.operate() r=-2 bl.length=0 2023-10-05T19:28:24.389 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.382+0000 7fa26411be40 20 realm 2023-10-05T19:28:24.389 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.382+0000 7fa26411be40 20 rados->read ofs=0 len=0 2023-10-05T19:28:24.389 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.382+0000 7fa26411be40 1 -- 172.21.15.167:0/3175752868 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:2 5.12 5:49953fa1:::default.realm:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x55841eb6ce70 con 0x55841eb3c7e0 2023-10-05T19:28:24.389 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.382+0000 7fa262451700 1 -- 172.21.15.167:0/3175752868 <== osd.1 v2:172.21.15.167:6800/30035 2 ==== osd_op_reply(2 default.realm [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 ==== 157+0+0 (crc 0 0 0) 0x7fa25400a500 con 0x55841eb3c7e0 2023-10-05T19:28:24.389 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.382+0000 7fa26411be40 20 rados_obj.operate() r=-2 bl.length=0 2023-10-05T19:28:24.389 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.382+0000 7fa26411be40 4 RGWPeriod::init failed to init realm id : (2) No such file or directory 2023-10-05T19:28:24.389 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.382+0000 7fa26411be40 20 rados->read ofs=0 len=0 2023-10-05T19:28:24.389 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.382+0000 7fa26411be40 1 -- 172.21.15.167:0/3175752868 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:3 5.12 5:49953fa1:::default.realm:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x55841eb709a0 con 0x55841eb3c7e0 2023-10-05T19:28:24.390 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.382+0000 7fa262451700 1 -- 172.21.15.167:0/3175752868 <== osd.1 v2:172.21.15.167:6800/30035 3 ==== osd_op_reply(3 default.realm [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 ==== 157+0+0 (crc 0 0 0) 0x7fa25400a500 con 0x55841eb3c7e0 2023-10-05T19:28:24.390 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.382+0000 7fa26411be40 20 rados_obj.operate() r=-2 bl.length=0 2023-10-05T19:28:24.390 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.382+0000 7fa26411be40 20 rados->read ofs=0 len=0 2023-10-05T19:28:24.390 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.382+0000 7fa26411be40 1 -- 172.21.15.167:0/3175752868 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:4 5.f 5:f43b6ece:::zone_names.default:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x55841eb705b0 con 0x55841eb3c7e0 2023-10-05T19:28:24.390 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.382+0000 7fa262451700 1 -- 172.21.15.167:0/3175752868 <== osd.1 v2:172.21.15.167:6800/30035 4 ==== osd_op_reply(4 zone_names.default [read 0~46 out=46b] v0'0 uv1 ondisk = 0) v8 ==== 162+0+46 (crc 0 0 0) 0x7fa25400a500 con 0x55841eb3c7e0 2023-10-05T19:28:24.390 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.382+0000 7fa26411be40 20 rados_obj.operate() r=0 bl.length=46 2023-10-05T19:28:24.390 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.382+0000 7fa26411be40 20 rados->read ofs=0 len=0 2023-10-05T19:28:24.390 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.382+0000 7fa26411be40 1 -- 172.21.15.167:0/3175752868 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:5 5.1d 5:b8374eb4:::zone_info.9a9df965-6612-45d4-a45f-6e3c889bacc2:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x55841eb71830 con 0x55841eb3c7e0 2023-10-05T19:28:24.391 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.382+0000 7fa262451700 1 -- 172.21.15.167:0/3175752868 <== osd.1 v2:172.21.15.167:6800/30035 5 ==== osd_op_reply(5 zone_info.9a9df965-6612-45d4-a45f-6e3c889bacc2 [read 0~874 out=874b] v0'0 uv1 ondisk = 0) v8 ==== 190+0+874 (crc 0 0 0) 0x7fa25400a500 con 0x55841eb3c7e0 2023-10-05T19:28:24.391 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.382+0000 7fa26411be40 20 rados_obj.operate() r=0 bl.length=874 2023-10-05T19:28:24.391 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.382+0000 7fa26411be40 20 searching for the correct realm 2023-10-05T19:28:24.391 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.382+0000 7fa26411be40 1 --2- 172.21.15.167:0/3175752868 >> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] conn(0x55841eb735f0 0x55841eb759d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-05T19:28:24.391 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.382+0000 7fa26411be40 1 -- 172.21.15.167:0/3175752868 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:6 5.0 5:00000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x55841eb75f10 con 0x55841eb735f0 2023-10-05T19:28:24.392 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.386+0000 7fa261c50700 1 --2- 172.21.15.167:0/3175752868 >> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] conn(0x55841eb735f0 0x55841eb759d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-05T19:28:24.392 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.386+0000 7fa261c50700 1 --2- 172.21.15.167:0/3175752868 >> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] conn(0x55841eb735f0 0x55841eb759d0 crc :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-05T19:28:24.393 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.386+0000 7fa261c50700 1 -- 172.21.15.167:0/3175752868 <== osd.3 v2:172.21.15.167:6808/30036 1 ==== osd_op_reply(6 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fa25c02b070 con 0x55841eb735f0 2023-10-05T19:28:24.393 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.386+0000 7fa261c50700 1 --2- 172.21.15.167:0/3175752868 >> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] conn(0x7fa25c010070 0x7fa25c01c6e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-05T19:28:24.393 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.386+0000 7fa261c50700 1 -- 172.21.15.167:0/3175752868 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:7 5.10 5:08000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x55841eb75f10 con 0x7fa25c010070 2023-10-05T19:28:24.393 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.386+0000 7fa26144f700 1 --2- 172.21.15.167:0/3175752868 >> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] conn(0x7fa25c010070 0x7fa25c01c6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-05T19:28:24.393 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.386+0000 7fa26144f700 1 --2- 172.21.15.167:0/3175752868 >> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] conn(0x7fa25c010070 0x7fa25c01c6e0 crc :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-05T19:28:24.394 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.390+0000 7fa26144f700 1 -- 172.21.15.167:0/3175752868 <== osd.2 v2:172.21.15.167:6816/30037 1 ==== osd_op_reply(7 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fa25800bc70 con 0x7fa25c010070 2023-10-05T19:28:24.394 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.390+0000 7fa26144f700 1 --2- 172.21.15.167:0/3175752868 >> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] conn(0x7fa25800f8d0 0x7fa25800e360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-05T19:28:24.394 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.390+0000 7fa26144f700 1 -- 172.21.15.167:0/3175752868 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:8 5.8 5:10000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x55841eb75f10 con 0x7fa25800f8d0 2023-10-05T19:28:24.395 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.390+0000 7fa262451700 1 --2- 172.21.15.167:0/3175752868 >> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] conn(0x7fa25800f8d0 0x7fa25800e360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-05T19:28:24.395 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.390+0000 7fa262451700 1 --2- 172.21.15.167:0/3175752868 >> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] conn(0x7fa25800f8d0 0x7fa25800e360 crc :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-05T19:28:24.396 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.390+0000 7fa262451700 1 -- 172.21.15.167:0/3175752868 <== osd.0 v2:172.21.15.118:6800/27690 1 ==== osd_op_reply(8 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fa254014040 con 0x7fa25800f8d0 2023-10-05T19:28:24.396 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.390+0000 7fa262451700 1 -- 172.21.15.167:0/3175752868 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:9 5.18 5:18000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x55841eb75f10 con 0x55841eb3c7e0 2023-10-05T19:28:24.397 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.390+0000 7fa262451700 1 -- 172.21.15.167:0/3175752868 <== osd.1 v2:172.21.15.167:6800/30035 6 ==== osd_op_reply(9 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fa254014040 con 0x55841eb3c7e0 2023-10-05T19:28:24.397 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.390+0000 7fa262451700 1 -- 172.21.15.167:0/3175752868 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:10 5.4 5:20000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x55841eb75f10 con 0x55841eb3c7e0 2023-10-05T19:28:24.397 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.390+0000 7fa262451700 1 -- 172.21.15.167:0/3175752868 <== osd.1 v2:172.21.15.167:6800/30035 7 ==== osd_op_reply(10 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fa254014040 con 0x55841eb3c7e0 2023-10-05T19:28:24.397 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.390+0000 7fa262451700 1 -- 172.21.15.167:0/3175752868 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:11 5.14 5:28000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x55841eb75f10 con 0x7fa25800f8d0 2023-10-05T19:28:24.398 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.394+0000 7fa262451700 1 -- 172.21.15.167:0/3175752868 <== osd.0 v2:172.21.15.118:6800/27690 2 ==== osd_op_reply(11 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fa254014040 con 0x7fa25800f8d0 2023-10-05T19:28:24.398 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.394+0000 7fa262451700 1 -- 172.21.15.167:0/3175752868 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:12 5.c 5:30000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x55841eb75f10 con 0x7fa25800f8d0 2023-10-05T19:28:24.399 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.394+0000 7fa262451700 1 -- 172.21.15.167:0/3175752868 <== osd.0 v2:172.21.15.118:6800/27690 3 ==== osd_op_reply(12 [pgnls start_epoch 30 out=112b] v23'1 uv1 ondisk = 1) v8 ==== 144+0+112 (crc 0 0 0) 0x7fa254014040 con 0x7fa25800f8d0 2023-10-05T19:28:24.399 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.394+0000 7fa262451700 1 -- 172.21.15.167:0/3175752868 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:13 5.1c 5:38000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x55841eb75f10 con 0x7fa25c010070 2023-10-05T19:28:24.400 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.394+0000 7fa26144f700 1 -- 172.21.15.167:0/3175752868 <== osd.2 v2:172.21.15.167:6816/30037 2 ==== osd_op_reply(13 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fa25800bc70 con 0x7fa25c010070 2023-10-05T19:28:24.400 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.394+0000 7fa26144f700 1 -- 172.21.15.167:0/3175752868 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:14 5.2 5:40000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x55841eb75f10 con 0x55841eb735f0 2023-10-05T19:28:24.400 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.394+0000 7fa261c50700 1 -- 172.21.15.167:0/3175752868 <== osd.3 v2:172.21.15.167:6808/30036 2 ==== osd_op_reply(14 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fa25c040040 con 0x55841eb735f0 2023-10-05T19:28:24.401 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.394+0000 7fa261c50700 1 -- 172.21.15.167:0/3175752868 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:15 5.12 5:48000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x55841eb75f10 con 0x55841eb3c7e0 2023-10-05T19:28:24.401 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.394+0000 7fa262451700 1 -- 172.21.15.167:0/3175752868 <== osd.1 v2:172.21.15.167:6800/30035 8 ==== osd_op_reply(15 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fa254014040 con 0x55841eb3c7e0 2023-10-05T19:28:24.401 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.394+0000 7fa262451700 1 -- 172.21.15.167:0/3175752868 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:16 5.a 5:50000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x55841eb75f10 con 0x7fa25800f8d0 2023-10-05T19:28:24.402 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.398+0000 7fa262451700 1 -- 172.21.15.167:0/3175752868 <== osd.0 v2:172.21.15.118:6800/27690 4 ==== osd_op_reply(16 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fa254022040 con 0x7fa25800f8d0 2023-10-05T19:28:24.402 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.398+0000 7fa262451700 1 -- 172.21.15.167:0/3175752868 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:17 5.1a 5:58000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x55841eb75f10 con 0x55841eb3c7e0 2023-10-05T19:28:24.403 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.398+0000 7fa262451700 1 -- 172.21.15.167:0/3175752868 <== osd.1 v2:172.21.15.167:6800/30035 9 ==== osd_op_reply(17 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fa254022040 con 0x55841eb3c7e0 2023-10-05T19:28:24.403 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.398+0000 7fa262451700 1 -- 172.21.15.167:0/3175752868 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:18 5.6 5:60000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x55841eb75f10 con 0x7fa25c010070 2023-10-05T19:28:24.403 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.398+0000 7fa26144f700 1 -- 172.21.15.167:0/3175752868 <== osd.2 v2:172.21.15.167:6816/30037 3 ==== osd_op_reply(18 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fa25800bc70 con 0x7fa25c010070 2023-10-05T19:28:24.403 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.398+0000 7fa26144f700 1 -- 172.21.15.167:0/3175752868 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:19 5.16 5:68000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x55841eb75f10 con 0x55841eb735f0 2023-10-05T19:28:24.404 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.398+0000 7fa261c50700 1 -- 172.21.15.167:0/3175752868 <== osd.3 v2:172.21.15.167:6808/30036 3 ==== osd_op_reply(19 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fa25c040040 con 0x55841eb735f0 2023-10-05T19:28:24.404 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.398+0000 7fa261c50700 1 -- 172.21.15.167:0/3175752868 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:20 5.e 5:70000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x55841eb75f10 con 0x55841eb735f0 2023-10-05T19:28:24.405 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.398+0000 7fa261c50700 1 -- 172.21.15.167:0/3175752868 <== osd.3 v2:172.21.15.167:6808/30036 4 ==== osd_op_reply(20 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fa25c040040 con 0x55841eb735f0 2023-10-05T19:28:24.405 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.398+0000 7fa261c50700 1 -- 172.21.15.167:0/3175752868 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:21 5.1e 5:78000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x55841eb75f10 con 0x7fa25800f8d0 2023-10-05T19:28:24.406 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.398+0000 7fa262451700 1 -- 172.21.15.167:0/3175752868 <== osd.0 v2:172.21.15.118:6800/27690 5 ==== osd_op_reply(21 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fa254022040 con 0x7fa25800f8d0 2023-10-05T19:28:24.406 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.398+0000 7fa262451700 1 -- 172.21.15.167:0/3175752868 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:22 5.1 5:80000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x55841eb75f10 con 0x55841eb735f0 2023-10-05T19:28:24.407 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.402+0000 7fa261c50700 1 -- 172.21.15.167:0/3175752868 <== osd.3 v2:172.21.15.167:6808/30036 5 ==== osd_op_reply(22 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fa25c040040 con 0x55841eb735f0 2023-10-05T19:28:24.407 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.402+0000 7fa261c50700 1 -- 172.21.15.167:0/3175752868 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:23 5.11 5:88000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x55841eb75f10 con 0x55841eb735f0 2023-10-05T19:28:24.407 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.402+0000 7fa261c50700 1 -- 172.21.15.167:0/3175752868 <== osd.3 v2:172.21.15.167:6808/30036 6 ==== osd_op_reply(23 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fa25c040040 con 0x55841eb735f0 2023-10-05T19:28:24.407 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.402+0000 7fa261c50700 1 -- 172.21.15.167:0/3175752868 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:24 5.9 5:90000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x55841eb75f10 con 0x55841eb3c7e0 2023-10-05T19:28:24.408 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.402+0000 7fa262451700 1 -- 172.21.15.167:0/3175752868 <== osd.1 v2:172.21.15.167:6800/30035 10 ==== osd_op_reply(24 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fa254022040 con 0x55841eb3c7e0 2023-10-05T19:28:24.408 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.402+0000 7fa262451700 1 -- 172.21.15.167:0/3175752868 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:25 5.19 5:98000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x55841eb75f10 con 0x55841eb3c7e0 2023-10-05T19:28:24.409 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.402+0000 7fa262451700 1 -- 172.21.15.167:0/3175752868 <== osd.1 v2:172.21.15.167:6800/30035 11 ==== osd_op_reply(25 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fa254022040 con 0x55841eb3c7e0 2023-10-05T19:28:24.409 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.402+0000 7fa262451700 1 -- 172.21.15.167:0/3175752868 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:26 5.5 5:a0000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x55841eb75f10 con 0x7fa25800f8d0 2023-10-05T19:28:24.410 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.406+0000 7fa262451700 1 -- 172.21.15.167:0/3175752868 <== osd.0 v2:172.21.15.118:6800/27690 6 ==== osd_op_reply(26 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fa254022040 con 0x7fa25800f8d0 2023-10-05T19:28:24.410 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.406+0000 7fa262451700 1 -- 172.21.15.167:0/3175752868 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:27 5.15 5:a8000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x55841eb75f10 con 0x55841eb735f0 2023-10-05T19:28:24.410 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.406+0000 7fa261c50700 1 -- 172.21.15.167:0/3175752868 <== osd.3 v2:172.21.15.167:6808/30036 7 ==== osd_op_reply(27 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fa25c040040 con 0x55841eb735f0 2023-10-05T19:28:24.411 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.406+0000 7fa261c50700 1 -- 172.21.15.167:0/3175752868 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:28 5.d 5:b0000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x55841eb75f10 con 0x7fa25c010070 2023-10-05T19:28:24.411 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.406+0000 7fa26144f700 1 -- 172.21.15.167:0/3175752868 <== osd.2 v2:172.21.15.167:6816/30037 4 ==== osd_op_reply(28 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fa25800bc70 con 0x7fa25c010070 2023-10-05T19:28:24.411 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.406+0000 7fa26144f700 1 -- 172.21.15.167:0/3175752868 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:29 5.1d 5:b8000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x55841eb75f10 con 0x55841eb3c7e0 2023-10-05T19:28:24.412 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.406+0000 7fa262451700 1 -- 172.21.15.167:0/3175752868 <== osd.1 v2:172.21.15.167:6800/30035 12 ==== osd_op_reply(29 [pgnls start_epoch 30 out=107b] v23'1 uv1 ondisk = 1) v8 ==== 144+0+107 (crc 0 0 0) 0x7fa254022040 con 0x55841eb3c7e0 2023-10-05T19:28:24.412 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.406+0000 7fa262451700 1 -- 172.21.15.167:0/3175752868 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:30 5.3 5:c0000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x55841eb75f10 con 0x7fa25800f8d0 2023-10-05T19:28:24.413 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.406+0000 7fa262451700 1 -- 172.21.15.167:0/3175752868 <== osd.0 v2:172.21.15.118:6800/27690 7 ==== osd_op_reply(30 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fa254022040 con 0x7fa25800f8d0 2023-10-05T19:28:24.413 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.406+0000 7fa262451700 1 -- 172.21.15.167:0/3175752868 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:31 5.13 5:c8000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x55841eb75f10 con 0x55841eb735f0 2023-10-05T19:28:24.414 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.410+0000 7fa261c50700 1 -- 172.21.15.167:0/3175752868 <== osd.3 v2:172.21.15.167:6808/30036 8 ==== osd_op_reply(31 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fa25c040040 con 0x55841eb735f0 2023-10-05T19:28:24.414 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.410+0000 7fa261c50700 1 -- 172.21.15.167:0/3175752868 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:32 5.b 5:d0000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x55841eb75f10 con 0x7fa25800f8d0 2023-10-05T19:28:24.415 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.410+0000 7fa262451700 1 -- 172.21.15.167:0/3175752868 <== osd.0 v2:172.21.15.118:6800/27690 8 ==== osd_op_reply(32 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fa254022040 con 0x7fa25800f8d0 2023-10-05T19:28:24.415 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.410+0000 7fa262451700 1 -- 172.21.15.167:0/3175752868 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:33 5.1b 5:d8000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x55841eb75f10 con 0x7fa25c010070 2023-10-05T19:28:24.416 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.410+0000 7fa26144f700 1 -- 172.21.15.167:0/3175752868 <== osd.2 v2:172.21.15.167:6816/30037 5 ==== osd_op_reply(33 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fa25800bc70 con 0x7fa25c010070 2023-10-05T19:28:24.416 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.410+0000 7fa26144f700 1 -- 172.21.15.167:0/3175752868 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:34 5.7 5:e0000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x55841eb75f10 con 0x55841eb735f0 2023-10-05T19:28:24.417 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.410+0000 7fa261c50700 1 -- 172.21.15.167:0/3175752868 <== osd.3 v2:172.21.15.167:6808/30036 9 ==== osd_op_reply(34 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fa25c040040 con 0x55841eb735f0 2023-10-05T19:28:24.417 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.410+0000 7fa261c50700 1 -- 172.21.15.167:0/3175752868 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:35 5.17 5:e8000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x55841eb75f10 con 0x7fa25800f8d0 2023-10-05T19:28:24.418 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.414+0000 7fa262451700 1 -- 172.21.15.167:0/3175752868 <== osd.0 v2:172.21.15.118:6800/27690 9 ==== osd_op_reply(35 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fa254022040 con 0x7fa25800f8d0 2023-10-05T19:28:24.418 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.414+0000 7fa262451700 1 -- 172.21.15.167:0/3175752868 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:36 5.f 5:f0000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x55841eb75f10 con 0x55841eb3c7e0 2023-10-05T19:28:24.419 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.414+0000 7fa262451700 1 -- 172.21.15.167:0/3175752868 <== osd.1 v2:172.21.15.167:6800/30035 13 ==== osd_op_reply(36 [pgnls start_epoch 30 out=115b] v23'2 uv2 ondisk = 1) v8 ==== 144+0+115 (crc 0 0 0) 0x7fa25402f080 con 0x55841eb3c7e0 2023-10-05T19:28:24.419 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.414+0000 7fa262451700 1 -- 172.21.15.167:0/3175752868 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:37 5.1f 5:f8000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x55841eb75f10 con 0x7fa25c010070 2023-10-05T19:28:24.420 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.414+0000 7fa26144f700 1 -- 172.21.15.167:0/3175752868 <== osd.2 v2:172.21.15.167:6816/30037 6 ==== osd_op_reply(37 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fa25800bc70 con 0x7fa25c010070 2023-10-05T19:28:24.420 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.414+0000 7fa26411be40 20 RGWRados::pool_iterate: got zonegroup_info.4f1f719d-9648-417b-b10c-e5d939825419 2023-10-05T19:28:24.420 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.414+0000 7fa26411be40 20 RGWRados::pool_iterate: got zone_info.9a9df965-6612-45d4-a45f-6e3c889bacc2 2023-10-05T19:28:24.420 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.414+0000 7fa26411be40 20 RGWRados::pool_iterate: got zone_names.default 2023-10-05T19:28:24.420 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.414+0000 7fa26411be40 20 RGWRados::pool_iterate: got zonegroups_names.default 2023-10-05T19:28:24.420 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.414+0000 7fa26411be40 20 rados->read ofs=0 len=0 2023-10-05T19:28:24.420 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.414+0000 7fa26411be40 1 -- 172.21.15.167:0/3175752868 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:38 5.12 5:49953fa1:::default.realm:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x55841eb723a0 con 0x55841eb3c7e0 2023-10-05T19:28:24.421 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.414+0000 7fa262451700 1 -- 172.21.15.167:0/3175752868 <== osd.1 v2:172.21.15.167:6800/30035 14 ==== osd_op_reply(38 default.realm [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 ==== 157+0+0 (crc 0 0 0) 0x7fa25402f080 con 0x55841eb3c7e0 2023-10-05T19:28:24.421 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.414+0000 7fa26411be40 20 rados_obj.operate() r=-2 bl.length=0 2023-10-05T19:28:24.421 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.414+0000 7fa26411be40 20 rados->read ofs=0 len=0 2023-10-05T19:28:24.421 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.414+0000 7fa26411be40 1 -- 172.21.15.167:0/3175752868 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:39 5.f 5:f4c53578:::zonegroups_names.default:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x55841eb72c90 con 0x55841eb3c7e0 2023-10-05T19:28:24.422 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.414+0000 7fa262451700 1 -- 172.21.15.167:0/3175752868 <== osd.1 v2:172.21.15.167:6800/30035 15 ==== osd_op_reply(39 zonegroups_names.default [read 0~46 out=46b] v0'0 uv2 ondisk = 0) v8 ==== 168+0+46 (crc 0 0 0) 0x7fa25402f080 con 0x55841eb3c7e0 2023-10-05T19:28:24.422 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.414+0000 7fa26411be40 20 rados_obj.operate() r=0 bl.length=46 2023-10-05T19:28:24.422 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.414+0000 7fa26411be40 20 rados->read ofs=0 len=0 2023-10-05T19:28:24.422 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.414+0000 7fa26411be40 1 -- 172.21.15.167:0/3175752868 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:40 5.c 5:30d299a5:::zonegroup_info.4f1f719d-9648-417b-b10c-e5d939825419:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x55841eb76350 con 0x7fa25800f8d0 2023-10-05T19:28:24.423 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.418+0000 7fa262451700 1 -- 172.21.15.167:0/3175752868 <== osd.0 v2:172.21.15.118:6800/27690 10 ==== osd_op_reply(40 zonegroup_info.4f1f719d-9648-417b-b10c-e5d939825419 [read 0~362 out=362b] v0'0 uv1 ondisk = 0) v8 ==== 195+0+362 (crc 0 0 0) 0x7fa25402f080 con 0x7fa25800f8d0 2023-10-05T19:28:24.423 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.418+0000 7fa26411be40 20 rados_obj.operate() r=0 bl.length=362 2023-10-05T19:28:24.423 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.418+0000 7fa26411be40 20 rados->read ofs=0 len=0 2023-10-05T19:28:24.423 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.418+0000 7fa26411be40 1 -- 172.21.15.167:0/3175752868 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:41 5.13 5:cbbf6c60:::converted:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x55841eb76350 con 0x55841eb735f0 2023-10-05T19:28:24.423 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.418+0000 7fa261c50700 1 -- 172.21.15.167:0/3175752868 <== osd.3 v2:172.21.15.167:6808/30036 10 ==== osd_op_reply(41 converted [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 ==== 153+0+0 (crc 0 0 0) 0x7fa25c040040 con 0x55841eb735f0 2023-10-05T19:28:24.423 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.418+0000 7fa26411be40 20 rados_obj.operate() r=-2 bl.length=0 2023-10-05T19:28:24.423 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.418+0000 7fa26411be40 20 rados->read ofs=0 len=0 2023-10-05T19:28:24.423 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.418+0000 7fa26411be40 1 -- 172.21.15.167:0/3175752868 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:42 5.12 5:49953fa1:::default.realm:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x55841eb76700 con 0x55841eb3c7e0 2023-10-05T19:28:24.424 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.418+0000 7fa262451700 1 -- 172.21.15.167:0/3175752868 <== osd.1 v2:172.21.15.167:6800/30035 16 ==== osd_op_reply(42 default.realm [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 ==== 157+0+0 (crc 0 0 0) 0x7fa25402f080 con 0x55841eb3c7e0 2023-10-05T19:28:24.424 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.418+0000 7fa26411be40 20 rados_obj.operate() r=-2 bl.length=0 2023-10-05T19:28:24.424 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.418+0000 7fa26411be40 20 rados->read ofs=0 len=0 2023-10-05T19:28:24.424 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.418+0000 7fa26411be40 1 -- 172.21.15.167:0/3175752868 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:43 5.f 5:f4c53578:::zonegroups_names.default:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x55841eb72890 con 0x55841eb3c7e0 2023-10-05T19:28:24.425 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.418+0000 7fa262451700 1 -- 172.21.15.167:0/3175752868 <== osd.1 v2:172.21.15.167:6800/30035 17 ==== osd_op_reply(43 zonegroups_names.default [read 0~46 out=46b] v0'0 uv2 ondisk = 0) v8 ==== 168+0+46 (crc 0 0 0) 0x7fa25402f080 con 0x55841eb3c7e0 2023-10-05T19:28:24.425 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.418+0000 7fa26411be40 20 rados_obj.operate() r=0 bl.length=46 2023-10-05T19:28:24.425 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.418+0000 7fa26411be40 1 -- 172.21.15.167:0/3175752868 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:44 5.0 5:00000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x55841eb72860 con 0x55841eb735f0 2023-10-05T19:28:24.427 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.422+0000 7fa261c50700 1 -- 172.21.15.167:0/3175752868 <== osd.3 v2:172.21.15.167:6808/30036 11 ==== osd_op_reply(44 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fa25c040040 con 0x55841eb735f0 2023-10-05T19:28:24.427 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.422+0000 7fa261c50700 1 -- 172.21.15.167:0/3175752868 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:45 5.10 5:08000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x55841eb72860 con 0x7fa25c010070 2023-10-05T19:28:24.428 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.422+0000 7fa26144f700 1 -- 172.21.15.167:0/3175752868 <== osd.2 v2:172.21.15.167:6816/30037 7 ==== osd_op_reply(45 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fa25800bc70 con 0x7fa25c010070 2023-10-05T19:28:24.428 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.422+0000 7fa26144f700 1 -- 172.21.15.167:0/3175752868 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:46 5.8 5:10000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x55841eb72860 con 0x7fa25800f8d0 2023-10-05T19:28:24.429 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.422+0000 7fa262451700 1 -- 172.21.15.167:0/3175752868 <== osd.0 v2:172.21.15.118:6800/27690 11 ==== osd_op_reply(46 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fa25402f080 con 0x7fa25800f8d0 2023-10-05T19:28:24.429 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.422+0000 7fa262451700 1 -- 172.21.15.167:0/3175752868 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:47 5.18 5:18000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x55841eb72c90 con 0x55841eb3c7e0 2023-10-05T19:28:24.430 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.422+0000 7fa262451700 1 -- 172.21.15.167:0/3175752868 <== osd.1 v2:172.21.15.167:6800/30035 18 ==== osd_op_reply(47 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fa25402f080 con 0x55841eb3c7e0 2023-10-05T19:28:24.430 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.426+0000 7fa262451700 1 -- 172.21.15.167:0/3175752868 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:48 5.4 5:20000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x55841eb72c90 con 0x55841eb3c7e0 2023-10-05T19:28:24.430 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.426+0000 7fa262451700 1 -- 172.21.15.167:0/3175752868 <== osd.1 v2:172.21.15.167:6800/30035 19 ==== osd_op_reply(48 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fa25402f080 con 0x55841eb3c7e0 2023-10-05T19:28:24.430 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.426+0000 7fa262451700 1 -- 172.21.15.167:0/3175752868 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:49 5.14 5:28000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x55841eb72c90 con 0x7fa25800f8d0 2023-10-05T19:28:24.432 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.426+0000 7fa262451700 1 -- 172.21.15.167:0/3175752868 <== osd.0 v2:172.21.15.118:6800/27690 12 ==== osd_op_reply(49 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fa25402f080 con 0x7fa25800f8d0 2023-10-05T19:28:24.432 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.426+0000 7fa262451700 1 -- 172.21.15.167:0/3175752868 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:50 5.c 5:30000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x55841eb72c90 con 0x7fa25800f8d0 2023-10-05T19:28:24.433 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.426+0000 7fa262451700 1 -- 172.21.15.167:0/3175752868 <== osd.0 v2:172.21.15.118:6800/27690 13 ==== osd_op_reply(50 [pgnls start_epoch 30 out=112b] v23'1 uv1 ondisk = 1) v8 ==== 144+0+112 (crc 0 0 0) 0x7fa25402f080 con 0x7fa25800f8d0 2023-10-05T19:28:24.433 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.426+0000 7fa262451700 1 -- 172.21.15.167:0/3175752868 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:51 5.1c 5:38000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x55841eb72c90 con 0x7fa25c010070 2023-10-05T19:28:24.434 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.426+0000 7fa26144f700 1 -- 172.21.15.167:0/3175752868 <== osd.2 v2:172.21.15.167:6816/30037 8 ==== osd_op_reply(51 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fa25800bc70 con 0x7fa25c010070 2023-10-05T19:28:24.434 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.426+0000 7fa26144f700 1 -- 172.21.15.167:0/3175752868 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:52 5.2 5:40000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x55841eb72c90 con 0x55841eb735f0 2023-10-05T19:28:24.434 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.430+0000 7fa261c50700 1 -- 172.21.15.167:0/3175752868 <== osd.3 v2:172.21.15.167:6808/30036 12 ==== osd_op_reply(52 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fa25c040040 con 0x55841eb735f0 2023-10-05T19:28:24.434 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.430+0000 7fa261c50700 1 -- 172.21.15.167:0/3175752868 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:53 5.12 5:48000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x55841eb72c90 con 0x55841eb3c7e0 2023-10-05T19:28:24.435 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.430+0000 7fa262451700 1 -- 172.21.15.167:0/3175752868 <== osd.1 v2:172.21.15.167:6800/30035 20 ==== osd_op_reply(53 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fa25402f080 con 0x55841eb3c7e0 2023-10-05T19:28:24.436 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.430+0000 7fa262451700 1 -- 172.21.15.167:0/3175752868 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:54 5.a 5:50000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x55841eb72c90 con 0x7fa25800f8d0 2023-10-05T19:28:24.436 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.430+0000 7fa262451700 1 -- 172.21.15.167:0/3175752868 <== osd.0 v2:172.21.15.118:6800/27690 14 ==== osd_op_reply(54 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fa25402f080 con 0x7fa25800f8d0 2023-10-05T19:28:24.436 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.430+0000 7fa262451700 1 -- 172.21.15.167:0/3175752868 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:55 5.1a 5:58000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x55841eb72c90 con 0x55841eb3c7e0 2023-10-05T19:28:24.437 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.430+0000 7fa262451700 1 -- 172.21.15.167:0/3175752868 <== osd.1 v2:172.21.15.167:6800/30035 21 ==== osd_op_reply(55 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fa25402f080 con 0x55841eb3c7e0 2023-10-05T19:28:24.437 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.430+0000 7fa262451700 1 -- 172.21.15.167:0/3175752868 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:56 5.6 5:60000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x55841eb72c90 con 0x7fa25c010070 2023-10-05T19:28:24.438 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.434+0000 7fa26144f700 1 -- 172.21.15.167:0/3175752868 <== osd.2 v2:172.21.15.167:6816/30037 9 ==== osd_op_reply(56 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fa25800bc70 con 0x7fa25c010070 2023-10-05T19:28:24.438 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.434+0000 7fa26144f700 1 -- 172.21.15.167:0/3175752868 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:57 5.16 5:68000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x55841eb72c90 con 0x55841eb735f0 2023-10-05T19:28:24.439 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.434+0000 7fa261c50700 1 -- 172.21.15.167:0/3175752868 <== osd.3 v2:172.21.15.167:6808/30036 13 ==== osd_op_reply(57 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fa25c040040 con 0x55841eb735f0 2023-10-05T19:28:24.439 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.434+0000 7fa261c50700 1 -- 172.21.15.167:0/3175752868 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:58 5.e 5:70000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x55841eb72c90 con 0x55841eb735f0 2023-10-05T19:28:24.440 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.434+0000 7fa261c50700 1 -- 172.21.15.167:0/3175752868 <== osd.3 v2:172.21.15.167:6808/30036 14 ==== osd_op_reply(58 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fa25c040040 con 0x55841eb735f0 2023-10-05T19:28:24.440 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.434+0000 7fa261c50700 1 -- 172.21.15.167:0/3175752868 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:59 5.1e 5:78000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x55841eb72c90 con 0x7fa25800f8d0 2023-10-05T19:28:24.441 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.434+0000 7fa262451700 1 -- 172.21.15.167:0/3175752868 <== osd.0 v2:172.21.15.118:6800/27690 15 ==== osd_op_reply(59 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fa25402f080 con 0x7fa25800f8d0 2023-10-05T19:28:24.441 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.434+0000 7fa262451700 1 -- 172.21.15.167:0/3175752868 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:60 5.1 5:80000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x55841eb72c90 con 0x55841eb735f0 2023-10-05T19:28:24.442 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.434+0000 7fa261c50700 1 -- 172.21.15.167:0/3175752868 <== osd.3 v2:172.21.15.167:6808/30036 15 ==== osd_op_reply(60 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fa25c040040 con 0x55841eb735f0 2023-10-05T19:28:24.442 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.434+0000 7fa261c50700 1 -- 172.21.15.167:0/3175752868 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:61 5.11 5:88000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x55841eb72c90 con 0x55841eb735f0 2023-10-05T19:28:24.442 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.438+0000 7fa261c50700 1 -- 172.21.15.167:0/3175752868 <== osd.3 v2:172.21.15.167:6808/30036 16 ==== osd_op_reply(61 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fa25c040040 con 0x55841eb735f0 2023-10-05T19:28:24.442 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.438+0000 7fa261c50700 1 -- 172.21.15.167:0/3175752868 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:62 5.9 5:90000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x55841eb72c90 con 0x55841eb3c7e0 2023-10-05T19:28:24.443 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.438+0000 7fa262451700 1 -- 172.21.15.167:0/3175752868 <== osd.1 v2:172.21.15.167:6800/30035 22 ==== osd_op_reply(62 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fa25402f080 con 0x55841eb3c7e0 2023-10-05T19:28:24.443 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.438+0000 7fa262451700 1 -- 172.21.15.167:0/3175752868 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:63 5.19 5:98000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x55841eb72c90 con 0x55841eb3c7e0 2023-10-05T19:28:24.444 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.438+0000 7fa262451700 1 -- 172.21.15.167:0/3175752868 <== osd.1 v2:172.21.15.167:6800/30035 23 ==== osd_op_reply(63 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fa25402f080 con 0x55841eb3c7e0 2023-10-05T19:28:24.444 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.438+0000 7fa262451700 1 -- 172.21.15.167:0/3175752868 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:64 5.5 5:a0000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x55841eb72c90 con 0x7fa25800f8d0 2023-10-05T19:28:24.445 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.438+0000 7fa262451700 1 -- 172.21.15.167:0/3175752868 <== osd.0 v2:172.21.15.118:6800/27690 16 ==== osd_op_reply(64 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fa25402f080 con 0x7fa25800f8d0 2023-10-05T19:28:24.445 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.438+0000 7fa262451700 1 -- 172.21.15.167:0/3175752868 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:65 5.15 5:a8000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x55841eb72c90 con 0x55841eb735f0 2023-10-05T19:28:24.446 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.442+0000 7fa261c50700 1 -- 172.21.15.167:0/3175752868 <== osd.3 v2:172.21.15.167:6808/30036 17 ==== osd_op_reply(65 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fa25c040040 con 0x55841eb735f0 2023-10-05T19:28:24.446 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.442+0000 7fa261c50700 1 -- 172.21.15.167:0/3175752868 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:66 5.d 5:b0000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x55841eb72c90 con 0x7fa25c010070 2023-10-05T19:28:24.447 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.442+0000 7fa26144f700 1 -- 172.21.15.167:0/3175752868 <== osd.2 v2:172.21.15.167:6816/30037 10 ==== osd_op_reply(66 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fa25800bc70 con 0x7fa25c010070 2023-10-05T19:28:24.447 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.442+0000 7fa26144f700 1 -- 172.21.15.167:0/3175752868 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:67 5.1d 5:b8000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x55841eb72c90 con 0x55841eb3c7e0 2023-10-05T19:28:24.448 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.442+0000 7fa262451700 1 -- 172.21.15.167:0/3175752868 <== osd.1 v2:172.21.15.167:6800/30035 24 ==== osd_op_reply(67 [pgnls start_epoch 30 out=107b] v23'1 uv1 ondisk = 1) v8 ==== 144+0+107 (crc 0 0 0) 0x7fa25402f080 con 0x55841eb3c7e0 2023-10-05T19:28:24.448 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.442+0000 7fa262451700 1 -- 172.21.15.167:0/3175752868 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:68 5.3 5:c0000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x55841eb72c90 con 0x7fa25800f8d0 2023-10-05T19:28:24.449 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.442+0000 7fa262451700 1 -- 172.21.15.167:0/3175752868 <== osd.0 v2:172.21.15.118:6800/27690 17 ==== osd_op_reply(68 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fa25402f080 con 0x7fa25800f8d0 2023-10-05T19:28:24.449 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.442+0000 7fa262451700 1 -- 172.21.15.167:0/3175752868 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:69 5.13 5:c8000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x55841eb72c90 con 0x55841eb735f0 2023-10-05T19:28:24.450 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.442+0000 7fa261c50700 1 -- 172.21.15.167:0/3175752868 <== osd.3 v2:172.21.15.167:6808/30036 18 ==== osd_op_reply(69 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fa25c040040 con 0x55841eb735f0 2023-10-05T19:28:24.450 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.442+0000 7fa261c50700 1 -- 172.21.15.167:0/3175752868 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:70 5.b 5:d0000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x55841eb72c90 con 0x7fa25800f8d0 2023-10-05T19:28:24.451 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.446+0000 7fa262451700 1 -- 172.21.15.167:0/3175752868 <== osd.0 v2:172.21.15.118:6800/27690 18 ==== osd_op_reply(70 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fa25402f080 con 0x7fa25800f8d0 2023-10-05T19:28:24.451 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.446+0000 7fa262451700 1 -- 172.21.15.167:0/3175752868 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:71 5.1b 5:d8000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x55841eb72c90 con 0x7fa25c010070 2023-10-05T19:28:24.452 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.446+0000 7fa26144f700 1 -- 172.21.15.167:0/3175752868 <== osd.2 v2:172.21.15.167:6816/30037 11 ==== osd_op_reply(71 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fa25800bc70 con 0x7fa25c010070 2023-10-05T19:28:24.452 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.446+0000 7fa26144f700 1 -- 172.21.15.167:0/3175752868 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:72 5.7 5:e0000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x55841eb72c90 con 0x55841eb735f0 2023-10-05T19:28:24.453 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.446+0000 7fa261c50700 1 -- 172.21.15.167:0/3175752868 <== osd.3 v2:172.21.15.167:6808/30036 19 ==== osd_op_reply(72 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fa25c040040 con 0x55841eb735f0 2023-10-05T19:28:24.453 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.446+0000 7fa261c50700 1 -- 172.21.15.167:0/3175752868 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:73 5.17 5:e8000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x55841eb72c90 con 0x7fa25800f8d0 2023-10-05T19:28:24.454 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.446+0000 7fa262451700 1 -- 172.21.15.167:0/3175752868 <== osd.0 v2:172.21.15.118:6800/27690 19 ==== osd_op_reply(73 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fa25402f080 con 0x7fa25800f8d0 2023-10-05T19:28:24.454 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.450+0000 7fa262451700 1 -- 172.21.15.167:0/3175752868 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:74 5.f 5:f0000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x55841eb72c90 con 0x55841eb3c7e0 2023-10-05T19:28:24.455 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.450+0000 7fa262451700 1 -- 172.21.15.167:0/3175752868 <== osd.1 v2:172.21.15.167:6800/30035 25 ==== osd_op_reply(74 [pgnls start_epoch 30 out=115b] v23'2 uv2 ondisk = 1) v8 ==== 144+0+115 (crc 0 0 0) 0x7fa25402f080 con 0x55841eb3c7e0 2023-10-05T19:28:24.455 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.450+0000 7fa262451700 1 -- 172.21.15.167:0/3175752868 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:75 5.1f 5:f8000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x55841eb72c90 con 0x7fa25c010070 2023-10-05T19:28:24.456 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.450+0000 7fa26144f700 1 -- 172.21.15.167:0/3175752868 <== osd.2 v2:172.21.15.167:6816/30037 12 ==== osd_op_reply(75 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fa25800bc70 con 0x7fa25c010070 2023-10-05T19:28:24.456 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.450+0000 7fa26411be40 20 RGWRados::pool_iterate: got zonegroup_info.4f1f719d-9648-417b-b10c-e5d939825419 2023-10-05T19:28:24.456 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.450+0000 7fa26411be40 20 RGWRados::pool_iterate: got zone_info.9a9df965-6612-45d4-a45f-6e3c889bacc2 2023-10-05T19:28:24.456 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.450+0000 7fa26411be40 20 RGWRados::pool_iterate: got zone_names.default 2023-10-05T19:28:24.456 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.450+0000 7fa26411be40 20 RGWRados::pool_iterate: got zonegroups_names.default 2023-10-05T19:28:24.456 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.450+0000 7fa26411be40 20 rados->read ofs=0 len=0 2023-10-05T19:28:24.456 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.450+0000 7fa26411be40 1 -- 172.21.15.167:0/3175752868 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:76 5.f 5:f43b6ece:::zone_names.default:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x55841eb72860 con 0x55841eb3c7e0 2023-10-05T19:28:24.457 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.450+0000 7fa262451700 1 -- 172.21.15.167:0/3175752868 <== osd.1 v2:172.21.15.167:6800/30035 26 ==== osd_op_reply(76 zone_names.default [read 0~46 out=46b] v0'0 uv1 ondisk = 0) v8 ==== 162+0+46 (crc 0 0 0) 0x7fa25402f080 con 0x55841eb3c7e0 2023-10-05T19:28:24.457 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.450+0000 7fa26411be40 20 rados_obj.operate() r=0 bl.length=46 2023-10-05T19:28:24.457 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.450+0000 7fa26411be40 20 rados->read ofs=0 len=0 2023-10-05T19:28:24.457 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.450+0000 7fa26411be40 1 -- 172.21.15.167:0/3175752868 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:77 5.1d 5:b8374eb4:::zone_info.9a9df965-6612-45d4-a45f-6e3c889bacc2:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x55841eb76c40 con 0x55841eb3c7e0 2023-10-05T19:28:24.458 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.450+0000 7fa262451700 1 -- 172.21.15.167:0/3175752868 <== osd.1 v2:172.21.15.167:6800/30035 27 ==== osd_op_reply(77 zone_info.9a9df965-6612-45d4-a45f-6e3c889bacc2 [read 0~874 out=874b] v0'0 uv1 ondisk = 0) v8 ==== 190+0+874 (crc 0 0 0) 0x7fa25402f080 con 0x55841eb3c7e0 2023-10-05T19:28:24.458 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.450+0000 7fa26411be40 20 rados_obj.operate() r=0 bl.length=874 2023-10-05T19:28:24.458 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.450+0000 7fa26411be40 20 rados->read ofs=0 len=0 2023-10-05T19:28:24.458 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.450+0000 7fa26411be40 1 -- 172.21.15.167:0/3175752868 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:78 5.f 5:f4c53578:::zonegroups_names.default:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x55841eb77130 con 0x55841eb3c7e0 2023-10-05T19:28:24.459 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.454+0000 7fa262451700 1 -- 172.21.15.167:0/3175752868 <== osd.1 v2:172.21.15.167:6800/30035 28 ==== osd_op_reply(78 zonegroups_names.default [read 0~46 out=46b] v0'0 uv2 ondisk = 0) v8 ==== 168+0+46 (crc 0 0 0) 0x7fa25402f080 con 0x55841eb3c7e0 2023-10-05T19:28:24.459 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.454+0000 7fa26411be40 20 rados_obj.operate() r=0 bl.length=46 2023-10-05T19:28:24.459 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.454+0000 7fa26411be40 20 rados->read ofs=0 len=0 2023-10-05T19:28:24.459 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.454+0000 7fa26411be40 1 -- 172.21.15.167:0/3175752868 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:79 5.c 5:30d299a5:::zonegroup_info.4f1f719d-9648-417b-b10c-e5d939825419:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x55841eb77130 con 0x7fa25800f8d0 2023-10-05T19:28:24.460 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.454+0000 7fa262451700 1 -- 172.21.15.167:0/3175752868 <== osd.0 v2:172.21.15.118:6800/27690 20 ==== osd_op_reply(79 zonegroup_info.4f1f719d-9648-417b-b10c-e5d939825419 [read 0~362 out=362b] v0'0 uv1 ondisk = 0) v8 ==== 195+0+362 (crc 0 0 0) 0x7fa25402f080 con 0x7fa25800f8d0 2023-10-05T19:28:24.460 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.454+0000 7fa26411be40 20 rados_obj.operate() r=0 bl.length=362 2023-10-05T19:28:24.460 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.454+0000 7fa26411be40 20 rados->read ofs=0 len=0 2023-10-05T19:28:24.460 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.454+0000 7fa26411be40 1 -- 172.21.15.167:0/3175752868 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:80 5.3 5:c0ad8cbd:::region_map:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x55841eb77230 con 0x7fa25800f8d0 2023-10-05T19:28:24.461 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.454+0000 7fa262451700 1 -- 172.21.15.167:0/3175752868 <== osd.0 v2:172.21.15.118:6800/27690 21 ==== osd_op_reply(80 region_map [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 ==== 154+0+0 (crc 0 0 0) 0x7fa25402f080 con 0x7fa25800f8d0 2023-10-05T19:28:24.461 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.454+0000 7fa26411be40 20 rados_obj.operate() r=-2 bl.length=0 2023-10-05T19:28:24.461 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.454+0000 7fa26411be40 20 zone default found 2023-10-05T19:28:24.461 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.454+0000 7fa26411be40 4 Realm: () 2023-10-05T19:28:24.461 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.454+0000 7fa26411be40 4 ZoneGroup: default (4f1f719d-9648-417b-b10c-e5d939825419) 2023-10-05T19:28:24.461 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.454+0000 7fa26411be40 4 Zone: default (9a9df965-6612-45d4-a45f-6e3c889bacc2) 2023-10-05T19:28:24.461 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.454+0000 7fa26411be40 10 cannot find current period zonegroup using local zonegroup configuration 2023-10-05T19:28:24.461 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.454+0000 7fa26411be40 20 zonegroup default 2023-10-05T19:28:24.461 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.454+0000 7fa26411be40 20 rados->read ofs=0 len=0 2023-10-05T19:28:24.461 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.454+0000 7fa26411be40 1 -- 172.21.15.167:0/3175752868 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:81 5.3 5:c52100b6:::period_config.default:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x55841eb77230 con 0x7fa25800f8d0 2023-10-05T19:28:24.462 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.454+0000 7fa262451700 1 -- 172.21.15.167:0/3175752868 <== osd.0 v2:172.21.15.118:6800/27690 22 ==== osd_op_reply(81 period_config.default [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 ==== 165+0+0 (crc 0 0 0) 0x7fa25402f080 con 0x7fa25800f8d0 2023-10-05T19:28:24.462 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.454+0000 7fa26411be40 20 rados_obj.operate() r=-2 bl.length=0 2023-10-05T19:28:24.462 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.454+0000 7fa26411be40 20 rados->read ofs=0 len=0 2023-10-05T19:28:24.463 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.454+0000 7fa26411be40 1 -- 172.21.15.167:0/3175752868 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:82 6.2 6:4347d321:::bucket.sync-source-hints.:head [call version.read in=11b,read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x55841eb78bb0 con 0x55841eb735f0 2023-10-05T19:28:24.463 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.458+0000 7fa261c50700 1 -- 172.21.15.167:0/3175752868 <== osd.3 v2:172.21.15.167:6808/30036 20 ==== osd_op_reply(82 bucket.sync-source-hints. [call,read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 ==== 211+0+0 (crc 0 0 0) 0x7fa25c040040 con 0x55841eb735f0 2023-10-05T19:28:24.463 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.458+0000 7fa26411be40 20 rados_obj.operate() r=-2 bl.length=0 2023-10-05T19:28:24.463 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.458+0000 7fa26411be40 20 rados->read ofs=0 len=0 2023-10-05T19:28:24.463 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.458+0000 7fa26411be40 1 -- 172.21.15.167:0/3175752868 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:83 6.b 6:d467b91b:::bucket.sync-target-hints.:head [call version.read in=11b,read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x55841eb78f60 con 0x55841eb735f0 2023-10-05T19:28:24.465 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.458+0000 7fa261c50700 1 -- 172.21.15.167:0/3175752868 <== osd.3 v2:172.21.15.167:6808/30036 21 ==== osd_op_reply(83 bucket.sync-target-hints. [call,read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 ==== 211+0+0 (crc 0 0 0) 0x7fa25c040040 con 0x55841eb735f0 2023-10-05T19:28:24.465 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.458+0000 7fa26411be40 20 rados_obj.operate() r=-2 bl.length=0 2023-10-05T19:28:24.466 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.458+0000 7fa26411be40 20 started sync module instance, tier type = 2023-10-05T19:28:24.466 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.458+0000 7fa26411be40 20 started zone id=9a9df965-6612-45d4-a45f-6e3c889bacc2 (name=default) with tier type = 2023-10-05T19:28:24.466 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.458+0000 7fa26411be40 1 -- 172.21.15.167:0/3175752868 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:84 7.1f 7:f95f44c2:::notify.0:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x55841eb78270 con 0x7fa25c010070 2023-10-05T19:28:24.466 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.462+0000 7fa26144f700 1 -- 172.21.15.167:0/3175752868 <== osd.2 v2:172.21.15.167:6816/30037 13 ==== osd_op_reply(84 notify.0 [create] v30'12 uv12 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7fa25800bc70 con 0x7fa25c010070 2023-10-05T19:28:24.466 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.462+0000 7fa26411be40 1 -- 172.21.15.167:0/3175752868 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:85 7.1f 7:f95f44c2:::notify.0:head [watch watch cookie 94025939388016] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x55841eb77ea0 con 0x7fa25c010070 2023-10-05T19:28:24.467 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.462+0000 7fa26411be40 1 -- 172.21.15.167:0/3175752868 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:86 7.0 7:05bf5b68:::notify.1:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x55841eb79cf0 con 0x55841eb3c7e0 2023-10-05T19:28:24.469 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.462+0000 7fa26144f700 1 -- 172.21.15.167:0/3175752868 <== osd.2 v2:172.21.15.167:6816/30037 14 ==== osd_op_reply(85 notify.0 [watch watch cookie 94025939388016] v30'13 uv12 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7fa25800bc70 con 0x7fa25c010070 2023-10-05T19:28:24.469 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.462+0000 7fa262451700 1 -- 172.21.15.167:0/3175752868 <== osd.1 v2:172.21.15.167:6800/30035 29 ==== osd_op_reply(86 notify.1 [create] v30'12 uv12 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7fa25402f080 con 0x55841eb3c7e0 2023-10-05T19:28:24.469 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.462+0000 7fa26411be40 1 -- 172.21.15.167:0/3175752868 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:87 7.0 7:05bf5b68:::notify.1:head [watch watch cookie 94025939391536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x55841eb7a0f0 con 0x55841eb3c7e0 2023-10-05T19:28:24.469 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.462+0000 7fa26411be40 1 -- 172.21.15.167:0/3175752868 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:88 7.15 7:a93a5511:::notify.2:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x55841eb7a990 con 0x55841eb735f0 2023-10-05T19:28:24.472 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.466+0000 7fa262451700 1 -- 172.21.15.167:0/3175752868 <== osd.1 v2:172.21.15.167:6800/30035 30 ==== osd_op_reply(87 notify.1 [watch watch cookie 94025939391536] v30'13 uv12 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7fa25402f080 con 0x55841eb3c7e0 2023-10-05T19:28:24.473 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.466+0000 7fa261c50700 1 -- 172.21.15.167:0/3175752868 <== osd.3 v2:172.21.15.167:6808/30036 22 ==== osd_op_reply(88 notify.2 [create] v30'12 uv12 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7fa25c040040 con 0x55841eb735f0 2023-10-05T19:28:24.473 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.466+0000 7fa26411be40 1 -- 172.21.15.167:0/3175752868 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:89 7.15 7:a93a5511:::notify.2:head [watch watch cookie 94025939392640] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x55841eb7a5e0 con 0x55841eb735f0 2023-10-05T19:28:24.473 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.466+0000 7fa26411be40 1 -- 172.21.15.167:0/3175752868 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:90 7.e 7:7759931f:::notify.3:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x55841eb7b4d0 con 0x7fa25c010070 2023-10-05T19:28:24.476 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.470+0000 7fa26144f700 1 -- 172.21.15.167:0/3175752868 <== osd.2 v2:172.21.15.167:6816/30037 15 ==== osd_op_reply(90 notify.3 [create] v30'12 uv12 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7fa25800bc70 con 0x7fa25c010070 2023-10-05T19:28:24.476 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.470+0000 7fa26411be40 1 -- 172.21.15.167:0/3175752868 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:91 7.e 7:7759931f:::notify.3:head [watch watch cookie 94025939400912] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x55841eb7c210 con 0x7fa25c010070 2023-10-05T19:28:24.476 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.470+0000 7fa26411be40 1 -- 172.21.15.167:0/3175752868 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:92 7.d 7:b4812045:::notify.4:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x55841eb7cc20 con 0x55841eb3c7e0 2023-10-05T19:28:24.476 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.470+0000 7fa261c50700 1 -- 172.21.15.167:0/3175752868 <== osd.3 v2:172.21.15.167:6808/30036 23 ==== osd_op_reply(89 notify.2 [watch watch cookie 94025939392640] v30'13 uv12 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7fa25c040040 con 0x55841eb735f0 2023-10-05T19:28:24.478 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.474+0000 7fa26144f700 1 -- 172.21.15.167:0/3175752868 <== osd.2 v2:172.21.15.167:6816/30037 16 ==== osd_op_reply(91 notify.3 [watch watch cookie 94025939400912] v30'13 uv12 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7fa25800bc70 con 0x7fa25c010070 2023-10-05T19:28:24.478 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.474+0000 7fa262451700 1 -- 172.21.15.167:0/3175752868 <== osd.1 v2:172.21.15.167:6800/30035 31 ==== osd_op_reply(92 notify.4 [create] v30'12 uv12 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7fa25402f080 con 0x55841eb3c7e0 2023-10-05T19:28:24.479 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.474+0000 7fa26411be40 1 -- 172.21.15.167:0/3175752868 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:93 7.d 7:b4812045:::notify.4:head [watch watch cookie 94025939394720] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x55841eb7cc20 con 0x55841eb3c7e0 2023-10-05T19:28:24.479 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.474+0000 7fa26411be40 1 -- 172.21.15.167:0/3175752868 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:94 7.3 7:c609908c:::notify.5:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x55841eb7d4c0 con 0x7fa25800f8d0 2023-10-05T19:28:24.481 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.474+0000 7fa262451700 1 -- 172.21.15.167:0/3175752868 <== osd.1 v2:172.21.15.167:6800/30035 32 ==== osd_op_reply(93 notify.4 [watch watch cookie 94025939394720] v30'13 uv12 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7fa25402f080 con 0x55841eb3c7e0 2023-10-05T19:28:24.482 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.474+0000 7fa262451700 1 -- 172.21.15.167:0/3175752868 <== osd.0 v2:172.21.15.118:6800/27690 23 ==== osd_op_reply(94 notify.5 [create] v30'12 uv12 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7fa25402f080 con 0x7fa25800f8d0 2023-10-05T19:28:24.482 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.474+0000 7fa26411be40 1 -- 172.21.15.167:0/3175752868 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:95 7.3 7:c609908c:::notify.5:head [watch watch cookie 94025939404864] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x55841eb7d180 con 0x7fa25800f8d0 2023-10-05T19:28:24.482 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.474+0000 7fa26411be40 1 -- 172.21.15.167:0/3175752868 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:96 7.14 7:2b04a3e9:::notify.6:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x55841eb7dbf0 con 0x7fa25800f8d0 2023-10-05T19:28:24.485 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.478+0000 7fa262451700 1 -- 172.21.15.167:0/3175752868 <== osd.0 v2:172.21.15.118:6800/27690 24 ==== osd_op_reply(96 notify.6 [create] v30'12 uv12 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7fa25402f080 con 0x7fa25800f8d0 2023-10-05T19:28:24.485 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.478+0000 7fa26411be40 1 -- 172.21.15.167:0/3175752868 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:97 7.14 7:2b04a3e9:::notify.6:head [watch watch cookie 94025939408640] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x55841eb7e040 con 0x7fa25800f8d0 2023-10-05T19:28:24.485 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.478+0000 7fa262451700 1 -- 172.21.15.167:0/3175752868 <== osd.0 v2:172.21.15.118:6800/27690 25 ==== osd_op_reply(95 notify.5 [watch watch cookie 94025939404864] v30'13 uv12 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7fa25402f080 con 0x7fa25800f8d0 2023-10-05T19:28:24.485 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.478+0000 7fa26411be40 1 -- 172.21.15.167:0/3175752868 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:98 7.9 7:93e5b521:::notify.7:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x55841eb7e9a0 con 0x7fa25800f8d0 2023-10-05T19:28:24.487 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.482+0000 7fa262451700 1 -- 172.21.15.167:0/3175752868 <== osd.0 v2:172.21.15.118:6800/27690 26 ==== osd_op_reply(97 notify.6 [watch watch cookie 94025939408640] v30'13 uv12 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7fa25402f080 con 0x7fa25800f8d0 2023-10-05T19:28:24.487 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.482+0000 7fa262451700 1 -- 172.21.15.167:0/3175752868 <== osd.0 v2:172.21.15.118:6800/27690 27 ==== osd_op_reply(98 notify.7 [create] v30'12 uv12 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7fa25402f080 con 0x7fa25800f8d0 2023-10-05T19:28:24.487 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.482+0000 7fa26411be40 1 -- 172.21.15.167:0/3175752868 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:99 7.9 7:93e5b521:::notify.7:head [watch watch cookie 94025939409744] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x55841eb7dfa0 con 0x7fa25800f8d0 2023-10-05T19:28:24.487 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.482+0000 7fa26411be40 20 add_watcher() i=0 2023-10-05T19:28:24.487 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.482+0000 7fa26411be40 20 add_watcher() i=1 2023-10-05T19:28:24.488 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.482+0000 7fa26411be40 20 add_watcher() i=2 2023-10-05T19:28:24.488 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.482+0000 7fa26411be40 20 add_watcher() i=3 2023-10-05T19:28:24.488 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.482+0000 7fa26411be40 20 add_watcher() i=4 2023-10-05T19:28:24.488 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.482+0000 7fa26411be40 20 add_watcher() i=5 2023-10-05T19:28:24.488 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.482+0000 7fa26411be40 20 add_watcher() i=6 2023-10-05T19:28:24.489 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.482+0000 7fa262451700 1 -- 172.21.15.167:0/3175752868 <== osd.0 v2:172.21.15.118:6800/27690 28 ==== osd_op_reply(99 notify.7 [watch watch cookie 94025939409744] v30'13 uv12 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7fa25402f080 con 0x7fa25800f8d0 2023-10-05T19:28:24.489 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.482+0000 7fa26411be40 20 add_watcher() i=7 2023-10-05T19:28:24.489 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.482+0000 7fa26411be40 2 all 8 watchers are set, enabling cache 2023-10-05T19:28:24.489 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.482+0000 7fa26411be40 1 -- 172.21.15.167:0/3175752868 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:100 6.4 6:22d26bf9:::data_loggenerations_metadata:head [call version.check_conds in=74b,call version.read in=11b,read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x55841eb819d0 con 0x55841eb3c7e0 2023-10-05T19:28:24.490 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.486+0000 7fa262451700 1 -- 172.21.15.167:0/3175752868 <== osd.1 v2:172.21.15.167:6800/30035 33 ==== osd_op_reply(100 data_loggenerations_metadata [call,call out=48b,read 0~28 out=28b] v0'0 uv1 ondisk = 0) v8 ==== 256+0+76 (crc 0 0 0) 0x7fa25402f080 con 0x55841eb3c7e0 2023-10-05T19:28:24.490 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.486+0000 7fa26411be40 1 -- 172.21.15.167:0/3175752868 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:101 6.4 6:22d26bf9:::data_loggenerations_metadata:head [watch watch cookie 94025939412144] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x55841eb7edf0 con 0x55841eb3c7e0 2023-10-05T19:28:24.492 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.486+0000 7fa262451700 1 -- 172.21.15.167:0/3175752868 <== osd.1 v2:172.21.15.167:6800/30035 34 ==== osd_op_reply(101 data_loggenerations_metadata [watch watch cookie 94025939412144] v30'36 uv1 ondisk = 0) v8 ==== 172+0+0 (crc 0 0 0) 0x7fa25402f080 con 0x55841eb3c7e0 2023-10-05T19:28:24.492 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.486+0000 7fa232fe5700 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2023-10-05T19:28:24.492 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.486+0000 7fa26411be40 20 check_secure_mon_conn(): auth registy supported: methods=[2] modes=[2,1] 2023-10-05T19:28:24.493 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.486+0000 7fa26411be40 20 check_secure_mon_conn(): mode 1 is insecure 2023-10-05T19:28:24.493 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.486+0000 7fa26411be40 1 Processor -- start 2023-10-05T19:28:24.493 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.486+0000 7fa26411be40 1 -- start start 2023-10-05T19:28:24.493 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.486+0000 7fa26411be40 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55841eb8c060 0x55841eb8c430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-05T19:28:24.494 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.486+0000 7fa26411be40 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x55841eb8c970 con 0x55841eb8c060 2023-10-05T19:28:24.494 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.490+0000 7fa26144f700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55841eb8c060 0x55841eb8c430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-05T19:28:24.494 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.490+0000 7fa26144f700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55841eb8c060 0x55841eb8c430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.118:3300/0 says I am v2:172.21.15.167:56950/0 (socket says 172.21.15.167:56950) 2023-10-05T19:28:24.494 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.490+0000 7fa26144f700 1 -- 172.21.15.167:0/3656866140 learned_addr learned my addr 172.21.15.167:0/3656866140 (peer_addr_for_me v2:172.21.15.167:0/0) 2023-10-05T19:28:24.495 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.490+0000 7fa26144f700 1 -- 172.21.15.167:0/3656866140 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55841eb8cab0 con 0x55841eb8c060 2023-10-05T19:28:24.495 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.490+0000 7fa26144f700 1 --2- 172.21.15.167:0/3656866140 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55841eb8c060 0x55841eb8c430 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x55841eb71090 tx=0x7fa258011000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bb34b68696a6a53e server_cookie=0 in_seq=0 out_seq=0 2023-10-05T19:28:24.495 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.490+0000 7fa2327e4700 1 -- 172.21.15.167:0/3656866140 <== mon.0 v2:172.21.15.118:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fa258010e40 con 0x55841eb8c060 2023-10-05T19:28:24.495 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.490+0000 7fa2327e4700 1 -- 172.21.15.167:0/3656866140 <== mon.0 v2:172.21.15.118:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa258026040 con 0x55841eb8c060 2023-10-05T19:28:24.496 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.490+0000 7fa2327e4700 1 -- 172.21.15.167:0/3656866140 <== mon.0 v2:172.21.15.118:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fa25800ed60 con 0x55841eb8c060 2023-10-05T19:28:24.497 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.490+0000 7fa26411be40 1 -- 172.21.15.167:0/3656866140 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55841eb8c060 msgr2=0x55841eb8c430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-05T19:28:24.497 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.490+0000 7fa26411be40 1 --2- 172.21.15.167:0/3656866140 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55841eb8c060 0x55841eb8c430 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x55841eb71090 tx=0x7fa258011000 comp rx=0 tx=0).stop 2023-10-05T19:28:24.497 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.490+0000 7fa26411be40 1 -- 172.21.15.167:0/3656866140 shutdown_connections 2023-10-05T19:28:24.497 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.490+0000 7fa26411be40 1 --2- 172.21.15.167:0/3656866140 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55841eb8c060 0x55841eb8c430 unknown :-1 s=CLOSED pgs=156 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-05T19:28:24.497 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.490+0000 7fa26411be40 1 -- 172.21.15.167:0/3656866140 >> 172.21.15.167:0/3656866140 conn(0x55841eb87420 msgr2=0x55841eb85570 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-05T19:28:24.497 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.490+0000 7fa26411be40 1 -- 172.21.15.167:0/3656866140 shutdown_connections 2023-10-05T19:28:24.497 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.490+0000 7fa26411be40 1 -- 172.21.15.167:0/3656866140 wait complete. 2023-10-05T19:28:24.498 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.490+0000 7fa26411be40 1 Processor -- start 2023-10-05T19:28:24.498 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.490+0000 7fa26411be40 1 -- start start 2023-10-05T19:28:24.498 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.490+0000 7fa26411be40 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55841eb8c060 0x55841eb91c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-05T19:28:24.498 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.490+0000 7fa26411be40 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x55841eb92c10 con 0x55841eb8c060 2023-10-05T19:28:24.498 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.494+0000 7fa26144f700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55841eb8c060 0x55841eb91c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-05T19:28:24.499 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.494+0000 7fa26144f700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55841eb8c060 0x55841eb91c10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.118:3300/0 says I am v2:172.21.15.167:56958/0 (socket says 172.21.15.167:56958) 2023-10-05T19:28:24.499 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.494+0000 7fa26144f700 1 -- 172.21.15.167:0/3590574458 learned_addr learned my addr 172.21.15.167:0/3590574458 (peer_addr_for_me v2:172.21.15.167:0/0) 2023-10-05T19:28:24.500 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.494+0000 7fa26144f700 1 -- 172.21.15.167:0/3590574458 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa25800a040 con 0x55841eb8c060 2023-10-05T19:28:24.500 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.494+0000 7fa26144f700 1 --2- 172.21.15.167:0/3590574458 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55841eb8c060 0x55841eb91c10 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7fa2580104c0 tx=0x7fa258031070 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-05T19:28:24.500 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.494+0000 7fa230fe1700 1 -- 172.21.15.167:0/3590574458 <== mon.0 v2:172.21.15.118:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fa258044070 con 0x55841eb8c060 2023-10-05T19:28:24.500 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.494+0000 7fa230fe1700 1 -- 172.21.15.167:0/3590574458 <== mon.0 v2:172.21.15.118:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa258020450 con 0x55841eb8c060 2023-10-05T19:28:24.500 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.494+0000 7fa26411be40 1 -- 172.21.15.167:0/3590574458 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x55841eb92210 con 0x55841eb8c060 2023-10-05T19:28:24.500 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.494+0000 7fa230fe1700 1 -- 172.21.15.167:0/3590574458 <== mon.0 v2:172.21.15.118:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fa258020740 con 0x55841eb8c060 2023-10-05T19:28:24.500 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.494+0000 7fa26411be40 1 -- 172.21.15.167:0/3590574458 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55841eb92650 con 0x55841eb8c060 2023-10-05T19:28:24.501 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.494+0000 7fa230fe1700 1 -- 172.21.15.167:0/3590574458 <== mon.0 v2:172.21.15.118:3300/0 4 ==== mgrmap(e 5) v1 ==== 40329+0+0 (secure 0 0 0) 0x7fa2580208a0 con 0x55841eb8c060 2023-10-05T19:28:24.501 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.494+0000 7fa230fe1700 1 --2- 172.21.15.167:0/3590574458 >> [v2:172.21.15.167:6824/29889,v1:172.21.15.167:6825/29889] conn(0x7fa1a8030c50 0x7fa1a8033100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-05T19:28:24.501 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.494+0000 7fa230fe1700 1 -- 172.21.15.167:0/3590574458 <== mon.0 v2:172.21.15.118:3300/0 5 ==== osd_map(30..30 src has 1..30) v4 ==== 6057+0+0 (secure 0 0 0) 0x7fa258010e40 con 0x55841eb8c060 2023-10-05T19:28:24.502 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.494+0000 7fa262451700 1 --2- 172.21.15.167:0/3590574458 >> [v2:172.21.15.167:6824/29889,v1:172.21.15.167:6825/29889] conn(0x7fa1a8030c50 0x7fa1a8033100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-05T19:28:24.502 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.498+0000 7fa26411be40 1 --2- 172.21.15.167:0/3590574458 >> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] conn(0x55841eb9c860 0x55841eb9ec70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-05T19:28:24.502 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.498+0000 7fa26411be40 1 -- 172.21.15.167:0/3590574458 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:1 6.19 6:9a6dd8c0:gc::gc.0:head [create,call version.check_conds in=74b,call rgw_gc.rgw_gc_queue_init in=45b,call version.set in=34b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x55841eb9f2c0 con 0x55841eb9c860 2023-10-05T19:28:24.502 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.498+0000 7fa261c50700 1 --2- 172.21.15.167:0/3590574458 >> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] conn(0x55841eb9c860 0x55841eb9ec70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-05T19:28:24.503 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.498+0000 7fa262451700 1 --2- 172.21.15.167:0/3590574458 >> [v2:172.21.15.167:6824/29889,v1:172.21.15.167:6825/29889] conn(0x7fa1a8030c50 0x7fa1a8033100 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x55841eb7e270 tx=0x7fa254000f90 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-05T19:28:24.503 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.498+0000 7fa261c50700 1 --2- 172.21.15.167:0/3590574458 >> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] conn(0x55841eb9c860 0x55841eb9ec70 crc :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-05T19:28:24.505 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.498+0000 7fa261c50700 1 -- 172.21.15.167:0/3590574458 <== osd.1 v2:172.21.15.167:6800/30035 1 ==== osd_op_reply(1 gc.0 [create,call,call,call] v30'18 uv3 ondisk = -125 ((125) Operation canceled)) v8 ==== 274+0+0 (crc 0 0 0) 0x7fa25c040040 con 0x55841eb9c860 2023-10-05T19:28:24.505 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.498+0000 7fa26411be40 1 --2- 172.21.15.167:0/3590574458 >> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] conn(0x55841eba0310 0x55841eba26f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-05T19:28:24.506 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.498+0000 7fa26411be40 1 -- 172.21.15.167:0/3590574458 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:2 6.18 6:1e85421f:gc::gc.1:head [create,call version.check_conds in=74b,call rgw_gc.rgw_gc_queue_init in=45b,call version.set in=34b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x55841eba2dd0 con 0x55841eba0310 2023-10-05T19:28:24.506 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.502+0000 7fa26144f700 1 --2- 172.21.15.167:0/3590574458 >> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] conn(0x55841eba0310 0x55841eba26f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-05T19:28:24.506 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.502+0000 7fa26144f700 1 --2- 172.21.15.167:0/3590574458 >> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] conn(0x55841eba0310 0x55841eba26f0 crc :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-05T19:28:24.509 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.502+0000 7fa26144f700 1 -- 172.21.15.167:0/3590574458 <== osd.0 v2:172.21.15.118:6800/27690 1 ==== osd_op_reply(2 gc.1 [create,call,call,call] v30'12 uv5 ondisk = -125 ((125) Operation canceled)) v8 ==== 274+0+0 (crc 0 0 0) 0x7fa25800bc70 con 0x55841eba0310 2023-10-05T19:28:24.510 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.502+0000 7fa26411be40 1 --2- 172.21.15.167:0/3590574458 >> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] conn(0x55841eba3aa0 0x55841eba5eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-05T19:28:24.510 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.502+0000 7fa26411be40 1 -- 172.21.15.167:0/3590574458 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:3 6.b 6:d242335b:gc::gc.2:head [create,call version.check_conds in=74b,call rgw_gc.rgw_gc_queue_init in=45b,call version.set in=34b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x55841eba65e0 con 0x55841eba3aa0 2023-10-05T19:28:24.510 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.502+0000 7fa262451700 1 --2- 172.21.15.167:0/3590574458 >> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] conn(0x55841eba3aa0 0x55841eba5eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-05T19:28:24.510 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.506+0000 7fa262451700 1 --2- 172.21.15.167:0/3590574458 >> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] conn(0x55841eba3aa0 0x55841eba5eb0 crc :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-05T19:28:24.512 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.506+0000 7fa262451700 1 -- 172.21.15.167:0/3590574458 <== osd.3 v2:172.21.15.167:6808/30036 1 ==== osd_op_reply(3 gc.2 [create,call,call,call] v30'12 uv3 ondisk = -125 ((125) Operation canceled)) v8 ==== 274+0+0 (crc 0 0 0) 0x7fa25402f080 con 0x55841eba3aa0 2023-10-05T19:28:24.513 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.506+0000 7fa26411be40 1 -- 172.21.15.167:0/3590574458 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:4 6.4 6:2166d8e1:gc::gc.3:head [create,call version.check_conds in=74b,call rgw_gc.rgw_gc_queue_init in=45b,call version.set in=34b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x55841eba6c30 con 0x55841eb9c860 2023-10-05T19:28:24.516 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.510+0000 7fa261c50700 1 -- 172.21.15.167:0/3590574458 <== osd.1 v2:172.21.15.167:6800/30035 2 ==== osd_op_reply(4 gc.3 [create,call,call,call] v30'37 uv7 ondisk = -125 ((125) Operation canceled)) v8 ==== 274+0+0 (crc 0 0 0) 0x7fa25c040040 con 0x55841eb9c860 2023-10-05T19:28:24.516 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.510+0000 7fa26411be40 1 --2- 172.21.15.167:0/3590574458 >> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] conn(0x55841eba78d0 0x55841eba9cb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-05T19:28:24.516 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.510+0000 7fa26411be40 1 -- 172.21.15.167:0/3590574458 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:5 6.3 6:c60da066:gc::gc.4:head [create,call version.check_conds in=74b,call rgw_gc.rgw_gc_queue_init in=45b,call version.set in=34b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x55841ebaa360 con 0x55841eba78d0 2023-10-05T19:28:24.516 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.510+0000 7fa261c50700 1 --2- 172.21.15.167:0/3590574458 >> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] conn(0x55841eba78d0 0x55841eba9cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-05T19:28:24.516 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.510+0000 7fa261c50700 1 --2- 172.21.15.167:0/3590574458 >> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] conn(0x55841eba78d0 0x55841eba9cb0 crc :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-05T19:28:24.519 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.514+0000 7fa261c50700 1 -- 172.21.15.167:0/3590574458 <== osd.2 v2:172.21.15.167:6816/30037 1 ==== osd_op_reply(5 gc.4 [create,call,call,call] v30'26 uv7 ondisk = -125 ((125) Operation canceled)) v8 ==== 274+0+0 (crc 0 0 0) 0x7fa25c040040 con 0x55841eba78d0 2023-10-05T19:28:24.519 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.514+0000 7fa26411be40 1 -- 172.21.15.167:0/3590574458 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:6 6.9 6:954be8d2:gc::gc.5:head [create,call version.check_conds in=74b,call rgw_gc.rgw_gc_queue_init in=45b,call version.set in=34b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x55841ebaa980 con 0x55841eba3aa0 2023-10-05T19:28:24.521 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.514+0000 7fa262451700 1 -- 172.21.15.167:0/3590574458 <== osd.3 v2:172.21.15.167:6808/30036 2 ==== osd_op_reply(6 gc.5 [create,call,call,call] v30'17 uv4 ondisk = -125 ((125) Operation canceled)) v8 ==== 274+0+0 (crc 0 0 0) 0x7fa25402f080 con 0x55841eba3aa0 2023-10-05T19:28:24.522 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.514+0000 7fa26411be40 1 -- 172.21.15.167:0/3590574458 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:7 6.e 6:748722ff:gc::gc.6:head [create,call version.check_conds in=74b,call rgw_gc.rgw_gc_queue_init in=45b,call version.set in=34b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x55841ebaafa0 con 0x55841eb9c860 2023-10-05T19:28:24.524 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.518+0000 7fa261c50700 1 -- 172.21.15.167:0/3590574458 <== osd.1 v2:172.21.15.167:6800/30035 3 ==== osd_op_reply(7 gc.6 [create,call,call,call] v30'14 uv5 ondisk = -125 ((125) Operation canceled)) v8 ==== 274+0+0 (crc 0 0 0) 0x7fa25c040040 con 0x55841eb9c860 2023-10-05T19:28:24.525 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.518+0000 7fa26411be40 1 -- 172.21.15.167:0/3590574458 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:8 6.1d 6:beaa4be4:gc::gc.7:head [create,call version.check_conds in=74b,call rgw_gc.rgw_gc_queue_init in=45b,call version.set in=34b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x55841ebab490 con 0x55841eba0310 2023-10-05T19:28:24.528 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.522+0000 7fa26144f700 1 -- 172.21.15.167:0/3590574458 <== osd.0 v2:172.21.15.118:6800/27690 2 ==== osd_op_reply(8 gc.7 [create,call,call,call] v30'17 uv4 ondisk = -125 ((125) Operation canceled)) v8 ==== 274+0+0 (crc 0 0 0) 0x7fa25800bc70 con 0x55841eba0310 2023-10-05T19:28:24.528 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.522+0000 7fa26411be40 1 -- 172.21.15.167:0/3590574458 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:9 6.12 6:4f78e9a2:gc::gc.8:head [create,call version.check_conds in=74b,call rgw_gc.rgw_gc_queue_init in=45b,call version.set in=34b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x55841eb98150 con 0x55841eba78d0 2023-10-05T19:28:24.530 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.526+0000 7fa261c50700 1 -- 172.21.15.167:0/3590574458 <== osd.2 v2:172.21.15.167:6816/30037 2 ==== osd_op_reply(9 gc.8 [create,call,call,call] v30'12 uv3 ondisk = -125 ((125) Operation canceled)) v8 ==== 274+0+0 (crc 0 0 0) 0x7fa25c040040 con 0x55841eba78d0 2023-10-05T19:28:24.530 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.526+0000 7fa26411be40 1 -- 172.21.15.167:0/3590574458 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:10 6.1c 6:3e426232:gc::gc.9:head [create,call version.check_conds in=74b,call rgw_gc.rgw_gc_queue_init in=45b,call version.set in=34b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x55841eb98150 con 0x55841eba3aa0 2023-10-05T19:28:24.533 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.526+0000 7fa262451700 1 -- 172.21.15.167:0/3590574458 <== osd.3 v2:172.21.15.167:6808/30036 3 ==== osd_op_reply(10 gc.9 [create,call,call,call] v30'18 uv3 ondisk = -125 ((125) Operation canceled)) v8 ==== 274+0+0 (crc 0 0 0) 0x7fa25402f080 con 0x55841eba3aa0 2023-10-05T19:28:24.533 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.526+0000 7fa26411be40 1 -- 172.21.15.167:0/3590574458 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:11 6.3 6:c31f41d7:gc::gc.10:head [create,call version.check_conds in=74b,call rgw_gc.rgw_gc_queue_init in=45b,call version.set in=34b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x55841ebabfb0 con 0x55841eba78d0 2023-10-05T19:28:24.536 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.530+0000 7fa261c50700 1 -- 172.21.15.167:0/3590574458 <== osd.2 v2:172.21.15.167:6816/30037 3 ==== osd_op_reply(11 gc.10 [create,call,call,call] v30'27 uv11 ondisk = -125 ((125) Operation canceled)) v8 ==== 275+0+0 (crc 0 0 0) 0x7fa25c040040 con 0x55841eba78d0 2023-10-05T19:28:24.536 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.530+0000 7fa26411be40 1 -- 172.21.15.167:0/3590574458 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:12 6.13 6:c8333aab:gc::gc.11:head [create,call version.check_conds in=74b,call rgw_gc.rgw_gc_queue_init in=45b,call version.set in=34b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x55841ebabfb0 con 0x55841eba0310 2023-10-05T19:28:24.538 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.534+0000 7fa26144f700 1 -- 172.21.15.167:0/3590574458 <== osd.0 v2:172.21.15.118:6800/27690 3 ==== osd_op_reply(12 gc.11 [create,call,call,call] v30'14 uv5 ondisk = -125 ((125) Operation canceled)) v8 ==== 275+0+0 (crc 0 0 0) 0x7fa25800bc70 con 0x55841eba0310 2023-10-05T19:28:24.538 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.534+0000 7fa26411be40 1 -- 172.21.15.167:0/3590574458 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:13 6.15 6:ab569f81:gc::gc.12:head [create,call version.check_conds in=74b,call rgw_gc.rgw_gc_queue_init in=45b,call version.set in=34b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x55841ebac4a0 con 0x55841eba78d0 2023-10-05T19:28:24.541 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.534+0000 7fa261c50700 1 -- 172.21.15.167:0/3590574458 <== osd.2 v2:172.21.15.167:6816/30037 4 ==== osd_op_reply(13 gc.12 [create,call,call,call] v30'12 uv3 ondisk = -125 ((125) Operation canceled)) v8 ==== 275+0+0 (crc 0 0 0) 0x7fa25c040040 con 0x55841eba78d0 2023-10-05T19:28:24.541 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.534+0000 7fa26411be40 1 -- 172.21.15.167:0/3590574458 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:14 6.0 6:0667bb13:gc::gc.13:head [create,call version.check_conds in=74b,call rgw_gc.rgw_gc_queue_init in=45b,call version.set in=34b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x55841ebac4a0 con 0x55841eba0310 2023-10-05T19:28:24.544 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.538+0000 7fa26144f700 1 -- 172.21.15.167:0/3590574458 <== osd.0 v2:172.21.15.118:6800/27690 4 ==== osd_op_reply(14 gc.13 [create,call,call,call] v30'23 uv6 ondisk = -125 ((125) Operation canceled)) v8 ==== 275+0+0 (crc 0 0 0) 0x7fa25800bc70 con 0x55841eba0310 2023-10-05T19:28:24.544 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.538+0000 7fa26411be40 1 -- 172.21.15.167:0/3590574458 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:15 6.4 6:227f63b5:gc::gc.14:head [create,call version.check_conds in=74b,call rgw_gc.rgw_gc_queue_init in=45b,call version.set in=34b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x55841ebac710 con 0x55841eb9c860 2023-10-05T19:28:24.547 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.542+0000 7fa261c50700 1 -- 172.21.15.167:0/3590574458 <== osd.1 v2:172.21.15.167:6800/30035 4 ==== osd_op_reply(15 gc.14 [create,call,call,call] v30'38 uv13 ondisk = -125 ((125) Operation canceled)) v8 ==== 275+0+0 (crc 0 0 0) 0x7fa25c040040 con 0x55841eb9c860 2023-10-05T19:28:24.547 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.542+0000 7fa26411be40 1 -- 172.21.15.167:0/3590574458 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:16 6.7 6:e07a6548:gc::gc.15:head [create,call version.check_conds in=74b,call rgw_gc.rgw_gc_queue_init in=45b,call version.set in=34b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x55841ebac710 con 0x55841eba3aa0 2023-10-05T19:28:24.550 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.546+0000 7fa262451700 1 -- 172.21.15.167:0/3590574458 <== osd.3 v2:172.21.15.167:6808/30036 4 ==== osd_op_reply(16 gc.15 [create,call,call,call] v30'20 uv7 ondisk = -125 ((125) Operation canceled)) v8 ==== 275+0+0 (crc 0 0 0) 0x7fa25402f080 con 0x55841eba3aa0 2023-10-05T19:28:24.550 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.546+0000 7fa26411be40 1 -- 172.21.15.167:0/3590574458 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:17 6.3 6:c623b173:gc::gc.16:head [create,call version.check_conds in=74b,call rgw_gc.rgw_gc_queue_init in=45b,call version.set in=34b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x55841ebac710 con 0x55841eba78d0 2023-10-05T19:28:24.552 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.546+0000 7fa261c50700 1 -- 172.21.15.167:0/3590574458 <== osd.2 v2:172.21.15.167:6816/30037 5 ==== osd_op_reply(17 gc.16 [create,call,call,call] v30'28 uv13 ondisk = -125 ((125) Operation canceled)) v8 ==== 275+0+0 (crc 0 0 0) 0x7fa25c040040 con 0x55841eba78d0 2023-10-05T19:28:24.553 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.546+0000 7fa26411be40 1 -- 172.21.15.167:0/3590574458 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:18 6.16 6:6849fd06:gc::gc.17:head [create,call version.check_conds in=74b,call rgw_gc.rgw_gc_queue_init in=45b,call version.set in=34b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x55841ebac710 con 0x55841eba3aa0 2023-10-05T19:28:24.555 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.550+0000 7fa262451700 1 -- 172.21.15.167:0/3590574458 <== osd.3 v2:172.21.15.167:6808/30036 5 ==== osd_op_reply(18 gc.17 [create,call,call,call] v30'12 uv7 ondisk = -125 ((125) Operation canceled)) v8 ==== 275+0+0 (crc 0 0 0) 0x7fa25402f080 con 0x55841eba3aa0 2023-10-05T19:28:24.555 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.550+0000 7fa26411be40 1 -- 172.21.15.167:0/3590574458 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:19 6.d 6:b019ea83:gc::gc.18:head [create,call version.check_conds in=74b,call rgw_gc.rgw_gc_queue_init in=45b,call version.set in=34b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x55841ebac7a0 con 0x55841eb9c860 2023-10-05T19:28:24.558 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.554+0000 7fa261c50700 1 -- 172.21.15.167:0/3590574458 <== osd.1 v2:172.21.15.167:6800/30035 5 ==== osd_op_reply(19 gc.18 [create,call,call,call] v30'20 uv5 ondisk = -125 ((125) Operation canceled)) v8 ==== 275+0+0 (crc 0 0 0) 0x7fa25c040040 con 0x55841eb9c860 2023-10-05T19:28:24.558 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.554+0000 7fa26411be40 1 -- 172.21.15.167:0/3590574458 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:20 6.4 6:24c8e1ae:gc::gc.19:head [create,call version.check_conds in=74b,call rgw_gc.rgw_gc_queue_init in=45b,call version.set in=34b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x55841ebac7a0 con 0x55841eb9c860 2023-10-05T19:28:24.561 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.554+0000 7fa261c50700 1 -- 172.21.15.167:0/3590574458 <== osd.1 v2:172.21.15.167:6800/30035 6 ==== osd_op_reply(20 gc.19 [create,call,call,call] v30'39 uv15 ondisk = -125 ((125) Operation canceled)) v8 ==== 275+0+0 (crc 0 0 0) 0x7fa25c040040 con 0x55841eb9c860 2023-10-05T19:28:24.561 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.554+0000 7fa26411be40 1 -- 172.21.15.167:0/3590574458 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:21 6.7 6:e622f76a:gc::gc.20:head [create,call version.check_conds in=74b,call rgw_gc.rgw_gc_queue_init in=45b,call version.set in=34b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x55841ebac7a0 con 0x55841eba3aa0 2023-10-05T19:28:24.564 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.558+0000 7fa262451700 1 -- 172.21.15.167:0/3590574458 <== osd.3 v2:172.21.15.167:6808/30036 6 ==== osd_op_reply(21 gc.20 [create,call,call,call] v30'21 uv9 ondisk = -125 ((125) Operation canceled)) v8 ==== 275+0+0 (crc 0 0 0) 0x7fa25402f080 con 0x55841eba3aa0 2023-10-05T19:28:24.564 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.558+0000 7fa26411be40 1 -- 172.21.15.167:0/3590574458 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:22 6.a 6:5127e642:gc::gc.21:head [create,call version.check_conds in=74b,call rgw_gc.rgw_gc_queue_init in=45b,call version.set in=34b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x55841ebac830 con 0x55841eba78d0 2023-10-05T19:28:24.566 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.562+0000 7fa261c50700 1 -- 172.21.15.167:0/3590574458 <== osd.2 v2:172.21.15.167:6816/30037 6 ==== osd_op_reply(22 gc.21 [create,call,call,call] v30'22 uv7 ondisk = -125 ((125) Operation canceled)) v8 ==== 275+0+0 (crc 0 0 0) 0x7fa25c040040 con 0x55841eba78d0 2023-10-05T19:28:24.567 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.562+0000 7fa26411be40 1 -- 172.21.15.167:0/3590574458 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:23 6.11 6:8eb99c53:gc::gc.22:head [create,call version.check_conds in=74b,call rgw_gc.rgw_gc_queue_init in=45b,call version.set in=34b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x55841ebac830 con 0x55841eba3aa0 2023-10-05T19:28:24.569 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.562+0000 7fa262451700 1 -- 172.21.15.167:0/3590574458 <== osd.3 v2:172.21.15.167:6808/30036 7 ==== osd_op_reply(23 gc.22 [create,call,call,call] v30'14 uv3 ondisk = -125 ((125) Operation canceled)) v8 ==== 275+0+0 (crc 0 0 0) 0x7fa25402f080 con 0x55841eba3aa0 2023-10-05T19:28:24.569 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.562+0000 7fa26411be40 1 -- 172.21.15.167:0/3590574458 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:24 6.8 6:17d7e8ed:gc::gc.23:head [create,call version.check_conds in=74b,call rgw_gc.rgw_gc_queue_init in=45b,call version.set in=34b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x55841ebac8c0 con 0x55841eba3aa0 2023-10-05T19:28:24.571 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.566+0000 7fa262451700 1 -- 172.21.15.167:0/3590574458 <== osd.3 v2:172.21.15.167:6808/30036 8 ==== osd_op_reply(24 gc.23 [create,call,call,call] v30'19 uv5 ondisk = -125 ((125) Operation canceled)) v8 ==== 275+0+0 (crc 0 0 0) 0x7fa25402f080 con 0x55841eba3aa0 2023-10-05T19:28:24.572 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.566+0000 7fa26411be40 1 -- 172.21.15.167:0/3590574458 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:25 6.1b 6:db9d130d:gc::gc.24:head [create,call version.check_conds in=74b,call rgw_gc.rgw_gc_queue_init in=45b,call version.set in=34b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x55841ebac8c0 con 0x55841eba3aa0 2023-10-05T19:28:24.575 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.570+0000 7fa262451700 1 -- 172.21.15.167:0/3590574458 <== osd.3 v2:172.21.15.167:6808/30036 9 ==== osd_op_reply(25 gc.24 [create,call,call,call] v30'8 uv3 ondisk = -125 ((125) Operation canceled)) v8 ==== 275+0+0 (crc 0 0 0) 0x7fa25402f080 con 0x55841eba3aa0 2023-10-05T19:28:24.575 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.570+0000 7fa26411be40 1 -- 172.21.15.167:0/3590574458 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:26 6.9 6:921ecdb1:gc::gc.25:head [create,call version.check_conds in=74b,call rgw_gc.rgw_gc_queue_init in=45b,call version.set in=34b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x55841ebac8c0 con 0x55841eba3aa0 2023-10-05T19:28:24.577 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.570+0000 7fa262451700 1 -- 172.21.15.167:0/3590574458 <== osd.3 v2:172.21.15.167:6808/30036 10 ==== osd_op_reply(26 gc.25 [create,call,call,call] v30'18 uv10 ondisk = -125 ((125) Operation canceled)) v8 ==== 275+0+0 (crc 0 0 0) 0x7fa25402f080 con 0x55841eba3aa0 2023-10-05T19:28:24.577 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.570+0000 7fa26411be40 1 -- 172.21.15.167:0/3590574458 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:27 6.1f 6:fde4dd55:gc::gc.26:head [create,call version.check_conds in=74b,call rgw_gc.rgw_gc_queue_init in=45b,call version.set in=34b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x55841ebac8c0 con 0x55841eba0310 2023-10-05T19:28:24.580 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.574+0000 7fa26144f700 1 -- 172.21.15.167:0/3590574458 <== osd.0 v2:172.21.15.118:6800/27690 5 ==== osd_op_reply(27 gc.26 [create,call,call,call] v30'14 uv5 ondisk = -125 ((125) Operation canceled)) v8 ==== 275+0+0 (crc 0 0 0) 0x7fa25800bc70 con 0x55841eba0310 2023-10-05T19:28:24.580 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.574+0000 7fa26411be40 1 -- 172.21.15.167:0/3590574458 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:28 6.1a 6:5909e671:gc::gc.27:head [create,call version.check_conds in=74b,call rgw_gc.rgw_gc_queue_init in=45b,call version.set in=34b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x55841ebac8c0 con 0x55841eba3aa0 2023-10-05T19:28:24.582 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.578+0000 7fa262451700 1 -- 172.21.15.167:0/3590574458 <== osd.3 v2:172.21.15.167:6808/30036 11 ==== osd_op_reply(28 gc.27 [create,call,call,call] v30'12 uv7 ondisk = -125 ((125) Operation canceled)) v8 ==== 275+0+0 (crc 0 0 0) 0x7fa25402f080 con 0x55841eba3aa0 2023-10-05T19:28:24.582 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.578+0000 7fa26411be40 1 -- 172.21.15.167:0/3590574458 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:29 6.1d 6:b88e3271:gc::gc.28:head [create,call version.check_conds in=74b,call rgw_gc.rgw_gc_queue_init in=45b,call version.set in=34b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x55841ebac8c0 con 0x55841eba0310 2023-10-05T19:28:24.585 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.578+0000 7fa26144f700 1 -- 172.21.15.167:0/3590574458 <== osd.0 v2:172.21.15.118:6800/27690 6 ==== osd_op_reply(29 gc.28 [create,call,call,call] v30'18 uv8 ondisk = -125 ((125) Operation canceled)) v8 ==== 275+0+0 (crc 0 0 0) 0x7fa25800bc70 con 0x55841eba0310 2023-10-05T19:28:24.586 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.578+0000 7fa26411be40 1 -- 172.21.15.167:0/3590574458 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:30 6.7 6:e47c9322:gc::gc.29:head [create,call version.check_conds in=74b,call rgw_gc.rgw_gc_queue_init in=45b,call version.set in=34b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x55841ebac8c0 con 0x55841eba3aa0 2023-10-05T19:28:24.588 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.582+0000 7fa262451700 1 -- 172.21.15.167:0/3590574458 <== osd.3 v2:172.21.15.167:6808/30036 12 ==== osd_op_reply(30 gc.29 [create,call,call,call] v30'22 uv11 ondisk = -125 ((125) Operation canceled)) v8 ==== 275+0+0 (crc 0 0 0) 0x7fa25402f080 con 0x55841eba3aa0 2023-10-05T19:28:24.588 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.582+0000 7fa26411be40 1 -- 172.21.15.167:0/3590574458 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:31 6.0 6:013d6c33:gc::gc.30:head [create,call version.check_conds in=74b,call rgw_gc.rgw_gc_queue_init in=45b,call version.set in=34b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x55841ebac8c0 con 0x55841eba0310 2023-10-05T19:28:24.591 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.586+0000 7fa26144f700 1 -- 172.21.15.167:0/3590574458 <== osd.0 v2:172.21.15.118:6800/27690 7 ==== osd_op_reply(31 gc.30 [create,call,call,call] v30'24 uv12 ondisk = -125 ((125) Operation canceled)) v8 ==== 275+0+0 (crc 0 0 0) 0x7fa25800bc70 con 0x55841eba0310 2023-10-05T19:28:24.592 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.586+0000 7fa26411be40 1 -- 172.21.15.167:0/3590574458 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:32 6.f 6:f24cb4a8:gc::gc.31:head [create,call version.check_conds in=74b,call rgw_gc.rgw_gc_queue_init in=45b,call version.set in=34b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x55841ebac8c0 con 0x55841eba78d0 2023-10-05T19:28:24.594 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.590+0000 7fa261c50700 1 -- 172.21.15.167:0/3590574458 <== osd.2 v2:172.21.15.167:6816/30037 7 ==== osd_op_reply(32 gc.31 [create,call,call,call] v30'18 uv5 ondisk = -125 ((125) Operation canceled)) v8 ==== 275+0+0 (crc 0 0 0) 0x7fa25c040040 con 0x55841eba78d0 2023-10-05T19:28:24.597 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.590+0000 7fa1a1ffb700 20 reqs_thread_entry: start 2023-10-05T19:28:24.597 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.590+0000 7fa1937fe700 20 reqs_thread_entry: start 2023-10-05T19:28:24.598 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.590+0000 7fa26411be40 20 init_complete bucket index max shards: 11 2023-10-05T19:28:24.598 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.594+0000 7fa26411be40 10 rgw notify: Started notification manager with: 1 workers 2023-10-05T19:28:24.598 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.594+0000 7fa191ffb700 20 rgw notify: INFO: next queues processing will happen at: Thu Oct 5 19:28:55 2023 2023-10-05T19:28:24.599 INFO:teuthology.orchestra.run.smithi167.stderr: 2023-10-05T19:28:24.599 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.594+0000 7fa1917fa700 20 reqs_thread_entry: start 2023-10-05T19:28:24.599 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.594+0000 7fa26411be40 20 get_system_obj_state: rctx=0x7fff8881ecb8 obj=default.rgw.meta:users.uid:fud state=0x55841ea150b0 s->prefetch_data=0 2023-10-05T19:28:24.599 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.594+0000 7fa26411be40 10 cache get: name=default.rgw.meta+users.uid+fud : miss 2023-10-05T19:28:24.599 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.594+0000 7fa26411be40 1 -- 172.21.15.167:0/3175752868 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:102 8.e 8:75f48285:users.uid::fud:head [call version.read in=11b,getxattrs,stat] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x55841ec0b2d0 con 0x7fa25800f8d0 2023-10-05T19:28:24.600 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.594+0000 7fa262451700 1 -- 172.21.15.167:0/3175752868 <== osd.0 v2:172.21.15.118:6800/27690 29 ==== osd_op_reply(102 fud [call,getxattrs,stat] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 ==== 231+0+0 (crc 0 0 0) 0x7fa25402f080 con 0x7fa25800f8d0 2023-10-05T19:28:24.600 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.594+0000 7fa26411be40 10 cache put: name=default.rgw.meta+users.uid+fud info.flags=0x0 2023-10-05T19:28:24.600 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.594+0000 7fa26411be40 10 adding default.rgw.meta+users.uid+fud to cache LRU end 2023-10-05T19:28:24.600 INFO:teuthology.orchestra.run.smithi167.stderr:could not remove user: unable to remove user, user does not exist 2023-10-05T19:28:24.600 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.594+0000 7fa26411be40 1 -- 172.21.15.167:0/3175752868 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:103 7.1f 7:f95f44c2:::notify.0:head [watch unwatch cookie 94025939388016] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x55841ec0b2d0 con 0x7fa25c010070 2023-10-05T19:28:24.603 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.598+0000 7fa26144f700 1 -- 172.21.15.167:0/3175752868 <== osd.2 v2:172.21.15.167:6816/30037 17 ==== osd_op_reply(103 notify.0 [watch unwatch cookie 94025939388016] v30'14 uv12 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7fa25800bc70 con 0x7fa25c010070 2023-10-05T19:28:24.603 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.598+0000 7fa26411be40 20 remove_watcher() i=0 2023-10-05T19:28:24.603 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.598+0000 7fa26411be40 2 removed watcher, disabling cache 2023-10-05T19:28:24.603 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.598+0000 7fa26411be40 1 -- 172.21.15.167:0/3175752868 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:104 7.0 7:05bf5b68:::notify.1:head [watch unwatch cookie 94025939391536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x55841ec0b2d0 con 0x55841eb3c7e0 2023-10-05T19:28:24.605 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.598+0000 7fa262451700 1 -- 172.21.15.167:0/3175752868 <== osd.1 v2:172.21.15.167:6800/30035 35 ==== osd_op_reply(104 notify.1 [watch unwatch cookie 94025939391536] v30'14 uv12 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7fa25402f080 con 0x55841eb3c7e0 2023-10-05T19:28:24.606 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.598+0000 7fa26411be40 20 remove_watcher() i=1 2023-10-05T19:28:24.606 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.598+0000 7fa26411be40 1 -- 172.21.15.167:0/3175752868 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:105 7.15 7:a93a5511:::notify.2:head [watch unwatch cookie 94025939392640] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x55841ec0b2d0 con 0x55841eb735f0 2023-10-05T19:28:24.608 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.602+0000 7fa261c50700 1 -- 172.21.15.167:0/3175752868 <== osd.3 v2:172.21.15.167:6808/30036 24 ==== osd_op_reply(105 notify.2 [watch unwatch cookie 94025939392640] v30'14 uv12 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7fa25c040040 con 0x55841eb735f0 2023-10-05T19:28:24.608 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.602+0000 7fa26411be40 20 remove_watcher() i=2 2023-10-05T19:28:24.609 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.602+0000 7fa26411be40 1 -- 172.21.15.167:0/3175752868 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:106 7.e 7:7759931f:::notify.3:head [watch unwatch cookie 94025939400912] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x55841ec0b2d0 con 0x7fa25c010070 2023-10-05T19:28:24.611 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.606+0000 7fa26144f700 1 -- 172.21.15.167:0/3175752868 <== osd.2 v2:172.21.15.167:6816/30037 18 ==== osd_op_reply(106 notify.3 [watch unwatch cookie 94025939400912] v30'14 uv12 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7fa25800bc70 con 0x7fa25c010070 2023-10-05T19:28:24.611 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.606+0000 7fa26411be40 20 remove_watcher() i=3 2023-10-05T19:28:24.611 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.606+0000 7fa26411be40 1 -- 172.21.15.167:0/3175752868 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:107 7.d 7:b4812045:::notify.4:head [watch unwatch cookie 94025939394720] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x55841ec0b2d0 con 0x55841eb3c7e0 2023-10-05T19:28:24.613 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.606+0000 7fa262451700 1 -- 172.21.15.167:0/3175752868 <== osd.1 v2:172.21.15.167:6800/30035 36 ==== osd_op_reply(107 notify.4 [watch unwatch cookie 94025939394720] v30'14 uv12 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7fa25402f080 con 0x55841eb3c7e0 2023-10-05T19:28:24.614 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.606+0000 7fa26411be40 20 remove_watcher() i=4 2023-10-05T19:28:24.614 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.606+0000 7fa26411be40 1 -- 172.21.15.167:0/3175752868 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:108 7.3 7:c609908c:::notify.5:head [watch unwatch cookie 94025939404864] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x55841ec0b2d0 con 0x7fa25800f8d0 2023-10-05T19:28:24.617 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.610+0000 7fa262451700 1 -- 172.21.15.167:0/3175752868 <== osd.0 v2:172.21.15.118:6800/27690 30 ==== osd_op_reply(108 notify.5 [watch unwatch cookie 94025939404864] v30'14 uv12 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7fa25402f080 con 0x7fa25800f8d0 2023-10-05T19:28:24.617 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.610+0000 7fa26411be40 20 remove_watcher() i=5 2023-10-05T19:28:24.617 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.610+0000 7fa26411be40 1 -- 172.21.15.167:0/3175752868 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:109 7.14 7:2b04a3e9:::notify.6:head [watch unwatch cookie 94025939408640] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x55841ec0b2d0 con 0x7fa25800f8d0 2023-10-05T19:28:24.620 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.614+0000 7fa262451700 1 -- 172.21.15.167:0/3175752868 <== osd.0 v2:172.21.15.118:6800/27690 31 ==== osd_op_reply(109 notify.6 [watch unwatch cookie 94025939408640] v30'14 uv12 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7fa25402f080 con 0x7fa25800f8d0 2023-10-05T19:28:24.620 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.614+0000 7fa26411be40 20 remove_watcher() i=6 2023-10-05T19:28:24.620 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.614+0000 7fa26411be40 1 -- 172.21.15.167:0/3175752868 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:110 7.9 7:93e5b521:::notify.7:head [watch unwatch cookie 94025939409744] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x55841ec0b2d0 con 0x7fa25800f8d0 2023-10-05T19:28:24.623 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.618+0000 7fa262451700 1 -- 172.21.15.167:0/3175752868 <== osd.0 v2:172.21.15.118:6800/27690 32 ==== osd_op_reply(110 notify.7 [watch unwatch cookie 94025939409744] v30'14 uv12 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7fa25402f080 con 0x7fa25800f8d0 2023-10-05T19:28:24.623 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.618+0000 7fa26411be40 20 remove_watcher() i=7 2023-10-05T19:28:24.624 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.618+0000 7fa232fe5700 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2023-10-05T19:28:24.625 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.618+0000 7fa26411be40 1 -- 172.21.15.167:0/3175752868 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:111 6.4 6:22d26bf9:::data_loggenerations_metadata:head [watch unwatch cookie 94025939412144] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x55841eb7dbf0 con 0x55841eb3c7e0 2023-10-05T19:28:24.628 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.622+0000 7fa262451700 1 -- 172.21.15.167:0/3175752868 <== osd.1 v2:172.21.15.167:6800/30035 37 ==== osd_op_reply(111 data_loggenerations_metadata [watch unwatch cookie 94025939412144] v30'40 uv1 ondisk = 0) v8 ==== 172+0+0 (crc 0 0 0) 0x7fa25402f080 con 0x55841eb3c7e0 2023-10-05T19:28:24.628 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.622+0000 7fa26411be40 1 -- 172.21.15.167:0/3175752868 >> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] conn(0x7fa25800f8d0 msgr2=0x7fa25800e360 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-05T19:28:24.629 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.622+0000 7fa26411be40 1 --2- 172.21.15.167:0/3175752868 >> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] conn(0x7fa25800f8d0 0x7fa25800e360 crc :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-05T19:28:24.629 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.622+0000 7fa26411be40 1 -- 172.21.15.167:0/3175752868 >> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] conn(0x55841eb3c7e0 msgr2=0x55841eb6f810 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-05T19:28:24.629 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.622+0000 7fa26411be40 1 --2- 172.21.15.167:0/3175752868 >> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] conn(0x55841eb3c7e0 0x55841eb6f810 crc :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-05T19:28:24.629 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.622+0000 7fa26411be40 1 -- 172.21.15.167:0/3175752868 >> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] conn(0x7fa25c010070 msgr2=0x7fa25c01c6e0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-05T19:28:24.629 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.622+0000 7fa26411be40 1 --2- 172.21.15.167:0/3175752868 >> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] conn(0x7fa25c010070 0x7fa25c01c6e0 crc :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-05T19:28:24.629 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.622+0000 7fa26411be40 1 -- 172.21.15.167:0/3175752868 >> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] conn(0x55841eb735f0 msgr2=0x55841eb759d0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-05T19:28:24.629 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.622+0000 7fa26411be40 1 --2- 172.21.15.167:0/3175752868 >> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] conn(0x55841eb735f0 0x55841eb759d0 crc :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-05T19:28:24.629 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.622+0000 7fa26411be40 1 -- 172.21.15.167:0/3175752868 >> [v2:172.21.15.167:6824/29889,v1:172.21.15.167:6825/29889] conn(0x7fa244030ca0 msgr2=0x7fa244033150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-05T19:28:24.629 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.622+0000 7fa26411be40 1 --2- 172.21.15.167:0/3175752868 >> [v2:172.21.15.167:6824/29889,v1:172.21.15.167:6825/29889] conn(0x7fa244030ca0 0x7fa244033150 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7fa25800b2b0 tx=0x7fa258001490 comp rx=0 tx=0).stop 2023-10-05T19:28:24.629 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.622+0000 7fa262451700 1 -- 172.21.15.167:0/3175752868 reap_dead start 2023-10-05T19:28:24.629 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.622+0000 7fa26411be40 1 -- 172.21.15.167:0/3175752868 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55841eb3c410 msgr2=0x55841eb63d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-05T19:28:24.630 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.622+0000 7fa26411be40 1 --2- 172.21.15.167:0/3175752868 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55841eb3c410 0x55841eb63d80 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7fa25c005860 tx=0x7fa25c01c000 comp rx=0 tx=0).stop 2023-10-05T19:28:24.630 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.622+0000 7fa26411be40 1 -- 172.21.15.167:0/3175752868 shutdown_connections 2023-10-05T19:28:24.630 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.622+0000 7fa26411be40 1 --2- 172.21.15.167:0/3175752868 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55841eb3c410 0x55841eb63d80 unknown :-1 s=CLOSED pgs=155 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-05T19:28:24.630 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.622+0000 7fa26411be40 1 -- 172.21.15.167:0/3175752868 >> 172.21.15.167:0/3175752868 conn(0x55841eb36720 msgr2=0x55841eb57790 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-05T19:28:24.630 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.622+0000 7fa26411be40 1 -- 172.21.15.167:0/3175752868 shutdown_connections 2023-10-05T19:28:24.630 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.622+0000 7fa26411be40 1 -- 172.21.15.167:0/3175752868 wait complete. 2023-10-05T19:28:24.631 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.622+0000 7fa26411be40 1 -- 172.21.15.167:0/3590574458 >> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] conn(0x55841eba0310 msgr2=0x55841eba26f0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-05T19:28:24.631 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.622+0000 7fa26411be40 1 --2- 172.21.15.167:0/3590574458 >> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] conn(0x55841eba0310 0x55841eba26f0 crc :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-05T19:28:24.631 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.622+0000 7fa26411be40 1 -- 172.21.15.167:0/3590574458 >> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] conn(0x55841eb9c860 msgr2=0x55841eb9ec70 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-05T19:28:24.631 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.622+0000 7fa26411be40 1 --2- 172.21.15.167:0/3590574458 >> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] conn(0x55841eb9c860 0x55841eb9ec70 crc :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-05T19:28:24.631 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.622+0000 7fa26411be40 1 -- 172.21.15.167:0/3590574458 >> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] conn(0x55841eba78d0 msgr2=0x55841eba9cb0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-05T19:28:24.632 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.622+0000 7fa26411be40 1 --2- 172.21.15.167:0/3590574458 >> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] conn(0x55841eba78d0 0x55841eba9cb0 crc :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-05T19:28:24.632 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.622+0000 7fa26411be40 1 -- 172.21.15.167:0/3590574458 >> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] conn(0x55841eba3aa0 msgr2=0x55841eba5eb0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-05T19:28:24.632 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.622+0000 7fa26411be40 1 --2- 172.21.15.167:0/3590574458 >> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] conn(0x55841eba3aa0 0x55841eba5eb0 crc :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-05T19:28:24.632 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.622+0000 7fa26411be40 1 -- 172.21.15.167:0/3590574458 >> [v2:172.21.15.167:6824/29889,v1:172.21.15.167:6825/29889] conn(0x7fa1a8030c50 msgr2=0x7fa1a8033100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-05T19:28:24.632 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.622+0000 7fa26411be40 1 --2- 172.21.15.167:0/3590574458 >> [v2:172.21.15.167:6824/29889,v1:172.21.15.167:6825/29889] conn(0x7fa1a8030c50 0x7fa1a8033100 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x55841eb7e270 tx=0x7fa254000f90 comp rx=0 tx=0).stop 2023-10-05T19:28:24.632 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.622+0000 7fa26411be40 1 -- 172.21.15.167:0/3590574458 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55841eb8c060 msgr2=0x55841eb91c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-05T19:28:24.632 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.622+0000 7fa261c50700 1 -- 172.21.15.167:0/3590574458 reap_dead start 2023-10-05T19:28:24.632 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.622+0000 7fa26411be40 1 --2- 172.21.15.167:0/3590574458 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55841eb8c060 0x55841eb91c10 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7fa2580104c0 tx=0x7fa258031070 comp rx=0 tx=0).stop 2023-10-05T19:28:24.632 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.622+0000 7fa26411be40 1 -- 172.21.15.167:0/3590574458 shutdown_connections 2023-10-05T19:28:24.632 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.622+0000 7fa26411be40 1 --2- 172.21.15.167:0/3590574458 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55841eb8c060 0x55841eb91c10 unknown :-1 s=CLOSED pgs=157 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-05T19:28:24.632 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.622+0000 7fa26411be40 1 -- 172.21.15.167:0/3590574458 >> 172.21.15.167:0/3590574458 conn(0x55841eb87420 msgr2=0x55841eb854c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-05T19:28:24.632 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.622+0000 7fa26411be40 1 -- 172.21.15.167:0/3590574458 shutdown_connections 2023-10-05T19:28:24.633 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.622+0000 7fa26411be40 1 -- 172.21.15.167:0/3590574458 wait complete. 2023-10-05T19:28:24.635 DEBUG:teuthology.orchestra.run:got remote process result: 2 2023-10-05T19:28:24.635 INFO:tasks.util.rgw:rgwadmin: client.0 : ['user', 'rm', '--uid', 'foo'] 2023-10-05T19:28:24.635 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'ceph', 'user', 'rm', '--uid', 'foo'] 2023-10-05T19:28:24.635 DEBUG:teuthology.orchestra.run.smithi167:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster ceph user rm --uid foo 2023-10-05T19:28:24.716 INFO:teuthology.orchestra.run.smithi167.stderr:ignoring --setuser ceph since I am not root 2023-10-05T19:28:24.716 INFO:teuthology.orchestra.run.smithi167.stderr:ignoring --setgroup ceph since I am not root 2023-10-05T19:28:24.717 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.710+0000 7fb92a165e40 1 Processor -- start 2023-10-05T19:28:24.717 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.710+0000 7fb92a165e40 1 -- start start 2023-10-05T19:28:24.717 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.710+0000 7fb92a165e40 1 --2- >> v2:172.21.15.118:3300/0 conn(0x55f9284501a0 0x55f9284e3f40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-05T19:28:24.717 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.710+0000 7fb92a165e40 1 -- --> v1:172.21.15.118:6789/0 -- auth(proto 0 26 bytes epoch 0) v1 -- 0x55f928427bd0 con 0x55f9284514b0 2023-10-05T19:28:24.717 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.710+0000 7fb92a165e40 1 -- --> v2:172.21.15.118:3300/0 -- mon_getmap magic: 0 v1 -- 0x55f9283e4ee0 con 0x55f9284501a0 2023-10-05T19:28:24.718 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.710+0000 7fb927499700 1 --2- >> v2:172.21.15.118:3300/0 conn(0x55f9284501a0 0x55f9284e3f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-05T19:28:24.718 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.710+0000 7fb927499700 1 --2- >> v2:172.21.15.118:3300/0 conn(0x55f9284501a0 0x55f9284e3f40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.118:3300/0 says I am v2:172.21.15.167:56974/0 (socket says 172.21.15.167:56974) 2023-10-05T19:28:24.718 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.714+0000 7fb927499700 1 -- 172.21.15.167:0/3299313984 learned_addr learned my addr 172.21.15.167:0/3299313984 (peer_addr_for_me v2:172.21.15.167:0/0) 2023-10-05T19:28:24.718 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.714+0000 7fb927c9a700 1 --1- 172.21.15.167:0/3299313984 >> v1:172.21.15.118:6789/0 conn(0x55f9284514b0 0x55f92844fa90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.118:6789/0 says I am v1:172.21.15.167:56324/0 (socket says 172.21.15.167:56324) 2023-10-05T19:28:24.718 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.714+0000 7fb927499700 1 -- 172.21.15.167:0/3299313984 >> v1:172.21.15.118:6789/0 conn(0x55f9284514b0 legacy=0x55f92844fa90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-05T19:28:24.719 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.714+0000 7fb927499700 1 -- 172.21.15.167:0/3299313984 --> v2:172.21.15.118:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55f928451940 con 0x55f9284501a0 2023-10-05T19:28:24.719 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.714+0000 7fb927499700 1 --2- 172.21.15.167:0/3299313984 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55f9284501a0 0x55f9284e3f40 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7fb92000a5c0 tx=0x7fb920005e00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ff46da400c2df022 server_cookie=0 in_seq=0 out_seq=0 2023-10-05T19:28:24.719 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.714+0000 7fb926c98700 1 -- 172.21.15.167:0/3299313984 <== mon.0 v2:172.21.15.118:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fb92000e510 con 0x55f9284501a0 2023-10-05T19:28:24.719 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.714+0000 7fb926c98700 1 -- 172.21.15.167:0/3299313984 <== mon.0 v2:172.21.15.118:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb92000e670 con 0x55f9284501a0 2023-10-05T19:28:24.719 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.714+0000 7fb926c98700 1 -- 172.21.15.167:0/3299313984 <== mon.0 v2:172.21.15.118:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fb92000e920 con 0x55f9284501a0 2023-10-05T19:28:24.720 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.714+0000 7fb92a165e40 1 -- 172.21.15.167:0/3299313984 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55f9284501a0 msgr2=0x55f9284e3f40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-05T19:28:24.721 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.714+0000 7fb92a165e40 1 --2- 172.21.15.167:0/3299313984 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55f9284501a0 0x55f9284e3f40 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7fb92000a5c0 tx=0x7fb920005e00 comp rx=0 tx=0).stop 2023-10-05T19:28:24.721 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.714+0000 7fb92a165e40 1 -- 172.21.15.167:0/3299313984 shutdown_connections 2023-10-05T19:28:24.721 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.714+0000 7fb92a165e40 1 --2- 172.21.15.167:0/3299313984 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55f9284501a0 0x55f9284e3f40 unknown :-1 s=CLOSED pgs=159 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-05T19:28:24.721 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.714+0000 7fb92a165e40 1 -- 172.21.15.167:0/3299313984 >> 172.21.15.167:0/3299313984 conn(0x55f9284d6830 msgr2=0x55f9284d8c40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-05T19:28:24.721 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.714+0000 7fb92a165e40 1 -- 172.21.15.167:0/3299313984 shutdown_connections 2023-10-05T19:28:24.721 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.714+0000 7fb92a165e40 1 -- 172.21.15.167:0/3299313984 wait complete. 2023-10-05T19:28:24.721 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.714+0000 7fb92a165e40 1 Processor -- start 2023-10-05T19:28:24.721 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.714+0000 7fb92a165e40 1 -- start start 2023-10-05T19:28:24.721 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.714+0000 7fb92a165e40 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55f9284501a0 0x55f9284fc130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-05T19:28:24.721 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.714+0000 7fb92a165e40 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f9283a8e30 con 0x55f9284501a0 2023-10-05T19:28:24.722 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.714+0000 7fb927c9a700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55f9284501a0 0x55f9284fc130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-05T19:28:24.722 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.714+0000 7fb927c9a700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55f9284501a0 0x55f9284fc130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.118:3300/0 says I am v2:172.21.15.167:56990/0 (socket says 172.21.15.167:56990) 2023-10-05T19:28:24.722 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.714+0000 7fb927c9a700 1 -- 172.21.15.167:0/1269678577 learned_addr learned my addr 172.21.15.167:0/1269678577 (peer_addr_for_me v2:172.21.15.167:0/0) 2023-10-05T19:28:24.722 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.718+0000 7fb927c9a700 1 -- 172.21.15.167:0/1269678577 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb92000a180 con 0x55f9284501a0 2023-10-05T19:28:24.723 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.718+0000 7fb927c9a700 1 --2- 172.21.15.167:0/1269678577 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55f9284501a0 0x55f9284fc130 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7fb910001cf0 tx=0x7fb9100018a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b363cc5bec6eebcb server_cookie=0 in_seq=0 out_seq=0 2023-10-05T19:28:24.724 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.718+0000 7fb925c96700 1 -- 172.21.15.167:0/1269678577 <== mon.0 v2:172.21.15.118:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fb910014070 con 0x55f9284501a0 2023-10-05T19:28:24.724 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.718+0000 7fb925c96700 1 -- 172.21.15.167:0/1269678577 <== mon.0 v2:172.21.15.118:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb91000be10 con 0x55f9284501a0 2023-10-05T19:28:24.724 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.718+0000 7fb925c96700 1 -- 172.21.15.167:0/1269678577 <== mon.0 v2:172.21.15.118:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fb91000f590 con 0x55f9284501a0 2023-10-05T19:28:24.724 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.718+0000 7fb92a165e40 1 -- 172.21.15.167:0/1269678577 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55f9284501a0 msgr2=0x55f9284fc130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-05T19:28:24.724 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.718+0000 7fb92a165e40 1 --2- 172.21.15.167:0/1269678577 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55f9284501a0 0x55f9284fc130 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7fb910001cf0 tx=0x7fb9100018a0 comp rx=0 tx=0).stop 2023-10-05T19:28:24.724 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.718+0000 7fb92a165e40 1 -- 172.21.15.167:0/1269678577 shutdown_connections 2023-10-05T19:28:24.724 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.718+0000 7fb92a165e40 1 --2- 172.21.15.167:0/1269678577 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55f9284501a0 0x55f9284fc130 unknown :-1 s=CLOSED pgs=160 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-05T19:28:24.724 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.718+0000 7fb92a165e40 1 -- 172.21.15.167:0/1269678577 >> 172.21.15.167:0/1269678577 conn(0x55f9284d6830 msgr2=0x55f9284f7cf0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-05T19:28:24.724 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.718+0000 7fb92a165e40 1 -- 172.21.15.167:0/1269678577 shutdown_connections 2023-10-05T19:28:24.724 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.718+0000 7fb92a165e40 1 -- 172.21.15.167:0/1269678577 wait complete. 2023-10-05T19:28:24.725 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.718+0000 7fb92a165e40 1 Processor -- start 2023-10-05T19:28:24.725 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.718+0000 7fb92a165e40 1 -- start start 2023-10-05T19:28:24.726 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.718+0000 7fb92a165e40 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55f9284501a0 0x55f928502f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-05T19:28:24.726 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.718+0000 7fb92a165e40 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f92837bed0 con 0x55f9284501a0 2023-10-05T19:28:24.726 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.718+0000 7fb927c9a700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55f9284501a0 0x55f928502f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-05T19:28:24.726 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.718+0000 7fb927c9a700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55f9284501a0 0x55f928502f00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.118:3300/0 says I am v2:172.21.15.167:56998/0 (socket says 172.21.15.167:56998) 2023-10-05T19:28:24.726 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.718+0000 7fb927c9a700 1 -- 172.21.15.167:0/2080866673 learned_addr learned my addr 172.21.15.167:0/2080866673 (peer_addr_for_me v2:172.21.15.167:0/0) 2023-10-05T19:28:24.726 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.718+0000 7fb927c9a700 1 -- 172.21.15.167:0/2080866673 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55f928427bd0 con 0x55f9284501a0 2023-10-05T19:28:24.726 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.718+0000 7fb927c9a700 1 --2- 172.21.15.167:0/2080866673 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55f9284501a0 0x55f928502f00 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7fb9100018e0 tx=0x7fb910016000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-05T19:28:24.727 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.722+0000 7fb91ffff700 1 -- 172.21.15.167:0/2080866673 <== mon.0 v2:172.21.15.118:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fb91001c670 con 0x55f9284501a0 2023-10-05T19:28:24.727 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.722+0000 7fb92a165e40 1 -- 172.21.15.167:0/2080866673 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x55f928503440 con 0x55f9284501a0 2023-10-05T19:28:24.727 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.722+0000 7fb91ffff700 1 -- 172.21.15.167:0/2080866673 <== mon.0 v2:172.21.15.118:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb91001c7f0 con 0x55f9284501a0 2023-10-05T19:28:24.727 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.722+0000 7fb91ffff700 1 -- 172.21.15.167:0/2080866673 <== mon.0 v2:172.21.15.118:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fb91001cae0 con 0x55f9284501a0 2023-10-05T19:28:24.727 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.722+0000 7fb92a165e40 1 -- 172.21.15.167:0/2080866673 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55f928503880 con 0x55f9284501a0 2023-10-05T19:28:24.727 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.722+0000 7fb92a165e40 20 rados->read ofs=0 len=0 2023-10-05T19:28:24.728 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.722+0000 7fb91ffff700 1 -- 172.21.15.167:0/2080866673 <== mon.0 v2:172.21.15.118:3300/0 4 ==== mgrmap(e 5) v1 ==== 40329+0+0 (secure 0 0 0) 0x7fb910023020 con 0x55f9284501a0 2023-10-05T19:28:24.728 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.722+0000 7fb91ffff700 1 --2- 172.21.15.167:0/2080866673 >> [v2:172.21.15.167:6824/29889,v1:172.21.15.167:6825/29889] conn(0x7fb90c030c50 0x7fb90c033100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-05T19:28:24.729 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.722+0000 7fb91ffff700 1 -- 172.21.15.167:0/2080866673 <== mon.0 v2:172.21.15.118:3300/0 5 ==== osd_map(30..30 src has 1..30) v4 ==== 6057+0+0 (secure 0 0 0) 0x7fb910008030 con 0x55f9284501a0 2023-10-05T19:28:24.729 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.722+0000 7fb927499700 1 --2- 172.21.15.167:0/2080866673 >> [v2:172.21.15.167:6824/29889,v1:172.21.15.167:6825/29889] conn(0x7fb90c030c50 0x7fb90c033100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-05T19:28:24.729 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.722+0000 7fb92a165e40 1 --2- 172.21.15.167:0/2080866673 >> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] conn(0x55f9284514b0 0x55f92850e9f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-05T19:28:24.729 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.722+0000 7fb92a165e40 1 -- 172.21.15.167:0/2080866673 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:1 5.12 5:49953fa1:::default.realm:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x55f92850f0f0 con 0x55f9284514b0 2023-10-05T19:28:24.729 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.722+0000 7fb92849b700 1 --2- 172.21.15.167:0/2080866673 >> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] conn(0x55f9284514b0 0x55f92850e9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-05T19:28:24.730 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.722+0000 7fb927499700 1 --2- 172.21.15.167:0/2080866673 >> [v2:172.21.15.167:6824/29889,v1:172.21.15.167:6825/29889] conn(0x7fb90c030c50 0x7fb90c033100 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7fb920000f80 tx=0x7fb920013000 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-05T19:28:24.730 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.726+0000 7fb92849b700 1 --2- 172.21.15.167:0/2080866673 >> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] conn(0x55f9284514b0 0x55f92850e9f0 crc :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-05T19:28:24.731 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.726+0000 7fb92849b700 1 -- 172.21.15.167:0/2080866673 <== osd.1 v2:172.21.15.167:6800/30035 1 ==== osd_op_reply(1 default.realm [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 ==== 157+0+0 (crc 0 0 0) 0x7fb91800a500 con 0x55f9284514b0 2023-10-05T19:28:24.731 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.726+0000 7fb92a165e40 20 rados_obj.operate() r=-2 bl.length=0 2023-10-05T19:28:24.731 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.726+0000 7fb92a165e40 20 realm 2023-10-05T19:28:24.731 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.726+0000 7fb92a165e40 20 rados->read ofs=0 len=0 2023-10-05T19:28:24.731 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.726+0000 7fb92a165e40 1 -- 172.21.15.167:0/2080866673 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:2 5.12 5:49953fa1:::default.realm:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x55f92850c050 con 0x55f9284514b0 2023-10-05T19:28:24.732 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.726+0000 7fb92849b700 1 -- 172.21.15.167:0/2080866673 <== osd.1 v2:172.21.15.167:6800/30035 2 ==== osd_op_reply(2 default.realm [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 ==== 157+0+0 (crc 0 0 0) 0x7fb91800a500 con 0x55f9284514b0 2023-10-05T19:28:24.732 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.726+0000 7fb92a165e40 20 rados_obj.operate() r=-2 bl.length=0 2023-10-05T19:28:24.732 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.726+0000 7fb92a165e40 4 RGWPeriod::init failed to init realm id : (2) No such file or directory 2023-10-05T19:28:24.732 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.726+0000 7fb92a165e40 20 rados->read ofs=0 len=0 2023-10-05T19:28:24.733 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.726+0000 7fb92a165e40 1 -- 172.21.15.167:0/2080866673 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:3 5.12 5:49953fa1:::default.realm:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x55f92850fbd0 con 0x55f9284514b0 2023-10-05T19:28:24.733 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.726+0000 7fb92849b700 1 -- 172.21.15.167:0/2080866673 <== osd.1 v2:172.21.15.167:6800/30035 3 ==== osd_op_reply(3 default.realm [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 ==== 157+0+0 (crc 0 0 0) 0x7fb91800a500 con 0x55f9284514b0 2023-10-05T19:28:24.733 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.726+0000 7fb92a165e40 20 rados_obj.operate() r=-2 bl.length=0 2023-10-05T19:28:24.733 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.726+0000 7fb92a165e40 20 rados->read ofs=0 len=0 2023-10-05T19:28:24.733 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.726+0000 7fb92a165e40 1 -- 172.21.15.167:0/2080866673 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:4 5.f 5:f43b6ece:::zone_names.default:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x55f92850f7e0 con 0x55f9284514b0 2023-10-05T19:28:24.734 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.726+0000 7fb92849b700 1 -- 172.21.15.167:0/2080866673 <== osd.1 v2:172.21.15.167:6800/30035 4 ==== osd_op_reply(4 zone_names.default [read 0~46 out=46b] v0'0 uv1 ondisk = 0) v8 ==== 162+0+46 (crc 0 0 0) 0x7fb91800a500 con 0x55f9284514b0 2023-10-05T19:28:24.734 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.726+0000 7fb92a165e40 20 rados_obj.operate() r=0 bl.length=46 2023-10-05T19:28:24.734 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.726+0000 7fb92a165e40 20 rados->read ofs=0 len=0 2023-10-05T19:28:24.734 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.726+0000 7fb92a165e40 1 -- 172.21.15.167:0/2080866673 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:5 5.1d 5:b8374eb4:::zone_info.9a9df965-6612-45d4-a45f-6e3c889bacc2:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x55f928510a60 con 0x55f9284514b0 2023-10-05T19:28:24.735 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.726+0000 7fb92849b700 1 -- 172.21.15.167:0/2080866673 <== osd.1 v2:172.21.15.167:6800/30035 5 ==== osd_op_reply(5 zone_info.9a9df965-6612-45d4-a45f-6e3c889bacc2 [read 0~874 out=874b] v0'0 uv1 ondisk = 0) v8 ==== 190+0+874 (crc 0 0 0) 0x7fb91800a500 con 0x55f9284514b0 2023-10-05T19:28:24.735 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.726+0000 7fb92a165e40 20 rados_obj.operate() r=0 bl.length=874 2023-10-05T19:28:24.735 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.726+0000 7fb92a165e40 20 searching for the correct realm 2023-10-05T19:28:24.735 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.726+0000 7fb92a165e40 1 --2- 172.21.15.167:0/2080866673 >> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] conn(0x55f928512820 0x55f928514c00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-05T19:28:24.735 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.726+0000 7fb92a165e40 1 -- 172.21.15.167:0/2080866673 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:6 5.0 5:00000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x55f928515140 con 0x55f928512820 2023-10-05T19:28:24.735 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.730+0000 7fb927c9a700 1 --2- 172.21.15.167:0/2080866673 >> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] conn(0x55f928512820 0x55f928514c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-05T19:28:24.735 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.730+0000 7fb927c9a700 1 --2- 172.21.15.167:0/2080866673 >> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] conn(0x55f928512820 0x55f928514c00 crc :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-05T19:28:24.737 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.730+0000 7fb927c9a700 1 -- 172.21.15.167:0/2080866673 <== osd.3 v2:172.21.15.167:6808/30036 1 ==== osd_op_reply(6 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fb910016070 con 0x55f928512820 2023-10-05T19:28:24.737 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.730+0000 7fb927c9a700 1 --2- 172.21.15.167:0/2080866673 >> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] conn(0x7fb91001a070 0x7fb910043750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-05T19:28:24.737 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.730+0000 7fb927c9a700 1 -- 172.21.15.167:0/2080866673 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:7 5.10 5:08000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x55f928515140 con 0x7fb91001a070 2023-10-05T19:28:24.737 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.730+0000 7fb927499700 1 --2- 172.21.15.167:0/2080866673 >> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] conn(0x7fb91001a070 0x7fb910043750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-05T19:28:24.737 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.730+0000 7fb927499700 1 --2- 172.21.15.167:0/2080866673 >> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] conn(0x7fb91001a070 0x7fb910043750 crc :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-05T19:28:24.738 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.734+0000 7fb927499700 1 -- 172.21.15.167:0/2080866673 <== osd.2 v2:172.21.15.167:6816/30037 1 ==== osd_op_reply(7 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fb920017050 con 0x7fb91001a070 2023-10-05T19:28:24.738 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.734+0000 7fb927499700 1 --2- 172.21.15.167:0/2080866673 >> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] conn(0x7fb920010070 0x7fb92001c430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-05T19:28:24.739 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.734+0000 7fb927499700 1 -- 172.21.15.167:0/2080866673 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:8 5.8 5:10000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x55f928515140 con 0x7fb920010070 2023-10-05T19:28:24.739 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.734+0000 7fb92849b700 1 --2- 172.21.15.167:0/2080866673 >> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] conn(0x7fb920010070 0x7fb92001c430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-05T19:28:24.739 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.734+0000 7fb92849b700 1 --2- 172.21.15.167:0/2080866673 >> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] conn(0x7fb920010070 0x7fb92001c430 crc :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-05T19:28:24.740 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.734+0000 7fb92849b700 1 -- 172.21.15.167:0/2080866673 <== osd.0 v2:172.21.15.118:6800/27690 1 ==== osd_op_reply(8 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fb918014040 con 0x7fb920010070 2023-10-05T19:28:24.740 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.734+0000 7fb92849b700 1 -- 172.21.15.167:0/2080866673 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:9 5.18 5:18000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x55f928515140 con 0x55f9284514b0 2023-10-05T19:28:24.741 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.734+0000 7fb92849b700 1 -- 172.21.15.167:0/2080866673 <== osd.1 v2:172.21.15.167:6800/30035 6 ==== osd_op_reply(9 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fb918014040 con 0x55f9284514b0 2023-10-05T19:28:24.741 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.734+0000 7fb92849b700 1 -- 172.21.15.167:0/2080866673 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:10 5.4 5:20000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x55f928515140 con 0x55f9284514b0 2023-10-05T19:28:24.742 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.738+0000 7fb92849b700 1 -- 172.21.15.167:0/2080866673 <== osd.1 v2:172.21.15.167:6800/30035 7 ==== osd_op_reply(10 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fb918014040 con 0x55f9284514b0 2023-10-05T19:28:24.742 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.738+0000 7fb92849b700 1 -- 172.21.15.167:0/2080866673 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:11 5.14 5:28000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x55f928515140 con 0x7fb920010070 2023-10-05T19:28:24.743 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.738+0000 7fb92849b700 1 -- 172.21.15.167:0/2080866673 <== osd.0 v2:172.21.15.118:6800/27690 2 ==== osd_op_reply(11 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fb918014040 con 0x7fb920010070 2023-10-05T19:28:24.743 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.738+0000 7fb92849b700 1 -- 172.21.15.167:0/2080866673 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:12 5.c 5:30000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x55f928515140 con 0x7fb920010070 2023-10-05T19:28:24.744 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.738+0000 7fb92849b700 1 -- 172.21.15.167:0/2080866673 <== osd.0 v2:172.21.15.118:6800/27690 3 ==== osd_op_reply(12 [pgnls start_epoch 30 out=112b] v23'1 uv1 ondisk = 1) v8 ==== 144+0+112 (crc 0 0 0) 0x7fb918014040 con 0x7fb920010070 2023-10-05T19:28:24.744 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.738+0000 7fb92849b700 1 -- 172.21.15.167:0/2080866673 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:13 5.1c 5:38000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x55f928515140 con 0x7fb91001a070 2023-10-05T19:28:24.745 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.738+0000 7fb927499700 1 -- 172.21.15.167:0/2080866673 <== osd.2 v2:172.21.15.167:6816/30037 2 ==== osd_op_reply(13 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fb920017050 con 0x7fb91001a070 2023-10-05T19:28:24.745 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.738+0000 7fb927499700 1 -- 172.21.15.167:0/2080866673 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:14 5.2 5:40000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x55f928515140 con 0x55f928512820 2023-10-05T19:28:24.746 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.738+0000 7fb927c9a700 1 -- 172.21.15.167:0/2080866673 <== osd.3 v2:172.21.15.167:6808/30036 2 ==== osd_op_reply(14 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fb910016070 con 0x55f928512820 2023-10-05T19:28:24.746 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.742+0000 7fb927c9a700 1 -- 172.21.15.167:0/2080866673 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:15 5.12 5:48000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x55f928515140 con 0x55f9284514b0 2023-10-05T19:28:24.747 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.742+0000 7fb92849b700 1 -- 172.21.15.167:0/2080866673 <== osd.1 v2:172.21.15.167:6800/30035 8 ==== osd_op_reply(15 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fb918014040 con 0x55f9284514b0 2023-10-05T19:28:24.747 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.742+0000 7fb92849b700 1 -- 172.21.15.167:0/2080866673 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:16 5.a 5:50000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x55f928515140 con 0x7fb920010070 2023-10-05T19:28:24.748 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.742+0000 7fb92849b700 1 -- 172.21.15.167:0/2080866673 <== osd.0 v2:172.21.15.118:6800/27690 4 ==== osd_op_reply(16 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fb918022040 con 0x7fb920010070 2023-10-05T19:28:24.748 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.742+0000 7fb92849b700 1 -- 172.21.15.167:0/2080866673 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:17 5.1a 5:58000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x55f928515140 con 0x55f9284514b0 2023-10-05T19:28:24.748 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.742+0000 7fb92849b700 1 -- 172.21.15.167:0/2080866673 <== osd.1 v2:172.21.15.167:6800/30035 9 ==== osd_op_reply(17 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fb918022040 con 0x55f9284514b0 2023-10-05T19:28:24.748 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.742+0000 7fb92849b700 1 -- 172.21.15.167:0/2080866673 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:18 5.6 5:60000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x55f928515140 con 0x7fb91001a070 2023-10-05T19:28:24.749 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.742+0000 7fb927499700 1 -- 172.21.15.167:0/2080866673 <== osd.2 v2:172.21.15.167:6816/30037 3 ==== osd_op_reply(18 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fb920017050 con 0x7fb91001a070 2023-10-05T19:28:24.749 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.742+0000 7fb927499700 1 -- 172.21.15.167:0/2080866673 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:19 5.16 5:68000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x55f928515140 con 0x55f928512820 2023-10-05T19:28:24.750 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.746+0000 7fb927c9a700 1 -- 172.21.15.167:0/2080866673 <== osd.3 v2:172.21.15.167:6808/30036 3 ==== osd_op_reply(19 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fb910016070 con 0x55f928512820 2023-10-05T19:28:24.750 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.746+0000 7fb927c9a700 1 -- 172.21.15.167:0/2080866673 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:20 5.e 5:70000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x55f928515140 con 0x55f928512820 2023-10-05T19:28:24.751 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.746+0000 7fb927c9a700 1 -- 172.21.15.167:0/2080866673 <== osd.3 v2:172.21.15.167:6808/30036 4 ==== osd_op_reply(20 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fb910016070 con 0x55f928512820 2023-10-05T19:28:24.751 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.746+0000 7fb927c9a700 1 -- 172.21.15.167:0/2080866673 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:21 5.1e 5:78000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x55f928515140 con 0x7fb920010070 2023-10-05T19:28:24.752 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.746+0000 7fb92849b700 1 -- 172.21.15.167:0/2080866673 <== osd.0 v2:172.21.15.118:6800/27690 5 ==== osd_op_reply(21 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fb918022040 con 0x7fb920010070 2023-10-05T19:28:24.752 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.746+0000 7fb92849b700 1 -- 172.21.15.167:0/2080866673 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:22 5.1 5:80000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x55f928515140 con 0x55f928512820 2023-10-05T19:28:24.753 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.746+0000 7fb927c9a700 1 -- 172.21.15.167:0/2080866673 <== osd.3 v2:172.21.15.167:6808/30036 5 ==== osd_op_reply(22 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fb910039040 con 0x55f928512820 2023-10-05T19:28:24.753 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.746+0000 7fb927c9a700 1 -- 172.21.15.167:0/2080866673 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:23 5.11 5:88000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x55f928515140 con 0x55f928512820 2023-10-05T19:28:24.754 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.746+0000 7fb927c9a700 1 -- 172.21.15.167:0/2080866673 <== osd.3 v2:172.21.15.167:6808/30036 6 ==== osd_op_reply(23 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fb910039040 con 0x55f928512820 2023-10-05T19:28:24.754 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.750+0000 7fb927c9a700 1 -- 172.21.15.167:0/2080866673 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:24 5.9 5:90000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x7fb91003d040 con 0x55f9284514b0 2023-10-05T19:28:24.754 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.750+0000 7fb92849b700 1 -- 172.21.15.167:0/2080866673 <== osd.1 v2:172.21.15.167:6800/30035 10 ==== osd_op_reply(24 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fb918022040 con 0x55f9284514b0 2023-10-05T19:28:24.755 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.750+0000 7fb92849b700 1 -- 172.21.15.167:0/2080866673 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:25 5.19 5:98000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x7fb91003d040 con 0x55f9284514b0 2023-10-05T19:28:24.755 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.750+0000 7fb92849b700 1 -- 172.21.15.167:0/2080866673 <== osd.1 v2:172.21.15.167:6800/30035 11 ==== osd_op_reply(25 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fb918022040 con 0x55f9284514b0 2023-10-05T19:28:24.756 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.750+0000 7fb92849b700 1 -- 172.21.15.167:0/2080866673 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:26 5.5 5:a0000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x7fb91003d040 con 0x7fb920010070 2023-10-05T19:28:24.756 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.750+0000 7fb92849b700 1 -- 172.21.15.167:0/2080866673 <== osd.0 v2:172.21.15.118:6800/27690 6 ==== osd_op_reply(26 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fb918022040 con 0x7fb920010070 2023-10-05T19:28:24.756 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.750+0000 7fb92849b700 1 -- 172.21.15.167:0/2080866673 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:27 5.15 5:a8000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x7fb91003d040 con 0x55f928512820 2023-10-05T19:28:24.757 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.750+0000 7fb927c9a700 1 -- 172.21.15.167:0/2080866673 <== osd.3 v2:172.21.15.167:6808/30036 7 ==== osd_op_reply(27 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fb910039040 con 0x55f928512820 2023-10-05T19:28:24.757 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.750+0000 7fb927c9a700 1 -- 172.21.15.167:0/2080866673 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:28 5.d 5:b0000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x7fb91003d040 con 0x7fb91001a070 2023-10-05T19:28:24.758 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.754+0000 7fb927499700 1 -- 172.21.15.167:0/2080866673 <== osd.2 v2:172.21.15.167:6816/30037 4 ==== osd_op_reply(28 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fb920017050 con 0x7fb91001a070 2023-10-05T19:28:24.758 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.754+0000 7fb927499700 1 -- 172.21.15.167:0/2080866673 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:29 5.1d 5:b8000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x7fb91003d040 con 0x55f9284514b0 2023-10-05T19:28:24.759 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.754+0000 7fb92849b700 1 -- 172.21.15.167:0/2080866673 <== osd.1 v2:172.21.15.167:6800/30035 12 ==== osd_op_reply(29 [pgnls start_epoch 30 out=107b] v23'1 uv1 ondisk = 1) v8 ==== 144+0+107 (crc 0 0 0) 0x7fb918022040 con 0x55f9284514b0 2023-10-05T19:28:24.759 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.754+0000 7fb92849b700 1 -- 172.21.15.167:0/2080866673 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:30 5.3 5:c0000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x7fb91003d040 con 0x7fb920010070 2023-10-05T19:28:24.760 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.754+0000 7fb92849b700 1 -- 172.21.15.167:0/2080866673 <== osd.0 v2:172.21.15.118:6800/27690 7 ==== osd_op_reply(30 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fb918022040 con 0x7fb920010070 2023-10-05T19:28:24.760 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.754+0000 7fb92849b700 1 -- 172.21.15.167:0/2080866673 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:31 5.13 5:c8000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x7fb91003d040 con 0x55f928512820 2023-10-05T19:28:24.760 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.754+0000 7fb927c9a700 1 -- 172.21.15.167:0/2080866673 <== osd.3 v2:172.21.15.167:6808/30036 8 ==== osd_op_reply(31 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fb910039040 con 0x55f928512820 2023-10-05T19:28:24.760 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.754+0000 7fb927c9a700 1 -- 172.21.15.167:0/2080866673 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:32 5.b 5:d0000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x7fb91003d040 con 0x7fb920010070 2023-10-05T19:28:24.761 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.754+0000 7fb92849b700 1 -- 172.21.15.167:0/2080866673 <== osd.0 v2:172.21.15.118:6800/27690 8 ==== osd_op_reply(32 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fb918022040 con 0x7fb920010070 2023-10-05T19:28:24.761 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.754+0000 7fb92849b700 1 -- 172.21.15.167:0/2080866673 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:33 5.1b 5:d8000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x7fb91003d040 con 0x7fb91001a070 2023-10-05T19:28:24.762 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.758+0000 7fb927499700 1 -- 172.21.15.167:0/2080866673 <== osd.2 v2:172.21.15.167:6816/30037 5 ==== osd_op_reply(33 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fb920017050 con 0x7fb91001a070 2023-10-05T19:28:24.762 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.758+0000 7fb927499700 1 -- 172.21.15.167:0/2080866673 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:34 5.7 5:e0000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x7fb91003d040 con 0x55f928512820 2023-10-05T19:28:24.763 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.758+0000 7fb927c9a700 1 -- 172.21.15.167:0/2080866673 <== osd.3 v2:172.21.15.167:6808/30036 9 ==== osd_op_reply(34 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fb910039040 con 0x55f928512820 2023-10-05T19:28:24.763 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.758+0000 7fb927c9a700 1 -- 172.21.15.167:0/2080866673 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:35 5.17 5:e8000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x7fb91003d040 con 0x7fb920010070 2023-10-05T19:28:24.764 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.758+0000 7fb92849b700 1 -- 172.21.15.167:0/2080866673 <== osd.0 v2:172.21.15.118:6800/27690 9 ==== osd_op_reply(35 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fb918022040 con 0x7fb920010070 2023-10-05T19:28:24.764 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.758+0000 7fb92849b700 1 -- 172.21.15.167:0/2080866673 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:36 5.f 5:f0000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x7fb91003d040 con 0x55f9284514b0 2023-10-05T19:28:24.765 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.758+0000 7fb92849b700 1 -- 172.21.15.167:0/2080866673 <== osd.1 v2:172.21.15.167:6800/30035 13 ==== osd_op_reply(36 [pgnls start_epoch 30 out=115b] v23'2 uv2 ondisk = 1) v8 ==== 144+0+115 (crc 0 0 0) 0x7fb91802f080 con 0x55f9284514b0 2023-10-05T19:28:24.765 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.758+0000 7fb92849b700 1 -- 172.21.15.167:0/2080866673 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:37 5.1f 5:f8000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x7fb91003d040 con 0x7fb91001a070 2023-10-05T19:28:24.766 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.762+0000 7fb927499700 1 -- 172.21.15.167:0/2080866673 <== osd.2 v2:172.21.15.167:6816/30037 6 ==== osd_op_reply(37 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fb920017050 con 0x7fb91001a070 2023-10-05T19:28:24.766 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.762+0000 7fb92a165e40 20 RGWRados::pool_iterate: got zonegroup_info.4f1f719d-9648-417b-b10c-e5d939825419 2023-10-05T19:28:24.766 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.762+0000 7fb92a165e40 20 RGWRados::pool_iterate: got zone_info.9a9df965-6612-45d4-a45f-6e3c889bacc2 2023-10-05T19:28:24.766 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.762+0000 7fb92a165e40 20 RGWRados::pool_iterate: got zone_names.default 2023-10-05T19:28:24.766 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.762+0000 7fb92a165e40 20 RGWRados::pool_iterate: got zonegroups_names.default 2023-10-05T19:28:24.767 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.762+0000 7fb92a165e40 20 rados->read ofs=0 len=0 2023-10-05T19:28:24.767 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.762+0000 7fb92a165e40 1 -- 172.21.15.167:0/2080866673 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:38 5.12 5:49953fa1:::default.realm:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x55f9285115d0 con 0x55f9284514b0 2023-10-05T19:28:24.767 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.762+0000 7fb92849b700 1 -- 172.21.15.167:0/2080866673 <== osd.1 v2:172.21.15.167:6800/30035 14 ==== osd_op_reply(38 default.realm [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 ==== 157+0+0 (crc 0 0 0) 0x7fb91802f080 con 0x55f9284514b0 2023-10-05T19:28:24.768 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.762+0000 7fb92a165e40 20 rados_obj.operate() r=-2 bl.length=0 2023-10-05T19:28:24.768 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.762+0000 7fb92a165e40 20 rados->read ofs=0 len=0 2023-10-05T19:28:24.768 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.762+0000 7fb92a165e40 1 -- 172.21.15.167:0/2080866673 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:39 5.f 5:f4c53578:::zonegroups_names.default:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x55f928511ec0 con 0x55f9284514b0 2023-10-05T19:28:24.768 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.762+0000 7fb92849b700 1 -- 172.21.15.167:0/2080866673 <== osd.1 v2:172.21.15.167:6800/30035 15 ==== osd_op_reply(39 zonegroups_names.default [read 0~46 out=46b] v0'0 uv2 ondisk = 0) v8 ==== 168+0+46 (crc 0 0 0) 0x7fb91802f080 con 0x55f9284514b0 2023-10-05T19:28:24.768 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.762+0000 7fb92a165e40 20 rados_obj.operate() r=0 bl.length=46 2023-10-05T19:28:24.768 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.762+0000 7fb92a165e40 20 rados->read ofs=0 len=0 2023-10-05T19:28:24.768 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.762+0000 7fb92a165e40 1 -- 172.21.15.167:0/2080866673 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:40 5.c 5:30d299a5:::zonegroup_info.4f1f719d-9648-417b-b10c-e5d939825419:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x55f928515580 con 0x7fb920010070 2023-10-05T19:28:24.769 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.762+0000 7fb92849b700 1 -- 172.21.15.167:0/2080866673 <== osd.0 v2:172.21.15.118:6800/27690 10 ==== osd_op_reply(40 zonegroup_info.4f1f719d-9648-417b-b10c-e5d939825419 [read 0~362 out=362b] v0'0 uv1 ondisk = 0) v8 ==== 195+0+362 (crc 0 0 0) 0x7fb91802f080 con 0x7fb920010070 2023-10-05T19:28:24.769 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.762+0000 7fb92a165e40 20 rados_obj.operate() r=0 bl.length=362 2023-10-05T19:28:24.770 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.762+0000 7fb92a165e40 20 rados->read ofs=0 len=0 2023-10-05T19:28:24.770 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.762+0000 7fb92a165e40 1 -- 172.21.15.167:0/2080866673 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:41 5.13 5:cbbf6c60:::converted:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x55f928515580 con 0x55f928512820 2023-10-05T19:28:24.770 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.762+0000 7fb927c9a700 1 -- 172.21.15.167:0/2080866673 <== osd.3 v2:172.21.15.167:6808/30036 10 ==== osd_op_reply(41 converted [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 ==== 153+0+0 (crc 0 0 0) 0x7fb910039040 con 0x55f928512820 2023-10-05T19:28:24.770 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.762+0000 7fb92a165e40 20 rados_obj.operate() r=-2 bl.length=0 2023-10-05T19:28:24.770 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.762+0000 7fb92a165e40 20 rados->read ofs=0 len=0 2023-10-05T19:28:24.770 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.762+0000 7fb92a165e40 1 -- 172.21.15.167:0/2080866673 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:42 5.12 5:49953fa1:::default.realm:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x55f928515930 con 0x55f9284514b0 2023-10-05T19:28:24.771 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.762+0000 7fb92849b700 1 -- 172.21.15.167:0/2080866673 <== osd.1 v2:172.21.15.167:6800/30035 16 ==== osd_op_reply(42 default.realm [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 ==== 157+0+0 (crc 0 0 0) 0x7fb91802f080 con 0x55f9284514b0 2023-10-05T19:28:24.771 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.762+0000 7fb92a165e40 20 rados_obj.operate() r=-2 bl.length=0 2023-10-05T19:28:24.771 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.762+0000 7fb92a165e40 20 rados->read ofs=0 len=0 2023-10-05T19:28:24.771 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.762+0000 7fb92a165e40 1 -- 172.21.15.167:0/2080866673 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:43 5.f 5:f4c53578:::zonegroups_names.default:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x55f928515d30 con 0x55f9284514b0 2023-10-05T19:28:24.771 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.766+0000 7fb92849b700 1 -- 172.21.15.167:0/2080866673 <== osd.1 v2:172.21.15.167:6800/30035 17 ==== osd_op_reply(43 zonegroups_names.default [read 0~46 out=46b] v0'0 uv2 ondisk = 0) v8 ==== 168+0+46 (crc 0 0 0) 0x7fb91802f080 con 0x55f9284514b0 2023-10-05T19:28:24.771 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.766+0000 7fb92a165e40 20 rados_obj.operate() r=0 bl.length=46 2023-10-05T19:28:24.771 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.766+0000 7fb92a165e40 1 -- 172.21.15.167:0/2080866673 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:44 5.0 5:00000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x55f928516d40 con 0x55f928512820 2023-10-05T19:28:24.773 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.766+0000 7fb927c9a700 1 -- 172.21.15.167:0/2080866673 <== osd.3 v2:172.21.15.167:6808/30036 11 ==== osd_op_reply(44 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fb910039040 con 0x55f928512820 2023-10-05T19:28:24.773 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.766+0000 7fb927c9a700 1 -- 172.21.15.167:0/2080866673 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:45 5.10 5:08000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x55f928516d40 con 0x7fb91001a070 2023-10-05T19:28:24.774 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.770+0000 7fb927499700 1 -- 172.21.15.167:0/2080866673 <== osd.2 v2:172.21.15.167:6816/30037 7 ==== osd_op_reply(45 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fb920017050 con 0x7fb91001a070 2023-10-05T19:28:24.774 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.770+0000 7fb927499700 1 -- 172.21.15.167:0/2080866673 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:46 5.8 5:10000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x55f928516d40 con 0x7fb920010070 2023-10-05T19:28:24.775 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.770+0000 7fb92849b700 1 -- 172.21.15.167:0/2080866673 <== osd.0 v2:172.21.15.118:6800/27690 11 ==== osd_op_reply(46 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fb91802f080 con 0x7fb920010070 2023-10-05T19:28:24.775 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.770+0000 7fb92849b700 1 -- 172.21.15.167:0/2080866673 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:47 5.18 5:18000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x55f928511ec0 con 0x55f9284514b0 2023-10-05T19:28:24.776 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.770+0000 7fb92849b700 1 -- 172.21.15.167:0/2080866673 <== osd.1 v2:172.21.15.167:6800/30035 18 ==== osd_op_reply(47 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fb91802f080 con 0x55f9284514b0 2023-10-05T19:28:24.776 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.770+0000 7fb92849b700 1 -- 172.21.15.167:0/2080866673 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:48 5.4 5:20000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x55f928511ec0 con 0x55f9284514b0 2023-10-05T19:28:24.777 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.770+0000 7fb92849b700 1 -- 172.21.15.167:0/2080866673 <== osd.1 v2:172.21.15.167:6800/30035 19 ==== osd_op_reply(48 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fb91802f080 con 0x55f9284514b0 2023-10-05T19:28:24.777 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.770+0000 7fb92849b700 1 -- 172.21.15.167:0/2080866673 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:49 5.14 5:28000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x55f928511ec0 con 0x7fb920010070 2023-10-05T19:28:24.777 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.770+0000 7fb92849b700 1 -- 172.21.15.167:0/2080866673 <== osd.0 v2:172.21.15.118:6800/27690 12 ==== osd_op_reply(49 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fb91802f080 con 0x7fb920010070 2023-10-05T19:28:24.778 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.770+0000 7fb92849b700 1 -- 172.21.15.167:0/2080866673 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:50 5.c 5:30000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x55f928511ec0 con 0x7fb920010070 2023-10-05T19:28:24.778 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.774+0000 7fb92849b700 1 -- 172.21.15.167:0/2080866673 <== osd.0 v2:172.21.15.118:6800/27690 13 ==== osd_op_reply(50 [pgnls start_epoch 30 out=112b] v23'1 uv1 ondisk = 1) v8 ==== 144+0+112 (crc 0 0 0) 0x7fb91802f080 con 0x7fb920010070 2023-10-05T19:28:24.779 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.774+0000 7fb92849b700 1 -- 172.21.15.167:0/2080866673 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:51 5.1c 5:38000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x55f928511ec0 con 0x7fb91001a070 2023-10-05T19:28:24.779 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.774+0000 7fb927499700 1 -- 172.21.15.167:0/2080866673 <== osd.2 v2:172.21.15.167:6816/30037 8 ==== osd_op_reply(51 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fb920017050 con 0x7fb91001a070 2023-10-05T19:28:24.779 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.774+0000 7fb927499700 1 -- 172.21.15.167:0/2080866673 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:52 5.2 5:40000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x55f928511ec0 con 0x55f928512820 2023-10-05T19:28:24.780 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.774+0000 7fb927c9a700 1 -- 172.21.15.167:0/2080866673 <== osd.3 v2:172.21.15.167:6808/30036 12 ==== osd_op_reply(52 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fb910039040 con 0x55f928512820 2023-10-05T19:28:24.780 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.774+0000 7fb927c9a700 1 -- 172.21.15.167:0/2080866673 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:53 5.12 5:48000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x55f928511ec0 con 0x55f9284514b0 2023-10-05T19:28:24.781 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.774+0000 7fb92849b700 1 -- 172.21.15.167:0/2080866673 <== osd.1 v2:172.21.15.167:6800/30035 20 ==== osd_op_reply(53 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fb91802f080 con 0x55f9284514b0 2023-10-05T19:28:24.781 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.774+0000 7fb92849b700 1 -- 172.21.15.167:0/2080866673 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:54 5.a 5:50000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x55f928511ec0 con 0x7fb920010070 2023-10-05T19:28:24.782 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.778+0000 7fb92849b700 1 -- 172.21.15.167:0/2080866673 <== osd.0 v2:172.21.15.118:6800/27690 14 ==== osd_op_reply(54 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fb91802f080 con 0x7fb920010070 2023-10-05T19:28:24.782 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.778+0000 7fb92849b700 1 -- 172.21.15.167:0/2080866673 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:55 5.1a 5:58000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x55f928511ec0 con 0x55f9284514b0 2023-10-05T19:28:24.783 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.778+0000 7fb92849b700 1 -- 172.21.15.167:0/2080866673 <== osd.1 v2:172.21.15.167:6800/30035 21 ==== osd_op_reply(55 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fb91802f080 con 0x55f9284514b0 2023-10-05T19:28:24.783 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.778+0000 7fb92849b700 1 -- 172.21.15.167:0/2080866673 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:56 5.6 5:60000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x55f928511ec0 con 0x7fb91001a070 2023-10-05T19:28:24.784 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.778+0000 7fb927499700 1 -- 172.21.15.167:0/2080866673 <== osd.2 v2:172.21.15.167:6816/30037 9 ==== osd_op_reply(56 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fb920017050 con 0x7fb91001a070 2023-10-05T19:28:24.784 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.778+0000 7fb927499700 1 -- 172.21.15.167:0/2080866673 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:57 5.16 5:68000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x55f928511ec0 con 0x55f928512820 2023-10-05T19:28:24.785 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.778+0000 7fb927c9a700 1 -- 172.21.15.167:0/2080866673 <== osd.3 v2:172.21.15.167:6808/30036 13 ==== osd_op_reply(57 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fb910039040 con 0x55f928512820 2023-10-05T19:28:24.785 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.778+0000 7fb927c9a700 1 -- 172.21.15.167:0/2080866673 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:58 5.e 5:70000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x55f928511ec0 con 0x55f928512820 2023-10-05T19:28:24.785 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.778+0000 7fb927c9a700 1 -- 172.21.15.167:0/2080866673 <== osd.3 v2:172.21.15.167:6808/30036 14 ==== osd_op_reply(58 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fb910039040 con 0x55f928512820 2023-10-05T19:28:24.786 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.778+0000 7fb927c9a700 1 -- 172.21.15.167:0/2080866673 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:59 5.1e 5:78000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x55f928511ec0 con 0x7fb920010070 2023-10-05T19:28:24.787 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.782+0000 7fb92849b700 1 -- 172.21.15.167:0/2080866673 <== osd.0 v2:172.21.15.118:6800/27690 15 ==== osd_op_reply(59 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fb91802f080 con 0x7fb920010070 2023-10-05T19:28:24.787 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.782+0000 7fb92849b700 1 -- 172.21.15.167:0/2080866673 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:60 5.1 5:80000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x55f928511ec0 con 0x55f928512820 2023-10-05T19:28:24.787 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.782+0000 7fb927c9a700 1 -- 172.21.15.167:0/2080866673 <== osd.3 v2:172.21.15.167:6808/30036 15 ==== osd_op_reply(60 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fb910039040 con 0x55f928512820 2023-10-05T19:28:24.787 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.782+0000 7fb927c9a700 1 -- 172.21.15.167:0/2080866673 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:61 5.11 5:88000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x55f928511ec0 con 0x55f928512820 2023-10-05T19:28:24.788 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.782+0000 7fb927c9a700 1 -- 172.21.15.167:0/2080866673 <== osd.3 v2:172.21.15.167:6808/30036 16 ==== osd_op_reply(61 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fb910039040 con 0x55f928512820 2023-10-05T19:28:24.788 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.782+0000 7fb927c9a700 1 -- 172.21.15.167:0/2080866673 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:62 5.9 5:90000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x55f928511ec0 con 0x55f9284514b0 2023-10-05T19:28:24.789 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.782+0000 7fb92849b700 1 -- 172.21.15.167:0/2080866673 <== osd.1 v2:172.21.15.167:6800/30035 22 ==== osd_op_reply(62 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fb91802f080 con 0x55f9284514b0 2023-10-05T19:28:24.789 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.782+0000 7fb92849b700 1 -- 172.21.15.167:0/2080866673 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:63 5.19 5:98000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x55f928511ec0 con 0x55f9284514b0 2023-10-05T19:28:24.790 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.786+0000 7fb92849b700 1 -- 172.21.15.167:0/2080866673 <== osd.1 v2:172.21.15.167:6800/30035 23 ==== osd_op_reply(63 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fb91802f080 con 0x55f9284514b0 2023-10-05T19:28:24.790 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.786+0000 7fb92849b700 1 -- 172.21.15.167:0/2080866673 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:64 5.5 5:a0000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x55f928511ec0 con 0x7fb920010070 2023-10-05T19:28:24.791 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.786+0000 7fb92849b700 1 -- 172.21.15.167:0/2080866673 <== osd.0 v2:172.21.15.118:6800/27690 16 ==== osd_op_reply(64 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fb91802f080 con 0x7fb920010070 2023-10-05T19:28:24.791 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.786+0000 7fb92849b700 1 -- 172.21.15.167:0/2080866673 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:65 5.15 5:a8000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x55f928511ec0 con 0x55f928512820 2023-10-05T19:28:24.792 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.786+0000 7fb927c9a700 1 -- 172.21.15.167:0/2080866673 <== osd.3 v2:172.21.15.167:6808/30036 17 ==== osd_op_reply(65 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fb910039040 con 0x55f928512820 2023-10-05T19:28:24.792 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.786+0000 7fb927c9a700 1 -- 172.21.15.167:0/2080866673 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:66 5.d 5:b0000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x55f928511ec0 con 0x7fb91001a070 2023-10-05T19:28:24.792 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.786+0000 7fb927499700 1 -- 172.21.15.167:0/2080866673 <== osd.2 v2:172.21.15.167:6816/30037 10 ==== osd_op_reply(66 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fb920017050 con 0x7fb91001a070 2023-10-05T19:28:24.793 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.786+0000 7fb927499700 1 -- 172.21.15.167:0/2080866673 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:67 5.1d 5:b8000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x55f928511ec0 con 0x55f9284514b0 2023-10-05T19:28:24.793 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.786+0000 7fb92849b700 1 -- 172.21.15.167:0/2080866673 <== osd.1 v2:172.21.15.167:6800/30035 24 ==== osd_op_reply(67 [pgnls start_epoch 30 out=107b] v23'1 uv1 ondisk = 1) v8 ==== 144+0+107 (crc 0 0 0) 0x7fb91802f080 con 0x55f9284514b0 2023-10-05T19:28:24.793 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.786+0000 7fb92849b700 1 -- 172.21.15.167:0/2080866673 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:68 5.3 5:c0000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x55f928511ec0 con 0x7fb920010070 2023-10-05T19:28:24.794 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.790+0000 7fb92849b700 1 -- 172.21.15.167:0/2080866673 <== osd.0 v2:172.21.15.118:6800/27690 17 ==== osd_op_reply(68 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fb91802f080 con 0x7fb920010070 2023-10-05T19:28:24.794 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.790+0000 7fb92849b700 1 -- 172.21.15.167:0/2080866673 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:69 5.13 5:c8000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x55f928511ec0 con 0x55f928512820 2023-10-05T19:28:24.795 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.790+0000 7fb927c9a700 1 -- 172.21.15.167:0/2080866673 <== osd.3 v2:172.21.15.167:6808/30036 18 ==== osd_op_reply(69 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fb910039040 con 0x55f928512820 2023-10-05T19:28:24.795 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.790+0000 7fb927c9a700 1 -- 172.21.15.167:0/2080866673 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:70 5.b 5:d0000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x55f928511ec0 con 0x7fb920010070 2023-10-05T19:28:24.796 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.790+0000 7fb92849b700 1 -- 172.21.15.167:0/2080866673 <== osd.0 v2:172.21.15.118:6800/27690 18 ==== osd_op_reply(70 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fb91802f080 con 0x7fb920010070 2023-10-05T19:28:24.796 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.790+0000 7fb92849b700 1 -- 172.21.15.167:0/2080866673 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:71 5.1b 5:d8000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x55f928511ec0 con 0x7fb91001a070 2023-10-05T19:28:24.796 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.790+0000 7fb927499700 1 -- 172.21.15.167:0/2080866673 <== osd.2 v2:172.21.15.167:6816/30037 11 ==== osd_op_reply(71 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fb920017050 con 0x7fb91001a070 2023-10-05T19:28:24.797 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.790+0000 7fb927499700 1 -- 172.21.15.167:0/2080866673 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:72 5.7 5:e0000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x55f928511ec0 con 0x55f928512820 2023-10-05T19:28:24.797 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.790+0000 7fb927c9a700 1 -- 172.21.15.167:0/2080866673 <== osd.3 v2:172.21.15.167:6808/30036 19 ==== osd_op_reply(72 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fb910039040 con 0x55f928512820 2023-10-05T19:28:24.797 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.790+0000 7fb927c9a700 1 -- 172.21.15.167:0/2080866673 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:73 5.17 5:e8000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x55f928511ec0 con 0x7fb920010070 2023-10-05T19:28:24.798 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.794+0000 7fb92849b700 1 -- 172.21.15.167:0/2080866673 <== osd.0 v2:172.21.15.118:6800/27690 19 ==== osd_op_reply(73 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fb91802f080 con 0x7fb920010070 2023-10-05T19:28:24.798 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.794+0000 7fb92849b700 1 -- 172.21.15.167:0/2080866673 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:74 5.f 5:f0000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x55f928511ec0 con 0x55f9284514b0 2023-10-05T19:28:24.799 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.794+0000 7fb92849b700 1 -- 172.21.15.167:0/2080866673 <== osd.1 v2:172.21.15.167:6800/30035 25 ==== osd_op_reply(74 [pgnls start_epoch 30 out=115b] v23'2 uv2 ondisk = 1) v8 ==== 144+0+115 (crc 0 0 0) 0x7fb91802f080 con 0x55f9284514b0 2023-10-05T19:28:24.799 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.794+0000 7fb92849b700 1 -- 172.21.15.167:0/2080866673 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:75 5.1f 5:f8000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x55f928511ec0 con 0x7fb91001a070 2023-10-05T19:28:24.800 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.794+0000 7fb927499700 1 -- 172.21.15.167:0/2080866673 <== osd.2 v2:172.21.15.167:6816/30037 12 ==== osd_op_reply(75 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fb920017050 con 0x7fb91001a070 2023-10-05T19:28:24.800 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.794+0000 7fb92a165e40 20 RGWRados::pool_iterate: got zonegroup_info.4f1f719d-9648-417b-b10c-e5d939825419 2023-10-05T19:28:24.800 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.794+0000 7fb92a165e40 20 RGWRados::pool_iterate: got zone_info.9a9df965-6612-45d4-a45f-6e3c889bacc2 2023-10-05T19:28:24.801 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.794+0000 7fb92a165e40 20 RGWRados::pool_iterate: got zone_names.default 2023-10-05T19:28:24.801 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.794+0000 7fb92a165e40 20 RGWRados::pool_iterate: got zonegroups_names.default 2023-10-05T19:28:24.801 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.794+0000 7fb92a165e40 20 rados->read ofs=0 len=0 2023-10-05T19:28:24.801 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.794+0000 7fb92a165e40 1 -- 172.21.15.167:0/2080866673 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:76 5.f 5:f43b6ece:::zone_names.default:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x55f928515d30 con 0x55f9284514b0 2023-10-05T19:28:24.801 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.794+0000 7fb92849b700 1 -- 172.21.15.167:0/2080866673 <== osd.1 v2:172.21.15.167:6800/30035 26 ==== osd_op_reply(76 zone_names.default [read 0~46 out=46b] v0'0 uv1 ondisk = 0) v8 ==== 162+0+46 (crc 0 0 0) 0x7fb91802f080 con 0x55f9284514b0 2023-10-05T19:28:24.802 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.794+0000 7fb92a165e40 20 rados_obj.operate() r=0 bl.length=46 2023-10-05T19:28:24.802 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.794+0000 7fb92a165e40 20 rados->read ofs=0 len=0 2023-10-05T19:28:24.802 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.794+0000 7fb92a165e40 1 -- 172.21.15.167:0/2080866673 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:77 5.1d 5:b8374eb4:::zone_info.9a9df965-6612-45d4-a45f-6e3c889bacc2:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x55f9285160e0 con 0x55f9284514b0 2023-10-05T19:28:24.803 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.798+0000 7fb92849b700 1 -- 172.21.15.167:0/2080866673 <== osd.1 v2:172.21.15.167:6800/30035 27 ==== osd_op_reply(77 zone_info.9a9df965-6612-45d4-a45f-6e3c889bacc2 [read 0~874 out=874b] v0'0 uv1 ondisk = 0) v8 ==== 190+0+874 (crc 0 0 0) 0x7fb91802f080 con 0x55f9284514b0 2023-10-05T19:28:24.803 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.798+0000 7fb92a165e40 20 rados_obj.operate() r=0 bl.length=874 2023-10-05T19:28:24.803 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.798+0000 7fb92a165e40 20 rados->read ofs=0 len=0 2023-10-05T19:28:24.803 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.798+0000 7fb92a165e40 1 -- 172.21.15.167:0/2080866673 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:78 5.f 5:f4c53578:::zonegroups_names.default:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x55f9285160e0 con 0x55f9284514b0 2023-10-05T19:28:24.803 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.798+0000 7fb92849b700 1 -- 172.21.15.167:0/2080866673 <== osd.1 v2:172.21.15.167:6800/30035 28 ==== osd_op_reply(78 zonegroups_names.default [read 0~46 out=46b] v0'0 uv2 ondisk = 0) v8 ==== 168+0+46 (crc 0 0 0) 0x7fb91802f080 con 0x55f9284514b0 2023-10-05T19:28:24.803 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.798+0000 7fb92a165e40 20 rados_obj.operate() r=0 bl.length=46 2023-10-05T19:28:24.803 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.798+0000 7fb92a165e40 20 rados->read ofs=0 len=0 2023-10-05T19:28:24.803 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.798+0000 7fb92a165e40 1 -- 172.21.15.167:0/2080866673 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:79 5.c 5:30d299a5:::zonegroup_info.4f1f719d-9648-417b-b10c-e5d939825419:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x55f9285160e0 con 0x7fb920010070 2023-10-05T19:28:24.804 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.798+0000 7fb92849b700 1 -- 172.21.15.167:0/2080866673 <== osd.0 v2:172.21.15.118:6800/27690 20 ==== osd_op_reply(79 zonegroup_info.4f1f719d-9648-417b-b10c-e5d939825419 [read 0~362 out=362b] v0'0 uv1 ondisk = 0) v8 ==== 195+0+362 (crc 0 0 0) 0x7fb91802f080 con 0x7fb920010070 2023-10-05T19:28:24.804 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.798+0000 7fb92a165e40 20 rados_obj.operate() r=0 bl.length=362 2023-10-05T19:28:24.804 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.798+0000 7fb92a165e40 20 rados->read ofs=0 len=0 2023-10-05T19:28:24.804 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.798+0000 7fb92a165e40 1 -- 172.21.15.167:0/2080866673 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:80 5.3 5:c0ad8cbd:::region_map:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x55f9285165a0 con 0x7fb920010070 2023-10-05T19:28:24.805 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.798+0000 7fb92849b700 1 -- 172.21.15.167:0/2080866673 <== osd.0 v2:172.21.15.118:6800/27690 21 ==== osd_op_reply(80 region_map [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 ==== 154+0+0 (crc 0 0 0) 0x7fb91802f080 con 0x7fb920010070 2023-10-05T19:28:24.805 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.798+0000 7fb92a165e40 20 rados_obj.operate() r=-2 bl.length=0 2023-10-05T19:28:24.805 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.798+0000 7fb92a165e40 20 zone default found 2023-10-05T19:28:24.805 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.798+0000 7fb92a165e40 4 Realm: () 2023-10-05T19:28:24.805 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.798+0000 7fb92a165e40 4 ZoneGroup: default (4f1f719d-9648-417b-b10c-e5d939825419) 2023-10-05T19:28:24.805 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.798+0000 7fb92a165e40 4 Zone: default (9a9df965-6612-45d4-a45f-6e3c889bacc2) 2023-10-05T19:28:24.805 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.798+0000 7fb92a165e40 10 cannot find current period zonegroup using local zonegroup configuration 2023-10-05T19:28:24.806 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.798+0000 7fb92a165e40 20 zonegroup default 2023-10-05T19:28:24.806 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.798+0000 7fb92a165e40 20 rados->read ofs=0 len=0 2023-10-05T19:28:24.806 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.798+0000 7fb92a165e40 1 -- 172.21.15.167:0/2080866673 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:81 5.3 5:c52100b6:::period_config.default:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x55f9285165a0 con 0x7fb920010070 2023-10-05T19:28:24.807 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.798+0000 7fb92849b700 1 -- 172.21.15.167:0/2080866673 <== osd.0 v2:172.21.15.118:6800/27690 22 ==== osd_op_reply(81 period_config.default [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 ==== 165+0+0 (crc 0 0 0) 0x7fb918050040 con 0x7fb920010070 2023-10-05T19:28:24.807 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.798+0000 7fb92a165e40 20 rados_obj.operate() r=-2 bl.length=0 2023-10-05T19:28:24.807 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.798+0000 7fb92a165e40 20 rados->read ofs=0 len=0 2023-10-05T19:28:24.807 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.798+0000 7fb92a165e40 1 -- 172.21.15.167:0/2080866673 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:82 6.2 6:4347d321:::bucket.sync-source-hints.:head [call version.read in=11b,read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x55f928517f20 con 0x55f928512820 2023-10-05T19:28:24.807 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.802+0000 7fb927c9a700 1 -- 172.21.15.167:0/2080866673 <== osd.3 v2:172.21.15.167:6808/30036 20 ==== osd_op_reply(82 bucket.sync-source-hints. [call,read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 ==== 211+0+0 (crc 0 0 0) 0x7fb910039040 con 0x55f928512820 2023-10-05T19:28:24.807 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.802+0000 7fb92a165e40 20 rados_obj.operate() r=-2 bl.length=0 2023-10-05T19:28:24.807 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.802+0000 7fb92a165e40 20 rados->read ofs=0 len=0 2023-10-05T19:28:24.807 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.802+0000 7fb92a165e40 1 -- 172.21.15.167:0/2080866673 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:83 6.b 6:d467b91b:::bucket.sync-target-hints.:head [call version.read in=11b,read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x55f9285182d0 con 0x55f928512820 2023-10-05T19:28:24.808 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.802+0000 7fb927c9a700 1 -- 172.21.15.167:0/2080866673 <== osd.3 v2:172.21.15.167:6808/30036 21 ==== osd_op_reply(83 bucket.sync-target-hints. [call,read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 ==== 211+0+0 (crc 0 0 0) 0x7fb910039040 con 0x55f928512820 2023-10-05T19:28:24.808 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.802+0000 7fb92a165e40 20 rados_obj.operate() r=-2 bl.length=0 2023-10-05T19:28:24.808 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.802+0000 7fb92a165e40 20 started sync module instance, tier type = 2023-10-05T19:28:24.808 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.802+0000 7fb92a165e40 20 started zone id=9a9df965-6612-45d4-a45f-6e3c889bacc2 (name=default) with tier type = 2023-10-05T19:28:24.808 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.802+0000 7fb92a165e40 1 -- 172.21.15.167:0/2080866673 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:84 7.1f 7:f95f44c2:::notify.0:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x55f9285175e0 con 0x7fb91001a070 2023-10-05T19:28:24.810 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.806+0000 7fb927499700 1 -- 172.21.15.167:0/2080866673 <== osd.2 v2:172.21.15.167:6816/30037 13 ==== osd_op_reply(84 notify.0 [create] v30'15 uv15 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7fb920017050 con 0x7fb91001a070 2023-10-05T19:28:24.810 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.806+0000 7fb92a165e40 1 -- 172.21.15.167:0/2080866673 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:85 7.1f 7:f95f44c2:::notify.0:head [watch watch cookie 94528611621328] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x55f928518680 con 0x7fb91001a070 2023-10-05T19:28:24.810 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.806+0000 7fb92a165e40 1 -- 172.21.15.167:0/2080866673 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:86 7.0 7:05bf5b68:::notify.1:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x55f928518f20 con 0x55f9284514b0 2023-10-05T19:28:24.813 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.806+0000 7fb927499700 1 -- 172.21.15.167:0/2080866673 <== osd.2 v2:172.21.15.167:6816/30037 14 ==== osd_op_reply(85 notify.0 [watch watch cookie 94528611621328] v30'16 uv15 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7fb920017050 con 0x7fb91001a070 2023-10-05T19:28:24.813 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.806+0000 7fb92849b700 1 -- 172.21.15.167:0/2080866673 <== osd.1 v2:172.21.15.167:6800/30035 29 ==== osd_op_reply(86 notify.1 [create] v30'15 uv15 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7fb918050040 con 0x55f9284514b0 2023-10-05T19:28:24.813 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.806+0000 7fb92a165e40 1 -- 172.21.15.167:0/2080866673 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:87 7.0 7:05bf5b68:::notify.1:head [watch watch cookie 94528611650096] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x55f928517b00 con 0x55f9284514b0 2023-10-05T19:28:24.813 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.806+0000 7fb92a165e40 1 -- 172.21.15.167:0/2080866673 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:88 7.15 7:a93a5511:::notify.2:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x55f9285199b0 con 0x55f928512820 2023-10-05T19:28:24.815 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.810+0000 7fb92849b700 1 -- 172.21.15.167:0/2080866673 <== osd.1 v2:172.21.15.167:6800/30035 30 ==== osd_op_reply(87 notify.1 [watch watch cookie 94528611650096] v30'16 uv15 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7fb918050040 con 0x55f9284514b0 2023-10-05T19:28:24.816 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.810+0000 7fb927c9a700 1 -- 172.21.15.167:0/2080866673 <== osd.3 v2:172.21.15.167:6808/30036 22 ==== osd_op_reply(88 notify.2 [create] v30'15 uv15 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7fb910039040 con 0x55f928512820 2023-10-05T19:28:24.816 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.810+0000 7fb92a165e40 1 -- 172.21.15.167:0/2080866673 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:89 7.15 7:a93a5511:::notify.2:head [watch watch cookie 94528611651200] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x55f928519db0 con 0x55f928512820 2023-10-05T19:28:24.816 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.810+0000 7fb92a165e40 1 -- 172.21.15.167:0/2080866673 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:90 7.e 7:7759931f:::notify.3:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x55f92851a650 con 0x7fb91001a070 2023-10-05T19:28:24.818 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.814+0000 7fb927499700 1 -- 172.21.15.167:0/2080866673 <== osd.2 v2:172.21.15.167:6816/30037 15 ==== osd_op_reply(90 notify.3 [create] v30'15 uv15 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7fb920017050 con 0x7fb91001a070 2023-10-05T19:28:24.818 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.814+0000 7fb92a165e40 1 -- 172.21.15.167:0/2080866673 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:91 7.e 7:7759931f:::notify.3:head [watch watch cookie 94528611658320] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x55f92851b390 con 0x7fb91001a070 2023-10-05T19:28:24.819 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.814+0000 7fb92a165e40 1 -- 172.21.15.167:0/2080866673 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:92 7.d 7:b4812045:::notify.4:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x55f92851bc70 con 0x55f9284514b0 2023-10-05T19:28:24.819 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.814+0000 7fb927c9a700 1 -- 172.21.15.167:0/2080866673 <== osd.3 v2:172.21.15.167:6808/30036 23 ==== osd_op_reply(89 notify.2 [watch watch cookie 94528611651200] v30'16 uv15 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7fb910039040 con 0x55f928512820 2023-10-05T19:28:24.820 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.814+0000 7fb927499700 1 -- 172.21.15.167:0/2080866673 <== osd.2 v2:172.21.15.167:6816/30037 16 ==== osd_op_reply(91 notify.3 [watch watch cookie 94528611658320] v30'16 uv15 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7fb920017050 con 0x7fb91001a070 2023-10-05T19:28:24.821 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.814+0000 7fb92849b700 1 -- 172.21.15.167:0/2080866673 <== osd.1 v2:172.21.15.167:6800/30035 31 ==== osd_op_reply(92 notify.4 [create] v30'15 uv15 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7fb918050040 con 0x55f9284514b0 2023-10-05T19:28:24.821 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.814+0000 7fb92a165e40 1 -- 172.21.15.167:0/2080866673 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:93 7.d 7:b4812045:::notify.4:head [watch watch cookie 94528611659424] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x55f92851b740 con 0x55f9284514b0 2023-10-05T19:28:24.821 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.814+0000 7fb92a165e40 1 -- 172.21.15.167:0/2080866673 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:94 7.3 7:c609908c:::notify.5:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x55f92851c560 con 0x7fb920010070 2023-10-05T19:28:24.823 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.818+0000 7fb92849b700 1 -- 172.21.15.167:0/2080866673 <== osd.1 v2:172.21.15.167:6800/30035 32 ==== osd_op_reply(93 notify.4 [watch watch cookie 94528611659424] v30'16 uv15 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7fb918050040 con 0x55f9284514b0 2023-10-05T19:28:24.824 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.818+0000 7fb92849b700 1 -- 172.21.15.167:0/2080866673 <== osd.0 v2:172.21.15.118:6800/27690 23 ==== osd_op_reply(94 notify.5 [create] v30'15 uv15 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7fb918050040 con 0x7fb920010070 2023-10-05T19:28:24.824 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.818+0000 7fb92a165e40 1 -- 172.21.15.167:0/2080866673 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:95 7.3 7:c609908c:::notify.5:head [watch watch cookie 94528611652736] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x55f92851c510 con 0x7fb920010070 2023-10-05T19:28:24.824 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.818+0000 7fb92a165e40 1 -- 172.21.15.167:0/2080866673 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:96 7.14 7:2b04a3e9:::notify.6:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x55f92851cdb0 con 0x7fb920010070 2023-10-05T19:28:24.826 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.822+0000 7fb92849b700 1 -- 172.21.15.167:0/2080866673 <== osd.0 v2:172.21.15.118:6800/27690 24 ==== osd_op_reply(95 notify.5 [watch watch cookie 94528611652736] v30'16 uv15 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7fb918050040 con 0x7fb920010070 2023-10-05T19:28:24.827 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.822+0000 7fb92849b700 1 -- 172.21.15.167:0/2080866673 <== osd.0 v2:172.21.15.118:6800/27690 25 ==== osd_op_reply(96 notify.6 [create] v30'15 uv15 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7fb918050040 con 0x7fb920010070 2023-10-05T19:28:24.827 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.822+0000 7fb92a165e40 1 -- 172.21.15.167:0/2080866673 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:97 7.14 7:2b04a3e9:::notify.6:head [watch watch cookie 94528611664464] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x55f92851cbd0 con 0x7fb920010070 2023-10-05T19:28:24.827 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.822+0000 7fb92a165e40 1 -- 172.21.15.167:0/2080866673 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:98 7.9 7:93e5b521:::notify.7:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x55f92851d730 con 0x7fb920010070 2023-10-05T19:28:24.830 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.826+0000 7fb92849b700 1 -- 172.21.15.167:0/2080866673 <== osd.0 v2:172.21.15.118:6800/27690 26 ==== osd_op_reply(98 notify.7 [create] v30'15 uv15 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7fb918050040 con 0x7fb920010070 2023-10-05T19:28:24.830 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.826+0000 7fb92849b700 1 -- 172.21.15.167:0/2080866673 <== osd.0 v2:172.21.15.118:6800/27690 27 ==== osd_op_reply(97 notify.6 [watch watch cookie 94528611664464] v30'16 uv15 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7fb918050040 con 0x7fb920010070 2023-10-05T19:28:24.831 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.826+0000 7fb92a165e40 1 -- 172.21.15.167:0/2080866673 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:99 7.9 7:93e5b521:::notify.7:head [watch watch cookie 94528611665568] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x55f92851d6f0 con 0x7fb920010070 2023-10-05T19:28:24.831 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.826+0000 7fb92a165e40 20 add_watcher() i=0 2023-10-05T19:28:24.831 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.826+0000 7fb92a165e40 20 add_watcher() i=1 2023-10-05T19:28:24.831 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.826+0000 7fb92a165e40 20 add_watcher() i=2 2023-10-05T19:28:24.831 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.826+0000 7fb92a165e40 20 add_watcher() i=3 2023-10-05T19:28:24.831 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.826+0000 7fb92a165e40 20 add_watcher() i=4 2023-10-05T19:28:24.831 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.826+0000 7fb92a165e40 20 add_watcher() i=5 2023-10-05T19:28:24.831 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.826+0000 7fb92a165e40 20 add_watcher() i=6 2023-10-05T19:28:24.832 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.826+0000 7fb92849b700 1 -- 172.21.15.167:0/2080866673 <== osd.0 v2:172.21.15.118:6800/27690 28 ==== osd_op_reply(99 notify.7 [watch watch cookie 94528611665568] v30'16 uv15 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7fb918050040 con 0x7fb920010070 2023-10-05T19:28:24.832 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.826+0000 7fb92a165e40 20 add_watcher() i=7 2023-10-05T19:28:24.833 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.826+0000 7fb92a165e40 2 all 8 watchers are set, enabling cache 2023-10-05T19:28:24.833 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.826+0000 7fb92a165e40 1 -- 172.21.15.167:0/2080866673 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:100 6.4 6:22d26bf9:::data_loggenerations_metadata:head [call version.check_conds in=74b,call version.read in=11b,read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x55f928520930 con 0x55f9284514b0 2023-10-05T19:28:24.833 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.826+0000 7fb92849b700 1 -- 172.21.15.167:0/2080866673 <== osd.1 v2:172.21.15.167:6800/30035 33 ==== osd_op_reply(100 data_loggenerations_metadata [call,call out=48b,read 0~28 out=28b] v0'0 uv1 ondisk = 0) v8 ==== 256+0+76 (crc 0 0 0) 0x7fb918050040 con 0x55f9284514b0 2023-10-05T19:28:24.833 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.826+0000 7fb92a165e40 1 -- 172.21.15.167:0/2080866673 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:101 6.4 6:22d26bf9:::data_loggenerations_metadata:head [watch watch cookie 94528611668848] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x55f92851dcb0 con 0x55f9284514b0 2023-10-05T19:28:24.836 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.830+0000 7fb92849b700 1 -- 172.21.15.167:0/2080866673 <== osd.1 v2:172.21.15.167:6800/30035 34 ==== osd_op_reply(101 data_loggenerations_metadata [watch watch cookie 94528611668848] v30'41 uv1 ondisk = 0) v8 ==== 172+0+0 (crc 0 0 0) 0x7fb918050040 con 0x55f9284514b0 2023-10-05T19:28:24.836 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.830+0000 7fb873fff700 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2023-10-05T19:28:24.836 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.830+0000 7fb92a165e40 20 check_secure_mon_conn(): auth registy supported: methods=[2] modes=[2,1] 2023-10-05T19:28:24.836 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.830+0000 7fb92a165e40 20 check_secure_mon_conn(): mode 1 is insecure 2023-10-05T19:28:24.837 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.830+0000 7fb92a165e40 1 Processor -- start 2023-10-05T19:28:24.837 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.830+0000 7fb92a165e40 1 -- start start 2023-10-05T19:28:24.837 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.830+0000 7fb92a165e40 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55f92852b1a0 0x55f92852b570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-05T19:28:24.837 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.830+0000 7fb92a165e40 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f92852bab0 con 0x55f92852b1a0 2023-10-05T19:28:24.837 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.830+0000 7fb927499700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55f92852b1a0 0x55f92852b570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-05T19:28:24.838 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.830+0000 7fb927499700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55f92852b1a0 0x55f92852b570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.118:3300/0 says I am v2:172.21.15.167:57008/0 (socket says 172.21.15.167:57008) 2023-10-05T19:28:24.838 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.830+0000 7fb927499700 1 -- 172.21.15.167:0/3404649153 learned_addr learned my addr 172.21.15.167:0/3404649153 (peer_addr_for_me v2:172.21.15.167:0/0) 2023-10-05T19:28:24.838 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.830+0000 7fb927499700 1 -- 172.21.15.167:0/3404649153 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55f92852bbf0 con 0x55f92852b1a0 2023-10-05T19:28:24.838 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.834+0000 7fb927499700 1 --2- 172.21.15.167:0/3404649153 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55f92852b1a0 0x55f92852b570 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x55f9285104b0 tx=0x7fb920010000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c2c34e0f3d69959a server_cookie=0 in_seq=0 out_seq=0 2023-10-05T19:28:24.839 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.834+0000 7fb8737fe700 1 -- 172.21.15.167:0/3404649153 <== mon.0 v2:172.21.15.118:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fb92001c9f0 con 0x55f92852b1a0 2023-10-05T19:28:24.839 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.834+0000 7fb8737fe700 1 -- 172.21.15.167:0/3404649153 <== mon.0 v2:172.21.15.118:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb92001cb50 con 0x55f92852b1a0 2023-10-05T19:28:24.839 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.834+0000 7fb8737fe700 1 -- 172.21.15.167:0/3404649153 <== mon.0 v2:172.21.15.118:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fb92001cd00 con 0x55f92852b1a0 2023-10-05T19:28:24.840 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.834+0000 7fb92a165e40 1 -- 172.21.15.167:0/3404649153 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55f92852b1a0 msgr2=0x55f92852b570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-05T19:28:24.840 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.834+0000 7fb92a165e40 1 --2- 172.21.15.167:0/3404649153 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55f92852b1a0 0x55f92852b570 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x55f9285104b0 tx=0x7fb920010000 comp rx=0 tx=0).stop 2023-10-05T19:28:24.840 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.834+0000 7fb92a165e40 1 -- 172.21.15.167:0/3404649153 shutdown_connections 2023-10-05T19:28:24.840 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.834+0000 7fb92a165e40 1 --2- 172.21.15.167:0/3404649153 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55f92852b1a0 0x55f92852b570 unknown :-1 s=CLOSED pgs=162 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-05T19:28:24.840 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.834+0000 7fb92a165e40 1 -- 172.21.15.167:0/3404649153 >> 172.21.15.167:0/3404649153 conn(0x55f928526560 msgr2=0x55f9285246b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-05T19:28:24.840 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.834+0000 7fb92a165e40 1 -- 172.21.15.167:0/3404649153 shutdown_connections 2023-10-05T19:28:24.840 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.834+0000 7fb92a165e40 1 -- 172.21.15.167:0/3404649153 wait complete. 2023-10-05T19:28:24.841 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.834+0000 7fb92a165e40 1 Processor -- start 2023-10-05T19:28:24.841 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.834+0000 7fb92a165e40 1 -- start start 2023-10-05T19:28:24.841 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.834+0000 7fb92a165e40 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55f92852b1a0 0x55f928530c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-05T19:28:24.841 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.834+0000 7fb92a165e40 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f928531c20 con 0x55f92852b1a0 2023-10-05T19:28:24.841 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.834+0000 7fb927499700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55f92852b1a0 0x55f928530c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-05T19:28:24.842 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.834+0000 7fb927499700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55f92852b1a0 0x55f928530c20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.118:3300/0 says I am v2:172.21.15.167:57012/0 (socket says 172.21.15.167:57012) 2023-10-05T19:28:24.842 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.834+0000 7fb927499700 1 -- 172.21.15.167:0/2876240720 learned_addr learned my addr 172.21.15.167:0/2876240720 (peer_addr_for_me v2:172.21.15.167:0/0) 2023-10-05T19:28:24.842 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.834+0000 7fb927499700 1 -- 172.21.15.167:0/2876240720 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55f928451940 con 0x55f92852b1a0 2023-10-05T19:28:24.843 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.838+0000 7fb927499700 1 --2- 172.21.15.167:0/2876240720 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55f92852b1a0 0x55f928530c20 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7fb920037040 tx=0x7fb92002f000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-05T19:28:24.843 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.838+0000 7fb871ffb700 1 -- 172.21.15.167:0/2876240720 <== mon.0 v2:172.21.15.118:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fb920047460 con 0x55f92852b1a0 2023-10-05T19:28:24.843 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.838+0000 7fb871ffb700 1 -- 172.21.15.167:0/2876240720 <== mon.0 v2:172.21.15.118:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb9200475c0 con 0x55f92852b1a0 2023-10-05T19:28:24.843 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.838+0000 7fb92a165e40 1 -- 172.21.15.167:0/2876240720 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x55f928531220 con 0x55f92852b1a0 2023-10-05T19:28:24.843 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.838+0000 7fb871ffb700 1 -- 172.21.15.167:0/2876240720 <== mon.0 v2:172.21.15.118:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fb9200477b0 con 0x55f92852b1a0 2023-10-05T19:28:24.843 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.838+0000 7fb92a165e40 1 -- 172.21.15.167:0/2876240720 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55f928531600 con 0x55f92852b1a0 2023-10-05T19:28:24.843 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.838+0000 7fb871ffb700 1 -- 172.21.15.167:0/2876240720 <== mon.0 v2:172.21.15.118:3300/0 4 ==== mgrmap(e 5) v1 ==== 40329+0+0 (secure 0 0 0) 0x7fb920047a20 con 0x55f92852b1a0 2023-10-05T19:28:24.844 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.838+0000 7fb871ffb700 1 --2- 172.21.15.167:0/2876240720 >> [v2:172.21.15.167:6824/29889,v1:172.21.15.167:6825/29889] conn(0x7fb864030c00 0x7fb8640330b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-05T19:28:24.844 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.838+0000 7fb871ffb700 1 -- 172.21.15.167:0/2876240720 <== mon.0 v2:172.21.15.118:3300/0 5 ==== osd_map(30..30 src has 1..30) v4 ==== 6057+0+0 (secure 0 0 0) 0x7fb92002f070 con 0x55f92852b1a0 2023-10-05T19:28:24.844 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.838+0000 7fb92849b700 1 --2- 172.21.15.167:0/2876240720 >> [v2:172.21.15.167:6824/29889,v1:172.21.15.167:6825/29889] conn(0x7fb864030c00 0x7fb8640330b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-05T19:28:24.845 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.838+0000 7fb92a165e40 1 --2- 172.21.15.167:0/2876240720 >> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] conn(0x55f92853b810 0x55f92853dc20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-05T19:28:24.845 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.838+0000 7fb92a165e40 1 -- 172.21.15.167:0/2876240720 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:1 6.19 6:9a6dd8c0:gc::gc.0:head [create,call version.check_conds in=74b,call rgw_gc.rgw_gc_queue_init in=45b,call version.set in=34b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x55f92853e2c0 con 0x55f92853b810 2023-10-05T19:28:24.845 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.838+0000 7fb927c9a700 1 --2- 172.21.15.167:0/2876240720 >> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] conn(0x55f92853b810 0x55f92853dc20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-05T19:28:24.845 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.838+0000 7fb92849b700 1 --2- 172.21.15.167:0/2876240720 >> [v2:172.21.15.167:6824/29889,v1:172.21.15.167:6825/29889] conn(0x7fb864030c00 0x7fb8640330b0 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x55f928520a00 tx=0x7fb918000f90 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-05T19:28:24.845 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.838+0000 7fb927c9a700 1 --2- 172.21.15.167:0/2876240720 >> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] conn(0x55f92853b810 0x55f92853dc20 crc :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-05T19:28:24.848 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.842+0000 7fb927c9a700 1 -- 172.21.15.167:0/2876240720 <== osd.1 v2:172.21.15.167:6800/30035 1 ==== osd_op_reply(1 gc.0 [create,call,call,call] v30'19 uv3 ondisk = -125 ((125) Operation canceled)) v8 ==== 274+0+0 (crc 0 0 0) 0x7fb910039040 con 0x55f92853b810 2023-10-05T19:28:24.848 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.842+0000 7fb92a165e40 1 --2- 172.21.15.167:0/2876240720 >> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] conn(0x55f928537450 0x55f928541930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-05T19:28:24.848 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.842+0000 7fb92a165e40 1 -- 172.21.15.167:0/2876240720 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:2 6.18 6:1e85421f:gc::gc.1:head [create,call version.check_conds in=74b,call rgw_gc.rgw_gc_queue_init in=45b,call version.set in=34b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x55f928541e70 con 0x55f928537450 2023-10-05T19:28:24.848 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.842+0000 7fb927499700 1 --2- 172.21.15.167:0/2876240720 >> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] conn(0x55f928537450 0x55f928541930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-05T19:28:24.849 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.842+0000 7fb927499700 1 --2- 172.21.15.167:0/2876240720 >> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] conn(0x55f928537450 0x55f928541930 crc :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-05T19:28:24.851 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.846+0000 7fb927499700 1 -- 172.21.15.167:0/2876240720 <== osd.0 v2:172.21.15.118:6800/27690 1 ==== osd_op_reply(2 gc.1 [create,call,call,call] v30'13 uv5 ondisk = -125 ((125) Operation canceled)) v8 ==== 274+0+0 (crc 0 0 0) 0x7fb920032040 con 0x55f928537450 2023-10-05T19:28:24.851 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.846+0000 7fb92a165e40 1 --2- 172.21.15.167:0/2876240720 >> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] conn(0x55f928542ec0 0x55f9285452a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-05T19:28:24.851 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.846+0000 7fb92a165e40 1 -- 172.21.15.167:0/2876240720 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:3 6.b 6:d242335b:gc::gc.2:head [create,call version.check_conds in=74b,call rgw_gc.rgw_gc_queue_init in=45b,call version.set in=34b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x55f928545900 con 0x55f928542ec0 2023-10-05T19:28:24.852 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.846+0000 7fb92849b700 1 --2- 172.21.15.167:0/2876240720 >> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] conn(0x55f928542ec0 0x55f9285452a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-05T19:28:24.852 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.846+0000 7fb92849b700 1 --2- 172.21.15.167:0/2876240720 >> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] conn(0x55f928542ec0 0x55f9285452a0 crc :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-05T19:28:24.854 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.850+0000 7fb92849b700 1 -- 172.21.15.167:0/2876240720 <== osd.3 v2:172.21.15.167:6808/30036 1 ==== osd_op_reply(3 gc.2 [create,call,call,call] v30'13 uv3 ondisk = -125 ((125) Operation canceled)) v8 ==== 274+0+0 (crc 0 0 0) 0x7fb918050040 con 0x55f928542ec0 2023-10-05T19:28:24.854 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.850+0000 7fb92a165e40 1 -- 172.21.15.167:0/2876240720 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:4 6.4 6:2166d8e1:gc::gc.3:head [create,call version.check_conds in=74b,call rgw_gc.rgw_gc_queue_init in=45b,call version.set in=34b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x55f928545f50 con 0x55f92853b810 2023-10-05T19:28:24.857 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.850+0000 7fb927c9a700 1 -- 172.21.15.167:0/2876240720 <== osd.1 v2:172.21.15.167:6800/30035 2 ==== osd_op_reply(4 gc.3 [create,call,call,call] v30'42 uv7 ondisk = -125 ((125) Operation canceled)) v8 ==== 274+0+0 (crc 0 0 0) 0x7fb910039040 con 0x55f92853b810 2023-10-05T19:28:24.857 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.850+0000 7fb92a165e40 1 --2- 172.21.15.167:0/2876240720 >> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] conn(0x55f928546bf0 0x55f928548fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-05T19:28:24.857 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.850+0000 7fb92a165e40 1 -- 172.21.15.167:0/2876240720 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:5 6.3 6:c60da066:gc::gc.4:head [create,call version.check_conds in=74b,call rgw_gc.rgw_gc_queue_init in=45b,call version.set in=34b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x55f928549620 con 0x55f928546bf0 2023-10-05T19:28:24.857 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.850+0000 7fb927c9a700 1 --2- 172.21.15.167:0/2876240720 >> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] conn(0x55f928546bf0 0x55f928548fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-05T19:28:24.858 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.850+0000 7fb927c9a700 1 --2- 172.21.15.167:0/2876240720 >> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] conn(0x55f928546bf0 0x55f928548fd0 crc :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-05T19:28:24.860 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.854+0000 7fb927c9a700 1 -- 172.21.15.167:0/2876240720 <== osd.2 v2:172.21.15.167:6816/30037 1 ==== osd_op_reply(5 gc.4 [create,call,call,call] v30'29 uv7 ondisk = -125 ((125) Operation canceled)) v8 ==== 274+0+0 (crc 0 0 0) 0x7fb910039040 con 0x55f928546bf0 2023-10-05T19:28:24.860 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.854+0000 7fb92a165e40 1 -- 172.21.15.167:0/2876240720 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:6 6.9 6:954be8d2:gc::gc.5:head [create,call version.check_conds in=74b,call rgw_gc.rgw_gc_queue_init in=45b,call version.set in=34b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x55f928549c70 con 0x55f928542ec0 2023-10-05T19:28:24.862 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.858+0000 7fb92849b700 1 -- 172.21.15.167:0/2876240720 <== osd.3 v2:172.21.15.167:6808/30036 2 ==== osd_op_reply(6 gc.5 [create,call,call,call] v30'19 uv4 ondisk = -125 ((125) Operation canceled)) v8 ==== 274+0+0 (crc 0 0 0) 0x7fb918050040 con 0x55f928542ec0 2023-10-05T19:28:24.862 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.858+0000 7fb92a165e40 1 -- 172.21.15.167:0/2876240720 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:7 6.e 6:748722ff:gc::gc.6:head [create,call version.check_conds in=74b,call rgw_gc.rgw_gc_queue_init in=45b,call version.set in=34b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x55f92854a290 con 0x55f92853b810 2023-10-05T19:28:24.866 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.858+0000 7fb927c9a700 1 -- 172.21.15.167:0/2876240720 <== osd.1 v2:172.21.15.167:6800/30035 3 ==== osd_op_reply(7 gc.6 [create,call,call,call] v30'15 uv5 ondisk = -125 ((125) Operation canceled)) v8 ==== 274+0+0 (crc 0 0 0) 0x7fb910039040 con 0x55f92853b810 2023-10-05T19:28:24.866 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.858+0000 7fb92a165e40 1 -- 172.21.15.167:0/2876240720 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:8 6.1d 6:beaa4be4:gc::gc.7:head [create,call version.check_conds in=74b,call rgw_gc.rgw_gc_queue_init in=45b,call version.set in=34b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x55f92854a290 con 0x55f928537450 2023-10-05T19:28:24.868 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.862+0000 7fb927499700 1 -- 172.21.15.167:0/2876240720 <== osd.0 v2:172.21.15.118:6800/27690 2 ==== osd_op_reply(8 gc.7 [create,call,call,call] v30'19 uv4 ondisk = -125 ((125) Operation canceled)) v8 ==== 274+0+0 (crc 0 0 0) 0x7fb920032040 con 0x55f928537450 2023-10-05T19:28:24.868 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.862+0000 7fb92a165e40 1 -- 172.21.15.167:0/2876240720 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:9 6.12 6:4f78e9a2:gc::gc.8:head [create,call version.check_conds in=74b,call rgw_gc.rgw_gc_queue_init in=45b,call version.set in=34b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x55f92854a8b0 con 0x55f928546bf0 2023-10-05T19:28:24.871 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.866+0000 7fb927c9a700 1 -- 172.21.15.167:0/2876240720 <== osd.2 v2:172.21.15.167:6816/30037 2 ==== osd_op_reply(9 gc.8 [create,call,call,call] v30'13 uv3 ondisk = -125 ((125) Operation canceled)) v8 ==== 274+0+0 (crc 0 0 0) 0x7fb910039040 con 0x55f928546bf0 2023-10-05T19:28:24.871 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.866+0000 7fb92a165e40 1 -- 172.21.15.167:0/2876240720 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:10 6.1c 6:3e426232:gc::gc.9:head [create,call version.check_conds in=74b,call rgw_gc.rgw_gc_queue_init in=45b,call version.set in=34b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x55f92854a8b0 con 0x55f928542ec0 2023-10-05T19:28:24.873 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.866+0000 7fb92849b700 1 -- 172.21.15.167:0/2876240720 <== osd.3 v2:172.21.15.167:6808/30036 3 ==== osd_op_reply(10 gc.9 [create,call,call,call] v30'19 uv3 ondisk = -125 ((125) Operation canceled)) v8 ==== 274+0+0 (crc 0 0 0) 0x7fb918050040 con 0x55f928542ec0 2023-10-05T19:28:24.874 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.866+0000 7fb92a165e40 1 -- 172.21.15.167:0/2876240720 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:11 6.3 6:c31f41d7:gc::gc.10:head [create,call version.check_conds in=74b,call rgw_gc.rgw_gc_queue_init in=45b,call version.set in=34b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x55f92854ada0 con 0x55f928546bf0 2023-10-05T19:28:24.877 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.870+0000 7fb927c9a700 1 -- 172.21.15.167:0/2876240720 <== osd.2 v2:172.21.15.167:6816/30037 3 ==== osd_op_reply(11 gc.10 [create,call,call,call] v30'30 uv11 ondisk = -125 ((125) Operation canceled)) v8 ==== 275+0+0 (crc 0 0 0) 0x7fb910039040 con 0x55f928546bf0 2023-10-05T19:28:24.877 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.870+0000 7fb92a165e40 1 -- 172.21.15.167:0/2876240720 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:12 6.13 6:c8333aab:gc::gc.11:head [create,call version.check_conds in=74b,call rgw_gc.rgw_gc_queue_init in=45b,call version.set in=34b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x55f92854ada0 con 0x55f928537450 2023-10-05T19:28:24.879 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.874+0000 7fb927499700 1 -- 172.21.15.167:0/2876240720 <== osd.0 v2:172.21.15.118:6800/27690 3 ==== osd_op_reply(12 gc.11 [create,call,call,call] v30'15 uv5 ondisk = -125 ((125) Operation canceled)) v8 ==== 275+0+0 (crc 0 0 0) 0x7fb920032040 con 0x55f928537450 2023-10-05T19:28:24.879 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.874+0000 7fb92a165e40 1 -- 172.21.15.167:0/2876240720 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:13 6.15 6:ab569f81:gc::gc.12:head [create,call version.check_conds in=74b,call rgw_gc.rgw_gc_queue_init in=45b,call version.set in=34b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x55f92854b3c0 con 0x55f928546bf0 2023-10-05T19:28:24.881 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.874+0000 7fb927c9a700 1 -- 172.21.15.167:0/2876240720 <== osd.2 v2:172.21.15.167:6816/30037 4 ==== osd_op_reply(13 gc.12 [create,call,call,call] v30'13 uv3 ondisk = -125 ((125) Operation canceled)) v8 ==== 275+0+0 (crc 0 0 0) 0x7fb910039040 con 0x55f928546bf0 2023-10-05T19:28:24.881 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.874+0000 7fb92a165e40 1 -- 172.21.15.167:0/2876240720 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:14 6.0 6:0667bb13:gc::gc.13:head [create,call version.check_conds in=74b,call rgw_gc.rgw_gc_queue_init in=45b,call version.set in=34b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x55f92854b3c0 con 0x55f928537450 2023-10-05T19:28:24.884 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.878+0000 7fb927499700 1 -- 172.21.15.167:0/2876240720 <== osd.0 v2:172.21.15.118:6800/27690 4 ==== osd_op_reply(14 gc.13 [create,call,call,call] v30'25 uv6 ondisk = -125 ((125) Operation canceled)) v8 ==== 275+0+0 (crc 0 0 0) 0x7fb920032040 con 0x55f928537450 2023-10-05T19:28:24.884 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.878+0000 7fb92a165e40 1 -- 172.21.15.167:0/2876240720 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:15 6.4 6:227f63b5:gc::gc.14:head [create,call version.check_conds in=74b,call rgw_gc.rgw_gc_queue_init in=45b,call version.set in=34b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x55f92854b8b0 con 0x55f92853b810 2023-10-05T19:28:24.887 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.882+0000 7fb927c9a700 1 -- 172.21.15.167:0/2876240720 <== osd.1 v2:172.21.15.167:6800/30035 4 ==== osd_op_reply(15 gc.14 [create,call,call,call] v30'43 uv13 ondisk = -125 ((125) Operation canceled)) v8 ==== 275+0+0 (crc 0 0 0) 0x7fb910039040 con 0x55f92853b810 2023-10-05T19:28:24.887 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.882+0000 7fb92a165e40 1 -- 172.21.15.167:0/2876240720 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:16 6.7 6:e07a6548:gc::gc.15:head [create,call version.check_conds in=74b,call rgw_gc.rgw_gc_queue_init in=45b,call version.set in=34b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x55f92854b8b0 con 0x55f928542ec0 2023-10-05T19:28:24.890 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.886+0000 7fb92849b700 1 -- 172.21.15.167:0/2876240720 <== osd.3 v2:172.21.15.167:6808/30036 4 ==== osd_op_reply(16 gc.15 [create,call,call,call] v30'23 uv7 ondisk = -125 ((125) Operation canceled)) v8 ==== 275+0+0 (crc 0 0 0) 0x7fb918050040 con 0x55f928542ec0 2023-10-05T19:28:24.890 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.886+0000 7fb92a165e40 1 -- 172.21.15.167:0/2876240720 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:17 6.3 6:c623b173:gc::gc.16:head [create,call version.check_conds in=74b,call rgw_gc.rgw_gc_queue_init in=45b,call version.set in=34b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x55f92854b8b0 con 0x55f928546bf0 2023-10-05T19:28:24.893 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.886+0000 7fb927c9a700 1 -- 172.21.15.167:0/2876240720 <== osd.2 v2:172.21.15.167:6816/30037 5 ==== osd_op_reply(17 gc.16 [create,call,call,call] v30'31 uv13 ondisk = -125 ((125) Operation canceled)) v8 ==== 275+0+0 (crc 0 0 0) 0x7fb910039040 con 0x55f928546bf0 2023-10-05T19:28:24.893 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.886+0000 7fb92a165e40 1 -- 172.21.15.167:0/2876240720 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:18 6.16 6:6849fd06:gc::gc.17:head [create,call version.check_conds in=74b,call rgw_gc.rgw_gc_queue_init in=45b,call version.set in=34b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x55f92854b8b0 con 0x55f928542ec0 2023-10-05T19:28:24.896 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.890+0000 7fb92849b700 1 -- 172.21.15.167:0/2876240720 <== osd.3 v2:172.21.15.167:6808/30036 5 ==== osd_op_reply(18 gc.17 [create,call,call,call] v30'13 uv7 ondisk = -125 ((125) Operation canceled)) v8 ==== 275+0+0 (crc 0 0 0) 0x7fb918050040 con 0x55f928542ec0 2023-10-05T19:28:24.896 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.890+0000 7fb92a165e40 1 -- 172.21.15.167:0/2876240720 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:19 6.d 6:b019ea83:gc::gc.18:head [create,call version.check_conds in=74b,call rgw_gc.rgw_gc_queue_init in=45b,call version.set in=34b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x55f92854b8b0 con 0x55f92853b810 2023-10-05T19:28:24.899 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.894+0000 7fb927c9a700 1 -- 172.21.15.167:0/2876240720 <== osd.1 v2:172.21.15.167:6800/30035 5 ==== osd_op_reply(19 gc.18 [create,call,call,call] v30'21 uv5 ondisk = -125 ((125) Operation canceled)) v8 ==== 275+0+0 (crc 0 0 0) 0x7fb910039040 con 0x55f92853b810 2023-10-05T19:28:24.899 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.894+0000 7fb92a165e40 1 -- 172.21.15.167:0/2876240720 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:20 6.4 6:24c8e1ae:gc::gc.19:head [create,call version.check_conds in=74b,call rgw_gc.rgw_gc_queue_init in=45b,call version.set in=34b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x55f92854b8b0 con 0x55f92853b810 2023-10-05T19:28:24.901 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.894+0000 7fb927c9a700 1 -- 172.21.15.167:0/2876240720 <== osd.1 v2:172.21.15.167:6800/30035 6 ==== osd_op_reply(20 gc.19 [create,call,call,call] v30'44 uv15 ondisk = -125 ((125) Operation canceled)) v8 ==== 275+0+0 (crc 0 0 0) 0x7fb910039040 con 0x55f92853b810 2023-10-05T19:28:24.901 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.894+0000 7fb92a165e40 1 -- 172.21.15.167:0/2876240720 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:21 6.7 6:e622f76a:gc::gc.20:head [create,call version.check_conds in=74b,call rgw_gc.rgw_gc_queue_init in=45b,call version.set in=34b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x55f92854b8b0 con 0x55f928542ec0 2023-10-05T19:28:24.905 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.898+0000 7fb92849b700 1 -- 172.21.15.167:0/2876240720 <== osd.3 v2:172.21.15.167:6808/30036 6 ==== osd_op_reply(21 gc.20 [create,call,call,call] v30'24 uv9 ondisk = -125 ((125) Operation canceled)) v8 ==== 275+0+0 (crc 0 0 0) 0x7fb918050040 con 0x55f928542ec0 2023-10-05T19:28:24.905 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.898+0000 7fb92a165e40 1 -- 172.21.15.167:0/2876240720 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:22 6.a 6:5127e642:gc::gc.21:head [create,call version.check_conds in=74b,call rgw_gc.rgw_gc_queue_init in=45b,call version.set in=34b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x55f92854b8b0 con 0x55f928546bf0 2023-10-05T19:28:24.907 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.902+0000 7fb927c9a700 1 -- 172.21.15.167:0/2876240720 <== osd.2 v2:172.21.15.167:6816/30037 6 ==== osd_op_reply(22 gc.21 [create,call,call,call] v30'23 uv7 ondisk = -125 ((125) Operation canceled)) v8 ==== 275+0+0 (crc 0 0 0) 0x7fb910039040 con 0x55f928546bf0 2023-10-05T19:28:24.907 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.902+0000 7fb92a165e40 1 -- 172.21.15.167:0/2876240720 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:23 6.11 6:8eb99c53:gc::gc.22:head [create,call version.check_conds in=74b,call rgw_gc.rgw_gc_queue_init in=45b,call version.set in=34b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x55f92854b8b0 con 0x55f928542ec0 2023-10-05T19:28:24.910 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.902+0000 7fb92849b700 1 -- 172.21.15.167:0/2876240720 <== osd.3 v2:172.21.15.167:6808/30036 7 ==== osd_op_reply(23 gc.22 [create,call,call,call] v30'15 uv3 ondisk = -125 ((125) Operation canceled)) v8 ==== 275+0+0 (crc 0 0 0) 0x7fb918050040 con 0x55f928542ec0 2023-10-05T19:28:24.910 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.902+0000 7fb92a165e40 1 -- 172.21.15.167:0/2876240720 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:24 6.8 6:17d7e8ed:gc::gc.23:head [create,call version.check_conds in=74b,call rgw_gc.rgw_gc_queue_init in=45b,call version.set in=34b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x55f92854b940 con 0x55f928542ec0 2023-10-05T19:28:24.912 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.906+0000 7fb92849b700 1 -- 172.21.15.167:0/2876240720 <== osd.3 v2:172.21.15.167:6808/30036 8 ==== osd_op_reply(24 gc.23 [create,call,call,call] v30'20 uv5 ondisk = -125 ((125) Operation canceled)) v8 ==== 275+0+0 (crc 0 0 0) 0x7fb918050040 con 0x55f928542ec0 2023-10-05T19:28:24.912 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.906+0000 7fb92a165e40 1 -- 172.21.15.167:0/2876240720 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:25 6.1b 6:db9d130d:gc::gc.24:head [create,call version.check_conds in=74b,call rgw_gc.rgw_gc_queue_init in=45b,call version.set in=34b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x55f92854b940 con 0x55f928542ec0 2023-10-05T19:28:24.914 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.906+0000 7fb92849b700 1 -- 172.21.15.167:0/2876240720 <== osd.3 v2:172.21.15.167:6808/30036 9 ==== osd_op_reply(25 gc.24 [create,call,call,call] v30'9 uv3 ondisk = -125 ((125) Operation canceled)) v8 ==== 275+0+0 (crc 0 0 0) 0x7fb918050040 con 0x55f928542ec0 2023-10-05T19:28:24.914 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.906+0000 7fb92a165e40 1 -- 172.21.15.167:0/2876240720 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:26 6.9 6:921ecdb1:gc::gc.25:head [create,call version.check_conds in=74b,call rgw_gc.rgw_gc_queue_init in=45b,call version.set in=34b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x55f92854b940 con 0x55f928542ec0 2023-10-05T19:28:24.916 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.910+0000 7fb92849b700 1 -- 172.21.15.167:0/2876240720 <== osd.3 v2:172.21.15.167:6808/30036 10 ==== osd_op_reply(26 gc.25 [create,call,call,call] v30'20 uv10 ondisk = -125 ((125) Operation canceled)) v8 ==== 275+0+0 (crc 0 0 0) 0x7fb918050040 con 0x55f928542ec0 2023-10-05T19:28:24.916 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.910+0000 7fb92a165e40 1 -- 172.21.15.167:0/2876240720 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:27 6.1f 6:fde4dd55:gc::gc.26:head [create,call version.check_conds in=74b,call rgw_gc.rgw_gc_queue_init in=45b,call version.set in=34b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x55f92854b940 con 0x55f928537450 2023-10-05T19:28:24.919 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.914+0000 7fb927499700 1 -- 172.21.15.167:0/2876240720 <== osd.0 v2:172.21.15.118:6800/27690 5 ==== osd_op_reply(27 gc.26 [create,call,call,call] v30'15 uv5 ondisk = -125 ((125) Operation canceled)) v8 ==== 275+0+0 (crc 0 0 0) 0x7fb920032040 con 0x55f928537450 2023-10-05T19:28:24.919 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.914+0000 7fb92a165e40 1 -- 172.21.15.167:0/2876240720 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:28 6.1a 6:5909e671:gc::gc.27:head [create,call version.check_conds in=74b,call rgw_gc.rgw_gc_queue_init in=45b,call version.set in=34b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x55f92854b940 con 0x55f928542ec0 2023-10-05T19:28:24.921 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.914+0000 7fb92849b700 1 -- 172.21.15.167:0/2876240720 <== osd.3 v2:172.21.15.167:6808/30036 11 ==== osd_op_reply(28 gc.27 [create,call,call,call] v30'13 uv7 ondisk = -125 ((125) Operation canceled)) v8 ==== 275+0+0 (crc 0 0 0) 0x7fb918050040 con 0x55f928542ec0 2023-10-05T19:28:24.921 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.914+0000 7fb92a165e40 1 -- 172.21.15.167:0/2876240720 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:29 6.1d 6:b88e3271:gc::gc.28:head [create,call version.check_conds in=74b,call rgw_gc.rgw_gc_queue_init in=45b,call version.set in=34b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x55f92854b940 con 0x55f928537450 2023-10-05T19:28:24.924 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.918+0000 7fb927499700 1 -- 172.21.15.167:0/2876240720 <== osd.0 v2:172.21.15.118:6800/27690 6 ==== osd_op_reply(29 gc.28 [create,call,call,call] v30'20 uv8 ondisk = -125 ((125) Operation canceled)) v8 ==== 275+0+0 (crc 0 0 0) 0x7fb920032040 con 0x55f928537450 2023-10-05T19:28:24.924 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.918+0000 7fb92a165e40 1 -- 172.21.15.167:0/2876240720 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:30 6.7 6:e47c9322:gc::gc.29:head [create,call version.check_conds in=74b,call rgw_gc.rgw_gc_queue_init in=45b,call version.set in=34b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x55f92854b940 con 0x55f928542ec0 2023-10-05T19:28:24.927 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.922+0000 7fb92849b700 1 -- 172.21.15.167:0/2876240720 <== osd.3 v2:172.21.15.167:6808/30036 12 ==== osd_op_reply(30 gc.29 [create,call,call,call] v30'25 uv11 ondisk = -125 ((125) Operation canceled)) v8 ==== 275+0+0 (crc 0 0 0) 0x7fb918050040 con 0x55f928542ec0 2023-10-05T19:28:24.927 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.922+0000 7fb92a165e40 1 -- 172.21.15.167:0/2876240720 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:31 6.0 6:013d6c33:gc::gc.30:head [create,call version.check_conds in=74b,call rgw_gc.rgw_gc_queue_init in=45b,call version.set in=34b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x55f92854b940 con 0x55f928537450 2023-10-05T19:28:24.930 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.926+0000 7fb927499700 1 -- 172.21.15.167:0/2876240720 <== osd.0 v2:172.21.15.118:6800/27690 7 ==== osd_op_reply(31 gc.30 [create,call,call,call] v30'26 uv12 ondisk = -125 ((125) Operation canceled)) v8 ==== 275+0+0 (crc 0 0 0) 0x7fb920032040 con 0x55f928537450 2023-10-05T19:28:24.930 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.926+0000 7fb92a165e40 1 -- 172.21.15.167:0/2876240720 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:32 6.f 6:f24cb4a8:gc::gc.31:head [create,call version.check_conds in=74b,call rgw_gc.rgw_gc_queue_init in=45b,call version.set in=34b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x55f92854b940 con 0x55f928546bf0 2023-10-05T19:28:24.933 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.926+0000 7fb927c9a700 1 -- 172.21.15.167:0/2876240720 <== osd.2 v2:172.21.15.167:6816/30037 7 ==== osd_op_reply(32 gc.31 [create,call,call,call] v30'19 uv5 ondisk = -125 ((125) Operation canceled)) v8 ==== 275+0+0 (crc 0 0 0) 0x7fb910039040 con 0x55f928546bf0 2023-10-05T19:28:24.935 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.930+0000 7fb86affd700 20 reqs_thread_entry: start 2023-10-05T19:28:24.936 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.930+0000 7fb868ff9700 20 reqs_thread_entry: start 2023-10-05T19:28:24.936 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.930+0000 7fb92a165e40 20 init_complete bucket index max shards: 11 2023-10-05T19:28:24.936 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.930+0000 7fb92a165e40 10 rgw notify: Started notification manager with: 1 workers 2023-10-05T19:28:24.936 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.930+0000 7fb856ffd700 20 rgw notify: INFO: next queues processing will happen at: Thu Oct 5 19:28:55 2023 2023-10-05T19:28:24.937 INFO:teuthology.orchestra.run.smithi167.stderr: 2023-10-05T19:28:24.937 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.930+0000 7fb8567fc700 20 reqs_thread_entry: start 2023-10-05T19:28:24.937 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.930+0000 7fb92a165e40 20 get_system_obj_state: rctx=0x7ffd8616fbb8 obj=default.rgw.meta:users.uid:foo state=0x55f9283b40b0 s->prefetch_data=0 2023-10-05T19:28:24.937 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.930+0000 7fb92a165e40 10 cache get: name=default.rgw.meta+users.uid+foo : miss 2023-10-05T19:28:24.937 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.930+0000 7fb92a165e40 1 -- 172.21.15.167:0/2080866673 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:102 8.18 8:1e6ca5ed:users.uid::foo:head [call version.read in=11b,getxattrs,stat] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x55f9285aa030 con 0x55f9284514b0 2023-10-05T19:28:24.938 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.930+0000 7fb92849b700 1 -- 172.21.15.167:0/2080866673 <== osd.1 v2:172.21.15.167:6800/30035 35 ==== osd_op_reply(102 foo [call,getxattrs,stat] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 ==== 231+0+0 (crc 0 0 0) 0x7fb918050040 con 0x55f9284514b0 2023-10-05T19:28:24.938 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.930+0000 7fb92a165e40 10 cache put: name=default.rgw.meta+users.uid+foo info.flags=0x0 2023-10-05T19:28:24.938 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.930+0000 7fb92a165e40 10 adding default.rgw.meta+users.uid+foo to cache LRU end 2023-10-05T19:28:24.938 INFO:teuthology.orchestra.run.smithi167.stderr:could not remove user: unable to remove user, user does not exist 2023-10-05T19:28:24.938 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.930+0000 7fb92a165e40 1 -- 172.21.15.167:0/2080866673 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:103 7.1f 7:f95f44c2:::notify.0:head [watch unwatch cookie 94528611621328] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x55f9285aa030 con 0x7fb91001a070 2023-10-05T19:28:24.940 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.934+0000 7fb927499700 1 -- 172.21.15.167:0/2080866673 <== osd.2 v2:172.21.15.167:6816/30037 17 ==== osd_op_reply(103 notify.0 [watch unwatch cookie 94528611621328] v30'17 uv15 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7fb920032040 con 0x7fb91001a070 2023-10-05T19:28:24.940 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.934+0000 7fb92a165e40 20 remove_watcher() i=0 2023-10-05T19:28:24.941 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.934+0000 7fb92a165e40 2 removed watcher, disabling cache 2023-10-05T19:28:24.941 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.934+0000 7fb92a165e40 1 -- 172.21.15.167:0/2080866673 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:104 7.0 7:05bf5b68:::notify.1:head [watch unwatch cookie 94528611650096] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x55f9285aa030 con 0x55f9284514b0 2023-10-05T19:28:24.943 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.938+0000 7fb92849b700 1 -- 172.21.15.167:0/2080866673 <== osd.1 v2:172.21.15.167:6800/30035 36 ==== osd_op_reply(104 notify.1 [watch unwatch cookie 94528611650096] v30'17 uv15 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7fb918050040 con 0x55f9284514b0 2023-10-05T19:28:24.944 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.938+0000 7fb92a165e40 20 remove_watcher() i=1 2023-10-05T19:28:24.944 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.938+0000 7fb92a165e40 1 -- 172.21.15.167:0/2080866673 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:105 7.15 7:a93a5511:::notify.2:head [watch unwatch cookie 94528611651200] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x55f9285aa030 con 0x55f928512820 2023-10-05T19:28:24.947 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.942+0000 7fb927c9a700 1 -- 172.21.15.167:0/2080866673 <== osd.3 v2:172.21.15.167:6808/30036 24 ==== osd_op_reply(105 notify.2 [watch unwatch cookie 94528611651200] v30'17 uv15 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7fb910039040 con 0x55f928512820 2023-10-05T19:28:24.947 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.942+0000 7fb92a165e40 20 remove_watcher() i=2 2023-10-05T19:28:24.947 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.942+0000 7fb92a165e40 1 -- 172.21.15.167:0/2080866673 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:106 7.e 7:7759931f:::notify.3:head [watch unwatch cookie 94528611658320] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x55f9285aa030 con 0x7fb91001a070 2023-10-05T19:28:24.949 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.942+0000 7fb927499700 1 -- 172.21.15.167:0/2080866673 <== osd.2 v2:172.21.15.167:6816/30037 18 ==== osd_op_reply(106 notify.3 [watch unwatch cookie 94528611658320] v30'17 uv15 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7fb920032040 con 0x7fb91001a070 2023-10-05T19:28:24.950 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.942+0000 7fb92a165e40 20 remove_watcher() i=3 2023-10-05T19:28:24.950 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.942+0000 7fb92a165e40 1 -- 172.21.15.167:0/2080866673 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:107 7.d 7:b4812045:::notify.4:head [watch unwatch cookie 94528611659424] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x55f9285aa030 con 0x55f9284514b0 2023-10-05T19:28:24.952 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.946+0000 7fb92849b700 1 -- 172.21.15.167:0/2080866673 <== osd.1 v2:172.21.15.167:6800/30035 37 ==== osd_op_reply(107 notify.4 [watch unwatch cookie 94528611659424] v30'17 uv15 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7fb918050040 con 0x55f9284514b0 2023-10-05T19:28:24.953 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.946+0000 7fb92a165e40 20 remove_watcher() i=4 2023-10-05T19:28:24.953 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.946+0000 7fb92a165e40 1 -- 172.21.15.167:0/2080866673 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:108 7.3 7:c609908c:::notify.5:head [watch unwatch cookie 94528611652736] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x55f9285aa030 con 0x7fb920010070 2023-10-05T19:28:24.955 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.950+0000 7fb92849b700 1 -- 172.21.15.167:0/2080866673 <== osd.0 v2:172.21.15.118:6800/27690 29 ==== osd_op_reply(108 notify.5 [watch unwatch cookie 94528611652736] v30'17 uv15 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7fb918050040 con 0x7fb920010070 2023-10-05T19:28:24.955 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.950+0000 7fb92a165e40 20 remove_watcher() i=5 2023-10-05T19:28:24.955 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.950+0000 7fb92a165e40 1 -- 172.21.15.167:0/2080866673 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:109 7.14 7:2b04a3e9:::notify.6:head [watch unwatch cookie 94528611664464] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x55f9285aa030 con 0x7fb920010070 2023-10-05T19:28:24.958 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.950+0000 7fb92849b700 1 -- 172.21.15.167:0/2080866673 <== osd.0 v2:172.21.15.118:6800/27690 30 ==== osd_op_reply(109 notify.6 [watch unwatch cookie 94528611664464] v30'17 uv15 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7fb918050040 con 0x7fb920010070 2023-10-05T19:28:24.958 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.950+0000 7fb92a165e40 20 remove_watcher() i=6 2023-10-05T19:28:24.958 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.954+0000 7fb92a165e40 1 -- 172.21.15.167:0/2080866673 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:110 7.9 7:93e5b521:::notify.7:head [watch unwatch cookie 94528611665568] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x55f9285aa030 con 0x7fb920010070 2023-10-05T19:28:24.960 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.954+0000 7fb92849b700 1 -- 172.21.15.167:0/2080866673 <== osd.0 v2:172.21.15.118:6800/27690 31 ==== osd_op_reply(110 notify.7 [watch unwatch cookie 94528611665568] v30'17 uv15 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7fb918050040 con 0x7fb920010070 2023-10-05T19:28:24.960 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.954+0000 7fb92a165e40 20 remove_watcher() i=7 2023-10-05T19:28:24.962 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.958+0000 7fb873fff700 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2023-10-05T19:28:24.962 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.958+0000 7fb92a165e40 1 -- 172.21.15.167:0/2080866673 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:111 6.4 6:22d26bf9:::data_loggenerations_metadata:head [watch unwatch cookie 94528611668848] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x55f92851c740 con 0x55f9284514b0 2023-10-05T19:28:24.966 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.958+0000 7fb92849b700 1 -- 172.21.15.167:0/2080866673 <== osd.1 v2:172.21.15.167:6800/30035 38 ==== osd_op_reply(111 data_loggenerations_metadata [watch unwatch cookie 94528611668848] v30'45 uv1 ondisk = 0) v8 ==== 172+0+0 (crc 0 0 0) 0x7fb918050040 con 0x55f9284514b0 2023-10-05T19:28:24.966 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.958+0000 7fb92a165e40 1 -- 172.21.15.167:0/2080866673 >> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] conn(0x7fb920010070 msgr2=0x7fb92001c430 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-05T19:28:24.966 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.958+0000 7fb92a165e40 1 --2- 172.21.15.167:0/2080866673 >> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] conn(0x7fb920010070 0x7fb92001c430 crc :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-05T19:28:24.966 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.958+0000 7fb92a165e40 1 -- 172.21.15.167:0/2080866673 >> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] conn(0x55f9284514b0 msgr2=0x55f92850e9f0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-05T19:28:24.966 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.958+0000 7fb92a165e40 1 --2- 172.21.15.167:0/2080866673 >> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] conn(0x55f9284514b0 0x55f92850e9f0 crc :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-05T19:28:24.966 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.958+0000 7fb92a165e40 1 -- 172.21.15.167:0/2080866673 >> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] conn(0x7fb91001a070 msgr2=0x7fb910043750 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-05T19:28:24.966 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.958+0000 7fb92a165e40 1 --2- 172.21.15.167:0/2080866673 >> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] conn(0x7fb91001a070 0x7fb910043750 crc :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-05T19:28:24.966 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.958+0000 7fb92a165e40 1 -- 172.21.15.167:0/2080866673 >> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] conn(0x55f928512820 msgr2=0x55f928514c00 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-05T19:28:24.967 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.958+0000 7fb92a165e40 1 --2- 172.21.15.167:0/2080866673 >> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] conn(0x55f928512820 0x55f928514c00 crc :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-05T19:28:24.968 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.962+0000 7fb92a165e40 1 -- 172.21.15.167:0/2080866673 >> [v2:172.21.15.167:6824/29889,v1:172.21.15.167:6825/29889] conn(0x7fb90c030c50 msgr2=0x7fb90c033100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-05T19:28:24.968 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.962+0000 7fb92a165e40 1 --2- 172.21.15.167:0/2080866673 >> [v2:172.21.15.167:6824/29889,v1:172.21.15.167:6825/29889] conn(0x7fb90c030c50 0x7fb90c033100 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7fb920000f80 tx=0x7fb920013000 comp rx=0 tx=0).stop 2023-10-05T19:28:24.968 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.962+0000 7fb92a165e40 1 -- 172.21.15.167:0/2080866673 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55f9284501a0 msgr2=0x55f928502f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-05T19:28:24.968 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.962+0000 7fb92a165e40 1 --2- 172.21.15.167:0/2080866673 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55f9284501a0 0x55f928502f00 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7fb9100018e0 tx=0x7fb910016000 comp rx=0 tx=0).stop 2023-10-05T19:28:24.969 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.962+0000 7fb92849b700 1 -- 172.21.15.167:0/2080866673 reap_dead start 2023-10-05T19:28:24.969 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.962+0000 7fb92a165e40 1 -- 172.21.15.167:0/2080866673 shutdown_connections 2023-10-05T19:28:24.969 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.962+0000 7fb92a165e40 1 --2- 172.21.15.167:0/2080866673 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55f9284501a0 0x55f928502f00 unknown :-1 s=CLOSED pgs=161 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-05T19:28:24.969 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.962+0000 7fb92a165e40 1 -- 172.21.15.167:0/2080866673 >> 172.21.15.167:0/2080866673 conn(0x55f9284d6830 msgr2=0x55f9284f6770 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-05T19:28:24.969 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.962+0000 7fb92a165e40 1 -- 172.21.15.167:0/2080866673 shutdown_connections 2023-10-05T19:28:24.969 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.962+0000 7fb92a165e40 1 -- 172.21.15.167:0/2080866673 wait complete. 2023-10-05T19:28:24.969 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.962+0000 7fb92a165e40 1 -- 172.21.15.167:0/2876240720 >> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] conn(0x55f928537450 msgr2=0x55f928541930 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-05T19:28:24.969 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.962+0000 7fb92a165e40 1 --2- 172.21.15.167:0/2876240720 >> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] conn(0x55f928537450 0x55f928541930 crc :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-05T19:28:24.969 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.962+0000 7fb92a165e40 1 -- 172.21.15.167:0/2876240720 >> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] conn(0x55f92853b810 msgr2=0x55f92853dc20 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-05T19:28:24.969 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.962+0000 7fb92a165e40 1 --2- 172.21.15.167:0/2876240720 >> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] conn(0x55f92853b810 0x55f92853dc20 crc :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-05T19:28:24.969 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.962+0000 7fb92a165e40 1 -- 172.21.15.167:0/2876240720 >> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] conn(0x55f928546bf0 msgr2=0x55f928548fd0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-05T19:28:24.970 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.962+0000 7fb92a165e40 1 --2- 172.21.15.167:0/2876240720 >> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] conn(0x55f928546bf0 0x55f928548fd0 crc :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-05T19:28:24.970 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.962+0000 7fb92a165e40 1 -- 172.21.15.167:0/2876240720 >> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] conn(0x55f928542ec0 msgr2=0x55f9285452a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-05T19:28:24.970 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.962+0000 7fb92a165e40 1 --2- 172.21.15.167:0/2876240720 >> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] conn(0x55f928542ec0 0x55f9285452a0 crc :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-05T19:28:24.970 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.962+0000 7fb92a165e40 1 -- 172.21.15.167:0/2876240720 >> [v2:172.21.15.167:6824/29889,v1:172.21.15.167:6825/29889] conn(0x7fb864030c00 msgr2=0x7fb8640330b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-05T19:28:24.970 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.962+0000 7fb92a165e40 1 --2- 172.21.15.167:0/2876240720 >> [v2:172.21.15.167:6824/29889,v1:172.21.15.167:6825/29889] conn(0x7fb864030c00 0x7fb8640330b0 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x55f928520a00 tx=0x7fb918000f90 comp rx=0 tx=0).stop 2023-10-05T19:28:24.970 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.962+0000 7fb927c9a700 1 -- 172.21.15.167:0/2876240720 reap_dead start 2023-10-05T19:28:24.970 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.962+0000 7fb92a165e40 1 -- 172.21.15.167:0/2876240720 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55f92852b1a0 msgr2=0x55f928530c20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-05T19:28:24.970 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.962+0000 7fb92a165e40 1 --2- 172.21.15.167:0/2876240720 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55f92852b1a0 0x55f928530c20 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7fb920037040 tx=0x7fb92002f000 comp rx=0 tx=0).stop 2023-10-05T19:28:24.970 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.962+0000 7fb92a165e40 1 -- 172.21.15.167:0/2876240720 shutdown_connections 2023-10-05T19:28:24.970 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.962+0000 7fb92a165e40 1 --2- 172.21.15.167:0/2876240720 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55f92852b1a0 0x55f928530c20 unknown :-1 s=CLOSED pgs=163 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-05T19:28:24.970 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.962+0000 7fb92a165e40 1 -- 172.21.15.167:0/2876240720 >> 172.21.15.167:0/2876240720 conn(0x55f928526560 msgr2=0x55f928524600 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-05T19:28:24.970 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.962+0000 7fb92a165e40 1 -- 172.21.15.167:0/2876240720 shutdown_connections 2023-10-05T19:28:24.971 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:24.962+0000 7fb92a165e40 1 -- 172.21.15.167:0/2876240720 wait complete. 2023-10-05T19:28:24.973 DEBUG:teuthology.orchestra.run:got remote process result: 2 2023-10-05T19:28:24.974 INFO:tasks.util.rgw:rgwadmin: client.0 : ['zone', 'placement', 'rm', '--rgw-zone', 'default', '--placement-id', 'new-placement'] 2023-10-05T19:28:24.974 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'ceph', 'zone', 'placement', 'rm', '--rgw-zone', 'default', '--placement-id', 'new-placement'] 2023-10-05T19:28:24.974 DEBUG:teuthology.orchestra.run.smithi167:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster ceph zone placement rm --rgw-zone default --placement-id new-placement 2023-10-05T19:28:25.016 INFO:teuthology.orchestra.run.smithi167.stderr:ignoring --setuser ceph since I am not root 2023-10-05T19:28:25.016 INFO:teuthology.orchestra.run.smithi167.stderr:ignoring --setgroup ceph since I am not root 2023-10-05T19:28:25.019 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.010+0000 7fde4dfd2e40 1 Processor -- start 2023-10-05T19:28:25.019 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.010+0000 7fde4dfd2e40 1 -- start start 2023-10-05T19:28:25.019 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.010+0000 7fde4dfd2e40 1 --2- >> v2:172.21.15.118:3300/0 conn(0x55af23f26320 0x55af23f28700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-05T19:28:25.019 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.010+0000 7fde4dfd2e40 1 -- --> v1:172.21.15.118:6789/0 -- auth(proto 0 26 bytes epoch 0) v1 -- 0x55af23e6ebd0 con 0x55af23e967c0 2023-10-05T19:28:25.019 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.010+0000 7fde4dfd2e40 1 -- --> v2:172.21.15.118:3300/0 -- mon_getmap magic: 0 v1 -- 0x55af23e2bee0 con 0x55af23f26320 2023-10-05T19:28:25.020 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.010+0000 7fde43fff700 1 --2- >> v2:172.21.15.118:3300/0 conn(0x55af23f26320 0x55af23f28700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-05T19:28:25.020 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.010+0000 7fde4bb07700 1 --1- >> v1:172.21.15.118:6789/0 conn(0x55af23e967c0 0x55af23f24c00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.118:6789/0 says I am v1:172.21.15.167:56334/0 (socket says 172.21.15.167:56334) 2023-10-05T19:28:25.020 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.010+0000 7fde4bb07700 1 -- 172.21.15.167:0/3289580428 learned_addr learned my addr 172.21.15.167:0/3289580428 (peer_addr_for_me v1:172.21.15.167:0/0) 2023-10-05T19:28:25.020 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.014+0000 7fde4b306700 1 -- 172.21.15.167:0/3289580428 <== mon.0 v1:172.21.15.118:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2686547935 0 0) 0x55af23e6ebd0 con 0x55af23e967c0 2023-10-05T19:28:25.020 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.014+0000 7fde4b306700 1 -- 172.21.15.167:0/3289580428 --> v1:172.21.15.118:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fde38001e10 con 0x55af23e967c0 2023-10-05T19:28:25.021 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.014+0000 7fde43fff700 1 -- 172.21.15.167:0/3289580428 >> v1:172.21.15.118:6789/0 conn(0x55af23e967c0 legacy=0x55af23f24c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-05T19:28:25.021 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.014+0000 7fde43fff700 1 -- 172.21.15.167:0/3289580428 --> v2:172.21.15.118:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55af23f29c50 con 0x55af23f26320 2023-10-05T19:28:25.021 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.014+0000 7fde43fff700 1 --2- 172.21.15.167:0/3289580428 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55af23f26320 0x55af23f28700 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7fde3400a5d0 tx=0x7fde34005a50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bbda1e50ea93044b server_cookie=0 in_seq=0 out_seq=0 2023-10-05T19:28:25.026 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.014+0000 7fde4b306700 1 -- 172.21.15.167:0/3289580428 <== mon.0 v2:172.21.15.118:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fde34007610 con 0x55af23f26320 2023-10-05T19:28:25.026 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.014+0000 7fde4b306700 1 -- 172.21.15.167:0/3289580428 <== mon.0 v2:172.21.15.118:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fde34007770 con 0x55af23f26320 2023-10-05T19:28:25.026 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.014+0000 7fde4b306700 1 -- 172.21.15.167:0/3289580428 <== mon.0 v2:172.21.15.118:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fde34007a20 con 0x55af23f26320 2023-10-05T19:28:25.027 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.014+0000 7fde4dfd2e40 1 -- 172.21.15.167:0/3289580428 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55af23f26320 msgr2=0x55af23f28700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-05T19:28:25.027 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.014+0000 7fde4dfd2e40 1 --2- 172.21.15.167:0/3289580428 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55af23f26320 0x55af23f28700 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7fde3400a5d0 tx=0x7fde34005a50 comp rx=0 tx=0).stop 2023-10-05T19:28:25.027 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.014+0000 7fde4dfd2e40 1 -- 172.21.15.167:0/3289580428 shutdown_connections 2023-10-05T19:28:25.027 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.014+0000 7fde4dfd2e40 1 --2- 172.21.15.167:0/3289580428 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55af23f26320 0x55af23f28700 unknown :-1 s=CLOSED pgs=165 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-05T19:28:25.027 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.014+0000 7fde4dfd2e40 1 -- 172.21.15.167:0/3289580428 >> 172.21.15.167:0/3289580428 conn(0x55af23f1c750 msgr2=0x55af23f1eb60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-05T19:28:25.027 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.014+0000 7fde4dfd2e40 1 -- 172.21.15.167:0/3289580428 shutdown_connections 2023-10-05T19:28:25.028 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.014+0000 7fde4dfd2e40 1 -- 172.21.15.167:0/3289580428 wait complete. 2023-10-05T19:28:25.028 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.014+0000 7fde4dfd2e40 1 Processor -- start 2023-10-05T19:28:25.028 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.014+0000 7fde4dfd2e40 1 -- start start 2023-10-05T19:28:25.028 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.014+0000 7fde4dfd2e40 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55af23e967c0 0x55af23f40dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-05T19:28:25.028 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.014+0000 7fde4dfd2e40 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x55af23defe30 con 0x55af23e967c0 2023-10-05T19:28:25.028 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.014+0000 7fde4bb07700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55af23e967c0 0x55af23f40dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-05T19:28:25.029 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.014+0000 7fde4bb07700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55af23e967c0 0x55af23f40dd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.118:3300/0 says I am v2:172.21.15.167:57026/0 (socket says 172.21.15.167:57026) 2023-10-05T19:28:25.029 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.014+0000 7fde4bb07700 1 -- 172.21.15.167:0/3595327133 learned_addr learned my addr 172.21.15.167:0/3595327133 (peer_addr_for_me v2:172.21.15.167:0/0) 2023-10-05T19:28:25.029 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.018+0000 7fde4bb07700 1 -- 172.21.15.167:0/3595327133 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fde34005cf0 con 0x55af23e967c0 2023-10-05T19:28:25.029 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.018+0000 7fde4bb07700 1 --2- 172.21.15.167:0/3595327133 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55af23e967c0 0x55af23f40dd0 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7fde3c006e30 tx=0x7fde3c006a10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=97fa2a40f1f1e139 server_cookie=0 in_seq=0 out_seq=0 2023-10-05T19:28:25.029 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.018+0000 7fde4a304700 1 -- 172.21.15.167:0/3595327133 <== mon.0 v2:172.21.15.118:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fde3c006110 con 0x55af23e967c0 2023-10-05T19:28:25.029 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.018+0000 7fde4a304700 1 -- 172.21.15.167:0/3595327133 <== mon.0 v2:172.21.15.118:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fde3c006290 con 0x55af23e967c0 2023-10-05T19:28:25.030 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.018+0000 7fde4a304700 1 -- 172.21.15.167:0/3595327133 <== mon.0 v2:172.21.15.118:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fde3c004520 con 0x55af23e967c0 2023-10-05T19:28:25.030 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.018+0000 7fde4dfd2e40 1 -- 172.21.15.167:0/3595327133 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55af23e967c0 msgr2=0x55af23f40dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-05T19:28:25.030 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.018+0000 7fde4dfd2e40 1 --2- 172.21.15.167:0/3595327133 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55af23e967c0 0x55af23f40dd0 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7fde3c006e30 tx=0x7fde3c006a10 comp rx=0 tx=0).stop 2023-10-05T19:28:25.030 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.018+0000 7fde4dfd2e40 1 -- 172.21.15.167:0/3595327133 shutdown_connections 2023-10-05T19:28:25.030 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.018+0000 7fde4dfd2e40 1 --2- 172.21.15.167:0/3595327133 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55af23e967c0 0x55af23f40dd0 unknown :-1 s=CLOSED pgs=166 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-05T19:28:25.030 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.018+0000 7fde4dfd2e40 1 -- 172.21.15.167:0/3595327133 >> 172.21.15.167:0/3595327133 conn(0x55af23f1c750 msgr2=0x55af23f3c910 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-05T19:28:25.031 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.018+0000 7fde4dfd2e40 1 -- 172.21.15.167:0/3595327133 shutdown_connections 2023-10-05T19:28:25.031 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.018+0000 7fde4dfd2e40 1 -- 172.21.15.167:0/3595327133 wait complete. 2023-10-05T19:28:25.031 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.018+0000 7fde4dfd2e40 1 Processor -- start 2023-10-05T19:28:25.031 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.018+0000 7fde4dfd2e40 1 -- start start 2023-10-05T19:28:25.031 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.018+0000 7fde4dfd2e40 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55af23e967c0 0x55af23f22390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-05T19:28:25.031 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.018+0000 7fde4dfd2e40 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x55af23dc2ed0 con 0x55af23e967c0 2023-10-05T19:28:25.032 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.018+0000 7fde4bb07700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55af23e967c0 0x55af23f22390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-05T19:28:25.032 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.018+0000 7fde4bb07700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55af23e967c0 0x55af23f22390 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.118:3300/0 says I am v2:172.21.15.167:57032/0 (socket says 172.21.15.167:57032) 2023-10-05T19:28:25.032 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.018+0000 7fde4bb07700 1 -- 172.21.15.167:0/3087593634 learned_addr learned my addr 172.21.15.167:0/3087593634 (peer_addr_for_me v2:172.21.15.167:0/0) 2023-10-05T19:28:25.032 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.018+0000 7fde4bb07700 1 -- 172.21.15.167:0/3087593634 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fde38001e10 con 0x55af23e967c0 2023-10-05T19:28:25.032 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.018+0000 7fde4bb07700 1 --2- 172.21.15.167:0/3087593634 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55af23e967c0 0x55af23f22390 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7fde3c001530 tx=0x7fde3c010000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-05T19:28:25.032 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.018+0000 7fde48b01700 1 -- 172.21.15.167:0/3087593634 <== mon.0 v2:172.21.15.118:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fde3c006110 con 0x55af23e967c0 2023-10-05T19:28:25.033 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.018+0000 7fde48b01700 1 -- 172.21.15.167:0/3087593634 <== mon.0 v2:172.21.15.118:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fde3c006400 con 0x55af23e967c0 2023-10-05T19:28:25.033 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.018+0000 7fde4dfd2e40 1 -- 172.21.15.167:0/3087593634 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x55af23f228d0 con 0x55af23e967c0 2023-10-05T19:28:25.033 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.018+0000 7fde48b01700 1 -- 172.21.15.167:0/3087593634 <== mon.0 v2:172.21.15.118:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fde3c014bd0 con 0x55af23e967c0 2023-10-05T19:28:25.033 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.018+0000 7fde4dfd2e40 1 -- 172.21.15.167:0/3087593634 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55af23f20c90 con 0x55af23e967c0 2023-10-05T19:28:25.038 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.022+0000 7fde48b01700 1 -- 172.21.15.167:0/3087593634 <== mon.0 v2:172.21.15.118:3300/0 4 ==== mgrmap(e 5) v1 ==== 40329+0+0 (secure 0 0 0) 0x7fde3c028460 con 0x55af23e967c0 2023-10-05T19:28:25.038 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.022+0000 7fde4dfd2e40 20 check_secure_mon_conn(): auth registy supported: methods=[2] modes=[2,1] 2023-10-05T19:28:25.038 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.022+0000 7fde4dfd2e40 20 check_secure_mon_conn(): mode 1 is insecure 2023-10-05T19:28:25.038 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.022+0000 7fde4dfd2e40 1 Processor -- start 2023-10-05T19:28:25.038 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.022+0000 7fde4dfd2e40 1 -- start start 2023-10-05T19:28:25.038 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.022+0000 7fde48b01700 1 --2- 172.21.15.167:0/3087593634 >> [v2:172.21.15.167:6824/29889,v1:172.21.15.167:6825/29889] conn(0x7fde2c030c40 0x7fde2c0330f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-05T19:28:25.039 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.022+0000 7fde48b01700 1 -- 172.21.15.167:0/3087593634 <== mon.0 v2:172.21.15.118:3300/0 5 ==== osd_map(30..30 src has 1..30) v4 ==== 6057+0+0 (secure 0 0 0) 0x7fde3c045130 con 0x55af23e967c0 2023-10-05T19:28:25.039 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.022+0000 7fde4c308700 1 --2- 172.21.15.167:0/3087593634 >> [v2:172.21.15.167:6824/29889,v1:172.21.15.167:6825/29889] conn(0x7fde2c030c40 0x7fde2c0330f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-05T19:28:25.039 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.022+0000 7fde4dfd2e40 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55af23f5c9c0 0x55af23f5cd90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-05T19:28:25.039 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.022+0000 7fde4dfd2e40 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x55af23f5d2d0 con 0x55af23f5c9c0 2023-10-05T19:28:25.039 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.022+0000 7fde4bb07700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55af23f5c9c0 0x55af23f5cd90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-05T19:28:25.039 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.022+0000 7fde4bb07700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55af23f5c9c0 0x55af23f5cd90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.118:3300/0 says I am v2:172.21.15.167:57048/0 (socket says 172.21.15.167:57048) 2023-10-05T19:28:25.039 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.022+0000 7fde4bb07700 1 -- 172.21.15.167:0/2345635336 learned_addr learned my addr 172.21.15.167:0/2345635336 (peer_addr_for_me v2:172.21.15.167:0/0) 2023-10-05T19:28:25.040 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.022+0000 7fde4c308700 1 --2- 172.21.15.167:0/3087593634 >> [v2:172.21.15.167:6824/29889,v1:172.21.15.167:6825/29889] conn(0x7fde2c030c40 0x7fde2c0330f0 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7fde4400a370 tx=0x7fde44005500 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-05T19:28:25.040 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.022+0000 7fde4bb07700 1 -- 172.21.15.167:0/2345635336 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55af23f5d410 con 0x55af23f5c9c0 2023-10-05T19:28:25.040 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.022+0000 7fde4bb07700 1 --2- 172.21.15.167:0/2345635336 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55af23f5c9c0 0x55af23f5cd90 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7fde3c043590 tx=0x7fde3c0434f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7d7df4da3ab65fa1 server_cookie=0 in_seq=0 out_seq=0 2023-10-05T19:28:25.040 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.022+0000 7fde1d7e2700 1 -- 172.21.15.167:0/2345635336 <== mon.0 v2:172.21.15.118:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fde3c0273c0 con 0x55af23f5c9c0 2023-10-05T19:28:25.040 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.022+0000 7fde1d7e2700 1 -- 172.21.15.167:0/2345635336 <== mon.0 v2:172.21.15.118:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fde3c0123c0 con 0x55af23f5c9c0 2023-10-05T19:28:25.040 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.022+0000 7fde1d7e2700 1 -- 172.21.15.167:0/2345635336 <== mon.0 v2:172.21.15.118:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fde3c01d350 con 0x55af23f5c9c0 2023-10-05T19:28:25.040 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.026+0000 7fde4dfd2e40 1 -- 172.21.15.167:0/2345635336 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55af23f5c9c0 msgr2=0x55af23f5cd90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-05T19:28:25.040 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.026+0000 7fde4dfd2e40 1 --2- 172.21.15.167:0/2345635336 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55af23f5c9c0 0x55af23f5cd90 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7fde3c043590 tx=0x7fde3c0434f0 comp rx=0 tx=0).stop 2023-10-05T19:28:25.041 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.026+0000 7fde4dfd2e40 1 -- 172.21.15.167:0/2345635336 shutdown_connections 2023-10-05T19:28:25.041 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.026+0000 7fde4dfd2e40 1 --2- 172.21.15.167:0/2345635336 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55af23f5c9c0 0x55af23f5cd90 unknown :-1 s=CLOSED pgs=168 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-05T19:28:25.041 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.026+0000 7fde4dfd2e40 1 -- 172.21.15.167:0/2345635336 >> 172.21.15.167:0/2345635336 conn(0x55af23f26320 msgr2=0x55af23f581c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-05T19:28:25.041 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.026+0000 7fde4dfd2e40 1 -- 172.21.15.167:0/2345635336 shutdown_connections 2023-10-05T19:28:25.041 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.026+0000 7fde4dfd2e40 1 -- 172.21.15.167:0/2345635336 wait complete. 2023-10-05T19:28:25.041 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.026+0000 7fde4dfd2e40 1 Processor -- start 2023-10-05T19:28:25.041 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.026+0000 7fde4dfd2e40 1 -- start start 2023-10-05T19:28:25.042 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.026+0000 7fde4dfd2e40 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55af23f5c9c0 0x55af23f62690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-05T19:28:25.042 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.026+0000 7fde4dfd2e40 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x55af23f63690 con 0x55af23f5c9c0 2023-10-05T19:28:25.042 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.026+0000 7fde4bb07700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55af23f5c9c0 0x55af23f62690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-05T19:28:25.042 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.026+0000 7fde4bb07700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55af23f5c9c0 0x55af23f62690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.118:3300/0 says I am v2:172.21.15.167:57058/0 (socket says 172.21.15.167:57058) 2023-10-05T19:28:25.042 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.026+0000 7fde4bb07700 1 -- 172.21.15.167:0/1070899594 learned_addr learned my addr 172.21.15.167:0/1070899594 (peer_addr_for_me v2:172.21.15.167:0/0) 2023-10-05T19:28:25.042 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.026+0000 7fde4bb07700 1 -- 172.21.15.167:0/1070899594 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55af23f20c90 con 0x55af23f5c9c0 2023-10-05T19:28:25.042 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.026+0000 7fde4bb07700 1 --2- 172.21.15.167:0/1070899594 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55af23f5c9c0 0x55af23f62690 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7fde3c042b30 tx=0x7fde3c039000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-05T19:28:25.042 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.026+0000 7fddd77fe700 1 -- 172.21.15.167:0/1070899594 <== mon.0 v2:172.21.15.118:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fde3c012aa0 con 0x55af23f5c9c0 2023-10-05T19:28:25.043 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.026+0000 7fddd77fe700 1 -- 172.21.15.167:0/1070899594 <== mon.0 v2:172.21.15.118:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fde3c012c00 con 0x55af23f5c9c0 2023-10-05T19:28:25.043 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.026+0000 7fde4dfd2e40 1 -- 172.21.15.167:0/1070899594 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x55af23f20c90 con 0x55af23f5c9c0 2023-10-05T19:28:25.043 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.026+0000 7fddd77fe700 1 -- 172.21.15.167:0/1070899594 <== mon.0 v2:172.21.15.118:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fde3c028710 con 0x55af23f5c9c0 2023-10-05T19:28:25.043 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.026+0000 7fde4dfd2e40 1 -- 172.21.15.167:0/1070899594 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55af23f62fa0 con 0x55af23f5c9c0 2023-10-05T19:28:25.043 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.030+0000 7fddd77fe700 1 -- 172.21.15.167:0/1070899594 <== mon.0 v2:172.21.15.118:3300/0 4 ==== mgrmap(e 5) v1 ==== 40329+0+0 (secure 0 0 0) 0x7fde3c028870 con 0x55af23f5c9c0 2023-10-05T19:28:25.043 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.030+0000 7fddd77fe700 1 --2- 172.21.15.167:0/1070899594 >> [v2:172.21.15.167:6824/29889,v1:172.21.15.167:6825/29889] conn(0x7fdd98030c00 0x7fdd980330b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-05T19:28:25.043 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.030+0000 7fddd77fe700 1 -- 172.21.15.167:0/1070899594 <== mon.0 v2:172.21.15.118:3300/0 5 ==== osd_map(30..30 src has 1..30) v4 ==== 6057+0+0 (secure 0 0 0) 0x7fde3c050c10 con 0x55af23f5c9c0 2023-10-05T19:28:25.044 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.030+0000 7fde43fff700 1 --2- 172.21.15.167:0/1070899594 >> [v2:172.21.15.167:6824/29889,v1:172.21.15.167:6825/29889] conn(0x7fdd98030c00 0x7fdd980330b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-05T19:28:25.044 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.030+0000 7fde43fff700 1 --2- 172.21.15.167:0/1070899594 >> [v2:172.21.15.167:6824/29889,v1:172.21.15.167:6825/29889] conn(0x7fdd98030c00 0x7fdd980330b0 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7fde340030c0 tx=0x7fde34014000 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-05T19:28:25.044 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.030+0000 7fddd57fa700 20 reqs_thread_entry: start 2023-10-05T19:28:25.044 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.030+0000 7fde4dfd2e40 20 try_to_resolve_local_entities(): before: realm_id= realm_name= zonegroup_id= zonegroup_name= zone_id= zone_name=default 2023-10-05T19:28:25.044 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.030+0000 7fde4dfd2e40 20 rados->read ofs=0 len=0 2023-10-05T19:28:25.044 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.030+0000 7fde4dfd2e40 1 --2- 172.21.15.167:0/3087593634 >> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] conn(0x55af23f7fec0 0x55af23f802d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-05T19:28:25.044 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.030+0000 7fde4dfd2e40 1 -- 172.21.15.167:0/3087593634 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:1 5.f 5:f43b6ece:::zone_names.default:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x55af23f80810 con 0x55af23f7fec0 2023-10-05T19:28:25.044 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.030+0000 7fde4c308700 1 --2- 172.21.15.167:0/3087593634 >> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] conn(0x55af23f7fec0 0x55af23f802d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-05T19:28:25.044 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.030+0000 7fde4c308700 1 --2- 172.21.15.167:0/3087593634 >> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] conn(0x55af23f7fec0 0x55af23f802d0 crc :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-05T19:28:25.047 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.034+0000 7fde4c308700 1 -- 172.21.15.167:0/3087593634 <== osd.1 v2:172.21.15.167:6800/30035 1 ==== osd_op_reply(1 zone_names.default [read 0~46 out=46b] v0'0 uv1 ondisk = 0) v8 ==== 162+0+46 (crc 0 0 0) 0x7fde44011040 con 0x55af23f7fec0 2023-10-05T19:28:25.047 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.034+0000 7fde4dfd2e40 20 rados_obj.operate() r=0 bl.length=46 2023-10-05T19:28:25.047 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.034+0000 7fde4dfd2e40 20 rados->read ofs=0 len=0 2023-10-05T19:28:25.047 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.034+0000 7fde4dfd2e40 1 -- 172.21.15.167:0/3087593634 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:2 5.1d 5:b8374eb4:::zone_info.9a9df965-6612-45d4-a45f-6e3c889bacc2:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x55af23f93380 con 0x55af23f7fec0 2023-10-05T19:28:25.047 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.034+0000 7fde4c308700 1 -- 172.21.15.167:0/3087593634 <== osd.1 v2:172.21.15.167:6800/30035 2 ==== osd_op_reply(2 zone_info.9a9df965-6612-45d4-a45f-6e3c889bacc2 [read 0~874 out=874b] v0'0 uv1 ondisk = 0) v8 ==== 190+0+874 (crc 0 0 0) 0x7fde44011040 con 0x55af23f7fec0 2023-10-05T19:28:25.047 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.034+0000 7fde4dfd2e40 20 rados_obj.operate() r=0 bl.length=874 2023-10-05T19:28:25.047 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.034+0000 7fde4dfd2e40 1 --2- 172.21.15.167:0/3087593634 >> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] conn(0x55af23f94c50 0x55af23f97030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-05T19:28:25.047 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.034+0000 7fde4dfd2e40 1 -- 172.21.15.167:0/3087593634 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:3 5.0 5:00000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x55af23f97570 con 0x55af23f94c50 2023-10-05T19:28:25.048 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.034+0000 7fde4bb07700 1 --2- 172.21.15.167:0/3087593634 >> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] conn(0x55af23f94c50 0x55af23f97030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-05T19:28:25.048 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.034+0000 7fde4bb07700 1 --2- 172.21.15.167:0/3087593634 >> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] conn(0x55af23f94c50 0x55af23f97030 crc :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-05T19:28:25.048 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.038+0000 7fde4bb07700 1 -- 172.21.15.167:0/3087593634 <== osd.3 v2:172.21.15.167:6808/30036 1 ==== osd_op_reply(3 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fde3c024020 con 0x55af23f94c50 2023-10-05T19:28:25.048 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.038+0000 7fde4bb07700 1 --2- 172.21.15.167:0/3087593634 >> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] conn(0x7fde3c059040 0x7fde3c02c160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-05T19:28:25.048 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.038+0000 7fde4bb07700 1 -- 172.21.15.167:0/3087593634 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:4 5.10 5:08000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x55af23f97570 con 0x7fde3c059040 2023-10-05T19:28:25.048 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.038+0000 7fde43fff700 1 --2- 172.21.15.167:0/3087593634 >> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] conn(0x7fde3c059040 0x7fde3c02c160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-05T19:28:25.048 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.038+0000 7fde43fff700 1 --2- 172.21.15.167:0/3087593634 >> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] conn(0x7fde3c059040 0x7fde3c02c160 crc :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-05T19:28:25.048 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.038+0000 7fde43fff700 1 -- 172.21.15.167:0/3087593634 <== osd.2 v2:172.21.15.167:6816/30037 1 ==== osd_op_reply(4 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fde34007d60 con 0x7fde3c059040 2023-10-05T19:28:25.048 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.038+0000 7fde43fff700 1 --2- 172.21.15.167:0/3087593634 >> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] conn(0x7fde34007030 0x7fde3401d430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-05T19:28:25.048 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.038+0000 7fde43fff700 1 -- 172.21.15.167:0/3087593634 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:5 5.8 5:10000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x55af23f97570 con 0x7fde34007030 2023-10-05T19:28:25.048 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.038+0000 7fde4c308700 1 --2- 172.21.15.167:0/3087593634 >> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] conn(0x7fde34007030 0x7fde3401d430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-05T19:28:25.049 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.038+0000 7fde4c308700 1 --2- 172.21.15.167:0/3087593634 >> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] conn(0x7fde34007030 0x7fde3401d430 crc :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-05T19:28:25.049 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.038+0000 7fde4c308700 1 -- 172.21.15.167:0/3087593634 <== osd.0 v2:172.21.15.118:6800/27690 1 ==== osd_op_reply(5 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fde44011040 con 0x7fde34007030 2023-10-05T19:28:25.049 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.038+0000 7fde4c308700 1 -- 172.21.15.167:0/3087593634 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:6 5.18 5:18000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x55af23f97570 con 0x55af23f7fec0 2023-10-05T19:28:25.049 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.042+0000 7fde4c308700 1 -- 172.21.15.167:0/3087593634 <== osd.1 v2:172.21.15.167:6800/30035 3 ==== osd_op_reply(6 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fde44011040 con 0x55af23f7fec0 2023-10-05T19:28:25.049 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.042+0000 7fde4c308700 1 -- 172.21.15.167:0/3087593634 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:7 5.4 5:20000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x55af23f97570 con 0x55af23f7fec0 2023-10-05T19:28:25.050 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.042+0000 7fde4c308700 1 -- 172.21.15.167:0/3087593634 <== osd.1 v2:172.21.15.167:6800/30035 4 ==== osd_op_reply(7 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fde44011040 con 0x55af23f7fec0 2023-10-05T19:28:25.050 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.042+0000 7fde4c308700 1 -- 172.21.15.167:0/3087593634 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:8 5.14 5:28000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x55af23f97570 con 0x7fde34007030 2023-10-05T19:28:25.050 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.042+0000 7fde4c308700 1 -- 172.21.15.167:0/3087593634 <== osd.0 v2:172.21.15.118:6800/27690 2 ==== osd_op_reply(8 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fde44011040 con 0x7fde34007030 2023-10-05T19:28:25.050 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.042+0000 7fde4c308700 1 -- 172.21.15.167:0/3087593634 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:9 5.c 5:30000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x55af23f97570 con 0x7fde34007030 2023-10-05T19:28:25.051 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.042+0000 7fde4c308700 1 -- 172.21.15.167:0/3087593634 <== osd.0 v2:172.21.15.118:6800/27690 3 ==== osd_op_reply(9 [pgnls start_epoch 30 out=112b] v23'1 uv1 ondisk = 1) v8 ==== 144+0+112 (crc 0 0 0) 0x7fde44011040 con 0x7fde34007030 2023-10-05T19:28:25.051 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.042+0000 7fde4c308700 1 -- 172.21.15.167:0/3087593634 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:10 5.1c 5:38000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x55af23f97570 con 0x7fde3c059040 2023-10-05T19:28:25.051 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.042+0000 7fde43fff700 1 -- 172.21.15.167:0/3087593634 <== osd.2 v2:172.21.15.167:6816/30037 2 ==== osd_op_reply(10 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fde34029040 con 0x7fde3c059040 2023-10-05T19:28:25.051 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.042+0000 7fde43fff700 1 -- 172.21.15.167:0/3087593634 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:11 5.2 5:40000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x55af23f97570 con 0x55af23f94c50 2023-10-05T19:28:25.051 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.046+0000 7fde4bb07700 1 -- 172.21.15.167:0/3087593634 <== osd.3 v2:172.21.15.167:6808/30036 2 ==== osd_op_reply(11 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fde3c024020 con 0x55af23f94c50 2023-10-05T19:28:25.051 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.046+0000 7fde4bb07700 1 -- 172.21.15.167:0/3087593634 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:12 5.12 5:48000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x55af23f97570 con 0x55af23f7fec0 2023-10-05T19:28:25.052 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.046+0000 7fde4c308700 1 -- 172.21.15.167:0/3087593634 <== osd.1 v2:172.21.15.167:6800/30035 5 ==== osd_op_reply(12 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fde44011040 con 0x55af23f7fec0 2023-10-05T19:28:25.052 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.046+0000 7fde4c308700 1 -- 172.21.15.167:0/3087593634 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:13 5.a 5:50000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x55af23f97570 con 0x7fde34007030 2023-10-05T19:28:25.052 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.046+0000 7fde4c308700 1 -- 172.21.15.167:0/3087593634 <== osd.0 v2:172.21.15.118:6800/27690 4 ==== osd_op_reply(13 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fde44011040 con 0x7fde34007030 2023-10-05T19:28:25.052 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.046+0000 7fde4c308700 1 -- 172.21.15.167:0/3087593634 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:14 5.1a 5:58000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x55af23f97570 con 0x55af23f7fec0 2023-10-05T19:28:25.053 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.046+0000 7fde4c308700 1 -- 172.21.15.167:0/3087593634 <== osd.1 v2:172.21.15.167:6800/30035 6 ==== osd_op_reply(14 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fde44011040 con 0x55af23f7fec0 2023-10-05T19:28:25.053 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.046+0000 7fde4c308700 1 -- 172.21.15.167:0/3087593634 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:15 5.6 5:60000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x55af23f97570 con 0x7fde3c059040 2023-10-05T19:28:25.054 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.046+0000 7fde43fff700 1 -- 172.21.15.167:0/3087593634 <== osd.2 v2:172.21.15.167:6816/30037 3 ==== osd_op_reply(15 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fde34029040 con 0x7fde3c059040 2023-10-05T19:28:25.054 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.046+0000 7fde43fff700 1 -- 172.21.15.167:0/3087593634 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:16 5.16 5:68000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x55af23f97570 con 0x55af23f94c50 2023-10-05T19:28:25.055 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.050+0000 7fde4bb07700 1 -- 172.21.15.167:0/3087593634 <== osd.3 v2:172.21.15.167:6808/30036 3 ==== osd_op_reply(16 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fde3c024020 con 0x55af23f94c50 2023-10-05T19:28:25.055 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.050+0000 7fde4bb07700 1 -- 172.21.15.167:0/3087593634 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:17 5.e 5:70000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x55af23f97570 con 0x55af23f94c50 2023-10-05T19:28:25.055 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.050+0000 7fde4bb07700 1 -- 172.21.15.167:0/3087593634 <== osd.3 v2:172.21.15.167:6808/30036 4 ==== osd_op_reply(17 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fde3c024020 con 0x55af23f94c50 2023-10-05T19:28:25.055 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.050+0000 7fde4bb07700 1 -- 172.21.15.167:0/3087593634 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:18 5.1e 5:78000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x55af23f97570 con 0x7fde34007030 2023-10-05T19:28:25.056 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.050+0000 7fde4c308700 1 -- 172.21.15.167:0/3087593634 <== osd.0 v2:172.21.15.118:6800/27690 5 ==== osd_op_reply(18 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fde44011040 con 0x7fde34007030 2023-10-05T19:28:25.056 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.050+0000 7fde4c308700 1 -- 172.21.15.167:0/3087593634 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:19 5.1 5:80000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x55af23f97570 con 0x55af23f94c50 2023-10-05T19:28:25.057 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.050+0000 7fde4bb07700 1 -- 172.21.15.167:0/3087593634 <== osd.3 v2:172.21.15.167:6808/30036 5 ==== osd_op_reply(19 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fde3c024020 con 0x55af23f94c50 2023-10-05T19:28:25.057 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.050+0000 7fde4bb07700 1 -- 172.21.15.167:0/3087593634 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:20 5.11 5:88000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x55af23f97570 con 0x55af23f94c50 2023-10-05T19:28:25.057 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.050+0000 7fde4bb07700 1 -- 172.21.15.167:0/3087593634 <== osd.3 v2:172.21.15.167:6808/30036 6 ==== osd_op_reply(20 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fde3c024020 con 0x55af23f94c50 2023-10-05T19:28:25.058 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.050+0000 7fde4bb07700 1 -- 172.21.15.167:0/3087593634 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:21 5.9 5:90000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x55af23f97570 con 0x55af23f7fec0 2023-10-05T19:28:25.058 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.054+0000 7fde4c308700 1 -- 172.21.15.167:0/3087593634 <== osd.1 v2:172.21.15.167:6800/30035 7 ==== osd_op_reply(21 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fde44011040 con 0x55af23f7fec0 2023-10-05T19:28:25.059 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.054+0000 7fde4c308700 1 -- 172.21.15.167:0/3087593634 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:22 5.19 5:98000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x55af23f97570 con 0x55af23f7fec0 2023-10-05T19:28:25.059 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.054+0000 7fde4c308700 1 -- 172.21.15.167:0/3087593634 <== osd.1 v2:172.21.15.167:6800/30035 8 ==== osd_op_reply(22 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fde44011040 con 0x55af23f7fec0 2023-10-05T19:28:25.059 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.054+0000 7fde4c308700 1 -- 172.21.15.167:0/3087593634 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:23 5.5 5:a0000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x55af23f97570 con 0x7fde34007030 2023-10-05T19:28:25.060 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.054+0000 7fde4c308700 1 -- 172.21.15.167:0/3087593634 <== osd.0 v2:172.21.15.118:6800/27690 6 ==== osd_op_reply(23 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fde44011040 con 0x7fde34007030 2023-10-05T19:28:25.060 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.054+0000 7fde4c308700 1 -- 172.21.15.167:0/3087593634 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:24 5.15 5:a8000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x55af23f97570 con 0x55af23f94c50 2023-10-05T19:28:25.061 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.054+0000 7fde4bb07700 1 -- 172.21.15.167:0/3087593634 <== osd.3 v2:172.21.15.167:6808/30036 7 ==== osd_op_reply(24 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fde3c024020 con 0x55af23f94c50 2023-10-05T19:28:25.061 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.054+0000 7fde4bb07700 1 -- 172.21.15.167:0/3087593634 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:25 5.d 5:b0000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x55af23f97570 con 0x7fde3c059040 2023-10-05T19:28:25.062 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.054+0000 7fde43fff700 1 -- 172.21.15.167:0/3087593634 <== osd.2 v2:172.21.15.167:6816/30037 4 ==== osd_op_reply(25 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fde34029040 con 0x7fde3c059040 2023-10-05T19:28:25.062 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.058+0000 7fde43fff700 1 -- 172.21.15.167:0/3087593634 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:26 5.1d 5:b8000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x55af23f97570 con 0x55af23f7fec0 2023-10-05T19:28:25.063 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.058+0000 7fde4c308700 1 -- 172.21.15.167:0/3087593634 <== osd.1 v2:172.21.15.167:6800/30035 9 ==== osd_op_reply(26 [pgnls start_epoch 30 out=107b] v23'1 uv1 ondisk = 1) v8 ==== 144+0+107 (crc 0 0 0) 0x7fde44011040 con 0x55af23f7fec0 2023-10-05T19:28:25.063 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.058+0000 7fde4c308700 1 -- 172.21.15.167:0/3087593634 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:27 5.3 5:c0000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x55af23f97570 con 0x7fde34007030 2023-10-05T19:28:25.064 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.058+0000 7fde4c308700 1 -- 172.21.15.167:0/3087593634 <== osd.0 v2:172.21.15.118:6800/27690 7 ==== osd_op_reply(27 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fde44011040 con 0x7fde34007030 2023-10-05T19:28:25.064 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.058+0000 7fde4c308700 1 -- 172.21.15.167:0/3087593634 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:28 5.13 5:c8000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x55af23f97570 con 0x55af23f94c50 2023-10-05T19:28:25.064 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.058+0000 7fde4bb07700 1 -- 172.21.15.167:0/3087593634 <== osd.3 v2:172.21.15.167:6808/30036 8 ==== osd_op_reply(28 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fde3c024020 con 0x55af23f94c50 2023-10-05T19:28:25.065 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.058+0000 7fde4bb07700 1 -- 172.21.15.167:0/3087593634 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:29 5.b 5:d0000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x55af23f97570 con 0x7fde34007030 2023-10-05T19:28:25.065 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.058+0000 7fde4c308700 1 -- 172.21.15.167:0/3087593634 <== osd.0 v2:172.21.15.118:6800/27690 8 ==== osd_op_reply(29 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fde44011040 con 0x7fde34007030 2023-10-05T19:28:25.065 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.058+0000 7fde4c308700 1 -- 172.21.15.167:0/3087593634 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:30 5.1b 5:d8000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x55af23f97570 con 0x7fde3c059040 2023-10-05T19:28:25.066 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.062+0000 7fde43fff700 1 -- 172.21.15.167:0/3087593634 <== osd.2 v2:172.21.15.167:6816/30037 5 ==== osd_op_reply(30 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fde34029040 con 0x7fde3c059040 2023-10-05T19:28:25.066 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.062+0000 7fde43fff700 1 -- 172.21.15.167:0/3087593634 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:31 5.7 5:e0000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x55af23f97570 con 0x55af23f94c50 2023-10-05T19:28:25.067 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.062+0000 7fde4bb07700 1 -- 172.21.15.167:0/3087593634 <== osd.3 v2:172.21.15.167:6808/30036 9 ==== osd_op_reply(31 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fde3c024020 con 0x55af23f94c50 2023-10-05T19:28:25.067 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.062+0000 7fde4bb07700 1 -- 172.21.15.167:0/3087593634 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:32 5.17 5:e8000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x55af23f97570 con 0x7fde34007030 2023-10-05T19:28:25.068 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.062+0000 7fde4c308700 1 -- 172.21.15.167:0/3087593634 <== osd.0 v2:172.21.15.118:6800/27690 9 ==== osd_op_reply(32 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fde44011040 con 0x7fde34007030 2023-10-05T19:28:25.068 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.062+0000 7fde4c308700 1 -- 172.21.15.167:0/3087593634 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:33 5.f 5:f0000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x55af23f97570 con 0x55af23f7fec0 2023-10-05T19:28:25.069 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.062+0000 7fde4c308700 1 -- 172.21.15.167:0/3087593634 <== osd.1 v2:172.21.15.167:6800/30035 10 ==== osd_op_reply(33 [pgnls start_epoch 30 out=115b] v23'2 uv2 ondisk = 1) v8 ==== 144+0+115 (crc 0 0 0) 0x7fde44011040 con 0x55af23f7fec0 2023-10-05T19:28:25.069 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.062+0000 7fde4c308700 1 -- 172.21.15.167:0/3087593634 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:34 5.1f 5:f8000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x55af23f97570 con 0x7fde3c059040 2023-10-05T19:28:25.070 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.062+0000 7fde43fff700 1 -- 172.21.15.167:0/3087593634 <== osd.2 v2:172.21.15.167:6816/30037 6 ==== osd_op_reply(34 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fde34029040 con 0x7fde3c059040 2023-10-05T19:28:25.070 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.062+0000 7fde4dfd2e40 20 RGWRados::pool_iterate: got zonegroup_info.4f1f719d-9648-417b-b10c-e5d939825419 2023-10-05T19:28:25.070 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.062+0000 7fde4dfd2e40 20 RGWRados::pool_iterate: got zone_info.9a9df965-6612-45d4-a45f-6e3c889bacc2 2023-10-05T19:28:25.070 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.066+0000 7fde4dfd2e40 20 RGWRados::pool_iterate: got zone_names.default 2023-10-05T19:28:25.070 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.066+0000 7fde4dfd2e40 20 RGWRados::pool_iterate: got zonegroups_names.default 2023-10-05T19:28:25.070 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.066+0000 7fde4dfd2e40 20 rados->read ofs=0 len=0 2023-10-05T19:28:25.071 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.066+0000 7fde4dfd2e40 1 -- 172.21.15.167:0/3087593634 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:35 5.1d 5:b8374eb4:::zone_info.9a9df965-6612-45d4-a45f-6e3c889bacc2:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x55af23f93730 con 0x55af23f7fec0 2023-10-05T19:28:25.071 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.066+0000 7fde4c308700 1 -- 172.21.15.167:0/3087593634 <== osd.1 v2:172.21.15.167:6800/30035 11 ==== osd_op_reply(35 zone_info.9a9df965-6612-45d4-a45f-6e3c889bacc2 [read 0~874 out=874b] v0'0 uv1 ondisk = 0) v8 ==== 190+0+874 (crc 0 0 0) 0x7fde44011040 con 0x55af23f7fec0 2023-10-05T19:28:25.071 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.066+0000 7fde4dfd2e40 20 rados_obj.operate() r=0 bl.length=874 2023-10-05T19:28:25.071 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.066+0000 7fde4dfd2e40 1 -- 172.21.15.167:0/3087593634 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:36 5.1d 5:b8374eb4:::zone_info.9a9df965-6612-45d4-a45f-6e3c889bacc2:head [delete,create,writefull 0~874 in=874b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x55af23f93b70 con 0x55af23f7fec0 2023-10-05T19:28:25.074 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.070+0000 7fde4c308700 1 -- 172.21.15.167:0/3087593634 <== osd.1 v2:172.21.15.167:6800/30035 12 ==== osd_op_reply(36 zone_info.9a9df965-6612-45d4-a45f-6e3c889bacc2 [delete,create,writefull 0~874] v30'2 uv2 ondisk = 0) v8 ==== 274+0+0 (crc 0 0 0) 0x7fde44011040 con 0x55af23f7fec0 2023-10-05T19:28:25.077 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.070+0000 7fde4dfd2e40 1 -- 172.21.15.167:0/3087593634 >> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] conn(0x7fde34007030 msgr2=0x7fde3401d430 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-05T19:28:25.077 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.070+0000 7fde4dfd2e40 1 --2- 172.21.15.167:0/3087593634 >> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] conn(0x7fde34007030 0x7fde3401d430 crc :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-05T19:28:25.077 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.070+0000 7fde4dfd2e40 1 -- 172.21.15.167:0/3087593634 >> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] conn(0x55af23f7fec0 msgr2=0x55af23f802d0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-05T19:28:25.077 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.070+0000 7fde4dfd2e40 1 --2- 172.21.15.167:0/3087593634 >> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] conn(0x55af23f7fec0 0x55af23f802d0 crc :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-05T19:28:25.077 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.070+0000 7fde4dfd2e40 1 -- 172.21.15.167:0/3087593634 >> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] conn(0x7fde3c059040 msgr2=0x7fde3c02c160 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-05T19:28:25.077 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.070+0000 7fde4dfd2e40 1 --2- 172.21.15.167:0/3087593634 >> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] conn(0x7fde3c059040 0x7fde3c02c160 crc :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-05T19:28:25.077 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.070+0000 7fde4dfd2e40 1 -- 172.21.15.167:0/3087593634 >> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] conn(0x55af23f94c50 msgr2=0x55af23f97030 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-05T19:28:25.077 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.070+0000 7fde4dfd2e40 1 --2- 172.21.15.167:0/3087593634 >> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] conn(0x55af23f94c50 0x55af23f97030 crc :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-05T19:28:25.077 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.070+0000 7fde4dfd2e40 1 -- 172.21.15.167:0/3087593634 >> [v2:172.21.15.167:6824/29889,v1:172.21.15.167:6825/29889] conn(0x7fde2c030c40 msgr2=0x7fde2c0330f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-05T19:28:25.078 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.070+0000 7fde4dfd2e40 1 --2- 172.21.15.167:0/3087593634 >> [v2:172.21.15.167:6824/29889,v1:172.21.15.167:6825/29889] conn(0x7fde2c030c40 0x7fde2c0330f0 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7fde4400a370 tx=0x7fde44005500 comp rx=0 tx=0).stop 2023-10-05T19:28:25.078 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.070+0000 7fde4dfd2e40 1 -- 172.21.15.167:0/3087593634 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55af23e967c0 msgr2=0x55af23f22390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-05T19:28:25.078 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.070+0000 7fde4c308700 1 -- 172.21.15.167:0/3087593634 reap_dead start 2023-10-05T19:28:25.078 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.070+0000 7fde4dfd2e40 1 --2- 172.21.15.167:0/3087593634 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55af23e967c0 0x55af23f22390 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7fde3c001530 tx=0x7fde3c010000 comp rx=0 tx=0).stop 2023-10-05T19:28:25.078 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.070+0000 7fde4dfd2e40 1 -- 172.21.15.167:0/3087593634 shutdown_connections 2023-10-05T19:28:25.078 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.070+0000 7fde4dfd2e40 1 --2- 172.21.15.167:0/3087593634 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55af23e967c0 0x55af23f22390 unknown :-1 s=CLOSED pgs=167 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-05T19:28:25.078 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.070+0000 7fde4dfd2e40 1 -- 172.21.15.167:0/3087593634 >> 172.21.15.167:0/3087593634 conn(0x55af23f1c750 msgr2=0x55af23f3b390 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-05T19:28:25.078 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.070+0000 7fde4dfd2e40 1 -- 172.21.15.167:0/3087593634 shutdown_connections 2023-10-05T19:28:25.078 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.070+0000 7fde4dfd2e40 1 -- 172.21.15.167:0/3087593634 wait complete. 2023-10-05T19:28:25.082 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.070+0000 7fde4dfd2e40 1 -- 172.21.15.167:0/1070899594 >> [v2:172.21.15.167:6824/29889,v1:172.21.15.167:6825/29889] conn(0x7fdd98030c00 msgr2=0x7fdd980330b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-05T19:28:25.082 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.070+0000 7fde4dfd2e40 1 --2- 172.21.15.167:0/1070899594 >> [v2:172.21.15.167:6824/29889,v1:172.21.15.167:6825/29889] conn(0x7fdd98030c00 0x7fdd980330b0 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7fde340030c0 tx=0x7fde34014000 comp rx=0 tx=0).stop 2023-10-05T19:28:25.082 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.070+0000 7fde4dfd2e40 1 -- 172.21.15.167:0/1070899594 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55af23f5c9c0 msgr2=0x55af23f62690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-05T19:28:25.082 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.070+0000 7fde4dfd2e40 1 --2- 172.21.15.167:0/1070899594 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55af23f5c9c0 0x55af23f62690 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7fde3c042b30 tx=0x7fde3c039000 comp rx=0 tx=0).stop 2023-10-05T19:28:25.082 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.070+0000 7fde4dfd2e40 1 -- 172.21.15.167:0/1070899594 shutdown_connections 2023-10-05T19:28:25.083 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.070+0000 7fde4dfd2e40 1 --2- 172.21.15.167:0/1070899594 >> [v2:172.21.15.167:6824/29889,v1:172.21.15.167:6825/29889] conn(0x7fdd98030c00 0x7fdd980330b0 unknown :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-05T19:28:25.083 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.070+0000 7fde4dfd2e40 1 --2- 172.21.15.167:0/1070899594 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55af23f5c9c0 0x55af23f62690 unknown :-1 s=CLOSED pgs=169 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-05T19:28:25.083 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.070+0000 7fde4dfd2e40 1 -- 172.21.15.167:0/1070899594 >> 172.21.15.167:0/1070899594 conn(0x55af23f26320 msgr2=0x55af23f57e00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-05T19:28:25.083 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.070+0000 7fde4dfd2e40 1 -- 172.21.15.167:0/1070899594 shutdown_connections 2023-10-05T19:28:25.083 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.070+0000 7fde4dfd2e40 1 -- 172.21.15.167:0/1070899594 wait complete. 2023-10-05T19:28:25.084 INFO:teuthology.orchestra.run.smithi167.stdout:{ 2023-10-05T19:28:25.084 INFO:teuthology.orchestra.run.smithi167.stdout: "id": "9a9df965-6612-45d4-a45f-6e3c889bacc2", 2023-10-05T19:28:25.084 INFO:teuthology.orchestra.run.smithi167.stdout: "name": "default", 2023-10-05T19:28:25.084 INFO:teuthology.orchestra.run.smithi167.stdout: "domain_root": "default.rgw.meta:root", 2023-10-05T19:28:25.085 INFO:teuthology.orchestra.run.smithi167.stdout: "control_pool": "default.rgw.control", 2023-10-05T19:28:25.085 INFO:teuthology.orchestra.run.smithi167.stdout: "gc_pool": "default.rgw.log:gc", 2023-10-05T19:28:25.085 INFO:teuthology.orchestra.run.smithi167.stdout: "lc_pool": "default.rgw.log:lc", 2023-10-05T19:28:25.085 INFO:teuthology.orchestra.run.smithi167.stdout: "log_pool": "default.rgw.log", 2023-10-05T19:28:25.085 INFO:teuthology.orchestra.run.smithi167.stdout: "intent_log_pool": "default.rgw.log:intent", 2023-10-05T19:28:25.085 INFO:teuthology.orchestra.run.smithi167.stdout: "usage_log_pool": "default.rgw.log:usage", 2023-10-05T19:28:25.085 INFO:teuthology.orchestra.run.smithi167.stdout: "roles_pool": "default.rgw.meta:roles", 2023-10-05T19:28:25.085 INFO:teuthology.orchestra.run.smithi167.stdout: "reshard_pool": "default.rgw.log:reshard", 2023-10-05T19:28:25.085 INFO:teuthology.orchestra.run.smithi167.stdout: "user_keys_pool": "default.rgw.meta:users.keys", 2023-10-05T19:28:25.085 INFO:teuthology.orchestra.run.smithi167.stdout: "user_email_pool": "default.rgw.meta:users.email", 2023-10-05T19:28:25.086 INFO:teuthology.orchestra.run.smithi167.stdout: "user_swift_pool": "default.rgw.meta:users.swift", 2023-10-05T19:28:25.086 INFO:teuthology.orchestra.run.smithi167.stdout: "user_uid_pool": "default.rgw.meta:users.uid", 2023-10-05T19:28:25.086 INFO:teuthology.orchestra.run.smithi167.stdout: "otp_pool": "default.rgw.otp", 2023-10-05T19:28:25.086 INFO:teuthology.orchestra.run.smithi167.stdout: "system_key": { 2023-10-05T19:28:25.086 INFO:teuthology.orchestra.run.smithi167.stdout: "access_key": "", 2023-10-05T19:28:25.086 INFO:teuthology.orchestra.run.smithi167.stdout: "secret_key": "" 2023-10-05T19:28:25.086 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2023-10-05T19:28:25.086 INFO:teuthology.orchestra.run.smithi167.stdout: "placement_pools": [ 2023-10-05T19:28:25.086 INFO:teuthology.orchestra.run.smithi167.stdout: { 2023-10-05T19:28:25.086 INFO:teuthology.orchestra.run.smithi167.stdout: "key": "default-placement", 2023-10-05T19:28:25.086 INFO:teuthology.orchestra.run.smithi167.stdout: "val": { 2023-10-05T19:28:25.087 INFO:teuthology.orchestra.run.smithi167.stdout: "index_pool": "default.rgw.buckets.index", 2023-10-05T19:28:25.087 INFO:teuthology.orchestra.run.smithi167.stdout: "storage_classes": { 2023-10-05T19:28:25.087 INFO:teuthology.orchestra.run.smithi167.stdout: "STANDARD": { 2023-10-05T19:28:25.087 INFO:teuthology.orchestra.run.smithi167.stdout: "data_pool": "default.rgw.buckets.data" 2023-10-05T19:28:25.087 INFO:teuthology.orchestra.run.smithi167.stdout: } 2023-10-05T19:28:25.087 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2023-10-05T19:28:25.087 INFO:teuthology.orchestra.run.smithi167.stdout: "data_extra_pool": "default.rgw.buckets.non-ec", 2023-10-05T19:28:25.087 INFO:teuthology.orchestra.run.smithi167.stdout: "index_type": 0, 2023-10-05T19:28:25.087 INFO:teuthology.orchestra.run.smithi167.stdout: "inline_data": "true" 2023-10-05T19:28:25.087 INFO:teuthology.orchestra.run.smithi167.stdout: } 2023-10-05T19:28:25.087 INFO:teuthology.orchestra.run.smithi167.stdout: } 2023-10-05T19:28:25.088 INFO:teuthology.orchestra.run.smithi167.stdout: ], 2023-10-05T19:28:25.088 INFO:teuthology.orchestra.run.smithi167.stdout: "realm_id": "", 2023-10-05T19:28:25.088 INFO:teuthology.orchestra.run.smithi167.stdout: "notif_pool": "default.rgw.log:notif" 2023-10-05T19:28:25.088 INFO:teuthology.orchestra.run.smithi167.stdout:} 2023-10-05T19:28:25.088 DEBUG:tasks.util.rgw: json result: {'id': '9a9df965-6612-45d4-a45f-6e3c889bacc2', 'name': 'default', 'domain_root': 'default.rgw.meta:root', 'control_pool': 'default.rgw.control', 'gc_pool': 'default.rgw.log:gc', 'lc_pool': 'default.rgw.log:lc', 'log_pool': 'default.rgw.log', 'intent_log_pool': 'default.rgw.log:intent', 'usage_log_pool': 'default.rgw.log:usage', 'roles_pool': 'default.rgw.meta:roles', 'reshard_pool': 'default.rgw.log:reshard', 'user_keys_pool': 'default.rgw.meta:users.keys', 'user_email_pool': 'default.rgw.meta:users.email', 'user_swift_pool': 'default.rgw.meta:users.swift', 'user_uid_pool': 'default.rgw.meta:users.uid', 'otp_pool': 'default.rgw.otp', 'system_key': {'access_key': '', 'secret_key': ''}, 'placement_pools': [{'key': 'default-placement', 'val': {'index_pool': 'default.rgw.buckets.index', 'storage_classes': {'STANDARD': {'data_pool': 'default.rgw.buckets.data'}}, 'data_extra_pool': 'default.rgw.buckets.non-ec', 'index_type': 0, 'inline_data': 'true'}}], 'realm_id': '', 'notif_pool': 'default.rgw.log:notif'} 2023-10-05T19:28:25.089 INFO:tasks.util.rgw:rgwadmin: client.0 : ['usage', 'show'] 2023-10-05T19:28:25.090 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'ceph', 'usage', 'show'] 2023-10-05T19:28:25.090 DEBUG:teuthology.orchestra.run.smithi167:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster ceph usage show 2023-10-05T19:28:25.128 INFO:teuthology.orchestra.run.smithi167.stderr:ignoring --setuser ceph since I am not root 2023-10-05T19:28:25.128 INFO:teuthology.orchestra.run.smithi167.stderr:ignoring --setgroup ceph since I am not root 2023-10-05T19:28:25.129 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.122+0000 7f541092fe40 1 Processor -- start 2023-10-05T19:28:25.129 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.122+0000 7f541092fe40 1 -- start start 2023-10-05T19:28:25.129 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.122+0000 7f541092fe40 1 --2- >> v2:172.21.15.118:3300/0 conn(0x55d84228abf0 0x55d8422891d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-05T19:28:25.129 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.122+0000 7f541092fe40 1 -- --> v1:172.21.15.118:6789/0 -- auth(proto 0 26 bytes epoch 0) v1 -- 0x55d842261c00 con 0x55d842289710 2023-10-05T19:28:25.129 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.122+0000 7f541092fe40 1 -- --> v2:172.21.15.118:3300/0 -- mon_getmap magic: 0 v1 -- 0x55d84221eee0 con 0x55d84228abf0 2023-10-05T19:28:25.130 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.122+0000 7f540e464700 1 --2- >> v2:172.21.15.118:3300/0 conn(0x55d84228abf0 0x55d8422891d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-05T19:28:25.130 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.126+0000 7f540e464700 1 --2- >> v2:172.21.15.118:3300/0 conn(0x55d84228abf0 0x55d8422891d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.118:3300/0 says I am v2:172.21.15.167:57060/0 (socket says 172.21.15.167:57060) 2023-10-05T19:28:25.130 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.126+0000 7f540dc63700 1 --1- >> v1:172.21.15.118:6789/0 conn(0x55d842289710 0x55d84231cf20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.118:6789/0 says I am v1:172.21.15.167:56338/0 (socket says 172.21.15.167:56338) 2023-10-05T19:28:25.130 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.126+0000 7f540e464700 1 -- 172.21.15.167:0/3774656944 learned_addr learned my addr 172.21.15.167:0/3774656944 (peer_addr_for_me v2:172.21.15.167:0/0) 2023-10-05T19:28:25.131 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.126+0000 7f540e464700 1 -- 172.21.15.167:0/3774656944 >> v1:172.21.15.118:6789/0 conn(0x55d842289710 legacy=0x55d84231cf20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-05T19:28:25.131 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.126+0000 7f540e464700 1 -- 172.21.15.167:0/3774656944 --> v2:172.21.15.118:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55d84228b080 con 0x55d84228abf0 2023-10-05T19:28:25.131 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.126+0000 7f540e464700 1 --2- 172.21.15.167:0/3774656944 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55d84228abf0 0x55d8422891d0 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7f540800a650 tx=0x7f5408005b10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f18fc701666914da server_cookie=0 in_seq=0 out_seq=0 2023-10-05T19:28:25.132 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.126+0000 7f540d462700 1 -- 172.21.15.167:0/3774656944 <== mon.0 v2:172.21.15.118:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f5408008760 con 0x55d84228abf0 2023-10-05T19:28:25.132 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.126+0000 7f540d462700 1 -- 172.21.15.167:0/3774656944 <== mon.0 v2:172.21.15.118:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f54080088c0 con 0x55d84228abf0 2023-10-05T19:28:25.132 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.126+0000 7f540d462700 1 -- 172.21.15.167:0/3774656944 <== mon.0 v2:172.21.15.118:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f5408008bb0 con 0x55d84228abf0 2023-10-05T19:28:25.133 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.126+0000 7f541092fe40 1 -- 172.21.15.167:0/3774656944 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55d84228abf0 msgr2=0x55d8422891d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-05T19:28:25.133 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.126+0000 7f541092fe40 1 --2- 172.21.15.167:0/3774656944 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55d84228abf0 0x55d8422891d0 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7f540800a650 tx=0x7f5408005b10 comp rx=0 tx=0).stop 2023-10-05T19:28:25.133 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.126+0000 7f541092fe40 1 -- 172.21.15.167:0/3774656944 shutdown_connections 2023-10-05T19:28:25.133 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.126+0000 7f541092fe40 1 --2- 172.21.15.167:0/3774656944 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55d84228abf0 0x55d8422891d0 unknown :-1 s=CLOSED pgs=171 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-05T19:28:25.133 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.126+0000 7f541092fe40 1 -- 172.21.15.167:0/3774656944 >> 172.21.15.167:0/3774656944 conn(0x55d842310820 msgr2=0x55d842312c30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-05T19:28:25.133 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.126+0000 7f541092fe40 1 -- 172.21.15.167:0/3774656944 shutdown_connections 2023-10-05T19:28:25.133 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.126+0000 7f541092fe40 1 -- 172.21.15.167:0/3774656944 wait complete. 2023-10-05T19:28:25.134 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.126+0000 7f541092fe40 1 Processor -- start 2023-10-05T19:28:25.134 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.126+0000 7f541092fe40 1 -- start start 2023-10-05T19:28:25.134 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.126+0000 7f541092fe40 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55d842289710 0x55d8423360b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-05T19:28:25.134 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.126+0000 7f541092fe40 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d8421e2e30 con 0x55d842289710 2023-10-05T19:28:25.134 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.130+0000 7f540e464700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55d842289710 0x55d8423360b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-05T19:28:25.135 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.130+0000 7f540e464700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55d842289710 0x55d8423360b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.118:3300/0 says I am v2:172.21.15.167:57072/0 (socket says 172.21.15.167:57072) 2023-10-05T19:28:25.135 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.130+0000 7f540e464700 1 -- 172.21.15.167:0/648533440 learned_addr learned my addr 172.21.15.167:0/648533440 (peer_addr_for_me v2:172.21.15.167:0/0) 2023-10-05T19:28:25.135 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.130+0000 7f540e464700 1 -- 172.21.15.167:0/648533440 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5408005db0 con 0x55d842289710 2023-10-05T19:28:25.135 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.130+0000 7f540e464700 1 --2- 172.21.15.167:0/648533440 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55d842289710 0x55d8423360b0 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7f5408005b50 tx=0x7f5408015000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=caddd69b28f4ba62 server_cookie=0 in_seq=0 out_seq=0 2023-10-05T19:28:25.136 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.130+0000 7f5407fff700 1 -- 172.21.15.167:0/648533440 <== mon.0 v2:172.21.15.118:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f54080055f0 con 0x55d842289710 2023-10-05T19:28:25.136 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.130+0000 7f5407fff700 1 -- 172.21.15.167:0/648533440 <== mon.0 v2:172.21.15.118:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5408008a30 con 0x55d842289710 2023-10-05T19:28:25.136 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.130+0000 7f5407fff700 1 -- 172.21.15.167:0/648533440 <== mon.0 v2:172.21.15.118:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f5408008d00 con 0x55d842289710 2023-10-05T19:28:25.137 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.130+0000 7f541092fe40 1 -- 172.21.15.167:0/648533440 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55d842289710 msgr2=0x55d8423360b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-05T19:28:25.137 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.130+0000 7f541092fe40 1 --2- 172.21.15.167:0/648533440 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55d842289710 0x55d8423360b0 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7f5408005b50 tx=0x7f5408015000 comp rx=0 tx=0).stop 2023-10-05T19:28:25.137 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.130+0000 7f541092fe40 1 -- 172.21.15.167:0/648533440 shutdown_connections 2023-10-05T19:28:25.137 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.130+0000 7f541092fe40 1 --2- 172.21.15.167:0/648533440 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55d842289710 0x55d8423360b0 unknown :-1 s=CLOSED pgs=172 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-05T19:28:25.137 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.130+0000 7f541092fe40 1 -- 172.21.15.167:0/648533440 >> 172.21.15.167:0/648533440 conn(0x55d842310820 msgr2=0x55d842331c70 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-05T19:28:25.137 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.130+0000 7f541092fe40 1 -- 172.21.15.167:0/648533440 shutdown_connections 2023-10-05T19:28:25.138 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.130+0000 7f541092fe40 1 -- 172.21.15.167:0/648533440 wait complete. 2023-10-05T19:28:25.138 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.134+0000 7f541092fe40 1 Processor -- start 2023-10-05T19:28:25.138 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.134+0000 7f541092fe40 1 -- start start 2023-10-05T19:28:25.138 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.134+0000 7f541092fe40 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55d842289710 0x55d84233ccc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-05T19:28:25.138 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.134+0000 7f541092fe40 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d8421b5ed0 con 0x55d842289710 2023-10-05T19:28:25.139 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.134+0000 7f540e464700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55d842289710 0x55d84233ccc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-05T19:28:25.139 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.134+0000 7f540e464700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55d842289710 0x55d84233ccc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.118:3300/0 says I am v2:172.21.15.167:57080/0 (socket says 172.21.15.167:57080) 2023-10-05T19:28:25.139 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.134+0000 7f540e464700 1 -- 172.21.15.167:0/803679881 learned_addr learned my addr 172.21.15.167:0/803679881 (peer_addr_for_me v2:172.21.15.167:0/0) 2023-10-05T19:28:25.140 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.134+0000 7f540e464700 1 -- 172.21.15.167:0/803679881 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55d84228b080 con 0x55d842289710 2023-10-05T19:28:25.140 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.134+0000 7f540e464700 1 --2- 172.21.15.167:0/803679881 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55d842289710 0x55d84233ccc0 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7f540800a650 tx=0x7f540801c000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-05T19:28:25.141 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.134+0000 7f54067fc700 1 -- 172.21.15.167:0/803679881 <== mon.0 v2:172.21.15.118:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f54080055b0 con 0x55d842289710 2023-10-05T19:28:25.141 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.134+0000 7f54067fc700 1 -- 172.21.15.167:0/803679881 <== mon.0 v2:172.21.15.118:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f54080269b0 con 0x55d842289710 2023-10-05T19:28:25.141 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.134+0000 7f541092fe40 1 -- 172.21.15.167:0/803679881 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x55d84233d200 con 0x55d842289710 2023-10-05T19:28:25.141 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.134+0000 7f54067fc700 1 -- 172.21.15.167:0/803679881 <== mon.0 v2:172.21.15.118:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f5408026ca0 con 0x55d842289710 2023-10-05T19:28:25.141 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.134+0000 7f541092fe40 1 -- 172.21.15.167:0/803679881 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55d84233d6a0 con 0x55d842289710 2023-10-05T19:28:25.142 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.138+0000 7f54067fc700 1 -- 172.21.15.167:0/803679881 <== mon.0 v2:172.21.15.118:3300/0 4 ==== mgrmap(e 5) v1 ==== 40329+0+0 (secure 0 0 0) 0x7f540802e030 con 0x55d842289710 2023-10-05T19:28:25.143 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.138+0000 7f54067fc700 1 --2- 172.21.15.167:0/803679881 >> [v2:172.21.15.167:6824/29889,v1:172.21.15.167:6825/29889] conn(0x7f53f0030c50 0x7f53f0033100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-05T19:28:25.143 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.138+0000 7f54067fc700 1 -- 172.21.15.167:0/803679881 <== mon.0 v2:172.21.15.118:3300/0 5 ==== osd_map(30..30 src has 1..30) v4 ==== 6057+0+0 (secure 0 0 0) 0x7f540803bc10 con 0x55d842289710 2023-10-05T19:28:25.143 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.138+0000 7f541092fe40 20 rados->read ofs=0 len=0 2023-10-05T19:28:25.144 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.138+0000 7f540dc63700 1 --2- 172.21.15.167:0/803679881 >> [v2:172.21.15.167:6824/29889,v1:172.21.15.167:6825/29889] conn(0x7f53f0030c50 0x7f53f0033100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-05T19:28:25.144 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.138+0000 7f541092fe40 1 --2- 172.21.15.167:0/803679881 >> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] conn(0x55d84228abf0 0x55d842348780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-05T19:28:25.144 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.138+0000 7f541092fe40 1 -- 172.21.15.167:0/803679881 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:1 5.12 5:49953fa1:::default.realm:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x55d842348df0 con 0x55d84228abf0 2023-10-05T19:28:25.144 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.138+0000 7f540ec65700 1 --2- 172.21.15.167:0/803679881 >> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] conn(0x55d84228abf0 0x55d842348780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-05T19:28:25.144 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.138+0000 7f540dc63700 1 --2- 172.21.15.167:0/803679881 >> [v2:172.21.15.167:6824/29889,v1:172.21.15.167:6825/29889] conn(0x7f53f0030c50 0x7f53f0033100 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f53f80016f0 tx=0x7f53f80067f0 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-05T19:28:25.145 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.138+0000 7f540ec65700 1 --2- 172.21.15.167:0/803679881 >> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] conn(0x55d84228abf0 0x55d842348780 crc :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-05T19:28:25.145 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.138+0000 7f540ec65700 1 -- 172.21.15.167:0/803679881 <== osd.1 v2:172.21.15.167:6800/30035 1 ==== osd_op_reply(1 default.realm [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 ==== 157+0+0 (crc 0 0 0) 0x7f540000a500 con 0x55d84228abf0 2023-10-05T19:28:25.145 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.138+0000 7f541092fe40 20 rados_obj.operate() r=-2 bl.length=0 2023-10-05T19:28:25.145 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.138+0000 7f541092fe40 20 realm 2023-10-05T19:28:25.145 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.138+0000 7f541092fe40 20 rados->read ofs=0 len=0 2023-10-05T19:28:25.146 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.138+0000 7f541092fe40 1 -- 172.21.15.167:0/803679881 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:2 5.12 5:49953fa1:::default.realm:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x55d842345de0 con 0x55d84228abf0 2023-10-05T19:28:25.147 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.138+0000 7f540ec65700 1 -- 172.21.15.167:0/803679881 <== osd.1 v2:172.21.15.167:6800/30035 2 ==== osd_op_reply(2 default.realm [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 ==== 157+0+0 (crc 0 0 0) 0x7f540000a500 con 0x55d84228abf0 2023-10-05T19:28:25.147 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.138+0000 7f541092fe40 20 rados_obj.operate() r=-2 bl.length=0 2023-10-05T19:28:25.147 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.138+0000 7f541092fe40 4 RGWPeriod::init failed to init realm id : (2) No such file or directory 2023-10-05T19:28:25.147 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.138+0000 7f541092fe40 20 rados->read ofs=0 len=0 2023-10-05T19:28:25.147 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.138+0000 7f541092fe40 1 -- 172.21.15.167:0/803679881 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:3 5.12 5:49953fa1:::default.realm:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x55d8423498d0 con 0x55d84228abf0 2023-10-05T19:28:25.147 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.142+0000 7f540ec65700 1 -- 172.21.15.167:0/803679881 <== osd.1 v2:172.21.15.167:6800/30035 3 ==== osd_op_reply(3 default.realm [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 ==== 157+0+0 (crc 0 0 0) 0x7f540000a500 con 0x55d84228abf0 2023-10-05T19:28:25.147 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.142+0000 7f541092fe40 20 rados_obj.operate() r=-2 bl.length=0 2023-10-05T19:28:25.147 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.142+0000 7f541092fe40 20 rados->read ofs=0 len=0 2023-10-05T19:28:25.147 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.142+0000 7f541092fe40 1 -- 172.21.15.167:0/803679881 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:4 5.f 5:f43b6ece:::zone_names.default:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x55d8423494b0 con 0x55d84228abf0 2023-10-05T19:28:25.147 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.142+0000 7f540ec65700 1 -- 172.21.15.167:0/803679881 <== osd.1 v2:172.21.15.167:6800/30035 4 ==== osd_op_reply(4 zone_names.default [read 0~46 out=46b] v0'0 uv1 ondisk = 0) v8 ==== 162+0+46 (crc 0 0 0) 0x7f540000a500 con 0x55d84228abf0 2023-10-05T19:28:25.148 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.142+0000 7f541092fe40 20 rados_obj.operate() r=0 bl.length=46 2023-10-05T19:28:25.148 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.142+0000 7f541092fe40 20 rados->read ofs=0 len=0 2023-10-05T19:28:25.148 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.142+0000 7f541092fe40 1 -- 172.21.15.167:0/803679881 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:5 5.1d 5:b8374eb4:::zone_info.9a9df965-6612-45d4-a45f-6e3c889bacc2:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x55d84234a340 con 0x55d84228abf0 2023-10-05T19:28:25.149 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.142+0000 7f540ec65700 1 -- 172.21.15.167:0/803679881 <== osd.1 v2:172.21.15.167:6800/30035 5 ==== osd_op_reply(5 zone_info.9a9df965-6612-45d4-a45f-6e3c889bacc2 [read 0~874 out=874b] v0'0 uv2 ondisk = 0) v8 ==== 190+0+874 (crc 0 0 0) 0x7f540000a500 con 0x55d84228abf0 2023-10-05T19:28:25.149 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.142+0000 7f541092fe40 20 rados_obj.operate() r=0 bl.length=874 2023-10-05T19:28:25.149 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.142+0000 7f541092fe40 20 searching for the correct realm 2023-10-05T19:28:25.149 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.142+0000 7f541092fe40 1 --2- 172.21.15.167:0/803679881 >> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] conn(0x55d84234c190 0x55d84234e570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-05T19:28:25.149 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.142+0000 7f541092fe40 1 -- 172.21.15.167:0/803679881 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:6 5.0 5:00000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x55d84234eab0 con 0x55d84234c190 2023-10-05T19:28:25.149 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.142+0000 7f540e464700 1 --2- 172.21.15.167:0/803679881 >> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] conn(0x55d84234c190 0x55d84234e570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-05T19:28:25.149 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.142+0000 7f540e464700 1 --2- 172.21.15.167:0/803679881 >> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] conn(0x55d84234c190 0x55d84234e570 crc :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-05T19:28:25.150 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.146+0000 7f540e464700 1 -- 172.21.15.167:0/803679881 <== osd.3 v2:172.21.15.167:6808/30036 1 ==== osd_op_reply(6 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f540801cc30 con 0x55d84234c190 2023-10-05T19:28:25.150 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.146+0000 7f540e464700 1 --2- 172.21.15.167:0/803679881 >> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] conn(0x7f5408007030 0x7f540801c410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-05T19:28:25.150 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.146+0000 7f540e464700 1 -- 172.21.15.167:0/803679881 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:7 5.10 5:08000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x55d84234eab0 con 0x7f5408007030 2023-10-05T19:28:25.150 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.146+0000 7f540dc63700 1 --2- 172.21.15.167:0/803679881 >> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] conn(0x7f5408007030 0x7f540801c410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-05T19:28:25.151 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.146+0000 7f540dc63700 1 --2- 172.21.15.167:0/803679881 >> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] conn(0x7f5408007030 0x7f540801c410 crc :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-05T19:28:25.151 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.146+0000 7f540dc63700 1 -- 172.21.15.167:0/803679881 <== osd.2 v2:172.21.15.167:6816/30037 1 ==== osd_op_reply(7 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f53f8003450 con 0x7f5408007030 2023-10-05T19:28:25.151 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.146+0000 7f540dc63700 1 --2- 172.21.15.167:0/803679881 >> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] conn(0x7f53f8011040 0x7f53f8015960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-05T19:28:25.152 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.146+0000 7f540dc63700 1 -- 172.21.15.167:0/803679881 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:8 5.8 5:10000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x55d84234eab0 con 0x7f53f8011040 2023-10-05T19:28:25.152 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.146+0000 7f540ec65700 1 --2- 172.21.15.167:0/803679881 >> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] conn(0x7f53f8011040 0x7f53f8015960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-05T19:28:25.152 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.146+0000 7f540ec65700 1 --2- 172.21.15.167:0/803679881 >> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] conn(0x7f53f8011040 0x7f53f8015960 crc :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-05T19:28:25.153 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.146+0000 7f540ec65700 1 -- 172.21.15.167:0/803679881 <== osd.0 v2:172.21.15.118:6800/27690 1 ==== osd_op_reply(8 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f5400014040 con 0x7f53f8011040 2023-10-05T19:28:25.153 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.146+0000 7f540ec65700 1 -- 172.21.15.167:0/803679881 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:9 5.18 5:18000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x55d84234eab0 con 0x55d84228abf0 2023-10-05T19:28:25.154 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.150+0000 7f540ec65700 1 -- 172.21.15.167:0/803679881 <== osd.1 v2:172.21.15.167:6800/30035 6 ==== osd_op_reply(9 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f5400014040 con 0x55d84228abf0 2023-10-05T19:28:25.154 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.150+0000 7f540ec65700 1 -- 172.21.15.167:0/803679881 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:10 5.4 5:20000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x55d84234eab0 con 0x55d84228abf0 2023-10-05T19:28:25.154 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.150+0000 7f540ec65700 1 -- 172.21.15.167:0/803679881 <== osd.1 v2:172.21.15.167:6800/30035 7 ==== osd_op_reply(10 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f5400014040 con 0x55d84228abf0 2023-10-05T19:28:25.154 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.150+0000 7f540ec65700 1 -- 172.21.15.167:0/803679881 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:11 5.14 5:28000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x55d84234eab0 con 0x7f53f8011040 2023-10-05T19:28:25.155 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.150+0000 7f540ec65700 1 -- 172.21.15.167:0/803679881 <== osd.0 v2:172.21.15.118:6800/27690 2 ==== osd_op_reply(11 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f5400014040 con 0x7f53f8011040 2023-10-05T19:28:25.155 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.150+0000 7f540ec65700 1 -- 172.21.15.167:0/803679881 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:12 5.c 5:30000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x55d84234eab0 con 0x7f53f8011040 2023-10-05T19:28:25.156 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.150+0000 7f540ec65700 1 -- 172.21.15.167:0/803679881 <== osd.0 v2:172.21.15.118:6800/27690 3 ==== osd_op_reply(12 [pgnls start_epoch 30 out=112b] v23'1 uv1 ondisk = 1) v8 ==== 144+0+112 (crc 0 0 0) 0x7f5400014040 con 0x7f53f8011040 2023-10-05T19:28:25.156 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.150+0000 7f540ec65700 1 -- 172.21.15.167:0/803679881 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:13 5.1c 5:38000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x55d84234eab0 con 0x7f5408007030 2023-10-05T19:28:25.156 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.150+0000 7f540dc63700 1 -- 172.21.15.167:0/803679881 <== osd.2 v2:172.21.15.167:6816/30037 2 ==== osd_op_reply(13 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f53f8003450 con 0x7f5408007030 2023-10-05T19:28:25.157 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.150+0000 7f540dc63700 1 -- 172.21.15.167:0/803679881 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:14 5.2 5:40000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x55d84234eab0 con 0x55d84234c190 2023-10-05T19:28:25.157 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.150+0000 7f540e464700 1 -- 172.21.15.167:0/803679881 <== osd.3 v2:172.21.15.167:6808/30036 2 ==== osd_op_reply(14 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f540801cc30 con 0x55d84234c190 2023-10-05T19:28:25.157 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.150+0000 7f540e464700 1 -- 172.21.15.167:0/803679881 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:15 5.12 5:48000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x55d84234eab0 con 0x55d84228abf0 2023-10-05T19:28:25.158 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.154+0000 7f540ec65700 1 -- 172.21.15.167:0/803679881 <== osd.1 v2:172.21.15.167:6800/30035 8 ==== osd_op_reply(15 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f5400014040 con 0x55d84228abf0 2023-10-05T19:28:25.158 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.154+0000 7f540ec65700 1 -- 172.21.15.167:0/803679881 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:16 5.a 5:50000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x55d84234eab0 con 0x7f53f8011040 2023-10-05T19:28:25.159 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.154+0000 7f540ec65700 1 -- 172.21.15.167:0/803679881 <== osd.0 v2:172.21.15.118:6800/27690 4 ==== osd_op_reply(16 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f5400014040 con 0x7f53f8011040 2023-10-05T19:28:25.159 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.154+0000 7f540ec65700 1 -- 172.21.15.167:0/803679881 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:17 5.1a 5:58000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x55d84234eab0 con 0x55d84228abf0 2023-10-05T19:28:25.159 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.154+0000 7f540ec65700 1 -- 172.21.15.167:0/803679881 <== osd.1 v2:172.21.15.167:6800/30035 9 ==== osd_op_reply(17 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f5400014040 con 0x55d84228abf0 2023-10-05T19:28:25.159 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.154+0000 7f540ec65700 1 -- 172.21.15.167:0/803679881 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:18 5.6 5:60000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x55d84234eab0 con 0x7f5408007030 2023-10-05T19:28:25.160 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.154+0000 7f540dc63700 1 -- 172.21.15.167:0/803679881 <== osd.2 v2:172.21.15.167:6816/30037 3 ==== osd_op_reply(18 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f53f8003450 con 0x7f5408007030 2023-10-05T19:28:25.160 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.154+0000 7f540dc63700 1 -- 172.21.15.167:0/803679881 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:19 5.16 5:68000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x55d84234eab0 con 0x55d84234c190 2023-10-05T19:28:25.161 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.154+0000 7f540e464700 1 -- 172.21.15.167:0/803679881 <== osd.3 v2:172.21.15.167:6808/30036 3 ==== osd_op_reply(19 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f540801cc30 con 0x55d84234c190 2023-10-05T19:28:25.161 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.154+0000 7f540e464700 1 -- 172.21.15.167:0/803679881 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:20 5.e 5:70000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x55d84234eab0 con 0x55d84234c190 2023-10-05T19:28:25.161 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.154+0000 7f540e464700 1 -- 172.21.15.167:0/803679881 <== osd.3 v2:172.21.15.167:6808/30036 4 ==== osd_op_reply(20 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f540801cc30 con 0x55d84234c190 2023-10-05T19:28:25.161 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.154+0000 7f540e464700 1 -- 172.21.15.167:0/803679881 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:21 5.1e 5:78000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x55d84234eab0 con 0x7f53f8011040 2023-10-05T19:28:25.162 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.158+0000 7f540ec65700 1 -- 172.21.15.167:0/803679881 <== osd.0 v2:172.21.15.118:6800/27690 5 ==== osd_op_reply(21 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f5400022040 con 0x7f53f8011040 2023-10-05T19:28:25.162 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.158+0000 7f540ec65700 1 -- 172.21.15.167:0/803679881 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:22 5.1 5:80000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x55d84234eab0 con 0x55d84234c190 2023-10-05T19:28:25.163 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.158+0000 7f540e464700 1 -- 172.21.15.167:0/803679881 <== osd.3 v2:172.21.15.167:6808/30036 5 ==== osd_op_reply(22 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f540801cc30 con 0x55d84234c190 2023-10-05T19:28:25.163 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.158+0000 7f540e464700 1 -- 172.21.15.167:0/803679881 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:23 5.11 5:88000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x55d84234eab0 con 0x55d84234c190 2023-10-05T19:28:25.163 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.158+0000 7f540e464700 1 -- 172.21.15.167:0/803679881 <== osd.3 v2:172.21.15.167:6808/30036 6 ==== osd_op_reply(23 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f540801cc30 con 0x55d84234c190 2023-10-05T19:28:25.163 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.158+0000 7f540e464700 1 -- 172.21.15.167:0/803679881 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:24 5.9 5:90000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x55d84234eab0 con 0x55d84228abf0 2023-10-05T19:28:25.164 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.158+0000 7f540ec65700 1 -- 172.21.15.167:0/803679881 <== osd.1 v2:172.21.15.167:6800/30035 10 ==== osd_op_reply(24 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f5400022040 con 0x55d84228abf0 2023-10-05T19:28:25.164 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.158+0000 7f540ec65700 1 -- 172.21.15.167:0/803679881 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:25 5.19 5:98000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x55d84234eab0 con 0x55d84228abf0 2023-10-05T19:28:25.165 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.158+0000 7f540ec65700 1 -- 172.21.15.167:0/803679881 <== osd.1 v2:172.21.15.167:6800/30035 11 ==== osd_op_reply(25 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f5400022040 con 0x55d84228abf0 2023-10-05T19:28:25.165 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.158+0000 7f540ec65700 1 -- 172.21.15.167:0/803679881 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:26 5.5 5:a0000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x55d84234eab0 con 0x7f53f8011040 2023-10-05T19:28:25.166 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.162+0000 7f540ec65700 1 -- 172.21.15.167:0/803679881 <== osd.0 v2:172.21.15.118:6800/27690 6 ==== osd_op_reply(26 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f5400022040 con 0x7f53f8011040 2023-10-05T19:28:25.166 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.162+0000 7f540ec65700 1 -- 172.21.15.167:0/803679881 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:27 5.15 5:a8000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x55d84234eab0 con 0x55d84234c190 2023-10-05T19:28:25.166 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.162+0000 7f540e464700 1 -- 172.21.15.167:0/803679881 <== osd.3 v2:172.21.15.167:6808/30036 7 ==== osd_op_reply(27 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f540801cc30 con 0x55d84234c190 2023-10-05T19:28:25.167 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.162+0000 7f540e464700 1 -- 172.21.15.167:0/803679881 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:28 5.d 5:b0000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x55d84234eab0 con 0x7f5408007030 2023-10-05T19:28:25.167 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.162+0000 7f540dc63700 1 -- 172.21.15.167:0/803679881 <== osd.2 v2:172.21.15.167:6816/30037 4 ==== osd_op_reply(28 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f53f8003450 con 0x7f5408007030 2023-10-05T19:28:25.167 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.162+0000 7f540dc63700 1 -- 172.21.15.167:0/803679881 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:29 5.1d 5:b8000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x55d84234eab0 con 0x55d84228abf0 2023-10-05T19:28:25.168 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.162+0000 7f540ec65700 1 -- 172.21.15.167:0/803679881 <== osd.1 v2:172.21.15.167:6800/30035 12 ==== osd_op_reply(29 [pgnls start_epoch 30 out=107b] v30'2 uv2 ondisk = 1) v8 ==== 144+0+107 (crc 0 0 0) 0x7f5400022040 con 0x55d84228abf0 2023-10-05T19:28:25.168 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.162+0000 7f540ec65700 1 -- 172.21.15.167:0/803679881 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:30 5.3 5:c0000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x55d84234eab0 con 0x7f53f8011040 2023-10-05T19:28:25.169 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.162+0000 7f540ec65700 1 -- 172.21.15.167:0/803679881 <== osd.0 v2:172.21.15.118:6800/27690 7 ==== osd_op_reply(30 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f5400022040 con 0x7f53f8011040 2023-10-05T19:28:25.169 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.162+0000 7f540ec65700 1 -- 172.21.15.167:0/803679881 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:31 5.13 5:c8000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x55d84234eab0 con 0x55d84234c190 2023-10-05T19:28:25.169 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.162+0000 7f540e464700 1 -- 172.21.15.167:0/803679881 <== osd.3 v2:172.21.15.167:6808/30036 8 ==== osd_op_reply(31 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f540801cc30 con 0x55d84234c190 2023-10-05T19:28:25.170 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.162+0000 7f540e464700 1 -- 172.21.15.167:0/803679881 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:32 5.b 5:d0000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x55d84234eab0 con 0x7f53f8011040 2023-10-05T19:28:25.171 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.166+0000 7f540ec65700 1 -- 172.21.15.167:0/803679881 <== osd.0 v2:172.21.15.118:6800/27690 8 ==== osd_op_reply(32 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f5400022040 con 0x7f53f8011040 2023-10-05T19:28:25.171 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.166+0000 7f540ec65700 1 -- 172.21.15.167:0/803679881 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:33 5.1b 5:d8000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x55d84234eab0 con 0x7f5408007030 2023-10-05T19:28:25.171 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.166+0000 7f540dc63700 1 -- 172.21.15.167:0/803679881 <== osd.2 v2:172.21.15.167:6816/30037 5 ==== osd_op_reply(33 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f53f8003450 con 0x7f5408007030 2023-10-05T19:28:25.171 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.166+0000 7f540dc63700 1 -- 172.21.15.167:0/803679881 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:34 5.7 5:e0000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x55d84234eab0 con 0x55d84234c190 2023-10-05T19:28:25.172 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.166+0000 7f540e464700 1 -- 172.21.15.167:0/803679881 <== osd.3 v2:172.21.15.167:6808/30036 9 ==== osd_op_reply(34 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f540801cc30 con 0x55d84234c190 2023-10-05T19:28:25.172 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.166+0000 7f540e464700 1 -- 172.21.15.167:0/803679881 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:35 5.17 5:e8000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x55d84234eab0 con 0x7f53f8011040 2023-10-05T19:28:25.173 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.166+0000 7f540ec65700 1 -- 172.21.15.167:0/803679881 <== osd.0 v2:172.21.15.118:6800/27690 9 ==== osd_op_reply(35 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f5400022040 con 0x7f53f8011040 2023-10-05T19:28:25.173 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.166+0000 7f540ec65700 1 -- 172.21.15.167:0/803679881 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:36 5.f 5:f0000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x55d84234eab0 con 0x55d84228abf0 2023-10-05T19:28:25.174 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.170+0000 7f540ec65700 1 -- 172.21.15.167:0/803679881 <== osd.1 v2:172.21.15.167:6800/30035 13 ==== osd_op_reply(36 [pgnls start_epoch 30 out=115b] v23'2 uv2 ondisk = 1) v8 ==== 144+0+115 (crc 0 0 0) 0x7f5400022040 con 0x55d84228abf0 2023-10-05T19:28:25.174 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.170+0000 7f540ec65700 1 -- 172.21.15.167:0/803679881 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:37 5.1f 5:f8000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x55d84234eab0 con 0x7f5408007030 2023-10-05T19:28:25.175 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.170+0000 7f540dc63700 1 -- 172.21.15.167:0/803679881 <== osd.2 v2:172.21.15.167:6816/30037 6 ==== osd_op_reply(37 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f53f8003450 con 0x7f5408007030 2023-10-05T19:28:25.175 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.170+0000 7f541092fe40 20 RGWRados::pool_iterate: got zonegroup_info.4f1f719d-9648-417b-b10c-e5d939825419 2023-10-05T19:28:25.175 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.170+0000 7f541092fe40 20 RGWRados::pool_iterate: got zone_info.9a9df965-6612-45d4-a45f-6e3c889bacc2 2023-10-05T19:28:25.175 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.170+0000 7f541092fe40 20 RGWRados::pool_iterate: got zone_names.default 2023-10-05T19:28:25.175 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.170+0000 7f541092fe40 20 RGWRados::pool_iterate: got zonegroups_names.default 2023-10-05T19:28:25.175 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.170+0000 7f541092fe40 20 rados->read ofs=0 len=0 2023-10-05T19:28:25.175 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.170+0000 7f541092fe40 1 -- 172.21.15.167:0/803679881 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:38 5.12 5:49953fa1:::default.realm:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x55d84234af40 con 0x55d84228abf0 2023-10-05T19:28:25.176 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.170+0000 7f540ec65700 1 -- 172.21.15.167:0/803679881 <== osd.1 v2:172.21.15.167:6800/30035 14 ==== osd_op_reply(38 default.realm [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 ==== 157+0+0 (crc 0 0 0) 0x7f5400022040 con 0x55d84228abf0 2023-10-05T19:28:25.176 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.170+0000 7f541092fe40 20 rados_obj.operate() r=-2 bl.length=0 2023-10-05T19:28:25.176 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.170+0000 7f541092fe40 20 rados->read ofs=0 len=0 2023-10-05T19:28:25.176 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.170+0000 7f541092fe40 1 -- 172.21.15.167:0/803679881 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:39 5.f 5:f4c53578:::zonegroups_names.default:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x55d84234b8c0 con 0x55d84228abf0 2023-10-05T19:28:25.177 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.170+0000 7f540ec65700 1 -- 172.21.15.167:0/803679881 <== osd.1 v2:172.21.15.167:6800/30035 15 ==== osd_op_reply(39 zonegroups_names.default [read 0~46 out=46b] v0'0 uv2 ondisk = 0) v8 ==== 168+0+46 (crc 0 0 0) 0x7f5400022040 con 0x55d84228abf0 2023-10-05T19:28:25.177 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.170+0000 7f541092fe40 20 rados_obj.operate() r=0 bl.length=46 2023-10-05T19:28:25.177 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.170+0000 7f541092fe40 20 rados->read ofs=0 len=0 2023-10-05T19:28:25.177 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.170+0000 7f541092fe40 1 -- 172.21.15.167:0/803679881 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:40 5.c 5:30d299a5:::zonegroup_info.4f1f719d-9648-417b-b10c-e5d939825419:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x55d84234ab30 con 0x7f53f8011040 2023-10-05T19:28:25.178 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.170+0000 7f540ec65700 1 -- 172.21.15.167:0/803679881 <== osd.0 v2:172.21.15.118:6800/27690 10 ==== osd_op_reply(40 zonegroup_info.4f1f719d-9648-417b-b10c-e5d939825419 [read 0~362 out=362b] v0'0 uv1 ondisk = 0) v8 ==== 195+0+362 (crc 0 0 0) 0x7f5400022040 con 0x7f53f8011040 2023-10-05T19:28:25.178 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.170+0000 7f541092fe40 20 rados_obj.operate() r=0 bl.length=362 2023-10-05T19:28:25.178 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.170+0000 7f541092fe40 20 rados->read ofs=0 len=0 2023-10-05T19:28:25.178 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.170+0000 7f541092fe40 1 -- 172.21.15.167:0/803679881 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:41 5.13 5:cbbf6c60:::converted:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x55d84234f3a0 con 0x55d84234c190 2023-10-05T19:28:25.178 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.170+0000 7f540e464700 1 -- 172.21.15.167:0/803679881 <== osd.3 v2:172.21.15.167:6808/30036 10 ==== osd_op_reply(41 converted [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 ==== 153+0+0 (crc 0 0 0) 0x7f540801cc30 con 0x55d84234c190 2023-10-05T19:28:25.178 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.170+0000 7f541092fe40 20 rados_obj.operate() r=-2 bl.length=0 2023-10-05T19:28:25.178 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.170+0000 7f541092fe40 20 rados->read ofs=0 len=0 2023-10-05T19:28:25.178 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.170+0000 7f541092fe40 1 -- 172.21.15.167:0/803679881 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:42 5.12 5:49953fa1:::default.realm:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x55d84234f750 con 0x55d84228abf0 2023-10-05T19:28:25.179 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.174+0000 7f540ec65700 1 -- 172.21.15.167:0/803679881 <== osd.1 v2:172.21.15.167:6800/30035 16 ==== osd_op_reply(42 default.realm [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 ==== 157+0+0 (crc 0 0 0) 0x7f5400022040 con 0x55d84228abf0 2023-10-05T19:28:25.179 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.174+0000 7f541092fe40 20 rados_obj.operate() r=-2 bl.length=0 2023-10-05T19:28:25.179 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.174+0000 7f541092fe40 20 rados->read ofs=0 len=0 2023-10-05T19:28:25.179 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.174+0000 7f541092fe40 1 -- 172.21.15.167:0/803679881 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:43 5.f 5:f4c53578:::zonegroups_names.default:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x55d84234f750 con 0x55d84228abf0 2023-10-05T19:28:25.179 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.174+0000 7f540ec65700 1 -- 172.21.15.167:0/803679881 <== osd.1 v2:172.21.15.167:6800/30035 17 ==== osd_op_reply(43 zonegroups_names.default [read 0~46 out=46b] v0'0 uv2 ondisk = 0) v8 ==== 168+0+46 (crc 0 0 0) 0x7f5400022040 con 0x55d84228abf0 2023-10-05T19:28:25.180 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.174+0000 7f541092fe40 20 rados_obj.operate() r=0 bl.length=46 2023-10-05T19:28:25.180 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.174+0000 7f541092fe40 1 -- 172.21.15.167:0/803679881 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:44 5.0 5:00000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x55d842350760 con 0x55d84234c190 2023-10-05T19:28:25.181 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.174+0000 7f540e464700 1 -- 172.21.15.167:0/803679881 <== osd.3 v2:172.21.15.167:6808/30036 11 ==== osd_op_reply(44 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f540801cc30 con 0x55d84234c190 2023-10-05T19:28:25.181 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.174+0000 7f540e464700 1 -- 172.21.15.167:0/803679881 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:45 5.10 5:08000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x55d842350760 con 0x7f5408007030 2023-10-05T19:28:25.182 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.178+0000 7f540dc63700 1 -- 172.21.15.167:0/803679881 <== osd.2 v2:172.21.15.167:6816/30037 7 ==== osd_op_reply(45 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f53f8003450 con 0x7f5408007030 2023-10-05T19:28:25.182 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.178+0000 7f540dc63700 1 -- 172.21.15.167:0/803679881 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:46 5.8 5:10000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x55d842350760 con 0x7f53f8011040 2023-10-05T19:28:25.183 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.178+0000 7f540ec65700 1 -- 172.21.15.167:0/803679881 <== osd.0 v2:172.21.15.118:6800/27690 11 ==== osd_op_reply(46 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f5400022040 con 0x7f53f8011040 2023-10-05T19:28:25.183 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.178+0000 7f540ec65700 1 -- 172.21.15.167:0/803679881 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:47 5.18 5:18000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x55d84234ab30 con 0x55d84228abf0 2023-10-05T19:28:25.184 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.178+0000 7f540ec65700 1 -- 172.21.15.167:0/803679881 <== osd.1 v2:172.21.15.167:6800/30035 18 ==== osd_op_reply(47 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f5400022040 con 0x55d84228abf0 2023-10-05T19:28:25.184 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.178+0000 7f540ec65700 1 -- 172.21.15.167:0/803679881 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:48 5.4 5:20000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x55d84234ab30 con 0x55d84228abf0 2023-10-05T19:28:25.184 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.178+0000 7f540ec65700 1 -- 172.21.15.167:0/803679881 <== osd.1 v2:172.21.15.167:6800/30035 19 ==== osd_op_reply(48 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f5400022040 con 0x55d84228abf0 2023-10-05T19:28:25.184 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.178+0000 7f540ec65700 1 -- 172.21.15.167:0/803679881 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:49 5.14 5:28000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x55d84234ab30 con 0x7f53f8011040 2023-10-05T19:28:25.185 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.178+0000 7f540ec65700 1 -- 172.21.15.167:0/803679881 <== osd.0 v2:172.21.15.118:6800/27690 12 ==== osd_op_reply(49 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f5400022040 con 0x7f53f8011040 2023-10-05T19:28:25.185 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.178+0000 7f540ec65700 1 -- 172.21.15.167:0/803679881 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:50 5.c 5:30000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x55d84234ab30 con 0x7f53f8011040 2023-10-05T19:28:25.186 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.182+0000 7f540ec65700 1 -- 172.21.15.167:0/803679881 <== osd.0 v2:172.21.15.118:6800/27690 13 ==== osd_op_reply(50 [pgnls start_epoch 30 out=112b] v23'1 uv1 ondisk = 1) v8 ==== 144+0+112 (crc 0 0 0) 0x7f5400022040 con 0x7f53f8011040 2023-10-05T19:28:25.186 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.182+0000 7f540ec65700 1 -- 172.21.15.167:0/803679881 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:51 5.1c 5:38000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x55d84234ab30 con 0x7f5408007030 2023-10-05T19:28:25.187 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.182+0000 7f540dc63700 1 -- 172.21.15.167:0/803679881 <== osd.2 v2:172.21.15.167:6816/30037 8 ==== osd_op_reply(51 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f53f8003450 con 0x7f5408007030 2023-10-05T19:28:25.187 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.182+0000 7f540dc63700 1 -- 172.21.15.167:0/803679881 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:52 5.2 5:40000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x55d84234ab30 con 0x55d84234c190 2023-10-05T19:28:25.188 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.182+0000 7f540e464700 1 -- 172.21.15.167:0/803679881 <== osd.3 v2:172.21.15.167:6808/30036 12 ==== osd_op_reply(52 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f540801cc30 con 0x55d84234c190 2023-10-05T19:28:25.188 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.182+0000 7f540e464700 1 -- 172.21.15.167:0/803679881 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:53 5.12 5:48000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x55d84234ab30 con 0x55d84228abf0 2023-10-05T19:28:25.188 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.182+0000 7f540ec65700 1 -- 172.21.15.167:0/803679881 <== osd.1 v2:172.21.15.167:6800/30035 20 ==== osd_op_reply(53 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f5400022040 con 0x55d84228abf0 2023-10-05T19:28:25.189 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.182+0000 7f540ec65700 1 -- 172.21.15.167:0/803679881 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:54 5.a 5:50000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x55d84234ab30 con 0x7f53f8011040 2023-10-05T19:28:25.189 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.182+0000 7f540ec65700 1 -- 172.21.15.167:0/803679881 <== osd.0 v2:172.21.15.118:6800/27690 14 ==== osd_op_reply(54 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f5400022040 con 0x7f53f8011040 2023-10-05T19:28:25.190 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.182+0000 7f540ec65700 1 -- 172.21.15.167:0/803679881 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:55 5.1a 5:58000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x55d84234ab30 con 0x55d84228abf0 2023-10-05T19:28:25.190 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.186+0000 7f540ec65700 1 -- 172.21.15.167:0/803679881 <== osd.1 v2:172.21.15.167:6800/30035 21 ==== osd_op_reply(55 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f5400022040 con 0x55d84228abf0 2023-10-05T19:28:25.191 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.186+0000 7f540ec65700 1 -- 172.21.15.167:0/803679881 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:56 5.6 5:60000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x55d84234ab30 con 0x7f5408007030 2023-10-05T19:28:25.191 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.186+0000 7f540dc63700 1 -- 172.21.15.167:0/803679881 <== osd.2 v2:172.21.15.167:6816/30037 9 ==== osd_op_reply(56 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f53f8003450 con 0x7f5408007030 2023-10-05T19:28:25.192 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.186+0000 7f540dc63700 1 -- 172.21.15.167:0/803679881 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:57 5.16 5:68000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x55d84234ab30 con 0x55d84234c190 2023-10-05T19:28:25.192 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.186+0000 7f540e464700 1 -- 172.21.15.167:0/803679881 <== osd.3 v2:172.21.15.167:6808/30036 13 ==== osd_op_reply(57 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f540801cc30 con 0x55d84234c190 2023-10-05T19:28:25.192 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.186+0000 7f540e464700 1 -- 172.21.15.167:0/803679881 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:58 5.e 5:70000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x55d84234ab30 con 0x55d84234c190 2023-10-05T19:28:25.193 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.186+0000 7f540e464700 1 -- 172.21.15.167:0/803679881 <== osd.3 v2:172.21.15.167:6808/30036 14 ==== osd_op_reply(58 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f540801cc30 con 0x55d84234c190 2023-10-05T19:28:25.193 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.186+0000 7f540e464700 1 -- 172.21.15.167:0/803679881 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:59 5.1e 5:78000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x55d84234ab30 con 0x7f53f8011040 2023-10-05T19:28:25.194 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.190+0000 7f540ec65700 1 -- 172.21.15.167:0/803679881 <== osd.0 v2:172.21.15.118:6800/27690 15 ==== osd_op_reply(59 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f5400022040 con 0x7f53f8011040 2023-10-05T19:28:25.194 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.190+0000 7f540ec65700 1 -- 172.21.15.167:0/803679881 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:60 5.1 5:80000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x55d84234ab30 con 0x55d84234c190 2023-10-05T19:28:25.195 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.190+0000 7f540e464700 1 -- 172.21.15.167:0/803679881 <== osd.3 v2:172.21.15.167:6808/30036 15 ==== osd_op_reply(60 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f540801cc30 con 0x55d84234c190 2023-10-05T19:28:25.195 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.190+0000 7f540e464700 1 -- 172.21.15.167:0/803679881 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:61 5.11 5:88000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x55d84234ab30 con 0x55d84234c190 2023-10-05T19:28:25.195 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.190+0000 7f540e464700 1 -- 172.21.15.167:0/803679881 <== osd.3 v2:172.21.15.167:6808/30036 16 ==== osd_op_reply(61 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f540801cc30 con 0x55d84234c190 2023-10-05T19:28:25.195 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.190+0000 7f540e464700 1 -- 172.21.15.167:0/803679881 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:62 5.9 5:90000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x55d84234ab30 con 0x55d84228abf0 2023-10-05T19:28:25.196 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.190+0000 7f540ec65700 1 -- 172.21.15.167:0/803679881 <== osd.1 v2:172.21.15.167:6800/30035 22 ==== osd_op_reply(62 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f5400022040 con 0x55d84228abf0 2023-10-05T19:28:25.196 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.190+0000 7f540ec65700 1 -- 172.21.15.167:0/803679881 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:63 5.19 5:98000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x55d84234ab30 con 0x55d84228abf0 2023-10-05T19:28:25.197 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.190+0000 7f540ec65700 1 -- 172.21.15.167:0/803679881 <== osd.1 v2:172.21.15.167:6800/30035 23 ==== osd_op_reply(63 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f5400022040 con 0x55d84228abf0 2023-10-05T19:28:25.197 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.190+0000 7f540ec65700 1 -- 172.21.15.167:0/803679881 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:64 5.5 5:a0000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x55d84234ab30 con 0x7f53f8011040 2023-10-05T19:28:25.198 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.194+0000 7f540ec65700 1 -- 172.21.15.167:0/803679881 <== osd.0 v2:172.21.15.118:6800/27690 16 ==== osd_op_reply(64 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f5400022040 con 0x7f53f8011040 2023-10-05T19:28:25.198 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.194+0000 7f540ec65700 1 -- 172.21.15.167:0/803679881 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:65 5.15 5:a8000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x55d84234ab30 con 0x55d84234c190 2023-10-05T19:28:25.198 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.194+0000 7f540e464700 1 -- 172.21.15.167:0/803679881 <== osd.3 v2:172.21.15.167:6808/30036 17 ==== osd_op_reply(65 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f540801cc30 con 0x55d84234c190 2023-10-05T19:28:25.199 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.194+0000 7f540e464700 1 -- 172.21.15.167:0/803679881 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:66 5.d 5:b0000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x55d84234ab30 con 0x7f5408007030 2023-10-05T19:28:25.199 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.194+0000 7f540dc63700 1 -- 172.21.15.167:0/803679881 <== osd.2 v2:172.21.15.167:6816/30037 10 ==== osd_op_reply(66 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f53f8003450 con 0x7f5408007030 2023-10-05T19:28:25.199 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.194+0000 7f540dc63700 1 -- 172.21.15.167:0/803679881 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:67 5.1d 5:b8000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x55d84234ab30 con 0x55d84228abf0 2023-10-05T19:28:25.200 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.194+0000 7f540ec65700 1 -- 172.21.15.167:0/803679881 <== osd.1 v2:172.21.15.167:6800/30035 24 ==== osd_op_reply(67 [pgnls start_epoch 30 out=107b] v30'2 uv2 ondisk = 1) v8 ==== 144+0+107 (crc 0 0 0) 0x7f5400022040 con 0x55d84228abf0 2023-10-05T19:28:25.200 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.194+0000 7f540ec65700 1 -- 172.21.15.167:0/803679881 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:68 5.3 5:c0000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x55d84234ab30 con 0x7f53f8011040 2023-10-05T19:28:25.201 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.194+0000 7f540ec65700 1 -- 172.21.15.167:0/803679881 <== osd.0 v2:172.21.15.118:6800/27690 17 ==== osd_op_reply(68 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f5400022040 con 0x7f53f8011040 2023-10-05T19:28:25.201 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.194+0000 7f540ec65700 1 -- 172.21.15.167:0/803679881 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:69 5.13 5:c8000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x55d84234ab30 con 0x55d84234c190 2023-10-05T19:28:25.202 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.194+0000 7f540e464700 1 -- 172.21.15.167:0/803679881 <== osd.3 v2:172.21.15.167:6808/30036 18 ==== osd_op_reply(69 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f540801cc30 con 0x55d84234c190 2023-10-05T19:28:25.202 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.194+0000 7f540e464700 1 -- 172.21.15.167:0/803679881 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:70 5.b 5:d0000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x55d84234ab30 con 0x7f53f8011040 2023-10-05T19:28:25.203 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.198+0000 7f540ec65700 1 -- 172.21.15.167:0/803679881 <== osd.0 v2:172.21.15.118:6800/27690 18 ==== osd_op_reply(70 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f5400022040 con 0x7f53f8011040 2023-10-05T19:28:25.203 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.198+0000 7f540ec65700 1 -- 172.21.15.167:0/803679881 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:71 5.1b 5:d8000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x55d84234ab30 con 0x7f5408007030 2023-10-05T19:28:25.203 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.198+0000 7f540dc63700 1 -- 172.21.15.167:0/803679881 <== osd.2 v2:172.21.15.167:6816/30037 11 ==== osd_op_reply(71 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f53f8003450 con 0x7f5408007030 2023-10-05T19:28:25.203 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.198+0000 7f540dc63700 1 -- 172.21.15.167:0/803679881 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:72 5.7 5:e0000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x55d84234ab30 con 0x55d84234c190 2023-10-05T19:28:25.204 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.198+0000 7f540e464700 1 -- 172.21.15.167:0/803679881 <== osd.3 v2:172.21.15.167:6808/30036 19 ==== osd_op_reply(72 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f540801cc30 con 0x55d84234c190 2023-10-05T19:28:25.204 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.198+0000 7f540e464700 1 -- 172.21.15.167:0/803679881 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:73 5.17 5:e8000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x55d84234ab30 con 0x7f53f8011040 2023-10-05T19:28:25.205 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.198+0000 7f540ec65700 1 -- 172.21.15.167:0/803679881 <== osd.0 v2:172.21.15.118:6800/27690 19 ==== osd_op_reply(73 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f5400022040 con 0x7f53f8011040 2023-10-05T19:28:25.205 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.198+0000 7f540ec65700 1 -- 172.21.15.167:0/803679881 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:74 5.f 5:f0000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x55d84234ab30 con 0x55d84228abf0 2023-10-05T19:28:25.206 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.202+0000 7f540ec65700 1 -- 172.21.15.167:0/803679881 <== osd.1 v2:172.21.15.167:6800/30035 25 ==== osd_op_reply(74 [pgnls start_epoch 30 out=115b] v23'2 uv2 ondisk = 1) v8 ==== 144+0+115 (crc 0 0 0) 0x7f5400022040 con 0x55d84228abf0 2023-10-05T19:28:25.206 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.202+0000 7f540ec65700 1 -- 172.21.15.167:0/803679881 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:75 5.1f 5:f8000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x55d84234ab30 con 0x7f5408007030 2023-10-05T19:28:25.207 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.202+0000 7f540dc63700 1 -- 172.21.15.167:0/803679881 <== osd.2 v2:172.21.15.167:6816/30037 12 ==== osd_op_reply(75 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f53f8003450 con 0x7f5408007030 2023-10-05T19:28:25.207 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.202+0000 7f541092fe40 20 RGWRados::pool_iterate: got zonegroup_info.4f1f719d-9648-417b-b10c-e5d939825419 2023-10-05T19:28:25.207 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.202+0000 7f541092fe40 20 RGWRados::pool_iterate: got zone_info.9a9df965-6612-45d4-a45f-6e3c889bacc2 2023-10-05T19:28:25.208 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.202+0000 7f541092fe40 20 RGWRados::pool_iterate: got zone_names.default 2023-10-05T19:28:25.208 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.202+0000 7f541092fe40 20 RGWRados::pool_iterate: got zonegroups_names.default 2023-10-05T19:28:25.208 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.202+0000 7f541092fe40 20 rados->read ofs=0 len=0 2023-10-05T19:28:25.208 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.202+0000 7f541092fe40 1 -- 172.21.15.167:0/803679881 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:76 5.f 5:f43b6ece:::zone_names.default:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x55d84234f750 con 0x55d84228abf0 2023-10-05T19:28:25.208 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.202+0000 7f540ec65700 1 -- 172.21.15.167:0/803679881 <== osd.1 v2:172.21.15.167:6800/30035 26 ==== osd_op_reply(76 zone_names.default [read 0~46 out=46b] v0'0 uv1 ondisk = 0) v8 ==== 162+0+46 (crc 0 0 0) 0x7f5400022040 con 0x55d84228abf0 2023-10-05T19:28:25.209 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.202+0000 7f541092fe40 20 rados_obj.operate() r=0 bl.length=46 2023-10-05T19:28:25.209 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.202+0000 7f541092fe40 20 rados->read ofs=0 len=0 2023-10-05T19:28:25.209 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.202+0000 7f541092fe40 1 -- 172.21.15.167:0/803679881 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:77 5.1d 5:b8374eb4:::zone_info.9a9df965-6612-45d4-a45f-6e3c889bacc2:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x55d84234fb00 con 0x55d84228abf0 2023-10-05T19:28:25.209 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.202+0000 7f540ec65700 1 -- 172.21.15.167:0/803679881 <== osd.1 v2:172.21.15.167:6800/30035 27 ==== osd_op_reply(77 zone_info.9a9df965-6612-45d4-a45f-6e3c889bacc2 [read 0~874 out=874b] v0'0 uv2 ondisk = 0) v8 ==== 190+0+874 (crc 0 0 0) 0x7f5400022040 con 0x55d84228abf0 2023-10-05T19:28:25.209 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.202+0000 7f541092fe40 20 rados_obj.operate() r=0 bl.length=874 2023-10-05T19:28:25.209 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.202+0000 7f541092fe40 20 rados->read ofs=0 len=0 2023-10-05T19:28:25.209 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.202+0000 7f541092fe40 1 -- 172.21.15.167:0/803679881 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:78 5.f 5:f4c53578:::zonegroups_names.default:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x55d84234fff0 con 0x55d84228abf0 2023-10-05T19:28:25.209 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.202+0000 7f540ec65700 1 -- 172.21.15.167:0/803679881 <== osd.1 v2:172.21.15.167:6800/30035 28 ==== osd_op_reply(78 zonegroups_names.default [read 0~46 out=46b] v0'0 uv2 ondisk = 0) v8 ==== 168+0+46 (crc 0 0 0) 0x7f5400022040 con 0x55d84228abf0 2023-10-05T19:28:25.210 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.202+0000 7f541092fe40 20 rados_obj.operate() r=0 bl.length=46 2023-10-05T19:28:25.210 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.202+0000 7f541092fe40 20 rados->read ofs=0 len=0 2023-10-05T19:28:25.210 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.202+0000 7f541092fe40 1 -- 172.21.15.167:0/803679881 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:79 5.c 5:30d299a5:::zonegroup_info.4f1f719d-9648-417b-b10c-e5d939825419:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x55d84234fff0 con 0x7f53f8011040 2023-10-05T19:28:25.210 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.206+0000 7f540ec65700 1 -- 172.21.15.167:0/803679881 <== osd.0 v2:172.21.15.118:6800/27690 20 ==== osd_op_reply(79 zonegroup_info.4f1f719d-9648-417b-b10c-e5d939825419 [read 0~362 out=362b] v0'0 uv1 ondisk = 0) v8 ==== 195+0+362 (crc 0 0 0) 0x7f5400045040 con 0x7f53f8011040 2023-10-05T19:28:25.211 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.206+0000 7f541092fe40 20 rados_obj.operate() r=0 bl.length=362 2023-10-05T19:28:25.211 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.206+0000 7f541092fe40 20 rados->read ofs=0 len=0 2023-10-05T19:28:25.211 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.206+0000 7f541092fe40 1 -- 172.21.15.167:0/803679881 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:80 5.3 5:c0ad8cbd:::region_map:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x55d8423500f0 con 0x7f53f8011040 2023-10-05T19:28:25.211 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.206+0000 7f540ec65700 1 -- 172.21.15.167:0/803679881 <== osd.0 v2:172.21.15.118:6800/27690 21 ==== osd_op_reply(80 region_map [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 ==== 154+0+0 (crc 0 0 0) 0x7f5400045040 con 0x7f53f8011040 2023-10-05T19:28:25.211 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.206+0000 7f541092fe40 20 rados_obj.operate() r=-2 bl.length=0 2023-10-05T19:28:25.211 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.206+0000 7f541092fe40 20 zone default found 2023-10-05T19:28:25.212 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.206+0000 7f541092fe40 4 Realm: () 2023-10-05T19:28:25.212 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.206+0000 7f541092fe40 4 ZoneGroup: default (4f1f719d-9648-417b-b10c-e5d939825419) 2023-10-05T19:28:25.212 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.206+0000 7f541092fe40 4 Zone: default (9a9df965-6612-45d4-a45f-6e3c889bacc2) 2023-10-05T19:28:25.212 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.206+0000 7f541092fe40 10 cannot find current period zonegroup using local zonegroup configuration 2023-10-05T19:28:25.212 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.206+0000 7f541092fe40 20 zonegroup default 2023-10-05T19:28:25.212 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.206+0000 7f541092fe40 20 rados->read ofs=0 len=0 2023-10-05T19:28:25.212 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.206+0000 7f541092fe40 1 -- 172.21.15.167:0/803679881 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:81 5.3 5:c52100b6:::period_config.default:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x55d8423500f0 con 0x7f53f8011040 2023-10-05T19:28:25.213 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.206+0000 7f540ec65700 1 -- 172.21.15.167:0/803679881 <== osd.0 v2:172.21.15.118:6800/27690 22 ==== osd_op_reply(81 period_config.default [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 ==== 165+0+0 (crc 0 0 0) 0x7f5400045040 con 0x7f53f8011040 2023-10-05T19:28:25.213 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.206+0000 7f541092fe40 20 rados_obj.operate() r=-2 bl.length=0 2023-10-05T19:28:25.213 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.206+0000 7f541092fe40 20 rados->read ofs=0 len=0 2023-10-05T19:28:25.214 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.206+0000 7f541092fe40 1 -- 172.21.15.167:0/803679881 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:82 6.2 6:4347d321:::bucket.sync-source-hints.:head [call version.read in=11b,read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x55d842351a70 con 0x55d84234c190 2023-10-05T19:28:25.214 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.206+0000 7f540e464700 1 -- 172.21.15.167:0/803679881 <== osd.3 v2:172.21.15.167:6808/30036 20 ==== osd_op_reply(82 bucket.sync-source-hints. [call,read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 ==== 211+0+0 (crc 0 0 0) 0x7f540801cc30 con 0x55d84234c190 2023-10-05T19:28:25.214 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.206+0000 7f541092fe40 20 rados_obj.operate() r=-2 bl.length=0 2023-10-05T19:28:25.214 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.206+0000 7f541092fe40 20 rados->read ofs=0 len=0 2023-10-05T19:28:25.214 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.206+0000 7f541092fe40 1 -- 172.21.15.167:0/803679881 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:83 6.b 6:d467b91b:::bucket.sync-target-hints.:head [call version.read in=11b,read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x55d842351e20 con 0x55d84234c190 2023-10-05T19:28:25.214 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.206+0000 7f540e464700 1 -- 172.21.15.167:0/803679881 <== osd.3 v2:172.21.15.167:6808/30036 21 ==== osd_op_reply(83 bucket.sync-target-hints. [call,read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 ==== 211+0+0 (crc 0 0 0) 0x7f540801cc30 con 0x55d84234c190 2023-10-05T19:28:25.214 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.206+0000 7f541092fe40 20 rados_obj.operate() r=-2 bl.length=0 2023-10-05T19:28:25.214 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.206+0000 7f541092fe40 20 started sync module instance, tier type = 2023-10-05T19:28:25.214 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.206+0000 7f541092fe40 20 started zone id=9a9df965-6612-45d4-a45f-6e3c889bacc2 (name=default) with tier type = 2023-10-05T19:28:25.214 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.206+0000 7f541092fe40 1 -- 172.21.15.167:0/803679881 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:84 7.1f 7:f95f44c2:::notify.0:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x55d842351130 con 0x7f5408007030 2023-10-05T19:28:25.216 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.210+0000 7f540dc63700 1 -- 172.21.15.167:0/803679881 <== osd.2 v2:172.21.15.167:6816/30037 13 ==== osd_op_reply(84 notify.0 [create] v30'18 uv18 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f53f802f040 con 0x7f5408007030 2023-10-05T19:28:25.216 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.210+0000 7f541092fe40 1 -- 172.21.15.167:0/803679881 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:85 7.1f 7:f95f44c2:::notify.0:head [watch watch cookie 94387312069696] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x55d8423521d0 con 0x7f5408007030 2023-10-05T19:28:25.217 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.210+0000 7f541092fe40 1 -- 172.21.15.167:0/803679881 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:86 7.0 7:05bf5b68:::notify.1:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x55d842352bc0 con 0x55d84228abf0 2023-10-05T19:28:25.219 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.214+0000 7f540dc63700 1 -- 172.21.15.167:0/803679881 <== osd.2 v2:172.21.15.167:6816/30037 14 ==== osd_op_reply(85 notify.0 [watch watch cookie 94387312069696] v30'19 uv18 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f53f802f040 con 0x7f5408007030 2023-10-05T19:28:25.220 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.214+0000 7f540ec65700 1 -- 172.21.15.167:0/803679881 <== osd.1 v2:172.21.15.167:6800/30035 29 ==== osd_op_reply(86 notify.1 [create] v30'18 uv18 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f5400045040 con 0x55d84228abf0 2023-10-05T19:28:25.220 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.214+0000 7f541092fe40 1 -- 172.21.15.167:0/803679881 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:87 7.0 7:05bf5b68:::notify.1:head [watch watch cookie 94387312076432] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x55d842352fc0 con 0x55d84228abf0 2023-10-05T19:28:25.220 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.214+0000 7f541092fe40 1 -- 172.21.15.167:0/803679881 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:88 7.15 7:a93a5511:::notify.2:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x55d842353860 con 0x55d84234c190 2023-10-05T19:28:25.223 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.218+0000 7f540ec65700 1 -- 172.21.15.167:0/803679881 <== osd.1 v2:172.21.15.167:6800/30035 30 ==== osd_op_reply(87 notify.1 [watch watch cookie 94387312076432] v30'19 uv18 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f5400045040 con 0x55d84228abf0 2023-10-05T19:28:25.223 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.218+0000 7f540e464700 1 -- 172.21.15.167:0/803679881 <== osd.3 v2:172.21.15.167:6808/30036 22 ==== osd_op_reply(88 notify.2 [create] v30'18 uv18 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f540801cc30 con 0x55d84234c190 2023-10-05T19:28:25.224 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.218+0000 7f541092fe40 1 -- 172.21.15.167:0/803679881 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:89 7.15 7:a93a5511:::notify.2:head [watch watch cookie 94387312072192] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x55d8423534b0 con 0x55d84234c190 2023-10-05T19:28:25.224 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.218+0000 7f541092fe40 1 -- 172.21.15.167:0/803679881 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:90 7.e 7:7759931f:::notify.3:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x55d8423543a0 con 0x7f5408007030 2023-10-05T19:28:25.226 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.222+0000 7f540dc63700 1 -- 172.21.15.167:0/803679881 <== osd.2 v2:172.21.15.167:6816/30037 15 ==== osd_op_reply(90 notify.3 [create] v30'18 uv18 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f53f802f040 con 0x7f5408007030 2023-10-05T19:28:25.226 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.222+0000 7f541092fe40 1 -- 172.21.15.167:0/803679881 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:91 7.e 7:7759931f:::notify.3:head [watch watch cookie 94387312084896] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x55d8423550e0 con 0x7f5408007030 2023-10-05T19:28:25.226 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.222+0000 7f541092fe40 1 -- 172.21.15.167:0/803679881 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:92 7.d 7:b4812045:::notify.4:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x55d8423559c0 con 0x55d84228abf0 2023-10-05T19:28:25.227 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.222+0000 7f540e464700 1 -- 172.21.15.167:0/803679881 <== osd.3 v2:172.21.15.167:6808/30036 23 ==== osd_op_reply(89 notify.2 [watch watch cookie 94387312072192] v30'19 uv18 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f540801cc30 con 0x55d84234c190 2023-10-05T19:28:25.228 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.222+0000 7f540dc63700 1 -- 172.21.15.167:0/803679881 <== osd.2 v2:172.21.15.167:6816/30037 16 ==== osd_op_reply(91 notify.3 [watch watch cookie 94387312084896] v30'19 uv18 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f53f802f040 con 0x7f5408007030 2023-10-05T19:28:25.229 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.222+0000 7f540ec65700 1 -- 172.21.15.167:0/803679881 <== osd.1 v2:172.21.15.167:6800/30035 31 ==== osd_op_reply(92 notify.4 [create] v30'18 uv18 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f5400045040 con 0x55d84228abf0 2023-10-05T19:28:25.229 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.222+0000 7f541092fe40 1 -- 172.21.15.167:0/803679881 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:93 7.d 7:b4812045:::notify.4:head [watch watch cookie 94387312086000] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x55d842352f70 con 0x55d84228abf0 2023-10-05T19:28:25.229 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.222+0000 7f541092fe40 1 -- 172.21.15.167:0/803679881 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:94 7.3 7:c609908c:::notify.5:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x55d8423562b0 con 0x7f53f8011040 2023-10-05T19:28:25.231 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.226+0000 7f540ec65700 1 -- 172.21.15.167:0/803679881 <== osd.0 v2:172.21.15.118:6800/27690 23 ==== osd_op_reply(94 notify.5 [create] v30'18 uv18 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f5400045040 con 0x7f53f8011040 2023-10-05T19:28:25.232 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.226+0000 7f541092fe40 1 -- 172.21.15.167:0/803679881 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:95 7.3 7:c609908c:::notify.5:head [watch watch cookie 94387312090480] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x55d8423566b0 con 0x7f53f8011040 2023-10-05T19:28:25.232 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.226+0000 7f541092fe40 1 -- 172.21.15.167:0/803679881 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:96 7.14 7:2b04a3e9:::notify.6:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x55d842357190 con 0x7f53f8011040 2023-10-05T19:28:25.232 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.226+0000 7f540ec65700 1 -- 172.21.15.167:0/803679881 <== osd.1 v2:172.21.15.167:6800/30035 32 ==== osd_op_reply(93 notify.4 [watch watch cookie 94387312086000] v30'19 uv18 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f5400045040 con 0x55d84228abf0 2023-10-05T19:28:25.234 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.230+0000 7f540ec65700 1 -- 172.21.15.167:0/803679881 <== osd.0 v2:172.21.15.118:6800/27690 24 ==== osd_op_reply(95 notify.5 [watch watch cookie 94387312090480] v30'19 uv18 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f5400045040 con 0x7f53f8011040 2023-10-05T19:28:25.235 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.230+0000 7f540ec65700 1 -- 172.21.15.167:0/803679881 <== osd.0 v2:172.21.15.118:6800/27690 25 ==== osd_op_reply(96 notify.6 [create] v30'18 uv18 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f5400045040 con 0x7f53f8011040 2023-10-05T19:28:25.235 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.230+0000 7f541092fe40 1 -- 172.21.15.167:0/803679881 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:97 7.14 7:2b04a3e9:::notify.6:head [watch watch cookie 94387312088208] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x55d8423561c0 con 0x7f53f8011040 2023-10-05T19:28:25.235 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.230+0000 7f541092fe40 1 -- 172.21.15.167:0/803679881 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:98 7.9 7:93e5b521:::notify.7:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x55d8423578b0 con 0x7f53f8011040 2023-10-05T19:28:25.238 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.230+0000 7f540ec65700 1 -- 172.21.15.167:0/803679881 <== osd.0 v2:172.21.15.118:6800/27690 26 ==== osd_op_reply(97 notify.6 [watch watch cookie 94387312088208] v30'19 uv18 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f5400045040 con 0x7f53f8011040 2023-10-05T19:28:25.238 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.234+0000 7f540ec65700 1 -- 172.21.15.167:0/803679881 <== osd.0 v2:172.21.15.118:6800/27690 27 ==== osd_op_reply(98 notify.7 [create] v30'18 uv18 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f5400045040 con 0x7f53f8011040 2023-10-05T19:28:25.238 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.234+0000 7f541092fe40 1 -- 172.21.15.167:0/803679881 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:99 7.9 7:93e5b521:::notify.7:head [watch watch cookie 94387312094304] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x55d8423561c0 con 0x7f53f8011040 2023-10-05T19:28:25.238 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.234+0000 7f541092fe40 20 add_watcher() i=0 2023-10-05T19:28:25.238 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.234+0000 7f541092fe40 20 add_watcher() i=1 2023-10-05T19:28:25.239 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.234+0000 7f541092fe40 20 add_watcher() i=2 2023-10-05T19:28:25.239 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.234+0000 7f541092fe40 20 add_watcher() i=3 2023-10-05T19:28:25.239 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.234+0000 7f541092fe40 20 add_watcher() i=4 2023-10-05T19:28:25.239 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.234+0000 7f541092fe40 20 add_watcher() i=5 2023-10-05T19:28:25.239 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.234+0000 7f541092fe40 20 add_watcher() i=6 2023-10-05T19:28:25.240 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.234+0000 7f540ec65700 1 -- 172.21.15.167:0/803679881 <== osd.0 v2:172.21.15.118:6800/27690 28 ==== osd_op_reply(99 notify.7 [watch watch cookie 94387312094304] v30'19 uv18 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f5400045040 con 0x7f53f8011040 2023-10-05T19:28:25.240 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.234+0000 7f541092fe40 20 add_watcher() i=7 2023-10-05T19:28:25.240 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.234+0000 7f541092fe40 2 all 8 watchers are set, enabling cache 2023-10-05T19:28:25.241 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.234+0000 7f541092fe40 1 -- 172.21.15.167:0/803679881 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:100 6.4 6:22d26bf9:::data_loggenerations_metadata:head [call version.check_conds in=74b,call version.read in=11b,read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x55d84235a940 con 0x55d84228abf0 2023-10-05T19:28:25.241 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.234+0000 7f540ec65700 1 -- 172.21.15.167:0/803679881 <== osd.1 v2:172.21.15.167:6800/30035 33 ==== osd_op_reply(100 data_loggenerations_metadata [call,call out=48b,read 0~28 out=28b] v0'0 uv1 ondisk = 0) v8 ==== 256+0+76 (crc 0 0 0) 0x7f5400045040 con 0x55d84228abf0 2023-10-05T19:28:25.241 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.234+0000 7f541092fe40 1 -- 172.21.15.167:0/803679881 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:101 6.4 6:22d26bf9:::data_loggenerations_metadata:head [watch watch cookie 94387312096192] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x55d842357e00 con 0x55d84228abf0 2023-10-05T19:28:25.243 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.238+0000 7f540ec65700 1 -- 172.21.15.167:0/803679881 <== osd.1 v2:172.21.15.167:6800/30035 34 ==== osd_op_reply(101 data_loggenerations_metadata [watch watch cookie 94387312096192] v30'46 uv1 ondisk = 0) v8 ==== 172+0+0 (crc 0 0 0) 0x7f5400045040 con 0x55d84228abf0 2023-10-05T19:28:25.244 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.238+0000 7f53dffdf700 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2023-10-05T19:28:25.244 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.238+0000 7f541092fe40 20 check_secure_mon_conn(): auth registy supported: methods=[2] modes=[2,1] 2023-10-05T19:28:25.244 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.238+0000 7f541092fe40 20 check_secure_mon_conn(): mode 1 is insecure 2023-10-05T19:28:25.244 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.238+0000 7f541092fe40 1 Processor -- start 2023-10-05T19:28:25.245 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.238+0000 7f541092fe40 1 -- start start 2023-10-05T19:28:25.245 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.238+0000 7f541092fe40 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55d842364d70 0x55d842365140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-05T19:28:25.245 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.238+0000 7f541092fe40 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d842365680 con 0x55d842364d70 2023-10-05T19:28:25.245 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.238+0000 7f540dc63700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55d842364d70 0x55d842365140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-05T19:28:25.245 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.238+0000 7f540dc63700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55d842364d70 0x55d842365140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.118:3300/0 says I am v2:172.21.15.167:57090/0 (socket says 172.21.15.167:57090) 2023-10-05T19:28:25.245 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.238+0000 7f540dc63700 1 -- 172.21.15.167:0/4060975404 learned_addr learned my addr 172.21.15.167:0/4060975404 (peer_addr_for_me v2:172.21.15.167:0/0) 2023-10-05T19:28:25.246 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.238+0000 7f540dc63700 1 -- 172.21.15.167:0/4060975404 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55d8423657c0 con 0x55d842364d70 2023-10-05T19:28:25.246 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.242+0000 7f540dc63700 1 --2- 172.21.15.167:0/4060975404 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55d842364d70 0x55d842365140 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x55d842349c80 tx=0x7f53f8006000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=53a6f2cb9c81adba server_cookie=0 in_seq=0 out_seq=0 2023-10-05T19:28:25.246 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.242+0000 7f53df7de700 1 -- 172.21.15.167:0/4060975404 <== mon.0 v2:172.21.15.118:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f53f802c040 con 0x55d842364d70 2023-10-05T19:28:25.247 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.242+0000 7f53df7de700 1 -- 172.21.15.167:0/4060975404 <== mon.0 v2:172.21.15.118:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f53f802d040 con 0x55d842364d70 2023-10-05T19:28:25.247 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.242+0000 7f53df7de700 1 -- 172.21.15.167:0/4060975404 <== mon.0 v2:172.21.15.118:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f53f80289c0 con 0x55d842364d70 2023-10-05T19:28:25.248 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.242+0000 7f541092fe40 1 -- 172.21.15.167:0/4060975404 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55d842364d70 msgr2=0x55d842365140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-05T19:28:25.248 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.242+0000 7f541092fe40 1 --2- 172.21.15.167:0/4060975404 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55d842364d70 0x55d842365140 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x55d842349c80 tx=0x7f53f8006000 comp rx=0 tx=0).stop 2023-10-05T19:28:25.248 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.242+0000 7f541092fe40 1 -- 172.21.15.167:0/4060975404 shutdown_connections 2023-10-05T19:28:25.248 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.242+0000 7f541092fe40 1 --2- 172.21.15.167:0/4060975404 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55d842364d70 0x55d842365140 unknown :-1 s=CLOSED pgs=174 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-05T19:28:25.248 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.242+0000 7f541092fe40 1 -- 172.21.15.167:0/4060975404 >> 172.21.15.167:0/4060975404 conn(0x55d842360130 msgr2=0x55d84235e2e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-05T19:28:25.248 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.242+0000 7f541092fe40 1 -- 172.21.15.167:0/4060975404 shutdown_connections 2023-10-05T19:28:25.248 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.242+0000 7f541092fe40 1 -- 172.21.15.167:0/4060975404 wait complete. 2023-10-05T19:28:25.249 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.242+0000 7f541092fe40 1 Processor -- start 2023-10-05T19:28:25.249 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.242+0000 7f541092fe40 1 -- start start 2023-10-05T19:28:25.249 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.242+0000 7f541092fe40 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55d842364d70 0x55d84236a8d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-05T19:28:25.249 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.242+0000 7f541092fe40 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d84236b8d0 con 0x55d842364d70 2023-10-05T19:28:25.249 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.242+0000 7f540dc63700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55d842364d70 0x55d84236a8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-05T19:28:25.249 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.242+0000 7f540dc63700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55d842364d70 0x55d84236a8d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.118:3300/0 says I am v2:172.21.15.167:57102/0 (socket says 172.21.15.167:57102) 2023-10-05T19:28:25.250 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.242+0000 7f540dc63700 1 -- 172.21.15.167:0/3217491777 learned_addr learned my addr 172.21.15.167:0/3217491777 (peer_addr_for_me v2:172.21.15.167:0/0) 2023-10-05T19:28:25.250 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.246+0000 7f540dc63700 1 -- 172.21.15.167:0/3217491777 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55d842261c00 con 0x55d842364d70 2023-10-05T19:28:25.250 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.246+0000 7f540dc63700 1 --2- 172.21.15.167:0/3217491777 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55d842364d70 0x55d84236a8d0 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7f53f8040040 tx=0x7f53f803c000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-05T19:28:25.251 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.246+0000 7f53ddfdb700 1 -- 172.21.15.167:0/3217491777 <== mon.0 v2:172.21.15.118:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f53f802c070 con 0x55d842364d70 2023-10-05T19:28:25.251 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.246+0000 7f53ddfdb700 1 -- 172.21.15.167:0/3217491777 <== mon.0 v2:172.21.15.118:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f53f801c040 con 0x55d842364d70 2023-10-05T19:28:25.251 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.246+0000 7f541092fe40 1 -- 172.21.15.167:0/3217491777 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x55d84236aed0 con 0x55d842364d70 2023-10-05T19:28:25.251 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.246+0000 7f53ddfdb700 1 -- 172.21.15.167:0/3217491777 <== mon.0 v2:172.21.15.118:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f53f80038a0 con 0x55d842364d70 2023-10-05T19:28:25.251 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.246+0000 7f541092fe40 1 -- 172.21.15.167:0/3217491777 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55d84236b370 con 0x55d842364d70 2023-10-05T19:28:25.252 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.246+0000 7f53ddfdb700 1 -- 172.21.15.167:0/3217491777 <== mon.0 v2:172.21.15.118:3300/0 4 ==== mgrmap(e 5) v1 ==== 40329+0+0 (secure 0 0 0) 0x7f53f8048420 con 0x55d842364d70 2023-10-05T19:28:25.253 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.246+0000 7f53ddfdb700 1 --2- 172.21.15.167:0/3217491777 >> [v2:172.21.15.167:6824/29889,v1:172.21.15.167:6825/29889] conn(0x7f5354030c50 0x7f5354033100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-05T19:28:25.253 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.246+0000 7f53ddfdb700 1 -- 172.21.15.167:0/3217491777 <== mon.0 v2:172.21.15.118:3300/0 5 ==== osd_map(30..30 src has 1..30) v4 ==== 6057+0+0 (secure 0 0 0) 0x7f53f8052c20 con 0x55d842364d70 2023-10-05T19:28:25.253 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.246+0000 7f540ec65700 1 --2- 172.21.15.167:0/3217491777 >> [v2:172.21.15.167:6824/29889,v1:172.21.15.167:6825/29889] conn(0x7f5354030c50 0x7f5354033100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-05T19:28:25.253 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.246+0000 7f541092fe40 5 note: GC not initialized 2023-10-05T19:28:25.253 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.246+0000 7f540ec65700 1 --2- 172.21.15.167:0/3217491777 >> [v2:172.21.15.167:6824/29889,v1:172.21.15.167:6825/29889] conn(0x7f5354030c50 0x7f5354033100 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x55d842356480 tx=0x7f5400000f90 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-05T19:28:25.255 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.250+0000 7f534effd700 20 reqs_thread_entry: start 2023-10-05T19:28:25.255 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.250+0000 7f541092fe40 20 init_complete bucket index max shards: 11 2023-10-05T19:28:25.255 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.250+0000 7f534cff9700 20 reqs_thread_entry: start 2023-10-05T19:28:25.256 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.250+0000 7f541092fe40 10 rgw notify: Started notification manager with: 1 workers 2023-10-05T19:28:25.256 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.250+0000 7f533effd700 20 rgw notify: INFO: next queues processing will happen at: Thu Oct 5 19:28:55 2023 2023-10-05T19:28:25.256 INFO:teuthology.orchestra.run.smithi167.stderr: 2023-10-05T19:28:25.256 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.250+0000 7f533e7fc700 20 reqs_thread_entry: start 2023-10-05T19:28:25.256 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.250+0000 7f541092fe40 1 -- 172.21.15.167:0/803679881 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:102 6.e 6:74cd889a:usage::usage.0:head [call rgw.user_usage_log_read in=60b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x55d8423d3990 con 0x55d84228abf0 2023-10-05T19:28:25.257 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.250+0000 7f540ec65700 1 -- 172.21.15.167:0/803679881 <== osd.1 v2:172.21.15.167:6800/30035 35 ==== osd_op_reply(102 usage.0 [call] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 ==== 151+0+0 (crc 0 0 0) 0x7f5400045040 con 0x55d84228abf0 2023-10-05T19:28:25.257 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.250+0000 7f541092fe40 1 -- 172.21.15.167:0/803679881 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:103 6.3 6:c0e78a64:usage::usage.1:head [call rgw.user_usage_log_read in=60b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x55d8423d3990 con 0x7f5408007030 2023-10-05T19:28:25.257 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.250+0000 7f540dc63700 1 -- 172.21.15.167:0/803679881 <== osd.2 v2:172.21.15.167:6816/30037 17 ==== osd_op_reply(103 usage.1 [call] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 ==== 151+0+0 (crc 0 0 0) 0x7f53f802f040 con 0x7f5408007030 2023-10-05T19:28:25.258 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.250+0000 7f541092fe40 1 -- 172.21.15.167:0/803679881 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:104 6.d 6:b564efa4:usage::usage.2:head [call rgw.user_usage_log_read in=60b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x55d8423d3d40 con 0x55d84228abf0 2023-10-05T19:28:25.258 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.254+0000 7f540ec65700 1 -- 172.21.15.167:0/803679881 <== osd.1 v2:172.21.15.167:6800/30035 36 ==== osd_op_reply(104 usage.2 [call] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 ==== 151+0+0 (crc 0 0 0) 0x7f5400045040 con 0x55d84228abf0 2023-10-05T19:28:25.259 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.254+0000 7f541092fe40 1 -- 172.21.15.167:0/803679881 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:105 6.2 6:42d64401:usage::usage.3:head [call rgw.user_usage_log_read in=60b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x55d8423d3d40 con 0x55d84234c190 2023-10-05T19:28:25.260 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.254+0000 7f540e464700 1 -- 172.21.15.167:0/803679881 <== osd.3 v2:172.21.15.167:6808/30036 24 ==== osd_op_reply(105 usage.3 [call] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 ==== 151+0+0 (crc 0 0 0) 0x7f540801cc30 con 0x55d84234c190 2023-10-05T19:28:25.260 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.254+0000 7f541092fe40 1 -- 172.21.15.167:0/803679881 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:106 6.7 6:e09b6f01:usage::usage.4:head [call rgw.user_usage_log_read in=60b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x55d8423d40f0 con 0x55d84234c190 2023-10-05T19:28:25.260 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.254+0000 7f540e464700 1 -- 172.21.15.167:0/803679881 <== osd.3 v2:172.21.15.167:6808/30036 25 ==== osd_op_reply(106 usage.4 [call] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 ==== 151+0+0 (crc 0 0 0) 0x7f540801cc30 con 0x55d84234c190 2023-10-05T19:28:25.260 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.254+0000 7f541092fe40 1 -- 172.21.15.167:0/803679881 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:107 6.6 6:672810f1:usage::usage.5:head [call rgw.user_usage_log_read in=60b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x55d8423d40f0 con 0x55d84234c190 2023-10-05T19:28:25.261 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.254+0000 7f540e464700 1 -- 172.21.15.167:0/803679881 <== osd.3 v2:172.21.15.167:6808/30036 26 ==== osd_op_reply(107 usage.5 [call] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 ==== 151+0+0 (crc 0 0 0) 0x7f540801cc30 con 0x55d84234c190 2023-10-05T19:28:25.261 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.254+0000 7f541092fe40 1 -- 172.21.15.167:0/803679881 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:108 6.4 6:25a86931:usage::usage.6:head [call rgw.user_usage_log_read in=60b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x55d8423d40f0 con 0x55d84228abf0 2023-10-05T19:28:25.261 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.254+0000 7f540ec65700 1 -- 172.21.15.167:0/803679881 <== osd.1 v2:172.21.15.167:6800/30035 37 ==== osd_op_reply(108 usage.6 [call] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 ==== 151+0+0 (crc 0 0 0) 0x7f5400045040 con 0x55d84228abf0 2023-10-05T19:28:25.262 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.254+0000 7f541092fe40 1 -- 172.21.15.167:0/803679881 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:109 6.17 6:ebbe7322:usage::usage.7:head [call rgw.user_usage_log_read in=60b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x55d8423d40f0 con 0x55d84228abf0 2023-10-05T19:28:25.262 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.254+0000 7f540ec65700 1 -- 172.21.15.167:0/803679881 <== osd.1 v2:172.21.15.167:6800/30035 38 ==== osd_op_reply(109 usage.7 [call] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 ==== 151+0+0 (crc 0 0 0) 0x7f5400045040 con 0x55d84228abf0 2023-10-05T19:28:25.262 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.254+0000 7f541092fe40 1 -- 172.21.15.167:0/803679881 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:110 6.6 6:607e1922:usage::usage.8:head [call rgw.user_usage_log_read in=60b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x55d8423d40f0 con 0x55d84234c190 2023-10-05T19:28:25.262 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.258+0000 7f540e464700 1 -- 172.21.15.167:0/803679881 <== osd.3 v2:172.21.15.167:6808/30036 27 ==== osd_op_reply(110 usage.8 [call] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 ==== 151+0+0 (crc 0 0 0) 0x7f540801cc30 con 0x55d84234c190 2023-10-05T19:28:25.262 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.258+0000 7f541092fe40 1 -- 172.21.15.167:0/803679881 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:111 6.b 6:d7809252:usage::usage.9:head [call rgw.user_usage_log_read in=60b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x55d8423d40f0 con 0x55d84234c190 2023-10-05T19:28:25.263 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.258+0000 7f540e464700 1 -- 172.21.15.167:0/803679881 <== osd.3 v2:172.21.15.167:6808/30036 28 ==== osd_op_reply(111 usage.9 [call] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 ==== 151+0+0 (crc 0 0 0) 0x7f540801cc30 con 0x55d84234c190 2023-10-05T19:28:25.263 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.258+0000 7f541092fe40 1 -- 172.21.15.167:0/803679881 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:112 6.13 6:cea599e4:usage::usage.10:head [call rgw.user_usage_log_read in=60b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x55d8423d40f0 con 0x7f53f8011040 2023-10-05T19:28:25.263 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.258+0000 7f540ec65700 1 -- 172.21.15.167:0/803679881 <== osd.0 v2:172.21.15.118:6800/27690 29 ==== osd_op_reply(112 usage.10 [call] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 ==== 152+0+0 (crc 0 0 0) 0x7f5400045040 con 0x7f53f8011040 2023-10-05T19:28:25.263 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.258+0000 7f541092fe40 1 -- 172.21.15.167:0/803679881 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:113 6.4 6:21f845ae:usage::usage.11:head [call rgw.user_usage_log_read in=60b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x55d8423d40f0 con 0x55d84228abf0 2023-10-05T19:28:25.264 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.258+0000 7f540ec65700 1 -- 172.21.15.167:0/803679881 <== osd.1 v2:172.21.15.167:6800/30035 39 ==== osd_op_reply(113 usage.11 [call] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 ==== 152+0+0 (crc 0 0 0) 0x7f5400045040 con 0x55d84228abf0 2023-10-05T19:28:25.264 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.258+0000 7f541092fe40 1 -- 172.21.15.167:0/803679881 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:114 6.f 6:f0386354:usage::usage.12:head [call rgw.user_usage_log_read in=60b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x55d8423d40f0 con 0x7f5408007030 2023-10-05T19:28:25.264 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.258+0000 7f540dc63700 1 -- 172.21.15.167:0/803679881 <== osd.2 v2:172.21.15.167:6816/30037 18 ==== osd_op_reply(114 usage.12 [call] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 ==== 152+0+0 (crc 0 0 0) 0x7f53f802f040 con 0x7f5408007030 2023-10-05T19:28:25.264 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.258+0000 7f541092fe40 1 -- 172.21.15.167:0/803679881 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:115 6.c 6:3735eec4:usage::usage.13:head [call rgw.user_usage_log_read in=60b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x55d8423d40f0 con 0x55d84234c190 2023-10-05T19:28:25.265 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.258+0000 7f540e464700 1 -- 172.21.15.167:0/803679881 <== osd.3 v2:172.21.15.167:6808/30036 29 ==== osd_op_reply(115 usage.13 [call] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 ==== 152+0+0 (crc 0 0 0) 0x7f540801cc30 con 0x55d84234c190 2023-10-05T19:28:25.265 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.258+0000 7f541092fe40 1 -- 172.21.15.167:0/803679881 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:116 6.18 6:1f520d0a:usage::usage.14:head [call rgw.user_usage_log_read in=60b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x55d8423d40f0 con 0x7f53f8011040 2023-10-05T19:28:25.266 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.258+0000 7f540ec65700 1 -- 172.21.15.167:0/803679881 <== osd.0 v2:172.21.15.118:6800/27690 30 ==== osd_op_reply(116 usage.14 [call] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 ==== 152+0+0 (crc 0 0 0) 0x7f5400045040 con 0x7f53f8011040 2023-10-05T19:28:25.266 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.262+0000 7f541092fe40 1 -- 172.21.15.167:0/803679881 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:117 6.0 6:029cee12:usage::usage.15:head [call rgw.user_usage_log_read in=60b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x55d8423d40f0 con 0x7f53f8011040 2023-10-05T19:28:25.266 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.262+0000 7f540ec65700 1 -- 172.21.15.167:0/803679881 <== osd.0 v2:172.21.15.118:6800/27690 31 ==== osd_op_reply(117 usage.15 [call] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 ==== 152+0+0 (crc 0 0 0) 0x7f5400045040 con 0x7f53f8011040 2023-10-05T19:28:25.266 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.262+0000 7f541092fe40 1 -- 172.21.15.167:0/803679881 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:118 6.1e 6:7f3e030e:usage::usage.16:head [call rgw.user_usage_log_read in=60b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x55d8423d40f0 con 0x55d84234c190 2023-10-05T19:28:25.267 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.262+0000 7f540e464700 1 -- 172.21.15.167:0/803679881 <== osd.3 v2:172.21.15.167:6808/30036 30 ==== osd_op_reply(118 usage.16 [call] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 ==== 152+0+0 (crc 0 0 0) 0x7f540801cc30 con 0x55d84234c190 2023-10-05T19:28:25.267 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.262+0000 7f541092fe40 1 -- 172.21.15.167:0/803679881 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:119 6.10 6:0fbdcb32:usage::usage.17:head [call rgw.user_usage_log_read in=60b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x55d8423d40f0 con 0x7f53f8011040 2023-10-05T19:28:25.268 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.262+0000 7f540ec65700 1 -- 172.21.15.167:0/803679881 <== osd.0 v2:172.21.15.118:6800/27690 32 ==== osd_op_reply(119 usage.17 [call] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 ==== 152+0+0 (crc 0 0 0) 0x7f5400045040 con 0x7f53f8011040 2023-10-05T19:28:25.268 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.262+0000 7f541092fe40 1 -- 172.21.15.167:0/803679881 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:120 6.19 6:993b03e9:usage::usage.18:head [call rgw.user_usage_log_read in=60b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x55d8423d40f0 con 0x55d84228abf0 2023-10-05T19:28:25.269 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.262+0000 7f540ec65700 1 -- 172.21.15.167:0/803679881 <== osd.1 v2:172.21.15.167:6800/30035 40 ==== osd_op_reply(120 usage.18 [call] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 ==== 152+0+0 (crc 0 0 0) 0x7f5400045040 con 0x55d84228abf0 2023-10-05T19:28:25.269 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.262+0000 7f541092fe40 1 -- 172.21.15.167:0/803679881 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:121 6.1a 6:58fcd270:usage::usage.19:head [call rgw.user_usage_log_read in=60b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x55d8423d40f0 con 0x55d84234c190 2023-10-05T19:28:25.269 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.262+0000 7f540e464700 1 -- 172.21.15.167:0/803679881 <== osd.3 v2:172.21.15.167:6808/30036 31 ==== osd_op_reply(121 usage.19 [call] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 ==== 152+0+0 (crc 0 0 0) 0x7f540801cc30 con 0x55d84234c190 2023-10-05T19:28:25.269 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.262+0000 7f541092fe40 1 -- 172.21.15.167:0/803679881 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:122 6.1f 6:fcd63d49:usage::usage.20:head [call rgw.user_usage_log_read in=60b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x55d8423d40f0 con 0x7f53f8011040 2023-10-05T19:28:25.270 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.266+0000 7f540ec65700 1 -- 172.21.15.167:0/803679881 <== osd.0 v2:172.21.15.118:6800/27690 33 ==== osd_op_reply(122 usage.20 [call] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 ==== 152+0+0 (crc 0 0 0) 0x7f5400045040 con 0x7f53f8011040 2023-10-05T19:28:25.270 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.266+0000 7f541092fe40 1 -- 172.21.15.167:0/803679881 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:123 6.5 6:a2c1a412:usage::usage.21:head [call rgw.user_usage_log_read in=60b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x55d8423d40f0 con 0x7f5408007030 2023-10-05T19:28:25.271 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.266+0000 7f540dc63700 1 -- 172.21.15.167:0/803679881 <== osd.2 v2:172.21.15.167:6816/30037 19 ==== osd_op_reply(123 usage.21 [call] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 ==== 152+0+0 (crc 0 0 0) 0x7f53f802f040 con 0x7f5408007030 2023-10-05T19:28:25.271 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.266+0000 7f541092fe40 1 -- 172.21.15.167:0/803679881 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:124 6.13 6:cdb71d68:usage::usage.22:head [call rgw.user_usage_log_read in=60b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x55d8423d40f0 con 0x7f53f8011040 2023-10-05T19:28:25.271 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.266+0000 7f540ec65700 1 -- 172.21.15.167:0/803679881 <== osd.0 v2:172.21.15.118:6800/27690 34 ==== osd_op_reply(124 usage.22 [call] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 ==== 152+0+0 (crc 0 0 0) 0x7f5400045040 con 0x7f53f8011040 2023-10-05T19:28:25.271 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.266+0000 7f541092fe40 1 -- 172.21.15.167:0/803679881 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:125 6.17 6:ef3bca03:usage::usage.23:head [call rgw.user_usage_log_read in=60b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x55d8423d40f0 con 0x55d84228abf0 2023-10-05T19:28:25.272 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.266+0000 7f540ec65700 1 -- 172.21.15.167:0/803679881 <== osd.1 v2:172.21.15.167:6800/30035 41 ==== osd_op_reply(125 usage.23 [call] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 ==== 152+0+0 (crc 0 0 0) 0x7f5400045040 con 0x55d84228abf0 2023-10-05T19:28:25.272 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.266+0000 7f541092fe40 1 -- 172.21.15.167:0/803679881 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:126 6.e 6:7412960a:usage::usage.24:head [call rgw.user_usage_log_read in=60b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x55d8423d40f0 con 0x55d84228abf0 2023-10-05T19:28:25.272 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.266+0000 7f540ec65700 1 -- 172.21.15.167:0/803679881 <== osd.1 v2:172.21.15.167:6800/30035 42 ==== osd_op_reply(126 usage.24 [call] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 ==== 152+0+0 (crc 0 0 0) 0x7f5400045040 con 0x55d84228abf0 2023-10-05T19:28:25.273 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.266+0000 7f541092fe40 1 -- 172.21.15.167:0/803679881 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:127 6.7 6:e62d1024:usage::usage.25:head [call rgw.user_usage_log_read in=60b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x55d8423d40f0 con 0x55d84234c190 2023-10-05T19:28:25.273 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.266+0000 7f540e464700 1 -- 172.21.15.167:0/803679881 <== osd.3 v2:172.21.15.167:6808/30036 32 ==== osd_op_reply(127 usage.25 [call] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 ==== 152+0+0 (crc 0 0 0) 0x7f540801cc30 con 0x55d84234c190 2023-10-05T19:28:25.273 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.266+0000 7f541092fe40 1 -- 172.21.15.167:0/803679881 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:128 6.d 6:b172cd59:usage::usage.26:head [call rgw.user_usage_log_read in=60b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x55d8423d40f0 con 0x55d84228abf0 2023-10-05T19:28:25.274 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.270+0000 7f540ec65700 1 -- 172.21.15.167:0/803679881 <== osd.1 v2:172.21.15.167:6800/30035 43 ==== osd_op_reply(128 usage.26 [call] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 ==== 152+0+0 (crc 0 0 0) 0x7f5400045040 con 0x55d84228abf0 2023-10-05T19:28:25.274 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.270+0000 7f541092fe40 1 -- 172.21.15.167:0/803679881 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:129 6.1 6:83912023:usage::usage.27:head [call rgw.user_usage_log_read in=60b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x55d8423d40f0 con 0x55d84228abf0 2023-10-05T19:28:25.274 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.270+0000 7f540ec65700 1 -- 172.21.15.167:0/803679881 <== osd.1 v2:172.21.15.167:6800/30035 44 ==== osd_op_reply(129 usage.27 [call] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 ==== 152+0+0 (crc 0 0 0) 0x7f5400045040 con 0x55d84228abf0 2023-10-05T19:28:25.275 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.270+0000 7f541092fe40 1 -- 172.21.15.167:0/803679881 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:130 6.1e 6:7c73bdb3:usage::usage.28:head [call rgw.user_usage_log_read in=60b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x55d8423d40f0 con 0x55d84234c190 2023-10-05T19:28:25.275 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.270+0000 7f540e464700 1 -- 172.21.15.167:0/803679881 <== osd.3 v2:172.21.15.167:6808/30036 33 ==== osd_op_reply(130 usage.28 [call] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 ==== 152+0+0 (crc 0 0 0) 0x7f540801cc30 con 0x55d84234c190 2023-10-05T19:28:25.275 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.270+0000 7f541092fe40 1 -- 172.21.15.167:0/803679881 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:131 6.16 6:6e59fa35:usage::usage.29:head [call rgw.user_usage_log_read in=60b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x55d8423d40f0 con 0x55d84234c190 2023-10-05T19:28:25.276 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.270+0000 7f540e464700 1 -- 172.21.15.167:0/803679881 <== osd.3 v2:172.21.15.167:6808/30036 34 ==== osd_op_reply(131 usage.29 [call] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 ==== 152+0+0 (crc 0 0 0) 0x7f540801cc30 con 0x55d84234c190 2023-10-05T19:28:25.276 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.270+0000 7f541092fe40 1 -- 172.21.15.167:0/803679881 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:132 6.1c 6:38fe6885:usage::usage.30:head [call rgw.user_usage_log_read in=60b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x55d8423d40f0 con 0x55d84234c190 2023-10-05T19:28:25.276 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.270+0000 7f540e464700 1 -- 172.21.15.167:0/803679881 <== osd.3 v2:172.21.15.167:6808/30036 35 ==== osd_op_reply(132 usage.30 [call] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 ==== 152+0+0 (crc 0 0 0) 0x7f540801cc30 con 0x55d84234c190 2023-10-05T19:28:25.276 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.270+0000 7f541092fe40 1 -- 172.21.15.167:0/803679881 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:133 6.1d 6:bb28dff0:usage::usage.31:head [call rgw.user_usage_log_read in=60b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x55d8423d40f0 con 0x7f53f8011040 2023-10-05T19:28:25.277 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.270+0000 7f540ec65700 1 -- 172.21.15.167:0/803679881 <== osd.0 v2:172.21.15.118:6800/27690 35 ==== osd_op_reply(133 usage.31 [call] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 ==== 152+0+0 (crc 0 0 0) 0x7f5400045040 con 0x7f53f8011040 2023-10-05T19:28:25.277 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.270+0000 7f541092fe40 1 -- 172.21.15.167:0/803679881 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:134 7.1f 7:f95f44c2:::notify.0:head [watch unwatch cookie 94387312069696] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x55d8423d40f0 con 0x7f5408007030 2023-10-05T19:28:25.280 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.274+0000 7f540dc63700 1 -- 172.21.15.167:0/803679881 <== osd.2 v2:172.21.15.167:6816/30037 20 ==== osd_op_reply(134 notify.0 [watch unwatch cookie 94387312069696] v30'20 uv18 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f53f802f040 con 0x7f5408007030 2023-10-05T19:28:25.280 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.274+0000 7f541092fe40 20 remove_watcher() i=0 2023-10-05T19:28:25.280 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.274+0000 7f541092fe40 2 removed watcher, disabling cache 2023-10-05T19:28:25.281 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.274+0000 7f541092fe40 1 -- 172.21.15.167:0/803679881 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:135 7.0 7:05bf5b68:::notify.1:head [watch unwatch cookie 94387312076432] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x55d8423d40f0 con 0x55d84228abf0 2023-10-05T19:28:25.283 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.278+0000 7f540ec65700 1 -- 172.21.15.167:0/803679881 <== osd.1 v2:172.21.15.167:6800/30035 45 ==== osd_op_reply(135 notify.1 [watch unwatch cookie 94387312076432] v30'20 uv18 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f5400045040 con 0x55d84228abf0 2023-10-05T19:28:25.283 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.278+0000 7f541092fe40 20 remove_watcher() i=1 2023-10-05T19:28:25.283 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.278+0000 7f541092fe40 1 -- 172.21.15.167:0/803679881 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:136 7.15 7:a93a5511:::notify.2:head [watch unwatch cookie 94387312072192] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x55d8423d40f0 con 0x55d84234c190 2023-10-05T19:28:25.286 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.282+0000 7f540e464700 1 -- 172.21.15.167:0/803679881 <== osd.3 v2:172.21.15.167:6808/30036 36 ==== osd_op_reply(136 notify.2 [watch unwatch cookie 94387312072192] v30'20 uv18 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f540801cc30 con 0x55d84234c190 2023-10-05T19:28:25.286 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.282+0000 7f541092fe40 20 remove_watcher() i=2 2023-10-05T19:28:25.286 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.282+0000 7f541092fe40 1 -- 172.21.15.167:0/803679881 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:137 7.e 7:7759931f:::notify.3:head [watch unwatch cookie 94387312084896] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x55d8423d40f0 con 0x7f5408007030 2023-10-05T19:28:25.289 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.282+0000 7f540dc63700 1 -- 172.21.15.167:0/803679881 <== osd.2 v2:172.21.15.167:6816/30037 21 ==== osd_op_reply(137 notify.3 [watch unwatch cookie 94387312084896] v30'20 uv18 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f53f802f040 con 0x7f5408007030 2023-10-05T19:28:25.289 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.282+0000 7f541092fe40 20 remove_watcher() i=3 2023-10-05T19:28:25.289 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.282+0000 7f541092fe40 1 -- 172.21.15.167:0/803679881 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:138 7.d 7:b4812045:::notify.4:head [watch unwatch cookie 94387312086000] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x55d8423d40f0 con 0x55d84228abf0 2023-10-05T19:28:25.291 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.286+0000 7f540ec65700 1 -- 172.21.15.167:0/803679881 <== osd.1 v2:172.21.15.167:6800/30035 46 ==== osd_op_reply(138 notify.4 [watch unwatch cookie 94387312086000] v30'20 uv18 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f5400045040 con 0x55d84228abf0 2023-10-05T19:28:25.292 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.286+0000 7f541092fe40 20 remove_watcher() i=4 2023-10-05T19:28:25.292 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.286+0000 7f541092fe40 1 -- 172.21.15.167:0/803679881 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:139 7.3 7:c609908c:::notify.5:head [watch unwatch cookie 94387312090480] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x55d8423d40f0 con 0x7f53f8011040 2023-10-05T19:28:25.294 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.290+0000 7f540ec65700 1 -- 172.21.15.167:0/803679881 <== osd.0 v2:172.21.15.118:6800/27690 36 ==== osd_op_reply(139 notify.5 [watch unwatch cookie 94387312090480] v30'20 uv18 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f5400045040 con 0x7f53f8011040 2023-10-05T19:28:25.294 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.290+0000 7f541092fe40 20 remove_watcher() i=5 2023-10-05T19:28:25.295 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.290+0000 7f541092fe40 1 -- 172.21.15.167:0/803679881 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:140 7.14 7:2b04a3e9:::notify.6:head [watch unwatch cookie 94387312088208] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x55d8423d40f0 con 0x7f53f8011040 2023-10-05T19:28:25.297 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.290+0000 7f540ec65700 1 -- 172.21.15.167:0/803679881 <== osd.0 v2:172.21.15.118:6800/27690 37 ==== osd_op_reply(140 notify.6 [watch unwatch cookie 94387312088208] v30'20 uv18 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f5400045040 con 0x7f53f8011040 2023-10-05T19:28:25.297 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.290+0000 7f541092fe40 20 remove_watcher() i=6 2023-10-05T19:28:25.297 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.290+0000 7f541092fe40 1 -- 172.21.15.167:0/803679881 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:141 7.9 7:93e5b521:::notify.7:head [watch unwatch cookie 94387312094304] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x55d8423d40f0 con 0x7f53f8011040 2023-10-05T19:28:25.300 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.294+0000 7f540ec65700 1 -- 172.21.15.167:0/803679881 <== osd.0 v2:172.21.15.118:6800/27690 38 ==== osd_op_reply(141 notify.7 [watch unwatch cookie 94387312094304] v30'20 uv18 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f5400045040 con 0x7f53f8011040 2023-10-05T19:28:25.300 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.294+0000 7f541092fe40 20 remove_watcher() i=7 2023-10-05T19:28:25.301 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.294+0000 7f53dffdf700 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2023-10-05T19:28:25.302 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.294+0000 7f541092fe40 1 -- 172.21.15.167:0/803679881 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:142 6.4 6:22d26bf9:::data_loggenerations_metadata:head [watch unwatch cookie 94387312096192] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x55d8423d40f0 con 0x55d84228abf0 2023-10-05T19:28:25.305 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.298+0000 7f540ec65700 1 -- 172.21.15.167:0/803679881 <== osd.1 v2:172.21.15.167:6800/30035 47 ==== osd_op_reply(142 data_loggenerations_metadata [watch unwatch cookie 94387312096192] v30'47 uv1 ondisk = 0) v8 ==== 172+0+0 (crc 0 0 0) 0x7f5400045040 con 0x55d84228abf0 2023-10-05T19:28:25.305 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.298+0000 7f541092fe40 1 -- 172.21.15.167:0/803679881 >> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] conn(0x7f53f8011040 msgr2=0x7f53f8015960 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-05T19:28:25.305 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.298+0000 7f541092fe40 1 --2- 172.21.15.167:0/803679881 >> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] conn(0x7f53f8011040 0x7f53f8015960 crc :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-05T19:28:25.305 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.298+0000 7f541092fe40 1 -- 172.21.15.167:0/803679881 >> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] conn(0x55d84228abf0 msgr2=0x55d842348780 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-05T19:28:25.305 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.298+0000 7f541092fe40 1 --2- 172.21.15.167:0/803679881 >> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] conn(0x55d84228abf0 0x55d842348780 crc :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-05T19:28:25.305 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.298+0000 7f541092fe40 1 -- 172.21.15.167:0/803679881 >> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] conn(0x7f5408007030 msgr2=0x7f540801c410 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-05T19:28:25.305 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.298+0000 7f541092fe40 1 --2- 172.21.15.167:0/803679881 >> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] conn(0x7f5408007030 0x7f540801c410 crc :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-05T19:28:25.306 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.298+0000 7f541092fe40 1 -- 172.21.15.167:0/803679881 >> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] conn(0x55d84234c190 msgr2=0x55d84234e570 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-05T19:28:25.306 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.298+0000 7f541092fe40 1 --2- 172.21.15.167:0/803679881 >> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] conn(0x55d84234c190 0x55d84234e570 crc :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-05T19:28:25.306 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.298+0000 7f541092fe40 1 -- 172.21.15.167:0/803679881 >> [v2:172.21.15.167:6824/29889,v1:172.21.15.167:6825/29889] conn(0x7f53f0030c50 msgr2=0x7f53f0033100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-05T19:28:25.306 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.298+0000 7f541092fe40 1 --2- 172.21.15.167:0/803679881 >> [v2:172.21.15.167:6824/29889,v1:172.21.15.167:6825/29889] conn(0x7f53f0030c50 0x7f53f0033100 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f53f80016f0 tx=0x7f53f80067f0 comp rx=0 tx=0).stop 2023-10-05T19:28:25.306 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.298+0000 7f540ec65700 1 -- 172.21.15.167:0/803679881 reap_dead start 2023-10-05T19:28:25.306 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.298+0000 7f541092fe40 1 -- 172.21.15.167:0/803679881 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55d842289710 msgr2=0x55d84233ccc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-05T19:28:25.306 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.298+0000 7f541092fe40 1 --2- 172.21.15.167:0/803679881 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55d842289710 0x55d84233ccc0 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7f540800a650 tx=0x7f540801c000 comp rx=0 tx=0).stop 2023-10-05T19:28:25.306 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.298+0000 7f541092fe40 1 -- 172.21.15.167:0/803679881 shutdown_connections 2023-10-05T19:28:25.306 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.298+0000 7f541092fe40 1 --2- 172.21.15.167:0/803679881 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55d842289710 0x55d84233ccc0 unknown :-1 s=CLOSED pgs=173 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-05T19:28:25.306 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.298+0000 7f541092fe40 1 -- 172.21.15.167:0/803679881 >> 172.21.15.167:0/803679881 conn(0x55d842310820 msgr2=0x55d8423306f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-05T19:28:25.306 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.298+0000 7f541092fe40 1 -- 172.21.15.167:0/803679881 shutdown_connections 2023-10-05T19:28:25.307 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.298+0000 7f541092fe40 1 -- 172.21.15.167:0/803679881 wait complete. 2023-10-05T19:28:25.308 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.298+0000 7f541092fe40 1 -- 172.21.15.167:0/3217491777 >> [v2:172.21.15.167:6824/29889,v1:172.21.15.167:6825/29889] conn(0x7f5354030c50 msgr2=0x7f5354033100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-05T19:28:25.308 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.298+0000 7f541092fe40 1 --2- 172.21.15.167:0/3217491777 >> [v2:172.21.15.167:6824/29889,v1:172.21.15.167:6825/29889] conn(0x7f5354030c50 0x7f5354033100 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x55d842356480 tx=0x7f5400000f90 comp rx=0 tx=0).stop 2023-10-05T19:28:25.308 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.298+0000 7f541092fe40 1 -- 172.21.15.167:0/3217491777 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55d842364d70 msgr2=0x55d84236a8d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-05T19:28:25.308 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.298+0000 7f541092fe40 1 --2- 172.21.15.167:0/3217491777 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55d842364d70 0x55d84236a8d0 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7f53f8040040 tx=0x7f53f803c000 comp rx=0 tx=0).stop 2023-10-05T19:28:25.308 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.302+0000 7f541092fe40 1 -- 172.21.15.167:0/3217491777 shutdown_connections 2023-10-05T19:28:25.308 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.302+0000 7f541092fe40 1 --2- 172.21.15.167:0/3217491777 >> [v2:172.21.15.167:6824/29889,v1:172.21.15.167:6825/29889] conn(0x7f5354030c50 0x7f5354033100 unknown :-1 s=CLOSED pgs=65 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-05T19:28:25.308 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.302+0000 7f541092fe40 1 --2- 172.21.15.167:0/3217491777 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55d842364d70 0x55d84236a8d0 unknown :-1 s=CLOSED pgs=175 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-05T19:28:25.308 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.302+0000 7f541092fe40 1 -- 172.21.15.167:0/3217491777 >> 172.21.15.167:0/3217491777 conn(0x55d842360130 msgr2=0x55d84235e230 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-05T19:28:25.308 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.302+0000 7f541092fe40 1 -- 172.21.15.167:0/3217491777 shutdown_connections 2023-10-05T19:28:25.308 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.302+0000 7f541092fe40 1 -- 172.21.15.167:0/3217491777 wait complete. 2023-10-05T19:28:25.309 INFO:teuthology.orchestra.run.smithi167.stdout:{ 2023-10-05T19:28:25.309 INFO:teuthology.orchestra.run.smithi167.stdout: "entries": [], 2023-10-05T19:28:25.309 INFO:teuthology.orchestra.run.smithi167.stdout: "summary": [] 2023-10-05T19:28:25.309 INFO:teuthology.orchestra.run.smithi167.stdout:} 2023-10-05T19:28:25.311 DEBUG:tasks.util.rgw: json result: {'entries': [], 'summary': []} 2023-10-05T19:28:25.312 INFO:tasks.util.rgw:rgwadmin: client.0 : ['user', 'info', '--uid', 'foo'] 2023-10-05T19:28:25.312 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'ceph', 'user', 'info', '--uid', 'foo'] 2023-10-05T19:28:25.312 DEBUG:teuthology.orchestra.run.smithi167:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster ceph user info --uid foo 2023-10-05T19:28:25.396 INFO:teuthology.orchestra.run.smithi167.stderr:ignoring --setuser ceph since I am not root 2023-10-05T19:28:25.396 INFO:teuthology.orchestra.run.smithi167.stderr:ignoring --setgroup ceph since I am not root 2023-10-05T19:28:25.397 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.390+0000 7fa4832e0e40 1 Processor -- start 2023-10-05T19:28:25.397 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.390+0000 7fa4832e0e40 1 -- start start 2023-10-05T19:28:25.397 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.390+0000 7fa4832e0e40 1 --2- >> v2:172.21.15.118:3300/0 conn(0x563c18a0a5f0 0x563c18a0c9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-05T19:28:25.397 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.390+0000 7fa4832e0e40 1 -- --> v1:172.21.15.118:6789/0 -- auth(proto 0 26 bytes epoch 0) v1 -- 0x563c18950bd0 con 0x563c18a06010 2023-10-05T19:28:25.397 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.390+0000 7fa4832e0e40 1 -- --> v2:172.21.15.118:3300/0 -- mon_getmap magic: 0 v1 -- 0x563c1890dee0 con 0x563c18a0a5f0 2023-10-05T19:28:25.398 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.390+0000 7fa480614700 1 --2- >> v2:172.21.15.118:3300/0 conn(0x563c18a0a5f0 0x563c18a0c9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-05T19:28:25.398 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.390+0000 7fa480e15700 1 --1- >> v1:172.21.15.118:6789/0 conn(0x563c18a06010 0x563c18a063e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.118:6789/0 says I am v1:172.21.15.167:56348/0 (socket says 172.21.15.167:56348) 2023-10-05T19:28:25.398 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.390+0000 7fa480e15700 1 -- 172.21.15.167:0/4093490210 learned_addr learned my addr 172.21.15.167:0/4093490210 (peer_addr_for_me v1:172.21.15.167:0/0) 2023-10-05T19:28:25.398 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.394+0000 7fa480614700 1 -- 172.21.15.167:0/4093490210 >> v1:172.21.15.118:6789/0 conn(0x563c18a06010 legacy=0x563c18a063e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-05T19:28:25.398 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.394+0000 7fa480614700 1 -- 172.21.15.167:0/4093490210 --> v2:172.21.15.118:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x563c18a06c20 con 0x563c18a0a5f0 2023-10-05T19:28:25.399 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.394+0000 7fa480614700 1 --2- 172.21.15.167:0/4093490210 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x563c18a0a5f0 0x563c18a0c9d0 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7fa47400a5d0 tx=0x7fa474005a50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=73248c887c7c374f server_cookie=0 in_seq=0 out_seq=0 2023-10-05T19:28:25.399 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.394+0000 7fa47fe13700 1 -- 172.21.15.167:0/4093490210 <== mon.0 v2:172.21.15.118:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fa4740086a0 con 0x563c18a0a5f0 2023-10-05T19:28:25.399 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.394+0000 7fa47fe13700 1 -- 172.21.15.167:0/4093490210 <== mon.0 v2:172.21.15.118:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa474008800 con 0x563c18a0a5f0 2023-10-05T19:28:25.399 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.394+0000 7fa47fe13700 1 -- 172.21.15.167:0/4093490210 <== mon.0 v2:172.21.15.118:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fa474008ad0 con 0x563c18a0a5f0 2023-10-05T19:28:25.400 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.394+0000 7fa4832e0e40 1 -- 172.21.15.167:0/4093490210 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x563c18a0a5f0 msgr2=0x563c18a0c9d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-05T19:28:25.400 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.394+0000 7fa4832e0e40 1 --2- 172.21.15.167:0/4093490210 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x563c18a0a5f0 0x563c18a0c9d0 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7fa47400a5d0 tx=0x7fa474005a50 comp rx=0 tx=0).stop 2023-10-05T19:28:25.400 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.394+0000 7fa4832e0e40 1 -- 172.21.15.167:0/4093490210 shutdown_connections 2023-10-05T19:28:25.400 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.394+0000 7fa4832e0e40 1 --2- 172.21.15.167:0/4093490210 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x563c18a0a5f0 0x563c18a0c9d0 unknown :-1 s=CLOSED pgs=177 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-05T19:28:25.400 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.394+0000 7fa4832e0e40 1 -- 172.21.15.167:0/4093490210 >> 172.21.15.167:0/4093490210 conn(0x563c18979860 msgr2=0x563c18979c60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-05T19:28:25.400 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.394+0000 7fa4832e0e40 1 -- 172.21.15.167:0/4093490210 shutdown_connections 2023-10-05T19:28:25.401 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.394+0000 7fa4832e0e40 1 -- 172.21.15.167:0/4093490210 wait complete. 2023-10-05T19:28:25.402 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.394+0000 7fa4832e0e40 1 Processor -- start 2023-10-05T19:28:25.402 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.394+0000 7fa4832e0e40 1 -- start start 2023-10-05T19:28:25.402 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.394+0000 7fa4832e0e40 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x563c18a06010 0x563c18a25190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-05T19:28:25.402 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.394+0000 7fa4832e0e40 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x563c188d1e30 con 0x563c18a06010 2023-10-05T19:28:25.402 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.394+0000 7fa480e15700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x563c18a06010 0x563c18a25190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-05T19:28:25.402 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.394+0000 7fa480e15700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x563c18a06010 0x563c18a25190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.118:3300/0 says I am v2:172.21.15.167:57118/0 (socket says 172.21.15.167:57118) 2023-10-05T19:28:25.402 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.394+0000 7fa480e15700 1 -- 172.21.15.167:0/4261513042 learned_addr learned my addr 172.21.15.167:0/4261513042 (peer_addr_for_me v2:172.21.15.167:0/0) 2023-10-05T19:28:25.402 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.394+0000 7fa480e15700 1 -- 172.21.15.167:0/4261513042 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa474005cf0 con 0x563c18a06010 2023-10-05T19:28:25.402 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.398+0000 7fa480e15700 1 --2- 172.21.15.167:0/4261513042 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x563c18a06010 0x563c18a25190 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7fa4780014a0 tx=0x7fa478006b90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e083c0b751b71254 server_cookie=0 in_seq=0 out_seq=0 2023-10-05T19:28:25.404 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.398+0000 7fa47ee11700 1 -- 172.21.15.167:0/4261513042 <== mon.0 v2:172.21.15.118:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fa478006110 con 0x563c18a06010 2023-10-05T19:28:25.404 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.398+0000 7fa47ee11700 1 -- 172.21.15.167:0/4261513042 <== mon.0 v2:172.21.15.118:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa478006270 con 0x563c18a06010 2023-10-05T19:28:25.404 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.398+0000 7fa47ee11700 1 -- 172.21.15.167:0/4261513042 <== mon.0 v2:172.21.15.118:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fa478004520 con 0x563c18a06010 2023-10-05T19:28:25.404 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.398+0000 7fa4832e0e40 1 -- 172.21.15.167:0/4261513042 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x563c18a06010 msgr2=0x563c18a25190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-05T19:28:25.404 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.398+0000 7fa4832e0e40 1 --2- 172.21.15.167:0/4261513042 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x563c18a06010 0x563c18a25190 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7fa4780014a0 tx=0x7fa478006b90 comp rx=0 tx=0).stop 2023-10-05T19:28:25.404 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.398+0000 7fa4832e0e40 1 -- 172.21.15.167:0/4261513042 shutdown_connections 2023-10-05T19:28:25.405 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.398+0000 7fa4832e0e40 1 --2- 172.21.15.167:0/4261513042 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x563c18a06010 0x563c18a25190 unknown :-1 s=CLOSED pgs=178 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-05T19:28:25.405 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.398+0000 7fa4832e0e40 1 -- 172.21.15.167:0/4261513042 >> 172.21.15.167:0/4261513042 conn(0x563c18979860 msgr2=0x563c18a20d50 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-05T19:28:25.405 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.398+0000 7fa4832e0e40 1 -- 172.21.15.167:0/4261513042 shutdown_connections 2023-10-05T19:28:25.405 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.398+0000 7fa4832e0e40 1 -- 172.21.15.167:0/4261513042 wait complete. 2023-10-05T19:28:25.405 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.398+0000 7fa4832e0e40 1 Processor -- start 2023-10-05T19:28:25.405 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.398+0000 7fa4832e0e40 1 -- start start 2023-10-05T19:28:25.405 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.398+0000 7fa4832e0e40 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x563c18a06010 0x563c18a2bf60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-05T19:28:25.405 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.398+0000 7fa4832e0e40 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x563c188a4ed0 con 0x563c18a06010 2023-10-05T19:28:25.405 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.398+0000 7fa480e15700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x563c18a06010 0x563c18a2bf60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-05T19:28:25.405 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.398+0000 7fa480e15700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x563c18a06010 0x563c18a2bf60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.118:3300/0 says I am v2:172.21.15.167:57122/0 (socket says 172.21.15.167:57122) 2023-10-05T19:28:25.405 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.398+0000 7fa480e15700 1 -- 172.21.15.167:0/4019939501 learned_addr learned my addr 172.21.15.167:0/4019939501 (peer_addr_for_me v2:172.21.15.167:0/0) 2023-10-05T19:28:25.406 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.398+0000 7fa480e15700 1 -- 172.21.15.167:0/4019939501 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x563c18950bd0 con 0x563c18a06010 2023-10-05T19:28:25.407 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.398+0000 7fa480e15700 1 --2- 172.21.15.167:0/4019939501 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x563c18a06010 0x563c18a2bf60 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7fa478006bd0 tx=0x7fa478014000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-05T19:28:25.407 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.398+0000 7fa47d60e700 1 -- 172.21.15.167:0/4019939501 <== mon.0 v2:172.21.15.118:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fa478006110 con 0x563c18a06010 2023-10-05T19:28:25.407 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.398+0000 7fa47d60e700 1 -- 172.21.15.167:0/4019939501 <== mon.0 v2:172.21.15.118:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa4780063e0 con 0x563c18a06010 2023-10-05T19:28:25.407 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.398+0000 7fa4832e0e40 1 -- 172.21.15.167:0/4019939501 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x563c18a2c4a0 con 0x563c18a06010 2023-10-05T19:28:25.407 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.398+0000 7fa47d60e700 1 -- 172.21.15.167:0/4019939501 <== mon.0 v2:172.21.15.118:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fa478013d00 con 0x563c18a06010 2023-10-05T19:28:25.407 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.398+0000 7fa4832e0e40 1 -- 172.21.15.167:0/4019939501 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x563c18a2c8e0 con 0x563c18a06010 2023-10-05T19:28:25.407 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.402+0000 7fa4832e0e40 20 rados->read ofs=0 len=0 2023-10-05T19:28:25.407 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.402+0000 7fa47d60e700 1 -- 172.21.15.167:0/4019939501 <== mon.0 v2:172.21.15.118:3300/0 4 ==== mgrmap(e 5) v1 ==== 40329+0+0 (secure 0 0 0) 0x7fa478004030 con 0x563c18a06010 2023-10-05T19:28:25.408 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.402+0000 7fa47d60e700 1 --2- 172.21.15.167:0/4019939501 >> [v2:172.21.15.167:6824/29889,v1:172.21.15.167:6825/29889] conn(0x7fa46c030c00 0x7fa46c0330b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-05T19:28:25.408 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.402+0000 7fa47d60e700 1 -- 172.21.15.167:0/4019939501 <== mon.0 v2:172.21.15.118:3300/0 5 ==== osd_map(30..30 src has 1..30) v4 ==== 6057+0+0 (secure 0 0 0) 0x7fa4780449f0 con 0x563c18a06010 2023-10-05T19:28:25.409 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.402+0000 7fa480614700 1 --2- 172.21.15.167:0/4019939501 >> [v2:172.21.15.167:6824/29889,v1:172.21.15.167:6825/29889] conn(0x7fa46c030c00 0x7fa46c0330b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-05T19:28:25.409 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.402+0000 7fa4832e0e40 1 --2- 172.21.15.167:0/4019939501 >> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] conn(0x563c18a0a5f0 0x563c18a379f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-05T19:28:25.409 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.402+0000 7fa4832e0e40 1 -- 172.21.15.167:0/4019939501 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:1 5.12 5:49953fa1:::default.realm:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x563c18a380f0 con 0x563c18a0a5f0 2023-10-05T19:28:25.409 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.402+0000 7fa481616700 1 --2- 172.21.15.167:0/4019939501 >> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] conn(0x563c18a0a5f0 0x563c18a379f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-05T19:28:25.409 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.402+0000 7fa480614700 1 --2- 172.21.15.167:0/4019939501 >> [v2:172.21.15.167:6824/29889,v1:172.21.15.167:6825/29889] conn(0x7fa46c030c00 0x7fa46c0330b0 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7fa474005a90 tx=0x7fa474015000 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-05T19:28:25.409 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.402+0000 7fa481616700 1 --2- 172.21.15.167:0/4019939501 >> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] conn(0x563c18a0a5f0 0x563c18a379f0 crc :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-05T19:28:25.410 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.402+0000 7fa481616700 1 -- 172.21.15.167:0/4019939501 <== osd.1 v2:172.21.15.167:6800/30035 1 ==== osd_op_reply(1 default.realm [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 ==== 157+0+0 (crc 0 0 0) 0x7fa47000a500 con 0x563c18a0a5f0 2023-10-05T19:28:25.410 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.402+0000 7fa4832e0e40 20 rados_obj.operate() r=-2 bl.length=0 2023-10-05T19:28:25.411 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.402+0000 7fa4832e0e40 20 realm 2023-10-05T19:28:25.411 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.402+0000 7fa4832e0e40 20 rados->read ofs=0 len=0 2023-10-05T19:28:25.411 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.402+0000 7fa4832e0e40 1 -- 172.21.15.167:0/4019939501 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:2 5.12 5:49953fa1:::default.realm:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x563c18a35050 con 0x563c18a0a5f0 2023-10-05T19:28:25.411 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.402+0000 7fa481616700 1 -- 172.21.15.167:0/4019939501 <== osd.1 v2:172.21.15.167:6800/30035 2 ==== osd_op_reply(2 default.realm [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 ==== 157+0+0 (crc 0 0 0) 0x7fa47000a500 con 0x563c18a0a5f0 2023-10-05T19:28:25.411 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.406+0000 7fa4832e0e40 20 rados_obj.operate() r=-2 bl.length=0 2023-10-05T19:28:25.411 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.406+0000 7fa4832e0e40 4 RGWPeriod::init failed to init realm id : (2) No such file or directory 2023-10-05T19:28:25.411 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.406+0000 7fa4832e0e40 20 rados->read ofs=0 len=0 2023-10-05T19:28:25.411 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.406+0000 7fa4832e0e40 1 -- 172.21.15.167:0/4019939501 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:3 5.12 5:49953fa1:::default.realm:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x563c18a38bd0 con 0x563c18a0a5f0 2023-10-05T19:28:25.411 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.406+0000 7fa481616700 1 -- 172.21.15.167:0/4019939501 <== osd.1 v2:172.21.15.167:6800/30035 3 ==== osd_op_reply(3 default.realm [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 ==== 157+0+0 (crc 0 0 0) 0x7fa47000a500 con 0x563c18a0a5f0 2023-10-05T19:28:25.411 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.406+0000 7fa4832e0e40 20 rados_obj.operate() r=-2 bl.length=0 2023-10-05T19:28:25.411 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.406+0000 7fa4832e0e40 20 rados->read ofs=0 len=0 2023-10-05T19:28:25.412 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.406+0000 7fa4832e0e40 1 -- 172.21.15.167:0/4019939501 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:4 5.f 5:f43b6ece:::zone_names.default:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x563c18a387b0 con 0x563c18a0a5f0 2023-10-05T19:28:25.413 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.406+0000 7fa481616700 1 -- 172.21.15.167:0/4019939501 <== osd.1 v2:172.21.15.167:6800/30035 4 ==== osd_op_reply(4 zone_names.default [read 0~46 out=46b] v0'0 uv1 ondisk = 0) v8 ==== 162+0+46 (crc 0 0 0) 0x7fa47000a500 con 0x563c18a0a5f0 2023-10-05T19:28:25.413 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.406+0000 7fa4832e0e40 20 rados_obj.operate() r=0 bl.length=46 2023-10-05T19:28:25.413 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.406+0000 7fa4832e0e40 20 rados->read ofs=0 len=0 2023-10-05T19:28:25.413 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.406+0000 7fa4832e0e40 1 -- 172.21.15.167:0/4019939501 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:5 5.1d 5:b8374eb4:::zone_info.9a9df965-6612-45d4-a45f-6e3c889bacc2:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x563c18a39640 con 0x563c18a0a5f0 2023-10-05T19:28:25.413 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.406+0000 7fa481616700 1 -- 172.21.15.167:0/4019939501 <== osd.1 v2:172.21.15.167:6800/30035 5 ==== osd_op_reply(5 zone_info.9a9df965-6612-45d4-a45f-6e3c889bacc2 [read 0~874 out=874b] v0'0 uv2 ondisk = 0) v8 ==== 190+0+874 (crc 0 0 0) 0x7fa47000a500 con 0x563c18a0a5f0 2023-10-05T19:28:25.413 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.406+0000 7fa4832e0e40 20 rados_obj.operate() r=0 bl.length=874 2023-10-05T19:28:25.413 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.406+0000 7fa4832e0e40 20 searching for the correct realm 2023-10-05T19:28:25.414 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.406+0000 7fa4832e0e40 1 --2- 172.21.15.167:0/4019939501 >> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] conn(0x563c18a3b490 0x563c18a3d870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-05T19:28:25.414 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.406+0000 7fa4832e0e40 1 -- 172.21.15.167:0/4019939501 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:6 5.0 5:00000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x563c18a3ddb0 con 0x563c18a3b490 2023-10-05T19:28:25.414 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.406+0000 7fa480e15700 1 --2- 172.21.15.167:0/4019939501 >> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] conn(0x563c18a3b490 0x563c18a3d870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-05T19:28:25.414 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.406+0000 7fa480e15700 1 --2- 172.21.15.167:0/4019939501 >> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] conn(0x563c18a3b490 0x563c18a3d870 crc :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-05T19:28:25.415 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.410+0000 7fa480e15700 1 -- 172.21.15.167:0/4019939501 <== osd.3 v2:172.21.15.167:6808/30036 1 ==== osd_op_reply(6 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fa478012c70 con 0x563c18a3b490 2023-10-05T19:28:25.415 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.410+0000 7fa480e15700 1 --2- 172.21.15.167:0/4019939501 >> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] conn(0x7fa4780276e0 0x7fa4780441e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-05T19:28:25.415 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.410+0000 7fa480e15700 1 -- 172.21.15.167:0/4019939501 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:7 5.10 5:08000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x563c18a3ddb0 con 0x7fa4780276e0 2023-10-05T19:28:25.415 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.410+0000 7fa480614700 1 --2- 172.21.15.167:0/4019939501 >> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] conn(0x7fa4780276e0 0x7fa4780441e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-05T19:28:25.415 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.410+0000 7fa480614700 1 --2- 172.21.15.167:0/4019939501 >> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] conn(0x7fa4780276e0 0x7fa4780441e0 crc :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-05T19:28:25.416 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.410+0000 7fa480614700 1 -- 172.21.15.167:0/4019939501 <== osd.2 v2:172.21.15.167:6816/30037 1 ==== osd_op_reply(7 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fa474008a80 con 0x7fa4780276e0 2023-10-05T19:28:25.416 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.410+0000 7fa480614700 1 --2- 172.21.15.167:0/4019939501 >> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] conn(0x7fa474007030 0x7fa47401c410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-05T19:28:25.416 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.410+0000 7fa480614700 1 -- 172.21.15.167:0/4019939501 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:8 5.8 5:10000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x563c18a3ddb0 con 0x7fa474007030 2023-10-05T19:28:25.417 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.410+0000 7fa481616700 1 --2- 172.21.15.167:0/4019939501 >> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] conn(0x7fa474007030 0x7fa47401c410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-05T19:28:25.417 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.410+0000 7fa481616700 1 --2- 172.21.15.167:0/4019939501 >> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] conn(0x7fa474007030 0x7fa47401c410 crc :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-05T19:28:25.418 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.414+0000 7fa481616700 1 -- 172.21.15.167:0/4019939501 <== osd.0 v2:172.21.15.118:6800/27690 1 ==== osd_op_reply(8 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fa470014040 con 0x7fa474007030 2023-10-05T19:28:25.418 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.414+0000 7fa481616700 1 -- 172.21.15.167:0/4019939501 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:9 5.18 5:18000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x563c18a3ddb0 con 0x563c18a0a5f0 2023-10-05T19:28:25.419 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.414+0000 7fa481616700 1 -- 172.21.15.167:0/4019939501 <== osd.1 v2:172.21.15.167:6800/30035 6 ==== osd_op_reply(9 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fa470014040 con 0x563c18a0a5f0 2023-10-05T19:28:25.419 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.414+0000 7fa481616700 1 -- 172.21.15.167:0/4019939501 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:10 5.4 5:20000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x563c18a3ddb0 con 0x563c18a0a5f0 2023-10-05T19:28:25.420 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.414+0000 7fa481616700 1 -- 172.21.15.167:0/4019939501 <== osd.1 v2:172.21.15.167:6800/30035 7 ==== osd_op_reply(10 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fa470014040 con 0x563c18a0a5f0 2023-10-05T19:28:25.420 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.414+0000 7fa481616700 1 -- 172.21.15.167:0/4019939501 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:11 5.14 5:28000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x563c18a3ddb0 con 0x7fa474007030 2023-10-05T19:28:25.421 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.414+0000 7fa481616700 1 -- 172.21.15.167:0/4019939501 <== osd.0 v2:172.21.15.118:6800/27690 2 ==== osd_op_reply(11 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fa470014040 con 0x7fa474007030 2023-10-05T19:28:25.421 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.414+0000 7fa481616700 1 -- 172.21.15.167:0/4019939501 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:12 5.c 5:30000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x563c18a3ddb0 con 0x7fa474007030 2023-10-05T19:28:25.421 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.414+0000 7fa481616700 1 -- 172.21.15.167:0/4019939501 <== osd.0 v2:172.21.15.118:6800/27690 3 ==== osd_op_reply(12 [pgnls start_epoch 30 out=112b] v23'1 uv1 ondisk = 1) v8 ==== 144+0+112 (crc 0 0 0) 0x7fa470014040 con 0x7fa474007030 2023-10-05T19:28:25.421 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.414+0000 7fa481616700 1 -- 172.21.15.167:0/4019939501 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:13 5.1c 5:38000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x563c18a3ddb0 con 0x7fa4780276e0 2023-10-05T19:28:25.422 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.418+0000 7fa480614700 1 -- 172.21.15.167:0/4019939501 <== osd.2 v2:172.21.15.167:6816/30037 2 ==== osd_op_reply(13 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fa474008a80 con 0x7fa4780276e0 2023-10-05T19:28:25.422 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.418+0000 7fa480614700 1 -- 172.21.15.167:0/4019939501 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:14 5.2 5:40000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x563c18a3ddb0 con 0x563c18a3b490 2023-10-05T19:28:25.423 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.418+0000 7fa480e15700 1 -- 172.21.15.167:0/4019939501 <== osd.3 v2:172.21.15.167:6808/30036 2 ==== osd_op_reply(14 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fa478012c70 con 0x563c18a3b490 2023-10-05T19:28:25.423 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.418+0000 7fa480e15700 1 -- 172.21.15.167:0/4019939501 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:15 5.12 5:48000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x563c18a3ddb0 con 0x563c18a0a5f0 2023-10-05T19:28:25.424 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.418+0000 7fa481616700 1 -- 172.21.15.167:0/4019939501 <== osd.1 v2:172.21.15.167:6800/30035 8 ==== osd_op_reply(15 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fa470014040 con 0x563c18a0a5f0 2023-10-05T19:28:25.424 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.418+0000 7fa481616700 1 -- 172.21.15.167:0/4019939501 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:16 5.a 5:50000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x563c18a3ddb0 con 0x7fa474007030 2023-10-05T19:28:25.424 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.418+0000 7fa481616700 1 -- 172.21.15.167:0/4019939501 <== osd.0 v2:172.21.15.118:6800/27690 4 ==== osd_op_reply(16 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fa470022040 con 0x7fa474007030 2023-10-05T19:28:25.425 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.418+0000 7fa481616700 1 -- 172.21.15.167:0/4019939501 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:17 5.1a 5:58000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x563c18a3ddb0 con 0x563c18a0a5f0 2023-10-05T19:28:25.425 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.418+0000 7fa481616700 1 -- 172.21.15.167:0/4019939501 <== osd.1 v2:172.21.15.167:6800/30035 9 ==== osd_op_reply(17 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fa470022040 con 0x563c18a0a5f0 2023-10-05T19:28:25.425 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.418+0000 7fa481616700 1 -- 172.21.15.167:0/4019939501 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:18 5.6 5:60000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x563c18a3ddb0 con 0x7fa4780276e0 2023-10-05T19:28:25.426 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.422+0000 7fa480614700 1 -- 172.21.15.167:0/4019939501 <== osd.2 v2:172.21.15.167:6816/30037 3 ==== osd_op_reply(18 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fa474008a80 con 0x7fa4780276e0 2023-10-05T19:28:25.426 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.422+0000 7fa480614700 1 -- 172.21.15.167:0/4019939501 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:19 5.16 5:68000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x563c18a3ddb0 con 0x563c18a3b490 2023-10-05T19:28:25.427 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.422+0000 7fa480e15700 1 -- 172.21.15.167:0/4019939501 <== osd.3 v2:172.21.15.167:6808/30036 3 ==== osd_op_reply(19 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fa478012c70 con 0x563c18a3b490 2023-10-05T19:28:25.427 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.422+0000 7fa480e15700 1 -- 172.21.15.167:0/4019939501 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:20 5.e 5:70000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x563c18a3ddb0 con 0x563c18a3b490 2023-10-05T19:28:25.427 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.422+0000 7fa480e15700 1 -- 172.21.15.167:0/4019939501 <== osd.3 v2:172.21.15.167:6808/30036 4 ==== osd_op_reply(20 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fa478012c70 con 0x563c18a3b490 2023-10-05T19:28:25.427 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.422+0000 7fa480e15700 1 -- 172.21.15.167:0/4019939501 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:21 5.1e 5:78000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x563c18a3ddb0 con 0x7fa474007030 2023-10-05T19:28:25.428 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.422+0000 7fa481616700 1 -- 172.21.15.167:0/4019939501 <== osd.0 v2:172.21.15.118:6800/27690 5 ==== osd_op_reply(21 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fa470022040 con 0x7fa474007030 2023-10-05T19:28:25.428 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.422+0000 7fa481616700 1 -- 172.21.15.167:0/4019939501 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:22 5.1 5:80000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x563c18a3ddb0 con 0x563c18a3b490 2023-10-05T19:28:25.429 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.422+0000 7fa480e15700 1 -- 172.21.15.167:0/4019939501 <== osd.3 v2:172.21.15.167:6808/30036 5 ==== osd_op_reply(22 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fa478012c70 con 0x563c18a3b490 2023-10-05T19:28:25.429 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.422+0000 7fa480e15700 1 -- 172.21.15.167:0/4019939501 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:23 5.11 5:88000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x563c18a3ddb0 con 0x563c18a3b490 2023-10-05T19:28:25.429 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.422+0000 7fa480e15700 1 -- 172.21.15.167:0/4019939501 <== osd.3 v2:172.21.15.167:6808/30036 6 ==== osd_op_reply(23 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fa478012c70 con 0x563c18a3b490 2023-10-05T19:28:25.430 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.422+0000 7fa480e15700 1 -- 172.21.15.167:0/4019939501 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:24 5.9 5:90000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x563c18a3ddb0 con 0x563c18a0a5f0 2023-10-05T19:28:25.430 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.426+0000 7fa481616700 1 -- 172.21.15.167:0/4019939501 <== osd.1 v2:172.21.15.167:6800/30035 10 ==== osd_op_reply(24 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fa470022040 con 0x563c18a0a5f0 2023-10-05T19:28:25.430 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.426+0000 7fa481616700 1 -- 172.21.15.167:0/4019939501 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:25 5.19 5:98000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x563c18a3ddb0 con 0x563c18a0a5f0 2023-10-05T19:28:25.431 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.426+0000 7fa481616700 1 -- 172.21.15.167:0/4019939501 <== osd.1 v2:172.21.15.167:6800/30035 11 ==== osd_op_reply(25 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fa470022040 con 0x563c18a0a5f0 2023-10-05T19:28:25.431 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.426+0000 7fa481616700 1 -- 172.21.15.167:0/4019939501 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:26 5.5 5:a0000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x563c18a3ddb0 con 0x7fa474007030 2023-10-05T19:28:25.431 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.426+0000 7fa481616700 1 -- 172.21.15.167:0/4019939501 <== osd.0 v2:172.21.15.118:6800/27690 6 ==== osd_op_reply(26 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fa470022040 con 0x7fa474007030 2023-10-05T19:28:25.431 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.426+0000 7fa481616700 1 -- 172.21.15.167:0/4019939501 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:27 5.15 5:a8000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x563c18a3ddb0 con 0x563c18a3b490 2023-10-05T19:28:25.432 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.426+0000 7fa480e15700 1 -- 172.21.15.167:0/4019939501 <== osd.3 v2:172.21.15.167:6808/30036 7 ==== osd_op_reply(27 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fa478012c70 con 0x563c18a3b490 2023-10-05T19:28:25.432 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.426+0000 7fa480e15700 1 -- 172.21.15.167:0/4019939501 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:28 5.d 5:b0000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x563c18a3ddb0 con 0x7fa4780276e0 2023-10-05T19:28:25.433 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.426+0000 7fa480614700 1 -- 172.21.15.167:0/4019939501 <== osd.2 v2:172.21.15.167:6816/30037 4 ==== osd_op_reply(28 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fa474008a80 con 0x7fa4780276e0 2023-10-05T19:28:25.433 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.426+0000 7fa480614700 1 -- 172.21.15.167:0/4019939501 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:29 5.1d 5:b8000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x563c18a3ddb0 con 0x563c18a0a5f0 2023-10-05T19:28:25.434 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.426+0000 7fa481616700 1 -- 172.21.15.167:0/4019939501 <== osd.1 v2:172.21.15.167:6800/30035 12 ==== osd_op_reply(29 [pgnls start_epoch 30 out=107b] v30'2 uv2 ondisk = 1) v8 ==== 144+0+107 (crc 0 0 0) 0x7fa470022040 con 0x563c18a0a5f0 2023-10-05T19:28:25.434 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.430+0000 7fa481616700 1 -- 172.21.15.167:0/4019939501 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:30 5.3 5:c0000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x563c18a3ddb0 con 0x7fa474007030 2023-10-05T19:28:25.435 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.430+0000 7fa481616700 1 -- 172.21.15.167:0/4019939501 <== osd.0 v2:172.21.15.118:6800/27690 7 ==== osd_op_reply(30 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fa470022040 con 0x7fa474007030 2023-10-05T19:28:25.435 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.430+0000 7fa481616700 1 -- 172.21.15.167:0/4019939501 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:31 5.13 5:c8000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x563c18a3ddb0 con 0x563c18a3b490 2023-10-05T19:28:25.435 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.430+0000 7fa480e15700 1 -- 172.21.15.167:0/4019939501 <== osd.3 v2:172.21.15.167:6808/30036 8 ==== osd_op_reply(31 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fa478012c70 con 0x563c18a3b490 2023-10-05T19:28:25.436 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.430+0000 7fa480e15700 1 -- 172.21.15.167:0/4019939501 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:32 5.b 5:d0000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x563c18a3ddb0 con 0x7fa474007030 2023-10-05T19:28:25.436 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.430+0000 7fa481616700 1 -- 172.21.15.167:0/4019939501 <== osd.0 v2:172.21.15.118:6800/27690 8 ==== osd_op_reply(32 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fa470022040 con 0x7fa474007030 2023-10-05T19:28:25.436 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.430+0000 7fa481616700 1 -- 172.21.15.167:0/4019939501 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:33 5.1b 5:d8000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x563c18a3ddb0 con 0x7fa4780276e0 2023-10-05T19:28:25.437 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.430+0000 7fa480614700 1 -- 172.21.15.167:0/4019939501 <== osd.2 v2:172.21.15.167:6816/30037 5 ==== osd_op_reply(33 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fa474008a80 con 0x7fa4780276e0 2023-10-05T19:28:25.437 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.430+0000 7fa480614700 1 -- 172.21.15.167:0/4019939501 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:34 5.7 5:e0000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x563c18a3ddb0 con 0x563c18a3b490 2023-10-05T19:28:25.438 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.434+0000 7fa480e15700 1 -- 172.21.15.167:0/4019939501 <== osd.3 v2:172.21.15.167:6808/30036 9 ==== osd_op_reply(34 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fa478012c70 con 0x563c18a3b490 2023-10-05T19:28:25.438 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.434+0000 7fa480e15700 1 -- 172.21.15.167:0/4019939501 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:35 5.17 5:e8000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x563c18a3ddb0 con 0x7fa474007030 2023-10-05T19:28:25.439 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.434+0000 7fa481616700 1 -- 172.21.15.167:0/4019939501 <== osd.0 v2:172.21.15.118:6800/27690 9 ==== osd_op_reply(35 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fa470022040 con 0x7fa474007030 2023-10-05T19:28:25.439 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.434+0000 7fa481616700 1 -- 172.21.15.167:0/4019939501 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:36 5.f 5:f0000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x563c18a3ddb0 con 0x563c18a0a5f0 2023-10-05T19:28:25.440 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.434+0000 7fa481616700 1 -- 172.21.15.167:0/4019939501 <== osd.1 v2:172.21.15.167:6800/30035 13 ==== osd_op_reply(36 [pgnls start_epoch 30 out=115b] v23'2 uv2 ondisk = 1) v8 ==== 144+0+115 (crc 0 0 0) 0x7fa47002f080 con 0x563c18a0a5f0 2023-10-05T19:28:25.440 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.434+0000 7fa481616700 1 -- 172.21.15.167:0/4019939501 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:37 5.1f 5:f8000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x563c18a3ddb0 con 0x7fa4780276e0 2023-10-05T19:28:25.441 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.434+0000 7fa480614700 1 -- 172.21.15.167:0/4019939501 <== osd.2 v2:172.21.15.167:6816/30037 6 ==== osd_op_reply(37 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fa474008a80 con 0x7fa4780276e0 2023-10-05T19:28:25.441 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.434+0000 7fa4832e0e40 20 RGWRados::pool_iterate: got zonegroup_info.4f1f719d-9648-417b-b10c-e5d939825419 2023-10-05T19:28:25.441 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.434+0000 7fa4832e0e40 20 RGWRados::pool_iterate: got zone_info.9a9df965-6612-45d4-a45f-6e3c889bacc2 2023-10-05T19:28:25.441 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.434+0000 7fa4832e0e40 20 RGWRados::pool_iterate: got zone_names.default 2023-10-05T19:28:25.441 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.434+0000 7fa4832e0e40 20 RGWRados::pool_iterate: got zonegroups_names.default 2023-10-05T19:28:25.441 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.434+0000 7fa4832e0e40 20 rados->read ofs=0 len=0 2023-10-05T19:28:25.441 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.434+0000 7fa4832e0e40 1 -- 172.21.15.167:0/4019939501 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:38 5.12 5:49953fa1:::default.realm:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x563c18a3a210 con 0x563c18a0a5f0 2023-10-05T19:28:25.442 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.434+0000 7fa481616700 1 -- 172.21.15.167:0/4019939501 <== osd.1 v2:172.21.15.167:6800/30035 14 ==== osd_op_reply(38 default.realm [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 ==== 157+0+0 (crc 0 0 0) 0x7fa47002f080 con 0x563c18a0a5f0 2023-10-05T19:28:25.442 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.434+0000 7fa4832e0e40 20 rados_obj.operate() r=-2 bl.length=0 2023-10-05T19:28:25.442 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.434+0000 7fa4832e0e40 20 rados->read ofs=0 len=0 2023-10-05T19:28:25.443 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.434+0000 7fa4832e0e40 1 -- 172.21.15.167:0/4019939501 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:39 5.f 5:f4c53578:::zonegroups_names.default:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x563c18a3ab90 con 0x563c18a0a5f0 2023-10-05T19:28:25.443 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.438+0000 7fa481616700 1 -- 172.21.15.167:0/4019939501 <== osd.1 v2:172.21.15.167:6800/30035 15 ==== osd_op_reply(39 zonegroups_names.default [read 0~46 out=46b] v0'0 uv2 ondisk = 0) v8 ==== 168+0+46 (crc 0 0 0) 0x7fa47002f080 con 0x563c18a0a5f0 2023-10-05T19:28:25.443 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.438+0000 7fa4832e0e40 20 rados_obj.operate() r=0 bl.length=46 2023-10-05T19:28:25.443 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.438+0000 7fa4832e0e40 20 rados->read ofs=0 len=0 2023-10-05T19:28:25.443 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.438+0000 7fa4832e0e40 1 -- 172.21.15.167:0/4019939501 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:40 5.c 5:30d299a5:::zonegroup_info.4f1f719d-9648-417b-b10c-e5d939825419:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x563c18a39dd0 con 0x7fa474007030 2023-10-05T19:28:25.444 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.438+0000 7fa481616700 1 -- 172.21.15.167:0/4019939501 <== osd.0 v2:172.21.15.118:6800/27690 10 ==== osd_op_reply(40 zonegroup_info.4f1f719d-9648-417b-b10c-e5d939825419 [read 0~362 out=362b] v0'0 uv1 ondisk = 0) v8 ==== 195+0+362 (crc 0 0 0) 0x7fa47002f080 con 0x7fa474007030 2023-10-05T19:28:25.444 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.438+0000 7fa4832e0e40 20 rados_obj.operate() r=0 bl.length=362 2023-10-05T19:28:25.444 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.438+0000 7fa4832e0e40 20 rados->read ofs=0 len=0 2023-10-05T19:28:25.444 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.438+0000 7fa4832e0e40 1 -- 172.21.15.167:0/4019939501 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:41 5.13 5:cbbf6c60:::converted:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x563c18a3e160 con 0x563c18a3b490 2023-10-05T19:28:25.444 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.438+0000 7fa480e15700 1 -- 172.21.15.167:0/4019939501 <== osd.3 v2:172.21.15.167:6808/30036 10 ==== osd_op_reply(41 converted [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 ==== 153+0+0 (crc 0 0 0) 0x7fa478012c70 con 0x563c18a3b490 2023-10-05T19:28:25.444 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.438+0000 7fa4832e0e40 20 rados_obj.operate() r=-2 bl.length=0 2023-10-05T19:28:25.444 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.438+0000 7fa4832e0e40 20 rados->read ofs=0 len=0 2023-10-05T19:28:25.444 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.438+0000 7fa4832e0e40 1 -- 172.21.15.167:0/4019939501 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:42 5.12 5:49953fa1:::default.realm:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x563c18a3e510 con 0x563c18a0a5f0 2023-10-05T19:28:25.445 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.438+0000 7fa481616700 1 -- 172.21.15.167:0/4019939501 <== osd.1 v2:172.21.15.167:6800/30035 16 ==== osd_op_reply(42 default.realm [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 ==== 157+0+0 (crc 0 0 0) 0x7fa47002f080 con 0x563c18a0a5f0 2023-10-05T19:28:25.445 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.438+0000 7fa4832e0e40 20 rados_obj.operate() r=-2 bl.length=0 2023-10-05T19:28:25.445 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.438+0000 7fa4832e0e40 20 rados->read ofs=0 len=0 2023-10-05T19:28:25.446 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.438+0000 7fa4832e0e40 1 -- 172.21.15.167:0/4019939501 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:43 5.f 5:f4c53578:::zonegroups_names.default:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x563c18a3e510 con 0x563c18a0a5f0 2023-10-05T19:28:25.446 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.438+0000 7fa481616700 1 -- 172.21.15.167:0/4019939501 <== osd.1 v2:172.21.15.167:6800/30035 17 ==== osd_op_reply(43 zonegroups_names.default [read 0~46 out=46b] v0'0 uv2 ondisk = 0) v8 ==== 168+0+46 (crc 0 0 0) 0x7fa47002f080 con 0x563c18a0a5f0 2023-10-05T19:28:25.446 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.438+0000 7fa4832e0e40 20 rados_obj.operate() r=0 bl.length=46 2023-10-05T19:28:25.446 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.438+0000 7fa4832e0e40 1 -- 172.21.15.167:0/4019939501 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:44 5.0 5:00000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x563c18a3f520 con 0x563c18a3b490 2023-10-05T19:28:25.447 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.442+0000 7fa480e15700 1 -- 172.21.15.167:0/4019939501 <== osd.3 v2:172.21.15.167:6808/30036 11 ==== osd_op_reply(44 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fa478012c70 con 0x563c18a3b490 2023-10-05T19:28:25.448 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.442+0000 7fa480e15700 1 -- 172.21.15.167:0/4019939501 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:45 5.10 5:08000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x563c18a3f520 con 0x7fa4780276e0 2023-10-05T19:28:25.448 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.442+0000 7fa480614700 1 -- 172.21.15.167:0/4019939501 <== osd.2 v2:172.21.15.167:6816/30037 7 ==== osd_op_reply(45 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fa474008a80 con 0x7fa4780276e0 2023-10-05T19:28:25.448 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.442+0000 7fa480614700 1 -- 172.21.15.167:0/4019939501 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:46 5.8 5:10000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x563c18a3f520 con 0x7fa474007030 2023-10-05T19:28:25.449 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.442+0000 7fa481616700 1 -- 172.21.15.167:0/4019939501 <== osd.0 v2:172.21.15.118:6800/27690 11 ==== osd_op_reply(46 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fa47002f080 con 0x7fa474007030 2023-10-05T19:28:25.449 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.442+0000 7fa481616700 1 -- 172.21.15.167:0/4019939501 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:47 5.18 5:18000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x563c18a3ab90 con 0x563c18a0a5f0 2023-10-05T19:28:25.450 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.446+0000 7fa481616700 1 -- 172.21.15.167:0/4019939501 <== osd.1 v2:172.21.15.167:6800/30035 18 ==== osd_op_reply(47 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fa47002f080 con 0x563c18a0a5f0 2023-10-05T19:28:25.450 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.446+0000 7fa481616700 1 -- 172.21.15.167:0/4019939501 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:48 5.4 5:20000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x563c18a3ab90 con 0x563c18a0a5f0 2023-10-05T19:28:25.451 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.446+0000 7fa481616700 1 -- 172.21.15.167:0/4019939501 <== osd.1 v2:172.21.15.167:6800/30035 19 ==== osd_op_reply(48 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fa47002f080 con 0x563c18a0a5f0 2023-10-05T19:28:25.451 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.446+0000 7fa481616700 1 -- 172.21.15.167:0/4019939501 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:49 5.14 5:28000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x563c18a3ab90 con 0x7fa474007030 2023-10-05T19:28:25.452 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.446+0000 7fa481616700 1 -- 172.21.15.167:0/4019939501 <== osd.0 v2:172.21.15.118:6800/27690 12 ==== osd_op_reply(49 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fa47002f080 con 0x7fa474007030 2023-10-05T19:28:25.452 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.446+0000 7fa481616700 1 -- 172.21.15.167:0/4019939501 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:50 5.c 5:30000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x563c18a3ab90 con 0x7fa474007030 2023-10-05T19:28:25.453 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.446+0000 7fa481616700 1 -- 172.21.15.167:0/4019939501 <== osd.0 v2:172.21.15.118:6800/27690 13 ==== osd_op_reply(50 [pgnls start_epoch 30 out=112b] v23'1 uv1 ondisk = 1) v8 ==== 144+0+112 (crc 0 0 0) 0x7fa47002f080 con 0x7fa474007030 2023-10-05T19:28:25.453 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.446+0000 7fa481616700 1 -- 172.21.15.167:0/4019939501 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:51 5.1c 5:38000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x563c18a3ab90 con 0x7fa4780276e0 2023-10-05T19:28:25.454 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.450+0000 7fa480614700 1 -- 172.21.15.167:0/4019939501 <== osd.2 v2:172.21.15.167:6816/30037 8 ==== osd_op_reply(51 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fa474008a80 con 0x7fa4780276e0 2023-10-05T19:28:25.454 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.450+0000 7fa480614700 1 -- 172.21.15.167:0/4019939501 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:52 5.2 5:40000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x563c18a3ab90 con 0x563c18a3b490 2023-10-05T19:28:25.454 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.450+0000 7fa480e15700 1 -- 172.21.15.167:0/4019939501 <== osd.3 v2:172.21.15.167:6808/30036 12 ==== osd_op_reply(52 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fa478012c70 con 0x563c18a3b490 2023-10-05T19:28:25.455 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.450+0000 7fa480e15700 1 -- 172.21.15.167:0/4019939501 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:53 5.12 5:48000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x563c18a3ab90 con 0x563c18a0a5f0 2023-10-05T19:28:25.455 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.450+0000 7fa481616700 1 -- 172.21.15.167:0/4019939501 <== osd.1 v2:172.21.15.167:6800/30035 20 ==== osd_op_reply(53 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fa47002f080 con 0x563c18a0a5f0 2023-10-05T19:28:25.455 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.450+0000 7fa481616700 1 -- 172.21.15.167:0/4019939501 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:54 5.a 5:50000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x563c18a3ab90 con 0x7fa474007030 2023-10-05T19:28:25.456 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.450+0000 7fa481616700 1 -- 172.21.15.167:0/4019939501 <== osd.0 v2:172.21.15.118:6800/27690 14 ==== osd_op_reply(54 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fa47002f080 con 0x7fa474007030 2023-10-05T19:28:25.456 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.450+0000 7fa481616700 1 -- 172.21.15.167:0/4019939501 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:55 5.1a 5:58000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x563c18a3ab90 con 0x563c18a0a5f0 2023-10-05T19:28:25.457 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.450+0000 7fa481616700 1 -- 172.21.15.167:0/4019939501 <== osd.1 v2:172.21.15.167:6800/30035 21 ==== osd_op_reply(55 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fa47002f080 con 0x563c18a0a5f0 2023-10-05T19:28:25.457 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.450+0000 7fa481616700 1 -- 172.21.15.167:0/4019939501 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:56 5.6 5:60000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x563c18a3ab90 con 0x7fa4780276e0 2023-10-05T19:28:25.458 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.454+0000 7fa480614700 1 -- 172.21.15.167:0/4019939501 <== osd.2 v2:172.21.15.167:6816/30037 9 ==== osd_op_reply(56 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fa474008a80 con 0x7fa4780276e0 2023-10-05T19:28:25.458 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.454+0000 7fa480614700 1 -- 172.21.15.167:0/4019939501 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:57 5.16 5:68000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x563c18a3ab90 con 0x563c18a3b490 2023-10-05T19:28:25.459 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.454+0000 7fa480e15700 1 -- 172.21.15.167:0/4019939501 <== osd.3 v2:172.21.15.167:6808/30036 13 ==== osd_op_reply(57 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fa478012c70 con 0x563c18a3b490 2023-10-05T19:28:25.459 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.454+0000 7fa480e15700 1 -- 172.21.15.167:0/4019939501 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:58 5.e 5:70000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x563c18a3ab90 con 0x563c18a3b490 2023-10-05T19:28:25.460 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.454+0000 7fa480e15700 1 -- 172.21.15.167:0/4019939501 <== osd.3 v2:172.21.15.167:6808/30036 14 ==== osd_op_reply(58 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fa478012c70 con 0x563c18a3b490 2023-10-05T19:28:25.460 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.454+0000 7fa480e15700 1 -- 172.21.15.167:0/4019939501 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:59 5.1e 5:78000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x563c18a3ab90 con 0x7fa474007030 2023-10-05T19:28:25.461 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.454+0000 7fa481616700 1 -- 172.21.15.167:0/4019939501 <== osd.0 v2:172.21.15.118:6800/27690 15 ==== osd_op_reply(59 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fa47002f080 con 0x7fa474007030 2023-10-05T19:28:25.461 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.454+0000 7fa481616700 1 -- 172.21.15.167:0/4019939501 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:60 5.1 5:80000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x563c18a3ab90 con 0x563c18a3b490 2023-10-05T19:28:25.461 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.454+0000 7fa480e15700 1 -- 172.21.15.167:0/4019939501 <== osd.3 v2:172.21.15.167:6808/30036 15 ==== osd_op_reply(60 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fa478012c70 con 0x563c18a3b490 2023-10-05T19:28:25.462 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.454+0000 7fa480e15700 1 -- 172.21.15.167:0/4019939501 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:61 5.11 5:88000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x563c18a3ab90 con 0x563c18a3b490 2023-10-05T19:28:25.462 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.458+0000 7fa480e15700 1 -- 172.21.15.167:0/4019939501 <== osd.3 v2:172.21.15.167:6808/30036 16 ==== osd_op_reply(61 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fa478012c70 con 0x563c18a3b490 2023-10-05T19:28:25.462 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.458+0000 7fa480e15700 1 -- 172.21.15.167:0/4019939501 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:62 5.9 5:90000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x563c18a3ab90 con 0x563c18a0a5f0 2023-10-05T19:28:25.463 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.458+0000 7fa481616700 1 -- 172.21.15.167:0/4019939501 <== osd.1 v2:172.21.15.167:6800/30035 22 ==== osd_op_reply(62 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fa47002f080 con 0x563c18a0a5f0 2023-10-05T19:28:25.463 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.458+0000 7fa481616700 1 -- 172.21.15.167:0/4019939501 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:63 5.19 5:98000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x563c18a3ab90 con 0x563c18a0a5f0 2023-10-05T19:28:25.464 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.458+0000 7fa481616700 1 -- 172.21.15.167:0/4019939501 <== osd.1 v2:172.21.15.167:6800/30035 23 ==== osd_op_reply(63 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fa47002f080 con 0x563c18a0a5f0 2023-10-05T19:28:25.464 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.458+0000 7fa481616700 1 -- 172.21.15.167:0/4019939501 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:64 5.5 5:a0000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x563c18a3ab90 con 0x7fa474007030 2023-10-05T19:28:25.465 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.458+0000 7fa481616700 1 -- 172.21.15.167:0/4019939501 <== osd.0 v2:172.21.15.118:6800/27690 16 ==== osd_op_reply(64 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fa47002f080 con 0x7fa474007030 2023-10-05T19:28:25.465 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.458+0000 7fa481616700 1 -- 172.21.15.167:0/4019939501 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:65 5.15 5:a8000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x563c18a3ab90 con 0x563c18a3b490 2023-10-05T19:28:25.466 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.458+0000 7fa480e15700 1 -- 172.21.15.167:0/4019939501 <== osd.3 v2:172.21.15.167:6808/30036 17 ==== osd_op_reply(65 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fa478012c70 con 0x563c18a3b490 2023-10-05T19:28:25.466 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.458+0000 7fa480e15700 1 -- 172.21.15.167:0/4019939501 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:66 5.d 5:b0000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x563c18a3ab90 con 0x7fa4780276e0 2023-10-05T19:28:25.467 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.462+0000 7fa480614700 1 -- 172.21.15.167:0/4019939501 <== osd.2 v2:172.21.15.167:6816/30037 10 ==== osd_op_reply(66 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fa474008a80 con 0x7fa4780276e0 2023-10-05T19:28:25.467 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.462+0000 7fa480614700 1 -- 172.21.15.167:0/4019939501 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:67 5.1d 5:b8000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x563c18a3ab90 con 0x563c18a0a5f0 2023-10-05T19:28:25.467 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.462+0000 7fa481616700 1 -- 172.21.15.167:0/4019939501 <== osd.1 v2:172.21.15.167:6800/30035 24 ==== osd_op_reply(67 [pgnls start_epoch 30 out=107b] v30'2 uv2 ondisk = 1) v8 ==== 144+0+107 (crc 0 0 0) 0x7fa47002f080 con 0x563c18a0a5f0 2023-10-05T19:28:25.467 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.462+0000 7fa481616700 1 -- 172.21.15.167:0/4019939501 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:68 5.3 5:c0000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x563c18a3ab90 con 0x7fa474007030 2023-10-05T19:28:25.468 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.462+0000 7fa481616700 1 -- 172.21.15.167:0/4019939501 <== osd.0 v2:172.21.15.118:6800/27690 17 ==== osd_op_reply(68 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fa47002f080 con 0x7fa474007030 2023-10-05T19:28:25.468 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.462+0000 7fa481616700 1 -- 172.21.15.167:0/4019939501 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:69 5.13 5:c8000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x563c18a3ab90 con 0x563c18a3b490 2023-10-05T19:28:25.469 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.462+0000 7fa480e15700 1 -- 172.21.15.167:0/4019939501 <== osd.3 v2:172.21.15.167:6808/30036 18 ==== osd_op_reply(69 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fa478012c70 con 0x563c18a3b490 2023-10-05T19:28:25.469 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.462+0000 7fa480e15700 1 -- 172.21.15.167:0/4019939501 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:70 5.b 5:d0000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x563c18a3ab90 con 0x7fa474007030 2023-10-05T19:28:25.470 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.466+0000 7fa481616700 1 -- 172.21.15.167:0/4019939501 <== osd.0 v2:172.21.15.118:6800/27690 18 ==== osd_op_reply(70 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fa47002f080 con 0x7fa474007030 2023-10-05T19:28:25.470 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.466+0000 7fa481616700 1 -- 172.21.15.167:0/4019939501 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:71 5.1b 5:d8000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x563c18a3ab90 con 0x7fa4780276e0 2023-10-05T19:28:25.471 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.466+0000 7fa480614700 1 -- 172.21.15.167:0/4019939501 <== osd.2 v2:172.21.15.167:6816/30037 11 ==== osd_op_reply(71 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fa474008a80 con 0x7fa4780276e0 2023-10-05T19:28:25.471 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.466+0000 7fa480614700 1 -- 172.21.15.167:0/4019939501 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:72 5.7 5:e0000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x563c18a3ab90 con 0x563c18a3b490 2023-10-05T19:28:25.472 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.466+0000 7fa480e15700 1 -- 172.21.15.167:0/4019939501 <== osd.3 v2:172.21.15.167:6808/30036 19 ==== osd_op_reply(72 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fa478012c70 con 0x563c18a3b490 2023-10-05T19:28:25.472 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.466+0000 7fa480e15700 1 -- 172.21.15.167:0/4019939501 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:73 5.17 5:e8000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x563c18a3ab90 con 0x7fa474007030 2023-10-05T19:28:25.473 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.466+0000 7fa481616700 1 -- 172.21.15.167:0/4019939501 <== osd.0 v2:172.21.15.118:6800/27690 19 ==== osd_op_reply(73 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fa47002f080 con 0x7fa474007030 2023-10-05T19:28:25.473 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.466+0000 7fa481616700 1 -- 172.21.15.167:0/4019939501 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:74 5.f 5:f0000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x563c18a3ab90 con 0x563c18a0a5f0 2023-10-05T19:28:25.474 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.466+0000 7fa481616700 1 -- 172.21.15.167:0/4019939501 <== osd.1 v2:172.21.15.167:6800/30035 25 ==== osd_op_reply(74 [pgnls start_epoch 30 out=115b] v23'2 uv2 ondisk = 1) v8 ==== 144+0+115 (crc 0 0 0) 0x7fa47002f080 con 0x563c18a0a5f0 2023-10-05T19:28:25.474 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.466+0000 7fa481616700 1 -- 172.21.15.167:0/4019939501 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:75 5.1f 5:f8000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x563c18a3ab90 con 0x7fa4780276e0 2023-10-05T19:28:25.475 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.470+0000 7fa480614700 1 -- 172.21.15.167:0/4019939501 <== osd.2 v2:172.21.15.167:6816/30037 12 ==== osd_op_reply(75 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fa474008a80 con 0x7fa4780276e0 2023-10-05T19:28:25.475 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.470+0000 7fa4832e0e40 20 RGWRados::pool_iterate: got zonegroup_info.4f1f719d-9648-417b-b10c-e5d939825419 2023-10-05T19:28:25.475 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.470+0000 7fa4832e0e40 20 RGWRados::pool_iterate: got zone_info.9a9df965-6612-45d4-a45f-6e3c889bacc2 2023-10-05T19:28:25.475 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.470+0000 7fa4832e0e40 20 RGWRados::pool_iterate: got zone_names.default 2023-10-05T19:28:25.475 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.470+0000 7fa4832e0e40 20 RGWRados::pool_iterate: got zonegroups_names.default 2023-10-05T19:28:25.475 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.470+0000 7fa4832e0e40 20 rados->read ofs=0 len=0 2023-10-05T19:28:25.475 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.470+0000 7fa4832e0e40 1 -- 172.21.15.167:0/4019939501 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:76 5.f 5:f43b6ece:::zone_names.default:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x563c18a3e510 con 0x563c18a0a5f0 2023-10-05T19:28:25.476 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.470+0000 7fa481616700 1 -- 172.21.15.167:0/4019939501 <== osd.1 v2:172.21.15.167:6800/30035 26 ==== osd_op_reply(76 zone_names.default [read 0~46 out=46b] v0'0 uv1 ondisk = 0) v8 ==== 162+0+46 (crc 0 0 0) 0x7fa47002f080 con 0x563c18a0a5f0 2023-10-05T19:28:25.476 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.470+0000 7fa4832e0e40 20 rados_obj.operate() r=0 bl.length=46 2023-10-05T19:28:25.476 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.470+0000 7fa4832e0e40 20 rados->read ofs=0 len=0 2023-10-05T19:28:25.477 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.470+0000 7fa4832e0e40 1 -- 172.21.15.167:0/4019939501 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:77 5.1d 5:b8374eb4:::zone_info.9a9df965-6612-45d4-a45f-6e3c889bacc2:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x563c18a3e8c0 con 0x563c18a0a5f0 2023-10-05T19:28:25.477 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.470+0000 7fa481616700 1 -- 172.21.15.167:0/4019939501 <== osd.1 v2:172.21.15.167:6800/30035 27 ==== osd_op_reply(77 zone_info.9a9df965-6612-45d4-a45f-6e3c889bacc2 [read 0~874 out=874b] v0'0 uv2 ondisk = 0) v8 ==== 190+0+874 (crc 0 0 0) 0x7fa47002f080 con 0x563c18a0a5f0 2023-10-05T19:28:25.477 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.470+0000 7fa4832e0e40 20 rados_obj.operate() r=0 bl.length=874 2023-10-05T19:28:25.477 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.470+0000 7fa4832e0e40 20 rados->read ofs=0 len=0 2023-10-05T19:28:25.477 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.470+0000 7fa4832e0e40 1 -- 172.21.15.167:0/4019939501 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:78 5.f 5:f4c53578:::zonegroups_names.default:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x563c18a3edb0 con 0x563c18a0a5f0 2023-10-05T19:28:25.477 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.470+0000 7fa481616700 1 -- 172.21.15.167:0/4019939501 <== osd.1 v2:172.21.15.167:6800/30035 28 ==== osd_op_reply(78 zonegroups_names.default [read 0~46 out=46b] v0'0 uv2 ondisk = 0) v8 ==== 168+0+46 (crc 0 0 0) 0x7fa47002f080 con 0x563c18a0a5f0 2023-10-05T19:28:25.478 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.470+0000 7fa4832e0e40 20 rados_obj.operate() r=0 bl.length=46 2023-10-05T19:28:25.478 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.470+0000 7fa4832e0e40 20 rados->read ofs=0 len=0 2023-10-05T19:28:25.478 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.470+0000 7fa4832e0e40 1 -- 172.21.15.167:0/4019939501 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:79 5.c 5:30d299a5:::zonegroup_info.4f1f719d-9648-417b-b10c-e5d939825419:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x563c18a3edb0 con 0x7fa474007030 2023-10-05T19:28:25.478 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.474+0000 7fa481616700 1 -- 172.21.15.167:0/4019939501 <== osd.0 v2:172.21.15.118:6800/27690 20 ==== osd_op_reply(79 zonegroup_info.4f1f719d-9648-417b-b10c-e5d939825419 [read 0~362 out=362b] v0'0 uv1 ondisk = 0) v8 ==== 195+0+362 (crc 0 0 0) 0x7fa47002f080 con 0x7fa474007030 2023-10-05T19:28:25.479 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.474+0000 7fa4832e0e40 20 rados_obj.operate() r=0 bl.length=362 2023-10-05T19:28:25.479 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.474+0000 7fa4832e0e40 20 rados->read ofs=0 len=0 2023-10-05T19:28:25.479 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.474+0000 7fa4832e0e40 1 -- 172.21.15.167:0/4019939501 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:80 5.3 5:c0ad8cbd:::region_map:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x563c18a3ee10 con 0x7fa474007030 2023-10-05T19:28:25.479 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.474+0000 7fa481616700 1 -- 172.21.15.167:0/4019939501 <== osd.0 v2:172.21.15.118:6800/27690 21 ==== osd_op_reply(80 region_map [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 ==== 154+0+0 (crc 0 0 0) 0x7fa470050040 con 0x7fa474007030 2023-10-05T19:28:25.480 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.474+0000 7fa4832e0e40 20 rados_obj.operate() r=-2 bl.length=0 2023-10-05T19:28:25.480 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.474+0000 7fa4832e0e40 20 zone default found 2023-10-05T19:28:25.480 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.474+0000 7fa4832e0e40 4 Realm: () 2023-10-05T19:28:25.480 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.474+0000 7fa4832e0e40 4 ZoneGroup: default (4f1f719d-9648-417b-b10c-e5d939825419) 2023-10-05T19:28:25.480 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.474+0000 7fa4832e0e40 4 Zone: default (9a9df965-6612-45d4-a45f-6e3c889bacc2) 2023-10-05T19:28:25.480 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.474+0000 7fa4832e0e40 10 cannot find current period zonegroup using local zonegroup configuration 2023-10-05T19:28:25.480 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.474+0000 7fa4832e0e40 20 zonegroup default 2023-10-05T19:28:25.480 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.474+0000 7fa4832e0e40 20 rados->read ofs=0 len=0 2023-10-05T19:28:25.480 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.474+0000 7fa4832e0e40 1 -- 172.21.15.167:0/4019939501 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:81 5.3 5:c52100b6:::period_config.default:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x563c18a3ee10 con 0x7fa474007030 2023-10-05T19:28:25.481 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.474+0000 7fa481616700 1 -- 172.21.15.167:0/4019939501 <== osd.0 v2:172.21.15.118:6800/27690 22 ==== osd_op_reply(81 period_config.default [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 ==== 165+0+0 (crc 0 0 0) 0x7fa470050040 con 0x7fa474007030 2023-10-05T19:28:25.482 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.474+0000 7fa4832e0e40 20 rados_obj.operate() r=-2 bl.length=0 2023-10-05T19:28:25.482 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.474+0000 7fa4832e0e40 20 rados->read ofs=0 len=0 2023-10-05T19:28:25.482 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.474+0000 7fa4832e0e40 1 -- 172.21.15.167:0/4019939501 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:82 6.2 6:4347d321:::bucket.sync-source-hints.:head [call version.read in=11b,read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x563c18a40790 con 0x563c18a3b490 2023-10-05T19:28:25.482 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.474+0000 7fa480e15700 1 -- 172.21.15.167:0/4019939501 <== osd.3 v2:172.21.15.167:6808/30036 20 ==== osd_op_reply(82 bucket.sync-source-hints. [call,read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 ==== 211+0+0 (crc 0 0 0) 0x7fa478012c70 con 0x563c18a3b490 2023-10-05T19:28:25.482 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.474+0000 7fa4832e0e40 20 rados_obj.operate() r=-2 bl.length=0 2023-10-05T19:28:25.482 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.474+0000 7fa4832e0e40 20 rados->read ofs=0 len=0 2023-10-05T19:28:25.482 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.474+0000 7fa4832e0e40 1 -- 172.21.15.167:0/4019939501 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:83 6.b 6:d467b91b:::bucket.sync-target-hints.:head [call version.read in=11b,read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x563c18a40b40 con 0x563c18a3b490 2023-10-05T19:28:25.482 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.474+0000 7fa480e15700 1 -- 172.21.15.167:0/4019939501 <== osd.3 v2:172.21.15.167:6808/30036 21 ==== osd_op_reply(83 bucket.sync-target-hints. [call,read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 ==== 211+0+0 (crc 0 0 0) 0x7fa478012c70 con 0x563c18a3b490 2023-10-05T19:28:25.482 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.474+0000 7fa4832e0e40 20 rados_obj.operate() r=-2 bl.length=0 2023-10-05T19:28:25.483 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.474+0000 7fa4832e0e40 20 started sync module instance, tier type = 2023-10-05T19:28:25.483 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.474+0000 7fa4832e0e40 20 started zone id=9a9df965-6612-45d4-a45f-6e3c889bacc2 (name=default) with tier type = 2023-10-05T19:28:25.483 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.474+0000 7fa4832e0e40 1 -- 172.21.15.167:0/4019939501 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:84 7.1f 7:f95f44c2:::notify.0:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x563c18a3fe50 con 0x7fa4780276e0 2023-10-05T19:28:25.484 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.478+0000 7fa480614700 1 -- 172.21.15.167:0/4019939501 <== osd.2 v2:172.21.15.167:6816/30037 13 ==== osd_op_reply(84 notify.0 [create] v30'21 uv21 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7fa474008a80 con 0x7fa4780276e0 2023-10-05T19:28:25.485 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.478+0000 7fa4832e0e40 1 -- 172.21.15.167:0/4019939501 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:85 7.1f 7:f95f44c2:::notify.0:head [watch watch cookie 94816111428224] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x563c18a3fa80 con 0x7fa4780276e0 2023-10-05T19:28:25.485 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.478+0000 7fa4832e0e40 1 -- 172.21.15.167:0/4019939501 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:86 7.0 7:05bf5b68:::notify.1:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x563c18a41900 con 0x563c18a0a5f0 2023-10-05T19:28:25.487 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.482+0000 7fa480614700 1 -- 172.21.15.167:0/4019939501 <== osd.2 v2:172.21.15.167:6816/30037 14 ==== osd_op_reply(85 notify.0 [watch watch cookie 94816111428224] v30'22 uv21 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7fa474008a80 con 0x7fa4780276e0 2023-10-05T19:28:25.488 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.482+0000 7fa481616700 1 -- 172.21.15.167:0/4019939501 <== osd.1 v2:172.21.15.167:6800/30035 29 ==== osd_op_reply(86 notify.1 [create] v30'21 uv21 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7fa470050040 con 0x563c18a0a5f0 2023-10-05T19:28:25.488 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.482+0000 7fa4832e0e40 1 -- 172.21.15.167:0/4019939501 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:87 7.0 7:05bf5b68:::notify.1:head [watch watch cookie 94816111432720] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x563c18a41d50 con 0x563c18a0a5f0 2023-10-05T19:28:25.488 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.482+0000 7fa4832e0e40 1 -- 172.21.15.167:0/4019939501 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:88 7.15 7:a93a5511:::notify.2:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x563c18a425f0 con 0x563c18a3b490 2023-10-05T19:28:25.490 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.486+0000 7fa480e15700 1 -- 172.21.15.167:0/4019939501 <== osd.3 v2:172.21.15.167:6808/30036 22 ==== osd_op_reply(88 notify.2 [create] v30'21 uv21 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7fa478012c70 con 0x563c18a3b490 2023-10-05T19:28:25.491 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.486+0000 7fa4832e0e40 1 -- 172.21.15.167:0/4019939501 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:89 7.15 7:a93a5511:::notify.2:head [watch watch cookie 94816111435088] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x563c18a42e90 con 0x563c18a3b490 2023-10-05T19:28:25.491 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.486+0000 7fa4832e0e40 1 -- 172.21.15.167:0/4019939501 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:90 7.e 7:7759931f:::notify.3:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x563c18a43730 con 0x7fa4780276e0 2023-10-05T19:28:25.491 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.486+0000 7fa481616700 1 -- 172.21.15.167:0/4019939501 <== osd.1 v2:172.21.15.167:6800/30035 30 ==== osd_op_reply(87 notify.1 [watch watch cookie 94816111432720] v30'22 uv21 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7fa470050040 con 0x563c18a0a5f0 2023-10-05T19:28:25.493 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.486+0000 7fa480614700 1 -- 172.21.15.167:0/4019939501 <== osd.2 v2:172.21.15.167:6816/30037 15 ==== osd_op_reply(90 notify.3 [create] v30'21 uv21 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7fa474008a80 con 0x7fa4780276e0 2023-10-05T19:28:25.493 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.486+0000 7fa4832e0e40 1 -- 172.21.15.167:0/4019939501 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:91 7.e 7:7759931f:::notify.3:head [watch watch cookie 94816111433824] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x563c18a43360 con 0x7fa4780276e0 2023-10-05T19:28:25.493 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.486+0000 7fa4832e0e40 1 -- 172.21.15.167:0/4019939501 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:92 7.d 7:b4812045:::notify.4:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x563c18a447b0 con 0x563c18a0a5f0 2023-10-05T19:28:25.494 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.486+0000 7fa480e15700 1 -- 172.21.15.167:0/4019939501 <== osd.3 v2:172.21.15.167:6808/30036 23 ==== osd_op_reply(89 notify.2 [watch watch cookie 94816111435088] v30'22 uv21 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7fa478012c70 con 0x563c18a3b490 2023-10-05T19:28:25.496 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.490+0000 7fa480614700 1 -- 172.21.15.167:0/4019939501 <== osd.2 v2:172.21.15.167:6816/30037 16 ==== osd_op_reply(91 notify.3 [watch watch cookie 94816111433824] v30'22 uv21 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7fa474008a80 con 0x7fa4780276e0 2023-10-05T19:28:25.496 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.490+0000 7fa481616700 1 -- 172.21.15.167:0/4019939501 <== osd.1 v2:172.21.15.167:6800/30035 31 ==== osd_op_reply(92 notify.4 [create] v30'21 uv21 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7fa470050040 con 0x563c18a0a5f0 2023-10-05T19:28:25.496 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.490+0000 7fa4832e0e40 1 -- 172.21.15.167:0/4019939501 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:93 7.d 7:b4812045:::notify.4:head [watch watch cookie 94816111442656] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x563c18a44280 con 0x563c18a0a5f0 2023-10-05T19:28:25.496 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.490+0000 7fa4832e0e40 1 -- 172.21.15.167:0/4019939501 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:94 7.3 7:c609908c:::notify.5:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x563c18a450a0 con 0x7fa474007030 2023-10-05T19:28:25.498 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.494+0000 7fa481616700 1 -- 172.21.15.167:0/4019939501 <== osd.0 v2:172.21.15.118:6800/27690 23 ==== osd_op_reply(94 notify.5 [create] v30'21 uv21 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7fa470050040 con 0x7fa474007030 2023-10-05T19:28:25.499 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.494+0000 7fa4832e0e40 1 -- 172.21.15.167:0/4019939501 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:95 7.3 7:c609908c:::notify.5:head [watch watch cookie 94816111446880] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x563c18a454a0 con 0x7fa474007030 2023-10-05T19:28:25.499 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.494+0000 7fa4832e0e40 1 -- 172.21.15.167:0/4019939501 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:96 7.14 7:2b04a3e9:::notify.6:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x563c18a45f40 con 0x7fa474007030 2023-10-05T19:28:25.499 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.494+0000 7fa481616700 1 -- 172.21.15.167:0/4019939501 <== osd.1 v2:172.21.15.167:6800/30035 32 ==== osd_op_reply(93 notify.4 [watch watch cookie 94816111442656] v30'22 uv21 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7fa470050040 con 0x563c18a0a5f0 2023-10-05T19:28:25.501 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.494+0000 7fa481616700 1 -- 172.21.15.167:0/4019939501 <== osd.0 v2:172.21.15.118:6800/27690 24 ==== osd_op_reply(96 notify.6 [create] v30'21 uv21 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7fa470050040 con 0x7fa474007030 2023-10-05T19:28:25.501 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.494+0000 7fa4832e0e40 1 -- 172.21.15.167:0/4019939501 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:97 7.14 7:2b04a3e9:::notify.6:head [watch watch cookie 94816111438240] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x563c18a45f40 con 0x7fa474007030 2023-10-05T19:28:25.501 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.494+0000 7fa4832e0e40 1 -- 172.21.15.167:0/4019939501 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:98 7.9 7:93e5b521:::notify.7:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x563c18a46950 con 0x7fa474007030 2023-10-05T19:28:25.501 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.494+0000 7fa481616700 1 -- 172.21.15.167:0/4019939501 <== osd.0 v2:172.21.15.118:6800/27690 25 ==== osd_op_reply(95 notify.5 [watch watch cookie 94816111446880] v30'22 uv21 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7fa470050040 con 0x7fa474007030 2023-10-05T19:28:25.504 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.498+0000 7fa481616700 1 -- 172.21.15.167:0/4019939501 <== osd.0 v2:172.21.15.118:6800/27690 26 ==== osd_op_reply(98 notify.7 [create] v30'21 uv21 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7fa470050040 con 0x7fa474007030 2023-10-05T19:28:25.504 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.498+0000 7fa4832e0e40 1 -- 172.21.15.167:0/4019939501 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:99 7.9 7:93e5b521:::notify.7:head [watch watch cookie 94816111447984] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x563c18a46950 con 0x7fa474007030 2023-10-05T19:28:25.504 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.498+0000 7fa4832e0e40 20 add_watcher() i=0 2023-10-05T19:28:25.504 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.498+0000 7fa4832e0e40 20 add_watcher() i=1 2023-10-05T19:28:25.504 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.498+0000 7fa4832e0e40 20 add_watcher() i=2 2023-10-05T19:28:25.504 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.498+0000 7fa4832e0e40 20 add_watcher() i=3 2023-10-05T19:28:25.504 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.498+0000 7fa4832e0e40 20 add_watcher() i=4 2023-10-05T19:28:25.504 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.498+0000 7fa4832e0e40 20 add_watcher() i=5 2023-10-05T19:28:25.505 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.498+0000 7fa481616700 1 -- 172.21.15.167:0/4019939501 <== osd.0 v2:172.21.15.118:6800/27690 27 ==== osd_op_reply(97 notify.6 [watch watch cookie 94816111438240] v30'22 uv21 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7fa470050040 con 0x7fa474007030 2023-10-05T19:28:25.505 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.498+0000 7fa4832e0e40 20 add_watcher() i=6 2023-10-05T19:28:25.506 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.498+0000 7fa481616700 1 -- 172.21.15.167:0/4019939501 <== osd.0 v2:172.21.15.118:6800/27690 28 ==== osd_op_reply(99 notify.7 [watch watch cookie 94816111447984] v30'22 uv21 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7fa470050040 con 0x7fa474007030 2023-10-05T19:28:25.506 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.498+0000 7fa4832e0e40 20 add_watcher() i=7 2023-10-05T19:28:25.506 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.498+0000 7fa4832e0e40 2 all 8 watchers are set, enabling cache 2023-10-05T19:28:25.506 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.498+0000 7fa4832e0e40 1 -- 172.21.15.167:0/4019939501 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:100 6.4 6:22d26bf9:::data_loggenerations_metadata:head [call version.check_conds in=74b,call version.read in=11b,read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x563c18a46950 con 0x563c18a0a5f0 2023-10-05T19:28:25.506 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.502+0000 7fa481616700 1 -- 172.21.15.167:0/4019939501 <== osd.1 v2:172.21.15.167:6800/30035 33 ==== osd_op_reply(100 data_loggenerations_metadata [call,call out=48b,read 0~28 out=28b] v0'0 uv1 ondisk = 0) v8 ==== 256+0+76 (crc 0 0 0) 0x7fa470050040 con 0x563c18a0a5f0 2023-10-05T19:28:25.507 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.502+0000 7fa4832e0e40 1 -- 172.21.15.167:0/4019939501 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:101 6.4 6:22d26bf9:::data_loggenerations_metadata:head [watch watch cookie 94816111453280] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x563c18a46da0 con 0x563c18a0a5f0 2023-10-05T19:28:25.509 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.502+0000 7fa481616700 1 -- 172.21.15.167:0/4019939501 <== osd.1 v2:172.21.15.167:6800/30035 34 ==== osd_op_reply(101 data_loggenerations_metadata [watch watch cookie 94816111453280] v30'48 uv1 ondisk = 0) v8 ==== 172+0+0 (crc 0 0 0) 0x7fa470050040 con 0x563c18a0a5f0 2023-10-05T19:28:25.509 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.502+0000 7fa4527e4700 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2023-10-05T19:28:25.510 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.502+0000 7fa4832e0e40 20 check_secure_mon_conn(): auth registy supported: methods=[2] modes=[2,1] 2023-10-05T19:28:25.510 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.502+0000 7fa4832e0e40 20 check_secure_mon_conn(): mode 1 is insecure 2023-10-05T19:28:25.510 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.502+0000 7fa4832e0e40 1 Processor -- start 2023-10-05T19:28:25.510 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.506+0000 7fa4832e0e40 1 -- start start 2023-10-05T19:28:25.510 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.506+0000 7fa4832e0e40 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x563c18a53a80 0x563c18a53e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-05T19:28:25.511 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.506+0000 7fa4832e0e40 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x563c18a54390 con 0x563c18a53a80 2023-10-05T19:28:25.511 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.506+0000 7fa480614700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x563c18a53a80 0x563c18a53e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-05T19:28:25.511 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.506+0000 7fa480614700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x563c18a53a80 0x563c18a53e50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.118:3300/0 says I am v2:172.21.15.167:57134/0 (socket says 172.21.15.167:57134) 2023-10-05T19:28:25.511 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.506+0000 7fa480614700 1 -- 172.21.15.167:0/1172499350 learned_addr learned my addr 172.21.15.167:0/1172499350 (peer_addr_for_me v2:172.21.15.167:0/0) 2023-10-05T19:28:25.511 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.506+0000 7fa480614700 1 -- 172.21.15.167:0/1172499350 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x563c18a544d0 con 0x563c18a53a80 2023-10-05T19:28:25.511 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.506+0000 7fa480614700 1 --2- 172.21.15.167:0/1172499350 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x563c18a53a80 0x563c18a53e50 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x563c18a07090 tx=0x7fa474012000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ecb453d433097c37 server_cookie=0 in_seq=0 out_seq=0 2023-10-05T19:28:25.512 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.506+0000 7fa451fe3700 1 -- 172.21.15.167:0/1172499350 <== mon.0 v2:172.21.15.118:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fa4740127d0 con 0x563c18a53a80 2023-10-05T19:28:25.512 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.506+0000 7fa451fe3700 1 -- 172.21.15.167:0/1172499350 <== mon.0 v2:172.21.15.118:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa47401cdf0 con 0x563c18a53a80 2023-10-05T19:28:25.512 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.506+0000 7fa451fe3700 1 -- 172.21.15.167:0/1172499350 <== mon.0 v2:172.21.15.118:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fa47402a8a0 con 0x563c18a53a80 2023-10-05T19:28:25.513 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.506+0000 7fa4832e0e40 1 -- 172.21.15.167:0/1172499350 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x563c18a53a80 msgr2=0x563c18a53e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-05T19:28:25.513 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.506+0000 7fa4832e0e40 1 --2- 172.21.15.167:0/1172499350 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x563c18a53a80 0x563c18a53e50 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x563c18a07090 tx=0x7fa474012000 comp rx=0 tx=0).stop 2023-10-05T19:28:25.513 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.506+0000 7fa4832e0e40 1 -- 172.21.15.167:0/1172499350 shutdown_connections 2023-10-05T19:28:25.513 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.506+0000 7fa4832e0e40 1 --2- 172.21.15.167:0/1172499350 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x563c18a53a80 0x563c18a53e50 unknown :-1 s=CLOSED pgs=180 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-05T19:28:25.514 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.506+0000 7fa4832e0e40 1 -- 172.21.15.167:0/1172499350 >> 172.21.15.167:0/1172499350 conn(0x563c18a4ee40 msgr2=0x563c18a4cff0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-05T19:28:25.514 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.506+0000 7fa4832e0e40 1 -- 172.21.15.167:0/1172499350 shutdown_connections 2023-10-05T19:28:25.514 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.506+0000 7fa4832e0e40 1 -- 172.21.15.167:0/1172499350 wait complete. 2023-10-05T19:28:25.515 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.506+0000 7fa4832e0e40 1 Processor -- start 2023-10-05T19:28:25.515 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.510+0000 7fa4832e0e40 1 -- start start 2023-10-05T19:28:25.515 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.510+0000 7fa4832e0e40 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x563c18a53a80 0x563c18a59550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-05T19:28:25.515 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.510+0000 7fa4832e0e40 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x563c18a5a550 con 0x563c18a53a80 2023-10-05T19:28:25.515 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.510+0000 7fa480614700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x563c18a53a80 0x563c18a59550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-05T19:28:25.515 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.510+0000 7fa480614700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x563c18a53a80 0x563c18a59550 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.118:3300/0 says I am v2:172.21.15.167:57140/0 (socket says 172.21.15.167:57140) 2023-10-05T19:28:25.515 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.510+0000 7fa480614700 1 -- 172.21.15.167:0/1757840981 learned_addr learned my addr 172.21.15.167:0/1757840981 (peer_addr_for_me v2:172.21.15.167:0/0) 2023-10-05T19:28:25.515 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.510+0000 7fa480614700 1 -- 172.21.15.167:0/1757840981 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x563c18a06c20 con 0x563c18a53a80 2023-10-05T19:28:25.516 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.510+0000 7fa480614700 1 --2- 172.21.15.167:0/1757840981 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x563c18a53a80 0x563c18a59550 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x563c18a4db60 tx=0x7fa47402f000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-05T19:28:25.516 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.510+0000 7fa3c3fff700 1 -- 172.21.15.167:0/1757840981 <== mon.0 v2:172.21.15.118:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fa47403e910 con 0x563c18a53a80 2023-10-05T19:28:25.516 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.510+0000 7fa3c3fff700 1 -- 172.21.15.167:0/1757840981 <== mon.0 v2:172.21.15.118:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa47403ea90 con 0x563c18a53a80 2023-10-05T19:28:25.517 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.510+0000 7fa4832e0e40 1 -- 172.21.15.167:0/1757840981 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x563c18a59b50 con 0x563c18a53a80 2023-10-05T19:28:25.517 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.510+0000 7fa3c3fff700 1 -- 172.21.15.167:0/1757840981 <== mon.0 v2:172.21.15.118:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fa47403ec40 con 0x563c18a53a80 2023-10-05T19:28:25.517 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.510+0000 7fa4832e0e40 1 -- 172.21.15.167:0/1757840981 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x563c18a59f90 con 0x563c18a53a80 2023-10-05T19:28:25.517 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.510+0000 7fa3c3fff700 1 -- 172.21.15.167:0/1757840981 <== mon.0 v2:172.21.15.118:3300/0 4 ==== mgrmap(e 5) v1 ==== 40329+0+0 (secure 0 0 0) 0x7fa474046070 con 0x563c18a53a80 2023-10-05T19:28:25.518 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.514+0000 7fa3c3fff700 1 --2- 172.21.15.167:0/1757840981 >> [v2:172.21.15.167:6824/29889,v1:172.21.15.167:6825/29889] conn(0x7fa3c8030c00 0x7fa3c80330b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-05T19:28:25.518 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.514+0000 7fa3c3fff700 1 -- 172.21.15.167:0/1757840981 <== mon.0 v2:172.21.15.118:3300/0 5 ==== osd_map(30..30 src has 1..30) v4 ==== 6057+0+0 (secure 0 0 0) 0x7fa47402f070 con 0x563c18a53a80 2023-10-05T19:28:25.518 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.514+0000 7fa481616700 1 --2- 172.21.15.167:0/1757840981 >> [v2:172.21.15.167:6824/29889,v1:172.21.15.167:6825/29889] conn(0x7fa3c8030c00 0x7fa3c80330b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-05T19:28:25.518 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.514+0000 7fa4832e0e40 5 note: GC not initialized 2023-10-05T19:28:25.519 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.514+0000 7fa481616700 1 --2- 172.21.15.167:0/1757840981 >> [v2:172.21.15.167:6824/29889,v1:172.21.15.167:6825/29889] conn(0x7fa3c8030c00 0x7fa3c80330b0 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x563c18a45c60 tx=0x7fa470000f90 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-05T19:28:25.520 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.514+0000 7fa3c17fa700 20 reqs_thread_entry: start 2023-10-05T19:28:25.520 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.514+0000 7fa3baffd700 20 reqs_thread_entry: start 2023-10-05T19:28:25.520 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.514+0000 7fa4832e0e40 20 init_complete bucket index max shards: 11 2023-10-05T19:28:25.521 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.514+0000 7fa4832e0e40 10 rgw notify: Started notification manager with: 1 workers 2023-10-05T19:28:25.521 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.514+0000 7fa3b97fa700 20 rgw notify: INFO: next queues processing will happen at: Thu Oct 5 19:28:55 2023 2023-10-05T19:28:25.521 INFO:teuthology.orchestra.run.smithi167.stderr: 2023-10-05T19:28:25.521 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.514+0000 7fa3b8ff9700 20 reqs_thread_entry: start 2023-10-05T19:28:25.521 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.514+0000 7fa4832e0e40 20 get_system_obj_state: rctx=0x7ffdbc0289a8 obj=default.rgw.meta:users.uid:foo state=0x563c188dd0b0 s->prefetch_data=0 2023-10-05T19:28:25.522 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.514+0000 7fa4832e0e40 1 -- 172.21.15.167:0/4019939501 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:102 8.18 8:1e6ca5ed:users.uid::foo:head [call version.read in=11b,getxattrs,stat] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x563c18ac1c80 con 0x563c18a0a5f0 2023-10-05T19:28:25.522 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.514+0000 7fa481616700 1 -- 172.21.15.167:0/4019939501 <== osd.1 v2:172.21.15.167:6800/30035 35 ==== osd_op_reply(102 foo [call,getxattrs,stat] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 ==== 231+0+0 (crc 0 0 0) 0x7fa470050040 con 0x563c18a0a5f0 2023-10-05T19:28:25.522 INFO:teuthology.orchestra.run.smithi167.stderr:could not fetch user info: no user info saved 2023-10-05T19:28:25.522 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.518+0000 7fa4832e0e40 1 -- 172.21.15.167:0/4019939501 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:103 7.1f 7:f95f44c2:::notify.0:head [watch unwatch cookie 94816111428224] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x563c18ac1c80 con 0x7fa4780276e0 2023-10-05T19:28:25.525 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.518+0000 7fa480614700 1 -- 172.21.15.167:0/4019939501 <== osd.2 v2:172.21.15.167:6816/30037 17 ==== osd_op_reply(103 notify.0 [watch unwatch cookie 94816111428224] v30'23 uv21 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7fa474032040 con 0x7fa4780276e0 2023-10-05T19:28:25.525 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.518+0000 7fa4832e0e40 20 remove_watcher() i=0 2023-10-05T19:28:25.525 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.518+0000 7fa4832e0e40 2 removed watcher, disabling cache 2023-10-05T19:28:25.525 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.518+0000 7fa4832e0e40 1 -- 172.21.15.167:0/4019939501 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:104 7.0 7:05bf5b68:::notify.1:head [watch unwatch cookie 94816111432720] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x563c18ac1c80 con 0x563c18a0a5f0 2023-10-05T19:28:25.528 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.522+0000 7fa481616700 1 -- 172.21.15.167:0/4019939501 <== osd.1 v2:172.21.15.167:6800/30035 36 ==== osd_op_reply(104 notify.1 [watch unwatch cookie 94816111432720] v30'23 uv21 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7fa470050040 con 0x563c18a0a5f0 2023-10-05T19:28:25.528 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.522+0000 7fa4832e0e40 20 remove_watcher() i=1 2023-10-05T19:28:25.528 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.522+0000 7fa4832e0e40 1 -- 172.21.15.167:0/4019939501 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:105 7.15 7:a93a5511:::notify.2:head [watch unwatch cookie 94816111435088] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x563c18ac1c80 con 0x563c18a3b490 2023-10-05T19:28:25.531 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.526+0000 7fa480e15700 1 -- 172.21.15.167:0/4019939501 <== osd.3 v2:172.21.15.167:6808/30036 24 ==== osd_op_reply(105 notify.2 [watch unwatch cookie 94816111435088] v30'23 uv21 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7fa478012c70 con 0x563c18a3b490 2023-10-05T19:28:25.531 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.526+0000 7fa4832e0e40 20 remove_watcher() i=2 2023-10-05T19:28:25.531 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.526+0000 7fa4832e0e40 1 -- 172.21.15.167:0/4019939501 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:106 7.e 7:7759931f:::notify.3:head [watch unwatch cookie 94816111433824] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x563c18a41410 con 0x7fa4780276e0 2023-10-05T19:28:25.533 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.526+0000 7fa480614700 1 -- 172.21.15.167:0/4019939501 <== osd.2 v2:172.21.15.167:6816/30037 18 ==== osd_op_reply(106 notify.3 [watch unwatch cookie 94816111433824] v30'23 uv21 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7fa474032040 con 0x7fa4780276e0 2023-10-05T19:28:25.533 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.526+0000 7fa4832e0e40 20 remove_watcher() i=3 2023-10-05T19:28:25.533 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.526+0000 7fa4832e0e40 1 -- 172.21.15.167:0/4019939501 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:107 7.d 7:b4812045:::notify.4:head [watch unwatch cookie 94816111442656] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x563c18a41d50 con 0x563c18a0a5f0 2023-10-05T19:28:25.535 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.530+0000 7fa481616700 1 -- 172.21.15.167:0/4019939501 <== osd.1 v2:172.21.15.167:6800/30035 37 ==== osd_op_reply(107 notify.4 [watch unwatch cookie 94816111442656] v30'23 uv21 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7fa470050040 con 0x563c18a0a5f0 2023-10-05T19:28:25.536 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.530+0000 7fa4832e0e40 20 remove_watcher() i=4 2023-10-05T19:28:25.536 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.530+0000 7fa4832e0e40 1 -- 172.21.15.167:0/4019939501 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:108 7.3 7:c609908c:::notify.5:head [watch unwatch cookie 94816111446880] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x563c18a43ae0 con 0x7fa474007030 2023-10-05T19:28:25.538 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.534+0000 7fa481616700 1 -- 172.21.15.167:0/4019939501 <== osd.0 v2:172.21.15.118:6800/27690 29 ==== osd_op_reply(108 notify.5 [watch unwatch cookie 94816111446880] v30'23 uv21 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7fa470050040 con 0x7fa474007030 2023-10-05T19:28:25.538 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.534+0000 7fa4832e0e40 20 remove_watcher() i=5 2023-10-05T19:28:25.538 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.534+0000 7fa4832e0e40 1 -- 172.21.15.167:0/4019939501 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:109 7.14 7:2b04a3e9:::notify.6:head [watch unwatch cookie 94816111438240] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x563c18a41d50 con 0x7fa474007030 2023-10-05T19:28:25.541 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.534+0000 7fa481616700 1 -- 172.21.15.167:0/4019939501 <== osd.0 v2:172.21.15.118:6800/27690 30 ==== osd_op_reply(109 notify.6 [watch unwatch cookie 94816111438240] v30'23 uv21 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7fa470050040 con 0x7fa474007030 2023-10-05T19:28:25.541 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.534+0000 7fa4832e0e40 20 remove_watcher() i=6 2023-10-05T19:28:25.541 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.534+0000 7fa4832e0e40 1 -- 172.21.15.167:0/4019939501 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:110 7.9 7:93e5b521:::notify.7:head [watch unwatch cookie 94816111447984] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x563c18a41d50 con 0x7fa474007030 2023-10-05T19:28:25.544 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.538+0000 7fa481616700 1 -- 172.21.15.167:0/4019939501 <== osd.0 v2:172.21.15.118:6800/27690 31 ==== osd_op_reply(110 notify.7 [watch unwatch cookie 94816111447984] v30'23 uv21 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7fa470050040 con 0x7fa474007030 2023-10-05T19:28:25.544 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.538+0000 7fa4832e0e40 20 remove_watcher() i=7 2023-10-05T19:28:25.546 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.538+0000 7fa4527e4700 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2023-10-05T19:28:25.546 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.538+0000 7fa4832e0e40 1 -- 172.21.15.167:0/4019939501 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:111 6.4 6:22d26bf9:::data_loggenerations_metadata:head [watch unwatch cookie 94816111453280] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x563c18a41d50 con 0x563c18a0a5f0 2023-10-05T19:28:25.549 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.542+0000 7fa481616700 1 -- 172.21.15.167:0/4019939501 <== osd.1 v2:172.21.15.167:6800/30035 38 ==== osd_op_reply(111 data_loggenerations_metadata [watch unwatch cookie 94816111453280] v30'49 uv1 ondisk = 0) v8 ==== 172+0+0 (crc 0 0 0) 0x7fa470050040 con 0x563c18a0a5f0 2023-10-05T19:28:25.549 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.542+0000 7fa4832e0e40 1 -- 172.21.15.167:0/4019939501 >> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] conn(0x7fa474007030 msgr2=0x7fa47401c410 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-05T19:28:25.550 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.542+0000 7fa4832e0e40 1 --2- 172.21.15.167:0/4019939501 >> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] conn(0x7fa474007030 0x7fa47401c410 crc :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-05T19:28:25.550 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.542+0000 7fa4832e0e40 1 -- 172.21.15.167:0/4019939501 >> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] conn(0x563c18a0a5f0 msgr2=0x563c18a379f0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-05T19:28:25.550 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.542+0000 7fa4832e0e40 1 --2- 172.21.15.167:0/4019939501 >> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] conn(0x563c18a0a5f0 0x563c18a379f0 crc :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-05T19:28:25.550 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.542+0000 7fa4832e0e40 1 -- 172.21.15.167:0/4019939501 >> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] conn(0x7fa4780276e0 msgr2=0x7fa4780441e0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-05T19:28:25.550 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.542+0000 7fa4832e0e40 1 --2- 172.21.15.167:0/4019939501 >> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] conn(0x7fa4780276e0 0x7fa4780441e0 crc :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-05T19:28:25.550 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.542+0000 7fa4832e0e40 1 -- 172.21.15.167:0/4019939501 >> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] conn(0x563c18a3b490 msgr2=0x563c18a3d870 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-05T19:28:25.550 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.542+0000 7fa4832e0e40 1 --2- 172.21.15.167:0/4019939501 >> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] conn(0x563c18a3b490 0x563c18a3d870 crc :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-05T19:28:25.550 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.542+0000 7fa4832e0e40 1 -- 172.21.15.167:0/4019939501 >> [v2:172.21.15.167:6824/29889,v1:172.21.15.167:6825/29889] conn(0x7fa46c030c00 msgr2=0x7fa46c0330b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-05T19:28:25.550 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.542+0000 7fa4832e0e40 1 --2- 172.21.15.167:0/4019939501 >> [v2:172.21.15.167:6824/29889,v1:172.21.15.167:6825/29889] conn(0x7fa46c030c00 0x7fa46c0330b0 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7fa474005a90 tx=0x7fa474015000 comp rx=0 tx=0).stop 2023-10-05T19:28:25.550 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.542+0000 7fa481616700 1 -- 172.21.15.167:0/4019939501 reap_dead start 2023-10-05T19:28:25.551 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.542+0000 7fa4832e0e40 1 -- 172.21.15.167:0/4019939501 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x563c18a06010 msgr2=0x563c18a2bf60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-05T19:28:25.551 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.542+0000 7fa4832e0e40 1 --2- 172.21.15.167:0/4019939501 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x563c18a06010 0x563c18a2bf60 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7fa478006bd0 tx=0x7fa478014000 comp rx=0 tx=0).stop 2023-10-05T19:28:25.551 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.542+0000 7fa4832e0e40 1 -- 172.21.15.167:0/4019939501 shutdown_connections 2023-10-05T19:28:25.551 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.542+0000 7fa4832e0e40 1 --2- 172.21.15.167:0/4019939501 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x563c18a06010 0x563c18a2bf60 unknown :-1 s=CLOSED pgs=179 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-05T19:28:25.551 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.542+0000 7fa4832e0e40 1 -- 172.21.15.167:0/4019939501 >> 172.21.15.167:0/4019939501 conn(0x563c18979860 msgr2=0x563c18a1f7d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-05T19:28:25.551 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.542+0000 7fa4832e0e40 1 -- 172.21.15.167:0/4019939501 shutdown_connections 2023-10-05T19:28:25.551 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.542+0000 7fa4832e0e40 1 -- 172.21.15.167:0/4019939501 wait complete. 2023-10-05T19:28:25.552 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.546+0000 7fa4832e0e40 1 -- 172.21.15.167:0/1757840981 >> [v2:172.21.15.167:6824/29889,v1:172.21.15.167:6825/29889] conn(0x7fa3c8030c00 msgr2=0x7fa3c80330b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-05T19:28:25.552 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.546+0000 7fa4832e0e40 1 --2- 172.21.15.167:0/1757840981 >> [v2:172.21.15.167:6824/29889,v1:172.21.15.167:6825/29889] conn(0x7fa3c8030c00 0x7fa3c80330b0 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x563c18a45c60 tx=0x7fa470000f90 comp rx=0 tx=0).stop 2023-10-05T19:28:25.552 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.546+0000 7fa4832e0e40 1 -- 172.21.15.167:0/1757840981 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x563c18a53a80 msgr2=0x563c18a59550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-05T19:28:25.552 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.546+0000 7fa4832e0e40 1 --2- 172.21.15.167:0/1757840981 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x563c18a53a80 0x563c18a59550 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x563c18a4db60 tx=0x7fa47402f000 comp rx=0 tx=0).stop 2023-10-05T19:28:25.552 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.546+0000 7fa4832e0e40 1 -- 172.21.15.167:0/1757840981 shutdown_connections 2023-10-05T19:28:25.552 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.546+0000 7fa4832e0e40 1 --2- 172.21.15.167:0/1757840981 >> [v2:172.21.15.167:6824/29889,v1:172.21.15.167:6825/29889] conn(0x7fa3c8030c00 0x7fa3c80330b0 unknown :-1 s=CLOSED pgs=67 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-05T19:28:25.552 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.546+0000 7fa4832e0e40 1 --2- 172.21.15.167:0/1757840981 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x563c18a53a80 0x563c18a59550 unknown :-1 s=CLOSED pgs=181 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-05T19:28:25.553 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.546+0000 7fa4832e0e40 1 -- 172.21.15.167:0/1757840981 >> 172.21.15.167:0/1757840981 conn(0x563c18a4ee40 msgr2=0x563c18a4cf40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-05T19:28:25.553 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.546+0000 7fa4832e0e40 1 -- 172.21.15.167:0/1757840981 shutdown_connections 2023-10-05T19:28:25.553 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.546+0000 7fa4832e0e40 1 -- 172.21.15.167:0/1757840981 wait complete. 2023-10-05T19:28:25.557 DEBUG:teuthology.orchestra.run:got remote process result: 22 2023-10-05T19:28:25.557 INFO:tasks.util.rgw:rgwadmin: client.0 : ['user', 'create', '--uid', 'foo', '--display-name', 'Foo', '--email', 'foo@foo.com', '--access-key', '9te6NH5mcdcq0Tc5i8i1', '--secret', 'Ny4IOauQoL18Gp2zM7lC1vLmoawgqcYP/YGcWfXu', '--max-buckets', '4'] 2023-10-05T19:28:25.557 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'ceph', 'user', 'create', '--uid', 'foo', '--display-name', 'Foo', '--email', 'foo@foo.com', '--access-key', '9te6NH5mcdcq0Tc5i8i1', '--secret', 'Ny4IOauQoL18Gp2zM7lC1vLmoawgqcYP/YGcWfXu', '--max-buckets', '4'] 2023-10-05T19:28:25.557 DEBUG:teuthology.orchestra.run.smithi167:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster ceph user create --uid foo --display-name Foo --email foo@foo.com --access-key 9te6NH5mcdcq0Tc5i8i1 --secret Ny4IOauQoL18Gp2zM7lC1vLmoawgqcYP/YGcWfXu --max-buckets 4 2023-10-05T19:28:25.602 INFO:teuthology.orchestra.run.smithi167.stderr:ignoring --setuser ceph since I am not root 2023-10-05T19:28:25.603 INFO:teuthology.orchestra.run.smithi167.stderr:ignoring --setgroup ceph since I am not root 2023-10-05T19:28:25.603 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.598+0000 7fade5b46e40 1 Processor -- start 2023-10-05T19:28:25.603 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.598+0000 7fade5b46e40 1 -- start start 2023-10-05T19:28:25.604 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.598+0000 7fade5b46e40 1 --2- >> v2:172.21.15.118:3300/0 conn(0x5559d5e3efc0 0x5559d5e3f390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-05T19:28:25.604 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.598+0000 7fade5b46e40 1 -- --> v1:172.21.15.118:6789/0 -- auth(proto 0 26 bytes epoch 0) v1 -- 0x5559d5d89bd0 con 0x5559d5e3f8d0 2023-10-05T19:28:25.604 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.598+0000 7fade5b46e40 1 -- --> v2:172.21.15.118:3300/0 -- mon_getmap magic: 0 v1 -- 0x5559d5d46ee0 con 0x5559d5e3efc0 2023-10-05T19:28:25.604 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.598+0000 7fade367b700 1 --2- >> v2:172.21.15.118:3300/0 conn(0x5559d5e3efc0 0x5559d5e3f390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-05T19:28:25.604 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.598+0000 7fade2e7a700 1 --1- >> v1:172.21.15.118:6789/0 conn(0x5559d5e3f8d0 0x5559d5e40370 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.118:6789/0 says I am v1:172.21.15.167:56358/0 (socket says 172.21.15.167:56358) 2023-10-05T19:28:25.604 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.598+0000 7fade2e7a700 1 -- 172.21.15.167:0/2578315676 learned_addr learned my addr 172.21.15.167:0/2578315676 (peer_addr_for_me v1:172.21.15.167:0/0) 2023-10-05T19:28:25.605 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.598+0000 7fade367b700 1 -- 172.21.15.167:0/2578315676 >> v1:172.21.15.118:6789/0 conn(0x5559d5e3f8d0 legacy=0x5559d5e40370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-05T19:28:25.605 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.598+0000 7fade367b700 1 -- 172.21.15.167:0/2578315676 --> v2:172.21.15.118:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5559d5e40a80 con 0x5559d5e3efc0 2023-10-05T19:28:25.605 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.598+0000 7fade367b700 1 --2- 172.21.15.167:0/2578315676 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5559d5e3efc0 0x5559d5e3f390 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7fadd400a5d0 tx=0x7fadd4005a50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1c0433f7bfcf4c14 server_cookie=0 in_seq=0 out_seq=0 2023-10-05T19:28:25.606 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.598+0000 7fade2679700 1 -- 172.21.15.167:0/2578315676 <== mon.0 v2:172.21.15.118:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fadd40086a0 con 0x5559d5e3efc0 2023-10-05T19:28:25.606 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.598+0000 7fade2679700 1 -- 172.21.15.167:0/2578315676 <== mon.0 v2:172.21.15.118:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fadd4008800 con 0x5559d5e3efc0 2023-10-05T19:28:25.606 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.598+0000 7fade2679700 1 -- 172.21.15.167:0/2578315676 <== mon.0 v2:172.21.15.118:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fadd4008ad0 con 0x5559d5e3efc0 2023-10-05T19:28:25.607 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.602+0000 7fade5b46e40 1 -- 172.21.15.167:0/2578315676 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5559d5e3efc0 msgr2=0x5559d5e3f390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-05T19:28:25.607 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.602+0000 7fade5b46e40 1 --2- 172.21.15.167:0/2578315676 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5559d5e3efc0 0x5559d5e3f390 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7fadd400a5d0 tx=0x7fadd4005a50 comp rx=0 tx=0).stop 2023-10-05T19:28:25.607 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.602+0000 7fade5b46e40 1 -- 172.21.15.167:0/2578315676 shutdown_connections 2023-10-05T19:28:25.607 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.602+0000 7fade5b46e40 1 --2- 172.21.15.167:0/2578315676 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5559d5e3efc0 0x5559d5e3f390 unknown :-1 s=CLOSED pgs=183 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-05T19:28:25.607 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.602+0000 7fade5b46e40 1 -- 172.21.15.167:0/2578315676 >> 172.21.15.167:0/2578315676 conn(0x5559d5db2810 msgr2=0x5559d5e3d4b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-05T19:28:25.607 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.602+0000 7fade5b46e40 1 -- 172.21.15.167:0/2578315676 shutdown_connections 2023-10-05T19:28:25.607 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.602+0000 7fade5b46e40 1 -- 172.21.15.167:0/2578315676 wait complete. 2023-10-05T19:28:25.608 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.602+0000 7fade5b46e40 1 Processor -- start 2023-10-05T19:28:25.608 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.602+0000 7fade5b46e40 1 -- start start 2023-10-05T19:28:25.608 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.602+0000 7fade5b46e40 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5559d5e3efc0 0x5559d5e5e130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-05T19:28:25.609 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.602+0000 7fade5b46e40 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x5559d5d0ae30 con 0x5559d5e3efc0 2023-10-05T19:28:25.609 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.602+0000 7fade367b700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5559d5e3efc0 0x5559d5e5e130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-05T19:28:25.609 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.602+0000 7fade367b700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5559d5e3efc0 0x5559d5e5e130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.118:3300/0 says I am v2:172.21.15.167:57162/0 (socket says 172.21.15.167:57162) 2023-10-05T19:28:25.609 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.602+0000 7fade367b700 1 -- 172.21.15.167:0/1134798388 learned_addr learned my addr 172.21.15.167:0/1134798388 (peer_addr_for_me v2:172.21.15.167:0/0) 2023-10-05T19:28:25.609 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.602+0000 7fade367b700 1 -- 172.21.15.167:0/1134798388 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fadd4005cf0 con 0x5559d5e3efc0 2023-10-05T19:28:25.610 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.602+0000 7fade367b700 1 --2- 172.21.15.167:0/1134798388 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5559d5e3efc0 0x5559d5e5e130 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7fadd4000c00 tx=0x7fadd4015000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7d4ec1ab85e086da server_cookie=0 in_seq=0 out_seq=0 2023-10-05T19:28:25.611 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.602+0000 7fade1677700 1 -- 172.21.15.167:0/1134798388 <== mon.0 v2:172.21.15.118:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fadd4008970 con 0x5559d5e3efc0 2023-10-05T19:28:25.611 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.602+0000 7fade1677700 1 -- 172.21.15.167:0/1134798388 <== mon.0 v2:172.21.15.118:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fadd4008af0 con 0x5559d5e3efc0 2023-10-05T19:28:25.611 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.602+0000 7fade1677700 1 -- 172.21.15.167:0/1134798388 <== mon.0 v2:172.21.15.118:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fadd4008de0 con 0x5559d5e3efc0 2023-10-05T19:28:25.611 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.602+0000 7fade5b46e40 1 -- 172.21.15.167:0/1134798388 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5559d5e3efc0 msgr2=0x5559d5e5e130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-05T19:28:25.611 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.602+0000 7fade5b46e40 1 --2- 172.21.15.167:0/1134798388 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5559d5e3efc0 0x5559d5e5e130 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7fadd4000c00 tx=0x7fadd4015000 comp rx=0 tx=0).stop 2023-10-05T19:28:25.611 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.606+0000 7fade5b46e40 1 -- 172.21.15.167:0/1134798388 shutdown_connections 2023-10-05T19:28:25.611 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.606+0000 7fade5b46e40 1 --2- 172.21.15.167:0/1134798388 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5559d5e3efc0 0x5559d5e5e130 unknown :-1 s=CLOSED pgs=184 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-05T19:28:25.611 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.606+0000 7fade5b46e40 1 -- 172.21.15.167:0/1134798388 >> 172.21.15.167:0/1134798388 conn(0x5559d5db2810 msgr2=0x5559d5e59cf0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-05T19:28:25.611 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.606+0000 7fade5b46e40 1 -- 172.21.15.167:0/1134798388 shutdown_connections 2023-10-05T19:28:25.611 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.606+0000 7fade5b46e40 1 -- 172.21.15.167:0/1134798388 wait complete. 2023-10-05T19:28:25.611 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.606+0000 7fade5b46e40 1 Processor -- start 2023-10-05T19:28:25.612 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.606+0000 7fade5b46e40 1 -- start start 2023-10-05T19:28:25.612 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.606+0000 7fade5b46e40 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5559d5e3efc0 0x5559d5e64d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-05T19:28:25.612 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.606+0000 7fade5b46e40 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x5559d5cdded0 con 0x5559d5e3efc0 2023-10-05T19:28:25.613 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.606+0000 7fade367b700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5559d5e3efc0 0x5559d5e64d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-05T19:28:25.613 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.606+0000 7fade367b700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5559d5e3efc0 0x5559d5e64d90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.118:3300/0 says I am v2:172.21.15.167:57168/0 (socket says 172.21.15.167:57168) 2023-10-05T19:28:25.613 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.606+0000 7fade367b700 1 -- 172.21.15.167:0/3427901904 learned_addr learned my addr 172.21.15.167:0/3427901904 (peer_addr_for_me v2:172.21.15.167:0/0) 2023-10-05T19:28:25.613 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.606+0000 7fade367b700 1 -- 172.21.15.167:0/3427901904 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5559d5e40a80 con 0x5559d5e3efc0 2023-10-05T19:28:25.613 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.606+0000 7fade367b700 1 --2- 172.21.15.167:0/3427901904 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5559d5e3efc0 0x5559d5e64d90 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7fadd4008fd0 tx=0x7fadd4003050 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-05T19:28:25.613 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.606+0000 7faddb7fe700 1 -- 172.21.15.167:0/3427901904 <== mon.0 v2:172.21.15.118:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fadd4012b70 con 0x5559d5e3efc0 2023-10-05T19:28:25.613 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.606+0000 7faddb7fe700 1 -- 172.21.15.167:0/3427901904 <== mon.0 v2:172.21.15.118:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fadd402a070 con 0x5559d5e3efc0 2023-10-05T19:28:25.614 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.606+0000 7fade5b46e40 1 -- 172.21.15.167:0/3427901904 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x5559d5e652d0 con 0x5559d5e3efc0 2023-10-05T19:28:25.614 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.606+0000 7faddb7fe700 1 -- 172.21.15.167:0/3427901904 <== mon.0 v2:172.21.15.118:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fadd4031480 con 0x5559d5e3efc0 2023-10-05T19:28:25.614 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.606+0000 7fade5b46e40 1 -- 172.21.15.167:0/3427901904 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x5559d5e65770 con 0x5559d5e3efc0 2023-10-05T19:28:25.614 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.610+0000 7fade5b46e40 20 rados->read ofs=0 len=0 2023-10-05T19:28:25.615 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.610+0000 7faddb7fe700 1 -- 172.21.15.167:0/3427901904 <== mon.0 v2:172.21.15.118:3300/0 4 ==== mgrmap(e 5) v1 ==== 40329+0+0 (secure 0 0 0) 0x7fadd4019070 con 0x5559d5e3efc0 2023-10-05T19:28:25.615 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.610+0000 7faddb7fe700 1 --2- 172.21.15.167:0/3427901904 >> [v2:172.21.15.167:6824/29889,v1:172.21.15.167:6825/29889] conn(0x7fadc8030c00 0x7fadc80330b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-05T19:28:25.615 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.610+0000 7faddb7fe700 1 -- 172.21.15.167:0/3427901904 <== mon.0 v2:172.21.15.118:3300/0 5 ==== osd_map(30..30 src has 1..30) v4 ==== 6057+0+0 (secure 0 0 0) 0x7fadd402c070 con 0x5559d5e3efc0 2023-10-05T19:28:25.616 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.610+0000 7fade2e7a700 1 --2- 172.21.15.167:0/3427901904 >> [v2:172.21.15.167:6824/29889,v1:172.21.15.167:6825/29889] conn(0x7fadc8030c00 0x7fadc80330b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-05T19:28:25.616 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.610+0000 7fade5b46e40 1 --2- 172.21.15.167:0/3427901904 >> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] conn(0x5559d5e3f8d0 0x5559d5e70980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-05T19:28:25.616 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.610+0000 7fade5b46e40 1 -- 172.21.15.167:0/3427901904 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:1 5.12 5:49953fa1:::default.realm:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x5559d5e70fa0 con 0x5559d5e3f8d0 2023-10-05T19:28:25.616 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.610+0000 7fade3e7c700 1 --2- 172.21.15.167:0/3427901904 >> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] conn(0x5559d5e3f8d0 0x5559d5e70980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-05T19:28:25.616 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.610+0000 7fade2e7a700 1 --2- 172.21.15.167:0/3427901904 >> [v2:172.21.15.167:6824/29889,v1:172.21.15.167:6825/29889] conn(0x7fadc8030c00 0x7fadc80330b0 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7faddc0044f0 tx=0x7faddc0067c0 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-05T19:28:25.616 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.610+0000 7fade3e7c700 1 --2- 172.21.15.167:0/3427901904 >> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] conn(0x5559d5e3f8d0 0x5559d5e70980 crc :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-05T19:28:25.617 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.610+0000 7fade3e7c700 1 -- 172.21.15.167:0/3427901904 <== osd.1 v2:172.21.15.167:6800/30035 1 ==== osd_op_reply(1 default.realm [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 ==== 157+0+0 (crc 0 0 0) 0x7fadcc00a500 con 0x5559d5e3f8d0 2023-10-05T19:28:25.617 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.610+0000 7fade5b46e40 20 rados_obj.operate() r=-2 bl.length=0 2023-10-05T19:28:25.617 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.610+0000 7fade5b46e40 20 realm 2023-10-05T19:28:25.617 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.610+0000 7fade5b46e40 20 rados->read ofs=0 len=0 2023-10-05T19:28:25.617 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.610+0000 7fade5b46e40 1 -- 172.21.15.167:0/3427901904 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:2 5.12 5:49953fa1:::default.realm:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x5559d5e6dfe0 con 0x5559d5e3f8d0 2023-10-05T19:28:25.618 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.610+0000 7fade3e7c700 1 -- 172.21.15.167:0/3427901904 <== osd.1 v2:172.21.15.167:6800/30035 2 ==== osd_op_reply(2 default.realm [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 ==== 157+0+0 (crc 0 0 0) 0x7fadcc00a500 con 0x5559d5e3f8d0 2023-10-05T19:28:25.618 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.610+0000 7fade5b46e40 20 rados_obj.operate() r=-2 bl.length=0 2023-10-05T19:28:25.619 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.610+0000 7fade5b46e40 4 RGWPeriod::init failed to init realm id : (2) No such file or directory 2023-10-05T19:28:25.619 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.610+0000 7fade5b46e40 20 rados->read ofs=0 len=0 2023-10-05T19:28:25.619 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.610+0000 7fade5b46e40 1 -- 172.21.15.167:0/3427901904 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:3 5.12 5:49953fa1:::default.realm:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x5559d5e71a80 con 0x5559d5e3f8d0 2023-10-05T19:28:25.619 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.614+0000 7fade3e7c700 1 -- 172.21.15.167:0/3427901904 <== osd.1 v2:172.21.15.167:6800/30035 3 ==== osd_op_reply(3 default.realm [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 ==== 157+0+0 (crc 0 0 0) 0x7fadcc00a500 con 0x5559d5e3f8d0 2023-10-05T19:28:25.619 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.614+0000 7fade5b46e40 20 rados_obj.operate() r=-2 bl.length=0 2023-10-05T19:28:25.619 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.614+0000 7fade5b46e40 20 rados->read ofs=0 len=0 2023-10-05T19:28:25.619 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.614+0000 7fade5b46e40 1 -- 172.21.15.167:0/3427901904 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:4 5.f 5:f43b6ece:::zone_names.default:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x5559d5e71690 con 0x5559d5e3f8d0 2023-10-05T19:28:25.620 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.614+0000 7fade3e7c700 1 -- 172.21.15.167:0/3427901904 <== osd.1 v2:172.21.15.167:6800/30035 4 ==== osd_op_reply(4 zone_names.default [read 0~46 out=46b] v0'0 uv1 ondisk = 0) v8 ==== 162+0+46 (crc 0 0 0) 0x7fadcc00a500 con 0x5559d5e3f8d0 2023-10-05T19:28:25.620 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.614+0000 7fade5b46e40 20 rados_obj.operate() r=0 bl.length=46 2023-10-05T19:28:25.621 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.614+0000 7fade5b46e40 20 rados->read ofs=0 len=0 2023-10-05T19:28:25.621 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.614+0000 7fade5b46e40 1 -- 172.21.15.167:0/3427901904 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:5 5.1d 5:b8374eb4:::zone_info.9a9df965-6612-45d4-a45f-6e3c889bacc2:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x5559d5e72910 con 0x5559d5e3f8d0 2023-10-05T19:28:25.621 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.614+0000 7fade3e7c700 1 -- 172.21.15.167:0/3427901904 <== osd.1 v2:172.21.15.167:6800/30035 5 ==== osd_op_reply(5 zone_info.9a9df965-6612-45d4-a45f-6e3c889bacc2 [read 0~874 out=874b] v0'0 uv2 ondisk = 0) v8 ==== 190+0+874 (crc 0 0 0) 0x7fadcc00a500 con 0x5559d5e3f8d0 2023-10-05T19:28:25.621 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.614+0000 7fade5b46e40 20 rados_obj.operate() r=0 bl.length=874 2023-10-05T19:28:25.621 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.614+0000 7fade5b46e40 20 searching for the correct realm 2023-10-05T19:28:25.621 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.614+0000 7fade5b46e40 1 --2- 172.21.15.167:0/3427901904 >> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] conn(0x5559d5e746d0 0x5559d5e76ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-05T19:28:25.621 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.614+0000 7fade5b46e40 1 -- 172.21.15.167:0/3427901904 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:6 5.0 5:00000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x5559d5e76ff0 con 0x5559d5e746d0 2023-10-05T19:28:25.621 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.614+0000 7fade367b700 1 --2- 172.21.15.167:0/3427901904 >> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] conn(0x5559d5e746d0 0x5559d5e76ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-05T19:28:25.621 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.614+0000 7fade367b700 1 --2- 172.21.15.167:0/3427901904 >> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] conn(0x5559d5e746d0 0x5559d5e76ab0 crc :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-05T19:28:25.622 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.618+0000 7fade367b700 1 -- 172.21.15.167:0/3427901904 <== osd.3 v2:172.21.15.167:6808/30036 1 ==== osd_op_reply(6 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fadd4022020 con 0x5559d5e746d0 2023-10-05T19:28:25.623 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.618+0000 7fade367b700 1 --2- 172.21.15.167:0/3427901904 >> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] conn(0x7fadd4007030 0x7fadd401ea00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-05T19:28:25.623 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.618+0000 7fade367b700 1 -- 172.21.15.167:0/3427901904 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:7 5.10 5:08000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x5559d5e76ff0 con 0x7fadd4007030 2023-10-05T19:28:25.623 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.618+0000 7fade2e7a700 1 --2- 172.21.15.167:0/3427901904 >> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] conn(0x7fadd4007030 0x7fadd401ea00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-05T19:28:25.623 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.618+0000 7fade2e7a700 1 --2- 172.21.15.167:0/3427901904 >> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] conn(0x7fadd4007030 0x7fadd401ea00 crc :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-05T19:28:25.624 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.618+0000 7fade2e7a700 1 -- 172.21.15.167:0/3427901904 <== osd.2 v2:172.21.15.167:6816/30037 1 ==== osd_op_reply(7 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7faddc004520 con 0x7fadd4007030 2023-10-05T19:28:25.624 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.618+0000 7fade2e7a700 1 --2- 172.21.15.167:0/3427901904 >> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] conn(0x7faddc011040 0x7faddc014950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-05T19:28:25.624 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.618+0000 7fade2e7a700 1 -- 172.21.15.167:0/3427901904 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:8 5.8 5:10000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x5559d5e76ff0 con 0x7faddc011040 2023-10-05T19:28:25.625 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.618+0000 7fade3e7c700 1 --2- 172.21.15.167:0/3427901904 >> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] conn(0x7faddc011040 0x7faddc014950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-05T19:28:25.625 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.618+0000 7fade3e7c700 1 --2- 172.21.15.167:0/3427901904 >> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] conn(0x7faddc011040 0x7faddc014950 crc :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-05T19:28:25.626 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.622+0000 7fade3e7c700 1 -- 172.21.15.167:0/3427901904 <== osd.0 v2:172.21.15.118:6800/27690 1 ==== osd_op_reply(8 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fadcc014040 con 0x7faddc011040 2023-10-05T19:28:25.626 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.622+0000 7fade3e7c700 1 -- 172.21.15.167:0/3427901904 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:9 5.18 5:18000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x5559d5e76ff0 con 0x5559d5e3f8d0 2023-10-05T19:28:25.627 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.622+0000 7fade3e7c700 1 -- 172.21.15.167:0/3427901904 <== osd.1 v2:172.21.15.167:6800/30035 6 ==== osd_op_reply(9 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fadcc014040 con 0x5559d5e3f8d0 2023-10-05T19:28:25.627 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.622+0000 7fade3e7c700 1 -- 172.21.15.167:0/3427901904 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:10 5.4 5:20000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x5559d5e76ff0 con 0x5559d5e3f8d0 2023-10-05T19:28:25.628 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.622+0000 7fade3e7c700 1 -- 172.21.15.167:0/3427901904 <== osd.1 v2:172.21.15.167:6800/30035 7 ==== osd_op_reply(10 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fadcc014040 con 0x5559d5e3f8d0 2023-10-05T19:28:25.628 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.622+0000 7fade3e7c700 1 -- 172.21.15.167:0/3427901904 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:11 5.14 5:28000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x5559d5e76ff0 con 0x7faddc011040 2023-10-05T19:28:25.628 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.622+0000 7fade3e7c700 1 -- 172.21.15.167:0/3427901904 <== osd.0 v2:172.21.15.118:6800/27690 2 ==== osd_op_reply(11 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fadcc014040 con 0x7faddc011040 2023-10-05T19:28:25.628 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.622+0000 7fade3e7c700 1 -- 172.21.15.167:0/3427901904 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:12 5.c 5:30000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x5559d5e76ff0 con 0x7faddc011040 2023-10-05T19:28:25.629 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.622+0000 7fade3e7c700 1 -- 172.21.15.167:0/3427901904 <== osd.0 v2:172.21.15.118:6800/27690 3 ==== osd_op_reply(12 [pgnls start_epoch 30 out=112b] v23'1 uv1 ondisk = 1) v8 ==== 144+0+112 (crc 0 0 0) 0x7fadcc014040 con 0x7faddc011040 2023-10-05T19:28:25.629 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.622+0000 7fade3e7c700 1 -- 172.21.15.167:0/3427901904 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:13 5.1c 5:38000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x5559d5e76ff0 con 0x7fadd4007030 2023-10-05T19:28:25.630 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.626+0000 7fade2e7a700 1 -- 172.21.15.167:0/3427901904 <== osd.2 v2:172.21.15.167:6816/30037 2 ==== osd_op_reply(13 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7faddc004520 con 0x7fadd4007030 2023-10-05T19:28:25.630 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.626+0000 7fade2e7a700 1 -- 172.21.15.167:0/3427901904 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:14 5.2 5:40000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x5559d5e76ff0 con 0x5559d5e746d0 2023-10-05T19:28:25.630 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.626+0000 7fade367b700 1 -- 172.21.15.167:0/3427901904 <== osd.3 v2:172.21.15.167:6808/30036 2 ==== osd_op_reply(14 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fadd4022020 con 0x5559d5e746d0 2023-10-05T19:28:25.631 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.626+0000 7fade367b700 1 -- 172.21.15.167:0/3427901904 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:15 5.12 5:48000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x5559d5e76ff0 con 0x5559d5e3f8d0 2023-10-05T19:28:25.631 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.626+0000 7fade3e7c700 1 -- 172.21.15.167:0/3427901904 <== osd.1 v2:172.21.15.167:6800/30035 8 ==== osd_op_reply(15 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fadcc014040 con 0x5559d5e3f8d0 2023-10-05T19:28:25.631 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.626+0000 7fade3e7c700 1 -- 172.21.15.167:0/3427901904 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:16 5.a 5:50000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x5559d5e76ff0 con 0x7faddc011040 2023-10-05T19:28:25.633 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.626+0000 7fade3e7c700 1 -- 172.21.15.167:0/3427901904 <== osd.0 v2:172.21.15.118:6800/27690 4 ==== osd_op_reply(16 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fadcc022040 con 0x7faddc011040 2023-10-05T19:28:25.633 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.626+0000 7fade3e7c700 1 -- 172.21.15.167:0/3427901904 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:17 5.1a 5:58000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x5559d5e76ff0 con 0x5559d5e3f8d0 2023-10-05T19:28:25.633 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.626+0000 7fade3e7c700 1 -- 172.21.15.167:0/3427901904 <== osd.1 v2:172.21.15.167:6800/30035 9 ==== osd_op_reply(17 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fadcc022040 con 0x5559d5e3f8d0 2023-10-05T19:28:25.633 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.626+0000 7fade3e7c700 1 -- 172.21.15.167:0/3427901904 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:18 5.6 5:60000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x5559d5e76ff0 con 0x7fadd4007030 2023-10-05T19:28:25.634 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.626+0000 7fade2e7a700 1 -- 172.21.15.167:0/3427901904 <== osd.2 v2:172.21.15.167:6816/30037 3 ==== osd_op_reply(18 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7faddc004520 con 0x7fadd4007030 2023-10-05T19:28:25.634 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.630+0000 7fade2e7a700 1 -- 172.21.15.167:0/3427901904 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:19 5.16 5:68000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x5559d5e76ff0 con 0x5559d5e746d0 2023-10-05T19:28:25.634 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.630+0000 7fade367b700 1 -- 172.21.15.167:0/3427901904 <== osd.3 v2:172.21.15.167:6808/30036 3 ==== osd_op_reply(19 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fadd4022020 con 0x5559d5e746d0 2023-10-05T19:28:25.635 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.630+0000 7fade367b700 1 -- 172.21.15.167:0/3427901904 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:20 5.e 5:70000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x7fadd4047040 con 0x5559d5e746d0 2023-10-05T19:28:25.635 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.630+0000 7fade367b700 1 -- 172.21.15.167:0/3427901904 <== osd.3 v2:172.21.15.167:6808/30036 4 ==== osd_op_reply(20 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fadd4022020 con 0x5559d5e746d0 2023-10-05T19:28:25.635 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.630+0000 7fade367b700 1 -- 172.21.15.167:0/3427901904 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:21 5.1e 5:78000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x7fadd4047040 con 0x7faddc011040 2023-10-05T19:28:25.636 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.630+0000 7fade3e7c700 1 -- 172.21.15.167:0/3427901904 <== osd.0 v2:172.21.15.118:6800/27690 5 ==== osd_op_reply(21 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fadcc022040 con 0x7faddc011040 2023-10-05T19:28:25.636 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.630+0000 7fade3e7c700 1 -- 172.21.15.167:0/3427901904 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:22 5.1 5:80000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x7fadd4047040 con 0x5559d5e746d0 2023-10-05T19:28:25.637 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.630+0000 7fade367b700 1 -- 172.21.15.167:0/3427901904 <== osd.3 v2:172.21.15.167:6808/30036 5 ==== osd_op_reply(22 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fadd4022020 con 0x5559d5e746d0 2023-10-05T19:28:25.637 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.630+0000 7fade367b700 1 -- 172.21.15.167:0/3427901904 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:23 5.11 5:88000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x7fadd4047040 con 0x5559d5e746d0 2023-10-05T19:28:25.638 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.630+0000 7fade367b700 1 -- 172.21.15.167:0/3427901904 <== osd.3 v2:172.21.15.167:6808/30036 6 ==== osd_op_reply(23 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fadd4022020 con 0x5559d5e746d0 2023-10-05T19:28:25.638 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.630+0000 7fade367b700 1 -- 172.21.15.167:0/3427901904 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:24 5.9 5:90000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x7fadd4047040 con 0x5559d5e3f8d0 2023-10-05T19:28:25.638 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.634+0000 7fade3e7c700 1 -- 172.21.15.167:0/3427901904 <== osd.1 v2:172.21.15.167:6800/30035 10 ==== osd_op_reply(24 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fadcc022040 con 0x5559d5e3f8d0 2023-10-05T19:28:25.638 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.634+0000 7fade3e7c700 1 -- 172.21.15.167:0/3427901904 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:25 5.19 5:98000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x7fadd4047040 con 0x5559d5e3f8d0 2023-10-05T19:28:25.639 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.634+0000 7fade3e7c700 1 -- 172.21.15.167:0/3427901904 <== osd.1 v2:172.21.15.167:6800/30035 11 ==== osd_op_reply(25 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fadcc022040 con 0x5559d5e3f8d0 2023-10-05T19:28:25.639 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.634+0000 7fade3e7c700 1 -- 172.21.15.167:0/3427901904 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:26 5.5 5:a0000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x7fadd4047040 con 0x7faddc011040 2023-10-05T19:28:25.640 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.634+0000 7fade3e7c700 1 -- 172.21.15.167:0/3427901904 <== osd.0 v2:172.21.15.118:6800/27690 6 ==== osd_op_reply(26 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fadcc022040 con 0x7faddc011040 2023-10-05T19:28:25.640 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.634+0000 7fade3e7c700 1 -- 172.21.15.167:0/3427901904 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:27 5.15 5:a8000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x7fadd4047040 con 0x5559d5e746d0 2023-10-05T19:28:25.641 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.634+0000 7fade367b700 1 -- 172.21.15.167:0/3427901904 <== osd.3 v2:172.21.15.167:6808/30036 7 ==== osd_op_reply(27 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fadd4022020 con 0x5559d5e746d0 2023-10-05T19:28:25.641 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.634+0000 7fade367b700 1 -- 172.21.15.167:0/3427901904 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:28 5.d 5:b0000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x7fadd4047040 con 0x7fadd4007030 2023-10-05T19:28:25.642 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.634+0000 7fade2e7a700 1 -- 172.21.15.167:0/3427901904 <== osd.2 v2:172.21.15.167:6816/30037 4 ==== osd_op_reply(28 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7faddc004520 con 0x7fadd4007030 2023-10-05T19:28:25.642 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.634+0000 7fade2e7a700 1 -- 172.21.15.167:0/3427901904 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:29 5.1d 5:b8000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x7fadd4047040 con 0x5559d5e3f8d0 2023-10-05T19:28:25.642 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.638+0000 7fade3e7c700 1 -- 172.21.15.167:0/3427901904 <== osd.1 v2:172.21.15.167:6800/30035 12 ==== osd_op_reply(29 [pgnls start_epoch 30 out=107b] v30'2 uv2 ondisk = 1) v8 ==== 144+0+107 (crc 0 0 0) 0x7fadcc022040 con 0x5559d5e3f8d0 2023-10-05T19:28:25.643 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.638+0000 7fade3e7c700 1 -- 172.21.15.167:0/3427901904 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:30 5.3 5:c0000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x7fadd4047040 con 0x7faddc011040 2023-10-05T19:28:25.643 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.638+0000 7fade3e7c700 1 -- 172.21.15.167:0/3427901904 <== osd.0 v2:172.21.15.118:6800/27690 7 ==== osd_op_reply(30 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fadcc022040 con 0x7faddc011040 2023-10-05T19:28:25.644 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.638+0000 7fade3e7c700 1 -- 172.21.15.167:0/3427901904 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:31 5.13 5:c8000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x7fadd4047040 con 0x5559d5e746d0 2023-10-05T19:28:25.644 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.638+0000 7fade367b700 1 -- 172.21.15.167:0/3427901904 <== osd.3 v2:172.21.15.167:6808/30036 8 ==== osd_op_reply(31 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fadd4022020 con 0x5559d5e746d0 2023-10-05T19:28:25.644 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.638+0000 7fade367b700 1 -- 172.21.15.167:0/3427901904 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:32 5.b 5:d0000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x7fadd4047040 con 0x7faddc011040 2023-10-05T19:28:25.645 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.638+0000 7fade3e7c700 1 -- 172.21.15.167:0/3427901904 <== osd.0 v2:172.21.15.118:6800/27690 8 ==== osd_op_reply(32 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fadcc022040 con 0x7faddc011040 2023-10-05T19:28:25.645 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.638+0000 7fade3e7c700 1 -- 172.21.15.167:0/3427901904 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:33 5.1b 5:d8000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x7fadd4047040 con 0x7fadd4007030 2023-10-05T19:28:25.646 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.642+0000 7fade2e7a700 1 -- 172.21.15.167:0/3427901904 <== osd.2 v2:172.21.15.167:6816/30037 5 ==== osd_op_reply(33 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7faddc004520 con 0x7fadd4007030 2023-10-05T19:28:25.646 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.642+0000 7fade2e7a700 1 -- 172.21.15.167:0/3427901904 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:34 5.7 5:e0000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x7fadd4047040 con 0x5559d5e746d0 2023-10-05T19:28:25.647 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.642+0000 7fade367b700 1 -- 172.21.15.167:0/3427901904 <== osd.3 v2:172.21.15.167:6808/30036 9 ==== osd_op_reply(34 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fadd4022020 con 0x5559d5e746d0 2023-10-05T19:28:25.647 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.642+0000 7fade367b700 1 -- 172.21.15.167:0/3427901904 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:35 5.17 5:e8000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x7fadd4047040 con 0x7faddc011040 2023-10-05T19:28:25.648 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.642+0000 7fade3e7c700 1 -- 172.21.15.167:0/3427901904 <== osd.0 v2:172.21.15.118:6800/27690 9 ==== osd_op_reply(35 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fadcc022040 con 0x7faddc011040 2023-10-05T19:28:25.648 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.642+0000 7fade3e7c700 1 -- 172.21.15.167:0/3427901904 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:36 5.f 5:f0000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x7fadd4047040 con 0x5559d5e3f8d0 2023-10-05T19:28:25.649 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.642+0000 7fade3e7c700 1 -- 172.21.15.167:0/3427901904 <== osd.1 v2:172.21.15.167:6800/30035 13 ==== osd_op_reply(36 [pgnls start_epoch 30 out=115b] v23'2 uv2 ondisk = 1) v8 ==== 144+0+115 (crc 0 0 0) 0x7fadcc02f080 con 0x5559d5e3f8d0 2023-10-05T19:28:25.649 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.642+0000 7fade3e7c700 1 -- 172.21.15.167:0/3427901904 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:37 5.1f 5:f8000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x7fadd4047040 con 0x7fadd4007030 2023-10-05T19:28:25.650 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.646+0000 7fade2e7a700 1 -- 172.21.15.167:0/3427901904 <== osd.2 v2:172.21.15.167:6816/30037 6 ==== osd_op_reply(37 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7faddc004520 con 0x7fadd4007030 2023-10-05T19:28:25.650 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.646+0000 7fade5b46e40 20 RGWRados::pool_iterate: got zonegroup_info.4f1f719d-9648-417b-b10c-e5d939825419 2023-10-05T19:28:25.650 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.646+0000 7fade5b46e40 20 RGWRados::pool_iterate: got zone_info.9a9df965-6612-45d4-a45f-6e3c889bacc2 2023-10-05T19:28:25.650 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.646+0000 7fade5b46e40 20 RGWRados::pool_iterate: got zone_names.default 2023-10-05T19:28:25.651 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.646+0000 7fade5b46e40 20 RGWRados::pool_iterate: got zonegroups_names.default 2023-10-05T19:28:25.651 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.646+0000 7fade5b46e40 20 rados->read ofs=0 len=0 2023-10-05T19:28:25.651 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.646+0000 7fade5b46e40 1 -- 172.21.15.167:0/3427901904 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:38 5.12 5:49953fa1:::default.realm:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x5559d5e73480 con 0x5559d5e3f8d0 2023-10-05T19:28:25.652 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.646+0000 7fade3e7c700 1 -- 172.21.15.167:0/3427901904 <== osd.1 v2:172.21.15.167:6800/30035 14 ==== osd_op_reply(38 default.realm [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 ==== 157+0+0 (crc 0 0 0) 0x7fadcc02f080 con 0x5559d5e3f8d0 2023-10-05T19:28:25.652 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.646+0000 7fade5b46e40 20 rados_obj.operate() r=-2 bl.length=0 2023-10-05T19:28:25.652 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.646+0000 7fade5b46e40 20 rados->read ofs=0 len=0 2023-10-05T19:28:25.652 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.646+0000 7fade5b46e40 1 -- 172.21.15.167:0/3427901904 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:39 5.f 5:f4c53578:::zonegroups_names.default:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x5559d5e73d70 con 0x5559d5e3f8d0 2023-10-05T19:28:25.652 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.646+0000 7fade3e7c700 1 -- 172.21.15.167:0/3427901904 <== osd.1 v2:172.21.15.167:6800/30035 15 ==== osd_op_reply(39 zonegroups_names.default [read 0~46 out=46b] v0'0 uv2 ondisk = 0) v8 ==== 168+0+46 (crc 0 0 0) 0x7fadcc02f080 con 0x5559d5e3f8d0 2023-10-05T19:28:25.652 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.646+0000 7fade5b46e40 20 rados_obj.operate() r=0 bl.length=46 2023-10-05T19:28:25.652 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.646+0000 7fade5b46e40 20 rados->read ofs=0 len=0 2023-10-05T19:28:25.652 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.646+0000 7fade5b46e40 1 -- 172.21.15.167:0/3427901904 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:40 5.c 5:30d299a5:::zonegroup_info.4f1f719d-9648-417b-b10c-e5d939825419:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x5559d5e77430 con 0x7faddc011040 2023-10-05T19:28:25.653 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.646+0000 7fade3e7c700 1 -- 172.21.15.167:0/3427901904 <== osd.0 v2:172.21.15.118:6800/27690 10 ==== osd_op_reply(40 zonegroup_info.4f1f719d-9648-417b-b10c-e5d939825419 [read 0~362 out=362b] v0'0 uv1 ondisk = 0) v8 ==== 195+0+362 (crc 0 0 0) 0x7fadcc02f080 con 0x7faddc011040 2023-10-05T19:28:25.653 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.646+0000 7fade5b46e40 20 rados_obj.operate() r=0 bl.length=362 2023-10-05T19:28:25.653 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.646+0000 7fade5b46e40 20 rados->read ofs=0 len=0 2023-10-05T19:28:25.653 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.646+0000 7fade5b46e40 1 -- 172.21.15.167:0/3427901904 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:41 5.13 5:cbbf6c60:::converted:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x5559d5e77430 con 0x5559d5e746d0 2023-10-05T19:28:25.654 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.646+0000 7fade367b700 1 -- 172.21.15.167:0/3427901904 <== osd.3 v2:172.21.15.167:6808/30036 10 ==== osd_op_reply(41 converted [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 ==== 153+0+0 (crc 0 0 0) 0x7fadd4022020 con 0x5559d5e746d0 2023-10-05T19:28:25.654 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.646+0000 7fade5b46e40 20 rados_obj.operate() r=-2 bl.length=0 2023-10-05T19:28:25.654 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.646+0000 7fade5b46e40 20 rados->read ofs=0 len=0 2023-10-05T19:28:25.654 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.646+0000 7fade5b46e40 1 -- 172.21.15.167:0/3427901904 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:42 5.12 5:49953fa1:::default.realm:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x5559d5e777e0 con 0x5559d5e3f8d0 2023-10-05T19:28:25.655 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.650+0000 7fade3e7c700 1 -- 172.21.15.167:0/3427901904 <== osd.1 v2:172.21.15.167:6800/30035 16 ==== osd_op_reply(42 default.realm [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 ==== 157+0+0 (crc 0 0 0) 0x7fadcc02f080 con 0x5559d5e3f8d0 2023-10-05T19:28:25.655 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.650+0000 7fade5b46e40 20 rados_obj.operate() r=-2 bl.length=0 2023-10-05T19:28:25.655 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.650+0000 7fade5b46e40 20 rados->read ofs=0 len=0 2023-10-05T19:28:25.655 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.650+0000 7fade5b46e40 1 -- 172.21.15.167:0/3427901904 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:43 5.f 5:f4c53578:::zonegroups_names.default:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x5559d5e77be0 con 0x5559d5e3f8d0 2023-10-05T19:28:25.655 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.650+0000 7fade3e7c700 1 -- 172.21.15.167:0/3427901904 <== osd.1 v2:172.21.15.167:6800/30035 17 ==== osd_op_reply(43 zonegroups_names.default [read 0~46 out=46b] v0'0 uv2 ondisk = 0) v8 ==== 168+0+46 (crc 0 0 0) 0x7fadcc02f080 con 0x5559d5e3f8d0 2023-10-05T19:28:25.655 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.650+0000 7fade5b46e40 20 rados_obj.operate() r=0 bl.length=46 2023-10-05T19:28:25.656 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.650+0000 7fade5b46e40 1 -- 172.21.15.167:0/3427901904 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:44 5.0 5:00000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x5559d5e78bf0 con 0x5559d5e746d0 2023-10-05T19:28:25.658 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.650+0000 7fade367b700 1 -- 172.21.15.167:0/3427901904 <== osd.3 v2:172.21.15.167:6808/30036 11 ==== osd_op_reply(44 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fadd4022020 con 0x5559d5e746d0 2023-10-05T19:28:25.658 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.650+0000 7fade367b700 1 -- 172.21.15.167:0/3427901904 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:45 5.10 5:08000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x5559d5e78bf0 con 0x7fadd4007030 2023-10-05T19:28:25.658 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.654+0000 7fade2e7a700 1 -- 172.21.15.167:0/3427901904 <== osd.2 v2:172.21.15.167:6816/30037 7 ==== osd_op_reply(45 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7faddc004520 con 0x7fadd4007030 2023-10-05T19:28:25.658 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.654+0000 7fade2e7a700 1 -- 172.21.15.167:0/3427901904 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:46 5.8 5:10000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x5559d5e78bf0 con 0x7faddc011040 2023-10-05T19:28:25.659 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.654+0000 7fade3e7c700 1 -- 172.21.15.167:0/3427901904 <== osd.0 v2:172.21.15.118:6800/27690 11 ==== osd_op_reply(46 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fadcc02f080 con 0x7faddc011040 2023-10-05T19:28:25.660 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.654+0000 7fade3e7c700 1 -- 172.21.15.167:0/3427901904 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:47 5.18 5:18000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x5559d5e73d70 con 0x5559d5e3f8d0 2023-10-05T19:28:25.660 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.654+0000 7fade3e7c700 1 -- 172.21.15.167:0/3427901904 <== osd.1 v2:172.21.15.167:6800/30035 18 ==== osd_op_reply(47 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fadcc02f080 con 0x5559d5e3f8d0 2023-10-05T19:28:25.660 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.654+0000 7fade3e7c700 1 -- 172.21.15.167:0/3427901904 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:48 5.4 5:20000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x5559d5e73d70 con 0x5559d5e3f8d0 2023-10-05T19:28:25.661 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.654+0000 7fade3e7c700 1 -- 172.21.15.167:0/3427901904 <== osd.1 v2:172.21.15.167:6800/30035 19 ==== osd_op_reply(48 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fadcc02f080 con 0x5559d5e3f8d0 2023-10-05T19:28:25.661 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.654+0000 7fade3e7c700 1 -- 172.21.15.167:0/3427901904 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:49 5.14 5:28000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x5559d5e73d70 con 0x7faddc011040 2023-10-05T19:28:25.662 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.658+0000 7fade3e7c700 1 -- 172.21.15.167:0/3427901904 <== osd.0 v2:172.21.15.118:6800/27690 12 ==== osd_op_reply(49 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fadcc02f080 con 0x7faddc011040 2023-10-05T19:28:25.662 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.658+0000 7fade3e7c700 1 -- 172.21.15.167:0/3427901904 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:50 5.c 5:30000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x5559d5e73d70 con 0x7faddc011040 2023-10-05T19:28:25.662 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.658+0000 7fade3e7c700 1 -- 172.21.15.167:0/3427901904 <== osd.0 v2:172.21.15.118:6800/27690 13 ==== osd_op_reply(50 [pgnls start_epoch 30 out=112b] v23'1 uv1 ondisk = 1) v8 ==== 144+0+112 (crc 0 0 0) 0x7fadcc02f080 con 0x7faddc011040 2023-10-05T19:28:25.663 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.658+0000 7fade3e7c700 1 -- 172.21.15.167:0/3427901904 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:51 5.1c 5:38000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x5559d5e73d70 con 0x7fadd4007030 2023-10-05T19:28:25.663 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.658+0000 7fade2e7a700 1 -- 172.21.15.167:0/3427901904 <== osd.2 v2:172.21.15.167:6816/30037 8 ==== osd_op_reply(51 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7faddc004520 con 0x7fadd4007030 2023-10-05T19:28:25.664 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.658+0000 7fade2e7a700 1 -- 172.21.15.167:0/3427901904 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:52 5.2 5:40000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x5559d5e73d70 con 0x5559d5e746d0 2023-10-05T19:28:25.664 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.658+0000 7fade367b700 1 -- 172.21.15.167:0/3427901904 <== osd.3 v2:172.21.15.167:6808/30036 12 ==== osd_op_reply(52 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fadd4022020 con 0x5559d5e746d0 2023-10-05T19:28:25.664 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.658+0000 7fade367b700 1 -- 172.21.15.167:0/3427901904 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:53 5.12 5:48000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x5559d5e73d70 con 0x5559d5e3f8d0 2023-10-05T19:28:25.665 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.658+0000 7fade3e7c700 1 -- 172.21.15.167:0/3427901904 <== osd.1 v2:172.21.15.167:6800/30035 20 ==== osd_op_reply(53 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fadcc02f080 con 0x5559d5e3f8d0 2023-10-05T19:28:25.665 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.658+0000 7fade3e7c700 1 -- 172.21.15.167:0/3427901904 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:54 5.a 5:50000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x5559d5e73d70 con 0x7faddc011040 2023-10-05T19:28:25.666 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.662+0000 7fade3e7c700 1 -- 172.21.15.167:0/3427901904 <== osd.0 v2:172.21.15.118:6800/27690 14 ==== osd_op_reply(54 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fadcc02f080 con 0x7faddc011040 2023-10-05T19:28:25.666 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.662+0000 7fade3e7c700 1 -- 172.21.15.167:0/3427901904 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:55 5.1a 5:58000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x5559d5e73d70 con 0x5559d5e3f8d0 2023-10-05T19:28:25.667 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.662+0000 7fade3e7c700 1 -- 172.21.15.167:0/3427901904 <== osd.1 v2:172.21.15.167:6800/30035 21 ==== osd_op_reply(55 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fadcc02f080 con 0x5559d5e3f8d0 2023-10-05T19:28:25.667 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.662+0000 7fade3e7c700 1 -- 172.21.15.167:0/3427901904 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:56 5.6 5:60000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x5559d5e73d70 con 0x7fadd4007030 2023-10-05T19:28:25.667 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.662+0000 7fade2e7a700 1 -- 172.21.15.167:0/3427901904 <== osd.2 v2:172.21.15.167:6816/30037 9 ==== osd_op_reply(56 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7faddc004520 con 0x7fadd4007030 2023-10-05T19:28:25.668 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.662+0000 7fade2e7a700 1 -- 172.21.15.167:0/3427901904 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:57 5.16 5:68000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x5559d5e73d70 con 0x5559d5e746d0 2023-10-05T19:28:25.668 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.662+0000 7fade367b700 1 -- 172.21.15.167:0/3427901904 <== osd.3 v2:172.21.15.167:6808/30036 13 ==== osd_op_reply(57 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fadd4022020 con 0x5559d5e746d0 2023-10-05T19:28:25.668 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.662+0000 7fade367b700 1 -- 172.21.15.167:0/3427901904 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:58 5.e 5:70000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x5559d5e73d70 con 0x5559d5e746d0 2023-10-05T19:28:25.669 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.662+0000 7fade367b700 1 -- 172.21.15.167:0/3427901904 <== osd.3 v2:172.21.15.167:6808/30036 14 ==== osd_op_reply(58 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fadd4022020 con 0x5559d5e746d0 2023-10-05T19:28:25.669 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.662+0000 7fade367b700 1 -- 172.21.15.167:0/3427901904 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:59 5.1e 5:78000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x5559d5e73d70 con 0x7faddc011040 2023-10-05T19:28:25.670 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.666+0000 7fade3e7c700 1 -- 172.21.15.167:0/3427901904 <== osd.0 v2:172.21.15.118:6800/27690 15 ==== osd_op_reply(59 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fadcc02f080 con 0x7faddc011040 2023-10-05T19:28:25.670 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.666+0000 7fade3e7c700 1 -- 172.21.15.167:0/3427901904 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:60 5.1 5:80000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x5559d5e73d70 con 0x5559d5e746d0 2023-10-05T19:28:25.671 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.666+0000 7fade367b700 1 -- 172.21.15.167:0/3427901904 <== osd.3 v2:172.21.15.167:6808/30036 15 ==== osd_op_reply(60 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fadd4022020 con 0x5559d5e746d0 2023-10-05T19:28:25.671 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.666+0000 7fade367b700 1 -- 172.21.15.167:0/3427901904 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:61 5.11 5:88000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x5559d5e73d70 con 0x5559d5e746d0 2023-10-05T19:28:25.671 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.666+0000 7fade367b700 1 -- 172.21.15.167:0/3427901904 <== osd.3 v2:172.21.15.167:6808/30036 16 ==== osd_op_reply(61 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fadd4022020 con 0x5559d5e746d0 2023-10-05T19:28:25.671 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.666+0000 7fade367b700 1 -- 172.21.15.167:0/3427901904 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:62 5.9 5:90000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x5559d5e73d70 con 0x5559d5e3f8d0 2023-10-05T19:28:25.672 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.666+0000 7fade3e7c700 1 -- 172.21.15.167:0/3427901904 <== osd.1 v2:172.21.15.167:6800/30035 22 ==== osd_op_reply(62 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fadcc02f080 con 0x5559d5e3f8d0 2023-10-05T19:28:25.672 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.666+0000 7fade3e7c700 1 -- 172.21.15.167:0/3427901904 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:63 5.19 5:98000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x5559d5e73d70 con 0x5559d5e3f8d0 2023-10-05T19:28:25.673 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.666+0000 7fade3e7c700 1 -- 172.21.15.167:0/3427901904 <== osd.1 v2:172.21.15.167:6800/30035 23 ==== osd_op_reply(63 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fadcc02f080 con 0x5559d5e3f8d0 2023-10-05T19:28:25.673 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.666+0000 7fade3e7c700 1 -- 172.21.15.167:0/3427901904 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:64 5.5 5:a0000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x5559d5e73d70 con 0x7faddc011040 2023-10-05T19:28:25.674 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.670+0000 7fade3e7c700 1 -- 172.21.15.167:0/3427901904 <== osd.0 v2:172.21.15.118:6800/27690 16 ==== osd_op_reply(64 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fadcc02f080 con 0x7faddc011040 2023-10-05T19:28:25.674 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.670+0000 7fade3e7c700 1 -- 172.21.15.167:0/3427901904 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:65 5.15 5:a8000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x5559d5e73d70 con 0x5559d5e746d0 2023-10-05T19:28:25.675 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.670+0000 7fade367b700 1 -- 172.21.15.167:0/3427901904 <== osd.3 v2:172.21.15.167:6808/30036 17 ==== osd_op_reply(65 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fadd4022020 con 0x5559d5e746d0 2023-10-05T19:28:25.675 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.670+0000 7fade367b700 1 -- 172.21.15.167:0/3427901904 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:66 5.d 5:b0000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x5559d5e73d70 con 0x7fadd4007030 2023-10-05T19:28:25.676 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.670+0000 7fade2e7a700 1 -- 172.21.15.167:0/3427901904 <== osd.2 v2:172.21.15.167:6816/30037 10 ==== osd_op_reply(66 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7faddc004520 con 0x7fadd4007030 2023-10-05T19:28:25.676 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.670+0000 7fade2e7a700 1 -- 172.21.15.167:0/3427901904 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:67 5.1d 5:b8000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x5559d5e73d70 con 0x5559d5e3f8d0 2023-10-05T19:28:25.676 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.670+0000 7fade3e7c700 1 -- 172.21.15.167:0/3427901904 <== osd.1 v2:172.21.15.167:6800/30035 24 ==== osd_op_reply(67 [pgnls start_epoch 30 out=107b] v30'2 uv2 ondisk = 1) v8 ==== 144+0+107 (crc 0 0 0) 0x7fadcc02f080 con 0x5559d5e3f8d0 2023-10-05T19:28:25.677 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.670+0000 7fade3e7c700 1 -- 172.21.15.167:0/3427901904 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:68 5.3 5:c0000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x5559d5e73d70 con 0x7faddc011040 2023-10-05T19:28:25.677 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.670+0000 7fade3e7c700 1 -- 172.21.15.167:0/3427901904 <== osd.0 v2:172.21.15.118:6800/27690 17 ==== osd_op_reply(68 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fadcc02f080 con 0x7faddc011040 2023-10-05T19:28:25.678 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.670+0000 7fade3e7c700 1 -- 172.21.15.167:0/3427901904 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:69 5.13 5:c8000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x5559d5e73d70 con 0x5559d5e746d0 2023-10-05T19:28:25.678 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.674+0000 7fade367b700 1 -- 172.21.15.167:0/3427901904 <== osd.3 v2:172.21.15.167:6808/30036 18 ==== osd_op_reply(69 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fadd4022020 con 0x5559d5e746d0 2023-10-05T19:28:25.678 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.674+0000 7fade367b700 1 -- 172.21.15.167:0/3427901904 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:70 5.b 5:d0000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x5559d5e73d70 con 0x7faddc011040 2023-10-05T19:28:25.679 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.674+0000 7fade3e7c700 1 -- 172.21.15.167:0/3427901904 <== osd.0 v2:172.21.15.118:6800/27690 18 ==== osd_op_reply(70 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fadcc02f080 con 0x7faddc011040 2023-10-05T19:28:25.679 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.674+0000 7fade3e7c700 1 -- 172.21.15.167:0/3427901904 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:71 5.1b 5:d8000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x5559d5e73d70 con 0x7fadd4007030 2023-10-05T19:28:25.680 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.674+0000 7fade2e7a700 1 -- 172.21.15.167:0/3427901904 <== osd.2 v2:172.21.15.167:6816/30037 11 ==== osd_op_reply(71 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7faddc004520 con 0x7fadd4007030 2023-10-05T19:28:25.680 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.674+0000 7fade2e7a700 1 -- 172.21.15.167:0/3427901904 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:72 5.7 5:e0000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x5559d5e73d70 con 0x5559d5e746d0 2023-10-05T19:28:25.681 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.674+0000 7fade367b700 1 -- 172.21.15.167:0/3427901904 <== osd.3 v2:172.21.15.167:6808/30036 19 ==== osd_op_reply(72 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fadd4022020 con 0x5559d5e746d0 2023-10-05T19:28:25.681 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.674+0000 7fade367b700 1 -- 172.21.15.167:0/3427901904 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:73 5.17 5:e8000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x5559d5e73d70 con 0x7faddc011040 2023-10-05T19:28:25.682 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.678+0000 7fade3e7c700 1 -- 172.21.15.167:0/3427901904 <== osd.0 v2:172.21.15.118:6800/27690 19 ==== osd_op_reply(73 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fadcc02f080 con 0x7faddc011040 2023-10-05T19:28:25.682 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.678+0000 7fade3e7c700 1 -- 172.21.15.167:0/3427901904 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:74 5.f 5:f0000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x5559d5e73d70 con 0x5559d5e3f8d0 2023-10-05T19:28:25.683 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.678+0000 7fade3e7c700 1 -- 172.21.15.167:0/3427901904 <== osd.1 v2:172.21.15.167:6800/30035 25 ==== osd_op_reply(74 [pgnls start_epoch 30 out=115b] v23'2 uv2 ondisk = 1) v8 ==== 144+0+115 (crc 0 0 0) 0x7fadcc02f080 con 0x5559d5e3f8d0 2023-10-05T19:28:25.683 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.678+0000 7fade3e7c700 1 -- 172.21.15.167:0/3427901904 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:75 5.1f 5:f8000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x5559d5e73d70 con 0x7fadd4007030 2023-10-05T19:28:25.684 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.678+0000 7fade2e7a700 1 -- 172.21.15.167:0/3427901904 <== osd.2 v2:172.21.15.167:6816/30037 12 ==== osd_op_reply(75 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7faddc004520 con 0x7fadd4007030 2023-10-05T19:28:25.684 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.678+0000 7fade5b46e40 20 RGWRados::pool_iterate: got zonegroup_info.4f1f719d-9648-417b-b10c-e5d939825419 2023-10-05T19:28:25.684 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.678+0000 7fade5b46e40 20 RGWRados::pool_iterate: got zone_info.9a9df965-6612-45d4-a45f-6e3c889bacc2 2023-10-05T19:28:25.684 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.678+0000 7fade5b46e40 20 RGWRados::pool_iterate: got zone_names.default 2023-10-05T19:28:25.684 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.678+0000 7fade5b46e40 20 RGWRados::pool_iterate: got zonegroups_names.default 2023-10-05T19:28:25.684 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.678+0000 7fade5b46e40 20 rados->read ofs=0 len=0 2023-10-05T19:28:25.684 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.678+0000 7fade5b46e40 1 -- 172.21.15.167:0/3427901904 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:76 5.f 5:f43b6ece:::zone_names.default:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x5559d5e77be0 con 0x5559d5e3f8d0 2023-10-05T19:28:25.685 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.678+0000 7fade3e7c700 1 -- 172.21.15.167:0/3427901904 <== osd.1 v2:172.21.15.167:6800/30035 26 ==== osd_op_reply(76 zone_names.default [read 0~46 out=46b] v0'0 uv1 ondisk = 0) v8 ==== 162+0+46 (crc 0 0 0) 0x7fadcc02f080 con 0x5559d5e3f8d0 2023-10-05T19:28:25.685 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.678+0000 7fade5b46e40 20 rados_obj.operate() r=0 bl.length=46 2023-10-05T19:28:25.685 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.678+0000 7fade5b46e40 20 rados->read ofs=0 len=0 2023-10-05T19:28:25.685 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.678+0000 7fade5b46e40 1 -- 172.21.15.167:0/3427901904 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:77 5.1d 5:b8374eb4:::zone_info.9a9df965-6612-45d4-a45f-6e3c889bacc2:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x5559d5e77f90 con 0x5559d5e3f8d0 2023-10-05T19:28:25.686 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.678+0000 7fade3e7c700 1 -- 172.21.15.167:0/3427901904 <== osd.1 v2:172.21.15.167:6800/30035 27 ==== osd_op_reply(77 zone_info.9a9df965-6612-45d4-a45f-6e3c889bacc2 [read 0~874 out=874b] v0'0 uv2 ondisk = 0) v8 ==== 190+0+874 (crc 0 0 0) 0x7fadcc02f080 con 0x5559d5e3f8d0 2023-10-05T19:28:25.686 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.678+0000 7fade5b46e40 20 rados_obj.operate() r=0 bl.length=874 2023-10-05T19:28:25.686 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.678+0000 7fade5b46e40 20 rados->read ofs=0 len=0 2023-10-05T19:28:25.686 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.678+0000 7fade5b46e40 1 -- 172.21.15.167:0/3427901904 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:78 5.f 5:f4c53578:::zonegroups_names.default:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x5559d5e739a0 con 0x5559d5e3f8d0 2023-10-05T19:28:25.687 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.682+0000 7fade3e7c700 1 -- 172.21.15.167:0/3427901904 <== osd.1 v2:172.21.15.167:6800/30035 28 ==== osd_op_reply(78 zonegroups_names.default [read 0~46 out=46b] v0'0 uv2 ondisk = 0) v8 ==== 168+0+46 (crc 0 0 0) 0x7fadcc02f080 con 0x5559d5e3f8d0 2023-10-05T19:28:25.687 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.682+0000 7fade5b46e40 20 rados_obj.operate() r=0 bl.length=46 2023-10-05T19:28:25.687 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.682+0000 7fade5b46e40 20 rados->read ofs=0 len=0 2023-10-05T19:28:25.687 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.682+0000 7fade5b46e40 1 -- 172.21.15.167:0/3427901904 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:79 5.c 5:30d299a5:::zonegroup_info.4f1f719d-9648-417b-b10c-e5d939825419:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x5559d5e739b0 con 0x7faddc011040 2023-10-05T19:28:25.688 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.682+0000 7fade3e7c700 1 -- 172.21.15.167:0/3427901904 <== osd.0 v2:172.21.15.118:6800/27690 20 ==== osd_op_reply(79 zonegroup_info.4f1f719d-9648-417b-b10c-e5d939825419 [read 0~362 out=362b] v0'0 uv1 ondisk = 0) v8 ==== 195+0+362 (crc 0 0 0) 0x7fadcc02f080 con 0x7faddc011040 2023-10-05T19:28:25.688 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.682+0000 7fade5b46e40 20 rados_obj.operate() r=0 bl.length=362 2023-10-05T19:28:25.688 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.682+0000 7fade5b46e40 20 rados->read ofs=0 len=0 2023-10-05T19:28:25.688 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.682+0000 7fade5b46e40 1 -- 172.21.15.167:0/3427901904 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:80 5.3 5:c0ad8cbd:::region_map:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x5559d5e78a00 con 0x7faddc011040 2023-10-05T19:28:25.689 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.682+0000 7fade3e7c700 1 -- 172.21.15.167:0/3427901904 <== osd.0 v2:172.21.15.118:6800/27690 21 ==== osd_op_reply(80 region_map [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 ==== 154+0+0 (crc 0 0 0) 0x7fadcc02f080 con 0x7faddc011040 2023-10-05T19:28:25.689 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.682+0000 7fade5b46e40 20 rados_obj.operate() r=-2 bl.length=0 2023-10-05T19:28:25.689 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.682+0000 7fade5b46e40 20 zone default found 2023-10-05T19:28:25.690 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.682+0000 7fade5b46e40 4 Realm: () 2023-10-05T19:28:25.690 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.682+0000 7fade5b46e40 4 ZoneGroup: default (4f1f719d-9648-417b-b10c-e5d939825419) 2023-10-05T19:28:25.690 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.682+0000 7fade5b46e40 4 Zone: default (9a9df965-6612-45d4-a45f-6e3c889bacc2) 2023-10-05T19:28:25.690 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.682+0000 7fade5b46e40 10 cannot find current period zonegroup using local zonegroup configuration 2023-10-05T19:28:25.690 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.682+0000 7fade5b46e40 20 zonegroup default 2023-10-05T19:28:25.690 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.682+0000 7fade5b46e40 20 rados->read ofs=0 len=0 2023-10-05T19:28:25.690 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.682+0000 7fade5b46e40 1 -- 172.21.15.167:0/3427901904 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:81 5.3 5:c52100b6:::period_config.default:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x5559d5e78a00 con 0x7faddc011040 2023-10-05T19:28:25.690 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.682+0000 7fade3e7c700 1 -- 172.21.15.167:0/3427901904 <== osd.0 v2:172.21.15.118:6800/27690 22 ==== osd_op_reply(81 period_config.default [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 ==== 165+0+0 (crc 0 0 0) 0x7fadcc02f080 con 0x7faddc011040 2023-10-05T19:28:25.690 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.682+0000 7fade5b46e40 20 rados_obj.operate() r=-2 bl.length=0 2023-10-05T19:28:25.690 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.682+0000 7fade5b46e40 20 rados->read ofs=0 len=0 2023-10-05T19:28:25.691 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.682+0000 7fade5b46e40 1 -- 172.21.15.167:0/3427901904 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:82 6.2 6:4347d321:::bucket.sync-source-hints.:head [call version.read in=11b,read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x5559d5e7a020 con 0x5559d5e746d0 2023-10-05T19:28:25.692 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.686+0000 7fade367b700 1 -- 172.21.15.167:0/3427901904 <== osd.3 v2:172.21.15.167:6808/30036 20 ==== osd_op_reply(82 bucket.sync-source-hints. [call,read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 ==== 211+0+0 (crc 0 0 0) 0x7fadd4022020 con 0x5559d5e746d0 2023-10-05T19:28:25.692 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.686+0000 7fade5b46e40 20 rados_obj.operate() r=-2 bl.length=0 2023-10-05T19:28:25.692 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.686+0000 7fade5b46e40 20 rados->read ofs=0 len=0 2023-10-05T19:28:25.692 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.686+0000 7fade5b46e40 1 -- 172.21.15.167:0/3427901904 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:83 6.b 6:d467b91b:::bucket.sync-target-hints.:head [call version.read in=11b,read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x5559d5e7a3d0 con 0x5559d5e746d0 2023-10-05T19:28:25.692 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.686+0000 7fade367b700 1 -- 172.21.15.167:0/3427901904 <== osd.3 v2:172.21.15.167:6808/30036 21 ==== osd_op_reply(83 bucket.sync-target-hints. [call,read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 ==== 211+0+0 (crc 0 0 0) 0x7fadd4022020 con 0x5559d5e746d0 2023-10-05T19:28:25.692 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.686+0000 7fade5b46e40 20 rados_obj.operate() r=-2 bl.length=0 2023-10-05T19:28:25.692 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.686+0000 7fade5b46e40 20 started sync module instance, tier type = 2023-10-05T19:28:25.692 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.686+0000 7fade5b46e40 20 started zone id=9a9df965-6612-45d4-a45f-6e3c889bacc2 (name=default) with tier type = 2023-10-05T19:28:25.692 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.686+0000 7fade5b46e40 1 -- 172.21.15.167:0/3427901904 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:84 7.1f 7:f95f44c2:::notify.0:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x5559d5e796e0 con 0x7fadd4007030 2023-10-05T19:28:25.694 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.690+0000 7fade2e7a700 1 -- 172.21.15.167:0/3427901904 <== osd.2 v2:172.21.15.167:6816/30037 13 ==== osd_op_reply(84 notify.0 [create] v30'24 uv24 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7faddc02f040 con 0x7fadd4007030 2023-10-05T19:28:25.694 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.690+0000 7fade5b46e40 1 -- 172.21.15.167:0/3427901904 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:85 7.1f 7:f95f44c2:::notify.0:head [watch watch cookie 93844329175776] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x5559d5e79310 con 0x7fadd4007030 2023-10-05T19:28:25.694 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.690+0000 7fade5b46e40 1 -- 172.21.15.167:0/3427901904 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:86 7.0 7:05bf5b68:::notify.1:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x5559d5e7b1d0 con 0x5559d5e3f8d0 2023-10-05T19:28:25.698 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.690+0000 7fade2e7a700 1 -- 172.21.15.167:0/3427901904 <== osd.2 v2:172.21.15.167:6816/30037 14 ==== osd_op_reply(85 notify.0 [watch watch cookie 93844329175776] v30'25 uv24 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7faddc02f040 con 0x7fadd4007030 2023-10-05T19:28:25.698 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.690+0000 7fade3e7c700 1 -- 172.21.15.167:0/3427901904 <== osd.1 v2:172.21.15.167:6800/30035 29 ==== osd_op_reply(86 notify.1 [create] v30'24 uv24 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7fadcc02f080 con 0x5559d5e3f8d0 2023-10-05T19:28:25.698 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.690+0000 7fade5b46e40 1 -- 172.21.15.167:0/3427901904 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:87 7.0 7:05bf5b68:::notify.1:head [watch watch cookie 93844329179008] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x5559d5e7b5d0 con 0x5559d5e3f8d0 2023-10-05T19:28:25.698 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.690+0000 7fade5b46e40 1 -- 172.21.15.167:0/3427901904 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:88 7.15 7:a93a5511:::notify.2:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x5559d5e7be70 con 0x5559d5e746d0 2023-10-05T19:28:25.701 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.694+0000 7fade367b700 1 -- 172.21.15.167:0/3427901904 <== osd.3 v2:172.21.15.167:6808/30036 22 ==== osd_op_reply(88 notify.2 [create] v30'24 uv24 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7fadd4022020 con 0x5559d5e746d0 2023-10-05T19:28:25.701 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.694+0000 7fade5b46e40 1 -- 172.21.15.167:0/3427901904 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:89 7.15 7:a93a5511:::notify.2:head [watch watch cookie 93844329182672] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x5559d5e7c710 con 0x5559d5e746d0 2023-10-05T19:28:25.701 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.694+0000 7fade5b46e40 1 -- 172.21.15.167:0/3427901904 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:90 7.e 7:7759931f:::notify.3:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x5559d5e7cfb0 con 0x7fadd4007030 2023-10-05T19:28:25.702 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.694+0000 7fade3e7c700 1 -- 172.21.15.167:0/3427901904 <== osd.1 v2:172.21.15.167:6800/30035 30 ==== osd_op_reply(87 notify.1 [watch watch cookie 93844329179008] v30'25 uv24 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7fadcc050040 con 0x5559d5e3f8d0 2023-10-05T19:28:25.703 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.698+0000 7fade2e7a700 1 -- 172.21.15.167:0/3427901904 <== osd.2 v2:172.21.15.167:6816/30037 15 ==== osd_op_reply(90 notify.3 [create] v30'24 uv24 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7faddc02f040 con 0x7fadd4007030 2023-10-05T19:28:25.703 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.698+0000 7fade5b46e40 1 -- 172.21.15.167:0/3427901904 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:91 7.e 7:7759931f:::notify.3:head [watch watch cookie 93844329187088] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x5559d5e7cb60 con 0x7fadd4007030 2023-10-05T19:28:25.703 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.698+0000 7fade5b46e40 1 -- 172.21.15.167:0/3427901904 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:92 7.d 7:b4812045:::notify.4:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x5559d5e7dd90 con 0x5559d5e3f8d0 2023-10-05T19:28:25.704 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.698+0000 7fade367b700 1 -- 172.21.15.167:0/3427901904 <== osd.3 v2:172.21.15.167:6808/30036 23 ==== osd_op_reply(89 notify.2 [watch watch cookie 93844329182672] v30'25 uv24 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7fadd4022020 con 0x5559d5e746d0 2023-10-05T19:28:25.705 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.698+0000 7fade2e7a700 1 -- 172.21.15.167:0/3427901904 <== osd.2 v2:172.21.15.167:6816/30037 16 ==== osd_op_reply(91 notify.3 [watch watch cookie 93844329187088] v30'25 uv24 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7faddc02f040 con 0x7fadd4007030 2023-10-05T19:28:25.705 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.698+0000 7fade3e7c700 1 -- 172.21.15.167:0/3427901904 <== osd.1 v2:172.21.15.167:6800/30035 31 ==== osd_op_reply(92 notify.4 [create] v30'24 uv24 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7fadcc050040 con 0x5559d5e3f8d0 2023-10-05T19:28:25.706 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.698+0000 7fade5b46e40 1 -- 172.21.15.167:0/3427901904 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:93 7.d 7:b4812045:::notify.4:head [watch watch cookie 93844329190320] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x5559d5e7dcf0 con 0x5559d5e3f8d0 2023-10-05T19:28:25.706 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.698+0000 7fade5b46e40 1 -- 172.21.15.167:0/3427901904 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:94 7.3 7:c609908c:::notify.5:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x5559d5e7e590 con 0x7faddc011040 2023-10-05T19:28:25.708 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.702+0000 7fade3e7c700 1 -- 172.21.15.167:0/3427901904 <== osd.1 v2:172.21.15.167:6800/30035 32 ==== osd_op_reply(93 notify.4 [watch watch cookie 93844329190320] v30'25 uv24 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7fadcc050040 con 0x5559d5e3f8d0 2023-10-05T19:28:25.709 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.702+0000 7fade3e7c700 1 -- 172.21.15.167:0/3427901904 <== osd.0 v2:172.21.15.118:6800/27690 23 ==== osd_op_reply(94 notify.5 [create] v30'24 uv24 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7fadcc050040 con 0x7faddc011040 2023-10-05T19:28:25.709 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.702+0000 7fade5b46e40 1 -- 172.21.15.167:0/3427901904 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:95 7.3 7:c609908c:::notify.5:head [watch watch cookie 93844329191984] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x5559d5e7e370 con 0x7faddc011040 2023-10-05T19:28:25.709 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.702+0000 7fade5b46e40 1 -- 172.21.15.167:0/3427901904 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:96 7.14 7:2b04a3e9:::notify.6:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x5559d5e7ee50 con 0x7faddc011040 2023-10-05T19:28:25.712 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.706+0000 7fade3e7c700 1 -- 172.21.15.167:0/3427901904 <== osd.0 v2:172.21.15.118:6800/27690 24 ==== osd_op_reply(96 notify.6 [create] v30'24 uv24 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7fadcc050040 con 0x7faddc011040 2023-10-05T19:28:25.712 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.706+0000 7fade5b46e40 1 -- 172.21.15.167:0/3427901904 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:97 7.14 7:2b04a3e9:::notify.6:head [watch watch cookie 93844329195808] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x5559d5e7f260 con 0x7faddc011040 2023-10-05T19:28:25.712 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.706+0000 7fade5b46e40 1 -- 172.21.15.167:0/3427901904 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:98 7.9 7:93e5b521:::notify.7:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x5559d5e7fd40 con 0x7faddc011040 2023-10-05T19:28:25.713 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.706+0000 7fade3e7c700 1 -- 172.21.15.167:0/3427901904 <== osd.0 v2:172.21.15.118:6800/27690 25 ==== osd_op_reply(95 notify.5 [watch watch cookie 93844329191984] v30'25 uv24 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7fadcc050040 con 0x7faddc011040 2023-10-05T19:28:25.715 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.710+0000 7fade3e7c700 1 -- 172.21.15.167:0/3427901904 <== osd.0 v2:172.21.15.118:6800/27690 26 ==== osd_op_reply(98 notify.7 [create] v30'24 uv24 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7fadcc050040 con 0x7faddc011040 2023-10-05T19:28:25.715 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.710+0000 7fade3e7c700 1 -- 172.21.15.167:0/3427901904 <== osd.0 v2:172.21.15.118:6800/27690 27 ==== osd_op_reply(97 notify.6 [watch watch cookie 93844329195808] v30'25 uv24 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7fadcc050040 con 0x7faddc011040 2023-10-05T19:28:25.715 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.710+0000 7fade5b46e40 1 -- 172.21.15.167:0/3427901904 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:99 7.9 7:93e5b521:::notify.7:head [watch watch cookie 93844329196912] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x5559d5e7f810 con 0x7faddc011040 2023-10-05T19:28:25.715 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.710+0000 7fade5b46e40 20 add_watcher() i=0 2023-10-05T19:28:25.715 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.710+0000 7fade5b46e40 20 add_watcher() i=1 2023-10-05T19:28:25.715 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.710+0000 7fade5b46e40 20 add_watcher() i=2 2023-10-05T19:28:25.716 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.710+0000 7fade5b46e40 20 add_watcher() i=3 2023-10-05T19:28:25.716 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.710+0000 7fade5b46e40 20 add_watcher() i=4 2023-10-05T19:28:25.716 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.710+0000 7fade5b46e40 20 add_watcher() i=5 2023-10-05T19:28:25.716 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.710+0000 7fade5b46e40 20 add_watcher() i=6 2023-10-05T19:28:25.717 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.710+0000 7fade3e7c700 1 -- 172.21.15.167:0/3427901904 <== osd.0 v2:172.21.15.118:6800/27690 28 ==== osd_op_reply(99 notify.7 [watch watch cookie 93844329196912] v30'25 uv24 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7fadcc050040 con 0x7faddc011040 2023-10-05T19:28:25.717 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.710+0000 7fade5b46e40 20 add_watcher() i=7 2023-10-05T19:28:25.717 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.710+0000 7fade5b46e40 2 all 8 watchers are set, enabling cache 2023-10-05T19:28:25.717 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.710+0000 7fade5b46e40 1 -- 172.21.15.167:0/3427901904 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:100 6.4 6:22d26bf9:::data_loggenerations_metadata:head [call version.check_conds in=74b,call version.read in=11b,read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x5559d5e82d30 con 0x5559d5e3f8d0 2023-10-05T19:28:25.718 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.714+0000 7fade3e7c700 1 -- 172.21.15.167:0/3427901904 <== osd.1 v2:172.21.15.167:6800/30035 33 ==== osd_op_reply(100 data_loggenerations_metadata [call,call out=48b,read 0~28 out=28b] v0'0 uv1 ondisk = 0) v8 ==== 256+0+76 (crc 0 0 0) 0x7fadcc050040 con 0x5559d5e3f8d0 2023-10-05T19:28:25.718 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.714+0000 7fade5b46e40 1 -- 172.21.15.167:0/3427901904 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:101 6.4 6:22d26bf9:::data_loggenerations_metadata:head [watch watch cookie 93844329199632] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x5559d5e80150 con 0x5559d5e3f8d0 2023-10-05T19:28:25.720 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.714+0000 7fade3e7c700 1 -- 172.21.15.167:0/3427901904 <== osd.1 v2:172.21.15.167:6800/30035 34 ==== osd_op_reply(101 data_loggenerations_metadata [watch watch cookie 93844329199632] v30'50 uv1 ondisk = 0) v8 ==== 172+0+0 (crc 0 0 0) 0x7fadcc050040 con 0x5559d5e3f8d0 2023-10-05T19:28:25.720 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.714+0000 7fadb3fef700 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2023-10-05T19:28:25.721 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.714+0000 7fade5b46e40 20 check_secure_mon_conn(): auth registy supported: methods=[2] modes=[2,1] 2023-10-05T19:28:25.721 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.714+0000 7fade5b46e40 20 check_secure_mon_conn(): mode 1 is insecure 2023-10-05T19:28:25.721 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.714+0000 7fade5b46e40 1 Processor -- start 2023-10-05T19:28:25.721 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.714+0000 7fade5b46e40 1 -- start start 2023-10-05T19:28:25.722 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.714+0000 7fade5b46e40 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5559d5e8d1d0 0x5559d5e8d5a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-05T19:28:25.722 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.714+0000 7fade5b46e40 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x5559d5e8dae0 con 0x5559d5e8d1d0 2023-10-05T19:28:25.722 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.714+0000 7fade2e7a700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5559d5e8d1d0 0x5559d5e8d5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-05T19:28:25.722 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.718+0000 7fade2e7a700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5559d5e8d1d0 0x5559d5e8d5a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.118:3300/0 says I am v2:172.21.15.167:57184/0 (socket says 172.21.15.167:57184) 2023-10-05T19:28:25.722 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.718+0000 7fade2e7a700 1 -- 172.21.15.167:0/2502406754 learned_addr learned my addr 172.21.15.167:0/2502406754 (peer_addr_for_me v2:172.21.15.167:0/0) 2023-10-05T19:28:25.723 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.718+0000 7fade2e7a700 1 -- 172.21.15.167:0/2502406754 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5559d5e8dc20 con 0x5559d5e8d1d0 2023-10-05T19:28:25.723 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.718+0000 7fade2e7a700 1 --2- 172.21.15.167:0/2502406754 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5559d5e8d1d0 0x5559d5e8d5a0 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x5559d5e72170 tx=0x7faddc006000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fe9cf109e8b9a232 server_cookie=0 in_seq=0 out_seq=0 2023-10-05T19:28:25.723 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.718+0000 7fadb37ee700 1 -- 172.21.15.167:0/2502406754 <== mon.0 v2:172.21.15.118:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7faddc02c040 con 0x5559d5e8d1d0 2023-10-05T19:28:25.723 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.718+0000 7fadb37ee700 1 -- 172.21.15.167:0/2502406754 <== mon.0 v2:172.21.15.118:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7faddc02d040 con 0x5559d5e8d1d0 2023-10-05T19:28:25.724 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.718+0000 7fadb37ee700 1 -- 172.21.15.167:0/2502406754 <== mon.0 v2:172.21.15.118:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7faddc0289c0 con 0x5559d5e8d1d0 2023-10-05T19:28:25.724 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.718+0000 7fade5b46e40 1 -- 172.21.15.167:0/2502406754 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5559d5e8d1d0 msgr2=0x5559d5e8d5a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-05T19:28:25.725 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.718+0000 7fade5b46e40 1 --2- 172.21.15.167:0/2502406754 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5559d5e8d1d0 0x5559d5e8d5a0 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x5559d5e72170 tx=0x7faddc006000 comp rx=0 tx=0).stop 2023-10-05T19:28:25.725 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.718+0000 7fade5b46e40 1 -- 172.21.15.167:0/2502406754 shutdown_connections 2023-10-05T19:28:25.725 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.718+0000 7fade5b46e40 1 --2- 172.21.15.167:0/2502406754 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5559d5e8d1d0 0x5559d5e8d5a0 unknown :-1 s=CLOSED pgs=186 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-05T19:28:25.725 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.718+0000 7fade5b46e40 1 -- 172.21.15.167:0/2502406754 >> 172.21.15.167:0/2502406754 conn(0x5559d5e88590 msgr2=0x5559d5e866e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-05T19:28:25.725 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.718+0000 7fade5b46e40 1 -- 172.21.15.167:0/2502406754 shutdown_connections 2023-10-05T19:28:25.725 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.718+0000 7fade5b46e40 1 -- 172.21.15.167:0/2502406754 wait complete. 2023-10-05T19:28:25.726 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.718+0000 7fade5b46e40 1 Processor -- start 2023-10-05T19:28:25.726 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.718+0000 7fade5b46e40 1 -- start start 2023-10-05T19:28:25.726 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.718+0000 7fade5b46e40 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5559d5e8d1d0 0x5559d5e92d60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-05T19:28:25.726 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.718+0000 7fade5b46e40 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x5559d5e93d60 con 0x5559d5e8d1d0 2023-10-05T19:28:25.726 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.722+0000 7fade2e7a700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5559d5e8d1d0 0x5559d5e92d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-05T19:28:25.726 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.722+0000 7fade2e7a700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5559d5e8d1d0 0x5559d5e92d60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.118:3300/0 says I am v2:172.21.15.167:57196/0 (socket says 172.21.15.167:57196) 2023-10-05T19:28:25.726 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.722+0000 7fade2e7a700 1 -- 172.21.15.167:0/231899477 learned_addr learned my addr 172.21.15.167:0/231899477 (peer_addr_for_me v2:172.21.15.167:0/0) 2023-10-05T19:28:25.727 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.722+0000 7fade2e7a700 1 -- 172.21.15.167:0/231899477 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5559d5d89bd0 con 0x5559d5e8d1d0 2023-10-05T19:28:25.727 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.722+0000 7fade2e7a700 1 --2- 172.21.15.167:0/231899477 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5559d5e8d1d0 0x5559d5e92d60 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7faddc041040 tx=0x7faddc03c000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-05T19:28:25.727 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.722+0000 7fadb1feb700 1 -- 172.21.15.167:0/231899477 <== mon.0 v2:172.21.15.118:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7faddc02c030 con 0x5559d5e8d1d0 2023-10-05T19:28:25.728 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.722+0000 7fadb1feb700 1 -- 172.21.15.167:0/231899477 <== mon.0 v2:172.21.15.118:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7faddc01c040 con 0x5559d5e8d1d0 2023-10-05T19:28:25.728 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.722+0000 7fadb1feb700 1 -- 172.21.15.167:0/231899477 <== mon.0 v2:172.21.15.118:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7faddc004050 con 0x5559d5e8d1d0 2023-10-05T19:28:25.728 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.722+0000 7fade5b46e40 1 -- 172.21.15.167:0/231899477 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x5559d5e93360 con 0x5559d5e8d1d0 2023-10-05T19:28:25.728 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.722+0000 7fade5b46e40 1 -- 172.21.15.167:0/231899477 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x5559d5e937a0 con 0x5559d5e8d1d0 2023-10-05T19:28:25.728 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.722+0000 7fadb1feb700 1 -- 172.21.15.167:0/231899477 <== mon.0 v2:172.21.15.118:3300/0 4 ==== mgrmap(e 5) v1 ==== 40329+0+0 (secure 0 0 0) 0x7faddc0041b0 con 0x5559d5e8d1d0 2023-10-05T19:28:25.729 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.722+0000 7fadb1feb700 1 --2- 172.21.15.167:0/231899477 >> [v2:172.21.15.167:6824/29889,v1:172.21.15.167:6825/29889] conn(0x7fad28030c50 0x7fad28033100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-05T19:28:25.729 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.722+0000 7fadb1feb700 1 -- 172.21.15.167:0/231899477 <== mon.0 v2:172.21.15.118:3300/0 5 ==== osd_map(30..30 src has 1..30) v4 ==== 6057+0+0 (secure 0 0 0) 0x7faddc053c60 con 0x5559d5e8d1d0 2023-10-05T19:28:25.729 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.722+0000 7fade5b46e40 5 note: GC not initialized 2023-10-05T19:28:25.729 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.722+0000 7fade3e7c700 1 --2- 172.21.15.167:0/231899477 >> [v2:172.21.15.167:6824/29889,v1:172.21.15.167:6825/29889] conn(0x7fad28030c50 0x7fad28033100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-05T19:28:25.730 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.726+0000 7fade3e7c700 1 --2- 172.21.15.167:0/231899477 >> [v2:172.21.15.167:6824/29889,v1:172.21.15.167:6825/29889] conn(0x7fad28030c50 0x7fad28033100 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x5559d5e7e0d0 tx=0x7fadcc000f90 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-05T19:28:25.732 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.726+0000 7fad22ffd700 20 reqs_thread_entry: start 2023-10-05T19:28:25.732 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.726+0000 7fad20ff9700 20 reqs_thread_entry: start 2023-10-05T19:28:25.732 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.726+0000 7fade5b46e40 20 init_complete bucket index max shards: 11 2023-10-05T19:28:25.733 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.726+0000 7fade5b46e40 10 rgw notify: Started notification manager with: 1 workers 2023-10-05T19:28:25.733 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.726+0000 7fade5b46e40 20 get_system_obj_state: rctx=0x7ffd74c8e028 obj=default.rgw.meta:users.uid:foo state=0x5559d5d160b0 s->prefetch_data=0 2023-10-05T19:28:25.733 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.726+0000 7fade5b46e40 10 cache get: name=default.rgw.meta+users.uid+foo : miss 2023-10-05T19:28:25.733 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.726+0000 7fad16ffd700 20 rgw notify: INFO: next queues processing will happen at: Thu Oct 5 19:28:56 2023 2023-10-05T19:28:25.733 INFO:teuthology.orchestra.run.smithi167.stderr: 2023-10-05T19:28:25.733 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.726+0000 7fade5b46e40 1 -- 172.21.15.167:0/3427901904 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:102 8.18 8:1e6ca5ed:users.uid::foo:head [call version.read in=11b,getxattrs,stat] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x5559d5efb550 con 0x5559d5e3f8d0 2023-10-05T19:28:25.734 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.726+0000 7fad167fc700 20 reqs_thread_entry: start 2023-10-05T19:28:25.735 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.730+0000 7fade3e7c700 1 -- 172.21.15.167:0/3427901904 <== osd.1 v2:172.21.15.167:6800/30035 35 ==== osd_op_reply(102 foo [call,getxattrs,stat] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 ==== 231+0+0 (crc 0 0 0) 0x7fadcc050040 con 0x5559d5e3f8d0 2023-10-05T19:28:25.735 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.730+0000 7fade5b46e40 10 cache put: name=default.rgw.meta+users.uid+foo info.flags=0x0 2023-10-05T19:28:25.735 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.730+0000 7fade5b46e40 10 adding default.rgw.meta+users.uid+foo to cache LRU end 2023-10-05T19:28:25.735 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.730+0000 7fade5b46e40 20 get_system_obj_state: rctx=0x7ffd74c8dd18 obj=default.rgw.meta:users.email:foo@foo.com state=0x5559d5d160b0 s->prefetch_data=0 2023-10-05T19:28:25.735 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.730+0000 7fade5b46e40 10 cache get: name=default.rgw.meta+users.email+foo@foo.com : miss 2023-10-05T19:28:25.735 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.730+0000 7fade5b46e40 1 -- 172.21.15.167:0/3427901904 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:103 8.3 8:c162d5d1:users.email::foo@foo.com:head [getxattrs,stat] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x5559d5efb550 con 0x5559d5e3f8d0 2023-10-05T19:28:25.735 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.730+0000 7fade3e7c700 1 -- 172.21.15.167:0/3427901904 <== osd.1 v2:172.21.15.167:6800/30035 36 ==== osd_op_reply(103 foo@foo.com [getxattrs,stat] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 ==== 197+0+0 (crc 0 0 0) 0x7fadcc050040 con 0x5559d5e3f8d0 2023-10-05T19:28:25.735 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.730+0000 7fade5b46e40 10 cache put: name=default.rgw.meta+users.email+foo@foo.com info.flags=0x0 2023-10-05T19:28:25.735 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.730+0000 7fade5b46e40 10 adding default.rgw.meta+users.email+foo@foo.com to cache LRU end 2023-10-05T19:28:25.735 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.730+0000 7fade5b46e40 20 get_system_obj_state: rctx=0x7ffd74c8dd18 obj=default.rgw.meta:users.keys:9te6NH5mcdcq0Tc5i8i1 state=0x5559d5d160b0 s->prefetch_data=0 2023-10-05T19:28:25.736 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.730+0000 7fade5b46e40 10 cache get: name=default.rgw.meta+users.keys+9te6NH5mcdcq0Tc5i8i1 : miss 2023-10-05T19:28:25.736 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.730+0000 7fade5b46e40 1 -- 172.21.15.167:0/3427901904 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:104 8.7 8:e4e0e270:users.keys::9te6NH5mcdcq0Tc5i8i1:head [getxattrs,stat] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x5559d5efb550 con 0x7faddc011040 2023-10-05T19:28:25.736 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.730+0000 7fade3e7c700 1 -- 172.21.15.167:0/3427901904 <== osd.0 v2:172.21.15.118:6800/27690 29 ==== osd_op_reply(104 9te6NH5mcdcq0Tc5i8i1 [getxattrs,stat] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 ==== 206+0+0 (crc 0 0 0) 0x7fadcc050040 con 0x7faddc011040 2023-10-05T19:28:25.737 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.730+0000 7fade5b46e40 10 cache put: name=default.rgw.meta+users.keys+9te6NH5mcdcq0Tc5i8i1 info.flags=0x0 2023-10-05T19:28:25.737 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.730+0000 7fade5b46e40 10 adding default.rgw.meta+users.keys+9te6NH5mcdcq0Tc5i8i1 to cache LRU end 2023-10-05T19:28:25.737 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.730+0000 7fade5b46e40 20 get_system_obj_state: rctx=0x7ffd74c8d5b8 obj=default.rgw.meta:users.keys:9te6NH5mcdcq0Tc5i8i1 state=0x5559d5d160b0 s->prefetch_data=0 2023-10-05T19:28:25.737 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.730+0000 7fade5b46e40 10 cache get: name=default.rgw.meta+users.keys+9te6NH5mcdcq0Tc5i8i1 : hit (negative entry) 2023-10-05T19:28:25.737 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.730+0000 7fade5b46e40 20 get_system_obj_state: rctx=0x7ffd74c8dc58 obj=default.rgw.meta:users.keys:9te6NH5mcdcq0Tc5i8i1 state=0x5559d5d160b0 s->prefetch_data=0 2023-10-05T19:28:25.737 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.730+0000 7fade5b46e40 10 cache get: name=default.rgw.meta+users.keys+9te6NH5mcdcq0Tc5i8i1 : hit (negative entry) 2023-10-05T19:28:25.737 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.730+0000 7fade5b46e40 1 -- 172.21.15.167:0/3427901904 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:105 8.18 8:1e6ca5ed:users.uid::foo:head [delete,create,call version.set in=58b,writefull 0~357 in=357b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x5559d5efdfe0 con 0x5559d5e3f8d0 2023-10-05T19:28:25.739 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.734+0000 7fade3e7c700 1 -- 172.21.15.167:0/3427901904 <== osd.1 v2:172.21.15.167:6800/30035 37 ==== osd_op_reply(105 foo [delete,create,call,writefull 0~357] v30'1 uv1 ondisk = 0) v8 ==== 273+0+0 (crc 0 0 0) 0x7fadcc050040 con 0x5559d5e3f8d0 2023-10-05T19:28:25.739 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.734+0000 7fade5b46e40 10 cache put: name=default.rgw.meta+users.uid+foo info.flags=0x17 2023-10-05T19:28:25.739 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.734+0000 7fade5b46e40 10 moving default.rgw.meta+users.uid+foo to cache LRU end 2023-10-05T19:28:25.739 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.734+0000 7fade5b46e40 10 distributing notification oid=default.rgw.control:notify.4 cni=[op: 0, obj: default.rgw.meta:users.uid:foo, ofs0, ns] 2023-10-05T19:28:25.739 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.734+0000 7fade5b46e40 1 -- 172.21.15.167:0/3427901904 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:106 7.d 7:b4812045:::notify.4:head [notify cookie 93844329712672 in=545b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x5559d5efe980 con 0x5559d5e3f8d0 2023-10-05T19:28:25.740 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.734+0000 7fade3e7c700 1 -- 172.21.15.167:0/3427901904 <== osd.1 v2:172.21.15.167:6800/30035 38 ==== watch-notify(notify (1) cookie 93844329190320 notify 128849018880 ret 0) v3 ==== 575+0+0 (crc 0 0 0) 0x7fadcc03d040 con 0x5559d5e3f8d0 2023-10-05T19:28:25.740 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.734+0000 7fade1677700 10 rgw watcher librados: RGWWatcher::handle_notify() notify_id 128849018880 cookie 93844329190320 notifier 4275 bl.length()=533 2023-10-05T19:28:25.740 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.734+0000 7fade3e7c700 1 -- 172.21.15.167:0/3427901904 <== osd.1 v2:172.21.15.167:6800/30035 39 ==== osd_op_reply(106 notify.4 [notify cookie 93844329712672 out=8b] v0'0 uv24 ondisk = 0) v8 ==== 152+0+8 (crc 0 0 0) 0x7fadcc050040 con 0x5559d5e3f8d0 2023-10-05T19:28:25.740 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.734+0000 7fade1677700 10 rgw watcher librados: cache put: name=default.rgw.meta+users.uid+foo info.flags=0x17 2023-10-05T19:28:25.740 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.734+0000 7fade1677700 10 rgw watcher librados: moving default.rgw.meta+users.uid+foo to cache LRU end 2023-10-05T19:28:25.740 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.734+0000 7fade1677700 1 -- 172.21.15.167:0/3427901904 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:107 7.d 7:b4812045:::notify.4:head [notify-ack in=20b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x7fadbc003d90 con 0x5559d5e3f8d0 2023-10-05T19:28:25.741 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.734+0000 7fade3e7c700 1 -- 172.21.15.167:0/3427901904 <== osd.1 v2:172.21.15.167:6800/30035 40 ==== watch-notify(notify_complete (2) cookie 93844329712672 notify 128849018880 ret 0) v3 ==== 42+0+48 (crc 0 0 0) 0x7fadcc033780 con 0x5559d5e3f8d0 2023-10-05T19:28:25.741 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.734+0000 7fade3e7c700 1 -- 172.21.15.167:0/3427901904 <== osd.1 v2:172.21.15.167:6800/30035 41 ==== osd_op_reply(107 notify.4 [notify-ack] v0'0 uv24 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7fadcc050040 con 0x5559d5e3f8d0 2023-10-05T19:28:25.741 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.734+0000 7fade5b46e40 1 -- 172.21.15.167:0/3427901904 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:108 8.3 8:c162d5d1:users.email::foo@foo.com:head [delete,create,writefull 0~7 in=7b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x5559d5efe880 con 0x5559d5e3f8d0 2023-10-05T19:28:25.745 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.738+0000 7fade3e7c700 1 -- 172.21.15.167:0/3427901904 <== osd.1 v2:172.21.15.167:6800/30035 42 ==== osd_op_reply(108 foo@foo.com [delete,create,writefull 0~7] v30'1 uv1 ondisk = 0) v8 ==== 239+0+0 (crc 0 0 0) 0x7fadcc050040 con 0x5559d5e3f8d0 2023-10-05T19:28:25.745 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.738+0000 7fade5b46e40 10 cache put: name=default.rgw.meta+users.email+foo@foo.com info.flags=0x7 2023-10-05T19:28:25.745 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.738+0000 7fade5b46e40 10 moving default.rgw.meta+users.email+foo@foo.com to cache LRU end 2023-10-05T19:28:25.746 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.738+0000 7fade5b46e40 10 distributing notification oid=default.rgw.control:notify.3 cni=[op: 0, obj: default.rgw.meta:users.email:foo@foo.com, ofs0, ns] 2023-10-05T19:28:25.746 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.738+0000 7fade5b46e40 1 -- 172.21.15.167:0/3427901904 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:109 7.e 7:7759931f:::notify.3:head [notify cookie 93844329712672 in=181b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x5559d5efe8e0 con 0x7fadd4007030 2023-10-05T19:28:25.746 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.738+0000 7fade2e7a700 1 -- 172.21.15.167:0/3427901904 <== osd.2 v2:172.21.15.167:6816/30037 17 ==== watch-notify(notify (1) cookie 93844329187088 notify 128849018880 ret 0) v3 ==== 211+0+0 (crc 0 0 0) 0x5559d5e937a0 con 0x7fadd4007030 2023-10-05T19:28:25.746 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.738+0000 7fade1677700 10 rgw watcher librados: RGWWatcher::handle_notify() notify_id 128849018880 cookie 93844329187088 notifier 4275 bl.length()=169 2023-10-05T19:28:25.746 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.738+0000 7fade1677700 10 rgw watcher librados: cache put: name=default.rgw.meta+users.email+foo@foo.com info.flags=0x7 2023-10-05T19:28:25.746 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.738+0000 7fade1677700 10 rgw watcher librados: moving default.rgw.meta+users.email+foo@foo.com to cache LRU end 2023-10-05T19:28:25.746 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.738+0000 7fade2e7a700 1 -- 172.21.15.167:0/3427901904 <== osd.2 v2:172.21.15.167:6816/30037 18 ==== osd_op_reply(109 notify.3 [notify cookie 93844329712672 out=8b] v0'0 uv24 ondisk = 0) v8 ==== 152+0+8 (crc 0 0 0) 0x7faddc045040 con 0x7fadd4007030 2023-10-05T19:28:25.746 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.738+0000 7fade1677700 1 -- 172.21.15.167:0/3427901904 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:110 7.e 7:7759931f:::notify.3:head [notify-ack in=20b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x7fadbc003d90 con 0x7fadd4007030 2023-10-05T19:28:25.747 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.738+0000 7fade2e7a700 1 -- 172.21.15.167:0/3427901904 <== osd.2 v2:172.21.15.167:6816/30037 19 ==== osd_op_reply(110 notify.3 [notify-ack] v0'0 uv24 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7faddc045040 con 0x7fadd4007030 2023-10-05T19:28:25.747 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.742+0000 7fade2e7a700 1 -- 172.21.15.167:0/3427901904 <== osd.2 v2:172.21.15.167:6816/30037 20 ==== watch-notify(notify_complete (2) cookie 93844329712672 notify 128849018880 ret 0) v3 ==== 42+0+48 (crc 0 0 0) 0x5559d5e93360 con 0x7fadd4007030 2023-10-05T19:28:25.747 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.742+0000 7fade5b46e40 1 -- 172.21.15.167:0/3427901904 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:111 8.7 8:e4e0e270:users.keys::9te6NH5mcdcq0Tc5i8i1:head [delete,create,writefull 0~7 in=7b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x5559d5efa540 con 0x7faddc011040 2023-10-05T19:28:25.750 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.742+0000 7fade3e7c700 1 -- 172.21.15.167:0/3427901904 <== osd.0 v2:172.21.15.118:6800/27690 30 ==== osd_op_reply(111 9te6NH5mcdcq0Tc5i8i1 [delete,create,writefull 0~7] v30'1 uv1 ondisk = 0) v8 ==== 248+0+0 (crc 0 0 0) 0x7fadcc050040 con 0x7faddc011040 2023-10-05T19:28:25.750 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.742+0000 7fade5b46e40 10 cache put: name=default.rgw.meta+users.keys+9te6NH5mcdcq0Tc5i8i1 info.flags=0x7 2023-10-05T19:28:25.750 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.742+0000 7fade5b46e40 10 moving default.rgw.meta+users.keys+9te6NH5mcdcq0Tc5i8i1 to cache LRU end 2023-10-05T19:28:25.750 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.742+0000 7fade5b46e40 10 distributing notification oid=default.rgw.control:notify.3 cni=[op: 0, obj: default.rgw.meta:users.keys:9te6NH5mcdcq0Tc5i8i1, ofs0, ns] 2023-10-05T19:28:25.750 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.742+0000 7fade5b46e40 1 -- 172.21.15.167:0/3427901904 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:112 7.e 7:7759931f:::notify.3:head [notify cookie 93844329712672 in=189b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x5559d5efa540 con 0x7fadd4007030 2023-10-05T19:28:25.751 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.746+0000 7fade2e7a700 1 -- 172.21.15.167:0/3427901904 <== osd.2 v2:172.21.15.167:6816/30037 21 ==== watch-notify(notify (1) cookie 93844329187088 notify 128849018881 ret 0) v3 ==== 219+0+0 (crc 0 0 0) 0x5559d5e93360 con 0x7fadd4007030 2023-10-05T19:28:25.751 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.746+0000 7fade1677700 10 rgw watcher librados: RGWWatcher::handle_notify() notify_id 128849018881 cookie 93844329187088 notifier 4275 bl.length()=177 2023-10-05T19:28:25.751 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.746+0000 7fade1677700 10 rgw watcher librados: cache put: name=default.rgw.meta+users.keys+9te6NH5mcdcq0Tc5i8i1 info.flags=0x7 2023-10-05T19:28:25.751 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.746+0000 7fade1677700 10 rgw watcher librados: moving default.rgw.meta+users.keys+9te6NH5mcdcq0Tc5i8i1 to cache LRU end 2023-10-05T19:28:25.752 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.746+0000 7fade1677700 1 -- 172.21.15.167:0/3427901904 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:113 7.e 7:7759931f:::notify.3:head [notify-ack in=20b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x7fadbc003d90 con 0x7fadd4007030 2023-10-05T19:28:25.752 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.746+0000 7fade2e7a700 1 -- 172.21.15.167:0/3427901904 <== osd.2 v2:172.21.15.167:6816/30037 22 ==== osd_op_reply(112 notify.3 [notify cookie 93844329712672 out=8b] v0'0 uv24 ondisk = 0) v8 ==== 152+0+8 (crc 0 0 0) 0x7faddc045040 con 0x7fadd4007030 2023-10-05T19:28:25.752 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.746+0000 7fade2e7a700 1 -- 172.21.15.167:0/3427901904 <== osd.2 v2:172.21.15.167:6816/30037 23 ==== osd_op_reply(113 notify.3 [notify-ack] v0'0 uv24 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7faddc045040 con 0x7fadd4007030 2023-10-05T19:28:25.752 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.746+0000 7fade2e7a700 1 -- 172.21.15.167:0/3427901904 <== osd.2 v2:172.21.15.167:6816/30037 24 ==== watch-notify(notify_complete (2) cookie 93844329712672 notify 128849018881 ret 0) v3 ==== 42+0+48 (crc 0 0 0) 0x5559d5d89bd0 con 0x7fadd4007030 2023-10-05T19:28:25.752 INFO:teuthology.orchestra.run.smithi167.stdout:{ 2023-10-05T19:28:25.752 INFO:teuthology.orchestra.run.smithi167.stdout: "user_id": "foo", 2023-10-05T19:28:25.752 INFO:teuthology.orchestra.run.smithi167.stdout: "display_name": "Foo", 2023-10-05T19:28:25.752 INFO:teuthology.orchestra.run.smithi167.stdout: "email": "foo@foo.com", 2023-10-05T19:28:25.752 INFO:teuthology.orchestra.run.smithi167.stdout: "suspended": 0, 2023-10-05T19:28:25.753 INFO:teuthology.orchestra.run.smithi167.stdout: "max_buckets": 4, 2023-10-05T19:28:25.753 INFO:teuthology.orchestra.run.smithi167.stdout: "subusers": [], 2023-10-05T19:28:25.753 INFO:teuthology.orchestra.run.smithi167.stdout: "keys": [ 2023-10-05T19:28:25.753 INFO:teuthology.orchestra.run.smithi167.stdout: { 2023-10-05T19:28:25.753 INFO:teuthology.orchestra.run.smithi167.stdout: "user": "foo", 2023-10-05T19:28:25.753 INFO:teuthology.orchestra.run.smithi167.stdout: "access_key": "9te6NH5mcdcq0Tc5i8i1", 2023-10-05T19:28:25.753 INFO:teuthology.orchestra.run.smithi167.stdout: "secret_key": "Ny4IOauQoL18Gp2zM7lC1vLmoawgqcYP/YGcWfXu" 2023-10-05T19:28:25.753 INFO:teuthology.orchestra.run.smithi167.stdout: } 2023-10-05T19:28:25.753 INFO:teuthology.orchestra.run.smithi167.stdout: ], 2023-10-05T19:28:25.753 INFO:teuthology.orchestra.run.smithi167.stdout: "swift_keys": [], 2023-10-05T19:28:25.753 INFO:teuthology.orchestra.run.smithi167.stdout: "caps": [], 2023-10-05T19:28:25.753 INFO:teuthology.orchestra.run.smithi167.stdout: "op_mask": "read, write, delete", 2023-10-05T19:28:25.754 INFO:teuthology.orchestra.run.smithi167.stdout: "default_placement": "", 2023-10-05T19:28:25.754 INFO:teuthology.orchestra.run.smithi167.stdout: "default_storage_class": "", 2023-10-05T19:28:25.754 INFO:teuthology.orchestra.run.smithi167.stdout: "placement_tags": [], 2023-10-05T19:28:25.754 INFO:teuthology.orchestra.run.smithi167.stdout: "bucket_quota": { 2023-10-05T19:28:25.754 INFO:teuthology.orchestra.run.smithi167.stdout: "enabled": false, 2023-10-05T19:28:25.754 INFO:teuthology.orchestra.run.smithi167.stdout: "check_on_raw": false, 2023-10-05T19:28:25.754 INFO:teuthology.orchestra.run.smithi167.stdout: "max_size": -1, 2023-10-05T19:28:25.754 INFO:teuthology.orchestra.run.smithi167.stdout: "max_size_kb": 0, 2023-10-05T19:28:25.754 INFO:teuthology.orchestra.run.smithi167.stdout: "max_objects": -1 2023-10-05T19:28:25.754 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2023-10-05T19:28:25.754 INFO:teuthology.orchestra.run.smithi167.stdout: "user_quota": { 2023-10-05T19:28:25.754 INFO:teuthology.orchestra.run.smithi167.stdout: "enabled": false, 2023-10-05T19:28:25.755 INFO:teuthology.orchestra.run.smithi167.stdout: "check_on_raw": false, 2023-10-05T19:28:25.755 INFO:teuthology.orchestra.run.smithi167.stdout: "max_size": -1, 2023-10-05T19:28:25.755 INFO:teuthology.orchestra.run.smithi167.stdout: "max_size_kb": 0, 2023-10-05T19:28:25.755 INFO:teuthology.orchestra.run.smithi167.stdout: "max_objects": -1 2023-10-05T19:28:25.755 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2023-10-05T19:28:25.755 INFO:teuthology.orchestra.run.smithi167.stdout: "temp_url_keys": [], 2023-10-05T19:28:25.755 INFO:teuthology.orchestra.run.smithi167.stdout: "type": "rgw", 2023-10-05T19:28:25.755 INFO:teuthology.orchestra.run.smithi167.stdout: "mfa_ids": [] 2023-10-05T19:28:25.755 INFO:teuthology.orchestra.run.smithi167.stdout:} 2023-10-05T19:28:25.755 INFO:teuthology.orchestra.run.smithi167.stdout: 2023-10-05T19:28:25.756 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.746+0000 7fade5b46e40 1 -- 172.21.15.167:0/3427901904 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:114 7.1f 7:f95f44c2:::notify.0:head [watch unwatch cookie 93844329175776] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x5559d5e54990 con 0x7fadd4007030 2023-10-05T19:28:25.756 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.750+0000 7fade2e7a700 1 -- 172.21.15.167:0/3427901904 <== osd.2 v2:172.21.15.167:6816/30037 25 ==== osd_op_reply(114 notify.0 [watch unwatch cookie 93844329175776] v30'26 uv24 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7faddc045040 con 0x7fadd4007030 2023-10-05T19:28:25.756 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.750+0000 7fade5b46e40 20 remove_watcher() i=0 2023-10-05T19:28:25.756 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.750+0000 7fade5b46e40 2 removed watcher, disabling cache 2023-10-05T19:28:25.756 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.750+0000 7fade5b46e40 1 -- 172.21.15.167:0/3427901904 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:115 7.0 7:05bf5b68:::notify.1:head [watch unwatch cookie 93844329179008] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x5559d5e54990 con 0x5559d5e3f8d0 2023-10-05T19:28:25.757 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.750+0000 7fade3e7c700 1 -- 172.21.15.167:0/3427901904 <== osd.1 v2:172.21.15.167:6800/30035 43 ==== osd_op_reply(115 notify.1 [watch unwatch cookie 93844329179008] v30'26 uv24 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7fadcc050040 con 0x5559d5e3f8d0 2023-10-05T19:28:25.758 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.750+0000 7fade5b46e40 20 remove_watcher() i=1 2023-10-05T19:28:25.758 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.750+0000 7fade5b46e40 1 -- 172.21.15.167:0/3427901904 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:116 7.15 7:a93a5511:::notify.2:head [watch unwatch cookie 93844329182672] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x5559d5e54990 con 0x5559d5e746d0 2023-10-05T19:28:25.761 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.754+0000 7fade367b700 1 -- 172.21.15.167:0/3427901904 <== osd.3 v2:172.21.15.167:6808/30036 24 ==== osd_op_reply(116 notify.2 [watch unwatch cookie 93844329182672] v30'26 uv24 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7fadd4022020 con 0x5559d5e746d0 2023-10-05T19:28:25.761 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.754+0000 7fade5b46e40 20 remove_watcher() i=2 2023-10-05T19:28:25.761 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.754+0000 7fade5b46e40 1 -- 172.21.15.167:0/3427901904 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:117 7.e 7:7759931f:::notify.3:head [watch unwatch cookie 93844329187088] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x5559d5e54990 con 0x7fadd4007030 2023-10-05T19:28:25.763 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.758+0000 7fade2e7a700 1 -- 172.21.15.167:0/3427901904 <== osd.2 v2:172.21.15.167:6816/30037 26 ==== osd_op_reply(117 notify.3 [watch unwatch cookie 93844329187088] v30'26 uv24 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7faddc045040 con 0x7fadd4007030 2023-10-05T19:28:25.763 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.758+0000 7fade5b46e40 20 remove_watcher() i=3 2023-10-05T19:28:25.763 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.758+0000 7fade5b46e40 1 -- 172.21.15.167:0/3427901904 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:118 7.d 7:b4812045:::notify.4:head [watch unwatch cookie 93844329190320] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x5559d5e54990 con 0x5559d5e3f8d0 2023-10-05T19:28:25.765 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.758+0000 7fade3e7c700 1 -- 172.21.15.167:0/3427901904 <== osd.1 v2:172.21.15.167:6800/30035 44 ==== osd_op_reply(118 notify.4 [watch unwatch cookie 93844329190320] v30'26 uv24 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7fadcc050040 con 0x5559d5e3f8d0 2023-10-05T19:28:25.766 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.758+0000 7fade5b46e40 20 remove_watcher() i=4 2023-10-05T19:28:25.766 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.758+0000 7fade5b46e40 1 -- 172.21.15.167:0/3427901904 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:119 7.3 7:c609908c:::notify.5:head [watch unwatch cookie 93844329191984] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x5559d5e54990 con 0x7faddc011040 2023-10-05T19:28:25.769 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.762+0000 7fade3e7c700 1 -- 172.21.15.167:0/3427901904 <== osd.0 v2:172.21.15.118:6800/27690 31 ==== osd_op_reply(119 notify.5 [watch unwatch cookie 93844329191984] v30'26 uv24 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7fadcc050040 con 0x7faddc011040 2023-10-05T19:28:25.769 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.762+0000 7fade5b46e40 20 remove_watcher() i=5 2023-10-05T19:28:25.769 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.762+0000 7fade5b46e40 1 -- 172.21.15.167:0/3427901904 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:120 7.14 7:2b04a3e9:::notify.6:head [watch unwatch cookie 93844329195808] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x5559d5e54990 con 0x7faddc011040 2023-10-05T19:28:25.771 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.766+0000 7fade3e7c700 1 -- 172.21.15.167:0/3427901904 <== osd.0 v2:172.21.15.118:6800/27690 32 ==== osd_op_reply(120 notify.6 [watch unwatch cookie 93844329195808] v30'26 uv24 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7fadcc050040 con 0x7faddc011040 2023-10-05T19:28:25.771 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.766+0000 7fade5b46e40 20 remove_watcher() i=6 2023-10-05T19:28:25.771 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.766+0000 7fade5b46e40 1 -- 172.21.15.167:0/3427901904 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:121 7.9 7:93e5b521:::notify.7:head [watch unwatch cookie 93844329196912] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x5559d5e54990 con 0x7faddc011040 2023-10-05T19:28:25.774 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.770+0000 7fade3e7c700 1 -- 172.21.15.167:0/3427901904 <== osd.0 v2:172.21.15.118:6800/27690 33 ==== osd_op_reply(121 notify.7 [watch unwatch cookie 93844329196912] v30'26 uv24 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7fadcc050040 con 0x7faddc011040 2023-10-05T19:28:25.774 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.770+0000 7fade5b46e40 20 remove_watcher() i=7 2023-10-05T19:28:25.776 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.770+0000 7fadb3fef700 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2023-10-05T19:28:25.776 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.770+0000 7fade5b46e40 1 -- 172.21.15.167:0/3427901904 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:122 6.4 6:22d26bf9:::data_loggenerations_metadata:head [watch unwatch cookie 93844329199632] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x5559d5e54990 con 0x5559d5e3f8d0 2023-10-05T19:28:25.778 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.774+0000 7fade3e7c700 1 -- 172.21.15.167:0/3427901904 <== osd.1 v2:172.21.15.167:6800/30035 45 ==== osd_op_reply(122 data_loggenerations_metadata [watch unwatch cookie 93844329199632] v30'51 uv1 ondisk = 0) v8 ==== 172+0+0 (crc 0 0 0) 0x7fadcc050040 con 0x5559d5e3f8d0 2023-10-05T19:28:25.779 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.774+0000 7fade5b46e40 1 -- 172.21.15.167:0/3427901904 >> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] conn(0x7faddc011040 msgr2=0x7faddc014950 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-05T19:28:25.779 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.774+0000 7fade5b46e40 1 --2- 172.21.15.167:0/3427901904 >> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] conn(0x7faddc011040 0x7faddc014950 crc :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-05T19:28:25.779 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.774+0000 7fade5b46e40 1 -- 172.21.15.167:0/3427901904 >> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] conn(0x5559d5e3f8d0 msgr2=0x5559d5e70980 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-05T19:28:25.779 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.774+0000 7fade5b46e40 1 --2- 172.21.15.167:0/3427901904 >> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] conn(0x5559d5e3f8d0 0x5559d5e70980 crc :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-05T19:28:25.779 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.774+0000 7fade5b46e40 1 -- 172.21.15.167:0/3427901904 >> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] conn(0x7fadd4007030 msgr2=0x7fadd401ea00 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-05T19:28:25.779 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.774+0000 7fade5b46e40 1 --2- 172.21.15.167:0/3427901904 >> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] conn(0x7fadd4007030 0x7fadd401ea00 crc :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-05T19:28:25.779 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.774+0000 7fade5b46e40 1 -- 172.21.15.167:0/3427901904 >> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] conn(0x5559d5e746d0 msgr2=0x5559d5e76ab0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-05T19:28:25.779 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.774+0000 7fade5b46e40 1 --2- 172.21.15.167:0/3427901904 >> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] conn(0x5559d5e746d0 0x5559d5e76ab0 crc :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-05T19:28:25.781 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.774+0000 7fade5b46e40 1 -- 172.21.15.167:0/3427901904 >> [v2:172.21.15.167:6824/29889,v1:172.21.15.167:6825/29889] conn(0x7fadc8030c00 msgr2=0x7fadc80330b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-05T19:28:25.781 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.774+0000 7fade5b46e40 1 --2- 172.21.15.167:0/3427901904 >> [v2:172.21.15.167:6824/29889,v1:172.21.15.167:6825/29889] conn(0x7fadc8030c00 0x7fadc80330b0 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7faddc0044f0 tx=0x7faddc0067c0 comp rx=0 tx=0).stop 2023-10-05T19:28:25.781 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.774+0000 7fade5b46e40 1 -- 172.21.15.167:0/3427901904 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5559d5e3efc0 msgr2=0x5559d5e64d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-05T19:28:25.781 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.774+0000 7fade3e7c700 1 -- 172.21.15.167:0/3427901904 reap_dead start 2023-10-05T19:28:25.781 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.774+0000 7fade5b46e40 1 --2- 172.21.15.167:0/3427901904 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5559d5e3efc0 0x5559d5e64d90 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7fadd4008fd0 tx=0x7fadd4003050 comp rx=0 tx=0).stop 2023-10-05T19:28:25.781 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.774+0000 7fade5b46e40 1 -- 172.21.15.167:0/3427901904 shutdown_connections 2023-10-05T19:28:25.781 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.774+0000 7fade5b46e40 1 --2- 172.21.15.167:0/3427901904 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5559d5e3efc0 0x5559d5e64d90 unknown :-1 s=CLOSED pgs=185 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-05T19:28:25.781 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.774+0000 7fade5b46e40 1 -- 172.21.15.167:0/3427901904 >> 172.21.15.167:0/3427901904 conn(0x5559d5db2810 msgr2=0x5559d5e58770 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-05T19:28:25.781 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.774+0000 7fade5b46e40 1 -- 172.21.15.167:0/3427901904 shutdown_connections 2023-10-05T19:28:25.781 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.774+0000 7fade5b46e40 1 -- 172.21.15.167:0/3427901904 wait complete. 2023-10-05T19:28:25.781 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.774+0000 7fade5b46e40 1 -- 172.21.15.167:0/231899477 >> [v2:172.21.15.167:6824/29889,v1:172.21.15.167:6825/29889] conn(0x7fad28030c50 msgr2=0x7fad28033100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-05T19:28:25.782 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.774+0000 7fade5b46e40 1 --2- 172.21.15.167:0/231899477 >> [v2:172.21.15.167:6824/29889,v1:172.21.15.167:6825/29889] conn(0x7fad28030c50 0x7fad28033100 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x5559d5e7e0d0 tx=0x7fadcc000f90 comp rx=0 tx=0).stop 2023-10-05T19:28:25.782 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.774+0000 7fade5b46e40 1 -- 172.21.15.167:0/231899477 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5559d5e8d1d0 msgr2=0x5559d5e92d60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-05T19:28:25.782 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.774+0000 7fade5b46e40 1 --2- 172.21.15.167:0/231899477 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5559d5e8d1d0 0x5559d5e92d60 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7faddc041040 tx=0x7faddc03c000 comp rx=0 tx=0).stop 2023-10-05T19:28:25.782 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.774+0000 7fade5b46e40 1 -- 172.21.15.167:0/231899477 shutdown_connections 2023-10-05T19:28:25.782 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.774+0000 7fade5b46e40 1 --2- 172.21.15.167:0/231899477 >> [v2:172.21.15.167:6824/29889,v1:172.21.15.167:6825/29889] conn(0x7fad28030c50 0x7fad28033100 unknown :-1 s=CLOSED pgs=69 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-05T19:28:25.782 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.774+0000 7fade5b46e40 1 --2- 172.21.15.167:0/231899477 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5559d5e8d1d0 0x5559d5e92d60 unknown :-1 s=CLOSED pgs=187 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-05T19:28:25.782 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.774+0000 7fade5b46e40 1 -- 172.21.15.167:0/231899477 >> 172.21.15.167:0/231899477 conn(0x5559d5e88590 msgr2=0x5559d5e86630 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-05T19:28:25.782 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.774+0000 7fade5b46e40 1 -- 172.21.15.167:0/231899477 shutdown_connections 2023-10-05T19:28:25.782 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.774+0000 7fade5b46e40 1 -- 172.21.15.167:0/231899477 wait complete. 2023-10-05T19:28:25.787 DEBUG:tasks.util.rgw: json result: {'user_id': 'foo', 'display_name': 'Foo', 'email': 'foo@foo.com', 'suspended': 0, 'max_buckets': 4, 'subusers': [], 'keys': [{'user': 'foo', 'access_key': '9te6NH5mcdcq0Tc5i8i1', 'secret_key': 'Ny4IOauQoL18Gp2zM7lC1vLmoawgqcYP/YGcWfXu'}], 'swift_keys': [], 'caps': [], 'op_mask': 'read, write, delete', 'default_placement': '', 'default_storage_class': '', 'placement_tags': [], 'bucket_quota': {'enabled': False, 'check_on_raw': False, 'max_size': -1, 'max_size_kb': 0, 'max_objects': -1}, 'user_quota': {'enabled': False, 'check_on_raw': False, 'max_size': -1, 'max_size_kb': 0, 'max_objects': -1}, 'temp_url_keys': [], 'type': 'rgw', 'mfa_ids': []} 2023-10-05T19:28:25.787 INFO:tasks.util.rgw:rgwadmin: client.0 : ['user', 'create', '--uid', 'fud', '--display-name', 'Fud', '--email', 'foo@foo.com'] 2023-10-05T19:28:25.787 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'ceph', 'user', 'create', '--uid', 'fud', '--display-name', 'Fud', '--email', 'foo@foo.com'] 2023-10-05T19:28:25.787 DEBUG:teuthology.orchestra.run.smithi167:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster ceph user create --uid fud --display-name Fud --email foo@foo.com 2023-10-05T19:28:25.832 INFO:teuthology.orchestra.run.smithi167.stderr:ignoring --setuser ceph since I am not root 2023-10-05T19:28:25.832 INFO:teuthology.orchestra.run.smithi167.stderr:ignoring --setgroup ceph since I am not root 2023-10-05T19:28:25.832 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.826+0000 7f78020b4e40 1 Processor -- start 2023-10-05T19:28:25.833 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.826+0000 7f78020b4e40 1 -- start start 2023-10-05T19:28:25.833 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.826+0000 7f78020b4e40 1 --2- >> v2:172.21.15.118:3300/0 conn(0x5563027442a0 0x556302746680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-05T19:28:25.833 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.826+0000 7f78020b4e40 1 -- --> v1:172.21.15.118:6789/0 -- auth(proto 0 26 bytes epoch 0) v1 -- 0x55630268dbd0 con 0x55630273fcc0 2023-10-05T19:28:25.834 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.826+0000 7f78020b4e40 1 -- --> v2:172.21.15.118:3300/0 -- mon_getmap magic: 0 v1 -- 0x55630264aee0 con 0x5563027442a0 2023-10-05T19:28:25.834 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.826+0000 7f77ff3e8700 1 --2- >> v2:172.21.15.118:3300/0 conn(0x5563027442a0 0x556302746680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-05T19:28:25.834 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.826+0000 7f77ff3e8700 1 --2- >> v2:172.21.15.118:3300/0 conn(0x5563027442a0 0x556302746680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.118:3300/0 says I am v2:172.21.15.167:57200/0 (socket says 172.21.15.167:57200) 2023-10-05T19:28:25.834 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.826+0000 7f77ffbe9700 1 --1- >> v1:172.21.15.118:6789/0 conn(0x55630273fcc0 0x556302740090 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.118:6789/0 says I am v1:172.21.15.167:56364/0 (socket says 172.21.15.167:56364) 2023-10-05T19:28:25.834 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.826+0000 7f77ff3e8700 1 -- 172.21.15.167:0/2445868235 learned_addr learned my addr 172.21.15.167:0/2445868235 (peer_addr_for_me v2:172.21.15.167:0/0) 2023-10-05T19:28:25.835 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.830+0000 7f77ff3e8700 1 -- 172.21.15.167:0/2445868235 >> v1:172.21.15.118:6789/0 conn(0x55630273fcc0 legacy=0x556302740090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-05T19:28:25.835 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.830+0000 7f77ff3e8700 1 -- 172.21.15.167:0/2445868235 --> v2:172.21.15.118:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5563027408d0 con 0x5563027442a0 2023-10-05T19:28:25.835 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.830+0000 7f77ff3e8700 1 --2- 172.21.15.167:0/2445868235 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5563027442a0 0x556302746680 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7f77e800a650 tx=0x7f77e8005b10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6ffd89c71ccfb92 server_cookie=0 in_seq=0 out_seq=0 2023-10-05T19:28:25.835 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.830+0000 7f77febe7700 1 -- 172.21.15.167:0/2445868235 <== mon.0 v2:172.21.15.118:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f77e8008760 con 0x5563027442a0 2023-10-05T19:28:25.835 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.830+0000 7f77febe7700 1 -- 172.21.15.167:0/2445868235 <== mon.0 v2:172.21.15.118:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f77e80088c0 con 0x5563027442a0 2023-10-05T19:28:25.835 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.830+0000 7f77febe7700 1 -- 172.21.15.167:0/2445868235 <== mon.0 v2:172.21.15.118:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f77e8008bb0 con 0x5563027442a0 2023-10-05T19:28:25.837 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.830+0000 7f78020b4e40 1 -- 172.21.15.167:0/2445868235 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5563027442a0 msgr2=0x556302746680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-05T19:28:25.837 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.830+0000 7f78020b4e40 1 --2- 172.21.15.167:0/2445868235 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5563027442a0 0x556302746680 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7f77e800a650 tx=0x7f77e8005b10 comp rx=0 tx=0).stop 2023-10-05T19:28:25.837 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.830+0000 7f78020b4e40 1 -- 172.21.15.167:0/2445868235 shutdown_connections 2023-10-05T19:28:25.837 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.830+0000 7f78020b4e40 1 --2- 172.21.15.167:0/2445868235 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5563027442a0 0x556302746680 unknown :-1 s=CLOSED pgs=189 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-05T19:28:25.837 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.830+0000 7f78020b4e40 1 -- 172.21.15.167:0/2445868235 >> 172.21.15.167:0/2445868235 conn(0x55630273b6e0 msgr2=0x55630273daf0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-05T19:28:25.837 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.830+0000 7f78020b4e40 1 -- 172.21.15.167:0/2445868235 shutdown_connections 2023-10-05T19:28:25.837 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.830+0000 7f78020b4e40 1 -- 172.21.15.167:0/2445868235 wait complete. 2023-10-05T19:28:25.838 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.830+0000 7f78020b4e40 1 Processor -- start 2023-10-05T19:28:25.838 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.830+0000 7f78020b4e40 1 -- start start 2023-10-05T19:28:25.838 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.830+0000 7f78020b4e40 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55630273fcc0 0x55630275ff10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-05T19:28:25.838 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.830+0000 7f78020b4e40 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x55630260ee30 con 0x55630273fcc0 2023-10-05T19:28:25.839 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.830+0000 7f77ffbe9700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55630273fcc0 0x55630275ff10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-05T19:28:25.839 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.830+0000 7f77ffbe9700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55630273fcc0 0x55630275ff10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.118:3300/0 says I am v2:172.21.15.167:57214/0 (socket says 172.21.15.167:57214) 2023-10-05T19:28:25.839 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.830+0000 7f77ffbe9700 1 -- 172.21.15.167:0/3096915601 learned_addr learned my addr 172.21.15.167:0/3096915601 (peer_addr_for_me v2:172.21.15.167:0/0) 2023-10-05T19:28:25.839 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.830+0000 7f77ffbe9700 1 -- 172.21.15.167:0/3096915601 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f77e8005db0 con 0x55630273fcc0 2023-10-05T19:28:25.840 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.834+0000 7f77ffbe9700 1 --2- 172.21.15.167:0/3096915601 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55630273fcc0 0x55630275ff10 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7f77f8001cb0 tx=0x7f77f8001860 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7852f527bb70193b server_cookie=0 in_seq=0 out_seq=0 2023-10-05T19:28:25.840 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.834+0000 7f77fdbe5700 1 -- 172.21.15.167:0/3096915601 <== mon.0 v2:172.21.15.118:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f77f8014070 con 0x55630273fcc0 2023-10-05T19:28:25.840 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.834+0000 7f77fdbe5700 1 -- 172.21.15.167:0/3096915601 <== mon.0 v2:172.21.15.118:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f77f800bd40 con 0x55630273fcc0 2023-10-05T19:28:25.840 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.834+0000 7f77fdbe5700 1 -- 172.21.15.167:0/3096915601 <== mon.0 v2:172.21.15.118:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f77f800f450 con 0x55630273fcc0 2023-10-05T19:28:25.841 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.834+0000 7f78020b4e40 1 -- 172.21.15.167:0/3096915601 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55630273fcc0 msgr2=0x55630275ff10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-05T19:28:25.841 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.834+0000 7f78020b4e40 1 --2- 172.21.15.167:0/3096915601 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55630273fcc0 0x55630275ff10 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7f77f8001cb0 tx=0x7f77f8001860 comp rx=0 tx=0).stop 2023-10-05T19:28:25.841 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.834+0000 7f78020b4e40 1 -- 172.21.15.167:0/3096915601 shutdown_connections 2023-10-05T19:28:25.841 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.834+0000 7f78020b4e40 1 --2- 172.21.15.167:0/3096915601 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55630273fcc0 0x55630275ff10 unknown :-1 s=CLOSED pgs=190 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-05T19:28:25.842 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.834+0000 7f78020b4e40 1 -- 172.21.15.167:0/3096915601 >> 172.21.15.167:0/3096915601 conn(0x55630273b6e0 msgr2=0x55630275ba50 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-05T19:28:25.842 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.834+0000 7f78020b4e40 1 -- 172.21.15.167:0/3096915601 shutdown_connections 2023-10-05T19:28:25.842 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.834+0000 7f78020b4e40 1 -- 172.21.15.167:0/3096915601 wait complete. 2023-10-05T19:28:25.842 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.834+0000 7f78020b4e40 1 Processor -- start 2023-10-05T19:28:25.842 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.834+0000 7f78020b4e40 1 -- start start 2023-10-05T19:28:25.842 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.834+0000 7f78020b4e40 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55630273fcc0 0x556302766ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-05T19:28:25.842 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.834+0000 7f78020b4e40 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x5563025e1ed0 con 0x55630273fcc0 2023-10-05T19:28:25.844 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.834+0000 7f77ffbe9700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55630273fcc0 0x556302766ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-05T19:28:25.844 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.834+0000 7f77ffbe9700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55630273fcc0 0x556302766ce0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.118:3300/0 says I am v2:172.21.15.167:57216/0 (socket says 172.21.15.167:57216) 2023-10-05T19:28:25.844 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.834+0000 7f77ffbe9700 1 -- 172.21.15.167:0/2969922888 learned_addr learned my addr 172.21.15.167:0/2969922888 (peer_addr_for_me v2:172.21.15.167:0/0) 2023-10-05T19:28:25.844 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.838+0000 7f77ffbe9700 1 -- 172.21.15.167:0/2969922888 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55630268dbd0 con 0x55630273fcc0 2023-10-05T19:28:25.844 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.838+0000 7f77ffbe9700 1 --2- 172.21.15.167:0/2969922888 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55630273fcc0 0x556302766ce0 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7f77f8000c00 tx=0x7f77f8016000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-05T19:28:25.844 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.838+0000 7f77f7fff700 1 -- 172.21.15.167:0/2969922888 <== mon.0 v2:172.21.15.118:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f77f800beb0 con 0x55630273fcc0 2023-10-05T19:28:25.844 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.838+0000 7f77f7fff700 1 -- 172.21.15.167:0/2969922888 <== mon.0 v2:172.21.15.118:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f77f800f450 con 0x55630273fcc0 2023-10-05T19:28:25.844 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.838+0000 7f78020b4e40 1 -- 172.21.15.167:0/2969922888 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x556302767220 con 0x55630273fcc0 2023-10-05T19:28:25.844 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.838+0000 7f77f7fff700 1 -- 172.21.15.167:0/2969922888 <== mon.0 v2:172.21.15.118:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f77f801cc80 con 0x55630273fcc0 2023-10-05T19:28:25.845 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.838+0000 7f78020b4e40 1 -- 172.21.15.167:0/2969922888 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x556302767660 con 0x55630273fcc0 2023-10-05T19:28:25.845 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.838+0000 7f77f7fff700 1 -- 172.21.15.167:0/2969922888 <== mon.0 v2:172.21.15.118:3300/0 4 ==== mgrmap(e 5) v1 ==== 40329+0+0 (secure 0 0 0) 0x7f77f8023020 con 0x55630273fcc0 2023-10-05T19:28:25.846 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.838+0000 7f78020b4e40 20 rados->read ofs=0 len=0 2023-10-05T19:28:25.846 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.838+0000 7f77f7fff700 1 --2- 172.21.15.167:0/2969922888 >> [v2:172.21.15.167:6824/29889,v1:172.21.15.167:6825/29889] conn(0x7f77e4030c50 0x7f77e4033100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-05T19:28:25.846 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.838+0000 7f77f7fff700 1 -- 172.21.15.167:0/2969922888 <== mon.0 v2:172.21.15.118:3300/0 5 ==== osd_map(30..30 src has 1..30) v4 ==== 6057+0+0 (secure 0 0 0) 0x7f77f8008030 con 0x55630273fcc0 2023-10-05T19:28:25.846 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.838+0000 7f77ff3e8700 1 --2- 172.21.15.167:0/2969922888 >> [v2:172.21.15.167:6824/29889,v1:172.21.15.167:6825/29889] conn(0x7f77e4030c50 0x7f77e4033100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-05T19:28:25.847 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.842+0000 7f78020b4e40 1 --2- 172.21.15.167:0/2969922888 >> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] conn(0x5563027442a0 0x5563027727d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-05T19:28:25.847 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.842+0000 7f78020b4e40 1 -- 172.21.15.167:0/2969922888 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:1 5.12 5:49953fa1:::default.realm:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x556302772e80 con 0x5563027442a0 2023-10-05T19:28:25.847 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.842+0000 7f78003ea700 1 --2- 172.21.15.167:0/2969922888 >> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] conn(0x5563027442a0 0x5563027727d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-05T19:28:25.847 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.842+0000 7f77ff3e8700 1 --2- 172.21.15.167:0/2969922888 >> [v2:172.21.15.167:6824/29889,v1:172.21.15.167:6825/29889] conn(0x7f77e4030c50 0x7f77e4033100 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f77e80030a0 tx=0x7f77e8015000 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-05T19:28:25.847 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.842+0000 7f78003ea700 1 --2- 172.21.15.167:0/2969922888 >> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] conn(0x5563027442a0 0x5563027727d0 crc :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-05T19:28:25.848 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.842+0000 7f78003ea700 1 -- 172.21.15.167:0/2969922888 <== osd.1 v2:172.21.15.167:6800/30035 1 ==== osd_op_reply(1 default.realm [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 ==== 157+0+0 (crc 0 0 0) 0x7f77f000a5a0 con 0x5563027442a0 2023-10-05T19:28:25.849 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.842+0000 7f78020b4e40 20 rados_obj.operate() r=-2 bl.length=0 2023-10-05T19:28:25.849 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.842+0000 7f78020b4e40 20 realm 2023-10-05T19:28:25.849 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.842+0000 7f78020b4e40 20 rados->read ofs=0 len=0 2023-10-05T19:28:25.849 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.842+0000 7f78020b4e40 1 -- 172.21.15.167:0/2969922888 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:2 5.12 5:49953fa1:::default.realm:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x55630276fdf0 con 0x5563027442a0 2023-10-05T19:28:25.849 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.842+0000 7f78003ea700 1 -- 172.21.15.167:0/2969922888 <== osd.1 v2:172.21.15.167:6800/30035 2 ==== osd_op_reply(2 default.realm [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 ==== 157+0+0 (crc 0 0 0) 0x7f77f000a5a0 con 0x5563027442a0 2023-10-05T19:28:25.849 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.842+0000 7f78020b4e40 20 rados_obj.operate() r=-2 bl.length=0 2023-10-05T19:28:25.849 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.842+0000 7f78020b4e40 4 RGWPeriod::init failed to init realm id : (2) No such file or directory 2023-10-05T19:28:25.849 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.842+0000 7f78020b4e40 20 rados->read ofs=0 len=0 2023-10-05T19:28:25.849 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.842+0000 7f78020b4e40 1 -- 172.21.15.167:0/2969922888 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:3 5.12 5:49953fa1:::default.realm:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x556302773960 con 0x5563027442a0 2023-10-05T19:28:25.849 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.842+0000 7f78003ea700 1 -- 172.21.15.167:0/2969922888 <== osd.1 v2:172.21.15.167:6800/30035 3 ==== osd_op_reply(3 default.realm [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 ==== 157+0+0 (crc 0 0 0) 0x7f77f000a5a0 con 0x5563027442a0 2023-10-05T19:28:25.850 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.842+0000 7f78020b4e40 20 rados_obj.operate() r=-2 bl.length=0 2023-10-05T19:28:25.850 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.842+0000 7f78020b4e40 20 rados->read ofs=0 len=0 2023-10-05T19:28:25.850 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.842+0000 7f78020b4e40 1 -- 172.21.15.167:0/2969922888 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:4 5.f 5:f43b6ece:::zone_names.default:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x556302773540 con 0x5563027442a0 2023-10-05T19:28:25.851 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.842+0000 7f78003ea700 1 -- 172.21.15.167:0/2969922888 <== osd.1 v2:172.21.15.167:6800/30035 4 ==== osd_op_reply(4 zone_names.default [read 0~46 out=46b] v0'0 uv1 ondisk = 0) v8 ==== 162+0+46 (crc 0 0 0) 0x7f77f000a5a0 con 0x5563027442a0 2023-10-05T19:28:25.851 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.842+0000 7f78020b4e40 20 rados_obj.operate() r=0 bl.length=46 2023-10-05T19:28:25.851 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.842+0000 7f78020b4e40 20 rados->read ofs=0 len=0 2023-10-05T19:28:25.851 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.842+0000 7f78020b4e40 1 -- 172.21.15.167:0/2969922888 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:5 5.1d 5:b8374eb4:::zone_info.9a9df965-6612-45d4-a45f-6e3c889bacc2:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x556302774500 con 0x5563027442a0 2023-10-05T19:28:25.852 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.846+0000 7f78003ea700 1 -- 172.21.15.167:0/2969922888 <== osd.1 v2:172.21.15.167:6800/30035 5 ==== osd_op_reply(5 zone_info.9a9df965-6612-45d4-a45f-6e3c889bacc2 [read 0~874 out=874b] v0'0 uv2 ondisk = 0) v8 ==== 190+0+874 (crc 0 0 0) 0x7f77f000a5a0 con 0x5563027442a0 2023-10-05T19:28:25.852 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.846+0000 7f78020b4e40 20 rados_obj.operate() r=0 bl.length=874 2023-10-05T19:28:25.852 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.846+0000 7f78020b4e40 20 searching for the correct realm 2023-10-05T19:28:25.852 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.846+0000 7f78020b4e40 1 --2- 172.21.15.167:0/2969922888 >> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] conn(0x556302776440 0x556302778850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-05T19:28:25.852 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.846+0000 7f78020b4e40 1 -- 172.21.15.167:0/2969922888 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:6 5.0 5:00000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x556302778e60 con 0x556302776440 2023-10-05T19:28:25.852 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.846+0000 7f77ffbe9700 1 --2- 172.21.15.167:0/2969922888 >> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] conn(0x556302776440 0x556302778850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-05T19:28:25.852 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.846+0000 7f77ffbe9700 1 --2- 172.21.15.167:0/2969922888 >> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] conn(0x556302776440 0x556302778850 crc :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-05T19:28:25.853 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.846+0000 7f77ffbe9700 1 -- 172.21.15.167:0/2969922888 <== osd.3 v2:172.21.15.167:6808/30036 1 ==== osd_op_reply(6 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f77f8016070 con 0x556302776440 2023-10-05T19:28:25.853 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.846+0000 7f77ffbe9700 1 --2- 172.21.15.167:0/2969922888 >> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] conn(0x7f77f801a070 0x7f77f8042220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-05T19:28:25.853 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.846+0000 7f77ffbe9700 1 -- 172.21.15.167:0/2969922888 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:7 5.10 5:08000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x556302778e60 con 0x7f77f801a070 2023-10-05T19:28:25.853 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.846+0000 7f77ff3e8700 1 --2- 172.21.15.167:0/2969922888 >> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] conn(0x7f77f801a070 0x7f77f8042220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-05T19:28:25.853 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.846+0000 7f77ff3e8700 1 --2- 172.21.15.167:0/2969922888 >> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] conn(0x7f77f801a070 0x7f77f8042220 crc :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-05T19:28:25.854 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.850+0000 7f77ff3e8700 1 -- 172.21.15.167:0/2969922888 <== osd.2 v2:172.21.15.167:6816/30037 1 ==== osd_op_reply(7 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f77e8012450 con 0x7f77f801a070 2023-10-05T19:28:25.854 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.850+0000 7f77ff3e8700 1 --2- 172.21.15.167:0/2969922888 >> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] conn(0x7f77e8007030 0x7f77e801c920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-05T19:28:25.854 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.850+0000 7f77ff3e8700 1 -- 172.21.15.167:0/2969922888 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:8 5.8 5:10000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x556302778e60 con 0x7f77e8007030 2023-10-05T19:28:25.855 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.850+0000 7f78003ea700 1 --2- 172.21.15.167:0/2969922888 >> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] conn(0x7f77e8007030 0x7f77e801c920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-05T19:28:25.855 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.850+0000 7f78003ea700 1 --2- 172.21.15.167:0/2969922888 >> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] conn(0x7f77e8007030 0x7f77e801c920 crc :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-05T19:28:25.856 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.850+0000 7f78003ea700 1 -- 172.21.15.167:0/2969922888 <== osd.0 v2:172.21.15.118:6800/27690 1 ==== osd_op_reply(8 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f77f000a5a0 con 0x7f77e8007030 2023-10-05T19:28:25.856 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.850+0000 7f78003ea700 1 -- 172.21.15.167:0/2969922888 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:9 5.18 5:18000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x556302778e60 con 0x5563027442a0 2023-10-05T19:28:25.857 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.850+0000 7f78003ea700 1 -- 172.21.15.167:0/2969922888 <== osd.1 v2:172.21.15.167:6800/30035 6 ==== osd_op_reply(9 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f77f000a5a0 con 0x5563027442a0 2023-10-05T19:28:25.857 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.850+0000 7f78003ea700 1 -- 172.21.15.167:0/2969922888 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:10 5.4 5:20000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x556302778e60 con 0x5563027442a0 2023-10-05T19:28:25.857 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.850+0000 7f78003ea700 1 -- 172.21.15.167:0/2969922888 <== osd.1 v2:172.21.15.167:6800/30035 7 ==== osd_op_reply(10 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f77f000a5a0 con 0x5563027442a0 2023-10-05T19:28:25.858 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.850+0000 7f78003ea700 1 -- 172.21.15.167:0/2969922888 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:11 5.14 5:28000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x556302778e60 con 0x7f77e8007030 2023-10-05T19:28:25.858 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.854+0000 7f78003ea700 1 -- 172.21.15.167:0/2969922888 <== osd.0 v2:172.21.15.118:6800/27690 2 ==== osd_op_reply(11 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f77f000a5a0 con 0x7f77e8007030 2023-10-05T19:28:25.859 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.854+0000 7f78003ea700 1 -- 172.21.15.167:0/2969922888 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:12 5.c 5:30000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x556302778e60 con 0x7f77e8007030 2023-10-05T19:28:25.859 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.854+0000 7f78003ea700 1 -- 172.21.15.167:0/2969922888 <== osd.0 v2:172.21.15.118:6800/27690 3 ==== osd_op_reply(12 [pgnls start_epoch 30 out=112b] v23'1 uv1 ondisk = 1) v8 ==== 144+0+112 (crc 0 0 0) 0x7f77f000a5a0 con 0x7f77e8007030 2023-10-05T19:28:25.860 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.854+0000 7f78003ea700 1 -- 172.21.15.167:0/2969922888 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:13 5.1c 5:38000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x556302778e60 con 0x7f77f801a070 2023-10-05T19:28:25.860 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.854+0000 7f77ff3e8700 1 -- 172.21.15.167:0/2969922888 <== osd.2 v2:172.21.15.167:6816/30037 2 ==== osd_op_reply(13 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f77e8012450 con 0x7f77f801a070 2023-10-05T19:28:25.860 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.854+0000 7f77ff3e8700 1 -- 172.21.15.167:0/2969922888 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:14 5.2 5:40000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x556302778e60 con 0x556302776440 2023-10-05T19:28:25.861 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.854+0000 7f77ffbe9700 1 -- 172.21.15.167:0/2969922888 <== osd.3 v2:172.21.15.167:6808/30036 2 ==== osd_op_reply(14 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f77f8016070 con 0x556302776440 2023-10-05T19:28:25.861 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.854+0000 7f77ffbe9700 1 -- 172.21.15.167:0/2969922888 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:15 5.12 5:48000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x556302778e60 con 0x5563027442a0 2023-10-05T19:28:25.862 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.858+0000 7f78003ea700 1 -- 172.21.15.167:0/2969922888 <== osd.1 v2:172.21.15.167:6800/30035 8 ==== osd_op_reply(15 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f77f000a5a0 con 0x5563027442a0 2023-10-05T19:28:25.862 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.858+0000 7f78003ea700 1 -- 172.21.15.167:0/2969922888 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:16 5.a 5:50000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x556302778e60 con 0x7f77e8007030 2023-10-05T19:28:25.863 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.858+0000 7f78003ea700 1 -- 172.21.15.167:0/2969922888 <== osd.0 v2:172.21.15.118:6800/27690 4 ==== osd_op_reply(16 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f77f000a5a0 con 0x7f77e8007030 2023-10-05T19:28:25.863 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.858+0000 7f78003ea700 1 -- 172.21.15.167:0/2969922888 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:17 5.1a 5:58000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x556302778e60 con 0x5563027442a0 2023-10-05T19:28:25.864 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.858+0000 7f78003ea700 1 -- 172.21.15.167:0/2969922888 <== osd.1 v2:172.21.15.167:6800/30035 9 ==== osd_op_reply(17 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f77f000a5a0 con 0x5563027442a0 2023-10-05T19:28:25.864 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.858+0000 7f78003ea700 1 -- 172.21.15.167:0/2969922888 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:18 5.6 5:60000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x556302778e60 con 0x7f77f801a070 2023-10-05T19:28:25.864 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.858+0000 7f77ff3e8700 1 -- 172.21.15.167:0/2969922888 <== osd.2 v2:172.21.15.167:6816/30037 3 ==== osd_op_reply(18 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f77e8012450 con 0x7f77f801a070 2023-10-05T19:28:25.865 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.858+0000 7f77ff3e8700 1 -- 172.21.15.167:0/2969922888 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:19 5.16 5:68000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x556302778e60 con 0x556302776440 2023-10-05T19:28:25.865 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.858+0000 7f77ffbe9700 1 -- 172.21.15.167:0/2969922888 <== osd.3 v2:172.21.15.167:6808/30036 3 ==== osd_op_reply(19 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f77f8016070 con 0x556302776440 2023-10-05T19:28:25.865 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.858+0000 7f77ffbe9700 1 -- 172.21.15.167:0/2969922888 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:20 5.e 5:70000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x556302778e60 con 0x556302776440 2023-10-05T19:28:25.866 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.862+0000 7f77ffbe9700 1 -- 172.21.15.167:0/2969922888 <== osd.3 v2:172.21.15.167:6808/30036 4 ==== osd_op_reply(20 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f77f8016070 con 0x556302776440 2023-10-05T19:28:25.866 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.862+0000 7f77ffbe9700 1 -- 172.21.15.167:0/2969922888 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:21 5.1e 5:78000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x556302778e60 con 0x7f77e8007030 2023-10-05T19:28:25.867 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.862+0000 7f78003ea700 1 -- 172.21.15.167:0/2969922888 <== osd.0 v2:172.21.15.118:6800/27690 5 ==== osd_op_reply(21 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f77f000a5a0 con 0x7f77e8007030 2023-10-05T19:28:25.867 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.862+0000 7f78003ea700 1 -- 172.21.15.167:0/2969922888 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:22 5.1 5:80000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x556302778e60 con 0x556302776440 2023-10-05T19:28:25.868 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.862+0000 7f77ffbe9700 1 -- 172.21.15.167:0/2969922888 <== osd.3 v2:172.21.15.167:6808/30036 5 ==== osd_op_reply(22 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f77f8039040 con 0x556302776440 2023-10-05T19:28:25.868 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.862+0000 7f77ffbe9700 1 -- 172.21.15.167:0/2969922888 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:23 5.11 5:88000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x556302778e60 con 0x556302776440 2023-10-05T19:28:25.869 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.862+0000 7f77ffbe9700 1 -- 172.21.15.167:0/2969922888 <== osd.3 v2:172.21.15.167:6808/30036 6 ==== osd_op_reply(23 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f77f8039040 con 0x556302776440 2023-10-05T19:28:25.869 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.862+0000 7f77ffbe9700 1 -- 172.21.15.167:0/2969922888 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:24 5.9 5:90000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x556302778e60 con 0x5563027442a0 2023-10-05T19:28:25.869 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.862+0000 7f78003ea700 1 -- 172.21.15.167:0/2969922888 <== osd.1 v2:172.21.15.167:6800/30035 10 ==== osd_op_reply(24 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f77f000a5a0 con 0x5563027442a0 2023-10-05T19:28:25.870 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.862+0000 7f78003ea700 1 -- 172.21.15.167:0/2969922888 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:25 5.19 5:98000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x556302778e60 con 0x5563027442a0 2023-10-05T19:28:25.870 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.866+0000 7f78003ea700 1 -- 172.21.15.167:0/2969922888 <== osd.1 v2:172.21.15.167:6800/30035 11 ==== osd_op_reply(25 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f77f000a5a0 con 0x5563027442a0 2023-10-05T19:28:25.870 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.866+0000 7f78003ea700 1 -- 172.21.15.167:0/2969922888 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:26 5.5 5:a0000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x556302778e60 con 0x7f77e8007030 2023-10-05T19:28:25.871 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.866+0000 7f78003ea700 1 -- 172.21.15.167:0/2969922888 <== osd.0 v2:172.21.15.118:6800/27690 6 ==== osd_op_reply(26 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f77f000a5a0 con 0x7f77e8007030 2023-10-05T19:28:25.872 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.866+0000 7f78003ea700 1 -- 172.21.15.167:0/2969922888 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:27 5.15 5:a8000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x556302778e60 con 0x556302776440 2023-10-05T19:28:25.872 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.866+0000 7f77ffbe9700 1 -- 172.21.15.167:0/2969922888 <== osd.3 v2:172.21.15.167:6808/30036 7 ==== osd_op_reply(27 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f77f8039040 con 0x556302776440 2023-10-05T19:28:25.872 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.866+0000 7f77ffbe9700 1 -- 172.21.15.167:0/2969922888 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:28 5.d 5:b0000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x556302778e60 con 0x7f77f801a070 2023-10-05T19:28:25.873 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.866+0000 7f77ff3e8700 1 -- 172.21.15.167:0/2969922888 <== osd.2 v2:172.21.15.167:6816/30037 4 ==== osd_op_reply(28 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f77e8012450 con 0x7f77f801a070 2023-10-05T19:28:25.873 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.866+0000 7f77ff3e8700 1 -- 172.21.15.167:0/2969922888 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:29 5.1d 5:b8000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x556302778e60 con 0x5563027442a0 2023-10-05T19:28:25.874 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.870+0000 7f78003ea700 1 -- 172.21.15.167:0/2969922888 <== osd.1 v2:172.21.15.167:6800/30035 12 ==== osd_op_reply(29 [pgnls start_epoch 30 out=107b] v30'2 uv2 ondisk = 1) v8 ==== 144+0+107 (crc 0 0 0) 0x7f77f000a5a0 con 0x5563027442a0 2023-10-05T19:28:25.874 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.870+0000 7f78003ea700 1 -- 172.21.15.167:0/2969922888 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:30 5.3 5:c0000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x556302778e60 con 0x7f77e8007030 2023-10-05T19:28:25.875 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.870+0000 7f78003ea700 1 -- 172.21.15.167:0/2969922888 <== osd.0 v2:172.21.15.118:6800/27690 7 ==== osd_op_reply(30 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f77f000a5a0 con 0x7f77e8007030 2023-10-05T19:28:25.875 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.870+0000 7f78003ea700 1 -- 172.21.15.167:0/2969922888 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:31 5.13 5:c8000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x556302778e60 con 0x556302776440 2023-10-05T19:28:25.876 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.870+0000 7f77ffbe9700 1 -- 172.21.15.167:0/2969922888 <== osd.3 v2:172.21.15.167:6808/30036 8 ==== osd_op_reply(31 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f77f8039040 con 0x556302776440 2023-10-05T19:28:25.876 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.870+0000 7f77ffbe9700 1 -- 172.21.15.167:0/2969922888 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:32 5.b 5:d0000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x556302778e60 con 0x7f77e8007030 2023-10-05T19:28:25.877 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.870+0000 7f78003ea700 1 -- 172.21.15.167:0/2969922888 <== osd.0 v2:172.21.15.118:6800/27690 8 ==== osd_op_reply(32 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f77f000a5a0 con 0x7f77e8007030 2023-10-05T19:28:25.877 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.870+0000 7f78003ea700 1 -- 172.21.15.167:0/2969922888 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:33 5.1b 5:d8000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x556302778e60 con 0x7f77f801a070 2023-10-05T19:28:25.878 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.874+0000 7f77ff3e8700 1 -- 172.21.15.167:0/2969922888 <== osd.2 v2:172.21.15.167:6816/30037 5 ==== osd_op_reply(33 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f77e8012450 con 0x7f77f801a070 2023-10-05T19:28:25.878 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.874+0000 7f77ff3e8700 1 -- 172.21.15.167:0/2969922888 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:34 5.7 5:e0000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x556302778e60 con 0x556302776440 2023-10-05T19:28:25.879 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.874+0000 7f77ffbe9700 1 -- 172.21.15.167:0/2969922888 <== osd.3 v2:172.21.15.167:6808/30036 9 ==== osd_op_reply(34 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f77f8039040 con 0x556302776440 2023-10-05T19:28:25.879 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.874+0000 7f77ffbe9700 1 -- 172.21.15.167:0/2969922888 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:35 5.17 5:e8000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x556302778e60 con 0x7f77e8007030 2023-10-05T19:28:25.880 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.874+0000 7f78003ea700 1 -- 172.21.15.167:0/2969922888 <== osd.0 v2:172.21.15.118:6800/27690 9 ==== osd_op_reply(35 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f77f000a5a0 con 0x7f77e8007030 2023-10-05T19:28:25.880 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.874+0000 7f78003ea700 1 -- 172.21.15.167:0/2969922888 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:36 5.f 5:f0000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x556302778e60 con 0x5563027442a0 2023-10-05T19:28:25.881 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.874+0000 7f78003ea700 1 -- 172.21.15.167:0/2969922888 <== osd.1 v2:172.21.15.167:6800/30035 13 ==== osd_op_reply(36 [pgnls start_epoch 30 out=115b] v23'2 uv2 ondisk = 1) v8 ==== 144+0+115 (crc 0 0 0) 0x7f77f000a5a0 con 0x5563027442a0 2023-10-05T19:28:25.881 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.874+0000 7f78003ea700 1 -- 172.21.15.167:0/2969922888 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:37 5.1f 5:f8000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x556302778e60 con 0x7f77f801a070 2023-10-05T19:28:25.882 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.874+0000 7f77ff3e8700 1 -- 172.21.15.167:0/2969922888 <== osd.2 v2:172.21.15.167:6816/30037 6 ==== osd_op_reply(37 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f77e8012450 con 0x7f77f801a070 2023-10-05T19:28:25.882 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.874+0000 7f78020b4e40 20 RGWRados::pool_iterate: got zonegroup_info.4f1f719d-9648-417b-b10c-e5d939825419 2023-10-05T19:28:25.882 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.874+0000 7f78020b4e40 20 RGWRados::pool_iterate: got zone_info.9a9df965-6612-45d4-a45f-6e3c889bacc2 2023-10-05T19:28:25.882 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.874+0000 7f78020b4e40 20 RGWRados::pool_iterate: got zone_names.default 2023-10-05T19:28:25.882 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.874+0000 7f78020b4e40 20 RGWRados::pool_iterate: got zonegroups_names.default 2023-10-05T19:28:25.882 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.874+0000 7f78020b4e40 20 rados->read ofs=0 len=0 2023-10-05T19:28:25.882 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.874+0000 7f78020b4e40 1 -- 172.21.15.167:0/2969922888 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:38 5.12 5:49953fa1:::default.realm:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x556302774ef0 con 0x5563027442a0 2023-10-05T19:28:25.883 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.878+0000 7f78003ea700 1 -- 172.21.15.167:0/2969922888 <== osd.1 v2:172.21.15.167:6800/30035 14 ==== osd_op_reply(38 default.realm [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 ==== 157+0+0 (crc 0 0 0) 0x7f77f000a5a0 con 0x5563027442a0 2023-10-05T19:28:25.883 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.878+0000 7f78020b4e40 20 rados_obj.operate() r=-2 bl.length=0 2023-10-05T19:28:25.884 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.878+0000 7f78020b4e40 20 rados->read ofs=0 len=0 2023-10-05T19:28:25.884 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.878+0000 7f78020b4e40 1 -- 172.21.15.167:0/2969922888 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:39 5.f 5:f4c53578:::zonegroups_names.default:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x5563027752f0 con 0x5563027442a0 2023-10-05T19:28:25.884 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.878+0000 7f78003ea700 1 -- 172.21.15.167:0/2969922888 <== osd.1 v2:172.21.15.167:6800/30035 15 ==== osd_op_reply(39 zonegroups_names.default [read 0~46 out=46b] v0'0 uv2 ondisk = 0) v8 ==== 168+0+46 (crc 0 0 0) 0x7f77f000a5a0 con 0x5563027442a0 2023-10-05T19:28:25.884 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.878+0000 7f78020b4e40 20 rados_obj.operate() r=0 bl.length=46 2023-10-05T19:28:25.884 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.878+0000 7f78020b4e40 20 rados->read ofs=0 len=0 2023-10-05T19:28:25.884 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.878+0000 7f78020b4e40 1 -- 172.21.15.167:0/2969922888 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:40 5.c 5:30d299a5:::zonegroup_info.4f1f719d-9648-417b-b10c-e5d939825419:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x5563027752f0 con 0x7f77e8007030 2023-10-05T19:28:25.885 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.878+0000 7f78003ea700 1 -- 172.21.15.167:0/2969922888 <== osd.0 v2:172.21.15.118:6800/27690 10 ==== osd_op_reply(40 zonegroup_info.4f1f719d-9648-417b-b10c-e5d939825419 [read 0~362 out=362b] v0'0 uv1 ondisk = 0) v8 ==== 195+0+362 (crc 0 0 0) 0x7f77f000a5a0 con 0x7f77e8007030 2023-10-05T19:28:25.885 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.878+0000 7f78020b4e40 20 rados_obj.operate() r=0 bl.length=362 2023-10-05T19:28:25.885 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.878+0000 7f78020b4e40 20 rados->read ofs=0 len=0 2023-10-05T19:28:25.885 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.878+0000 7f78020b4e40 1 -- 172.21.15.167:0/2969922888 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:41 5.13 5:cbbf6c60:::converted:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x5563027752f0 con 0x556302776440 2023-10-05T19:28:25.886 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.878+0000 7f77ffbe9700 1 -- 172.21.15.167:0/2969922888 <== osd.3 v2:172.21.15.167:6808/30036 10 ==== osd_op_reply(41 converted [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 ==== 153+0+0 (crc 0 0 0) 0x7f77f8039040 con 0x556302776440 2023-10-05T19:28:25.886 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.878+0000 7f78020b4e40 20 rados_obj.operate() r=-2 bl.length=0 2023-10-05T19:28:25.886 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.878+0000 7f78020b4e40 20 rados->read ofs=0 len=0 2023-10-05T19:28:25.886 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.878+0000 7f78020b4e40 1 -- 172.21.15.167:0/2969922888 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:42 5.12 5:49953fa1:::default.realm:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x5563027792d0 con 0x5563027442a0 2023-10-05T19:28:25.886 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.878+0000 7f78003ea700 1 -- 172.21.15.167:0/2969922888 <== osd.1 v2:172.21.15.167:6800/30035 16 ==== osd_op_reply(42 default.realm [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 ==== 157+0+0 (crc 0 0 0) 0x7f77f000a5a0 con 0x5563027442a0 2023-10-05T19:28:25.886 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.882+0000 7f78020b4e40 20 rados_obj.operate() r=-2 bl.length=0 2023-10-05T19:28:25.886 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.882+0000 7f78020b4e40 20 rados->read ofs=0 len=0 2023-10-05T19:28:25.886 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.882+0000 7f78020b4e40 1 -- 172.21.15.167:0/2969922888 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:43 5.f 5:f4c53578:::zonegroups_names.default:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x556302774af0 con 0x5563027442a0 2023-10-05T19:28:25.887 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.882+0000 7f78003ea700 1 -- 172.21.15.167:0/2969922888 <== osd.1 v2:172.21.15.167:6800/30035 17 ==== osd_op_reply(43 zonegroups_names.default [read 0~46 out=46b] v0'0 uv2 ondisk = 0) v8 ==== 168+0+46 (crc 0 0 0) 0x7f77f000a5a0 con 0x5563027442a0 2023-10-05T19:28:25.887 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.882+0000 7f78020b4e40 20 rados_obj.operate() r=0 bl.length=46 2023-10-05T19:28:25.887 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.882+0000 7f78020b4e40 1 -- 172.21.15.167:0/2969922888 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:44 5.0 5:00000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x556302774ac0 con 0x556302776440 2023-10-05T19:28:25.889 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.882+0000 7f77ffbe9700 1 -- 172.21.15.167:0/2969922888 <== osd.3 v2:172.21.15.167:6808/30036 11 ==== osd_op_reply(44 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f77f8039040 con 0x556302776440 2023-10-05T19:28:25.889 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.882+0000 7f77ffbe9700 1 -- 172.21.15.167:0/2969922888 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:45 5.10 5:08000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x556302774ac0 con 0x7f77f801a070 2023-10-05T19:28:25.890 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.886+0000 7f77ff3e8700 1 -- 172.21.15.167:0/2969922888 <== osd.2 v2:172.21.15.167:6816/30037 7 ==== osd_op_reply(45 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f77e8012450 con 0x7f77f801a070 2023-10-05T19:28:25.890 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.886+0000 7f77ff3e8700 1 -- 172.21.15.167:0/2969922888 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:46 5.8 5:10000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x556302774ac0 con 0x7f77e8007030 2023-10-05T19:28:25.891 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.886+0000 7f78003ea700 1 -- 172.21.15.167:0/2969922888 <== osd.0 v2:172.21.15.118:6800/27690 11 ==== osd_op_reply(46 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f77f000a5a0 con 0x7f77e8007030 2023-10-05T19:28:25.891 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.886+0000 7f78003ea700 1 -- 172.21.15.167:0/2969922888 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:47 5.18 5:18000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x556302774ef0 con 0x5563027442a0 2023-10-05T19:28:25.892 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.886+0000 7f78003ea700 1 -- 172.21.15.167:0/2969922888 <== osd.1 v2:172.21.15.167:6800/30035 18 ==== osd_op_reply(47 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f77f000a5a0 con 0x5563027442a0 2023-10-05T19:28:25.892 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.886+0000 7f78003ea700 1 -- 172.21.15.167:0/2969922888 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:48 5.4 5:20000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x556302774ef0 con 0x5563027442a0 2023-10-05T19:28:25.892 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.886+0000 7f78003ea700 1 -- 172.21.15.167:0/2969922888 <== osd.1 v2:172.21.15.167:6800/30035 19 ==== osd_op_reply(48 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f77f000a5a0 con 0x5563027442a0 2023-10-05T19:28:25.893 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.886+0000 7f78003ea700 1 -- 172.21.15.167:0/2969922888 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:49 5.14 5:28000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x556302774ef0 con 0x7f77e8007030 2023-10-05T19:28:25.894 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.886+0000 7f78003ea700 1 -- 172.21.15.167:0/2969922888 <== osd.0 v2:172.21.15.118:6800/27690 12 ==== osd_op_reply(49 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f77f000a5a0 con 0x7f77e8007030 2023-10-05T19:28:25.894 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.886+0000 7f78003ea700 1 -- 172.21.15.167:0/2969922888 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:50 5.c 5:30000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x556302774ef0 con 0x7f77e8007030 2023-10-05T19:28:25.894 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.890+0000 7f78003ea700 1 -- 172.21.15.167:0/2969922888 <== osd.0 v2:172.21.15.118:6800/27690 13 ==== osd_op_reply(50 [pgnls start_epoch 30 out=112b] v23'1 uv1 ondisk = 1) v8 ==== 144+0+112 (crc 0 0 0) 0x7f77f000a5a0 con 0x7f77e8007030 2023-10-05T19:28:25.894 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.890+0000 7f78003ea700 1 -- 172.21.15.167:0/2969922888 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:51 5.1c 5:38000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x556302774ef0 con 0x7f77f801a070 2023-10-05T19:28:25.895 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.890+0000 7f77ff3e8700 1 -- 172.21.15.167:0/2969922888 <== osd.2 v2:172.21.15.167:6816/30037 8 ==== osd_op_reply(51 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f77e8012450 con 0x7f77f801a070 2023-10-05T19:28:25.895 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.890+0000 7f77ff3e8700 1 -- 172.21.15.167:0/2969922888 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:52 5.2 5:40000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x556302774ef0 con 0x556302776440 2023-10-05T19:28:25.896 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.890+0000 7f77ffbe9700 1 -- 172.21.15.167:0/2969922888 <== osd.3 v2:172.21.15.167:6808/30036 12 ==== osd_op_reply(52 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f77f8039040 con 0x556302776440 2023-10-05T19:28:25.896 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.890+0000 7f77ffbe9700 1 -- 172.21.15.167:0/2969922888 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:53 5.12 5:48000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x556302774ef0 con 0x5563027442a0 2023-10-05T19:28:25.897 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.890+0000 7f78003ea700 1 -- 172.21.15.167:0/2969922888 <== osd.1 v2:172.21.15.167:6800/30035 20 ==== osd_op_reply(53 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f77f000a5a0 con 0x5563027442a0 2023-10-05T19:28:25.897 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.890+0000 7f78003ea700 1 -- 172.21.15.167:0/2969922888 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:54 5.a 5:50000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x556302774ef0 con 0x7f77e8007030 2023-10-05T19:28:25.898 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.894+0000 7f78003ea700 1 -- 172.21.15.167:0/2969922888 <== osd.0 v2:172.21.15.118:6800/27690 14 ==== osd_op_reply(54 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f77f000a5a0 con 0x7f77e8007030 2023-10-05T19:28:25.898 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.894+0000 7f78003ea700 1 -- 172.21.15.167:0/2969922888 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:55 5.1a 5:58000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x556302774ef0 con 0x5563027442a0 2023-10-05T19:28:25.902 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.894+0000 7f78003ea700 1 -- 172.21.15.167:0/2969922888 <== osd.1 v2:172.21.15.167:6800/30035 21 ==== osd_op_reply(55 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f77f000a5a0 con 0x5563027442a0 2023-10-05T19:28:25.902 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.894+0000 7f78003ea700 1 -- 172.21.15.167:0/2969922888 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:56 5.6 5:60000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x556302774ef0 con 0x7f77f801a070 2023-10-05T19:28:25.902 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.894+0000 7f77ff3e8700 1 -- 172.21.15.167:0/2969922888 <== osd.2 v2:172.21.15.167:6816/30037 9 ==== osd_op_reply(56 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f77e8012450 con 0x7f77f801a070 2023-10-05T19:28:25.902 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.894+0000 7f77ff3e8700 1 -- 172.21.15.167:0/2969922888 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:57 5.16 5:68000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x556302774ef0 con 0x556302776440 2023-10-05T19:28:25.902 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.894+0000 7f77ffbe9700 1 -- 172.21.15.167:0/2969922888 <== osd.3 v2:172.21.15.167:6808/30036 13 ==== osd_op_reply(57 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f77f8039040 con 0x556302776440 2023-10-05T19:28:25.902 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.894+0000 7f77ffbe9700 1 -- 172.21.15.167:0/2969922888 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:58 5.e 5:70000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x556302774ef0 con 0x556302776440 2023-10-05T19:28:25.902 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.894+0000 7f77ffbe9700 1 -- 172.21.15.167:0/2969922888 <== osd.3 v2:172.21.15.167:6808/30036 14 ==== osd_op_reply(58 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f77f8039040 con 0x556302776440 2023-10-05T19:28:25.902 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.894+0000 7f77ffbe9700 1 -- 172.21.15.167:0/2969922888 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:59 5.1e 5:78000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x556302774ef0 con 0x7f77e8007030 2023-10-05T19:28:25.903 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.898+0000 7f78003ea700 1 -- 172.21.15.167:0/2969922888 <== osd.0 v2:172.21.15.118:6800/27690 15 ==== osd_op_reply(59 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f77f003f040 con 0x7f77e8007030 2023-10-05T19:28:25.903 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.898+0000 7f78003ea700 1 -- 172.21.15.167:0/2969922888 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:60 5.1 5:80000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x556302774ef0 con 0x556302776440 2023-10-05T19:28:25.903 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.898+0000 7f77ffbe9700 1 -- 172.21.15.167:0/2969922888 <== osd.3 v2:172.21.15.167:6808/30036 15 ==== osd_op_reply(60 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f77f8039040 con 0x556302776440 2023-10-05T19:28:25.903 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.898+0000 7f77ffbe9700 1 -- 172.21.15.167:0/2969922888 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:61 5.11 5:88000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x556302774ef0 con 0x556302776440 2023-10-05T19:28:25.904 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.898+0000 7f77ffbe9700 1 -- 172.21.15.167:0/2969922888 <== osd.3 v2:172.21.15.167:6808/30036 16 ==== osd_op_reply(61 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f77f8039040 con 0x556302776440 2023-10-05T19:28:25.904 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.898+0000 7f77ffbe9700 1 -- 172.21.15.167:0/2969922888 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:62 5.9 5:90000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x556302774ef0 con 0x5563027442a0 2023-10-05T19:28:25.905 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.898+0000 7f78003ea700 1 -- 172.21.15.167:0/2969922888 <== osd.1 v2:172.21.15.167:6800/30035 22 ==== osd_op_reply(62 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f77f003f040 con 0x5563027442a0 2023-10-05T19:28:25.905 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.898+0000 7f78003ea700 1 -- 172.21.15.167:0/2969922888 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:63 5.19 5:98000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x556302774ef0 con 0x5563027442a0 2023-10-05T19:28:25.905 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.898+0000 7f78003ea700 1 -- 172.21.15.167:0/2969922888 <== osd.1 v2:172.21.15.167:6800/30035 23 ==== osd_op_reply(63 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f77f003f040 con 0x5563027442a0 2023-10-05T19:28:25.905 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.898+0000 7f78003ea700 1 -- 172.21.15.167:0/2969922888 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:64 5.5 5:a0000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x556302774ef0 con 0x7f77e8007030 2023-10-05T19:28:25.906 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.902+0000 7f78003ea700 1 -- 172.21.15.167:0/2969922888 <== osd.0 v2:172.21.15.118:6800/27690 16 ==== osd_op_reply(64 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f77f003f040 con 0x7f77e8007030 2023-10-05T19:28:25.906 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.902+0000 7f78003ea700 1 -- 172.21.15.167:0/2969922888 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:65 5.15 5:a8000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x556302774ef0 con 0x556302776440 2023-10-05T19:28:25.907 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.902+0000 7f77ffbe9700 1 -- 172.21.15.167:0/2969922888 <== osd.3 v2:172.21.15.167:6808/30036 17 ==== osd_op_reply(65 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f77f8039040 con 0x556302776440 2023-10-05T19:28:25.907 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.902+0000 7f77ffbe9700 1 -- 172.21.15.167:0/2969922888 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:66 5.d 5:b0000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x556302774ef0 con 0x7f77f801a070 2023-10-05T19:28:25.908 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.902+0000 7f77ff3e8700 1 -- 172.21.15.167:0/2969922888 <== osd.2 v2:172.21.15.167:6816/30037 10 ==== osd_op_reply(66 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f77e8012450 con 0x7f77f801a070 2023-10-05T19:28:25.908 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.902+0000 7f77ff3e8700 1 -- 172.21.15.167:0/2969922888 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:67 5.1d 5:b8000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x556302774ef0 con 0x5563027442a0 2023-10-05T19:28:25.909 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.902+0000 7f78003ea700 1 -- 172.21.15.167:0/2969922888 <== osd.1 v2:172.21.15.167:6800/30035 24 ==== osd_op_reply(67 [pgnls start_epoch 30 out=107b] v30'2 uv2 ondisk = 1) v8 ==== 144+0+107 (crc 0 0 0) 0x7f77f003f040 con 0x5563027442a0 2023-10-05T19:28:25.909 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.902+0000 7f78003ea700 1 -- 172.21.15.167:0/2969922888 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:68 5.3 5:c0000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x556302774ef0 con 0x7f77e8007030 2023-10-05T19:28:25.910 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.906+0000 7f78003ea700 1 -- 172.21.15.167:0/2969922888 <== osd.0 v2:172.21.15.118:6800/27690 17 ==== osd_op_reply(68 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f77f003f040 con 0x7f77e8007030 2023-10-05T19:28:25.910 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.906+0000 7f78003ea700 1 -- 172.21.15.167:0/2969922888 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:69 5.13 5:c8000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x556302774ef0 con 0x556302776440 2023-10-05T19:28:25.911 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.906+0000 7f77ffbe9700 1 -- 172.21.15.167:0/2969922888 <== osd.3 v2:172.21.15.167:6808/30036 18 ==== osd_op_reply(69 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f77f8039040 con 0x556302776440 2023-10-05T19:28:25.911 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.906+0000 7f77ffbe9700 1 -- 172.21.15.167:0/2969922888 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:70 5.b 5:d0000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x556302774ef0 con 0x7f77e8007030 2023-10-05T19:28:25.912 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.906+0000 7f78003ea700 1 -- 172.21.15.167:0/2969922888 <== osd.0 v2:172.21.15.118:6800/27690 18 ==== osd_op_reply(70 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f77f003f040 con 0x7f77e8007030 2023-10-05T19:28:25.912 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.906+0000 7f78003ea700 1 -- 172.21.15.167:0/2969922888 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:71 5.1b 5:d8000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x556302774ef0 con 0x7f77f801a070 2023-10-05T19:28:25.913 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.906+0000 7f77ff3e8700 1 -- 172.21.15.167:0/2969922888 <== osd.2 v2:172.21.15.167:6816/30037 11 ==== osd_op_reply(71 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f77e8012450 con 0x7f77f801a070 2023-10-05T19:28:25.913 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.906+0000 7f77ff3e8700 1 -- 172.21.15.167:0/2969922888 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:72 5.7 5:e0000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x556302774ef0 con 0x556302776440 2023-10-05T19:28:25.914 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.910+0000 7f77ffbe9700 1 -- 172.21.15.167:0/2969922888 <== osd.3 v2:172.21.15.167:6808/30036 19 ==== osd_op_reply(72 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f77f8039040 con 0x556302776440 2023-10-05T19:28:25.914 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.910+0000 7f77ffbe9700 1 -- 172.21.15.167:0/2969922888 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:73 5.17 5:e8000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x556302774ef0 con 0x7f77e8007030 2023-10-05T19:28:25.915 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.910+0000 7f78003ea700 1 -- 172.21.15.167:0/2969922888 <== osd.0 v2:172.21.15.118:6800/27690 19 ==== osd_op_reply(73 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f77f003f040 con 0x7f77e8007030 2023-10-05T19:28:25.915 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.910+0000 7f78003ea700 1 -- 172.21.15.167:0/2969922888 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:74 5.f 5:f0000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x556302774ef0 con 0x5563027442a0 2023-10-05T19:28:25.916 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.910+0000 7f78003ea700 1 -- 172.21.15.167:0/2969922888 <== osd.1 v2:172.21.15.167:6800/30035 25 ==== osd_op_reply(74 [pgnls start_epoch 30 out=115b] v23'2 uv2 ondisk = 1) v8 ==== 144+0+115 (crc 0 0 0) 0x7f77f003f040 con 0x5563027442a0 2023-10-05T19:28:25.916 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.910+0000 7f78003ea700 1 -- 172.21.15.167:0/2969922888 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:75 5.1f 5:f8000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x556302774ef0 con 0x7f77f801a070 2023-10-05T19:28:25.917 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.910+0000 7f77ff3e8700 1 -- 172.21.15.167:0/2969922888 <== osd.2 v2:172.21.15.167:6816/30037 12 ==== osd_op_reply(75 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f77e8012450 con 0x7f77f801a070 2023-10-05T19:28:25.917 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.910+0000 7f78020b4e40 20 RGWRados::pool_iterate: got zonegroup_info.4f1f719d-9648-417b-b10c-e5d939825419 2023-10-05T19:28:25.917 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.910+0000 7f78020b4e40 20 RGWRados::pool_iterate: got zone_info.9a9df965-6612-45d4-a45f-6e3c889bacc2 2023-10-05T19:28:25.917 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.910+0000 7f78020b4e40 20 RGWRados::pool_iterate: got zone_names.default 2023-10-05T19:28:25.917 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.910+0000 7f78020b4e40 20 RGWRados::pool_iterate: got zonegroups_names.default 2023-10-05T19:28:25.917 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.910+0000 7f78020b4e40 20 rados->read ofs=0 len=0 2023-10-05T19:28:25.918 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.910+0000 7f78020b4e40 1 -- 172.21.15.167:0/2969922888 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:76 5.f 5:f43b6ece:::zone_names.default:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x556302774ac0 con 0x5563027442a0 2023-10-05T19:28:25.918 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.910+0000 7f78003ea700 1 -- 172.21.15.167:0/2969922888 <== osd.1 v2:172.21.15.167:6800/30035 26 ==== osd_op_reply(76 zone_names.default [read 0~46 out=46b] v0'0 uv1 ondisk = 0) v8 ==== 162+0+46 (crc 0 0 0) 0x7f77f003f040 con 0x5563027442a0 2023-10-05T19:28:25.918 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.910+0000 7f78020b4e40 20 rados_obj.operate() r=0 bl.length=46 2023-10-05T19:28:25.918 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.910+0000 7f78020b4e40 20 rados->read ofs=0 len=0 2023-10-05T19:28:25.918 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.914+0000 7f78020b4e40 1 -- 172.21.15.167:0/2969922888 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:77 5.1d 5:b8374eb4:::zone_info.9a9df965-6612-45d4-a45f-6e3c889bacc2:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x556302779840 con 0x5563027442a0 2023-10-05T19:28:25.919 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.914+0000 7f78003ea700 1 -- 172.21.15.167:0/2969922888 <== osd.1 v2:172.21.15.167:6800/30035 27 ==== osd_op_reply(77 zone_info.9a9df965-6612-45d4-a45f-6e3c889bacc2 [read 0~874 out=874b] v0'0 uv2 ondisk = 0) v8 ==== 190+0+874 (crc 0 0 0) 0x7f77f003f040 con 0x5563027442a0 2023-10-05T19:28:25.919 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.914+0000 7f78020b4e40 20 rados_obj.operate() r=0 bl.length=874 2023-10-05T19:28:25.919 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.914+0000 7f78020b4e40 20 rados->read ofs=0 len=0 2023-10-05T19:28:25.920 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.914+0000 7f78020b4e40 1 -- 172.21.15.167:0/2969922888 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:78 5.f 5:f4c53578:::zonegroups_names.default:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x556302779d30 con 0x5563027442a0 2023-10-05T19:28:25.920 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.914+0000 7f78003ea700 1 -- 172.21.15.167:0/2969922888 <== osd.1 v2:172.21.15.167:6800/30035 28 ==== osd_op_reply(78 zonegroups_names.default [read 0~46 out=46b] v0'0 uv2 ondisk = 0) v8 ==== 168+0+46 (crc 0 0 0) 0x7f77f003f040 con 0x5563027442a0 2023-10-05T19:28:25.920 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.914+0000 7f78020b4e40 20 rados_obj.operate() r=0 bl.length=46 2023-10-05T19:28:25.920 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.914+0000 7f78020b4e40 20 rados->read ofs=0 len=0 2023-10-05T19:28:25.920 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.914+0000 7f78020b4e40 1 -- 172.21.15.167:0/2969922888 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:79 5.c 5:30d299a5:::zonegroup_info.4f1f719d-9648-417b-b10c-e5d939825419:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x556302779d30 con 0x7f77e8007030 2023-10-05T19:28:25.921 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.914+0000 7f78003ea700 1 -- 172.21.15.167:0/2969922888 <== osd.0 v2:172.21.15.118:6800/27690 20 ==== osd_op_reply(79 zonegroup_info.4f1f719d-9648-417b-b10c-e5d939825419 [read 0~362 out=362b] v0'0 uv1 ondisk = 0) v8 ==== 195+0+362 (crc 0 0 0) 0x7f77f003f040 con 0x7f77e8007030 2023-10-05T19:28:25.921 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.914+0000 7f78020b4e40 20 rados_obj.operate() r=0 bl.length=362 2023-10-05T19:28:25.921 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.914+0000 7f78020b4e40 20 rados->read ofs=0 len=0 2023-10-05T19:28:25.921 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.914+0000 7f78020b4e40 1 -- 172.21.15.167:0/2969922888 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:80 5.3 5:c0ad8cbd:::region_map:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x556302779d30 con 0x7f77e8007030 2023-10-05T19:28:25.922 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.914+0000 7f78003ea700 1 -- 172.21.15.167:0/2969922888 <== osd.0 v2:172.21.15.118:6800/27690 21 ==== osd_op_reply(80 region_map [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 ==== 154+0+0 (crc 0 0 0) 0x7f77f003f040 con 0x7f77e8007030 2023-10-05T19:28:25.922 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.914+0000 7f78020b4e40 20 rados_obj.operate() r=-2 bl.length=0 2023-10-05T19:28:25.922 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.914+0000 7f78020b4e40 20 zone default found 2023-10-05T19:28:25.922 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.914+0000 7f78020b4e40 4 Realm: () 2023-10-05T19:28:25.922 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.914+0000 7f78020b4e40 4 ZoneGroup: default (4f1f719d-9648-417b-b10c-e5d939825419) 2023-10-05T19:28:25.922 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.914+0000 7f78020b4e40 4 Zone: default (9a9df965-6612-45d4-a45f-6e3c889bacc2) 2023-10-05T19:28:25.922 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.914+0000 7f78020b4e40 10 cannot find current period zonegroup using local zonegroup configuration 2023-10-05T19:28:25.922 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.914+0000 7f78020b4e40 20 zonegroup default 2023-10-05T19:28:25.922 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.914+0000 7f78020b4e40 20 rados->read ofs=0 len=0 2023-10-05T19:28:25.922 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.914+0000 7f78020b4e40 1 -- 172.21.15.167:0/2969922888 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:81 5.3 5:c52100b6:::period_config.default:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x556302779d30 con 0x7f77e8007030 2023-10-05T19:28:25.923 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.918+0000 7f78003ea700 1 -- 172.21.15.167:0/2969922888 <== osd.0 v2:172.21.15.118:6800/27690 22 ==== osd_op_reply(81 period_config.default [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 ==== 165+0+0 (crc 0 0 0) 0x7f77f0052040 con 0x7f77e8007030 2023-10-05T19:28:25.924 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.918+0000 7f78020b4e40 20 rados_obj.operate() r=-2 bl.length=0 2023-10-05T19:28:25.924 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.918+0000 7f78020b4e40 20 rados->read ofs=0 len=0 2023-10-05T19:28:25.924 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.918+0000 7f78020b4e40 1 -- 172.21.15.167:0/2969922888 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:82 6.2 6:4347d321:::bucket.sync-source-hints.:head [call version.read in=11b,read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x55630277b6b0 con 0x556302776440 2023-10-05T19:28:25.924 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.918+0000 7f77ffbe9700 1 -- 172.21.15.167:0/2969922888 <== osd.3 v2:172.21.15.167:6808/30036 20 ==== osd_op_reply(82 bucket.sync-source-hints. [call,read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 ==== 211+0+0 (crc 0 0 0) 0x7f77f8039040 con 0x556302776440 2023-10-05T19:28:25.924 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.918+0000 7f78020b4e40 20 rados_obj.operate() r=-2 bl.length=0 2023-10-05T19:28:25.924 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.918+0000 7f78020b4e40 20 rados->read ofs=0 len=0 2023-10-05T19:28:25.924 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.918+0000 7f78020b4e40 1 -- 172.21.15.167:0/2969922888 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:83 6.b 6:d467b91b:::bucket.sync-target-hints.:head [call version.read in=11b,read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x55630277ba60 con 0x556302776440 2023-10-05T19:28:25.925 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.918+0000 7f77ffbe9700 1 -- 172.21.15.167:0/2969922888 <== osd.3 v2:172.21.15.167:6808/30036 21 ==== osd_op_reply(83 bucket.sync-target-hints. [call,read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 ==== 211+0+0 (crc 0 0 0) 0x7f77f8039040 con 0x556302776440 2023-10-05T19:28:25.925 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.918+0000 7f78020b4e40 20 rados_obj.operate() r=-2 bl.length=0 2023-10-05T19:28:25.925 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.918+0000 7f78020b4e40 20 started sync module instance, tier type = 2023-10-05T19:28:25.925 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.918+0000 7f78020b4e40 20 started zone id=9a9df965-6612-45d4-a45f-6e3c889bacc2 (name=default) with tier type = 2023-10-05T19:28:25.925 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.918+0000 7f78020b4e40 1 -- 172.21.15.167:0/2969922888 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:84 7.1f 7:f95f44c2:::notify.0:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x55630277ad70 con 0x7f77f801a070 2023-10-05T19:28:25.927 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.922+0000 7f77ff3e8700 1 -- 172.21.15.167:0/2969922888 <== osd.2 v2:172.21.15.167:6816/30037 13 ==== osd_op_reply(84 notify.0 [create] v30'27 uv27 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f77e8012450 con 0x7f77f801a070 2023-10-05T19:28:25.928 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.922+0000 7f78020b4e40 1 -- 172.21.15.167:0/2969922888 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:85 7.1f 7:f95f44c2:::notify.0:head [watch watch cookie 93883731521904] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x55630277a9a0 con 0x7f77f801a070 2023-10-05T19:28:25.928 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.922+0000 7f78020b4e40 1 -- 172.21.15.167:0/2969922888 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:86 7.0 7:05bf5b68:::notify.1:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x55630277c7f0 con 0x5563027442a0 2023-10-05T19:28:25.930 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.926+0000 7f77ff3e8700 1 -- 172.21.15.167:0/2969922888 <== osd.2 v2:172.21.15.167:6816/30037 14 ==== osd_op_reply(85 notify.0 [watch watch cookie 93883731521904] v30'28 uv27 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f77e8012450 con 0x7f77f801a070 2023-10-05T19:28:25.931 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.926+0000 7f78003ea700 1 -- 172.21.15.167:0/2969922888 <== osd.1 v2:172.21.15.167:6800/30035 29 ==== osd_op_reply(86 notify.1 [create] v30'27 uv27 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f77f0052040 con 0x5563027442a0 2023-10-05T19:28:25.931 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.926+0000 7f78020b4e40 1 -- 172.21.15.167:0/2969922888 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:87 7.0 7:05bf5b68:::notify.1:head [watch watch cookie 93883731525424] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x55630277c8f0 con 0x5563027442a0 2023-10-05T19:28:25.931 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.926+0000 7f78020b4e40 1 -- 172.21.15.167:0/2969922888 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:88 7.15 7:a93a5511:::notify.2:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x55630277d2b0 con 0x556302776440 2023-10-05T19:28:25.934 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.926+0000 7f77ffbe9700 1 -- 172.21.15.167:0/2969922888 <== osd.3 v2:172.21.15.167:6808/30036 22 ==== osd_op_reply(88 notify.2 [create] v30'27 uv27 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f77f8039040 con 0x556302776440 2023-10-05T19:28:25.934 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.926+0000 7f78020b4e40 1 -- 172.21.15.167:0/2969922888 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:89 7.15 7:a93a5511:::notify.2:head [watch watch cookie 93883731529088] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x55630277dba0 con 0x556302776440 2023-10-05T19:28:25.934 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.926+0000 7f78020b4e40 1 -- 172.21.15.167:0/2969922888 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:90 7.e 7:7759931f:::notify.3:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x55630277e5b0 con 0x7f77f801a070 2023-10-05T19:28:25.934 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.930+0000 7f78003ea700 1 -- 172.21.15.167:0/2969922888 <== osd.1 v2:172.21.15.167:6800/30035 30 ==== osd_op_reply(87 notify.1 [watch watch cookie 93883731525424] v30'28 uv27 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f77f0052040 con 0x5563027442a0 2023-10-05T19:28:25.936 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.930+0000 7f77ff3e8700 1 -- 172.21.15.167:0/2969922888 <== osd.2 v2:172.21.15.167:6816/30037 15 ==== osd_op_reply(90 notify.3 [create] v30'27 uv27 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f77e8012450 con 0x7f77f801a070 2023-10-05T19:28:25.936 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.930+0000 7f78020b4e40 1 -- 172.21.15.167:0/2969922888 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:91 7.e 7:7759931f:::notify.3:head [watch watch cookie 93883731526528] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x55630277e1e0 con 0x7f77f801a070 2023-10-05T19:28:25.936 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.930+0000 7f78020b4e40 1 -- 172.21.15.167:0/2969922888 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:92 7.d 7:b4812045:::notify.4:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x55630277f600 con 0x5563027442a0 2023-10-05T19:28:25.937 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.930+0000 7f77ffbe9700 1 -- 172.21.15.167:0/2969922888 <== osd.3 v2:172.21.15.167:6808/30036 23 ==== osd_op_reply(89 notify.2 [watch watch cookie 93883731529088] v30'28 uv27 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f77f8039040 con 0x556302776440 2023-10-05T19:28:25.939 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.934+0000 7f78003ea700 1 -- 172.21.15.167:0/2969922888 <== osd.1 v2:172.21.15.167:6800/30035 31 ==== osd_op_reply(92 notify.4 [create] v30'27 uv27 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f77f0052040 con 0x5563027442a0 2023-10-05T19:28:25.939 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.934+0000 7f78020b4e40 1 -- 172.21.15.167:0/2969922888 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:93 7.d 7:b4812045:::notify.4:head [watch watch cookie 93883731531440] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x55630277fa00 con 0x5563027442a0 2023-10-05T19:28:25.939 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.934+0000 7f78020b4e40 1 -- 172.21.15.167:0/2969922888 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:94 7.3 7:c609908c:::notify.5:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x556302780410 con 0x7f77e8007030 2023-10-05T19:28:25.940 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.934+0000 7f77ff3e8700 1 -- 172.21.15.167:0/2969922888 <== osd.2 v2:172.21.15.167:6816/30037 16 ==== osd_op_reply(91 notify.3 [watch watch cookie 93883731526528] v30'28 uv27 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f77e8012450 con 0x7f77f801a070 2023-10-05T19:28:25.941 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.934+0000 7f78003ea700 1 -- 172.21.15.167:0/2969922888 <== osd.0 v2:172.21.15.118:6800/27690 23 ==== osd_op_reply(94 notify.5 [create] v30'27 uv27 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f77f0052040 con 0x7f77e8007030 2023-10-05T19:28:25.941 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.934+0000 7f78020b4e40 1 -- 172.21.15.167:0/2969922888 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:95 7.3 7:c609908c:::notify.5:head [watch watch cookie 93883731536224] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x556302780410 con 0x7f77e8007030 2023-10-05T19:28:25.941 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.934+0000 7f78020b4e40 1 -- 172.21.15.167:0/2969922888 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:96 7.14 7:2b04a3e9:::notify.6:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x556302780cf0 con 0x7f77e8007030 2023-10-05T19:28:25.942 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.938+0000 7f78003ea700 1 -- 172.21.15.167:0/2969922888 <== osd.1 v2:172.21.15.167:6800/30035 32 ==== osd_op_reply(93 notify.4 [watch watch cookie 93883731531440] v30'28 uv27 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f77f0052040 con 0x5563027442a0 2023-10-05T19:28:25.943 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.938+0000 7f78003ea700 1 -- 172.21.15.167:0/2969922888 <== osd.0 v2:172.21.15.118:6800/27690 24 ==== osd_op_reply(95 notify.5 [watch watch cookie 93883731536224] v30'28 uv27 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f77f0052040 con 0x7f77e8007030 2023-10-05T19:28:25.944 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.938+0000 7f78003ea700 1 -- 172.21.15.167:0/2969922888 <== osd.0 v2:172.21.15.118:6800/27690 25 ==== osd_op_reply(96 notify.6 [create] v30'27 uv27 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f77f0052040 con 0x7f77e8007030 2023-10-05T19:28:25.944 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.938+0000 7f78020b4e40 1 -- 172.21.15.167:0/2969922888 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:97 7.14 7:2b04a3e9:::notify.6:head [watch watch cookie 93883731538128] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x556302780410 con 0x7f77e8007030 2023-10-05T19:28:25.944 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.938+0000 7f78020b4e40 1 -- 172.21.15.167:0/2969922888 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:98 7.9 7:93e5b521:::notify.7:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x5563027815e0 con 0x7f77e8007030 2023-10-05T19:28:25.947 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.942+0000 7f78003ea700 1 -- 172.21.15.167:0/2969922888 <== osd.0 v2:172.21.15.118:6800/27690 26 ==== osd_op_reply(98 notify.7 [create] v30'27 uv27 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f77f0052040 con 0x7f77e8007030 2023-10-05T19:28:25.947 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.942+0000 7f78020b4e40 1 -- 172.21.15.167:0/2969922888 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:99 7.9 7:93e5b521:::notify.7:head [watch watch cookie 93883731545888] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x556302780410 con 0x7f77e8007030 2023-10-05T19:28:25.947 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.942+0000 7f78020b4e40 20 add_watcher() i=0 2023-10-05T19:28:25.947 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.942+0000 7f78020b4e40 20 add_watcher() i=1 2023-10-05T19:28:25.947 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.942+0000 7f78020b4e40 20 add_watcher() i=2 2023-10-05T19:28:25.947 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.942+0000 7f78020b4e40 20 add_watcher() i=3 2023-10-05T19:28:25.947 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.942+0000 7f78020b4e40 20 add_watcher() i=4 2023-10-05T19:28:25.947 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.942+0000 7f78020b4e40 20 add_watcher() i=5 2023-10-05T19:28:25.947 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.942+0000 7f78003ea700 1 -- 172.21.15.167:0/2969922888 <== osd.0 v2:172.21.15.118:6800/27690 27 ==== osd_op_reply(97 notify.6 [watch watch cookie 93883731538128] v30'28 uv27 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f77f0052040 con 0x7f77e8007030 2023-10-05T19:28:25.947 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.942+0000 7f78020b4e40 20 add_watcher() i=6 2023-10-05T19:28:25.949 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.942+0000 7f78003ea700 1 -- 172.21.15.167:0/2969922888 <== osd.0 v2:172.21.15.118:6800/27690 28 ==== osd_op_reply(99 notify.7 [watch watch cookie 93883731545888] v30'28 uv27 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f77f0052040 con 0x7f77e8007030 2023-10-05T19:28:25.949 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.942+0000 7f78020b4e40 20 add_watcher() i=7 2023-10-05T19:28:25.949 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.942+0000 7f78020b4e40 2 all 8 watchers are set, enabling cache 2023-10-05T19:28:25.949 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.942+0000 7f78020b4e40 1 -- 172.21.15.167:0/2969922888 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:100 6.4 6:22d26bf9:::data_loggenerations_metadata:head [call version.check_conds in=74b,call version.read in=11b,read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x5563027849a0 con 0x5563027442a0 2023-10-05T19:28:25.949 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.942+0000 7f78003ea700 1 -- 172.21.15.167:0/2969922888 <== osd.1 v2:172.21.15.167:6800/30035 33 ==== osd_op_reply(100 data_loggenerations_metadata [call,call out=48b,read 0~28 out=28b] v0'0 uv1 ondisk = 0) v8 ==== 256+0+76 (crc 0 0 0) 0x7f77f0052040 con 0x5563027442a0 2023-10-05T19:28:25.949 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.942+0000 7f78020b4e40 1 -- 172.21.15.167:0/2969922888 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:101 6.4 6:22d26bf9:::data_loggenerations_metadata:head [watch watch cookie 93883731541840] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x5563027803a0 con 0x5563027442a0 2023-10-05T19:28:25.951 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.946+0000 7f78003ea700 1 -- 172.21.15.167:0/2969922888 <== osd.1 v2:172.21.15.167:6800/30035 34 ==== osd_op_reply(101 data_loggenerations_metadata [watch watch cookie 93883731541840] v30'52 uv1 ondisk = 0) v8 ==== 172+0+0 (crc 0 0 0) 0x7f77f0052040 con 0x5563027442a0 2023-10-05T19:28:25.951 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.946+0000 7f774bfff700 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2023-10-05T19:28:25.952 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.946+0000 7f78020b4e40 20 check_secure_mon_conn(): auth registy supported: methods=[2] modes=[2,1] 2023-10-05T19:28:25.952 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.946+0000 7f78020b4e40 20 check_secure_mon_conn(): mode 1 is insecure 2023-10-05T19:28:25.953 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.946+0000 7f78020b4e40 1 Processor -- start 2023-10-05T19:28:25.953 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.946+0000 7f78020b4e40 1 -- start start 2023-10-05T19:28:25.953 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.946+0000 7f78020b4e40 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55630278eb00 0x55630278eed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-05T19:28:25.953 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.946+0000 7f78020b4e40 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x55630278f410 con 0x55630278eb00 2023-10-05T19:28:25.953 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.946+0000 7f77ff3e8700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55630278eb00 0x55630278eed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-05T19:28:25.953 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.946+0000 7f77ff3e8700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55630278eb00 0x55630278eed0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.118:3300/0 says I am v2:172.21.15.167:57228/0 (socket says 172.21.15.167:57228) 2023-10-05T19:28:25.953 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.946+0000 7f77ff3e8700 1 -- 172.21.15.167:0/366001959 learned_addr learned my addr 172.21.15.167:0/366001959 (peer_addr_for_me v2:172.21.15.167:0/0) 2023-10-05T19:28:25.954 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.946+0000 7f77ff3e8700 1 -- 172.21.15.167:0/366001959 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55630278f550 con 0x55630278eb00 2023-10-05T19:28:25.954 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.946+0000 7f77ff3e8700 1 --2- 172.21.15.167:0/366001959 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55630278eb00 0x55630278eed0 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x556302740d40 tx=0x7f77e8012000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=978d43fa98a13ce8 server_cookie=0 in_seq=0 out_seq=0 2023-10-05T19:28:25.954 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.950+0000 7f774b7fe700 1 -- 172.21.15.167:0/366001959 <== mon.0 v2:172.21.15.118:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f77e801cec0 con 0x55630278eb00 2023-10-05T19:28:25.954 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.950+0000 7f774b7fe700 1 -- 172.21.15.167:0/366001959 <== mon.0 v2:172.21.15.118:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f77e8012e50 con 0x55630278eb00 2023-10-05T19:28:25.954 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.950+0000 7f774b7fe700 1 -- 172.21.15.167:0/366001959 <== mon.0 v2:172.21.15.118:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f77e802a8a0 con 0x55630278eb00 2023-10-05T19:28:25.955 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.950+0000 7f78020b4e40 1 -- 172.21.15.167:0/366001959 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55630278eb00 msgr2=0x55630278eed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-05T19:28:25.955 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.950+0000 7f78020b4e40 1 --2- 172.21.15.167:0/366001959 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55630278eb00 0x55630278eed0 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x556302740d40 tx=0x7f77e8012000 comp rx=0 tx=0).stop 2023-10-05T19:28:25.955 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.950+0000 7f78020b4e40 1 -- 172.21.15.167:0/366001959 shutdown_connections 2023-10-05T19:28:25.956 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.950+0000 7f78020b4e40 1 --2- 172.21.15.167:0/366001959 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55630278eb00 0x55630278eed0 unknown :-1 s=CLOSED pgs=192 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-05T19:28:25.956 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.950+0000 7f78020b4e40 1 -- 172.21.15.167:0/366001959 >> 172.21.15.167:0/366001959 conn(0x556302789ec0 msgr2=0x556302788010 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-05T19:28:25.956 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.950+0000 7f78020b4e40 1 -- 172.21.15.167:0/366001959 shutdown_connections 2023-10-05T19:28:25.956 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.950+0000 7f78020b4e40 1 -- 172.21.15.167:0/366001959 wait complete. 2023-10-05T19:28:25.957 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.950+0000 7f78020b4e40 1 Processor -- start 2023-10-05T19:28:25.957 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.950+0000 7f78020b4e40 1 -- start start 2023-10-05T19:28:25.957 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.950+0000 7f78020b4e40 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55630278eb00 0x5563027495a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-05T19:28:25.957 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.950+0000 7f78020b4e40 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x556302748ea0 con 0x55630278eb00 2023-10-05T19:28:25.957 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.950+0000 7f77ff3e8700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55630278eb00 0x5563027495a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-05T19:28:25.957 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.950+0000 7f77ff3e8700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55630278eb00 0x5563027495a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.118:3300/0 says I am v2:172.21.15.167:57232/0 (socket says 172.21.15.167:57232) 2023-10-05T19:28:25.957 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.950+0000 7f77ff3e8700 1 -- 172.21.15.167:0/3383862639 learned_addr learned my addr 172.21.15.167:0/3383862639 (peer_addr_for_me v2:172.21.15.167:0/0) 2023-10-05T19:28:25.958 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.950+0000 7f77ff3e8700 1 -- 172.21.15.167:0/3383862639 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5563027408d0 con 0x55630278eb00 2023-10-05T19:28:25.958 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.950+0000 7f77ff3e8700 1 --2- 172.21.15.167:0/3383862639 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55630278eb00 0x5563027495a0 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x556302788b80 tx=0x7f77e802f000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-05T19:28:25.958 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.954+0000 7f7749ffb700 1 -- 172.21.15.167:0/3383862639 <== mon.0 v2:172.21.15.118:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f77e8009890 con 0x55630278eb00 2023-10-05T19:28:25.959 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.954+0000 7f7749ffb700 1 -- 172.21.15.167:0/3383862639 <== mon.0 v2:172.21.15.118:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f77e80099f0 con 0x55630278eb00 2023-10-05T19:28:25.959 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.954+0000 7f78020b4e40 1 -- 172.21.15.167:0/3383862639 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x556302749ba0 con 0x55630278eb00 2023-10-05T19:28:25.959 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.954+0000 7f7749ffb700 1 -- 172.21.15.167:0/3383862639 <== mon.0 v2:172.21.15.118:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f77e8009ba0 con 0x55630278eb00 2023-10-05T19:28:25.959 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.954+0000 7f78020b4e40 1 -- 172.21.15.167:0/3383862639 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x556302747ea0 con 0x55630278eb00 2023-10-05T19:28:25.959 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.954+0000 7f7749ffb700 1 -- 172.21.15.167:0/3383862639 <== mon.0 v2:172.21.15.118:3300/0 4 ==== mgrmap(e 5) v1 ==== 40329+0+0 (secure 0 0 0) 0x7f77e8035400 con 0x55630278eb00 2023-10-05T19:28:25.960 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.954+0000 7f7749ffb700 1 --2- 172.21.15.167:0/3383862639 >> [v2:172.21.15.167:6824/29889,v1:172.21.15.167:6825/29889] conn(0x7f773c030c00 0x7f773c0330b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-05T19:28:25.960 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.954+0000 7f7749ffb700 1 -- 172.21.15.167:0/3383862639 <== mon.0 v2:172.21.15.118:3300/0 5 ==== osd_map(30..30 src has 1..30) v4 ==== 6057+0+0 (secure 0 0 0) 0x7f77e802f070 con 0x55630278eb00 2023-10-05T19:28:25.960 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.954+0000 7f78003ea700 1 --2- 172.21.15.167:0/3383862639 >> [v2:172.21.15.167:6824/29889,v1:172.21.15.167:6825/29889] conn(0x7f773c030c00 0x7f773c0330b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-05T19:28:25.960 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.954+0000 7f78020b4e40 5 note: GC not initialized 2023-10-05T19:28:25.961 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.954+0000 7f78003ea700 1 --2- 172.21.15.167:0/3383862639 >> [v2:172.21.15.167:6824/29889,v1:172.21.15.167:6825/29889] conn(0x7f773c030c00 0x7f773c0330b0 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x556302781620 tx=0x7f77f000aec0 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-05T19:28:25.963 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.958+0000 7f7742ffd700 20 reqs_thread_entry: start 2023-10-05T19:28:25.963 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.958+0000 7f7740ff9700 20 reqs_thread_entry: start 2023-10-05T19:28:25.963 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.958+0000 7f78020b4e40 20 init_complete bucket index max shards: 11 2023-10-05T19:28:25.964 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.958+0000 7f78020b4e40 10 rgw notify: Started notification manager with: 1 workers 2023-10-05T19:28:25.964 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.958+0000 7f772effd700 20 rgw notify: INFO: next queues processing will happen at: Thu Oct 5 19:28:56 2023 2023-10-05T19:28:25.964 INFO:teuthology.orchestra.run.smithi167.stderr: 2023-10-05T19:28:25.964 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.958+0000 7f772e7fc700 20 reqs_thread_entry: start 2023-10-05T19:28:25.964 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.958+0000 7f78020b4e40 20 get_system_obj_state: rctx=0x7ffed4168ab8 obj=default.rgw.meta:users.uid:fud state=0x55630261a0b0 s->prefetch_data=0 2023-10-05T19:28:25.965 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.958+0000 7f78020b4e40 10 cache get: name=default.rgw.meta+users.uid+fud : miss 2023-10-05T19:28:25.965 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.958+0000 7f78020b4e40 1 -- 172.21.15.167:0/2969922888 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:102 8.e 8:75f48285:users.uid::fud:head [call version.read in=11b,getxattrs,stat] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x5563027ff060 con 0x7f77e8007030 2023-10-05T19:28:25.965 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.958+0000 7f78003ea700 1 -- 172.21.15.167:0/2969922888 <== osd.0 v2:172.21.15.118:6800/27690 29 ==== osd_op_reply(102 fud [call,getxattrs,stat] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 ==== 231+0+0 (crc 0 0 0) 0x7f77f0052040 con 0x7f77e8007030 2023-10-05T19:28:25.965 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.958+0000 7f78020b4e40 10 cache put: name=default.rgw.meta+users.uid+fud info.flags=0x0 2023-10-05T19:28:25.966 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.958+0000 7f78020b4e40 10 adding default.rgw.meta+users.uid+fud to cache LRU end 2023-10-05T19:28:25.966 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.958+0000 7f78020b4e40 20 get_system_obj_state: rctx=0x7ffed41687a8 obj=default.rgw.meta:users.email:foo@foo.com state=0x55630261a0b0 s->prefetch_data=0 2023-10-05T19:28:25.966 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.958+0000 7f78020b4e40 10 cache get: name=default.rgw.meta+users.email+foo@foo.com : miss 2023-10-05T19:28:25.966 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.958+0000 7f78020b4e40 1 -- 172.21.15.167:0/2969922888 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:103 8.3 8:c162d5d1:users.email::foo@foo.com:head [getxattrs,stat] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x5563027ff060 con 0x5563027442a0 2023-10-05T19:28:25.967 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.962+0000 7f78003ea700 1 -- 172.21.15.167:0/2969922888 <== osd.1 v2:172.21.15.167:6800/30035 35 ==== osd_op_reply(103 foo@foo.com [getxattrs out=4b,stat out=16b] v0'0 uv1 ondisk = 0) v8 ==== 197+0+20 (crc 0 0 0) 0x7f77f0052040 con 0x5563027442a0 2023-10-05T19:28:25.967 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.962+0000 7f78020b4e40 10 cache put: name=default.rgw.meta+users.email+foo@foo.com info.flags=0x6 2023-10-05T19:28:25.967 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.962+0000 7f78020b4e40 10 adding default.rgw.meta+users.email+foo@foo.com to cache LRU end 2023-10-05T19:28:25.967 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.962+0000 7f78020b4e40 20 get_system_obj_state: s->obj_tag was set empty 2023-10-05T19:28:25.967 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.962+0000 7f78020b4e40 10 cache get: name=default.rgw.meta+users.email+foo@foo.com : type miss (requested=0x1, cached=0x6) 2023-10-05T19:28:25.967 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.962+0000 7f78020b4e40 20 rados->read ofs=0 len=0 2023-10-05T19:28:25.967 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.962+0000 7f78020b4e40 1 -- 172.21.15.167:0/2969922888 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:104 8.3 8:c162d5d1:users.email::foo@foo.com:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x5563027ff060 con 0x5563027442a0 2023-10-05T19:28:25.967 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.962+0000 7f78003ea700 1 -- 172.21.15.167:0/2969922888 <== osd.1 v2:172.21.15.167:6800/30035 36 ==== osd_op_reply(104 foo@foo.com [read 0~7 out=7b] v0'0 uv1 ondisk = 0) v8 ==== 155+0+7 (crc 0 0 0) 0x7f77f0052040 con 0x5563027442a0 2023-10-05T19:28:25.967 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.962+0000 7f78020b4e40 20 rados_obj.operate() r=0 bl.length=7 2023-10-05T19:28:25.967 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.962+0000 7f78020b4e40 10 cache put: name=default.rgw.meta+users.email+foo@foo.com info.flags=0x1 2023-10-05T19:28:25.968 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.962+0000 7f78020b4e40 10 moving default.rgw.meta+users.email+foo@foo.com to cache LRU end 2023-10-05T19:28:25.968 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.962+0000 7f78020b4e40 20 get_system_obj_state: rctx=0x7ffed41687a8 obj=default.rgw.meta:users.uid:foo state=0x5563025f02b0 s->prefetch_data=0 2023-10-05T19:28:25.968 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.962+0000 7f78020b4e40 10 cache get: name=default.rgw.meta+users.uid+foo : miss 2023-10-05T19:28:25.968 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.962+0000 7f78020b4e40 1 -- 172.21.15.167:0/2969922888 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:105 8.18 8:1e6ca5ed:users.uid::foo:head [call version.read in=11b,getxattrs,stat] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x5563028005f0 con 0x5563027442a0 2023-10-05T19:28:25.969 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.962+0000 7f78003ea700 1 -- 172.21.15.167:0/2969922888 <== osd.1 v2:172.21.15.167:6800/30035 37 ==== osd_op_reply(105 foo [call out=48b,getxattrs out=75b,stat out=16b] v0'0 uv1 ondisk = 0) v8 ==== 231+0+139 (crc 0 0 0) 0x7f77f0052040 con 0x5563027442a0 2023-10-05T19:28:25.969 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.962+0000 7f78020b4e40 10 cache put: name=default.rgw.meta+users.uid+foo info.flags=0x16 2023-10-05T19:28:25.969 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.962+0000 7f78020b4e40 10 adding default.rgw.meta+users.uid+foo to cache LRU end 2023-10-05T19:28:25.969 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.962+0000 7f78020b4e40 10 updating xattr: name=ceph.objclass.version bl.length()=42 2023-10-05T19:28:25.969 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.962+0000 7f78020b4e40 20 get_system_obj_state: s->obj_tag was set empty 2023-10-05T19:28:25.969 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.962+0000 7f78020b4e40 10 cache get: name=default.rgw.meta+users.uid+foo : type miss (requested=0x11, cached=0x16) 2023-10-05T19:28:25.969 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.962+0000 7f78020b4e40 20 rados->read ofs=0 len=0 2023-10-05T19:28:25.969 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.962+0000 7f78020b4e40 1 -- 172.21.15.167:0/2969922888 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:106 8.18 8:1e6ca5ed:users.uid::foo:head [call version.check_conds in=122b,call version.read in=11b,read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x5563027fefd0 con 0x5563027442a0 2023-10-05T19:28:25.969 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.962+0000 7f78003ea700 1 -- 172.21.15.167:0/2969922888 <== osd.1 v2:172.21.15.167:6800/30035 38 ==== osd_op_reply(106 foo [call,call out=48b,read 0~357 out=357b] v0'0 uv1 ondisk = 0) v8 ==== 231+0+405 (crc 0 0 0) 0x7f77f0052040 con 0x5563027442a0 2023-10-05T19:28:25.970 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.962+0000 7f78020b4e40 20 rados_obj.operate() r=0 bl.length=357 2023-10-05T19:28:25.970 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.962+0000 7f78020b4e40 10 cache put: name=default.rgw.meta+users.uid+foo info.flags=0x11 2023-10-05T19:28:25.970 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.962+0000 7f78020b4e40 10 moving default.rgw.meta+users.uid+foo to cache LRU end 2023-10-05T19:28:25.970 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.962+0000 7f78020b4e40 10 chain_cache_entry: cache_locator=default.rgw.meta+users.uid+foo 2023-10-05T19:28:25.970 INFO:teuthology.orchestra.run.smithi167.stderr:could not create user: unable to parse parameters, email: foo@foo.com is the email address of an existing user 2023-10-05T19:28:25.970 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.962+0000 7f78020b4e40 1 -- 172.21.15.167:0/2969922888 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:107 7.1f 7:f95f44c2:::notify.0:head [watch unwatch cookie 93883731521904] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x5563027bd450 con 0x7f77f801a070 2023-10-05T19:28:25.971 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.966+0000 7f77ff3e8700 1 -- 172.21.15.167:0/2969922888 <== osd.2 v2:172.21.15.167:6816/30037 17 ==== osd_op_reply(107 notify.0 [watch unwatch cookie 93883731521904] v30'29 uv27 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f77e8032040 con 0x7f77f801a070 2023-10-05T19:28:25.971 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.966+0000 7f78020b4e40 20 remove_watcher() i=0 2023-10-05T19:28:25.971 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.966+0000 7f78020b4e40 2 removed watcher, disabling cache 2023-10-05T19:28:25.971 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.966+0000 7f78020b4e40 1 -- 172.21.15.167:0/2969922888 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:108 7.0 7:05bf5b68:::notify.1:head [watch unwatch cookie 93883731525424] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x55630277b170 con 0x5563027442a0 2023-10-05T19:28:25.974 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.966+0000 7f78003ea700 1 -- 172.21.15.167:0/2969922888 <== osd.1 v2:172.21.15.167:6800/30035 39 ==== osd_op_reply(108 notify.1 [watch unwatch cookie 93883731525424] v30'29 uv27 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f77f0052040 con 0x5563027442a0 2023-10-05T19:28:25.974 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.970+0000 7f78020b4e40 20 remove_watcher() i=1 2023-10-05T19:28:25.974 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.970+0000 7f78020b4e40 1 -- 172.21.15.167:0/2969922888 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:109 7.15 7:a93a5511:::notify.2:head [watch unwatch cookie 93883731529088] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x55630277b170 con 0x556302776440 2023-10-05T19:28:25.976 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.970+0000 7f77ffbe9700 1 -- 172.21.15.167:0/2969922888 <== osd.3 v2:172.21.15.167:6808/30036 24 ==== osd_op_reply(109 notify.2 [watch unwatch cookie 93883731529088] v30'29 uv27 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f77f8039040 con 0x556302776440 2023-10-05T19:28:25.977 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.970+0000 7f78020b4e40 20 remove_watcher() i=2 2023-10-05T19:28:25.977 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.970+0000 7f78020b4e40 1 -- 172.21.15.167:0/2969922888 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:110 7.e 7:7759931f:::notify.3:head [watch unwatch cookie 93883731526528] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x55630277cd80 con 0x7f77f801a070 2023-10-05T19:28:25.979 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.974+0000 7f77ff3e8700 1 -- 172.21.15.167:0/2969922888 <== osd.2 v2:172.21.15.167:6816/30037 18 ==== osd_op_reply(110 notify.3 [watch unwatch cookie 93883731526528] v30'29 uv27 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f77e8032040 con 0x7f77f801a070 2023-10-05T19:28:25.979 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.974+0000 7f78020b4e40 20 remove_watcher() i=3 2023-10-05T19:28:25.979 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.974+0000 7f78020b4e40 1 -- 172.21.15.167:0/2969922888 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:111 7.d 7:b4812045:::notify.4:head [watch unwatch cookie 93883731531440] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x55630277cd80 con 0x5563027442a0 2023-10-05T19:28:25.982 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.974+0000 7f78003ea700 1 -- 172.21.15.167:0/2969922888 <== osd.1 v2:172.21.15.167:6800/30035 40 ==== osd_op_reply(111 notify.4 [watch unwatch cookie 93883731531440] v30'29 uv27 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f77f0052040 con 0x5563027442a0 2023-10-05T19:28:25.982 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.974+0000 7f78020b4e40 20 remove_watcher() i=4 2023-10-05T19:28:25.982 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.974+0000 7f78020b4e40 1 -- 172.21.15.167:0/2969922888 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:112 7.3 7:c609908c:::notify.5:head [watch unwatch cookie 93883731536224] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x55630277cd80 con 0x7f77e8007030 2023-10-05T19:28:25.984 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.978+0000 7f78003ea700 1 -- 172.21.15.167:0/2969922888 <== osd.0 v2:172.21.15.118:6800/27690 30 ==== osd_op_reply(112 notify.5 [watch unwatch cookie 93883731536224] v30'29 uv27 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f77f0052040 con 0x7f77e8007030 2023-10-05T19:28:25.984 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.978+0000 7f78020b4e40 20 remove_watcher() i=5 2023-10-05T19:28:25.984 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.978+0000 7f78020b4e40 1 -- 172.21.15.167:0/2969922888 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:113 7.14 7:2b04a3e9:::notify.6:head [watch unwatch cookie 93883731538128] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x55630277e960 con 0x7f77e8007030 2023-10-05T19:28:25.987 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.982+0000 7f78003ea700 1 -- 172.21.15.167:0/2969922888 <== osd.0 v2:172.21.15.118:6800/27690 31 ==== osd_op_reply(113 notify.6 [watch unwatch cookie 93883731538128] v30'29 uv27 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f77f0052040 con 0x7f77e8007030 2023-10-05T19:28:25.987 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.982+0000 7f78020b4e40 20 remove_watcher() i=6 2023-10-05T19:28:25.987 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.982+0000 7f78020b4e40 1 -- 172.21.15.167:0/2969922888 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:114 7.9 7:93e5b521:::notify.7:head [watch unwatch cookie 93883731545888] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x55630277cd80 con 0x7f77e8007030 2023-10-05T19:28:25.990 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.982+0000 7f78003ea700 1 -- 172.21.15.167:0/2969922888 <== osd.0 v2:172.21.15.118:6800/27690 32 ==== osd_op_reply(114 notify.7 [watch unwatch cookie 93883731545888] v30'29 uv27 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f77f0052040 con 0x7f77e8007030 2023-10-05T19:28:25.990 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.986+0000 7f78020b4e40 20 remove_watcher() i=7 2023-10-05T19:28:25.991 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.986+0000 7f774bfff700 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2023-10-05T19:28:25.992 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.986+0000 7f78020b4e40 1 -- 172.21.15.167:0/2969922888 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:115 6.4 6:22d26bf9:::data_loggenerations_metadata:head [watch unwatch cookie 93883731541840] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x556302780f20 con 0x5563027442a0 2023-10-05T19:28:25.995 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.990+0000 7f78003ea700 1 -- 172.21.15.167:0/2969922888 <== osd.1 v2:172.21.15.167:6800/30035 41 ==== osd_op_reply(115 data_loggenerations_metadata [watch unwatch cookie 93883731541840] v30'53 uv1 ondisk = 0) v8 ==== 172+0+0 (crc 0 0 0) 0x7f77f0052040 con 0x5563027442a0 2023-10-05T19:28:25.995 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.990+0000 7f78020b4e40 1 -- 172.21.15.167:0/2969922888 >> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] conn(0x7f77e8007030 msgr2=0x7f77e801c920 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-05T19:28:25.995 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.990+0000 7f78020b4e40 1 --2- 172.21.15.167:0/2969922888 >> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] conn(0x7f77e8007030 0x7f77e801c920 crc :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-05T19:28:25.995 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.990+0000 7f78020b4e40 1 -- 172.21.15.167:0/2969922888 >> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] conn(0x5563027442a0 msgr2=0x5563027727d0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-05T19:28:25.995 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.990+0000 7f78020b4e40 1 --2- 172.21.15.167:0/2969922888 >> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] conn(0x5563027442a0 0x5563027727d0 crc :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-05T19:28:25.995 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.990+0000 7f78020b4e40 1 -- 172.21.15.167:0/2969922888 >> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] conn(0x7f77f801a070 msgr2=0x7f77f8042220 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-05T19:28:25.995 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.990+0000 7f78020b4e40 1 --2- 172.21.15.167:0/2969922888 >> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] conn(0x7f77f801a070 0x7f77f8042220 crc :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-05T19:28:25.996 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.990+0000 7f78020b4e40 1 -- 172.21.15.167:0/2969922888 >> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] conn(0x556302776440 msgr2=0x556302778850 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-05T19:28:25.996 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.990+0000 7f78020b4e40 1 --2- 172.21.15.167:0/2969922888 >> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] conn(0x556302776440 0x556302778850 crc :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-05T19:28:25.996 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.990+0000 7f78020b4e40 1 -- 172.21.15.167:0/2969922888 >> [v2:172.21.15.167:6824/29889,v1:172.21.15.167:6825/29889] conn(0x7f77e4030c50 msgr2=0x7f77e4033100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-05T19:28:25.996 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.990+0000 7f78020b4e40 1 --2- 172.21.15.167:0/2969922888 >> [v2:172.21.15.167:6824/29889,v1:172.21.15.167:6825/29889] conn(0x7f77e4030c50 0x7f77e4033100 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f77e80030a0 tx=0x7f77e8015000 comp rx=0 tx=0).stop 2023-10-05T19:28:25.996 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.990+0000 7f78020b4e40 1 -- 172.21.15.167:0/2969922888 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55630273fcc0 msgr2=0x556302766ce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-05T19:28:25.996 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.990+0000 7f78020b4e40 1 --2- 172.21.15.167:0/2969922888 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55630273fcc0 0x556302766ce0 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7f77f8000c00 tx=0x7f77f8016000 comp rx=0 tx=0).stop 2023-10-05T19:28:25.996 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.990+0000 7f78003ea700 1 -- 172.21.15.167:0/2969922888 reap_dead start 2023-10-05T19:28:25.996 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.990+0000 7f78020b4e40 1 -- 172.21.15.167:0/2969922888 shutdown_connections 2023-10-05T19:28:25.996 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.990+0000 7f78020b4e40 1 --2- 172.21.15.167:0/2969922888 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55630273fcc0 0x556302766ce0 unknown :-1 s=CLOSED pgs=191 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-05T19:28:25.996 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.990+0000 7f78020b4e40 1 -- 172.21.15.167:0/2969922888 >> 172.21.15.167:0/2969922888 conn(0x55630273b6e0 msgr2=0x55630275a4d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-05T19:28:25.996 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.990+0000 7f78020b4e40 1 -- 172.21.15.167:0/2969922888 shutdown_connections 2023-10-05T19:28:25.996 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.990+0000 7f78020b4e40 1 -- 172.21.15.167:0/2969922888 wait complete. 2023-10-05T19:28:25.997 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.990+0000 7f78020b4e40 1 -- 172.21.15.167:0/3383862639 >> [v2:172.21.15.167:6824/29889,v1:172.21.15.167:6825/29889] conn(0x7f773c030c00 msgr2=0x7f773c0330b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-05T19:28:25.997 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.990+0000 7f78020b4e40 1 --2- 172.21.15.167:0/3383862639 >> [v2:172.21.15.167:6824/29889,v1:172.21.15.167:6825/29889] conn(0x7f773c030c00 0x7f773c0330b0 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x556302781620 tx=0x7f77f000aec0 comp rx=0 tx=0).stop 2023-10-05T19:28:25.998 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.990+0000 7f78020b4e40 1 -- 172.21.15.167:0/3383862639 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55630278eb00 msgr2=0x5563027495a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-05T19:28:25.998 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.990+0000 7f78020b4e40 1 --2- 172.21.15.167:0/3383862639 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55630278eb00 0x5563027495a0 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x556302788b80 tx=0x7f77e802f000 comp rx=0 tx=0).stop 2023-10-05T19:28:25.998 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.990+0000 7f78020b4e40 1 -- 172.21.15.167:0/3383862639 shutdown_connections 2023-10-05T19:28:25.998 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.990+0000 7f78020b4e40 1 --2- 172.21.15.167:0/3383862639 >> [v2:172.21.15.167:6824/29889,v1:172.21.15.167:6825/29889] conn(0x7f773c030c00 0x7f773c0330b0 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-05T19:28:25.998 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.990+0000 7f78020b4e40 1 --2- 172.21.15.167:0/3383862639 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55630278eb00 0x5563027495a0 unknown :-1 s=CLOSED pgs=193 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-05T19:28:25.998 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.990+0000 7f78020b4e40 1 -- 172.21.15.167:0/3383862639 >> 172.21.15.167:0/3383862639 conn(0x556302789ec0 msgr2=0x556302787f60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-05T19:28:25.998 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.990+0000 7f78020b4e40 1 -- 172.21.15.167:0/3383862639 shutdown_connections 2023-10-05T19:28:25.998 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:25.990+0000 7f78020b4e40 1 -- 172.21.15.167:0/3383862639 wait complete. 2023-10-05T19:28:26.002 DEBUG:teuthology.orchestra.run:got remote process result: 240 2023-10-05T19:28:26.003 INFO:tasks.util.rgw:rgwadmin: client.0 : ['user', 'info', '--uid', 'foo'] 2023-10-05T19:28:26.003 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'ceph', 'user', 'info', '--uid', 'foo'] 2023-10-05T19:28:26.003 DEBUG:teuthology.orchestra.run.smithi167:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster ceph user info --uid foo 2023-10-05T19:28:26.038 INFO:teuthology.orchestra.run.smithi167.stderr:ignoring --setuser ceph since I am not root 2023-10-05T19:28:26.038 INFO:teuthology.orchestra.run.smithi167.stderr:ignoring --setgroup ceph since I am not root 2023-10-05T19:28:26.039 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.034+0000 7f90cfe1ae40 1 Processor -- start 2023-10-05T19:28:26.040 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.034+0000 7f90cfe1ae40 1 -- start start 2023-10-05T19:28:26.040 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.034+0000 7f90cfe1ae40 1 --2- >> v2:172.21.15.118:3300/0 conn(0x559e90e6d020 0x559e90e6d3f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-05T19:28:26.040 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.034+0000 7f90cfe1ae40 1 -- --> v1:172.21.15.118:6789/0 -- auth(proto 0 26 bytes epoch 0) v1 -- 0x559e90db7bd0 con 0x559e90e6d930 2023-10-05T19:28:26.040 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.034+0000 7f90cfe1ae40 1 -- --> v2:172.21.15.118:3300/0 -- mon_getmap magic: 0 v1 -- 0x559e90d74ee0 con 0x559e90e6d020 2023-10-05T19:28:26.041 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.034+0000 7f90cd94f700 1 --2- >> v2:172.21.15.118:3300/0 conn(0x559e90e6d020 0x559e90e6d3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-05T19:28:26.041 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.034+0000 7f90cd14e700 1 --1- >> v1:172.21.15.118:6789/0 conn(0x559e90e6d930 0x559e90e6e3d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.118:6789/0 says I am v1:172.21.15.167:56372/0 (socket says 172.21.15.167:56372) 2023-10-05T19:28:26.041 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.034+0000 7f90cd94f700 1 --2- >> v2:172.21.15.118:3300/0 conn(0x559e90e6d020 0x559e90e6d3f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.118:3300/0 says I am v2:172.21.15.167:57248/0 (socket says 172.21.15.167:57248) 2023-10-05T19:28:26.041 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.034+0000 7f90cd14e700 1 -- 172.21.15.167:0/2925547227 learned_addr learned my addr 172.21.15.167:0/2925547227 (peer_addr_for_me v1:172.21.15.167:0/0) 2023-10-05T19:28:26.041 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.034+0000 7f90cd94f700 1 -- 172.21.15.167:0/2925547227 >> v1:172.21.15.118:6789/0 conn(0x559e90e6d930 legacy=0x559e90e6e3d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-05T19:28:26.041 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.034+0000 7f90cd94f700 1 -- 172.21.15.167:0/2925547227 --> v2:172.21.15.118:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x559e90e6eae0 con 0x559e90e6d020 2023-10-05T19:28:26.042 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.034+0000 7f90cd94f700 1 --2- 172.21.15.167:0/2925547227 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x559e90e6d020 0x559e90e6d3f0 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7f90c000a5d0 tx=0x7f90c0005a50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=321ffcb05b8e133c server_cookie=0 in_seq=0 out_seq=0 2023-10-05T19:28:26.043 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.034+0000 7f90cc94d700 1 -- 172.21.15.167:0/2925547227 <== mon.0 v2:172.21.15.118:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f90c00086a0 con 0x559e90e6d020 2023-10-05T19:28:26.043 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.034+0000 7f90cc94d700 1 -- 172.21.15.167:0/2925547227 <== mon.0 v2:172.21.15.118:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f90c0008800 con 0x559e90e6d020 2023-10-05T19:28:26.043 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.034+0000 7f90cc94d700 1 -- 172.21.15.167:0/2925547227 <== mon.0 v2:172.21.15.118:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f90c0008ad0 con 0x559e90e6d020 2023-10-05T19:28:26.043 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.038+0000 7f90cfe1ae40 1 -- 172.21.15.167:0/2925547227 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x559e90e6d020 msgr2=0x559e90e6d3f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-05T19:28:26.043 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.038+0000 7f90cfe1ae40 1 --2- 172.21.15.167:0/2925547227 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x559e90e6d020 0x559e90e6d3f0 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7f90c000a5d0 tx=0x7f90c0005a50 comp rx=0 tx=0).stop 2023-10-05T19:28:26.043 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.038+0000 7f90cfe1ae40 1 -- 172.21.15.167:0/2925547227 shutdown_connections 2023-10-05T19:28:26.043 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.038+0000 7f90cfe1ae40 1 --2- 172.21.15.167:0/2925547227 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x559e90e6d020 0x559e90e6d3f0 unknown :-1 s=CLOSED pgs=195 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-05T19:28:26.043 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.038+0000 7f90cfe1ae40 1 -- 172.21.15.167:0/2925547227 >> 172.21.15.167:0/2925547227 conn(0x559e90de0870 msgr2=0x559e90e6b510 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-05T19:28:26.043 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.038+0000 7f90cfe1ae40 1 -- 172.21.15.167:0/2925547227 shutdown_connections 2023-10-05T19:28:26.043 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.038+0000 7f90cfe1ae40 1 -- 172.21.15.167:0/2925547227 wait complete. 2023-10-05T19:28:26.044 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.038+0000 7f90cfe1ae40 1 Processor -- start 2023-10-05T19:28:26.044 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.038+0000 7f90cfe1ae40 1 -- start start 2023-10-05T19:28:26.045 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.038+0000 7f90cfe1ae40 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x559e90e6d020 0x559e90e8c0f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-05T19:28:26.045 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.038+0000 7f90cfe1ae40 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x559e90d38e30 con 0x559e90e6d020 2023-10-05T19:28:26.045 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.038+0000 7f90cd94f700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x559e90e6d020 0x559e90e8c0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-05T19:28:26.045 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.038+0000 7f90cd94f700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x559e90e6d020 0x559e90e8c0f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.118:3300/0 says I am v2:172.21.15.167:57260/0 (socket says 172.21.15.167:57260) 2023-10-05T19:28:26.045 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.038+0000 7f90cd94f700 1 -- 172.21.15.167:0/1690724709 learned_addr learned my addr 172.21.15.167:0/1690724709 (peer_addr_for_me v2:172.21.15.167:0/0) 2023-10-05T19:28:26.045 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.038+0000 7f90cd94f700 1 -- 172.21.15.167:0/1690724709 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f90c0005cf0 con 0x559e90e6d020 2023-10-05T19:28:26.046 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.038+0000 7f90cd94f700 1 --2- 172.21.15.167:0/1690724709 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x559e90e6d020 0x559e90e8c0f0 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7f90c0000c00 tx=0x7f90c0015000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c74f40192ab8af28 server_cookie=0 in_seq=0 out_seq=0 2023-10-05T19:28:26.046 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.038+0000 7f90c77fe700 1 -- 172.21.15.167:0/1690724709 <== mon.0 v2:172.21.15.118:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f90c0008970 con 0x559e90e6d020 2023-10-05T19:28:26.046 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.038+0000 7f90c77fe700 1 -- 172.21.15.167:0/1690724709 <== mon.0 v2:172.21.15.118:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f90c0008af0 con 0x559e90e6d020 2023-10-05T19:28:26.046 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.038+0000 7f90c77fe700 1 -- 172.21.15.167:0/1690724709 <== mon.0 v2:172.21.15.118:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f90c0008de0 con 0x559e90e6d020 2023-10-05T19:28:26.047 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.042+0000 7f90cfe1ae40 1 -- 172.21.15.167:0/1690724709 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x559e90e6d020 msgr2=0x559e90e8c0f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-05T19:28:26.047 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.042+0000 7f90cfe1ae40 1 --2- 172.21.15.167:0/1690724709 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x559e90e6d020 0x559e90e8c0f0 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7f90c0000c00 tx=0x7f90c0015000 comp rx=0 tx=0).stop 2023-10-05T19:28:26.047 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.042+0000 7f90cfe1ae40 1 -- 172.21.15.167:0/1690724709 shutdown_connections 2023-10-05T19:28:26.047 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.042+0000 7f90cfe1ae40 1 --2- 172.21.15.167:0/1690724709 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x559e90e6d020 0x559e90e8c0f0 unknown :-1 s=CLOSED pgs=196 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-05T19:28:26.047 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.042+0000 7f90cfe1ae40 1 -- 172.21.15.167:0/1690724709 >> 172.21.15.167:0/1690724709 conn(0x559e90de0870 msgr2=0x559e90e87cb0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-05T19:28:26.047 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.042+0000 7f90cfe1ae40 1 -- 172.21.15.167:0/1690724709 shutdown_connections 2023-10-05T19:28:26.047 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.042+0000 7f90cfe1ae40 1 -- 172.21.15.167:0/1690724709 wait complete. 2023-10-05T19:28:26.048 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.042+0000 7f90cfe1ae40 1 Processor -- start 2023-10-05T19:28:26.048 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.042+0000 7f90cfe1ae40 1 -- start start 2023-10-05T19:28:26.048 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.042+0000 7f90cfe1ae40 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x559e90e6d020 0x559e90e92d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-05T19:28:26.048 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.042+0000 7f90cfe1ae40 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x559e90d0bed0 con 0x559e90e6d020 2023-10-05T19:28:26.049 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.042+0000 7f90cd94f700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x559e90e6d020 0x559e90e92d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-05T19:28:26.049 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.042+0000 7f90cd94f700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x559e90e6d020 0x559e90e92d00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.118:3300/0 says I am v2:172.21.15.167:57262/0 (socket says 172.21.15.167:57262) 2023-10-05T19:28:26.049 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.042+0000 7f90cd94f700 1 -- 172.21.15.167:0/3718246546 learned_addr learned my addr 172.21.15.167:0/3718246546 (peer_addr_for_me v2:172.21.15.167:0/0) 2023-10-05T19:28:26.050 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.042+0000 7f90cd94f700 1 -- 172.21.15.167:0/3718246546 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x559e90e6eae0 con 0x559e90e6d020 2023-10-05T19:28:26.050 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.042+0000 7f90cd94f700 1 --2- 172.21.15.167:0/3718246546 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x559e90e6d020 0x559e90e92d00 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7f90c000c010 tx=0x7f90c0003050 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-05T19:28:26.050 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.046+0000 7f90c5ffb700 1 -- 172.21.15.167:0/3718246546 <== mon.0 v2:172.21.15.118:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f90c0012ba0 con 0x559e90e6d020 2023-10-05T19:28:26.051 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.046+0000 7f90c5ffb700 1 -- 172.21.15.167:0/3718246546 <== mon.0 v2:172.21.15.118:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f90c002a070 con 0x559e90e6d020 2023-10-05T19:28:26.051 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.046+0000 7f90cfe1ae40 1 -- 172.21.15.167:0/3718246546 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x559e90e93240 con 0x559e90e6d020 2023-10-05T19:28:26.051 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.046+0000 7f90c5ffb700 1 -- 172.21.15.167:0/3718246546 <== mon.0 v2:172.21.15.118:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f90c0012ea0 con 0x559e90e6d020 2023-10-05T19:28:26.051 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.046+0000 7f90cfe1ae40 1 -- 172.21.15.167:0/3718246546 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x559e90e936e0 con 0x559e90e6d020 2023-10-05T19:28:26.052 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.046+0000 7f90c5ffb700 1 -- 172.21.15.167:0/3718246546 <== mon.0 v2:172.21.15.118:3300/0 4 ==== mgrmap(e 5) v1 ==== 40329+0+0 (secure 0 0 0) 0x7f90c0019070 con 0x559e90e6d020 2023-10-05T19:28:26.053 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.046+0000 7f90cfe1ae40 20 rados->read ofs=0 len=0 2023-10-05T19:28:26.053 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.046+0000 7f90c5ffb700 1 --2- 172.21.15.167:0/3718246546 >> [v2:172.21.15.167:6824/29889,v1:172.21.15.167:6825/29889] conn(0x7f90b0030bf0 0x7f90b00330a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-05T19:28:26.053 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.046+0000 7f90c5ffb700 1 -- 172.21.15.167:0/3718246546 <== mon.0 v2:172.21.15.118:3300/0 5 ==== osd_map(30..30 src has 1..30) v4 ==== 6057+0+0 (secure 0 0 0) 0x7f90c002c070 con 0x559e90e6d020 2023-10-05T19:28:26.053 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.046+0000 7f90cd14e700 1 --2- 172.21.15.167:0/3718246546 >> [v2:172.21.15.167:6824/29889,v1:172.21.15.167:6825/29889] conn(0x7f90b0030bf0 0x7f90b00330a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-05T19:28:26.053 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.046+0000 7f90cfe1ae40 1 --2- 172.21.15.167:0/3718246546 >> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] conn(0x559e90e6d930 0x559e90e9e770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-05T19:28:26.053 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.046+0000 7f90cfe1ae40 1 -- 172.21.15.167:0/3718246546 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:1 5.12 5:49953fa1:::default.realm:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x559e90e9ee20 con 0x559e90e6d930 2023-10-05T19:28:26.054 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.046+0000 7f90ce150700 1 --2- 172.21.15.167:0/3718246546 >> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] conn(0x559e90e6d930 0x559e90e9e770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-05T19:28:26.054 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.046+0000 7f90cd14e700 1 --2- 172.21.15.167:0/3718246546 >> [v2:172.21.15.167:6824/29889,v1:172.21.15.167:6825/29889] conn(0x7f90b0030bf0 0x7f90b00330a0 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f90b80016f0 tx=0x7f90b80067f0 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-05T19:28:26.054 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.050+0000 7f90ce150700 1 --2- 172.21.15.167:0/3718246546 >> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] conn(0x559e90e6d930 0x559e90e9e770 crc :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-05T19:28:26.055 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.050+0000 7f90ce150700 1 -- 172.21.15.167:0/3718246546 <== osd.1 v2:172.21.15.167:6800/30035 1 ==== osd_op_reply(1 default.realm [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 ==== 157+0+0 (crc 0 0 0) 0x7f90c800a500 con 0x559e90e6d930 2023-10-05T19:28:26.055 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.050+0000 7f90cfe1ae40 20 rados_obj.operate() r=-2 bl.length=0 2023-10-05T19:28:26.055 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.050+0000 7f90cfe1ae40 20 realm 2023-10-05T19:28:26.055 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.050+0000 7f90cfe1ae40 20 rados->read ofs=0 len=0 2023-10-05T19:28:26.055 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.050+0000 7f90cfe1ae40 1 -- 172.21.15.167:0/3718246546 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:2 5.12 5:49953fa1:::default.realm:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x559e90e9bdd0 con 0x559e90e6d930 2023-10-05T19:28:26.056 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.050+0000 7f90ce150700 1 -- 172.21.15.167:0/3718246546 <== osd.1 v2:172.21.15.167:6800/30035 2 ==== osd_op_reply(2 default.realm [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 ==== 157+0+0 (crc 0 0 0) 0x7f90c800a500 con 0x559e90e6d930 2023-10-05T19:28:26.056 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.050+0000 7f90cfe1ae40 20 rados_obj.operate() r=-2 bl.length=0 2023-10-05T19:28:26.056 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.050+0000 7f90cfe1ae40 4 RGWPeriod::init failed to init realm id : (2) No such file or directory 2023-10-05T19:28:26.056 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.050+0000 7f90cfe1ae40 20 rados->read ofs=0 len=0 2023-10-05T19:28:26.056 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.050+0000 7f90cfe1ae40 1 -- 172.21.15.167:0/3718246546 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:3 5.12 5:49953fa1:::default.realm:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x559e90e9f900 con 0x559e90e6d930 2023-10-05T19:28:26.057 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.050+0000 7f90ce150700 1 -- 172.21.15.167:0/3718246546 <== osd.1 v2:172.21.15.167:6800/30035 3 ==== osd_op_reply(3 default.realm [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 ==== 157+0+0 (crc 0 0 0) 0x7f90c800a500 con 0x559e90e6d930 2023-10-05T19:28:26.057 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.050+0000 7f90cfe1ae40 20 rados_obj.operate() r=-2 bl.length=0 2023-10-05T19:28:26.057 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.050+0000 7f90cfe1ae40 20 rados->read ofs=0 len=0 2023-10-05T19:28:26.057 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.050+0000 7f90cfe1ae40 1 -- 172.21.15.167:0/3718246546 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:4 5.f 5:f43b6ece:::zone_names.default:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x559e90e9f4e0 con 0x559e90e6d930 2023-10-05T19:28:26.057 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.050+0000 7f90ce150700 1 -- 172.21.15.167:0/3718246546 <== osd.1 v2:172.21.15.167:6800/30035 4 ==== osd_op_reply(4 zone_names.default [read 0~46 out=46b] v0'0 uv1 ondisk = 0) v8 ==== 162+0+46 (crc 0 0 0) 0x7f90c800a500 con 0x559e90e6d930 2023-10-05T19:28:26.057 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.050+0000 7f90cfe1ae40 20 rados_obj.operate() r=0 bl.length=46 2023-10-05T19:28:26.058 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.050+0000 7f90cfe1ae40 20 rados->read ofs=0 len=0 2023-10-05T19:28:26.058 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.050+0000 7f90cfe1ae40 1 -- 172.21.15.167:0/3718246546 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:5 5.1d 5:b8374eb4:::zone_info.9a9df965-6612-45d4-a45f-6e3c889bacc2:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x559e90ea0370 con 0x559e90e6d930 2023-10-05T19:28:26.058 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.050+0000 7f90ce150700 1 -- 172.21.15.167:0/3718246546 <== osd.1 v2:172.21.15.167:6800/30035 5 ==== osd_op_reply(5 zone_info.9a9df965-6612-45d4-a45f-6e3c889bacc2 [read 0~874 out=874b] v0'0 uv2 ondisk = 0) v8 ==== 190+0+874 (crc 0 0 0) 0x7f90c800a500 con 0x559e90e6d930 2023-10-05T19:28:26.059 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.050+0000 7f90cfe1ae40 20 rados_obj.operate() r=0 bl.length=874 2023-10-05T19:28:26.059 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.050+0000 7f90cfe1ae40 20 searching for the correct realm 2023-10-05T19:28:26.059 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.050+0000 7f90cfe1ae40 1 --2- 172.21.15.167:0/3718246546 >> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] conn(0x559e90ea21c0 0x559e90ea45a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-05T19:28:26.059 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.050+0000 7f90cfe1ae40 1 -- 172.21.15.167:0/3718246546 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:6 5.0 5:00000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x559e90ea4ae0 con 0x559e90ea21c0 2023-10-05T19:28:26.059 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.054+0000 7f90cd94f700 1 --2- 172.21.15.167:0/3718246546 >> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] conn(0x559e90ea21c0 0x559e90ea45a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-05T19:28:26.059 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.054+0000 7f90cd94f700 1 --2- 172.21.15.167:0/3718246546 >> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] conn(0x559e90ea21c0 0x559e90ea45a0 crc :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-05T19:28:26.060 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.054+0000 7f90cd94f700 1 -- 172.21.15.167:0/3718246546 <== osd.3 v2:172.21.15.167:6808/30036 1 ==== osd_op_reply(6 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f90c0022020 con 0x559e90ea21c0 2023-10-05T19:28:26.060 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.054+0000 7f90cd94f700 1 --2- 172.21.15.167:0/3718246546 >> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] conn(0x7f90c0007030 0x7f90c00249c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-05T19:28:26.060 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.054+0000 7f90cd94f700 1 -- 172.21.15.167:0/3718246546 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:7 5.10 5:08000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x559e90ea4ae0 con 0x7f90c0007030 2023-10-05T19:28:26.060 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.054+0000 7f90cd14e700 1 --2- 172.21.15.167:0/3718246546 >> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] conn(0x7f90c0007030 0x7f90c00249c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-05T19:28:26.061 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.054+0000 7f90cd14e700 1 --2- 172.21.15.167:0/3718246546 >> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] conn(0x7f90c0007030 0x7f90c00249c0 crc :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-05T19:28:26.061 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.054+0000 7f90cd14e700 1 -- 172.21.15.167:0/3718246546 <== osd.2 v2:172.21.15.167:6816/30037 1 ==== osd_op_reply(7 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f90b80060d0 con 0x7f90c0007030 2023-10-05T19:28:26.061 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.054+0000 7f90cd14e700 1 --2- 172.21.15.167:0/3718246546 >> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] conn(0x7f90b8011040 0x7f90b8014950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-05T19:28:26.061 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.054+0000 7f90cd14e700 1 -- 172.21.15.167:0/3718246546 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:8 5.8 5:10000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x559e90ea4ae0 con 0x7f90b8011040 2023-10-05T19:28:26.062 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.054+0000 7f90ce150700 1 --2- 172.21.15.167:0/3718246546 >> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] conn(0x7f90b8011040 0x7f90b8014950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-05T19:28:26.062 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.058+0000 7f90ce150700 1 --2- 172.21.15.167:0/3718246546 >> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] conn(0x7f90b8011040 0x7f90b8014950 crc :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-05T19:28:26.063 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.058+0000 7f90ce150700 1 -- 172.21.15.167:0/3718246546 <== osd.0 v2:172.21.15.118:6800/27690 1 ==== osd_op_reply(8 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f90c8014040 con 0x7f90b8011040 2023-10-05T19:28:26.063 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.058+0000 7f90ce150700 1 -- 172.21.15.167:0/3718246546 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:9 5.18 5:18000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x559e90ea4ae0 con 0x559e90e6d930 2023-10-05T19:28:26.064 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.058+0000 7f90ce150700 1 -- 172.21.15.167:0/3718246546 <== osd.1 v2:172.21.15.167:6800/30035 6 ==== osd_op_reply(9 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f90c8014040 con 0x559e90e6d930 2023-10-05T19:28:26.064 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.058+0000 7f90ce150700 1 -- 172.21.15.167:0/3718246546 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:10 5.4 5:20000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x559e90ea4ae0 con 0x559e90e6d930 2023-10-05T19:28:26.065 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.058+0000 7f90ce150700 1 -- 172.21.15.167:0/3718246546 <== osd.1 v2:172.21.15.167:6800/30035 7 ==== osd_op_reply(10 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f90c8014040 con 0x559e90e6d930 2023-10-05T19:28:26.065 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.058+0000 7f90ce150700 1 -- 172.21.15.167:0/3718246546 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:11 5.14 5:28000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x559e90ea4ae0 con 0x7f90b8011040 2023-10-05T19:28:26.066 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.062+0000 7f90ce150700 1 -- 172.21.15.167:0/3718246546 <== osd.0 v2:172.21.15.118:6800/27690 2 ==== osd_op_reply(11 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f90c8014040 con 0x7f90b8011040 2023-10-05T19:28:26.066 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.062+0000 7f90ce150700 1 -- 172.21.15.167:0/3718246546 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:12 5.c 5:30000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x559e90ea4ae0 con 0x7f90b8011040 2023-10-05T19:28:26.067 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.062+0000 7f90ce150700 1 -- 172.21.15.167:0/3718246546 <== osd.0 v2:172.21.15.118:6800/27690 3 ==== osd_op_reply(12 [pgnls start_epoch 30 out=112b] v23'1 uv1 ondisk = 1) v8 ==== 144+0+112 (crc 0 0 0) 0x7f90c8014040 con 0x7f90b8011040 2023-10-05T19:28:26.067 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.062+0000 7f90ce150700 1 -- 172.21.15.167:0/3718246546 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:13 5.1c 5:38000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x559e90ea4ae0 con 0x7f90c0007030 2023-10-05T19:28:26.067 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.062+0000 7f90cd14e700 1 -- 172.21.15.167:0/3718246546 <== osd.2 v2:172.21.15.167:6816/30037 2 ==== osd_op_reply(13 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f90b8004b80 con 0x7f90c0007030 2023-10-05T19:28:26.067 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.062+0000 7f90cd14e700 1 -- 172.21.15.167:0/3718246546 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:14 5.2 5:40000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x559e90ea4ae0 con 0x559e90ea21c0 2023-10-05T19:28:26.068 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.062+0000 7f90cd94f700 1 -- 172.21.15.167:0/3718246546 <== osd.3 v2:172.21.15.167:6808/30036 2 ==== osd_op_reply(14 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f90c0022020 con 0x559e90ea21c0 2023-10-05T19:28:26.068 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.062+0000 7f90cd94f700 1 -- 172.21.15.167:0/3718246546 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:15 5.12 5:48000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x559e90ea4ae0 con 0x559e90e6d930 2023-10-05T19:28:26.068 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.062+0000 7f90ce150700 1 -- 172.21.15.167:0/3718246546 <== osd.1 v2:172.21.15.167:6800/30035 8 ==== osd_op_reply(15 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f90c8014040 con 0x559e90e6d930 2023-10-05T19:28:26.068 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.062+0000 7f90ce150700 1 -- 172.21.15.167:0/3718246546 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:16 5.a 5:50000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x559e90ea4ae0 con 0x7f90b8011040 2023-10-05T19:28:26.069 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.062+0000 7f90ce150700 1 -- 172.21.15.167:0/3718246546 <== osd.0 v2:172.21.15.118:6800/27690 4 ==== osd_op_reply(16 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f90c8022040 con 0x7f90b8011040 2023-10-05T19:28:26.069 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.062+0000 7f90ce150700 1 -- 172.21.15.167:0/3718246546 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:17 5.1a 5:58000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x559e90ea4ae0 con 0x559e90e6d930 2023-10-05T19:28:26.070 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.066+0000 7f90ce150700 1 -- 172.21.15.167:0/3718246546 <== osd.1 v2:172.21.15.167:6800/30035 9 ==== osd_op_reply(17 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f90c8022040 con 0x559e90e6d930 2023-10-05T19:28:26.070 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.066+0000 7f90ce150700 1 -- 172.21.15.167:0/3718246546 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:18 5.6 5:60000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x559e90ea4ae0 con 0x7f90c0007030 2023-10-05T19:28:26.070 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.066+0000 7f90cd14e700 1 -- 172.21.15.167:0/3718246546 <== osd.2 v2:172.21.15.167:6816/30037 3 ==== osd_op_reply(18 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f90b8004b80 con 0x7f90c0007030 2023-10-05T19:28:26.070 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.066+0000 7f90cd14e700 1 -- 172.21.15.167:0/3718246546 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:19 5.16 5:68000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x559e90ea4ae0 con 0x559e90ea21c0 2023-10-05T19:28:26.071 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.066+0000 7f90cd94f700 1 -- 172.21.15.167:0/3718246546 <== osd.3 v2:172.21.15.167:6808/30036 3 ==== osd_op_reply(19 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f90c0022020 con 0x559e90ea21c0 2023-10-05T19:28:26.071 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.066+0000 7f90cd94f700 1 -- 172.21.15.167:0/3718246546 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:20 5.e 5:70000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x559e90ea4ae0 con 0x559e90ea21c0 2023-10-05T19:28:26.071 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.066+0000 7f90cd94f700 1 -- 172.21.15.167:0/3718246546 <== osd.3 v2:172.21.15.167:6808/30036 4 ==== osd_op_reply(20 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f90c0022020 con 0x559e90ea21c0 2023-10-05T19:28:26.072 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.066+0000 7f90cd94f700 1 -- 172.21.15.167:0/3718246546 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:21 5.1e 5:78000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x559e90ea4ae0 con 0x7f90b8011040 2023-10-05T19:28:26.072 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.066+0000 7f90ce150700 1 -- 172.21.15.167:0/3718246546 <== osd.0 v2:172.21.15.118:6800/27690 5 ==== osd_op_reply(21 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f90c8022040 con 0x7f90b8011040 2023-10-05T19:28:26.073 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.066+0000 7f90ce150700 1 -- 172.21.15.167:0/3718246546 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:22 5.1 5:80000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x559e90ea4ae0 con 0x559e90ea21c0 2023-10-05T19:28:26.073 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.066+0000 7f90cd94f700 1 -- 172.21.15.167:0/3718246546 <== osd.3 v2:172.21.15.167:6808/30036 5 ==== osd_op_reply(22 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f90c0022020 con 0x559e90ea21c0 2023-10-05T19:28:26.073 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.066+0000 7f90cd94f700 1 -- 172.21.15.167:0/3718246546 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:23 5.11 5:88000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x559e90ea4ae0 con 0x559e90ea21c0 2023-10-05T19:28:26.074 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.070+0000 7f90cd94f700 1 -- 172.21.15.167:0/3718246546 <== osd.3 v2:172.21.15.167:6808/30036 6 ==== osd_op_reply(23 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f90c0022020 con 0x559e90ea21c0 2023-10-05T19:28:26.074 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.070+0000 7f90cd94f700 1 -- 172.21.15.167:0/3718246546 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:24 5.9 5:90000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x559e90ea4ae0 con 0x559e90e6d930 2023-10-05T19:28:26.074 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.070+0000 7f90ce150700 1 -- 172.21.15.167:0/3718246546 <== osd.1 v2:172.21.15.167:6800/30035 10 ==== osd_op_reply(24 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f90c8022040 con 0x559e90e6d930 2023-10-05T19:28:26.075 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.070+0000 7f90ce150700 1 -- 172.21.15.167:0/3718246546 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:25 5.19 5:98000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x559e90ea4ae0 con 0x559e90e6d930 2023-10-05T19:28:26.075 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.070+0000 7f90ce150700 1 -- 172.21.15.167:0/3718246546 <== osd.1 v2:172.21.15.167:6800/30035 11 ==== osd_op_reply(25 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f90c8022040 con 0x559e90e6d930 2023-10-05T19:28:26.075 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.070+0000 7f90ce150700 1 -- 172.21.15.167:0/3718246546 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:26 5.5 5:a0000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x559e90ea4ae0 con 0x7f90b8011040 2023-10-05T19:28:26.076 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.070+0000 7f90ce150700 1 -- 172.21.15.167:0/3718246546 <== osd.0 v2:172.21.15.118:6800/27690 6 ==== osd_op_reply(26 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f90c8022040 con 0x7f90b8011040 2023-10-05T19:28:26.076 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.070+0000 7f90ce150700 1 -- 172.21.15.167:0/3718246546 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:27 5.15 5:a8000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x559e90ea4ae0 con 0x559e90ea21c0 2023-10-05T19:28:26.077 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.070+0000 7f90cd94f700 1 -- 172.21.15.167:0/3718246546 <== osd.3 v2:172.21.15.167:6808/30036 7 ==== osd_op_reply(27 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f90c0022020 con 0x559e90ea21c0 2023-10-05T19:28:26.077 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.070+0000 7f90cd94f700 1 -- 172.21.15.167:0/3718246546 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:28 5.d 5:b0000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x559e90ea4ae0 con 0x7f90c0007030 2023-10-05T19:28:26.078 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.070+0000 7f90cd14e700 1 -- 172.21.15.167:0/3718246546 <== osd.2 v2:172.21.15.167:6816/30037 4 ==== osd_op_reply(28 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f90b8004b80 con 0x7f90c0007030 2023-10-05T19:28:26.078 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.070+0000 7f90cd14e700 1 -- 172.21.15.167:0/3718246546 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:29 5.1d 5:b8000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x559e90ea4ae0 con 0x559e90e6d930 2023-10-05T19:28:26.078 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.074+0000 7f90ce150700 1 -- 172.21.15.167:0/3718246546 <== osd.1 v2:172.21.15.167:6800/30035 12 ==== osd_op_reply(29 [pgnls start_epoch 30 out=107b] v30'2 uv2 ondisk = 1) v8 ==== 144+0+107 (crc 0 0 0) 0x7f90c8022040 con 0x559e90e6d930 2023-10-05T19:28:26.079 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.074+0000 7f90ce150700 1 -- 172.21.15.167:0/3718246546 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:30 5.3 5:c0000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x559e90ea4ae0 con 0x7f90b8011040 2023-10-05T19:28:26.079 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.074+0000 7f90ce150700 1 -- 172.21.15.167:0/3718246546 <== osd.0 v2:172.21.15.118:6800/27690 7 ==== osd_op_reply(30 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f90c8022040 con 0x7f90b8011040 2023-10-05T19:28:26.079 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.074+0000 7f90ce150700 1 -- 172.21.15.167:0/3718246546 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:31 5.13 5:c8000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x559e90ea4ae0 con 0x559e90ea21c0 2023-10-05T19:28:26.080 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.074+0000 7f90cd94f700 1 -- 172.21.15.167:0/3718246546 <== osd.3 v2:172.21.15.167:6808/30036 8 ==== osd_op_reply(31 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f90c0022020 con 0x559e90ea21c0 2023-10-05T19:28:26.080 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.074+0000 7f90cd94f700 1 -- 172.21.15.167:0/3718246546 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:32 5.b 5:d0000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x559e90ea4ae0 con 0x7f90b8011040 2023-10-05T19:28:26.081 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.074+0000 7f90ce150700 1 -- 172.21.15.167:0/3718246546 <== osd.0 v2:172.21.15.118:6800/27690 8 ==== osd_op_reply(32 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f90c8022040 con 0x7f90b8011040 2023-10-05T19:28:26.081 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.074+0000 7f90ce150700 1 -- 172.21.15.167:0/3718246546 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:33 5.1b 5:d8000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x559e90ea4ae0 con 0x7f90c0007030 2023-10-05T19:28:26.082 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.078+0000 7f90cd14e700 1 -- 172.21.15.167:0/3718246546 <== osd.2 v2:172.21.15.167:6816/30037 5 ==== osd_op_reply(33 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f90b8004b80 con 0x7f90c0007030 2023-10-05T19:28:26.082 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.078+0000 7f90cd14e700 1 -- 172.21.15.167:0/3718246546 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:34 5.7 5:e0000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x559e90ea4ae0 con 0x559e90ea21c0 2023-10-05T19:28:26.082 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.078+0000 7f90cd94f700 1 -- 172.21.15.167:0/3718246546 <== osd.3 v2:172.21.15.167:6808/30036 9 ==== osd_op_reply(34 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f90c0022020 con 0x559e90ea21c0 2023-10-05T19:28:26.083 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.078+0000 7f90cd94f700 1 -- 172.21.15.167:0/3718246546 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:35 5.17 5:e8000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x559e90ea4ae0 con 0x7f90b8011040 2023-10-05T19:28:26.083 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.078+0000 7f90ce150700 1 -- 172.21.15.167:0/3718246546 <== osd.0 v2:172.21.15.118:6800/27690 9 ==== osd_op_reply(35 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f90c8022040 con 0x7f90b8011040 2023-10-05T19:28:26.084 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.078+0000 7f90ce150700 1 -- 172.21.15.167:0/3718246546 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:36 5.f 5:f0000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x559e90ea4ae0 con 0x559e90e6d930 2023-10-05T19:28:26.084 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.078+0000 7f90ce150700 1 -- 172.21.15.167:0/3718246546 <== osd.1 v2:172.21.15.167:6800/30035 13 ==== osd_op_reply(36 [pgnls start_epoch 30 out=115b] v23'2 uv2 ondisk = 1) v8 ==== 144+0+115 (crc 0 0 0) 0x7f90c802f080 con 0x559e90e6d930 2023-10-05T19:28:26.084 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.078+0000 7f90ce150700 1 -- 172.21.15.167:0/3718246546 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:37 5.1f 5:f8000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x559e90ea4ae0 con 0x7f90c0007030 2023-10-05T19:28:26.085 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.078+0000 7f90cd14e700 1 -- 172.21.15.167:0/3718246546 <== osd.2 v2:172.21.15.167:6816/30037 6 ==== osd_op_reply(37 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f90b8004b80 con 0x7f90c0007030 2023-10-05T19:28:26.086 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.078+0000 7f90cfe1ae40 20 RGWRados::pool_iterate: got zonegroup_info.4f1f719d-9648-417b-b10c-e5d939825419 2023-10-05T19:28:26.086 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.078+0000 7f90cfe1ae40 20 RGWRados::pool_iterate: got zone_info.9a9df965-6612-45d4-a45f-6e3c889bacc2 2023-10-05T19:28:26.086 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.078+0000 7f90cfe1ae40 20 RGWRados::pool_iterate: got zone_names.default 2023-10-05T19:28:26.086 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.078+0000 7f90cfe1ae40 20 RGWRados::pool_iterate: got zonegroups_names.default 2023-10-05T19:28:26.086 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.078+0000 7f90cfe1ae40 20 rados->read ofs=0 len=0 2023-10-05T19:28:26.086 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.078+0000 7f90cfe1ae40 1 -- 172.21.15.167:0/3718246546 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:38 5.12 5:49953fa1:::default.realm:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x559e90ea0f70 con 0x559e90e6d930 2023-10-05T19:28:26.087 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.082+0000 7f90ce150700 1 -- 172.21.15.167:0/3718246546 <== osd.1 v2:172.21.15.167:6800/30035 14 ==== osd_op_reply(38 default.realm [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 ==== 157+0+0 (crc 0 0 0) 0x7f90c802f080 con 0x559e90e6d930 2023-10-05T19:28:26.087 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.082+0000 7f90cfe1ae40 20 rados_obj.operate() r=-2 bl.length=0 2023-10-05T19:28:26.087 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.082+0000 7f90cfe1ae40 20 rados->read ofs=0 len=0 2023-10-05T19:28:26.087 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.082+0000 7f90cfe1ae40 1 -- 172.21.15.167:0/3718246546 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:39 5.f 5:f4c53578:::zonegroups_names.default:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x559e90ea18f0 con 0x559e90e6d930 2023-10-05T19:28:26.087 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.082+0000 7f90ce150700 1 -- 172.21.15.167:0/3718246546 <== osd.1 v2:172.21.15.167:6800/30035 15 ==== osd_op_reply(39 zonegroups_names.default [read 0~46 out=46b] v0'0 uv2 ondisk = 0) v8 ==== 168+0+46 (crc 0 0 0) 0x7f90c802f080 con 0x559e90e6d930 2023-10-05T19:28:26.087 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.082+0000 7f90cfe1ae40 20 rados_obj.operate() r=0 bl.length=46 2023-10-05T19:28:26.087 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.082+0000 7f90cfe1ae40 20 rados->read ofs=0 len=0 2023-10-05T19:28:26.088 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.082+0000 7f90cfe1ae40 1 -- 172.21.15.167:0/3718246546 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:40 5.c 5:30d299a5:::zonegroup_info.4f1f719d-9648-417b-b10c-e5d939825419:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x559e90ea0b60 con 0x7f90b8011040 2023-10-05T19:28:26.088 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.082+0000 7f90ce150700 1 -- 172.21.15.167:0/3718246546 <== osd.0 v2:172.21.15.118:6800/27690 10 ==== osd_op_reply(40 zonegroup_info.4f1f719d-9648-417b-b10c-e5d939825419 [read 0~362 out=362b] v0'0 uv1 ondisk = 0) v8 ==== 195+0+362 (crc 0 0 0) 0x7f90c802f080 con 0x7f90b8011040 2023-10-05T19:28:26.088 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.082+0000 7f90cfe1ae40 20 rados_obj.operate() r=0 bl.length=362 2023-10-05T19:28:26.088 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.082+0000 7f90cfe1ae40 20 rados->read ofs=0 len=0 2023-10-05T19:28:26.088 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.082+0000 7f90cfe1ae40 1 -- 172.21.15.167:0/3718246546 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:41 5.13 5:cbbf6c60:::converted:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x559e90ea5380 con 0x559e90ea21c0 2023-10-05T19:28:26.089 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.082+0000 7f90cd94f700 1 -- 172.21.15.167:0/3718246546 <== osd.3 v2:172.21.15.167:6808/30036 10 ==== osd_op_reply(41 converted [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 ==== 153+0+0 (crc 0 0 0) 0x7f90c0022020 con 0x559e90ea21c0 2023-10-05T19:28:26.089 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.082+0000 7f90cfe1ae40 20 rados_obj.operate() r=-2 bl.length=0 2023-10-05T19:28:26.089 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.082+0000 7f90cfe1ae40 20 rados->read ofs=0 len=0 2023-10-05T19:28:26.089 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.082+0000 7f90cfe1ae40 1 -- 172.21.15.167:0/3718246546 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:42 5.12 5:49953fa1:::default.realm:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x559e90ea5730 con 0x559e90e6d930 2023-10-05T19:28:26.090 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.082+0000 7f90ce150700 1 -- 172.21.15.167:0/3718246546 <== osd.1 v2:172.21.15.167:6800/30035 16 ==== osd_op_reply(42 default.realm [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 ==== 157+0+0 (crc 0 0 0) 0x7f90c802f080 con 0x559e90e6d930 2023-10-05T19:28:26.090 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.082+0000 7f90cfe1ae40 20 rados_obj.operate() r=-2 bl.length=0 2023-10-05T19:28:26.090 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.082+0000 7f90cfe1ae40 20 rados->read ofs=0 len=0 2023-10-05T19:28:26.090 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.082+0000 7f90cfe1ae40 1 -- 172.21.15.167:0/3718246546 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:43 5.f 5:f4c53578:::zonegroups_names.default:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x559e90ea5730 con 0x559e90e6d930 2023-10-05T19:28:26.091 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.086+0000 7f90ce150700 1 -- 172.21.15.167:0/3718246546 <== osd.1 v2:172.21.15.167:6800/30035 17 ==== osd_op_reply(43 zonegroups_names.default [read 0~46 out=46b] v0'0 uv2 ondisk = 0) v8 ==== 168+0+46 (crc 0 0 0) 0x7f90c802f080 con 0x559e90e6d930 2023-10-05T19:28:26.091 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.086+0000 7f90cfe1ae40 20 rados_obj.operate() r=0 bl.length=46 2023-10-05T19:28:26.091 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.086+0000 7f90cfe1ae40 1 -- 172.21.15.167:0/3718246546 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:44 5.0 5:00000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x559e90ea6740 con 0x559e90ea21c0 2023-10-05T19:28:26.093 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.086+0000 7f90cd94f700 1 -- 172.21.15.167:0/3718246546 <== osd.3 v2:172.21.15.167:6808/30036 11 ==== osd_op_reply(44 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f90c0054040 con 0x559e90ea21c0 2023-10-05T19:28:26.093 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.086+0000 7f90cd94f700 1 -- 172.21.15.167:0/3718246546 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:45 5.10 5:08000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x559e90ea6740 con 0x7f90c0007030 2023-10-05T19:28:26.094 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.090+0000 7f90cd14e700 1 -- 172.21.15.167:0/3718246546 <== osd.2 v2:172.21.15.167:6816/30037 7 ==== osd_op_reply(45 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f90b8004b80 con 0x7f90c0007030 2023-10-05T19:28:26.094 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.090+0000 7f90cd14e700 1 -- 172.21.15.167:0/3718246546 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:46 5.8 5:10000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x559e90ea6740 con 0x7f90b8011040 2023-10-05T19:28:26.095 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.090+0000 7f90ce150700 1 -- 172.21.15.167:0/3718246546 <== osd.0 v2:172.21.15.118:6800/27690 11 ==== osd_op_reply(46 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f90c802f080 con 0x7f90b8011040 2023-10-05T19:28:26.095 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.090+0000 7f90ce150700 1 -- 172.21.15.167:0/3718246546 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:47 5.18 5:18000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x559e90ea0b60 con 0x559e90e6d930 2023-10-05T19:28:26.096 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.090+0000 7f90ce150700 1 -- 172.21.15.167:0/3718246546 <== osd.1 v2:172.21.15.167:6800/30035 18 ==== osd_op_reply(47 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f90c802f080 con 0x559e90e6d930 2023-10-05T19:28:26.096 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.090+0000 7f90ce150700 1 -- 172.21.15.167:0/3718246546 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:48 5.4 5:20000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x559e90ea0b60 con 0x559e90e6d930 2023-10-05T19:28:26.096 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.090+0000 7f90ce150700 1 -- 172.21.15.167:0/3718246546 <== osd.1 v2:172.21.15.167:6800/30035 19 ==== osd_op_reply(48 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f90c802f080 con 0x559e90e6d930 2023-10-05T19:28:26.097 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.090+0000 7f90ce150700 1 -- 172.21.15.167:0/3718246546 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:49 5.14 5:28000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x559e90ea0b60 con 0x7f90b8011040 2023-10-05T19:28:26.098 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.090+0000 7f90ce150700 1 -- 172.21.15.167:0/3718246546 <== osd.0 v2:172.21.15.118:6800/27690 12 ==== osd_op_reply(49 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f90c802f080 con 0x7f90b8011040 2023-10-05T19:28:26.098 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.090+0000 7f90ce150700 1 -- 172.21.15.167:0/3718246546 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:50 5.c 5:30000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x559e90ea0b60 con 0x7f90b8011040 2023-10-05T19:28:26.099 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.094+0000 7f90ce150700 1 -- 172.21.15.167:0/3718246546 <== osd.0 v2:172.21.15.118:6800/27690 13 ==== osd_op_reply(50 [pgnls start_epoch 30 out=112b] v23'1 uv1 ondisk = 1) v8 ==== 144+0+112 (crc 0 0 0) 0x7f90c802f080 con 0x7f90b8011040 2023-10-05T19:28:26.099 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.094+0000 7f90ce150700 1 -- 172.21.15.167:0/3718246546 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:51 5.1c 5:38000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x559e90ea0b60 con 0x7f90c0007030 2023-10-05T19:28:26.100 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.094+0000 7f90cd14e700 1 -- 172.21.15.167:0/3718246546 <== osd.2 v2:172.21.15.167:6816/30037 8 ==== osd_op_reply(51 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f90b8004b80 con 0x7f90c0007030 2023-10-05T19:28:26.100 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.094+0000 7f90cd14e700 1 -- 172.21.15.167:0/3718246546 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:52 5.2 5:40000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x559e90ea0b60 con 0x559e90ea21c0 2023-10-05T19:28:26.100 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.094+0000 7f90cd94f700 1 -- 172.21.15.167:0/3718246546 <== osd.3 v2:172.21.15.167:6808/30036 12 ==== osd_op_reply(52 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f90c0054040 con 0x559e90ea21c0 2023-10-05T19:28:26.100 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.094+0000 7f90cd94f700 1 -- 172.21.15.167:0/3718246546 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:53 5.12 5:48000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x559e90ea0b60 con 0x559e90e6d930 2023-10-05T19:28:26.101 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.094+0000 7f90ce150700 1 -- 172.21.15.167:0/3718246546 <== osd.1 v2:172.21.15.167:6800/30035 20 ==== osd_op_reply(53 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f90c802f080 con 0x559e90e6d930 2023-10-05T19:28:26.101 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.094+0000 7f90ce150700 1 -- 172.21.15.167:0/3718246546 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:54 5.a 5:50000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x559e90ea0b60 con 0x7f90b8011040 2023-10-05T19:28:26.102 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.098+0000 7f90ce150700 1 -- 172.21.15.167:0/3718246546 <== osd.0 v2:172.21.15.118:6800/27690 14 ==== osd_op_reply(54 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f90c802f080 con 0x7f90b8011040 2023-10-05T19:28:26.102 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.098+0000 7f90ce150700 1 -- 172.21.15.167:0/3718246546 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:55 5.1a 5:58000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x559e90ea0b60 con 0x559e90e6d930 2023-10-05T19:28:26.103 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.098+0000 7f90ce150700 1 -- 172.21.15.167:0/3718246546 <== osd.1 v2:172.21.15.167:6800/30035 21 ==== osd_op_reply(55 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f90c802f080 con 0x559e90e6d930 2023-10-05T19:28:26.103 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.098+0000 7f90ce150700 1 -- 172.21.15.167:0/3718246546 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:56 5.6 5:60000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x559e90ea0b60 con 0x7f90c0007030 2023-10-05T19:28:26.105 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.098+0000 7f90cd14e700 1 -- 172.21.15.167:0/3718246546 <== osd.2 v2:172.21.15.167:6816/30037 9 ==== osd_op_reply(56 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f90b8004b80 con 0x7f90c0007030 2023-10-05T19:28:26.105 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.098+0000 7f90cd14e700 1 -- 172.21.15.167:0/3718246546 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:57 5.16 5:68000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x559e90ea0b60 con 0x559e90ea21c0 2023-10-05T19:28:26.106 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.098+0000 7f90cd94f700 1 -- 172.21.15.167:0/3718246546 <== osd.3 v2:172.21.15.167:6808/30036 13 ==== osd_op_reply(57 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f90c0054040 con 0x559e90ea21c0 2023-10-05T19:28:26.106 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.098+0000 7f90cd94f700 1 -- 172.21.15.167:0/3718246546 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:58 5.e 5:70000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x559e90ea0b60 con 0x559e90ea21c0 2023-10-05T19:28:26.106 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.102+0000 7f90cd94f700 1 -- 172.21.15.167:0/3718246546 <== osd.3 v2:172.21.15.167:6808/30036 14 ==== osd_op_reply(58 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f90c0054040 con 0x559e90ea21c0 2023-10-05T19:28:26.106 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.102+0000 7f90cd94f700 1 -- 172.21.15.167:0/3718246546 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:59 5.1e 5:78000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x7f90c0058040 con 0x7f90b8011040 2023-10-05T19:28:26.107 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.102+0000 7f90ce150700 1 -- 172.21.15.167:0/3718246546 <== osd.0 v2:172.21.15.118:6800/27690 15 ==== osd_op_reply(59 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f90c802f080 con 0x7f90b8011040 2023-10-05T19:28:26.107 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.102+0000 7f90ce150700 1 -- 172.21.15.167:0/3718246546 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:60 5.1 5:80000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x7f90c0058040 con 0x559e90ea21c0 2023-10-05T19:28:26.108 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.102+0000 7f90cd94f700 1 -- 172.21.15.167:0/3718246546 <== osd.3 v2:172.21.15.167:6808/30036 15 ==== osd_op_reply(60 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f90c0054040 con 0x559e90ea21c0 2023-10-05T19:28:26.108 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.102+0000 7f90cd94f700 1 -- 172.21.15.167:0/3718246546 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:61 5.11 5:88000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x7f90c0058040 con 0x559e90ea21c0 2023-10-05T19:28:26.108 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.102+0000 7f90cd94f700 1 -- 172.21.15.167:0/3718246546 <== osd.3 v2:172.21.15.167:6808/30036 16 ==== osd_op_reply(61 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f90c0054040 con 0x559e90ea21c0 2023-10-05T19:28:26.108 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.102+0000 7f90cd94f700 1 -- 172.21.15.167:0/3718246546 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:62 5.9 5:90000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x7f90c0058040 con 0x559e90e6d930 2023-10-05T19:28:26.109 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.102+0000 7f90ce150700 1 -- 172.21.15.167:0/3718246546 <== osd.1 v2:172.21.15.167:6800/30035 22 ==== osd_op_reply(62 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f90c802f080 con 0x559e90e6d930 2023-10-05T19:28:26.109 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.102+0000 7f90ce150700 1 -- 172.21.15.167:0/3718246546 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:63 5.19 5:98000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x7f90c0058040 con 0x559e90e6d930 2023-10-05T19:28:26.110 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.106+0000 7f90ce150700 1 -- 172.21.15.167:0/3718246546 <== osd.1 v2:172.21.15.167:6800/30035 23 ==== osd_op_reply(63 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f90c802f080 con 0x559e90e6d930 2023-10-05T19:28:26.110 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.106+0000 7f90ce150700 1 -- 172.21.15.167:0/3718246546 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:64 5.5 5:a0000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x7f90c0058040 con 0x7f90b8011040 2023-10-05T19:28:26.111 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.106+0000 7f90ce150700 1 -- 172.21.15.167:0/3718246546 <== osd.0 v2:172.21.15.118:6800/27690 16 ==== osd_op_reply(64 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f90c802f080 con 0x7f90b8011040 2023-10-05T19:28:26.111 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.106+0000 7f90ce150700 1 -- 172.21.15.167:0/3718246546 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:65 5.15 5:a8000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x7f90c0058040 con 0x559e90ea21c0 2023-10-05T19:28:26.112 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.106+0000 7f90cd94f700 1 -- 172.21.15.167:0/3718246546 <== osd.3 v2:172.21.15.167:6808/30036 17 ==== osd_op_reply(65 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f90c0054040 con 0x559e90ea21c0 2023-10-05T19:28:26.112 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.106+0000 7f90cd94f700 1 -- 172.21.15.167:0/3718246546 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:66 5.d 5:b0000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x7f90c0058040 con 0x7f90c0007030 2023-10-05T19:28:26.113 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.106+0000 7f90cd14e700 1 -- 172.21.15.167:0/3718246546 <== osd.2 v2:172.21.15.167:6816/30037 10 ==== osd_op_reply(66 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f90b8004b80 con 0x7f90c0007030 2023-10-05T19:28:26.113 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.106+0000 7f90cd14e700 1 -- 172.21.15.167:0/3718246546 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:67 5.1d 5:b8000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x7f90c0058040 con 0x559e90e6d930 2023-10-05T19:28:26.114 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.110+0000 7f90ce150700 1 -- 172.21.15.167:0/3718246546 <== osd.1 v2:172.21.15.167:6800/30035 24 ==== osd_op_reply(67 [pgnls start_epoch 30 out=107b] v30'2 uv2 ondisk = 1) v8 ==== 144+0+107 (crc 0 0 0) 0x7f90c802f080 con 0x559e90e6d930 2023-10-05T19:28:26.114 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.110+0000 7f90ce150700 1 -- 172.21.15.167:0/3718246546 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:68 5.3 5:c0000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x7f90c0058040 con 0x7f90b8011040 2023-10-05T19:28:26.115 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.110+0000 7f90ce150700 1 -- 172.21.15.167:0/3718246546 <== osd.0 v2:172.21.15.118:6800/27690 17 ==== osd_op_reply(68 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f90c802f080 con 0x7f90b8011040 2023-10-05T19:28:26.115 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.110+0000 7f90ce150700 1 -- 172.21.15.167:0/3718246546 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:69 5.13 5:c8000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x7f90c0058040 con 0x559e90ea21c0 2023-10-05T19:28:26.115 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.110+0000 7f90cd94f700 1 -- 172.21.15.167:0/3718246546 <== osd.3 v2:172.21.15.167:6808/30036 18 ==== osd_op_reply(69 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f90c0054040 con 0x559e90ea21c0 2023-10-05T19:28:26.116 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.110+0000 7f90cd94f700 1 -- 172.21.15.167:0/3718246546 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:70 5.b 5:d0000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x7f90c0058040 con 0x7f90b8011040 2023-10-05T19:28:26.117 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.110+0000 7f90ce150700 1 -- 172.21.15.167:0/3718246546 <== osd.0 v2:172.21.15.118:6800/27690 18 ==== osd_op_reply(70 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f90c802f080 con 0x7f90b8011040 2023-10-05T19:28:26.117 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.110+0000 7f90ce150700 1 -- 172.21.15.167:0/3718246546 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:71 5.1b 5:d8000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x7f90c0058040 con 0x7f90c0007030 2023-10-05T19:28:26.118 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.110+0000 7f90cd14e700 1 -- 172.21.15.167:0/3718246546 <== osd.2 v2:172.21.15.167:6816/30037 11 ==== osd_op_reply(71 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f90b8004b80 con 0x7f90c0007030 2023-10-05T19:28:26.118 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.110+0000 7f90cd14e700 1 -- 172.21.15.167:0/3718246546 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:72 5.7 5:e0000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x7f90c0058040 con 0x559e90ea21c0 2023-10-05T19:28:26.118 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.114+0000 7f90cd94f700 1 -- 172.21.15.167:0/3718246546 <== osd.3 v2:172.21.15.167:6808/30036 19 ==== osd_op_reply(72 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f90c0054040 con 0x559e90ea21c0 2023-10-05T19:28:26.118 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.114+0000 7f90cd94f700 1 -- 172.21.15.167:0/3718246546 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:73 5.17 5:e8000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x7f90c0058040 con 0x7f90b8011040 2023-10-05T19:28:26.119 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.114+0000 7f90ce150700 1 -- 172.21.15.167:0/3718246546 <== osd.0 v2:172.21.15.118:6800/27690 19 ==== osd_op_reply(73 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f90c802f080 con 0x7f90b8011040 2023-10-05T19:28:26.119 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.114+0000 7f90ce150700 1 -- 172.21.15.167:0/3718246546 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:74 5.f 5:f0000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x7f90c0058040 con 0x559e90e6d930 2023-10-05T19:28:26.121 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.114+0000 7f90ce150700 1 -- 172.21.15.167:0/3718246546 <== osd.1 v2:172.21.15.167:6800/30035 25 ==== osd_op_reply(74 [pgnls start_epoch 30 out=115b] v23'2 uv2 ondisk = 1) v8 ==== 144+0+115 (crc 0 0 0) 0x7f90c802f080 con 0x559e90e6d930 2023-10-05T19:28:26.121 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.114+0000 7f90ce150700 1 -- 172.21.15.167:0/3718246546 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:75 5.1f 5:f8000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x7f90c0058040 con 0x7f90c0007030 2023-10-05T19:28:26.121 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.114+0000 7f90cd14e700 1 -- 172.21.15.167:0/3718246546 <== osd.2 v2:172.21.15.167:6816/30037 12 ==== osd_op_reply(75 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f90b8004b80 con 0x7f90c0007030 2023-10-05T19:28:26.121 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.114+0000 7f90cfe1ae40 20 RGWRados::pool_iterate: got zonegroup_info.4f1f719d-9648-417b-b10c-e5d939825419 2023-10-05T19:28:26.122 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.114+0000 7f90cfe1ae40 20 RGWRados::pool_iterate: got zone_info.9a9df965-6612-45d4-a45f-6e3c889bacc2 2023-10-05T19:28:26.122 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.114+0000 7f90cfe1ae40 20 RGWRados::pool_iterate: got zone_names.default 2023-10-05T19:28:26.122 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.114+0000 7f90cfe1ae40 20 RGWRados::pool_iterate: got zonegroups_names.default 2023-10-05T19:28:26.122 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.114+0000 7f90cfe1ae40 20 rados->read ofs=0 len=0 2023-10-05T19:28:26.122 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.114+0000 7f90cfe1ae40 1 -- 172.21.15.167:0/3718246546 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:76 5.f 5:f43b6ece:::zone_names.default:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x559e90ea5730 con 0x559e90e6d930 2023-10-05T19:28:26.123 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.118+0000 7f90ce150700 1 -- 172.21.15.167:0/3718246546 <== osd.1 v2:172.21.15.167:6800/30035 26 ==== osd_op_reply(76 zone_names.default [read 0~46 out=46b] v0'0 uv1 ondisk = 0) v8 ==== 162+0+46 (crc 0 0 0) 0x7f90c802f080 con 0x559e90e6d930 2023-10-05T19:28:26.123 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.118+0000 7f90cfe1ae40 20 rados_obj.operate() r=0 bl.length=46 2023-10-05T19:28:26.123 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.118+0000 7f90cfe1ae40 20 rados->read ofs=0 len=0 2023-10-05T19:28:26.123 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.118+0000 7f90cfe1ae40 1 -- 172.21.15.167:0/3718246546 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:77 5.1d 5:b8374eb4:::zone_info.9a9df965-6612-45d4-a45f-6e3c889bacc2:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x559e90ea5ae0 con 0x559e90e6d930 2023-10-05T19:28:26.123 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.118+0000 7f90ce150700 1 -- 172.21.15.167:0/3718246546 <== osd.1 v2:172.21.15.167:6800/30035 27 ==== osd_op_reply(77 zone_info.9a9df965-6612-45d4-a45f-6e3c889bacc2 [read 0~874 out=874b] v0'0 uv2 ondisk = 0) v8 ==== 190+0+874 (crc 0 0 0) 0x7f90c802f080 con 0x559e90e6d930 2023-10-05T19:28:26.124 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.118+0000 7f90cfe1ae40 20 rados_obj.operate() r=0 bl.length=874 2023-10-05T19:28:26.124 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.118+0000 7f90cfe1ae40 20 rados->read ofs=0 len=0 2023-10-05T19:28:26.124 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.118+0000 7f90cfe1ae40 1 -- 172.21.15.167:0/3718246546 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:78 5.f 5:f4c53578:::zonegroups_names.default:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x559e90ea5fd0 con 0x559e90e6d930 2023-10-05T19:28:26.124 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.118+0000 7f90ce150700 1 -- 172.21.15.167:0/3718246546 <== osd.1 v2:172.21.15.167:6800/30035 28 ==== osd_op_reply(78 zonegroups_names.default [read 0~46 out=46b] v0'0 uv2 ondisk = 0) v8 ==== 168+0+46 (crc 0 0 0) 0x7f90c802f080 con 0x559e90e6d930 2023-10-05T19:28:26.124 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.118+0000 7f90cfe1ae40 20 rados_obj.operate() r=0 bl.length=46 2023-10-05T19:28:26.124 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.118+0000 7f90cfe1ae40 20 rados->read ofs=0 len=0 2023-10-05T19:28:26.125 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.118+0000 7f90cfe1ae40 1 -- 172.21.15.167:0/3718246546 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:79 5.c 5:30d299a5:::zonegroup_info.4f1f719d-9648-417b-b10c-e5d939825419:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x559e90ea5fd0 con 0x7f90b8011040 2023-10-05T19:28:26.125 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.118+0000 7f90ce150700 1 -- 172.21.15.167:0/3718246546 <== osd.0 v2:172.21.15.118:6800/27690 20 ==== osd_op_reply(79 zonegroup_info.4f1f719d-9648-417b-b10c-e5d939825419 [read 0~362 out=362b] v0'0 uv1 ondisk = 0) v8 ==== 195+0+362 (crc 0 0 0) 0x7f90c802f080 con 0x7f90b8011040 2023-10-05T19:28:26.125 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.118+0000 7f90cfe1ae40 20 rados_obj.operate() r=0 bl.length=362 2023-10-05T19:28:26.125 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.118+0000 7f90cfe1ae40 20 rados->read ofs=0 len=0 2023-10-05T19:28:26.125 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.118+0000 7f90cfe1ae40 1 -- 172.21.15.167:0/3718246546 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:80 5.3 5:c0ad8cbd:::region_map:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x559e90ea60d0 con 0x7f90b8011040 2023-10-05T19:28:26.126 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.122+0000 7f90ce150700 1 -- 172.21.15.167:0/3718246546 <== osd.0 v2:172.21.15.118:6800/27690 21 ==== osd_op_reply(80 region_map [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 ==== 154+0+0 (crc 0 0 0) 0x7f90c8050040 con 0x7f90b8011040 2023-10-05T19:28:26.126 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.122+0000 7f90cfe1ae40 20 rados_obj.operate() r=-2 bl.length=0 2023-10-05T19:28:26.126 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.122+0000 7f90cfe1ae40 20 zone default found 2023-10-05T19:28:26.126 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.122+0000 7f90cfe1ae40 4 Realm: () 2023-10-05T19:28:26.127 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.122+0000 7f90cfe1ae40 4 ZoneGroup: default (4f1f719d-9648-417b-b10c-e5d939825419) 2023-10-05T19:28:26.127 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.122+0000 7f90cfe1ae40 4 Zone: default (9a9df965-6612-45d4-a45f-6e3c889bacc2) 2023-10-05T19:28:26.127 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.122+0000 7f90cfe1ae40 10 cannot find current period zonegroup using local zonegroup configuration 2023-10-05T19:28:26.127 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.122+0000 7f90cfe1ae40 20 zonegroup default 2023-10-05T19:28:26.127 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.122+0000 7f90cfe1ae40 20 rados->read ofs=0 len=0 2023-10-05T19:28:26.127 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.122+0000 7f90cfe1ae40 1 -- 172.21.15.167:0/3718246546 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:81 5.3 5:c52100b6:::period_config.default:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x559e90ea60d0 con 0x7f90b8011040 2023-10-05T19:28:26.128 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.122+0000 7f90ce150700 1 -- 172.21.15.167:0/3718246546 <== osd.0 v2:172.21.15.118:6800/27690 22 ==== osd_op_reply(81 period_config.default [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 ==== 165+0+0 (crc 0 0 0) 0x7f90c8050040 con 0x7f90b8011040 2023-10-05T19:28:26.128 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.122+0000 7f90cfe1ae40 20 rados_obj.operate() r=-2 bl.length=0 2023-10-05T19:28:26.128 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.122+0000 7f90cfe1ae40 20 rados->read ofs=0 len=0 2023-10-05T19:28:26.128 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.122+0000 7f90cfe1ae40 1 -- 172.21.15.167:0/3718246546 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:82 6.2 6:4347d321:::bucket.sync-source-hints.:head [call version.read in=11b,read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x559e90ea7a50 con 0x559e90ea21c0 2023-10-05T19:28:26.128 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.122+0000 7f90cd94f700 1 -- 172.21.15.167:0/3718246546 <== osd.3 v2:172.21.15.167:6808/30036 20 ==== osd_op_reply(82 bucket.sync-source-hints. [call,read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 ==== 211+0+0 (crc 0 0 0) 0x7f90c0054040 con 0x559e90ea21c0 2023-10-05T19:28:26.128 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.122+0000 7f90cfe1ae40 20 rados_obj.operate() r=-2 bl.length=0 2023-10-05T19:28:26.128 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.122+0000 7f90cfe1ae40 20 rados->read ofs=0 len=0 2023-10-05T19:28:26.129 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.122+0000 7f90cfe1ae40 1 -- 172.21.15.167:0/3718246546 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:83 6.b 6:d467b91b:::bucket.sync-target-hints.:head [call version.read in=11b,read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x559e90ea7e00 con 0x559e90ea21c0 2023-10-05T19:28:26.129 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.122+0000 7f90cd94f700 1 -- 172.21.15.167:0/3718246546 <== osd.3 v2:172.21.15.167:6808/30036 21 ==== osd_op_reply(83 bucket.sync-target-hints. [call,read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 ==== 211+0+0 (crc 0 0 0) 0x7f90c0054040 con 0x559e90ea21c0 2023-10-05T19:28:26.129 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.122+0000 7f90cfe1ae40 20 rados_obj.operate() r=-2 bl.length=0 2023-10-05T19:28:26.129 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.122+0000 7f90cfe1ae40 20 started sync module instance, tier type = 2023-10-05T19:28:26.129 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.122+0000 7f90cfe1ae40 20 started zone id=9a9df965-6612-45d4-a45f-6e3c889bacc2 (name=default) with tier type = 2023-10-05T19:28:26.130 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.122+0000 7f90cfe1ae40 1 -- 172.21.15.167:0/3718246546 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:84 7.1f 7:f95f44c2:::notify.0:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x559e90ea7110 con 0x7f90c0007030 2023-10-05T19:28:26.132 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.126+0000 7f90cd14e700 1 -- 172.21.15.167:0/3718246546 <== osd.2 v2:172.21.15.167:6816/30037 13 ==== osd_op_reply(84 notify.0 [create] v30'30 uv30 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f90b802f040 con 0x7f90c0007030 2023-10-05T19:28:26.132 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.126+0000 7f90cfe1ae40 1 -- 172.21.15.167:0/3718246546 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:85 7.1f 7:f95f44c2:::notify.0:head [watch watch cookie 94139524475936] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x559e90ea81b0 con 0x7f90c0007030 2023-10-05T19:28:26.132 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.126+0000 7f90cfe1ae40 1 -- 172.21.15.167:0/3718246546 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:86 7.0 7:05bf5b68:::notify.1:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x559e90ea8bd0 con 0x559e90e6d930 2023-10-05T19:28:26.135 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.130+0000 7f90cd14e700 1 -- 172.21.15.167:0/3718246546 <== osd.2 v2:172.21.15.167:6816/30037 14 ==== osd_op_reply(85 notify.0 [watch watch cookie 94139524475936] v30'31 uv30 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f90b802f040 con 0x7f90c0007030 2023-10-05T19:28:26.135 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.130+0000 7f90ce150700 1 -- 172.21.15.167:0/3718246546 <== osd.1 v2:172.21.15.167:6800/30035 29 ==== osd_op_reply(86 notify.1 [create] v30'30 uv30 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f90c8050040 con 0x559e90e6d930 2023-10-05T19:28:26.135 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.130+0000 7f90cfe1ae40 1 -- 172.21.15.167:0/3718246546 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:87 7.0 7:05bf5b68:::notify.1:head [watch watch cookie 94139524478224] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x559e90ea8fd0 con 0x559e90e6d930 2023-10-05T19:28:26.135 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.130+0000 7f90cfe1ae40 1 -- 172.21.15.167:0/3718246546 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:88 7.15 7:a93a5511:::notify.2:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x559e90ea9870 con 0x559e90ea21c0 2023-10-05T19:28:26.138 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.130+0000 7f90ce150700 1 -- 172.21.15.167:0/3718246546 <== osd.1 v2:172.21.15.167:6800/30035 30 ==== osd_op_reply(87 notify.1 [watch watch cookie 94139524478224] v30'31 uv30 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f90c8050040 con 0x559e90e6d930 2023-10-05T19:28:26.138 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.134+0000 7f90cd94f700 1 -- 172.21.15.167:0/3718246546 <== osd.3 v2:172.21.15.167:6808/30036 22 ==== osd_op_reply(88 notify.2 [create] v30'30 uv30 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f90c0054040 con 0x559e90ea21c0 2023-10-05T19:28:26.138 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.134+0000 7f90cfe1ae40 1 -- 172.21.15.167:0/3718246546 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:89 7.15 7:a93a5511:::notify.2:head [watch watch cookie 94139524485072] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x559e90ea87c0 con 0x559e90ea21c0 2023-10-05T19:28:26.139 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.134+0000 7f90cfe1ae40 1 -- 172.21.15.167:0/3718246546 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:90 7.e 7:7759931f:::notify.3:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x559e90eaa600 con 0x7f90c0007030 2023-10-05T19:28:26.141 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.134+0000 7f90cd94f700 1 -- 172.21.15.167:0/3718246546 <== osd.3 v2:172.21.15.167:6808/30036 23 ==== osd_op_reply(89 notify.2 [watch watch cookie 94139524485072] v30'31 uv30 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f90c0054040 con 0x559e90ea21c0 2023-10-05T19:28:26.141 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.134+0000 7f90cd14e700 1 -- 172.21.15.167:0/3718246546 <== osd.2 v2:172.21.15.167:6816/30037 15 ==== osd_op_reply(90 notify.3 [create] v30'30 uv30 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f90b802f040 con 0x7f90c0007030 2023-10-05T19:28:26.141 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.134+0000 7f90cfe1ae40 1 -- 172.21.15.167:0/3718246546 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:91 7.e 7:7759931f:::notify.3:head [watch watch cookie 94139524488224] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x559e90ea87c0 con 0x7f90c0007030 2023-10-05T19:28:26.142 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.134+0000 7f90cfe1ae40 1 -- 172.21.15.167:0/3718246546 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:92 7.d 7:b4812045:::notify.4:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x559e90eab120 con 0x559e90e6d930 2023-10-05T19:28:26.143 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.138+0000 7f90cd14e700 1 -- 172.21.15.167:0/3718246546 <== osd.2 v2:172.21.15.167:6816/30037 16 ==== osd_op_reply(91 notify.3 [watch watch cookie 94139524488224] v30'31 uv30 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f90b802f040 con 0x7f90c0007030 2023-10-05T19:28:26.144 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.138+0000 7f90ce150700 1 -- 172.21.15.167:0/3718246546 <== osd.1 v2:172.21.15.167:6800/30035 31 ==== osd_op_reply(92 notify.4 [create] v30'30 uv30 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f90c8050040 con 0x559e90e6d930 2023-10-05T19:28:26.144 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.138+0000 7f90cfe1ae40 1 -- 172.21.15.167:0/3718246546 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:93 7.d 7:b4812045:::notify.4:head [watch watch cookie 94139524492336] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x559e90eab080 con 0x559e90e6d930 2023-10-05T19:28:26.144 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.138+0000 7f90cfe1ae40 1 -- 172.21.15.167:0/3718246546 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:94 7.3 7:c609908c:::notify.5:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x559e90eabf00 con 0x7f90b8011040 2023-10-05T19:28:26.147 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.142+0000 7f90ce150700 1 -- 172.21.15.167:0/3718246546 <== osd.0 v2:172.21.15.118:6800/27690 23 ==== osd_op_reply(94 notify.5 [create] v30'30 uv30 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f90c8050040 con 0x7f90b8011040 2023-10-05T19:28:26.147 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.142+0000 7f90cfe1ae40 1 -- 172.21.15.167:0/3718246546 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:95 7.3 7:c609908c:::notify.5:head [watch watch cookie 94139524495888] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x559e90eabe60 con 0x7f90b8011040 2023-10-05T19:28:26.147 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.142+0000 7f90cfe1ae40 1 -- 172.21.15.167:0/3718246546 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:96 7.14 7:2b04a3e9:::notify.6:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x559e90eabe60 con 0x7f90b8011040 2023-10-05T19:28:26.147 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.142+0000 7f90ce150700 1 -- 172.21.15.167:0/3718246546 <== osd.1 v2:172.21.15.167:6800/30035 32 ==== osd_op_reply(93 notify.4 [watch watch cookie 94139524492336] v30'31 uv30 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f90c8050040 con 0x559e90e6d930 2023-10-05T19:28:26.149 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.142+0000 7f90ce150700 1 -- 172.21.15.167:0/3718246546 <== osd.0 v2:172.21.15.118:6800/27690 24 ==== osd_op_reply(95 notify.5 [watch watch cookie 94139524495888] v30'31 uv30 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f90c8050040 con 0x7f90b8011040 2023-10-05T19:28:26.149 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.142+0000 7f90ce150700 1 -- 172.21.15.167:0/3718246546 <== osd.0 v2:172.21.15.118:6800/27690 25 ==== osd_op_reply(96 notify.6 [create] v30'30 uv30 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f90c8050040 con 0x7f90b8011040 2023-10-05T19:28:26.150 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.142+0000 7f90cfe1ae40 1 -- 172.21.15.167:0/3718246546 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:97 7.14 7:2b04a3e9:::notify.6:head [watch watch cookie 94139524494576] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x559e90eac960 con 0x7f90b8011040 2023-10-05T19:28:26.150 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.142+0000 7f90cfe1ae40 1 -- 172.21.15.167:0/3718246546 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:98 7.9 7:93e5b521:::notify.7:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x559e90ead200 con 0x7f90b8011040 2023-10-05T19:28:26.151 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.146+0000 7f90ce150700 1 -- 172.21.15.167:0/3718246546 <== osd.0 v2:172.21.15.118:6800/27690 26 ==== osd_op_reply(97 notify.6 [watch watch cookie 94139524494576] v30'31 uv30 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f90c8050040 con 0x7f90b8011040 2023-10-05T19:28:26.152 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.146+0000 7f90ce150700 1 -- 172.21.15.167:0/3718246546 <== osd.0 v2:172.21.15.118:6800/27690 27 ==== osd_op_reply(98 notify.7 [create] v30'30 uv30 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f90c8050040 con 0x7f90b8011040 2023-10-05T19:28:26.152 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.146+0000 7f90cfe1ae40 1 -- 172.21.15.167:0/3718246546 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:99 7.9 7:93e5b521:::notify.7:head [watch watch cookie 94139524499808] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x559e90ead2a0 con 0x7f90b8011040 2023-10-05T19:28:26.152 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.146+0000 7f90cfe1ae40 20 add_watcher() i=0 2023-10-05T19:28:26.152 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.146+0000 7f90cfe1ae40 20 add_watcher() i=1 2023-10-05T19:28:26.153 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.146+0000 7f90cfe1ae40 20 add_watcher() i=2 2023-10-05T19:28:26.153 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.146+0000 7f90cfe1ae40 20 add_watcher() i=3 2023-10-05T19:28:26.153 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.146+0000 7f90cfe1ae40 20 add_watcher() i=4 2023-10-05T19:28:26.153 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.146+0000 7f90cfe1ae40 20 add_watcher() i=5 2023-10-05T19:28:26.153 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.146+0000 7f90cfe1ae40 20 add_watcher() i=6 2023-10-05T19:28:26.154 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.150+0000 7f90ce150700 1 -- 172.21.15.167:0/3718246546 <== osd.0 v2:172.21.15.118:6800/27690 28 ==== osd_op_reply(99 notify.7 [watch watch cookie 94139524499808] v30'31 uv30 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f90c8050040 con 0x7f90b8011040 2023-10-05T19:28:26.155 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.150+0000 7f90cfe1ae40 20 add_watcher() i=7 2023-10-05T19:28:26.155 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.150+0000 7f90cfe1ae40 2 all 8 watchers are set, enabling cache 2023-10-05T19:28:26.155 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.150+0000 7f90cfe1ae40 1 -- 172.21.15.167:0/3718246546 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:100 6.4 6:22d26bf9:::data_loggenerations_metadata:head [call version.check_conds in=74b,call version.read in=11b,read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x559e90eb03e0 con 0x559e90e6d930 2023-10-05T19:28:26.155 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.150+0000 7f90ce150700 1 -- 172.21.15.167:0/3718246546 <== osd.1 v2:172.21.15.167:6800/30035 33 ==== osd_op_reply(100 data_loggenerations_metadata [call,call out=48b,read 0~28 out=28b] v0'0 uv1 ondisk = 0) v8 ==== 256+0+76 (crc 0 0 0) 0x7f90c8050040 con 0x559e90e6d930 2023-10-05T19:28:26.155 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.150+0000 7f90cfe1ae40 1 -- 172.21.15.167:0/3718246546 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:101 6.4 6:22d26bf9:::data_loggenerations_metadata:head [watch watch cookie 94139524500912] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x559e90ead6f0 con 0x559e90e6d930 2023-10-05T19:28:26.158 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.150+0000 7f90ce150700 1 -- 172.21.15.167:0/3718246546 <== osd.1 v2:172.21.15.167:6800/30035 34 ==== osd_op_reply(101 data_loggenerations_metadata [watch watch cookie 94139524500912] v30'54 uv1 ondisk = 0) v8 ==== 172+0+0 (crc 0 0 0) 0x7f90c8050040 con 0x559e90e6d930 2023-10-05T19:28:26.158 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.154+0000 7f909f7de700 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2023-10-05T19:28:26.158 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.154+0000 7f90cfe1ae40 20 check_secure_mon_conn(): auth registy supported: methods=[2] modes=[2,1] 2023-10-05T19:28:26.158 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.154+0000 7f90cfe1ae40 20 check_secure_mon_conn(): mode 1 is insecure 2023-10-05T19:28:26.159 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.154+0000 7f90cfe1ae40 1 Processor -- start 2023-10-05T19:28:26.159 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.154+0000 7f90cfe1ae40 1 -- start start 2023-10-05T19:28:26.159 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.154+0000 7f90cfe1ae40 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x559e90eba890 0x559e90ebac60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-05T19:28:26.159 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.154+0000 7f90cfe1ae40 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x559e90ebb1a0 con 0x559e90eba890 2023-10-05T19:28:26.160 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.154+0000 7f90cd14e700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x559e90eba890 0x559e90ebac60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-05T19:28:26.160 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.154+0000 7f90cd14e700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x559e90eba890 0x559e90ebac60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.118:3300/0 says I am v2:172.21.15.167:57264/0 (socket says 172.21.15.167:57264) 2023-10-05T19:28:26.160 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.154+0000 7f90cd14e700 1 -- 172.21.15.167:0/426686188 learned_addr learned my addr 172.21.15.167:0/426686188 (peer_addr_for_me v2:172.21.15.167:0/0) 2023-10-05T19:28:26.160 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.154+0000 7f90cd14e700 1 -- 172.21.15.167:0/426686188 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x559e90ebb2e0 con 0x559e90eba890 2023-10-05T19:28:26.161 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.154+0000 7f90cd14e700 1 --2- 172.21.15.167:0/426686188 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x559e90eba890 0x559e90ebac60 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x559e90e9ff10 tx=0x7f90b8006000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e844fd58f199871d server_cookie=0 in_seq=0 out_seq=0 2023-10-05T19:28:26.161 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.154+0000 7f909efdd700 1 -- 172.21.15.167:0/426686188 <== mon.0 v2:172.21.15.118:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f90b802c040 con 0x559e90eba890 2023-10-05T19:28:26.161 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.154+0000 7f909efdd700 1 -- 172.21.15.167:0/426686188 <== mon.0 v2:172.21.15.118:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f90b802d040 con 0x559e90eba890 2023-10-05T19:28:26.161 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.154+0000 7f909efdd700 1 -- 172.21.15.167:0/426686188 <== mon.0 v2:172.21.15.118:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f90b8028cc0 con 0x559e90eba890 2023-10-05T19:28:26.162 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.158+0000 7f90cfe1ae40 1 -- 172.21.15.167:0/426686188 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x559e90eba890 msgr2=0x559e90ebac60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-05T19:28:26.162 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.158+0000 7f90cfe1ae40 1 --2- 172.21.15.167:0/426686188 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x559e90eba890 0x559e90ebac60 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x559e90e9ff10 tx=0x7f90b8006000 comp rx=0 tx=0).stop 2023-10-05T19:28:26.162 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.158+0000 7f90cfe1ae40 1 -- 172.21.15.167:0/426686188 shutdown_connections 2023-10-05T19:28:26.163 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.158+0000 7f90cfe1ae40 1 --2- 172.21.15.167:0/426686188 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x559e90eba890 0x559e90ebac60 unknown :-1 s=CLOSED pgs=198 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-05T19:28:26.163 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.158+0000 7f90cfe1ae40 1 -- 172.21.15.167:0/426686188 >> 172.21.15.167:0/426686188 conn(0x559e90eb5c50 msgr2=0x559e90eb3e00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-05T19:28:26.163 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.158+0000 7f90cfe1ae40 1 -- 172.21.15.167:0/426686188 shutdown_connections 2023-10-05T19:28:26.163 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.158+0000 7f90cfe1ae40 1 -- 172.21.15.167:0/426686188 wait complete. 2023-10-05T19:28:26.164 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.158+0000 7f90cfe1ae40 1 Processor -- start 2023-10-05T19:28:26.164 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.158+0000 7f90cfe1ae40 1 -- start start 2023-10-05T19:28:26.164 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.158+0000 7f90cfe1ae40 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x559e90eba890 0x559e90ec03f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-05T19:28:26.164 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.158+0000 7f90cfe1ae40 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x559e90ec13f0 con 0x559e90eba890 2023-10-05T19:28:26.164 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.158+0000 7f90cd14e700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x559e90eba890 0x559e90ec03f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-05T19:28:26.164 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.158+0000 7f90cd14e700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x559e90eba890 0x559e90ec03f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.118:3300/0 says I am v2:172.21.15.167:57268/0 (socket says 172.21.15.167:57268) 2023-10-05T19:28:26.164 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.158+0000 7f90cd14e700 1 -- 172.21.15.167:0/2107617791 learned_addr learned my addr 172.21.15.167:0/2107617791 (peer_addr_for_me v2:172.21.15.167:0/0) 2023-10-05T19:28:26.165 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.158+0000 7f90cd14e700 1 -- 172.21.15.167:0/2107617791 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x559e90db7bd0 con 0x559e90eba890 2023-10-05T19:28:26.165 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.158+0000 7f90cd14e700 1 --2- 172.21.15.167:0/2107617791 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x559e90eba890 0x559e90ec03f0 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7f90b80049e0 tx=0x7f90b803c000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-05T19:28:26.165 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.158+0000 7f909d7da700 1 -- 172.21.15.167:0/2107617791 <== mon.0 v2:172.21.15.118:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f90b802c070 con 0x559e90eba890 2023-10-05T19:28:26.166 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.158+0000 7f909d7da700 1 -- 172.21.15.167:0/2107617791 <== mon.0 v2:172.21.15.118:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f90b801c040 con 0x559e90eba890 2023-10-05T19:28:26.166 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.158+0000 7f909d7da700 1 -- 172.21.15.167:0/2107617791 <== mon.0 v2:172.21.15.118:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f90b8033e80 con 0x559e90eba890 2023-10-05T19:28:26.166 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.158+0000 7f90cfe1ae40 1 -- 172.21.15.167:0/2107617791 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x559e90ec09f0 con 0x559e90eba890 2023-10-05T19:28:26.166 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.158+0000 7f90cfe1ae40 1 -- 172.21.15.167:0/2107617791 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x559e90ec0e30 con 0x559e90eba890 2023-10-05T19:28:26.167 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.162+0000 7f909d7da700 1 -- 172.21.15.167:0/2107617791 <== mon.0 v2:172.21.15.118:3300/0 4 ==== mgrmap(e 5) v1 ==== 40329+0+0 (secure 0 0 0) 0x7f90b8045050 con 0x559e90eba890 2023-10-05T19:28:26.167 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.162+0000 7f909d7da700 1 --2- 172.21.15.167:0/2107617791 >> [v2:172.21.15.167:6824/29889,v1:172.21.15.167:6825/29889] conn(0x7f9014030c00 0x7f90140330b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-05T19:28:26.167 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.162+0000 7f909d7da700 1 -- 172.21.15.167:0/2107617791 <== mon.0 v2:172.21.15.118:3300/0 5 ==== osd_map(30..30 src has 1..30) v4 ==== 6057+0+0 (secure 0 0 0) 0x7f90b8052c20 con 0x559e90eba890 2023-10-05T19:28:26.168 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.162+0000 7f90ce150700 1 --2- 172.21.15.167:0/2107617791 >> [v2:172.21.15.167:6824/29889,v1:172.21.15.167:6825/29889] conn(0x7f9014030c00 0x7f90140330b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-05T19:28:26.168 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.162+0000 7f90cfe1ae40 5 note: GC not initialized 2023-10-05T19:28:26.168 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.162+0000 7f90ce150700 1 --2- 172.21.15.167:0/2107617791 >> [v2:172.21.15.167:6824/29889,v1:172.21.15.167:6825/29889] conn(0x7f9014030c00 0x7f90140330b0 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x559e90eb0460 tx=0x7f90c8000f90 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-05T19:28:26.170 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.166+0000 7f900e7fc700 20 reqs_thread_entry: start 2023-10-05T19:28:26.171 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.166+0000 7f90cfe1ae40 20 init_complete bucket index max shards: 11 2023-10-05T19:28:26.171 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.166+0000 7f8ffffff700 20 reqs_thread_entry: start 2023-10-05T19:28:26.171 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.166+0000 7f90cfe1ae40 10 rgw notify: Started notification manager with: 1 workers 2023-10-05T19:28:26.171 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.166+0000 7f8ffe7fc700 20 rgw notify: INFO: next queues processing will happen at: Thu Oct 5 19:28:56 2023 2023-10-05T19:28:26.171 INFO:teuthology.orchestra.run.smithi167.stderr: 2023-10-05T19:28:26.172 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.166+0000 7f8ffdffb700 20 reqs_thread_entry: start 2023-10-05T19:28:26.173 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.166+0000 7f90cfe1ae40 20 get_system_obj_state: rctx=0x7ffc7d5b3ad8 obj=default.rgw.meta:users.uid:foo state=0x559e90d440b0 s->prefetch_data=0 2023-10-05T19:28:26.173 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.166+0000 7f90cfe1ae40 1 -- 172.21.15.167:0/3718246546 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:102 8.18 8:1e6ca5ed:users.uid::foo:head [call version.read in=11b,getxattrs,stat] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x559e90f28c40 con 0x559e90e6d930 2023-10-05T19:28:26.173 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.166+0000 7f90ce150700 1 -- 172.21.15.167:0/3718246546 <== osd.1 v2:172.21.15.167:6800/30035 35 ==== osd_op_reply(102 foo [call out=48b,getxattrs out=75b,stat out=16b] v0'0 uv1 ondisk = 0) v8 ==== 231+0+139 (crc 0 0 0) 0x7f90c8050040 con 0x559e90e6d930 2023-10-05T19:28:26.173 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.166+0000 7f90cfe1ae40 20 get_system_obj_state: s->obj_tag was set empty 2023-10-05T19:28:26.173 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.166+0000 7f90cfe1ae40 20 Read xattr: user.rgw.idtag 2023-10-05T19:28:26.173 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.166+0000 7f90cfe1ae40 20 rados->read ofs=0 len=0 2023-10-05T19:28:26.173 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.166+0000 7f90cfe1ae40 1 -- 172.21.15.167:0/3718246546 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:103 8.18 8:1e6ca5ed:users.uid::foo:head [call version.check_conds in=122b,call version.read in=11b,read 0~0,getxattrs] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x559e90f2b6d0 con 0x559e90e6d930 2023-10-05T19:28:26.174 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.166+0000 7f90ce150700 1 -- 172.21.15.167:0/3718246546 <== osd.1 v2:172.21.15.167:6800/30035 36 ==== osd_op_reply(103 foo [call,call out=48b,read 0~357 out=357b,getxattrs out=75b] v0'0 uv1 ondisk = 0) v8 ==== 273+0+480 (crc 0 0 0) 0x7f90c8050040 con 0x559e90e6d930 2023-10-05T19:28:26.174 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.166+0000 7f90cfe1ae40 20 rados_obj.operate() r=0 bl.length=357 2023-10-05T19:28:26.174 INFO:teuthology.orchestra.run.smithi167.stdout:{ 2023-10-05T19:28:26.174 INFO:teuthology.orchestra.run.smithi167.stdout: "user_id": "foo", 2023-10-05T19:28:26.174 INFO:teuthology.orchestra.run.smithi167.stdout: "display_name": "Foo", 2023-10-05T19:28:26.174 INFO:teuthology.orchestra.run.smithi167.stdout: "email": "foo@foo.com", 2023-10-05T19:28:26.174 INFO:teuthology.orchestra.run.smithi167.stdout: "suspended": 0, 2023-10-05T19:28:26.175 INFO:teuthology.orchestra.run.smithi167.stdout: "max_buckets": 4, 2023-10-05T19:28:26.175 INFO:teuthology.orchestra.run.smithi167.stdout: "subusers": [], 2023-10-05T19:28:26.175 INFO:teuthology.orchestra.run.smithi167.stdout: "keys": [ 2023-10-05T19:28:26.175 INFO:teuthology.orchestra.run.smithi167.stdout: { 2023-10-05T19:28:26.175 INFO:teuthology.orchestra.run.smithi167.stdout: "user": "foo", 2023-10-05T19:28:26.175 INFO:teuthology.orchestra.run.smithi167.stdout: "access_key": "9te6NH5mcdcq0Tc5i8i1", 2023-10-05T19:28:26.175 INFO:teuthology.orchestra.run.smithi167.stdout: "secret_key": "Ny4IOauQoL18Gp2zM7lC1vLmoawgqcYP/YGcWfXu" 2023-10-05T19:28:26.175 INFO:teuthology.orchestra.run.smithi167.stdout: } 2023-10-05T19:28:26.175 INFO:teuthology.orchestra.run.smithi167.stdout: ], 2023-10-05T19:28:26.175 INFO:teuthology.orchestra.run.smithi167.stdout: "swift_keys": [], 2023-10-05T19:28:26.175 INFO:teuthology.orchestra.run.smithi167.stdout: "caps": [], 2023-10-05T19:28:26.175 INFO:teuthology.orchestra.run.smithi167.stdout: "op_mask": "read, write, delete", 2023-10-05T19:28:26.176 INFO:teuthology.orchestra.run.smithi167.stdout: "default_placement": "", 2023-10-05T19:28:26.176 INFO:teuthology.orchestra.run.smithi167.stdout: "default_storage_class": "", 2023-10-05T19:28:26.176 INFO:teuthology.orchestra.run.smithi167.stdout: "placement_tags": [], 2023-10-05T19:28:26.176 INFO:teuthology.orchestra.run.smithi167.stdout: "bucket_quota": { 2023-10-05T19:28:26.176 INFO:teuthology.orchestra.run.smithi167.stdout: "enabled": false, 2023-10-05T19:28:26.176 INFO:teuthology.orchestra.run.smithi167.stdout: "check_on_raw": false, 2023-10-05T19:28:26.176 INFO:teuthology.orchestra.run.smithi167.stdout: "max_size": -1, 2023-10-05T19:28:26.176 INFO:teuthology.orchestra.run.smithi167.stdout: "max_size_kb": 0, 2023-10-05T19:28:26.176 INFO:teuthology.orchestra.run.smithi167.stdout: "max_objects": -1 2023-10-05T19:28:26.176 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2023-10-05T19:28:26.176 INFO:teuthology.orchestra.run.smithi167.stdout: "user_quota": { 2023-10-05T19:28:26.176 INFO:teuthology.orchestra.run.smithi167.stdout: "enabled": false, 2023-10-05T19:28:26.176 INFO:teuthology.orchestra.run.smithi167.stdout: "check_on_raw": false, 2023-10-05T19:28:26.177 INFO:teuthology.orchestra.run.smithi167.stdout: "max_size": -1, 2023-10-05T19:28:26.177 INFO:teuthology.orchestra.run.smithi167.stdout: "max_size_kb": 0, 2023-10-05T19:28:26.177 INFO:teuthology.orchestra.run.smithi167.stdout: "max_objects": -1 2023-10-05T19:28:26.177 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2023-10-05T19:28:26.177 INFO:teuthology.orchestra.run.smithi167.stdout: "temp_url_keys": [], 2023-10-05T19:28:26.177 INFO:teuthology.orchestra.run.smithi167.stdout: "type": "rgw", 2023-10-05T19:28:26.177 INFO:teuthology.orchestra.run.smithi167.stdout: "mfa_ids": [] 2023-10-05T19:28:26.177 INFO:teuthology.orchestra.run.smithi167.stdout:} 2023-10-05T19:28:26.177 INFO:teuthology.orchestra.run.smithi167.stdout: 2023-10-05T19:28:26.178 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.170+0000 7f90cfe1ae40 1 -- 172.21.15.167:0/3718246546 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:104 7.1f 7:f95f44c2:::notify.0:head [watch unwatch cookie 94139524475936] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x559e90e82990 con 0x7f90c0007030 2023-10-05T19:28:26.178 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.170+0000 7f90cd14e700 1 -- 172.21.15.167:0/3718246546 <== osd.2 v2:172.21.15.167:6816/30037 17 ==== osd_op_reply(104 notify.0 [watch unwatch cookie 94139524475936] v30'32 uv30 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f90b802f040 con 0x7f90c0007030 2023-10-05T19:28:26.178 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.170+0000 7f90cfe1ae40 20 remove_watcher() i=0 2023-10-05T19:28:26.178 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.170+0000 7f90cfe1ae40 2 removed watcher, disabling cache 2023-10-05T19:28:26.178 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.170+0000 7f90cfe1ae40 1 -- 172.21.15.167:0/3718246546 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:105 7.0 7:05bf5b68:::notify.1:head [watch unwatch cookie 94139524478224] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x559e90ea6c20 con 0x559e90e6d930 2023-10-05T19:28:26.179 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.174+0000 7f90ce150700 1 -- 172.21.15.167:0/3718246546 <== osd.1 v2:172.21.15.167:6800/30035 37 ==== osd_op_reply(105 notify.1 [watch unwatch cookie 94139524478224] v30'32 uv30 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f90c8050040 con 0x559e90e6d930 2023-10-05T19:28:26.179 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.174+0000 7f90cfe1ae40 20 remove_watcher() i=1 2023-10-05T19:28:26.179 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.174+0000 7f90cfe1ae40 1 -- 172.21.15.167:0/3718246546 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:106 7.15 7:a93a5511:::notify.2:head [watch unwatch cookie 94139524485072] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x559e90ea7510 con 0x559e90ea21c0 2023-10-05T19:28:26.182 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.178+0000 7f90cd94f700 1 -- 172.21.15.167:0/3718246546 <== osd.3 v2:172.21.15.167:6808/30036 24 ==== osd_op_reply(106 notify.2 [watch unwatch cookie 94139524485072] v30'32 uv30 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f90c0054040 con 0x559e90ea21c0 2023-10-05T19:28:26.182 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.178+0000 7f90cfe1ae40 20 remove_watcher() i=2 2023-10-05T19:28:26.182 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.178+0000 7f90cfe1ae40 1 -- 172.21.15.167:0/3718246546 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:107 7.e 7:7759931f:::notify.3:head [watch unwatch cookie 94139524488224] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x559e90ea8fd0 con 0x7f90c0007030 2023-10-05T19:28:26.185 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.178+0000 7f90cd14e700 1 -- 172.21.15.167:0/3718246546 <== osd.2 v2:172.21.15.167:6816/30037 18 ==== osd_op_reply(107 notify.3 [watch unwatch cookie 94139524488224] v30'32 uv30 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f90b802f040 con 0x7f90c0007030 2023-10-05T19:28:26.185 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.178+0000 7f90cfe1ae40 20 remove_watcher() i=3 2023-10-05T19:28:26.185 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.178+0000 7f90cfe1ae40 1 -- 172.21.15.167:0/3718246546 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:108 7.d 7:b4812045:::notify.4:head [watch unwatch cookie 94139524492336] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x559e90ea8fd0 con 0x559e90e6d930 2023-10-05T19:28:26.188 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.182+0000 7f90ce150700 1 -- 172.21.15.167:0/3718246546 <== osd.1 v2:172.21.15.167:6800/30035 38 ==== osd_op_reply(108 notify.4 [watch unwatch cookie 94139524492336] v30'32 uv30 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f90c8050040 con 0x559e90e6d930 2023-10-05T19:28:26.188 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.182+0000 7f90cfe1ae40 20 remove_watcher() i=4 2023-10-05T19:28:26.188 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.182+0000 7f90cfe1ae40 1 -- 172.21.15.167:0/3718246546 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:109 7.3 7:c609908c:::notify.5:head [watch unwatch cookie 94139524495888] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x559e90eaac30 con 0x7f90b8011040 2023-10-05T19:28:26.190 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.186+0000 7f90ce150700 1 -- 172.21.15.167:0/3718246546 <== osd.0 v2:172.21.15.118:6800/27690 29 ==== osd_op_reply(109 notify.5 [watch unwatch cookie 94139524495888] v30'32 uv30 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f90c8050040 con 0x7f90b8011040 2023-10-05T19:28:26.191 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.186+0000 7f90cfe1ae40 20 remove_watcher() i=5 2023-10-05T19:28:26.191 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.186+0000 7f90cfe1ae40 1 -- 172.21.15.167:0/3718246546 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:110 7.14 7:2b04a3e9:::notify.6:head [watch unwatch cookie 94139524494576] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x559e90ea8fd0 con 0x7f90b8011040 2023-10-05T19:28:26.194 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.186+0000 7f90ce150700 1 -- 172.21.15.167:0/3718246546 <== osd.0 v2:172.21.15.118:6800/27690 30 ==== osd_op_reply(110 notify.6 [watch unwatch cookie 94139524494576] v30'32 uv30 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f90c8050040 con 0x7f90b8011040 2023-10-05T19:28:26.194 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.186+0000 7f90cfe1ae40 20 remove_watcher() i=6 2023-10-05T19:28:26.194 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.186+0000 7f90cfe1ae40 1 -- 172.21.15.167:0/3718246546 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:111 7.9 7:93e5b521:::notify.7:head [watch unwatch cookie 94139524499808] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x559e90eaba10 con 0x7f90b8011040 2023-10-05T19:28:26.197 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.190+0000 7f90ce150700 1 -- 172.21.15.167:0/3718246546 <== osd.0 v2:172.21.15.118:6800/27690 31 ==== osd_op_reply(111 notify.7 [watch unwatch cookie 94139524499808] v30'32 uv30 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f90c8050040 con 0x7f90b8011040 2023-10-05T19:28:26.197 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.190+0000 7f90cfe1ae40 20 remove_watcher() i=7 2023-10-05T19:28:26.199 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.194+0000 7f909f7de700 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2023-10-05T19:28:26.199 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.194+0000 7f90cfe1ae40 1 -- 172.21.15.167:0/3718246546 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:112 6.4 6:22d26bf9:::data_loggenerations_metadata:head [watch unwatch cookie 94139524500912] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x559e90ee7fc0 con 0x559e90e6d930 2023-10-05T19:28:26.202 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.194+0000 7f90ce150700 1 -- 172.21.15.167:0/3718246546 <== osd.1 v2:172.21.15.167:6800/30035 39 ==== osd_op_reply(112 data_loggenerations_metadata [watch unwatch cookie 94139524500912] v30'55 uv1 ondisk = 0) v8 ==== 172+0+0 (crc 0 0 0) 0x7f90c8050040 con 0x559e90e6d930 2023-10-05T19:28:26.202 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.194+0000 7f90cfe1ae40 1 -- 172.21.15.167:0/3718246546 >> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] conn(0x7f90b8011040 msgr2=0x7f90b8014950 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-05T19:28:26.203 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.194+0000 7f90cfe1ae40 1 --2- 172.21.15.167:0/3718246546 >> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] conn(0x7f90b8011040 0x7f90b8014950 crc :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-05T19:28:26.203 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.194+0000 7f90cfe1ae40 1 -- 172.21.15.167:0/3718246546 >> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] conn(0x559e90e6d930 msgr2=0x559e90e9e770 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-05T19:28:26.203 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.194+0000 7f90cfe1ae40 1 --2- 172.21.15.167:0/3718246546 >> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] conn(0x559e90e6d930 0x559e90e9e770 crc :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-05T19:28:26.203 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.194+0000 7f90cfe1ae40 1 -- 172.21.15.167:0/3718246546 >> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] conn(0x7f90c0007030 msgr2=0x7f90c00249c0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-05T19:28:26.203 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.194+0000 7f90cfe1ae40 1 --2- 172.21.15.167:0/3718246546 >> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] conn(0x7f90c0007030 0x7f90c00249c0 crc :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-05T19:28:26.203 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.194+0000 7f90cfe1ae40 1 -- 172.21.15.167:0/3718246546 >> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] conn(0x559e90ea21c0 msgr2=0x559e90ea45a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-05T19:28:26.203 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.194+0000 7f90cfe1ae40 1 --2- 172.21.15.167:0/3718246546 >> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] conn(0x559e90ea21c0 0x559e90ea45a0 crc :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-05T19:28:26.203 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.194+0000 7f90cfe1ae40 1 -- 172.21.15.167:0/3718246546 >> [v2:172.21.15.167:6824/29889,v1:172.21.15.167:6825/29889] conn(0x7f90b0030bf0 msgr2=0x7f90b00330a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-05T19:28:26.204 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.194+0000 7f90cfe1ae40 1 --2- 172.21.15.167:0/3718246546 >> [v2:172.21.15.167:6824/29889,v1:172.21.15.167:6825/29889] conn(0x7f90b0030bf0 0x7f90b00330a0 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f90b80016f0 tx=0x7f90b80067f0 comp rx=0 tx=0).stop 2023-10-05T19:28:26.204 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.194+0000 7f90ce150700 1 -- 172.21.15.167:0/3718246546 reap_dead start 2023-10-05T19:28:26.204 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.194+0000 7f90cfe1ae40 1 -- 172.21.15.167:0/3718246546 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x559e90e6d020 msgr2=0x559e90e92d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-05T19:28:26.204 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.194+0000 7f90cfe1ae40 1 --2- 172.21.15.167:0/3718246546 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x559e90e6d020 0x559e90e92d00 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7f90c000c010 tx=0x7f90c0003050 comp rx=0 tx=0).stop 2023-10-05T19:28:26.204 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.198+0000 7f90cfe1ae40 1 -- 172.21.15.167:0/3718246546 shutdown_connections 2023-10-05T19:28:26.204 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.198+0000 7f90cfe1ae40 1 --2- 172.21.15.167:0/3718246546 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x559e90e6d020 0x559e90e92d00 unknown :-1 s=CLOSED pgs=197 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-05T19:28:26.204 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.198+0000 7f90cfe1ae40 1 -- 172.21.15.167:0/3718246546 >> 172.21.15.167:0/3718246546 conn(0x559e90de0870 msgr2=0x559e90e86730 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-05T19:28:26.204 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.198+0000 7f90cfe1ae40 1 -- 172.21.15.167:0/3718246546 shutdown_connections 2023-10-05T19:28:26.204 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.198+0000 7f90cfe1ae40 1 -- 172.21.15.167:0/3718246546 wait complete. 2023-10-05T19:28:26.205 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.198+0000 7f90cfe1ae40 1 -- 172.21.15.167:0/2107617791 >> [v2:172.21.15.167:6824/29889,v1:172.21.15.167:6825/29889] conn(0x7f9014030c00 msgr2=0x7f90140330b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-05T19:28:26.206 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.198+0000 7f90cfe1ae40 1 --2- 172.21.15.167:0/2107617791 >> [v2:172.21.15.167:6824/29889,v1:172.21.15.167:6825/29889] conn(0x7f9014030c00 0x7f90140330b0 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x559e90eb0460 tx=0x7f90c8000f90 comp rx=0 tx=0).stop 2023-10-05T19:28:26.206 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.198+0000 7f90cfe1ae40 1 -- 172.21.15.167:0/2107617791 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x559e90eba890 msgr2=0x559e90ec03f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-05T19:28:26.206 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.198+0000 7f90cfe1ae40 1 --2- 172.21.15.167:0/2107617791 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x559e90eba890 0x559e90ec03f0 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7f90b80049e0 tx=0x7f90b803c000 comp rx=0 tx=0).stop 2023-10-05T19:28:26.206 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.198+0000 7f90cfe1ae40 1 -- 172.21.15.167:0/2107617791 shutdown_connections 2023-10-05T19:28:26.206 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.198+0000 7f90cfe1ae40 1 --2- 172.21.15.167:0/2107617791 >> [v2:172.21.15.167:6824/29889,v1:172.21.15.167:6825/29889] conn(0x7f9014030c00 0x7f90140330b0 unknown :-1 s=CLOSED pgs=73 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-05T19:28:26.206 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.198+0000 7f90cfe1ae40 1 --2- 172.21.15.167:0/2107617791 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x559e90eba890 0x559e90ec03f0 unknown :-1 s=CLOSED pgs=199 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-05T19:28:26.207 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.198+0000 7f90cfe1ae40 1 -- 172.21.15.167:0/2107617791 >> 172.21.15.167:0/2107617791 conn(0x559e90eb5c50 msgr2=0x559e90eb3d50 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-05T19:28:26.207 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.198+0000 7f90cfe1ae40 1 -- 172.21.15.167:0/2107617791 shutdown_connections 2023-10-05T19:28:26.207 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.198+0000 7f90cfe1ae40 1 -- 172.21.15.167:0/2107617791 wait complete. 2023-10-05T19:28:26.209 DEBUG:tasks.util.rgw: json result: {'user_id': 'foo', 'display_name': 'Foo', 'email': 'foo@foo.com', 'suspended': 0, 'max_buckets': 4, 'subusers': [], 'keys': [{'user': 'foo', 'access_key': '9te6NH5mcdcq0Tc5i8i1', 'secret_key': 'Ny4IOauQoL18Gp2zM7lC1vLmoawgqcYP/YGcWfXu'}], 'swift_keys': [], 'caps': [], 'op_mask': 'read, write, delete', 'default_placement': '', 'default_storage_class': '', 'placement_tags': [], 'bucket_quota': {'enabled': False, 'check_on_raw': False, 'max_size': -1, 'max_size_kb': 0, 'max_objects': -1}, 'user_quota': {'enabled': False, 'check_on_raw': False, 'max_size': -1, 'max_size_kb': 0, 'max_objects': -1}, 'temp_url_keys': [], 'type': 'rgw', 'mfa_ids': []} 2023-10-05T19:28:26.209 INFO:tasks.util.rgw:rgwadmin: client.0 : ['user', 'suspend', '--uid', 'foo'] 2023-10-05T19:28:26.209 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'ceph', 'user', 'suspend', '--uid', 'foo'] 2023-10-05T19:28:26.210 DEBUG:teuthology.orchestra.run.smithi167:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster ceph user suspend --uid foo 2023-10-05T19:28:26.290 INFO:teuthology.orchestra.run.smithi167.stderr:ignoring --setuser ceph since I am not root 2023-10-05T19:28:26.291 INFO:teuthology.orchestra.run.smithi167.stderr:ignoring --setgroup ceph since I am not root 2023-10-05T19:28:26.291 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.286+0000 7f6b5f2a3e40 1 Processor -- start 2023-10-05T19:28:26.291 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.286+0000 7f6b5f2a3e40 1 -- start start 2023-10-05T19:28:26.291 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.286+0000 7f6b5f2a3e40 1 --2- >> v2:172.21.15.118:3300/0 conn(0x5584aaad42f0 0x5584aaad66d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-05T19:28:26.292 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.286+0000 7f6b5f2a3e40 1 -- --> v1:172.21.15.118:6789/0 -- auth(proto 0 26 bytes epoch 0) v1 -- 0x5584aaa1cbd0 con 0x5584aaa443d0 2023-10-05T19:28:26.292 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.286+0000 7f6b5f2a3e40 1 -- --> v2:172.21.15.118:3300/0 -- mon_getmap magic: 0 v1 -- 0x5584aa9d9ee0 con 0x5584aaad42f0 2023-10-05T19:28:26.292 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.286+0000 7f6b5c5d7700 1 --2- >> v2:172.21.15.118:3300/0 conn(0x5584aaad42f0 0x5584aaad66d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-05T19:28:26.292 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.286+0000 7f6b5c5d7700 1 --2- >> v2:172.21.15.118:3300/0 conn(0x5584aaad42f0 0x5584aaad66d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.118:3300/0 says I am v2:172.21.15.167:57274/0 (socket says 172.21.15.167:57274) 2023-10-05T19:28:26.293 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.286+0000 7f6b5c5d7700 1 -- 172.21.15.167:0/3621157824 learned_addr learned my addr 172.21.15.167:0/3621157824 (peer_addr_for_me v2:172.21.15.167:0/0) 2023-10-05T19:28:26.293 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.286+0000 7f6b5c5d7700 1 -- 172.21.15.167:0/3621157824 >> v1:172.21.15.118:6789/0 conn(0x5584aaa443d0 legacy=0x5584aaad2bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-05T19:28:26.293 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.286+0000 7f6b5c5d7700 1 -- 172.21.15.167:0/3621157824 --> v2:172.21.15.118:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5584aaad7c20 con 0x5584aaad42f0 2023-10-05T19:28:26.294 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.286+0000 7f6b5c5d7700 1 --2- 172.21.15.167:0/3621157824 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5584aaad42f0 0x5584aaad66d0 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7f6b4400a5c0 tx=0x7f6b44005e00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3977872ac6b8d1ad server_cookie=0 in_seq=0 out_seq=0 2023-10-05T19:28:26.294 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.286+0000 7f6b5bdd6700 1 -- 172.21.15.167:0/3621157824 <== mon.0 v2:172.21.15.118:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f6b4400e510 con 0x5584aaad42f0 2023-10-05T19:28:26.294 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.286+0000 7f6b5bdd6700 1 -- 172.21.15.167:0/3621157824 <== mon.0 v2:172.21.15.118:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6b4400e670 con 0x5584aaad42f0 2023-10-05T19:28:26.295 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.286+0000 7f6b5bdd6700 1 -- 172.21.15.167:0/3621157824 <== mon.0 v2:172.21.15.118:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f6b4400e920 con 0x5584aaad42f0 2023-10-05T19:28:26.295 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.286+0000 7f6b5f2a3e40 1 -- 172.21.15.167:0/3621157824 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5584aaad42f0 msgr2=0x5584aaad66d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-05T19:28:26.295 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.286+0000 7f6b5f2a3e40 1 --2- 172.21.15.167:0/3621157824 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5584aaad42f0 0x5584aaad66d0 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7f6b4400a5c0 tx=0x7f6b44005e00 comp rx=0 tx=0).stop 2023-10-05T19:28:26.295 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.286+0000 7f6b5f2a3e40 1 -- 172.21.15.167:0/3621157824 shutdown_connections 2023-10-05T19:28:26.295 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.286+0000 7f6b5f2a3e40 1 --2- 172.21.15.167:0/3621157824 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5584aaad42f0 0x5584aaad66d0 unknown :-1 s=CLOSED pgs=201 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-05T19:28:26.295 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.290+0000 7f6b5f2a3e40 1 -- 172.21.15.167:0/3621157824 >> 172.21.15.167:0/3621157824 conn(0x5584aaaca720 msgr2=0x5584aaaccb30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-05T19:28:26.295 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.290+0000 7f6b5f2a3e40 1 -- 172.21.15.167:0/3621157824 shutdown_connections 2023-10-05T19:28:26.295 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.290+0000 7f6b5f2a3e40 1 -- 172.21.15.167:0/3621157824 wait complete. 2023-10-05T19:28:26.296 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.290+0000 7f6b5f2a3e40 1 Processor -- start 2023-10-05T19:28:26.296 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.290+0000 7f6b5f2a3e40 1 -- start start 2023-10-05T19:28:26.296 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.290+0000 7f6b5f2a3e40 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5584aaa443d0 0x5584aaaeef90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-05T19:28:26.296 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.290+0000 7f6b5f2a3e40 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x5584aa99de30 con 0x5584aaa443d0 2023-10-05T19:28:26.297 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.290+0000 7f6b5cdd8700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5584aaa443d0 0x5584aaaeef90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-05T19:28:26.297 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.290+0000 7f6b5cdd8700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5584aaa443d0 0x5584aaaeef90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.118:3300/0 says I am v2:172.21.15.167:57288/0 (socket says 172.21.15.167:57288) 2023-10-05T19:28:26.297 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.290+0000 7f6b5cdd8700 1 -- 172.21.15.167:0/3889897730 learned_addr learned my addr 172.21.15.167:0/3889897730 (peer_addr_for_me v2:172.21.15.167:0/0) 2023-10-05T19:28:26.298 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.290+0000 7f6b5cdd8700 1 -- 172.21.15.167:0/3889897730 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6b4400a180 con 0x5584aaa443d0 2023-10-05T19:28:26.298 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.290+0000 7f6b5cdd8700 1 --2- 172.21.15.167:0/3889897730 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5584aaa443d0 0x5584aaaeef90 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7f6b4c001d30 tx=0x7f6b4c001900 comp rx=0 tx=0).ready entity=mon.0 client_cookie=da274fa62d22a9e4 server_cookie=0 in_seq=0 out_seq=0 2023-10-05T19:28:26.298 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.290+0000 7f6b5add4700 1 -- 172.21.15.167:0/3889897730 <== mon.0 v2:172.21.15.118:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f6b4c014070 con 0x5584aaa443d0 2023-10-05T19:28:26.298 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.294+0000 7f6b5add4700 1 -- 172.21.15.167:0/3889897730 <== mon.0 v2:172.21.15.118:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6b4c008070 con 0x5584aaa443d0 2023-10-05T19:28:26.298 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.294+0000 7f6b5add4700 1 -- 172.21.15.167:0/3889897730 <== mon.0 v2:172.21.15.118:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f6b4c00bd30 con 0x5584aaa443d0 2023-10-05T19:28:26.299 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.294+0000 7f6b5f2a3e40 1 -- 172.21.15.167:0/3889897730 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5584aaa443d0 msgr2=0x5584aaaeef90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-05T19:28:26.299 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.294+0000 7f6b5f2a3e40 1 --2- 172.21.15.167:0/3889897730 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5584aaa443d0 0x5584aaaeef90 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7f6b4c001d30 tx=0x7f6b4c001900 comp rx=0 tx=0).stop 2023-10-05T19:28:26.300 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.294+0000 7f6b5f2a3e40 1 -- 172.21.15.167:0/3889897730 shutdown_connections 2023-10-05T19:28:26.300 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.294+0000 7f6b5f2a3e40 1 --2- 172.21.15.167:0/3889897730 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5584aaa443d0 0x5584aaaeef90 unknown :-1 s=CLOSED pgs=202 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-05T19:28:26.300 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.294+0000 7f6b5f2a3e40 1 -- 172.21.15.167:0/3889897730 >> 172.21.15.167:0/3889897730 conn(0x5584aaaca720 msgr2=0x5584aaaeab50 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-05T19:28:26.300 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.294+0000 7f6b5f2a3e40 1 -- 172.21.15.167:0/3889897730 shutdown_connections 2023-10-05T19:28:26.300 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.294+0000 7f6b5f2a3e40 1 -- 172.21.15.167:0/3889897730 wait complete. 2023-10-05T19:28:26.301 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.294+0000 7f6b5f2a3e40 1 Processor -- start 2023-10-05T19:28:26.301 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.294+0000 7f6b5f2a3e40 1 -- start start 2023-10-05T19:28:26.301 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.294+0000 7f6b5f2a3e40 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5584aaa443d0 0x5584aaaf5d60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-05T19:28:26.301 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.294+0000 7f6b5f2a3e40 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x5584aa970ed0 con 0x5584aaa443d0 2023-10-05T19:28:26.302 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.294+0000 7f6b5cdd8700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5584aaa443d0 0x5584aaaf5d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-05T19:28:26.302 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.294+0000 7f6b5cdd8700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5584aaa443d0 0x5584aaaf5d60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.118:3300/0 says I am v2:172.21.15.167:57292/0 (socket says 172.21.15.167:57292) 2023-10-05T19:28:26.302 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.294+0000 7f6b5cdd8700 1 -- 172.21.15.167:0/581366395 learned_addr learned my addr 172.21.15.167:0/581366395 (peer_addr_for_me v2:172.21.15.167:0/0) 2023-10-05T19:28:26.303 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.294+0000 7f6b5cdd8700 1 -- 172.21.15.167:0/581366395 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6b4c00a040 con 0x5584aaa443d0 2023-10-05T19:28:26.303 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.294+0000 7f6b5cdd8700 1 --2- 172.21.15.167:0/581366395 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5584aaa443d0 0x5584aaaf5d60 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x5584aaaeb670 tx=0x7f6b4c016000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-05T19:28:26.303 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.298+0000 7f6b595d1700 1 -- 172.21.15.167:0/581366395 <== mon.0 v2:172.21.15.118:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f6b4c00bee0 con 0x5584aaa443d0 2023-10-05T19:28:26.303 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.298+0000 7f6b595d1700 1 -- 172.21.15.167:0/581366395 <== mon.0 v2:172.21.15.118:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6b4c021070 con 0x5584aaa443d0 2023-10-05T19:28:26.303 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.298+0000 7f6b5f2a3e40 1 -- 172.21.15.167:0/581366395 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x5584aaaf62a0 con 0x5584aaa443d0 2023-10-05T19:28:26.303 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.298+0000 7f6b595d1700 1 -- 172.21.15.167:0/581366395 <== mon.0 v2:172.21.15.118:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f6b4c01a600 con 0x5584aaa443d0 2023-10-05T19:28:26.303 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.298+0000 7f6b5f2a3e40 1 -- 172.21.15.167:0/581366395 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x5584aaaf6740 con 0x5584aaa443d0 2023-10-05T19:28:26.304 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.298+0000 7f6b595d1700 1 -- 172.21.15.167:0/581366395 <== mon.0 v2:172.21.15.118:3300/0 4 ==== mgrmap(e 5) v1 ==== 40329+0+0 (secure 0 0 0) 0x7f6b4c01a800 con 0x5584aaa443d0 2023-10-05T19:28:26.304 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.298+0000 7f6b595d1700 1 --2- 172.21.15.167:0/581366395 >> [v2:172.21.15.167:6824/29889,v1:172.21.15.167:6825/29889] conn(0x7f6b3c030c00 0x7f6b3c0330b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-05T19:28:26.304 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.298+0000 7f6b595d1700 1 -- 172.21.15.167:0/581366395 <== mon.0 v2:172.21.15.118:3300/0 5 ==== osd_map(30..30 src has 1..30) v4 ==== 6057+0+0 (secure 0 0 0) 0x7f6b4c044e90 con 0x5584aaa443d0 2023-10-05T19:28:26.304 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.298+0000 7f6b5c5d7700 1 --2- 172.21.15.167:0/581366395 >> [v2:172.21.15.167:6824/29889,v1:172.21.15.167:6825/29889] conn(0x7f6b3c030c00 0x7f6b3c0330b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-05T19:28:26.305 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.298+0000 7f6b5c5d7700 1 --2- 172.21.15.167:0/581366395 >> [v2:172.21.15.167:6824/29889,v1:172.21.15.167:6825/29889] conn(0x7f6b3c030c00 0x7f6b3c0330b0 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f6b44000f80 tx=0x7f6b44013000 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-05T19:28:26.305 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.298+0000 7f6b5f2a3e40 20 rados->read ofs=0 len=0 2023-10-05T19:28:26.306 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.298+0000 7f6b5f2a3e40 1 --2- 172.21.15.167:0/581366395 >> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] conn(0x5584aaad42f0 0x5584aab01800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-05T19:28:26.306 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.298+0000 7f6b5f2a3e40 1 -- 172.21.15.167:0/581366395 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:1 5.12 5:49953fa1:::default.realm:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x5584aab01f30 con 0x5584aaad42f0 2023-10-05T19:28:26.306 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.298+0000 7f6b5d5d9700 1 --2- 172.21.15.167:0/581366395 >> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] conn(0x5584aaad42f0 0x5584aab01800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-05T19:28:26.306 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.298+0000 7f6b5d5d9700 1 --2- 172.21.15.167:0/581366395 >> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] conn(0x5584aaad42f0 0x5584aab01800 crc :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-05T19:28:26.307 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.302+0000 7f6b5d5d9700 1 -- 172.21.15.167:0/581366395 <== osd.1 v2:172.21.15.167:6800/30035 1 ==== osd_op_reply(1 default.realm [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 ==== 157+0+0 (crc 0 0 0) 0x7f6b5400a500 con 0x5584aaad42f0 2023-10-05T19:28:26.307 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.302+0000 7f6b5f2a3e40 20 rados_obj.operate() r=-2 bl.length=0 2023-10-05T19:28:26.307 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.302+0000 7f6b5f2a3e40 20 realm 2023-10-05T19:28:26.307 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.302+0000 7f6b5f2a3e40 20 rados->read ofs=0 len=0 2023-10-05T19:28:26.307 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.302+0000 7f6b5f2a3e40 1 -- 172.21.15.167:0/581366395 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:2 5.12 5:49953fa1:::default.realm:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x5584aaafede0 con 0x5584aaad42f0 2023-10-05T19:28:26.308 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.302+0000 7f6b5d5d9700 1 -- 172.21.15.167:0/581366395 <== osd.1 v2:172.21.15.167:6800/30035 2 ==== osd_op_reply(2 default.realm [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 ==== 157+0+0 (crc 0 0 0) 0x7f6b5400a500 con 0x5584aaad42f0 2023-10-05T19:28:26.308 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.302+0000 7f6b5f2a3e40 20 rados_obj.operate() r=-2 bl.length=0 2023-10-05T19:28:26.308 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.302+0000 7f6b5f2a3e40 4 RGWPeriod::init failed to init realm id : (2) No such file or directory 2023-10-05T19:28:26.308 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.302+0000 7f6b5f2a3e40 20 rados->read ofs=0 len=0 2023-10-05T19:28:26.308 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.302+0000 7f6b5f2a3e40 1 -- 172.21.15.167:0/581366395 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:3 5.12 5:49953fa1:::default.realm:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x5584aab02a10 con 0x5584aaad42f0 2023-10-05T19:28:26.308 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.302+0000 7f6b5d5d9700 1 -- 172.21.15.167:0/581366395 <== osd.1 v2:172.21.15.167:6800/30035 3 ==== osd_op_reply(3 default.realm [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 ==== 157+0+0 (crc 0 0 0) 0x7f6b5400a500 con 0x5584aaad42f0 2023-10-05T19:28:26.308 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.302+0000 7f6b5f2a3e40 20 rados_obj.operate() r=-2 bl.length=0 2023-10-05T19:28:26.308 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.302+0000 7f6b5f2a3e40 20 rados->read ofs=0 len=0 2023-10-05T19:28:26.308 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.302+0000 7f6b5f2a3e40 1 -- 172.21.15.167:0/581366395 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:4 5.f 5:f43b6ece:::zone_names.default:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x5584aab02620 con 0x5584aaad42f0 2023-10-05T19:28:26.310 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.302+0000 7f6b5d5d9700 1 -- 172.21.15.167:0/581366395 <== osd.1 v2:172.21.15.167:6800/30035 4 ==== osd_op_reply(4 zone_names.default [read 0~46 out=46b] v0'0 uv1 ondisk = 0) v8 ==== 162+0+46 (crc 0 0 0) 0x7f6b5400a500 con 0x5584aaad42f0 2023-10-05T19:28:26.310 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.302+0000 7f6b5f2a3e40 20 rados_obj.operate() r=0 bl.length=46 2023-10-05T19:28:26.310 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.302+0000 7f6b5f2a3e40 20 rados->read ofs=0 len=0 2023-10-05T19:28:26.310 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.302+0000 7f6b5f2a3e40 1 -- 172.21.15.167:0/581366395 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:5 5.1d 5:b8374eb4:::zone_info.9a9df965-6612-45d4-a45f-6e3c889bacc2:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x5584aab038a0 con 0x5584aaad42f0 2023-10-05T19:28:26.310 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.302+0000 7f6b5d5d9700 1 -- 172.21.15.167:0/581366395 <== osd.1 v2:172.21.15.167:6800/30035 5 ==== osd_op_reply(5 zone_info.9a9df965-6612-45d4-a45f-6e3c889bacc2 [read 0~874 out=874b] v0'0 uv2 ondisk = 0) v8 ==== 190+0+874 (crc 0 0 0) 0x7f6b5400a500 con 0x5584aaad42f0 2023-10-05T19:28:26.310 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.302+0000 7f6b5f2a3e40 20 rados_obj.operate() r=0 bl.length=874 2023-10-05T19:28:26.310 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.302+0000 7f6b5f2a3e40 20 searching for the correct realm 2023-10-05T19:28:26.310 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.302+0000 7f6b5f2a3e40 1 --2- 172.21.15.167:0/581366395 >> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] conn(0x5584aab05660 0x5584aab07a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-05T19:28:26.310 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.302+0000 7f6b5f2a3e40 1 -- 172.21.15.167:0/581366395 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:6 5.0 5:00000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x5584aab07f80 con 0x5584aab05660 2023-10-05T19:28:26.313 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.302+0000 7f6b5cdd8700 1 --2- 172.21.15.167:0/581366395 >> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] conn(0x5584aab05660 0x5584aab07a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-05T19:28:26.313 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.302+0000 7f6b5cdd8700 1 --2- 172.21.15.167:0/581366395 >> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] conn(0x5584aab05660 0x5584aab07a40 crc :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-05T19:28:26.315 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.306+0000 7f6b5cdd8700 1 -- 172.21.15.167:0/581366395 <== osd.3 v2:172.21.15.167:6808/30036 1 ==== osd_op_reply(6 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f6b4c016070 con 0x5584aab05660 2023-10-05T19:28:26.315 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.306+0000 7f6b5cdd8700 1 --2- 172.21.15.167:0/581366395 >> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] conn(0x7f6b4c0176e0 0x7f6b4c0420e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-05T19:28:26.315 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.306+0000 7f6b5cdd8700 1 -- 172.21.15.167:0/581366395 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:7 5.10 5:08000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x5584aab07f80 con 0x7f6b4c0176e0 2023-10-05T19:28:26.315 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.306+0000 7f6b5c5d7700 1 --2- 172.21.15.167:0/581366395 >> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] conn(0x7f6b4c0176e0 0x7f6b4c0420e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-05T19:28:26.316 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.306+0000 7f6b5c5d7700 1 --2- 172.21.15.167:0/581366395 >> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] conn(0x7f6b4c0176e0 0x7f6b4c0420e0 crc :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-05T19:28:26.316 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.306+0000 7f6b5c5d7700 1 -- 172.21.15.167:0/581366395 <== osd.2 v2:172.21.15.167:6816/30037 1 ==== osd_op_reply(7 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f6b44017050 con 0x7f6b4c0176e0 2023-10-05T19:28:26.316 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.306+0000 7f6b5c5d7700 1 --2- 172.21.15.167:0/581366395 >> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] conn(0x7f6b44010070 0x7f6b4401c430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-05T19:28:26.316 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.306+0000 7f6b5c5d7700 1 -- 172.21.15.167:0/581366395 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:8 5.8 5:10000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x5584aab07f80 con 0x7f6b44010070 2023-10-05T19:28:26.316 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.306+0000 7f6b5d5d9700 1 --2- 172.21.15.167:0/581366395 >> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] conn(0x7f6b44010070 0x7f6b4401c430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-05T19:28:26.316 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.306+0000 7f6b5d5d9700 1 --2- 172.21.15.167:0/581366395 >> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] conn(0x7f6b44010070 0x7f6b4401c430 crc :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-05T19:28:26.316 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.310+0000 7f6b5d5d9700 1 -- 172.21.15.167:0/581366395 <== osd.0 v2:172.21.15.118:6800/27690 1 ==== osd_op_reply(8 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f6b54014040 con 0x7f6b44010070 2023-10-05T19:28:26.317 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.310+0000 7f6b5d5d9700 1 -- 172.21.15.167:0/581366395 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:9 5.18 5:18000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x5584aab07f80 con 0x5584aaad42f0 2023-10-05T19:28:26.317 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.310+0000 7f6b5d5d9700 1 -- 172.21.15.167:0/581366395 <== osd.1 v2:172.21.15.167:6800/30035 6 ==== osd_op_reply(9 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f6b54014040 con 0x5584aaad42f0 2023-10-05T19:28:26.317 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.310+0000 7f6b5d5d9700 1 -- 172.21.15.167:0/581366395 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:10 5.4 5:20000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x5584aab07f80 con 0x5584aaad42f0 2023-10-05T19:28:26.317 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.310+0000 7f6b5d5d9700 1 -- 172.21.15.167:0/581366395 <== osd.1 v2:172.21.15.167:6800/30035 7 ==== osd_op_reply(10 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f6b54014040 con 0x5584aaad42f0 2023-10-05T19:28:26.317 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.310+0000 7f6b5d5d9700 1 -- 172.21.15.167:0/581366395 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:11 5.14 5:28000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x5584aab07f80 con 0x7f6b44010070 2023-10-05T19:28:26.319 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.310+0000 7f6b5d5d9700 1 -- 172.21.15.167:0/581366395 <== osd.0 v2:172.21.15.118:6800/27690 2 ==== osd_op_reply(11 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f6b54014040 con 0x7f6b44010070 2023-10-05T19:28:26.319 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.310+0000 7f6b5d5d9700 1 -- 172.21.15.167:0/581366395 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:12 5.c 5:30000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x5584aab07f80 con 0x7f6b44010070 2023-10-05T19:28:26.319 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.310+0000 7f6b5d5d9700 1 -- 172.21.15.167:0/581366395 <== osd.0 v2:172.21.15.118:6800/27690 3 ==== osd_op_reply(12 [pgnls start_epoch 30 out=112b] v23'1 uv1 ondisk = 1) v8 ==== 144+0+112 (crc 0 0 0) 0x7f6b54014040 con 0x7f6b44010070 2023-10-05T19:28:26.319 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.310+0000 7f6b5d5d9700 1 -- 172.21.15.167:0/581366395 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:13 5.1c 5:38000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x5584aab07f80 con 0x7f6b4c0176e0 2023-10-05T19:28:26.319 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.310+0000 7f6b5c5d7700 1 -- 172.21.15.167:0/581366395 <== osd.2 v2:172.21.15.167:6816/30037 2 ==== osd_op_reply(13 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f6b44017050 con 0x7f6b4c0176e0 2023-10-05T19:28:26.319 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.310+0000 7f6b5c5d7700 1 -- 172.21.15.167:0/581366395 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:14 5.2 5:40000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x5584aab07f80 con 0x5584aab05660 2023-10-05T19:28:26.320 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.314+0000 7f6b5cdd8700 1 -- 172.21.15.167:0/581366395 <== osd.3 v2:172.21.15.167:6808/30036 2 ==== osd_op_reply(14 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f6b4c016070 con 0x5584aab05660 2023-10-05T19:28:26.320 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.314+0000 7f6b5cdd8700 1 -- 172.21.15.167:0/581366395 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:15 5.12 5:48000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x5584aab07f80 con 0x5584aaad42f0 2023-10-05T19:28:26.320 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.314+0000 7f6b5d5d9700 1 -- 172.21.15.167:0/581366395 <== osd.1 v2:172.21.15.167:6800/30035 8 ==== osd_op_reply(15 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f6b54014040 con 0x5584aaad42f0 2023-10-05T19:28:26.320 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.314+0000 7f6b5d5d9700 1 -- 172.21.15.167:0/581366395 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:16 5.a 5:50000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x5584aab07f80 con 0x7f6b44010070 2023-10-05T19:28:26.321 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.314+0000 7f6b5d5d9700 1 -- 172.21.15.167:0/581366395 <== osd.0 v2:172.21.15.118:6800/27690 4 ==== osd_op_reply(16 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f6b54022040 con 0x7f6b44010070 2023-10-05T19:28:26.321 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.314+0000 7f6b5d5d9700 1 -- 172.21.15.167:0/581366395 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:17 5.1a 5:58000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x5584aab07f80 con 0x5584aaad42f0 2023-10-05T19:28:26.321 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.314+0000 7f6b5d5d9700 1 -- 172.21.15.167:0/581366395 <== osd.1 v2:172.21.15.167:6800/30035 9 ==== osd_op_reply(17 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f6b54022040 con 0x5584aaad42f0 2023-10-05T19:28:26.321 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.314+0000 7f6b5d5d9700 1 -- 172.21.15.167:0/581366395 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:18 5.6 5:60000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x5584aab07f80 con 0x7f6b4c0176e0 2023-10-05T19:28:26.321 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.314+0000 7f6b5c5d7700 1 -- 172.21.15.167:0/581366395 <== osd.2 v2:172.21.15.167:6816/30037 3 ==== osd_op_reply(18 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f6b44017050 con 0x7f6b4c0176e0 2023-10-05T19:28:26.322 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.314+0000 7f6b5c5d7700 1 -- 172.21.15.167:0/581366395 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:19 5.16 5:68000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x5584aab07f80 con 0x5584aab05660 2023-10-05T19:28:26.322 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.314+0000 7f6b5cdd8700 1 -- 172.21.15.167:0/581366395 <== osd.3 v2:172.21.15.167:6808/30036 3 ==== osd_op_reply(19 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f6b4c016070 con 0x5584aab05660 2023-10-05T19:28:26.322 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.314+0000 7f6b5cdd8700 1 -- 172.21.15.167:0/581366395 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:20 5.e 5:70000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x5584aab07f80 con 0x5584aab05660 2023-10-05T19:28:26.323 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.318+0000 7f6b5cdd8700 1 -- 172.21.15.167:0/581366395 <== osd.3 v2:172.21.15.167:6808/30036 4 ==== osd_op_reply(20 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f6b4c016070 con 0x5584aab05660 2023-10-05T19:28:26.323 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.318+0000 7f6b5cdd8700 1 -- 172.21.15.167:0/581366395 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:21 5.1e 5:78000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x5584aab07f80 con 0x7f6b44010070 2023-10-05T19:28:26.323 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.318+0000 7f6b5d5d9700 1 -- 172.21.15.167:0/581366395 <== osd.0 v2:172.21.15.118:6800/27690 5 ==== osd_op_reply(21 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f6b54022040 con 0x7f6b44010070 2023-10-05T19:28:26.323 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.318+0000 7f6b5d5d9700 1 -- 172.21.15.167:0/581366395 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:22 5.1 5:80000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x5584aab07f80 con 0x5584aab05660 2023-10-05T19:28:26.324 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.318+0000 7f6b5cdd8700 1 -- 172.21.15.167:0/581366395 <== osd.3 v2:172.21.15.167:6808/30036 5 ==== osd_op_reply(22 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f6b4c039040 con 0x5584aab05660 2023-10-05T19:28:26.324 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.318+0000 7f6b5cdd8700 1 -- 172.21.15.167:0/581366395 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:23 5.11 5:88000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x5584aab07f80 con 0x5584aab05660 2023-10-05T19:28:26.325 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.318+0000 7f6b5cdd8700 1 -- 172.21.15.167:0/581366395 <== osd.3 v2:172.21.15.167:6808/30036 6 ==== osd_op_reply(23 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f6b4c039040 con 0x5584aab05660 2023-10-05T19:28:26.325 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.318+0000 7f6b5cdd8700 1 -- 172.21.15.167:0/581366395 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:24 5.9 5:90000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x7f6b4c03d040 con 0x5584aaad42f0 2023-10-05T19:28:26.326 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.318+0000 7f6b5d5d9700 1 -- 172.21.15.167:0/581366395 <== osd.1 v2:172.21.15.167:6800/30035 10 ==== osd_op_reply(24 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f6b54022040 con 0x5584aaad42f0 2023-10-05T19:28:26.326 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.318+0000 7f6b5d5d9700 1 -- 172.21.15.167:0/581366395 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:25 5.19 5:98000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x7f6b4c03d040 con 0x5584aaad42f0 2023-10-05T19:28:26.326 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.322+0000 7f6b5d5d9700 1 -- 172.21.15.167:0/581366395 <== osd.1 v2:172.21.15.167:6800/30035 11 ==== osd_op_reply(25 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f6b54022040 con 0x5584aaad42f0 2023-10-05T19:28:26.326 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.322+0000 7f6b5d5d9700 1 -- 172.21.15.167:0/581366395 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:26 5.5 5:a0000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x7f6b4c03d040 con 0x7f6b44010070 2023-10-05T19:28:26.327 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.322+0000 7f6b5d5d9700 1 -- 172.21.15.167:0/581366395 <== osd.0 v2:172.21.15.118:6800/27690 6 ==== osd_op_reply(26 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f6b54022040 con 0x7f6b44010070 2023-10-05T19:28:26.327 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.322+0000 7f6b5d5d9700 1 -- 172.21.15.167:0/581366395 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:27 5.15 5:a8000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x7f6b4c03d040 con 0x5584aab05660 2023-10-05T19:28:26.328 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.322+0000 7f6b5cdd8700 1 -- 172.21.15.167:0/581366395 <== osd.3 v2:172.21.15.167:6808/30036 7 ==== osd_op_reply(27 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f6b4c039040 con 0x5584aab05660 2023-10-05T19:28:26.328 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.322+0000 7f6b5cdd8700 1 -- 172.21.15.167:0/581366395 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:28 5.d 5:b0000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x7f6b4c03d040 con 0x7f6b4c0176e0 2023-10-05T19:28:26.329 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.322+0000 7f6b5c5d7700 1 -- 172.21.15.167:0/581366395 <== osd.2 v2:172.21.15.167:6816/30037 4 ==== osd_op_reply(28 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f6b44017050 con 0x7f6b4c0176e0 2023-10-05T19:28:26.329 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.322+0000 7f6b5c5d7700 1 -- 172.21.15.167:0/581366395 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:29 5.1d 5:b8000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x7f6b4c03d040 con 0x5584aaad42f0 2023-10-05T19:28:26.329 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.322+0000 7f6b5d5d9700 1 -- 172.21.15.167:0/581366395 <== osd.1 v2:172.21.15.167:6800/30035 12 ==== osd_op_reply(29 [pgnls start_epoch 30 out=107b] v30'2 uv2 ondisk = 1) v8 ==== 144+0+107 (crc 0 0 0) 0x7f6b54022040 con 0x5584aaad42f0 2023-10-05T19:28:26.330 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.322+0000 7f6b5d5d9700 1 -- 172.21.15.167:0/581366395 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:30 5.3 5:c0000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x7f6b4c03d040 con 0x7f6b44010070 2023-10-05T19:28:26.330 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.326+0000 7f6b5d5d9700 1 -- 172.21.15.167:0/581366395 <== osd.0 v2:172.21.15.118:6800/27690 7 ==== osd_op_reply(30 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f6b54022040 con 0x7f6b44010070 2023-10-05T19:28:26.330 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.326+0000 7f6b5d5d9700 1 -- 172.21.15.167:0/581366395 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:31 5.13 5:c8000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x7f6b4c03d040 con 0x5584aab05660 2023-10-05T19:28:26.331 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.326+0000 7f6b5cdd8700 1 -- 172.21.15.167:0/581366395 <== osd.3 v2:172.21.15.167:6808/30036 8 ==== osd_op_reply(31 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f6b4c039040 con 0x5584aab05660 2023-10-05T19:28:26.331 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.326+0000 7f6b5cdd8700 1 -- 172.21.15.167:0/581366395 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:32 5.b 5:d0000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x7f6b4c03d040 con 0x7f6b44010070 2023-10-05T19:28:26.332 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.326+0000 7f6b5d5d9700 1 -- 172.21.15.167:0/581366395 <== osd.0 v2:172.21.15.118:6800/27690 8 ==== osd_op_reply(32 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f6b54022040 con 0x7f6b44010070 2023-10-05T19:28:26.332 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.326+0000 7f6b5d5d9700 1 -- 172.21.15.167:0/581366395 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:33 5.1b 5:d8000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x7f6b4c03d040 con 0x7f6b4c0176e0 2023-10-05T19:28:26.333 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.326+0000 7f6b5c5d7700 1 -- 172.21.15.167:0/581366395 <== osd.2 v2:172.21.15.167:6816/30037 5 ==== osd_op_reply(33 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f6b44017050 con 0x7f6b4c0176e0 2023-10-05T19:28:26.333 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.326+0000 7f6b5c5d7700 1 -- 172.21.15.167:0/581366395 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:34 5.7 5:e0000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x7f6b4c03d040 con 0x5584aab05660 2023-10-05T19:28:26.334 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.330+0000 7f6b5cdd8700 1 -- 172.21.15.167:0/581366395 <== osd.3 v2:172.21.15.167:6808/30036 9 ==== osd_op_reply(34 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f6b4c039040 con 0x5584aab05660 2023-10-05T19:28:26.334 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.330+0000 7f6b5cdd8700 1 -- 172.21.15.167:0/581366395 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:35 5.17 5:e8000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x7f6b4c03d040 con 0x7f6b44010070 2023-10-05T19:28:26.335 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.330+0000 7f6b5d5d9700 1 -- 172.21.15.167:0/581366395 <== osd.0 v2:172.21.15.118:6800/27690 9 ==== osd_op_reply(35 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f6b54022040 con 0x7f6b44010070 2023-10-05T19:28:26.335 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.330+0000 7f6b5d5d9700 1 -- 172.21.15.167:0/581366395 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:36 5.f 5:f0000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x7f6b4c03d040 con 0x5584aaad42f0 2023-10-05T19:28:26.336 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.330+0000 7f6b5d5d9700 1 -- 172.21.15.167:0/581366395 <== osd.1 v2:172.21.15.167:6800/30035 13 ==== osd_op_reply(36 [pgnls start_epoch 30 out=115b] v23'2 uv2 ondisk = 1) v8 ==== 144+0+115 (crc 0 0 0) 0x7f6b5402f080 con 0x5584aaad42f0 2023-10-05T19:28:26.336 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.330+0000 7f6b5d5d9700 1 -- 172.21.15.167:0/581366395 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:37 5.1f 5:f8000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x7f6b4c03d040 con 0x7f6b4c0176e0 2023-10-05T19:28:26.338 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.330+0000 7f6b5c5d7700 1 -- 172.21.15.167:0/581366395 <== osd.2 v2:172.21.15.167:6816/30037 6 ==== osd_op_reply(37 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f6b44017050 con 0x7f6b4c0176e0 2023-10-05T19:28:26.338 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.330+0000 7f6b5f2a3e40 20 RGWRados::pool_iterate: got zonegroup_info.4f1f719d-9648-417b-b10c-e5d939825419 2023-10-05T19:28:26.338 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.330+0000 7f6b5f2a3e40 20 RGWRados::pool_iterate: got zone_info.9a9df965-6612-45d4-a45f-6e3c889bacc2 2023-10-05T19:28:26.338 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.330+0000 7f6b5f2a3e40 20 RGWRados::pool_iterate: got zone_names.default 2023-10-05T19:28:26.338 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.330+0000 7f6b5f2a3e40 20 RGWRados::pool_iterate: got zonegroups_names.default 2023-10-05T19:28:26.338 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.330+0000 7f6b5f2a3e40 20 rados->read ofs=0 len=0 2023-10-05T19:28:26.338 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.330+0000 7f6b5f2a3e40 1 -- 172.21.15.167:0/581366395 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:38 5.12 5:49953fa1:::default.realm:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x5584aab04410 con 0x5584aaad42f0 2023-10-05T19:28:26.339 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.330+0000 7f6b5d5d9700 1 -- 172.21.15.167:0/581366395 <== osd.1 v2:172.21.15.167:6800/30035 14 ==== osd_op_reply(38 default.realm [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 ==== 157+0+0 (crc 0 0 0) 0x7f6b5402f080 con 0x5584aaad42f0 2023-10-05T19:28:26.339 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.330+0000 7f6b5f2a3e40 20 rados_obj.operate() r=-2 bl.length=0 2023-10-05T19:28:26.339 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.330+0000 7f6b5f2a3e40 20 rados->read ofs=0 len=0 2023-10-05T19:28:26.339 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.330+0000 7f6b5f2a3e40 1 -- 172.21.15.167:0/581366395 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:39 5.f 5:f4c53578:::zonegroups_names.default:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x5584aab04d00 con 0x5584aaad42f0 2023-10-05T19:28:26.342 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.334+0000 7f6b5d5d9700 1 -- 172.21.15.167:0/581366395 <== osd.1 v2:172.21.15.167:6800/30035 15 ==== osd_op_reply(39 zonegroups_names.default [read 0~46 out=46b] v0'0 uv2 ondisk = 0) v8 ==== 168+0+46 (crc 0 0 0) 0x7f6b5402f080 con 0x5584aaad42f0 2023-10-05T19:28:26.343 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.334+0000 7f6b5f2a3e40 20 rados_obj.operate() r=0 bl.length=46 2023-10-05T19:28:26.343 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.334+0000 7f6b5f2a3e40 20 rados->read ofs=0 len=0 2023-10-05T19:28:26.343 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.334+0000 7f6b5f2a3e40 1 -- 172.21.15.167:0/581366395 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:40 5.c 5:30d299a5:::zonegroup_info.4f1f719d-9648-417b-b10c-e5d939825419:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x5584aab083c0 con 0x7f6b44010070 2023-10-05T19:28:26.343 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.334+0000 7f6b5d5d9700 1 -- 172.21.15.167:0/581366395 <== osd.0 v2:172.21.15.118:6800/27690 10 ==== osd_op_reply(40 zonegroup_info.4f1f719d-9648-417b-b10c-e5d939825419 [read 0~362 out=362b] v0'0 uv1 ondisk = 0) v8 ==== 195+0+362 (crc 0 0 0) 0x7f6b5402f080 con 0x7f6b44010070 2023-10-05T19:28:26.343 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.334+0000 7f6b5f2a3e40 20 rados_obj.operate() r=0 bl.length=362 2023-10-05T19:28:26.344 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.334+0000 7f6b5f2a3e40 20 rados->read ofs=0 len=0 2023-10-05T19:28:26.344 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.334+0000 7f6b5f2a3e40 1 -- 172.21.15.167:0/581366395 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:41 5.13 5:cbbf6c60:::converted:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x5584aab083c0 con 0x5584aab05660 2023-10-05T19:28:26.344 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.334+0000 7f6b5cdd8700 1 -- 172.21.15.167:0/581366395 <== osd.3 v2:172.21.15.167:6808/30036 10 ==== osd_op_reply(41 converted [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 ==== 153+0+0 (crc 0 0 0) 0x7f6b4c039040 con 0x5584aab05660 2023-10-05T19:28:26.344 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.334+0000 7f6b5f2a3e40 20 rados_obj.operate() r=-2 bl.length=0 2023-10-05T19:28:26.344 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.334+0000 7f6b5f2a3e40 20 rados->read ofs=0 len=0 2023-10-05T19:28:26.345 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.334+0000 7f6b5f2a3e40 1 -- 172.21.15.167:0/581366395 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:42 5.12 5:49953fa1:::default.realm:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x5584aab08770 con 0x5584aaad42f0 2023-10-05T19:28:26.345 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.334+0000 7f6b5d5d9700 1 -- 172.21.15.167:0/581366395 <== osd.1 v2:172.21.15.167:6800/30035 16 ==== osd_op_reply(42 default.realm [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 ==== 157+0+0 (crc 0 0 0) 0x7f6b5402f080 con 0x5584aaad42f0 2023-10-05T19:28:26.345 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.334+0000 7f6b5f2a3e40 20 rados_obj.operate() r=-2 bl.length=0 2023-10-05T19:28:26.345 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.334+0000 7f6b5f2a3e40 20 rados->read ofs=0 len=0 2023-10-05T19:28:26.345 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.334+0000 7f6b5f2a3e40 1 -- 172.21.15.167:0/581366395 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:43 5.f 5:f4c53578:::zonegroups_names.default:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x5584aab08b70 con 0x5584aaad42f0 2023-10-05T19:28:26.345 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.334+0000 7f6b5d5d9700 1 -- 172.21.15.167:0/581366395 <== osd.1 v2:172.21.15.167:6800/30035 17 ==== osd_op_reply(43 zonegroups_names.default [read 0~46 out=46b] v0'0 uv2 ondisk = 0) v8 ==== 168+0+46 (crc 0 0 0) 0x7f6b5402f080 con 0x5584aaad42f0 2023-10-05T19:28:26.346 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.334+0000 7f6b5f2a3e40 20 rados_obj.operate() r=0 bl.length=46 2023-10-05T19:28:26.346 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.334+0000 7f6b5f2a3e40 1 -- 172.21.15.167:0/581366395 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:44 5.0 5:00000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x5584aab09b80 con 0x5584aab05660 2023-10-05T19:28:26.348 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.338+0000 7f6b5cdd8700 1 -- 172.21.15.167:0/581366395 <== osd.3 v2:172.21.15.167:6808/30036 11 ==== osd_op_reply(44 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f6b4c039040 con 0x5584aab05660 2023-10-05T19:28:26.348 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.338+0000 7f6b5cdd8700 1 -- 172.21.15.167:0/581366395 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:45 5.10 5:08000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x5584aab09b80 con 0x7f6b4c0176e0 2023-10-05T19:28:26.348 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.338+0000 7f6b5c5d7700 1 -- 172.21.15.167:0/581366395 <== osd.2 v2:172.21.15.167:6816/30037 7 ==== osd_op_reply(45 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f6b44017050 con 0x7f6b4c0176e0 2023-10-05T19:28:26.348 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.338+0000 7f6b5c5d7700 1 -- 172.21.15.167:0/581366395 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:46 5.8 5:10000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x5584aab09b80 con 0x7f6b44010070 2023-10-05T19:28:26.348 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.342+0000 7f6b5d5d9700 1 -- 172.21.15.167:0/581366395 <== osd.0 v2:172.21.15.118:6800/27690 11 ==== osd_op_reply(46 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f6b5402f080 con 0x7f6b44010070 2023-10-05T19:28:26.348 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.342+0000 7f6b5d5d9700 1 -- 172.21.15.167:0/581366395 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:47 5.18 5:18000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x5584aab04d00 con 0x5584aaad42f0 2023-10-05T19:28:26.349 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.342+0000 7f6b5d5d9700 1 -- 172.21.15.167:0/581366395 <== osd.1 v2:172.21.15.167:6800/30035 18 ==== osd_op_reply(47 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f6b5402f080 con 0x5584aaad42f0 2023-10-05T19:28:26.349 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.342+0000 7f6b5d5d9700 1 -- 172.21.15.167:0/581366395 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:48 5.4 5:20000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x5584aab04d00 con 0x5584aaad42f0 2023-10-05T19:28:26.349 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.342+0000 7f6b5d5d9700 1 -- 172.21.15.167:0/581366395 <== osd.1 v2:172.21.15.167:6800/30035 19 ==== osd_op_reply(48 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f6b5402f080 con 0x5584aaad42f0 2023-10-05T19:28:26.349 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.342+0000 7f6b5d5d9700 1 -- 172.21.15.167:0/581366395 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:49 5.14 5:28000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x5584aab04d00 con 0x7f6b44010070 2023-10-05T19:28:26.351 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.342+0000 7f6b5d5d9700 1 -- 172.21.15.167:0/581366395 <== osd.0 v2:172.21.15.118:6800/27690 12 ==== osd_op_reply(49 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f6b5402f080 con 0x7f6b44010070 2023-10-05T19:28:26.351 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.342+0000 7f6b5d5d9700 1 -- 172.21.15.167:0/581366395 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:50 5.c 5:30000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x5584aab04d00 con 0x7f6b44010070 2023-10-05T19:28:26.351 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.342+0000 7f6b5d5d9700 1 -- 172.21.15.167:0/581366395 <== osd.0 v2:172.21.15.118:6800/27690 13 ==== osd_op_reply(50 [pgnls start_epoch 30 out=112b] v23'1 uv1 ondisk = 1) v8 ==== 144+0+112 (crc 0 0 0) 0x7f6b5402f080 con 0x7f6b44010070 2023-10-05T19:28:26.352 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.342+0000 7f6b5d5d9700 1 -- 172.21.15.167:0/581366395 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:51 5.1c 5:38000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x5584aab04d00 con 0x7f6b4c0176e0 2023-10-05T19:28:26.352 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.346+0000 7f6b5c5d7700 1 -- 172.21.15.167:0/581366395 <== osd.2 v2:172.21.15.167:6816/30037 8 ==== osd_op_reply(51 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f6b44017050 con 0x7f6b4c0176e0 2023-10-05T19:28:26.352 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.346+0000 7f6b5c5d7700 1 -- 172.21.15.167:0/581366395 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:52 5.2 5:40000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x5584aab04d00 con 0x5584aab05660 2023-10-05T19:28:26.354 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.346+0000 7f6b5cdd8700 1 -- 172.21.15.167:0/581366395 <== osd.3 v2:172.21.15.167:6808/30036 12 ==== osd_op_reply(52 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f6b4c039040 con 0x5584aab05660 2023-10-05T19:28:26.354 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.346+0000 7f6b5cdd8700 1 -- 172.21.15.167:0/581366395 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:53 5.12 5:48000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x5584aab04d00 con 0x5584aaad42f0 2023-10-05T19:28:26.354 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.346+0000 7f6b5d5d9700 1 -- 172.21.15.167:0/581366395 <== osd.1 v2:172.21.15.167:6800/30035 20 ==== osd_op_reply(53 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f6b5402f080 con 0x5584aaad42f0 2023-10-05T19:28:26.354 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.346+0000 7f6b5d5d9700 1 -- 172.21.15.167:0/581366395 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:54 5.a 5:50000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x5584aab04d00 con 0x7f6b44010070 2023-10-05T19:28:26.354 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.346+0000 7f6b5d5d9700 1 -- 172.21.15.167:0/581366395 <== osd.0 v2:172.21.15.118:6800/27690 14 ==== osd_op_reply(54 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f6b5402f080 con 0x7f6b44010070 2023-10-05T19:28:26.354 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.346+0000 7f6b5d5d9700 1 -- 172.21.15.167:0/581366395 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:55 5.1a 5:58000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x5584aab04d00 con 0x5584aaad42f0 2023-10-05T19:28:26.356 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.350+0000 7f6b5d5d9700 1 -- 172.21.15.167:0/581366395 <== osd.1 v2:172.21.15.167:6800/30035 21 ==== osd_op_reply(55 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f6b5402f080 con 0x5584aaad42f0 2023-10-05T19:28:26.356 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.350+0000 7f6b5d5d9700 1 -- 172.21.15.167:0/581366395 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:56 5.6 5:60000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x5584aab04d00 con 0x7f6b4c0176e0 2023-10-05T19:28:26.356 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.350+0000 7f6b5c5d7700 1 -- 172.21.15.167:0/581366395 <== osd.2 v2:172.21.15.167:6816/30037 9 ==== osd_op_reply(56 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f6b44017050 con 0x7f6b4c0176e0 2023-10-05T19:28:26.357 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.350+0000 7f6b5c5d7700 1 -- 172.21.15.167:0/581366395 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:57 5.16 5:68000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x5584aab04d00 con 0x5584aab05660 2023-10-05T19:28:26.357 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.350+0000 7f6b5cdd8700 1 -- 172.21.15.167:0/581366395 <== osd.3 v2:172.21.15.167:6808/30036 13 ==== osd_op_reply(57 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f6b4c039040 con 0x5584aab05660 2023-10-05T19:28:26.357 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.350+0000 7f6b5cdd8700 1 -- 172.21.15.167:0/581366395 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:58 5.e 5:70000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x5584aab04d00 con 0x5584aab05660 2023-10-05T19:28:26.358 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.350+0000 7f6b5cdd8700 1 -- 172.21.15.167:0/581366395 <== osd.3 v2:172.21.15.167:6808/30036 14 ==== osd_op_reply(58 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f6b4c039040 con 0x5584aab05660 2023-10-05T19:28:26.359 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.350+0000 7f6b5cdd8700 1 -- 172.21.15.167:0/581366395 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:59 5.1e 5:78000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x5584aab04d00 con 0x7f6b44010070 2023-10-05T19:28:26.359 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.350+0000 7f6b5d5d9700 1 -- 172.21.15.167:0/581366395 <== osd.0 v2:172.21.15.118:6800/27690 15 ==== osd_op_reply(59 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f6b5402f080 con 0x7f6b44010070 2023-10-05T19:28:26.359 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.350+0000 7f6b5d5d9700 1 -- 172.21.15.167:0/581366395 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:60 5.1 5:80000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x5584aab04d00 con 0x5584aab05660 2023-10-05T19:28:26.359 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.354+0000 7f6b5cdd8700 1 -- 172.21.15.167:0/581366395 <== osd.3 v2:172.21.15.167:6808/30036 15 ==== osd_op_reply(60 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f6b4c039040 con 0x5584aab05660 2023-10-05T19:28:26.359 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.354+0000 7f6b5cdd8700 1 -- 172.21.15.167:0/581366395 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:61 5.11 5:88000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x5584aab04d00 con 0x5584aab05660 2023-10-05T19:28:26.360 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.354+0000 7f6b5cdd8700 1 -- 172.21.15.167:0/581366395 <== osd.3 v2:172.21.15.167:6808/30036 16 ==== osd_op_reply(61 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f6b4c039040 con 0x5584aab05660 2023-10-05T19:28:26.361 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.354+0000 7f6b5cdd8700 1 -- 172.21.15.167:0/581366395 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:62 5.9 5:90000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x5584aab04d00 con 0x5584aaad42f0 2023-10-05T19:28:26.361 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.354+0000 7f6b5d5d9700 1 -- 172.21.15.167:0/581366395 <== osd.1 v2:172.21.15.167:6800/30035 22 ==== osd_op_reply(62 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f6b5402f080 con 0x5584aaad42f0 2023-10-05T19:28:26.361 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.354+0000 7f6b5d5d9700 1 -- 172.21.15.167:0/581366395 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:63 5.19 5:98000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x5584aab04d00 con 0x5584aaad42f0 2023-10-05T19:28:26.362 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.354+0000 7f6b5d5d9700 1 -- 172.21.15.167:0/581366395 <== osd.1 v2:172.21.15.167:6800/30035 23 ==== osd_op_reply(63 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f6b5402f080 con 0x5584aaad42f0 2023-10-05T19:28:26.362 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.354+0000 7f6b5d5d9700 1 -- 172.21.15.167:0/581366395 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:64 5.5 5:a0000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x5584aab04d00 con 0x7f6b44010070 2023-10-05T19:28:26.362 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.354+0000 7f6b5d5d9700 1 -- 172.21.15.167:0/581366395 <== osd.0 v2:172.21.15.118:6800/27690 16 ==== osd_op_reply(64 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f6b5402f080 con 0x7f6b44010070 2023-10-05T19:28:26.362 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.354+0000 7f6b5d5d9700 1 -- 172.21.15.167:0/581366395 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:65 5.15 5:a8000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x5584aab04d00 con 0x5584aab05660 2023-10-05T19:28:26.364 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.358+0000 7f6b5cdd8700 1 -- 172.21.15.167:0/581366395 <== osd.3 v2:172.21.15.167:6808/30036 17 ==== osd_op_reply(65 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f6b4c039040 con 0x5584aab05660 2023-10-05T19:28:26.364 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.358+0000 7f6b5cdd8700 1 -- 172.21.15.167:0/581366395 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:66 5.d 5:b0000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x5584aab04d00 con 0x7f6b4c0176e0 2023-10-05T19:28:26.364 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.358+0000 7f6b5c5d7700 1 -- 172.21.15.167:0/581366395 <== osd.2 v2:172.21.15.167:6816/30037 10 ==== osd_op_reply(66 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f6b44017050 con 0x7f6b4c0176e0 2023-10-05T19:28:26.364 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.358+0000 7f6b5c5d7700 1 -- 172.21.15.167:0/581366395 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:67 5.1d 5:b8000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x5584aab04d00 con 0x5584aaad42f0 2023-10-05T19:28:26.365 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.358+0000 7f6b5d5d9700 1 -- 172.21.15.167:0/581366395 <== osd.1 v2:172.21.15.167:6800/30035 24 ==== osd_op_reply(67 [pgnls start_epoch 30 out=107b] v30'2 uv2 ondisk = 1) v8 ==== 144+0+107 (crc 0 0 0) 0x7f6b5402f080 con 0x5584aaad42f0 2023-10-05T19:28:26.365 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.358+0000 7f6b5d5d9700 1 -- 172.21.15.167:0/581366395 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:68 5.3 5:c0000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x5584aab04d00 con 0x7f6b44010070 2023-10-05T19:28:26.365 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.358+0000 7f6b5d5d9700 1 -- 172.21.15.167:0/581366395 <== osd.0 v2:172.21.15.118:6800/27690 17 ==== osd_op_reply(68 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f6b5402f080 con 0x7f6b44010070 2023-10-05T19:28:26.366 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.358+0000 7f6b5d5d9700 1 -- 172.21.15.167:0/581366395 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:69 5.13 5:c8000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x5584aab04d00 con 0x5584aab05660 2023-10-05T19:28:26.366 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.362+0000 7f6b5cdd8700 1 -- 172.21.15.167:0/581366395 <== osd.3 v2:172.21.15.167:6808/30036 18 ==== osd_op_reply(69 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f6b4c052040 con 0x5584aab05660 2023-10-05T19:28:26.367 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.362+0000 7f6b5cdd8700 1 -- 172.21.15.167:0/581366395 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:70 5.b 5:d0000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x5584aab04d00 con 0x7f6b44010070 2023-10-05T19:28:26.367 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.362+0000 7f6b5d5d9700 1 -- 172.21.15.167:0/581366395 <== osd.0 v2:172.21.15.118:6800/27690 18 ==== osd_op_reply(70 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f6b5402f080 con 0x7f6b44010070 2023-10-05T19:28:26.367 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.362+0000 7f6b5d5d9700 1 -- 172.21.15.167:0/581366395 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:71 5.1b 5:d8000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x5584aab04d00 con 0x7f6b4c0176e0 2023-10-05T19:28:26.368 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.362+0000 7f6b5c5d7700 1 -- 172.21.15.167:0/581366395 <== osd.2 v2:172.21.15.167:6816/30037 11 ==== osd_op_reply(71 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f6b44017050 con 0x7f6b4c0176e0 2023-10-05T19:28:26.368 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.362+0000 7f6b5c5d7700 1 -- 172.21.15.167:0/581366395 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:72 5.7 5:e0000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x5584aab04d00 con 0x5584aab05660 2023-10-05T19:28:26.369 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.362+0000 7f6b5cdd8700 1 -- 172.21.15.167:0/581366395 <== osd.3 v2:172.21.15.167:6808/30036 19 ==== osd_op_reply(72 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f6b4c052040 con 0x5584aab05660 2023-10-05T19:28:26.369 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.362+0000 7f6b5cdd8700 1 -- 172.21.15.167:0/581366395 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:73 5.17 5:e8000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x5584aab04d00 con 0x7f6b44010070 2023-10-05T19:28:26.370 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.362+0000 7f6b5d5d9700 1 -- 172.21.15.167:0/581366395 <== osd.0 v2:172.21.15.118:6800/27690 19 ==== osd_op_reply(73 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f6b5402f080 con 0x7f6b44010070 2023-10-05T19:28:26.370 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.362+0000 7f6b5d5d9700 1 -- 172.21.15.167:0/581366395 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:74 5.f 5:f0000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x5584aab04d00 con 0x5584aaad42f0 2023-10-05T19:28:26.371 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.366+0000 7f6b5d5d9700 1 -- 172.21.15.167:0/581366395 <== osd.1 v2:172.21.15.167:6800/30035 25 ==== osd_op_reply(74 [pgnls start_epoch 30 out=115b] v23'2 uv2 ondisk = 1) v8 ==== 144+0+115 (crc 0 0 0) 0x7f6b5402f080 con 0x5584aaad42f0 2023-10-05T19:28:26.371 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.366+0000 7f6b5d5d9700 1 -- 172.21.15.167:0/581366395 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:75 5.1f 5:f8000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x5584aab04d00 con 0x7f6b4c0176e0 2023-10-05T19:28:26.373 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.366+0000 7f6b5c5d7700 1 -- 172.21.15.167:0/581366395 <== osd.2 v2:172.21.15.167:6816/30037 12 ==== osd_op_reply(75 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f6b44017050 con 0x7f6b4c0176e0 2023-10-05T19:28:26.374 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.366+0000 7f6b5f2a3e40 20 RGWRados::pool_iterate: got zonegroup_info.4f1f719d-9648-417b-b10c-e5d939825419 2023-10-05T19:28:26.374 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.366+0000 7f6b5f2a3e40 20 RGWRados::pool_iterate: got zone_info.9a9df965-6612-45d4-a45f-6e3c889bacc2 2023-10-05T19:28:26.374 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.366+0000 7f6b5f2a3e40 20 RGWRados::pool_iterate: got zone_names.default 2023-10-05T19:28:26.374 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.366+0000 7f6b5f2a3e40 20 RGWRados::pool_iterate: got zonegroups_names.default 2023-10-05T19:28:26.374 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.366+0000 7f6b5f2a3e40 20 rados->read ofs=0 len=0 2023-10-05T19:28:26.374 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.366+0000 7f6b5f2a3e40 1 -- 172.21.15.167:0/581366395 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:76 5.f 5:f43b6ece:::zone_names.default:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x5584aab08b70 con 0x5584aaad42f0 2023-10-05T19:28:26.374 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.366+0000 7f6b5d5d9700 1 -- 172.21.15.167:0/581366395 <== osd.1 v2:172.21.15.167:6800/30035 26 ==== osd_op_reply(76 zone_names.default [read 0~46 out=46b] v0'0 uv1 ondisk = 0) v8 ==== 162+0+46 (crc 0 0 0) 0x7f6b5402f080 con 0x5584aaad42f0 2023-10-05T19:28:26.374 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.366+0000 7f6b5f2a3e40 20 rados_obj.operate() r=0 bl.length=46 2023-10-05T19:28:26.374 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.366+0000 7f6b5f2a3e40 20 rados->read ofs=0 len=0 2023-10-05T19:28:26.375 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.366+0000 7f6b5f2a3e40 1 -- 172.21.15.167:0/581366395 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:77 5.1d 5:b8374eb4:::zone_info.9a9df965-6612-45d4-a45f-6e3c889bacc2:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x5584aab08f20 con 0x5584aaad42f0 2023-10-05T19:28:26.375 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.366+0000 7f6b5d5d9700 1 -- 172.21.15.167:0/581366395 <== osd.1 v2:172.21.15.167:6800/30035 27 ==== osd_op_reply(77 zone_info.9a9df965-6612-45d4-a45f-6e3c889bacc2 [read 0~874 out=874b] v0'0 uv2 ondisk = 0) v8 ==== 190+0+874 (crc 0 0 0) 0x7f6b5402f080 con 0x5584aaad42f0 2023-10-05T19:28:26.375 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.366+0000 7f6b5f2a3e40 20 rados_obj.operate() r=0 bl.length=874 2023-10-05T19:28:26.375 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.366+0000 7f6b5f2a3e40 20 rados->read ofs=0 len=0 2023-10-05T19:28:26.375 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.366+0000 7f6b5f2a3e40 1 -- 172.21.15.167:0/581366395 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:78 5.f 5:f4c53578:::zonegroups_names.default:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x5584aab08f20 con 0x5584aaad42f0 2023-10-05T19:28:26.376 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.370+0000 7f6b5d5d9700 1 -- 172.21.15.167:0/581366395 <== osd.1 v2:172.21.15.167:6800/30035 28 ==== osd_op_reply(78 zonegroups_names.default [read 0~46 out=46b] v0'0 uv2 ondisk = 0) v8 ==== 168+0+46 (crc 0 0 0) 0x7f6b5402f080 con 0x5584aaad42f0 2023-10-05T19:28:26.376 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.370+0000 7f6b5f2a3e40 20 rados_obj.operate() r=0 bl.length=46 2023-10-05T19:28:26.376 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.370+0000 7f6b5f2a3e40 20 rados->read ofs=0 len=0 2023-10-05T19:28:26.376 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.370+0000 7f6b5f2a3e40 1 -- 172.21.15.167:0/581366395 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:79 5.c 5:30d299a5:::zonegroup_info.4f1f719d-9648-417b-b10c-e5d939825419:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x5584aab08f20 con 0x7f6b44010070 2023-10-05T19:28:26.377 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.370+0000 7f6b5d5d9700 1 -- 172.21.15.167:0/581366395 <== osd.0 v2:172.21.15.118:6800/27690 20 ==== osd_op_reply(79 zonegroup_info.4f1f719d-9648-417b-b10c-e5d939825419 [read 0~362 out=362b] v0'0 uv1 ondisk = 0) v8 ==== 195+0+362 (crc 0 0 0) 0x7f6b5402f080 con 0x7f6b44010070 2023-10-05T19:28:26.377 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.370+0000 7f6b5f2a3e40 20 rados_obj.operate() r=0 bl.length=362 2023-10-05T19:28:26.377 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.370+0000 7f6b5f2a3e40 20 rados->read ofs=0 len=0 2023-10-05T19:28:26.377 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.370+0000 7f6b5f2a3e40 1 -- 172.21.15.167:0/581366395 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:80 5.3 5:c0ad8cbd:::region_map:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x5584aab09370 con 0x7f6b44010070 2023-10-05T19:28:26.377 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.370+0000 7f6b5d5d9700 1 -- 172.21.15.167:0/581366395 <== osd.0 v2:172.21.15.118:6800/27690 21 ==== osd_op_reply(80 region_map [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 ==== 154+0+0 (crc 0 0 0) 0x7f6b54050040 con 0x7f6b44010070 2023-10-05T19:28:26.377 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.370+0000 7f6b5f2a3e40 20 rados_obj.operate() r=-2 bl.length=0 2023-10-05T19:28:26.377 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.370+0000 7f6b5f2a3e40 20 zone default found 2023-10-05T19:28:26.377 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.370+0000 7f6b5f2a3e40 4 Realm: () 2023-10-05T19:28:26.377 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.370+0000 7f6b5f2a3e40 4 ZoneGroup: default (4f1f719d-9648-417b-b10c-e5d939825419) 2023-10-05T19:28:26.377 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.370+0000 7f6b5f2a3e40 4 Zone: default (9a9df965-6612-45d4-a45f-6e3c889bacc2) 2023-10-05T19:28:26.378 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.370+0000 7f6b5f2a3e40 10 cannot find current period zonegroup using local zonegroup configuration 2023-10-05T19:28:26.378 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.370+0000 7f6b5f2a3e40 20 zonegroup default 2023-10-05T19:28:26.378 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.370+0000 7f6b5f2a3e40 20 rados->read ofs=0 len=0 2023-10-05T19:28:26.378 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.370+0000 7f6b5f2a3e40 1 -- 172.21.15.167:0/581366395 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:81 5.3 5:c52100b6:::period_config.default:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x5584aab09370 con 0x7f6b44010070 2023-10-05T19:28:26.379 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.370+0000 7f6b5d5d9700 1 -- 172.21.15.167:0/581366395 <== osd.0 v2:172.21.15.118:6800/27690 22 ==== osd_op_reply(81 period_config.default [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 ==== 165+0+0 (crc 0 0 0) 0x7f6b54050040 con 0x7f6b44010070 2023-10-05T19:28:26.379 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.370+0000 7f6b5f2a3e40 20 rados_obj.operate() r=-2 bl.length=0 2023-10-05T19:28:26.379 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.370+0000 7f6b5f2a3e40 20 rados->read ofs=0 len=0 2023-10-05T19:28:26.380 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.370+0000 7f6b5f2a3e40 1 -- 172.21.15.167:0/581366395 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:82 6.2 6:4347d321:::bucket.sync-source-hints.:head [call version.read in=11b,read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x5584aab0aac0 con 0x5584aab05660 2023-10-05T19:28:26.380 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.370+0000 7f6b5cdd8700 1 -- 172.21.15.167:0/581366395 <== osd.3 v2:172.21.15.167:6808/30036 20 ==== osd_op_reply(82 bucket.sync-source-hints. [call,read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 ==== 211+0+0 (crc 0 0 0) 0x7f6b4c052040 con 0x5584aab05660 2023-10-05T19:28:26.380 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.374+0000 7f6b5f2a3e40 20 rados_obj.operate() r=-2 bl.length=0 2023-10-05T19:28:26.380 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.374+0000 7f6b5f2a3e40 20 rados->read ofs=0 len=0 2023-10-05T19:28:26.380 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.374+0000 7f6b5f2a3e40 1 -- 172.21.15.167:0/581366395 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:83 6.b 6:d467b91b:::bucket.sync-target-hints.:head [call version.read in=11b,read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x5584aab0ae70 con 0x5584aab05660 2023-10-05T19:28:26.380 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.374+0000 7f6b5cdd8700 1 -- 172.21.15.167:0/581366395 <== osd.3 v2:172.21.15.167:6808/30036 21 ==== osd_op_reply(83 bucket.sync-target-hints. [call,read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 ==== 211+0+0 (crc 0 0 0) 0x7f6b4c052040 con 0x5584aab05660 2023-10-05T19:28:26.380 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.374+0000 7f6b5f2a3e40 20 rados_obj.operate() r=-2 bl.length=0 2023-10-05T19:28:26.380 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.374+0000 7f6b5f2a3e40 20 started sync module instance, tier type = 2023-10-05T19:28:26.380 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.374+0000 7f6b5f2a3e40 20 started zone id=9a9df965-6612-45d4-a45f-6e3c889bacc2 (name=default) with tier type = 2023-10-05T19:28:26.380 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.374+0000 7f6b5f2a3e40 1 -- 172.21.15.167:0/581366395 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:84 7.1f 7:f95f44c2:::notify.0:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x5584aab0a180 con 0x7f6b4c0176e0 2023-10-05T19:28:26.382 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.374+0000 7f6b5c5d7700 1 -- 172.21.15.167:0/581366395 <== osd.2 v2:172.21.15.167:6816/30037 13 ==== osd_op_reply(84 notify.0 [create] v30'33 uv33 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f6b44017050 con 0x7f6b4c0176e0 2023-10-05T19:28:26.382 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.374+0000 7f6b5f2a3e40 1 -- 172.21.15.167:0/581366395 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:85 7.1f 7:f95f44c2:::notify.0:head [watch watch cookie 94028287747456] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x5584aab04930 con 0x7f6b4c0176e0 2023-10-05T19:28:26.382 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.378+0000 7f6b5f2a3e40 1 -- 172.21.15.167:0/581366395 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:86 7.0 7:05bf5b68:::notify.1:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x5584aab0bc30 con 0x5584aaad42f0 2023-10-05T19:28:26.385 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.378+0000 7f6b5c5d7700 1 -- 172.21.15.167:0/581366395 <== osd.2 v2:172.21.15.167:6816/30037 14 ==== osd_op_reply(85 notify.0 [watch watch cookie 94028287747456] v30'34 uv33 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f6b44017050 con 0x7f6b4c0176e0 2023-10-05T19:28:26.385 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.378+0000 7f6b5d5d9700 1 -- 172.21.15.167:0/581366395 <== osd.1 v2:172.21.15.167:6800/30035 29 ==== osd_op_reply(86 notify.1 [create] v30'33 uv33 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f6b54050040 con 0x5584aaad42f0 2023-10-05T19:28:26.386 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.378+0000 7f6b5f2a3e40 1 -- 172.21.15.167:0/581366395 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:87 7.0 7:05bf5b68:::notify.1:head [watch watch cookie 94028287752000] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x5584aab0bb90 con 0x5584aaad42f0 2023-10-05T19:28:26.386 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.378+0000 7f6b5f2a3e40 1 -- 172.21.15.167:0/581366395 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:88 7.15 7:a93a5511:::notify.2:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x5584aab0ca10 con 0x5584aab05660 2023-10-05T19:28:26.388 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.382+0000 7f6b5cdd8700 1 -- 172.21.15.167:0/581366395 <== osd.3 v2:172.21.15.167:6808/30036 22 ==== osd_op_reply(88 notify.2 [create] v30'33 uv33 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f6b4c052040 con 0x5584aab05660 2023-10-05T19:28:26.389 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.382+0000 7f6b5f2a3e40 1 -- 172.21.15.167:0/581366395 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:89 7.15 7:a93a5511:::notify.2:head [watch watch cookie 94028287755552] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x5584aab0bb90 con 0x5584aab05660 2023-10-05T19:28:26.389 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.382+0000 7f6b5f2a3e40 1 -- 172.21.15.167:0/581366395 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:90 7.e 7:7759931f:::notify.3:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x5584aab0da40 con 0x7f6b4c0176e0 2023-10-05T19:28:26.389 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.382+0000 7f6b5d5d9700 1 -- 172.21.15.167:0/581366395 <== osd.1 v2:172.21.15.167:6800/30035 30 ==== osd_op_reply(87 notify.1 [watch watch cookie 94028287752000] v30'34 uv33 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f6b54050040 con 0x5584aaad42f0 2023-10-05T19:28:26.391 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.386+0000 7f6b5c5d7700 1 -- 172.21.15.167:0/581366395 <== osd.2 v2:172.21.15.167:6816/30037 15 ==== osd_op_reply(90 notify.3 [create] v30'33 uv33 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f6b44017050 con 0x7f6b4c0176e0 2023-10-05T19:28:26.392 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.386+0000 7f6b5f2a3e40 1 -- 172.21.15.167:0/581366395 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:91 7.e 7:7759931f:::notify.3:head [watch watch cookie 94028287754288] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x5584aab0d630 con 0x7f6b4c0176e0 2023-10-05T19:28:26.392 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.386+0000 7f6b5f2a3e40 1 -- 172.21.15.167:0/581366395 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:92 7.d 7:b4812045:::notify.4:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x5584aab0eae0 con 0x5584aaad42f0 2023-10-05T19:28:26.392 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.386+0000 7f6b5cdd8700 1 -- 172.21.15.167:0/581366395 <== osd.3 v2:172.21.15.167:6808/30036 23 ==== osd_op_reply(89 notify.2 [watch watch cookie 94028287755552] v30'34 uv33 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f6b4c052040 con 0x5584aab05660 2023-10-05T19:28:26.393 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.386+0000 7f6b5c5d7700 1 -- 172.21.15.167:0/581366395 <== osd.2 v2:172.21.15.167:6816/30037 16 ==== osd_op_reply(91 notify.3 [watch watch cookie 94028287754288] v30'34 uv33 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f6b44017050 con 0x7f6b4c0176e0 2023-10-05T19:28:26.394 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.390+0000 7f6b5d5d9700 1 -- 172.21.15.167:0/581366395 <== osd.1 v2:172.21.15.167:6800/30035 31 ==== osd_op_reply(92 notify.4 [create] v30'33 uv33 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f6b54050040 con 0x5584aaad42f0 2023-10-05T19:28:26.395 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.390+0000 7f6b5f2a3e40 1 -- 172.21.15.167:0/581366395 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:93 7.d 7:b4812045:::notify.4:head [watch watch cookie 94028287761984] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x5584aab0e5b0 con 0x5584aaad42f0 2023-10-05T19:28:26.395 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.390+0000 7f6b5f2a3e40 1 -- 172.21.15.167:0/581366395 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:94 7.3 7:c609908c:::notify.5:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x5584aab0f3d0 con 0x7f6b44010070 2023-10-05T19:28:26.397 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.390+0000 7f6b5d5d9700 1 -- 172.21.15.167:0/581366395 <== osd.0 v2:172.21.15.118:6800/27690 23 ==== osd_op_reply(94 notify.5 [create] v30'33 uv33 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f6b54050040 con 0x7f6b44010070 2023-10-05T19:28:26.397 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.390+0000 7f6b5f2a3e40 1 -- 172.21.15.167:0/581366395 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:95 7.3 7:c609908c:::notify.5:head [watch watch cookie 94028287766160] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x5584aab0f7d0 con 0x7f6b44010070 2023-10-05T19:28:26.397 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.390+0000 7f6b5f2a3e40 1 -- 172.21.15.167:0/581366395 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:96 7.14 7:2b04a3e9:::notify.6:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x5584aab10220 con 0x7f6b44010070 2023-10-05T19:28:26.397 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.390+0000 7f6b5d5d9700 1 -- 172.21.15.167:0/581366395 <== osd.1 v2:172.21.15.167:6800/30035 32 ==== osd_op_reply(93 notify.4 [watch watch cookie 94028287761984] v30'34 uv33 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f6b54050040 con 0x5584aaad42f0 2023-10-05T19:28:26.400 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.394+0000 7f6b5d5d9700 1 -- 172.21.15.167:0/581366395 <== osd.0 v2:172.21.15.118:6800/27690 24 ==== osd_op_reply(96 notify.6 [create] v30'33 uv33 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f6b54050040 con 0x7f6b44010070 2023-10-05T19:28:26.400 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.394+0000 7f6b5d5d9700 1 -- 172.21.15.167:0/581366395 <== osd.0 v2:172.21.15.118:6800/27690 25 ==== osd_op_reply(95 notify.5 [watch watch cookie 94028287766160] v30'34 uv33 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f6b54050040 con 0x7f6b44010070 2023-10-05T19:28:26.400 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.394+0000 7f6b5f2a3e40 1 -- 172.21.15.167:0/581366395 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:97 7.14 7:2b04a3e9:::notify.6:head [watch watch cookie 94028287757760] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x5584aab0fcf0 con 0x7f6b44010070 2023-10-05T19:28:26.400 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.394+0000 7f6b5f2a3e40 1 -- 172.21.15.167:0/581366395 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:98 7.9 7:93e5b521:::notify.7:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x5584aab10b10 con 0x7f6b44010070 2023-10-05T19:28:26.402 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.394+0000 7f6b5d5d9700 1 -- 172.21.15.167:0/581366395 <== osd.0 v2:172.21.15.118:6800/27690 26 ==== osd_op_reply(97 notify.6 [watch watch cookie 94028287757760] v30'34 uv33 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f6b54050040 con 0x7f6b44010070 2023-10-05T19:28:26.402 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.398+0000 7f6b5d5d9700 1 -- 172.21.15.167:0/581366395 <== osd.0 v2:172.21.15.118:6800/27690 27 ==== osd_op_reply(98 notify.7 [create] v30'33 uv33 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f6b54050040 con 0x7f6b44010070 2023-10-05T19:28:26.402 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.398+0000 7f6b5f2a3e40 1 -- 172.21.15.167:0/581366395 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:99 7.9 7:93e5b521:::notify.7:head [watch watch cookie 94028287767264] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x5584aab0fcf0 con 0x7f6b44010070 2023-10-05T19:28:26.403 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.398+0000 7f6b5f2a3e40 20 add_watcher() i=0 2023-10-05T19:28:26.403 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.398+0000 7f6b5f2a3e40 20 add_watcher() i=1 2023-10-05T19:28:26.403 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.398+0000 7f6b5f2a3e40 20 add_watcher() i=2 2023-10-05T19:28:26.403 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.398+0000 7f6b5f2a3e40 20 add_watcher() i=3 2023-10-05T19:28:26.403 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.398+0000 7f6b5f2a3e40 20 add_watcher() i=4 2023-10-05T19:28:26.403 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.398+0000 7f6b5f2a3e40 20 add_watcher() i=5 2023-10-05T19:28:26.403 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.398+0000 7f6b5f2a3e40 20 add_watcher() i=6 2023-10-05T19:28:26.405 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.398+0000 7f6b5d5d9700 1 -- 172.21.15.167:0/581366395 <== osd.0 v2:172.21.15.118:6800/27690 28 ==== osd_op_reply(99 notify.7 [watch watch cookie 94028287767264] v30'34 uv33 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f6b54050040 con 0x7f6b44010070 2023-10-05T19:28:26.405 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.398+0000 7f6b5f2a3e40 20 add_watcher() i=7 2023-10-05T19:28:26.405 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.398+0000 7f6b5f2a3e40 2 all 8 watchers are set, enabling cache 2023-10-05T19:28:26.405 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.398+0000 7f6b5f2a3e40 1 -- 172.21.15.167:0/581366395 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:100 6.4 6:22d26bf9:::data_loggenerations_metadata:head [call version.check_conds in=74b,call version.read in=11b,read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x5584aab139f0 con 0x5584aaad42f0 2023-10-05T19:28:26.406 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.398+0000 7f6b5d5d9700 1 -- 172.21.15.167:0/581366395 <== osd.1 v2:172.21.15.167:6800/30035 33 ==== osd_op_reply(100 data_loggenerations_metadata [call,call out=48b,read 0~28 out=28b] v0'0 uv1 ondisk = 0) v8 ==== 256+0+76 (crc 0 0 0) 0x7f6b54050040 con 0x5584aaad42f0 2023-10-05T19:28:26.406 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.402+0000 7f6b5f2a3e40 1 -- 172.21.15.167:0/581366395 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:101 6.4 6:22d26bf9:::data_loggenerations_metadata:head [watch watch cookie 94028287771856] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x5584aab10e80 con 0x5584aaad42f0 2023-10-05T19:28:26.408 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.402+0000 7f6b5d5d9700 1 -- 172.21.15.167:0/581366395 <== osd.1 v2:172.21.15.167:6800/30035 34 ==== osd_op_reply(101 data_loggenerations_metadata [watch watch cookie 94028287771856] v30'56 uv1 ondisk = 0) v8 ==== 172+0+0 (crc 0 0 0) 0x7f6b54050040 con 0x5584aaad42f0 2023-10-05T19:28:26.408 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.402+0000 7f6b2e7e4700 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2023-10-05T19:28:26.408 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.402+0000 7f6b5f2a3e40 20 check_secure_mon_conn(): auth registy supported: methods=[2] modes=[2,1] 2023-10-05T19:28:26.408 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.402+0000 7f6b5f2a3e40 20 check_secure_mon_conn(): mode 1 is insecure 2023-10-05T19:28:26.410 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.402+0000 7f6b5f2a3e40 1 Processor -- start 2023-10-05T19:28:26.410 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.402+0000 7f6b5f2a3e40 1 -- start start 2023-10-05T19:28:26.410 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.402+0000 7f6b5f2a3e40 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5584aab1e0e0 0x5584aab1e4b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-05T19:28:26.410 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.402+0000 7f6b5f2a3e40 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x5584aab1e9f0 con 0x5584aab1e0e0 2023-10-05T19:28:26.410 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.402+0000 7f6b5c5d7700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5584aab1e0e0 0x5584aab1e4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-05T19:28:26.411 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.402+0000 7f6b5c5d7700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5584aab1e0e0 0x5584aab1e4b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.118:3300/0 says I am v2:172.21.15.167:57304/0 (socket says 172.21.15.167:57304) 2023-10-05T19:28:26.411 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.402+0000 7f6b5c5d7700 1 -- 172.21.15.167:0/970092738 learned_addr learned my addr 172.21.15.167:0/970092738 (peer_addr_for_me v2:172.21.15.167:0/0) 2023-10-05T19:28:26.411 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.402+0000 7f6b5c5d7700 1 -- 172.21.15.167:0/970092738 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5584aab1eb30 con 0x5584aab1e0e0 2023-10-05T19:28:26.414 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.406+0000 7f6b5c5d7700 1 --2- 172.21.15.167:0/970092738 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5584aab1e0e0 0x5584aab1e4b0 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7f6b44003000 tx=0x7f6b44010000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=74b494a024429634 server_cookie=0 in_seq=0 out_seq=0 2023-10-05T19:28:26.414 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.406+0000 7f6b2dfe3700 1 -- 172.21.15.167:0/970092738 <== mon.0 v2:172.21.15.118:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f6b4401c9f0 con 0x5584aab1e0e0 2023-10-05T19:28:26.414 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.406+0000 7f6b2dfe3700 1 -- 172.21.15.167:0/970092738 <== mon.0 v2:172.21.15.118:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6b4401cb70 con 0x5584aab1e0e0 2023-10-05T19:28:26.414 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.406+0000 7f6b2dfe3700 1 -- 172.21.15.167:0/970092738 <== mon.0 v2:172.21.15.118:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f6b4401cd60 con 0x5584aab1e0e0 2023-10-05T19:28:26.414 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.406+0000 7f6b5f2a3e40 1 -- 172.21.15.167:0/970092738 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5584aab1e0e0 msgr2=0x5584aab1e4b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-05T19:28:26.414 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.406+0000 7f6b5f2a3e40 1 --2- 172.21.15.167:0/970092738 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5584aab1e0e0 0x5584aab1e4b0 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7f6b44003000 tx=0x7f6b44010000 comp rx=0 tx=0).stop 2023-10-05T19:28:26.414 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.406+0000 7f6b5f2a3e40 1 -- 172.21.15.167:0/970092738 shutdown_connections 2023-10-05T19:28:26.415 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.406+0000 7f6b5f2a3e40 1 --2- 172.21.15.167:0/970092738 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5584aab1e0e0 0x5584aab1e4b0 unknown :-1 s=CLOSED pgs=204 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-05T19:28:26.415 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.406+0000 7f6b5f2a3e40 1 -- 172.21.15.167:0/970092738 >> 172.21.15.167:0/970092738 conn(0x5584aab194a0 msgr2=0x5584aab175f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-05T19:28:26.415 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.406+0000 7f6b5f2a3e40 1 -- 172.21.15.167:0/970092738 shutdown_connections 2023-10-05T19:28:26.415 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.406+0000 7f6b5f2a3e40 1 -- 172.21.15.167:0/970092738 wait complete. 2023-10-05T19:28:26.415 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.406+0000 7f6b5f2a3e40 1 Processor -- start 2023-10-05T19:28:26.415 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.406+0000 7f6b5f2a3e40 1 -- start start 2023-10-05T19:28:26.415 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.406+0000 7f6b5f2a3e40 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5584aab1e0e0 0x5584aab23c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-05T19:28:26.415 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.406+0000 7f6b5f2a3e40 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x5584aab24c10 con 0x5584aab1e0e0 2023-10-05T19:28:26.415 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.406+0000 7f6b5c5d7700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5584aab1e0e0 0x5584aab23c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-05T19:28:26.416 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.406+0000 7f6b5c5d7700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5584aab1e0e0 0x5584aab23c10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.118:3300/0 says I am v2:172.21.15.167:57306/0 (socket says 172.21.15.167:57306) 2023-10-05T19:28:26.416 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.406+0000 7f6b5c5d7700 1 -- 172.21.15.167:0/3411784563 learned_addr learned my addr 172.21.15.167:0/3411784563 (peer_addr_for_me v2:172.21.15.167:0/0) 2023-10-05T19:28:26.416 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.410+0000 7f6b5c5d7700 1 -- 172.21.15.167:0/3411784563 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5584aaad7c20 con 0x5584aab1e0e0 2023-10-05T19:28:26.417 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.410+0000 7f6b5c5d7700 1 --2- 172.21.15.167:0/3411784563 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5584aab1e0e0 0x5584aab23c10 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7f6b4400efd0 tx=0x7f6b4402f000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-05T19:28:26.417 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.410+0000 7f6a9ffff700 1 -- 172.21.15.167:0/3411784563 <== mon.0 v2:172.21.15.118:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f6b4402ab20 con 0x5584aab1e0e0 2023-10-05T19:28:26.417 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.410+0000 7f6a9ffff700 1 -- 172.21.15.167:0/3411784563 <== mon.0 v2:172.21.15.118:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6b4402ac80 con 0x5584aab1e0e0 2023-10-05T19:28:26.418 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.410+0000 7f6b5f2a3e40 1 -- 172.21.15.167:0/3411784563 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x5584aab24210 con 0x5584aab1e0e0 2023-10-05T19:28:26.418 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.410+0000 7f6a9ffff700 1 -- 172.21.15.167:0/3411784563 <== mon.0 v2:172.21.15.118:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f6b4402ae30 con 0x5584aab1e0e0 2023-10-05T19:28:26.418 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.410+0000 7f6b5f2a3e40 1 -- 172.21.15.167:0/3411784563 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x5584aab246b0 con 0x5584aab1e0e0 2023-10-05T19:28:26.418 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.410+0000 7f6a9ffff700 1 -- 172.21.15.167:0/3411784563 <== mon.0 v2:172.21.15.118:3300/0 4 ==== mgrmap(e 5) v1 ==== 40329+0+0 (secure 0 0 0) 0x7f6b44050440 con 0x5584aab1e0e0 2023-10-05T19:28:26.418 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.410+0000 7f6a9ffff700 1 --2- 172.21.15.167:0/3411784563 >> [v2:172.21.15.167:6824/29889,v1:172.21.15.167:6825/29889] conn(0x7f6aa4030c00 0x7f6aa40330b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-05T19:28:26.418 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.410+0000 7f6a9ffff700 1 -- 172.21.15.167:0/3411784563 <== mon.0 v2:172.21.15.118:3300/0 5 ==== osd_map(30..30 src has 1..30) v4 ==== 6057+0+0 (secure 0 0 0) 0x7f6b4402f070 con 0x5584aab1e0e0 2023-10-05T19:28:26.418 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.410+0000 7f6b5d5d9700 1 --2- 172.21.15.167:0/3411784563 >> [v2:172.21.15.167:6824/29889,v1:172.21.15.167:6825/29889] conn(0x7f6aa4030c00 0x7f6aa40330b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-05T19:28:26.418 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.410+0000 7f6b5f2a3e40 5 note: GC not initialized 2023-10-05T19:28:26.419 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.414+0000 7f6b5d5d9700 1 --2- 172.21.15.167:0/3411784563 >> [v2:172.21.15.167:6824/29889,v1:172.21.15.167:6825/29889] conn(0x7f6aa4030c00 0x7f6aa40330b0 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x5584aab10030 tx=0x7f6b54000f90 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-05T19:28:26.420 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.414+0000 7f6a9d7fa700 20 reqs_thread_entry: start 2023-10-05T19:28:26.420 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.414+0000 7f6a96ffd700 20 reqs_thread_entry: start 2023-10-05T19:28:26.420 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.414+0000 7f6b5f2a3e40 20 init_complete bucket index max shards: 11 2023-10-05T19:28:26.421 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.414+0000 7f6b5f2a3e40 10 rgw notify: Started notification manager with: 1 workers 2023-10-05T19:28:26.421 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.414+0000 7f6a957fa700 20 rgw notify: INFO: next queues processing will happen at: Thu Oct 5 19:28:56 2023 2023-10-05T19:28:26.421 INFO:teuthology.orchestra.run.smithi167.stderr: 2023-10-05T19:28:26.421 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.414+0000 7f6a94ff9700 20 reqs_thread_entry: start 2023-10-05T19:28:26.421 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.414+0000 7f6b5f2a3e40 20 get_system_obj_state: rctx=0x7ffe47869e18 obj=default.rgw.meta:users.uid:foo state=0x5584aa9a90b0 s->prefetch_data=0 2023-10-05T19:28:26.422 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.414+0000 7f6b5f2a3e40 10 cache get: name=default.rgw.meta+users.uid+foo : miss 2023-10-05T19:28:26.422 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.414+0000 7f6b5f2a3e40 1 -- 172.21.15.167:0/581366395 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:102 8.18 8:1e6ca5ed:users.uid::foo:head [call version.read in=11b,getxattrs,stat] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x5584aab8c320 con 0x5584aaad42f0 2023-10-05T19:28:26.422 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.414+0000 7f6b5d5d9700 1 -- 172.21.15.167:0/581366395 <== osd.1 v2:172.21.15.167:6800/30035 35 ==== osd_op_reply(102 foo [call out=48b,getxattrs out=75b,stat out=16b] v0'0 uv1 ondisk = 0) v8 ==== 231+0+139 (crc 0 0 0) 0x7f6b54050040 con 0x5584aaad42f0 2023-10-05T19:28:26.423 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.414+0000 7f6b5f2a3e40 10 cache put: name=default.rgw.meta+users.uid+foo info.flags=0x16 2023-10-05T19:28:26.423 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.414+0000 7f6b5f2a3e40 10 adding default.rgw.meta+users.uid+foo to cache LRU end 2023-10-05T19:28:26.423 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.414+0000 7f6b5f2a3e40 10 updating xattr: name=ceph.objclass.version bl.length()=42 2023-10-05T19:28:26.423 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.414+0000 7f6b5f2a3e40 20 get_system_obj_state: s->obj_tag was set empty 2023-10-05T19:28:26.423 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.414+0000 7f6b5f2a3e40 20 Read xattr: user.rgw.idtag 2023-10-05T19:28:26.423 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.414+0000 7f6b5f2a3e40 10 cache get: name=default.rgw.meta+users.uid+foo : type miss (requested=0x13, cached=0x16) 2023-10-05T19:28:26.423 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.414+0000 7f6b5f2a3e40 20 rados->read ofs=0 len=0 2023-10-05T19:28:26.423 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.418+0000 7f6b5f2a3e40 1 -- 172.21.15.167:0/581366395 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:103 8.18 8:1e6ca5ed:users.uid::foo:head [call version.check_conds in=122b,call version.read in=11b,read 0~0,getxattrs] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x5584aab8edb0 con 0x5584aaad42f0 2023-10-05T19:28:26.424 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.418+0000 7f6b5d5d9700 1 -- 172.21.15.167:0/581366395 <== osd.1 v2:172.21.15.167:6800/30035 36 ==== osd_op_reply(103 foo [call,call out=48b,read 0~357 out=357b,getxattrs out=75b] v0'0 uv1 ondisk = 0) v8 ==== 273+0+480 (crc 0 0 0) 0x7f6b54050040 con 0x5584aaad42f0 2023-10-05T19:28:26.424 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.418+0000 7f6b5f2a3e40 20 rados_obj.operate() r=0 bl.length=357 2023-10-05T19:28:26.424 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.418+0000 7f6b5f2a3e40 10 cache put: name=default.rgw.meta+users.uid+foo info.flags=0x13 2023-10-05T19:28:26.425 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.418+0000 7f6b5f2a3e40 10 moving default.rgw.meta+users.uid+foo to cache LRU end 2023-10-05T19:28:26.425 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.418+0000 7f6b5f2a3e40 10 updating xattr: name=ceph.objclass.version bl.length()=42 2023-10-05T19:28:26.425 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.418+0000 7f6b5f2a3e40 1 -- 172.21.15.167:0/581366395 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:104 8.9 8:96d98480:users.uid::foo.buckets:head [call user.list_buckets in=34b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x5584aab8edb0 con 0x7f6b4c0176e0 2023-10-05T19:28:26.425 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.418+0000 7f6b5c5d7700 1 -- 172.21.15.167:0/581366395 <== osd.2 v2:172.21.15.167:6816/30037 17 ==== osd_op_reply(104 foo.buckets [call] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 ==== 155+0+0 (crc 0 0 0) 0x7f6b44032040 con 0x7f6b4c0176e0 2023-10-05T19:28:26.425 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.418+0000 7f6b5f2a3e40 1 -- 172.21.15.167:0/581366395 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:105 8.18 8:1e6ca5ed:users.uid::foo:head [delete,create,call version.set in=58b,writefull 0~357 in=357b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x5584aab90660 con 0x5584aaad42f0 2023-10-05T19:28:26.427 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.422+0000 7f6b5d5d9700 1 -- 172.21.15.167:0/581366395 <== osd.1 v2:172.21.15.167:6800/30035 37 ==== osd_op_reply(105 foo [delete,create,call,writefull 0~357] v30'2 uv2 ondisk = 0) v8 ==== 273+0+0 (crc 0 0 0) 0x7f6b54050040 con 0x5584aaad42f0 2023-10-05T19:28:26.427 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.422+0000 7f6b5f2a3e40 10 cache put: name=default.rgw.meta+users.uid+foo info.flags=0x17 2023-10-05T19:28:26.428 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.422+0000 7f6b5f2a3e40 10 moving default.rgw.meta+users.uid+foo to cache LRU end 2023-10-05T19:28:26.428 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.422+0000 7f6b5f2a3e40 10 distributing notification oid=default.rgw.control:notify.4 cni=[op: 0, obj: default.rgw.meta:users.uid:foo, ofs0, ns] 2023-10-05T19:28:26.428 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.422+0000 7f6b5f2a3e40 1 -- 172.21.15.167:0/581366395 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:106 7.d 7:b4812045:::notify.4:head [notify cookie 94028288273488 in=545b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x5584aab8b0a0 con 0x5584aaad42f0 2023-10-05T19:28:26.428 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.422+0000 7f6b5d5d9700 1 -- 172.21.15.167:0/581366395 <== osd.1 v2:172.21.15.167:6800/30035 38 ==== watch-notify(notify (1) cookie 94028287761984 notify 128849018881 ret 0) v3 ==== 575+0+0 (crc 0 0 0) 0x7f6b5403d040 con 0x5584aaad42f0 2023-10-05T19:28:26.429 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.422+0000 7f6b5add4700 10 rgw watcher librados: RGWWatcher::handle_notify() notify_id 128849018881 cookie 94028287761984 notifier 4293 bl.length()=533 2023-10-05T19:28:26.429 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.422+0000 7f6b5add4700 10 rgw watcher librados: cache put: name=default.rgw.meta+users.uid+foo info.flags=0x17 2023-10-05T19:28:26.429 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.422+0000 7f6b5add4700 10 rgw watcher librados: moving default.rgw.meta+users.uid+foo to cache LRU end 2023-10-05T19:28:26.429 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.422+0000 7f6b5add4700 1 -- 172.21.15.167:0/581366395 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:107 7.d 7:b4812045:::notify.4:head [notify-ack in=20b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x7f6b40003d90 con 0x5584aaad42f0 2023-10-05T19:28:26.429 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.422+0000 7f6b5d5d9700 1 -- 172.21.15.167:0/581366395 <== osd.1 v2:172.21.15.167:6800/30035 39 ==== osd_op_reply(106 notify.4 [notify cookie 94028288273488 out=8b] v0'0 uv33 ondisk = 0) v8 ==== 152+0+8 (crc 0 0 0) 0x7f6b54050040 con 0x5584aaad42f0 2023-10-05T19:28:26.429 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.422+0000 7f6b5d5d9700 1 -- 172.21.15.167:0/581366395 <== osd.1 v2:172.21.15.167:6800/30035 40 ==== osd_op_reply(107 notify.4 [notify-ack] v0'0 uv33 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f6b54050040 con 0x5584aaad42f0 2023-10-05T19:28:26.429 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.422+0000 7f6b5d5d9700 1 -- 172.21.15.167:0/581366395 <== osd.1 v2:172.21.15.167:6800/30035 41 ==== watch-notify(notify_complete (2) cookie 94028288273488 notify 128849018881 ret 0) v3 ==== 42+0+48 (crc 0 0 0) 0x7f6b54002b70 con 0x5584aaad42f0 2023-10-05T19:28:26.430 INFO:teuthology.orchestra.run.smithi167.stdout:{ 2023-10-05T19:28:26.430 INFO:teuthology.orchestra.run.smithi167.stdout: "user_id": "foo", 2023-10-05T19:28:26.430 INFO:teuthology.orchestra.run.smithi167.stdout: "display_name": "Foo", 2023-10-05T19:28:26.430 INFO:teuthology.orchestra.run.smithi167.stdout: "email": "foo@foo.com", 2023-10-05T19:28:26.430 INFO:teuthology.orchestra.run.smithi167.stdout: "suspended": 1, 2023-10-05T19:28:26.430 INFO:teuthology.orchestra.run.smithi167.stdout: "max_buckets": 4, 2023-10-05T19:28:26.430 INFO:teuthology.orchestra.run.smithi167.stdout: "subusers": [], 2023-10-05T19:28:26.430 INFO:teuthology.orchestra.run.smithi167.stdout: "keys": [ 2023-10-05T19:28:26.431 INFO:teuthology.orchestra.run.smithi167.stdout: { 2023-10-05T19:28:26.431 INFO:teuthology.orchestra.run.smithi167.stdout: "user": "foo", 2023-10-05T19:28:26.431 INFO:teuthology.orchestra.run.smithi167.stdout: "access_key": "9te6NH5mcdcq0Tc5i8i1", 2023-10-05T19:28:26.431 INFO:teuthology.orchestra.run.smithi167.stdout: "secret_key": "Ny4IOauQoL18Gp2zM7lC1vLmoawgqcYP/YGcWfXu" 2023-10-05T19:28:26.431 INFO:teuthology.orchestra.run.smithi167.stdout: } 2023-10-05T19:28:26.431 INFO:teuthology.orchestra.run.smithi167.stdout: ], 2023-10-05T19:28:26.431 INFO:teuthology.orchestra.run.smithi167.stdout: "swift_keys": [], 2023-10-05T19:28:26.431 INFO:teuthology.orchestra.run.smithi167.stdout: "caps": [], 2023-10-05T19:28:26.431 INFO:teuthology.orchestra.run.smithi167.stdout: "op_mask": "read, write, delete", 2023-10-05T19:28:26.431 INFO:teuthology.orchestra.run.smithi167.stdout: "default_placement": "", 2023-10-05T19:28:26.432 INFO:teuthology.orchestra.run.smithi167.stdout: "default_storage_class": "", 2023-10-05T19:28:26.432 INFO:teuthology.orchestra.run.smithi167.stdout: "placement_tags": [], 2023-10-05T19:28:26.432 INFO:teuthology.orchestra.run.smithi167.stdout: "bucket_quota": { 2023-10-05T19:28:26.432 INFO:teuthology.orchestra.run.smithi167.stdout: "enabled": false, 2023-10-05T19:28:26.432 INFO:teuthology.orchestra.run.smithi167.stdout: "check_on_raw": false, 2023-10-05T19:28:26.432 INFO:teuthology.orchestra.run.smithi167.stdout: "max_size": -1, 2023-10-05T19:28:26.432 INFO:teuthology.orchestra.run.smithi167.stdout: "max_size_kb": 0, 2023-10-05T19:28:26.432 INFO:teuthology.orchestra.run.smithi167.stdout: "max_objects": -1 2023-10-05T19:28:26.432 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2023-10-05T19:28:26.432 INFO:teuthology.orchestra.run.smithi167.stdout: "user_quota": { 2023-10-05T19:28:26.433 INFO:teuthology.orchestra.run.smithi167.stdout: "enabled": false, 2023-10-05T19:28:26.433 INFO:teuthology.orchestra.run.smithi167.stdout: "check_on_raw": false, 2023-10-05T19:28:26.433 INFO:teuthology.orchestra.run.smithi167.stdout: "max_size": -1, 2023-10-05T19:28:26.433 INFO:teuthology.orchestra.run.smithi167.stdout: "max_size_kb": 0, 2023-10-05T19:28:26.433 INFO:teuthology.orchestra.run.smithi167.stdout: "max_objects": -1 2023-10-05T19:28:26.433 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2023-10-05T19:28:26.433 INFO:teuthology.orchestra.run.smithi167.stdout: "temp_url_keys": [], 2023-10-05T19:28:26.433 INFO:teuthology.orchestra.run.smithi167.stdout: "type": "rgw", 2023-10-05T19:28:26.433 INFO:teuthology.orchestra.run.smithi167.stdout: "mfa_ids": [] 2023-10-05T19:28:26.433 INFO:teuthology.orchestra.run.smithi167.stdout:} 2023-10-05T19:28:26.433 INFO:teuthology.orchestra.run.smithi167.stdout: 2023-10-05T19:28:26.434 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.422+0000 7f6b5f2a3e40 1 -- 172.21.15.167:0/581366395 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:108 7.1f 7:f95f44c2:::notify.0:head [watch unwatch cookie 94028287747456] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x5584aaae5830 con 0x7f6b4c0176e0 2023-10-05T19:28:26.434 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.426+0000 7f6b5c5d7700 1 -- 172.21.15.167:0/581366395 <== osd.2 v2:172.21.15.167:6816/30037 18 ==== osd_op_reply(108 notify.0 [watch unwatch cookie 94028287747456] v30'35 uv33 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f6b44032040 con 0x7f6b4c0176e0 2023-10-05T19:28:26.434 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.426+0000 7f6b5f2a3e40 20 remove_watcher() i=0 2023-10-05T19:28:26.434 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.426+0000 7f6b5f2a3e40 2 removed watcher, disabling cache 2023-10-05T19:28:26.434 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.426+0000 7f6b5f2a3e40 1 -- 172.21.15.167:0/581366395 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:109 7.0 7:05bf5b68:::notify.1:head [watch unwatch cookie 94028287752000] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x5584aaae5830 con 0x5584aaad42f0 2023-10-05T19:28:26.435 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.430+0000 7f6b5d5d9700 1 -- 172.21.15.167:0/581366395 <== osd.1 v2:172.21.15.167:6800/30035 42 ==== osd_op_reply(109 notify.1 [watch unwatch cookie 94028287752000] v30'35 uv33 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f6b54050040 con 0x5584aaad42f0 2023-10-05T19:28:26.436 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.430+0000 7f6b5f2a3e40 20 remove_watcher() i=1 2023-10-05T19:28:26.436 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.430+0000 7f6b5f2a3e40 1 -- 172.21.15.167:0/581366395 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:110 7.15 7:a93a5511:::notify.2:head [watch unwatch cookie 94028287755552] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x5584aaae5830 con 0x5584aab05660 2023-10-05T19:28:26.438 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.434+0000 7f6b5cdd8700 1 -- 172.21.15.167:0/581366395 <== osd.3 v2:172.21.15.167:6808/30036 24 ==== osd_op_reply(110 notify.2 [watch unwatch cookie 94028287755552] v30'35 uv33 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f6b4c052040 con 0x5584aab05660 2023-10-05T19:28:26.438 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.434+0000 7f6b5f2a3e40 20 remove_watcher() i=2 2023-10-05T19:28:26.439 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.434+0000 7f6b5f2a3e40 1 -- 172.21.15.167:0/581366395 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:111 7.e 7:7759931f:::notify.3:head [watch unwatch cookie 94028287754288] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x5584aaae5830 con 0x7f6b4c0176e0 2023-10-05T19:28:26.441 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.434+0000 7f6b5c5d7700 1 -- 172.21.15.167:0/581366395 <== osd.2 v2:172.21.15.167:6816/30037 19 ==== osd_op_reply(111 notify.3 [watch unwatch cookie 94028287754288] v30'35 uv33 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f6b44032040 con 0x7f6b4c0176e0 2023-10-05T19:28:26.441 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.434+0000 7f6b5f2a3e40 20 remove_watcher() i=3 2023-10-05T19:28:26.441 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.434+0000 7f6b5f2a3e40 1 -- 172.21.15.167:0/581366395 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:112 7.d 7:b4812045:::notify.4:head [watch unwatch cookie 94028287761984] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x5584aaae5830 con 0x5584aaad42f0 2023-10-05T19:28:26.443 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.438+0000 7f6b5d5d9700 1 -- 172.21.15.167:0/581366395 <== osd.1 v2:172.21.15.167:6800/30035 43 ==== osd_op_reply(112 notify.4 [watch unwatch cookie 94028287761984] v30'35 uv33 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f6b54050040 con 0x5584aaad42f0 2023-10-05T19:28:26.444 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.438+0000 7f6b5f2a3e40 20 remove_watcher() i=4 2023-10-05T19:28:26.444 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.438+0000 7f6b5f2a3e40 1 -- 172.21.15.167:0/581366395 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:113 7.3 7:c609908c:::notify.5:head [watch unwatch cookie 94028287766160] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x5584aab90660 con 0x7f6b44010070 2023-10-05T19:28:26.446 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.442+0000 7f6b5d5d9700 1 -- 172.21.15.167:0/581366395 <== osd.0 v2:172.21.15.118:6800/27690 29 ==== osd_op_reply(113 notify.5 [watch unwatch cookie 94028287766160] v30'35 uv33 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f6b54050040 con 0x7f6b44010070 2023-10-05T19:28:26.447 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.442+0000 7f6b5f2a3e40 20 remove_watcher() i=5 2023-10-05T19:28:26.447 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.442+0000 7f6b5f2a3e40 1 -- 172.21.15.167:0/581366395 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:114 7.14 7:2b04a3e9:::notify.6:head [watch unwatch cookie 94028287757760] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x5584aab90660 con 0x7f6b44010070 2023-10-05T19:28:26.449 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.442+0000 7f6b5d5d9700 1 -- 172.21.15.167:0/581366395 <== osd.0 v2:172.21.15.118:6800/27690 30 ==== osd_op_reply(114 notify.6 [watch unwatch cookie 94028287757760] v30'35 uv33 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f6b54050040 con 0x7f6b44010070 2023-10-05T19:28:26.449 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.442+0000 7f6b5f2a3e40 20 remove_watcher() i=6 2023-10-05T19:28:26.449 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.442+0000 7f6b5f2a3e40 1 -- 172.21.15.167:0/581366395 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:115 7.9 7:93e5b521:::notify.7:head [watch unwatch cookie 94028287767264] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x5584aab90660 con 0x7f6b44010070 2023-10-05T19:28:26.452 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.446+0000 7f6b5d5d9700 1 -- 172.21.15.167:0/581366395 <== osd.0 v2:172.21.15.118:6800/27690 31 ==== osd_op_reply(115 notify.7 [watch unwatch cookie 94028287767264] v30'35 uv33 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f6b54050040 con 0x7f6b44010070 2023-10-05T19:28:26.452 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.446+0000 7f6b5f2a3e40 20 remove_watcher() i=7 2023-10-05T19:28:26.453 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.446+0000 7f6b2e7e4700 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2023-10-05T19:28:26.454 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.446+0000 7f6b5f2a3e40 1 -- 172.21.15.167:0/581366395 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:116 6.4 6:22d26bf9:::data_loggenerations_metadata:head [watch unwatch cookie 94028287771856] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x5584aab90660 con 0x5584aaad42f0 2023-10-05T19:28:26.457 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.450+0000 7f6b5d5d9700 1 -- 172.21.15.167:0/581366395 <== osd.1 v2:172.21.15.167:6800/30035 44 ==== osd_op_reply(116 data_loggenerations_metadata [watch unwatch cookie 94028287771856] v30'57 uv1 ondisk = 0) v8 ==== 172+0+0 (crc 0 0 0) 0x7f6b54050040 con 0x5584aaad42f0 2023-10-05T19:28:26.457 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.450+0000 7f6b5f2a3e40 1 -- 172.21.15.167:0/581366395 >> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] conn(0x7f6b44010070 msgr2=0x7f6b4401c430 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-05T19:28:26.457 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.450+0000 7f6b5f2a3e40 1 --2- 172.21.15.167:0/581366395 >> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] conn(0x7f6b44010070 0x7f6b4401c430 crc :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-05T19:28:26.458 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.450+0000 7f6b5f2a3e40 1 -- 172.21.15.167:0/581366395 >> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] conn(0x5584aaad42f0 msgr2=0x5584aab01800 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-05T19:28:26.458 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.450+0000 7f6b5f2a3e40 1 --2- 172.21.15.167:0/581366395 >> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] conn(0x5584aaad42f0 0x5584aab01800 crc :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-05T19:28:26.458 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.450+0000 7f6b5f2a3e40 1 -- 172.21.15.167:0/581366395 >> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] conn(0x7f6b4c0176e0 msgr2=0x7f6b4c0420e0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-05T19:28:26.458 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.450+0000 7f6b5f2a3e40 1 --2- 172.21.15.167:0/581366395 >> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] conn(0x7f6b4c0176e0 0x7f6b4c0420e0 crc :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-05T19:28:26.458 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.450+0000 7f6b5f2a3e40 1 -- 172.21.15.167:0/581366395 >> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] conn(0x5584aab05660 msgr2=0x5584aab07a40 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-05T19:28:26.458 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.450+0000 7f6b5f2a3e40 1 --2- 172.21.15.167:0/581366395 >> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] conn(0x5584aab05660 0x5584aab07a40 crc :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-05T19:28:26.458 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.450+0000 7f6b5f2a3e40 1 -- 172.21.15.167:0/581366395 >> [v2:172.21.15.167:6824/29889,v1:172.21.15.167:6825/29889] conn(0x7f6b3c030c00 msgr2=0x7f6b3c0330b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-05T19:28:26.458 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.450+0000 7f6b5f2a3e40 1 --2- 172.21.15.167:0/581366395 >> [v2:172.21.15.167:6824/29889,v1:172.21.15.167:6825/29889] conn(0x7f6b3c030c00 0x7f6b3c0330b0 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f6b44000f80 tx=0x7f6b44013000 comp rx=0 tx=0).stop 2023-10-05T19:28:26.458 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.450+0000 7f6b5f2a3e40 1 -- 172.21.15.167:0/581366395 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5584aaa443d0 msgr2=0x5584aaaf5d60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-05T19:28:26.458 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.450+0000 7f6b5d5d9700 1 -- 172.21.15.167:0/581366395 reap_dead start 2023-10-05T19:28:26.459 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.450+0000 7f6b5f2a3e40 1 --2- 172.21.15.167:0/581366395 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5584aaa443d0 0x5584aaaf5d60 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x5584aaaeb670 tx=0x7f6b4c016000 comp rx=0 tx=0).stop 2023-10-05T19:28:26.459 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.450+0000 7f6b5f2a3e40 1 -- 172.21.15.167:0/581366395 shutdown_connections 2023-10-05T19:28:26.459 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.450+0000 7f6b5f2a3e40 1 --2- 172.21.15.167:0/581366395 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5584aaa443d0 0x5584aaaf5d60 unknown :-1 s=CLOSED pgs=203 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-05T19:28:26.459 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.450+0000 7f6b5f2a3e40 1 -- 172.21.15.167:0/581366395 >> 172.21.15.167:0/581366395 conn(0x5584aaaca720 msgr2=0x5584aaae95d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-05T19:28:26.459 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.450+0000 7f6b5f2a3e40 1 -- 172.21.15.167:0/581366395 shutdown_connections 2023-10-05T19:28:26.459 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.450+0000 7f6b5f2a3e40 1 -- 172.21.15.167:0/581366395 wait complete. 2023-10-05T19:28:26.460 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.454+0000 7f6b5f2a3e40 1 -- 172.21.15.167:0/3411784563 >> [v2:172.21.15.167:6824/29889,v1:172.21.15.167:6825/29889] conn(0x7f6aa4030c00 msgr2=0x7f6aa40330b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-05T19:28:26.460 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.454+0000 7f6b5f2a3e40 1 --2- 172.21.15.167:0/3411784563 >> [v2:172.21.15.167:6824/29889,v1:172.21.15.167:6825/29889] conn(0x7f6aa4030c00 0x7f6aa40330b0 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x5584aab10030 tx=0x7f6b54000f90 comp rx=0 tx=0).stop 2023-10-05T19:28:26.460 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.454+0000 7f6b5f2a3e40 1 -- 172.21.15.167:0/3411784563 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5584aab1e0e0 msgr2=0x5584aab23c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-05T19:28:26.460 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.454+0000 7f6b5f2a3e40 1 --2- 172.21.15.167:0/3411784563 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5584aab1e0e0 0x5584aab23c10 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7f6b4400efd0 tx=0x7f6b4402f000 comp rx=0 tx=0).stop 2023-10-05T19:28:26.460 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.454+0000 7f6b5f2a3e40 1 -- 172.21.15.167:0/3411784563 shutdown_connections 2023-10-05T19:28:26.461 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.454+0000 7f6b5f2a3e40 1 --2- 172.21.15.167:0/3411784563 >> [v2:172.21.15.167:6824/29889,v1:172.21.15.167:6825/29889] conn(0x7f6aa4030c00 0x7f6aa40330b0 unknown :-1 s=CLOSED pgs=75 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-05T19:28:26.461 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.454+0000 7f6b5f2a3e40 1 --2- 172.21.15.167:0/3411784563 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5584aab1e0e0 0x5584aab23c10 unknown :-1 s=CLOSED pgs=205 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-05T19:28:26.461 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.454+0000 7f6b5f2a3e40 1 -- 172.21.15.167:0/3411784563 >> 172.21.15.167:0/3411784563 conn(0x5584aab194a0 msgr2=0x5584aab17540 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-05T19:28:26.461 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.454+0000 7f6b5f2a3e40 1 -- 172.21.15.167:0/3411784563 shutdown_connections 2023-10-05T19:28:26.461 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.454+0000 7f6b5f2a3e40 1 -- 172.21.15.167:0/3411784563 wait complete. 2023-10-05T19:28:26.464 DEBUG:tasks.util.rgw: json result: {'user_id': 'foo', 'display_name': 'Foo', 'email': 'foo@foo.com', 'suspended': 1, 'max_buckets': 4, 'subusers': [], 'keys': [{'user': 'foo', 'access_key': '9te6NH5mcdcq0Tc5i8i1', 'secret_key': 'Ny4IOauQoL18Gp2zM7lC1vLmoawgqcYP/YGcWfXu'}], 'swift_keys': [], 'caps': [], 'op_mask': 'read, write, delete', 'default_placement': '', 'default_storage_class': '', 'placement_tags': [], 'bucket_quota': {'enabled': False, 'check_on_raw': False, 'max_size': -1, 'max_size_kb': 0, 'max_objects': -1}, 'user_quota': {'enabled': False, 'check_on_raw': False, 'max_size': -1, 'max_size_kb': 0, 'max_objects': -1}, 'temp_url_keys': [], 'type': 'rgw', 'mfa_ids': []} 2023-10-05T19:28:26.464 INFO:tasks.util.rgw:rgwadmin: client.0 : ['user', 'info', '--uid', 'foo'] 2023-10-05T19:28:26.464 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'ceph', 'user', 'info', '--uid', 'foo'] 2023-10-05T19:28:26.465 DEBUG:teuthology.orchestra.run.smithi167:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster ceph user info --uid foo 2023-10-05T19:28:26.504 INFO:teuthology.orchestra.run.smithi167.stderr:ignoring --setuser ceph since I am not root 2023-10-05T19:28:26.504 INFO:teuthology.orchestra.run.smithi167.stderr:ignoring --setgroup ceph since I am not root 2023-10-05T19:28:26.504 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.498+0000 7fe1047a9e40 1 Processor -- start 2023-10-05T19:28:26.504 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.498+0000 7fe1047a9e40 1 -- start start 2023-10-05T19:28:26.505 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.498+0000 7fe1047a9e40 1 --2- >> v2:172.21.15.118:3300/0 conn(0x561c5a44e7e0 0x561c5a44d6d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-05T19:28:26.505 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.498+0000 7fe1047a9e40 1 -- --> v1:172.21.15.118:6789/0 -- auth(proto 0 26 bytes epoch 0) v1 -- 0x561c5a39abd0 con 0x561c5a44e410 2023-10-05T19:28:26.505 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.498+0000 7fe1047a9e40 1 -- --> v2:172.21.15.118:3300/0 -- mon_getmap magic: 0 v1 -- 0x561c5a357ee0 con 0x561c5a44e7e0 2023-10-05T19:28:26.505 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.498+0000 7fe101add700 1 --2- >> v2:172.21.15.118:3300/0 conn(0x561c5a44e7e0 0x561c5a44d6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-05T19:28:26.506 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.498+0000 7fe1022de700 1 --1- >> v1:172.21.15.118:6789/0 conn(0x561c5a44e410 0x561c5a44cfc0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.118:6789/0 says I am v1:172.21.15.167:56398/0 (socket says 172.21.15.167:56398) 2023-10-05T19:28:26.506 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.498+0000 7fe1022de700 1 -- 172.21.15.167:0/1366458948 learned_addr learned my addr 172.21.15.167:0/1366458948 (peer_addr_for_me v1:172.21.15.167:0/0) 2023-10-05T19:28:26.506 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.502+0000 7fe101add700 1 -- 172.21.15.167:0/1366458948 >> v1:172.21.15.118:6789/0 conn(0x561c5a44e410 legacy=0x561c5a44cfc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-05T19:28:26.507 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.502+0000 7fe101add700 1 -- 172.21.15.167:0/1366458948 --> v2:172.21.15.118:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x561c5a44dcd0 con 0x561c5a44e7e0 2023-10-05T19:28:26.507 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.502+0000 7fe101add700 1 --2- 172.21.15.167:0/1366458948 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x561c5a44e7e0 0x561c5a44d6d0 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7fe0ec00a5d0 tx=0x7fe0ec005a50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=70c54e6a102aa48f server_cookie=0 in_seq=0 out_seq=0 2023-10-05T19:28:26.507 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.502+0000 7fe1012dc700 1 -- 172.21.15.167:0/1366458948 <== mon.0 v2:172.21.15.118:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fe0ec0086a0 con 0x561c5a44e7e0 2023-10-05T19:28:26.507 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.502+0000 7fe1012dc700 1 -- 172.21.15.167:0/1366458948 <== mon.0 v2:172.21.15.118:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe0ec008800 con 0x561c5a44e7e0 2023-10-05T19:28:26.507 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.502+0000 7fe1012dc700 1 -- 172.21.15.167:0/1366458948 <== mon.0 v2:172.21.15.118:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fe0ec008ad0 con 0x561c5a44e7e0 2023-10-05T19:28:26.508 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.502+0000 7fe1047a9e40 1 -- 172.21.15.167:0/1366458948 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x561c5a44e7e0 msgr2=0x561c5a44d6d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-05T19:28:26.508 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.502+0000 7fe1047a9e40 1 --2- 172.21.15.167:0/1366458948 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x561c5a44e7e0 0x561c5a44d6d0 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7fe0ec00a5d0 tx=0x7fe0ec005a50 comp rx=0 tx=0).stop 2023-10-05T19:28:26.508 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.502+0000 7fe1047a9e40 1 -- 172.21.15.167:0/1366458948 shutdown_connections 2023-10-05T19:28:26.508 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.502+0000 7fe1047a9e40 1 --2- 172.21.15.167:0/1366458948 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x561c5a44e7e0 0x561c5a44d6d0 unknown :-1 s=CLOSED pgs=207 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-05T19:28:26.508 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.502+0000 7fe1047a9e40 1 -- 172.21.15.167:0/1366458948 >> 172.21.15.167:0/1366458948 conn(0x561c5a448720 msgr2=0x561c5a44ab30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-05T19:28:26.508 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.502+0000 7fe1047a9e40 1 -- 172.21.15.167:0/1366458948 shutdown_connections 2023-10-05T19:28:26.508 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.502+0000 7fe1047a9e40 1 -- 172.21.15.167:0/1366458948 wait complete. 2023-10-05T19:28:26.509 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.502+0000 7fe1047a9e40 1 Processor -- start 2023-10-05T19:28:26.509 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.502+0000 7fe1047a9e40 1 -- start start 2023-10-05T19:28:26.509 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.502+0000 7fe1047a9e40 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x561c5a44e410 0x561c5a46f120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-05T19:28:26.510 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.502+0000 7fe1047a9e40 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x561c5a31be30 con 0x561c5a44e410 2023-10-05T19:28:26.510 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.502+0000 7fe1022de700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x561c5a44e410 0x561c5a46f120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-05T19:28:26.510 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.502+0000 7fe1022de700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x561c5a44e410 0x561c5a46f120 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.118:3300/0 says I am v2:172.21.15.167:57336/0 (socket says 172.21.15.167:57336) 2023-10-05T19:28:26.510 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.502+0000 7fe1022de700 1 -- 172.21.15.167:0/4110757938 learned_addr learned my addr 172.21.15.167:0/4110757938 (peer_addr_for_me v2:172.21.15.167:0/0) 2023-10-05T19:28:26.510 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.502+0000 7fe1022de700 1 -- 172.21.15.167:0/4110757938 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe0ec005cf0 con 0x561c5a44e410 2023-10-05T19:28:26.511 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.502+0000 7fe1022de700 1 --2- 172.21.15.167:0/4110757938 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x561c5a44e410 0x561c5a46f120 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7fe0f4001bf0 tx=0x7fe0f40017a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=53e4387188dbc1f5 server_cookie=0 in_seq=0 out_seq=0 2023-10-05T19:28:26.511 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.506+0000 7fe0fbfff700 1 -- 172.21.15.167:0/4110757938 <== mon.0 v2:172.21.15.118:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fe0f400bbf0 con 0x561c5a44e410 2023-10-05T19:28:26.511 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.506+0000 7fe0fbfff700 1 -- 172.21.15.167:0/4110757938 <== mon.0 v2:172.21.15.118:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe0f400bd50 con 0x561c5a44e410 2023-10-05T19:28:26.511 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.506+0000 7fe0fbfff700 1 -- 172.21.15.167:0/4110757938 <== mon.0 v2:172.21.15.118:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fe0f400e470 con 0x561c5a44e410 2023-10-05T19:28:26.515 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.506+0000 7fe1047a9e40 1 -- 172.21.15.167:0/4110757938 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x561c5a44e410 msgr2=0x561c5a46f120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-05T19:28:26.515 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.506+0000 7fe1047a9e40 1 --2- 172.21.15.167:0/4110757938 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x561c5a44e410 0x561c5a46f120 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7fe0f4001bf0 tx=0x7fe0f40017a0 comp rx=0 tx=0).stop 2023-10-05T19:28:26.515 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.506+0000 7fe1047a9e40 1 -- 172.21.15.167:0/4110757938 shutdown_connections 2023-10-05T19:28:26.515 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.506+0000 7fe1047a9e40 1 --2- 172.21.15.167:0/4110757938 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x561c5a44e410 0x561c5a46f120 unknown :-1 s=CLOSED pgs=208 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-05T19:28:26.515 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.506+0000 7fe1047a9e40 1 -- 172.21.15.167:0/4110757938 >> 172.21.15.167:0/4110757938 conn(0x561c5a448720 msgr2=0x561c5a46ace0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-05T19:28:26.515 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.506+0000 7fe1047a9e40 1 -- 172.21.15.167:0/4110757938 shutdown_connections 2023-10-05T19:28:26.516 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.506+0000 7fe1047a9e40 1 -- 172.21.15.167:0/4110757938 wait complete. 2023-10-05T19:28:26.516 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.506+0000 7fe1047a9e40 1 Processor -- start 2023-10-05T19:28:26.516 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.506+0000 7fe1047a9e40 1 -- start start 2023-10-05T19:28:26.516 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.506+0000 7fe1047a9e40 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x561c5a44e7e0 0x561c5a476060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-05T19:28:26.516 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.506+0000 7fe1047a9e40 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x561c5a2eeed0 con 0x561c5a44e7e0 2023-10-05T19:28:26.516 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.506+0000 7fe1022de700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x561c5a44e7e0 0x561c5a476060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-05T19:28:26.516 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.506+0000 7fe1022de700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x561c5a44e7e0 0x561c5a476060 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.118:3300/0 says I am v2:172.21.15.167:57352/0 (socket says 172.21.15.167:57352) 2023-10-05T19:28:26.516 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.506+0000 7fe1022de700 1 -- 172.21.15.167:0/399129457 learned_addr learned my addr 172.21.15.167:0/399129457 (peer_addr_for_me v2:172.21.15.167:0/0) 2023-10-05T19:28:26.516 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.506+0000 7fe1022de700 1 -- 172.21.15.167:0/399129457 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x561c5a39abd0 con 0x561c5a44e7e0 2023-10-05T19:28:26.516 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.510+0000 7fe1022de700 1 --2- 172.21.15.167:0/399129457 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x561c5a44e7e0 0x561c5a476060 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x561c5a46b800 tx=0x7fe0f4016000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-05T19:28:26.516 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.510+0000 7fe0fa7fc700 1 -- 172.21.15.167:0/399129457 <== mon.0 v2:172.21.15.118:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fe0f401f070 con 0x561c5a44e7e0 2023-10-05T19:28:26.516 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.510+0000 7fe0fa7fc700 1 -- 172.21.15.167:0/399129457 <== mon.0 v2:172.21.15.118:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe0f401a2e0 con 0x561c5a44e7e0 2023-10-05T19:28:26.517 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.510+0000 7fe1047a9e40 1 -- 172.21.15.167:0/399129457 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x561c5a4765a0 con 0x561c5a44e7e0 2023-10-05T19:28:26.517 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.510+0000 7fe0fa7fc700 1 -- 172.21.15.167:0/399129457 <== mon.0 v2:172.21.15.118:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fe0f400bec0 con 0x561c5a44e7e0 2023-10-05T19:28:26.517 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.510+0000 7fe1047a9e40 1 -- 172.21.15.167:0/399129457 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x561c5a4769e0 con 0x561c5a44e7e0 2023-10-05T19:28:26.518 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.510+0000 7fe1047a9e40 20 rados->read ofs=0 len=0 2023-10-05T19:28:26.518 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.510+0000 7fe0fa7fc700 1 -- 172.21.15.167:0/399129457 <== mon.0 v2:172.21.15.118:3300/0 4 ==== mgrmap(e 5) v1 ==== 40329+0+0 (secure 0 0 0) 0x7fe0f401a960 con 0x561c5a44e7e0 2023-10-05T19:28:26.518 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.510+0000 7fe0fa7fc700 1 --2- 172.21.15.167:0/399129457 >> [v2:172.21.15.167:6824/29889,v1:172.21.15.167:6825/29889] conn(0x7fe0e4030bb0 0x7fe0e4033060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-05T19:28:26.519 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.510+0000 7fe0fa7fc700 1 -- 172.21.15.167:0/399129457 <== mon.0 v2:172.21.15.118:3300/0 5 ==== osd_map(30..30 src has 1..30) v4 ==== 6057+0+0 (secure 0 0 0) 0x7fe0f4045020 con 0x561c5a44e7e0 2023-10-05T19:28:26.519 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.510+0000 7fe101add700 1 --2- 172.21.15.167:0/399129457 >> [v2:172.21.15.167:6824/29889,v1:172.21.15.167:6825/29889] conn(0x7fe0e4030bb0 0x7fe0e4033060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-05T19:28:26.519 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.510+0000 7fe1047a9e40 1 --2- 172.21.15.167:0/399129457 >> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] conn(0x561c5a47fae0 0x561c5a481ec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-05T19:28:26.519 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.510+0000 7fe1047a9e40 1 -- 172.21.15.167:0/399129457 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:1 5.12 5:49953fa1:::default.realm:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x561c5a482530 con 0x561c5a47fae0 2023-10-05T19:28:26.519 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.510+0000 7fe102adf700 1 --2- 172.21.15.167:0/399129457 >> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] conn(0x561c5a47fae0 0x561c5a481ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-05T19:28:26.519 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.514+0000 7fe101add700 1 --2- 172.21.15.167:0/399129457 >> [v2:172.21.15.167:6824/29889,v1:172.21.15.167:6825/29889] conn(0x7fe0e4030bb0 0x7fe0e4033060 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7fe0ec005a90 tx=0x7fe0ec015000 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-05T19:28:26.519 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.514+0000 7fe102adf700 1 --2- 172.21.15.167:0/399129457 >> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] conn(0x561c5a47fae0 0x561c5a481ec0 crc :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-05T19:28:26.521 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.514+0000 7fe102adf700 1 -- 172.21.15.167:0/399129457 <== osd.1 v2:172.21.15.167:6800/30035 1 ==== osd_op_reply(1 default.realm [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 ==== 157+0+0 (crc 0 0 0) 0x7fe0fc00a5a0 con 0x561c5a47fae0 2023-10-05T19:28:26.521 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.514+0000 7fe1047a9e40 20 rados_obj.operate() r=-2 bl.length=0 2023-10-05T19:28:26.521 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.514+0000 7fe1047a9e40 20 realm 2023-10-05T19:28:26.521 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.514+0000 7fe1047a9e40 20 rados->read ofs=0 len=0 2023-10-05T19:28:26.521 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.514+0000 7fe1047a9e40 1 -- 172.21.15.167:0/399129457 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:2 5.12 5:49953fa1:::default.realm:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x561c5a47f150 con 0x561c5a47fae0 2023-10-05T19:28:26.521 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.514+0000 7fe102adf700 1 -- 172.21.15.167:0/399129457 <== osd.1 v2:172.21.15.167:6800/30035 2 ==== osd_op_reply(2 default.realm [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 ==== 157+0+0 (crc 0 0 0) 0x7fe0fc00a5a0 con 0x561c5a47fae0 2023-10-05T19:28:26.521 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.514+0000 7fe1047a9e40 20 rados_obj.operate() r=-2 bl.length=0 2023-10-05T19:28:26.521 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.514+0000 7fe1047a9e40 4 RGWPeriod::init failed to init realm id : (2) No such file or directory 2023-10-05T19:28:26.522 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.514+0000 7fe1047a9e40 20 rados->read ofs=0 len=0 2023-10-05T19:28:26.522 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.514+0000 7fe1047a9e40 1 -- 172.21.15.167:0/399129457 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:3 5.12 5:49953fa1:::default.realm:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x561c5a483010 con 0x561c5a47fae0 2023-10-05T19:28:26.522 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.514+0000 7fe102adf700 1 -- 172.21.15.167:0/399129457 <== osd.1 v2:172.21.15.167:6800/30035 3 ==== osd_op_reply(3 default.realm [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 ==== 157+0+0 (crc 0 0 0) 0x7fe0fc00a5a0 con 0x561c5a47fae0 2023-10-05T19:28:26.522 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.514+0000 7fe1047a9e40 20 rados_obj.operate() r=-2 bl.length=0 2023-10-05T19:28:26.522 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.514+0000 7fe1047a9e40 20 rados->read ofs=0 len=0 2023-10-05T19:28:26.522 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.514+0000 7fe1047a9e40 1 -- 172.21.15.167:0/399129457 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:4 5.f 5:f43b6ece:::zone_names.default:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x561c5a482bf0 con 0x561c5a47fae0 2023-10-05T19:28:26.522 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.514+0000 7fe102adf700 1 -- 172.21.15.167:0/399129457 <== osd.1 v2:172.21.15.167:6800/30035 4 ==== osd_op_reply(4 zone_names.default [read 0~46 out=46b] v0'0 uv1 ondisk = 0) v8 ==== 162+0+46 (crc 0 0 0) 0x7fe0fc00a5a0 con 0x561c5a47fae0 2023-10-05T19:28:26.522 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.514+0000 7fe1047a9e40 20 rados_obj.operate() r=0 bl.length=46 2023-10-05T19:28:26.522 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.514+0000 7fe1047a9e40 20 rados->read ofs=0 len=0 2023-10-05T19:28:26.522 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.514+0000 7fe1047a9e40 1 -- 172.21.15.167:0/399129457 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:5 5.1d 5:b8374eb4:::zone_info.9a9df965-6612-45d4-a45f-6e3c889bacc2:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x561c5a483a80 con 0x561c5a47fae0 2023-10-05T19:28:26.523 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.514+0000 7fe102adf700 1 -- 172.21.15.167:0/399129457 <== osd.1 v2:172.21.15.167:6800/30035 5 ==== osd_op_reply(5 zone_info.9a9df965-6612-45d4-a45f-6e3c889bacc2 [read 0~874 out=874b] v0'0 uv2 ondisk = 0) v8 ==== 190+0+874 (crc 0 0 0) 0x7fe0fc00a5a0 con 0x561c5a47fae0 2023-10-05T19:28:26.523 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.514+0000 7fe1047a9e40 20 rados_obj.operate() r=0 bl.length=874 2023-10-05T19:28:26.524 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.514+0000 7fe1047a9e40 20 searching for the correct realm 2023-10-05T19:28:26.524 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.514+0000 7fe1047a9e40 1 --2- 172.21.15.167:0/399129457 >> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] conn(0x561c5a4858d0 0x561c5a487cb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-05T19:28:26.524 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.514+0000 7fe1047a9e40 1 -- 172.21.15.167:0/399129457 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:6 5.0 5:00000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x561c5a4881f0 con 0x561c5a4858d0 2023-10-05T19:28:26.524 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.518+0000 7fe1022de700 1 --2- 172.21.15.167:0/399129457 >> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] conn(0x561c5a4858d0 0x561c5a487cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-05T19:28:26.524 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.518+0000 7fe1022de700 1 --2- 172.21.15.167:0/399129457 >> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] conn(0x561c5a4858d0 0x561c5a487cb0 crc :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-05T19:28:26.525 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.518+0000 7fe1022de700 1 -- 172.21.15.167:0/399129457 <== osd.3 v2:172.21.15.167:6808/30036 1 ==== osd_op_reply(6 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fe0f4014070 con 0x561c5a4858d0 2023-10-05T19:28:26.525 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.518+0000 7fe1022de700 1 --2- 172.21.15.167:0/399129457 >> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] conn(0x7fe0f402d020 0x7fe0f403d790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-05T19:28:26.525 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.518+0000 7fe1022de700 1 -- 172.21.15.167:0/399129457 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:7 5.10 5:08000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x561c5a4881f0 con 0x7fe0f402d020 2023-10-05T19:28:26.526 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.518+0000 7fe101add700 1 --2- 172.21.15.167:0/399129457 >> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] conn(0x7fe0f402d020 0x7fe0f403d790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-05T19:28:26.526 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.522+0000 7fe101add700 1 --2- 172.21.15.167:0/399129457 >> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] conn(0x7fe0f402d020 0x7fe0f403d790 crc :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-05T19:28:26.527 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.522+0000 7fe101add700 1 -- 172.21.15.167:0/399129457 <== osd.2 v2:172.21.15.167:6816/30037 1 ==== osd_op_reply(7 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fe0ec012450 con 0x7fe0f402d020 2023-10-05T19:28:26.527 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.522+0000 7fe101add700 1 --2- 172.21.15.167:0/399129457 >> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] conn(0x7fe0ec007030 0x7fe0ec01c920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-05T19:28:26.527 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.522+0000 7fe101add700 1 -- 172.21.15.167:0/399129457 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:8 5.8 5:10000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x561c5a4881f0 con 0x7fe0ec007030 2023-10-05T19:28:26.528 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.522+0000 7fe102adf700 1 --2- 172.21.15.167:0/399129457 >> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] conn(0x7fe0ec007030 0x7fe0ec01c920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-05T19:28:26.528 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.522+0000 7fe102adf700 1 --2- 172.21.15.167:0/399129457 >> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] conn(0x7fe0ec007030 0x7fe0ec01c920 crc :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-05T19:28:26.529 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.522+0000 7fe102adf700 1 -- 172.21.15.167:0/399129457 <== osd.0 v2:172.21.15.118:6800/27690 1 ==== osd_op_reply(8 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fe0fc00a5a0 con 0x7fe0ec007030 2023-10-05T19:28:26.529 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.522+0000 7fe102adf700 1 -- 172.21.15.167:0/399129457 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:9 5.18 5:18000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x561c5a4881f0 con 0x561c5a47fae0 2023-10-05T19:28:26.530 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.522+0000 7fe102adf700 1 -- 172.21.15.167:0/399129457 <== osd.1 v2:172.21.15.167:6800/30035 6 ==== osd_op_reply(9 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fe0fc00a5a0 con 0x561c5a47fae0 2023-10-05T19:28:26.530 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.522+0000 7fe102adf700 1 -- 172.21.15.167:0/399129457 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:10 5.4 5:20000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x561c5a4881f0 con 0x561c5a47fae0 2023-10-05T19:28:26.530 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.526+0000 7fe102adf700 1 -- 172.21.15.167:0/399129457 <== osd.1 v2:172.21.15.167:6800/30035 7 ==== osd_op_reply(10 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fe0fc00a5a0 con 0x561c5a47fae0 2023-10-05T19:28:26.530 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.526+0000 7fe102adf700 1 -- 172.21.15.167:0/399129457 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:11 5.14 5:28000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x561c5a4881f0 con 0x7fe0ec007030 2023-10-05T19:28:26.531 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.526+0000 7fe102adf700 1 -- 172.21.15.167:0/399129457 <== osd.0 v2:172.21.15.118:6800/27690 2 ==== osd_op_reply(11 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fe0fc00a5a0 con 0x7fe0ec007030 2023-10-05T19:28:26.531 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.526+0000 7fe102adf700 1 -- 172.21.15.167:0/399129457 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:12 5.c 5:30000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x561c5a4881f0 con 0x7fe0ec007030 2023-10-05T19:28:26.532 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.526+0000 7fe102adf700 1 -- 172.21.15.167:0/399129457 <== osd.0 v2:172.21.15.118:6800/27690 3 ==== osd_op_reply(12 [pgnls start_epoch 30 out=112b] v23'1 uv1 ondisk = 1) v8 ==== 144+0+112 (crc 0 0 0) 0x7fe0fc00a5a0 con 0x7fe0ec007030 2023-10-05T19:28:26.532 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.526+0000 7fe102adf700 1 -- 172.21.15.167:0/399129457 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:13 5.1c 5:38000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x561c5a4881f0 con 0x7fe0f402d020 2023-10-05T19:28:26.533 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.526+0000 7fe101add700 1 -- 172.21.15.167:0/399129457 <== osd.2 v2:172.21.15.167:6816/30037 2 ==== osd_op_reply(13 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fe0ec012450 con 0x7fe0f402d020 2023-10-05T19:28:26.533 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.526+0000 7fe101add700 1 -- 172.21.15.167:0/399129457 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:14 5.2 5:40000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x561c5a4881f0 con 0x561c5a4858d0 2023-10-05T19:28:26.533 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.526+0000 7fe1022de700 1 -- 172.21.15.167:0/399129457 <== osd.3 v2:172.21.15.167:6808/30036 2 ==== osd_op_reply(14 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fe0f4014070 con 0x561c5a4858d0 2023-10-05T19:28:26.534 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.526+0000 7fe1022de700 1 -- 172.21.15.167:0/399129457 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:15 5.12 5:48000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x561c5a4881f0 con 0x561c5a47fae0 2023-10-05T19:28:26.534 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.530+0000 7fe102adf700 1 -- 172.21.15.167:0/399129457 <== osd.1 v2:172.21.15.167:6800/30035 8 ==== osd_op_reply(15 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fe0fc00a5a0 con 0x561c5a47fae0 2023-10-05T19:28:26.534 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.530+0000 7fe102adf700 1 -- 172.21.15.167:0/399129457 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:16 5.a 5:50000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x561c5a4881f0 con 0x7fe0ec007030 2023-10-05T19:28:26.535 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.530+0000 7fe102adf700 1 -- 172.21.15.167:0/399129457 <== osd.0 v2:172.21.15.118:6800/27690 4 ==== osd_op_reply(16 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fe0fc00a5a0 con 0x7fe0ec007030 2023-10-05T19:28:26.535 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.530+0000 7fe102adf700 1 -- 172.21.15.167:0/399129457 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:17 5.1a 5:58000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x561c5a4881f0 con 0x561c5a47fae0 2023-10-05T19:28:26.536 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.530+0000 7fe102adf700 1 -- 172.21.15.167:0/399129457 <== osd.1 v2:172.21.15.167:6800/30035 9 ==== osd_op_reply(17 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fe0fc00a5a0 con 0x561c5a47fae0 2023-10-05T19:28:26.536 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.530+0000 7fe102adf700 1 -- 172.21.15.167:0/399129457 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:18 5.6 5:60000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x561c5a4881f0 con 0x7fe0f402d020 2023-10-05T19:28:26.536 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.530+0000 7fe101add700 1 -- 172.21.15.167:0/399129457 <== osd.2 v2:172.21.15.167:6816/30037 3 ==== osd_op_reply(18 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fe0ec012450 con 0x7fe0f402d020 2023-10-05T19:28:26.537 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.530+0000 7fe101add700 1 -- 172.21.15.167:0/399129457 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:19 5.16 5:68000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x561c5a4881f0 con 0x561c5a4858d0 2023-10-05T19:28:26.537 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.530+0000 7fe1022de700 1 -- 172.21.15.167:0/399129457 <== osd.3 v2:172.21.15.167:6808/30036 3 ==== osd_op_reply(19 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fe0f4014070 con 0x561c5a4858d0 2023-10-05T19:28:26.537 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.530+0000 7fe1022de700 1 -- 172.21.15.167:0/399129457 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:20 5.e 5:70000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x561c5a4881f0 con 0x561c5a4858d0 2023-10-05T19:28:26.538 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.534+0000 7fe1022de700 1 -- 172.21.15.167:0/399129457 <== osd.3 v2:172.21.15.167:6808/30036 4 ==== osd_op_reply(20 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fe0f4014070 con 0x561c5a4858d0 2023-10-05T19:28:26.538 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.534+0000 7fe1022de700 1 -- 172.21.15.167:0/399129457 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:21 5.1e 5:78000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x561c5a4881f0 con 0x7fe0ec007030 2023-10-05T19:28:26.539 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.534+0000 7fe102adf700 1 -- 172.21.15.167:0/399129457 <== osd.0 v2:172.21.15.118:6800/27690 5 ==== osd_op_reply(21 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fe0fc00a5a0 con 0x7fe0ec007030 2023-10-05T19:28:26.539 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.534+0000 7fe102adf700 1 -- 172.21.15.167:0/399129457 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:22 5.1 5:80000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x561c5a4881f0 con 0x561c5a4858d0 2023-10-05T19:28:26.540 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.534+0000 7fe1022de700 1 -- 172.21.15.167:0/399129457 <== osd.3 v2:172.21.15.167:6808/30036 5 ==== osd_op_reply(22 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fe0f4014070 con 0x561c5a4858d0 2023-10-05T19:28:26.540 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.534+0000 7fe1022de700 1 -- 172.21.15.167:0/399129457 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:23 5.11 5:88000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x561c5a4881f0 con 0x561c5a4858d0 2023-10-05T19:28:26.540 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.534+0000 7fe1022de700 1 -- 172.21.15.167:0/399129457 <== osd.3 v2:172.21.15.167:6808/30036 6 ==== osd_op_reply(23 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fe0f4014070 con 0x561c5a4858d0 2023-10-05T19:28:26.540 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.534+0000 7fe1022de700 1 -- 172.21.15.167:0/399129457 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:24 5.9 5:90000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x561c5a4881f0 con 0x561c5a47fae0 2023-10-05T19:28:26.541 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.534+0000 7fe102adf700 1 -- 172.21.15.167:0/399129457 <== osd.1 v2:172.21.15.167:6800/30035 10 ==== osd_op_reply(24 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fe0fc00a5a0 con 0x561c5a47fae0 2023-10-05T19:28:26.541 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.534+0000 7fe102adf700 1 -- 172.21.15.167:0/399129457 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:25 5.19 5:98000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x561c5a4881f0 con 0x561c5a47fae0 2023-10-05T19:28:26.542 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.538+0000 7fe102adf700 1 -- 172.21.15.167:0/399129457 <== osd.1 v2:172.21.15.167:6800/30035 11 ==== osd_op_reply(25 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fe0fc00a5a0 con 0x561c5a47fae0 2023-10-05T19:28:26.542 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.538+0000 7fe102adf700 1 -- 172.21.15.167:0/399129457 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:26 5.5 5:a0000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x561c5a4881f0 con 0x7fe0ec007030 2023-10-05T19:28:26.543 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.538+0000 7fe102adf700 1 -- 172.21.15.167:0/399129457 <== osd.0 v2:172.21.15.118:6800/27690 6 ==== osd_op_reply(26 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fe0fc00a5a0 con 0x7fe0ec007030 2023-10-05T19:28:26.543 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.538+0000 7fe102adf700 1 -- 172.21.15.167:0/399129457 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:27 5.15 5:a8000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x561c5a4881f0 con 0x561c5a4858d0 2023-10-05T19:28:26.544 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.538+0000 7fe1022de700 1 -- 172.21.15.167:0/399129457 <== osd.3 v2:172.21.15.167:6808/30036 7 ==== osd_op_reply(27 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fe0f4014070 con 0x561c5a4858d0 2023-10-05T19:28:26.544 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.538+0000 7fe1022de700 1 -- 172.21.15.167:0/399129457 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:28 5.d 5:b0000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x561c5a4881f0 con 0x7fe0f402d020 2023-10-05T19:28:26.544 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.538+0000 7fe101add700 1 -- 172.21.15.167:0/399129457 <== osd.2 v2:172.21.15.167:6816/30037 4 ==== osd_op_reply(28 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fe0ec012450 con 0x7fe0f402d020 2023-10-05T19:28:26.545 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.538+0000 7fe101add700 1 -- 172.21.15.167:0/399129457 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:29 5.1d 5:b8000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x561c5a4881f0 con 0x561c5a47fae0 2023-10-05T19:28:26.545 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.538+0000 7fe102adf700 1 -- 172.21.15.167:0/399129457 <== osd.1 v2:172.21.15.167:6800/30035 12 ==== osd_op_reply(29 [pgnls start_epoch 30 out=107b] v30'2 uv2 ondisk = 1) v8 ==== 144+0+107 (crc 0 0 0) 0x7fe0fc00a5a0 con 0x561c5a47fae0 2023-10-05T19:28:26.545 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.538+0000 7fe102adf700 1 -- 172.21.15.167:0/399129457 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:30 5.3 5:c0000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x561c5a4881f0 con 0x7fe0ec007030 2023-10-05T19:28:26.546 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.542+0000 7fe102adf700 1 -- 172.21.15.167:0/399129457 <== osd.0 v2:172.21.15.118:6800/27690 7 ==== osd_op_reply(30 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fe0fc00a5a0 con 0x7fe0ec007030 2023-10-05T19:28:26.546 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.542+0000 7fe102adf700 1 -- 172.21.15.167:0/399129457 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:31 5.13 5:c8000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x561c5a4881f0 con 0x561c5a4858d0 2023-10-05T19:28:26.547 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.542+0000 7fe1022de700 1 -- 172.21.15.167:0/399129457 <== osd.3 v2:172.21.15.167:6808/30036 8 ==== osd_op_reply(31 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fe0f4014070 con 0x561c5a4858d0 2023-10-05T19:28:26.547 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.542+0000 7fe1022de700 1 -- 172.21.15.167:0/399129457 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:32 5.b 5:d0000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x561c5a4881f0 con 0x7fe0ec007030 2023-10-05T19:28:26.548 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.542+0000 7fe102adf700 1 -- 172.21.15.167:0/399129457 <== osd.0 v2:172.21.15.118:6800/27690 8 ==== osd_op_reply(32 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fe0fc00a5a0 con 0x7fe0ec007030 2023-10-05T19:28:26.548 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.542+0000 7fe102adf700 1 -- 172.21.15.167:0/399129457 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:33 5.1b 5:d8000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x561c5a4881f0 con 0x7fe0f402d020 2023-10-05T19:28:26.549 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.542+0000 7fe101add700 1 -- 172.21.15.167:0/399129457 <== osd.2 v2:172.21.15.167:6816/30037 5 ==== osd_op_reply(33 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fe0ec012450 con 0x7fe0f402d020 2023-10-05T19:28:26.549 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.542+0000 7fe101add700 1 -- 172.21.15.167:0/399129457 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:34 5.7 5:e0000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x561c5a4881f0 con 0x561c5a4858d0 2023-10-05T19:28:26.550 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.546+0000 7fe1022de700 1 -- 172.21.15.167:0/399129457 <== osd.3 v2:172.21.15.167:6808/30036 9 ==== osd_op_reply(34 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fe0f4014070 con 0x561c5a4858d0 2023-10-05T19:28:26.550 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.546+0000 7fe1022de700 1 -- 172.21.15.167:0/399129457 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:35 5.17 5:e8000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x561c5a4881f0 con 0x7fe0ec007030 2023-10-05T19:28:26.551 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.546+0000 7fe102adf700 1 -- 172.21.15.167:0/399129457 <== osd.0 v2:172.21.15.118:6800/27690 9 ==== osd_op_reply(35 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fe0fc00a5a0 con 0x7fe0ec007030 2023-10-05T19:28:26.551 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.546+0000 7fe102adf700 1 -- 172.21.15.167:0/399129457 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:36 5.f 5:f0000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x561c5a4881f0 con 0x561c5a47fae0 2023-10-05T19:28:26.552 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.546+0000 7fe102adf700 1 -- 172.21.15.167:0/399129457 <== osd.1 v2:172.21.15.167:6800/30035 13 ==== osd_op_reply(36 [pgnls start_epoch 30 out=115b] v23'2 uv2 ondisk = 1) v8 ==== 144+0+115 (crc 0 0 0) 0x7fe0fc00a5a0 con 0x561c5a47fae0 2023-10-05T19:28:26.552 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.546+0000 7fe102adf700 1 -- 172.21.15.167:0/399129457 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:37 5.1f 5:f8000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x561c5a4881f0 con 0x7fe0f402d020 2023-10-05T19:28:26.553 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.546+0000 7fe101add700 1 -- 172.21.15.167:0/399129457 <== osd.2 v2:172.21.15.167:6816/30037 6 ==== osd_op_reply(37 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fe0ec012450 con 0x7fe0f402d020 2023-10-05T19:28:26.553 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.546+0000 7fe1047a9e40 20 RGWRados::pool_iterate: got zonegroup_info.4f1f719d-9648-417b-b10c-e5d939825419 2023-10-05T19:28:26.553 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.546+0000 7fe1047a9e40 20 RGWRados::pool_iterate: got zone_info.9a9df965-6612-45d4-a45f-6e3c889bacc2 2023-10-05T19:28:26.553 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.546+0000 7fe1047a9e40 20 RGWRados::pool_iterate: got zone_names.default 2023-10-05T19:28:26.553 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.546+0000 7fe1047a9e40 20 RGWRados::pool_iterate: got zonegroups_names.default 2023-10-05T19:28:26.553 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.546+0000 7fe1047a9e40 20 rados->read ofs=0 len=0 2023-10-05T19:28:26.553 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.546+0000 7fe1047a9e40 1 -- 172.21.15.167:0/399129457 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:38 5.12 5:49953fa1:::default.realm:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x561c5a4845d0 con 0x561c5a47fae0 2023-10-05T19:28:26.554 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.546+0000 7fe102adf700 1 -- 172.21.15.167:0/399129457 <== osd.1 v2:172.21.15.167:6800/30035 14 ==== osd_op_reply(38 default.realm [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 ==== 157+0+0 (crc 0 0 0) 0x7fe0fc00a5a0 con 0x561c5a47fae0 2023-10-05T19:28:26.554 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.546+0000 7fe1047a9e40 20 rados_obj.operate() r=-2 bl.length=0 2023-10-05T19:28:26.555 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.546+0000 7fe1047a9e40 20 rados->read ofs=0 len=0 2023-10-05T19:28:26.555 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.546+0000 7fe1047a9e40 1 -- 172.21.15.167:0/399129457 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:39 5.f 5:f4c53578:::zonegroups_names.default:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x561c5a4841d0 con 0x561c5a47fae0 2023-10-05T19:28:26.555 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.550+0000 7fe102adf700 1 -- 172.21.15.167:0/399129457 <== osd.1 v2:172.21.15.167:6800/30035 15 ==== osd_op_reply(39 zonegroups_names.default [read 0~46 out=46b] v0'0 uv2 ondisk = 0) v8 ==== 168+0+46 (crc 0 0 0) 0x7fe0fc00a5a0 con 0x561c5a47fae0 2023-10-05T19:28:26.555 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.550+0000 7fe1047a9e40 20 rados_obj.operate() r=0 bl.length=46 2023-10-05T19:28:26.555 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.550+0000 7fe1047a9e40 20 rados->read ofs=0 len=0 2023-10-05T19:28:26.555 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.550+0000 7fe1047a9e40 1 -- 172.21.15.167:0/399129457 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:40 5.c 5:30d299a5:::zonegroup_info.4f1f719d-9648-417b-b10c-e5d939825419:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x561c5a4841a0 con 0x7fe0ec007030 2023-10-05T19:28:26.556 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.550+0000 7fe102adf700 1 -- 172.21.15.167:0/399129457 <== osd.0 v2:172.21.15.118:6800/27690 10 ==== osd_op_reply(40 zonegroup_info.4f1f719d-9648-417b-b10c-e5d939825419 [read 0~362 out=362b] v0'0 uv1 ondisk = 0) v8 ==== 195+0+362 (crc 0 0 0) 0x7fe0fc00a5a0 con 0x7fe0ec007030 2023-10-05T19:28:26.556 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.550+0000 7fe1047a9e40 20 rados_obj.operate() r=0 bl.length=362 2023-10-05T19:28:26.556 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.550+0000 7fe1047a9e40 20 rados->read ofs=0 len=0 2023-10-05T19:28:26.556 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.550+0000 7fe1047a9e40 1 -- 172.21.15.167:0/399129457 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:41 5.13 5:cbbf6c60:::converted:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x561c5a484f60 con 0x561c5a4858d0 2023-10-05T19:28:26.557 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.550+0000 7fe1022de700 1 -- 172.21.15.167:0/399129457 <== osd.3 v2:172.21.15.167:6808/30036 10 ==== osd_op_reply(41 converted [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 ==== 153+0+0 (crc 0 0 0) 0x7fe0f4014070 con 0x561c5a4858d0 2023-10-05T19:28:26.557 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.550+0000 7fe1047a9e40 20 rados_obj.operate() r=-2 bl.length=0 2023-10-05T19:28:26.557 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.550+0000 7fe1047a9e40 20 rados->read ofs=0 len=0 2023-10-05T19:28:26.557 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.550+0000 7fe1047a9e40 1 -- 172.21.15.167:0/399129457 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:42 5.12 5:49953fa1:::default.realm:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x561c5a488630 con 0x561c5a47fae0 2023-10-05T19:28:26.557 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.550+0000 7fe102adf700 1 -- 172.21.15.167:0/399129457 <== osd.1 v2:172.21.15.167:6800/30035 16 ==== osd_op_reply(42 default.realm [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 ==== 157+0+0 (crc 0 0 0) 0x7fe0fc00a5a0 con 0x561c5a47fae0 2023-10-05T19:28:26.558 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.550+0000 7fe1047a9e40 20 rados_obj.operate() r=-2 bl.length=0 2023-10-05T19:28:26.558 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.550+0000 7fe1047a9e40 20 rados->read ofs=0 len=0 2023-10-05T19:28:26.558 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.550+0000 7fe1047a9e40 1 -- 172.21.15.167:0/399129457 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:43 5.f 5:f4c53578:::zonegroups_names.default:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x561c5a488630 con 0x561c5a47fae0 2023-10-05T19:28:26.558 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.554+0000 7fe102adf700 1 -- 172.21.15.167:0/399129457 <== osd.1 v2:172.21.15.167:6800/30035 17 ==== osd_op_reply(43 zonegroups_names.default [read 0~46 out=46b] v0'0 uv2 ondisk = 0) v8 ==== 168+0+46 (crc 0 0 0) 0x7fe0fc00a5a0 con 0x561c5a47fae0 2023-10-05T19:28:26.558 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.554+0000 7fe1047a9e40 20 rados_obj.operate() r=0 bl.length=46 2023-10-05T19:28:26.558 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.554+0000 7fe1047a9e40 1 -- 172.21.15.167:0/399129457 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:44 5.0 5:00000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x561c5a489640 con 0x561c5a4858d0 2023-10-05T19:28:26.561 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.554+0000 7fe1022de700 1 -- 172.21.15.167:0/399129457 <== osd.3 v2:172.21.15.167:6808/30036 11 ==== osd_op_reply(44 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fe0f4014070 con 0x561c5a4858d0 2023-10-05T19:28:26.561 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.554+0000 7fe1022de700 1 -- 172.21.15.167:0/399129457 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:45 5.10 5:08000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x561c5a489640 con 0x7fe0f402d020 2023-10-05T19:28:26.562 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.554+0000 7fe101add700 1 -- 172.21.15.167:0/399129457 <== osd.2 v2:172.21.15.167:6816/30037 7 ==== osd_op_reply(45 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fe0ec012450 con 0x7fe0f402d020 2023-10-05T19:28:26.562 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.554+0000 7fe101add700 1 -- 172.21.15.167:0/399129457 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:46 5.8 5:10000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x561c5a489640 con 0x7fe0ec007030 2023-10-05T19:28:26.563 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.558+0000 7fe102adf700 1 -- 172.21.15.167:0/399129457 <== osd.0 v2:172.21.15.118:6800/27690 11 ==== osd_op_reply(46 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fe0fc00a5a0 con 0x7fe0ec007030 2023-10-05T19:28:26.563 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.558+0000 7fe102adf700 1 -- 172.21.15.167:0/399129457 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:47 5.18 5:18000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x561c5a4845d0 con 0x561c5a47fae0 2023-10-05T19:28:26.563 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.558+0000 7fe102adf700 1 -- 172.21.15.167:0/399129457 <== osd.1 v2:172.21.15.167:6800/30035 18 ==== osd_op_reply(47 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fe0fc00a5a0 con 0x561c5a47fae0 2023-10-05T19:28:26.564 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.558+0000 7fe102adf700 1 -- 172.21.15.167:0/399129457 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:48 5.4 5:20000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x561c5a4845d0 con 0x561c5a47fae0 2023-10-05T19:28:26.564 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.558+0000 7fe102adf700 1 -- 172.21.15.167:0/399129457 <== osd.1 v2:172.21.15.167:6800/30035 19 ==== osd_op_reply(48 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fe0fc00a5a0 con 0x561c5a47fae0 2023-10-05T19:28:26.564 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.558+0000 7fe102adf700 1 -- 172.21.15.167:0/399129457 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:49 5.14 5:28000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x561c5a4845d0 con 0x7fe0ec007030 2023-10-05T19:28:26.565 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.558+0000 7fe102adf700 1 -- 172.21.15.167:0/399129457 <== osd.0 v2:172.21.15.118:6800/27690 12 ==== osd_op_reply(49 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fe0fc00a5a0 con 0x7fe0ec007030 2023-10-05T19:28:26.565 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.558+0000 7fe102adf700 1 -- 172.21.15.167:0/399129457 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:50 5.c 5:30000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x561c5a4845d0 con 0x7fe0ec007030 2023-10-05T19:28:26.566 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.562+0000 7fe102adf700 1 -- 172.21.15.167:0/399129457 <== osd.0 v2:172.21.15.118:6800/27690 13 ==== osd_op_reply(50 [pgnls start_epoch 30 out=112b] v23'1 uv1 ondisk = 1) v8 ==== 144+0+112 (crc 0 0 0) 0x7fe0fc00a5a0 con 0x7fe0ec007030 2023-10-05T19:28:26.566 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.562+0000 7fe102adf700 1 -- 172.21.15.167:0/399129457 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:51 5.1c 5:38000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x561c5a4845d0 con 0x7fe0f402d020 2023-10-05T19:28:26.567 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.562+0000 7fe101add700 1 -- 172.21.15.167:0/399129457 <== osd.2 v2:172.21.15.167:6816/30037 8 ==== osd_op_reply(51 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fe0ec012450 con 0x7fe0f402d020 2023-10-05T19:28:26.568 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.562+0000 7fe101add700 1 -- 172.21.15.167:0/399129457 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:52 5.2 5:40000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x561c5a4845d0 con 0x561c5a4858d0 2023-10-05T19:28:26.568 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.562+0000 7fe1022de700 1 -- 172.21.15.167:0/399129457 <== osd.3 v2:172.21.15.167:6808/30036 12 ==== osd_op_reply(52 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fe0f4014070 con 0x561c5a4858d0 2023-10-05T19:28:26.568 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.562+0000 7fe1022de700 1 -- 172.21.15.167:0/399129457 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:53 5.12 5:48000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x561c5a4845d0 con 0x561c5a47fae0 2023-10-05T19:28:26.569 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.562+0000 7fe102adf700 1 -- 172.21.15.167:0/399129457 <== osd.1 v2:172.21.15.167:6800/30035 20 ==== osd_op_reply(53 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fe0fc00a5a0 con 0x561c5a47fae0 2023-10-05T19:28:26.569 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.562+0000 7fe102adf700 1 -- 172.21.15.167:0/399129457 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:54 5.a 5:50000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x561c5a4845d0 con 0x7fe0ec007030 2023-10-05T19:28:26.570 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.566+0000 7fe102adf700 1 -- 172.21.15.167:0/399129457 <== osd.0 v2:172.21.15.118:6800/27690 14 ==== osd_op_reply(54 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fe0fc00a5a0 con 0x7fe0ec007030 2023-10-05T19:28:26.570 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.566+0000 7fe102adf700 1 -- 172.21.15.167:0/399129457 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:55 5.1a 5:58000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x561c5a4845d0 con 0x561c5a47fae0 2023-10-05T19:28:26.571 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.566+0000 7fe102adf700 1 -- 172.21.15.167:0/399129457 <== osd.1 v2:172.21.15.167:6800/30035 21 ==== osd_op_reply(55 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fe0fc00a5a0 con 0x561c5a47fae0 2023-10-05T19:28:26.571 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.566+0000 7fe102adf700 1 -- 172.21.15.167:0/399129457 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:56 5.6 5:60000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x561c5a4845d0 con 0x7fe0f402d020 2023-10-05T19:28:26.572 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.566+0000 7fe101add700 1 -- 172.21.15.167:0/399129457 <== osd.2 v2:172.21.15.167:6816/30037 9 ==== osd_op_reply(56 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fe0ec012450 con 0x7fe0f402d020 2023-10-05T19:28:26.572 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.566+0000 7fe101add700 1 -- 172.21.15.167:0/399129457 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:57 5.16 5:68000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x561c5a4845d0 con 0x561c5a4858d0 2023-10-05T19:28:26.573 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.566+0000 7fe1022de700 1 -- 172.21.15.167:0/399129457 <== osd.3 v2:172.21.15.167:6808/30036 13 ==== osd_op_reply(57 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fe0f4014070 con 0x561c5a4858d0 2023-10-05T19:28:26.573 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.566+0000 7fe1022de700 1 -- 172.21.15.167:0/399129457 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:58 5.e 5:70000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x561c5a4845d0 con 0x561c5a4858d0 2023-10-05T19:28:26.573 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.566+0000 7fe1022de700 1 -- 172.21.15.167:0/399129457 <== osd.3 v2:172.21.15.167:6808/30036 14 ==== osd_op_reply(58 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fe0f4014070 con 0x561c5a4858d0 2023-10-05T19:28:26.574 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.566+0000 7fe1022de700 1 -- 172.21.15.167:0/399129457 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:59 5.1e 5:78000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x561c5a4845d0 con 0x7fe0ec007030 2023-10-05T19:28:26.575 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.570+0000 7fe102adf700 1 -- 172.21.15.167:0/399129457 <== osd.0 v2:172.21.15.118:6800/27690 15 ==== osd_op_reply(59 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fe0fc03f040 con 0x7fe0ec007030 2023-10-05T19:28:26.575 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.570+0000 7fe102adf700 1 -- 172.21.15.167:0/399129457 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:60 5.1 5:80000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x561c5a4845d0 con 0x561c5a4858d0 2023-10-05T19:28:26.575 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.570+0000 7fe1022de700 1 -- 172.21.15.167:0/399129457 <== osd.3 v2:172.21.15.167:6808/30036 15 ==== osd_op_reply(60 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fe0f4014070 con 0x561c5a4858d0 2023-10-05T19:28:26.575 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.570+0000 7fe1022de700 1 -- 172.21.15.167:0/399129457 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:61 5.11 5:88000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x561c5a4845d0 con 0x561c5a4858d0 2023-10-05T19:28:26.576 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.570+0000 7fe1022de700 1 -- 172.21.15.167:0/399129457 <== osd.3 v2:172.21.15.167:6808/30036 16 ==== osd_op_reply(61 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fe0f4014070 con 0x561c5a4858d0 2023-10-05T19:28:26.576 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.570+0000 7fe1022de700 1 -- 172.21.15.167:0/399129457 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:62 5.9 5:90000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x561c5a4845d0 con 0x561c5a47fae0 2023-10-05T19:28:26.577 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.570+0000 7fe102adf700 1 -- 172.21.15.167:0/399129457 <== osd.1 v2:172.21.15.167:6800/30035 22 ==== osd_op_reply(62 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fe0fc03f040 con 0x561c5a47fae0 2023-10-05T19:28:26.577 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.570+0000 7fe102adf700 1 -- 172.21.15.167:0/399129457 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:63 5.19 5:98000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x561c5a4845d0 con 0x561c5a47fae0 2023-10-05T19:28:26.578 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.574+0000 7fe102adf700 1 -- 172.21.15.167:0/399129457 <== osd.1 v2:172.21.15.167:6800/30035 23 ==== osd_op_reply(63 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fe0fc03f040 con 0x561c5a47fae0 2023-10-05T19:28:26.578 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.574+0000 7fe102adf700 1 -- 172.21.15.167:0/399129457 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:64 5.5 5:a0000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x561c5a4845d0 con 0x7fe0ec007030 2023-10-05T19:28:26.579 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.574+0000 7fe102adf700 1 -- 172.21.15.167:0/399129457 <== osd.0 v2:172.21.15.118:6800/27690 16 ==== osd_op_reply(64 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fe0fc03f040 con 0x7fe0ec007030 2023-10-05T19:28:26.579 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.574+0000 7fe102adf700 1 -- 172.21.15.167:0/399129457 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:65 5.15 5:a8000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x561c5a4845d0 con 0x561c5a4858d0 2023-10-05T19:28:26.580 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.574+0000 7fe1022de700 1 -- 172.21.15.167:0/399129457 <== osd.3 v2:172.21.15.167:6808/30036 17 ==== osd_op_reply(65 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fe0f4014070 con 0x561c5a4858d0 2023-10-05T19:28:26.580 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.574+0000 7fe1022de700 1 -- 172.21.15.167:0/399129457 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:66 5.d 5:b0000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x561c5a4845d0 con 0x7fe0f402d020 2023-10-05T19:28:26.581 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.574+0000 7fe101add700 1 -- 172.21.15.167:0/399129457 <== osd.2 v2:172.21.15.167:6816/30037 10 ==== osd_op_reply(66 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fe0ec012450 con 0x7fe0f402d020 2023-10-05T19:28:26.581 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.574+0000 7fe101add700 1 -- 172.21.15.167:0/399129457 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:67 5.1d 5:b8000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x561c5a4845d0 con 0x561c5a47fae0 2023-10-05T19:28:26.581 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.574+0000 7fe102adf700 1 -- 172.21.15.167:0/399129457 <== osd.1 v2:172.21.15.167:6800/30035 24 ==== osd_op_reply(67 [pgnls start_epoch 30 out=107b] v30'2 uv2 ondisk = 1) v8 ==== 144+0+107 (crc 0 0 0) 0x7fe0fc03f040 con 0x561c5a47fae0 2023-10-05T19:28:26.582 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.574+0000 7fe102adf700 1 -- 172.21.15.167:0/399129457 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:68 5.3 5:c0000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x561c5a4845d0 con 0x7fe0ec007030 2023-10-05T19:28:26.582 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.578+0000 7fe102adf700 1 -- 172.21.15.167:0/399129457 <== osd.0 v2:172.21.15.118:6800/27690 17 ==== osd_op_reply(68 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fe0fc03f040 con 0x7fe0ec007030 2023-10-05T19:28:26.583 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.578+0000 7fe102adf700 1 -- 172.21.15.167:0/399129457 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:69 5.13 5:c8000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x561c5a4845d0 con 0x561c5a4858d0 2023-10-05T19:28:26.583 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.578+0000 7fe1022de700 1 -- 172.21.15.167:0/399129457 <== osd.3 v2:172.21.15.167:6808/30036 18 ==== osd_op_reply(69 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fe0f4014070 con 0x561c5a4858d0 2023-10-05T19:28:26.583 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.578+0000 7fe1022de700 1 -- 172.21.15.167:0/399129457 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:70 5.b 5:d0000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x561c5a4845d0 con 0x7fe0ec007030 2023-10-05T19:28:26.585 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.578+0000 7fe102adf700 1 -- 172.21.15.167:0/399129457 <== osd.0 v2:172.21.15.118:6800/27690 18 ==== osd_op_reply(70 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fe0fc03f040 con 0x7fe0ec007030 2023-10-05T19:28:26.585 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.578+0000 7fe102adf700 1 -- 172.21.15.167:0/399129457 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:71 5.1b 5:d8000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x561c5a4845d0 con 0x7fe0f402d020 2023-10-05T19:28:26.585 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.578+0000 7fe101add700 1 -- 172.21.15.167:0/399129457 <== osd.2 v2:172.21.15.167:6816/30037 11 ==== osd_op_reply(71 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fe0ec012450 con 0x7fe0f402d020 2023-10-05T19:28:26.585 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.578+0000 7fe101add700 1 -- 172.21.15.167:0/399129457 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:72 5.7 5:e0000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x561c5a4845d0 con 0x561c5a4858d0 2023-10-05T19:28:26.586 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.582+0000 7fe1022de700 1 -- 172.21.15.167:0/399129457 <== osd.3 v2:172.21.15.167:6808/30036 19 ==== osd_op_reply(72 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fe0f4014070 con 0x561c5a4858d0 2023-10-05T19:28:26.586 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.582+0000 7fe1022de700 1 -- 172.21.15.167:0/399129457 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:73 5.17 5:e8000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x561c5a4845d0 con 0x7fe0ec007030 2023-10-05T19:28:26.587 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.582+0000 7fe102adf700 1 -- 172.21.15.167:0/399129457 <== osd.0 v2:172.21.15.118:6800/27690 19 ==== osd_op_reply(73 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fe0fc03f040 con 0x7fe0ec007030 2023-10-05T19:28:26.587 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.582+0000 7fe102adf700 1 -- 172.21.15.167:0/399129457 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:74 5.f 5:f0000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x561c5a4845d0 con 0x561c5a47fae0 2023-10-05T19:28:26.588 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.582+0000 7fe102adf700 1 -- 172.21.15.167:0/399129457 <== osd.1 v2:172.21.15.167:6800/30035 25 ==== osd_op_reply(74 [pgnls start_epoch 30 out=115b] v23'2 uv2 ondisk = 1) v8 ==== 144+0+115 (crc 0 0 0) 0x7fe0fc03f040 con 0x561c5a47fae0 2023-10-05T19:28:26.588 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.582+0000 7fe102adf700 1 -- 172.21.15.167:0/399129457 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:75 5.1f 5:f8000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x561c5a4845d0 con 0x7fe0f402d020 2023-10-05T19:28:26.589 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.582+0000 7fe101add700 1 -- 172.21.15.167:0/399129457 <== osd.2 v2:172.21.15.167:6816/30037 12 ==== osd_op_reply(75 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fe0ec012450 con 0x7fe0f402d020 2023-10-05T19:28:26.589 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.582+0000 7fe1047a9e40 20 RGWRados::pool_iterate: got zonegroup_info.4f1f719d-9648-417b-b10c-e5d939825419 2023-10-05T19:28:26.590 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.582+0000 7fe1047a9e40 20 RGWRados::pool_iterate: got zone_info.9a9df965-6612-45d4-a45f-6e3c889bacc2 2023-10-05T19:28:26.590 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.582+0000 7fe1047a9e40 20 RGWRados::pool_iterate: got zone_names.default 2023-10-05T19:28:26.590 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.582+0000 7fe1047a9e40 20 RGWRados::pool_iterate: got zonegroups_names.default 2023-10-05T19:28:26.590 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.582+0000 7fe1047a9e40 20 rados->read ofs=0 len=0 2023-10-05T19:28:26.590 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.582+0000 7fe1047a9e40 1 -- 172.21.15.167:0/399129457 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:76 5.f 5:f43b6ece:::zone_names.default:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x561c5a488630 con 0x561c5a47fae0 2023-10-05T19:28:26.593 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.586+0000 7fe102adf700 1 -- 172.21.15.167:0/399129457 <== osd.1 v2:172.21.15.167:6800/30035 26 ==== osd_op_reply(76 zone_names.default [read 0~46 out=46b] v0'0 uv1 ondisk = 0) v8 ==== 162+0+46 (crc 0 0 0) 0x7fe0fc03f040 con 0x561c5a47fae0 2023-10-05T19:28:26.593 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.586+0000 7fe1047a9e40 20 rados_obj.operate() r=0 bl.length=46 2023-10-05T19:28:26.593 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.586+0000 7fe1047a9e40 20 rados->read ofs=0 len=0 2023-10-05T19:28:26.594 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.586+0000 7fe1047a9e40 1 -- 172.21.15.167:0/399129457 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:77 5.1d 5:b8374eb4:::zone_info.9a9df965-6612-45d4-a45f-6e3c889bacc2:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x561c5a4889e0 con 0x561c5a47fae0 2023-10-05T19:28:26.594 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.586+0000 7fe102adf700 1 -- 172.21.15.167:0/399129457 <== osd.1 v2:172.21.15.167:6800/30035 27 ==== osd_op_reply(77 zone_info.9a9df965-6612-45d4-a45f-6e3c889bacc2 [read 0~874 out=874b] v0'0 uv2 ondisk = 0) v8 ==== 190+0+874 (crc 0 0 0) 0x7fe0fc03f040 con 0x561c5a47fae0 2023-10-05T19:28:26.594 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.586+0000 7fe1047a9e40 20 rados_obj.operate() r=0 bl.length=874 2023-10-05T19:28:26.594 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.586+0000 7fe1047a9e40 20 rados->read ofs=0 len=0 2023-10-05T19:28:26.594 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.586+0000 7fe1047a9e40 1 -- 172.21.15.167:0/399129457 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:78 5.f 5:f4c53578:::zonegroups_names.default:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x561c5a488ed0 con 0x561c5a47fae0 2023-10-05T19:28:26.594 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.586+0000 7fe102adf700 1 -- 172.21.15.167:0/399129457 <== osd.1 v2:172.21.15.167:6800/30035 28 ==== osd_op_reply(78 zonegroups_names.default [read 0~46 out=46b] v0'0 uv2 ondisk = 0) v8 ==== 168+0+46 (crc 0 0 0) 0x7fe0fc03f040 con 0x561c5a47fae0 2023-10-05T19:28:26.594 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.586+0000 7fe1047a9e40 20 rados_obj.operate() r=0 bl.length=46 2023-10-05T19:28:26.595 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.586+0000 7fe1047a9e40 20 rados->read ofs=0 len=0 2023-10-05T19:28:26.595 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.586+0000 7fe1047a9e40 1 -- 172.21.15.167:0/399129457 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:79 5.c 5:30d299a5:::zonegroup_info.4f1f719d-9648-417b-b10c-e5d939825419:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x561c5a488ed0 con 0x7fe0ec007030 2023-10-05T19:28:26.595 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.586+0000 7fe102adf700 1 -- 172.21.15.167:0/399129457 <== osd.0 v2:172.21.15.118:6800/27690 20 ==== osd_op_reply(79 zonegroup_info.4f1f719d-9648-417b-b10c-e5d939825419 [read 0~362 out=362b] v0'0 uv1 ondisk = 0) v8 ==== 195+0+362 (crc 0 0 0) 0x7fe0fc052040 con 0x7fe0ec007030 2023-10-05T19:28:26.595 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.586+0000 7fe1047a9e40 20 rados_obj.operate() r=0 bl.length=362 2023-10-05T19:28:26.595 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.586+0000 7fe1047a9e40 20 rados->read ofs=0 len=0 2023-10-05T19:28:26.596 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.586+0000 7fe1047a9e40 1 -- 172.21.15.167:0/399129457 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:80 5.3 5:c0ad8cbd:::region_map:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x561c5a488ed0 con 0x7fe0ec007030 2023-10-05T19:28:26.598 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.586+0000 7fe102adf700 1 -- 172.21.15.167:0/399129457 <== osd.0 v2:172.21.15.118:6800/27690 21 ==== osd_op_reply(80 region_map [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 ==== 154+0+0 (crc 0 0 0) 0x7fe0fc052040 con 0x7fe0ec007030 2023-10-05T19:28:26.598 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.586+0000 7fe1047a9e40 20 rados_obj.operate() r=-2 bl.length=0 2023-10-05T19:28:26.598 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.586+0000 7fe1047a9e40 20 zone default found 2023-10-05T19:28:26.599 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.586+0000 7fe1047a9e40 4 Realm: () 2023-10-05T19:28:26.599 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.586+0000 7fe1047a9e40 4 ZoneGroup: default (4f1f719d-9648-417b-b10c-e5d939825419) 2023-10-05T19:28:26.599 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.586+0000 7fe1047a9e40 4 Zone: default (9a9df965-6612-45d4-a45f-6e3c889bacc2) 2023-10-05T19:28:26.599 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.586+0000 7fe1047a9e40 10 cannot find current period zonegroup using local zonegroup configuration 2023-10-05T19:28:26.599 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.586+0000 7fe1047a9e40 20 zonegroup default 2023-10-05T19:28:26.599 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.586+0000 7fe1047a9e40 20 rados->read ofs=0 len=0 2023-10-05T19:28:26.599 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.586+0000 7fe1047a9e40 1 -- 172.21.15.167:0/399129457 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:81 5.3 5:c52100b6:::period_config.default:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x561c5a488ed0 con 0x7fe0ec007030 2023-10-05T19:28:26.599 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.590+0000 7fe102adf700 1 -- 172.21.15.167:0/399129457 <== osd.0 v2:172.21.15.118:6800/27690 22 ==== osd_op_reply(81 period_config.default [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 ==== 165+0+0 (crc 0 0 0) 0x7fe0fc052040 con 0x7fe0ec007030 2023-10-05T19:28:26.599 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.590+0000 7fe1047a9e40 20 rados_obj.operate() r=-2 bl.length=0 2023-10-05T19:28:26.599 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.590+0000 7fe1047a9e40 20 rados->read ofs=0 len=0 2023-10-05T19:28:26.599 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.590+0000 7fe1047a9e40 1 -- 172.21.15.167:0/399129457 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:82 6.2 6:4347d321:::bucket.sync-source-hints.:head [call version.read in=11b,read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x561c5a48a850 con 0x561c5a4858d0 2023-10-05T19:28:26.600 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.590+0000 7fe1022de700 1 -- 172.21.15.167:0/399129457 <== osd.3 v2:172.21.15.167:6808/30036 20 ==== osd_op_reply(82 bucket.sync-source-hints. [call,read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 ==== 211+0+0 (crc 0 0 0) 0x7fe0f4014070 con 0x561c5a4858d0 2023-10-05T19:28:26.600 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.590+0000 7fe1047a9e40 20 rados_obj.operate() r=-2 bl.length=0 2023-10-05T19:28:26.600 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.590+0000 7fe1047a9e40 20 rados->read ofs=0 len=0 2023-10-05T19:28:26.600 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.590+0000 7fe1047a9e40 1 -- 172.21.15.167:0/399129457 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:83 6.b 6:d467b91b:::bucket.sync-target-hints.:head [call version.read in=11b,read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x561c5a48ac00 con 0x561c5a4858d0 2023-10-05T19:28:26.600 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.590+0000 7fe1022de700 1 -- 172.21.15.167:0/399129457 <== osd.3 v2:172.21.15.167:6808/30036 21 ==== osd_op_reply(83 bucket.sync-target-hints. [call,read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 ==== 211+0+0 (crc 0 0 0) 0x7fe0f4014070 con 0x561c5a4858d0 2023-10-05T19:28:26.600 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.590+0000 7fe1047a9e40 20 rados_obj.operate() r=-2 bl.length=0 2023-10-05T19:28:26.600 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.590+0000 7fe1047a9e40 20 started sync module instance, tier type = 2023-10-05T19:28:26.600 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.590+0000 7fe1047a9e40 20 started zone id=9a9df965-6612-45d4-a45f-6e3c889bacc2 (name=default) with tier type = 2023-10-05T19:28:26.600 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.590+0000 7fe1047a9e40 1 -- 172.21.15.167:0/399129457 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:84 7.1f 7:f95f44c2:::notify.0:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x561c5a489f10 con 0x7fe0f402d020 2023-10-05T19:28:26.601 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.594+0000 7fe101add700 1 -- 172.21.15.167:0/399129457 <== osd.2 v2:172.21.15.167:6816/30037 13 ==== osd_op_reply(84 notify.0 [create] v30'36 uv36 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7fe0ec012450 con 0x7fe0f402d020 2023-10-05T19:28:26.601 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.594+0000 7fe1047a9e40 1 -- 172.21.15.167:0/399129457 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:85 7.1f 7:f95f44c2:::notify.0:head [watch watch cookie 94679773780512] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x561c5a48aff0 con 0x7fe0f402d020 2023-10-05T19:28:26.601 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.594+0000 7fe1047a9e40 1 -- 172.21.15.167:0/399129457 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:86 7.0 7:05bf5b68:::notify.1:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x561c5a48ba10 con 0x561c5a47fae0 2023-10-05T19:28:26.602 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.598+0000 7fe102adf700 1 -- 172.21.15.167:0/399129457 <== osd.1 v2:172.21.15.167:6800/30035 29 ==== osd_op_reply(86 notify.1 [create] v30'36 uv36 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7fe0fc052040 con 0x561c5a47fae0 2023-10-05T19:28:26.602 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.598+0000 7fe1047a9e40 1 -- 172.21.15.167:0/399129457 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:87 7.0 7:05bf5b68:::notify.1:head [watch watch cookie 94679773787360] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x561c5a48c300 con 0x561c5a47fae0 2023-10-05T19:28:26.602 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.598+0000 7fe1047a9e40 1 -- 172.21.15.167:0/399129457 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:88 7.15 7:a93a5511:::notify.2:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x561c5a48cba0 con 0x561c5a4858d0 2023-10-05T19:28:26.602 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.598+0000 7fe101add700 1 -- 172.21.15.167:0/399129457 <== osd.2 v2:172.21.15.167:6816/30037 14 ==== osd_op_reply(85 notify.0 [watch watch cookie 94679773780512] v30'37 uv36 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7fe0ec012450 con 0x7fe0f402d020 2023-10-05T19:28:26.605 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.598+0000 7fe102adf700 1 -- 172.21.15.167:0/399129457 <== osd.1 v2:172.21.15.167:6800/30035 30 ==== osd_op_reply(87 notify.1 [watch watch cookie 94679773787360] v30'37 uv36 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7fe0fc052040 con 0x561c5a47fae0 2023-10-05T19:28:26.605 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.598+0000 7fe1022de700 1 -- 172.21.15.167:0/399129457 <== osd.3 v2:172.21.15.167:6808/30036 22 ==== osd_op_reply(88 notify.2 [create] v30'36 uv36 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7fe0f4014070 con 0x561c5a4858d0 2023-10-05T19:28:26.606 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.598+0000 7fe1047a9e40 1 -- 172.21.15.167:0/399129457 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:89 7.15 7:a93a5511:::notify.2:head [watch watch cookie 94679773789712] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x561c5a48a470 con 0x561c5a4858d0 2023-10-05T19:28:26.606 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.598+0000 7fe1047a9e40 1 -- 172.21.15.167:0/399129457 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:90 7.e 7:7759931f:::notify.3:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x561c5a48d440 con 0x7fe0f402d020 2023-10-05T19:28:26.608 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.602+0000 7fe1022de700 1 -- 172.21.15.167:0/399129457 <== osd.3 v2:172.21.15.167:6808/30036 23 ==== osd_op_reply(89 notify.2 [watch watch cookie 94679773789712] v30'37 uv36 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7fe0f4014070 con 0x561c5a4858d0 2023-10-05T19:28:26.608 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.602+0000 7fe101add700 1 -- 172.21.15.167:0/399129457 <== osd.2 v2:172.21.15.167:6816/30037 15 ==== osd_op_reply(90 notify.3 [create] v30'36 uv36 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7fe0ec012450 con 0x7fe0f402d020 2023-10-05T19:28:26.608 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.602+0000 7fe1047a9e40 1 -- 172.21.15.167:0/399129457 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:91 7.e 7:7759931f:::notify.3:head [watch watch cookie 94679773794128] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x561c5a48d7f0 con 0x7fe0f402d020 2023-10-05T19:28:26.608 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.602+0000 7fe1047a9e40 1 -- 172.21.15.167:0/399129457 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:92 7.d 7:b4812045:::notify.4:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x561c5a48e260 con 0x561c5a47fae0 2023-10-05T19:28:26.611 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.606+0000 7fe101add700 1 -- 172.21.15.167:0/399129457 <== osd.2 v2:172.21.15.167:6816/30037 16 ==== osd_op_reply(91 notify.3 [watch watch cookie 94679773794128] v30'37 uv36 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7fe0ec012450 con 0x7fe0f402d020 2023-10-05T19:28:26.611 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.606+0000 7fe102adf700 1 -- 172.21.15.167:0/399129457 <== osd.1 v2:172.21.15.167:6800/30035 31 ==== osd_op_reply(92 notify.4 [create] v30'36 uv36 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7fe0fc052040 con 0x561c5a47fae0 2023-10-05T19:28:26.611 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.606+0000 7fe1047a9e40 1 -- 172.21.15.167:0/399129457 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:93 7.d 7:b4812045:::notify.4:head [watch watch cookie 94679773797744] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x561c5a48e1c0 con 0x561c5a47fae0 2023-10-05T19:28:26.611 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.606+0000 7fe1047a9e40 1 -- 172.21.15.167:0/399129457 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:94 7.3 7:c609908c:::notify.5:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x561c5a48f040 con 0x7fe0ec007030 2023-10-05T19:28:26.613 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.606+0000 7fe102adf700 1 -- 172.21.15.167:0/399129457 <== osd.1 v2:172.21.15.167:6800/30035 32 ==== osd_op_reply(93 notify.4 [watch watch cookie 94679773797744] v30'37 uv36 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7fe0fc052040 con 0x561c5a47fae0 2023-10-05T19:28:26.614 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.606+0000 7fe102adf700 1 -- 172.21.15.167:0/399129457 <== osd.0 v2:172.21.15.118:6800/27690 23 ==== osd_op_reply(94 notify.5 [create] v30'36 uv36 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7fe0fc052040 con 0x7fe0ec007030 2023-10-05T19:28:26.614 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.606+0000 7fe1047a9e40 1 -- 172.21.15.167:0/399129457 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:95 7.3 7:c609908c:::notify.5:head [watch watch cookie 94679773799920] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x561c5a48ef30 con 0x7fe0ec007030 2023-10-05T19:28:26.614 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.606+0000 7fe1047a9e40 1 -- 172.21.15.167:0/399129457 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:96 7.14 7:2b04a3e9:::notify.6:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x561c5a48faa0 con 0x7fe0ec007030 2023-10-05T19:28:26.616 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.610+0000 7fe102adf700 1 -- 172.21.15.167:0/399129457 <== osd.0 v2:172.21.15.118:6800/27690 24 ==== osd_op_reply(95 notify.5 [watch watch cookie 94679773799920] v30'37 uv36 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7fe0fc052040 con 0x7fe0ec007030 2023-10-05T19:28:26.616 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.610+0000 7fe102adf700 1 -- 172.21.15.167:0/399129457 <== osd.0 v2:172.21.15.118:6800/27690 25 ==== osd_op_reply(96 notify.6 [create] v30'36 uv36 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7fe0fc052040 con 0x7fe0ec007030 2023-10-05T19:28:26.617 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.610+0000 7fe1047a9e40 1 -- 172.21.15.167:0/399129457 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:97 7.14 7:2b04a3e9:::notify.6:head [watch watch cookie 94679773801296] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x561c5a48faa0 con 0x7fe0ec007030 2023-10-05T19:28:26.617 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.610+0000 7fe1047a9e40 1 -- 172.21.15.167:0/399129457 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:98 7.9 7:93e5b521:::notify.7:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x561c5a490340 con 0x7fe0ec007030 2023-10-05T19:28:26.619 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.614+0000 7fe102adf700 1 -- 172.21.15.167:0/399129457 <== osd.0 v2:172.21.15.118:6800/27690 26 ==== osd_op_reply(97 notify.6 [watch watch cookie 94679773801296] v30'37 uv36 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7fe0fc052040 con 0x7fe0ec007030 2023-10-05T19:28:26.619 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.614+0000 7fe102adf700 1 -- 172.21.15.167:0/399129457 <== osd.0 v2:172.21.15.118:6800/27690 27 ==== osd_op_reply(98 notify.7 [create] v30'36 uv36 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7fe0fc052040 con 0x7fe0ec007030 2023-10-05T19:28:26.620 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.614+0000 7fe1047a9e40 1 -- 172.21.15.167:0/399129457 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:99 7.9 7:93e5b521:::notify.7:head [watch watch cookie 94679773804512] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x561c5a490160 con 0x7fe0ec007030 2023-10-05T19:28:26.620 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.614+0000 7fe1047a9e40 20 add_watcher() i=0 2023-10-05T19:28:26.620 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.614+0000 7fe1047a9e40 20 add_watcher() i=1 2023-10-05T19:28:26.620 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.614+0000 7fe1047a9e40 20 add_watcher() i=2 2023-10-05T19:28:26.620 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.614+0000 7fe1047a9e40 20 add_watcher() i=3 2023-10-05T19:28:26.620 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.614+0000 7fe1047a9e40 20 add_watcher() i=4 2023-10-05T19:28:26.621 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.614+0000 7fe1047a9e40 20 add_watcher() i=5 2023-10-05T19:28:26.621 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.614+0000 7fe1047a9e40 20 add_watcher() i=6 2023-10-05T19:28:26.622 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.614+0000 7fe102adf700 1 -- 172.21.15.167:0/399129457 <== osd.0 v2:172.21.15.118:6800/27690 28 ==== osd_op_reply(99 notify.7 [watch watch cookie 94679773804512] v30'37 uv36 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7fe0fc052040 con 0x7fe0ec007030 2023-10-05T19:28:26.622 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.614+0000 7fe1047a9e40 20 add_watcher() i=7 2023-10-05T19:28:26.622 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.614+0000 7fe1047a9e40 2 all 8 watchers are set, enabling cache 2023-10-05T19:28:26.622 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.614+0000 7fe1047a9e40 1 -- 172.21.15.167:0/399129457 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:100 6.4 6:22d26bf9:::data_loggenerations_metadata:head [call version.check_conds in=74b,call version.read in=11b,read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x561c5a493770 con 0x561c5a47fae0 2023-10-05T19:28:26.622 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.618+0000 7fe102adf700 1 -- 172.21.15.167:0/399129457 <== osd.1 v2:172.21.15.167:6800/30035 33 ==== osd_op_reply(100 data_loggenerations_metadata [call,call out=48b,read 0~28 out=28b] v0'0 uv1 ondisk = 0) v8 ==== 256+0+76 (crc 0 0 0) 0x7fe0fc052040 con 0x561c5a47fae0 2023-10-05T19:28:26.623 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.618+0000 7fe1047a9e40 1 -- 172.21.15.167:0/399129457 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:101 6.4 6:22d26bf9:::data_loggenerations_metadata:head [watch watch cookie 94679773805616] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x561c5a490ba0 con 0x561c5a47fae0 2023-10-05T19:28:26.625 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.618+0000 7fe102adf700 1 -- 172.21.15.167:0/399129457 <== osd.1 v2:172.21.15.167:6800/30035 34 ==== osd_op_reply(101 data_loggenerations_metadata [watch watch cookie 94679773805616] v30'58 uv1 ondisk = 0) v8 ==== 172+0+0 (crc 0 0 0) 0x7fe0fc052040 con 0x561c5a47fae0 2023-10-05T19:28:26.625 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.618+0000 7fe0d3fdf700 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2023-10-05T19:28:26.625 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.618+0000 7fe1047a9e40 20 check_secure_mon_conn(): auth registy supported: methods=[2] modes=[2,1] 2023-10-05T19:28:26.625 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.618+0000 7fe1047a9e40 20 check_secure_mon_conn(): mode 1 is insecure 2023-10-05T19:28:26.626 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.618+0000 7fe1047a9e40 1 Processor -- start 2023-10-05T19:28:26.626 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.618+0000 7fe1047a9e40 1 -- start start 2023-10-05T19:28:26.626 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.618+0000 7fe1047a9e40 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x561c5a49d9d0 0x561c5a49dda0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-05T19:28:26.626 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.618+0000 7fe1047a9e40 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x561c5a49e2e0 con 0x561c5a49d9d0 2023-10-05T19:28:26.626 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.622+0000 7fe101add700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x561c5a49d9d0 0x561c5a49dda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-05T19:28:26.626 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.622+0000 7fe101add700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x561c5a49d9d0 0x561c5a49dda0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.118:3300/0 says I am v2:172.21.15.167:57356/0 (socket says 172.21.15.167:57356) 2023-10-05T19:28:26.626 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.622+0000 7fe101add700 1 -- 172.21.15.167:0/3050246546 learned_addr learned my addr 172.21.15.167:0/3050246546 (peer_addr_for_me v2:172.21.15.167:0/0) 2023-10-05T19:28:26.627 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.622+0000 7fe101add700 1 -- 172.21.15.167:0/3050246546 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x561c5a49e420 con 0x561c5a49d9d0 2023-10-05T19:28:26.627 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.622+0000 7fe101add700 1 --2- 172.21.15.167:0/3050246546 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x561c5a49d9d0 0x561c5a49dda0 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x561c5a44dca0 tx=0x7fe0ec012000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ac29dce6b0268bea server_cookie=0 in_seq=0 out_seq=0 2023-10-05T19:28:26.627 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.622+0000 7fe0d37de700 1 -- 172.21.15.167:0/3050246546 <== mon.0 v2:172.21.15.118:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fe0ec02a540 con 0x561c5a49d9d0 2023-10-05T19:28:26.628 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.622+0000 7fe0d37de700 1 -- 172.21.15.167:0/3050246546 <== mon.0 v2:172.21.15.118:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe0ec02a6c0 con 0x561c5a49d9d0 2023-10-05T19:28:26.628 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.622+0000 7fe0d37de700 1 -- 172.21.15.167:0/3050246546 <== mon.0 v2:172.21.15.118:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fe0ec02a870 con 0x561c5a49d9d0 2023-10-05T19:28:26.629 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.622+0000 7fe1047a9e40 1 -- 172.21.15.167:0/3050246546 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x561c5a49d9d0 msgr2=0x561c5a49dda0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-05T19:28:26.629 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.622+0000 7fe1047a9e40 1 --2- 172.21.15.167:0/3050246546 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x561c5a49d9d0 0x561c5a49dda0 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x561c5a44dca0 tx=0x7fe0ec012000 comp rx=0 tx=0).stop 2023-10-05T19:28:26.629 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.622+0000 7fe1047a9e40 1 -- 172.21.15.167:0/3050246546 shutdown_connections 2023-10-05T19:28:26.629 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.622+0000 7fe1047a9e40 1 --2- 172.21.15.167:0/3050246546 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x561c5a49d9d0 0x561c5a49dda0 unknown :-1 s=CLOSED pgs=210 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-05T19:28:26.629 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.622+0000 7fe1047a9e40 1 -- 172.21.15.167:0/3050246546 >> 172.21.15.167:0/3050246546 conn(0x561c5a498d90 msgr2=0x561c5a496f40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-05T19:28:26.629 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.622+0000 7fe1047a9e40 1 -- 172.21.15.167:0/3050246546 shutdown_connections 2023-10-05T19:28:26.629 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.622+0000 7fe1047a9e40 1 -- 172.21.15.167:0/3050246546 wait complete. 2023-10-05T19:28:26.630 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.622+0000 7fe1047a9e40 1 Processor -- start 2023-10-05T19:28:26.630 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.622+0000 7fe1047a9e40 1 -- start start 2023-10-05T19:28:26.630 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.622+0000 7fe1047a9e40 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x561c5a49d9d0 0x561c5a4a3490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-05T19:28:26.630 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.622+0000 7fe1047a9e40 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x561c5a4a4490 con 0x561c5a49d9d0 2023-10-05T19:28:26.630 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.626+0000 7fe101add700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x561c5a49d9d0 0x561c5a4a3490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-05T19:28:26.631 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.626+0000 7fe101add700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x561c5a49d9d0 0x561c5a4a3490 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.118:3300/0 says I am v2:172.21.15.167:57366/0 (socket says 172.21.15.167:57366) 2023-10-05T19:28:26.631 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.626+0000 7fe101add700 1 -- 172.21.15.167:0/1783565513 learned_addr learned my addr 172.21.15.167:0/1783565513 (peer_addr_for_me v2:172.21.15.167:0/0) 2023-10-05T19:28:26.632 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.626+0000 7fe101add700 1 -- 172.21.15.167:0/1783565513 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x561c5a44dcd0 con 0x561c5a49d9d0 2023-10-05T19:28:26.632 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.626+0000 7fe101add700 1 --2- 172.21.15.167:0/1783565513 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x561c5a49d9d0 0x561c5a4a3490 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7fe0ec037040 tx=0x7fe0ec02f000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-05T19:28:26.632 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.626+0000 7fe0d1fdb700 1 -- 172.21.15.167:0/1783565513 <== mon.0 v2:172.21.15.118:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fe0ec03e950 con 0x561c5a49d9d0 2023-10-05T19:28:26.632 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.626+0000 7fe0d1fdb700 1 -- 172.21.15.167:0/1783565513 <== mon.0 v2:172.21.15.118:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe0ec03ead0 con 0x561c5a49d9d0 2023-10-05T19:28:26.632 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.626+0000 7fe1047a9e40 1 -- 172.21.15.167:0/1783565513 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x561c5a4a3a90 con 0x561c5a49d9d0 2023-10-05T19:28:26.632 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.626+0000 7fe0d1fdb700 1 -- 172.21.15.167:0/1783565513 <== mon.0 v2:172.21.15.118:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fe0ec03ec80 con 0x561c5a49d9d0 2023-10-05T19:28:26.632 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.626+0000 7fe1047a9e40 1 -- 172.21.15.167:0/1783565513 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x561c5a4a3ed0 con 0x561c5a49d9d0 2023-10-05T19:28:26.633 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.626+0000 7fe0d1fdb700 1 -- 172.21.15.167:0/1783565513 <== mon.0 v2:172.21.15.118:3300/0 4 ==== mgrmap(e 5) v1 ==== 40329+0+0 (secure 0 0 0) 0x7fe0ec046070 con 0x561c5a49d9d0 2023-10-05T19:28:26.634 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.626+0000 7fe0d1fdb700 1 --2- 172.21.15.167:0/1783565513 >> [v2:172.21.15.167:6824/29889,v1:172.21.15.167:6825/29889] conn(0x7fe048030c00 0x7fe0480330b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-05T19:28:26.634 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.626+0000 7fe0d1fdb700 1 -- 172.21.15.167:0/1783565513 <== mon.0 v2:172.21.15.118:3300/0 5 ==== osd_map(30..30 src has 1..30) v4 ==== 6057+0+0 (secure 0 0 0) 0x7fe0ec02f070 con 0x561c5a49d9d0 2023-10-05T19:28:26.634 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.626+0000 7fe102adf700 1 --2- 172.21.15.167:0/1783565513 >> [v2:172.21.15.167:6824/29889,v1:172.21.15.167:6825/29889] conn(0x7fe048030c00 0x7fe0480330b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-05T19:28:26.634 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.626+0000 7fe1047a9e40 5 note: GC not initialized 2023-10-05T19:28:26.634 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.630+0000 7fe102adf700 1 --2- 172.21.15.167:0/1783565513 >> [v2:172.21.15.167:6824/29889,v1:172.21.15.167:6825/29889] conn(0x7fe048030c00 0x7fe0480330b0 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x561c5a490370 tx=0x7fe0fc00aec0 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-05T19:28:26.636 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.630+0000 7fe042ffd700 20 reqs_thread_entry: start 2023-10-05T19:28:26.637 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.630+0000 7fe1047a9e40 20 init_complete bucket index max shards: 11 2023-10-05T19:28:26.637 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.630+0000 7fe040ff9700 20 reqs_thread_entry: start 2023-10-05T19:28:26.637 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.630+0000 7fe1047a9e40 10 rgw notify: Started notification manager with: 1 workers 2023-10-05T19:28:26.637 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.630+0000 7fe032ffd700 20 rgw notify: INFO: next queues processing will happen at: Thu Oct 5 19:28:57 2023 2023-10-05T19:28:26.638 INFO:teuthology.orchestra.run.smithi167.stderr: 2023-10-05T19:28:26.638 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.630+0000 7fe0327fc700 20 reqs_thread_entry: start 2023-10-05T19:28:26.638 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.630+0000 7fe1047a9e40 20 get_system_obj_state: rctx=0x7ffe02a1fd98 obj=default.rgw.meta:users.uid:foo state=0x561c5a3270b0 s->prefetch_data=0 2023-10-05T19:28:26.638 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.630+0000 7fe1047a9e40 1 -- 172.21.15.167:0/399129457 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:102 8.18 8:1e6ca5ed:users.uid::foo:head [call version.read in=11b,getxattrs,stat] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x561c5a50bc80 con 0x561c5a47fae0 2023-10-05T19:28:26.638 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.634+0000 7fe102adf700 1 -- 172.21.15.167:0/399129457 <== osd.1 v2:172.21.15.167:6800/30035 35 ==== osd_op_reply(102 foo [call out=48b,getxattrs out=75b,stat out=16b] v0'0 uv2 ondisk = 0) v8 ==== 231+0+139 (crc 0 0 0) 0x7fe0fc052040 con 0x561c5a47fae0 2023-10-05T19:28:26.639 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.634+0000 7fe1047a9e40 20 get_system_obj_state: s->obj_tag was set empty 2023-10-05T19:28:26.639 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.634+0000 7fe1047a9e40 20 Read xattr: user.rgw.idtag 2023-10-05T19:28:26.639 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.634+0000 7fe1047a9e40 20 rados->read ofs=0 len=0 2023-10-05T19:28:26.639 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.634+0000 7fe1047a9e40 1 -- 172.21.15.167:0/399129457 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:103 8.18 8:1e6ca5ed:users.uid::foo:head [call version.check_conds in=122b,call version.read in=11b,read 0~0,getxattrs] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x561c5a50bc80 con 0x561c5a47fae0 2023-10-05T19:28:26.639 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.634+0000 7fe102adf700 1 -- 172.21.15.167:0/399129457 <== osd.1 v2:172.21.15.167:6800/30035 36 ==== osd_op_reply(103 foo [call,call out=48b,read 0~357 out=357b,getxattrs out=75b] v0'0 uv2 ondisk = 0) v8 ==== 273+0+480 (crc 0 0 0) 0x7fe0fc052040 con 0x561c5a47fae0 2023-10-05T19:28:26.639 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.634+0000 7fe1047a9e40 20 rados_obj.operate() r=0 bl.length=357 2023-10-05T19:28:26.640 INFO:teuthology.orchestra.run.smithi167.stdout:{ 2023-10-05T19:28:26.640 INFO:teuthology.orchestra.run.smithi167.stdout: "user_id": "foo", 2023-10-05T19:28:26.640 INFO:teuthology.orchestra.run.smithi167.stdout: "display_name": "Foo", 2023-10-05T19:28:26.640 INFO:teuthology.orchestra.run.smithi167.stdout: "email": "foo@foo.com", 2023-10-05T19:28:26.640 INFO:teuthology.orchestra.run.smithi167.stdout: "suspended": 1, 2023-10-05T19:28:26.640 INFO:teuthology.orchestra.run.smithi167.stdout: "max_buckets": 4, 2023-10-05T19:28:26.640 INFO:teuthology.orchestra.run.smithi167.stdout: "subusers": [], 2023-10-05T19:28:26.640 INFO:teuthology.orchestra.run.smithi167.stdout: "keys": [ 2023-10-05T19:28:26.641 INFO:teuthology.orchestra.run.smithi167.stdout: { 2023-10-05T19:28:26.641 INFO:teuthology.orchestra.run.smithi167.stdout: "user": "foo", 2023-10-05T19:28:26.641 INFO:teuthology.orchestra.run.smithi167.stdout: "access_key": "9te6NH5mcdcq0Tc5i8i1", 2023-10-05T19:28:26.641 INFO:teuthology.orchestra.run.smithi167.stdout: "secret_key": "Ny4IOauQoL18Gp2zM7lC1vLmoawgqcYP/YGcWfXu" 2023-10-05T19:28:26.641 INFO:teuthology.orchestra.run.smithi167.stdout: } 2023-10-05T19:28:26.641 INFO:teuthology.orchestra.run.smithi167.stdout: ], 2023-10-05T19:28:26.641 INFO:teuthology.orchestra.run.smithi167.stdout: "swift_keys": [], 2023-10-05T19:28:26.641 INFO:teuthology.orchestra.run.smithi167.stdout: "caps": [], 2023-10-05T19:28:26.641 INFO:teuthology.orchestra.run.smithi167.stdout: "op_mask": "read, write, delete", 2023-10-05T19:28:26.641 INFO:teuthology.orchestra.run.smithi167.stdout: "default_placement": "", 2023-10-05T19:28:26.641 INFO:teuthology.orchestra.run.smithi167.stdout: "default_storage_class": "", 2023-10-05T19:28:26.641 INFO:teuthology.orchestra.run.smithi167.stdout: "placement_tags": [], 2023-10-05T19:28:26.641 INFO:teuthology.orchestra.run.smithi167.stdout: "bucket_quota": { 2023-10-05T19:28:26.642 INFO:teuthology.orchestra.run.smithi167.stdout: "enabled": false, 2023-10-05T19:28:26.642 INFO:teuthology.orchestra.run.smithi167.stdout: "check_on_raw": false, 2023-10-05T19:28:26.642 INFO:teuthology.orchestra.run.smithi167.stdout: "max_size": -1, 2023-10-05T19:28:26.642 INFO:teuthology.orchestra.run.smithi167.stdout: "max_size_kb": 0, 2023-10-05T19:28:26.642 INFO:teuthology.orchestra.run.smithi167.stdout: "max_objects": -1 2023-10-05T19:28:26.642 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2023-10-05T19:28:26.642 INFO:teuthology.orchestra.run.smithi167.stdout: "user_quota": { 2023-10-05T19:28:26.642 INFO:teuthology.orchestra.run.smithi167.stdout: "enabled": false, 2023-10-05T19:28:26.642 INFO:teuthology.orchestra.run.smithi167.stdout: "check_on_raw": false, 2023-10-05T19:28:26.642 INFO:teuthology.orchestra.run.smithi167.stdout: "max_size": -1, 2023-10-05T19:28:26.642 INFO:teuthology.orchestra.run.smithi167.stdout: "max_size_kb": 0, 2023-10-05T19:28:26.642 INFO:teuthology.orchestra.run.smithi167.stdout: "max_objects": -1 2023-10-05T19:28:26.643 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2023-10-05T19:28:26.643 INFO:teuthology.orchestra.run.smithi167.stdout: "temp_url_keys": [], 2023-10-05T19:28:26.643 INFO:teuthology.orchestra.run.smithi167.stdout: "type": "rgw", 2023-10-05T19:28:26.643 INFO:teuthology.orchestra.run.smithi167.stdout: "mfa_ids": [] 2023-10-05T19:28:26.643 INFO:teuthology.orchestra.run.smithi167.stdout:} 2023-10-05T19:28:26.643 INFO:teuthology.orchestra.run.smithi167.stdout: 2023-10-05T19:28:26.643 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.634+0000 7fe1047a9e40 1 -- 172.21.15.167:0/399129457 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:104 7.1f 7:f95f44c2:::notify.0:head [watch unwatch cookie 94679773780512] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x561c5a50bc80 con 0x7fe0f402d020 2023-10-05T19:28:26.644 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.638+0000 7fe101add700 1 -- 172.21.15.167:0/399129457 <== osd.2 v2:172.21.15.167:6816/30037 17 ==== osd_op_reply(104 notify.0 [watch unwatch cookie 94679773780512] v30'38 uv36 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7fe0ec032040 con 0x7fe0f402d020 2023-10-05T19:28:26.644 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.638+0000 7fe1047a9e40 20 remove_watcher() i=0 2023-10-05T19:28:26.644 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.638+0000 7fe1047a9e40 2 removed watcher, disabling cache 2023-10-05T19:28:26.644 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.638+0000 7fe1047a9e40 1 -- 172.21.15.167:0/399129457 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:105 7.0 7:05bf5b68:::notify.1:head [watch unwatch cookie 94679773787360] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x561c5a465a00 con 0x561c5a47fae0 2023-10-05T19:28:26.646 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.642+0000 7fe102adf700 1 -- 172.21.15.167:0/399129457 <== osd.1 v2:172.21.15.167:6800/30035 37 ==== osd_op_reply(105 notify.1 [watch unwatch cookie 94679773787360] v30'38 uv36 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7fe0fc052040 con 0x561c5a47fae0 2023-10-05T19:28:26.646 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.642+0000 7fe1047a9e40 20 remove_watcher() i=1 2023-10-05T19:28:26.646 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.642+0000 7fe1047a9e40 1 -- 172.21.15.167:0/399129457 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:106 7.15 7:a93a5511:::notify.2:head [watch unwatch cookie 94679773789712] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x561c5a465a00 con 0x561c5a4858d0 2023-10-05T19:28:26.649 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.642+0000 7fe1022de700 1 -- 172.21.15.167:0/399129457 <== osd.3 v2:172.21.15.167:6808/30036 24 ==== osd_op_reply(106 notify.2 [watch unwatch cookie 94679773789712] v30'38 uv36 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7fe0f4014070 con 0x561c5a4858d0 2023-10-05T19:28:26.649 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.642+0000 7fe1047a9e40 20 remove_watcher() i=2 2023-10-05T19:28:26.649 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.642+0000 7fe1047a9e40 1 -- 172.21.15.167:0/399129457 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:107 7.e 7:7759931f:::notify.3:head [watch unwatch cookie 94679773794128] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x561c5a48b4e0 con 0x7fe0f402d020 2023-10-05T19:28:26.651 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.646+0000 7fe101add700 1 -- 172.21.15.167:0/399129457 <== osd.2 v2:172.21.15.167:6816/30037 18 ==== osd_op_reply(107 notify.3 [watch unwatch cookie 94679773794128] v30'38 uv36 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7fe0ec032040 con 0x7fe0f402d020 2023-10-05T19:28:26.652 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.646+0000 7fe1047a9e40 20 remove_watcher() i=3 2023-10-05T19:28:26.652 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.646+0000 7fe1047a9e40 1 -- 172.21.15.167:0/399129457 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:108 7.d 7:b4812045:::notify.4:head [watch unwatch cookie 94679773797744] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x561c5a48be10 con 0x561c5a47fae0 2023-10-05T19:28:26.654 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.650+0000 7fe102adf700 1 -- 172.21.15.167:0/399129457 <== osd.1 v2:172.21.15.167:6800/30035 38 ==== osd_op_reply(108 notify.4 [watch unwatch cookie 94679773797744] v30'38 uv36 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7fe0fc052040 con 0x561c5a47fae0 2023-10-05T19:28:26.654 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.650+0000 7fe1047a9e40 20 remove_watcher() i=4 2023-10-05T19:28:26.654 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.650+0000 7fe1047a9e40 1 -- 172.21.15.167:0/399129457 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:109 7.3 7:c609908c:::notify.5:head [watch unwatch cookie 94679773799920] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x561c5a48cf50 con 0x7fe0ec007030 2023-10-05T19:28:26.657 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.650+0000 7fe102adf700 1 -- 172.21.15.167:0/399129457 <== osd.0 v2:172.21.15.118:6800/27690 29 ==== osd_op_reply(109 notify.5 [watch unwatch cookie 94679773799920] v30'38 uv36 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7fe0fc052040 con 0x7fe0ec007030 2023-10-05T19:28:26.657 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.650+0000 7fe1047a9e40 20 remove_watcher() i=5 2023-10-05T19:28:26.657 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.650+0000 7fe1047a9e40 1 -- 172.21.15.167:0/399129457 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:110 7.14 7:2b04a3e9:::notify.6:head [watch unwatch cookie 94679773801296] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x561c5a48b4e0 con 0x7fe0ec007030 2023-10-05T19:28:26.659 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.654+0000 7fe102adf700 1 -- 172.21.15.167:0/399129457 <== osd.0 v2:172.21.15.118:6800/27690 30 ==== osd_op_reply(110 notify.6 [watch unwatch cookie 94679773801296] v30'38 uv36 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7fe0fc052040 con 0x7fe0ec007030 2023-10-05T19:28:26.660 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.654+0000 7fe1047a9e40 20 remove_watcher() i=6 2023-10-05T19:28:26.660 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.654+0000 7fe1047a9e40 1 -- 172.21.15.167:0/399129457 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:111 7.9 7:93e5b521:::notify.7:head [watch unwatch cookie 94679773804512] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x561c5a48dd70 con 0x7fe0ec007030 2023-10-05T19:28:26.662 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.658+0000 7fe102adf700 1 -- 172.21.15.167:0/399129457 <== osd.0 v2:172.21.15.118:6800/27690 31 ==== osd_op_reply(111 notify.7 [watch unwatch cookie 94679773804512] v30'38 uv36 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7fe0fc052040 con 0x7fe0ec007030 2023-10-05T19:28:26.662 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.658+0000 7fe1047a9e40 20 remove_watcher() i=7 2023-10-05T19:28:26.664 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.658+0000 7fe0d3fdf700 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2023-10-05T19:28:26.664 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.658+0000 7fe1047a9e40 1 -- 172.21.15.167:0/399129457 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:112 6.4 6:22d26bf9:::data_loggenerations_metadata:head [watch unwatch cookie 94679773805616] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x561c5a48dd70 con 0x561c5a47fae0 2023-10-05T19:28:26.667 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.662+0000 7fe102adf700 1 -- 172.21.15.167:0/399129457 <== osd.1 v2:172.21.15.167:6800/30035 39 ==== osd_op_reply(112 data_loggenerations_metadata [watch unwatch cookie 94679773805616] v30'59 uv1 ondisk = 0) v8 ==== 172+0+0 (crc 0 0 0) 0x7fe0fc052040 con 0x561c5a47fae0 2023-10-05T19:28:26.667 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.662+0000 7fe1047a9e40 1 -- 172.21.15.167:0/399129457 >> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] conn(0x7fe0ec007030 msgr2=0x7fe0ec01c920 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-05T19:28:26.667 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.662+0000 7fe1047a9e40 1 --2- 172.21.15.167:0/399129457 >> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] conn(0x7fe0ec007030 0x7fe0ec01c920 crc :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-05T19:28:26.667 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.662+0000 7fe1047a9e40 1 -- 172.21.15.167:0/399129457 >> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] conn(0x561c5a47fae0 msgr2=0x561c5a481ec0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-05T19:28:26.667 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.662+0000 7fe1047a9e40 1 --2- 172.21.15.167:0/399129457 >> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] conn(0x561c5a47fae0 0x561c5a481ec0 crc :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-05T19:28:26.667 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.662+0000 7fe1047a9e40 1 -- 172.21.15.167:0/399129457 >> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] conn(0x7fe0f402d020 msgr2=0x7fe0f403d790 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-05T19:28:26.667 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.662+0000 7fe1047a9e40 1 --2- 172.21.15.167:0/399129457 >> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] conn(0x7fe0f402d020 0x7fe0f403d790 crc :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-05T19:28:26.668 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.662+0000 7fe1047a9e40 1 -- 172.21.15.167:0/399129457 >> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] conn(0x561c5a4858d0 msgr2=0x561c5a487cb0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-05T19:28:26.668 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.662+0000 7fe1047a9e40 1 --2- 172.21.15.167:0/399129457 >> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] conn(0x561c5a4858d0 0x561c5a487cb0 crc :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-05T19:28:26.668 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.662+0000 7fe1047a9e40 1 -- 172.21.15.167:0/399129457 >> [v2:172.21.15.167:6824/29889,v1:172.21.15.167:6825/29889] conn(0x7fe0e4030bb0 msgr2=0x7fe0e4033060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-05T19:28:26.668 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.662+0000 7fe1047a9e40 1 --2- 172.21.15.167:0/399129457 >> [v2:172.21.15.167:6824/29889,v1:172.21.15.167:6825/29889] conn(0x7fe0e4030bb0 0x7fe0e4033060 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7fe0ec005a90 tx=0x7fe0ec015000 comp rx=0 tx=0).stop 2023-10-05T19:28:26.668 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.662+0000 7fe1047a9e40 1 -- 172.21.15.167:0/399129457 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x561c5a44e7e0 msgr2=0x561c5a476060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-05T19:28:26.668 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.662+0000 7fe102adf700 1 -- 172.21.15.167:0/399129457 reap_dead start 2023-10-05T19:28:26.668 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.662+0000 7fe1047a9e40 1 --2- 172.21.15.167:0/399129457 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x561c5a44e7e0 0x561c5a476060 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x561c5a46b800 tx=0x7fe0f4016000 comp rx=0 tx=0).stop 2023-10-05T19:28:26.669 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.662+0000 7fe1047a9e40 1 -- 172.21.15.167:0/399129457 shutdown_connections 2023-10-05T19:28:26.669 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.662+0000 7fe1047a9e40 1 --2- 172.21.15.167:0/399129457 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x561c5a44e7e0 0x561c5a476060 unknown :-1 s=CLOSED pgs=209 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-05T19:28:26.669 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.662+0000 7fe1047a9e40 1 -- 172.21.15.167:0/399129457 >> 172.21.15.167:0/399129457 conn(0x561c5a448720 msgr2=0x561c5a469760 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-05T19:28:26.670 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.662+0000 7fe1047a9e40 1 -- 172.21.15.167:0/399129457 shutdown_connections 2023-10-05T19:28:26.670 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.662+0000 7fe1047a9e40 1 -- 172.21.15.167:0/399129457 wait complete. 2023-10-05T19:28:26.670 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.662+0000 7fe1047a9e40 1 -- 172.21.15.167:0/1783565513 >> [v2:172.21.15.167:6824/29889,v1:172.21.15.167:6825/29889] conn(0x7fe048030c00 msgr2=0x7fe0480330b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-05T19:28:26.670 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.662+0000 7fe1047a9e40 1 --2- 172.21.15.167:0/1783565513 >> [v2:172.21.15.167:6824/29889,v1:172.21.15.167:6825/29889] conn(0x7fe048030c00 0x7fe0480330b0 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x561c5a490370 tx=0x7fe0fc00aec0 comp rx=0 tx=0).stop 2023-10-05T19:28:26.670 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.662+0000 7fe1047a9e40 1 -- 172.21.15.167:0/1783565513 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x561c5a49d9d0 msgr2=0x561c5a4a3490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-05T19:28:26.670 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.662+0000 7fe1047a9e40 1 --2- 172.21.15.167:0/1783565513 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x561c5a49d9d0 0x561c5a4a3490 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7fe0ec037040 tx=0x7fe0ec02f000 comp rx=0 tx=0).stop 2023-10-05T19:28:26.670 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.662+0000 7fe1047a9e40 1 -- 172.21.15.167:0/1783565513 shutdown_connections 2023-10-05T19:28:26.670 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.662+0000 7fe1047a9e40 1 --2- 172.21.15.167:0/1783565513 >> [v2:172.21.15.167:6824/29889,v1:172.21.15.167:6825/29889] conn(0x7fe048030c00 0x7fe0480330b0 unknown :-1 s=CLOSED pgs=77 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-05T19:28:26.670 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.662+0000 7fe1047a9e40 1 --2- 172.21.15.167:0/1783565513 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x561c5a49d9d0 0x561c5a4a3490 unknown :-1 s=CLOSED pgs=211 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-05T19:28:26.670 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.662+0000 7fe1047a9e40 1 -- 172.21.15.167:0/1783565513 >> 172.21.15.167:0/1783565513 conn(0x561c5a498d90 msgr2=0x561c5a496e90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-05T19:28:26.670 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.662+0000 7fe1047a9e40 1 -- 172.21.15.167:0/1783565513 shutdown_connections 2023-10-05T19:28:26.671 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.662+0000 7fe1047a9e40 1 -- 172.21.15.167:0/1783565513 wait complete. 2023-10-05T19:28:26.674 DEBUG:tasks.util.rgw: json result: {'user_id': 'foo', 'display_name': 'Foo', 'email': 'foo@foo.com', 'suspended': 1, 'max_buckets': 4, 'subusers': [], 'keys': [{'user': 'foo', 'access_key': '9te6NH5mcdcq0Tc5i8i1', 'secret_key': 'Ny4IOauQoL18Gp2zM7lC1vLmoawgqcYP/YGcWfXu'}], 'swift_keys': [], 'caps': [], 'op_mask': 'read, write, delete', 'default_placement': '', 'default_storage_class': '', 'placement_tags': [], 'bucket_quota': {'enabled': False, 'check_on_raw': False, 'max_size': -1, 'max_size_kb': 0, 'max_objects': -1}, 'user_quota': {'enabled': False, 'check_on_raw': False, 'max_size': -1, 'max_size_kb': 0, 'max_objects': -1}, 'temp_url_keys': [], 'type': 'rgw', 'mfa_ids': []} 2023-10-05T19:28:26.675 INFO:tasks.util.rgw:rgwadmin: client.0 : ['user', 'enable', '--uid', 'foo'] 2023-10-05T19:28:26.675 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'ceph', 'user', 'enable', '--uid', 'foo'] 2023-10-05T19:28:26.675 DEBUG:teuthology.orchestra.run.smithi167:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster ceph user enable --uid foo 2023-10-05T19:28:26.713 INFO:teuthology.orchestra.run.smithi167.stderr:ignoring --setuser ceph since I am not root 2023-10-05T19:28:26.713 INFO:teuthology.orchestra.run.smithi167.stderr:ignoring --setgroup ceph since I am not root 2023-10-05T19:28:26.714 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.706+0000 7f6a11c38e40 1 Processor -- start 2023-10-05T19:28:26.714 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.710+0000 7f6a11c38e40 1 -- start start 2023-10-05T19:28:26.714 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.710+0000 7f6a11c38e40 1 --2- >> v2:172.21.15.118:3300/0 conn(0x55875ee8b410 0x55875ee8d7f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-05T19:28:26.714 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.710+0000 7f6a11c38e40 1 -- --> v1:172.21.15.118:6789/0 -- auth(proto 0 26 bytes epoch 0) v1 -- 0x55875edd3bd0 con 0x55875ee86e30 2023-10-05T19:28:26.714 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.710+0000 7f6a11c38e40 1 -- --> v2:172.21.15.118:3300/0 -- mon_getmap magic: 0 v1 -- 0x55875ed90ee0 con 0x55875ee8b410 2023-10-05T19:28:26.715 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.710+0000 7f6a0ef6c700 1 --2- >> v2:172.21.15.118:3300/0 conn(0x55875ee8b410 0x55875ee8d7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-05T19:28:26.715 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.710+0000 7f6a0f76d700 1 --1- >> v1:172.21.15.118:6789/0 conn(0x55875ee86e30 0x55875ee87200 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.118:6789/0 says I am v1:172.21.15.167:56402/0 (socket says 172.21.15.167:56402) 2023-10-05T19:28:26.716 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.710+0000 7f6a0f76d700 1 -- 172.21.15.167:0/2553961154 learned_addr learned my addr 172.21.15.167:0/2553961154 (peer_addr_for_me v1:172.21.15.167:0/0) 2023-10-05T19:28:26.716 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.710+0000 7f6a0ef6c700 1 --2- >> v2:172.21.15.118:3300/0 conn(0x55875ee8b410 0x55875ee8d7f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.118:3300/0 says I am v2:172.21.15.167:57382/0 (socket says 172.21.15.167:57382) 2023-10-05T19:28:26.716 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.710+0000 7f6a0e76b700 1 -- 172.21.15.167:0/2553961154 <== mon.0 v1:172.21.15.118:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2061353655 0 0) 0x55875edd3bd0 con 0x55875ee86e30 2023-10-05T19:28:26.716 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.710+0000 7f6a0e76b700 1 -- 172.21.15.167:0/2553961154 --> v1:172.21.15.118:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f69fc001e10 con 0x55875ee86e30 2023-10-05T19:28:26.716 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.710+0000 7f6a0ef6c700 1 -- 172.21.15.167:0/2553961154 >> v1:172.21.15.118:6789/0 conn(0x55875ee86e30 legacy=0x55875ee87200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-05T19:28:26.716 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.710+0000 7f6a0ef6c700 1 -- 172.21.15.167:0/2553961154 --> v2:172.21.15.118:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55875ee87a40 con 0x55875ee8b410 2023-10-05T19:28:26.717 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.710+0000 7f6a0ef6c700 1 --2- 172.21.15.167:0/2553961154 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55875ee8b410 0x55875ee8d7f0 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7f69f800a5d0 tx=0x7f69f8005a50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=44d86efca75c97b4 server_cookie=0 in_seq=0 out_seq=0 2023-10-05T19:28:26.717 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.710+0000 7f6a0e76b700 1 -- 172.21.15.167:0/2553961154 <== mon.0 v2:172.21.15.118:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f69f8007610 con 0x55875ee8b410 2023-10-05T19:28:26.717 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.710+0000 7f6a0e76b700 1 -- 172.21.15.167:0/2553961154 <== mon.0 v2:172.21.15.118:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f69f8007770 con 0x55875ee8b410 2023-10-05T19:28:26.717 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.710+0000 7f6a0e76b700 1 -- 172.21.15.167:0/2553961154 <== mon.0 v2:172.21.15.118:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f69f8007a40 con 0x55875ee8b410 2023-10-05T19:28:26.717 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.710+0000 7f6a11c38e40 1 -- 172.21.15.167:0/2553961154 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55875ee8b410 msgr2=0x55875ee8d7f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-05T19:28:26.718 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.710+0000 7f6a11c38e40 1 --2- 172.21.15.167:0/2553961154 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55875ee8b410 0x55875ee8d7f0 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7f69f800a5d0 tx=0x7f69f8005a50 comp rx=0 tx=0).stop 2023-10-05T19:28:26.718 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.710+0000 7f6a11c38e40 1 -- 172.21.15.167:0/2553961154 shutdown_connections 2023-10-05T19:28:26.718 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.710+0000 7f6a11c38e40 1 --2- 172.21.15.167:0/2553961154 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55875ee8b410 0x55875ee8d7f0 unknown :-1 s=CLOSED pgs=213 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-05T19:28:26.718 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.710+0000 7f6a11c38e40 1 -- 172.21.15.167:0/2553961154 >> 172.21.15.167:0/2553961154 conn(0x55875ee82850 msgr2=0x55875ee84c60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-05T19:28:26.718 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.710+0000 7f6a11c38e40 1 -- 172.21.15.167:0/2553961154 shutdown_connections 2023-10-05T19:28:26.718 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.710+0000 7f6a11c38e40 1 -- 172.21.15.167:0/2553961154 wait complete. 2023-10-05T19:28:26.719 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.714+0000 7f6a11c38e40 1 Processor -- start 2023-10-05T19:28:26.719 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.714+0000 7f6a11c38e40 1 -- start start 2023-10-05T19:28:26.720 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.714+0000 7f6a11c38e40 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55875ee86e30 0x55875eea5d60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-05T19:28:26.720 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.714+0000 7f6a11c38e40 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x55875ed54e30 con 0x55875ee86e30 2023-10-05T19:28:26.720 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.714+0000 7f6a0f76d700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55875ee86e30 0x55875eea5d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-05T19:28:26.720 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.714+0000 7f6a0f76d700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55875ee86e30 0x55875eea5d60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.118:3300/0 says I am v2:172.21.15.167:57388/0 (socket says 172.21.15.167:57388) 2023-10-05T19:28:26.720 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.714+0000 7f6a0f76d700 1 -- 172.21.15.167:0/1318847170 learned_addr learned my addr 172.21.15.167:0/1318847170 (peer_addr_for_me v2:172.21.15.167:0/0) 2023-10-05T19:28:26.720 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.714+0000 7f6a0f76d700 1 -- 172.21.15.167:0/1318847170 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f69f8005cf0 con 0x55875ee86e30 2023-10-05T19:28:26.720 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.714+0000 7f6a0f76d700 1 --2- 172.21.15.167:0/1318847170 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55875ee86e30 0x55875eea5d60 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7f6a00006e70 tx=0x7f6a00006a20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=67c3241959e2ece7 server_cookie=0 in_seq=0 out_seq=0 2023-10-05T19:28:26.720 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.714+0000 7f6a0d769700 1 -- 172.21.15.167:0/1318847170 <== mon.0 v2:172.21.15.118:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f6a00006110 con 0x55875ee86e30 2023-10-05T19:28:26.720 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.714+0000 7f6a0d769700 1 -- 172.21.15.167:0/1318847170 <== mon.0 v2:172.21.15.118:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6a00006290 con 0x55875ee86e30 2023-10-05T19:28:26.720 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.714+0000 7f6a0d769700 1 -- 172.21.15.167:0/1318847170 <== mon.0 v2:172.21.15.118:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f6a00004520 con 0x55875ee86e30 2023-10-05T19:28:26.722 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.714+0000 7f6a11c38e40 1 -- 172.21.15.167:0/1318847170 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55875ee86e30 msgr2=0x55875eea5d60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-05T19:28:26.722 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.714+0000 7f6a11c38e40 1 --2- 172.21.15.167:0/1318847170 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55875ee86e30 0x55875eea5d60 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7f6a00006e70 tx=0x7f6a00006a20 comp rx=0 tx=0).stop 2023-10-05T19:28:26.722 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.714+0000 7f6a11c38e40 1 -- 172.21.15.167:0/1318847170 shutdown_connections 2023-10-05T19:28:26.722 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.714+0000 7f6a11c38e40 1 --2- 172.21.15.167:0/1318847170 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55875ee86e30 0x55875eea5d60 unknown :-1 s=CLOSED pgs=214 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-05T19:28:26.722 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.714+0000 7f6a11c38e40 1 -- 172.21.15.167:0/1318847170 >> 172.21.15.167:0/1318847170 conn(0x55875ee82850 msgr2=0x55875eea1920 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-05T19:28:26.722 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.714+0000 7f6a11c38e40 1 -- 172.21.15.167:0/1318847170 shutdown_connections 2023-10-05T19:28:26.722 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.714+0000 7f6a11c38e40 1 -- 172.21.15.167:0/1318847170 wait complete. 2023-10-05T19:28:26.723 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.714+0000 7f6a11c38e40 1 Processor -- start 2023-10-05T19:28:26.723 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.714+0000 7f6a11c38e40 1 -- start start 2023-10-05T19:28:26.723 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.714+0000 7f6a11c38e40 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55875ee86e30 0x55875eeacb40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-05T19:28:26.723 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.714+0000 7f6a11c38e40 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x55875ed27ed0 con 0x55875ee86e30 2023-10-05T19:28:26.723 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.714+0000 7f6a0f76d700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55875ee86e30 0x55875eeacb40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-05T19:28:26.723 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.714+0000 7f6a0f76d700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55875ee86e30 0x55875eeacb40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.118:3300/0 says I am v2:172.21.15.167:57390/0 (socket says 172.21.15.167:57390) 2023-10-05T19:28:26.723 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.714+0000 7f6a0f76d700 1 -- 172.21.15.167:0/1870492997 learned_addr learned my addr 172.21.15.167:0/1870492997 (peer_addr_for_me v2:172.21.15.167:0/0) 2023-10-05T19:28:26.723 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.718+0000 7f6a0f76d700 1 -- 172.21.15.167:0/1870492997 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f69fc001e10 con 0x55875ee86e30 2023-10-05T19:28:26.723 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.718+0000 7f6a0f76d700 1 --2- 172.21.15.167:0/1870492997 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55875ee86e30 0x55875eeacb40 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7f6a00001350 tx=0x7f6a00014000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-05T19:28:26.724 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.718+0000 7f6a077fe700 1 -- 172.21.15.167:0/1870492997 <== mon.0 v2:172.21.15.118:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f6a00006110 con 0x55875ee86e30 2023-10-05T19:28:26.724 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.718+0000 7f6a077fe700 1 -- 172.21.15.167:0/1870492997 <== mon.0 v2:172.21.15.118:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6a00006400 con 0x55875ee86e30 2023-10-05T19:28:26.725 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.718+0000 7f6a11c38e40 1 -- 172.21.15.167:0/1870492997 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x55875eead080 con 0x55875ee86e30 2023-10-05T19:28:26.725 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.718+0000 7f6a077fe700 1 -- 172.21.15.167:0/1870492997 <== mon.0 v2:172.21.15.118:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f6a00013970 con 0x55875ee86e30 2023-10-05T19:28:26.725 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.718+0000 7f6a11c38e40 1 -- 172.21.15.167:0/1870492997 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55875eead4c0 con 0x55875ee86e30 2023-10-05T19:28:26.725 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.718+0000 7f6a11c38e40 20 rados->read ofs=0 len=0 2023-10-05T19:28:26.725 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.718+0000 7f6a077fe700 1 -- 172.21.15.167:0/1870492997 <== mon.0 v2:172.21.15.118:3300/0 4 ==== mgrmap(e 5) v1 ==== 40329+0+0 (secure 0 0 0) 0x7f6a0001a070 con 0x55875ee86e30 2023-10-05T19:28:26.725 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.718+0000 7f6a077fe700 1 --2- 172.21.15.167:0/1870492997 >> [v2:172.21.15.167:6824/29889,v1:172.21.15.167:6825/29889] conn(0x7f69f0030c00 0x7f69f00330b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-05T19:28:26.725 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.718+0000 7f6a077fe700 1 -- 172.21.15.167:0/1870492997 <== mon.0 v2:172.21.15.118:3300/0 5 ==== osd_map(30..30 src has 1..30) v4 ==== 6057+0+0 (secure 0 0 0) 0x7f6a00044e50 con 0x55875ee86e30 2023-10-05T19:28:26.726 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.718+0000 7f6a0ef6c700 1 --2- 172.21.15.167:0/1870492997 >> [v2:172.21.15.167:6824/29889,v1:172.21.15.167:6825/29889] conn(0x7f69f0030c00 0x7f69f00330b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-05T19:28:26.726 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.718+0000 7f6a11c38e40 1 --2- 172.21.15.167:0/1870492997 >> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] conn(0x55875ee8b410 0x55875eeb8600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-05T19:28:26.726 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.718+0000 7f6a11c38e40 1 -- 172.21.15.167:0/1870492997 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:1 5.12 5:49953fa1:::default.realm:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x55875eeb8d00 con 0x55875ee8b410 2023-10-05T19:28:26.726 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.718+0000 7f6a0ff6e700 1 --2- 172.21.15.167:0/1870492997 >> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] conn(0x55875ee8b410 0x55875eeb8600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-05T19:28:26.726 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.718+0000 7f6a0ef6c700 1 --2- 172.21.15.167:0/1870492997 >> [v2:172.21.15.167:6824/29889,v1:172.21.15.167:6825/29889] conn(0x7f69f0030c00 0x7f69f00330b0 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f69f800afd0 tx=0x7f69f8014000 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-05T19:28:26.726 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.722+0000 7f6a0ff6e700 1 --2- 172.21.15.167:0/1870492997 >> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] conn(0x55875ee8b410 0x55875eeb8600 crc :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-05T19:28:26.728 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.722+0000 7f6a0ff6e700 1 -- 172.21.15.167:0/1870492997 <== osd.1 v2:172.21.15.167:6800/30035 1 ==== osd_op_reply(1 default.realm [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 ==== 157+0+0 (crc 0 0 0) 0x7f6a0800a500 con 0x55875ee8b410 2023-10-05T19:28:26.728 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.722+0000 7f6a11c38e40 20 rados_obj.operate() r=-2 bl.length=0 2023-10-05T19:28:26.728 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.722+0000 7f6a11c38e40 20 realm 2023-10-05T19:28:26.728 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.722+0000 7f6a11c38e40 20 rados->read ofs=0 len=0 2023-10-05T19:28:26.728 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.722+0000 7f6a11c38e40 1 -- 172.21.15.167:0/1870492997 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:2 5.12 5:49953fa1:::default.realm:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x55875eeb5c60 con 0x55875ee8b410 2023-10-05T19:28:26.728 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.722+0000 7f6a0ff6e700 1 -- 172.21.15.167:0/1870492997 <== osd.1 v2:172.21.15.167:6800/30035 2 ==== osd_op_reply(2 default.realm [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 ==== 157+0+0 (crc 0 0 0) 0x7f6a0800a500 con 0x55875ee8b410 2023-10-05T19:28:26.728 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.722+0000 7f6a11c38e40 20 rados_obj.operate() r=-2 bl.length=0 2023-10-05T19:28:26.728 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.722+0000 7f6a11c38e40 4 RGWPeriod::init failed to init realm id : (2) No such file or directory 2023-10-05T19:28:26.729 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.722+0000 7f6a11c38e40 20 rados->read ofs=0 len=0 2023-10-05T19:28:26.729 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.722+0000 7f6a11c38e40 1 -- 172.21.15.167:0/1870492997 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:3 5.12 5:49953fa1:::default.realm:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x55875eeb97e0 con 0x55875ee8b410 2023-10-05T19:28:26.729 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.722+0000 7f6a0ff6e700 1 -- 172.21.15.167:0/1870492997 <== osd.1 v2:172.21.15.167:6800/30035 3 ==== osd_op_reply(3 default.realm [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 ==== 157+0+0 (crc 0 0 0) 0x7f6a0800a500 con 0x55875ee8b410 2023-10-05T19:28:26.729 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.722+0000 7f6a11c38e40 20 rados_obj.operate() r=-2 bl.length=0 2023-10-05T19:28:26.729 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.722+0000 7f6a11c38e40 20 rados->read ofs=0 len=0 2023-10-05T19:28:26.729 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.722+0000 7f6a11c38e40 1 -- 172.21.15.167:0/1870492997 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:4 5.f 5:f43b6ece:::zone_names.default:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x55875eeb93f0 con 0x55875ee8b410 2023-10-05T19:28:26.729 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.722+0000 7f6a0ff6e700 1 -- 172.21.15.167:0/1870492997 <== osd.1 v2:172.21.15.167:6800/30035 4 ==== osd_op_reply(4 zone_names.default [read 0~46 out=46b] v0'0 uv1 ondisk = 0) v8 ==== 162+0+46 (crc 0 0 0) 0x7f6a0800a500 con 0x55875ee8b410 2023-10-05T19:28:26.729 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.722+0000 7f6a11c38e40 20 rados_obj.operate() r=0 bl.length=46 2023-10-05T19:28:26.729 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.722+0000 7f6a11c38e40 20 rados->read ofs=0 len=0 2023-10-05T19:28:26.729 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.722+0000 7f6a11c38e40 1 -- 172.21.15.167:0/1870492997 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:5 5.1d 5:b8374eb4:::zone_info.9a9df965-6612-45d4-a45f-6e3c889bacc2:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x55875eeba670 con 0x55875ee8b410 2023-10-05T19:28:26.730 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.722+0000 7f6a0ff6e700 1 -- 172.21.15.167:0/1870492997 <== osd.1 v2:172.21.15.167:6800/30035 5 ==== osd_op_reply(5 zone_info.9a9df965-6612-45d4-a45f-6e3c889bacc2 [read 0~874 out=874b] v0'0 uv2 ondisk = 0) v8 ==== 190+0+874 (crc 0 0 0) 0x7f6a0800a500 con 0x55875ee8b410 2023-10-05T19:28:26.730 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.722+0000 7f6a11c38e40 20 rados_obj.operate() r=0 bl.length=874 2023-10-05T19:28:26.730 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.722+0000 7f6a11c38e40 20 searching for the correct realm 2023-10-05T19:28:26.731 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.722+0000 7f6a11c38e40 1 --2- 172.21.15.167:0/1870492997 >> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] conn(0x55875eebc430 0x55875eebe810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-05T19:28:26.731 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.722+0000 7f6a11c38e40 1 -- 172.21.15.167:0/1870492997 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:6 5.0 5:00000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x55875eebed50 con 0x55875eebc430 2023-10-05T19:28:26.731 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.722+0000 7f6a0f76d700 1 --2- 172.21.15.167:0/1870492997 >> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] conn(0x55875eebc430 0x55875eebe810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-05T19:28:26.731 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.722+0000 7f6a0f76d700 1 --2- 172.21.15.167:0/1870492997 >> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] conn(0x55875eebc430 0x55875eebe810 crc :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-05T19:28:26.731 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.726+0000 7f6a0f76d700 1 -- 172.21.15.167:0/1870492997 <== osd.3 v2:172.21.15.167:6808/30036 1 ==== osd_op_reply(6 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f6a00025020 con 0x55875eebc430 2023-10-05T19:28:26.732 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.726+0000 7f6a0f76d700 1 --2- 172.21.15.167:0/1870492997 >> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] conn(0x7f6a0001fc20 0x7f6a0002a000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-05T19:28:26.732 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.726+0000 7f6a0f76d700 1 -- 172.21.15.167:0/1870492997 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:7 5.10 5:08000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x55875eebed50 con 0x7f6a0001fc20 2023-10-05T19:28:26.732 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.726+0000 7f6a0ef6c700 1 --2- 172.21.15.167:0/1870492997 >> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] conn(0x7f6a0001fc20 0x7f6a0002a000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-05T19:28:26.732 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.726+0000 7f6a0ef6c700 1 --2- 172.21.15.167:0/1870492997 >> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] conn(0x7f6a0001fc20 0x7f6a0002a000 crc :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-05T19:28:26.733 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.726+0000 7f6a0ef6c700 1 -- 172.21.15.167:0/1870492997 <== osd.2 v2:172.21.15.167:6816/30037 1 ==== osd_op_reply(7 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f69f8007da0 con 0x7f6a0001fc20 2023-10-05T19:28:26.733 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.726+0000 7f6a0ef6c700 1 --2- 172.21.15.167:0/1870492997 >> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] conn(0x7f69f8007030 0x7f69f801d430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-05T19:28:26.733 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.726+0000 7f6a0ef6c700 1 -- 172.21.15.167:0/1870492997 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:8 5.8 5:10000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x55875eebed50 con 0x7f69f8007030 2023-10-05T19:28:26.733 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.726+0000 7f6a0ff6e700 1 --2- 172.21.15.167:0/1870492997 >> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] conn(0x7f69f8007030 0x7f69f801d430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-05T19:28:26.734 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.730+0000 7f6a0ff6e700 1 --2- 172.21.15.167:0/1870492997 >> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] conn(0x7f69f8007030 0x7f69f801d430 crc :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-05T19:28:26.735 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.730+0000 7f6a0ff6e700 1 -- 172.21.15.167:0/1870492997 <== osd.0 v2:172.21.15.118:6800/27690 1 ==== osd_op_reply(8 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f6a08014040 con 0x7f69f8007030 2023-10-05T19:28:26.735 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.730+0000 7f6a0ff6e700 1 -- 172.21.15.167:0/1870492997 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:9 5.18 5:18000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x55875eebed50 con 0x55875ee8b410 2023-10-05T19:28:26.735 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.730+0000 7f6a0ff6e700 1 -- 172.21.15.167:0/1870492997 <== osd.1 v2:172.21.15.167:6800/30035 6 ==== osd_op_reply(9 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f6a08014040 con 0x55875ee8b410 2023-10-05T19:28:26.735 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.730+0000 7f6a0ff6e700 1 -- 172.21.15.167:0/1870492997 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:10 5.4 5:20000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x55875eebed50 con 0x55875ee8b410 2023-10-05T19:28:26.736 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.730+0000 7f6a0ff6e700 1 -- 172.21.15.167:0/1870492997 <== osd.1 v2:172.21.15.167:6800/30035 7 ==== osd_op_reply(10 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f6a08014040 con 0x55875ee8b410 2023-10-05T19:28:26.736 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.730+0000 7f6a0ff6e700 1 -- 172.21.15.167:0/1870492997 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:11 5.14 5:28000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x55875eebed50 con 0x7f69f8007030 2023-10-05T19:28:26.737 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.730+0000 7f6a0ff6e700 1 -- 172.21.15.167:0/1870492997 <== osd.0 v2:172.21.15.118:6800/27690 2 ==== osd_op_reply(11 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f6a08014040 con 0x7f69f8007030 2023-10-05T19:28:26.737 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.730+0000 7f6a0ff6e700 1 -- 172.21.15.167:0/1870492997 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:12 5.c 5:30000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x55875eebed50 con 0x7f69f8007030 2023-10-05T19:28:26.738 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.730+0000 7f6a0ff6e700 1 -- 172.21.15.167:0/1870492997 <== osd.0 v2:172.21.15.118:6800/27690 3 ==== osd_op_reply(12 [pgnls start_epoch 30 out=112b] v23'1 uv1 ondisk = 1) v8 ==== 144+0+112 (crc 0 0 0) 0x7f6a08014040 con 0x7f69f8007030 2023-10-05T19:28:26.738 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.730+0000 7f6a0ff6e700 1 -- 172.21.15.167:0/1870492997 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:13 5.1c 5:38000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x55875eebed50 con 0x7f6a0001fc20 2023-10-05T19:28:26.738 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.734+0000 7f6a0ef6c700 1 -- 172.21.15.167:0/1870492997 <== osd.2 v2:172.21.15.167:6816/30037 2 ==== osd_op_reply(13 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f69f8029040 con 0x7f6a0001fc20 2023-10-05T19:28:26.738 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.734+0000 7f6a0ef6c700 1 -- 172.21.15.167:0/1870492997 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:14 5.2 5:40000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x55875eebed50 con 0x55875eebc430 2023-10-05T19:28:26.739 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.734+0000 7f6a0f76d700 1 -- 172.21.15.167:0/1870492997 <== osd.3 v2:172.21.15.167:6808/30036 2 ==== osd_op_reply(14 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f6a00025020 con 0x55875eebc430 2023-10-05T19:28:26.739 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.734+0000 7f6a0f76d700 1 -- 172.21.15.167:0/1870492997 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:15 5.12 5:48000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x55875eebed50 con 0x55875ee8b410 2023-10-05T19:28:26.739 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.734+0000 7f6a0ff6e700 1 -- 172.21.15.167:0/1870492997 <== osd.1 v2:172.21.15.167:6800/30035 8 ==== osd_op_reply(15 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f6a08014040 con 0x55875ee8b410 2023-10-05T19:28:26.740 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.734+0000 7f6a0ff6e700 1 -- 172.21.15.167:0/1870492997 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:16 5.a 5:50000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x55875eebed50 con 0x7f69f8007030 2023-10-05T19:28:26.740 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.734+0000 7f6a0ff6e700 1 -- 172.21.15.167:0/1870492997 <== osd.0 v2:172.21.15.118:6800/27690 4 ==== osd_op_reply(16 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f6a08022040 con 0x7f69f8007030 2023-10-05T19:28:26.741 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.734+0000 7f6a0ff6e700 1 -- 172.21.15.167:0/1870492997 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:17 5.1a 5:58000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x55875eebed50 con 0x55875ee8b410 2023-10-05T19:28:26.741 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.734+0000 7f6a0ff6e700 1 -- 172.21.15.167:0/1870492997 <== osd.1 v2:172.21.15.167:6800/30035 9 ==== osd_op_reply(17 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f6a08022040 con 0x55875ee8b410 2023-10-05T19:28:26.741 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.734+0000 7f6a0ff6e700 1 -- 172.21.15.167:0/1870492997 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:18 5.6 5:60000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x55875eebed50 con 0x7f6a0001fc20 2023-10-05T19:28:26.742 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.734+0000 7f6a0ef6c700 1 -- 172.21.15.167:0/1870492997 <== osd.2 v2:172.21.15.167:6816/30037 3 ==== osd_op_reply(18 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f69f8029040 con 0x7f6a0001fc20 2023-10-05T19:28:26.742 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.734+0000 7f6a0ef6c700 1 -- 172.21.15.167:0/1870492997 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:19 5.16 5:68000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x55875eebed50 con 0x55875eebc430 2023-10-05T19:28:26.742 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.738+0000 7f6a0f76d700 1 -- 172.21.15.167:0/1870492997 <== osd.3 v2:172.21.15.167:6808/30036 3 ==== osd_op_reply(19 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f6a00025020 con 0x55875eebc430 2023-10-05T19:28:26.742 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.738+0000 7f6a0f76d700 1 -- 172.21.15.167:0/1870492997 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:20 5.e 5:70000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x55875eebed50 con 0x55875eebc430 2023-10-05T19:28:26.743 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.738+0000 7f6a0f76d700 1 -- 172.21.15.167:0/1870492997 <== osd.3 v2:172.21.15.167:6808/30036 4 ==== osd_op_reply(20 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f6a00025020 con 0x55875eebc430 2023-10-05T19:28:26.743 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.738+0000 7f6a0f76d700 1 -- 172.21.15.167:0/1870492997 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:21 5.1e 5:78000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x55875eebed50 con 0x7f69f8007030 2023-10-05T19:28:26.744 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.738+0000 7f6a0ff6e700 1 -- 172.21.15.167:0/1870492997 <== osd.0 v2:172.21.15.118:6800/27690 5 ==== osd_op_reply(21 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f6a08022040 con 0x7f69f8007030 2023-10-05T19:28:26.744 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.738+0000 7f6a0ff6e700 1 -- 172.21.15.167:0/1870492997 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:22 5.1 5:80000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x55875eebed50 con 0x55875eebc430 2023-10-05T19:28:26.744 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.738+0000 7f6a0f76d700 1 -- 172.21.15.167:0/1870492997 <== osd.3 v2:172.21.15.167:6808/30036 5 ==== osd_op_reply(22 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f6a00025020 con 0x55875eebc430 2023-10-05T19:28:26.744 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.738+0000 7f6a0f76d700 1 -- 172.21.15.167:0/1870492997 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:23 5.11 5:88000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x55875eebed50 con 0x55875eebc430 2023-10-05T19:28:26.745 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.738+0000 7f6a0f76d700 1 -- 172.21.15.167:0/1870492997 <== osd.3 v2:172.21.15.167:6808/30036 6 ==== osd_op_reply(23 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f6a00025020 con 0x55875eebc430 2023-10-05T19:28:26.745 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.738+0000 7f6a0f76d700 1 -- 172.21.15.167:0/1870492997 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:24 5.9 5:90000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x55875eebed50 con 0x55875ee8b410 2023-10-05T19:28:26.746 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.742+0000 7f6a0ff6e700 1 -- 172.21.15.167:0/1870492997 <== osd.1 v2:172.21.15.167:6800/30035 10 ==== osd_op_reply(24 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f6a08022040 con 0x55875ee8b410 2023-10-05T19:28:26.746 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.742+0000 7f6a0ff6e700 1 -- 172.21.15.167:0/1870492997 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:25 5.19 5:98000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x55875eebed50 con 0x55875ee8b410 2023-10-05T19:28:26.746 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.742+0000 7f6a0ff6e700 1 -- 172.21.15.167:0/1870492997 <== osd.1 v2:172.21.15.167:6800/30035 11 ==== osd_op_reply(25 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f6a08022040 con 0x55875ee8b410 2023-10-05T19:28:26.746 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.742+0000 7f6a0ff6e700 1 -- 172.21.15.167:0/1870492997 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:26 5.5 5:a0000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x55875eebed50 con 0x7f69f8007030 2023-10-05T19:28:26.747 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.742+0000 7f6a0ff6e700 1 -- 172.21.15.167:0/1870492997 <== osd.0 v2:172.21.15.118:6800/27690 6 ==== osd_op_reply(26 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f6a08022040 con 0x7f69f8007030 2023-10-05T19:28:26.747 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.742+0000 7f6a0ff6e700 1 -- 172.21.15.167:0/1870492997 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:27 5.15 5:a8000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x55875eebed50 con 0x55875eebc430 2023-10-05T19:28:26.748 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.742+0000 7f6a0f76d700 1 -- 172.21.15.167:0/1870492997 <== osd.3 v2:172.21.15.167:6808/30036 7 ==== osd_op_reply(27 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f6a00025020 con 0x55875eebc430 2023-10-05T19:28:26.748 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.742+0000 7f6a0f76d700 1 -- 172.21.15.167:0/1870492997 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:28 5.d 5:b0000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x55875eebed50 con 0x7f6a0001fc20 2023-10-05T19:28:26.749 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.742+0000 7f6a0ef6c700 1 -- 172.21.15.167:0/1870492997 <== osd.2 v2:172.21.15.167:6816/30037 4 ==== osd_op_reply(28 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f69f8029040 con 0x7f6a0001fc20 2023-10-05T19:28:26.749 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.742+0000 7f6a0ef6c700 1 -- 172.21.15.167:0/1870492997 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:29 5.1d 5:b8000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x55875eebed50 con 0x55875ee8b410 2023-10-05T19:28:26.749 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.742+0000 7f6a0ff6e700 1 -- 172.21.15.167:0/1870492997 <== osd.1 v2:172.21.15.167:6800/30035 12 ==== osd_op_reply(29 [pgnls start_epoch 30 out=107b] v30'2 uv2 ondisk = 1) v8 ==== 144+0+107 (crc 0 0 0) 0x7f6a08022040 con 0x55875ee8b410 2023-10-05T19:28:26.750 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.742+0000 7f6a0ff6e700 1 -- 172.21.15.167:0/1870492997 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:30 5.3 5:c0000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x55875eebed50 con 0x7f69f8007030 2023-10-05T19:28:26.750 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.746+0000 7f6a0ff6e700 1 -- 172.21.15.167:0/1870492997 <== osd.0 v2:172.21.15.118:6800/27690 7 ==== osd_op_reply(30 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f6a08022040 con 0x7f69f8007030 2023-10-05T19:28:26.751 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.746+0000 7f6a0ff6e700 1 -- 172.21.15.167:0/1870492997 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:31 5.13 5:c8000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x55875eebed50 con 0x55875eebc430 2023-10-05T19:28:26.751 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.746+0000 7f6a0f76d700 1 -- 172.21.15.167:0/1870492997 <== osd.3 v2:172.21.15.167:6808/30036 8 ==== osd_op_reply(31 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f6a00025020 con 0x55875eebc430 2023-10-05T19:28:26.751 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.746+0000 7f6a0f76d700 1 -- 172.21.15.167:0/1870492997 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:32 5.b 5:d0000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x55875eebed50 con 0x7f69f8007030 2023-10-05T19:28:26.752 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.746+0000 7f6a0ff6e700 1 -- 172.21.15.167:0/1870492997 <== osd.0 v2:172.21.15.118:6800/27690 8 ==== osd_op_reply(32 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f6a08022040 con 0x7f69f8007030 2023-10-05T19:28:26.752 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.746+0000 7f6a0ff6e700 1 -- 172.21.15.167:0/1870492997 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:33 5.1b 5:d8000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x55875eebed50 con 0x7f6a0001fc20 2023-10-05T19:28:26.753 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.746+0000 7f6a0ef6c700 1 -- 172.21.15.167:0/1870492997 <== osd.2 v2:172.21.15.167:6816/30037 5 ==== osd_op_reply(33 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f69f8029040 con 0x7f6a0001fc20 2023-10-05T19:28:26.753 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.746+0000 7f6a0ef6c700 1 -- 172.21.15.167:0/1870492997 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:34 5.7 5:e0000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x55875eebed50 con 0x55875eebc430 2023-10-05T19:28:26.754 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.750+0000 7f6a0f76d700 1 -- 172.21.15.167:0/1870492997 <== osd.3 v2:172.21.15.167:6808/30036 9 ==== osd_op_reply(34 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f6a00025020 con 0x55875eebc430 2023-10-05T19:28:26.754 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.750+0000 7f6a0f76d700 1 -- 172.21.15.167:0/1870492997 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:35 5.17 5:e8000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x55875eebed50 con 0x7f69f8007030 2023-10-05T19:28:26.755 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.750+0000 7f6a0ff6e700 1 -- 172.21.15.167:0/1870492997 <== osd.0 v2:172.21.15.118:6800/27690 9 ==== osd_op_reply(35 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f6a08022040 con 0x7f69f8007030 2023-10-05T19:28:26.755 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.750+0000 7f6a0ff6e700 1 -- 172.21.15.167:0/1870492997 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:36 5.f 5:f0000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x55875eebed50 con 0x55875ee8b410 2023-10-05T19:28:26.755 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.750+0000 7f6a0ff6e700 1 -- 172.21.15.167:0/1870492997 <== osd.1 v2:172.21.15.167:6800/30035 13 ==== osd_op_reply(36 [pgnls start_epoch 30 out=115b] v23'2 uv2 ondisk = 1) v8 ==== 144+0+115 (crc 0 0 0) 0x7f6a0802f080 con 0x55875ee8b410 2023-10-05T19:28:26.756 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.750+0000 7f6a0ff6e700 1 -- 172.21.15.167:0/1870492997 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:37 5.1f 5:f8000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x55875eebed50 con 0x7f6a0001fc20 2023-10-05T19:28:26.756 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.750+0000 7f6a0ef6c700 1 -- 172.21.15.167:0/1870492997 <== osd.2 v2:172.21.15.167:6816/30037 6 ==== osd_op_reply(37 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f69f8029040 con 0x7f6a0001fc20 2023-10-05T19:28:26.757 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.750+0000 7f6a11c38e40 20 RGWRados::pool_iterate: got zonegroup_info.4f1f719d-9648-417b-b10c-e5d939825419 2023-10-05T19:28:26.757 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.750+0000 7f6a11c38e40 20 RGWRados::pool_iterate: got zone_info.9a9df965-6612-45d4-a45f-6e3c889bacc2 2023-10-05T19:28:26.757 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.750+0000 7f6a11c38e40 20 RGWRados::pool_iterate: got zone_names.default 2023-10-05T19:28:26.757 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.750+0000 7f6a11c38e40 20 RGWRados::pool_iterate: got zonegroups_names.default 2023-10-05T19:28:26.757 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.750+0000 7f6a11c38e40 20 rados->read ofs=0 len=0 2023-10-05T19:28:26.757 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.750+0000 7f6a11c38e40 1 -- 172.21.15.167:0/1870492997 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:38 5.12 5:49953fa1:::default.realm:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x55875eebb1e0 con 0x55875ee8b410 2023-10-05T19:28:26.758 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.750+0000 7f6a0ff6e700 1 -- 172.21.15.167:0/1870492997 <== osd.1 v2:172.21.15.167:6800/30035 14 ==== osd_op_reply(38 default.realm [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 ==== 157+0+0 (crc 0 0 0) 0x7f6a0802f080 con 0x55875ee8b410 2023-10-05T19:28:26.758 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.750+0000 7f6a11c38e40 20 rados_obj.operate() r=-2 bl.length=0 2023-10-05T19:28:26.758 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.750+0000 7f6a11c38e40 20 rados->read ofs=0 len=0 2023-10-05T19:28:26.758 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.750+0000 7f6a11c38e40 1 -- 172.21.15.167:0/1870492997 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:39 5.f 5:f4c53578:::zonegroups_names.default:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x55875eebbad0 con 0x55875ee8b410 2023-10-05T19:28:26.758 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.750+0000 7f6a0ff6e700 1 -- 172.21.15.167:0/1870492997 <== osd.1 v2:172.21.15.167:6800/30035 15 ==== osd_op_reply(39 zonegroups_names.default [read 0~46 out=46b] v0'0 uv2 ondisk = 0) v8 ==== 168+0+46 (crc 0 0 0) 0x7f6a0802f080 con 0x55875ee8b410 2023-10-05T19:28:26.758 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.750+0000 7f6a11c38e40 20 rados_obj.operate() r=0 bl.length=46 2023-10-05T19:28:26.758 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.750+0000 7f6a11c38e40 20 rados->read ofs=0 len=0 2023-10-05T19:28:26.758 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.750+0000 7f6a11c38e40 1 -- 172.21.15.167:0/1870492997 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:40 5.c 5:30d299a5:::zonegroup_info.4f1f719d-9648-417b-b10c-e5d939825419:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x55875eebf190 con 0x7f69f8007030 2023-10-05T19:28:26.760 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.754+0000 7f6a0ff6e700 1 -- 172.21.15.167:0/1870492997 <== osd.0 v2:172.21.15.118:6800/27690 10 ==== osd_op_reply(40 zonegroup_info.4f1f719d-9648-417b-b10c-e5d939825419 [read 0~362 out=362b] v0'0 uv1 ondisk = 0) v8 ==== 195+0+362 (crc 0 0 0) 0x7f6a0802f080 con 0x7f69f8007030 2023-10-05T19:28:26.760 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.754+0000 7f6a11c38e40 20 rados_obj.operate() r=0 bl.length=362 2023-10-05T19:28:26.760 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.754+0000 7f6a11c38e40 20 rados->read ofs=0 len=0 2023-10-05T19:28:26.760 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.754+0000 7f6a11c38e40 1 -- 172.21.15.167:0/1870492997 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:41 5.13 5:cbbf6c60:::converted:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x55875eebf190 con 0x55875eebc430 2023-10-05T19:28:26.760 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.754+0000 7f6a0f76d700 1 -- 172.21.15.167:0/1870492997 <== osd.3 v2:172.21.15.167:6808/30036 10 ==== osd_op_reply(41 converted [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 ==== 153+0+0 (crc 0 0 0) 0x7f6a00025020 con 0x55875eebc430 2023-10-05T19:28:26.760 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.754+0000 7f6a11c38e40 20 rados_obj.operate() r=-2 bl.length=0 2023-10-05T19:28:26.760 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.754+0000 7f6a11c38e40 20 rados->read ofs=0 len=0 2023-10-05T19:28:26.760 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.754+0000 7f6a11c38e40 1 -- 172.21.15.167:0/1870492997 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:42 5.12 5:49953fa1:::default.realm:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x55875eebf540 con 0x55875ee8b410 2023-10-05T19:28:26.760 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.754+0000 7f6a0ff6e700 1 -- 172.21.15.167:0/1870492997 <== osd.1 v2:172.21.15.167:6800/30035 16 ==== osd_op_reply(42 default.realm [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 ==== 157+0+0 (crc 0 0 0) 0x7f6a0802f080 con 0x55875ee8b410 2023-10-05T19:28:26.761 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.754+0000 7f6a11c38e40 20 rados_obj.operate() r=-2 bl.length=0 2023-10-05T19:28:26.761 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.754+0000 7f6a11c38e40 20 rados->read ofs=0 len=0 2023-10-05T19:28:26.761 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.754+0000 7f6a11c38e40 1 -- 172.21.15.167:0/1870492997 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:43 5.f 5:f4c53578:::zonegroups_names.default:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x55875eebf940 con 0x55875ee8b410 2023-10-05T19:28:26.761 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.754+0000 7f6a0ff6e700 1 -- 172.21.15.167:0/1870492997 <== osd.1 v2:172.21.15.167:6800/30035 17 ==== osd_op_reply(43 zonegroups_names.default [read 0~46 out=46b] v0'0 uv2 ondisk = 0) v8 ==== 168+0+46 (crc 0 0 0) 0x7f6a0802f080 con 0x55875ee8b410 2023-10-05T19:28:26.761 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.754+0000 7f6a11c38e40 20 rados_obj.operate() r=0 bl.length=46 2023-10-05T19:28:26.761 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.754+0000 7f6a11c38e40 1 -- 172.21.15.167:0/1870492997 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:44 5.0 5:00000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x55875eec0950 con 0x55875eebc430 2023-10-05T19:28:26.763 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.758+0000 7f6a0f76d700 1 -- 172.21.15.167:0/1870492997 <== osd.3 v2:172.21.15.167:6808/30036 11 ==== osd_op_reply(44 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f6a00025020 con 0x55875eebc430 2023-10-05T19:28:26.763 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.758+0000 7f6a0f76d700 1 -- 172.21.15.167:0/1870492997 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:45 5.10 5:08000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x55875eec0950 con 0x7f6a0001fc20 2023-10-05T19:28:26.764 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.758+0000 7f6a0ef6c700 1 -- 172.21.15.167:0/1870492997 <== osd.2 v2:172.21.15.167:6816/30037 7 ==== osd_op_reply(45 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f69f8029040 con 0x7f6a0001fc20 2023-10-05T19:28:26.764 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.758+0000 7f6a0ef6c700 1 -- 172.21.15.167:0/1870492997 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:46 5.8 5:10000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x55875eec0950 con 0x7f69f8007030 2023-10-05T19:28:26.765 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.758+0000 7f6a0ff6e700 1 -- 172.21.15.167:0/1870492997 <== osd.0 v2:172.21.15.118:6800/27690 11 ==== osd_op_reply(46 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f6a0802f080 con 0x7f69f8007030 2023-10-05T19:28:26.765 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.758+0000 7f6a0ff6e700 1 -- 172.21.15.167:0/1870492997 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:47 5.18 5:18000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x55875eebbad0 con 0x55875ee8b410 2023-10-05T19:28:26.766 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.762+0000 7f6a0ff6e700 1 -- 172.21.15.167:0/1870492997 <== osd.1 v2:172.21.15.167:6800/30035 18 ==== osd_op_reply(47 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f6a0802f080 con 0x55875ee8b410 2023-10-05T19:28:26.766 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.762+0000 7f6a0ff6e700 1 -- 172.21.15.167:0/1870492997 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:48 5.4 5:20000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x55875eebbad0 con 0x55875ee8b410 2023-10-05T19:28:26.767 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.762+0000 7f6a0ff6e700 1 -- 172.21.15.167:0/1870492997 <== osd.1 v2:172.21.15.167:6800/30035 19 ==== osd_op_reply(48 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f6a0802f080 con 0x55875ee8b410 2023-10-05T19:28:26.767 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.762+0000 7f6a0ff6e700 1 -- 172.21.15.167:0/1870492997 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:49 5.14 5:28000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x55875eebbad0 con 0x7f69f8007030 2023-10-05T19:28:26.768 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.762+0000 7f6a0ff6e700 1 -- 172.21.15.167:0/1870492997 <== osd.0 v2:172.21.15.118:6800/27690 12 ==== osd_op_reply(49 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f6a0802f080 con 0x7f69f8007030 2023-10-05T19:28:26.768 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.762+0000 7f6a0ff6e700 1 -- 172.21.15.167:0/1870492997 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:50 5.c 5:30000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x55875eebbad0 con 0x7f69f8007030 2023-10-05T19:28:26.769 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.762+0000 7f6a0ff6e700 1 -- 172.21.15.167:0/1870492997 <== osd.0 v2:172.21.15.118:6800/27690 13 ==== osd_op_reply(50 [pgnls start_epoch 30 out=112b] v23'1 uv1 ondisk = 1) v8 ==== 144+0+112 (crc 0 0 0) 0x7f6a0802f080 con 0x7f69f8007030 2023-10-05T19:28:26.769 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.762+0000 7f6a0ff6e700 1 -- 172.21.15.167:0/1870492997 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:51 5.1c 5:38000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x55875eebbad0 con 0x7f6a0001fc20 2023-10-05T19:28:26.770 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.762+0000 7f6a0ef6c700 1 -- 172.21.15.167:0/1870492997 <== osd.2 v2:172.21.15.167:6816/30037 8 ==== osd_op_reply(51 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f69f8029040 con 0x7f6a0001fc20 2023-10-05T19:28:26.770 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.762+0000 7f6a0ef6c700 1 -- 172.21.15.167:0/1870492997 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:52 5.2 5:40000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x55875eebbad0 con 0x55875eebc430 2023-10-05T19:28:26.771 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.766+0000 7f6a0f76d700 1 -- 172.21.15.167:0/1870492997 <== osd.3 v2:172.21.15.167:6808/30036 12 ==== osd_op_reply(52 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f6a00025020 con 0x55875eebc430 2023-10-05T19:28:26.771 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.766+0000 7f6a0f76d700 1 -- 172.21.15.167:0/1870492997 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:53 5.12 5:48000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x55875eebbad0 con 0x55875ee8b410 2023-10-05T19:28:26.772 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.766+0000 7f6a0ff6e700 1 -- 172.21.15.167:0/1870492997 <== osd.1 v2:172.21.15.167:6800/30035 20 ==== osd_op_reply(53 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f6a0802f080 con 0x55875ee8b410 2023-10-05T19:28:26.772 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.766+0000 7f6a0ff6e700 1 -- 172.21.15.167:0/1870492997 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:54 5.a 5:50000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x55875eebbad0 con 0x7f69f8007030 2023-10-05T19:28:26.772 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.766+0000 7f6a0ff6e700 1 -- 172.21.15.167:0/1870492997 <== osd.0 v2:172.21.15.118:6800/27690 14 ==== osd_op_reply(54 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f6a0802f080 con 0x7f69f8007030 2023-10-05T19:28:26.773 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.766+0000 7f6a0ff6e700 1 -- 172.21.15.167:0/1870492997 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:55 5.1a 5:58000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x55875eebbad0 con 0x55875ee8b410 2023-10-05T19:28:26.773 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.766+0000 7f6a0ff6e700 1 -- 172.21.15.167:0/1870492997 <== osd.1 v2:172.21.15.167:6800/30035 21 ==== osd_op_reply(55 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f6a0802f080 con 0x55875ee8b410 2023-10-05T19:28:26.773 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.766+0000 7f6a0ff6e700 1 -- 172.21.15.167:0/1870492997 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:56 5.6 5:60000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x55875eebbad0 con 0x7f6a0001fc20 2023-10-05T19:28:26.774 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.770+0000 7f6a0ef6c700 1 -- 172.21.15.167:0/1870492997 <== osd.2 v2:172.21.15.167:6816/30037 9 ==== osd_op_reply(56 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f69f8029040 con 0x7f6a0001fc20 2023-10-05T19:28:26.774 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.770+0000 7f6a0ef6c700 1 -- 172.21.15.167:0/1870492997 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:57 5.16 5:68000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x55875eebbad0 con 0x55875eebc430 2023-10-05T19:28:26.775 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.770+0000 7f6a0f76d700 1 -- 172.21.15.167:0/1870492997 <== osd.3 v2:172.21.15.167:6808/30036 13 ==== osd_op_reply(57 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f6a00025020 con 0x55875eebc430 2023-10-05T19:28:26.775 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.770+0000 7f6a0f76d700 1 -- 172.21.15.167:0/1870492997 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:58 5.e 5:70000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x55875eebbad0 con 0x55875eebc430 2023-10-05T19:28:26.776 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.770+0000 7f6a0f76d700 1 -- 172.21.15.167:0/1870492997 <== osd.3 v2:172.21.15.167:6808/30036 14 ==== osd_op_reply(58 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f6a00025020 con 0x55875eebc430 2023-10-05T19:28:26.776 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.770+0000 7f6a0f76d700 1 -- 172.21.15.167:0/1870492997 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:59 5.1e 5:78000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x55875eebbad0 con 0x7f69f8007030 2023-10-05T19:28:26.777 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.770+0000 7f6a0ff6e700 1 -- 172.21.15.167:0/1870492997 <== osd.0 v2:172.21.15.118:6800/27690 15 ==== osd_op_reply(59 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f6a0802f080 con 0x7f69f8007030 2023-10-05T19:28:26.777 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.770+0000 7f6a0ff6e700 1 -- 172.21.15.167:0/1870492997 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:60 5.1 5:80000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x55875eebbad0 con 0x55875eebc430 2023-10-05T19:28:26.778 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.774+0000 7f6a0f76d700 1 -- 172.21.15.167:0/1870492997 <== osd.3 v2:172.21.15.167:6808/30036 15 ==== osd_op_reply(60 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f6a00025020 con 0x55875eebc430 2023-10-05T19:28:26.778 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.774+0000 7f6a0f76d700 1 -- 172.21.15.167:0/1870492997 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:61 5.11 5:88000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x55875eebbad0 con 0x55875eebc430 2023-10-05T19:28:26.779 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.774+0000 7f6a0f76d700 1 -- 172.21.15.167:0/1870492997 <== osd.3 v2:172.21.15.167:6808/30036 16 ==== osd_op_reply(61 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f6a00025020 con 0x55875eebc430 2023-10-05T19:28:26.779 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.774+0000 7f6a0f76d700 1 -- 172.21.15.167:0/1870492997 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:62 5.9 5:90000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x55875eebbad0 con 0x55875ee8b410 2023-10-05T19:28:26.780 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.774+0000 7f6a0ff6e700 1 -- 172.21.15.167:0/1870492997 <== osd.1 v2:172.21.15.167:6800/30035 22 ==== osd_op_reply(62 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f6a0802f080 con 0x55875ee8b410 2023-10-05T19:28:26.780 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.774+0000 7f6a0ff6e700 1 -- 172.21.15.167:0/1870492997 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:63 5.19 5:98000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x55875eebbad0 con 0x55875ee8b410 2023-10-05T19:28:26.781 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.774+0000 7f6a0ff6e700 1 -- 172.21.15.167:0/1870492997 <== osd.1 v2:172.21.15.167:6800/30035 23 ==== osd_op_reply(63 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f6a0802f080 con 0x55875ee8b410 2023-10-05T19:28:26.781 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.774+0000 7f6a0ff6e700 1 -- 172.21.15.167:0/1870492997 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:64 5.5 5:a0000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x55875eebbad0 con 0x7f69f8007030 2023-10-05T19:28:26.782 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.774+0000 7f6a0ff6e700 1 -- 172.21.15.167:0/1870492997 <== osd.0 v2:172.21.15.118:6800/27690 16 ==== osd_op_reply(64 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f6a0802f080 con 0x7f69f8007030 2023-10-05T19:28:26.782 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.778+0000 7f6a0ff6e700 1 -- 172.21.15.167:0/1870492997 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:65 5.15 5:a8000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x55875eebbad0 con 0x55875eebc430 2023-10-05T19:28:26.783 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.778+0000 7f6a0f76d700 1 -- 172.21.15.167:0/1870492997 <== osd.3 v2:172.21.15.167:6808/30036 17 ==== osd_op_reply(65 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f6a00025020 con 0x55875eebc430 2023-10-05T19:28:26.783 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.778+0000 7f6a0f76d700 1 -- 172.21.15.167:0/1870492997 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:66 5.d 5:b0000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x55875eebbad0 con 0x7f6a0001fc20 2023-10-05T19:28:26.783 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.778+0000 7f6a0ef6c700 1 -- 172.21.15.167:0/1870492997 <== osd.2 v2:172.21.15.167:6816/30037 10 ==== osd_op_reply(66 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f69f8029040 con 0x7f6a0001fc20 2023-10-05T19:28:26.784 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.778+0000 7f6a0ef6c700 1 -- 172.21.15.167:0/1870492997 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:67 5.1d 5:b8000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x55875eebbad0 con 0x55875ee8b410 2023-10-05T19:28:26.784 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.778+0000 7f6a0ff6e700 1 -- 172.21.15.167:0/1870492997 <== osd.1 v2:172.21.15.167:6800/30035 24 ==== osd_op_reply(67 [pgnls start_epoch 30 out=107b] v30'2 uv2 ondisk = 1) v8 ==== 144+0+107 (crc 0 0 0) 0x7f6a0802f080 con 0x55875ee8b410 2023-10-05T19:28:26.784 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.778+0000 7f6a0ff6e700 1 -- 172.21.15.167:0/1870492997 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:68 5.3 5:c0000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x55875eebbad0 con 0x7f69f8007030 2023-10-05T19:28:26.785 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.778+0000 7f6a0ff6e700 1 -- 172.21.15.167:0/1870492997 <== osd.0 v2:172.21.15.118:6800/27690 17 ==== osd_op_reply(68 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f6a0802f080 con 0x7f69f8007030 2023-10-05T19:28:26.785 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.778+0000 7f6a0ff6e700 1 -- 172.21.15.167:0/1870492997 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:69 5.13 5:c8000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x55875eebbad0 con 0x55875eebc430 2023-10-05T19:28:26.786 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.782+0000 7f6a0f76d700 1 -- 172.21.15.167:0/1870492997 <== osd.3 v2:172.21.15.167:6808/30036 18 ==== osd_op_reply(69 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f6a00025020 con 0x55875eebc430 2023-10-05T19:28:26.786 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.782+0000 7f6a0f76d700 1 -- 172.21.15.167:0/1870492997 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:70 5.b 5:d0000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x55875eebbad0 con 0x7f69f8007030 2023-10-05T19:28:26.787 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.782+0000 7f6a0ff6e700 1 -- 172.21.15.167:0/1870492997 <== osd.0 v2:172.21.15.118:6800/27690 18 ==== osd_op_reply(70 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f6a0802f080 con 0x7f69f8007030 2023-10-05T19:28:26.787 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.782+0000 7f6a0ff6e700 1 -- 172.21.15.167:0/1870492997 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:71 5.1b 5:d8000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x55875eebbad0 con 0x7f6a0001fc20 2023-10-05T19:28:26.788 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.782+0000 7f6a0ef6c700 1 -- 172.21.15.167:0/1870492997 <== osd.2 v2:172.21.15.167:6816/30037 11 ==== osd_op_reply(71 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f69f8029040 con 0x7f6a0001fc20 2023-10-05T19:28:26.788 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.782+0000 7f6a0ef6c700 1 -- 172.21.15.167:0/1870492997 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:72 5.7 5:e0000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x55875eebbad0 con 0x55875eebc430 2023-10-05T19:28:26.789 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.782+0000 7f6a0f76d700 1 -- 172.21.15.167:0/1870492997 <== osd.3 v2:172.21.15.167:6808/30036 19 ==== osd_op_reply(72 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f6a00025020 con 0x55875eebc430 2023-10-05T19:28:26.789 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.782+0000 7f6a0f76d700 1 -- 172.21.15.167:0/1870492997 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:73 5.17 5:e8000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x55875eebbad0 con 0x7f69f8007030 2023-10-05T19:28:26.790 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.786+0000 7f6a0ff6e700 1 -- 172.21.15.167:0/1870492997 <== osd.0 v2:172.21.15.118:6800/27690 19 ==== osd_op_reply(73 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f6a0802f080 con 0x7f69f8007030 2023-10-05T19:28:26.790 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.786+0000 7f6a0ff6e700 1 -- 172.21.15.167:0/1870492997 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:74 5.f 5:f0000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x55875eebbad0 con 0x55875ee8b410 2023-10-05T19:28:26.791 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.786+0000 7f6a0ff6e700 1 -- 172.21.15.167:0/1870492997 <== osd.1 v2:172.21.15.167:6800/30035 25 ==== osd_op_reply(74 [pgnls start_epoch 30 out=115b] v23'2 uv2 ondisk = 1) v8 ==== 144+0+115 (crc 0 0 0) 0x7f6a0802f080 con 0x55875ee8b410 2023-10-05T19:28:26.791 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.786+0000 7f6a0ff6e700 1 -- 172.21.15.167:0/1870492997 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:75 5.1f 5:f8000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x55875eebbad0 con 0x7f6a0001fc20 2023-10-05T19:28:26.792 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.786+0000 7f6a0ef6c700 1 -- 172.21.15.167:0/1870492997 <== osd.2 v2:172.21.15.167:6816/30037 12 ==== osd_op_reply(75 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f69f8029040 con 0x7f6a0001fc20 2023-10-05T19:28:26.792 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.786+0000 7f6a11c38e40 20 RGWRados::pool_iterate: got zonegroup_info.4f1f719d-9648-417b-b10c-e5d939825419 2023-10-05T19:28:26.792 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.786+0000 7f6a11c38e40 20 RGWRados::pool_iterate: got zone_info.9a9df965-6612-45d4-a45f-6e3c889bacc2 2023-10-05T19:28:26.792 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.786+0000 7f6a11c38e40 20 RGWRados::pool_iterate: got zone_names.default 2023-10-05T19:28:26.792 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.786+0000 7f6a11c38e40 20 RGWRados::pool_iterate: got zonegroups_names.default 2023-10-05T19:28:26.792 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.786+0000 7f6a11c38e40 20 rados->read ofs=0 len=0 2023-10-05T19:28:26.793 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.786+0000 7f6a11c38e40 1 -- 172.21.15.167:0/1870492997 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:76 5.f 5:f43b6ece:::zone_names.default:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x55875eebf940 con 0x55875ee8b410 2023-10-05T19:28:26.794 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.786+0000 7f6a0ff6e700 1 -- 172.21.15.167:0/1870492997 <== osd.1 v2:172.21.15.167:6800/30035 26 ==== osd_op_reply(76 zone_names.default [read 0~46 out=46b] v0'0 uv1 ondisk = 0) v8 ==== 162+0+46 (crc 0 0 0) 0x7f6a0802f080 con 0x55875ee8b410 2023-10-05T19:28:26.794 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.786+0000 7f6a11c38e40 20 rados_obj.operate() r=0 bl.length=46 2023-10-05T19:28:26.794 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.786+0000 7f6a11c38e40 20 rados->read ofs=0 len=0 2023-10-05T19:28:26.794 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.786+0000 7f6a11c38e40 1 -- 172.21.15.167:0/1870492997 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:77 5.1d 5:b8374eb4:::zone_info.9a9df965-6612-45d4-a45f-6e3c889bacc2:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x55875eebfcf0 con 0x55875ee8b410 2023-10-05T19:28:26.794 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.786+0000 7f6a0ff6e700 1 -- 172.21.15.167:0/1870492997 <== osd.1 v2:172.21.15.167:6800/30035 27 ==== osd_op_reply(77 zone_info.9a9df965-6612-45d4-a45f-6e3c889bacc2 [read 0~874 out=874b] v0'0 uv2 ondisk = 0) v8 ==== 190+0+874 (crc 0 0 0) 0x7f6a0802f080 con 0x55875ee8b410 2023-10-05T19:28:26.794 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.786+0000 7f6a11c38e40 20 rados_obj.operate() r=0 bl.length=874 2023-10-05T19:28:26.794 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.786+0000 7f6a11c38e40 20 rados->read ofs=0 len=0 2023-10-05T19:28:26.794 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.786+0000 7f6a11c38e40 1 -- 172.21.15.167:0/1870492997 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:78 5.f 5:f4c53578:::zonegroups_names.default:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x55875eebfd20 con 0x55875ee8b410 2023-10-05T19:28:26.795 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.790+0000 7f6a0ff6e700 1 -- 172.21.15.167:0/1870492997 <== osd.1 v2:172.21.15.167:6800/30035 28 ==== osd_op_reply(78 zonegroups_names.default [read 0~46 out=46b] v0'0 uv2 ondisk = 0) v8 ==== 168+0+46 (crc 0 0 0) 0x7f6a0802f080 con 0x55875ee8b410 2023-10-05T19:28:26.795 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.790+0000 7f6a11c38e40 20 rados_obj.operate() r=0 bl.length=46 2023-10-05T19:28:26.795 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.790+0000 7f6a11c38e40 20 rados->read ofs=0 len=0 2023-10-05T19:28:26.795 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.790+0000 7f6a11c38e40 1 -- 172.21.15.167:0/1870492997 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:79 5.c 5:30d299a5:::zonegroup_info.4f1f719d-9648-417b-b10c-e5d939825419:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x55875eebfd30 con 0x7f69f8007030 2023-10-05T19:28:26.796 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.790+0000 7f6a0ff6e700 1 -- 172.21.15.167:0/1870492997 <== osd.0 v2:172.21.15.118:6800/27690 20 ==== osd_op_reply(79 zonegroup_info.4f1f719d-9648-417b-b10c-e5d939825419 [read 0~362 out=362b] v0'0 uv1 ondisk = 0) v8 ==== 195+0+362 (crc 0 0 0) 0x7f6a0802f080 con 0x7f69f8007030 2023-10-05T19:28:26.796 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.790+0000 7f6a11c38e40 20 rados_obj.operate() r=0 bl.length=362 2023-10-05T19:28:26.796 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.790+0000 7f6a11c38e40 20 rados->read ofs=0 len=0 2023-10-05T19:28:26.796 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.790+0000 7f6a11c38e40 1 -- 172.21.15.167:0/1870492997 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:80 5.3 5:c0ad8cbd:::region_map:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x55875eec01e0 con 0x7f69f8007030 2023-10-05T19:28:26.797 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.790+0000 7f6a0ff6e700 1 -- 172.21.15.167:0/1870492997 <== osd.0 v2:172.21.15.118:6800/27690 21 ==== osd_op_reply(80 region_map [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 ==== 154+0+0 (crc 0 0 0) 0x7f6a0802f080 con 0x7f69f8007030 2023-10-05T19:28:26.797 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.790+0000 7f6a11c38e40 20 rados_obj.operate() r=-2 bl.length=0 2023-10-05T19:28:26.797 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.790+0000 7f6a11c38e40 20 zone default found 2023-10-05T19:28:26.797 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.790+0000 7f6a11c38e40 4 Realm: () 2023-10-05T19:28:26.797 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.790+0000 7f6a11c38e40 4 ZoneGroup: default (4f1f719d-9648-417b-b10c-e5d939825419) 2023-10-05T19:28:26.797 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.790+0000 7f6a11c38e40 4 Zone: default (9a9df965-6612-45d4-a45f-6e3c889bacc2) 2023-10-05T19:28:26.797 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.790+0000 7f6a11c38e40 10 cannot find current period zonegroup using local zonegroup configuration 2023-10-05T19:28:26.797 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.790+0000 7f6a11c38e40 20 zonegroup default 2023-10-05T19:28:26.797 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.790+0000 7f6a11c38e40 20 rados->read ofs=0 len=0 2023-10-05T19:28:26.797 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.790+0000 7f6a11c38e40 1 -- 172.21.15.167:0/1870492997 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:81 5.3 5:c52100b6:::period_config.default:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x55875eec01e0 con 0x7f69f8007030 2023-10-05T19:28:26.798 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.790+0000 7f6a0ff6e700 1 -- 172.21.15.167:0/1870492997 <== osd.0 v2:172.21.15.118:6800/27690 22 ==== osd_op_reply(81 period_config.default [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 ==== 165+0+0 (crc 0 0 0) 0x7f6a0802f080 con 0x7f69f8007030 2023-10-05T19:28:26.798 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.790+0000 7f6a11c38e40 20 rados_obj.operate() r=-2 bl.length=0 2023-10-05T19:28:26.798 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.794+0000 7f6a11c38e40 20 rados->read ofs=0 len=0 2023-10-05T19:28:26.798 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.794+0000 7f6a11c38e40 1 -- 172.21.15.167:0/1870492997 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:82 6.2 6:4347d321:::bucket.sync-source-hints.:head [call version.read in=11b,read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x55875eec18d0 con 0x55875eebc430 2023-10-05T19:28:26.799 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.794+0000 7f6a0f76d700 1 -- 172.21.15.167:0/1870492997 <== osd.3 v2:172.21.15.167:6808/30036 20 ==== osd_op_reply(82 bucket.sync-source-hints. [call,read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 ==== 211+0+0 (crc 0 0 0) 0x7f6a00025020 con 0x55875eebc430 2023-10-05T19:28:26.799 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.794+0000 7f6a11c38e40 20 rados_obj.operate() r=-2 bl.length=0 2023-10-05T19:28:26.799 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.794+0000 7f6a11c38e40 20 rados->read ofs=0 len=0 2023-10-05T19:28:26.799 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.794+0000 7f6a11c38e40 1 -- 172.21.15.167:0/1870492997 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:83 6.b 6:d467b91b:::bucket.sync-target-hints.:head [call version.read in=11b,read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x55875eec1c80 con 0x55875eebc430 2023-10-05T19:28:26.800 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.794+0000 7f6a0f76d700 1 -- 172.21.15.167:0/1870492997 <== osd.3 v2:172.21.15.167:6808/30036 21 ==== osd_op_reply(83 bucket.sync-target-hints. [call,read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 ==== 211+0+0 (crc 0 0 0) 0x7f6a00025020 con 0x55875eebc430 2023-10-05T19:28:26.800 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.794+0000 7f6a11c38e40 20 rados_obj.operate() r=-2 bl.length=0 2023-10-05T19:28:26.800 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.794+0000 7f6a11c38e40 20 started sync module instance, tier type = 2023-10-05T19:28:26.800 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.794+0000 7f6a11c38e40 20 started zone id=9a9df965-6612-45d4-a45f-6e3c889bacc2 (name=default) with tier type = 2023-10-05T19:28:26.800 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.794+0000 7f6a11c38e40 1 -- 172.21.15.167:0/1870492997 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:84 7.1f 7:f95f44c2:::notify.0:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x55875eec0f90 con 0x7f6a0001fc20 2023-10-05T19:28:26.803 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.798+0000 7f6a0ef6c700 1 -- 172.21.15.167:0/1870492997 <== osd.2 v2:172.21.15.167:6816/30037 13 ==== osd_op_reply(84 notify.0 [create] v30'39 uv39 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f69f8029040 con 0x7f6a0001fc20 2023-10-05T19:28:26.803 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.798+0000 7f6a11c38e40 1 -- 172.21.15.167:0/1870492997 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:85 7.1f 7:f95f44c2:::notify.0:head [watch watch cookie 94039901475728] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x55875eebb700 con 0x7f6a0001fc20 2023-10-05T19:28:26.803 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.798+0000 7f6a11c38e40 1 -- 172.21.15.167:0/1870492997 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:86 7.0 7:05bf5b68:::notify.1:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x55875eec2ad0 con 0x55875ee8b410 2023-10-05T19:28:26.805 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.798+0000 7f6a0ef6c700 1 -- 172.21.15.167:0/1870492997 <== osd.2 v2:172.21.15.167:6816/30037 14 ==== osd_op_reply(85 notify.0 [watch watch cookie 94039901475728] v30'40 uv39 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f69f8029040 con 0x7f6a0001fc20 2023-10-05T19:28:26.806 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.802+0000 7f6a0ff6e700 1 -- 172.21.15.167:0/1870492997 <== osd.1 v2:172.21.15.167:6800/30035 29 ==== osd_op_reply(86 notify.1 [create] v30'39 uv39 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f6a08050040 con 0x55875ee8b410 2023-10-05T19:28:26.806 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.802+0000 7f6a11c38e40 1 -- 172.21.15.167:0/1870492997 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:87 7.0 7:05bf5b68:::notify.1:head [watch watch cookie 94039901480416] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x55875eec2150 con 0x55875ee8b410 2023-10-05T19:28:26.806 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.802+0000 7f6a11c38e40 1 -- 172.21.15.167:0/1870492997 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:88 7.15 7:a93a5511:::notify.2:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x55875eec38b0 con 0x55875eebc430 2023-10-05T19:28:26.810 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.802+0000 7f6a0f76d700 1 -- 172.21.15.167:0/1870492997 <== osd.3 v2:172.21.15.167:6808/30036 22 ==== osd_op_reply(88 notify.2 [create] v30'39 uv39 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f6a00025020 con 0x55875eebc430 2023-10-05T19:28:26.810 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.802+0000 7f6a11c38e40 1 -- 172.21.15.167:0/1870492997 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:89 7.15 7:a93a5511:::notify.2:head [watch watch cookie 94039901483968] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x55875eec2150 con 0x55875eebc430 2023-10-05T19:28:26.810 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.802+0000 7f6a11c38e40 1 -- 172.21.15.167:0/1870492997 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:90 7.e 7:7759931f:::notify.3:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x55875eec4820 con 0x7f6a0001fc20 2023-10-05T19:28:26.810 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.802+0000 7f6a0ff6e700 1 -- 172.21.15.167:0/1870492997 <== osd.1 v2:172.21.15.167:6800/30035 30 ==== osd_op_reply(87 notify.1 [watch watch cookie 94039901480416] v30'40 uv39 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f6a08050040 con 0x55875ee8b410 2023-10-05T19:28:26.812 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.806+0000 7f6a0ef6c700 1 -- 172.21.15.167:0/1870492997 <== osd.2 v2:172.21.15.167:6816/30037 15 ==== osd_op_reply(90 notify.3 [create] v30'39 uv39 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f69f8029040 con 0x7f6a0001fc20 2023-10-05T19:28:26.812 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.806+0000 7f6a11c38e40 1 -- 172.21.15.167:0/1870492997 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:91 7.e 7:7759931f:::notify.3:head [watch watch cookie 94039901482704] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x55875eec4410 con 0x7f6a0001fc20 2023-10-05T19:28:26.812 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.806+0000 7f6a11c38e40 1 -- 172.21.15.167:0/1870492997 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:92 7.d 7:b4812045:::notify.4:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x55875eec58c0 con 0x55875ee8b410 2023-10-05T19:28:26.813 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.806+0000 7f6a0f76d700 1 -- 172.21.15.167:0/1870492997 <== osd.3 v2:172.21.15.167:6808/30036 23 ==== osd_op_reply(89 notify.2 [watch watch cookie 94039901483968] v30'40 uv39 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f6a00025020 con 0x55875eebc430 2023-10-05T19:28:26.815 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.810+0000 7f6a0ef6c700 1 -- 172.21.15.167:0/1870492997 <== osd.2 v2:172.21.15.167:6816/30037 16 ==== osd_op_reply(91 notify.3 [watch watch cookie 94039901482704] v30'40 uv39 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f69f8029040 con 0x7f6a0001fc20 2023-10-05T19:28:26.815 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.810+0000 7f6a0ff6e700 1 -- 172.21.15.167:0/1870492997 <== osd.1 v2:172.21.15.167:6800/30035 31 ==== osd_op_reply(92 notify.4 [create] v30'39 uv39 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f6a08050040 con 0x55875ee8b410 2023-10-05T19:28:26.815 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.810+0000 7f6a11c38e40 1 -- 172.21.15.167:0/1870492997 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:93 7.d 7:b4812045:::notify.4:head [watch watch cookie 94039901490208] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x55875eec5390 con 0x55875ee8b410 2023-10-05T19:28:26.815 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.810+0000 7f6a11c38e40 1 -- 172.21.15.167:0/1870492997 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:94 7.3 7:c609908c:::notify.5:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x55875eec61b0 con 0x7f69f8007030 2023-10-05T19:28:26.817 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.810+0000 7f6a0ff6e700 1 -- 172.21.15.167:0/1870492997 <== osd.0 v2:172.21.15.118:6800/27690 23 ==== osd_op_reply(94 notify.5 [create] v30'39 uv39 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f6a08050040 con 0x7f69f8007030 2023-10-05T19:28:26.817 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.810+0000 7f6a11c38e40 1 -- 172.21.15.167:0/1870492997 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:95 7.3 7:c609908c:::notify.5:head [watch watch cookie 94039901494384] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x55875eec65b0 con 0x7f69f8007030 2023-10-05T19:28:26.818 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.810+0000 7f6a11c38e40 1 -- 172.21.15.167:0/1870492997 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:96 7.14 7:2b04a3e9:::notify.6:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x55875eec6fc0 con 0x7f69f8007030 2023-10-05T19:28:26.818 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.814+0000 7f6a0ff6e700 1 -- 172.21.15.167:0/1870492997 <== osd.1 v2:172.21.15.167:6800/30035 32 ==== osd_op_reply(93 notify.4 [watch watch cookie 94039901490208] v30'40 uv39 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f6a08050040 con 0x55875ee8b410 2023-10-05T19:28:26.820 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.814+0000 7f6a0ff6e700 1 -- 172.21.15.167:0/1870492997 <== osd.0 v2:172.21.15.118:6800/27690 24 ==== osd_op_reply(95 notify.5 [watch watch cookie 94039901494384] v30'40 uv39 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f6a08050040 con 0x7f69f8007030 2023-10-05T19:28:26.820 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.814+0000 7f6a0ff6e700 1 -- 172.21.15.167:0/1870492997 <== osd.0 v2:172.21.15.118:6800/27690 25 ==== osd_op_reply(96 notify.6 [create] v30'39 uv39 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f6a08050040 con 0x7f69f8007030 2023-10-05T19:28:26.821 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.814+0000 7f6a11c38e40 1 -- 172.21.15.167:0/1870492997 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:97 7.14 7:2b04a3e9:::notify.6:head [watch watch cookie 94039901495488] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x55875eec6510 con 0x7f69f8007030 2023-10-05T19:28:26.821 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.814+0000 7f6a11c38e40 1 -- 172.21.15.167:0/1870492997 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:98 7.9 7:93e5b521:::notify.7:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x55875eec74b0 con 0x7f69f8007030 2023-10-05T19:28:26.823 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.818+0000 7f6a0ff6e700 1 -- 172.21.15.167:0/1870492997 <== osd.0 v2:172.21.15.118:6800/27690 26 ==== osd_op_reply(98 notify.7 [create] v30'39 uv39 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f6a08050040 con 0x7f69f8007030 2023-10-05T19:28:26.823 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.818+0000 7f6a11c38e40 1 -- 172.21.15.167:0/1870492997 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:99 7.9 7:93e5b521:::notify.7:head [watch watch cookie 94039901499328] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x55875eec7900 con 0x7f69f8007030 2023-10-05T19:28:26.823 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.818+0000 7f6a11c38e40 20 add_watcher() i=0 2023-10-05T19:28:26.824 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.818+0000 7f6a11c38e40 20 add_watcher() i=1 2023-10-05T19:28:26.824 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.818+0000 7f6a11c38e40 20 add_watcher() i=2 2023-10-05T19:28:26.824 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.818+0000 7f6a11c38e40 20 add_watcher() i=3 2023-10-05T19:28:26.824 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.818+0000 7f6a11c38e40 20 add_watcher() i=4 2023-10-05T19:28:26.824 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.818+0000 7f6a11c38e40 20 add_watcher() i=5 2023-10-05T19:28:26.824 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.818+0000 7f6a0ff6e700 1 -- 172.21.15.167:0/1870492997 <== osd.0 v2:172.21.15.118:6800/27690 27 ==== osd_op_reply(97 notify.6 [watch watch cookie 94039901495488] v30'40 uv39 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f6a08050040 con 0x7f69f8007030 2023-10-05T19:28:26.824 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.818+0000 7f6a11c38e40 20 add_watcher() i=6 2023-10-05T19:28:26.825 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.818+0000 7f6a0ff6e700 1 -- 172.21.15.167:0/1870492997 <== osd.0 v2:172.21.15.118:6800/27690 28 ==== osd_op_reply(99 notify.7 [watch watch cookie 94039901499328] v30'40 uv39 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f6a08050040 con 0x7f69f8007030 2023-10-05T19:28:26.825 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.818+0000 7f6a11c38e40 20 add_watcher() i=7 2023-10-05T19:28:26.826 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.818+0000 7f6a11c38e40 2 all 8 watchers are set, enabling cache 2023-10-05T19:28:26.826 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.818+0000 7f6a11c38e40 1 -- 172.21.15.167:0/1870492997 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:100 6.4 6:22d26bf9:::data_loggenerations_metadata:head [call version.check_conds in=74b,call version.read in=11b,read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x55875eec6ad0 con 0x55875ee8b410 2023-10-05T19:28:26.826 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.822+0000 7f6a0ff6e700 1 -- 172.21.15.167:0/1870492997 <== osd.1 v2:172.21.15.167:6800/30035 33 ==== osd_op_reply(100 data_loggenerations_metadata [call,call out=48b,read 0~28 out=28b] v0'0 uv1 ondisk = 0) v8 ==== 256+0+76 (crc 0 0 0) 0x7f6a08050040 con 0x55875ee8b410 2023-10-05T19:28:26.826 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.822+0000 7f6a11c38e40 1 -- 172.21.15.167:0/1870492997 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:101 6.4 6:22d26bf9:::data_loggenerations_metadata:head [watch watch cookie 94039901500432] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x55875eec7d10 con 0x55875ee8b410 2023-10-05T19:28:26.829 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.822+0000 7f6a0ff6e700 1 -- 172.21.15.167:0/1870492997 <== osd.1 v2:172.21.15.167:6800/30035 34 ==== osd_op_reply(101 data_loggenerations_metadata [watch watch cookie 94039901500432] v30'60 uv1 ondisk = 0) v8 ==== 172+0+0 (crc 0 0 0) 0x7f6a08050040 con 0x55875ee8b410 2023-10-05T19:28:26.829 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.822+0000 7f69e0fe1700 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2023-10-05T19:28:26.829 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.822+0000 7f6a11c38e40 20 check_secure_mon_conn(): auth registy supported: methods=[2] modes=[2,1] 2023-10-05T19:28:26.829 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.822+0000 7f6a11c38e40 20 check_secure_mon_conn(): mode 1 is insecure 2023-10-05T19:28:26.830 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.822+0000 7f6a11c38e40 1 Processor -- start 2023-10-05T19:28:26.830 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.822+0000 7f6a11c38e40 1 -- start start 2023-10-05T19:28:26.830 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.826+0000 7f6a11c38e40 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55875eed4e90 0x55875eed5260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-05T19:28:26.830 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.826+0000 7f6a11c38e40 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x55875eed57a0 con 0x55875eed4e90 2023-10-05T19:28:26.831 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.826+0000 7f6a0ef6c700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55875eed4e90 0x55875eed5260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-05T19:28:26.831 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.826+0000 7f6a0ef6c700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55875eed4e90 0x55875eed5260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.118:3300/0 says I am v2:172.21.15.167:57404/0 (socket says 172.21.15.167:57404) 2023-10-05T19:28:26.831 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.826+0000 7f6a0ef6c700 1 -- 172.21.15.167:0/2470739383 learned_addr learned my addr 172.21.15.167:0/2470739383 (peer_addr_for_me v2:172.21.15.167:0/0) 2023-10-05T19:28:26.832 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.826+0000 7f6a0ef6c700 1 -- 172.21.15.167:0/2470739383 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55875eed58e0 con 0x55875eed4e90 2023-10-05T19:28:26.832 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.826+0000 7f6a0ef6c700 1 --2- 172.21.15.167:0/2470739383 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55875eed4e90 0x55875eed5260 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x55875ee87eb0 tx=0x7f69f8011000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d6a577e6383308cb server_cookie=0 in_seq=0 out_seq=0 2023-10-05T19:28:26.832 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.826+0000 7f695bfff700 1 -- 172.21.15.167:0/2470739383 <== mon.0 v2:172.21.15.118:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f69f8025040 con 0x55875eed4e90 2023-10-05T19:28:26.832 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.826+0000 7f695bfff700 1 -- 172.21.15.167:0/2470739383 <== mon.0 v2:172.21.15.118:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f69f8039450 con 0x55875eed4e90 2023-10-05T19:28:26.832 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.826+0000 7f695bfff700 1 -- 172.21.15.167:0/2470739383 <== mon.0 v2:172.21.15.118:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f69f8039740 con 0x55875eed4e90 2023-10-05T19:28:26.833 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.826+0000 7f6a11c38e40 1 -- 172.21.15.167:0/2470739383 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55875eed4e90 msgr2=0x55875eed5260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-05T19:28:26.833 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.826+0000 7f6a11c38e40 1 --2- 172.21.15.167:0/2470739383 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55875eed4e90 0x55875eed5260 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x55875ee87eb0 tx=0x7f69f8011000 comp rx=0 tx=0).stop 2023-10-05T19:28:26.833 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.826+0000 7f6a11c38e40 1 -- 172.21.15.167:0/2470739383 shutdown_connections 2023-10-05T19:28:26.834 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.826+0000 7f6a11c38e40 1 --2- 172.21.15.167:0/2470739383 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55875eed4e90 0x55875eed5260 unknown :-1 s=CLOSED pgs=216 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-05T19:28:26.834 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.826+0000 7f6a11c38e40 1 -- 172.21.15.167:0/2470739383 >> 172.21.15.167:0/2470739383 conn(0x55875eed0250 msgr2=0x55875eece3a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-05T19:28:26.834 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.826+0000 7f6a11c38e40 1 -- 172.21.15.167:0/2470739383 shutdown_connections 2023-10-05T19:28:26.834 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.826+0000 7f6a11c38e40 1 -- 172.21.15.167:0/2470739383 wait complete. 2023-10-05T19:28:26.835 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.830+0000 7f6a11c38e40 1 Processor -- start 2023-10-05T19:28:26.835 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.830+0000 7f6a11c38e40 1 -- start start 2023-10-05T19:28:26.835 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.830+0000 7f6a11c38e40 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55875eed4e90 0x55875eeda950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-05T19:28:26.835 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.830+0000 7f6a11c38e40 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x55875eedb950 con 0x55875eed4e90 2023-10-05T19:28:26.835 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.830+0000 7f6a0ef6c700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55875eed4e90 0x55875eeda950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-05T19:28:26.835 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.830+0000 7f6a0ef6c700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55875eed4e90 0x55875eeda950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.118:3300/0 says I am v2:172.21.15.167:57414/0 (socket says 172.21.15.167:57414) 2023-10-05T19:28:26.835 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.830+0000 7f6a0ef6c700 1 -- 172.21.15.167:0/467276925 learned_addr learned my addr 172.21.15.167:0/467276925 (peer_addr_for_me v2:172.21.15.167:0/0) 2023-10-05T19:28:26.835 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.830+0000 7f6a0ef6c700 1 -- 172.21.15.167:0/467276925 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55875ee87a40 con 0x55875eed4e90 2023-10-05T19:28:26.836 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.830+0000 7f6a0ef6c700 1 --2- 172.21.15.167:0/467276925 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55875eed4e90 0x55875eeda950 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x55875eecef10 tx=0x7f69f8039000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-05T19:28:26.836 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.830+0000 7f695a7fc700 1 -- 172.21.15.167:0/467276925 <== mon.0 v2:172.21.15.118:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f69f8025040 con 0x55875eed4e90 2023-10-05T19:28:26.837 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.830+0000 7f695a7fc700 1 -- 172.21.15.167:0/467276925 <== mon.0 v2:172.21.15.118:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f69f8047070 con 0x55875eed4e90 2023-10-05T19:28:26.837 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.830+0000 7f6a11c38e40 1 -- 172.21.15.167:0/467276925 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x55875eedaf50 con 0x55875eed4e90 2023-10-05T19:28:26.837 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.830+0000 7f695a7fc700 1 -- 172.21.15.167:0/467276925 <== mon.0 v2:172.21.15.118:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f69f802d680 con 0x55875eed4e90 2023-10-05T19:28:26.837 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.830+0000 7f6a11c38e40 1 -- 172.21.15.167:0/467276925 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55875eedb3f0 con 0x55875eed4e90 2023-10-05T19:28:26.837 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.830+0000 7f695a7fc700 1 -- 172.21.15.167:0/467276925 <== mon.0 v2:172.21.15.118:3300/0 4 ==== mgrmap(e 5) v1 ==== 40329+0+0 (secure 0 0 0) 0x7f69f8049070 con 0x55875eed4e90 2023-10-05T19:28:26.838 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.830+0000 7f695a7fc700 1 --2- 172.21.15.167:0/467276925 >> [v2:172.21.15.167:6824/29889,v1:172.21.15.167:6825/29889] conn(0x7f6950030c50 0x7f6950033100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-05T19:28:26.838 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.830+0000 7f695a7fc700 1 -- 172.21.15.167:0/467276925 <== mon.0 v2:172.21.15.118:3300/0 5 ==== osd_map(30..30 src has 1..30) v4 ==== 6057+0+0 (secure 0 0 0) 0x7f69f8032030 con 0x55875eed4e90 2023-10-05T19:28:26.838 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.830+0000 7f6a11c38e40 5 note: GC not initialized 2023-10-05T19:28:26.838 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.830+0000 7f6a0ff6e700 1 --2- 172.21.15.167:0/467276925 >> [v2:172.21.15.167:6824/29889,v1:172.21.15.167:6825/29889] conn(0x7f6950030c50 0x7f6950033100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-05T19:28:26.838 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.834+0000 7f6a0ff6e700 1 --2- 172.21.15.167:0/467276925 >> [v2:172.21.15.167:6824/29889,v1:172.21.15.167:6825/29889] conn(0x7f6950030c50 0x7f6950033100 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x55875eec4120 tx=0x7f6a08000f90 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-05T19:28:26.840 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.834+0000 7f694f7fe700 20 reqs_thread_entry: start 2023-10-05T19:28:26.840 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.834+0000 7f694d7fa700 20 reqs_thread_entry: start 2023-10-05T19:28:26.840 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.834+0000 7f6a11c38e40 20 init_complete bucket index max shards: 11 2023-10-05T19:28:26.840 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.834+0000 7f6a11c38e40 10 rgw notify: Started notification manager with: 1 workers 2023-10-05T19:28:26.841 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.834+0000 7f693affd700 20 reqs_thread_entry: start 2023-10-05T19:28:26.841 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.834+0000 7f6a11c38e40 20 get_system_obj_state: rctx=0x7fff980bb718 obj=default.rgw.meta:users.uid:foo state=0x55875ed600b0 s->prefetch_data=0 2023-10-05T19:28:26.841 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.834+0000 7f6a11c38e40 10 cache get: name=default.rgw.meta+users.uid+foo : miss 2023-10-05T19:28:26.841 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.834+0000 7f6a11c38e40 1 -- 172.21.15.167:0/1870492997 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:102 8.18 8:1e6ca5ed:users.uid::foo:head [call version.read in=11b,getxattrs,stat] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x55875ef42fe0 con 0x55875ee8b410 2023-10-05T19:28:26.842 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.834+0000 7f693b7fe700 20 rgw notify: INFO: next queues processing will happen at: Thu Oct 5 19:28:57 2023 2023-10-05T19:28:26.842 INFO:teuthology.orchestra.run.smithi167.stderr: 2023-10-05T19:28:26.842 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.834+0000 7f6a0ff6e700 1 -- 172.21.15.167:0/1870492997 <== osd.1 v2:172.21.15.167:6800/30035 35 ==== osd_op_reply(102 foo [call out=48b,getxattrs out=75b,stat out=16b] v0'0 uv2 ondisk = 0) v8 ==== 231+0+139 (crc 0 0 0) 0x7f6a08050040 con 0x55875ee8b410 2023-10-05T19:28:26.842 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.834+0000 7f6a11c38e40 10 cache put: name=default.rgw.meta+users.uid+foo info.flags=0x16 2023-10-05T19:28:26.842 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.834+0000 7f6a11c38e40 10 adding default.rgw.meta+users.uid+foo to cache LRU end 2023-10-05T19:28:26.842 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.834+0000 7f6a11c38e40 10 updating xattr: name=ceph.objclass.version bl.length()=42 2023-10-05T19:28:26.842 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.834+0000 7f6a11c38e40 20 get_system_obj_state: s->obj_tag was set empty 2023-10-05T19:28:26.842 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.834+0000 7f6a11c38e40 20 Read xattr: user.rgw.idtag 2023-10-05T19:28:26.842 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.834+0000 7f6a11c38e40 10 cache get: name=default.rgw.meta+users.uid+foo : type miss (requested=0x13, cached=0x16) 2023-10-05T19:28:26.842 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.834+0000 7f6a11c38e40 20 rados->read ofs=0 len=0 2023-10-05T19:28:26.842 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.834+0000 7f6a11c38e40 1 -- 172.21.15.167:0/1870492997 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:103 8.18 8:1e6ca5ed:users.uid::foo:head [call version.check_conds in=122b,call version.read in=11b,read 0~0,getxattrs] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x55875ef45a70 con 0x55875ee8b410 2023-10-05T19:28:26.843 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.834+0000 7f6a0ff6e700 1 -- 172.21.15.167:0/1870492997 <== osd.1 v2:172.21.15.167:6800/30035 36 ==== osd_op_reply(103 foo [call,call out=48b,read 0~357 out=357b,getxattrs out=75b] v0'0 uv2 ondisk = 0) v8 ==== 273+0+480 (crc 0 0 0) 0x7f6a08050040 con 0x55875ee8b410 2023-10-05T19:28:26.843 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.834+0000 7f6a11c38e40 20 rados_obj.operate() r=0 bl.length=357 2023-10-05T19:28:26.843 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.834+0000 7f6a11c38e40 10 cache put: name=default.rgw.meta+users.uid+foo info.flags=0x13 2023-10-05T19:28:26.843 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.834+0000 7f6a11c38e40 10 moving default.rgw.meta+users.uid+foo to cache LRU end 2023-10-05T19:28:26.843 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.834+0000 7f6a11c38e40 10 updating xattr: name=ceph.objclass.version bl.length()=42 2023-10-05T19:28:26.844 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.834+0000 7f6a11c38e40 1 -- 172.21.15.167:0/1870492997 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:104 8.9 8:96d98480:users.uid::foo.buckets:head [call user.list_buckets in=34b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x55875ef45a70 con 0x7f6a0001fc20 2023-10-05T19:28:26.844 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.838+0000 7f6a0ef6c700 1 -- 172.21.15.167:0/1870492997 <== osd.2 v2:172.21.15.167:6816/30037 17 ==== osd_op_reply(104 foo.buckets [call] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 ==== 155+0+0 (crc 0 0 0) 0x7f69f803f020 con 0x7f6a0001fc20 2023-10-05T19:28:26.844 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.838+0000 7f6a11c38e40 1 -- 172.21.15.167:0/1870492997 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:105 8.18 8:1e6ca5ed:users.uid::foo:head [delete,create,call version.set in=58b,writefull 0~357 in=357b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x55875ef46cf0 con 0x55875ee8b410 2023-10-05T19:28:26.844 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.838+0000 7f6a0ff6e700 1 -- 172.21.15.167:0/1870492997 <== osd.1 v2:172.21.15.167:6800/30035 37 ==== osd_op_reply(105 foo [delete,create,call,writefull 0~357] v30'3 uv3 ondisk = 0) v8 ==== 273+0+0 (crc 0 0 0) 0x7f6a08050040 con 0x55875ee8b410 2023-10-05T19:28:26.844 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.838+0000 7f6a11c38e40 10 cache put: name=default.rgw.meta+users.uid+foo info.flags=0x17 2023-10-05T19:28:26.845 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.838+0000 7f6a11c38e40 10 moving default.rgw.meta+users.uid+foo to cache LRU end 2023-10-05T19:28:26.845 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.838+0000 7f6a11c38e40 10 distributing notification oid=default.rgw.control:notify.4 cni=[op: 0, obj: default.rgw.meta:users.uid:foo, ofs0, ns] 2023-10-05T19:28:26.845 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.838+0000 7f6a11c38e40 1 -- 172.21.15.167:0/1870492997 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:106 7.d 7:b4812045:::notify.4:head [notify cookie 94039902021632 in=545b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x55875ef45a70 con 0x55875ee8b410 2023-10-05T19:28:26.846 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.838+0000 7f6a0ff6e700 1 -- 172.21.15.167:0/1870492997 <== osd.1 v2:172.21.15.167:6800/30035 38 ==== watch-notify(notify (1) cookie 94039901490208 notify 128849018882 ret 0) v3 ==== 575+0+0 (crc 0 0 0) 0x7f6a0803d040 con 0x55875ee8b410 2023-10-05T19:28:26.846 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.838+0000 7f6a0d769700 10 rgw watcher librados: RGWWatcher::handle_notify() notify_id 128849018882 cookie 94039901490208 notifier 4305 bl.length()=533 2023-10-05T19:28:26.846 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.838+0000 7f6a0d769700 10 rgw watcher librados: cache put: name=default.rgw.meta+users.uid+foo info.flags=0x17 2023-10-05T19:28:26.846 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.838+0000 7f6a0d769700 10 rgw watcher librados: moving default.rgw.meta+users.uid+foo to cache LRU end 2023-10-05T19:28:26.846 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.838+0000 7f6a0d769700 1 -- 172.21.15.167:0/1870492997 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:107 7.d 7:b4812045:::notify.4:head [notify-ack in=20b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x7f69f4003d90 con 0x55875ee8b410 2023-10-05T19:28:26.846 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.838+0000 7f6a0ff6e700 1 -- 172.21.15.167:0/1870492997 <== osd.1 v2:172.21.15.167:6800/30035 39 ==== osd_op_reply(106 notify.4 [notify cookie 94039902021632 out=8b] v0'0 uv39 ondisk = 0) v8 ==== 152+0+8 (crc 0 0 0) 0x7f6a08050040 con 0x55875ee8b410 2023-10-05T19:28:26.846 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.838+0000 7f6a0ff6e700 1 -- 172.21.15.167:0/1870492997 <== osd.1 v2:172.21.15.167:6800/30035 40 ==== osd_op_reply(107 notify.4 [notify-ack] v0'0 uv39 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f6a08050040 con 0x55875ee8b410 2023-10-05T19:28:26.846 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.842+0000 7f6a0ff6e700 1 -- 172.21.15.167:0/1870492997 <== osd.1 v2:172.21.15.167:6800/30035 41 ==== watch-notify(notify_complete (2) cookie 94039902021632 notify 128849018882 ret 0) v3 ==== 42+0+48 (crc 0 0 0) 0x7f6a080024c0 con 0x55875ee8b410 2023-10-05T19:28:26.847 INFO:teuthology.orchestra.run.smithi167.stdout:{ 2023-10-05T19:28:26.847 INFO:teuthology.orchestra.run.smithi167.stdout: "user_id": "foo", 2023-10-05T19:28:26.847 INFO:teuthology.orchestra.run.smithi167.stdout: "display_name": "Foo", 2023-10-05T19:28:26.847 INFO:teuthology.orchestra.run.smithi167.stdout: "email": "foo@foo.com", 2023-10-05T19:28:26.847 INFO:teuthology.orchestra.run.smithi167.stdout: "suspended": 0, 2023-10-05T19:28:26.847 INFO:teuthology.orchestra.run.smithi167.stdout: "max_buckets": 4, 2023-10-05T19:28:26.847 INFO:teuthology.orchestra.run.smithi167.stdout: "subusers": [], 2023-10-05T19:28:26.847 INFO:teuthology.orchestra.run.smithi167.stdout: "keys": [ 2023-10-05T19:28:26.847 INFO:teuthology.orchestra.run.smithi167.stdout: { 2023-10-05T19:28:26.847 INFO:teuthology.orchestra.run.smithi167.stdout: "user": "foo", 2023-10-05T19:28:26.847 INFO:teuthology.orchestra.run.smithi167.stdout: "access_key": "9te6NH5mcdcq0Tc5i8i1", 2023-10-05T19:28:26.848 INFO:teuthology.orchestra.run.smithi167.stdout: "secret_key": "Ny4IOauQoL18Gp2zM7lC1vLmoawgqcYP/YGcWfXu" 2023-10-05T19:28:26.848 INFO:teuthology.orchestra.run.smithi167.stdout: } 2023-10-05T19:28:26.848 INFO:teuthology.orchestra.run.smithi167.stdout: ], 2023-10-05T19:28:26.848 INFO:teuthology.orchestra.run.smithi167.stdout: "swift_keys": [], 2023-10-05T19:28:26.848 INFO:teuthology.orchestra.run.smithi167.stdout: "caps": [], 2023-10-05T19:28:26.848 INFO:teuthology.orchestra.run.smithi167.stdout: "op_mask": "read, write, delete", 2023-10-05T19:28:26.848 INFO:teuthology.orchestra.run.smithi167.stdout: "default_placement": "", 2023-10-05T19:28:26.848 INFO:teuthology.orchestra.run.smithi167.stdout: "default_storage_class": "", 2023-10-05T19:28:26.848 INFO:teuthology.orchestra.run.smithi167.stdout: "placement_tags": [], 2023-10-05T19:28:26.848 INFO:teuthology.orchestra.run.smithi167.stdout: "bucket_quota": { 2023-10-05T19:28:26.848 INFO:teuthology.orchestra.run.smithi167.stdout: "enabled": false, 2023-10-05T19:28:26.848 INFO:teuthology.orchestra.run.smithi167.stdout: "check_on_raw": false, 2023-10-05T19:28:26.848 INFO:teuthology.orchestra.run.smithi167.stdout: "max_size": -1, 2023-10-05T19:28:26.849 INFO:teuthology.orchestra.run.smithi167.stdout: "max_size_kb": 0, 2023-10-05T19:28:26.849 INFO:teuthology.orchestra.run.smithi167.stdout: "max_objects": -1 2023-10-05T19:28:26.849 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2023-10-05T19:28:26.849 INFO:teuthology.orchestra.run.smithi167.stdout: "user_quota": { 2023-10-05T19:28:26.849 INFO:teuthology.orchestra.run.smithi167.stdout: "enabled": false, 2023-10-05T19:28:26.849 INFO:teuthology.orchestra.run.smithi167.stdout: "check_on_raw": false, 2023-10-05T19:28:26.849 INFO:teuthology.orchestra.run.smithi167.stdout: "max_size": -1, 2023-10-05T19:28:26.849 INFO:teuthology.orchestra.run.smithi167.stdout: "max_size_kb": 0, 2023-10-05T19:28:26.849 INFO:teuthology.orchestra.run.smithi167.stdout: "max_objects": -1 2023-10-05T19:28:26.849 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2023-10-05T19:28:26.849 INFO:teuthology.orchestra.run.smithi167.stdout: "temp_url_keys": [], 2023-10-05T19:28:26.849 INFO:teuthology.orchestra.run.smithi167.stdout: "type": "rgw", 2023-10-05T19:28:26.850 INFO:teuthology.orchestra.run.smithi167.stdout: "mfa_ids": [] 2023-10-05T19:28:26.850 INFO:teuthology.orchestra.run.smithi167.stdout:} 2023-10-05T19:28:26.850 INFO:teuthology.orchestra.run.smithi167.stdout: 2023-10-05T19:28:26.850 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.842+0000 7f6a11c38e40 1 -- 172.21.15.167:0/1870492997 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:108 7.1f 7:f95f44c2:::notify.0:head [watch unwatch cookie 94039901475728] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x55875ee9c640 con 0x7f6a0001fc20 2023-10-05T19:28:26.850 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.842+0000 7f6a0ef6c700 1 -- 172.21.15.167:0/1870492997 <== osd.2 v2:172.21.15.167:6816/30037 18 ==== osd_op_reply(108 notify.0 [watch unwatch cookie 94039901475728] v30'41 uv39 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f69f803f020 con 0x7f6a0001fc20 2023-10-05T19:28:26.851 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.842+0000 7f6a11c38e40 20 remove_watcher() i=0 2023-10-05T19:28:26.851 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.842+0000 7f6a11c38e40 2 removed watcher, disabling cache 2023-10-05T19:28:26.851 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.842+0000 7f6a11c38e40 1 -- 172.21.15.167:0/1870492997 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:109 7.0 7:05bf5b68:::notify.1:head [watch unwatch cookie 94039901480416] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x55875ee9c640 con 0x55875ee8b410 2023-10-05T19:28:26.851 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.846+0000 7f6a0ff6e700 1 -- 172.21.15.167:0/1870492997 <== osd.1 v2:172.21.15.167:6800/30035 42 ==== osd_op_reply(109 notify.1 [watch unwatch cookie 94039901480416] v30'41 uv39 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f6a08050040 con 0x55875ee8b410 2023-10-05T19:28:26.851 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.846+0000 7f6a11c38e40 20 remove_watcher() i=1 2023-10-05T19:28:26.851 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.846+0000 7f6a11c38e40 1 -- 172.21.15.167:0/1870492997 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:110 7.15 7:a93a5511:::notify.2:head [watch unwatch cookie 94039901483968] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x55875ee9c640 con 0x55875eebc430 2023-10-05T19:28:26.853 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.846+0000 7f6a0f76d700 1 -- 172.21.15.167:0/1870492997 <== osd.3 v2:172.21.15.167:6808/30036 24 ==== osd_op_reply(110 notify.2 [watch unwatch cookie 94039901483968] v30'41 uv39 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f6a00025020 con 0x55875eebc430 2023-10-05T19:28:26.854 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.846+0000 7f6a11c38e40 20 remove_watcher() i=2 2023-10-05T19:28:26.854 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.846+0000 7f6a11c38e40 1 -- 172.21.15.167:0/1870492997 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:111 7.e 7:7759931f:::notify.3:head [watch unwatch cookie 94039901482704] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x55875eec1390 con 0x7f6a0001fc20 2023-10-05T19:28:26.856 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.850+0000 7f6a0ef6c700 1 -- 172.21.15.167:0/1870492997 <== osd.2 v2:172.21.15.167:6816/30037 19 ==== osd_op_reply(111 notify.3 [watch unwatch cookie 94039901482704] v30'41 uv39 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f69f803f020 con 0x7f6a0001fc20 2023-10-05T19:28:26.856 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.850+0000 7f6a11c38e40 20 remove_watcher() i=3 2023-10-05T19:28:26.856 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.850+0000 7f6a11c38e40 1 -- 172.21.15.167:0/1870492997 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:112 7.d 7:b4812045:::notify.4:head [watch unwatch cookie 94039901490208] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x55875eec33c0 con 0x55875ee8b410 2023-10-05T19:28:26.858 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.854+0000 7f6a0ff6e700 1 -- 172.21.15.167:0/1870492997 <== osd.1 v2:172.21.15.167:6800/30035 43 ==== osd_op_reply(112 notify.4 [watch unwatch cookie 94039901490208] v30'41 uv39 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f6a08050040 con 0x55875ee8b410 2023-10-05T19:28:26.858 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.854+0000 7f6a11c38e40 20 remove_watcher() i=4 2023-10-05T19:28:26.858 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.854+0000 7f6a11c38e40 1 -- 172.21.15.167:0/1870492997 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:113 7.3 7:c609908c:::notify.5:head [watch unwatch cookie 94039901494384] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x55875eec1390 con 0x7f69f8007030 2023-10-05T19:28:26.861 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.854+0000 7f6a0ff6e700 1 -- 172.21.15.167:0/1870492997 <== osd.0 v2:172.21.15.118:6800/27690 29 ==== osd_op_reply(113 notify.5 [watch unwatch cookie 94039901494384] v30'41 uv39 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f6a08050040 con 0x7f69f8007030 2023-10-05T19:28:26.861 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.854+0000 7f6a11c38e40 20 remove_watcher() i=5 2023-10-05T19:28:26.861 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.854+0000 7f6a11c38e40 1 -- 172.21.15.167:0/1870492997 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:114 7.14 7:2b04a3e9:::notify.6:head [watch unwatch cookie 94039901495488] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x55875eec4c20 con 0x7f69f8007030 2023-10-05T19:28:26.863 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.858+0000 7f6a0ff6e700 1 -- 172.21.15.167:0/1870492997 <== osd.0 v2:172.21.15.118:6800/27690 30 ==== osd_op_reply(114 notify.6 [watch unwatch cookie 94039901495488] v30'41 uv39 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f6a08050040 con 0x7f69f8007030 2023-10-05T19:28:26.864 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.858+0000 7f6a11c38e40 20 remove_watcher() i=6 2023-10-05T19:28:26.864 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.858+0000 7f6a11c38e40 1 -- 172.21.15.167:0/1870492997 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:115 7.9 7:93e5b521:::notify.7:head [watch unwatch cookie 94039901499328] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x55875eec6160 con 0x7f69f8007030 2023-10-05T19:28:26.866 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.862+0000 7f6a0ff6e700 1 -- 172.21.15.167:0/1870492997 <== osd.0 v2:172.21.15.118:6800/27690 31 ==== osd_op_reply(115 notify.7 [watch unwatch cookie 94039901499328] v30'41 uv39 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f6a08050040 con 0x7f69f8007030 2023-10-05T19:28:26.866 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.862+0000 7f6a11c38e40 20 remove_watcher() i=7 2023-10-05T19:28:26.868 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.862+0000 7f69e0fe1700 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2023-10-05T19:28:26.868 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.862+0000 7f6a11c38e40 1 -- 172.21.15.167:0/1870492997 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:116 6.4 6:22d26bf9:::data_loggenerations_metadata:head [watch unwatch cookie 94039901500432] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x55875eec33c0 con 0x55875ee8b410 2023-10-05T19:28:26.871 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.866+0000 7f6a0ff6e700 1 -- 172.21.15.167:0/1870492997 <== osd.1 v2:172.21.15.167:6800/30035 44 ==== osd_op_reply(116 data_loggenerations_metadata [watch unwatch cookie 94039901500432] v30'61 uv1 ondisk = 0) v8 ==== 172+0+0 (crc 0 0 0) 0x7f6a08050040 con 0x55875ee8b410 2023-10-05T19:28:26.872 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.866+0000 7f6a11c38e40 1 -- 172.21.15.167:0/1870492997 >> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] conn(0x7f69f8007030 msgr2=0x7f69f801d430 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-05T19:28:26.872 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.866+0000 7f6a11c38e40 1 --2- 172.21.15.167:0/1870492997 >> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] conn(0x7f69f8007030 0x7f69f801d430 crc :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-05T19:28:26.872 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.866+0000 7f6a11c38e40 1 -- 172.21.15.167:0/1870492997 >> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] conn(0x55875ee8b410 msgr2=0x55875eeb8600 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-05T19:28:26.872 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.866+0000 7f6a11c38e40 1 --2- 172.21.15.167:0/1870492997 >> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] conn(0x55875ee8b410 0x55875eeb8600 crc :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-05T19:28:26.872 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.866+0000 7f6a11c38e40 1 -- 172.21.15.167:0/1870492997 >> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] conn(0x7f6a0001fc20 msgr2=0x7f6a0002a000 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-05T19:28:26.872 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.866+0000 7f6a11c38e40 1 --2- 172.21.15.167:0/1870492997 >> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] conn(0x7f6a0001fc20 0x7f6a0002a000 crc :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-05T19:28:26.872 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.866+0000 7f6a11c38e40 1 -- 172.21.15.167:0/1870492997 >> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] conn(0x55875eebc430 msgr2=0x55875eebe810 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-05T19:28:26.872 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.866+0000 7f6a11c38e40 1 --2- 172.21.15.167:0/1870492997 >> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] conn(0x55875eebc430 0x55875eebe810 crc :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-05T19:28:26.872 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.866+0000 7f6a11c38e40 1 -- 172.21.15.167:0/1870492997 >> [v2:172.21.15.167:6824/29889,v1:172.21.15.167:6825/29889] conn(0x7f69f0030c00 msgr2=0x7f69f00330b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-05T19:28:26.872 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.866+0000 7f6a11c38e40 1 --2- 172.21.15.167:0/1870492997 >> [v2:172.21.15.167:6824/29889,v1:172.21.15.167:6825/29889] conn(0x7f69f0030c00 0x7f69f00330b0 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f69f800afd0 tx=0x7f69f8014000 comp rx=0 tx=0).stop 2023-10-05T19:28:26.872 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.866+0000 7f6a11c38e40 1 -- 172.21.15.167:0/1870492997 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55875ee86e30 msgr2=0x55875eeacb40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-05T19:28:26.873 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.866+0000 7f6a0ff6e700 1 -- 172.21.15.167:0/1870492997 reap_dead start 2023-10-05T19:28:26.873 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.866+0000 7f6a11c38e40 1 --2- 172.21.15.167:0/1870492997 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55875ee86e30 0x55875eeacb40 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7f6a00001350 tx=0x7f6a00014000 comp rx=0 tx=0).stop 2023-10-05T19:28:26.873 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.866+0000 7f6a11c38e40 1 -- 172.21.15.167:0/1870492997 shutdown_connections 2023-10-05T19:28:26.873 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.866+0000 7f6a11c38e40 1 --2- 172.21.15.167:0/1870492997 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55875ee86e30 0x55875eeacb40 unknown :-1 s=CLOSED pgs=215 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-05T19:28:26.873 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.866+0000 7f6a11c38e40 1 -- 172.21.15.167:0/1870492997 >> 172.21.15.167:0/1870492997 conn(0x55875ee82850 msgr2=0x55875eea03a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-05T19:28:26.873 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.866+0000 7f6a11c38e40 1 -- 172.21.15.167:0/1870492997 shutdown_connections 2023-10-05T19:28:26.873 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.866+0000 7f6a11c38e40 1 -- 172.21.15.167:0/1870492997 wait complete. 2023-10-05T19:28:26.874 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.866+0000 7f6a11c38e40 1 -- 172.21.15.167:0/467276925 >> [v2:172.21.15.167:6824/29889,v1:172.21.15.167:6825/29889] conn(0x7f6950030c50 msgr2=0x7f6950033100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-05T19:28:26.874 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.866+0000 7f6a11c38e40 1 --2- 172.21.15.167:0/467276925 >> [v2:172.21.15.167:6824/29889,v1:172.21.15.167:6825/29889] conn(0x7f6950030c50 0x7f6950033100 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x55875eec4120 tx=0x7f6a08000f90 comp rx=0 tx=0).stop 2023-10-05T19:28:26.874 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.866+0000 7f6a11c38e40 1 -- 172.21.15.167:0/467276925 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55875eed4e90 msgr2=0x55875eeda950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-05T19:28:26.874 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.866+0000 7f6a11c38e40 1 --2- 172.21.15.167:0/467276925 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55875eed4e90 0x55875eeda950 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x55875eecef10 tx=0x7f69f8039000 comp rx=0 tx=0).stop 2023-10-05T19:28:26.875 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.866+0000 7f6a11c38e40 1 -- 172.21.15.167:0/467276925 shutdown_connections 2023-10-05T19:28:26.875 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.866+0000 7f6a11c38e40 1 --2- 172.21.15.167:0/467276925 >> [v2:172.21.15.167:6824/29889,v1:172.21.15.167:6825/29889] conn(0x7f6950030c50 0x7f6950033100 unknown :-1 s=CLOSED pgs=79 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-05T19:28:26.875 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.866+0000 7f6a11c38e40 1 --2- 172.21.15.167:0/467276925 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55875eed4e90 0x55875eeda950 unknown :-1 s=CLOSED pgs=217 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-05T19:28:26.875 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.866+0000 7f6a11c38e40 1 -- 172.21.15.167:0/467276925 >> 172.21.15.167:0/467276925 conn(0x55875eed0250 msgr2=0x55875eece280 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-05T19:28:26.875 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.866+0000 7f6a11c38e40 1 -- 172.21.15.167:0/467276925 shutdown_connections 2023-10-05T19:28:26.875 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.866+0000 7f6a11c38e40 1 -- 172.21.15.167:0/467276925 wait complete. 2023-10-05T19:28:26.879 DEBUG:tasks.util.rgw: json result: {'user_id': 'foo', 'display_name': 'Foo', 'email': 'foo@foo.com', 'suspended': 0, 'max_buckets': 4, 'subusers': [], 'keys': [{'user': 'foo', 'access_key': '9te6NH5mcdcq0Tc5i8i1', 'secret_key': 'Ny4IOauQoL18Gp2zM7lC1vLmoawgqcYP/YGcWfXu'}], 'swift_keys': [], 'caps': [], 'op_mask': 'read, write, delete', 'default_placement': '', 'default_storage_class': '', 'placement_tags': [], 'bucket_quota': {'enabled': False, 'check_on_raw': False, 'max_size': -1, 'max_size_kb': 0, 'max_objects': -1}, 'user_quota': {'enabled': False, 'check_on_raw': False, 'max_size': -1, 'max_size_kb': 0, 'max_objects': -1}, 'temp_url_keys': [], 'type': 'rgw', 'mfa_ids': []} 2023-10-05T19:28:26.879 INFO:tasks.util.rgw:rgwadmin: client.0 : ['user', 'info', '--uid', 'foo'] 2023-10-05T19:28:26.879 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'ceph', 'user', 'info', '--uid', 'foo'] 2023-10-05T19:28:26.879 DEBUG:teuthology.orchestra.run.smithi167:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster ceph user info --uid foo 2023-10-05T19:28:26.961 INFO:teuthology.orchestra.run.smithi167.stderr:ignoring --setuser ceph since I am not root 2023-10-05T19:28:26.962 INFO:teuthology.orchestra.run.smithi167.stderr:ignoring --setgroup ceph since I am not root 2023-10-05T19:28:26.962 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.958+0000 7f6b1aa70e40 1 Processor -- start 2023-10-05T19:28:26.962 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.958+0000 7f6b1aa70e40 1 -- start start 2023-10-05T19:28:26.962 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.958+0000 7f6b1aa70e40 1 --2- >> v2:172.21.15.118:3300/0 conn(0x5573f47694b0 0x5573f4767a90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-05T19:28:26.963 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.958+0000 7f6b1aa70e40 1 -- --> v1:172.21.15.118:6789/0 -- auth(proto 0 26 bytes epoch 0) v1 -- 0x5573f473fbd0 con 0x5573f4767fd0 2023-10-05T19:28:26.963 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.958+0000 7f6b1aa70e40 1 -- --> v2:172.21.15.118:3300/0 -- mon_getmap magic: 0 v1 -- 0x5573f46fcee0 con 0x5573f47694b0 2023-10-05T19:28:26.964 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.958+0000 7f6b185a5700 1 --2- >> v2:172.21.15.118:3300/0 conn(0x5573f47694b0 0x5573f4767a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-05T19:28:26.964 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.958+0000 7f6b17da4700 1 --1- >> v1:172.21.15.118:6789/0 conn(0x5573f4767fd0 0x5573f47faf30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.118:6789/0 says I am v1:172.21.15.167:56418/0 (socket says 172.21.15.167:56418) 2023-10-05T19:28:26.964 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.958+0000 7f6b17da4700 1 -- 172.21.15.167:0/2557124705 learned_addr learned my addr 172.21.15.167:0/2557124705 (peer_addr_for_me v1:172.21.15.167:0/0) 2023-10-05T19:28:26.964 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.958+0000 7f6b175a3700 1 -- 172.21.15.167:0/2557124705 <== mon.0 v1:172.21.15.118:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4133749604 0 0) 0x5573f473fbd0 con 0x5573f4767fd0 2023-10-05T19:28:26.964 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.958+0000 7f6b175a3700 1 -- 172.21.15.167:0/2557124705 --> v1:172.21.15.118:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6b00001e10 con 0x5573f4767fd0 2023-10-05T19:28:26.964 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.958+0000 7f6b185a5700 1 -- 172.21.15.167:0/2557124705 >> v1:172.21.15.118:6789/0 conn(0x5573f4767fd0 legacy=0x5573f47faf30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-05T19:28:26.964 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.958+0000 7f6b185a5700 1 -- 172.21.15.167:0/2557124705 --> v2:172.21.15.118:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5573f4769940 con 0x5573f47694b0 2023-10-05T19:28:26.965 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.958+0000 7f6b185a5700 1 --2- 172.21.15.167:0/2557124705 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5573f47694b0 0x5573f4767a90 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7f6b0800a5d0 tx=0x7f6b08005a50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ed58deac15a62750 server_cookie=0 in_seq=0 out_seq=0 2023-10-05T19:28:26.965 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.958+0000 7f6b175a3700 1 -- 172.21.15.167:0/2557124705 <== mon.0 v2:172.21.15.118:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f6b08007610 con 0x5573f47694b0 2023-10-05T19:28:26.965 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.958+0000 7f6b175a3700 1 -- 172.21.15.167:0/2557124705 <== mon.0 v2:172.21.15.118:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6b08007770 con 0x5573f47694b0 2023-10-05T19:28:26.965 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.958+0000 7f6b175a3700 1 -- 172.21.15.167:0/2557124705 <== mon.0 v2:172.21.15.118:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f6b08007a40 con 0x5573f47694b0 2023-10-05T19:28:26.966 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.958+0000 7f6b1aa70e40 1 -- 172.21.15.167:0/2557124705 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5573f47694b0 msgr2=0x5573f4767a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-05T19:28:26.966 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.958+0000 7f6b1aa70e40 1 --2- 172.21.15.167:0/2557124705 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5573f47694b0 0x5573f4767a90 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7f6b0800a5d0 tx=0x7f6b08005a50 comp rx=0 tx=0).stop 2023-10-05T19:28:26.966 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.958+0000 7f6b1aa70e40 1 -- 172.21.15.167:0/2557124705 shutdown_connections 2023-10-05T19:28:26.966 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.958+0000 7f6b1aa70e40 1 --2- 172.21.15.167:0/2557124705 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5573f47694b0 0x5573f4767a90 unknown :-1 s=CLOSED pgs=219 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-05T19:28:26.966 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.958+0000 7f6b1aa70e40 1 -- 172.21.15.167:0/2557124705 >> 172.21.15.167:0/2557124705 conn(0x5573f47ee830 msgr2=0x5573f47f0c40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-05T19:28:26.966 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.958+0000 7f6b1aa70e40 1 -- 172.21.15.167:0/2557124705 shutdown_connections 2023-10-05T19:28:26.966 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.958+0000 7f6b1aa70e40 1 -- 172.21.15.167:0/2557124705 wait complete. 2023-10-05T19:28:26.967 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.962+0000 7f6b1aa70e40 1 Processor -- start 2023-10-05T19:28:26.967 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.962+0000 7f6b1aa70e40 1 -- start start 2023-10-05T19:28:26.967 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.962+0000 7f6b1aa70e40 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5573f4767fd0 0x5573f4813f50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-05T19:28:26.968 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.962+0000 7f6b1aa70e40 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x5573f46c0e30 con 0x5573f4767fd0 2023-10-05T19:28:26.968 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.962+0000 7f6b185a5700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5573f4767fd0 0x5573f4813f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-05T19:28:26.968 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.962+0000 7f6b185a5700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5573f4767fd0 0x5573f4813f50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.118:3300/0 says I am v2:172.21.15.167:57432/0 (socket says 172.21.15.167:57432) 2023-10-05T19:28:26.968 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.962+0000 7f6b185a5700 1 -- 172.21.15.167:0/2749475858 learned_addr learned my addr 172.21.15.167:0/2749475858 (peer_addr_for_me v2:172.21.15.167:0/0) 2023-10-05T19:28:26.968 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.962+0000 7f6b185a5700 1 -- 172.21.15.167:0/2749475858 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6b08005cf0 con 0x5573f4767fd0 2023-10-05T19:28:26.968 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.962+0000 7f6b185a5700 1 --2- 172.21.15.167:0/2749475858 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5573f4767fd0 0x5573f4813f50 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7f6b08000c00 tx=0x7f6b08014000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=71324838737f8630 server_cookie=0 in_seq=0 out_seq=0 2023-10-05T19:28:26.969 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.962+0000 7f6b165a1700 1 -- 172.21.15.167:0/2749475858 <== mon.0 v2:172.21.15.118:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f6b080078e0 con 0x5573f4767fd0 2023-10-05T19:28:26.969 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.962+0000 7f6b165a1700 1 -- 172.21.15.167:0/2749475858 <== mon.0 v2:172.21.15.118:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6b08007a40 con 0x5573f4767fd0 2023-10-05T19:28:26.969 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.962+0000 7f6b165a1700 1 -- 172.21.15.167:0/2749475858 <== mon.0 v2:172.21.15.118:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f6b08007bd0 con 0x5573f4767fd0 2023-10-05T19:28:26.969 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.962+0000 7f6b1aa70e40 1 -- 172.21.15.167:0/2749475858 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5573f4767fd0 msgr2=0x5573f4813f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-05T19:28:26.970 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.962+0000 7f6b1aa70e40 1 --2- 172.21.15.167:0/2749475858 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5573f4767fd0 0x5573f4813f50 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7f6b08000c00 tx=0x7f6b08014000 comp rx=0 tx=0).stop 2023-10-05T19:28:26.970 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.962+0000 7f6b1aa70e40 1 -- 172.21.15.167:0/2749475858 shutdown_connections 2023-10-05T19:28:26.970 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.962+0000 7f6b1aa70e40 1 --2- 172.21.15.167:0/2749475858 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5573f4767fd0 0x5573f4813f50 unknown :-1 s=CLOSED pgs=220 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-05T19:28:26.970 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.962+0000 7f6b1aa70e40 1 -- 172.21.15.167:0/2749475858 >> 172.21.15.167:0/2749475858 conn(0x5573f47ee830 msgr2=0x5573f480fb10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-05T19:28:26.970 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.962+0000 7f6b1aa70e40 1 -- 172.21.15.167:0/2749475858 shutdown_connections 2023-10-05T19:28:26.970 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.962+0000 7f6b1aa70e40 1 -- 172.21.15.167:0/2749475858 wait complete. 2023-10-05T19:28:26.970 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.966+0000 7f6b1aa70e40 1 Processor -- start 2023-10-05T19:28:26.971 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.966+0000 7f6b1aa70e40 1 -- start start 2023-10-05T19:28:26.971 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.966+0000 7f6b1aa70e40 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5573f4767fd0 0x5573f481ac10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-05T19:28:26.971 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.966+0000 7f6b1aa70e40 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x5573f4693ed0 con 0x5573f4767fd0 2023-10-05T19:28:26.971 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.966+0000 7f6b185a5700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5573f4767fd0 0x5573f481ac10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-05T19:28:26.972 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.966+0000 7f6b185a5700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5573f4767fd0 0x5573f481ac10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.118:3300/0 says I am v2:172.21.15.167:57440/0 (socket says 172.21.15.167:57440) 2023-10-05T19:28:26.972 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.966+0000 7f6b185a5700 1 -- 172.21.15.167:0/1447055810 learned_addr learned my addr 172.21.15.167:0/1447055810 (peer_addr_for_me v2:172.21.15.167:0/0) 2023-10-05T19:28:26.973 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.966+0000 7f6b185a5700 1 -- 172.21.15.167:0/1447055810 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5573f4769940 con 0x5573f4767fd0 2023-10-05T19:28:26.973 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.966+0000 7f6b185a5700 1 --2- 172.21.15.167:0/1447055810 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5573f4767fd0 0x5573f481ac10 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7f6b0800afd0 tx=0x7f6b0801c000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-05T19:28:26.973 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.966+0000 7f6b14d9e700 1 -- 172.21.15.167:0/1447055810 <== mon.0 v2:172.21.15.118:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f6b08007e00 con 0x5573f4767fd0 2023-10-05T19:28:26.973 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.966+0000 7f6b14d9e700 1 -- 172.21.15.167:0/1447055810 <== mon.0 v2:172.21.15.118:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6b08007bb0 con 0x5573f4767fd0 2023-10-05T19:28:26.973 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.966+0000 7f6b1aa70e40 1 -- 172.21.15.167:0/1447055810 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x5573f481b150 con 0x5573f4767fd0 2023-10-05T19:28:26.973 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.966+0000 7f6b14d9e700 1 -- 172.21.15.167:0/1447055810 <== mon.0 v2:172.21.15.118:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f6b08026650 con 0x5573f4767fd0 2023-10-05T19:28:26.973 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.966+0000 7f6b1aa70e40 1 -- 172.21.15.167:0/1447055810 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x5573f481b590 con 0x5573f4767fd0 2023-10-05T19:28:26.974 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.970+0000 7f6b14d9e700 1 -- 172.21.15.167:0/1447055810 <== mon.0 v2:172.21.15.118:3300/0 4 ==== mgrmap(e 5) v1 ==== 40329+0+0 (secure 0 0 0) 0x7f6b080267b0 con 0x5573f4767fd0 2023-10-05T19:28:26.975 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.970+0000 7f6b14d9e700 1 --2- 172.21.15.167:0/1447055810 >> [v2:172.21.15.167:6824/29889,v1:172.21.15.167:6825/29889] conn(0x7f6b04030c40 0x7f6b040330f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-05T19:28:26.975 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.970+0000 7f6b14d9e700 1 -- 172.21.15.167:0/1447055810 <== mon.0 v2:172.21.15.118:3300/0 5 ==== osd_map(30..30 src has 1..30) v4 ==== 6057+0+0 (secure 0 0 0) 0x7f6b08046650 con 0x5573f4767fd0 2023-10-05T19:28:26.975 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.970+0000 7f6b1aa70e40 20 rados->read ofs=0 len=0 2023-10-05T19:28:26.975 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.970+0000 7f6b17da4700 1 --2- 172.21.15.167:0/1447055810 >> [v2:172.21.15.167:6824/29889,v1:172.21.15.167:6825/29889] conn(0x7f6b04030c40 0x7f6b040330f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-05T19:28:26.976 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.970+0000 7f6b1aa70e40 1 --2- 172.21.15.167:0/1447055810 >> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] conn(0x5573f47694b0 0x5573f48266a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-05T19:28:26.976 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.970+0000 7f6b1aa70e40 1 -- 172.21.15.167:0/1447055810 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:1 5.12 5:49953fa1:::default.realm:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x5573f4826d10 con 0x5573f47694b0 2023-10-05T19:28:26.976 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.970+0000 7f6b18da6700 1 --2- 172.21.15.167:0/1447055810 >> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] conn(0x5573f47694b0 0x5573f48266a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-05T19:28:26.976 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.970+0000 7f6b17da4700 1 --2- 172.21.15.167:0/1447055810 >> [v2:172.21.15.167:6824/29889,v1:172.21.15.167:6825/29889] conn(0x7f6b04030c40 0x7f6b040330f0 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f6b0c004170 tx=0x7f6b0c001400 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-05T19:28:26.976 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.970+0000 7f6b18da6700 1 --2- 172.21.15.167:0/1447055810 >> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] conn(0x5573f47694b0 0x5573f48266a0 crc :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-05T19:28:26.977 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.970+0000 7f6b18da6700 1 -- 172.21.15.167:0/1447055810 <== osd.1 v2:172.21.15.167:6800/30035 1 ==== osd_op_reply(1 default.realm [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 ==== 157+0+0 (crc 0 0 0) 0x7f6b1000a5a0 con 0x5573f47694b0 2023-10-05T19:28:26.977 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.970+0000 7f6b1aa70e40 20 rados_obj.operate() r=-2 bl.length=0 2023-10-05T19:28:26.977 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.970+0000 7f6b1aa70e40 20 realm 2023-10-05T19:28:26.977 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.970+0000 7f6b1aa70e40 20 rados->read ofs=0 len=0 2023-10-05T19:28:26.977 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.970+0000 7f6b1aa70e40 1 -- 172.21.15.167:0/1447055810 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:2 5.12 5:49953fa1:::default.realm:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x5573f4823d00 con 0x5573f47694b0 2023-10-05T19:28:26.978 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.970+0000 7f6b18da6700 1 -- 172.21.15.167:0/1447055810 <== osd.1 v2:172.21.15.167:6800/30035 2 ==== osd_op_reply(2 default.realm [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 ==== 157+0+0 (crc 0 0 0) 0x7f6b1000a5a0 con 0x5573f47694b0 2023-10-05T19:28:26.978 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.970+0000 7f6b1aa70e40 20 rados_obj.operate() r=-2 bl.length=0 2023-10-05T19:28:26.978 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.970+0000 7f6b1aa70e40 4 RGWPeriod::init failed to init realm id : (2) No such file or directory 2023-10-05T19:28:26.978 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.970+0000 7f6b1aa70e40 20 rados->read ofs=0 len=0 2023-10-05T19:28:26.979 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.970+0000 7f6b1aa70e40 1 -- 172.21.15.167:0/1447055810 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:3 5.12 5:49953fa1:::default.realm:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x5573f48277f0 con 0x5573f47694b0 2023-10-05T19:28:26.979 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.974+0000 7f6b18da6700 1 -- 172.21.15.167:0/1447055810 <== osd.1 v2:172.21.15.167:6800/30035 3 ==== osd_op_reply(3 default.realm [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 ==== 157+0+0 (crc 0 0 0) 0x7f6b1000a5a0 con 0x5573f47694b0 2023-10-05T19:28:26.979 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.974+0000 7f6b1aa70e40 20 rados_obj.operate() r=-2 bl.length=0 2023-10-05T19:28:26.979 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.974+0000 7f6b1aa70e40 20 rados->read ofs=0 len=0 2023-10-05T19:28:26.979 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.974+0000 7f6b1aa70e40 1 -- 172.21.15.167:0/1447055810 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:4 5.f 5:f43b6ece:::zone_names.default:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x5573f48273d0 con 0x5573f47694b0 2023-10-05T19:28:26.980 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.974+0000 7f6b18da6700 1 -- 172.21.15.167:0/1447055810 <== osd.1 v2:172.21.15.167:6800/30035 4 ==== osd_op_reply(4 zone_names.default [read 0~46 out=46b] v0'0 uv1 ondisk = 0) v8 ==== 162+0+46 (crc 0 0 0) 0x7f6b1000a5a0 con 0x5573f47694b0 2023-10-05T19:28:26.980 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.974+0000 7f6b1aa70e40 20 rados_obj.operate() r=0 bl.length=46 2023-10-05T19:28:26.980 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.974+0000 7f6b1aa70e40 20 rados->read ofs=0 len=0 2023-10-05T19:28:26.980 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.974+0000 7f6b1aa70e40 1 -- 172.21.15.167:0/1447055810 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:5 5.1d 5:b8374eb4:::zone_info.9a9df965-6612-45d4-a45f-6e3c889bacc2:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x5573f4828260 con 0x5573f47694b0 2023-10-05T19:28:26.980 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.974+0000 7f6b18da6700 1 -- 172.21.15.167:0/1447055810 <== osd.1 v2:172.21.15.167:6800/30035 5 ==== osd_op_reply(5 zone_info.9a9df965-6612-45d4-a45f-6e3c889bacc2 [read 0~874 out=874b] v0'0 uv2 ondisk = 0) v8 ==== 190+0+874 (crc 0 0 0) 0x7f6b1000a5a0 con 0x5573f47694b0 2023-10-05T19:28:26.980 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.974+0000 7f6b1aa70e40 20 rados_obj.operate() r=0 bl.length=874 2023-10-05T19:28:26.980 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.974+0000 7f6b1aa70e40 20 searching for the correct realm 2023-10-05T19:28:26.981 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.974+0000 7f6b1aa70e40 1 --2- 172.21.15.167:0/1447055810 >> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] conn(0x5573f482a0b0 0x5573f482c490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-05T19:28:26.981 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.974+0000 7f6b1aa70e40 1 -- 172.21.15.167:0/1447055810 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:6 5.0 5:00000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x5573f482c9d0 con 0x5573f482a0b0 2023-10-05T19:28:26.981 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.974+0000 7f6b185a5700 1 --2- 172.21.15.167:0/1447055810 >> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] conn(0x5573f482a0b0 0x5573f482c490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-05T19:28:26.981 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.974+0000 7f6b185a5700 1 --2- 172.21.15.167:0/1447055810 >> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] conn(0x5573f482a0b0 0x5573f482c490 crc :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-05T19:28:26.982 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.978+0000 7f6b185a5700 1 -- 172.21.15.167:0/1447055810 <== osd.3 v2:172.21.15.167:6808/30036 1 ==== osd_op_reply(6 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f6b0801c7f0 con 0x5573f482a0b0 2023-10-05T19:28:26.982 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.978+0000 7f6b185a5700 1 --2- 172.21.15.167:0/1447055810 >> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] conn(0x7f6b08038060 0x7f6b08025710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-05T19:28:26.982 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.978+0000 7f6b185a5700 1 -- 172.21.15.167:0/1447055810 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:7 5.10 5:08000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x5573f482c9d0 con 0x7f6b08038060 2023-10-05T19:28:26.982 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.978+0000 7f6b17da4700 1 --2- 172.21.15.167:0/1447055810 >> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] conn(0x7f6b08038060 0x7f6b08025710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-05T19:28:26.983 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.978+0000 7f6b17da4700 1 --2- 172.21.15.167:0/1447055810 >> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] conn(0x7f6b08038060 0x7f6b08025710 crc :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-05T19:28:26.983 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.978+0000 7f6b17da4700 1 -- 172.21.15.167:0/1447055810 <== osd.2 v2:172.21.15.167:6816/30037 1 ==== osd_op_reply(7 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f6b0c006070 con 0x7f6b08038060 2023-10-05T19:28:26.983 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.978+0000 7f6b17da4700 1 --2- 172.21.15.167:0/1447055810 >> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] conn(0x7f6b0c00eb60 0x7f6b0c00d8e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-05T19:28:26.983 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.978+0000 7f6b17da4700 1 -- 172.21.15.167:0/1447055810 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:8 5.8 5:10000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x5573f482c9d0 con 0x7f6b0c00eb60 2023-10-05T19:28:26.984 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.978+0000 7f6b18da6700 1 --2- 172.21.15.167:0/1447055810 >> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] conn(0x7f6b0c00eb60 0x7f6b0c00d8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-05T19:28:26.984 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.978+0000 7f6b18da6700 1 --2- 172.21.15.167:0/1447055810 >> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] conn(0x7f6b0c00eb60 0x7f6b0c00d8e0 crc :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-05T19:28:26.985 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.978+0000 7f6b18da6700 1 -- 172.21.15.167:0/1447055810 <== osd.0 v2:172.21.15.118:6800/27690 1 ==== osd_op_reply(8 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f6b1000a5a0 con 0x7f6b0c00eb60 2023-10-05T19:28:26.985 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.978+0000 7f6b18da6700 1 -- 172.21.15.167:0/1447055810 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:9 5.18 5:18000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x5573f482c9d0 con 0x5573f47694b0 2023-10-05T19:28:26.986 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.982+0000 7f6b18da6700 1 -- 172.21.15.167:0/1447055810 <== osd.1 v2:172.21.15.167:6800/30035 6 ==== osd_op_reply(9 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f6b1000a5a0 con 0x5573f47694b0 2023-10-05T19:28:26.986 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.982+0000 7f6b18da6700 1 -- 172.21.15.167:0/1447055810 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:10 5.4 5:20000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x5573f482c9d0 con 0x5573f47694b0 2023-10-05T19:28:26.986 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.982+0000 7f6b18da6700 1 -- 172.21.15.167:0/1447055810 <== osd.1 v2:172.21.15.167:6800/30035 7 ==== osd_op_reply(10 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f6b1000a5a0 con 0x5573f47694b0 2023-10-05T19:28:26.987 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.982+0000 7f6b18da6700 1 -- 172.21.15.167:0/1447055810 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:11 5.14 5:28000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x5573f482c9d0 con 0x7f6b0c00eb60 2023-10-05T19:28:26.988 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.982+0000 7f6b18da6700 1 -- 172.21.15.167:0/1447055810 <== osd.0 v2:172.21.15.118:6800/27690 2 ==== osd_op_reply(11 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f6b1000a5a0 con 0x7f6b0c00eb60 2023-10-05T19:28:26.988 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.982+0000 7f6b18da6700 1 -- 172.21.15.167:0/1447055810 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:12 5.c 5:30000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x5573f482c9d0 con 0x7f6b0c00eb60 2023-10-05T19:28:26.989 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.982+0000 7f6b18da6700 1 -- 172.21.15.167:0/1447055810 <== osd.0 v2:172.21.15.118:6800/27690 3 ==== osd_op_reply(12 [pgnls start_epoch 30 out=112b] v23'1 uv1 ondisk = 1) v8 ==== 144+0+112 (crc 0 0 0) 0x7f6b1000a5a0 con 0x7f6b0c00eb60 2023-10-05T19:28:26.989 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.982+0000 7f6b18da6700 1 -- 172.21.15.167:0/1447055810 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:13 5.1c 5:38000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x5573f482c9d0 con 0x7f6b08038060 2023-10-05T19:28:26.989 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.982+0000 7f6b17da4700 1 -- 172.21.15.167:0/1447055810 <== osd.2 v2:172.21.15.167:6816/30037 2 ==== osd_op_reply(13 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f6b0c006070 con 0x7f6b08038060 2023-10-05T19:28:26.990 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.982+0000 7f6b17da4700 1 -- 172.21.15.167:0/1447055810 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:14 5.2 5:40000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x5573f482c9d0 con 0x5573f482a0b0 2023-10-05T19:28:26.990 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.986+0000 7f6b185a5700 1 -- 172.21.15.167:0/1447055810 <== osd.3 v2:172.21.15.167:6808/30036 2 ==== osd_op_reply(14 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f6b0801c7f0 con 0x5573f482a0b0 2023-10-05T19:28:26.990 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.986+0000 7f6b185a5700 1 -- 172.21.15.167:0/1447055810 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:15 5.12 5:48000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x5573f482c9d0 con 0x5573f47694b0 2023-10-05T19:28:26.991 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.986+0000 7f6b18da6700 1 -- 172.21.15.167:0/1447055810 <== osd.1 v2:172.21.15.167:6800/30035 8 ==== osd_op_reply(15 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f6b1000a5a0 con 0x5573f47694b0 2023-10-05T19:28:26.991 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.986+0000 7f6b18da6700 1 -- 172.21.15.167:0/1447055810 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:16 5.a 5:50000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x5573f482c9d0 con 0x7f6b0c00eb60 2023-10-05T19:28:26.992 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.986+0000 7f6b18da6700 1 -- 172.21.15.167:0/1447055810 <== osd.0 v2:172.21.15.118:6800/27690 4 ==== osd_op_reply(16 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f6b1000a5a0 con 0x7f6b0c00eb60 2023-10-05T19:28:26.992 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.986+0000 7f6b18da6700 1 -- 172.21.15.167:0/1447055810 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:17 5.1a 5:58000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x5573f482c9d0 con 0x5573f47694b0 2023-10-05T19:28:26.993 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.986+0000 7f6b18da6700 1 -- 172.21.15.167:0/1447055810 <== osd.1 v2:172.21.15.167:6800/30035 9 ==== osd_op_reply(17 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f6b1000a5a0 con 0x5573f47694b0 2023-10-05T19:28:26.993 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.986+0000 7f6b18da6700 1 -- 172.21.15.167:0/1447055810 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:18 5.6 5:60000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x5573f482c9d0 con 0x7f6b08038060 2023-10-05T19:28:26.993 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.986+0000 7f6b17da4700 1 -- 172.21.15.167:0/1447055810 <== osd.2 v2:172.21.15.167:6816/30037 3 ==== osd_op_reply(18 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f6b0c006070 con 0x7f6b08038060 2023-10-05T19:28:26.993 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.986+0000 7f6b17da4700 1 -- 172.21.15.167:0/1447055810 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:19 5.16 5:68000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x5573f482c9d0 con 0x5573f482a0b0 2023-10-05T19:28:26.994 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.990+0000 7f6b185a5700 1 -- 172.21.15.167:0/1447055810 <== osd.3 v2:172.21.15.167:6808/30036 3 ==== osd_op_reply(19 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f6b0801c7f0 con 0x5573f482a0b0 2023-10-05T19:28:26.994 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.990+0000 7f6b185a5700 1 -- 172.21.15.167:0/1447055810 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:20 5.e 5:70000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x5573f482c9d0 con 0x5573f482a0b0 2023-10-05T19:28:26.995 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.990+0000 7f6b185a5700 1 -- 172.21.15.167:0/1447055810 <== osd.3 v2:172.21.15.167:6808/30036 4 ==== osd_op_reply(20 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f6b0801c7f0 con 0x5573f482a0b0 2023-10-05T19:28:26.995 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.990+0000 7f6b185a5700 1 -- 172.21.15.167:0/1447055810 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:21 5.1e 5:78000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x5573f482c9d0 con 0x7f6b0c00eb60 2023-10-05T19:28:26.996 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.990+0000 7f6b18da6700 1 -- 172.21.15.167:0/1447055810 <== osd.0 v2:172.21.15.118:6800/27690 5 ==== osd_op_reply(21 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f6b1000a5a0 con 0x7f6b0c00eb60 2023-10-05T19:28:26.996 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.990+0000 7f6b18da6700 1 -- 172.21.15.167:0/1447055810 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:22 5.1 5:80000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x5573f482c9d0 con 0x5573f482a0b0 2023-10-05T19:28:26.996 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.990+0000 7f6b185a5700 1 -- 172.21.15.167:0/1447055810 <== osd.3 v2:172.21.15.167:6808/30036 5 ==== osd_op_reply(22 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f6b0801c7f0 con 0x5573f482a0b0 2023-10-05T19:28:26.997 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.990+0000 7f6b185a5700 1 -- 172.21.15.167:0/1447055810 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:23 5.11 5:88000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x5573f482c9d0 con 0x5573f482a0b0 2023-10-05T19:28:26.997 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.990+0000 7f6b185a5700 1 -- 172.21.15.167:0/1447055810 <== osd.3 v2:172.21.15.167:6808/30036 6 ==== osd_op_reply(23 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f6b0801c7f0 con 0x5573f482a0b0 2023-10-05T19:28:26.997 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.990+0000 7f6b185a5700 1 -- 172.21.15.167:0/1447055810 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:24 5.9 5:90000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x5573f482c9d0 con 0x5573f47694b0 2023-10-05T19:28:26.998 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.994+0000 7f6b18da6700 1 -- 172.21.15.167:0/1447055810 <== osd.1 v2:172.21.15.167:6800/30035 10 ==== osd_op_reply(24 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f6b1000a5a0 con 0x5573f47694b0 2023-10-05T19:28:26.998 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.994+0000 7f6b18da6700 1 -- 172.21.15.167:0/1447055810 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:25 5.19 5:98000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x5573f482c9d0 con 0x5573f47694b0 2023-10-05T19:28:26.999 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.994+0000 7f6b18da6700 1 -- 172.21.15.167:0/1447055810 <== osd.1 v2:172.21.15.167:6800/30035 11 ==== osd_op_reply(25 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f6b1000a5a0 con 0x5573f47694b0 2023-10-05T19:28:26.999 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.994+0000 7f6b18da6700 1 -- 172.21.15.167:0/1447055810 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:26 5.5 5:a0000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x5573f482c9d0 con 0x7f6b0c00eb60 2023-10-05T19:28:27.000 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.994+0000 7f6b18da6700 1 -- 172.21.15.167:0/1447055810 <== osd.0 v2:172.21.15.118:6800/27690 6 ==== osd_op_reply(26 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f6b1000a5a0 con 0x7f6b0c00eb60 2023-10-05T19:28:27.000 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.994+0000 7f6b18da6700 1 -- 172.21.15.167:0/1447055810 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:27 5.15 5:a8000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x5573f482c9d0 con 0x5573f482a0b0 2023-10-05T19:28:27.001 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.994+0000 7f6b185a5700 1 -- 172.21.15.167:0/1447055810 <== osd.3 v2:172.21.15.167:6808/30036 7 ==== osd_op_reply(27 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f6b0801c7f0 con 0x5573f482a0b0 2023-10-05T19:28:27.001 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.994+0000 7f6b185a5700 1 -- 172.21.15.167:0/1447055810 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:28 5.d 5:b0000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x5573f482c9d0 con 0x7f6b08038060 2023-10-05T19:28:27.001 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.994+0000 7f6b17da4700 1 -- 172.21.15.167:0/1447055810 <== osd.2 v2:172.21.15.167:6816/30037 4 ==== osd_op_reply(28 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f6b0c006070 con 0x7f6b08038060 2023-10-05T19:28:27.001 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.994+0000 7f6b17da4700 1 -- 172.21.15.167:0/1447055810 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:29 5.1d 5:b8000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x5573f482c9d0 con 0x5573f47694b0 2023-10-05T19:28:27.002 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.998+0000 7f6b18da6700 1 -- 172.21.15.167:0/1447055810 <== osd.1 v2:172.21.15.167:6800/30035 12 ==== osd_op_reply(29 [pgnls start_epoch 30 out=107b] v30'2 uv2 ondisk = 1) v8 ==== 144+0+107 (crc 0 0 0) 0x7f6b1000a5a0 con 0x5573f47694b0 2023-10-05T19:28:27.002 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.998+0000 7f6b18da6700 1 -- 172.21.15.167:0/1447055810 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:30 5.3 5:c0000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x5573f482c9d0 con 0x7f6b0c00eb60 2023-10-05T19:28:27.003 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.998+0000 7f6b18da6700 1 -- 172.21.15.167:0/1447055810 <== osd.0 v2:172.21.15.118:6800/27690 7 ==== osd_op_reply(30 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f6b1000a5a0 con 0x7f6b0c00eb60 2023-10-05T19:28:27.004 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.998+0000 7f6b18da6700 1 -- 172.21.15.167:0/1447055810 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:31 5.13 5:c8000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x5573f482c9d0 con 0x5573f482a0b0 2023-10-05T19:28:27.004 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.998+0000 7f6b185a5700 1 -- 172.21.15.167:0/1447055810 <== osd.3 v2:172.21.15.167:6808/30036 8 ==== osd_op_reply(31 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f6b0801c7f0 con 0x5573f482a0b0 2023-10-05T19:28:27.004 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.998+0000 7f6b185a5700 1 -- 172.21.15.167:0/1447055810 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:32 5.b 5:d0000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x5573f482c9d0 con 0x7f6b0c00eb60 2023-10-05T19:28:27.005 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.998+0000 7f6b18da6700 1 -- 172.21.15.167:0/1447055810 <== osd.0 v2:172.21.15.118:6800/27690 8 ==== osd_op_reply(32 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f6b1000a5a0 con 0x7f6b0c00eb60 2023-10-05T19:28:27.005 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:26.998+0000 7f6b18da6700 1 -- 172.21.15.167:0/1447055810 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:33 5.1b 5:d8000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x5573f482c9d0 con 0x7f6b08038060 2023-10-05T19:28:27.006 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.002+0000 7f6b17da4700 1 -- 172.21.15.167:0/1447055810 <== osd.2 v2:172.21.15.167:6816/30037 5 ==== osd_op_reply(33 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f6b0c006070 con 0x7f6b08038060 2023-10-05T19:28:27.006 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.002+0000 7f6b17da4700 1 -- 172.21.15.167:0/1447055810 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:34 5.7 5:e0000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x5573f482c9d0 con 0x5573f482a0b0 2023-10-05T19:28:27.006 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.002+0000 7f6b185a5700 1 -- 172.21.15.167:0/1447055810 <== osd.3 v2:172.21.15.167:6808/30036 9 ==== osd_op_reply(34 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f6b0801c7f0 con 0x5573f482a0b0 2023-10-05T19:28:27.007 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.002+0000 7f6b185a5700 1 -- 172.21.15.167:0/1447055810 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:35 5.17 5:e8000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x5573f482c9d0 con 0x7f6b0c00eb60 2023-10-05T19:28:27.007 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.002+0000 7f6b18da6700 1 -- 172.21.15.167:0/1447055810 <== osd.0 v2:172.21.15.118:6800/27690 9 ==== osd_op_reply(35 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f6b1000a5a0 con 0x7f6b0c00eb60 2023-10-05T19:28:27.008 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.002+0000 7f6b18da6700 1 -- 172.21.15.167:0/1447055810 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:36 5.f 5:f0000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x5573f482c9d0 con 0x5573f47694b0 2023-10-05T19:28:27.008 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.002+0000 7f6b18da6700 1 -- 172.21.15.167:0/1447055810 <== osd.1 v2:172.21.15.167:6800/30035 13 ==== osd_op_reply(36 [pgnls start_epoch 30 out=115b] v23'2 uv2 ondisk = 1) v8 ==== 144+0+115 (crc 0 0 0) 0x7f6b1000a5a0 con 0x5573f47694b0 2023-10-05T19:28:27.009 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.002+0000 7f6b18da6700 1 -- 172.21.15.167:0/1447055810 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:37 5.1f 5:f8000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x5573f482c9d0 con 0x7f6b08038060 2023-10-05T19:28:27.009 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.002+0000 7f6b17da4700 1 -- 172.21.15.167:0/1447055810 <== osd.2 v2:172.21.15.167:6816/30037 6 ==== osd_op_reply(37 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f6b0c006070 con 0x7f6b08038060 2023-10-05T19:28:27.010 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.002+0000 7f6b1aa70e40 20 RGWRados::pool_iterate: got zonegroup_info.4f1f719d-9648-417b-b10c-e5d939825419 2023-10-05T19:28:27.010 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.002+0000 7f6b1aa70e40 20 RGWRados::pool_iterate: got zone_info.9a9df965-6612-45d4-a45f-6e3c889bacc2 2023-10-05T19:28:27.010 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.002+0000 7f6b1aa70e40 20 RGWRados::pool_iterate: got zone_names.default 2023-10-05T19:28:27.010 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.002+0000 7f6b1aa70e40 20 RGWRados::pool_iterate: got zonegroups_names.default 2023-10-05T19:28:27.010 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.002+0000 7f6b1aa70e40 20 rados->read ofs=0 len=0 2023-10-05T19:28:27.010 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.002+0000 7f6b1aa70e40 1 -- 172.21.15.167:0/1447055810 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:38 5.12 5:49953fa1:::default.realm:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x5573f4828de0 con 0x5573f47694b0 2023-10-05T19:28:27.011 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.006+0000 7f6b18da6700 1 -- 172.21.15.167:0/1447055810 <== osd.1 v2:172.21.15.167:6800/30035 14 ==== osd_op_reply(38 default.realm [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 ==== 157+0+0 (crc 0 0 0) 0x7f6b1000a5a0 con 0x5573f47694b0 2023-10-05T19:28:27.011 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.006+0000 7f6b1aa70e40 20 rados_obj.operate() r=-2 bl.length=0 2023-10-05T19:28:27.011 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.006+0000 7f6b1aa70e40 20 rados->read ofs=0 len=0 2023-10-05T19:28:27.011 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.006+0000 7f6b1aa70e40 1 -- 172.21.15.167:0/1447055810 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:39 5.f 5:f4c53578:::zonegroups_names.default:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x5573f48289e0 con 0x5573f47694b0 2023-10-05T19:28:27.011 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.006+0000 7f6b18da6700 1 -- 172.21.15.167:0/1447055810 <== osd.1 v2:172.21.15.167:6800/30035 15 ==== osd_op_reply(39 zonegroups_names.default [read 0~46 out=46b] v0'0 uv2 ondisk = 0) v8 ==== 168+0+46 (crc 0 0 0) 0x7f6b1000a5a0 con 0x5573f47694b0 2023-10-05T19:28:27.011 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.006+0000 7f6b1aa70e40 20 rados_obj.operate() r=0 bl.length=46 2023-10-05T19:28:27.012 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.006+0000 7f6b1aa70e40 20 rados->read ofs=0 len=0 2023-10-05T19:28:27.012 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.006+0000 7f6b1aa70e40 1 -- 172.21.15.167:0/1447055810 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:40 5.c 5:30d299a5:::zonegroup_info.4f1f719d-9648-417b-b10c-e5d939825419:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x5573f48289b0 con 0x7f6b0c00eb60 2023-10-05T19:28:27.012 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.006+0000 7f6b18da6700 1 -- 172.21.15.167:0/1447055810 <== osd.0 v2:172.21.15.118:6800/27690 10 ==== osd_op_reply(40 zonegroup_info.4f1f719d-9648-417b-b10c-e5d939825419 [read 0~362 out=362b] v0'0 uv1 ondisk = 0) v8 ==== 195+0+362 (crc 0 0 0) 0x7f6b1000a5a0 con 0x7f6b0c00eb60 2023-10-05T19:28:27.012 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.006+0000 7f6b1aa70e40 20 rados_obj.operate() r=0 bl.length=362 2023-10-05T19:28:27.012 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.006+0000 7f6b1aa70e40 20 rados->read ofs=0 len=0 2023-10-05T19:28:27.013 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.006+0000 7f6b1aa70e40 1 -- 172.21.15.167:0/1447055810 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:41 5.13 5:cbbf6c60:::converted:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x5573f4829770 con 0x5573f482a0b0 2023-10-05T19:28:27.013 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.006+0000 7f6b185a5700 1 -- 172.21.15.167:0/1447055810 <== osd.3 v2:172.21.15.167:6808/30036 10 ==== osd_op_reply(41 converted [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 ==== 153+0+0 (crc 0 0 0) 0x7f6b0801c7f0 con 0x5573f482a0b0 2023-10-05T19:28:27.013 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.006+0000 7f6b1aa70e40 20 rados_obj.operate() r=-2 bl.length=0 2023-10-05T19:28:27.013 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.006+0000 7f6b1aa70e40 20 rados->read ofs=0 len=0 2023-10-05T19:28:27.013 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.006+0000 7f6b1aa70e40 1 -- 172.21.15.167:0/1447055810 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:42 5.12 5:49953fa1:::default.realm:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x5573f482ce10 con 0x5573f47694b0 2023-10-05T19:28:27.014 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.006+0000 7f6b18da6700 1 -- 172.21.15.167:0/1447055810 <== osd.1 v2:172.21.15.167:6800/30035 16 ==== osd_op_reply(42 default.realm [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 ==== 157+0+0 (crc 0 0 0) 0x7f6b1000a5a0 con 0x5573f47694b0 2023-10-05T19:28:27.014 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.010+0000 7f6b1aa70e40 20 rados_obj.operate() r=-2 bl.length=0 2023-10-05T19:28:27.014 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.010+0000 7f6b1aa70e40 20 rados->read ofs=0 len=0 2023-10-05T19:28:27.014 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.010+0000 7f6b1aa70e40 1 -- 172.21.15.167:0/1447055810 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:43 5.f 5:f4c53578:::zonegroups_names.default:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x5573f482ce10 con 0x5573f47694b0 2023-10-05T19:28:27.015 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.010+0000 7f6b18da6700 1 -- 172.21.15.167:0/1447055810 <== osd.1 v2:172.21.15.167:6800/30035 17 ==== osd_op_reply(43 zonegroups_names.default [read 0~46 out=46b] v0'0 uv2 ondisk = 0) v8 ==== 168+0+46 (crc 0 0 0) 0x7f6b1000a5a0 con 0x5573f47694b0 2023-10-05T19:28:27.015 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.010+0000 7f6b1aa70e40 20 rados_obj.operate() r=0 bl.length=46 2023-10-05T19:28:27.015 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.010+0000 7f6b1aa70e40 1 -- 172.21.15.167:0/1447055810 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:44 5.0 5:00000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x5573f482de20 con 0x5573f482a0b0 2023-10-05T19:28:27.017 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.010+0000 7f6b185a5700 1 -- 172.21.15.167:0/1447055810 <== osd.3 v2:172.21.15.167:6808/30036 11 ==== osd_op_reply(44 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f6b0801c7f0 con 0x5573f482a0b0 2023-10-05T19:28:27.017 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.010+0000 7f6b185a5700 1 -- 172.21.15.167:0/1447055810 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:45 5.10 5:08000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x5573f482de20 con 0x7f6b08038060 2023-10-05T19:28:27.018 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.014+0000 7f6b17da4700 1 -- 172.21.15.167:0/1447055810 <== osd.2 v2:172.21.15.167:6816/30037 7 ==== osd_op_reply(45 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f6b0c006070 con 0x7f6b08038060 2023-10-05T19:28:27.018 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.014+0000 7f6b17da4700 1 -- 172.21.15.167:0/1447055810 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:46 5.8 5:10000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x5573f482de20 con 0x7f6b0c00eb60 2023-10-05T19:28:27.019 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.014+0000 7f6b18da6700 1 -- 172.21.15.167:0/1447055810 <== osd.0 v2:172.21.15.118:6800/27690 11 ==== osd_op_reply(46 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f6b1000a5a0 con 0x7f6b0c00eb60 2023-10-05T19:28:27.019 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.014+0000 7f6b18da6700 1 -- 172.21.15.167:0/1447055810 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:47 5.18 5:18000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x5573f4828de0 con 0x5573f47694b0 2023-10-05T19:28:27.020 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.014+0000 7f6b18da6700 1 -- 172.21.15.167:0/1447055810 <== osd.1 v2:172.21.15.167:6800/30035 18 ==== osd_op_reply(47 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f6b1000a5a0 con 0x5573f47694b0 2023-10-05T19:28:27.020 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.014+0000 7f6b18da6700 1 -- 172.21.15.167:0/1447055810 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:48 5.4 5:20000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x5573f4828de0 con 0x5573f47694b0 2023-10-05T19:28:27.020 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.014+0000 7f6b18da6700 1 -- 172.21.15.167:0/1447055810 <== osd.1 v2:172.21.15.167:6800/30035 19 ==== osd_op_reply(48 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f6b1000a5a0 con 0x5573f47694b0 2023-10-05T19:28:27.021 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.014+0000 7f6b18da6700 1 -- 172.21.15.167:0/1447055810 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:49 5.14 5:28000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x5573f4828de0 con 0x7f6b0c00eb60 2023-10-05T19:28:27.022 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.014+0000 7f6b18da6700 1 -- 172.21.15.167:0/1447055810 <== osd.0 v2:172.21.15.118:6800/27690 12 ==== osd_op_reply(49 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f6b1000a5a0 con 0x7f6b0c00eb60 2023-10-05T19:28:27.022 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.014+0000 7f6b18da6700 1 -- 172.21.15.167:0/1447055810 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:50 5.c 5:30000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x5573f4828de0 con 0x7f6b0c00eb60 2023-10-05T19:28:27.023 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.018+0000 7f6b18da6700 1 -- 172.21.15.167:0/1447055810 <== osd.0 v2:172.21.15.118:6800/27690 13 ==== osd_op_reply(50 [pgnls start_epoch 30 out=112b] v23'1 uv1 ondisk = 1) v8 ==== 144+0+112 (crc 0 0 0) 0x7f6b1000a5a0 con 0x7f6b0c00eb60 2023-10-05T19:28:27.023 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.018+0000 7f6b18da6700 1 -- 172.21.15.167:0/1447055810 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:51 5.1c 5:38000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x5573f4828de0 con 0x7f6b08038060 2023-10-05T19:28:27.024 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.018+0000 7f6b17da4700 1 -- 172.21.15.167:0/1447055810 <== osd.2 v2:172.21.15.167:6816/30037 8 ==== osd_op_reply(51 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f6b0c006070 con 0x7f6b08038060 2023-10-05T19:28:27.024 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.018+0000 7f6b17da4700 1 -- 172.21.15.167:0/1447055810 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:52 5.2 5:40000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x5573f4828de0 con 0x5573f482a0b0 2023-10-05T19:28:27.025 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.018+0000 7f6b185a5700 1 -- 172.21.15.167:0/1447055810 <== osd.3 v2:172.21.15.167:6808/30036 12 ==== osd_op_reply(52 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f6b0801c7f0 con 0x5573f482a0b0 2023-10-05T19:28:27.025 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.018+0000 7f6b185a5700 1 -- 172.21.15.167:0/1447055810 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:53 5.12 5:48000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x5573f4828de0 con 0x5573f47694b0 2023-10-05T19:28:27.025 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.018+0000 7f6b18da6700 1 -- 172.21.15.167:0/1447055810 <== osd.1 v2:172.21.15.167:6800/30035 20 ==== osd_op_reply(53 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f6b1000a5a0 con 0x5573f47694b0 2023-10-05T19:28:27.025 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.018+0000 7f6b18da6700 1 -- 172.21.15.167:0/1447055810 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:54 5.a 5:50000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x5573f4828de0 con 0x7f6b0c00eb60 2023-10-05T19:28:27.026 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.022+0000 7f6b18da6700 1 -- 172.21.15.167:0/1447055810 <== osd.0 v2:172.21.15.118:6800/27690 14 ==== osd_op_reply(54 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f6b1000a5a0 con 0x7f6b0c00eb60 2023-10-05T19:28:27.027 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.022+0000 7f6b18da6700 1 -- 172.21.15.167:0/1447055810 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:55 5.1a 5:58000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x5573f4828de0 con 0x5573f47694b0 2023-10-05T19:28:27.028 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.022+0000 7f6b18da6700 1 -- 172.21.15.167:0/1447055810 <== osd.1 v2:172.21.15.167:6800/30035 21 ==== osd_op_reply(55 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f6b1000a5a0 con 0x5573f47694b0 2023-10-05T19:28:27.028 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.022+0000 7f6b18da6700 1 -- 172.21.15.167:0/1447055810 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:56 5.6 5:60000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x5573f4828de0 con 0x7f6b08038060 2023-10-05T19:28:27.028 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.022+0000 7f6b17da4700 1 -- 172.21.15.167:0/1447055810 <== osd.2 v2:172.21.15.167:6816/30037 9 ==== osd_op_reply(56 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f6b0c006070 con 0x7f6b08038060 2023-10-05T19:28:27.028 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.022+0000 7f6b17da4700 1 -- 172.21.15.167:0/1447055810 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:57 5.16 5:68000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x5573f4828de0 con 0x5573f482a0b0 2023-10-05T19:28:27.029 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.022+0000 7f6b185a5700 1 -- 172.21.15.167:0/1447055810 <== osd.3 v2:172.21.15.167:6808/30036 13 ==== osd_op_reply(57 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f6b0801c7f0 con 0x5573f482a0b0 2023-10-05T19:28:27.029 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.022+0000 7f6b185a5700 1 -- 172.21.15.167:0/1447055810 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:58 5.e 5:70000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x5573f4828de0 con 0x5573f482a0b0 2023-10-05T19:28:27.030 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.026+0000 7f6b185a5700 1 -- 172.21.15.167:0/1447055810 <== osd.3 v2:172.21.15.167:6808/30036 14 ==== osd_op_reply(58 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f6b0801c7f0 con 0x5573f482a0b0 2023-10-05T19:28:27.030 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.026+0000 7f6b185a5700 1 -- 172.21.15.167:0/1447055810 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:59 5.1e 5:78000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x5573f4828de0 con 0x7f6b0c00eb60 2023-10-05T19:28:27.031 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.026+0000 7f6b18da6700 1 -- 172.21.15.167:0/1447055810 <== osd.0 v2:172.21.15.118:6800/27690 15 ==== osd_op_reply(59 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f6b1003f040 con 0x7f6b0c00eb60 2023-10-05T19:28:27.031 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.026+0000 7f6b18da6700 1 -- 172.21.15.167:0/1447055810 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:60 5.1 5:80000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x5573f4828de0 con 0x5573f482a0b0 2023-10-05T19:28:27.032 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.026+0000 7f6b185a5700 1 -- 172.21.15.167:0/1447055810 <== osd.3 v2:172.21.15.167:6808/30036 15 ==== osd_op_reply(60 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f6b0801c7f0 con 0x5573f482a0b0 2023-10-05T19:28:27.032 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.026+0000 7f6b185a5700 1 -- 172.21.15.167:0/1447055810 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:61 5.11 5:88000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x5573f4828de0 con 0x5573f482a0b0 2023-10-05T19:28:27.033 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.026+0000 7f6b185a5700 1 -- 172.21.15.167:0/1447055810 <== osd.3 v2:172.21.15.167:6808/30036 16 ==== osd_op_reply(61 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f6b0801c7f0 con 0x5573f482a0b0 2023-10-05T19:28:27.033 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.026+0000 7f6b185a5700 1 -- 172.21.15.167:0/1447055810 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:62 5.9 5:90000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x5573f4828de0 con 0x5573f47694b0 2023-10-05T19:28:27.034 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.030+0000 7f6b18da6700 1 -- 172.21.15.167:0/1447055810 <== osd.1 v2:172.21.15.167:6800/30035 22 ==== osd_op_reply(62 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f6b1003f040 con 0x5573f47694b0 2023-10-05T19:28:27.034 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.030+0000 7f6b18da6700 1 -- 172.21.15.167:0/1447055810 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:63 5.19 5:98000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x5573f4828de0 con 0x5573f47694b0 2023-10-05T19:28:27.035 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.030+0000 7f6b18da6700 1 -- 172.21.15.167:0/1447055810 <== osd.1 v2:172.21.15.167:6800/30035 23 ==== osd_op_reply(63 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f6b1003f040 con 0x5573f47694b0 2023-10-05T19:28:27.035 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.030+0000 7f6b18da6700 1 -- 172.21.15.167:0/1447055810 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:64 5.5 5:a0000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x5573f4828de0 con 0x7f6b0c00eb60 2023-10-05T19:28:27.036 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.030+0000 7f6b18da6700 1 -- 172.21.15.167:0/1447055810 <== osd.0 v2:172.21.15.118:6800/27690 16 ==== osd_op_reply(64 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f6b1003f040 con 0x7f6b0c00eb60 2023-10-05T19:28:27.036 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.030+0000 7f6b18da6700 1 -- 172.21.15.167:0/1447055810 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:65 5.15 5:a8000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x5573f4828de0 con 0x5573f482a0b0 2023-10-05T19:28:27.036 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.030+0000 7f6b185a5700 1 -- 172.21.15.167:0/1447055810 <== osd.3 v2:172.21.15.167:6808/30036 17 ==== osd_op_reply(65 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f6b0801c7f0 con 0x5573f482a0b0 2023-10-05T19:28:27.036 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.030+0000 7f6b185a5700 1 -- 172.21.15.167:0/1447055810 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:66 5.d 5:b0000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x5573f4828de0 con 0x7f6b08038060 2023-10-05T19:28:27.037 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.030+0000 7f6b17da4700 1 -- 172.21.15.167:0/1447055810 <== osd.2 v2:172.21.15.167:6816/30037 10 ==== osd_op_reply(66 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f6b0c006070 con 0x7f6b08038060 2023-10-05T19:28:27.037 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.030+0000 7f6b17da4700 1 -- 172.21.15.167:0/1447055810 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:67 5.1d 5:b8000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x5573f4828de0 con 0x5573f47694b0 2023-10-05T19:28:27.038 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.034+0000 7f6b18da6700 1 -- 172.21.15.167:0/1447055810 <== osd.1 v2:172.21.15.167:6800/30035 24 ==== osd_op_reply(67 [pgnls start_epoch 30 out=107b] v30'2 uv2 ondisk = 1) v8 ==== 144+0+107 (crc 0 0 0) 0x7f6b1003f040 con 0x5573f47694b0 2023-10-05T19:28:27.038 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.034+0000 7f6b18da6700 1 -- 172.21.15.167:0/1447055810 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:68 5.3 5:c0000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x5573f4828de0 con 0x7f6b0c00eb60 2023-10-05T19:28:27.039 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.034+0000 7f6b18da6700 1 -- 172.21.15.167:0/1447055810 <== osd.0 v2:172.21.15.118:6800/27690 17 ==== osd_op_reply(68 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f6b1003f040 con 0x7f6b0c00eb60 2023-10-05T19:28:27.039 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.034+0000 7f6b18da6700 1 -- 172.21.15.167:0/1447055810 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:69 5.13 5:c8000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x5573f4828de0 con 0x5573f482a0b0 2023-10-05T19:28:27.040 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.034+0000 7f6b185a5700 1 -- 172.21.15.167:0/1447055810 <== osd.3 v2:172.21.15.167:6808/30036 18 ==== osd_op_reply(69 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f6b0801c7f0 con 0x5573f482a0b0 2023-10-05T19:28:27.040 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.034+0000 7f6b185a5700 1 -- 172.21.15.167:0/1447055810 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:70 5.b 5:d0000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x5573f4828de0 con 0x7f6b0c00eb60 2023-10-05T19:28:27.041 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.034+0000 7f6b18da6700 1 -- 172.21.15.167:0/1447055810 <== osd.0 v2:172.21.15.118:6800/27690 18 ==== osd_op_reply(70 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f6b1003f040 con 0x7f6b0c00eb60 2023-10-05T19:28:27.041 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.034+0000 7f6b18da6700 1 -- 172.21.15.167:0/1447055810 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:71 5.1b 5:d8000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x5573f4828de0 con 0x7f6b08038060 2023-10-05T19:28:27.042 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.038+0000 7f6b17da4700 1 -- 172.21.15.167:0/1447055810 <== osd.2 v2:172.21.15.167:6816/30037 11 ==== osd_op_reply(71 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f6b0c006070 con 0x7f6b08038060 2023-10-05T19:28:27.042 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.038+0000 7f6b17da4700 1 -- 172.21.15.167:0/1447055810 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:72 5.7 5:e0000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x5573f4828de0 con 0x5573f482a0b0 2023-10-05T19:28:27.043 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.038+0000 7f6b185a5700 1 -- 172.21.15.167:0/1447055810 <== osd.3 v2:172.21.15.167:6808/30036 19 ==== osd_op_reply(72 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f6b0801c7f0 con 0x5573f482a0b0 2023-10-05T19:28:27.043 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.038+0000 7f6b185a5700 1 -- 172.21.15.167:0/1447055810 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:73 5.17 5:e8000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x5573f4828de0 con 0x7f6b0c00eb60 2023-10-05T19:28:27.044 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.038+0000 7f6b18da6700 1 -- 172.21.15.167:0/1447055810 <== osd.0 v2:172.21.15.118:6800/27690 19 ==== osd_op_reply(73 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f6b1003f040 con 0x7f6b0c00eb60 2023-10-05T19:28:27.044 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.038+0000 7f6b18da6700 1 -- 172.21.15.167:0/1447055810 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:74 5.f 5:f0000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x5573f4828de0 con 0x5573f47694b0 2023-10-05T19:28:27.045 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.038+0000 7f6b18da6700 1 -- 172.21.15.167:0/1447055810 <== osd.1 v2:172.21.15.167:6800/30035 25 ==== osd_op_reply(74 [pgnls start_epoch 30 out=115b] v23'2 uv2 ondisk = 1) v8 ==== 144+0+115 (crc 0 0 0) 0x7f6b1003f040 con 0x5573f47694b0 2023-10-05T19:28:27.045 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.038+0000 7f6b18da6700 1 -- 172.21.15.167:0/1447055810 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:75 5.1f 5:f8000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x5573f4828de0 con 0x7f6b08038060 2023-10-05T19:28:27.046 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.038+0000 7f6b17da4700 1 -- 172.21.15.167:0/1447055810 <== osd.2 v2:172.21.15.167:6816/30037 12 ==== osd_op_reply(75 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f6b0c006070 con 0x7f6b08038060 2023-10-05T19:28:27.046 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.042+0000 7f6b1aa70e40 20 RGWRados::pool_iterate: got zonegroup_info.4f1f719d-9648-417b-b10c-e5d939825419 2023-10-05T19:28:27.046 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.042+0000 7f6b1aa70e40 20 RGWRados::pool_iterate: got zone_info.9a9df965-6612-45d4-a45f-6e3c889bacc2 2023-10-05T19:28:27.046 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.042+0000 7f6b1aa70e40 20 RGWRados::pool_iterate: got zone_names.default 2023-10-05T19:28:27.046 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.042+0000 7f6b1aa70e40 20 RGWRados::pool_iterate: got zonegroups_names.default 2023-10-05T19:28:27.046 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.042+0000 7f6b1aa70e40 20 rados->read ofs=0 len=0 2023-10-05T19:28:27.046 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.042+0000 7f6b1aa70e40 1 -- 172.21.15.167:0/1447055810 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:76 5.f 5:f43b6ece:::zone_names.default:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x5573f482ce10 con 0x5573f47694b0 2023-10-05T19:28:27.047 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.042+0000 7f6b18da6700 1 -- 172.21.15.167:0/1447055810 <== osd.1 v2:172.21.15.167:6800/30035 26 ==== osd_op_reply(76 zone_names.default [read 0~46 out=46b] v0'0 uv1 ondisk = 0) v8 ==== 162+0+46 (crc 0 0 0) 0x7f6b1003f040 con 0x5573f47694b0 2023-10-05T19:28:27.047 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.042+0000 7f6b1aa70e40 20 rados_obj.operate() r=0 bl.length=46 2023-10-05T19:28:27.047 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.042+0000 7f6b1aa70e40 20 rados->read ofs=0 len=0 2023-10-05T19:28:27.047 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.042+0000 7f6b1aa70e40 1 -- 172.21.15.167:0/1447055810 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:77 5.1d 5:b8374eb4:::zone_info.9a9df965-6612-45d4-a45f-6e3c889bacc2:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x5573f482d1c0 con 0x5573f47694b0 2023-10-05T19:28:27.048 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.042+0000 7f6b18da6700 1 -- 172.21.15.167:0/1447055810 <== osd.1 v2:172.21.15.167:6800/30035 27 ==== osd_op_reply(77 zone_info.9a9df965-6612-45d4-a45f-6e3c889bacc2 [read 0~874 out=874b] v0'0 uv2 ondisk = 0) v8 ==== 190+0+874 (crc 0 0 0) 0x7f6b1003f040 con 0x5573f47694b0 2023-10-05T19:28:27.048 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.042+0000 7f6b1aa70e40 20 rados_obj.operate() r=0 bl.length=874 2023-10-05T19:28:27.048 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.042+0000 7f6b1aa70e40 20 rados->read ofs=0 len=0 2023-10-05T19:28:27.048 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.042+0000 7f6b1aa70e40 1 -- 172.21.15.167:0/1447055810 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:78 5.f 5:f4c53578:::zonegroups_names.default:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x5573f482d6b0 con 0x5573f47694b0 2023-10-05T19:28:27.049 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.042+0000 7f6b18da6700 1 -- 172.21.15.167:0/1447055810 <== osd.1 v2:172.21.15.167:6800/30035 28 ==== osd_op_reply(78 zonegroups_names.default [read 0~46 out=46b] v0'0 uv2 ondisk = 0) v8 ==== 168+0+46 (crc 0 0 0) 0x7f6b1003f040 con 0x5573f47694b0 2023-10-05T19:28:27.049 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.042+0000 7f6b1aa70e40 20 rados_obj.operate() r=0 bl.length=46 2023-10-05T19:28:27.049 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.042+0000 7f6b1aa70e40 20 rados->read ofs=0 len=0 2023-10-05T19:28:27.049 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.042+0000 7f6b1aa70e40 1 -- 172.21.15.167:0/1447055810 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:79 5.c 5:30d299a5:::zonegroup_info.4f1f719d-9648-417b-b10c-e5d939825419:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x5573f482d6b0 con 0x7f6b0c00eb60 2023-10-05T19:28:27.050 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.042+0000 7f6b18da6700 1 -- 172.21.15.167:0/1447055810 <== osd.0 v2:172.21.15.118:6800/27690 20 ==== osd_op_reply(79 zonegroup_info.4f1f719d-9648-417b-b10c-e5d939825419 [read 0~362 out=362b] v0'0 uv1 ondisk = 0) v8 ==== 195+0+362 (crc 0 0 0) 0x7f6b1003f040 con 0x7f6b0c00eb60 2023-10-05T19:28:27.050 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.042+0000 7f6b1aa70e40 20 rados_obj.operate() r=0 bl.length=362 2023-10-05T19:28:27.050 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.042+0000 7f6b1aa70e40 20 rados->read ofs=0 len=0 2023-10-05T19:28:27.050 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.042+0000 7f6b1aa70e40 1 -- 172.21.15.167:0/1447055810 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:80 5.3 5:c0ad8cbd:::region_map:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x5573f482d7b0 con 0x7f6b0c00eb60 2023-10-05T19:28:27.051 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.046+0000 7f6b18da6700 1 -- 172.21.15.167:0/1447055810 <== osd.0 v2:172.21.15.118:6800/27690 21 ==== osd_op_reply(80 region_map [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 ==== 154+0+0 (crc 0 0 0) 0x7f6b1003f040 con 0x7f6b0c00eb60 2023-10-05T19:28:27.051 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.046+0000 7f6b1aa70e40 20 rados_obj.operate() r=-2 bl.length=0 2023-10-05T19:28:27.051 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.046+0000 7f6b1aa70e40 20 zone default found 2023-10-05T19:28:27.051 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.046+0000 7f6b1aa70e40 4 Realm: () 2023-10-05T19:28:27.051 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.046+0000 7f6b1aa70e40 4 ZoneGroup: default (4f1f719d-9648-417b-b10c-e5d939825419) 2023-10-05T19:28:27.051 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.046+0000 7f6b1aa70e40 4 Zone: default (9a9df965-6612-45d4-a45f-6e3c889bacc2) 2023-10-05T19:28:27.051 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.046+0000 7f6b1aa70e40 10 cannot find current period zonegroup using local zonegroup configuration 2023-10-05T19:28:27.051 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.046+0000 7f6b1aa70e40 20 zonegroup default 2023-10-05T19:28:27.051 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.046+0000 7f6b1aa70e40 20 rados->read ofs=0 len=0 2023-10-05T19:28:27.051 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.046+0000 7f6b1aa70e40 1 -- 172.21.15.167:0/1447055810 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:81 5.3 5:c52100b6:::period_config.default:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x5573f482d7b0 con 0x7f6b0c00eb60 2023-10-05T19:28:27.052 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.046+0000 7f6b18da6700 1 -- 172.21.15.167:0/1447055810 <== osd.0 v2:172.21.15.118:6800/27690 22 ==== osd_op_reply(81 period_config.default [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 ==== 165+0+0 (crc 0 0 0) 0x7f6b1003f040 con 0x7f6b0c00eb60 2023-10-05T19:28:27.052 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.046+0000 7f6b1aa70e40 20 rados_obj.operate() r=-2 bl.length=0 2023-10-05T19:28:27.052 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.046+0000 7f6b1aa70e40 20 rados->read ofs=0 len=0 2023-10-05T19:28:27.052 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.046+0000 7f6b1aa70e40 1 -- 172.21.15.167:0/1447055810 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:82 6.2 6:4347d321:::bucket.sync-source-hints.:head [call version.read in=11b,read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x5573f482f130 con 0x5573f482a0b0 2023-10-05T19:28:27.052 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.046+0000 7f6b185a5700 1 -- 172.21.15.167:0/1447055810 <== osd.3 v2:172.21.15.167:6808/30036 20 ==== osd_op_reply(82 bucket.sync-source-hints. [call,read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 ==== 211+0+0 (crc 0 0 0) 0x7f6b0801c7f0 con 0x5573f482a0b0 2023-10-05T19:28:27.053 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.046+0000 7f6b1aa70e40 20 rados_obj.operate() r=-2 bl.length=0 2023-10-05T19:28:27.053 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.046+0000 7f6b1aa70e40 20 rados->read ofs=0 len=0 2023-10-05T19:28:27.053 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.046+0000 7f6b1aa70e40 1 -- 172.21.15.167:0/1447055810 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:83 6.b 6:d467b91b:::bucket.sync-target-hints.:head [call version.read in=11b,read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x5573f482f4e0 con 0x5573f482a0b0 2023-10-05T19:28:27.054 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.046+0000 7f6b185a5700 1 -- 172.21.15.167:0/1447055810 <== osd.3 v2:172.21.15.167:6808/30036 21 ==== osd_op_reply(83 bucket.sync-target-hints. [call,read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 ==== 211+0+0 (crc 0 0 0) 0x7f6b0801c7f0 con 0x5573f482a0b0 2023-10-05T19:28:27.054 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.046+0000 7f6b1aa70e40 20 rados_obj.operate() r=-2 bl.length=0 2023-10-05T19:28:27.054 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.046+0000 7f6b1aa70e40 20 started sync module instance, tier type = 2023-10-05T19:28:27.054 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.046+0000 7f6b1aa70e40 20 started zone id=9a9df965-6612-45d4-a45f-6e3c889bacc2 (name=default) with tier type = 2023-10-05T19:28:27.054 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.046+0000 7f6b1aa70e40 1 -- 172.21.15.167:0/1447055810 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:84 7.1f 7:f95f44c2:::notify.0:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x5573f482e7f0 con 0x7f6b08038060 2023-10-05T19:28:27.056 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.050+0000 7f6b17da4700 1 -- 172.21.15.167:0/1447055810 <== osd.2 v2:172.21.15.167:6816/30037 13 ==== osd_op_reply(84 notify.0 [create] v30'42 uv42 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f6b0c006070 con 0x7f6b08038060 2023-10-05T19:28:27.056 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.050+0000 7f6b1aa70e40 1 -- 172.21.15.167:0/1447055810 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:85 7.1f 7:f95f44c2:::notify.0:head [watch watch cookie 93956511820704] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x5573f482e420 con 0x7f6b08038060 2023-10-05T19:28:27.056 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.050+0000 7f6b1aa70e40 1 -- 172.21.15.167:0/1447055810 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:86 7.0 7:05bf5b68:::notify.1:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x5573f48302e0 con 0x5573f47694b0 2023-10-05T19:28:27.059 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.054+0000 7f6b17da4700 1 -- 172.21.15.167:0/1447055810 <== osd.2 v2:172.21.15.167:6816/30037 14 ==== osd_op_reply(85 notify.0 [watch watch cookie 93956511820704] v30'43 uv42 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f6b0c006070 con 0x7f6b08038060 2023-10-05T19:28:27.060 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.054+0000 7f6b18da6700 1 -- 172.21.15.167:0/1447055810 <== osd.1 v2:172.21.15.167:6800/30035 29 ==== osd_op_reply(86 notify.1 [create] v30'42 uv42 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f6b1003f040 con 0x5573f47694b0 2023-10-05T19:28:27.060 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.054+0000 7f6b1aa70e40 1 -- 172.21.15.167:0/1447055810 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:87 7.0 7:05bf5b68:::notify.1:head [watch watch cookie 93956511824016] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x5573f48306e0 con 0x5573f47694b0 2023-10-05T19:28:27.060 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.054+0000 7f6b1aa70e40 1 -- 172.21.15.167:0/1447055810 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:88 7.15 7:a93a5511:::notify.2:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x5573f4830f80 con 0x5573f482a0b0 2023-10-05T19:28:27.062 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.058+0000 7f6b18da6700 1 -- 172.21.15.167:0/1447055810 <== osd.1 v2:172.21.15.167:6800/30035 30 ==== osd_op_reply(87 notify.1 [watch watch cookie 93956511824016] v30'43 uv42 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f6b1003f040 con 0x5573f47694b0 2023-10-05T19:28:27.063 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.058+0000 7f6b185a5700 1 -- 172.21.15.167:0/1447055810 <== osd.3 v2:172.21.15.167:6808/30036 22 ==== osd_op_reply(88 notify.2 [create] v30'42 uv42 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f6b0801c7f0 con 0x5573f482a0b0 2023-10-05T19:28:27.063 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.058+0000 7f6b1aa70e40 1 -- 172.21.15.167:0/1447055810 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:89 7.15 7:a93a5511:::notify.2:head [watch watch cookie 93956511827680] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x5573f482fed0 con 0x5573f482a0b0 2023-10-05T19:28:27.063 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.058+0000 7f6b1aa70e40 1 -- 172.21.15.167:0/1447055810 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:90 7.e 7:7759931f:::notify.3:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x5573f4831d10 con 0x7f6b08038060 2023-10-05T19:28:27.066 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.058+0000 7f6b17da4700 1 -- 172.21.15.167:0/1447055810 <== osd.2 v2:172.21.15.167:6816/30037 15 ==== osd_op_reply(90 notify.3 [create] v30'42 uv42 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f6b0c006070 con 0x7f6b08038060 2023-10-05T19:28:27.066 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.058+0000 7f6b1aa70e40 1 -- 172.21.15.167:0/1447055810 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:91 7.e 7:7759931f:::notify.3:head [watch watch cookie 93956511832096] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x5573f4832630 con 0x7f6b08038060 2023-10-05T19:28:27.066 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.062+0000 7f6b1aa70e40 1 -- 172.21.15.167:0/1447055810 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:92 7.d 7:b4812045:::notify.4:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x5573f4833160 con 0x5573f47694b0 2023-10-05T19:28:27.066 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.062+0000 7f6b185a5700 1 -- 172.21.15.167:0/1447055810 <== osd.3 v2:172.21.15.167:6808/30036 23 ==== osd_op_reply(89 notify.2 [watch watch cookie 93956511827680] v30'43 uv42 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f6b0801c7f0 con 0x5573f482a0b0 2023-10-05T19:28:27.068 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.062+0000 7f6b17da4700 1 -- 172.21.15.167:0/1447055810 <== osd.2 v2:172.21.15.167:6816/30037 16 ==== osd_op_reply(91 notify.3 [watch watch cookie 93956511832096] v30'43 uv42 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f6b0c006070 con 0x7f6b08038060 2023-10-05T19:28:27.069 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.062+0000 7f6b18da6700 1 -- 172.21.15.167:0/1447055810 <== osd.1 v2:172.21.15.167:6800/30035 31 ==== osd_op_reply(92 notify.4 [create] v30'42 uv42 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f6b1003f040 con 0x5573f47694b0 2023-10-05T19:28:27.069 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.062+0000 7f6b1aa70e40 1 -- 172.21.15.167:0/1447055810 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:93 7.d 7:b4812045:::notify.4:head [watch watch cookie 93956511830832] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x5573f48320c0 con 0x5573f47694b0 2023-10-05T19:28:27.069 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.062+0000 7f6b1aa70e40 1 -- 172.21.15.167:0/1447055810 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:94 7.3 7:c609908c:::notify.5:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x5573f4833a50 con 0x7f6b0c00eb60 2023-10-05T19:28:27.072 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.066+0000 7f6b18da6700 1 -- 172.21.15.167:0/1447055810 <== osd.0 v2:172.21.15.118:6800/27690 23 ==== osd_op_reply(94 notify.5 [create] v30'42 uv42 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f6b1003f040 con 0x7f6b0c00eb60 2023-10-05T19:28:27.072 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.066+0000 7f6b1aa70e40 1 -- 172.21.15.167:0/1447055810 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:95 7.3 7:c609908c:::notify.5:head [watch watch cookie 93956511839504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x5573f4833e50 con 0x7f6b0c00eb60 2023-10-05T19:28:27.072 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.066+0000 7f6b1aa70e40 1 -- 172.21.15.167:0/1447055810 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:96 7.14 7:2b04a3e9:::notify.6:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x5573f48348a0 con 0x7f6b0c00eb60 2023-10-05T19:28:27.072 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.066+0000 7f6b18da6700 1 -- 172.21.15.167:0/1447055810 <== osd.1 v2:172.21.15.167:6800/30035 32 ==== osd_op_reply(93 notify.4 [watch watch cookie 93956511830832] v30'43 uv42 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f6b1003f040 con 0x5573f47694b0 2023-10-05T19:28:27.074 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.070+0000 7f6b18da6700 1 -- 172.21.15.167:0/1447055810 <== osd.0 v2:172.21.15.118:6800/27690 24 ==== osd_op_reply(95 notify.5 [watch watch cookie 93956511839504] v30'43 uv42 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f6b1003f040 con 0x7f6b0c00eb60 2023-10-05T19:28:27.074 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.070+0000 7f6b18da6700 1 -- 172.21.15.167:0/1447055810 <== osd.0 v2:172.21.15.118:6800/27690 25 ==== osd_op_reply(96 notify.6 [create] v30'42 uv42 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f6b1003f040 con 0x7f6b0c00eb60 2023-10-05T19:28:27.075 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.070+0000 7f6b1aa70e40 1 -- 172.21.15.167:0/1447055810 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:97 7.14 7:2b04a3e9:::notify.6:head [watch watch cookie 93956511837232] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x5573f4833960 con 0x7f6b0c00eb60 2023-10-05T19:28:27.075 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.070+0000 7f6b1aa70e40 1 -- 172.21.15.167:0/1447055810 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:98 7.9 7:93e5b521:::notify.7:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x5573f4835190 con 0x7f6b0c00eb60 2023-10-05T19:28:27.077 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.070+0000 7f6b18da6700 1 -- 172.21.15.167:0/1447055810 <== osd.0 v2:172.21.15.118:6800/27690 26 ==== osd_op_reply(97 notify.6 [watch watch cookie 93956511837232] v30'43 uv42 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f6b1003f040 con 0x7f6b0c00eb60 2023-10-05T19:28:27.077 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.070+0000 7f6b18da6700 1 -- 172.21.15.167:0/1447055810 <== osd.0 v2:172.21.15.118:6800/27690 27 ==== osd_op_reply(98 notify.7 [create] v30'42 uv42 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f6b1003f040 con 0x7f6b0c00eb60 2023-10-05T19:28:27.078 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.070+0000 7f6b1aa70e40 1 -- 172.21.15.167:0/1447055810 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:99 7.9 7:93e5b521:::notify.7:head [watch watch cookie 93956511843184] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x5573f4833960 con 0x7f6b0c00eb60 2023-10-05T19:28:27.078 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.070+0000 7f6b1aa70e40 20 add_watcher() i=0 2023-10-05T19:28:27.078 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.070+0000 7f6b1aa70e40 20 add_watcher() i=1 2023-10-05T19:28:27.078 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.070+0000 7f6b1aa70e40 20 add_watcher() i=2 2023-10-05T19:28:27.078 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.070+0000 7f6b1aa70e40 20 add_watcher() i=3 2023-10-05T19:28:27.078 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.070+0000 7f6b1aa70e40 20 add_watcher() i=4 2023-10-05T19:28:27.078 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.070+0000 7f6b1aa70e40 20 add_watcher() i=5 2023-10-05T19:28:27.078 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.070+0000 7f6b1aa70e40 20 add_watcher() i=6 2023-10-05T19:28:27.080 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.074+0000 7f6b18da6700 1 -- 172.21.15.167:0/1447055810 <== osd.0 v2:172.21.15.118:6800/27690 28 ==== osd_op_reply(99 notify.7 [watch watch cookie 93956511843184] v30'43 uv42 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f6b1003f040 con 0x7f6b0c00eb60 2023-10-05T19:28:27.080 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.074+0000 7f6b1aa70e40 20 add_watcher() i=7 2023-10-05T19:28:27.080 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.074+0000 7f6b1aa70e40 2 all 8 watchers are set, enabling cache 2023-10-05T19:28:27.081 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.074+0000 7f6b1aa70e40 1 -- 172.21.15.167:0/1447055810 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:100 6.4 6:22d26bf9:::data_loggenerations_metadata:head [call version.check_conds in=74b,call version.read in=11b,read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x5573f4838020 con 0x5573f47694b0 2023-10-05T19:28:27.081 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.074+0000 7f6b18da6700 1 -- 172.21.15.167:0/1447055810 <== osd.1 v2:172.21.15.167:6800/30035 33 ==== osd_op_reply(100 data_loggenerations_metadata [call,call out=48b,read 0~28 out=28b] v0'0 uv1 ondisk = 0) v8 ==== 256+0+76 (crc 0 0 0) 0x7f6b1003f040 con 0x5573f47694b0 2023-10-05T19:28:27.081 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.074+0000 7f6b1aa70e40 1 -- 172.21.15.167:0/1447055810 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:101 6.4 6:22d26bf9:::data_loggenerations_metadata:head [watch watch cookie 93956511845072] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x5573f48354e0 con 0x5573f47694b0 2023-10-05T19:28:27.083 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.078+0000 7f6b18da6700 1 -- 172.21.15.167:0/1447055810 <== osd.1 v2:172.21.15.167:6800/30035 34 ==== osd_op_reply(101 data_loggenerations_metadata [watch watch cookie 93956511845072] v30'62 uv1 ondisk = 0) v8 ==== 172+0+0 (crc 0 0 0) 0x7f6b1003f040 con 0x5573f47694b0 2023-10-05T19:28:27.083 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.078+0000 7f6ae9fe3700 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2023-10-05T19:28:27.084 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.078+0000 7f6b1aa70e40 20 check_secure_mon_conn(): auth registy supported: methods=[2] modes=[2,1] 2023-10-05T19:28:27.084 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.078+0000 7f6b1aa70e40 20 check_secure_mon_conn(): mode 1 is insecure 2023-10-05T19:28:27.084 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.078+0000 7f6b1aa70e40 1 Processor -- start 2023-10-05T19:28:27.084 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.078+0000 7f6b1aa70e40 1 -- start start 2023-10-05T19:28:27.084 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.078+0000 7f6b1aa70e40 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5573f48423b0 0x5573f4842780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-05T19:28:27.085 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.078+0000 7f6b1aa70e40 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x5573f4842cc0 con 0x5573f48423b0 2023-10-05T19:28:27.085 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.078+0000 7f6b17da4700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5573f48423b0 0x5573f4842780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-05T19:28:27.085 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.078+0000 7f6b17da4700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5573f48423b0 0x5573f4842780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.118:3300/0 says I am v2:172.21.15.167:57444/0 (socket says 172.21.15.167:57444) 2023-10-05T19:28:27.085 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.078+0000 7f6b17da4700 1 -- 172.21.15.167:0/33528356 learned_addr learned my addr 172.21.15.167:0/33528356 (peer_addr_for_me v2:172.21.15.167:0/0) 2023-10-05T19:28:27.085 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.078+0000 7f6b17da4700 1 -- 172.21.15.167:0/33528356 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5573f4842e00 con 0x5573f48423b0 2023-10-05T19:28:27.086 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.082+0000 7f6b17da4700 1 --2- 172.21.15.167:0/33528356 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5573f48423b0 0x5573f4842780 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7f6b0c000fb0 tx=0x7f6b0c006000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=274e1350d0747262 server_cookie=0 in_seq=0 out_seq=0 2023-10-05T19:28:27.086 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.082+0000 7f6ae97e2700 1 -- 172.21.15.167:0/33528356 <== mon.0 v2:172.21.15.118:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f6b0c023040 con 0x5573f48423b0 2023-10-05T19:28:27.086 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.082+0000 7f6ae97e2700 1 -- 172.21.15.167:0/33528356 <== mon.0 v2:172.21.15.118:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6b0c024040 con 0x5573f48423b0 2023-10-05T19:28:27.086 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.082+0000 7f6ae97e2700 1 -- 172.21.15.167:0/33528356 <== mon.0 v2:172.21.15.118:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f6b0c013040 con 0x5573f48423b0 2023-10-05T19:28:27.088 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.082+0000 7f6b1aa70e40 1 -- 172.21.15.167:0/33528356 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5573f48423b0 msgr2=0x5573f4842780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-05T19:28:27.088 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.082+0000 7f6b1aa70e40 1 --2- 172.21.15.167:0/33528356 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5573f48423b0 0x5573f4842780 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7f6b0c000fb0 tx=0x7f6b0c006000 comp rx=0 tx=0).stop 2023-10-05T19:28:27.088 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.082+0000 7f6b1aa70e40 1 -- 172.21.15.167:0/33528356 shutdown_connections 2023-10-05T19:28:27.088 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.082+0000 7f6b1aa70e40 1 --2- 172.21.15.167:0/33528356 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5573f48423b0 0x5573f4842780 unknown :-1 s=CLOSED pgs=222 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-05T19:28:27.088 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.082+0000 7f6b1aa70e40 1 -- 172.21.15.167:0/33528356 >> 172.21.15.167:0/33528356 conn(0x5573f483d770 msgr2=0x5573f483b920 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-05T19:28:27.088 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.082+0000 7f6b1aa70e40 1 -- 172.21.15.167:0/33528356 shutdown_connections 2023-10-05T19:28:27.088 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.082+0000 7f6b1aa70e40 1 -- 172.21.15.167:0/33528356 wait complete. 2023-10-05T19:28:27.089 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.082+0000 7f6b1aa70e40 1 Processor -- start 2023-10-05T19:28:27.089 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.082+0000 7f6b1aa70e40 1 -- start start 2023-10-05T19:28:27.089 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.082+0000 7f6b1aa70e40 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5573f48423b0 0x5573f4847f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-05T19:28:27.089 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.082+0000 7f6b1aa70e40 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x5573f4848f50 con 0x5573f48423b0 2023-10-05T19:28:27.090 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.082+0000 7f6b17da4700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5573f48423b0 0x5573f4847f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-05T19:28:27.090 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.082+0000 7f6b17da4700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5573f48423b0 0x5573f4847f50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.118:3300/0 says I am v2:172.21.15.167:57448/0 (socket says 172.21.15.167:57448) 2023-10-05T19:28:27.090 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.082+0000 7f6b17da4700 1 -- 172.21.15.167:0/3597287746 learned_addr learned my addr 172.21.15.167:0/3597287746 (peer_addr_for_me v2:172.21.15.167:0/0) 2023-10-05T19:28:27.090 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.086+0000 7f6b17da4700 1 -- 172.21.15.167:0/3597287746 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6b00001e10 con 0x5573f48423b0 2023-10-05T19:28:27.090 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.086+0000 7f6b17da4700 1 --2- 172.21.15.167:0/3597287746 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5573f48423b0 0x5573f4847f50 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x5573f4827ba0 tx=0x7f6b0c030000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-05T19:28:27.091 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.086+0000 7f6a637fe700 1 -- 172.21.15.167:0/3597287746 <== mon.0 v2:172.21.15.118:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f6b0c023040 con 0x5573f48423b0 2023-10-05T19:28:27.091 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.086+0000 7f6a637fe700 1 -- 172.21.15.167:0/3597287746 <== mon.0 v2:172.21.15.118:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6b0c01e6b0 con 0x5573f48423b0 2023-10-05T19:28:27.091 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.086+0000 7f6a637fe700 1 -- 172.21.15.167:0/3597287746 <== mon.0 v2:172.21.15.118:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f6b0c01e860 con 0x5573f48423b0 2023-10-05T19:28:27.091 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.086+0000 7f6b1aa70e40 1 -- 172.21.15.167:0/3597287746 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x5573f4848550 con 0x5573f48423b0 2023-10-05T19:28:27.091 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.086+0000 7f6b1aa70e40 1 -- 172.21.15.167:0/3597287746 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x5573f4848990 con 0x5573f48423b0 2023-10-05T19:28:27.093 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.086+0000 7f6a637fe700 1 -- 172.21.15.167:0/3597287746 <== mon.0 v2:172.21.15.118:3300/0 4 ==== mgrmap(e 5) v1 ==== 40329+0+0 (secure 0 0 0) 0x7f6b0c01e9c0 con 0x5573f48423b0 2023-10-05T19:28:27.093 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.086+0000 7f6a637fe700 1 --2- 172.21.15.167:0/3597287746 >> [v2:172.21.15.167:6824/29889,v1:172.21.15.167:6825/29889] conn(0x7f6a58030ca0 0x7f6a58033150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-05T19:28:27.094 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.086+0000 7f6a637fe700 1 -- 172.21.15.167:0/3597287746 <== mon.0 v2:172.21.15.118:3300/0 5 ==== osd_map(30..30 src has 1..30) v4 ==== 6057+0+0 (secure 0 0 0) 0x7f6b0c043c20 con 0x5573f48423b0 2023-10-05T19:28:27.094 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.090+0000 7f6b1aa70e40 5 note: GC not initialized 2023-10-05T19:28:27.094 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.090+0000 7f6b18da6700 1 --2- 172.21.15.167:0/3597287746 >> [v2:172.21.15.167:6824/29889,v1:172.21.15.167:6825/29889] conn(0x7f6a58030ca0 0x7f6a58033150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-05T19:28:27.095 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.090+0000 7f6b18da6700 1 --2- 172.21.15.167:0/3597287746 >> [v2:172.21.15.167:6824/29889,v1:172.21.15.167:6825/29889] conn(0x7f6a58030ca0 0x7f6a58033150 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x5573f4833c30 tx=0x7f6b1000aec0 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-05T19:28:27.096 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.090+0000 7f6a60ff9700 20 reqs_thread_entry: start 2023-10-05T19:28:27.097 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.090+0000 7f6b1aa70e40 20 init_complete bucket index max shards: 11 2023-10-05T19:28:27.097 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.090+0000 7f6a567fc700 20 reqs_thread_entry: start 2023-10-05T19:28:27.097 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.090+0000 7f6b1aa70e40 10 rgw notify: Started notification manager with: 1 workers 2023-10-05T19:28:27.098 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.090+0000 7f6a54ff9700 20 rgw notify: INFO: next queues processing will happen at: Thu Oct 5 19:28:57 2023 2023-10-05T19:28:27.098 INFO:teuthology.orchestra.run.smithi167.stderr: 2023-10-05T19:28:27.098 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.090+0000 7f6a3bfff700 20 reqs_thread_entry: start 2023-10-05T19:28:27.098 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.090+0000 7f6b1aa70e40 20 get_system_obj_state: rctx=0x7ffd8395ca58 obj=default.rgw.meta:users.uid:foo state=0x5573f46cc0b0 s->prefetch_data=0 2023-10-05T19:28:27.098 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.090+0000 7f6b1aa70e40 1 -- 172.21.15.167:0/1447055810 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:102 8.18 8:1e6ca5ed:users.uid::foo:head [call version.read in=11b,getxattrs,stat] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x5573f48b0740 con 0x5573f47694b0 2023-10-05T19:28:27.098 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.094+0000 7f6b18da6700 1 -- 172.21.15.167:0/1447055810 <== osd.1 v2:172.21.15.167:6800/30035 35 ==== osd_op_reply(102 foo [call out=48b,getxattrs out=75b,stat out=16b] v0'0 uv3 ondisk = 0) v8 ==== 231+0+139 (crc 0 0 0) 0x7f6b1003f040 con 0x5573f47694b0 2023-10-05T19:28:27.099 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.094+0000 7f6b1aa70e40 20 get_system_obj_state: s->obj_tag was set empty 2023-10-05T19:28:27.099 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.094+0000 7f6b1aa70e40 20 Read xattr: user.rgw.idtag 2023-10-05T19:28:27.099 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.094+0000 7f6b1aa70e40 20 rados->read ofs=0 len=0 2023-10-05T19:28:27.099 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.094+0000 7f6b1aa70e40 1 -- 172.21.15.167:0/1447055810 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:103 8.18 8:1e6ca5ed:users.uid::foo:head [call version.check_conds in=122b,call version.read in=11b,read 0~0,getxattrs] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x5573f48b06c0 con 0x5573f47694b0 2023-10-05T19:28:27.099 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.094+0000 7f6b18da6700 1 -- 172.21.15.167:0/1447055810 <== osd.1 v2:172.21.15.167:6800/30035 36 ==== osd_op_reply(103 foo [call,call out=48b,read 0~357 out=357b,getxattrs out=75b] v0'0 uv3 ondisk = 0) v8 ==== 273+0+480 (crc 0 0 0) 0x7f6b1003f040 con 0x5573f47694b0 2023-10-05T19:28:27.100 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.094+0000 7f6b1aa70e40 20 rados_obj.operate() r=0 bl.length=357 2023-10-05T19:28:27.100 INFO:teuthology.orchestra.run.smithi167.stdout:{ 2023-10-05T19:28:27.100 INFO:teuthology.orchestra.run.smithi167.stdout: "user_id": "foo", 2023-10-05T19:28:27.100 INFO:teuthology.orchestra.run.smithi167.stdout: "display_name": "Foo", 2023-10-05T19:28:27.100 INFO:teuthology.orchestra.run.smithi167.stdout: "email": "foo@foo.com", 2023-10-05T19:28:27.100 INFO:teuthology.orchestra.run.smithi167.stdout: "suspended": 0, 2023-10-05T19:28:27.100 INFO:teuthology.orchestra.run.smithi167.stdout: "max_buckets": 4, 2023-10-05T19:28:27.100 INFO:teuthology.orchestra.run.smithi167.stdout: "subusers": [], 2023-10-05T19:28:27.100 INFO:teuthology.orchestra.run.smithi167.stdout: "keys": [ 2023-10-05T19:28:27.100 INFO:teuthology.orchestra.run.smithi167.stdout: { 2023-10-05T19:28:27.101 INFO:teuthology.orchestra.run.smithi167.stdout: "user": "foo", 2023-10-05T19:28:27.101 INFO:teuthology.orchestra.run.smithi167.stdout: "access_key": "9te6NH5mcdcq0Tc5i8i1", 2023-10-05T19:28:27.101 INFO:teuthology.orchestra.run.smithi167.stdout: "secret_key": "Ny4IOauQoL18Gp2zM7lC1vLmoawgqcYP/YGcWfXu" 2023-10-05T19:28:27.101 INFO:teuthology.orchestra.run.smithi167.stdout: } 2023-10-05T19:28:27.101 INFO:teuthology.orchestra.run.smithi167.stdout: ], 2023-10-05T19:28:27.101 INFO:teuthology.orchestra.run.smithi167.stdout: "swift_keys": [], 2023-10-05T19:28:27.101 INFO:teuthology.orchestra.run.smithi167.stdout: "caps": [], 2023-10-05T19:28:27.101 INFO:teuthology.orchestra.run.smithi167.stdout: "op_mask": "read, write, delete", 2023-10-05T19:28:27.101 INFO:teuthology.orchestra.run.smithi167.stdout: "default_placement": "", 2023-10-05T19:28:27.101 INFO:teuthology.orchestra.run.smithi167.stdout: "default_storage_class": "", 2023-10-05T19:28:27.101 INFO:teuthology.orchestra.run.smithi167.stdout: "placement_tags": [], 2023-10-05T19:28:27.101 INFO:teuthology.orchestra.run.smithi167.stdout: "bucket_quota": { 2023-10-05T19:28:27.101 INFO:teuthology.orchestra.run.smithi167.stdout: "enabled": false, 2023-10-05T19:28:27.102 INFO:teuthology.orchestra.run.smithi167.stdout: "check_on_raw": false, 2023-10-05T19:28:27.102 INFO:teuthology.orchestra.run.smithi167.stdout: "max_size": -1, 2023-10-05T19:28:27.102 INFO:teuthology.orchestra.run.smithi167.stdout: "max_size_kb": 0, 2023-10-05T19:28:27.102 INFO:teuthology.orchestra.run.smithi167.stdout: "max_objects": -1 2023-10-05T19:28:27.102 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2023-10-05T19:28:27.102 INFO:teuthology.orchestra.run.smithi167.stdout: "user_quota": { 2023-10-05T19:28:27.102 INFO:teuthology.orchestra.run.smithi167.stdout: "enabled": false, 2023-10-05T19:28:27.102 INFO:teuthology.orchestra.run.smithi167.stdout: "check_on_raw": false, 2023-10-05T19:28:27.102 INFO:teuthology.orchestra.run.smithi167.stdout: "max_size": -1, 2023-10-05T19:28:27.102 INFO:teuthology.orchestra.run.smithi167.stdout: "max_size_kb": 0, 2023-10-05T19:28:27.102 INFO:teuthology.orchestra.run.smithi167.stdout: "max_objects": -1 2023-10-05T19:28:27.102 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2023-10-05T19:28:27.103 INFO:teuthology.orchestra.run.smithi167.stdout: "temp_url_keys": [], 2023-10-05T19:28:27.103 INFO:teuthology.orchestra.run.smithi167.stdout: "type": "rgw", 2023-10-05T19:28:27.103 INFO:teuthology.orchestra.run.smithi167.stdout: "mfa_ids": [] 2023-10-05T19:28:27.103 INFO:teuthology.orchestra.run.smithi167.stdout:} 2023-10-05T19:28:27.103 INFO:teuthology.orchestra.run.smithi167.stdout: 2023-10-05T19:28:27.103 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.094+0000 7f6b1aa70e40 1 -- 172.21.15.167:0/1447055810 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:104 7.1f 7:f95f44c2:::notify.0:head [watch unwatch cookie 93956511820704] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x5573f48b06c0 con 0x7f6b08038060 2023-10-05T19:28:27.104 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.098+0000 7f6b17da4700 1 -- 172.21.15.167:0/1447055810 <== osd.2 v2:172.21.15.167:6816/30037 17 ==== osd_op_reply(104 notify.0 [watch unwatch cookie 93956511820704] v30'44 uv42 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f6b0c03f040 con 0x7f6b08038060 2023-10-05T19:28:27.104 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.098+0000 7f6b1aa70e40 20 remove_watcher() i=0 2023-10-05T19:28:27.104 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.098+0000 7f6b1aa70e40 2 removed watcher, disabling cache 2023-10-05T19:28:27.104 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.098+0000 7f6b1aa70e40 1 -- 172.21.15.167:0/1447055810 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:105 7.0 7:05bf5b68:::notify.1:head [watch unwatch cookie 93956511824016] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x5573f480a7f0 con 0x5573f47694b0 2023-10-05T19:28:27.106 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.102+0000 7f6b18da6700 1 -- 172.21.15.167:0/1447055810 <== osd.1 v2:172.21.15.167:6800/30035 37 ==== osd_op_reply(105 notify.1 [watch unwatch cookie 93956511824016] v30'44 uv42 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f6b1003f040 con 0x5573f47694b0 2023-10-05T19:28:27.106 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.102+0000 7f6b1aa70e40 20 remove_watcher() i=1 2023-10-05T19:28:27.106 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.102+0000 7f6b1aa70e40 1 -- 172.21.15.167:0/1447055810 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:106 7.15 7:a93a5511:::notify.2:head [watch unwatch cookie 93956511827680] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x5573f482f890 con 0x5573f482a0b0 2023-10-05T19:28:27.109 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.102+0000 7f6b185a5700 1 -- 172.21.15.167:0/1447055810 <== osd.3 v2:172.21.15.167:6808/30036 24 ==== osd_op_reply(106 notify.2 [watch unwatch cookie 93956511827680] v30'44 uv42 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f6b0801c7f0 con 0x5573f482a0b0 2023-10-05T19:28:27.109 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.102+0000 7f6b1aa70e40 20 remove_watcher() i=2 2023-10-05T19:28:27.109 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.102+0000 7f6b1aa70e40 1 -- 172.21.15.167:0/1447055810 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:107 7.e 7:7759931f:::notify.3:head [watch unwatch cookie 93956511832096] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x5573f48306b0 con 0x7f6b08038060 2023-10-05T19:28:27.111 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.106+0000 7f6b17da4700 1 -- 172.21.15.167:0/1447055810 <== osd.2 v2:172.21.15.167:6816/30037 18 ==== osd_op_reply(107 notify.3 [watch unwatch cookie 93956511832096] v30'44 uv42 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f6b0c03f040 con 0x7f6b08038060 2023-10-05T19:28:27.111 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.106+0000 7f6b1aa70e40 20 remove_watcher() i=3 2023-10-05T19:28:27.111 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.106+0000 7f6b1aa70e40 1 -- 172.21.15.167:0/1447055810 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:108 7.d 7:b4812045:::notify.4:head [watch unwatch cookie 93956511830832] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x5573f4831820 con 0x5573f47694b0 2023-10-05T19:28:27.114 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.110+0000 7f6b18da6700 1 -- 172.21.15.167:0/1447055810 <== osd.1 v2:172.21.15.167:6800/30035 38 ==== osd_op_reply(108 notify.4 [watch unwatch cookie 93956511830832] v30'44 uv42 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f6b1003f040 con 0x5573f47694b0 2023-10-05T19:28:27.114 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.110+0000 7f6b1aa70e40 20 remove_watcher() i=4 2023-10-05T19:28:27.114 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.110+0000 7f6b1aa70e40 1 -- 172.21.15.167:0/1447055810 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:109 7.3 7:c609908c:::notify.5:head [watch unwatch cookie 93956511839504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x5573f4831820 con 0x7f6b0c00eb60 2023-10-05T19:28:27.117 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.110+0000 7f6b18da6700 1 -- 172.21.15.167:0/1447055810 <== osd.0 v2:172.21.15.118:6800/27690 29 ==== osd_op_reply(109 notify.5 [watch unwatch cookie 93956511839504] v30'44 uv42 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f6b1003f040 con 0x7f6b0c00eb60 2023-10-05T19:28:27.117 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.110+0000 7f6b1aa70e40 20 remove_watcher() i=5 2023-10-05T19:28:27.117 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.110+0000 7f6b1aa70e40 1 -- 172.21.15.167:0/1447055810 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:110 7.14 7:2b04a3e9:::notify.6:head [watch unwatch cookie 93956511837232] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x5573f4831820 con 0x7f6b0c00eb60 2023-10-05T19:28:27.120 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.114+0000 7f6b18da6700 1 -- 172.21.15.167:0/1447055810 <== osd.0 v2:172.21.15.118:6800/27690 30 ==== osd_op_reply(110 notify.6 [watch unwatch cookie 93956511837232] v30'44 uv42 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f6b1003f040 con 0x7f6b0c00eb60 2023-10-05T19:28:27.120 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.114+0000 7f6b1aa70e40 20 remove_watcher() i=6 2023-10-05T19:28:27.120 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.114+0000 7f6b1aa70e40 1 -- 172.21.15.167:0/1447055810 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:111 7.9 7:93e5b521:::notify.7:head [watch unwatch cookie 93956511843184] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x5573f4831820 con 0x7f6b0c00eb60 2023-10-05T19:28:27.123 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.118+0000 7f6b18da6700 1 -- 172.21.15.167:0/1447055810 <== osd.0 v2:172.21.15.118:6800/27690 31 ==== osd_op_reply(111 notify.7 [watch unwatch cookie 93956511843184] v30'44 uv42 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f6b1003f040 con 0x7f6b0c00eb60 2023-10-05T19:28:27.123 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.118+0000 7f6b1aa70e40 20 remove_watcher() i=7 2023-10-05T19:28:27.125 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.118+0000 7f6ae9fe3700 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2023-10-05T19:28:27.125 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.118+0000 7f6b1aa70e40 1 -- 172.21.15.167:0/1447055810 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:112 6.4 6:22d26bf9:::data_loggenerations_metadata:head [watch unwatch cookie 93956511845072] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x5573f486fac0 con 0x5573f47694b0 2023-10-05T19:28:27.131 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.122+0000 7f6b18da6700 1 -- 172.21.15.167:0/1447055810 <== osd.1 v2:172.21.15.167:6800/30035 39 ==== osd_op_reply(112 data_loggenerations_metadata [watch unwatch cookie 93956511845072] v30'63 uv1 ondisk = 0) v8 ==== 172+0+0 (crc 0 0 0) 0x7f6b1003f040 con 0x5573f47694b0 2023-10-05T19:28:27.131 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.122+0000 7f6b1aa70e40 1 -- 172.21.15.167:0/1447055810 >> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] conn(0x7f6b0c00eb60 msgr2=0x7f6b0c00d8e0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-05T19:28:27.131 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.122+0000 7f6b1aa70e40 1 --2- 172.21.15.167:0/1447055810 >> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] conn(0x7f6b0c00eb60 0x7f6b0c00d8e0 crc :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-05T19:28:27.131 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.122+0000 7f6b1aa70e40 1 -- 172.21.15.167:0/1447055810 >> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] conn(0x5573f47694b0 msgr2=0x5573f48266a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-05T19:28:27.132 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.122+0000 7f6b1aa70e40 1 --2- 172.21.15.167:0/1447055810 >> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] conn(0x5573f47694b0 0x5573f48266a0 crc :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-05T19:28:27.132 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.122+0000 7f6b1aa70e40 1 -- 172.21.15.167:0/1447055810 >> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] conn(0x7f6b08038060 msgr2=0x7f6b08025710 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-05T19:28:27.132 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.122+0000 7f6b1aa70e40 1 --2- 172.21.15.167:0/1447055810 >> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] conn(0x7f6b08038060 0x7f6b08025710 crc :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-05T19:28:27.132 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.122+0000 7f6b1aa70e40 1 -- 172.21.15.167:0/1447055810 >> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] conn(0x5573f482a0b0 msgr2=0x5573f482c490 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-05T19:28:27.132 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.122+0000 7f6b1aa70e40 1 --2- 172.21.15.167:0/1447055810 >> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] conn(0x5573f482a0b0 0x5573f482c490 crc :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-05T19:28:27.132 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.122+0000 7f6b1aa70e40 1 -- 172.21.15.167:0/1447055810 >> [v2:172.21.15.167:6824/29889,v1:172.21.15.167:6825/29889] conn(0x7f6b04030c40 msgr2=0x7f6b040330f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-05T19:28:27.132 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.122+0000 7f6b1aa70e40 1 --2- 172.21.15.167:0/1447055810 >> [v2:172.21.15.167:6824/29889,v1:172.21.15.167:6825/29889] conn(0x7f6b04030c40 0x7f6b040330f0 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f6b0c004170 tx=0x7f6b0c001400 comp rx=0 tx=0).stop 2023-10-05T19:28:27.132 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.122+0000 7f6b18da6700 1 -- 172.21.15.167:0/1447055810 reap_dead start 2023-10-05T19:28:27.132 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.122+0000 7f6b1aa70e40 1 -- 172.21.15.167:0/1447055810 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5573f4767fd0 msgr2=0x5573f481ac10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-05T19:28:27.132 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.122+0000 7f6b1aa70e40 1 --2- 172.21.15.167:0/1447055810 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5573f4767fd0 0x5573f481ac10 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7f6b0800afd0 tx=0x7f6b0801c000 comp rx=0 tx=0).stop 2023-10-05T19:28:27.132 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.122+0000 7f6b1aa70e40 1 -- 172.21.15.167:0/1447055810 shutdown_connections 2023-10-05T19:28:27.133 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.122+0000 7f6b1aa70e40 1 --2- 172.21.15.167:0/1447055810 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5573f4767fd0 0x5573f481ac10 unknown :-1 s=CLOSED pgs=221 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-05T19:28:27.133 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.122+0000 7f6b1aa70e40 1 -- 172.21.15.167:0/1447055810 >> 172.21.15.167:0/1447055810 conn(0x5573f47ee830 msgr2=0x5573f480e590 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-05T19:28:27.133 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.122+0000 7f6b1aa70e40 1 -- 172.21.15.167:0/1447055810 shutdown_connections 2023-10-05T19:28:27.133 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.122+0000 7f6b1aa70e40 1 -- 172.21.15.167:0/1447055810 wait complete. 2023-10-05T19:28:27.133 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.122+0000 7f6b1aa70e40 1 -- 172.21.15.167:0/3597287746 >> [v2:172.21.15.167:6824/29889,v1:172.21.15.167:6825/29889] conn(0x7f6a58030ca0 msgr2=0x7f6a58033150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-05T19:28:27.133 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.122+0000 7f6b1aa70e40 1 --2- 172.21.15.167:0/3597287746 >> [v2:172.21.15.167:6824/29889,v1:172.21.15.167:6825/29889] conn(0x7f6a58030ca0 0x7f6a58033150 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x5573f4833c30 tx=0x7f6b1000aec0 comp rx=0 tx=0).stop 2023-10-05T19:28:27.133 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.122+0000 7f6b1aa70e40 1 -- 172.21.15.167:0/3597287746 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5573f48423b0 msgr2=0x5573f4847f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-05T19:28:27.133 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.122+0000 7f6b1aa70e40 1 --2- 172.21.15.167:0/3597287746 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5573f48423b0 0x5573f4847f50 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x5573f4827ba0 tx=0x7f6b0c030000 comp rx=0 tx=0).stop 2023-10-05T19:28:27.133 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.126+0000 7f6b1aa70e40 1 -- 172.21.15.167:0/3597287746 shutdown_connections 2023-10-05T19:28:27.133 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.126+0000 7f6b1aa70e40 1 --2- 172.21.15.167:0/3597287746 >> [v2:172.21.15.167:6824/29889,v1:172.21.15.167:6825/29889] conn(0x7f6a58030ca0 0x7f6a58033150 unknown :-1 s=CLOSED pgs=81 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-05T19:28:27.133 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.126+0000 7f6b1aa70e40 1 --2- 172.21.15.167:0/3597287746 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5573f48423b0 0x5573f4847f50 unknown :-1 s=CLOSED pgs=223 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-05T19:28:27.134 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.126+0000 7f6b1aa70e40 1 -- 172.21.15.167:0/3597287746 >> 172.21.15.167:0/3597287746 conn(0x5573f483d770 msgr2=0x5573f483b870 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-05T19:28:27.134 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.126+0000 7f6b1aa70e40 1 -- 172.21.15.167:0/3597287746 shutdown_connections 2023-10-05T19:28:27.134 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.126+0000 7f6b1aa70e40 1 -- 172.21.15.167:0/3597287746 wait complete. 2023-10-05T19:28:27.137 DEBUG:tasks.util.rgw: json result: {'user_id': 'foo', 'display_name': 'Foo', 'email': 'foo@foo.com', 'suspended': 0, 'max_buckets': 4, 'subusers': [], 'keys': [{'user': 'foo', 'access_key': '9te6NH5mcdcq0Tc5i8i1', 'secret_key': 'Ny4IOauQoL18Gp2zM7lC1vLmoawgqcYP/YGcWfXu'}], 'swift_keys': [], 'caps': [], 'op_mask': 'read, write, delete', 'default_placement': '', 'default_storage_class': '', 'placement_tags': [], 'bucket_quota': {'enabled': False, 'check_on_raw': False, 'max_size': -1, 'max_size_kb': 0, 'max_objects': -1}, 'user_quota': {'enabled': False, 'check_on_raw': False, 'max_size': -1, 'max_size_kb': 0, 'max_objects': -1}, 'temp_url_keys': [], 'type': 'rgw', 'mfa_ids': []} 2023-10-05T19:28:27.137 INFO:tasks.util.rgw:rgwadmin: client.0 : ['key', 'create', '--uid', 'foo', '--access-key', 'p5YnriCv1nAtykxBrupQ', '--secret', 'Q8Tk6Q/27hfbFSYdSkPtUqhqx1GgzvpXa4WARozh'] 2023-10-05T19:28:27.137 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'ceph', 'key', 'create', '--uid', 'foo', '--access-key', 'p5YnriCv1nAtykxBrupQ', '--secret', 'Q8Tk6Q/27hfbFSYdSkPtUqhqx1GgzvpXa4WARozh'] 2023-10-05T19:28:27.137 DEBUG:teuthology.orchestra.run.smithi167:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster ceph key create --uid foo --access-key p5YnriCv1nAtykxBrupQ --secret Q8Tk6Q/27hfbFSYdSkPtUqhqx1GgzvpXa4WARozh 2023-10-05T19:28:27.180 INFO:teuthology.orchestra.run.smithi167.stderr:ignoring --setuser ceph since I am not root 2023-10-05T19:28:27.181 INFO:teuthology.orchestra.run.smithi167.stderr:ignoring --setgroup ceph since I am not root 2023-10-05T19:28:27.182 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.174+0000 7f92a8e6ce40 1 Processor -- start 2023-10-05T19:28:27.182 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.174+0000 7f92a8e6ce40 1 -- start start 2023-10-05T19:28:27.182 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.174+0000 7f92a8e6ce40 1 --2- >> v2:172.21.15.118:3300/0 conn(0x556e04bbb3b0 0x556e04bbd790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-05T19:28:27.182 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.174+0000 7f92a8e6ce40 1 -- --> v1:172.21.15.118:6789/0 -- auth(proto 0 26 bytes epoch 0) v1 -- 0x556e04b03bd0 con 0x556e04bb6dd0 2023-10-05T19:28:27.182 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.174+0000 7f92a8e6ce40 1 -- --> v2:172.21.15.118:3300/0 -- mon_getmap magic: 0 v1 -- 0x556e04ac0ee0 con 0x556e04bbb3b0 2023-10-05T19:28:27.182 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.174+0000 7f92a61a0700 1 --2- >> v2:172.21.15.118:3300/0 conn(0x556e04bbb3b0 0x556e04bbd790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-05T19:28:27.182 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.178+0000 7f92a61a0700 1 --2- >> v2:172.21.15.118:3300/0 conn(0x556e04bbb3b0 0x556e04bbd790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.118:3300/0 says I am v2:172.21.15.167:57454/0 (socket says 172.21.15.167:57454) 2023-10-05T19:28:27.182 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.178+0000 7f92a61a0700 1 -- 172.21.15.167:0/732317912 learned_addr learned my addr 172.21.15.167:0/732317912 (peer_addr_for_me v2:172.21.15.167:0/0) 2023-10-05T19:28:27.184 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.178+0000 7f92a61a0700 1 -- 172.21.15.167:0/732317912 >> v1:172.21.15.118:6789/0 conn(0x556e04bb6dd0 legacy=0x556e04bb71a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-05T19:28:27.184 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.178+0000 7f92a61a0700 1 -- 172.21.15.167:0/732317912 --> v2:172.21.15.118:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x556e04bb79e0 con 0x556e04bbb3b0 2023-10-05T19:28:27.184 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.178+0000 7f92a61a0700 1 --2- 172.21.15.167:0/732317912 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x556e04bbb3b0 0x556e04bbd790 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7f92a000a5c0 tx=0x7f92a0005e00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2a397fbe20fdd588 server_cookie=0 in_seq=0 out_seq=0 2023-10-05T19:28:27.184 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.178+0000 7f92a599f700 1 -- 172.21.15.167:0/732317912 <== mon.0 v2:172.21.15.118:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f92a000e510 con 0x556e04bbb3b0 2023-10-05T19:28:27.184 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.178+0000 7f92a599f700 1 -- 172.21.15.167:0/732317912 <== mon.0 v2:172.21.15.118:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f92a000e670 con 0x556e04bbb3b0 2023-10-05T19:28:27.184 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.178+0000 7f92a599f700 1 -- 172.21.15.167:0/732317912 <== mon.0 v2:172.21.15.118:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f92a000e920 con 0x556e04bbb3b0 2023-10-05T19:28:27.184 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.178+0000 7f92a8e6ce40 1 -- 172.21.15.167:0/732317912 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x556e04bbb3b0 msgr2=0x556e04bbd790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-05T19:28:27.184 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.178+0000 7f92a8e6ce40 1 --2- 172.21.15.167:0/732317912 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x556e04bbb3b0 0x556e04bbd790 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7f92a000a5c0 tx=0x7f92a0005e00 comp rx=0 tx=0).stop 2023-10-05T19:28:27.184 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.178+0000 7f92a8e6ce40 1 -- 172.21.15.167:0/732317912 shutdown_connections 2023-10-05T19:28:27.184 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.178+0000 7f92a8e6ce40 1 --2- 172.21.15.167:0/732317912 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x556e04bbb3b0 0x556e04bbd790 unknown :-1 s=CLOSED pgs=225 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-05T19:28:27.184 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.178+0000 7f92a8e6ce40 1 -- 172.21.15.167:0/732317912 >> 172.21.15.167:0/732317912 conn(0x556e04bb27f0 msgr2=0x556e04bb4c00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-05T19:28:27.185 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.178+0000 7f92a8e6ce40 1 -- 172.21.15.167:0/732317912 shutdown_connections 2023-10-05T19:28:27.185 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.178+0000 7f92a8e6ce40 1 -- 172.21.15.167:0/732317912 wait complete. 2023-10-05T19:28:27.186 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.178+0000 7f92a8e6ce40 1 Processor -- start 2023-10-05T19:28:27.186 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.178+0000 7f92a8e6ce40 1 -- start start 2023-10-05T19:28:27.186 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.178+0000 7f92a8e6ce40 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x556e04bb6dd0 0x556e04bd6050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-05T19:28:27.186 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.178+0000 7f92a8e6ce40 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x556e04a84e30 con 0x556e04bb6dd0 2023-10-05T19:28:27.186 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.178+0000 7f92a69a1700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x556e04bb6dd0 0x556e04bd6050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-05T19:28:27.186 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.178+0000 7f92a69a1700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x556e04bb6dd0 0x556e04bd6050 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.118:3300/0 says I am v2:172.21.15.167:57466/0 (socket says 172.21.15.167:57466) 2023-10-05T19:28:27.186 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.178+0000 7f92a69a1700 1 -- 172.21.15.167:0/1388536432 learned_addr learned my addr 172.21.15.167:0/1388536432 (peer_addr_for_me v2:172.21.15.167:0/0) 2023-10-05T19:28:27.186 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.178+0000 7f92a69a1700 1 -- 172.21.15.167:0/1388536432 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f92a000a180 con 0x556e04bb6dd0 2023-10-05T19:28:27.187 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.182+0000 7f92a69a1700 1 --2- 172.21.15.167:0/1388536432 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x556e04bb6dd0 0x556e04bd6050 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7f9298001d30 tx=0x7f9298001900 comp rx=0 tx=0).ready entity=mon.0 client_cookie=747310428a967a6a server_cookie=0 in_seq=0 out_seq=0 2023-10-05T19:28:27.188 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.182+0000 7f92a499d700 1 -- 172.21.15.167:0/1388536432 <== mon.0 v2:172.21.15.118:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f9298014070 con 0x556e04bb6dd0 2023-10-05T19:28:27.188 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.182+0000 7f92a499d700 1 -- 172.21.15.167:0/1388536432 <== mon.0 v2:172.21.15.118:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9298008070 con 0x556e04bb6dd0 2023-10-05T19:28:27.188 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.182+0000 7f92a499d700 1 -- 172.21.15.167:0/1388536432 <== mon.0 v2:172.21.15.118:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f929800bd30 con 0x556e04bb6dd0 2023-10-05T19:28:27.188 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.182+0000 7f92a8e6ce40 1 -- 172.21.15.167:0/1388536432 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x556e04bb6dd0 msgr2=0x556e04bd6050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-05T19:28:27.188 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.182+0000 7f92a8e6ce40 1 --2- 172.21.15.167:0/1388536432 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x556e04bb6dd0 0x556e04bd6050 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7f9298001d30 tx=0x7f9298001900 comp rx=0 tx=0).stop 2023-10-05T19:28:27.188 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.182+0000 7f92a8e6ce40 1 -- 172.21.15.167:0/1388536432 shutdown_connections 2023-10-05T19:28:27.189 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.182+0000 7f92a8e6ce40 1 --2- 172.21.15.167:0/1388536432 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x556e04bb6dd0 0x556e04bd6050 unknown :-1 s=CLOSED pgs=226 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-05T19:28:27.189 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.182+0000 7f92a8e6ce40 1 -- 172.21.15.167:0/1388536432 >> 172.21.15.167:0/1388536432 conn(0x556e04bb27f0 msgr2=0x556e04bd1b90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-05T19:28:27.189 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.182+0000 7f92a8e6ce40 1 -- 172.21.15.167:0/1388536432 shutdown_connections 2023-10-05T19:28:27.189 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.182+0000 7f92a8e6ce40 1 -- 172.21.15.167:0/1388536432 wait complete. 2023-10-05T19:28:27.189 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.182+0000 7f92a8e6ce40 1 Processor -- start 2023-10-05T19:28:27.189 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.182+0000 7f92a8e6ce40 1 -- start start 2023-10-05T19:28:27.189 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.182+0000 7f92a8e6ce40 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x556e04bb6dd0 0x556e04b2ca90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-05T19:28:27.189 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.182+0000 7f92a8e6ce40 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x556e04a57ed0 con 0x556e04bb6dd0 2023-10-05T19:28:27.189 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.182+0000 7f92a69a1700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x556e04bb6dd0 0x556e04b2ca90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-05T19:28:27.189 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.182+0000 7f92a69a1700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x556e04bb6dd0 0x556e04b2ca90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.118:3300/0 says I am v2:172.21.15.167:57470/0 (socket says 172.21.15.167:57470) 2023-10-05T19:28:27.189 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.182+0000 7f92a69a1700 1 -- 172.21.15.167:0/1223402549 learned_addr learned my addr 172.21.15.167:0/1223402549 (peer_addr_for_me v2:172.21.15.167:0/0) 2023-10-05T19:28:27.190 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.182+0000 7f92a69a1700 1 -- 172.21.15.167:0/1223402549 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f929800a040 con 0x556e04bb6dd0 2023-10-05T19:28:27.190 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.182+0000 7f92a69a1700 1 --2- 172.21.15.167:0/1223402549 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x556e04bb6dd0 0x556e04b2ca90 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x556e04bd26b0 tx=0x7f9298016000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-05T19:28:27.190 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.182+0000 7f929effd700 1 -- 172.21.15.167:0/1223402549 <== mon.0 v2:172.21.15.118:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f929800bee0 con 0x556e04bb6dd0 2023-10-05T19:28:27.190 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.182+0000 7f929effd700 1 -- 172.21.15.167:0/1223402549 <== mon.0 v2:172.21.15.118:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9298021070 con 0x556e04bb6dd0 2023-10-05T19:28:27.191 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.182+0000 7f92a8e6ce40 1 -- 172.21.15.167:0/1223402549 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x556e04b2cfd0 con 0x556e04bb6dd0 2023-10-05T19:28:27.191 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.182+0000 7f929effd700 1 -- 172.21.15.167:0/1223402549 <== mon.0 v2:172.21.15.118:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f929801a600 con 0x556e04bb6dd0 2023-10-05T19:28:27.191 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.182+0000 7f92a8e6ce40 1 -- 172.21.15.167:0/1223402549 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x556e04b2b390 con 0x556e04bb6dd0 2023-10-05T19:28:27.191 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.186+0000 7f92a8e6ce40 20 rados->read ofs=0 len=0 2023-10-05T19:28:27.192 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.186+0000 7f929effd700 1 -- 172.21.15.167:0/1223402549 <== mon.0 v2:172.21.15.118:3300/0 4 ==== mgrmap(e 5) v1 ==== 40329+0+0 (secure 0 0 0) 0x7f929801a800 con 0x556e04bb6dd0 2023-10-05T19:28:27.192 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.186+0000 7f929effd700 1 --2- 172.21.15.167:0/1223402549 >> [v2:172.21.15.167:6824/29889,v1:172.21.15.167:6825/29889] conn(0x7f9288030c00 0x7f92880330b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-05T19:28:27.192 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.186+0000 7f929effd700 1 -- 172.21.15.167:0/1223402549 <== mon.0 v2:172.21.15.118:3300/0 5 ==== osd_map(30..30 src has 1..30) v4 ==== 6057+0+0 (secure 0 0 0) 0x7f9298044df0 con 0x556e04bb6dd0 2023-10-05T19:28:27.192 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.186+0000 7f92a8e6ce40 1 --2- 172.21.15.167:0/1223402549 >> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] conn(0x556e04bbb3b0 0x556e04beac20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-05T19:28:27.193 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.186+0000 7f92a61a0700 1 --2- 172.21.15.167:0/1223402549 >> [v2:172.21.15.167:6824/29889,v1:172.21.15.167:6825/29889] conn(0x7f9288030c00 0x7f92880330b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-05T19:28:27.193 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.186+0000 7f92a8e6ce40 1 -- 172.21.15.167:0/1223402549 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:1 5.12 5:49953fa1:::default.realm:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x556e04beb240 con 0x556e04bbb3b0 2023-10-05T19:28:27.193 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.186+0000 7f92a71a2700 1 --2- 172.21.15.167:0/1223402549 >> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] conn(0x556e04bbb3b0 0x556e04beac20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-05T19:28:27.193 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.186+0000 7f92a61a0700 1 --2- 172.21.15.167:0/1223402549 >> [v2:172.21.15.167:6824/29889,v1:172.21.15.167:6825/29889] conn(0x7f9288030c00 0x7f92880330b0 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f92a0000f80 tx=0x7f92a0013000 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-05T19:28:27.193 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.186+0000 7f92a71a2700 1 --2- 172.21.15.167:0/1223402549 >> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] conn(0x556e04bbb3b0 0x556e04beac20 crc :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-05T19:28:27.194 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.186+0000 7f92a71a2700 1 -- 172.21.15.167:0/1223402549 <== osd.1 v2:172.21.15.167:6800/30035 1 ==== osd_op_reply(1 default.realm [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 ==== 157+0+0 (crc 0 0 0) 0x7f929000a500 con 0x556e04bbb3b0 2023-10-05T19:28:27.194 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.190+0000 7f92a8e6ce40 20 rados_obj.operate() r=-2 bl.length=0 2023-10-05T19:28:27.194 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.190+0000 7f92a8e6ce40 20 realm 2023-10-05T19:28:27.194 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.190+0000 7f92a8e6ce40 20 rados->read ofs=0 len=0 2023-10-05T19:28:27.194 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.190+0000 7f92a8e6ce40 1 -- 172.21.15.167:0/1223402549 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:2 5.12 5:49953fa1:::default.realm:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x556e04be8280 con 0x556e04bbb3b0 2023-10-05T19:28:27.195 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.190+0000 7f92a71a2700 1 -- 172.21.15.167:0/1223402549 <== osd.1 v2:172.21.15.167:6800/30035 2 ==== osd_op_reply(2 default.realm [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 ==== 157+0+0 (crc 0 0 0) 0x7f929000a500 con 0x556e04bbb3b0 2023-10-05T19:28:27.195 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.190+0000 7f92a8e6ce40 20 rados_obj.operate() r=-2 bl.length=0 2023-10-05T19:28:27.195 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.190+0000 7f92a8e6ce40 4 RGWPeriod::init failed to init realm id : (2) No such file or directory 2023-10-05T19:28:27.195 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.190+0000 7f92a8e6ce40 20 rados->read ofs=0 len=0 2023-10-05T19:28:27.196 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.190+0000 7f92a8e6ce40 1 -- 172.21.15.167:0/1223402549 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:3 5.12 5:49953fa1:::default.realm:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x556e04bebcf0 con 0x556e04bbb3b0 2023-10-05T19:28:27.196 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.190+0000 7f92a71a2700 1 -- 172.21.15.167:0/1223402549 <== osd.1 v2:172.21.15.167:6800/30035 3 ==== osd_op_reply(3 default.realm [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 ==== 157+0+0 (crc 0 0 0) 0x7f929000a500 con 0x556e04bbb3b0 2023-10-05T19:28:27.196 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.190+0000 7f92a8e6ce40 20 rados_obj.operate() r=-2 bl.length=0 2023-10-05T19:28:27.196 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.190+0000 7f92a8e6ce40 20 rados->read ofs=0 len=0 2023-10-05T19:28:27.196 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.190+0000 7f92a8e6ce40 1 -- 172.21.15.167:0/1223402549 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:4 5.f 5:f43b6ece:::zone_names.default:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x556e04beb900 con 0x556e04bbb3b0 2023-10-05T19:28:27.196 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.190+0000 7f92a71a2700 1 -- 172.21.15.167:0/1223402549 <== osd.1 v2:172.21.15.167:6800/30035 4 ==== osd_op_reply(4 zone_names.default [read 0~46 out=46b] v0'0 uv1 ondisk = 0) v8 ==== 162+0+46 (crc 0 0 0) 0x7f929000a500 con 0x556e04bbb3b0 2023-10-05T19:28:27.197 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.190+0000 7f92a8e6ce40 20 rados_obj.operate() r=0 bl.length=46 2023-10-05T19:28:27.197 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.190+0000 7f92a8e6ce40 20 rados->read ofs=0 len=0 2023-10-05T19:28:27.197 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.190+0000 7f92a8e6ce40 1 -- 172.21.15.167:0/1223402549 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:5 5.1d 5:b8374eb4:::zone_info.9a9df965-6612-45d4-a45f-6e3c889bacc2:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x556e04becb80 con 0x556e04bbb3b0 2023-10-05T19:28:27.198 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.190+0000 7f92a71a2700 1 -- 172.21.15.167:0/1223402549 <== osd.1 v2:172.21.15.167:6800/30035 5 ==== osd_op_reply(5 zone_info.9a9df965-6612-45d4-a45f-6e3c889bacc2 [read 0~874 out=874b] v0'0 uv2 ondisk = 0) v8 ==== 190+0+874 (crc 0 0 0) 0x7f929000a500 con 0x556e04bbb3b0 2023-10-05T19:28:27.198 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.190+0000 7f92a8e6ce40 20 rados_obj.operate() r=0 bl.length=874 2023-10-05T19:28:27.198 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.190+0000 7f92a8e6ce40 20 searching for the correct realm 2023-10-05T19:28:27.198 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.190+0000 7f92a8e6ce40 1 --2- 172.21.15.167:0/1223402549 >> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] conn(0x556e04bee940 0x556e04bf0d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-05T19:28:27.198 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.190+0000 7f92a8e6ce40 1 -- 172.21.15.167:0/1223402549 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:6 5.0 5:00000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x556e04bf1260 con 0x556e04bee940 2023-10-05T19:28:27.198 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.190+0000 7f92a69a1700 1 --2- 172.21.15.167:0/1223402549 >> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] conn(0x556e04bee940 0x556e04bf0d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-05T19:28:27.198 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.194+0000 7f92a69a1700 1 --2- 172.21.15.167:0/1223402549 >> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] conn(0x556e04bee940 0x556e04bf0d20 crc :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-05T19:28:27.201 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.194+0000 7f92a69a1700 1 -- 172.21.15.167:0/1223402549 <== osd.3 v2:172.21.15.167:6808/30036 1 ==== osd_op_reply(6 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f9298016070 con 0x556e04bee940 2023-10-05T19:28:27.201 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.194+0000 7f92a69a1700 1 --2- 172.21.15.167:0/1223402549 >> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] conn(0x7f92980176e0 0x7f929803d580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-05T19:28:27.201 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.194+0000 7f92a69a1700 1 -- 172.21.15.167:0/1223402549 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:7 5.10 5:08000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x556e04bf1260 con 0x7f92980176e0 2023-10-05T19:28:27.201 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.194+0000 7f92a61a0700 1 --2- 172.21.15.167:0/1223402549 >> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] conn(0x7f92980176e0 0x7f929803d580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-05T19:28:27.202 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.194+0000 7f92a61a0700 1 --2- 172.21.15.167:0/1223402549 >> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] conn(0x7f92980176e0 0x7f929803d580 crc :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-05T19:28:27.202 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.194+0000 7f92a61a0700 1 -- 172.21.15.167:0/1223402549 <== osd.2 v2:172.21.15.167:6816/30037 1 ==== osd_op_reply(7 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f92a0017050 con 0x7f92980176e0 2023-10-05T19:28:27.202 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.194+0000 7f92a61a0700 1 --2- 172.21.15.167:0/1223402549 >> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] conn(0x7f92a0010070 0x7f92a001c430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-05T19:28:27.202 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.194+0000 7f92a61a0700 1 -- 172.21.15.167:0/1223402549 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:8 5.8 5:10000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x556e04bf1260 con 0x7f92a0010070 2023-10-05T19:28:27.202 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.198+0000 7f92a71a2700 1 --2- 172.21.15.167:0/1223402549 >> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] conn(0x7f92a0010070 0x7f92a001c430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-05T19:28:27.203 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.198+0000 7f92a71a2700 1 --2- 172.21.15.167:0/1223402549 >> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] conn(0x7f92a0010070 0x7f92a001c430 crc :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-05T19:28:27.203 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.198+0000 7f92a71a2700 1 -- 172.21.15.167:0/1223402549 <== osd.0 v2:172.21.15.118:6800/27690 1 ==== osd_op_reply(8 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f9290014040 con 0x7f92a0010070 2023-10-05T19:28:27.204 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.198+0000 7f92a71a2700 1 -- 172.21.15.167:0/1223402549 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:9 5.18 5:18000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x556e04bf1260 con 0x556e04bbb3b0 2023-10-05T19:28:27.204 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.198+0000 7f92a71a2700 1 -- 172.21.15.167:0/1223402549 <== osd.1 v2:172.21.15.167:6800/30035 6 ==== osd_op_reply(9 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f9290014040 con 0x556e04bbb3b0 2023-10-05T19:28:27.204 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.198+0000 7f92a71a2700 1 -- 172.21.15.167:0/1223402549 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:10 5.4 5:20000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x556e04bf1260 con 0x556e04bbb3b0 2023-10-05T19:28:27.205 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.198+0000 7f92a71a2700 1 -- 172.21.15.167:0/1223402549 <== osd.1 v2:172.21.15.167:6800/30035 7 ==== osd_op_reply(10 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f9290014040 con 0x556e04bbb3b0 2023-10-05T19:28:27.205 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.198+0000 7f92a71a2700 1 -- 172.21.15.167:0/1223402549 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:11 5.14 5:28000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x556e04bf1260 con 0x7f92a0010070 2023-10-05T19:28:27.205 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.198+0000 7f92a71a2700 1 -- 172.21.15.167:0/1223402549 <== osd.0 v2:172.21.15.118:6800/27690 2 ==== osd_op_reply(11 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f9290014040 con 0x7f92a0010070 2023-10-05T19:28:27.206 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.198+0000 7f92a71a2700 1 -- 172.21.15.167:0/1223402549 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:12 5.c 5:30000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x556e04bf1260 con 0x7f92a0010070 2023-10-05T19:28:27.206 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.202+0000 7f92a71a2700 1 -- 172.21.15.167:0/1223402549 <== osd.0 v2:172.21.15.118:6800/27690 3 ==== osd_op_reply(12 [pgnls start_epoch 30 out=112b] v23'1 uv1 ondisk = 1) v8 ==== 144+0+112 (crc 0 0 0) 0x7f9290014040 con 0x7f92a0010070 2023-10-05T19:28:27.206 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.202+0000 7f92a71a2700 1 -- 172.21.15.167:0/1223402549 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:13 5.1c 5:38000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x556e04bf1260 con 0x7f92980176e0 2023-10-05T19:28:27.207 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.202+0000 7f92a61a0700 1 -- 172.21.15.167:0/1223402549 <== osd.2 v2:172.21.15.167:6816/30037 2 ==== osd_op_reply(13 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f92a0017050 con 0x7f92980176e0 2023-10-05T19:28:27.207 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.202+0000 7f92a61a0700 1 -- 172.21.15.167:0/1223402549 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:14 5.2 5:40000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x556e04bf1260 con 0x556e04bee940 2023-10-05T19:28:27.208 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.202+0000 7f92a69a1700 1 -- 172.21.15.167:0/1223402549 <== osd.3 v2:172.21.15.167:6808/30036 2 ==== osd_op_reply(14 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f9298016070 con 0x556e04bee940 2023-10-05T19:28:27.208 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.202+0000 7f92a69a1700 1 -- 172.21.15.167:0/1223402549 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:15 5.12 5:48000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x556e04bf1260 con 0x556e04bbb3b0 2023-10-05T19:28:27.209 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.202+0000 7f92a71a2700 1 -- 172.21.15.167:0/1223402549 <== osd.1 v2:172.21.15.167:6800/30035 8 ==== osd_op_reply(15 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f9290014040 con 0x556e04bbb3b0 2023-10-05T19:28:27.209 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.202+0000 7f92a71a2700 1 -- 172.21.15.167:0/1223402549 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:16 5.a 5:50000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x556e04bf1260 con 0x7f92a0010070 2023-10-05T19:28:27.209 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.202+0000 7f92a71a2700 1 -- 172.21.15.167:0/1223402549 <== osd.0 v2:172.21.15.118:6800/27690 4 ==== osd_op_reply(16 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f9290022040 con 0x7f92a0010070 2023-10-05T19:28:27.209 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.202+0000 7f92a71a2700 1 -- 172.21.15.167:0/1223402549 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:17 5.1a 5:58000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x556e04bf1260 con 0x556e04bbb3b0 2023-10-05T19:28:27.210 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.206+0000 7f92a71a2700 1 -- 172.21.15.167:0/1223402549 <== osd.1 v2:172.21.15.167:6800/30035 9 ==== osd_op_reply(17 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f9290022040 con 0x556e04bbb3b0 2023-10-05T19:28:27.210 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.206+0000 7f92a71a2700 1 -- 172.21.15.167:0/1223402549 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:18 5.6 5:60000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x556e04bf1260 con 0x7f92980176e0 2023-10-05T19:28:27.211 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.206+0000 7f92a61a0700 1 -- 172.21.15.167:0/1223402549 <== osd.2 v2:172.21.15.167:6816/30037 3 ==== osd_op_reply(18 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f92a0017050 con 0x7f92980176e0 2023-10-05T19:28:27.211 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.206+0000 7f92a61a0700 1 -- 172.21.15.167:0/1223402549 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:19 5.16 5:68000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x556e04bf1260 con 0x556e04bee940 2023-10-05T19:28:27.211 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.206+0000 7f92a69a1700 1 -- 172.21.15.167:0/1223402549 <== osd.3 v2:172.21.15.167:6808/30036 3 ==== osd_op_reply(19 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f9298016070 con 0x556e04bee940 2023-10-05T19:28:27.211 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.206+0000 7f92a69a1700 1 -- 172.21.15.167:0/1223402549 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:20 5.e 5:70000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x556e04bf1260 con 0x556e04bee940 2023-10-05T19:28:27.212 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.206+0000 7f92a69a1700 1 -- 172.21.15.167:0/1223402549 <== osd.3 v2:172.21.15.167:6808/30036 4 ==== osd_op_reply(20 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f9298016070 con 0x556e04bee940 2023-10-05T19:28:27.212 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.206+0000 7f92a69a1700 1 -- 172.21.15.167:0/1223402549 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:21 5.1e 5:78000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x556e04bf1260 con 0x7f92a0010070 2023-10-05T19:28:27.213 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.206+0000 7f92a71a2700 1 -- 172.21.15.167:0/1223402549 <== osd.0 v2:172.21.15.118:6800/27690 5 ==== osd_op_reply(21 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f9290022040 con 0x7f92a0010070 2023-10-05T19:28:27.213 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.206+0000 7f92a71a2700 1 -- 172.21.15.167:0/1223402549 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:22 5.1 5:80000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x556e04bf1260 con 0x556e04bee940 2023-10-05T19:28:27.213 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.206+0000 7f92a69a1700 1 -- 172.21.15.167:0/1223402549 <== osd.3 v2:172.21.15.167:6808/30036 5 ==== osd_op_reply(22 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f9298039040 con 0x556e04bee940 2023-10-05T19:28:27.213 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.206+0000 7f92a69a1700 1 -- 172.21.15.167:0/1223402549 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:23 5.11 5:88000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x556e04bf1260 con 0x556e04bee940 2023-10-05T19:28:27.214 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.210+0000 7f92a69a1700 1 -- 172.21.15.167:0/1223402549 <== osd.3 v2:172.21.15.167:6808/30036 6 ==== osd_op_reply(23 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f9298039040 con 0x556e04bee940 2023-10-05T19:28:27.214 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.210+0000 7f92a69a1700 1 -- 172.21.15.167:0/1223402549 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:24 5.9 5:90000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x7f929803d040 con 0x556e04bbb3b0 2023-10-05T19:28:27.215 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.210+0000 7f92a71a2700 1 -- 172.21.15.167:0/1223402549 <== osd.1 v2:172.21.15.167:6800/30035 10 ==== osd_op_reply(24 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f9290022040 con 0x556e04bbb3b0 2023-10-05T19:28:27.215 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.210+0000 7f92a71a2700 1 -- 172.21.15.167:0/1223402549 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:25 5.19 5:98000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x7f929803d040 con 0x556e04bbb3b0 2023-10-05T19:28:27.216 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.210+0000 7f92a71a2700 1 -- 172.21.15.167:0/1223402549 <== osd.1 v2:172.21.15.167:6800/30035 11 ==== osd_op_reply(25 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f9290022040 con 0x556e04bbb3b0 2023-10-05T19:28:27.216 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.210+0000 7f92a71a2700 1 -- 172.21.15.167:0/1223402549 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:26 5.5 5:a0000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x7f929803d040 con 0x7f92a0010070 2023-10-05T19:28:27.216 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.210+0000 7f92a71a2700 1 -- 172.21.15.167:0/1223402549 <== osd.0 v2:172.21.15.118:6800/27690 6 ==== osd_op_reply(26 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f9290022040 con 0x7f92a0010070 2023-10-05T19:28:27.217 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.210+0000 7f92a71a2700 1 -- 172.21.15.167:0/1223402549 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:27 5.15 5:a8000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x7f929803d040 con 0x556e04bee940 2023-10-05T19:28:27.217 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.210+0000 7f92a69a1700 1 -- 172.21.15.167:0/1223402549 <== osd.3 v2:172.21.15.167:6808/30036 7 ==== osd_op_reply(27 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f9298039040 con 0x556e04bee940 2023-10-05T19:28:27.218 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.210+0000 7f92a69a1700 1 -- 172.21.15.167:0/1223402549 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:28 5.d 5:b0000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x7f929803d040 con 0x7f92980176e0 2023-10-05T19:28:27.218 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.214+0000 7f92a61a0700 1 -- 172.21.15.167:0/1223402549 <== osd.2 v2:172.21.15.167:6816/30037 4 ==== osd_op_reply(28 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f92a0017050 con 0x7f92980176e0 2023-10-05T19:28:27.218 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.214+0000 7f92a61a0700 1 -- 172.21.15.167:0/1223402549 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:29 5.1d 5:b8000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x7f929803d040 con 0x556e04bbb3b0 2023-10-05T19:28:27.219 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.214+0000 7f92a71a2700 1 -- 172.21.15.167:0/1223402549 <== osd.1 v2:172.21.15.167:6800/30035 12 ==== osd_op_reply(29 [pgnls start_epoch 30 out=107b] v30'2 uv2 ondisk = 1) v8 ==== 144+0+107 (crc 0 0 0) 0x7f9290022040 con 0x556e04bbb3b0 2023-10-05T19:28:27.219 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.214+0000 7f92a71a2700 1 -- 172.21.15.167:0/1223402549 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:30 5.3 5:c0000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x7f929803d040 con 0x7f92a0010070 2023-10-05T19:28:27.220 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.214+0000 7f92a71a2700 1 -- 172.21.15.167:0/1223402549 <== osd.0 v2:172.21.15.118:6800/27690 7 ==== osd_op_reply(30 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f9290022040 con 0x7f92a0010070 2023-10-05T19:28:27.220 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.214+0000 7f92a71a2700 1 -- 172.21.15.167:0/1223402549 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:31 5.13 5:c8000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x7f929803d040 con 0x556e04bee940 2023-10-05T19:28:27.221 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.214+0000 7f92a69a1700 1 -- 172.21.15.167:0/1223402549 <== osd.3 v2:172.21.15.167:6808/30036 8 ==== osd_op_reply(31 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f9298039040 con 0x556e04bee940 2023-10-05T19:28:27.221 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.214+0000 7f92a69a1700 1 -- 172.21.15.167:0/1223402549 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:32 5.b 5:d0000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x7f929803d040 con 0x7f92a0010070 2023-10-05T19:28:27.222 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.218+0000 7f92a71a2700 1 -- 172.21.15.167:0/1223402549 <== osd.0 v2:172.21.15.118:6800/27690 8 ==== osd_op_reply(32 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f9290022040 con 0x7f92a0010070 2023-10-05T19:28:27.222 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.218+0000 7f92a71a2700 1 -- 172.21.15.167:0/1223402549 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:33 5.1b 5:d8000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x7f929803d040 con 0x7f92980176e0 2023-10-05T19:28:27.223 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.218+0000 7f92a61a0700 1 -- 172.21.15.167:0/1223402549 <== osd.2 v2:172.21.15.167:6816/30037 5 ==== osd_op_reply(33 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f92a0017050 con 0x7f92980176e0 2023-10-05T19:28:27.223 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.218+0000 7f92a61a0700 1 -- 172.21.15.167:0/1223402549 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:34 5.7 5:e0000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x7f929803d040 con 0x556e04bee940 2023-10-05T19:28:27.224 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.218+0000 7f92a69a1700 1 -- 172.21.15.167:0/1223402549 <== osd.3 v2:172.21.15.167:6808/30036 9 ==== osd_op_reply(34 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f9298039040 con 0x556e04bee940 2023-10-05T19:28:27.224 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.218+0000 7f92a69a1700 1 -- 172.21.15.167:0/1223402549 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:35 5.17 5:e8000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x7f929803d040 con 0x7f92a0010070 2023-10-05T19:28:27.225 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.218+0000 7f92a71a2700 1 -- 172.21.15.167:0/1223402549 <== osd.0 v2:172.21.15.118:6800/27690 9 ==== osd_op_reply(35 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f9290022040 con 0x7f92a0010070 2023-10-05T19:28:27.225 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.218+0000 7f92a71a2700 1 -- 172.21.15.167:0/1223402549 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:36 5.f 5:f0000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x7f929803d040 con 0x556e04bbb3b0 2023-10-05T19:28:27.226 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.218+0000 7f92a71a2700 1 -- 172.21.15.167:0/1223402549 <== osd.1 v2:172.21.15.167:6800/30035 13 ==== osd_op_reply(36 [pgnls start_epoch 30 out=115b] v23'2 uv2 ondisk = 1) v8 ==== 144+0+115 (crc 0 0 0) 0x7f929002f080 con 0x556e04bbb3b0 2023-10-05T19:28:27.226 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.222+0000 7f92a71a2700 1 -- 172.21.15.167:0/1223402549 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:37 5.1f 5:f8000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x7f929803d040 con 0x7f92980176e0 2023-10-05T19:28:27.227 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.222+0000 7f92a61a0700 1 -- 172.21.15.167:0/1223402549 <== osd.2 v2:172.21.15.167:6816/30037 6 ==== osd_op_reply(37 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f92a0017050 con 0x7f92980176e0 2023-10-05T19:28:27.227 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.222+0000 7f92a8e6ce40 20 RGWRados::pool_iterate: got zonegroup_info.4f1f719d-9648-417b-b10c-e5d939825419 2023-10-05T19:28:27.227 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.222+0000 7f92a8e6ce40 20 RGWRados::pool_iterate: got zone_info.9a9df965-6612-45d4-a45f-6e3c889bacc2 2023-10-05T19:28:27.227 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.222+0000 7f92a8e6ce40 20 RGWRados::pool_iterate: got zone_names.default 2023-10-05T19:28:27.227 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.222+0000 7f92a8e6ce40 20 RGWRados::pool_iterate: got zonegroups_names.default 2023-10-05T19:28:27.227 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.222+0000 7f92a8e6ce40 20 rados->read ofs=0 len=0 2023-10-05T19:28:27.227 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.222+0000 7f92a8e6ce40 1 -- 172.21.15.167:0/1223402549 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:38 5.12 5:49953fa1:::default.realm:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x556e04bed6f0 con 0x556e04bbb3b0 2023-10-05T19:28:27.228 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.222+0000 7f92a71a2700 1 -- 172.21.15.167:0/1223402549 <== osd.1 v2:172.21.15.167:6800/30035 14 ==== osd_op_reply(38 default.realm [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 ==== 157+0+0 (crc 0 0 0) 0x7f929002f080 con 0x556e04bbb3b0 2023-10-05T19:28:27.228 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.222+0000 7f92a8e6ce40 20 rados_obj.operate() r=-2 bl.length=0 2023-10-05T19:28:27.229 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.222+0000 7f92a8e6ce40 20 rados->read ofs=0 len=0 2023-10-05T19:28:27.229 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.222+0000 7f92a8e6ce40 1 -- 172.21.15.167:0/1223402549 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:39 5.f 5:f4c53578:::zonegroups_names.default:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x556e04bedfe0 con 0x556e04bbb3b0 2023-10-05T19:28:27.229 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.222+0000 7f92a71a2700 1 -- 172.21.15.167:0/1223402549 <== osd.1 v2:172.21.15.167:6800/30035 15 ==== osd_op_reply(39 zonegroups_names.default [read 0~46 out=46b] v0'0 uv2 ondisk = 0) v8 ==== 168+0+46 (crc 0 0 0) 0x7f929002f080 con 0x556e04bbb3b0 2023-10-05T19:28:27.229 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.222+0000 7f92a8e6ce40 20 rados_obj.operate() r=0 bl.length=46 2023-10-05T19:28:27.229 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.222+0000 7f92a8e6ce40 20 rados->read ofs=0 len=0 2023-10-05T19:28:27.229 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.222+0000 7f92a8e6ce40 1 -- 172.21.15.167:0/1223402549 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:40 5.c 5:30d299a5:::zonegroup_info.4f1f719d-9648-417b-b10c-e5d939825419:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x556e04bf16a0 con 0x7f92a0010070 2023-10-05T19:28:27.230 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.222+0000 7f92a71a2700 1 -- 172.21.15.167:0/1223402549 <== osd.0 v2:172.21.15.118:6800/27690 10 ==== osd_op_reply(40 zonegroup_info.4f1f719d-9648-417b-b10c-e5d939825419 [read 0~362 out=362b] v0'0 uv1 ondisk = 0) v8 ==== 195+0+362 (crc 0 0 0) 0x7f929002f080 con 0x7f92a0010070 2023-10-05T19:28:27.230 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.222+0000 7f92a8e6ce40 20 rados_obj.operate() r=0 bl.length=362 2023-10-05T19:28:27.230 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.222+0000 7f92a8e6ce40 20 rados->read ofs=0 len=0 2023-10-05T19:28:27.231 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.222+0000 7f92a8e6ce40 1 -- 172.21.15.167:0/1223402549 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:41 5.13 5:cbbf6c60:::converted:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x556e04bf16a0 con 0x556e04bee940 2023-10-05T19:28:27.231 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.226+0000 7f92a69a1700 1 -- 172.21.15.167:0/1223402549 <== osd.3 v2:172.21.15.167:6808/30036 10 ==== osd_op_reply(41 converted [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 ==== 153+0+0 (crc 0 0 0) 0x7f9298039040 con 0x556e04bee940 2023-10-05T19:28:27.231 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.226+0000 7f92a8e6ce40 20 rados_obj.operate() r=-2 bl.length=0 2023-10-05T19:28:27.231 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.226+0000 7f92a8e6ce40 20 rados->read ofs=0 len=0 2023-10-05T19:28:27.231 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.226+0000 7f92a8e6ce40 1 -- 172.21.15.167:0/1223402549 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:42 5.12 5:49953fa1:::default.realm:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x556e04bf1a50 con 0x556e04bbb3b0 2023-10-05T19:28:27.232 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.226+0000 7f92a71a2700 1 -- 172.21.15.167:0/1223402549 <== osd.1 v2:172.21.15.167:6800/30035 16 ==== osd_op_reply(42 default.realm [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 ==== 157+0+0 (crc 0 0 0) 0x7f929002f080 con 0x556e04bbb3b0 2023-10-05T19:28:27.232 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.226+0000 7f92a8e6ce40 20 rados_obj.operate() r=-2 bl.length=0 2023-10-05T19:28:27.232 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.226+0000 7f92a8e6ce40 20 rados->read ofs=0 len=0 2023-10-05T19:28:27.232 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.226+0000 7f92a8e6ce40 1 -- 172.21.15.167:0/1223402549 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:43 5.f 5:f4c53578:::zonegroups_names.default:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x556e04bf1e50 con 0x556e04bbb3b0 2023-10-05T19:28:27.232 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.226+0000 7f92a71a2700 1 -- 172.21.15.167:0/1223402549 <== osd.1 v2:172.21.15.167:6800/30035 17 ==== osd_op_reply(43 zonegroups_names.default [read 0~46 out=46b] v0'0 uv2 ondisk = 0) v8 ==== 168+0+46 (crc 0 0 0) 0x7f929002f080 con 0x556e04bbb3b0 2023-10-05T19:28:27.232 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.226+0000 7f92a8e6ce40 20 rados_obj.operate() r=0 bl.length=46 2023-10-05T19:28:27.233 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.226+0000 7f92a8e6ce40 1 -- 172.21.15.167:0/1223402549 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:44 5.0 5:00000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x556e04bf2e60 con 0x556e04bee940 2023-10-05T19:28:27.234 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.230+0000 7f92a69a1700 1 -- 172.21.15.167:0/1223402549 <== osd.3 v2:172.21.15.167:6808/30036 11 ==== osd_op_reply(44 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f9298039040 con 0x556e04bee940 2023-10-05T19:28:27.234 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.230+0000 7f92a69a1700 1 -- 172.21.15.167:0/1223402549 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:45 5.10 5:08000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x556e04bf2e60 con 0x7f92980176e0 2023-10-05T19:28:27.234 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.230+0000 7f92a61a0700 1 -- 172.21.15.167:0/1223402549 <== osd.2 v2:172.21.15.167:6816/30037 7 ==== osd_op_reply(45 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f92a0017050 con 0x7f92980176e0 2023-10-05T19:28:27.235 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.230+0000 7f92a61a0700 1 -- 172.21.15.167:0/1223402549 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:46 5.8 5:10000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x556e04bf2e60 con 0x7f92a0010070 2023-10-05T19:28:27.235 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.230+0000 7f92a71a2700 1 -- 172.21.15.167:0/1223402549 <== osd.0 v2:172.21.15.118:6800/27690 11 ==== osd_op_reply(46 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f929002f080 con 0x7f92a0010070 2023-10-05T19:28:27.236 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.230+0000 7f92a71a2700 1 -- 172.21.15.167:0/1223402549 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:47 5.18 5:18000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x556e04bedfe0 con 0x556e04bbb3b0 2023-10-05T19:28:27.236 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.230+0000 7f92a71a2700 1 -- 172.21.15.167:0/1223402549 <== osd.1 v2:172.21.15.167:6800/30035 18 ==== osd_op_reply(47 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f929002f080 con 0x556e04bbb3b0 2023-10-05T19:28:27.236 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.230+0000 7f92a71a2700 1 -- 172.21.15.167:0/1223402549 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:48 5.4 5:20000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x556e04bedfe0 con 0x556e04bbb3b0 2023-10-05T19:28:27.237 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.230+0000 7f92a71a2700 1 -- 172.21.15.167:0/1223402549 <== osd.1 v2:172.21.15.167:6800/30035 19 ==== osd_op_reply(48 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f929002f080 con 0x556e04bbb3b0 2023-10-05T19:28:27.237 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.230+0000 7f92a71a2700 1 -- 172.21.15.167:0/1223402549 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:49 5.14 5:28000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x556e04bedfe0 con 0x7f92a0010070 2023-10-05T19:28:27.238 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.234+0000 7f92a71a2700 1 -- 172.21.15.167:0/1223402549 <== osd.0 v2:172.21.15.118:6800/27690 12 ==== osd_op_reply(49 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f929002f080 con 0x7f92a0010070 2023-10-05T19:28:27.238 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.234+0000 7f92a71a2700 1 -- 172.21.15.167:0/1223402549 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:50 5.c 5:30000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x556e04bedfe0 con 0x7f92a0010070 2023-10-05T19:28:27.239 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.234+0000 7f92a71a2700 1 -- 172.21.15.167:0/1223402549 <== osd.0 v2:172.21.15.118:6800/27690 13 ==== osd_op_reply(50 [pgnls start_epoch 30 out=112b] v23'1 uv1 ondisk = 1) v8 ==== 144+0+112 (crc 0 0 0) 0x7f929002f080 con 0x7f92a0010070 2023-10-05T19:28:27.239 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.234+0000 7f92a71a2700 1 -- 172.21.15.167:0/1223402549 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:51 5.1c 5:38000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x556e04bedfe0 con 0x7f92980176e0 2023-10-05T19:28:27.240 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.234+0000 7f92a61a0700 1 -- 172.21.15.167:0/1223402549 <== osd.2 v2:172.21.15.167:6816/30037 8 ==== osd_op_reply(51 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f92a0017050 con 0x7f92980176e0 2023-10-05T19:28:27.240 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.234+0000 7f92a61a0700 1 -- 172.21.15.167:0/1223402549 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:52 5.2 5:40000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x556e04bedfe0 con 0x556e04bee940 2023-10-05T19:28:27.240 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.234+0000 7f92a69a1700 1 -- 172.21.15.167:0/1223402549 <== osd.3 v2:172.21.15.167:6808/30036 12 ==== osd_op_reply(52 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f9298039040 con 0x556e04bee940 2023-10-05T19:28:27.241 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.234+0000 7f92a69a1700 1 -- 172.21.15.167:0/1223402549 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:53 5.12 5:48000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x556e04bedfe0 con 0x556e04bbb3b0 2023-10-05T19:28:27.241 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.234+0000 7f92a71a2700 1 -- 172.21.15.167:0/1223402549 <== osd.1 v2:172.21.15.167:6800/30035 20 ==== osd_op_reply(53 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f929002f080 con 0x556e04bbb3b0 2023-10-05T19:28:27.241 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.234+0000 7f92a71a2700 1 -- 172.21.15.167:0/1223402549 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:54 5.a 5:50000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x556e04bedfe0 con 0x7f92a0010070 2023-10-05T19:28:27.242 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.238+0000 7f92a71a2700 1 -- 172.21.15.167:0/1223402549 <== osd.0 v2:172.21.15.118:6800/27690 14 ==== osd_op_reply(54 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f929002f080 con 0x7f92a0010070 2023-10-05T19:28:27.242 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.238+0000 7f92a71a2700 1 -- 172.21.15.167:0/1223402549 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:55 5.1a 5:58000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x556e04bedfe0 con 0x556e04bbb3b0 2023-10-05T19:28:27.243 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.238+0000 7f92a71a2700 1 -- 172.21.15.167:0/1223402549 <== osd.1 v2:172.21.15.167:6800/30035 21 ==== osd_op_reply(55 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f929002f080 con 0x556e04bbb3b0 2023-10-05T19:28:27.243 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.238+0000 7f92a71a2700 1 -- 172.21.15.167:0/1223402549 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:56 5.6 5:60000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x556e04bedfe0 con 0x7f92980176e0 2023-10-05T19:28:27.244 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.238+0000 7f92a61a0700 1 -- 172.21.15.167:0/1223402549 <== osd.2 v2:172.21.15.167:6816/30037 9 ==== osd_op_reply(56 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f92a0017050 con 0x7f92980176e0 2023-10-05T19:28:27.244 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.238+0000 7f92a61a0700 1 -- 172.21.15.167:0/1223402549 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:57 5.16 5:68000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x556e04bedfe0 con 0x556e04bee940 2023-10-05T19:28:27.245 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.238+0000 7f92a69a1700 1 -- 172.21.15.167:0/1223402549 <== osd.3 v2:172.21.15.167:6808/30036 13 ==== osd_op_reply(57 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f9298039040 con 0x556e04bee940 2023-10-05T19:28:27.245 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.238+0000 7f92a69a1700 1 -- 172.21.15.167:0/1223402549 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:58 5.e 5:70000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x556e04bedfe0 con 0x556e04bee940 2023-10-05T19:28:27.245 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.238+0000 7f92a69a1700 1 -- 172.21.15.167:0/1223402549 <== osd.3 v2:172.21.15.167:6808/30036 14 ==== osd_op_reply(58 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f9298039040 con 0x556e04bee940 2023-10-05T19:28:27.245 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.238+0000 7f92a69a1700 1 -- 172.21.15.167:0/1223402549 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:59 5.1e 5:78000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x556e04bedfe0 con 0x7f92a0010070 2023-10-05T19:28:27.246 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.242+0000 7f92a71a2700 1 -- 172.21.15.167:0/1223402549 <== osd.0 v2:172.21.15.118:6800/27690 15 ==== osd_op_reply(59 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f929002f080 con 0x7f92a0010070 2023-10-05T19:28:27.247 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.242+0000 7f92a71a2700 1 -- 172.21.15.167:0/1223402549 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:60 5.1 5:80000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x556e04bedfe0 con 0x556e04bee940 2023-10-05T19:28:27.247 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.242+0000 7f92a69a1700 1 -- 172.21.15.167:0/1223402549 <== osd.3 v2:172.21.15.167:6808/30036 15 ==== osd_op_reply(60 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f9298039040 con 0x556e04bee940 2023-10-05T19:28:27.247 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.242+0000 7f92a69a1700 1 -- 172.21.15.167:0/1223402549 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:61 5.11 5:88000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x556e04bedfe0 con 0x556e04bee940 2023-10-05T19:28:27.248 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.242+0000 7f92a69a1700 1 -- 172.21.15.167:0/1223402549 <== osd.3 v2:172.21.15.167:6808/30036 16 ==== osd_op_reply(61 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f9298039040 con 0x556e04bee940 2023-10-05T19:28:27.248 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.242+0000 7f92a69a1700 1 -- 172.21.15.167:0/1223402549 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:62 5.9 5:90000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x556e04bedfe0 con 0x556e04bbb3b0 2023-10-05T19:28:27.249 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.242+0000 7f92a71a2700 1 -- 172.21.15.167:0/1223402549 <== osd.1 v2:172.21.15.167:6800/30035 22 ==== osd_op_reply(62 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f929002f080 con 0x556e04bbb3b0 2023-10-05T19:28:27.249 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.242+0000 7f92a71a2700 1 -- 172.21.15.167:0/1223402549 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:63 5.19 5:98000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x556e04bedfe0 con 0x556e04bbb3b0 2023-10-05T19:28:27.249 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.242+0000 7f92a71a2700 1 -- 172.21.15.167:0/1223402549 <== osd.1 v2:172.21.15.167:6800/30035 23 ==== osd_op_reply(63 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f929002f080 con 0x556e04bbb3b0 2023-10-05T19:28:27.250 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.242+0000 7f92a71a2700 1 -- 172.21.15.167:0/1223402549 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:64 5.5 5:a0000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x556e04bedfe0 con 0x7f92a0010070 2023-10-05T19:28:27.251 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.246+0000 7f92a71a2700 1 -- 172.21.15.167:0/1223402549 <== osd.0 v2:172.21.15.118:6800/27690 16 ==== osd_op_reply(64 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f929002f080 con 0x7f92a0010070 2023-10-05T19:28:27.251 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.246+0000 7f92a71a2700 1 -- 172.21.15.167:0/1223402549 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:65 5.15 5:a8000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x556e04bedfe0 con 0x556e04bee940 2023-10-05T19:28:27.252 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.246+0000 7f92a69a1700 1 -- 172.21.15.167:0/1223402549 <== osd.3 v2:172.21.15.167:6808/30036 17 ==== osd_op_reply(65 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f9298039040 con 0x556e04bee940 2023-10-05T19:28:27.252 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.246+0000 7f92a69a1700 1 -- 172.21.15.167:0/1223402549 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:66 5.d 5:b0000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x556e04bedfe0 con 0x7f92980176e0 2023-10-05T19:28:27.252 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.246+0000 7f92a61a0700 1 -- 172.21.15.167:0/1223402549 <== osd.2 v2:172.21.15.167:6816/30037 10 ==== osd_op_reply(66 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f92a0017050 con 0x7f92980176e0 2023-10-05T19:28:27.253 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.246+0000 7f92a61a0700 1 -- 172.21.15.167:0/1223402549 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:67 5.1d 5:b8000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x556e04bedfe0 con 0x556e04bbb3b0 2023-10-05T19:28:27.253 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.246+0000 7f92a71a2700 1 -- 172.21.15.167:0/1223402549 <== osd.1 v2:172.21.15.167:6800/30035 24 ==== osd_op_reply(67 [pgnls start_epoch 30 out=107b] v30'2 uv2 ondisk = 1) v8 ==== 144+0+107 (crc 0 0 0) 0x7f929002f080 con 0x556e04bbb3b0 2023-10-05T19:28:27.253 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.246+0000 7f92a71a2700 1 -- 172.21.15.167:0/1223402549 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:68 5.3 5:c0000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x556e04bedfe0 con 0x7f92a0010070 2023-10-05T19:28:27.254 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.250+0000 7f92a71a2700 1 -- 172.21.15.167:0/1223402549 <== osd.0 v2:172.21.15.118:6800/27690 17 ==== osd_op_reply(68 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f929002f080 con 0x7f92a0010070 2023-10-05T19:28:27.255 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.250+0000 7f92a71a2700 1 -- 172.21.15.167:0/1223402549 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:69 5.13 5:c8000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x556e04bedfe0 con 0x556e04bee940 2023-10-05T19:28:27.256 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.250+0000 7f92a69a1700 1 -- 172.21.15.167:0/1223402549 <== osd.3 v2:172.21.15.167:6808/30036 18 ==== osd_op_reply(69 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f9298053040 con 0x556e04bee940 2023-10-05T19:28:27.256 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.250+0000 7f92a69a1700 1 -- 172.21.15.167:0/1223402549 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:70 5.b 5:d0000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x556e04bedfe0 con 0x7f92a0010070 2023-10-05T19:28:27.256 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.250+0000 7f92a71a2700 1 -- 172.21.15.167:0/1223402549 <== osd.0 v2:172.21.15.118:6800/27690 18 ==== osd_op_reply(70 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f929002f080 con 0x7f92a0010070 2023-10-05T19:28:27.257 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.250+0000 7f92a71a2700 1 -- 172.21.15.167:0/1223402549 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:71 5.1b 5:d8000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x556e04bedfe0 con 0x7f92980176e0 2023-10-05T19:28:27.257 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.250+0000 7f92a61a0700 1 -- 172.21.15.167:0/1223402549 <== osd.2 v2:172.21.15.167:6816/30037 11 ==== osd_op_reply(71 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f92a0017050 con 0x7f92980176e0 2023-10-05T19:28:27.257 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.250+0000 7f92a61a0700 1 -- 172.21.15.167:0/1223402549 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:72 5.7 5:e0000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x556e04bedfe0 con 0x556e04bee940 2023-10-05T19:28:27.258 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.254+0000 7f92a69a1700 1 -- 172.21.15.167:0/1223402549 <== osd.3 v2:172.21.15.167:6808/30036 19 ==== osd_op_reply(72 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f9298053040 con 0x556e04bee940 2023-10-05T19:28:27.258 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.254+0000 7f92a69a1700 1 -- 172.21.15.167:0/1223402549 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:73 5.17 5:e8000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x556e04bedfe0 con 0x7f92a0010070 2023-10-05T19:28:27.259 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.254+0000 7f92a71a2700 1 -- 172.21.15.167:0/1223402549 <== osd.0 v2:172.21.15.118:6800/27690 19 ==== osd_op_reply(73 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f929002f080 con 0x7f92a0010070 2023-10-05T19:28:27.259 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.254+0000 7f92a71a2700 1 -- 172.21.15.167:0/1223402549 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:74 5.f 5:f0000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x556e04bedfe0 con 0x556e04bbb3b0 2023-10-05T19:28:27.260 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.254+0000 7f92a71a2700 1 -- 172.21.15.167:0/1223402549 <== osd.1 v2:172.21.15.167:6800/30035 25 ==== osd_op_reply(74 [pgnls start_epoch 30 out=115b] v23'2 uv2 ondisk = 1) v8 ==== 144+0+115 (crc 0 0 0) 0x7f929002f080 con 0x556e04bbb3b0 2023-10-05T19:28:27.260 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.254+0000 7f92a71a2700 1 -- 172.21.15.167:0/1223402549 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:75 5.1f 5:f8000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x556e04bedfe0 con 0x7f92980176e0 2023-10-05T19:28:27.261 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.254+0000 7f92a61a0700 1 -- 172.21.15.167:0/1223402549 <== osd.2 v2:172.21.15.167:6816/30037 12 ==== osd_op_reply(75 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f92a0017050 con 0x7f92980176e0 2023-10-05T19:28:27.261 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.254+0000 7f92a8e6ce40 20 RGWRados::pool_iterate: got zonegroup_info.4f1f719d-9648-417b-b10c-e5d939825419 2023-10-05T19:28:27.261 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.254+0000 7f92a8e6ce40 20 RGWRados::pool_iterate: got zone_info.9a9df965-6612-45d4-a45f-6e3c889bacc2 2023-10-05T19:28:27.262 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.254+0000 7f92a8e6ce40 20 RGWRados::pool_iterate: got zone_names.default 2023-10-05T19:28:27.262 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.254+0000 7f92a8e6ce40 20 RGWRados::pool_iterate: got zonegroups_names.default 2023-10-05T19:28:27.262 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.254+0000 7f92a8e6ce40 20 rados->read ofs=0 len=0 2023-10-05T19:28:27.262 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.254+0000 7f92a8e6ce40 1 -- 172.21.15.167:0/1223402549 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:76 5.f 5:f43b6ece:::zone_names.default:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x556e04bf1e50 con 0x556e04bbb3b0 2023-10-05T19:28:27.263 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.258+0000 7f92a71a2700 1 -- 172.21.15.167:0/1223402549 <== osd.1 v2:172.21.15.167:6800/30035 26 ==== osd_op_reply(76 zone_names.default [read 0~46 out=46b] v0'0 uv1 ondisk = 0) v8 ==== 162+0+46 (crc 0 0 0) 0x7f929002f080 con 0x556e04bbb3b0 2023-10-05T19:28:27.263 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.258+0000 7f92a8e6ce40 20 rados_obj.operate() r=0 bl.length=46 2023-10-05T19:28:27.263 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.258+0000 7f92a8e6ce40 20 rados->read ofs=0 len=0 2023-10-05T19:28:27.263 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.258+0000 7f92a8e6ce40 1 -- 172.21.15.167:0/1223402549 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:77 5.1d 5:b8374eb4:::zone_info.9a9df965-6612-45d4-a45f-6e3c889bacc2:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x556e04bf2200 con 0x556e04bbb3b0 2023-10-05T19:28:27.263 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.258+0000 7f92a71a2700 1 -- 172.21.15.167:0/1223402549 <== osd.1 v2:172.21.15.167:6800/30035 27 ==== osd_op_reply(77 zone_info.9a9df965-6612-45d4-a45f-6e3c889bacc2 [read 0~874 out=874b] v0'0 uv2 ondisk = 0) v8 ==== 190+0+874 (crc 0 0 0) 0x7f929002f080 con 0x556e04bbb3b0 2023-10-05T19:28:27.263 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.258+0000 7f92a8e6ce40 20 rados_obj.operate() r=0 bl.length=874 2023-10-05T19:28:27.263 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.258+0000 7f92a8e6ce40 20 rados->read ofs=0 len=0 2023-10-05T19:28:27.263 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.258+0000 7f92a8e6ce40 1 -- 172.21.15.167:0/1223402549 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:78 5.f 5:f4c53578:::zonegroups_names.default:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x556e04bf2200 con 0x556e04bbb3b0 2023-10-05T19:28:27.264 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.258+0000 7f92a71a2700 1 -- 172.21.15.167:0/1223402549 <== osd.1 v2:172.21.15.167:6800/30035 28 ==== osd_op_reply(78 zonegroups_names.default [read 0~46 out=46b] v0'0 uv2 ondisk = 0) v8 ==== 168+0+46 (crc 0 0 0) 0x7f929002f080 con 0x556e04bbb3b0 2023-10-05T19:28:27.264 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.258+0000 7f92a8e6ce40 20 rados_obj.operate() r=0 bl.length=46 2023-10-05T19:28:27.264 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.258+0000 7f92a8e6ce40 20 rados->read ofs=0 len=0 2023-10-05T19:28:27.265 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.258+0000 7f92a8e6ce40 1 -- 172.21.15.167:0/1223402549 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:79 5.c 5:30d299a5:::zonegroup_info.4f1f719d-9648-417b-b10c-e5d939825419:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x556e04bf2200 con 0x7f92a0010070 2023-10-05T19:28:27.265 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.258+0000 7f92a71a2700 1 -- 172.21.15.167:0/1223402549 <== osd.0 v2:172.21.15.118:6800/27690 20 ==== osd_op_reply(79 zonegroup_info.4f1f719d-9648-417b-b10c-e5d939825419 [read 0~362 out=362b] v0'0 uv1 ondisk = 0) v8 ==== 195+0+362 (crc 0 0 0) 0x7f929002f080 con 0x7f92a0010070 2023-10-05T19:28:27.265 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.258+0000 7f92a8e6ce40 20 rados_obj.operate() r=0 bl.length=362 2023-10-05T19:28:27.265 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.258+0000 7f92a8e6ce40 20 rados->read ofs=0 len=0 2023-10-05T19:28:27.265 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.258+0000 7f92a8e6ce40 1 -- 172.21.15.167:0/1223402549 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:80 5.3 5:c0ad8cbd:::region_map:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x556e04bf2650 con 0x7f92a0010070 2023-10-05T19:28:27.265 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.258+0000 7f92a71a2700 1 -- 172.21.15.167:0/1223402549 <== osd.0 v2:172.21.15.118:6800/27690 21 ==== osd_op_reply(80 region_map [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 ==== 154+0+0 (crc 0 0 0) 0x7f9290050040 con 0x7f92a0010070 2023-10-05T19:28:27.266 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.258+0000 7f92a8e6ce40 20 rados_obj.operate() r=-2 bl.length=0 2023-10-05T19:28:27.266 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.258+0000 7f92a8e6ce40 20 zone default found 2023-10-05T19:28:27.266 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.258+0000 7f92a8e6ce40 4 Realm: () 2023-10-05T19:28:27.266 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.258+0000 7f92a8e6ce40 4 ZoneGroup: default (4f1f719d-9648-417b-b10c-e5d939825419) 2023-10-05T19:28:27.266 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.258+0000 7f92a8e6ce40 4 Zone: default (9a9df965-6612-45d4-a45f-6e3c889bacc2) 2023-10-05T19:28:27.266 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.258+0000 7f92a8e6ce40 10 cannot find current period zonegroup using local zonegroup configuration 2023-10-05T19:28:27.266 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.258+0000 7f92a8e6ce40 20 zonegroup default 2023-10-05T19:28:27.266 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.258+0000 7f92a8e6ce40 20 rados->read ofs=0 len=0 2023-10-05T19:28:27.266 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.258+0000 7f92a8e6ce40 1 -- 172.21.15.167:0/1223402549 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:81 5.3 5:c52100b6:::period_config.default:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x556e04bf2650 con 0x7f92a0010070 2023-10-05T19:28:27.267 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.262+0000 7f92a71a2700 1 -- 172.21.15.167:0/1223402549 <== osd.0 v2:172.21.15.118:6800/27690 22 ==== osd_op_reply(81 period_config.default [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 ==== 165+0+0 (crc 0 0 0) 0x7f9290050040 con 0x7f92a0010070 2023-10-05T19:28:27.267 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.262+0000 7f92a8e6ce40 20 rados_obj.operate() r=-2 bl.length=0 2023-10-05T19:28:27.267 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.262+0000 7f92a8e6ce40 20 rados->read ofs=0 len=0 2023-10-05T19:28:27.268 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.262+0000 7f92a8e6ce40 1 -- 172.21.15.167:0/1223402549 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:82 6.2 6:4347d321:::bucket.sync-source-hints.:head [call version.read in=11b,read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x556e04bf3da0 con 0x556e04bee940 2023-10-05T19:28:27.268 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.262+0000 7f92a69a1700 1 -- 172.21.15.167:0/1223402549 <== osd.3 v2:172.21.15.167:6808/30036 20 ==== osd_op_reply(82 bucket.sync-source-hints. [call,read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 ==== 211+0+0 (crc 0 0 0) 0x7f9298053040 con 0x556e04bee940 2023-10-05T19:28:27.268 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.262+0000 7f92a8e6ce40 20 rados_obj.operate() r=-2 bl.length=0 2023-10-05T19:28:27.268 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.262+0000 7f92a8e6ce40 20 rados->read ofs=0 len=0 2023-10-05T19:28:27.268 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.262+0000 7f92a8e6ce40 1 -- 172.21.15.167:0/1223402549 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:83 6.b 6:d467b91b:::bucket.sync-target-hints.:head [call version.read in=11b,read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x556e04bf4150 con 0x556e04bee940 2023-10-05T19:28:27.268 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.262+0000 7f92a69a1700 1 -- 172.21.15.167:0/1223402549 <== osd.3 v2:172.21.15.167:6808/30036 21 ==== osd_op_reply(83 bucket.sync-target-hints. [call,read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 ==== 211+0+0 (crc 0 0 0) 0x7f9298053040 con 0x556e04bee940 2023-10-05T19:28:27.269 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.262+0000 7f92a8e6ce40 20 rados_obj.operate() r=-2 bl.length=0 2023-10-05T19:28:27.269 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.262+0000 7f92a8e6ce40 20 started sync module instance, tier type = 2023-10-05T19:28:27.269 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.262+0000 7f92a8e6ce40 20 started zone id=9a9df965-6612-45d4-a45f-6e3c889bacc2 (name=default) with tier type = 2023-10-05T19:28:27.269 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.262+0000 7f92a8e6ce40 1 -- 172.21.15.167:0/1223402549 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:84 7.1f 7:f95f44c2:::notify.0:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x556e04bf3460 con 0x7f92980176e0 2023-10-05T19:28:27.271 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.266+0000 7f92a61a0700 1 -- 172.21.15.167:0/1223402549 <== osd.2 v2:172.21.15.167:6816/30037 13 ==== osd_op_reply(84 notify.0 [create] v30'45 uv45 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f92a0017050 con 0x7f92980176e0 2023-10-05T19:28:27.271 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.266+0000 7f92a8e6ce40 1 -- 172.21.15.167:0/1223402549 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:85 7.1f 7:f95f44c2:::notify.0:head [watch watch cookie 93931014404192] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x556e04bedc10 con 0x7f92980176e0 2023-10-05T19:28:27.271 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.266+0000 7f92a8e6ce40 1 -- 172.21.15.167:0/1223402549 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:86 7.0 7:05bf5b68:::notify.1:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x556e04bf4ee0 con 0x556e04bbb3b0 2023-10-05T19:28:27.274 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.266+0000 7f92a61a0700 1 -- 172.21.15.167:0/1223402549 <== osd.2 v2:172.21.15.167:6816/30037 14 ==== osd_op_reply(85 notify.0 [watch watch cookie 93931014404192] v30'46 uv45 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f92a0017050 con 0x7f92980176e0 2023-10-05T19:28:27.274 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.270+0000 7f92a71a2700 1 -- 172.21.15.167:0/1223402549 <== osd.1 v2:172.21.15.167:6800/30035 29 ==== osd_op_reply(86 notify.1 [create] v30'45 uv45 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f9290050040 con 0x556e04bbb3b0 2023-10-05T19:28:27.274 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.270+0000 7f92a8e6ce40 1 -- 172.21.15.167:0/1223402549 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:87 7.0 7:05bf5b68:::notify.1:head [watch watch cookie 93931014407712] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x556e04bf52e0 con 0x556e04bbb3b0 2023-10-05T19:28:27.275 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.270+0000 7f92a8e6ce40 1 -- 172.21.15.167:0/1223402549 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:88 7.15 7:a93a5511:::notify.2:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x556e04bf5b80 con 0x556e04bee940 2023-10-05T19:28:27.277 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.270+0000 7f92a71a2700 1 -- 172.21.15.167:0/1223402549 <== osd.1 v2:172.21.15.167:6800/30035 30 ==== osd_op_reply(87 notify.1 [watch watch cookie 93931014407712] v30'46 uv45 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f9290050040 con 0x556e04bbb3b0 2023-10-05T19:28:27.278 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.274+0000 7f92a69a1700 1 -- 172.21.15.167:0/1223402549 <== osd.3 v2:172.21.15.167:6808/30036 22 ==== osd_op_reply(88 notify.2 [create] v30'45 uv45 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f9298053040 con 0x556e04bee940 2023-10-05T19:28:27.278 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.274+0000 7f92a8e6ce40 1 -- 172.21.15.167:0/1223402549 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:89 7.15 7:a93a5511:::notify.2:head [watch watch cookie 93931014410976] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x556e04bf4bd0 con 0x556e04bee940 2023-10-05T19:28:27.278 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.274+0000 7f92a8e6ce40 1 -- 172.21.15.167:0/1223402549 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:90 7.e 7:7759931f:::notify.3:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x556e04bf6910 con 0x7f92980176e0 2023-10-05T19:28:27.281 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.274+0000 7f92a61a0700 1 -- 172.21.15.167:0/1223402549 <== osd.2 v2:172.21.15.167:6816/30037 15 ==== osd_op_reply(90 notify.3 [create] v30'45 uv45 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f92a0017050 con 0x7f92980176e0 2023-10-05T19:28:27.281 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.274+0000 7f92a8e6ce40 1 -- 172.21.15.167:0/1223402549 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:91 7.e 7:7759931f:::notify.3:head [watch watch cookie 93931014415392] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x556e04bf71f0 con 0x7f92980176e0 2023-10-05T19:28:27.281 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.274+0000 7f92a8e6ce40 1 -- 172.21.15.167:0/1223402549 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:92 7.d 7:b4812045:::notify.4:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x556e04bf7d50 con 0x556e04bbb3b0 2023-10-05T19:28:27.281 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.274+0000 7f92a69a1700 1 -- 172.21.15.167:0/1223402549 <== osd.3 v2:172.21.15.167:6808/30036 23 ==== osd_op_reply(89 notify.2 [watch watch cookie 93931014410976] v30'46 uv45 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f9298053040 con 0x556e04bee940 2023-10-05T19:28:27.283 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.278+0000 7f92a61a0700 1 -- 172.21.15.167:0/1223402549 <== osd.2 v2:172.21.15.167:6816/30037 16 ==== osd_op_reply(91 notify.3 [watch watch cookie 93931014415392] v30'46 uv45 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f92a0017050 con 0x7f92980176e0 2023-10-05T19:28:27.284 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.278+0000 7f92a71a2700 1 -- 172.21.15.167:0/1223402549 <== osd.1 v2:172.21.15.167:6800/30035 31 ==== osd_op_reply(92 notify.4 [create] v30'45 uv45 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f9290050040 con 0x556e04bbb3b0 2023-10-05T19:28:27.284 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.278+0000 7f92a8e6ce40 1 -- 172.21.15.167:0/1223402549 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:93 7.d 7:b4812045:::notify.4:head [watch watch cookie 93931014414128] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x556e04bf7820 con 0x556e04bbb3b0 2023-10-05T19:28:27.284 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.278+0000 7f92a8e6ce40 1 -- 172.21.15.167:0/1223402549 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:94 7.3 7:c609908c:::notify.5:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x556e04bf8640 con 0x7f92a0010070 2023-10-05T19:28:27.287 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.282+0000 7f92a71a2700 1 -- 172.21.15.167:0/1223402549 <== osd.0 v2:172.21.15.118:6800/27690 23 ==== osd_op_reply(94 notify.5 [create] v30'45 uv45 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f9290050040 con 0x7f92a0010070 2023-10-05T19:28:27.287 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.282+0000 7f92a8e6ce40 1 -- 172.21.15.167:0/1223402549 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:95 7.3 7:c609908c:::notify.5:head [watch watch cookie 93931014422784] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x556e04bf8a40 con 0x7f92a0010070 2023-10-05T19:28:27.287 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.282+0000 7f92a8e6ce40 1 -- 172.21.15.167:0/1223402549 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:96 7.14 7:2b04a3e9:::notify.6:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x556e04bf94c0 con 0x7f92a0010070 2023-10-05T19:28:27.287 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.282+0000 7f92a71a2700 1 -- 172.21.15.167:0/1223402549 <== osd.1 v2:172.21.15.167:6800/30035 32 ==== osd_op_reply(93 notify.4 [watch watch cookie 93931014414128] v30'46 uv45 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f9290050040 con 0x556e04bbb3b0 2023-10-05T19:28:27.289 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.282+0000 7f92a71a2700 1 -- 172.21.15.167:0/1223402549 <== osd.0 v2:172.21.15.118:6800/27690 24 ==== osd_op_reply(96 notify.6 [create] v30'45 uv45 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f9290050040 con 0x7f92a0010070 2023-10-05T19:28:27.289 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.282+0000 7f92a8e6ce40 1 -- 172.21.15.167:0/1223402549 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:97 7.14 7:2b04a3e9:::notify.6:head [watch watch cookie 93931014417600] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x556e04bf9480 con 0x7f92a0010070 2023-10-05T19:28:27.290 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.282+0000 7f92a8e6ce40 1 -- 172.21.15.167:0/1223402549 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:98 7.9 7:93e5b521:::notify.7:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x556e04bf9e90 con 0x7f92a0010070 2023-10-05T19:28:27.290 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.282+0000 7f92a71a2700 1 -- 172.21.15.167:0/1223402549 <== osd.0 v2:172.21.15.118:6800/27690 25 ==== osd_op_reply(95 notify.5 [watch watch cookie 93931014422784] v30'46 uv45 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f9290050040 con 0x7f92a0010070 2023-10-05T19:28:27.291 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.286+0000 7f92a71a2700 1 -- 172.21.15.167:0/1223402549 <== osd.0 v2:172.21.15.118:6800/27690 26 ==== osd_op_reply(97 notify.6 [watch watch cookie 93931014417600] v30'46 uv45 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f9290050040 con 0x7f92a0010070 2023-10-05T19:28:27.292 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.286+0000 7f92a71a2700 1 -- 172.21.15.167:0/1223402549 <== osd.0 v2:172.21.15.118:6800/27690 27 ==== osd_op_reply(98 notify.7 [create] v30'45 uv45 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f9290050040 con 0x7f92a0010070 2023-10-05T19:28:27.292 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.286+0000 7f92a8e6ce40 1 -- 172.21.15.167:0/1223402549 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:99 7.9 7:93e5b521:::notify.7:head [watch watch cookie 93931014426512] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x556e04bf93e0 con 0x7f92a0010070 2023-10-05T19:28:27.292 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.286+0000 7f92a8e6ce40 20 add_watcher() i=0 2023-10-05T19:28:27.293 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.286+0000 7f92a8e6ce40 20 add_watcher() i=1 2023-10-05T19:28:27.293 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.286+0000 7f92a8e6ce40 20 add_watcher() i=2 2023-10-05T19:28:27.293 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.286+0000 7f92a8e6ce40 20 add_watcher() i=3 2023-10-05T19:28:27.293 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.286+0000 7f92a8e6ce40 20 add_watcher() i=4 2023-10-05T19:28:27.293 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.286+0000 7f92a8e6ce40 20 add_watcher() i=5 2023-10-05T19:28:27.293 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.286+0000 7f92a8e6ce40 20 add_watcher() i=6 2023-10-05T19:28:27.294 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.290+0000 7f92a71a2700 1 -- 172.21.15.167:0/1223402549 <== osd.0 v2:172.21.15.118:6800/27690 28 ==== osd_op_reply(99 notify.7 [watch watch cookie 93931014426512] v30'46 uv45 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f9290050040 con 0x7f92a0010070 2023-10-05T19:28:27.294 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.290+0000 7f92a8e6ce40 20 add_watcher() i=7 2023-10-05T19:28:27.294 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.290+0000 7f92a8e6ce40 2 all 8 watchers are set, enabling cache 2023-10-05T19:28:27.294 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.290+0000 7f92a8e6ce40 1 -- 172.21.15.167:0/1223402549 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:100 6.4 6:22d26bf9:::data_loggenerations_metadata:head [call version.check_conds in=74b,call version.read in=11b,read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x556e04bf8670 con 0x556e04bbb3b0 2023-10-05T19:28:27.295 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.290+0000 7f92a71a2700 1 -- 172.21.15.167:0/1223402549 <== osd.1 v2:172.21.15.167:6800/30035 33 ==== osd_op_reply(100 data_loggenerations_metadata [call,call out=48b,read 0~28 out=28b] v0'0 uv1 ondisk = 0) v8 ==== 256+0+76 (crc 0 0 0) 0x7f9290050040 con 0x556e04bbb3b0 2023-10-05T19:28:27.295 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.290+0000 7f92a8e6ce40 1 -- 172.21.15.167:0/1223402549 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:101 6.4 6:22d26bf9:::data_loggenerations_metadata:head [watch watch cookie 93931014429088] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x556e04bfa2e0 con 0x556e04bbb3b0 2023-10-05T19:28:27.298 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.290+0000 7f92a71a2700 1 -- 172.21.15.167:0/1223402549 <== osd.1 v2:172.21.15.167:6800/30035 34 ==== osd_op_reply(101 data_loggenerations_metadata [watch watch cookie 93931014429088] v30'64 uv1 ondisk = 0) v8 ==== 172+0+0 (crc 0 0 0) 0x7f9290050040 con 0x556e04bbb3b0 2023-10-05T19:28:27.298 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.290+0000 7f91f3fff700 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2023-10-05T19:28:27.298 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.294+0000 7f92a8e6ce40 20 check_secure_mon_conn(): auth registy supported: methods=[2] modes=[2,1] 2023-10-05T19:28:27.299 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.294+0000 7f92a8e6ce40 20 check_secure_mon_conn(): mode 1 is insecure 2023-10-05T19:28:27.299 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.294+0000 7f92a8e6ce40 1 Processor -- start 2023-10-05T19:28:27.299 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.294+0000 7f92a8e6ce40 1 -- start start 2023-10-05T19:28:27.299 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.294+0000 7f92a8e6ce40 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x556e04c072e0 0x556e04c076b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-05T19:28:27.299 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.294+0000 7f92a8e6ce40 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x556e04c07bf0 con 0x556e04c072e0 2023-10-05T19:28:27.300 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.294+0000 7f92a61a0700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x556e04c072e0 0x556e04c076b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-05T19:28:27.300 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.294+0000 7f92a61a0700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x556e04c072e0 0x556e04c076b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.118:3300/0 says I am v2:172.21.15.167:57472/0 (socket says 172.21.15.167:57472) 2023-10-05T19:28:27.300 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.294+0000 7f92a61a0700 1 -- 172.21.15.167:0/4079599137 learned_addr learned my addr 172.21.15.167:0/4079599137 (peer_addr_for_me v2:172.21.15.167:0/0) 2023-10-05T19:28:27.300 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.294+0000 7f92a61a0700 1 -- 172.21.15.167:0/4079599137 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x556e04c07d30 con 0x556e04c072e0 2023-10-05T19:28:27.300 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.294+0000 7f92a61a0700 1 --2- 172.21.15.167:0/4079599137 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x556e04c072e0 0x556e04c076b0 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7f92a0003000 tx=0x7f92a0010000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5f6d4a0482795cee server_cookie=0 in_seq=0 out_seq=0 2023-10-05T19:28:27.301 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.294+0000 7f91f37fe700 1 -- 172.21.15.167:0/4079599137 <== mon.0 v2:172.21.15.118:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f92a001c9f0 con 0x556e04c072e0 2023-10-05T19:28:27.301 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.294+0000 7f91f37fe700 1 -- 172.21.15.167:0/4079599137 <== mon.0 v2:172.21.15.118:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f92a001cb50 con 0x556e04c072e0 2023-10-05T19:28:27.301 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.294+0000 7f91f37fe700 1 -- 172.21.15.167:0/4079599137 <== mon.0 v2:172.21.15.118:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f92a001cd00 con 0x556e04c072e0 2023-10-05T19:28:27.302 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.294+0000 7f92a8e6ce40 1 -- 172.21.15.167:0/4079599137 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x556e04c072e0 msgr2=0x556e04c076b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-05T19:28:27.302 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.294+0000 7f92a8e6ce40 1 --2- 172.21.15.167:0/4079599137 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x556e04c072e0 0x556e04c076b0 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7f92a0003000 tx=0x7f92a0010000 comp rx=0 tx=0).stop 2023-10-05T19:28:27.302 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.294+0000 7f92a8e6ce40 1 -- 172.21.15.167:0/4079599137 shutdown_connections 2023-10-05T19:28:27.302 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.294+0000 7f92a8e6ce40 1 --2- 172.21.15.167:0/4079599137 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x556e04c072e0 0x556e04c076b0 unknown :-1 s=CLOSED pgs=228 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-05T19:28:27.302 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.294+0000 7f92a8e6ce40 1 -- 172.21.15.167:0/4079599137 >> 172.21.15.167:0/4079599137 conn(0x556e04c026a0 msgr2=0x556e04c007f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-05T19:28:27.302 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.298+0000 7f92a8e6ce40 1 -- 172.21.15.167:0/4079599137 shutdown_connections 2023-10-05T19:28:27.302 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.298+0000 7f92a8e6ce40 1 -- 172.21.15.167:0/4079599137 wait complete. 2023-10-05T19:28:27.303 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.298+0000 7f92a8e6ce40 1 Processor -- start 2023-10-05T19:28:27.303 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.298+0000 7f92a8e6ce40 1 -- start start 2023-10-05T19:28:27.303 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.298+0000 7f92a8e6ce40 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x556e04c072e0 0x556e04c0ce10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-05T19:28:27.304 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.298+0000 7f92a8e6ce40 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x556e04c0de10 con 0x556e04c072e0 2023-10-05T19:28:27.304 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.298+0000 7f92a61a0700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x556e04c072e0 0x556e04c0ce10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-05T19:28:27.304 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.298+0000 7f92a61a0700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x556e04c072e0 0x556e04c0ce10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.118:3300/0 says I am v2:172.21.15.167:57482/0 (socket says 172.21.15.167:57482) 2023-10-05T19:28:27.304 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.298+0000 7f92a61a0700 1 -- 172.21.15.167:0/584546410 learned_addr learned my addr 172.21.15.167:0/584546410 (peer_addr_for_me v2:172.21.15.167:0/0) 2023-10-05T19:28:27.304 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.298+0000 7f92a61a0700 1 -- 172.21.15.167:0/584546410 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x556e04bb79e0 con 0x556e04c072e0 2023-10-05T19:28:27.304 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.298+0000 7f92a61a0700 1 --2- 172.21.15.167:0/584546410 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x556e04c072e0 0x556e04c0ce10 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x556e04c01360 tx=0x7f92a002f000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-05T19:28:27.305 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.298+0000 7f91f1ffb700 1 -- 172.21.15.167:0/584546410 <== mon.0 v2:172.21.15.118:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f92a0047460 con 0x556e04c072e0 2023-10-05T19:28:27.305 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.298+0000 7f91f1ffb700 1 -- 172.21.15.167:0/584546410 <== mon.0 v2:172.21.15.118:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f92a00475e0 con 0x556e04c072e0 2023-10-05T19:28:27.305 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.298+0000 7f91f1ffb700 1 -- 172.21.15.167:0/584546410 <== mon.0 v2:172.21.15.118:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f92a0047790 con 0x556e04c072e0 2023-10-05T19:28:27.306 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.298+0000 7f92a8e6ce40 1 -- 172.21.15.167:0/584546410 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x556e04c0d410 con 0x556e04c072e0 2023-10-05T19:28:27.306 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.298+0000 7f92a8e6ce40 1 -- 172.21.15.167:0/584546410 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x556e04c0d850 con 0x556e04c072e0 2023-10-05T19:28:27.307 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.302+0000 7f91f1ffb700 1 -- 172.21.15.167:0/584546410 <== mon.0 v2:172.21.15.118:3300/0 4 ==== mgrmap(e 5) v1 ==== 40329+0+0 (secure 0 0 0) 0x7f92a0035400 con 0x556e04c072e0 2023-10-05T19:28:27.307 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.302+0000 7f91f1ffb700 1 --2- 172.21.15.167:0/584546410 >> [v2:172.21.15.167:6824/29889,v1:172.21.15.167:6825/29889] conn(0x7f91e4030c00 0x7f91e40330b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-05T19:28:27.308 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.302+0000 7f91f1ffb700 1 -- 172.21.15.167:0/584546410 <== mon.0 v2:172.21.15.118:3300/0 5 ==== osd_map(30..30 src has 1..30) v4 ==== 6057+0+0 (secure 0 0 0) 0x7f92a002f070 con 0x556e04c072e0 2023-10-05T19:28:27.308 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.302+0000 7f92a8e6ce40 5 note: GC not initialized 2023-10-05T19:28:27.308 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.302+0000 7f92a71a2700 1 --2- 172.21.15.167:0/584546410 >> [v2:172.21.15.167:6824/29889,v1:172.21.15.167:6825/29889] conn(0x7f91e4030c00 0x7f91e40330b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-05T19:28:27.308 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.302+0000 7f92a71a2700 1 --2- 172.21.15.167:0/584546410 >> [v2:172.21.15.167:6824/29889,v1:172.21.15.167:6825/29889] conn(0x7f91e4030c00 0x7f91e40330b0 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x556e04bf8730 tx=0x7f9290000f90 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-05T19:28:27.310 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.302+0000 7f91e2ffd700 20 reqs_thread_entry: start 2023-10-05T19:28:27.310 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.306+0000 7f92a8e6ce40 20 init_complete bucket index max shards: 11 2023-10-05T19:28:27.311 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.306+0000 7f92a8e6ce40 10 rgw notify: Started notification manager with: 1 workers 2023-10-05T19:28:27.311 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.306+0000 7f91daffd700 20 rgw notify: INFO: next queues processing will happen at: Thu Oct 5 19:28:57 2023 2023-10-05T19:28:27.311 INFO:teuthology.orchestra.run.smithi167.stderr: 2023-10-05T19:28:27.311 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.306+0000 7f91da7fc700 20 reqs_thread_entry: start 2023-10-05T19:28:27.311 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.306+0000 7f92a8e6ce40 20 get_system_obj_state: rctx=0x7fffab7e2058 obj=default.rgw.meta:users.uid:foo state=0x556e04a900b0 s->prefetch_data=0 2023-10-05T19:28:27.311 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.306+0000 7f92a8e6ce40 10 cache get: name=default.rgw.meta+users.uid+foo : miss 2023-10-05T19:28:27.312 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.306+0000 7f92a8e6ce40 1 -- 172.21.15.167:0/1223402549 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:102 8.18 8:1e6ca5ed:users.uid::foo:head [call version.read in=11b,getxattrs,stat] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x556e04c755c0 con 0x556e04bbb3b0 2023-10-05T19:28:27.312 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.306+0000 7f91e0ff9700 20 reqs_thread_entry: start 2023-10-05T19:28:27.313 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.306+0000 7f92a71a2700 1 -- 172.21.15.167:0/1223402549 <== osd.1 v2:172.21.15.167:6800/30035 35 ==== osd_op_reply(102 foo [call out=48b,getxattrs out=75b,stat out=16b] v0'0 uv3 ondisk = 0) v8 ==== 231+0+139 (crc 0 0 0) 0x7f9290050040 con 0x556e04bbb3b0 2023-10-05T19:28:27.313 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.306+0000 7f92a8e6ce40 10 cache put: name=default.rgw.meta+users.uid+foo info.flags=0x16 2023-10-05T19:28:27.313 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.306+0000 7f92a8e6ce40 10 adding default.rgw.meta+users.uid+foo to cache LRU end 2023-10-05T19:28:27.313 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.306+0000 7f92a8e6ce40 10 updating xattr: name=ceph.objclass.version bl.length()=42 2023-10-05T19:28:27.313 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.306+0000 7f92a8e6ce40 20 get_system_obj_state: s->obj_tag was set empty 2023-10-05T19:28:27.313 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.306+0000 7f92a8e6ce40 20 Read xattr: user.rgw.idtag 2023-10-05T19:28:27.313 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.306+0000 7f92a8e6ce40 10 cache get: name=default.rgw.meta+users.uid+foo : type miss (requested=0x13, cached=0x16) 2023-10-05T19:28:27.313 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.306+0000 7f92a8e6ce40 20 rados->read ofs=0 len=0 2023-10-05T19:28:27.313 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.306+0000 7f92a8e6ce40 1 -- 172.21.15.167:0/1223402549 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:103 8.18 8:1e6ca5ed:users.uid::foo:head [call version.check_conds in=122b,call version.read in=11b,read 0~0,getxattrs] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x556e04c78050 con 0x556e04bbb3b0 2023-10-05T19:28:27.314 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.306+0000 7f92a71a2700 1 -- 172.21.15.167:0/1223402549 <== osd.1 v2:172.21.15.167:6800/30035 36 ==== osd_op_reply(103 foo [call,call out=48b,read 0~357 out=357b,getxattrs out=75b] v0'0 uv3 ondisk = 0) v8 ==== 273+0+480 (crc 0 0 0) 0x7f9290050040 con 0x556e04bbb3b0 2023-10-05T19:28:27.314 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.306+0000 7f92a8e6ce40 20 rados_obj.operate() r=0 bl.length=357 2023-10-05T19:28:27.314 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.306+0000 7f92a8e6ce40 10 cache put: name=default.rgw.meta+users.uid+foo info.flags=0x13 2023-10-05T19:28:27.314 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.306+0000 7f92a8e6ce40 10 moving default.rgw.meta+users.uid+foo to cache LRU end 2023-10-05T19:28:27.314 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.306+0000 7f92a8e6ce40 10 updating xattr: name=ceph.objclass.version bl.length()=42 2023-10-05T19:28:27.314 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.306+0000 7f92a8e6ce40 20 get_system_obj_state: rctx=0x7fffab7e1af8 obj=default.rgw.meta:users.keys:p5YnriCv1nAtykxBrupQ state=0x556e04a900b0 s->prefetch_data=0 2023-10-05T19:28:27.314 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.306+0000 7f92a8e6ce40 10 cache get: name=default.rgw.meta+users.keys+p5YnriCv1nAtykxBrupQ : miss 2023-10-05T19:28:27.314 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.306+0000 7f92a8e6ce40 1 -- 172.21.15.167:0/1223402549 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:104 8.13 8:cc695d75:users.keys::p5YnriCv1nAtykxBrupQ:head [getxattrs,stat] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x556e04c76030 con 0x7f92a0010070 2023-10-05T19:28:27.315 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.306+0000 7f92a71a2700 1 -- 172.21.15.167:0/1223402549 <== osd.0 v2:172.21.15.118:6800/27690 29 ==== osd_op_reply(104 p5YnriCv1nAtykxBrupQ [getxattrs,stat] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 ==== 206+0+0 (crc 0 0 0) 0x7f9290050040 con 0x7f92a0010070 2023-10-05T19:28:27.315 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.306+0000 7f92a8e6ce40 10 cache put: name=default.rgw.meta+users.keys+p5YnriCv1nAtykxBrupQ info.flags=0x0 2023-10-05T19:28:27.315 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.306+0000 7f92a8e6ce40 10 adding default.rgw.meta+users.keys+p5YnriCv1nAtykxBrupQ to cache LRU end 2023-10-05T19:28:27.315 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.306+0000 7f92a8e6ce40 20 get_system_obj_state: rctx=0x7fffab7e2018 obj=default.rgw.meta:users.keys:p5YnriCv1nAtykxBrupQ state=0x556e04a900b0 s->prefetch_data=0 2023-10-05T19:28:27.315 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.306+0000 7f92a8e6ce40 10 cache get: name=default.rgw.meta+users.keys+p5YnriCv1nAtykxBrupQ : hit (negative entry) 2023-10-05T19:28:27.315 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.306+0000 7f92a8e6ce40 1 -- 172.21.15.167:0/1223402549 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:105 8.18 8:1e6ca5ed:users.uid::foo:head [delete,create,call version.set in=58b,writefull 0~459 in=459b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x556e04c794c0 con 0x556e04bbb3b0 2023-10-05T19:28:27.317 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.310+0000 7f92a71a2700 1 -- 172.21.15.167:0/1223402549 <== osd.1 v2:172.21.15.167:6800/30035 37 ==== osd_op_reply(105 foo [delete,create,call,writefull 0~459] v30'4 uv4 ondisk = 0) v8 ==== 273+0+0 (crc 0 0 0) 0x7f9290050040 con 0x556e04bbb3b0 2023-10-05T19:28:27.317 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.310+0000 7f92a8e6ce40 10 cache put: name=default.rgw.meta+users.uid+foo info.flags=0x17 2023-10-05T19:28:27.317 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.310+0000 7f92a8e6ce40 10 moving default.rgw.meta+users.uid+foo to cache LRU end 2023-10-05T19:28:27.317 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.310+0000 7f92a8e6ce40 10 distributing notification oid=default.rgw.control:notify.4 cni=[op: 0, obj: default.rgw.meta:users.uid:foo, ofs0, ns] 2023-10-05T19:28:27.318 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.310+0000 7f92a8e6ce40 1 -- 172.21.15.167:0/1223402549 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:106 7.d 7:b4812045:::notify.4:head [notify cookie 93931014950576 in=647b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x556e04c780f0 con 0x556e04bbb3b0 2023-10-05T19:28:27.318 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.314+0000 7f92a71a2700 1 -- 172.21.15.167:0/1223402549 <== osd.1 v2:172.21.15.167:6800/30035 38 ==== watch-notify(notify (1) cookie 93931014414128 notify 128849018883 ret 0) v3 ==== 677+0+0 (crc 0 0 0) 0x7f929003d040 con 0x556e04bbb3b0 2023-10-05T19:28:27.318 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.314+0000 7f92a499d700 10 rgw watcher librados: RGWWatcher::handle_notify() notify_id 128849018883 cookie 93931014414128 notifier 4317 bl.length()=635 2023-10-05T19:28:27.318 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.314+0000 7f92a499d700 10 rgw watcher librados: cache put: name=default.rgw.meta+users.uid+foo info.flags=0x17 2023-10-05T19:28:27.319 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.314+0000 7f92a499d700 10 rgw watcher librados: moving default.rgw.meta+users.uid+foo to cache LRU end 2023-10-05T19:28:27.319 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.314+0000 7f92a499d700 1 -- 172.21.15.167:0/1223402549 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:107 7.d 7:b4812045:::notify.4:head [notify-ack in=20b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x7f928c003d90 con 0x556e04bbb3b0 2023-10-05T19:28:27.319 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.314+0000 7f92a71a2700 1 -- 172.21.15.167:0/1223402549 <== osd.1 v2:172.21.15.167:6800/30035 39 ==== osd_op_reply(106 notify.4 [notify cookie 93931014950576 out=8b] v0'0 uv45 ondisk = 0) v8 ==== 152+0+8 (crc 0 0 0) 0x7f9290050040 con 0x556e04bbb3b0 2023-10-05T19:28:27.319 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.314+0000 7f92a71a2700 1 -- 172.21.15.167:0/1223402549 <== osd.1 v2:172.21.15.167:6800/30035 40 ==== osd_op_reply(107 notify.4 [notify-ack] v0'0 uv45 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f9290050040 con 0x556e04bbb3b0 2023-10-05T19:28:27.320 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.314+0000 7f92a71a2700 1 -- 172.21.15.167:0/1223402549 <== osd.1 v2:172.21.15.167:6800/30035 41 ==== watch-notify(notify_complete (2) cookie 93931014950576 notify 128849018883 ret 0) v3 ==== 42+0+48 (crc 0 0 0) 0x7f92900337e0 con 0x556e04bbb3b0 2023-10-05T19:28:27.320 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.314+0000 7f92a8e6ce40 1 -- 172.21.15.167:0/1223402549 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:108 8.13 8:cc695d75:users.keys::p5YnriCv1nAtykxBrupQ:head [delete,create,writefull 0~7 in=7b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x556e04c78eb0 con 0x7f92a0010070 2023-10-05T19:28:27.322 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.318+0000 7f92a71a2700 1 -- 172.21.15.167:0/1223402549 <== osd.0 v2:172.21.15.118:6800/27690 30 ==== osd_op_reply(108 p5YnriCv1nAtykxBrupQ [delete,create,writefull 0~7] v30'1 uv1 ondisk = 0) v8 ==== 248+0+0 (crc 0 0 0) 0x7f9290050040 con 0x7f92a0010070 2023-10-05T19:28:27.323 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.318+0000 7f92a8e6ce40 10 cache put: name=default.rgw.meta+users.keys+p5YnriCv1nAtykxBrupQ info.flags=0x7 2023-10-05T19:28:27.323 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.318+0000 7f92a8e6ce40 10 moving default.rgw.meta+users.keys+p5YnriCv1nAtykxBrupQ to cache LRU end 2023-10-05T19:28:27.323 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.318+0000 7f92a8e6ce40 10 distributing notification oid=default.rgw.control:notify.2 cni=[op: 0, obj: default.rgw.meta:users.keys:p5YnriCv1nAtykxBrupQ, ofs0, ns] 2023-10-05T19:28:27.323 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.318+0000 7f92a8e6ce40 1 -- 172.21.15.167:0/1223402549 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:109 7.15 7:a93a5511:::notify.2:head [notify cookie 93931014950576 in=189b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x556e04c77d30 con 0x556e04bee940 2023-10-05T19:28:27.324 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.318+0000 7f92a69a1700 1 -- 172.21.15.167:0/1223402549 <== osd.3 v2:172.21.15.167:6808/30036 24 ==== watch-notify(notify (1) cookie 93931014410976 notify 128849018880 ret 0) v3 ==== 219+0+0 (crc 0 0 0) 0x556e04b2b390 con 0x556e04bee940 2023-10-05T19:28:27.324 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.318+0000 7f929ffff700 10 rgw watcher librados: RGWWatcher::handle_notify() notify_id 128849018880 cookie 93931014410976 notifier 4317 bl.length()=177 2023-10-05T19:28:27.324 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.318+0000 7f929ffff700 10 rgw watcher librados: cache put: name=default.rgw.meta+users.keys+p5YnriCv1nAtykxBrupQ info.flags=0x7 2023-10-05T19:28:27.324 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.318+0000 7f929ffff700 10 rgw watcher librados: moving default.rgw.meta+users.keys+p5YnriCv1nAtykxBrupQ to cache LRU end 2023-10-05T19:28:27.324 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.318+0000 7f929ffff700 1 -- 172.21.15.167:0/1223402549 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:110 7.15 7:a93a5511:::notify.2:head [notify-ack in=20b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x7f91f8003450 con 0x556e04bee940 2023-10-05T19:28:27.324 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.318+0000 7f92a69a1700 1 -- 172.21.15.167:0/1223402549 <== osd.3 v2:172.21.15.167:6808/30036 25 ==== osd_op_reply(109 notify.2 [notify cookie 93931014950576 out=8b] v0'0 uv45 ondisk = 0) v8 ==== 152+0+8 (crc 0 0 0) 0x7f9298053040 con 0x556e04bee940 2023-10-05T19:28:27.324 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.318+0000 7f92a69a1700 1 -- 172.21.15.167:0/1223402549 <== osd.3 v2:172.21.15.167:6808/30036 26 ==== osd_op_reply(110 notify.2 [notify-ack] v0'0 uv45 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f9298053040 con 0x556e04bee940 2023-10-05T19:28:27.325 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.318+0000 7f92a69a1700 1 -- 172.21.15.167:0/1223402549 <== osd.3 v2:172.21.15.167:6808/30036 27 ==== watch-notify(notify_complete (2) cookie 93931014950576 notify 128849018880 ret 0) v3 ==== 42+0+48 (crc 0 0 0) 0x556e04b2cfd0 con 0x556e04bee940 2023-10-05T19:28:27.325 INFO:teuthology.orchestra.run.smithi167.stdout:{ 2023-10-05T19:28:27.325 INFO:teuthology.orchestra.run.smithi167.stdout: "user_id": "foo", 2023-10-05T19:28:27.325 INFO:teuthology.orchestra.run.smithi167.stdout: "display_name": "Foo", 2023-10-05T19:28:27.325 INFO:teuthology.orchestra.run.smithi167.stdout: "email": "foo@foo.com", 2023-10-05T19:28:27.325 INFO:teuthology.orchestra.run.smithi167.stdout: "suspended": 0, 2023-10-05T19:28:27.325 INFO:teuthology.orchestra.run.smithi167.stdout: "max_buckets": 4, 2023-10-05T19:28:27.326 INFO:teuthology.orchestra.run.smithi167.stdout: "subusers": [], 2023-10-05T19:28:27.326 INFO:teuthology.orchestra.run.smithi167.stdout: "keys": [ 2023-10-05T19:28:27.326 INFO:teuthology.orchestra.run.smithi167.stdout: { 2023-10-05T19:28:27.326 INFO:teuthology.orchestra.run.smithi167.stdout: "user": "foo", 2023-10-05T19:28:27.326 INFO:teuthology.orchestra.run.smithi167.stdout: "access_key": "9te6NH5mcdcq0Tc5i8i1", 2023-10-05T19:28:27.326 INFO:teuthology.orchestra.run.smithi167.stdout: "secret_key": "Ny4IOauQoL18Gp2zM7lC1vLmoawgqcYP/YGcWfXu" 2023-10-05T19:28:27.326 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2023-10-05T19:28:27.326 INFO:teuthology.orchestra.run.smithi167.stdout: { 2023-10-05T19:28:27.326 INFO:teuthology.orchestra.run.smithi167.stdout: "user": "foo", 2023-10-05T19:28:27.326 INFO:teuthology.orchestra.run.smithi167.stdout: "access_key": "p5YnriCv1nAtykxBrupQ", 2023-10-05T19:28:27.326 INFO:teuthology.orchestra.run.smithi167.stdout: "secret_key": "Q8Tk6Q/27hfbFSYdSkPtUqhqx1GgzvpXa4WARozh" 2023-10-05T19:28:27.327 INFO:teuthology.orchestra.run.smithi167.stdout: } 2023-10-05T19:28:27.327 INFO:teuthology.orchestra.run.smithi167.stdout: ], 2023-10-05T19:28:27.327 INFO:teuthology.orchestra.run.smithi167.stdout: "swift_keys": [], 2023-10-05T19:28:27.327 INFO:teuthology.orchestra.run.smithi167.stdout: "caps": [], 2023-10-05T19:28:27.327 INFO:teuthology.orchestra.run.smithi167.stdout: "op_mask": "read, write, delete", 2023-10-05T19:28:27.327 INFO:teuthology.orchestra.run.smithi167.stdout: "default_placement": "", 2023-10-05T19:28:27.327 INFO:teuthology.orchestra.run.smithi167.stdout: "default_storage_class": "", 2023-10-05T19:28:27.327 INFO:teuthology.orchestra.run.smithi167.stdout: "placement_tags": [], 2023-10-05T19:28:27.327 INFO:teuthology.orchestra.run.smithi167.stdout: "bucket_quota": { 2023-10-05T19:28:27.327 INFO:teuthology.orchestra.run.smithi167.stdout: "enabled": false, 2023-10-05T19:28:27.328 INFO:teuthology.orchestra.run.smithi167.stdout: "check_on_raw": false, 2023-10-05T19:28:27.328 INFO:teuthology.orchestra.run.smithi167.stdout: "max_size": -1, 2023-10-05T19:28:27.328 INFO:teuthology.orchestra.run.smithi167.stdout: "max_size_kb": 0, 2023-10-05T19:28:27.328 INFO:teuthology.orchestra.run.smithi167.stdout: "max_objects": -1 2023-10-05T19:28:27.328 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2023-10-05T19:28:27.328 INFO:teuthology.orchestra.run.smithi167.stdout: "user_quota": { 2023-10-05T19:28:27.328 INFO:teuthology.orchestra.run.smithi167.stdout: "enabled": false, 2023-10-05T19:28:27.328 INFO:teuthology.orchestra.run.smithi167.stdout: "check_on_raw": false, 2023-10-05T19:28:27.328 INFO:teuthology.orchestra.run.smithi167.stdout: "max_size": -1, 2023-10-05T19:28:27.328 INFO:teuthology.orchestra.run.smithi167.stdout: "max_size_kb": 0, 2023-10-05T19:28:27.328 INFO:teuthology.orchestra.run.smithi167.stdout: "max_objects": -1 2023-10-05T19:28:27.329 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2023-10-05T19:28:27.329 INFO:teuthology.orchestra.run.smithi167.stdout: "temp_url_keys": [], 2023-10-05T19:28:27.329 INFO:teuthology.orchestra.run.smithi167.stdout: "type": "rgw", 2023-10-05T19:28:27.329 INFO:teuthology.orchestra.run.smithi167.stdout: "mfa_ids": [] 2023-10-05T19:28:27.329 INFO:teuthology.orchestra.run.smithi167.stdout:} 2023-10-05T19:28:27.329 INFO:teuthology.orchestra.run.smithi167.stdout: 2023-10-05T19:28:27.329 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.318+0000 7f92a8e6ce40 1 -- 172.21.15.167:0/1223402549 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:111 7.1f 7:f95f44c2:::notify.0:head [watch unwatch cookie 93931014404192] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x556e04bcc830 con 0x7f92980176e0 2023-10-05T19:28:27.330 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.322+0000 7f92a61a0700 1 -- 172.21.15.167:0/1223402549 <== osd.2 v2:172.21.15.167:6816/30037 17 ==== osd_op_reply(111 notify.0 [watch unwatch cookie 93931014404192] v30'47 uv45 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f92a0032040 con 0x7f92980176e0 2023-10-05T19:28:27.330 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.322+0000 7f92a8e6ce40 20 remove_watcher() i=0 2023-10-05T19:28:27.330 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.322+0000 7f92a8e6ce40 2 removed watcher, disabling cache 2023-10-05T19:28:27.330 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.322+0000 7f92a8e6ce40 1 -- 172.21.15.167:0/1223402549 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:112 7.0 7:05bf5b68:::notify.1:head [watch unwatch cookie 93931014407712] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x556e04bf3860 con 0x556e04bbb3b0 2023-10-05T19:28:27.331 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.326+0000 7f92a71a2700 1 -- 172.21.15.167:0/1223402549 <== osd.1 v2:172.21.15.167:6800/30035 42 ==== osd_op_reply(112 notify.1 [watch unwatch cookie 93931014407712] v30'47 uv45 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f9290050040 con 0x556e04bbb3b0 2023-10-05T19:28:27.331 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.326+0000 7f92a8e6ce40 20 remove_watcher() i=1 2023-10-05T19:28:27.331 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.326+0000 7f92a8e6ce40 1 -- 172.21.15.167:0/1223402549 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:113 7.15 7:a93a5511:::notify.2:head [watch unwatch cookie 93931014410976] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x556e04bf3860 con 0x556e04bee940 2023-10-05T19:28:27.334 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.326+0000 7f92a69a1700 1 -- 172.21.15.167:0/1223402549 <== osd.3 v2:172.21.15.167:6808/30036 28 ==== osd_op_reply(113 notify.2 [watch unwatch cookie 93931014410976] v30'47 uv45 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f9298053040 con 0x556e04bee940 2023-10-05T19:28:27.334 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.326+0000 7f92a8e6ce40 20 remove_watcher() i=2 2023-10-05T19:28:27.334 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.326+0000 7f92a8e6ce40 1 -- 172.21.15.167:0/1223402549 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:114 7.e 7:7759931f:::notify.3:head [watch unwatch cookie 93931014415392] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x556e04bf3860 con 0x7f92980176e0 2023-10-05T19:28:27.336 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.330+0000 7f92a61a0700 1 -- 172.21.15.167:0/1223402549 <== osd.2 v2:172.21.15.167:6816/30037 18 ==== osd_op_reply(114 notify.3 [watch unwatch cookie 93931014415392] v30'47 uv45 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f92a0032040 con 0x7f92980176e0 2023-10-05T19:28:27.336 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.330+0000 7f92a8e6ce40 20 remove_watcher() i=3 2023-10-05T19:28:27.336 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.330+0000 7f92a8e6ce40 1 -- 172.21.15.167:0/1223402549 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:115 7.d 7:b4812045:::notify.4:head [watch unwatch cookie 93931014414128] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x556e04bf3860 con 0x556e04bbb3b0 2023-10-05T19:28:27.339 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.334+0000 7f92a71a2700 1 -- 172.21.15.167:0/1223402549 <== osd.1 v2:172.21.15.167:6800/30035 43 ==== osd_op_reply(115 notify.4 [watch unwatch cookie 93931014414128] v30'47 uv45 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f9290050040 con 0x556e04bbb3b0 2023-10-05T19:28:27.339 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.334+0000 7f92a8e6ce40 20 remove_watcher() i=4 2023-10-05T19:28:27.339 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.334+0000 7f92a8e6ce40 1 -- 172.21.15.167:0/1223402549 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:116 7.3 7:c609908c:::notify.5:head [watch unwatch cookie 93931014422784] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x556e04bf3860 con 0x7f92a0010070 2023-10-05T19:28:27.342 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.338+0000 7f92a71a2700 1 -- 172.21.15.167:0/1223402549 <== osd.0 v2:172.21.15.118:6800/27690 31 ==== osd_op_reply(116 notify.5 [watch unwatch cookie 93931014422784] v30'47 uv45 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f9290050040 con 0x7f92a0010070 2023-10-05T19:28:27.342 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.338+0000 7f92a8e6ce40 20 remove_watcher() i=5 2023-10-05T19:28:27.342 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.338+0000 7f92a8e6ce40 1 -- 172.21.15.167:0/1223402549 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:117 7.14 7:2b04a3e9:::notify.6:head [watch unwatch cookie 93931014417600] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x556e04bf8100 con 0x7f92a0010070 2023-10-05T19:28:27.345 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.338+0000 7f92a71a2700 1 -- 172.21.15.167:0/1223402549 <== osd.0 v2:172.21.15.118:6800/27690 32 ==== osd_op_reply(117 notify.6 [watch unwatch cookie 93931014417600] v30'47 uv45 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f9290050040 con 0x7f92a0010070 2023-10-05T19:28:27.345 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.338+0000 7f92a8e6ce40 20 remove_watcher() i=6 2023-10-05T19:28:27.345 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.338+0000 7f92a8e6ce40 1 -- 172.21.15.167:0/1223402549 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:118 7.9 7:93e5b521:::notify.7:head [watch unwatch cookie 93931014426512] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x556e04bf3860 con 0x7f92a0010070 2023-10-05T19:28:27.348 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.342+0000 7f92a71a2700 1 -- 172.21.15.167:0/1223402549 <== osd.0 v2:172.21.15.118:6800/27690 33 ==== osd_op_reply(118 notify.7 [watch unwatch cookie 93931014426512] v30'47 uv45 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f9290050040 con 0x7f92a0010070 2023-10-05T19:28:27.348 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.342+0000 7f92a8e6ce40 20 remove_watcher() i=7 2023-10-05T19:28:27.349 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.342+0000 7f91f3fff700 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2023-10-05T19:28:27.349 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.342+0000 7f92a8e6ce40 1 -- 172.21.15.167:0/1223402549 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:119 6.4 6:22d26bf9:::data_loggenerations_metadata:head [watch unwatch cookie 93931014429088] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x556e04bf8f90 con 0x556e04bbb3b0 2023-10-05T19:28:27.353 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.346+0000 7f92a71a2700 1 -- 172.21.15.167:0/1223402549 <== osd.1 v2:172.21.15.167:6800/30035 44 ==== osd_op_reply(119 data_loggenerations_metadata [watch unwatch cookie 93931014429088] v30'65 uv1 ondisk = 0) v8 ==== 172+0+0 (crc 0 0 0) 0x7f9290050040 con 0x556e04bbb3b0 2023-10-05T19:28:27.353 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.346+0000 7f92a8e6ce40 1 -- 172.21.15.167:0/1223402549 >> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] conn(0x7f92a0010070 msgr2=0x7f92a001c430 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-05T19:28:27.353 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.346+0000 7f92a8e6ce40 1 --2- 172.21.15.167:0/1223402549 >> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] conn(0x7f92a0010070 0x7f92a001c430 crc :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-05T19:28:27.353 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.346+0000 7f92a8e6ce40 1 -- 172.21.15.167:0/1223402549 >> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] conn(0x556e04bbb3b0 msgr2=0x556e04beac20 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-05T19:28:27.353 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.346+0000 7f92a8e6ce40 1 --2- 172.21.15.167:0/1223402549 >> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] conn(0x556e04bbb3b0 0x556e04beac20 crc :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-05T19:28:27.353 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.346+0000 7f92a8e6ce40 1 -- 172.21.15.167:0/1223402549 >> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] conn(0x7f92980176e0 msgr2=0x7f929803d580 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-05T19:28:27.353 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.346+0000 7f92a8e6ce40 1 --2- 172.21.15.167:0/1223402549 >> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] conn(0x7f92980176e0 0x7f929803d580 crc :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-05T19:28:27.354 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.346+0000 7f92a8e6ce40 1 -- 172.21.15.167:0/1223402549 >> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] conn(0x556e04bee940 msgr2=0x556e04bf0d20 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-05T19:28:27.354 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.346+0000 7f92a8e6ce40 1 --2- 172.21.15.167:0/1223402549 >> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] conn(0x556e04bee940 0x556e04bf0d20 crc :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-05T19:28:27.354 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.346+0000 7f92a8e6ce40 1 -- 172.21.15.167:0/1223402549 >> [v2:172.21.15.167:6824/29889,v1:172.21.15.167:6825/29889] conn(0x7f9288030c00 msgr2=0x7f92880330b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-05T19:28:27.354 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.346+0000 7f92a8e6ce40 1 --2- 172.21.15.167:0/1223402549 >> [v2:172.21.15.167:6824/29889,v1:172.21.15.167:6825/29889] conn(0x7f9288030c00 0x7f92880330b0 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f92a0000f80 tx=0x7f92a0013000 comp rx=0 tx=0).stop 2023-10-05T19:28:27.354 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.346+0000 7f92a71a2700 1 -- 172.21.15.167:0/1223402549 reap_dead start 2023-10-05T19:28:27.354 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.346+0000 7f92a8e6ce40 1 -- 172.21.15.167:0/1223402549 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x556e04bb6dd0 msgr2=0x556e04b2ca90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-05T19:28:27.354 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.346+0000 7f92a8e6ce40 1 --2- 172.21.15.167:0/1223402549 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x556e04bb6dd0 0x556e04b2ca90 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x556e04bd26b0 tx=0x7f9298016000 comp rx=0 tx=0).stop 2023-10-05T19:28:27.354 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.346+0000 7f92a8e6ce40 1 -- 172.21.15.167:0/1223402549 shutdown_connections 2023-10-05T19:28:27.354 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.346+0000 7f92a8e6ce40 1 --2- 172.21.15.167:0/1223402549 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x556e04bb6dd0 0x556e04b2ca90 unknown :-1 s=CLOSED pgs=227 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-05T19:28:27.354 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.346+0000 7f92a8e6ce40 1 -- 172.21.15.167:0/1223402549 >> 172.21.15.167:0/1223402549 conn(0x556e04bb27f0 msgr2=0x556e04bd0610 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-05T19:28:27.354 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.346+0000 7f92a8e6ce40 1 -- 172.21.15.167:0/1223402549 shutdown_connections 2023-10-05T19:28:27.355 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.346+0000 7f92a8e6ce40 1 -- 172.21.15.167:0/1223402549 wait complete. 2023-10-05T19:28:27.355 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.346+0000 7f92a8e6ce40 1 -- 172.21.15.167:0/584546410 >> [v2:172.21.15.167:6824/29889,v1:172.21.15.167:6825/29889] conn(0x7f91e4030c00 msgr2=0x7f91e40330b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-05T19:28:27.355 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.346+0000 7f92a8e6ce40 1 --2- 172.21.15.167:0/584546410 >> [v2:172.21.15.167:6824/29889,v1:172.21.15.167:6825/29889] conn(0x7f91e4030c00 0x7f91e40330b0 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x556e04bf8730 tx=0x7f9290000f90 comp rx=0 tx=0).stop 2023-10-05T19:28:27.356 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.346+0000 7f92a8e6ce40 1 -- 172.21.15.167:0/584546410 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x556e04c072e0 msgr2=0x556e04c0ce10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-05T19:28:27.356 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.346+0000 7f92a8e6ce40 1 --2- 172.21.15.167:0/584546410 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x556e04c072e0 0x556e04c0ce10 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x556e04c01360 tx=0x7f92a002f000 comp rx=0 tx=0).stop 2023-10-05T19:28:27.356 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.350+0000 7f92a8e6ce40 1 -- 172.21.15.167:0/584546410 shutdown_connections 2023-10-05T19:28:27.356 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.350+0000 7f92a8e6ce40 1 --2- 172.21.15.167:0/584546410 >> [v2:172.21.15.167:6824/29889,v1:172.21.15.167:6825/29889] conn(0x7f91e4030c00 0x7f91e40330b0 unknown :-1 s=CLOSED pgs=83 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-05T19:28:27.356 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.350+0000 7f92a8e6ce40 1 --2- 172.21.15.167:0/584546410 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x556e04c072e0 0x556e04c0ce10 unknown :-1 s=CLOSED pgs=229 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-05T19:28:27.356 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.350+0000 7f92a8e6ce40 1 -- 172.21.15.167:0/584546410 >> 172.21.15.167:0/584546410 conn(0x556e04c026a0 msgr2=0x556e04c00740 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-05T19:28:27.356 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.350+0000 7f92a8e6ce40 1 -- 172.21.15.167:0/584546410 shutdown_connections 2023-10-05T19:28:27.356 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.350+0000 7f92a8e6ce40 1 -- 172.21.15.167:0/584546410 wait complete. 2023-10-05T19:28:27.360 DEBUG:tasks.util.rgw: json result: {'user_id': 'foo', 'display_name': 'Foo', 'email': 'foo@foo.com', 'suspended': 0, 'max_buckets': 4, 'subusers': [], 'keys': [{'user': 'foo', 'access_key': '9te6NH5mcdcq0Tc5i8i1', 'secret_key': 'Ny4IOauQoL18Gp2zM7lC1vLmoawgqcYP/YGcWfXu'}, {'user': 'foo', 'access_key': 'p5YnriCv1nAtykxBrupQ', 'secret_key': 'Q8Tk6Q/27hfbFSYdSkPtUqhqx1GgzvpXa4WARozh'}], 'swift_keys': [], 'caps': [], 'op_mask': 'read, write, delete', 'default_placement': '', 'default_storage_class': '', 'placement_tags': [], 'bucket_quota': {'enabled': False, 'check_on_raw': False, 'max_size': -1, 'max_size_kb': 0, 'max_objects': -1}, 'user_quota': {'enabled': False, 'check_on_raw': False, 'max_size': -1, 'max_size_kb': 0, 'max_objects': -1}, 'temp_url_keys': [], 'type': 'rgw', 'mfa_ids': []} 2023-10-05T19:28:27.360 INFO:tasks.util.rgw:rgwadmin: client.0 : ['user', 'info', '--uid', 'foo'] 2023-10-05T19:28:27.360 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'ceph', 'user', 'info', '--uid', 'foo'] 2023-10-05T19:28:27.361 DEBUG:teuthology.orchestra.run.smithi167:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster ceph user info --uid foo 2023-10-05T19:28:27.446 INFO:teuthology.orchestra.run.smithi167.stderr:ignoring --setuser ceph since I am not root 2023-10-05T19:28:27.446 INFO:teuthology.orchestra.run.smithi167.stderr:ignoring --setgroup ceph since I am not root 2023-10-05T19:28:27.447 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.442+0000 7f2117119e40 1 Processor -- start 2023-10-05T19:28:27.447 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.442+0000 7f2117119e40 1 -- start start 2023-10-05T19:28:27.447 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.442+0000 7f2117119e40 1 --2- >> v2:172.21.15.118:3300/0 conn(0x556bf4a02020 0x556bf4a023f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-05T19:28:27.447 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.442+0000 7f2117119e40 1 -- --> v1:172.21.15.118:6789/0 -- auth(proto 0 26 bytes epoch 0) v1 -- 0x556bf494cbd0 con 0x556bf4a02930 2023-10-05T19:28:27.447 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.442+0000 7f2117119e40 1 -- --> v2:172.21.15.118:3300/0 -- mon_getmap magic: 0 v1 -- 0x556bf4909ee0 con 0x556bf4a02020 2023-10-05T19:28:27.448 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.442+0000 7f2114c4e700 1 --2- >> v2:172.21.15.118:3300/0 conn(0x556bf4a02020 0x556bf4a023f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-05T19:28:27.448 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.442+0000 7f211444d700 1 --1- >> v1:172.21.15.118:6789/0 conn(0x556bf4a02930 0x556bf4a033d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.118:6789/0 says I am v1:172.21.15.167:56430/0 (socket says 172.21.15.167:56430) 2023-10-05T19:28:27.448 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.442+0000 7f211444d700 1 -- 172.21.15.167:0/388733938 learned_addr learned my addr 172.21.15.167:0/388733938 (peer_addr_for_me v1:172.21.15.167:0/0) 2023-10-05T19:28:27.449 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.442+0000 7f2113c4c700 1 -- 172.21.15.167:0/388733938 <== mon.0 v1:172.21.15.118:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1612916410 0 0) 0x556bf494cbd0 con 0x556bf4a02930 2023-10-05T19:28:27.449 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.442+0000 7f2113c4c700 1 -- 172.21.15.167:0/388733938 --> v1:172.21.15.118:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f20f4001e10 con 0x556bf4a02930 2023-10-05T19:28:27.449 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.442+0000 7f2114c4e700 1 -- 172.21.15.167:0/388733938 >> v1:172.21.15.118:6789/0 conn(0x556bf4a02930 legacy=0x556bf4a033d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-05T19:28:27.449 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.442+0000 7f2114c4e700 1 -- 172.21.15.167:0/388733938 --> v2:172.21.15.118:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x556bf4a03ae0 con 0x556bf4a02020 2023-10-05T19:28:27.450 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.442+0000 7f2114c4e700 1 --2- 172.21.15.167:0/388733938 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x556bf4a02020 0x556bf4a023f0 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7f210c00a520 tx=0x7f210c005d40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3b9ab402e4d6524e server_cookie=0 in_seq=0 out_seq=0 2023-10-05T19:28:27.450 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.442+0000 7f2113c4c700 1 -- 172.21.15.167:0/388733938 <== mon.0 v2:172.21.15.118:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f210c00f590 con 0x556bf4a02020 2023-10-05T19:28:27.450 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.442+0000 7f2113c4c700 1 -- 172.21.15.167:0/388733938 <== mon.0 v2:172.21.15.118:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f210c00f6f0 con 0x556bf4a02020 2023-10-05T19:28:27.450 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.442+0000 7f2113c4c700 1 -- 172.21.15.167:0/388733938 <== mon.0 v2:172.21.15.118:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f210c00f860 con 0x556bf4a02020 2023-10-05T19:28:27.450 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.446+0000 7f2117119e40 1 -- 172.21.15.167:0/388733938 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x556bf4a02020 msgr2=0x556bf4a023f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-05T19:28:27.450 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.446+0000 7f2117119e40 1 --2- 172.21.15.167:0/388733938 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x556bf4a02020 0x556bf4a023f0 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7f210c00a520 tx=0x7f210c005d40 comp rx=0 tx=0).stop 2023-10-05T19:28:27.451 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.446+0000 7f2117119e40 1 -- 172.21.15.167:0/388733938 shutdown_connections 2023-10-05T19:28:27.451 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.446+0000 7f2117119e40 1 --2- 172.21.15.167:0/388733938 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x556bf4a02020 0x556bf4a023f0 unknown :-1 s=CLOSED pgs=231 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-05T19:28:27.451 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.446+0000 7f2117119e40 1 -- 172.21.15.167:0/388733938 >> 172.21.15.167:0/388733938 conn(0x556bf4975870 msgr2=0x556bf4a00510 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-05T19:28:27.451 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.446+0000 7f2117119e40 1 -- 172.21.15.167:0/388733938 shutdown_connections 2023-10-05T19:28:27.451 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.446+0000 7f2117119e40 1 -- 172.21.15.167:0/388733938 wait complete. 2023-10-05T19:28:27.452 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.446+0000 7f2117119e40 1 Processor -- start 2023-10-05T19:28:27.452 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.446+0000 7f2117119e40 1 -- start start 2023-10-05T19:28:27.452 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.446+0000 7f2117119e40 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x556bf4a02020 0x556bf4a20e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-05T19:28:27.452 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.446+0000 7f2117119e40 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x556bf48cde30 con 0x556bf4a02020 2023-10-05T19:28:27.452 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.446+0000 7f2114c4e700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x556bf4a02020 0x556bf4a20e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-05T19:28:27.452 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.446+0000 7f2114c4e700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x556bf4a02020 0x556bf4a20e90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.118:3300/0 says I am v2:172.21.15.167:57508/0 (socket says 172.21.15.167:57508) 2023-10-05T19:28:27.452 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.446+0000 7f2114c4e700 1 -- 172.21.15.167:0/1915898080 learned_addr learned my addr 172.21.15.167:0/1915898080 (peer_addr_for_me v2:172.21.15.167:0/0) 2023-10-05T19:28:27.453 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.446+0000 7f2114c4e700 1 -- 172.21.15.167:0/1915898080 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f210c00a0e0 con 0x556bf4a02020 2023-10-05T19:28:27.453 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.446+0000 7f2114c4e700 1 --2- 172.21.15.167:0/1915898080 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x556bf4a02020 0x556bf4a20e90 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7f210c005d80 tx=0x7f210c014000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=be9e68c1a9685ecc server_cookie=0 in_seq=0 out_seq=0 2023-10-05T19:28:27.453 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.446+0000 7f2112c4a700 1 -- 172.21.15.167:0/1915898080 <== mon.0 v2:172.21.15.118:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f210c00f980 con 0x556bf4a02020 2023-10-05T19:28:27.453 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.446+0000 7f2112c4a700 1 -- 172.21.15.167:0/1915898080 <== mon.0 v2:172.21.15.118:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f210c00fb20 con 0x556bf4a02020 2023-10-05T19:28:27.453 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.446+0000 7f2112c4a700 1 -- 172.21.15.167:0/1915898080 <== mon.0 v2:172.21.15.118:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f210c00fcd0 con 0x556bf4a02020 2023-10-05T19:28:27.454 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.446+0000 7f2117119e40 1 -- 172.21.15.167:0/1915898080 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x556bf4a02020 msgr2=0x556bf4a20e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-05T19:28:27.455 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.446+0000 7f2117119e40 1 --2- 172.21.15.167:0/1915898080 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x556bf4a02020 0x556bf4a20e90 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7f210c005d80 tx=0x7f210c014000 comp rx=0 tx=0).stop 2023-10-05T19:28:27.455 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.446+0000 7f2117119e40 1 -- 172.21.15.167:0/1915898080 shutdown_connections 2023-10-05T19:28:27.455 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.446+0000 7f2117119e40 1 --2- 172.21.15.167:0/1915898080 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x556bf4a02020 0x556bf4a20e90 unknown :-1 s=CLOSED pgs=232 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-05T19:28:27.455 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.446+0000 7f2117119e40 1 -- 172.21.15.167:0/1915898080 >> 172.21.15.167:0/1915898080 conn(0x556bf4975870 msgr2=0x556bf4a1ca50 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-05T19:28:27.455 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.446+0000 7f2117119e40 1 -- 172.21.15.167:0/1915898080 shutdown_connections 2023-10-05T19:28:27.455 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.450+0000 7f2117119e40 1 -- 172.21.15.167:0/1915898080 wait complete. 2023-10-05T19:28:27.455 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.450+0000 7f2117119e40 1 Processor -- start 2023-10-05T19:28:27.455 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.450+0000 7f2117119e40 1 -- start start 2023-10-05T19:28:27.455 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.450+0000 7f2117119e40 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x556bf4a02020 0x556bf4a27b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-05T19:28:27.455 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.450+0000 7f2117119e40 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x556bf48a0ed0 con 0x556bf4a02020 2023-10-05T19:28:27.456 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.450+0000 7f2114c4e700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x556bf4a02020 0x556bf4a27b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-05T19:28:27.457 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.450+0000 7f2114c4e700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x556bf4a02020 0x556bf4a27b90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.118:3300/0 says I am v2:172.21.15.167:57518/0 (socket says 172.21.15.167:57518) 2023-10-05T19:28:27.457 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.450+0000 7f2114c4e700 1 -- 172.21.15.167:0/1497151981 learned_addr learned my addr 172.21.15.167:0/1497151981 (peer_addr_for_me v2:172.21.15.167:0/0) 2023-10-05T19:28:27.457 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.450+0000 7f2114c4e700 1 -- 172.21.15.167:0/1497151981 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x556bf4a03ae0 con 0x556bf4a02020 2023-10-05T19:28:27.457 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.450+0000 7f2114c4e700 1 --2- 172.21.15.167:0/1497151981 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x556bf4a02020 0x556bf4a27b90 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x556bf4a1d570 tx=0x7f210c01d000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-05T19:28:27.457 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.450+0000 7f2111447700 1 -- 172.21.15.167:0/1497151981 <== mon.0 v2:172.21.15.118:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f210c02c070 con 0x556bf4a02020 2023-10-05T19:28:27.457 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.450+0000 7f2111447700 1 -- 172.21.15.167:0/1497151981 <== mon.0 v2:172.21.15.118:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f210c031460 con 0x556bf4a02020 2023-10-05T19:28:27.457 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.450+0000 7f2117119e40 1 -- 172.21.15.167:0/1497151981 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x556bf4a280d0 con 0x556bf4a02020 2023-10-05T19:28:27.457 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.450+0000 7f2111447700 1 -- 172.21.15.167:0/1497151981 <== mon.0 v2:172.21.15.118:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f210c00fea0 con 0x556bf4a02020 2023-10-05T19:28:27.458 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.450+0000 7f2117119e40 1 -- 172.21.15.167:0/1497151981 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x556bf4a28510 con 0x556bf4a02020 2023-10-05T19:28:27.458 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.454+0000 7f2117119e40 20 rados->read ofs=0 len=0 2023-10-05T19:28:27.458 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.454+0000 7f2111447700 1 -- 172.21.15.167:0/1497151981 <== mon.0 v2:172.21.15.118:3300/0 4 ==== mgrmap(e 5) v1 ==== 40329+0+0 (secure 0 0 0) 0x7f210c010420 con 0x556bf4a02020 2023-10-05T19:28:27.459 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.454+0000 7f2111447700 1 --2- 172.21.15.167:0/1497151981 >> [v2:172.21.15.167:6824/29889,v1:172.21.15.167:6825/29889] conn(0x7f20f8030c50 0x7f20f8033100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-05T19:28:27.459 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.454+0000 7f2111447700 1 -- 172.21.15.167:0/1497151981 <== mon.0 v2:172.21.15.118:3300/0 5 ==== osd_map(30..30 src has 1..30) v4 ==== 6057+0+0 (secure 0 0 0) 0x7f210c046b00 con 0x556bf4a02020 2023-10-05T19:28:27.459 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.454+0000 7f211444d700 1 --2- 172.21.15.167:0/1497151981 >> [v2:172.21.15.167:6824/29889,v1:172.21.15.167:6825/29889] conn(0x7f20f8030c50 0x7f20f8033100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-05T19:28:27.460 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.454+0000 7f2117119e40 1 --2- 172.21.15.167:0/1497151981 >> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] conn(0x556bf4a02930 0x556bf4a335d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-05T19:28:27.460 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.454+0000 7f2117119e40 1 -- 172.21.15.167:0/1497151981 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:1 5.12 5:49953fa1:::default.realm:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x556bf4a33c40 con 0x556bf4a02930 2023-10-05T19:28:27.460 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.454+0000 7f211544f700 1 --2- 172.21.15.167:0/1497151981 >> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] conn(0x556bf4a02930 0x556bf4a335d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-05T19:28:27.460 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.454+0000 7f211444d700 1 --2- 172.21.15.167:0/1497151981 >> [v2:172.21.15.167:6824/29889,v1:172.21.15.167:6825/29889] conn(0x7f20f8030c50 0x7f20f8033100 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f20fc004170 tx=0x7f20fc001400 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-05T19:28:27.462 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.454+0000 7f211544f700 1 --2- 172.21.15.167:0/1497151981 >> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] conn(0x556bf4a02930 0x556bf4a335d0 crc :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-05T19:28:27.462 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.454+0000 7f211544f700 1 -- 172.21.15.167:0/1497151981 <== osd.1 v2:172.21.15.167:6800/30035 1 ==== osd_op_reply(1 default.realm [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 ==== 157+0+0 (crc 0 0 0) 0x7f210400a500 con 0x556bf4a02930 2023-10-05T19:28:27.462 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.454+0000 7f2117119e40 20 rados_obj.operate() r=-2 bl.length=0 2023-10-05T19:28:27.462 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.454+0000 7f2117119e40 20 realm 2023-10-05T19:28:27.462 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.454+0000 7f2117119e40 20 rados->read ofs=0 len=0 2023-10-05T19:28:27.462 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.454+0000 7f2117119e40 1 -- 172.21.15.167:0/1497151981 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:2 5.12 5:49953fa1:::default.realm:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x556bf4a30c30 con 0x556bf4a02930 2023-10-05T19:28:27.462 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.454+0000 7f211544f700 1 -- 172.21.15.167:0/1497151981 <== osd.1 v2:172.21.15.167:6800/30035 2 ==== osd_op_reply(2 default.realm [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 ==== 157+0+0 (crc 0 0 0) 0x7f210400a500 con 0x556bf4a02930 2023-10-05T19:28:27.462 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.454+0000 7f2117119e40 20 rados_obj.operate() r=-2 bl.length=0 2023-10-05T19:28:27.462 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.454+0000 7f2117119e40 4 RGWPeriod::init failed to init realm id : (2) No such file or directory 2023-10-05T19:28:27.462 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.454+0000 7f2117119e40 20 rados->read ofs=0 len=0 2023-10-05T19:28:27.463 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.454+0000 7f2117119e40 1 -- 172.21.15.167:0/1497151981 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:3 5.12 5:49953fa1:::default.realm:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x556bf4a34720 con 0x556bf4a02930 2023-10-05T19:28:27.463 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.454+0000 7f211544f700 1 -- 172.21.15.167:0/1497151981 <== osd.1 v2:172.21.15.167:6800/30035 3 ==== osd_op_reply(3 default.realm [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 ==== 157+0+0 (crc 0 0 0) 0x7f210400a500 con 0x556bf4a02930 2023-10-05T19:28:27.463 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.454+0000 7f2117119e40 20 rados_obj.operate() r=-2 bl.length=0 2023-10-05T19:28:27.463 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.454+0000 7f2117119e40 20 rados->read ofs=0 len=0 2023-10-05T19:28:27.463 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.454+0000 7f2117119e40 1 -- 172.21.15.167:0/1497151981 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:4 5.f 5:f43b6ece:::zone_names.default:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x556bf4a34300 con 0x556bf4a02930 2023-10-05T19:28:27.464 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.458+0000 7f211544f700 1 -- 172.21.15.167:0/1497151981 <== osd.1 v2:172.21.15.167:6800/30035 4 ==== osd_op_reply(4 zone_names.default [read 0~46 out=46b] v0'0 uv1 ondisk = 0) v8 ==== 162+0+46 (crc 0 0 0) 0x7f210400a500 con 0x556bf4a02930 2023-10-05T19:28:27.464 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.458+0000 7f2117119e40 20 rados_obj.operate() r=0 bl.length=46 2023-10-05T19:28:27.464 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.458+0000 7f2117119e40 20 rados->read ofs=0 len=0 2023-10-05T19:28:27.464 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.458+0000 7f2117119e40 1 -- 172.21.15.167:0/1497151981 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:5 5.1d 5:b8374eb4:::zone_info.9a9df965-6612-45d4-a45f-6e3c889bacc2:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x556bf4a35190 con 0x556bf4a02930 2023-10-05T19:28:27.464 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.458+0000 7f211544f700 1 -- 172.21.15.167:0/1497151981 <== osd.1 v2:172.21.15.167:6800/30035 5 ==== osd_op_reply(5 zone_info.9a9df965-6612-45d4-a45f-6e3c889bacc2 [read 0~874 out=874b] v0'0 uv2 ondisk = 0) v8 ==== 190+0+874 (crc 0 0 0) 0x7f210400a500 con 0x556bf4a02930 2023-10-05T19:28:27.465 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.458+0000 7f2117119e40 20 rados_obj.operate() r=0 bl.length=874 2023-10-05T19:28:27.465 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.458+0000 7f2117119e40 20 searching for the correct realm 2023-10-05T19:28:27.465 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.458+0000 7f2117119e40 1 --2- 172.21.15.167:0/1497151981 >> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] conn(0x556bf4a36fe0 0x556bf4a393c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-05T19:28:27.465 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.458+0000 7f2117119e40 1 -- 172.21.15.167:0/1497151981 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:6 5.0 5:00000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x556bf4a39900 con 0x556bf4a36fe0 2023-10-05T19:28:27.465 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.458+0000 7f2114c4e700 1 --2- 172.21.15.167:0/1497151981 >> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] conn(0x556bf4a36fe0 0x556bf4a393c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-05T19:28:27.465 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.458+0000 7f2114c4e700 1 --2- 172.21.15.167:0/1497151981 >> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] conn(0x556bf4a36fe0 0x556bf4a393c0 crc :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-05T19:28:27.466 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.458+0000 7f2114c4e700 1 -- 172.21.15.167:0/1497151981 <== osd.3 v2:172.21.15.167:6808/30036 1 ==== osd_op_reply(6 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f210c0397c0 con 0x556bf4a36fe0 2023-10-05T19:28:27.466 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.458+0000 7f2114c4e700 1 --2- 172.21.15.167:0/1497151981 >> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] conn(0x7f210c03a020 0x7f210c03a730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-05T19:28:27.466 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.458+0000 7f2114c4e700 1 -- 172.21.15.167:0/1497151981 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:7 5.10 5:08000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x556bf4a39900 con 0x7f210c03a020 2023-10-05T19:28:27.466 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.458+0000 7f211444d700 1 --2- 172.21.15.167:0/1497151981 >> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] conn(0x7f210c03a020 0x7f210c03a730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-05T19:28:27.466 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.458+0000 7f211444d700 1 --2- 172.21.15.167:0/1497151981 >> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] conn(0x7f210c03a020 0x7f210c03a730 crc :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-05T19:28:27.467 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.462+0000 7f211444d700 1 -- 172.21.15.167:0/1497151981 <== osd.2 v2:172.21.15.167:6816/30037 1 ==== osd_op_reply(7 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f20fc006070 con 0x7f210c03a020 2023-10-05T19:28:27.467 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.462+0000 7f211444d700 1 --2- 172.21.15.167:0/1497151981 >> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] conn(0x7f20fc00ebe0 0x7f20fc00d8e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-05T19:28:27.467 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.462+0000 7f211444d700 1 -- 172.21.15.167:0/1497151981 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:8 5.8 5:10000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x556bf4a39900 con 0x7f20fc00ebe0 2023-10-05T19:28:27.467 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.462+0000 7f211544f700 1 --2- 172.21.15.167:0/1497151981 >> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] conn(0x7f20fc00ebe0 0x7f20fc00d8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-05T19:28:27.467 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.462+0000 7f211544f700 1 --2- 172.21.15.167:0/1497151981 >> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] conn(0x7f20fc00ebe0 0x7f20fc00d8e0 crc :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-05T19:28:27.468 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.462+0000 7f211544f700 1 -- 172.21.15.167:0/1497151981 <== osd.0 v2:172.21.15.118:6800/27690 1 ==== osd_op_reply(8 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f2104014040 con 0x7f20fc00ebe0 2023-10-05T19:28:27.468 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.462+0000 7f211544f700 1 -- 172.21.15.167:0/1497151981 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:9 5.18 5:18000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x556bf4a39900 con 0x556bf4a02930 2023-10-05T19:28:27.469 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.462+0000 7f211544f700 1 -- 172.21.15.167:0/1497151981 <== osd.1 v2:172.21.15.167:6800/30035 6 ==== osd_op_reply(9 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f2104014040 con 0x556bf4a02930 2023-10-05T19:28:27.469 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.462+0000 7f211544f700 1 -- 172.21.15.167:0/1497151981 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:10 5.4 5:20000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x556bf4a39900 con 0x556bf4a02930 2023-10-05T19:28:27.469 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.462+0000 7f211544f700 1 -- 172.21.15.167:0/1497151981 <== osd.1 v2:172.21.15.167:6800/30035 7 ==== osd_op_reply(10 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f2104014040 con 0x556bf4a02930 2023-10-05T19:28:27.469 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.462+0000 7f211544f700 1 -- 172.21.15.167:0/1497151981 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:11 5.14 5:28000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x556bf4a39900 con 0x7f20fc00ebe0 2023-10-05T19:28:27.470 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.466+0000 7f211544f700 1 -- 172.21.15.167:0/1497151981 <== osd.0 v2:172.21.15.118:6800/27690 2 ==== osd_op_reply(11 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f2104014040 con 0x7f20fc00ebe0 2023-10-05T19:28:27.470 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.466+0000 7f211544f700 1 -- 172.21.15.167:0/1497151981 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:12 5.c 5:30000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x556bf4a39900 con 0x7f20fc00ebe0 2023-10-05T19:28:27.471 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.466+0000 7f211544f700 1 -- 172.21.15.167:0/1497151981 <== osd.0 v2:172.21.15.118:6800/27690 3 ==== osd_op_reply(12 [pgnls start_epoch 30 out=112b] v23'1 uv1 ondisk = 1) v8 ==== 144+0+112 (crc 0 0 0) 0x7f2104014040 con 0x7f20fc00ebe0 2023-10-05T19:28:27.471 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.466+0000 7f211544f700 1 -- 172.21.15.167:0/1497151981 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:13 5.1c 5:38000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x556bf4a39900 con 0x7f210c03a020 2023-10-05T19:28:27.472 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.466+0000 7f211444d700 1 -- 172.21.15.167:0/1497151981 <== osd.2 v2:172.21.15.167:6816/30037 2 ==== osd_op_reply(13 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f20fc006070 con 0x7f210c03a020 2023-10-05T19:28:27.472 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.466+0000 7f211444d700 1 -- 172.21.15.167:0/1497151981 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:14 5.2 5:40000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x556bf4a39900 con 0x556bf4a36fe0 2023-10-05T19:28:27.472 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.466+0000 7f2114c4e700 1 -- 172.21.15.167:0/1497151981 <== osd.3 v2:172.21.15.167:6808/30036 2 ==== osd_op_reply(14 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f210c0397c0 con 0x556bf4a36fe0 2023-10-05T19:28:27.472 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.466+0000 7f2114c4e700 1 -- 172.21.15.167:0/1497151981 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:15 5.12 5:48000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x556bf4a39900 con 0x556bf4a02930 2023-10-05T19:28:27.473 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.466+0000 7f211544f700 1 -- 172.21.15.167:0/1497151981 <== osd.1 v2:172.21.15.167:6800/30035 8 ==== osd_op_reply(15 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f2104014040 con 0x556bf4a02930 2023-10-05T19:28:27.473 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.466+0000 7f211544f700 1 -- 172.21.15.167:0/1497151981 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:16 5.a 5:50000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x556bf4a39900 con 0x7f20fc00ebe0 2023-10-05T19:28:27.474 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.470+0000 7f211544f700 1 -- 172.21.15.167:0/1497151981 <== osd.0 v2:172.21.15.118:6800/27690 4 ==== osd_op_reply(16 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f2104022040 con 0x7f20fc00ebe0 2023-10-05T19:28:27.474 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.470+0000 7f211544f700 1 -- 172.21.15.167:0/1497151981 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:17 5.1a 5:58000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x556bf4a39900 con 0x556bf4a02930 2023-10-05T19:28:27.475 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.470+0000 7f211544f700 1 -- 172.21.15.167:0/1497151981 <== osd.1 v2:172.21.15.167:6800/30035 9 ==== osd_op_reply(17 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f2104022040 con 0x556bf4a02930 2023-10-05T19:28:27.475 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.470+0000 7f211544f700 1 -- 172.21.15.167:0/1497151981 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:18 5.6 5:60000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x556bf4a39900 con 0x7f210c03a020 2023-10-05T19:28:27.475 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.470+0000 7f211444d700 1 -- 172.21.15.167:0/1497151981 <== osd.2 v2:172.21.15.167:6816/30037 3 ==== osd_op_reply(18 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f20fc006070 con 0x7f210c03a020 2023-10-05T19:28:27.475 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.470+0000 7f211444d700 1 -- 172.21.15.167:0/1497151981 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:19 5.16 5:68000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x556bf4a39900 con 0x556bf4a36fe0 2023-10-05T19:28:27.476 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.470+0000 7f2114c4e700 1 -- 172.21.15.167:0/1497151981 <== osd.3 v2:172.21.15.167:6808/30036 3 ==== osd_op_reply(19 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f210c0397c0 con 0x556bf4a36fe0 2023-10-05T19:28:27.476 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.470+0000 7f2114c4e700 1 -- 172.21.15.167:0/1497151981 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:20 5.e 5:70000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x556bf4a39900 con 0x556bf4a36fe0 2023-10-05T19:28:27.477 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.470+0000 7f2114c4e700 1 -- 172.21.15.167:0/1497151981 <== osd.3 v2:172.21.15.167:6808/30036 4 ==== osd_op_reply(20 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f210c0397c0 con 0x556bf4a36fe0 2023-10-05T19:28:27.477 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.470+0000 7f2114c4e700 1 -- 172.21.15.167:0/1497151981 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:21 5.1e 5:78000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x556bf4a39900 con 0x7f20fc00ebe0 2023-10-05T19:28:27.478 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.470+0000 7f211544f700 1 -- 172.21.15.167:0/1497151981 <== osd.0 v2:172.21.15.118:6800/27690 5 ==== osd_op_reply(21 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f2104022040 con 0x7f20fc00ebe0 2023-10-05T19:28:27.478 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.474+0000 7f211544f700 1 -- 172.21.15.167:0/1497151981 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:22 5.1 5:80000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x556bf4a39900 con 0x556bf4a36fe0 2023-10-05T19:28:27.478 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.474+0000 7f2114c4e700 1 -- 172.21.15.167:0/1497151981 <== osd.3 v2:172.21.15.167:6808/30036 5 ==== osd_op_reply(22 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f210c0397c0 con 0x556bf4a36fe0 2023-10-05T19:28:27.479 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.474+0000 7f2114c4e700 1 -- 172.21.15.167:0/1497151981 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:23 5.11 5:88000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x556bf4a39900 con 0x556bf4a36fe0 2023-10-05T19:28:27.479 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.474+0000 7f2114c4e700 1 -- 172.21.15.167:0/1497151981 <== osd.3 v2:172.21.15.167:6808/30036 6 ==== osd_op_reply(23 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f210c0397c0 con 0x556bf4a36fe0 2023-10-05T19:28:27.479 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.474+0000 7f2114c4e700 1 -- 172.21.15.167:0/1497151981 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:24 5.9 5:90000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x556bf4a39900 con 0x556bf4a02930 2023-10-05T19:28:27.480 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.474+0000 7f211544f700 1 -- 172.21.15.167:0/1497151981 <== osd.1 v2:172.21.15.167:6800/30035 10 ==== osd_op_reply(24 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f2104022040 con 0x556bf4a02930 2023-10-05T19:28:27.480 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.474+0000 7f211544f700 1 -- 172.21.15.167:0/1497151981 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:25 5.19 5:98000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x556bf4a39900 con 0x556bf4a02930 2023-10-05T19:28:27.481 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.474+0000 7f211544f700 1 -- 172.21.15.167:0/1497151981 <== osd.1 v2:172.21.15.167:6800/30035 11 ==== osd_op_reply(25 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f2104022040 con 0x556bf4a02930 2023-10-05T19:28:27.481 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.474+0000 7f211544f700 1 -- 172.21.15.167:0/1497151981 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:26 5.5 5:a0000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x556bf4a39900 con 0x7f20fc00ebe0 2023-10-05T19:28:27.482 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.474+0000 7f211544f700 1 -- 172.21.15.167:0/1497151981 <== osd.0 v2:172.21.15.118:6800/27690 6 ==== osd_op_reply(26 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f2104022040 con 0x7f20fc00ebe0 2023-10-05T19:28:27.482 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.474+0000 7f211544f700 1 -- 172.21.15.167:0/1497151981 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:27 5.15 5:a8000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x556bf4a39900 con 0x556bf4a36fe0 2023-10-05T19:28:27.482 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.478+0000 7f2114c4e700 1 -- 172.21.15.167:0/1497151981 <== osd.3 v2:172.21.15.167:6808/30036 7 ==== osd_op_reply(27 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f210c0397c0 con 0x556bf4a36fe0 2023-10-05T19:28:27.483 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.478+0000 7f2114c4e700 1 -- 172.21.15.167:0/1497151981 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:28 5.d 5:b0000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x556bf4a39900 con 0x7f210c03a020 2023-10-05T19:28:27.483 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.478+0000 7f211444d700 1 -- 172.21.15.167:0/1497151981 <== osd.2 v2:172.21.15.167:6816/30037 4 ==== osd_op_reply(28 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f20fc006070 con 0x7f210c03a020 2023-10-05T19:28:27.483 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.478+0000 7f211444d700 1 -- 172.21.15.167:0/1497151981 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:29 5.1d 5:b8000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x556bf4a39900 con 0x556bf4a02930 2023-10-05T19:28:27.484 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.478+0000 7f211544f700 1 -- 172.21.15.167:0/1497151981 <== osd.1 v2:172.21.15.167:6800/30035 12 ==== osd_op_reply(29 [pgnls start_epoch 30 out=107b] v30'2 uv2 ondisk = 1) v8 ==== 144+0+107 (crc 0 0 0) 0x7f2104022040 con 0x556bf4a02930 2023-10-05T19:28:27.485 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.478+0000 7f211544f700 1 -- 172.21.15.167:0/1497151981 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:30 5.3 5:c0000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x556bf4a39900 con 0x7f20fc00ebe0 2023-10-05T19:28:27.485 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.478+0000 7f211544f700 1 -- 172.21.15.167:0/1497151981 <== osd.0 v2:172.21.15.118:6800/27690 7 ==== osd_op_reply(30 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f2104022040 con 0x7f20fc00ebe0 2023-10-05T19:28:27.485 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.478+0000 7f211544f700 1 -- 172.21.15.167:0/1497151981 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:31 5.13 5:c8000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x556bf4a39900 con 0x556bf4a36fe0 2023-10-05T19:28:27.486 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.482+0000 7f2114c4e700 1 -- 172.21.15.167:0/1497151981 <== osd.3 v2:172.21.15.167:6808/30036 8 ==== osd_op_reply(31 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f210c0397c0 con 0x556bf4a36fe0 2023-10-05T19:28:27.486 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.482+0000 7f2114c4e700 1 -- 172.21.15.167:0/1497151981 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:32 5.b 5:d0000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x556bf4a39900 con 0x7f20fc00ebe0 2023-10-05T19:28:27.487 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.482+0000 7f211544f700 1 -- 172.21.15.167:0/1497151981 <== osd.0 v2:172.21.15.118:6800/27690 8 ==== osd_op_reply(32 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f2104022040 con 0x7f20fc00ebe0 2023-10-05T19:28:27.488 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.482+0000 7f211544f700 1 -- 172.21.15.167:0/1497151981 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:33 5.1b 5:d8000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x556bf4a39900 con 0x7f210c03a020 2023-10-05T19:28:27.488 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.482+0000 7f211444d700 1 -- 172.21.15.167:0/1497151981 <== osd.2 v2:172.21.15.167:6816/30037 5 ==== osd_op_reply(33 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f20fc006070 con 0x7f210c03a020 2023-10-05T19:28:27.488 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.482+0000 7f211444d700 1 -- 172.21.15.167:0/1497151981 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:34 5.7 5:e0000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x556bf4a39900 con 0x556bf4a36fe0 2023-10-05T19:28:27.489 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.482+0000 7f2114c4e700 1 -- 172.21.15.167:0/1497151981 <== osd.3 v2:172.21.15.167:6808/30036 9 ==== osd_op_reply(34 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f210c0397c0 con 0x556bf4a36fe0 2023-10-05T19:28:27.489 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.482+0000 7f2114c4e700 1 -- 172.21.15.167:0/1497151981 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:35 5.17 5:e8000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x556bf4a39900 con 0x7f20fc00ebe0 2023-10-05T19:28:27.490 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.486+0000 7f211544f700 1 -- 172.21.15.167:0/1497151981 <== osd.0 v2:172.21.15.118:6800/27690 9 ==== osd_op_reply(35 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f2104022040 con 0x7f20fc00ebe0 2023-10-05T19:28:27.490 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.486+0000 7f211544f700 1 -- 172.21.15.167:0/1497151981 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:36 5.f 5:f0000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x556bf4a39900 con 0x556bf4a02930 2023-10-05T19:28:27.491 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.486+0000 7f211544f700 1 -- 172.21.15.167:0/1497151981 <== osd.1 v2:172.21.15.167:6800/30035 13 ==== osd_op_reply(36 [pgnls start_epoch 30 out=115b] v23'2 uv2 ondisk = 1) v8 ==== 144+0+115 (crc 0 0 0) 0x7f210402f080 con 0x556bf4a02930 2023-10-05T19:28:27.491 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.486+0000 7f211544f700 1 -- 172.21.15.167:0/1497151981 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:37 5.1f 5:f8000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x556bf4a39900 con 0x7f210c03a020 2023-10-05T19:28:27.492 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.486+0000 7f211444d700 1 -- 172.21.15.167:0/1497151981 <== osd.2 v2:172.21.15.167:6816/30037 6 ==== osd_op_reply(37 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f20fc006070 con 0x7f210c03a020 2023-10-05T19:28:27.492 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.486+0000 7f2117119e40 20 RGWRados::pool_iterate: got zonegroup_info.4f1f719d-9648-417b-b10c-e5d939825419 2023-10-05T19:28:27.492 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.486+0000 7f2117119e40 20 RGWRados::pool_iterate: got zone_info.9a9df965-6612-45d4-a45f-6e3c889bacc2 2023-10-05T19:28:27.492 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.486+0000 7f2117119e40 20 RGWRados::pool_iterate: got zone_names.default 2023-10-05T19:28:27.493 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.486+0000 7f2117119e40 20 RGWRados::pool_iterate: got zonegroups_names.default 2023-10-05T19:28:27.493 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.486+0000 7f2117119e40 20 rados->read ofs=0 len=0 2023-10-05T19:28:27.493 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.486+0000 7f2117119e40 1 -- 172.21.15.167:0/1497151981 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:38 5.12 5:49953fa1:::default.realm:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x556bf4a35d90 con 0x556bf4a02930 2023-10-05T19:28:27.494 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.486+0000 7f211544f700 1 -- 172.21.15.167:0/1497151981 <== osd.1 v2:172.21.15.167:6800/30035 14 ==== osd_op_reply(38 default.realm [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 ==== 157+0+0 (crc 0 0 0) 0x7f210402f080 con 0x556bf4a02930 2023-10-05T19:28:27.494 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.486+0000 7f2117119e40 20 rados_obj.operate() r=-2 bl.length=0 2023-10-05T19:28:27.494 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.486+0000 7f2117119e40 20 rados->read ofs=0 len=0 2023-10-05T19:28:27.494 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.486+0000 7f2117119e40 1 -- 172.21.15.167:0/1497151981 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:39 5.f 5:f4c53578:::zonegroups_names.default:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x556bf4a36710 con 0x556bf4a02930 2023-10-05T19:28:27.494 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.486+0000 7f211544f700 1 -- 172.21.15.167:0/1497151981 <== osd.1 v2:172.21.15.167:6800/30035 15 ==== osd_op_reply(39 zonegroups_names.default [read 0~46 out=46b] v0'0 uv2 ondisk = 0) v8 ==== 168+0+46 (crc 0 0 0) 0x7f210402f080 con 0x556bf4a02930 2023-10-05T19:28:27.494 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.486+0000 7f2117119e40 20 rados_obj.operate() r=0 bl.length=46 2023-10-05T19:28:27.494 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.486+0000 7f2117119e40 20 rados->read ofs=0 len=0 2023-10-05T19:28:27.494 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.486+0000 7f2117119e40 1 -- 172.21.15.167:0/1497151981 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:40 5.c 5:30d299a5:::zonegroup_info.4f1f719d-9648-417b-b10c-e5d939825419:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x556bf4a35980 con 0x7f20fc00ebe0 2023-10-05T19:28:27.495 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.490+0000 7f211544f700 1 -- 172.21.15.167:0/1497151981 <== osd.0 v2:172.21.15.118:6800/27690 10 ==== osd_op_reply(40 zonegroup_info.4f1f719d-9648-417b-b10c-e5d939825419 [read 0~362 out=362b] v0'0 uv1 ondisk = 0) v8 ==== 195+0+362 (crc 0 0 0) 0x7f210402f080 con 0x7f20fc00ebe0 2023-10-05T19:28:27.496 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.490+0000 7f2117119e40 20 rados_obj.operate() r=0 bl.length=362 2023-10-05T19:28:27.496 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.490+0000 7f2117119e40 20 rados->read ofs=0 len=0 2023-10-05T19:28:27.496 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.490+0000 7f2117119e40 1 -- 172.21.15.167:0/1497151981 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:41 5.13 5:cbbf6c60:::converted:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x556bf4a3a1f0 con 0x556bf4a36fe0 2023-10-05T19:28:27.496 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.490+0000 7f2114c4e700 1 -- 172.21.15.167:0/1497151981 <== osd.3 v2:172.21.15.167:6808/30036 10 ==== osd_op_reply(41 converted [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 ==== 153+0+0 (crc 0 0 0) 0x7f210c0397c0 con 0x556bf4a36fe0 2023-10-05T19:28:27.496 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.490+0000 7f2117119e40 20 rados_obj.operate() r=-2 bl.length=0 2023-10-05T19:28:27.496 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.490+0000 7f2117119e40 20 rados->read ofs=0 len=0 2023-10-05T19:28:27.496 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.490+0000 7f2117119e40 1 -- 172.21.15.167:0/1497151981 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:42 5.12 5:49953fa1:::default.realm:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x556bf4a3a5a0 con 0x556bf4a02930 2023-10-05T19:28:27.497 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.490+0000 7f211544f700 1 -- 172.21.15.167:0/1497151981 <== osd.1 v2:172.21.15.167:6800/30035 16 ==== osd_op_reply(42 default.realm [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 ==== 157+0+0 (crc 0 0 0) 0x7f210402f080 con 0x556bf4a02930 2023-10-05T19:28:27.497 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.490+0000 7f2117119e40 20 rados_obj.operate() r=-2 bl.length=0 2023-10-05T19:28:27.497 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.490+0000 7f2117119e40 20 rados->read ofs=0 len=0 2023-10-05T19:28:27.498 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.490+0000 7f2117119e40 1 -- 172.21.15.167:0/1497151981 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:43 5.f 5:f4c53578:::zonegroups_names.default:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x556bf4a3a5a0 con 0x556bf4a02930 2023-10-05T19:28:27.498 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.490+0000 7f211544f700 1 -- 172.21.15.167:0/1497151981 <== osd.1 v2:172.21.15.167:6800/30035 17 ==== osd_op_reply(43 zonegroups_names.default [read 0~46 out=46b] v0'0 uv2 ondisk = 0) v8 ==== 168+0+46 (crc 0 0 0) 0x7f210402f080 con 0x556bf4a02930 2023-10-05T19:28:27.498 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.490+0000 7f2117119e40 20 rados_obj.operate() r=0 bl.length=46 2023-10-05T19:28:27.498 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.490+0000 7f2117119e40 1 -- 172.21.15.167:0/1497151981 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:44 5.0 5:00000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x556bf4a3b5b0 con 0x556bf4a36fe0 2023-10-05T19:28:27.499 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.494+0000 7f2114c4e700 1 -- 172.21.15.167:0/1497151981 <== osd.3 v2:172.21.15.167:6808/30036 11 ==== osd_op_reply(44 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f210c0397c0 con 0x556bf4a36fe0 2023-10-05T19:28:27.499 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.494+0000 7f2114c4e700 1 -- 172.21.15.167:0/1497151981 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:45 5.10 5:08000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x556bf4a3b5b0 con 0x7f210c03a020 2023-10-05T19:28:27.501 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.494+0000 7f211444d700 1 -- 172.21.15.167:0/1497151981 <== osd.2 v2:172.21.15.167:6816/30037 7 ==== osd_op_reply(45 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f20fc006070 con 0x7f210c03a020 2023-10-05T19:28:27.501 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.494+0000 7f211444d700 1 -- 172.21.15.167:0/1497151981 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:46 5.8 5:10000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x556bf4a3b5b0 con 0x7f20fc00ebe0 2023-10-05T19:28:27.501 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.494+0000 7f211544f700 1 -- 172.21.15.167:0/1497151981 <== osd.0 v2:172.21.15.118:6800/27690 11 ==== osd_op_reply(46 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f210402f080 con 0x7f20fc00ebe0 2023-10-05T19:28:27.502 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.494+0000 7f211544f700 1 -- 172.21.15.167:0/1497151981 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:47 5.18 5:18000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x556bf4a35980 con 0x556bf4a02930 2023-10-05T19:28:27.502 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.498+0000 7f211544f700 1 -- 172.21.15.167:0/1497151981 <== osd.1 v2:172.21.15.167:6800/30035 18 ==== osd_op_reply(47 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f210402f080 con 0x556bf4a02930 2023-10-05T19:28:27.502 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.498+0000 7f211544f700 1 -- 172.21.15.167:0/1497151981 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:48 5.4 5:20000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x556bf4a35980 con 0x556bf4a02930 2023-10-05T19:28:27.503 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.498+0000 7f211544f700 1 -- 172.21.15.167:0/1497151981 <== osd.1 v2:172.21.15.167:6800/30035 19 ==== osd_op_reply(48 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f210402f080 con 0x556bf4a02930 2023-10-05T19:28:27.503 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.498+0000 7f211544f700 1 -- 172.21.15.167:0/1497151981 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:49 5.14 5:28000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x556bf4a35980 con 0x7f20fc00ebe0 2023-10-05T19:28:27.504 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.498+0000 7f211544f700 1 -- 172.21.15.167:0/1497151981 <== osd.0 v2:172.21.15.118:6800/27690 12 ==== osd_op_reply(49 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f210402f080 con 0x7f20fc00ebe0 2023-10-05T19:28:27.504 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.498+0000 7f211544f700 1 -- 172.21.15.167:0/1497151981 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:50 5.c 5:30000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x556bf4a35980 con 0x7f20fc00ebe0 2023-10-05T19:28:27.505 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.498+0000 7f211544f700 1 -- 172.21.15.167:0/1497151981 <== osd.0 v2:172.21.15.118:6800/27690 13 ==== osd_op_reply(50 [pgnls start_epoch 30 out=112b] v23'1 uv1 ondisk = 1) v8 ==== 144+0+112 (crc 0 0 0) 0x7f210402f080 con 0x7f20fc00ebe0 2023-10-05T19:28:27.505 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.498+0000 7f211544f700 1 -- 172.21.15.167:0/1497151981 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:51 5.1c 5:38000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x556bf4a35980 con 0x7f210c03a020 2023-10-05T19:28:27.506 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.502+0000 7f211444d700 1 -- 172.21.15.167:0/1497151981 <== osd.2 v2:172.21.15.167:6816/30037 8 ==== osd_op_reply(51 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f20fc006070 con 0x7f210c03a020 2023-10-05T19:28:27.506 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.502+0000 7f211444d700 1 -- 172.21.15.167:0/1497151981 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:52 5.2 5:40000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x556bf4a35980 con 0x556bf4a36fe0 2023-10-05T19:28:27.507 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.502+0000 7f2114c4e700 1 -- 172.21.15.167:0/1497151981 <== osd.3 v2:172.21.15.167:6808/30036 12 ==== osd_op_reply(52 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f210c0397c0 con 0x556bf4a36fe0 2023-10-05T19:28:27.507 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.502+0000 7f2114c4e700 1 -- 172.21.15.167:0/1497151981 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:53 5.12 5:48000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x556bf4a35980 con 0x556bf4a02930 2023-10-05T19:28:27.508 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.502+0000 7f211544f700 1 -- 172.21.15.167:0/1497151981 <== osd.1 v2:172.21.15.167:6800/30035 20 ==== osd_op_reply(53 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f210402f080 con 0x556bf4a02930 2023-10-05T19:28:27.508 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.502+0000 7f211544f700 1 -- 172.21.15.167:0/1497151981 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:54 5.a 5:50000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x556bf4a35980 con 0x7f20fc00ebe0 2023-10-05T19:28:27.509 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.502+0000 7f211544f700 1 -- 172.21.15.167:0/1497151981 <== osd.0 v2:172.21.15.118:6800/27690 14 ==== osd_op_reply(54 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f210402f080 con 0x7f20fc00ebe0 2023-10-05T19:28:27.509 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.502+0000 7f211544f700 1 -- 172.21.15.167:0/1497151981 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:55 5.1a 5:58000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x556bf4a35980 con 0x556bf4a02930 2023-10-05T19:28:27.510 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.506+0000 7f211544f700 1 -- 172.21.15.167:0/1497151981 <== osd.1 v2:172.21.15.167:6800/30035 21 ==== osd_op_reply(55 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f210402f080 con 0x556bf4a02930 2023-10-05T19:28:27.510 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.506+0000 7f211544f700 1 -- 172.21.15.167:0/1497151981 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:56 5.6 5:60000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x556bf4a35980 con 0x7f210c03a020 2023-10-05T19:28:27.511 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.506+0000 7f211444d700 1 -- 172.21.15.167:0/1497151981 <== osd.2 v2:172.21.15.167:6816/30037 9 ==== osd_op_reply(56 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f20fc006070 con 0x7f210c03a020 2023-10-05T19:28:27.511 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.506+0000 7f211444d700 1 -- 172.21.15.167:0/1497151981 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:57 5.16 5:68000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x556bf4a35980 con 0x556bf4a36fe0 2023-10-05T19:28:27.511 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.506+0000 7f2114c4e700 1 -- 172.21.15.167:0/1497151981 <== osd.3 v2:172.21.15.167:6808/30036 13 ==== osd_op_reply(57 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f210c0397c0 con 0x556bf4a36fe0 2023-10-05T19:28:27.511 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.506+0000 7f2114c4e700 1 -- 172.21.15.167:0/1497151981 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:58 5.e 5:70000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x556bf4a35980 con 0x556bf4a36fe0 2023-10-05T19:28:27.512 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.506+0000 7f2114c4e700 1 -- 172.21.15.167:0/1497151981 <== osd.3 v2:172.21.15.167:6808/30036 14 ==== osd_op_reply(58 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f210c0397c0 con 0x556bf4a36fe0 2023-10-05T19:28:27.512 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.506+0000 7f2114c4e700 1 -- 172.21.15.167:0/1497151981 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:59 5.1e 5:78000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x556bf4a35980 con 0x7f20fc00ebe0 2023-10-05T19:28:27.513 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.506+0000 7f211544f700 1 -- 172.21.15.167:0/1497151981 <== osd.0 v2:172.21.15.118:6800/27690 15 ==== osd_op_reply(59 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f210402f080 con 0x7f20fc00ebe0 2023-10-05T19:28:27.513 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.506+0000 7f211544f700 1 -- 172.21.15.167:0/1497151981 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:60 5.1 5:80000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x556bf4a35980 con 0x556bf4a36fe0 2023-10-05T19:28:27.514 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.510+0000 7f2114c4e700 1 -- 172.21.15.167:0/1497151981 <== osd.3 v2:172.21.15.167:6808/30036 15 ==== osd_op_reply(60 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f210c0397c0 con 0x556bf4a36fe0 2023-10-05T19:28:27.514 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.510+0000 7f2114c4e700 1 -- 172.21.15.167:0/1497151981 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:61 5.11 5:88000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x556bf4a35980 con 0x556bf4a36fe0 2023-10-05T19:28:27.516 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.510+0000 7f2114c4e700 1 -- 172.21.15.167:0/1497151981 <== osd.3 v2:172.21.15.167:6808/30036 16 ==== osd_op_reply(61 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f210c0397c0 con 0x556bf4a36fe0 2023-10-05T19:28:27.516 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.510+0000 7f2114c4e700 1 -- 172.21.15.167:0/1497151981 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:62 5.9 5:90000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x556bf4a35980 con 0x556bf4a02930 2023-10-05T19:28:27.516 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.510+0000 7f211544f700 1 -- 172.21.15.167:0/1497151981 <== osd.1 v2:172.21.15.167:6800/30035 22 ==== osd_op_reply(62 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f210402f080 con 0x556bf4a02930 2023-10-05T19:28:27.516 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.510+0000 7f211544f700 1 -- 172.21.15.167:0/1497151981 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:63 5.19 5:98000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x556bf4a35980 con 0x556bf4a02930 2023-10-05T19:28:27.517 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.510+0000 7f211544f700 1 -- 172.21.15.167:0/1497151981 <== osd.1 v2:172.21.15.167:6800/30035 23 ==== osd_op_reply(63 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f210402f080 con 0x556bf4a02930 2023-10-05T19:28:27.517 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.510+0000 7f211544f700 1 -- 172.21.15.167:0/1497151981 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:64 5.5 5:a0000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x556bf4a35980 con 0x7f20fc00ebe0 2023-10-05T19:28:27.517 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.510+0000 7f211544f700 1 -- 172.21.15.167:0/1497151981 <== osd.0 v2:172.21.15.118:6800/27690 16 ==== osd_op_reply(64 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f210402f080 con 0x7f20fc00ebe0 2023-10-05T19:28:27.517 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.510+0000 7f211544f700 1 -- 172.21.15.167:0/1497151981 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:65 5.15 5:a8000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x556bf4a35980 con 0x556bf4a36fe0 2023-10-05T19:28:27.518 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.514+0000 7f2114c4e700 1 -- 172.21.15.167:0/1497151981 <== osd.3 v2:172.21.15.167:6808/30036 17 ==== osd_op_reply(65 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f210c0397c0 con 0x556bf4a36fe0 2023-10-05T19:28:27.518 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.514+0000 7f2114c4e700 1 -- 172.21.15.167:0/1497151981 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:66 5.d 5:b0000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x556bf4a35980 con 0x7f210c03a020 2023-10-05T19:28:27.519 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.514+0000 7f211444d700 1 -- 172.21.15.167:0/1497151981 <== osd.2 v2:172.21.15.167:6816/30037 10 ==== osd_op_reply(66 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f20fc006070 con 0x7f210c03a020 2023-10-05T19:28:27.519 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.514+0000 7f211444d700 1 -- 172.21.15.167:0/1497151981 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:67 5.1d 5:b8000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x556bf4a35980 con 0x556bf4a02930 2023-10-05T19:28:27.520 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.514+0000 7f211544f700 1 -- 172.21.15.167:0/1497151981 <== osd.1 v2:172.21.15.167:6800/30035 24 ==== osd_op_reply(67 [pgnls start_epoch 30 out=107b] v30'2 uv2 ondisk = 1) v8 ==== 144+0+107 (crc 0 0 0) 0x7f210402f080 con 0x556bf4a02930 2023-10-05T19:28:27.520 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.514+0000 7f211544f700 1 -- 172.21.15.167:0/1497151981 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:68 5.3 5:c0000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x556bf4a35980 con 0x7f20fc00ebe0 2023-10-05T19:28:27.521 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.514+0000 7f211544f700 1 -- 172.21.15.167:0/1497151981 <== osd.0 v2:172.21.15.118:6800/27690 17 ==== osd_op_reply(68 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f210402f080 con 0x7f20fc00ebe0 2023-10-05T19:28:27.521 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.514+0000 7f211544f700 1 -- 172.21.15.167:0/1497151981 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:69 5.13 5:c8000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x556bf4a35980 con 0x556bf4a36fe0 2023-10-05T19:28:27.522 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.518+0000 7f2114c4e700 1 -- 172.21.15.167:0/1497151981 <== osd.3 v2:172.21.15.167:6808/30036 18 ==== osd_op_reply(69 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f210c0397c0 con 0x556bf4a36fe0 2023-10-05T19:28:27.522 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.518+0000 7f2114c4e700 1 -- 172.21.15.167:0/1497151981 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:70 5.b 5:d0000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x556bf4a35980 con 0x7f20fc00ebe0 2023-10-05T19:28:27.523 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.518+0000 7f211544f700 1 -- 172.21.15.167:0/1497151981 <== osd.0 v2:172.21.15.118:6800/27690 18 ==== osd_op_reply(70 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f210402f080 con 0x7f20fc00ebe0 2023-10-05T19:28:27.523 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.518+0000 7f211544f700 1 -- 172.21.15.167:0/1497151981 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:71 5.1b 5:d8000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x556bf4a35980 con 0x7f210c03a020 2023-10-05T19:28:27.524 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.518+0000 7f211444d700 1 -- 172.21.15.167:0/1497151981 <== osd.2 v2:172.21.15.167:6816/30037 11 ==== osd_op_reply(71 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f20fc006070 con 0x7f210c03a020 2023-10-05T19:28:27.524 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.518+0000 7f211444d700 1 -- 172.21.15.167:0/1497151981 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:72 5.7 5:e0000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x556bf4a35980 con 0x556bf4a36fe0 2023-10-05T19:28:27.525 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.518+0000 7f2114c4e700 1 -- 172.21.15.167:0/1497151981 <== osd.3 v2:172.21.15.167:6808/30036 19 ==== osd_op_reply(72 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f210c0397c0 con 0x556bf4a36fe0 2023-10-05T19:28:27.525 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.518+0000 7f2114c4e700 1 -- 172.21.15.167:0/1497151981 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:73 5.17 5:e8000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x556bf4a35980 con 0x7f20fc00ebe0 2023-10-05T19:28:27.526 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.522+0000 7f211544f700 1 -- 172.21.15.167:0/1497151981 <== osd.0 v2:172.21.15.118:6800/27690 19 ==== osd_op_reply(73 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f210402f080 con 0x7f20fc00ebe0 2023-10-05T19:28:27.526 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.522+0000 7f211544f700 1 -- 172.21.15.167:0/1497151981 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:74 5.f 5:f0000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x556bf4a35980 con 0x556bf4a02930 2023-10-05T19:28:27.527 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.522+0000 7f211544f700 1 -- 172.21.15.167:0/1497151981 <== osd.1 v2:172.21.15.167:6800/30035 25 ==== osd_op_reply(74 [pgnls start_epoch 30 out=115b] v23'2 uv2 ondisk = 1) v8 ==== 144+0+115 (crc 0 0 0) 0x7f210402f080 con 0x556bf4a02930 2023-10-05T19:28:27.527 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.522+0000 7f211544f700 1 -- 172.21.15.167:0/1497151981 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:75 5.1f 5:f8000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x556bf4a35980 con 0x7f210c03a020 2023-10-05T19:28:27.528 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.522+0000 7f211444d700 1 -- 172.21.15.167:0/1497151981 <== osd.2 v2:172.21.15.167:6816/30037 12 ==== osd_op_reply(75 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f20fc006070 con 0x7f210c03a020 2023-10-05T19:28:27.528 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.522+0000 7f2117119e40 20 RGWRados::pool_iterate: got zonegroup_info.4f1f719d-9648-417b-b10c-e5d939825419 2023-10-05T19:28:27.528 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.522+0000 7f2117119e40 20 RGWRados::pool_iterate: got zone_info.9a9df965-6612-45d4-a45f-6e3c889bacc2 2023-10-05T19:28:27.528 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.522+0000 7f2117119e40 20 RGWRados::pool_iterate: got zone_names.default 2023-10-05T19:28:27.528 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.522+0000 7f2117119e40 20 RGWRados::pool_iterate: got zonegroups_names.default 2023-10-05T19:28:27.529 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.522+0000 7f2117119e40 20 rados->read ofs=0 len=0 2023-10-05T19:28:27.529 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.522+0000 7f2117119e40 1 -- 172.21.15.167:0/1497151981 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:76 5.f 5:f43b6ece:::zone_names.default:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x556bf4a3a5a0 con 0x556bf4a02930 2023-10-05T19:28:27.529 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.522+0000 7f211544f700 1 -- 172.21.15.167:0/1497151981 <== osd.1 v2:172.21.15.167:6800/30035 26 ==== osd_op_reply(76 zone_names.default [read 0~46 out=46b] v0'0 uv1 ondisk = 0) v8 ==== 162+0+46 (crc 0 0 0) 0x7f210402f080 con 0x556bf4a02930 2023-10-05T19:28:27.529 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.522+0000 7f2117119e40 20 rados_obj.operate() r=0 bl.length=46 2023-10-05T19:28:27.530 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.522+0000 7f2117119e40 20 rados->read ofs=0 len=0 2023-10-05T19:28:27.530 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.522+0000 7f2117119e40 1 -- 172.21.15.167:0/1497151981 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:77 5.1d 5:b8374eb4:::zone_info.9a9df965-6612-45d4-a45f-6e3c889bacc2:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x556bf4a3a950 con 0x556bf4a02930 2023-10-05T19:28:27.530 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.526+0000 7f211544f700 1 -- 172.21.15.167:0/1497151981 <== osd.1 v2:172.21.15.167:6800/30035 27 ==== osd_op_reply(77 zone_info.9a9df965-6612-45d4-a45f-6e3c889bacc2 [read 0~874 out=874b] v0'0 uv2 ondisk = 0) v8 ==== 190+0+874 (crc 0 0 0) 0x7f210402f080 con 0x556bf4a02930 2023-10-05T19:28:27.530 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.526+0000 7f2117119e40 20 rados_obj.operate() r=0 bl.length=874 2023-10-05T19:28:27.530 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.526+0000 7f2117119e40 20 rados->read ofs=0 len=0 2023-10-05T19:28:27.531 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.526+0000 7f2117119e40 1 -- 172.21.15.167:0/1497151981 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:78 5.f 5:f4c53578:::zonegroups_names.default:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x556bf4a3ae40 con 0x556bf4a02930 2023-10-05T19:28:27.531 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.526+0000 7f211544f700 1 -- 172.21.15.167:0/1497151981 <== osd.1 v2:172.21.15.167:6800/30035 28 ==== osd_op_reply(78 zonegroups_names.default [read 0~46 out=46b] v0'0 uv2 ondisk = 0) v8 ==== 168+0+46 (crc 0 0 0) 0x7f210402f080 con 0x556bf4a02930 2023-10-05T19:28:27.531 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.526+0000 7f2117119e40 20 rados_obj.operate() r=0 bl.length=46 2023-10-05T19:28:27.531 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.526+0000 7f2117119e40 20 rados->read ofs=0 len=0 2023-10-05T19:28:27.531 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.526+0000 7f2117119e40 1 -- 172.21.15.167:0/1497151981 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:79 5.c 5:30d299a5:::zonegroup_info.4f1f719d-9648-417b-b10c-e5d939825419:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x556bf4a3ae40 con 0x7f20fc00ebe0 2023-10-05T19:28:27.532 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.526+0000 7f211544f700 1 -- 172.21.15.167:0/1497151981 <== osd.0 v2:172.21.15.118:6800/27690 20 ==== osd_op_reply(79 zonegroup_info.4f1f719d-9648-417b-b10c-e5d939825419 [read 0~362 out=362b] v0'0 uv1 ondisk = 0) v8 ==== 195+0+362 (crc 0 0 0) 0x7f210402f080 con 0x7f20fc00ebe0 2023-10-05T19:28:27.532 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.526+0000 7f2117119e40 20 rados_obj.operate() r=0 bl.length=362 2023-10-05T19:28:27.532 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.526+0000 7f2117119e40 20 rados->read ofs=0 len=0 2023-10-05T19:28:27.532 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.526+0000 7f2117119e40 1 -- 172.21.15.167:0/1497151981 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:80 5.3 5:c0ad8cbd:::region_map:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x556bf4a3af40 con 0x7f20fc00ebe0 2023-10-05T19:28:27.533 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.526+0000 7f211544f700 1 -- 172.21.15.167:0/1497151981 <== osd.0 v2:172.21.15.118:6800/27690 21 ==== osd_op_reply(80 region_map [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 ==== 154+0+0 (crc 0 0 0) 0x7f2104050040 con 0x7f20fc00ebe0 2023-10-05T19:28:27.533 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.526+0000 7f2117119e40 20 rados_obj.operate() r=-2 bl.length=0 2023-10-05T19:28:27.533 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.526+0000 7f2117119e40 20 zone default found 2023-10-05T19:28:27.533 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.526+0000 7f2117119e40 4 Realm: () 2023-10-05T19:28:27.533 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.526+0000 7f2117119e40 4 ZoneGroup: default (4f1f719d-9648-417b-b10c-e5d939825419) 2023-10-05T19:28:27.534 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.526+0000 7f2117119e40 4 Zone: default (9a9df965-6612-45d4-a45f-6e3c889bacc2) 2023-10-05T19:28:27.534 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.526+0000 7f2117119e40 10 cannot find current period zonegroup using local zonegroup configuration 2023-10-05T19:28:27.534 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.526+0000 7f2117119e40 20 zonegroup default 2023-10-05T19:28:27.534 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.526+0000 7f2117119e40 20 rados->read ofs=0 len=0 2023-10-05T19:28:27.534 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.526+0000 7f2117119e40 1 -- 172.21.15.167:0/1497151981 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:81 5.3 5:c52100b6:::period_config.default:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x556bf4a3af40 con 0x7f20fc00ebe0 2023-10-05T19:28:27.538 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.526+0000 7f211544f700 1 -- 172.21.15.167:0/1497151981 <== osd.0 v2:172.21.15.118:6800/27690 22 ==== osd_op_reply(81 period_config.default [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 ==== 165+0+0 (crc 0 0 0) 0x7f2104050040 con 0x7f20fc00ebe0 2023-10-05T19:28:27.538 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.526+0000 7f2117119e40 20 rados_obj.operate() r=-2 bl.length=0 2023-10-05T19:28:27.538 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.526+0000 7f2117119e40 20 rados->read ofs=0 len=0 2023-10-05T19:28:27.538 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.526+0000 7f2117119e40 1 -- 172.21.15.167:0/1497151981 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:82 6.2 6:4347d321:::bucket.sync-source-hints.:head [call version.read in=11b,read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x556bf4a3c8c0 con 0x556bf4a36fe0 2023-10-05T19:28:27.538 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.530+0000 7f2114c4e700 1 -- 172.21.15.167:0/1497151981 <== osd.3 v2:172.21.15.167:6808/30036 20 ==== osd_op_reply(82 bucket.sync-source-hints. [call,read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 ==== 211+0+0 (crc 0 0 0) 0x7f210c0397c0 con 0x556bf4a36fe0 2023-10-05T19:28:27.538 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.530+0000 7f2117119e40 20 rados_obj.operate() r=-2 bl.length=0 2023-10-05T19:28:27.539 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.530+0000 7f2117119e40 20 rados->read ofs=0 len=0 2023-10-05T19:28:27.539 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.530+0000 7f2117119e40 1 -- 172.21.15.167:0/1497151981 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:83 6.b 6:d467b91b:::bucket.sync-target-hints.:head [call version.read in=11b,read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x556bf4a3cc70 con 0x556bf4a36fe0 2023-10-05T19:28:27.539 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.530+0000 7f2114c4e700 1 -- 172.21.15.167:0/1497151981 <== osd.3 v2:172.21.15.167:6808/30036 21 ==== osd_op_reply(83 bucket.sync-target-hints. [call,read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 ==== 211+0+0 (crc 0 0 0) 0x7f210c0397c0 con 0x556bf4a36fe0 2023-10-05T19:28:27.539 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.530+0000 7f2117119e40 20 rados_obj.operate() r=-2 bl.length=0 2023-10-05T19:28:27.539 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.530+0000 7f2117119e40 20 started sync module instance, tier type = 2023-10-05T19:28:27.539 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.530+0000 7f2117119e40 20 started zone id=9a9df965-6612-45d4-a45f-6e3c889bacc2 (name=default) with tier type = 2023-10-05T19:28:27.539 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.530+0000 7f2117119e40 1 -- 172.21.15.167:0/1497151981 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:84 7.1f 7:f95f44c2:::notify.0:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x556bf4a3bf80 con 0x7f210c03a020 2023-10-05T19:28:27.540 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.534+0000 7f211444d700 1 -- 172.21.15.167:0/1497151981 <== osd.2 v2:172.21.15.167:6816/30037 13 ==== osd_op_reply(84 notify.0 [create] v30'48 uv48 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f20fc006070 con 0x7f210c03a020 2023-10-05T19:28:27.540 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.534+0000 7f2117119e40 1 -- 172.21.15.167:0/1497151981 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:85 7.1f 7:f95f44c2:::notify.0:head [watch watch cookie 93922154234752] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x556bf4a3bbb0 con 0x7f210c03a020 2023-10-05T19:28:27.540 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.534+0000 7f2117119e40 1 -- 172.21.15.167:0/1497151981 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:86 7.0 7:05bf5b68:::notify.1:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x556bf4a3da00 con 0x556bf4a02930 2023-10-05T19:28:27.541 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.534+0000 7f211444d700 1 -- 172.21.15.167:0/1497151981 <== osd.2 v2:172.21.15.167:6816/30037 14 ==== osd_op_reply(85 notify.0 [watch watch cookie 93922154234752] v30'49 uv48 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f20fc006070 con 0x7f210c03a020 2023-10-05T19:28:27.541 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.534+0000 7f211544f700 1 -- 172.21.15.167:0/1497151981 <== osd.1 v2:172.21.15.167:6800/30035 29 ==== osd_op_reply(86 notify.1 [create] v30'48 uv48 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f2104050040 con 0x556bf4a02930 2023-10-05T19:28:27.542 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.534+0000 7f2117119e40 1 -- 172.21.15.167:0/1497151981 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:87 7.0 7:05bf5b68:::notify.1:head [watch watch cookie 93922154238272] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x556bf4a3de00 con 0x556bf4a02930 2023-10-05T19:28:27.542 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.534+0000 7f2117119e40 1 -- 172.21.15.167:0/1497151981 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:88 7.15 7:a93a5511:::notify.2:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x556bf4a3e6a0 con 0x556bf4a36fe0 2023-10-05T19:28:27.545 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.538+0000 7f211544f700 1 -- 172.21.15.167:0/1497151981 <== osd.1 v2:172.21.15.167:6800/30035 30 ==== osd_op_reply(87 notify.1 [watch watch cookie 93922154238272] v30'49 uv48 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f2104050040 con 0x556bf4a02930 2023-10-05T19:28:27.545 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.538+0000 7f2114c4e700 1 -- 172.21.15.167:0/1497151981 <== osd.3 v2:172.21.15.167:6808/30036 22 ==== osd_op_reply(88 notify.2 [create] v30'48 uv48 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f210c0397c0 con 0x556bf4a36fe0 2023-10-05T19:28:27.545 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.538+0000 7f2117119e40 1 -- 172.21.15.167:0/1497151981 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:89 7.15 7:a93a5511:::notify.2:head [watch watch cookie 93922154241536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x556bf4a3d6f0 con 0x556bf4a36fe0 2023-10-05T19:28:27.546 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.538+0000 7f2117119e40 1 -- 172.21.15.167:0/1497151981 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:90 7.e 7:7759931f:::notify.3:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x556bf4a3f430 con 0x7f210c03a020 2023-10-05T19:28:27.547 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.542+0000 7f2114c4e700 1 -- 172.21.15.167:0/1497151981 <== osd.3 v2:172.21.15.167:6808/30036 23 ==== osd_op_reply(89 notify.2 [watch watch cookie 93922154241536] v30'49 uv48 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f210c0397c0 con 0x556bf4a36fe0 2023-10-05T19:28:27.548 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.542+0000 7f211444d700 1 -- 172.21.15.167:0/1497151981 <== osd.2 v2:172.21.15.167:6816/30037 15 ==== osd_op_reply(90 notify.3 [create] v30'48 uv48 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f20fc006070 con 0x7f210c03a020 2023-10-05T19:28:27.548 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.542+0000 7f2117119e40 1 -- 172.21.15.167:0/1497151981 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:91 7.e 7:7759931f:::notify.3:head [watch watch cookie 93922154244688] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x556bf4a3f7e0 con 0x7f210c03a020 2023-10-05T19:28:27.548 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.542+0000 7f2117119e40 1 -- 172.21.15.167:0/1497151981 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:92 7.d 7:b4812045:::notify.4:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x556bf4a40320 con 0x556bf4a02930 2023-10-05T19:28:27.550 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.546+0000 7f211444d700 1 -- 172.21.15.167:0/1497151981 <== osd.2 v2:172.21.15.167:6816/30037 16 ==== osd_op_reply(91 notify.3 [watch watch cookie 93922154244688] v30'49 uv48 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f20fc006070 con 0x7f210c03a020 2023-10-05T19:28:27.551 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.546+0000 7f211544f700 1 -- 172.21.15.167:0/1497151981 <== osd.1 v2:172.21.15.167:6800/30035 31 ==== osd_op_reply(92 notify.4 [create] v30'48 uv48 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f2104050040 con 0x556bf4a02930 2023-10-05T19:28:27.551 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.546+0000 7f2117119e40 1 -- 172.21.15.167:0/1497151981 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:93 7.d 7:b4812045:::notify.4:head [watch watch cookie 93922154245792] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x556bf4a40720 con 0x556bf4a02930 2023-10-05T19:28:27.551 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.546+0000 7f2117119e40 1 -- 172.21.15.167:0/1497151981 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:94 7.3 7:c609908c:::notify.5:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x556bf4a41130 con 0x7f20fc00ebe0 2023-10-05T19:28:27.553 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.546+0000 7f211544f700 1 -- 172.21.15.167:0/1497151981 <== osd.1 v2:172.21.15.167:6800/30035 32 ==== osd_op_reply(93 notify.4 [watch watch cookie 93922154245792] v30'49 uv48 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f2104050040 con 0x556bf4a02930 2023-10-05T19:28:27.553 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.546+0000 7f211544f700 1 -- 172.21.15.167:0/1497151981 <== osd.0 v2:172.21.15.118:6800/27690 23 ==== osd_op_reply(94 notify.5 [create] v30'48 uv48 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f2104050040 con 0x7f20fc00ebe0 2023-10-05T19:28:27.554 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.546+0000 7f2117119e40 1 -- 172.21.15.167:0/1497151981 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:95 7.3 7:c609908c:::notify.5:head [watch watch cookie 93922154249712] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x556bf4a41130 con 0x7f20fc00ebe0 2023-10-05T19:28:27.554 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.546+0000 7f2117119e40 1 -- 172.21.15.167:0/1497151981 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:96 7.14 7:2b04a3e9:::notify.6:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x556bf4a41b40 con 0x7f20fc00ebe0 2023-10-05T19:28:27.556 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.550+0000 7f211544f700 1 -- 172.21.15.167:0/1497151981 <== osd.0 v2:172.21.15.118:6800/27690 24 ==== osd_op_reply(95 notify.5 [watch watch cookie 93922154249712] v30'49 uv48 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f2104050040 con 0x7f20fc00ebe0 2023-10-05T19:28:27.556 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.550+0000 7f211544f700 1 -- 172.21.15.167:0/1497151981 <== osd.0 v2:172.21.15.118:6800/27690 25 ==== osd_op_reply(96 notify.6 [create] v30'48 uv48 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f2104050040 con 0x7f20fc00ebe0 2023-10-05T19:28:27.556 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.550+0000 7f2117119e40 1 -- 172.21.15.167:0/1497151981 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:97 7.14 7:2b04a3e9:::notify.6:head [watch watch cookie 93922154253936] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x556bf4a41b40 con 0x7f20fc00ebe0 2023-10-05T19:28:27.556 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.550+0000 7f2117119e40 1 -- 172.21.15.167:0/1497151981 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:98 7.9 7:93e5b521:::notify.7:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x556bf4a42550 con 0x7f20fc00ebe0 2023-10-05T19:28:27.559 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.554+0000 7f211544f700 1 -- 172.21.15.167:0/1497151981 <== osd.0 v2:172.21.15.118:6800/27690 26 ==== osd_op_reply(98 notify.7 [create] v30'48 uv48 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f2104050040 con 0x7f20fc00ebe0 2023-10-05T19:28:27.559 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.554+0000 7f2117119e40 1 -- 172.21.15.167:0/1497151981 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:99 7.9 7:93e5b521:::notify.7:head [watch watch cookie 93922154258528] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x556bf4a429a0 con 0x7f20fc00ebe0 2023-10-05T19:28:27.559 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.554+0000 7f2117119e40 20 add_watcher() i=0 2023-10-05T19:28:27.559 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.554+0000 7f2117119e40 20 add_watcher() i=1 2023-10-05T19:28:27.559 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.554+0000 7f2117119e40 20 add_watcher() i=2 2023-10-05T19:28:27.559 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.554+0000 7f2117119e40 20 add_watcher() i=3 2023-10-05T19:28:27.559 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.554+0000 7f2117119e40 20 add_watcher() i=4 2023-10-05T19:28:27.559 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.554+0000 7f2117119e40 20 add_watcher() i=5 2023-10-05T19:28:27.559 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.554+0000 7f211544f700 1 -- 172.21.15.167:0/1497151981 <== osd.0 v2:172.21.15.118:6800/27690 27 ==== osd_op_reply(97 notify.6 [watch watch cookie 93922154253936] v30'49 uv48 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f2104050040 con 0x7f20fc00ebe0 2023-10-05T19:28:27.560 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.554+0000 7f2117119e40 20 add_watcher() i=6 2023-10-05T19:28:27.561 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.554+0000 7f211544f700 1 -- 172.21.15.167:0/1497151981 <== osd.0 v2:172.21.15.118:6800/27690 28 ==== osd_op_reply(99 notify.7 [watch watch cookie 93922154258528] v30'49 uv48 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f2104050040 con 0x7f20fc00ebe0 2023-10-05T19:28:27.561 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.554+0000 7f2117119e40 20 add_watcher() i=7 2023-10-05T19:28:27.562 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.554+0000 7f2117119e40 2 all 8 watchers are set, enabling cache 2023-10-05T19:28:27.562 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.554+0000 7f2117119e40 1 -- 172.21.15.167:0/1497151981 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:100 6.4 6:22d26bf9:::data_loggenerations_metadata:head [call version.check_conds in=74b,call version.read in=11b,read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x556bf4a429a0 con 0x556bf4a02930 2023-10-05T19:28:27.562 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.558+0000 7f211544f700 1 -- 172.21.15.167:0/1497151981 <== osd.1 v2:172.21.15.167:6800/30035 33 ==== osd_op_reply(100 data_loggenerations_metadata [call,call out=48b,read 0~28 out=28b] v0'0 uv1 ondisk = 0) v8 ==== 256+0+76 (crc 0 0 0) 0x7f2104050040 con 0x556bf4a02930 2023-10-05T19:28:27.562 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.558+0000 7f2117119e40 1 -- 172.21.15.167:0/1497151981 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:101 6.4 6:22d26bf9:::data_loggenerations_metadata:head [watch watch cookie 93922154259632] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x556bf4a42900 con 0x556bf4a02930 2023-10-05T19:28:27.564 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.558+0000 7f211544f700 1 -- 172.21.15.167:0/1497151981 <== osd.1 v2:172.21.15.167:6800/30035 34 ==== osd_op_reply(101 data_loggenerations_metadata [watch watch cookie 93922154259632] v30'66 uv1 ondisk = 0) v8 ==== 172+0+0 (crc 0 0 0) 0x7f2104050040 con 0x556bf4a02930 2023-10-05T19:28:27.564 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.558+0000 7f20e57f2700 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2023-10-05T19:28:27.564 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.558+0000 7f2117119e40 20 check_secure_mon_conn(): auth registy supported: methods=[2] modes=[2,1] 2023-10-05T19:28:27.565 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.558+0000 7f2117119e40 20 check_secure_mon_conn(): mode 1 is insecure 2023-10-05T19:28:27.566 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.558+0000 7f2117119e40 1 Processor -- start 2023-10-05T19:28:27.566 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.558+0000 7f2117119e40 1 -- start start 2023-10-05T19:28:27.566 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.558+0000 7f2117119e40 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x556bf4a4fbd0 0x556bf4a4ffa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-05T19:28:27.566 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.558+0000 7f2117119e40 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x556bf4a504e0 con 0x556bf4a4fbd0 2023-10-05T19:28:27.566 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.558+0000 7f211444d700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x556bf4a4fbd0 0x556bf4a4ffa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-05T19:28:27.567 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.558+0000 7f211444d700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x556bf4a4fbd0 0x556bf4a4ffa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.118:3300/0 says I am v2:172.21.15.167:57522/0 (socket says 172.21.15.167:57522) 2023-10-05T19:28:27.567 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.558+0000 7f211444d700 1 -- 172.21.15.167:0/858913483 learned_addr learned my addr 172.21.15.167:0/858913483 (peer_addr_for_me v2:172.21.15.167:0/0) 2023-10-05T19:28:27.568 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.562+0000 7f211444d700 1 -- 172.21.15.167:0/858913483 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x556bf4a50620 con 0x556bf4a4fbd0 2023-10-05T19:28:27.568 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.562+0000 7f211444d700 1 --2- 172.21.15.167:0/858913483 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x556bf4a4fbd0 0x556bf4a4ffa0 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x556bf4a34ad0 tx=0x7f20fc006000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=39a471d8cfcbb227 server_cookie=0 in_seq=0 out_seq=0 2023-10-05T19:28:27.568 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.562+0000 7f20e4ff1700 1 -- 172.21.15.167:0/858913483 <== mon.0 v2:172.21.15.118:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f20fc023040 con 0x556bf4a4fbd0 2023-10-05T19:28:27.568 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.562+0000 7f20e4ff1700 1 -- 172.21.15.167:0/858913483 <== mon.0 v2:172.21.15.118:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f20fc024040 con 0x556bf4a4fbd0 2023-10-05T19:28:27.569 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.562+0000 7f20e4ff1700 1 -- 172.21.15.167:0/858913483 <== mon.0 v2:172.21.15.118:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f20fc013040 con 0x556bf4a4fbd0 2023-10-05T19:28:27.569 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.562+0000 7f2117119e40 1 -- 172.21.15.167:0/858913483 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x556bf4a4fbd0 msgr2=0x556bf4a4ffa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-05T19:28:27.569 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.562+0000 7f2117119e40 1 --2- 172.21.15.167:0/858913483 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x556bf4a4fbd0 0x556bf4a4ffa0 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x556bf4a34ad0 tx=0x7f20fc006000 comp rx=0 tx=0).stop 2023-10-05T19:28:27.569 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.562+0000 7f2117119e40 1 -- 172.21.15.167:0/858913483 shutdown_connections 2023-10-05T19:28:27.569 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.562+0000 7f2117119e40 1 --2- 172.21.15.167:0/858913483 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x556bf4a4fbd0 0x556bf4a4ffa0 unknown :-1 s=CLOSED pgs=234 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-05T19:28:27.569 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.562+0000 7f2117119e40 1 -- 172.21.15.167:0/858913483 >> 172.21.15.167:0/858913483 conn(0x556bf4a4af90 msgr2=0x556bf4a49140 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-05T19:28:27.569 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.562+0000 7f2117119e40 1 -- 172.21.15.167:0/858913483 shutdown_connections 2023-10-05T19:28:27.569 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.562+0000 7f2117119e40 1 -- 172.21.15.167:0/858913483 wait complete. 2023-10-05T19:28:27.571 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.562+0000 7f2117119e40 1 Processor -- start 2023-10-05T19:28:27.572 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.562+0000 7f2117119e40 1 -- start start 2023-10-05T19:28:27.572 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.562+0000 7f2117119e40 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x556bf4a4fbd0 0x556bf4a55720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-05T19:28:27.572 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.562+0000 7f2117119e40 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x556bf4a56720 con 0x556bf4a4fbd0 2023-10-05T19:28:27.572 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.562+0000 7f211444d700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x556bf4a4fbd0 0x556bf4a55720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-05T19:28:27.572 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.562+0000 7f211444d700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x556bf4a4fbd0 0x556bf4a55720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.118:3300/0 says I am v2:172.21.15.167:57534/0 (socket says 172.21.15.167:57534) 2023-10-05T19:28:27.572 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.562+0000 7f211444d700 1 -- 172.21.15.167:0/3229428815 learned_addr learned my addr 172.21.15.167:0/3229428815 (peer_addr_for_me v2:172.21.15.167:0/0) 2023-10-05T19:28:27.572 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.566+0000 7f211444d700 1 -- 172.21.15.167:0/3229428815 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f20f4001e10 con 0x556bf4a4fbd0 2023-10-05T19:28:27.572 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.566+0000 7f211444d700 1 --2- 172.21.15.167:0/3229428815 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x556bf4a4fbd0 0x556bf4a55720 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x556bf4a49cb0 tx=0x7f20fc036000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-05T19:28:27.572 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.566+0000 7f205effd700 1 -- 172.21.15.167:0/3229428815 <== mon.0 v2:172.21.15.118:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f20fc023070 con 0x556bf4a4fbd0 2023-10-05T19:28:27.572 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.566+0000 7f205effd700 1 -- 172.21.15.167:0/3229428815 <== mon.0 v2:172.21.15.118:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f20fc01e630 con 0x556bf4a4fbd0 2023-10-05T19:28:27.572 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.566+0000 7f205effd700 1 -- 172.21.15.167:0/3229428815 <== mon.0 v2:172.21.15.118:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f20fc01e7c0 con 0x556bf4a4fbd0 2023-10-05T19:28:27.573 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.566+0000 7f2117119e40 1 -- 172.21.15.167:0/3229428815 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x556bf4a55d20 con 0x556bf4a4fbd0 2023-10-05T19:28:27.573 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.566+0000 7f2117119e40 1 -- 172.21.15.167:0/3229428815 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x556bf4a56160 con 0x556bf4a4fbd0 2023-10-05T19:28:27.574 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.566+0000 7f205effd700 1 -- 172.21.15.167:0/3229428815 <== mon.0 v2:172.21.15.118:3300/0 4 ==== mgrmap(e 5) v1 ==== 40329+0+0 (secure 0 0 0) 0x7f20fc03f040 con 0x556bf4a4fbd0 2023-10-05T19:28:27.574 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.566+0000 7f205effd700 1 --2- 172.21.15.167:0/3229428815 >> [v2:172.21.15.167:6824/29889,v1:172.21.15.167:6825/29889] conn(0x7f2054030ca0 0x7f2054033150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-05T19:28:27.574 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.566+0000 7f205effd700 1 -- 172.21.15.167:0/3229428815 <== mon.0 v2:172.21.15.118:3300/0 5 ==== osd_map(30..30 src has 1..30) v4 ==== 6057+0+0 (secure 0 0 0) 0x7f20fc042c10 con 0x556bf4a4fbd0 2023-10-05T19:28:27.574 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.566+0000 7f211544f700 1 --2- 172.21.15.167:0/3229428815 >> [v2:172.21.15.167:6824/29889,v1:172.21.15.167:6825/29889] conn(0x7f2054030ca0 0x7f2054033150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-05T19:28:27.574 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.566+0000 7f2117119e40 5 note: GC not initialized 2023-10-05T19:28:27.574 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.570+0000 7f211544f700 1 --2- 172.21.15.167:0/3229428815 >> [v2:172.21.15.167:6824/29889,v1:172.21.15.167:6825/29889] conn(0x7f2054030ca0 0x7f2054033150 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x556bf4a41890 tx=0x7f2104000f90 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-05T19:28:27.576 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.570+0000 7f2051ffb700 20 reqs_thread_entry: start 2023-10-05T19:28:27.576 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.570+0000 7f2117119e40 20 init_complete bucket index max shards: 11 2023-10-05T19:28:27.576 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.570+0000 7f2053fff700 20 reqs_thread_entry: start 2023-10-05T19:28:27.578 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.570+0000 7f2117119e40 10 rgw notify: Started notification manager with: 1 workers 2023-10-05T19:28:27.578 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.570+0000 7f203f7fe700 20 reqs_thread_entry: start 2023-10-05T19:28:27.578 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.570+0000 7f2117119e40 20 get_system_obj_state: rctx=0x7fffc4b76cd8 obj=default.rgw.meta:users.uid:foo state=0x556bf48d90b0 s->prefetch_data=0 2023-10-05T19:28:27.578 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.570+0000 7f2117119e40 1 -- 172.21.15.167:0/1497151981 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:102 8.18 8:1e6ca5ed:users.uid::foo:head [call version.read in=11b,getxattrs,stat] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x556bf4abde70 con 0x556bf4a02930 2023-10-05T19:28:27.578 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.570+0000 7f203ffff700 20 rgw notify: INFO: next queues processing will happen at: Thu Oct 5 19:28:57 2023 2023-10-05T19:28:27.578 INFO:teuthology.orchestra.run.smithi167.stderr: 2023-10-05T19:28:27.578 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.570+0000 7f211544f700 1 -- 172.21.15.167:0/1497151981 <== osd.1 v2:172.21.15.167:6800/30035 35 ==== osd_op_reply(102 foo [call out=48b,getxattrs out=75b,stat out=16b] v0'0 uv4 ondisk = 0) v8 ==== 231+0+139 (crc 0 0 0) 0x7f2104050040 con 0x556bf4a02930 2023-10-05T19:28:27.578 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.570+0000 7f2117119e40 20 get_system_obj_state: s->obj_tag was set empty 2023-10-05T19:28:27.579 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.570+0000 7f2117119e40 20 Read xattr: user.rgw.idtag 2023-10-05T19:28:27.579 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.570+0000 7f2117119e40 20 rados->read ofs=0 len=0 2023-10-05T19:28:27.579 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.570+0000 7f2117119e40 1 -- 172.21.15.167:0/1497151981 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:103 8.18 8:1e6ca5ed:users.uid::foo:head [call version.check_conds in=122b,call version.read in=11b,read 0~0,getxattrs] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x556bf4abde30 con 0x556bf4a02930 2023-10-05T19:28:27.579 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.574+0000 7f211544f700 1 -- 172.21.15.167:0/1497151981 <== osd.1 v2:172.21.15.167:6800/30035 36 ==== osd_op_reply(103 foo [call,call out=48b,read 0~459 out=459b,getxattrs out=75b] v0'0 uv4 ondisk = 0) v8 ==== 273+0+582 (crc 0 0 0) 0x7f2104050040 con 0x556bf4a02930 2023-10-05T19:28:27.579 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.574+0000 7f2117119e40 20 rados_obj.operate() r=0 bl.length=459 2023-10-05T19:28:27.580 INFO:teuthology.orchestra.run.smithi167.stdout:{ 2023-10-05T19:28:27.580 INFO:teuthology.orchestra.run.smithi167.stdout: "user_id": "foo", 2023-10-05T19:28:27.580 INFO:teuthology.orchestra.run.smithi167.stdout: "display_name": "Foo", 2023-10-05T19:28:27.580 INFO:teuthology.orchestra.run.smithi167.stdout: "email": "foo@foo.com", 2023-10-05T19:28:27.580 INFO:teuthology.orchestra.run.smithi167.stdout: "suspended": 0, 2023-10-05T19:28:27.580 INFO:teuthology.orchestra.run.smithi167.stdout: "max_buckets": 4, 2023-10-05T19:28:27.580 INFO:teuthology.orchestra.run.smithi167.stdout: "subusers": [], 2023-10-05T19:28:27.580 INFO:teuthology.orchestra.run.smithi167.stdout: "keys": [ 2023-10-05T19:28:27.580 INFO:teuthology.orchestra.run.smithi167.stdout: { 2023-10-05T19:28:27.580 INFO:teuthology.orchestra.run.smithi167.stdout: "user": "foo", 2023-10-05T19:28:27.581 INFO:teuthology.orchestra.run.smithi167.stdout: "access_key": "9te6NH5mcdcq0Tc5i8i1", 2023-10-05T19:28:27.581 INFO:teuthology.orchestra.run.smithi167.stdout: "secret_key": "Ny4IOauQoL18Gp2zM7lC1vLmoawgqcYP/YGcWfXu" 2023-10-05T19:28:27.581 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2023-10-05T19:28:27.581 INFO:teuthology.orchestra.run.smithi167.stdout: { 2023-10-05T19:28:27.581 INFO:teuthology.orchestra.run.smithi167.stdout: "user": "foo", 2023-10-05T19:28:27.581 INFO:teuthology.orchestra.run.smithi167.stdout: "access_key": "p5YnriCv1nAtykxBrupQ", 2023-10-05T19:28:27.581 INFO:teuthology.orchestra.run.smithi167.stdout: "secret_key": "Q8Tk6Q/27hfbFSYdSkPtUqhqx1GgzvpXa4WARozh" 2023-10-05T19:28:27.581 INFO:teuthology.orchestra.run.smithi167.stdout: } 2023-10-05T19:28:27.581 INFO:teuthology.orchestra.run.smithi167.stdout: ], 2023-10-05T19:28:27.581 INFO:teuthology.orchestra.run.smithi167.stdout: "swift_keys": [], 2023-10-05T19:28:27.582 INFO:teuthology.orchestra.run.smithi167.stdout: "caps": [], 2023-10-05T19:28:27.582 INFO:teuthology.orchestra.run.smithi167.stdout: "op_mask": "read, write, delete", 2023-10-05T19:28:27.582 INFO:teuthology.orchestra.run.smithi167.stdout: "default_placement": "", 2023-10-05T19:28:27.582 INFO:teuthology.orchestra.run.smithi167.stdout: "default_storage_class": "", 2023-10-05T19:28:27.582 INFO:teuthology.orchestra.run.smithi167.stdout: "placement_tags": [], 2023-10-05T19:28:27.582 INFO:teuthology.orchestra.run.smithi167.stdout: "bucket_quota": { 2023-10-05T19:28:27.582 INFO:teuthology.orchestra.run.smithi167.stdout: "enabled": false, 2023-10-05T19:28:27.582 INFO:teuthology.orchestra.run.smithi167.stdout: "check_on_raw": false, 2023-10-05T19:28:27.582 INFO:teuthology.orchestra.run.smithi167.stdout: "max_size": -1, 2023-10-05T19:28:27.582 INFO:teuthology.orchestra.run.smithi167.stdout: "max_size_kb": 0, 2023-10-05T19:28:27.582 INFO:teuthology.orchestra.run.smithi167.stdout: "max_objects": -1 2023-10-05T19:28:27.582 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2023-10-05T19:28:27.583 INFO:teuthology.orchestra.run.smithi167.stdout: "user_quota": { 2023-10-05T19:28:27.583 INFO:teuthology.orchestra.run.smithi167.stdout: "enabled": false, 2023-10-05T19:28:27.583 INFO:teuthology.orchestra.run.smithi167.stdout: "check_on_raw": false, 2023-10-05T19:28:27.583 INFO:teuthology.orchestra.run.smithi167.stdout: "max_size": -1, 2023-10-05T19:28:27.583 INFO:teuthology.orchestra.run.smithi167.stdout: "max_size_kb": 0, 2023-10-05T19:28:27.583 INFO:teuthology.orchestra.run.smithi167.stdout: "max_objects": -1 2023-10-05T19:28:27.583 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2023-10-05T19:28:27.583 INFO:teuthology.orchestra.run.smithi167.stdout: "temp_url_keys": [], 2023-10-05T19:28:27.583 INFO:teuthology.orchestra.run.smithi167.stdout: "type": "rgw", 2023-10-05T19:28:27.583 INFO:teuthology.orchestra.run.smithi167.stdout: "mfa_ids": [] 2023-10-05T19:28:27.583 INFO:teuthology.orchestra.run.smithi167.stdout:} 2023-10-05T19:28:27.584 INFO:teuthology.orchestra.run.smithi167.stdout: 2023-10-05T19:28:27.584 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.574+0000 7f2117119e40 1 -- 172.21.15.167:0/1497151981 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:104 7.1f 7:f95f44c2:::notify.0:head [watch unwatch cookie 93922154234752] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x556bf4abde30 con 0x7f210c03a020 2023-10-05T19:28:27.584 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.574+0000 7f211444d700 1 -- 172.21.15.167:0/1497151981 <== osd.2 v2:172.21.15.167:6816/30037 17 ==== osd_op_reply(104 notify.0 [watch unwatch cookie 93922154234752] v30'50 uv48 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f20fc006070 con 0x7f210c03a020 2023-10-05T19:28:27.584 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.578+0000 7f2117119e40 20 remove_watcher() i=0 2023-10-05T19:28:27.584 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.578+0000 7f2117119e40 2 removed watcher, disabling cache 2023-10-05T19:28:27.585 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.578+0000 7f2117119e40 1 -- 172.21.15.167:0/1497151981 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:105 7.0 7:05bf5b68:::notify.1:head [watch unwatch cookie 93922154238272] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x556bf4a17730 con 0x556bf4a02930 2023-10-05T19:28:27.585 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.578+0000 7f211544f700 1 -- 172.21.15.167:0/1497151981 <== osd.1 v2:172.21.15.167:6800/30035 37 ==== osd_op_reply(105 notify.1 [watch unwatch cookie 93922154238272] v30'50 uv48 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f2104050040 con 0x556bf4a02930 2023-10-05T19:28:27.585 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.578+0000 7f2117119e40 20 remove_watcher() i=1 2023-10-05T19:28:27.585 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.578+0000 7f2117119e40 1 -- 172.21.15.167:0/1497151981 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:106 7.15 7:a93a5511:::notify.2:head [watch unwatch cookie 93922154241536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x556bf4a17730 con 0x556bf4a36fe0 2023-10-05T19:28:27.587 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.582+0000 7f2114c4e700 1 -- 172.21.15.167:0/1497151981 <== osd.3 v2:172.21.15.167:6808/30036 24 ==== osd_op_reply(106 notify.2 [watch unwatch cookie 93922154241536] v30'50 uv48 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f210c0397c0 con 0x556bf4a36fe0 2023-10-05T19:28:27.587 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.582+0000 7f2117119e40 20 remove_watcher() i=2 2023-10-05T19:28:27.587 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.582+0000 7f2117119e40 1 -- 172.21.15.167:0/1497151981 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:107 7.e 7:7759931f:::notify.3:head [watch unwatch cookie 93922154244688] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x556bf4a3de00 con 0x7f210c03a020 2023-10-05T19:28:27.589 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.582+0000 7f211444d700 1 -- 172.21.15.167:0/1497151981 <== osd.2 v2:172.21.15.167:6816/30037 18 ==== osd_op_reply(107 notify.3 [watch unwatch cookie 93922154244688] v30'50 uv48 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f20fc006070 con 0x7f210c03a020 2023-10-05T19:28:27.590 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.582+0000 7f2117119e40 20 remove_watcher() i=3 2023-10-05T19:28:27.590 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.582+0000 7f2117119e40 1 -- 172.21.15.167:0/1497151981 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:108 7.d 7:b4812045:::notify.4:head [watch unwatch cookie 93922154245792] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x556bf4a3de00 con 0x556bf4a02930 2023-10-05T19:28:27.592 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.586+0000 7f211544f700 1 -- 172.21.15.167:0/1497151981 <== osd.1 v2:172.21.15.167:6800/30035 38 ==== osd_op_reply(108 notify.4 [watch unwatch cookie 93922154245792] v30'50 uv48 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f2104050040 con 0x556bf4a02930 2023-10-05T19:28:27.592 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.586+0000 7f2117119e40 20 remove_watcher() i=4 2023-10-05T19:28:27.592 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.586+0000 7f2117119e40 1 -- 172.21.15.167:0/1497151981 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:109 7.3 7:c609908c:::notify.5:head [watch unwatch cookie 93922154249712] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x556bf4a3de00 con 0x7f20fc00ebe0 2023-10-05T19:28:27.595 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.590+0000 7f211544f700 1 -- 172.21.15.167:0/1497151981 <== osd.0 v2:172.21.15.118:6800/27690 29 ==== osd_op_reply(109 notify.5 [watch unwatch cookie 93922154249712] v30'50 uv48 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f2104050040 con 0x7f20fc00ebe0 2023-10-05T19:28:27.595 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.590+0000 7f2117119e40 20 remove_watcher() i=5 2023-10-05T19:28:27.595 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.590+0000 7f2117119e40 1 -- 172.21.15.167:0/1497151981 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:110 7.14 7:2b04a3e9:::notify.6:head [watch unwatch cookie 93922154253936] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x556bf4a3de00 con 0x7f20fc00ebe0 2023-10-05T19:28:27.598 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.594+0000 7f211544f700 1 -- 172.21.15.167:0/1497151981 <== osd.0 v2:172.21.15.118:6800/27690 30 ==== osd_op_reply(110 notify.6 [watch unwatch cookie 93922154253936] v30'50 uv48 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f2104050040 con 0x7f20fc00ebe0 2023-10-05T19:28:27.598 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.594+0000 7f2117119e40 20 remove_watcher() i=6 2023-10-05T19:28:27.598 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.594+0000 7f2117119e40 1 -- 172.21.15.167:0/1497151981 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:111 7.9 7:93e5b521:::notify.7:head [watch unwatch cookie 93922154258528] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x556bf4a3de00 con 0x7f20fc00ebe0 2023-10-05T19:28:27.601 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.594+0000 7f211544f700 1 -- 172.21.15.167:0/1497151981 <== osd.0 v2:172.21.15.118:6800/27690 31 ==== osd_op_reply(111 notify.7 [watch unwatch cookie 93922154258528] v30'50 uv48 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f2104050040 con 0x7f20fc00ebe0 2023-10-05T19:28:27.601 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.594+0000 7f2117119e40 20 remove_watcher() i=7 2023-10-05T19:28:27.603 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.598+0000 7f20e57f2700 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2023-10-05T19:28:27.603 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.598+0000 7f2117119e40 1 -- 172.21.15.167:0/1497151981 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:112 6.4 6:22d26bf9:::data_loggenerations_metadata:head [watch unwatch cookie 93922154259632] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x556bf4a3de00 con 0x556bf4a02930 2023-10-05T19:28:27.607 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.598+0000 7f211544f700 1 -- 172.21.15.167:0/1497151981 <== osd.1 v2:172.21.15.167:6800/30035 39 ==== osd_op_reply(112 data_loggenerations_metadata [watch unwatch cookie 93922154259632] v30'67 uv1 ondisk = 0) v8 ==== 172+0+0 (crc 0 0 0) 0x7f2104050040 con 0x556bf4a02930 2023-10-05T19:28:27.607 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.602+0000 7f2117119e40 1 -- 172.21.15.167:0/1497151981 >> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] conn(0x7f20fc00ebe0 msgr2=0x7f20fc00d8e0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-05T19:28:27.607 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.602+0000 7f2117119e40 1 --2- 172.21.15.167:0/1497151981 >> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] conn(0x7f20fc00ebe0 0x7f20fc00d8e0 crc :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-05T19:28:27.607 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.602+0000 7f2117119e40 1 -- 172.21.15.167:0/1497151981 >> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] conn(0x556bf4a02930 msgr2=0x556bf4a335d0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-05T19:28:27.607 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.602+0000 7f2117119e40 1 --2- 172.21.15.167:0/1497151981 >> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] conn(0x556bf4a02930 0x556bf4a335d0 crc :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-05T19:28:27.607 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.602+0000 7f2117119e40 1 -- 172.21.15.167:0/1497151981 >> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] conn(0x7f210c03a020 msgr2=0x7f210c03a730 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-05T19:28:27.608 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.602+0000 7f2117119e40 1 --2- 172.21.15.167:0/1497151981 >> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] conn(0x7f210c03a020 0x7f210c03a730 crc :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-05T19:28:27.608 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.602+0000 7f2117119e40 1 -- 172.21.15.167:0/1497151981 >> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] conn(0x556bf4a36fe0 msgr2=0x556bf4a393c0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-05T19:28:27.608 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.602+0000 7f2117119e40 1 --2- 172.21.15.167:0/1497151981 >> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] conn(0x556bf4a36fe0 0x556bf4a393c0 crc :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-05T19:28:27.608 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.602+0000 7f2117119e40 1 -- 172.21.15.167:0/1497151981 >> [v2:172.21.15.167:6824/29889,v1:172.21.15.167:6825/29889] conn(0x7f20f8030c50 msgr2=0x7f20f8033100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-05T19:28:27.608 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.602+0000 7f2117119e40 1 --2- 172.21.15.167:0/1497151981 >> [v2:172.21.15.167:6824/29889,v1:172.21.15.167:6825/29889] conn(0x7f20f8030c50 0x7f20f8033100 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f20fc004170 tx=0x7f20fc001400 comp rx=0 tx=0).stop 2023-10-05T19:28:27.608 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.602+0000 7f211544f700 1 -- 172.21.15.167:0/1497151981 reap_dead start 2023-10-05T19:28:27.608 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.602+0000 7f2117119e40 1 -- 172.21.15.167:0/1497151981 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x556bf4a02020 msgr2=0x556bf4a27b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-05T19:28:27.608 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.602+0000 7f2117119e40 1 --2- 172.21.15.167:0/1497151981 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x556bf4a02020 0x556bf4a27b90 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x556bf4a1d570 tx=0x7f210c01d000 comp rx=0 tx=0).stop 2023-10-05T19:28:27.608 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.602+0000 7f2117119e40 1 -- 172.21.15.167:0/1497151981 shutdown_connections 2023-10-05T19:28:27.608 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.602+0000 7f2117119e40 1 --2- 172.21.15.167:0/1497151981 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x556bf4a02020 0x556bf4a27b90 unknown :-1 s=CLOSED pgs=233 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-05T19:28:27.609 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.602+0000 7f2117119e40 1 -- 172.21.15.167:0/1497151981 >> 172.21.15.167:0/1497151981 conn(0x556bf4975870 msgr2=0x556bf4a1b4d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-05T19:28:27.609 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.602+0000 7f2117119e40 1 -- 172.21.15.167:0/1497151981 shutdown_connections 2023-10-05T19:28:27.609 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.602+0000 7f2117119e40 1 -- 172.21.15.167:0/1497151981 wait complete. 2023-10-05T19:28:27.610 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.602+0000 7f2117119e40 1 -- 172.21.15.167:0/3229428815 >> [v2:172.21.15.167:6824/29889,v1:172.21.15.167:6825/29889] conn(0x7f2054030ca0 msgr2=0x7f2054033150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-05T19:28:27.610 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.602+0000 7f2117119e40 1 --2- 172.21.15.167:0/3229428815 >> [v2:172.21.15.167:6824/29889,v1:172.21.15.167:6825/29889] conn(0x7f2054030ca0 0x7f2054033150 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x556bf4a41890 tx=0x7f2104000f90 comp rx=0 tx=0).stop 2023-10-05T19:28:27.610 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.602+0000 7f2117119e40 1 -- 172.21.15.167:0/3229428815 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x556bf4a4fbd0 msgr2=0x556bf4a55720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-05T19:28:27.610 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.602+0000 7f2117119e40 1 --2- 172.21.15.167:0/3229428815 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x556bf4a4fbd0 0x556bf4a55720 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x556bf4a49cb0 tx=0x7f20fc036000 comp rx=0 tx=0).stop 2023-10-05T19:28:27.610 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.602+0000 7f2117119e40 1 -- 172.21.15.167:0/3229428815 shutdown_connections 2023-10-05T19:28:27.610 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.602+0000 7f2117119e40 1 --2- 172.21.15.167:0/3229428815 >> [v2:172.21.15.167:6824/29889,v1:172.21.15.167:6825/29889] conn(0x7f2054030ca0 0x7f2054033150 unknown :-1 s=CLOSED pgs=85 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-05T19:28:27.610 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.602+0000 7f2117119e40 1 --2- 172.21.15.167:0/3229428815 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x556bf4a4fbd0 0x556bf4a55720 unknown :-1 s=CLOSED pgs=235 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-05T19:28:27.610 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.602+0000 7f2117119e40 1 -- 172.21.15.167:0/3229428815 >> 172.21.15.167:0/3229428815 conn(0x556bf4a4af90 msgr2=0x556bf4a49090 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-05T19:28:27.611 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.602+0000 7f2117119e40 1 -- 172.21.15.167:0/3229428815 shutdown_connections 2023-10-05T19:28:27.611 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.602+0000 7f2117119e40 1 -- 172.21.15.167:0/3229428815 wait complete. 2023-10-05T19:28:27.614 DEBUG:tasks.util.rgw: json result: {'user_id': 'foo', 'display_name': 'Foo', 'email': 'foo@foo.com', 'suspended': 0, 'max_buckets': 4, 'subusers': [], 'keys': [{'user': 'foo', 'access_key': '9te6NH5mcdcq0Tc5i8i1', 'secret_key': 'Ny4IOauQoL18Gp2zM7lC1vLmoawgqcYP/YGcWfXu'}, {'user': 'foo', 'access_key': 'p5YnriCv1nAtykxBrupQ', 'secret_key': 'Q8Tk6Q/27hfbFSYdSkPtUqhqx1GgzvpXa4WARozh'}], 'swift_keys': [], 'caps': [], 'op_mask': 'read, write, delete', 'default_placement': '', 'default_storage_class': '', 'placement_tags': [], 'bucket_quota': {'enabled': False, 'check_on_raw': False, 'max_size': -1, 'max_size_kb': 0, 'max_objects': -1}, 'user_quota': {'enabled': False, 'check_on_raw': False, 'max_size': -1, 'max_size_kb': 0, 'max_objects': -1}, 'temp_url_keys': [], 'type': 'rgw', 'mfa_ids': []} 2023-10-05T19:28:27.614 INFO:tasks.util.rgw:rgwadmin: client.0 : ['key', 'rm', '--uid', 'foo', '--access-key', 'p5YnriCv1nAtykxBrupQ'] 2023-10-05T19:28:27.614 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'ceph', 'key', 'rm', '--uid', 'foo', '--access-key', 'p5YnriCv1nAtykxBrupQ'] 2023-10-05T19:28:27.615 DEBUG:teuthology.orchestra.run.smithi167:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster ceph key rm --uid foo --access-key p5YnriCv1nAtykxBrupQ 2023-10-05T19:28:27.654 INFO:teuthology.orchestra.run.smithi167.stderr:ignoring --setuser ceph since I am not root 2023-10-05T19:28:27.655 INFO:teuthology.orchestra.run.smithi167.stderr:ignoring --setgroup ceph since I am not root 2023-10-05T19:28:27.655 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.650+0000 7f8063037e40 1 Processor -- start 2023-10-05T19:28:27.656 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.650+0000 7f8063037e40 1 -- start start 2023-10-05T19:28:27.656 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.650+0000 7f8063037e40 1 --2- >> v2:172.21.15.118:3300/0 conn(0x55ad9fb5e7f0 0x55ad9fb5d6e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-05T19:28:27.656 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.650+0000 7f8063037e40 1 -- --> v1:172.21.15.118:6789/0 -- auth(proto 0 26 bytes epoch 0) v1 -- 0x55ad9faaabd0 con 0x55ad9fb5e420 2023-10-05T19:28:27.656 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.650+0000 7f8063037e40 1 -- --> v2:172.21.15.118:3300/0 -- mon_getmap magic: 0 v1 -- 0x55ad9fa67ee0 con 0x55ad9fb5e7f0 2023-10-05T19:28:27.657 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.650+0000 7f806036b700 1 --2- >> v2:172.21.15.118:3300/0 conn(0x55ad9fb5e7f0 0x55ad9fb5d6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-05T19:28:27.657 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.650+0000 7f806036b700 1 --2- >> v2:172.21.15.118:3300/0 conn(0x55ad9fb5e7f0 0x55ad9fb5d6e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.118:3300/0 says I am v2:172.21.15.167:57550/0 (socket says 172.21.15.167:57550) 2023-10-05T19:28:27.657 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.650+0000 7f806036b700 1 -- 172.21.15.167:0/1779075998 learned_addr learned my addr 172.21.15.167:0/1779075998 (peer_addr_for_me v2:172.21.15.167:0/0) 2023-10-05T19:28:27.657 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.650+0000 7f8060b6c700 1 --1- 172.21.15.167:0/1779075998 >> v1:172.21.15.118:6789/0 conn(0x55ad9fb5e420 0x55ad9fb5cfd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.118:6789/0 says I am v1:172.21.15.167:56438/0 (socket says 172.21.15.167:56438) 2023-10-05T19:28:27.658 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.650+0000 7f806036b700 1 -- 172.21.15.167:0/1779075998 >> v1:172.21.15.118:6789/0 conn(0x55ad9fb5e420 legacy=0x55ad9fb5cfd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-05T19:28:27.658 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.650+0000 7f805fb6a700 1 -- 172.21.15.167:0/1779075998 <== mon.0 v1:172.21.15.118:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2665709430 0 0) 0x55ad9faaabd0 con 0x55ad9fb5e420 2023-10-05T19:28:27.658 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.650+0000 7f806036b700 1 -- 172.21.15.167:0/1779075998 --> v2:172.21.15.118:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55ad9fb5dce0 con 0x55ad9fb5e7f0 2023-10-05T19:28:27.658 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.650+0000 7f806036b700 1 --2- 172.21.15.167:0/1779075998 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55ad9fb5e7f0 0x55ad9fb5d6e0 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7f805400a650 tx=0x7f8054005b10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=90f4450865f1ae30 server_cookie=0 in_seq=0 out_seq=0 2023-10-05T19:28:27.659 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.654+0000 7f805fb6a700 1 -- 172.21.15.167:0/1779075998 <== mon.0 v2:172.21.15.118:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f8054008760 con 0x55ad9fb5e7f0 2023-10-05T19:28:27.659 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.654+0000 7f805fb6a700 1 -- 172.21.15.167:0/1779075998 <== mon.0 v2:172.21.15.118:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f80540088c0 con 0x55ad9fb5e7f0 2023-10-05T19:28:27.659 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.654+0000 7f805fb6a700 1 -- 172.21.15.167:0/1779075998 <== mon.0 v2:172.21.15.118:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f8054008bb0 con 0x55ad9fb5e7f0 2023-10-05T19:28:27.659 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.654+0000 7f8063037e40 1 -- 172.21.15.167:0/1779075998 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55ad9fb5e7f0 msgr2=0x55ad9fb5d6e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-05T19:28:27.659 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.654+0000 7f8063037e40 1 --2- 172.21.15.167:0/1779075998 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55ad9fb5e7f0 0x55ad9fb5d6e0 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7f805400a650 tx=0x7f8054005b10 comp rx=0 tx=0).stop 2023-10-05T19:28:27.660 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.654+0000 7f8063037e40 1 -- 172.21.15.167:0/1779075998 shutdown_connections 2023-10-05T19:28:27.660 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.654+0000 7f8063037e40 1 --2- 172.21.15.167:0/1779075998 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55ad9fb5e7f0 0x55ad9fb5d6e0 unknown :-1 s=CLOSED pgs=237 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-05T19:28:27.660 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.654+0000 7f8063037e40 1 -- 172.21.15.167:0/1779075998 >> 172.21.15.167:0/1779075998 conn(0x55ad9fb58730 msgr2=0x55ad9fb5ab40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-05T19:28:27.660 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.654+0000 7f8063037e40 1 -- 172.21.15.167:0/1779075998 shutdown_connections 2023-10-05T19:28:27.660 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.654+0000 7f8063037e40 1 -- 172.21.15.167:0/1779075998 wait complete. 2023-10-05T19:28:27.661 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.654+0000 7f8063037e40 1 Processor -- start 2023-10-05T19:28:27.661 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.654+0000 7f8063037e40 1 -- start start 2023-10-05T19:28:27.661 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.654+0000 7f8063037e40 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55ad9fb5e420 0x55ad9fb7f040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-05T19:28:27.661 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.654+0000 7f8063037e40 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ad9fa2be30 con 0x55ad9fb5e420 2023-10-05T19:28:27.661 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.654+0000 7f8060b6c700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55ad9fb5e420 0x55ad9fb7f040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-05T19:28:27.662 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.654+0000 7f8060b6c700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55ad9fb5e420 0x55ad9fb7f040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.118:3300/0 says I am v2:172.21.15.167:57564/0 (socket says 172.21.15.167:57564) 2023-10-05T19:28:27.662 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.654+0000 7f8060b6c700 1 -- 172.21.15.167:0/214245381 learned_addr learned my addr 172.21.15.167:0/214245381 (peer_addr_for_me v2:172.21.15.167:0/0) 2023-10-05T19:28:27.662 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.654+0000 7f8060b6c700 1 -- 172.21.15.167:0/214245381 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8054005db0 con 0x55ad9fb5e420 2023-10-05T19:28:27.663 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.654+0000 7f8060b6c700 1 --2- 172.21.15.167:0/214245381 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55ad9fb5e420 0x55ad9fb7f040 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7f8050001e80 tx=0x7f8050001850 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a5765c70e58700ee server_cookie=0 in_seq=0 out_seq=0 2023-10-05T19:28:27.663 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.654+0000 7f805eb68700 1 -- 172.21.15.167:0/214245381 <== mon.0 v2:172.21.15.118:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f8050014070 con 0x55ad9fb5e420 2023-10-05T19:28:27.663 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.658+0000 7f805eb68700 1 -- 172.21.15.167:0/214245381 <== mon.0 v2:172.21.15.118:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f805000f450 con 0x55ad9fb5e420 2023-10-05T19:28:27.663 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.658+0000 7f805eb68700 1 -- 172.21.15.167:0/214245381 <== mon.0 v2:172.21.15.118:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f805000f740 con 0x55ad9fb5e420 2023-10-05T19:28:27.663 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.658+0000 7f8063037e40 1 -- 172.21.15.167:0/214245381 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55ad9fb5e420 msgr2=0x55ad9fb7f040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-05T19:28:27.663 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.658+0000 7f8063037e40 1 --2- 172.21.15.167:0/214245381 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55ad9fb5e420 0x55ad9fb7f040 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7f8050001e80 tx=0x7f8050001850 comp rx=0 tx=0).stop 2023-10-05T19:28:27.664 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.658+0000 7f8063037e40 1 -- 172.21.15.167:0/214245381 shutdown_connections 2023-10-05T19:28:27.664 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.658+0000 7f8063037e40 1 --2- 172.21.15.167:0/214245381 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55ad9fb5e420 0x55ad9fb7f040 unknown :-1 s=CLOSED pgs=238 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-05T19:28:27.664 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.658+0000 7f8063037e40 1 -- 172.21.15.167:0/214245381 >> 172.21.15.167:0/214245381 conn(0x55ad9fb58730 msgr2=0x55ad9fb7ac00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-05T19:28:27.664 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.658+0000 7f8063037e40 1 -- 172.21.15.167:0/214245381 shutdown_connections 2023-10-05T19:28:27.664 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.658+0000 7f8063037e40 1 -- 172.21.15.167:0/214245381 wait complete. 2023-10-05T19:28:27.664 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.658+0000 7f8063037e40 1 Processor -- start 2023-10-05T19:28:27.664 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.658+0000 7f8063037e40 1 -- start start 2023-10-05T19:28:27.664 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.658+0000 7f8063037e40 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55ad9fb5e420 0x55ad9fb85dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-05T19:28:27.664 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.658+0000 7f8063037e40 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ad9f9feed0 con 0x55ad9fb5e420 2023-10-05T19:28:27.665 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.658+0000 7f8060b6c700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55ad9fb5e420 0x55ad9fb85dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-05T19:28:27.666 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.658+0000 7f8060b6c700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55ad9fb5e420 0x55ad9fb85dd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.118:3300/0 says I am v2:172.21.15.167:57572/0 (socket says 172.21.15.167:57572) 2023-10-05T19:28:27.666 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.658+0000 7f8060b6c700 1 -- 172.21.15.167:0/355163605 learned_addr learned my addr 172.21.15.167:0/355163605 (peer_addr_for_me v2:172.21.15.167:0/0) 2023-10-05T19:28:27.666 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.658+0000 7f8060b6c700 1 -- 172.21.15.167:0/355163605 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8050001b30 con 0x55ad9fb5e420 2023-10-05T19:28:27.666 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.658+0000 7f8060b6c700 1 --2- 172.21.15.167:0/355163605 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55ad9fb5e420 0x55ad9fb85dd0 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7f80500011e0 tx=0x7f8050016000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-05T19:28:27.666 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.658+0000 7f805d365700 1 -- 172.21.15.167:0/355163605 <== mon.0 v2:172.21.15.118:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f805000fe50 con 0x55ad9fb5e420 2023-10-05T19:28:27.666 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.658+0000 7f805d365700 1 -- 172.21.15.167:0/355163605 <== mon.0 v2:172.21.15.118:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f805001cb40 con 0x55ad9fb5e420 2023-10-05T19:28:27.666 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.658+0000 7f8063037e40 1 -- 172.21.15.167:0/355163605 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x55ad9fb86310 con 0x55ad9fb5e420 2023-10-05T19:28:27.666 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.658+0000 7f805d365700 1 -- 172.21.15.167:0/355163605 <== mon.0 v2:172.21.15.118:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f805001ce30 con 0x55ad9fb5e420 2023-10-05T19:28:27.666 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.658+0000 7f8063037e40 1 -- 172.21.15.167:0/355163605 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55ad9fb86750 con 0x55ad9fb5e420 2023-10-05T19:28:27.667 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.662+0000 7f805d365700 1 -- 172.21.15.167:0/355163605 <== mon.0 v2:172.21.15.118:3300/0 4 ==== mgrmap(e 5) v1 ==== 40329+0+0 (secure 0 0 0) 0x7f8050023020 con 0x55ad9fb5e420 2023-10-05T19:28:27.668 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.662+0000 7f8063037e40 20 rados->read ofs=0 len=0 2023-10-05T19:28:27.668 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.662+0000 7f805d365700 1 --2- 172.21.15.167:0/355163605 >> [v2:172.21.15.167:6824/29889,v1:172.21.15.167:6825/29889] conn(0x7f804c030c50 0x7f804c033100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-05T19:28:27.668 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.662+0000 7f805d365700 1 -- 172.21.15.167:0/355163605 <== mon.0 v2:172.21.15.118:3300/0 5 ==== osd_map(30..30 src has 1..30) v4 ==== 6057+0+0 (secure 0 0 0) 0x7f8050008030 con 0x55ad9fb5e420 2023-10-05T19:28:27.668 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.662+0000 7f806036b700 1 --2- 172.21.15.167:0/355163605 >> [v2:172.21.15.167:6824/29889,v1:172.21.15.167:6825/29889] conn(0x7f804c030c50 0x7f804c033100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-05T19:28:27.668 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.662+0000 7f8063037e40 1 --2- 172.21.15.167:0/355163605 >> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] conn(0x55ad9fb5e7f0 0x55ad9fb91890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-05T19:28:27.668 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.662+0000 7f8063037e40 1 -- 172.21.15.167:0/355163605 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:1 5.12 5:49953fa1:::default.realm:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x55ad9fb91f90 con 0x55ad9fb5e7f0 2023-10-05T19:28:27.668 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.662+0000 7f806136d700 1 --2- 172.21.15.167:0/355163605 >> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] conn(0x55ad9fb5e7f0 0x55ad9fb91890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-05T19:28:27.669 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.662+0000 7f806036b700 1 --2- 172.21.15.167:0/355163605 >> [v2:172.21.15.167:6824/29889,v1:172.21.15.167:6825/29889] conn(0x7f804c030c50 0x7f804c033100 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f80540030a0 tx=0x7f8054015000 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-05T19:28:27.669 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.662+0000 7f806136d700 1 --2- 172.21.15.167:0/355163605 >> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] conn(0x55ad9fb5e7f0 0x55ad9fb91890 crc :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-05T19:28:27.669 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.662+0000 7f806136d700 1 -- 172.21.15.167:0/355163605 <== osd.1 v2:172.21.15.167:6800/30035 1 ==== osd_op_reply(1 default.realm [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 ==== 157+0+0 (crc 0 0 0) 0x7f805800a5a0 con 0x55ad9fb5e7f0 2023-10-05T19:28:27.669 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.662+0000 7f8063037e40 20 rados_obj.operate() r=-2 bl.length=0 2023-10-05T19:28:27.670 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.662+0000 7f8063037e40 20 realm 2023-10-05T19:28:27.670 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.662+0000 7f8063037e40 20 rados->read ofs=0 len=0 2023-10-05T19:28:27.670 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.662+0000 7f8063037e40 1 -- 172.21.15.167:0/355163605 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:2 5.12 5:49953fa1:::default.realm:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x55ad9fb8eeb0 con 0x55ad9fb5e7f0 2023-10-05T19:28:27.670 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.662+0000 7f806136d700 1 -- 172.21.15.167:0/355163605 <== osd.1 v2:172.21.15.167:6800/30035 2 ==== osd_op_reply(2 default.realm [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 ==== 157+0+0 (crc 0 0 0) 0x7f805800a5a0 con 0x55ad9fb5e7f0 2023-10-05T19:28:27.670 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.662+0000 7f8063037e40 20 rados_obj.operate() r=-2 bl.length=0 2023-10-05T19:28:27.670 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.662+0000 7f8063037e40 4 RGWPeriod::init failed to init realm id : (2) No such file or directory 2023-10-05T19:28:27.670 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.662+0000 7f8063037e40 20 rados->read ofs=0 len=0 2023-10-05T19:28:27.670 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.662+0000 7f8063037e40 1 -- 172.21.15.167:0/355163605 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:3 5.12 5:49953fa1:::default.realm:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x55ad9fb92a70 con 0x55ad9fb5e7f0 2023-10-05T19:28:27.672 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.662+0000 7f806136d700 1 -- 172.21.15.167:0/355163605 <== osd.1 v2:172.21.15.167:6800/30035 3 ==== osd_op_reply(3 default.realm [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 ==== 157+0+0 (crc 0 0 0) 0x7f805800a5a0 con 0x55ad9fb5e7f0 2023-10-05T19:28:27.672 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.662+0000 7f8063037e40 20 rados_obj.operate() r=-2 bl.length=0 2023-10-05T19:28:27.672 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.662+0000 7f8063037e40 20 rados->read ofs=0 len=0 2023-10-05T19:28:27.672 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.662+0000 7f8063037e40 1 -- 172.21.15.167:0/355163605 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:4 5.f 5:f43b6ece:::zone_names.default:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x55ad9fb92650 con 0x55ad9fb5e7f0 2023-10-05T19:28:27.672 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.666+0000 7f806136d700 1 -- 172.21.15.167:0/355163605 <== osd.1 v2:172.21.15.167:6800/30035 4 ==== osd_op_reply(4 zone_names.default [read 0~46 out=46b] v0'0 uv1 ondisk = 0) v8 ==== 162+0+46 (crc 0 0 0) 0x7f805800a5a0 con 0x55ad9fb5e7f0 2023-10-05T19:28:27.672 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.666+0000 7f8063037e40 20 rados_obj.operate() r=0 bl.length=46 2023-10-05T19:28:27.672 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.666+0000 7f8063037e40 20 rados->read ofs=0 len=0 2023-10-05T19:28:27.673 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.666+0000 7f8063037e40 1 -- 172.21.15.167:0/355163605 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:5 5.1d 5:b8374eb4:::zone_info.9a9df965-6612-45d4-a45f-6e3c889bacc2:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x55ad9fb93610 con 0x55ad9fb5e7f0 2023-10-05T19:28:27.673 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.666+0000 7f806136d700 1 -- 172.21.15.167:0/355163605 <== osd.1 v2:172.21.15.167:6800/30035 5 ==== osd_op_reply(5 zone_info.9a9df965-6612-45d4-a45f-6e3c889bacc2 [read 0~874 out=874b] v0'0 uv2 ondisk = 0) v8 ==== 190+0+874 (crc 0 0 0) 0x7f805800a5a0 con 0x55ad9fb5e7f0 2023-10-05T19:28:27.673 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.666+0000 7f8063037e40 20 rados_obj.operate() r=0 bl.length=874 2023-10-05T19:28:27.673 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.666+0000 7f8063037e40 20 searching for the correct realm 2023-10-05T19:28:27.673 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.666+0000 7f8063037e40 1 --2- 172.21.15.167:0/355163605 >> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] conn(0x55ad9fb95550 0x55ad9fb97960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-05T19:28:27.673 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.666+0000 7f8063037e40 1 -- 172.21.15.167:0/355163605 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:6 5.0 5:00000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x55ad9fb97f70 con 0x55ad9fb95550 2023-10-05T19:28:27.673 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.666+0000 7f8060b6c700 1 --2- 172.21.15.167:0/355163605 >> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] conn(0x55ad9fb95550 0x55ad9fb97960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-05T19:28:27.673 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.666+0000 7f8060b6c700 1 --2- 172.21.15.167:0/355163605 >> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] conn(0x55ad9fb95550 0x55ad9fb97960 crc :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-05T19:28:27.674 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.666+0000 7f8060b6c700 1 -- 172.21.15.167:0/355163605 <== osd.3 v2:172.21.15.167:6808/30036 1 ==== osd_op_reply(6 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f8050016070 con 0x55ad9fb95550 2023-10-05T19:28:27.674 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.666+0000 7f8060b6c700 1 --2- 172.21.15.167:0/355163605 >> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] conn(0x7f805001a070 0x7f805001d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-05T19:28:27.674 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.666+0000 7f8060b6c700 1 -- 172.21.15.167:0/355163605 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:7 5.10 5:08000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x55ad9fb97f70 con 0x7f805001a070 2023-10-05T19:28:27.675 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.666+0000 7f806036b700 1 --2- 172.21.15.167:0/355163605 >> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] conn(0x7f805001a070 0x7f805001d450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-05T19:28:27.675 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.666+0000 7f806036b700 1 --2- 172.21.15.167:0/355163605 >> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] conn(0x7f805001a070 0x7f805001d450 crc :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-05T19:28:27.675 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.670+0000 7f806036b700 1 -- 172.21.15.167:0/355163605 <== osd.2 v2:172.21.15.167:6816/30037 1 ==== osd_op_reply(7 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f8054012450 con 0x7f805001a070 2023-10-05T19:28:27.675 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.670+0000 7f806036b700 1 --2- 172.21.15.167:0/355163605 >> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] conn(0x7f8054007030 0x7f805401c920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-05T19:28:27.675 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.670+0000 7f806036b700 1 -- 172.21.15.167:0/355163605 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:8 5.8 5:10000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x55ad9fb97f70 con 0x7f8054007030 2023-10-05T19:28:27.675 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.670+0000 7f806136d700 1 --2- 172.21.15.167:0/355163605 >> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] conn(0x7f8054007030 0x7f805401c920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-05T19:28:27.676 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.670+0000 7f806136d700 1 --2- 172.21.15.167:0/355163605 >> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] conn(0x7f8054007030 0x7f805401c920 crc :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-05T19:28:27.676 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.670+0000 7f806136d700 1 -- 172.21.15.167:0/355163605 <== osd.0 v2:172.21.15.118:6800/27690 1 ==== osd_op_reply(8 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f805800a5a0 con 0x7f8054007030 2023-10-05T19:28:27.677 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.670+0000 7f806136d700 1 -- 172.21.15.167:0/355163605 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:9 5.18 5:18000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x55ad9fb97f70 con 0x55ad9fb5e7f0 2023-10-05T19:28:27.677 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.670+0000 7f806136d700 1 -- 172.21.15.167:0/355163605 <== osd.1 v2:172.21.15.167:6800/30035 6 ==== osd_op_reply(9 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f805800a5a0 con 0x55ad9fb5e7f0 2023-10-05T19:28:27.677 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.670+0000 7f806136d700 1 -- 172.21.15.167:0/355163605 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:10 5.4 5:20000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x55ad9fb97f70 con 0x55ad9fb5e7f0 2023-10-05T19:28:27.678 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.670+0000 7f806136d700 1 -- 172.21.15.167:0/355163605 <== osd.1 v2:172.21.15.167:6800/30035 7 ==== osd_op_reply(10 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f805800a5a0 con 0x55ad9fb5e7f0 2023-10-05T19:28:27.678 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.674+0000 7f806136d700 1 -- 172.21.15.167:0/355163605 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:11 5.14 5:28000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x55ad9fb97f70 con 0x7f8054007030 2023-10-05T19:28:27.679 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.674+0000 7f806136d700 1 -- 172.21.15.167:0/355163605 <== osd.0 v2:172.21.15.118:6800/27690 2 ==== osd_op_reply(11 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f805800a5a0 con 0x7f8054007030 2023-10-05T19:28:27.679 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.674+0000 7f806136d700 1 -- 172.21.15.167:0/355163605 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:12 5.c 5:30000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x55ad9fb97f70 con 0x7f8054007030 2023-10-05T19:28:27.679 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.674+0000 7f806136d700 1 -- 172.21.15.167:0/355163605 <== osd.0 v2:172.21.15.118:6800/27690 3 ==== osd_op_reply(12 [pgnls start_epoch 30 out=112b] v23'1 uv1 ondisk = 1) v8 ==== 144+0+112 (crc 0 0 0) 0x7f805800a5a0 con 0x7f8054007030 2023-10-05T19:28:27.680 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.674+0000 7f806136d700 1 -- 172.21.15.167:0/355163605 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:13 5.1c 5:38000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x55ad9fb97f70 con 0x7f805001a070 2023-10-05T19:28:27.680 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.674+0000 7f806036b700 1 -- 172.21.15.167:0/355163605 <== osd.2 v2:172.21.15.167:6816/30037 2 ==== osd_op_reply(13 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f8054012450 con 0x7f805001a070 2023-10-05T19:28:27.680 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.674+0000 7f806036b700 1 -- 172.21.15.167:0/355163605 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:14 5.2 5:40000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x55ad9fb97f70 con 0x55ad9fb95550 2023-10-05T19:28:27.681 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.674+0000 7f8060b6c700 1 -- 172.21.15.167:0/355163605 <== osd.3 v2:172.21.15.167:6808/30036 2 ==== osd_op_reply(14 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f8050016070 con 0x55ad9fb95550 2023-10-05T19:28:27.681 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.674+0000 7f8060b6c700 1 -- 172.21.15.167:0/355163605 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:15 5.12 5:48000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x55ad9fb97f70 con 0x55ad9fb5e7f0 2023-10-05T19:28:27.682 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.678+0000 7f806136d700 1 -- 172.21.15.167:0/355163605 <== osd.1 v2:172.21.15.167:6800/30035 8 ==== osd_op_reply(15 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f805800a5a0 con 0x55ad9fb5e7f0 2023-10-05T19:28:27.682 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.678+0000 7f806136d700 1 -- 172.21.15.167:0/355163605 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:16 5.a 5:50000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x55ad9fb97f70 con 0x7f8054007030 2023-10-05T19:28:27.683 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.678+0000 7f806136d700 1 -- 172.21.15.167:0/355163605 <== osd.0 v2:172.21.15.118:6800/27690 4 ==== osd_op_reply(16 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f805800a5a0 con 0x7f8054007030 2023-10-05T19:28:27.683 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.678+0000 7f806136d700 1 -- 172.21.15.167:0/355163605 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:17 5.1a 5:58000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x55ad9fb97f70 con 0x55ad9fb5e7f0 2023-10-05T19:28:27.683 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.678+0000 7f806136d700 1 -- 172.21.15.167:0/355163605 <== osd.1 v2:172.21.15.167:6800/30035 9 ==== osd_op_reply(17 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f805800a5a0 con 0x55ad9fb5e7f0 2023-10-05T19:28:27.684 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.678+0000 7f806136d700 1 -- 172.21.15.167:0/355163605 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:18 5.6 5:60000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x55ad9fb97f70 con 0x7f805001a070 2023-10-05T19:28:27.684 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.678+0000 7f806036b700 1 -- 172.21.15.167:0/355163605 <== osd.2 v2:172.21.15.167:6816/30037 3 ==== osd_op_reply(18 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f8054012450 con 0x7f805001a070 2023-10-05T19:28:27.684 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.678+0000 7f806036b700 1 -- 172.21.15.167:0/355163605 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:19 5.16 5:68000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x55ad9fb97f70 con 0x55ad9fb95550 2023-10-05T19:28:27.685 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.678+0000 7f8060b6c700 1 -- 172.21.15.167:0/355163605 <== osd.3 v2:172.21.15.167:6808/30036 3 ==== osd_op_reply(19 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f8050016070 con 0x55ad9fb95550 2023-10-05T19:28:27.685 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.678+0000 7f8060b6c700 1 -- 172.21.15.167:0/355163605 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:20 5.e 5:70000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x55ad9fb97f70 con 0x55ad9fb95550 2023-10-05T19:28:27.686 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.678+0000 7f8060b6c700 1 -- 172.21.15.167:0/355163605 <== osd.3 v2:172.21.15.167:6808/30036 4 ==== osd_op_reply(20 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f8050016070 con 0x55ad9fb95550 2023-10-05T19:28:27.686 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.678+0000 7f8060b6c700 1 -- 172.21.15.167:0/355163605 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:21 5.1e 5:78000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x55ad9fb97f70 con 0x7f8054007030 2023-10-05T19:28:27.686 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.682+0000 7f806136d700 1 -- 172.21.15.167:0/355163605 <== osd.0 v2:172.21.15.118:6800/27690 5 ==== osd_op_reply(21 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f805800a5a0 con 0x7f8054007030 2023-10-05T19:28:27.686 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.682+0000 7f806136d700 1 -- 172.21.15.167:0/355163605 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:22 5.1 5:80000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x55ad9fb97f70 con 0x55ad9fb95550 2023-10-05T19:28:27.687 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.682+0000 7f8060b6c700 1 -- 172.21.15.167:0/355163605 <== osd.3 v2:172.21.15.167:6808/30036 5 ==== osd_op_reply(22 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f8050016070 con 0x55ad9fb95550 2023-10-05T19:28:27.687 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.682+0000 7f8060b6c700 1 -- 172.21.15.167:0/355163605 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:23 5.11 5:88000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x55ad9fb97f70 con 0x55ad9fb95550 2023-10-05T19:28:27.688 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.682+0000 7f8060b6c700 1 -- 172.21.15.167:0/355163605 <== osd.3 v2:172.21.15.167:6808/30036 6 ==== osd_op_reply(23 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f8050016070 con 0x55ad9fb95550 2023-10-05T19:28:27.688 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.682+0000 7f8060b6c700 1 -- 172.21.15.167:0/355163605 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:24 5.9 5:90000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x55ad9fb97f70 con 0x55ad9fb5e7f0 2023-10-05T19:28:27.689 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.682+0000 7f806136d700 1 -- 172.21.15.167:0/355163605 <== osd.1 v2:172.21.15.167:6800/30035 10 ==== osd_op_reply(24 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f805800a5a0 con 0x55ad9fb5e7f0 2023-10-05T19:28:27.689 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.682+0000 7f806136d700 1 -- 172.21.15.167:0/355163605 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:25 5.19 5:98000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x55ad9fb97f70 con 0x55ad9fb5e7f0 2023-10-05T19:28:27.689 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.682+0000 7f806136d700 1 -- 172.21.15.167:0/355163605 <== osd.1 v2:172.21.15.167:6800/30035 11 ==== osd_op_reply(25 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f805800a5a0 con 0x55ad9fb5e7f0 2023-10-05T19:28:27.689 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.682+0000 7f806136d700 1 -- 172.21.15.167:0/355163605 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:26 5.5 5:a0000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x55ad9fb97f70 con 0x7f8054007030 2023-10-05T19:28:27.690 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.686+0000 7f806136d700 1 -- 172.21.15.167:0/355163605 <== osd.0 v2:172.21.15.118:6800/27690 6 ==== osd_op_reply(26 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f805800a5a0 con 0x7f8054007030 2023-10-05T19:28:27.690 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.686+0000 7f806136d700 1 -- 172.21.15.167:0/355163605 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:27 5.15 5:a8000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x55ad9fb97f70 con 0x55ad9fb95550 2023-10-05T19:28:27.691 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.686+0000 7f8060b6c700 1 -- 172.21.15.167:0/355163605 <== osd.3 v2:172.21.15.167:6808/30036 7 ==== osd_op_reply(27 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f8050016070 con 0x55ad9fb95550 2023-10-05T19:28:27.691 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.686+0000 7f8060b6c700 1 -- 172.21.15.167:0/355163605 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:28 5.d 5:b0000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x55ad9fb97f70 con 0x7f805001a070 2023-10-05T19:28:27.692 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.686+0000 7f806036b700 1 -- 172.21.15.167:0/355163605 <== osd.2 v2:172.21.15.167:6816/30037 4 ==== osd_op_reply(28 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f8054012450 con 0x7f805001a070 2023-10-05T19:28:27.692 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.686+0000 7f806036b700 1 -- 172.21.15.167:0/355163605 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:29 5.1d 5:b8000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x55ad9fb97f70 con 0x55ad9fb5e7f0 2023-10-05T19:28:27.693 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.686+0000 7f806136d700 1 -- 172.21.15.167:0/355163605 <== osd.1 v2:172.21.15.167:6800/30035 12 ==== osd_op_reply(29 [pgnls start_epoch 30 out=107b] v30'2 uv2 ondisk = 1) v8 ==== 144+0+107 (crc 0 0 0) 0x7f805800a5a0 con 0x55ad9fb5e7f0 2023-10-05T19:28:27.693 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.686+0000 7f806136d700 1 -- 172.21.15.167:0/355163605 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:30 5.3 5:c0000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x55ad9fb97f70 con 0x7f8054007030 2023-10-05T19:28:27.694 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.690+0000 7f806136d700 1 -- 172.21.15.167:0/355163605 <== osd.0 v2:172.21.15.118:6800/27690 7 ==== osd_op_reply(30 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f805800a5a0 con 0x7f8054007030 2023-10-05T19:28:27.694 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.690+0000 7f806136d700 1 -- 172.21.15.167:0/355163605 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:31 5.13 5:c8000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x55ad9fb97f70 con 0x55ad9fb95550 2023-10-05T19:28:27.694 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.690+0000 7f8060b6c700 1 -- 172.21.15.167:0/355163605 <== osd.3 v2:172.21.15.167:6808/30036 8 ==== osd_op_reply(31 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f8050016070 con 0x55ad9fb95550 2023-10-05T19:28:27.695 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.690+0000 7f8060b6c700 1 -- 172.21.15.167:0/355163605 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:32 5.b 5:d0000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x55ad9fb97f70 con 0x7f8054007030 2023-10-05T19:28:27.695 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.690+0000 7f806136d700 1 -- 172.21.15.167:0/355163605 <== osd.0 v2:172.21.15.118:6800/27690 8 ==== osd_op_reply(32 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f805800a5a0 con 0x7f8054007030 2023-10-05T19:28:27.695 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.690+0000 7f806136d700 1 -- 172.21.15.167:0/355163605 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:33 5.1b 5:d8000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x55ad9fb97f70 con 0x7f805001a070 2023-10-05T19:28:27.696 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.690+0000 7f806036b700 1 -- 172.21.15.167:0/355163605 <== osd.2 v2:172.21.15.167:6816/30037 5 ==== osd_op_reply(33 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f8054012450 con 0x7f805001a070 2023-10-05T19:28:27.696 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.690+0000 7f806036b700 1 -- 172.21.15.167:0/355163605 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:34 5.7 5:e0000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x55ad9fb97f70 con 0x55ad9fb95550 2023-10-05T19:28:27.697 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.690+0000 7f8060b6c700 1 -- 172.21.15.167:0/355163605 <== osd.3 v2:172.21.15.167:6808/30036 9 ==== osd_op_reply(34 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f8050016070 con 0x55ad9fb95550 2023-10-05T19:28:27.697 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.690+0000 7f8060b6c700 1 -- 172.21.15.167:0/355163605 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:35 5.17 5:e8000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x55ad9fb97f70 con 0x7f8054007030 2023-10-05T19:28:27.698 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.694+0000 7f806136d700 1 -- 172.21.15.167:0/355163605 <== osd.0 v2:172.21.15.118:6800/27690 9 ==== osd_op_reply(35 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f805800a5a0 con 0x7f8054007030 2023-10-05T19:28:27.698 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.694+0000 7f806136d700 1 -- 172.21.15.167:0/355163605 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:36 5.f 5:f0000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x55ad9fb97f70 con 0x55ad9fb5e7f0 2023-10-05T19:28:27.699 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.694+0000 7f806136d700 1 -- 172.21.15.167:0/355163605 <== osd.1 v2:172.21.15.167:6800/30035 13 ==== osd_op_reply(36 [pgnls start_epoch 30 out=115b] v23'2 uv2 ondisk = 1) v8 ==== 144+0+115 (crc 0 0 0) 0x7f805800a5a0 con 0x55ad9fb5e7f0 2023-10-05T19:28:27.699 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.694+0000 7f806136d700 1 -- 172.21.15.167:0/355163605 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:37 5.1f 5:f8000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x55ad9fb97f70 con 0x7f805001a070 2023-10-05T19:28:27.700 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.694+0000 7f806036b700 1 -- 172.21.15.167:0/355163605 <== osd.2 v2:172.21.15.167:6816/30037 6 ==== osd_op_reply(37 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f8054012450 con 0x7f805001a070 2023-10-05T19:28:27.700 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.694+0000 7f8063037e40 20 RGWRados::pool_iterate: got zonegroup_info.4f1f719d-9648-417b-b10c-e5d939825419 2023-10-05T19:28:27.700 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.694+0000 7f8063037e40 20 RGWRados::pool_iterate: got zone_info.9a9df965-6612-45d4-a45f-6e3c889bacc2 2023-10-05T19:28:27.700 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.694+0000 7f8063037e40 20 RGWRados::pool_iterate: got zone_names.default 2023-10-05T19:28:27.700 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.694+0000 7f8063037e40 20 RGWRados::pool_iterate: got zonegroups_names.default 2023-10-05T19:28:27.701 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.694+0000 7f8063037e40 20 rados->read ofs=0 len=0 2023-10-05T19:28:27.701 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.694+0000 7f8063037e40 1 -- 172.21.15.167:0/355163605 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:38 5.12 5:49953fa1:::default.realm:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x55ad9fb94000 con 0x55ad9fb5e7f0 2023-10-05T19:28:27.702 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.694+0000 7f806136d700 1 -- 172.21.15.167:0/355163605 <== osd.1 v2:172.21.15.167:6800/30035 14 ==== osd_op_reply(38 default.realm [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 ==== 157+0+0 (crc 0 0 0) 0x7f805800a5a0 con 0x55ad9fb5e7f0 2023-10-05T19:28:27.702 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.694+0000 7f8063037e40 20 rados_obj.operate() r=-2 bl.length=0 2023-10-05T19:28:27.702 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.694+0000 7f8063037e40 20 rados->read ofs=0 len=0 2023-10-05T19:28:27.702 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.694+0000 7f8063037e40 1 -- 172.21.15.167:0/355163605 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:39 5.f 5:f4c53578:::zonegroups_names.default:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x55ad9fb94400 con 0x55ad9fb5e7f0 2023-10-05T19:28:27.702 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.694+0000 7f806136d700 1 -- 172.21.15.167:0/355163605 <== osd.1 v2:172.21.15.167:6800/30035 15 ==== osd_op_reply(39 zonegroups_names.default [read 0~46 out=46b] v0'0 uv2 ondisk = 0) v8 ==== 168+0+46 (crc 0 0 0) 0x7f805800a5a0 con 0x55ad9fb5e7f0 2023-10-05T19:28:27.702 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.694+0000 7f8063037e40 20 rados_obj.operate() r=0 bl.length=46 2023-10-05T19:28:27.702 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.694+0000 7f8063037e40 20 rados->read ofs=0 len=0 2023-10-05T19:28:27.702 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.694+0000 7f8063037e40 1 -- 172.21.15.167:0/355163605 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:40 5.c 5:30d299a5:::zonegroup_info.4f1f719d-9648-417b-b10c-e5d939825419:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x55ad9fb94400 con 0x7f8054007030 2023-10-05T19:28:27.703 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.698+0000 7f806136d700 1 -- 172.21.15.167:0/355163605 <== osd.0 v2:172.21.15.118:6800/27690 10 ==== osd_op_reply(40 zonegroup_info.4f1f719d-9648-417b-b10c-e5d939825419 [read 0~362 out=362b] v0'0 uv1 ondisk = 0) v8 ==== 195+0+362 (crc 0 0 0) 0x7f805800a5a0 con 0x7f8054007030 2023-10-05T19:28:27.703 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.698+0000 7f8063037e40 20 rados_obj.operate() r=0 bl.length=362 2023-10-05T19:28:27.703 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.698+0000 7f8063037e40 20 rados->read ofs=0 len=0 2023-10-05T19:28:27.703 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.698+0000 7f8063037e40 1 -- 172.21.15.167:0/355163605 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:41 5.13 5:cbbf6c60:::converted:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x55ad9fb94400 con 0x55ad9fb95550 2023-10-05T19:28:27.704 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.698+0000 7f8060b6c700 1 -- 172.21.15.167:0/355163605 <== osd.3 v2:172.21.15.167:6808/30036 10 ==== osd_op_reply(41 converted [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 ==== 153+0+0 (crc 0 0 0) 0x7f8050016070 con 0x55ad9fb95550 2023-10-05T19:28:27.704 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.698+0000 7f8063037e40 20 rados_obj.operate() r=-2 bl.length=0 2023-10-05T19:28:27.704 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.698+0000 7f8063037e40 20 rados->read ofs=0 len=0 2023-10-05T19:28:27.704 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.698+0000 7f8063037e40 1 -- 172.21.15.167:0/355163605 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:42 5.12 5:49953fa1:::default.realm:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x55ad9fb983e0 con 0x55ad9fb5e7f0 2023-10-05T19:28:27.705 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.698+0000 7f806136d700 1 -- 172.21.15.167:0/355163605 <== osd.1 v2:172.21.15.167:6800/30035 16 ==== osd_op_reply(42 default.realm [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 ==== 157+0+0 (crc 0 0 0) 0x7f805800a5a0 con 0x55ad9fb5e7f0 2023-10-05T19:28:27.705 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.698+0000 7f8063037e40 20 rados_obj.operate() r=-2 bl.length=0 2023-10-05T19:28:27.705 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.698+0000 7f8063037e40 20 rados->read ofs=0 len=0 2023-10-05T19:28:27.705 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.698+0000 7f8063037e40 1 -- 172.21.15.167:0/355163605 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:43 5.f 5:f4c53578:::zonegroups_names.default:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x55ad9fb987e0 con 0x55ad9fb5e7f0 2023-10-05T19:28:27.706 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.698+0000 7f806136d700 1 -- 172.21.15.167:0/355163605 <== osd.1 v2:172.21.15.167:6800/30035 17 ==== osd_op_reply(43 zonegroups_names.default [read 0~46 out=46b] v0'0 uv2 ondisk = 0) v8 ==== 168+0+46 (crc 0 0 0) 0x7f805800a5a0 con 0x55ad9fb5e7f0 2023-10-05T19:28:27.706 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.698+0000 7f8063037e40 20 rados_obj.operate() r=0 bl.length=46 2023-10-05T19:28:27.706 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.698+0000 7f8063037e40 1 -- 172.21.15.167:0/355163605 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:44 5.0 5:00000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x55ad9fb997f0 con 0x55ad9fb95550 2023-10-05T19:28:27.708 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.702+0000 7f8060b6c700 1 -- 172.21.15.167:0/355163605 <== osd.3 v2:172.21.15.167:6808/30036 11 ==== osd_op_reply(44 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f8050016070 con 0x55ad9fb95550 2023-10-05T19:28:27.708 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.702+0000 7f8060b6c700 1 -- 172.21.15.167:0/355163605 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:45 5.10 5:08000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x55ad9fb997f0 con 0x7f805001a070 2023-10-05T19:28:27.709 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.702+0000 7f806036b700 1 -- 172.21.15.167:0/355163605 <== osd.2 v2:172.21.15.167:6816/30037 7 ==== osd_op_reply(45 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f8054012450 con 0x7f805001a070 2023-10-05T19:28:27.709 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.702+0000 7f806036b700 1 -- 172.21.15.167:0/355163605 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:46 5.8 5:10000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x55ad9fb997f0 con 0x7f8054007030 2023-10-05T19:28:27.710 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.706+0000 7f806136d700 1 -- 172.21.15.167:0/355163605 <== osd.0 v2:172.21.15.118:6800/27690 11 ==== osd_op_reply(46 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f805800a5a0 con 0x7f8054007030 2023-10-05T19:28:27.710 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.706+0000 7f806136d700 1 -- 172.21.15.167:0/355163605 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:47 5.18 5:18000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x55ad9fb94000 con 0x55ad9fb5e7f0 2023-10-05T19:28:27.711 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.706+0000 7f806136d700 1 -- 172.21.15.167:0/355163605 <== osd.1 v2:172.21.15.167:6800/30035 18 ==== osd_op_reply(47 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f805800a5a0 con 0x55ad9fb5e7f0 2023-10-05T19:28:27.711 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.706+0000 7f806136d700 1 -- 172.21.15.167:0/355163605 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:48 5.4 5:20000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x55ad9fb94000 con 0x55ad9fb5e7f0 2023-10-05T19:28:27.712 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.706+0000 7f806136d700 1 -- 172.21.15.167:0/355163605 <== osd.1 v2:172.21.15.167:6800/30035 19 ==== osd_op_reply(48 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f805800a5a0 con 0x55ad9fb5e7f0 2023-10-05T19:28:27.712 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.706+0000 7f806136d700 1 -- 172.21.15.167:0/355163605 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:49 5.14 5:28000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x55ad9fb94000 con 0x7f8054007030 2023-10-05T19:28:27.713 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.706+0000 7f806136d700 1 -- 172.21.15.167:0/355163605 <== osd.0 v2:172.21.15.118:6800/27690 12 ==== osd_op_reply(49 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f805800a5a0 con 0x7f8054007030 2023-10-05T19:28:27.713 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.706+0000 7f806136d700 1 -- 172.21.15.167:0/355163605 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:50 5.c 5:30000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x55ad9fb94000 con 0x7f8054007030 2023-10-05T19:28:27.714 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.710+0000 7f806136d700 1 -- 172.21.15.167:0/355163605 <== osd.0 v2:172.21.15.118:6800/27690 13 ==== osd_op_reply(50 [pgnls start_epoch 30 out=112b] v23'1 uv1 ondisk = 1) v8 ==== 144+0+112 (crc 0 0 0) 0x7f805800a5a0 con 0x7f8054007030 2023-10-05T19:28:27.714 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.710+0000 7f806136d700 1 -- 172.21.15.167:0/355163605 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:51 5.1c 5:38000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x55ad9fb94000 con 0x7f805001a070 2023-10-05T19:28:27.715 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.710+0000 7f806036b700 1 -- 172.21.15.167:0/355163605 <== osd.2 v2:172.21.15.167:6816/30037 8 ==== osd_op_reply(51 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f8054012450 con 0x7f805001a070 2023-10-05T19:28:27.715 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.710+0000 7f806036b700 1 -- 172.21.15.167:0/355163605 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:52 5.2 5:40000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x55ad9fb94000 con 0x55ad9fb95550 2023-10-05T19:28:27.716 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.710+0000 7f8060b6c700 1 -- 172.21.15.167:0/355163605 <== osd.3 v2:172.21.15.167:6808/30036 12 ==== osd_op_reply(52 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f8050016070 con 0x55ad9fb95550 2023-10-05T19:28:27.716 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.710+0000 7f8060b6c700 1 -- 172.21.15.167:0/355163605 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:53 5.12 5:48000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x55ad9fb94000 con 0x55ad9fb5e7f0 2023-10-05T19:28:27.717 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.710+0000 7f806136d700 1 -- 172.21.15.167:0/355163605 <== osd.1 v2:172.21.15.167:6800/30035 20 ==== osd_op_reply(53 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f805800a5a0 con 0x55ad9fb5e7f0 2023-10-05T19:28:27.717 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.710+0000 7f806136d700 1 -- 172.21.15.167:0/355163605 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:54 5.a 5:50000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x55ad9fb94000 con 0x7f8054007030 2023-10-05T19:28:27.718 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.710+0000 7f806136d700 1 -- 172.21.15.167:0/355163605 <== osd.0 v2:172.21.15.118:6800/27690 14 ==== osd_op_reply(54 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f805800a5a0 con 0x7f8054007030 2023-10-05T19:28:27.718 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.710+0000 7f806136d700 1 -- 172.21.15.167:0/355163605 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:55 5.1a 5:58000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x55ad9fb94000 con 0x55ad9fb5e7f0 2023-10-05T19:28:27.719 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.714+0000 7f806136d700 1 -- 172.21.15.167:0/355163605 <== osd.1 v2:172.21.15.167:6800/30035 21 ==== osd_op_reply(55 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f805800a5a0 con 0x55ad9fb5e7f0 2023-10-05T19:28:27.719 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.714+0000 7f806136d700 1 -- 172.21.15.167:0/355163605 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:56 5.6 5:60000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x55ad9fb94000 con 0x7f805001a070 2023-10-05T19:28:27.720 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.714+0000 7f806036b700 1 -- 172.21.15.167:0/355163605 <== osd.2 v2:172.21.15.167:6816/30037 9 ==== osd_op_reply(56 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f8054012450 con 0x7f805001a070 2023-10-05T19:28:27.720 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.714+0000 7f806036b700 1 -- 172.21.15.167:0/355163605 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:57 5.16 5:68000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x55ad9fb94000 con 0x55ad9fb95550 2023-10-05T19:28:27.720 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.714+0000 7f8060b6c700 1 -- 172.21.15.167:0/355163605 <== osd.3 v2:172.21.15.167:6808/30036 13 ==== osd_op_reply(57 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f8050016070 con 0x55ad9fb95550 2023-10-05T19:28:27.720 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.714+0000 7f8060b6c700 1 -- 172.21.15.167:0/355163605 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:58 5.e 5:70000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x55ad9fb94000 con 0x55ad9fb95550 2023-10-05T19:28:27.721 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.714+0000 7f8060b6c700 1 -- 172.21.15.167:0/355163605 <== osd.3 v2:172.21.15.167:6808/30036 14 ==== osd_op_reply(58 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f8050016070 con 0x55ad9fb95550 2023-10-05T19:28:27.721 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.714+0000 7f8060b6c700 1 -- 172.21.15.167:0/355163605 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:59 5.1e 5:78000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x55ad9fb94000 con 0x7f8054007030 2023-10-05T19:28:27.722 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.718+0000 7f806136d700 1 -- 172.21.15.167:0/355163605 <== osd.0 v2:172.21.15.118:6800/27690 15 ==== osd_op_reply(59 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f805803f040 con 0x7f8054007030 2023-10-05T19:28:27.722 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.718+0000 7f806136d700 1 -- 172.21.15.167:0/355163605 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:60 5.1 5:80000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x55ad9fb94000 con 0x55ad9fb95550 2023-10-05T19:28:27.723 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.718+0000 7f8060b6c700 1 -- 172.21.15.167:0/355163605 <== osd.3 v2:172.21.15.167:6808/30036 15 ==== osd_op_reply(60 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f8050016070 con 0x55ad9fb95550 2023-10-05T19:28:27.723 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.718+0000 7f8060b6c700 1 -- 172.21.15.167:0/355163605 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:61 5.11 5:88000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x55ad9fb94000 con 0x55ad9fb95550 2023-10-05T19:28:27.723 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.718+0000 7f8060b6c700 1 -- 172.21.15.167:0/355163605 <== osd.3 v2:172.21.15.167:6808/30036 16 ==== osd_op_reply(61 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f8050016070 con 0x55ad9fb95550 2023-10-05T19:28:27.723 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.718+0000 7f8060b6c700 1 -- 172.21.15.167:0/355163605 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:62 5.9 5:90000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x55ad9fb94000 con 0x55ad9fb5e7f0 2023-10-05T19:28:27.724 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.718+0000 7f806136d700 1 -- 172.21.15.167:0/355163605 <== osd.1 v2:172.21.15.167:6800/30035 22 ==== osd_op_reply(62 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f805803f040 con 0x55ad9fb5e7f0 2023-10-05T19:28:27.724 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.718+0000 7f806136d700 1 -- 172.21.15.167:0/355163605 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:63 5.19 5:98000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x55ad9fb94000 con 0x55ad9fb5e7f0 2023-10-05T19:28:27.725 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.718+0000 7f806136d700 1 -- 172.21.15.167:0/355163605 <== osd.1 v2:172.21.15.167:6800/30035 23 ==== osd_op_reply(63 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f805803f040 con 0x55ad9fb5e7f0 2023-10-05T19:28:27.725 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.718+0000 7f806136d700 1 -- 172.21.15.167:0/355163605 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:64 5.5 5:a0000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x55ad9fb94000 con 0x7f8054007030 2023-10-05T19:28:27.726 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.722+0000 7f806136d700 1 -- 172.21.15.167:0/355163605 <== osd.0 v2:172.21.15.118:6800/27690 16 ==== osd_op_reply(64 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f805803f040 con 0x7f8054007030 2023-10-05T19:28:27.726 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.722+0000 7f806136d700 1 -- 172.21.15.167:0/355163605 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:65 5.15 5:a8000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x55ad9fb94000 con 0x55ad9fb95550 2023-10-05T19:28:27.727 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.722+0000 7f8060b6c700 1 -- 172.21.15.167:0/355163605 <== osd.3 v2:172.21.15.167:6808/30036 17 ==== osd_op_reply(65 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f8050016070 con 0x55ad9fb95550 2023-10-05T19:28:27.727 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.722+0000 7f8060b6c700 1 -- 172.21.15.167:0/355163605 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:66 5.d 5:b0000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x55ad9fb94000 con 0x7f805001a070 2023-10-05T19:28:27.728 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.722+0000 7f806036b700 1 -- 172.21.15.167:0/355163605 <== osd.2 v2:172.21.15.167:6816/30037 10 ==== osd_op_reply(66 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f8054012450 con 0x7f805001a070 2023-10-05T19:28:27.728 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.722+0000 7f806036b700 1 -- 172.21.15.167:0/355163605 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:67 5.1d 5:b8000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x55ad9fb94000 con 0x55ad9fb5e7f0 2023-10-05T19:28:27.728 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.722+0000 7f806136d700 1 -- 172.21.15.167:0/355163605 <== osd.1 v2:172.21.15.167:6800/30035 24 ==== osd_op_reply(67 [pgnls start_epoch 30 out=107b] v30'2 uv2 ondisk = 1) v8 ==== 144+0+107 (crc 0 0 0) 0x7f805803f040 con 0x55ad9fb5e7f0 2023-10-05T19:28:27.729 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.722+0000 7f806136d700 1 -- 172.21.15.167:0/355163605 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:68 5.3 5:c0000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x55ad9fb94000 con 0x7f8054007030 2023-10-05T19:28:27.729 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.722+0000 7f806136d700 1 -- 172.21.15.167:0/355163605 <== osd.0 v2:172.21.15.118:6800/27690 17 ==== osd_op_reply(68 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f805803f040 con 0x7f8054007030 2023-10-05T19:28:27.730 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.722+0000 7f806136d700 1 -- 172.21.15.167:0/355163605 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:69 5.13 5:c8000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x55ad9fb94000 con 0x55ad9fb95550 2023-10-05T19:28:27.730 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.726+0000 7f8060b6c700 1 -- 172.21.15.167:0/355163605 <== osd.3 v2:172.21.15.167:6808/30036 18 ==== osd_op_reply(69 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f8050016070 con 0x55ad9fb95550 2023-10-05T19:28:27.730 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.726+0000 7f8060b6c700 1 -- 172.21.15.167:0/355163605 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:70 5.b 5:d0000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x55ad9fb94000 con 0x7f8054007030 2023-10-05T19:28:27.731 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.726+0000 7f806136d700 1 -- 172.21.15.167:0/355163605 <== osd.0 v2:172.21.15.118:6800/27690 18 ==== osd_op_reply(70 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f805803f040 con 0x7f8054007030 2023-10-05T19:28:27.731 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.726+0000 7f806136d700 1 -- 172.21.15.167:0/355163605 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:71 5.1b 5:d8000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x55ad9fb94000 con 0x7f805001a070 2023-10-05T19:28:27.732 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.726+0000 7f806036b700 1 -- 172.21.15.167:0/355163605 <== osd.2 v2:172.21.15.167:6816/30037 11 ==== osd_op_reply(71 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f8054012450 con 0x7f805001a070 2023-10-05T19:28:27.732 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.726+0000 7f806036b700 1 -- 172.21.15.167:0/355163605 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:72 5.7 5:e0000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x55ad9fb94000 con 0x55ad9fb95550 2023-10-05T19:28:27.733 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.726+0000 7f8060b6c700 1 -- 172.21.15.167:0/355163605 <== osd.3 v2:172.21.15.167:6808/30036 19 ==== osd_op_reply(72 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f8050016070 con 0x55ad9fb95550 2023-10-05T19:28:27.733 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.726+0000 7f8060b6c700 1 -- 172.21.15.167:0/355163605 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:73 5.17 5:e8000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x55ad9fb94000 con 0x7f8054007030 2023-10-05T19:28:27.734 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.730+0000 7f806136d700 1 -- 172.21.15.167:0/355163605 <== osd.0 v2:172.21.15.118:6800/27690 19 ==== osd_op_reply(73 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f805803f040 con 0x7f8054007030 2023-10-05T19:28:27.734 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.730+0000 7f806136d700 1 -- 172.21.15.167:0/355163605 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:74 5.f 5:f0000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x55ad9fb94000 con 0x55ad9fb5e7f0 2023-10-05T19:28:27.735 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.730+0000 7f806136d700 1 -- 172.21.15.167:0/355163605 <== osd.1 v2:172.21.15.167:6800/30035 25 ==== osd_op_reply(74 [pgnls start_epoch 30 out=115b] v23'2 uv2 ondisk = 1) v8 ==== 144+0+115 (crc 0 0 0) 0x7f805803f040 con 0x55ad9fb5e7f0 2023-10-05T19:28:27.735 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.730+0000 7f806136d700 1 -- 172.21.15.167:0/355163605 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:75 5.1f 5:f8000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x55ad9fb94000 con 0x7f805001a070 2023-10-05T19:28:27.736 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.730+0000 7f806036b700 1 -- 172.21.15.167:0/355163605 <== osd.2 v2:172.21.15.167:6816/30037 12 ==== osd_op_reply(75 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f8054012450 con 0x7f805001a070 2023-10-05T19:28:27.736 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.730+0000 7f8063037e40 20 RGWRados::pool_iterate: got zonegroup_info.4f1f719d-9648-417b-b10c-e5d939825419 2023-10-05T19:28:27.736 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.730+0000 7f8063037e40 20 RGWRados::pool_iterate: got zone_info.9a9df965-6612-45d4-a45f-6e3c889bacc2 2023-10-05T19:28:27.736 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.730+0000 7f8063037e40 20 RGWRados::pool_iterate: got zone_names.default 2023-10-05T19:28:27.737 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.730+0000 7f8063037e40 20 RGWRados::pool_iterate: got zonegroups_names.default 2023-10-05T19:28:27.737 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.730+0000 7f8063037e40 20 rados->read ofs=0 len=0 2023-10-05T19:28:27.737 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.730+0000 7f8063037e40 1 -- 172.21.15.167:0/355163605 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:76 5.f 5:f43b6ece:::zone_names.default:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x55ad9fb987e0 con 0x55ad9fb5e7f0 2023-10-05T19:28:27.738 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.730+0000 7f806136d700 1 -- 172.21.15.167:0/355163605 <== osd.1 v2:172.21.15.167:6800/30035 26 ==== osd_op_reply(76 zone_names.default [read 0~46 out=46b] v0'0 uv1 ondisk = 0) v8 ==== 162+0+46 (crc 0 0 0) 0x7f805803f040 con 0x55ad9fb5e7f0 2023-10-05T19:28:27.738 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.730+0000 7f8063037e40 20 rados_obj.operate() r=0 bl.length=46 2023-10-05T19:28:27.738 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.730+0000 7f8063037e40 20 rados->read ofs=0 len=0 2023-10-05T19:28:27.738 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.730+0000 7f8063037e40 1 -- 172.21.15.167:0/355163605 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:77 5.1d 5:b8374eb4:::zone_info.9a9df965-6612-45d4-a45f-6e3c889bacc2:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x55ad9fb98b90 con 0x55ad9fb5e7f0 2023-10-05T19:28:27.738 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.730+0000 7f806136d700 1 -- 172.21.15.167:0/355163605 <== osd.1 v2:172.21.15.167:6800/30035 27 ==== osd_op_reply(77 zone_info.9a9df965-6612-45d4-a45f-6e3c889bacc2 [read 0~874 out=874b] v0'0 uv2 ondisk = 0) v8 ==== 190+0+874 (crc 0 0 0) 0x7f805803f040 con 0x55ad9fb5e7f0 2023-10-05T19:28:27.738 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.730+0000 7f8063037e40 20 rados_obj.operate() r=0 bl.length=874 2023-10-05T19:28:27.738 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.730+0000 7f8063037e40 20 rados->read ofs=0 len=0 2023-10-05T19:28:27.738 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.730+0000 7f8063037e40 1 -- 172.21.15.167:0/355163605 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:78 5.f 5:f4c53578:::zonegroups_names.default:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x55ad9fb98b90 con 0x55ad9fb5e7f0 2023-10-05T19:28:27.739 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.734+0000 7f806136d700 1 -- 172.21.15.167:0/355163605 <== osd.1 v2:172.21.15.167:6800/30035 28 ==== osd_op_reply(78 zonegroups_names.default [read 0~46 out=46b] v0'0 uv2 ondisk = 0) v8 ==== 168+0+46 (crc 0 0 0) 0x7f805803f040 con 0x55ad9fb5e7f0 2023-10-05T19:28:27.739 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.734+0000 7f8063037e40 20 rados_obj.operate() r=0 bl.length=46 2023-10-05T19:28:27.739 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.734+0000 7f8063037e40 20 rados->read ofs=0 len=0 2023-10-05T19:28:27.739 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.734+0000 7f8063037e40 1 -- 172.21.15.167:0/355163605 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:79 5.c 5:30d299a5:::zonegroup_info.4f1f719d-9648-417b-b10c-e5d939825419:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x55ad9fb98b90 con 0x7f8054007030 2023-10-05T19:28:27.742 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.734+0000 7f806136d700 1 -- 172.21.15.167:0/355163605 <== osd.0 v2:172.21.15.118:6800/27690 20 ==== osd_op_reply(79 zonegroup_info.4f1f719d-9648-417b-b10c-e5d939825419 [read 0~362 out=362b] v0'0 uv1 ondisk = 0) v8 ==== 195+0+362 (crc 0 0 0) 0x7f805803f040 con 0x7f8054007030 2023-10-05T19:28:27.742 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.734+0000 7f8063037e40 20 rados_obj.operate() r=0 bl.length=362 2023-10-05T19:28:27.742 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.734+0000 7f8063037e40 20 rados->read ofs=0 len=0 2023-10-05T19:28:27.742 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.734+0000 7f8063037e40 1 -- 172.21.15.167:0/355163605 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:80 5.3 5:c0ad8cbd:::region_map:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x55ad9fb99020 con 0x7f8054007030 2023-10-05T19:28:27.742 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.734+0000 7f806136d700 1 -- 172.21.15.167:0/355163605 <== osd.0 v2:172.21.15.118:6800/27690 21 ==== osd_op_reply(80 region_map [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 ==== 154+0+0 (crc 0 0 0) 0x7f805803f040 con 0x7f8054007030 2023-10-05T19:28:27.742 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.734+0000 7f8063037e40 20 rados_obj.operate() r=-2 bl.length=0 2023-10-05T19:28:27.743 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.734+0000 7f8063037e40 20 zone default found 2023-10-05T19:28:27.743 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.734+0000 7f8063037e40 4 Realm: () 2023-10-05T19:28:27.743 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.734+0000 7f8063037e40 4 ZoneGroup: default (4f1f719d-9648-417b-b10c-e5d939825419) 2023-10-05T19:28:27.743 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.734+0000 7f8063037e40 4 Zone: default (9a9df965-6612-45d4-a45f-6e3c889bacc2) 2023-10-05T19:28:27.743 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.734+0000 7f8063037e40 10 cannot find current period zonegroup using local zonegroup configuration 2023-10-05T19:28:27.743 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.734+0000 7f8063037e40 20 zonegroup default 2023-10-05T19:28:27.743 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.734+0000 7f8063037e40 20 rados->read ofs=0 len=0 2023-10-05T19:28:27.743 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.734+0000 7f8063037e40 1 -- 172.21.15.167:0/355163605 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:81 5.3 5:c52100b6:::period_config.default:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x55ad9fb99020 con 0x7f8054007030 2023-10-05T19:28:27.744 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.738+0000 7f806136d700 1 -- 172.21.15.167:0/355163605 <== osd.0 v2:172.21.15.118:6800/27690 22 ==== osd_op_reply(81 period_config.default [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 ==== 165+0+0 (crc 0 0 0) 0x7f805803f040 con 0x7f8054007030 2023-10-05T19:28:27.744 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.738+0000 7f8063037e40 20 rados_obj.operate() r=-2 bl.length=0 2023-10-05T19:28:27.744 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.738+0000 7f8063037e40 20 rados->read ofs=0 len=0 2023-10-05T19:28:27.744 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.738+0000 7f8063037e40 1 -- 172.21.15.167:0/355163605 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:82 6.2 6:4347d321:::bucket.sync-source-hints.:head [call version.read in=11b,read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x55ad9fb9a730 con 0x55ad9fb95550 2023-10-05T19:28:27.745 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.738+0000 7f8060b6c700 1 -- 172.21.15.167:0/355163605 <== osd.3 v2:172.21.15.167:6808/30036 20 ==== osd_op_reply(82 bucket.sync-source-hints. [call,read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 ==== 211+0+0 (crc 0 0 0) 0x7f8050016070 con 0x55ad9fb95550 2023-10-05T19:28:27.745 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.738+0000 7f8063037e40 20 rados_obj.operate() r=-2 bl.length=0 2023-10-05T19:28:27.745 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.738+0000 7f8063037e40 20 rados->read ofs=0 len=0 2023-10-05T19:28:27.745 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.738+0000 7f8063037e40 1 -- 172.21.15.167:0/355163605 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:83 6.b 6:d467b91b:::bucket.sync-target-hints.:head [call version.read in=11b,read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x55ad9fb9aae0 con 0x55ad9fb95550 2023-10-05T19:28:27.746 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.738+0000 7f8060b6c700 1 -- 172.21.15.167:0/355163605 <== osd.3 v2:172.21.15.167:6808/30036 21 ==== osd_op_reply(83 bucket.sync-target-hints. [call,read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 ==== 211+0+0 (crc 0 0 0) 0x7f8050016070 con 0x55ad9fb95550 2023-10-05T19:28:27.746 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.738+0000 7f8063037e40 20 rados_obj.operate() r=-2 bl.length=0 2023-10-05T19:28:27.746 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.738+0000 7f8063037e40 20 started sync module instance, tier type = 2023-10-05T19:28:27.746 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.738+0000 7f8063037e40 20 started zone id=9a9df965-6612-45d4-a45f-6e3c889bacc2 (name=default) with tier type = 2023-10-05T19:28:27.746 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.738+0000 7f8063037e40 1 -- 172.21.15.167:0/355163605 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:84 7.1f 7:f95f44c2:::notify.0:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x55ad9fb99df0 con 0x7f805001a070 2023-10-05T19:28:27.748 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.742+0000 7f806036b700 1 -- 172.21.15.167:0/355163605 <== osd.2 v2:172.21.15.167:6816/30037 13 ==== osd_op_reply(84 notify.0 [create] v30'51 uv51 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f8054012450 con 0x7f805001a070 2023-10-05T19:28:27.748 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.742+0000 7f8063037e40 1 -- 172.21.15.167:0/355163605 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:85 7.1f 7:f95f44c2:::notify.0:head [watch watch cookie 94204197446048] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x55ad9fb93c00 con 0x7f805001a070 2023-10-05T19:28:27.748 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.742+0000 7f8063037e40 1 -- 172.21.15.167:0/355163605 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:86 7.0 7:05bf5b68:::notify.1:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x55ad9fb9b940 con 0x55ad9fb5e7f0 2023-10-05T19:28:27.751 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.746+0000 7f806036b700 1 -- 172.21.15.167:0/355163605 <== osd.2 v2:172.21.15.167:6816/30037 14 ==== osd_op_reply(85 notify.0 [watch watch cookie 94204197446048] v30'52 uv51 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f8054012450 con 0x7f805001a070 2023-10-05T19:28:27.751 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.746+0000 7f806136d700 1 -- 172.21.15.167:0/355163605 <== osd.1 v2:172.21.15.167:6800/30035 29 ==== osd_op_reply(86 notify.1 [create] v30'51 uv51 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f8058052040 con 0x55ad9fb5e7f0 2023-10-05T19:28:27.751 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.746+0000 7f8063037e40 1 -- 172.21.15.167:0/355163605 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:87 7.0 7:05bf5b68:::notify.1:head [watch watch cookie 94204197449360] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x55ad9fb9bd40 con 0x55ad9fb5e7f0 2023-10-05T19:28:27.751 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.746+0000 7f8063037e40 1 -- 172.21.15.167:0/355163605 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:88 7.15 7:a93a5511:::notify.2:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x55ad9fb9c5e0 con 0x55ad9fb95550 2023-10-05T19:28:27.754 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.750+0000 7f806136d700 1 -- 172.21.15.167:0/355163605 <== osd.1 v2:172.21.15.167:6800/30035 30 ==== osd_op_reply(87 notify.1 [watch watch cookie 94204197449360] v30'52 uv51 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f8058052040 con 0x55ad9fb5e7f0 2023-10-05T19:28:27.755 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.750+0000 7f8060b6c700 1 -- 172.21.15.167:0/355163605 <== osd.3 v2:172.21.15.167:6808/30036 22 ==== osd_op_reply(88 notify.2 [create] v30'51 uv51 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f8050016070 con 0x55ad9fb95550 2023-10-05T19:28:27.755 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.750+0000 7f8063037e40 1 -- 172.21.15.167:0/355163605 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:89 7.15 7:a93a5511:::notify.2:head [watch watch cookie 94204197450768] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x55ad9fb9c230 con 0x55ad9fb95550 2023-10-05T19:28:27.755 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.750+0000 7f8063037e40 1 -- 172.21.15.167:0/355163605 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:90 7.e 7:7759931f:::notify.3:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x55ad9fb9d0f0 con 0x7f805001a070 2023-10-05T19:28:27.758 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.750+0000 7f806036b700 1 -- 172.21.15.167:0/355163605 <== osd.2 v2:172.21.15.167:6816/30037 15 ==== osd_op_reply(90 notify.3 [create] v30'51 uv51 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f8054012450 con 0x7f805001a070 2023-10-05T19:28:27.758 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.750+0000 7f8060b6c700 1 -- 172.21.15.167:0/355163605 <== osd.3 v2:172.21.15.167:6808/30036 23 ==== osd_op_reply(89 notify.2 [watch watch cookie 94204197450768] v30'52 uv51 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f8050016070 con 0x55ad9fb95550 2023-10-05T19:28:27.758 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.750+0000 7f8063037e40 1 -- 172.21.15.167:0/355163605 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:91 7.e 7:7759931f:::notify.3:head [watch watch cookie 94204197459184] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x55ad9fb9de30 con 0x7f805001a070 2023-10-05T19:28:27.758 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.750+0000 7f8063037e40 1 -- 172.21.15.167:0/355163605 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:92 7.d 7:b4812045:::notify.4:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x55ad9fb9e1e0 con 0x55ad9fb5e7f0 2023-10-05T19:28:27.760 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.754+0000 7f806036b700 1 -- 172.21.15.167:0/355163605 <== osd.2 v2:172.21.15.167:6816/30037 16 ==== osd_op_reply(91 notify.3 [watch watch cookie 94204197459184] v30'52 uv51 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f8054012450 con 0x7f805001a070 2023-10-05T19:28:27.760 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.754+0000 7f806136d700 1 -- 172.21.15.167:0/355163605 <== osd.1 v2:172.21.15.167:6800/30035 31 ==== osd_op_reply(92 notify.4 [create] v30'51 uv51 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f8058052040 con 0x55ad9fb5e7f0 2023-10-05T19:28:27.760 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.754+0000 7f8063037e40 1 -- 172.21.15.167:0/355163605 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:93 7.d 7:b4812045:::notify.4:head [watch watch cookie 94204197460288] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x55ad9fb9e5e0 con 0x55ad9fb5e7f0 2023-10-05T19:28:27.761 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.754+0000 7f8063037e40 1 -- 172.21.15.167:0/355163605 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:94 7.3 7:c609908c:::notify.5:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x55ad9fb9eff0 con 0x7f8054007030 2023-10-05T19:28:27.763 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.758+0000 7f806136d700 1 -- 172.21.15.167:0/355163605 <== osd.1 v2:172.21.15.167:6800/30035 32 ==== osd_op_reply(93 notify.4 [watch watch cookie 94204197460288] v30'52 uv51 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f8058052040 con 0x55ad9fb5e7f0 2023-10-05T19:28:27.764 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.758+0000 7f806136d700 1 -- 172.21.15.167:0/355163605 <== osd.0 v2:172.21.15.118:6800/27690 23 ==== osd_op_reply(94 notify.5 [create] v30'51 uv51 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f8058052040 con 0x7f8054007030 2023-10-05T19:28:27.764 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.758+0000 7f8063037e40 1 -- 172.21.15.167:0/355163605 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:95 7.3 7:c609908c:::notify.5:head [watch watch cookie 94204197453040] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x55ad9fb9eff0 con 0x7f8054007030 2023-10-05T19:28:27.764 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.758+0000 7f8063037e40 1 -- 172.21.15.167:0/355163605 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:96 7.14 7:2b04a3e9:::notify.6:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x55ad9fb9fa00 con 0x7f8054007030 2023-10-05T19:28:27.766 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.762+0000 7f806136d700 1 -- 172.21.15.167:0/355163605 <== osd.0 v2:172.21.15.118:6800/27690 24 ==== osd_op_reply(96 notify.6 [create] v30'51 uv51 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f8058052040 con 0x7f8054007030 2023-10-05T19:28:27.767 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.762+0000 7f8063037e40 1 -- 172.21.15.167:0/355163605 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:97 7.14 7:2b04a3e9:::notify.6:head [watch watch cookie 94204197467408] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x55ad9fb9fe50 con 0x7f8054007030 2023-10-05T19:28:27.767 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.762+0000 7f8063037e40 1 -- 172.21.15.167:0/355163605 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:98 7.9 7:93e5b521:::notify.7:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x55ad9fba0860 con 0x7f8054007030 2023-10-05T19:28:27.767 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.762+0000 7f806136d700 1 -- 172.21.15.167:0/355163605 <== osd.0 v2:172.21.15.118:6800/27690 25 ==== osd_op_reply(95 notify.5 [watch watch cookie 94204197453040] v30'52 uv51 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f8058052040 con 0x7f8054007030 2023-10-05T19:28:27.770 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.762+0000 7f806136d700 1 -- 172.21.15.167:0/355163605 <== osd.0 v2:172.21.15.118:6800/27690 26 ==== osd_op_reply(98 notify.7 [create] v30'51 uv51 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f8058052040 con 0x7f8054007030 2023-10-05T19:28:27.770 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.762+0000 7f8063037e40 1 -- 172.21.15.167:0/355163605 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:99 7.9 7:93e5b521:::notify.7:head [watch watch cookie 94204197464832] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x55ad9fba0860 con 0x7f8054007030 2023-10-05T19:28:27.770 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.762+0000 7f8063037e40 20 add_watcher() i=0 2023-10-05T19:28:27.770 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.762+0000 7f8063037e40 20 add_watcher() i=1 2023-10-05T19:28:27.770 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.762+0000 7f8063037e40 20 add_watcher() i=2 2023-10-05T19:28:27.770 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.762+0000 7f8063037e40 20 add_watcher() i=3 2023-10-05T19:28:27.770 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.762+0000 7f8063037e40 20 add_watcher() i=4 2023-10-05T19:28:27.770 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.762+0000 7f8063037e40 20 add_watcher() i=5 2023-10-05T19:28:27.771 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.766+0000 7f806136d700 1 -- 172.21.15.167:0/355163605 <== osd.0 v2:172.21.15.118:6800/27690 27 ==== osd_op_reply(97 notify.6 [watch watch cookie 94204197467408] v30'52 uv51 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f8058052040 con 0x7f8054007030 2023-10-05T19:28:27.771 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.766+0000 7f8063037e40 20 add_watcher() i=6 2023-10-05T19:28:27.772 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.766+0000 7f806136d700 1 -- 172.21.15.167:0/355163605 <== osd.0 v2:172.21.15.118:6800/27690 28 ==== osd_op_reply(99 notify.7 [watch watch cookie 94204197464832] v30'52 uv51 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f8058052040 con 0x7f8054007030 2023-10-05T19:28:27.772 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.766+0000 7f8063037e40 20 add_watcher() i=7 2023-10-05T19:28:27.772 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.766+0000 7f8063037e40 2 all 8 watchers are set, enabling cache 2023-10-05T19:28:27.772 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.766+0000 7f8063037e40 1 -- 172.21.15.167:0/355163605 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:100 6.4 6:22d26bf9:::data_loggenerations_metadata:head [call version.check_conds in=74b,call version.read in=11b,read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x55ad9fba3890 con 0x55ad9fb5e7f0 2023-10-05T19:28:27.772 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.766+0000 7f806136d700 1 -- 172.21.15.167:0/355163605 <== osd.1 v2:172.21.15.167:6800/30035 33 ==== osd_op_reply(100 data_loggenerations_metadata [call,call out=48b,read 0~28 out=28b] v0'0 uv1 ondisk = 0) v8 ==== 256+0+76 (crc 0 0 0) 0x7f8058052040 con 0x55ad9fb5e7f0 2023-10-05T19:28:27.773 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.766+0000 7f8063037e40 1 -- 172.21.15.167:0/355163605 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:101 6.4 6:22d26bf9:::data_loggenerations_metadata:head [watch watch cookie 94204197471088] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x55ad9fba0cb0 con 0x55ad9fb5e7f0 2023-10-05T19:28:27.775 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.770+0000 7f806136d700 1 -- 172.21.15.167:0/355163605 <== osd.1 v2:172.21.15.167:6800/30035 34 ==== osd_op_reply(101 data_loggenerations_metadata [watch watch cookie 94204197471088] v30'68 uv1 ondisk = 0) v8 ==== 172+0+0 (crc 0 0 0) 0x7f8058052040 con 0x55ad9fb5e7f0 2023-10-05T19:28:27.775 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.770+0000 7f80327e4700 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2023-10-05T19:28:27.775 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.770+0000 7f8063037e40 20 check_secure_mon_conn(): auth registy supported: methods=[2] modes=[2,1] 2023-10-05T19:28:27.775 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.770+0000 7f8063037e40 20 check_secure_mon_conn(): mode 1 is insecure 2023-10-05T19:28:27.776 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.770+0000 7f8063037e40 1 Processor -- start 2023-10-05T19:28:27.776 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.770+0000 7f8063037e40 1 -- start start 2023-10-05T19:28:27.776 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.770+0000 7f8063037e40 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55ad9fbadd50 0x55ad9fbae120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-05T19:28:27.776 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.770+0000 7f8063037e40 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ad9fbae660 con 0x55ad9fbadd50 2023-10-05T19:28:27.776 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.770+0000 7f806036b700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55ad9fbadd50 0x55ad9fbae120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-05T19:28:27.777 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.770+0000 7f806036b700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55ad9fbadd50 0x55ad9fbae120 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.118:3300/0 says I am v2:172.21.15.167:57584/0 (socket says 172.21.15.167:57584) 2023-10-05T19:28:27.777 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.770+0000 7f806036b700 1 -- 172.21.15.167:0/4029357878 learned_addr learned my addr 172.21.15.167:0/4029357878 (peer_addr_for_me v2:172.21.15.167:0/0) 2023-10-05T19:28:27.777 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.770+0000 7f806036b700 1 -- 172.21.15.167:0/4029357878 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55ad9fbae7a0 con 0x55ad9fbadd50 2023-10-05T19:28:27.777 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.770+0000 7f806036b700 1 --2- 172.21.15.167:0/4029357878 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55ad9fbadd50 0x55ad9fbae120 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7f8054002000 tx=0x7f8054012000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cfa75baa2a4be2a1 server_cookie=0 in_seq=0 out_seq=0 2023-10-05T19:28:27.778 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.770+0000 7f8031fe3700 1 -- 172.21.15.167:0/4029357878 <== mon.0 v2:172.21.15.118:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f8054012da0 con 0x55ad9fbadd50 2023-10-05T19:28:27.778 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.770+0000 7f8031fe3700 1 -- 172.21.15.167:0/4029357878 <== mon.0 v2:172.21.15.118:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f805402a8a0 con 0x55ad9fbadd50 2023-10-05T19:28:27.778 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.770+0000 7f8031fe3700 1 -- 172.21.15.167:0/4029357878 <== mon.0 v2:172.21.15.118:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f805402aa50 con 0x55ad9fbadd50 2023-10-05T19:28:27.779 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.774+0000 7f8063037e40 1 -- 172.21.15.167:0/4029357878 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55ad9fbadd50 msgr2=0x55ad9fbae120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-05T19:28:27.779 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.774+0000 7f8063037e40 1 --2- 172.21.15.167:0/4029357878 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55ad9fbadd50 0x55ad9fbae120 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7f8054002000 tx=0x7f8054012000 comp rx=0 tx=0).stop 2023-10-05T19:28:27.779 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.774+0000 7f8063037e40 1 -- 172.21.15.167:0/4029357878 shutdown_connections 2023-10-05T19:28:27.779 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.774+0000 7f8063037e40 1 --2- 172.21.15.167:0/4029357878 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55ad9fbadd50 0x55ad9fbae120 unknown :-1 s=CLOSED pgs=240 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-05T19:28:27.779 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.774+0000 7f8063037e40 1 -- 172.21.15.167:0/4029357878 >> 172.21.15.167:0/4029357878 conn(0x55ad9fba9110 msgr2=0x55ad9fba7260 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-05T19:28:27.779 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.774+0000 7f8063037e40 1 -- 172.21.15.167:0/4029357878 shutdown_connections 2023-10-05T19:28:27.780 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.774+0000 7f8063037e40 1 -- 172.21.15.167:0/4029357878 wait complete. 2023-10-05T19:28:27.780 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.774+0000 7f8063037e40 1 Processor -- start 2023-10-05T19:28:27.780 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.774+0000 7f8063037e40 1 -- start start 2023-10-05T19:28:27.780 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.774+0000 7f8063037e40 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55ad9fbadd50 0x55ad9fbb37f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-05T19:28:27.780 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.774+0000 7f8063037e40 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ad9fbb47f0 con 0x55ad9fbadd50 2023-10-05T19:28:27.781 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.774+0000 7f806036b700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55ad9fbadd50 0x55ad9fbb37f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-05T19:28:27.781 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.774+0000 7f806036b700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55ad9fbadd50 0x55ad9fbb37f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.118:3300/0 says I am v2:172.21.15.167:57590/0 (socket says 172.21.15.167:57590) 2023-10-05T19:28:27.781 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.774+0000 7f806036b700 1 -- 172.21.15.167:0/3340626072 learned_addr learned my addr 172.21.15.167:0/3340626072 (peer_addr_for_me v2:172.21.15.167:0/0) 2023-10-05T19:28:27.782 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.774+0000 7f806036b700 1 -- 172.21.15.167:0/3340626072 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55ad9fb5dce0 con 0x55ad9fbadd50 2023-10-05T19:28:27.782 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.774+0000 7f806036b700 1 --2- 172.21.15.167:0/3340626072 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55ad9fbadd50 0x55ad9fbb37f0 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7f8054008fd0 tx=0x7f805402f000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-05T19:28:27.782 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.778+0000 7f7fa3fff700 1 -- 172.21.15.167:0/3340626072 <== mon.0 v2:172.21.15.118:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f80540098c0 con 0x55ad9fbadd50 2023-10-05T19:28:27.783 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.778+0000 7f7fa3fff700 1 -- 172.21.15.167:0/3340626072 <== mon.0 v2:172.21.15.118:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8054009a20 con 0x55ad9fbadd50 2023-10-05T19:28:27.783 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.778+0000 7f8063037e40 1 -- 172.21.15.167:0/3340626072 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x55ad9fbb3df0 con 0x55ad9fbadd50 2023-10-05T19:28:27.783 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.778+0000 7f7fa3fff700 1 -- 172.21.15.167:0/3340626072 <== mon.0 v2:172.21.15.118:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f8054009bb0 con 0x55ad9fbadd50 2023-10-05T19:28:27.783 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.778+0000 7f8063037e40 1 -- 172.21.15.167:0/3340626072 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55ad9fbb4290 con 0x55ad9fbadd50 2023-10-05T19:28:27.784 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.778+0000 7f7fa3fff700 1 -- 172.21.15.167:0/3340626072 <== mon.0 v2:172.21.15.118:3300/0 4 ==== mgrmap(e 5) v1 ==== 40329+0+0 (secure 0 0 0) 0x7f8054050440 con 0x55ad9fbadd50 2023-10-05T19:28:27.784 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.778+0000 7f7fa3fff700 1 --2- 172.21.15.167:0/3340626072 >> [v2:172.21.15.167:6824/29889,v1:172.21.15.167:6825/29889] conn(0x7f7fa8030c50 0x7f7fa8033100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-05T19:28:27.784 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.778+0000 7f7fa3fff700 1 -- 172.21.15.167:0/3340626072 <== mon.0 v2:172.21.15.118:3300/0 5 ==== osd_map(30..30 src has 1..30) v4 ==== 6057+0+0 (secure 0 0 0) 0x7f805402f070 con 0x55ad9fbadd50 2023-10-05T19:28:27.785 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.778+0000 7f8063037e40 5 note: GC not initialized 2023-10-05T19:28:27.785 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.778+0000 7f806136d700 1 --2- 172.21.15.167:0/3340626072 >> [v2:172.21.15.167:6824/29889,v1:172.21.15.167:6825/29889] conn(0x7f7fa8030c50 0x7f7fa8033100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-05T19:28:27.785 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.778+0000 7f806136d700 1 --2- 172.21.15.167:0/3340626072 >> [v2:172.21.15.167:6824/29889,v1:172.21.15.167:6825/29889] conn(0x7f7fa8030c50 0x7f7fa8033100 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x55ad9fb9fba0 tx=0x7f805800a280 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-05T19:28:27.787 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.782+0000 7f7fa17fa700 20 reqs_thread_entry: start 2023-10-05T19:28:27.788 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.782+0000 7f7f9affd700 20 reqs_thread_entry: start 2023-10-05T19:28:27.788 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.782+0000 7f8063037e40 20 init_complete bucket index max shards: 11 2023-10-05T19:28:27.788 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.782+0000 7f8063037e40 10 rgw notify: Started notification manager with: 1 workers 2023-10-05T19:28:27.789 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.782+0000 7f7f997fa700 20 rgw notify: INFO: next queues processing will happen at: Thu Oct 5 19:28:58 2023 2023-10-05T19:28:27.789 INFO:teuthology.orchestra.run.smithi167.stderr: 2023-10-05T19:28:27.789 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.782+0000 7f7f98ff9700 20 reqs_thread_entry: start 2023-10-05T19:28:27.789 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.782+0000 7f8063037e40 20 get_system_obj_state: rctx=0x7ffeed21fdf8 obj=default.rgw.meta:users.uid:foo state=0x55ad9fa370b0 s->prefetch_data=0 2023-10-05T19:28:27.789 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.782+0000 7f8063037e40 10 cache get: name=default.rgw.meta+users.uid+foo : miss 2023-10-05T19:28:27.789 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.782+0000 7f8063037e40 1 -- 172.21.15.167:0/355163605 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:102 8.18 8:1e6ca5ed:users.uid::foo:head [call version.read in=11b,getxattrs,stat] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x55ad9fc1bf60 con 0x55ad9fb5e7f0 2023-10-05T19:28:27.790 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.782+0000 7f806136d700 1 -- 172.21.15.167:0/355163605 <== osd.1 v2:172.21.15.167:6800/30035 35 ==== osd_op_reply(102 foo [call out=48b,getxattrs out=75b,stat out=16b] v0'0 uv4 ondisk = 0) v8 ==== 231+0+139 (crc 0 0 0) 0x7f8058052040 con 0x55ad9fb5e7f0 2023-10-05T19:28:27.790 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.782+0000 7f8063037e40 10 cache put: name=default.rgw.meta+users.uid+foo info.flags=0x16 2023-10-05T19:28:27.790 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.782+0000 7f8063037e40 10 adding default.rgw.meta+users.uid+foo to cache LRU end 2023-10-05T19:28:27.790 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.782+0000 7f8063037e40 10 updating xattr: name=ceph.objclass.version bl.length()=42 2023-10-05T19:28:27.790 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.782+0000 7f8063037e40 20 get_system_obj_state: s->obj_tag was set empty 2023-10-05T19:28:27.790 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.782+0000 7f8063037e40 20 Read xattr: user.rgw.idtag 2023-10-05T19:28:27.790 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.782+0000 7f8063037e40 10 cache get: name=default.rgw.meta+users.uid+foo : type miss (requested=0x13, cached=0x16) 2023-10-05T19:28:27.791 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.782+0000 7f8063037e40 20 rados->read ofs=0 len=0 2023-10-05T19:28:27.791 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.782+0000 7f8063037e40 1 -- 172.21.15.167:0/355163605 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:103 8.18 8:1e6ca5ed:users.uid::foo:head [call version.check_conds in=122b,call version.read in=11b,read 0~0,getxattrs] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x55ad9fc1e9f0 con 0x55ad9fb5e7f0 2023-10-05T19:28:27.791 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.786+0000 7f806136d700 1 -- 172.21.15.167:0/355163605 <== osd.1 v2:172.21.15.167:6800/30035 36 ==== osd_op_reply(103 foo [call,call out=48b,read 0~459 out=459b,getxattrs out=75b] v0'0 uv4 ondisk = 0) v8 ==== 273+0+582 (crc 0 0 0) 0x7f8058052040 con 0x55ad9fb5e7f0 2023-10-05T19:28:27.791 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.786+0000 7f8063037e40 20 rados_obj.operate() r=0 bl.length=459 2023-10-05T19:28:27.791 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.786+0000 7f8063037e40 10 cache put: name=default.rgw.meta+users.uid+foo info.flags=0x13 2023-10-05T19:28:27.791 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.786+0000 7f8063037e40 10 moving default.rgw.meta+users.uid+foo to cache LRU end 2023-10-05T19:28:27.792 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.786+0000 7f8063037e40 10 updating xattr: name=ceph.objclass.version bl.length()=42 2023-10-05T19:28:27.792 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.786+0000 7f8063037e40 1 -- 172.21.15.167:0/355163605 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:104 8.18 8:1e6ca5ed:users.uid::foo:head [delete,create,call version.set in=58b,writefull 0~357 in=357b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x55ad9fc1b230 con 0x55ad9fb5e7f0 2023-10-05T19:28:27.793 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.786+0000 7f806136d700 1 -- 172.21.15.167:0/355163605 <== osd.1 v2:172.21.15.167:6800/30035 37 ==== osd_op_reply(104 foo [delete,create,call,writefull 0~357] v30'5 uv5 ondisk = 0) v8 ==== 273+0+0 (crc 0 0 0) 0x7f8058052040 con 0x55ad9fb5e7f0 2023-10-05T19:28:27.794 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.786+0000 7f8063037e40 10 cache put: name=default.rgw.meta+users.uid+foo info.flags=0x17 2023-10-05T19:28:27.794 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.786+0000 7f8063037e40 10 moving default.rgw.meta+users.uid+foo to cache LRU end 2023-10-05T19:28:27.794 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.786+0000 7f8063037e40 10 distributing notification oid=default.rgw.control:notify.4 cni=[op: 0, obj: default.rgw.meta:users.uid:foo, ofs0, ns] 2023-10-05T19:28:27.794 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.786+0000 7f8063037e40 1 -- 172.21.15.167:0/355163605 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:105 7.d 7:b4812045:::notify.4:head [notify cookie 94204197973312 in=545b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x55ad9fc1fa00 con 0x55ad9fb5e7f0 2023-10-05T19:28:27.795 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.790+0000 7f806136d700 1 -- 172.21.15.167:0/355163605 <== osd.1 v2:172.21.15.167:6800/30035 38 ==== watch-notify(notify (1) cookie 94204197460288 notify 128849018884 ret 0) v3 ==== 575+0+0 (crc 0 0 0) 0x7f805804e380 con 0x55ad9fb5e7f0 2023-10-05T19:28:27.795 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.790+0000 7f805eb68700 10 rgw watcher librados: RGWWatcher::handle_notify() notify_id 128849018884 cookie 94204197460288 notifier 4329 bl.length()=533 2023-10-05T19:28:27.795 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.790+0000 7f805eb68700 10 rgw watcher librados: cache put: name=default.rgw.meta+users.uid+foo info.flags=0x17 2023-10-05T19:28:27.795 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.790+0000 7f805eb68700 10 rgw watcher librados: moving default.rgw.meta+users.uid+foo to cache LRU end 2023-10-05T19:28:27.795 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.790+0000 7f805eb68700 1 -- 172.21.15.167:0/355163605 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:106 7.d 7:b4812045:::notify.4:head [notify-ack in=20b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x7f8040003d90 con 0x55ad9fb5e7f0 2023-10-05T19:28:27.795 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.790+0000 7f806136d700 1 -- 172.21.15.167:0/355163605 <== osd.1 v2:172.21.15.167:6800/30035 39 ==== osd_op_reply(105 notify.4 [notify cookie 94204197973312 out=8b] v0'0 uv51 ondisk = 0) v8 ==== 152+0+8 (crc 0 0 0) 0x7f8058052040 con 0x55ad9fb5e7f0 2023-10-05T19:28:27.796 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.790+0000 7f806136d700 1 -- 172.21.15.167:0/355163605 <== osd.1 v2:172.21.15.167:6800/30035 40 ==== osd_op_reply(106 notify.4 [notify-ack] v0'0 uv51 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f8058052040 con 0x55ad9fb5e7f0 2023-10-05T19:28:27.796 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.790+0000 7f806136d700 1 -- 172.21.15.167:0/355163605 <== osd.1 v2:172.21.15.167:6800/30035 41 ==== watch-notify(notify_complete (2) cookie 94204197973312 notify 128849018884 ret 0) v3 ==== 42+0+48 (crc 0 0 0) 0x7f8058036c80 con 0x55ad9fb5e7f0 2023-10-05T19:28:27.796 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.790+0000 7f8063037e40 10 distributing notification oid=default.rgw.control:notify.2 cni=[op: 1, obj: default.rgw.meta:users.keys:p5YnriCv1nAtykxBrupQ, ofs0, ns] 2023-10-05T19:28:27.796 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.790+0000 7f8063037e40 1 -- 172.21.15.167:0/355163605 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:107 7.15 7:a93a5511:::notify.2:head [notify cookie 94204197973312 in=182b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x55ad9fc1fa00 con 0x55ad9fb95550 2023-10-05T19:28:27.796 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.790+0000 7f8060b6c700 1 -- 172.21.15.167:0/355163605 <== osd.3 v2:172.21.15.167:6808/30036 24 ==== watch-notify(notify (1) cookie 94204197450768 notify 128849018881 ret 0) v3 ==== 212+0+0 (crc 0 0 0) 0x55ad9fb86750 con 0x55ad9fb95550 2023-10-05T19:28:27.796 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.790+0000 7f805e367700 10 rgw watcher librados: RGWWatcher::handle_notify() notify_id 128849018881 cookie 94204197450768 notifier 4329 bl.length()=170 2023-10-05T19:28:27.796 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.790+0000 7f805e367700 1 -- 172.21.15.167:0/355163605 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:108 7.15 7:a93a5511:::notify.2:head [notify-ack in=20b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x7f7fb00032c0 con 0x55ad9fb95550 2023-10-05T19:28:27.797 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.790+0000 7f8060b6c700 1 -- 172.21.15.167:0/355163605 <== osd.3 v2:172.21.15.167:6808/30036 25 ==== osd_op_reply(107 notify.2 [notify cookie 94204197973312 out=8b] v0'0 uv51 ondisk = 0) v8 ==== 152+0+8 (crc 0 0 0) 0x7f8050016070 con 0x55ad9fb95550 2023-10-05T19:28:27.797 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.790+0000 7f8060b6c700 1 -- 172.21.15.167:0/355163605 <== osd.3 v2:172.21.15.167:6808/30036 26 ==== osd_op_reply(108 notify.2 [notify-ack] v0'0 uv51 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f8050016070 con 0x55ad9fb95550 2023-10-05T19:28:27.797 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.790+0000 7f8060b6c700 1 -- 172.21.15.167:0/355163605 <== osd.3 v2:172.21.15.167:6808/30036 27 ==== watch-notify(notify_complete (2) cookie 94204197973312 notify 128849018881 ret 0) v3 ==== 42+0+48 (crc 0 0 0) 0x55ad9fb86310 con 0x55ad9fb95550 2023-10-05T19:28:27.797 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.790+0000 7f8063037e40 1 -- 172.21.15.167:0/355163605 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:109 8.13 8:cc695d75:users.keys::p5YnriCv1nAtykxBrupQ:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x55ad9fc1ad40 con 0x7f8054007030 2023-10-05T19:28:27.800 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.794+0000 7f806136d700 1 -- 172.21.15.167:0/355163605 <== osd.0 v2:172.21.15.118:6800/27690 29 ==== osd_op_reply(109 p5YnriCv1nAtykxBrupQ [delete] v30'2 uv2 ondisk = 0) v8 ==== 164+0+0 (crc 0 0 0) 0x7f8058052040 con 0x7f8054007030 2023-10-05T19:28:27.800 INFO:teuthology.orchestra.run.smithi167.stdout:{ 2023-10-05T19:28:27.800 INFO:teuthology.orchestra.run.smithi167.stdout: "user_id": "foo", 2023-10-05T19:28:27.800 INFO:teuthology.orchestra.run.smithi167.stdout: "display_name": "Foo", 2023-10-05T19:28:27.800 INFO:teuthology.orchestra.run.smithi167.stdout: "email": "foo@foo.com", 2023-10-05T19:28:27.800 INFO:teuthology.orchestra.run.smithi167.stdout: "suspended": 0, 2023-10-05T19:28:27.801 INFO:teuthology.orchestra.run.smithi167.stdout: "max_buckets": 4, 2023-10-05T19:28:27.801 INFO:teuthology.orchestra.run.smithi167.stdout: "subusers": [], 2023-10-05T19:28:27.801 INFO:teuthology.orchestra.run.smithi167.stdout: "keys": [ 2023-10-05T19:28:27.801 INFO:teuthology.orchestra.run.smithi167.stdout: { 2023-10-05T19:28:27.801 INFO:teuthology.orchestra.run.smithi167.stdout: "user": "foo", 2023-10-05T19:28:27.801 INFO:teuthology.orchestra.run.smithi167.stdout: "access_key": "9te6NH5mcdcq0Tc5i8i1", 2023-10-05T19:28:27.801 INFO:teuthology.orchestra.run.smithi167.stdout: "secret_key": "Ny4IOauQoL18Gp2zM7lC1vLmoawgqcYP/YGcWfXu" 2023-10-05T19:28:27.801 INFO:teuthology.orchestra.run.smithi167.stdout: } 2023-10-05T19:28:27.801 INFO:teuthology.orchestra.run.smithi167.stdout: ], 2023-10-05T19:28:27.801 INFO:teuthology.orchestra.run.smithi167.stdout: "swift_keys": [], 2023-10-05T19:28:27.801 INFO:teuthology.orchestra.run.smithi167.stdout: "caps": [], 2023-10-05T19:28:27.801 INFO:teuthology.orchestra.run.smithi167.stdout: "op_mask": "read, write, delete", 2023-10-05T19:28:27.802 INFO:teuthology.orchestra.run.smithi167.stdout: "default_placement": "", 2023-10-05T19:28:27.802 INFO:teuthology.orchestra.run.smithi167.stdout: "default_storage_class": "", 2023-10-05T19:28:27.802 INFO:teuthology.orchestra.run.smithi167.stdout: "placement_tags": [], 2023-10-05T19:28:27.802 INFO:teuthology.orchestra.run.smithi167.stdout: "bucket_quota": { 2023-10-05T19:28:27.802 INFO:teuthology.orchestra.run.smithi167.stdout: "enabled": false, 2023-10-05T19:28:27.802 INFO:teuthology.orchestra.run.smithi167.stdout: "check_on_raw": false, 2023-10-05T19:28:27.802 INFO:teuthology.orchestra.run.smithi167.stdout: "max_size": -1, 2023-10-05T19:28:27.802 INFO:teuthology.orchestra.run.smithi167.stdout: "max_size_kb": 0, 2023-10-05T19:28:27.802 INFO:teuthology.orchestra.run.smithi167.stdout: "max_objects": -1 2023-10-05T19:28:27.802 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2023-10-05T19:28:27.802 INFO:teuthology.orchestra.run.smithi167.stdout: "user_quota": { 2023-10-05T19:28:27.802 INFO:teuthology.orchestra.run.smithi167.stdout: "enabled": false, 2023-10-05T19:28:27.803 INFO:teuthology.orchestra.run.smithi167.stdout: "check_on_raw": false, 2023-10-05T19:28:27.803 INFO:teuthology.orchestra.run.smithi167.stdout: "max_size": -1, 2023-10-05T19:28:27.803 INFO:teuthology.orchestra.run.smithi167.stdout: "max_size_kb": 0, 2023-10-05T19:28:27.803 INFO:teuthology.orchestra.run.smithi167.stdout: "max_objects": -1 2023-10-05T19:28:27.803 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2023-10-05T19:28:27.803 INFO:teuthology.orchestra.run.smithi167.stdout: "temp_url_keys": [], 2023-10-05T19:28:27.803 INFO:teuthology.orchestra.run.smithi167.stdout: "type": "rgw", 2023-10-05T19:28:27.803 INFO:teuthology.orchestra.run.smithi167.stdout: "mfa_ids": [] 2023-10-05T19:28:27.803 INFO:teuthology.orchestra.run.smithi167.stdout:} 2023-10-05T19:28:27.803 INFO:teuthology.orchestra.run.smithi167.stdout: 2023-10-05T19:28:27.804 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.794+0000 7f8063037e40 1 -- 172.21.15.167:0/355163605 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:110 7.1f 7:f95f44c2:::notify.0:head [watch unwatch cookie 94204197446048] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x55ad9fb75920 con 0x7f805001a070 2023-10-05T19:28:27.804 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.798+0000 7f806036b700 1 -- 172.21.15.167:0/355163605 <== osd.2 v2:172.21.15.167:6816/30037 17 ==== osd_op_reply(110 notify.0 [watch unwatch cookie 94204197446048] v30'53 uv51 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f8054032040 con 0x7f805001a070 2023-10-05T19:28:27.804 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.798+0000 7f8063037e40 20 remove_watcher() i=0 2023-10-05T19:28:27.804 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.798+0000 7f8063037e40 2 removed watcher, disabling cache 2023-10-05T19:28:27.804 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.798+0000 7f8063037e40 1 -- 172.21.15.167:0/355163605 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:111 7.0 7:05bf5b68:::notify.1:head [watch unwatch cookie 94204197449360] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x55ad9fb9a1a0 con 0x55ad9fb5e7f0 2023-10-05T19:28:27.806 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.802+0000 7f806136d700 1 -- 172.21.15.167:0/355163605 <== osd.1 v2:172.21.15.167:6800/30035 42 ==== osd_op_reply(111 notify.1 [watch unwatch cookie 94204197449360] v30'53 uv51 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f8058052040 con 0x55ad9fb5e7f0 2023-10-05T19:28:27.806 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.802+0000 7f8063037e40 20 remove_watcher() i=1 2023-10-05T19:28:27.806 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.802+0000 7f8063037e40 1 -- 172.21.15.167:0/355163605 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:112 7.15 7:a93a5511:::notify.2:head [watch unwatch cookie 94204197450768] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x55ad9fb9ae90 con 0x55ad9fb95550 2023-10-05T19:28:27.809 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.802+0000 7f8060b6c700 1 -- 172.21.15.167:0/355163605 <== osd.3 v2:172.21.15.167:6808/30036 28 ==== osd_op_reply(112 notify.2 [watch unwatch cookie 94204197450768] v30'53 uv51 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f8050016070 con 0x55ad9fb95550 2023-10-05T19:28:27.809 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.802+0000 7f8063037e40 20 remove_watcher() i=2 2023-10-05T19:28:27.809 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.802+0000 7f8063037e40 1 -- 172.21.15.167:0/355163605 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:113 7.e 7:7759931f:::notify.3:head [watch unwatch cookie 94204197459184] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x55ad9fc1b230 con 0x7f805001a070 2023-10-05T19:28:27.812 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.806+0000 7f806036b700 1 -- 172.21.15.167:0/355163605 <== osd.2 v2:172.21.15.167:6816/30037 18 ==== osd_op_reply(113 notify.3 [watch unwatch cookie 94204197459184] v30'53 uv51 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f8054032040 con 0x7f805001a070 2023-10-05T19:28:27.812 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.806+0000 7f8063037e40 20 remove_watcher() i=3 2023-10-05T19:28:27.812 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.806+0000 7f8063037e40 1 -- 172.21.15.167:0/355163605 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:114 7.d 7:b4812045:::notify.4:head [watch unwatch cookie 94204197460288] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x55ad9fc1b230 con 0x55ad9fb5e7f0 2023-10-05T19:28:27.815 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.810+0000 7f806136d700 1 -- 172.21.15.167:0/355163605 <== osd.1 v2:172.21.15.167:6800/30035 43 ==== osd_op_reply(114 notify.4 [watch unwatch cookie 94204197460288] v30'53 uv51 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f8058052040 con 0x55ad9fb5e7f0 2023-10-05T19:28:27.815 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.810+0000 7f8063037e40 20 remove_watcher() i=4 2023-10-05T19:28:27.815 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.810+0000 7f8063037e40 1 -- 172.21.15.167:0/355163605 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:115 7.3 7:c609908c:::notify.5:head [watch unwatch cookie 94204197453040] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x55ad9fc1b230 con 0x7f8054007030 2023-10-05T19:28:27.818 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.814+0000 7f806136d700 1 -- 172.21.15.167:0/355163605 <== osd.0 v2:172.21.15.118:6800/27690 30 ==== osd_op_reply(115 notify.5 [watch unwatch cookie 94204197453040] v30'53 uv51 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f8058052040 con 0x7f8054007030 2023-10-05T19:28:27.818 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.814+0000 7f8063037e40 20 remove_watcher() i=5 2023-10-05T19:28:27.818 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.814+0000 7f8063037e40 1 -- 172.21.15.167:0/355163605 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:116 7.14 7:2b04a3e9:::notify.6:head [watch unwatch cookie 94204197467408] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x55ad9fc1b230 con 0x7f8054007030 2023-10-05T19:28:27.821 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.814+0000 7f806136d700 1 -- 172.21.15.167:0/355163605 <== osd.0 v2:172.21.15.118:6800/27690 31 ==== osd_op_reply(116 notify.6 [watch unwatch cookie 94204197467408] v30'53 uv51 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f8058052040 con 0x7f8054007030 2023-10-05T19:28:27.821 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.814+0000 7f8063037e40 20 remove_watcher() i=6 2023-10-05T19:28:27.821 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.814+0000 7f8063037e40 1 -- 172.21.15.167:0/355163605 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:117 7.9 7:93e5b521:::notify.7:head [watch unwatch cookie 94204197464832] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x55ad9fc1b230 con 0x7f8054007030 2023-10-05T19:28:27.824 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.818+0000 7f806136d700 1 -- 172.21.15.167:0/355163605 <== osd.0 v2:172.21.15.118:6800/27690 32 ==== osd_op_reply(117 notify.7 [watch unwatch cookie 94204197464832] v30'53 uv51 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f8058052040 con 0x7f8054007030 2023-10-05T19:28:27.824 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.818+0000 7f8063037e40 20 remove_watcher() i=7 2023-10-05T19:28:27.826 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.822+0000 7f80327e4700 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2023-10-05T19:28:27.826 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.822+0000 7f8063037e40 1 -- 172.21.15.167:0/355163605 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:118 6.4 6:22d26bf9:::data_loggenerations_metadata:head [watch unwatch cookie 94204197471088] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x55ad9fb9eb00 con 0x55ad9fb5e7f0 2023-10-05T19:28:27.830 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.822+0000 7f806136d700 1 -- 172.21.15.167:0/355163605 <== osd.1 v2:172.21.15.167:6800/30035 44 ==== osd_op_reply(118 data_loggenerations_metadata [watch unwatch cookie 94204197471088] v30'69 uv1 ondisk = 0) v8 ==== 172+0+0 (crc 0 0 0) 0x7f8058052040 con 0x55ad9fb5e7f0 2023-10-05T19:28:27.830 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.822+0000 7f8063037e40 1 -- 172.21.15.167:0/355163605 >> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] conn(0x7f8054007030 msgr2=0x7f805401c920 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-05T19:28:27.830 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.822+0000 7f8063037e40 1 --2- 172.21.15.167:0/355163605 >> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] conn(0x7f8054007030 0x7f805401c920 crc :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-05T19:28:27.830 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.822+0000 7f8063037e40 1 -- 172.21.15.167:0/355163605 >> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] conn(0x55ad9fb5e7f0 msgr2=0x55ad9fb91890 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-05T19:28:27.830 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.822+0000 7f8063037e40 1 --2- 172.21.15.167:0/355163605 >> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] conn(0x55ad9fb5e7f0 0x55ad9fb91890 crc :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-05T19:28:27.830 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.822+0000 7f8063037e40 1 -- 172.21.15.167:0/355163605 >> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] conn(0x7f805001a070 msgr2=0x7f805001d450 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-05T19:28:27.830 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.822+0000 7f8063037e40 1 --2- 172.21.15.167:0/355163605 >> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] conn(0x7f805001a070 0x7f805001d450 crc :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-05T19:28:27.830 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.822+0000 7f8063037e40 1 -- 172.21.15.167:0/355163605 >> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] conn(0x55ad9fb95550 msgr2=0x55ad9fb97960 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-05T19:28:27.830 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.822+0000 7f8063037e40 1 --2- 172.21.15.167:0/355163605 >> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] conn(0x55ad9fb95550 0x55ad9fb97960 crc :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-05T19:28:27.832 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.826+0000 7f8063037e40 1 -- 172.21.15.167:0/355163605 >> [v2:172.21.15.167:6824/29889,v1:172.21.15.167:6825/29889] conn(0x7f804c030c50 msgr2=0x7f804c033100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-05T19:28:27.832 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.826+0000 7f8063037e40 1 --2- 172.21.15.167:0/355163605 >> [v2:172.21.15.167:6824/29889,v1:172.21.15.167:6825/29889] conn(0x7f804c030c50 0x7f804c033100 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f80540030a0 tx=0x7f8054015000 comp rx=0 tx=0).stop 2023-10-05T19:28:27.832 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.826+0000 7f8063037e40 1 -- 172.21.15.167:0/355163605 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55ad9fb5e420 msgr2=0x55ad9fb85dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-05T19:28:27.832 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.826+0000 7f806136d700 1 -- 172.21.15.167:0/355163605 reap_dead start 2023-10-05T19:28:27.832 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.826+0000 7f8063037e40 1 --2- 172.21.15.167:0/355163605 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55ad9fb5e420 0x55ad9fb85dd0 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7f80500011e0 tx=0x7f8050016000 comp rx=0 tx=0).stop 2023-10-05T19:28:27.832 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.826+0000 7f8063037e40 1 -- 172.21.15.167:0/355163605 shutdown_connections 2023-10-05T19:28:27.832 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.826+0000 7f8063037e40 1 --2- 172.21.15.167:0/355163605 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55ad9fb5e420 0x55ad9fb85dd0 unknown :-1 s=CLOSED pgs=239 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-05T19:28:27.832 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.826+0000 7f8063037e40 1 -- 172.21.15.167:0/355163605 >> 172.21.15.167:0/355163605 conn(0x55ad9fb58730 msgr2=0x55ad9fb79680 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-05T19:28:27.833 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.826+0000 7f8063037e40 1 -- 172.21.15.167:0/355163605 shutdown_connections 2023-10-05T19:28:27.833 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.826+0000 7f8063037e40 1 -- 172.21.15.167:0/355163605 wait complete. 2023-10-05T19:28:27.833 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.826+0000 7f8063037e40 1 -- 172.21.15.167:0/3340626072 >> [v2:172.21.15.167:6824/29889,v1:172.21.15.167:6825/29889] conn(0x7f7fa8030c50 msgr2=0x7f7fa8033100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-05T19:28:27.833 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.826+0000 7f8063037e40 1 --2- 172.21.15.167:0/3340626072 >> [v2:172.21.15.167:6824/29889,v1:172.21.15.167:6825/29889] conn(0x7f7fa8030c50 0x7f7fa8033100 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x55ad9fb9fba0 tx=0x7f805800a280 comp rx=0 tx=0).stop 2023-10-05T19:28:27.833 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.826+0000 7f8063037e40 1 -- 172.21.15.167:0/3340626072 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55ad9fbadd50 msgr2=0x55ad9fbb37f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-05T19:28:27.833 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.826+0000 7f8063037e40 1 --2- 172.21.15.167:0/3340626072 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55ad9fbadd50 0x55ad9fbb37f0 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7f8054008fd0 tx=0x7f805402f000 comp rx=0 tx=0).stop 2023-10-05T19:28:27.833 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.826+0000 7f8063037e40 1 -- 172.21.15.167:0/3340626072 shutdown_connections 2023-10-05T19:28:27.833 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.826+0000 7f8063037e40 1 --2- 172.21.15.167:0/3340626072 >> [v2:172.21.15.167:6824/29889,v1:172.21.15.167:6825/29889] conn(0x7f7fa8030c50 0x7f7fa8033100 unknown :-1 s=CLOSED pgs=87 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-05T19:28:27.833 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.826+0000 7f8063037e40 1 --2- 172.21.15.167:0/3340626072 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55ad9fbadd50 0x55ad9fbb37f0 unknown :-1 s=CLOSED pgs=241 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-05T19:28:27.833 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.826+0000 7f8063037e40 1 -- 172.21.15.167:0/3340626072 >> 172.21.15.167:0/3340626072 conn(0x55ad9fba9110 msgr2=0x55ad9fba71b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-05T19:28:27.833 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.826+0000 7f8063037e40 1 -- 172.21.15.167:0/3340626072 shutdown_connections 2023-10-05T19:28:27.834 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.826+0000 7f8063037e40 1 -- 172.21.15.167:0/3340626072 wait complete. 2023-10-05T19:28:27.838 DEBUG:tasks.util.rgw: json result: {'user_id': 'foo', 'display_name': 'Foo', 'email': 'foo@foo.com', 'suspended': 0, 'max_buckets': 4, 'subusers': [], 'keys': [{'user': 'foo', 'access_key': '9te6NH5mcdcq0Tc5i8i1', 'secret_key': 'Ny4IOauQoL18Gp2zM7lC1vLmoawgqcYP/YGcWfXu'}], 'swift_keys': [], 'caps': [], 'op_mask': 'read, write, delete', 'default_placement': '', 'default_storage_class': '', 'placement_tags': [], 'bucket_quota': {'enabled': False, 'check_on_raw': False, 'max_size': -1, 'max_size_kb': 0, 'max_objects': -1}, 'user_quota': {'enabled': False, 'check_on_raw': False, 'max_size': -1, 'max_size_kb': 0, 'max_objects': -1}, 'temp_url_keys': [], 'type': 'rgw', 'mfa_ids': []} 2023-10-05T19:28:27.838 INFO:tasks.util.rgw:rgwadmin: client.0 : ['subuser', 'create', '--subuser', 'foo:foo1', '--access', 'full'] 2023-10-05T19:28:27.839 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'ceph', 'subuser', 'create', '--subuser', 'foo:foo1', '--access', 'full'] 2023-10-05T19:28:27.839 DEBUG:teuthology.orchestra.run.smithi167:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster ceph subuser create --subuser foo:foo1 --access full 2023-10-05T19:28:27.881 INFO:teuthology.orchestra.run.smithi167.stderr:ignoring --setuser ceph since I am not root 2023-10-05T19:28:27.881 INFO:teuthology.orchestra.run.smithi167.stderr:ignoring --setgroup ceph since I am not root 2023-10-05T19:28:27.882 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.878+0000 7f5d3f786e40 1 Processor -- start 2023-10-05T19:28:27.882 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.878+0000 7f5d3f786e40 1 -- start start 2023-10-05T19:28:27.882 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.878+0000 7f5d3f786e40 1 --2- >> v2:172.21.15.118:3300/0 conn(0x56161bf087f0 0x56161bf076e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-05T19:28:27.882 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.878+0000 7f5d3f786e40 1 -- --> v1:172.21.15.118:6789/0 -- auth(proto 0 26 bytes epoch 0) v1 -- 0x56161be54bd0 con 0x56161bf08420 2023-10-05T19:28:27.882 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.878+0000 7f5d3f786e40 1 -- --> v2:172.21.15.118:3300/0 -- mon_getmap magic: 0 v1 -- 0x56161be11ee0 con 0x56161bf087f0 2023-10-05T19:28:27.883 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.878+0000 7f5d3caba700 1 --2- >> v2:172.21.15.118:3300/0 conn(0x56161bf087f0 0x56161bf076e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-05T19:28:27.883 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.878+0000 7f5d3d2bb700 1 --1- >> v1:172.21.15.118:6789/0 conn(0x56161bf08420 0x56161bf06fd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.118:6789/0 says I am v1:172.21.15.167:56442/0 (socket says 172.21.15.167:56442) 2023-10-05T19:28:27.883 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.878+0000 7f5d3d2bb700 1 -- 172.21.15.167:0/4015713226 learned_addr learned my addr 172.21.15.167:0/4015713226 (peer_addr_for_me v1:172.21.15.167:0/0) 2023-10-05T19:28:27.884 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.878+0000 7f5d3caba700 1 -- 172.21.15.167:0/4015713226 >> v1:172.21.15.118:6789/0 conn(0x56161bf08420 legacy=0x56161bf06fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-05T19:28:27.884 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.878+0000 7f5d3caba700 1 -- 172.21.15.167:0/4015713226 --> v2:172.21.15.118:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x56161bf07ce0 con 0x56161bf087f0 2023-10-05T19:28:27.884 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.878+0000 7f5d3caba700 1 --2- 172.21.15.167:0/4015713226 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x56161bf087f0 0x56161bf076e0 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7f5d2800a5d0 tx=0x7f5d28005a50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d7b37880308e0464 server_cookie=0 in_seq=0 out_seq=0 2023-10-05T19:28:27.885 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.878+0000 7f5d37fff700 1 -- 172.21.15.167:0/4015713226 <== mon.0 v2:172.21.15.118:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f5d280086a0 con 0x56161bf087f0 2023-10-05T19:28:27.885 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.878+0000 7f5d37fff700 1 -- 172.21.15.167:0/4015713226 <== mon.0 v2:172.21.15.118:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5d28008800 con 0x56161bf087f0 2023-10-05T19:28:27.885 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.878+0000 7f5d37fff700 1 -- 172.21.15.167:0/4015713226 <== mon.0 v2:172.21.15.118:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f5d28008ad0 con 0x56161bf087f0 2023-10-05T19:28:27.885 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.878+0000 7f5d3f786e40 1 -- 172.21.15.167:0/4015713226 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x56161bf087f0 msgr2=0x56161bf076e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-05T19:28:27.885 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.878+0000 7f5d3f786e40 1 --2- 172.21.15.167:0/4015713226 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x56161bf087f0 0x56161bf076e0 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7f5d2800a5d0 tx=0x7f5d28005a50 comp rx=0 tx=0).stop 2023-10-05T19:28:27.886 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.878+0000 7f5d3f786e40 1 -- 172.21.15.167:0/4015713226 shutdown_connections 2023-10-05T19:28:27.886 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.878+0000 7f5d3f786e40 1 --2- 172.21.15.167:0/4015713226 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x56161bf087f0 0x56161bf076e0 unknown :-1 s=CLOSED pgs=243 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-05T19:28:27.886 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.878+0000 7f5d3f786e40 1 -- 172.21.15.167:0/4015713226 >> 172.21.15.167:0/4015713226 conn(0x56161bf02730 msgr2=0x56161bf04b40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-05T19:28:27.886 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.878+0000 7f5d3f786e40 1 -- 172.21.15.167:0/4015713226 shutdown_connections 2023-10-05T19:28:27.886 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.878+0000 7f5d3f786e40 1 -- 172.21.15.167:0/4015713226 wait complete. 2023-10-05T19:28:27.887 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.882+0000 7f5d3f786e40 1 Processor -- start 2023-10-05T19:28:27.887 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.882+0000 7f5d3f786e40 1 -- start start 2023-10-05T19:28:27.887 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.882+0000 7f5d3f786e40 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x56161bf08420 0x56161bf29100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-05T19:28:27.887 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.882+0000 7f5d3f786e40 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x56161bdd5e30 con 0x56161bf08420 2023-10-05T19:28:27.887 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.882+0000 7f5d3d2bb700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x56161bf08420 0x56161bf29100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-05T19:28:27.887 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.882+0000 7f5d3d2bb700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x56161bf08420 0x56161bf29100 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.118:3300/0 says I am v2:172.21.15.167:57608/0 (socket says 172.21.15.167:57608) 2023-10-05T19:28:27.887 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.882+0000 7f5d3d2bb700 1 -- 172.21.15.167:0/1707443511 learned_addr learned my addr 172.21.15.167:0/1707443511 (peer_addr_for_me v2:172.21.15.167:0/0) 2023-10-05T19:28:27.888 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.882+0000 7f5d3d2bb700 1 -- 172.21.15.167:0/1707443511 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5d28005cf0 con 0x56161bf08420 2023-10-05T19:28:27.888 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.882+0000 7f5d3d2bb700 1 --2- 172.21.15.167:0/1707443511 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x56161bf08420 0x56161bf29100 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7f5d30001c30 tx=0x7f5d300017e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8d0f7759308d66f6 server_cookie=0 in_seq=0 out_seq=0 2023-10-05T19:28:27.888 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.882+0000 7f5d36ffd700 1 -- 172.21.15.167:0/1707443511 <== mon.0 v2:172.21.15.118:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f5d3000bbf0 con 0x56161bf08420 2023-10-05T19:28:27.888 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.882+0000 7f5d36ffd700 1 -- 172.21.15.167:0/1707443511 <== mon.0 v2:172.21.15.118:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5d3000bd50 con 0x56161bf08420 2023-10-05T19:28:27.888 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.882+0000 7f5d36ffd700 1 -- 172.21.15.167:0/1707443511 <== mon.0 v2:172.21.15.118:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f5d3000e470 con 0x56161bf08420 2023-10-05T19:28:27.890 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.882+0000 7f5d3f786e40 1 -- 172.21.15.167:0/1707443511 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x56161bf08420 msgr2=0x56161bf29100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-05T19:28:27.890 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.882+0000 7f5d3f786e40 1 --2- 172.21.15.167:0/1707443511 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x56161bf08420 0x56161bf29100 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7f5d30001c30 tx=0x7f5d300017e0 comp rx=0 tx=0).stop 2023-10-05T19:28:27.890 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.882+0000 7f5d3f786e40 1 -- 172.21.15.167:0/1707443511 shutdown_connections 2023-10-05T19:28:27.890 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.882+0000 7f5d3f786e40 1 --2- 172.21.15.167:0/1707443511 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x56161bf08420 0x56161bf29100 unknown :-1 s=CLOSED pgs=244 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-05T19:28:27.890 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.882+0000 7f5d3f786e40 1 -- 172.21.15.167:0/1707443511 >> 172.21.15.167:0/1707443511 conn(0x56161bf02730 msgr2=0x56161bf24cc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-05T19:28:27.890 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.882+0000 7f5d3f786e40 1 -- 172.21.15.167:0/1707443511 shutdown_connections 2023-10-05T19:28:27.890 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.882+0000 7f5d3f786e40 1 -- 172.21.15.167:0/1707443511 wait complete. 2023-10-05T19:28:27.891 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.882+0000 7f5d3f786e40 1 Processor -- start 2023-10-05T19:28:27.891 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.882+0000 7f5d3f786e40 1 -- start start 2023-10-05T19:28:27.891 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.882+0000 7f5d3f786e40 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x56161bf08420 0x56161bf2fe70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-05T19:28:27.891 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.882+0000 7f5d3f786e40 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x56161bda8ed0 con 0x56161bf08420 2023-10-05T19:28:27.891 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.886+0000 7f5d3d2bb700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x56161bf08420 0x56161bf2fe70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-05T19:28:27.891 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.886+0000 7f5d3d2bb700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x56161bf08420 0x56161bf2fe70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.118:3300/0 says I am v2:172.21.15.167:57614/0 (socket says 172.21.15.167:57614) 2023-10-05T19:28:27.891 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.886+0000 7f5d3d2bb700 1 -- 172.21.15.167:0/2568822831 learned_addr learned my addr 172.21.15.167:0/2568822831 (peer_addr_for_me v2:172.21.15.167:0/0) 2023-10-05T19:28:27.892 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.886+0000 7f5d3d2bb700 1 -- 172.21.15.167:0/2568822831 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x56161be54bd0 con 0x56161bf08420 2023-10-05T19:28:27.892 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.886+0000 7f5d3d2bb700 1 --2- 172.21.15.167:0/2568822831 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x56161bf08420 0x56161bf2fe70 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7f5d30001820 tx=0x7f5d30016000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-05T19:28:27.892 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.886+0000 7f5d357fa700 1 -- 172.21.15.167:0/2568822831 <== mon.0 v2:172.21.15.118:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f5d3001f070 con 0x56161bf08420 2023-10-05T19:28:27.892 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.886+0000 7f5d357fa700 1 -- 172.21.15.167:0/2568822831 <== mon.0 v2:172.21.15.118:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5d30019760 con 0x56161bf08420 2023-10-05T19:28:27.892 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.886+0000 7f5d3f786e40 1 -- 172.21.15.167:0/2568822831 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x56161bf303b0 con 0x56161bf08420 2023-10-05T19:28:27.892 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.886+0000 7f5d357fa700 1 -- 172.21.15.167:0/2568822831 <== mon.0 v2:172.21.15.118:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f5d300198d0 con 0x56161bf08420 2023-10-05T19:28:27.892 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.886+0000 7f5d3f786e40 1 -- 172.21.15.167:0/2568822831 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x56161bf307f0 con 0x56161bf08420 2023-10-05T19:28:27.893 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.886+0000 7f5d357fa700 1 -- 172.21.15.167:0/2568822831 <== mon.0 v2:172.21.15.118:3300/0 4 ==== mgrmap(e 5) v1 ==== 40329+0+0 (secure 0 0 0) 0x7f5d30022020 con 0x56161bf08420 2023-10-05T19:28:27.893 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.886+0000 7f5d357fa700 1 --2- 172.21.15.167:0/2568822831 >> [v2:172.21.15.167:6824/29889,v1:172.21.15.167:6825/29889] conn(0x7f5d20030c00 0x7f5d200330b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-05T19:28:27.894 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.886+0000 7f5d357fa700 1 -- 172.21.15.167:0/2568822831 <== mon.0 v2:172.21.15.118:3300/0 5 ==== osd_map(30..30 src has 1..30) v4 ==== 6057+0+0 (secure 0 0 0) 0x7f5d30044f00 con 0x56161bf08420 2023-10-05T19:28:27.894 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.886+0000 7f5d3f786e40 20 rados->read ofs=0 len=0 2023-10-05T19:28:27.894 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.890+0000 7f5d3f786e40 1 --2- 172.21.15.167:0/2568822831 >> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] conn(0x56161bf087f0 0x56161bf3b930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-05T19:28:27.894 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.890+0000 7f5d3caba700 1 --2- 172.21.15.167:0/2568822831 >> [v2:172.21.15.167:6824/29889,v1:172.21.15.167:6825/29889] conn(0x7f5d20030c00 0x7f5d200330b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-05T19:28:27.895 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.890+0000 7f5d3f786e40 1 -- 172.21.15.167:0/2568822831 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:1 5.12 5:49953fa1:::default.realm:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x56161bf3bfe0 con 0x56161bf087f0 2023-10-05T19:28:27.895 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.890+0000 7f5d3dabc700 1 --2- 172.21.15.167:0/2568822831 >> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] conn(0x56161bf087f0 0x56161bf3b930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-05T19:28:27.895 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.890+0000 7f5d3caba700 1 --2- 172.21.15.167:0/2568822831 >> [v2:172.21.15.167:6824/29889,v1:172.21.15.167:6825/29889] conn(0x7f5d20030c00 0x7f5d200330b0 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f5d28005a90 tx=0x7f5d28015000 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-05T19:28:27.895 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.890+0000 7f5d3dabc700 1 --2- 172.21.15.167:0/2568822831 >> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] conn(0x56161bf087f0 0x56161bf3b930 crc :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-05T19:28:27.896 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.890+0000 7f5d3dabc700 1 -- 172.21.15.167:0/2568822831 <== osd.1 v2:172.21.15.167:6800/30035 1 ==== osd_op_reply(1 default.realm [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 ==== 157+0+0 (crc 0 0 0) 0x7f5d3800a5a0 con 0x56161bf087f0 2023-10-05T19:28:27.896 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.890+0000 7f5d3f786e40 20 rados_obj.operate() r=-2 bl.length=0 2023-10-05T19:28:27.897 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.890+0000 7f5d3f786e40 20 realm 2023-10-05T19:28:27.897 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.890+0000 7f5d3f786e40 20 rados->read ofs=0 len=0 2023-10-05T19:28:27.897 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.890+0000 7f5d3f786e40 1 -- 172.21.15.167:0/2568822831 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:2 5.12 5:49953fa1:::default.realm:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x56161bf38f90 con 0x56161bf087f0 2023-10-05T19:28:27.897 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.890+0000 7f5d3dabc700 1 -- 172.21.15.167:0/2568822831 <== osd.1 v2:172.21.15.167:6800/30035 2 ==== osd_op_reply(2 default.realm [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 ==== 157+0+0 (crc 0 0 0) 0x7f5d3800a5a0 con 0x56161bf087f0 2023-10-05T19:28:27.897 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.890+0000 7f5d3f786e40 20 rados_obj.operate() r=-2 bl.length=0 2023-10-05T19:28:27.897 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.890+0000 7f5d3f786e40 4 RGWPeriod::init failed to init realm id : (2) No such file or directory 2023-10-05T19:28:27.897 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.890+0000 7f5d3f786e40 20 rados->read ofs=0 len=0 2023-10-05T19:28:27.897 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.890+0000 7f5d3f786e40 1 -- 172.21.15.167:0/2568822831 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:3 5.12 5:49953fa1:::default.realm:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x56161bf3cac0 con 0x56161bf087f0 2023-10-05T19:28:27.899 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.890+0000 7f5d3dabc700 1 -- 172.21.15.167:0/2568822831 <== osd.1 v2:172.21.15.167:6800/30035 3 ==== osd_op_reply(3 default.realm [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 ==== 157+0+0 (crc 0 0 0) 0x7f5d3800a5a0 con 0x56161bf087f0 2023-10-05T19:28:27.899 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.890+0000 7f5d3f786e40 20 rados_obj.operate() r=-2 bl.length=0 2023-10-05T19:28:27.899 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.890+0000 7f5d3f786e40 20 rados->read ofs=0 len=0 2023-10-05T19:28:27.899 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.890+0000 7f5d3f786e40 1 -- 172.21.15.167:0/2568822831 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:4 5.f 5:f43b6ece:::zone_names.default:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x56161bf3c6d0 con 0x56161bf087f0 2023-10-05T19:28:27.899 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.890+0000 7f5d3dabc700 1 -- 172.21.15.167:0/2568822831 <== osd.1 v2:172.21.15.167:6800/30035 4 ==== osd_op_reply(4 zone_names.default [read 0~46 out=46b] v0'0 uv1 ondisk = 0) v8 ==== 162+0+46 (crc 0 0 0) 0x7f5d3800a5a0 con 0x56161bf087f0 2023-10-05T19:28:27.899 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.890+0000 7f5d3f786e40 20 rados_obj.operate() r=0 bl.length=46 2023-10-05T19:28:27.899 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.890+0000 7f5d3f786e40 20 rados->read ofs=0 len=0 2023-10-05T19:28:27.899 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.890+0000 7f5d3f786e40 1 -- 172.21.15.167:0/2568822831 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:5 5.1d 5:b8374eb4:::zone_info.9a9df965-6612-45d4-a45f-6e3c889bacc2:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x56161bf3d950 con 0x56161bf087f0 2023-10-05T19:28:27.899 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.890+0000 7f5d3dabc700 1 -- 172.21.15.167:0/2568822831 <== osd.1 v2:172.21.15.167:6800/30035 5 ==== osd_op_reply(5 zone_info.9a9df965-6612-45d4-a45f-6e3c889bacc2 [read 0~874 out=874b] v0'0 uv2 ondisk = 0) v8 ==== 190+0+874 (crc 0 0 0) 0x7f5d3800a5a0 con 0x56161bf087f0 2023-10-05T19:28:27.900 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.890+0000 7f5d3f786e40 20 rados_obj.operate() r=0 bl.length=874 2023-10-05T19:28:27.900 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.890+0000 7f5d3f786e40 20 searching for the correct realm 2023-10-05T19:28:27.900 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.894+0000 7f5d3f786e40 1 --2- 172.21.15.167:0/2568822831 >> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] conn(0x56161bf3f710 0x56161bf41af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-05T19:28:27.900 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.894+0000 7f5d3f786e40 1 -- 172.21.15.167:0/2568822831 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:6 5.0 5:00000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x56161bf42030 con 0x56161bf3f710 2023-10-05T19:28:27.900 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.894+0000 7f5d3d2bb700 1 --2- 172.21.15.167:0/2568822831 >> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] conn(0x56161bf3f710 0x56161bf41af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-05T19:28:27.900 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.894+0000 7f5d3d2bb700 1 --2- 172.21.15.167:0/2568822831 >> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] conn(0x56161bf3f710 0x56161bf41af0 crc :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-05T19:28:27.901 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.894+0000 7f5d3d2bb700 1 -- 172.21.15.167:0/2568822831 <== osd.3 v2:172.21.15.167:6808/30036 1 ==== osd_op_reply(6 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f5d300173e0 con 0x56161bf3f710 2023-10-05T19:28:27.901 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.894+0000 7f5d3d2bb700 1 --2- 172.21.15.167:0/2568822831 >> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] conn(0x7f5d3000d3d0 0x7f5d300420d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-05T19:28:27.901 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.894+0000 7f5d3d2bb700 1 -- 172.21.15.167:0/2568822831 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:7 5.10 5:08000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x56161bf42030 con 0x7f5d3000d3d0 2023-10-05T19:28:27.901 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.894+0000 7f5d3caba700 1 --2- 172.21.15.167:0/2568822831 >> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] conn(0x7f5d3000d3d0 0x7f5d300420d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-05T19:28:27.902 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.894+0000 7f5d3caba700 1 --2- 172.21.15.167:0/2568822831 >> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] conn(0x7f5d3000d3d0 0x7f5d300420d0 crc :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-05T19:28:27.902 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.894+0000 7f5d3caba700 1 -- 172.21.15.167:0/2568822831 <== osd.2 v2:172.21.15.167:6816/30037 1 ==== osd_op_reply(7 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f5d28008ae0 con 0x7f5d3000d3d0 2023-10-05T19:28:27.902 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.894+0000 7f5d3caba700 1 --2- 172.21.15.167:0/2568822831 >> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] conn(0x7f5d28007030 0x7f5d2801c920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-05T19:28:27.902 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.894+0000 7f5d3caba700 1 -- 172.21.15.167:0/2568822831 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:8 5.8 5:10000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x56161bf42030 con 0x7f5d28007030 2023-10-05T19:28:27.902 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.898+0000 7f5d3dabc700 1 --2- 172.21.15.167:0/2568822831 >> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] conn(0x7f5d28007030 0x7f5d2801c920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-05T19:28:27.903 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.898+0000 7f5d3dabc700 1 --2- 172.21.15.167:0/2568822831 >> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] conn(0x7f5d28007030 0x7f5d2801c920 crc :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-05T19:28:27.904 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.898+0000 7f5d3dabc700 1 -- 172.21.15.167:0/2568822831 <== osd.0 v2:172.21.15.118:6800/27690 1 ==== osd_op_reply(8 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f5d3800a5a0 con 0x7f5d28007030 2023-10-05T19:28:27.904 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.898+0000 7f5d3dabc700 1 -- 172.21.15.167:0/2568822831 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:9 5.18 5:18000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x56161bf42030 con 0x56161bf087f0 2023-10-05T19:28:27.905 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.898+0000 7f5d3dabc700 1 -- 172.21.15.167:0/2568822831 <== osd.1 v2:172.21.15.167:6800/30035 6 ==== osd_op_reply(9 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f5d3800a5a0 con 0x56161bf087f0 2023-10-05T19:28:27.905 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.898+0000 7f5d3dabc700 1 -- 172.21.15.167:0/2568822831 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:10 5.4 5:20000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x56161bf42030 con 0x56161bf087f0 2023-10-05T19:28:27.906 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.898+0000 7f5d3dabc700 1 -- 172.21.15.167:0/2568822831 <== osd.1 v2:172.21.15.167:6800/30035 7 ==== osd_op_reply(10 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f5d3800a5a0 con 0x56161bf087f0 2023-10-05T19:28:27.906 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.898+0000 7f5d3dabc700 1 -- 172.21.15.167:0/2568822831 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:11 5.14 5:28000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x56161bf42030 con 0x7f5d28007030 2023-10-05T19:28:27.906 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.902+0000 7f5d3dabc700 1 -- 172.21.15.167:0/2568822831 <== osd.0 v2:172.21.15.118:6800/27690 2 ==== osd_op_reply(11 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f5d3800a5a0 con 0x7f5d28007030 2023-10-05T19:28:27.906 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.902+0000 7f5d3dabc700 1 -- 172.21.15.167:0/2568822831 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:12 5.c 5:30000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x56161bf42030 con 0x7f5d28007030 2023-10-05T19:28:27.907 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.902+0000 7f5d3dabc700 1 -- 172.21.15.167:0/2568822831 <== osd.0 v2:172.21.15.118:6800/27690 3 ==== osd_op_reply(12 [pgnls start_epoch 30 out=112b] v23'1 uv1 ondisk = 1) v8 ==== 144+0+112 (crc 0 0 0) 0x7f5d3800a5a0 con 0x7f5d28007030 2023-10-05T19:28:27.907 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.902+0000 7f5d3dabc700 1 -- 172.21.15.167:0/2568822831 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:13 5.1c 5:38000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x56161bf42030 con 0x7f5d3000d3d0 2023-10-05T19:28:27.908 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.902+0000 7f5d3caba700 1 -- 172.21.15.167:0/2568822831 <== osd.2 v2:172.21.15.167:6816/30037 2 ==== osd_op_reply(13 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f5d28008ae0 con 0x7f5d3000d3d0 2023-10-05T19:28:27.908 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.902+0000 7f5d3caba700 1 -- 172.21.15.167:0/2568822831 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:14 5.2 5:40000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x56161bf42030 con 0x56161bf3f710 2023-10-05T19:28:27.908 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.902+0000 7f5d3d2bb700 1 -- 172.21.15.167:0/2568822831 <== osd.3 v2:172.21.15.167:6808/30036 2 ==== osd_op_reply(14 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f5d300173e0 con 0x56161bf3f710 2023-10-05T19:28:27.909 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.902+0000 7f5d3d2bb700 1 -- 172.21.15.167:0/2568822831 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:15 5.12 5:48000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x56161bf42030 con 0x56161bf087f0 2023-10-05T19:28:27.909 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.902+0000 7f5d3dabc700 1 -- 172.21.15.167:0/2568822831 <== osd.1 v2:172.21.15.167:6800/30035 8 ==== osd_op_reply(15 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f5d3800a5a0 con 0x56161bf087f0 2023-10-05T19:28:27.909 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.902+0000 7f5d3dabc700 1 -- 172.21.15.167:0/2568822831 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:16 5.a 5:50000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x56161bf42030 con 0x7f5d28007030 2023-10-05T19:28:27.910 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.906+0000 7f5d3dabc700 1 -- 172.21.15.167:0/2568822831 <== osd.0 v2:172.21.15.118:6800/27690 4 ==== osd_op_reply(16 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f5d3800a5a0 con 0x7f5d28007030 2023-10-05T19:28:27.910 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.906+0000 7f5d3dabc700 1 -- 172.21.15.167:0/2568822831 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:17 5.1a 5:58000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x56161bf42030 con 0x56161bf087f0 2023-10-05T19:28:27.911 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.906+0000 7f5d3dabc700 1 -- 172.21.15.167:0/2568822831 <== osd.1 v2:172.21.15.167:6800/30035 9 ==== osd_op_reply(17 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f5d3800a5a0 con 0x56161bf087f0 2023-10-05T19:28:27.911 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.906+0000 7f5d3dabc700 1 -- 172.21.15.167:0/2568822831 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:18 5.6 5:60000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x56161bf42030 con 0x7f5d3000d3d0 2023-10-05T19:28:27.911 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.906+0000 7f5d3caba700 1 -- 172.21.15.167:0/2568822831 <== osd.2 v2:172.21.15.167:6816/30037 3 ==== osd_op_reply(18 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f5d28008ae0 con 0x7f5d3000d3d0 2023-10-05T19:28:27.911 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.906+0000 7f5d3caba700 1 -- 172.21.15.167:0/2568822831 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:19 5.16 5:68000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x56161bf42030 con 0x56161bf3f710 2023-10-05T19:28:27.912 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.906+0000 7f5d3d2bb700 1 -- 172.21.15.167:0/2568822831 <== osd.3 v2:172.21.15.167:6808/30036 3 ==== osd_op_reply(19 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f5d300173e0 con 0x56161bf3f710 2023-10-05T19:28:27.912 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.906+0000 7f5d3d2bb700 1 -- 172.21.15.167:0/2568822831 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:20 5.e 5:70000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x56161bf42030 con 0x56161bf3f710 2023-10-05T19:28:27.913 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.906+0000 7f5d3d2bb700 1 -- 172.21.15.167:0/2568822831 <== osd.3 v2:172.21.15.167:6808/30036 4 ==== osd_op_reply(20 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f5d300173e0 con 0x56161bf3f710 2023-10-05T19:28:27.913 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.906+0000 7f5d3d2bb700 1 -- 172.21.15.167:0/2568822831 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:21 5.1e 5:78000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x56161bf42030 con 0x7f5d28007030 2023-10-05T19:28:27.914 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.906+0000 7f5d3dabc700 1 -- 172.21.15.167:0/2568822831 <== osd.0 v2:172.21.15.118:6800/27690 5 ==== osd_op_reply(21 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f5d3800a5a0 con 0x7f5d28007030 2023-10-05T19:28:27.914 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.906+0000 7f5d3dabc700 1 -- 172.21.15.167:0/2568822831 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:22 5.1 5:80000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x56161bf42030 con 0x56161bf3f710 2023-10-05T19:28:27.914 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.910+0000 7f5d3d2bb700 1 -- 172.21.15.167:0/2568822831 <== osd.3 v2:172.21.15.167:6808/30036 5 ==== osd_op_reply(22 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f5d300173e0 con 0x56161bf3f710 2023-10-05T19:28:27.914 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.910+0000 7f5d3d2bb700 1 -- 172.21.15.167:0/2568822831 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:23 5.11 5:88000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x56161bf42030 con 0x56161bf3f710 2023-10-05T19:28:27.915 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.910+0000 7f5d3d2bb700 1 -- 172.21.15.167:0/2568822831 <== osd.3 v2:172.21.15.167:6808/30036 6 ==== osd_op_reply(23 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f5d300173e0 con 0x56161bf3f710 2023-10-05T19:28:27.915 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.910+0000 7f5d3d2bb700 1 -- 172.21.15.167:0/2568822831 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:24 5.9 5:90000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x56161bf42030 con 0x56161bf087f0 2023-10-05T19:28:27.916 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.910+0000 7f5d3dabc700 1 -- 172.21.15.167:0/2568822831 <== osd.1 v2:172.21.15.167:6800/30035 10 ==== osd_op_reply(24 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f5d3800a5a0 con 0x56161bf087f0 2023-10-05T19:28:27.916 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.910+0000 7f5d3dabc700 1 -- 172.21.15.167:0/2568822831 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:25 5.19 5:98000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x56161bf42030 con 0x56161bf087f0 2023-10-05T19:28:27.916 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.910+0000 7f5d3dabc700 1 -- 172.21.15.167:0/2568822831 <== osd.1 v2:172.21.15.167:6800/30035 11 ==== osd_op_reply(25 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f5d3800a5a0 con 0x56161bf087f0 2023-10-05T19:28:27.916 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.910+0000 7f5d3dabc700 1 -- 172.21.15.167:0/2568822831 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:26 5.5 5:a0000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x56161bf42030 con 0x7f5d28007030 2023-10-05T19:28:27.917 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.910+0000 7f5d3dabc700 1 -- 172.21.15.167:0/2568822831 <== osd.0 v2:172.21.15.118:6800/27690 6 ==== osd_op_reply(26 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f5d3800a5a0 con 0x7f5d28007030 2023-10-05T19:28:27.917 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.910+0000 7f5d3dabc700 1 -- 172.21.15.167:0/2568822831 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:27 5.15 5:a8000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x56161bf42030 con 0x56161bf3f710 2023-10-05T19:28:27.918 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.914+0000 7f5d3d2bb700 1 -- 172.21.15.167:0/2568822831 <== osd.3 v2:172.21.15.167:6808/30036 7 ==== osd_op_reply(27 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f5d300173e0 con 0x56161bf3f710 2023-10-05T19:28:27.918 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.914+0000 7f5d3d2bb700 1 -- 172.21.15.167:0/2568822831 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:28 5.d 5:b0000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x56161bf42030 con 0x7f5d3000d3d0 2023-10-05T19:28:27.919 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.914+0000 7f5d3caba700 1 -- 172.21.15.167:0/2568822831 <== osd.2 v2:172.21.15.167:6816/30037 4 ==== osd_op_reply(28 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f5d28008ae0 con 0x7f5d3000d3d0 2023-10-05T19:28:27.919 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.914+0000 7f5d3caba700 1 -- 172.21.15.167:0/2568822831 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:29 5.1d 5:b8000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x56161bf42030 con 0x56161bf087f0 2023-10-05T19:28:27.919 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.914+0000 7f5d3dabc700 1 -- 172.21.15.167:0/2568822831 <== osd.1 v2:172.21.15.167:6800/30035 12 ==== osd_op_reply(29 [pgnls start_epoch 30 out=107b] v30'2 uv2 ondisk = 1) v8 ==== 144+0+107 (crc 0 0 0) 0x7f5d3800a5a0 con 0x56161bf087f0 2023-10-05T19:28:27.920 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.914+0000 7f5d3dabc700 1 -- 172.21.15.167:0/2568822831 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:30 5.3 5:c0000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x56161bf42030 con 0x7f5d28007030 2023-10-05T19:28:27.920 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.914+0000 7f5d3dabc700 1 -- 172.21.15.167:0/2568822831 <== osd.0 v2:172.21.15.118:6800/27690 7 ==== osd_op_reply(30 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f5d3800a5a0 con 0x7f5d28007030 2023-10-05T19:28:27.921 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.914+0000 7f5d3dabc700 1 -- 172.21.15.167:0/2568822831 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:31 5.13 5:c8000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x56161bf42030 con 0x56161bf3f710 2023-10-05T19:28:27.921 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.914+0000 7f5d3d2bb700 1 -- 172.21.15.167:0/2568822831 <== osd.3 v2:172.21.15.167:6808/30036 8 ==== osd_op_reply(31 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f5d300173e0 con 0x56161bf3f710 2023-10-05T19:28:27.921 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.914+0000 7f5d3d2bb700 1 -- 172.21.15.167:0/2568822831 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:32 5.b 5:d0000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x56161bf42030 con 0x7f5d28007030 2023-10-05T19:28:27.922 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.918+0000 7f5d3dabc700 1 -- 172.21.15.167:0/2568822831 <== osd.0 v2:172.21.15.118:6800/27690 8 ==== osd_op_reply(32 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f5d3800a5a0 con 0x7f5d28007030 2023-10-05T19:28:27.922 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.918+0000 7f5d3dabc700 1 -- 172.21.15.167:0/2568822831 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:33 5.1b 5:d8000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x56161bf42030 con 0x7f5d3000d3d0 2023-10-05T19:28:27.923 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.918+0000 7f5d3caba700 1 -- 172.21.15.167:0/2568822831 <== osd.2 v2:172.21.15.167:6816/30037 5 ==== osd_op_reply(33 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f5d28008ae0 con 0x7f5d3000d3d0 2023-10-05T19:28:27.923 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.918+0000 7f5d3caba700 1 -- 172.21.15.167:0/2568822831 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:34 5.7 5:e0000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x56161bf42030 con 0x56161bf3f710 2023-10-05T19:28:27.924 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.918+0000 7f5d3d2bb700 1 -- 172.21.15.167:0/2568822831 <== osd.3 v2:172.21.15.167:6808/30036 9 ==== osd_op_reply(34 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f5d300173e0 con 0x56161bf3f710 2023-10-05T19:28:27.924 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.918+0000 7f5d3d2bb700 1 -- 172.21.15.167:0/2568822831 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:35 5.17 5:e8000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x56161bf42030 con 0x7f5d28007030 2023-10-05T19:28:27.925 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.918+0000 7f5d3dabc700 1 -- 172.21.15.167:0/2568822831 <== osd.0 v2:172.21.15.118:6800/27690 9 ==== osd_op_reply(35 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f5d3800a5a0 con 0x7f5d28007030 2023-10-05T19:28:27.925 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.918+0000 7f5d3dabc700 1 -- 172.21.15.167:0/2568822831 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:36 5.f 5:f0000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x56161bf42030 con 0x56161bf087f0 2023-10-05T19:28:27.925 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.918+0000 7f5d3dabc700 1 -- 172.21.15.167:0/2568822831 <== osd.1 v2:172.21.15.167:6800/30035 13 ==== osd_op_reply(36 [pgnls start_epoch 30 out=115b] v23'2 uv2 ondisk = 1) v8 ==== 144+0+115 (crc 0 0 0) 0x7f5d3800a5a0 con 0x56161bf087f0 2023-10-05T19:28:27.926 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.918+0000 7f5d3dabc700 1 -- 172.21.15.167:0/2568822831 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:37 5.1f 5:f8000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x56161bf42030 con 0x7f5d3000d3d0 2023-10-05T19:28:27.926 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.922+0000 7f5d3caba700 1 -- 172.21.15.167:0/2568822831 <== osd.2 v2:172.21.15.167:6816/30037 6 ==== osd_op_reply(37 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f5d28008ae0 con 0x7f5d3000d3d0 2023-10-05T19:28:27.927 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.922+0000 7f5d3f786e40 20 RGWRados::pool_iterate: got zonegroup_info.4f1f719d-9648-417b-b10c-e5d939825419 2023-10-05T19:28:27.927 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.922+0000 7f5d3f786e40 20 RGWRados::pool_iterate: got zone_info.9a9df965-6612-45d4-a45f-6e3c889bacc2 2023-10-05T19:28:27.927 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.922+0000 7f5d3f786e40 20 RGWRados::pool_iterate: got zone_names.default 2023-10-05T19:28:27.927 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.922+0000 7f5d3f786e40 20 RGWRados::pool_iterate: got zonegroups_names.default 2023-10-05T19:28:27.927 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.922+0000 7f5d3f786e40 20 rados->read ofs=0 len=0 2023-10-05T19:28:27.927 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.922+0000 7f5d3f786e40 1 -- 172.21.15.167:0/2568822831 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:38 5.12 5:49953fa1:::default.realm:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x56161bf3e490 con 0x56161bf087f0 2023-10-05T19:28:27.928 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.922+0000 7f5d3dabc700 1 -- 172.21.15.167:0/2568822831 <== osd.1 v2:172.21.15.167:6800/30035 14 ==== osd_op_reply(38 default.realm [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 ==== 157+0+0 (crc 0 0 0) 0x7f5d3800a5a0 con 0x56161bf087f0 2023-10-05T19:28:27.928 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.922+0000 7f5d3f786e40 20 rados_obj.operate() r=-2 bl.length=0 2023-10-05T19:28:27.929 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.922+0000 7f5d3f786e40 20 rados->read ofs=0 len=0 2023-10-05T19:28:27.929 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.922+0000 7f5d3f786e40 1 -- 172.21.15.167:0/2568822831 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:39 5.f 5:f4c53578:::zonegroups_names.default:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x56161bf3ed80 con 0x56161bf087f0 2023-10-05T19:28:27.929 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.922+0000 7f5d3dabc700 1 -- 172.21.15.167:0/2568822831 <== osd.1 v2:172.21.15.167:6800/30035 15 ==== osd_op_reply(39 zonegroups_names.default [read 0~46 out=46b] v0'0 uv2 ondisk = 0) v8 ==== 168+0+46 (crc 0 0 0) 0x7f5d3800a5a0 con 0x56161bf087f0 2023-10-05T19:28:27.929 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.922+0000 7f5d3f786e40 20 rados_obj.operate() r=0 bl.length=46 2023-10-05T19:28:27.929 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.922+0000 7f5d3f786e40 20 rados->read ofs=0 len=0 2023-10-05T19:28:27.929 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.922+0000 7f5d3f786e40 1 -- 172.21.15.167:0/2568822831 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:40 5.c 5:30d299a5:::zonegroup_info.4f1f719d-9648-417b-b10c-e5d939825419:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x56161bf3ed80 con 0x7f5d28007030 2023-10-05T19:28:27.930 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.922+0000 7f5d3dabc700 1 -- 172.21.15.167:0/2568822831 <== osd.0 v2:172.21.15.118:6800/27690 10 ==== osd_op_reply(40 zonegroup_info.4f1f719d-9648-417b-b10c-e5d939825419 [read 0~362 out=362b] v0'0 uv1 ondisk = 0) v8 ==== 195+0+362 (crc 0 0 0) 0x7f5d3800a5a0 con 0x7f5d28007030 2023-10-05T19:28:27.930 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.922+0000 7f5d3f786e40 20 rados_obj.operate() r=0 bl.length=362 2023-10-05T19:28:27.930 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.922+0000 7f5d3f786e40 20 rados->read ofs=0 len=0 2023-10-05T19:28:27.930 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.922+0000 7f5d3f786e40 1 -- 172.21.15.167:0/2568822831 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:41 5.13 5:cbbf6c60:::converted:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x56161bf3ed80 con 0x56161bf3f710 2023-10-05T19:28:27.930 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.922+0000 7f5d3d2bb700 1 -- 172.21.15.167:0/2568822831 <== osd.3 v2:172.21.15.167:6808/30036 10 ==== osd_op_reply(41 converted [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 ==== 153+0+0 (crc 0 0 0) 0x7f5d300173e0 con 0x56161bf3f710 2023-10-05T19:28:27.930 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.926+0000 7f5d3f786e40 20 rados_obj.operate() r=-2 bl.length=0 2023-10-05T19:28:27.931 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.926+0000 7f5d3f786e40 20 rados->read ofs=0 len=0 2023-10-05T19:28:27.931 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.926+0000 7f5d3f786e40 1 -- 172.21.15.167:0/2568822831 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:42 5.12 5:49953fa1:::default.realm:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x56161bf424a0 con 0x56161bf087f0 2023-10-05T19:28:27.931 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.926+0000 7f5d3dabc700 1 -- 172.21.15.167:0/2568822831 <== osd.1 v2:172.21.15.167:6800/30035 16 ==== osd_op_reply(42 default.realm [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 ==== 157+0+0 (crc 0 0 0) 0x7f5d3800a5a0 con 0x56161bf087f0 2023-10-05T19:28:27.932 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.926+0000 7f5d3f786e40 20 rados_obj.operate() r=-2 bl.length=0 2023-10-05T19:28:27.932 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.926+0000 7f5d3f786e40 20 rados->read ofs=0 len=0 2023-10-05T19:28:27.932 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.926+0000 7f5d3f786e40 1 -- 172.21.15.167:0/2568822831 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:43 5.f 5:f4c53578:::zonegroups_names.default:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x56161bf3e920 con 0x56161bf087f0 2023-10-05T19:28:27.932 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.926+0000 7f5d3dabc700 1 -- 172.21.15.167:0/2568822831 <== osd.1 v2:172.21.15.167:6800/30035 17 ==== osd_op_reply(43 zonegroups_names.default [read 0~46 out=46b] v0'0 uv2 ondisk = 0) v8 ==== 168+0+46 (crc 0 0 0) 0x7f5d3800a5a0 con 0x56161bf087f0 2023-10-05T19:28:27.932 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.926+0000 7f5d3f786e40 20 rados_obj.operate() r=0 bl.length=46 2023-10-05T19:28:27.932 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.926+0000 7f5d3f786e40 1 -- 172.21.15.167:0/2568822831 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:44 5.0 5:00000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x56161bf3e8f0 con 0x56161bf3f710 2023-10-05T19:28:27.933 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.926+0000 7f5d3d2bb700 1 -- 172.21.15.167:0/2568822831 <== osd.3 v2:172.21.15.167:6808/30036 11 ==== osd_op_reply(44 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f5d300173e0 con 0x56161bf3f710 2023-10-05T19:28:27.934 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.926+0000 7f5d3d2bb700 1 -- 172.21.15.167:0/2568822831 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:45 5.10 5:08000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x56161bf3e8f0 con 0x7f5d3000d3d0 2023-10-05T19:28:27.934 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.930+0000 7f5d3caba700 1 -- 172.21.15.167:0/2568822831 <== osd.2 v2:172.21.15.167:6816/30037 7 ==== osd_op_reply(45 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f5d28008ae0 con 0x7f5d3000d3d0 2023-10-05T19:28:27.934 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.930+0000 7f5d3caba700 1 -- 172.21.15.167:0/2568822831 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:46 5.8 5:10000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x56161bf3e8f0 con 0x7f5d28007030 2023-10-05T19:28:27.935 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.930+0000 7f5d3dabc700 1 -- 172.21.15.167:0/2568822831 <== osd.0 v2:172.21.15.118:6800/27690 11 ==== osd_op_reply(46 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f5d3800a5a0 con 0x7f5d28007030 2023-10-05T19:28:27.936 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.930+0000 7f5d3dabc700 1 -- 172.21.15.167:0/2568822831 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:47 5.18 5:18000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x56161bf3e490 con 0x56161bf087f0 2023-10-05T19:28:27.936 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.930+0000 7f5d3dabc700 1 -- 172.21.15.167:0/2568822831 <== osd.1 v2:172.21.15.167:6800/30035 18 ==== osd_op_reply(47 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f5d3800a5a0 con 0x56161bf087f0 2023-10-05T19:28:27.936 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.930+0000 7f5d3dabc700 1 -- 172.21.15.167:0/2568822831 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:48 5.4 5:20000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x56161bf3e490 con 0x56161bf087f0 2023-10-05T19:28:27.937 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.930+0000 7f5d3dabc700 1 -- 172.21.15.167:0/2568822831 <== osd.1 v2:172.21.15.167:6800/30035 19 ==== osd_op_reply(48 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f5d3800a5a0 con 0x56161bf087f0 2023-10-05T19:28:27.937 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.930+0000 7f5d3dabc700 1 -- 172.21.15.167:0/2568822831 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:49 5.14 5:28000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x56161bf3e490 con 0x7f5d28007030 2023-10-05T19:28:27.939 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.934+0000 7f5d3dabc700 1 -- 172.21.15.167:0/2568822831 <== osd.0 v2:172.21.15.118:6800/27690 12 ==== osd_op_reply(49 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f5d3800a5a0 con 0x7f5d28007030 2023-10-05T19:28:27.939 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.934+0000 7f5d3dabc700 1 -- 172.21.15.167:0/2568822831 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:50 5.c 5:30000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x56161bf3e490 con 0x7f5d28007030 2023-10-05T19:28:27.939 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.934+0000 7f5d3dabc700 1 -- 172.21.15.167:0/2568822831 <== osd.0 v2:172.21.15.118:6800/27690 13 ==== osd_op_reply(50 [pgnls start_epoch 30 out=112b] v23'1 uv1 ondisk = 1) v8 ==== 144+0+112 (crc 0 0 0) 0x7f5d3800a5a0 con 0x7f5d28007030 2023-10-05T19:28:27.939 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.934+0000 7f5d3dabc700 1 -- 172.21.15.167:0/2568822831 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:51 5.1c 5:38000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x56161bf3e490 con 0x7f5d3000d3d0 2023-10-05T19:28:27.940 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.934+0000 7f5d3caba700 1 -- 172.21.15.167:0/2568822831 <== osd.2 v2:172.21.15.167:6816/30037 8 ==== osd_op_reply(51 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f5d28008ae0 con 0x7f5d3000d3d0 2023-10-05T19:28:27.940 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.934+0000 7f5d3caba700 1 -- 172.21.15.167:0/2568822831 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:52 5.2 5:40000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x56161bf3e490 con 0x56161bf3f710 2023-10-05T19:28:27.941 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.934+0000 7f5d3d2bb700 1 -- 172.21.15.167:0/2568822831 <== osd.3 v2:172.21.15.167:6808/30036 12 ==== osd_op_reply(52 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f5d300173e0 con 0x56161bf3f710 2023-10-05T19:28:27.941 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.934+0000 7f5d3d2bb700 1 -- 172.21.15.167:0/2568822831 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:53 5.12 5:48000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x56161bf3e490 con 0x56161bf087f0 2023-10-05T19:28:27.942 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.938+0000 7f5d3dabc700 1 -- 172.21.15.167:0/2568822831 <== osd.1 v2:172.21.15.167:6800/30035 20 ==== osd_op_reply(53 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f5d3800a5a0 con 0x56161bf087f0 2023-10-05T19:28:27.942 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.938+0000 7f5d3dabc700 1 -- 172.21.15.167:0/2568822831 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:54 5.a 5:50000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x56161bf3e490 con 0x7f5d28007030 2023-10-05T19:28:27.943 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.938+0000 7f5d3dabc700 1 -- 172.21.15.167:0/2568822831 <== osd.0 v2:172.21.15.118:6800/27690 14 ==== osd_op_reply(54 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f5d3800a5a0 con 0x7f5d28007030 2023-10-05T19:28:27.943 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.938+0000 7f5d3dabc700 1 -- 172.21.15.167:0/2568822831 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:55 5.1a 5:58000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x56161bf3e490 con 0x56161bf087f0 2023-10-05T19:28:27.944 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.938+0000 7f5d3dabc700 1 -- 172.21.15.167:0/2568822831 <== osd.1 v2:172.21.15.167:6800/30035 21 ==== osd_op_reply(55 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f5d3800a5a0 con 0x56161bf087f0 2023-10-05T19:28:27.944 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.938+0000 7f5d3dabc700 1 -- 172.21.15.167:0/2568822831 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:56 5.6 5:60000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x56161bf3e490 con 0x7f5d3000d3d0 2023-10-05T19:28:27.945 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.938+0000 7f5d3caba700 1 -- 172.21.15.167:0/2568822831 <== osd.2 v2:172.21.15.167:6816/30037 9 ==== osd_op_reply(56 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f5d28008ae0 con 0x7f5d3000d3d0 2023-10-05T19:28:27.945 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.938+0000 7f5d3caba700 1 -- 172.21.15.167:0/2568822831 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:57 5.16 5:68000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x56161bf3e490 con 0x56161bf3f710 2023-10-05T19:28:27.946 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.938+0000 7f5d3d2bb700 1 -- 172.21.15.167:0/2568822831 <== osd.3 v2:172.21.15.167:6808/30036 13 ==== osd_op_reply(57 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f5d300173e0 con 0x56161bf3f710 2023-10-05T19:28:27.946 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.938+0000 7f5d3d2bb700 1 -- 172.21.15.167:0/2568822831 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:58 5.e 5:70000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x56161bf3e490 con 0x56161bf3f710 2023-10-05T19:28:27.946 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.942+0000 7f5d3d2bb700 1 -- 172.21.15.167:0/2568822831 <== osd.3 v2:172.21.15.167:6808/30036 14 ==== osd_op_reply(58 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f5d300173e0 con 0x56161bf3f710 2023-10-05T19:28:27.946 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.942+0000 7f5d3d2bb700 1 -- 172.21.15.167:0/2568822831 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:59 5.1e 5:78000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x56161bf3e490 con 0x7f5d28007030 2023-10-05T19:28:27.947 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.942+0000 7f5d3dabc700 1 -- 172.21.15.167:0/2568822831 <== osd.0 v2:172.21.15.118:6800/27690 15 ==== osd_op_reply(59 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f5d3803f040 con 0x7f5d28007030 2023-10-05T19:28:27.947 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.942+0000 7f5d3dabc700 1 -- 172.21.15.167:0/2568822831 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:60 5.1 5:80000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x56161bf3e490 con 0x56161bf3f710 2023-10-05T19:28:27.948 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.942+0000 7f5d3d2bb700 1 -- 172.21.15.167:0/2568822831 <== osd.3 v2:172.21.15.167:6808/30036 15 ==== osd_op_reply(60 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f5d300173e0 con 0x56161bf3f710 2023-10-05T19:28:27.948 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.942+0000 7f5d3d2bb700 1 -- 172.21.15.167:0/2568822831 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:61 5.11 5:88000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x56161bf3e490 con 0x56161bf3f710 2023-10-05T19:28:27.949 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.942+0000 7f5d3d2bb700 1 -- 172.21.15.167:0/2568822831 <== osd.3 v2:172.21.15.167:6808/30036 16 ==== osd_op_reply(61 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f5d300173e0 con 0x56161bf3f710 2023-10-05T19:28:27.949 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.942+0000 7f5d3d2bb700 1 -- 172.21.15.167:0/2568822831 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:62 5.9 5:90000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x56161bf3e490 con 0x56161bf087f0 2023-10-05T19:28:27.949 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.942+0000 7f5d3dabc700 1 -- 172.21.15.167:0/2568822831 <== osd.1 v2:172.21.15.167:6800/30035 22 ==== osd_op_reply(62 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f5d3803f040 con 0x56161bf087f0 2023-10-05T19:28:27.950 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.942+0000 7f5d3dabc700 1 -- 172.21.15.167:0/2568822831 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:63 5.19 5:98000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x56161bf3e490 con 0x56161bf087f0 2023-10-05T19:28:27.950 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.946+0000 7f5d3dabc700 1 -- 172.21.15.167:0/2568822831 <== osd.1 v2:172.21.15.167:6800/30035 23 ==== osd_op_reply(63 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f5d3803f040 con 0x56161bf087f0 2023-10-05T19:28:27.950 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.946+0000 7f5d3dabc700 1 -- 172.21.15.167:0/2568822831 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:64 5.5 5:a0000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x56161bf3e490 con 0x7f5d28007030 2023-10-05T19:28:27.951 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.946+0000 7f5d3dabc700 1 -- 172.21.15.167:0/2568822831 <== osd.0 v2:172.21.15.118:6800/27690 16 ==== osd_op_reply(64 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f5d3803f040 con 0x7f5d28007030 2023-10-05T19:28:27.951 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.946+0000 7f5d3dabc700 1 -- 172.21.15.167:0/2568822831 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:65 5.15 5:a8000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x56161bf3e490 con 0x56161bf3f710 2023-10-05T19:28:27.952 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.946+0000 7f5d3d2bb700 1 -- 172.21.15.167:0/2568822831 <== osd.3 v2:172.21.15.167:6808/30036 17 ==== osd_op_reply(65 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f5d300173e0 con 0x56161bf3f710 2023-10-05T19:28:27.952 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.946+0000 7f5d3d2bb700 1 -- 172.21.15.167:0/2568822831 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:66 5.d 5:b0000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x56161bf3e490 con 0x7f5d3000d3d0 2023-10-05T19:28:27.953 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.946+0000 7f5d3caba700 1 -- 172.21.15.167:0/2568822831 <== osd.2 v2:172.21.15.167:6816/30037 10 ==== osd_op_reply(66 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f5d28008ae0 con 0x7f5d3000d3d0 2023-10-05T19:28:27.953 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.946+0000 7f5d3caba700 1 -- 172.21.15.167:0/2568822831 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:67 5.1d 5:b8000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x56161bf3e490 con 0x56161bf087f0 2023-10-05T19:28:27.953 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.946+0000 7f5d3dabc700 1 -- 172.21.15.167:0/2568822831 <== osd.1 v2:172.21.15.167:6800/30035 24 ==== osd_op_reply(67 [pgnls start_epoch 30 out=107b] v30'2 uv2 ondisk = 1) v8 ==== 144+0+107 (crc 0 0 0) 0x7f5d3803f040 con 0x56161bf087f0 2023-10-05T19:28:27.954 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.946+0000 7f5d3dabc700 1 -- 172.21.15.167:0/2568822831 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:68 5.3 5:c0000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x56161bf3e490 con 0x7f5d28007030 2023-10-05T19:28:27.955 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.950+0000 7f5d3dabc700 1 -- 172.21.15.167:0/2568822831 <== osd.0 v2:172.21.15.118:6800/27690 17 ==== osd_op_reply(68 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f5d3803f040 con 0x7f5d28007030 2023-10-05T19:28:27.955 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.950+0000 7f5d3dabc700 1 -- 172.21.15.167:0/2568822831 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:69 5.13 5:c8000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x56161bf3e490 con 0x56161bf3f710 2023-10-05T19:28:27.955 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.950+0000 7f5d3d2bb700 1 -- 172.21.15.167:0/2568822831 <== osd.3 v2:172.21.15.167:6808/30036 18 ==== osd_op_reply(69 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f5d300173e0 con 0x56161bf3f710 2023-10-05T19:28:27.955 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.950+0000 7f5d3d2bb700 1 -- 172.21.15.167:0/2568822831 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:70 5.b 5:d0000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x56161bf3e490 con 0x7f5d28007030 2023-10-05T19:28:27.956 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.950+0000 7f5d3dabc700 1 -- 172.21.15.167:0/2568822831 <== osd.0 v2:172.21.15.118:6800/27690 18 ==== osd_op_reply(70 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f5d3803f040 con 0x7f5d28007030 2023-10-05T19:28:27.956 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.950+0000 7f5d3dabc700 1 -- 172.21.15.167:0/2568822831 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:71 5.1b 5:d8000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x56161bf3e490 con 0x7f5d3000d3d0 2023-10-05T19:28:27.957 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.950+0000 7f5d3caba700 1 -- 172.21.15.167:0/2568822831 <== osd.2 v2:172.21.15.167:6816/30037 11 ==== osd_op_reply(71 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f5d28008ae0 con 0x7f5d3000d3d0 2023-10-05T19:28:27.957 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.950+0000 7f5d3caba700 1 -- 172.21.15.167:0/2568822831 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:72 5.7 5:e0000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x56161bf3e490 con 0x56161bf3f710 2023-10-05T19:28:27.958 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.954+0000 7f5d3d2bb700 1 -- 172.21.15.167:0/2568822831 <== osd.3 v2:172.21.15.167:6808/30036 19 ==== osd_op_reply(72 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f5d30050040 con 0x56161bf3f710 2023-10-05T19:28:27.958 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.954+0000 7f5d3d2bb700 1 -- 172.21.15.167:0/2568822831 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:73 5.17 5:e8000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x56161bf3e490 con 0x7f5d28007030 2023-10-05T19:28:27.959 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.954+0000 7f5d3dabc700 1 -- 172.21.15.167:0/2568822831 <== osd.0 v2:172.21.15.118:6800/27690 19 ==== osd_op_reply(73 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f5d3803f040 con 0x7f5d28007030 2023-10-05T19:28:27.959 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.954+0000 7f5d3dabc700 1 -- 172.21.15.167:0/2568822831 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:74 5.f 5:f0000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x56161bf3e490 con 0x56161bf087f0 2023-10-05T19:28:27.960 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.954+0000 7f5d3dabc700 1 -- 172.21.15.167:0/2568822831 <== osd.1 v2:172.21.15.167:6800/30035 25 ==== osd_op_reply(74 [pgnls start_epoch 30 out=115b] v23'2 uv2 ondisk = 1) v8 ==== 144+0+115 (crc 0 0 0) 0x7f5d3803f040 con 0x56161bf087f0 2023-10-05T19:28:27.960 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.954+0000 7f5d3dabc700 1 -- 172.21.15.167:0/2568822831 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:75 5.1f 5:f8000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x56161bf3e490 con 0x7f5d3000d3d0 2023-10-05T19:28:27.961 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.954+0000 7f5d3caba700 1 -- 172.21.15.167:0/2568822831 <== osd.2 v2:172.21.15.167:6816/30037 12 ==== osd_op_reply(75 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f5d28008ae0 con 0x7f5d3000d3d0 2023-10-05T19:28:27.961 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.954+0000 7f5d3f786e40 20 RGWRados::pool_iterate: got zonegroup_info.4f1f719d-9648-417b-b10c-e5d939825419 2023-10-05T19:28:27.961 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.954+0000 7f5d3f786e40 20 RGWRados::pool_iterate: got zone_info.9a9df965-6612-45d4-a45f-6e3c889bacc2 2023-10-05T19:28:27.961 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.954+0000 7f5d3f786e40 20 RGWRados::pool_iterate: got zone_names.default 2023-10-05T19:28:27.962 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.954+0000 7f5d3f786e40 20 RGWRados::pool_iterate: got zonegroups_names.default 2023-10-05T19:28:27.962 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.954+0000 7f5d3f786e40 20 rados->read ofs=0 len=0 2023-10-05T19:28:27.962 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.954+0000 7f5d3f786e40 1 -- 172.21.15.167:0/2568822831 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:76 5.f 5:f43b6ece:::zone_names.default:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x56161bf3e8f0 con 0x56161bf087f0 2023-10-05T19:28:27.963 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.954+0000 7f5d3dabc700 1 -- 172.21.15.167:0/2568822831 <== osd.1 v2:172.21.15.167:6800/30035 26 ==== osd_op_reply(76 zone_names.default [read 0~46 out=46b] v0'0 uv1 ondisk = 0) v8 ==== 162+0+46 (crc 0 0 0) 0x7f5d3803f040 con 0x56161bf087f0 2023-10-05T19:28:27.963 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.958+0000 7f5d3f786e40 20 rados_obj.operate() r=0 bl.length=46 2023-10-05T19:28:27.963 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.958+0000 7f5d3f786e40 20 rados->read ofs=0 len=0 2023-10-05T19:28:27.963 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.958+0000 7f5d3f786e40 1 -- 172.21.15.167:0/2568822831 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:77 5.1d 5:b8374eb4:::zone_info.9a9df965-6612-45d4-a45f-6e3c889bacc2:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x56161bf42990 con 0x56161bf087f0 2023-10-05T19:28:27.963 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.958+0000 7f5d3dabc700 1 -- 172.21.15.167:0/2568822831 <== osd.1 v2:172.21.15.167:6800/30035 27 ==== osd_op_reply(77 zone_info.9a9df965-6612-45d4-a45f-6e3c889bacc2 [read 0~874 out=874b] v0'0 uv2 ondisk = 0) v8 ==== 190+0+874 (crc 0 0 0) 0x7f5d3803f040 con 0x56161bf087f0 2023-10-05T19:28:27.963 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.958+0000 7f5d3f786e40 20 rados_obj.operate() r=0 bl.length=874 2023-10-05T19:28:27.963 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.958+0000 7f5d3f786e40 20 rados->read ofs=0 len=0 2023-10-05T19:28:27.964 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.958+0000 7f5d3f786e40 1 -- 172.21.15.167:0/2568822831 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:78 5.f 5:f4c53578:::zonegroups_names.default:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x56161bf42ba0 con 0x56161bf087f0 2023-10-05T19:28:27.964 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.958+0000 7f5d3dabc700 1 -- 172.21.15.167:0/2568822831 <== osd.1 v2:172.21.15.167:6800/30035 28 ==== osd_op_reply(78 zonegroups_names.default [read 0~46 out=46b] v0'0 uv2 ondisk = 0) v8 ==== 168+0+46 (crc 0 0 0) 0x7f5d3803f040 con 0x56161bf087f0 2023-10-05T19:28:27.964 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.958+0000 7f5d3f786e40 20 rados_obj.operate() r=0 bl.length=46 2023-10-05T19:28:27.965 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.958+0000 7f5d3f786e40 20 rados->read ofs=0 len=0 2023-10-05T19:28:27.965 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.958+0000 7f5d3f786e40 1 -- 172.21.15.167:0/2568822831 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:79 5.c 5:30d299a5:::zonegroup_info.4f1f719d-9648-417b-b10c-e5d939825419:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x56161bf42b80 con 0x7f5d28007030 2023-10-05T19:28:27.966 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.958+0000 7f5d3dabc700 1 -- 172.21.15.167:0/2568822831 <== osd.0 v2:172.21.15.118:6800/27690 20 ==== osd_op_reply(79 zonegroup_info.4f1f719d-9648-417b-b10c-e5d939825419 [read 0~362 out=362b] v0'0 uv1 ondisk = 0) v8 ==== 195+0+362 (crc 0 0 0) 0x7f5d3803f040 con 0x7f5d28007030 2023-10-05T19:28:27.966 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.958+0000 7f5d3f786e40 20 rados_obj.operate() r=0 bl.length=362 2023-10-05T19:28:27.966 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.958+0000 7f5d3f786e40 20 rados->read ofs=0 len=0 2023-10-05T19:28:27.966 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.958+0000 7f5d3f786e40 1 -- 172.21.15.167:0/2568822831 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:80 5.3 5:c0ad8cbd:::region_map:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x56161bf43020 con 0x7f5d28007030 2023-10-05T19:28:27.966 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.958+0000 7f5d3dabc700 1 -- 172.21.15.167:0/2568822831 <== osd.0 v2:172.21.15.118:6800/27690 21 ==== osd_op_reply(80 region_map [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 ==== 154+0+0 (crc 0 0 0) 0x7f5d3803f040 con 0x7f5d28007030 2023-10-05T19:28:27.966 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.958+0000 7f5d3f786e40 20 rados_obj.operate() r=-2 bl.length=0 2023-10-05T19:28:27.966 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.958+0000 7f5d3f786e40 20 zone default found 2023-10-05T19:28:27.967 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.958+0000 7f5d3f786e40 4 Realm: () 2023-10-05T19:28:27.967 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.958+0000 7f5d3f786e40 4 ZoneGroup: default (4f1f719d-9648-417b-b10c-e5d939825419) 2023-10-05T19:28:27.967 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.958+0000 7f5d3f786e40 4 Zone: default (9a9df965-6612-45d4-a45f-6e3c889bacc2) 2023-10-05T19:28:27.967 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.958+0000 7f5d3f786e40 10 cannot find current period zonegroup using local zonegroup configuration 2023-10-05T19:28:27.967 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.958+0000 7f5d3f786e40 20 zonegroup default 2023-10-05T19:28:27.967 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.958+0000 7f5d3f786e40 20 rados->read ofs=0 len=0 2023-10-05T19:28:27.967 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.962+0000 7f5d3f786e40 1 -- 172.21.15.167:0/2568822831 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:81 5.3 5:c52100b6:::period_config.default:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x56161bf43020 con 0x7f5d28007030 2023-10-05T19:28:27.969 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.962+0000 7f5d3dabc700 1 -- 172.21.15.167:0/2568822831 <== osd.0 v2:172.21.15.118:6800/27690 22 ==== osd_op_reply(81 period_config.default [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 ==== 165+0+0 (crc 0 0 0) 0x7f5d3803f040 con 0x7f5d28007030 2023-10-05T19:28:27.969 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.962+0000 7f5d3f786e40 20 rados_obj.operate() r=-2 bl.length=0 2023-10-05T19:28:27.969 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.962+0000 7f5d3f786e40 20 rados->read ofs=0 len=0 2023-10-05T19:28:27.969 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.962+0000 7f5d3f786e40 1 -- 172.21.15.167:0/2568822831 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:82 6.2 6:4347d321:::bucket.sync-source-hints.:head [call version.read in=11b,read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x56161bf44720 con 0x56161bf3f710 2023-10-05T19:28:27.969 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.962+0000 7f5d3d2bb700 1 -- 172.21.15.167:0/2568822831 <== osd.3 v2:172.21.15.167:6808/30036 20 ==== osd_op_reply(82 bucket.sync-source-hints. [call,read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 ==== 211+0+0 (crc 0 0 0) 0x7f5d30050040 con 0x56161bf3f710 2023-10-05T19:28:27.969 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.962+0000 7f5d3f786e40 20 rados_obj.operate() r=-2 bl.length=0 2023-10-05T19:28:27.969 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.962+0000 7f5d3f786e40 20 rados->read ofs=0 len=0 2023-10-05T19:28:27.969 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.962+0000 7f5d3f786e40 1 -- 172.21.15.167:0/2568822831 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:83 6.b 6:d467b91b:::bucket.sync-target-hints.:head [call version.read in=11b,read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x56161bf44ad0 con 0x56161bf3f710 2023-10-05T19:28:27.969 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.962+0000 7f5d3d2bb700 1 -- 172.21.15.167:0/2568822831 <== osd.3 v2:172.21.15.167:6808/30036 21 ==== osd_op_reply(83 bucket.sync-target-hints. [call,read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 ==== 211+0+0 (crc 0 0 0) 0x7f5d30050040 con 0x56161bf3f710 2023-10-05T19:28:27.969 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.962+0000 7f5d3f786e40 20 rados_obj.operate() r=-2 bl.length=0 2023-10-05T19:28:27.969 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.962+0000 7f5d3f786e40 20 started sync module instance, tier type = 2023-10-05T19:28:27.970 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.962+0000 7f5d3f786e40 20 started zone id=9a9df965-6612-45d4-a45f-6e3c889bacc2 (name=default) with tier type = 2023-10-05T19:28:27.970 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.962+0000 7f5d3f786e40 1 -- 172.21.15.167:0/2568822831 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:84 7.1f 7:f95f44c2:::notify.0:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x56161bf442d0 con 0x7f5d3000d3d0 2023-10-05T19:28:27.971 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.966+0000 7f5d3caba700 1 -- 172.21.15.167:0/2568822831 <== osd.2 v2:172.21.15.167:6816/30037 13 ==== osd_op_reply(84 notify.0 [create] v30'54 uv54 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f5d28008ae0 con 0x7f5d3000d3d0 2023-10-05T19:28:27.971 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.966+0000 7f5d3f786e40 1 -- 172.21.15.167:0/2568822831 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:85 7.1f 7:f95f44c2:::notify.0:head [watch watch cookie 94652958260704] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x56161bf427c0 con 0x7f5d3000d3d0 2023-10-05T19:28:27.972 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.966+0000 7f5d3f786e40 1 -- 172.21.15.167:0/2568822831 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:86 7.0 7:05bf5b68:::notify.1:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x56161bf45960 con 0x56161bf087f0 2023-10-05T19:28:27.974 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.970+0000 7f5d3caba700 1 -- 172.21.15.167:0/2568822831 <== osd.2 v2:172.21.15.167:6816/30037 14 ==== osd_op_reply(85 notify.0 [watch watch cookie 94652958260704] v30'55 uv54 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f5d28008ae0 con 0x7f5d3000d3d0 2023-10-05T19:28:27.975 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.970+0000 7f5d3dabc700 1 -- 172.21.15.167:0/2568822831 <== osd.1 v2:172.21.15.167:6800/30035 29 ==== osd_op_reply(86 notify.1 [create] v30'54 uv54 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f5d38052040 con 0x56161bf087f0 2023-10-05T19:28:27.975 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.970+0000 7f5d3f786e40 1 -- 172.21.15.167:0/2568822831 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:87 7.0 7:05bf5b68:::notify.1:head [watch watch cookie 94652958264960] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x56161bf45d60 con 0x56161bf087f0 2023-10-05T19:28:27.975 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.970+0000 7f5d3f786e40 1 -- 172.21.15.167:0/2568822831 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:88 7.15 7:a93a5511:::notify.2:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x56161bf46600 con 0x56161bf3f710 2023-10-05T19:28:27.978 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.974+0000 7f5d3dabc700 1 -- 172.21.15.167:0/2568822831 <== osd.1 v2:172.21.15.167:6800/30035 30 ==== osd_op_reply(87 notify.1 [watch watch cookie 94652958264960] v30'55 uv54 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f5d38052040 con 0x56161bf087f0 2023-10-05T19:28:27.979 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.974+0000 7f5d3d2bb700 1 -- 172.21.15.167:0/2568822831 <== osd.3 v2:172.21.15.167:6808/30036 22 ==== osd_op_reply(88 notify.2 [create] v30'54 uv54 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f5d30050040 con 0x56161bf3f710 2023-10-05T19:28:27.979 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.974+0000 7f5d3f786e40 1 -- 172.21.15.167:0/2568822831 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:89 7.15 7:a93a5511:::notify.2:head [watch watch cookie 94652958266416] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x56161bf46250 con 0x56161bf3f710 2023-10-05T19:28:27.979 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.974+0000 7f5d3f786e40 1 -- 172.21.15.167:0/2568822831 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:90 7.e 7:7759931f:::notify.3:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x56161bf47110 con 0x7f5d3000d3d0 2023-10-05T19:28:27.981 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.974+0000 7f5d3caba700 1 -- 172.21.15.167:0/2568822831 <== osd.2 v2:172.21.15.167:6816/30037 15 ==== osd_op_reply(90 notify.3 [create] v30'54 uv54 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f5d28008ae0 con 0x7f5d3000d3d0 2023-10-05T19:28:27.982 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.974+0000 7f5d3f786e40 1 -- 172.21.15.167:0/2568822831 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:91 7.e 7:7759931f:::notify.3:head [watch watch cookie 94652958274832] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x56161bf47e50 con 0x7f5d3000d3d0 2023-10-05T19:28:27.982 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.974+0000 7f5d3f786e40 1 -- 172.21.15.167:0/2568822831 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:92 7.d 7:b4812045:::notify.4:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x56161bf48860 con 0x56161bf087f0 2023-10-05T19:28:27.982 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.978+0000 7f5d3d2bb700 1 -- 172.21.15.167:0/2568822831 <== osd.3 v2:172.21.15.167:6808/30036 23 ==== osd_op_reply(89 notify.2 [watch watch cookie 94652958266416] v30'55 uv54 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f5d30050040 con 0x56161bf3f710 2023-10-05T19:28:27.984 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.978+0000 7f5d3caba700 1 -- 172.21.15.167:0/2568822831 <== osd.2 v2:172.21.15.167:6816/30037 16 ==== osd_op_reply(91 notify.3 [watch watch cookie 94652958274832] v30'55 uv54 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f5d28008ae0 con 0x7f5d3000d3d0 2023-10-05T19:28:27.984 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.978+0000 7f5d3dabc700 1 -- 172.21.15.167:0/2568822831 <== osd.1 v2:172.21.15.167:6800/30035 31 ==== osd_op_reply(92 notify.4 [create] v30'54 uv54 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f5d38052040 con 0x56161bf087f0 2023-10-05T19:28:27.985 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.978+0000 7f5d3f786e40 1 -- 172.21.15.167:0/2568822831 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:93 7.d 7:b4812045:::notify.4:head [watch watch cookie 94652958268688] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x56161bf48860 con 0x56161bf087f0 2023-10-05T19:28:27.985 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.978+0000 7f5d3f786e40 1 -- 172.21.15.167:0/2568822831 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:94 7.3 7:c609908c:::notify.5:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x56161bf49100 con 0x7f5d28007030 2023-10-05T19:28:27.987 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.982+0000 7f5d3dabc700 1 -- 172.21.15.167:0/2568822831 <== osd.1 v2:172.21.15.167:6800/30035 32 ==== osd_op_reply(93 notify.4 [watch watch cookie 94652958268688] v30'55 uv54 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f5d38052040 con 0x56161bf087f0 2023-10-05T19:28:27.987 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.982+0000 7f5d3dabc700 1 -- 172.21.15.167:0/2568822831 <== osd.0 v2:172.21.15.118:6800/27690 23 ==== osd_op_reply(94 notify.5 [create] v30'54 uv54 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f5d38052040 con 0x7f5d28007030 2023-10-05T19:28:27.987 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.982+0000 7f5d3f786e40 1 -- 172.21.15.167:0/2568822831 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:95 7.3 7:c609908c:::notify.5:head [watch watch cookie 94652958279072] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x56161bf48ee0 con 0x7f5d28007030 2023-10-05T19:28:27.987 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.982+0000 7f5d3f786e40 1 -- 172.21.15.167:0/2568822831 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:96 7.14 7:2b04a3e9:::notify.6:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x56161bf49960 con 0x7f5d28007030 2023-10-05T19:28:27.989 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.982+0000 7f5d3dabc700 1 -- 172.21.15.167:0/2568822831 <== osd.0 v2:172.21.15.118:6800/27690 24 ==== osd_op_reply(95 notify.5 [watch watch cookie 94652958279072] v30'55 uv54 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f5d38052040 con 0x7f5d28007030 2023-10-05T19:28:27.991 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.986+0000 7f5d3dabc700 1 -- 172.21.15.167:0/2568822831 <== osd.0 v2:172.21.15.118:6800/27690 25 ==== osd_op_reply(96 notify.6 [create] v30'54 uv54 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f5d38052040 con 0x7f5d28007030 2023-10-05T19:28:27.991 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.986+0000 7f5d3f786e40 1 -- 172.21.15.167:0/2568822831 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:97 7.14 7:2b04a3e9:::notify.6:head [watch watch cookie 94652958282800] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x56161bf48ee0 con 0x7f5d28007030 2023-10-05T19:28:27.991 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.986+0000 7f5d3f786e40 1 -- 172.21.15.167:0/2568822831 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:98 7.9 7:93e5b521:::notify.7:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x56161bf4a320 con 0x7f5d28007030 2023-10-05T19:28:27.993 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.986+0000 7f5d3dabc700 1 -- 172.21.15.167:0/2568822831 <== osd.0 v2:172.21.15.118:6800/27690 26 ==== osd_op_reply(98 notify.7 [create] v30'54 uv54 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f5d38052040 con 0x7f5d28007030 2023-10-05T19:28:27.994 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.986+0000 7f5d3f786e40 1 -- 172.21.15.167:0/2568822831 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:99 7.9 7:93e5b521:::notify.7:head [watch watch cookie 94652958285296] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x56161bf4a730 con 0x7f5d28007030 2023-10-05T19:28:27.994 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.986+0000 7f5d3f786e40 20 add_watcher() i=0 2023-10-05T19:28:27.994 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.986+0000 7f5d3f786e40 20 add_watcher() i=1 2023-10-05T19:28:27.994 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.986+0000 7f5d3f786e40 20 add_watcher() i=2 2023-10-05T19:28:27.994 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.986+0000 7f5d3f786e40 20 add_watcher() i=3 2023-10-05T19:28:27.994 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.986+0000 7f5d3f786e40 20 add_watcher() i=4 2023-10-05T19:28:27.994 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.986+0000 7f5d3f786e40 20 add_watcher() i=5 2023-10-05T19:28:27.994 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.986+0000 7f5d3dabc700 1 -- 172.21.15.167:0/2568822831 <== osd.0 v2:172.21.15.118:6800/27690 27 ==== osd_op_reply(97 notify.6 [watch watch cookie 94652958282800] v30'55 uv54 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f5d38052040 con 0x7f5d28007030 2023-10-05T19:28:27.994 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.986+0000 7f5d3f786e40 20 add_watcher() i=6 2023-10-05T19:28:27.996 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.990+0000 7f5d3dabc700 1 -- 172.21.15.167:0/2568822831 <== osd.0 v2:172.21.15.118:6800/27690 28 ==== osd_op_reply(99 notify.7 [watch watch cookie 94652958285296] v30'55 uv54 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f5d38052040 con 0x7f5d28007030 2023-10-05T19:28:27.996 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.990+0000 7f5d3f786e40 20 add_watcher() i=7 2023-10-05T19:28:27.996 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.990+0000 7f5d3f786e40 2 all 8 watchers are set, enabling cache 2023-10-05T19:28:27.996 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.990+0000 7f5d3f786e40 1 -- 172.21.15.167:0/2568822831 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:100 6.4 6:22d26bf9:::data_loggenerations_metadata:head [call version.check_conds in=74b,call version.read in=11b,read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x56161bf4a730 con 0x56161bf087f0 2023-10-05T19:28:27.997 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.990+0000 7f5d3dabc700 1 -- 172.21.15.167:0/2568822831 <== osd.1 v2:172.21.15.167:6800/30035 33 ==== osd_op_reply(100 data_loggenerations_metadata [call,call out=48b,read 0~28 out=28b] v0'0 uv1 ondisk = 0) v8 ==== 256+0+76 (crc 0 0 0) 0x7f5d38052040 con 0x56161bf087f0 2023-10-05T19:28:27.997 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.990+0000 7f5d3f786e40 1 -- 172.21.15.167:0/2568822831 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:101 6.4 6:22d26bf9:::data_loggenerations_metadata:head [watch watch cookie 94652958286400] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x56161bf4a690 con 0x56161bf087f0 2023-10-05T19:28:27.999 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.994+0000 7f5d3dabc700 1 -- 172.21.15.167:0/2568822831 <== osd.1 v2:172.21.15.167:6800/30035 34 ==== osd_op_reply(101 data_loggenerations_metadata [watch watch cookie 94652958286400] v30'70 uv1 ondisk = 0) v8 ==== 172+0+0 (crc 0 0 0) 0x7f5d38052040 con 0x56161bf087f0 2023-10-05T19:28:27.999 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.994+0000 7f5d0e7e4700 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2023-10-05T19:28:27.999 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.994+0000 7f5d3f786e40 20 check_secure_mon_conn(): auth registy supported: methods=[2] modes=[2,1] 2023-10-05T19:28:28.000 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.994+0000 7f5d3f786e40 20 check_secure_mon_conn(): mode 1 is insecure 2023-10-05T19:28:28.000 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.994+0000 7f5d3f786e40 1 Processor -- start 2023-10-05T19:28:28.000 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.994+0000 7f5d3f786e40 1 -- start start 2023-10-05T19:28:28.000 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.994+0000 7f5d3f786e40 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x56161bf57d60 0x56161bf58130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-05T19:28:28.001 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.994+0000 7f5d3f786e40 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x56161bf58670 con 0x56161bf57d60 2023-10-05T19:28:28.001 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.994+0000 7f5d3caba700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x56161bf57d60 0x56161bf58130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-05T19:28:28.001 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.994+0000 7f5d3caba700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x56161bf57d60 0x56161bf58130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.118:3300/0 says I am v2:172.21.15.167:57620/0 (socket says 172.21.15.167:57620) 2023-10-05T19:28:28.002 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.994+0000 7f5d3caba700 1 -- 172.21.15.167:0/3415519348 learned_addr learned my addr 172.21.15.167:0/3415519348 (peer_addr_for_me v2:172.21.15.167:0/0) 2023-10-05T19:28:28.002 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.994+0000 7f5d3caba700 1 -- 172.21.15.167:0/3415519348 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x56161bf587b0 con 0x56161bf57d60 2023-10-05T19:28:28.002 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.994+0000 7f5d3caba700 1 --2- 172.21.15.167:0/3415519348 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x56161bf57d60 0x56161bf58130 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x56161bf07cb0 tx=0x7f5d28012000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7533a6be4f436f08 server_cookie=0 in_seq=0 out_seq=0 2023-10-05T19:28:28.002 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.998+0000 7f5d0dfe3700 1 -- 172.21.15.167:0/3415519348 <== mon.0 v2:172.21.15.118:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f5d28012c70 con 0x56161bf57d60 2023-10-05T19:28:28.002 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.998+0000 7f5d0dfe3700 1 -- 172.21.15.167:0/3415519348 <== mon.0 v2:172.21.15.118:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5d28012dd0 con 0x56161bf57d60 2023-10-05T19:28:28.003 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.998+0000 7f5d0dfe3700 1 -- 172.21.15.167:0/3415519348 <== mon.0 v2:172.21.15.118:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f5d2802a8a0 con 0x56161bf57d60 2023-10-05T19:28:28.003 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.998+0000 7f5d3f786e40 1 -- 172.21.15.167:0/3415519348 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x56161bf57d60 msgr2=0x56161bf58130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-05T19:28:28.003 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.998+0000 7f5d3f786e40 1 --2- 172.21.15.167:0/3415519348 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x56161bf57d60 0x56161bf58130 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x56161bf07cb0 tx=0x7f5d28012000 comp rx=0 tx=0).stop 2023-10-05T19:28:28.004 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.998+0000 7f5d3f786e40 1 -- 172.21.15.167:0/3415519348 shutdown_connections 2023-10-05T19:28:28.004 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.998+0000 7f5d3f786e40 1 --2- 172.21.15.167:0/3415519348 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x56161bf57d60 0x56161bf58130 unknown :-1 s=CLOSED pgs=246 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-05T19:28:28.004 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.998+0000 7f5d3f786e40 1 -- 172.21.15.167:0/3415519348 >> 172.21.15.167:0/3415519348 conn(0x56161bf53120 msgr2=0x56161bf51270 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-05T19:28:28.004 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.998+0000 7f5d3f786e40 1 -- 172.21.15.167:0/3415519348 shutdown_connections 2023-10-05T19:28:28.004 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.998+0000 7f5d3f786e40 1 -- 172.21.15.167:0/3415519348 wait complete. 2023-10-05T19:28:28.004 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.998+0000 7f5d3f786e40 1 Processor -- start 2023-10-05T19:28:28.005 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.998+0000 7f5d3f786e40 1 -- start start 2023-10-05T19:28:28.005 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.998+0000 7f5d3f786e40 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x56161bf57d60 0x56161bf5d890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-05T19:28:28.005 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.998+0000 7f5d3f786e40 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x56161bf5e890 con 0x56161bf57d60 2023-10-05T19:28:28.005 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.998+0000 7f5d3caba700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x56161bf57d60 0x56161bf5d890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-05T19:28:28.006 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.998+0000 7f5d3caba700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x56161bf57d60 0x56161bf5d890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.118:3300/0 says I am v2:172.21.15.167:57626/0 (socket says 172.21.15.167:57626) 2023-10-05T19:28:28.006 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.998+0000 7f5d3caba700 1 -- 172.21.15.167:0/2929312097 learned_addr learned my addr 172.21.15.167:0/2929312097 (peer_addr_for_me v2:172.21.15.167:0/0) 2023-10-05T19:28:28.006 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:27.998+0000 7f5d3caba700 1 -- 172.21.15.167:0/2929312097 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x56161bf07ce0 con 0x56161bf57d60 2023-10-05T19:28:28.007 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.002+0000 7f5d3caba700 1 --2- 172.21.15.167:0/2929312097 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x56161bf57d60 0x56161bf5d890 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x56161bf51de0 tx=0x7f5d2802f000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-05T19:28:28.007 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.002+0000 7f5c7ffff700 1 -- 172.21.15.167:0/2929312097 <== mon.0 v2:172.21.15.118:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f5d2803e5d0 con 0x56161bf57d60 2023-10-05T19:28:28.007 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.002+0000 7f5c7ffff700 1 -- 172.21.15.167:0/2929312097 <== mon.0 v2:172.21.15.118:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5d2803e730 con 0x56161bf57d60 2023-10-05T19:28:28.007 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.002+0000 7f5d3f786e40 1 -- 172.21.15.167:0/2929312097 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x56161bf5de90 con 0x56161bf57d60 2023-10-05T19:28:28.007 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.002+0000 7f5c7ffff700 1 -- 172.21.15.167:0/2929312097 <== mon.0 v2:172.21.15.118:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f5d2803e8e0 con 0x56161bf57d60 2023-10-05T19:28:28.007 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.002+0000 7f5d3f786e40 1 -- 172.21.15.167:0/2929312097 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x56161bf5e270 con 0x56161bf57d60 2023-10-05T19:28:28.008 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.002+0000 7f5c7ffff700 1 -- 172.21.15.167:0/2929312097 <== mon.0 v2:172.21.15.118:3300/0 4 ==== mgrmap(e 5) v1 ==== 40329+0+0 (secure 0 0 0) 0x7f5d28046070 con 0x56161bf57d60 2023-10-05T19:28:28.008 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.002+0000 7f5c7ffff700 1 --2- 172.21.15.167:0/2929312097 >> [v2:172.21.15.167:6824/29889,v1:172.21.15.167:6825/29889] conn(0x7f5c84030c50 0x7f5c84033100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-05T19:28:28.009 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.002+0000 7f5c7ffff700 1 -- 172.21.15.167:0/2929312097 <== mon.0 v2:172.21.15.118:3300/0 5 ==== osd_map(30..30 src has 1..30) v4 ==== 6057+0+0 (secure 0 0 0) 0x7f5d2802f070 con 0x56161bf57d60 2023-10-05T19:28:28.009 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.002+0000 7f5d3dabc700 1 --2- 172.21.15.167:0/2929312097 >> [v2:172.21.15.167:6824/29889,v1:172.21.15.167:6825/29889] conn(0x7f5c84030c50 0x7f5c84033100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-05T19:28:28.009 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.002+0000 7f5d3f786e40 5 note: GC not initialized 2023-10-05T19:28:28.009 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.002+0000 7f5d3dabc700 1 --2- 172.21.15.167:0/2929312097 >> [v2:172.21.15.167:6824/29889,v1:172.21.15.167:6825/29889] conn(0x7f5c84030c50 0x7f5c84033100 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x56161bf48c90 tx=0x7f5d3800a280 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-05T19:28:28.011 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.006+0000 7f5c7d7fa700 20 reqs_thread_entry: start 2023-10-05T19:28:28.012 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.006+0000 7f5c76ffd700 20 reqs_thread_entry: start 2023-10-05T19:28:28.012 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.006+0000 7f5d3f786e40 20 init_complete bucket index max shards: 11 2023-10-05T19:28:28.013 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.006+0000 7f5d3f786e40 10 rgw notify: Started notification manager with: 1 workers 2023-10-05T19:28:28.013 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.006+0000 7f5c757fa700 20 rgw notify: INFO: next queues processing will happen at: Thu Oct 5 19:28:58 2023 2023-10-05T19:28:28.013 INFO:teuthology.orchestra.run.smithi167.stderr: 2023-10-05T19:28:28.013 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.006+0000 7f5c74ff9700 20 reqs_thread_entry: start 2023-10-05T19:28:28.013 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.006+0000 7f5d3f786e40 20 get_system_obj_state: rctx=0x7ffefdc2f7d8 obj=default.rgw.meta:users.uid:foo state=0x56161bde10b0 s->prefetch_data=0 2023-10-05T19:28:28.013 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.006+0000 7f5d3f786e40 10 cache get: name=default.rgw.meta+users.uid+foo : miss 2023-10-05T19:28:28.013 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.006+0000 7f5d3f786e40 1 -- 172.21.15.167:0/2568822831 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:102 8.18 8:1e6ca5ed:users.uid::foo:head [call version.read in=11b,getxattrs,stat] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x56161bfc6080 con 0x56161bf087f0 2023-10-05T19:28:28.015 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.006+0000 7f5d3dabc700 1 -- 172.21.15.167:0/2568822831 <== osd.1 v2:172.21.15.167:6800/30035 35 ==== osd_op_reply(102 foo [call out=48b,getxattrs out=75b,stat out=16b] v0'0 uv5 ondisk = 0) v8 ==== 231+0+139 (crc 0 0 0) 0x7f5d38052040 con 0x56161bf087f0 2023-10-05T19:28:28.015 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.006+0000 7f5d3f786e40 10 cache put: name=default.rgw.meta+users.uid+foo info.flags=0x16 2023-10-05T19:28:28.015 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.006+0000 7f5d3f786e40 10 adding default.rgw.meta+users.uid+foo to cache LRU end 2023-10-05T19:28:28.015 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.006+0000 7f5d3f786e40 10 updating xattr: name=ceph.objclass.version bl.length()=42 2023-10-05T19:28:28.015 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.006+0000 7f5d3f786e40 20 get_system_obj_state: s->obj_tag was set empty 2023-10-05T19:28:28.015 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.006+0000 7f5d3f786e40 20 Read xattr: user.rgw.idtag 2023-10-05T19:28:28.015 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.006+0000 7f5d3f786e40 10 cache get: name=default.rgw.meta+users.uid+foo : type miss (requested=0x13, cached=0x16) 2023-10-05T19:28:28.015 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.006+0000 7f5d3f786e40 20 rados->read ofs=0 len=0 2023-10-05T19:28:28.015 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.006+0000 7f5d3f786e40 1 -- 172.21.15.167:0/2568822831 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:103 8.18 8:1e6ca5ed:users.uid::foo:head [call version.check_conds in=122b,call version.read in=11b,read 0~0,getxattrs] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x56161bfc8b10 con 0x56161bf087f0 2023-10-05T19:28:28.015 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.010+0000 7f5d3dabc700 1 -- 172.21.15.167:0/2568822831 <== osd.1 v2:172.21.15.167:6800/30035 36 ==== osd_op_reply(103 foo [call,call out=48b,read 0~357 out=357b,getxattrs out=75b] v0'0 uv5 ondisk = 0) v8 ==== 273+0+480 (crc 0 0 0) 0x7f5d38052040 con 0x56161bf087f0 2023-10-05T19:28:28.016 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.010+0000 7f5d3f786e40 20 rados_obj.operate() r=0 bl.length=357 2023-10-05T19:28:28.016 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.010+0000 7f5d3f786e40 10 cache put: name=default.rgw.meta+users.uid+foo info.flags=0x13 2023-10-05T19:28:28.016 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.010+0000 7f5d3f786e40 10 moving default.rgw.meta+users.uid+foo to cache LRU end 2023-10-05T19:28:28.016 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.010+0000 7f5d3f786e40 10 updating xattr: name=ceph.objclass.version bl.length()=42 2023-10-05T19:28:28.016 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.010+0000 7f5d3f786e40 20 get_system_obj_state: rctx=0x7ffefdc2f028 obj=default.rgw.meta:users.swift:foo:foo1 state=0x56161bde10b0 s->prefetch_data=0 2023-10-05T19:28:28.016 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.010+0000 7f5d3f786e40 10 cache get: name=default.rgw.meta+users.swift+foo:foo1 : miss 2023-10-05T19:28:28.016 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.010+0000 7f5d3f786e40 1 -- 172.21.15.167:0/2568822831 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:104 8.8 8:17d68d04:users.swift::foo%3afoo1:head [getxattrs,stat] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x56161bfc6af0 con 0x56161bf087f0 2023-10-05T19:28:28.017 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.010+0000 7f5d3dabc700 1 -- 172.21.15.167:0/2568822831 <== osd.1 v2:172.21.15.167:6800/30035 37 ==== osd_op_reply(104 foo:foo1 [getxattrs,stat] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 ==== 194+0+0 (crc 0 0 0) 0x7f5d38052040 con 0x56161bf087f0 2023-10-05T19:28:28.017 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.010+0000 7f5d3f786e40 10 cache put: name=default.rgw.meta+users.swift+foo:foo1 info.flags=0x0 2023-10-05T19:28:28.017 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.010+0000 7f5d3f786e40 10 adding default.rgw.meta+users.swift+foo:foo1 to cache LRU end 2023-10-05T19:28:28.017 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.010+0000 7f5d3f786e40 20 get_system_obj_state: rctx=0x7ffefdc2f6c8 obj=default.rgw.meta:users.swift:foo:foo1 state=0x56161bde10b0 s->prefetch_data=0 2023-10-05T19:28:28.017 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.010+0000 7f5d3f786e40 10 cache get: name=default.rgw.meta+users.swift+foo:foo1 : hit (negative entry) 2023-10-05T19:28:28.017 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.010+0000 7f5d3f786e40 1 -- 172.21.15.167:0/2568822831 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:105 8.18 8:1e6ca5ed:users.uid::foo:head [delete,create,call version.set in=58b,writefull 0~513 in=513b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x56161bfc9e50 con 0x56161bf087f0 2023-10-05T19:28:28.018 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.010+0000 7f5d3dabc700 1 -- 172.21.15.167:0/2568822831 <== osd.1 v2:172.21.15.167:6800/30035 38 ==== osd_op_reply(105 foo [delete,create,call,writefull 0~513] v30'6 uv6 ondisk = 0) v8 ==== 273+0+0 (crc 0 0 0) 0x7f5d38052040 con 0x56161bf087f0 2023-10-05T19:28:28.018 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.010+0000 7f5d3f786e40 10 cache put: name=default.rgw.meta+users.uid+foo info.flags=0x17 2023-10-05T19:28:28.018 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.010+0000 7f5d3f786e40 10 moving default.rgw.meta+users.uid+foo to cache LRU end 2023-10-05T19:28:28.018 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.010+0000 7f5d3f786e40 10 distributing notification oid=default.rgw.control:notify.4 cni=[op: 0, obj: default.rgw.meta:users.uid:foo, ofs0, ns] 2023-10-05T19:28:28.019 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.010+0000 7f5d3f786e40 1 -- 172.21.15.167:0/2568822831 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:106 7.d 7:b4812045:::notify.4:head [notify cookie 94652958808224 in=701b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x56161bfc8b50 con 0x56161bf087f0 2023-10-05T19:28:28.019 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.014+0000 7f5d3dabc700 1 -- 172.21.15.167:0/2568822831 <== osd.1 v2:172.21.15.167:6800/30035 39 ==== watch-notify(notify (1) cookie 94652958268688 notify 128849018885 ret 0) v3 ==== 731+0+0 (crc 0 0 0) 0x7f5d3804e380 con 0x56161bf087f0 2023-10-05T19:28:28.019 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.014+0000 7f5d36ffd700 10 rgw watcher librados: RGWWatcher::handle_notify() notify_id 128849018885 cookie 94652958268688 notifier 4335 bl.length()=689 2023-10-05T19:28:28.019 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.014+0000 7f5d3dabc700 1 -- 172.21.15.167:0/2568822831 <== osd.1 v2:172.21.15.167:6800/30035 40 ==== osd_op_reply(106 notify.4 [notify cookie 94652958808224 out=8b] v0'0 uv54 ondisk = 0) v8 ==== 152+0+8 (crc 0 0 0) 0x7f5d38052040 con 0x56161bf087f0 2023-10-05T19:28:28.019 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.014+0000 7f5d36ffd700 10 rgw watcher librados: cache put: name=default.rgw.meta+users.uid+foo info.flags=0x17 2023-10-05T19:28:28.019 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.014+0000 7f5d36ffd700 10 rgw watcher librados: moving default.rgw.meta+users.uid+foo to cache LRU end 2023-10-05T19:28:28.019 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.014+0000 7f5d36ffd700 1 -- 172.21.15.167:0/2568822831 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:107 7.d 7:b4812045:::notify.4:head [notify-ack in=20b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x7f5d1c003d90 con 0x56161bf087f0 2023-10-05T19:28:28.020 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.014+0000 7f5d3dabc700 1 -- 172.21.15.167:0/2568822831 <== osd.1 v2:172.21.15.167:6800/30035 41 ==== osd_op_reply(107 notify.4 [notify-ack] v0'0 uv54 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f5d38052040 con 0x56161bf087f0 2023-10-05T19:28:28.020 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.014+0000 7f5d3dabc700 1 -- 172.21.15.167:0/2568822831 <== osd.1 v2:172.21.15.167:6800/30035 42 ==== watch-notify(notify_complete (2) cookie 94652958808224 notify 128849018885 ret 0) v3 ==== 42+0+48 (crc 0 0 0) 0x7f5d380369c0 con 0x56161bf087f0 2023-10-05T19:28:28.020 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.014+0000 7f5d3f786e40 1 -- 172.21.15.167:0/2568822831 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:108 8.8 8:17d68d04:users.swift::foo%3afoo1:head [delete,create,writefull 0~7 in=7b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x56161bfc98a0 con 0x56161bf087f0 2023-10-05T19:28:28.021 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.014+0000 7f5d3dabc700 1 -- 172.21.15.167:0/2568822831 <== osd.1 v2:172.21.15.167:6800/30035 43 ==== osd_op_reply(108 foo:foo1 [delete,create,writefull 0~7] v30'1 uv1 ondisk = 0) v8 ==== 236+0+0 (crc 0 0 0) 0x7f5d38052040 con 0x56161bf087f0 2023-10-05T19:28:28.022 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.014+0000 7f5d3f786e40 10 cache put: name=default.rgw.meta+users.swift+foo:foo1 info.flags=0x7 2023-10-05T19:28:28.022 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.014+0000 7f5d3f786e40 10 moving default.rgw.meta+users.swift+foo:foo1 to cache LRU end 2023-10-05T19:28:28.022 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.014+0000 7f5d3f786e40 10 distributing notification oid=default.rgw.control:notify.0 cni=[op: 0, obj: default.rgw.meta:users.swift:foo:foo1, ofs0, ns] 2023-10-05T19:28:28.022 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.014+0000 7f5d3f786e40 1 -- 172.21.15.167:0/2568822831 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:109 7.1f 7:f95f44c2:::notify.0:head [notify cookie 94652958808224 in=178b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x56161bfc87f0 con 0x7f5d3000d3d0 2023-10-05T19:28:28.022 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.018+0000 7f5d3caba700 1 -- 172.21.15.167:0/2568822831 <== osd.2 v2:172.21.15.167:6816/30037 17 ==== watch-notify(notify (1) cookie 94652958260704 notify 128849018882 ret 0) v3 ==== 208+0+0 (crc 0 0 0) 0x56161bf5e270 con 0x7f5d3000d3d0 2023-10-05T19:28:28.023 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.018+0000 7f5d36ffd700 10 rgw watcher librados: RGWWatcher::handle_notify() notify_id 128849018882 cookie 94652958260704 notifier 4335 bl.length()=166 2023-10-05T19:28:28.023 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.018+0000 7f5d36ffd700 10 rgw watcher librados: cache put: name=default.rgw.meta+users.swift+foo:foo1 info.flags=0x7 2023-10-05T19:28:28.023 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.018+0000 7f5d36ffd700 10 rgw watcher librados: moving default.rgw.meta+users.swift+foo:foo1 to cache LRU end 2023-10-05T19:28:28.023 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.018+0000 7f5d36ffd700 1 -- 172.21.15.167:0/2568822831 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:110 7.1f 7:f95f44c2:::notify.0:head [notify-ack in=20b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x7f5d1c003d90 con 0x7f5d3000d3d0 2023-10-05T19:28:28.023 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.018+0000 7f5d3caba700 1 -- 172.21.15.167:0/2568822831 <== osd.2 v2:172.21.15.167:6816/30037 18 ==== osd_op_reply(109 notify.0 [notify cookie 94652958808224 out=8b] v0'0 uv54 ondisk = 0) v8 ==== 152+0+8 (crc 0 0 0) 0x7f5d28032040 con 0x7f5d3000d3d0 2023-10-05T19:28:28.024 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.018+0000 7f5d3caba700 1 -- 172.21.15.167:0/2568822831 <== osd.2 v2:172.21.15.167:6816/30037 19 ==== osd_op_reply(110 notify.0 [notify-ack] v0'0 uv54 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f5d28032040 con 0x7f5d3000d3d0 2023-10-05T19:28:28.024 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.018+0000 7f5d3caba700 1 -- 172.21.15.167:0/2568822831 <== osd.2 v2:172.21.15.167:6816/30037 20 ==== watch-notify(notify_complete (2) cookie 94652958808224 notify 128849018882 ret 0) v3 ==== 42+0+48 (crc 0 0 0) 0x56161bf5de90 con 0x7f5d3000d3d0 2023-10-05T19:28:28.024 INFO:teuthology.orchestra.run.smithi167.stdout:{ 2023-10-05T19:28:28.024 INFO:teuthology.orchestra.run.smithi167.stdout: "user_id": "foo", 2023-10-05T19:28:28.024 INFO:teuthology.orchestra.run.smithi167.stdout: "display_name": "Foo", 2023-10-05T19:28:28.024 INFO:teuthology.orchestra.run.smithi167.stdout: "email": "foo@foo.com", 2023-10-05T19:28:28.024 INFO:teuthology.orchestra.run.smithi167.stdout: "suspended": 0, 2023-10-05T19:28:28.025 INFO:teuthology.orchestra.run.smithi167.stdout: "max_buckets": 4, 2023-10-05T19:28:28.025 INFO:teuthology.orchestra.run.smithi167.stdout: "subusers": [ 2023-10-05T19:28:28.025 INFO:teuthology.orchestra.run.smithi167.stdout: { 2023-10-05T19:28:28.025 INFO:teuthology.orchestra.run.smithi167.stdout: "id": "foo:foo1", 2023-10-05T19:28:28.025 INFO:teuthology.orchestra.run.smithi167.stdout: "permissions": "full-control" 2023-10-05T19:28:28.025 INFO:teuthology.orchestra.run.smithi167.stdout: } 2023-10-05T19:28:28.025 INFO:teuthology.orchestra.run.smithi167.stdout: ], 2023-10-05T19:28:28.025 INFO:teuthology.orchestra.run.smithi167.stdout: "keys": [ 2023-10-05T19:28:28.025 INFO:teuthology.orchestra.run.smithi167.stdout: { 2023-10-05T19:28:28.025 INFO:teuthology.orchestra.run.smithi167.stdout: "user": "foo", 2023-10-05T19:28:28.025 INFO:teuthology.orchestra.run.smithi167.stdout: "access_key": "9te6NH5mcdcq0Tc5i8i1", 2023-10-05T19:28:28.025 INFO:teuthology.orchestra.run.smithi167.stdout: "secret_key": "Ny4IOauQoL18Gp2zM7lC1vLmoawgqcYP/YGcWfXu" 2023-10-05T19:28:28.026 INFO:teuthology.orchestra.run.smithi167.stdout: } 2023-10-05T19:28:28.026 INFO:teuthology.orchestra.run.smithi167.stdout: ], 2023-10-05T19:28:28.026 INFO:teuthology.orchestra.run.smithi167.stdout: "swift_keys": [ 2023-10-05T19:28:28.026 INFO:teuthology.orchestra.run.smithi167.stdout: { 2023-10-05T19:28:28.026 INFO:teuthology.orchestra.run.smithi167.stdout: "user": "foo:foo1", 2023-10-05T19:28:28.026 INFO:teuthology.orchestra.run.smithi167.stdout: "secret_key": "GKaEU2w0XSW9ownT1gGkBHsVLBtq1pIammfMP8WA" 2023-10-05T19:28:28.026 INFO:teuthology.orchestra.run.smithi167.stdout: } 2023-10-05T19:28:28.026 INFO:teuthology.orchestra.run.smithi167.stdout: ], 2023-10-05T19:28:28.026 INFO:teuthology.orchestra.run.smithi167.stdout: "caps": [], 2023-10-05T19:28:28.026 INFO:teuthology.orchestra.run.smithi167.stdout: "op_mask": "read, write, delete", 2023-10-05T19:28:28.027 INFO:teuthology.orchestra.run.smithi167.stdout: "default_placement": "", 2023-10-05T19:28:28.027 INFO:teuthology.orchestra.run.smithi167.stdout: "default_storage_class": "", 2023-10-05T19:28:28.027 INFO:teuthology.orchestra.run.smithi167.stdout: "placement_tags": [], 2023-10-05T19:28:28.027 INFO:teuthology.orchestra.run.smithi167.stdout: "bucket_quota": { 2023-10-05T19:28:28.027 INFO:teuthology.orchestra.run.smithi167.stdout: "enabled": false, 2023-10-05T19:28:28.027 INFO:teuthology.orchestra.run.smithi167.stdout: "check_on_raw": false, 2023-10-05T19:28:28.027 INFO:teuthology.orchestra.run.smithi167.stdout: "max_size": -1, 2023-10-05T19:28:28.027 INFO:teuthology.orchestra.run.smithi167.stdout: "max_size_kb": 0, 2023-10-05T19:28:28.027 INFO:teuthology.orchestra.run.smithi167.stdout: "max_objects": -1 2023-10-05T19:28:28.027 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2023-10-05T19:28:28.027 INFO:teuthology.orchestra.run.smithi167.stdout: "user_quota": { 2023-10-05T19:28:28.027 INFO:teuthology.orchestra.run.smithi167.stdout: "enabled": false, 2023-10-05T19:28:28.028 INFO:teuthology.orchestra.run.smithi167.stdout: "check_on_raw": false, 2023-10-05T19:28:28.028 INFO:teuthology.orchestra.run.smithi167.stdout: "max_size": -1, 2023-10-05T19:28:28.028 INFO:teuthology.orchestra.run.smithi167.stdout: "max_size_kb": 0, 2023-10-05T19:28:28.028 INFO:teuthology.orchestra.run.smithi167.stdout: "max_objects": -1 2023-10-05T19:28:28.028 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2023-10-05T19:28:28.028 INFO:teuthology.orchestra.run.smithi167.stdout: "temp_url_keys": [], 2023-10-05T19:28:28.028 INFO:teuthology.orchestra.run.smithi167.stdout: "type": "rgw", 2023-10-05T19:28:28.028 INFO:teuthology.orchestra.run.smithi167.stdout: "mfa_ids": [] 2023-10-05T19:28:28.028 INFO:teuthology.orchestra.run.smithi167.stdout:} 2023-10-05T19:28:28.028 INFO:teuthology.orchestra.run.smithi167.stdout: 2023-10-05T19:28:28.029 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.018+0000 7f5d3f786e40 1 -- 172.21.15.167:0/2568822831 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:111 7.1f 7:f95f44c2:::notify.0:head [watch unwatch cookie 94652958260704] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x56161bf1fa90 con 0x7f5d3000d3d0 2023-10-05T19:28:28.029 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.022+0000 7f5d3caba700 1 -- 172.21.15.167:0/2568822831 <== osd.2 v2:172.21.15.167:6816/30037 21 ==== osd_op_reply(111 notify.0 [watch unwatch cookie 94652958260704] v30'56 uv54 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f5d28032040 con 0x7f5d3000d3d0 2023-10-05T19:28:28.029 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.022+0000 7f5d3f786e40 20 remove_watcher() i=0 2023-10-05T19:28:28.029 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.022+0000 7f5d3f786e40 2 removed watcher, disabling cache 2023-10-05T19:28:28.029 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.022+0000 7f5d3f786e40 1 -- 172.21.15.167:0/2568822831 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:112 7.0 7:05bf5b68:::notify.1:head [watch unwatch cookie 94652958264960] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x56161bf1fa90 con 0x56161bf087f0 2023-10-05T19:28:28.030 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.022+0000 7f5d3dabc700 1 -- 172.21.15.167:0/2568822831 <== osd.1 v2:172.21.15.167:6800/30035 44 ==== osd_op_reply(112 notify.1 [watch unwatch cookie 94652958264960] v30'56 uv54 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f5d38052040 con 0x56161bf087f0 2023-10-05T19:28:28.030 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.022+0000 7f5d3f786e40 20 remove_watcher() i=1 2023-10-05T19:28:28.030 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.022+0000 7f5d3f786e40 1 -- 172.21.15.167:0/2568822831 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:113 7.15 7:a93a5511:::notify.2:head [watch unwatch cookie 94652958266416] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x56161bf1fa90 con 0x56161bf3f710 2023-10-05T19:28:28.031 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.026+0000 7f5d3d2bb700 1 -- 172.21.15.167:0/2568822831 <== osd.3 v2:172.21.15.167:6808/30036 24 ==== osd_op_reply(113 notify.2 [watch unwatch cookie 94652958266416] v30'56 uv54 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f5d30050040 con 0x56161bf3f710 2023-10-05T19:28:28.032 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.026+0000 7f5d3f786e40 20 remove_watcher() i=2 2023-10-05T19:28:28.032 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.026+0000 7f5d3f786e40 1 -- 172.21.15.167:0/2568822831 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:114 7.e 7:7759931f:::notify.3:head [watch unwatch cookie 94652958274832] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x56161bf43de0 con 0x7f5d3000d3d0 2023-10-05T19:28:28.034 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.030+0000 7f5d3caba700 1 -- 172.21.15.167:0/2568822831 <== osd.2 v2:172.21.15.167:6816/30037 22 ==== osd_op_reply(114 notify.3 [watch unwatch cookie 94652958274832] v30'56 uv54 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f5d28032040 con 0x7f5d3000d3d0 2023-10-05T19:28:28.034 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.030+0000 7f5d3f786e40 20 remove_watcher() i=3 2023-10-05T19:28:28.034 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.030+0000 7f5d3f786e40 1 -- 172.21.15.167:0/2568822831 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:115 7.d 7:b4812045:::notify.4:head [watch unwatch cookie 94652958268688] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x56161bf45430 con 0x56161bf087f0 2023-10-05T19:28:28.036 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.030+0000 7f5d3dabc700 1 -- 172.21.15.167:0/2568822831 <== osd.1 v2:172.21.15.167:6800/30035 45 ==== osd_op_reply(115 notify.4 [watch unwatch cookie 94652958268688] v30'56 uv54 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f5d38052040 con 0x56161bf087f0 2023-10-05T19:28:28.036 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.030+0000 7f5d3f786e40 20 remove_watcher() i=4 2023-10-05T19:28:28.037 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.030+0000 7f5d3f786e40 1 -- 172.21.15.167:0/2568822831 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:116 7.3 7:c609908c:::notify.5:head [watch unwatch cookie 94652958279072] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x56161bf47510 con 0x7f5d28007030 2023-10-05T19:28:28.039 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.034+0000 7f5d3dabc700 1 -- 172.21.15.167:0/2568822831 <== osd.0 v2:172.21.15.118:6800/27690 29 ==== osd_op_reply(116 notify.5 [watch unwatch cookie 94652958279072] v30'56 uv54 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f5d38052040 con 0x7f5d28007030 2023-10-05T19:28:28.039 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.034+0000 7f5d3f786e40 20 remove_watcher() i=5 2023-10-05T19:28:28.039 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.034+0000 7f5d3f786e40 1 -- 172.21.15.167:0/2568822831 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:117 7.14 7:2b04a3e9:::notify.6:head [watch unwatch cookie 94652958282800] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x56161bf43de0 con 0x7f5d28007030 2023-10-05T19:28:28.042 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.038+0000 7f5d3dabc700 1 -- 172.21.15.167:0/2568822831 <== osd.0 v2:172.21.15.118:6800/27690 30 ==== osd_op_reply(117 notify.6 [watch unwatch cookie 94652958282800] v30'56 uv54 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f5d38052040 con 0x7f5d28007030 2023-10-05T19:28:28.042 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.038+0000 7f5d3f786e40 20 remove_watcher() i=6 2023-10-05T19:28:28.042 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.038+0000 7f5d3f786e40 1 -- 172.21.15.167:0/2568822831 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:118 7.9 7:93e5b521:::notify.7:head [watch unwatch cookie 94652958285296] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x56161bf43de0 con 0x7f5d28007030 2023-10-05T19:28:28.045 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.038+0000 7f5d3dabc700 1 -- 172.21.15.167:0/2568822831 <== osd.0 v2:172.21.15.118:6800/27690 31 ==== osd_op_reply(118 notify.7 [watch unwatch cookie 94652958285296] v30'56 uv54 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f5d38052040 con 0x7f5d28007030 2023-10-05T19:28:28.045 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.038+0000 7f5d3f786e40 20 remove_watcher() i=7 2023-10-05T19:28:28.046 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.042+0000 7f5d0e7e4700 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2023-10-05T19:28:28.047 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.042+0000 7f5d3f786e40 1 -- 172.21.15.167:0/2568822831 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:119 6.4 6:22d26bf9:::data_loggenerations_metadata:head [watch unwatch cookie 94652958286400] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x56161bf43de0 con 0x56161bf087f0 2023-10-05T19:28:28.050 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.042+0000 7f5d3dabc700 1 -- 172.21.15.167:0/2568822831 <== osd.1 v2:172.21.15.167:6800/30035 46 ==== osd_op_reply(119 data_loggenerations_metadata [watch unwatch cookie 94652958286400] v30'71 uv1 ondisk = 0) v8 ==== 172+0+0 (crc 0 0 0) 0x7f5d38052040 con 0x56161bf087f0 2023-10-05T19:28:28.050 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.042+0000 7f5d3f786e40 1 -- 172.21.15.167:0/2568822831 >> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] conn(0x7f5d28007030 msgr2=0x7f5d2801c920 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-05T19:28:28.050 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.042+0000 7f5d3f786e40 1 --2- 172.21.15.167:0/2568822831 >> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] conn(0x7f5d28007030 0x7f5d2801c920 crc :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-05T19:28:28.050 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.042+0000 7f5d3f786e40 1 -- 172.21.15.167:0/2568822831 >> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] conn(0x56161bf087f0 msgr2=0x56161bf3b930 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-05T19:28:28.050 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.042+0000 7f5d3f786e40 1 --2- 172.21.15.167:0/2568822831 >> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] conn(0x56161bf087f0 0x56161bf3b930 crc :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-05T19:28:28.051 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.042+0000 7f5d3f786e40 1 -- 172.21.15.167:0/2568822831 >> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] conn(0x7f5d3000d3d0 msgr2=0x7f5d300420d0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-05T19:28:28.051 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.042+0000 7f5d3f786e40 1 --2- 172.21.15.167:0/2568822831 >> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] conn(0x7f5d3000d3d0 0x7f5d300420d0 crc :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-05T19:28:28.051 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.042+0000 7f5d3f786e40 1 -- 172.21.15.167:0/2568822831 >> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] conn(0x56161bf3f710 msgr2=0x56161bf41af0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-05T19:28:28.051 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.042+0000 7f5d3f786e40 1 --2- 172.21.15.167:0/2568822831 >> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] conn(0x56161bf3f710 0x56161bf41af0 crc :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-05T19:28:28.051 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.042+0000 7f5d3f786e40 1 -- 172.21.15.167:0/2568822831 >> [v2:172.21.15.167:6824/29889,v1:172.21.15.167:6825/29889] conn(0x7f5d20030c00 msgr2=0x7f5d200330b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-05T19:28:28.051 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.042+0000 7f5d3f786e40 1 --2- 172.21.15.167:0/2568822831 >> [v2:172.21.15.167:6824/29889,v1:172.21.15.167:6825/29889] conn(0x7f5d20030c00 0x7f5d200330b0 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f5d28005a90 tx=0x7f5d28015000 comp rx=0 tx=0).stop 2023-10-05T19:28:28.051 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.042+0000 7f5d3f786e40 1 -- 172.21.15.167:0/2568822831 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x56161bf08420 msgr2=0x56161bf2fe70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-05T19:28:28.051 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.042+0000 7f5d3dabc700 1 -- 172.21.15.167:0/2568822831 reap_dead start 2023-10-05T19:28:28.051 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.042+0000 7f5d3f786e40 1 --2- 172.21.15.167:0/2568822831 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x56161bf08420 0x56161bf2fe70 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7f5d30001820 tx=0x7f5d30016000 comp rx=0 tx=0).stop 2023-10-05T19:28:28.051 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.046+0000 7f5d3f786e40 1 -- 172.21.15.167:0/2568822831 shutdown_connections 2023-10-05T19:28:28.051 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.046+0000 7f5d3f786e40 1 --2- 172.21.15.167:0/2568822831 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x56161bf08420 0x56161bf2fe70 unknown :-1 s=CLOSED pgs=245 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-05T19:28:28.052 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.046+0000 7f5d3f786e40 1 -- 172.21.15.167:0/2568822831 >> 172.21.15.167:0/2568822831 conn(0x56161bf02730 msgr2=0x56161bf23740 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-05T19:28:28.052 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.046+0000 7f5d3f786e40 1 -- 172.21.15.167:0/2568822831 shutdown_connections 2023-10-05T19:28:28.052 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.046+0000 7f5d3f786e40 1 -- 172.21.15.167:0/2568822831 wait complete. 2023-10-05T19:28:28.053 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.046+0000 7f5d3f786e40 1 -- 172.21.15.167:0/2929312097 >> [v2:172.21.15.167:6824/29889,v1:172.21.15.167:6825/29889] conn(0x7f5c84030c50 msgr2=0x7f5c84033100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-05T19:28:28.053 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.046+0000 7f5d3f786e40 1 --2- 172.21.15.167:0/2929312097 >> [v2:172.21.15.167:6824/29889,v1:172.21.15.167:6825/29889] conn(0x7f5c84030c50 0x7f5c84033100 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x56161bf48c90 tx=0x7f5d3800a280 comp rx=0 tx=0).stop 2023-10-05T19:28:28.053 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.046+0000 7f5d3f786e40 1 -- 172.21.15.167:0/2929312097 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x56161bf57d60 msgr2=0x56161bf5d890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-05T19:28:28.053 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.046+0000 7f5d3f786e40 1 --2- 172.21.15.167:0/2929312097 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x56161bf57d60 0x56161bf5d890 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x56161bf51de0 tx=0x7f5d2802f000 comp rx=0 tx=0).stop 2023-10-05T19:28:28.053 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.046+0000 7f5d3f786e40 1 -- 172.21.15.167:0/2929312097 shutdown_connections 2023-10-05T19:28:28.053 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.046+0000 7f5d3f786e40 1 --2- 172.21.15.167:0/2929312097 >> [v2:172.21.15.167:6824/29889,v1:172.21.15.167:6825/29889] conn(0x7f5c84030c50 0x7f5c84033100 unknown :-1 s=CLOSED pgs=89 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-05T19:28:28.053 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.046+0000 7f5d3f786e40 1 --2- 172.21.15.167:0/2929312097 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x56161bf57d60 0x56161bf5d890 unknown :-1 s=CLOSED pgs=247 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-05T19:28:28.053 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.046+0000 7f5d3f786e40 1 -- 172.21.15.167:0/2929312097 >> 172.21.15.167:0/2929312097 conn(0x56161bf53120 msgr2=0x56161bf511c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-05T19:28:28.053 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.046+0000 7f5d3f786e40 1 -- 172.21.15.167:0/2929312097 shutdown_connections 2023-10-05T19:28:28.054 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.046+0000 7f5d3f786e40 1 -- 172.21.15.167:0/2929312097 wait complete. 2023-10-05T19:28:28.056 DEBUG:tasks.util.rgw: json result: {'user_id': 'foo', 'display_name': 'Foo', 'email': 'foo@foo.com', 'suspended': 0, 'max_buckets': 4, 'subusers': [{'id': 'foo:foo1', 'permissions': 'full-control'}], 'keys': [{'user': 'foo', 'access_key': '9te6NH5mcdcq0Tc5i8i1', 'secret_key': 'Ny4IOauQoL18Gp2zM7lC1vLmoawgqcYP/YGcWfXu'}], 'swift_keys': [{'user': 'foo:foo1', 'secret_key': 'GKaEU2w0XSW9ownT1gGkBHsVLBtq1pIammfMP8WA'}], 'caps': [], 'op_mask': 'read, write, delete', 'default_placement': '', 'default_storage_class': '', 'placement_tags': [], 'bucket_quota': {'enabled': False, 'check_on_raw': False, 'max_size': -1, 'max_size_kb': 0, 'max_objects': -1}, 'user_quota': {'enabled': False, 'check_on_raw': False, 'max_size': -1, 'max_size_kb': 0, 'max_objects': -1}, 'temp_url_keys': [], 'type': 'rgw', 'mfa_ids': []} 2023-10-05T19:28:28.057 INFO:tasks.util.rgw:rgwadmin: client.0 : ['subuser', 'modify', '--subuser', 'foo:foo1', '--secret', 'gpS2G9RREMrnbqlp29PP2D36kgPR1tm72n5fPYfL', '--key-type', 'swift'] 2023-10-05T19:28:28.057 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'ceph', 'subuser', 'modify', '--subuser', 'foo:foo1', '--secret', 'gpS2G9RREMrnbqlp29PP2D36kgPR1tm72n5fPYfL', '--key-type', 'swift'] 2023-10-05T19:28:28.057 DEBUG:teuthology.orchestra.run.smithi167:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster ceph subuser modify --subuser foo:foo1 --secret gpS2G9RREMrnbqlp29PP2D36kgPR1tm72n5fPYfL --key-type swift 2023-10-05T19:28:28.100 INFO:teuthology.orchestra.run.smithi167.stderr:ignoring --setuser ceph since I am not root 2023-10-05T19:28:28.100 INFO:teuthology.orchestra.run.smithi167.stderr:ignoring --setgroup ceph since I am not root 2023-10-05T19:28:28.101 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.094+0000 7f18f2e04e40 1 Processor -- start 2023-10-05T19:28:28.101 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.094+0000 7f18f2e04e40 1 -- start start 2023-10-05T19:28:28.101 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.094+0000 7f18f2e04e40 1 --2- >> v2:172.21.15.118:3300/0 conn(0x556bff5addd0 0x556bff5ae1a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-05T19:28:28.101 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.094+0000 7f18f2e04e40 1 -- --> v1:172.21.15.118:6789/0 -- auth(proto 0 26 bytes epoch 0) v1 -- 0x556bff4fabd0 con 0x556bff5ae6e0 2023-10-05T19:28:28.101 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.094+0000 7f18f2e04e40 1 -- --> v2:172.21.15.118:3300/0 -- mon_getmap magic: 0 v1 -- 0x556bff4b7ee0 con 0x556bff5addd0 2023-10-05T19:28:28.102 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.094+0000 7f18f0939700 1 --2- >> v2:172.21.15.118:3300/0 conn(0x556bff5addd0 0x556bff5ae1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-05T19:28:28.102 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.094+0000 7f18f0138700 1 --1- >> v1:172.21.15.118:6789/0 conn(0x556bff5ae6e0 0x556bff5af180 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.118:6789/0 says I am v1:172.21.15.167:56454/0 (socket says 172.21.15.167:56454) 2023-10-05T19:28:28.102 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.094+0000 7f18f0939700 1 --2- >> v2:172.21.15.118:3300/0 conn(0x556bff5addd0 0x556bff5ae1a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.118:3300/0 says I am v2:172.21.15.167:57640/0 (socket says 172.21.15.167:57640) 2023-10-05T19:28:28.102 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.094+0000 7f18f0138700 1 -- 172.21.15.167:0/3607957701 learned_addr learned my addr 172.21.15.167:0/3607957701 (peer_addr_for_me v1:172.21.15.167:0/0) 2023-10-05T19:28:28.103 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.098+0000 7f18f0939700 1 -- 172.21.15.167:0/3607957701 >> v1:172.21.15.118:6789/0 conn(0x556bff5ae6e0 legacy=0x556bff5af180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-05T19:28:28.103 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.098+0000 7f18ef937700 1 -- 172.21.15.167:0/3607957701 <== mon.0 v1:172.21.15.118:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 349728979 0 0) 0x556bff4fabd0 con 0x556bff5ae6e0 2023-10-05T19:28:28.103 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.098+0000 7f18f0939700 1 -- 172.21.15.167:0/3607957701 --> v2:172.21.15.118:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x556bff5af890 con 0x556bff5addd0 2023-10-05T19:28:28.104 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.098+0000 7f18f0939700 1 --2- 172.21.15.167:0/3607957701 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x556bff5addd0 0x556bff5ae1a0 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x7f18e000a520 tx=0x7f18e0005d40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f23d4648e0b176b7 server_cookie=0 in_seq=0 out_seq=0 2023-10-05T19:28:28.104 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.098+0000 7f18ef937700 1 -- 172.21.15.167:0/3607957701 <== mon.0 v2:172.21.15.118:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f18e000f590 con 0x556bff5addd0 2023-10-05T19:28:28.104 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.098+0000 7f18ef937700 1 -- 172.21.15.167:0/3607957701 <== mon.0 v2:172.21.15.118:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f18e000f6f0 con 0x556bff5addd0 2023-10-05T19:28:28.104 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.098+0000 7f18ef937700 1 -- 172.21.15.167:0/3607957701 <== mon.0 v2:172.21.15.118:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f18e000f880 con 0x556bff5addd0 2023-10-05T19:28:28.104 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.098+0000 7f18f2e04e40 1 -- 172.21.15.167:0/3607957701 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x556bff5addd0 msgr2=0x556bff5ae1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-05T19:28:28.104 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.098+0000 7f18f2e04e40 1 --2- 172.21.15.167:0/3607957701 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x556bff5addd0 0x556bff5ae1a0 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x7f18e000a520 tx=0x7f18e0005d40 comp rx=0 tx=0).stop 2023-10-05T19:28:28.105 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.098+0000 7f18f2e04e40 1 -- 172.21.15.167:0/3607957701 shutdown_connections 2023-10-05T19:28:28.105 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.098+0000 7f18f2e04e40 1 --2- 172.21.15.167:0/3607957701 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x556bff5addd0 0x556bff5ae1a0 unknown :-1 s=CLOSED pgs=249 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-05T19:28:28.105 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.098+0000 7f18f2e04e40 1 -- 172.21.15.167:0/3607957701 >> 172.21.15.167:0/3607957701 conn(0x556bff5a97f0 msgr2=0x556bff5abc00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-05T19:28:28.105 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.098+0000 7f18f2e04e40 1 -- 172.21.15.167:0/3607957701 shutdown_connections 2023-10-05T19:28:28.105 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.098+0000 7f18f2e04e40 1 -- 172.21.15.167:0/3607957701 wait complete. 2023-10-05T19:28:28.106 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.098+0000 7f18f2e04e40 1 Processor -- start 2023-10-05T19:28:28.106 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.098+0000 7f18f2e04e40 1 -- start start 2023-10-05T19:28:28.106 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.098+0000 7f18f2e04e40 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x556bff5addd0 0x556bff5ccd30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-05T19:28:28.106 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.098+0000 7f18f2e04e40 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x556bff47be30 con 0x556bff5addd0 2023-10-05T19:28:28.106 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.098+0000 7f18f0939700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x556bff5addd0 0x556bff5ccd30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-05T19:28:28.106 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.098+0000 7f18f0939700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x556bff5addd0 0x556bff5ccd30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.118:3300/0 says I am v2:172.21.15.167:57650/0 (socket says 172.21.15.167:57650) 2023-10-05T19:28:28.107 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.098+0000 7f18f0939700 1 -- 172.21.15.167:0/2282544254 learned_addr learned my addr 172.21.15.167:0/2282544254 (peer_addr_for_me v2:172.21.15.167:0/0) 2023-10-05T19:28:28.107 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.102+0000 7f18f0939700 1 -- 172.21.15.167:0/2282544254 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f18e000a0e0 con 0x556bff5addd0 2023-10-05T19:28:28.107 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.102+0000 7f18f0939700 1 --2- 172.21.15.167:0/2282544254 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x556bff5addd0 0x556bff5ccd30 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x7f18e0005b20 tx=0x7f18e0014000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a7efa424ac46d980 server_cookie=0 in_seq=0 out_seq=0 2023-10-05T19:28:28.107 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.102+0000 7f18ee935700 1 -- 172.21.15.167:0/2282544254 <== mon.0 v2:172.21.15.118:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f18e000f860 con 0x556bff5addd0 2023-10-05T19:28:28.107 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.102+0000 7f18ee935700 1 -- 172.21.15.167:0/2282544254 <== mon.0 v2:172.21.15.118:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f18e000f9c0 con 0x556bff5addd0 2023-10-05T19:28:28.107 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.102+0000 7f18ee935700 1 -- 172.21.15.167:0/2282544254 <== mon.0 v2:172.21.15.118:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f18e000fb70 con 0x556bff5addd0 2023-10-05T19:28:28.108 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.102+0000 7f18f2e04e40 1 -- 172.21.15.167:0/2282544254 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x556bff5addd0 msgr2=0x556bff5ccd30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-05T19:28:28.108 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.102+0000 7f18f2e04e40 1 --2- 172.21.15.167:0/2282544254 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x556bff5addd0 0x556bff5ccd30 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x7f18e0005b20 tx=0x7f18e0014000 comp rx=0 tx=0).stop 2023-10-05T19:28:28.108 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.102+0000 7f18f2e04e40 1 -- 172.21.15.167:0/2282544254 shutdown_connections 2023-10-05T19:28:28.109 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.102+0000 7f18f2e04e40 1 --2- 172.21.15.167:0/2282544254 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x556bff5addd0 0x556bff5ccd30 unknown :-1 s=CLOSED pgs=250 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-05T19:28:28.109 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.102+0000 7f18f2e04e40 1 -- 172.21.15.167:0/2282544254 >> 172.21.15.167:0/2282544254 conn(0x556bff5a97f0 msgr2=0x556bff5c8870 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-05T19:28:28.109 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.102+0000 7f18f2e04e40 1 -- 172.21.15.167:0/2282544254 shutdown_connections 2023-10-05T19:28:28.109 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.102+0000 7f18f2e04e40 1 -- 172.21.15.167:0/2282544254 wait complete. 2023-10-05T19:28:28.110 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.102+0000 7f18f2e04e40 1 Processor -- start 2023-10-05T19:28:28.110 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.102+0000 7f18f2e04e40 1 -- start start 2023-10-05T19:28:28.110 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.102+0000 7f18f2e04e40 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x556bff5addd0 0x556bff5d39b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-05T19:28:28.110 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.102+0000 7f18f2e04e40 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x556bff44eed0 con 0x556bff5addd0 2023-10-05T19:28:28.110 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.106+0000 7f18f0939700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x556bff5addd0 0x556bff5d39b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-05T19:28:28.110 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.106+0000 7f18f0939700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x556bff5addd0 0x556bff5d39b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.118:3300/0 says I am v2:172.21.15.167:57652/0 (socket says 172.21.15.167:57652) 2023-10-05T19:28:28.110 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.106+0000 7f18f0939700 1 -- 172.21.15.167:0/1910438506 learned_addr learned my addr 172.21.15.167:0/1910438506 (peer_addr_for_me v2:172.21.15.167:0/0) 2023-10-05T19:28:28.111 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.106+0000 7f18f0939700 1 -- 172.21.15.167:0/1910438506 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x556bff5af890 con 0x556bff5addd0 2023-10-05T19:28:28.112 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.106+0000 7f18f0939700 1 --2- 172.21.15.167:0/1910438506 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x556bff5addd0 0x556bff5d39b0 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7f18e0000f80 tx=0x7f18e001d000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-05T19:28:28.112 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.106+0000 7f18ed132700 1 -- 172.21.15.167:0/1910438506 <== mon.0 v2:172.21.15.118:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f18e001ce00 con 0x556bff5addd0 2023-10-05T19:28:28.112 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.106+0000 7f18ed132700 1 -- 172.21.15.167:0/1910438506 <== mon.0 v2:172.21.15.118:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f18e0010630 con 0x556bff5addd0 2023-10-05T19:28:28.112 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.106+0000 7f18f2e04e40 1 -- 172.21.15.167:0/1910438506 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x556bff5d3ef0 con 0x556bff5addd0 2023-10-05T19:28:28.112 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.106+0000 7f18ed132700 1 -- 172.21.15.167:0/1910438506 <== mon.0 v2:172.21.15.118:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f18e00107e0 con 0x556bff5addd0 2023-10-05T19:28:28.112 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.106+0000 7f18f2e04e40 1 -- 172.21.15.167:0/1910438506 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x556bff5d4390 con 0x556bff5addd0 2023-10-05T19:28:28.113 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.106+0000 7f18ed132700 1 -- 172.21.15.167:0/1910438506 <== mon.0 v2:172.21.15.118:3300/0 4 ==== mgrmap(e 5) v1 ==== 40329+0+0 (secure 0 0 0) 0x7f18e0010a10 con 0x556bff5addd0 2023-10-05T19:28:28.113 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.106+0000 7f18ed132700 1 --2- 172.21.15.167:0/1910438506 >> [v2:172.21.15.167:6824/29889,v1:172.21.15.167:6825/29889] conn(0x7f18dc030c00 0x7f18dc0330b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-05T19:28:28.114 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.106+0000 7f18ed132700 1 -- 172.21.15.167:0/1910438506 <== mon.0 v2:172.21.15.118:3300/0 5 ==== osd_map(30..30 src has 1..30) v4 ==== 6057+0+0 (secure 0 0 0) 0x7f18e002b070 con 0x556bff5addd0 2023-10-05T19:28:28.114 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.106+0000 7f18f2e04e40 20 rados->read ofs=0 len=0 2023-10-05T19:28:28.115 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.106+0000 7f18f0138700 1 --2- 172.21.15.167:0/1910438506 >> [v2:172.21.15.167:6824/29889,v1:172.21.15.167:6825/29889] conn(0x7f18dc030c00 0x7f18dc0330b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-05T19:28:28.115 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.110+0000 7f18f2e04e40 1 --2- 172.21.15.167:0/1910438506 >> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] conn(0x556bff5ae6e0 0x556bff5df430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-05T19:28:28.115 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.110+0000 7f18f2e04e40 1 -- 172.21.15.167:0/1910438506 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:1 5.12 5:49953fa1:::default.realm:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x556bff5dfb60 con 0x556bff5ae6e0 2023-10-05T19:28:28.115 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.110+0000 7f18f113a700 1 --2- 172.21.15.167:0/1910438506 >> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] conn(0x556bff5ae6e0 0x556bff5df430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-05T19:28:28.115 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.110+0000 7f18f113a700 1 --2- 172.21.15.167:0/1910438506 >> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] conn(0x556bff5ae6e0 0x556bff5df430 crc :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-05T19:28:28.115 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.110+0000 7f18f0138700 1 --2- 172.21.15.167:0/1910438506 >> [v2:172.21.15.167:6824/29889,v1:172.21.15.167:6825/29889] conn(0x7f18dc030c00 0x7f18dc0330b0 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f18e40011e0 tx=0x7f18e40013a0 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-05T19:28:28.117 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.110+0000 7f18f113a700 1 -- 172.21.15.167:0/1910438506 <== osd.1 v2:172.21.15.167:6800/30035 1 ==== osd_op_reply(1 default.realm [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 ==== 157+0+0 (crc 0 0 0) 0x7f18e800a500 con 0x556bff5ae6e0 2023-10-05T19:28:28.117 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.110+0000 7f18f2e04e40 20 rados_obj.operate() r=-2 bl.length=0 2023-10-05T19:28:28.117 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.110+0000 7f18f2e04e40 20 realm 2023-10-05T19:28:28.117 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.110+0000 7f18f2e04e40 20 rados->read ofs=0 len=0 2023-10-05T19:28:28.117 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.110+0000 7f18f2e04e40 1 -- 172.21.15.167:0/1910438506 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:2 5.12 5:49953fa1:::default.realm:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x556bff5dc9d0 con 0x556bff5ae6e0 2023-10-05T19:28:28.118 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.110+0000 7f18f113a700 1 -- 172.21.15.167:0/1910438506 <== osd.1 v2:172.21.15.167:6800/30035 2 ==== osd_op_reply(2 default.realm [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 ==== 157+0+0 (crc 0 0 0) 0x7f18e800a500 con 0x556bff5ae6e0 2023-10-05T19:28:28.118 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.110+0000 7f18f2e04e40 20 rados_obj.operate() r=-2 bl.length=0 2023-10-05T19:28:28.118 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.110+0000 7f18f2e04e40 4 RGWPeriod::init failed to init realm id : (2) No such file or directory 2023-10-05T19:28:28.118 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.110+0000 7f18f2e04e40 20 rados->read ofs=0 len=0 2023-10-05T19:28:28.118 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.110+0000 7f18f2e04e40 1 -- 172.21.15.167:0/1910438506 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:3 5.12 5:49953fa1:::default.realm:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x556bff5e0640 con 0x556bff5ae6e0 2023-10-05T19:28:28.118 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.110+0000 7f18f113a700 1 -- 172.21.15.167:0/1910438506 <== osd.1 v2:172.21.15.167:6800/30035 3 ==== osd_op_reply(3 default.realm [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 ==== 157+0+0 (crc 0 0 0) 0x7f18e800a500 con 0x556bff5ae6e0 2023-10-05T19:28:28.118 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.110+0000 7f18f2e04e40 20 rados_obj.operate() r=-2 bl.length=0 2023-10-05T19:28:28.118 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.110+0000 7f18f2e04e40 20 rados->read ofs=0 len=0 2023-10-05T19:28:28.118 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.110+0000 7f18f2e04e40 1 -- 172.21.15.167:0/1910438506 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:4 5.f 5:f43b6ece:::zone_names.default:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x556bff5e0250 con 0x556bff5ae6e0 2023-10-05T19:28:28.118 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.110+0000 7f18f113a700 1 -- 172.21.15.167:0/1910438506 <== osd.1 v2:172.21.15.167:6800/30035 4 ==== osd_op_reply(4 zone_names.default [read 0~46 out=46b] v0'0 uv1 ondisk = 0) v8 ==== 162+0+46 (crc 0 0 0) 0x7f18e800a500 con 0x556bff5ae6e0 2023-10-05T19:28:28.119 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.110+0000 7f18f2e04e40 20 rados_obj.operate() r=0 bl.length=46 2023-10-05T19:28:28.119 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.110+0000 7f18f2e04e40 20 rados->read ofs=0 len=0 2023-10-05T19:28:28.119 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.110+0000 7f18f2e04e40 1 -- 172.21.15.167:0/1910438506 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:5 5.1d 5:b8374eb4:::zone_info.9a9df965-6612-45d4-a45f-6e3c889bacc2:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x556bff5e14d0 con 0x556bff5ae6e0 2023-10-05T19:28:28.119 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.110+0000 7f18f113a700 1 -- 172.21.15.167:0/1910438506 <== osd.1 v2:172.21.15.167:6800/30035 5 ==== osd_op_reply(5 zone_info.9a9df965-6612-45d4-a45f-6e3c889bacc2 [read 0~874 out=874b] v0'0 uv2 ondisk = 0) v8 ==== 190+0+874 (crc 0 0 0) 0x7f18e800a500 con 0x556bff5ae6e0 2023-10-05T19:28:28.119 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.110+0000 7f18f2e04e40 20 rados_obj.operate() r=0 bl.length=874 2023-10-05T19:28:28.119 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.110+0000 7f18f2e04e40 20 searching for the correct realm 2023-10-05T19:28:28.119 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.110+0000 7f18f2e04e40 1 --2- 172.21.15.167:0/1910438506 >> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] conn(0x556bff5e3290 0x556bff5e5670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-05T19:28:28.119 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.110+0000 7f18f2e04e40 1 -- 172.21.15.167:0/1910438506 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:6 5.0 5:00000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x556bff5e5bb0 con 0x556bff5e3290 2023-10-05T19:28:28.119 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.110+0000 7f18f0939700 1 --2- 172.21.15.167:0/1910438506 >> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] conn(0x556bff5e3290 0x556bff5e5670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-05T19:28:28.120 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.114+0000 7f18f0939700 1 --2- 172.21.15.167:0/1910438506 >> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] conn(0x556bff5e3290 0x556bff5e5670 crc :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-05T19:28:28.120 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.114+0000 7f18f0939700 1 -- 172.21.15.167:0/1910438506 <== osd.3 v2:172.21.15.167:6808/30036 1 ==== osd_op_reply(6 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f18e0038c40 con 0x556bff5e3290 2023-10-05T19:28:28.120 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.114+0000 7f18f0939700 1 --2- 172.21.15.167:0/1910438506 >> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] conn(0x7f18e001b740 0x7f18e0030440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-05T19:28:28.120 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.114+0000 7f18f0939700 1 -- 172.21.15.167:0/1910438506 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:7 5.10 5:08000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x556bff5e5bb0 con 0x7f18e001b740 2023-10-05T19:28:28.121 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.114+0000 7f18f0138700 1 --2- 172.21.15.167:0/1910438506 >> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] conn(0x7f18e001b740 0x7f18e0030440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-05T19:28:28.121 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.114+0000 7f18f0138700 1 --2- 172.21.15.167:0/1910438506 >> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] conn(0x7f18e001b740 0x7f18e0030440 crc :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-05T19:28:28.121 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.114+0000 7f18f0138700 1 -- 172.21.15.167:0/1910438506 <== osd.2 v2:172.21.15.167:6816/30037 1 ==== osd_op_reply(7 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f18e4006070 con 0x7f18e001b740 2023-10-05T19:28:28.121 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.114+0000 7f18f0138700 1 --2- 172.21.15.167:0/1910438506 >> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] conn(0x7f18e400d8e0 0x7f18e400c360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-05T19:28:28.122 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.114+0000 7f18f0138700 1 -- 172.21.15.167:0/1910438506 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:8 5.8 5:10000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x556bff5e5bb0 con 0x7f18e400d8e0 2023-10-05T19:28:28.122 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.114+0000 7f18f113a700 1 --2- 172.21.15.167:0/1910438506 >> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] conn(0x7f18e400d8e0 0x7f18e400c360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-05T19:28:28.122 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.118+0000 7f18f113a700 1 --2- 172.21.15.167:0/1910438506 >> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] conn(0x7f18e400d8e0 0x7f18e400c360 crc :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-05T19:28:28.123 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.118+0000 7f18f113a700 1 -- 172.21.15.167:0/1910438506 <== osd.0 v2:172.21.15.118:6800/27690 1 ==== osd_op_reply(8 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f18e8014040 con 0x7f18e400d8e0 2023-10-05T19:28:28.123 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.118+0000 7f18f113a700 1 -- 172.21.15.167:0/1910438506 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:9 5.18 5:18000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x556bff5e5bb0 con 0x556bff5ae6e0 2023-10-05T19:28:28.124 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.118+0000 7f18f113a700 1 -- 172.21.15.167:0/1910438506 <== osd.1 v2:172.21.15.167:6800/30035 6 ==== osd_op_reply(9 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f18e8014040 con 0x556bff5ae6e0 2023-10-05T19:28:28.124 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.118+0000 7f18f113a700 1 -- 172.21.15.167:0/1910438506 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:10 5.4 5:20000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x556bff5e5bb0 con 0x556bff5ae6e0 2023-10-05T19:28:28.124 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.118+0000 7f18f113a700 1 -- 172.21.15.167:0/1910438506 <== osd.1 v2:172.21.15.167:6800/30035 7 ==== osd_op_reply(10 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f18e8014040 con 0x556bff5ae6e0 2023-10-05T19:28:28.124 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.118+0000 7f18f113a700 1 -- 172.21.15.167:0/1910438506 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:11 5.14 5:28000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x556bff5e5bb0 con 0x7f18e400d8e0 2023-10-05T19:28:28.125 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.118+0000 7f18f113a700 1 -- 172.21.15.167:0/1910438506 <== osd.0 v2:172.21.15.118:6800/27690 2 ==== osd_op_reply(11 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f18e8014040 con 0x7f18e400d8e0 2023-10-05T19:28:28.125 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.118+0000 7f18f113a700 1 -- 172.21.15.167:0/1910438506 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:12 5.c 5:30000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x556bff5e5bb0 con 0x7f18e400d8e0 2023-10-05T19:28:28.126 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.122+0000 7f18f113a700 1 -- 172.21.15.167:0/1910438506 <== osd.0 v2:172.21.15.118:6800/27690 3 ==== osd_op_reply(12 [pgnls start_epoch 30 out=112b] v23'1 uv1 ondisk = 1) v8 ==== 144+0+112 (crc 0 0 0) 0x7f18e8014040 con 0x7f18e400d8e0 2023-10-05T19:28:28.126 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.122+0000 7f18f113a700 1 -- 172.21.15.167:0/1910438506 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:13 5.1c 5:38000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x556bff5e5bb0 con 0x7f18e001b740 2023-10-05T19:28:28.126 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.122+0000 7f18f0138700 1 -- 172.21.15.167:0/1910438506 <== osd.2 v2:172.21.15.167:6816/30037 2 ==== osd_op_reply(13 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f18e4006070 con 0x7f18e001b740 2023-10-05T19:28:28.126 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.122+0000 7f18f0138700 1 -- 172.21.15.167:0/1910438506 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:14 5.2 5:40000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x556bff5e5bb0 con 0x556bff5e3290 2023-10-05T19:28:28.127 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.122+0000 7f18f0939700 1 -- 172.21.15.167:0/1910438506 <== osd.3 v2:172.21.15.167:6808/30036 2 ==== osd_op_reply(14 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f18e0038c40 con 0x556bff5e3290 2023-10-05T19:28:28.127 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.122+0000 7f18f0939700 1 -- 172.21.15.167:0/1910438506 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:15 5.12 5:48000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x556bff5e5bb0 con 0x556bff5ae6e0 2023-10-05T19:28:28.128 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.122+0000 7f18f113a700 1 -- 172.21.15.167:0/1910438506 <== osd.1 v2:172.21.15.167:6800/30035 8 ==== osd_op_reply(15 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f18e8014040 con 0x556bff5ae6e0 2023-10-05T19:28:28.128 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.122+0000 7f18f113a700 1 -- 172.21.15.167:0/1910438506 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:16 5.a 5:50000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x556bff5e5bb0 con 0x7f18e400d8e0 2023-10-05T19:28:28.129 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.122+0000 7f18f113a700 1 -- 172.21.15.167:0/1910438506 <== osd.0 v2:172.21.15.118:6800/27690 4 ==== osd_op_reply(16 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f18e8022040 con 0x7f18e400d8e0 2023-10-05T19:28:28.129 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.122+0000 7f18f113a700 1 -- 172.21.15.167:0/1910438506 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:17 5.1a 5:58000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x556bff5e5bb0 con 0x556bff5ae6e0 2023-10-05T19:28:28.129 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.122+0000 7f18f113a700 1 -- 172.21.15.167:0/1910438506 <== osd.1 v2:172.21.15.167:6800/30035 9 ==== osd_op_reply(17 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f18e8022040 con 0x556bff5ae6e0 2023-10-05T19:28:28.130 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.122+0000 7f18f113a700 1 -- 172.21.15.167:0/1910438506 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:18 5.6 5:60000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x556bff5e5bb0 con 0x7f18e001b740 2023-10-05T19:28:28.130 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.126+0000 7f18f0138700 1 -- 172.21.15.167:0/1910438506 <== osd.2 v2:172.21.15.167:6816/30037 3 ==== osd_op_reply(18 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f18e4006070 con 0x7f18e001b740 2023-10-05T19:28:28.130 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.126+0000 7f18f0138700 1 -- 172.21.15.167:0/1910438506 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:19 5.16 5:68000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x556bff5e5bb0 con 0x556bff5e3290 2023-10-05T19:28:28.131 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.126+0000 7f18f0939700 1 -- 172.21.15.167:0/1910438506 <== osd.3 v2:172.21.15.167:6808/30036 3 ==== osd_op_reply(19 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f18e0038c40 con 0x556bff5e3290 2023-10-05T19:28:28.131 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.126+0000 7f18f0939700 1 -- 172.21.15.167:0/1910438506 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:20 5.e 5:70000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x556bff5e5bb0 con 0x556bff5e3290 2023-10-05T19:28:28.132 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.126+0000 7f18f0939700 1 -- 172.21.15.167:0/1910438506 <== osd.3 v2:172.21.15.167:6808/30036 4 ==== osd_op_reply(20 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f18e0038c40 con 0x556bff5e3290 2023-10-05T19:28:28.132 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.126+0000 7f18f0939700 1 -- 172.21.15.167:0/1910438506 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:21 5.1e 5:78000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x556bff5e5bb0 con 0x7f18e400d8e0 2023-10-05T19:28:28.133 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.126+0000 7f18f113a700 1 -- 172.21.15.167:0/1910438506 <== osd.0 v2:172.21.15.118:6800/27690 5 ==== osd_op_reply(21 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f18e8022040 con 0x7f18e400d8e0 2023-10-05T19:28:28.133 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.126+0000 7f18f113a700 1 -- 172.21.15.167:0/1910438506 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:22 5.1 5:80000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x556bff5e5bb0 con 0x556bff5e3290 2023-10-05T19:28:28.134 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.130+0000 7f18f0939700 1 -- 172.21.15.167:0/1910438506 <== osd.3 v2:172.21.15.167:6808/30036 5 ==== osd_op_reply(22 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f18e0038c40 con 0x556bff5e3290 2023-10-05T19:28:28.134 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.130+0000 7f18f0939700 1 -- 172.21.15.167:0/1910438506 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:23 5.11 5:88000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x556bff5e5bb0 con 0x556bff5e3290 2023-10-05T19:28:28.135 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.130+0000 7f18f0939700 1 -- 172.21.15.167:0/1910438506 <== osd.3 v2:172.21.15.167:6808/30036 6 ==== osd_op_reply(23 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f18e0038c40 con 0x556bff5e3290 2023-10-05T19:28:28.135 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.130+0000 7f18f0939700 1 -- 172.21.15.167:0/1910438506 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:24 5.9 5:90000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x556bff5e5bb0 con 0x556bff5ae6e0 2023-10-05T19:28:28.135 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.130+0000 7f18f113a700 1 -- 172.21.15.167:0/1910438506 <== osd.1 v2:172.21.15.167:6800/30035 10 ==== osd_op_reply(24 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f18e8022040 con 0x556bff5ae6e0 2023-10-05T19:28:28.136 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.130+0000 7f18f113a700 1 -- 172.21.15.167:0/1910438506 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:25 5.19 5:98000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x556bff5e5bb0 con 0x556bff5ae6e0 2023-10-05T19:28:28.136 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.130+0000 7f18f113a700 1 -- 172.21.15.167:0/1910438506 <== osd.1 v2:172.21.15.167:6800/30035 11 ==== osd_op_reply(25 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f18e8022040 con 0x556bff5ae6e0 2023-10-05T19:28:28.136 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.130+0000 7f18f113a700 1 -- 172.21.15.167:0/1910438506 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:26 5.5 5:a0000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x556bff5e5bb0 con 0x7f18e400d8e0 2023-10-05T19:28:28.137 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.130+0000 7f18f113a700 1 -- 172.21.15.167:0/1910438506 <== osd.0 v2:172.21.15.118:6800/27690 6 ==== osd_op_reply(26 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f18e8022040 con 0x7f18e400d8e0 2023-10-05T19:28:28.137 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.130+0000 7f18f113a700 1 -- 172.21.15.167:0/1910438506 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:27 5.15 5:a8000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x556bff5e5bb0 con 0x556bff5e3290 2023-10-05T19:28:28.138 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.134+0000 7f18f0939700 1 -- 172.21.15.167:0/1910438506 <== osd.3 v2:172.21.15.167:6808/30036 7 ==== osd_op_reply(27 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f18e0038c40 con 0x556bff5e3290 2023-10-05T19:28:28.138 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.134+0000 7f18f0939700 1 -- 172.21.15.167:0/1910438506 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:28 5.d 5:b0000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x556bff5e5bb0 con 0x7f18e001b740 2023-10-05T19:28:28.139 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.134+0000 7f18f0138700 1 -- 172.21.15.167:0/1910438506 <== osd.2 v2:172.21.15.167:6816/30037 4 ==== osd_op_reply(28 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f18e4006070 con 0x7f18e001b740 2023-10-05T19:28:28.139 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.134+0000 7f18f0138700 1 -- 172.21.15.167:0/1910438506 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:29 5.1d 5:b8000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x556bff5e5bb0 con 0x556bff5ae6e0 2023-10-05T19:28:28.140 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.134+0000 7f18f113a700 1 -- 172.21.15.167:0/1910438506 <== osd.1 v2:172.21.15.167:6800/30035 12 ==== osd_op_reply(29 [pgnls start_epoch 30 out=107b] v30'2 uv2 ondisk = 1) v8 ==== 144+0+107 (crc 0 0 0) 0x7f18e8022040 con 0x556bff5ae6e0 2023-10-05T19:28:28.140 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.134+0000 7f18f113a700 1 -- 172.21.15.167:0/1910438506 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:30 5.3 5:c0000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x556bff5e5bb0 con 0x7f18e400d8e0 2023-10-05T19:28:28.141 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.134+0000 7f18f113a700 1 -- 172.21.15.167:0/1910438506 <== osd.0 v2:172.21.15.118:6800/27690 7 ==== osd_op_reply(30 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f18e8022040 con 0x7f18e400d8e0 2023-10-05T19:28:28.141 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.134+0000 7f18f113a700 1 -- 172.21.15.167:0/1910438506 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:31 5.13 5:c8000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x556bff5e5bb0 con 0x556bff5e3290 2023-10-05T19:28:28.141 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.134+0000 7f18f0939700 1 -- 172.21.15.167:0/1910438506 <== osd.3 v2:172.21.15.167:6808/30036 8 ==== osd_op_reply(31 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f18e0038c40 con 0x556bff5e3290 2023-10-05T19:28:28.142 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.134+0000 7f18f0939700 1 -- 172.21.15.167:0/1910438506 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:32 5.b 5:d0000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x556bff5e5bb0 con 0x7f18e400d8e0 2023-10-05T19:28:28.143 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.138+0000 7f18f113a700 1 -- 172.21.15.167:0/1910438506 <== osd.0 v2:172.21.15.118:6800/27690 8 ==== osd_op_reply(32 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f18e8022040 con 0x7f18e400d8e0 2023-10-05T19:28:28.143 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.138+0000 7f18f113a700 1 -- 172.21.15.167:0/1910438506 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:33 5.1b 5:d8000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x556bff5e5bb0 con 0x7f18e001b740 2023-10-05T19:28:28.143 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.138+0000 7f18f0138700 1 -- 172.21.15.167:0/1910438506 <== osd.2 v2:172.21.15.167:6816/30037 5 ==== osd_op_reply(33 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f18e4006070 con 0x7f18e001b740 2023-10-05T19:28:28.143 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.138+0000 7f18f0138700 1 -- 172.21.15.167:0/1910438506 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:34 5.7 5:e0000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x556bff5e5bb0 con 0x556bff5e3290 2023-10-05T19:28:28.144 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.138+0000 7f18f0939700 1 -- 172.21.15.167:0/1910438506 <== osd.3 v2:172.21.15.167:6808/30036 9 ==== osd_op_reply(34 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f18e0038c40 con 0x556bff5e3290 2023-10-05T19:28:28.144 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.138+0000 7f18f0939700 1 -- 172.21.15.167:0/1910438506 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:35 5.17 5:e8000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x556bff5e5bb0 con 0x7f18e400d8e0 2023-10-05T19:28:28.146 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.138+0000 7f18f113a700 1 -- 172.21.15.167:0/1910438506 <== osd.0 v2:172.21.15.118:6800/27690 9 ==== osd_op_reply(35 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f18e8022040 con 0x7f18e400d8e0 2023-10-05T19:28:28.146 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.138+0000 7f18f113a700 1 -- 172.21.15.167:0/1910438506 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:36 5.f 5:f0000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x556bff5e5bb0 con 0x556bff5ae6e0 2023-10-05T19:28:28.146 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.142+0000 7f18f113a700 1 -- 172.21.15.167:0/1910438506 <== osd.1 v2:172.21.15.167:6800/30035 13 ==== osd_op_reply(36 [pgnls start_epoch 30 out=115b] v23'2 uv2 ondisk = 1) v8 ==== 144+0+115 (crc 0 0 0) 0x7f18e802f080 con 0x556bff5ae6e0 2023-10-05T19:28:28.147 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.142+0000 7f18f113a700 1 -- 172.21.15.167:0/1910438506 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:37 5.1f 5:f8000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x556bff5e5bb0 con 0x7f18e001b740 2023-10-05T19:28:28.148 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.142+0000 7f18f0138700 1 -- 172.21.15.167:0/1910438506 <== osd.2 v2:172.21.15.167:6816/30037 6 ==== osd_op_reply(37 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f18e4006070 con 0x7f18e001b740 2023-10-05T19:28:28.148 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.142+0000 7f18f2e04e40 20 RGWRados::pool_iterate: got zonegroup_info.4f1f719d-9648-417b-b10c-e5d939825419 2023-10-05T19:28:28.148 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.142+0000 7f18f2e04e40 20 RGWRados::pool_iterate: got zone_info.9a9df965-6612-45d4-a45f-6e3c889bacc2 2023-10-05T19:28:28.148 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.142+0000 7f18f2e04e40 20 RGWRados::pool_iterate: got zone_names.default 2023-10-05T19:28:28.148 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.142+0000 7f18f2e04e40 20 RGWRados::pool_iterate: got zonegroups_names.default 2023-10-05T19:28:28.148 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.142+0000 7f18f2e04e40 20 rados->read ofs=0 len=0 2023-10-05T19:28:28.148 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.142+0000 7f18f2e04e40 1 -- 172.21.15.167:0/1910438506 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:38 5.12 5:49953fa1:::default.realm:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x556bff5e2040 con 0x556bff5ae6e0 2023-10-05T19:28:28.149 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.142+0000 7f18f113a700 1 -- 172.21.15.167:0/1910438506 <== osd.1 v2:172.21.15.167:6800/30035 14 ==== osd_op_reply(38 default.realm [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 ==== 157+0+0 (crc 0 0 0) 0x7f18e802f080 con 0x556bff5ae6e0 2023-10-05T19:28:28.149 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.142+0000 7f18f2e04e40 20 rados_obj.operate() r=-2 bl.length=0 2023-10-05T19:28:28.149 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.142+0000 7f18f2e04e40 20 rados->read ofs=0 len=0 2023-10-05T19:28:28.149 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.142+0000 7f18f2e04e40 1 -- 172.21.15.167:0/1910438506 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:39 5.f 5:f4c53578:::zonegroups_names.default:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x556bff5e2930 con 0x556bff5ae6e0 2023-10-05T19:28:28.150 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.142+0000 7f18f113a700 1 -- 172.21.15.167:0/1910438506 <== osd.1 v2:172.21.15.167:6800/30035 15 ==== osd_op_reply(39 zonegroups_names.default [read 0~46 out=46b] v0'0 uv2 ondisk = 0) v8 ==== 168+0+46 (crc 0 0 0) 0x7f18e802f080 con 0x556bff5ae6e0 2023-10-05T19:28:28.150 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.142+0000 7f18f2e04e40 20 rados_obj.operate() r=0 bl.length=46 2023-10-05T19:28:28.150 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.142+0000 7f18f2e04e40 20 rados->read ofs=0 len=0 2023-10-05T19:28:28.150 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.142+0000 7f18f2e04e40 1 -- 172.21.15.167:0/1910438506 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:40 5.c 5:30d299a5:::zonegroup_info.4f1f719d-9648-417b-b10c-e5d939825419:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x556bff5e5ff0 con 0x7f18e400d8e0 2023-10-05T19:28:28.154 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.146+0000 7f18f113a700 1 -- 172.21.15.167:0/1910438506 <== osd.0 v2:172.21.15.118:6800/27690 10 ==== osd_op_reply(40 zonegroup_info.4f1f719d-9648-417b-b10c-e5d939825419 [read 0~362 out=362b] v0'0 uv1 ondisk = 0) v8 ==== 195+0+362 (crc 0 0 0) 0x7f18e802f080 con 0x7f18e400d8e0 2023-10-05T19:28:28.154 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.146+0000 7f18f2e04e40 20 rados_obj.operate() r=0 bl.length=362 2023-10-05T19:28:28.154 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.146+0000 7f18f2e04e40 20 rados->read ofs=0 len=0 2023-10-05T19:28:28.154 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.146+0000 7f18f2e04e40 1 -- 172.21.15.167:0/1910438506 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:41 5.13 5:cbbf6c60:::converted:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x556bff5e5ff0 con 0x556bff5e3290 2023-10-05T19:28:28.154 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.146+0000 7f18f0939700 1 -- 172.21.15.167:0/1910438506 <== osd.3 v2:172.21.15.167:6808/30036 10 ==== osd_op_reply(41 converted [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 ==== 153+0+0 (crc 0 0 0) 0x7f18e0038c40 con 0x556bff5e3290 2023-10-05T19:28:28.154 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.146+0000 7f18f2e04e40 20 rados_obj.operate() r=-2 bl.length=0 2023-10-05T19:28:28.155 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.146+0000 7f18f2e04e40 20 rados->read ofs=0 len=0 2023-10-05T19:28:28.155 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.146+0000 7f18f2e04e40 1 -- 172.21.15.167:0/1910438506 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:42 5.12 5:49953fa1:::default.realm:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x556bff5e63a0 con 0x556bff5ae6e0 2023-10-05T19:28:28.155 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.146+0000 7f18f113a700 1 -- 172.21.15.167:0/1910438506 <== osd.1 v2:172.21.15.167:6800/30035 16 ==== osd_op_reply(42 default.realm [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 ==== 157+0+0 (crc 0 0 0) 0x7f18e802f080 con 0x556bff5ae6e0 2023-10-05T19:28:28.155 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.146+0000 7f18f2e04e40 20 rados_obj.operate() r=-2 bl.length=0 2023-10-05T19:28:28.155 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.146+0000 7f18f2e04e40 20 rados->read ofs=0 len=0 2023-10-05T19:28:28.155 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.146+0000 7f18f2e04e40 1 -- 172.21.15.167:0/1910438506 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:43 5.f 5:f4c53578:::zonegroups_names.default:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x556bff5e67d0 con 0x556bff5ae6e0 2023-10-05T19:28:28.155 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.146+0000 7f18f113a700 1 -- 172.21.15.167:0/1910438506 <== osd.1 v2:172.21.15.167:6800/30035 17 ==== osd_op_reply(43 zonegroups_names.default [read 0~46 out=46b] v0'0 uv2 ondisk = 0) v8 ==== 168+0+46 (crc 0 0 0) 0x7f18e802f080 con 0x556bff5ae6e0 2023-10-05T19:28:28.155 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.146+0000 7f18f2e04e40 20 rados_obj.operate() r=0 bl.length=46 2023-10-05T19:28:28.155 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.146+0000 7f18f2e04e40 1 -- 172.21.15.167:0/1910438506 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:44 5.0 5:00000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x556bff5e77e0 con 0x556bff5e3290 2023-10-05T19:28:28.156 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.150+0000 7f18f0939700 1 -- 172.21.15.167:0/1910438506 <== osd.3 v2:172.21.15.167:6808/30036 11 ==== osd_op_reply(44 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f18e0038c40 con 0x556bff5e3290 2023-10-05T19:28:28.156 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.150+0000 7f18f0939700 1 -- 172.21.15.167:0/1910438506 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:45 5.10 5:08000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x556bff5e77e0 con 0x7f18e001b740 2023-10-05T19:28:28.156 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.150+0000 7f18f0138700 1 -- 172.21.15.167:0/1910438506 <== osd.2 v2:172.21.15.167:6816/30037 7 ==== osd_op_reply(45 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f18e4006070 con 0x7f18e001b740 2023-10-05T19:28:28.156 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.150+0000 7f18f0138700 1 -- 172.21.15.167:0/1910438506 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:46 5.8 5:10000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x556bff5e77e0 con 0x7f18e400d8e0 2023-10-05T19:28:28.157 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.150+0000 7f18f113a700 1 -- 172.21.15.167:0/1910438506 <== osd.0 v2:172.21.15.118:6800/27690 11 ==== osd_op_reply(46 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f18e802f080 con 0x7f18e400d8e0 2023-10-05T19:28:28.157 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.150+0000 7f18f113a700 1 -- 172.21.15.167:0/1910438506 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:47 5.18 5:18000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x556bff5e2930 con 0x556bff5ae6e0 2023-10-05T19:28:28.157 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.150+0000 7f18f113a700 1 -- 172.21.15.167:0/1910438506 <== osd.1 v2:172.21.15.167:6800/30035 18 ==== osd_op_reply(47 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f18e802f080 con 0x556bff5ae6e0 2023-10-05T19:28:28.157 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.150+0000 7f18f113a700 1 -- 172.21.15.167:0/1910438506 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:48 5.4 5:20000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x556bff5e2930 con 0x556bff5ae6e0 2023-10-05T19:28:28.158 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.154+0000 7f18f113a700 1 -- 172.21.15.167:0/1910438506 <== osd.1 v2:172.21.15.167:6800/30035 19 ==== osd_op_reply(48 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f18e802f080 con 0x556bff5ae6e0 2023-10-05T19:28:28.158 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.154+0000 7f18f113a700 1 -- 172.21.15.167:0/1910438506 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:49 5.14 5:28000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x556bff5e2930 con 0x7f18e400d8e0 2023-10-05T19:28:28.159 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.154+0000 7f18f113a700 1 -- 172.21.15.167:0/1910438506 <== osd.0 v2:172.21.15.118:6800/27690 12 ==== osd_op_reply(49 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f18e802f080 con 0x7f18e400d8e0 2023-10-05T19:28:28.159 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.154+0000 7f18f113a700 1 -- 172.21.15.167:0/1910438506 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:50 5.c 5:30000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x556bff5e2930 con 0x7f18e400d8e0 2023-10-05T19:28:28.160 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.154+0000 7f18f113a700 1 -- 172.21.15.167:0/1910438506 <== osd.0 v2:172.21.15.118:6800/27690 13 ==== osd_op_reply(50 [pgnls start_epoch 30 out=112b] v23'1 uv1 ondisk = 1) v8 ==== 144+0+112 (crc 0 0 0) 0x7f18e802f080 con 0x7f18e400d8e0 2023-10-05T19:28:28.160 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.154+0000 7f18f113a700 1 -- 172.21.15.167:0/1910438506 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:51 5.1c 5:38000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x556bff5e2930 con 0x7f18e001b740 2023-10-05T19:28:28.161 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.154+0000 7f18f0138700 1 -- 172.21.15.167:0/1910438506 <== osd.2 v2:172.21.15.167:6816/30037 8 ==== osd_op_reply(51 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f18e4006070 con 0x7f18e001b740 2023-10-05T19:28:28.161 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.154+0000 7f18f0138700 1 -- 172.21.15.167:0/1910438506 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:52 5.2 5:40000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x556bff5e2930 con 0x556bff5e3290 2023-10-05T19:28:28.162 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.154+0000 7f18f0939700 1 -- 172.21.15.167:0/1910438506 <== osd.3 v2:172.21.15.167:6808/30036 12 ==== osd_op_reply(52 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f18e0038c40 con 0x556bff5e3290 2023-10-05T19:28:28.162 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.154+0000 7f18f0939700 1 -- 172.21.15.167:0/1910438506 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:53 5.12 5:48000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x556bff5e2930 con 0x556bff5ae6e0 2023-10-05T19:28:28.162 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.158+0000 7f18f113a700 1 -- 172.21.15.167:0/1910438506 <== osd.1 v2:172.21.15.167:6800/30035 20 ==== osd_op_reply(53 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f18e802f080 con 0x556bff5ae6e0 2023-10-05T19:28:28.163 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.158+0000 7f18f113a700 1 -- 172.21.15.167:0/1910438506 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:54 5.a 5:50000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x556bff5e2930 con 0x7f18e400d8e0 2023-10-05T19:28:28.163 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.158+0000 7f18f113a700 1 -- 172.21.15.167:0/1910438506 <== osd.0 v2:172.21.15.118:6800/27690 14 ==== osd_op_reply(54 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f18e802f080 con 0x7f18e400d8e0 2023-10-05T19:28:28.164 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.158+0000 7f18f113a700 1 -- 172.21.15.167:0/1910438506 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:55 5.1a 5:58000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x556bff5e2930 con 0x556bff5ae6e0 2023-10-05T19:28:28.164 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.158+0000 7f18f113a700 1 -- 172.21.15.167:0/1910438506 <== osd.1 v2:172.21.15.167:6800/30035 21 ==== osd_op_reply(55 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f18e802f080 con 0x556bff5ae6e0 2023-10-05T19:28:28.164 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.158+0000 7f18f113a700 1 -- 172.21.15.167:0/1910438506 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:56 5.6 5:60000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x556bff5e2930 con 0x7f18e001b740 2023-10-05T19:28:28.165 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.158+0000 7f18f0138700 1 -- 172.21.15.167:0/1910438506 <== osd.2 v2:172.21.15.167:6816/30037 9 ==== osd_op_reply(56 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f18e4006070 con 0x7f18e001b740 2023-10-05T19:28:28.165 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.158+0000 7f18f0138700 1 -- 172.21.15.167:0/1910438506 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:57 5.16 5:68000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x556bff5e2930 con 0x556bff5e3290 2023-10-05T19:28:28.166 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.162+0000 7f18f0939700 1 -- 172.21.15.167:0/1910438506 <== osd.3 v2:172.21.15.167:6808/30036 13 ==== osd_op_reply(57 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f18e0038c40 con 0x556bff5e3290 2023-10-05T19:28:28.166 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.162+0000 7f18f0939700 1 -- 172.21.15.167:0/1910438506 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:58 5.e 5:70000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x556bff5e2930 con 0x556bff5e3290 2023-10-05T19:28:28.167 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.162+0000 7f18f0939700 1 -- 172.21.15.167:0/1910438506 <== osd.3 v2:172.21.15.167:6808/30036 14 ==== osd_op_reply(58 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f18e0038c40 con 0x556bff5e3290 2023-10-05T19:28:28.167 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.162+0000 7f18f0939700 1 -- 172.21.15.167:0/1910438506 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:59 5.1e 5:78000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x556bff5e2930 con 0x7f18e400d8e0 2023-10-05T19:28:28.168 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.162+0000 7f18f113a700 1 -- 172.21.15.167:0/1910438506 <== osd.0 v2:172.21.15.118:6800/27690 15 ==== osd_op_reply(59 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f18e802f080 con 0x7f18e400d8e0 2023-10-05T19:28:28.168 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.162+0000 7f18f113a700 1 -- 172.21.15.167:0/1910438506 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:60 5.1 5:80000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x556bff5e2930 con 0x556bff5e3290 2023-10-05T19:28:28.169 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.162+0000 7f18f0939700 1 -- 172.21.15.167:0/1910438506 <== osd.3 v2:172.21.15.167:6808/30036 15 ==== osd_op_reply(60 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f18e0038c40 con 0x556bff5e3290 2023-10-05T19:28:28.169 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.162+0000 7f18f0939700 1 -- 172.21.15.167:0/1910438506 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:61 5.11 5:88000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x556bff5e2930 con 0x556bff5e3290 2023-10-05T19:28:28.169 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.162+0000 7f18f0939700 1 -- 172.21.15.167:0/1910438506 <== osd.3 v2:172.21.15.167:6808/30036 16 ==== osd_op_reply(61 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f18e0038c40 con 0x556bff5e3290 2023-10-05T19:28:28.169 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.162+0000 7f18f0939700 1 -- 172.21.15.167:0/1910438506 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:62 5.9 5:90000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x556bff5e2930 con 0x556bff5ae6e0 2023-10-05T19:28:28.170 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.166+0000 7f18f113a700 1 -- 172.21.15.167:0/1910438506 <== osd.1 v2:172.21.15.167:6800/30035 22 ==== osd_op_reply(62 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f18e802f080 con 0x556bff5ae6e0 2023-10-05T19:28:28.170 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.166+0000 7f18f113a700 1 -- 172.21.15.167:0/1910438506 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:63 5.19 5:98000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x556bff5e2930 con 0x556bff5ae6e0 2023-10-05T19:28:28.171 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.166+0000 7f18f113a700 1 -- 172.21.15.167:0/1910438506 <== osd.1 v2:172.21.15.167:6800/30035 23 ==== osd_op_reply(63 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f18e802f080 con 0x556bff5ae6e0 2023-10-05T19:28:28.171 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.166+0000 7f18f113a700 1 -- 172.21.15.167:0/1910438506 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:64 5.5 5:a0000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x556bff5e2930 con 0x7f18e400d8e0 2023-10-05T19:28:28.172 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.166+0000 7f18f113a700 1 -- 172.21.15.167:0/1910438506 <== osd.0 v2:172.21.15.118:6800/27690 16 ==== osd_op_reply(64 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f18e802f080 con 0x7f18e400d8e0 2023-10-05T19:28:28.172 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.166+0000 7f18f113a700 1 -- 172.21.15.167:0/1910438506 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:65 5.15 5:a8000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x556bff5e2930 con 0x556bff5e3290 2023-10-05T19:28:28.173 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.166+0000 7f18f0939700 1 -- 172.21.15.167:0/1910438506 <== osd.3 v2:172.21.15.167:6808/30036 17 ==== osd_op_reply(65 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f18e0038c40 con 0x556bff5e3290 2023-10-05T19:28:28.173 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.166+0000 7f18f0939700 1 -- 172.21.15.167:0/1910438506 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:66 5.d 5:b0000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x556bff5e2930 con 0x7f18e001b740 2023-10-05T19:28:28.174 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.170+0000 7f18f0138700 1 -- 172.21.15.167:0/1910438506 <== osd.2 v2:172.21.15.167:6816/30037 10 ==== osd_op_reply(66 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f18e4006070 con 0x7f18e001b740 2023-10-05T19:28:28.174 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.170+0000 7f18f0138700 1 -- 172.21.15.167:0/1910438506 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:67 5.1d 5:b8000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x556bff5e2930 con 0x556bff5ae6e0 2023-10-05T19:28:28.175 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.170+0000 7f18f113a700 1 -- 172.21.15.167:0/1910438506 <== osd.1 v2:172.21.15.167:6800/30035 24 ==== osd_op_reply(67 [pgnls start_epoch 30 out=107b] v30'2 uv2 ondisk = 1) v8 ==== 144+0+107 (crc 0 0 0) 0x7f18e802f080 con 0x556bff5ae6e0 2023-10-05T19:28:28.175 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.170+0000 7f18f113a700 1 -- 172.21.15.167:0/1910438506 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:68 5.3 5:c0000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x556bff5e2930 con 0x7f18e400d8e0 2023-10-05T19:28:28.176 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.170+0000 7f18f113a700 1 -- 172.21.15.167:0/1910438506 <== osd.0 v2:172.21.15.118:6800/27690 17 ==== osd_op_reply(68 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f18e802f080 con 0x7f18e400d8e0 2023-10-05T19:28:28.176 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.170+0000 7f18f113a700 1 -- 172.21.15.167:0/1910438506 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:69 5.13 5:c8000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x556bff5e2930 con 0x556bff5e3290 2023-10-05T19:28:28.177 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.170+0000 7f18f0939700 1 -- 172.21.15.167:0/1910438506 <== osd.3 v2:172.21.15.167:6808/30036 18 ==== osd_op_reply(69 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f18e0038c40 con 0x556bff5e3290 2023-10-05T19:28:28.177 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.170+0000 7f18f0939700 1 -- 172.21.15.167:0/1910438506 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:70 5.b 5:d0000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x556bff5e2930 con 0x7f18e400d8e0 2023-10-05T19:28:28.178 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.174+0000 7f18f113a700 1 -- 172.21.15.167:0/1910438506 <== osd.0 v2:172.21.15.118:6800/27690 18 ==== osd_op_reply(70 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f18e802f080 con 0x7f18e400d8e0 2023-10-05T19:28:28.178 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.174+0000 7f18f113a700 1 -- 172.21.15.167:0/1910438506 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:71 5.1b 5:d8000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x556bff5e2930 con 0x7f18e001b740 2023-10-05T19:28:28.179 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.174+0000 7f18f0138700 1 -- 172.21.15.167:0/1910438506 <== osd.2 v2:172.21.15.167:6816/30037 11 ==== osd_op_reply(71 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f18e4006070 con 0x7f18e001b740 2023-10-05T19:28:28.179 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.174+0000 7f18f0138700 1 -- 172.21.15.167:0/1910438506 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:72 5.7 5:e0000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x556bff5e2930 con 0x556bff5e3290 2023-10-05T19:28:28.180 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.174+0000 7f18f0939700 1 -- 172.21.15.167:0/1910438506 <== osd.3 v2:172.21.15.167:6808/30036 19 ==== osd_op_reply(72 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f18e0038c40 con 0x556bff5e3290 2023-10-05T19:28:28.180 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.174+0000 7f18f0939700 1 -- 172.21.15.167:0/1910438506 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:73 5.17 5:e8000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x556bff5e2930 con 0x7f18e400d8e0 2023-10-05T19:28:28.181 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.174+0000 7f18f113a700 1 -- 172.21.15.167:0/1910438506 <== osd.0 v2:172.21.15.118:6800/27690 19 ==== osd_op_reply(73 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f18e802f080 con 0x7f18e400d8e0 2023-10-05T19:28:28.181 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.174+0000 7f18f113a700 1 -- 172.21.15.167:0/1910438506 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:74 5.f 5:f0000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x556bff5e2930 con 0x556bff5ae6e0 2023-10-05T19:28:28.182 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.174+0000 7f18f113a700 1 -- 172.21.15.167:0/1910438506 <== osd.1 v2:172.21.15.167:6800/30035 25 ==== osd_op_reply(74 [pgnls start_epoch 30 out=115b] v23'2 uv2 ondisk = 1) v8 ==== 144+0+115 (crc 0 0 0) 0x7f18e802f080 con 0x556bff5ae6e0 2023-10-05T19:28:28.182 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.174+0000 7f18f113a700 1 -- 172.21.15.167:0/1910438506 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:75 5.1f 5:f8000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x556bff5e2930 con 0x7f18e001b740 2023-10-05T19:28:28.183 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.178+0000 7f18f0138700 1 -- 172.21.15.167:0/1910438506 <== osd.2 v2:172.21.15.167:6816/30037 12 ==== osd_op_reply(75 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f18e4006070 con 0x7f18e001b740 2023-10-05T19:28:28.183 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.178+0000 7f18f2e04e40 20 RGWRados::pool_iterate: got zonegroup_info.4f1f719d-9648-417b-b10c-e5d939825419 2023-10-05T19:28:28.183 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.178+0000 7f18f2e04e40 20 RGWRados::pool_iterate: got zone_info.9a9df965-6612-45d4-a45f-6e3c889bacc2 2023-10-05T19:28:28.183 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.178+0000 7f18f2e04e40 20 RGWRados::pool_iterate: got zone_names.default 2023-10-05T19:28:28.183 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.178+0000 7f18f2e04e40 20 RGWRados::pool_iterate: got zonegroups_names.default 2023-10-05T19:28:28.183 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.178+0000 7f18f2e04e40 20 rados->read ofs=0 len=0 2023-10-05T19:28:28.183 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.178+0000 7f18f2e04e40 1 -- 172.21.15.167:0/1910438506 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:76 5.f 5:f43b6ece:::zone_names.default:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x556bff5e67d0 con 0x556bff5ae6e0 2023-10-05T19:28:28.184 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.178+0000 7f18f113a700 1 -- 172.21.15.167:0/1910438506 <== osd.1 v2:172.21.15.167:6800/30035 26 ==== osd_op_reply(76 zone_names.default [read 0~46 out=46b] v0'0 uv1 ondisk = 0) v8 ==== 162+0+46 (crc 0 0 0) 0x7f18e802f080 con 0x556bff5ae6e0 2023-10-05T19:28:28.184 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.178+0000 7f18f2e04e40 20 rados_obj.operate() r=0 bl.length=46 2023-10-05T19:28:28.184 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.178+0000 7f18f2e04e40 20 rados->read ofs=0 len=0 2023-10-05T19:28:28.184 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.178+0000 7f18f2e04e40 1 -- 172.21.15.167:0/1910438506 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:77 5.1d 5:b8374eb4:::zone_info.9a9df965-6612-45d4-a45f-6e3c889bacc2:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x556bff5e6b80 con 0x556bff5ae6e0 2023-10-05T19:28:28.185 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.178+0000 7f18f113a700 1 -- 172.21.15.167:0/1910438506 <== osd.1 v2:172.21.15.167:6800/30035 27 ==== osd_op_reply(77 zone_info.9a9df965-6612-45d4-a45f-6e3c889bacc2 [read 0~874 out=874b] v0'0 uv2 ondisk = 0) v8 ==== 190+0+874 (crc 0 0 0) 0x7f18e802f080 con 0x556bff5ae6e0 2023-10-05T19:28:28.185 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.178+0000 7f18f2e04e40 20 rados_obj.operate() r=0 bl.length=874 2023-10-05T19:28:28.185 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.178+0000 7f18f2e04e40 20 rados->read ofs=0 len=0 2023-10-05T19:28:28.185 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.178+0000 7f18f2e04e40 1 -- 172.21.15.167:0/1910438506 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:78 5.f 5:f4c53578:::zonegroups_names.default:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x556bff5e6bb0 con 0x556bff5ae6e0 2023-10-05T19:28:28.185 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.178+0000 7f18f113a700 1 -- 172.21.15.167:0/1910438506 <== osd.1 v2:172.21.15.167:6800/30035 28 ==== osd_op_reply(78 zonegroups_names.default [read 0~46 out=46b] v0'0 uv2 ondisk = 0) v8 ==== 168+0+46 (crc 0 0 0) 0x7f18e802f080 con 0x556bff5ae6e0 2023-10-05T19:28:28.185 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.178+0000 7f18f2e04e40 20 rados_obj.operate() r=0 bl.length=46 2023-10-05T19:28:28.186 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.178+0000 7f18f2e04e40 20 rados->read ofs=0 len=0 2023-10-05T19:28:28.186 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.178+0000 7f18f2e04e40 1 -- 172.21.15.167:0/1910438506 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:79 5.c 5:30d299a5:::zonegroup_info.4f1f719d-9648-417b-b10c-e5d939825419:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x556bff5e6fe0 con 0x7f18e400d8e0 2023-10-05T19:28:28.186 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.178+0000 7f18f113a700 1 -- 172.21.15.167:0/1910438506 <== osd.0 v2:172.21.15.118:6800/27690 20 ==== osd_op_reply(79 zonegroup_info.4f1f719d-9648-417b-b10c-e5d939825419 [read 0~362 out=362b] v0'0 uv1 ondisk = 0) v8 ==== 195+0+362 (crc 0 0 0) 0x7f18e802f080 con 0x7f18e400d8e0 2023-10-05T19:28:28.186 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.182+0000 7f18f2e04e40 20 rados_obj.operate() r=0 bl.length=362 2023-10-05T19:28:28.187 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.182+0000 7f18f2e04e40 20 rados->read ofs=0 len=0 2023-10-05T19:28:28.187 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.182+0000 7f18f2e04e40 1 -- 172.21.15.167:0/1910438506 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:80 5.3 5:c0ad8cbd:::region_map:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x556bff5e6fe0 con 0x7f18e400d8e0 2023-10-05T19:28:28.187 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.182+0000 7f18f113a700 1 -- 172.21.15.167:0/1910438506 <== osd.0 v2:172.21.15.118:6800/27690 21 ==== osd_op_reply(80 region_map [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 ==== 154+0+0 (crc 0 0 0) 0x7f18e802f080 con 0x7f18e400d8e0 2023-10-05T19:28:28.187 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.182+0000 7f18f2e04e40 20 rados_obj.operate() r=-2 bl.length=0 2023-10-05T19:28:28.187 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.182+0000 7f18f2e04e40 20 zone default found 2023-10-05T19:28:28.187 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.182+0000 7f18f2e04e40 4 Realm: () 2023-10-05T19:28:28.188 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.182+0000 7f18f2e04e40 4 ZoneGroup: default (4f1f719d-9648-417b-b10c-e5d939825419) 2023-10-05T19:28:28.188 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.182+0000 7f18f2e04e40 4 Zone: default (9a9df965-6612-45d4-a45f-6e3c889bacc2) 2023-10-05T19:28:28.188 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.182+0000 7f18f2e04e40 10 cannot find current period zonegroup using local zonegroup configuration 2023-10-05T19:28:28.188 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.182+0000 7f18f2e04e40 20 zonegroup default 2023-10-05T19:28:28.188 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.182+0000 7f18f2e04e40 20 rados->read ofs=0 len=0 2023-10-05T19:28:28.188 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.182+0000 7f18f2e04e40 1 -- 172.21.15.167:0/1910438506 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:81 5.3 5:c52100b6:::period_config.default:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x556bff5e6eb0 con 0x7f18e400d8e0 2023-10-05T19:28:28.189 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.182+0000 7f18f113a700 1 -- 172.21.15.167:0/1910438506 <== osd.0 v2:172.21.15.118:6800/27690 22 ==== osd_op_reply(81 period_config.default [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 ==== 165+0+0 (crc 0 0 0) 0x7f18e8050040 con 0x7f18e400d8e0 2023-10-05T19:28:28.189 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.182+0000 7f18f2e04e40 20 rados_obj.operate() r=-2 bl.length=0 2023-10-05T19:28:28.189 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.182+0000 7f18f2e04e40 20 rados->read ofs=0 len=0 2023-10-05T19:28:28.189 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.182+0000 7f18f2e04e40 1 -- 172.21.15.167:0/1910438506 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:82 6.2 6:4347d321:::bucket.sync-source-hints.:head [call version.read in=11b,read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x556bff5e8830 con 0x556bff5e3290 2023-10-05T19:28:28.189 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.182+0000 7f18f0939700 1 -- 172.21.15.167:0/1910438506 <== osd.3 v2:172.21.15.167:6808/30036 20 ==== osd_op_reply(82 bucket.sync-source-hints. [call,read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 ==== 211+0+0 (crc 0 0 0) 0x7f18e0038c40 con 0x556bff5e3290 2023-10-05T19:28:28.190 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.182+0000 7f18f2e04e40 20 rados_obj.operate() r=-2 bl.length=0 2023-10-05T19:28:28.190 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.182+0000 7f18f2e04e40 20 rados->read ofs=0 len=0 2023-10-05T19:28:28.190 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.182+0000 7f18f2e04e40 1 -- 172.21.15.167:0/1910438506 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:83 6.b 6:d467b91b:::bucket.sync-target-hints.:head [call version.read in=11b,read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x556bff5e8be0 con 0x556bff5e3290 2023-10-05T19:28:28.190 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.182+0000 7f18f0939700 1 -- 172.21.15.167:0/1910438506 <== osd.3 v2:172.21.15.167:6808/30036 21 ==== osd_op_reply(83 bucket.sync-target-hints. [call,read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 ==== 211+0+0 (crc 0 0 0) 0x7f18e0038c40 con 0x556bff5e3290 2023-10-05T19:28:28.190 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.182+0000 7f18f2e04e40 20 rados_obj.operate() r=-2 bl.length=0 2023-10-05T19:28:28.190 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.182+0000 7f18f2e04e40 20 started sync module instance, tier type = 2023-10-05T19:28:28.190 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.182+0000 7f18f2e04e40 20 started zone id=9a9df965-6612-45d4-a45f-6e3c889bacc2 (name=default) with tier type = 2023-10-05T19:28:28.190 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.182+0000 7f18f2e04e40 1 -- 172.21.15.167:0/1910438506 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:84 7.1f 7:f95f44c2:::notify.0:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x556bff5e7ef0 con 0x7f18e001b740 2023-10-05T19:28:28.192 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.186+0000 7f18f0138700 1 -- 172.21.15.167:0/1910438506 <== osd.2 v2:172.21.15.167:6816/30037 13 ==== osd_op_reply(84 notify.0 [create] v30'57 uv57 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f18e4006070 con 0x7f18e001b740 2023-10-05T19:28:28.192 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.186+0000 7f18f2e04e40 1 -- 172.21.15.167:0/1910438506 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:85 7.1f 7:f95f44c2:::notify.0:head [watch watch cookie 93922334245536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x556bff5e2560 con 0x7f18e001b740 2023-10-05T19:28:28.192 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.186+0000 7f18f2e04e40 1 -- 172.21.15.167:0/1910438506 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:86 7.0 7:05bf5b68:::notify.1:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x556bff5e9970 con 0x556bff5ae6e0 2023-10-05T19:28:28.194 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.190+0000 7f18f0138700 1 -- 172.21.15.167:0/1910438506 <== osd.2 v2:172.21.15.167:6816/30037 14 ==== osd_op_reply(85 notify.0 [watch watch cookie 93922334245536] v30'58 uv57 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f18e4006070 con 0x7f18e001b740 2023-10-05T19:28:28.195 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.190+0000 7f18f113a700 1 -- 172.21.15.167:0/1910438506 <== osd.1 v2:172.21.15.167:6800/30035 29 ==== osd_op_reply(86 notify.1 [create] v30'57 uv57 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f18e8050040 con 0x556bff5ae6e0 2023-10-05T19:28:28.195 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.190+0000 7f18f2e04e40 1 -- 172.21.15.167:0/1910438506 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:87 7.0 7:05bf5b68:::notify.1:head [watch watch cookie 93922334249136] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x556bff5e9a70 con 0x556bff5ae6e0 2023-10-05T19:28:28.195 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.190+0000 7f18f2e04e40 1 -- 172.21.15.167:0/1910438506 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:88 7.15 7:a93a5511:::notify.2:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x556bff5ea410 con 0x556bff5e3290 2023-10-05T19:28:28.198 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.194+0000 7f18f113a700 1 -- 172.21.15.167:0/1910438506 <== osd.1 v2:172.21.15.167:6800/30035 30 ==== osd_op_reply(87 notify.1 [watch watch cookie 93922334249136] v30'58 uv57 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f18e8050040 con 0x556bff5ae6e0 2023-10-05T19:28:28.199 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.194+0000 7f18f0939700 1 -- 172.21.15.167:0/1910438506 <== osd.3 v2:172.21.15.167:6808/30036 22 ==== osd_op_reply(88 notify.2 [create] v30'57 uv57 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f18e0038c40 con 0x556bff5e3290 2023-10-05T19:28:28.199 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.194+0000 7f18f2e04e40 1 -- 172.21.15.167:0/1910438506 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:89 7.15 7:a93a5511:::notify.2:head [watch watch cookie 93922334252768] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x556bff5ea810 con 0x556bff5e3290 2023-10-05T19:28:28.199 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.194+0000 7f18f2e04e40 1 -- 172.21.15.167:0/1910438506 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:90 7.e 7:7759931f:::notify.3:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x556bff5eb0b0 con 0x7f18e001b740 2023-10-05T19:28:28.201 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.194+0000 7f18f0138700 1 -- 172.21.15.167:0/1910438506 <== osd.2 v2:172.21.15.167:6816/30037 15 ==== osd_op_reply(90 notify.3 [create] v30'57 uv57 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f18e4006070 con 0x7f18e001b740 2023-10-05T19:28:28.201 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.194+0000 7f18f2e04e40 1 -- 172.21.15.167:0/1910438506 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:91 7.e 7:7759931f:::notify.3:head [watch watch cookie 93922334258352] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x556bff5ebdf0 con 0x7f18e001b740 2023-10-05T19:28:28.201 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.194+0000 7f18f2e04e40 1 -- 172.21.15.167:0/1910438506 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:92 7.d 7:b4812045:::notify.4:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x556bff5ec800 con 0x556bff5ae6e0 2023-10-05T19:28:28.202 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.194+0000 7f18f0939700 1 -- 172.21.15.167:0/1910438506 <== osd.3 v2:172.21.15.167:6808/30036 23 ==== osd_op_reply(89 notify.2 [watch watch cookie 93922334252768] v30'58 uv57 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f18e0038c40 con 0x556bff5e3290 2023-10-05T19:28:28.204 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.198+0000 7f18f113a700 1 -- 172.21.15.167:0/1910438506 <== osd.1 v2:172.21.15.167:6800/30035 31 ==== osd_op_reply(92 notify.4 [create] v30'57 uv57 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f18e8050040 con 0x556bff5ae6e0 2023-10-05T19:28:28.204 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.198+0000 7f18f2e04e40 1 -- 172.21.15.167:0/1910438506 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:93 7.d 7:b4812045:::notify.4:head [watch watch cookie 93922334262032] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x556bff5ecc50 con 0x556bff5ae6e0 2023-10-05T19:28:28.204 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.198+0000 7f18f2e04e40 1 -- 172.21.15.167:0/1910438506 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:94 7.3 7:c609908c:::notify.5:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x556bff5ed4f0 con 0x7f18e400d8e0 2023-10-05T19:28:28.204 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.198+0000 7f18f0138700 1 -- 172.21.15.167:0/1910438506 <== osd.2 v2:172.21.15.167:6816/30037 16 ==== osd_op_reply(91 notify.3 [watch watch cookie 93922334258352] v30'58 uv57 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f18e4006070 con 0x7f18e001b740 2023-10-05T19:28:28.206 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.202+0000 7f18f113a700 1 -- 172.21.15.167:0/1910438506 <== osd.0 v2:172.21.15.118:6800/27690 23 ==== osd_op_reply(94 notify.5 [create] v30'57 uv57 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f18e8050040 con 0x7f18e400d8e0 2023-10-05T19:28:28.207 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.202+0000 7f18f2e04e40 1 -- 172.21.15.167:0/1910438506 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:95 7.3 7:c609908c:::notify.5:head [watch watch cookie 93922334264400] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x556bff5ed590 con 0x7f18e400d8e0 2023-10-05T19:28:28.207 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.202+0000 7f18f2e04e40 1 -- 172.21.15.167:0/1910438506 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:96 7.14 7:2b04a3e9:::notify.6:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x556bff5ede30 con 0x7f18e400d8e0 2023-10-05T19:28:28.207 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.202+0000 7f18f113a700 1 -- 172.21.15.167:0/1910438506 <== osd.1 v2:172.21.15.167:6800/30035 32 ==== osd_op_reply(93 notify.4 [watch watch cookie 93922334262032] v30'58 uv57 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f18e8050040 con 0x556bff5ae6e0 2023-10-05T19:28:28.210 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.202+0000 7f18f113a700 1 -- 172.21.15.167:0/1910438506 <== osd.0 v2:172.21.15.118:6800/27690 24 ==== osd_op_reply(96 notify.6 [create] v30'57 uv57 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f18e8050040 con 0x7f18e400d8e0 2023-10-05T19:28:28.210 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.202+0000 7f18f2e04e40 1 -- 172.21.15.167:0/1910438506 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:97 7.14 7:2b04a3e9:::notify.6:head [watch watch cookie 93922334266768] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x556bff5eded0 con 0x7f18e400d8e0 2023-10-05T19:28:28.210 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.206+0000 7f18f2e04e40 1 -- 172.21.15.167:0/1910438506 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:98 7.9 7:93e5b521:::notify.7:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x556bff5ee7b0 con 0x7f18e400d8e0 2023-10-05T19:28:28.210 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.206+0000 7f18f113a700 1 -- 172.21.15.167:0/1910438506 <== osd.0 v2:172.21.15.118:6800/27690 25 ==== osd_op_reply(95 notify.5 [watch watch cookie 93922334264400] v30'58 uv57 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f18e8050040 con 0x7f18e400d8e0 2023-10-05T19:28:28.212 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.206+0000 7f18f113a700 1 -- 172.21.15.167:0/1910438506 <== osd.0 v2:172.21.15.118:6800/27690 26 ==== osd_op_reply(98 notify.7 [create] v30'57 uv57 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f18e8050040 con 0x7f18e400d8e0 2023-10-05T19:28:28.212 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.206+0000 7f18f2e04e40 1 -- 172.21.15.167:0/1910438506 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:99 7.9 7:93e5b521:::notify.7:head [watch watch cookie 93922334265504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x556bff5ee3c0 con 0x7f18e400d8e0 2023-10-05T19:28:28.212 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.206+0000 7f18f113a700 1 -- 172.21.15.167:0/1910438506 <== osd.0 v2:172.21.15.118:6800/27690 27 ==== osd_op_reply(97 notify.6 [watch watch cookie 93922334266768] v30'58 uv57 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f18e8050040 con 0x7f18e400d8e0 2023-10-05T19:28:28.212 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.206+0000 7f18f2e04e40 20 add_watcher() i=0 2023-10-05T19:28:28.212 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.206+0000 7f18f2e04e40 20 add_watcher() i=1 2023-10-05T19:28:28.213 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.206+0000 7f18f2e04e40 20 add_watcher() i=2 2023-10-05T19:28:28.213 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.206+0000 7f18f2e04e40 20 add_watcher() i=3 2023-10-05T19:28:28.213 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.206+0000 7f18f2e04e40 20 add_watcher() i=4 2023-10-05T19:28:28.213 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.206+0000 7f18f2e04e40 20 add_watcher() i=5 2023-10-05T19:28:28.213 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.206+0000 7f18f2e04e40 20 add_watcher() i=6 2023-10-05T19:28:28.214 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.210+0000 7f18f113a700 1 -- 172.21.15.167:0/1910438506 <== osd.0 v2:172.21.15.118:6800/27690 28 ==== osd_op_reply(99 notify.7 [watch watch cookie 93922334265504] v30'58 uv57 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f18e8050040 con 0x7f18e400d8e0 2023-10-05T19:28:28.214 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.210+0000 7f18f2e04e40 20 add_watcher() i=7 2023-10-05T19:28:28.214 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.210+0000 7f18f2e04e40 2 all 8 watchers are set, enabling cache 2023-10-05T19:28:28.214 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.210+0000 7f18f2e04e40 1 -- 172.21.15.167:0/1910438506 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:100 6.4 6:22d26bf9:::data_loggenerations_metadata:head [call version.check_conds in=74b,call version.read in=11b,read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x556bff5ee0a0 con 0x556bff5ae6e0 2023-10-05T19:28:28.215 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.210+0000 7f18f113a700 1 -- 172.21.15.167:0/1910438506 <== osd.1 v2:172.21.15.167:6800/30035 33 ==== osd_op_reply(100 data_loggenerations_metadata [call,call out=48b,read 0~28 out=28b] v0'0 uv1 ondisk = 0) v8 ==== 256+0+76 (crc 0 0 0) 0x7f18e8050040 con 0x556bff5ae6e0 2023-10-05T19:28:28.215 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.210+0000 7f18f2e04e40 1 -- 172.21.15.167:0/1910438506 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:101 6.4 6:22d26bf9:::data_loggenerations_metadata:head [watch watch cookie 93922334268336] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x556bff5eebb0 con 0x556bff5ae6e0 2023-10-05T19:28:28.217 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.210+0000 7f18f113a700 1 -- 172.21.15.167:0/1910438506 <== osd.1 v2:172.21.15.167:6800/30035 34 ==== osd_op_reply(101 data_loggenerations_metadata [watch watch cookie 93922334268336] v30'72 uv1 ondisk = 0) v8 ==== 172+0+0 (crc 0 0 0) 0x7f18e8050040 con 0x556bff5ae6e0 2023-10-05T19:28:28.217 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.210+0000 7f18c27e4700 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2023-10-05T19:28:28.217 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.210+0000 7f18f2e04e40 20 check_secure_mon_conn(): auth registy supported: methods=[2] modes=[2,1] 2023-10-05T19:28:28.217 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.210+0000 7f18f2e04e40 20 check_secure_mon_conn(): mode 1 is insecure 2023-10-05T19:28:28.218 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.210+0000 7f18f2e04e40 1 Processor -- start 2023-10-05T19:28:28.218 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.214+0000 7f18f2e04e40 1 -- start start 2023-10-05T19:28:28.218 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.214+0000 7f18f2e04e40 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x556bff5fbcb0 0x556bff5fc080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-05T19:28:28.218 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.214+0000 7f18f2e04e40 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x556bff5fc5c0 con 0x556bff5fbcb0 2023-10-05T19:28:28.219 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.214+0000 7f18f0138700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x556bff5fbcb0 0x556bff5fc080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-05T19:28:28.219 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.214+0000 7f18f0138700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x556bff5fbcb0 0x556bff5fc080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.118:3300/0 says I am v2:172.21.15.167:57662/0 (socket says 172.21.15.167:57662) 2023-10-05T19:28:28.219 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.214+0000 7f18f0138700 1 -- 172.21.15.167:0/3324883167 learned_addr learned my addr 172.21.15.167:0/3324883167 (peer_addr_for_me v2:172.21.15.167:0/0) 2023-10-05T19:28:28.220 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.214+0000 7f18f0138700 1 -- 172.21.15.167:0/3324883167 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x556bff5fc700 con 0x556bff5fbcb0 2023-10-05T19:28:28.220 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.214+0000 7f18f0138700 1 --2- 172.21.15.167:0/3324883167 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x556bff5fbcb0 0x556bff5fc080 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x556bff5e09f0 tx=0x7f18e4006000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d286e3d46197015d server_cookie=0 in_seq=0 out_seq=0 2023-10-05T19:28:28.220 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.214+0000 7f18c1fe3700 1 -- 172.21.15.167:0/3324883167 <== mon.0 v2:172.21.15.118:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f18e4023040 con 0x556bff5fbcb0 2023-10-05T19:28:28.220 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.214+0000 7f18c1fe3700 1 -- 172.21.15.167:0/3324883167 <== mon.0 v2:172.21.15.118:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f18e4024040 con 0x556bff5fbcb0 2023-10-05T19:28:28.220 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.214+0000 7f18c1fe3700 1 -- 172.21.15.167:0/3324883167 <== mon.0 v2:172.21.15.118:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f18e4013040 con 0x556bff5fbcb0 2023-10-05T19:28:28.221 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.214+0000 7f18f2e04e40 1 -- 172.21.15.167:0/3324883167 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x556bff5fbcb0 msgr2=0x556bff5fc080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-05T19:28:28.222 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.214+0000 7f18f2e04e40 1 --2- 172.21.15.167:0/3324883167 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x556bff5fbcb0 0x556bff5fc080 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x556bff5e09f0 tx=0x7f18e4006000 comp rx=0 tx=0).stop 2023-10-05T19:28:28.222 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.214+0000 7f18f2e04e40 1 -- 172.21.15.167:0/3324883167 shutdown_connections 2023-10-05T19:28:28.222 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.214+0000 7f18f2e04e40 1 --2- 172.21.15.167:0/3324883167 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x556bff5fbcb0 0x556bff5fc080 unknown :-1 s=CLOSED pgs=252 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-05T19:28:28.222 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.214+0000 7f18f2e04e40 1 -- 172.21.15.167:0/3324883167 >> 172.21.15.167:0/3324883167 conn(0x556bff5f7070 msgr2=0x556bff5f51c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-05T19:28:28.222 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.214+0000 7f18f2e04e40 1 -- 172.21.15.167:0/3324883167 shutdown_connections 2023-10-05T19:28:28.222 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.214+0000 7f18f2e04e40 1 -- 172.21.15.167:0/3324883167 wait complete. 2023-10-05T19:28:28.223 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.218+0000 7f18f2e04e40 1 Processor -- start 2023-10-05T19:28:28.223 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.218+0000 7f18f2e04e40 1 -- start start 2023-10-05T19:28:28.223 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.218+0000 7f18f2e04e40 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x556bff5fbcb0 0x556bff601810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-05T19:28:28.223 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.218+0000 7f18f2e04e40 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x556bff602810 con 0x556bff5fbcb0 2023-10-05T19:28:28.223 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.218+0000 7f18f0138700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x556bff5fbcb0 0x556bff601810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-05T19:28:28.223 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.218+0000 7f18f0138700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x556bff5fbcb0 0x556bff601810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.118:3300/0 says I am v2:172.21.15.167:57676/0 (socket says 172.21.15.167:57676) 2023-10-05T19:28:28.224 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.218+0000 7f18f0138700 1 -- 172.21.15.167:0/2152878927 learned_addr learned my addr 172.21.15.167:0/2152878927 (peer_addr_for_me v2:172.21.15.167:0/0) 2023-10-05T19:28:28.224 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.218+0000 7f18f0138700 1 -- 172.21.15.167:0/2152878927 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f18e400ece0 con 0x556bff5fbcb0 2023-10-05T19:28:28.225 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.218+0000 7f18f0138700 1 --2- 172.21.15.167:0/2152878927 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x556bff5fbcb0 0x556bff601810 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7f18e0031000 tx=0x7f18e4030000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-05T19:28:28.225 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.218+0000 7f1833fff700 1 -- 172.21.15.167:0/2152878927 <== mon.0 v2:172.21.15.118:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f18e4023040 con 0x556bff5fbcb0 2023-10-05T19:28:28.225 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.218+0000 7f1833fff700 1 -- 172.21.15.167:0/2152878927 <== mon.0 v2:172.21.15.118:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f18e401e570 con 0x556bff5fbcb0 2023-10-05T19:28:28.225 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.218+0000 7f18f2e04e40 1 -- 172.21.15.167:0/2152878927 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x556bff601e10 con 0x556bff5fbcb0 2023-10-05T19:28:28.225 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.218+0000 7f1833fff700 1 -- 172.21.15.167:0/2152878927 <== mon.0 v2:172.21.15.118:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f18e401e700 con 0x556bff5fbcb0 2023-10-05T19:28:28.225 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.218+0000 7f18f2e04e40 1 -- 172.21.15.167:0/2152878927 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x556bff602250 con 0x556bff5fbcb0 2023-10-05T19:28:28.226 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.222+0000 7f1833fff700 1 -- 172.21.15.167:0/2152878927 <== mon.0 v2:172.21.15.118:3300/0 4 ==== mgrmap(e 5) v1 ==== 40329+0+0 (secure 0 0 0) 0x7f18e401e860 con 0x556bff5fbcb0 2023-10-05T19:28:28.226 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.222+0000 7f1833fff700 1 --2- 172.21.15.167:0/2152878927 >> [v2:172.21.15.167:6824/29889,v1:172.21.15.167:6825/29889] conn(0x7f1838030c50 0x7f1838033100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-05T19:28:28.227 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.222+0000 7f1833fff700 1 -- 172.21.15.167:0/2152878927 <== mon.0 v2:172.21.15.118:3300/0 5 ==== osd_map(30..30 src has 1..30) v4 ==== 6057+0+0 (secure 0 0 0) 0x7f18e404fc60 con 0x556bff5fbcb0 2023-10-05T19:28:28.227 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.222+0000 7f18f113a700 1 --2- 172.21.15.167:0/2152878927 >> [v2:172.21.15.167:6824/29889,v1:172.21.15.167:6825/29889] conn(0x7f1838030c50 0x7f1838033100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-05T19:28:28.227 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.222+0000 7f18f2e04e40 5 note: GC not initialized 2023-10-05T19:28:28.227 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.222+0000 7f18f113a700 1 --2- 172.21.15.167:0/2152878927 >> [v2:172.21.15.167:6824/29889,v1:172.21.15.167:6825/29889] conn(0x7f1838030c50 0x7f1838033100 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x556bff5ee9b0 tx=0x7f18e8000f90 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-05T19:28:28.229 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.222+0000 7f18317fa700 20 reqs_thread_entry: start 2023-10-05T19:28:28.229 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.222+0000 7f182affd700 20 reqs_thread_entry: start 2023-10-05T19:28:28.230 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.222+0000 7f18f2e04e40 20 init_complete bucket index max shards: 11 2023-10-05T19:28:28.230 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.226+0000 7f18f2e04e40 10 rgw notify: Started notification manager with: 1 workers 2023-10-05T19:28:28.230 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.226+0000 7f18297fa700 20 rgw notify: INFO: next queues processing will happen at: Thu Oct 5 19:28:58 2023 2023-10-05T19:28:28.230 INFO:teuthology.orchestra.run.smithi167.stderr: 2023-10-05T19:28:28.231 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.226+0000 7f1828ff9700 20 reqs_thread_entry: start 2023-10-05T19:28:28.231 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.226+0000 7f18f2e04e40 20 get_system_obj_state: rctx=0x7ffd290876a8 obj=default.rgw.meta:users.uid:foo state=0x556bff4870b0 s->prefetch_data=0 2023-10-05T19:28:28.231 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.226+0000 7f18f2e04e40 10 cache get: name=default.rgw.meta+users.uid+foo : miss 2023-10-05T19:28:28.231 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.226+0000 7f18f2e04e40 1 -- 172.21.15.167:0/1910438506 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:102 8.18 8:1e6ca5ed:users.uid::foo:head [call version.read in=11b,getxattrs,stat] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x556bff66a040 con 0x556bff5ae6e0 2023-10-05T19:28:28.232 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.226+0000 7f18f113a700 1 -- 172.21.15.167:0/1910438506 <== osd.1 v2:172.21.15.167:6800/30035 35 ==== osd_op_reply(102 foo [call out=48b,getxattrs out=75b,stat out=16b] v0'0 uv6 ondisk = 0) v8 ==== 231+0+139 (crc 0 0 0) 0x7f18e8050040 con 0x556bff5ae6e0 2023-10-05T19:28:28.232 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.226+0000 7f18f2e04e40 10 cache put: name=default.rgw.meta+users.uid+foo info.flags=0x16 2023-10-05T19:28:28.232 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.226+0000 7f18f2e04e40 10 adding default.rgw.meta+users.uid+foo to cache LRU end 2023-10-05T19:28:28.232 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.226+0000 7f18f2e04e40 10 updating xattr: name=ceph.objclass.version bl.length()=42 2023-10-05T19:28:28.233 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.226+0000 7f18f2e04e40 20 get_system_obj_state: s->obj_tag was set empty 2023-10-05T19:28:28.233 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.226+0000 7f18f2e04e40 20 Read xattr: user.rgw.idtag 2023-10-05T19:28:28.233 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.226+0000 7f18f2e04e40 10 cache get: name=default.rgw.meta+users.uid+foo : type miss (requested=0x13, cached=0x16) 2023-10-05T19:28:28.233 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.226+0000 7f18f2e04e40 20 rados->read ofs=0 len=0 2023-10-05T19:28:28.233 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.226+0000 7f18f2e04e40 1 -- 172.21.15.167:0/1910438506 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:103 8.18 8:1e6ca5ed:users.uid::foo:head [call version.check_conds in=122b,call version.read in=11b,read 0~0,getxattrs] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x556bff66cad0 con 0x556bff5ae6e0 2023-10-05T19:28:28.233 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.226+0000 7f18f113a700 1 -- 172.21.15.167:0/1910438506 <== osd.1 v2:172.21.15.167:6800/30035 36 ==== osd_op_reply(103 foo [call,call out=48b,read 0~513 out=513b,getxattrs out=75b] v0'0 uv6 ondisk = 0) v8 ==== 273+0+636 (crc 0 0 0) 0x7f18e8050040 con 0x556bff5ae6e0 2023-10-05T19:28:28.233 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.226+0000 7f18f2e04e40 20 rados_obj.operate() r=0 bl.length=513 2023-10-05T19:28:28.233 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.226+0000 7f18f2e04e40 10 cache put: name=default.rgw.meta+users.uid+foo info.flags=0x13 2023-10-05T19:28:28.233 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.226+0000 7f18f2e04e40 10 moving default.rgw.meta+users.uid+foo to cache LRU end 2023-10-05T19:28:28.233 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.226+0000 7f18f2e04e40 10 updating xattr: name=ceph.objclass.version bl.length()=42 2023-10-05T19:28:28.234 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.226+0000 7f18f2e04e40 1 -- 172.21.15.167:0/1910438506 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:104 8.18 8:1e6ca5ed:users.uid::foo:head [delete,create,call version.set in=58b,writefull 0~513 in=513b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x556bff6692b0 con 0x556bff5ae6e0 2023-10-05T19:28:28.235 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.230+0000 7f18f113a700 1 -- 172.21.15.167:0/1910438506 <== osd.1 v2:172.21.15.167:6800/30035 37 ==== osd_op_reply(104 foo [delete,create,call,writefull 0~513] v30'7 uv7 ondisk = 0) v8 ==== 273+0+0 (crc 0 0 0) 0x7f18e8050040 con 0x556bff5ae6e0 2023-10-05T19:28:28.236 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.230+0000 7f18f2e04e40 10 cache put: name=default.rgw.meta+users.uid+foo info.flags=0x17 2023-10-05T19:28:28.236 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.230+0000 7f18f2e04e40 10 moving default.rgw.meta+users.uid+foo to cache LRU end 2023-10-05T19:28:28.236 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.230+0000 7f18f2e04e40 10 distributing notification oid=default.rgw.control:notify.4 cni=[op: 0, obj: default.rgw.meta:users.uid:foo, ofs0, ns] 2023-10-05T19:28:28.236 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.230+0000 7f18f2e04e40 1 -- 172.21.15.167:0/1910438506 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:105 7.d 7:b4812045:::notify.4:head [notify cookie 93922334772432 in=701b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x556bff66dae0 con 0x556bff5ae6e0 2023-10-05T19:28:28.237 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.230+0000 7f18f113a700 1 -- 172.21.15.167:0/1910438506 <== osd.1 v2:172.21.15.167:6800/30035 38 ==== watch-notify(notify (1) cookie 93922334262032 notify 128849018886 ret 0) v3 ==== 731+0+0 (crc 0 0 0) 0x7f18e803d040 con 0x556bff5ae6e0 2023-10-05T19:28:28.237 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.230+0000 7f18ee935700 10 rgw watcher librados: RGWWatcher::handle_notify() notify_id 128849018886 cookie 93922334262032 notifier 4341 bl.length()=689 2023-10-05T19:28:28.237 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.230+0000 7f18ee935700 10 rgw watcher librados: cache put: name=default.rgw.meta+users.uid+foo info.flags=0x17 2023-10-05T19:28:28.237 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.230+0000 7f18ee935700 10 rgw watcher librados: moving default.rgw.meta+users.uid+foo to cache LRU end 2023-10-05T19:28:28.237 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.230+0000 7f18ee935700 1 -- 172.21.15.167:0/1910438506 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:106 7.d 7:b4812045:::notify.4:head [notify-ack in=20b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x7f18d0003d90 con 0x556bff5ae6e0 2023-10-05T19:28:28.237 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.230+0000 7f18f113a700 1 -- 172.21.15.167:0/1910438506 <== osd.1 v2:172.21.15.167:6800/30035 39 ==== osd_op_reply(105 notify.4 [notify cookie 93922334772432 out=8b] v0'0 uv57 ondisk = 0) v8 ==== 152+0+8 (crc 0 0 0) 0x7f18e8050040 con 0x556bff5ae6e0 2023-10-05T19:28:28.237 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.230+0000 7f18f113a700 1 -- 172.21.15.167:0/1910438506 <== osd.1 v2:172.21.15.167:6800/30035 40 ==== osd_op_reply(106 notify.4 [notify-ack] v0'0 uv57 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f18e8050040 con 0x556bff5ae6e0 2023-10-05T19:28:28.238 INFO:teuthology.orchestra.run.smithi167.stdout:{ 2023-10-05T19:28:28.238 INFO:teuthology.orchestra.run.smithi167.stdout: "user_id": "foo", 2023-10-05T19:28:28.238 INFO:teuthology.orchestra.run.smithi167.stdout: "display_name": "Foo", 2023-10-05T19:28:28.238 INFO:teuthology.orchestra.run.smithi167.stdout: "email": "foo@foo.com", 2023-10-05T19:28:28.238 INFO:teuthology.orchestra.run.smithi167.stdout: "suspended": 0, 2023-10-05T19:28:28.238 INFO:teuthology.orchestra.run.smithi167.stdout: "max_buckets": 4, 2023-10-05T19:28:28.238 INFO:teuthology.orchestra.run.smithi167.stdout: "subusers": [ 2023-10-05T19:28:28.238 INFO:teuthology.orchestra.run.smithi167.stdout: { 2023-10-05T19:28:28.239 INFO:teuthology.orchestra.run.smithi167.stdout: "id": "foo:foo1", 2023-10-05T19:28:28.239 INFO:teuthology.orchestra.run.smithi167.stdout: "permissions": "full-control" 2023-10-05T19:28:28.239 INFO:teuthology.orchestra.run.smithi167.stdout: } 2023-10-05T19:28:28.239 INFO:teuthology.orchestra.run.smithi167.stdout: ], 2023-10-05T19:28:28.239 INFO:teuthology.orchestra.run.smithi167.stdout: "keys": [ 2023-10-05T19:28:28.239 INFO:teuthology.orchestra.run.smithi167.stdout: { 2023-10-05T19:28:28.239 INFO:teuthology.orchestra.run.smithi167.stdout: "user": "foo", 2023-10-05T19:28:28.239 INFO:teuthology.orchestra.run.smithi167.stdout: "access_key": "9te6NH5mcdcq0Tc5i8i1", 2023-10-05T19:28:28.239 INFO:teuthology.orchestra.run.smithi167.stdout: "secret_key": "Ny4IOauQoL18Gp2zM7lC1vLmoawgqcYP/YGcWfXu" 2023-10-05T19:28:28.239 INFO:teuthology.orchestra.run.smithi167.stdout: } 2023-10-05T19:28:28.239 INFO:teuthology.orchestra.run.smithi167.stdout: ], 2023-10-05T19:28:28.240 INFO:teuthology.orchestra.run.smithi167.stdout: "swift_keys": [ 2023-10-05T19:28:28.240 INFO:teuthology.orchestra.run.smithi167.stdout: { 2023-10-05T19:28:28.240 INFO:teuthology.orchestra.run.smithi167.stdout: "user": "foo:foo1", 2023-10-05T19:28:28.240 INFO:teuthology.orchestra.run.smithi167.stdout: "secret_key": "gpS2G9RREMrnbqlp29PP2D36kgPR1tm72n5fPYfL" 2023-10-05T19:28:28.240 INFO:teuthology.orchestra.run.smithi167.stdout: } 2023-10-05T19:28:28.240 INFO:teuthology.orchestra.run.smithi167.stdout: ], 2023-10-05T19:28:28.240 INFO:teuthology.orchestra.run.smithi167.stdout: "caps": [], 2023-10-05T19:28:28.240 INFO:teuthology.orchestra.run.smithi167.stdout: "op_mask": "read, write, delete", 2023-10-05T19:28:28.240 INFO:teuthology.orchestra.run.smithi167.stdout: "default_placement": "", 2023-10-05T19:28:28.241 INFO:teuthology.orchestra.run.smithi167.stdout: "default_storage_class": "", 2023-10-05T19:28:28.241 INFO:teuthology.orchestra.run.smithi167.stdout: "placement_tags": [], 2023-10-05T19:28:28.241 INFO:teuthology.orchestra.run.smithi167.stdout: "bucket_quota": { 2023-10-05T19:28:28.241 INFO:teuthology.orchestra.run.smithi167.stdout: "enabled": false, 2023-10-05T19:28:28.241 INFO:teuthology.orchestra.run.smithi167.stdout: "check_on_raw": false, 2023-10-05T19:28:28.241 INFO:teuthology.orchestra.run.smithi167.stdout: "max_size": -1, 2023-10-05T19:28:28.241 INFO:teuthology.orchestra.run.smithi167.stdout: "max_size_kb": 0, 2023-10-05T19:28:28.241 INFO:teuthology.orchestra.run.smithi167.stdout: "max_objects": -1 2023-10-05T19:28:28.241 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2023-10-05T19:28:28.241 INFO:teuthology.orchestra.run.smithi167.stdout: "user_quota": { 2023-10-05T19:28:28.241 INFO:teuthology.orchestra.run.smithi167.stdout: "enabled": false, 2023-10-05T19:28:28.242 INFO:teuthology.orchestra.run.smithi167.stdout: "check_on_raw": false, 2023-10-05T19:28:28.242 INFO:teuthology.orchestra.run.smithi167.stdout: "max_size": -1, 2023-10-05T19:28:28.242 INFO:teuthology.orchestra.run.smithi167.stdout: "max_size_kb": 0, 2023-10-05T19:28:28.242 INFO:teuthology.orchestra.run.smithi167.stdout: "max_objects": -1 2023-10-05T19:28:28.242 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2023-10-05T19:28:28.242 INFO:teuthology.orchestra.run.smithi167.stdout: "temp_url_keys": [], 2023-10-05T19:28:28.242 INFO:teuthology.orchestra.run.smithi167.stdout: "type": "rgw", 2023-10-05T19:28:28.242 INFO:teuthology.orchestra.run.smithi167.stdout: "mfa_ids": [] 2023-10-05T19:28:28.242 INFO:teuthology.orchestra.run.smithi167.stdout:} 2023-10-05T19:28:28.242 INFO:teuthology.orchestra.run.smithi167.stdout: 2023-10-05T19:28:28.243 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.230+0000 7f18f113a700 1 -- 172.21.15.167:0/1910438506 <== osd.1 v2:172.21.15.167:6800/30035 41 ==== watch-notify(notify_complete (2) cookie 93922334772432 notify 128849018886 ret 0) v3 ==== 42+0+48 (crc 0 0 0) 0x7f18e8002660 con 0x556bff5ae6e0 2023-10-05T19:28:28.243 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.230+0000 7f18f2e04e40 1 -- 172.21.15.167:0/1910438506 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:107 7.1f 7:f95f44c2:::notify.0:head [watch unwatch cookie 93922334245536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x556bff5c34d0 con 0x7f18e001b740 2023-10-05T19:28:28.244 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.234+0000 7f18f0138700 1 -- 172.21.15.167:0/1910438506 <== osd.2 v2:172.21.15.167:6816/30037 17 ==== osd_op_reply(107 notify.0 [watch unwatch cookie 93922334245536] v30'59 uv57 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f18e4006070 con 0x7f18e001b740 2023-10-05T19:28:28.244 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.234+0000 7f18f2e04e40 20 remove_watcher() i=0 2023-10-05T19:28:28.244 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.234+0000 7f18f2e04e40 2 removed watcher, disabling cache 2023-10-05T19:28:28.244 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.234+0000 7f18f2e04e40 1 -- 172.21.15.167:0/1910438506 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:108 7.0 7:05bf5b68:::notify.1:head [watch unwatch cookie 93922334249136] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x556bff5e82a0 con 0x556bff5ae6e0 2023-10-05T19:28:28.244 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.238+0000 7f18f113a700 1 -- 172.21.15.167:0/1910438506 <== osd.1 v2:172.21.15.167:6800/30035 42 ==== osd_op_reply(108 notify.1 [watch unwatch cookie 93922334249136] v30'59 uv57 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f18e8050040 con 0x556bff5ae6e0 2023-10-05T19:28:28.244 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.238+0000 7f18f2e04e40 20 remove_watcher() i=1 2023-10-05T19:28:28.244 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.238+0000 7f18f2e04e40 1 -- 172.21.15.167:0/1910438506 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:109 7.15 7:a93a5511:::notify.2:head [watch unwatch cookie 93922334252768] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x556bff5e82a0 con 0x556bff5e3290 2023-10-05T19:28:28.245 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.238+0000 7f18f0939700 1 -- 172.21.15.167:0/1910438506 <== osd.3 v2:172.21.15.167:6808/30036 24 ==== osd_op_reply(109 notify.2 [watch unwatch cookie 93922334252768] v30'59 uv57 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f18e0038c40 con 0x556bff5e3290 2023-10-05T19:28:28.246 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.238+0000 7f18f2e04e40 20 remove_watcher() i=2 2023-10-05T19:28:28.246 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.238+0000 7f18f2e04e40 1 -- 172.21.15.167:0/1910438506 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:110 7.e 7:7759931f:::notify.3:head [watch unwatch cookie 93922334258352] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x556bff5e9ee0 con 0x7f18e001b740 2023-10-05T19:28:28.248 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.242+0000 7f18f0138700 1 -- 172.21.15.167:0/1910438506 <== osd.2 v2:172.21.15.167:6816/30037 18 ==== osd_op_reply(110 notify.3 [watch unwatch cookie 93922334258352] v30'59 uv57 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f18e4006070 con 0x7f18e001b740 2023-10-05T19:28:28.248 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.242+0000 7f18f2e04e40 20 remove_watcher() i=3 2023-10-05T19:28:28.248 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.242+0000 7f18f2e04e40 1 -- 172.21.15.167:0/1910438506 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:111 7.d 7:b4812045:::notify.4:head [watch unwatch cookie 93922334262032] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x556bff5eb4b0 con 0x556bff5ae6e0 2023-10-05T19:28:28.250 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.246+0000 7f18f113a700 1 -- 172.21.15.167:0/1910438506 <== osd.1 v2:172.21.15.167:6800/30035 43 ==== osd_op_reply(111 notify.4 [watch unwatch cookie 93922334262032] v30'59 uv57 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f18e8050040 con 0x556bff5ae6e0 2023-10-05T19:28:28.250 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.246+0000 7f18f2e04e40 20 remove_watcher() i=4 2023-10-05T19:28:28.251 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.246+0000 7f18f2e04e40 1 -- 172.21.15.167:0/1910438506 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:112 7.3 7:c609908c:::notify.5:head [watch unwatch cookie 93922334264400] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x556bff5e9ee0 con 0x7f18e400d8e0 2023-10-05T19:28:28.253 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.246+0000 7f18f113a700 1 -- 172.21.15.167:0/1910438506 <== osd.0 v2:172.21.15.118:6800/27690 29 ==== osd_op_reply(112 notify.5 [watch unwatch cookie 93922334264400] v30'59 uv57 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f18e8050040 con 0x7f18e400d8e0 2023-10-05T19:28:28.253 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.246+0000 7f18f2e04e40 20 remove_watcher() i=5 2023-10-05T19:28:28.253 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.246+0000 7f18f2e04e40 1 -- 172.21.15.167:0/1910438506 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:113 7.14 7:2b04a3e9:::notify.6:head [watch unwatch cookie 93922334266768] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x556bff5ec310 con 0x7f18e400d8e0 2023-10-05T19:28:28.256 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.250+0000 7f18f113a700 1 -- 172.21.15.167:0/1910438506 <== osd.0 v2:172.21.15.118:6800/27690 30 ==== osd_op_reply(113 notify.6 [watch unwatch cookie 93922334266768] v30'59 uv57 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f18e8050040 con 0x7f18e400d8e0 2023-10-05T19:28:28.256 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.250+0000 7f18f2e04e40 20 remove_watcher() i=6 2023-10-05T19:28:28.256 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.250+0000 7f18f2e04e40 1 -- 172.21.15.167:0/1910438506 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:114 7.9 7:93e5b521:::notify.7:head [watch unwatch cookie 93922334265504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x556bff5eb4b0 con 0x7f18e400d8e0 2023-10-05T19:28:28.258 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.254+0000 7f18f113a700 1 -- 172.21.15.167:0/1910438506 <== osd.0 v2:172.21.15.118:6800/27690 31 ==== osd_op_reply(114 notify.7 [watch unwatch cookie 93922334265504] v30'59 uv57 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f18e8050040 con 0x7f18e400d8e0 2023-10-05T19:28:28.258 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.254+0000 7f18f2e04e40 20 remove_watcher() i=7 2023-10-05T19:28:28.260 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.254+0000 7f18c27e4700 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2023-10-05T19:28:28.260 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.254+0000 7f18f2e04e40 1 -- 172.21.15.167:0/1910438506 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:115 6.4 6:22d26bf9:::data_loggenerations_metadata:head [watch unwatch cookie 93922334268336] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x556bff6692b0 con 0x556bff5ae6e0 2023-10-05T19:28:28.263 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.258+0000 7f18f113a700 1 -- 172.21.15.167:0/1910438506 <== osd.1 v2:172.21.15.167:6800/30035 44 ==== osd_op_reply(115 data_loggenerations_metadata [watch unwatch cookie 93922334268336] v30'73 uv1 ondisk = 0) v8 ==== 172+0+0 (crc 0 0 0) 0x7f18e8050040 con 0x556bff5ae6e0 2023-10-05T19:28:28.263 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.258+0000 7f18f2e04e40 1 -- 172.21.15.167:0/1910438506 >> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] conn(0x7f18e400d8e0 msgr2=0x7f18e400c360 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-05T19:28:28.264 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.258+0000 7f18f2e04e40 1 --2- 172.21.15.167:0/1910438506 >> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] conn(0x7f18e400d8e0 0x7f18e400c360 crc :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-05T19:28:28.264 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.258+0000 7f18f2e04e40 1 -- 172.21.15.167:0/1910438506 >> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] conn(0x556bff5ae6e0 msgr2=0x556bff5df430 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-05T19:28:28.264 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.258+0000 7f18f2e04e40 1 --2- 172.21.15.167:0/1910438506 >> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] conn(0x556bff5ae6e0 0x556bff5df430 crc :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-05T19:28:28.264 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.258+0000 7f18f2e04e40 1 -- 172.21.15.167:0/1910438506 >> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] conn(0x7f18e001b740 msgr2=0x7f18e0030440 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-05T19:28:28.264 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.258+0000 7f18f2e04e40 1 --2- 172.21.15.167:0/1910438506 >> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] conn(0x7f18e001b740 0x7f18e0030440 crc :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-05T19:28:28.264 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.258+0000 7f18f2e04e40 1 -- 172.21.15.167:0/1910438506 >> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] conn(0x556bff5e3290 msgr2=0x556bff5e5670 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-05T19:28:28.264 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.258+0000 7f18f2e04e40 1 --2- 172.21.15.167:0/1910438506 >> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] conn(0x556bff5e3290 0x556bff5e5670 crc :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-05T19:28:28.264 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.258+0000 7f18f2e04e40 1 -- 172.21.15.167:0/1910438506 >> [v2:172.21.15.167:6824/29889,v1:172.21.15.167:6825/29889] conn(0x7f18dc030c00 msgr2=0x7f18dc0330b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-05T19:28:28.264 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.258+0000 7f18f2e04e40 1 --2- 172.21.15.167:0/1910438506 >> [v2:172.21.15.167:6824/29889,v1:172.21.15.167:6825/29889] conn(0x7f18dc030c00 0x7f18dc0330b0 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f18e40011e0 tx=0x7f18e40013a0 comp rx=0 tx=0).stop 2023-10-05T19:28:28.264 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.258+0000 7f18f2e04e40 1 -- 172.21.15.167:0/1910438506 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x556bff5addd0 msgr2=0x556bff5d39b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-05T19:28:28.264 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.258+0000 7f18f113a700 1 -- 172.21.15.167:0/1910438506 reap_dead start 2023-10-05T19:28:28.265 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.258+0000 7f18f2e04e40 1 --2- 172.21.15.167:0/1910438506 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x556bff5addd0 0x556bff5d39b0 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7f18e0000f80 tx=0x7f18e001d000 comp rx=0 tx=0).stop 2023-10-05T19:28:28.265 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.258+0000 7f18f2e04e40 1 -- 172.21.15.167:0/1910438506 shutdown_connections 2023-10-05T19:28:28.265 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.258+0000 7f18f2e04e40 1 --2- 172.21.15.167:0/1910438506 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x556bff5addd0 0x556bff5d39b0 unknown :-1 s=CLOSED pgs=251 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-05T19:28:28.265 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.258+0000 7f18f2e04e40 1 -- 172.21.15.167:0/1910438506 >> 172.21.15.167:0/1910438506 conn(0x556bff5a97f0 msgr2=0x556bff5c72f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-05T19:28:28.265 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.258+0000 7f18f2e04e40 1 -- 172.21.15.167:0/1910438506 shutdown_connections 2023-10-05T19:28:28.265 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.258+0000 7f18f2e04e40 1 -- 172.21.15.167:0/1910438506 wait complete. 2023-10-05T19:28:28.266 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.258+0000 7f18f2e04e40 1 -- 172.21.15.167:0/2152878927 >> [v2:172.21.15.167:6824/29889,v1:172.21.15.167:6825/29889] conn(0x7f1838030c50 msgr2=0x7f1838033100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-05T19:28:28.266 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.258+0000 7f18f2e04e40 1 --2- 172.21.15.167:0/2152878927 >> [v2:172.21.15.167:6824/29889,v1:172.21.15.167:6825/29889] conn(0x7f1838030c50 0x7f1838033100 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x556bff5ee9b0 tx=0x7f18e8000f90 comp rx=0 tx=0).stop 2023-10-05T19:28:28.266 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.258+0000 7f18f2e04e40 1 -- 172.21.15.167:0/2152878927 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x556bff5fbcb0 msgr2=0x556bff601810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-05T19:28:28.266 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.258+0000 7f18f2e04e40 1 --2- 172.21.15.167:0/2152878927 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x556bff5fbcb0 0x556bff601810 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7f18e0031000 tx=0x7f18e4030000 comp rx=0 tx=0).stop 2023-10-05T19:28:28.266 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.258+0000 7f18f2e04e40 1 -- 172.21.15.167:0/2152878927 shutdown_connections 2023-10-05T19:28:28.266 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.258+0000 7f18f2e04e40 1 --2- 172.21.15.167:0/2152878927 >> [v2:172.21.15.167:6824/29889,v1:172.21.15.167:6825/29889] conn(0x7f1838030c50 0x7f1838033100 unknown :-1 s=CLOSED pgs=91 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-05T19:28:28.267 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.258+0000 7f18f2e04e40 1 --2- 172.21.15.167:0/2152878927 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x556bff5fbcb0 0x556bff601810 unknown :-1 s=CLOSED pgs=253 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-05T19:28:28.267 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.258+0000 7f18f2e04e40 1 -- 172.21.15.167:0/2152878927 >> 172.21.15.167:0/2152878927 conn(0x556bff5f7070 msgr2=0x556bff5f5110 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-05T19:28:28.267 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.258+0000 7f18f2e04e40 1 -- 172.21.15.167:0/2152878927 shutdown_connections 2023-10-05T19:28:28.267 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.258+0000 7f18f2e04e40 1 -- 172.21.15.167:0/2152878927 wait complete. 2023-10-05T19:28:28.269 DEBUG:tasks.util.rgw: json result: {'user_id': 'foo', 'display_name': 'Foo', 'email': 'foo@foo.com', 'suspended': 0, 'max_buckets': 4, 'subusers': [{'id': 'foo:foo1', 'permissions': 'full-control'}], 'keys': [{'user': 'foo', 'access_key': '9te6NH5mcdcq0Tc5i8i1', 'secret_key': 'Ny4IOauQoL18Gp2zM7lC1vLmoawgqcYP/YGcWfXu'}], 'swift_keys': [{'user': 'foo:foo1', 'secret_key': 'gpS2G9RREMrnbqlp29PP2D36kgPR1tm72n5fPYfL'}], 'caps': [], 'op_mask': 'read, write, delete', 'default_placement': '', 'default_storage_class': '', 'placement_tags': [], 'bucket_quota': {'enabled': False, 'check_on_raw': False, 'max_size': -1, 'max_size_kb': 0, 'max_objects': -1}, 'user_quota': {'enabled': False, 'check_on_raw': False, 'max_size': -1, 'max_size_kb': 0, 'max_objects': -1}, 'temp_url_keys': [], 'type': 'rgw', 'mfa_ids': []} 2023-10-05T19:28:28.270 INFO:tasks.util.rgw:rgwadmin: client.0 : ['user', 'info', '--uid', 'foo'] 2023-10-05T19:28:28.270 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'ceph', 'user', 'info', '--uid', 'foo'] 2023-10-05T19:28:28.270 DEBUG:teuthology.orchestra.run.smithi167:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster ceph user info --uid foo 2023-10-05T19:28:28.354 INFO:teuthology.orchestra.run.smithi167.stderr:ignoring --setuser ceph since I am not root 2023-10-05T19:28:28.354 INFO:teuthology.orchestra.run.smithi167.stderr:ignoring --setgroup ceph since I am not root 2023-10-05T19:28:28.355 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.350+0000 7f1931bace40 1 Processor -- start 2023-10-05T19:28:28.355 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.350+0000 7f1931bace40 1 -- start start 2023-10-05T19:28:28.355 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.350+0000 7f1931bace40 1 --2- >> v2:172.21.15.118:3300/0 conn(0x555f00f37910 0x555f00fc5b40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-05T19:28:28.355 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.350+0000 7f1931bace40 1 -- --> v1:172.21.15.118:6789/0 -- auth(proto 0 26 bytes epoch 0) v1 -- 0x555f00f0fbd0 con 0x555f00f38c20 2023-10-05T19:28:28.355 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.350+0000 7f1931bace40 1 -- --> v2:172.21.15.118:3300/0 -- mon_getmap magic: 0 v1 -- 0x555f00eccee0 con 0x555f00f37910 2023-10-05T19:28:28.358 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.350+0000 7f192eee0700 1 --2- >> v2:172.21.15.118:3300/0 conn(0x555f00f37910 0x555f00fc5b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-05T19:28:28.358 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.350+0000 7f192f6e1700 1 --1- >> v1:172.21.15.118:6789/0 conn(0x555f00f38c20 0x555f00f37200 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.118:6789/0 says I am v1:172.21.15.167:56464/0 (socket says 172.21.15.167:56464) 2023-10-05T19:28:28.358 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.350+0000 7f192eee0700 1 --2- >> v2:172.21.15.118:3300/0 conn(0x555f00f37910 0x555f00fc5b40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.118:3300/0 says I am v2:172.21.15.167:57682/0 (socket says 172.21.15.167:57682) 2023-10-05T19:28:28.358 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.350+0000 7f192f6e1700 1 -- 172.21.15.167:0/405646671 learned_addr learned my addr 172.21.15.167:0/405646671 (peer_addr_for_me v1:172.21.15.167:0/0) 2023-10-05T19:28:28.359 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.350+0000 7f192eee0700 1 -- 172.21.15.167:0/405646671 >> v1:172.21.15.118:6789/0 conn(0x555f00f38c20 legacy=0x555f00f37200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-05T19:28:28.359 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.350+0000 7f192eee0700 1 -- 172.21.15.167:0/405646671 --> v2:172.21.15.118:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x555f00f390b0 con 0x555f00f37910 2023-10-05T19:28:28.359 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.350+0000 7f192eee0700 1 --2- 172.21.15.167:0/405646671 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x555f00f37910 0x555f00fc5b40 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7f191800a5d0 tx=0x7f1918005a50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=80579705eb9bc747 server_cookie=0 in_seq=0 out_seq=0 2023-10-05T19:28:28.359 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.350+0000 7f192e6df700 1 -- 172.21.15.167:0/405646671 <== mon.0 v2:172.21.15.118:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f19180086a0 con 0x555f00f37910 2023-10-05T19:28:28.359 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.350+0000 7f192e6df700 1 -- 172.21.15.167:0/405646671 <== mon.0 v2:172.21.15.118:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1918008800 con 0x555f00f37910 2023-10-05T19:28:28.359 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.350+0000 7f192e6df700 1 -- 172.21.15.167:0/405646671 <== mon.0 v2:172.21.15.118:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f1918008ad0 con 0x555f00f37910 2023-10-05T19:28:28.359 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.350+0000 7f1931bace40 1 -- 172.21.15.167:0/405646671 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x555f00f37910 msgr2=0x555f00fc5b40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-05T19:28:28.359 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.350+0000 7f1931bace40 1 --2- 172.21.15.167:0/405646671 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x555f00f37910 0x555f00fc5b40 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7f191800a5d0 tx=0x7f1918005a50 comp rx=0 tx=0).stop 2023-10-05T19:28:28.359 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.350+0000 7f1931bace40 1 -- 172.21.15.167:0/405646671 shutdown_connections 2023-10-05T19:28:28.359 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.350+0000 7f1931bace40 1 --2- 172.21.15.167:0/405646671 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x555f00f37910 0x555f00fc5b40 unknown :-1 s=CLOSED pgs=255 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-05T19:28:28.359 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.350+0000 7f1931bace40 1 -- 172.21.15.167:0/405646671 >> 172.21.15.167:0/405646671 conn(0x555f00fbe850 msgr2=0x555f00fc0c60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-05T19:28:28.360 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.354+0000 7f1931bace40 1 -- 172.21.15.167:0/405646671 shutdown_connections 2023-10-05T19:28:28.360 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.354+0000 7f1931bace40 1 -- 172.21.15.167:0/405646671 wait complete. 2023-10-05T19:28:28.361 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.354+0000 7f1931bace40 1 Processor -- start 2023-10-05T19:28:28.362 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.354+0000 7f1931bace40 1 -- start start 2023-10-05T19:28:28.362 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.354+0000 7f1931bace40 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x555f00f37910 0x555f00fe8500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-05T19:28:28.362 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.354+0000 7f1931bace40 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x555f00e90e30 con 0x555f00f37910 2023-10-05T19:28:28.362 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.354+0000 7f192f6e1700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x555f00f37910 0x555f00fe8500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-05T19:28:28.362 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.354+0000 7f192f6e1700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x555f00f37910 0x555f00fe8500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.118:3300/0 says I am v2:172.21.15.167:57698/0 (socket says 172.21.15.167:57698) 2023-10-05T19:28:28.362 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.354+0000 7f192f6e1700 1 -- 172.21.15.167:0/3683854565 learned_addr learned my addr 172.21.15.167:0/3683854565 (peer_addr_for_me v2:172.21.15.167:0/0) 2023-10-05T19:28:28.362 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.354+0000 7f192f6e1700 1 -- 172.21.15.167:0/3683854565 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1918005cf0 con 0x555f00f37910 2023-10-05T19:28:28.362 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.354+0000 7f192f6e1700 1 --2- 172.21.15.167:0/3683854565 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x555f00f37910 0x555f00fe8500 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x7f19280014a0 tx=0x7f1928006b90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f4cbbd8b06269 server_cookie=0 in_seq=0 out_seq=0 2023-10-05T19:28:28.362 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.354+0000 7f192d6dd700 1 -- 172.21.15.167:0/3683854565 <== mon.0 v2:172.21.15.118:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f1928006110 con 0x555f00f37910 2023-10-05T19:28:28.363 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.354+0000 7f192d6dd700 1 -- 172.21.15.167:0/3683854565 <== mon.0 v2:172.21.15.118:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1928006270 con 0x555f00f37910 2023-10-05T19:28:28.363 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.354+0000 7f192d6dd700 1 -- 172.21.15.167:0/3683854565 <== mon.0 v2:172.21.15.118:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f1928004520 con 0x555f00f37910 2023-10-05T19:28:28.363 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.354+0000 7f1931bace40 1 -- 172.21.15.167:0/3683854565 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x555f00f37910 msgr2=0x555f00fe8500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-05T19:28:28.363 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.354+0000 7f1931bace40 1 --2- 172.21.15.167:0/3683854565 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x555f00f37910 0x555f00fe8500 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x7f19280014a0 tx=0x7f1928006b90 comp rx=0 tx=0).stop 2023-10-05T19:28:28.363 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.354+0000 7f1931bace40 1 -- 172.21.15.167:0/3683854565 shutdown_connections 2023-10-05T19:28:28.363 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.354+0000 7f1931bace40 1 --2- 172.21.15.167:0/3683854565 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x555f00f37910 0x555f00fe8500 unknown :-1 s=CLOSED pgs=256 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-05T19:28:28.363 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.354+0000 7f1931bace40 1 -- 172.21.15.167:0/3683854565 >> 172.21.15.167:0/3683854565 conn(0x555f00fbe850 msgr2=0x555f00fe40c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-05T19:28:28.363 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.354+0000 7f1931bace40 1 -- 172.21.15.167:0/3683854565 shutdown_connections 2023-10-05T19:28:28.363 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.354+0000 7f1931bace40 1 -- 172.21.15.167:0/3683854565 wait complete. 2023-10-05T19:28:28.365 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.358+0000 7f1931bace40 1 Processor -- start 2023-10-05T19:28:28.365 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.358+0000 7f1931bace40 1 -- start start 2023-10-05T19:28:28.365 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.358+0000 7f1931bace40 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x555f00f37910 0x555f00fef2d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-05T19:28:28.365 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.358+0000 7f1931bace40 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x555f00e63ed0 con 0x555f00f37910 2023-10-05T19:28:28.365 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.358+0000 7f192f6e1700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x555f00f37910 0x555f00fef2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-05T19:28:28.365 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.358+0000 7f192f6e1700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x555f00f37910 0x555f00fef2d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.118:3300/0 says I am v2:172.21.15.167:57702/0 (socket says 172.21.15.167:57702) 2023-10-05T19:28:28.365 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.358+0000 7f192f6e1700 1 -- 172.21.15.167:0/2517686516 learned_addr learned my addr 172.21.15.167:0/2517686516 (peer_addr_for_me v2:172.21.15.167:0/0) 2023-10-05T19:28:28.366 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.358+0000 7f192f6e1700 1 -- 172.21.15.167:0/2517686516 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x555f00f0fbd0 con 0x555f00f37910 2023-10-05T19:28:28.366 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.358+0000 7f192f6e1700 1 --2- 172.21.15.167:0/2517686516 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x555f00f37910 0x555f00fef2d0 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7f1928006bd0 tx=0x7f1928014000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-05T19:28:28.366 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.358+0000 7f19277fe700 1 -- 172.21.15.167:0/2517686516 <== mon.0 v2:172.21.15.118:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f1928006110 con 0x555f00f37910 2023-10-05T19:28:28.366 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.358+0000 7f19277fe700 1 -- 172.21.15.167:0/2517686516 <== mon.0 v2:172.21.15.118:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f19280063e0 con 0x555f00f37910 2023-10-05T19:28:28.366 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.358+0000 7f1931bace40 1 -- 172.21.15.167:0/2517686516 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x555f00fef810 con 0x555f00f37910 2023-10-05T19:28:28.366 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.358+0000 7f19277fe700 1 -- 172.21.15.167:0/2517686516 <== mon.0 v2:172.21.15.118:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f1928013d00 con 0x555f00f37910 2023-10-05T19:28:28.366 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.358+0000 7f1931bace40 1 -- 172.21.15.167:0/2517686516 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x555f00fefc50 con 0x555f00f37910 2023-10-05T19:28:28.367 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.358+0000 7f1931bace40 20 rados->read ofs=0 len=0 2023-10-05T19:28:28.368 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.362+0000 7f19277fe700 1 -- 172.21.15.167:0/2517686516 <== mon.0 v2:172.21.15.118:3300/0 4 ==== mgrmap(e 5) v1 ==== 40329+0+0 (secure 0 0 0) 0x7f1928004030 con 0x555f00f37910 2023-10-05T19:28:28.368 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.362+0000 7f19277fe700 1 --2- 172.21.15.167:0/2517686516 >> [v2:172.21.15.167:6824/29889,v1:172.21.15.167:6825/29889] conn(0x7f1910030c00 0x7f19100330b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-05T19:28:28.368 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.362+0000 7f19277fe700 1 -- 172.21.15.167:0/2517686516 <== mon.0 v2:172.21.15.118:3300/0 5 ==== osd_map(30..30 src has 1..30) v4 ==== 6057+0+0 (secure 0 0 0) 0x7f1928044950 con 0x555f00f37910 2023-10-05T19:28:28.368 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.362+0000 7f1931bace40 1 --2- 172.21.15.167:0/2517686516 >> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] conn(0x555f00f38c20 0x555f00ffad60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-05T19:28:28.368 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.362+0000 7f192eee0700 1 --2- 172.21.15.167:0/2517686516 >> [v2:172.21.15.167:6824/29889,v1:172.21.15.167:6825/29889] conn(0x7f1910030c00 0x7f19100330b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-05T19:28:28.368 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.362+0000 7f1931bace40 1 -- 172.21.15.167:0/2517686516 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:1 5.12 5:49953fa1:::default.realm:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x555f00ffb460 con 0x555f00f38c20 2023-10-05T19:28:28.368 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.362+0000 7f192fee2700 1 --2- 172.21.15.167:0/2517686516 >> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] conn(0x555f00f38c20 0x555f00ffad60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-05T19:28:28.368 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.362+0000 7f192eee0700 1 --2- 172.21.15.167:0/2517686516 >> [v2:172.21.15.167:6824/29889,v1:172.21.15.167:6825/29889] conn(0x7f1910030c00 0x7f19100330b0 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7f1918005a90 tx=0x7f1918015000 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-05T19:28:28.369 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.362+0000 7f192fee2700 1 --2- 172.21.15.167:0/2517686516 >> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] conn(0x555f00f38c20 0x555f00ffad60 crc :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-05T19:28:28.370 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.362+0000 7f192fee2700 1 -- 172.21.15.167:0/2517686516 <== osd.1 v2:172.21.15.167:6800/30035 1 ==== osd_op_reply(1 default.realm [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 ==== 157+0+0 (crc 0 0 0) 0x7f192000a500 con 0x555f00f38c20 2023-10-05T19:28:28.370 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.362+0000 7f1931bace40 20 rados_obj.operate() r=-2 bl.length=0 2023-10-05T19:28:28.370 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.362+0000 7f1931bace40 20 realm 2023-10-05T19:28:28.371 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.362+0000 7f1931bace40 20 rados->read ofs=0 len=0 2023-10-05T19:28:28.371 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.362+0000 7f1931bace40 1 -- 172.21.15.167:0/2517686516 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:2 5.12 5:49953fa1:::default.realm:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x555f00ff83c0 con 0x555f00f38c20 2023-10-05T19:28:28.371 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.362+0000 7f192fee2700 1 -- 172.21.15.167:0/2517686516 <== osd.1 v2:172.21.15.167:6800/30035 2 ==== osd_op_reply(2 default.realm [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 ==== 157+0+0 (crc 0 0 0) 0x7f192000a500 con 0x555f00f38c20 2023-10-05T19:28:28.371 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.362+0000 7f1931bace40 20 rados_obj.operate() r=-2 bl.length=0 2023-10-05T19:28:28.371 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.362+0000 7f1931bace40 4 RGWPeriod::init failed to init realm id : (2) No such file or directory 2023-10-05T19:28:28.371 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.362+0000 7f1931bace40 20 rados->read ofs=0 len=0 2023-10-05T19:28:28.371 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.362+0000 7f1931bace40 1 -- 172.21.15.167:0/2517686516 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:3 5.12 5:49953fa1:::default.realm:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x555f00ffbf40 con 0x555f00f38c20 2023-10-05T19:28:28.371 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.362+0000 7f192fee2700 1 -- 172.21.15.167:0/2517686516 <== osd.1 v2:172.21.15.167:6800/30035 3 ==== osd_op_reply(3 default.realm [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 ==== 157+0+0 (crc 0 0 0) 0x7f192000a500 con 0x555f00f38c20 2023-10-05T19:28:28.371 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.362+0000 7f1931bace40 20 rados_obj.operate() r=-2 bl.length=0 2023-10-05T19:28:28.371 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.362+0000 7f1931bace40 20 rados->read ofs=0 len=0 2023-10-05T19:28:28.372 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.362+0000 7f1931bace40 1 -- 172.21.15.167:0/2517686516 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:4 5.f 5:f43b6ece:::zone_names.default:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x555f00ffbb20 con 0x555f00f38c20 2023-10-05T19:28:28.372 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.366+0000 7f192fee2700 1 -- 172.21.15.167:0/2517686516 <== osd.1 v2:172.21.15.167:6800/30035 4 ==== osd_op_reply(4 zone_names.default [read 0~46 out=46b] v0'0 uv1 ondisk = 0) v8 ==== 162+0+46 (crc 0 0 0) 0x7f192000a500 con 0x555f00f38c20 2023-10-05T19:28:28.372 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.366+0000 7f1931bace40 20 rados_obj.operate() r=0 bl.length=46 2023-10-05T19:28:28.372 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.366+0000 7f1931bace40 20 rados->read ofs=0 len=0 2023-10-05T19:28:28.372 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.366+0000 7f1931bace40 1 -- 172.21.15.167:0/2517686516 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:5 5.1d 5:b8374eb4:::zone_info.9a9df965-6612-45d4-a45f-6e3c889bacc2:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x555f00ffc9b0 con 0x555f00f38c20 2023-10-05T19:28:28.373 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.366+0000 7f192fee2700 1 -- 172.21.15.167:0/2517686516 <== osd.1 v2:172.21.15.167:6800/30035 5 ==== osd_op_reply(5 zone_info.9a9df965-6612-45d4-a45f-6e3c889bacc2 [read 0~874 out=874b] v0'0 uv2 ondisk = 0) v8 ==== 190+0+874 (crc 0 0 0) 0x7f192000a500 con 0x555f00f38c20 2023-10-05T19:28:28.373 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.366+0000 7f1931bace40 20 rados_obj.operate() r=0 bl.length=874 2023-10-05T19:28:28.373 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.366+0000 7f1931bace40 20 searching for the correct realm 2023-10-05T19:28:28.373 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.366+0000 7f1931bace40 1 --2- 172.21.15.167:0/2517686516 >> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] conn(0x555f00ffe800 0x555f01000be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-05T19:28:28.373 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.366+0000 7f1931bace40 1 -- 172.21.15.167:0/2517686516 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:6 5.0 5:00000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x555f01001120 con 0x555f00ffe800 2023-10-05T19:28:28.373 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.366+0000 7f192f6e1700 1 --2- 172.21.15.167:0/2517686516 >> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] conn(0x555f00ffe800 0x555f01000be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-05T19:28:28.373 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.366+0000 7f192f6e1700 1 --2- 172.21.15.167:0/2517686516 >> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] conn(0x555f00ffe800 0x555f01000be0 crc :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-05T19:28:28.374 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.366+0000 7f192f6e1700 1 -- 172.21.15.167:0/2517686516 <== osd.3 v2:172.21.15.167:6808/30036 1 ==== osd_op_reply(6 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f1928012c70 con 0x555f00ffe800 2023-10-05T19:28:28.374 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.366+0000 7f192f6e1700 1 --2- 172.21.15.167:0/2517686516 >> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] conn(0x7f19280276e0 0x7f19280441a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-05T19:28:28.375 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.366+0000 7f192f6e1700 1 -- 172.21.15.167:0/2517686516 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:7 5.10 5:08000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x555f01001120 con 0x7f19280276e0 2023-10-05T19:28:28.375 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.370+0000 7f192eee0700 1 --2- 172.21.15.167:0/2517686516 >> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] conn(0x7f19280276e0 0x7f19280441a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-05T19:28:28.375 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.370+0000 7f192eee0700 1 --2- 172.21.15.167:0/2517686516 >> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] conn(0x7f19280276e0 0x7f19280441a0 crc :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-05T19:28:28.375 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.370+0000 7f192eee0700 1 -- 172.21.15.167:0/2517686516 <== osd.2 v2:172.21.15.167:6816/30037 1 ==== osd_op_reply(7 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f1918008ae0 con 0x7f19280276e0 2023-10-05T19:28:28.375 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.370+0000 7f192eee0700 1 --2- 172.21.15.167:0/2517686516 >> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] conn(0x7f1918007030 0x7f191801c920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-05T19:28:28.376 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.370+0000 7f192eee0700 1 -- 172.21.15.167:0/2517686516 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:8 5.8 5:10000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x555f01001120 con 0x7f1918007030 2023-10-05T19:28:28.376 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.370+0000 7f192fee2700 1 --2- 172.21.15.167:0/2517686516 >> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] conn(0x7f1918007030 0x7f191801c920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-05T19:28:28.376 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.370+0000 7f192fee2700 1 --2- 172.21.15.167:0/2517686516 >> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] conn(0x7f1918007030 0x7f191801c920 crc :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-05T19:28:28.377 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.370+0000 7f192fee2700 1 -- 172.21.15.167:0/2517686516 <== osd.0 v2:172.21.15.118:6800/27690 1 ==== osd_op_reply(8 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f1920014040 con 0x7f1918007030 2023-10-05T19:28:28.377 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.370+0000 7f192fee2700 1 -- 172.21.15.167:0/2517686516 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:9 5.18 5:18000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x555f01001120 con 0x555f00f38c20 2023-10-05T19:28:28.378 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.370+0000 7f192fee2700 1 -- 172.21.15.167:0/2517686516 <== osd.1 v2:172.21.15.167:6800/30035 6 ==== osd_op_reply(9 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f1920014040 con 0x555f00f38c20 2023-10-05T19:28:28.378 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.370+0000 7f192fee2700 1 -- 172.21.15.167:0/2517686516 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:10 5.4 5:20000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x555f01001120 con 0x555f00f38c20 2023-10-05T19:28:28.378 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.374+0000 7f192fee2700 1 -- 172.21.15.167:0/2517686516 <== osd.1 v2:172.21.15.167:6800/30035 7 ==== osd_op_reply(10 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f1920014040 con 0x555f00f38c20 2023-10-05T19:28:28.378 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.374+0000 7f192fee2700 1 -- 172.21.15.167:0/2517686516 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:11 5.14 5:28000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x555f01001120 con 0x7f1918007030 2023-10-05T19:28:28.379 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.374+0000 7f192fee2700 1 -- 172.21.15.167:0/2517686516 <== osd.0 v2:172.21.15.118:6800/27690 2 ==== osd_op_reply(11 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f1920014040 con 0x7f1918007030 2023-10-05T19:28:28.379 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.374+0000 7f192fee2700 1 -- 172.21.15.167:0/2517686516 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:12 5.c 5:30000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x555f01001120 con 0x7f1918007030 2023-10-05T19:28:28.380 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.374+0000 7f192fee2700 1 -- 172.21.15.167:0/2517686516 <== osd.0 v2:172.21.15.118:6800/27690 3 ==== osd_op_reply(12 [pgnls start_epoch 30 out=112b] v23'1 uv1 ondisk = 1) v8 ==== 144+0+112 (crc 0 0 0) 0x7f1920014040 con 0x7f1918007030 2023-10-05T19:28:28.380 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.374+0000 7f192fee2700 1 -- 172.21.15.167:0/2517686516 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:13 5.1c 5:38000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x555f01001120 con 0x7f19280276e0 2023-10-05T19:28:28.380 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.374+0000 7f192eee0700 1 -- 172.21.15.167:0/2517686516 <== osd.2 v2:172.21.15.167:6816/30037 2 ==== osd_op_reply(13 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f1918008ae0 con 0x7f19280276e0 2023-10-05T19:28:28.381 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.374+0000 7f192eee0700 1 -- 172.21.15.167:0/2517686516 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:14 5.2 5:40000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x555f01001120 con 0x555f00ffe800 2023-10-05T19:28:28.381 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.374+0000 7f192f6e1700 1 -- 172.21.15.167:0/2517686516 <== osd.3 v2:172.21.15.167:6808/30036 2 ==== osd_op_reply(14 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f1928012c70 con 0x555f00ffe800 2023-10-05T19:28:28.382 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.374+0000 7f192f6e1700 1 -- 172.21.15.167:0/2517686516 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:15 5.12 5:48000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x555f01001120 con 0x555f00f38c20 2023-10-05T19:28:28.382 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.378+0000 7f192fee2700 1 -- 172.21.15.167:0/2517686516 <== osd.1 v2:172.21.15.167:6800/30035 8 ==== osd_op_reply(15 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f1920014040 con 0x555f00f38c20 2023-10-05T19:28:28.382 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.378+0000 7f192fee2700 1 -- 172.21.15.167:0/2517686516 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:16 5.a 5:50000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x555f01001120 con 0x7f1918007030 2023-10-05T19:28:28.383 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.378+0000 7f192fee2700 1 -- 172.21.15.167:0/2517686516 <== osd.0 v2:172.21.15.118:6800/27690 4 ==== osd_op_reply(16 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f1920022040 con 0x7f1918007030 2023-10-05T19:28:28.383 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.378+0000 7f192fee2700 1 -- 172.21.15.167:0/2517686516 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:17 5.1a 5:58000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x555f01001120 con 0x555f00f38c20 2023-10-05T19:28:28.384 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.378+0000 7f192fee2700 1 -- 172.21.15.167:0/2517686516 <== osd.1 v2:172.21.15.167:6800/30035 9 ==== osd_op_reply(17 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f1920022040 con 0x555f00f38c20 2023-10-05T19:28:28.384 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.378+0000 7f192fee2700 1 -- 172.21.15.167:0/2517686516 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:18 5.6 5:60000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x555f01001120 con 0x7f19280276e0 2023-10-05T19:28:28.384 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.378+0000 7f192eee0700 1 -- 172.21.15.167:0/2517686516 <== osd.2 v2:172.21.15.167:6816/30037 3 ==== osd_op_reply(18 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f1918008ae0 con 0x7f19280276e0 2023-10-05T19:28:28.384 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.378+0000 7f192eee0700 1 -- 172.21.15.167:0/2517686516 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:19 5.16 5:68000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x555f01001120 con 0x555f00ffe800 2023-10-05T19:28:28.385 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.378+0000 7f192f6e1700 1 -- 172.21.15.167:0/2517686516 <== osd.3 v2:172.21.15.167:6808/30036 3 ==== osd_op_reply(19 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f1928012c70 con 0x555f00ffe800 2023-10-05T19:28:28.385 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.378+0000 7f192f6e1700 1 -- 172.21.15.167:0/2517686516 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:20 5.e 5:70000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x555f01001120 con 0x555f00ffe800 2023-10-05T19:28:28.386 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.382+0000 7f192f6e1700 1 -- 172.21.15.167:0/2517686516 <== osd.3 v2:172.21.15.167:6808/30036 4 ==== osd_op_reply(20 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f1928012c70 con 0x555f00ffe800 2023-10-05T19:28:28.386 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.382+0000 7f192f6e1700 1 -- 172.21.15.167:0/2517686516 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:21 5.1e 5:78000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x555f01001120 con 0x7f1918007030 2023-10-05T19:28:28.387 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.382+0000 7f192fee2700 1 -- 172.21.15.167:0/2517686516 <== osd.0 v2:172.21.15.118:6800/27690 5 ==== osd_op_reply(21 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f1920022040 con 0x7f1918007030 2023-10-05T19:28:28.387 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.382+0000 7f192fee2700 1 -- 172.21.15.167:0/2517686516 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:22 5.1 5:80000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x555f01001120 con 0x555f00ffe800 2023-10-05T19:28:28.388 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.382+0000 7f192f6e1700 1 -- 172.21.15.167:0/2517686516 <== osd.3 v2:172.21.15.167:6808/30036 5 ==== osd_op_reply(22 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f1928012c70 con 0x555f00ffe800 2023-10-05T19:28:28.388 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.382+0000 7f192f6e1700 1 -- 172.21.15.167:0/2517686516 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:23 5.11 5:88000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x555f01001120 con 0x555f00ffe800 2023-10-05T19:28:28.389 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.382+0000 7f192f6e1700 1 -- 172.21.15.167:0/2517686516 <== osd.3 v2:172.21.15.167:6808/30036 6 ==== osd_op_reply(23 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f1928012c70 con 0x555f00ffe800 2023-10-05T19:28:28.389 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.382+0000 7f192f6e1700 1 -- 172.21.15.167:0/2517686516 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:24 5.9 5:90000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x555f01001120 con 0x555f00f38c20 2023-10-05T19:28:28.389 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.382+0000 7f192fee2700 1 -- 172.21.15.167:0/2517686516 <== osd.1 v2:172.21.15.167:6800/30035 10 ==== osd_op_reply(24 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f1920022040 con 0x555f00f38c20 2023-10-05T19:28:28.389 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.382+0000 7f192fee2700 1 -- 172.21.15.167:0/2517686516 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:25 5.19 5:98000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x555f01001120 con 0x555f00f38c20 2023-10-05T19:28:28.390 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.386+0000 7f192fee2700 1 -- 172.21.15.167:0/2517686516 <== osd.1 v2:172.21.15.167:6800/30035 11 ==== osd_op_reply(25 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f1920022040 con 0x555f00f38c20 2023-10-05T19:28:28.390 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.386+0000 7f192fee2700 1 -- 172.21.15.167:0/2517686516 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:26 5.5 5:a0000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x555f01001120 con 0x7f1918007030 2023-10-05T19:28:28.391 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.386+0000 7f192fee2700 1 -- 172.21.15.167:0/2517686516 <== osd.0 v2:172.21.15.118:6800/27690 6 ==== osd_op_reply(26 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f1920022040 con 0x7f1918007030 2023-10-05T19:28:28.391 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.386+0000 7f192fee2700 1 -- 172.21.15.167:0/2517686516 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:27 5.15 5:a8000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x555f01001120 con 0x555f00ffe800 2023-10-05T19:28:28.392 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.386+0000 7f192f6e1700 1 -- 172.21.15.167:0/2517686516 <== osd.3 v2:172.21.15.167:6808/30036 7 ==== osd_op_reply(27 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f1928012c70 con 0x555f00ffe800 2023-10-05T19:28:28.392 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.386+0000 7f192f6e1700 1 -- 172.21.15.167:0/2517686516 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:28 5.d 5:b0000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x555f01001120 con 0x7f19280276e0 2023-10-05T19:28:28.393 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.386+0000 7f192eee0700 1 -- 172.21.15.167:0/2517686516 <== osd.2 v2:172.21.15.167:6816/30037 4 ==== osd_op_reply(28 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f1918008ae0 con 0x7f19280276e0 2023-10-05T19:28:28.393 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.386+0000 7f192eee0700 1 -- 172.21.15.167:0/2517686516 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:29 5.1d 5:b8000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x555f01001120 con 0x555f00f38c20 2023-10-05T19:28:28.393 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.386+0000 7f192fee2700 1 -- 172.21.15.167:0/2517686516 <== osd.1 v2:172.21.15.167:6800/30035 12 ==== osd_op_reply(29 [pgnls start_epoch 30 out=107b] v30'2 uv2 ondisk = 1) v8 ==== 144+0+107 (crc 0 0 0) 0x7f1920022040 con 0x555f00f38c20 2023-10-05T19:28:28.393 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.386+0000 7f192fee2700 1 -- 172.21.15.167:0/2517686516 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:30 5.3 5:c0000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x555f01001120 con 0x7f1918007030 2023-10-05T19:28:28.394 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.390+0000 7f192fee2700 1 -- 172.21.15.167:0/2517686516 <== osd.0 v2:172.21.15.118:6800/27690 7 ==== osd_op_reply(30 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f1920022040 con 0x7f1918007030 2023-10-05T19:28:28.395 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.390+0000 7f192fee2700 1 -- 172.21.15.167:0/2517686516 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:31 5.13 5:c8000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x555f01001120 con 0x555f00ffe800 2023-10-05T19:28:28.395 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.390+0000 7f192f6e1700 1 -- 172.21.15.167:0/2517686516 <== osd.3 v2:172.21.15.167:6808/30036 8 ==== osd_op_reply(31 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f1928012c70 con 0x555f00ffe800 2023-10-05T19:28:28.395 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.390+0000 7f192f6e1700 1 -- 172.21.15.167:0/2517686516 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:32 5.b 5:d0000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x555f01001120 con 0x7f1918007030 2023-10-05T19:28:28.397 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.390+0000 7f192fee2700 1 -- 172.21.15.167:0/2517686516 <== osd.0 v2:172.21.15.118:6800/27690 8 ==== osd_op_reply(32 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f1920022040 con 0x7f1918007030 2023-10-05T19:28:28.397 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.390+0000 7f192fee2700 1 -- 172.21.15.167:0/2517686516 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:33 5.1b 5:d8000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x555f01001120 con 0x7f19280276e0 2023-10-05T19:28:28.397 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.390+0000 7f192eee0700 1 -- 172.21.15.167:0/2517686516 <== osd.2 v2:172.21.15.167:6816/30037 5 ==== osd_op_reply(33 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f1918008ae0 con 0x7f19280276e0 2023-10-05T19:28:28.397 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.390+0000 7f192eee0700 1 -- 172.21.15.167:0/2517686516 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:34 5.7 5:e0000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x555f01001120 con 0x555f00ffe800 2023-10-05T19:28:28.398 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.394+0000 7f192f6e1700 1 -- 172.21.15.167:0/2517686516 <== osd.3 v2:172.21.15.167:6808/30036 9 ==== osd_op_reply(34 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f1928012c70 con 0x555f00ffe800 2023-10-05T19:28:28.398 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.394+0000 7f192f6e1700 1 -- 172.21.15.167:0/2517686516 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:35 5.17 5:e8000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x555f01001120 con 0x7f1918007030 2023-10-05T19:28:28.399 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.394+0000 7f192fee2700 1 -- 172.21.15.167:0/2517686516 <== osd.0 v2:172.21.15.118:6800/27690 9 ==== osd_op_reply(35 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f1920022040 con 0x7f1918007030 2023-10-05T19:28:28.399 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.394+0000 7f192fee2700 1 -- 172.21.15.167:0/2517686516 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:36 5.f 5:f0000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x555f01001120 con 0x555f00f38c20 2023-10-05T19:28:28.400 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.394+0000 7f192fee2700 1 -- 172.21.15.167:0/2517686516 <== osd.1 v2:172.21.15.167:6800/30035 13 ==== osd_op_reply(36 [pgnls start_epoch 30 out=115b] v23'2 uv2 ondisk = 1) v8 ==== 144+0+115 (crc 0 0 0) 0x7f192002f080 con 0x555f00f38c20 2023-10-05T19:28:28.400 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.394+0000 7f192fee2700 1 -- 172.21.15.167:0/2517686516 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:37 5.1f 5:f8000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x555f01001120 con 0x7f19280276e0 2023-10-05T19:28:28.401 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.394+0000 7f192eee0700 1 -- 172.21.15.167:0/2517686516 <== osd.2 v2:172.21.15.167:6816/30037 6 ==== osd_op_reply(37 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f1918008ae0 con 0x7f19280276e0 2023-10-05T19:28:28.401 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.394+0000 7f1931bace40 20 RGWRados::pool_iterate: got zonegroup_info.4f1f719d-9648-417b-b10c-e5d939825419 2023-10-05T19:28:28.401 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.394+0000 7f1931bace40 20 RGWRados::pool_iterate: got zone_info.9a9df965-6612-45d4-a45f-6e3c889bacc2 2023-10-05T19:28:28.401 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.394+0000 7f1931bace40 20 RGWRados::pool_iterate: got zone_names.default 2023-10-05T19:28:28.401 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.394+0000 7f1931bace40 20 RGWRados::pool_iterate: got zonegroups_names.default 2023-10-05T19:28:28.401 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.394+0000 7f1931bace40 20 rados->read ofs=0 len=0 2023-10-05T19:28:28.401 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.394+0000 7f1931bace40 1 -- 172.21.15.167:0/2517686516 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:38 5.12 5:49953fa1:::default.realm:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x555f00ffd580 con 0x555f00f38c20 2023-10-05T19:28:28.402 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.394+0000 7f192fee2700 1 -- 172.21.15.167:0/2517686516 <== osd.1 v2:172.21.15.167:6800/30035 14 ==== osd_op_reply(38 default.realm [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 ==== 157+0+0 (crc 0 0 0) 0x7f192002f080 con 0x555f00f38c20 2023-10-05T19:28:28.402 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.394+0000 7f1931bace40 20 rados_obj.operate() r=-2 bl.length=0 2023-10-05T19:28:28.402 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.394+0000 7f1931bace40 20 rados->read ofs=0 len=0 2023-10-05T19:28:28.403 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.394+0000 7f1931bace40 1 -- 172.21.15.167:0/2517686516 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:39 5.f 5:f4c53578:::zonegroups_names.default:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x555f00ffdf00 con 0x555f00f38c20 2023-10-05T19:28:28.403 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.398+0000 7f192fee2700 1 -- 172.21.15.167:0/2517686516 <== osd.1 v2:172.21.15.167:6800/30035 15 ==== osd_op_reply(39 zonegroups_names.default [read 0~46 out=46b] v0'0 uv2 ondisk = 0) v8 ==== 168+0+46 (crc 0 0 0) 0x7f192002f080 con 0x555f00f38c20 2023-10-05T19:28:28.403 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.398+0000 7f1931bace40 20 rados_obj.operate() r=0 bl.length=46 2023-10-05T19:28:28.403 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.398+0000 7f1931bace40 20 rados->read ofs=0 len=0 2023-10-05T19:28:28.403 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.398+0000 7f1931bace40 1 -- 172.21.15.167:0/2517686516 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:40 5.c 5:30d299a5:::zonegroup_info.4f1f719d-9648-417b-b10c-e5d939825419:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x555f00ffd140 con 0x7f1918007030 2023-10-05T19:28:28.403 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.398+0000 7f192fee2700 1 -- 172.21.15.167:0/2517686516 <== osd.0 v2:172.21.15.118:6800/27690 10 ==== osd_op_reply(40 zonegroup_info.4f1f719d-9648-417b-b10c-e5d939825419 [read 0~362 out=362b] v0'0 uv1 ondisk = 0) v8 ==== 195+0+362 (crc 0 0 0) 0x7f192002f080 con 0x7f1918007030 2023-10-05T19:28:28.404 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.398+0000 7f1931bace40 20 rados_obj.operate() r=0 bl.length=362 2023-10-05T19:28:28.404 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.398+0000 7f1931bace40 20 rados->read ofs=0 len=0 2023-10-05T19:28:28.404 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.398+0000 7f1931bace40 1 -- 172.21.15.167:0/2517686516 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:41 5.13 5:cbbf6c60:::converted:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x555f010014d0 con 0x555f00ffe800 2023-10-05T19:28:28.405 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.398+0000 7f192f6e1700 1 -- 172.21.15.167:0/2517686516 <== osd.3 v2:172.21.15.167:6808/30036 10 ==== osd_op_reply(41 converted [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 ==== 153+0+0 (crc 0 0 0) 0x7f1928012c70 con 0x555f00ffe800 2023-10-05T19:28:28.405 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.398+0000 7f1931bace40 20 rados_obj.operate() r=-2 bl.length=0 2023-10-05T19:28:28.405 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.398+0000 7f1931bace40 20 rados->read ofs=0 len=0 2023-10-05T19:28:28.405 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.398+0000 7f1931bace40 1 -- 172.21.15.167:0/2517686516 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:42 5.12 5:49953fa1:::default.realm:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x555f01001880 con 0x555f00f38c20 2023-10-05T19:28:28.405 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.398+0000 7f192fee2700 1 -- 172.21.15.167:0/2517686516 <== osd.1 v2:172.21.15.167:6800/30035 16 ==== osd_op_reply(42 default.realm [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 ==== 157+0+0 (crc 0 0 0) 0x7f192002f080 con 0x555f00f38c20 2023-10-05T19:28:28.405 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.398+0000 7f1931bace40 20 rados_obj.operate() r=-2 bl.length=0 2023-10-05T19:28:28.405 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.398+0000 7f1931bace40 20 rados->read ofs=0 len=0 2023-10-05T19:28:28.405 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.398+0000 7f1931bace40 1 -- 172.21.15.167:0/2517686516 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:43 5.f 5:f4c53578:::zonegroups_names.default:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x555f01001880 con 0x555f00f38c20 2023-10-05T19:28:28.406 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.398+0000 7f192fee2700 1 -- 172.21.15.167:0/2517686516 <== osd.1 v2:172.21.15.167:6800/30035 17 ==== osd_op_reply(43 zonegroups_names.default [read 0~46 out=46b] v0'0 uv2 ondisk = 0) v8 ==== 168+0+46 (crc 0 0 0) 0x7f192002f080 con 0x555f00f38c20 2023-10-05T19:28:28.406 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.398+0000 7f1931bace40 20 rados_obj.operate() r=0 bl.length=46 2023-10-05T19:28:28.406 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.398+0000 7f1931bace40 1 -- 172.21.15.167:0/2517686516 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:44 5.0 5:00000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x555f01002890 con 0x555f00ffe800 2023-10-05T19:28:28.408 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.402+0000 7f192f6e1700 1 -- 172.21.15.167:0/2517686516 <== osd.3 v2:172.21.15.167:6808/30036 11 ==== osd_op_reply(44 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f1928012c70 con 0x555f00ffe800 2023-10-05T19:28:28.408 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.402+0000 7f192f6e1700 1 -- 172.21.15.167:0/2517686516 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:45 5.10 5:08000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x555f01002890 con 0x7f19280276e0 2023-10-05T19:28:28.409 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.402+0000 7f192eee0700 1 -- 172.21.15.167:0/2517686516 <== osd.2 v2:172.21.15.167:6816/30037 7 ==== osd_op_reply(45 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f1918008ae0 con 0x7f19280276e0 2023-10-05T19:28:28.409 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.402+0000 7f192eee0700 1 -- 172.21.15.167:0/2517686516 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:46 5.8 5:10000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x555f01002890 con 0x7f1918007030 2023-10-05T19:28:28.410 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.406+0000 7f192fee2700 1 -- 172.21.15.167:0/2517686516 <== osd.0 v2:172.21.15.118:6800/27690 11 ==== osd_op_reply(46 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f192002f080 con 0x7f1918007030 2023-10-05T19:28:28.410 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.406+0000 7f192fee2700 1 -- 172.21.15.167:0/2517686516 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:47 5.18 5:18000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x555f00ffdf00 con 0x555f00f38c20 2023-10-05T19:28:28.411 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.406+0000 7f192fee2700 1 -- 172.21.15.167:0/2517686516 <== osd.1 v2:172.21.15.167:6800/30035 18 ==== osd_op_reply(47 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f192002f080 con 0x555f00f38c20 2023-10-05T19:28:28.411 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.406+0000 7f192fee2700 1 -- 172.21.15.167:0/2517686516 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:48 5.4 5:20000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x555f00ffdf00 con 0x555f00f38c20 2023-10-05T19:28:28.411 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.406+0000 7f192fee2700 1 -- 172.21.15.167:0/2517686516 <== osd.1 v2:172.21.15.167:6800/30035 19 ==== osd_op_reply(48 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f192002f080 con 0x555f00f38c20 2023-10-05T19:28:28.412 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.406+0000 7f192fee2700 1 -- 172.21.15.167:0/2517686516 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:49 5.14 5:28000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x555f00ffdf00 con 0x7f1918007030 2023-10-05T19:28:28.412 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.406+0000 7f192fee2700 1 -- 172.21.15.167:0/2517686516 <== osd.0 v2:172.21.15.118:6800/27690 12 ==== osd_op_reply(49 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f192002f080 con 0x7f1918007030 2023-10-05T19:28:28.413 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.406+0000 7f192fee2700 1 -- 172.21.15.167:0/2517686516 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:50 5.c 5:30000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x555f00ffdf00 con 0x7f1918007030 2023-10-05T19:28:28.414 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.406+0000 7f192fee2700 1 -- 172.21.15.167:0/2517686516 <== osd.0 v2:172.21.15.118:6800/27690 13 ==== osd_op_reply(50 [pgnls start_epoch 30 out=112b] v23'1 uv1 ondisk = 1) v8 ==== 144+0+112 (crc 0 0 0) 0x7f192002f080 con 0x7f1918007030 2023-10-05T19:28:28.414 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.406+0000 7f192fee2700 1 -- 172.21.15.167:0/2517686516 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:51 5.1c 5:38000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x555f00ffdf00 con 0x7f19280276e0 2023-10-05T19:28:28.415 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.410+0000 7f192eee0700 1 -- 172.21.15.167:0/2517686516 <== osd.2 v2:172.21.15.167:6816/30037 8 ==== osd_op_reply(51 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f1918008ae0 con 0x7f19280276e0 2023-10-05T19:28:28.415 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.410+0000 7f192eee0700 1 -- 172.21.15.167:0/2517686516 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:52 5.2 5:40000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x555f00ffdf00 con 0x555f00ffe800 2023-10-05T19:28:28.415 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.410+0000 7f192f6e1700 1 -- 172.21.15.167:0/2517686516 <== osd.3 v2:172.21.15.167:6808/30036 12 ==== osd_op_reply(52 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f1928012c70 con 0x555f00ffe800 2023-10-05T19:28:28.416 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.410+0000 7f192f6e1700 1 -- 172.21.15.167:0/2517686516 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:53 5.12 5:48000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x555f00ffdf00 con 0x555f00f38c20 2023-10-05T19:28:28.416 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.410+0000 7f192fee2700 1 -- 172.21.15.167:0/2517686516 <== osd.1 v2:172.21.15.167:6800/30035 20 ==== osd_op_reply(53 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f192002f080 con 0x555f00f38c20 2023-10-05T19:28:28.417 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.410+0000 7f192fee2700 1 -- 172.21.15.167:0/2517686516 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:54 5.a 5:50000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x555f00ffdf00 con 0x7f1918007030 2023-10-05T19:28:28.417 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.410+0000 7f192fee2700 1 -- 172.21.15.167:0/2517686516 <== osd.0 v2:172.21.15.118:6800/27690 14 ==== osd_op_reply(54 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f192002f080 con 0x7f1918007030 2023-10-05T19:28:28.417 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.410+0000 7f192fee2700 1 -- 172.21.15.167:0/2517686516 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:55 5.1a 5:58000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x555f00ffdf00 con 0x555f00f38c20 2023-10-05T19:28:28.418 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.414+0000 7f192fee2700 1 -- 172.21.15.167:0/2517686516 <== osd.1 v2:172.21.15.167:6800/30035 21 ==== osd_op_reply(55 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f192002f080 con 0x555f00f38c20 2023-10-05T19:28:28.418 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.414+0000 7f192fee2700 1 -- 172.21.15.167:0/2517686516 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:56 5.6 5:60000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x555f00ffdf00 con 0x7f19280276e0 2023-10-05T19:28:28.419 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.414+0000 7f192eee0700 1 -- 172.21.15.167:0/2517686516 <== osd.2 v2:172.21.15.167:6816/30037 9 ==== osd_op_reply(56 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f1918008ae0 con 0x7f19280276e0 2023-10-05T19:28:28.419 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.414+0000 7f192eee0700 1 -- 172.21.15.167:0/2517686516 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:57 5.16 5:68000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x555f00ffdf00 con 0x555f00ffe800 2023-10-05T19:28:28.420 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.414+0000 7f192f6e1700 1 -- 172.21.15.167:0/2517686516 <== osd.3 v2:172.21.15.167:6808/30036 13 ==== osd_op_reply(57 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f1928012c70 con 0x555f00ffe800 2023-10-05T19:28:28.420 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.414+0000 7f192f6e1700 1 -- 172.21.15.167:0/2517686516 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:58 5.e 5:70000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x555f00ffdf00 con 0x555f00ffe800 2023-10-05T19:28:28.421 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.414+0000 7f192f6e1700 1 -- 172.21.15.167:0/2517686516 <== osd.3 v2:172.21.15.167:6808/30036 14 ==== osd_op_reply(58 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f1928012c70 con 0x555f00ffe800 2023-10-05T19:28:28.421 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.414+0000 7f192f6e1700 1 -- 172.21.15.167:0/2517686516 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:59 5.1e 5:78000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x555f00ffdf00 con 0x7f1918007030 2023-10-05T19:28:28.422 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.418+0000 7f192fee2700 1 -- 172.21.15.167:0/2517686516 <== osd.0 v2:172.21.15.118:6800/27690 15 ==== osd_op_reply(59 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f192002f080 con 0x7f1918007030 2023-10-05T19:28:28.422 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.418+0000 7f192fee2700 1 -- 172.21.15.167:0/2517686516 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:60 5.1 5:80000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x555f00ffdf00 con 0x555f00ffe800 2023-10-05T19:28:28.423 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.418+0000 7f192f6e1700 1 -- 172.21.15.167:0/2517686516 <== osd.3 v2:172.21.15.167:6808/30036 15 ==== osd_op_reply(60 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f1928012c70 con 0x555f00ffe800 2023-10-05T19:28:28.423 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.418+0000 7f192f6e1700 1 -- 172.21.15.167:0/2517686516 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:61 5.11 5:88000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x555f00ffdf00 con 0x555f00ffe800 2023-10-05T19:28:28.423 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.418+0000 7f192f6e1700 1 -- 172.21.15.167:0/2517686516 <== osd.3 v2:172.21.15.167:6808/30036 16 ==== osd_op_reply(61 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f1928012c70 con 0x555f00ffe800 2023-10-05T19:28:28.424 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.418+0000 7f192f6e1700 1 -- 172.21.15.167:0/2517686516 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:62 5.9 5:90000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x555f00ffdf00 con 0x555f00f38c20 2023-10-05T19:28:28.424 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.418+0000 7f192fee2700 1 -- 172.21.15.167:0/2517686516 <== osd.1 v2:172.21.15.167:6800/30035 22 ==== osd_op_reply(62 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f192002f080 con 0x555f00f38c20 2023-10-05T19:28:28.424 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.418+0000 7f192fee2700 1 -- 172.21.15.167:0/2517686516 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:63 5.19 5:98000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x555f00ffdf00 con 0x555f00f38c20 2023-10-05T19:28:28.425 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.418+0000 7f192fee2700 1 -- 172.21.15.167:0/2517686516 <== osd.1 v2:172.21.15.167:6800/30035 23 ==== osd_op_reply(63 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f192002f080 con 0x555f00f38c20 2023-10-05T19:28:28.425 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.418+0000 7f192fee2700 1 -- 172.21.15.167:0/2517686516 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:64 5.5 5:a0000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x555f00ffdf00 con 0x7f1918007030 2023-10-05T19:28:28.426 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.422+0000 7f192fee2700 1 -- 172.21.15.167:0/2517686516 <== osd.0 v2:172.21.15.118:6800/27690 16 ==== osd_op_reply(64 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f192002f080 con 0x7f1918007030 2023-10-05T19:28:28.426 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.422+0000 7f192fee2700 1 -- 172.21.15.167:0/2517686516 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:65 5.15 5:a8000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x555f00ffdf00 con 0x555f00ffe800 2023-10-05T19:28:28.427 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.422+0000 7f192f6e1700 1 -- 172.21.15.167:0/2517686516 <== osd.3 v2:172.21.15.167:6808/30036 17 ==== osd_op_reply(65 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f1928012c70 con 0x555f00ffe800 2023-10-05T19:28:28.427 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.422+0000 7f192f6e1700 1 -- 172.21.15.167:0/2517686516 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:66 5.d 5:b0000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x555f00ffdf00 con 0x7f19280276e0 2023-10-05T19:28:28.427 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.422+0000 7f192eee0700 1 -- 172.21.15.167:0/2517686516 <== osd.2 v2:172.21.15.167:6816/30037 10 ==== osd_op_reply(66 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f1918008ae0 con 0x7f19280276e0 2023-10-05T19:28:28.428 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.422+0000 7f192eee0700 1 -- 172.21.15.167:0/2517686516 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:67 5.1d 5:b8000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x555f00ffdf00 con 0x555f00f38c20 2023-10-05T19:28:28.428 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.422+0000 7f192fee2700 1 -- 172.21.15.167:0/2517686516 <== osd.1 v2:172.21.15.167:6800/30035 24 ==== osd_op_reply(67 [pgnls start_epoch 30 out=107b] v30'2 uv2 ondisk = 1) v8 ==== 144+0+107 (crc 0 0 0) 0x7f192002f080 con 0x555f00f38c20 2023-10-05T19:28:28.428 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.422+0000 7f192fee2700 1 -- 172.21.15.167:0/2517686516 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:68 5.3 5:c0000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x555f00ffdf00 con 0x7f1918007030 2023-10-05T19:28:28.429 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.422+0000 7f192fee2700 1 -- 172.21.15.167:0/2517686516 <== osd.0 v2:172.21.15.118:6800/27690 17 ==== osd_op_reply(68 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f192002f080 con 0x7f1918007030 2023-10-05T19:28:28.429 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.422+0000 7f192fee2700 1 -- 172.21.15.167:0/2517686516 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:69 5.13 5:c8000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x555f00ffdf00 con 0x555f00ffe800 2023-10-05T19:28:28.430 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.426+0000 7f192f6e1700 1 -- 172.21.15.167:0/2517686516 <== osd.3 v2:172.21.15.167:6808/30036 18 ==== osd_op_reply(69 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f1928012c70 con 0x555f00ffe800 2023-10-05T19:28:28.430 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.426+0000 7f192f6e1700 1 -- 172.21.15.167:0/2517686516 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:70 5.b 5:d0000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x555f00ffdf00 con 0x7f1918007030 2023-10-05T19:28:28.431 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.426+0000 7f192fee2700 1 -- 172.21.15.167:0/2517686516 <== osd.0 v2:172.21.15.118:6800/27690 18 ==== osd_op_reply(70 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f192002f080 con 0x7f1918007030 2023-10-05T19:28:28.431 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.426+0000 7f192fee2700 1 -- 172.21.15.167:0/2517686516 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:71 5.1b 5:d8000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x555f00ffdf00 con 0x7f19280276e0 2023-10-05T19:28:28.432 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.426+0000 7f192eee0700 1 -- 172.21.15.167:0/2517686516 <== osd.2 v2:172.21.15.167:6816/30037 11 ==== osd_op_reply(71 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f1918008ae0 con 0x7f19280276e0 2023-10-05T19:28:28.432 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.426+0000 7f192eee0700 1 -- 172.21.15.167:0/2517686516 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:72 5.7 5:e0000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x555f00ffdf00 con 0x555f00ffe800 2023-10-05T19:28:28.433 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.426+0000 7f192f6e1700 1 -- 172.21.15.167:0/2517686516 <== osd.3 v2:172.21.15.167:6808/30036 19 ==== osd_op_reply(72 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f1928012c70 con 0x555f00ffe800 2023-10-05T19:28:28.433 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.426+0000 7f192f6e1700 1 -- 172.21.15.167:0/2517686516 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:73 5.17 5:e8000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x555f00ffdf00 con 0x7f1918007030 2023-10-05T19:28:28.433 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.426+0000 7f192fee2700 1 -- 172.21.15.167:0/2517686516 <== osd.0 v2:172.21.15.118:6800/27690 19 ==== osd_op_reply(73 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f192002f080 con 0x7f1918007030 2023-10-05T19:28:28.434 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.426+0000 7f192fee2700 1 -- 172.21.15.167:0/2517686516 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:74 5.f 5:f0000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x555f00ffdf00 con 0x555f00f38c20 2023-10-05T19:28:28.434 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.430+0000 7f192fee2700 1 -- 172.21.15.167:0/2517686516 <== osd.1 v2:172.21.15.167:6800/30035 25 ==== osd_op_reply(74 [pgnls start_epoch 30 out=115b] v23'2 uv2 ondisk = 1) v8 ==== 144+0+115 (crc 0 0 0) 0x7f192002f080 con 0x555f00f38c20 2023-10-05T19:28:28.434 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.430+0000 7f192fee2700 1 -- 172.21.15.167:0/2517686516 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:75 5.1f 5:f8000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x555f00ffdf00 con 0x7f19280276e0 2023-10-05T19:28:28.435 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.430+0000 7f192eee0700 1 -- 172.21.15.167:0/2517686516 <== osd.2 v2:172.21.15.167:6816/30037 12 ==== osd_op_reply(75 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f1918008ae0 con 0x7f19280276e0 2023-10-05T19:28:28.435 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.430+0000 7f1931bace40 20 RGWRados::pool_iterate: got zonegroup_info.4f1f719d-9648-417b-b10c-e5d939825419 2023-10-05T19:28:28.436 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.430+0000 7f1931bace40 20 RGWRados::pool_iterate: got zone_info.9a9df965-6612-45d4-a45f-6e3c889bacc2 2023-10-05T19:28:28.436 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.430+0000 7f1931bace40 20 RGWRados::pool_iterate: got zone_names.default 2023-10-05T19:28:28.436 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.430+0000 7f1931bace40 20 RGWRados::pool_iterate: got zonegroups_names.default 2023-10-05T19:28:28.436 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.430+0000 7f1931bace40 20 rados->read ofs=0 len=0 2023-10-05T19:28:28.436 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.430+0000 7f1931bace40 1 -- 172.21.15.167:0/2517686516 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:76 5.f 5:f43b6ece:::zone_names.default:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x555f01001880 con 0x555f00f38c20 2023-10-05T19:28:28.437 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.430+0000 7f192fee2700 1 -- 172.21.15.167:0/2517686516 <== osd.1 v2:172.21.15.167:6800/30035 26 ==== osd_op_reply(76 zone_names.default [read 0~46 out=46b] v0'0 uv1 ondisk = 0) v8 ==== 162+0+46 (crc 0 0 0) 0x7f192002f080 con 0x555f00f38c20 2023-10-05T19:28:28.437 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.430+0000 7f1931bace40 20 rados_obj.operate() r=0 bl.length=46 2023-10-05T19:28:28.437 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.430+0000 7f1931bace40 20 rados->read ofs=0 len=0 2023-10-05T19:28:28.437 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.430+0000 7f1931bace40 1 -- 172.21.15.167:0/2517686516 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:77 5.1d 5:b8374eb4:::zone_info.9a9df965-6612-45d4-a45f-6e3c889bacc2:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x555f01001c30 con 0x555f00f38c20 2023-10-05T19:28:28.437 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.430+0000 7f192fee2700 1 -- 172.21.15.167:0/2517686516 <== osd.1 v2:172.21.15.167:6800/30035 27 ==== osd_op_reply(77 zone_info.9a9df965-6612-45d4-a45f-6e3c889bacc2 [read 0~874 out=874b] v0'0 uv2 ondisk = 0) v8 ==== 190+0+874 (crc 0 0 0) 0x7f192002f080 con 0x555f00f38c20 2023-10-05T19:28:28.438 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.430+0000 7f1931bace40 20 rados_obj.operate() r=0 bl.length=874 2023-10-05T19:28:28.438 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.430+0000 7f1931bace40 20 rados->read ofs=0 len=0 2023-10-05T19:28:28.438 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.430+0000 7f1931bace40 1 -- 172.21.15.167:0/2517686516 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:78 5.f 5:f4c53578:::zonegroups_names.default:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x555f01002120 con 0x555f00f38c20 2023-10-05T19:28:28.439 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.430+0000 7f192fee2700 1 -- 172.21.15.167:0/2517686516 <== osd.1 v2:172.21.15.167:6800/30035 28 ==== osd_op_reply(78 zonegroups_names.default [read 0~46 out=46b] v0'0 uv2 ondisk = 0) v8 ==== 168+0+46 (crc 0 0 0) 0x7f192002f080 con 0x555f00f38c20 2023-10-05T19:28:28.439 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.430+0000 7f1931bace40 20 rados_obj.operate() r=0 bl.length=46 2023-10-05T19:28:28.439 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.430+0000 7f1931bace40 20 rados->read ofs=0 len=0 2023-10-05T19:28:28.439 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.430+0000 7f1931bace40 1 -- 172.21.15.167:0/2517686516 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:79 5.c 5:30d299a5:::zonegroup_info.4f1f719d-9648-417b-b10c-e5d939825419:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x555f01002120 con 0x7f1918007030 2023-10-05T19:28:28.439 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.434+0000 7f192fee2700 1 -- 172.21.15.167:0/2517686516 <== osd.0 v2:172.21.15.118:6800/27690 20 ==== osd_op_reply(79 zonegroup_info.4f1f719d-9648-417b-b10c-e5d939825419 [read 0~362 out=362b] v0'0 uv1 ondisk = 0) v8 ==== 195+0+362 (crc 0 0 0) 0x7f192002f080 con 0x7f1918007030 2023-10-05T19:28:28.439 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.434+0000 7f1931bace40 20 rados_obj.operate() r=0 bl.length=362 2023-10-05T19:28:28.439 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.434+0000 7f1931bace40 20 rados->read ofs=0 len=0 2023-10-05T19:28:28.439 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.434+0000 7f1931bace40 1 -- 172.21.15.167:0/2517686516 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:80 5.3 5:c0ad8cbd:::region_map:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x555f01002220 con 0x7f1918007030 2023-10-05T19:28:28.440 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.434+0000 7f192fee2700 1 -- 172.21.15.167:0/2517686516 <== osd.0 v2:172.21.15.118:6800/27690 21 ==== osd_op_reply(80 region_map [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 ==== 154+0+0 (crc 0 0 0) 0x7f1920050040 con 0x7f1918007030 2023-10-05T19:28:28.440 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.434+0000 7f1931bace40 20 rados_obj.operate() r=-2 bl.length=0 2023-10-05T19:28:28.440 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.434+0000 7f1931bace40 20 zone default found 2023-10-05T19:28:28.440 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.434+0000 7f1931bace40 4 Realm: () 2023-10-05T19:28:28.440 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.434+0000 7f1931bace40 4 ZoneGroup: default (4f1f719d-9648-417b-b10c-e5d939825419) 2023-10-05T19:28:28.440 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.434+0000 7f1931bace40 4 Zone: default (9a9df965-6612-45d4-a45f-6e3c889bacc2) 2023-10-05T19:28:28.441 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.434+0000 7f1931bace40 10 cannot find current period zonegroup using local zonegroup configuration 2023-10-05T19:28:28.441 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.434+0000 7f1931bace40 20 zonegroup default 2023-10-05T19:28:28.441 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.434+0000 7f1931bace40 20 rados->read ofs=0 len=0 2023-10-05T19:28:28.441 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.434+0000 7f1931bace40 1 -- 172.21.15.167:0/2517686516 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:81 5.3 5:c52100b6:::period_config.default:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x555f01002220 con 0x7f1918007030 2023-10-05T19:28:28.442 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.434+0000 7f192fee2700 1 -- 172.21.15.167:0/2517686516 <== osd.0 v2:172.21.15.118:6800/27690 22 ==== osd_op_reply(81 period_config.default [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 ==== 165+0+0 (crc 0 0 0) 0x7f1920050040 con 0x7f1918007030 2023-10-05T19:28:28.442 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.434+0000 7f1931bace40 20 rados_obj.operate() r=-2 bl.length=0 2023-10-05T19:28:28.442 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.434+0000 7f1931bace40 20 rados->read ofs=0 len=0 2023-10-05T19:28:28.442 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.434+0000 7f1931bace40 1 -- 172.21.15.167:0/2517686516 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:82 6.2 6:4347d321:::bucket.sync-source-hints.:head [call version.read in=11b,read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x555f01003ba0 con 0x555f00ffe800 2023-10-05T19:28:28.442 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.434+0000 7f192f6e1700 1 -- 172.21.15.167:0/2517686516 <== osd.3 v2:172.21.15.167:6808/30036 20 ==== osd_op_reply(82 bucket.sync-source-hints. [call,read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 ==== 211+0+0 (crc 0 0 0) 0x7f1928012c70 con 0x555f00ffe800 2023-10-05T19:28:28.442 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.434+0000 7f1931bace40 20 rados_obj.operate() r=-2 bl.length=0 2023-10-05T19:28:28.442 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.434+0000 7f1931bace40 20 rados->read ofs=0 len=0 2023-10-05T19:28:28.442 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.438+0000 7f1931bace40 1 -- 172.21.15.167:0/2517686516 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:83 6.b 6:d467b91b:::bucket.sync-target-hints.:head [call version.read in=11b,read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x555f01003f50 con 0x555f00ffe800 2023-10-05T19:28:28.443 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.438+0000 7f192f6e1700 1 -- 172.21.15.167:0/2517686516 <== osd.3 v2:172.21.15.167:6808/30036 21 ==== osd_op_reply(83 bucket.sync-target-hints. [call,read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 ==== 211+0+0 (crc 0 0 0) 0x7f1928012c70 con 0x555f00ffe800 2023-10-05T19:28:28.443 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.438+0000 7f1931bace40 20 rados_obj.operate() r=-2 bl.length=0 2023-10-05T19:28:28.443 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.438+0000 7f1931bace40 20 started sync module instance, tier type = 2023-10-05T19:28:28.444 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.438+0000 7f1931bace40 20 started zone id=9a9df965-6612-45d4-a45f-6e3c889bacc2 (name=default) with tier type = 2023-10-05T19:28:28.444 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.438+0000 7f1931bace40 1 -- 172.21.15.167:0/2517686516 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:84 7.1f 7:f95f44c2:::notify.0:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x555f01003260 con 0x7f19280276e0 2023-10-05T19:28:28.446 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.438+0000 7f192eee0700 1 -- 172.21.15.167:0/2517686516 <== osd.2 v2:172.21.15.167:6816/30037 13 ==== osd_op_reply(84 notify.0 [create] v30'60 uv60 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f1918008ae0 con 0x7f19280276e0 2023-10-05T19:28:28.446 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.438+0000 7f1931bace40 1 -- 172.21.15.167:0/2517686516 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:85 7.1f 7:f95f44c2:::notify.0:head [watch watch cookie 93866527042928] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x555f01004300 con 0x7f19280276e0 2023-10-05T19:28:28.446 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.438+0000 7f1931bace40 1 -- 172.21.15.167:0/2517686516 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:86 7.0 7:05bf5b68:::notify.1:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x555f01004bd0 con 0x555f00f38c20 2023-10-05T19:28:28.449 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.442+0000 7f192fee2700 1 -- 172.21.15.167:0/2517686516 <== osd.1 v2:172.21.15.167:6800/30035 29 ==== osd_op_reply(86 notify.1 [create] v30'60 uv60 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f1920050040 con 0x555f00f38c20 2023-10-05T19:28:28.449 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.442+0000 7f1931bace40 1 -- 172.21.15.167:0/2517686516 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:87 7.0 7:05bf5b68:::notify.1:head [watch watch cookie 93866527049728] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x555f010054c0 con 0x555f00f38c20 2023-10-05T19:28:28.449 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.442+0000 7f1931bace40 1 -- 172.21.15.167:0/2517686516 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:88 7.15 7:a93a5511:::notify.2:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x555f01005d60 con 0x555f00ffe800 2023-10-05T19:28:28.449 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.442+0000 7f192eee0700 1 -- 172.21.15.167:0/2517686516 <== osd.2 v2:172.21.15.167:6816/30037 14 ==== osd_op_reply(85 notify.0 [watch watch cookie 93866527042928] v30'61 uv60 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f1918008ae0 con 0x7f19280276e0 2023-10-05T19:28:28.452 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.446+0000 7f192fee2700 1 -- 172.21.15.167:0/2517686516 <== osd.1 v2:172.21.15.167:6800/30035 30 ==== osd_op_reply(87 notify.1 [watch watch cookie 93866527049728] v30'61 uv60 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f1920050040 con 0x555f00f38c20 2023-10-05T19:28:28.452 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.446+0000 7f192f6e1700 1 -- 172.21.15.167:0/2517686516 <== osd.3 v2:172.21.15.167:6808/30036 22 ==== osd_op_reply(88 notify.2 [create] v30'60 uv60 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f1928012c70 con 0x555f00ffe800 2023-10-05T19:28:28.452 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.446+0000 7f1931bace40 1 -- 172.21.15.167:0/2517686516 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:89 7.15 7:a93a5511:::notify.2:head [watch watch cookie 93866527051728] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x555f01003780 con 0x555f00ffe800 2023-10-05T19:28:28.453 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.446+0000 7f1931bace40 1 -- 172.21.15.167:0/2517686516 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:90 7.e 7:7759931f:::notify.3:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x555f01006600 con 0x7f19280276e0 2023-10-05T19:28:28.455 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.450+0000 7f192eee0700 1 -- 172.21.15.167:0/2517686516 <== osd.2 v2:172.21.15.167:6816/30037 15 ==== osd_op_reply(90 notify.3 [create] v30'60 uv60 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f1918008ae0 con 0x7f19280276e0 2023-10-05T19:28:28.455 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.450+0000 7f1931bace40 1 -- 172.21.15.167:0/2517686516 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:91 7.e 7:7759931f:::notify.3:head [watch watch cookie 93866527058352] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x555f01006230 con 0x7f19280276e0 2023-10-05T19:28:28.455 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.450+0000 7f1931bace40 1 -- 172.21.15.167:0/2517686516 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:92 7.d 7:b4812045:::notify.4:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x555f01007b00 con 0x555f00f38c20 2023-10-05T19:28:28.455 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.450+0000 7f192f6e1700 1 -- 172.21.15.167:0/2517686516 <== osd.3 v2:172.21.15.167:6808/30036 23 ==== osd_op_reply(89 notify.2 [watch watch cookie 93866527051728] v30'61 uv60 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f1928012c70 con 0x555f00ffe800 2023-10-05T19:28:28.457 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.450+0000 7f192fee2700 1 -- 172.21.15.167:0/2517686516 <== osd.1 v2:172.21.15.167:6800/30035 31 ==== osd_op_reply(92 notify.4 [create] v30'60 uv60 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f1920050040 con 0x555f00f38c20 2023-10-05T19:28:28.457 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.450+0000 7f192eee0700 1 -- 172.21.15.167:0/2517686516 <== osd.2 v2:172.21.15.167:6816/30037 16 ==== osd_op_reply(91 notify.3 [watch watch cookie 93866527058352] v30'61 uv60 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f1918008ae0 con 0x7f19280276e0 2023-10-05T19:28:28.457 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.450+0000 7f1931bace40 1 -- 172.21.15.167:0/2517686516 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:93 7.d 7:b4812045:::notify.4:head [watch watch cookie 93866527052832] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x555f01007f00 con 0x555f00f38c20 2023-10-05T19:28:28.457 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.450+0000 7f1931bace40 1 -- 172.21.15.167:0/2517686516 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:94 7.3 7:c609908c:::notify.5:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x555f01008910 con 0x7f1918007030 2023-10-05T19:28:28.459 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.454+0000 7f192fee2700 1 -- 172.21.15.167:0/2517686516 <== osd.1 v2:172.21.15.167:6800/30035 32 ==== osd_op_reply(93 notify.4 [watch watch cookie 93866527052832] v30'61 uv60 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f1920050040 con 0x555f00f38c20 2023-10-05T19:28:28.460 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.454+0000 7f192fee2700 1 -- 172.21.15.167:0/2517686516 <== osd.0 v2:172.21.15.118:6800/27690 23 ==== osd_op_reply(94 notify.5 [create] v30'60 uv60 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f1920050040 con 0x7f1918007030 2023-10-05T19:28:28.460 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.454+0000 7f1931bace40 1 -- 172.21.15.167:0/2517686516 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:95 7.3 7:c609908c:::notify.5:head [watch watch cookie 93866527059456] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x555f01008420 con 0x7f1918007030 2023-10-05T19:28:28.460 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.454+0000 7f1931bace40 1 -- 172.21.15.167:0/2517686516 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:96 7.14 7:2b04a3e9:::notify.6:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x555f01008cc0 con 0x7f1918007030 2023-10-05T19:28:28.462 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.458+0000 7f192fee2700 1 -- 172.21.15.167:0/2517686516 <== osd.0 v2:172.21.15.118:6800/27690 24 ==== osd_op_reply(95 notify.5 [watch watch cookie 93866527059456] v30'61 uv60 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f1920050040 con 0x7f1918007030 2023-10-05T19:28:28.463 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.458+0000 7f192fee2700 1 -- 172.21.15.167:0/2517686516 <== osd.0 v2:172.21.15.118:6800/27690 25 ==== osd_op_reply(96 notify.6 [create] v30'60 uv60 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f1920050040 con 0x7f1918007030 2023-10-05T19:28:28.463 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.458+0000 7f1931bace40 1 -- 172.21.15.167:0/2517686516 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:97 7.14 7:2b04a3e9:::notify.6:head [watch watch cookie 93866527065120] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x555f01008d60 con 0x7f1918007030 2023-10-05T19:28:28.463 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.458+0000 7f1931bace40 1 -- 172.21.15.167:0/2517686516 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:98 7.9 7:93e5b521:::notify.7:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x555f01009600 con 0x7f1918007030 2023-10-05T19:28:28.466 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.458+0000 7f192fee2700 1 -- 172.21.15.167:0/2517686516 <== osd.0 v2:172.21.15.118:6800/27690 26 ==== osd_op_reply(98 notify.7 [create] v30'60 uv60 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f1920050040 con 0x7f1918007030 2023-10-05T19:28:28.466 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.458+0000 7f1931bace40 1 -- 172.21.15.167:0/2517686516 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:99 7.9 7:93e5b521:::notify.7:head [watch watch cookie 93866527068048] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x555f01009990 con 0x7f1918007030 2023-10-05T19:28:28.466 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.458+0000 7f1931bace40 20 add_watcher() i=0 2023-10-05T19:28:28.466 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.458+0000 7f1931bace40 20 add_watcher() i=1 2023-10-05T19:28:28.466 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.458+0000 7f1931bace40 20 add_watcher() i=2 2023-10-05T19:28:28.466 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.458+0000 7f1931bace40 20 add_watcher() i=3 2023-10-05T19:28:28.467 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.458+0000 7f1931bace40 20 add_watcher() i=4 2023-10-05T19:28:28.467 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.458+0000 7f1931bace40 20 add_watcher() i=5 2023-10-05T19:28:28.467 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.462+0000 7f192fee2700 1 -- 172.21.15.167:0/2517686516 <== osd.0 v2:172.21.15.118:6800/27690 27 ==== osd_op_reply(97 notify.6 [watch watch cookie 93866527065120] v30'61 uv60 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f1920050040 con 0x7f1918007030 2023-10-05T19:28:28.467 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.462+0000 7f1931bace40 20 add_watcher() i=6 2023-10-05T19:28:28.468 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.462+0000 7f192fee2700 1 -- 172.21.15.167:0/2517686516 <== osd.0 v2:172.21.15.118:6800/27690 28 ==== osd_op_reply(99 notify.7 [watch watch cookie 93866527068048] v30'61 uv60 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f1920050040 con 0x7f1918007030 2023-10-05T19:28:28.468 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.462+0000 7f1931bace40 20 add_watcher() i=7 2023-10-05T19:28:28.469 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.462+0000 7f1931bace40 2 all 8 watchers are set, enabling cache 2023-10-05T19:28:28.469 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.462+0000 7f1931bace40 1 -- 172.21.15.167:0/2517686516 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:100 6.4 6:22d26bf9:::data_loggenerations_metadata:head [call version.check_conds in=74b,call version.read in=11b,read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x555f01008870 con 0x555f00f38c20 2023-10-05T19:28:28.469 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.462+0000 7f192fee2700 1 -- 172.21.15.167:0/2517686516 <== osd.1 v2:172.21.15.167:6800/30035 33 ==== osd_op_reply(100 data_loggenerations_metadata [call,call out=48b,read 0~28 out=28b] v0'0 uv1 ondisk = 0) v8 ==== 256+0+76 (crc 0 0 0) 0x7f1920050040 con 0x555f00f38c20 2023-10-05T19:28:28.469 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.462+0000 7f1931bace40 1 -- 172.21.15.167:0/2517686516 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:101 6.4 6:22d26bf9:::data_loggenerations_metadata:head [watch watch cookie 93866527069280] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x555f01009eb0 con 0x555f00f38c20 2023-10-05T19:28:28.471 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.466+0000 7f192fee2700 1 -- 172.21.15.167:0/2517686516 <== osd.1 v2:172.21.15.167:6800/30035 34 ==== osd_op_reply(101 data_loggenerations_metadata [watch watch cookie 93866527069280] v30'74 uv1 ondisk = 0) v8 ==== 172+0+0 (crc 0 0 0) 0x7f1920050040 con 0x555f00f38c20 2023-10-05T19:28:28.471 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.466+0000 7f1900fe1700 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2023-10-05T19:28:28.471 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.466+0000 7f1931bace40 20 check_secure_mon_conn(): auth registy supported: methods=[2] modes=[2,1] 2023-10-05T19:28:28.471 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.466+0000 7f1931bace40 20 check_secure_mon_conn(): mode 1 is insecure 2023-10-05T19:28:28.472 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.466+0000 7f1931bace40 1 Processor -- start 2023-10-05T19:28:28.472 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.466+0000 7f1931bace40 1 -- start start 2023-10-05T19:28:28.472 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.466+0000 7f1931bace40 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x555f01016d70 0x555f01017140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-05T19:28:28.472 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.466+0000 7f1931bace40 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x555f01017680 con 0x555f01016d70 2023-10-05T19:28:28.473 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.466+0000 7f192eee0700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x555f01016d70 0x555f01017140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-05T19:28:28.473 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.466+0000 7f192eee0700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x555f01016d70 0x555f01017140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.118:3300/0 says I am v2:172.21.15.167:57712/0 (socket says 172.21.15.167:57712) 2023-10-05T19:28:28.473 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.466+0000 7f192eee0700 1 -- 172.21.15.167:0/1758860885 learned_addr learned my addr 172.21.15.167:0/1758860885 (peer_addr_for_me v2:172.21.15.167:0/0) 2023-10-05T19:28:28.474 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.466+0000 7f192eee0700 1 -- 172.21.15.167:0/1758860885 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x555f010177c0 con 0x555f01016d70 2023-10-05T19:28:28.474 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.466+0000 7f192eee0700 1 --2- 172.21.15.167:0/1758860885 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x555f01016d70 0x555f01017140 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 crypto rx=0x555f00f39080 tx=0x7f1918012000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bd5bb185285b7084 server_cookie=0 in_seq=0 out_seq=0 2023-10-05T19:28:28.474 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.470+0000 7f187bfff700 1 -- 172.21.15.167:0/1758860885 <== mon.0 v2:172.21.15.118:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f1918012c70 con 0x555f01016d70 2023-10-05T19:28:28.474 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.470+0000 7f187bfff700 1 -- 172.21.15.167:0/1758860885 <== mon.0 v2:172.21.15.118:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1918012dd0 con 0x555f01016d70 2023-10-05T19:28:28.474 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.470+0000 7f187bfff700 1 -- 172.21.15.167:0/1758860885 <== mon.0 v2:172.21.15.118:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f191802a8a0 con 0x555f01016d70 2023-10-05T19:28:28.475 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.470+0000 7f1931bace40 1 -- 172.21.15.167:0/1758860885 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x555f01016d70 msgr2=0x555f01017140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-05T19:28:28.475 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.470+0000 7f1931bace40 1 --2- 172.21.15.167:0/1758860885 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x555f01016d70 0x555f01017140 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 crypto rx=0x555f00f39080 tx=0x7f1918012000 comp rx=0 tx=0).stop 2023-10-05T19:28:28.476 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.470+0000 7f1931bace40 1 -- 172.21.15.167:0/1758860885 shutdown_connections 2023-10-05T19:28:28.476 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.470+0000 7f1931bace40 1 --2- 172.21.15.167:0/1758860885 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x555f01016d70 0x555f01017140 unknown :-1 s=CLOSED pgs=258 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-05T19:28:28.476 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.470+0000 7f1931bace40 1 -- 172.21.15.167:0/1758860885 >> 172.21.15.167:0/1758860885 conn(0x555f01012130 msgr2=0x555f010102e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-05T19:28:28.476 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.470+0000 7f1931bace40 1 -- 172.21.15.167:0/1758860885 shutdown_connections 2023-10-05T19:28:28.476 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.470+0000 7f1931bace40 1 -- 172.21.15.167:0/1758860885 wait complete. 2023-10-05T19:28:28.477 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.470+0000 7f1931bace40 1 Processor -- start 2023-10-05T19:28:28.477 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.470+0000 7f1931bace40 1 -- start start 2023-10-05T19:28:28.477 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.470+0000 7f1931bace40 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x555f01016d70 0x555f0101c8a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-05T19:28:28.477 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.470+0000 7f1931bace40 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x555f0101d8a0 con 0x555f01016d70 2023-10-05T19:28:28.477 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.470+0000 7f192eee0700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x555f01016d70 0x555f0101c8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-05T19:28:28.477 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.470+0000 7f192eee0700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x555f01016d70 0x555f0101c8a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.118:3300/0 says I am v2:172.21.15.167:57718/0 (socket says 172.21.15.167:57718) 2023-10-05T19:28:28.477 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.470+0000 7f192eee0700 1 -- 172.21.15.167:0/3092055014 learned_addr learned my addr 172.21.15.167:0/3092055014 (peer_addr_for_me v2:172.21.15.167:0/0) 2023-10-05T19:28:28.478 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.470+0000 7f192eee0700 1 -- 172.21.15.167:0/3092055014 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x555f00f390b0 con 0x555f01016d70 2023-10-05T19:28:28.478 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.474+0000 7f192eee0700 1 --2- 172.21.15.167:0/3092055014 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x555f01016d70 0x555f0101c8a0 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 crypto rx=0x555f01010e50 tx=0x7f191802f000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-05T19:28:28.479 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.474+0000 7f187a7fc700 1 -- 172.21.15.167:0/3092055014 <== mon.0 v2:172.21.15.118:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f191803e910 con 0x555f01016d70 2023-10-05T19:28:28.479 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.474+0000 7f187a7fc700 1 -- 172.21.15.167:0/3092055014 <== mon.0 v2:172.21.15.118:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f191803eab0 con 0x555f01016d70 2023-10-05T19:28:28.479 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.474+0000 7f1931bace40 1 -- 172.21.15.167:0/3092055014 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x555f0101cea0 con 0x555f01016d70 2023-10-05T19:28:28.479 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.474+0000 7f187a7fc700 1 -- 172.21.15.167:0/3092055014 <== mon.0 v2:172.21.15.118:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f191803ec60 con 0x555f01016d70 2023-10-05T19:28:28.479 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.474+0000 7f1931bace40 1 -- 172.21.15.167:0/3092055014 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x555f0101d340 con 0x555f01016d70 2023-10-05T19:28:28.479 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.474+0000 7f187a7fc700 1 -- 172.21.15.167:0/3092055014 <== mon.0 v2:172.21.15.118:3300/0 4 ==== mgrmap(e 5) v1 ==== 40329+0+0 (secure 0 0 0) 0x7f1918046070 con 0x555f01016d70 2023-10-05T19:28:28.480 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.474+0000 7f187a7fc700 1 --2- 172.21.15.167:0/3092055014 >> [v2:172.21.15.167:6824/29889,v1:172.21.15.167:6825/29889] conn(0x7f1870030c50 0x7f1870033100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-05T19:28:28.480 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.474+0000 7f187a7fc700 1 -- 172.21.15.167:0/3092055014 <== mon.0 v2:172.21.15.118:3300/0 5 ==== osd_map(30..30 src has 1..30) v4 ==== 6057+0+0 (secure 0 0 0) 0x7f191802f070 con 0x555f01016d70 2023-10-05T19:28:28.480 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.474+0000 7f1931bace40 5 note: GC not initialized 2023-10-05T19:28:28.481 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.474+0000 7f192fee2700 1 --2- 172.21.15.167:0/3092055014 >> [v2:172.21.15.167:6824/29889,v1:172.21.15.167:6825/29889] conn(0x7f1870030c50 0x7f1870033100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-05T19:28:28.481 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.474+0000 7f192fee2700 1 --2- 172.21.15.167:0/3092055014 >> [v2:172.21.15.167:6824/29889,v1:172.21.15.167:6825/29889] conn(0x7f1870030c50 0x7f1870033100 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x555f01008250 tx=0x7f1920000f90 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-05T19:28:28.483 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.478+0000 7f186f7fe700 20 reqs_thread_entry: start 2023-10-05T19:28:28.483 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.478+0000 7f186d7fa700 20 reqs_thread_entry: start 2023-10-05T19:28:28.484 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.478+0000 7f1931bace40 20 init_complete bucket index max shards: 11 2023-10-05T19:28:28.484 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.478+0000 7f1931bace40 10 rgw notify: Started notification manager with: 1 workers 2023-10-05T19:28:28.484 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.478+0000 7f185b7fe700 20 rgw notify: INFO: next queues processing will happen at: Thu Oct 5 19:28:58 2023 2023-10-05T19:28:28.485 INFO:teuthology.orchestra.run.smithi167.stderr: 2023-10-05T19:28:28.485 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.478+0000 7f185affd700 20 reqs_thread_entry: start 2023-10-05T19:28:28.485 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.478+0000 7f1931bace40 20 get_system_obj_state: rctx=0x7ffc19be9c48 obj=default.rgw.meta:users.uid:foo state=0x555f00e9c0b0 s->prefetch_data=0 2023-10-05T19:28:28.485 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.478+0000 7f1931bace40 1 -- 172.21.15.167:0/2517686516 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:102 8.18 8:1e6ca5ed:users.uid::foo:head [call version.read in=11b,getxattrs,stat] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x555f01085010 con 0x555f00f38c20 2023-10-05T19:28:28.486 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.478+0000 7f192fee2700 1 -- 172.21.15.167:0/2517686516 <== osd.1 v2:172.21.15.167:6800/30035 35 ==== osd_op_reply(102 foo [call out=48b,getxattrs out=75b,stat out=16b] v0'0 uv7 ondisk = 0) v8 ==== 231+0+139 (crc 0 0 0) 0x7f1920050040 con 0x555f00f38c20 2023-10-05T19:28:28.486 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.478+0000 7f1931bace40 20 get_system_obj_state: s->obj_tag was set empty 2023-10-05T19:28:28.486 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.478+0000 7f1931bace40 20 Read xattr: user.rgw.idtag 2023-10-05T19:28:28.487 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.478+0000 7f1931bace40 20 rados->read ofs=0 len=0 2023-10-05T19:28:28.487 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.478+0000 7f1931bace40 1 -- 172.21.15.167:0/2517686516 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:103 8.18 8:1e6ca5ed:users.uid::foo:head [call version.check_conds in=122b,call version.read in=11b,read 0~0,getxattrs] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x555f01087aa0 con 0x555f00f38c20 2023-10-05T19:28:28.487 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.482+0000 7f192fee2700 1 -- 172.21.15.167:0/2517686516 <== osd.1 v2:172.21.15.167:6800/30035 36 ==== osd_op_reply(103 foo [call,call out=48b,read 0~513 out=513b,getxattrs out=75b] v0'0 uv7 ondisk = 0) v8 ==== 273+0+636 (crc 0 0 0) 0x7f1920050040 con 0x555f00f38c20 2023-10-05T19:28:28.487 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.482+0000 7f1931bace40 20 rados_obj.operate() r=0 bl.length=513 2023-10-05T19:28:28.487 INFO:teuthology.orchestra.run.smithi167.stdout:{ 2023-10-05T19:28:28.487 INFO:teuthology.orchestra.run.smithi167.stdout: "user_id": "foo", 2023-10-05T19:28:28.487 INFO:teuthology.orchestra.run.smithi167.stdout: "display_name": "Foo", 2023-10-05T19:28:28.487 INFO:teuthology.orchestra.run.smithi167.stdout: "email": "foo@foo.com", 2023-10-05T19:28:28.488 INFO:teuthology.orchestra.run.smithi167.stdout: "suspended": 0, 2023-10-05T19:28:28.488 INFO:teuthology.orchestra.run.smithi167.stdout: "max_buckets": 4, 2023-10-05T19:28:28.488 INFO:teuthology.orchestra.run.smithi167.stdout: "subusers": [ 2023-10-05T19:28:28.488 INFO:teuthology.orchestra.run.smithi167.stdout: { 2023-10-05T19:28:28.488 INFO:teuthology.orchestra.run.smithi167.stdout: "id": "foo:foo1", 2023-10-05T19:28:28.488 INFO:teuthology.orchestra.run.smithi167.stdout: "permissions": "full-control" 2023-10-05T19:28:28.488 INFO:teuthology.orchestra.run.smithi167.stdout: } 2023-10-05T19:28:28.488 INFO:teuthology.orchestra.run.smithi167.stdout: ], 2023-10-05T19:28:28.488 INFO:teuthology.orchestra.run.smithi167.stdout: "keys": [ 2023-10-05T19:28:28.488 INFO:teuthology.orchestra.run.smithi167.stdout: { 2023-10-05T19:28:28.488 INFO:teuthology.orchestra.run.smithi167.stdout: "user": "foo", 2023-10-05T19:28:28.488 INFO:teuthology.orchestra.run.smithi167.stdout: "access_key": "9te6NH5mcdcq0Tc5i8i1", 2023-10-05T19:28:28.489 INFO:teuthology.orchestra.run.smithi167.stdout: "secret_key": "Ny4IOauQoL18Gp2zM7lC1vLmoawgqcYP/YGcWfXu" 2023-10-05T19:28:28.489 INFO:teuthology.orchestra.run.smithi167.stdout: } 2023-10-05T19:28:28.489 INFO:teuthology.orchestra.run.smithi167.stdout: ], 2023-10-05T19:28:28.489 INFO:teuthology.orchestra.run.smithi167.stdout: "swift_keys": [ 2023-10-05T19:28:28.489 INFO:teuthology.orchestra.run.smithi167.stdout: { 2023-10-05T19:28:28.489 INFO:teuthology.orchestra.run.smithi167.stdout: "user": "foo:foo1", 2023-10-05T19:28:28.489 INFO:teuthology.orchestra.run.smithi167.stdout: "secret_key": "gpS2G9RREMrnbqlp29PP2D36kgPR1tm72n5fPYfL" 2023-10-05T19:28:28.489 INFO:teuthology.orchestra.run.smithi167.stdout: } 2023-10-05T19:28:28.489 INFO:teuthology.orchestra.run.smithi167.stdout: ], 2023-10-05T19:28:28.489 INFO:teuthology.orchestra.run.smithi167.stdout: "caps": [], 2023-10-05T19:28:28.489 INFO:teuthology.orchestra.run.smithi167.stdout: "op_mask": "read, write, delete", 2023-10-05T19:28:28.489 INFO:teuthology.orchestra.run.smithi167.stdout: "default_placement": "", 2023-10-05T19:28:28.490 INFO:teuthology.orchestra.run.smithi167.stdout: "default_storage_class": "", 2023-10-05T19:28:28.490 INFO:teuthology.orchestra.run.smithi167.stdout: "placement_tags": [], 2023-10-05T19:28:28.490 INFO:teuthology.orchestra.run.smithi167.stdout: "bucket_quota": { 2023-10-05T19:28:28.490 INFO:teuthology.orchestra.run.smithi167.stdout: "enabled": false, 2023-10-05T19:28:28.490 INFO:teuthology.orchestra.run.smithi167.stdout: "check_on_raw": false, 2023-10-05T19:28:28.490 INFO:teuthology.orchestra.run.smithi167.stdout: "max_size": -1, 2023-10-05T19:28:28.490 INFO:teuthology.orchestra.run.smithi167.stdout: "max_size_kb": 0, 2023-10-05T19:28:28.490 INFO:teuthology.orchestra.run.smithi167.stdout: "max_objects": -1 2023-10-05T19:28:28.490 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2023-10-05T19:28:28.491 INFO:teuthology.orchestra.run.smithi167.stdout: "user_quota": { 2023-10-05T19:28:28.491 INFO:teuthology.orchestra.run.smithi167.stdout: "enabled": false, 2023-10-05T19:28:28.491 INFO:teuthology.orchestra.run.smithi167.stdout: "check_on_raw": false, 2023-10-05T19:28:28.491 INFO:teuthology.orchestra.run.smithi167.stdout: "max_size": -1, 2023-10-05T19:28:28.491 INFO:teuthology.orchestra.run.smithi167.stdout: "max_size_kb": 0, 2023-10-05T19:28:28.491 INFO:teuthology.orchestra.run.smithi167.stdout: "max_objects": -1 2023-10-05T19:28:28.491 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2023-10-05T19:28:28.491 INFO:teuthology.orchestra.run.smithi167.stdout: "temp_url_keys": [], 2023-10-05T19:28:28.491 INFO:teuthology.orchestra.run.smithi167.stdout: "type": "rgw", 2023-10-05T19:28:28.491 INFO:teuthology.orchestra.run.smithi167.stdout: "mfa_ids": [] 2023-10-05T19:28:28.491 INFO:teuthology.orchestra.run.smithi167.stdout:} 2023-10-05T19:28:28.491 INFO:teuthology.orchestra.run.smithi167.stdout: 2023-10-05T19:28:28.492 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.482+0000 7f1931bace40 1 -- 172.21.15.167:0/2517686516 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:104 7.1f 7:f95f44c2:::notify.0:head [watch unwatch cookie 93866527042928] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x555f00fdeda0 con 0x7f19280276e0 2023-10-05T19:28:28.492 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.482+0000 7f192eee0700 1 -- 172.21.15.167:0/2517686516 <== osd.2 v2:172.21.15.167:6816/30037 17 ==== osd_op_reply(104 notify.0 [watch unwatch cookie 93866527042928] v30'62 uv60 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f1918032040 con 0x7f19280276e0 2023-10-05T19:28:28.493 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.482+0000 7f1931bace40 20 remove_watcher() i=0 2023-10-05T19:28:28.493 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.482+0000 7f1931bace40 2 removed watcher, disabling cache 2023-10-05T19:28:28.493 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.482+0000 7f1931bace40 1 -- 172.21.15.167:0/2517686516 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:105 7.0 7:05bf5b68:::notify.1:head [watch unwatch cookie 93866527049728] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x555f00fdeda0 con 0x555f00f38c20 2023-10-05T19:28:28.493 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.486+0000 7f192fee2700 1 -- 172.21.15.167:0/2517686516 <== osd.1 v2:172.21.15.167:6800/30035 37 ==== osd_op_reply(105 notify.1 [watch unwatch cookie 93866527049728] v30'62 uv60 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f1920050040 con 0x555f00f38c20 2023-10-05T19:28:28.493 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.486+0000 7f1931bace40 20 remove_watcher() i=1 2023-10-05T19:28:28.493 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.486+0000 7f1931bace40 1 -- 172.21.15.167:0/2517686516 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:106 7.15 7:a93a5511:::notify.2:head [watch unwatch cookie 93866527051728] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x555f00fdeda0 con 0x555f00ffe800 2023-10-05T19:28:28.494 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.490+0000 7f192f6e1700 1 -- 172.21.15.167:0/2517686516 <== osd.3 v2:172.21.15.167:6808/30036 24 ==== osd_op_reply(106 notify.2 [watch unwatch cookie 93866527051728] v30'62 uv60 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f1928012c70 con 0x555f00ffe800 2023-10-05T19:28:28.494 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.490+0000 7f1931bace40 20 remove_watcher() i=2 2023-10-05T19:28:28.494 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.490+0000 7f1931bace40 1 -- 172.21.15.167:0/2517686516 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:107 7.e 7:7759931f:::notify.3:head [watch unwatch cookie 93866527058352] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x555f01004fd0 con 0x7f19280276e0 2023-10-05T19:28:28.496 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.490+0000 7f192eee0700 1 -- 172.21.15.167:0/2517686516 <== osd.2 v2:172.21.15.167:6816/30037 18 ==== osd_op_reply(107 notify.3 [watch unwatch cookie 93866527058352] v30'62 uv60 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f1918032040 con 0x7f19280276e0 2023-10-05T19:28:28.496 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.490+0000 7f1931bace40 20 remove_watcher() i=3 2023-10-05T19:28:28.497 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.490+0000 7f1931bace40 1 -- 172.21.15.167:0/2517686516 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:108 7.d 7:b4812045:::notify.4:head [watch unwatch cookie 93866527052832] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x555f010069b0 con 0x555f00f38c20 2023-10-05T19:28:28.499 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.494+0000 7f192fee2700 1 -- 172.21.15.167:0/2517686516 <== osd.1 v2:172.21.15.167:6800/30035 38 ==== osd_op_reply(108 notify.4 [watch unwatch cookie 93866527052832] v30'62 uv60 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f1920050040 con 0x555f00f38c20 2023-10-05T19:28:28.500 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.494+0000 7f1931bace40 20 remove_watcher() i=4 2023-10-05T19:28:28.500 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.494+0000 7f1931bace40 1 -- 172.21.15.167:0/2517686516 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:109 7.3 7:c609908c:::notify.5:head [watch unwatch cookie 93866527059456] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x555f010069b0 con 0x7f1918007030 2023-10-05T19:28:28.503 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.498+0000 7f192fee2700 1 -- 172.21.15.167:0/2517686516 <== osd.0 v2:172.21.15.118:6800/27690 29 ==== osd_op_reply(109 notify.5 [watch unwatch cookie 93866527059456] v30'62 uv60 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f1920050040 con 0x7f1918007030 2023-10-05T19:28:28.503 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.498+0000 7f1931bace40 20 remove_watcher() i=5 2023-10-05T19:28:28.504 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.498+0000 7f1931bace40 1 -- 172.21.15.167:0/2517686516 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:110 7.14 7:2b04a3e9:::notify.6:head [watch unwatch cookie 93866527065120] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x555f010069b0 con 0x7f1918007030 2023-10-05T19:28:28.505 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.498+0000 7f192fee2700 1 -- 172.21.15.167:0/2517686516 <== osd.0 v2:172.21.15.118:6800/27690 30 ==== osd_op_reply(110 notify.6 [watch unwatch cookie 93866527065120] v30'62 uv60 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f1920050040 con 0x7f1918007030 2023-10-05T19:28:28.506 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.498+0000 7f1931bace40 20 remove_watcher() i=6 2023-10-05T19:28:28.506 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.498+0000 7f1931bace40 1 -- 172.21.15.167:0/2517686516 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:111 7.9 7:93e5b521:::notify.7:head [watch unwatch cookie 93866527068048] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x555f010069b0 con 0x7f1918007030 2023-10-05T19:28:28.508 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.502+0000 7f192fee2700 1 -- 172.21.15.167:0/2517686516 <== osd.0 v2:172.21.15.118:6800/27690 31 ==== osd_op_reply(111 notify.7 [watch unwatch cookie 93866527068048] v30'62 uv60 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f1920050040 con 0x7f1918007030 2023-10-05T19:28:28.508 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.502+0000 7f1931bace40 20 remove_watcher() i=7 2023-10-05T19:28:28.510 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.502+0000 7f1900fe1700 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2023-10-05T19:28:28.510 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.506+0000 7f1931bace40 1 -- 172.21.15.167:0/2517686516 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:112 6.4 6:22d26bf9:::data_loggenerations_metadata:head [watch unwatch cookie 93866527069280] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x555f010069b0 con 0x555f00f38c20 2023-10-05T19:28:28.513 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.506+0000 7f192fee2700 1 -- 172.21.15.167:0/2517686516 <== osd.1 v2:172.21.15.167:6800/30035 39 ==== osd_op_reply(112 data_loggenerations_metadata [watch unwatch cookie 93866527069280] v30'75 uv1 ondisk = 0) v8 ==== 172+0+0 (crc 0 0 0) 0x7f1920050040 con 0x555f00f38c20 2023-10-05T19:28:28.514 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.506+0000 7f1931bace40 1 -- 172.21.15.167:0/2517686516 >> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] conn(0x7f1918007030 msgr2=0x7f191801c920 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-05T19:28:28.514 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.506+0000 7f1931bace40 1 --2- 172.21.15.167:0/2517686516 >> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] conn(0x7f1918007030 0x7f191801c920 crc :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-05T19:28:28.514 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.506+0000 7f1931bace40 1 -- 172.21.15.167:0/2517686516 >> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] conn(0x555f00f38c20 msgr2=0x555f00ffad60 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-05T19:28:28.514 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.506+0000 7f1931bace40 1 --2- 172.21.15.167:0/2517686516 >> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] conn(0x555f00f38c20 0x555f00ffad60 crc :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-05T19:28:28.514 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.506+0000 7f1931bace40 1 -- 172.21.15.167:0/2517686516 >> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] conn(0x7f19280276e0 msgr2=0x7f19280441a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-05T19:28:28.514 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.506+0000 7f1931bace40 1 --2- 172.21.15.167:0/2517686516 >> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] conn(0x7f19280276e0 0x7f19280441a0 crc :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-05T19:28:28.514 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.506+0000 7f1931bace40 1 -- 172.21.15.167:0/2517686516 >> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] conn(0x555f00ffe800 msgr2=0x555f01000be0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-05T19:28:28.514 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.506+0000 7f1931bace40 1 --2- 172.21.15.167:0/2517686516 >> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] conn(0x555f00ffe800 0x555f01000be0 crc :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-05T19:28:28.514 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.506+0000 7f1931bace40 1 -- 172.21.15.167:0/2517686516 >> [v2:172.21.15.167:6824/29889,v1:172.21.15.167:6825/29889] conn(0x7f1910030c00 msgr2=0x7f19100330b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-05T19:28:28.514 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.506+0000 7f1931bace40 1 --2- 172.21.15.167:0/2517686516 >> [v2:172.21.15.167:6824/29889,v1:172.21.15.167:6825/29889] conn(0x7f1910030c00 0x7f19100330b0 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7f1918005a90 tx=0x7f1918015000 comp rx=0 tx=0).stop 2023-10-05T19:28:28.515 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.506+0000 7f192fee2700 1 -- 172.21.15.167:0/2517686516 reap_dead start 2023-10-05T19:28:28.515 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.506+0000 7f1931bace40 1 -- 172.21.15.167:0/2517686516 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x555f00f37910 msgr2=0x555f00fef2d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-05T19:28:28.515 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.506+0000 7f1931bace40 1 --2- 172.21.15.167:0/2517686516 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x555f00f37910 0x555f00fef2d0 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7f1928006bd0 tx=0x7f1928014000 comp rx=0 tx=0).stop 2023-10-05T19:28:28.515 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.506+0000 7f1931bace40 1 -- 172.21.15.167:0/2517686516 shutdown_connections 2023-10-05T19:28:28.515 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.506+0000 7f1931bace40 1 --2- 172.21.15.167:0/2517686516 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x555f00f37910 0x555f00fef2d0 unknown :-1 s=CLOSED pgs=257 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-05T19:28:28.515 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.506+0000 7f1931bace40 1 -- 172.21.15.167:0/2517686516 >> 172.21.15.167:0/2517686516 conn(0x555f00fbe850 msgr2=0x555f00fe2b40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-05T19:28:28.515 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.506+0000 7f1931bace40 1 -- 172.21.15.167:0/2517686516 shutdown_connections 2023-10-05T19:28:28.515 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.506+0000 7f1931bace40 1 -- 172.21.15.167:0/2517686516 wait complete. 2023-10-05T19:28:28.516 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.510+0000 7f1931bace40 1 -- 172.21.15.167:0/3092055014 >> [v2:172.21.15.167:6824/29889,v1:172.21.15.167:6825/29889] conn(0x7f1870030c50 msgr2=0x7f1870033100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-05T19:28:28.516 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.510+0000 7f1931bace40 1 --2- 172.21.15.167:0/3092055014 >> [v2:172.21.15.167:6824/29889,v1:172.21.15.167:6825/29889] conn(0x7f1870030c50 0x7f1870033100 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x555f01008250 tx=0x7f1920000f90 comp rx=0 tx=0).stop 2023-10-05T19:28:28.516 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.510+0000 7f1931bace40 1 -- 172.21.15.167:0/3092055014 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x555f01016d70 msgr2=0x555f0101c8a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-05T19:28:28.516 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.510+0000 7f1931bace40 1 --2- 172.21.15.167:0/3092055014 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x555f01016d70 0x555f0101c8a0 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 crypto rx=0x555f01010e50 tx=0x7f191802f000 comp rx=0 tx=0).stop 2023-10-05T19:28:28.516 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.510+0000 7f1931bace40 1 -- 172.21.15.167:0/3092055014 shutdown_connections 2023-10-05T19:28:28.517 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.510+0000 7f1931bace40 1 --2- 172.21.15.167:0/3092055014 >> [v2:172.21.15.167:6824/29889,v1:172.21.15.167:6825/29889] conn(0x7f1870030c50 0x7f1870033100 unknown :-1 s=CLOSED pgs=93 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-05T19:28:28.517 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.510+0000 7f1931bace40 1 --2- 172.21.15.167:0/3092055014 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x555f01016d70 0x555f0101c8a0 unknown :-1 s=CLOSED pgs=259 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-05T19:28:28.517 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.510+0000 7f1931bace40 1 -- 172.21.15.167:0/3092055014 >> 172.21.15.167:0/3092055014 conn(0x555f01012130 msgr2=0x555f01010230 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-05T19:28:28.517 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.510+0000 7f1931bace40 1 -- 172.21.15.167:0/3092055014 shutdown_connections 2023-10-05T19:28:28.517 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.510+0000 7f1931bace40 1 -- 172.21.15.167:0/3092055014 wait complete. 2023-10-05T19:28:28.520 DEBUG:tasks.util.rgw: json result: {'user_id': 'foo', 'display_name': 'Foo', 'email': 'foo@foo.com', 'suspended': 0, 'max_buckets': 4, 'subusers': [{'id': 'foo:foo1', 'permissions': 'full-control'}], 'keys': [{'user': 'foo', 'access_key': '9te6NH5mcdcq0Tc5i8i1', 'secret_key': 'Ny4IOauQoL18Gp2zM7lC1vLmoawgqcYP/YGcWfXu'}], 'swift_keys': [{'user': 'foo:foo1', 'secret_key': 'gpS2G9RREMrnbqlp29PP2D36kgPR1tm72n5fPYfL'}], 'caps': [], 'op_mask': 'read, write, delete', 'default_placement': '', 'default_storage_class': '', 'placement_tags': [], 'bucket_quota': {'enabled': False, 'check_on_raw': False, 'max_size': -1, 'max_size_kb': 0, 'max_objects': -1}, 'user_quota': {'enabled': False, 'check_on_raw': False, 'max_size': -1, 'max_size_kb': 0, 'max_objects': -1}, 'temp_url_keys': [], 'type': 'rgw', 'mfa_ids': []} 2023-10-05T19:28:28.520 INFO:tasks.util.rgw:rgwadmin: client.0 : ['user', 'info', '--uid', 'foo'] 2023-10-05T19:28:28.520 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'ceph', 'user', 'info', '--uid', 'foo'] 2023-10-05T19:28:28.520 DEBUG:teuthology.orchestra.run.smithi167:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster ceph user info --uid foo 2023-10-05T19:28:28.564 INFO:teuthology.orchestra.run.smithi167.stderr:ignoring --setuser ceph since I am not root 2023-10-05T19:28:28.564 INFO:teuthology.orchestra.run.smithi167.stderr:ignoring --setgroup ceph since I am not root 2023-10-05T19:28:28.564 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.554+0000 7f97c0b82e40 1 Processor -- start 2023-10-05T19:28:28.564 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.554+0000 7f97c0b82e40 1 -- start start 2023-10-05T19:28:28.564 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.554+0000 7f97c0b82e40 1 --2- >> v2:172.21.15.118:3300/0 conn(0x555a089707e0 0x555a0896f6d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-05T19:28:28.564 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.554+0000 7f97c0b82e40 1 -- --> v1:172.21.15.118:6789/0 -- auth(proto 0 26 bytes epoch 0) v1 -- 0x555a088bcbd0 con 0x555a08970410 2023-10-05T19:28:28.564 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.554+0000 7f97c0b82e40 1 -- --> v2:172.21.15.118:3300/0 -- mon_getmap magic: 0 v1 -- 0x555a08879ee0 con 0x555a089707e0 2023-10-05T19:28:28.564 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.554+0000 7f97bdeb6700 1 --2- >> v2:172.21.15.118:3300/0 conn(0x555a089707e0 0x555a0896f6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-05T19:28:28.565 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.554+0000 7f97be6b7700 1 --1- >> v1:172.21.15.118:6789/0 conn(0x555a08970410 0x555a0896efc0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.118:6789/0 says I am v1:172.21.15.167:56470/0 (socket says 172.21.15.167:56470) 2023-10-05T19:28:28.565 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.554+0000 7f97be6b7700 1 -- 172.21.15.167:0/1329554993 learned_addr learned my addr 172.21.15.167:0/1329554993 (peer_addr_for_me v1:172.21.15.167:0/0) 2023-10-05T19:28:28.565 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.554+0000 7f97bdeb6700 1 -- 172.21.15.167:0/1329554993 >> v1:172.21.15.118:6789/0 conn(0x555a08970410 legacy=0x555a0896efc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-05T19:28:28.565 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.554+0000 7f97bdeb6700 1 -- 172.21.15.167:0/1329554993 --> v2:172.21.15.118:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x555a0896fcd0 con 0x555a089707e0 2023-10-05T19:28:28.565 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.554+0000 7f97bdeb6700 1 --2- 172.21.15.167:0/1329554993 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x555a089707e0 0x555a0896f6d0 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 crypto rx=0x7f97b400a5d0 tx=0x7f97b4005a50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4e30b6f687b65d8e server_cookie=0 in_seq=0 out_seq=0 2023-10-05T19:28:28.565 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.558+0000 7f97bd6b5700 1 -- 172.21.15.167:0/1329554993 <== mon.0 v2:172.21.15.118:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f97b40086a0 con 0x555a089707e0 2023-10-05T19:28:28.565 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.558+0000 7f97bd6b5700 1 -- 172.21.15.167:0/1329554993 <== mon.0 v2:172.21.15.118:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f97b4008800 con 0x555a089707e0 2023-10-05T19:28:28.565 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.558+0000 7f97bd6b5700 1 -- 172.21.15.167:0/1329554993 <== mon.0 v2:172.21.15.118:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f97b4008ad0 con 0x555a089707e0 2023-10-05T19:28:28.565 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.558+0000 7f97c0b82e40 1 -- 172.21.15.167:0/1329554993 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x555a089707e0 msgr2=0x555a0896f6d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-05T19:28:28.565 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.558+0000 7f97c0b82e40 1 --2- 172.21.15.167:0/1329554993 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x555a089707e0 0x555a0896f6d0 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 crypto rx=0x7f97b400a5d0 tx=0x7f97b4005a50 comp rx=0 tx=0).stop 2023-10-05T19:28:28.565 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.558+0000 7f97c0b82e40 1 -- 172.21.15.167:0/1329554993 shutdown_connections 2023-10-05T19:28:28.565 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.558+0000 7f97c0b82e40 1 --2- 172.21.15.167:0/1329554993 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x555a089707e0 0x555a0896f6d0 unknown :-1 s=CLOSED pgs=261 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-05T19:28:28.566 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.558+0000 7f97c0b82e40 1 -- 172.21.15.167:0/1329554993 >> 172.21.15.167:0/1329554993 conn(0x555a0896a720 msgr2=0x555a0896cb30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-05T19:28:28.566 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.558+0000 7f97c0b82e40 1 -- 172.21.15.167:0/1329554993 shutdown_connections 2023-10-05T19:28:28.566 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.558+0000 7f97c0b82e40 1 -- 172.21.15.167:0/1329554993 wait complete. 2023-10-05T19:28:28.566 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.558+0000 7f97c0b82e40 1 Processor -- start 2023-10-05T19:28:28.566 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.558+0000 7f97c0b82e40 1 -- start start 2023-10-05T19:28:28.568 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.558+0000 7f97c0b82e40 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x555a08970410 0x555a08991160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-05T19:28:28.568 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.558+0000 7f97c0b82e40 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x555a0883de30 con 0x555a08970410 2023-10-05T19:28:28.568 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.558+0000 7f97be6b7700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x555a08970410 0x555a08991160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-05T19:28:28.568 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.558+0000 7f97be6b7700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x555a08970410 0x555a08991160 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.118:3300/0 says I am v2:172.21.15.167:57736/0 (socket says 172.21.15.167:57736) 2023-10-05T19:28:28.568 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.558+0000 7f97be6b7700 1 -- 172.21.15.167:0/752842280 learned_addr learned my addr 172.21.15.167:0/752842280 (peer_addr_for_me v2:172.21.15.167:0/0) 2023-10-05T19:28:28.568 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.558+0000 7f97be6b7700 1 -- 172.21.15.167:0/752842280 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f97b4005cf0 con 0x555a08970410 2023-10-05T19:28:28.568 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.558+0000 7f97be6b7700 1 --2- 172.21.15.167:0/752842280 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x555a08970410 0x555a08991160 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 crypto rx=0x7f97b80014a0 tx=0x7f97b8006b90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f8c13ebacaf962c server_cookie=0 in_seq=0 out_seq=0 2023-10-05T19:28:28.568 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.558+0000 7f97affff700 1 -- 172.21.15.167:0/752842280 <== mon.0 v2:172.21.15.118:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f97b8006110 con 0x555a08970410 2023-10-05T19:28:28.569 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.558+0000 7f97affff700 1 -- 172.21.15.167:0/752842280 <== mon.0 v2:172.21.15.118:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f97b8006270 con 0x555a08970410 2023-10-05T19:28:28.569 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.558+0000 7f97affff700 1 -- 172.21.15.167:0/752842280 <== mon.0 v2:172.21.15.118:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f97b8004520 con 0x555a08970410 2023-10-05T19:28:28.569 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.562+0000 7f97c0b82e40 1 -- 172.21.15.167:0/752842280 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x555a08970410 msgr2=0x555a08991160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-05T19:28:28.569 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.562+0000 7f97c0b82e40 1 --2- 172.21.15.167:0/752842280 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x555a08970410 0x555a08991160 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 crypto rx=0x7f97b80014a0 tx=0x7f97b8006b90 comp rx=0 tx=0).stop 2023-10-05T19:28:28.569 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.562+0000 7f97c0b82e40 1 -- 172.21.15.167:0/752842280 shutdown_connections 2023-10-05T19:28:28.569 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.562+0000 7f97c0b82e40 1 --2- 172.21.15.167:0/752842280 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x555a08970410 0x555a08991160 unknown :-1 s=CLOSED pgs=262 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-05T19:28:28.569 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.562+0000 7f97c0b82e40 1 -- 172.21.15.167:0/752842280 >> 172.21.15.167:0/752842280 conn(0x555a0896a720 msgr2=0x555a0898cd20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-05T19:28:28.569 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.562+0000 7f97c0b82e40 1 -- 172.21.15.167:0/752842280 shutdown_connections 2023-10-05T19:28:28.569 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.562+0000 7f97c0b82e40 1 -- 172.21.15.167:0/752842280 wait complete. 2023-10-05T19:28:28.569 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.562+0000 7f97c0b82e40 1 Processor -- start 2023-10-05T19:28:28.569 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.562+0000 7f97c0b82e40 1 -- start start 2023-10-05T19:28:28.571 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.562+0000 7f97c0b82e40 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x555a08970410 0x555a08997f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-05T19:28:28.571 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.562+0000 7f97c0b82e40 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x555a08810ed0 con 0x555a08970410 2023-10-05T19:28:28.571 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.562+0000 7f97be6b7700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x555a08970410 0x555a08997f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-05T19:28:28.571 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.562+0000 7f97be6b7700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x555a08970410 0x555a08997f30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.118:3300/0 says I am v2:172.21.15.167:57738/0 (socket says 172.21.15.167:57738) 2023-10-05T19:28:28.571 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.562+0000 7f97be6b7700 1 -- 172.21.15.167:0/16775053 learned_addr learned my addr 172.21.15.167:0/16775053 (peer_addr_for_me v2:172.21.15.167:0/0) 2023-10-05T19:28:28.571 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.562+0000 7f97be6b7700 1 -- 172.21.15.167:0/16775053 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x555a088bcbd0 con 0x555a08970410 2023-10-05T19:28:28.571 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.562+0000 7f97be6b7700 1 --2- 172.21.15.167:0/16775053 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x555a08970410 0x555a08997f30 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 crypto rx=0x7f97b8006bd0 tx=0x7f97b8014000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-05T19:28:28.571 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.562+0000 7f97ae7fc700 1 -- 172.21.15.167:0/16775053 <== mon.0 v2:172.21.15.118:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f97b8013750 con 0x555a08970410 2023-10-05T19:28:28.572 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.562+0000 7f97ae7fc700 1 -- 172.21.15.167:0/16775053 <== mon.0 v2:172.21.15.118:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f97b80138b0 con 0x555a08970410 2023-10-05T19:28:28.572 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.562+0000 7f97c0b82e40 1 -- 172.21.15.167:0/16775053 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x555a08998470 con 0x555a08970410 2023-10-05T19:28:28.572 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.562+0000 7f97ae7fc700 1 -- 172.21.15.167:0/16775053 <== mon.0 v2:172.21.15.118:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f97b8013b80 con 0x555a08970410 2023-10-05T19:28:28.572 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.566+0000 7f97c0b82e40 1 -- 172.21.15.167:0/16775053 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x555a089988b0 con 0x555a08970410 2023-10-05T19:28:28.572 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.566+0000 7f97ae7fc700 1 -- 172.21.15.167:0/16775053 <== mon.0 v2:172.21.15.118:3300/0 4 ==== mgrmap(e 5) v1 ==== 40329+0+0 (secure 0 0 0) 0x7f97b8013d30 con 0x555a08970410 2023-10-05T19:28:28.573 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.566+0000 7f97ae7fc700 1 --2- 172.21.15.167:0/16775053 >> [v2:172.21.15.167:6824/29889,v1:172.21.15.167:6825/29889] conn(0x7f97a0030c50 0x7f97a0033100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-05T19:28:28.573 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.566+0000 7f97ae7fc700 1 -- 172.21.15.167:0/16775053 <== mon.0 v2:172.21.15.118:3300/0 5 ==== osd_map(30..30 src has 1..30) v4 ==== 6057+0+0 (secure 0 0 0) 0x7f97b8045190 con 0x555a08970410 2023-10-05T19:28:28.573 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.566+0000 7f97c0b82e40 20 rados->read ofs=0 len=0 2023-10-05T19:28:28.573 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.566+0000 7f97bdeb6700 1 --2- 172.21.15.167:0/16775053 >> [v2:172.21.15.167:6824/29889,v1:172.21.15.167:6825/29889] conn(0x7f97a0030c50 0x7f97a0033100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-05T19:28:28.573 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.566+0000 7f97c0b82e40 1 --2- 172.21.15.167:0/16775053 >> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] conn(0x555a089707e0 0x555a089a39f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-05T19:28:28.574 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.566+0000 7f97c0b82e40 1 -- 172.21.15.167:0/16775053 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:1 5.12 5:49953fa1:::default.realm:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x555a089a40f0 con 0x555a089707e0 2023-10-05T19:28:28.574 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.566+0000 7f97bdeb6700 1 --2- 172.21.15.167:0/16775053 >> [v2:172.21.15.167:6824/29889,v1:172.21.15.167:6825/29889] conn(0x7f97a0030c50 0x7f97a0033100 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7f97b4005a90 tx=0x7f97b4015000 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-05T19:28:28.574 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.566+0000 7f97beeb8700 1 --2- 172.21.15.167:0/16775053 >> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] conn(0x555a089707e0 0x555a089a39f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-05T19:28:28.574 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.566+0000 7f97beeb8700 1 --2- 172.21.15.167:0/16775053 >> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] conn(0x555a089707e0 0x555a089a39f0 crc :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-05T19:28:28.575 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.566+0000 7f97beeb8700 1 -- 172.21.15.167:0/16775053 <== osd.1 v2:172.21.15.167:6800/30035 1 ==== osd_op_reply(1 default.realm [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 ==== 157+0+0 (crc 0 0 0) 0x7f97b000a500 con 0x555a089707e0 2023-10-05T19:28:28.576 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.566+0000 7f97c0b82e40 20 rados_obj.operate() r=-2 bl.length=0 2023-10-05T19:28:28.576 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.566+0000 7f97c0b82e40 20 realm 2023-10-05T19:28:28.576 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.570+0000 7f97c0b82e40 20 rados->read ofs=0 len=0 2023-10-05T19:28:28.576 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.570+0000 7f97c0b82e40 1 -- 172.21.15.167:0/16775053 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:2 5.12 5:49953fa1:::default.realm:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x555a089a1050 con 0x555a089707e0 2023-10-05T19:28:28.576 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.570+0000 7f97beeb8700 1 -- 172.21.15.167:0/16775053 <== osd.1 v2:172.21.15.167:6800/30035 2 ==== osd_op_reply(2 default.realm [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 ==== 157+0+0 (crc 0 0 0) 0x7f97b000a500 con 0x555a089707e0 2023-10-05T19:28:28.576 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.570+0000 7f97c0b82e40 20 rados_obj.operate() r=-2 bl.length=0 2023-10-05T19:28:28.576 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.570+0000 7f97c0b82e40 4 RGWPeriod::init failed to init realm id : (2) No such file or directory 2023-10-05T19:28:28.576 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.570+0000 7f97c0b82e40 20 rados->read ofs=0 len=0 2023-10-05T19:28:28.576 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.570+0000 7f97c0b82e40 1 -- 172.21.15.167:0/16775053 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:3 5.12 5:49953fa1:::default.realm:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x555a089a4bd0 con 0x555a089707e0 2023-10-05T19:28:28.576 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.570+0000 7f97beeb8700 1 -- 172.21.15.167:0/16775053 <== osd.1 v2:172.21.15.167:6800/30035 3 ==== osd_op_reply(3 default.realm [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 ==== 157+0+0 (crc 0 0 0) 0x7f97b000a500 con 0x555a089707e0 2023-10-05T19:28:28.576 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.570+0000 7f97c0b82e40 20 rados_obj.operate() r=-2 bl.length=0 2023-10-05T19:28:28.577 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.570+0000 7f97c0b82e40 20 rados->read ofs=0 len=0 2023-10-05T19:28:28.577 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.570+0000 7f97c0b82e40 1 -- 172.21.15.167:0/16775053 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:4 5.f 5:f43b6ece:::zone_names.default:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x555a089a47b0 con 0x555a089707e0 2023-10-05T19:28:28.577 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.570+0000 7f97beeb8700 1 -- 172.21.15.167:0/16775053 <== osd.1 v2:172.21.15.167:6800/30035 4 ==== osd_op_reply(4 zone_names.default [read 0~46 out=46b] v0'0 uv1 ondisk = 0) v8 ==== 162+0+46 (crc 0 0 0) 0x7f97b000a500 con 0x555a089707e0 2023-10-05T19:28:28.577 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.570+0000 7f97c0b82e40 20 rados_obj.operate() r=0 bl.length=46 2023-10-05T19:28:28.577 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.570+0000 7f97c0b82e40 20 rados->read ofs=0 len=0 2023-10-05T19:28:28.577 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.570+0000 7f97c0b82e40 1 -- 172.21.15.167:0/16775053 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:5 5.1d 5:b8374eb4:::zone_info.9a9df965-6612-45d4-a45f-6e3c889bacc2:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x555a089a5640 con 0x555a089707e0 2023-10-05T19:28:28.578 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.570+0000 7f97beeb8700 1 -- 172.21.15.167:0/16775053 <== osd.1 v2:172.21.15.167:6800/30035 5 ==== osd_op_reply(5 zone_info.9a9df965-6612-45d4-a45f-6e3c889bacc2 [read 0~874 out=874b] v0'0 uv2 ondisk = 0) v8 ==== 190+0+874 (crc 0 0 0) 0x7f97b000a500 con 0x555a089707e0 2023-10-05T19:28:28.578 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.570+0000 7f97c0b82e40 20 rados_obj.operate() r=0 bl.length=874 2023-10-05T19:28:28.578 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.570+0000 7f97c0b82e40 20 searching for the correct realm 2023-10-05T19:28:28.578 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.570+0000 7f97c0b82e40 1 --2- 172.21.15.167:0/16775053 >> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] conn(0x555a089a7490 0x555a089a9870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-05T19:28:28.578 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.570+0000 7f97c0b82e40 1 -- 172.21.15.167:0/16775053 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:6 5.0 5:00000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x555a089a9db0 con 0x555a089a7490 2023-10-05T19:28:28.578 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.570+0000 7f97be6b7700 1 --2- 172.21.15.167:0/16775053 >> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] conn(0x555a089a7490 0x555a089a9870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-05T19:28:28.578 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.570+0000 7f97be6b7700 1 --2- 172.21.15.167:0/16775053 >> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] conn(0x555a089a7490 0x555a089a9870 crc :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-05T19:28:28.580 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.574+0000 7f97be6b7700 1 -- 172.21.15.167:0/16775053 <== osd.3 v2:172.21.15.167:6808/30036 1 ==== osd_op_reply(6 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f97b8024020 con 0x555a089a7490 2023-10-05T19:28:28.580 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.574+0000 7f97be6b7700 1 --2- 172.21.15.167:0/16775053 >> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] conn(0x7f97b8027be0 0x7f97b8042ba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-05T19:28:28.580 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.574+0000 7f97be6b7700 1 -- 172.21.15.167:0/16775053 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:7 5.10 5:08000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x555a089a9db0 con 0x7f97b8027be0 2023-10-05T19:28:28.580 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.574+0000 7f97bdeb6700 1 --2- 172.21.15.167:0/16775053 >> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] conn(0x7f97b8027be0 0x7f97b8042ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-05T19:28:28.581 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.574+0000 7f97bdeb6700 1 --2- 172.21.15.167:0/16775053 >> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] conn(0x7f97b8027be0 0x7f97b8042ba0 crc :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-05T19:28:28.581 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.574+0000 7f97bdeb6700 1 -- 172.21.15.167:0/16775053 <== osd.2 v2:172.21.15.167:6816/30037 1 ==== osd_op_reply(7 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f97b4008ae0 con 0x7f97b8027be0 2023-10-05T19:28:28.581 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.574+0000 7f97bdeb6700 1 --2- 172.21.15.167:0/16775053 >> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] conn(0x7f97b4007030 0x7f97b401c920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-05T19:28:28.581 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.574+0000 7f97bdeb6700 1 -- 172.21.15.167:0/16775053 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:8 5.8 5:10000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x555a089a9db0 con 0x7f97b4007030 2023-10-05T19:28:28.581 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.574+0000 7f97beeb8700 1 --2- 172.21.15.167:0/16775053 >> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] conn(0x7f97b4007030 0x7f97b401c920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-05T19:28:28.582 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.578+0000 7f97beeb8700 1 --2- 172.21.15.167:0/16775053 >> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] conn(0x7f97b4007030 0x7f97b401c920 crc :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-05T19:28:28.583 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.578+0000 7f97beeb8700 1 -- 172.21.15.167:0/16775053 <== osd.0 v2:172.21.15.118:6800/27690 1 ==== osd_op_reply(8 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f97b0014040 con 0x7f97b4007030 2023-10-05T19:28:28.583 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.578+0000 7f97beeb8700 1 -- 172.21.15.167:0/16775053 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:9 5.18 5:18000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x555a089a9db0 con 0x555a089707e0 2023-10-05T19:28:28.583 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.578+0000 7f97beeb8700 1 -- 172.21.15.167:0/16775053 <== osd.1 v2:172.21.15.167:6800/30035 6 ==== osd_op_reply(9 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f97b0014040 con 0x555a089707e0 2023-10-05T19:28:28.584 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.578+0000 7f97beeb8700 1 -- 172.21.15.167:0/16775053 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:10 5.4 5:20000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x555a089a9db0 con 0x555a089707e0 2023-10-05T19:28:28.584 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.578+0000 7f97beeb8700 1 -- 172.21.15.167:0/16775053 <== osd.1 v2:172.21.15.167:6800/30035 7 ==== osd_op_reply(10 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f97b0014040 con 0x555a089707e0 2023-10-05T19:28:28.584 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.578+0000 7f97beeb8700 1 -- 172.21.15.167:0/16775053 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:11 5.14 5:28000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x555a089a9db0 con 0x7f97b4007030 2023-10-05T19:28:28.585 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.578+0000 7f97beeb8700 1 -- 172.21.15.167:0/16775053 <== osd.0 v2:172.21.15.118:6800/27690 2 ==== osd_op_reply(11 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f97b0014040 con 0x7f97b4007030 2023-10-05T19:28:28.585 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.578+0000 7f97beeb8700 1 -- 172.21.15.167:0/16775053 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:12 5.c 5:30000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x555a089a9db0 con 0x7f97b4007030 2023-10-05T19:28:28.586 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.582+0000 7f97beeb8700 1 -- 172.21.15.167:0/16775053 <== osd.0 v2:172.21.15.118:6800/27690 3 ==== osd_op_reply(12 [pgnls start_epoch 30 out=112b] v23'1 uv1 ondisk = 1) v8 ==== 144+0+112 (crc 0 0 0) 0x7f97b0014040 con 0x7f97b4007030 2023-10-05T19:28:28.586 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.582+0000 7f97beeb8700 1 -- 172.21.15.167:0/16775053 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:13 5.1c 5:38000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x555a089a9db0 con 0x7f97b8027be0 2023-10-05T19:28:28.587 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.582+0000 7f97bdeb6700 1 -- 172.21.15.167:0/16775053 <== osd.2 v2:172.21.15.167:6816/30037 2 ==== osd_op_reply(13 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f97b4008ae0 con 0x7f97b8027be0 2023-10-05T19:28:28.587 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.582+0000 7f97bdeb6700 1 -- 172.21.15.167:0/16775053 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:14 5.2 5:40000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x555a089a9db0 con 0x555a089a7490 2023-10-05T19:28:28.587 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.582+0000 7f97be6b7700 1 -- 172.21.15.167:0/16775053 <== osd.3 v2:172.21.15.167:6808/30036 2 ==== osd_op_reply(14 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f97b8024020 con 0x555a089a7490 2023-10-05T19:28:28.588 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.582+0000 7f97be6b7700 1 -- 172.21.15.167:0/16775053 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:15 5.12 5:48000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x555a089a9db0 con 0x555a089707e0 2023-10-05T19:28:28.588 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.582+0000 7f97beeb8700 1 -- 172.21.15.167:0/16775053 <== osd.1 v2:172.21.15.167:6800/30035 8 ==== osd_op_reply(15 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f97b0014040 con 0x555a089707e0 2023-10-05T19:28:28.588 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.582+0000 7f97beeb8700 1 -- 172.21.15.167:0/16775053 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:16 5.a 5:50000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x555a089a9db0 con 0x7f97b4007030 2023-10-05T19:28:28.589 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.582+0000 7f97beeb8700 1 -- 172.21.15.167:0/16775053 <== osd.0 v2:172.21.15.118:6800/27690 4 ==== osd_op_reply(16 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f97b0022040 con 0x7f97b4007030 2023-10-05T19:28:28.589 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.582+0000 7f97beeb8700 1 -- 172.21.15.167:0/16775053 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:17 5.1a 5:58000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x555a089a9db0 con 0x555a089707e0 2023-10-05T19:28:28.590 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.582+0000 7f97beeb8700 1 -- 172.21.15.167:0/16775053 <== osd.1 v2:172.21.15.167:6800/30035 9 ==== osd_op_reply(17 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f97b0022040 con 0x555a089707e0 2023-10-05T19:28:28.590 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.586+0000 7f97beeb8700 1 -- 172.21.15.167:0/16775053 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:18 5.6 5:60000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x555a089a9db0 con 0x7f97b8027be0 2023-10-05T19:28:28.590 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.586+0000 7f97bdeb6700 1 -- 172.21.15.167:0/16775053 <== osd.2 v2:172.21.15.167:6816/30037 3 ==== osd_op_reply(18 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f97b4008ae0 con 0x7f97b8027be0 2023-10-05T19:28:28.591 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.586+0000 7f97bdeb6700 1 -- 172.21.15.167:0/16775053 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:19 5.16 5:68000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x555a089a9db0 con 0x555a089a7490 2023-10-05T19:28:28.591 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.586+0000 7f97be6b7700 1 -- 172.21.15.167:0/16775053 <== osd.3 v2:172.21.15.167:6808/30036 3 ==== osd_op_reply(19 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f97b8024020 con 0x555a089a7490 2023-10-05T19:28:28.591 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.586+0000 7f97be6b7700 1 -- 172.21.15.167:0/16775053 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:20 5.e 5:70000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x555a089a9db0 con 0x555a089a7490 2023-10-05T19:28:28.592 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.586+0000 7f97be6b7700 1 -- 172.21.15.167:0/16775053 <== osd.3 v2:172.21.15.167:6808/30036 4 ==== osd_op_reply(20 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f97b8024020 con 0x555a089a7490 2023-10-05T19:28:28.592 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.586+0000 7f97be6b7700 1 -- 172.21.15.167:0/16775053 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:21 5.1e 5:78000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x555a089a9db0 con 0x7f97b4007030 2023-10-05T19:28:28.593 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.586+0000 7f97beeb8700 1 -- 172.21.15.167:0/16775053 <== osd.0 v2:172.21.15.118:6800/27690 5 ==== osd_op_reply(21 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f97b0022040 con 0x7f97b4007030 2023-10-05T19:28:28.593 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.586+0000 7f97beeb8700 1 -- 172.21.15.167:0/16775053 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:22 5.1 5:80000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x555a089a9db0 con 0x555a089a7490 2023-10-05T19:28:28.594 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.586+0000 7f97be6b7700 1 -- 172.21.15.167:0/16775053 <== osd.3 v2:172.21.15.167:6808/30036 5 ==== osd_op_reply(22 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f97b8024020 con 0x555a089a7490 2023-10-05T19:28:28.594 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.586+0000 7f97be6b7700 1 -- 172.21.15.167:0/16775053 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:23 5.11 5:88000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x555a089a9db0 con 0x555a089a7490 2023-10-05T19:28:28.594 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.590+0000 7f97be6b7700 1 -- 172.21.15.167:0/16775053 <== osd.3 v2:172.21.15.167:6808/30036 6 ==== osd_op_reply(23 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f97b8024020 con 0x555a089a7490 2023-10-05T19:28:28.594 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.590+0000 7f97be6b7700 1 -- 172.21.15.167:0/16775053 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:24 5.9 5:90000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x555a089a9db0 con 0x555a089707e0 2023-10-05T19:28:28.595 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.590+0000 7f97beeb8700 1 -- 172.21.15.167:0/16775053 <== osd.1 v2:172.21.15.167:6800/30035 10 ==== osd_op_reply(24 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f97b0022040 con 0x555a089707e0 2023-10-05T19:28:28.595 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.590+0000 7f97beeb8700 1 -- 172.21.15.167:0/16775053 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:25 5.19 5:98000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x555a089a9db0 con 0x555a089707e0 2023-10-05T19:28:28.596 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.590+0000 7f97beeb8700 1 -- 172.21.15.167:0/16775053 <== osd.1 v2:172.21.15.167:6800/30035 11 ==== osd_op_reply(25 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f97b0022040 con 0x555a089707e0 2023-10-05T19:28:28.596 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.590+0000 7f97beeb8700 1 -- 172.21.15.167:0/16775053 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:26 5.5 5:a0000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x555a089a9db0 con 0x7f97b4007030 2023-10-05T19:28:28.597 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.590+0000 7f97beeb8700 1 -- 172.21.15.167:0/16775053 <== osd.0 v2:172.21.15.118:6800/27690 6 ==== osd_op_reply(26 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f97b0022040 con 0x7f97b4007030 2023-10-05T19:28:28.597 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.590+0000 7f97beeb8700 1 -- 172.21.15.167:0/16775053 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:27 5.15 5:a8000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x555a089a9db0 con 0x555a089a7490 2023-10-05T19:28:28.597 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.590+0000 7f97be6b7700 1 -- 172.21.15.167:0/16775053 <== osd.3 v2:172.21.15.167:6808/30036 7 ==== osd_op_reply(27 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f97b8024020 con 0x555a089a7490 2023-10-05T19:28:28.598 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.590+0000 7f97be6b7700 1 -- 172.21.15.167:0/16775053 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:28 5.d 5:b0000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x555a089a9db0 con 0x7f97b8027be0 2023-10-05T19:28:28.598 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.594+0000 7f97bdeb6700 1 -- 172.21.15.167:0/16775053 <== osd.2 v2:172.21.15.167:6816/30037 4 ==== osd_op_reply(28 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f97b4008ae0 con 0x7f97b8027be0 2023-10-05T19:28:28.598 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.594+0000 7f97bdeb6700 1 -- 172.21.15.167:0/16775053 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:29 5.1d 5:b8000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x555a089a9db0 con 0x555a089707e0 2023-10-05T19:28:28.599 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.594+0000 7f97beeb8700 1 -- 172.21.15.167:0/16775053 <== osd.1 v2:172.21.15.167:6800/30035 12 ==== osd_op_reply(29 [pgnls start_epoch 30 out=107b] v30'2 uv2 ondisk = 1) v8 ==== 144+0+107 (crc 0 0 0) 0x7f97b0022040 con 0x555a089707e0 2023-10-05T19:28:28.599 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.594+0000 7f97beeb8700 1 -- 172.21.15.167:0/16775053 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:30 5.3 5:c0000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x555a089a9db0 con 0x7f97b4007030 2023-10-05T19:28:28.600 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.594+0000 7f97beeb8700 1 -- 172.21.15.167:0/16775053 <== osd.0 v2:172.21.15.118:6800/27690 7 ==== osd_op_reply(30 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f97b0022040 con 0x7f97b4007030 2023-10-05T19:28:28.600 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.594+0000 7f97beeb8700 1 -- 172.21.15.167:0/16775053 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:31 5.13 5:c8000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x555a089a9db0 con 0x555a089a7490 2023-10-05T19:28:28.601 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.594+0000 7f97be6b7700 1 -- 172.21.15.167:0/16775053 <== osd.3 v2:172.21.15.167:6808/30036 8 ==== osd_op_reply(31 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f97b8024020 con 0x555a089a7490 2023-10-05T19:28:28.601 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.594+0000 7f97be6b7700 1 -- 172.21.15.167:0/16775053 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:32 5.b 5:d0000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x555a089a9db0 con 0x7f97b4007030 2023-10-05T19:28:28.602 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.598+0000 7f97beeb8700 1 -- 172.21.15.167:0/16775053 <== osd.0 v2:172.21.15.118:6800/27690 8 ==== osd_op_reply(32 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f97b0022040 con 0x7f97b4007030 2023-10-05T19:28:28.602 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.598+0000 7f97beeb8700 1 -- 172.21.15.167:0/16775053 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:33 5.1b 5:d8000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x555a089a9db0 con 0x7f97b8027be0 2023-10-05T19:28:28.603 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.598+0000 7f97bdeb6700 1 -- 172.21.15.167:0/16775053 <== osd.2 v2:172.21.15.167:6816/30037 5 ==== osd_op_reply(33 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f97b4008ae0 con 0x7f97b8027be0 2023-10-05T19:28:28.603 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.598+0000 7f97bdeb6700 1 -- 172.21.15.167:0/16775053 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:34 5.7 5:e0000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x555a089a9db0 con 0x555a089a7490 2023-10-05T19:28:28.603 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.598+0000 7f97be6b7700 1 -- 172.21.15.167:0/16775053 <== osd.3 v2:172.21.15.167:6808/30036 9 ==== osd_op_reply(34 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f97b8024020 con 0x555a089a7490 2023-10-05T19:28:28.604 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.598+0000 7f97be6b7700 1 -- 172.21.15.167:0/16775053 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:35 5.17 5:e8000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x555a089a9db0 con 0x7f97b4007030 2023-10-05T19:28:28.605 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.598+0000 7f97beeb8700 1 -- 172.21.15.167:0/16775053 <== osd.0 v2:172.21.15.118:6800/27690 9 ==== osd_op_reply(35 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f97b0022040 con 0x7f97b4007030 2023-10-05T19:28:28.605 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.598+0000 7f97beeb8700 1 -- 172.21.15.167:0/16775053 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:36 5.f 5:f0000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x555a089a9db0 con 0x555a089707e0 2023-10-05T19:28:28.606 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.598+0000 7f97beeb8700 1 -- 172.21.15.167:0/16775053 <== osd.1 v2:172.21.15.167:6800/30035 13 ==== osd_op_reply(36 [pgnls start_epoch 30 out=115b] v23'2 uv2 ondisk = 1) v8 ==== 144+0+115 (crc 0 0 0) 0x7f97b002f080 con 0x555a089707e0 2023-10-05T19:28:28.606 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.598+0000 7f97beeb8700 1 -- 172.21.15.167:0/16775053 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:37 5.1f 5:f8000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x555a089a9db0 con 0x7f97b8027be0 2023-10-05T19:28:28.607 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.602+0000 7f97bdeb6700 1 -- 172.21.15.167:0/16775053 <== osd.2 v2:172.21.15.167:6816/30037 6 ==== osd_op_reply(37 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f97b4008ae0 con 0x7f97b8027be0 2023-10-05T19:28:28.607 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.602+0000 7f97c0b82e40 20 RGWRados::pool_iterate: got zonegroup_info.4f1f719d-9648-417b-b10c-e5d939825419 2023-10-05T19:28:28.607 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.602+0000 7f97c0b82e40 20 RGWRados::pool_iterate: got zone_info.9a9df965-6612-45d4-a45f-6e3c889bacc2 2023-10-05T19:28:28.607 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.602+0000 7f97c0b82e40 20 RGWRados::pool_iterate: got zone_names.default 2023-10-05T19:28:28.607 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.602+0000 7f97c0b82e40 20 RGWRados::pool_iterate: got zonegroups_names.default 2023-10-05T19:28:28.607 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.602+0000 7f97c0b82e40 20 rados->read ofs=0 len=0 2023-10-05T19:28:28.607 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.602+0000 7f97c0b82e40 1 -- 172.21.15.167:0/16775053 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:38 5.12 5:49953fa1:::default.realm:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x555a089a6210 con 0x555a089707e0 2023-10-05T19:28:28.608 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.602+0000 7f97beeb8700 1 -- 172.21.15.167:0/16775053 <== osd.1 v2:172.21.15.167:6800/30035 14 ==== osd_op_reply(38 default.realm [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 ==== 157+0+0 (crc 0 0 0) 0x7f97b002f080 con 0x555a089707e0 2023-10-05T19:28:28.608 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.602+0000 7f97c0b82e40 20 rados_obj.operate() r=-2 bl.length=0 2023-10-05T19:28:28.608 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.602+0000 7f97c0b82e40 20 rados->read ofs=0 len=0 2023-10-05T19:28:28.608 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.602+0000 7f97c0b82e40 1 -- 172.21.15.167:0/16775053 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:39 5.f 5:f4c53578:::zonegroups_names.default:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x555a089a6b90 con 0x555a089707e0 2023-10-05T19:28:28.609 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.602+0000 7f97beeb8700 1 -- 172.21.15.167:0/16775053 <== osd.1 v2:172.21.15.167:6800/30035 15 ==== osd_op_reply(39 zonegroups_names.default [read 0~46 out=46b] v0'0 uv2 ondisk = 0) v8 ==== 168+0+46 (crc 0 0 0) 0x7f97b002f080 con 0x555a089707e0 2023-10-05T19:28:28.609 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.602+0000 7f97c0b82e40 20 rados_obj.operate() r=0 bl.length=46 2023-10-05T19:28:28.609 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.602+0000 7f97c0b82e40 20 rados->read ofs=0 len=0 2023-10-05T19:28:28.609 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.602+0000 7f97c0b82e40 1 -- 172.21.15.167:0/16775053 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:40 5.c 5:30d299a5:::zonegroup_info.4f1f719d-9648-417b-b10c-e5d939825419:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x555a089a5dd0 con 0x7f97b4007030 2023-10-05T19:28:28.610 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.602+0000 7f97beeb8700 1 -- 172.21.15.167:0/16775053 <== osd.0 v2:172.21.15.118:6800/27690 10 ==== osd_op_reply(40 zonegroup_info.4f1f719d-9648-417b-b10c-e5d939825419 [read 0~362 out=362b] v0'0 uv1 ondisk = 0) v8 ==== 195+0+362 (crc 0 0 0) 0x7f97b002f080 con 0x7f97b4007030 2023-10-05T19:28:28.610 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.602+0000 7f97c0b82e40 20 rados_obj.operate() r=0 bl.length=362 2023-10-05T19:28:28.610 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.602+0000 7f97c0b82e40 20 rados->read ofs=0 len=0 2023-10-05T19:28:28.610 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.602+0000 7f97c0b82e40 1 -- 172.21.15.167:0/16775053 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:41 5.13 5:cbbf6c60:::converted:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x555a089aa160 con 0x555a089a7490 2023-10-05T19:28:28.611 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.606+0000 7f97be6b7700 1 -- 172.21.15.167:0/16775053 <== osd.3 v2:172.21.15.167:6808/30036 10 ==== osd_op_reply(41 converted [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 ==== 153+0+0 (crc 0 0 0) 0x7f97b8024020 con 0x555a089a7490 2023-10-05T19:28:28.611 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.606+0000 7f97c0b82e40 20 rados_obj.operate() r=-2 bl.length=0 2023-10-05T19:28:28.611 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.606+0000 7f97c0b82e40 20 rados->read ofs=0 len=0 2023-10-05T19:28:28.611 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.606+0000 7f97c0b82e40 1 -- 172.21.15.167:0/16775053 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:42 5.12 5:49953fa1:::default.realm:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x555a089aa510 con 0x555a089707e0 2023-10-05T19:28:28.611 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.606+0000 7f97beeb8700 1 -- 172.21.15.167:0/16775053 <== osd.1 v2:172.21.15.167:6800/30035 16 ==== osd_op_reply(42 default.realm [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 ==== 157+0+0 (crc 0 0 0) 0x7f97b002f080 con 0x555a089707e0 2023-10-05T19:28:28.611 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.606+0000 7f97c0b82e40 20 rados_obj.operate() r=-2 bl.length=0 2023-10-05T19:28:28.611 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.606+0000 7f97c0b82e40 20 rados->read ofs=0 len=0 2023-10-05T19:28:28.612 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.606+0000 7f97c0b82e40 1 -- 172.21.15.167:0/16775053 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:43 5.f 5:f4c53578:::zonegroups_names.default:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x555a089aa510 con 0x555a089707e0 2023-10-05T19:28:28.612 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.606+0000 7f97beeb8700 1 -- 172.21.15.167:0/16775053 <== osd.1 v2:172.21.15.167:6800/30035 17 ==== osd_op_reply(43 zonegroups_names.default [read 0~46 out=46b] v0'0 uv2 ondisk = 0) v8 ==== 168+0+46 (crc 0 0 0) 0x7f97b002f080 con 0x555a089707e0 2023-10-05T19:28:28.612 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.606+0000 7f97c0b82e40 20 rados_obj.operate() r=0 bl.length=46 2023-10-05T19:28:28.612 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.606+0000 7f97c0b82e40 1 -- 172.21.15.167:0/16775053 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:44 5.0 5:00000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x555a089aa510 con 0x555a089a7490 2023-10-05T19:28:28.614 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.610+0000 7f97be6b7700 1 -- 172.21.15.167:0/16775053 <== osd.3 v2:172.21.15.167:6808/30036 11 ==== osd_op_reply(44 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f97b8024020 con 0x555a089a7490 2023-10-05T19:28:28.614 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.610+0000 7f97be6b7700 1 -- 172.21.15.167:0/16775053 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:45 5.10 5:08000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x555a089aa510 con 0x7f97b8027be0 2023-10-05T19:28:28.615 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.610+0000 7f97bdeb6700 1 -- 172.21.15.167:0/16775053 <== osd.2 v2:172.21.15.167:6816/30037 7 ==== osd_op_reply(45 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f97b4008ae0 con 0x7f97b8027be0 2023-10-05T19:28:28.615 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.610+0000 7f97bdeb6700 1 -- 172.21.15.167:0/16775053 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:46 5.8 5:10000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x555a089aa510 con 0x7f97b4007030 2023-10-05T19:28:28.616 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.610+0000 7f97beeb8700 1 -- 172.21.15.167:0/16775053 <== osd.0 v2:172.21.15.118:6800/27690 11 ==== osd_op_reply(46 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f97b002f080 con 0x7f97b4007030 2023-10-05T19:28:28.616 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.610+0000 7f97beeb8700 1 -- 172.21.15.167:0/16775053 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:47 5.18 5:18000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x555a089a6b90 con 0x555a089707e0 2023-10-05T19:28:28.617 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.610+0000 7f97beeb8700 1 -- 172.21.15.167:0/16775053 <== osd.1 v2:172.21.15.167:6800/30035 18 ==== osd_op_reply(47 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f97b002f080 con 0x555a089707e0 2023-10-05T19:28:28.617 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.610+0000 7f97beeb8700 1 -- 172.21.15.167:0/16775053 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:48 5.4 5:20000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x555a089a6b90 con 0x555a089707e0 2023-10-05T19:28:28.618 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.610+0000 7f97beeb8700 1 -- 172.21.15.167:0/16775053 <== osd.1 v2:172.21.15.167:6800/30035 19 ==== osd_op_reply(48 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f97b002f080 con 0x555a089707e0 2023-10-05T19:28:28.618 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.610+0000 7f97beeb8700 1 -- 172.21.15.167:0/16775053 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:49 5.14 5:28000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x555a089a6b90 con 0x7f97b4007030 2023-10-05T19:28:28.619 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.614+0000 7f97beeb8700 1 -- 172.21.15.167:0/16775053 <== osd.0 v2:172.21.15.118:6800/27690 12 ==== osd_op_reply(49 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f97b002f080 con 0x7f97b4007030 2023-10-05T19:28:28.619 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.614+0000 7f97beeb8700 1 -- 172.21.15.167:0/16775053 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:50 5.c 5:30000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x555a089a6b90 con 0x7f97b4007030 2023-10-05T19:28:28.620 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.614+0000 7f97beeb8700 1 -- 172.21.15.167:0/16775053 <== osd.0 v2:172.21.15.118:6800/27690 13 ==== osd_op_reply(50 [pgnls start_epoch 30 out=112b] v23'1 uv1 ondisk = 1) v8 ==== 144+0+112 (crc 0 0 0) 0x7f97b002f080 con 0x7f97b4007030 2023-10-05T19:28:28.620 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.614+0000 7f97beeb8700 1 -- 172.21.15.167:0/16775053 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:51 5.1c 5:38000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x555a089a6b90 con 0x7f97b8027be0 2023-10-05T19:28:28.621 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.614+0000 7f97bdeb6700 1 -- 172.21.15.167:0/16775053 <== osd.2 v2:172.21.15.167:6816/30037 8 ==== osd_op_reply(51 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f97b4008ae0 con 0x7f97b8027be0 2023-10-05T19:28:28.621 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.614+0000 7f97bdeb6700 1 -- 172.21.15.167:0/16775053 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:52 5.2 5:40000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x555a089a6b90 con 0x555a089a7490 2023-10-05T19:28:28.622 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.618+0000 7f97be6b7700 1 -- 172.21.15.167:0/16775053 <== osd.3 v2:172.21.15.167:6808/30036 12 ==== osd_op_reply(52 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f97b8024020 con 0x555a089a7490 2023-10-05T19:28:28.622 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.618+0000 7f97be6b7700 1 -- 172.21.15.167:0/16775053 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:53 5.12 5:48000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x555a089a6b90 con 0x555a089707e0 2023-10-05T19:28:28.623 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.618+0000 7f97beeb8700 1 -- 172.21.15.167:0/16775053 <== osd.1 v2:172.21.15.167:6800/30035 20 ==== osd_op_reply(53 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f97b002f080 con 0x555a089707e0 2023-10-05T19:28:28.623 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.618+0000 7f97beeb8700 1 -- 172.21.15.167:0/16775053 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:54 5.a 5:50000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x555a089a6b90 con 0x7f97b4007030 2023-10-05T19:28:28.624 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.618+0000 7f97beeb8700 1 -- 172.21.15.167:0/16775053 <== osd.0 v2:172.21.15.118:6800/27690 14 ==== osd_op_reply(54 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f97b002f080 con 0x7f97b4007030 2023-10-05T19:28:28.624 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.618+0000 7f97beeb8700 1 -- 172.21.15.167:0/16775053 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:55 5.1a 5:58000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x555a089a6b90 con 0x555a089707e0 2023-10-05T19:28:28.625 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.618+0000 7f97beeb8700 1 -- 172.21.15.167:0/16775053 <== osd.1 v2:172.21.15.167:6800/30035 21 ==== osd_op_reply(55 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f97b002f080 con 0x555a089707e0 2023-10-05T19:28:28.625 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.618+0000 7f97beeb8700 1 -- 172.21.15.167:0/16775053 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:56 5.6 5:60000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x555a089a6b90 con 0x7f97b8027be0 2023-10-05T19:28:28.626 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.618+0000 7f97bdeb6700 1 -- 172.21.15.167:0/16775053 <== osd.2 v2:172.21.15.167:6816/30037 9 ==== osd_op_reply(56 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f97b4008ae0 con 0x7f97b8027be0 2023-10-05T19:28:28.626 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.618+0000 7f97bdeb6700 1 -- 172.21.15.167:0/16775053 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:57 5.16 5:68000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x555a089a6b90 con 0x555a089a7490 2023-10-05T19:28:28.626 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.622+0000 7f97be6b7700 1 -- 172.21.15.167:0/16775053 <== osd.3 v2:172.21.15.167:6808/30036 13 ==== osd_op_reply(57 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f97b8024020 con 0x555a089a7490 2023-10-05T19:28:28.626 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.622+0000 7f97be6b7700 1 -- 172.21.15.167:0/16775053 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:58 5.e 5:70000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x555a089a6b90 con 0x555a089a7490 2023-10-05T19:28:28.627 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.622+0000 7f97be6b7700 1 -- 172.21.15.167:0/16775053 <== osd.3 v2:172.21.15.167:6808/30036 14 ==== osd_op_reply(58 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f97b8024020 con 0x555a089a7490 2023-10-05T19:28:28.627 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.622+0000 7f97be6b7700 1 -- 172.21.15.167:0/16775053 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:59 5.1e 5:78000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x555a089a6b90 con 0x7f97b4007030 2023-10-05T19:28:28.628 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.622+0000 7f97beeb8700 1 -- 172.21.15.167:0/16775053 <== osd.0 v2:172.21.15.118:6800/27690 15 ==== osd_op_reply(59 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f97b002f080 con 0x7f97b4007030 2023-10-05T19:28:28.628 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.622+0000 7f97beeb8700 1 -- 172.21.15.167:0/16775053 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:60 5.1 5:80000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x555a089a6b90 con 0x555a089a7490 2023-10-05T19:28:28.629 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.622+0000 7f97be6b7700 1 -- 172.21.15.167:0/16775053 <== osd.3 v2:172.21.15.167:6808/30036 15 ==== osd_op_reply(60 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f97b8024020 con 0x555a089a7490 2023-10-05T19:28:28.629 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.622+0000 7f97be6b7700 1 -- 172.21.15.167:0/16775053 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:61 5.11 5:88000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x555a089a6b90 con 0x555a089a7490 2023-10-05T19:28:28.630 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.626+0000 7f97be6b7700 1 -- 172.21.15.167:0/16775053 <== osd.3 v2:172.21.15.167:6808/30036 16 ==== osd_op_reply(61 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f97b8024020 con 0x555a089a7490 2023-10-05T19:28:28.630 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.626+0000 7f97be6b7700 1 -- 172.21.15.167:0/16775053 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:62 5.9 5:90000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x555a089a6b90 con 0x555a089707e0 2023-10-05T19:28:28.631 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.626+0000 7f97beeb8700 1 -- 172.21.15.167:0/16775053 <== osd.1 v2:172.21.15.167:6800/30035 22 ==== osd_op_reply(62 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f97b002f080 con 0x555a089707e0 2023-10-05T19:28:28.631 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.626+0000 7f97beeb8700 1 -- 172.21.15.167:0/16775053 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:63 5.19 5:98000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x555a089a6b90 con 0x555a089707e0 2023-10-05T19:28:28.631 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.626+0000 7f97beeb8700 1 -- 172.21.15.167:0/16775053 <== osd.1 v2:172.21.15.167:6800/30035 23 ==== osd_op_reply(63 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f97b002f080 con 0x555a089707e0 2023-10-05T19:28:28.632 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.626+0000 7f97beeb8700 1 -- 172.21.15.167:0/16775053 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:64 5.5 5:a0000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x555a089a6b90 con 0x7f97b4007030 2023-10-05T19:28:28.632 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.626+0000 7f97beeb8700 1 -- 172.21.15.167:0/16775053 <== osd.0 v2:172.21.15.118:6800/27690 16 ==== osd_op_reply(64 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f97b002f080 con 0x7f97b4007030 2023-10-05T19:28:28.632 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.626+0000 7f97beeb8700 1 -- 172.21.15.167:0/16775053 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:65 5.15 5:a8000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x555a089a6b90 con 0x555a089a7490 2023-10-05T19:28:28.633 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.626+0000 7f97be6b7700 1 -- 172.21.15.167:0/16775053 <== osd.3 v2:172.21.15.167:6808/30036 17 ==== osd_op_reply(65 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f97b8024020 con 0x555a089a7490 2023-10-05T19:28:28.634 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.626+0000 7f97be6b7700 1 -- 172.21.15.167:0/16775053 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:66 5.d 5:b0000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x555a089a6b90 con 0x7f97b8027be0 2023-10-05T19:28:28.634 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.630+0000 7f97bdeb6700 1 -- 172.21.15.167:0/16775053 <== osd.2 v2:172.21.15.167:6816/30037 10 ==== osd_op_reply(66 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f97b4008ae0 con 0x7f97b8027be0 2023-10-05T19:28:28.634 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.630+0000 7f97bdeb6700 1 -- 172.21.15.167:0/16775053 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:67 5.1d 5:b8000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x555a089a6b90 con 0x555a089707e0 2023-10-05T19:28:28.635 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.630+0000 7f97beeb8700 1 -- 172.21.15.167:0/16775053 <== osd.1 v2:172.21.15.167:6800/30035 24 ==== osd_op_reply(67 [pgnls start_epoch 30 out=107b] v30'2 uv2 ondisk = 1) v8 ==== 144+0+107 (crc 0 0 0) 0x7f97b002f080 con 0x555a089707e0 2023-10-05T19:28:28.635 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.630+0000 7f97beeb8700 1 -- 172.21.15.167:0/16775053 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:68 5.3 5:c0000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x555a089a6b90 con 0x7f97b4007030 2023-10-05T19:28:28.636 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.630+0000 7f97beeb8700 1 -- 172.21.15.167:0/16775053 <== osd.0 v2:172.21.15.118:6800/27690 17 ==== osd_op_reply(68 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f97b002f080 con 0x7f97b4007030 2023-10-05T19:28:28.636 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.630+0000 7f97beeb8700 1 -- 172.21.15.167:0/16775053 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:69 5.13 5:c8000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x555a089a6b90 con 0x555a089a7490 2023-10-05T19:28:28.637 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.630+0000 7f97be6b7700 1 -- 172.21.15.167:0/16775053 <== osd.3 v2:172.21.15.167:6808/30036 18 ==== osd_op_reply(69 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f97b8024020 con 0x555a089a7490 2023-10-05T19:28:28.637 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.630+0000 7f97be6b7700 1 -- 172.21.15.167:0/16775053 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:70 5.b 5:d0000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x555a089a6b90 con 0x7f97b4007030 2023-10-05T19:28:28.638 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.634+0000 7f97beeb8700 1 -- 172.21.15.167:0/16775053 <== osd.0 v2:172.21.15.118:6800/27690 18 ==== osd_op_reply(70 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f97b002f080 con 0x7f97b4007030 2023-10-05T19:28:28.638 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.634+0000 7f97beeb8700 1 -- 172.21.15.167:0/16775053 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:71 5.1b 5:d8000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x555a089a6b90 con 0x7f97b8027be0 2023-10-05T19:28:28.639 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.634+0000 7f97bdeb6700 1 -- 172.21.15.167:0/16775053 <== osd.2 v2:172.21.15.167:6816/30037 11 ==== osd_op_reply(71 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f97b4008ae0 con 0x7f97b8027be0 2023-10-05T19:28:28.639 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.634+0000 7f97bdeb6700 1 -- 172.21.15.167:0/16775053 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:72 5.7 5:e0000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x555a089a6b90 con 0x555a089a7490 2023-10-05T19:28:28.640 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.634+0000 7f97be6b7700 1 -- 172.21.15.167:0/16775053 <== osd.3 v2:172.21.15.167:6808/30036 19 ==== osd_op_reply(72 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f97b8024020 con 0x555a089a7490 2023-10-05T19:28:28.640 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.634+0000 7f97be6b7700 1 -- 172.21.15.167:0/16775053 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:73 5.17 5:e8000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x555a089a6b90 con 0x7f97b4007030 2023-10-05T19:28:28.641 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.634+0000 7f97beeb8700 1 -- 172.21.15.167:0/16775053 <== osd.0 v2:172.21.15.118:6800/27690 19 ==== osd_op_reply(73 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f97b002f080 con 0x7f97b4007030 2023-10-05T19:28:28.641 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.634+0000 7f97beeb8700 1 -- 172.21.15.167:0/16775053 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:74 5.f 5:f0000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x555a089a6b90 con 0x555a089707e0 2023-10-05T19:28:28.642 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.634+0000 7f97beeb8700 1 -- 172.21.15.167:0/16775053 <== osd.1 v2:172.21.15.167:6800/30035 25 ==== osd_op_reply(74 [pgnls start_epoch 30 out=115b] v23'2 uv2 ondisk = 1) v8 ==== 144+0+115 (crc 0 0 0) 0x7f97b002f080 con 0x555a089707e0 2023-10-05T19:28:28.642 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.638+0000 7f97beeb8700 1 -- 172.21.15.167:0/16775053 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:75 5.1f 5:f8000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x555a089a6b90 con 0x7f97b8027be0 2023-10-05T19:28:28.643 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.638+0000 7f97bdeb6700 1 -- 172.21.15.167:0/16775053 <== osd.2 v2:172.21.15.167:6816/30037 12 ==== osd_op_reply(75 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f97b4008ae0 con 0x7f97b8027be0 2023-10-05T19:28:28.643 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.638+0000 7f97c0b82e40 20 RGWRados::pool_iterate: got zonegroup_info.4f1f719d-9648-417b-b10c-e5d939825419 2023-10-05T19:28:28.643 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.638+0000 7f97c0b82e40 20 RGWRados::pool_iterate: got zone_info.9a9df965-6612-45d4-a45f-6e3c889bacc2 2023-10-05T19:28:28.643 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.638+0000 7f97c0b82e40 20 RGWRados::pool_iterate: got zone_names.default 2023-10-05T19:28:28.643 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.638+0000 7f97c0b82e40 20 RGWRados::pool_iterate: got zonegroups_names.default 2023-10-05T19:28:28.643 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.638+0000 7f97c0b82e40 20 rados->read ofs=0 len=0 2023-10-05T19:28:28.643 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.638+0000 7f97c0b82e40 1 -- 172.21.15.167:0/16775053 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:76 5.f 5:f43b6ece:::zone_names.default:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x555a089aa510 con 0x555a089707e0 2023-10-05T19:28:28.645 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.638+0000 7f97beeb8700 1 -- 172.21.15.167:0/16775053 <== osd.1 v2:172.21.15.167:6800/30035 26 ==== osd_op_reply(76 zone_names.default [read 0~46 out=46b] v0'0 uv1 ondisk = 0) v8 ==== 162+0+46 (crc 0 0 0) 0x7f97b002f080 con 0x555a089707e0 2023-10-05T19:28:28.645 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.638+0000 7f97c0b82e40 20 rados_obj.operate() r=0 bl.length=46 2023-10-05T19:28:28.645 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.638+0000 7f97c0b82e40 20 rados->read ofs=0 len=0 2023-10-05T19:28:28.645 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.638+0000 7f97c0b82e40 1 -- 172.21.15.167:0/16775053 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:77 5.1d 5:b8374eb4:::zone_info.9a9df965-6612-45d4-a45f-6e3c889bacc2:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x555a089aa910 con 0x555a089707e0 2023-10-05T19:28:28.645 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.638+0000 7f97beeb8700 1 -- 172.21.15.167:0/16775053 <== osd.1 v2:172.21.15.167:6800/30035 27 ==== osd_op_reply(77 zone_info.9a9df965-6612-45d4-a45f-6e3c889bacc2 [read 0~874 out=874b] v0'0 uv2 ondisk = 0) v8 ==== 190+0+874 (crc 0 0 0) 0x7f97b002f080 con 0x555a089707e0 2023-10-05T19:28:28.645 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.638+0000 7f97c0b82e40 20 rados_obj.operate() r=0 bl.length=874 2023-10-05T19:28:28.645 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.638+0000 7f97c0b82e40 20 rados->read ofs=0 len=0 2023-10-05T19:28:28.645 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.638+0000 7f97c0b82e40 1 -- 172.21.15.167:0/16775053 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:78 5.f 5:f4c53578:::zonegroups_names.default:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x555a089aae00 con 0x555a089707e0 2023-10-05T19:28:28.646 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.638+0000 7f97beeb8700 1 -- 172.21.15.167:0/16775053 <== osd.1 v2:172.21.15.167:6800/30035 28 ==== osd_op_reply(78 zonegroups_names.default [read 0~46 out=46b] v0'0 uv2 ondisk = 0) v8 ==== 168+0+46 (crc 0 0 0) 0x7f97b002f080 con 0x555a089707e0 2023-10-05T19:28:28.646 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.638+0000 7f97c0b82e40 20 rados_obj.operate() r=0 bl.length=46 2023-10-05T19:28:28.646 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.638+0000 7f97c0b82e40 20 rados->read ofs=0 len=0 2023-10-05T19:28:28.646 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.638+0000 7f97c0b82e40 1 -- 172.21.15.167:0/16775053 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:79 5.c 5:30d299a5:::zonegroup_info.4f1f719d-9648-417b-b10c-e5d939825419:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x555a089aae00 con 0x7f97b4007030 2023-10-05T19:28:28.647 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.642+0000 7f97beeb8700 1 -- 172.21.15.167:0/16775053 <== osd.0 v2:172.21.15.118:6800/27690 20 ==== osd_op_reply(79 zonegroup_info.4f1f719d-9648-417b-b10c-e5d939825419 [read 0~362 out=362b] v0'0 uv1 ondisk = 0) v8 ==== 195+0+362 (crc 0 0 0) 0x7f97b002f080 con 0x7f97b4007030 2023-10-05T19:28:28.647 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.642+0000 7f97c0b82e40 20 rados_obj.operate() r=0 bl.length=362 2023-10-05T19:28:28.647 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.642+0000 7f97c0b82e40 20 rados->read ofs=0 len=0 2023-10-05T19:28:28.647 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.642+0000 7f97c0b82e40 1 -- 172.21.15.167:0/16775053 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:80 5.3 5:c0ad8cbd:::region_map:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x555a089aaf00 con 0x7f97b4007030 2023-10-05T19:28:28.647 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.642+0000 7f97beeb8700 1 -- 172.21.15.167:0/16775053 <== osd.0 v2:172.21.15.118:6800/27690 21 ==== osd_op_reply(80 region_map [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 ==== 154+0+0 (crc 0 0 0) 0x7f97b0050040 con 0x7f97b4007030 2023-10-05T19:28:28.648 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.642+0000 7f97c0b82e40 20 rados_obj.operate() r=-2 bl.length=0 2023-10-05T19:28:28.648 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.642+0000 7f97c0b82e40 20 zone default found 2023-10-05T19:28:28.648 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.642+0000 7f97c0b82e40 4 Realm: () 2023-10-05T19:28:28.648 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.642+0000 7f97c0b82e40 4 ZoneGroup: default (4f1f719d-9648-417b-b10c-e5d939825419) 2023-10-05T19:28:28.648 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.642+0000 7f97c0b82e40 4 Zone: default (9a9df965-6612-45d4-a45f-6e3c889bacc2) 2023-10-05T19:28:28.648 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.642+0000 7f97c0b82e40 10 cannot find current period zonegroup using local zonegroup configuration 2023-10-05T19:28:28.648 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.642+0000 7f97c0b82e40 20 zonegroup default 2023-10-05T19:28:28.648 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.642+0000 7f97c0b82e40 20 rados->read ofs=0 len=0 2023-10-05T19:28:28.648 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.642+0000 7f97c0b82e40 1 -- 172.21.15.167:0/16775053 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:81 5.3 5:c52100b6:::period_config.default:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x555a089aaf00 con 0x7f97b4007030 2023-10-05T19:28:28.650 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.642+0000 7f97beeb8700 1 -- 172.21.15.167:0/16775053 <== osd.0 v2:172.21.15.118:6800/27690 22 ==== osd_op_reply(81 period_config.default [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 ==== 165+0+0 (crc 0 0 0) 0x7f97b0050040 con 0x7f97b4007030 2023-10-05T19:28:28.650 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.642+0000 7f97c0b82e40 20 rados_obj.operate() r=-2 bl.length=0 2023-10-05T19:28:28.650 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.642+0000 7f97c0b82e40 20 rados->read ofs=0 len=0 2023-10-05T19:28:28.650 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.642+0000 7f97c0b82e40 1 -- 172.21.15.167:0/16775053 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:82 6.2 6:4347d321:::bucket.sync-source-hints.:head [call version.read in=11b,read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x555a089ac880 con 0x555a089a7490 2023-10-05T19:28:28.650 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.642+0000 7f97be6b7700 1 -- 172.21.15.167:0/16775053 <== osd.3 v2:172.21.15.167:6808/30036 20 ==== osd_op_reply(82 bucket.sync-source-hints. [call,read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 ==== 211+0+0 (crc 0 0 0) 0x7f97b8024020 con 0x555a089a7490 2023-10-05T19:28:28.650 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.642+0000 7f97c0b82e40 20 rados_obj.operate() r=-2 bl.length=0 2023-10-05T19:28:28.650 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.642+0000 7f97c0b82e40 20 rados->read ofs=0 len=0 2023-10-05T19:28:28.651 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.642+0000 7f97c0b82e40 1 -- 172.21.15.167:0/16775053 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:83 6.b 6:d467b91b:::bucket.sync-target-hints.:head [call version.read in=11b,read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x555a089acc30 con 0x555a089a7490 2023-10-05T19:28:28.651 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.642+0000 7f97be6b7700 1 -- 172.21.15.167:0/16775053 <== osd.3 v2:172.21.15.167:6808/30036 21 ==== osd_op_reply(83 bucket.sync-target-hints. [call,read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 ==== 211+0+0 (crc 0 0 0) 0x7f97b8024020 con 0x555a089a7490 2023-10-05T19:28:28.651 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.642+0000 7f97c0b82e40 20 rados_obj.operate() r=-2 bl.length=0 2023-10-05T19:28:28.651 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.642+0000 7f97c0b82e40 20 started sync module instance, tier type = 2023-10-05T19:28:28.651 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.642+0000 7f97c0b82e40 20 started zone id=9a9df965-6612-45d4-a45f-6e3c889bacc2 (name=default) with tier type = 2023-10-05T19:28:28.651 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.642+0000 7f97c0b82e40 1 -- 172.21.15.167:0/16775053 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:84 7.1f 7:f95f44c2:::notify.0:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x555a089abf40 con 0x7f97b8027be0 2023-10-05T19:28:28.652 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.646+0000 7f97bdeb6700 1 -- 172.21.15.167:0/16775053 <== osd.2 v2:172.21.15.167:6816/30037 13 ==== osd_op_reply(84 notify.0 [create] v30'63 uv63 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f97b4008ae0 con 0x7f97b8027be0 2023-10-05T19:28:28.653 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.646+0000 7f97c0b82e40 1 -- 172.21.15.167:0/16775053 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:85 7.1f 7:f95f44c2:::notify.0:head [watch watch cookie 93845179775568] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x555a089acfe0 con 0x7f97b8027be0 2023-10-05T19:28:28.653 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.646+0000 7f97c0b82e40 1 -- 172.21.15.167:0/16775053 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:86 7.0 7:05bf5b68:::notify.1:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x555a089ad950 con 0x555a089707e0 2023-10-05T19:28:28.656 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.650+0000 7f97bdeb6700 1 -- 172.21.15.167:0/16775053 <== osd.2 v2:172.21.15.167:6816/30037 14 ==== osd_op_reply(85 notify.0 [watch watch cookie 93845179775568] v30'64 uv63 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f97b4008ae0 con 0x7f97b8027be0 2023-10-05T19:28:28.656 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.650+0000 7f97beeb8700 1 -- 172.21.15.167:0/16775053 <== osd.1 v2:172.21.15.167:6800/30035 29 ==== osd_op_reply(86 notify.1 [create] v30'63 uv63 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f97b0050040 con 0x555a089707e0 2023-10-05T19:28:28.656 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.650+0000 7f97c0b82e40 1 -- 172.21.15.167:0/16775053 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:87 7.0 7:05bf5b68:::notify.1:head [watch watch cookie 93845179777856] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x555a089add50 con 0x555a089707e0 2023-10-05T19:28:28.656 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.650+0000 7f97c0b82e40 1 -- 172.21.15.167:0/16775053 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:88 7.15 7:a93a5511:::notify.2:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x555a089ae5f0 con 0x555a089a7490 2023-10-05T19:28:28.658 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.654+0000 7f97beeb8700 1 -- 172.21.15.167:0/16775053 <== osd.1 v2:172.21.15.167:6800/30035 30 ==== osd_op_reply(87 notify.1 [watch watch cookie 93845179777856] v30'64 uv63 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f97b0050040 con 0x555a089707e0 2023-10-05T19:28:28.659 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.654+0000 7f97be6b7700 1 -- 172.21.15.167:0/16775053 <== osd.3 v2:172.21.15.167:6808/30036 22 ==== osd_op_reply(88 notify.2 [create] v30'63 uv63 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f97b8024020 con 0x555a089a7490 2023-10-05T19:28:28.659 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.654+0000 7f97c0b82e40 1 -- 172.21.15.167:0/16775053 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:89 7.15 7:a93a5511:::notify.2:head [watch watch cookie 93845179784528] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x555a089ad540 con 0x555a089a7490 2023-10-05T19:28:28.659 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.654+0000 7f97c0b82e40 1 -- 172.21.15.167:0/16775053 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:90 7.e 7:7759931f:::notify.3:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x555a089af380 con 0x7f97b8027be0 2023-10-05T19:28:28.662 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.654+0000 7f97bdeb6700 1 -- 172.21.15.167:0/16775053 <== osd.2 v2:172.21.15.167:6816/30037 15 ==== osd_op_reply(90 notify.3 [create] v30'63 uv63 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f97b4008ae0 con 0x7f97b8027be0 2023-10-05T19:28:28.662 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.654+0000 7f97c0b82e40 1 -- 172.21.15.167:0/16775053 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:91 7.e 7:7759931f:::notify.3:head [watch watch cookie 93845179788944] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x555a089afc20 con 0x7f97b8027be0 2023-10-05T19:28:28.662 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.658+0000 7f97c0b82e40 1 -- 172.21.15.167:0/16775053 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:92 7.d 7:b4812045:::notify.4:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x555a089b0780 con 0x555a089707e0 2023-10-05T19:28:28.662 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.658+0000 7f97be6b7700 1 -- 172.21.15.167:0/16775053 <== osd.3 v2:172.21.15.167:6808/30036 23 ==== osd_op_reply(89 notify.2 [watch watch cookie 93845179784528] v30'64 uv63 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f97b8024020 con 0x555a089a7490 2023-10-05T19:28:28.664 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.658+0000 7f97bdeb6700 1 -- 172.21.15.167:0/16775053 <== osd.2 v2:172.21.15.167:6816/30037 16 ==== osd_op_reply(91 notify.3 [watch watch cookie 93845179788944] v30'64 uv63 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f97b4008ae0 con 0x7f97b8027be0 2023-10-05T19:28:28.665 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.658+0000 7f97beeb8700 1 -- 172.21.15.167:0/16775053 <== osd.1 v2:172.21.15.167:6800/30035 31 ==== osd_op_reply(92 notify.4 [create] v30'63 uv63 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f97b0050040 con 0x555a089707e0 2023-10-05T19:28:28.665 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.658+0000 7f97c0b82e40 1 -- 172.21.15.167:0/16775053 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:93 7.d 7:b4812045:::notify.4:head [watch watch cookie 93845179791152] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x555a089b0250 con 0x555a089707e0 2023-10-05T19:28:28.665 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.658+0000 7f97c0b82e40 1 -- 172.21.15.167:0/16775053 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:94 7.3 7:c609908c:::notify.5:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x555a089b1070 con 0x7f97b4007030 2023-10-05T19:28:28.667 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.662+0000 7f97beeb8700 1 -- 172.21.15.167:0/16775053 <== osd.0 v2:172.21.15.118:6800/27690 23 ==== osd_op_reply(94 notify.5 [create] v30'63 uv63 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f97b0050040 con 0x7f97b4007030 2023-10-05T19:28:28.667 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.662+0000 7f97c0b82e40 1 -- 172.21.15.167:0/16775053 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:95 7.3 7:c609908c:::notify.5:head [watch watch cookie 93845179796272] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x555a089b1470 con 0x7f97b4007030 2023-10-05T19:28:28.667 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.662+0000 7f97c0b82e40 1 -- 172.21.15.167:0/16775053 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:96 7.14 7:2b04a3e9:::notify.6:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x555a089b1f50 con 0x7f97b4007030 2023-10-05T19:28:28.668 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.662+0000 7f97beeb8700 1 -- 172.21.15.167:0/16775053 <== osd.1 v2:172.21.15.167:6800/30035 32 ==== osd_op_reply(93 notify.4 [watch watch cookie 93845179791152] v30'64 uv63 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f97b0050040 con 0x555a089707e0 2023-10-05T19:28:28.669 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.662+0000 7f97beeb8700 1 -- 172.21.15.167:0/16775053 <== osd.0 v2:172.21.15.118:6800/27690 24 ==== osd_op_reply(95 notify.5 [watch watch cookie 93845179796272] v30'64 uv63 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f97b0050040 con 0x7f97b4007030 2023-10-05T19:28:28.670 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.666+0000 7f97beeb8700 1 -- 172.21.15.167:0/16775053 <== osd.0 v2:172.21.15.118:6800/27690 25 ==== osd_op_reply(96 notify.6 [create] v30'63 uv63 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f97b0050040 con 0x7f97b4007030 2023-10-05T19:28:28.670 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.666+0000 7f97c0b82e40 1 -- 172.21.15.167:0/16775053 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:97 7.14 7:2b04a3e9:::notify.6:head [watch watch cookie 93845179787680] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x555a089b0f80 con 0x7f97b4007030 2023-10-05T19:28:28.670 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.666+0000 7f97c0b82e40 1 -- 172.21.15.167:0/16775053 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:98 7.9 7:93e5b521:::notify.7:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x555a089b2670 con 0x7f97b4007030 2023-10-05T19:28:28.672 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.666+0000 7f97beeb8700 1 -- 172.21.15.167:0/16775053 <== osd.0 v2:172.21.15.118:6800/27690 26 ==== osd_op_reply(97 notify.6 [watch watch cookie 93845179787680] v30'64 uv63 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f97b0050040 con 0x7f97b4007030 2023-10-05T19:28:28.673 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.666+0000 7f97beeb8700 1 -- 172.21.15.167:0/16775053 <== osd.0 v2:172.21.15.118:6800/27690 27 ==== osd_op_reply(98 notify.7 [create] v30'63 uv63 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f97b0050040 con 0x7f97b4007030 2023-10-05T19:28:28.673 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.666+0000 7f97c0b82e40 1 -- 172.21.15.167:0/16775053 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:99 7.9 7:93e5b521:::notify.7:head [watch watch cookie 93845179800096] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x555a089b0f80 con 0x7f97b4007030 2023-10-05T19:28:28.673 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.666+0000 7f97c0b82e40 20 add_watcher() i=0 2023-10-05T19:28:28.673 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.666+0000 7f97c0b82e40 20 add_watcher() i=1 2023-10-05T19:28:28.673 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.666+0000 7f97c0b82e40 20 add_watcher() i=2 2023-10-05T19:28:28.673 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.666+0000 7f97c0b82e40 20 add_watcher() i=3 2023-10-05T19:28:28.673 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.666+0000 7f97c0b82e40 20 add_watcher() i=4 2023-10-05T19:28:28.673 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.666+0000 7f97c0b82e40 20 add_watcher() i=5 2023-10-05T19:28:28.673 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.666+0000 7f97c0b82e40 20 add_watcher() i=6 2023-10-05T19:28:28.674 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.670+0000 7f97beeb8700 1 -- 172.21.15.167:0/16775053 <== osd.0 v2:172.21.15.118:6800/27690 28 ==== osd_op_reply(99 notify.7 [watch watch cookie 93845179800096] v30'64 uv63 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f97b0050040 con 0x7f97b4007030 2023-10-05T19:28:28.674 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.670+0000 7f97c0b82e40 20 add_watcher() i=7 2023-10-05T19:28:28.675 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.670+0000 7f97c0b82e40 2 all 8 watchers are set, enabling cache 2023-10-05T19:28:28.675 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.670+0000 7f97c0b82e40 1 -- 172.21.15.167:0/16775053 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:100 6.4 6:22d26bf9:::data_loggenerations_metadata:head [call version.check_conds in=74b,call version.read in=11b,read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x555a089b5730 con 0x555a089707e0 2023-10-05T19:28:28.675 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.670+0000 7f97beeb8700 1 -- 172.21.15.167:0/16775053 <== osd.1 v2:172.21.15.167:6800/30035 33 ==== osd_op_reply(100 data_loggenerations_metadata [call,call out=48b,read 0~28 out=28b] v0'0 uv1 ondisk = 0) v8 ==== 256+0+76 (crc 0 0 0) 0x7f97b0050040 con 0x555a089707e0 2023-10-05T19:28:28.675 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.670+0000 7f97c0b82e40 1 -- 172.21.15.167:0/16775053 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:101 6.4 6:22d26bf9:::data_loggenerations_metadata:head [watch watch cookie 93845179801984] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x555a089b2bf0 con 0x555a089707e0 2023-10-05T19:28:28.677 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.670+0000 7f97beeb8700 1 -- 172.21.15.167:0/16775053 <== osd.1 v2:172.21.15.167:6800/30035 34 ==== osd_op_reply(101 data_loggenerations_metadata [watch watch cookie 93845179801984] v30'76 uv1 ondisk = 0) v8 ==== 172+0+0 (crc 0 0 0) 0x7f97b0050040 con 0x555a089707e0 2023-10-05T19:28:28.677 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.670+0000 7f978ffdf700 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2023-10-05T19:28:28.677 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.670+0000 7f97c0b82e40 20 check_secure_mon_conn(): auth registy supported: methods=[2] modes=[2,1] 2023-10-05T19:28:28.677 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.670+0000 7f97c0b82e40 20 check_secure_mon_conn(): mode 1 is insecure 2023-10-05T19:28:28.678 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.670+0000 7f97c0b82e40 1 Processor -- start 2023-10-05T19:28:28.678 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.674+0000 7f97c0b82e40 1 -- start start 2023-10-05T19:28:28.678 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.674+0000 7f97c0b82e40 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x555a089bfb10 0x555a089bfee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-05T19:28:28.678 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.674+0000 7f97c0b82e40 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x555a089c0420 con 0x555a089bfb10 2023-10-05T19:28:28.679 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.674+0000 7f97bdeb6700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x555a089bfb10 0x555a089bfee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-05T19:28:28.679 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.674+0000 7f97bdeb6700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x555a089bfb10 0x555a089bfee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.118:3300/0 says I am v2:172.21.15.167:57750/0 (socket says 172.21.15.167:57750) 2023-10-05T19:28:28.679 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.674+0000 7f97bdeb6700 1 -- 172.21.15.167:0/3754723222 learned_addr learned my addr 172.21.15.167:0/3754723222 (peer_addr_for_me v2:172.21.15.167:0/0) 2023-10-05T19:28:28.680 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.674+0000 7f97bdeb6700 1 -- 172.21.15.167:0/3754723222 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x555a089c0560 con 0x555a089bfb10 2023-10-05T19:28:28.680 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.674+0000 7f97bdeb6700 1 --2- 172.21.15.167:0/3754723222 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x555a089bfb10 0x555a089bfee0 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 crypto rx=0x7f97b4002000 tx=0x7f97b4012000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e7236b38f01ee8b4 server_cookie=0 in_seq=0 out_seq=0 2023-10-05T19:28:28.680 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.674+0000 7f978f7de700 1 -- 172.21.15.167:0/3754723222 <== mon.0 v2:172.21.15.118:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f97b4012c90 con 0x555a089bfb10 2023-10-05T19:28:28.680 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.674+0000 7f978f7de700 1 -- 172.21.15.167:0/3754723222 <== mon.0 v2:172.21.15.118:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f97b4012df0 con 0x555a089bfb10 2023-10-05T19:28:28.680 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.674+0000 7f978f7de700 1 -- 172.21.15.167:0/3754723222 <== mon.0 v2:172.21.15.118:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f97b402a8a0 con 0x555a089bfb10 2023-10-05T19:28:28.681 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.674+0000 7f97c0b82e40 1 -- 172.21.15.167:0/3754723222 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x555a089bfb10 msgr2=0x555a089bfee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-05T19:28:28.681 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.674+0000 7f97c0b82e40 1 --2- 172.21.15.167:0/3754723222 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x555a089bfb10 0x555a089bfee0 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 crypto rx=0x7f97b4002000 tx=0x7f97b4012000 comp rx=0 tx=0).stop 2023-10-05T19:28:28.682 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.674+0000 7f97c0b82e40 1 -- 172.21.15.167:0/3754723222 shutdown_connections 2023-10-05T19:28:28.682 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.674+0000 7f97c0b82e40 1 --2- 172.21.15.167:0/3754723222 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x555a089bfb10 0x555a089bfee0 unknown :-1 s=CLOSED pgs=264 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-05T19:28:28.682 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.674+0000 7f97c0b82e40 1 -- 172.21.15.167:0/3754723222 >> 172.21.15.167:0/3754723222 conn(0x555a089baed0 msgr2=0x555a089b9080 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-05T19:28:28.682 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.674+0000 7f97c0b82e40 1 -- 172.21.15.167:0/3754723222 shutdown_connections 2023-10-05T19:28:28.682 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.674+0000 7f97c0b82e40 1 -- 172.21.15.167:0/3754723222 wait complete. 2023-10-05T19:28:28.682 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.674+0000 7f97c0b82e40 1 Processor -- start 2023-10-05T19:28:28.683 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.678+0000 7f97c0b82e40 1 -- start start 2023-10-05T19:28:28.683 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.678+0000 7f97c0b82e40 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x555a089bfb10 0x555a089c5550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-05T19:28:28.683 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.678+0000 7f97c0b82e40 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x555a089c6550 con 0x555a089bfb10 2023-10-05T19:28:28.683 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.678+0000 7f97bdeb6700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x555a089bfb10 0x555a089c5550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-05T19:28:28.684 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.678+0000 7f97bdeb6700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x555a089bfb10 0x555a089c5550 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.118:3300/0 says I am v2:172.21.15.167:57754/0 (socket says 172.21.15.167:57754) 2023-10-05T19:28:28.684 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.678+0000 7f97bdeb6700 1 -- 172.21.15.167:0/1911393562 learned_addr learned my addr 172.21.15.167:0/1911393562 (peer_addr_for_me v2:172.21.15.167:0/0) 2023-10-05T19:28:28.684 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.678+0000 7f97bdeb6700 1 -- 172.21.15.167:0/1911393562 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x555a0896fcd0 con 0x555a089bfb10 2023-10-05T19:28:28.684 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.678+0000 7f97bdeb6700 1 --2- 172.21.15.167:0/1911393562 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x555a089bfb10 0x555a089c5550 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 crypto rx=0x555a089b9bf0 tx=0x7f97b402f000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-05T19:28:28.684 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.678+0000 7f978dfdb700 1 -- 172.21.15.167:0/1911393562 <== mon.0 v2:172.21.15.118:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f97b403eb20 con 0x555a089bfb10 2023-10-05T19:28:28.685 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.678+0000 7f978dfdb700 1 -- 172.21.15.167:0/1911393562 <== mon.0 v2:172.21.15.118:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f97b403ec80 con 0x555a089bfb10 2023-10-05T19:28:28.685 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.678+0000 7f978dfdb700 1 -- 172.21.15.167:0/1911393562 <== mon.0 v2:172.21.15.118:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f97b403ee10 con 0x555a089bfb10 2023-10-05T19:28:28.685 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.678+0000 7f97c0b82e40 1 -- 172.21.15.167:0/1911393562 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x555a089c5b50 con 0x555a089bfb10 2023-10-05T19:28:28.685 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.678+0000 7f97c0b82e40 1 -- 172.21.15.167:0/1911393562 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x555a089c5f90 con 0x555a089bfb10 2023-10-05T19:28:28.685 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.678+0000 7f978dfdb700 1 -- 172.21.15.167:0/1911393562 <== mon.0 v2:172.21.15.118:3300/0 4 ==== mgrmap(e 5) v1 ==== 40329+0+0 (secure 0 0 0) 0x7f97b4035400 con 0x555a089bfb10 2023-10-05T19:28:28.686 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.678+0000 7f978dfdb700 1 --2- 172.21.15.167:0/1911393562 >> [v2:172.21.15.167:6824/29889,v1:172.21.15.167:6825/29889] conn(0x7f9704030c00 0x7f97040330b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-05T19:28:28.686 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.678+0000 7f978dfdb700 1 -- 172.21.15.167:0/1911393562 <== mon.0 v2:172.21.15.118:3300/0 5 ==== osd_map(30..30 src has 1..30) v4 ==== 6057+0+0 (secure 0 0 0) 0x7f97b402f070 con 0x555a089bfb10 2023-10-05T19:28:28.686 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.682+0000 7f97c0b82e40 5 note: GC not initialized 2023-10-05T19:28:28.686 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.682+0000 7f97beeb8700 1 --2- 172.21.15.167:0/1911393562 >> [v2:172.21.15.167:6824/29889,v1:172.21.15.167:6825/29889] conn(0x7f9704030c00 0x7f97040330b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-05T19:28:28.687 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.682+0000 7f97beeb8700 1 --2- 172.21.15.167:0/1911393562 >> [v2:172.21.15.167:6824/29889,v1:172.21.15.167:6825/29889] conn(0x7f9704030c00 0x7f97040330b0 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x555a089b1200 tx=0x7f97b0000f90 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-05T19:28:28.688 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.682+0000 7f96feffd700 20 reqs_thread_entry: start 2023-10-05T19:28:28.688 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.682+0000 7f96fcff9700 20 reqs_thread_entry: start 2023-10-05T19:28:28.689 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.682+0000 7f97c0b82e40 20 init_complete bucket index max shards: 11 2023-10-05T19:28:28.689 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.682+0000 7f97c0b82e40 10 rgw notify: Started notification manager with: 1 workers 2023-10-05T19:28:28.689 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.682+0000 7f96f2ffd700 20 rgw notify: INFO: next queues processing will happen at: Thu Oct 5 19:28:58 2023 2023-10-05T19:28:28.690 INFO:teuthology.orchestra.run.smithi167.stderr: 2023-10-05T19:28:28.690 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.682+0000 7f97c0b82e40 20 get_system_obj_state: rctx=0x7ffe8828c0d8 obj=default.rgw.meta:users.uid:foo state=0x555a088490b0 s->prefetch_data=0 2023-10-05T19:28:28.690 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.682+0000 7f97c0b82e40 1 -- 172.21.15.167:0/16775053 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:102 8.18 8:1e6ca5ed:users.uid::foo:head [call version.read in=11b,getxattrs,stat] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x555a08a2dcc0 con 0x555a089707e0 2023-10-05T19:28:28.690 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.682+0000 7f96f27fc700 20 reqs_thread_entry: start 2023-10-05T19:28:28.691 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.686+0000 7f97beeb8700 1 -- 172.21.15.167:0/16775053 <== osd.1 v2:172.21.15.167:6800/30035 35 ==== osd_op_reply(102 foo [call out=48b,getxattrs out=75b,stat out=16b] v0'0 uv7 ondisk = 0) v8 ==== 231+0+139 (crc 0 0 0) 0x7f97b0050040 con 0x555a089707e0 2023-10-05T19:28:28.691 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.686+0000 7f97c0b82e40 20 get_system_obj_state: s->obj_tag was set empty 2023-10-05T19:28:28.691 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.686+0000 7f97c0b82e40 20 Read xattr: user.rgw.idtag 2023-10-05T19:28:28.691 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.686+0000 7f97c0b82e40 20 rados->read ofs=0 len=0 2023-10-05T19:28:28.691 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.686+0000 7f97c0b82e40 1 -- 172.21.15.167:0/16775053 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:103 8.18 8:1e6ca5ed:users.uid::foo:head [call version.check_conds in=122b,call version.read in=11b,read 0~0,getxattrs] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x555a08a2dcc0 con 0x555a089707e0 2023-10-05T19:28:28.691 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.686+0000 7f97beeb8700 1 -- 172.21.15.167:0/16775053 <== osd.1 v2:172.21.15.167:6800/30035 36 ==== osd_op_reply(103 foo [call,call out=48b,read 0~513 out=513b,getxattrs out=75b] v0'0 uv7 ondisk = 0) v8 ==== 273+0+636 (crc 0 0 0) 0x7f97b0050040 con 0x555a089707e0 2023-10-05T19:28:28.691 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.686+0000 7f97c0b82e40 20 rados_obj.operate() r=0 bl.length=513 2023-10-05T19:28:28.692 INFO:teuthology.orchestra.run.smithi167.stdout:{ 2023-10-05T19:28:28.692 INFO:teuthology.orchestra.run.smithi167.stdout: "user_id": "foo", 2023-10-05T19:28:28.692 INFO:teuthology.orchestra.run.smithi167.stdout: "display_name": "Foo", 2023-10-05T19:28:28.692 INFO:teuthology.orchestra.run.smithi167.stdout: "email": "foo@foo.com", 2023-10-05T19:28:28.692 INFO:teuthology.orchestra.run.smithi167.stdout: "suspended": 0, 2023-10-05T19:28:28.692 INFO:teuthology.orchestra.run.smithi167.stdout: "max_buckets": 4, 2023-10-05T19:28:28.692 INFO:teuthology.orchestra.run.smithi167.stdout: "subusers": [ 2023-10-05T19:28:28.692 INFO:teuthology.orchestra.run.smithi167.stdout: { 2023-10-05T19:28:28.692 INFO:teuthology.orchestra.run.smithi167.stdout: "id": "foo:foo1", 2023-10-05T19:28:28.692 INFO:teuthology.orchestra.run.smithi167.stdout: "permissions": "full-control" 2023-10-05T19:28:28.692 INFO:teuthology.orchestra.run.smithi167.stdout: } 2023-10-05T19:28:28.693 INFO:teuthology.orchestra.run.smithi167.stdout: ], 2023-10-05T19:28:28.693 INFO:teuthology.orchestra.run.smithi167.stdout: "keys": [ 2023-10-05T19:28:28.693 INFO:teuthology.orchestra.run.smithi167.stdout: { 2023-10-05T19:28:28.693 INFO:teuthology.orchestra.run.smithi167.stdout: "user": "foo", 2023-10-05T19:28:28.693 INFO:teuthology.orchestra.run.smithi167.stdout: "access_key": "9te6NH5mcdcq0Tc5i8i1", 2023-10-05T19:28:28.693 INFO:teuthology.orchestra.run.smithi167.stdout: "secret_key": "Ny4IOauQoL18Gp2zM7lC1vLmoawgqcYP/YGcWfXu" 2023-10-05T19:28:28.693 INFO:teuthology.orchestra.run.smithi167.stdout: } 2023-10-05T19:28:28.693 INFO:teuthology.orchestra.run.smithi167.stdout: ], 2023-10-05T19:28:28.693 INFO:teuthology.orchestra.run.smithi167.stdout: "swift_keys": [ 2023-10-05T19:28:28.693 INFO:teuthology.orchestra.run.smithi167.stdout: { 2023-10-05T19:28:28.693 INFO:teuthology.orchestra.run.smithi167.stdout: "user": "foo:foo1", 2023-10-05T19:28:28.693 INFO:teuthology.orchestra.run.smithi167.stdout: "secret_key": "gpS2G9RREMrnbqlp29PP2D36kgPR1tm72n5fPYfL" 2023-10-05T19:28:28.694 INFO:teuthology.orchestra.run.smithi167.stdout: } 2023-10-05T19:28:28.694 INFO:teuthology.orchestra.run.smithi167.stdout: ], 2023-10-05T19:28:28.694 INFO:teuthology.orchestra.run.smithi167.stdout: "caps": [], 2023-10-05T19:28:28.694 INFO:teuthology.orchestra.run.smithi167.stdout: "op_mask": "read, write, delete", 2023-10-05T19:28:28.694 INFO:teuthology.orchestra.run.smithi167.stdout: "default_placement": "", 2023-10-05T19:28:28.694 INFO:teuthology.orchestra.run.smithi167.stdout: "default_storage_class": "", 2023-10-05T19:28:28.694 INFO:teuthology.orchestra.run.smithi167.stdout: "placement_tags": [], 2023-10-05T19:28:28.694 INFO:teuthology.orchestra.run.smithi167.stdout: "bucket_quota": { 2023-10-05T19:28:28.694 INFO:teuthology.orchestra.run.smithi167.stdout: "enabled": false, 2023-10-05T19:28:28.694 INFO:teuthology.orchestra.run.smithi167.stdout: "check_on_raw": false, 2023-10-05T19:28:28.694 INFO:teuthology.orchestra.run.smithi167.stdout: "max_size": -1, 2023-10-05T19:28:28.695 INFO:teuthology.orchestra.run.smithi167.stdout: "max_size_kb": 0, 2023-10-05T19:28:28.695 INFO:teuthology.orchestra.run.smithi167.stdout: "max_objects": -1 2023-10-05T19:28:28.695 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2023-10-05T19:28:28.695 INFO:teuthology.orchestra.run.smithi167.stdout: "user_quota": { 2023-10-05T19:28:28.695 INFO:teuthology.orchestra.run.smithi167.stdout: "enabled": false, 2023-10-05T19:28:28.695 INFO:teuthology.orchestra.run.smithi167.stdout: "check_on_raw": false, 2023-10-05T19:28:28.695 INFO:teuthology.orchestra.run.smithi167.stdout: "max_size": -1, 2023-10-05T19:28:28.695 INFO:teuthology.orchestra.run.smithi167.stdout: "max_size_kb": 0, 2023-10-05T19:28:28.695 INFO:teuthology.orchestra.run.smithi167.stdout: "max_objects": -1 2023-10-05T19:28:28.695 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2023-10-05T19:28:28.695 INFO:teuthology.orchestra.run.smithi167.stdout: "temp_url_keys": [], 2023-10-05T19:28:28.695 INFO:teuthology.orchestra.run.smithi167.stdout: "type": "rgw", 2023-10-05T19:28:28.695 INFO:teuthology.orchestra.run.smithi167.stdout: "mfa_ids": [] 2023-10-05T19:28:28.696 INFO:teuthology.orchestra.run.smithi167.stdout:} 2023-10-05T19:28:28.696 INFO:teuthology.orchestra.run.smithi167.stdout: 2023-10-05T19:28:28.696 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.686+0000 7f97c0b82e40 1 -- 172.21.15.167:0/16775053 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:104 7.1f 7:f95f44c2:::notify.0:head [watch unwatch cookie 93845179775568] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x555a08a2dcc0 con 0x7f97b8027be0 2023-10-05T19:28:28.696 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.690+0000 7f97bdeb6700 1 -- 172.21.15.167:0/16775053 <== osd.2 v2:172.21.15.167:6816/30037 17 ==== osd_op_reply(104 notify.0 [watch unwatch cookie 93845179775568] v30'65 uv63 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f97b4032040 con 0x7f97b8027be0 2023-10-05T19:28:28.696 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.690+0000 7f97c0b82e40 20 remove_watcher() i=0 2023-10-05T19:28:28.696 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.690+0000 7f97c0b82e40 2 removed watcher, disabling cache 2023-10-05T19:28:28.697 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.690+0000 7f97c0b82e40 1 -- 172.21.15.167:0/16775053 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:105 7.0 7:05bf5b68:::notify.1:head [watch unwatch cookie 93845179777856] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x555a08987a80 con 0x555a089707e0 2023-10-05T19:28:28.697 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.690+0000 7f97beeb8700 1 -- 172.21.15.167:0/16775053 <== osd.1 v2:172.21.15.167:6800/30035 37 ==== osd_op_reply(105 notify.1 [watch unwatch cookie 93845179777856] v30'65 uv63 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f97b0050040 con 0x555a089707e0 2023-10-05T19:28:28.697 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.690+0000 7f97c0b82e40 20 remove_watcher() i=1 2023-10-05T19:28:28.697 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.690+0000 7f97c0b82e40 1 -- 172.21.15.167:0/16775053 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:106 7.15 7:a93a5511:::notify.2:head [watch unwatch cookie 93845179784528] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x555a08987a80 con 0x555a089a7490 2023-10-05T19:28:28.701 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.694+0000 7f97be6b7700 1 -- 172.21.15.167:0/16775053 <== osd.3 v2:172.21.15.167:6808/30036 24 ==== osd_op_reply(106 notify.2 [watch unwatch cookie 93845179784528] v30'65 uv63 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f97b8024020 con 0x555a089a7490 2023-10-05T19:28:28.701 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.694+0000 7f97c0b82e40 20 remove_watcher() i=2 2023-10-05T19:28:28.701 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.694+0000 7f97c0b82e40 1 -- 172.21.15.167:0/16775053 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:107 7.e 7:7759931f:::notify.3:head [watch unwatch cookie 93845179788944] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x555a089add50 con 0x7f97b8027be0 2023-10-05T19:28:28.702 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.698+0000 7f97bdeb6700 1 -- 172.21.15.167:0/16775053 <== osd.2 v2:172.21.15.167:6816/30037 18 ==== osd_op_reply(107 notify.3 [watch unwatch cookie 93845179788944] v30'65 uv63 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f97b4032040 con 0x7f97b8027be0 2023-10-05T19:28:28.702 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.698+0000 7f97c0b82e40 20 remove_watcher() i=3 2023-10-05T19:28:28.702 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.698+0000 7f97c0b82e40 1 -- 172.21.15.167:0/16775053 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:108 7.d 7:b4812045:::notify.4:head [watch unwatch cookie 93845179791152] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x555a089aee90 con 0x555a089707e0 2023-10-05T19:28:28.705 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.698+0000 7f97beeb8700 1 -- 172.21.15.167:0/16775053 <== osd.1 v2:172.21.15.167:6800/30035 38 ==== osd_op_reply(108 notify.4 [watch unwatch cookie 93845179791152] v30'65 uv63 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f97b0050040 con 0x555a089707e0 2023-10-05T19:28:28.705 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.698+0000 7f97c0b82e40 20 remove_watcher() i=4 2023-10-05T19:28:28.705 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.698+0000 7f97c0b82e40 1 -- 172.21.15.167:0/16775053 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:109 7.3 7:c609908c:::notify.5:head [watch unwatch cookie 93845179796272] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x555a089af730 con 0x7f97b4007030 2023-10-05T19:28:28.708 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.702+0000 7f97beeb8700 1 -- 172.21.15.167:0/16775053 <== osd.0 v2:172.21.15.118:6800/27690 29 ==== osd_op_reply(109 notify.5 [watch unwatch cookie 93845179796272] v30'65 uv63 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f97b0050040 con 0x7f97b4007030 2023-10-05T19:28:28.708 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.702+0000 7f97c0b82e40 20 remove_watcher() i=5 2023-10-05T19:28:28.708 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.702+0000 7f97c0b82e40 1 -- 172.21.15.167:0/16775053 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:110 7.14 7:2b04a3e9:::notify.6:head [watch unwatch cookie 93845179787680] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x555a089aee90 con 0x7f97b4007030 2023-10-05T19:28:28.711 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.706+0000 7f97beeb8700 1 -- 172.21.15.167:0/16775053 <== osd.0 v2:172.21.15.118:6800/27690 30 ==== osd_op_reply(110 notify.6 [watch unwatch cookie 93845179787680] v30'65 uv63 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f97b0050040 con 0x7f97b4007030 2023-10-05T19:28:28.711 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.706+0000 7f97c0b82e40 20 remove_watcher() i=6 2023-10-05T19:28:28.711 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.706+0000 7f97c0b82e40 1 -- 172.21.15.167:0/16775053 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:111 7.9 7:93e5b521:::notify.7:head [watch unwatch cookie 93845179800096] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x555a089b0b30 con 0x7f97b4007030 2023-10-05T19:28:28.714 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.710+0000 7f97beeb8700 1 -- 172.21.15.167:0/16775053 <== osd.0 v2:172.21.15.118:6800/27690 31 ==== osd_op_reply(111 notify.7 [watch unwatch cookie 93845179800096] v30'65 uv63 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f97b0050040 con 0x7f97b4007030 2023-10-05T19:28:28.714 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.710+0000 7f97c0b82e40 20 remove_watcher() i=7 2023-10-05T19:28:28.716 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.710+0000 7f978ffdf700 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2023-10-05T19:28:28.716 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.710+0000 7f97c0b82e40 1 -- 172.21.15.167:0/16775053 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:112 6.4 6:22d26bf9:::data_loggenerations_metadata:head [watch unwatch cookie 93845179801984] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x555a089ee830 con 0x555a089707e0 2023-10-05T19:28:28.719 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.714+0000 7f97beeb8700 1 -- 172.21.15.167:0/16775053 <== osd.1 v2:172.21.15.167:6800/30035 39 ==== osd_op_reply(112 data_loggenerations_metadata [watch unwatch cookie 93845179801984] v30'77 uv1 ondisk = 0) v8 ==== 172+0+0 (crc 0 0 0) 0x7f97b0050040 con 0x555a089707e0 2023-10-05T19:28:28.719 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.714+0000 7f97c0b82e40 1 -- 172.21.15.167:0/16775053 >> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] conn(0x7f97b4007030 msgr2=0x7f97b401c920 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-05T19:28:28.720 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.714+0000 7f97c0b82e40 1 --2- 172.21.15.167:0/16775053 >> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] conn(0x7f97b4007030 0x7f97b401c920 crc :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-05T19:28:28.720 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.714+0000 7f97c0b82e40 1 -- 172.21.15.167:0/16775053 >> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] conn(0x555a089707e0 msgr2=0x555a089a39f0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-05T19:28:28.720 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.714+0000 7f97c0b82e40 1 --2- 172.21.15.167:0/16775053 >> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] conn(0x555a089707e0 0x555a089a39f0 crc :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-05T19:28:28.720 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.714+0000 7f97c0b82e40 1 -- 172.21.15.167:0/16775053 >> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] conn(0x7f97b8027be0 msgr2=0x7f97b8042ba0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-05T19:28:28.720 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.714+0000 7f97c0b82e40 1 --2- 172.21.15.167:0/16775053 >> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] conn(0x7f97b8027be0 0x7f97b8042ba0 crc :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-05T19:28:28.720 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.714+0000 7f97c0b82e40 1 -- 172.21.15.167:0/16775053 >> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] conn(0x555a089a7490 msgr2=0x555a089a9870 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-05T19:28:28.720 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.714+0000 7f97c0b82e40 1 --2- 172.21.15.167:0/16775053 >> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] conn(0x555a089a7490 0x555a089a9870 crc :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-05T19:28:28.720 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.714+0000 7f97c0b82e40 1 -- 172.21.15.167:0/16775053 >> [v2:172.21.15.167:6824/29889,v1:172.21.15.167:6825/29889] conn(0x7f97a0030c50 msgr2=0x7f97a0033100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-05T19:28:28.720 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.714+0000 7f97c0b82e40 1 --2- 172.21.15.167:0/16775053 >> [v2:172.21.15.167:6824/29889,v1:172.21.15.167:6825/29889] conn(0x7f97a0030c50 0x7f97a0033100 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7f97b4005a90 tx=0x7f97b4015000 comp rx=0 tx=0).stop 2023-10-05T19:28:28.720 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.714+0000 7f97beeb8700 1 -- 172.21.15.167:0/16775053 reap_dead start 2023-10-05T19:28:28.721 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.714+0000 7f97c0b82e40 1 -- 172.21.15.167:0/16775053 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x555a08970410 msgr2=0x555a08997f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-05T19:28:28.721 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.714+0000 7f97c0b82e40 1 --2- 172.21.15.167:0/16775053 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x555a08970410 0x555a08997f30 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 crypto rx=0x7f97b8006bd0 tx=0x7f97b8014000 comp rx=0 tx=0).stop 2023-10-05T19:28:28.721 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.714+0000 7f97c0b82e40 1 -- 172.21.15.167:0/16775053 shutdown_connections 2023-10-05T19:28:28.721 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.714+0000 7f97c0b82e40 1 --2- 172.21.15.167:0/16775053 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x555a08970410 0x555a08997f30 unknown :-1 s=CLOSED pgs=263 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-05T19:28:28.721 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.714+0000 7f97c0b82e40 1 -- 172.21.15.167:0/16775053 >> 172.21.15.167:0/16775053 conn(0x555a0896a720 msgr2=0x555a0898b7a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-05T19:28:28.721 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.714+0000 7f97c0b82e40 1 -- 172.21.15.167:0/16775053 shutdown_connections 2023-10-05T19:28:28.721 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.714+0000 7f97c0b82e40 1 -- 172.21.15.167:0/16775053 wait complete. 2023-10-05T19:28:28.722 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.714+0000 7f97c0b82e40 1 -- 172.21.15.167:0/1911393562 >> [v2:172.21.15.167:6824/29889,v1:172.21.15.167:6825/29889] conn(0x7f9704030c00 msgr2=0x7f97040330b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-05T19:28:28.722 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.714+0000 7f97c0b82e40 1 --2- 172.21.15.167:0/1911393562 >> [v2:172.21.15.167:6824/29889,v1:172.21.15.167:6825/29889] conn(0x7f9704030c00 0x7f97040330b0 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x555a089b1200 tx=0x7f97b0000f90 comp rx=0 tx=0).stop 2023-10-05T19:28:28.722 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.714+0000 7f97c0b82e40 1 -- 172.21.15.167:0/1911393562 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x555a089bfb10 msgr2=0x555a089c5550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-05T19:28:28.722 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.714+0000 7f97c0b82e40 1 --2- 172.21.15.167:0/1911393562 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x555a089bfb10 0x555a089c5550 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 crypto rx=0x555a089b9bf0 tx=0x7f97b402f000 comp rx=0 tx=0).stop 2023-10-05T19:28:28.723 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.714+0000 7f97c0b82e40 1 -- 172.21.15.167:0/1911393562 shutdown_connections 2023-10-05T19:28:28.723 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.714+0000 7f97c0b82e40 1 --2- 172.21.15.167:0/1911393562 >> [v2:172.21.15.167:6824/29889,v1:172.21.15.167:6825/29889] conn(0x7f9704030c00 0x7f97040330b0 unknown :-1 s=CLOSED pgs=95 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-05T19:28:28.723 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.714+0000 7f97c0b82e40 1 --2- 172.21.15.167:0/1911393562 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x555a089bfb10 0x555a089c5550 unknown :-1 s=CLOSED pgs=265 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-05T19:28:28.723 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.714+0000 7f97c0b82e40 1 -- 172.21.15.167:0/1911393562 >> 172.21.15.167:0/1911393562 conn(0x555a089baed0 msgr2=0x555a089b8fd0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-05T19:28:28.723 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.714+0000 7f97c0b82e40 1 -- 172.21.15.167:0/1911393562 shutdown_connections 2023-10-05T19:28:28.723 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.714+0000 7f97c0b82e40 1 -- 172.21.15.167:0/1911393562 wait complete. 2023-10-05T19:28:28.726 DEBUG:tasks.util.rgw: json result: {'user_id': 'foo', 'display_name': 'Foo', 'email': 'foo@foo.com', 'suspended': 0, 'max_buckets': 4, 'subusers': [{'id': 'foo:foo1', 'permissions': 'full-control'}], 'keys': [{'user': 'foo', 'access_key': '9te6NH5mcdcq0Tc5i8i1', 'secret_key': 'Ny4IOauQoL18Gp2zM7lC1vLmoawgqcYP/YGcWfXu'}], 'swift_keys': [{'user': 'foo:foo1', 'secret_key': 'gpS2G9RREMrnbqlp29PP2D36kgPR1tm72n5fPYfL'}], 'caps': [], 'op_mask': 'read, write, delete', 'default_placement': '', 'default_storage_class': '', 'placement_tags': [], 'bucket_quota': {'enabled': False, 'check_on_raw': False, 'max_size': -1, 'max_size_kb': 0, 'max_objects': -1}, 'user_quota': {'enabled': False, 'check_on_raw': False, 'max_size': -1, 'max_size_kb': 0, 'max_objects': -1}, 'temp_url_keys': [], 'type': 'rgw', 'mfa_ids': []} 2023-10-05T19:28:28.726 INFO:tasks.util.rgw:rgwadmin: client.0 : ['subuser', 'create', '--subuser', 'foo:foo2', '--secret', 'ri2VJQcKSYATOY6uaDUX7pxgkW+W1YmC6OCxPHwy', '--key-type', 'swift'] 2023-10-05T19:28:28.727 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'ceph', 'subuser', 'create', '--subuser', 'foo:foo2', '--secret', 'ri2VJQcKSYATOY6uaDUX7pxgkW+W1YmC6OCxPHwy', '--key-type', 'swift'] 2023-10-05T19:28:28.727 DEBUG:teuthology.orchestra.run.smithi167:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster ceph subuser create --subuser foo:foo2 --secret ri2VJQcKSYATOY6uaDUX7pxgkW+W1YmC6OCxPHwy --key-type swift 2023-10-05T19:28:28.771 INFO:teuthology.orchestra.run.smithi167.stderr:ignoring --setuser ceph since I am not root 2023-10-05T19:28:28.771 INFO:teuthology.orchestra.run.smithi167.stderr:ignoring --setgroup ceph since I am not root 2023-10-05T19:28:28.771 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.766+0000 7f7e9cd41e40 1 Processor -- start 2023-10-05T19:28:28.772 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.766+0000 7f7e9cd41e40 1 -- start start 2023-10-05T19:28:28.772 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.766+0000 7f7e9cd41e40 1 --2- >> v2:172.21.15.118:3300/0 conn(0x5652a2f40590 0x5652a2f42970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-05T19:28:28.772 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.766+0000 7f7e9cd41e40 1 -- --> v1:172.21.15.118:6789/0 -- auth(proto 0 26 bytes epoch 0) v1 -- 0x5652a2e86bd0 con 0x5652a2f3bfb0 2023-10-05T19:28:28.772 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.766+0000 7f7e9cd41e40 1 -- --> v2:172.21.15.118:3300/0 -- mon_getmap magic: 0 v1 -- 0x5652a2e43ee0 con 0x5652a2f40590 2023-10-05T19:28:28.773 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.766+0000 7f7e9a876700 1 --1- >> v1:172.21.15.118:6789/0 conn(0x5652a2f3bfb0 0x5652a2f3c380 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.118:6789/0 says I am v1:172.21.15.167:56472/0 (socket says 172.21.15.167:56472) 2023-10-05T19:28:28.773 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.766+0000 7f7e9a075700 1 --2- >> v2:172.21.15.118:3300/0 conn(0x5652a2f40590 0x5652a2f42970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-05T19:28:28.773 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.766+0000 7f7e9a876700 1 -- 172.21.15.167:0/58976891 learned_addr learned my addr 172.21.15.167:0/58976891 (peer_addr_for_me v1:172.21.15.167:0/0) 2023-10-05T19:28:28.774 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.766+0000 7f7e99874700 1 -- 172.21.15.167:0/58976891 <== mon.0 v1:172.21.15.118:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 228024277 0 0) 0x5652a2e86bd0 con 0x5652a2f3bfb0 2023-10-05T19:28:28.774 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.766+0000 7f7e99874700 1 -- 172.21.15.167:0/58976891 --> v1:172.21.15.118:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7e7c001e10 con 0x5652a2f3bfb0 2023-10-05T19:28:28.774 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.766+0000 7f7e9a075700 1 -- 172.21.15.167:0/58976891 >> v1:172.21.15.118:6789/0 conn(0x5652a2f3bfb0 legacy=0x5652a2f3c380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-05T19:28:28.774 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.766+0000 7f7e9a075700 1 -- 172.21.15.167:0/58976891 --> v2:172.21.15.118:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5652a2f3cbc0 con 0x5652a2f40590 2023-10-05T19:28:28.774 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.770+0000 7f7e9a075700 1 --2- 172.21.15.167:0/58976891 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5652a2f40590 0x5652a2f42970 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 crypto rx=0x7f7e8400a5d0 tx=0x7f7e84005a50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c26825eefd879bfd server_cookie=0 in_seq=0 out_seq=0 2023-10-05T19:28:28.775 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.770+0000 7f7e99874700 1 -- 172.21.15.167:0/58976891 <== mon.0 v2:172.21.15.118:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f7e84007610 con 0x5652a2f40590 2023-10-05T19:28:28.775 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.770+0000 7f7e99874700 1 -- 172.21.15.167:0/58976891 <== mon.0 v2:172.21.15.118:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7e84007770 con 0x5652a2f40590 2023-10-05T19:28:28.775 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.770+0000 7f7e99874700 1 -- 172.21.15.167:0/58976891 <== mon.0 v2:172.21.15.118:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f7e84007a20 con 0x5652a2f40590 2023-10-05T19:28:28.776 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.770+0000 7f7e9cd41e40 1 -- 172.21.15.167:0/58976891 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5652a2f40590 msgr2=0x5652a2f42970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-05T19:28:28.776 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.770+0000 7f7e9cd41e40 1 --2- 172.21.15.167:0/58976891 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5652a2f40590 0x5652a2f42970 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 crypto rx=0x7f7e8400a5d0 tx=0x7f7e84005a50 comp rx=0 tx=0).stop 2023-10-05T19:28:28.776 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.770+0000 7f7e9cd41e40 1 -- 172.21.15.167:0/58976891 shutdown_connections 2023-10-05T19:28:28.776 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.770+0000 7f7e9cd41e40 1 --2- 172.21.15.167:0/58976891 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5652a2f40590 0x5652a2f42970 unknown :-1 s=CLOSED pgs=267 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-05T19:28:28.776 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.770+0000 7f7e9cd41e40 1 -- 172.21.15.167:0/58976891 >> 172.21.15.167:0/58976891 conn(0x5652a2eaf800 msgr2=0x5652a2eafc00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-05T19:28:28.776 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.770+0000 7f7e9cd41e40 1 -- 172.21.15.167:0/58976891 shutdown_connections 2023-10-05T19:28:28.776 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.770+0000 7f7e9cd41e40 1 -- 172.21.15.167:0/58976891 wait complete. 2023-10-05T19:28:28.777 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.770+0000 7f7e9cd41e40 1 Processor -- start 2023-10-05T19:28:28.777 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.770+0000 7f7e9cd41e40 1 -- start start 2023-10-05T19:28:28.778 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.774+0000 7f7e9cd41e40 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5652a2f3bfb0 0x5652a2f5b020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-05T19:28:28.778 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.774+0000 7f7e9cd41e40 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x5652a2e07e30 con 0x5652a2f3bfb0 2023-10-05T19:28:28.779 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.774+0000 7f7e9a876700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5652a2f3bfb0 0x5652a2f5b020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-05T19:28:28.779 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.774+0000 7f7e9a876700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5652a2f3bfb0 0x5652a2f5b020 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.118:3300/0 says I am v2:172.21.15.167:57774/0 (socket says 172.21.15.167:57774) 2023-10-05T19:28:28.779 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.774+0000 7f7e9a876700 1 -- 172.21.15.167:0/2464542108 learned_addr learned my addr 172.21.15.167:0/2464542108 (peer_addr_for_me v2:172.21.15.167:0/0) 2023-10-05T19:28:28.779 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.774+0000 7f7e9a876700 1 -- 172.21.15.167:0/2464542108 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7e84005cf0 con 0x5652a2f3bfb0 2023-10-05T19:28:28.779 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.774+0000 7f7e9a876700 1 --2- 172.21.15.167:0/2464542108 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5652a2f3bfb0 0x5652a2f5b020 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 crypto rx=0x7f7e8c001c30 tx=0x7f7e8c001810 comp rx=0 tx=0).ready entity=mon.0 client_cookie=865a929e2f379107 server_cookie=0 in_seq=0 out_seq=0 2023-10-05T19:28:28.780 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.774+0000 7f7e98872700 1 -- 172.21.15.167:0/2464542108 <== mon.0 v2:172.21.15.118:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f7e8c00bbf0 con 0x5652a2f3bfb0 2023-10-05T19:28:28.780 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.774+0000 7f7e98872700 1 -- 172.21.15.167:0/2464542108 <== mon.0 v2:172.21.15.118:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7e8c00bd50 con 0x5652a2f3bfb0 2023-10-05T19:28:28.780 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.774+0000 7f7e98872700 1 -- 172.21.15.167:0/2464542108 <== mon.0 v2:172.21.15.118:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f7e8c00e470 con 0x5652a2f3bfb0 2023-10-05T19:28:28.781 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.774+0000 7f7e9cd41e40 1 -- 172.21.15.167:0/2464542108 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5652a2f3bfb0 msgr2=0x5652a2f5b020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-05T19:28:28.781 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.774+0000 7f7e9cd41e40 1 --2- 172.21.15.167:0/2464542108 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5652a2f3bfb0 0x5652a2f5b020 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 crypto rx=0x7f7e8c001c30 tx=0x7f7e8c001810 comp rx=0 tx=0).stop 2023-10-05T19:28:28.781 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.774+0000 7f7e9cd41e40 1 -- 172.21.15.167:0/2464542108 shutdown_connections 2023-10-05T19:28:28.781 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.774+0000 7f7e9cd41e40 1 --2- 172.21.15.167:0/2464542108 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5652a2f3bfb0 0x5652a2f5b020 unknown :-1 s=CLOSED pgs=268 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-05T19:28:28.781 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.774+0000 7f7e9cd41e40 1 -- 172.21.15.167:0/2464542108 >> 172.21.15.167:0/2464542108 conn(0x5652a2eaf800 msgr2=0x5652a2f56b60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-05T19:28:28.782 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.774+0000 7f7e9cd41e40 1 -- 172.21.15.167:0/2464542108 shutdown_connections 2023-10-05T19:28:28.782 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.774+0000 7f7e9cd41e40 1 -- 172.21.15.167:0/2464542108 wait complete. 2023-10-05T19:28:28.782 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.774+0000 7f7e9cd41e40 1 Processor -- start 2023-10-05T19:28:28.782 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.778+0000 7f7e9cd41e40 1 -- start start 2023-10-05T19:28:28.782 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.778+0000 7f7e9cd41e40 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5652a2f3bfb0 0x5652a2f61db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-05T19:28:28.782 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.778+0000 7f7e9cd41e40 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x5652a2ddaed0 con 0x5652a2f3bfb0 2023-10-05T19:28:28.783 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.778+0000 7f7e9a876700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5652a2f3bfb0 0x5652a2f61db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-05T19:28:28.783 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.778+0000 7f7e9a876700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5652a2f3bfb0 0x5652a2f61db0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.118:3300/0 says I am v2:172.21.15.167:57788/0 (socket says 172.21.15.167:57788) 2023-10-05T19:28:28.783 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.778+0000 7f7e9a876700 1 -- 172.21.15.167:0/4093925617 learned_addr learned my addr 172.21.15.167:0/4093925617 (peer_addr_for_me v2:172.21.15.167:0/0) 2023-10-05T19:28:28.783 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.778+0000 7f7e9a876700 1 -- 172.21.15.167:0/4093925617 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7e7c001e10 con 0x5652a2f3bfb0 2023-10-05T19:28:28.784 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.778+0000 7f7e9a876700 1 --2- 172.21.15.167:0/4093925617 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5652a2f3bfb0 0x5652a2f61db0 secure :-1 s=READY pgs=269 cs=0 l=1 rev1=1 crypto rx=0x7f7e8c00b960 tx=0x7f7e8c014000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-05T19:28:28.784 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.778+0000 7f7e92ffd700 1 -- 172.21.15.167:0/4093925617 <== mon.0 v2:172.21.15.118:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f7e8c01f070 con 0x5652a2f3bfb0 2023-10-05T19:28:28.784 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.778+0000 7f7e92ffd700 1 -- 172.21.15.167:0/4093925617 <== mon.0 v2:172.21.15.118:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7e8c01a2e0 con 0x5652a2f3bfb0 2023-10-05T19:28:28.785 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.778+0000 7f7e9cd41e40 1 -- 172.21.15.167:0/4093925617 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x5652a2f622f0 con 0x5652a2f3bfb0 2023-10-05T19:28:28.785 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.778+0000 7f7e92ffd700 1 -- 172.21.15.167:0/4093925617 <== mon.0 v2:172.21.15.118:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f7e8c00e470 con 0x5652a2f3bfb0 2023-10-05T19:28:28.785 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.778+0000 7f7e9cd41e40 1 -- 172.21.15.167:0/4093925617 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x5652a2f62730 con 0x5652a2f3bfb0 2023-10-05T19:28:28.786 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.782+0000 7f7e92ffd700 1 -- 172.21.15.167:0/4093925617 <== mon.0 v2:172.21.15.118:3300/0 4 ==== mgrmap(e 5) v1 ==== 40329+0+0 (secure 0 0 0) 0x7f7e8c01aa10 con 0x5652a2f3bfb0 2023-10-05T19:28:28.786 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.782+0000 7f7e9cd41e40 20 rados->read ofs=0 len=0 2023-10-05T19:28:28.786 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.782+0000 7f7e92ffd700 1 --2- 172.21.15.167:0/4093925617 >> [v2:172.21.15.167:6824/29889,v1:172.21.15.167:6825/29889] conn(0x7f7e80030c40 0x7f7e800330f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-05T19:28:28.787 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.782+0000 7f7e92ffd700 1 -- 172.21.15.167:0/4093925617 <== mon.0 v2:172.21.15.118:3300/0 5 ==== osd_map(30..30 src has 1..30) v4 ==== 6057+0+0 (secure 0 0 0) 0x7f7e8c0451b0 con 0x5652a2f3bfb0 2023-10-05T19:28:28.787 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.782+0000 7f7e9a075700 1 --2- 172.21.15.167:0/4093925617 >> [v2:172.21.15.167:6824/29889,v1:172.21.15.167:6825/29889] conn(0x7f7e80030c40 0x7f7e800330f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-05T19:28:28.787 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.782+0000 7f7e9cd41e40 1 --2- 172.21.15.167:0/4093925617 >> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] conn(0x5652a2f40590 0x5652a2f6d840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-05T19:28:28.787 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.782+0000 7f7e9cd41e40 1 -- 172.21.15.167:0/4093925617 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:1 5.12 5:49953fa1:::default.realm:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x5652a2f6deb0 con 0x5652a2f40590 2023-10-05T19:28:28.788 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.782+0000 7f7e9b077700 1 --2- 172.21.15.167:0/4093925617 >> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] conn(0x5652a2f40590 0x5652a2f6d840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-05T19:28:28.788 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.782+0000 7f7e9a075700 1 --2- 172.21.15.167:0/4093925617 >> [v2:172.21.15.167:6824/29889,v1:172.21.15.167:6825/29889] conn(0x7f7e80030c40 0x7f7e800330f0 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7f7e84005cc0 tx=0x7f7e84014000 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-05T19:28:28.788 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.782+0000 7f7e9b077700 1 --2- 172.21.15.167:0/4093925617 >> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] conn(0x5652a2f40590 0x5652a2f6d840 crc :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-05T19:28:28.789 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.782+0000 7f7e9b077700 1 -- 172.21.15.167:0/4093925617 <== osd.1 v2:172.21.15.167:6800/30035 1 ==== osd_op_reply(1 default.realm [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 ==== 157+0+0 (crc 0 0 0) 0x7f7e9400a5a0 con 0x5652a2f40590 2023-10-05T19:28:28.789 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.782+0000 7f7e9cd41e40 20 rados_obj.operate() r=-2 bl.length=0 2023-10-05T19:28:28.789 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.782+0000 7f7e9cd41e40 20 realm 2023-10-05T19:28:28.789 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.782+0000 7f7e9cd41e40 20 rados->read ofs=0 len=0 2023-10-05T19:28:28.789 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.782+0000 7f7e9cd41e40 1 -- 172.21.15.167:0/4093925617 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:2 5.12 5:49953fa1:::default.realm:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x5652a2f6ae60 con 0x5652a2f40590 2023-10-05T19:28:28.791 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.782+0000 7f7e9b077700 1 -- 172.21.15.167:0/4093925617 <== osd.1 v2:172.21.15.167:6800/30035 2 ==== osd_op_reply(2 default.realm [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 ==== 157+0+0 (crc 0 0 0) 0x7f7e9400a5a0 con 0x5652a2f40590 2023-10-05T19:28:28.791 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.782+0000 7f7e9cd41e40 20 rados_obj.operate() r=-2 bl.length=0 2023-10-05T19:28:28.791 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.782+0000 7f7e9cd41e40 4 RGWPeriod::init failed to init realm id : (2) No such file or directory 2023-10-05T19:28:28.792 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.782+0000 7f7e9cd41e40 20 rados->read ofs=0 len=0 2023-10-05T19:28:28.792 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.782+0000 7f7e9cd41e40 1 -- 172.21.15.167:0/4093925617 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:3 5.12 5:49953fa1:::default.realm:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x5652a2f6e990 con 0x5652a2f40590 2023-10-05T19:28:28.792 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.782+0000 7f7e9b077700 1 -- 172.21.15.167:0/4093925617 <== osd.1 v2:172.21.15.167:6800/30035 3 ==== osd_op_reply(3 default.realm [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 ==== 157+0+0 (crc 0 0 0) 0x7f7e9400a5a0 con 0x5652a2f40590 2023-10-05T19:28:28.792 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.786+0000 7f7e9cd41e40 20 rados_obj.operate() r=-2 bl.length=0 2023-10-05T19:28:28.792 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.786+0000 7f7e9cd41e40 20 rados->read ofs=0 len=0 2023-10-05T19:28:28.792 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.786+0000 7f7e9cd41e40 1 -- 172.21.15.167:0/4093925617 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:4 5.f 5:f43b6ece:::zone_names.default:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x5652a2f6e570 con 0x5652a2f40590 2023-10-05T19:28:28.792 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.786+0000 7f7e9b077700 1 -- 172.21.15.167:0/4093925617 <== osd.1 v2:172.21.15.167:6800/30035 4 ==== osd_op_reply(4 zone_names.default [read 0~46 out=46b] v0'0 uv1 ondisk = 0) v8 ==== 162+0+46 (crc 0 0 0) 0x7f7e9400a5a0 con 0x5652a2f40590 2023-10-05T19:28:28.792 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.786+0000 7f7e9cd41e40 20 rados_obj.operate() r=0 bl.length=46 2023-10-05T19:28:28.792 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.786+0000 7f7e9cd41e40 20 rados->read ofs=0 len=0 2023-10-05T19:28:28.792 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.786+0000 7f7e9cd41e40 1 -- 172.21.15.167:0/4093925617 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:5 5.1d 5:b8374eb4:::zone_info.9a9df965-6612-45d4-a45f-6e3c889bacc2:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x5652a2f6f530 con 0x5652a2f40590 2023-10-05T19:28:28.792 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.786+0000 7f7e9b077700 1 -- 172.21.15.167:0/4093925617 <== osd.1 v2:172.21.15.167:6800/30035 5 ==== osd_op_reply(5 zone_info.9a9df965-6612-45d4-a45f-6e3c889bacc2 [read 0~874 out=874b] v0'0 uv2 ondisk = 0) v8 ==== 190+0+874 (crc 0 0 0) 0x7f7e9400a5a0 con 0x5652a2f40590 2023-10-05T19:28:28.793 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.786+0000 7f7e9cd41e40 20 rados_obj.operate() r=0 bl.length=874 2023-10-05T19:28:28.793 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.786+0000 7f7e9cd41e40 20 searching for the correct realm 2023-10-05T19:28:28.793 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.786+0000 7f7e9cd41e40 1 --2- 172.21.15.167:0/4093925617 >> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] conn(0x5652a2f71470 0x5652a2f73880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-05T19:28:28.793 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.786+0000 7f7e9cd41e40 1 -- 172.21.15.167:0/4093925617 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:6 5.0 5:00000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x5652a2f73e90 con 0x5652a2f71470 2023-10-05T19:28:28.793 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.786+0000 7f7e9a876700 1 --2- 172.21.15.167:0/4093925617 >> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] conn(0x5652a2f71470 0x5652a2f73880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-05T19:28:28.794 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.786+0000 7f7e9a876700 1 --2- 172.21.15.167:0/4093925617 >> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] conn(0x5652a2f71470 0x5652a2f73880 crc :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-05T19:28:28.794 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.786+0000 7f7e9a876700 1 -- 172.21.15.167:0/4093925617 <== osd.3 v2:172.21.15.167:6808/30036 1 ==== osd_op_reply(6 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f7e8c014070 con 0x5652a2f71470 2023-10-05T19:28:28.794 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.786+0000 7f7e9a876700 1 --2- 172.21.15.167:0/4093925617 >> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] conn(0x7f7e8c02d020 0x7f7e8c02c5d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-05T19:28:28.794 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.786+0000 7f7e9a876700 1 -- 172.21.15.167:0/4093925617 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:7 5.10 5:08000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x5652a2f73e90 con 0x7f7e8c02d020 2023-10-05T19:28:28.794 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.790+0000 7f7e9a075700 1 --2- 172.21.15.167:0/4093925617 >> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] conn(0x7f7e8c02d020 0x7f7e8c02c5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-05T19:28:28.794 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.790+0000 7f7e9a075700 1 --2- 172.21.15.167:0/4093925617 >> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] conn(0x7f7e8c02d020 0x7f7e8c02c5d0 crc :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-05T19:28:28.795 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.790+0000 7f7e9a075700 1 -- 172.21.15.167:0/4093925617 <== osd.2 v2:172.21.15.167:6816/30037 1 ==== osd_op_reply(7 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f7e84007a50 con 0x7f7e8c02d020 2023-10-05T19:28:28.795 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.790+0000 7f7e9a075700 1 --2- 172.21.15.167:0/4093925617 >> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] conn(0x7f7e84007030 0x7f7e8401d430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-05T19:28:28.795 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.790+0000 7f7e9a075700 1 -- 172.21.15.167:0/4093925617 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:8 5.8 5:10000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x5652a2f73e90 con 0x7f7e84007030 2023-10-05T19:28:28.795 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.790+0000 7f7e9b077700 1 --2- 172.21.15.167:0/4093925617 >> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] conn(0x7f7e84007030 0x7f7e8401d430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-05T19:28:28.796 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.790+0000 7f7e9b077700 1 --2- 172.21.15.167:0/4093925617 >> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] conn(0x7f7e84007030 0x7f7e8401d430 crc :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-05T19:28:28.797 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.790+0000 7f7e9b077700 1 -- 172.21.15.167:0/4093925617 <== osd.0 v2:172.21.15.118:6800/27690 1 ==== osd_op_reply(8 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f7e9400a5a0 con 0x7f7e84007030 2023-10-05T19:28:28.797 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.790+0000 7f7e9b077700 1 -- 172.21.15.167:0/4093925617 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:9 5.18 5:18000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x5652a2f73e90 con 0x5652a2f40590 2023-10-05T19:28:28.797 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.790+0000 7f7e9b077700 1 -- 172.21.15.167:0/4093925617 <== osd.1 v2:172.21.15.167:6800/30035 6 ==== osd_op_reply(9 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f7e9400a5a0 con 0x5652a2f40590 2023-10-05T19:28:28.797 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.790+0000 7f7e9b077700 1 -- 172.21.15.167:0/4093925617 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:10 5.4 5:20000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x5652a2f73e90 con 0x5652a2f40590 2023-10-05T19:28:28.798 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.794+0000 7f7e9b077700 1 -- 172.21.15.167:0/4093925617 <== osd.1 v2:172.21.15.167:6800/30035 7 ==== osd_op_reply(10 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f7e9400a5a0 con 0x5652a2f40590 2023-10-05T19:28:28.798 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.794+0000 7f7e9b077700 1 -- 172.21.15.167:0/4093925617 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:11 5.14 5:28000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x5652a2f73e90 con 0x7f7e84007030 2023-10-05T19:28:28.799 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.794+0000 7f7e9b077700 1 -- 172.21.15.167:0/4093925617 <== osd.0 v2:172.21.15.118:6800/27690 2 ==== osd_op_reply(11 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f7e9400a5a0 con 0x7f7e84007030 2023-10-05T19:28:28.799 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.794+0000 7f7e9b077700 1 -- 172.21.15.167:0/4093925617 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:12 5.c 5:30000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x5652a2f73e90 con 0x7f7e84007030 2023-10-05T19:28:28.799 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.794+0000 7f7e9b077700 1 -- 172.21.15.167:0/4093925617 <== osd.0 v2:172.21.15.118:6800/27690 3 ==== osd_op_reply(12 [pgnls start_epoch 30 out=112b] v23'1 uv1 ondisk = 1) v8 ==== 144+0+112 (crc 0 0 0) 0x7f7e9400a5a0 con 0x7f7e84007030 2023-10-05T19:28:28.800 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.794+0000 7f7e9b077700 1 -- 172.21.15.167:0/4093925617 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:13 5.1c 5:38000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x5652a2f73e90 con 0x7f7e8c02d020 2023-10-05T19:28:28.800 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.794+0000 7f7e9a075700 1 -- 172.21.15.167:0/4093925617 <== osd.2 v2:172.21.15.167:6816/30037 2 ==== osd_op_reply(13 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f7e84029040 con 0x7f7e8c02d020 2023-10-05T19:28:28.800 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.794+0000 7f7e9a075700 1 -- 172.21.15.167:0/4093925617 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:14 5.2 5:40000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x5652a2f73e90 con 0x5652a2f71470 2023-10-05T19:28:28.801 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.794+0000 7f7e9a876700 1 -- 172.21.15.167:0/4093925617 <== osd.3 v2:172.21.15.167:6808/30036 2 ==== osd_op_reply(14 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f7e8c014070 con 0x5652a2f71470 2023-10-05T19:28:28.801 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.794+0000 7f7e9a876700 1 -- 172.21.15.167:0/4093925617 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:15 5.12 5:48000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x5652a2f73e90 con 0x5652a2f40590 2023-10-05T19:28:28.801 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.794+0000 7f7e9b077700 1 -- 172.21.15.167:0/4093925617 <== osd.1 v2:172.21.15.167:6800/30035 8 ==== osd_op_reply(15 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f7e9400a5a0 con 0x5652a2f40590 2023-10-05T19:28:28.801 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.794+0000 7f7e9b077700 1 -- 172.21.15.167:0/4093925617 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:16 5.a 5:50000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x5652a2f73e90 con 0x7f7e84007030 2023-10-05T19:28:28.802 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.798+0000 7f7e9b077700 1 -- 172.21.15.167:0/4093925617 <== osd.0 v2:172.21.15.118:6800/27690 4 ==== osd_op_reply(16 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f7e9400a5a0 con 0x7f7e84007030 2023-10-05T19:28:28.803 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.798+0000 7f7e9b077700 1 -- 172.21.15.167:0/4093925617 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:17 5.1a 5:58000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x5652a2f73e90 con 0x5652a2f40590 2023-10-05T19:28:28.803 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.798+0000 7f7e9b077700 1 -- 172.21.15.167:0/4093925617 <== osd.1 v2:172.21.15.167:6800/30035 9 ==== osd_op_reply(17 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f7e9400a5a0 con 0x5652a2f40590 2023-10-05T19:28:28.803 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.798+0000 7f7e9b077700 1 -- 172.21.15.167:0/4093925617 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:18 5.6 5:60000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x5652a2f73e90 con 0x7f7e8c02d020 2023-10-05T19:28:28.804 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.798+0000 7f7e9a075700 1 -- 172.21.15.167:0/4093925617 <== osd.2 v2:172.21.15.167:6816/30037 3 ==== osd_op_reply(18 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f7e84029040 con 0x7f7e8c02d020 2023-10-05T19:28:28.804 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.798+0000 7f7e9a075700 1 -- 172.21.15.167:0/4093925617 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:19 5.16 5:68000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x5652a2f73e90 con 0x5652a2f71470 2023-10-05T19:28:28.805 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.798+0000 7f7e9a876700 1 -- 172.21.15.167:0/4093925617 <== osd.3 v2:172.21.15.167:6808/30036 3 ==== osd_op_reply(19 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f7e8c014070 con 0x5652a2f71470 2023-10-05T19:28:28.805 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.798+0000 7f7e9a876700 1 -- 172.21.15.167:0/4093925617 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:20 5.e 5:70000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x5652a2f73e90 con 0x5652a2f71470 2023-10-05T19:28:28.805 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.798+0000 7f7e9a876700 1 -- 172.21.15.167:0/4093925617 <== osd.3 v2:172.21.15.167:6808/30036 4 ==== osd_op_reply(20 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f7e8c014070 con 0x5652a2f71470 2023-10-05T19:28:28.805 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.798+0000 7f7e9a876700 1 -- 172.21.15.167:0/4093925617 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:21 5.1e 5:78000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x5652a2f73e90 con 0x7f7e84007030 2023-10-05T19:28:28.806 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.802+0000 7f7e9b077700 1 -- 172.21.15.167:0/4093925617 <== osd.0 v2:172.21.15.118:6800/27690 5 ==== osd_op_reply(21 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f7e9400a5a0 con 0x7f7e84007030 2023-10-05T19:28:28.806 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.802+0000 7f7e9b077700 1 -- 172.21.15.167:0/4093925617 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:22 5.1 5:80000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x5652a2f73e90 con 0x5652a2f71470 2023-10-05T19:28:28.807 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.802+0000 7f7e9a876700 1 -- 172.21.15.167:0/4093925617 <== osd.3 v2:172.21.15.167:6808/30036 5 ==== osd_op_reply(22 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f7e8c014070 con 0x5652a2f71470 2023-10-05T19:28:28.807 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.802+0000 7f7e9a876700 1 -- 172.21.15.167:0/4093925617 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:23 5.11 5:88000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x5652a2f73e90 con 0x5652a2f71470 2023-10-05T19:28:28.808 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.802+0000 7f7e9a876700 1 -- 172.21.15.167:0/4093925617 <== osd.3 v2:172.21.15.167:6808/30036 6 ==== osd_op_reply(23 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f7e8c014070 con 0x5652a2f71470 2023-10-05T19:28:28.808 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.802+0000 7f7e9a876700 1 -- 172.21.15.167:0/4093925617 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:24 5.9 5:90000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x5652a2f73e90 con 0x5652a2f40590 2023-10-05T19:28:28.809 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.802+0000 7f7e9b077700 1 -- 172.21.15.167:0/4093925617 <== osd.1 v2:172.21.15.167:6800/30035 10 ==== osd_op_reply(24 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f7e9400a5a0 con 0x5652a2f40590 2023-10-05T19:28:28.809 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.802+0000 7f7e9b077700 1 -- 172.21.15.167:0/4093925617 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:25 5.19 5:98000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x5652a2f73e90 con 0x5652a2f40590 2023-10-05T19:28:28.810 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.806+0000 7f7e9b077700 1 -- 172.21.15.167:0/4093925617 <== osd.1 v2:172.21.15.167:6800/30035 11 ==== osd_op_reply(25 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f7e9400a5a0 con 0x5652a2f40590 2023-10-05T19:28:28.810 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.806+0000 7f7e9b077700 1 -- 172.21.15.167:0/4093925617 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:26 5.5 5:a0000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x5652a2f73e90 con 0x7f7e84007030 2023-10-05T19:28:28.811 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.806+0000 7f7e9b077700 1 -- 172.21.15.167:0/4093925617 <== osd.0 v2:172.21.15.118:6800/27690 6 ==== osd_op_reply(26 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f7e9400a5a0 con 0x7f7e84007030 2023-10-05T19:28:28.811 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.806+0000 7f7e9b077700 1 -- 172.21.15.167:0/4093925617 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:27 5.15 5:a8000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x5652a2f73e90 con 0x5652a2f71470 2023-10-05T19:28:28.811 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.806+0000 7f7e9a876700 1 -- 172.21.15.167:0/4093925617 <== osd.3 v2:172.21.15.167:6808/30036 7 ==== osd_op_reply(27 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f7e8c014070 con 0x5652a2f71470 2023-10-05T19:28:28.812 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.806+0000 7f7e9a876700 1 -- 172.21.15.167:0/4093925617 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:28 5.d 5:b0000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x5652a2f73e90 con 0x7f7e8c02d020 2023-10-05T19:28:28.812 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.806+0000 7f7e9a075700 1 -- 172.21.15.167:0/4093925617 <== osd.2 v2:172.21.15.167:6816/30037 4 ==== osd_op_reply(28 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f7e84029040 con 0x7f7e8c02d020 2023-10-05T19:28:28.812 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.806+0000 7f7e9a075700 1 -- 172.21.15.167:0/4093925617 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:29 5.1d 5:b8000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x5652a2f73e90 con 0x5652a2f40590 2023-10-05T19:28:28.813 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.806+0000 7f7e9b077700 1 -- 172.21.15.167:0/4093925617 <== osd.1 v2:172.21.15.167:6800/30035 12 ==== osd_op_reply(29 [pgnls start_epoch 30 out=107b] v30'2 uv2 ondisk = 1) v8 ==== 144+0+107 (crc 0 0 0) 0x7f7e9400a5a0 con 0x5652a2f40590 2023-10-05T19:28:28.813 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.806+0000 7f7e9b077700 1 -- 172.21.15.167:0/4093925617 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:30 5.3 5:c0000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x5652a2f73e90 con 0x7f7e84007030 2023-10-05T19:28:28.814 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.806+0000 7f7e9b077700 1 -- 172.21.15.167:0/4093925617 <== osd.0 v2:172.21.15.118:6800/27690 7 ==== osd_op_reply(30 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f7e9400a5a0 con 0x7f7e84007030 2023-10-05T19:28:28.814 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.806+0000 7f7e9b077700 1 -- 172.21.15.167:0/4093925617 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:31 5.13 5:c8000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x5652a2f73e90 con 0x5652a2f71470 2023-10-05T19:28:28.814 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.810+0000 7f7e9a876700 1 -- 172.21.15.167:0/4093925617 <== osd.3 v2:172.21.15.167:6808/30036 8 ==== osd_op_reply(31 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f7e8c014070 con 0x5652a2f71470 2023-10-05T19:28:28.814 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.810+0000 7f7e9a876700 1 -- 172.21.15.167:0/4093925617 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:32 5.b 5:d0000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x5652a2f73e90 con 0x7f7e84007030 2023-10-05T19:28:28.815 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.810+0000 7f7e9b077700 1 -- 172.21.15.167:0/4093925617 <== osd.0 v2:172.21.15.118:6800/27690 8 ==== osd_op_reply(32 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f7e9400a5a0 con 0x7f7e84007030 2023-10-05T19:28:28.815 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.810+0000 7f7e9b077700 1 -- 172.21.15.167:0/4093925617 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:33 5.1b 5:d8000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x5652a2f73e90 con 0x7f7e8c02d020 2023-10-05T19:28:28.816 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.810+0000 7f7e9a075700 1 -- 172.21.15.167:0/4093925617 <== osd.2 v2:172.21.15.167:6816/30037 5 ==== osd_op_reply(33 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f7e84029040 con 0x7f7e8c02d020 2023-10-05T19:28:28.816 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.810+0000 7f7e9a075700 1 -- 172.21.15.167:0/4093925617 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:34 5.7 5:e0000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x5652a2f73e90 con 0x5652a2f71470 2023-10-05T19:28:28.817 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.810+0000 7f7e9a876700 1 -- 172.21.15.167:0/4093925617 <== osd.3 v2:172.21.15.167:6808/30036 9 ==== osd_op_reply(34 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f7e8c014070 con 0x5652a2f71470 2023-10-05T19:28:28.817 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.810+0000 7f7e9a876700 1 -- 172.21.15.167:0/4093925617 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:35 5.17 5:e8000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x5652a2f73e90 con 0x7f7e84007030 2023-10-05T19:28:28.818 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.814+0000 7f7e9b077700 1 -- 172.21.15.167:0/4093925617 <== osd.0 v2:172.21.15.118:6800/27690 9 ==== osd_op_reply(35 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f7e9400a5a0 con 0x7f7e84007030 2023-10-05T19:28:28.818 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.814+0000 7f7e9b077700 1 -- 172.21.15.167:0/4093925617 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:36 5.f 5:f0000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x5652a2f73e90 con 0x5652a2f40590 2023-10-05T19:28:28.818 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.814+0000 7f7e9b077700 1 -- 172.21.15.167:0/4093925617 <== osd.1 v2:172.21.15.167:6800/30035 13 ==== osd_op_reply(36 [pgnls start_epoch 30 out=115b] v23'2 uv2 ondisk = 1) v8 ==== 144+0+115 (crc 0 0 0) 0x7f7e9400a5a0 con 0x5652a2f40590 2023-10-05T19:28:28.819 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.814+0000 7f7e9b077700 1 -- 172.21.15.167:0/4093925617 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:37 5.1f 5:f8000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x5652a2f73e90 con 0x7f7e8c02d020 2023-10-05T19:28:28.820 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.814+0000 7f7e9a075700 1 -- 172.21.15.167:0/4093925617 <== osd.2 v2:172.21.15.167:6816/30037 6 ==== osd_op_reply(37 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f7e84029040 con 0x7f7e8c02d020 2023-10-05T19:28:28.820 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.814+0000 7f7e9cd41e40 20 RGWRados::pool_iterate: got zonegroup_info.4f1f719d-9648-417b-b10c-e5d939825419 2023-10-05T19:28:28.820 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.814+0000 7f7e9cd41e40 20 RGWRados::pool_iterate: got zone_info.9a9df965-6612-45d4-a45f-6e3c889bacc2 2023-10-05T19:28:28.820 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.814+0000 7f7e9cd41e40 20 RGWRados::pool_iterate: got zone_names.default 2023-10-05T19:28:28.820 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.814+0000 7f7e9cd41e40 20 RGWRados::pool_iterate: got zonegroups_names.default 2023-10-05T19:28:28.820 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.814+0000 7f7e9cd41e40 20 rados->read ofs=0 len=0 2023-10-05T19:28:28.820 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.814+0000 7f7e9cd41e40 1 -- 172.21.15.167:0/4093925617 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:38 5.12 5:49953fa1:::default.realm:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x5652a2f6ff20 con 0x5652a2f40590 2023-10-05T19:28:28.820 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.814+0000 7f7e9b077700 1 -- 172.21.15.167:0/4093925617 <== osd.1 v2:172.21.15.167:6800/30035 14 ==== osd_op_reply(38 default.realm [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 ==== 157+0+0 (crc 0 0 0) 0x7f7e9400a5a0 con 0x5652a2f40590 2023-10-05T19:28:28.820 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.814+0000 7f7e9cd41e40 20 rados_obj.operate() r=-2 bl.length=0 2023-10-05T19:28:28.821 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.814+0000 7f7e9cd41e40 20 rados->read ofs=0 len=0 2023-10-05T19:28:28.821 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.814+0000 7f7e9cd41e40 1 -- 172.21.15.167:0/4093925617 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:39 5.f 5:f4c53578:::zonegroups_names.default:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x5652a2f70320 con 0x5652a2f40590 2023-10-05T19:28:28.822 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.814+0000 7f7e9b077700 1 -- 172.21.15.167:0/4093925617 <== osd.1 v2:172.21.15.167:6800/30035 15 ==== osd_op_reply(39 zonegroups_names.default [read 0~46 out=46b] v0'0 uv2 ondisk = 0) v8 ==== 168+0+46 (crc 0 0 0) 0x7f7e9400a5a0 con 0x5652a2f40590 2023-10-05T19:28:28.822 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.814+0000 7f7e9cd41e40 20 rados_obj.operate() r=0 bl.length=46 2023-10-05T19:28:28.822 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.814+0000 7f7e9cd41e40 20 rados->read ofs=0 len=0 2023-10-05T19:28:28.822 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.814+0000 7f7e9cd41e40 1 -- 172.21.15.167:0/4093925617 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:40 5.c 5:30d299a5:::zonegroup_info.4f1f719d-9648-417b-b10c-e5d939825419:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x5652a2f70320 con 0x7f7e84007030 2023-10-05T19:28:28.822 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.814+0000 7f7e9b077700 1 -- 172.21.15.167:0/4093925617 <== osd.0 v2:172.21.15.118:6800/27690 10 ==== osd_op_reply(40 zonegroup_info.4f1f719d-9648-417b-b10c-e5d939825419 [read 0~362 out=362b] v0'0 uv1 ondisk = 0) v8 ==== 195+0+362 (crc 0 0 0) 0x7f7e9400a5a0 con 0x7f7e84007030 2023-10-05T19:28:28.822 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.814+0000 7f7e9cd41e40 20 rados_obj.operate() r=0 bl.length=362 2023-10-05T19:28:28.823 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.814+0000 7f7e9cd41e40 20 rados->read ofs=0 len=0 2023-10-05T19:28:28.823 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.814+0000 7f7e9cd41e40 1 -- 172.21.15.167:0/4093925617 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:41 5.13 5:cbbf6c60:::converted:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x5652a2f70320 con 0x5652a2f71470 2023-10-05T19:28:28.823 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.814+0000 7f7e9a876700 1 -- 172.21.15.167:0/4093925617 <== osd.3 v2:172.21.15.167:6808/30036 10 ==== osd_op_reply(41 converted [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 ==== 153+0+0 (crc 0 0 0) 0x7f7e8c014070 con 0x5652a2f71470 2023-10-05T19:28:28.823 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.814+0000 7f7e9cd41e40 20 rados_obj.operate() r=-2 bl.length=0 2023-10-05T19:28:28.823 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.814+0000 7f7e9cd41e40 20 rados->read ofs=0 len=0 2023-10-05T19:28:28.823 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.814+0000 7f7e9cd41e40 1 -- 172.21.15.167:0/4093925617 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:42 5.12 5:49953fa1:::default.realm:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x5652a2f74300 con 0x5652a2f40590 2023-10-05T19:28:28.824 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.818+0000 7f7e9b077700 1 -- 172.21.15.167:0/4093925617 <== osd.1 v2:172.21.15.167:6800/30035 16 ==== osd_op_reply(42 default.realm [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 ==== 157+0+0 (crc 0 0 0) 0x7f7e9400a5a0 con 0x5652a2f40590 2023-10-05T19:28:28.824 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.818+0000 7f7e9cd41e40 20 rados_obj.operate() r=-2 bl.length=0 2023-10-05T19:28:28.824 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.818+0000 7f7e9cd41e40 20 rados->read ofs=0 len=0 2023-10-05T19:28:28.824 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.818+0000 7f7e9cd41e40 1 -- 172.21.15.167:0/4093925617 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:43 5.f 5:f4c53578:::zonegroups_names.default:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x5652a2f6fac0 con 0x5652a2f40590 2023-10-05T19:28:28.824 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.818+0000 7f7e9b077700 1 -- 172.21.15.167:0/4093925617 <== osd.1 v2:172.21.15.167:6800/30035 17 ==== osd_op_reply(43 zonegroups_names.default [read 0~46 out=46b] v0'0 uv2 ondisk = 0) v8 ==== 168+0+46 (crc 0 0 0) 0x7f7e9400a5a0 con 0x5652a2f40590 2023-10-05T19:28:28.824 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.818+0000 7f7e9cd41e40 20 rados_obj.operate() r=0 bl.length=46 2023-10-05T19:28:28.824 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.818+0000 7f7e9cd41e40 1 -- 172.21.15.167:0/4093925617 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:44 5.0 5:00000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x5652a2f6fa90 con 0x5652a2f71470 2023-10-05T19:28:28.825 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.818+0000 7f7e9a876700 1 -- 172.21.15.167:0/4093925617 <== osd.3 v2:172.21.15.167:6808/30036 11 ==== osd_op_reply(44 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f7e8c014070 con 0x5652a2f71470 2023-10-05T19:28:28.825 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.818+0000 7f7e9a876700 1 -- 172.21.15.167:0/4093925617 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:45 5.10 5:08000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x5652a2f6fa90 con 0x7f7e8c02d020 2023-10-05T19:28:28.826 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.822+0000 7f7e9a075700 1 -- 172.21.15.167:0/4093925617 <== osd.2 v2:172.21.15.167:6816/30037 7 ==== osd_op_reply(45 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f7e84029040 con 0x7f7e8c02d020 2023-10-05T19:28:28.826 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.822+0000 7f7e9a075700 1 -- 172.21.15.167:0/4093925617 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:46 5.8 5:10000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x5652a2f6fa90 con 0x7f7e84007030 2023-10-05T19:28:28.827 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.822+0000 7f7e9b077700 1 -- 172.21.15.167:0/4093925617 <== osd.0 v2:172.21.15.118:6800/27690 11 ==== osd_op_reply(46 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f7e9400a5a0 con 0x7f7e84007030 2023-10-05T19:28:28.827 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.822+0000 7f7e9b077700 1 -- 172.21.15.167:0/4093925617 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:47 5.18 5:18000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x5652a2f6ff20 con 0x5652a2f40590 2023-10-05T19:28:28.828 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.822+0000 7f7e9b077700 1 -- 172.21.15.167:0/4093925617 <== osd.1 v2:172.21.15.167:6800/30035 18 ==== osd_op_reply(47 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f7e9400a5a0 con 0x5652a2f40590 2023-10-05T19:28:28.828 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.822+0000 7f7e9b077700 1 -- 172.21.15.167:0/4093925617 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:48 5.4 5:20000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x5652a2f6ff20 con 0x5652a2f40590 2023-10-05T19:28:28.828 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.822+0000 7f7e9b077700 1 -- 172.21.15.167:0/4093925617 <== osd.1 v2:172.21.15.167:6800/30035 19 ==== osd_op_reply(48 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f7e9400a5a0 con 0x5652a2f40590 2023-10-05T19:28:28.829 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.822+0000 7f7e9b077700 1 -- 172.21.15.167:0/4093925617 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:49 5.14 5:28000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x5652a2f6ff20 con 0x7f7e84007030 2023-10-05T19:28:28.830 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.822+0000 7f7e9b077700 1 -- 172.21.15.167:0/4093925617 <== osd.0 v2:172.21.15.118:6800/27690 12 ==== osd_op_reply(49 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f7e9400a5a0 con 0x7f7e84007030 2023-10-05T19:28:28.830 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.822+0000 7f7e9b077700 1 -- 172.21.15.167:0/4093925617 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:50 5.c 5:30000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x5652a2f6ff20 con 0x7f7e84007030 2023-10-05T19:28:28.830 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.826+0000 7f7e9b077700 1 -- 172.21.15.167:0/4093925617 <== osd.0 v2:172.21.15.118:6800/27690 13 ==== osd_op_reply(50 [pgnls start_epoch 30 out=112b] v23'1 uv1 ondisk = 1) v8 ==== 144+0+112 (crc 0 0 0) 0x7f7e9400a5a0 con 0x7f7e84007030 2023-10-05T19:28:28.830 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.826+0000 7f7e9b077700 1 -- 172.21.15.167:0/4093925617 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:51 5.1c 5:38000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x5652a2f6ff20 con 0x7f7e8c02d020 2023-10-05T19:28:28.831 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.826+0000 7f7e9a075700 1 -- 172.21.15.167:0/4093925617 <== osd.2 v2:172.21.15.167:6816/30037 8 ==== osd_op_reply(51 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f7e84029040 con 0x7f7e8c02d020 2023-10-05T19:28:28.831 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.826+0000 7f7e9a075700 1 -- 172.21.15.167:0/4093925617 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:52 5.2 5:40000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x5652a2f6ff20 con 0x5652a2f71470 2023-10-05T19:28:28.832 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.826+0000 7f7e9a876700 1 -- 172.21.15.167:0/4093925617 <== osd.3 v2:172.21.15.167:6808/30036 12 ==== osd_op_reply(52 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f7e8c014070 con 0x5652a2f71470 2023-10-05T19:28:28.832 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.826+0000 7f7e9a876700 1 -- 172.21.15.167:0/4093925617 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:53 5.12 5:48000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x5652a2f6ff20 con 0x5652a2f40590 2023-10-05T19:28:28.832 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.826+0000 7f7e9b077700 1 -- 172.21.15.167:0/4093925617 <== osd.1 v2:172.21.15.167:6800/30035 20 ==== osd_op_reply(53 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f7e9400a5a0 con 0x5652a2f40590 2023-10-05T19:28:28.832 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.826+0000 7f7e9b077700 1 -- 172.21.15.167:0/4093925617 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:54 5.a 5:50000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x5652a2f6ff20 con 0x7f7e84007030 2023-10-05T19:28:28.833 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.826+0000 7f7e9b077700 1 -- 172.21.15.167:0/4093925617 <== osd.0 v2:172.21.15.118:6800/27690 14 ==== osd_op_reply(54 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f7e9400a5a0 con 0x7f7e84007030 2023-10-05T19:28:28.833 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.826+0000 7f7e9b077700 1 -- 172.21.15.167:0/4093925617 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:55 5.1a 5:58000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x5652a2f6ff20 con 0x5652a2f40590 2023-10-05T19:28:28.834 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.830+0000 7f7e9b077700 1 -- 172.21.15.167:0/4093925617 <== osd.1 v2:172.21.15.167:6800/30035 21 ==== osd_op_reply(55 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f7e9400a5a0 con 0x5652a2f40590 2023-10-05T19:28:28.834 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.830+0000 7f7e9b077700 1 -- 172.21.15.167:0/4093925617 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:56 5.6 5:60000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x5652a2f6ff20 con 0x7f7e8c02d020 2023-10-05T19:28:28.834 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.830+0000 7f7e9a075700 1 -- 172.21.15.167:0/4093925617 <== osd.2 v2:172.21.15.167:6816/30037 9 ==== osd_op_reply(56 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f7e84029040 con 0x7f7e8c02d020 2023-10-05T19:28:28.835 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.830+0000 7f7e9a075700 1 -- 172.21.15.167:0/4093925617 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:57 5.16 5:68000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x5652a2f6ff20 con 0x5652a2f71470 2023-10-05T19:28:28.835 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.830+0000 7f7e9a876700 1 -- 172.21.15.167:0/4093925617 <== osd.3 v2:172.21.15.167:6808/30036 13 ==== osd_op_reply(57 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f7e8c014070 con 0x5652a2f71470 2023-10-05T19:28:28.835 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.830+0000 7f7e9a876700 1 -- 172.21.15.167:0/4093925617 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:58 5.e 5:70000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x5652a2f6ff20 con 0x5652a2f71470 2023-10-05T19:28:28.835 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.830+0000 7f7e9a876700 1 -- 172.21.15.167:0/4093925617 <== osd.3 v2:172.21.15.167:6808/30036 14 ==== osd_op_reply(58 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f7e8c014070 con 0x5652a2f71470 2023-10-05T19:28:28.836 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.830+0000 7f7e9a876700 1 -- 172.21.15.167:0/4093925617 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:59 5.1e 5:78000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x5652a2f6ff20 con 0x7f7e84007030 2023-10-05T19:28:28.836 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.830+0000 7f7e9b077700 1 -- 172.21.15.167:0/4093925617 <== osd.0 v2:172.21.15.118:6800/27690 15 ==== osd_op_reply(59 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f7e9403f040 con 0x7f7e84007030 2023-10-05T19:28:28.837 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.830+0000 7f7e9b077700 1 -- 172.21.15.167:0/4093925617 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:60 5.1 5:80000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x5652a2f6ff20 con 0x5652a2f71470 2023-10-05T19:28:28.837 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.830+0000 7f7e9a876700 1 -- 172.21.15.167:0/4093925617 <== osd.3 v2:172.21.15.167:6808/30036 15 ==== osd_op_reply(60 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f7e8c014070 con 0x5652a2f71470 2023-10-05T19:28:28.837 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.830+0000 7f7e9a876700 1 -- 172.21.15.167:0/4093925617 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:61 5.11 5:88000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x5652a2f6ff20 con 0x5652a2f71470 2023-10-05T19:28:28.837 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.830+0000 7f7e9a876700 1 -- 172.21.15.167:0/4093925617 <== osd.3 v2:172.21.15.167:6808/30036 16 ==== osd_op_reply(61 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f7e8c014070 con 0x5652a2f71470 2023-10-05T19:28:28.837 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.830+0000 7f7e9a876700 1 -- 172.21.15.167:0/4093925617 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:62 5.9 5:90000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x5652a2f6ff20 con 0x5652a2f40590 2023-10-05T19:28:28.838 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.830+0000 7f7e9b077700 1 -- 172.21.15.167:0/4093925617 <== osd.1 v2:172.21.15.167:6800/30035 22 ==== osd_op_reply(62 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f7e9403f040 con 0x5652a2f40590 2023-10-05T19:28:28.838 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.830+0000 7f7e9b077700 1 -- 172.21.15.167:0/4093925617 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:63 5.19 5:98000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x5652a2f6ff20 con 0x5652a2f40590 2023-10-05T19:28:28.838 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.830+0000 7f7e9b077700 1 -- 172.21.15.167:0/4093925617 <== osd.1 v2:172.21.15.167:6800/30035 23 ==== osd_op_reply(63 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f7e9403f040 con 0x5652a2f40590 2023-10-05T19:28:28.838 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.830+0000 7f7e9b077700 1 -- 172.21.15.167:0/4093925617 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:64 5.5 5:a0000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x5652a2f6ff20 con 0x7f7e84007030 2023-10-05T19:28:28.839 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.834+0000 7f7e9b077700 1 -- 172.21.15.167:0/4093925617 <== osd.0 v2:172.21.15.118:6800/27690 16 ==== osd_op_reply(64 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f7e9403f040 con 0x7f7e84007030 2023-10-05T19:28:28.839 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.834+0000 7f7e9b077700 1 -- 172.21.15.167:0/4093925617 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:65 5.15 5:a8000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x5652a2f6ff20 con 0x5652a2f71470 2023-10-05T19:28:28.839 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.834+0000 7f7e9a876700 1 -- 172.21.15.167:0/4093925617 <== osd.3 v2:172.21.15.167:6808/30036 17 ==== osd_op_reply(65 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f7e8c014070 con 0x5652a2f71470 2023-10-05T19:28:28.839 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.834+0000 7f7e9a876700 1 -- 172.21.15.167:0/4093925617 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:66 5.d 5:b0000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x5652a2f6ff20 con 0x7f7e8c02d020 2023-10-05T19:28:28.840 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.834+0000 7f7e9a075700 1 -- 172.21.15.167:0/4093925617 <== osd.2 v2:172.21.15.167:6816/30037 10 ==== osd_op_reply(66 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f7e84029040 con 0x7f7e8c02d020 2023-10-05T19:28:28.840 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.834+0000 7f7e9a075700 1 -- 172.21.15.167:0/4093925617 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:67 5.1d 5:b8000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x5652a2f6ff20 con 0x5652a2f40590 2023-10-05T19:28:28.841 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.834+0000 7f7e9b077700 1 -- 172.21.15.167:0/4093925617 <== osd.1 v2:172.21.15.167:6800/30035 24 ==== osd_op_reply(67 [pgnls start_epoch 30 out=107b] v30'2 uv2 ondisk = 1) v8 ==== 144+0+107 (crc 0 0 0) 0x7f7e9403f040 con 0x5652a2f40590 2023-10-05T19:28:28.841 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.834+0000 7f7e9b077700 1 -- 172.21.15.167:0/4093925617 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:68 5.3 5:c0000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x5652a2f6ff20 con 0x7f7e84007030 2023-10-05T19:28:28.841 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.834+0000 7f7e9b077700 1 -- 172.21.15.167:0/4093925617 <== osd.0 v2:172.21.15.118:6800/27690 17 ==== osd_op_reply(68 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f7e9403f040 con 0x7f7e84007030 2023-10-05T19:28:28.841 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.834+0000 7f7e9b077700 1 -- 172.21.15.167:0/4093925617 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:69 5.13 5:c8000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x5652a2f6ff20 con 0x5652a2f71470 2023-10-05T19:28:28.842 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.838+0000 7f7e9a876700 1 -- 172.21.15.167:0/4093925617 <== osd.3 v2:172.21.15.167:6808/30036 18 ==== osd_op_reply(69 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f7e8c014070 con 0x5652a2f71470 2023-10-05T19:28:28.842 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.838+0000 7f7e9a876700 1 -- 172.21.15.167:0/4093925617 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:70 5.b 5:d0000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x5652a2f6ff20 con 0x7f7e84007030 2023-10-05T19:28:28.843 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.838+0000 7f7e9b077700 1 -- 172.21.15.167:0/4093925617 <== osd.0 v2:172.21.15.118:6800/27690 18 ==== osd_op_reply(70 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f7e9403f040 con 0x7f7e84007030 2023-10-05T19:28:28.843 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.838+0000 7f7e9b077700 1 -- 172.21.15.167:0/4093925617 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:71 5.1b 5:d8000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x5652a2f6ff20 con 0x7f7e8c02d020 2023-10-05T19:28:28.844 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.838+0000 7f7e9a075700 1 -- 172.21.15.167:0/4093925617 <== osd.2 v2:172.21.15.167:6816/30037 11 ==== osd_op_reply(71 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f7e84029040 con 0x7f7e8c02d020 2023-10-05T19:28:28.844 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.838+0000 7f7e9a075700 1 -- 172.21.15.167:0/4093925617 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:72 5.7 5:e0000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x5652a2f6ff20 con 0x5652a2f71470 2023-10-05T19:28:28.844 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.838+0000 7f7e9a876700 1 -- 172.21.15.167:0/4093925617 <== osd.3 v2:172.21.15.167:6808/30036 19 ==== osd_op_reply(72 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f7e8c014070 con 0x5652a2f71470 2023-10-05T19:28:28.844 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.838+0000 7f7e9a876700 1 -- 172.21.15.167:0/4093925617 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:73 5.17 5:e8000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x5652a2f6ff20 con 0x7f7e84007030 2023-10-05T19:28:28.845 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.838+0000 7f7e9b077700 1 -- 172.21.15.167:0/4093925617 <== osd.0 v2:172.21.15.118:6800/27690 19 ==== osd_op_reply(73 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f7e9403f040 con 0x7f7e84007030 2023-10-05T19:28:28.845 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.838+0000 7f7e9b077700 1 -- 172.21.15.167:0/4093925617 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:74 5.f 5:f0000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x5652a2f6ff20 con 0x5652a2f40590 2023-10-05T19:28:28.846 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.838+0000 7f7e9b077700 1 -- 172.21.15.167:0/4093925617 <== osd.1 v2:172.21.15.167:6800/30035 25 ==== osd_op_reply(74 [pgnls start_epoch 30 out=115b] v23'2 uv2 ondisk = 1) v8 ==== 144+0+115 (crc 0 0 0) 0x7f7e9403f040 con 0x5652a2f40590 2023-10-05T19:28:28.846 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.838+0000 7f7e9b077700 1 -- 172.21.15.167:0/4093925617 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:75 5.1f 5:f8000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x5652a2f6ff20 con 0x7f7e8c02d020 2023-10-05T19:28:28.847 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.842+0000 7f7e9a075700 1 -- 172.21.15.167:0/4093925617 <== osd.2 v2:172.21.15.167:6816/30037 12 ==== osd_op_reply(75 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f7e84029040 con 0x7f7e8c02d020 2023-10-05T19:28:28.847 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.842+0000 7f7e9cd41e40 20 RGWRados::pool_iterate: got zonegroup_info.4f1f719d-9648-417b-b10c-e5d939825419 2023-10-05T19:28:28.847 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.842+0000 7f7e9cd41e40 20 RGWRados::pool_iterate: got zone_info.9a9df965-6612-45d4-a45f-6e3c889bacc2 2023-10-05T19:28:28.847 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.842+0000 7f7e9cd41e40 20 RGWRados::pool_iterate: got zone_names.default 2023-10-05T19:28:28.847 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.842+0000 7f7e9cd41e40 20 RGWRados::pool_iterate: got zonegroups_names.default 2023-10-05T19:28:28.847 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.842+0000 7f7e9cd41e40 20 rados->read ofs=0 len=0 2023-10-05T19:28:28.848 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.842+0000 7f7e9cd41e40 1 -- 172.21.15.167:0/4093925617 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:76 5.f 5:f43b6ece:::zone_names.default:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x5652a2f6fa90 con 0x5652a2f40590 2023-10-05T19:28:28.848 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.842+0000 7f7e9b077700 1 -- 172.21.15.167:0/4093925617 <== osd.1 v2:172.21.15.167:6800/30035 26 ==== osd_op_reply(76 zone_names.default [read 0~46 out=46b] v0'0 uv1 ondisk = 0) v8 ==== 162+0+46 (crc 0 0 0) 0x7f7e9403f040 con 0x5652a2f40590 2023-10-05T19:28:28.848 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.842+0000 7f7e9cd41e40 20 rados_obj.operate() r=0 bl.length=46 2023-10-05T19:28:28.848 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.842+0000 7f7e9cd41e40 20 rados->read ofs=0 len=0 2023-10-05T19:28:28.848 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.842+0000 7f7e9cd41e40 1 -- 172.21.15.167:0/4093925617 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:77 5.1d 5:b8374eb4:::zone_info.9a9df965-6612-45d4-a45f-6e3c889bacc2:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x5652a2f747f0 con 0x5652a2f40590 2023-10-05T19:28:28.849 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.842+0000 7f7e9b077700 1 -- 172.21.15.167:0/4093925617 <== osd.1 v2:172.21.15.167:6800/30035 27 ==== osd_op_reply(77 zone_info.9a9df965-6612-45d4-a45f-6e3c889bacc2 [read 0~874 out=874b] v0'0 uv2 ondisk = 0) v8 ==== 190+0+874 (crc 0 0 0) 0x7f7e9403f040 con 0x5652a2f40590 2023-10-05T19:28:28.849 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.842+0000 7f7e9cd41e40 20 rados_obj.operate() r=0 bl.length=874 2023-10-05T19:28:28.849 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.842+0000 7f7e9cd41e40 20 rados->read ofs=0 len=0 2023-10-05T19:28:28.849 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.842+0000 7f7e9cd41e40 1 -- 172.21.15.167:0/4093925617 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:78 5.f 5:f4c53578:::zonegroups_names.default:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x5652a2f74a60 con 0x5652a2f40590 2023-10-05T19:28:28.849 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.842+0000 7f7e9b077700 1 -- 172.21.15.167:0/4093925617 <== osd.1 v2:172.21.15.167:6800/30035 28 ==== osd_op_reply(78 zonegroups_names.default [read 0~46 out=46b] v0'0 uv2 ondisk = 0) v8 ==== 168+0+46 (crc 0 0 0) 0x7f7e9403f040 con 0x5652a2f40590 2023-10-05T19:28:28.850 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.842+0000 7f7e9cd41e40 20 rados_obj.operate() r=0 bl.length=46 2023-10-05T19:28:28.850 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.842+0000 7f7e9cd41e40 20 rados->read ofs=0 len=0 2023-10-05T19:28:28.850 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.842+0000 7f7e9cd41e40 1 -- 172.21.15.167:0/4093925617 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:79 5.c 5:30d299a5:::zonegroup_info.4f1f719d-9648-417b-b10c-e5d939825419:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x5652a2f74a80 con 0x7f7e84007030 2023-10-05T19:28:28.850 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.842+0000 7f7e9b077700 1 -- 172.21.15.167:0/4093925617 <== osd.0 v2:172.21.15.118:6800/27690 20 ==== osd_op_reply(79 zonegroup_info.4f1f719d-9648-417b-b10c-e5d939825419 [read 0~362 out=362b] v0'0 uv1 ondisk = 0) v8 ==== 195+0+362 (crc 0 0 0) 0x7f7e94052040 con 0x7f7e84007030 2023-10-05T19:28:28.850 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.842+0000 7f7e9cd41e40 20 rados_obj.operate() r=0 bl.length=362 2023-10-05T19:28:28.850 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.842+0000 7f7e9cd41e40 20 rados->read ofs=0 len=0 2023-10-05T19:28:28.850 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.842+0000 7f7e9cd41e40 1 -- 172.21.15.167:0/4093925617 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:80 5.3 5:c0ad8cbd:::region_map:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x5652a2f75410 con 0x7f7e84007030 2023-10-05T19:28:28.852 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.842+0000 7f7e9b077700 1 -- 172.21.15.167:0/4093925617 <== osd.0 v2:172.21.15.118:6800/27690 21 ==== osd_op_reply(80 region_map [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 ==== 154+0+0 (crc 0 0 0) 0x7f7e94052040 con 0x7f7e84007030 2023-10-05T19:28:28.852 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.842+0000 7f7e9cd41e40 20 rados_obj.operate() r=-2 bl.length=0 2023-10-05T19:28:28.852 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.842+0000 7f7e9cd41e40 20 zone default found 2023-10-05T19:28:28.852 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.842+0000 7f7e9cd41e40 4 Realm: () 2023-10-05T19:28:28.852 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.842+0000 7f7e9cd41e40 4 ZoneGroup: default (4f1f719d-9648-417b-b10c-e5d939825419) 2023-10-05T19:28:28.852 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.842+0000 7f7e9cd41e40 4 Zone: default (9a9df965-6612-45d4-a45f-6e3c889bacc2) 2023-10-05T19:28:28.852 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.842+0000 7f7e9cd41e40 10 cannot find current period zonegroup using local zonegroup configuration 2023-10-05T19:28:28.852 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.842+0000 7f7e9cd41e40 20 zonegroup default 2023-10-05T19:28:28.852 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.842+0000 7f7e9cd41e40 20 rados->read ofs=0 len=0 2023-10-05T19:28:28.852 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.842+0000 7f7e9cd41e40 1 -- 172.21.15.167:0/4093925617 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:81 5.3 5:c52100b6:::period_config.default:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x5652a2f75410 con 0x7f7e84007030 2023-10-05T19:28:28.852 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.846+0000 7f7e9b077700 1 -- 172.21.15.167:0/4093925617 <== osd.0 v2:172.21.15.118:6800/27690 22 ==== osd_op_reply(81 period_config.default [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 ==== 165+0+0 (crc 0 0 0) 0x7f7e94052040 con 0x7f7e84007030 2023-10-05T19:28:28.853 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.846+0000 7f7e9cd41e40 20 rados_obj.operate() r=-2 bl.length=0 2023-10-05T19:28:28.853 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.846+0000 7f7e9cd41e40 20 rados->read ofs=0 len=0 2023-10-05T19:28:28.853 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.846+0000 7f7e9cd41e40 1 -- 172.21.15.167:0/4093925617 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:82 6.2 6:4347d321:::bucket.sync-source-hints.:head [call version.read in=11b,read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x5652a2f76cb0 con 0x5652a2f71470 2023-10-05T19:28:28.853 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.846+0000 7f7e9a876700 1 -- 172.21.15.167:0/4093925617 <== osd.3 v2:172.21.15.167:6808/30036 20 ==== osd_op_reply(82 bucket.sync-source-hints. [call,read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 ==== 211+0+0 (crc 0 0 0) 0x7f7e8c014070 con 0x5652a2f71470 2023-10-05T19:28:28.853 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.846+0000 7f7e9cd41e40 20 rados_obj.operate() r=-2 bl.length=0 2023-10-05T19:28:28.853 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.846+0000 7f7e9cd41e40 20 rados->read ofs=0 len=0 2023-10-05T19:28:28.853 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.846+0000 7f7e9cd41e40 1 -- 172.21.15.167:0/4093925617 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:83 6.b 6:d467b91b:::bucket.sync-target-hints.:head [call version.read in=11b,read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x5652a2f77060 con 0x5652a2f71470 2023-10-05T19:28:28.853 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.846+0000 7f7e9a876700 1 -- 172.21.15.167:0/4093925617 <== osd.3 v2:172.21.15.167:6808/30036 21 ==== osd_op_reply(83 bucket.sync-target-hints. [call,read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 ==== 211+0+0 (crc 0 0 0) 0x7f7e8c014070 con 0x5652a2f71470 2023-10-05T19:28:28.853 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.846+0000 7f7e9cd41e40 20 rados_obj.operate() r=-2 bl.length=0 2023-10-05T19:28:28.853 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.846+0000 7f7e9cd41e40 20 started sync module instance, tier type = 2023-10-05T19:28:28.853 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.846+0000 7f7e9cd41e40 20 started zone id=9a9df965-6612-45d4-a45f-6e3c889bacc2 (name=default) with tier type = 2023-10-05T19:28:28.853 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.846+0000 7f7e9cd41e40 1 -- 172.21.15.167:0/4093925617 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:84 7.1f 7:f95f44c2:::notify.0:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x5652a2f76370 con 0x7f7e8c02d020 2023-10-05T19:28:28.854 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.846+0000 7f7e9a075700 1 -- 172.21.15.167:0/4093925617 <== osd.2 v2:172.21.15.167:6816/30037 13 ==== osd_op_reply(84 notify.0 [create] v30'66 uv66 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f7e84029040 con 0x7f7e8c02d020 2023-10-05T19:28:28.854 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.850+0000 7f7e9cd41e40 1 -- 172.21.15.167:0/4093925617 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:85 7.1f 7:f95f44c2:::notify.0:head [watch watch cookie 94912921427584] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x5652a2f77410 con 0x7f7e8c02d020 2023-10-05T19:28:28.854 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.850+0000 7f7e9cd41e40 1 -- 172.21.15.167:0/4093925617 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:86 7.0 7:05bf5b68:::notify.1:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x5652a2f77cb0 con 0x5652a2f40590 2023-10-05T19:28:28.856 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.850+0000 7f7e9a075700 1 -- 172.21.15.167:0/4093925617 <== osd.2 v2:172.21.15.167:6816/30037 14 ==== osd_op_reply(85 notify.0 [watch watch cookie 94912921427584] v30'67 uv66 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f7e84029040 con 0x7f7e8c02d020 2023-10-05T19:28:28.856 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.850+0000 7f7e9b077700 1 -- 172.21.15.167:0/4093925617 <== osd.1 v2:172.21.15.167:6800/30035 29 ==== osd_op_reply(86 notify.1 [create] v30'66 uv66 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f7e94052040 con 0x5652a2f40590 2023-10-05T19:28:28.856 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.850+0000 7f7e9cd41e40 1 -- 172.21.15.167:0/4093925617 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:87 7.0 7:05bf5b68:::notify.1:head [watch watch cookie 94912921434048] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x5652a2f76890 con 0x5652a2f40590 2023-10-05T19:28:28.856 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.850+0000 7f7e9cd41e40 1 -- 172.21.15.167:0/4093925617 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:88 7.15 7:a93a5511:::notify.2:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x5652a2f78720 con 0x5652a2f71470 2023-10-05T19:28:28.859 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.854+0000 7f7e9a876700 1 -- 172.21.15.167:0/4093925617 <== osd.3 v2:172.21.15.167:6808/30036 22 ==== osd_op_reply(88 notify.2 [create] v30'66 uv66 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f7e8c014070 con 0x5652a2f71470 2023-10-05T19:28:28.859 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.854+0000 7f7e9cd41e40 1 -- 172.21.15.167:0/4093925617 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:89 7.15 7:a93a5511:::notify.2:head [watch watch cookie 94912921436656] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x5652a2f79010 con 0x5652a2f71470 2023-10-05T19:28:28.859 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.854+0000 7f7e9cd41e40 1 -- 172.21.15.167:0/4093925617 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:90 7.e 7:7759931f:::notify.3:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x5652a2f79a20 con 0x7f7e8c02d020 2023-10-05T19:28:28.859 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.854+0000 7f7e9b077700 1 -- 172.21.15.167:0/4093925617 <== osd.1 v2:172.21.15.167:6800/30035 30 ==== osd_op_reply(87 notify.1 [watch watch cookie 94912921434048] v30'67 uv66 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f7e94052040 con 0x5652a2f40590 2023-10-05T19:28:28.860 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.854+0000 7f7e9a075700 1 -- 172.21.15.167:0/4093925617 <== osd.2 v2:172.21.15.167:6816/30037 15 ==== osd_op_reply(90 notify.3 [create] v30'66 uv66 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f7e84029040 con 0x7f7e8c02d020 2023-10-05T19:28:28.860 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.854+0000 7f7e9cd41e40 1 -- 172.21.15.167:0/4093925617 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:91 7.e 7:7759931f:::notify.3:head [watch watch cookie 94912921435152] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x5652a2f79650 con 0x7f7e8c02d020 2023-10-05T19:28:28.861 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.854+0000 7f7e9cd41e40 1 -- 172.21.15.167:0/4093925617 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:92 7.d 7:b4812045:::notify.4:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x5652a2f7aa20 con 0x5652a2f40590 2023-10-05T19:28:28.861 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.854+0000 7f7e9a876700 1 -- 172.21.15.167:0/4093925617 <== osd.3 v2:172.21.15.167:6808/30036 23 ==== osd_op_reply(89 notify.2 [watch watch cookie 94912921436656] v30'67 uv66 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f7e8c014070 con 0x5652a2f71470 2023-10-05T19:28:28.862 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.858+0000 7f7e9a075700 1 -- 172.21.15.167:0/4093925617 <== osd.2 v2:172.21.15.167:6816/30037 16 ==== osd_op_reply(91 notify.3 [watch watch cookie 94912921435152] v30'67 uv66 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f7e84029040 con 0x7f7e8c02d020 2023-10-05T19:28:28.862 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.858+0000 7f7e9b077700 1 -- 172.21.15.167:0/4093925617 <== osd.1 v2:172.21.15.167:6800/30035 31 ==== osd_op_reply(92 notify.4 [create] v30'66 uv66 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f7e94052040 con 0x5652a2f40590 2023-10-05T19:28:28.862 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.858+0000 7f7e9cd41e40 1 -- 172.21.15.167:0/4093925617 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:93 7.d 7:b4812045:::notify.4:head [watch watch cookie 94912921439008] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x5652a2f7a4f0 con 0x5652a2f40590 2023-10-05T19:28:28.863 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.858+0000 7f7e9cd41e40 1 -- 172.21.15.167:0/4093925617 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:94 7.3 7:c609908c:::notify.5:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x5652a2f7b310 con 0x7f7e84007030 2023-10-05T19:28:28.864 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.858+0000 7f7e9b077700 1 -- 172.21.15.167:0/4093925617 <== osd.0 v2:172.21.15.118:6800/27690 23 ==== osd_op_reply(94 notify.5 [create] v30'66 uv66 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f7e94052040 con 0x7f7e84007030 2023-10-05T19:28:28.864 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.858+0000 7f7e9cd41e40 1 -- 172.21.15.167:0/4093925617 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:95 7.3 7:c609908c:::notify.5:head [watch watch cookie 94912921447888] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x5652a2f7b710 con 0x7f7e84007030 2023-10-05T19:28:28.865 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.858+0000 7f7e9cd41e40 1 -- 172.21.15.167:0/4093925617 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:96 7.14 7:2b04a3e9:::notify.6:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x5652a2f7b710 con 0x7f7e84007030 2023-10-05T19:28:28.865 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.858+0000 7f7e9b077700 1 -- 172.21.15.167:0/4093925617 <== osd.1 v2:172.21.15.167:6800/30035 32 ==== osd_op_reply(93 notify.4 [watch watch cookie 94912921439008] v30'67 uv66 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f7e94052040 con 0x5652a2f40590 2023-10-05T19:28:28.866 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.862+0000 7f7e9b077700 1 -- 172.21.15.167:0/4093925617 <== osd.0 v2:172.21.15.118:6800/27690 24 ==== osd_op_reply(95 notify.5 [watch watch cookie 94912921447888] v30'67 uv66 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f7e94052040 con 0x7f7e84007030 2023-10-05T19:28:28.867 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.862+0000 7f7e9b077700 1 -- 172.21.15.167:0/4093925617 <== osd.0 v2:172.21.15.118:6800/27690 25 ==== osd_op_reply(96 notify.6 [create] v30'66 uv66 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f7e94052040 con 0x7f7e84007030 2023-10-05T19:28:28.867 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.862+0000 7f7e9cd41e40 1 -- 172.21.15.167:0/4093925617 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:97 7.14 7:2b04a3e9:::notify.6:head [watch watch cookie 94912921448992] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x5652a2f7b670 con 0x7f7e84007030 2023-10-05T19:28:28.867 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.862+0000 7f7e9cd41e40 1 -- 172.21.15.167:0/4093925617 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:98 7.9 7:93e5b521:::notify.7:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x5652a2f7c610 con 0x7f7e84007030 2023-10-05T19:28:28.869 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.862+0000 7f7e9b077700 1 -- 172.21.15.167:0/4093925617 <== osd.0 v2:172.21.15.118:6800/27690 26 ==== osd_op_reply(97 notify.6 [watch watch cookie 94912921448992] v30'67 uv66 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f7e94052040 con 0x7f7e84007030 2023-10-05T19:28:28.870 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.866+0000 7f7e9b077700 1 -- 172.21.15.167:0/4093925617 <== osd.0 v2:172.21.15.118:6800/27690 27 ==== osd_op_reply(98 notify.7 [create] v30'66 uv66 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f7e94052040 con 0x7f7e84007030 2023-10-05T19:28:28.870 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.866+0000 7f7e9cd41e40 1 -- 172.21.15.167:0/4093925617 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:99 7.9 7:93e5b521:::notify.7:head [watch watch cookie 94912921451568] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x5652a2f7c570 con 0x7f7e84007030 2023-10-05T19:28:28.870 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.866+0000 7f7e9cd41e40 20 add_watcher() i=0 2023-10-05T19:28:28.870 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.866+0000 7f7e9cd41e40 20 add_watcher() i=1 2023-10-05T19:28:28.870 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.866+0000 7f7e9cd41e40 20 add_watcher() i=2 2023-10-05T19:28:28.870 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.866+0000 7f7e9cd41e40 20 add_watcher() i=3 2023-10-05T19:28:28.871 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.866+0000 7f7e9cd41e40 20 add_watcher() i=4 2023-10-05T19:28:28.871 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.866+0000 7f7e9cd41e40 20 add_watcher() i=5 2023-10-05T19:28:28.871 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.866+0000 7f7e9cd41e40 20 add_watcher() i=6 2023-10-05T19:28:28.872 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.866+0000 7f7e9b077700 1 -- 172.21.15.167:0/4093925617 <== osd.0 v2:172.21.15.118:6800/27690 28 ==== osd_op_reply(99 notify.7 [watch watch cookie 94912921451568] v30'67 uv66 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f7e94052040 con 0x7f7e84007030 2023-10-05T19:28:28.872 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.866+0000 7f7e9cd41e40 20 add_watcher() i=7 2023-10-05T19:28:28.872 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.866+0000 7f7e9cd41e40 2 all 8 watchers are set, enabling cache 2023-10-05T19:28:28.872 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.866+0000 7f7e9cd41e40 1 -- 172.21.15.167:0/4093925617 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:100 6.4 6:22d26bf9:::data_loggenerations_metadata:head [call version.check_conds in=74b,call version.read in=11b,read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x5652a2f7f710 con 0x5652a2f40590 2023-10-05T19:28:28.873 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.866+0000 7f7e9b077700 1 -- 172.21.15.167:0/4093925617 <== osd.1 v2:172.21.15.167:6800/30035 33 ==== osd_op_reply(100 data_loggenerations_metadata [call,call out=48b,read 0~28 out=28b] v0'0 uv1 ondisk = 0) v8 ==== 256+0+76 (crc 0 0 0) 0x7f7e94052040 con 0x5652a2f40590 2023-10-05T19:28:28.873 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.866+0000 7f7e9cd41e40 1 -- 172.21.15.167:0/4093925617 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:101 6.4 6:22d26bf9:::data_loggenerations_metadata:head [watch watch cookie 94912921453040] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x5652a2f7ccb0 con 0x5652a2f40590 2023-10-05T19:28:28.874 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.870+0000 7f7e9b077700 1 -- 172.21.15.167:0/4093925617 <== osd.1 v2:172.21.15.167:6800/30035 34 ==== osd_op_reply(101 data_loggenerations_metadata [watch watch cookie 94912921453040] v30'78 uv1 ondisk = 0) v8 ==== 172+0+0 (crc 0 0 0) 0x7f7e94052040 con 0x5652a2f40590 2023-10-05T19:28:28.875 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.870+0000 7f7e6b7ee700 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2023-10-05T19:28:28.875 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.870+0000 7f7e9cd41e40 20 check_secure_mon_conn(): auth registy supported: methods=[2] modes=[2,1] 2023-10-05T19:28:28.875 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.870+0000 7f7e9cd41e40 20 check_secure_mon_conn(): mode 1 is insecure 2023-10-05T19:28:28.876 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.870+0000 7f7e9cd41e40 1 Processor -- start 2023-10-05T19:28:28.876 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.870+0000 7f7e9cd41e40 1 -- start start 2023-10-05T19:28:28.876 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.870+0000 7f7e9cd41e40 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5652a2f89f30 0x5652a2f8a300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-05T19:28:28.876 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.870+0000 7f7e9cd41e40 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x5652a2f8a840 con 0x5652a2f89f30 2023-10-05T19:28:28.876 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.870+0000 7f7e9a075700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5652a2f89f30 0x5652a2f8a300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-05T19:28:28.876 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.870+0000 7f7e9a075700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5652a2f89f30 0x5652a2f8a300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.118:3300/0 says I am v2:172.21.15.167:57804/0 (socket says 172.21.15.167:57804) 2023-10-05T19:28:28.876 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.870+0000 7f7e9a075700 1 -- 172.21.15.167:0/3980068692 learned_addr learned my addr 172.21.15.167:0/3980068692 (peer_addr_for_me v2:172.21.15.167:0/0) 2023-10-05T19:28:28.877 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.870+0000 7f7e9a075700 1 -- 172.21.15.167:0/3980068692 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5652a2f8a980 con 0x5652a2f89f30 2023-10-05T19:28:28.877 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.870+0000 7f7e9a075700 1 --2- 172.21.15.167:0/3980068692 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5652a2f89f30 0x5652a2f8a300 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 crypto rx=0x7f7e84002000 tx=0x7f7e84030040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cff8a011d7cae105 server_cookie=0 in_seq=0 out_seq=0 2023-10-05T19:28:28.877 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.870+0000 7f7e6afed700 1 -- 172.21.15.167:0/3980068692 <== mon.0 v2:172.21.15.118:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f7e84025040 con 0x5652a2f89f30 2023-10-05T19:28:28.877 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.870+0000 7f7e6afed700 1 -- 172.21.15.167:0/3980068692 <== mon.0 v2:172.21.15.118:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7e84034050 con 0x5652a2f89f30 2023-10-05T19:28:28.878 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.870+0000 7f7e6afed700 1 -- 172.21.15.167:0/3980068692 <== mon.0 v2:172.21.15.118:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f7e8402e4e0 con 0x5652a2f89f30 2023-10-05T19:28:28.878 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.874+0000 7f7e9cd41e40 1 -- 172.21.15.167:0/3980068692 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5652a2f89f30 msgr2=0x5652a2f8a300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-05T19:28:28.879 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.874+0000 7f7e9cd41e40 1 --2- 172.21.15.167:0/3980068692 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5652a2f89f30 0x5652a2f8a300 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 crypto rx=0x7f7e84002000 tx=0x7f7e84030040 comp rx=0 tx=0).stop 2023-10-05T19:28:28.879 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.874+0000 7f7e9cd41e40 1 -- 172.21.15.167:0/3980068692 shutdown_connections 2023-10-05T19:28:28.879 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.874+0000 7f7e9cd41e40 1 --2- 172.21.15.167:0/3980068692 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5652a2f89f30 0x5652a2f8a300 unknown :-1 s=CLOSED pgs=270 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-05T19:28:28.879 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.874+0000 7f7e9cd41e40 1 -- 172.21.15.167:0/3980068692 >> 172.21.15.167:0/3980068692 conn(0x5652a2f852f0 msgr2=0x5652a2f83440 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-05T19:28:28.879 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.874+0000 7f7e9cd41e40 1 -- 172.21.15.167:0/3980068692 shutdown_connections 2023-10-05T19:28:28.879 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.874+0000 7f7e9cd41e40 1 -- 172.21.15.167:0/3980068692 wait complete. 2023-10-05T19:28:28.881 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.874+0000 7f7e9cd41e40 1 Processor -- start 2023-10-05T19:28:28.881 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.874+0000 7f7e9cd41e40 1 -- start start 2023-10-05T19:28:28.881 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.874+0000 7f7e9cd41e40 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5652a2f89f30 0x5652a2f8fa20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-05T19:28:28.881 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.874+0000 7f7e9cd41e40 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x5652a2f90a20 con 0x5652a2f89f30 2023-10-05T19:28:28.881 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.874+0000 7f7e9a075700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5652a2f89f30 0x5652a2f8fa20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-05T19:28:28.881 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.874+0000 7f7e9a075700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5652a2f89f30 0x5652a2f8fa20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.118:3300/0 says I am v2:172.21.15.167:57812/0 (socket says 172.21.15.167:57812) 2023-10-05T19:28:28.881 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.874+0000 7f7e9a075700 1 -- 172.21.15.167:0/236670960 learned_addr learned my addr 172.21.15.167:0/236670960 (peer_addr_for_me v2:172.21.15.167:0/0) 2023-10-05T19:28:28.882 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.874+0000 7f7e9a075700 1 -- 172.21.15.167:0/236670960 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5652a2f3cbc0 con 0x5652a2f89f30 2023-10-05T19:28:28.882 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.874+0000 7f7e9a075700 1 --2- 172.21.15.167:0/236670960 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5652a2f89f30 0x5652a2f8fa20 secure :-1 s=READY pgs=271 cs=0 l=1 rev1=1 crypto rx=0x5652a2f83fb0 tx=0x7f7e84038000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-05T19:28:28.882 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.874+0000 7f7e697ea700 1 -- 172.21.15.167:0/236670960 <== mon.0 v2:172.21.15.118:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f7e84025040 con 0x5652a2f89f30 2023-10-05T19:28:28.882 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.874+0000 7f7e697ea700 1 -- 172.21.15.167:0/236670960 <== mon.0 v2:172.21.15.118:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7e84039850 con 0x5652a2f89f30 2023-10-05T19:28:28.882 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.874+0000 7f7e9cd41e40 1 -- 172.21.15.167:0/236670960 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x5652a2f90020 con 0x5652a2f89f30 2023-10-05T19:28:28.882 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.874+0000 7f7e697ea700 1 -- 172.21.15.167:0/236670960 <== mon.0 v2:172.21.15.118:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f7e84039a00 con 0x5652a2f89f30 2023-10-05T19:28:28.882 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.874+0000 7f7e9cd41e40 1 -- 172.21.15.167:0/236670960 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x5652a2f904c0 con 0x5652a2f89f30 2023-10-05T19:28:28.883 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.878+0000 7f7e697ea700 1 -- 172.21.15.167:0/236670960 <== mon.0 v2:172.21.15.118:3300/0 4 ==== mgrmap(e 5) v1 ==== 40329+0+0 (secure 0 0 0) 0x7f7e84039b60 con 0x5652a2f89f30 2023-10-05T19:28:28.883 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.878+0000 7f7e697ea700 1 --2- 172.21.15.167:0/236670960 >> [v2:172.21.15.167:6824/29889,v1:172.21.15.167:6825/29889] conn(0x7f7de0030c50 0x7f7de0033100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-05T19:28:28.883 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.878+0000 7f7e697ea700 1 -- 172.21.15.167:0/236670960 <== mon.0 v2:172.21.15.118:3300/0 5 ==== osd_map(30..30 src has 1..30) v4 ==== 6057+0+0 (secure 0 0 0) 0x7f7e84008070 con 0x5652a2f89f30 2023-10-05T19:28:28.883 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.878+0000 7f7e9b077700 1 --2- 172.21.15.167:0/236670960 >> [v2:172.21.15.167:6824/29889,v1:172.21.15.167:6825/29889] conn(0x7f7de0030c50 0x7f7de0033100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-05T19:28:28.883 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.878+0000 7f7e9cd41e40 5 note: GC not initialized 2023-10-05T19:28:28.884 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.878+0000 7f7e9b077700 1 --2- 172.21.15.167:0/236670960 >> [v2:172.21.15.167:6824/29889,v1:172.21.15.167:6825/29889] conn(0x7f7de0030c50 0x7f7de0033100 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x5652a2f7cc10 tx=0x7f7e9400aec0 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-05T19:28:28.885 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.878+0000 7f7dda7fc700 20 reqs_thread_entry: start 2023-10-05T19:28:28.885 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.878+0000 7f7e9cd41e40 20 init_complete bucket index max shards: 11 2023-10-05T19:28:28.886 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.878+0000 7f7e9cd41e40 10 rgw notify: Started notification manager with: 1 workers 2023-10-05T19:28:28.886 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.878+0000 7f7dca7fc700 20 rgw notify: INFO: next queues processing will happen at: Thu Oct 5 19:28:59 2023 2023-10-05T19:28:28.886 INFO:teuthology.orchestra.run.smithi167.stderr: 2023-10-05T19:28:28.887 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.882+0000 7f7dcbfff700 20 reqs_thread_entry: start 2023-10-05T19:28:28.887 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.882+0000 7f7dc9ffb700 20 reqs_thread_entry: start 2023-10-05T19:28:28.887 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.882+0000 7f7e9cd41e40 20 get_system_obj_state: rctx=0x7ffd4f87a748 obj=default.rgw.meta:users.uid:foo state=0x5652a2e130b0 s->prefetch_data=0 2023-10-05T19:28:28.887 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.882+0000 7f7e9cd41e40 10 cache get: name=default.rgw.meta+users.uid+foo : miss 2023-10-05T19:28:28.887 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.882+0000 7f7e9cd41e40 1 -- 172.21.15.167:0/4093925617 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:102 8.18 8:1e6ca5ed:users.uid::foo:head [call version.read in=11b,getxattrs,stat] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x5652a2ff82f0 con 0x5652a2f40590 2023-10-05T19:28:28.888 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.882+0000 7f7e9b077700 1 -- 172.21.15.167:0/4093925617 <== osd.1 v2:172.21.15.167:6800/30035 35 ==== osd_op_reply(102 foo [call out=48b,getxattrs out=75b,stat out=16b] v0'0 uv7 ondisk = 0) v8 ==== 231+0+139 (crc 0 0 0) 0x7f7e94052040 con 0x5652a2f40590 2023-10-05T19:28:28.889 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.882+0000 7f7e9cd41e40 10 cache put: name=default.rgw.meta+users.uid+foo info.flags=0x16 2023-10-05T19:28:28.889 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.882+0000 7f7e9cd41e40 10 adding default.rgw.meta+users.uid+foo to cache LRU end 2023-10-05T19:28:28.889 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.882+0000 7f7e9cd41e40 10 updating xattr: name=ceph.objclass.version bl.length()=42 2023-10-05T19:28:28.889 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.882+0000 7f7e9cd41e40 20 get_system_obj_state: s->obj_tag was set empty 2023-10-05T19:28:28.889 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.882+0000 7f7e9cd41e40 20 Read xattr: user.rgw.idtag 2023-10-05T19:28:28.889 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.882+0000 7f7e9cd41e40 10 cache get: name=default.rgw.meta+users.uid+foo : type miss (requested=0x13, cached=0x16) 2023-10-05T19:28:28.889 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.882+0000 7f7e9cd41e40 20 rados->read ofs=0 len=0 2023-10-05T19:28:28.889 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.882+0000 7f7e9cd41e40 1 -- 172.21.15.167:0/4093925617 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:103 8.18 8:1e6ca5ed:users.uid::foo:head [call version.check_conds in=122b,call version.read in=11b,read 0~0,getxattrs] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x5652a2ffad80 con 0x5652a2f40590 2023-10-05T19:28:28.889 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.882+0000 7f7e9b077700 1 -- 172.21.15.167:0/4093925617 <== osd.1 v2:172.21.15.167:6800/30035 36 ==== osd_op_reply(103 foo [call,call out=48b,read 0~513 out=513b,getxattrs out=75b] v0'0 uv7 ondisk = 0) v8 ==== 273+0+636 (crc 0 0 0) 0x7f7e94052040 con 0x5652a2f40590 2023-10-05T19:28:28.889 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.882+0000 7f7e9cd41e40 20 rados_obj.operate() r=0 bl.length=513 2023-10-05T19:28:28.890 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.882+0000 7f7e9cd41e40 10 cache put: name=default.rgw.meta+users.uid+foo info.flags=0x13 2023-10-05T19:28:28.890 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.882+0000 7f7e9cd41e40 10 moving default.rgw.meta+users.uid+foo to cache LRU end 2023-10-05T19:28:28.890 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.882+0000 7f7e9cd41e40 10 updating xattr: name=ceph.objclass.version bl.length()=42 2023-10-05T19:28:28.890 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.882+0000 7f7e9cd41e40 20 get_system_obj_state: rctx=0x7ffd4f879f98 obj=default.rgw.meta:users.swift:foo:foo2 state=0x5652a2e130b0 s->prefetch_data=0 2023-10-05T19:28:28.890 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.882+0000 7f7e9cd41e40 10 cache get: name=default.rgw.meta+users.swift+foo:foo2 : miss 2023-10-05T19:28:28.890 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.882+0000 7f7e9cd41e40 1 -- 172.21.15.167:0/4093925617 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:104 8.2 8:45671577:users.swift::foo%3afoo2:head [getxattrs,stat] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x5652a2ff7380 con 0x7f7e8c02d020 2023-10-05T19:28:28.890 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.882+0000 7f7e9a075700 1 -- 172.21.15.167:0/4093925617 <== osd.2 v2:172.21.15.167:6816/30037 17 ==== osd_op_reply(104 foo:foo2 [getxattrs,stat] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 ==== 194+0+0 (crc 0 0 0) 0x7f7e8403f020 con 0x7f7e8c02d020 2023-10-05T19:28:28.890 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.882+0000 7f7e9cd41e40 10 cache put: name=default.rgw.meta+users.swift+foo:foo2 info.flags=0x0 2023-10-05T19:28:28.890 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.882+0000 7f7e9cd41e40 10 adding default.rgw.meta+users.swift+foo:foo2 to cache LRU end 2023-10-05T19:28:28.890 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.882+0000 7f7e9cd41e40 20 get_system_obj_state: rctx=0x7ffd4f87a638 obj=default.rgw.meta:users.swift:foo:foo2 state=0x5652a2e130b0 s->prefetch_data=0 2023-10-05T19:28:28.891 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.882+0000 7f7e9cd41e40 10 cache get: name=default.rgw.meta+users.swift+foo:foo2 : hit (negative entry) 2023-10-05T19:28:28.891 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.882+0000 7f7e9cd41e40 1 -- 172.21.15.167:0/4093925617 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:105 8.18 8:1e6ca5ed:users.uid::foo:head [delete,create,call version.set in=58b,writefull 0~621 in=621b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x5652a2ffc8b0 con 0x5652a2f40590 2023-10-05T19:28:28.892 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.886+0000 7f7e9b077700 1 -- 172.21.15.167:0/4093925617 <== osd.1 v2:172.21.15.167:6800/30035 37 ==== osd_op_reply(105 foo [delete,create,call,writefull 0~621] v30'8 uv8 ondisk = 0) v8 ==== 273+0+0 (crc 0 0 0) 0x7f7e94052040 con 0x5652a2f40590 2023-10-05T19:28:28.892 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.886+0000 7f7e9cd41e40 10 cache put: name=default.rgw.meta+users.uid+foo info.flags=0x17 2023-10-05T19:28:28.892 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.886+0000 7f7e9cd41e40 10 moving default.rgw.meta+users.uid+foo to cache LRU end 2023-10-05T19:28:28.892 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.886+0000 7f7e9cd41e40 10 distributing notification oid=default.rgw.control:notify.4 cni=[op: 0, obj: default.rgw.meta:users.uid:foo, ofs0, ns] 2023-10-05T19:28:28.892 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.886+0000 7f7e9cd41e40 1 -- 172.21.15.167:0/4093925617 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:106 7.d 7:b4812045:::notify.4:head [notify cookie 94912921970832 in=809b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x5652a2ffc1e0 con 0x5652a2f40590 2023-10-05T19:28:28.892 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.886+0000 7f7e9b077700 1 -- 172.21.15.167:0/4093925617 <== osd.1 v2:172.21.15.167:6800/30035 38 ==== watch-notify(notify (1) cookie 94912921439008 notify 128849018887 ret 0) v3 ==== 839+0+0 (crc 0 0 0) 0x7f7e9404e380 con 0x5652a2f40590 2023-10-05T19:28:28.892 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.886+0000 7f7e98872700 10 rgw watcher librados: RGWWatcher::handle_notify() notify_id 128849018887 cookie 94912921439008 notifier 4359 bl.length()=797 2023-10-05T19:28:28.892 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.886+0000 7f7e98872700 10 rgw watcher librados: cache put: name=default.rgw.meta+users.uid+foo info.flags=0x17 2023-10-05T19:28:28.892 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.886+0000 7f7e98872700 10 rgw watcher librados: moving default.rgw.meta+users.uid+foo to cache LRU end 2023-10-05T19:28:28.893 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.886+0000 7f7e98872700 1 -- 172.21.15.167:0/4093925617 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:107 7.d 7:b4812045:::notify.4:head [notify-ack in=20b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x7f7e74003d90 con 0x5652a2f40590 2023-10-05T19:28:28.893 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.886+0000 7f7e9b077700 1 -- 172.21.15.167:0/4093925617 <== osd.1 v2:172.21.15.167:6800/30035 39 ==== osd_op_reply(106 notify.4 [notify cookie 94912921970832 out=8b] v0'0 uv66 ondisk = 0) v8 ==== 152+0+8 (crc 0 0 0) 0x7f7e94052040 con 0x5652a2f40590 2023-10-05T19:28:28.893 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.886+0000 7f7e9b077700 1 -- 172.21.15.167:0/4093925617 <== osd.1 v2:172.21.15.167:6800/30035 40 ==== osd_op_reply(107 notify.4 [notify-ack] v0'0 uv66 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f7e94052040 con 0x5652a2f40590 2023-10-05T19:28:28.893 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.886+0000 7f7e9b077700 1 -- 172.21.15.167:0/4093925617 <== osd.1 v2:172.21.15.167:6800/30035 41 ==== watch-notify(notify_complete (2) cookie 94912921970832 notify 128849018887 ret 0) v3 ==== 42+0+48 (crc 0 0 0) 0x7f7e940369d0 con 0x5652a2f40590 2023-10-05T19:28:28.893 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.886+0000 7f7e9cd41e40 1 -- 172.21.15.167:0/4093925617 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:108 8.2 8:45671577:users.swift::foo%3afoo2:head [delete,create,writefull 0~7 in=7b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x5652a2ff9d70 con 0x7f7e8c02d020 2023-10-05T19:28:28.896 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.890+0000 7f7e9a075700 1 -- 172.21.15.167:0/4093925617 <== osd.2 v2:172.21.15.167:6816/30037 18 ==== osd_op_reply(108 foo:foo2 [delete,create,writefull 0~7] v30'1 uv1 ondisk = 0) v8 ==== 236+0+0 (crc 0 0 0) 0x7f7e8403f020 con 0x7f7e8c02d020 2023-10-05T19:28:28.896 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.890+0000 7f7e9cd41e40 10 cache put: name=default.rgw.meta+users.swift+foo:foo2 info.flags=0x7 2023-10-05T19:28:28.896 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.890+0000 7f7e9cd41e40 10 moving default.rgw.meta+users.swift+foo:foo2 to cache LRU end 2023-10-05T19:28:28.896 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.890+0000 7f7e9cd41e40 10 distributing notification oid=default.rgw.control:notify.0 cni=[op: 0, obj: default.rgw.meta:users.swift:foo:foo2, ofs0, ns] 2023-10-05T19:28:28.896 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.890+0000 7f7e9cd41e40 1 -- 172.21.15.167:0/4093925617 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:109 7.1f 7:f95f44c2:::notify.0:head [notify cookie 94912921966720 in=178b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x5652a2ff9cd0 con 0x7f7e8c02d020 2023-10-05T19:28:28.897 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.890+0000 7f7e9a075700 1 -- 172.21.15.167:0/4093925617 <== osd.2 v2:172.21.15.167:6816/30037 19 ==== watch-notify(notify (1) cookie 94912921427584 notify 128849018883 ret 0) v3 ==== 208+0+0 (crc 0 0 0) 0x5652a2f904c0 con 0x7f7e8c02d020 2023-10-05T19:28:28.897 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.890+0000 7f7e93fff700 10 rgw watcher librados: RGWWatcher::handle_notify() notify_id 128849018883 cookie 94912921427584 notifier 4359 bl.length()=166 2023-10-05T19:28:28.897 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.890+0000 7f7e9a075700 1 -- 172.21.15.167:0/4093925617 <== osd.2 v2:172.21.15.167:6816/30037 20 ==== osd_op_reply(109 notify.0 [notify cookie 94912921966720 out=8b] v0'0 uv66 ondisk = 0) v8 ==== 152+0+8 (crc 0 0 0) 0x7f7e8403f020 con 0x7f7e8c02d020 2023-10-05T19:28:28.898 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.890+0000 7f7e93fff700 10 rgw watcher librados: cache put: name=default.rgw.meta+users.swift+foo:foo2 info.flags=0x7 2023-10-05T19:28:28.898 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.890+0000 7f7e93fff700 10 rgw watcher librados: moving default.rgw.meta+users.swift+foo:foo2 to cache LRU end 2023-10-05T19:28:28.898 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.890+0000 7f7e93fff700 1 -- 172.21.15.167:0/4093925617 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:110 7.1f 7:f95f44c2:::notify.0:head [notify-ack in=20b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x7f7de8003410 con 0x7f7e8c02d020 2023-10-05T19:28:28.898 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.890+0000 7f7e9a075700 1 -- 172.21.15.167:0/4093925617 <== osd.2 v2:172.21.15.167:6816/30037 21 ==== osd_op_reply(110 notify.0 [notify-ack] v0'0 uv66 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f7e8403f020 con 0x7f7e8c02d020 2023-10-05T19:28:28.898 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.890+0000 7f7e9a075700 1 -- 172.21.15.167:0/4093925617 <== osd.2 v2:172.21.15.167:6816/30037 22 ==== watch-notify(notify_complete (2) cookie 94912921966720 notify 128849018883 ret 0) v3 ==== 42+0+48 (crc 0 0 0) 0x5652a2f90020 con 0x7f7e8c02d020 2023-10-05T19:28:28.898 INFO:teuthology.orchestra.run.smithi167.stdout:{ 2023-10-05T19:28:28.898 INFO:teuthology.orchestra.run.smithi167.stdout: "user_id": "foo", 2023-10-05T19:28:28.898 INFO:teuthology.orchestra.run.smithi167.stdout: "display_name": "Foo", 2023-10-05T19:28:28.899 INFO:teuthology.orchestra.run.smithi167.stdout: "email": "foo@foo.com", 2023-10-05T19:28:28.899 INFO:teuthology.orchestra.run.smithi167.stdout: "suspended": 0, 2023-10-05T19:28:28.899 INFO:teuthology.orchestra.run.smithi167.stdout: "max_buckets": 4, 2023-10-05T19:28:28.899 INFO:teuthology.orchestra.run.smithi167.stdout: "subusers": [ 2023-10-05T19:28:28.899 INFO:teuthology.orchestra.run.smithi167.stdout: { 2023-10-05T19:28:28.899 INFO:teuthology.orchestra.run.smithi167.stdout: "id": "foo:foo1", 2023-10-05T19:28:28.899 INFO:teuthology.orchestra.run.smithi167.stdout: "permissions": "full-control" 2023-10-05T19:28:28.899 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2023-10-05T19:28:28.899 INFO:teuthology.orchestra.run.smithi167.stdout: { 2023-10-05T19:28:28.899 INFO:teuthology.orchestra.run.smithi167.stdout: "id": "foo:foo2", 2023-10-05T19:28:28.899 INFO:teuthology.orchestra.run.smithi167.stdout: "permissions": "" 2023-10-05T19:28:28.899 INFO:teuthology.orchestra.run.smithi167.stdout: } 2023-10-05T19:28:28.900 INFO:teuthology.orchestra.run.smithi167.stdout: ], 2023-10-05T19:28:28.900 INFO:teuthology.orchestra.run.smithi167.stdout: "keys": [ 2023-10-05T19:28:28.900 INFO:teuthology.orchestra.run.smithi167.stdout: { 2023-10-05T19:28:28.900 INFO:teuthology.orchestra.run.smithi167.stdout: "user": "foo", 2023-10-05T19:28:28.900 INFO:teuthology.orchestra.run.smithi167.stdout: "access_key": "9te6NH5mcdcq0Tc5i8i1", 2023-10-05T19:28:28.900 INFO:teuthology.orchestra.run.smithi167.stdout: "secret_key": "Ny4IOauQoL18Gp2zM7lC1vLmoawgqcYP/YGcWfXu" 2023-10-05T19:28:28.900 INFO:teuthology.orchestra.run.smithi167.stdout: } 2023-10-05T19:28:28.900 INFO:teuthology.orchestra.run.smithi167.stdout: ], 2023-10-05T19:28:28.900 INFO:teuthology.orchestra.run.smithi167.stdout: "swift_keys": [ 2023-10-05T19:28:28.900 INFO:teuthology.orchestra.run.smithi167.stdout: { 2023-10-05T19:28:28.900 INFO:teuthology.orchestra.run.smithi167.stdout: "user": "foo:foo1", 2023-10-05T19:28:28.900 INFO:teuthology.orchestra.run.smithi167.stdout: "secret_key": "gpS2G9RREMrnbqlp29PP2D36kgPR1tm72n5fPYfL" 2023-10-05T19:28:28.901 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2023-10-05T19:28:28.901 INFO:teuthology.orchestra.run.smithi167.stdout: { 2023-10-05T19:28:28.901 INFO:teuthology.orchestra.run.smithi167.stdout: "user": "foo:foo2", 2023-10-05T19:28:28.901 INFO:teuthology.orchestra.run.smithi167.stdout: "secret_key": "ri2VJQcKSYATOY6uaDUX7pxgkW+W1YmC6OCxPHwy" 2023-10-05T19:28:28.901 INFO:teuthology.orchestra.run.smithi167.stdout: } 2023-10-05T19:28:28.901 INFO:teuthology.orchestra.run.smithi167.stdout: ], 2023-10-05T19:28:28.901 INFO:teuthology.orchestra.run.smithi167.stdout: "caps": [], 2023-10-05T19:28:28.901 INFO:teuthology.orchestra.run.smithi167.stdout: "op_mask": "read, write, delete", 2023-10-05T19:28:28.901 INFO:teuthology.orchestra.run.smithi167.stdout: "default_placement": "", 2023-10-05T19:28:28.901 INFO:teuthology.orchestra.run.smithi167.stdout: "default_storage_class": "", 2023-10-05T19:28:28.901 INFO:teuthology.orchestra.run.smithi167.stdout: "placement_tags": [], 2023-10-05T19:28:28.901 INFO:teuthology.orchestra.run.smithi167.stdout: "bucket_quota": { 2023-10-05T19:28:28.901 INFO:teuthology.orchestra.run.smithi167.stdout: "enabled": false, 2023-10-05T19:28:28.902 INFO:teuthology.orchestra.run.smithi167.stdout: "check_on_raw": false, 2023-10-05T19:28:28.902 INFO:teuthology.orchestra.run.smithi167.stdout: "max_size": -1, 2023-10-05T19:28:28.902 INFO:teuthology.orchestra.run.smithi167.stdout: "max_size_kb": 0, 2023-10-05T19:28:28.902 INFO:teuthology.orchestra.run.smithi167.stdout: "max_objects": -1 2023-10-05T19:28:28.902 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2023-10-05T19:28:28.902 INFO:teuthology.orchestra.run.smithi167.stdout: "user_quota": { 2023-10-05T19:28:28.902 INFO:teuthology.orchestra.run.smithi167.stdout: "enabled": false, 2023-10-05T19:28:28.902 INFO:teuthology.orchestra.run.smithi167.stdout: "check_on_raw": false, 2023-10-05T19:28:28.902 INFO:teuthology.orchestra.run.smithi167.stdout: "max_size": -1, 2023-10-05T19:28:28.902 INFO:teuthology.orchestra.run.smithi167.stdout: "max_size_kb": 0, 2023-10-05T19:28:28.902 INFO:teuthology.orchestra.run.smithi167.stdout: "max_objects": -1 2023-10-05T19:28:28.902 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2023-10-05T19:28:28.903 INFO:teuthology.orchestra.run.smithi167.stdout: "temp_url_keys": [], 2023-10-05T19:28:28.903 INFO:teuthology.orchestra.run.smithi167.stdout: "type": "rgw", 2023-10-05T19:28:28.903 INFO:teuthology.orchestra.run.smithi167.stdout: "mfa_ids": [] 2023-10-05T19:28:28.903 INFO:teuthology.orchestra.run.smithi167.stdout:} 2023-10-05T19:28:28.903 INFO:teuthology.orchestra.run.smithi167.stdout: 2023-10-05T19:28:28.904 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.894+0000 7f7e9cd41e40 1 -- 172.21.15.167:0/4093925617 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:111 7.1f 7:f95f44c2:::notify.0:head [watch unwatch cookie 94912921427584] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x5652a2f51880 con 0x7f7e8c02d020 2023-10-05T19:28:28.904 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.894+0000 7f7e9a075700 1 -- 172.21.15.167:0/4093925617 <== osd.2 v2:172.21.15.167:6816/30037 23 ==== osd_op_reply(111 notify.0 [watch unwatch cookie 94912921427584] v30'68 uv66 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f7e8403f020 con 0x7f7e8c02d020 2023-10-05T19:28:28.904 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.894+0000 7f7e9cd41e40 20 remove_watcher() i=0 2023-10-05T19:28:28.904 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.894+0000 7f7e9cd41e40 2 removed watcher, disabling cache 2023-10-05T19:28:28.904 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.894+0000 7f7e9cd41e40 1 -- 172.21.15.167:0/4093925617 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:112 7.0 7:05bf5b68:::notify.1:head [watch unwatch cookie 94912921434048] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x5652a2f6fa90 con 0x5652a2f40590 2023-10-05T19:28:28.904 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.898+0000 7f7e9b077700 1 -- 172.21.15.167:0/4093925617 <== osd.1 v2:172.21.15.167:6800/30035 42 ==== osd_op_reply(112 notify.1 [watch unwatch cookie 94912921434048] v30'68 uv66 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f7e94052040 con 0x5652a2f40590 2023-10-05T19:28:28.904 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.898+0000 7f7e9cd41e40 20 remove_watcher() i=1 2023-10-05T19:28:28.904 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.898+0000 7f7e9cd41e40 1 -- 172.21.15.167:0/4093925617 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:113 7.15 7:a93a5511:::notify.2:head [watch unwatch cookie 94912921436656] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x5652a2f6fa90 con 0x5652a2f71470 2023-10-05T19:28:28.906 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.898+0000 7f7e9a876700 1 -- 172.21.15.167:0/4093925617 <== osd.3 v2:172.21.15.167:6808/30036 24 ==== osd_op_reply(113 notify.2 [watch unwatch cookie 94912921436656] v30'68 uv66 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f7e8c014070 con 0x5652a2f71470 2023-10-05T19:28:28.906 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.898+0000 7f7e9cd41e40 20 remove_watcher() i=2 2023-10-05T19:28:28.906 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.898+0000 7f7e9cd41e40 1 -- 172.21.15.167:0/4093925617 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:114 7.e 7:7759931f:::notify.3:head [watch unwatch cookie 94912921435152] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x5652a2f51880 con 0x7f7e8c02d020 2023-10-05T19:28:28.908 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.902+0000 7f7e9a075700 1 -- 172.21.15.167:0/4093925617 <== osd.2 v2:172.21.15.167:6816/30037 24 ==== osd_op_reply(114 notify.3 [watch unwatch cookie 94912921435152] v30'68 uv66 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f7e8403f020 con 0x7f7e8c02d020 2023-10-05T19:28:28.908 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.902+0000 7f7e9cd41e40 20 remove_watcher() i=3 2023-10-05T19:28:28.908 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.902+0000 7f7e9cd41e40 1 -- 172.21.15.167:0/4093925617 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:115 7.d 7:b4812045:::notify.4:head [watch unwatch cookie 94912921439008] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x5652a2f77cb0 con 0x5652a2f40590 2023-10-05T19:28:28.910 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.906+0000 7f7e9b077700 1 -- 172.21.15.167:0/4093925617 <== osd.1 v2:172.21.15.167:6800/30035 43 ==== osd_op_reply(115 notify.4 [watch unwatch cookie 94912921439008] v30'68 uv66 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f7e94052040 con 0x5652a2f40590 2023-10-05T19:28:28.910 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.906+0000 7f7e9cd41e40 20 remove_watcher() i=4 2023-10-05T19:28:28.911 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.906+0000 7f7e9cd41e40 1 -- 172.21.15.167:0/4093925617 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:116 7.3 7:c609908c:::notify.5:head [watch unwatch cookie 94912921447888] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x5652a2f77cb0 con 0x7f7e84007030 2023-10-05T19:28:28.913 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.906+0000 7f7e9b077700 1 -- 172.21.15.167:0/4093925617 <== osd.0 v2:172.21.15.118:6800/27690 29 ==== osd_op_reply(116 notify.5 [watch unwatch cookie 94912921447888] v30'68 uv66 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f7e94052040 con 0x7f7e84007030 2023-10-05T19:28:28.913 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.906+0000 7f7e9cd41e40 20 remove_watcher() i=5 2023-10-05T19:28:28.913 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.906+0000 7f7e9cd41e40 1 -- 172.21.15.167:0/4093925617 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:117 7.14 7:2b04a3e9:::notify.6:head [watch unwatch cookie 94912921448992] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x5652a2f77cb0 con 0x7f7e84007030 2023-10-05T19:28:28.917 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.910+0000 7f7e9b077700 1 -- 172.21.15.167:0/4093925617 <== osd.0 v2:172.21.15.118:6800/27690 30 ==== osd_op_reply(117 notify.6 [watch unwatch cookie 94912921448992] v30'68 uv66 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f7e94052040 con 0x7f7e84007030 2023-10-05T19:28:28.917 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.910+0000 7f7e9cd41e40 20 remove_watcher() i=6 2023-10-05T19:28:28.917 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.910+0000 7f7e9cd41e40 1 -- 172.21.15.167:0/4093925617 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:118 7.9 7:93e5b521:::notify.7:head [watch unwatch cookie 94912921451568] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x5652a2f77cb0 con 0x7f7e84007030 2023-10-05T19:28:28.919 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.914+0000 7f7e9b077700 1 -- 172.21.15.167:0/4093925617 <== osd.0 v2:172.21.15.118:6800/27690 31 ==== osd_op_reply(118 notify.7 [watch unwatch cookie 94912921451568] v30'68 uv66 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f7e94052040 con 0x7f7e84007030 2023-10-05T19:28:28.920 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.914+0000 7f7e9cd41e40 20 remove_watcher() i=7 2023-10-05T19:28:28.921 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.914+0000 7f7e6b7ee700 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2023-10-05T19:28:28.921 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.914+0000 7f7e9cd41e40 1 -- 172.21.15.167:0/4093925617 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:119 6.4 6:22d26bf9:::data_loggenerations_metadata:head [watch unwatch cookie 94912921453040] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x5652a2f77cb0 con 0x5652a2f40590 2023-10-05T19:28:28.925 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.918+0000 7f7e9b077700 1 -- 172.21.15.167:0/4093925617 <== osd.1 v2:172.21.15.167:6800/30035 44 ==== osd_op_reply(119 data_loggenerations_metadata [watch unwatch cookie 94912921453040] v30'79 uv1 ondisk = 0) v8 ==== 172+0+0 (crc 0 0 0) 0x7f7e94052040 con 0x5652a2f40590 2023-10-05T19:28:28.926 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.918+0000 7f7e9cd41e40 1 -- 172.21.15.167:0/4093925617 >> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] conn(0x7f7e84007030 msgr2=0x7f7e8401d430 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-05T19:28:28.926 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.918+0000 7f7e9cd41e40 1 --2- 172.21.15.167:0/4093925617 >> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] conn(0x7f7e84007030 0x7f7e8401d430 crc :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-05T19:28:28.926 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.918+0000 7f7e9cd41e40 1 -- 172.21.15.167:0/4093925617 >> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] conn(0x5652a2f40590 msgr2=0x5652a2f6d840 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-05T19:28:28.926 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.918+0000 7f7e9cd41e40 1 --2- 172.21.15.167:0/4093925617 >> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] conn(0x5652a2f40590 0x5652a2f6d840 crc :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-05T19:28:28.926 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.918+0000 7f7e9cd41e40 1 -- 172.21.15.167:0/4093925617 >> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] conn(0x7f7e8c02d020 msgr2=0x7f7e8c02c5d0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-05T19:28:28.926 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.918+0000 7f7e9cd41e40 1 --2- 172.21.15.167:0/4093925617 >> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] conn(0x7f7e8c02d020 0x7f7e8c02c5d0 crc :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-05T19:28:28.926 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.918+0000 7f7e9cd41e40 1 -- 172.21.15.167:0/4093925617 >> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] conn(0x5652a2f71470 msgr2=0x5652a2f73880 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-05T19:28:28.926 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.918+0000 7f7e9cd41e40 1 --2- 172.21.15.167:0/4093925617 >> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] conn(0x5652a2f71470 0x5652a2f73880 crc :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-05T19:28:28.926 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.918+0000 7f7e9cd41e40 1 -- 172.21.15.167:0/4093925617 >> [v2:172.21.15.167:6824/29889,v1:172.21.15.167:6825/29889] conn(0x7f7e80030c40 msgr2=0x7f7e800330f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-05T19:28:28.926 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.918+0000 7f7e9cd41e40 1 --2- 172.21.15.167:0/4093925617 >> [v2:172.21.15.167:6824/29889,v1:172.21.15.167:6825/29889] conn(0x7f7e80030c40 0x7f7e800330f0 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7f7e84005cc0 tx=0x7f7e84014000 comp rx=0 tx=0).stop 2023-10-05T19:28:28.927 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.918+0000 7f7e9b077700 1 -- 172.21.15.167:0/4093925617 reap_dead start 2023-10-05T19:28:28.927 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.918+0000 7f7e9cd41e40 1 -- 172.21.15.167:0/4093925617 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5652a2f3bfb0 msgr2=0x5652a2f61db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-05T19:28:28.927 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.918+0000 7f7e9cd41e40 1 --2- 172.21.15.167:0/4093925617 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5652a2f3bfb0 0x5652a2f61db0 secure :-1 s=READY pgs=269 cs=0 l=1 rev1=1 crypto rx=0x7f7e8c00b960 tx=0x7f7e8c014000 comp rx=0 tx=0).stop 2023-10-05T19:28:28.927 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.918+0000 7f7e9cd41e40 1 -- 172.21.15.167:0/4093925617 shutdown_connections 2023-10-05T19:28:28.927 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.918+0000 7f7e9cd41e40 1 --2- 172.21.15.167:0/4093925617 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5652a2f3bfb0 0x5652a2f61db0 unknown :-1 s=CLOSED pgs=269 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-05T19:28:28.927 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.918+0000 7f7e9cd41e40 1 -- 172.21.15.167:0/4093925617 >> 172.21.15.167:0/4093925617 conn(0x5652a2eaf800 msgr2=0x5652a2f555e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-05T19:28:28.927 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.918+0000 7f7e9cd41e40 1 -- 172.21.15.167:0/4093925617 shutdown_connections 2023-10-05T19:28:28.927 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.918+0000 7f7e9cd41e40 1 -- 172.21.15.167:0/4093925617 wait complete. 2023-10-05T19:28:28.928 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.922+0000 7f7e9cd41e40 1 -- 172.21.15.167:0/236670960 >> [v2:172.21.15.167:6824/29889,v1:172.21.15.167:6825/29889] conn(0x7f7de0030c50 msgr2=0x7f7de0033100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-05T19:28:28.928 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.922+0000 7f7e9cd41e40 1 --2- 172.21.15.167:0/236670960 >> [v2:172.21.15.167:6824/29889,v1:172.21.15.167:6825/29889] conn(0x7f7de0030c50 0x7f7de0033100 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x5652a2f7cc10 tx=0x7f7e9400aec0 comp rx=0 tx=0).stop 2023-10-05T19:28:28.928 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.922+0000 7f7e9cd41e40 1 -- 172.21.15.167:0/236670960 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5652a2f89f30 msgr2=0x5652a2f8fa20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-05T19:28:28.928 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.922+0000 7f7e9cd41e40 1 --2- 172.21.15.167:0/236670960 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5652a2f89f30 0x5652a2f8fa20 secure :-1 s=READY pgs=271 cs=0 l=1 rev1=1 crypto rx=0x5652a2f83fb0 tx=0x7f7e84038000 comp rx=0 tx=0).stop 2023-10-05T19:28:28.928 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.922+0000 7f7e9cd41e40 1 -- 172.21.15.167:0/236670960 shutdown_connections 2023-10-05T19:28:28.928 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.922+0000 7f7e9cd41e40 1 --2- 172.21.15.167:0/236670960 >> [v2:172.21.15.167:6824/29889,v1:172.21.15.167:6825/29889] conn(0x7f7de0030c50 0x7f7de0033100 unknown :-1 s=CLOSED pgs=97 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-05T19:28:28.929 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.922+0000 7f7e9cd41e40 1 --2- 172.21.15.167:0/236670960 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5652a2f89f30 0x5652a2f8fa20 unknown :-1 s=CLOSED pgs=271 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-05T19:28:28.929 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.922+0000 7f7e9cd41e40 1 -- 172.21.15.167:0/236670960 >> 172.21.15.167:0/236670960 conn(0x5652a2f852f0 msgr2=0x5652a2f83390 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-05T19:28:28.929 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.922+0000 7f7e9cd41e40 1 -- 172.21.15.167:0/236670960 shutdown_connections 2023-10-05T19:28:28.929 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.922+0000 7f7e9cd41e40 1 -- 172.21.15.167:0/236670960 wait complete. 2023-10-05T19:28:28.932 DEBUG:tasks.util.rgw: json result: {'user_id': 'foo', 'display_name': 'Foo', 'email': 'foo@foo.com', 'suspended': 0, 'max_buckets': 4, 'subusers': [{'id': 'foo:foo1', 'permissions': 'full-control'}, {'id': 'foo:foo2', 'permissions': ''}], 'keys': [{'user': 'foo', 'access_key': '9te6NH5mcdcq0Tc5i8i1', 'secret_key': 'Ny4IOauQoL18Gp2zM7lC1vLmoawgqcYP/YGcWfXu'}], 'swift_keys': [{'user': 'foo:foo1', 'secret_key': 'gpS2G9RREMrnbqlp29PP2D36kgPR1tm72n5fPYfL'}, {'user': 'foo:foo2', 'secret_key': 'ri2VJQcKSYATOY6uaDUX7pxgkW+W1YmC6OCxPHwy'}], 'caps': [], 'op_mask': 'read, write, delete', 'default_placement': '', 'default_storage_class': '', 'placement_tags': [], 'bucket_quota': {'enabled': False, 'check_on_raw': False, 'max_size': -1, 'max_size_kb': 0, 'max_objects': -1}, 'user_quota': {'enabled': False, 'check_on_raw': False, 'max_size': -1, 'max_size_kb': 0, 'max_objects': -1}, 'temp_url_keys': [], 'type': 'rgw', 'mfa_ids': []} 2023-10-05T19:28:28.932 INFO:tasks.util.rgw:rgwadmin: client.0 : ['user', 'info', '--uid', 'foo'] 2023-10-05T19:28:28.932 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'ceph', 'user', 'info', '--uid', 'foo'] 2023-10-05T19:28:28.932 DEBUG:teuthology.orchestra.run.smithi167:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster ceph user info --uid foo 2023-10-05T19:28:28.970 INFO:teuthology.orchestra.run.smithi167.stderr:ignoring --setuser ceph since I am not root 2023-10-05T19:28:28.971 INFO:teuthology.orchestra.run.smithi167.stderr:ignoring --setgroup ceph since I am not root 2023-10-05T19:28:28.971 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.966+0000 7f4ba20dce40 1 Processor -- start 2023-10-05T19:28:28.971 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.966+0000 7f4ba20dce40 1 -- start start 2023-10-05T19:28:28.972 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.966+0000 7f4ba20dce40 1 --2- >> v2:172.21.15.118:3300/0 conn(0x56015b0122f0 0x56015b0146d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-05T19:28:28.972 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.966+0000 7f4ba20dce40 1 -- --> v1:172.21.15.118:6789/0 -- auth(proto 0 26 bytes epoch 0) v1 -- 0x56015af5abd0 con 0x56015af823d0 2023-10-05T19:28:28.972 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.966+0000 7f4ba20dce40 1 -- --> v2:172.21.15.118:3300/0 -- mon_getmap magic: 0 v1 -- 0x56015af17ee0 con 0x56015b0122f0 2023-10-05T19:28:28.972 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.966+0000 7f4b9f410700 1 --2- >> v2:172.21.15.118:3300/0 conn(0x56015b0122f0 0x56015b0146d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-05T19:28:28.972 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.966+0000 7f4b9fc11700 1 --1- >> v1:172.21.15.118:6789/0 conn(0x56015af823d0 0x56015b010bd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.118:6789/0 says I am v1:172.21.15.167:56484/0 (socket says 172.21.15.167:56484) 2023-10-05T19:28:28.972 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.966+0000 7f4b9fc11700 1 -- 172.21.15.167:0/3116762842 learned_addr learned my addr 172.21.15.167:0/3116762842 (peer_addr_for_me v1:172.21.15.167:0/0) 2023-10-05T19:28:28.973 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.966+0000 7f4b9f410700 1 -- 172.21.15.167:0/3116762842 >> v1:172.21.15.118:6789/0 conn(0x56015af823d0 legacy=0x56015b010bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-05T19:28:28.973 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.966+0000 7f4b9ec0f700 1 -- 172.21.15.167:0/3116762842 <== mon.0 v1:172.21.15.118:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2203069464 0 0) 0x56015af5abd0 con 0x56015af823d0 2023-10-05T19:28:28.973 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.966+0000 7f4b9f410700 1 -- 172.21.15.167:0/3116762842 --> v2:172.21.15.118:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x56015b015c20 con 0x56015b0122f0 2023-10-05T19:28:28.973 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.966+0000 7f4b9f410700 1 --2- 172.21.15.167:0/3116762842 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x56015b0122f0 0x56015b0146d0 secure :-1 s=READY pgs=273 cs=0 l=1 rev1=1 crypto rx=0x7f4b8800a5d0 tx=0x7f4b88005a50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e22fc79254f32059 server_cookie=0 in_seq=0 out_seq=0 2023-10-05T19:28:28.974 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.966+0000 7f4b9ec0f700 1 -- 172.21.15.167:0/3116762842 <== mon.0 v2:172.21.15.118:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f4b880086a0 con 0x56015b0122f0 2023-10-05T19:28:28.974 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.966+0000 7f4b9ec0f700 1 -- 172.21.15.167:0/3116762842 <== mon.0 v2:172.21.15.118:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4b88008800 con 0x56015b0122f0 2023-10-05T19:28:28.975 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.966+0000 7f4b9ec0f700 1 -- 172.21.15.167:0/3116762842 <== mon.0 v2:172.21.15.118:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f4b88008ad0 con 0x56015b0122f0 2023-10-05T19:28:28.975 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.970+0000 7f4ba20dce40 1 -- 172.21.15.167:0/3116762842 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x56015b0122f0 msgr2=0x56015b0146d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-05T19:28:28.975 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.970+0000 7f4ba20dce40 1 --2- 172.21.15.167:0/3116762842 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x56015b0122f0 0x56015b0146d0 secure :-1 s=READY pgs=273 cs=0 l=1 rev1=1 crypto rx=0x7f4b8800a5d0 tx=0x7f4b88005a50 comp rx=0 tx=0).stop 2023-10-05T19:28:28.975 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.970+0000 7f4ba20dce40 1 -- 172.21.15.167:0/3116762842 shutdown_connections 2023-10-05T19:28:28.975 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.970+0000 7f4ba20dce40 1 --2- 172.21.15.167:0/3116762842 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x56015b0122f0 0x56015b0146d0 unknown :-1 s=CLOSED pgs=273 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-05T19:28:28.975 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.970+0000 7f4ba20dce40 1 -- 172.21.15.167:0/3116762842 >> 172.21.15.167:0/3116762842 conn(0x56015b008720 msgr2=0x56015b00ab30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-05T19:28:28.975 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.970+0000 7f4ba20dce40 1 -- 172.21.15.167:0/3116762842 shutdown_connections 2023-10-05T19:28:28.975 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.970+0000 7f4ba20dce40 1 -- 172.21.15.167:0/3116762842 wait complete. 2023-10-05T19:28:28.976 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.970+0000 7f4ba20dce40 1 Processor -- start 2023-10-05T19:28:28.976 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.970+0000 7f4ba20dce40 1 -- start start 2023-10-05T19:28:28.976 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.970+0000 7f4ba20dce40 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x56015af823d0 0x56015b00e150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-05T19:28:28.976 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.970+0000 7f4ba20dce40 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x56015aedbe30 con 0x56015af823d0 2023-10-05T19:28:28.977 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.970+0000 7f4b9fc11700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x56015af823d0 0x56015b00e150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-05T19:28:28.977 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.970+0000 7f4b9fc11700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x56015af823d0 0x56015b00e150 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.118:3300/0 says I am v2:172.21.15.167:57834/0 (socket says 172.21.15.167:57834) 2023-10-05T19:28:28.977 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.970+0000 7f4b9fc11700 1 -- 172.21.15.167:0/885828751 learned_addr learned my addr 172.21.15.167:0/885828751 (peer_addr_for_me v2:172.21.15.167:0/0) 2023-10-05T19:28:28.977 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.970+0000 7f4b9fc11700 1 -- 172.21.15.167:0/885828751 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4b88005cf0 con 0x56015af823d0 2023-10-05T19:28:28.978 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.970+0000 7f4b9fc11700 1 --2- 172.21.15.167:0/885828751 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x56015af823d0 0x56015b00e150 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 crypto rx=0x7f4b90001630 tx=0x7f4b900069b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8b2ea3e4995ba7dc server_cookie=0 in_seq=0 out_seq=0 2023-10-05T19:28:28.978 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.970+0000 7f4b9dc0d700 1 -- 172.21.15.167:0/885828751 <== mon.0 v2:172.21.15.118:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f4b90006320 con 0x56015af823d0 2023-10-05T19:28:28.978 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.970+0000 7f4b9dc0d700 1 -- 172.21.15.167:0/885828751 <== mon.0 v2:172.21.15.118:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4b900044c0 con 0x56015af823d0 2023-10-05T19:28:28.978 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.970+0000 7f4b9dc0d700 1 -- 172.21.15.167:0/885828751 <== mon.0 v2:172.21.15.118:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f4b90004650 con 0x56015af823d0 2023-10-05T19:28:28.978 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.970+0000 7f4ba20dce40 1 -- 172.21.15.167:0/885828751 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x56015af823d0 msgr2=0x56015b00e150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-05T19:28:28.978 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.970+0000 7f4ba20dce40 1 --2- 172.21.15.167:0/885828751 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x56015af823d0 0x56015b00e150 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 crypto rx=0x7f4b90001630 tx=0x7f4b900069b0 comp rx=0 tx=0).stop 2023-10-05T19:28:28.978 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.974+0000 7f4ba20dce40 1 -- 172.21.15.167:0/885828751 shutdown_connections 2023-10-05T19:28:28.979 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.974+0000 7f4ba20dce40 1 --2- 172.21.15.167:0/885828751 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x56015af823d0 0x56015b00e150 unknown :-1 s=CLOSED pgs=274 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-05T19:28:28.979 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.974+0000 7f4ba20dce40 1 -- 172.21.15.167:0/885828751 >> 172.21.15.167:0/885828751 conn(0x56015b008720 msgr2=0x56015b028a30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-05T19:28:28.979 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.974+0000 7f4ba20dce40 1 -- 172.21.15.167:0/885828751 shutdown_connections 2023-10-05T19:28:28.979 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.974+0000 7f4ba20dce40 1 -- 172.21.15.167:0/885828751 wait complete. 2023-10-05T19:28:28.980 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.974+0000 7f4ba20dce40 1 Processor -- start 2023-10-05T19:28:28.980 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.974+0000 7f4ba20dce40 1 -- start start 2023-10-05T19:28:28.980 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.974+0000 7f4ba20dce40 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x56015b035f90 0x56015b036360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-05T19:28:28.980 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.974+0000 7f4ba20dce40 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x56015aeaeed0 con 0x56015b035f90 2023-10-05T19:28:28.980 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.974+0000 7f4b9fc11700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x56015b035f90 0x56015b036360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-05T19:28:28.980 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.974+0000 7f4b9fc11700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x56015b035f90 0x56015b036360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.118:3300/0 says I am v2:172.21.15.167:57838/0 (socket says 172.21.15.167:57838) 2023-10-05T19:28:28.980 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.974+0000 7f4b9fc11700 1 -- 172.21.15.167:0/4265138048 learned_addr learned my addr 172.21.15.167:0/4265138048 (peer_addr_for_me v2:172.21.15.167:0/0) 2023-10-05T19:28:28.980 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.974+0000 7f4b9fc11700 1 -- 172.21.15.167:0/4265138048 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4b90006c90 con 0x56015b035f90 2023-10-05T19:28:28.981 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.974+0000 7f4b9fc11700 1 --2- 172.21.15.167:0/4265138048 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x56015b035f90 0x56015b036360 secure :-1 s=READY pgs=275 cs=0 l=1 rev1=1 crypto rx=0x7f4b90001560 tx=0x7f4b90014000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-05T19:28:28.981 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.974+0000 7f4b97fff700 1 -- 172.21.15.167:0/4265138048 <== mon.0 v2:172.21.15.118:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f4b90013690 con 0x56015b035f90 2023-10-05T19:28:28.981 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.974+0000 7f4b97fff700 1 -- 172.21.15.167:0/4265138048 <== mon.0 v2:172.21.15.118:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4b900137f0 con 0x56015b035f90 2023-10-05T19:28:28.982 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.974+0000 7f4b97fff700 1 -- 172.21.15.167:0/4265138048 <== mon.0 v2:172.21.15.118:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f4b90013ac0 con 0x56015b035f90 2023-10-05T19:28:28.982 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.974+0000 7f4ba20dce40 1 -- 172.21.15.167:0/4265138048 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x56015b0368a0 con 0x56015b035f90 2023-10-05T19:28:28.982 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.974+0000 7f4ba20dce40 1 -- 172.21.15.167:0/4265138048 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x56015b0394b0 con 0x56015b035f90 2023-10-05T19:28:28.982 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.978+0000 7f4b97fff700 1 -- 172.21.15.167:0/4265138048 <== mon.0 v2:172.21.15.118:3300/0 4 ==== mgrmap(e 5) v1 ==== 40329+0+0 (secure 0 0 0) 0x7f4b9001a070 con 0x56015b035f90 2023-10-05T19:28:28.983 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.978+0000 7f4b97fff700 1 --2- 172.21.15.167:0/4265138048 >> [v2:172.21.15.167:6824/29889,v1:172.21.15.167:6825/29889] conn(0x7f4b80030c00 0x7f4b800330b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-05T19:28:28.984 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.978+0000 7f4b97fff700 1 -- 172.21.15.167:0/4265138048 <== mon.0 v2:172.21.15.118:3300/0 5 ==== osd_map(30..30 src has 1..30) v4 ==== 6057+0+0 (secure 0 0 0) 0x7f4b90044fa0 con 0x56015b035f90 2023-10-05T19:28:28.984 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.978+0000 7f4b9f410700 1 --2- 172.21.15.167:0/4265138048 >> [v2:172.21.15.167:6824/29889,v1:172.21.15.167:6825/29889] conn(0x7f4b80030c00 0x7f4b800330b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-05T19:28:28.984 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.978+0000 7f4ba20dce40 20 rados->read ofs=0 len=0 2023-10-05T19:28:28.984 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.978+0000 7f4ba20dce40 1 --2- 172.21.15.167:0/4265138048 >> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] conn(0x56015b03fd60 0x56015b042140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-05T19:28:28.984 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.978+0000 7f4ba20dce40 1 -- 172.21.15.167:0/4265138048 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:1 5.12 5:49953fa1:::default.realm:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x56015b042870 con 0x56015b03fd60 2023-10-05T19:28:28.985 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.978+0000 7f4ba0412700 1 --2- 172.21.15.167:0/4265138048 >> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] conn(0x56015b03fd60 0x56015b042140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-05T19:28:28.985 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.978+0000 7f4b9f410700 1 --2- 172.21.15.167:0/4265138048 >> [v2:172.21.15.167:6824/29889,v1:172.21.15.167:6825/29889] conn(0x7f4b80030c00 0x7f4b800330b0 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x56015af81c30 tx=0x7f4b88015000 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-05T19:28:28.985 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.978+0000 7f4ba0412700 1 --2- 172.21.15.167:0/4265138048 >> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] conn(0x56015b03fd60 0x56015b042140 crc :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-05T19:28:28.986 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.978+0000 7f4ba0412700 1 -- 172.21.15.167:0/4265138048 <== osd.1 v2:172.21.15.167:6800/30035 1 ==== osd_op_reply(1 default.realm [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 ==== 157+0+0 (crc 0 0 0) 0x7f4b9800a500 con 0x56015b03fd60 2023-10-05T19:28:28.986 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.978+0000 7f4ba20dce40 20 rados_obj.operate() r=-2 bl.length=0 2023-10-05T19:28:28.986 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.978+0000 7f4ba20dce40 20 realm 2023-10-05T19:28:28.986 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.978+0000 7f4ba20dce40 20 rados->read ofs=0 len=0 2023-10-05T19:28:28.986 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.978+0000 7f4ba20dce40 1 -- 172.21.15.167:0/4265138048 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:2 5.12 5:49953fa1:::default.realm:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x56015b03f390 con 0x56015b03fd60 2023-10-05T19:28:28.987 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.982+0000 7f4ba0412700 1 -- 172.21.15.167:0/4265138048 <== osd.1 v2:172.21.15.167:6800/30035 2 ==== osd_op_reply(2 default.realm [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 ==== 157+0+0 (crc 0 0 0) 0x7f4b9800a500 con 0x56015b03fd60 2023-10-05T19:28:28.987 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.982+0000 7f4ba20dce40 20 rados_obj.operate() r=-2 bl.length=0 2023-10-05T19:28:28.987 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.982+0000 7f4ba20dce40 4 RGWPeriod::init failed to init realm id : (2) No such file or directory 2023-10-05T19:28:28.987 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.982+0000 7f4ba20dce40 20 rados->read ofs=0 len=0 2023-10-05T19:28:28.987 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.982+0000 7f4ba20dce40 1 -- 172.21.15.167:0/4265138048 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:3 5.12 5:49953fa1:::default.realm:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x56015b043350 con 0x56015b03fd60 2023-10-05T19:28:28.987 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.982+0000 7f4ba0412700 1 -- 172.21.15.167:0/4265138048 <== osd.1 v2:172.21.15.167:6800/30035 3 ==== osd_op_reply(3 default.realm [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 ==== 157+0+0 (crc 0 0 0) 0x7f4b9800a500 con 0x56015b03fd60 2023-10-05T19:28:28.987 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.982+0000 7f4ba20dce40 20 rados_obj.operate() r=-2 bl.length=0 2023-10-05T19:28:28.988 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.982+0000 7f4ba20dce40 20 rados->read ofs=0 len=0 2023-10-05T19:28:28.988 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.982+0000 7f4ba20dce40 1 -- 172.21.15.167:0/4265138048 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:4 5.f 5:f43b6ece:::zone_names.default:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x56015b042ef0 con 0x56015b03fd60 2023-10-05T19:28:28.989 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.982+0000 7f4ba0412700 1 -- 172.21.15.167:0/4265138048 <== osd.1 v2:172.21.15.167:6800/30035 4 ==== osd_op_reply(4 zone_names.default [read 0~46 out=46b] v0'0 uv1 ondisk = 0) v8 ==== 162+0+46 (crc 0 0 0) 0x7f4b9800a500 con 0x56015b03fd60 2023-10-05T19:28:28.989 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.982+0000 7f4ba20dce40 20 rados_obj.operate() r=0 bl.length=46 2023-10-05T19:28:28.989 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.982+0000 7f4ba20dce40 20 rados->read ofs=0 len=0 2023-10-05T19:28:28.989 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.982+0000 7f4ba20dce40 1 -- 172.21.15.167:0/4265138048 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:5 5.1d 5:b8374eb4:::zone_info.9a9df965-6612-45d4-a45f-6e3c889bacc2:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x56015b043db0 con 0x56015b03fd60 2023-10-05T19:28:28.989 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.982+0000 7f4ba0412700 1 -- 172.21.15.167:0/4265138048 <== osd.1 v2:172.21.15.167:6800/30035 5 ==== osd_op_reply(5 zone_info.9a9df965-6612-45d4-a45f-6e3c889bacc2 [read 0~874 out=874b] v0'0 uv2 ondisk = 0) v8 ==== 190+0+874 (crc 0 0 0) 0x7f4b9800a500 con 0x56015b03fd60 2023-10-05T19:28:28.989 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.982+0000 7f4ba20dce40 20 rados_obj.operate() r=0 bl.length=874 2023-10-05T19:28:28.990 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.982+0000 7f4ba20dce40 20 searching for the correct realm 2023-10-05T19:28:28.990 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.982+0000 7f4ba20dce40 1 --2- 172.21.15.167:0/4265138048 >> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] conn(0x56015b045c00 0x56015b047fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-05T19:28:28.990 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.982+0000 7f4ba20dce40 1 -- 172.21.15.167:0/4265138048 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:6 5.0 5:00000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x56015b048520 con 0x56015b045c00 2023-10-05T19:28:28.990 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.982+0000 7f4b9fc11700 1 --2- 172.21.15.167:0/4265138048 >> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] conn(0x56015b045c00 0x56015b047fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-05T19:28:28.990 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.982+0000 7f4b9fc11700 1 --2- 172.21.15.167:0/4265138048 >> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] conn(0x56015b045c00 0x56015b047fe0 crc :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-05T19:28:28.991 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.986+0000 7f4b9fc11700 1 -- 172.21.15.167:0/4265138048 <== osd.3 v2:172.21.15.167:6808/30036 1 ==== osd_op_reply(6 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f4b90025020 con 0x56015b045c00 2023-10-05T19:28:28.991 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.986+0000 7f4b9fc11700 1 --2- 172.21.15.167:0/4265138048 >> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] conn(0x56015af823d0 0x7f4b9001e590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-05T19:28:28.991 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.986+0000 7f4b9fc11700 1 -- 172.21.15.167:0/4265138048 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:7 5.10 5:08000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x56015b048520 con 0x56015af823d0 2023-10-05T19:28:28.991 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.986+0000 7f4b9f410700 1 --2- 172.21.15.167:0/4265138048 >> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] conn(0x56015af823d0 0x7f4b9001e590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-05T19:28:28.991 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.986+0000 7f4b9f410700 1 --2- 172.21.15.167:0/4265138048 >> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] conn(0x56015af823d0 0x7f4b9001e590 crc :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-05T19:28:28.992 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.986+0000 7f4b9f410700 1 -- 172.21.15.167:0/4265138048 <== osd.2 v2:172.21.15.167:6816/30037 1 ==== osd_op_reply(7 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f4b88008d50 con 0x56015af823d0 2023-10-05T19:28:28.992 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.986+0000 7f4b9f410700 1 --2- 172.21.15.167:0/4265138048 >> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] conn(0x56015b0122f0 0x7f4b8801c410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-05T19:28:28.992 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.986+0000 7f4b9f410700 1 -- 172.21.15.167:0/4265138048 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:8 5.8 5:10000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x56015b048520 con 0x56015b0122f0 2023-10-05T19:28:28.993 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.986+0000 7f4ba0412700 1 --2- 172.21.15.167:0/4265138048 >> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] conn(0x56015b0122f0 0x7f4b8801c410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-05T19:28:28.993 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.986+0000 7f4ba0412700 1 --2- 172.21.15.167:0/4265138048 >> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] conn(0x56015b0122f0 0x7f4b8801c410 crc :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-05T19:28:28.994 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.990+0000 7f4ba0412700 1 -- 172.21.15.167:0/4265138048 <== osd.0 v2:172.21.15.118:6800/27690 1 ==== osd_op_reply(8 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f4b98014040 con 0x56015b0122f0 2023-10-05T19:28:28.994 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.990+0000 7f4ba0412700 1 -- 172.21.15.167:0/4265138048 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:9 5.18 5:18000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x56015b048520 con 0x56015b03fd60 2023-10-05T19:28:28.995 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.990+0000 7f4ba0412700 1 -- 172.21.15.167:0/4265138048 <== osd.1 v2:172.21.15.167:6800/30035 6 ==== osd_op_reply(9 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f4b98014040 con 0x56015b03fd60 2023-10-05T19:28:28.995 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.990+0000 7f4ba0412700 1 -- 172.21.15.167:0/4265138048 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:10 5.4 5:20000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x56015b048520 con 0x56015b03fd60 2023-10-05T19:28:28.995 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.990+0000 7f4ba0412700 1 -- 172.21.15.167:0/4265138048 <== osd.1 v2:172.21.15.167:6800/30035 7 ==== osd_op_reply(10 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f4b98014040 con 0x56015b03fd60 2023-10-05T19:28:28.996 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.990+0000 7f4ba0412700 1 -- 172.21.15.167:0/4265138048 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:11 5.14 5:28000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x56015b048520 con 0x56015b0122f0 2023-10-05T19:28:28.996 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.990+0000 7f4ba0412700 1 -- 172.21.15.167:0/4265138048 <== osd.0 v2:172.21.15.118:6800/27690 2 ==== osd_op_reply(11 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f4b98014040 con 0x56015b0122f0 2023-10-05T19:28:28.996 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.990+0000 7f4ba0412700 1 -- 172.21.15.167:0/4265138048 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:12 5.c 5:30000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x56015b048520 con 0x56015b0122f0 2023-10-05T19:28:28.997 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.990+0000 7f4ba0412700 1 -- 172.21.15.167:0/4265138048 <== osd.0 v2:172.21.15.118:6800/27690 3 ==== osd_op_reply(12 [pgnls start_epoch 30 out=112b] v23'1 uv1 ondisk = 1) v8 ==== 144+0+112 (crc 0 0 0) 0x7f4b98014040 con 0x56015b0122f0 2023-10-05T19:28:28.998 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.990+0000 7f4ba0412700 1 -- 172.21.15.167:0/4265138048 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:13 5.1c 5:38000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x56015b048520 con 0x56015af823d0 2023-10-05T19:28:28.999 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.994+0000 7f4b9f410700 1 -- 172.21.15.167:0/4265138048 <== osd.2 v2:172.21.15.167:6816/30037 2 ==== osd_op_reply(13 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f4b88008d50 con 0x56015af823d0 2023-10-05T19:28:28.999 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.994+0000 7f4b9f410700 1 -- 172.21.15.167:0/4265138048 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:14 5.2 5:40000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x56015b048520 con 0x56015b045c00 2023-10-05T19:28:28.999 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.994+0000 7f4b9fc11700 1 -- 172.21.15.167:0/4265138048 <== osd.3 v2:172.21.15.167:6808/30036 2 ==== osd_op_reply(14 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f4b90025020 con 0x56015b045c00 2023-10-05T19:28:28.999 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.994+0000 7f4b9fc11700 1 -- 172.21.15.167:0/4265138048 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:15 5.12 5:48000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x56015b048520 con 0x56015b03fd60 2023-10-05T19:28:29.000 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.994+0000 7f4ba0412700 1 -- 172.21.15.167:0/4265138048 <== osd.1 v2:172.21.15.167:6800/30035 8 ==== osd_op_reply(15 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f4b98014040 con 0x56015b03fd60 2023-10-05T19:28:29.000 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.994+0000 7f4ba0412700 1 -- 172.21.15.167:0/4265138048 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:16 5.a 5:50000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x56015b048520 con 0x56015b0122f0 2023-10-05T19:28:29.000 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.994+0000 7f4ba0412700 1 -- 172.21.15.167:0/4265138048 <== osd.0 v2:172.21.15.118:6800/27690 4 ==== osd_op_reply(16 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f4b98022040 con 0x56015b0122f0 2023-10-05T19:28:29.001 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.994+0000 7f4ba0412700 1 -- 172.21.15.167:0/4265138048 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:17 5.1a 5:58000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x56015b048520 con 0x56015b03fd60 2023-10-05T19:28:29.001 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.994+0000 7f4ba0412700 1 -- 172.21.15.167:0/4265138048 <== osd.1 v2:172.21.15.167:6800/30035 9 ==== osd_op_reply(17 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f4b98022040 con 0x56015b03fd60 2023-10-05T19:28:29.001 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.994+0000 7f4ba0412700 1 -- 172.21.15.167:0/4265138048 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:18 5.6 5:60000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x56015b048520 con 0x56015af823d0 2023-10-05T19:28:29.002 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.998+0000 7f4b9f410700 1 -- 172.21.15.167:0/4265138048 <== osd.2 v2:172.21.15.167:6816/30037 3 ==== osd_op_reply(18 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f4b88008d50 con 0x56015af823d0 2023-10-05T19:28:29.002 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.998+0000 7f4b9f410700 1 -- 172.21.15.167:0/4265138048 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:19 5.16 5:68000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x56015b048520 con 0x56015b045c00 2023-10-05T19:28:29.003 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.998+0000 7f4b9fc11700 1 -- 172.21.15.167:0/4265138048 <== osd.3 v2:172.21.15.167:6808/30036 3 ==== osd_op_reply(19 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f4b90025020 con 0x56015b045c00 2023-10-05T19:28:29.003 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.998+0000 7f4b9fc11700 1 -- 172.21.15.167:0/4265138048 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:20 5.e 5:70000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x56015b048520 con 0x56015b045c00 2023-10-05T19:28:29.003 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.998+0000 7f4b9fc11700 1 -- 172.21.15.167:0/4265138048 <== osd.3 v2:172.21.15.167:6808/30036 4 ==== osd_op_reply(20 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f4b90025020 con 0x56015b045c00 2023-10-05T19:28:29.003 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.998+0000 7f4b9fc11700 1 -- 172.21.15.167:0/4265138048 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:21 5.1e 5:78000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x56015b048520 con 0x56015b0122f0 2023-10-05T19:28:29.004 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.998+0000 7f4ba0412700 1 -- 172.21.15.167:0/4265138048 <== osd.0 v2:172.21.15.118:6800/27690 5 ==== osd_op_reply(21 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f4b98022040 con 0x56015b0122f0 2023-10-05T19:28:29.004 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.998+0000 7f4ba0412700 1 -- 172.21.15.167:0/4265138048 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:22 5.1 5:80000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x56015b048520 con 0x56015b045c00 2023-10-05T19:28:29.005 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.998+0000 7f4b9fc11700 1 -- 172.21.15.167:0/4265138048 <== osd.3 v2:172.21.15.167:6808/30036 5 ==== osd_op_reply(22 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f4b90025020 con 0x56015b045c00 2023-10-05T19:28:29.006 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.998+0000 7f4b9fc11700 1 -- 172.21.15.167:0/4265138048 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:23 5.11 5:88000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x56015b048520 con 0x56015b045c00 2023-10-05T19:28:29.006 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.998+0000 7f4b9fc11700 1 -- 172.21.15.167:0/4265138048 <== osd.3 v2:172.21.15.167:6808/30036 6 ==== osd_op_reply(23 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f4b90025020 con 0x56015b045c00 2023-10-05T19:28:29.006 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:28.998+0000 7f4b9fc11700 1 -- 172.21.15.167:0/4265138048 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:24 5.9 5:90000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x56015b048520 con 0x56015b03fd60 2023-10-05T19:28:29.006 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.002+0000 7f4ba0412700 1 -- 172.21.15.167:0/4265138048 <== osd.1 v2:172.21.15.167:6800/30035 10 ==== osd_op_reply(24 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f4b98022040 con 0x56015b03fd60 2023-10-05T19:28:29.006 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.002+0000 7f4ba0412700 1 -- 172.21.15.167:0/4265138048 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:25 5.19 5:98000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x56015b048520 con 0x56015b03fd60 2023-10-05T19:28:29.007 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.002+0000 7f4ba0412700 1 -- 172.21.15.167:0/4265138048 <== osd.1 v2:172.21.15.167:6800/30035 11 ==== osd_op_reply(25 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f4b98022040 con 0x56015b03fd60 2023-10-05T19:28:29.007 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.002+0000 7f4ba0412700 1 -- 172.21.15.167:0/4265138048 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:26 5.5 5:a0000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x56015b048520 con 0x56015b0122f0 2023-10-05T19:28:29.008 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.002+0000 7f4ba0412700 1 -- 172.21.15.167:0/4265138048 <== osd.0 v2:172.21.15.118:6800/27690 6 ==== osd_op_reply(26 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f4b98022040 con 0x56015b0122f0 2023-10-05T19:28:29.008 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.002+0000 7f4ba0412700 1 -- 172.21.15.167:0/4265138048 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:27 5.15 5:a8000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x56015b048520 con 0x56015b045c00 2023-10-05T19:28:29.009 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.002+0000 7f4b9fc11700 1 -- 172.21.15.167:0/4265138048 <== osd.3 v2:172.21.15.167:6808/30036 7 ==== osd_op_reply(27 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f4b90025020 con 0x56015b045c00 2023-10-05T19:28:29.009 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.002+0000 7f4b9fc11700 1 -- 172.21.15.167:0/4265138048 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:28 5.d 5:b0000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x56015b048520 con 0x56015af823d0 2023-10-05T19:28:29.010 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.002+0000 7f4b9f410700 1 -- 172.21.15.167:0/4265138048 <== osd.2 v2:172.21.15.167:6816/30037 4 ==== osd_op_reply(28 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f4b88008d50 con 0x56015af823d0 2023-10-05T19:28:29.010 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.002+0000 7f4b9f410700 1 -- 172.21.15.167:0/4265138048 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:29 5.1d 5:b8000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x56015b048520 con 0x56015b03fd60 2023-10-05T19:28:29.010 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.006+0000 7f4ba0412700 1 -- 172.21.15.167:0/4265138048 <== osd.1 v2:172.21.15.167:6800/30035 12 ==== osd_op_reply(29 [pgnls start_epoch 30 out=107b] v30'2 uv2 ondisk = 1) v8 ==== 144+0+107 (crc 0 0 0) 0x7f4b98022040 con 0x56015b03fd60 2023-10-05T19:28:29.011 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.006+0000 7f4ba0412700 1 -- 172.21.15.167:0/4265138048 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:30 5.3 5:c0000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x56015b048520 con 0x56015b0122f0 2023-10-05T19:28:29.011 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.006+0000 7f4ba0412700 1 -- 172.21.15.167:0/4265138048 <== osd.0 v2:172.21.15.118:6800/27690 7 ==== osd_op_reply(30 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f4b98022040 con 0x56015b0122f0 2023-10-05T19:28:29.011 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.006+0000 7f4ba0412700 1 -- 172.21.15.167:0/4265138048 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:31 5.13 5:c8000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x56015b048520 con 0x56015b045c00 2023-10-05T19:28:29.012 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.006+0000 7f4b9fc11700 1 -- 172.21.15.167:0/4265138048 <== osd.3 v2:172.21.15.167:6808/30036 8 ==== osd_op_reply(31 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f4b90025020 con 0x56015b045c00 2023-10-05T19:28:29.012 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.006+0000 7f4b9fc11700 1 -- 172.21.15.167:0/4265138048 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:32 5.b 5:d0000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x56015b048520 con 0x56015b0122f0 2023-10-05T19:28:29.013 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.006+0000 7f4ba0412700 1 -- 172.21.15.167:0/4265138048 <== osd.0 v2:172.21.15.118:6800/27690 8 ==== osd_op_reply(32 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f4b98022040 con 0x56015b0122f0 2023-10-05T19:28:29.013 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.006+0000 7f4ba0412700 1 -- 172.21.15.167:0/4265138048 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:33 5.1b 5:d8000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x56015b048520 con 0x56015af823d0 2023-10-05T19:28:29.014 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.010+0000 7f4b9f410700 1 -- 172.21.15.167:0/4265138048 <== osd.2 v2:172.21.15.167:6816/30037 5 ==== osd_op_reply(33 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f4b88008d50 con 0x56015af823d0 2023-10-05T19:28:29.014 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.010+0000 7f4b9f410700 1 -- 172.21.15.167:0/4265138048 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:34 5.7 5:e0000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x56015b048520 con 0x56015b045c00 2023-10-05T19:28:29.015 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.010+0000 7f4b9fc11700 1 -- 172.21.15.167:0/4265138048 <== osd.3 v2:172.21.15.167:6808/30036 9 ==== osd_op_reply(34 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f4b90025020 con 0x56015b045c00 2023-10-05T19:28:29.015 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.010+0000 7f4b9fc11700 1 -- 172.21.15.167:0/4265138048 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:35 5.17 5:e8000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x56015b048520 con 0x56015b0122f0 2023-10-05T19:28:29.016 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.010+0000 7f4ba0412700 1 -- 172.21.15.167:0/4265138048 <== osd.0 v2:172.21.15.118:6800/27690 9 ==== osd_op_reply(35 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f4b98022040 con 0x56015b0122f0 2023-10-05T19:28:29.016 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.010+0000 7f4ba0412700 1 -- 172.21.15.167:0/4265138048 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:36 5.f 5:f0000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x56015b048520 con 0x56015b03fd60 2023-10-05T19:28:29.017 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.010+0000 7f4ba0412700 1 -- 172.21.15.167:0/4265138048 <== osd.1 v2:172.21.15.167:6800/30035 13 ==== osd_op_reply(36 [pgnls start_epoch 30 out=115b] v23'2 uv2 ondisk = 1) v8 ==== 144+0+115 (crc 0 0 0) 0x7f4b9802f080 con 0x56015b03fd60 2023-10-05T19:28:29.017 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.010+0000 7f4ba0412700 1 -- 172.21.15.167:0/4265138048 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:37 5.1f 5:f8000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x56015b048520 con 0x56015af823d0 2023-10-05T19:28:29.018 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.010+0000 7f4b9f410700 1 -- 172.21.15.167:0/4265138048 <== osd.2 v2:172.21.15.167:6816/30037 6 ==== osd_op_reply(37 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f4b88008d50 con 0x56015af823d0 2023-10-05T19:28:29.018 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.010+0000 7f4ba20dce40 20 RGWRados::pool_iterate: got zonegroup_info.4f1f719d-9648-417b-b10c-e5d939825419 2023-10-05T19:28:29.018 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.010+0000 7f4ba20dce40 20 RGWRados::pool_iterate: got zone_info.9a9df965-6612-45d4-a45f-6e3c889bacc2 2023-10-05T19:28:29.018 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.010+0000 7f4ba20dce40 20 RGWRados::pool_iterate: got zone_names.default 2023-10-05T19:28:29.018 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.010+0000 7f4ba20dce40 20 RGWRados::pool_iterate: got zonegroups_names.default 2023-10-05T19:28:29.018 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.010+0000 7f4ba20dce40 20 rados->read ofs=0 len=0 2023-10-05T19:28:29.018 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.010+0000 7f4ba20dce40 1 -- 172.21.15.167:0/4265138048 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:38 5.12 5:49953fa1:::default.realm:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x56015b0449b0 con 0x56015b03fd60 2023-10-05T19:28:29.019 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.014+0000 7f4ba0412700 1 -- 172.21.15.167:0/4265138048 <== osd.1 v2:172.21.15.167:6800/30035 14 ==== osd_op_reply(38 default.realm [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 ==== 157+0+0 (crc 0 0 0) 0x7f4b9802f080 con 0x56015b03fd60 2023-10-05T19:28:29.019 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.014+0000 7f4ba20dce40 20 rados_obj.operate() r=-2 bl.length=0 2023-10-05T19:28:29.019 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.014+0000 7f4ba20dce40 20 rados->read ofs=0 len=0 2023-10-05T19:28:29.019 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.014+0000 7f4ba20dce40 1 -- 172.21.15.167:0/4265138048 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:39 5.f 5:f4c53578:::zonegroups_names.default:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x56015b0445b0 con 0x56015b03fd60 2023-10-05T19:28:29.019 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.014+0000 7f4ba0412700 1 -- 172.21.15.167:0/4265138048 <== osd.1 v2:172.21.15.167:6800/30035 15 ==== osd_op_reply(39 zonegroups_names.default [read 0~46 out=46b] v0'0 uv2 ondisk = 0) v8 ==== 168+0+46 (crc 0 0 0) 0x7f4b9802f080 con 0x56015b03fd60 2023-10-05T19:28:29.020 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.014+0000 7f4ba20dce40 20 rados_obj.operate() r=0 bl.length=46 2023-10-05T19:28:29.020 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.014+0000 7f4ba20dce40 20 rados->read ofs=0 len=0 2023-10-05T19:28:29.020 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.014+0000 7f4ba20dce40 1 -- 172.21.15.167:0/4265138048 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:40 5.c 5:30d299a5:::zonegroup_info.4f1f719d-9648-417b-b10c-e5d939825419:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x56015b0452a0 con 0x56015b0122f0 2023-10-05T19:28:29.021 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.014+0000 7f4ba0412700 1 -- 172.21.15.167:0/4265138048 <== osd.0 v2:172.21.15.118:6800/27690 10 ==== osd_op_reply(40 zonegroup_info.4f1f719d-9648-417b-b10c-e5d939825419 [read 0~362 out=362b] v0'0 uv1 ondisk = 0) v8 ==== 195+0+362 (crc 0 0 0) 0x7f4b9802f080 con 0x56015b0122f0 2023-10-05T19:28:29.021 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.014+0000 7f4ba20dce40 20 rados_obj.operate() r=0 bl.length=362 2023-10-05T19:28:29.021 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.014+0000 7f4ba20dce40 20 rados->read ofs=0 len=0 2023-10-05T19:28:29.021 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.014+0000 7f4ba20dce40 1 -- 172.21.15.167:0/4265138048 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:41 5.13 5:cbbf6c60:::converted:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x56015b0488d0 con 0x56015b045c00 2023-10-05T19:28:29.021 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.014+0000 7f4b9fc11700 1 -- 172.21.15.167:0/4265138048 <== osd.3 v2:172.21.15.167:6808/30036 10 ==== osd_op_reply(41 converted [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 ==== 153+0+0 (crc 0 0 0) 0x7f4b90025020 con 0x56015b045c00 2023-10-05T19:28:29.021 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.014+0000 7f4ba20dce40 20 rados_obj.operate() r=-2 bl.length=0 2023-10-05T19:28:29.021 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.014+0000 7f4ba20dce40 20 rados->read ofs=0 len=0 2023-10-05T19:28:29.021 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.014+0000 7f4ba20dce40 1 -- 172.21.15.167:0/4265138048 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:42 5.12 5:49953fa1:::default.realm:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x56015b048c80 con 0x56015b03fd60 2023-10-05T19:28:29.022 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.014+0000 7f4ba0412700 1 -- 172.21.15.167:0/4265138048 <== osd.1 v2:172.21.15.167:6800/30035 16 ==== osd_op_reply(42 default.realm [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 ==== 157+0+0 (crc 0 0 0) 0x7f4b9802f080 con 0x56015b03fd60 2023-10-05T19:28:29.022 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.014+0000 7f4ba20dce40 20 rados_obj.operate() r=-2 bl.length=0 2023-10-05T19:28:29.022 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.014+0000 7f4ba20dce40 20 rados->read ofs=0 len=0 2023-10-05T19:28:29.022 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.014+0000 7f4ba20dce40 1 -- 172.21.15.167:0/4265138048 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:43 5.f 5:f4c53578:::zonegroups_names.default:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x56015b048c80 con 0x56015b03fd60 2023-10-05T19:28:29.023 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.018+0000 7f4ba0412700 1 -- 172.21.15.167:0/4265138048 <== osd.1 v2:172.21.15.167:6800/30035 17 ==== osd_op_reply(43 zonegroups_names.default [read 0~46 out=46b] v0'0 uv2 ondisk = 0) v8 ==== 168+0+46 (crc 0 0 0) 0x7f4b9802f080 con 0x56015b03fd60 2023-10-05T19:28:29.023 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.018+0000 7f4ba20dce40 20 rados_obj.operate() r=0 bl.length=46 2023-10-05T19:28:29.023 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.018+0000 7f4ba20dce40 1 -- 172.21.15.167:0/4265138048 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:44 5.0 5:00000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x56015b049c90 con 0x56015b045c00 2023-10-05T19:28:29.025 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.018+0000 7f4b9fc11700 1 -- 172.21.15.167:0/4265138048 <== osd.3 v2:172.21.15.167:6808/30036 11 ==== osd_op_reply(44 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f4b90025020 con 0x56015b045c00 2023-10-05T19:28:29.025 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.018+0000 7f4b9fc11700 1 -- 172.21.15.167:0/4265138048 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:45 5.10 5:08000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x56015b049c90 con 0x56015af823d0 2023-10-05T19:28:29.025 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.018+0000 7f4b9f410700 1 -- 172.21.15.167:0/4265138048 <== osd.2 v2:172.21.15.167:6816/30037 7 ==== osd_op_reply(45 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f4b88008d50 con 0x56015af823d0 2023-10-05T19:28:29.026 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.018+0000 7f4b9f410700 1 -- 172.21.15.167:0/4265138048 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:46 5.8 5:10000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x56015b049c90 con 0x56015b0122f0 2023-10-05T19:28:29.026 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.022+0000 7f4ba0412700 1 -- 172.21.15.167:0/4265138048 <== osd.0 v2:172.21.15.118:6800/27690 11 ==== osd_op_reply(46 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f4b9802f080 con 0x56015b0122f0 2023-10-05T19:28:29.027 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.022+0000 7f4ba0412700 1 -- 172.21.15.167:0/4265138048 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:47 5.18 5:18000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x56015b0445b0 con 0x56015b03fd60 2023-10-05T19:28:29.027 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.022+0000 7f4ba0412700 1 -- 172.21.15.167:0/4265138048 <== osd.1 v2:172.21.15.167:6800/30035 18 ==== osd_op_reply(47 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f4b9802f080 con 0x56015b03fd60 2023-10-05T19:28:29.027 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.022+0000 7f4ba0412700 1 -- 172.21.15.167:0/4265138048 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:48 5.4 5:20000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x56015b0445b0 con 0x56015b03fd60 2023-10-05T19:28:29.028 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.022+0000 7f4ba0412700 1 -- 172.21.15.167:0/4265138048 <== osd.1 v2:172.21.15.167:6800/30035 19 ==== osd_op_reply(48 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f4b9802f080 con 0x56015b03fd60 2023-10-05T19:28:29.028 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.022+0000 7f4ba0412700 1 -- 172.21.15.167:0/4265138048 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:49 5.14 5:28000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x56015b0445b0 con 0x56015b0122f0 2023-10-05T19:28:29.029 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.022+0000 7f4ba0412700 1 -- 172.21.15.167:0/4265138048 <== osd.0 v2:172.21.15.118:6800/27690 12 ==== osd_op_reply(49 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f4b9802f080 con 0x56015b0122f0 2023-10-05T19:28:29.029 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.022+0000 7f4ba0412700 1 -- 172.21.15.167:0/4265138048 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:50 5.c 5:30000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x56015b0445b0 con 0x56015b0122f0 2023-10-05T19:28:29.030 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.022+0000 7f4ba0412700 1 -- 172.21.15.167:0/4265138048 <== osd.0 v2:172.21.15.118:6800/27690 13 ==== osd_op_reply(50 [pgnls start_epoch 30 out=112b] v23'1 uv1 ondisk = 1) v8 ==== 144+0+112 (crc 0 0 0) 0x7f4b9802f080 con 0x56015b0122f0 2023-10-05T19:28:29.030 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.026+0000 7f4ba0412700 1 -- 172.21.15.167:0/4265138048 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:51 5.1c 5:38000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x56015b0445b0 con 0x56015af823d0 2023-10-05T19:28:29.030 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.026+0000 7f4b9f410700 1 -- 172.21.15.167:0/4265138048 <== osd.2 v2:172.21.15.167:6816/30037 8 ==== osd_op_reply(51 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f4b88008d50 con 0x56015af823d0 2023-10-05T19:28:29.031 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.026+0000 7f4b9f410700 1 -- 172.21.15.167:0/4265138048 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:52 5.2 5:40000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x56015b0445b0 con 0x56015b045c00 2023-10-05T19:28:29.031 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.026+0000 7f4b9fc11700 1 -- 172.21.15.167:0/4265138048 <== osd.3 v2:172.21.15.167:6808/30036 12 ==== osd_op_reply(52 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f4b90025020 con 0x56015b045c00 2023-10-05T19:28:29.031 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.026+0000 7f4b9fc11700 1 -- 172.21.15.167:0/4265138048 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:53 5.12 5:48000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x56015b0445b0 con 0x56015b03fd60 2023-10-05T19:28:29.032 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.026+0000 7f4ba0412700 1 -- 172.21.15.167:0/4265138048 <== osd.1 v2:172.21.15.167:6800/30035 20 ==== osd_op_reply(53 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f4b9802f080 con 0x56015b03fd60 2023-10-05T19:28:29.032 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.026+0000 7f4ba0412700 1 -- 172.21.15.167:0/4265138048 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:54 5.a 5:50000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x56015b0445b0 con 0x56015b0122f0 2023-10-05T19:28:29.033 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.026+0000 7f4ba0412700 1 -- 172.21.15.167:0/4265138048 <== osd.0 v2:172.21.15.118:6800/27690 14 ==== osd_op_reply(54 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f4b9802f080 con 0x56015b0122f0 2023-10-05T19:28:29.033 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.026+0000 7f4ba0412700 1 -- 172.21.15.167:0/4265138048 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:55 5.1a 5:58000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x56015b0445b0 con 0x56015b03fd60 2023-10-05T19:28:29.034 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.026+0000 7f4ba0412700 1 -- 172.21.15.167:0/4265138048 <== osd.1 v2:172.21.15.167:6800/30035 21 ==== osd_op_reply(55 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f4b9802f080 con 0x56015b03fd60 2023-10-05T19:28:29.034 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.026+0000 7f4ba0412700 1 -- 172.21.15.167:0/4265138048 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:56 5.6 5:60000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x56015b0445b0 con 0x56015af823d0 2023-10-05T19:28:29.034 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.030+0000 7f4b9f410700 1 -- 172.21.15.167:0/4265138048 <== osd.2 v2:172.21.15.167:6816/30037 9 ==== osd_op_reply(56 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f4b88008d50 con 0x56015af823d0 2023-10-05T19:28:29.035 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.030+0000 7f4b9f410700 1 -- 172.21.15.167:0/4265138048 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:57 5.16 5:68000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x56015b0445b0 con 0x56015b045c00 2023-10-05T19:28:29.035 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.030+0000 7f4b9fc11700 1 -- 172.21.15.167:0/4265138048 <== osd.3 v2:172.21.15.167:6808/30036 13 ==== osd_op_reply(57 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f4b90025020 con 0x56015b045c00 2023-10-05T19:28:29.035 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.030+0000 7f4b9fc11700 1 -- 172.21.15.167:0/4265138048 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:58 5.e 5:70000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x56015b0445b0 con 0x56015b045c00 2023-10-05T19:28:29.036 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.030+0000 7f4b9fc11700 1 -- 172.21.15.167:0/4265138048 <== osd.3 v2:172.21.15.167:6808/30036 14 ==== osd_op_reply(58 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f4b90025020 con 0x56015b045c00 2023-10-05T19:28:29.036 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.030+0000 7f4b9fc11700 1 -- 172.21.15.167:0/4265138048 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:59 5.1e 5:78000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x56015b0445b0 con 0x56015b0122f0 2023-10-05T19:28:29.037 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.030+0000 7f4ba0412700 1 -- 172.21.15.167:0/4265138048 <== osd.0 v2:172.21.15.118:6800/27690 15 ==== osd_op_reply(59 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f4b9802f080 con 0x56015b0122f0 2023-10-05T19:28:29.037 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.030+0000 7f4ba0412700 1 -- 172.21.15.167:0/4265138048 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:60 5.1 5:80000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x56015b0445b0 con 0x56015b045c00 2023-10-05T19:28:29.038 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.030+0000 7f4b9fc11700 1 -- 172.21.15.167:0/4265138048 <== osd.3 v2:172.21.15.167:6808/30036 15 ==== osd_op_reply(60 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f4b90025020 con 0x56015b045c00 2023-10-05T19:28:29.038 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.030+0000 7f4b9fc11700 1 -- 172.21.15.167:0/4265138048 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:61 5.11 5:88000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x56015b0445b0 con 0x56015b045c00 2023-10-05T19:28:29.038 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.034+0000 7f4b9fc11700 1 -- 172.21.15.167:0/4265138048 <== osd.3 v2:172.21.15.167:6808/30036 16 ==== osd_op_reply(61 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f4b90025020 con 0x56015b045c00 2023-10-05T19:28:29.038 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.034+0000 7f4b9fc11700 1 -- 172.21.15.167:0/4265138048 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:62 5.9 5:90000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x56015b0445b0 con 0x56015b03fd60 2023-10-05T19:28:29.039 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.034+0000 7f4ba0412700 1 -- 172.21.15.167:0/4265138048 <== osd.1 v2:172.21.15.167:6800/30035 22 ==== osd_op_reply(62 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f4b9802f080 con 0x56015b03fd60 2023-10-05T19:28:29.039 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.034+0000 7f4ba0412700 1 -- 172.21.15.167:0/4265138048 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:63 5.19 5:98000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x56015b0445b0 con 0x56015b03fd60 2023-10-05T19:28:29.040 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.034+0000 7f4ba0412700 1 -- 172.21.15.167:0/4265138048 <== osd.1 v2:172.21.15.167:6800/30035 23 ==== osd_op_reply(63 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f4b9802f080 con 0x56015b03fd60 2023-10-05T19:28:29.040 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.034+0000 7f4ba0412700 1 -- 172.21.15.167:0/4265138048 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:64 5.5 5:a0000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x56015b0445b0 con 0x56015b0122f0 2023-10-05T19:28:29.041 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.034+0000 7f4ba0412700 1 -- 172.21.15.167:0/4265138048 <== osd.0 v2:172.21.15.118:6800/27690 16 ==== osd_op_reply(64 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f4b9802f080 con 0x56015b0122f0 2023-10-05T19:28:29.041 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.034+0000 7f4ba0412700 1 -- 172.21.15.167:0/4265138048 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:65 5.15 5:a8000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x56015b0445b0 con 0x56015b045c00 2023-10-05T19:28:29.042 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.038+0000 7f4b9fc11700 1 -- 172.21.15.167:0/4265138048 <== osd.3 v2:172.21.15.167:6808/30036 17 ==== osd_op_reply(65 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f4b90025020 con 0x56015b045c00 2023-10-05T19:28:29.042 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.038+0000 7f4b9fc11700 1 -- 172.21.15.167:0/4265138048 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:66 5.d 5:b0000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x56015b0445b0 con 0x56015af823d0 2023-10-05T19:28:29.043 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.038+0000 7f4b9f410700 1 -- 172.21.15.167:0/4265138048 <== osd.2 v2:172.21.15.167:6816/30037 10 ==== osd_op_reply(66 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f4b88008d50 con 0x56015af823d0 2023-10-05T19:28:29.043 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.038+0000 7f4b9f410700 1 -- 172.21.15.167:0/4265138048 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:67 5.1d 5:b8000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x56015b0445b0 con 0x56015b03fd60 2023-10-05T19:28:29.044 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.038+0000 7f4ba0412700 1 -- 172.21.15.167:0/4265138048 <== osd.1 v2:172.21.15.167:6800/30035 24 ==== osd_op_reply(67 [pgnls start_epoch 30 out=107b] v30'2 uv2 ondisk = 1) v8 ==== 144+0+107 (crc 0 0 0) 0x7f4b9802f080 con 0x56015b03fd60 2023-10-05T19:28:29.044 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.038+0000 7f4ba0412700 1 -- 172.21.15.167:0/4265138048 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:68 5.3 5:c0000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x56015b0445b0 con 0x56015b0122f0 2023-10-05T19:28:29.045 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.038+0000 7f4ba0412700 1 -- 172.21.15.167:0/4265138048 <== osd.0 v2:172.21.15.118:6800/27690 17 ==== osd_op_reply(68 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f4b9802f080 con 0x56015b0122f0 2023-10-05T19:28:29.045 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.038+0000 7f4ba0412700 1 -- 172.21.15.167:0/4265138048 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:69 5.13 5:c8000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x56015b0445b0 con 0x56015b045c00 2023-10-05T19:28:29.045 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.038+0000 7f4b9fc11700 1 -- 172.21.15.167:0/4265138048 <== osd.3 v2:172.21.15.167:6808/30036 18 ==== osd_op_reply(69 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f4b90025020 con 0x56015b045c00 2023-10-05T19:28:29.045 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.038+0000 7f4b9fc11700 1 -- 172.21.15.167:0/4265138048 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:70 5.b 5:d0000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x56015b0445b0 con 0x56015b0122f0 2023-10-05T19:28:29.046 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.042+0000 7f4ba0412700 1 -- 172.21.15.167:0/4265138048 <== osd.0 v2:172.21.15.118:6800/27690 18 ==== osd_op_reply(70 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f4b9802f080 con 0x56015b0122f0 2023-10-05T19:28:29.046 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.042+0000 7f4ba0412700 1 -- 172.21.15.167:0/4265138048 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:71 5.1b 5:d8000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x56015b0445b0 con 0x56015af823d0 2023-10-05T19:28:29.047 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.042+0000 7f4b9f410700 1 -- 172.21.15.167:0/4265138048 <== osd.2 v2:172.21.15.167:6816/30037 11 ==== osd_op_reply(71 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f4b88008d50 con 0x56015af823d0 2023-10-05T19:28:29.048 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.042+0000 7f4b9f410700 1 -- 172.21.15.167:0/4265138048 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:72 5.7 5:e0000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x56015b0445b0 con 0x56015b045c00 2023-10-05T19:28:29.048 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.042+0000 7f4b9fc11700 1 -- 172.21.15.167:0/4265138048 <== osd.3 v2:172.21.15.167:6808/30036 19 ==== osd_op_reply(72 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f4b90025020 con 0x56015b045c00 2023-10-05T19:28:29.048 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.042+0000 7f4b9fc11700 1 -- 172.21.15.167:0/4265138048 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:73 5.17 5:e8000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x56015b0445b0 con 0x56015b0122f0 2023-10-05T19:28:29.049 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.042+0000 7f4ba0412700 1 -- 172.21.15.167:0/4265138048 <== osd.0 v2:172.21.15.118:6800/27690 19 ==== osd_op_reply(73 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f4b9802f080 con 0x56015b0122f0 2023-10-05T19:28:29.049 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.042+0000 7f4ba0412700 1 -- 172.21.15.167:0/4265138048 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:74 5.f 5:f0000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x56015b0445b0 con 0x56015b03fd60 2023-10-05T19:28:29.050 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.046+0000 7f4ba0412700 1 -- 172.21.15.167:0/4265138048 <== osd.1 v2:172.21.15.167:6800/30035 25 ==== osd_op_reply(74 [pgnls start_epoch 30 out=115b] v23'2 uv2 ondisk = 1) v8 ==== 144+0+115 (crc 0 0 0) 0x7f4b9802f080 con 0x56015b03fd60 2023-10-05T19:28:29.050 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.046+0000 7f4ba0412700 1 -- 172.21.15.167:0/4265138048 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:75 5.1f 5:f8000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x56015b0445b0 con 0x56015af823d0 2023-10-05T19:28:29.051 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.046+0000 7f4b9f410700 1 -- 172.21.15.167:0/4265138048 <== osd.2 v2:172.21.15.167:6816/30037 12 ==== osd_op_reply(75 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f4b88008d50 con 0x56015af823d0 2023-10-05T19:28:29.051 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.046+0000 7f4ba20dce40 20 RGWRados::pool_iterate: got zonegroup_info.4f1f719d-9648-417b-b10c-e5d939825419 2023-10-05T19:28:29.051 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.046+0000 7f4ba20dce40 20 RGWRados::pool_iterate: got zone_info.9a9df965-6612-45d4-a45f-6e3c889bacc2 2023-10-05T19:28:29.052 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.046+0000 7f4ba20dce40 20 RGWRados::pool_iterate: got zone_names.default 2023-10-05T19:28:29.052 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.046+0000 7f4ba20dce40 20 RGWRados::pool_iterate: got zonegroups_names.default 2023-10-05T19:28:29.052 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.046+0000 7f4ba20dce40 20 rados->read ofs=0 len=0 2023-10-05T19:28:29.052 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.046+0000 7f4ba20dce40 1 -- 172.21.15.167:0/4265138048 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:76 5.f 5:f43b6ece:::zone_names.default:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x56015b048c80 con 0x56015b03fd60 2023-10-05T19:28:29.052 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.046+0000 7f4ba0412700 1 -- 172.21.15.167:0/4265138048 <== osd.1 v2:172.21.15.167:6800/30035 26 ==== osd_op_reply(76 zone_names.default [read 0~46 out=46b] v0'0 uv1 ondisk = 0) v8 ==== 162+0+46 (crc 0 0 0) 0x7f4b9802f080 con 0x56015b03fd60 2023-10-05T19:28:29.052 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.046+0000 7f4ba20dce40 20 rados_obj.operate() r=0 bl.length=46 2023-10-05T19:28:29.053 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.046+0000 7f4ba20dce40 20 rados->read ofs=0 len=0 2023-10-05T19:28:29.053 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.046+0000 7f4ba20dce40 1 -- 172.21.15.167:0/4265138048 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:77 5.1d 5:b8374eb4:::zone_info.9a9df965-6612-45d4-a45f-6e3c889bacc2:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x56015b049030 con 0x56015b03fd60 2023-10-05T19:28:29.053 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.046+0000 7f4ba0412700 1 -- 172.21.15.167:0/4265138048 <== osd.1 v2:172.21.15.167:6800/30035 27 ==== osd_op_reply(77 zone_info.9a9df965-6612-45d4-a45f-6e3c889bacc2 [read 0~874 out=874b] v0'0 uv2 ondisk = 0) v8 ==== 190+0+874 (crc 0 0 0) 0x7f4b9802f080 con 0x56015b03fd60 2023-10-05T19:28:29.054 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.046+0000 7f4ba20dce40 20 rados_obj.operate() r=0 bl.length=874 2023-10-05T19:28:29.054 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.046+0000 7f4ba20dce40 20 rados->read ofs=0 len=0 2023-10-05T19:28:29.054 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.046+0000 7f4ba20dce40 1 -- 172.21.15.167:0/4265138048 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:78 5.f 5:f4c53578:::zonegroups_names.default:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x56015b049470 con 0x56015b03fd60 2023-10-05T19:28:29.054 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.046+0000 7f4ba0412700 1 -- 172.21.15.167:0/4265138048 <== osd.1 v2:172.21.15.167:6800/30035 28 ==== osd_op_reply(78 zonegroups_names.default [read 0~46 out=46b] v0'0 uv2 ondisk = 0) v8 ==== 168+0+46 (crc 0 0 0) 0x7f4b9802f080 con 0x56015b03fd60 2023-10-05T19:28:29.054 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.046+0000 7f4ba20dce40 20 rados_obj.operate() r=0 bl.length=46 2023-10-05T19:28:29.054 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.046+0000 7f4ba20dce40 20 rados->read ofs=0 len=0 2023-10-05T19:28:29.054 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.046+0000 7f4ba20dce40 1 -- 172.21.15.167:0/4265138048 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:79 5.c 5:30d299a5:::zonegroup_info.4f1f719d-9648-417b-b10c-e5d939825419:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x56015b049470 con 0x56015b0122f0 2023-10-05T19:28:29.055 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.050+0000 7f4ba0412700 1 -- 172.21.15.167:0/4265138048 <== osd.0 v2:172.21.15.118:6800/27690 20 ==== osd_op_reply(79 zonegroup_info.4f1f719d-9648-417b-b10c-e5d939825419 [read 0~362 out=362b] v0'0 uv1 ondisk = 0) v8 ==== 195+0+362 (crc 0 0 0) 0x7f4b9802f080 con 0x56015b0122f0 2023-10-05T19:28:29.055 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.050+0000 7f4ba20dce40 20 rados_obj.operate() r=0 bl.length=362 2023-10-05T19:28:29.055 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.050+0000 7f4ba20dce40 20 rados->read ofs=0 len=0 2023-10-05T19:28:29.055 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.050+0000 7f4ba20dce40 1 -- 172.21.15.167:0/4265138048 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:80 5.3 5:c0ad8cbd:::region_map:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x56015b0498a0 con 0x56015b0122f0 2023-10-05T19:28:29.056 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.050+0000 7f4ba0412700 1 -- 172.21.15.167:0/4265138048 <== osd.0 v2:172.21.15.118:6800/27690 21 ==== osd_op_reply(80 region_map [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 ==== 154+0+0 (crc 0 0 0) 0x7f4b98050040 con 0x56015b0122f0 2023-10-05T19:28:29.056 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.050+0000 7f4ba20dce40 20 rados_obj.operate() r=-2 bl.length=0 2023-10-05T19:28:29.056 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.050+0000 7f4ba20dce40 20 zone default found 2023-10-05T19:28:29.056 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.050+0000 7f4ba20dce40 4 Realm: () 2023-10-05T19:28:29.056 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.050+0000 7f4ba20dce40 4 ZoneGroup: default (4f1f719d-9648-417b-b10c-e5d939825419) 2023-10-05T19:28:29.056 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.050+0000 7f4ba20dce40 4 Zone: default (9a9df965-6612-45d4-a45f-6e3c889bacc2) 2023-10-05T19:28:29.056 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.050+0000 7f4ba20dce40 10 cannot find current period zonegroup using local zonegroup configuration 2023-10-05T19:28:29.057 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.050+0000 7f4ba20dce40 20 zonegroup default 2023-10-05T19:28:29.057 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.050+0000 7f4ba20dce40 20 rados->read ofs=0 len=0 2023-10-05T19:28:29.057 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.050+0000 7f4ba20dce40 1 -- 172.21.15.167:0/4265138048 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:81 5.3 5:c52100b6:::period_config.default:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x56015b0495d0 con 0x56015b0122f0 2023-10-05T19:28:29.058 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.050+0000 7f4ba0412700 1 -- 172.21.15.167:0/4265138048 <== osd.0 v2:172.21.15.118:6800/27690 22 ==== osd_op_reply(81 period_config.default [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 ==== 165+0+0 (crc 0 0 0) 0x7f4b98050040 con 0x56015b0122f0 2023-10-05T19:28:29.058 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.050+0000 7f4ba20dce40 20 rados_obj.operate() r=-2 bl.length=0 2023-10-05T19:28:29.058 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.050+0000 7f4ba20dce40 20 rados->read ofs=0 len=0 2023-10-05T19:28:29.058 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.050+0000 7f4ba20dce40 1 -- 172.21.15.167:0/4265138048 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:82 6.2 6:4347d321:::bucket.sync-source-hints.:head [call version.read in=11b,read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x56015b04af50 con 0x56015b045c00 2023-10-05T19:28:29.059 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.050+0000 7f4b9fc11700 1 -- 172.21.15.167:0/4265138048 <== osd.3 v2:172.21.15.167:6808/30036 20 ==== osd_op_reply(82 bucket.sync-source-hints. [call,read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 ==== 211+0+0 (crc 0 0 0) 0x7f4b90025020 con 0x56015b045c00 2023-10-05T19:28:29.059 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.050+0000 7f4ba20dce40 20 rados_obj.operate() r=-2 bl.length=0 2023-10-05T19:28:29.059 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.050+0000 7f4ba20dce40 20 rados->read ofs=0 len=0 2023-10-05T19:28:29.059 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.050+0000 7f4ba20dce40 1 -- 172.21.15.167:0/4265138048 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:83 6.b 6:d467b91b:::bucket.sync-target-hints.:head [call version.read in=11b,read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x56015b04b300 con 0x56015b045c00 2023-10-05T19:28:29.059 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.054+0000 7f4b9fc11700 1 -- 172.21.15.167:0/4265138048 <== osd.3 v2:172.21.15.167:6808/30036 21 ==== osd_op_reply(83 bucket.sync-target-hints. [call,read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 ==== 211+0+0 (crc 0 0 0) 0x7f4b90025020 con 0x56015b045c00 2023-10-05T19:28:29.059 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.054+0000 7f4ba20dce40 20 rados_obj.operate() r=-2 bl.length=0 2023-10-05T19:28:29.059 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.054+0000 7f4ba20dce40 20 started sync module instance, tier type = 2023-10-05T19:28:29.059 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.054+0000 7f4ba20dce40 20 started zone id=9a9df965-6612-45d4-a45f-6e3c889bacc2 (name=default) with tier type = 2023-10-05T19:28:29.059 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.054+0000 7f4ba20dce40 1 -- 172.21.15.167:0/4265138048 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:84 7.1f 7:f95f44c2:::notify.0:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x56015b04a610 con 0x56015af823d0 2023-10-05T19:28:29.061 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.054+0000 7f4b9f410700 1 -- 172.21.15.167:0/4265138048 <== osd.2 v2:172.21.15.167:6816/30037 13 ==== osd_op_reply(84 notify.0 [create] v30'69 uv69 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f4b88008d50 con 0x56015af823d0 2023-10-05T19:28:29.061 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.054+0000 7f4ba20dce40 1 -- 172.21.15.167:0/4265138048 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:85 7.1f 7:f95f44c2:::notify.0:head [watch watch cookie 94563821988368] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x56015b044e80 con 0x56015af823d0 2023-10-05T19:28:29.061 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.054+0000 7f4ba20dce40 1 -- 172.21.15.167:0/4265138048 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:86 7.0 7:05bf5b68:::notify.1:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x56015b04c120 con 0x56015b03fd60 2023-10-05T19:28:29.064 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.058+0000 7f4ba0412700 1 -- 172.21.15.167:0/4265138048 <== osd.1 v2:172.21.15.167:6800/30035 29 ==== osd_op_reply(86 notify.1 [create] v30'69 uv69 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f4b98050040 con 0x56015b03fd60 2023-10-05T19:28:29.064 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.058+0000 7f4ba20dce40 1 -- 172.21.15.167:0/4265138048 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:87 7.0 7:05bf5b68:::notify.1:head [watch watch cookie 94563821993296] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x56015b04ca10 con 0x56015b03fd60 2023-10-05T19:28:29.064 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.058+0000 7f4ba20dce40 1 -- 172.21.15.167:0/4265138048 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:88 7.15 7:a93a5511:::notify.2:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x56015b04d2b0 con 0x56015b045c00 2023-10-05T19:28:29.064 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.058+0000 7f4b9f410700 1 -- 172.21.15.167:0/4265138048 <== osd.2 v2:172.21.15.167:6816/30037 14 ==== osd_op_reply(85 notify.0 [watch watch cookie 94563821988368] v30'70 uv69 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f4b88008d50 con 0x56015af823d0 2023-10-05T19:28:29.067 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.062+0000 7f4ba0412700 1 -- 172.21.15.167:0/4265138048 <== osd.1 v2:172.21.15.167:6800/30035 30 ==== osd_op_reply(87 notify.1 [watch watch cookie 94563821993296] v30'70 uv69 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f4b98050040 con 0x56015b03fd60 2023-10-05T19:28:29.068 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.062+0000 7f4b9fc11700 1 -- 172.21.15.167:0/4265138048 <== osd.3 v2:172.21.15.167:6808/30036 22 ==== osd_op_reply(88 notify.2 [create] v30'69 uv69 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f4b90025020 con 0x56015b045c00 2023-10-05T19:28:29.068 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.062+0000 7f4ba20dce40 1 -- 172.21.15.167:0/4265138048 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:89 7.15 7:a93a5511:::notify.2:head [watch watch cookie 94563821995296] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x56015b04b7d0 con 0x56015b045c00 2023-10-05T19:28:29.068 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.062+0000 7f4ba20dce40 1 -- 172.21.15.167:0/4265138048 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:90 7.e 7:7759931f:::notify.3:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x56015b04db50 con 0x56015af823d0 2023-10-05T19:28:29.070 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.066+0000 7f4b9f410700 1 -- 172.21.15.167:0/4265138048 <== osd.2 v2:172.21.15.167:6816/30037 15 ==== osd_op_reply(90 notify.3 [create] v30'69 uv69 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f4b88008d50 con 0x56015af823d0 2023-10-05T19:28:29.071 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.066+0000 7f4ba20dce40 1 -- 172.21.15.167:0/4265138048 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:91 7.e 7:7759931f:::notify.3:head [watch watch cookie 94563821999712] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x56015b04e3f0 con 0x56015af823d0 2023-10-05T19:28:29.071 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.066+0000 7f4ba20dce40 1 -- 172.21.15.167:0/4265138048 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:92 7.d 7:b4812045:::notify.4:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x56015b04ef50 con 0x56015b03fd60 2023-10-05T19:28:29.071 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.066+0000 7f4b9fc11700 1 -- 172.21.15.167:0/4265138048 <== osd.3 v2:172.21.15.167:6808/30036 23 ==== osd_op_reply(89 notify.2 [watch watch cookie 94563821995296] v30'70 uv69 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f4b90025020 con 0x56015b045c00 2023-10-05T19:28:29.073 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.066+0000 7f4b9f410700 1 -- 172.21.15.167:0/4265138048 <== osd.2 v2:172.21.15.167:6816/30037 16 ==== osd_op_reply(91 notify.3 [watch watch cookie 94563821999712] v30'70 uv69 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f4b88008d50 con 0x56015af823d0 2023-10-05T19:28:29.073 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.066+0000 7f4ba0412700 1 -- 172.21.15.167:0/4265138048 <== osd.1 v2:172.21.15.167:6800/30035 31 ==== osd_op_reply(92 notify.4 [create] v30'69 uv69 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f4b98050040 con 0x56015b03fd60 2023-10-05T19:28:29.074 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.066+0000 7f4ba20dce40 1 -- 172.21.15.167:0/4265138048 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:93 7.d 7:b4812045:::notify.4:head [watch watch cookie 94563822001920] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x56015b04ea20 con 0x56015b03fd60 2023-10-05T19:28:29.074 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.066+0000 7f4ba20dce40 1 -- 172.21.15.167:0/4265138048 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:94 7.3 7:c609908c:::notify.5:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x56015b04f840 con 0x56015b0122f0 2023-10-05T19:28:29.076 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.070+0000 7f4ba0412700 1 -- 172.21.15.167:0/4265138048 <== osd.0 v2:172.21.15.118:6800/27690 23 ==== osd_op_reply(94 notify.5 [create] v30'69 uv69 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f4b98050040 con 0x56015b0122f0 2023-10-05T19:28:29.076 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.070+0000 7f4ba20dce40 1 -- 172.21.15.167:0/4265138048 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:95 7.3 7:c609908c:::notify.5:head [watch watch cookie 94563822007040] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x56015b04fc40 con 0x56015b0122f0 2023-10-05T19:28:29.076 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.070+0000 7f4ba20dce40 1 -- 172.21.15.167:0/4265138048 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:96 7.14 7:2b04a3e9:::notify.6:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x56015b0506b0 con 0x56015b0122f0 2023-10-05T19:28:29.076 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.070+0000 7f4ba0412700 1 -- 172.21.15.167:0/4265138048 <== osd.1 v2:172.21.15.167:6800/30035 32 ==== osd_op_reply(93 notify.4 [watch watch cookie 94563822001920] v30'70 uv69 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f4b98050040 con 0x56015b03fd60 2023-10-05T19:28:29.078 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.074+0000 7f4ba0412700 1 -- 172.21.15.167:0/4265138048 <== osd.0 v2:172.21.15.118:6800/27690 24 ==== osd_op_reply(95 notify.5 [watch watch cookie 94563822007040] v30'70 uv69 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f4b98050040 con 0x56015b0122f0 2023-10-05T19:28:29.079 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.074+0000 7f4ba0412700 1 -- 172.21.15.167:0/4265138048 <== osd.0 v2:172.21.15.118:6800/27690 25 ==== osd_op_reply(96 notify.6 [create] v30'69 uv69 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f4b98050040 con 0x56015b0122f0 2023-10-05T19:28:29.079 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.074+0000 7f4ba20dce40 1 -- 172.21.15.167:0/4265138048 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:97 7.14 7:2b04a3e9:::notify.6:head [watch watch cookie 94563822008144] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x56015b04fba0 con 0x56015b0122f0 2023-10-05T19:28:29.079 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.074+0000 7f4ba20dce40 1 -- 172.21.15.167:0/4265138048 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:98 7.9 7:93e5b521:::notify.7:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x56015b050ba0 con 0x56015b0122f0 2023-10-05T19:28:29.081 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.074+0000 7f4ba0412700 1 -- 172.21.15.167:0/4265138048 <== osd.0 v2:172.21.15.118:6800/27690 26 ==== osd_op_reply(97 notify.6 [watch watch cookie 94563822008144] v30'70 uv69 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f4b98050040 con 0x56015b0122f0 2023-10-05T19:28:29.081 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.074+0000 7f4ba0412700 1 -- 172.21.15.167:0/4265138048 <== osd.0 v2:172.21.15.118:6800/27690 27 ==== osd_op_reply(98 notify.7 [create] v30'69 uv69 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f4b98050040 con 0x56015b0122f0 2023-10-05T19:28:29.081 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.074+0000 7f4ba20dce40 1 -- 172.21.15.167:0/4265138048 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:99 7.9 7:93e5b521:::notify.7:head [watch watch cookie 94563822010816] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x56015b050b00 con 0x56015b0122f0 2023-10-05T19:28:29.082 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.074+0000 7f4ba20dce40 20 add_watcher() i=0 2023-10-05T19:28:29.082 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.074+0000 7f4ba20dce40 20 add_watcher() i=1 2023-10-05T19:28:29.082 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.074+0000 7f4ba20dce40 20 add_watcher() i=2 2023-10-05T19:28:29.082 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.074+0000 7f4ba20dce40 20 add_watcher() i=3 2023-10-05T19:28:29.082 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.074+0000 7f4ba20dce40 20 add_watcher() i=4 2023-10-05T19:28:29.082 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.074+0000 7f4ba20dce40 20 add_watcher() i=5 2023-10-05T19:28:29.082 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.074+0000 7f4ba20dce40 20 add_watcher() i=6 2023-10-05T19:28:29.084 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.078+0000 7f4ba0412700 1 -- 172.21.15.167:0/4265138048 <== osd.0 v2:172.21.15.118:6800/27690 28 ==== osd_op_reply(99 notify.7 [watch watch cookie 94563822010816] v30'70 uv69 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f4b98050040 con 0x56015b0122f0 2023-10-05T19:28:29.084 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.078+0000 7f4ba20dce40 20 add_watcher() i=7 2023-10-05T19:28:29.084 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.078+0000 7f4ba20dce40 2 all 8 watchers are set, enabling cache 2023-10-05T19:28:29.084 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.078+0000 7f4ba20dce40 1 -- 172.21.15.167:0/4265138048 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:100 6.4 6:22d26bf9:::data_loggenerations_metadata:head [call version.check_conds in=74b,call version.read in=11b,read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x56015b053d40 con 0x56015b03fd60 2023-10-05T19:28:29.084 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.078+0000 7f4ba0412700 1 -- 172.21.15.167:0/4265138048 <== osd.1 v2:172.21.15.167:6800/30035 33 ==== osd_op_reply(100 data_loggenerations_metadata [call,call out=48b,read 0~28 out=28b] v0'0 uv1 ondisk = 0) v8 ==== 256+0+76 (crc 0 0 0) 0x7f4b98050040 con 0x56015b03fd60 2023-10-05T19:28:29.085 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.078+0000 7f4ba20dce40 1 -- 172.21.15.167:0/4265138048 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:101 6.4 6:22d26bf9:::data_loggenerations_metadata:head [watch watch cookie 94563822012288] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x56015b0510c0 con 0x56015b03fd60 2023-10-05T19:28:29.087 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.082+0000 7f4ba0412700 1 -- 172.21.15.167:0/4265138048 <== osd.1 v2:172.21.15.167:6800/30035 34 ==== osd_op_reply(101 data_loggenerations_metadata [watch watch cookie 94563822012288] v30'80 uv1 ondisk = 0) v8 ==== 172+0+0 (crc 0 0 0) 0x7f4b98050040 con 0x56015b03fd60 2023-10-05T19:28:29.087 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.082+0000 7f4b717e2700 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2023-10-05T19:28:29.087 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.082+0000 7f4ba20dce40 20 check_secure_mon_conn(): auth registy supported: methods=[2] modes=[2,1] 2023-10-05T19:28:29.087 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.082+0000 7f4ba20dce40 20 check_secure_mon_conn(): mode 1 is insecure 2023-10-05T19:28:29.088 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.082+0000 7f4ba20dce40 1 Processor -- start 2023-10-05T19:28:29.088 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.082+0000 7f4ba20dce40 1 -- start start 2023-10-05T19:28:29.088 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.082+0000 7f4ba20dce40 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x56015b05e230 0x56015b05e600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-05T19:28:29.088 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.082+0000 7f4ba20dce40 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x56015b05eb40 con 0x56015b05e230 2023-10-05T19:28:29.088 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.082+0000 7f4b9f410700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x56015b05e230 0x56015b05e600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-05T19:28:29.088 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.082+0000 7f4b9f410700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x56015b05e230 0x56015b05e600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.118:3300/0 says I am v2:172.21.15.167:57848/0 (socket says 172.21.15.167:57848) 2023-10-05T19:28:29.089 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.082+0000 7f4b9f410700 1 -- 172.21.15.167:0/1500058409 learned_addr learned my addr 172.21.15.167:0/1500058409 (peer_addr_for_me v2:172.21.15.167:0/0) 2023-10-05T19:28:29.089 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.082+0000 7f4b9f410700 1 -- 172.21.15.167:0/1500058409 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x56015b05ec80 con 0x56015b05e230 2023-10-05T19:28:29.089 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.082+0000 7f4b9f410700 1 --2- 172.21.15.167:0/1500058409 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x56015b05e230 0x56015b05e600 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 crypto rx=0x56015af81c60 tx=0x7f4b88012000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d65e81d6d3c5f987 server_cookie=0 in_seq=0 out_seq=0 2023-10-05T19:28:29.090 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.082+0000 7f4b70fe1700 1 -- 172.21.15.167:0/1500058409 <== mon.0 v2:172.21.15.118:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f4b88012510 con 0x56015b05e230 2023-10-05T19:28:29.090 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.082+0000 7f4b70fe1700 1 -- 172.21.15.167:0/1500058409 <== mon.0 v2:172.21.15.118:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4b8802a440 con 0x56015b05e230 2023-10-05T19:28:29.090 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.082+0000 7f4b70fe1700 1 -- 172.21.15.167:0/1500058409 <== mon.0 v2:172.21.15.118:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f4b8802a5d0 con 0x56015b05e230 2023-10-05T19:28:29.091 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.086+0000 7f4ba20dce40 1 -- 172.21.15.167:0/1500058409 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x56015b05e230 msgr2=0x56015b05e600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-05T19:28:29.091 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.086+0000 7f4ba20dce40 1 --2- 172.21.15.167:0/1500058409 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x56015b05e230 0x56015b05e600 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 crypto rx=0x56015af81c60 tx=0x7f4b88012000 comp rx=0 tx=0).stop 2023-10-05T19:28:29.091 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.086+0000 7f4ba20dce40 1 -- 172.21.15.167:0/1500058409 shutdown_connections 2023-10-05T19:28:29.091 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.086+0000 7f4ba20dce40 1 --2- 172.21.15.167:0/1500058409 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x56015b05e230 0x56015b05e600 unknown :-1 s=CLOSED pgs=276 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-05T19:28:29.091 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.086+0000 7f4ba20dce40 1 -- 172.21.15.167:0/1500058409 >> 172.21.15.167:0/1500058409 conn(0x56015b0595f0 msgr2=0x56015b0577a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-05T19:28:29.091 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.086+0000 7f4ba20dce40 1 -- 172.21.15.167:0/1500058409 shutdown_connections 2023-10-05T19:28:29.091 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.086+0000 7f4ba20dce40 1 -- 172.21.15.167:0/1500058409 wait complete. 2023-10-05T19:28:29.092 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.086+0000 7f4ba20dce40 1 Processor -- start 2023-10-05T19:28:29.092 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.086+0000 7f4ba20dce40 1 -- start start 2023-10-05T19:28:29.092 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.086+0000 7f4ba20dce40 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x56015b05e230 0x56015b063c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-05T19:28:29.092 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.086+0000 7f4ba20dce40 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x56015b064c50 con 0x56015b05e230 2023-10-05T19:28:29.092 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.086+0000 7f4b9f410700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x56015b05e230 0x56015b063c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-05T19:28:29.092 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.086+0000 7f4b9f410700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x56015b05e230 0x56015b063c50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.118:3300/0 says I am v2:172.21.15.167:57862/0 (socket says 172.21.15.167:57862) 2023-10-05T19:28:29.093 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.086+0000 7f4b9f410700 1 -- 172.21.15.167:0/1954600958 learned_addr learned my addr 172.21.15.167:0/1954600958 (peer_addr_for_me v2:172.21.15.167:0/0) 2023-10-05T19:28:29.093 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.086+0000 7f4b9f410700 1 -- 172.21.15.167:0/1954600958 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x56015b015c20 con 0x56015b05e230 2023-10-05T19:28:29.094 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.086+0000 7f4b9f410700 1 --2- 172.21.15.167:0/1954600958 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x56015b05e230 0x56015b063c50 secure :-1 s=READY pgs=277 cs=0 l=1 rev1=1 crypto rx=0x7f4b88037040 tx=0x7f4b8802f000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-05T19:28:29.094 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.086+0000 7f4aeaffd700 1 -- 172.21.15.167:0/1954600958 <== mon.0 v2:172.21.15.118:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f4b88012510 con 0x56015b05e230 2023-10-05T19:28:29.094 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.086+0000 7f4aeaffd700 1 -- 172.21.15.167:0/1954600958 <== mon.0 v2:172.21.15.118:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4b8803eb20 con 0x56015b05e230 2023-10-05T19:28:29.094 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.086+0000 7f4ba20dce40 1 -- 172.21.15.167:0/1954600958 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x56015b064250 con 0x56015b05e230 2023-10-05T19:28:29.094 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.086+0000 7f4aeaffd700 1 -- 172.21.15.167:0/1954600958 <== mon.0 v2:172.21.15.118:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f4b8803ecd0 con 0x56015b05e230 2023-10-05T19:28:29.094 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.086+0000 7f4ba20dce40 1 -- 172.21.15.167:0/1954600958 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x56015b064690 con 0x56015b05e230 2023-10-05T19:28:29.095 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.090+0000 7f4aeaffd700 1 -- 172.21.15.167:0/1954600958 <== mon.0 v2:172.21.15.118:3300/0 4 ==== mgrmap(e 5) v1 ==== 40329+0+0 (secure 0 0 0) 0x7f4b88051440 con 0x56015b05e230 2023-10-05T19:28:29.095 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.090+0000 7f4aeaffd700 1 --2- 172.21.15.167:0/1954600958 >> [v2:172.21.15.167:6824/29889,v1:172.21.15.167:6825/29889] conn(0x7f4ae0030c50 0x7f4ae0033100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-05T19:28:29.095 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.090+0000 7f4aeaffd700 1 -- 172.21.15.167:0/1954600958 <== mon.0 v2:172.21.15.118:3300/0 5 ==== osd_map(30..30 src has 1..30) v4 ==== 6057+0+0 (secure 0 0 0) 0x7f4b8802f070 con 0x56015b05e230 2023-10-05T19:28:29.095 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.090+0000 7f4ba20dce40 5 note: GC not initialized 2023-10-05T19:28:29.096 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.090+0000 7f4ba0412700 1 --2- 172.21.15.167:0/1954600958 >> [v2:172.21.15.167:6824/29889,v1:172.21.15.167:6825/29889] conn(0x7f4ae0030c50 0x7f4ae0033100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-05T19:28:29.096 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.090+0000 7f4ba0412700 1 --2- 172.21.15.167:0/1954600958 >> [v2:172.21.15.167:6824/29889,v1:172.21.15.167:6825/29889] conn(0x7f4ae0030c50 0x7f4ae0033100 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x56015b04fda0 tx=0x7f4b98000f90 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-05T19:28:29.098 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.094+0000 7f4adffff700 20 reqs_thread_entry: start 2023-10-05T19:28:29.099 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.094+0000 7f4addffb700 20 reqs_thread_entry: start 2023-10-05T19:28:29.099 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.094+0000 7f4ba20dce40 20 init_complete bucket index max shards: 11 2023-10-05T19:28:29.099 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.094+0000 7f4ba20dce40 10 rgw notify: Started notification manager with: 1 workers 2023-10-05T19:28:29.099 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.094+0000 7f4acbfff700 20 rgw notify: INFO: next queues processing will happen at: Thu Oct 5 19:28:59 2023 2023-10-05T19:28:29.099 INFO:teuthology.orchestra.run.smithi167.stderr: 2023-10-05T19:28:29.099 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.094+0000 7f4acb7fe700 20 reqs_thread_entry: start 2023-10-05T19:28:29.099 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.094+0000 7f4ba20dce40 20 get_system_obj_state: rctx=0x7ffe322a4738 obj=default.rgw.meta:users.uid:foo state=0x56015aee70b0 s->prefetch_data=0 2023-10-05T19:28:29.099 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.094+0000 7f4ba20dce40 1 -- 172.21.15.167:0/4265138048 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:102 8.18 8:1e6ca5ed:users.uid::foo:head [call version.read in=11b,getxattrs,stat] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x56015b0cc460 con 0x56015b03fd60 2023-10-05T19:28:29.100 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.094+0000 7f4ba0412700 1 -- 172.21.15.167:0/4265138048 <== osd.1 v2:172.21.15.167:6800/30035 35 ==== osd_op_reply(102 foo [call out=48b,getxattrs out=75b,stat out=16b] v0'0 uv8 ondisk = 0) v8 ==== 231+0+139 (crc 0 0 0) 0x7f4b98050040 con 0x56015b03fd60 2023-10-05T19:28:29.100 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.094+0000 7f4ba20dce40 20 get_system_obj_state: s->obj_tag was set empty 2023-10-05T19:28:29.100 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.094+0000 7f4ba20dce40 20 Read xattr: user.rgw.idtag 2023-10-05T19:28:29.100 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.094+0000 7f4ba20dce40 20 rados->read ofs=0 len=0 2023-10-05T19:28:29.101 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.094+0000 7f4ba20dce40 1 -- 172.21.15.167:0/4265138048 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:103 8.18 8:1e6ca5ed:users.uid::foo:head [call version.check_conds in=122b,call version.read in=11b,read 0~0,getxattrs] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x56015b0ceef0 con 0x56015b03fd60 2023-10-05T19:28:29.101 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.094+0000 7f4ba0412700 1 -- 172.21.15.167:0/4265138048 <== osd.1 v2:172.21.15.167:6800/30035 36 ==== osd_op_reply(103 foo [call,call out=48b,read 0~621 out=621b,getxattrs out=75b] v0'0 uv8 ondisk = 0) v8 ==== 273+0+744 (crc 0 0 0) 0x7f4b98050040 con 0x56015b03fd60 2023-10-05T19:28:29.101 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.094+0000 7f4ba20dce40 20 rados_obj.operate() r=0 bl.length=621 2023-10-05T19:28:29.102 INFO:teuthology.orchestra.run.smithi167.stdout:{ 2023-10-05T19:28:29.102 INFO:teuthology.orchestra.run.smithi167.stdout: "user_id": "foo", 2023-10-05T19:28:29.102 INFO:teuthology.orchestra.run.smithi167.stdout: "display_name": "Foo", 2023-10-05T19:28:29.102 INFO:teuthology.orchestra.run.smithi167.stdout: "email": "foo@foo.com", 2023-10-05T19:28:29.102 INFO:teuthology.orchestra.run.smithi167.stdout: "suspended": 0, 2023-10-05T19:28:29.102 INFO:teuthology.orchestra.run.smithi167.stdout: "max_buckets": 4, 2023-10-05T19:28:29.102 INFO:teuthology.orchestra.run.smithi167.stdout: "subusers": [ 2023-10-05T19:28:29.102 INFO:teuthology.orchestra.run.smithi167.stdout: { 2023-10-05T19:28:29.102 INFO:teuthology.orchestra.run.smithi167.stdout: "id": "foo:foo1", 2023-10-05T19:28:29.102 INFO:teuthology.orchestra.run.smithi167.stdout: "permissions": "full-control" 2023-10-05T19:28:29.102 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2023-10-05T19:28:29.103 INFO:teuthology.orchestra.run.smithi167.stdout: { 2023-10-05T19:28:29.103 INFO:teuthology.orchestra.run.smithi167.stdout: "id": "foo:foo2", 2023-10-05T19:28:29.103 INFO:teuthology.orchestra.run.smithi167.stdout: "permissions": "" 2023-10-05T19:28:29.103 INFO:teuthology.orchestra.run.smithi167.stdout: } 2023-10-05T19:28:29.103 INFO:teuthology.orchestra.run.smithi167.stdout: ], 2023-10-05T19:28:29.103 INFO:teuthology.orchestra.run.smithi167.stdout: "keys": [ 2023-10-05T19:28:29.103 INFO:teuthology.orchestra.run.smithi167.stdout: { 2023-10-05T19:28:29.103 INFO:teuthology.orchestra.run.smithi167.stdout: "user": "foo", 2023-10-05T19:28:29.103 INFO:teuthology.orchestra.run.smithi167.stdout: "access_key": "9te6NH5mcdcq0Tc5i8i1", 2023-10-05T19:28:29.103 INFO:teuthology.orchestra.run.smithi167.stdout: "secret_key": "Ny4IOauQoL18Gp2zM7lC1vLmoawgqcYP/YGcWfXu" 2023-10-05T19:28:29.103 INFO:teuthology.orchestra.run.smithi167.stdout: } 2023-10-05T19:28:29.103 INFO:teuthology.orchestra.run.smithi167.stdout: ], 2023-10-05T19:28:29.104 INFO:teuthology.orchestra.run.smithi167.stdout: "swift_keys": [ 2023-10-05T19:28:29.104 INFO:teuthology.orchestra.run.smithi167.stdout: { 2023-10-05T19:28:29.104 INFO:teuthology.orchestra.run.smithi167.stdout: "user": "foo:foo1", 2023-10-05T19:28:29.104 INFO:teuthology.orchestra.run.smithi167.stdout: "secret_key": "gpS2G9RREMrnbqlp29PP2D36kgPR1tm72n5fPYfL" 2023-10-05T19:28:29.104 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2023-10-05T19:28:29.104 INFO:teuthology.orchestra.run.smithi167.stdout: { 2023-10-05T19:28:29.104 INFO:teuthology.orchestra.run.smithi167.stdout: "user": "foo:foo2", 2023-10-05T19:28:29.104 INFO:teuthology.orchestra.run.smithi167.stdout: "secret_key": "ri2VJQcKSYATOY6uaDUX7pxgkW+W1YmC6OCxPHwy" 2023-10-05T19:28:29.104 INFO:teuthology.orchestra.run.smithi167.stdout: } 2023-10-05T19:28:29.104 INFO:teuthology.orchestra.run.smithi167.stdout: ], 2023-10-05T19:28:29.104 INFO:teuthology.orchestra.run.smithi167.stdout: "caps": [], 2023-10-05T19:28:29.104 INFO:teuthology.orchestra.run.smithi167.stdout: "op_mask": "read, write, delete", 2023-10-05T19:28:29.105 INFO:teuthology.orchestra.run.smithi167.stdout: "default_placement": "", 2023-10-05T19:28:29.105 INFO:teuthology.orchestra.run.smithi167.stdout: "default_storage_class": "", 2023-10-05T19:28:29.105 INFO:teuthology.orchestra.run.smithi167.stdout: "placement_tags": [], 2023-10-05T19:28:29.105 INFO:teuthology.orchestra.run.smithi167.stdout: "bucket_quota": { 2023-10-05T19:28:29.105 INFO:teuthology.orchestra.run.smithi167.stdout: "enabled": false, 2023-10-05T19:28:29.105 INFO:teuthology.orchestra.run.smithi167.stdout: "check_on_raw": false, 2023-10-05T19:28:29.105 INFO:teuthology.orchestra.run.smithi167.stdout: "max_size": -1, 2023-10-05T19:28:29.105 INFO:teuthology.orchestra.run.smithi167.stdout: "max_size_kb": 0, 2023-10-05T19:28:29.105 INFO:teuthology.orchestra.run.smithi167.stdout: "max_objects": -1 2023-10-05T19:28:29.105 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2023-10-05T19:28:29.105 INFO:teuthology.orchestra.run.smithi167.stdout: "user_quota": { 2023-10-05T19:28:29.105 INFO:teuthology.orchestra.run.smithi167.stdout: "enabled": false, 2023-10-05T19:28:29.106 INFO:teuthology.orchestra.run.smithi167.stdout: "check_on_raw": false, 2023-10-05T19:28:29.106 INFO:teuthology.orchestra.run.smithi167.stdout: "max_size": -1, 2023-10-05T19:28:29.106 INFO:teuthology.orchestra.run.smithi167.stdout: "max_size_kb": 0, 2023-10-05T19:28:29.106 INFO:teuthology.orchestra.run.smithi167.stdout: "max_objects": -1 2023-10-05T19:28:29.106 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2023-10-05T19:28:29.106 INFO:teuthology.orchestra.run.smithi167.stdout: "temp_url_keys": [], 2023-10-05T19:28:29.106 INFO:teuthology.orchestra.run.smithi167.stdout: "type": "rgw", 2023-10-05T19:28:29.106 INFO:teuthology.orchestra.run.smithi167.stdout: "mfa_ids": [] 2023-10-05T19:28:29.106 INFO:teuthology.orchestra.run.smithi167.stdout:} 2023-10-05T19:28:29.106 INFO:teuthology.orchestra.run.smithi167.stdout: 2023-10-05T19:28:29.107 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.094+0000 7f4ba20dce40 1 -- 172.21.15.167:0/4265138048 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:104 7.1f 7:f95f44c2:::notify.0:head [watch unwatch cookie 94563821988368] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x56015b0236b0 con 0x56015af823d0 2023-10-05T19:28:29.107 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.098+0000 7f4b9f410700 1 -- 172.21.15.167:0/4265138048 <== osd.2 v2:172.21.15.167:6816/30037 17 ==== osd_op_reply(104 notify.0 [watch unwatch cookie 94563821988368] v30'71 uv69 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f4b88032040 con 0x56015af823d0 2023-10-05T19:28:29.107 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.098+0000 7f4ba20dce40 20 remove_watcher() i=0 2023-10-05T19:28:29.108 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.098+0000 7f4ba20dce40 2 removed watcher, disabling cache 2023-10-05T19:28:29.108 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.098+0000 7f4ba20dce40 1 -- 172.21.15.167:0/4265138048 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:105 7.0 7:05bf5b68:::notify.1:head [watch unwatch cookie 94563821993296] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x56015b04aa10 con 0x56015b03fd60 2023-10-05T19:28:29.108 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.102+0000 7f4ba0412700 1 -- 172.21.15.167:0/4265138048 <== osd.1 v2:172.21.15.167:6800/30035 37 ==== osd_op_reply(105 notify.1 [watch unwatch cookie 94563821993296] v30'71 uv69 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f4b98050040 con 0x56015b03fd60 2023-10-05T19:28:29.108 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.102+0000 7f4ba20dce40 20 remove_watcher() i=1 2023-10-05T19:28:29.108 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.102+0000 7f4ba20dce40 1 -- 172.21.15.167:0/4265138048 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:106 7.15 7:a93a5511:::notify.2:head [watch unwatch cookie 94563821995296] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x56015b04aa10 con 0x56015b045c00 2023-10-05T19:28:29.109 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.102+0000 7f4b9fc11700 1 -- 172.21.15.167:0/4265138048 <== osd.3 v2:172.21.15.167:6808/30036 24 ==== osd_op_reply(106 notify.2 [watch unwatch cookie 94563821995296] v30'71 uv69 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f4b90025020 con 0x56015b045c00 2023-10-05T19:28:29.109 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.102+0000 7f4ba20dce40 20 remove_watcher() i=2 2023-10-05T19:28:29.110 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.102+0000 7f4ba20dce40 1 -- 172.21.15.167:0/4265138048 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:107 7.e 7:7759931f:::notify.3:head [watch unwatch cookie 94563821999712] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x56015b04c520 con 0x56015af823d0 2023-10-05T19:28:29.112 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.106+0000 7f4b9f410700 1 -- 172.21.15.167:0/4265138048 <== osd.2 v2:172.21.15.167:6816/30037 18 ==== osd_op_reply(107 notify.3 [watch unwatch cookie 94563821999712] v30'71 uv69 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f4b88032040 con 0x56015af823d0 2023-10-05T19:28:29.112 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.106+0000 7f4ba20dce40 20 remove_watcher() i=3 2023-10-05T19:28:29.112 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.106+0000 7f4ba20dce40 1 -- 172.21.15.167:0/4265138048 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:108 7.d 7:b4812045:::notify.4:head [watch unwatch cookie 94563822001920] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x56015b04c520 con 0x56015b03fd60 2023-10-05T19:28:29.114 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.110+0000 7f4ba0412700 1 -- 172.21.15.167:0/4265138048 <== osd.1 v2:172.21.15.167:6800/30035 38 ==== osd_op_reply(108 notify.4 [watch unwatch cookie 94563822001920] v30'71 uv69 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f4b98050040 con 0x56015b03fd60 2023-10-05T19:28:29.115 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.110+0000 7f4ba20dce40 20 remove_watcher() i=4 2023-10-05T19:28:29.115 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.110+0000 7f4ba20dce40 1 -- 172.21.15.167:0/4265138048 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:109 7.3 7:c609908c:::notify.5:head [watch unwatch cookie 94563822007040] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x56015b04c520 con 0x56015b0122f0 2023-10-05T19:28:29.117 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.110+0000 7f4ba0412700 1 -- 172.21.15.167:0/4265138048 <== osd.0 v2:172.21.15.118:6800/27690 29 ==== osd_op_reply(109 notify.5 [watch unwatch cookie 94563822007040] v30'71 uv69 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f4b98050040 con 0x56015b0122f0 2023-10-05T19:28:29.117 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.110+0000 7f4ba20dce40 20 remove_watcher() i=5 2023-10-05T19:28:29.117 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.110+0000 7f4ba20dce40 1 -- 172.21.15.167:0/4265138048 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:110 7.14 7:2b04a3e9:::notify.6:head [watch unwatch cookie 94563822008144] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x56015b04c520 con 0x56015b0122f0 2023-10-05T19:28:29.120 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.114+0000 7f4ba0412700 1 -- 172.21.15.167:0/4265138048 <== osd.0 v2:172.21.15.118:6800/27690 30 ==== osd_op_reply(110 notify.6 [watch unwatch cookie 94563822008144] v30'71 uv69 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f4b98050040 con 0x56015b0122f0 2023-10-05T19:28:29.120 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.114+0000 7f4ba20dce40 20 remove_watcher() i=6 2023-10-05T19:28:29.120 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.114+0000 7f4ba20dce40 1 -- 172.21.15.167:0/4265138048 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:111 7.9 7:93e5b521:::notify.7:head [watch unwatch cookie 94563822010816] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x56015b04c520 con 0x56015b0122f0 2023-10-05T19:28:29.122 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.118+0000 7f4ba0412700 1 -- 172.21.15.167:0/4265138048 <== osd.0 v2:172.21.15.118:6800/27690 31 ==== osd_op_reply(111 notify.7 [watch unwatch cookie 94563822010816] v30'71 uv69 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f4b98050040 con 0x56015b0122f0 2023-10-05T19:28:29.122 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.118+0000 7f4ba20dce40 20 remove_watcher() i=7 2023-10-05T19:28:29.124 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.118+0000 7f4b717e2700 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2023-10-05T19:28:29.124 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.118+0000 7f4ba20dce40 1 -- 172.21.15.167:0/4265138048 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:112 6.4 6:22d26bf9:::data_loggenerations_metadata:head [watch unwatch cookie 94563822012288] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x56015b04c520 con 0x56015b03fd60 2023-10-05T19:28:29.128 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.122+0000 7f4ba0412700 1 -- 172.21.15.167:0/4265138048 <== osd.1 v2:172.21.15.167:6800/30035 39 ==== osd_op_reply(112 data_loggenerations_metadata [watch unwatch cookie 94563822012288] v30'81 uv1 ondisk = 0) v8 ==== 172+0+0 (crc 0 0 0) 0x7f4b98050040 con 0x56015b03fd60 2023-10-05T19:28:29.128 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.122+0000 7f4ba20dce40 1 -- 172.21.15.167:0/4265138048 >> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] conn(0x56015b0122f0 msgr2=0x7f4b8801c410 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-05T19:28:29.128 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.122+0000 7f4ba20dce40 1 --2- 172.21.15.167:0/4265138048 >> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] conn(0x56015b0122f0 0x7f4b8801c410 crc :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-05T19:28:29.128 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.122+0000 7f4ba20dce40 1 -- 172.21.15.167:0/4265138048 >> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] conn(0x56015b03fd60 msgr2=0x56015b042140 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-05T19:28:29.128 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.122+0000 7f4ba20dce40 1 --2- 172.21.15.167:0/4265138048 >> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] conn(0x56015b03fd60 0x56015b042140 crc :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-05T19:28:29.128 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.122+0000 7f4ba20dce40 1 -- 172.21.15.167:0/4265138048 >> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] conn(0x56015af823d0 msgr2=0x7f4b9001e590 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-05T19:28:29.128 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.122+0000 7f4ba20dce40 1 --2- 172.21.15.167:0/4265138048 >> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] conn(0x56015af823d0 0x7f4b9001e590 crc :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-05T19:28:29.128 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.122+0000 7f4ba20dce40 1 -- 172.21.15.167:0/4265138048 >> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] conn(0x56015b045c00 msgr2=0x56015b047fe0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-05T19:28:29.128 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.122+0000 7f4ba20dce40 1 --2- 172.21.15.167:0/4265138048 >> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] conn(0x56015b045c00 0x56015b047fe0 crc :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-05T19:28:29.129 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.122+0000 7f4ba20dce40 1 -- 172.21.15.167:0/4265138048 >> [v2:172.21.15.167:6824/29889,v1:172.21.15.167:6825/29889] conn(0x7f4b80030c00 msgr2=0x7f4b800330b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-05T19:28:29.129 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.122+0000 7f4ba20dce40 1 --2- 172.21.15.167:0/4265138048 >> [v2:172.21.15.167:6824/29889,v1:172.21.15.167:6825/29889] conn(0x7f4b80030c00 0x7f4b800330b0 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x56015af81c30 tx=0x7f4b88015000 comp rx=0 tx=0).stop 2023-10-05T19:28:29.129 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.122+0000 7f4ba20dce40 1 -- 172.21.15.167:0/4265138048 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x56015b035f90 msgr2=0x56015b036360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-05T19:28:29.129 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.122+0000 7f4ba0412700 1 -- 172.21.15.167:0/4265138048 reap_dead start 2023-10-05T19:28:29.129 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.122+0000 7f4ba20dce40 1 --2- 172.21.15.167:0/4265138048 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x56015b035f90 0x56015b036360 secure :-1 s=READY pgs=275 cs=0 l=1 rev1=1 crypto rx=0x7f4b90001560 tx=0x7f4b90014000 comp rx=0 tx=0).stop 2023-10-05T19:28:29.129 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.122+0000 7f4ba20dce40 1 -- 172.21.15.167:0/4265138048 shutdown_connections 2023-10-05T19:28:29.129 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.122+0000 7f4ba20dce40 1 --2- 172.21.15.167:0/4265138048 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x56015b035f90 0x56015b036360 unknown :-1 s=CLOSED pgs=275 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-05T19:28:29.129 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.122+0000 7f4ba20dce40 1 -- 172.21.15.167:0/4265138048 >> 172.21.15.167:0/4265138048 conn(0x56015b008720 msgr2=0x56015b0274b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-05T19:28:29.129 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.122+0000 7f4ba20dce40 1 -- 172.21.15.167:0/4265138048 shutdown_connections 2023-10-05T19:28:29.129 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.122+0000 7f4ba20dce40 1 -- 172.21.15.167:0/4265138048 wait complete. 2023-10-05T19:28:29.130 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.122+0000 7f4ba20dce40 1 -- 172.21.15.167:0/1954600958 >> [v2:172.21.15.167:6824/29889,v1:172.21.15.167:6825/29889] conn(0x7f4ae0030c50 msgr2=0x7f4ae0033100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-05T19:28:29.130 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.122+0000 7f4ba20dce40 1 --2- 172.21.15.167:0/1954600958 >> [v2:172.21.15.167:6824/29889,v1:172.21.15.167:6825/29889] conn(0x7f4ae0030c50 0x7f4ae0033100 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x56015b04fda0 tx=0x7f4b98000f90 comp rx=0 tx=0).stop 2023-10-05T19:28:29.131 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.122+0000 7f4ba20dce40 1 -- 172.21.15.167:0/1954600958 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x56015b05e230 msgr2=0x56015b063c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-05T19:28:29.131 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.122+0000 7f4ba20dce40 1 --2- 172.21.15.167:0/1954600958 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x56015b05e230 0x56015b063c50 secure :-1 s=READY pgs=277 cs=0 l=1 rev1=1 crypto rx=0x7f4b88037040 tx=0x7f4b8802f000 comp rx=0 tx=0).stop 2023-10-05T19:28:29.131 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.122+0000 7f4ba20dce40 1 -- 172.21.15.167:0/1954600958 shutdown_connections 2023-10-05T19:28:29.131 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.122+0000 7f4ba20dce40 1 --2- 172.21.15.167:0/1954600958 >> [v2:172.21.15.167:6824/29889,v1:172.21.15.167:6825/29889] conn(0x7f4ae0030c50 0x7f4ae0033100 unknown :-1 s=CLOSED pgs=99 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-05T19:28:29.131 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.122+0000 7f4ba20dce40 1 --2- 172.21.15.167:0/1954600958 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x56015b05e230 0x56015b063c50 unknown :-1 s=CLOSED pgs=277 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-05T19:28:29.131 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.122+0000 7f4ba20dce40 1 -- 172.21.15.167:0/1954600958 >> 172.21.15.167:0/1954600958 conn(0x56015b0595f0 msgr2=0x56015b0576b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-05T19:28:29.131 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.122+0000 7f4ba20dce40 1 -- 172.21.15.167:0/1954600958 shutdown_connections 2023-10-05T19:28:29.131 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.122+0000 7f4ba20dce40 1 -- 172.21.15.167:0/1954600958 wait complete. 2023-10-05T19:28:29.135 DEBUG:tasks.util.rgw: json result: {'user_id': 'foo', 'display_name': 'Foo', 'email': 'foo@foo.com', 'suspended': 0, 'max_buckets': 4, 'subusers': [{'id': 'foo:foo1', 'permissions': 'full-control'}, {'id': 'foo:foo2', 'permissions': ''}], 'keys': [{'user': 'foo', 'access_key': '9te6NH5mcdcq0Tc5i8i1', 'secret_key': 'Ny4IOauQoL18Gp2zM7lC1vLmoawgqcYP/YGcWfXu'}], 'swift_keys': [{'user': 'foo:foo1', 'secret_key': 'gpS2G9RREMrnbqlp29PP2D36kgPR1tm72n5fPYfL'}, {'user': 'foo:foo2', 'secret_key': 'ri2VJQcKSYATOY6uaDUX7pxgkW+W1YmC6OCxPHwy'}], 'caps': [], 'op_mask': 'read, write, delete', 'default_placement': '', 'default_storage_class': '', 'placement_tags': [], 'bucket_quota': {'enabled': False, 'check_on_raw': False, 'max_size': -1, 'max_size_kb': 0, 'max_objects': -1}, 'user_quota': {'enabled': False, 'check_on_raw': False, 'max_size': -1, 'max_size_kb': 0, 'max_objects': -1}, 'temp_url_keys': [], 'type': 'rgw', 'mfa_ids': []} 2023-10-05T19:28:29.135 INFO:tasks.util.rgw:rgwadmin: client.0 : ['key', 'rm', '--subuser', 'foo:foo1', '--key-type', 'swift'] 2023-10-05T19:28:29.135 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'ceph', 'key', 'rm', '--subuser', 'foo:foo1', '--key-type', 'swift'] 2023-10-05T19:28:29.135 DEBUG:teuthology.orchestra.run.smithi167:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster ceph key rm --subuser foo:foo1 --key-type swift 2023-10-05T19:28:29.177 INFO:teuthology.orchestra.run.smithi167.stderr:ignoring --setuser ceph since I am not root 2023-10-05T19:28:29.177 INFO:teuthology.orchestra.run.smithi167.stderr:ignoring --setgroup ceph since I am not root 2023-10-05T19:28:29.178 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.170+0000 7f487c2d9e40 1 Processor -- start 2023-10-05T19:28:29.178 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.170+0000 7f487c2d9e40 1 -- start start 2023-10-05T19:28:29.178 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.170+0000 7f487c2d9e40 1 --2- >> v2:172.21.15.118:3300/0 conn(0x56187144ce20 0x56187144d1f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-05T19:28:29.178 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.170+0000 7f487c2d9e40 1 -- --> v1:172.21.15.118:6789/0 -- auth(proto 0 26 bytes epoch 0) v1 -- 0x561871399bd0 con 0x56187144d730 2023-10-05T19:28:29.178 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.170+0000 7f487c2d9e40 1 -- --> v2:172.21.15.118:3300/0 -- mon_getmap magic: 0 v1 -- 0x561871356ee0 con 0x56187144ce20 2023-10-05T19:28:29.179 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.174+0000 7f4879e0e700 1 --2- >> v2:172.21.15.118:3300/0 conn(0x56187144ce20 0x56187144d1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-05T19:28:29.179 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.174+0000 7f487960d700 1 --1- >> v1:172.21.15.118:6789/0 conn(0x56187144d730 0x56187144e1d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.118:6789/0 says I am v1:172.21.15.167:56498/0 (socket says 172.21.15.167:56498) 2023-10-05T19:28:29.179 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.174+0000 7f487960d700 1 -- 172.21.15.167:0/4240626718 learned_addr learned my addr 172.21.15.167:0/4240626718 (peer_addr_for_me v1:172.21.15.167:0/0) 2023-10-05T19:28:29.179 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.174+0000 7f4879e0e700 1 -- 172.21.15.167:0/4240626718 >> v1:172.21.15.118:6789/0 conn(0x56187144d730 legacy=0x56187144e1d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-05T19:28:29.179 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.174+0000 7f4878e0c700 1 -- 172.21.15.167:0/4240626718 <== mon.0 v1:172.21.15.118:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1735193259 0 0) 0x561871399bd0 con 0x56187144d730 2023-10-05T19:28:29.179 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.174+0000 7f4879e0e700 1 -- 172.21.15.167:0/4240626718 --> v2:172.21.15.118:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x56187144e8e0 con 0x56187144ce20 2023-10-05T19:28:29.180 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.174+0000 7f4879e0e700 1 --2- 172.21.15.167:0/4240626718 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x56187144ce20 0x56187144d1f0 secure :-1 s=READY pgs=279 cs=0 l=1 rev1=1 crypto rx=0x7f486c00a5d0 tx=0x7f486c005a50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2932e051717f399e server_cookie=0 in_seq=0 out_seq=0 2023-10-05T19:28:29.181 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.174+0000 7f4878e0c700 1 -- 172.21.15.167:0/4240626718 <== mon.0 v2:172.21.15.118:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f486c0086a0 con 0x56187144ce20 2023-10-05T19:28:29.181 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.174+0000 7f4878e0c700 1 -- 172.21.15.167:0/4240626718 <== mon.0 v2:172.21.15.118:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f486c008800 con 0x56187144ce20 2023-10-05T19:28:29.181 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.174+0000 7f4878e0c700 1 -- 172.21.15.167:0/4240626718 <== mon.0 v2:172.21.15.118:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f486c008ad0 con 0x56187144ce20 2023-10-05T19:28:29.181 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.174+0000 7f487c2d9e40 1 -- 172.21.15.167:0/4240626718 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x56187144ce20 msgr2=0x56187144d1f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-05T19:28:29.181 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.174+0000 7f487c2d9e40 1 --2- 172.21.15.167:0/4240626718 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x56187144ce20 0x56187144d1f0 secure :-1 s=READY pgs=279 cs=0 l=1 rev1=1 crypto rx=0x7f486c00a5d0 tx=0x7f486c005a50 comp rx=0 tx=0).stop 2023-10-05T19:28:29.181 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.174+0000 7f487c2d9e40 1 -- 172.21.15.167:0/4240626718 shutdown_connections 2023-10-05T19:28:29.181 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.174+0000 7f487c2d9e40 1 --2- 172.21.15.167:0/4240626718 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x56187144ce20 0x56187144d1f0 unknown :-1 s=CLOSED pgs=279 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-05T19:28:29.181 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.174+0000 7f487c2d9e40 1 -- 172.21.15.167:0/4240626718 >> 172.21.15.167:0/4240626718 conn(0x561871448840 msgr2=0x56187144ac50 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-05T19:28:29.181 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.174+0000 7f487c2d9e40 1 -- 172.21.15.167:0/4240626718 shutdown_connections 2023-10-05T19:28:29.181 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.174+0000 7f487c2d9e40 1 -- 172.21.15.167:0/4240626718 wait complete. 2023-10-05T19:28:29.182 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.174+0000 7f487c2d9e40 1 Processor -- start 2023-10-05T19:28:29.182 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.174+0000 7f487c2d9e40 1 -- start start 2023-10-05T19:28:29.183 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.174+0000 7f487c2d9e40 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x56187144ce20 0x5618713c3180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-05T19:28:29.183 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.174+0000 7f487c2d9e40 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x56187131ae30 con 0x56187144ce20 2023-10-05T19:28:29.183 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.178+0000 7f4879e0e700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x56187144ce20 0x5618713c3180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-05T19:28:29.183 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.178+0000 7f4879e0e700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x56187144ce20 0x5618713c3180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.118:3300/0 says I am v2:172.21.15.167:57870/0 (socket says 172.21.15.167:57870) 2023-10-05T19:28:29.183 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.178+0000 7f4879e0e700 1 -- 172.21.15.167:0/2735935100 learned_addr learned my addr 172.21.15.167:0/2735935100 (peer_addr_for_me v2:172.21.15.167:0/0) 2023-10-05T19:28:29.184 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.178+0000 7f4879e0e700 1 -- 172.21.15.167:0/2735935100 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f486c005cf0 con 0x56187144ce20 2023-10-05T19:28:29.184 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.178+0000 7f4879e0e700 1 --2- 172.21.15.167:0/2735935100 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x56187144ce20 0x5618713c3180 secure :-1 s=READY pgs=280 cs=0 l=1 rev1=1 crypto rx=0x7f486c005a90 tx=0x7f486c015000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6721df4c25102200 server_cookie=0 in_seq=0 out_seq=0 2023-10-05T19:28:29.184 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.178+0000 7f48737fe700 1 -- 172.21.15.167:0/2735935100 <== mon.0 v2:172.21.15.118:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f486c008a40 con 0x56187144ce20 2023-10-05T19:28:29.184 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.178+0000 7f48737fe700 1 -- 172.21.15.167:0/2735935100 <== mon.0 v2:172.21.15.118:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f486c008ba0 con 0x56187144ce20 2023-10-05T19:28:29.184 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.178+0000 7f48737fe700 1 -- 172.21.15.167:0/2735935100 <== mon.0 v2:172.21.15.118:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f486c008e70 con 0x56187144ce20 2023-10-05T19:28:29.185 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.178+0000 7f487c2d9e40 1 -- 172.21.15.167:0/2735935100 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x56187144ce20 msgr2=0x5618713c3180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-05T19:28:29.185 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.178+0000 7f487c2d9e40 1 --2- 172.21.15.167:0/2735935100 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x56187144ce20 0x5618713c3180 secure :-1 s=READY pgs=280 cs=0 l=1 rev1=1 crypto rx=0x7f486c005a90 tx=0x7f486c015000 comp rx=0 tx=0).stop 2023-10-05T19:28:29.185 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.178+0000 7f487c2d9e40 1 -- 172.21.15.167:0/2735935100 shutdown_connections 2023-10-05T19:28:29.185 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.178+0000 7f487c2d9e40 1 --2- 172.21.15.167:0/2735935100 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x56187144ce20 0x5618713c3180 unknown :-1 s=CLOSED pgs=280 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-05T19:28:29.186 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.178+0000 7f487c2d9e40 1 -- 172.21.15.167:0/2735935100 >> 172.21.15.167:0/2735935100 conn(0x561871448840 msgr2=0x5618714678b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-05T19:28:29.186 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.178+0000 7f487c2d9e40 1 -- 172.21.15.167:0/2735935100 shutdown_connections 2023-10-05T19:28:29.186 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.178+0000 7f487c2d9e40 1 -- 172.21.15.167:0/2735935100 wait complete. 2023-10-05T19:28:29.186 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.182+0000 7f487c2d9e40 1 Processor -- start 2023-10-05T19:28:29.187 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.182+0000 7f487c2d9e40 1 -- start start 2023-10-05T19:28:29.187 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.182+0000 7f487c2d9e40 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x56187144ce20 0x561871474b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-05T19:28:29.187 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.182+0000 7f487c2d9e40 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x5618712eded0 con 0x56187144ce20 2023-10-05T19:28:29.187 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.182+0000 7f4879e0e700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x56187144ce20 0x561871474b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-05T19:28:29.188 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.182+0000 7f4879e0e700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x56187144ce20 0x561871474b20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.118:3300/0 says I am v2:172.21.15.167:57880/0 (socket says 172.21.15.167:57880) 2023-10-05T19:28:29.188 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.182+0000 7f4879e0e700 1 -- 172.21.15.167:0/2494787467 learned_addr learned my addr 172.21.15.167:0/2494787467 (peer_addr_for_me v2:172.21.15.167:0/0) 2023-10-05T19:28:29.188 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.182+0000 7f4879e0e700 1 -- 172.21.15.167:0/2494787467 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x56187144e8e0 con 0x56187144ce20 2023-10-05T19:28:29.189 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.182+0000 7f4879e0e700 1 --2- 172.21.15.167:0/2494787467 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x56187144ce20 0x561871474b20 secure :-1 s=READY pgs=281 cs=0 l=1 rev1=1 crypto rx=0x7f486c01e040 tx=0x7f486c003050 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-05T19:28:29.189 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.182+0000 7f4871ffb700 1 -- 172.21.15.167:0/2494787467 <== mon.0 v2:172.21.15.118:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f486c012b50 con 0x56187144ce20 2023-10-05T19:28:29.189 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.182+0000 7f4871ffb700 1 -- 172.21.15.167:0/2494787467 <== mon.0 v2:172.21.15.118:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f486c02a070 con 0x56187144ce20 2023-10-05T19:28:29.189 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.182+0000 7f4871ffb700 1 -- 172.21.15.167:0/2494787467 <== mon.0 v2:172.21.15.118:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f486c012e10 con 0x56187144ce20 2023-10-05T19:28:29.189 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.182+0000 7f487c2d9e40 1 -- 172.21.15.167:0/2494787467 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x561871475060 con 0x56187144ce20 2023-10-05T19:28:29.189 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.182+0000 7f487c2d9e40 1 -- 172.21.15.167:0/2494787467 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x561871475500 con 0x56187144ce20 2023-10-05T19:28:29.191 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.186+0000 7f4871ffb700 1 -- 172.21.15.167:0/2494787467 <== mon.0 v2:172.21.15.118:3300/0 4 ==== mgrmap(e 5) v1 ==== 40329+0+0 (secure 0 0 0) 0x7f486c019070 con 0x56187144ce20 2023-10-05T19:28:29.191 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.186+0000 7f4871ffb700 1 --2- 172.21.15.167:0/2494787467 >> [v2:172.21.15.167:6824/29889,v1:172.21.15.167:6825/29889] conn(0x7f4860030c00 0x7f48600330b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-05T19:28:29.191 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.186+0000 7f4871ffb700 1 -- 172.21.15.167:0/2494787467 <== mon.0 v2:172.21.15.118:3300/0 5 ==== osd_map(30..30 src has 1..30) v4 ==== 6057+0+0 (secure 0 0 0) 0x7f486c02c070 con 0x56187144ce20 2023-10-05T19:28:29.191 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.186+0000 7f487c2d9e40 20 rados->read ofs=0 len=0 2023-10-05T19:28:29.192 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.186+0000 7f487960d700 1 --2- 172.21.15.167:0/2494787467 >> [v2:172.21.15.167:6824/29889,v1:172.21.15.167:6825/29889] conn(0x7f4860030c00 0x7f48600330b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-05T19:28:29.192 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.186+0000 7f487c2d9e40 1 --2- 172.21.15.167:0/2494787467 >> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] conn(0x56187144d730 0x5618714805f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-05T19:28:29.192 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.186+0000 7f487c2d9e40 1 -- 172.21.15.167:0/2494787467 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:1 5.12 5:49953fa1:::default.realm:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x561871480cf0 con 0x56187144d730 2023-10-05T19:28:29.192 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.186+0000 7f487a60f700 1 --2- 172.21.15.167:0/2494787467 >> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] conn(0x56187144d730 0x5618714805f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-05T19:28:29.193 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.186+0000 7f487960d700 1 --2- 172.21.15.167:0/2494787467 >> [v2:172.21.15.167:6824/29889,v1:172.21.15.167:6825/29889] conn(0x7f4860030c00 0x7f48600330b0 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7f4864001560 tx=0x7f4864006650 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-05T19:28:29.193 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.186+0000 7f487a60f700 1 --2- 172.21.15.167:0/2494787467 >> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] conn(0x56187144d730 0x5618714805f0 crc :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-05T19:28:29.193 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.186+0000 7f487a60f700 1 -- 172.21.15.167:0/2494787467 <== osd.1 v2:172.21.15.167:6800/30035 1 ==== osd_op_reply(1 default.realm [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 ==== 157+0+0 (crc 0 0 0) 0x7f487400a500 con 0x56187144d730 2023-10-05T19:28:29.193 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.186+0000 7f487c2d9e40 20 rados_obj.operate() r=-2 bl.length=0 2023-10-05T19:28:29.193 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.186+0000 7f487c2d9e40 20 realm 2023-10-05T19:28:29.194 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.186+0000 7f487c2d9e40 20 rados->read ofs=0 len=0 2023-10-05T19:28:29.194 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.186+0000 7f487c2d9e40 1 -- 172.21.15.167:0/2494787467 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:2 5.12 5:49953fa1:::default.realm:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x56187147dc50 con 0x56187144d730 2023-10-05T19:28:29.195 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.186+0000 7f487a60f700 1 -- 172.21.15.167:0/2494787467 <== osd.1 v2:172.21.15.167:6800/30035 2 ==== osd_op_reply(2 default.realm [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 ==== 157+0+0 (crc 0 0 0) 0x7f487400a500 con 0x56187144d730 2023-10-05T19:28:29.195 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.186+0000 7f487c2d9e40 20 rados_obj.operate() r=-2 bl.length=0 2023-10-05T19:28:29.195 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.186+0000 7f487c2d9e40 4 RGWPeriod::init failed to init realm id : (2) No such file or directory 2023-10-05T19:28:29.195 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.186+0000 7f487c2d9e40 20 rados->read ofs=0 len=0 2023-10-05T19:28:29.195 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.186+0000 7f487c2d9e40 1 -- 172.21.15.167:0/2494787467 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:3 5.12 5:49953fa1:::default.realm:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x561871481820 con 0x56187144d730 2023-10-05T19:28:29.196 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.190+0000 7f487a60f700 1 -- 172.21.15.167:0/2494787467 <== osd.1 v2:172.21.15.167:6800/30035 3 ==== osd_op_reply(3 default.realm [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 ==== 157+0+0 (crc 0 0 0) 0x7f487400a500 con 0x56187144d730 2023-10-05T19:28:29.196 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.190+0000 7f487c2d9e40 20 rados_obj.operate() r=-2 bl.length=0 2023-10-05T19:28:29.196 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.190+0000 7f487c2d9e40 20 rados->read ofs=0 len=0 2023-10-05T19:28:29.196 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.190+0000 7f487c2d9e40 1 -- 172.21.15.167:0/2494787467 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:4 5.f 5:f43b6ece:::zone_names.default:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x561871481430 con 0x56187144d730 2023-10-05T19:28:29.196 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.190+0000 7f487a60f700 1 -- 172.21.15.167:0/2494787467 <== osd.1 v2:172.21.15.167:6800/30035 4 ==== osd_op_reply(4 zone_names.default [read 0~46 out=46b] v0'0 uv1 ondisk = 0) v8 ==== 162+0+46 (crc 0 0 0) 0x7f487400a500 con 0x56187144d730 2023-10-05T19:28:29.196 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.190+0000 7f487c2d9e40 20 rados_obj.operate() r=0 bl.length=46 2023-10-05T19:28:29.196 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.190+0000 7f487c2d9e40 20 rados->read ofs=0 len=0 2023-10-05T19:28:29.196 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.190+0000 7f487c2d9e40 1 -- 172.21.15.167:0/2494787467 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:5 5.1d 5:b8374eb4:::zone_info.9a9df965-6612-45d4-a45f-6e3c889bacc2:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x5618714826b0 con 0x56187144d730 2023-10-05T19:28:29.197 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.190+0000 7f487a60f700 1 -- 172.21.15.167:0/2494787467 <== osd.1 v2:172.21.15.167:6800/30035 5 ==== osd_op_reply(5 zone_info.9a9df965-6612-45d4-a45f-6e3c889bacc2 [read 0~874 out=874b] v0'0 uv2 ondisk = 0) v8 ==== 190+0+874 (crc 0 0 0) 0x7f487400a500 con 0x56187144d730 2023-10-05T19:28:29.197 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.190+0000 7f487c2d9e40 20 rados_obj.operate() r=0 bl.length=874 2023-10-05T19:28:29.197 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.190+0000 7f487c2d9e40 20 searching for the correct realm 2023-10-05T19:28:29.197 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.190+0000 7f487c2d9e40 1 --2- 172.21.15.167:0/2494787467 >> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] conn(0x561871484470 0x561871486850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-05T19:28:29.198 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.190+0000 7f487c2d9e40 1 -- 172.21.15.167:0/2494787467 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:6 5.0 5:00000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x561871486d90 con 0x561871484470 2023-10-05T19:28:29.198 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.190+0000 7f4879e0e700 1 --2- 172.21.15.167:0/2494787467 >> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] conn(0x561871484470 0x561871486850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-05T19:28:29.198 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.190+0000 7f4879e0e700 1 --2- 172.21.15.167:0/2494787467 >> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] conn(0x561871484470 0x561871486850 crc :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-05T19:28:29.198 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.194+0000 7f4879e0e700 1 -- 172.21.15.167:0/2494787467 <== osd.3 v2:172.21.15.167:6808/30036 1 ==== osd_op_reply(6 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f486c022020 con 0x561871484470 2023-10-05T19:28:29.198 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.194+0000 7f4879e0e700 1 --2- 172.21.15.167:0/2494787467 >> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] conn(0x7f486c007030 0x7f486c01c410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-05T19:28:29.199 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.194+0000 7f4879e0e700 1 -- 172.21.15.167:0/2494787467 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:7 5.10 5:08000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x561871486d90 con 0x7f486c007030 2023-10-05T19:28:29.199 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.194+0000 7f487960d700 1 --2- 172.21.15.167:0/2494787467 >> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] conn(0x7f486c007030 0x7f486c01c410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-05T19:28:29.199 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.194+0000 7f487960d700 1 --2- 172.21.15.167:0/2494787467 >> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] conn(0x7f486c007030 0x7f486c01c410 crc :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-05T19:28:29.199 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.194+0000 7f487960d700 1 -- 172.21.15.167:0/2494787467 <== osd.2 v2:172.21.15.167:6816/30037 1 ==== osd_op_reply(7 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f4864004520 con 0x7f486c007030 2023-10-05T19:28:29.200 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.194+0000 7f487960d700 1 --2- 172.21.15.167:0/2494787467 >> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] conn(0x7f4864011040 0x7f4864014950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-05T19:28:29.200 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.194+0000 7f487960d700 1 -- 172.21.15.167:0/2494787467 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:8 5.8 5:10000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x561871486d90 con 0x7f4864011040 2023-10-05T19:28:29.200 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.194+0000 7f487a60f700 1 --2- 172.21.15.167:0/2494787467 >> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] conn(0x7f4864011040 0x7f4864014950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-05T19:28:29.200 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.194+0000 7f487a60f700 1 --2- 172.21.15.167:0/2494787467 >> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] conn(0x7f4864011040 0x7f4864014950 crc :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-05T19:28:29.201 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.194+0000 7f487a60f700 1 -- 172.21.15.167:0/2494787467 <== osd.0 v2:172.21.15.118:6800/27690 1 ==== osd_op_reply(8 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f4874014040 con 0x7f4864011040 2023-10-05T19:28:29.201 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.194+0000 7f487a60f700 1 -- 172.21.15.167:0/2494787467 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:9 5.18 5:18000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x561871486d90 con 0x56187144d730 2023-10-05T19:28:29.202 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.198+0000 7f487a60f700 1 -- 172.21.15.167:0/2494787467 <== osd.1 v2:172.21.15.167:6800/30035 6 ==== osd_op_reply(9 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f4874014040 con 0x56187144d730 2023-10-05T19:28:29.202 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.198+0000 7f487a60f700 1 -- 172.21.15.167:0/2494787467 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:10 5.4 5:20000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x561871486d90 con 0x56187144d730 2023-10-05T19:28:29.203 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.198+0000 7f487a60f700 1 -- 172.21.15.167:0/2494787467 <== osd.1 v2:172.21.15.167:6800/30035 7 ==== osd_op_reply(10 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f4874014040 con 0x56187144d730 2023-10-05T19:28:29.203 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.198+0000 7f487a60f700 1 -- 172.21.15.167:0/2494787467 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:11 5.14 5:28000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x561871486d90 con 0x7f4864011040 2023-10-05T19:28:29.203 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.198+0000 7f487a60f700 1 -- 172.21.15.167:0/2494787467 <== osd.0 v2:172.21.15.118:6800/27690 2 ==== osd_op_reply(11 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f4874014040 con 0x7f4864011040 2023-10-05T19:28:29.203 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.198+0000 7f487a60f700 1 -- 172.21.15.167:0/2494787467 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:12 5.c 5:30000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x561871486d90 con 0x7f4864011040 2023-10-05T19:28:29.204 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.198+0000 7f487a60f700 1 -- 172.21.15.167:0/2494787467 <== osd.0 v2:172.21.15.118:6800/27690 3 ==== osd_op_reply(12 [pgnls start_epoch 30 out=112b] v23'1 uv1 ondisk = 1) v8 ==== 144+0+112 (crc 0 0 0) 0x7f4874014040 con 0x7f4864011040 2023-10-05T19:28:29.204 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.198+0000 7f487a60f700 1 -- 172.21.15.167:0/2494787467 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:13 5.1c 5:38000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x561871486d90 con 0x7f486c007030 2023-10-05T19:28:29.205 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.198+0000 7f487960d700 1 -- 172.21.15.167:0/2494787467 <== osd.2 v2:172.21.15.167:6816/30037 2 ==== osd_op_reply(13 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f4864004520 con 0x7f486c007030 2023-10-05T19:28:29.205 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.198+0000 7f487960d700 1 -- 172.21.15.167:0/2494787467 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:14 5.2 5:40000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x561871486d90 con 0x561871484470 2023-10-05T19:28:29.206 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.202+0000 7f4879e0e700 1 -- 172.21.15.167:0/2494787467 <== osd.3 v2:172.21.15.167:6808/30036 2 ==== osd_op_reply(14 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f486c022020 con 0x561871484470 2023-10-05T19:28:29.206 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.202+0000 7f4879e0e700 1 -- 172.21.15.167:0/2494787467 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:15 5.12 5:48000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x561871486d90 con 0x56187144d730 2023-10-05T19:28:29.206 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.202+0000 7f487a60f700 1 -- 172.21.15.167:0/2494787467 <== osd.1 v2:172.21.15.167:6800/30035 8 ==== osd_op_reply(15 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f4874014040 con 0x56187144d730 2023-10-05T19:28:29.207 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.202+0000 7f487a60f700 1 -- 172.21.15.167:0/2494787467 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:16 5.a 5:50000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x561871486d90 con 0x7f4864011040 2023-10-05T19:28:29.207 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.202+0000 7f487a60f700 1 -- 172.21.15.167:0/2494787467 <== osd.0 v2:172.21.15.118:6800/27690 4 ==== osd_op_reply(16 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f4874022040 con 0x7f4864011040 2023-10-05T19:28:29.207 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.202+0000 7f487a60f700 1 -- 172.21.15.167:0/2494787467 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:17 5.1a 5:58000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x561871486d90 con 0x56187144d730 2023-10-05T19:28:29.208 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.202+0000 7f487a60f700 1 -- 172.21.15.167:0/2494787467 <== osd.1 v2:172.21.15.167:6800/30035 9 ==== osd_op_reply(17 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f4874022040 con 0x56187144d730 2023-10-05T19:28:29.208 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.202+0000 7f487a60f700 1 -- 172.21.15.167:0/2494787467 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:18 5.6 5:60000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x561871486d90 con 0x7f486c007030 2023-10-05T19:28:29.209 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.202+0000 7f487960d700 1 -- 172.21.15.167:0/2494787467 <== osd.2 v2:172.21.15.167:6816/30037 3 ==== osd_op_reply(18 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f4864004520 con 0x7f486c007030 2023-10-05T19:28:29.209 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.202+0000 7f487960d700 1 -- 172.21.15.167:0/2494787467 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:19 5.16 5:68000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x561871486d90 con 0x561871484470 2023-10-05T19:28:29.209 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.202+0000 7f4879e0e700 1 -- 172.21.15.167:0/2494787467 <== osd.3 v2:172.21.15.167:6808/30036 3 ==== osd_op_reply(19 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f486c022020 con 0x561871484470 2023-10-05T19:28:29.209 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.202+0000 7f4879e0e700 1 -- 172.21.15.167:0/2494787467 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:20 5.e 5:70000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x561871486d90 con 0x561871484470 2023-10-05T19:28:29.210 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.206+0000 7f4879e0e700 1 -- 172.21.15.167:0/2494787467 <== osd.3 v2:172.21.15.167:6808/30036 4 ==== osd_op_reply(20 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f486c022020 con 0x561871484470 2023-10-05T19:28:29.210 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.206+0000 7f4879e0e700 1 -- 172.21.15.167:0/2494787467 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:21 5.1e 5:78000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x561871486d90 con 0x7f4864011040 2023-10-05T19:28:29.211 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.206+0000 7f487a60f700 1 -- 172.21.15.167:0/2494787467 <== osd.0 v2:172.21.15.118:6800/27690 5 ==== osd_op_reply(21 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f4874022040 con 0x7f4864011040 2023-10-05T19:28:29.211 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.206+0000 7f487a60f700 1 -- 172.21.15.167:0/2494787467 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:22 5.1 5:80000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x561871486d90 con 0x561871484470 2023-10-05T19:28:29.212 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.206+0000 7f4879e0e700 1 -- 172.21.15.167:0/2494787467 <== osd.3 v2:172.21.15.167:6808/30036 5 ==== osd_op_reply(22 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f486c022020 con 0x561871484470 2023-10-05T19:28:29.212 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.206+0000 7f4879e0e700 1 -- 172.21.15.167:0/2494787467 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:23 5.11 5:88000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x561871486d90 con 0x561871484470 2023-10-05T19:28:29.212 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.206+0000 7f4879e0e700 1 -- 172.21.15.167:0/2494787467 <== osd.3 v2:172.21.15.167:6808/30036 6 ==== osd_op_reply(23 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f486c022020 con 0x561871484470 2023-10-05T19:28:29.212 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.206+0000 7f4879e0e700 1 -- 172.21.15.167:0/2494787467 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:24 5.9 5:90000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x561871486d90 con 0x56187144d730 2023-10-05T19:28:29.213 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.206+0000 7f487a60f700 1 -- 172.21.15.167:0/2494787467 <== osd.1 v2:172.21.15.167:6800/30035 10 ==== osd_op_reply(24 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f4874022040 con 0x56187144d730 2023-10-05T19:28:29.213 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.206+0000 7f487a60f700 1 -- 172.21.15.167:0/2494787467 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:25 5.19 5:98000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x561871486d90 con 0x56187144d730 2023-10-05T19:28:29.214 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.210+0000 7f487a60f700 1 -- 172.21.15.167:0/2494787467 <== osd.1 v2:172.21.15.167:6800/30035 11 ==== osd_op_reply(25 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f4874022040 con 0x56187144d730 2023-10-05T19:28:29.214 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.210+0000 7f487a60f700 1 -- 172.21.15.167:0/2494787467 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:26 5.5 5:a0000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x561871486d90 con 0x7f4864011040 2023-10-05T19:28:29.215 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.210+0000 7f487a60f700 1 -- 172.21.15.167:0/2494787467 <== osd.0 v2:172.21.15.118:6800/27690 6 ==== osd_op_reply(26 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f4874022040 con 0x7f4864011040 2023-10-05T19:28:29.215 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.210+0000 7f487a60f700 1 -- 172.21.15.167:0/2494787467 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:27 5.15 5:a8000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x561871486d90 con 0x561871484470 2023-10-05T19:28:29.216 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.210+0000 7f4879e0e700 1 -- 172.21.15.167:0/2494787467 <== osd.3 v2:172.21.15.167:6808/30036 7 ==== osd_op_reply(27 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f486c022020 con 0x561871484470 2023-10-05T19:28:29.216 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.210+0000 7f4879e0e700 1 -- 172.21.15.167:0/2494787467 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:28 5.d 5:b0000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x561871486d90 con 0x7f486c007030 2023-10-05T19:28:29.217 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.210+0000 7f487960d700 1 -- 172.21.15.167:0/2494787467 <== osd.2 v2:172.21.15.167:6816/30037 4 ==== osd_op_reply(28 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f4864004520 con 0x7f486c007030 2023-10-05T19:28:29.217 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.210+0000 7f487960d700 1 -- 172.21.15.167:0/2494787467 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:29 5.1d 5:b8000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x561871486d90 con 0x56187144d730 2023-10-05T19:28:29.217 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.210+0000 7f487a60f700 1 -- 172.21.15.167:0/2494787467 <== osd.1 v2:172.21.15.167:6800/30035 12 ==== osd_op_reply(29 [pgnls start_epoch 30 out=107b] v30'2 uv2 ondisk = 1) v8 ==== 144+0+107 (crc 0 0 0) 0x7f4874022040 con 0x56187144d730 2023-10-05T19:28:29.217 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.210+0000 7f487a60f700 1 -- 172.21.15.167:0/2494787467 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:30 5.3 5:c0000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x561871486d90 con 0x7f4864011040 2023-10-05T19:28:29.218 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.214+0000 7f487a60f700 1 -- 172.21.15.167:0/2494787467 <== osd.0 v2:172.21.15.118:6800/27690 7 ==== osd_op_reply(30 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f4874022040 con 0x7f4864011040 2023-10-05T19:28:29.219 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.214+0000 7f487a60f700 1 -- 172.21.15.167:0/2494787467 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:31 5.13 5:c8000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x561871486d90 con 0x561871484470 2023-10-05T19:28:29.219 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.214+0000 7f4879e0e700 1 -- 172.21.15.167:0/2494787467 <== osd.3 v2:172.21.15.167:6808/30036 8 ==== osd_op_reply(31 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f486c022020 con 0x561871484470 2023-10-05T19:28:29.219 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.214+0000 7f4879e0e700 1 -- 172.21.15.167:0/2494787467 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:32 5.b 5:d0000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x561871486d90 con 0x7f4864011040 2023-10-05T19:28:29.220 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.214+0000 7f487a60f700 1 -- 172.21.15.167:0/2494787467 <== osd.0 v2:172.21.15.118:6800/27690 8 ==== osd_op_reply(32 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f4874022040 con 0x7f4864011040 2023-10-05T19:28:29.221 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.214+0000 7f487a60f700 1 -- 172.21.15.167:0/2494787467 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:33 5.1b 5:d8000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x561871486d90 con 0x7f486c007030 2023-10-05T19:28:29.221 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.214+0000 7f487960d700 1 -- 172.21.15.167:0/2494787467 <== osd.2 v2:172.21.15.167:6816/30037 5 ==== osd_op_reply(33 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f4864004520 con 0x7f486c007030 2023-10-05T19:28:29.221 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.214+0000 7f487960d700 1 -- 172.21.15.167:0/2494787467 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:34 5.7 5:e0000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x561871486d90 con 0x561871484470 2023-10-05T19:28:29.222 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.218+0000 7f4879e0e700 1 -- 172.21.15.167:0/2494787467 <== osd.3 v2:172.21.15.167:6808/30036 9 ==== osd_op_reply(34 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f486c022020 con 0x561871484470 2023-10-05T19:28:29.222 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.218+0000 7f4879e0e700 1 -- 172.21.15.167:0/2494787467 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:35 5.17 5:e8000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x561871486d90 con 0x7f4864011040 2023-10-05T19:28:29.223 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.218+0000 7f487a60f700 1 -- 172.21.15.167:0/2494787467 <== osd.0 v2:172.21.15.118:6800/27690 9 ==== osd_op_reply(35 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f4874022040 con 0x7f4864011040 2023-10-05T19:28:29.223 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.218+0000 7f487a60f700 1 -- 172.21.15.167:0/2494787467 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:36 5.f 5:f0000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x561871486d90 con 0x56187144d730 2023-10-05T19:28:29.224 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.218+0000 7f487a60f700 1 -- 172.21.15.167:0/2494787467 <== osd.1 v2:172.21.15.167:6800/30035 13 ==== osd_op_reply(36 [pgnls start_epoch 30 out=115b] v23'2 uv2 ondisk = 1) v8 ==== 144+0+115 (crc 0 0 0) 0x7f487402f080 con 0x56187144d730 2023-10-05T19:28:29.224 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.218+0000 7f487a60f700 1 -- 172.21.15.167:0/2494787467 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:37 5.1f 5:f8000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x561871486d90 con 0x7f486c007030 2023-10-05T19:28:29.225 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.218+0000 7f487960d700 1 -- 172.21.15.167:0/2494787467 <== osd.2 v2:172.21.15.167:6816/30037 6 ==== osd_op_reply(37 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f4864004520 con 0x7f486c007030 2023-10-05T19:28:29.225 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.218+0000 7f487c2d9e40 20 RGWRados::pool_iterate: got zonegroup_info.4f1f719d-9648-417b-b10c-e5d939825419 2023-10-05T19:28:29.225 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.218+0000 7f487c2d9e40 20 RGWRados::pool_iterate: got zone_info.9a9df965-6612-45d4-a45f-6e3c889bacc2 2023-10-05T19:28:29.225 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.218+0000 7f487c2d9e40 20 RGWRados::pool_iterate: got zone_names.default 2023-10-05T19:28:29.226 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.218+0000 7f487c2d9e40 20 RGWRados::pool_iterate: got zonegroups_names.default 2023-10-05T19:28:29.226 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.218+0000 7f487c2d9e40 20 rados->read ofs=0 len=0 2023-10-05T19:28:29.226 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.218+0000 7f487c2d9e40 1 -- 172.21.15.167:0/2494787467 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:38 5.12 5:49953fa1:::default.realm:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x561871483220 con 0x56187144d730 2023-10-05T19:28:29.227 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.218+0000 7f487a60f700 1 -- 172.21.15.167:0/2494787467 <== osd.1 v2:172.21.15.167:6800/30035 14 ==== osd_op_reply(38 default.realm [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 ==== 157+0+0 (crc 0 0 0) 0x7f487402f080 con 0x56187144d730 2023-10-05T19:28:29.227 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.218+0000 7f487c2d9e40 20 rados_obj.operate() r=-2 bl.length=0 2023-10-05T19:28:29.227 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.218+0000 7f487c2d9e40 20 rados->read ofs=0 len=0 2023-10-05T19:28:29.227 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.218+0000 7f487c2d9e40 1 -- 172.21.15.167:0/2494787467 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:39 5.f 5:f4c53578:::zonegroups_names.default:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x561871483b10 con 0x56187144d730 2023-10-05T19:28:29.227 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.222+0000 7f487a60f700 1 -- 172.21.15.167:0/2494787467 <== osd.1 v2:172.21.15.167:6800/30035 15 ==== osd_op_reply(39 zonegroups_names.default [read 0~46 out=46b] v0'0 uv2 ondisk = 0) v8 ==== 168+0+46 (crc 0 0 0) 0x7f487402f080 con 0x56187144d730 2023-10-05T19:28:29.227 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.222+0000 7f487c2d9e40 20 rados_obj.operate() r=0 bl.length=46 2023-10-05T19:28:29.227 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.222+0000 7f487c2d9e40 20 rados->read ofs=0 len=0 2023-10-05T19:28:29.227 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.222+0000 7f487c2d9e40 1 -- 172.21.15.167:0/2494787467 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:40 5.c 5:30d299a5:::zonegroup_info.4f1f719d-9648-417b-b10c-e5d939825419:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x5618714871d0 con 0x7f4864011040 2023-10-05T19:28:29.228 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.222+0000 7f487a60f700 1 -- 172.21.15.167:0/2494787467 <== osd.0 v2:172.21.15.118:6800/27690 10 ==== osd_op_reply(40 zonegroup_info.4f1f719d-9648-417b-b10c-e5d939825419 [read 0~362 out=362b] v0'0 uv1 ondisk = 0) v8 ==== 195+0+362 (crc 0 0 0) 0x7f487402f080 con 0x7f4864011040 2023-10-05T19:28:29.228 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.222+0000 7f487c2d9e40 20 rados_obj.operate() r=0 bl.length=362 2023-10-05T19:28:29.228 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.222+0000 7f487c2d9e40 20 rados->read ofs=0 len=0 2023-10-05T19:28:29.229 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.222+0000 7f487c2d9e40 1 -- 172.21.15.167:0/2494787467 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:41 5.13 5:cbbf6c60:::converted:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x5618714871d0 con 0x561871484470 2023-10-05T19:28:29.229 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.222+0000 7f4879e0e700 1 -- 172.21.15.167:0/2494787467 <== osd.3 v2:172.21.15.167:6808/30036 10 ==== osd_op_reply(41 converted [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 ==== 153+0+0 (crc 0 0 0) 0x7f486c022020 con 0x561871484470 2023-10-05T19:28:29.229 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.222+0000 7f487c2d9e40 20 rados_obj.operate() r=-2 bl.length=0 2023-10-05T19:28:29.229 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.222+0000 7f487c2d9e40 20 rados->read ofs=0 len=0 2023-10-05T19:28:29.229 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.222+0000 7f487c2d9e40 1 -- 172.21.15.167:0/2494787467 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:42 5.12 5:49953fa1:::default.realm:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x561871487580 con 0x56187144d730 2023-10-05T19:28:29.229 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.222+0000 7f487a60f700 1 -- 172.21.15.167:0/2494787467 <== osd.1 v2:172.21.15.167:6800/30035 16 ==== osd_op_reply(42 default.realm [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 ==== 157+0+0 (crc 0 0 0) 0x7f487402f080 con 0x56187144d730 2023-10-05T19:28:29.230 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.222+0000 7f487c2d9e40 20 rados_obj.operate() r=-2 bl.length=0 2023-10-05T19:28:29.230 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.222+0000 7f487c2d9e40 20 rados->read ofs=0 len=0 2023-10-05T19:28:29.230 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.222+0000 7f487c2d9e40 1 -- 172.21.15.167:0/2494787467 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:43 5.f 5:f4c53578:::zonegroups_names.default:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x561871487980 con 0x56187144d730 2023-10-05T19:28:29.230 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.226+0000 7f487a60f700 1 -- 172.21.15.167:0/2494787467 <== osd.1 v2:172.21.15.167:6800/30035 17 ==== osd_op_reply(43 zonegroups_names.default [read 0~46 out=46b] v0'0 uv2 ondisk = 0) v8 ==== 168+0+46 (crc 0 0 0) 0x7f487402f080 con 0x56187144d730 2023-10-05T19:28:29.230 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.226+0000 7f487c2d9e40 20 rados_obj.operate() r=0 bl.length=46 2023-10-05T19:28:29.231 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.226+0000 7f487c2d9e40 1 -- 172.21.15.167:0/2494787467 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:44 5.0 5:00000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x561871488990 con 0x561871484470 2023-10-05T19:28:29.232 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.226+0000 7f4879e0e700 1 -- 172.21.15.167:0/2494787467 <== osd.3 v2:172.21.15.167:6808/30036 11 ==== osd_op_reply(44 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f486c022020 con 0x561871484470 2023-10-05T19:28:29.233 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.226+0000 7f4879e0e700 1 -- 172.21.15.167:0/2494787467 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:45 5.10 5:08000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x561871488990 con 0x7f486c007030 2023-10-05T19:28:29.233 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.226+0000 7f487960d700 1 -- 172.21.15.167:0/2494787467 <== osd.2 v2:172.21.15.167:6816/30037 7 ==== osd_op_reply(45 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f4864004520 con 0x7f486c007030 2023-10-05T19:28:29.234 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.226+0000 7f487960d700 1 -- 172.21.15.167:0/2494787467 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:46 5.8 5:10000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x561871488990 con 0x7f4864011040 2023-10-05T19:28:29.235 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.230+0000 7f487a60f700 1 -- 172.21.15.167:0/2494787467 <== osd.0 v2:172.21.15.118:6800/27690 11 ==== osd_op_reply(46 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f487402f080 con 0x7f4864011040 2023-10-05T19:28:29.235 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.230+0000 7f487a60f700 1 -- 172.21.15.167:0/2494787467 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:47 5.18 5:18000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x561871483b10 con 0x56187144d730 2023-10-05T19:28:29.235 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.230+0000 7f487a60f700 1 -- 172.21.15.167:0/2494787467 <== osd.1 v2:172.21.15.167:6800/30035 18 ==== osd_op_reply(47 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f487402f080 con 0x56187144d730 2023-10-05T19:28:29.236 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.230+0000 7f487a60f700 1 -- 172.21.15.167:0/2494787467 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:48 5.4 5:20000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x561871483b10 con 0x56187144d730 2023-10-05T19:28:29.236 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.230+0000 7f487a60f700 1 -- 172.21.15.167:0/2494787467 <== osd.1 v2:172.21.15.167:6800/30035 19 ==== osd_op_reply(48 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f487402f080 con 0x56187144d730 2023-10-05T19:28:29.236 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.230+0000 7f487a60f700 1 -- 172.21.15.167:0/2494787467 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:49 5.14 5:28000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x561871483b10 con 0x7f4864011040 2023-10-05T19:28:29.237 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.230+0000 7f487a60f700 1 -- 172.21.15.167:0/2494787467 <== osd.0 v2:172.21.15.118:6800/27690 12 ==== osd_op_reply(49 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f487402f080 con 0x7f4864011040 2023-10-05T19:28:29.237 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.230+0000 7f487a60f700 1 -- 172.21.15.167:0/2494787467 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:50 5.c 5:30000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x561871483b10 con 0x7f4864011040 2023-10-05T19:28:29.238 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.234+0000 7f487a60f700 1 -- 172.21.15.167:0/2494787467 <== osd.0 v2:172.21.15.118:6800/27690 13 ==== osd_op_reply(50 [pgnls start_epoch 30 out=112b] v23'1 uv1 ondisk = 1) v8 ==== 144+0+112 (crc 0 0 0) 0x7f487402f080 con 0x7f4864011040 2023-10-05T19:28:29.239 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.234+0000 7f487a60f700 1 -- 172.21.15.167:0/2494787467 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:51 5.1c 5:38000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x561871483b10 con 0x7f486c007030 2023-10-05T19:28:29.239 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.234+0000 7f487960d700 1 -- 172.21.15.167:0/2494787467 <== osd.2 v2:172.21.15.167:6816/30037 8 ==== osd_op_reply(51 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f4864004520 con 0x7f486c007030 2023-10-05T19:28:29.239 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.234+0000 7f487960d700 1 -- 172.21.15.167:0/2494787467 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:52 5.2 5:40000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x561871483b10 con 0x561871484470 2023-10-05T19:28:29.240 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.234+0000 7f4879e0e700 1 -- 172.21.15.167:0/2494787467 <== osd.3 v2:172.21.15.167:6808/30036 12 ==== osd_op_reply(52 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f486c022020 con 0x561871484470 2023-10-05T19:28:29.240 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.234+0000 7f4879e0e700 1 -- 172.21.15.167:0/2494787467 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:53 5.12 5:48000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x561871483b10 con 0x56187144d730 2023-10-05T19:28:29.241 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.234+0000 7f487a60f700 1 -- 172.21.15.167:0/2494787467 <== osd.1 v2:172.21.15.167:6800/30035 20 ==== osd_op_reply(53 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f487402f080 con 0x56187144d730 2023-10-05T19:28:29.241 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.234+0000 7f487a60f700 1 -- 172.21.15.167:0/2494787467 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:54 5.a 5:50000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x561871483b10 con 0x7f4864011040 2023-10-05T19:28:29.242 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.238+0000 7f487a60f700 1 -- 172.21.15.167:0/2494787467 <== osd.0 v2:172.21.15.118:6800/27690 14 ==== osd_op_reply(54 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f487403d040 con 0x7f4864011040 2023-10-05T19:28:29.242 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.238+0000 7f487a60f700 1 -- 172.21.15.167:0/2494787467 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:55 5.1a 5:58000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x561871483b10 con 0x56187144d730 2023-10-05T19:28:29.243 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.238+0000 7f487a60f700 1 -- 172.21.15.167:0/2494787467 <== osd.1 v2:172.21.15.167:6800/30035 21 ==== osd_op_reply(55 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f487403d040 con 0x56187144d730 2023-10-05T19:28:29.243 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.238+0000 7f487a60f700 1 -- 172.21.15.167:0/2494787467 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:56 5.6 5:60000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x561871483b10 con 0x7f486c007030 2023-10-05T19:28:29.244 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.238+0000 7f487960d700 1 -- 172.21.15.167:0/2494787467 <== osd.2 v2:172.21.15.167:6816/30037 9 ==== osd_op_reply(56 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f4864004520 con 0x7f486c007030 2023-10-05T19:28:29.244 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.238+0000 7f487960d700 1 -- 172.21.15.167:0/2494787467 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:57 5.16 5:68000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x561871483b10 con 0x561871484470 2023-10-05T19:28:29.244 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.238+0000 7f4879e0e700 1 -- 172.21.15.167:0/2494787467 <== osd.3 v2:172.21.15.167:6808/30036 13 ==== osd_op_reply(57 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f486c022020 con 0x561871484470 2023-10-05T19:28:29.244 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.238+0000 7f4879e0e700 1 -- 172.21.15.167:0/2494787467 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:58 5.e 5:70000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x561871483b10 con 0x561871484470 2023-10-05T19:28:29.245 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.238+0000 7f4879e0e700 1 -- 172.21.15.167:0/2494787467 <== osd.3 v2:172.21.15.167:6808/30036 14 ==== osd_op_reply(58 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f486c022020 con 0x561871484470 2023-10-05T19:28:29.245 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.238+0000 7f4879e0e700 1 -- 172.21.15.167:0/2494787467 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:59 5.1e 5:78000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x561871483b10 con 0x7f4864011040 2023-10-05T19:28:29.246 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.242+0000 7f487a60f700 1 -- 172.21.15.167:0/2494787467 <== osd.0 v2:172.21.15.118:6800/27690 15 ==== osd_op_reply(59 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f487403d040 con 0x7f4864011040 2023-10-05T19:28:29.246 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.242+0000 7f487a60f700 1 -- 172.21.15.167:0/2494787467 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:60 5.1 5:80000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x561871483b10 con 0x561871484470 2023-10-05T19:28:29.247 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.242+0000 7f4879e0e700 1 -- 172.21.15.167:0/2494787467 <== osd.3 v2:172.21.15.167:6808/30036 15 ==== osd_op_reply(60 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f486c022020 con 0x561871484470 2023-10-05T19:28:29.247 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.242+0000 7f4879e0e700 1 -- 172.21.15.167:0/2494787467 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:61 5.11 5:88000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x561871483b10 con 0x561871484470 2023-10-05T19:28:29.247 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.242+0000 7f4879e0e700 1 -- 172.21.15.167:0/2494787467 <== osd.3 v2:172.21.15.167:6808/30036 16 ==== osd_op_reply(61 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f486c022020 con 0x561871484470 2023-10-05T19:28:29.248 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.242+0000 7f4879e0e700 1 -- 172.21.15.167:0/2494787467 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:62 5.9 5:90000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x561871483b10 con 0x56187144d730 2023-10-05T19:28:29.248 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.242+0000 7f487a60f700 1 -- 172.21.15.167:0/2494787467 <== osd.1 v2:172.21.15.167:6800/30035 22 ==== osd_op_reply(62 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f487403d040 con 0x56187144d730 2023-10-05T19:28:29.248 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.242+0000 7f487a60f700 1 -- 172.21.15.167:0/2494787467 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:63 5.19 5:98000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x561871483b10 con 0x56187144d730 2023-10-05T19:28:29.249 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.242+0000 7f487a60f700 1 -- 172.21.15.167:0/2494787467 <== osd.1 v2:172.21.15.167:6800/30035 23 ==== osd_op_reply(63 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f487403d040 con 0x56187144d730 2023-10-05T19:28:29.249 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.242+0000 7f487a60f700 1 -- 172.21.15.167:0/2494787467 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:64 5.5 5:a0000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x561871483b10 con 0x7f4864011040 2023-10-05T19:28:29.250 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.246+0000 7f487a60f700 1 -- 172.21.15.167:0/2494787467 <== osd.0 v2:172.21.15.118:6800/27690 16 ==== osd_op_reply(64 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f487403d040 con 0x7f4864011040 2023-10-05T19:28:29.250 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.246+0000 7f487a60f700 1 -- 172.21.15.167:0/2494787467 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:65 5.15 5:a8000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x561871483b10 con 0x561871484470 2023-10-05T19:28:29.251 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.246+0000 7f4879e0e700 1 -- 172.21.15.167:0/2494787467 <== osd.3 v2:172.21.15.167:6808/30036 17 ==== osd_op_reply(65 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f486c022020 con 0x561871484470 2023-10-05T19:28:29.251 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.246+0000 7f4879e0e700 1 -- 172.21.15.167:0/2494787467 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:66 5.d 5:b0000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x561871483b10 con 0x7f486c007030 2023-10-05T19:28:29.252 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.246+0000 7f487960d700 1 -- 172.21.15.167:0/2494787467 <== osd.2 v2:172.21.15.167:6816/30037 10 ==== osd_op_reply(66 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f4864004520 con 0x7f486c007030 2023-10-05T19:28:29.252 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.246+0000 7f487960d700 1 -- 172.21.15.167:0/2494787467 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:67 5.1d 5:b8000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x561871483b10 con 0x56187144d730 2023-10-05T19:28:29.253 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.246+0000 7f487a60f700 1 -- 172.21.15.167:0/2494787467 <== osd.1 v2:172.21.15.167:6800/30035 24 ==== osd_op_reply(67 [pgnls start_epoch 30 out=107b] v30'2 uv2 ondisk = 1) v8 ==== 144+0+107 (crc 0 0 0) 0x7f487403d040 con 0x56187144d730 2023-10-05T19:28:29.253 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.246+0000 7f487a60f700 1 -- 172.21.15.167:0/2494787467 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:68 5.3 5:c0000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x561871483b10 con 0x7f4864011040 2023-10-05T19:28:29.254 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.246+0000 7f487a60f700 1 -- 172.21.15.167:0/2494787467 <== osd.0 v2:172.21.15.118:6800/27690 17 ==== osd_op_reply(68 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f487403d040 con 0x7f4864011040 2023-10-05T19:28:29.254 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.246+0000 7f487a60f700 1 -- 172.21.15.167:0/2494787467 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:69 5.13 5:c8000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x561871483b10 con 0x561871484470 2023-10-05T19:28:29.254 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.250+0000 7f4879e0e700 1 -- 172.21.15.167:0/2494787467 <== osd.3 v2:172.21.15.167:6808/30036 18 ==== osd_op_reply(69 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f486c022020 con 0x561871484470 2023-10-05T19:28:29.254 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.250+0000 7f4879e0e700 1 -- 172.21.15.167:0/2494787467 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:70 5.b 5:d0000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x561871483b10 con 0x7f4864011040 2023-10-05T19:28:29.255 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.250+0000 7f487a60f700 1 -- 172.21.15.167:0/2494787467 <== osd.0 v2:172.21.15.118:6800/27690 18 ==== osd_op_reply(70 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f487403d040 con 0x7f4864011040 2023-10-05T19:28:29.255 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.250+0000 7f487a60f700 1 -- 172.21.15.167:0/2494787467 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:71 5.1b 5:d8000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x561871483b10 con 0x7f486c007030 2023-10-05T19:28:29.256 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.250+0000 7f487960d700 1 -- 172.21.15.167:0/2494787467 <== osd.2 v2:172.21.15.167:6816/30037 11 ==== osd_op_reply(71 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f4864004520 con 0x7f486c007030 2023-10-05T19:28:29.256 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.250+0000 7f487960d700 1 -- 172.21.15.167:0/2494787467 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:72 5.7 5:e0000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x561871483b10 con 0x561871484470 2023-10-05T19:28:29.257 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.250+0000 7f4879e0e700 1 -- 172.21.15.167:0/2494787467 <== osd.3 v2:172.21.15.167:6808/30036 19 ==== osd_op_reply(72 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f486c022020 con 0x561871484470 2023-10-05T19:28:29.257 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.250+0000 7f4879e0e700 1 -- 172.21.15.167:0/2494787467 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:73 5.17 5:e8000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x561871483b10 con 0x7f4864011040 2023-10-05T19:28:29.258 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.254+0000 7f487a60f700 1 -- 172.21.15.167:0/2494787467 <== osd.0 v2:172.21.15.118:6800/27690 19 ==== osd_op_reply(73 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f487403d040 con 0x7f4864011040 2023-10-05T19:28:29.258 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.254+0000 7f487a60f700 1 -- 172.21.15.167:0/2494787467 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:74 5.f 5:f0000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x561871483b10 con 0x56187144d730 2023-10-05T19:28:29.259 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.254+0000 7f487a60f700 1 -- 172.21.15.167:0/2494787467 <== osd.1 v2:172.21.15.167:6800/30035 25 ==== osd_op_reply(74 [pgnls start_epoch 30 out=115b] v23'2 uv2 ondisk = 1) v8 ==== 144+0+115 (crc 0 0 0) 0x7f487403d040 con 0x56187144d730 2023-10-05T19:28:29.259 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.254+0000 7f487a60f700 1 -- 172.21.15.167:0/2494787467 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:75 5.1f 5:f8000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x561871483b10 con 0x7f486c007030 2023-10-05T19:28:29.260 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.254+0000 7f487960d700 1 -- 172.21.15.167:0/2494787467 <== osd.2 v2:172.21.15.167:6816/30037 12 ==== osd_op_reply(75 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f4864004520 con 0x7f486c007030 2023-10-05T19:28:29.260 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.254+0000 7f487c2d9e40 20 RGWRados::pool_iterate: got zonegroup_info.4f1f719d-9648-417b-b10c-e5d939825419 2023-10-05T19:28:29.260 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.254+0000 7f487c2d9e40 20 RGWRados::pool_iterate: got zone_info.9a9df965-6612-45d4-a45f-6e3c889bacc2 2023-10-05T19:28:29.260 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.254+0000 7f487c2d9e40 20 RGWRados::pool_iterate: got zone_names.default 2023-10-05T19:28:29.260 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.254+0000 7f487c2d9e40 20 RGWRados::pool_iterate: got zonegroups_names.default 2023-10-05T19:28:29.261 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.254+0000 7f487c2d9e40 20 rados->read ofs=0 len=0 2023-10-05T19:28:29.261 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.254+0000 7f487c2d9e40 1 -- 172.21.15.167:0/2494787467 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:76 5.f 5:f43b6ece:::zone_names.default:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x561871487980 con 0x56187144d730 2023-10-05T19:28:29.262 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.254+0000 7f487a60f700 1 -- 172.21.15.167:0/2494787467 <== osd.1 v2:172.21.15.167:6800/30035 26 ==== osd_op_reply(76 zone_names.default [read 0~46 out=46b] v0'0 uv1 ondisk = 0) v8 ==== 162+0+46 (crc 0 0 0) 0x7f487403d040 con 0x56187144d730 2023-10-05T19:28:29.262 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.254+0000 7f487c2d9e40 20 rados_obj.operate() r=0 bl.length=46 2023-10-05T19:28:29.262 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.254+0000 7f487c2d9e40 20 rados->read ofs=0 len=0 2023-10-05T19:28:29.262 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.254+0000 7f487c2d9e40 1 -- 172.21.15.167:0/2494787467 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:77 5.1d 5:b8374eb4:::zone_info.9a9df965-6612-45d4-a45f-6e3c889bacc2:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x561871487d30 con 0x56187144d730 2023-10-05T19:28:29.262 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.254+0000 7f487a60f700 1 -- 172.21.15.167:0/2494787467 <== osd.1 v2:172.21.15.167:6800/30035 27 ==== osd_op_reply(77 zone_info.9a9df965-6612-45d4-a45f-6e3c889bacc2 [read 0~874 out=874b] v0'0 uv2 ondisk = 0) v8 ==== 190+0+874 (crc 0 0 0) 0x7f487403d040 con 0x56187144d730 2023-10-05T19:28:29.262 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.254+0000 7f487c2d9e40 20 rados_obj.operate() r=0 bl.length=874 2023-10-05T19:28:29.262 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.254+0000 7f487c2d9e40 20 rados->read ofs=0 len=0 2023-10-05T19:28:29.262 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.254+0000 7f487c2d9e40 1 -- 172.21.15.167:0/2494787467 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:78 5.f 5:f4c53578:::zonegroups_names.default:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x5618714836b0 con 0x56187144d730 2023-10-05T19:28:29.263 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.258+0000 7f487a60f700 1 -- 172.21.15.167:0/2494787467 <== osd.1 v2:172.21.15.167:6800/30035 28 ==== osd_op_reply(78 zonegroups_names.default [read 0~46 out=46b] v0'0 uv2 ondisk = 0) v8 ==== 168+0+46 (crc 0 0 0) 0x7f487403d040 con 0x56187144d730 2023-10-05T19:28:29.263 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.258+0000 7f487c2d9e40 20 rados_obj.operate() r=0 bl.length=46 2023-10-05T19:28:29.263 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.258+0000 7f487c2d9e40 20 rados->read ofs=0 len=0 2023-10-05T19:28:29.263 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.258+0000 7f487c2d9e40 1 -- 172.21.15.167:0/2494787467 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:79 5.c 5:30d299a5:::zonegroup_info.4f1f719d-9648-417b-b10c-e5d939825419:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x5618714836d0 con 0x7f4864011040 2023-10-05T19:28:29.264 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.258+0000 7f487a60f700 1 -- 172.21.15.167:0/2494787467 <== osd.0 v2:172.21.15.118:6800/27690 20 ==== osd_op_reply(79 zonegroup_info.4f1f719d-9648-417b-b10c-e5d939825419 [read 0~362 out=362b] v0'0 uv1 ondisk = 0) v8 ==== 195+0+362 (crc 0 0 0) 0x7f487403d040 con 0x7f4864011040 2023-10-05T19:28:29.264 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.258+0000 7f487c2d9e40 20 rados_obj.operate() r=0 bl.length=362 2023-10-05T19:28:29.264 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.258+0000 7f487c2d9e40 20 rados->read ofs=0 len=0 2023-10-05T19:28:29.264 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.258+0000 7f487c2d9e40 1 -- 172.21.15.167:0/2494787467 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:80 5.3 5:c0ad8cbd:::region_map:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x5618714887a0 con 0x7f4864011040 2023-10-05T19:28:29.265 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.258+0000 7f487a60f700 1 -- 172.21.15.167:0/2494787467 <== osd.0 v2:172.21.15.118:6800/27690 21 ==== osd_op_reply(80 region_map [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 ==== 154+0+0 (crc 0 0 0) 0x7f487403d040 con 0x7f4864011040 2023-10-05T19:28:29.265 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.258+0000 7f487c2d9e40 20 rados_obj.operate() r=-2 bl.length=0 2023-10-05T19:28:29.265 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.258+0000 7f487c2d9e40 20 zone default found 2023-10-05T19:28:29.265 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.258+0000 7f487c2d9e40 4 Realm: () 2023-10-05T19:28:29.265 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.258+0000 7f487c2d9e40 4 ZoneGroup: default (4f1f719d-9648-417b-b10c-e5d939825419) 2023-10-05T19:28:29.265 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.258+0000 7f487c2d9e40 4 Zone: default (9a9df965-6612-45d4-a45f-6e3c889bacc2) 2023-10-05T19:28:29.265 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.258+0000 7f487c2d9e40 10 cannot find current period zonegroup using local zonegroup configuration 2023-10-05T19:28:29.265 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.258+0000 7f487c2d9e40 20 zonegroup default 2023-10-05T19:28:29.265 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.258+0000 7f487c2d9e40 20 rados->read ofs=0 len=0 2023-10-05T19:28:29.265 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.258+0000 7f487c2d9e40 1 -- 172.21.15.167:0/2494787467 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:81 5.3 5:c52100b6:::period_config.default:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x5618714887a0 con 0x7f4864011040 2023-10-05T19:28:29.266 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.258+0000 7f487a60f700 1 -- 172.21.15.167:0/2494787467 <== osd.0 v2:172.21.15.118:6800/27690 22 ==== osd_op_reply(81 period_config.default [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 ==== 165+0+0 (crc 0 0 0) 0x7f487403d040 con 0x7f4864011040 2023-10-05T19:28:29.266 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.258+0000 7f487c2d9e40 20 rados_obj.operate() r=-2 bl.length=0 2023-10-05T19:28:29.266 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.258+0000 7f487c2d9e40 20 rados->read ofs=0 len=0 2023-10-05T19:28:29.267 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.258+0000 7f487c2d9e40 1 -- 172.21.15.167:0/2494787467 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:82 6.2 6:4347d321:::bucket.sync-source-hints.:head [call version.read in=11b,read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x561871489dc0 con 0x561871484470 2023-10-05T19:28:29.267 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.262+0000 7f4879e0e700 1 -- 172.21.15.167:0/2494787467 <== osd.3 v2:172.21.15.167:6808/30036 20 ==== osd_op_reply(82 bucket.sync-source-hints. [call,read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 ==== 211+0+0 (crc 0 0 0) 0x7f486c022020 con 0x561871484470 2023-10-05T19:28:29.267 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.262+0000 7f487c2d9e40 20 rados_obj.operate() r=-2 bl.length=0 2023-10-05T19:28:29.267 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.262+0000 7f487c2d9e40 20 rados->read ofs=0 len=0 2023-10-05T19:28:29.267 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.262+0000 7f487c2d9e40 1 -- 172.21.15.167:0/2494787467 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:83 6.b 6:d467b91b:::bucket.sync-target-hints.:head [call version.read in=11b,read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x56187148a170 con 0x561871484470 2023-10-05T19:28:29.268 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.262+0000 7f4879e0e700 1 -- 172.21.15.167:0/2494787467 <== osd.3 v2:172.21.15.167:6808/30036 21 ==== osd_op_reply(83 bucket.sync-target-hints. [call,read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 ==== 211+0+0 (crc 0 0 0) 0x7f486c062040 con 0x561871484470 2023-10-05T19:28:29.268 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.262+0000 7f487c2d9e40 20 rados_obj.operate() r=-2 bl.length=0 2023-10-05T19:28:29.268 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.262+0000 7f487c2d9e40 20 started sync module instance, tier type = 2023-10-05T19:28:29.268 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.262+0000 7f487c2d9e40 20 started zone id=9a9df965-6612-45d4-a45f-6e3c889bacc2 (name=default) with tier type = 2023-10-05T19:28:29.268 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.262+0000 7f487c2d9e40 1 -- 172.21.15.167:0/2494787467 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:84 7.1f 7:f95f44c2:::notify.0:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x561871489480 con 0x7f486c007030 2023-10-05T19:28:29.270 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.266+0000 7f487960d700 1 -- 172.21.15.167:0/2494787467 <== osd.2 v2:172.21.15.167:6816/30037 13 ==== osd_op_reply(84 notify.0 [create] v30'72 uv72 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f486402f040 con 0x7f486c007030 2023-10-05T19:28:29.270 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.266+0000 7f487c2d9e40 1 -- 172.21.15.167:0/2494787467 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:85 7.1f 7:f95f44c2:::notify.0:head [watch watch cookie 94662979786800] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x5618714890b0 con 0x7f486c007030 2023-10-05T19:28:29.270 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.266+0000 7f487c2d9e40 1 -- 172.21.15.167:0/2494787467 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:86 7.0 7:05bf5b68:::notify.1:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x56187148af30 con 0x56187144d730 2023-10-05T19:28:29.273 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.266+0000 7f487a60f700 1 -- 172.21.15.167:0/2494787467 <== osd.1 v2:172.21.15.167:6800/30035 29 ==== osd_op_reply(86 notify.1 [create] v30'72 uv72 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f487403d040 con 0x56187144d730 2023-10-05T19:28:29.273 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.266+0000 7f487960d700 1 -- 172.21.15.167:0/2494787467 <== osd.2 v2:172.21.15.167:6816/30037 14 ==== osd_op_reply(85 notify.0 [watch watch cookie 94662979786800] v30'73 uv72 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f486402f040 con 0x7f486c007030 2023-10-05T19:28:29.274 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.266+0000 7f487c2d9e40 1 -- 172.21.15.167:0/2494787467 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:87 7.0 7:05bf5b68:::notify.1:head [watch watch cookie 94662979791712] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x56187148b820 con 0x56187144d730 2023-10-05T19:28:29.274 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.266+0000 7f487c2d9e40 1 -- 172.21.15.167:0/2494787467 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:88 7.15 7:a93a5511:::notify.2:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x56187148a520 con 0x561871484470 2023-10-05T19:28:29.276 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.270+0000 7f487a60f700 1 -- 172.21.15.167:0/2494787467 <== osd.1 v2:172.21.15.167:6800/30035 30 ==== osd_op_reply(87 notify.1 [watch watch cookie 94662979791712] v30'73 uv72 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f487403d040 con 0x56187144d730 2023-10-05T19:28:29.277 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.270+0000 7f4879e0e700 1 -- 172.21.15.167:0/2494787467 <== osd.3 v2:172.21.15.167:6808/30036 22 ==== osd_op_reply(88 notify.2 [create] v30'72 uv72 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f486c062040 con 0x561871484470 2023-10-05T19:28:29.277 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.270+0000 7f487c2d9e40 1 -- 172.21.15.167:0/2494787467 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:89 7.15 7:a93a5511:::notify.2:head [watch watch cookie 94662979793712] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x56187148bc70 con 0x561871484470 2023-10-05T19:28:29.277 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.270+0000 7f487c2d9e40 1 -- 172.21.15.167:0/2494787467 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:90 7.e 7:7759931f:::notify.3:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x56187148c6e0 con 0x7f486c007030 2023-10-05T19:28:29.280 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.274+0000 7f487960d700 1 -- 172.21.15.167:0/2494787467 <== osd.2 v2:172.21.15.167:6816/30037 15 ==== osd_op_reply(90 notify.3 [create] v30'72 uv72 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f486402f040 con 0x7f486c007030 2023-10-05T19:28:29.280 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.274+0000 7f487c2d9e40 1 -- 172.21.15.167:0/2494787467 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:91 7.e 7:7759931f:::notify.3:head [watch watch cookie 94662979799776] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x56187148d420 con 0x7f486c007030 2023-10-05T19:28:29.280 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.274+0000 7f4879e0e700 1 -- 172.21.15.167:0/2494787467 <== osd.3 v2:172.21.15.167:6808/30036 23 ==== osd_op_reply(89 notify.2 [watch watch cookie 94662979793712] v30'73 uv72 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f486c062040 con 0x561871484470 2023-10-05T19:28:29.280 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.274+0000 7f487c2d9e40 1 -- 172.21.15.167:0/2494787467 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:92 7.d 7:b4812045:::notify.4:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x56187148d7d0 con 0x56187144d730 2023-10-05T19:28:29.282 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.278+0000 7f487960d700 1 -- 172.21.15.167:0/2494787467 <== osd.2 v2:172.21.15.167:6816/30037 16 ==== osd_op_reply(91 notify.3 [watch watch cookie 94662979799776] v30'73 uv72 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f486402f040 con 0x7f486c007030 2023-10-05T19:28:29.283 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.278+0000 7f487a60f700 1 -- 172.21.15.167:0/2494787467 <== osd.1 v2:172.21.15.167:6800/30035 31 ==== osd_op_reply(92 notify.4 [create] v30'72 uv72 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f487403d040 con 0x56187144d730 2023-10-05T19:28:29.283 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.278+0000 7f487c2d9e40 1 -- 172.21.15.167:0/2494787467 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:93 7.d 7:b4812045:::notify.4:head [watch watch cookie 94662979800880] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x56187148dcc0 con 0x56187144d730 2023-10-05T19:28:29.283 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.278+0000 7f487c2d9e40 1 -- 172.21.15.167:0/2494787467 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:94 7.3 7:c609908c:::notify.5:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x56187148e560 con 0x7f4864011040 2023-10-05T19:28:29.285 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.278+0000 7f487a60f700 1 -- 172.21.15.167:0/2494787467 <== osd.0 v2:172.21.15.118:6800/27690 23 ==== osd_op_reply(94 notify.5 [create] v30'72 uv72 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f487403d040 con 0x7f4864011040 2023-10-05T19:28:29.285 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.278+0000 7f487c2d9e40 1 -- 172.21.15.167:0/2494787467 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:95 7.3 7:c609908c:::notify.5:head [watch watch cookie 94662979804912] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x56187148e960 con 0x7f4864011040 2023-10-05T19:28:29.285 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.278+0000 7f487c2d9e40 1 -- 172.21.15.167:0/2494787467 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:96 7.14 7:2b04a3e9:::notify.6:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x56187148f4c0 con 0x7f4864011040 2023-10-05T19:28:29.286 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.278+0000 7f487a60f700 1 -- 172.21.15.167:0/2494787467 <== osd.1 v2:172.21.15.167:6800/30035 32 ==== osd_op_reply(93 notify.4 [watch watch cookie 94662979800880] v30'73 uv72 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f487403d040 con 0x56187144d730 2023-10-05T19:28:29.287 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.282+0000 7f487a60f700 1 -- 172.21.15.167:0/2494787467 <== osd.0 v2:172.21.15.118:6800/27690 24 ==== osd_op_reply(95 notify.5 [watch watch cookie 94662979804912] v30'73 uv72 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f487403d040 con 0x7f4864011040 2023-10-05T19:28:29.288 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.282+0000 7f487a60f700 1 -- 172.21.15.167:0/2494787467 <== osd.0 v2:172.21.15.118:6800/27690 25 ==== osd_op_reply(96 notify.6 [create] v30'72 uv72 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f487403d040 con 0x7f4864011040 2023-10-05T19:28:29.288 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.282+0000 7f487c2d9e40 1 -- 172.21.15.167:0/2494787467 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:97 7.14 7:2b04a3e9:::notify.6:head [watch watch cookie 94662979803088] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x56187148ef90 con 0x7f4864011040 2023-10-05T19:28:29.288 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.282+0000 7f487c2d9e40 1 -- 172.21.15.167:0/2494787467 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:98 7.9 7:93e5b521:::notify.7:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x56187148f870 con 0x7f4864011040 2023-10-05T19:28:29.290 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.286+0000 7f487a60f700 1 -- 172.21.15.167:0/2494787467 <== osd.0 v2:172.21.15.118:6800/27690 26 ==== osd_op_reply(97 notify.6 [watch watch cookie 94662979803088] v30'73 uv72 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f487403d040 con 0x7f4864011040 2023-10-05T19:28:29.291 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.286+0000 7f487a60f700 1 -- 172.21.15.167:0/2494787467 <== osd.0 v2:172.21.15.118:6800/27690 27 ==== osd_op_reply(98 notify.7 [create] v30'72 uv72 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f487403d040 con 0x7f4864011040 2023-10-05T19:28:29.291 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.286+0000 7f487c2d9e40 1 -- 172.21.15.167:0/2494787467 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:99 7.9 7:93e5b521:::notify.7:head [watch watch cookie 94662979806080] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x56187148f480 con 0x7f4864011040 2023-10-05T19:28:29.291 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.286+0000 7f487c2d9e40 20 add_watcher() i=0 2023-10-05T19:28:29.291 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.286+0000 7f487c2d9e40 20 add_watcher() i=1 2023-10-05T19:28:29.291 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.286+0000 7f487c2d9e40 20 add_watcher() i=2 2023-10-05T19:28:29.291 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.286+0000 7f487c2d9e40 20 add_watcher() i=3 2023-10-05T19:28:29.291 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.286+0000 7f487c2d9e40 20 add_watcher() i=4 2023-10-05T19:28:29.291 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.286+0000 7f487c2d9e40 20 add_watcher() i=5 2023-10-05T19:28:29.291 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.286+0000 7f487c2d9e40 20 add_watcher() i=6 2023-10-05T19:28:29.293 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.286+0000 7f487a60f700 1 -- 172.21.15.167:0/2494787467 <== osd.0 v2:172.21.15.118:6800/27690 28 ==== osd_op_reply(99 notify.7 [watch watch cookie 94662979806080] v30'73 uv72 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f487403d040 con 0x7f4864011040 2023-10-05T19:28:29.293 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.286+0000 7f487c2d9e40 20 add_watcher() i=7 2023-10-05T19:28:29.293 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.286+0000 7f487c2d9e40 2 all 8 watchers are set, enabling cache 2023-10-05T19:28:29.293 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.286+0000 7f487c2d9e40 1 -- 172.21.15.167:0/2494787467 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:100 6.4 6:22d26bf9:::data_loggenerations_metadata:head [call version.check_conds in=74b,call version.read in=11b,read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x561871492c10 con 0x56187144d730 2023-10-05T19:28:29.294 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.286+0000 7f487a60f700 1 -- 172.21.15.167:0/2494787467 <== osd.1 v2:172.21.15.167:6800/30035 33 ==== osd_op_reply(100 data_loggenerations_metadata [call,call out=48b,read 0~28 out=28b] v0'0 uv1 ondisk = 0) v8 ==== 256+0+76 (crc 0 0 0) 0x7f487403d040 con 0x56187144d730 2023-10-05T19:28:29.294 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.286+0000 7f487c2d9e40 1 -- 172.21.15.167:0/2494787467 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:101 6.4 6:22d26bf9:::data_loggenerations_metadata:head [watch watch cookie 94662979809632] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x5618714900d0 con 0x56187144d730 2023-10-05T19:28:29.296 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.290+0000 7f487a60f700 1 -- 172.21.15.167:0/2494787467 <== osd.1 v2:172.21.15.167:6800/30035 34 ==== osd_op_reply(101 data_loggenerations_metadata [watch watch cookie 94662979809632] v30'82 uv1 ondisk = 0) v8 ==== 172+0+0 (crc 0 0 0) 0x7f487403d040 con 0x56187144d730 2023-10-05T19:28:29.296 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.290+0000 7f484a7ec700 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2023-10-05T19:28:29.296 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.290+0000 7f487c2d9e40 20 check_secure_mon_conn(): auth registy supported: methods=[2] modes=[2,1] 2023-10-05T19:28:29.296 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.290+0000 7f487c2d9e40 20 check_secure_mon_conn(): mode 1 is insecure 2023-10-05T19:28:29.297 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.290+0000 7f487c2d9e40 1 Processor -- start 2023-10-05T19:28:29.297 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.290+0000 7f487c2d9e40 1 -- start start 2023-10-05T19:28:29.297 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.290+0000 7f487c2d9e40 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x56187149d2b0 0x56187149d680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-05T19:28:29.297 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.290+0000 7f487c2d9e40 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x56187149dbc0 con 0x56187149d2b0 2023-10-05T19:28:29.298 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.290+0000 7f487960d700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x56187149d2b0 0x56187149d680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-05T19:28:29.298 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.290+0000 7f487960d700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x56187149d2b0 0x56187149d680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.118:3300/0 says I am v2:172.21.15.167:57888/0 (socket says 172.21.15.167:57888) 2023-10-05T19:28:29.298 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.290+0000 7f487960d700 1 -- 172.21.15.167:0/594925766 learned_addr learned my addr 172.21.15.167:0/594925766 (peer_addr_for_me v2:172.21.15.167:0/0) 2023-10-05T19:28:29.298 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.290+0000 7f487960d700 1 -- 172.21.15.167:0/594925766 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x56187149dd00 con 0x56187149d2b0 2023-10-05T19:28:29.299 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.294+0000 7f487960d700 1 --2- 172.21.15.167:0/594925766 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x56187149d2b0 0x56187149d680 secure :-1 s=READY pgs=282 cs=0 l=1 rev1=1 crypto rx=0x561871481f10 tx=0x7f4864006000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=419b41fe69a75750 server_cookie=0 in_seq=0 out_seq=0 2023-10-05T19:28:29.299 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.294+0000 7f4849feb700 1 -- 172.21.15.167:0/594925766 <== mon.0 v2:172.21.15.118:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f486402c040 con 0x56187149d2b0 2023-10-05T19:28:29.299 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.294+0000 7f4849feb700 1 -- 172.21.15.167:0/594925766 <== mon.0 v2:172.21.15.118:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f486402d040 con 0x56187149d2b0 2023-10-05T19:28:29.299 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.294+0000 7f4849feb700 1 -- 172.21.15.167:0/594925766 <== mon.0 v2:172.21.15.118:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f4864028c80 con 0x56187149d2b0 2023-10-05T19:28:29.300 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.294+0000 7f487c2d9e40 1 -- 172.21.15.167:0/594925766 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x56187149d2b0 msgr2=0x56187149d680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-05T19:28:29.300 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.294+0000 7f487c2d9e40 1 --2- 172.21.15.167:0/594925766 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x56187149d2b0 0x56187149d680 secure :-1 s=READY pgs=282 cs=0 l=1 rev1=1 crypto rx=0x561871481f10 tx=0x7f4864006000 comp rx=0 tx=0).stop 2023-10-05T19:28:29.300 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.294+0000 7f487c2d9e40 1 -- 172.21.15.167:0/594925766 shutdown_connections 2023-10-05T19:28:29.300 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.294+0000 7f487c2d9e40 1 --2- 172.21.15.167:0/594925766 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x56187149d2b0 0x56187149d680 unknown :-1 s=CLOSED pgs=282 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-05T19:28:29.300 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.294+0000 7f487c2d9e40 1 -- 172.21.15.167:0/594925766 >> 172.21.15.167:0/594925766 conn(0x561871498670 msgr2=0x5618714967c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-05T19:28:29.300 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.294+0000 7f487c2d9e40 1 -- 172.21.15.167:0/594925766 shutdown_connections 2023-10-05T19:28:29.300 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.294+0000 7f487c2d9e40 1 -- 172.21.15.167:0/594925766 wait complete. 2023-10-05T19:28:29.301 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.294+0000 7f487c2d9e40 1 Processor -- start 2023-10-05T19:28:29.301 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.294+0000 7f487c2d9e40 1 -- start start 2023-10-05T19:28:29.301 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.294+0000 7f487c2d9e40 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x56187149d2b0 0x5618714a2ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-05T19:28:29.301 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.294+0000 7f487c2d9e40 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x5618714a3ee0 con 0x56187149d2b0 2023-10-05T19:28:29.301 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.294+0000 7f487960d700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x56187149d2b0 0x5618714a2ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-05T19:28:29.301 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.294+0000 7f487960d700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x56187149d2b0 0x5618714a2ee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.118:3300/0 says I am v2:172.21.15.167:57902/0 (socket says 172.21.15.167:57902) 2023-10-05T19:28:29.302 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.294+0000 7f487960d700 1 -- 172.21.15.167:0/3268647138 learned_addr learned my addr 172.21.15.167:0/3268647138 (peer_addr_for_me v2:172.21.15.167:0/0) 2023-10-05T19:28:29.302 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.294+0000 7f487960d700 1 -- 172.21.15.167:0/3268647138 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4864004a80 con 0x56187149d2b0 2023-10-05T19:28:29.303 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.298+0000 7f487960d700 1 --2- 172.21.15.167:0/3268647138 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x56187149d2b0 0x5618714a2ee0 secure :-1 s=READY pgs=283 cs=0 l=1 rev1=1 crypto rx=0x7f48640060a0 tx=0x7f486403c000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-05T19:28:29.303 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.298+0000 7f47bbfff700 1 -- 172.21.15.167:0/3268647138 <== mon.0 v2:172.21.15.118:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f486402c070 con 0x56187149d2b0 2023-10-05T19:28:29.303 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.298+0000 7f487c2d9e40 1 -- 172.21.15.167:0/3268647138 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x5618714a34e0 con 0x56187149d2b0 2023-10-05T19:28:29.303 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.298+0000 7f47bbfff700 1 -- 172.21.15.167:0/3268647138 <== mon.0 v2:172.21.15.118:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f486401c040 con 0x56187149d2b0 2023-10-05T19:28:29.303 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.298+0000 7f47bbfff700 1 -- 172.21.15.167:0/3268647138 <== mon.0 v2:172.21.15.118:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f486404a5d0 con 0x56187149d2b0 2023-10-05T19:28:29.303 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.298+0000 7f487c2d9e40 1 -- 172.21.15.167:0/3268647138 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x5618714a38c0 con 0x56187149d2b0 2023-10-05T19:28:29.304 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.298+0000 7f47bbfff700 1 -- 172.21.15.167:0/3268647138 <== mon.0 v2:172.21.15.118:3300/0 4 ==== mgrmap(e 5) v1 ==== 40329+0+0 (secure 0 0 0) 0x7f4864011410 con 0x56187149d2b0 2023-10-05T19:28:29.304 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.298+0000 7f47bbfff700 1 --2- 172.21.15.167:0/3268647138 >> [v2:172.21.15.167:6824/29889,v1:172.21.15.167:6825/29889] conn(0x7f47c0030c00 0x7f47c00330b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-05T19:28:29.304 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.298+0000 7f47bbfff700 1 -- 172.21.15.167:0/3268647138 <== mon.0 v2:172.21.15.118:3300/0 5 ==== osd_map(30..30 src has 1..30) v4 ==== 6057+0+0 (secure 0 0 0) 0x7f4864005030 con 0x56187149d2b0 2023-10-05T19:28:29.304 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.298+0000 7f487a60f700 1 --2- 172.21.15.167:0/3268647138 >> [v2:172.21.15.167:6824/29889,v1:172.21.15.167:6825/29889] conn(0x7f47c0030c00 0x7f47c00330b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-05T19:28:29.305 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.298+0000 7f487c2d9e40 5 note: GC not initialized 2023-10-05T19:28:29.305 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.298+0000 7f487a60f700 1 --2- 172.21.15.167:0/3268647138 >> [v2:172.21.15.167:6824/29889,v1:172.21.15.167:6825/29889] conn(0x7f47c0030c00 0x7f47c00330b0 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x56187148f700 tx=0x7f487400e040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-05T19:28:29.308 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.302+0000 7f47b2ffd700 20 reqs_thread_entry: start 2023-10-05T19:28:29.308 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.302+0000 7f487c2d9e40 20 init_complete bucket index max shards: 11 2023-10-05T19:28:29.308 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.302+0000 7f47b97fa700 20 reqs_thread_entry: start 2023-10-05T19:28:29.308 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.302+0000 7f487c2d9e40 10 rgw notify: Started notification manager with: 1 workers 2023-10-05T19:28:29.309 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.302+0000 7f487c2d9e40 20 get_system_obj_state: rctx=0x7ffd6afeb248 obj=default.rgw.meta:users.uid:foo state=0x5618713260b0 s->prefetch_data=0 2023-10-05T19:28:29.309 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.302+0000 7f487c2d9e40 10 cache get: name=default.rgw.meta+users.uid+foo : miss 2023-10-05T19:28:29.309 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.302+0000 7f487c2d9e40 1 -- 172.21.15.167:0/2494787467 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:102 8.18 8:1e6ca5ed:users.uid::foo:head [call version.read in=11b,getxattrs,stat] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x56187150b790 con 0x56187144d730 2023-10-05T19:28:29.309 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.302+0000 7f47b0ff9700 20 reqs_thread_entry: start 2023-10-05T19:28:29.309 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.302+0000 7f47b17fa700 20 rgw notify: INFO: next queues processing will happen at: Thu Oct 5 19:28:59 2023 2023-10-05T19:28:29.309 INFO:teuthology.orchestra.run.smithi167.stderr: 2023-10-05T19:28:29.310 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.302+0000 7f487a60f700 1 -- 172.21.15.167:0/2494787467 <== osd.1 v2:172.21.15.167:6800/30035 35 ==== osd_op_reply(102 foo [call out=48b,getxattrs out=75b,stat out=16b] v0'0 uv8 ondisk = 0) v8 ==== 231+0+139 (crc 0 0 0) 0x7f487403d040 con 0x56187144d730 2023-10-05T19:28:29.310 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.302+0000 7f487c2d9e40 10 cache put: name=default.rgw.meta+users.uid+foo info.flags=0x16 2023-10-05T19:28:29.310 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.302+0000 7f487c2d9e40 10 adding default.rgw.meta+users.uid+foo to cache LRU end 2023-10-05T19:28:29.310 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.302+0000 7f487c2d9e40 10 updating xattr: name=ceph.objclass.version bl.length()=42 2023-10-05T19:28:29.310 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.302+0000 7f487c2d9e40 20 get_system_obj_state: s->obj_tag was set empty 2023-10-05T19:28:29.310 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.302+0000 7f487c2d9e40 20 Read xattr: user.rgw.idtag 2023-10-05T19:28:29.311 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.302+0000 7f487c2d9e40 10 cache get: name=default.rgw.meta+users.uid+foo : type miss (requested=0x13, cached=0x16) 2023-10-05T19:28:29.311 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.302+0000 7f487c2d9e40 20 rados->read ofs=0 len=0 2023-10-05T19:28:29.311 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.302+0000 7f487c2d9e40 1 -- 172.21.15.167:0/2494787467 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:103 8.18 8:1e6ca5ed:users.uid::foo:head [call version.check_conds in=122b,call version.read in=11b,read 0~0,getxattrs] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x56187150e220 con 0x56187144d730 2023-10-05T19:28:29.311 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.306+0000 7f487a60f700 1 -- 172.21.15.167:0/2494787467 <== osd.1 v2:172.21.15.167:6800/30035 36 ==== osd_op_reply(103 foo [call,call out=48b,read 0~621 out=621b,getxattrs out=75b] v0'0 uv8 ondisk = 0) v8 ==== 273+0+744 (crc 0 0 0) 0x7f487403d040 con 0x56187144d730 2023-10-05T19:28:29.311 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.306+0000 7f487c2d9e40 20 rados_obj.operate() r=0 bl.length=621 2023-10-05T19:28:29.311 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.306+0000 7f487c2d9e40 10 cache put: name=default.rgw.meta+users.uid+foo info.flags=0x13 2023-10-05T19:28:29.311 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.306+0000 7f487c2d9e40 10 moving default.rgw.meta+users.uid+foo to cache LRU end 2023-10-05T19:28:29.311 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.306+0000 7f487c2d9e40 10 updating xattr: name=ceph.objclass.version bl.length()=42 2023-10-05T19:28:29.311 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.306+0000 7f487c2d9e40 1 -- 172.21.15.167:0/2494787467 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:104 8.18 8:1e6ca5ed:users.uid::foo:head [delete,create,call version.set in=58b,writefull 0~539 in=539b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x56187150fd50 con 0x56187144d730 2023-10-05T19:28:29.313 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.306+0000 7f487a60f700 1 -- 172.21.15.167:0/2494787467 <== osd.1 v2:172.21.15.167:6800/30035 37 ==== osd_op_reply(104 foo [delete,create,call,writefull 0~539] v30'9 uv9 ondisk = 0) v8 ==== 273+0+0 (crc 0 0 0) 0x7f487403d040 con 0x56187144d730 2023-10-05T19:28:29.314 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.306+0000 7f487c2d9e40 10 cache put: name=default.rgw.meta+users.uid+foo info.flags=0x17 2023-10-05T19:28:29.314 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.306+0000 7f487c2d9e40 10 moving default.rgw.meta+users.uid+foo to cache LRU end 2023-10-05T19:28:29.314 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.306+0000 7f487c2d9e40 10 distributing notification oid=default.rgw.control:notify.4 cni=[op: 0, obj: default.rgw.meta:users.uid:foo, ofs0, ns] 2023-10-05T19:28:29.314 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.306+0000 7f487c2d9e40 1 -- 172.21.15.167:0/2494787467 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:105 7.d 7:b4812045:::notify.4:head [notify cookie 94662980328752 in=727b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x56187150f680 con 0x56187144d730 2023-10-05T19:28:29.315 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.310+0000 7f487a60f700 1 -- 172.21.15.167:0/2494787467 <== osd.1 v2:172.21.15.167:6800/30035 38 ==== watch-notify(notify (1) cookie 94662979800880 notify 128849018888 ret 0) v3 ==== 757+0+0 (crc 0 0 0) 0x7f4874002e10 con 0x56187144d730 2023-10-05T19:28:29.315 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.310+0000 7f48737fe700 10 rgw watcher librados: RGWWatcher::handle_notify() notify_id 128849018888 cookie 94662979800880 notifier 4371 bl.length()=715 2023-10-05T19:28:29.315 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.310+0000 7f48737fe700 10 rgw watcher librados: cache put: name=default.rgw.meta+users.uid+foo info.flags=0x17 2023-10-05T19:28:29.315 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.310+0000 7f48737fe700 10 rgw watcher librados: moving default.rgw.meta+users.uid+foo to cache LRU end 2023-10-05T19:28:29.315 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.310+0000 7f48737fe700 1 -- 172.21.15.167:0/2494787467 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:106 7.d 7:b4812045:::notify.4:head [notify-ack in=20b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x7f4854003d90 con 0x56187144d730 2023-10-05T19:28:29.315 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.310+0000 7f487a60f700 1 -- 172.21.15.167:0/2494787467 <== osd.1 v2:172.21.15.167:6800/30035 39 ==== osd_op_reply(105 notify.4 [notify cookie 94662980328752 out=8b] v0'0 uv72 ondisk = 0) v8 ==== 152+0+8 (crc 0 0 0) 0x7f487403d040 con 0x56187144d730 2023-10-05T19:28:29.315 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.310+0000 7f487a60f700 1 -- 172.21.15.167:0/2494787467 <== osd.1 v2:172.21.15.167:6800/30035 40 ==== osd_op_reply(106 notify.4 [notify-ack] v0'0 uv72 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f487403d040 con 0x56187144d730 2023-10-05T19:28:29.316 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.310+0000 7f487a60f700 1 -- 172.21.15.167:0/2494787467 <== osd.1 v2:172.21.15.167:6800/30035 41 ==== watch-notify(notify_complete (2) cookie 94662980328752 notify 128849018888 ret 0) v3 ==== 42+0+48 (crc 0 0 0) 0x7f4874002b70 con 0x56187144d730 2023-10-05T19:28:29.316 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.310+0000 7f487c2d9e40 10 distributing notification oid=default.rgw.control:notify.0 cni=[op: 1, obj: default.rgw.meta:users.swift:foo:foo1, ofs0, ns] 2023-10-05T19:28:29.316 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.310+0000 7f487c2d9e40 1 -- 172.21.15.167:0/2494787467 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:107 7.1f 7:f95f44c2:::notify.0:head [notify cookie 94662980328752 in=171b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x56187150f680 con 0x7f486c007030 2023-10-05T19:28:29.316 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.310+0000 7f487960d700 1 -- 172.21.15.167:0/2494787467 <== osd.2 v2:172.21.15.167:6816/30037 17 ==== watch-notify(notify (1) cookie 94662979786800 notify 128849018884 ret 0) v3 ==== 201+0+0 (crc 0 0 0) 0x5618714a38c0 con 0x7f486c007030 2023-10-05T19:28:29.316 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.310+0000 7f4872ffd700 10 rgw watcher librados: RGWWatcher::handle_notify() notify_id 128849018884 cookie 94662979786800 notifier 4371 bl.length()=159 2023-10-05T19:28:29.317 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.310+0000 7f4872ffd700 1 -- 172.21.15.167:0/2494787467 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:108 7.1f 7:f95f44c2:::notify.0:head [notify-ack in=20b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x7f47c80032d0 con 0x7f486c007030 2023-10-05T19:28:29.317 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.310+0000 7f487960d700 1 -- 172.21.15.167:0/2494787467 <== osd.2 v2:172.21.15.167:6816/30037 18 ==== osd_op_reply(107 notify.0 [notify cookie 94662980328752 out=8b] v0'0 uv72 ondisk = 0) v8 ==== 152+0+8 (crc 0 0 0) 0x7f4864045040 con 0x7f486c007030 2023-10-05T19:28:29.317 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.310+0000 7f487960d700 1 -- 172.21.15.167:0/2494787467 <== osd.2 v2:172.21.15.167:6816/30037 19 ==== osd_op_reply(108 notify.0 [notify-ack] v0'0 uv72 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f4864045040 con 0x7f486c007030 2023-10-05T19:28:29.317 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.310+0000 7f487960d700 1 -- 172.21.15.167:0/2494787467 <== osd.2 v2:172.21.15.167:6816/30037 20 ==== watch-notify(notify_complete (2) cookie 94662980328752 notify 128849018884 ret 0) v3 ==== 42+0+48 (crc 0 0 0) 0x5618714a34e0 con 0x7f486c007030 2023-10-05T19:28:29.317 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.310+0000 7f487c2d9e40 1 -- 172.21.15.167:0/2494787467 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:109 8.8 8:17d68d04:users.swift::foo%3afoo1:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x56187150d210 con 0x56187144d730 2023-10-05T19:28:29.319 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.314+0000 7f487a60f700 1 -- 172.21.15.167:0/2494787467 <== osd.1 v2:172.21.15.167:6800/30035 42 ==== osd_op_reply(109 foo:foo1 [delete] v30'2 uv2 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f487403d040 con 0x56187144d730 2023-10-05T19:28:29.319 INFO:teuthology.orchestra.run.smithi167.stdout:{ 2023-10-05T19:28:29.319 INFO:teuthology.orchestra.run.smithi167.stdout: "user_id": "foo", 2023-10-05T19:28:29.319 INFO:teuthology.orchestra.run.smithi167.stdout: "display_name": "Foo", 2023-10-05T19:28:29.319 INFO:teuthology.orchestra.run.smithi167.stdout: "email": "foo@foo.com", 2023-10-05T19:28:29.319 INFO:teuthology.orchestra.run.smithi167.stdout: "suspended": 0, 2023-10-05T19:28:29.320 INFO:teuthology.orchestra.run.smithi167.stdout: "max_buckets": 4, 2023-10-05T19:28:29.320 INFO:teuthology.orchestra.run.smithi167.stdout: "subusers": [ 2023-10-05T19:28:29.320 INFO:teuthology.orchestra.run.smithi167.stdout: { 2023-10-05T19:28:29.320 INFO:teuthology.orchestra.run.smithi167.stdout: "id": "foo:foo1", 2023-10-05T19:28:29.320 INFO:teuthology.orchestra.run.smithi167.stdout: "permissions": "full-control" 2023-10-05T19:28:29.320 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2023-10-05T19:28:29.320 INFO:teuthology.orchestra.run.smithi167.stdout: { 2023-10-05T19:28:29.320 INFO:teuthology.orchestra.run.smithi167.stdout: "id": "foo:foo2", 2023-10-05T19:28:29.320 INFO:teuthology.orchestra.run.smithi167.stdout: "permissions": "" 2023-10-05T19:28:29.320 INFO:teuthology.orchestra.run.smithi167.stdout: } 2023-10-05T19:28:29.320 INFO:teuthology.orchestra.run.smithi167.stdout: ], 2023-10-05T19:28:29.320 INFO:teuthology.orchestra.run.smithi167.stdout: "keys": [ 2023-10-05T19:28:29.321 INFO:teuthology.orchestra.run.smithi167.stdout: { 2023-10-05T19:28:29.321 INFO:teuthology.orchestra.run.smithi167.stdout: "user": "foo", 2023-10-05T19:28:29.321 INFO:teuthology.orchestra.run.smithi167.stdout: "access_key": "9te6NH5mcdcq0Tc5i8i1", 2023-10-05T19:28:29.321 INFO:teuthology.orchestra.run.smithi167.stdout: "secret_key": "Ny4IOauQoL18Gp2zM7lC1vLmoawgqcYP/YGcWfXu" 2023-10-05T19:28:29.321 INFO:teuthology.orchestra.run.smithi167.stdout: } 2023-10-05T19:28:29.321 INFO:teuthology.orchestra.run.smithi167.stdout: ], 2023-10-05T19:28:29.321 INFO:teuthology.orchestra.run.smithi167.stdout: "swift_keys": [ 2023-10-05T19:28:29.321 INFO:teuthology.orchestra.run.smithi167.stdout: { 2023-10-05T19:28:29.321 INFO:teuthology.orchestra.run.smithi167.stdout: "user": "foo:foo2", 2023-10-05T19:28:29.321 INFO:teuthology.orchestra.run.smithi167.stdout: "secret_key": "ri2VJQcKSYATOY6uaDUX7pxgkW+W1YmC6OCxPHwy" 2023-10-05T19:28:29.321 INFO:teuthology.orchestra.run.smithi167.stdout: } 2023-10-05T19:28:29.321 INFO:teuthology.orchestra.run.smithi167.stdout: ], 2023-10-05T19:28:29.321 INFO:teuthology.orchestra.run.smithi167.stdout: "caps": [], 2023-10-05T19:28:29.322 INFO:teuthology.orchestra.run.smithi167.stdout: "op_mask": "read, write, delete", 2023-10-05T19:28:29.322 INFO:teuthology.orchestra.run.smithi167.stdout: "default_placement": "", 2023-10-05T19:28:29.322 INFO:teuthology.orchestra.run.smithi167.stdout: "default_storage_class": "", 2023-10-05T19:28:29.322 INFO:teuthology.orchestra.run.smithi167.stdout: "placement_tags": [], 2023-10-05T19:28:29.322 INFO:teuthology.orchestra.run.smithi167.stdout: "bucket_quota": { 2023-10-05T19:28:29.322 INFO:teuthology.orchestra.run.smithi167.stdout: "enabled": false, 2023-10-05T19:28:29.322 INFO:teuthology.orchestra.run.smithi167.stdout: "check_on_raw": false, 2023-10-05T19:28:29.322 INFO:teuthology.orchestra.run.smithi167.stdout: "max_size": -1, 2023-10-05T19:28:29.322 INFO:teuthology.orchestra.run.smithi167.stdout: "max_size_kb": 0, 2023-10-05T19:28:29.322 INFO:teuthology.orchestra.run.smithi167.stdout: "max_objects": -1 2023-10-05T19:28:29.322 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2023-10-05T19:28:29.322 INFO:teuthology.orchestra.run.smithi167.stdout: "user_quota": { 2023-10-05T19:28:29.323 INFO:teuthology.orchestra.run.smithi167.stdout: "enabled": false, 2023-10-05T19:28:29.323 INFO:teuthology.orchestra.run.smithi167.stdout: "check_on_raw": false, 2023-10-05T19:28:29.323 INFO:teuthology.orchestra.run.smithi167.stdout: "max_size": -1, 2023-10-05T19:28:29.323 INFO:teuthology.orchestra.run.smithi167.stdout: "max_size_kb": 0, 2023-10-05T19:28:29.323 INFO:teuthology.orchestra.run.smithi167.stdout: "max_objects": -1 2023-10-05T19:28:29.323 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2023-10-05T19:28:29.323 INFO:teuthology.orchestra.run.smithi167.stdout: "temp_url_keys": [], 2023-10-05T19:28:29.323 INFO:teuthology.orchestra.run.smithi167.stdout: "type": "rgw", 2023-10-05T19:28:29.323 INFO:teuthology.orchestra.run.smithi167.stdout: "mfa_ids": [] 2023-10-05T19:28:29.323 INFO:teuthology.orchestra.run.smithi167.stdout:} 2023-10-05T19:28:29.323 INFO:teuthology.orchestra.run.smithi167.stdout: 2023-10-05T19:28:29.324 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.314+0000 7f487c2d9e40 1 -- 172.21.15.167:0/2494787467 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:110 7.1f 7:f95f44c2:::notify.0:head [watch unwatch cookie 94662979786800] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x561871462550 con 0x7f486c007030 2023-10-05T19:28:29.324 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.318+0000 7f487960d700 1 -- 172.21.15.167:0/2494787467 <== osd.2 v2:172.21.15.167:6816/30037 21 ==== osd_op_reply(110 notify.0 [watch unwatch cookie 94662979786800] v30'74 uv72 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f4864045040 con 0x7f486c007030 2023-10-05T19:28:29.324 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.318+0000 7f487c2d9e40 20 remove_watcher() i=0 2023-10-05T19:28:29.324 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.318+0000 7f487c2d9e40 2 removed watcher, disabling cache 2023-10-05T19:28:29.324 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.318+0000 7f487c2d9e40 1 -- 172.21.15.167:0/2494787467 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:111 7.0 7:05bf5b68:::notify.1:head [watch unwatch cookie 94662979791712] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x561871462550 con 0x56187144d730 2023-10-05T19:28:29.325 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.318+0000 7f487a60f700 1 -- 172.21.15.167:0/2494787467 <== osd.1 v2:172.21.15.167:6800/30035 43 ==== osd_op_reply(111 notify.1 [watch unwatch cookie 94662979791712] v30'74 uv72 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f487403d040 con 0x56187144d730 2023-10-05T19:28:29.325 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.318+0000 7f487c2d9e40 20 remove_watcher() i=1 2023-10-05T19:28:29.325 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.318+0000 7f487c2d9e40 1 -- 172.21.15.167:0/2494787467 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:112 7.15 7:a93a5511:::notify.2:head [watch unwatch cookie 94662979793712] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x561871462550 con 0x561871484470 2023-10-05T19:28:29.327 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.322+0000 7f4879e0e700 1 -- 172.21.15.167:0/2494787467 <== osd.3 v2:172.21.15.167:6808/30036 24 ==== osd_op_reply(112 notify.2 [watch unwatch cookie 94662979793712] v30'74 uv72 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f486c062040 con 0x561871484470 2023-10-05T19:28:29.327 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.322+0000 7f487c2d9e40 20 remove_watcher() i=2 2023-10-05T19:28:29.328 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.322+0000 7f487c2d9e40 1 -- 172.21.15.167:0/2494787467 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:113 7.e 7:7759931f:::notify.3:head [watch unwatch cookie 94662979799776] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x561871462550 con 0x7f486c007030 2023-10-05T19:28:29.329 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.322+0000 7f487960d700 1 -- 172.21.15.167:0/2494787467 <== osd.2 v2:172.21.15.167:6816/30037 22 ==== osd_op_reply(113 notify.3 [watch unwatch cookie 94662979799776] v30'74 uv72 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f4864045040 con 0x7f486c007030 2023-10-05T19:28:29.330 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.322+0000 7f487c2d9e40 20 remove_watcher() i=3 2023-10-05T19:28:29.330 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.322+0000 7f487c2d9e40 1 -- 172.21.15.167:0/2494787467 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:114 7.d 7:b4812045:::notify.4:head [watch unwatch cookie 94662979800880] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x561871462550 con 0x56187144d730 2023-10-05T19:28:29.332 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.326+0000 7f487a60f700 1 -- 172.21.15.167:0/2494787467 <== osd.1 v2:172.21.15.167:6800/30035 44 ==== osd_op_reply(114 notify.4 [watch unwatch cookie 94662979800880] v30'74 uv72 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f487403d040 con 0x56187144d730 2023-10-05T19:28:29.332 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.326+0000 7f487c2d9e40 20 remove_watcher() i=4 2023-10-05T19:28:29.332 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.326+0000 7f487c2d9e40 1 -- 172.21.15.167:0/2494787467 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:115 7.3 7:c609908c:::notify.5:head [watch unwatch cookie 94662979804912] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x56187148cfd0 con 0x7f4864011040 2023-10-05T19:28:29.335 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.330+0000 7f487a60f700 1 -- 172.21.15.167:0/2494787467 <== osd.0 v2:172.21.15.118:6800/27690 29 ==== osd_op_reply(115 notify.5 [watch unwatch cookie 94662979804912] v30'74 uv72 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f487403d040 con 0x7f4864011040 2023-10-05T19:28:29.335 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.330+0000 7f487c2d9e40 20 remove_watcher() i=5 2023-10-05T19:28:29.335 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.330+0000 7f487c2d9e40 1 -- 172.21.15.167:0/2494787467 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:116 7.14 7:2b04a3e9:::notify.6:head [watch unwatch cookie 94662979803088] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x56187148cfd0 con 0x7f4864011040 2023-10-05T19:28:29.337 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.330+0000 7f487a60f700 1 -- 172.21.15.167:0/2494787467 <== osd.0 v2:172.21.15.118:6800/27690 30 ==== osd_op_reply(116 notify.6 [watch unwatch cookie 94662979803088] v30'74 uv72 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f487403d040 con 0x7f4864011040 2023-10-05T19:28:29.338 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.330+0000 7f487c2d9e40 20 remove_watcher() i=6 2023-10-05T19:28:29.338 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.330+0000 7f487c2d9e40 1 -- 172.21.15.167:0/2494787467 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:117 7.9 7:93e5b521:::notify.7:head [watch unwatch cookie 94662979806080] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x561871462550 con 0x7f4864011040 2023-10-05T19:28:29.340 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.334+0000 7f487a60f700 1 -- 172.21.15.167:0/2494787467 <== osd.0 v2:172.21.15.118:6800/27690 31 ==== osd_op_reply(117 notify.7 [watch unwatch cookie 94662979806080] v30'74 uv72 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f487403d040 con 0x7f4864011040 2023-10-05T19:28:29.340 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.334+0000 7f487c2d9e40 20 remove_watcher() i=7 2023-10-05T19:28:29.342 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.338+0000 7f484a7ec700 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2023-10-05T19:28:29.342 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.338+0000 7f487c2d9e40 1 -- 172.21.15.167:0/2494787467 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:118 6.4 6:22d26bf9:::data_loggenerations_metadata:head [watch unwatch cookie 94662979809632] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x561871462550 con 0x56187144d730 2023-10-05T19:28:29.345 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.338+0000 7f487a60f700 1 -- 172.21.15.167:0/2494787467 <== osd.1 v2:172.21.15.167:6800/30035 45 ==== osd_op_reply(118 data_loggenerations_metadata [watch unwatch cookie 94662979809632] v30'83 uv1 ondisk = 0) v8 ==== 172+0+0 (crc 0 0 0) 0x7f487403d040 con 0x56187144d730 2023-10-05T19:28:29.345 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.338+0000 7f487c2d9e40 1 -- 172.21.15.167:0/2494787467 >> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] conn(0x7f4864011040 msgr2=0x7f4864014950 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-05T19:28:29.345 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.338+0000 7f487c2d9e40 1 --2- 172.21.15.167:0/2494787467 >> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] conn(0x7f4864011040 0x7f4864014950 crc :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-05T19:28:29.345 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.338+0000 7f487c2d9e40 1 -- 172.21.15.167:0/2494787467 >> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] conn(0x56187144d730 msgr2=0x5618714805f0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-05T19:28:29.346 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.338+0000 7f487c2d9e40 1 --2- 172.21.15.167:0/2494787467 >> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] conn(0x56187144d730 0x5618714805f0 crc :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-05T19:28:29.346 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.338+0000 7f487c2d9e40 1 -- 172.21.15.167:0/2494787467 >> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] conn(0x7f486c007030 msgr2=0x7f486c01c410 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-05T19:28:29.346 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.338+0000 7f487c2d9e40 1 --2- 172.21.15.167:0/2494787467 >> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] conn(0x7f486c007030 0x7f486c01c410 crc :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-05T19:28:29.346 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.338+0000 7f487c2d9e40 1 -- 172.21.15.167:0/2494787467 >> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] conn(0x561871484470 msgr2=0x561871486850 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-05T19:28:29.346 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.338+0000 7f487c2d9e40 1 --2- 172.21.15.167:0/2494787467 >> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] conn(0x561871484470 0x561871486850 crc :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-05T19:28:29.346 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.338+0000 7f487c2d9e40 1 -- 172.21.15.167:0/2494787467 >> [v2:172.21.15.167:6824/29889,v1:172.21.15.167:6825/29889] conn(0x7f4860030c00 msgr2=0x7f48600330b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-05T19:28:29.346 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.338+0000 7f487c2d9e40 1 --2- 172.21.15.167:0/2494787467 >> [v2:172.21.15.167:6824/29889,v1:172.21.15.167:6825/29889] conn(0x7f4860030c00 0x7f48600330b0 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7f4864001560 tx=0x7f4864006650 comp rx=0 tx=0).stop 2023-10-05T19:28:29.346 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.338+0000 7f487c2d9e40 1 -- 172.21.15.167:0/2494787467 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x56187144ce20 msgr2=0x561871474b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-05T19:28:29.346 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.338+0000 7f487c2d9e40 1 --2- 172.21.15.167:0/2494787467 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x56187144ce20 0x561871474b20 secure :-1 s=READY pgs=281 cs=0 l=1 rev1=1 crypto rx=0x7f486c01e040 tx=0x7f486c003050 comp rx=0 tx=0).stop 2023-10-05T19:28:29.346 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.338+0000 7f487a60f700 1 -- 172.21.15.167:0/2494787467 reap_dead start 2023-10-05T19:28:29.346 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.338+0000 7f487c2d9e40 1 -- 172.21.15.167:0/2494787467 shutdown_connections 2023-10-05T19:28:29.346 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.338+0000 7f487c2d9e40 1 -- 172.21.15.167:0/2494787467 >> 172.21.15.167:0/2494787467 conn(0x561871448840 msgr2=0x561871466330 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-05T19:28:29.347 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.338+0000 7f487c2d9e40 1 -- 172.21.15.167:0/2494787467 shutdown_connections 2023-10-05T19:28:29.347 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.338+0000 7f487c2d9e40 1 -- 172.21.15.167:0/2494787467 wait complete. 2023-10-05T19:28:29.347 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.342+0000 7f487c2d9e40 1 -- 172.21.15.167:0/3268647138 >> [v2:172.21.15.167:6824/29889,v1:172.21.15.167:6825/29889] conn(0x7f47c0030c00 msgr2=0x7f47c00330b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-05T19:28:29.347 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.342+0000 7f487c2d9e40 1 --2- 172.21.15.167:0/3268647138 >> [v2:172.21.15.167:6824/29889,v1:172.21.15.167:6825/29889] conn(0x7f47c0030c00 0x7f47c00330b0 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x56187148f700 tx=0x7f487400e040 comp rx=0 tx=0).stop 2023-10-05T19:28:29.348 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.342+0000 7f487c2d9e40 1 -- 172.21.15.167:0/3268647138 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x56187149d2b0 msgr2=0x5618714a2ee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-05T19:28:29.348 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.342+0000 7f487c2d9e40 1 --2- 172.21.15.167:0/3268647138 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x56187149d2b0 0x5618714a2ee0 secure :-1 s=READY pgs=283 cs=0 l=1 rev1=1 crypto rx=0x7f48640060a0 tx=0x7f486403c000 comp rx=0 tx=0).stop 2023-10-05T19:28:29.348 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.342+0000 7f487c2d9e40 1 -- 172.21.15.167:0/3268647138 shutdown_connections 2023-10-05T19:28:29.348 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.342+0000 7f487c2d9e40 1 --2- 172.21.15.167:0/3268647138 >> [v2:172.21.15.167:6824/29889,v1:172.21.15.167:6825/29889] conn(0x7f47c0030c00 0x7f47c00330b0 unknown :-1 s=CLOSED pgs=101 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-05T19:28:29.348 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.342+0000 7f487c2d9e40 1 --2- 172.21.15.167:0/3268647138 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x56187149d2b0 0x5618714a2ee0 unknown :-1 s=CLOSED pgs=283 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-05T19:28:29.348 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.342+0000 7f487c2d9e40 1 -- 172.21.15.167:0/3268647138 >> 172.21.15.167:0/3268647138 conn(0x561871498670 msgr2=0x561871496710 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-05T19:28:29.348 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.342+0000 7f487c2d9e40 1 -- 172.21.15.167:0/3268647138 shutdown_connections 2023-10-05T19:28:29.348 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.342+0000 7f487c2d9e40 1 -- 172.21.15.167:0/3268647138 wait complete. 2023-10-05T19:28:29.352 DEBUG:tasks.util.rgw: json result: {'user_id': 'foo', 'display_name': 'Foo', 'email': 'foo@foo.com', 'suspended': 0, 'max_buckets': 4, 'subusers': [{'id': 'foo:foo1', 'permissions': 'full-control'}, {'id': 'foo:foo2', 'permissions': ''}], 'keys': [{'user': 'foo', 'access_key': '9te6NH5mcdcq0Tc5i8i1', 'secret_key': 'Ny4IOauQoL18Gp2zM7lC1vLmoawgqcYP/YGcWfXu'}], 'swift_keys': [{'user': 'foo:foo2', 'secret_key': 'ri2VJQcKSYATOY6uaDUX7pxgkW+W1YmC6OCxPHwy'}], 'caps': [], 'op_mask': 'read, write, delete', 'default_placement': '', 'default_storage_class': '', 'placement_tags': [], 'bucket_quota': {'enabled': False, 'check_on_raw': False, 'max_size': -1, 'max_size_kb': 0, 'max_objects': -1}, 'user_quota': {'enabled': False, 'check_on_raw': False, 'max_size': -1, 'max_size_kb': 0, 'max_objects': -1}, 'temp_url_keys': [], 'type': 'rgw', 'mfa_ids': []} 2023-10-05T19:28:29.352 INFO:tasks.util.rgw:rgwadmin: client.0 : ['subuser', 'rm', '--subuser', 'foo:foo1'] 2023-10-05T19:28:29.352 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'ceph', 'subuser', 'rm', '--subuser', 'foo:foo1'] 2023-10-05T19:28:29.352 DEBUG:teuthology.orchestra.run.smithi167:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster ceph subuser rm --subuser foo:foo1 2023-10-05T19:28:29.390 INFO:teuthology.orchestra.run.smithi167.stderr:ignoring --setuser ceph since I am not root 2023-10-05T19:28:29.391 INFO:teuthology.orchestra.run.smithi167.stderr:ignoring --setgroup ceph since I am not root 2023-10-05T19:28:29.391 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.386+0000 7f35b13bae40 1 Processor -- start 2023-10-05T19:28:29.391 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.386+0000 7f35b13bae40 1 -- start start 2023-10-05T19:28:29.392 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.386+0000 7f35b13bae40 1 --2- >> v2:172.21.15.118:3300/0 conn(0x55590b2dbe30 0x55590b2dc200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-05T19:28:29.392 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.386+0000 7f35b13bae40 1 -- --> v1:172.21.15.118:6789/0 -- auth(proto 0 26 bytes epoch 0) v1 -- 0x55590b228bd0 con 0x55590b2dc740 2023-10-05T19:28:29.392 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.386+0000 7f35b13bae40 1 -- --> v2:172.21.15.118:3300/0 -- mon_getmap magic: 0 v1 -- 0x55590b1e5ee0 con 0x55590b2dbe30 2023-10-05T19:28:29.392 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.386+0000 7f35aeeef700 1 --2- >> v2:172.21.15.118:3300/0 conn(0x55590b2dbe30 0x55590b2dc200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-05T19:28:29.392 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.386+0000 7f35ae6ee700 1 --1- >> v1:172.21.15.118:6789/0 conn(0x55590b2dc740 0x55590b2dd1e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.118:6789/0 says I am v1:172.21.15.167:56512/0 (socket says 172.21.15.167:56512) 2023-10-05T19:28:29.392 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.386+0000 7f35ae6ee700 1 -- 172.21.15.167:0/2015647164 learned_addr learned my addr 172.21.15.167:0/2015647164 (peer_addr_for_me v1:172.21.15.167:0/0) 2023-10-05T19:28:29.393 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.386+0000 7f35aeeef700 1 -- 172.21.15.167:0/2015647164 >> v1:172.21.15.118:6789/0 conn(0x55590b2dc740 legacy=0x55590b2dd1e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-05T19:28:29.393 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.386+0000 7f35adeed700 1 -- 172.21.15.167:0/2015647164 <== mon.0 v1:172.21.15.118:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4003675455 0 0) 0x55590b228bd0 con 0x55590b2dc740 2023-10-05T19:28:29.393 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.386+0000 7f35aeeef700 1 -- 172.21.15.167:0/2015647164 --> v2:172.21.15.118:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55590b2dd8f0 con 0x55590b2dbe30 2023-10-05T19:28:29.394 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.386+0000 7f35aeeef700 1 --2- 172.21.15.167:0/2015647164 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55590b2dbe30 0x55590b2dc200 secure :-1 s=READY pgs=285 cs=0 l=1 rev1=1 crypto rx=0x7f35a000a5d0 tx=0x7f35a0005a50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2e86293019dc2ae5 server_cookie=0 in_seq=0 out_seq=0 2023-10-05T19:28:29.394 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.386+0000 7f35adeed700 1 -- 172.21.15.167:0/2015647164 <== mon.0 v2:172.21.15.118:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f35a00086a0 con 0x55590b2dbe30 2023-10-05T19:28:29.394 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.386+0000 7f35adeed700 1 -- 172.21.15.167:0/2015647164 <== mon.0 v2:172.21.15.118:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f35a0008800 con 0x55590b2dbe30 2023-10-05T19:28:29.394 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.386+0000 7f35adeed700 1 -- 172.21.15.167:0/2015647164 <== mon.0 v2:172.21.15.118:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f35a0008ad0 con 0x55590b2dbe30 2023-10-05T19:28:29.395 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.390+0000 7f35b13bae40 1 -- 172.21.15.167:0/2015647164 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55590b2dbe30 msgr2=0x55590b2dc200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-05T19:28:29.395 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.390+0000 7f35b13bae40 1 --2- 172.21.15.167:0/2015647164 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55590b2dbe30 0x55590b2dc200 secure :-1 s=READY pgs=285 cs=0 l=1 rev1=1 crypto rx=0x7f35a000a5d0 tx=0x7f35a0005a50 comp rx=0 tx=0).stop 2023-10-05T19:28:29.395 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.390+0000 7f35b13bae40 1 -- 172.21.15.167:0/2015647164 shutdown_connections 2023-10-05T19:28:29.395 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.390+0000 7f35b13bae40 1 --2- 172.21.15.167:0/2015647164 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55590b2dbe30 0x55590b2dc200 unknown :-1 s=CLOSED pgs=285 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-05T19:28:29.395 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.390+0000 7f35b13bae40 1 -- 172.21.15.167:0/2015647164 >> 172.21.15.167:0/2015647164 conn(0x55590b2d7850 msgr2=0x55590b2d9c60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-05T19:28:29.395 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.390+0000 7f35b13bae40 1 -- 172.21.15.167:0/2015647164 shutdown_connections 2023-10-05T19:28:29.395 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.390+0000 7f35b13bae40 1 -- 172.21.15.167:0/2015647164 wait complete. 2023-10-05T19:28:29.396 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.390+0000 7f35b13bae40 1 Processor -- start 2023-10-05T19:28:29.396 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.390+0000 7f35b13bae40 1 -- start start 2023-10-05T19:28:29.396 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.390+0000 7f35b13bae40 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55590b2dbe30 0x55590b2518e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-05T19:28:29.396 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.390+0000 7f35b13bae40 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x55590b1a9e30 con 0x55590b2dbe30 2023-10-05T19:28:29.397 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.390+0000 7f35aeeef700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55590b2dbe30 0x55590b2518e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-05T19:28:29.397 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.390+0000 7f35aeeef700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55590b2dbe30 0x55590b2518e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.118:3300/0 says I am v2:172.21.15.167:57914/0 (socket says 172.21.15.167:57914) 2023-10-05T19:28:29.397 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.390+0000 7f35aeeef700 1 -- 172.21.15.167:0/207058475 learned_addr learned my addr 172.21.15.167:0/207058475 (peer_addr_for_me v2:172.21.15.167:0/0) 2023-10-05T19:28:29.398 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.390+0000 7f35aeeef700 1 -- 172.21.15.167:0/207058475 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f35a0005cf0 con 0x55590b2dbe30 2023-10-05T19:28:29.398 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.390+0000 7f35aeeef700 1 --2- 172.21.15.167:0/207058475 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55590b2dbe30 0x55590b2518e0 secure :-1 s=READY pgs=286 cs=0 l=1 rev1=1 crypto rx=0x7f35a0005a90 tx=0x7f35a0015000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=12197e67470dd4a3 server_cookie=0 in_seq=0 out_seq=0 2023-10-05T19:28:29.398 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.390+0000 7f35aceeb700 1 -- 172.21.15.167:0/207058475 <== mon.0 v2:172.21.15.118:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f35a0008a40 con 0x55590b2dbe30 2023-10-05T19:28:29.398 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.390+0000 7f35aceeb700 1 -- 172.21.15.167:0/207058475 <== mon.0 v2:172.21.15.118:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f35a0008ba0 con 0x55590b2dbe30 2023-10-05T19:28:29.398 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.390+0000 7f35aceeb700 1 -- 172.21.15.167:0/207058475 <== mon.0 v2:172.21.15.118:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f35a0008e70 con 0x55590b2dbe30 2023-10-05T19:28:29.399 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.394+0000 7f35b13bae40 1 -- 172.21.15.167:0/207058475 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55590b2dbe30 msgr2=0x55590b2518e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-05T19:28:29.399 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.394+0000 7f35b13bae40 1 --2- 172.21.15.167:0/207058475 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55590b2dbe30 0x55590b2518e0 secure :-1 s=READY pgs=286 cs=0 l=1 rev1=1 crypto rx=0x7f35a0005a90 tx=0x7f35a0015000 comp rx=0 tx=0).stop 2023-10-05T19:28:29.399 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.394+0000 7f35b13bae40 1 -- 172.21.15.167:0/207058475 shutdown_connections 2023-10-05T19:28:29.399 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.394+0000 7f35b13bae40 1 --2- 172.21.15.167:0/207058475 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55590b2dbe30 0x55590b2518e0 unknown :-1 s=CLOSED pgs=286 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-05T19:28:29.399 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.394+0000 7f35b13bae40 1 -- 172.21.15.167:0/207058475 >> 172.21.15.167:0/207058475 conn(0x55590b2d7850 msgr2=0x55590b2f6850 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-05T19:28:29.399 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.394+0000 7f35b13bae40 1 -- 172.21.15.167:0/207058475 shutdown_connections 2023-10-05T19:28:29.399 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.394+0000 7f35b13bae40 1 -- 172.21.15.167:0/207058475 wait complete. 2023-10-05T19:28:29.400 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.394+0000 7f35b13bae40 1 Processor -- start 2023-10-05T19:28:29.400 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.394+0000 7f35b13bae40 1 -- start start 2023-10-05T19:28:29.400 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.394+0000 7f35b13bae40 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55590b2dbe30 0x55590b303ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-05T19:28:29.400 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.394+0000 7f35b13bae40 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x55590b17ced0 con 0x55590b2dbe30 2023-10-05T19:28:29.400 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.394+0000 7f35aeeef700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55590b2dbe30 0x55590b303ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-05T19:28:29.400 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.394+0000 7f35aeeef700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55590b2dbe30 0x55590b303ac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.118:3300/0 says I am v2:172.21.15.167:57924/0 (socket says 172.21.15.167:57924) 2023-10-05T19:28:29.400 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.394+0000 7f35aeeef700 1 -- 172.21.15.167:0/95972159 learned_addr learned my addr 172.21.15.167:0/95972159 (peer_addr_for_me v2:172.21.15.167:0/0) 2023-10-05T19:28:29.401 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.394+0000 7f35aeeef700 1 -- 172.21.15.167:0/95972159 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55590b2dd8f0 con 0x55590b2dbe30 2023-10-05T19:28:29.401 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.394+0000 7f35aeeef700 1 --2- 172.21.15.167:0/95972159 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55590b2dbe30 0x55590b303ac0 secure :-1 s=READY pgs=287 cs=0 l=1 rev1=1 crypto rx=0x7f35a001e040 tx=0x7f35a0003050 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-05T19:28:29.401 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.394+0000 7f35a6ffd700 1 -- 172.21.15.167:0/95972159 <== mon.0 v2:172.21.15.118:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f35a0012b50 con 0x55590b2dbe30 2023-10-05T19:28:29.401 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.394+0000 7f35a6ffd700 1 -- 172.21.15.167:0/95972159 <== mon.0 v2:172.21.15.118:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f35a002a070 con 0x55590b2dbe30 2023-10-05T19:28:29.402 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.394+0000 7f35b13bae40 1 -- 172.21.15.167:0/95972159 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x55590b304000 con 0x55590b2dbe30 2023-10-05T19:28:29.402 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.394+0000 7f35a6ffd700 1 -- 172.21.15.167:0/95972159 <== mon.0 v2:172.21.15.118:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f35a0012e10 con 0x55590b2dbe30 2023-10-05T19:28:29.402 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.394+0000 7f35b13bae40 1 -- 172.21.15.167:0/95972159 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55590b3044a0 con 0x55590b2dbe30 2023-10-05T19:28:29.402 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.398+0000 7f35b13bae40 20 rados->read ofs=0 len=0 2023-10-05T19:28:29.402 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.398+0000 7f35a6ffd700 1 -- 172.21.15.167:0/95972159 <== mon.0 v2:172.21.15.118:3300/0 4 ==== mgrmap(e 5) v1 ==== 40329+0+0 (secure 0 0 0) 0x7f35a0019070 con 0x55590b2dbe30 2023-10-05T19:28:29.404 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.398+0000 7f35a6ffd700 1 --2- 172.21.15.167:0/95972159 >> [v2:172.21.15.167:6824/29889,v1:172.21.15.167:6825/29889] conn(0x7f3590030c00 0x7f35900330b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-05T19:28:29.404 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.398+0000 7f35a6ffd700 1 -- 172.21.15.167:0/95972159 <== mon.0 v2:172.21.15.118:3300/0 5 ==== osd_map(30..30 src has 1..30) v4 ==== 6057+0+0 (secure 0 0 0) 0x7f35a002c070 con 0x55590b2dbe30 2023-10-05T19:28:29.404 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.398+0000 7f35ae6ee700 1 --2- 172.21.15.167:0/95972159 >> [v2:172.21.15.167:6824/29889,v1:172.21.15.167:6825/29889] conn(0x7f3590030c00 0x7f35900330b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-05T19:28:29.404 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.398+0000 7f35b13bae40 1 --2- 172.21.15.167:0/95972159 >> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] conn(0x55590b2dc740 0x55590b30f5f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-05T19:28:29.404 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.398+0000 7f35b13bae40 1 -- 172.21.15.167:0/95972159 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:1 5.12 5:49953fa1:::default.realm:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x55590b30fca0 con 0x55590b2dc740 2023-10-05T19:28:29.405 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.398+0000 7f35af6f0700 1 --2- 172.21.15.167:0/95972159 >> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] conn(0x55590b2dc740 0x55590b30f5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-05T19:28:29.405 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.398+0000 7f35ae6ee700 1 --2- 172.21.15.167:0/95972159 >> [v2:172.21.15.167:6824/29889,v1:172.21.15.167:6825/29889] conn(0x7f3590030c00 0x7f35900330b0 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7f3598001560 tx=0x7f3598006650 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-05T19:28:29.405 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.398+0000 7f35af6f0700 1 --2- 172.21.15.167:0/95972159 >> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] conn(0x55590b2dc740 0x55590b30f5f0 crc :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-05T19:28:29.405 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.398+0000 7f35af6f0700 1 -- 172.21.15.167:0/95972159 <== osd.1 v2:172.21.15.167:6800/30035 1 ==== osd_op_reply(1 default.realm [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 ==== 157+0+0 (crc 0 0 0) 0x7f35a800a500 con 0x55590b2dc740 2023-10-05T19:28:29.405 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.398+0000 7f35b13bae40 20 rados_obj.operate() r=-2 bl.length=0 2023-10-05T19:28:29.406 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.398+0000 7f35b13bae40 20 realm 2023-10-05T19:28:29.406 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.398+0000 7f35b13bae40 20 rados->read ofs=0 len=0 2023-10-05T19:28:29.406 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.398+0000 7f35b13bae40 1 -- 172.21.15.167:0/95972159 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:2 5.12 5:49953fa1:::default.realm:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x55590b30cc50 con 0x55590b2dc740 2023-10-05T19:28:29.406 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.402+0000 7f35af6f0700 1 -- 172.21.15.167:0/95972159 <== osd.1 v2:172.21.15.167:6800/30035 2 ==== osd_op_reply(2 default.realm [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 ==== 157+0+0 (crc 0 0 0) 0x7f35a800a500 con 0x55590b2dc740 2023-10-05T19:28:29.407 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.402+0000 7f35b13bae40 20 rados_obj.operate() r=-2 bl.length=0 2023-10-05T19:28:29.407 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.402+0000 7f35b13bae40 4 RGWPeriod::init failed to init realm id : (2) No such file or directory 2023-10-05T19:28:29.407 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.402+0000 7f35b13bae40 20 rados->read ofs=0 len=0 2023-10-05T19:28:29.407 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.402+0000 7f35b13bae40 1 -- 172.21.15.167:0/95972159 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:3 5.12 5:49953fa1:::default.realm:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x55590b310780 con 0x55590b2dc740 2023-10-05T19:28:29.407 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.402+0000 7f35af6f0700 1 -- 172.21.15.167:0/95972159 <== osd.1 v2:172.21.15.167:6800/30035 3 ==== osd_op_reply(3 default.realm [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 ==== 157+0+0 (crc 0 0 0) 0x7f35a800a500 con 0x55590b2dc740 2023-10-05T19:28:29.407 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.402+0000 7f35b13bae40 20 rados_obj.operate() r=-2 bl.length=0 2023-10-05T19:28:29.407 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.402+0000 7f35b13bae40 20 rados->read ofs=0 len=0 2023-10-05T19:28:29.407 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.402+0000 7f35b13bae40 1 -- 172.21.15.167:0/95972159 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:4 5.f 5:f43b6ece:::zone_names.default:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x55590b310390 con 0x55590b2dc740 2023-10-05T19:28:29.408 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.402+0000 7f35af6f0700 1 -- 172.21.15.167:0/95972159 <== osd.1 v2:172.21.15.167:6800/30035 4 ==== osd_op_reply(4 zone_names.default [read 0~46 out=46b] v0'0 uv1 ondisk = 0) v8 ==== 162+0+46 (crc 0 0 0) 0x7f35a800a500 con 0x55590b2dc740 2023-10-05T19:28:29.408 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.402+0000 7f35b13bae40 20 rados_obj.operate() r=0 bl.length=46 2023-10-05T19:28:29.409 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.402+0000 7f35b13bae40 20 rados->read ofs=0 len=0 2023-10-05T19:28:29.409 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.402+0000 7f35b13bae40 1 -- 172.21.15.167:0/95972159 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:5 5.1d 5:b8374eb4:::zone_info.9a9df965-6612-45d4-a45f-6e3c889bacc2:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x55590b311610 con 0x55590b2dc740 2023-10-05T19:28:29.409 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.402+0000 7f35af6f0700 1 -- 172.21.15.167:0/95972159 <== osd.1 v2:172.21.15.167:6800/30035 5 ==== osd_op_reply(5 zone_info.9a9df965-6612-45d4-a45f-6e3c889bacc2 [read 0~874 out=874b] v0'0 uv2 ondisk = 0) v8 ==== 190+0+874 (crc 0 0 0) 0x7f35a800a500 con 0x55590b2dc740 2023-10-05T19:28:29.409 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.402+0000 7f35b13bae40 20 rados_obj.operate() r=0 bl.length=874 2023-10-05T19:28:29.409 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.402+0000 7f35b13bae40 20 searching for the correct realm 2023-10-05T19:28:29.409 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.402+0000 7f35b13bae40 1 --2- 172.21.15.167:0/95972159 >> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] conn(0x55590b3133d0 0x55590b3157b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-05T19:28:29.409 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.402+0000 7f35b13bae40 1 -- 172.21.15.167:0/95972159 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:6 5.0 5:00000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x55590b315cf0 con 0x55590b3133d0 2023-10-05T19:28:29.409 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.402+0000 7f35aeeef700 1 --2- 172.21.15.167:0/95972159 >> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] conn(0x55590b3133d0 0x55590b3157b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-05T19:28:29.410 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.402+0000 7f35aeeef700 1 --2- 172.21.15.167:0/95972159 >> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] conn(0x55590b3133d0 0x55590b3157b0 crc :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-05T19:28:29.411 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.406+0000 7f35aeeef700 1 -- 172.21.15.167:0/95972159 <== osd.3 v2:172.21.15.167:6808/30036 1 ==== osd_op_reply(6 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f35a0042040 con 0x55590b3133d0 2023-10-05T19:28:29.411 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.406+0000 7f35aeeef700 1 --2- 172.21.15.167:0/95972159 >> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] conn(0x7f35a0007030 0x7f35a0031480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-05T19:28:29.411 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.406+0000 7f35aeeef700 1 -- 172.21.15.167:0/95972159 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:7 5.10 5:08000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x55590b315cf0 con 0x7f35a0007030 2023-10-05T19:28:29.411 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.406+0000 7f35ae6ee700 1 --2- 172.21.15.167:0/95972159 >> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] conn(0x7f35a0007030 0x7f35a0031480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-05T19:28:29.411 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.406+0000 7f35ae6ee700 1 --2- 172.21.15.167:0/95972159 >> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] conn(0x7f35a0007030 0x7f35a0031480 crc :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-05T19:28:29.412 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.406+0000 7f35ae6ee700 1 -- 172.21.15.167:0/95972159 <== osd.2 v2:172.21.15.167:6816/30037 1 ==== osd_op_reply(7 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f3598004520 con 0x7f35a0007030 2023-10-05T19:28:29.412 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.406+0000 7f35ae6ee700 1 --2- 172.21.15.167:0/95972159 >> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] conn(0x7f3598011040 0x7f3598014950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-05T19:28:29.412 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.406+0000 7f35ae6ee700 1 -- 172.21.15.167:0/95972159 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:8 5.8 5:10000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x55590b315cf0 con 0x7f3598011040 2023-10-05T19:28:29.413 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.406+0000 7f35af6f0700 1 --2- 172.21.15.167:0/95972159 >> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] conn(0x7f3598011040 0x7f3598014950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-05T19:28:29.413 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.406+0000 7f35af6f0700 1 --2- 172.21.15.167:0/95972159 >> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] conn(0x7f3598011040 0x7f3598014950 crc :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-05T19:28:29.414 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.410+0000 7f35af6f0700 1 -- 172.21.15.167:0/95972159 <== osd.0 v2:172.21.15.118:6800/27690 1 ==== osd_op_reply(8 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f35a8014040 con 0x7f3598011040 2023-10-05T19:28:29.414 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.410+0000 7f35af6f0700 1 -- 172.21.15.167:0/95972159 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:9 5.18 5:18000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x55590b315cf0 con 0x55590b2dc740 2023-10-05T19:28:29.415 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.410+0000 7f35af6f0700 1 -- 172.21.15.167:0/95972159 <== osd.1 v2:172.21.15.167:6800/30035 6 ==== osd_op_reply(9 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f35a8014040 con 0x55590b2dc740 2023-10-05T19:28:29.415 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.410+0000 7f35af6f0700 1 -- 172.21.15.167:0/95972159 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:10 5.4 5:20000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x55590b315cf0 con 0x55590b2dc740 2023-10-05T19:28:29.416 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.410+0000 7f35af6f0700 1 -- 172.21.15.167:0/95972159 <== osd.1 v2:172.21.15.167:6800/30035 7 ==== osd_op_reply(10 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f35a8014040 con 0x55590b2dc740 2023-10-05T19:28:29.416 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.410+0000 7f35af6f0700 1 -- 172.21.15.167:0/95972159 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:11 5.14 5:28000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x55590b315cf0 con 0x7f3598011040 2023-10-05T19:28:29.417 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.410+0000 7f35af6f0700 1 -- 172.21.15.167:0/95972159 <== osd.0 v2:172.21.15.118:6800/27690 2 ==== osd_op_reply(11 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f35a8014040 con 0x7f3598011040 2023-10-05T19:28:29.417 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.410+0000 7f35af6f0700 1 -- 172.21.15.167:0/95972159 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:12 5.c 5:30000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x55590b315cf0 con 0x7f3598011040 2023-10-05T19:28:29.417 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.410+0000 7f35af6f0700 1 -- 172.21.15.167:0/95972159 <== osd.0 v2:172.21.15.118:6800/27690 3 ==== osd_op_reply(12 [pgnls start_epoch 30 out=112b] v23'1 uv1 ondisk = 1) v8 ==== 144+0+112 (crc 0 0 0) 0x7f35a8014040 con 0x7f3598011040 2023-10-05T19:28:29.418 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.410+0000 7f35af6f0700 1 -- 172.21.15.167:0/95972159 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:13 5.1c 5:38000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x55590b315cf0 con 0x7f35a0007030 2023-10-05T19:28:29.418 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.414+0000 7f35ae6ee700 1 -- 172.21.15.167:0/95972159 <== osd.2 v2:172.21.15.167:6816/30037 2 ==== osd_op_reply(13 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f3598004520 con 0x7f35a0007030 2023-10-05T19:28:29.418 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.414+0000 7f35ae6ee700 1 -- 172.21.15.167:0/95972159 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:14 5.2 5:40000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x55590b315cf0 con 0x55590b3133d0 2023-10-05T19:28:29.419 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.414+0000 7f35aeeef700 1 -- 172.21.15.167:0/95972159 <== osd.3 v2:172.21.15.167:6808/30036 2 ==== osd_op_reply(14 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f35a0042040 con 0x55590b3133d0 2023-10-05T19:28:29.419 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.414+0000 7f35aeeef700 1 -- 172.21.15.167:0/95972159 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:15 5.12 5:48000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x7f35a0047040 con 0x55590b2dc740 2023-10-05T19:28:29.420 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.414+0000 7f35af6f0700 1 -- 172.21.15.167:0/95972159 <== osd.1 v2:172.21.15.167:6800/30035 8 ==== osd_op_reply(15 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f35a8014040 con 0x55590b2dc740 2023-10-05T19:28:29.420 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.414+0000 7f35af6f0700 1 -- 172.21.15.167:0/95972159 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:16 5.a 5:50000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x7f35a0047040 con 0x7f3598011040 2023-10-05T19:28:29.421 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.414+0000 7f35af6f0700 1 -- 172.21.15.167:0/95972159 <== osd.0 v2:172.21.15.118:6800/27690 4 ==== osd_op_reply(16 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f35a8022040 con 0x7f3598011040 2023-10-05T19:28:29.421 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.414+0000 7f35af6f0700 1 -- 172.21.15.167:0/95972159 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:17 5.1a 5:58000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x7f35a0047040 con 0x55590b2dc740 2023-10-05T19:28:29.421 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.414+0000 7f35af6f0700 1 -- 172.21.15.167:0/95972159 <== osd.1 v2:172.21.15.167:6800/30035 9 ==== osd_op_reply(17 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f35a8022040 con 0x55590b2dc740 2023-10-05T19:28:29.421 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.414+0000 7f35af6f0700 1 -- 172.21.15.167:0/95972159 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:18 5.6 5:60000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x7f35a0047040 con 0x7f35a0007030 2023-10-05T19:28:29.422 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.418+0000 7f35ae6ee700 1 -- 172.21.15.167:0/95972159 <== osd.2 v2:172.21.15.167:6816/30037 3 ==== osd_op_reply(18 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f3598004520 con 0x7f35a0007030 2023-10-05T19:28:29.422 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.418+0000 7f35ae6ee700 1 -- 172.21.15.167:0/95972159 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:19 5.16 5:68000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x7f35a0047040 con 0x55590b3133d0 2023-10-05T19:28:29.423 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.418+0000 7f35aeeef700 1 -- 172.21.15.167:0/95972159 <== osd.3 v2:172.21.15.167:6808/30036 3 ==== osd_op_reply(19 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f35a0042040 con 0x55590b3133d0 2023-10-05T19:28:29.423 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.418+0000 7f35aeeef700 1 -- 172.21.15.167:0/95972159 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:20 5.e 5:70000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x7f35a0047040 con 0x55590b3133d0 2023-10-05T19:28:29.424 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.418+0000 7f35aeeef700 1 -- 172.21.15.167:0/95972159 <== osd.3 v2:172.21.15.167:6808/30036 4 ==== osd_op_reply(20 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f35a0042040 con 0x55590b3133d0 2023-10-05T19:28:29.424 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.418+0000 7f35aeeef700 1 -- 172.21.15.167:0/95972159 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:21 5.1e 5:78000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x7f35a0047040 con 0x7f3598011040 2023-10-05T19:28:29.425 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.418+0000 7f35af6f0700 1 -- 172.21.15.167:0/95972159 <== osd.0 v2:172.21.15.118:6800/27690 5 ==== osd_op_reply(21 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f35a8022040 con 0x7f3598011040 2023-10-05T19:28:29.425 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.418+0000 7f35af6f0700 1 -- 172.21.15.167:0/95972159 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:22 5.1 5:80000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x7f35a0047040 con 0x55590b3133d0 2023-10-05T19:28:29.426 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.418+0000 7f35aeeef700 1 -- 172.21.15.167:0/95972159 <== osd.3 v2:172.21.15.167:6808/30036 5 ==== osd_op_reply(22 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f35a0042040 con 0x55590b3133d0 2023-10-05T19:28:29.426 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.418+0000 7f35aeeef700 1 -- 172.21.15.167:0/95972159 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:23 5.11 5:88000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x7f35a0047040 con 0x55590b3133d0 2023-10-05T19:28:29.427 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.422+0000 7f35aeeef700 1 -- 172.21.15.167:0/95972159 <== osd.3 v2:172.21.15.167:6808/30036 6 ==== osd_op_reply(23 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f35a0042040 con 0x55590b3133d0 2023-10-05T19:28:29.427 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.422+0000 7f35aeeef700 1 -- 172.21.15.167:0/95972159 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:24 5.9 5:90000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x7f35a0047040 con 0x55590b2dc740 2023-10-05T19:28:29.427 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.422+0000 7f35af6f0700 1 -- 172.21.15.167:0/95972159 <== osd.1 v2:172.21.15.167:6800/30035 10 ==== osd_op_reply(24 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f35a8022040 con 0x55590b2dc740 2023-10-05T19:28:29.427 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.422+0000 7f35af6f0700 1 -- 172.21.15.167:0/95972159 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:25 5.19 5:98000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x7f35a0047040 con 0x55590b2dc740 2023-10-05T19:28:29.428 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.422+0000 7f35af6f0700 1 -- 172.21.15.167:0/95972159 <== osd.1 v2:172.21.15.167:6800/30035 11 ==== osd_op_reply(25 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f35a8022040 con 0x55590b2dc740 2023-10-05T19:28:29.428 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.422+0000 7f35af6f0700 1 -- 172.21.15.167:0/95972159 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:26 5.5 5:a0000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x7f35a0047040 con 0x7f3598011040 2023-10-05T19:28:29.429 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.422+0000 7f35af6f0700 1 -- 172.21.15.167:0/95972159 <== osd.0 v2:172.21.15.118:6800/27690 6 ==== osd_op_reply(26 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f35a8022040 con 0x7f3598011040 2023-10-05T19:28:29.429 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.422+0000 7f35af6f0700 1 -- 172.21.15.167:0/95972159 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:27 5.15 5:a8000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x7f35a0047040 con 0x55590b3133d0 2023-10-05T19:28:29.430 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.422+0000 7f35aeeef700 1 -- 172.21.15.167:0/95972159 <== osd.3 v2:172.21.15.167:6808/30036 7 ==== osd_op_reply(27 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f35a0042040 con 0x55590b3133d0 2023-10-05T19:28:29.430 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.422+0000 7f35aeeef700 1 -- 172.21.15.167:0/95972159 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:28 5.d 5:b0000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x7f35a0047040 con 0x7f35a0007030 2023-10-05T19:28:29.430 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.426+0000 7f35ae6ee700 1 -- 172.21.15.167:0/95972159 <== osd.2 v2:172.21.15.167:6816/30037 4 ==== osd_op_reply(28 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f3598004520 con 0x7f35a0007030 2023-10-05T19:28:29.430 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.426+0000 7f35ae6ee700 1 -- 172.21.15.167:0/95972159 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:29 5.1d 5:b8000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x7f35a0047040 con 0x55590b2dc740 2023-10-05T19:28:29.431 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.426+0000 7f35af6f0700 1 -- 172.21.15.167:0/95972159 <== osd.1 v2:172.21.15.167:6800/30035 12 ==== osd_op_reply(29 [pgnls start_epoch 30 out=107b] v30'2 uv2 ondisk = 1) v8 ==== 144+0+107 (crc 0 0 0) 0x7f35a8022040 con 0x55590b2dc740 2023-10-05T19:28:29.431 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.426+0000 7f35af6f0700 1 -- 172.21.15.167:0/95972159 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:30 5.3 5:c0000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x7f35a0047040 con 0x7f3598011040 2023-10-05T19:28:29.432 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.426+0000 7f35af6f0700 1 -- 172.21.15.167:0/95972159 <== osd.0 v2:172.21.15.118:6800/27690 7 ==== osd_op_reply(30 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f35a8022040 con 0x7f3598011040 2023-10-05T19:28:29.432 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.426+0000 7f35af6f0700 1 -- 172.21.15.167:0/95972159 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:31 5.13 5:c8000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x7f35a0047040 con 0x55590b3133d0 2023-10-05T19:28:29.433 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.426+0000 7f35aeeef700 1 -- 172.21.15.167:0/95972159 <== osd.3 v2:172.21.15.167:6808/30036 8 ==== osd_op_reply(31 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f35a0042040 con 0x55590b3133d0 2023-10-05T19:28:29.433 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.426+0000 7f35aeeef700 1 -- 172.21.15.167:0/95972159 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:32 5.b 5:d0000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x7f35a0047040 con 0x7f3598011040 2023-10-05T19:28:29.434 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.430+0000 7f35af6f0700 1 -- 172.21.15.167:0/95972159 <== osd.0 v2:172.21.15.118:6800/27690 8 ==== osd_op_reply(32 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f35a8022040 con 0x7f3598011040 2023-10-05T19:28:29.434 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.430+0000 7f35af6f0700 1 -- 172.21.15.167:0/95972159 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:33 5.1b 5:d8000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x7f35a0047040 con 0x7f35a0007030 2023-10-05T19:28:29.434 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.430+0000 7f35ae6ee700 1 -- 172.21.15.167:0/95972159 <== osd.2 v2:172.21.15.167:6816/30037 5 ==== osd_op_reply(33 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f3598004520 con 0x7f35a0007030 2023-10-05T19:28:29.434 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.430+0000 7f35ae6ee700 1 -- 172.21.15.167:0/95972159 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:34 5.7 5:e0000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x7f35a0047040 con 0x55590b3133d0 2023-10-05T19:28:29.435 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.430+0000 7f35aeeef700 1 -- 172.21.15.167:0/95972159 <== osd.3 v2:172.21.15.167:6808/30036 9 ==== osd_op_reply(34 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f35a0042040 con 0x55590b3133d0 2023-10-05T19:28:29.435 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.430+0000 7f35aeeef700 1 -- 172.21.15.167:0/95972159 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:35 5.17 5:e8000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x7f35a0047040 con 0x7f3598011040 2023-10-05T19:28:29.436 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.430+0000 7f35af6f0700 1 -- 172.21.15.167:0/95972159 <== osd.0 v2:172.21.15.118:6800/27690 9 ==== osd_op_reply(35 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f35a8022040 con 0x7f3598011040 2023-10-05T19:28:29.436 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.430+0000 7f35af6f0700 1 -- 172.21.15.167:0/95972159 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:36 5.f 5:f0000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x7f35a0047040 con 0x55590b2dc740 2023-10-05T19:28:29.437 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.430+0000 7f35af6f0700 1 -- 172.21.15.167:0/95972159 <== osd.1 v2:172.21.15.167:6800/30035 13 ==== osd_op_reply(36 [pgnls start_epoch 30 out=115b] v23'2 uv2 ondisk = 1) v8 ==== 144+0+115 (crc 0 0 0) 0x7f35a802f080 con 0x55590b2dc740 2023-10-05T19:28:29.437 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.430+0000 7f35af6f0700 1 -- 172.21.15.167:0/95972159 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:37 5.1f 5:f8000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x7f35a0047040 con 0x7f35a0007030 2023-10-05T19:28:29.438 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.434+0000 7f35ae6ee700 1 -- 172.21.15.167:0/95972159 <== osd.2 v2:172.21.15.167:6816/30037 6 ==== osd_op_reply(37 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f3598004520 con 0x7f35a0007030 2023-10-05T19:28:29.438 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.434+0000 7f35b13bae40 20 RGWRados::pool_iterate: got zonegroup_info.4f1f719d-9648-417b-b10c-e5d939825419 2023-10-05T19:28:29.439 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.434+0000 7f35b13bae40 20 RGWRados::pool_iterate: got zone_info.9a9df965-6612-45d4-a45f-6e3c889bacc2 2023-10-05T19:28:29.439 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.434+0000 7f35b13bae40 20 RGWRados::pool_iterate: got zone_names.default 2023-10-05T19:28:29.439 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.434+0000 7f35b13bae40 20 RGWRados::pool_iterate: got zonegroups_names.default 2023-10-05T19:28:29.439 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.434+0000 7f35b13bae40 20 rados->read ofs=0 len=0 2023-10-05T19:28:29.439 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.434+0000 7f35b13bae40 1 -- 172.21.15.167:0/95972159 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:38 5.12 5:49953fa1:::default.realm:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x55590b312180 con 0x55590b2dc740 2023-10-05T19:28:29.440 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.434+0000 7f35af6f0700 1 -- 172.21.15.167:0/95972159 <== osd.1 v2:172.21.15.167:6800/30035 14 ==== osd_op_reply(38 default.realm [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 ==== 157+0+0 (crc 0 0 0) 0x7f35a802f080 con 0x55590b2dc740 2023-10-05T19:28:29.440 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.434+0000 7f35b13bae40 20 rados_obj.operate() r=-2 bl.length=0 2023-10-05T19:28:29.440 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.434+0000 7f35b13bae40 20 rados->read ofs=0 len=0 2023-10-05T19:28:29.440 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.434+0000 7f35b13bae40 1 -- 172.21.15.167:0/95972159 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:39 5.f 5:f4c53578:::zonegroups_names.default:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x55590b312a70 con 0x55590b2dc740 2023-10-05T19:28:29.440 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.434+0000 7f35af6f0700 1 -- 172.21.15.167:0/95972159 <== osd.1 v2:172.21.15.167:6800/30035 15 ==== osd_op_reply(39 zonegroups_names.default [read 0~46 out=46b] v0'0 uv2 ondisk = 0) v8 ==== 168+0+46 (crc 0 0 0) 0x7f35a802f080 con 0x55590b2dc740 2023-10-05T19:28:29.440 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.434+0000 7f35b13bae40 20 rados_obj.operate() r=0 bl.length=46 2023-10-05T19:28:29.440 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.434+0000 7f35b13bae40 20 rados->read ofs=0 len=0 2023-10-05T19:28:29.441 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.434+0000 7f35b13bae40 1 -- 172.21.15.167:0/95972159 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:40 5.c 5:30d299a5:::zonegroup_info.4f1f719d-9648-417b-b10c-e5d939825419:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x55590b316130 con 0x7f3598011040 2023-10-05T19:28:29.441 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.434+0000 7f35af6f0700 1 -- 172.21.15.167:0/95972159 <== osd.0 v2:172.21.15.118:6800/27690 10 ==== osd_op_reply(40 zonegroup_info.4f1f719d-9648-417b-b10c-e5d939825419 [read 0~362 out=362b] v0'0 uv1 ondisk = 0) v8 ==== 195+0+362 (crc 0 0 0) 0x7f35a802f080 con 0x7f3598011040 2023-10-05T19:28:29.442 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.434+0000 7f35b13bae40 20 rados_obj.operate() r=0 bl.length=362 2023-10-05T19:28:29.442 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.434+0000 7f35b13bae40 20 rados->read ofs=0 len=0 2023-10-05T19:28:29.442 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.434+0000 7f35b13bae40 1 -- 172.21.15.167:0/95972159 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:41 5.13 5:cbbf6c60:::converted:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x55590b316130 con 0x55590b3133d0 2023-10-05T19:28:29.442 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.434+0000 7f35aeeef700 1 -- 172.21.15.167:0/95972159 <== osd.3 v2:172.21.15.167:6808/30036 10 ==== osd_op_reply(41 converted [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 ==== 153+0+0 (crc 0 0 0) 0x7f35a0042040 con 0x55590b3133d0 2023-10-05T19:28:29.442 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.434+0000 7f35b13bae40 20 rados_obj.operate() r=-2 bl.length=0 2023-10-05T19:28:29.442 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.434+0000 7f35b13bae40 20 rados->read ofs=0 len=0 2023-10-05T19:28:29.442 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.434+0000 7f35b13bae40 1 -- 172.21.15.167:0/95972159 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:42 5.12 5:49953fa1:::default.realm:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x55590b3164e0 con 0x55590b2dc740 2023-10-05T19:28:29.443 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.438+0000 7f35af6f0700 1 -- 172.21.15.167:0/95972159 <== osd.1 v2:172.21.15.167:6800/30035 16 ==== osd_op_reply(42 default.realm [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 ==== 157+0+0 (crc 0 0 0) 0x7f35a802f080 con 0x55590b2dc740 2023-10-05T19:28:29.443 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.438+0000 7f35b13bae40 20 rados_obj.operate() r=-2 bl.length=0 2023-10-05T19:28:29.443 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.438+0000 7f35b13bae40 20 rados->read ofs=0 len=0 2023-10-05T19:28:29.443 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.438+0000 7f35b13bae40 1 -- 172.21.15.167:0/95972159 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:43 5.f 5:f4c53578:::zonegroups_names.default:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x55590b3168e0 con 0x55590b2dc740 2023-10-05T19:28:29.444 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.438+0000 7f35af6f0700 1 -- 172.21.15.167:0/95972159 <== osd.1 v2:172.21.15.167:6800/30035 17 ==== osd_op_reply(43 zonegroups_names.default [read 0~46 out=46b] v0'0 uv2 ondisk = 0) v8 ==== 168+0+46 (crc 0 0 0) 0x7f35a802f080 con 0x55590b2dc740 2023-10-05T19:28:29.444 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.438+0000 7f35b13bae40 20 rados_obj.operate() r=0 bl.length=46 2023-10-05T19:28:29.444 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.438+0000 7f35b13bae40 1 -- 172.21.15.167:0/95972159 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:44 5.0 5:00000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x55590b3178f0 con 0x55590b3133d0 2023-10-05T19:28:29.446 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.442+0000 7f35aeeef700 1 -- 172.21.15.167:0/95972159 <== osd.3 v2:172.21.15.167:6808/30036 11 ==== osd_op_reply(44 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f35a0042040 con 0x55590b3133d0 2023-10-05T19:28:29.446 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.442+0000 7f35aeeef700 1 -- 172.21.15.167:0/95972159 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:45 5.10 5:08000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x55590b3178f0 con 0x7f35a0007030 2023-10-05T19:28:29.447 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.442+0000 7f35ae6ee700 1 -- 172.21.15.167:0/95972159 <== osd.2 v2:172.21.15.167:6816/30037 7 ==== osd_op_reply(45 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f3598004520 con 0x7f35a0007030 2023-10-05T19:28:29.447 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.442+0000 7f35ae6ee700 1 -- 172.21.15.167:0/95972159 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:46 5.8 5:10000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x55590b3178f0 con 0x7f3598011040 2023-10-05T19:28:29.448 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.442+0000 7f35af6f0700 1 -- 172.21.15.167:0/95972159 <== osd.0 v2:172.21.15.118:6800/27690 11 ==== osd_op_reply(46 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f35a802f080 con 0x7f3598011040 2023-10-05T19:28:29.448 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.442+0000 7f35af6f0700 1 -- 172.21.15.167:0/95972159 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:47 5.18 5:18000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x55590b312a70 con 0x55590b2dc740 2023-10-05T19:28:29.448 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.442+0000 7f35af6f0700 1 -- 172.21.15.167:0/95972159 <== osd.1 v2:172.21.15.167:6800/30035 18 ==== osd_op_reply(47 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f35a802f080 con 0x55590b2dc740 2023-10-05T19:28:29.448 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.442+0000 7f35af6f0700 1 -- 172.21.15.167:0/95972159 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:48 5.4 5:20000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x55590b312a70 con 0x55590b2dc740 2023-10-05T19:28:29.449 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.442+0000 7f35af6f0700 1 -- 172.21.15.167:0/95972159 <== osd.1 v2:172.21.15.167:6800/30035 19 ==== osd_op_reply(48 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f35a802f080 con 0x55590b2dc740 2023-10-05T19:28:29.449 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.442+0000 7f35af6f0700 1 -- 172.21.15.167:0/95972159 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:49 5.14 5:28000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x55590b312a70 con 0x7f3598011040 2023-10-05T19:28:29.450 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.446+0000 7f35af6f0700 1 -- 172.21.15.167:0/95972159 <== osd.0 v2:172.21.15.118:6800/27690 12 ==== osd_op_reply(49 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f35a802f080 con 0x7f3598011040 2023-10-05T19:28:29.450 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.446+0000 7f35af6f0700 1 -- 172.21.15.167:0/95972159 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:50 5.c 5:30000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x55590b312a70 con 0x7f3598011040 2023-10-05T19:28:29.451 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.446+0000 7f35af6f0700 1 -- 172.21.15.167:0/95972159 <== osd.0 v2:172.21.15.118:6800/27690 13 ==== osd_op_reply(50 [pgnls start_epoch 30 out=112b] v23'1 uv1 ondisk = 1) v8 ==== 144+0+112 (crc 0 0 0) 0x7f35a802f080 con 0x7f3598011040 2023-10-05T19:28:29.451 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.446+0000 7f35af6f0700 1 -- 172.21.15.167:0/95972159 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:51 5.1c 5:38000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x55590b312a70 con 0x7f35a0007030 2023-10-05T19:28:29.452 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.446+0000 7f35ae6ee700 1 -- 172.21.15.167:0/95972159 <== osd.2 v2:172.21.15.167:6816/30037 8 ==== osd_op_reply(51 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f3598004520 con 0x7f35a0007030 2023-10-05T19:28:29.452 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.446+0000 7f35ae6ee700 1 -- 172.21.15.167:0/95972159 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:52 5.2 5:40000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x55590b312a70 con 0x55590b3133d0 2023-10-05T19:28:29.452 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.446+0000 7f35aeeef700 1 -- 172.21.15.167:0/95972159 <== osd.3 v2:172.21.15.167:6808/30036 12 ==== osd_op_reply(52 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f35a0042040 con 0x55590b3133d0 2023-10-05T19:28:29.453 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.446+0000 7f35aeeef700 1 -- 172.21.15.167:0/95972159 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:53 5.12 5:48000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x55590b312a70 con 0x55590b2dc740 2023-10-05T19:28:29.453 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.446+0000 7f35af6f0700 1 -- 172.21.15.167:0/95972159 <== osd.1 v2:172.21.15.167:6800/30035 20 ==== osd_op_reply(53 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f35a802f080 con 0x55590b2dc740 2023-10-05T19:28:29.453 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.446+0000 7f35af6f0700 1 -- 172.21.15.167:0/95972159 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:54 5.a 5:50000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x55590b312a70 con 0x7f3598011040 2023-10-05T19:28:29.454 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.450+0000 7f35af6f0700 1 -- 172.21.15.167:0/95972159 <== osd.0 v2:172.21.15.118:6800/27690 14 ==== osd_op_reply(54 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f35a802f080 con 0x7f3598011040 2023-10-05T19:28:29.454 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.450+0000 7f35af6f0700 1 -- 172.21.15.167:0/95972159 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:55 5.1a 5:58000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x55590b312a70 con 0x55590b2dc740 2023-10-05T19:28:29.455 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.450+0000 7f35af6f0700 1 -- 172.21.15.167:0/95972159 <== osd.1 v2:172.21.15.167:6800/30035 21 ==== osd_op_reply(55 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f35a802f080 con 0x55590b2dc740 2023-10-05T19:28:29.455 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.450+0000 7f35af6f0700 1 -- 172.21.15.167:0/95972159 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:56 5.6 5:60000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x55590b312a70 con 0x7f35a0007030 2023-10-05T19:28:29.456 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.450+0000 7f35ae6ee700 1 -- 172.21.15.167:0/95972159 <== osd.2 v2:172.21.15.167:6816/30037 9 ==== osd_op_reply(56 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f3598004520 con 0x7f35a0007030 2023-10-05T19:28:29.456 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.450+0000 7f35ae6ee700 1 -- 172.21.15.167:0/95972159 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:57 5.16 5:68000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x55590b312a70 con 0x55590b3133d0 2023-10-05T19:28:29.457 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.450+0000 7f35aeeef700 1 -- 172.21.15.167:0/95972159 <== osd.3 v2:172.21.15.167:6808/30036 13 ==== osd_op_reply(57 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f35a0042040 con 0x55590b3133d0 2023-10-05T19:28:29.457 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.450+0000 7f35aeeef700 1 -- 172.21.15.167:0/95972159 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:58 5.e 5:70000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x55590b312a70 con 0x55590b3133d0 2023-10-05T19:28:29.457 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.450+0000 7f35aeeef700 1 -- 172.21.15.167:0/95972159 <== osd.3 v2:172.21.15.167:6808/30036 14 ==== osd_op_reply(58 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f35a0042040 con 0x55590b3133d0 2023-10-05T19:28:29.457 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.450+0000 7f35aeeef700 1 -- 172.21.15.167:0/95972159 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:59 5.1e 5:78000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x55590b312a70 con 0x7f3598011040 2023-10-05T19:28:29.459 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.454+0000 7f35af6f0700 1 -- 172.21.15.167:0/95972159 <== osd.0 v2:172.21.15.118:6800/27690 15 ==== osd_op_reply(59 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f35a802f080 con 0x7f3598011040 2023-10-05T19:28:29.459 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.454+0000 7f35af6f0700 1 -- 172.21.15.167:0/95972159 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:60 5.1 5:80000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x55590b312a70 con 0x55590b3133d0 2023-10-05T19:28:29.459 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.454+0000 7f35aeeef700 1 -- 172.21.15.167:0/95972159 <== osd.3 v2:172.21.15.167:6808/30036 15 ==== osd_op_reply(60 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f35a0042040 con 0x55590b3133d0 2023-10-05T19:28:29.459 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.454+0000 7f35aeeef700 1 -- 172.21.15.167:0/95972159 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:61 5.11 5:88000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x55590b312a70 con 0x55590b3133d0 2023-10-05T19:28:29.460 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.454+0000 7f35aeeef700 1 -- 172.21.15.167:0/95972159 <== osd.3 v2:172.21.15.167:6808/30036 16 ==== osd_op_reply(61 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f35a0042040 con 0x55590b3133d0 2023-10-05T19:28:29.460 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.454+0000 7f35aeeef700 1 -- 172.21.15.167:0/95972159 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:62 5.9 5:90000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x55590b312a70 con 0x55590b2dc740 2023-10-05T19:28:29.461 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.454+0000 7f35af6f0700 1 -- 172.21.15.167:0/95972159 <== osd.1 v2:172.21.15.167:6800/30035 22 ==== osd_op_reply(62 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f35a802f080 con 0x55590b2dc740 2023-10-05T19:28:29.461 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.454+0000 7f35af6f0700 1 -- 172.21.15.167:0/95972159 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:63 5.19 5:98000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x55590b312a70 con 0x55590b2dc740 2023-10-05T19:28:29.462 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.454+0000 7f35af6f0700 1 -- 172.21.15.167:0/95972159 <== osd.1 v2:172.21.15.167:6800/30035 23 ==== osd_op_reply(63 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f35a802f080 con 0x55590b2dc740 2023-10-05T19:28:29.462 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.454+0000 7f35af6f0700 1 -- 172.21.15.167:0/95972159 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:64 5.5 5:a0000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x55590b312a70 con 0x7f3598011040 2023-10-05T19:28:29.462 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.458+0000 7f35af6f0700 1 -- 172.21.15.167:0/95972159 <== osd.0 v2:172.21.15.118:6800/27690 16 ==== osd_op_reply(64 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f35a802f080 con 0x7f3598011040 2023-10-05T19:28:29.463 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.458+0000 7f35af6f0700 1 -- 172.21.15.167:0/95972159 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:65 5.15 5:a8000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x55590b312a70 con 0x55590b3133d0 2023-10-05T19:28:29.463 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.458+0000 7f35aeeef700 1 -- 172.21.15.167:0/95972159 <== osd.3 v2:172.21.15.167:6808/30036 17 ==== osd_op_reply(65 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f35a0042040 con 0x55590b3133d0 2023-10-05T19:28:29.463 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.458+0000 7f35aeeef700 1 -- 172.21.15.167:0/95972159 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:66 5.d 5:b0000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x55590b312a70 con 0x7f35a0007030 2023-10-05T19:28:29.464 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.458+0000 7f35ae6ee700 1 -- 172.21.15.167:0/95972159 <== osd.2 v2:172.21.15.167:6816/30037 10 ==== osd_op_reply(66 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f3598004520 con 0x7f35a0007030 2023-10-05T19:28:29.464 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.458+0000 7f35ae6ee700 1 -- 172.21.15.167:0/95972159 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:67 5.1d 5:b8000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x55590b312a70 con 0x55590b2dc740 2023-10-05T19:28:29.465 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.458+0000 7f35af6f0700 1 -- 172.21.15.167:0/95972159 <== osd.1 v2:172.21.15.167:6800/30035 24 ==== osd_op_reply(67 [pgnls start_epoch 30 out=107b] v30'2 uv2 ondisk = 1) v8 ==== 144+0+107 (crc 0 0 0) 0x7f35a802f080 con 0x55590b2dc740 2023-10-05T19:28:29.465 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.458+0000 7f35af6f0700 1 -- 172.21.15.167:0/95972159 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:68 5.3 5:c0000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x55590b312a70 con 0x7f3598011040 2023-10-05T19:28:29.466 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.462+0000 7f35af6f0700 1 -- 172.21.15.167:0/95972159 <== osd.0 v2:172.21.15.118:6800/27690 17 ==== osd_op_reply(68 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f35a802f080 con 0x7f3598011040 2023-10-05T19:28:29.466 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.462+0000 7f35af6f0700 1 -- 172.21.15.167:0/95972159 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:69 5.13 5:c8000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x55590b312a70 con 0x55590b3133d0 2023-10-05T19:28:29.467 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.462+0000 7f35aeeef700 1 -- 172.21.15.167:0/95972159 <== osd.3 v2:172.21.15.167:6808/30036 18 ==== osd_op_reply(69 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f35a0042040 con 0x55590b3133d0 2023-10-05T19:28:29.467 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.462+0000 7f35aeeef700 1 -- 172.21.15.167:0/95972159 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:70 5.b 5:d0000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x55590b312a70 con 0x7f3598011040 2023-10-05T19:28:29.468 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.462+0000 7f35af6f0700 1 -- 172.21.15.167:0/95972159 <== osd.0 v2:172.21.15.118:6800/27690 18 ==== osd_op_reply(70 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f35a802f080 con 0x7f3598011040 2023-10-05T19:28:29.468 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.462+0000 7f35af6f0700 1 -- 172.21.15.167:0/95972159 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:71 5.1b 5:d8000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x55590b312a70 con 0x7f35a0007030 2023-10-05T19:28:29.469 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.462+0000 7f35ae6ee700 1 -- 172.21.15.167:0/95972159 <== osd.2 v2:172.21.15.167:6816/30037 11 ==== osd_op_reply(71 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f3598004520 con 0x7f35a0007030 2023-10-05T19:28:29.469 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.462+0000 7f35ae6ee700 1 -- 172.21.15.167:0/95972159 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:72 5.7 5:e0000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x55590b312a70 con 0x55590b3133d0 2023-10-05T19:28:29.469 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.462+0000 7f35aeeef700 1 -- 172.21.15.167:0/95972159 <== osd.3 v2:172.21.15.167:6808/30036 19 ==== osd_op_reply(72 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f35a0042040 con 0x55590b3133d0 2023-10-05T19:28:29.469 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.462+0000 7f35aeeef700 1 -- 172.21.15.167:0/95972159 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:73 5.17 5:e8000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x55590b312a70 con 0x7f3598011040 2023-10-05T19:28:29.470 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.466+0000 7f35af6f0700 1 -- 172.21.15.167:0/95972159 <== osd.0 v2:172.21.15.118:6800/27690 19 ==== osd_op_reply(73 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f35a802f080 con 0x7f3598011040 2023-10-05T19:28:29.471 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.466+0000 7f35af6f0700 1 -- 172.21.15.167:0/95972159 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:74 5.f 5:f0000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x55590b312a70 con 0x55590b2dc740 2023-10-05T19:28:29.471 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.466+0000 7f35af6f0700 1 -- 172.21.15.167:0/95972159 <== osd.1 v2:172.21.15.167:6800/30035 25 ==== osd_op_reply(74 [pgnls start_epoch 30 out=115b] v23'2 uv2 ondisk = 1) v8 ==== 144+0+115 (crc 0 0 0) 0x7f35a802f080 con 0x55590b2dc740 2023-10-05T19:28:29.472 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.466+0000 7f35af6f0700 1 -- 172.21.15.167:0/95972159 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:75 5.1f 5:f8000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x55590b312a70 con 0x7f35a0007030 2023-10-05T19:28:29.472 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.466+0000 7f35ae6ee700 1 -- 172.21.15.167:0/95972159 <== osd.2 v2:172.21.15.167:6816/30037 12 ==== osd_op_reply(75 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f3598004520 con 0x7f35a0007030 2023-10-05T19:28:29.473 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.466+0000 7f35b13bae40 20 RGWRados::pool_iterate: got zonegroup_info.4f1f719d-9648-417b-b10c-e5d939825419 2023-10-05T19:28:29.473 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.466+0000 7f35b13bae40 20 RGWRados::pool_iterate: got zone_info.9a9df965-6612-45d4-a45f-6e3c889bacc2 2023-10-05T19:28:29.473 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.466+0000 7f35b13bae40 20 RGWRados::pool_iterate: got zone_names.default 2023-10-05T19:28:29.473 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.466+0000 7f35b13bae40 20 RGWRados::pool_iterate: got zonegroups_names.default 2023-10-05T19:28:29.473 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.466+0000 7f35b13bae40 20 rados->read ofs=0 len=0 2023-10-05T19:28:29.473 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.466+0000 7f35b13bae40 1 -- 172.21.15.167:0/95972159 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:76 5.f 5:f43b6ece:::zone_names.default:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x55590b3168e0 con 0x55590b2dc740 2023-10-05T19:28:29.474 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.466+0000 7f35af6f0700 1 -- 172.21.15.167:0/95972159 <== osd.1 v2:172.21.15.167:6800/30035 26 ==== osd_op_reply(76 zone_names.default [read 0~46 out=46b] v0'0 uv1 ondisk = 0) v8 ==== 162+0+46 (crc 0 0 0) 0x7f35a802f080 con 0x55590b2dc740 2023-10-05T19:28:29.474 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.466+0000 7f35b13bae40 20 rados_obj.operate() r=0 bl.length=46 2023-10-05T19:28:29.474 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.466+0000 7f35b13bae40 20 rados->read ofs=0 len=0 2023-10-05T19:28:29.474 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.466+0000 7f35b13bae40 1 -- 172.21.15.167:0/95972159 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:77 5.1d 5:b8374eb4:::zone_info.9a9df965-6612-45d4-a45f-6e3c889bacc2:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x55590b316c90 con 0x55590b2dc740 2023-10-05T19:28:29.475 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.470+0000 7f35af6f0700 1 -- 172.21.15.167:0/95972159 <== osd.1 v2:172.21.15.167:6800/30035 27 ==== osd_op_reply(77 zone_info.9a9df965-6612-45d4-a45f-6e3c889bacc2 [read 0~874 out=874b] v0'0 uv2 ondisk = 0) v8 ==== 190+0+874 (crc 0 0 0) 0x7f35a802f080 con 0x55590b2dc740 2023-10-05T19:28:29.475 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.470+0000 7f35b13bae40 20 rados_obj.operate() r=0 bl.length=874 2023-10-05T19:28:29.475 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.470+0000 7f35b13bae40 20 rados->read ofs=0 len=0 2023-10-05T19:28:29.475 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.470+0000 7f35b13bae40 1 -- 172.21.15.167:0/95972159 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:78 5.f 5:f4c53578:::zonegroups_names.default:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x55590b312610 con 0x55590b2dc740 2023-10-05T19:28:29.475 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.470+0000 7f35af6f0700 1 -- 172.21.15.167:0/95972159 <== osd.1 v2:172.21.15.167:6800/30035 28 ==== osd_op_reply(78 zonegroups_names.default [read 0~46 out=46b] v0'0 uv2 ondisk = 0) v8 ==== 168+0+46 (crc 0 0 0) 0x7f35a802f080 con 0x55590b2dc740 2023-10-05T19:28:29.475 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.470+0000 7f35b13bae40 20 rados_obj.operate() r=0 bl.length=46 2023-10-05T19:28:29.476 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.470+0000 7f35b13bae40 20 rados->read ofs=0 len=0 2023-10-05T19:28:29.476 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.470+0000 7f35b13bae40 1 -- 172.21.15.167:0/95972159 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:79 5.c 5:30d299a5:::zonegroup_info.4f1f719d-9648-417b-b10c-e5d939825419:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x55590b312630 con 0x7f3598011040 2023-10-05T19:28:29.476 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.470+0000 7f35af6f0700 1 -- 172.21.15.167:0/95972159 <== osd.0 v2:172.21.15.118:6800/27690 20 ==== osd_op_reply(79 zonegroup_info.4f1f719d-9648-417b-b10c-e5d939825419 [read 0~362 out=362b] v0'0 uv1 ondisk = 0) v8 ==== 195+0+362 (crc 0 0 0) 0x7f35a802f080 con 0x7f3598011040 2023-10-05T19:28:29.476 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.470+0000 7f35b13bae40 20 rados_obj.operate() r=0 bl.length=362 2023-10-05T19:28:29.476 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.470+0000 7f35b13bae40 20 rados->read ofs=0 len=0 2023-10-05T19:28:29.477 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.470+0000 7f35b13bae40 1 -- 172.21.15.167:0/95972159 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:80 5.3 5:c0ad8cbd:::region_map:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x55590b317700 con 0x7f3598011040 2023-10-05T19:28:29.477 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.470+0000 7f35af6f0700 1 -- 172.21.15.167:0/95972159 <== osd.0 v2:172.21.15.118:6800/27690 21 ==== osd_op_reply(80 region_map [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 ==== 154+0+0 (crc 0 0 0) 0x7f35a802f080 con 0x7f3598011040 2023-10-05T19:28:29.477 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.470+0000 7f35b13bae40 20 rados_obj.operate() r=-2 bl.length=0 2023-10-05T19:28:29.477 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.470+0000 7f35b13bae40 20 zone default found 2023-10-05T19:28:29.478 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.470+0000 7f35b13bae40 4 Realm: () 2023-10-05T19:28:29.478 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.470+0000 7f35b13bae40 4 ZoneGroup: default (4f1f719d-9648-417b-b10c-e5d939825419) 2023-10-05T19:28:29.478 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.470+0000 7f35b13bae40 4 Zone: default (9a9df965-6612-45d4-a45f-6e3c889bacc2) 2023-10-05T19:28:29.478 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.470+0000 7f35b13bae40 10 cannot find current period zonegroup using local zonegroup configuration 2023-10-05T19:28:29.478 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.470+0000 7f35b13bae40 20 zonegroup default 2023-10-05T19:28:29.478 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.470+0000 7f35b13bae40 20 rados->read ofs=0 len=0 2023-10-05T19:28:29.478 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.470+0000 7f35b13bae40 1 -- 172.21.15.167:0/95972159 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:81 5.3 5:c52100b6:::period_config.default:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x55590b317700 con 0x7f3598011040 2023-10-05T19:28:29.479 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.474+0000 7f35af6f0700 1 -- 172.21.15.167:0/95972159 <== osd.0 v2:172.21.15.118:6800/27690 22 ==== osd_op_reply(81 period_config.default [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 ==== 165+0+0 (crc 0 0 0) 0x7f35a802f080 con 0x7f3598011040 2023-10-05T19:28:29.479 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.474+0000 7f35b13bae40 20 rados_obj.operate() r=-2 bl.length=0 2023-10-05T19:28:29.479 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.474+0000 7f35b13bae40 20 rados->read ofs=0 len=0 2023-10-05T19:28:29.479 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.474+0000 7f35b13bae40 1 -- 172.21.15.167:0/95972159 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:82 6.2 6:4347d321:::bucket.sync-source-hints.:head [call version.read in=11b,read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x55590b318d20 con 0x55590b3133d0 2023-10-05T19:28:29.479 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.474+0000 7f35aeeef700 1 -- 172.21.15.167:0/95972159 <== osd.3 v2:172.21.15.167:6808/30036 20 ==== osd_op_reply(82 bucket.sync-source-hints. [call,read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 ==== 211+0+0 (crc 0 0 0) 0x7f35a0042040 con 0x55590b3133d0 2023-10-05T19:28:29.480 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.474+0000 7f35b13bae40 20 rados_obj.operate() r=-2 bl.length=0 2023-10-05T19:28:29.480 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.474+0000 7f35b13bae40 20 rados->read ofs=0 len=0 2023-10-05T19:28:29.480 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.474+0000 7f35b13bae40 1 -- 172.21.15.167:0/95972159 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:83 6.b 6:d467b91b:::bucket.sync-target-hints.:head [call version.read in=11b,read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x55590b3190d0 con 0x55590b3133d0 2023-10-05T19:28:29.481 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.474+0000 7f35aeeef700 1 -- 172.21.15.167:0/95972159 <== osd.3 v2:172.21.15.167:6808/30036 21 ==== osd_op_reply(83 bucket.sync-target-hints. [call,read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 ==== 211+0+0 (crc 0 0 0) 0x7f35a0042040 con 0x55590b3133d0 2023-10-05T19:28:29.481 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.474+0000 7f35b13bae40 20 rados_obj.operate() r=-2 bl.length=0 2023-10-05T19:28:29.481 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.474+0000 7f35b13bae40 20 started sync module instance, tier type = 2023-10-05T19:28:29.481 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.474+0000 7f35b13bae40 20 started zone id=9a9df965-6612-45d4-a45f-6e3c889bacc2 (name=default) with tier type = 2023-10-05T19:28:29.481 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.474+0000 7f35b13bae40 1 -- 172.21.15.167:0/95972159 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:84 7.1f 7:f95f44c2:::notify.0:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x55590b3183e0 con 0x7f35a0007030 2023-10-05T19:28:29.483 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.478+0000 7f35ae6ee700 1 -- 172.21.15.167:0/95972159 <== osd.2 v2:172.21.15.167:6816/30037 13 ==== osd_op_reply(84 notify.0 [create] v30'75 uv75 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f359802f040 con 0x7f35a0007030 2023-10-05T19:28:29.483 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.478+0000 7f35b13bae40 1 -- 172.21.15.167:0/95972159 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:85 7.1f 7:f95f44c2:::notify.0:head [watch watch cookie 93840928245648] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x55590b318010 con 0x7f35a0007030 2023-10-05T19:28:29.483 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.478+0000 7f35b13bae40 1 -- 172.21.15.167:0/95972159 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:86 7.0 7:05bf5b68:::notify.1:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x55590b319e90 con 0x55590b2dc740 2023-10-05T19:28:29.485 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.478+0000 7f35ae6ee700 1 -- 172.21.15.167:0/95972159 <== osd.2 v2:172.21.15.167:6816/30037 14 ==== osd_op_reply(85 notify.0 [watch watch cookie 93840928245648] v30'76 uv75 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f359802f040 con 0x7f35a0007030 2023-10-05T19:28:29.486 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.482+0000 7f35af6f0700 1 -- 172.21.15.167:0/95972159 <== osd.1 v2:172.21.15.167:6800/30035 29 ==== osd_op_reply(86 notify.1 [create] v30'75 uv75 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f35a802f080 con 0x55590b2dc740 2023-10-05T19:28:29.487 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.482+0000 7f35b13bae40 1 -- 172.21.15.167:0/95972159 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:87 7.0 7:05bf5b68:::notify.1:head [watch watch cookie 93840928250272] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x55590b3195a0 con 0x55590b2dc740 2023-10-05T19:28:29.487 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.482+0000 7f35b13bae40 1 -- 172.21.15.167:0/95972159 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:88 7.15 7:a93a5511:::notify.2:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x55590b31ac70 con 0x55590b3133d0 2023-10-05T19:28:29.489 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.482+0000 7f35af6f0700 1 -- 172.21.15.167:0/95972159 <== osd.1 v2:172.21.15.167:6800/30035 30 ==== osd_op_reply(87 notify.1 [watch watch cookie 93840928250272] v30'76 uv75 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f35a8050040 con 0x55590b2dc740 2023-10-05T19:28:29.490 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.486+0000 7f35aeeef700 1 -- 172.21.15.167:0/95972159 <== osd.3 v2:172.21.15.167:6808/30036 22 ==== osd_op_reply(88 notify.2 [create] v30'75 uv75 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f35a0042040 con 0x55590b3133d0 2023-10-05T19:28:29.490 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.486+0000 7f35b13bae40 1 -- 172.21.15.167:0/95972159 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:89 7.15 7:a93a5511:::notify.2:head [watch watch cookie 93840928252560] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x55590b3195a0 con 0x55590b3133d0 2023-10-05T19:28:29.490 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.486+0000 7f35b13bae40 1 -- 172.21.15.167:0/95972159 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:90 7.e 7:7759931f:::notify.3:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x55590b31b680 con 0x7f35a0007030 2023-10-05T19:28:29.493 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.486+0000 7f35ae6ee700 1 -- 172.21.15.167:0/95972159 <== osd.2 v2:172.21.15.167:6816/30037 15 ==== osd_op_reply(90 notify.3 [create] v30'75 uv75 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f359802f040 con 0x7f35a0007030 2023-10-05T19:28:29.493 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.486+0000 7f35b13bae40 1 -- 172.21.15.167:0/95972159 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:91 7.e 7:7759931f:::notify.3:head [watch watch cookie 93840928256400] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x55590b31bf70 con 0x7f35a0007030 2023-10-05T19:28:29.493 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.486+0000 7f35b13bae40 1 -- 172.21.15.167:0/95972159 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:92 7.d 7:b4812045:::notify.4:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x55590b31c980 con 0x55590b2dc740 2023-10-05T19:28:29.494 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.490+0000 7f35aeeef700 1 -- 172.21.15.167:0/95972159 <== osd.3 v2:172.21.15.167:6808/30036 23 ==== osd_op_reply(89 notify.2 [watch watch cookie 93840928252560] v30'76 uv75 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f35a0042040 con 0x55590b3133d0 2023-10-05T19:28:29.495 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.490+0000 7f35ae6ee700 1 -- 172.21.15.167:0/95972159 <== osd.2 v2:172.21.15.167:6816/30037 16 ==== osd_op_reply(91 notify.3 [watch watch cookie 93840928256400] v30'76 uv75 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f359802f040 con 0x7f35a0007030 2023-10-05T19:28:29.495 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.490+0000 7f35af6f0700 1 -- 172.21.15.167:0/95972159 <== osd.1 v2:172.21.15.167:6800/30035 31 ==== osd_op_reply(92 notify.4 [create] v30'75 uv75 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f35a8050040 con 0x55590b2dc740 2023-10-05T19:28:29.496 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.490+0000 7f35b13bae40 1 -- 172.21.15.167:0/95972159 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:93 7.d 7:b4812045:::notify.4:head [watch watch cookie 93840928253664] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x55590b31c980 con 0x55590b2dc740 2023-10-05T19:28:29.496 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.490+0000 7f35b13bae40 1 -- 172.21.15.167:0/95972159 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:94 7.3 7:c609908c:::notify.5:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x55590b31d270 con 0x7f3598011040 2023-10-05T19:28:29.498 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.494+0000 7f35af6f0700 1 -- 172.21.15.167:0/95972159 <== osd.1 v2:172.21.15.167:6800/30035 32 ==== osd_op_reply(93 notify.4 [watch watch cookie 93840928253664] v30'76 uv75 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f35a8050040 con 0x55590b2dc740 2023-10-05T19:28:29.499 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.494+0000 7f35af6f0700 1 -- 172.21.15.167:0/95972159 <== osd.0 v2:172.21.15.118:6800/27690 23 ==== osd_op_reply(94 notify.5 [create] v30'75 uv75 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f35a8050040 con 0x7f3598011040 2023-10-05T19:28:29.499 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.494+0000 7f35b13bae40 1 -- 172.21.15.167:0/95972159 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:95 7.3 7:c609908c:::notify.5:head [watch watch cookie 93840928261824] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x55590b31d000 con 0x7f3598011040 2023-10-05T19:28:29.499 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.494+0000 7f35b13bae40 1 -- 172.21.15.167:0/95972159 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:96 7.14 7:2b04a3e9:::notify.6:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x55590b31db30 con 0x7f3598011040 2023-10-05T19:28:29.501 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.494+0000 7f35af6f0700 1 -- 172.21.15.167:0/95972159 <== osd.0 v2:172.21.15.118:6800/27690 24 ==== osd_op_reply(96 notify.6 [create] v30'75 uv75 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f35a8050040 con 0x7f3598011040 2023-10-05T19:28:29.501 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.494+0000 7f35b13bae40 1 -- 172.21.15.167:0/95972159 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:97 7.14 7:2b04a3e9:::notify.6:head [watch watch cookie 93840928265728] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x55590b31df40 con 0x7f3598011040 2023-10-05T19:28:29.501 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.494+0000 7f35b13bae40 1 -- 172.21.15.167:0/95972159 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:98 7.9 7:93e5b521:::notify.7:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x55590b31ea80 con 0x7f3598011040 2023-10-05T19:28:29.502 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.494+0000 7f35af6f0700 1 -- 172.21.15.167:0/95972159 <== osd.0 v2:172.21.15.118:6800/27690 25 ==== osd_op_reply(95 notify.5 [watch watch cookie 93840928261824] v30'76 uv75 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f35a8050040 con 0x7f3598011040 2023-10-05T19:28:29.504 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.498+0000 7f35af6f0700 1 -- 172.21.15.167:0/95972159 <== osd.0 v2:172.21.15.118:6800/27690 26 ==== osd_op_reply(98 notify.7 [create] v30'75 uv75 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f35a8050040 con 0x7f3598011040 2023-10-05T19:28:29.504 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.498+0000 7f35b13bae40 1 -- 172.21.15.167:0/95972159 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:99 7.9 7:93e5b521:::notify.7:head [watch watch cookie 93840928262928] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x55590b31ea40 con 0x7f3598011040 2023-10-05T19:28:29.504 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.498+0000 7f35b13bae40 20 add_watcher() i=0 2023-10-05T19:28:29.504 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.498+0000 7f35b13bae40 20 add_watcher() i=1 2023-10-05T19:28:29.504 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.498+0000 7f35b13bae40 20 add_watcher() i=2 2023-10-05T19:28:29.504 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.498+0000 7f35b13bae40 20 add_watcher() i=3 2023-10-05T19:28:29.505 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.498+0000 7f35b13bae40 20 add_watcher() i=4 2023-10-05T19:28:29.505 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.498+0000 7f35b13bae40 20 add_watcher() i=5 2023-10-05T19:28:29.505 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.498+0000 7f35af6f0700 1 -- 172.21.15.167:0/95972159 <== osd.0 v2:172.21.15.118:6800/27690 27 ==== osd_op_reply(97 notify.6 [watch watch cookie 93840928265728] v30'76 uv75 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f35a8050040 con 0x7f3598011040 2023-10-05T19:28:29.505 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.498+0000 7f35b13bae40 20 add_watcher() i=6 2023-10-05T19:28:29.506 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.502+0000 7f35af6f0700 1 -- 172.21.15.167:0/95972159 <== osd.0 v2:172.21.15.118:6800/27690 28 ==== osd_op_reply(99 notify.7 [watch watch cookie 93840928262928] v30'76 uv75 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f35a8050040 con 0x7f3598011040 2023-10-05T19:28:29.506 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.502+0000 7f35b13bae40 20 add_watcher() i=7 2023-10-05T19:28:29.506 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.502+0000 7f35b13bae40 2 all 8 watchers are set, enabling cache 2023-10-05T19:28:29.506 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.502+0000 7f35b13bae40 1 -- 172.21.15.167:0/95972159 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:100 6.4 6:22d26bf9:::data_loggenerations_metadata:head [call version.check_conds in=74b,call version.read in=11b,read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x55590b321a70 con 0x55590b2dc740 2023-10-05T19:28:29.507 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.502+0000 7f35af6f0700 1 -- 172.21.15.167:0/95972159 <== osd.1 v2:172.21.15.167:6800/30035 33 ==== osd_op_reply(100 data_loggenerations_metadata [call,call out=48b,read 0~28 out=28b] v0'0 uv1 ondisk = 0) v8 ==== 256+0+76 (crc 0 0 0) 0x7f35a8050040 con 0x55590b2dc740 2023-10-05T19:28:29.507 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.502+0000 7f35b13bae40 1 -- 172.21.15.167:0/95972159 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:101 6.4 6:22d26bf9:::data_loggenerations_metadata:head [watch watch cookie 93840928269648] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x55590b31ee90 con 0x55590b2dc740 2023-10-05T19:28:29.509 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.502+0000 7f35af6f0700 1 -- 172.21.15.167:0/95972159 <== osd.1 v2:172.21.15.167:6800/30035 34 ==== osd_op_reply(101 data_loggenerations_metadata [watch watch cookie 93840928269648] v30'84 uv1 ondisk = 0) v8 ==== 172+0+0 (crc 0 0 0) 0x7f35a8050040 con 0x55590b2dc740 2023-10-05T19:28:29.510 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.502+0000 7f34fbfff700 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2023-10-05T19:28:29.510 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.502+0000 7f35b13bae40 20 check_secure_mon_conn(): auth registy supported: methods=[2] modes=[2,1] 2023-10-05T19:28:29.510 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.502+0000 7f35b13bae40 20 check_secure_mon_conn(): mode 1 is insecure 2023-10-05T19:28:29.511 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.506+0000 7f35b13bae40 1 Processor -- start 2023-10-05T19:28:29.511 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.506+0000 7f35b13bae40 1 -- start start 2023-10-05T19:28:29.511 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.506+0000 7f35b13bae40 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55590b32be60 0x55590b32c230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-05T19:28:29.511 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.506+0000 7f35b13bae40 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x55590b32c770 con 0x55590b32be60 2023-10-05T19:28:29.511 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.506+0000 7f35ae6ee700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55590b32be60 0x55590b32c230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-05T19:28:29.511 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.506+0000 7f35ae6ee700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55590b32be60 0x55590b32c230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.118:3300/0 says I am v2:172.21.15.167:57932/0 (socket says 172.21.15.167:57932) 2023-10-05T19:28:29.511 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.506+0000 7f35ae6ee700 1 -- 172.21.15.167:0/1313695879 learned_addr learned my addr 172.21.15.167:0/1313695879 (peer_addr_for_me v2:172.21.15.167:0/0) 2023-10-05T19:28:29.512 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.506+0000 7f35ae6ee700 1 -- 172.21.15.167:0/1313695879 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55590b32c8b0 con 0x55590b32be60 2023-10-05T19:28:29.512 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.506+0000 7f35ae6ee700 1 --2- 172.21.15.167:0/1313695879 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55590b32be60 0x55590b32c230 secure :-1 s=READY pgs=288 cs=0 l=1 rev1=1 crypto rx=0x55590b30d080 tx=0x7f3598006000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d1a4d140f0bc2752 server_cookie=0 in_seq=0 out_seq=0 2023-10-05T19:28:29.512 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.506+0000 7f34fb7fe700 1 -- 172.21.15.167:0/1313695879 <== mon.0 v2:172.21.15.118:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f359802c040 con 0x55590b32be60 2023-10-05T19:28:29.512 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.506+0000 7f34fb7fe700 1 -- 172.21.15.167:0/1313695879 <== mon.0 v2:172.21.15.118:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f359802d040 con 0x55590b32be60 2023-10-05T19:28:29.513 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.506+0000 7f34fb7fe700 1 -- 172.21.15.167:0/1313695879 <== mon.0 v2:172.21.15.118:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f3598004e10 con 0x55590b32be60 2023-10-05T19:28:29.513 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.506+0000 7f35b13bae40 1 -- 172.21.15.167:0/1313695879 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55590b32be60 msgr2=0x55590b32c230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-05T19:28:29.513 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.506+0000 7f35b13bae40 1 --2- 172.21.15.167:0/1313695879 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55590b32be60 0x55590b32c230 secure :-1 s=READY pgs=288 cs=0 l=1 rev1=1 crypto rx=0x55590b30d080 tx=0x7f3598006000 comp rx=0 tx=0).stop 2023-10-05T19:28:29.513 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.506+0000 7f35b13bae40 1 -- 172.21.15.167:0/1313695879 shutdown_connections 2023-10-05T19:28:29.514 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.506+0000 7f35b13bae40 1 --2- 172.21.15.167:0/1313695879 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55590b32be60 0x55590b32c230 unknown :-1 s=CLOSED pgs=288 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-05T19:28:29.514 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.506+0000 7f35b13bae40 1 -- 172.21.15.167:0/1313695879 >> 172.21.15.167:0/1313695879 conn(0x55590b327220 msgr2=0x55590b325370 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-05T19:28:29.514 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.506+0000 7f35b13bae40 1 -- 172.21.15.167:0/1313695879 shutdown_connections 2023-10-05T19:28:29.514 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.506+0000 7f35b13bae40 1 -- 172.21.15.167:0/1313695879 wait complete. 2023-10-05T19:28:29.515 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.506+0000 7f35b13bae40 1 Processor -- start 2023-10-05T19:28:29.515 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.510+0000 7f35b13bae40 1 -- start start 2023-10-05T19:28:29.515 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.510+0000 7f35b13bae40 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55590b32be60 0x55590b3319b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-05T19:28:29.515 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.510+0000 7f35b13bae40 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x55590b3329b0 con 0x55590b32be60 2023-10-05T19:28:29.515 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.510+0000 7f35ae6ee700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55590b32be60 0x55590b3319b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-05T19:28:29.515 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.510+0000 7f35ae6ee700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55590b32be60 0x55590b3319b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.118:3300/0 says I am v2:172.21.15.167:57940/0 (socket says 172.21.15.167:57940) 2023-10-05T19:28:29.515 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.510+0000 7f35ae6ee700 1 -- 172.21.15.167:0/3021234063 learned_addr learned my addr 172.21.15.167:0/3021234063 (peer_addr_for_me v2:172.21.15.167:0/0) 2023-10-05T19:28:29.516 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.510+0000 7f35ae6ee700 1 -- 172.21.15.167:0/3021234063 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3598004b10 con 0x55590b32be60 2023-10-05T19:28:29.516 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.510+0000 7f35ae6ee700 1 --2- 172.21.15.167:0/3021234063 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55590b32be60 0x55590b3319b0 secure :-1 s=READY pgs=289 cs=0 l=1 rev1=1 crypto rx=0x7f3598006040 tx=0x7f359803c000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-05T19:28:29.516 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.510+0000 7f34f9ffb700 1 -- 172.21.15.167:0/3021234063 <== mon.0 v2:172.21.15.118:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f359802c070 con 0x55590b32be60 2023-10-05T19:28:29.517 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.510+0000 7f34f9ffb700 1 -- 172.21.15.167:0/3021234063 <== mon.0 v2:172.21.15.118:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f359801c040 con 0x55590b32be60 2023-10-05T19:28:29.517 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.510+0000 7f35b13bae40 1 -- 172.21.15.167:0/3021234063 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x55590b331fb0 con 0x55590b32be60 2023-10-05T19:28:29.517 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.510+0000 7f34f9ffb700 1 -- 172.21.15.167:0/3021234063 <== mon.0 v2:172.21.15.118:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f359804a5b0 con 0x55590b32be60 2023-10-05T19:28:29.517 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.510+0000 7f35b13bae40 1 -- 172.21.15.167:0/3021234063 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55590b3323f0 con 0x55590b32be60 2023-10-05T19:28:29.517 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.510+0000 7f34f9ffb700 1 -- 172.21.15.167:0/3021234063 <== mon.0 v2:172.21.15.118:3300/0 4 ==== mgrmap(e 5) v1 ==== 40329+0+0 (secure 0 0 0) 0x7f3598048440 con 0x55590b32be60 2023-10-05T19:28:29.518 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.510+0000 7f34f9ffb700 1 --2- 172.21.15.167:0/3021234063 >> [v2:172.21.15.167:6824/29889,v1:172.21.15.167:6825/29889] conn(0x7f34ec030c50 0x7f34ec033100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-05T19:28:29.518 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.510+0000 7f34f9ffb700 1 -- 172.21.15.167:0/3021234063 <== mon.0 v2:172.21.15.118:3300/0 5 ==== osd_map(30..30 src has 1..30) v4 ==== 6057+0+0 (secure 0 0 0) 0x7f359800b0e0 con 0x55590b32be60 2023-10-05T19:28:29.518 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.514+0000 7f35af6f0700 1 --2- 172.21.15.167:0/3021234063 >> [v2:172.21.15.167:6824/29889,v1:172.21.15.167:6825/29889] conn(0x7f34ec030c50 0x7f34ec033100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-05T19:28:29.518 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.514+0000 7f35b13bae40 5 note: GC not initialized 2023-10-05T19:28:29.519 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.514+0000 7f35af6f0700 1 --2- 172.21.15.167:0/3021234063 >> [v2:172.21.15.167:6824/29889,v1:172.21.15.167:6825/29889] conn(0x7f34ec030c50 0x7f34ec033100 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x55590b31dc50 tx=0x7f35a8000f90 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-05T19:28:29.520 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.514+0000 7f34f2ffd700 20 reqs_thread_entry: start 2023-10-05T19:28:29.520 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.514+0000 7f35b13bae40 20 init_complete bucket index max shards: 11 2023-10-05T19:28:29.520 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.514+0000 7f34f0ff9700 20 reqs_thread_entry: start 2023-10-05T19:28:29.521 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.514+0000 7f35b13bae40 10 rgw notify: Started notification manager with: 1 workers 2023-10-05T19:28:29.521 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.514+0000 7f34deffd700 20 rgw notify: INFO: next queues processing will happen at: Thu Oct 5 19:28:59 2023 2023-10-05T19:28:29.521 INFO:teuthology.orchestra.run.smithi167.stderr: 2023-10-05T19:28:29.521 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.514+0000 7f34de7fc700 20 reqs_thread_entry: start 2023-10-05T19:28:29.522 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.514+0000 7f35b13bae40 20 get_system_obj_state: rctx=0x7ffc0618c638 obj=default.rgw.meta:users.uid:foo state=0x55590b1b50b0 s->prefetch_data=0 2023-10-05T19:28:29.522 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.514+0000 7f35b13bae40 10 cache get: name=default.rgw.meta+users.uid+foo : miss 2023-10-05T19:28:29.522 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.514+0000 7f35b13bae40 1 -- 172.21.15.167:0/95972159 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:102 8.18 8:1e6ca5ed:users.uid::foo:head [call version.read in=11b,getxattrs,stat] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x55590b39a2c0 con 0x55590b2dc740 2023-10-05T19:28:29.523 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.514+0000 7f35af6f0700 1 -- 172.21.15.167:0/95972159 <== osd.1 v2:172.21.15.167:6800/30035 35 ==== osd_op_reply(102 foo [call out=48b,getxattrs out=75b,stat out=16b] v0'0 uv9 ondisk = 0) v8 ==== 231+0+139 (crc 0 0 0) 0x7f35a8050040 con 0x55590b2dc740 2023-10-05T19:28:29.523 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.518+0000 7f35b13bae40 10 cache put: name=default.rgw.meta+users.uid+foo info.flags=0x16 2023-10-05T19:28:29.523 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.518+0000 7f35b13bae40 10 adding default.rgw.meta+users.uid+foo to cache LRU end 2023-10-05T19:28:29.523 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.518+0000 7f35b13bae40 10 updating xattr: name=ceph.objclass.version bl.length()=42 2023-10-05T19:28:29.523 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.518+0000 7f35b13bae40 20 get_system_obj_state: s->obj_tag was set empty 2023-10-05T19:28:29.523 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.518+0000 7f35b13bae40 20 Read xattr: user.rgw.idtag 2023-10-05T19:28:29.523 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.518+0000 7f35b13bae40 10 cache get: name=default.rgw.meta+users.uid+foo : type miss (requested=0x13, cached=0x16) 2023-10-05T19:28:29.524 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.518+0000 7f35b13bae40 20 rados->read ofs=0 len=0 2023-10-05T19:28:29.524 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.518+0000 7f35b13bae40 1 -- 172.21.15.167:0/95972159 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:103 8.18 8:1e6ca5ed:users.uid::foo:head [call version.check_conds in=122b,call version.read in=11b,read 0~0,getxattrs] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x55590b39cd50 con 0x55590b2dc740 2023-10-05T19:28:29.524 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.518+0000 7f35af6f0700 1 -- 172.21.15.167:0/95972159 <== osd.1 v2:172.21.15.167:6800/30035 36 ==== osd_op_reply(103 foo [call,call out=48b,read 0~539 out=539b,getxattrs out=75b] v0'0 uv9 ondisk = 0) v8 ==== 273+0+662 (crc 0 0 0) 0x7f35a8050040 con 0x55590b2dc740 2023-10-05T19:28:29.524 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.518+0000 7f35b13bae40 20 rados_obj.operate() r=0 bl.length=539 2023-10-05T19:28:29.524 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.518+0000 7f35b13bae40 10 cache put: name=default.rgw.meta+users.uid+foo info.flags=0x13 2023-10-05T19:28:29.524 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.518+0000 7f35b13bae40 10 moving default.rgw.meta+users.uid+foo to cache LRU end 2023-10-05T19:28:29.524 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.518+0000 7f35b13bae40 10 updating xattr: name=ceph.objclass.version bl.length()=42 2023-10-05T19:28:29.524 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.518+0000 7f35b13bae40 1 -- 172.21.15.167:0/95972159 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:104 8.18 8:1e6ca5ed:users.uid::foo:head [delete,create,call version.set in=58b,writefull 0~513 in=513b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x55590b39dfd0 con 0x55590b2dc740 2023-10-05T19:28:29.526 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.518+0000 7f35af6f0700 1 -- 172.21.15.167:0/95972159 <== osd.1 v2:172.21.15.167:6800/30035 37 ==== osd_op_reply(104 foo [delete,create,call,writefull 0~513] v30'10 uv10 ondisk = 0) v8 ==== 273+0+0 (crc 0 0 0) 0x7f35a8050040 con 0x55590b2dc740 2023-10-05T19:28:29.526 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.518+0000 7f35b13bae40 10 cache put: name=default.rgw.meta+users.uid+foo info.flags=0x17 2023-10-05T19:28:29.526 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.518+0000 7f35b13bae40 10 moving default.rgw.meta+users.uid+foo to cache LRU end 2023-10-05T19:28:29.526 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.518+0000 7f35b13bae40 10 distributing notification oid=default.rgw.control:notify.4 cni=[op: 0, obj: default.rgw.meta:users.uid:foo, ofs0, ns] 2023-10-05T19:28:29.526 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.518+0000 7f35b13bae40 1 -- 172.21.15.167:0/95972159 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:105 7.d 7:b4812045:::notify.4:head [notify cookie 93840928791264 in=701b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x55590b39cd50 con 0x55590b2dc740 2023-10-05T19:28:29.527 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.522+0000 7f35af6f0700 1 -- 172.21.15.167:0/95972159 <== osd.1 v2:172.21.15.167:6800/30035 38 ==== watch-notify(notify (1) cookie 93840928253664 notify 128849018889 ret 0) v3 ==== 731+0+0 (crc 0 0 0) 0x7f35a803d040 con 0x55590b2dc740 2023-10-05T19:28:29.527 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.522+0000 7f35a7fff700 10 rgw watcher librados: RGWWatcher::handle_notify() notify_id 128849018889 cookie 93840928253664 notifier 4377 bl.length()=689 2023-10-05T19:28:29.527 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.522+0000 7f35a7fff700 10 rgw watcher librados: cache put: name=default.rgw.meta+users.uid+foo info.flags=0x17 2023-10-05T19:28:29.527 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.522+0000 7f35a7fff700 10 rgw watcher librados: moving default.rgw.meta+users.uid+foo to cache LRU end 2023-10-05T19:28:29.527 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.522+0000 7f35a7fff700 1 -- 172.21.15.167:0/95972159 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:106 7.d 7:b4812045:::notify.4:head [notify-ack in=20b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x7f3594003d90 con 0x55590b2dc740 2023-10-05T19:28:29.527 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.522+0000 7f35af6f0700 1 -- 172.21.15.167:0/95972159 <== osd.1 v2:172.21.15.167:6800/30035 39 ==== osd_op_reply(105 notify.4 [notify cookie 93840928791264 out=8b] v0'0 uv75 ondisk = 0) v8 ==== 152+0+8 (crc 0 0 0) 0x7f35a8050040 con 0x55590b2dc740 2023-10-05T19:28:29.528 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.522+0000 7f35af6f0700 1 -- 172.21.15.167:0/95972159 <== osd.1 v2:172.21.15.167:6800/30035 40 ==== watch-notify(notify_complete (2) cookie 93840928791264 notify 128849018889 ret 0) v3 ==== 42+0+48 (crc 0 0 0) 0x7f35a8002bd0 con 0x55590b2dc740 2023-10-05T19:28:29.528 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.522+0000 7f35af6f0700 1 -- 172.21.15.167:0/95972159 <== osd.1 v2:172.21.15.167:6800/30035 41 ==== osd_op_reply(106 notify.4 [notify-ack] v0'0 uv75 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f35a8050040 con 0x55590b2dc740 2023-10-05T19:28:29.528 INFO:teuthology.orchestra.run.smithi167.stdout:{ 2023-10-05T19:28:29.528 INFO:teuthology.orchestra.run.smithi167.stdout: "user_id": "foo", 2023-10-05T19:28:29.528 INFO:teuthology.orchestra.run.smithi167.stdout: "display_name": "Foo", 2023-10-05T19:28:29.529 INFO:teuthology.orchestra.run.smithi167.stdout: "email": "foo@foo.com", 2023-10-05T19:28:29.529 INFO:teuthology.orchestra.run.smithi167.stdout: "suspended": 0, 2023-10-05T19:28:29.529 INFO:teuthology.orchestra.run.smithi167.stdout: "max_buckets": 4, 2023-10-05T19:28:29.529 INFO:teuthology.orchestra.run.smithi167.stdout: "subusers": [ 2023-10-05T19:28:29.529 INFO:teuthology.orchestra.run.smithi167.stdout: { 2023-10-05T19:28:29.529 INFO:teuthology.orchestra.run.smithi167.stdout: "id": "foo:foo2", 2023-10-05T19:28:29.529 INFO:teuthology.orchestra.run.smithi167.stdout: "permissions": "" 2023-10-05T19:28:29.529 INFO:teuthology.orchestra.run.smithi167.stdout: } 2023-10-05T19:28:29.529 INFO:teuthology.orchestra.run.smithi167.stdout: ], 2023-10-05T19:28:29.529 INFO:teuthology.orchestra.run.smithi167.stdout: "keys": [ 2023-10-05T19:28:29.530 INFO:teuthology.orchestra.run.smithi167.stdout: { 2023-10-05T19:28:29.530 INFO:teuthology.orchestra.run.smithi167.stdout: "user": "foo", 2023-10-05T19:28:29.530 INFO:teuthology.orchestra.run.smithi167.stdout: "access_key": "9te6NH5mcdcq0Tc5i8i1", 2023-10-05T19:28:29.530 INFO:teuthology.orchestra.run.smithi167.stdout: "secret_key": "Ny4IOauQoL18Gp2zM7lC1vLmoawgqcYP/YGcWfXu" 2023-10-05T19:28:29.530 INFO:teuthology.orchestra.run.smithi167.stdout: } 2023-10-05T19:28:29.530 INFO:teuthology.orchestra.run.smithi167.stdout: ], 2023-10-05T19:28:29.530 INFO:teuthology.orchestra.run.smithi167.stdout: "swift_keys": [ 2023-10-05T19:28:29.530 INFO:teuthology.orchestra.run.smithi167.stdout: { 2023-10-05T19:28:29.530 INFO:teuthology.orchestra.run.smithi167.stdout: "user": "foo:foo2", 2023-10-05T19:28:29.530 INFO:teuthology.orchestra.run.smithi167.stdout: "secret_key": "ri2VJQcKSYATOY6uaDUX7pxgkW+W1YmC6OCxPHwy" 2023-10-05T19:28:29.531 INFO:teuthology.orchestra.run.smithi167.stdout: } 2023-10-05T19:28:29.531 INFO:teuthology.orchestra.run.smithi167.stdout: ], 2023-10-05T19:28:29.531 INFO:teuthology.orchestra.run.smithi167.stdout: "caps": [], 2023-10-05T19:28:29.531 INFO:teuthology.orchestra.run.smithi167.stdout: "op_mask": "read, write, delete", 2023-10-05T19:28:29.531 INFO:teuthology.orchestra.run.smithi167.stdout: "default_placement": "", 2023-10-05T19:28:29.531 INFO:teuthology.orchestra.run.smithi167.stdout: "default_storage_class": "", 2023-10-05T19:28:29.531 INFO:teuthology.orchestra.run.smithi167.stdout: "placement_tags": [], 2023-10-05T19:28:29.531 INFO:teuthology.orchestra.run.smithi167.stdout: "bucket_quota": { 2023-10-05T19:28:29.531 INFO:teuthology.orchestra.run.smithi167.stdout: "enabled": false, 2023-10-05T19:28:29.531 INFO:teuthology.orchestra.run.smithi167.stdout: "check_on_raw": false, 2023-10-05T19:28:29.531 INFO:teuthology.orchestra.run.smithi167.stdout: "max_size": -1, 2023-10-05T19:28:29.531 INFO:teuthology.orchestra.run.smithi167.stdout: "max_size_kb": 0, 2023-10-05T19:28:29.531 INFO:teuthology.orchestra.run.smithi167.stdout: "max_objects": -1 2023-10-05T19:28:29.532 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2023-10-05T19:28:29.532 INFO:teuthology.orchestra.run.smithi167.stdout: "user_quota": { 2023-10-05T19:28:29.532 INFO:teuthology.orchestra.run.smithi167.stdout: "enabled": false, 2023-10-05T19:28:29.532 INFO:teuthology.orchestra.run.smithi167.stdout: "check_on_raw": false, 2023-10-05T19:28:29.532 INFO:teuthology.orchestra.run.smithi167.stdout: "max_size": -1, 2023-10-05T19:28:29.532 INFO:teuthology.orchestra.run.smithi167.stdout: "max_size_kb": 0, 2023-10-05T19:28:29.532 INFO:teuthology.orchestra.run.smithi167.stdout: "max_objects": -1 2023-10-05T19:28:29.532 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2023-10-05T19:28:29.532 INFO:teuthology.orchestra.run.smithi167.stdout: "temp_url_keys": [], 2023-10-05T19:28:29.532 INFO:teuthology.orchestra.run.smithi167.stdout: "type": "rgw", 2023-10-05T19:28:29.532 INFO:teuthology.orchestra.run.smithi167.stdout: "mfa_ids": [] 2023-10-05T19:28:29.533 INFO:teuthology.orchestra.run.smithi167.stdout:} 2023-10-05T19:28:29.533 INFO:teuthology.orchestra.run.smithi167.stdout: 2023-10-05T19:28:29.534 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.522+0000 7f35b13bae40 1 -- 172.21.15.167:0/95972159 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:107 7.1f 7:f95f44c2:::notify.0:head [watch unwatch cookie 93840928245648] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x55590b2f1530 con 0x7f35a0007030 2023-10-05T19:28:29.534 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.526+0000 7f35ae6ee700 1 -- 172.21.15.167:0/95972159 <== osd.2 v2:172.21.15.167:6816/30037 17 ==== osd_op_reply(107 notify.0 [watch unwatch cookie 93840928245648] v30'77 uv75 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f3598045040 con 0x7f35a0007030 2023-10-05T19:28:29.534 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.526+0000 7f35b13bae40 20 remove_watcher() i=0 2023-10-05T19:28:29.534 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.526+0000 7f35b13bae40 2 removed watcher, disabling cache 2023-10-05T19:28:29.534 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.526+0000 7f35b13bae40 1 -- 172.21.15.167:0/95972159 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:108 7.0 7:05bf5b68:::notify.1:head [watch unwatch cookie 93840928250272] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x55590b318790 con 0x55590b2dc740 2023-10-05T19:28:29.534 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.526+0000 7f35af6f0700 1 -- 172.21.15.167:0/95972159 <== osd.1 v2:172.21.15.167:6800/30035 42 ==== osd_op_reply(108 notify.1 [watch unwatch cookie 93840928250272] v30'77 uv75 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f35a8050040 con 0x55590b2dc740 2023-10-05T19:28:29.534 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.526+0000 7f35b13bae40 20 remove_watcher() i=1 2023-10-05T19:28:29.534 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.526+0000 7f35b13bae40 1 -- 172.21.15.167:0/95972159 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:109 7.15 7:a93a5511:::notify.2:head [watch unwatch cookie 93840928252560] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x55590b3199a0 con 0x55590b3133d0 2023-10-05T19:28:29.536 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.530+0000 7f35aeeef700 1 -- 172.21.15.167:0/95972159 <== osd.3 v2:172.21.15.167:6808/30036 24 ==== osd_op_reply(109 notify.2 [watch unwatch cookie 93840928252560] v30'77 uv75 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f35a0042040 con 0x55590b3133d0 2023-10-05T19:28:29.536 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.530+0000 7f35b13bae40 20 remove_watcher() i=2 2023-10-05T19:28:29.536 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.530+0000 7f35b13bae40 1 -- 172.21.15.167:0/95972159 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:110 7.e 7:7759931f:::notify.3:head [watch unwatch cookie 93840928256400] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x55590b318790 con 0x7f35a0007030 2023-10-05T19:28:29.538 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.534+0000 7f35ae6ee700 1 -- 172.21.15.167:0/95972159 <== osd.2 v2:172.21.15.167:6816/30037 18 ==== osd_op_reply(110 notify.3 [watch unwatch cookie 93840928256400] v30'77 uv75 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f3598045040 con 0x7f35a0007030 2023-10-05T19:28:29.538 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.534+0000 7f35b13bae40 20 remove_watcher() i=3 2023-10-05T19:28:29.538 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.534+0000 7f35b13bae40 1 -- 172.21.15.167:0/95972159 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:111 7.d 7:b4812045:::notify.4:head [watch unwatch cookie 93840928253664] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x55590b31a290 con 0x55590b2dc740 2023-10-05T19:28:29.541 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.534+0000 7f35af6f0700 1 -- 172.21.15.167:0/95972159 <== osd.1 v2:172.21.15.167:6800/30035 43 ==== osd_op_reply(111 notify.4 [watch unwatch cookie 93840928253664] v30'77 uv75 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f35a8050040 con 0x55590b2dc740 2023-10-05T19:28:29.541 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.534+0000 7f35b13bae40 20 remove_watcher() i=4 2023-10-05T19:28:29.541 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.534+0000 7f35b13bae40 1 -- 172.21.15.167:0/95972159 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:112 7.3 7:c609908c:::notify.5:head [watch unwatch cookie 93840928261824] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x55590b3199a0 con 0x7f3598011040 2023-10-05T19:28:29.544 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.538+0000 7f35af6f0700 1 -- 172.21.15.167:0/95972159 <== osd.0 v2:172.21.15.118:6800/27690 29 ==== osd_op_reply(112 notify.5 [watch unwatch cookie 93840928261824] v30'77 uv75 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f35a8050040 con 0x7f3598011040 2023-10-05T19:28:29.544 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.538+0000 7f35b13bae40 20 remove_watcher() i=5 2023-10-05T19:28:29.544 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.538+0000 7f35b13bae40 1 -- 172.21.15.167:0/95972159 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:113 7.14 7:2b04a3e9:::notify.6:head [watch unwatch cookie 93840928265728] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x55590b318790 con 0x7f3598011040 2023-10-05T19:28:29.547 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.542+0000 7f35af6f0700 1 -- 172.21.15.167:0/95972159 <== osd.0 v2:172.21.15.118:6800/27690 30 ==== osd_op_reply(113 notify.6 [watch unwatch cookie 93840928265728] v30'77 uv75 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f35a8050040 con 0x7f3598011040 2023-10-05T19:28:29.547 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.542+0000 7f35b13bae40 20 remove_watcher() i=6 2023-10-05T19:28:29.547 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.542+0000 7f35b13bae40 1 -- 172.21.15.167:0/95972159 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:114 7.9 7:93e5b521:::notify.7:head [watch unwatch cookie 93840928262928] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x55590b31c6c0 con 0x7f3598011040 2023-10-05T19:28:29.549 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.542+0000 7f35af6f0700 1 -- 172.21.15.167:0/95972159 <== osd.0 v2:172.21.15.118:6800/27690 31 ==== osd_op_reply(114 notify.7 [watch unwatch cookie 93840928262928] v30'77 uv75 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f35a8050040 con 0x7f3598011040 2023-10-05T19:28:29.550 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.542+0000 7f35b13bae40 20 remove_watcher() i=7 2023-10-05T19:28:29.551 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.546+0000 7f34fbfff700 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2023-10-05T19:28:29.551 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.546+0000 7f35b13bae40 1 -- 172.21.15.167:0/95972159 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:115 6.4 6:22d26bf9:::data_loggenerations_metadata:head [watch unwatch cookie 93840928269648] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x55590b3199a0 con 0x55590b2dc740 2023-10-05T19:28:29.555 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.550+0000 7f35af6f0700 1 -- 172.21.15.167:0/95972159 <== osd.1 v2:172.21.15.167:6800/30035 44 ==== osd_op_reply(115 data_loggenerations_metadata [watch unwatch cookie 93840928269648] v30'85 uv1 ondisk = 0) v8 ==== 172+0+0 (crc 0 0 0) 0x7f35a8050040 con 0x55590b2dc740 2023-10-05T19:28:29.555 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.550+0000 7f35b13bae40 1 -- 172.21.15.167:0/95972159 >> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] conn(0x7f3598011040 msgr2=0x7f3598014950 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-05T19:28:29.555 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.550+0000 7f35b13bae40 1 --2- 172.21.15.167:0/95972159 >> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] conn(0x7f3598011040 0x7f3598014950 crc :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-05T19:28:29.555 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.550+0000 7f35b13bae40 1 -- 172.21.15.167:0/95972159 >> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] conn(0x55590b2dc740 msgr2=0x55590b30f5f0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-05T19:28:29.555 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.550+0000 7f35b13bae40 1 --2- 172.21.15.167:0/95972159 >> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] conn(0x55590b2dc740 0x55590b30f5f0 crc :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-05T19:28:29.556 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.550+0000 7f35b13bae40 1 -- 172.21.15.167:0/95972159 >> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] conn(0x7f35a0007030 msgr2=0x7f35a0031480 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-05T19:28:29.556 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.550+0000 7f35b13bae40 1 --2- 172.21.15.167:0/95972159 >> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] conn(0x7f35a0007030 0x7f35a0031480 crc :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-05T19:28:29.556 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.550+0000 7f35b13bae40 1 -- 172.21.15.167:0/95972159 >> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] conn(0x55590b3133d0 msgr2=0x55590b3157b0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-05T19:28:29.556 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.550+0000 7f35b13bae40 1 --2- 172.21.15.167:0/95972159 >> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] conn(0x55590b3133d0 0x55590b3157b0 crc :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-05T19:28:29.556 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.550+0000 7f35b13bae40 1 -- 172.21.15.167:0/95972159 >> [v2:172.21.15.167:6824/29889,v1:172.21.15.167:6825/29889] conn(0x7f3590030c00 msgr2=0x7f35900330b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-05T19:28:29.556 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.550+0000 7f35b13bae40 1 --2- 172.21.15.167:0/95972159 >> [v2:172.21.15.167:6824/29889,v1:172.21.15.167:6825/29889] conn(0x7f3590030c00 0x7f35900330b0 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7f3598001560 tx=0x7f3598006650 comp rx=0 tx=0).stop 2023-10-05T19:28:29.556 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.550+0000 7f35af6f0700 1 -- 172.21.15.167:0/95972159 reap_dead start 2023-10-05T19:28:29.556 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.550+0000 7f35b13bae40 1 -- 172.21.15.167:0/95972159 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55590b2dbe30 msgr2=0x55590b303ac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-05T19:28:29.556 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.550+0000 7f35b13bae40 1 --2- 172.21.15.167:0/95972159 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55590b2dbe30 0x55590b303ac0 secure :-1 s=READY pgs=287 cs=0 l=1 rev1=1 crypto rx=0x7f35a001e040 tx=0x7f35a0003050 comp rx=0 tx=0).stop 2023-10-05T19:28:29.556 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.550+0000 7f35b13bae40 1 -- 172.21.15.167:0/95972159 shutdown_connections 2023-10-05T19:28:29.556 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.550+0000 7f35b13bae40 1 --2- 172.21.15.167:0/95972159 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55590b2dbe30 0x55590b303ac0 secure :-1 s=CLOSED pgs=287 cs=0 l=1 rev1=1 crypto rx=0x7f35a001e040 tx=0x7f35a0003050 comp rx=0 tx=0).stop 2023-10-05T19:28:29.557 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.550+0000 7f35b13bae40 1 -- 172.21.15.167:0/95972159 >> 172.21.15.167:0/95972159 conn(0x55590b2d7850 msgr2=0x55590b2f52d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-05T19:28:29.557 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.550+0000 7f35b13bae40 1 -- 172.21.15.167:0/95972159 shutdown_connections 2023-10-05T19:28:29.557 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.550+0000 7f35b13bae40 1 -- 172.21.15.167:0/95972159 wait complete. 2023-10-05T19:28:29.558 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.550+0000 7f35b13bae40 1 -- 172.21.15.167:0/3021234063 >> [v2:172.21.15.167:6824/29889,v1:172.21.15.167:6825/29889] conn(0x7f34ec030c50 msgr2=0x7f34ec033100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-05T19:28:29.558 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.550+0000 7f35b13bae40 1 --2- 172.21.15.167:0/3021234063 >> [v2:172.21.15.167:6824/29889,v1:172.21.15.167:6825/29889] conn(0x7f34ec030c50 0x7f34ec033100 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x55590b31dc50 tx=0x7f35a8000f90 comp rx=0 tx=0).stop 2023-10-05T19:28:29.558 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.550+0000 7f35b13bae40 1 -- 172.21.15.167:0/3021234063 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55590b32be60 msgr2=0x55590b3319b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-05T19:28:29.558 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.550+0000 7f35b13bae40 1 --2- 172.21.15.167:0/3021234063 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55590b32be60 0x55590b3319b0 secure :-1 s=READY pgs=289 cs=0 l=1 rev1=1 crypto rx=0x7f3598006040 tx=0x7f359803c000 comp rx=0 tx=0).stop 2023-10-05T19:28:29.558 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.550+0000 7f35b13bae40 1 -- 172.21.15.167:0/3021234063 shutdown_connections 2023-10-05T19:28:29.558 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.550+0000 7f35b13bae40 1 --2- 172.21.15.167:0/3021234063 >> [v2:172.21.15.167:6824/29889,v1:172.21.15.167:6825/29889] conn(0x7f34ec030c50 0x7f34ec033100 unknown :-1 s=CLOSED pgs=103 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-05T19:28:29.558 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.550+0000 7f35b13bae40 1 --2- 172.21.15.167:0/3021234063 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55590b32be60 0x55590b3319b0 unknown :-1 s=CLOSED pgs=289 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-05T19:28:29.558 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.550+0000 7f35b13bae40 1 -- 172.21.15.167:0/3021234063 >> 172.21.15.167:0/3021234063 conn(0x55590b327220 msgr2=0x55590b3252c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-05T19:28:29.559 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.550+0000 7f35b13bae40 1 -- 172.21.15.167:0/3021234063 shutdown_connections 2023-10-05T19:28:29.559 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.550+0000 7f35b13bae40 1 -- 172.21.15.167:0/3021234063 wait complete. 2023-10-05T19:28:29.561 DEBUG:tasks.util.rgw: json result: {'user_id': 'foo', 'display_name': 'Foo', 'email': 'foo@foo.com', 'suspended': 0, 'max_buckets': 4, 'subusers': [{'id': 'foo:foo2', 'permissions': ''}], 'keys': [{'user': 'foo', 'access_key': '9te6NH5mcdcq0Tc5i8i1', 'secret_key': 'Ny4IOauQoL18Gp2zM7lC1vLmoawgqcYP/YGcWfXu'}], 'swift_keys': [{'user': 'foo:foo2', 'secret_key': 'ri2VJQcKSYATOY6uaDUX7pxgkW+W1YmC6OCxPHwy'}], 'caps': [], 'op_mask': 'read, write, delete', 'default_placement': '', 'default_storage_class': '', 'placement_tags': [], 'bucket_quota': {'enabled': False, 'check_on_raw': False, 'max_size': -1, 'max_size_kb': 0, 'max_objects': -1}, 'user_quota': {'enabled': False, 'check_on_raw': False, 'max_size': -1, 'max_size_kb': 0, 'max_objects': -1}, 'temp_url_keys': [], 'type': 'rgw', 'mfa_ids': []} 2023-10-05T19:28:29.562 INFO:tasks.util.rgw:rgwadmin: client.0 : ['subuser', 'rm', '--subuser', 'foo:foo2', '--key-type', 'swift', '--purge-keys'] 2023-10-05T19:28:29.562 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'ceph', 'subuser', 'rm', '--subuser', 'foo:foo2', '--key-type', 'swift', '--purge-keys'] 2023-10-05T19:28:29.562 DEBUG:teuthology.orchestra.run.smithi167:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster ceph subuser rm --subuser foo:foo2 --key-type swift --purge-keys 2023-10-05T19:28:29.603 INFO:teuthology.orchestra.run.smithi167.stderr:ignoring --setuser ceph since I am not root 2023-10-05T19:28:29.603 INFO:teuthology.orchestra.run.smithi167.stderr:ignoring --setgroup ceph since I am not root 2023-10-05T19:28:29.604 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.598+0000 7f1d97962e40 1 Processor -- start 2023-10-05T19:28:29.604 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.598+0000 7f1d97962e40 1 -- start start 2023-10-05T19:28:29.604 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.598+0000 7f1d97962e40 1 --2- >> v2:172.21.15.118:3300/0 conn(0x55fc75210fb0 0x55fc75211380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-05T19:28:29.604 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.598+0000 7f1d97962e40 1 -- --> v1:172.21.15.118:6789/0 -- auth(proto 0 26 bytes epoch 0) v1 -- 0x55fc7515bbd0 con 0x55fc752118c0 2023-10-05T19:28:29.605 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.598+0000 7f1d97962e40 1 -- --> v2:172.21.15.118:3300/0 -- mon_getmap magic: 0 v1 -- 0x55fc75118ee0 con 0x55fc75210fb0 2023-10-05T19:28:29.605 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.598+0000 7f1d95497700 1 --2- >> v2:172.21.15.118:3300/0 conn(0x55fc75210fb0 0x55fc75211380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-05T19:28:29.605 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.598+0000 7f1d8cc96700 1 --1- >> v1:172.21.15.118:6789/0 conn(0x55fc752118c0 0x55fc75212360 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.118:6789/0 says I am v1:172.21.15.167:56528/0 (socket says 172.21.15.167:56528) 2023-10-05T19:28:29.606 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.598+0000 7f1d8cc96700 1 -- 172.21.15.167:0/882433413 learned_addr learned my addr 172.21.15.167:0/882433413 (peer_addr_for_me v1:172.21.15.167:0/0) 2023-10-05T19:28:29.606 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.598+0000 7f1d94c96700 1 -- 172.21.15.167:0/882433413 <== mon.0 v1:172.21.15.118:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3152603946 0 0) 0x55fc7515bbd0 con 0x55fc752118c0 2023-10-05T19:28:29.606 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.598+0000 7f1d94c96700 1 -- 172.21.15.167:0/882433413 --> v1:172.21.15.118:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1d84001e10 con 0x55fc752118c0 2023-10-05T19:28:29.606 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.598+0000 7f1d95497700 1 -- 172.21.15.167:0/882433413 >> v1:172.21.15.118:6789/0 conn(0x55fc752118c0 legacy=0x55fc75212360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-05T19:28:29.606 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.598+0000 7f1d95497700 1 -- 172.21.15.167:0/882433413 --> v2:172.21.15.118:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55fc75212a70 con 0x55fc75210fb0 2023-10-05T19:28:29.607 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.602+0000 7f1d95497700 1 --2- 172.21.15.167:0/882433413 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55fc75210fb0 0x55fc75211380 secure :-1 s=READY pgs=291 cs=0 l=1 rev1=1 crypto rx=0x7f1d8800a5d0 tx=0x7f1d88005a50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=223b5482c165dba2 server_cookie=0 in_seq=0 out_seq=0 2023-10-05T19:28:29.607 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.602+0000 7f1d94c96700 1 -- 172.21.15.167:0/882433413 <== mon.0 v2:172.21.15.118:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f1d88007610 con 0x55fc75210fb0 2023-10-05T19:28:29.607 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.602+0000 7f1d94c96700 1 -- 172.21.15.167:0/882433413 <== mon.0 v2:172.21.15.118:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1d88007770 con 0x55fc75210fb0 2023-10-05T19:28:29.607 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.602+0000 7f1d94c96700 1 -- 172.21.15.167:0/882433413 <== mon.0 v2:172.21.15.118:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f1d88007a40 con 0x55fc75210fb0 2023-10-05T19:28:29.608 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.602+0000 7f1d97962e40 1 -- 172.21.15.167:0/882433413 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55fc75210fb0 msgr2=0x55fc75211380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-05T19:28:29.608 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.602+0000 7f1d97962e40 1 --2- 172.21.15.167:0/882433413 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55fc75210fb0 0x55fc75211380 secure :-1 s=READY pgs=291 cs=0 l=1 rev1=1 crypto rx=0x7f1d8800a5d0 tx=0x7f1d88005a50 comp rx=0 tx=0).stop 2023-10-05T19:28:29.608 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.602+0000 7f1d97962e40 1 -- 172.21.15.167:0/882433413 shutdown_connections 2023-10-05T19:28:29.608 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.602+0000 7f1d97962e40 1 --2- 172.21.15.167:0/882433413 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55fc75210fb0 0x55fc75211380 unknown :-1 s=CLOSED pgs=291 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-05T19:28:29.608 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.602+0000 7f1d97962e40 1 -- 172.21.15.167:0/882433413 >> 172.21.15.167:0/882433413 conn(0x55fc75184800 msgr2=0x55fc75184c00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-05T19:28:29.608 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.602+0000 7f1d97962e40 1 -- 172.21.15.167:0/882433413 shutdown_connections 2023-10-05T19:28:29.608 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.602+0000 7f1d97962e40 1 -- 172.21.15.167:0/882433413 wait complete. 2023-10-05T19:28:29.609 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.602+0000 7f1d97962e40 1 Processor -- start 2023-10-05T19:28:29.609 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.602+0000 7f1d97962e40 1 -- start start 2023-10-05T19:28:29.609 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.602+0000 7f1d97962e40 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55fc75210fb0 0x55fc7522ff30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-05T19:28:29.609 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.602+0000 7f1d97962e40 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x55fc750dce30 con 0x55fc75210fb0 2023-10-05T19:28:29.609 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.602+0000 7f1d95497700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55fc75210fb0 0x55fc7522ff30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-05T19:28:29.610 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.602+0000 7f1d95497700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55fc75210fb0 0x55fc7522ff30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.118:3300/0 says I am v2:172.21.15.167:57946/0 (socket says 172.21.15.167:57946) 2023-10-05T19:28:29.610 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.602+0000 7f1d95497700 1 -- 172.21.15.167:0/1218699573 learned_addr learned my addr 172.21.15.167:0/1218699573 (peer_addr_for_me v2:172.21.15.167:0/0) 2023-10-05T19:28:29.610 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.602+0000 7f1d95497700 1 -- 172.21.15.167:0/1218699573 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1d88005cf0 con 0x55fc75210fb0 2023-10-05T19:28:29.610 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.602+0000 7f1d95497700 1 --2- 172.21.15.167:0/1218699573 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55fc75210fb0 0x55fc7522ff30 secure :-1 s=READY pgs=292 cs=0 l=1 rev1=1 crypto rx=0x7f1d88000c00 tx=0x7f1d88014000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2ab517918368f3aa server_cookie=0 in_seq=0 out_seq=0 2023-10-05T19:28:29.610 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.606+0000 7f1d8f7fe700 1 -- 172.21.15.167:0/1218699573 <== mon.0 v2:172.21.15.118:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f1d880078e0 con 0x55fc75210fb0 2023-10-05T19:28:29.610 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.606+0000 7f1d8f7fe700 1 -- 172.21.15.167:0/1218699573 <== mon.0 v2:172.21.15.118:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1d88007a40 con 0x55fc75210fb0 2023-10-05T19:28:29.611 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.606+0000 7f1d8f7fe700 1 -- 172.21.15.167:0/1218699573 <== mon.0 v2:172.21.15.118:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f1d88007bd0 con 0x55fc75210fb0 2023-10-05T19:28:29.612 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.606+0000 7f1d97962e40 1 -- 172.21.15.167:0/1218699573 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55fc75210fb0 msgr2=0x55fc7522ff30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-05T19:28:29.612 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.606+0000 7f1d97962e40 1 --2- 172.21.15.167:0/1218699573 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55fc75210fb0 0x55fc7522ff30 secure :-1 s=READY pgs=292 cs=0 l=1 rev1=1 crypto rx=0x7f1d88000c00 tx=0x7f1d88014000 comp rx=0 tx=0).stop 2023-10-05T19:28:29.612 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.606+0000 7f1d97962e40 1 -- 172.21.15.167:0/1218699573 shutdown_connections 2023-10-05T19:28:29.612 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.606+0000 7f1d97962e40 1 --2- 172.21.15.167:0/1218699573 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55fc75210fb0 0x55fc7522ff30 unknown :-1 s=CLOSED pgs=292 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-05T19:28:29.612 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.606+0000 7f1d97962e40 1 -- 172.21.15.167:0/1218699573 >> 172.21.15.167:0/1218699573 conn(0x55fc75184800 msgr2=0x55fc7522ba70 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-05T19:28:29.612 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.606+0000 7f1d97962e40 1 -- 172.21.15.167:0/1218699573 shutdown_connections 2023-10-05T19:28:29.612 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.606+0000 7f1d97962e40 1 -- 172.21.15.167:0/1218699573 wait complete. 2023-10-05T19:28:29.613 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.606+0000 7f1d97962e40 1 Processor -- start 2023-10-05T19:28:29.613 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.606+0000 7f1d97962e40 1 -- start start 2023-10-05T19:28:29.613 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.606+0000 7f1d97962e40 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55fc75210fb0 0x55fc75236c30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-05T19:28:29.613 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.606+0000 7f1d97962e40 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x55fc750afed0 con 0x55fc75210fb0 2023-10-05T19:28:29.614 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.606+0000 7f1d95497700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55fc75210fb0 0x55fc75236c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-05T19:28:29.614 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.606+0000 7f1d95497700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55fc75210fb0 0x55fc75236c30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.118:3300/0 says I am v2:172.21.15.167:57962/0 (socket says 172.21.15.167:57962) 2023-10-05T19:28:29.614 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.606+0000 7f1d95497700 1 -- 172.21.15.167:0/1453987703 learned_addr learned my addr 172.21.15.167:0/1453987703 (peer_addr_for_me v2:172.21.15.167:0/0) 2023-10-05T19:28:29.614 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.610+0000 7f1d95497700 1 -- 172.21.15.167:0/1453987703 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55fc75212a70 con 0x55fc75210fb0 2023-10-05T19:28:29.614 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.610+0000 7f1d95497700 1 --2- 172.21.15.167:0/1453987703 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55fc75210fb0 0x55fc75236c30 secure :-1 s=READY pgs=293 cs=0 l=1 rev1=1 crypto rx=0x7f1d8801f040 tx=0x7f1d8801c000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-05T19:28:29.615 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.610+0000 7f1d8dffb700 1 -- 172.21.15.167:0/1453987703 <== mon.0 v2:172.21.15.118:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f1d88007e50 con 0x55fc75210fb0 2023-10-05T19:28:29.615 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.610+0000 7f1d8dffb700 1 -- 172.21.15.167:0/1453987703 <== mon.0 v2:172.21.15.118:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1d88029070 con 0x55fc75210fb0 2023-10-05T19:28:29.615 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.610+0000 7f1d97962e40 1 -- 172.21.15.167:0/1453987703 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x55fc75237170 con 0x55fc75210fb0 2023-10-05T19:28:29.615 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.610+0000 7f1d8dffb700 1 -- 172.21.15.167:0/1453987703 <== mon.0 v2:172.21.15.118:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f1d88030480 con 0x55fc75210fb0 2023-10-05T19:28:29.615 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.610+0000 7f1d97962e40 1 -- 172.21.15.167:0/1453987703 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55fc752375b0 con 0x55fc75210fb0 2023-10-05T19:28:29.616 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.610+0000 7f1d8dffb700 1 -- 172.21.15.167:0/1453987703 <== mon.0 v2:172.21.15.118:3300/0 4 ==== mgrmap(e 5) v1 ==== 40329+0+0 (secure 0 0 0) 0x7f1d88012400 con 0x55fc75210fb0 2023-10-05T19:28:29.616 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.610+0000 7f1d8dffb700 1 --2- 172.21.15.167:0/1453987703 >> [v2:172.21.15.167:6824/29889,v1:172.21.15.167:6825/29889] conn(0x7f1d78030bf0 0x7f1d780330a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-05T19:28:29.616 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.610+0000 7f1d8dffb700 1 -- 172.21.15.167:0/1453987703 <== mon.0 v2:172.21.15.118:3300/0 5 ==== osd_map(30..30 src has 1..30) v4 ==== 6057+0+0 (secure 0 0 0) 0x7f1d8802b070 con 0x55fc75210fb0 2023-10-05T19:28:29.617 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.610+0000 7f1d8cc96700 1 --2- 172.21.15.167:0/1453987703 >> [v2:172.21.15.167:6824/29889,v1:172.21.15.167:6825/29889] conn(0x7f1d78030bf0 0x7f1d780330a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-05T19:28:29.617 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.610+0000 7f1d97962e40 20 rados->read ofs=0 len=0 2023-10-05T19:28:29.618 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.610+0000 7f1d97962e40 1 --2- 172.21.15.167:0/1453987703 >> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] conn(0x55fc752118c0 0x55fc75242670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-05T19:28:29.618 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.610+0000 7f1d97962e40 1 -- 172.21.15.167:0/1453987703 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:1 5.12 5:49953fa1:::default.realm:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x55fc75242da0 con 0x55fc752118c0 2023-10-05T19:28:29.618 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.610+0000 7f1d95c98700 1 --2- 172.21.15.167:0/1453987703 >> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] conn(0x55fc752118c0 0x55fc75242670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-05T19:28:29.618 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.614+0000 7f1d8cc96700 1 --2- 172.21.15.167:0/1453987703 >> [v2:172.21.15.167:6824/29889,v1:172.21.15.167:6825/29889] conn(0x7f1d78030bf0 0x7f1d780330a0 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7f1d80004170 tx=0x7f1d80001400 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-05T19:28:29.618 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.614+0000 7f1d95c98700 1 --2- 172.21.15.167:0/1453987703 >> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] conn(0x55fc752118c0 0x55fc75242670 crc :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-05T19:28:29.619 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.614+0000 7f1d95c98700 1 -- 172.21.15.167:0/1453987703 <== osd.1 v2:172.21.15.167:6800/30035 1 ==== osd_op_reply(1 default.realm [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 ==== 157+0+0 (crc 0 0 0) 0x7f1d9000a5a0 con 0x55fc752118c0 2023-10-05T19:28:29.619 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.614+0000 7f1d97962e40 20 rados_obj.operate() r=-2 bl.length=0 2023-10-05T19:28:29.619 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.614+0000 7f1d97962e40 20 realm 2023-10-05T19:28:29.619 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.614+0000 7f1d97962e40 20 rados->read ofs=0 len=0 2023-10-05T19:28:29.620 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.614+0000 7f1d97962e40 1 -- 172.21.15.167:0/1453987703 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:2 5.12 5:49953fa1:::default.realm:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x55fc7523fc50 con 0x55fc752118c0 2023-10-05T19:28:29.620 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.614+0000 7f1d95c98700 1 -- 172.21.15.167:0/1453987703 <== osd.1 v2:172.21.15.167:6800/30035 2 ==== osd_op_reply(2 default.realm [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 ==== 157+0+0 (crc 0 0 0) 0x7f1d9000a5a0 con 0x55fc752118c0 2023-10-05T19:28:29.621 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.614+0000 7f1d97962e40 20 rados_obj.operate() r=-2 bl.length=0 2023-10-05T19:28:29.621 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.614+0000 7f1d97962e40 4 RGWPeriod::init failed to init realm id : (2) No such file or directory 2023-10-05T19:28:29.621 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.614+0000 7f1d97962e40 20 rados->read ofs=0 len=0 2023-10-05T19:28:29.621 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.614+0000 7f1d97962e40 1 -- 172.21.15.167:0/1453987703 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:3 5.12 5:49953fa1:::default.realm:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x55fc75243880 con 0x55fc752118c0 2023-10-05T19:28:29.621 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.614+0000 7f1d95c98700 1 -- 172.21.15.167:0/1453987703 <== osd.1 v2:172.21.15.167:6800/30035 3 ==== osd_op_reply(3 default.realm [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 ==== 157+0+0 (crc 0 0 0) 0x7f1d9000a5a0 con 0x55fc752118c0 2023-10-05T19:28:29.621 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.614+0000 7f1d97962e40 20 rados_obj.operate() r=-2 bl.length=0 2023-10-05T19:28:29.621 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.614+0000 7f1d97962e40 20 rados->read ofs=0 len=0 2023-10-05T19:28:29.621 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.614+0000 7f1d97962e40 1 -- 172.21.15.167:0/1453987703 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:4 5.f 5:f43b6ece:::zone_names.default:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x55fc75243490 con 0x55fc752118c0 2023-10-05T19:28:29.622 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.614+0000 7f1d95c98700 1 -- 172.21.15.167:0/1453987703 <== osd.1 v2:172.21.15.167:6800/30035 4 ==== osd_op_reply(4 zone_names.default [read 0~46 out=46b] v0'0 uv1 ondisk = 0) v8 ==== 162+0+46 (crc 0 0 0) 0x7f1d9000a5a0 con 0x55fc752118c0 2023-10-05T19:28:29.622 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.614+0000 7f1d97962e40 20 rados_obj.operate() r=0 bl.length=46 2023-10-05T19:28:29.622 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.614+0000 7f1d97962e40 20 rados->read ofs=0 len=0 2023-10-05T19:28:29.622 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.614+0000 7f1d97962e40 1 -- 172.21.15.167:0/1453987703 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:5 5.1d 5:b8374eb4:::zone_info.9a9df965-6612-45d4-a45f-6e3c889bacc2:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x55fc75244710 con 0x55fc752118c0 2023-10-05T19:28:29.623 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.618+0000 7f1d95c98700 1 -- 172.21.15.167:0/1453987703 <== osd.1 v2:172.21.15.167:6800/30035 5 ==== osd_op_reply(5 zone_info.9a9df965-6612-45d4-a45f-6e3c889bacc2 [read 0~874 out=874b] v0'0 uv2 ondisk = 0) v8 ==== 190+0+874 (crc 0 0 0) 0x7f1d9000a5a0 con 0x55fc752118c0 2023-10-05T19:28:29.623 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.618+0000 7f1d97962e40 20 rados_obj.operate() r=0 bl.length=874 2023-10-05T19:28:29.623 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.618+0000 7f1d97962e40 20 searching for the correct realm 2023-10-05T19:28:29.623 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.618+0000 7f1d97962e40 1 --2- 172.21.15.167:0/1453987703 >> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] conn(0x55fc752464d0 0x55fc752488b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-05T19:28:29.623 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.618+0000 7f1d97962e40 1 -- 172.21.15.167:0/1453987703 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:6 5.0 5:00000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x55fc75248df0 con 0x55fc752464d0 2023-10-05T19:28:29.624 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.618+0000 7f1d95497700 1 --2- 172.21.15.167:0/1453987703 >> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] conn(0x55fc752464d0 0x55fc752488b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-05T19:28:29.624 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.618+0000 7f1d95497700 1 --2- 172.21.15.167:0/1453987703 >> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] conn(0x55fc752464d0 0x55fc752488b0 crc :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-05T19:28:29.626 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.622+0000 7f1d95497700 1 -- 172.21.15.167:0/1453987703 <== osd.3 v2:172.21.15.167:6808/30036 1 ==== osd_op_reply(6 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f1d8801c7f0 con 0x55fc752464d0 2023-10-05T19:28:29.626 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.622+0000 7f1d95497700 1 --2- 172.21.15.167:0/1453987703 >> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] conn(0x7f1d88007030 0x7f1d88006820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-05T19:28:29.626 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.622+0000 7f1d95497700 1 -- 172.21.15.167:0/1453987703 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:7 5.10 5:08000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x55fc75248df0 con 0x7f1d88007030 2023-10-05T19:28:29.627 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.622+0000 7f1d8cc96700 1 --2- 172.21.15.167:0/1453987703 >> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] conn(0x7f1d88007030 0x7f1d88006820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-05T19:28:29.627 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.622+0000 7f1d8cc96700 1 --2- 172.21.15.167:0/1453987703 >> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] conn(0x7f1d88007030 0x7f1d88006820 crc :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-05T19:28:29.628 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.622+0000 7f1d8cc96700 1 -- 172.21.15.167:0/1453987703 <== osd.2 v2:172.21.15.167:6816/30037 1 ==== osd_op_reply(7 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f1d80006070 con 0x7f1d88007030 2023-10-05T19:28:29.628 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.622+0000 7f1d8cc96700 1 --2- 172.21.15.167:0/1453987703 >> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] conn(0x7f1d8000eb60 0x7f1d8000d8e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-05T19:28:29.628 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.622+0000 7f1d8cc96700 1 -- 172.21.15.167:0/1453987703 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:8 5.8 5:10000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x55fc75248df0 con 0x7f1d8000eb60 2023-10-05T19:28:29.628 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.622+0000 7f1d95c98700 1 --2- 172.21.15.167:0/1453987703 >> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] conn(0x7f1d8000eb60 0x7f1d8000d8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-05T19:28:29.629 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.622+0000 7f1d95c98700 1 --2- 172.21.15.167:0/1453987703 >> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] conn(0x7f1d8000eb60 0x7f1d8000d8e0 crc :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-05T19:28:29.630 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.622+0000 7f1d95c98700 1 -- 172.21.15.167:0/1453987703 <== osd.0 v2:172.21.15.118:6800/27690 1 ==== osd_op_reply(8 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f1d9000a5a0 con 0x7f1d8000eb60 2023-10-05T19:28:29.630 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.622+0000 7f1d95c98700 1 -- 172.21.15.167:0/1453987703 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:9 5.18 5:18000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x55fc75248df0 con 0x55fc752118c0 2023-10-05T19:28:29.630 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.626+0000 7f1d95c98700 1 -- 172.21.15.167:0/1453987703 <== osd.1 v2:172.21.15.167:6800/30035 6 ==== osd_op_reply(9 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f1d9000a5a0 con 0x55fc752118c0 2023-10-05T19:28:29.631 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.626+0000 7f1d95c98700 1 -- 172.21.15.167:0/1453987703 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:10 5.4 5:20000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x55fc75248df0 con 0x55fc752118c0 2023-10-05T19:28:29.631 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.626+0000 7f1d95c98700 1 -- 172.21.15.167:0/1453987703 <== osd.1 v2:172.21.15.167:6800/30035 7 ==== osd_op_reply(10 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f1d9000a5a0 con 0x55fc752118c0 2023-10-05T19:28:29.631 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.626+0000 7f1d95c98700 1 -- 172.21.15.167:0/1453987703 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:11 5.14 5:28000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x55fc75248df0 con 0x7f1d8000eb60 2023-10-05T19:28:29.632 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.626+0000 7f1d95c98700 1 -- 172.21.15.167:0/1453987703 <== osd.0 v2:172.21.15.118:6800/27690 2 ==== osd_op_reply(11 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f1d9000a5a0 con 0x7f1d8000eb60 2023-10-05T19:28:29.632 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.626+0000 7f1d95c98700 1 -- 172.21.15.167:0/1453987703 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:12 5.c 5:30000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x55fc75248df0 con 0x7f1d8000eb60 2023-10-05T19:28:29.632 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.626+0000 7f1d95c98700 1 -- 172.21.15.167:0/1453987703 <== osd.0 v2:172.21.15.118:6800/27690 3 ==== osd_op_reply(12 [pgnls start_epoch 30 out=112b] v23'1 uv1 ondisk = 1) v8 ==== 144+0+112 (crc 0 0 0) 0x7f1d9000a5a0 con 0x7f1d8000eb60 2023-10-05T19:28:29.632 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.626+0000 7f1d95c98700 1 -- 172.21.15.167:0/1453987703 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:13 5.1c 5:38000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x55fc75248df0 con 0x7f1d88007030 2023-10-05T19:28:29.633 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.626+0000 7f1d8cc96700 1 -- 172.21.15.167:0/1453987703 <== osd.2 v2:172.21.15.167:6816/30037 2 ==== osd_op_reply(13 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f1d80006070 con 0x7f1d88007030 2023-10-05T19:28:29.633 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.626+0000 7f1d8cc96700 1 -- 172.21.15.167:0/1453987703 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:14 5.2 5:40000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x55fc75248df0 con 0x55fc752464d0 2023-10-05T19:28:29.634 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.630+0000 7f1d95497700 1 -- 172.21.15.167:0/1453987703 <== osd.3 v2:172.21.15.167:6808/30036 2 ==== osd_op_reply(14 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f1d8801c7f0 con 0x55fc752464d0 2023-10-05T19:28:29.634 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.630+0000 7f1d95497700 1 -- 172.21.15.167:0/1453987703 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:15 5.12 5:48000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x55fc75248df0 con 0x55fc752118c0 2023-10-05T19:28:29.634 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.630+0000 7f1d95c98700 1 -- 172.21.15.167:0/1453987703 <== osd.1 v2:172.21.15.167:6800/30035 8 ==== osd_op_reply(15 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f1d9000a5a0 con 0x55fc752118c0 2023-10-05T19:28:29.634 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.630+0000 7f1d95c98700 1 -- 172.21.15.167:0/1453987703 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:16 5.a 5:50000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x55fc75248df0 con 0x7f1d8000eb60 2023-10-05T19:28:29.635 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.630+0000 7f1d95c98700 1 -- 172.21.15.167:0/1453987703 <== osd.0 v2:172.21.15.118:6800/27690 4 ==== osd_op_reply(16 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f1d9000a5a0 con 0x7f1d8000eb60 2023-10-05T19:28:29.636 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.630+0000 7f1d95c98700 1 -- 172.21.15.167:0/1453987703 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:17 5.1a 5:58000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x55fc75248df0 con 0x55fc752118c0 2023-10-05T19:28:29.636 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.630+0000 7f1d95c98700 1 -- 172.21.15.167:0/1453987703 <== osd.1 v2:172.21.15.167:6800/30035 9 ==== osd_op_reply(17 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f1d9000a5a0 con 0x55fc752118c0 2023-10-05T19:28:29.636 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.630+0000 7f1d95c98700 1 -- 172.21.15.167:0/1453987703 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:18 5.6 5:60000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x55fc75248df0 con 0x7f1d88007030 2023-10-05T19:28:29.636 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.630+0000 7f1d8cc96700 1 -- 172.21.15.167:0/1453987703 <== osd.2 v2:172.21.15.167:6816/30037 3 ==== osd_op_reply(18 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f1d80006070 con 0x7f1d88007030 2023-10-05T19:28:29.637 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.630+0000 7f1d8cc96700 1 -- 172.21.15.167:0/1453987703 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:19 5.16 5:68000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x55fc75248df0 con 0x55fc752464d0 2023-10-05T19:28:29.638 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.630+0000 7f1d95497700 1 -- 172.21.15.167:0/1453987703 <== osd.3 v2:172.21.15.167:6808/30036 3 ==== osd_op_reply(19 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f1d8801c7f0 con 0x55fc752464d0 2023-10-05T19:28:29.638 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.630+0000 7f1d95497700 1 -- 172.21.15.167:0/1453987703 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:20 5.e 5:70000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x55fc75248df0 con 0x55fc752464d0 2023-10-05T19:28:29.638 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.634+0000 7f1d95497700 1 -- 172.21.15.167:0/1453987703 <== osd.3 v2:172.21.15.167:6808/30036 4 ==== osd_op_reply(20 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f1d8801c7f0 con 0x55fc752464d0 2023-10-05T19:28:29.638 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.634+0000 7f1d95497700 1 -- 172.21.15.167:0/1453987703 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:21 5.1e 5:78000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x55fc75248df0 con 0x7f1d8000eb60 2023-10-05T19:28:29.639 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.634+0000 7f1d95c98700 1 -- 172.21.15.167:0/1453987703 <== osd.0 v2:172.21.15.118:6800/27690 5 ==== osd_op_reply(21 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f1d9000a5a0 con 0x7f1d8000eb60 2023-10-05T19:28:29.639 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.634+0000 7f1d95c98700 1 -- 172.21.15.167:0/1453987703 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:22 5.1 5:80000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x55fc75248df0 con 0x55fc752464d0 2023-10-05T19:28:29.640 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.634+0000 7f1d95497700 1 -- 172.21.15.167:0/1453987703 <== osd.3 v2:172.21.15.167:6808/30036 5 ==== osd_op_reply(22 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f1d8801c7f0 con 0x55fc752464d0 2023-10-05T19:28:29.640 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.634+0000 7f1d95497700 1 -- 172.21.15.167:0/1453987703 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:23 5.11 5:88000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x55fc75248df0 con 0x55fc752464d0 2023-10-05T19:28:29.641 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.634+0000 7f1d95497700 1 -- 172.21.15.167:0/1453987703 <== osd.3 v2:172.21.15.167:6808/30036 6 ==== osd_op_reply(23 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f1d8801c7f0 con 0x55fc752464d0 2023-10-05T19:28:29.641 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.634+0000 7f1d95497700 1 -- 172.21.15.167:0/1453987703 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:24 5.9 5:90000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x55fc75248df0 con 0x55fc752118c0 2023-10-05T19:28:29.642 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.634+0000 7f1d95c98700 1 -- 172.21.15.167:0/1453987703 <== osd.1 v2:172.21.15.167:6800/30035 10 ==== osd_op_reply(24 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f1d9000a5a0 con 0x55fc752118c0 2023-10-05T19:28:29.642 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.634+0000 7f1d95c98700 1 -- 172.21.15.167:0/1453987703 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:25 5.19 5:98000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x55fc75248df0 con 0x55fc752118c0 2023-10-05T19:28:29.642 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.638+0000 7f1d95c98700 1 -- 172.21.15.167:0/1453987703 <== osd.1 v2:172.21.15.167:6800/30035 11 ==== osd_op_reply(25 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f1d9000a5a0 con 0x55fc752118c0 2023-10-05T19:28:29.642 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.638+0000 7f1d95c98700 1 -- 172.21.15.167:0/1453987703 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:26 5.5 5:a0000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x55fc75248df0 con 0x7f1d8000eb60 2023-10-05T19:28:29.643 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.638+0000 7f1d95c98700 1 -- 172.21.15.167:0/1453987703 <== osd.0 v2:172.21.15.118:6800/27690 6 ==== osd_op_reply(26 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f1d9000a5a0 con 0x7f1d8000eb60 2023-10-05T19:28:29.643 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.638+0000 7f1d95c98700 1 -- 172.21.15.167:0/1453987703 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:27 5.15 5:a8000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x55fc75248df0 con 0x55fc752464d0 2023-10-05T19:28:29.644 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.638+0000 7f1d95497700 1 -- 172.21.15.167:0/1453987703 <== osd.3 v2:172.21.15.167:6808/30036 7 ==== osd_op_reply(27 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f1d8801c7f0 con 0x55fc752464d0 2023-10-05T19:28:29.644 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.638+0000 7f1d95497700 1 -- 172.21.15.167:0/1453987703 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:28 5.d 5:b0000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x55fc75248df0 con 0x7f1d88007030 2023-10-05T19:28:29.645 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.638+0000 7f1d8cc96700 1 -- 172.21.15.167:0/1453987703 <== osd.2 v2:172.21.15.167:6816/30037 4 ==== osd_op_reply(28 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f1d80006070 con 0x7f1d88007030 2023-10-05T19:28:29.645 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.638+0000 7f1d8cc96700 1 -- 172.21.15.167:0/1453987703 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:29 5.1d 5:b8000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x55fc75248df0 con 0x55fc752118c0 2023-10-05T19:28:29.645 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.638+0000 7f1d95c98700 1 -- 172.21.15.167:0/1453987703 <== osd.1 v2:172.21.15.167:6800/30035 12 ==== osd_op_reply(29 [pgnls start_epoch 30 out=107b] v30'2 uv2 ondisk = 1) v8 ==== 144+0+107 (crc 0 0 0) 0x7f1d9000a5a0 con 0x55fc752118c0 2023-10-05T19:28:29.645 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.638+0000 7f1d95c98700 1 -- 172.21.15.167:0/1453987703 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:30 5.3 5:c0000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x55fc75248df0 con 0x7f1d8000eb60 2023-10-05T19:28:29.646 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.642+0000 7f1d95c98700 1 -- 172.21.15.167:0/1453987703 <== osd.0 v2:172.21.15.118:6800/27690 7 ==== osd_op_reply(30 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f1d9000a5a0 con 0x7f1d8000eb60 2023-10-05T19:28:29.646 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.642+0000 7f1d95c98700 1 -- 172.21.15.167:0/1453987703 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:31 5.13 5:c8000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x55fc75248df0 con 0x55fc752464d0 2023-10-05T19:28:29.647 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.642+0000 7f1d95497700 1 -- 172.21.15.167:0/1453987703 <== osd.3 v2:172.21.15.167:6808/30036 8 ==== osd_op_reply(31 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f1d8801c7f0 con 0x55fc752464d0 2023-10-05T19:28:29.647 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.642+0000 7f1d95497700 1 -- 172.21.15.167:0/1453987703 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:32 5.b 5:d0000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x55fc75248df0 con 0x7f1d8000eb60 2023-10-05T19:28:29.648 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.642+0000 7f1d95c98700 1 -- 172.21.15.167:0/1453987703 <== osd.0 v2:172.21.15.118:6800/27690 8 ==== osd_op_reply(32 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f1d9000a5a0 con 0x7f1d8000eb60 2023-10-05T19:28:29.648 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.642+0000 7f1d95c98700 1 -- 172.21.15.167:0/1453987703 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:33 5.1b 5:d8000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x55fc75248df0 con 0x7f1d88007030 2023-10-05T19:28:29.649 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.642+0000 7f1d8cc96700 1 -- 172.21.15.167:0/1453987703 <== osd.2 v2:172.21.15.167:6816/30037 5 ==== osd_op_reply(33 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f1d80006070 con 0x7f1d88007030 2023-10-05T19:28:29.649 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.642+0000 7f1d8cc96700 1 -- 172.21.15.167:0/1453987703 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:34 5.7 5:e0000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x55fc75248df0 con 0x55fc752464d0 2023-10-05T19:28:29.649 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.642+0000 7f1d95497700 1 -- 172.21.15.167:0/1453987703 <== osd.3 v2:172.21.15.167:6808/30036 9 ==== osd_op_reply(34 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f1d8801c7f0 con 0x55fc752464d0 2023-10-05T19:28:29.650 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.642+0000 7f1d95497700 1 -- 172.21.15.167:0/1453987703 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:35 5.17 5:e8000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x55fc75248df0 con 0x7f1d8000eb60 2023-10-05T19:28:29.651 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.646+0000 7f1d95c98700 1 -- 172.21.15.167:0/1453987703 <== osd.0 v2:172.21.15.118:6800/27690 9 ==== osd_op_reply(35 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f1d9000a5a0 con 0x7f1d8000eb60 2023-10-05T19:28:29.651 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.646+0000 7f1d95c98700 1 -- 172.21.15.167:0/1453987703 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:36 5.f 5:f0000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x55fc75248df0 con 0x55fc752118c0 2023-10-05T19:28:29.652 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.646+0000 7f1d95c98700 1 -- 172.21.15.167:0/1453987703 <== osd.1 v2:172.21.15.167:6800/30035 13 ==== osd_op_reply(36 [pgnls start_epoch 30 out=115b] v23'2 uv2 ondisk = 1) v8 ==== 144+0+115 (crc 0 0 0) 0x7f1d9000a5a0 con 0x55fc752118c0 2023-10-05T19:28:29.652 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.646+0000 7f1d95c98700 1 -- 172.21.15.167:0/1453987703 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:37 5.1f 5:f8000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x55fc75248df0 con 0x7f1d88007030 2023-10-05T19:28:29.653 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.646+0000 7f1d8cc96700 1 -- 172.21.15.167:0/1453987703 <== osd.2 v2:172.21.15.167:6816/30037 6 ==== osd_op_reply(37 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f1d80006070 con 0x7f1d88007030 2023-10-05T19:28:29.653 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.646+0000 7f1d97962e40 20 RGWRados::pool_iterate: got zonegroup_info.4f1f719d-9648-417b-b10c-e5d939825419 2023-10-05T19:28:29.653 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.646+0000 7f1d97962e40 20 RGWRados::pool_iterate: got zone_info.9a9df965-6612-45d4-a45f-6e3c889bacc2 2023-10-05T19:28:29.653 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.646+0000 7f1d97962e40 20 RGWRados::pool_iterate: got zone_names.default 2023-10-05T19:28:29.653 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.646+0000 7f1d97962e40 20 RGWRados::pool_iterate: got zonegroups_names.default 2023-10-05T19:28:29.653 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.646+0000 7f1d97962e40 20 rados->read ofs=0 len=0 2023-10-05T19:28:29.653 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.646+0000 7f1d97962e40 1 -- 172.21.15.167:0/1453987703 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:38 5.12 5:49953fa1:::default.realm:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x55fc75245280 con 0x55fc752118c0 2023-10-05T19:28:29.654 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.646+0000 7f1d95c98700 1 -- 172.21.15.167:0/1453987703 <== osd.1 v2:172.21.15.167:6800/30035 14 ==== osd_op_reply(38 default.realm [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 ==== 157+0+0 (crc 0 0 0) 0x7f1d9000a5a0 con 0x55fc752118c0 2023-10-05T19:28:29.654 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.646+0000 7f1d97962e40 20 rados_obj.operate() r=-2 bl.length=0 2023-10-05T19:28:29.654 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.646+0000 7f1d97962e40 20 rados->read ofs=0 len=0 2023-10-05T19:28:29.654 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.646+0000 7f1d97962e40 1 -- 172.21.15.167:0/1453987703 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:39 5.f 5:f4c53578:::zonegroups_names.default:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x55fc75245b70 con 0x55fc752118c0 2023-10-05T19:28:29.654 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.650+0000 7f1d95c98700 1 -- 172.21.15.167:0/1453987703 <== osd.1 v2:172.21.15.167:6800/30035 15 ==== osd_op_reply(39 zonegroups_names.default [read 0~46 out=46b] v0'0 uv2 ondisk = 0) v8 ==== 168+0+46 (crc 0 0 0) 0x7f1d9000a5a0 con 0x55fc752118c0 2023-10-05T19:28:29.655 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.650+0000 7f1d97962e40 20 rados_obj.operate() r=0 bl.length=46 2023-10-05T19:28:29.655 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.650+0000 7f1d97962e40 20 rados->read ofs=0 len=0 2023-10-05T19:28:29.655 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.650+0000 7f1d97962e40 1 -- 172.21.15.167:0/1453987703 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:40 5.c 5:30d299a5:::zonegroup_info.4f1f719d-9648-417b-b10c-e5d939825419:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x55fc75245b70 con 0x7f1d8000eb60 2023-10-05T19:28:29.655 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.650+0000 7f1d95c98700 1 -- 172.21.15.167:0/1453987703 <== osd.0 v2:172.21.15.118:6800/27690 10 ==== osd_op_reply(40 zonegroup_info.4f1f719d-9648-417b-b10c-e5d939825419 [read 0~362 out=362b] v0'0 uv1 ondisk = 0) v8 ==== 195+0+362 (crc 0 0 0) 0x7f1d9000a5a0 con 0x7f1d8000eb60 2023-10-05T19:28:29.655 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.650+0000 7f1d97962e40 20 rados_obj.operate() r=0 bl.length=362 2023-10-05T19:28:29.656 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.650+0000 7f1d97962e40 20 rados->read ofs=0 len=0 2023-10-05T19:28:29.656 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.650+0000 7f1d97962e40 1 -- 172.21.15.167:0/1453987703 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:41 5.13 5:cbbf6c60:::converted:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x55fc75245b70 con 0x55fc752464d0 2023-10-05T19:28:29.656 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.650+0000 7f1d95497700 1 -- 172.21.15.167:0/1453987703 <== osd.3 v2:172.21.15.167:6808/30036 10 ==== osd_op_reply(41 converted [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 ==== 153+0+0 (crc 0 0 0) 0x7f1d8801c7f0 con 0x55fc752464d0 2023-10-05T19:28:29.656 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.650+0000 7f1d97962e40 20 rados_obj.operate() r=-2 bl.length=0 2023-10-05T19:28:29.656 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.650+0000 7f1d97962e40 20 rados->read ofs=0 len=0 2023-10-05T19:28:29.656 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.650+0000 7f1d97962e40 1 -- 172.21.15.167:0/1453987703 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:42 5.12 5:49953fa1:::default.realm:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x55fc75249260 con 0x55fc752118c0 2023-10-05T19:28:29.657 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.650+0000 7f1d95c98700 1 -- 172.21.15.167:0/1453987703 <== osd.1 v2:172.21.15.167:6800/30035 16 ==== osd_op_reply(42 default.realm [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 ==== 157+0+0 (crc 0 0 0) 0x7f1d9000a5a0 con 0x55fc752118c0 2023-10-05T19:28:29.657 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.650+0000 7f1d97962e40 20 rados_obj.operate() r=-2 bl.length=0 2023-10-05T19:28:29.657 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.650+0000 7f1d97962e40 20 rados->read ofs=0 len=0 2023-10-05T19:28:29.657 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.650+0000 7f1d97962e40 1 -- 172.21.15.167:0/1453987703 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:43 5.f 5:f4c53578:::zonegroups_names.default:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x55fc75249690 con 0x55fc752118c0 2023-10-05T19:28:29.658 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.650+0000 7f1d95c98700 1 -- 172.21.15.167:0/1453987703 <== osd.1 v2:172.21.15.167:6800/30035 17 ==== osd_op_reply(43 zonegroups_names.default [read 0~46 out=46b] v0'0 uv2 ondisk = 0) v8 ==== 168+0+46 (crc 0 0 0) 0x7f1d9000a5a0 con 0x55fc752118c0 2023-10-05T19:28:29.658 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.650+0000 7f1d97962e40 20 rados_obj.operate() r=0 bl.length=46 2023-10-05T19:28:29.658 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.650+0000 7f1d97962e40 1 -- 172.21.15.167:0/1453987703 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:44 5.0 5:00000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x55fc7524a6a0 con 0x55fc752464d0 2023-10-05T19:28:29.659 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.654+0000 7f1d95497700 1 -- 172.21.15.167:0/1453987703 <== osd.3 v2:172.21.15.167:6808/30036 11 ==== osd_op_reply(44 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f1d8801c7f0 con 0x55fc752464d0 2023-10-05T19:28:29.659 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.654+0000 7f1d95497700 1 -- 172.21.15.167:0/1453987703 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:45 5.10 5:08000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x55fc7524a6a0 con 0x7f1d88007030 2023-10-05T19:28:29.660 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.654+0000 7f1d8cc96700 1 -- 172.21.15.167:0/1453987703 <== osd.2 v2:172.21.15.167:6816/30037 7 ==== osd_op_reply(45 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f1d80006070 con 0x7f1d88007030 2023-10-05T19:28:29.660 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.654+0000 7f1d8cc96700 1 -- 172.21.15.167:0/1453987703 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:46 5.8 5:10000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x55fc7524a6a0 con 0x7f1d8000eb60 2023-10-05T19:28:29.661 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.654+0000 7f1d95c98700 1 -- 172.21.15.167:0/1453987703 <== osd.0 v2:172.21.15.118:6800/27690 11 ==== osd_op_reply(46 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f1d9000a5a0 con 0x7f1d8000eb60 2023-10-05T19:28:29.661 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.654+0000 7f1d95c98700 1 -- 172.21.15.167:0/1453987703 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:47 5.18 5:18000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x55fc75245280 con 0x55fc752118c0 2023-10-05T19:28:29.662 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.658+0000 7f1d95c98700 1 -- 172.21.15.167:0/1453987703 <== osd.1 v2:172.21.15.167:6800/30035 18 ==== osd_op_reply(47 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f1d9000a5a0 con 0x55fc752118c0 2023-10-05T19:28:29.662 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.658+0000 7f1d95c98700 1 -- 172.21.15.167:0/1453987703 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:48 5.4 5:20000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x55fc75245280 con 0x55fc752118c0 2023-10-05T19:28:29.663 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.658+0000 7f1d95c98700 1 -- 172.21.15.167:0/1453987703 <== osd.1 v2:172.21.15.167:6800/30035 19 ==== osd_op_reply(48 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f1d9000a5a0 con 0x55fc752118c0 2023-10-05T19:28:29.663 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.658+0000 7f1d95c98700 1 -- 172.21.15.167:0/1453987703 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:49 5.14 5:28000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x55fc75245280 con 0x7f1d8000eb60 2023-10-05T19:28:29.664 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.658+0000 7f1d95c98700 1 -- 172.21.15.167:0/1453987703 <== osd.0 v2:172.21.15.118:6800/27690 12 ==== osd_op_reply(49 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f1d9000a5a0 con 0x7f1d8000eb60 2023-10-05T19:28:29.664 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.658+0000 7f1d95c98700 1 -- 172.21.15.167:0/1453987703 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:50 5.c 5:30000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x55fc75245280 con 0x7f1d8000eb60 2023-10-05T19:28:29.665 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.658+0000 7f1d95c98700 1 -- 172.21.15.167:0/1453987703 <== osd.0 v2:172.21.15.118:6800/27690 13 ==== osd_op_reply(50 [pgnls start_epoch 30 out=112b] v23'1 uv1 ondisk = 1) v8 ==== 144+0+112 (crc 0 0 0) 0x7f1d9000a5a0 con 0x7f1d8000eb60 2023-10-05T19:28:29.665 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.658+0000 7f1d95c98700 1 -- 172.21.15.167:0/1453987703 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:51 5.1c 5:38000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x55fc75245280 con 0x7f1d88007030 2023-10-05T19:28:29.666 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.662+0000 7f1d8cc96700 1 -- 172.21.15.167:0/1453987703 <== osd.2 v2:172.21.15.167:6816/30037 8 ==== osd_op_reply(51 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f1d80006070 con 0x7f1d88007030 2023-10-05T19:28:29.666 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.662+0000 7f1d8cc96700 1 -- 172.21.15.167:0/1453987703 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:52 5.2 5:40000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x55fc75245280 con 0x55fc752464d0 2023-10-05T19:28:29.666 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.662+0000 7f1d95497700 1 -- 172.21.15.167:0/1453987703 <== osd.3 v2:172.21.15.167:6808/30036 12 ==== osd_op_reply(52 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f1d8801c7f0 con 0x55fc752464d0 2023-10-05T19:28:29.667 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.662+0000 7f1d95497700 1 -- 172.21.15.167:0/1453987703 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:53 5.12 5:48000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x55fc75245280 con 0x55fc752118c0 2023-10-05T19:28:29.667 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.662+0000 7f1d95c98700 1 -- 172.21.15.167:0/1453987703 <== osd.1 v2:172.21.15.167:6800/30035 20 ==== osd_op_reply(53 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f1d9000a5a0 con 0x55fc752118c0 2023-10-05T19:28:29.668 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.662+0000 7f1d95c98700 1 -- 172.21.15.167:0/1453987703 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:54 5.a 5:50000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x55fc75245280 con 0x7f1d8000eb60 2023-10-05T19:28:29.668 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.662+0000 7f1d95c98700 1 -- 172.21.15.167:0/1453987703 <== osd.0 v2:172.21.15.118:6800/27690 14 ==== osd_op_reply(54 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f1d9000a5a0 con 0x7f1d8000eb60 2023-10-05T19:28:29.668 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.662+0000 7f1d95c98700 1 -- 172.21.15.167:0/1453987703 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:55 5.1a 5:58000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x55fc75245280 con 0x55fc752118c0 2023-10-05T19:28:29.669 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.662+0000 7f1d95c98700 1 -- 172.21.15.167:0/1453987703 <== osd.1 v2:172.21.15.167:6800/30035 21 ==== osd_op_reply(55 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f1d9000a5a0 con 0x55fc752118c0 2023-10-05T19:28:29.669 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.662+0000 7f1d95c98700 1 -- 172.21.15.167:0/1453987703 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:56 5.6 5:60000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x55fc75245280 con 0x7f1d88007030 2023-10-05T19:28:29.670 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.666+0000 7f1d8cc96700 1 -- 172.21.15.167:0/1453987703 <== osd.2 v2:172.21.15.167:6816/30037 9 ==== osd_op_reply(56 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f1d80006070 con 0x7f1d88007030 2023-10-05T19:28:29.670 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.666+0000 7f1d8cc96700 1 -- 172.21.15.167:0/1453987703 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:57 5.16 5:68000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x55fc75245280 con 0x55fc752464d0 2023-10-05T19:28:29.670 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.666+0000 7f1d95497700 1 -- 172.21.15.167:0/1453987703 <== osd.3 v2:172.21.15.167:6808/30036 13 ==== osd_op_reply(57 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f1d8801c7f0 con 0x55fc752464d0 2023-10-05T19:28:29.671 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.666+0000 7f1d95497700 1 -- 172.21.15.167:0/1453987703 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:58 5.e 5:70000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x55fc75245280 con 0x55fc752464d0 2023-10-05T19:28:29.671 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.666+0000 7f1d95497700 1 -- 172.21.15.167:0/1453987703 <== osd.3 v2:172.21.15.167:6808/30036 14 ==== osd_op_reply(58 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f1d8801c7f0 con 0x55fc752464d0 2023-10-05T19:28:29.671 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.666+0000 7f1d95497700 1 -- 172.21.15.167:0/1453987703 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:59 5.1e 5:78000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x55fc75245280 con 0x7f1d8000eb60 2023-10-05T19:28:29.672 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.666+0000 7f1d95c98700 1 -- 172.21.15.167:0/1453987703 <== osd.0 v2:172.21.15.118:6800/27690 15 ==== osd_op_reply(59 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f1d9003f040 con 0x7f1d8000eb60 2023-10-05T19:28:29.672 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.666+0000 7f1d95c98700 1 -- 172.21.15.167:0/1453987703 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:60 5.1 5:80000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x55fc75245280 con 0x55fc752464d0 2023-10-05T19:28:29.673 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.666+0000 7f1d95497700 1 -- 172.21.15.167:0/1453987703 <== osd.3 v2:172.21.15.167:6808/30036 15 ==== osd_op_reply(60 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f1d8801c7f0 con 0x55fc752464d0 2023-10-05T19:28:29.673 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.666+0000 7f1d95497700 1 -- 172.21.15.167:0/1453987703 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:61 5.11 5:88000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x55fc75245280 con 0x55fc752464d0 2023-10-05T19:28:29.673 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.666+0000 7f1d95497700 1 -- 172.21.15.167:0/1453987703 <== osd.3 v2:172.21.15.167:6808/30036 16 ==== osd_op_reply(61 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f1d8801c7f0 con 0x55fc752464d0 2023-10-05T19:28:29.673 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.666+0000 7f1d95497700 1 -- 172.21.15.167:0/1453987703 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:62 5.9 5:90000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x55fc75245280 con 0x55fc752118c0 2023-10-05T19:28:29.674 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.670+0000 7f1d95c98700 1 -- 172.21.15.167:0/1453987703 <== osd.1 v2:172.21.15.167:6800/30035 22 ==== osd_op_reply(62 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f1d9003f040 con 0x55fc752118c0 2023-10-05T19:28:29.674 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.670+0000 7f1d95c98700 1 -- 172.21.15.167:0/1453987703 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:63 5.19 5:98000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x55fc75245280 con 0x55fc752118c0 2023-10-05T19:28:29.675 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.670+0000 7f1d95c98700 1 -- 172.21.15.167:0/1453987703 <== osd.1 v2:172.21.15.167:6800/30035 23 ==== osd_op_reply(63 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f1d9003f040 con 0x55fc752118c0 2023-10-05T19:28:29.675 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.670+0000 7f1d95c98700 1 -- 172.21.15.167:0/1453987703 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:64 5.5 5:a0000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x55fc75245280 con 0x7f1d8000eb60 2023-10-05T19:28:29.676 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.670+0000 7f1d95c98700 1 -- 172.21.15.167:0/1453987703 <== osd.0 v2:172.21.15.118:6800/27690 16 ==== osd_op_reply(64 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f1d9003f040 con 0x7f1d8000eb60 2023-10-05T19:28:29.676 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.670+0000 7f1d95c98700 1 -- 172.21.15.167:0/1453987703 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:65 5.15 5:a8000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x55fc75245280 con 0x55fc752464d0 2023-10-05T19:28:29.677 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.670+0000 7f1d95497700 1 -- 172.21.15.167:0/1453987703 <== osd.3 v2:172.21.15.167:6808/30036 17 ==== osd_op_reply(65 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f1d8801c7f0 con 0x55fc752464d0 2023-10-05T19:28:29.677 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.670+0000 7f1d95497700 1 -- 172.21.15.167:0/1453987703 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:66 5.d 5:b0000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x55fc75245280 con 0x7f1d88007030 2023-10-05T19:28:29.677 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.670+0000 7f1d8cc96700 1 -- 172.21.15.167:0/1453987703 <== osd.2 v2:172.21.15.167:6816/30037 10 ==== osd_op_reply(66 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f1d80006070 con 0x7f1d88007030 2023-10-05T19:28:29.678 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.670+0000 7f1d8cc96700 1 -- 172.21.15.167:0/1453987703 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:67 5.1d 5:b8000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x55fc75245280 con 0x55fc752118c0 2023-10-05T19:28:29.678 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.674+0000 7f1d95c98700 1 -- 172.21.15.167:0/1453987703 <== osd.1 v2:172.21.15.167:6800/30035 24 ==== osd_op_reply(67 [pgnls start_epoch 30 out=107b] v30'2 uv2 ondisk = 1) v8 ==== 144+0+107 (crc 0 0 0) 0x7f1d9003f040 con 0x55fc752118c0 2023-10-05T19:28:29.678 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.674+0000 7f1d95c98700 1 -- 172.21.15.167:0/1453987703 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:68 5.3 5:c0000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x55fc75245280 con 0x7f1d8000eb60 2023-10-05T19:28:29.679 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.674+0000 7f1d95c98700 1 -- 172.21.15.167:0/1453987703 <== osd.0 v2:172.21.15.118:6800/27690 17 ==== osd_op_reply(68 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f1d9003f040 con 0x7f1d8000eb60 2023-10-05T19:28:29.679 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.674+0000 7f1d95c98700 1 -- 172.21.15.167:0/1453987703 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:69 5.13 5:c8000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x55fc75245280 con 0x55fc752464d0 2023-10-05T19:28:29.680 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.674+0000 7f1d95497700 1 -- 172.21.15.167:0/1453987703 <== osd.3 v2:172.21.15.167:6808/30036 18 ==== osd_op_reply(69 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f1d8801c7f0 con 0x55fc752464d0 2023-10-05T19:28:29.680 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.674+0000 7f1d95497700 1 -- 172.21.15.167:0/1453987703 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:70 5.b 5:d0000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x55fc75245280 con 0x7f1d8000eb60 2023-10-05T19:28:29.681 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.674+0000 7f1d95c98700 1 -- 172.21.15.167:0/1453987703 <== osd.0 v2:172.21.15.118:6800/27690 18 ==== osd_op_reply(70 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f1d9003f040 con 0x7f1d8000eb60 2023-10-05T19:28:29.681 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.674+0000 7f1d95c98700 1 -- 172.21.15.167:0/1453987703 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:71 5.1b 5:d8000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x55fc75245280 con 0x7f1d88007030 2023-10-05T19:28:29.682 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.674+0000 7f1d8cc96700 1 -- 172.21.15.167:0/1453987703 <== osd.2 v2:172.21.15.167:6816/30037 11 ==== osd_op_reply(71 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f1d80006070 con 0x7f1d88007030 2023-10-05T19:28:29.682 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.674+0000 7f1d8cc96700 1 -- 172.21.15.167:0/1453987703 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:72 5.7 5:e0000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x55fc75245280 con 0x55fc752464d0 2023-10-05T19:28:29.682 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.678+0000 7f1d95497700 1 -- 172.21.15.167:0/1453987703 <== osd.3 v2:172.21.15.167:6808/30036 19 ==== osd_op_reply(72 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f1d8801c7f0 con 0x55fc752464d0 2023-10-05T19:28:29.682 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.678+0000 7f1d95497700 1 -- 172.21.15.167:0/1453987703 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:73 5.17 5:e8000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x55fc75245280 con 0x7f1d8000eb60 2023-10-05T19:28:29.683 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.678+0000 7f1d95c98700 1 -- 172.21.15.167:0/1453987703 <== osd.0 v2:172.21.15.118:6800/27690 19 ==== osd_op_reply(73 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f1d9003f040 con 0x7f1d8000eb60 2023-10-05T19:28:29.683 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.678+0000 7f1d95c98700 1 -- 172.21.15.167:0/1453987703 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:74 5.f 5:f0000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x55fc75245280 con 0x55fc752118c0 2023-10-05T19:28:29.684 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.678+0000 7f1d95c98700 1 -- 172.21.15.167:0/1453987703 <== osd.1 v2:172.21.15.167:6800/30035 25 ==== osd_op_reply(74 [pgnls start_epoch 30 out=115b] v23'2 uv2 ondisk = 1) v8 ==== 144+0+115 (crc 0 0 0) 0x7f1d9003f040 con 0x55fc752118c0 2023-10-05T19:28:29.684 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.678+0000 7f1d95c98700 1 -- 172.21.15.167:0/1453987703 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:75 5.1f 5:f8000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x55fc75245280 con 0x7f1d88007030 2023-10-05T19:28:29.685 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.678+0000 7f1d8cc96700 1 -- 172.21.15.167:0/1453987703 <== osd.2 v2:172.21.15.167:6816/30037 12 ==== osd_op_reply(75 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f1d80006070 con 0x7f1d88007030 2023-10-05T19:28:29.685 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.678+0000 7f1d97962e40 20 RGWRados::pool_iterate: got zonegroup_info.4f1f719d-9648-417b-b10c-e5d939825419 2023-10-05T19:28:29.685 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.678+0000 7f1d97962e40 20 RGWRados::pool_iterate: got zone_info.9a9df965-6612-45d4-a45f-6e3c889bacc2 2023-10-05T19:28:29.686 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.678+0000 7f1d97962e40 20 RGWRados::pool_iterate: got zone_names.default 2023-10-05T19:28:29.686 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.678+0000 7f1d97962e40 20 RGWRados::pool_iterate: got zonegroups_names.default 2023-10-05T19:28:29.686 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.678+0000 7f1d97962e40 20 rados->read ofs=0 len=0 2023-10-05T19:28:29.686 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.678+0000 7f1d97962e40 1 -- 172.21.15.167:0/1453987703 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:76 5.f 5:f43b6ece:::zone_names.default:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x55fc75249690 con 0x55fc752118c0 2023-10-05T19:28:29.687 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.682+0000 7f1d95c98700 1 -- 172.21.15.167:0/1453987703 <== osd.1 v2:172.21.15.167:6800/30035 26 ==== osd_op_reply(76 zone_names.default [read 0~46 out=46b] v0'0 uv1 ondisk = 0) v8 ==== 162+0+46 (crc 0 0 0) 0x7f1d9003f040 con 0x55fc752118c0 2023-10-05T19:28:29.687 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.682+0000 7f1d97962e40 20 rados_obj.operate() r=0 bl.length=46 2023-10-05T19:28:29.687 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.682+0000 7f1d97962e40 20 rados->read ofs=0 len=0 2023-10-05T19:28:29.687 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.682+0000 7f1d97962e40 1 -- 172.21.15.167:0/1453987703 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:77 5.1d 5:b8374eb4:::zone_info.9a9df965-6612-45d4-a45f-6e3c889bacc2:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x55fc75249a40 con 0x55fc752118c0 2023-10-05T19:28:29.687 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.682+0000 7f1d95c98700 1 -- 172.21.15.167:0/1453987703 <== osd.1 v2:172.21.15.167:6800/30035 27 ==== osd_op_reply(77 zone_info.9a9df965-6612-45d4-a45f-6e3c889bacc2 [read 0~874 out=874b] v0'0 uv2 ondisk = 0) v8 ==== 190+0+874 (crc 0 0 0) 0x7f1d9003f040 con 0x55fc752118c0 2023-10-05T19:28:29.687 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.682+0000 7f1d97962e40 20 rados_obj.operate() r=0 bl.length=874 2023-10-05T19:28:29.687 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.682+0000 7f1d97962e40 20 rados->read ofs=0 len=0 2023-10-05T19:28:29.687 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.682+0000 7f1d97962e40 1 -- 172.21.15.167:0/1453987703 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:78 5.f 5:f4c53578:::zonegroups_names.default:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x55fc75245710 con 0x55fc752118c0 2023-10-05T19:28:29.688 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.682+0000 7f1d95c98700 1 -- 172.21.15.167:0/1453987703 <== osd.1 v2:172.21.15.167:6800/30035 28 ==== osd_op_reply(78 zonegroups_names.default [read 0~46 out=46b] v0'0 uv2 ondisk = 0) v8 ==== 168+0+46 (crc 0 0 0) 0x7f1d9003f040 con 0x55fc752118c0 2023-10-05T19:28:29.688 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.682+0000 7f1d97962e40 20 rados_obj.operate() r=0 bl.length=46 2023-10-05T19:28:29.688 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.682+0000 7f1d97962e40 20 rados->read ofs=0 len=0 2023-10-05T19:28:29.688 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.682+0000 7f1d97962e40 1 -- 172.21.15.167:0/1453987703 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:79 5.c 5:30d299a5:::zonegroup_info.4f1f719d-9648-417b-b10c-e5d939825419:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x55fc75245730 con 0x7f1d8000eb60 2023-10-05T19:28:29.689 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.682+0000 7f1d95c98700 1 -- 172.21.15.167:0/1453987703 <== osd.0 v2:172.21.15.118:6800/27690 20 ==== osd_op_reply(79 zonegroup_info.4f1f719d-9648-417b-b10c-e5d939825419 [read 0~362 out=362b] v0'0 uv1 ondisk = 0) v8 ==== 195+0+362 (crc 0 0 0) 0x7f1d9003f040 con 0x7f1d8000eb60 2023-10-05T19:28:29.689 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.682+0000 7f1d97962e40 20 rados_obj.operate() r=0 bl.length=362 2023-10-05T19:28:29.689 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.682+0000 7f1d97962e40 20 rados->read ofs=0 len=0 2023-10-05T19:28:29.689 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.682+0000 7f1d97962e40 1 -- 172.21.15.167:0/1453987703 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:80 5.3 5:c0ad8cbd:::region_map:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x55fc7524a4b0 con 0x7f1d8000eb60 2023-10-05T19:28:29.690 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.682+0000 7f1d95c98700 1 -- 172.21.15.167:0/1453987703 <== osd.0 v2:172.21.15.118:6800/27690 21 ==== osd_op_reply(80 region_map [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 ==== 154+0+0 (crc 0 0 0) 0x7f1d9003f040 con 0x7f1d8000eb60 2023-10-05T19:28:29.690 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.682+0000 7f1d97962e40 20 rados_obj.operate() r=-2 bl.length=0 2023-10-05T19:28:29.690 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.682+0000 7f1d97962e40 20 zone default found 2023-10-05T19:28:29.690 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.682+0000 7f1d97962e40 4 Realm: () 2023-10-05T19:28:29.690 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.682+0000 7f1d97962e40 4 ZoneGroup: default (4f1f719d-9648-417b-b10c-e5d939825419) 2023-10-05T19:28:29.690 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.682+0000 7f1d97962e40 4 Zone: default (9a9df965-6612-45d4-a45f-6e3c889bacc2) 2023-10-05T19:28:29.690 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.682+0000 7f1d97962e40 10 cannot find current period zonegroup using local zonegroup configuration 2023-10-05T19:28:29.690 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.682+0000 7f1d97962e40 20 zonegroup default 2023-10-05T19:28:29.690 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.682+0000 7f1d97962e40 20 rados->read ofs=0 len=0 2023-10-05T19:28:29.690 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.682+0000 7f1d97962e40 1 -- 172.21.15.167:0/1453987703 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:81 5.3 5:c52100b6:::period_config.default:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x55fc7524a4b0 con 0x7f1d8000eb60 2023-10-05T19:28:29.691 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.686+0000 7f1d95c98700 1 -- 172.21.15.167:0/1453987703 <== osd.0 v2:172.21.15.118:6800/27690 22 ==== osd_op_reply(81 period_config.default [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 ==== 165+0+0 (crc 0 0 0) 0x7f1d9003f040 con 0x7f1d8000eb60 2023-10-05T19:28:29.691 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.686+0000 7f1d97962e40 20 rados_obj.operate() r=-2 bl.length=0 2023-10-05T19:28:29.692 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.686+0000 7f1d97962e40 20 rados->read ofs=0 len=0 2023-10-05T19:28:29.692 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.686+0000 7f1d97962e40 1 -- 172.21.15.167:0/1453987703 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:82 6.2 6:4347d321:::bucket.sync-source-hints.:head [call version.read in=11b,read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x55fc7524bad0 con 0x55fc752464d0 2023-10-05T19:28:29.692 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.686+0000 7f1d95497700 1 -- 172.21.15.167:0/1453987703 <== osd.3 v2:172.21.15.167:6808/30036 20 ==== osd_op_reply(82 bucket.sync-source-hints. [call,read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 ==== 211+0+0 (crc 0 0 0) 0x7f1d8805f040 con 0x55fc752464d0 2023-10-05T19:28:29.692 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.686+0000 7f1d97962e40 20 rados_obj.operate() r=-2 bl.length=0 2023-10-05T19:28:29.692 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.686+0000 7f1d97962e40 20 rados->read ofs=0 len=0 2023-10-05T19:28:29.692 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.686+0000 7f1d97962e40 1 -- 172.21.15.167:0/1453987703 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:83 6.b 6:d467b91b:::bucket.sync-target-hints.:head [call version.read in=11b,read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x55fc7524be80 con 0x55fc752464d0 2023-10-05T19:28:29.693 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.686+0000 7f1d95497700 1 -- 172.21.15.167:0/1453987703 <== osd.3 v2:172.21.15.167:6808/30036 21 ==== osd_op_reply(83 bucket.sync-target-hints. [call,read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 ==== 211+0+0 (crc 0 0 0) 0x7f1d8805f040 con 0x55fc752464d0 2023-10-05T19:28:29.693 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.686+0000 7f1d97962e40 20 rados_obj.operate() r=-2 bl.length=0 2023-10-05T19:28:29.693 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.686+0000 7f1d97962e40 20 started sync module instance, tier type = 2023-10-05T19:28:29.693 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.686+0000 7f1d97962e40 20 started zone id=9a9df965-6612-45d4-a45f-6e3c889bacc2 (name=default) with tier type = 2023-10-05T19:28:29.693 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.686+0000 7f1d97962e40 1 -- 172.21.15.167:0/1453987703 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:84 7.1f 7:f95f44c2:::notify.0:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x55fc7524b190 con 0x7f1d88007030 2023-10-05T19:28:29.695 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.690+0000 7f1d8cc96700 1 -- 172.21.15.167:0/1453987703 <== osd.2 v2:172.21.15.167:6816/30037 13 ==== osd_op_reply(84 notify.0 [create] v30'78 uv78 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f1d80006070 con 0x7f1d88007030 2023-10-05T19:28:29.695 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.690+0000 7f1d97962e40 1 -- 172.21.15.167:0/1453987703 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:85 7.1f 7:f95f44c2:::notify.0:head [watch watch cookie 94542785459520] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x55fc7524adc0 con 0x7f1d88007030 2023-10-05T19:28:29.695 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.690+0000 7f1d97962e40 1 -- 172.21.15.167:0/1453987703 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:86 7.0 7:05bf5b68:::notify.1:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x55fc7524cc10 con 0x55fc752118c0 2023-10-05T19:28:29.698 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.694+0000 7f1d95c98700 1 -- 172.21.15.167:0/1453987703 <== osd.1 v2:172.21.15.167:6800/30035 29 ==== osd_op_reply(86 notify.1 [create] v30'78 uv78 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f1d9003f040 con 0x55fc752118c0 2023-10-05T19:28:29.698 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.694+0000 7f1d8cc96700 1 -- 172.21.15.167:0/1453987703 <== osd.2 v2:172.21.15.167:6816/30037 14 ==== osd_op_reply(85 notify.0 [watch watch cookie 94542785459520] v30'79 uv78 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f1d80006070 con 0x7f1d88007030 2023-10-05T19:28:29.699 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.694+0000 7f1d97962e40 1 -- 172.21.15.167:0/1453987703 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:87 7.0 7:05bf5b68:::notify.1:head [watch watch cookie 94542785463120] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x55fc7524d010 con 0x55fc752118c0 2023-10-05T19:28:29.699 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.694+0000 7f1d97962e40 1 -- 172.21.15.167:0/1453987703 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:88 7.15 7:a93a5511:::notify.2:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x55fc7524d8b0 con 0x55fc752464d0 2023-10-05T19:28:29.701 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.694+0000 7f1d95497700 1 -- 172.21.15.167:0/1453987703 <== osd.3 v2:172.21.15.167:6808/30036 22 ==== osd_op_reply(88 notify.2 [create] v30'78 uv78 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f1d8805f040 con 0x55fc752464d0 2023-10-05T19:28:29.701 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.694+0000 7f1d97962e40 1 -- 172.21.15.167:0/1453987703 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:89 7.15 7:a93a5511:::notify.2:head [watch watch cookie 94542785466672] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x55fc7524e150 con 0x55fc752464d0 2023-10-05T19:28:29.701 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.694+0000 7f1d97962e40 1 -- 172.21.15.167:0/1453987703 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:90 7.e 7:7759931f:::notify.3:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x55fc7524e9f0 con 0x7f1d88007030 2023-10-05T19:28:29.702 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.694+0000 7f1d95c98700 1 -- 172.21.15.167:0/1453987703 <== osd.1 v2:172.21.15.167:6800/30035 30 ==== osd_op_reply(87 notify.1 [watch watch cookie 94542785463120] v30'79 uv78 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f1d9003f040 con 0x55fc752118c0 2023-10-05T19:28:29.704 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.698+0000 7f1d8cc96700 1 -- 172.21.15.167:0/1453987703 <== osd.2 v2:172.21.15.167:6816/30037 15 ==== osd_op_reply(90 notify.3 [create] v30'78 uv78 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f1d80006070 con 0x7f1d88007030 2023-10-05T19:28:29.704 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.698+0000 7f1d97962e40 1 -- 172.21.15.167:0/1453987703 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:91 7.e 7:7759931f:::notify.3:head [watch watch cookie 94542785464224] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x55fc7524e620 con 0x7f1d88007030 2023-10-05T19:28:29.704 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.698+0000 7f1d97962e40 1 -- 172.21.15.167:0/1453987703 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:92 7.d 7:b4812045:::notify.4:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x55fc7524fa10 con 0x55fc752118c0 2023-10-05T19:28:29.704 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.698+0000 7f1d95497700 1 -- 172.21.15.167:0/1453987703 <== osd.3 v2:172.21.15.167:6808/30036 23 ==== osd_op_reply(89 notify.2 [watch watch cookie 94542785466672] v30'79 uv78 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f1d8805f040 con 0x55fc752464d0 2023-10-05T19:28:29.706 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.702+0000 7f1d8cc96700 1 -- 172.21.15.167:0/1453987703 <== osd.2 v2:172.21.15.167:6816/30037 16 ==== osd_op_reply(91 notify.3 [watch watch cookie 94542785464224] v30'79 uv78 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f1d80006070 con 0x7f1d88007030 2023-10-05T19:28:29.706 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.702+0000 7f1d95c98700 1 -- 172.21.15.167:0/1453987703 <== osd.1 v2:172.21.15.167:6800/30035 31 ==== osd_op_reply(92 notify.4 [create] v30'78 uv78 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f1d9003f040 con 0x55fc752118c0 2023-10-05T19:28:29.707 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.702+0000 7f1d97962e40 1 -- 172.21.15.167:0/1453987703 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:93 7.d 7:b4812045:::notify.4:head [watch watch cookie 94542785474032] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x55fc7524f4e0 con 0x55fc752118c0 2023-10-05T19:28:29.707 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.702+0000 7f1d97962e40 1 -- 172.21.15.167:0/1453987703 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:94 7.3 7:c609908c:::notify.5:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x55fc75250300 con 0x7f1d8000eb60 2023-10-05T19:28:29.709 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.702+0000 7f1d95c98700 1 -- 172.21.15.167:0/1453987703 <== osd.0 v2:172.21.15.118:6800/27690 23 ==== osd_op_reply(94 notify.5 [create] v30'78 uv78 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f1d9003f040 con 0x7f1d8000eb60 2023-10-05T19:28:29.709 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.702+0000 7f1d97962e40 1 -- 172.21.15.167:0/1453987703 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:95 7.3 7:c609908c:::notify.5:head [watch watch cookie 94542785478080] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x55fc75250210 con 0x7f1d8000eb60 2023-10-05T19:28:29.709 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.702+0000 7f1d97962e40 1 -- 172.21.15.167:0/1453987703 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:96 7.14 7:2b04a3e9:::notify.6:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x55fc75251250 con 0x7f1d8000eb60 2023-10-05T19:28:29.709 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.702+0000 7f1d95c98700 1 -- 172.21.15.167:0/1453987703 <== osd.1 v2:172.21.15.167:6800/30035 32 ==== osd_op_reply(93 notify.4 [watch watch cookie 94542785474032] v30'79 uv78 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f1d9003f040 con 0x55fc752118c0 2023-10-05T19:28:29.712 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.706+0000 7f1d95c98700 1 -- 172.21.15.167:0/1453987703 <== osd.0 v2:172.21.15.118:6800/27690 24 ==== osd_op_reply(96 notify.6 [create] v30'78 uv78 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f1d9003f040 con 0x7f1d8000eb60 2023-10-05T19:28:29.712 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.706+0000 7f1d97962e40 1 -- 172.21.15.167:0/1453987703 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:97 7.14 7:2b04a3e9:::notify.6:head [watch watch cookie 94542785469536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x55fc75251250 con 0x7f1d8000eb60 2023-10-05T19:28:29.712 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.706+0000 7f1d97962e40 1 -- 172.21.15.167:0/1453987703 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:98 7.9 7:93e5b521:::notify.7:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x55fc75251c60 con 0x7f1d8000eb60 2023-10-05T19:28:29.712 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.706+0000 7f1d95c98700 1 -- 172.21.15.167:0/1453987703 <== osd.0 v2:172.21.15.118:6800/27690 25 ==== osd_op_reply(95 notify.5 [watch watch cookie 94542785478080] v30'79 uv78 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f1d9003f040 con 0x7f1d8000eb60 2023-10-05T19:28:29.714 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.710+0000 7f1d95c98700 1 -- 172.21.15.167:0/1453987703 <== osd.0 v2:172.21.15.118:6800/27690 26 ==== osd_op_reply(98 notify.7 [create] v30'78 uv78 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f1d9003f040 con 0x7f1d8000eb60 2023-10-05T19:28:29.714 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.710+0000 7f1d97962e40 1 -- 172.21.15.167:0/1453987703 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:99 7.9 7:93e5b521:::notify.7:head [watch watch cookie 94542785480336] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x55fc75251c60 con 0x7f1d8000eb60 2023-10-05T19:28:29.714 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.710+0000 7f1d97962e40 20 add_watcher() i=0 2023-10-05T19:28:29.714 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.710+0000 7f1d97962e40 20 add_watcher() i=1 2023-10-05T19:28:29.715 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.710+0000 7f1d97962e40 20 add_watcher() i=2 2023-10-05T19:28:29.715 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.710+0000 7f1d97962e40 20 add_watcher() i=3 2023-10-05T19:28:29.715 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.710+0000 7f1d97962e40 20 add_watcher() i=4 2023-10-05T19:28:29.715 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.710+0000 7f1d97962e40 20 add_watcher() i=5 2023-10-05T19:28:29.715 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.710+0000 7f1d95c98700 1 -- 172.21.15.167:0/1453987703 <== osd.0 v2:172.21.15.118:6800/27690 27 ==== osd_op_reply(97 notify.6 [watch watch cookie 94542785469536] v30'79 uv78 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f1d9003f040 con 0x7f1d8000eb60 2023-10-05T19:28:29.715 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.710+0000 7f1d97962e40 20 add_watcher() i=6 2023-10-05T19:28:29.716 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.710+0000 7f1d95c98700 1 -- 172.21.15.167:0/1453987703 <== osd.0 v2:172.21.15.118:6800/27690 28 ==== osd_op_reply(99 notify.7 [watch watch cookie 94542785480336] v30'79 uv78 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f1d9003f040 con 0x7f1d8000eb60 2023-10-05T19:28:29.716 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.710+0000 7f1d97962e40 20 add_watcher() i=7 2023-10-05T19:28:29.716 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.710+0000 7f1d97962e40 2 all 8 watchers are set, enabling cache 2023-10-05T19:28:29.716 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.710+0000 7f1d97962e40 1 -- 172.21.15.167:0/1453987703 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:100 6.4 6:22d26bf9:::data_loggenerations_metadata:head [call version.check_conds in=74b,call version.read in=11b,read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x55fc75254c90 con 0x55fc752118c0 2023-10-05T19:28:29.717 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.710+0000 7f1d95c98700 1 -- 172.21.15.167:0/1453987703 <== osd.1 v2:172.21.15.167:6800/30035 33 ==== osd_op_reply(100 data_loggenerations_metadata [call,call out=48b,read 0~28 out=28b] v0'0 uv1 ondisk = 0) v8 ==== 256+0+76 (crc 0 0 0) 0x7f1d9003f040 con 0x55fc752118c0 2023-10-05T19:28:29.717 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.710+0000 7f1d97962e40 1 -- 172.21.15.167:0/1453987703 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:101 6.4 6:22d26bf9:::data_loggenerations_metadata:head [watch watch cookie 94542785484656] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x55fc752520b0 con 0x55fc752118c0 2023-10-05T19:28:29.719 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.714+0000 7f1d95c98700 1 -- 172.21.15.167:0/1453987703 <== osd.1 v2:172.21.15.167:6800/30035 34 ==== osd_op_reply(101 data_loggenerations_metadata [watch watch cookie 94542785484656] v30'86 uv1 ondisk = 0) v8 ==== 172+0+0 (crc 0 0 0) 0x7f1d9003f040 con 0x55fc752118c0 2023-10-05T19:28:29.719 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.714+0000 7f1d667e4700 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2023-10-05T19:28:29.719 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.714+0000 7f1d97962e40 20 check_secure_mon_conn(): auth registy supported: methods=[2] modes=[2,1] 2023-10-05T19:28:29.719 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.714+0000 7f1d97962e40 20 check_secure_mon_conn(): mode 1 is insecure 2023-10-05T19:28:29.720 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.714+0000 7f1d97962e40 1 Processor -- start 2023-10-05T19:28:29.720 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.714+0000 7f1d97962e40 1 -- start start 2023-10-05T19:28:29.720 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.714+0000 7f1d97962e40 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55fc7525f060 0x55fc7525f430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-05T19:28:29.720 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.714+0000 7f1d97962e40 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x55fc7525f970 con 0x55fc7525f060 2023-10-05T19:28:29.721 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.714+0000 7f1d8cc96700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55fc7525f060 0x55fc7525f430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-05T19:28:29.721 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.714+0000 7f1d8cc96700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55fc7525f060 0x55fc7525f430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.118:3300/0 says I am v2:172.21.15.167:57966/0 (socket says 172.21.15.167:57966) 2023-10-05T19:28:29.721 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.714+0000 7f1d8cc96700 1 -- 172.21.15.167:0/460542418 learned_addr learned my addr 172.21.15.167:0/460542418 (peer_addr_for_me v2:172.21.15.167:0/0) 2023-10-05T19:28:29.721 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.714+0000 7f1d8cc96700 1 -- 172.21.15.167:0/460542418 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55fc7525fab0 con 0x55fc7525f060 2023-10-05T19:28:29.722 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.714+0000 7f1d8cc96700 1 --2- 172.21.15.167:0/460542418 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55fc7525f060 0x55fc7525f430 secure :-1 s=READY pgs=294 cs=0 l=1 rev1=1 crypto rx=0x55fc75240080 tx=0x7f1d80006000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8c463e1bfe18a386 server_cookie=0 in_seq=0 out_seq=0 2023-10-05T19:28:29.722 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.718+0000 7f1d65fe3700 1 -- 172.21.15.167:0/460542418 <== mon.0 v2:172.21.15.118:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f1d80023040 con 0x55fc7525f060 2023-10-05T19:28:29.722 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.718+0000 7f1d65fe3700 1 -- 172.21.15.167:0/460542418 <== mon.0 v2:172.21.15.118:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1d80024040 con 0x55fc7525f060 2023-10-05T19:28:29.722 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.718+0000 7f1d65fe3700 1 -- 172.21.15.167:0/460542418 <== mon.0 v2:172.21.15.118:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f1d80013040 con 0x55fc7525f060 2023-10-05T19:28:29.723 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.718+0000 7f1d97962e40 1 -- 172.21.15.167:0/460542418 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55fc7525f060 msgr2=0x55fc7525f430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-05T19:28:29.723 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.718+0000 7f1d97962e40 1 --2- 172.21.15.167:0/460542418 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55fc7525f060 0x55fc7525f430 secure :-1 s=READY pgs=294 cs=0 l=1 rev1=1 crypto rx=0x55fc75240080 tx=0x7f1d80006000 comp rx=0 tx=0).stop 2023-10-05T19:28:29.723 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.718+0000 7f1d97962e40 1 -- 172.21.15.167:0/460542418 shutdown_connections 2023-10-05T19:28:29.724 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.718+0000 7f1d97962e40 1 --2- 172.21.15.167:0/460542418 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55fc7525f060 0x55fc7525f430 unknown :-1 s=CLOSED pgs=294 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-05T19:28:29.724 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.718+0000 7f1d97962e40 1 -- 172.21.15.167:0/460542418 >> 172.21.15.167:0/460542418 conn(0x55fc7525a420 msgr2=0x55fc75258570 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-05T19:28:29.724 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.718+0000 7f1d97962e40 1 -- 172.21.15.167:0/460542418 shutdown_connections 2023-10-05T19:28:29.724 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.718+0000 7f1d97962e40 1 -- 172.21.15.167:0/460542418 wait complete. 2023-10-05T19:28:29.725 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.718+0000 7f1d97962e40 1 Processor -- start 2023-10-05T19:28:29.725 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.718+0000 7f1d97962e40 1 -- start start 2023-10-05T19:28:29.725 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.718+0000 7f1d97962e40 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55fc7525f060 0x55fc75264c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-05T19:28:29.725 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.718+0000 7f1d97962e40 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x55fc75265c50 con 0x55fc7525f060 2023-10-05T19:28:29.725 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.718+0000 7f1d8cc96700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55fc7525f060 0x55fc75264c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-05T19:28:29.725 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.718+0000 7f1d8cc96700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55fc7525f060 0x55fc75264c50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.118:3300/0 says I am v2:172.21.15.167:57976/0 (socket says 172.21.15.167:57976) 2023-10-05T19:28:29.725 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.718+0000 7f1d8cc96700 1 -- 172.21.15.167:0/2822868296 learned_addr learned my addr 172.21.15.167:0/2822868296 (peer_addr_for_me v2:172.21.15.167:0/0) 2023-10-05T19:28:29.726 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.718+0000 7f1d8cc96700 1 -- 172.21.15.167:0/2822868296 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1d84001e10 con 0x55fc7525f060 2023-10-05T19:28:29.726 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.718+0000 7f1d8cc96700 1 --2- 172.21.15.167:0/2822868296 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55fc7525f060 0x55fc75264c50 secure :-1 s=READY pgs=295 cs=0 l=1 rev1=1 crypto rx=0x55fc752590e0 tx=0x7f1d80030000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-05T19:28:29.726 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.722+0000 7f1cd7fff700 1 -- 172.21.15.167:0/2822868296 <== mon.0 v2:172.21.15.118:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f1d80023070 con 0x55fc7525f060 2023-10-05T19:28:29.726 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.722+0000 7f1cd7fff700 1 -- 172.21.15.167:0/2822868296 <== mon.0 v2:172.21.15.118:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1d8001e630 con 0x55fc7525f060 2023-10-05T19:28:29.727 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.722+0000 7f1d97962e40 1 -- 172.21.15.167:0/2822868296 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x55fc75265250 con 0x55fc7525f060 2023-10-05T19:28:29.727 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.722+0000 7f1cd7fff700 1 -- 172.21.15.167:0/2822868296 <== mon.0 v2:172.21.15.118:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f1d8000e5d0 con 0x55fc7525f060 2023-10-05T19:28:29.727 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.722+0000 7f1d97962e40 1 -- 172.21.15.167:0/2822868296 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55fc75265690 con 0x55fc7525f060 2023-10-05T19:28:29.727 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.722+0000 7f1cd7fff700 1 -- 172.21.15.167:0/2822868296 <== mon.0 v2:172.21.15.118:3300/0 4 ==== mgrmap(e 5) v1 ==== 40329+0+0 (secure 0 0 0) 0x7f1d8001e7a0 con 0x55fc7525f060 2023-10-05T19:28:29.728 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.722+0000 7f1cd7fff700 1 --2- 172.21.15.167:0/2822868296 >> [v2:172.21.15.167:6824/29889,v1:172.21.15.167:6825/29889] conn(0x7f1cdc030c50 0x7f1cdc033100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-05T19:28:29.728 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.722+0000 7f1cd7fff700 1 -- 172.21.15.167:0/2822868296 <== mon.0 v2:172.21.15.118:3300/0 5 ==== osd_map(30..30 src has 1..30) v4 ==== 6057+0+0 (secure 0 0 0) 0x7f1d8004dc60 con 0x55fc7525f060 2023-10-05T19:28:29.728 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.722+0000 7f1d95c98700 1 --2- 172.21.15.167:0/2822868296 >> [v2:172.21.15.167:6824/29889,v1:172.21.15.167:6825/29889] conn(0x7f1cdc030c50 0x7f1cdc033100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-05T19:28:29.728 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.722+0000 7f1d97962e40 5 note: GC not initialized 2023-10-05T19:28:29.729 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.722+0000 7f1d95c98700 1 --2- 172.21.15.167:0/2822868296 >> [v2:172.21.15.167:6824/29889,v1:172.21.15.167:6825/29889] conn(0x7f1cdc030c50 0x7f1cdc033100 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x55fc75250f80 tx=0x7f1d9000aec0 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-05T19:28:29.730 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.726+0000 7f1cceffd700 20 reqs_thread_entry: start 2023-10-05T19:28:29.730 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.726+0000 7f1d97962e40 20 init_complete bucket index max shards: 11 2023-10-05T19:28:29.731 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.726+0000 7f1d97962e40 10 rgw notify: Started notification manager with: 1 workers 2023-10-05T19:28:29.731 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.726+0000 7f1ccd7fa700 20 rgw notify: INFO: next queues processing will happen at: Thu Oct 5 19:28:59 2023 2023-10-05T19:28:29.732 INFO:teuthology.orchestra.run.smithi167.stderr: 2023-10-05T19:28:29.732 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.726+0000 7f1cccff9700 20 reqs_thread_entry: start 2023-10-05T19:28:29.732 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.726+0000 7f1d97962e40 20 get_system_obj_state: rctx=0x7ffeed5583f8 obj=default.rgw.meta:users.uid:foo state=0x55fc750e80b0 s->prefetch_data=0 2023-10-05T19:28:29.732 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.726+0000 7f1d97962e40 10 cache get: name=default.rgw.meta+users.uid+foo : miss 2023-10-05T19:28:29.732 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.726+0000 7f1d97962e40 1 -- 172.21.15.167:0/1453987703 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:102 8.18 8:1e6ca5ed:users.uid::foo:head [call version.read in=11b,getxattrs,stat] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x55fc752cd480 con 0x55fc752118c0 2023-10-05T19:28:29.732 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.726+0000 7f1cd57fa700 20 reqs_thread_entry: start 2023-10-05T19:28:29.733 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.726+0000 7f1d95c98700 1 -- 172.21.15.167:0/1453987703 <== osd.1 v2:172.21.15.167:6800/30035 35 ==== osd_op_reply(102 foo [call out=48b,getxattrs out=75b,stat out=16b] v0'0 uv10 ondisk = 0) v8 ==== 231+0+139 (crc 0 0 0) 0x7f1d9003f040 con 0x55fc752118c0 2023-10-05T19:28:29.733 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.726+0000 7f1d97962e40 10 cache put: name=default.rgw.meta+users.uid+foo info.flags=0x16 2023-10-05T19:28:29.733 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.726+0000 7f1d97962e40 10 adding default.rgw.meta+users.uid+foo to cache LRU end 2023-10-05T19:28:29.733 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.726+0000 7f1d97962e40 10 updating xattr: name=ceph.objclass.version bl.length()=42 2023-10-05T19:28:29.733 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.726+0000 7f1d97962e40 20 get_system_obj_state: s->obj_tag was set empty 2023-10-05T19:28:29.733 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.726+0000 7f1d97962e40 20 Read xattr: user.rgw.idtag 2023-10-05T19:28:29.733 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.726+0000 7f1d97962e40 10 cache get: name=default.rgw.meta+users.uid+foo : type miss (requested=0x13, cached=0x16) 2023-10-05T19:28:29.733 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.726+0000 7f1d97962e40 20 rados->read ofs=0 len=0 2023-10-05T19:28:29.733 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.726+0000 7f1d97962e40 1 -- 172.21.15.167:0/1453987703 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:103 8.18 8:1e6ca5ed:users.uid::foo:head [call version.check_conds in=122b,call version.read in=11b,read 0~0,getxattrs] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x55fc752cff10 con 0x55fc752118c0 2023-10-05T19:28:29.734 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.726+0000 7f1d95c98700 1 -- 172.21.15.167:0/1453987703 <== osd.1 v2:172.21.15.167:6800/30035 36 ==== osd_op_reply(103 foo [call,call out=48b,read 0~513 out=513b,getxattrs out=75b] v0'0 uv10 ondisk = 0) v8 ==== 273+0+636 (crc 0 0 0) 0x7f1d9003f040 con 0x55fc752118c0 2023-10-05T19:28:29.734 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.726+0000 7f1d97962e40 20 rados_obj.operate() r=0 bl.length=513 2023-10-05T19:28:29.734 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.726+0000 7f1d97962e40 10 cache put: name=default.rgw.meta+users.uid+foo info.flags=0x13 2023-10-05T19:28:29.734 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.726+0000 7f1d97962e40 10 moving default.rgw.meta+users.uid+foo to cache LRU end 2023-10-05T19:28:29.734 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.726+0000 7f1d97962e40 10 updating xattr: name=ceph.objclass.version bl.length()=42 2023-10-05T19:28:29.734 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.726+0000 7f1d97962e40 1 -- 172.21.15.167:0/1453987703 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:104 8.18 8:1e6ca5ed:users.uid::foo:head [delete,create,call version.set in=58b,writefull 0~357 in=357b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x55fc752cc510 con 0x55fc752118c0 2023-10-05T19:28:29.736 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.730+0000 7f1d95c98700 1 -- 172.21.15.167:0/1453987703 <== osd.1 v2:172.21.15.167:6800/30035 37 ==== osd_op_reply(104 foo [delete,create,call,writefull 0~357] v30'11 uv11 ondisk = 0) v8 ==== 273+0+0 (crc 0 0 0) 0x7f1d9003f040 con 0x55fc752118c0 2023-10-05T19:28:29.736 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.730+0000 7f1d97962e40 10 cache put: name=default.rgw.meta+users.uid+foo info.flags=0x17 2023-10-05T19:28:29.736 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.730+0000 7f1d97962e40 10 moving default.rgw.meta+users.uid+foo to cache LRU end 2023-10-05T19:28:29.737 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.730+0000 7f1d97962e40 10 distributing notification oid=default.rgw.control:notify.4 cni=[op: 0, obj: default.rgw.meta:users.uid:foo, ofs0, ns] 2023-10-05T19:28:29.737 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.730+0000 7f1d97962e40 1 -- 172.21.15.167:0/1453987703 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:105 7.d 7:b4812045:::notify.4:head [notify cookie 94542785986592 in=545b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x55fc752cc4d0 con 0x55fc752118c0 2023-10-05T19:28:29.737 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.730+0000 7f1d95c98700 1 -- 172.21.15.167:0/1453987703 <== osd.1 v2:172.21.15.167:6800/30035 38 ==== watch-notify(notify (1) cookie 94542785474032 notify 128849018890 ret 0) v3 ==== 575+0+0 (crc 0 0 0) 0x7f1d9004e040 con 0x55fc752118c0 2023-10-05T19:28:29.737 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.730+0000 7f1d8f7fe700 10 rgw watcher librados: RGWWatcher::handle_notify() notify_id 128849018890 cookie 94542785474032 notifier 4383 bl.length()=533 2023-10-05T19:28:29.738 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.730+0000 7f1d8f7fe700 10 rgw watcher librados: cache put: name=default.rgw.meta+users.uid+foo info.flags=0x17 2023-10-05T19:28:29.738 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.730+0000 7f1d8f7fe700 10 rgw watcher librados: moving default.rgw.meta+users.uid+foo to cache LRU end 2023-10-05T19:28:29.738 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.730+0000 7f1d95c98700 1 -- 172.21.15.167:0/1453987703 <== osd.1 v2:172.21.15.167:6800/30035 39 ==== osd_op_reply(105 notify.4 [notify cookie 94542785986592 out=8b] v0'0 uv78 ondisk = 0) v8 ==== 152+0+8 (crc 0 0 0) 0x7f1d9003f040 con 0x55fc752118c0 2023-10-05T19:28:29.738 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.730+0000 7f1d8f7fe700 1 -- 172.21.15.167:0/1453987703 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:106 7.d 7:b4812045:::notify.4:head [notify-ack in=20b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x7f1d74003d90 con 0x55fc752118c0 2023-10-05T19:28:29.738 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.734+0000 7f1d95c98700 1 -- 172.21.15.167:0/1453987703 <== osd.1 v2:172.21.15.167:6800/30035 40 ==== watch-notify(notify_complete (2) cookie 94542785986592 notify 128849018890 ret 0) v3 ==== 42+0+48 (crc 0 0 0) 0x7f1d90036920 con 0x55fc752118c0 2023-10-05T19:28:29.738 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.734+0000 7f1d95c98700 1 -- 172.21.15.167:0/1453987703 <== osd.1 v2:172.21.15.167:6800/30035 41 ==== osd_op_reply(106 notify.4 [notify-ack] v0'0 uv78 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f1d9003f040 con 0x55fc752118c0 2023-10-05T19:28:29.739 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.734+0000 7f1d97962e40 10 distributing notification oid=default.rgw.control:notify.0 cni=[op: 1, obj: default.rgw.meta:users.swift:foo:foo2, ofs0, ns] 2023-10-05T19:28:29.739 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.734+0000 7f1d97962e40 1 -- 172.21.15.167:0/1453987703 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:107 7.1f 7:f95f44c2:::notify.0:head [notify cookie 94542785986592 in=171b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x55fc752cc470 con 0x7f1d88007030 2023-10-05T19:28:29.739 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.734+0000 7f1d8cc96700 1 -- 172.21.15.167:0/1453987703 <== osd.2 v2:172.21.15.167:6816/30037 17 ==== watch-notify(notify (1) cookie 94542785459520 notify 128849018885 ret 0) v3 ==== 201+0+0 (crc 0 0 0) 0x55fc75265690 con 0x7f1d88007030 2023-10-05T19:28:29.739 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.734+0000 7f1d8effd700 10 rgw watcher librados: RGWWatcher::handle_notify() notify_id 128849018885 cookie 94542785459520 notifier 4383 bl.length()=159 2023-10-05T19:28:29.739 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.734+0000 7f1d8effd700 1 -- 172.21.15.167:0/1453987703 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:108 7.1f 7:f95f44c2:::notify.0:head [notify-ack in=20b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x7f1ce40032d0 con 0x7f1d88007030 2023-10-05T19:28:29.740 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.734+0000 7f1d8cc96700 1 -- 172.21.15.167:0/1453987703 <== osd.2 v2:172.21.15.167:6816/30037 18 ==== osd_op_reply(107 notify.0 [notify cookie 94542785986592 out=8b] v0'0 uv78 ondisk = 0) v8 ==== 152+0+8 (crc 0 0 0) 0x7f1d8003f040 con 0x7f1d88007030 2023-10-05T19:28:29.740 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.734+0000 7f1d8cc96700 1 -- 172.21.15.167:0/1453987703 <== osd.2 v2:172.21.15.167:6816/30037 19 ==== watch-notify(notify_complete (2) cookie 94542785986592 notify 128849018885 ret 0) v3 ==== 42+0+48 (crc 0 0 0) 0x55fc75265250 con 0x7f1d88007030 2023-10-05T19:28:29.740 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.734+0000 7f1d8cc96700 1 -- 172.21.15.167:0/1453987703 <== osd.2 v2:172.21.15.167:6816/30037 20 ==== osd_op_reply(108 notify.0 [notify-ack] v0'0 uv78 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f1d8003f040 con 0x7f1d88007030 2023-10-05T19:28:29.740 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.734+0000 7f1d97962e40 1 -- 172.21.15.167:0/1453987703 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:109 8.2 8:45671577:users.swift::foo%3afoo2:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x55fc752cc020 con 0x7f1d88007030 2023-10-05T19:28:29.743 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.738+0000 7f1d8cc96700 1 -- 172.21.15.167:0/1453987703 <== osd.2 v2:172.21.15.167:6816/30037 21 ==== osd_op_reply(109 foo:foo2 [delete] v30'2 uv2 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f1d8003f040 con 0x7f1d88007030 2023-10-05T19:28:29.743 INFO:teuthology.orchestra.run.smithi167.stdout:{ 2023-10-05T19:28:29.743 INFO:teuthology.orchestra.run.smithi167.stdout: "user_id": "foo", 2023-10-05T19:28:29.743 INFO:teuthology.orchestra.run.smithi167.stdout: "display_name": "Foo", 2023-10-05T19:28:29.743 INFO:teuthology.orchestra.run.smithi167.stdout: "email": "foo@foo.com", 2023-10-05T19:28:29.743 INFO:teuthology.orchestra.run.smithi167.stdout: "suspended": 0, 2023-10-05T19:28:29.743 INFO:teuthology.orchestra.run.smithi167.stdout: "max_buckets": 4, 2023-10-05T19:28:29.744 INFO:teuthology.orchestra.run.smithi167.stdout: "subusers": [], 2023-10-05T19:28:29.744 INFO:teuthology.orchestra.run.smithi167.stdout: "keys": [ 2023-10-05T19:28:29.744 INFO:teuthology.orchestra.run.smithi167.stdout: { 2023-10-05T19:28:29.744 INFO:teuthology.orchestra.run.smithi167.stdout: "user": "foo", 2023-10-05T19:28:29.744 INFO:teuthology.orchestra.run.smithi167.stdout: "access_key": "9te6NH5mcdcq0Tc5i8i1", 2023-10-05T19:28:29.744 INFO:teuthology.orchestra.run.smithi167.stdout: "secret_key": "Ny4IOauQoL18Gp2zM7lC1vLmoawgqcYP/YGcWfXu" 2023-10-05T19:28:29.744 INFO:teuthology.orchestra.run.smithi167.stdout: } 2023-10-05T19:28:29.744 INFO:teuthology.orchestra.run.smithi167.stdout: ], 2023-10-05T19:28:29.744 INFO:teuthology.orchestra.run.smithi167.stdout: "swift_keys": [], 2023-10-05T19:28:29.744 INFO:teuthology.orchestra.run.smithi167.stdout: "caps": [], 2023-10-05T19:28:29.744 INFO:teuthology.orchestra.run.smithi167.stdout: "op_mask": "read, write, delete", 2023-10-05T19:28:29.744 INFO:teuthology.orchestra.run.smithi167.stdout: "default_placement": "", 2023-10-05T19:28:29.745 INFO:teuthology.orchestra.run.smithi167.stdout: "default_storage_class": "", 2023-10-05T19:28:29.745 INFO:teuthology.orchestra.run.smithi167.stdout: "placement_tags": [], 2023-10-05T19:28:29.745 INFO:teuthology.orchestra.run.smithi167.stdout: "bucket_quota": { 2023-10-05T19:28:29.745 INFO:teuthology.orchestra.run.smithi167.stdout: "enabled": false, 2023-10-05T19:28:29.745 INFO:teuthology.orchestra.run.smithi167.stdout: "check_on_raw": false, 2023-10-05T19:28:29.745 INFO:teuthology.orchestra.run.smithi167.stdout: "max_size": -1, 2023-10-05T19:28:29.745 INFO:teuthology.orchestra.run.smithi167.stdout: "max_size_kb": 0, 2023-10-05T19:28:29.745 INFO:teuthology.orchestra.run.smithi167.stdout: "max_objects": -1 2023-10-05T19:28:29.745 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2023-10-05T19:28:29.745 INFO:teuthology.orchestra.run.smithi167.stdout: "user_quota": { 2023-10-05T19:28:29.745 INFO:teuthology.orchestra.run.smithi167.stdout: "enabled": false, 2023-10-05T19:28:29.745 INFO:teuthology.orchestra.run.smithi167.stdout: "check_on_raw": false, 2023-10-05T19:28:29.745 INFO:teuthology.orchestra.run.smithi167.stdout: "max_size": -1, 2023-10-05T19:28:29.746 INFO:teuthology.orchestra.run.smithi167.stdout: "max_size_kb": 0, 2023-10-05T19:28:29.746 INFO:teuthology.orchestra.run.smithi167.stdout: "max_objects": -1 2023-10-05T19:28:29.746 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2023-10-05T19:28:29.746 INFO:teuthology.orchestra.run.smithi167.stdout: "temp_url_keys": [], 2023-10-05T19:28:29.746 INFO:teuthology.orchestra.run.smithi167.stdout: "type": "rgw", 2023-10-05T19:28:29.746 INFO:teuthology.orchestra.run.smithi167.stdout: "mfa_ids": [] 2023-10-05T19:28:29.746 INFO:teuthology.orchestra.run.smithi167.stdout:} 2023-10-05T19:28:29.746 INFO:teuthology.orchestra.run.smithi167.stdout: 2023-10-05T19:28:29.747 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.738+0000 7f1d97962e40 1 -- 172.21.15.167:0/1453987703 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:110 7.1f 7:f95f44c2:::notify.0:head [watch unwatch cookie 94542785459520] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x55fc752cc020 con 0x7f1d88007030 2023-10-05T19:28:29.747 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.742+0000 7f1d8cc96700 1 -- 172.21.15.167:0/1453987703 <== osd.2 v2:172.21.15.167:6816/30037 22 ==== osd_op_reply(110 notify.0 [watch unwatch cookie 94542785459520] v30'80 uv78 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f1d8003f040 con 0x7f1d88007030 2023-10-05T19:28:29.747 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.742+0000 7f1d97962e40 20 remove_watcher() i=0 2023-10-05T19:28:29.747 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.742+0000 7f1d97962e40 2 removed watcher, disabling cache 2023-10-05T19:28:29.747 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.742+0000 7f1d97962e40 1 -- 172.21.15.167:0/1453987703 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:111 7.0 7:05bf5b68:::notify.1:head [watch unwatch cookie 94542785463120] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x55fc752cc020 con 0x55fc752118c0 2023-10-05T19:28:29.749 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.742+0000 7f1d95c98700 1 -- 172.21.15.167:0/1453987703 <== osd.1 v2:172.21.15.167:6800/30035 42 ==== osd_op_reply(111 notify.1 [watch unwatch cookie 94542785463120] v30'80 uv78 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f1d9003f040 con 0x55fc752118c0 2023-10-05T19:28:29.749 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.742+0000 7f1d97962e40 20 remove_watcher() i=1 2023-10-05T19:28:29.749 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.742+0000 7f1d97962e40 1 -- 172.21.15.167:0/1453987703 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:112 7.15 7:a93a5511:::notify.2:head [watch unwatch cookie 94542785466672] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x55fc752cc020 con 0x55fc752464d0 2023-10-05T19:28:29.751 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.746+0000 7f1d95497700 1 -- 172.21.15.167:0/1453987703 <== osd.3 v2:172.21.15.167:6808/30036 24 ==== osd_op_reply(112 notify.2 [watch unwatch cookie 94542785466672] v30'80 uv78 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f1d8805f040 con 0x55fc752464d0 2023-10-05T19:28:29.752 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.746+0000 7f1d97962e40 20 remove_watcher() i=2 2023-10-05T19:28:29.752 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.746+0000 7f1d97962e40 1 -- 172.21.15.167:0/1453987703 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:113 7.e 7:7759931f:::notify.3:head [watch unwatch cookie 94542785464224] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x55fc752267d0 con 0x7f1d88007030 2023-10-05T19:28:29.755 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.750+0000 7f1d8cc96700 1 -- 172.21.15.167:0/1453987703 <== osd.2 v2:172.21.15.167:6816/30037 23 ==== osd_op_reply(113 notify.3 [watch unwatch cookie 94542785464224] v30'80 uv78 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f1d8003f040 con 0x7f1d88007030 2023-10-05T19:28:29.755 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.750+0000 7f1d97962e40 20 remove_watcher() i=3 2023-10-05T19:28:29.755 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.750+0000 7f1d97962e40 1 -- 172.21.15.167:0/1453987703 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:114 7.d 7:b4812045:::notify.4:head [watch unwatch cookie 94542785474032] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x55fc7524c840 con 0x55fc752118c0 2023-10-05T19:28:29.758 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.754+0000 7f1d95c98700 1 -- 172.21.15.167:0/1453987703 <== osd.1 v2:172.21.15.167:6800/30035 43 ==== osd_op_reply(114 notify.4 [watch unwatch cookie 94542785474032] v30'80 uv78 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f1d9003f040 con 0x55fc752118c0 2023-10-05T19:28:29.758 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.754+0000 7f1d97962e40 20 remove_watcher() i=4 2023-10-05T19:28:29.758 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.754+0000 7f1d97962e40 1 -- 172.21.15.167:0/1453987703 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:115 7.3 7:c609908c:::notify.5:head [watch unwatch cookie 94542785478080] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x55fc7524c840 con 0x7f1d8000eb60 2023-10-05T19:28:29.761 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.754+0000 7f1d95c98700 1 -- 172.21.15.167:0/1453987703 <== osd.0 v2:172.21.15.118:6800/27690 29 ==== osd_op_reply(115 notify.5 [watch unwatch cookie 94542785478080] v30'80 uv78 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f1d9003f040 con 0x7f1d8000eb60 2023-10-05T19:28:29.761 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.754+0000 7f1d97962e40 20 remove_watcher() i=5 2023-10-05T19:28:29.761 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.754+0000 7f1d97962e40 1 -- 172.21.15.167:0/1453987703 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:116 7.14 7:2b04a3e9:::notify.6:head [watch unwatch cookie 94542785469536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x55fc7524c840 con 0x7f1d8000eb60 2023-10-05T19:28:29.763 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.758+0000 7f1d95c98700 1 -- 172.21.15.167:0/1453987703 <== osd.0 v2:172.21.15.118:6800/27690 30 ==== osd_op_reply(116 notify.6 [watch unwatch cookie 94542785469536] v30'80 uv78 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f1d9003f040 con 0x7f1d8000eb60 2023-10-05T19:28:29.763 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.758+0000 7f1d97962e40 20 remove_watcher() i=6 2023-10-05T19:28:29.764 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.758+0000 7f1d97962e40 1 -- 172.21.15.167:0/1453987703 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:117 7.9 7:93e5b521:::notify.7:head [watch unwatch cookie 94542785480336] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x55fc7524c840 con 0x7f1d8000eb60 2023-10-05T19:28:29.766 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.762+0000 7f1d95c98700 1 -- 172.21.15.167:0/1453987703 <== osd.0 v2:172.21.15.118:6800/27690 31 ==== osd_op_reply(117 notify.7 [watch unwatch cookie 94542785480336] v30'80 uv78 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f1d9003f040 con 0x7f1d8000eb60 2023-10-05T19:28:29.766 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.762+0000 7f1d97962e40 20 remove_watcher() i=7 2023-10-05T19:28:29.768 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.762+0000 7f1d667e4700 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2023-10-05T19:28:29.768 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.762+0000 7f1d97962e40 1 -- 172.21.15.167:0/1453987703 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:118 6.4 6:22d26bf9:::data_loggenerations_metadata:head [watch unwatch cookie 94542785484656] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x55fc752267d0 con 0x55fc752118c0 2023-10-05T19:28:29.771 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.766+0000 7f1d95c98700 1 -- 172.21.15.167:0/1453987703 <== osd.1 v2:172.21.15.167:6800/30035 44 ==== osd_op_reply(118 data_loggenerations_metadata [watch unwatch cookie 94542785484656] v30'87 uv1 ondisk = 0) v8 ==== 172+0+0 (crc 0 0 0) 0x7f1d9003f040 con 0x55fc752118c0 2023-10-05T19:28:29.771 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.766+0000 7f1d97962e40 1 -- 172.21.15.167:0/1453987703 >> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] conn(0x7f1d8000eb60 msgr2=0x7f1d8000d8e0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-05T19:28:29.771 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.766+0000 7f1d97962e40 1 --2- 172.21.15.167:0/1453987703 >> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] conn(0x7f1d8000eb60 0x7f1d8000d8e0 crc :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-05T19:28:29.771 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.766+0000 7f1d97962e40 1 -- 172.21.15.167:0/1453987703 >> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] conn(0x55fc752118c0 msgr2=0x55fc75242670 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-05T19:28:29.771 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.766+0000 7f1d97962e40 1 --2- 172.21.15.167:0/1453987703 >> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] conn(0x55fc752118c0 0x55fc75242670 crc :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-05T19:28:29.771 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.766+0000 7f1d97962e40 1 -- 172.21.15.167:0/1453987703 >> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] conn(0x7f1d88007030 msgr2=0x7f1d88006820 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-05T19:28:29.771 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.766+0000 7f1d97962e40 1 --2- 172.21.15.167:0/1453987703 >> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] conn(0x7f1d88007030 0x7f1d88006820 crc :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-05T19:28:29.772 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.766+0000 7f1d97962e40 1 -- 172.21.15.167:0/1453987703 >> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] conn(0x55fc752464d0 msgr2=0x55fc752488b0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-05T19:28:29.772 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.766+0000 7f1d97962e40 1 --2- 172.21.15.167:0/1453987703 >> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] conn(0x55fc752464d0 0x55fc752488b0 crc :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-05T19:28:29.772 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.766+0000 7f1d97962e40 1 -- 172.21.15.167:0/1453987703 >> [v2:172.21.15.167:6824/29889,v1:172.21.15.167:6825/29889] conn(0x7f1d78030bf0 msgr2=0x7f1d780330a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-05T19:28:29.772 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.766+0000 7f1d97962e40 1 --2- 172.21.15.167:0/1453987703 >> [v2:172.21.15.167:6824/29889,v1:172.21.15.167:6825/29889] conn(0x7f1d78030bf0 0x7f1d780330a0 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7f1d80004170 tx=0x7f1d80001400 comp rx=0 tx=0).stop 2023-10-05T19:28:29.772 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.766+0000 7f1d97962e40 1 -- 172.21.15.167:0/1453987703 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55fc75210fb0 msgr2=0x55fc75236c30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-05T19:28:29.772 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.766+0000 7f1d95c98700 1 -- 172.21.15.167:0/1453987703 reap_dead start 2023-10-05T19:28:29.772 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.766+0000 7f1d97962e40 1 --2- 172.21.15.167:0/1453987703 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55fc75210fb0 0x55fc75236c30 secure :-1 s=READY pgs=293 cs=0 l=1 rev1=1 crypto rx=0x7f1d8801f040 tx=0x7f1d8801c000 comp rx=0 tx=0).stop 2023-10-05T19:28:29.773 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.766+0000 7f1d97962e40 1 -- 172.21.15.167:0/1453987703 shutdown_connections 2023-10-05T19:28:29.773 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.766+0000 7f1d97962e40 1 --2- 172.21.15.167:0/1453987703 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55fc75210fb0 0x55fc75236c30 unknown :-1 s=CLOSED pgs=293 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-05T19:28:29.774 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.766+0000 7f1d97962e40 1 -- 172.21.15.167:0/1453987703 >> 172.21.15.167:0/1453987703 conn(0x55fc75184800 msgr2=0x55fc7522a4f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-05T19:28:29.774 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.766+0000 7f1d97962e40 1 -- 172.21.15.167:0/1453987703 shutdown_connections 2023-10-05T19:28:29.774 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.766+0000 7f1d97962e40 1 -- 172.21.15.167:0/1453987703 wait complete. 2023-10-05T19:28:29.774 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.766+0000 7f1d97962e40 1 -- 172.21.15.167:0/2822868296 >> [v2:172.21.15.167:6824/29889,v1:172.21.15.167:6825/29889] conn(0x7f1cdc030c50 msgr2=0x7f1cdc033100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-05T19:28:29.774 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.766+0000 7f1d97962e40 1 --2- 172.21.15.167:0/2822868296 >> [v2:172.21.15.167:6824/29889,v1:172.21.15.167:6825/29889] conn(0x7f1cdc030c50 0x7f1cdc033100 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x55fc75250f80 tx=0x7f1d9000aec0 comp rx=0 tx=0).stop 2023-10-05T19:28:29.774 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.766+0000 7f1d97962e40 1 -- 172.21.15.167:0/2822868296 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55fc7525f060 msgr2=0x55fc75264c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-05T19:28:29.774 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.766+0000 7f1d97962e40 1 --2- 172.21.15.167:0/2822868296 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55fc7525f060 0x55fc75264c50 secure :-1 s=READY pgs=295 cs=0 l=1 rev1=1 crypto rx=0x55fc752590e0 tx=0x7f1d80030000 comp rx=0 tx=0).stop 2023-10-05T19:28:29.774 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.766+0000 7f1d97962e40 1 -- 172.21.15.167:0/2822868296 shutdown_connections 2023-10-05T19:28:29.774 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.766+0000 7f1d97962e40 1 --2- 172.21.15.167:0/2822868296 >> [v2:172.21.15.167:6824/29889,v1:172.21.15.167:6825/29889] conn(0x7f1cdc030c50 0x7f1cdc033100 unknown :-1 s=CLOSED pgs=105 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-05T19:28:29.774 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.766+0000 7f1d97962e40 1 --2- 172.21.15.167:0/2822868296 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55fc7525f060 0x55fc75264c50 unknown :-1 s=CLOSED pgs=295 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-05T19:28:29.774 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.766+0000 7f1d97962e40 1 -- 172.21.15.167:0/2822868296 >> 172.21.15.167:0/2822868296 conn(0x55fc7525a420 msgr2=0x55fc752584c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-05T19:28:29.775 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.766+0000 7f1d97962e40 1 -- 172.21.15.167:0/2822868296 shutdown_connections 2023-10-05T19:28:29.775 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.766+0000 7f1d97962e40 1 -- 172.21.15.167:0/2822868296 wait complete. 2023-10-05T19:28:29.779 DEBUG:tasks.util.rgw: json result: {'user_id': 'foo', 'display_name': 'Foo', 'email': 'foo@foo.com', 'suspended': 0, 'max_buckets': 4, 'subusers': [], 'keys': [{'user': 'foo', 'access_key': '9te6NH5mcdcq0Tc5i8i1', 'secret_key': 'Ny4IOauQoL18Gp2zM7lC1vLmoawgqcYP/YGcWfXu'}], 'swift_keys': [], 'caps': [], 'op_mask': 'read, write, delete', 'default_placement': '', 'default_storage_class': '', 'placement_tags': [], 'bucket_quota': {'enabled': False, 'check_on_raw': False, 'max_size': -1, 'max_size_kb': 0, 'max_objects': -1}, 'user_quota': {'enabled': False, 'check_on_raw': False, 'max_size': -1, 'max_size_kb': 0, 'max_objects': -1}, 'temp_url_keys': [], 'type': 'rgw', 'mfa_ids': []} 2023-10-05T19:28:29.779 INFO:tasks.util.rgw:rgwadmin: client.0 : ['bucket', 'stats', '--uid', 'foo'] 2023-10-05T19:28:29.779 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'ceph', 'bucket', 'stats', '--uid', 'foo'] 2023-10-05T19:28:29.779 DEBUG:teuthology.orchestra.run.smithi167:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster ceph bucket stats --uid foo 2023-10-05T19:28:29.860 INFO:teuthology.orchestra.run.smithi167.stderr:ignoring --setuser ceph since I am not root 2023-10-05T19:28:29.860 INFO:teuthology.orchestra.run.smithi167.stderr:ignoring --setgroup ceph since I am not root 2023-10-05T19:28:29.861 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.854+0000 7fa6eb1ebe40 1 Processor -- start 2023-10-05T19:28:29.861 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.854+0000 7fa6eb1ebe40 1 -- start start 2023-10-05T19:28:29.861 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.854+0000 7fa6eb1ebe40 1 --2- >> v2:172.21.15.118:3300/0 conn(0x55bc13c70e30 0x55bc13c71200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-05T19:28:29.862 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.854+0000 7fa6eb1ebe40 1 -- --> v1:172.21.15.118:6789/0 -- auth(proto 0 26 bytes epoch 0) v1 -- 0x55bc13bbdbd0 con 0x55bc13c71740 2023-10-05T19:28:29.862 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.854+0000 7fa6eb1ebe40 1 -- --> v2:172.21.15.118:3300/0 -- mon_getmap magic: 0 v1 -- 0x55bc13b7aee0 con 0x55bc13c70e30 2023-10-05T19:28:29.863 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.854+0000 7fa6e8d20700 1 --2- >> v2:172.21.15.118:3300/0 conn(0x55bc13c70e30 0x55bc13c71200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-05T19:28:29.863 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.854+0000 7fa6e851f700 1 --1- >> v1:172.21.15.118:6789/0 conn(0x55bc13c71740 0x55bc13c721e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.118:6789/0 says I am v1:172.21.15.167:56532/0 (socket says 172.21.15.167:56532) 2023-10-05T19:28:29.863 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.854+0000 7fa6e851f700 1 -- 172.21.15.167:0/4176803749 learned_addr learned my addr 172.21.15.167:0/4176803749 (peer_addr_for_me v1:172.21.15.167:0/0) 2023-10-05T19:28:29.863 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.858+0000 7fa6e8d20700 1 -- 172.21.15.167:0/4176803749 >> v1:172.21.15.118:6789/0 conn(0x55bc13c71740 legacy=0x55bc13c721e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-05T19:28:29.863 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.858+0000 7fa6e7d1e700 1 -- 172.21.15.167:0/4176803749 <== mon.0 v1:172.21.15.118:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 11776058 0 0) 0x55bc13bbdbd0 con 0x55bc13c71740 2023-10-05T19:28:29.863 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.858+0000 7fa6e8d20700 1 -- 172.21.15.167:0/4176803749 --> v2:172.21.15.118:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55bc13c728f0 con 0x55bc13c70e30 2023-10-05T19:28:29.863 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.858+0000 7fa6e8d20700 1 --2- 172.21.15.167:0/4176803749 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55bc13c70e30 0x55bc13c71200 secure :-1 s=READY pgs=297 cs=0 l=1 rev1=1 crypto rx=0x7fa6e000a520 tx=0x7fa6e0005d40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9d024ddc9085057b server_cookie=0 in_seq=0 out_seq=0 2023-10-05T19:28:29.864 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.858+0000 7fa6e7d1e700 1 -- 172.21.15.167:0/4176803749 <== mon.0 v2:172.21.15.118:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fa6e000f590 con 0x55bc13c70e30 2023-10-05T19:28:29.864 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.858+0000 7fa6e7d1e700 1 -- 172.21.15.167:0/4176803749 <== mon.0 v2:172.21.15.118:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa6e000f6f0 con 0x55bc13c70e30 2023-10-05T19:28:29.865 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.858+0000 7fa6e7d1e700 1 -- 172.21.15.167:0/4176803749 <== mon.0 v2:172.21.15.118:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fa6e000f880 con 0x55bc13c70e30 2023-10-05T19:28:29.865 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.858+0000 7fa6eb1ebe40 1 -- 172.21.15.167:0/4176803749 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55bc13c70e30 msgr2=0x55bc13c71200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-05T19:28:29.865 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.858+0000 7fa6eb1ebe40 1 --2- 172.21.15.167:0/4176803749 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55bc13c70e30 0x55bc13c71200 secure :-1 s=READY pgs=297 cs=0 l=1 rev1=1 crypto rx=0x7fa6e000a520 tx=0x7fa6e0005d40 comp rx=0 tx=0).stop 2023-10-05T19:28:29.865 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.858+0000 7fa6eb1ebe40 1 -- 172.21.15.167:0/4176803749 shutdown_connections 2023-10-05T19:28:29.865 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.858+0000 7fa6eb1ebe40 1 --2- 172.21.15.167:0/4176803749 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55bc13c70e30 0x55bc13c71200 unknown :-1 s=CLOSED pgs=297 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-05T19:28:29.865 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.858+0000 7fa6eb1ebe40 1 -- 172.21.15.167:0/4176803749 >> 172.21.15.167:0/4176803749 conn(0x55bc13c6c850 msgr2=0x55bc13c6ec60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-05T19:28:29.865 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.858+0000 7fa6eb1ebe40 1 -- 172.21.15.167:0/4176803749 shutdown_connections 2023-10-05T19:28:29.865 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.858+0000 7fa6eb1ebe40 1 -- 172.21.15.167:0/4176803749 wait complete. 2023-10-05T19:28:29.866 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.858+0000 7fa6eb1ebe40 1 Processor -- start 2023-10-05T19:28:29.866 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.858+0000 7fa6eb1ebe40 1 -- start start 2023-10-05T19:28:29.866 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.858+0000 7fa6eb1ebe40 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55bc13c70e30 0x55bc13c8fc90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-05T19:28:29.866 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.858+0000 7fa6eb1ebe40 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x55bc13b3ee30 con 0x55bc13c70e30 2023-10-05T19:28:29.866 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.858+0000 7fa6e8d20700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55bc13c70e30 0x55bc13c8fc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-05T19:28:29.867 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.858+0000 7fa6e8d20700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55bc13c70e30 0x55bc13c8fc90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.118:3300/0 says I am v2:172.21.15.167:57988/0 (socket says 172.21.15.167:57988) 2023-10-05T19:28:29.867 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.858+0000 7fa6e8d20700 1 -- 172.21.15.167:0/995842106 learned_addr learned my addr 172.21.15.167:0/995842106 (peer_addr_for_me v2:172.21.15.167:0/0) 2023-10-05T19:28:29.867 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.862+0000 7fa6e8d20700 1 -- 172.21.15.167:0/995842106 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa6e000a0e0 con 0x55bc13c70e30 2023-10-05T19:28:29.867 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.862+0000 7fa6e8d20700 1 --2- 172.21.15.167:0/995842106 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55bc13c70e30 0x55bc13c8fc90 secure :-1 s=READY pgs=298 cs=0 l=1 rev1=1 crypto rx=0x7fa6e0005b20 tx=0x7fa6e0014000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5c6f27f099934414 server_cookie=0 in_seq=0 out_seq=0 2023-10-05T19:28:29.867 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.862+0000 7fa6e6d1c700 1 -- 172.21.15.167:0/995842106 <== mon.0 v2:172.21.15.118:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fa6e000f860 con 0x55bc13c70e30 2023-10-05T19:28:29.868 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.862+0000 7fa6e6d1c700 1 -- 172.21.15.167:0/995842106 <== mon.0 v2:172.21.15.118:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa6e000f9c0 con 0x55bc13c70e30 2023-10-05T19:28:29.868 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.862+0000 7fa6e6d1c700 1 -- 172.21.15.167:0/995842106 <== mon.0 v2:172.21.15.118:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fa6e000fb50 con 0x55bc13c70e30 2023-10-05T19:28:29.868 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.862+0000 7fa6eb1ebe40 1 -- 172.21.15.167:0/995842106 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55bc13c70e30 msgr2=0x55bc13c8fc90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-05T19:28:29.869 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.862+0000 7fa6eb1ebe40 1 --2- 172.21.15.167:0/995842106 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55bc13c70e30 0x55bc13c8fc90 secure :-1 s=READY pgs=298 cs=0 l=1 rev1=1 crypto rx=0x7fa6e0005b20 tx=0x7fa6e0014000 comp rx=0 tx=0).stop 2023-10-05T19:28:29.869 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.862+0000 7fa6eb1ebe40 1 -- 172.21.15.167:0/995842106 shutdown_connections 2023-10-05T19:28:29.869 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.862+0000 7fa6eb1ebe40 1 --2- 172.21.15.167:0/995842106 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55bc13c70e30 0x55bc13c8fc90 unknown :-1 s=CLOSED pgs=298 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-05T19:28:29.869 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.862+0000 7fa6eb1ebe40 1 -- 172.21.15.167:0/995842106 >> 172.21.15.167:0/995842106 conn(0x55bc13c6c850 msgr2=0x55bc13c8b850 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-05T19:28:29.869 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.862+0000 7fa6eb1ebe40 1 -- 172.21.15.167:0/995842106 shutdown_connections 2023-10-05T19:28:29.869 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.862+0000 7fa6eb1ebe40 1 -- 172.21.15.167:0/995842106 wait complete. 2023-10-05T19:28:29.870 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.862+0000 7fa6eb1ebe40 1 Processor -- start 2023-10-05T19:28:29.870 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.862+0000 7fa6eb1ebe40 1 -- start start 2023-10-05T19:28:29.870 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.866+0000 7fa6eb1ebe40 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55bc13c70e30 0x55bc13c96990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-05T19:28:29.870 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.866+0000 7fa6eb1ebe40 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x55bc13b11ed0 con 0x55bc13c70e30 2023-10-05T19:28:29.871 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.866+0000 7fa6e8d20700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55bc13c70e30 0x55bc13c96990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-05T19:28:29.871 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.866+0000 7fa6e8d20700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55bc13c70e30 0x55bc13c96990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.118:3300/0 says I am v2:172.21.15.167:58002/0 (socket says 172.21.15.167:58002) 2023-10-05T19:28:29.871 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.866+0000 7fa6e8d20700 1 -- 172.21.15.167:0/1248727001 learned_addr learned my addr 172.21.15.167:0/1248727001 (peer_addr_for_me v2:172.21.15.167:0/0) 2023-10-05T19:28:29.871 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.866+0000 7fa6e8d20700 1 -- 172.21.15.167:0/1248727001 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55bc13c728f0 con 0x55bc13c70e30 2023-10-05T19:28:29.872 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.866+0000 7fa6e8d20700 1 --2- 172.21.15.167:0/1248727001 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55bc13c70e30 0x55bc13c96990 secure :-1 s=READY pgs=299 cs=0 l=1 rev1=1 crypto rx=0x7fa6e0005b20 tx=0x7fa6e001d000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-05T19:28:29.872 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.866+0000 7fa6e5519700 1 -- 172.21.15.167:0/1248727001 <== mon.0 v2:172.21.15.118:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fa6e0010420 con 0x55bc13c70e30 2023-10-05T19:28:29.872 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.866+0000 7fa6e5519700 1 -- 172.21.15.167:0/1248727001 <== mon.0 v2:172.21.15.118:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa6e0010580 con 0x55bc13c70e30 2023-10-05T19:28:29.873 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.866+0000 7fa6eb1ebe40 1 -- 172.21.15.167:0/1248727001 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x55bc13c96ed0 con 0x55bc13c70e30 2023-10-05T19:28:29.873 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.866+0000 7fa6e5519700 1 -- 172.21.15.167:0/1248727001 <== mon.0 v2:172.21.15.118:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fa6e0010710 con 0x55bc13c70e30 2023-10-05T19:28:29.873 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.866+0000 7fa6eb1ebe40 1 -- 172.21.15.167:0/1248727001 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55bc13c97310 con 0x55bc13c70e30 2023-10-05T19:28:29.873 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.866+0000 7fa6e5519700 1 -- 172.21.15.167:0/1248727001 <== mon.0 v2:172.21.15.118:3300/0 4 ==== mgrmap(e 5) v1 ==== 40329+0+0 (secure 0 0 0) 0x7fa6e0010870 con 0x55bc13c70e30 2023-10-05T19:28:29.874 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.866+0000 7fa6e5519700 1 --2- 172.21.15.167:0/1248727001 >> [v2:172.21.15.167:6824/29889,v1:172.21.15.167:6825/29889] conn(0x7fa6c8030c50 0x7fa6c8033100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-05T19:28:29.874 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.866+0000 7fa6e5519700 1 -- 172.21.15.167:0/1248727001 <== mon.0 v2:172.21.15.118:3300/0 5 ==== osd_map(30..30 src has 1..30) v4 ==== 6057+0+0 (secure 0 0 0) 0x7fa6e002b070 con 0x55bc13c70e30 2023-10-05T19:28:29.874 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.870+0000 7fa6eb1ebe40 20 rados->read ofs=0 len=0 2023-10-05T19:28:29.874 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.870+0000 7fa6e851f700 1 --2- 172.21.15.167:0/1248727001 >> [v2:172.21.15.167:6824/29889,v1:172.21.15.167:6825/29889] conn(0x7fa6c8030c50 0x7fa6c8033100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-05T19:28:29.875 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.870+0000 7fa6eb1ebe40 1 --2- 172.21.15.167:0/1248727001 >> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] conn(0x55bc13c71740 0x55bc13ca2300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-05T19:28:29.875 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.870+0000 7fa6eb1ebe40 1 -- 172.21.15.167:0/1248727001 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:1 5.12 5:49953fa1:::default.realm:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x55bc13ca2a00 con 0x55bc13c71740 2023-10-05T19:28:29.875 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.870+0000 7fa6e9521700 1 --2- 172.21.15.167:0/1248727001 >> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] conn(0x55bc13c71740 0x55bc13ca2300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-05T19:28:29.875 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.870+0000 7fa6e851f700 1 --2- 172.21.15.167:0/1248727001 >> [v2:172.21.15.167:6824/29889,v1:172.21.15.167:6825/29889] conn(0x7fa6c8030c50 0x7fa6c8033100 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7fa6d00011e0 tx=0x7fa6d00013a0 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-05T19:28:29.875 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.870+0000 7fa6e9521700 1 --2- 172.21.15.167:0/1248727001 >> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] conn(0x55bc13c71740 0x55bc13ca2300 crc :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-05T19:28:29.877 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.870+0000 7fa6e9521700 1 -- 172.21.15.167:0/1248727001 <== osd.1 v2:172.21.15.167:6800/30035 1 ==== osd_op_reply(1 default.realm [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 ==== 157+0+0 (crc 0 0 0) 0x7fa6d800a500 con 0x55bc13c71740 2023-10-05T19:28:29.877 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.870+0000 7fa6eb1ebe40 20 rados_obj.operate() r=-2 bl.length=0 2023-10-05T19:28:29.877 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.870+0000 7fa6eb1ebe40 20 realm 2023-10-05T19:28:29.877 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.870+0000 7fa6eb1ebe40 20 rados->read ofs=0 len=0 2023-10-05T19:28:29.877 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.870+0000 7fa6eb1ebe40 1 -- 172.21.15.167:0/1248727001 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:2 5.12 5:49953fa1:::default.realm:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x55bc13c9f960 con 0x55bc13c71740 2023-10-05T19:28:29.877 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.870+0000 7fa6e9521700 1 -- 172.21.15.167:0/1248727001 <== osd.1 v2:172.21.15.167:6800/30035 2 ==== osd_op_reply(2 default.realm [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 ==== 157+0+0 (crc 0 0 0) 0x7fa6d800a500 con 0x55bc13c71740 2023-10-05T19:28:29.878 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.870+0000 7fa6eb1ebe40 20 rados_obj.operate() r=-2 bl.length=0 2023-10-05T19:28:29.878 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.870+0000 7fa6eb1ebe40 4 RGWPeriod::init failed to init realm id : (2) No such file or directory 2023-10-05T19:28:29.878 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.870+0000 7fa6eb1ebe40 20 rados->read ofs=0 len=0 2023-10-05T19:28:29.878 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.870+0000 7fa6eb1ebe40 1 -- 172.21.15.167:0/1248727001 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:3 5.12 5:49953fa1:::default.realm:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x55bc13ca34e0 con 0x55bc13c71740 2023-10-05T19:28:29.878 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.870+0000 7fa6e9521700 1 -- 172.21.15.167:0/1248727001 <== osd.1 v2:172.21.15.167:6800/30035 3 ==== osd_op_reply(3 default.realm [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 ==== 157+0+0 (crc 0 0 0) 0x7fa6d800a500 con 0x55bc13c71740 2023-10-05T19:28:29.878 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.870+0000 7fa6eb1ebe40 20 rados_obj.operate() r=-2 bl.length=0 2023-10-05T19:28:29.878 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.870+0000 7fa6eb1ebe40 20 rados->read ofs=0 len=0 2023-10-05T19:28:29.878 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.870+0000 7fa6eb1ebe40 1 -- 172.21.15.167:0/1248727001 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:4 5.f 5:f43b6ece:::zone_names.default:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x55bc13ca30c0 con 0x55bc13c71740 2023-10-05T19:28:29.879 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.870+0000 7fa6e9521700 1 -- 172.21.15.167:0/1248727001 <== osd.1 v2:172.21.15.167:6800/30035 4 ==== osd_op_reply(4 zone_names.default [read 0~46 out=46b] v0'0 uv1 ondisk = 0) v8 ==== 162+0+46 (crc 0 0 0) 0x7fa6d800a500 con 0x55bc13c71740 2023-10-05T19:28:29.880 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.870+0000 7fa6eb1ebe40 20 rados_obj.operate() r=0 bl.length=46 2023-10-05T19:28:29.880 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.870+0000 7fa6eb1ebe40 20 rados->read ofs=0 len=0 2023-10-05T19:28:29.880 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.870+0000 7fa6eb1ebe40 1 -- 172.21.15.167:0/1248727001 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:5 5.1d 5:b8374eb4:::zone_info.9a9df965-6612-45d4-a45f-6e3c889bacc2:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x55bc13ca3f50 con 0x55bc13c71740 2023-10-05T19:28:29.880 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.874+0000 7fa6e9521700 1 -- 172.21.15.167:0/1248727001 <== osd.1 v2:172.21.15.167:6800/30035 5 ==== osd_op_reply(5 zone_info.9a9df965-6612-45d4-a45f-6e3c889bacc2 [read 0~874 out=874b] v0'0 uv2 ondisk = 0) v8 ==== 190+0+874 (crc 0 0 0) 0x7fa6d800a500 con 0x55bc13c71740 2023-10-05T19:28:29.880 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.874+0000 7fa6eb1ebe40 20 rados_obj.operate() r=0 bl.length=874 2023-10-05T19:28:29.880 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.874+0000 7fa6eb1ebe40 20 searching for the correct realm 2023-10-05T19:28:29.880 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.874+0000 7fa6eb1ebe40 1 --2- 172.21.15.167:0/1248727001 >> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] conn(0x55bc13ca5da0 0x55bc13ca8180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-05T19:28:29.880 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.874+0000 7fa6eb1ebe40 1 -- 172.21.15.167:0/1248727001 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:6 5.0 5:00000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x55bc13ca86c0 con 0x55bc13ca5da0 2023-10-05T19:28:29.880 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.874+0000 7fa6e8d20700 1 --2- 172.21.15.167:0/1248727001 >> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] conn(0x55bc13ca5da0 0x55bc13ca8180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-05T19:28:29.880 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.874+0000 7fa6e8d20700 1 --2- 172.21.15.167:0/1248727001 >> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] conn(0x55bc13ca5da0 0x55bc13ca8180 crc :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-05T19:28:29.881 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.874+0000 7fa6e8d20700 1 -- 172.21.15.167:0/1248727001 <== osd.3 v2:172.21.15.167:6808/30036 1 ==== osd_op_reply(6 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fa6e0018c60 con 0x55bc13ca5da0 2023-10-05T19:28:29.881 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.874+0000 7fa6e8d20700 1 --2- 172.21.15.167:0/1248727001 >> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] conn(0x7fa6e001a740 0x7fa6e00398d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-05T19:28:29.881 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.874+0000 7fa6e8d20700 1 -- 172.21.15.167:0/1248727001 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:7 5.10 5:08000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x55bc13ca86c0 con 0x7fa6e001a740 2023-10-05T19:28:29.881 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.874+0000 7fa6e851f700 1 --2- 172.21.15.167:0/1248727001 >> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] conn(0x7fa6e001a740 0x7fa6e00398d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-05T19:28:29.882 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.874+0000 7fa6e851f700 1 --2- 172.21.15.167:0/1248727001 >> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] conn(0x7fa6e001a740 0x7fa6e00398d0 crc :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-05T19:28:29.882 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.878+0000 7fa6e851f700 1 -- 172.21.15.167:0/1248727001 <== osd.2 v2:172.21.15.167:6816/30037 1 ==== osd_op_reply(7 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fa6d0006070 con 0x7fa6e001a740 2023-10-05T19:28:29.882 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.878+0000 7fa6e851f700 1 --2- 172.21.15.167:0/1248727001 >> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] conn(0x7fa6d000d8e0 0x7fa6d000c360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-05T19:28:29.883 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.878+0000 7fa6e851f700 1 -- 172.21.15.167:0/1248727001 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:8 5.8 5:10000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x55bc13ca86c0 con 0x7fa6d000d8e0 2023-10-05T19:28:29.883 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.878+0000 7fa6e9521700 1 --2- 172.21.15.167:0/1248727001 >> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] conn(0x7fa6d000d8e0 0x7fa6d000c360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-05T19:28:29.883 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.878+0000 7fa6e9521700 1 --2- 172.21.15.167:0/1248727001 >> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] conn(0x7fa6d000d8e0 0x7fa6d000c360 crc :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-05T19:28:29.884 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.878+0000 7fa6e9521700 1 -- 172.21.15.167:0/1248727001 <== osd.0 v2:172.21.15.118:6800/27690 1 ==== osd_op_reply(8 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fa6d8014040 con 0x7fa6d000d8e0 2023-10-05T19:28:29.884 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.878+0000 7fa6e9521700 1 -- 172.21.15.167:0/1248727001 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:9 5.18 5:18000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x55bc13ca86c0 con 0x55bc13c71740 2023-10-05T19:28:29.885 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.878+0000 7fa6e9521700 1 -- 172.21.15.167:0/1248727001 <== osd.1 v2:172.21.15.167:6800/30035 6 ==== osd_op_reply(9 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fa6d8014040 con 0x55bc13c71740 2023-10-05T19:28:29.885 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.878+0000 7fa6e9521700 1 -- 172.21.15.167:0/1248727001 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:10 5.4 5:20000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x55bc13ca86c0 con 0x55bc13c71740 2023-10-05T19:28:29.885 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.878+0000 7fa6e9521700 1 -- 172.21.15.167:0/1248727001 <== osd.1 v2:172.21.15.167:6800/30035 7 ==== osd_op_reply(10 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fa6d8014040 con 0x55bc13c71740 2023-10-05T19:28:29.885 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.878+0000 7fa6e9521700 1 -- 172.21.15.167:0/1248727001 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:11 5.14 5:28000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x55bc13ca86c0 con 0x7fa6d000d8e0 2023-10-05T19:28:29.886 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.882+0000 7fa6e9521700 1 -- 172.21.15.167:0/1248727001 <== osd.0 v2:172.21.15.118:6800/27690 2 ==== osd_op_reply(11 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fa6d8014040 con 0x7fa6d000d8e0 2023-10-05T19:28:29.886 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.882+0000 7fa6e9521700 1 -- 172.21.15.167:0/1248727001 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:12 5.c 5:30000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x55bc13ca86c0 con 0x7fa6d000d8e0 2023-10-05T19:28:29.887 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.882+0000 7fa6e9521700 1 -- 172.21.15.167:0/1248727001 <== osd.0 v2:172.21.15.118:6800/27690 3 ==== osd_op_reply(12 [pgnls start_epoch 30 out=112b] v23'1 uv1 ondisk = 1) v8 ==== 144+0+112 (crc 0 0 0) 0x7fa6d8014040 con 0x7fa6d000d8e0 2023-10-05T19:28:29.887 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.882+0000 7fa6e9521700 1 -- 172.21.15.167:0/1248727001 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:13 5.1c 5:38000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x55bc13ca86c0 con 0x7fa6e001a740 2023-10-05T19:28:29.887 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.882+0000 7fa6e851f700 1 -- 172.21.15.167:0/1248727001 <== osd.2 v2:172.21.15.167:6816/30037 2 ==== osd_op_reply(13 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fa6d0006070 con 0x7fa6e001a740 2023-10-05T19:28:29.887 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.882+0000 7fa6e851f700 1 -- 172.21.15.167:0/1248727001 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:14 5.2 5:40000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x55bc13ca86c0 con 0x55bc13ca5da0 2023-10-05T19:28:29.888 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.882+0000 7fa6e8d20700 1 -- 172.21.15.167:0/1248727001 <== osd.3 v2:172.21.15.167:6808/30036 2 ==== osd_op_reply(14 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fa6e0018c60 con 0x55bc13ca5da0 2023-10-05T19:28:29.888 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.882+0000 7fa6e8d20700 1 -- 172.21.15.167:0/1248727001 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:15 5.12 5:48000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x55bc13ca86c0 con 0x55bc13c71740 2023-10-05T19:28:29.889 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.882+0000 7fa6e9521700 1 -- 172.21.15.167:0/1248727001 <== osd.1 v2:172.21.15.167:6800/30035 8 ==== osd_op_reply(15 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fa6d8014040 con 0x55bc13c71740 2023-10-05T19:28:29.889 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.882+0000 7fa6e9521700 1 -- 172.21.15.167:0/1248727001 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:16 5.a 5:50000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x55bc13ca86c0 con 0x7fa6d000d8e0 2023-10-05T19:28:29.890 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.882+0000 7fa6e9521700 1 -- 172.21.15.167:0/1248727001 <== osd.0 v2:172.21.15.118:6800/27690 4 ==== osd_op_reply(16 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fa6d8022040 con 0x7fa6d000d8e0 2023-10-05T19:28:29.890 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.886+0000 7fa6e9521700 1 -- 172.21.15.167:0/1248727001 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:17 5.1a 5:58000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x55bc13ca86c0 con 0x55bc13c71740 2023-10-05T19:28:29.890 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.886+0000 7fa6e9521700 1 -- 172.21.15.167:0/1248727001 <== osd.1 v2:172.21.15.167:6800/30035 9 ==== osd_op_reply(17 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fa6d8022040 con 0x55bc13c71740 2023-10-05T19:28:29.890 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.886+0000 7fa6e9521700 1 -- 172.21.15.167:0/1248727001 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:18 5.6 5:60000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x55bc13ca86c0 con 0x7fa6e001a740 2023-10-05T19:28:29.891 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.886+0000 7fa6e851f700 1 -- 172.21.15.167:0/1248727001 <== osd.2 v2:172.21.15.167:6816/30037 3 ==== osd_op_reply(18 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fa6d0006070 con 0x7fa6e001a740 2023-10-05T19:28:29.891 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.886+0000 7fa6e851f700 1 -- 172.21.15.167:0/1248727001 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:19 5.16 5:68000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x55bc13ca86c0 con 0x55bc13ca5da0 2023-10-05T19:28:29.892 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.886+0000 7fa6e8d20700 1 -- 172.21.15.167:0/1248727001 <== osd.3 v2:172.21.15.167:6808/30036 3 ==== osd_op_reply(19 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fa6e0018c60 con 0x55bc13ca5da0 2023-10-05T19:28:29.892 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.886+0000 7fa6e8d20700 1 -- 172.21.15.167:0/1248727001 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:20 5.e 5:70000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x55bc13ca86c0 con 0x55bc13ca5da0 2023-10-05T19:28:29.893 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.886+0000 7fa6e8d20700 1 -- 172.21.15.167:0/1248727001 <== osd.3 v2:172.21.15.167:6808/30036 4 ==== osd_op_reply(20 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fa6e0018c60 con 0x55bc13ca5da0 2023-10-05T19:28:29.893 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.886+0000 7fa6e8d20700 1 -- 172.21.15.167:0/1248727001 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:21 5.1e 5:78000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x55bc13ca86c0 con 0x7fa6d000d8e0 2023-10-05T19:28:29.893 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.886+0000 7fa6e9521700 1 -- 172.21.15.167:0/1248727001 <== osd.0 v2:172.21.15.118:6800/27690 5 ==== osd_op_reply(21 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fa6d8022040 con 0x7fa6d000d8e0 2023-10-05T19:28:29.894 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.886+0000 7fa6e9521700 1 -- 172.21.15.167:0/1248727001 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:22 5.1 5:80000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x55bc13ca86c0 con 0x55bc13ca5da0 2023-10-05T19:28:29.894 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.890+0000 7fa6e8d20700 1 -- 172.21.15.167:0/1248727001 <== osd.3 v2:172.21.15.167:6808/30036 5 ==== osd_op_reply(22 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fa6e0018c60 con 0x55bc13ca5da0 2023-10-05T19:28:29.894 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.890+0000 7fa6e8d20700 1 -- 172.21.15.167:0/1248727001 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:23 5.11 5:88000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x55bc13ca86c0 con 0x55bc13ca5da0 2023-10-05T19:28:29.895 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.890+0000 7fa6e8d20700 1 -- 172.21.15.167:0/1248727001 <== osd.3 v2:172.21.15.167:6808/30036 6 ==== osd_op_reply(23 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fa6e0018c60 con 0x55bc13ca5da0 2023-10-05T19:28:29.895 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.890+0000 7fa6e8d20700 1 -- 172.21.15.167:0/1248727001 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:24 5.9 5:90000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x55bc13ca86c0 con 0x55bc13c71740 2023-10-05T19:28:29.896 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.890+0000 7fa6e9521700 1 -- 172.21.15.167:0/1248727001 <== osd.1 v2:172.21.15.167:6800/30035 10 ==== osd_op_reply(24 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fa6d8022040 con 0x55bc13c71740 2023-10-05T19:28:29.896 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.890+0000 7fa6e9521700 1 -- 172.21.15.167:0/1248727001 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:25 5.19 5:98000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x55bc13ca86c0 con 0x55bc13c71740 2023-10-05T19:28:29.896 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.890+0000 7fa6e9521700 1 -- 172.21.15.167:0/1248727001 <== osd.1 v2:172.21.15.167:6800/30035 11 ==== osd_op_reply(25 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fa6d8022040 con 0x55bc13c71740 2023-10-05T19:28:29.896 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.890+0000 7fa6e9521700 1 -- 172.21.15.167:0/1248727001 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:26 5.5 5:a0000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x55bc13ca86c0 con 0x7fa6d000d8e0 2023-10-05T19:28:29.897 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.890+0000 7fa6e9521700 1 -- 172.21.15.167:0/1248727001 <== osd.0 v2:172.21.15.118:6800/27690 6 ==== osd_op_reply(26 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fa6d8022040 con 0x7fa6d000d8e0 2023-10-05T19:28:29.898 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.890+0000 7fa6e9521700 1 -- 172.21.15.167:0/1248727001 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:27 5.15 5:a8000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x55bc13ca86c0 con 0x55bc13ca5da0 2023-10-05T19:28:29.898 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.894+0000 7fa6e8d20700 1 -- 172.21.15.167:0/1248727001 <== osd.3 v2:172.21.15.167:6808/30036 7 ==== osd_op_reply(27 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fa6e0018c60 con 0x55bc13ca5da0 2023-10-05T19:28:29.898 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.894+0000 7fa6e8d20700 1 -- 172.21.15.167:0/1248727001 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:28 5.d 5:b0000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x55bc13ca86c0 con 0x7fa6e001a740 2023-10-05T19:28:29.899 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.894+0000 7fa6e851f700 1 -- 172.21.15.167:0/1248727001 <== osd.2 v2:172.21.15.167:6816/30037 4 ==== osd_op_reply(28 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fa6d0006070 con 0x7fa6e001a740 2023-10-05T19:28:29.899 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.894+0000 7fa6e851f700 1 -- 172.21.15.167:0/1248727001 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:29 5.1d 5:b8000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x55bc13ca86c0 con 0x55bc13c71740 2023-10-05T19:28:29.900 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.894+0000 7fa6e9521700 1 -- 172.21.15.167:0/1248727001 <== osd.1 v2:172.21.15.167:6800/30035 12 ==== osd_op_reply(29 [pgnls start_epoch 30 out=107b] v30'2 uv2 ondisk = 1) v8 ==== 144+0+107 (crc 0 0 0) 0x7fa6d8022040 con 0x55bc13c71740 2023-10-05T19:28:29.900 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.894+0000 7fa6e9521700 1 -- 172.21.15.167:0/1248727001 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:30 5.3 5:c0000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x55bc13ca86c0 con 0x7fa6d000d8e0 2023-10-05T19:28:29.900 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.894+0000 7fa6e9521700 1 -- 172.21.15.167:0/1248727001 <== osd.0 v2:172.21.15.118:6800/27690 7 ==== osd_op_reply(30 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fa6d8022040 con 0x7fa6d000d8e0 2023-10-05T19:28:29.900 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.894+0000 7fa6e9521700 1 -- 172.21.15.167:0/1248727001 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:31 5.13 5:c8000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x55bc13ca86c0 con 0x55bc13ca5da0 2023-10-05T19:28:29.901 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.894+0000 7fa6e8d20700 1 -- 172.21.15.167:0/1248727001 <== osd.3 v2:172.21.15.167:6808/30036 8 ==== osd_op_reply(31 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fa6e0018c60 con 0x55bc13ca5da0 2023-10-05T19:28:29.901 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.894+0000 7fa6e8d20700 1 -- 172.21.15.167:0/1248727001 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:32 5.b 5:d0000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x55bc13ca86c0 con 0x7fa6d000d8e0 2023-10-05T19:28:29.902 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.898+0000 7fa6e9521700 1 -- 172.21.15.167:0/1248727001 <== osd.0 v2:172.21.15.118:6800/27690 8 ==== osd_op_reply(32 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fa6d8022040 con 0x7fa6d000d8e0 2023-10-05T19:28:29.902 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.898+0000 7fa6e9521700 1 -- 172.21.15.167:0/1248727001 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:33 5.1b 5:d8000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x55bc13ca86c0 con 0x7fa6e001a740 2023-10-05T19:28:29.903 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.898+0000 7fa6e851f700 1 -- 172.21.15.167:0/1248727001 <== osd.2 v2:172.21.15.167:6816/30037 5 ==== osd_op_reply(33 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fa6d0006070 con 0x7fa6e001a740 2023-10-05T19:28:29.903 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.898+0000 7fa6e851f700 1 -- 172.21.15.167:0/1248727001 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:34 5.7 5:e0000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x55bc13ca86c0 con 0x55bc13ca5da0 2023-10-05T19:28:29.904 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.898+0000 7fa6e8d20700 1 -- 172.21.15.167:0/1248727001 <== osd.3 v2:172.21.15.167:6808/30036 9 ==== osd_op_reply(34 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fa6e0018c60 con 0x55bc13ca5da0 2023-10-05T19:28:29.904 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.898+0000 7fa6e8d20700 1 -- 172.21.15.167:0/1248727001 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:35 5.17 5:e8000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x55bc13ca86c0 con 0x7fa6d000d8e0 2023-10-05T19:28:29.905 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.898+0000 7fa6e9521700 1 -- 172.21.15.167:0/1248727001 <== osd.0 v2:172.21.15.118:6800/27690 9 ==== osd_op_reply(35 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fa6d8022040 con 0x7fa6d000d8e0 2023-10-05T19:28:29.905 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.898+0000 7fa6e9521700 1 -- 172.21.15.167:0/1248727001 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:36 5.f 5:f0000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x55bc13ca86c0 con 0x55bc13c71740 2023-10-05T19:28:29.906 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.898+0000 7fa6e9521700 1 -- 172.21.15.167:0/1248727001 <== osd.1 v2:172.21.15.167:6800/30035 13 ==== osd_op_reply(36 [pgnls start_epoch 30 out=115b] v23'2 uv2 ondisk = 1) v8 ==== 144+0+115 (crc 0 0 0) 0x7fa6d802f080 con 0x55bc13c71740 2023-10-05T19:28:29.906 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.898+0000 7fa6e9521700 1 -- 172.21.15.167:0/1248727001 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:37 5.1f 5:f8000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x55bc13ca86c0 con 0x7fa6e001a740 2023-10-05T19:28:29.907 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.902+0000 7fa6e851f700 1 -- 172.21.15.167:0/1248727001 <== osd.2 v2:172.21.15.167:6816/30037 6 ==== osd_op_reply(37 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fa6d0006070 con 0x7fa6e001a740 2023-10-05T19:28:29.907 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.902+0000 7fa6eb1ebe40 20 RGWRados::pool_iterate: got zonegroup_info.4f1f719d-9648-417b-b10c-e5d939825419 2023-10-05T19:28:29.907 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.902+0000 7fa6eb1ebe40 20 RGWRados::pool_iterate: got zone_info.9a9df965-6612-45d4-a45f-6e3c889bacc2 2023-10-05T19:28:29.907 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.902+0000 7fa6eb1ebe40 20 RGWRados::pool_iterate: got zone_names.default 2023-10-05T19:28:29.907 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.902+0000 7fa6eb1ebe40 20 RGWRados::pool_iterate: got zonegroups_names.default 2023-10-05T19:28:29.907 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.902+0000 7fa6eb1ebe40 20 rados->read ofs=0 len=0 2023-10-05T19:28:29.907 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.902+0000 7fa6eb1ebe40 1 -- 172.21.15.167:0/1248727001 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:38 5.12 5:49953fa1:::default.realm:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x55bc13ca4ad0 con 0x55bc13c71740 2023-10-05T19:28:29.908 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.902+0000 7fa6e9521700 1 -- 172.21.15.167:0/1248727001 <== osd.1 v2:172.21.15.167:6800/30035 14 ==== osd_op_reply(38 default.realm [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 ==== 157+0+0 (crc 0 0 0) 0x7fa6d802f080 con 0x55bc13c71740 2023-10-05T19:28:29.908 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.902+0000 7fa6eb1ebe40 20 rados_obj.operate() r=-2 bl.length=0 2023-10-05T19:28:29.908 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.902+0000 7fa6eb1ebe40 20 rados->read ofs=0 len=0 2023-10-05T19:28:29.909 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.902+0000 7fa6eb1ebe40 1 -- 172.21.15.167:0/1248727001 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:39 5.f 5:f4c53578:::zonegroups_names.default:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x55bc13ca46d0 con 0x55bc13c71740 2023-10-05T19:28:29.909 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.902+0000 7fa6e9521700 1 -- 172.21.15.167:0/1248727001 <== osd.1 v2:172.21.15.167:6800/30035 15 ==== osd_op_reply(39 zonegroups_names.default [read 0~46 out=46b] v0'0 uv2 ondisk = 0) v8 ==== 168+0+46 (crc 0 0 0) 0x7fa6d802f080 con 0x55bc13c71740 2023-10-05T19:28:29.909 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.902+0000 7fa6eb1ebe40 20 rados_obj.operate() r=0 bl.length=46 2023-10-05T19:28:29.909 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.902+0000 7fa6eb1ebe40 20 rados->read ofs=0 len=0 2023-10-05T19:28:29.909 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.902+0000 7fa6eb1ebe40 1 -- 172.21.15.167:0/1248727001 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:40 5.c 5:30d299a5:::zonegroup_info.4f1f719d-9648-417b-b10c-e5d939825419:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x55bc13ca5410 con 0x7fa6d000d8e0 2023-10-05T19:28:29.910 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.902+0000 7fa6e9521700 1 -- 172.21.15.167:0/1248727001 <== osd.0 v2:172.21.15.118:6800/27690 10 ==== osd_op_reply(40 zonegroup_info.4f1f719d-9648-417b-b10c-e5d939825419 [read 0~362 out=362b] v0'0 uv1 ondisk = 0) v8 ==== 195+0+362 (crc 0 0 0) 0x7fa6d802f080 con 0x7fa6d000d8e0 2023-10-05T19:28:29.910 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.902+0000 7fa6eb1ebe40 20 rados_obj.operate() r=0 bl.length=362 2023-10-05T19:28:29.910 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.902+0000 7fa6eb1ebe40 20 rados->read ofs=0 len=0 2023-10-05T19:28:29.910 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.902+0000 7fa6eb1ebe40 1 -- 172.21.15.167:0/1248727001 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:41 5.13 5:cbbf6c60:::converted:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x55bc13ca8b00 con 0x55bc13ca5da0 2023-10-05T19:28:29.910 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.902+0000 7fa6e8d20700 1 -- 172.21.15.167:0/1248727001 <== osd.3 v2:172.21.15.167:6808/30036 10 ==== osd_op_reply(41 converted [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 ==== 153+0+0 (crc 0 0 0) 0x7fa6e0018c60 con 0x55bc13ca5da0 2023-10-05T19:28:29.910 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.902+0000 7fa6eb1ebe40 20 rados_obj.operate() r=-2 bl.length=0 2023-10-05T19:28:29.910 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.902+0000 7fa6eb1ebe40 20 rados->read ofs=0 len=0 2023-10-05T19:28:29.911 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.902+0000 7fa6eb1ebe40 1 -- 172.21.15.167:0/1248727001 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:42 5.12 5:49953fa1:::default.realm:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x55bc13ca8eb0 con 0x55bc13c71740 2023-10-05T19:28:29.911 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.906+0000 7fa6e9521700 1 -- 172.21.15.167:0/1248727001 <== osd.1 v2:172.21.15.167:6800/30035 16 ==== osd_op_reply(42 default.realm [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 ==== 157+0+0 (crc 0 0 0) 0x7fa6d802f080 con 0x55bc13c71740 2023-10-05T19:28:29.911 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.906+0000 7fa6eb1ebe40 20 rados_obj.operate() r=-2 bl.length=0 2023-10-05T19:28:29.912 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.906+0000 7fa6eb1ebe40 20 rados->read ofs=0 len=0 2023-10-05T19:28:29.912 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.906+0000 7fa6eb1ebe40 1 -- 172.21.15.167:0/1248727001 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:43 5.f 5:f4c53578:::zonegroups_names.default:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x55bc13ca8eb0 con 0x55bc13c71740 2023-10-05T19:28:29.912 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.906+0000 7fa6e9521700 1 -- 172.21.15.167:0/1248727001 <== osd.1 v2:172.21.15.167:6800/30035 17 ==== osd_op_reply(43 zonegroups_names.default [read 0~46 out=46b] v0'0 uv2 ondisk = 0) v8 ==== 168+0+46 (crc 0 0 0) 0x7fa6d802f080 con 0x55bc13c71740 2023-10-05T19:28:29.912 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.906+0000 7fa6eb1ebe40 20 rados_obj.operate() r=0 bl.length=46 2023-10-05T19:28:29.912 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.906+0000 7fa6eb1ebe40 1 -- 172.21.15.167:0/1248727001 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:44 5.0 5:00000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x55bc13ca9ec0 con 0x55bc13ca5da0 2023-10-05T19:28:29.914 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.906+0000 7fa6e8d20700 1 -- 172.21.15.167:0/1248727001 <== osd.3 v2:172.21.15.167:6808/30036 11 ==== osd_op_reply(44 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fa6e0018c60 con 0x55bc13ca5da0 2023-10-05T19:28:29.914 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.906+0000 7fa6e8d20700 1 -- 172.21.15.167:0/1248727001 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:45 5.10 5:08000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x55bc13ca9ec0 con 0x7fa6e001a740 2023-10-05T19:28:29.915 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.910+0000 7fa6e851f700 1 -- 172.21.15.167:0/1248727001 <== osd.2 v2:172.21.15.167:6816/30037 7 ==== osd_op_reply(45 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fa6d0006070 con 0x7fa6e001a740 2023-10-05T19:28:29.915 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.910+0000 7fa6e851f700 1 -- 172.21.15.167:0/1248727001 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:46 5.8 5:10000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x55bc13ca9ec0 con 0x7fa6d000d8e0 2023-10-05T19:28:29.916 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.910+0000 7fa6e9521700 1 -- 172.21.15.167:0/1248727001 <== osd.0 v2:172.21.15.118:6800/27690 11 ==== osd_op_reply(46 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fa6d802f080 con 0x7fa6d000d8e0 2023-10-05T19:28:29.916 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.910+0000 7fa6e9521700 1 -- 172.21.15.167:0/1248727001 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:47 5.18 5:18000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x55bc13ca46d0 con 0x55bc13c71740 2023-10-05T19:28:29.916 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.910+0000 7fa6e9521700 1 -- 172.21.15.167:0/1248727001 <== osd.1 v2:172.21.15.167:6800/30035 18 ==== osd_op_reply(47 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fa6d802f080 con 0x55bc13c71740 2023-10-05T19:28:29.916 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.910+0000 7fa6e9521700 1 -- 172.21.15.167:0/1248727001 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:48 5.4 5:20000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x55bc13ca46d0 con 0x55bc13c71740 2023-10-05T19:28:29.917 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.910+0000 7fa6e9521700 1 -- 172.21.15.167:0/1248727001 <== osd.1 v2:172.21.15.167:6800/30035 19 ==== osd_op_reply(48 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fa6d802f080 con 0x55bc13c71740 2023-10-05T19:28:29.917 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.910+0000 7fa6e9521700 1 -- 172.21.15.167:0/1248727001 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:49 5.14 5:28000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x55bc13ca46d0 con 0x7fa6d000d8e0 2023-10-05T19:28:29.918 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.914+0000 7fa6e9521700 1 -- 172.21.15.167:0/1248727001 <== osd.0 v2:172.21.15.118:6800/27690 12 ==== osd_op_reply(49 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fa6d802f080 con 0x7fa6d000d8e0 2023-10-05T19:28:29.918 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.914+0000 7fa6e9521700 1 -- 172.21.15.167:0/1248727001 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:50 5.c 5:30000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x55bc13ca46d0 con 0x7fa6d000d8e0 2023-10-05T19:28:29.918 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.914+0000 7fa6e9521700 1 -- 172.21.15.167:0/1248727001 <== osd.0 v2:172.21.15.118:6800/27690 13 ==== osd_op_reply(50 [pgnls start_epoch 30 out=112b] v23'1 uv1 ondisk = 1) v8 ==== 144+0+112 (crc 0 0 0) 0x7fa6d802f080 con 0x7fa6d000d8e0 2023-10-05T19:28:29.919 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.914+0000 7fa6e9521700 1 -- 172.21.15.167:0/1248727001 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:51 5.1c 5:38000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x55bc13ca46d0 con 0x7fa6e001a740 2023-10-05T19:28:29.919 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.914+0000 7fa6e851f700 1 -- 172.21.15.167:0/1248727001 <== osd.2 v2:172.21.15.167:6816/30037 8 ==== osd_op_reply(51 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fa6d0006070 con 0x7fa6e001a740 2023-10-05T19:28:29.920 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.914+0000 7fa6e851f700 1 -- 172.21.15.167:0/1248727001 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:52 5.2 5:40000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x55bc13ca46d0 con 0x55bc13ca5da0 2023-10-05T19:28:29.920 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.914+0000 7fa6e8d20700 1 -- 172.21.15.167:0/1248727001 <== osd.3 v2:172.21.15.167:6808/30036 12 ==== osd_op_reply(52 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fa6e0018c60 con 0x55bc13ca5da0 2023-10-05T19:28:29.920 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.914+0000 7fa6e8d20700 1 -- 172.21.15.167:0/1248727001 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:53 5.12 5:48000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x55bc13ca46d0 con 0x55bc13c71740 2023-10-05T19:28:29.921 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.914+0000 7fa6e9521700 1 -- 172.21.15.167:0/1248727001 <== osd.1 v2:172.21.15.167:6800/30035 20 ==== osd_op_reply(53 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fa6d802f080 con 0x55bc13c71740 2023-10-05T19:28:29.921 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.914+0000 7fa6e9521700 1 -- 172.21.15.167:0/1248727001 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:54 5.a 5:50000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x55bc13ca46d0 con 0x7fa6d000d8e0 2023-10-05T19:28:29.922 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.918+0000 7fa6e9521700 1 -- 172.21.15.167:0/1248727001 <== osd.0 v2:172.21.15.118:6800/27690 14 ==== osd_op_reply(54 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fa6d802f080 con 0x7fa6d000d8e0 2023-10-05T19:28:29.922 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.918+0000 7fa6e9521700 1 -- 172.21.15.167:0/1248727001 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:55 5.1a 5:58000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x55bc13ca46d0 con 0x55bc13c71740 2023-10-05T19:28:29.923 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.918+0000 7fa6e9521700 1 -- 172.21.15.167:0/1248727001 <== osd.1 v2:172.21.15.167:6800/30035 21 ==== osd_op_reply(55 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fa6d802f080 con 0x55bc13c71740 2023-10-05T19:28:29.923 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.918+0000 7fa6e9521700 1 -- 172.21.15.167:0/1248727001 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:56 5.6 5:60000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x55bc13ca46d0 con 0x7fa6e001a740 2023-10-05T19:28:29.924 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.918+0000 7fa6e851f700 1 -- 172.21.15.167:0/1248727001 <== osd.2 v2:172.21.15.167:6816/30037 9 ==== osd_op_reply(56 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fa6d0006070 con 0x7fa6e001a740 2023-10-05T19:28:29.924 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.918+0000 7fa6e851f700 1 -- 172.21.15.167:0/1248727001 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:57 5.16 5:68000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x55bc13ca46d0 con 0x55bc13ca5da0 2023-10-05T19:28:29.925 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.918+0000 7fa6e8d20700 1 -- 172.21.15.167:0/1248727001 <== osd.3 v2:172.21.15.167:6808/30036 13 ==== osd_op_reply(57 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fa6e0018c60 con 0x55bc13ca5da0 2023-10-05T19:28:29.925 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.918+0000 7fa6e8d20700 1 -- 172.21.15.167:0/1248727001 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:58 5.e 5:70000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x55bc13ca46d0 con 0x55bc13ca5da0 2023-10-05T19:28:29.925 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.918+0000 7fa6e8d20700 1 -- 172.21.15.167:0/1248727001 <== osd.3 v2:172.21.15.167:6808/30036 14 ==== osd_op_reply(58 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fa6e0018c60 con 0x55bc13ca5da0 2023-10-05T19:28:29.925 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.918+0000 7fa6e8d20700 1 -- 172.21.15.167:0/1248727001 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:59 5.1e 5:78000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x55bc13ca46d0 con 0x7fa6d000d8e0 2023-10-05T19:28:29.926 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.922+0000 7fa6e9521700 1 -- 172.21.15.167:0/1248727001 <== osd.0 v2:172.21.15.118:6800/27690 15 ==== osd_op_reply(59 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fa6d802f080 con 0x7fa6d000d8e0 2023-10-05T19:28:29.926 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.922+0000 7fa6e9521700 1 -- 172.21.15.167:0/1248727001 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:60 5.1 5:80000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x55bc13ca46d0 con 0x55bc13ca5da0 2023-10-05T19:28:29.927 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.922+0000 7fa6e8d20700 1 -- 172.21.15.167:0/1248727001 <== osd.3 v2:172.21.15.167:6808/30036 15 ==== osd_op_reply(60 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fa6e0018c60 con 0x55bc13ca5da0 2023-10-05T19:28:29.927 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.922+0000 7fa6e8d20700 1 -- 172.21.15.167:0/1248727001 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:61 5.11 5:88000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x55bc13ca46d0 con 0x55bc13ca5da0 2023-10-05T19:28:29.928 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.922+0000 7fa6e8d20700 1 -- 172.21.15.167:0/1248727001 <== osd.3 v2:172.21.15.167:6808/30036 16 ==== osd_op_reply(61 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fa6e0018c60 con 0x55bc13ca5da0 2023-10-05T19:28:29.928 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.922+0000 7fa6e8d20700 1 -- 172.21.15.167:0/1248727001 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:62 5.9 5:90000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x55bc13ca46d0 con 0x55bc13c71740 2023-10-05T19:28:29.929 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.922+0000 7fa6e9521700 1 -- 172.21.15.167:0/1248727001 <== osd.1 v2:172.21.15.167:6800/30035 22 ==== osd_op_reply(62 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fa6d802f080 con 0x55bc13c71740 2023-10-05T19:28:29.929 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.922+0000 7fa6e9521700 1 -- 172.21.15.167:0/1248727001 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:63 5.19 5:98000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x55bc13ca46d0 con 0x55bc13c71740 2023-10-05T19:28:29.930 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.922+0000 7fa6e9521700 1 -- 172.21.15.167:0/1248727001 <== osd.1 v2:172.21.15.167:6800/30035 23 ==== osd_op_reply(63 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fa6d802f080 con 0x55bc13c71740 2023-10-05T19:28:29.930 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.922+0000 7fa6e9521700 1 -- 172.21.15.167:0/1248727001 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:64 5.5 5:a0000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x55bc13ca46d0 con 0x7fa6d000d8e0 2023-10-05T19:28:29.931 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.926+0000 7fa6e9521700 1 -- 172.21.15.167:0/1248727001 <== osd.0 v2:172.21.15.118:6800/27690 16 ==== osd_op_reply(64 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fa6d802f080 con 0x7fa6d000d8e0 2023-10-05T19:28:29.931 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.926+0000 7fa6e9521700 1 -- 172.21.15.167:0/1248727001 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:65 5.15 5:a8000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x55bc13ca46d0 con 0x55bc13ca5da0 2023-10-05T19:28:29.932 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.926+0000 7fa6e8d20700 1 -- 172.21.15.167:0/1248727001 <== osd.3 v2:172.21.15.167:6808/30036 17 ==== osd_op_reply(65 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fa6e0018c60 con 0x55bc13ca5da0 2023-10-05T19:28:29.932 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.926+0000 7fa6e8d20700 1 -- 172.21.15.167:0/1248727001 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:66 5.d 5:b0000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x55bc13ca46d0 con 0x7fa6e001a740 2023-10-05T19:28:29.932 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.926+0000 7fa6e851f700 1 -- 172.21.15.167:0/1248727001 <== osd.2 v2:172.21.15.167:6816/30037 10 ==== osd_op_reply(66 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fa6d0006070 con 0x7fa6e001a740 2023-10-05T19:28:29.932 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.926+0000 7fa6e851f700 1 -- 172.21.15.167:0/1248727001 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:67 5.1d 5:b8000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x55bc13ca46d0 con 0x55bc13c71740 2023-10-05T19:28:29.933 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.926+0000 7fa6e9521700 1 -- 172.21.15.167:0/1248727001 <== osd.1 v2:172.21.15.167:6800/30035 24 ==== osd_op_reply(67 [pgnls start_epoch 30 out=107b] v30'2 uv2 ondisk = 1) v8 ==== 144+0+107 (crc 0 0 0) 0x7fa6d802f080 con 0x55bc13c71740 2023-10-05T19:28:29.933 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.926+0000 7fa6e9521700 1 -- 172.21.15.167:0/1248727001 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:68 5.3 5:c0000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x55bc13ca46d0 con 0x7fa6d000d8e0 2023-10-05T19:28:29.934 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.930+0000 7fa6e9521700 1 -- 172.21.15.167:0/1248727001 <== osd.0 v2:172.21.15.118:6800/27690 17 ==== osd_op_reply(68 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fa6d802f080 con 0x7fa6d000d8e0 2023-10-05T19:28:29.934 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.930+0000 7fa6e9521700 1 -- 172.21.15.167:0/1248727001 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:69 5.13 5:c8000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x55bc13ca46d0 con 0x55bc13ca5da0 2023-10-05T19:28:29.935 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.930+0000 7fa6e8d20700 1 -- 172.21.15.167:0/1248727001 <== osd.3 v2:172.21.15.167:6808/30036 18 ==== osd_op_reply(69 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fa6e0018c60 con 0x55bc13ca5da0 2023-10-05T19:28:29.935 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.930+0000 7fa6e8d20700 1 -- 172.21.15.167:0/1248727001 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:70 5.b 5:d0000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x55bc13ca46d0 con 0x7fa6d000d8e0 2023-10-05T19:28:29.936 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.930+0000 7fa6e9521700 1 -- 172.21.15.167:0/1248727001 <== osd.0 v2:172.21.15.118:6800/27690 18 ==== osd_op_reply(70 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fa6d802f080 con 0x7fa6d000d8e0 2023-10-05T19:28:29.936 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.930+0000 7fa6e9521700 1 -- 172.21.15.167:0/1248727001 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:71 5.1b 5:d8000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x55bc13ca46d0 con 0x7fa6e001a740 2023-10-05T19:28:29.937 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.930+0000 7fa6e851f700 1 -- 172.21.15.167:0/1248727001 <== osd.2 v2:172.21.15.167:6816/30037 11 ==== osd_op_reply(71 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fa6d0006070 con 0x7fa6e001a740 2023-10-05T19:28:29.937 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.930+0000 7fa6e851f700 1 -- 172.21.15.167:0/1248727001 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:72 5.7 5:e0000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x55bc13ca46d0 con 0x55bc13ca5da0 2023-10-05T19:28:29.938 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.934+0000 7fa6e8d20700 1 -- 172.21.15.167:0/1248727001 <== osd.3 v2:172.21.15.167:6808/30036 19 ==== osd_op_reply(72 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fa6e0018c60 con 0x55bc13ca5da0 2023-10-05T19:28:29.938 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.934+0000 7fa6e8d20700 1 -- 172.21.15.167:0/1248727001 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:73 5.17 5:e8000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x55bc13ca46d0 con 0x7fa6d000d8e0 2023-10-05T19:28:29.939 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.934+0000 7fa6e9521700 1 -- 172.21.15.167:0/1248727001 <== osd.0 v2:172.21.15.118:6800/27690 19 ==== osd_op_reply(73 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fa6d802f080 con 0x7fa6d000d8e0 2023-10-05T19:28:29.939 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.934+0000 7fa6e9521700 1 -- 172.21.15.167:0/1248727001 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:74 5.f 5:f0000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x55bc13ca46d0 con 0x55bc13c71740 2023-10-05T19:28:29.940 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.934+0000 7fa6e9521700 1 -- 172.21.15.167:0/1248727001 <== osd.1 v2:172.21.15.167:6800/30035 25 ==== osd_op_reply(74 [pgnls start_epoch 30 out=115b] v23'2 uv2 ondisk = 1) v8 ==== 144+0+115 (crc 0 0 0) 0x7fa6d802f080 con 0x55bc13c71740 2023-10-05T19:28:29.940 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.934+0000 7fa6e9521700 1 -- 172.21.15.167:0/1248727001 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:75 5.1f 5:f8000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x55bc13ca46d0 con 0x7fa6e001a740 2023-10-05T19:28:29.941 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.934+0000 7fa6e851f700 1 -- 172.21.15.167:0/1248727001 <== osd.2 v2:172.21.15.167:6816/30037 12 ==== osd_op_reply(75 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fa6d0006070 con 0x7fa6e001a740 2023-10-05T19:28:29.941 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.934+0000 7fa6eb1ebe40 20 RGWRados::pool_iterate: got zonegroup_info.4f1f719d-9648-417b-b10c-e5d939825419 2023-10-05T19:28:29.941 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.934+0000 7fa6eb1ebe40 20 RGWRados::pool_iterate: got zone_info.9a9df965-6612-45d4-a45f-6e3c889bacc2 2023-10-05T19:28:29.941 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.934+0000 7fa6eb1ebe40 20 RGWRados::pool_iterate: got zone_names.default 2023-10-05T19:28:29.941 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.934+0000 7fa6eb1ebe40 20 RGWRados::pool_iterate: got zonegroups_names.default 2023-10-05T19:28:29.941 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.934+0000 7fa6eb1ebe40 20 rados->read ofs=0 len=0 2023-10-05T19:28:29.941 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.934+0000 7fa6eb1ebe40 1 -- 172.21.15.167:0/1248727001 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:76 5.f 5:f43b6ece:::zone_names.default:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x55bc13ca8eb0 con 0x55bc13c71740 2023-10-05T19:28:29.942 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.934+0000 7fa6e9521700 1 -- 172.21.15.167:0/1248727001 <== osd.1 v2:172.21.15.167:6800/30035 26 ==== osd_op_reply(76 zone_names.default [read 0~46 out=46b] v0'0 uv1 ondisk = 0) v8 ==== 162+0+46 (crc 0 0 0) 0x7fa6d802f080 con 0x55bc13c71740 2023-10-05T19:28:29.943 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.934+0000 7fa6eb1ebe40 20 rados_obj.operate() r=0 bl.length=46 2023-10-05T19:28:29.943 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.934+0000 7fa6eb1ebe40 20 rados->read ofs=0 len=0 2023-10-05T19:28:29.943 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.934+0000 7fa6eb1ebe40 1 -- 172.21.15.167:0/1248727001 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:77 5.1d 5:b8374eb4:::zone_info.9a9df965-6612-45d4-a45f-6e3c889bacc2:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x55bc13ca9260 con 0x55bc13c71740 2023-10-05T19:28:29.943 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.938+0000 7fa6e9521700 1 -- 172.21.15.167:0/1248727001 <== osd.1 v2:172.21.15.167:6800/30035 27 ==== osd_op_reply(77 zone_info.9a9df965-6612-45d4-a45f-6e3c889bacc2 [read 0~874 out=874b] v0'0 uv2 ondisk = 0) v8 ==== 190+0+874 (crc 0 0 0) 0x7fa6d802f080 con 0x55bc13c71740 2023-10-05T19:28:29.943 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.938+0000 7fa6eb1ebe40 20 rados_obj.operate() r=0 bl.length=874 2023-10-05T19:28:29.943 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.938+0000 7fa6eb1ebe40 20 rados->read ofs=0 len=0 2023-10-05T19:28:29.943 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.938+0000 7fa6eb1ebe40 1 -- 172.21.15.167:0/1248727001 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:78 5.f 5:f4c53578:::zonegroups_names.default:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x55bc13ca9750 con 0x55bc13c71740 2023-10-05T19:28:29.944 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.938+0000 7fa6e9521700 1 -- 172.21.15.167:0/1248727001 <== osd.1 v2:172.21.15.167:6800/30035 28 ==== osd_op_reply(78 zonegroups_names.default [read 0~46 out=46b] v0'0 uv2 ondisk = 0) v8 ==== 168+0+46 (crc 0 0 0) 0x7fa6d802f080 con 0x55bc13c71740 2023-10-05T19:28:29.944 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.938+0000 7fa6eb1ebe40 20 rados_obj.operate() r=0 bl.length=46 2023-10-05T19:28:29.944 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.938+0000 7fa6eb1ebe40 20 rados->read ofs=0 len=0 2023-10-05T19:28:29.944 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.938+0000 7fa6eb1ebe40 1 -- 172.21.15.167:0/1248727001 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:79 5.c 5:30d299a5:::zonegroup_info.4f1f719d-9648-417b-b10c-e5d939825419:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x55bc13ca9750 con 0x7fa6d000d8e0 2023-10-05T19:28:29.945 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.938+0000 7fa6e9521700 1 -- 172.21.15.167:0/1248727001 <== osd.0 v2:172.21.15.118:6800/27690 20 ==== osd_op_reply(79 zonegroup_info.4f1f719d-9648-417b-b10c-e5d939825419 [read 0~362 out=362b] v0'0 uv1 ondisk = 0) v8 ==== 195+0+362 (crc 0 0 0) 0x7fa6d802f080 con 0x7fa6d000d8e0 2023-10-05T19:28:29.945 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.938+0000 7fa6eb1ebe40 20 rados_obj.operate() r=0 bl.length=362 2023-10-05T19:28:29.946 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.938+0000 7fa6eb1ebe40 20 rados->read ofs=0 len=0 2023-10-05T19:28:29.946 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.938+0000 7fa6eb1ebe40 1 -- 172.21.15.167:0/1248727001 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:80 5.3 5:c0ad8cbd:::region_map:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x55bc13ca9850 con 0x7fa6d000d8e0 2023-10-05T19:28:29.946 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.938+0000 7fa6e9521700 1 -- 172.21.15.167:0/1248727001 <== osd.0 v2:172.21.15.118:6800/27690 21 ==== osd_op_reply(80 region_map [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 ==== 154+0+0 (crc 0 0 0) 0x7fa6d8050040 con 0x7fa6d000d8e0 2023-10-05T19:28:29.946 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.938+0000 7fa6eb1ebe40 20 rados_obj.operate() r=-2 bl.length=0 2023-10-05T19:28:29.946 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.938+0000 7fa6eb1ebe40 20 zone default found 2023-10-05T19:28:29.946 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.938+0000 7fa6eb1ebe40 4 Realm: () 2023-10-05T19:28:29.946 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.938+0000 7fa6eb1ebe40 4 ZoneGroup: default (4f1f719d-9648-417b-b10c-e5d939825419) 2023-10-05T19:28:29.946 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.938+0000 7fa6eb1ebe40 4 Zone: default (9a9df965-6612-45d4-a45f-6e3c889bacc2) 2023-10-05T19:28:29.946 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.938+0000 7fa6eb1ebe40 10 cannot find current period zonegroup using local zonegroup configuration 2023-10-05T19:28:29.946 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.938+0000 7fa6eb1ebe40 20 zonegroup default 2023-10-05T19:28:29.947 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.938+0000 7fa6eb1ebe40 20 rados->read ofs=0 len=0 2023-10-05T19:28:29.947 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.938+0000 7fa6eb1ebe40 1 -- 172.21.15.167:0/1248727001 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:81 5.3 5:c52100b6:::period_config.default:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x55bc13ca9850 con 0x7fa6d000d8e0 2023-10-05T19:28:29.948 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.942+0000 7fa6e9521700 1 -- 172.21.15.167:0/1248727001 <== osd.0 v2:172.21.15.118:6800/27690 22 ==== osd_op_reply(81 period_config.default [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 ==== 165+0+0 (crc 0 0 0) 0x7fa6d8050040 con 0x7fa6d000d8e0 2023-10-05T19:28:29.948 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.942+0000 7fa6eb1ebe40 20 rados_obj.operate() r=-2 bl.length=0 2023-10-05T19:28:29.948 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.942+0000 7fa6eb1ebe40 20 rados->read ofs=0 len=0 2023-10-05T19:28:29.948 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.942+0000 7fa6eb1ebe40 1 -- 172.21.15.167:0/1248727001 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:82 6.2 6:4347d321:::bucket.sync-source-hints.:head [call version.read in=11b,read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x55bc13cab1d0 con 0x55bc13ca5da0 2023-10-05T19:28:29.948 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.942+0000 7fa6e8d20700 1 -- 172.21.15.167:0/1248727001 <== osd.3 v2:172.21.15.167:6808/30036 20 ==== osd_op_reply(82 bucket.sync-source-hints. [call,read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 ==== 211+0+0 (crc 0 0 0) 0x7fa6e0018c60 con 0x55bc13ca5da0 2023-10-05T19:28:29.949 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.942+0000 7fa6eb1ebe40 20 rados_obj.operate() r=-2 bl.length=0 2023-10-05T19:28:29.949 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.942+0000 7fa6eb1ebe40 20 rados->read ofs=0 len=0 2023-10-05T19:28:29.949 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.942+0000 7fa6eb1ebe40 1 -- 172.21.15.167:0/1248727001 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:83 6.b 6:d467b91b:::bucket.sync-target-hints.:head [call version.read in=11b,read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x55bc13cab580 con 0x55bc13ca5da0 2023-10-05T19:28:29.949 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.942+0000 7fa6e8d20700 1 -- 172.21.15.167:0/1248727001 <== osd.3 v2:172.21.15.167:6808/30036 21 ==== osd_op_reply(83 bucket.sync-target-hints. [call,read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 ==== 211+0+0 (crc 0 0 0) 0x7fa6e005e040 con 0x55bc13ca5da0 2023-10-05T19:28:29.949 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.942+0000 7fa6eb1ebe40 20 rados_obj.operate() r=-2 bl.length=0 2023-10-05T19:28:29.949 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.942+0000 7fa6eb1ebe40 20 started sync module instance, tier type = 2023-10-05T19:28:29.949 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.942+0000 7fa6eb1ebe40 20 started zone id=9a9df965-6612-45d4-a45f-6e3c889bacc2 (name=default) with tier type = 2023-10-05T19:28:29.949 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.942+0000 7fa6eb1ebe40 1 -- 172.21.15.167:0/1248727001 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:84 7.1f 7:f95f44c2:::notify.0:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x55bc13caa890 con 0x7fa6e001a740 2023-10-05T19:28:29.951 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.946+0000 7fa6e851f700 1 -- 172.21.15.167:0/1248727001 <== osd.2 v2:172.21.15.167:6816/30037 13 ==== osd_op_reply(84 notify.0 [create] v30'81 uv81 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7fa6d0006070 con 0x7fa6e001a740 2023-10-05T19:28:29.951 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.946+0000 7fa6eb1ebe40 1 -- 172.21.15.167:0/1248727001 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:85 7.1f 7:f95f44c2:::notify.0:head [watch watch cookie 94266274262208] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x55bc13caa4c0 con 0x7fa6e001a740 2023-10-05T19:28:29.951 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.946+0000 7fa6eb1ebe40 1 -- 172.21.15.167:0/1248727001 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:86 7.0 7:05bf5b68:::notify.1:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x55bc13cac350 con 0x55bc13c71740 2023-10-05T19:28:29.953 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.946+0000 7fa6e9521700 1 -- 172.21.15.167:0/1248727001 <== osd.1 v2:172.21.15.167:6800/30035 29 ==== osd_op_reply(86 notify.1 [create] v30'81 uv81 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7fa6d8050040 con 0x55bc13c71740 2023-10-05T19:28:29.954 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.946+0000 7fa6eb1ebe40 1 -- 172.21.15.167:0/1248727001 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:87 7.0 7:05bf5b68:::notify.1:head [watch watch cookie 94266274266656] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x55bc13cacc40 con 0x55bc13c71740 2023-10-05T19:28:29.954 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.946+0000 7fa6eb1ebe40 1 -- 172.21.15.167:0/1248727001 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:88 7.15 7:a93a5511:::notify.2:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x55bc13cad4e0 con 0x55bc13ca5da0 2023-10-05T19:28:29.954 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.950+0000 7fa6e851f700 1 -- 172.21.15.167:0/1248727001 <== osd.2 v2:172.21.15.167:6816/30037 14 ==== osd_op_reply(85 notify.0 [watch watch cookie 94266274262208] v30'82 uv81 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7fa6d0006070 con 0x7fa6e001a740 2023-10-05T19:28:29.956 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.950+0000 7fa6e9521700 1 -- 172.21.15.167:0/1248727001 <== osd.1 v2:172.21.15.167:6800/30035 30 ==== osd_op_reply(87 notify.1 [watch watch cookie 94266274266656] v30'82 uv81 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7fa6d8050040 con 0x55bc13c71740 2023-10-05T19:28:29.957 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.950+0000 7fa6e8d20700 1 -- 172.21.15.167:0/1248727001 <== osd.3 v2:172.21.15.167:6808/30036 22 ==== osd_op_reply(88 notify.2 [create] v30'81 uv81 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7fa6e005e040 con 0x55bc13ca5da0 2023-10-05T19:28:29.957 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.950+0000 7fa6eb1ebe40 1 -- 172.21.15.167:0/1248727001 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:89 7.15 7:a93a5511:::notify.2:head [watch watch cookie 94266274269008] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x55bc13caba50 con 0x55bc13ca5da0 2023-10-05T19:28:29.957 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.950+0000 7fa6eb1ebe40 1 -- 172.21.15.167:0/1248727001 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:90 7.e 7:7759931f:::notify.3:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x55bc13cadd80 con 0x7fa6e001a740 2023-10-05T19:28:29.960 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.954+0000 7fa6e8d20700 1 -- 172.21.15.167:0/1248727001 <== osd.3 v2:172.21.15.167:6808/30036 23 ==== osd_op_reply(89 notify.2 [watch watch cookie 94266274269008] v30'82 uv81 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7fa6e005e040 con 0x55bc13ca5da0 2023-10-05T19:28:29.960 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.954+0000 7fa6e851f700 1 -- 172.21.15.167:0/1248727001 <== osd.2 v2:172.21.15.167:6816/30037 15 ==== osd_op_reply(90 notify.3 [create] v30'81 uv81 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7fa6d0006070 con 0x7fa6e001a740 2023-10-05T19:28:29.960 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.954+0000 7fa6eb1ebe40 1 -- 172.21.15.167:0/1248727001 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:91 7.e 7:7759931f:::notify.3:head [watch watch cookie 94266274273424] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x55bc13cae130 con 0x7fa6e001a740 2023-10-05T19:28:29.960 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.954+0000 7fa6eb1ebe40 1 -- 172.21.15.167:0/1248727001 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:92 7.d 7:b4812045:::notify.4:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x55bc13caec40 con 0x55bc13c71740 2023-10-05T19:28:29.962 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.958+0000 7fa6e851f700 1 -- 172.21.15.167:0/1248727001 <== osd.2 v2:172.21.15.167:6816/30037 16 ==== osd_op_reply(91 notify.3 [watch watch cookie 94266274273424] v30'82 uv81 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7fa6d0006070 con 0x7fa6e001a740 2023-10-05T19:28:29.963 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.958+0000 7fa6e9521700 1 -- 172.21.15.167:0/1248727001 <== osd.1 v2:172.21.15.167:6800/30035 31 ==== osd_op_reply(92 notify.4 [create] v30'81 uv81 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7fa6d8050040 con 0x55bc13c71740 2023-10-05T19:28:29.963 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.958+0000 7fa6eb1ebe40 1 -- 172.21.15.167:0/1248727001 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:93 7.d 7:b4812045:::notify.4:head [watch watch cookie 94266274277136] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x55bc13caf040 con 0x55bc13c71740 2023-10-05T19:28:29.963 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.958+0000 7fa6eb1ebe40 1 -- 172.21.15.167:0/1248727001 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:94 7.3 7:c609908c:::notify.5:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x55bc13cafa50 con 0x7fa6d000d8e0 2023-10-05T19:28:29.965 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.958+0000 7fa6e9521700 1 -- 172.21.15.167:0/1248727001 <== osd.0 v2:172.21.15.118:6800/27690 23 ==== osd_op_reply(94 notify.5 [create] v30'81 uv81 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7fa6d8050040 con 0x7fa6d000d8e0 2023-10-05T19:28:29.965 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.958+0000 7fa6eb1ebe40 1 -- 172.21.15.167:0/1248727001 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:95 7.3 7:c609908c:::notify.5:head [watch watch cookie 94266274280800] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x55bc13caf9b0 con 0x7fa6d000d8e0 2023-10-05T19:28:29.966 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.958+0000 7fa6eb1ebe40 1 -- 172.21.15.167:0/1248727001 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:96 7.14 7:2b04a3e9:::notify.6:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x55bc13cb09a0 con 0x7fa6d000d8e0 2023-10-05T19:28:29.966 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.962+0000 7fa6e9521700 1 -- 172.21.15.167:0/1248727001 <== osd.1 v2:172.21.15.167:6800/30035 32 ==== osd_op_reply(93 notify.4 [watch watch cookie 94266274277136] v30'82 uv81 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7fa6d8050040 con 0x55bc13c71740 2023-10-05T19:28:29.968 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.962+0000 7fa6e9521700 1 -- 172.21.15.167:0/1248727001 <== osd.0 v2:172.21.15.118:6800/27690 24 ==== osd_op_reply(96 notify.6 [create] v30'81 uv81 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7fa6d8050040 con 0x7fa6d000d8e0 2023-10-05T19:28:29.968 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.962+0000 7fa6eb1ebe40 1 -- 172.21.15.167:0/1248727001 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:97 7.14 7:2b04a3e9:::notify.6:head [watch watch cookie 94266274270112] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x55bc13cb09a0 con 0x7fa6d000d8e0 2023-10-05T19:28:29.968 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.962+0000 7fa6eb1ebe40 1 -- 172.21.15.167:0/1248727001 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:98 7.9 7:93e5b521:::notify.7:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x55bc13cb13b0 con 0x7fa6d000d8e0 2023-10-05T19:28:29.968 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.962+0000 7fa6e9521700 1 -- 172.21.15.167:0/1248727001 <== osd.0 v2:172.21.15.118:6800/27690 25 ==== osd_op_reply(95 notify.5 [watch watch cookie 94266274280800] v30'82 uv81 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7fa6d8050040 con 0x7fa6d000d8e0 2023-10-05T19:28:29.970 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.966+0000 7fa6e9521700 1 -- 172.21.15.167:0/1248727001 <== osd.0 v2:172.21.15.118:6800/27690 26 ==== osd_op_reply(97 notify.6 [watch watch cookie 94266274270112] v30'82 uv81 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7fa6d8050040 con 0x7fa6d000d8e0 2023-10-05T19:28:29.970 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.966+0000 7fa6e9521700 1 -- 172.21.15.167:0/1248727001 <== osd.0 v2:172.21.15.118:6800/27690 27 ==== osd_op_reply(98 notify.7 [create] v30'81 uv81 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7fa6d8050040 con 0x7fa6d000d8e0 2023-10-05T19:28:29.971 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.966+0000 7fa6eb1ebe40 1 -- 172.21.15.167:0/1248727001 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:99 7.9 7:93e5b521:::notify.7:head [watch watch cookie 94266274283200] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x55bc13cb0ec0 con 0x7fa6d000d8e0 2023-10-05T19:28:29.971 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.966+0000 7fa6eb1ebe40 20 add_watcher() i=0 2023-10-05T19:28:29.971 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.966+0000 7fa6eb1ebe40 20 add_watcher() i=1 2023-10-05T19:28:29.971 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.966+0000 7fa6eb1ebe40 20 add_watcher() i=2 2023-10-05T19:28:29.971 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.966+0000 7fa6eb1ebe40 20 add_watcher() i=3 2023-10-05T19:28:29.971 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.966+0000 7fa6eb1ebe40 20 add_watcher() i=4 2023-10-05T19:28:29.971 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.966+0000 7fa6eb1ebe40 20 add_watcher() i=5 2023-10-05T19:28:29.971 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.966+0000 7fa6eb1ebe40 20 add_watcher() i=6 2023-10-05T19:28:29.973 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.966+0000 7fa6e9521700 1 -- 172.21.15.167:0/1248727001 <== osd.0 v2:172.21.15.118:6800/27690 28 ==== osd_op_reply(99 notify.7 [watch watch cookie 94266274283200] v30'82 uv81 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7fa6d8050040 con 0x7fa6d000d8e0 2023-10-05T19:28:29.973 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.966+0000 7fa6eb1ebe40 20 add_watcher() i=7 2023-10-05T19:28:29.973 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.966+0000 7fa6eb1ebe40 2 all 8 watchers are set, enabling cache 2023-10-05T19:28:29.973 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.966+0000 7fa6eb1ebe40 1 -- 172.21.15.167:0/1248727001 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:100 6.4 6:22d26bf9:::data_loggenerations_metadata:head [call version.check_conds in=74b,call version.read in=11b,read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x55bc13cb3f20 con 0x55bc13c71740 2023-10-05T19:28:29.974 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.966+0000 7fa6e9521700 1 -- 172.21.15.167:0/1248727001 <== osd.1 v2:172.21.15.167:6800/30035 33 ==== osd_op_reply(100 data_loggenerations_metadata [call,call out=48b,read 0~28 out=28b] v0'0 uv1 ondisk = 0) v8 ==== 256+0+76 (crc 0 0 0) 0x7fa6d8050040 con 0x55bc13c71740 2023-10-05T19:28:29.974 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.966+0000 7fa6eb1ebe40 1 -- 172.21.15.167:0/1248727001 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:101 6.4 6:22d26bf9:::data_loggenerations_metadata:head [watch watch cookie 94266274284720] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x55bc13cb13b0 con 0x55bc13c71740 2023-10-05T19:28:29.976 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.970+0000 7fa6e9521700 1 -- 172.21.15.167:0/1248727001 <== osd.1 v2:172.21.15.167:6800/30035 34 ==== osd_op_reply(101 data_loggenerations_metadata [watch watch cookie 94266274284720] v30'88 uv1 ondisk = 0) v8 ==== 172+0+0 (crc 0 0 0) 0x7fa6d8050040 con 0x55bc13c71740 2023-10-05T19:28:29.976 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.970+0000 7fa6eb1ebe40 20 check_secure_mon_conn(): auth registy supported: methods=[2] modes=[2,1] 2023-10-05T19:28:29.976 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.970+0000 7fa6eb1ebe40 20 check_secure_mon_conn(): mode 1 is insecure 2023-10-05T19:28:29.977 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.970+0000 7fa6eb1ebe40 1 Processor -- start 2023-10-05T19:28:29.977 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.970+0000 7fa6ba7e4700 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2023-10-05T19:28:29.977 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.970+0000 7fa6eb1ebe40 1 -- start start 2023-10-05T19:28:29.977 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.970+0000 7fa6eb1ebe40 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55bc13cbe400 0x55bc13cbe7d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-05T19:28:29.977 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.970+0000 7fa6eb1ebe40 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x55bc13cbed10 con 0x55bc13cbe400 2023-10-05T19:28:29.978 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.970+0000 7fa6e851f700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55bc13cbe400 0x55bc13cbe7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-05T19:28:29.978 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.970+0000 7fa6e851f700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55bc13cbe400 0x55bc13cbe7d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.118:3300/0 says I am v2:172.21.15.167:58004/0 (socket says 172.21.15.167:58004) 2023-10-05T19:28:29.978 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.970+0000 7fa6e851f700 1 -- 172.21.15.167:0/2461480057 learned_addr learned my addr 172.21.15.167:0/2461480057 (peer_addr_for_me v2:172.21.15.167:0/0) 2023-10-05T19:28:29.978 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.974+0000 7fa6e851f700 1 -- 172.21.15.167:0/2461480057 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55bc13cbee50 con 0x55bc13cbe400 2023-10-05T19:28:29.979 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.974+0000 7fa6e851f700 1 --2- 172.21.15.167:0/2461480057 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55bc13cbe400 0x55bc13cbe7d0 secure :-1 s=READY pgs=300 cs=0 l=1 rev1=1 crypto rx=0x7fa6e0018c10 tx=0x7fa6d0006000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=86fb8f3b3f04ee0e server_cookie=0 in_seq=0 out_seq=0 2023-10-05T19:28:29.979 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.974+0000 7fa6b9fe3700 1 -- 172.21.15.167:0/2461480057 <== mon.0 v2:172.21.15.118:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fa6d0023040 con 0x55bc13cbe400 2023-10-05T19:28:29.979 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.974+0000 7fa6b9fe3700 1 -- 172.21.15.167:0/2461480057 <== mon.0 v2:172.21.15.118:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa6d0024040 con 0x55bc13cbe400 2023-10-05T19:28:29.979 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.974+0000 7fa6b9fe3700 1 -- 172.21.15.167:0/2461480057 <== mon.0 v2:172.21.15.118:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fa6d0013040 con 0x55bc13cbe400 2023-10-05T19:28:29.980 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.974+0000 7fa6eb1ebe40 1 -- 172.21.15.167:0/2461480057 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55bc13cbe400 msgr2=0x55bc13cbe7d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-05T19:28:29.981 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.974+0000 7fa6eb1ebe40 1 --2- 172.21.15.167:0/2461480057 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55bc13cbe400 0x55bc13cbe7d0 secure :-1 s=READY pgs=300 cs=0 l=1 rev1=1 crypto rx=0x7fa6e0018c10 tx=0x7fa6d0006000 comp rx=0 tx=0).stop 2023-10-05T19:28:29.981 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.974+0000 7fa6eb1ebe40 1 -- 172.21.15.167:0/2461480057 shutdown_connections 2023-10-05T19:28:29.981 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.974+0000 7fa6eb1ebe40 1 --2- 172.21.15.167:0/2461480057 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55bc13cbe400 0x55bc13cbe7d0 unknown :-1 s=CLOSED pgs=300 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-05T19:28:29.981 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.974+0000 7fa6eb1ebe40 1 -- 172.21.15.167:0/2461480057 >> 172.21.15.167:0/2461480057 conn(0x55bc13cb97c0 msgr2=0x55bc13cb7970 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-05T19:28:29.981 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.974+0000 7fa6eb1ebe40 1 -- 172.21.15.167:0/2461480057 shutdown_connections 2023-10-05T19:28:29.981 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.974+0000 7fa6eb1ebe40 1 -- 172.21.15.167:0/2461480057 wait complete. 2023-10-05T19:28:29.982 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.974+0000 7fa6eb1ebe40 1 Processor -- start 2023-10-05T19:28:29.982 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.974+0000 7fa6eb1ebe40 1 -- start start 2023-10-05T19:28:29.982 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.974+0000 7fa6eb1ebe40 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55bc13cbe400 0x55bc13cc3e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-05T19:28:29.982 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.974+0000 7fa6eb1ebe40 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x55bc13cc4e70 con 0x55bc13cbe400 2023-10-05T19:28:29.982 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.978+0000 7fa6e851f700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55bc13cbe400 0x55bc13cc3e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-05T19:28:29.983 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.978+0000 7fa6e851f700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55bc13cbe400 0x55bc13cc3e70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.118:3300/0 says I am v2:172.21.15.167:58020/0 (socket says 172.21.15.167:58020) 2023-10-05T19:28:29.983 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.978+0000 7fa6e851f700 1 -- 172.21.15.167:0/4273665405 learned_addr learned my addr 172.21.15.167:0/4273665405 (peer_addr_for_me v2:172.21.15.167:0/0) 2023-10-05T19:28:29.983 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.978+0000 7fa6e851f700 1 -- 172.21.15.167:0/4273665405 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa6d000ed60 con 0x55bc13cbe400 2023-10-05T19:28:29.984 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.978+0000 7fa6e851f700 1 --2- 172.21.15.167:0/4273665405 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55bc13cbe400 0x55bc13cc3e70 secure :-1 s=READY pgs=301 cs=0 l=1 rev1=1 crypto rx=0x55bc13cb84e0 tx=0x7fa6d0030000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-05T19:28:29.984 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.978+0000 7fa62bfff700 1 -- 172.21.15.167:0/4273665405 <== mon.0 v2:172.21.15.118:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fa6d0023040 con 0x55bc13cbe400 2023-10-05T19:28:29.984 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.978+0000 7fa62bfff700 1 -- 172.21.15.167:0/4273665405 <== mon.0 v2:172.21.15.118:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa6d003b5d0 con 0x55bc13cbe400 2023-10-05T19:28:29.984 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.978+0000 7fa6eb1ebe40 1 -- 172.21.15.167:0/4273665405 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x55bc13cc4470 con 0x55bc13cbe400 2023-10-05T19:28:29.984 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.978+0000 7fa62bfff700 1 -- 172.21.15.167:0/4273665405 <== mon.0 v2:172.21.15.118:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fa6d003b760 con 0x55bc13cbe400 2023-10-05T19:28:29.984 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.978+0000 7fa6eb1ebe40 1 -- 172.21.15.167:0/4273665405 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55bc13cc48b0 con 0x55bc13cbe400 2023-10-05T19:28:29.985 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.978+0000 7fa62bfff700 1 -- 172.21.15.167:0/4273665405 <== mon.0 v2:172.21.15.118:3300/0 4 ==== mgrmap(e 5) v1 ==== 40329+0+0 (secure 0 0 0) 0x7fa6d003b8c0 con 0x55bc13cbe400 2023-10-05T19:28:29.985 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.978+0000 7fa62bfff700 1 --2- 172.21.15.167:0/4273665405 >> [v2:172.21.15.167:6824/29889,v1:172.21.15.167:6825/29889] conn(0x7fa630030c00 0x7fa6300330b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-05T19:28:29.985 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.978+0000 7fa62bfff700 1 -- 172.21.15.167:0/4273665405 <== mon.0 v2:172.21.15.118:3300/0 5 ==== osd_map(30..30 src has 1..30) v4 ==== 6057+0+0 (secure 0 0 0) 0x7fa6d004ec60 con 0x55bc13cbe400 2023-10-05T19:28:29.986 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.978+0000 7fa6eb1ebe40 5 note: GC not initialized 2023-10-05T19:28:29.986 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.978+0000 7fa6e9521700 1 --2- 172.21.15.167:0/4273665405 >> [v2:172.21.15.167:6824/29889,v1:172.21.15.167:6825/29889] conn(0x7fa630030c00 0x7fa6300330b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-05T19:28:29.986 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.982+0000 7fa6e9521700 1 --2- 172.21.15.167:0/4273665405 >> [v2:172.21.15.167:6824/29889,v1:172.21.15.167:6825/29889] conn(0x7fa630030c00 0x7fa6300330b0 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x55bc13cb3f20 tx=0x7fa6d8000f90 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-05T19:28:29.988 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.982+0000 7fa6297fa700 20 reqs_thread_entry: start 2023-10-05T19:28:29.988 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.982+0000 7fa622ffd700 20 reqs_thread_entry: start 2023-10-05T19:28:29.988 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.982+0000 7fa6eb1ebe40 20 init_complete bucket index max shards: 11 2023-10-05T19:28:29.989 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.982+0000 7fa6eb1ebe40 10 rgw notify: Started notification manager with: 1 workers 2023-10-05T19:28:29.989 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.982+0000 7fa6217fa700 20 rgw notify: INFO: next queues processing will happen at: Thu Oct 5 19:29:00 2023 2023-10-05T19:28:29.989 INFO:teuthology.orchestra.run.smithi167.stderr: 2023-10-05T19:28:29.990 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.982+0000 7fa620ff9700 20 reqs_thread_entry: start 2023-10-05T19:28:29.990 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.982+0000 7fa6eb1ebe40 20 get_system_obj_state: rctx=0x7ffc2d9ce9c8 obj=default.rgw.meta:users.uid:foo state=0x55bc13b4a0b0 s->prefetch_data=0 2023-10-05T19:28:29.990 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.982+0000 7fa6eb1ebe40 1 -- 172.21.15.167:0/1248727001 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:102 8.18 8:1e6ca5ed:users.uid::foo:head [call version.read in=11b,getxattrs,stat] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x55bc13d2c6c0 con 0x55bc13c71740 2023-10-05T19:28:29.990 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.986+0000 7fa6e9521700 1 -- 172.21.15.167:0/1248727001 <== osd.1 v2:172.21.15.167:6800/30035 35 ==== osd_op_reply(102 foo [call out=48b,getxattrs out=75b,stat out=16b] v0'0 uv11 ondisk = 0) v8 ==== 231+0+139 (crc 0 0 0) 0x7fa6d8050040 con 0x55bc13c71740 2023-10-05T19:28:29.990 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.986+0000 7fa6eb1ebe40 20 get_system_obj_state: s->obj_tag was set empty 2023-10-05T19:28:29.990 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.986+0000 7fa6eb1ebe40 20 Read xattr: user.rgw.idtag 2023-10-05T19:28:29.991 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.986+0000 7fa6eb1ebe40 20 rados->read ofs=0 len=0 2023-10-05T19:28:29.991 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.986+0000 7fa6eb1ebe40 1 -- 172.21.15.167:0/1248727001 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:103 8.18 8:1e6ca5ed:users.uid::foo:head [call version.check_conds in=122b,call version.read in=11b,read 0~0,getxattrs] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x55bc13d2f150 con 0x55bc13c71740 2023-10-05T19:28:29.991 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.986+0000 7fa6e9521700 1 -- 172.21.15.167:0/1248727001 <== osd.1 v2:172.21.15.167:6800/30035 36 ==== osd_op_reply(103 foo [call,call out=48b,read 0~357 out=357b,getxattrs out=75b] v0'0 uv11 ondisk = 0) v8 ==== 273+0+480 (crc 0 0 0) 0x7fa6d8050040 con 0x55bc13c71740 2023-10-05T19:28:29.991 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.986+0000 7fa6eb1ebe40 20 rados_obj.operate() r=0 bl.length=357 2023-10-05T19:28:29.992 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.986+0000 7fa6eb1ebe40 1 -- 172.21.15.167:0/1248727001 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:104 8.9 8:96d98480:users.uid::foo.buckets:head [call user.list_buckets in=34b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x55bc13d2e140 con 0x7fa6e001a740 2023-10-05T19:28:29.992 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.986+0000 7fa6e851f700 1 -- 172.21.15.167:0/1248727001 <== osd.2 v2:172.21.15.167:6816/30037 17 ==== osd_op_reply(104 foo.buckets [call] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 ==== 155+0+0 (crc 0 0 0) 0x7fa6d0006070 con 0x7fa6e001a740 2023-10-05T19:28:29.992 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.986+0000 7fa6eb1ebe40 1 -- 172.21.15.167:0/1248727001 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:105 7.1f 7:f95f44c2:::notify.0:head [watch unwatch cookie 94266274262208] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x55bc13ceaab0 con 0x7fa6e001a740 2023-10-05T19:28:29.995 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.990+0000 7fa6e851f700 1 -- 172.21.15.167:0/1248727001 <== osd.2 v2:172.21.15.167:6816/30037 18 ==== osd_op_reply(105 notify.0 [watch unwatch cookie 94266274262208] v30'83 uv81 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7fa6d0006070 con 0x7fa6e001a740 2023-10-05T19:28:29.995 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.990+0000 7fa6eb1ebe40 20 remove_watcher() i=0 2023-10-05T19:28:29.995 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.990+0000 7fa6eb1ebe40 2 removed watcher, disabling cache 2023-10-05T19:28:29.995 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.990+0000 7fa6eb1ebe40 1 -- 172.21.15.167:0/1248727001 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:106 7.0 7:05bf5b68:::notify.1:head [watch unwatch cookie 94266274266656] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x55bc13caacc0 con 0x55bc13c71740 2023-10-05T19:28:29.998 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.994+0000 7fa6e9521700 1 -- 172.21.15.167:0/1248727001 <== osd.1 v2:172.21.15.167:6800/30035 37 ==== osd_op_reply(106 notify.1 [watch unwatch cookie 94266274266656] v30'83 uv81 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7fa6d8050040 con 0x55bc13c71740 2023-10-05T19:28:29.998 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.994+0000 7fa6eb1ebe40 20 remove_watcher() i=1 2023-10-05T19:28:29.998 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.994+0000 7fa6eb1ebe40 1 -- 172.21.15.167:0/1248727001 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:107 7.15 7:a93a5511:::notify.2:head [watch unwatch cookie 94266274269008] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x55bc13caacc0 con 0x55bc13ca5da0 2023-10-05T19:28:30.001 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.994+0000 7fa6e8d20700 1 -- 172.21.15.167:0/1248727001 <== osd.3 v2:172.21.15.167:6808/30036 24 ==== osd_op_reply(107 notify.2 [watch unwatch cookie 94266274269008] v30'83 uv81 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7fa6e005e040 con 0x55bc13ca5da0 2023-10-05T19:28:30.001 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.994+0000 7fa6eb1ebe40 20 remove_watcher() i=2 2023-10-05T19:28:30.001 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.994+0000 7fa6eb1ebe40 1 -- 172.21.15.167:0/1248727001 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:108 7.e 7:7759931f:::notify.3:head [watch unwatch cookie 94266274273424] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x55bc13cac750 con 0x7fa6e001a740 2023-10-05T19:28:30.004 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.998+0000 7fa6e851f700 1 -- 172.21.15.167:0/1248727001 <== osd.2 v2:172.21.15.167:6816/30037 19 ==== osd_op_reply(108 notify.3 [watch unwatch cookie 94266274273424] v30'83 uv81 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7fa6d0006070 con 0x7fa6e001a740 2023-10-05T19:28:30.004 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.998+0000 7fa6eb1ebe40 20 remove_watcher() i=3 2023-10-05T19:28:30.004 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:29.998+0000 7fa6eb1ebe40 1 -- 172.21.15.167:0/1248727001 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:109 7.d 7:b4812045:::notify.4:head [watch unwatch cookie 94266274277136] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x55bc13cad890 con 0x55bc13c71740 2023-10-05T19:28:30.006 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:30.002+0000 7fa6e9521700 1 -- 172.21.15.167:0/1248727001 <== osd.1 v2:172.21.15.167:6800/30035 38 ==== osd_op_reply(109 notify.4 [watch unwatch cookie 94266274277136] v30'83 uv81 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7fa6d8050040 con 0x55bc13c71740 2023-10-05T19:28:30.007 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:30.002+0000 7fa6eb1ebe40 20 remove_watcher() i=4 2023-10-05T19:28:30.007 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:30.002+0000 7fa6eb1ebe40 1 -- 172.21.15.167:0/1248727001 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:110 7.3 7:c609908c:::notify.5:head [watch unwatch cookie 94266274280800] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x55bc13cad890 con 0x7fa6d000d8e0 2023-10-05T19:28:30.009 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:30.002+0000 7fa6e9521700 1 -- 172.21.15.167:0/1248727001 <== osd.0 v2:172.21.15.118:6800/27690 29 ==== osd_op_reply(110 notify.5 [watch unwatch cookie 94266274280800] v30'83 uv81 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7fa6d8050040 con 0x7fa6d000d8e0 2023-10-05T19:28:30.009 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:30.002+0000 7fa6eb1ebe40 20 remove_watcher() i=5 2023-10-05T19:28:30.010 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:30.002+0000 7fa6eb1ebe40 1 -- 172.21.15.167:0/1248727001 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:111 7.14 7:2b04a3e9:::notify.6:head [watch unwatch cookie 94266274270112] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x55bc13cad890 con 0x7fa6d000d8e0 2023-10-05T19:28:30.012 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:30.006+0000 7fa6e9521700 1 -- 172.21.15.167:0/1248727001 <== osd.0 v2:172.21.15.118:6800/27690 30 ==== osd_op_reply(111 notify.6 [watch unwatch cookie 94266274270112] v30'83 uv81 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7fa6d8050040 con 0x7fa6d000d8e0 2023-10-05T19:28:30.013 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:30.006+0000 7fa6eb1ebe40 20 remove_watcher() i=6 2023-10-05T19:28:30.013 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:30.006+0000 7fa6eb1ebe40 1 -- 172.21.15.167:0/1248727001 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:112 7.9 7:93e5b521:::notify.7:head [watch unwatch cookie 94266274283200] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x55bc13cad890 con 0x7fa6d000d8e0 2023-10-05T19:28:30.015 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:30.010+0000 7fa6e9521700 1 -- 172.21.15.167:0/1248727001 <== osd.0 v2:172.21.15.118:6800/27690 31 ==== osd_op_reply(112 notify.7 [watch unwatch cookie 94266274283200] v30'83 uv81 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7fa6d8050040 con 0x7fa6d000d8e0 2023-10-05T19:28:30.015 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:30.010+0000 7fa6eb1ebe40 20 remove_watcher() i=7 2023-10-05T19:28:30.017 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:30.010+0000 7fa6ba7e4700 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2023-10-05T19:28:30.017 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:30.010+0000 7fa6eb1ebe40 1 -- 172.21.15.167:0/1248727001 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:113 6.4 6:22d26bf9:::data_loggenerations_metadata:head [watch unwatch cookie 94266274284720] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x55bc13cad890 con 0x55bc13c71740 2023-10-05T19:28:30.021 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:30.014+0000 7fa6e9521700 1 -- 172.21.15.167:0/1248727001 <== osd.1 v2:172.21.15.167:6800/30035 39 ==== osd_op_reply(113 data_loggenerations_metadata [watch unwatch cookie 94266274284720] v30'89 uv1 ondisk = 0) v8 ==== 172+0+0 (crc 0 0 0) 0x7fa6d8050040 con 0x55bc13c71740 2023-10-05T19:28:30.021 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:30.014+0000 7fa6eb1ebe40 1 -- 172.21.15.167:0/1248727001 >> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] conn(0x7fa6d000d8e0 msgr2=0x7fa6d000c360 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-05T19:28:30.021 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:30.014+0000 7fa6eb1ebe40 1 --2- 172.21.15.167:0/1248727001 >> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] conn(0x7fa6d000d8e0 0x7fa6d000c360 crc :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-05T19:28:30.021 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:30.014+0000 7fa6eb1ebe40 1 -- 172.21.15.167:0/1248727001 >> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] conn(0x55bc13c71740 msgr2=0x55bc13ca2300 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-05T19:28:30.021 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:30.014+0000 7fa6eb1ebe40 1 --2- 172.21.15.167:0/1248727001 >> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] conn(0x55bc13c71740 0x55bc13ca2300 crc :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-05T19:28:30.021 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:30.014+0000 7fa6eb1ebe40 1 -- 172.21.15.167:0/1248727001 >> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] conn(0x7fa6e001a740 msgr2=0x7fa6e00398d0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-05T19:28:30.021 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:30.014+0000 7fa6eb1ebe40 1 --2- 172.21.15.167:0/1248727001 >> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] conn(0x7fa6e001a740 0x7fa6e00398d0 crc :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-05T19:28:30.021 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:30.014+0000 7fa6eb1ebe40 1 -- 172.21.15.167:0/1248727001 >> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] conn(0x55bc13ca5da0 msgr2=0x55bc13ca8180 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-05T19:28:30.021 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:30.014+0000 7fa6eb1ebe40 1 --2- 172.21.15.167:0/1248727001 >> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] conn(0x55bc13ca5da0 0x55bc13ca8180 crc :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-05T19:28:30.021 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:30.014+0000 7fa6eb1ebe40 1 -- 172.21.15.167:0/1248727001 >> [v2:172.21.15.167:6824/29889,v1:172.21.15.167:6825/29889] conn(0x7fa6c8030c50 msgr2=0x7fa6c8033100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-05T19:28:30.021 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:30.014+0000 7fa6eb1ebe40 1 --2- 172.21.15.167:0/1248727001 >> [v2:172.21.15.167:6824/29889,v1:172.21.15.167:6825/29889] conn(0x7fa6c8030c50 0x7fa6c8033100 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7fa6d00011e0 tx=0x7fa6d00013a0 comp rx=0 tx=0).stop 2023-10-05T19:28:30.022 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:30.014+0000 7fa6eb1ebe40 1 -- 172.21.15.167:0/1248727001 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55bc13c70e30 msgr2=0x55bc13c96990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-05T19:28:30.022 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:30.014+0000 7fa6eb1ebe40 1 --2- 172.21.15.167:0/1248727001 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55bc13c70e30 0x55bc13c96990 secure :-1 s=READY pgs=299 cs=0 l=1 rev1=1 crypto rx=0x7fa6e0005b20 tx=0x7fa6e001d000 comp rx=0 tx=0).stop 2023-10-05T19:28:30.022 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:30.014+0000 7fa6e9521700 1 -- 172.21.15.167:0/1248727001 reap_dead start 2023-10-05T19:28:30.022 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:30.014+0000 7fa6e9521700 1 -- 172.21.15.167:0/1248727001 reap_dead start 2023-10-05T19:28:30.022 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:30.014+0000 7fa6eb1ebe40 1 -- 172.21.15.167:0/1248727001 shutdown_connections 2023-10-05T19:28:30.022 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:30.014+0000 7fa6eb1ebe40 1 -- 172.21.15.167:0/1248727001 >> 172.21.15.167:0/1248727001 conn(0x55bc13c6c850 msgr2=0x55bc13c8a2d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-05T19:28:30.022 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:30.014+0000 7fa6eb1ebe40 1 -- 172.21.15.167:0/1248727001 shutdown_connections 2023-10-05T19:28:30.022 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:30.014+0000 7fa6eb1ebe40 1 -- 172.21.15.167:0/1248727001 wait complete. 2023-10-05T19:28:30.023 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:30.014+0000 7fa6eb1ebe40 1 -- 172.21.15.167:0/4273665405 >> [v2:172.21.15.167:6824/29889,v1:172.21.15.167:6825/29889] conn(0x7fa630030c00 msgr2=0x7fa6300330b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-05T19:28:30.023 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:30.014+0000 7fa6eb1ebe40 1 --2- 172.21.15.167:0/4273665405 >> [v2:172.21.15.167:6824/29889,v1:172.21.15.167:6825/29889] conn(0x7fa630030c00 0x7fa6300330b0 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x55bc13cb3f20 tx=0x7fa6d8000f90 comp rx=0 tx=0).stop 2023-10-05T19:28:30.023 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:30.014+0000 7fa6eb1ebe40 1 -- 172.21.15.167:0/4273665405 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55bc13cbe400 msgr2=0x55bc13cc3e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-05T19:28:30.023 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:30.014+0000 7fa6eb1ebe40 1 --2- 172.21.15.167:0/4273665405 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55bc13cbe400 0x55bc13cc3e70 secure :-1 s=READY pgs=301 cs=0 l=1 rev1=1 crypto rx=0x55bc13cb84e0 tx=0x7fa6d0030000 comp rx=0 tx=0).stop 2023-10-05T19:28:30.023 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:30.014+0000 7fa6eb1ebe40 1 -- 172.21.15.167:0/4273665405 shutdown_connections 2023-10-05T19:28:30.024 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:30.014+0000 7fa6eb1ebe40 1 --2- 172.21.15.167:0/4273665405 >> [v2:172.21.15.167:6824/29889,v1:172.21.15.167:6825/29889] conn(0x7fa630030c00 0x7fa6300330b0 unknown :-1 s=CLOSED pgs=107 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-05T19:28:30.024 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:30.014+0000 7fa6eb1ebe40 1 --2- 172.21.15.167:0/4273665405 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55bc13cbe400 0x55bc13cc3e70 unknown :-1 s=CLOSED pgs=301 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-05T19:28:30.024 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:30.014+0000 7fa6eb1ebe40 1 -- 172.21.15.167:0/4273665405 >> 172.21.15.167:0/4273665405 conn(0x55bc13cb97c0 msgr2=0x55bc13cb78c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-05T19:28:30.024 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:30.014+0000 7fa6eb1ebe40 1 -- 172.21.15.167:0/4273665405 shutdown_connections 2023-10-05T19:28:30.024 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:30.014+0000 7fa6eb1ebe40 1 -- 172.21.15.167:0/4273665405 wait complete. 2023-10-05T19:28:30.024 INFO:teuthology.orchestra.run.smithi167.stdout:[] 2023-10-05T19:28:30.027 DEBUG:tasks.util.rgw: json result: [] 2023-10-05T19:28:30.027 INFO:tasks.util.rgw:rgwadmin: client.0 : ['bucket', 'list', '--uid', 'foo'] 2023-10-05T19:28:30.027 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'ceph', 'bucket', 'list', '--uid', 'foo'] 2023-10-05T19:28:30.027 DEBUG:teuthology.orchestra.run.smithi167:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster ceph bucket list --uid foo 2023-10-05T19:28:30.067 INFO:teuthology.orchestra.run.smithi167.stderr:ignoring --setuser ceph since I am not root 2023-10-05T19:28:30.067 INFO:teuthology.orchestra.run.smithi167.stderr:ignoring --setgroup ceph since I am not root 2023-10-05T19:28:30.068 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:30.062+0000 7f2d06502e40 1 Processor -- start 2023-10-05T19:28:30.068 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:30.062+0000 7f2d06502e40 1 -- start start 2023-10-05T19:28:30.068 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:30.062+0000 7f2d06502e40 1 --2- >> v2:172.21.15.118:3300/0 conn(0x55aeed15ed00 0x55aeed15f0d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-05T19:28:30.068 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:30.062+0000 7f2d06502e40 1 -- --> v1:172.21.15.118:6789/0 -- auth(proto 0 26 bytes epoch 0) v1 -- 0x55aeed0acbd0 con 0x55aeed15f610 2023-10-05T19:28:30.068 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:30.062+0000 7f2d06502e40 1 -- --> v2:172.21.15.118:3300/0 -- mon_getmap magic: 0 v1 -- 0x55aeed069ee0 con 0x55aeed15ed00 2023-10-05T19:28:30.069 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:30.062+0000 7f2d04037700 1 --2- >> v2:172.21.15.118:3300/0 conn(0x55aeed15ed00 0x55aeed15f0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-05T19:28:30.069 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:30.062+0000 7f2d03836700 1 --1- >> v1:172.21.15.118:6789/0 conn(0x55aeed15f610 0x55aeed1600b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.118:6789/0 says I am v1:172.21.15.167:56534/0 (socket says 172.21.15.167:56534) 2023-10-05T19:28:30.069 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:30.062+0000 7f2d03836700 1 -- 172.21.15.167:0/4136903216 learned_addr learned my addr 172.21.15.167:0/4136903216 (peer_addr_for_me v1:172.21.15.167:0/0) 2023-10-05T19:28:30.069 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:30.062+0000 7f2d04037700 1 -- 172.21.15.167:0/4136903216 >> v1:172.21.15.118:6789/0 conn(0x55aeed15f610 legacy=0x55aeed1600b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-05T19:28:30.070 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:30.062+0000 7f2d04037700 1 -- 172.21.15.167:0/4136903216 --> v2:172.21.15.118:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55aeed1607c0 con 0x55aeed15ed00 2023-10-05T19:28:30.071 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:30.062+0000 7f2d04037700 1 --2- 172.21.15.167:0/4136903216 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55aeed15ed00 0x55aeed15f0d0 secure :-1 s=READY pgs=303 cs=0 l=1 rev1=1 crypto rx=0x7f2cfc00a520 tx=0x7f2cfc005d40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=14610f510ee536f8 server_cookie=0 in_seq=0 out_seq=0 2023-10-05T19:28:30.071 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:30.066+0000 7f2d03035700 1 -- 172.21.15.167:0/4136903216 <== mon.0 v2:172.21.15.118:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f2cfc00f590 con 0x55aeed15ed00 2023-10-05T19:28:30.071 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:30.066+0000 7f2d03035700 1 -- 172.21.15.167:0/4136903216 <== mon.0 v2:172.21.15.118:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2cfc00f6f0 con 0x55aeed15ed00 2023-10-05T19:28:30.071 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:30.066+0000 7f2d03035700 1 -- 172.21.15.167:0/4136903216 <== mon.0 v2:172.21.15.118:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f2cfc00f880 con 0x55aeed15ed00 2023-10-05T19:28:30.071 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:30.066+0000 7f2d06502e40 1 -- 172.21.15.167:0/4136903216 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55aeed15ed00 msgr2=0x55aeed15f0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-05T19:28:30.071 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:30.066+0000 7f2d06502e40 1 --2- 172.21.15.167:0/4136903216 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55aeed15ed00 0x55aeed15f0d0 secure :-1 s=READY pgs=303 cs=0 l=1 rev1=1 crypto rx=0x7f2cfc00a520 tx=0x7f2cfc005d40 comp rx=0 tx=0).stop 2023-10-05T19:28:30.071 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:30.066+0000 7f2d06502e40 1 -- 172.21.15.167:0/4136903216 shutdown_connections 2023-10-05T19:28:30.071 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:30.066+0000 7f2d06502e40 1 --2- 172.21.15.167:0/4136903216 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55aeed15ed00 0x55aeed15f0d0 unknown :-1 s=CLOSED pgs=303 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-05T19:28:30.072 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:30.066+0000 7f2d06502e40 1 -- 172.21.15.167:0/4136903216 >> 172.21.15.167:0/4136903216 conn(0x55aeed15a720 msgr2=0x55aeed15cb30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-05T19:28:30.072 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:30.066+0000 7f2d06502e40 1 -- 172.21.15.167:0/4136903216 shutdown_connections 2023-10-05T19:28:30.072 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:30.066+0000 7f2d06502e40 1 -- 172.21.15.167:0/4136903216 wait complete. 2023-10-05T19:28:30.073 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:30.066+0000 7f2d06502e40 1 Processor -- start 2023-10-05T19:28:30.073 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:30.066+0000 7f2d06502e40 1 -- start start 2023-10-05T19:28:30.073 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:30.066+0000 7f2d06502e40 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55aeed15ed00 0x55aeed164aa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-05T19:28:30.073 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:30.066+0000 7f2d06502e40 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x55aeed02de30 con 0x55aeed15ed00 2023-10-05T19:28:30.073 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:30.066+0000 7f2d04037700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55aeed15ed00 0x55aeed164aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-05T19:28:30.073 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:30.066+0000 7f2d04037700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55aeed15ed00 0x55aeed164aa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.118:3300/0 says I am v2:172.21.15.167:58036/0 (socket says 172.21.15.167:58036) 2023-10-05T19:28:30.073 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:30.066+0000 7f2d04037700 1 -- 172.21.15.167:0/318972555 learned_addr learned my addr 172.21.15.167:0/318972555 (peer_addr_for_me v2:172.21.15.167:0/0) 2023-10-05T19:28:30.073 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:30.066+0000 7f2d04037700 1 -- 172.21.15.167:0/318972555 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2cfc00a0e0 con 0x55aeed15ed00 2023-10-05T19:28:30.074 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:30.066+0000 7f2d04037700 1 --2- 172.21.15.167:0/318972555 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55aeed15ed00 0x55aeed164aa0 secure :-1 s=READY pgs=304 cs=0 l=1 rev1=1 crypto rx=0x7f2cfc00a470 tx=0x7f2cfc014000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9d3eb16308e75c74 server_cookie=0 in_seq=0 out_seq=0 2023-10-05T19:28:30.074 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:30.066+0000 7f2d02033700 1 -- 172.21.15.167:0/318972555 <== mon.0 v2:172.21.15.118:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f2cfc00f860 con 0x55aeed15ed00 2023-10-05T19:28:30.074 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:30.066+0000 7f2d02033700 1 -- 172.21.15.167:0/318972555 <== mon.0 v2:172.21.15.118:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2cfc00f9e0 con 0x55aeed15ed00 2023-10-05T19:28:30.074 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:30.066+0000 7f2d02033700 1 -- 172.21.15.167:0/318972555 <== mon.0 v2:172.21.15.118:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f2cfc00fb90 con 0x55aeed15ed00 2023-10-05T19:28:30.075 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:30.070+0000 7f2d06502e40 1 -- 172.21.15.167:0/318972555 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55aeed15ed00 msgr2=0x55aeed164aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-05T19:28:30.075 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:30.070+0000 7f2d06502e40 1 --2- 172.21.15.167:0/318972555 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55aeed15ed00 0x55aeed164aa0 secure :-1 s=READY pgs=304 cs=0 l=1 rev1=1 crypto rx=0x7f2cfc00a470 tx=0x7f2cfc014000 comp rx=0 tx=0).stop 2023-10-05T19:28:30.075 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:30.070+0000 7f2d06502e40 1 -- 172.21.15.167:0/318972555 shutdown_connections 2023-10-05T19:28:30.076 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:30.070+0000 7f2d06502e40 1 --2- 172.21.15.167:0/318972555 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55aeed15ed00 0x55aeed164aa0 unknown :-1 s=CLOSED pgs=304 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-05T19:28:30.076 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:30.070+0000 7f2d06502e40 1 -- 172.21.15.167:0/318972555 >> 172.21.15.167:0/318972555 conn(0x55aeed15a720 msgr2=0x55aeed17aa80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-05T19:28:30.076 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:30.070+0000 7f2d06502e40 1 -- 172.21.15.167:0/318972555 shutdown_connections 2023-10-05T19:28:30.076 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:30.070+0000 7f2d06502e40 1 -- 172.21.15.167:0/318972555 wait complete. 2023-10-05T19:28:30.076 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:30.070+0000 7f2d06502e40 1 Processor -- start 2023-10-05T19:28:30.076 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:30.070+0000 7f2d06502e40 1 -- start start 2023-10-05T19:28:30.077 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:30.070+0000 7f2d06502e40 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55aeed15ed00 0x55aeed187cb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-05T19:28:30.077 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:30.070+0000 7f2d06502e40 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x55aeed000ed0 con 0x55aeed15ed00 2023-10-05T19:28:30.077 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:30.070+0000 7f2d04037700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55aeed15ed00 0x55aeed187cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-05T19:28:30.077 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:30.070+0000 7f2d04037700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55aeed15ed00 0x55aeed187cb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.118:3300/0 says I am v2:172.21.15.167:58042/0 (socket says 172.21.15.167:58042) 2023-10-05T19:28:30.078 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:30.070+0000 7f2d04037700 1 -- 172.21.15.167:0/3084951824 learned_addr learned my addr 172.21.15.167:0/3084951824 (peer_addr_for_me v2:172.21.15.167:0/0) 2023-10-05T19:28:30.078 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:30.070+0000 7f2d04037700 1 -- 172.21.15.167:0/3084951824 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55aeed1607c0 con 0x55aeed15ed00 2023-10-05T19:28:30.078 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:30.074+0000 7f2d04037700 1 --2- 172.21.15.167:0/3084951824 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55aeed15ed00 0x55aeed187cb0 secure :-1 s=READY pgs=305 cs=0 l=1 rev1=1 crypto rx=0x7f2cfc00a470 tx=0x7f2cfc01d000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-05T19:28:30.079 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:30.074+0000 7f2d00830700 1 -- 172.21.15.167:0/3084951824 <== mon.0 v2:172.21.15.118:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f2cfc01ce00 con 0x55aeed15ed00 2023-10-05T19:28:30.079 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:30.074+0000 7f2d00830700 1 -- 172.21.15.167:0/3084951824 <== mon.0 v2:172.21.15.118:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2cfc010630 con 0x55aeed15ed00 2023-10-05T19:28:30.079 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:30.074+0000 7f2d00830700 1 -- 172.21.15.167:0/3084951824 <== mon.0 v2:172.21.15.118:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f2cfc010820 con 0x55aeed15ed00 2023-10-05T19:28:30.079 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:30.074+0000 7f2d06502e40 1 -- 172.21.15.167:0/3084951824 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x55aeed1881f0 con 0x55aeed15ed00 2023-10-05T19:28:30.079 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:30.074+0000 7f2d06502e40 1 -- 172.21.15.167:0/3084951824 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55aeed188630 con 0x55aeed15ed00 2023-10-05T19:28:30.080 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:30.074+0000 7f2d00830700 1 -- 172.21.15.167:0/3084951824 <== mon.0 v2:172.21.15.118:3300/0 4 ==== mgrmap(e 5) v1 ==== 40329+0+0 (secure 0 0 0) 0x7f2cfc010980 con 0x55aeed15ed00 2023-10-05T19:28:30.081 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:30.074+0000 7f2d06502e40 20 rados->read ofs=0 len=0 2023-10-05T19:28:30.081 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:30.074+0000 7f2d00830700 1 --2- 172.21.15.167:0/3084951824 >> [v2:172.21.15.167:6824/29889,v1:172.21.15.167:6825/29889] conn(0x7f2cf0030c50 0x7f2cf0033100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-05T19:28:30.081 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:30.074+0000 7f2d00830700 1 -- 172.21.15.167:0/3084951824 <== mon.0 v2:172.21.15.118:3300/0 5 ==== osd_map(30..30 src has 1..30) v4 ==== 6057+0+0 (secure 0 0 0) 0x7f2cfc02b070 con 0x55aeed15ed00 2023-10-05T19:28:30.081 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:30.074+0000 7f2d03836700 1 --2- 172.21.15.167:0/3084951824 >> [v2:172.21.15.167:6824/29889,v1:172.21.15.167:6825/29889] conn(0x7f2cf0030c50 0x7f2cf0033100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-05T19:28:30.081 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:30.074+0000 7f2d06502e40 1 --2- 172.21.15.167:0/3084951824 >> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] conn(0x55aeed15f610 0x55aeed193710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-05T19:28:30.081 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:30.074+0000 7f2d06502e40 1 -- 172.21.15.167:0/3084951824 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:1 5.12 5:49953fa1:::default.realm:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x55aeed193d80 con 0x55aeed15f610 2023-10-05T19:28:30.082 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:30.074+0000 7f2d04838700 1 --2- 172.21.15.167:0/3084951824 >> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] conn(0x55aeed15f610 0x55aeed193710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-05T19:28:30.082 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:30.078+0000 7f2d03836700 1 --2- 172.21.15.167:0/3084951824 >> [v2:172.21.15.167:6824/29889,v1:172.21.15.167:6825/29889] conn(0x7f2cf0030c50 0x7f2cf0033100 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7f2cf8001e90 tx=0x7f2cf80013b0 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-05T19:28:30.082 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:30.078+0000 7f2d04838700 1 --2- 172.21.15.167:0/3084951824 >> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] conn(0x55aeed15f610 0x55aeed193710 crc :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-05T19:28:30.083 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:30.078+0000 7f2d04838700 1 -- 172.21.15.167:0/3084951824 <== osd.1 v2:172.21.15.167:6800/30035 1 ==== osd_op_reply(1 default.realm [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 ==== 157+0+0 (crc 0 0 0) 0x7f2cf400a500 con 0x55aeed15f610 2023-10-05T19:28:30.083 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:30.078+0000 7f2d06502e40 20 rados_obj.operate() r=-2 bl.length=0 2023-10-05T19:28:30.083 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:30.078+0000 7f2d06502e40 20 realm 2023-10-05T19:28:30.083 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:30.078+0000 7f2d06502e40 20 rados->read ofs=0 len=0 2023-10-05T19:28:30.083 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:30.078+0000 7f2d06502e40 1 -- 172.21.15.167:0/3084951824 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:2 5.12 5:49953fa1:::default.realm:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x55aeed190d70 con 0x55aeed15f610 2023-10-05T19:28:30.084 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:30.078+0000 7f2d04838700 1 -- 172.21.15.167:0/3084951824 <== osd.1 v2:172.21.15.167:6800/30035 2 ==== osd_op_reply(2 default.realm [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 ==== 157+0+0 (crc 0 0 0) 0x7f2cf400a500 con 0x55aeed15f610 2023-10-05T19:28:30.084 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:30.078+0000 7f2d06502e40 20 rados_obj.operate() r=-2 bl.length=0 2023-10-05T19:28:30.084 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:30.078+0000 7f2d06502e40 4 RGWPeriod::init failed to init realm id : (2) No such file or directory 2023-10-05T19:28:30.084 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:30.078+0000 7f2d06502e40 20 rados->read ofs=0 len=0 2023-10-05T19:28:30.084 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:30.078+0000 7f2d06502e40 1 -- 172.21.15.167:0/3084951824 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:3 5.12 5:49953fa1:::default.realm:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x55aeed194860 con 0x55aeed15f610 2023-10-05T19:28:30.085 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:30.078+0000 7f2d04838700 1 -- 172.21.15.167:0/3084951824 <== osd.1 v2:172.21.15.167:6800/30035 3 ==== osd_op_reply(3 default.realm [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 ==== 157+0+0 (crc 0 0 0) 0x7f2cf400a500 con 0x55aeed15f610 2023-10-05T19:28:30.085 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:30.078+0000 7f2d06502e40 20 rados_obj.operate() r=-2 bl.length=0 2023-10-05T19:28:30.085 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:30.078+0000 7f2d06502e40 20 rados->read ofs=0 len=0 2023-10-05T19:28:30.085 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:30.078+0000 7f2d06502e40 1 -- 172.21.15.167:0/3084951824 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:4 5.f 5:f43b6ece:::zone_names.default:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x55aeed194440 con 0x55aeed15f610 2023-10-05T19:28:30.086 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:30.078+0000 7f2d04838700 1 -- 172.21.15.167:0/3084951824 <== osd.1 v2:172.21.15.167:6800/30035 4 ==== osd_op_reply(4 zone_names.default [read 0~46 out=46b] v0'0 uv1 ondisk = 0) v8 ==== 162+0+46 (crc 0 0 0) 0x7f2cf400a500 con 0x55aeed15f610 2023-10-05T19:28:30.086 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:30.078+0000 7f2d06502e40 20 rados_obj.operate() r=0 bl.length=46 2023-10-05T19:28:30.086 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:30.078+0000 7f2d06502e40 20 rados->read ofs=0 len=0 2023-10-05T19:28:30.086 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:30.078+0000 7f2d06502e40 1 -- 172.21.15.167:0/3084951824 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:5 5.1d 5:b8374eb4:::zone_info.9a9df965-6612-45d4-a45f-6e3c889bacc2:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x55aeed1952d0 con 0x55aeed15f610 2023-10-05T19:28:30.086 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:30.078+0000 7f2d04838700 1 -- 172.21.15.167:0/3084951824 <== osd.1 v2:172.21.15.167:6800/30035 5 ==== osd_op_reply(5 zone_info.9a9df965-6612-45d4-a45f-6e3c889bacc2 [read 0~874 out=874b] v0'0 uv2 ondisk = 0) v8 ==== 190+0+874 (crc 0 0 0) 0x7f2cf400a500 con 0x55aeed15f610 2023-10-05T19:28:30.086 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:30.078+0000 7f2d06502e40 20 rados_obj.operate() r=0 bl.length=874 2023-10-05T19:28:30.086 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:30.078+0000 7f2d06502e40 20 searching for the correct realm 2023-10-05T19:28:30.087 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:30.078+0000 7f2d06502e40 1 --2- 172.21.15.167:0/3084951824 >> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] conn(0x55aeed197120 0x55aeed199500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-05T19:28:30.087 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:30.078+0000 7f2d06502e40 1 -- 172.21.15.167:0/3084951824 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:6 5.0 5:00000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x55aeed199a40 con 0x55aeed197120 2023-10-05T19:28:30.087 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:30.078+0000 7f2d04037700 1 --2- 172.21.15.167:0/3084951824 >> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] conn(0x55aeed197120 0x55aeed199500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-05T19:28:30.087 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:30.082+0000 7f2d04037700 1 --2- 172.21.15.167:0/3084951824 >> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] conn(0x55aeed197120 0x55aeed199500 crc :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-05T19:28:30.088 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:30.082+0000 7f2d04037700 1 -- 172.21.15.167:0/3084951824 <== osd.3 v2:172.21.15.167:6808/30036 1 ==== osd_op_reply(6 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f2cfc024920 con 0x55aeed197120 2023-10-05T19:28:30.088 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:30.082+0000 7f2d04037700 1 --2- 172.21.15.167:0/3084951824 >> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] conn(0x7f2cfc037c00 0x7f2cfc013120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-05T19:28:30.088 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:30.082+0000 7f2d04037700 1 -- 172.21.15.167:0/3084951824 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:7 5.10 5:08000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x55aeed199a40 con 0x7f2cfc037c00 2023-10-05T19:28:30.088 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:30.082+0000 7f2d03836700 1 --2- 172.21.15.167:0/3084951824 >> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] conn(0x7f2cfc037c00 0x7f2cfc013120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-05T19:28:30.088 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:30.082+0000 7f2d03836700 1 --2- 172.21.15.167:0/3084951824 >> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] conn(0x7f2cfc037c00 0x7f2cfc013120 crc :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-05T19:28:30.089 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:30.082+0000 7f2d03836700 1 -- 172.21.15.167:0/3084951824 <== osd.2 v2:172.21.15.167:6816/30037 1 ==== osd_op_reply(7 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f2cf8006070 con 0x7f2cfc037c00 2023-10-05T19:28:30.089 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:30.082+0000 7f2d03836700 1 --2- 172.21.15.167:0/3084951824 >> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] conn(0x7f2cf800d8e0 0x7f2cf800c360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-05T19:28:30.089 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:30.082+0000 7f2d03836700 1 -- 172.21.15.167:0/3084951824 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:8 5.8 5:10000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x55aeed199a40 con 0x7f2cf800d8e0 2023-10-05T19:28:30.090 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:30.082+0000 7f2d04838700 1 --2- 172.21.15.167:0/3084951824 >> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] conn(0x7f2cf800d8e0 0x7f2cf800c360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-05T19:28:30.090 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:30.086+0000 7f2d04838700 1 --2- 172.21.15.167:0/3084951824 >> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] conn(0x7f2cf800d8e0 0x7f2cf800c360 crc :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-05T19:28:30.091 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:30.086+0000 7f2d04838700 1 -- 172.21.15.167:0/3084951824 <== osd.0 v2:172.21.15.118:6800/27690 1 ==== osd_op_reply(8 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f2cf4014040 con 0x7f2cf800d8e0 2023-10-05T19:28:30.091 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:30.086+0000 7f2d04838700 1 -- 172.21.15.167:0/3084951824 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:9 5.18 5:18000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x55aeed199a40 con 0x55aeed15f610 2023-10-05T19:28:30.091 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:30.086+0000 7f2d04838700 1 -- 172.21.15.167:0/3084951824 <== osd.1 v2:172.21.15.167:6800/30035 6 ==== osd_op_reply(9 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f2cf4014040 con 0x55aeed15f610 2023-10-05T19:28:30.092 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:30.086+0000 7f2d04838700 1 -- 172.21.15.167:0/3084951824 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:10 5.4 5:20000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x55aeed199a40 con 0x55aeed15f610 2023-10-05T19:28:30.092 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:30.086+0000 7f2d04838700 1 -- 172.21.15.167:0/3084951824 <== osd.1 v2:172.21.15.167:6800/30035 7 ==== osd_op_reply(10 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f2cf4014040 con 0x55aeed15f610 2023-10-05T19:28:30.092 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:30.086+0000 7f2d04838700 1 -- 172.21.15.167:0/3084951824 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:11 5.14 5:28000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x55aeed199a40 con 0x7f2cf800d8e0 2023-10-05T19:28:30.093 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:30.086+0000 7f2d04838700 1 -- 172.21.15.167:0/3084951824 <== osd.0 v2:172.21.15.118:6800/27690 2 ==== osd_op_reply(11 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f2cf4014040 con 0x7f2cf800d8e0 2023-10-05T19:28:30.093 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:30.086+0000 7f2d04838700 1 -- 172.21.15.167:0/3084951824 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:12 5.c 5:30000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x55aeed199a40 con 0x7f2cf800d8e0 2023-10-05T19:28:30.094 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:30.086+0000 7f2d04838700 1 -- 172.21.15.167:0/3084951824 <== osd.0 v2:172.21.15.118:6800/27690 3 ==== osd_op_reply(12 [pgnls start_epoch 30 out=112b] v23'1 uv1 ondisk = 1) v8 ==== 144+0+112 (crc 0 0 0) 0x7f2cf4014040 con 0x7f2cf800d8e0 2023-10-05T19:28:30.094 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:30.086+0000 7f2d04838700 1 -- 172.21.15.167:0/3084951824 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:13 5.1c 5:38000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x55aeed199a40 con 0x7f2cfc037c00 2023-10-05T19:28:30.094 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:30.090+0000 7f2d03836700 1 -- 172.21.15.167:0/3084951824 <== osd.2 v2:172.21.15.167:6816/30037 2 ==== osd_op_reply(13 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f2cf8006070 con 0x7f2cfc037c00 2023-10-05T19:28:30.094 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:30.090+0000 7f2d03836700 1 -- 172.21.15.167:0/3084951824 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:14 5.2 5:40000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x55aeed199a40 con 0x55aeed197120 2023-10-05T19:28:30.095 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:30.090+0000 7f2d04037700 1 -- 172.21.15.167:0/3084951824 <== osd.3 v2:172.21.15.167:6808/30036 2 ==== osd_op_reply(14 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f2cfc024920 con 0x55aeed197120 2023-10-05T19:28:30.095 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:30.090+0000 7f2d04037700 1 -- 172.21.15.167:0/3084951824 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:15 5.12 5:48000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x55aeed199a40 con 0x55aeed15f610 2023-10-05T19:28:30.096 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:30.090+0000 7f2d04838700 1 -- 172.21.15.167:0/3084951824 <== osd.1 v2:172.21.15.167:6800/30035 8 ==== osd_op_reply(15 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f2cf4014040 con 0x55aeed15f610 2023-10-05T19:28:30.096 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:30.090+0000 7f2d04838700 1 -- 172.21.15.167:0/3084951824 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:16 5.a 5:50000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x55aeed199a40 con 0x7f2cf800d8e0 2023-10-05T19:28:30.096 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:30.090+0000 7f2d04838700 1 -- 172.21.15.167:0/3084951824 <== osd.0 v2:172.21.15.118:6800/27690 4 ==== osd_op_reply(16 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f2cf4022040 con 0x7f2cf800d8e0 2023-10-05T19:28:30.097 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:30.090+0000 7f2d04838700 1 -- 172.21.15.167:0/3084951824 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:17 5.1a 5:58000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x55aeed199a40 con 0x55aeed15f610 2023-10-05T19:28:30.097 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:30.090+0000 7f2d04838700 1 -- 172.21.15.167:0/3084951824 <== osd.1 v2:172.21.15.167:6800/30035 9 ==== osd_op_reply(17 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f2cf4022040 con 0x55aeed15f610 2023-10-05T19:28:30.097 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:30.090+0000 7f2d04838700 1 -- 172.21.15.167:0/3084951824 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:18 5.6 5:60000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x55aeed199a40 con 0x7f2cfc037c00 2023-10-05T19:28:30.098 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:30.094+0000 7f2d03836700 1 -- 172.21.15.167:0/3084951824 <== osd.2 v2:172.21.15.167:6816/30037 3 ==== osd_op_reply(18 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f2cf8006070 con 0x7f2cfc037c00 2023-10-05T19:28:30.098 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:30.094+0000 7f2d03836700 1 -- 172.21.15.167:0/3084951824 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:19 5.16 5:68000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x55aeed199a40 con 0x55aeed197120 2023-10-05T19:28:30.098 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:30.094+0000 7f2d04037700 1 -- 172.21.15.167:0/3084951824 <== osd.3 v2:172.21.15.167:6808/30036 3 ==== osd_op_reply(19 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f2cfc024920 con 0x55aeed197120 2023-10-05T19:28:30.099 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:30.094+0000 7f2d04037700 1 -- 172.21.15.167:0/3084951824 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:20 5.e 5:70000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x55aeed199a40 con 0x55aeed197120 2023-10-05T19:28:30.099 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:30.094+0000 7f2d04037700 1 -- 172.21.15.167:0/3084951824 <== osd.3 v2:172.21.15.167:6808/30036 4 ==== osd_op_reply(20 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f2cfc024920 con 0x55aeed197120 2023-10-05T19:28:30.099 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:30.094+0000 7f2d04037700 1 -- 172.21.15.167:0/3084951824 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:21 5.1e 5:78000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x55aeed199a40 con 0x7f2cf800d8e0 2023-10-05T19:28:30.100 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:30.094+0000 7f2d04838700 1 -- 172.21.15.167:0/3084951824 <== osd.0 v2:172.21.15.118:6800/27690 5 ==== osd_op_reply(21 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f2cf4022040 con 0x7f2cf800d8e0 2023-10-05T19:28:30.100 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:30.094+0000 7f2d04838700 1 -- 172.21.15.167:0/3084951824 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:22 5.1 5:80000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x55aeed199a40 con 0x55aeed197120 2023-10-05T19:28:30.101 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:30.094+0000 7f2d04037700 1 -- 172.21.15.167:0/3084951824 <== osd.3 v2:172.21.15.167:6808/30036 5 ==== osd_op_reply(22 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f2cfc024920 con 0x55aeed197120 2023-10-05T19:28:30.101 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:30.094+0000 7f2d04037700 1 -- 172.21.15.167:0/3084951824 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:23 5.11 5:88000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x55aeed199a40 con 0x55aeed197120 2023-10-05T19:28:30.101 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:30.094+0000 7f2d04037700 1 -- 172.21.15.167:0/3084951824 <== osd.3 v2:172.21.15.167:6808/30036 6 ==== osd_op_reply(23 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f2cfc024920 con 0x55aeed197120 2023-10-05T19:28:30.101 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:30.094+0000 7f2d04037700 1 -- 172.21.15.167:0/3084951824 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:24 5.9 5:90000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x55aeed199a40 con 0x55aeed15f610 2023-10-05T19:28:30.102 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:30.098+0000 7f2d04838700 1 -- 172.21.15.167:0/3084951824 <== osd.1 v2:172.21.15.167:6800/30035 10 ==== osd_op_reply(24 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f2cf4022040 con 0x55aeed15f610 2023-10-05T19:28:30.102 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:30.098+0000 7f2d04838700 1 -- 172.21.15.167:0/3084951824 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:25 5.19 5:98000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x55aeed199a40 con 0x55aeed15f610 2023-10-05T19:28:30.103 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:30.098+0000 7f2d04838700 1 -- 172.21.15.167:0/3084951824 <== osd.1 v2:172.21.15.167:6800/30035 11 ==== osd_op_reply(25 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f2cf4022040 con 0x55aeed15f610 2023-10-05T19:28:30.103 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:30.098+0000 7f2d04838700 1 -- 172.21.15.167:0/3084951824 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:26 5.5 5:a0000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x55aeed199a40 con 0x7f2cf800d8e0 2023-10-05T19:28:30.104 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:30.098+0000 7f2d04838700 1 -- 172.21.15.167:0/3084951824 <== osd.0 v2:172.21.15.118:6800/27690 6 ==== osd_op_reply(26 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f2cf4022040 con 0x7f2cf800d8e0 2023-10-05T19:28:30.104 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:30.098+0000 7f2d04838700 1 -- 172.21.15.167:0/3084951824 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:27 5.15 5:a8000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x55aeed199a40 con 0x55aeed197120 2023-10-05T19:28:30.104 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:30.098+0000 7f2d04037700 1 -- 172.21.15.167:0/3084951824 <== osd.3 v2:172.21.15.167:6808/30036 7 ==== osd_op_reply(27 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f2cfc024920 con 0x55aeed197120 2023-10-05T19:28:30.104 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:30.098+0000 7f2d04037700 1 -- 172.21.15.167:0/3084951824 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:28 5.d 5:b0000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x55aeed199a40 con 0x7f2cfc037c00 2023-10-05T19:28:30.106 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:30.098+0000 7f2d03836700 1 -- 172.21.15.167:0/3084951824 <== osd.2 v2:172.21.15.167:6816/30037 4 ==== osd_op_reply(28 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f2cf8006070 con 0x7f2cfc037c00 2023-10-05T19:28:30.106 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:30.098+0000 7f2d03836700 1 -- 172.21.15.167:0/3084951824 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:29 5.1d 5:b8000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x55aeed199a40 con 0x55aeed15f610 2023-10-05T19:28:30.106 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:30.102+0000 7f2d04838700 1 -- 172.21.15.167:0/3084951824 <== osd.1 v2:172.21.15.167:6800/30035 12 ==== osd_op_reply(29 [pgnls start_epoch 30 out=107b] v30'2 uv2 ondisk = 1) v8 ==== 144+0+107 (crc 0 0 0) 0x7f2cf4022040 con 0x55aeed15f610 2023-10-05T19:28:30.106 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:30.102+0000 7f2d04838700 1 -- 172.21.15.167:0/3084951824 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:30 5.3 5:c0000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x55aeed199a40 con 0x7f2cf800d8e0 2023-10-05T19:28:30.107 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:30.102+0000 7f2d04838700 1 -- 172.21.15.167:0/3084951824 <== osd.0 v2:172.21.15.118:6800/27690 7 ==== osd_op_reply(30 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f2cf4022040 con 0x7f2cf800d8e0 2023-10-05T19:28:30.107 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:30.102+0000 7f2d04838700 1 -- 172.21.15.167:0/3084951824 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:31 5.13 5:c8000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x55aeed199a40 con 0x55aeed197120 2023-10-05T19:28:30.108 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:30.102+0000 7f2d04037700 1 -- 172.21.15.167:0/3084951824 <== osd.3 v2:172.21.15.167:6808/30036 8 ==== osd_op_reply(31 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f2cfc024920 con 0x55aeed197120 2023-10-05T19:28:30.108 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:30.102+0000 7f2d04037700 1 -- 172.21.15.167:0/3084951824 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:32 5.b 5:d0000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x55aeed199a40 con 0x7f2cf800d8e0 2023-10-05T19:28:30.108 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:30.102+0000 7f2d04838700 1 -- 172.21.15.167:0/3084951824 <== osd.0 v2:172.21.15.118:6800/27690 8 ==== osd_op_reply(32 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f2cf4022040 con 0x7f2cf800d8e0 2023-10-05T19:28:30.109 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:30.102+0000 7f2d04838700 1 -- 172.21.15.167:0/3084951824 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:33 5.1b 5:d8000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x55aeed199a40 con 0x7f2cfc037c00 2023-10-05T19:28:30.109 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:30.102+0000 7f2d03836700 1 -- 172.21.15.167:0/3084951824 <== osd.2 v2:172.21.15.167:6816/30037 5 ==== osd_op_reply(33 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f2cf8006070 con 0x7f2cfc037c00 2023-10-05T19:28:30.110 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:30.102+0000 7f2d03836700 1 -- 172.21.15.167:0/3084951824 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:34 5.7 5:e0000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x55aeed199a40 con 0x55aeed197120 2023-10-05T19:28:30.110 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:30.106+0000 7f2d04037700 1 -- 172.21.15.167:0/3084951824 <== osd.3 v2:172.21.15.167:6808/30036 9 ==== osd_op_reply(34 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f2cfc024920 con 0x55aeed197120 2023-10-05T19:28:30.110 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:30.106+0000 7f2d04037700 1 -- 172.21.15.167:0/3084951824 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:35 5.17 5:e8000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x55aeed199a40 con 0x7f2cf800d8e0 2023-10-05T19:28:30.111 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:30.106+0000 7f2d04838700 1 -- 172.21.15.167:0/3084951824 <== osd.0 v2:172.21.15.118:6800/27690 9 ==== osd_op_reply(35 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f2cf4022040 con 0x7f2cf800d8e0 2023-10-05T19:28:30.111 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:30.106+0000 7f2d04838700 1 -- 172.21.15.167:0/3084951824 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:36 5.f 5:f0000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x55aeed199a40 con 0x55aeed15f610 2023-10-05T19:28:30.112 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:30.106+0000 7f2d04838700 1 -- 172.21.15.167:0/3084951824 <== osd.1 v2:172.21.15.167:6800/30035 13 ==== osd_op_reply(36 [pgnls start_epoch 30 out=115b] v23'2 uv2 ondisk = 1) v8 ==== 144+0+115 (crc 0 0 0) 0x7f2cf402f080 con 0x55aeed15f610 2023-10-05T19:28:30.112 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:30.106+0000 7f2d04838700 1 -- 172.21.15.167:0/3084951824 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:37 5.1f 5:f8000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x55aeed199a40 con 0x7f2cfc037c00 2023-10-05T19:28:30.113 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:30.106+0000 7f2d03836700 1 -- 172.21.15.167:0/3084951824 <== osd.2 v2:172.21.15.167:6816/30037 6 ==== osd_op_reply(37 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f2cf8006070 con 0x7f2cfc037c00 2023-10-05T19:28:30.113 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:30.106+0000 7f2d06502e40 20 RGWRados::pool_iterate: got zonegroup_info.4f1f719d-9648-417b-b10c-e5d939825419 2023-10-05T19:28:30.113 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:30.106+0000 7f2d06502e40 20 RGWRados::pool_iterate: got zone_info.9a9df965-6612-45d4-a45f-6e3c889bacc2 2023-10-05T19:28:30.114 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:30.106+0000 7f2d06502e40 20 RGWRados::pool_iterate: got zone_names.default 2023-10-05T19:28:30.114 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:30.106+0000 7f2d06502e40 20 RGWRados::pool_iterate: got zonegroups_names.default 2023-10-05T19:28:30.114 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:30.106+0000 7f2d06502e40 20 rados->read ofs=0 len=0 2023-10-05T19:28:30.114 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:30.106+0000 7f2d06502e40 1 -- 172.21.15.167:0/3084951824 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:38 5.12 5:49953fa1:::default.realm:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x55aeed195ed0 con 0x55aeed15f610 2023-10-05T19:28:30.115 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:30.110+0000 7f2d04838700 1 -- 172.21.15.167:0/3084951824 <== osd.1 v2:172.21.15.167:6800/30035 14 ==== osd_op_reply(38 default.realm [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 ==== 157+0+0 (crc 0 0 0) 0x7f2cf402f080 con 0x55aeed15f610 2023-10-05T19:28:30.115 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:30.110+0000 7f2d06502e40 20 rados_obj.operate() r=-2 bl.length=0 2023-10-05T19:28:30.115 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:30.110+0000 7f2d06502e40 20 rados->read ofs=0 len=0 2023-10-05T19:28:30.115 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:30.110+0000 7f2d06502e40 1 -- 172.21.15.167:0/3084951824 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:39 5.f 5:f4c53578:::zonegroups_names.default:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x55aeed196850 con 0x55aeed15f610 2023-10-05T19:28:30.115 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:30.110+0000 7f2d04838700 1 -- 172.21.15.167:0/3084951824 <== osd.1 v2:172.21.15.167:6800/30035 15 ==== osd_op_reply(39 zonegroups_names.default [read 0~46 out=46b] v0'0 uv2 ondisk = 0) v8 ==== 168+0+46 (crc 0 0 0) 0x7f2cf402f080 con 0x55aeed15f610 2023-10-05T19:28:30.115 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:30.110+0000 7f2d06502e40 20 rados_obj.operate() r=0 bl.length=46 2023-10-05T19:28:30.116 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:30.110+0000 7f2d06502e40 20 rados->read ofs=0 len=0 2023-10-05T19:28:30.116 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:30.110+0000 7f2d06502e40 1 -- 172.21.15.167:0/3084951824 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:40 5.c 5:30d299a5:::zonegroup_info.4f1f719d-9648-417b-b10c-e5d939825419:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x55aeed195ac0 con 0x7f2cf800d8e0 2023-10-05T19:28:30.117 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:30.110+0000 7f2d04838700 1 -- 172.21.15.167:0/3084951824 <== osd.0 v2:172.21.15.118:6800/27690 10 ==== osd_op_reply(40 zonegroup_info.4f1f719d-9648-417b-b10c-e5d939825419 [read 0~362 out=362b] v0'0 uv1 ondisk = 0) v8 ==== 195+0+362 (crc 0 0 0) 0x7f2cf402f080 con 0x7f2cf800d8e0 2023-10-05T19:28:30.117 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:30.110+0000 7f2d06502e40 20 rados_obj.operate() r=0 bl.length=362 2023-10-05T19:28:30.117 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:30.110+0000 7f2d06502e40 20 rados->read ofs=0 len=0 2023-10-05T19:28:30.117 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:30.110+0000 7f2d06502e40 1 -- 172.21.15.167:0/3084951824 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:41 5.13 5:cbbf6c60:::converted:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x55aeed19a330 con 0x55aeed197120 2023-10-05T19:28:30.117 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:30.110+0000 7f2d04037700 1 -- 172.21.15.167:0/3084951824 <== osd.3 v2:172.21.15.167:6808/30036 10 ==== osd_op_reply(41 converted [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 ==== 153+0+0 (crc 0 0 0) 0x7f2cfc024920 con 0x55aeed197120 2023-10-05T19:28:30.117 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:30.110+0000 7f2d06502e40 20 rados_obj.operate() r=-2 bl.length=0 2023-10-05T19:28:30.117 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:30.110+0000 7f2d06502e40 20 rados->read ofs=0 len=0 2023-10-05T19:28:30.117 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:30.110+0000 7f2d06502e40 1 -- 172.21.15.167:0/3084951824 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:42 5.12 5:49953fa1:::default.realm:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x55aeed19a6e0 con 0x55aeed15f610 2023-10-05T19:28:30.118 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:30.110+0000 7f2d04838700 1 -- 172.21.15.167:0/3084951824 <== osd.1 v2:172.21.15.167:6800/30035 16 ==== osd_op_reply(42 default.realm [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 ==== 157+0+0 (crc 0 0 0) 0x7f2cf402f080 con 0x55aeed15f610 2023-10-05T19:28:30.118 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:30.110+0000 7f2d06502e40 20 rados_obj.operate() r=-2 bl.length=0 2023-10-05T19:28:30.118 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:30.110+0000 7f2d06502e40 20 rados->read ofs=0 len=0 2023-10-05T19:28:30.119 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:30.110+0000 7f2d06502e40 1 -- 172.21.15.167:0/3084951824 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:43 5.f 5:f4c53578:::zonegroups_names.default:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x55aeed19a6e0 con 0x55aeed15f610 2023-10-05T19:28:30.119 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:30.110+0000 7f2d04838700 1 -- 172.21.15.167:0/3084951824 <== osd.1 v2:172.21.15.167:6800/30035 17 ==== osd_op_reply(43 zonegroups_names.default [read 0~46 out=46b] v0'0 uv2 ondisk = 0) v8 ==== 168+0+46 (crc 0 0 0) 0x7f2cf402f080 con 0x55aeed15f610 2023-10-05T19:28:30.119 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:30.114+0000 7f2d06502e40 20 rados_obj.operate() r=0 bl.length=46 2023-10-05T19:28:30.119 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:30.114+0000 7f2d06502e40 1 -- 172.21.15.167:0/3084951824 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:44 5.0 5:00000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x55aeed19b6f0 con 0x55aeed197120 2023-10-05T19:28:30.120 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:30.114+0000 7f2d04037700 1 -- 172.21.15.167:0/3084951824 <== osd.3 v2:172.21.15.167:6808/30036 11 ==== osd_op_reply(44 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f2cfc024920 con 0x55aeed197120 2023-10-05T19:28:30.121 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:30.114+0000 7f2d04037700 1 -- 172.21.15.167:0/3084951824 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:45 5.10 5:08000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x55aeed19b6f0 con 0x7f2cfc037c00 2023-10-05T19:28:30.121 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:30.114+0000 7f2d03836700 1 -- 172.21.15.167:0/3084951824 <== osd.2 v2:172.21.15.167:6816/30037 7 ==== osd_op_reply(45 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f2cf8006070 con 0x7f2cfc037c00 2023-10-05T19:28:30.121 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:30.114+0000 7f2d03836700 1 -- 172.21.15.167:0/3084951824 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:46 5.8 5:10000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x55aeed19b6f0 con 0x7f2cf800d8e0 2023-10-05T19:28:30.122 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:30.118+0000 7f2d04838700 1 -- 172.21.15.167:0/3084951824 <== osd.0 v2:172.21.15.118:6800/27690 11 ==== osd_op_reply(46 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f2cf402f080 con 0x7f2cf800d8e0 2023-10-05T19:28:30.122 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:30.118+0000 7f2d04838700 1 -- 172.21.15.167:0/3084951824 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:47 5.18 5:18000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x55aeed195ac0 con 0x55aeed15f610 2023-10-05T19:28:30.123 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:30.118+0000 7f2d04838700 1 -- 172.21.15.167:0/3084951824 <== osd.1 v2:172.21.15.167:6800/30035 18 ==== osd_op_reply(47 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f2cf402f080 con 0x55aeed15f610 2023-10-05T19:28:30.123 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:30.118+0000 7f2d04838700 1 -- 172.21.15.167:0/3084951824 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:48 5.4 5:20000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x55aeed195ac0 con 0x55aeed15f610 2023-10-05T19:28:30.124 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:30.118+0000 7f2d04838700 1 -- 172.21.15.167:0/3084951824 <== osd.1 v2:172.21.15.167:6800/30035 19 ==== osd_op_reply(48 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f2cf402f080 con 0x55aeed15f610 2023-10-05T19:28:30.124 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:30.118+0000 7f2d04838700 1 -- 172.21.15.167:0/3084951824 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:49 5.14 5:28000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x55aeed195ac0 con 0x7f2cf800d8e0 2023-10-05T19:28:30.125 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:30.118+0000 7f2d04838700 1 -- 172.21.15.167:0/3084951824 <== osd.0 v2:172.21.15.118:6800/27690 12 ==== osd_op_reply(49 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f2cf402f080 con 0x7f2cf800d8e0 2023-10-05T19:28:30.125 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:30.118+0000 7f2d04838700 1 -- 172.21.15.167:0/3084951824 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:50 5.c 5:30000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x55aeed195ac0 con 0x7f2cf800d8e0 2023-10-05T19:28:30.126 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:30.122+0000 7f2d04838700 1 -- 172.21.15.167:0/3084951824 <== osd.0 v2:172.21.15.118:6800/27690 13 ==== osd_op_reply(50 [pgnls start_epoch 30 out=112b] v23'1 uv1 ondisk = 1) v8 ==== 144+0+112 (crc 0 0 0) 0x7f2cf402f080 con 0x7f2cf800d8e0 2023-10-05T19:28:30.126 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:30.122+0000 7f2d04838700 1 -- 172.21.15.167:0/3084951824 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:51 5.1c 5:38000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x55aeed195ac0 con 0x7f2cfc037c00 2023-10-05T19:28:30.127 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:30.122+0000 7f2d03836700 1 -- 172.21.15.167:0/3084951824 <== osd.2 v2:172.21.15.167:6816/30037 8 ==== osd_op_reply(51 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f2cf8006070 con 0x7f2cfc037c00 2023-10-05T19:28:30.127 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:30.122+0000 7f2d03836700 1 -- 172.21.15.167:0/3084951824 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:52 5.2 5:40000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x55aeed195ac0 con 0x55aeed197120 2023-10-05T19:28:30.128 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:30.122+0000 7f2d04037700 1 -- 172.21.15.167:0/3084951824 <== osd.3 v2:172.21.15.167:6808/30036 12 ==== osd_op_reply(52 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f2cfc024920 con 0x55aeed197120 2023-10-05T19:28:30.128 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:30.122+0000 7f2d04037700 1 -- 172.21.15.167:0/3084951824 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:53 5.12 5:48000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x55aeed195ac0 con 0x55aeed15f610 2023-10-05T19:28:30.129 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:30.122+0000 7f2d04838700 1 -- 172.21.15.167:0/3084951824 <== osd.1 v2:172.21.15.167:6800/30035 20 ==== osd_op_reply(53 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f2cf402f080 con 0x55aeed15f610 2023-10-05T19:28:30.129 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:30.122+0000 7f2d04838700 1 -- 172.21.15.167:0/3084951824 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:54 5.a 5:50000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x55aeed195ac0 con 0x7f2cf800d8e0 2023-10-05T19:28:30.129 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:30.122+0000 7f2d04838700 1 -- 172.21.15.167:0/3084951824 <== osd.0 v2:172.21.15.118:6800/27690 14 ==== osd_op_reply(54 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f2cf402f080 con 0x7f2cf800d8e0 2023-10-05T19:28:30.130 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:30.122+0000 7f2d04838700 1 -- 172.21.15.167:0/3084951824 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:55 5.1a 5:58000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x55aeed195ac0 con 0x55aeed15f610 2023-10-05T19:28:30.130 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:30.126+0000 7f2d04838700 1 -- 172.21.15.167:0/3084951824 <== osd.1 v2:172.21.15.167:6800/30035 21 ==== osd_op_reply(55 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f2cf402f080 con 0x55aeed15f610 2023-10-05T19:28:30.130 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:30.126+0000 7f2d04838700 1 -- 172.21.15.167:0/3084951824 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:56 5.6 5:60000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x55aeed195ac0 con 0x7f2cfc037c00 2023-10-05T19:28:30.131 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:30.126+0000 7f2d03836700 1 -- 172.21.15.167:0/3084951824 <== osd.2 v2:172.21.15.167:6816/30037 9 ==== osd_op_reply(56 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f2cf8006070 con 0x7f2cfc037c00 2023-10-05T19:28:30.131 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:30.126+0000 7f2d03836700 1 -- 172.21.15.167:0/3084951824 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:57 5.16 5:68000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x55aeed195ac0 con 0x55aeed197120 2023-10-05T19:28:30.132 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:30.126+0000 7f2d04037700 1 -- 172.21.15.167:0/3084951824 <== osd.3 v2:172.21.15.167:6808/30036 13 ==== osd_op_reply(57 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f2cfc024920 con 0x55aeed197120 2023-10-05T19:28:30.132 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:30.126+0000 7f2d04037700 1 -- 172.21.15.167:0/3084951824 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:58 5.e 5:70000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x55aeed195ac0 con 0x55aeed197120 2023-10-05T19:28:30.133 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:30.126+0000 7f2d04037700 1 -- 172.21.15.167:0/3084951824 <== osd.3 v2:172.21.15.167:6808/30036 14 ==== osd_op_reply(58 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f2cfc024920 con 0x55aeed197120 2023-10-05T19:28:30.133 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:30.126+0000 7f2d04037700 1 -- 172.21.15.167:0/3084951824 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:59 5.1e 5:78000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x55aeed195ac0 con 0x7f2cf800d8e0 2023-10-05T19:28:30.134 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:30.130+0000 7f2d04838700 1 -- 172.21.15.167:0/3084951824 <== osd.0 v2:172.21.15.118:6800/27690 15 ==== osd_op_reply(59 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f2cf402f080 con 0x7f2cf800d8e0 2023-10-05T19:28:30.134 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:30.130+0000 7f2d04838700 1 -- 172.21.15.167:0/3084951824 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:60 5.1 5:80000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x55aeed195ac0 con 0x55aeed197120 2023-10-05T19:28:30.135 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:30.130+0000 7f2d04037700 1 -- 172.21.15.167:0/3084951824 <== osd.3 v2:172.21.15.167:6808/30036 15 ==== osd_op_reply(60 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f2cfc024920 con 0x55aeed197120 2023-10-05T19:28:30.135 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:30.130+0000 7f2d04037700 1 -- 172.21.15.167:0/3084951824 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:61 5.11 5:88000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x55aeed195ac0 con 0x55aeed197120 2023-10-05T19:28:30.136 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:30.130+0000 7f2d04037700 1 -- 172.21.15.167:0/3084951824 <== osd.3 v2:172.21.15.167:6808/30036 16 ==== osd_op_reply(61 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f2cfc024920 con 0x55aeed197120 2023-10-05T19:28:30.136 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:30.130+0000 7f2d04037700 1 -- 172.21.15.167:0/3084951824 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:62 5.9 5:90000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x55aeed195ac0 con 0x55aeed15f610 2023-10-05T19:28:30.136 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:30.130+0000 7f2d04838700 1 -- 172.21.15.167:0/3084951824 <== osd.1 v2:172.21.15.167:6800/30035 22 ==== osd_op_reply(62 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f2cf402f080 con 0x55aeed15f610 2023-10-05T19:28:30.137 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:30.130+0000 7f2d04838700 1 -- 172.21.15.167:0/3084951824 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:63 5.19 5:98000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x55aeed195ac0 con 0x55aeed15f610 2023-10-05T19:28:30.137 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:30.130+0000 7f2d04838700 1 -- 172.21.15.167:0/3084951824 <== osd.1 v2:172.21.15.167:6800/30035 23 ==== osd_op_reply(63 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f2cf402f080 con 0x55aeed15f610 2023-10-05T19:28:30.137 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:30.130+0000 7f2d04838700 1 -- 172.21.15.167:0/3084951824 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:64 5.5 5:a0000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x55aeed195ac0 con 0x7f2cf800d8e0 2023-10-05T19:28:30.138 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:30.134+0000 7f2d04838700 1 -- 172.21.15.167:0/3084951824 <== osd.0 v2:172.21.15.118:6800/27690 16 ==== osd_op_reply(64 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f2cf402f080 con 0x7f2cf800d8e0 2023-10-05T19:28:30.138 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:30.134+0000 7f2d04838700 1 -- 172.21.15.167:0/3084951824 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:65 5.15 5:a8000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x55aeed195ac0 con 0x55aeed197120 2023-10-05T19:28:30.139 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:30.134+0000 7f2d04037700 1 -- 172.21.15.167:0/3084951824 <== osd.3 v2:172.21.15.167:6808/30036 17 ==== osd_op_reply(65 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f2cfc024920 con 0x55aeed197120 2023-10-05T19:28:30.139 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:30.134+0000 7f2d04037700 1 -- 172.21.15.167:0/3084951824 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:66 5.d 5:b0000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x55aeed195ac0 con 0x7f2cfc037c00 2023-10-05T19:28:30.140 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:30.134+0000 7f2d03836700 1 -- 172.21.15.167:0/3084951824 <== osd.2 v2:172.21.15.167:6816/30037 10 ==== osd_op_reply(66 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f2cf8006070 con 0x7f2cfc037c00 2023-10-05T19:28:30.140 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:30.134+0000 7f2d03836700 1 -- 172.21.15.167:0/3084951824 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:67 5.1d 5:b8000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x55aeed195ac0 con 0x55aeed15f610 2023-10-05T19:28:30.141 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:30.134+0000 7f2d04838700 1 -- 172.21.15.167:0/3084951824 <== osd.1 v2:172.21.15.167:6800/30035 24 ==== osd_op_reply(67 [pgnls start_epoch 30 out=107b] v30'2 uv2 ondisk = 1) v8 ==== 144+0+107 (crc 0 0 0) 0x7f2cf402f080 con 0x55aeed15f610 2023-10-05T19:28:30.141 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:30.134+0000 7f2d04838700 1 -- 172.21.15.167:0/3084951824 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:68 5.3 5:c0000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x55aeed195ac0 con 0x7f2cf800d8e0 2023-10-05T19:28:30.142 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:30.138+0000 7f2d04838700 1 -- 172.21.15.167:0/3084951824 <== osd.0 v2:172.21.15.118:6800/27690 17 ==== osd_op_reply(68 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f2cf402f080 con 0x7f2cf800d8e0 2023-10-05T19:28:30.142 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:30.138+0000 7f2d04838700 1 -- 172.21.15.167:0/3084951824 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:69 5.13 5:c8000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x55aeed195ac0 con 0x55aeed197120 2023-10-05T19:28:30.143 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:30.138+0000 7f2d04037700 1 -- 172.21.15.167:0/3084951824 <== osd.3 v2:172.21.15.167:6808/30036 18 ==== osd_op_reply(69 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f2cfc024920 con 0x55aeed197120 2023-10-05T19:28:30.143 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:30.138+0000 7f2d04037700 1 -- 172.21.15.167:0/3084951824 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:70 5.b 5:d0000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x55aeed195ac0 con 0x7f2cf800d8e0 2023-10-05T19:28:30.144 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:30.138+0000 7f2d04838700 1 -- 172.21.15.167:0/3084951824 <== osd.0 v2:172.21.15.118:6800/27690 18 ==== osd_op_reply(70 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f2cf402f080 con 0x7f2cf800d8e0 2023-10-05T19:28:30.144 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:30.138+0000 7f2d04838700 1 -- 172.21.15.167:0/3084951824 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:71 5.1b 5:d8000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x55aeed195ac0 con 0x7f2cfc037c00 2023-10-05T19:28:30.145 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:30.138+0000 7f2d03836700 1 -- 172.21.15.167:0/3084951824 <== osd.2 v2:172.21.15.167:6816/30037 11 ==== osd_op_reply(71 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f2cf8006070 con 0x7f2cfc037c00 2023-10-05T19:28:30.145 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:30.138+0000 7f2d03836700 1 -- 172.21.15.167:0/3084951824 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:72 5.7 5:e0000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x55aeed195ac0 con 0x55aeed197120 2023-10-05T19:28:30.146 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:30.138+0000 7f2d04037700 1 -- 172.21.15.167:0/3084951824 <== osd.3 v2:172.21.15.167:6808/30036 19 ==== osd_op_reply(72 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f2cfc024920 con 0x55aeed197120 2023-10-05T19:28:30.146 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:30.138+0000 7f2d04037700 1 -- 172.21.15.167:0/3084951824 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:73 5.17 5:e8000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x55aeed195ac0 con 0x7f2cf800d8e0 2023-10-05T19:28:30.147 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:30.142+0000 7f2d04838700 1 -- 172.21.15.167:0/3084951824 <== osd.0 v2:172.21.15.118:6800/27690 19 ==== osd_op_reply(73 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f2cf402f080 con 0x7f2cf800d8e0 2023-10-05T19:28:30.147 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:30.142+0000 7f2d04838700 1 -- 172.21.15.167:0/3084951824 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:74 5.f 5:f0000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x55aeed195ac0 con 0x55aeed15f610 2023-10-05T19:28:30.147 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:30.142+0000 7f2d04838700 1 -- 172.21.15.167:0/3084951824 <== osd.1 v2:172.21.15.167:6800/30035 25 ==== osd_op_reply(74 [pgnls start_epoch 30 out=115b] v23'2 uv2 ondisk = 1) v8 ==== 144+0+115 (crc 0 0 0) 0x7f2cf402f080 con 0x55aeed15f610 2023-10-05T19:28:30.148 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:30.142+0000 7f2d04838700 1 -- 172.21.15.167:0/3084951824 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:75 5.1f 5:f8000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x55aeed195ac0 con 0x7f2cfc037c00 2023-10-05T19:28:30.149 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:30.142+0000 7f2d03836700 1 -- 172.21.15.167:0/3084951824 <== osd.2 v2:172.21.15.167:6816/30037 12 ==== osd_op_reply(75 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f2cf8006070 con 0x7f2cfc037c00 2023-10-05T19:28:30.149 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:30.142+0000 7f2d06502e40 20 RGWRados::pool_iterate: got zonegroup_info.4f1f719d-9648-417b-b10c-e5d939825419 2023-10-05T19:28:30.149 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:30.142+0000 7f2d06502e40 20 RGWRados::pool_iterate: got zone_info.9a9df965-6612-45d4-a45f-6e3c889bacc2 2023-10-05T19:28:30.149 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:30.142+0000 7f2d06502e40 20 RGWRados::pool_iterate: got zone_names.default 2023-10-05T19:28:30.149 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:30.142+0000 7f2d06502e40 20 RGWRados::pool_iterate: got zonegroups_names.default 2023-10-05T19:28:30.149 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:30.142+0000 7f2d06502e40 20 rados->read ofs=0 len=0 2023-10-05T19:28:30.149 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:30.142+0000 7f2d06502e40 1 -- 172.21.15.167:0/3084951824 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:76 5.f 5:f43b6ece:::zone_names.default:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x55aeed19a6e0 con 0x55aeed15f610 2023-10-05T19:28:30.150 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:30.142+0000 7f2d04838700 1 -- 172.21.15.167:0/3084951824 <== osd.1 v2:172.21.15.167:6800/30035 26 ==== osd_op_reply(76 zone_names.default [read 0~46 out=46b] v0'0 uv1 ondisk = 0) v8 ==== 162+0+46 (crc 0 0 0) 0x7f2cf402f080 con 0x55aeed15f610 2023-10-05T19:28:30.150 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:30.142+0000 7f2d06502e40 20 rados_obj.operate() r=0 bl.length=46 2023-10-05T19:28:30.150 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:30.142+0000 7f2d06502e40 20 rados->read ofs=0 len=0 2023-10-05T19:28:30.150 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:30.142+0000 7f2d06502e40 1 -- 172.21.15.167:0/3084951824 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:77 5.1d 5:b8374eb4:::zone_info.9a9df965-6612-45d4-a45f-6e3c889bacc2:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x55aeed19aa90 con 0x55aeed15f610 2023-10-05T19:28:30.151 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:30.146+0000 7f2d04838700 1 -- 172.21.15.167:0/3084951824 <== osd.1 v2:172.21.15.167:6800/30035 27 ==== osd_op_reply(77 zone_info.9a9df965-6612-45d4-a45f-6e3c889bacc2 [read 0~874 out=874b] v0'0 uv2 ondisk = 0) v8 ==== 190+0+874 (crc 0 0 0) 0x7f2cf402f080 con 0x55aeed15f610 2023-10-05T19:28:30.151 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:30.146+0000 7f2d06502e40 20 rados_obj.operate() r=0 bl.length=874 2023-10-05T19:28:30.151 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:30.146+0000 7f2d06502e40 20 rados->read ofs=0 len=0 2023-10-05T19:28:30.151 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:30.146+0000 7f2d06502e40 1 -- 172.21.15.167:0/3084951824 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:78 5.f 5:f4c53578:::zonegroups_names.default:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x55aeed19af80 con 0x55aeed15f610 2023-10-05T19:28:30.151 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:30.146+0000 7f2d04838700 1 -- 172.21.15.167:0/3084951824 <== osd.1 v2:172.21.15.167:6800/30035 28 ==== osd_op_reply(78 zonegroups_names.default [read 0~46 out=46b] v0'0 uv2 ondisk = 0) v8 ==== 168+0+46 (crc 0 0 0) 0x7f2cf402f080 con 0x55aeed15f610 2023-10-05T19:28:30.151 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:30.146+0000 7f2d06502e40 20 rados_obj.operate() r=0 bl.length=46 2023-10-05T19:28:30.151 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:30.146+0000 7f2d06502e40 20 rados->read ofs=0 len=0 2023-10-05T19:28:30.152 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:30.146+0000 7f2d06502e40 1 -- 172.21.15.167:0/3084951824 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:79 5.c 5:30d299a5:::zonegroup_info.4f1f719d-9648-417b-b10c-e5d939825419:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x55aeed19af80 con 0x7f2cf800d8e0 2023-10-05T19:28:30.152 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:30.146+0000 7f2d04838700 1 -- 172.21.15.167:0/3084951824 <== osd.0 v2:172.21.15.118:6800/27690 20 ==== osd_op_reply(79 zonegroup_info.4f1f719d-9648-417b-b10c-e5d939825419 [read 0~362 out=362b] v0'0 uv1 ondisk = 0) v8 ==== 195+0+362 (crc 0 0 0) 0x7f2cf402f080 con 0x7f2cf800d8e0 2023-10-05T19:28:30.152 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:30.146+0000 7f2d06502e40 20 rados_obj.operate() r=0 bl.length=362 2023-10-05T19:28:30.152 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:30.146+0000 7f2d06502e40 20 rados->read ofs=0 len=0 2023-10-05T19:28:30.152 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:30.146+0000 7f2d06502e40 1 -- 172.21.15.167:0/3084951824 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:80 5.3 5:c0ad8cbd:::region_map:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x55aeed19afb0 con 0x7f2cf800d8e0 2023-10-05T19:28:30.153 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:30.146+0000 7f2d04838700 1 -- 172.21.15.167:0/3084951824 <== osd.0 v2:172.21.15.118:6800/27690 21 ==== osd_op_reply(80 region_map [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 ==== 154+0+0 (crc 0 0 0) 0x7f2cf4050040 con 0x7f2cf800d8e0 2023-10-05T19:28:30.153 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:30.146+0000 7f2d06502e40 20 rados_obj.operate() r=-2 bl.length=0 2023-10-05T19:28:30.153 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:30.146+0000 7f2d06502e40 20 zone default found 2023-10-05T19:28:30.153 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:30.146+0000 7f2d06502e40 4 Realm: () 2023-10-05T19:28:30.153 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:30.146+0000 7f2d06502e40 4 ZoneGroup: default (4f1f719d-9648-417b-b10c-e5d939825419) 2023-10-05T19:28:30.153 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:30.146+0000 7f2d06502e40 4 Zone: default (9a9df965-6612-45d4-a45f-6e3c889bacc2) 2023-10-05T19:28:30.153 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:30.146+0000 7f2d06502e40 10 cannot find current period zonegroup using local zonegroup configuration 2023-10-05T19:28:30.153 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:30.146+0000 7f2d06502e40 20 zonegroup default 2023-10-05T19:28:30.154 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:30.146+0000 7f2d06502e40 20 rados->read ofs=0 len=0 2023-10-05T19:28:30.154 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:30.146+0000 7f2d06502e40 1 -- 172.21.15.167:0/3084951824 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:81 5.3 5:c52100b6:::period_config.default:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x55aeed19afb0 con 0x7f2cf800d8e0 2023-10-05T19:28:30.154 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:30.146+0000 7f2d04838700 1 -- 172.21.15.167:0/3084951824 <== osd.0 v2:172.21.15.118:6800/27690 22 ==== osd_op_reply(81 period_config.default [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 ==== 165+0+0 (crc 0 0 0) 0x7f2cf4050040 con 0x7f2cf800d8e0 2023-10-05T19:28:30.155 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:30.146+0000 7f2d06502e40 20 rados_obj.operate() r=-2 bl.length=0 2023-10-05T19:28:30.155 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:30.146+0000 7f2d06502e40 20 rados->read ofs=0 len=0 2023-10-05T19:28:30.155 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:30.146+0000 7f2d06502e40 1 -- 172.21.15.167:0/3084951824 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:82 6.2 6:4347d321:::bucket.sync-source-hints.:head [call version.read in=11b,read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x55aeed19c930 con 0x55aeed197120 2023-10-05T19:28:30.155 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:30.150+0000 7f2d04037700 1 -- 172.21.15.167:0/3084951824 <== osd.3 v2:172.21.15.167:6808/30036 20 ==== osd_op_reply(82 bucket.sync-source-hints. [call,read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 ==== 211+0+0 (crc 0 0 0) 0x7f2cfc024920 con 0x55aeed197120 2023-10-05T19:28:30.155 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:30.150+0000 7f2d06502e40 20 rados_obj.operate() r=-2 bl.length=0 2023-10-05T19:28:30.155 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:30.150+0000 7f2d06502e40 20 rados->read ofs=0 len=0 2023-10-05T19:28:30.155 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:30.150+0000 7f2d06502e40 1 -- 172.21.15.167:0/3084951824 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:83 6.b 6:d467b91b:::bucket.sync-target-hints.:head [call version.read in=11b,read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x55aeed19cce0 con 0x55aeed197120 2023-10-05T19:28:30.156 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:30.150+0000 7f2d04037700 1 -- 172.21.15.167:0/3084951824 <== osd.3 v2:172.21.15.167:6808/30036 21 ==== osd_op_reply(83 bucket.sync-target-hints. [call,read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 ==== 211+0+0 (crc 0 0 0) 0x7f2cfc05f040 con 0x55aeed197120 2023-10-05T19:28:30.156 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:30.150+0000 7f2d06502e40 20 rados_obj.operate() r=-2 bl.length=0 2023-10-05T19:28:30.156 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:30.150+0000 7f2d06502e40 20 started sync module instance, tier type = 2023-10-05T19:28:30.156 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:30.150+0000 7f2d06502e40 20 started zone id=9a9df965-6612-45d4-a45f-6e3c889bacc2 (name=default) with tier type = 2023-10-05T19:28:30.156 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:30.150+0000 7f2d06502e40 1 -- 172.21.15.167:0/3084951824 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:84 7.1f 7:f95f44c2:::notify.0:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x55aeed19bff0 con 0x7f2cfc037c00 2023-10-05T19:28:30.158 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:30.154+0000 7f2d03836700 1 -- 172.21.15.167:0/3084951824 <== osd.2 v2:172.21.15.167:6816/30037 13 ==== osd_op_reply(84 notify.0 [create] v30'84 uv84 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f2cf8006070 con 0x7f2cfc037c00 2023-10-05T19:28:30.158 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:30.154+0000 7f2d06502e40 1 -- 172.21.15.167:0/3084951824 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:85 7.1f 7:f95f44c2:::notify.0:head [watch watch cookie 94209790559216] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x55aeed19bc20 con 0x7f2cfc037c00 2023-10-05T19:28:30.159 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:30.154+0000 7f2d06502e40 1 -- 172.21.15.167:0/3084951824 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:86 7.0 7:05bf5b68:::notify.1:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x55aeed19db00 con 0x55aeed15f610 2023-10-05T19:28:30.161 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:30.154+0000 7f2d04838700 1 -- 172.21.15.167:0/3084951824 <== osd.1 v2:172.21.15.167:6800/30035 29 ==== osd_op_reply(86 notify.1 [create] v30'84 uv84 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f2cf4050040 con 0x55aeed15f610 2023-10-05T19:28:30.162 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:30.154+0000 7f2d06502e40 1 -- 172.21.15.167:0/3084951824 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:87 7.0 7:05bf5b68:::notify.1:head [watch watch cookie 94209790564144] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x55aeed19e3f0 con 0x55aeed15f610 2023-10-05T19:28:30.162 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:30.154+0000 7f2d03836700 1 -- 172.21.15.167:0/3084951824 <== osd.2 v2:172.21.15.167:6816/30037 14 ==== osd_op_reply(85 notify.0 [watch watch cookie 94209790559216] v30'85 uv84 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f2cf8006070 con 0x7f2cfc037c00 2023-10-05T19:28:30.162 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:30.154+0000 7f2d06502e40 1 -- 172.21.15.167:0/3084951824 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:88 7.15 7:a93a5511:::notify.2:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x55aeed19ec90 con 0x55aeed197120 2023-10-05T19:28:30.164 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:30.158+0000 7f2d04838700 1 -- 172.21.15.167:0/3084951824 <== osd.1 v2:172.21.15.167:6800/30035 30 ==== osd_op_reply(87 notify.1 [watch watch cookie 94209790564144] v30'85 uv84 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f2cf4050040 con 0x55aeed15f610 2023-10-05T19:28:30.165 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:30.158+0000 7f2d04037700 1 -- 172.21.15.167:0/3084951824 <== osd.3 v2:172.21.15.167:6808/30036 22 ==== osd_op_reply(88 notify.2 [create] v30'84 uv84 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f2cfc05f040 con 0x55aeed197120 2023-10-05T19:28:30.165 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:30.158+0000 7f2d06502e40 1 -- 172.21.15.167:0/3084951824 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:89 7.15 7:a93a5511:::notify.2:head [watch watch cookie 94209790566144] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x55aeed19d1b0 con 0x55aeed197120 2023-10-05T19:28:30.165 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:30.158+0000 7f2d06502e40 1 -- 172.21.15.167:0/3084951824 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:90 7.e 7:7759931f:::notify.3:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x55aeed19f530 con 0x7f2cfc037c00 2023-10-05T19:28:30.168 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:30.162+0000 7f2d03836700 1 -- 172.21.15.167:0/3084951824 <== osd.2 v2:172.21.15.167:6816/30037 15 ==== osd_op_reply(90 notify.3 [create] v30'84 uv84 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f2cf8006070 con 0x7f2cfc037c00 2023-10-05T19:28:30.168 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:30.162+0000 7f2d06502e40 1 -- 172.21.15.167:0/3084951824 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:91 7.e 7:7759931f:::notify.3:head [watch watch cookie 94209790570560] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x55aeed19fdd0 con 0x7f2cfc037c00 2023-10-05T19:28:30.168 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:30.162+0000 7f2d06502e40 1 -- 172.21.15.167:0/3084951824 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:92 7.d 7:b4812045:::notify.4:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x55aeed1a0960 con 0x55aeed15f610 2023-10-05T19:28:30.168 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:30.162+0000 7f2d04037700 1 -- 172.21.15.167:0/3084951824 <== osd.3 v2:172.21.15.167:6808/30036 23 ==== osd_op_reply(89 notify.2 [watch watch cookie 94209790566144] v30'85 uv84 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f2cfc05f040 con 0x55aeed197120 2023-10-05T19:28:30.170 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:30.166+0000 7f2d03836700 1 -- 172.21.15.167:0/3084951824 <== osd.2 v2:172.21.15.167:6816/30037 16 ==== osd_op_reply(91 notify.3 [watch watch cookie 94209790570560] v30'85 uv84 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f2cf8006070 con 0x7f2cfc037c00 2023-10-05T19:28:30.170 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:30.166+0000 7f2d04838700 1 -- 172.21.15.167:0/3084951824 <== osd.1 v2:172.21.15.167:6800/30035 31 ==== osd_op_reply(92 notify.4 [create] v30'84 uv84 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f2cf4050040 con 0x55aeed15f610 2023-10-05T19:28:30.170 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:30.166+0000 7f2d06502e40 1 -- 172.21.15.167:0/3084951824 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:93 7.d 7:b4812045:::notify.4:head [watch watch cookie 94209790572768] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x55aeed1a0430 con 0x55aeed15f610 2023-10-05T19:28:30.171 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:30.166+0000 7f2d06502e40 1 -- 172.21.15.167:0/3084951824 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:94 7.3 7:c609908c:::notify.5:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x55aeed1a1250 con 0x7f2cf800d8e0 2023-10-05T19:28:30.172 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:30.166+0000 7f2d04838700 1 -- 172.21.15.167:0/3084951824 <== osd.1 v2:172.21.15.167:6800/30035 32 ==== osd_op_reply(93 notify.4 [watch watch cookie 94209790572768] v30'85 uv84 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f2cf4050040 con 0x55aeed15f610 2023-10-05T19:28:30.173 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:30.166+0000 7f2d04838700 1 -- 172.21.15.167:0/3084951824 <== osd.0 v2:172.21.15.118:6800/27690 23 ==== osd_op_reply(94 notify.5 [create] v30'84 uv84 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f2cf4050040 con 0x7f2cf800d8e0 2023-10-05T19:28:30.173 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:30.166+0000 7f2d06502e40 1 -- 172.21.15.167:0/3084951824 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:95 7.3 7:c609908c:::notify.5:head [watch watch cookie 94209790567248] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x55aeed1a1200 con 0x7f2cf800d8e0 2023-10-05T19:28:30.173 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:30.166+0000 7f2d06502e40 1 -- 172.21.15.167:0/3084951824 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:96 7.14 7:2b04a3e9:::notify.6:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x55aeed1a1c10 con 0x7f2cf800d8e0 2023-10-05T19:28:30.176 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:30.170+0000 7f2d04838700 1 -- 172.21.15.167:0/3084951824 <== osd.0 v2:172.21.15.118:6800/27690 24 ==== osd_op_reply(95 notify.5 [watch watch cookie 94209790567248] v30'85 uv84 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f2cf4050040 con 0x7f2cf800d8e0 2023-10-05T19:28:30.176 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:30.170+0000 7f2d04838700 1 -- 172.21.15.167:0/3084951824 <== osd.0 v2:172.21.15.118:6800/27690 25 ==== osd_op_reply(96 notify.6 [create] v30'84 uv84 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f2cf4050040 con 0x7f2cf800d8e0 2023-10-05T19:28:30.176 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:30.170+0000 7f2d06502e40 1 -- 172.21.15.167:0/3084951824 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:97 7.14 7:2b04a3e9:::notify.6:head [watch watch cookie 94209790578208] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x55aeed1a1c10 con 0x7f2cf800d8e0 2023-10-05T19:28:30.177 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:30.170+0000 7f2d06502e40 1 -- 172.21.15.167:0/3084951824 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:98 7.9 7:93e5b521:::notify.7:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x55aeed1a2620 con 0x7f2cf800d8e0 2023-10-05T19:28:30.179 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:30.174+0000 7f2d04838700 1 -- 172.21.15.167:0/3084951824 <== osd.0 v2:172.21.15.118:6800/27690 26 ==== osd_op_reply(98 notify.7 [create] v30'84 uv84 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f2cf4050040 con 0x7f2cf800d8e0 2023-10-05T19:28:30.179 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:30.174+0000 7f2d06502e40 1 -- 172.21.15.167:0/3084951824 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:99 7.9 7:93e5b521:::notify.7:head [watch watch cookie 94209790583088] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x55aeed1a2a70 con 0x7f2cf800d8e0 2023-10-05T19:28:30.179 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:30.174+0000 7f2d06502e40 20 add_watcher() i=0 2023-10-05T19:28:30.179 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:30.174+0000 7f2d06502e40 20 add_watcher() i=1 2023-10-05T19:28:30.179 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:30.174+0000 7f2d06502e40 20 add_watcher() i=2 2023-10-05T19:28:30.179 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:30.174+0000 7f2d06502e40 20 add_watcher() i=3 2023-10-05T19:28:30.179 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:30.174+0000 7f2d06502e40 20 add_watcher() i=4 2023-10-05T19:28:30.180 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:30.174+0000 7f2d06502e40 20 add_watcher() i=5 2023-10-05T19:28:30.180 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:30.174+0000 7f2d04838700 1 -- 172.21.15.167:0/3084951824 <== osd.0 v2:172.21.15.118:6800/27690 27 ==== osd_op_reply(97 notify.6 [watch watch cookie 94209790578208] v30'85 uv84 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f2cf4050040 con 0x7f2cf800d8e0 2023-10-05T19:28:30.180 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:30.174+0000 7f2d06502e40 20 add_watcher() i=6 2023-10-05T19:28:30.181 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:30.174+0000 7f2d04838700 1 -- 172.21.15.167:0/3084951824 <== osd.0 v2:172.21.15.118:6800/27690 28 ==== osd_op_reply(99 notify.7 [watch watch cookie 94209790583088] v30'85 uv84 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f2cf4050040 con 0x7f2cf800d8e0 2023-10-05T19:28:30.181 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:30.174+0000 7f2d06502e40 20 add_watcher() i=7 2023-10-05T19:28:30.181 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:30.174+0000 7f2d06502e40 2 all 8 watchers are set, enabling cache 2023-10-05T19:28:30.181 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:30.174+0000 7f2d06502e40 1 -- 172.21.15.167:0/3084951824 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:100 6.4 6:22d26bf9:::data_loggenerations_metadata:head [call version.check_conds in=74b,call version.read in=11b,read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x55aeed1a5aa0 con 0x55aeed15f610 2023-10-05T19:28:30.182 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:30.178+0000 7f2d04838700 1 -- 172.21.15.167:0/3084951824 <== osd.1 v2:172.21.15.167:6800/30035 33 ==== osd_op_reply(100 data_loggenerations_metadata [call,call out=48b,read 0~28 out=28b] v0'0 uv1 ondisk = 0) v8 ==== 256+0+76 (crc 0 0 0) 0x7f2cf4050040 con 0x55aeed15f610 2023-10-05T19:28:30.182 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:30.178+0000 7f2d06502e40 1 -- 172.21.15.167:0/3084951824 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:101 6.4 6:22d26bf9:::data_loggenerations_metadata:head [watch watch cookie 94209790584192] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x55aeed1a2ec0 con 0x55aeed15f610 2023-10-05T19:28:30.184 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:30.178+0000 7f2d04838700 1 -- 172.21.15.167:0/3084951824 <== osd.1 v2:172.21.15.167:6800/30035 34 ==== osd_op_reply(101 data_loggenerations_metadata [watch watch cookie 94209790584192] v30'90 uv1 ondisk = 0) v8 ==== 172+0+0 (crc 0 0 0) 0x7f2cf4050040 con 0x55aeed15f610 2023-10-05T19:28:30.185 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:30.178+0000 7f2cd5fe3700 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2023-10-05T19:28:30.185 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:30.178+0000 7f2d06502e40 20 check_secure_mon_conn(): auth registy supported: methods=[2] modes=[2,1] 2023-10-05T19:28:30.185 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:30.178+0000 7f2d06502e40 20 check_secure_mon_conn(): mode 1 is insecure 2023-10-05T19:28:30.185 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:30.178+0000 7f2d06502e40 1 Processor -- start 2023-10-05T19:28:30.186 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:30.178+0000 7f2d06502e40 1 -- start start 2023-10-05T19:28:30.187 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:30.178+0000 7f2d06502e40 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55aeed1afc70 0x55aeed1b0040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-05T19:28:30.187 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:30.178+0000 7f2d06502e40 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x55aeed1b0580 con 0x55aeed1afc70 2023-10-05T19:28:30.187 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:30.182+0000 7f2d03836700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55aeed1afc70 0x55aeed1b0040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-05T19:28:30.187 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:30.182+0000 7f2d03836700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55aeed1afc70 0x55aeed1b0040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.118:3300/0 says I am v2:172.21.15.167:58050/0 (socket says 172.21.15.167:58050) 2023-10-05T19:28:30.187 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:30.182+0000 7f2d03836700 1 -- 172.21.15.167:0/4047647753 learned_addr learned my addr 172.21.15.167:0/4047647753 (peer_addr_for_me v2:172.21.15.167:0/0) 2023-10-05T19:28:30.187 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:30.182+0000 7f2d03836700 1 -- 172.21.15.167:0/4047647753 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55aeed1b06c0 con 0x55aeed1afc70 2023-10-05T19:28:30.187 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:30.182+0000 7f2d03836700 1 --2- 172.21.15.167:0/4047647753 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55aeed1afc70 0x55aeed1b0040 secure :-1 s=READY pgs=306 cs=0 l=1 rev1=1 crypto rx=0x7f2cf800be60 tx=0x7f2cf8006000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6af8825496889fe5 server_cookie=0 in_seq=0 out_seq=0 2023-10-05T19:28:30.188 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:30.182+0000 7f2cd57e2700 1 -- 172.21.15.167:0/4047647753 <== mon.0 v2:172.21.15.118:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f2cf8023040 con 0x55aeed1afc70 2023-10-05T19:28:30.188 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:30.182+0000 7f2cd57e2700 1 -- 172.21.15.167:0/4047647753 <== mon.0 v2:172.21.15.118:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2cf8024040 con 0x55aeed1afc70 2023-10-05T19:28:30.188 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:30.182+0000 7f2cd57e2700 1 -- 172.21.15.167:0/4047647753 <== mon.0 v2:172.21.15.118:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f2cf8013040 con 0x55aeed1afc70 2023-10-05T19:28:30.189 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:30.182+0000 7f2d06502e40 1 -- 172.21.15.167:0/4047647753 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55aeed1afc70 msgr2=0x55aeed1b0040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-05T19:28:30.189 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:30.182+0000 7f2d06502e40 1 --2- 172.21.15.167:0/4047647753 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55aeed1afc70 0x55aeed1b0040 secure :-1 s=READY pgs=306 cs=0 l=1 rev1=1 crypto rx=0x7f2cf800be60 tx=0x7f2cf8006000 comp rx=0 tx=0).stop 2023-10-05T19:28:30.189 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:30.182+0000 7f2d06502e40 1 -- 172.21.15.167:0/4047647753 shutdown_connections 2023-10-05T19:28:30.189 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:30.182+0000 7f2d06502e40 1 --2- 172.21.15.167:0/4047647753 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55aeed1afc70 0x55aeed1b0040 unknown :-1 s=CLOSED pgs=306 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-05T19:28:30.189 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:30.182+0000 7f2d06502e40 1 -- 172.21.15.167:0/4047647753 >> 172.21.15.167:0/4047647753 conn(0x55aeed1ab030 msgr2=0x55aeed1a91e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-05T19:28:30.189 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:30.182+0000 7f2d06502e40 1 -- 172.21.15.167:0/4047647753 shutdown_connections 2023-10-05T19:28:30.189 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:30.182+0000 7f2d06502e40 1 -- 172.21.15.167:0/4047647753 wait complete. 2023-10-05T19:28:30.190 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:30.182+0000 7f2d06502e40 1 Processor -- start 2023-10-05T19:28:30.191 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:30.182+0000 7f2d06502e40 1 -- start start 2023-10-05T19:28:30.193 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:30.182+0000 7f2d06502e40 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55aeed1afc70 0x55aeed1b5690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-05T19:28:30.193 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:30.182+0000 7f2d06502e40 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x55aeed1b6690 con 0x55aeed1afc70 2023-10-05T19:28:30.193 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:30.186+0000 7f2d03836700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55aeed1afc70 0x55aeed1b5690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-05T19:28:30.193 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:30.186+0000 7f2d03836700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55aeed1afc70 0x55aeed1b5690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.118:3300/0 says I am v2:172.21.15.167:58056/0 (socket says 172.21.15.167:58056) 2023-10-05T19:28:30.193 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:30.186+0000 7f2d03836700 1 -- 172.21.15.167:0/3205601873 learned_addr learned my addr 172.21.15.167:0/3205601873 (peer_addr_for_me v2:172.21.15.167:0/0) 2023-10-05T19:28:30.196 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:30.186+0000 7f2d03836700 1 -- 172.21.15.167:0/3205601873 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55aeed0acbd0 con 0x55aeed1afc70 2023-10-05T19:28:30.196 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:30.186+0000 7f2d03836700 1 --2- 172.21.15.167:0/3205601873 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55aeed1afc70 0x55aeed1b5690 secure :-1 s=READY pgs=307 cs=0 l=1 rev1=1 crypto rx=0x7f2cf800be60 tx=0x7f2cf8030000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-05T19:28:30.196 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:30.186+0000 7f2c4f7fe700 1 -- 172.21.15.167:0/3205601873 <== mon.0 v2:172.21.15.118:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f2cf8023040 con 0x55aeed1afc70 2023-10-05T19:28:30.196 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:30.186+0000 7f2d06502e40 1 -- 172.21.15.167:0/3205601873 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x55aeed1b5c90 con 0x55aeed1afc70 2023-10-05T19:28:30.196 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:30.186+0000 7f2c4f7fe700 1 -- 172.21.15.167:0/3205601873 <== mon.0 v2:172.21.15.118:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2cf803a420 con 0x55aeed1afc70 2023-10-05T19:28:30.196 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:30.186+0000 7f2c4f7fe700 1 -- 172.21.15.167:0/3205601873 <== mon.0 v2:172.21.15.118:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f2cf803a5b0 con 0x55aeed1afc70 2023-10-05T19:28:30.196 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:30.186+0000 7f2d06502e40 1 -- 172.21.15.167:0/3205601873 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55aeed1b60d0 con 0x55aeed1afc70 2023-10-05T19:28:30.196 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:30.186+0000 7f2c4f7fe700 1 -- 172.21.15.167:0/3205601873 <== mon.0 v2:172.21.15.118:3300/0 4 ==== mgrmap(e 5) v1 ==== 40329+0+0 (secure 0 0 0) 0x7f2cf803a820 con 0x55aeed1afc70 2023-10-05T19:28:30.197 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:30.186+0000 7f2c4f7fe700 1 --2- 172.21.15.167:0/3205601873 >> [v2:172.21.15.167:6824/29889,v1:172.21.15.167:6825/29889] conn(0x7f2c44030c00 0x7f2c440330b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-05T19:28:30.197 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:30.186+0000 7f2c4f7fe700 1 -- 172.21.15.167:0/3205601873 <== mon.0 v2:172.21.15.118:3300/0 5 ==== osd_map(30..30 src has 1..30) v4 ==== 6057+0+0 (secure 0 0 0) 0x7f2cf804fc60 con 0x55aeed1afc70 2023-10-05T19:28:30.197 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:30.186+0000 7f2d04838700 1 --2- 172.21.15.167:0/3205601873 >> [v2:172.21.15.167:6824/29889,v1:172.21.15.167:6825/29889] conn(0x7f2c44030c00 0x7f2c440330b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-05T19:28:30.197 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:30.186+0000 7f2d06502e40 5 note: GC not initialized 2023-10-05T19:28:30.197 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:30.190+0000 7f2d04838700 1 --2- 172.21.15.167:0/3205601873 >> [v2:172.21.15.167:6824/29889,v1:172.21.15.167:6825/29889] conn(0x7f2c44030c00 0x7f2c440330b0 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x55aeed1a1ad0 tx=0x7f2cf4000f90 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-05T19:28:30.197 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:30.190+0000 7f2c4cff9700 20 reqs_thread_entry: start 2023-10-05T19:28:30.198 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:30.190+0000 7f2c427fc700 20 reqs_thread_entry: start 2023-10-05T19:28:30.199 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:30.190+0000 7f2d06502e40 20 init_complete bucket index max shards: 11 2023-10-05T19:28:30.199 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:30.190+0000 7f2d06502e40 10 rgw notify: Started notification manager with: 1 workers 2023-10-05T19:28:30.199 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:30.190+0000 7f2c40ff9700 20 rgw notify: INFO: next queues processing will happen at: Thu Oct 5 19:29:00 2023 2023-10-05T19:28:30.199 INFO:teuthology.orchestra.run.smithi167.stderr: 2023-10-05T19:28:30.199 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:30.190+0000 7f2d06502e40 20 get_system_obj_state: rctx=0x7ffffcfaed78 obj=default.rgw.meta:users.uid:foo state=0x55aeed0390b0 s->prefetch_data=0 2023-10-05T19:28:30.199 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:30.190+0000 7f2d06502e40 1 -- 172.21.15.167:0/3084951824 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:102 8.18 8:1e6ca5ed:users.uid::foo:head [call version.read in=11b,getxattrs,stat] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x55aeed21dea0 con 0x55aeed15f610 2023-10-05T19:28:30.199 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:30.190+0000 7f2c27fff700 20 reqs_thread_entry: start 2023-10-05T19:28:30.199 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:30.190+0000 7f2d04838700 1 -- 172.21.15.167:0/3084951824 <== osd.1 v2:172.21.15.167:6800/30035 35 ==== osd_op_reply(102 foo [call out=48b,getxattrs out=75b,stat out=16b] v0'0 uv11 ondisk = 0) v8 ==== 231+0+139 (crc 0 0 0) 0x7f2cf4050040 con 0x55aeed15f610 2023-10-05T19:28:30.199 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:30.190+0000 7f2d06502e40 20 get_system_obj_state: s->obj_tag was set empty 2023-10-05T19:28:30.199 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:30.190+0000 7f2d06502e40 20 Read xattr: user.rgw.idtag 2023-10-05T19:28:30.199 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:30.190+0000 7f2d06502e40 20 rados->read ofs=0 len=0 2023-10-05T19:28:30.199 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:30.190+0000 7f2d06502e40 1 -- 172.21.15.167:0/3084951824 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:103 8.18 8:1e6ca5ed:users.uid::foo:head [call version.check_conds in=122b,call version.read in=11b,read 0~0,getxattrs] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x55aeed220930 con 0x55aeed15f610 2023-10-05T19:28:30.200 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:30.194+0000 7f2d04838700 1 -- 172.21.15.167:0/3084951824 <== osd.1 v2:172.21.15.167:6800/30035 36 ==== osd_op_reply(103 foo [call,call out=48b,read 0~357 out=357b,getxattrs out=75b] v0'0 uv11 ondisk = 0) v8 ==== 273+0+480 (crc 0 0 0) 0x7f2cf4050040 con 0x55aeed15f610 2023-10-05T19:28:30.200 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:30.194+0000 7f2d06502e40 20 rados_obj.operate() r=0 bl.length=357 2023-10-05T19:28:30.200 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:30.194+0000 7f2d06502e40 1 -- 172.21.15.167:0/3084951824 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:104 8.9 8:96d98480:users.uid::foo.buckets:head [call user.list_buckets in=34b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x55aeed21f920 con 0x7f2cfc037c00 2023-10-05T19:28:30.200 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:30.194+0000 7f2d03836700 1 -- 172.21.15.167:0/3084951824 <== osd.2 v2:172.21.15.167:6816/30037 17 ==== osd_op_reply(104 foo.buckets [call] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 ==== 155+0+0 (crc 0 0 0) 0x7f2cf8006070 con 0x7f2cfc037c00 2023-10-05T19:28:30.200 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:30.194+0000 7f2d06502e40 1 -- 172.21.15.167:0/3084951824 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:105 7.1f 7:f95f44c2:::notify.0:head [watch unwatch cookie 94209790559216] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x55aeed1dc290 con 0x7f2cfc037c00 2023-10-05T19:28:30.202 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:30.198+0000 7f2d03836700 1 -- 172.21.15.167:0/3084951824 <== osd.2 v2:172.21.15.167:6816/30037 18 ==== osd_op_reply(105 notify.0 [watch unwatch cookie 94209790559216] v30'86 uv84 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f2cf8006070 con 0x7f2cfc037c00 2023-10-05T19:28:30.202 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:30.198+0000 7f2d06502e40 20 remove_watcher() i=0 2023-10-05T19:28:30.203 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:30.198+0000 7f2d06502e40 2 removed watcher, disabling cache 2023-10-05T19:28:30.203 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:30.198+0000 7f2d06502e40 1 -- 172.21.15.167:0/3084951824 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:106 7.0 7:05bf5b68:::notify.1:head [watch unwatch cookie 94209790564144] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x55aeed19c3f0 con 0x55aeed15f610 2023-10-05T19:28:30.205 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:30.198+0000 7f2d04838700 1 -- 172.21.15.167:0/3084951824 <== osd.1 v2:172.21.15.167:6800/30035 37 ==== osd_op_reply(106 notify.1 [watch unwatch cookie 94209790564144] v30'86 uv84 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f2cf4050040 con 0x55aeed15f610 2023-10-05T19:28:30.205 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:30.198+0000 7f2d06502e40 20 remove_watcher() i=1 2023-10-05T19:28:30.205 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:30.198+0000 7f2d06502e40 1 -- 172.21.15.167:0/3084951824 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:107 7.15 7:a93a5511:::notify.2:head [watch unwatch cookie 94209790566144] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x55aeed19c3f0 con 0x55aeed197120 2023-10-05T19:28:30.207 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:30.202+0000 7f2d04037700 1 -- 172.21.15.167:0/3084951824 <== osd.3 v2:172.21.15.167:6808/30036 24 ==== osd_op_reply(107 notify.2 [watch unwatch cookie 94209790566144] v30'86 uv84 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f2cfc05f040 con 0x55aeed197120 2023-10-05T19:28:30.207 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:30.202+0000 7f2d06502e40 20 remove_watcher() i=2 2023-10-05T19:28:30.207 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:30.202+0000 7f2d06502e40 1 -- 172.21.15.167:0/3084951824 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:108 7.e 7:7759931f:::notify.3:head [watch unwatch cookie 94209790570560] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x55aeed19df00 con 0x7f2cfc037c00 2023-10-05T19:28:30.210 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:30.206+0000 7f2d03836700 1 -- 172.21.15.167:0/3084951824 <== osd.2 v2:172.21.15.167:6816/30037 19 ==== osd_op_reply(108 notify.3 [watch unwatch cookie 94209790570560] v30'86 uv84 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f2cf8006070 con 0x7f2cfc037c00 2023-10-05T19:28:30.210 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:30.206+0000 7f2d06502e40 20 remove_watcher() i=3 2023-10-05T19:28:30.210 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:30.206+0000 7f2d06502e40 1 -- 172.21.15.167:0/3084951824 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:109 7.d 7:b4812045:::notify.4:head [watch unwatch cookie 94209790572768] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x55aeed19df00 con 0x55aeed15f610 2023-10-05T19:28:30.213 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:30.206+0000 7f2d04838700 1 -- 172.21.15.167:0/3084951824 <== osd.1 v2:172.21.15.167:6800/30035 38 ==== osd_op_reply(109 notify.4 [watch unwatch cookie 94209790572768] v30'86 uv84 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f2cf4050040 con 0x55aeed15f610 2023-10-05T19:28:30.213 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:30.206+0000 7f2d06502e40 20 remove_watcher() i=4 2023-10-05T19:28:30.213 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:30.206+0000 7f2d06502e40 1 -- 172.21.15.167:0/3084951824 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:110 7.3 7:c609908c:::notify.5:head [watch unwatch cookie 94209790567248] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x55aeed19f8e0 con 0x7f2cf800d8e0 2023-10-05T19:28:30.216 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:30.210+0000 7f2d04838700 1 -- 172.21.15.167:0/3084951824 <== osd.0 v2:172.21.15.118:6800/27690 29 ==== osd_op_reply(110 notify.5 [watch unwatch cookie 94209790567248] v30'86 uv84 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f2cf4050040 con 0x7f2cf800d8e0 2023-10-05T19:28:30.216 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:30.210+0000 7f2d06502e40 20 remove_watcher() i=5 2023-10-05T19:28:30.216 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:30.210+0000 7f2d06502e40 1 -- 172.21.15.167:0/3084951824 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:111 7.14 7:2b04a3e9:::notify.6:head [watch unwatch cookie 94209790578208] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x55aeed19e3f0 con 0x7f2cf800d8e0 2023-10-05T19:28:30.219 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:30.214+0000 7f2d04838700 1 -- 172.21.15.167:0/3084951824 <== osd.0 v2:172.21.15.118:6800/27690 30 ==== osd_op_reply(111 notify.6 [watch unwatch cookie 94209790578208] v30'86 uv84 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f2cf4050040 con 0x7f2cf800d8e0 2023-10-05T19:28:30.219 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:30.214+0000 7f2d06502e40 20 remove_watcher() i=6 2023-10-05T19:28:30.219 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:30.214+0000 7f2d06502e40 1 -- 172.21.15.167:0/3084951824 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:112 7.9 7:93e5b521:::notify.7:head [watch unwatch cookie 94209790583088] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x55aeed19e3f0 con 0x7f2cf800d8e0 2023-10-05T19:28:30.221 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:30.214+0000 7f2d04838700 1 -- 172.21.15.167:0/3084951824 <== osd.0 v2:172.21.15.118:6800/27690 31 ==== osd_op_reply(112 notify.7 [watch unwatch cookie 94209790583088] v30'86 uv84 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f2cf4050040 con 0x7f2cf800d8e0 2023-10-05T19:28:30.221 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:30.214+0000 7f2d06502e40 20 remove_watcher() i=7 2023-10-05T19:28:30.223 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:30.218+0000 7f2cd5fe3700 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2023-10-05T19:28:30.223 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:30.218+0000 7f2d06502e40 1 -- 172.21.15.167:0/3084951824 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:113 6.4 6:22d26bf9:::data_loggenerations_metadata:head [watch unwatch cookie 94209790584192] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x55aeed19e3f0 con 0x55aeed15f610 2023-10-05T19:28:30.227 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:30.222+0000 7f2d04838700 1 -- 172.21.15.167:0/3084951824 <== osd.1 v2:172.21.15.167:6800/30035 39 ==== osd_op_reply(113 data_loggenerations_metadata [watch unwatch cookie 94209790584192] v30'91 uv1 ondisk = 0) v8 ==== 172+0+0 (crc 0 0 0) 0x7f2cf4050040 con 0x55aeed15f610 2023-10-05T19:28:30.227 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:30.222+0000 7f2d06502e40 1 -- 172.21.15.167:0/3084951824 >> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] conn(0x7f2cf800d8e0 msgr2=0x7f2cf800c360 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-05T19:28:30.227 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:30.222+0000 7f2d06502e40 1 --2- 172.21.15.167:0/3084951824 >> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] conn(0x7f2cf800d8e0 0x7f2cf800c360 crc :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-05T19:28:30.228 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:30.222+0000 7f2d06502e40 1 -- 172.21.15.167:0/3084951824 >> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] conn(0x55aeed15f610 msgr2=0x55aeed193710 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-05T19:28:30.228 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:30.222+0000 7f2d06502e40 1 --2- 172.21.15.167:0/3084951824 >> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] conn(0x55aeed15f610 0x55aeed193710 crc :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-05T19:28:30.228 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:30.222+0000 7f2d06502e40 1 -- 172.21.15.167:0/3084951824 >> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] conn(0x7f2cfc037c00 msgr2=0x7f2cfc013120 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-05T19:28:30.228 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:30.222+0000 7f2d06502e40 1 --2- 172.21.15.167:0/3084951824 >> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] conn(0x7f2cfc037c00 0x7f2cfc013120 crc :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-05T19:28:30.228 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:30.222+0000 7f2d06502e40 1 -- 172.21.15.167:0/3084951824 >> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] conn(0x55aeed197120 msgr2=0x55aeed199500 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-05T19:28:30.228 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:30.222+0000 7f2d06502e40 1 --2- 172.21.15.167:0/3084951824 >> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] conn(0x55aeed197120 0x55aeed199500 crc :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-05T19:28:30.228 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:30.222+0000 7f2d06502e40 1 -- 172.21.15.167:0/3084951824 >> [v2:172.21.15.167:6824/29889,v1:172.21.15.167:6825/29889] conn(0x7f2cf0030c50 msgr2=0x7f2cf0033100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-05T19:28:30.228 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:30.222+0000 7f2d06502e40 1 --2- 172.21.15.167:0/3084951824 >> [v2:172.21.15.167:6824/29889,v1:172.21.15.167:6825/29889] conn(0x7f2cf0030c50 0x7f2cf0033100 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7f2cf8001e90 tx=0x7f2cf80013b0 comp rx=0 tx=0).stop 2023-10-05T19:28:30.228 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:30.222+0000 7f2d06502e40 1 -- 172.21.15.167:0/3084951824 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55aeed15ed00 msgr2=0x55aeed187cb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-05T19:28:30.228 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:30.222+0000 7f2d06502e40 1 --2- 172.21.15.167:0/3084951824 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55aeed15ed00 0x55aeed187cb0 secure :-1 s=READY pgs=305 cs=0 l=1 rev1=1 crypto rx=0x7f2cfc00a470 tx=0x7f2cfc01d000 comp rx=0 tx=0).stop 2023-10-05T19:28:30.229 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:30.222+0000 7f2d04838700 1 -- 172.21.15.167:0/3084951824 reap_dead start 2023-10-05T19:28:30.229 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:30.222+0000 7f2d04838700 1 -- 172.21.15.167:0/3084951824 reap_dead start 2023-10-05T19:28:30.229 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:30.222+0000 7f2d06502e40 1 -- 172.21.15.167:0/3084951824 shutdown_connections 2023-10-05T19:28:30.229 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:30.222+0000 7f2d06502e40 1 -- 172.21.15.167:0/3084951824 >> 172.21.15.167:0/3084951824 conn(0x55aeed15a720 msgr2=0x55aeed179500 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-05T19:28:30.229 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:30.222+0000 7f2d06502e40 1 -- 172.21.15.167:0/3084951824 shutdown_connections 2023-10-05T19:28:30.229 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:30.222+0000 7f2d06502e40 1 -- 172.21.15.167:0/3084951824 wait complete. 2023-10-05T19:28:30.230 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:30.222+0000 7f2d06502e40 1 -- 172.21.15.167:0/3205601873 >> [v2:172.21.15.167:6824/29889,v1:172.21.15.167:6825/29889] conn(0x7f2c44030c00 msgr2=0x7f2c440330b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-05T19:28:30.230 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:30.222+0000 7f2d06502e40 1 --2- 172.21.15.167:0/3205601873 >> [v2:172.21.15.167:6824/29889,v1:172.21.15.167:6825/29889] conn(0x7f2c44030c00 0x7f2c440330b0 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x55aeed1a1ad0 tx=0x7f2cf4000f90 comp rx=0 tx=0).stop 2023-10-05T19:28:30.230 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:30.222+0000 7f2d06502e40 1 -- 172.21.15.167:0/3205601873 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55aeed1afc70 msgr2=0x55aeed1b5690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-05T19:28:30.230 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:30.222+0000 7f2d06502e40 1 --2- 172.21.15.167:0/3205601873 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55aeed1afc70 0x55aeed1b5690 secure :-1 s=READY pgs=307 cs=0 l=1 rev1=1 crypto rx=0x7f2cf800be60 tx=0x7f2cf8030000 comp rx=0 tx=0).stop 2023-10-05T19:28:30.231 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:30.222+0000 7f2d06502e40 1 -- 172.21.15.167:0/3205601873 shutdown_connections 2023-10-05T19:28:30.231 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:30.222+0000 7f2d06502e40 1 --2- 172.21.15.167:0/3205601873 >> [v2:172.21.15.167:6824/29889,v1:172.21.15.167:6825/29889] conn(0x7f2c44030c00 0x7f2c440330b0 unknown :-1 s=CLOSED pgs=109 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-05T19:28:30.231 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:30.222+0000 7f2d06502e40 1 --2- 172.21.15.167:0/3205601873 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55aeed1afc70 0x55aeed1b5690 unknown :-1 s=CLOSED pgs=307 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-05T19:28:30.231 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:30.222+0000 7f2d06502e40 1 -- 172.21.15.167:0/3205601873 >> 172.21.15.167:0/3205601873 conn(0x55aeed1ab030 msgr2=0x55aeed1a9130 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-05T19:28:30.231 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:30.222+0000 7f2d06502e40 1 -- 172.21.15.167:0/3205601873 shutdown_connections 2023-10-05T19:28:30.231 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:30.222+0000 7f2d06502e40 1 -- 172.21.15.167:0/3205601873 wait complete. 2023-10-05T19:28:30.231 INFO:teuthology.orchestra.run.smithi167.stdout:[] 2023-10-05T19:28:30.234 DEBUG:tasks.util.rgw: json result: [] 2023-10-05T19:28:30.273 INFO:tasks.radosgw_admin:RL: create_bucket myfoo foo bytes_out=0 bytes_in=0 failed=False 2023-10-05T19:28:30.273 INFO:tasks.util.rgw:rgwadmin: client.0 : ['bucket', 'list', '--uid', 'foo'] 2023-10-05T19:28:30.273 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'ceph', 'bucket', 'list', '--uid', 'foo'] 2023-10-05T19:28:30.273 DEBUG:teuthology.orchestra.run.smithi167:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster ceph bucket list --uid foo 2023-10-05T19:28:30.309 INFO:teuthology.orchestra.run.smithi167.stderr:ignoring --setuser ceph since I am not root 2023-10-05T19:28:30.309 INFO:teuthology.orchestra.run.smithi167.stderr:ignoring --setgroup ceph since I am not root 2023-10-05T19:28:30.310 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:30.306+0000 7effb2d2ae40 1 Processor -- start 2023-10-05T19:28:30.311 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:30.306+0000 7effb2d2ae40 1 -- start start 2023-10-05T19:28:30.311 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:30.306+0000 7effb2d2ae40 1 --2- >> v2:172.21.15.118:3300/0 conn(0x55e1218462e0 0x55e1218486c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-05T19:28:30.311 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:30.306+0000 7effb2d2ae40 1 -- --> v1:172.21.15.118:6789/0 -- auth(proto 0 26 bytes epoch 0) v1 -- 0x55e12178fbd0 con 0x55e121841d00 2023-10-05T19:28:30.311 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:30.306+0000 7effb2d2ae40 1 -- --> v2:172.21.15.118:3300/0 -- mon_getmap magic: 0 v1 -- 0x55e12174cee0 con 0x55e1218462e0 2023-10-05T19:28:30.312 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:30.306+0000 7effb085f700 1 --1- >> v1:172.21.15.118:6789/0 conn(0x55e121841d00 0x55e1218420d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.118:6789/0 says I am v1:172.21.15.167:56550/0 (socket says 172.21.15.167:56550) 2023-10-05T19:28:30.312 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:30.306+0000 7effb085f700 1 -- 172.21.15.167:0/2729011653 learned_addr learned my addr 172.21.15.167:0/2729011653 (peer_addr_for_me v1:172.21.15.167:0/0) 2023-10-05T19:28:30.313 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:30.306+0000 7effab7fe700 1 -- 172.21.15.167:0/2729011653 <== mon.0 v1:172.21.15.118:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4271237440 0 0) 0x55e12178fbd0 con 0x55e121841d00 2023-10-05T19:28:30.313 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:30.306+0000 7effab7fe700 1 -- 172.21.15.167:0/2729011653 --> v1:172.21.15.118:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7effa0001e10 con 0x55e121841d00 2023-10-05T19:28:30.313 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:30.306+0000 7effabfff700 1 --2- 172.21.15.167:0/2729011653 >> v2:172.21.15.118:3300/0 conn(0x55e1218462e0 0x55e1218486c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-05T19:28:30.313 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:30.306+0000 7effab7fe700 1 -- 172.21.15.167:0/2729011653 <== mon.0 v1:172.21.15.118:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1861916050 0 0) 0x7effa0001e10 con 0x55e121841d00 2023-10-05T19:28:30.313 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:30.306+0000 7effab7fe700 1 -- 172.21.15.167:0/2729011653 >> v2:172.21.15.118:3300/0 conn(0x55e1218462e0 msgr2=0x55e1218486c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-05T19:28:30.313 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:30.306+0000 7effab7fe700 1 --2- 172.21.15.167:0/2729011653 >> v2:172.21.15.118:3300/0 conn(0x55e1218462e0 0x55e1218486c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-05T19:28:30.313 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:30.306+0000 7effab7fe700 1 -- 172.21.15.167:0/2729011653 --> v1:172.21.15.118:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55e121842910 con 0x55e121841d00 2023-10-05T19:28:30.313 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:30.306+0000 7effab7fe700 1 -- 172.21.15.167:0/2729011653 <== mon.0 v1:172.21.15.118:6789/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (unknown 2681859047 0 0) 0x7effa40015b0 con 0x55e121841d00 2023-10-05T19:28:30.313 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:30.306+0000 7effab7fe700 1 -- 172.21.15.167:0/2729011653 >> v1:172.21.15.118:6789/0 conn(0x55e121841d00 legacy=0x55e1218420d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-05T19:28:30.313 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:30.306+0000 7effab7fe700 1 --2- 172.21.15.167:0/2729011653 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7effa0002300 0x7effa00026f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-05T19:28:30.313 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:30.306+0000 7effab7fe700 1 -- 172.21.15.167:0/2729011653 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e12174cee0 con 0x7effa0002300 2023-10-05T19:28:30.314 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:30.306+0000 7effb085f700 1 --2- 172.21.15.167:0/2729011653 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7effa0002300 0x7effa00026f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-05T19:28:30.315 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:30.306+0000 7effb085f700 1 -- 172.21.15.167:0/2729011653 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7effa0001e10 con 0x7effa0002300 2023-10-05T19:28:30.315 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:30.306+0000 7effb085f700 1 --2- 172.21.15.167:0/2729011653 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7effa0002300 0x7effa00026f0 secure :-1 s=READY pgs=309 cs=0 l=1 rev1=1 crypto rx=0x7effa4009a60 tx=0x7effa4001e70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=42602ea767ae3bf4 server_cookie=0 in_seq=0 out_seq=0 2023-10-05T19:28:30.315 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:30.306+0000 7effab7fe700 1 -- 172.21.15.167:0/2729011653 <== mon.0 v2:172.21.15.118:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7effa400b7e0 con 0x7effa0002300 2023-10-05T19:28:30.315 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:30.306+0000 7effab7fe700 1 -- 172.21.15.167:0/2729011653 <== mon.0 v2:172.21.15.118:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7effa400b940 con 0x7effa0002300 2023-10-05T19:28:30.315 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:30.306+0000 7effab7fe700 1 -- 172.21.15.167:0/2729011653 <== mon.0 v2:172.21.15.118:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7effa400baf0 con 0x7effa0002300 2023-10-05T19:28:30.315 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:30.310+0000 7effb2d2ae40 1 -- 172.21.15.167:0/2729011653 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7effa0002300 msgr2=0x7effa00026f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-05T19:28:30.315 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:30.310+0000 7effb2d2ae40 1 --2- 172.21.15.167:0/2729011653 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7effa0002300 0x7effa00026f0 secure :-1 s=READY pgs=309 cs=0 l=1 rev1=1 crypto rx=0x7effa4009a60 tx=0x7effa4001e70 comp rx=0 tx=0).stop 2023-10-05T19:28:30.315 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:30.310+0000 7effb2d2ae40 1 -- 172.21.15.167:0/2729011653 shutdown_connections 2023-10-05T19:28:30.316 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:30.310+0000 7effb2d2ae40 1 --2- 172.21.15.167:0/2729011653 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7effa0002300 0x7effa00026f0 unknown :-1 s=CLOSED pgs=309 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-05T19:28:30.316 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:30.310+0000 7effb2d2ae40 1 --2- 172.21.15.167:0/2729011653 >> v2:172.21.15.118:3300/0 conn(0x55e1218462e0 0x55e1218486c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-05T19:28:30.316 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:30.310+0000 7effb2d2ae40 1 -- 172.21.15.167:0/2729011653 >> 172.21.15.167:0/2729011653 conn(0x55e12183d720 msgr2=0x55e12183fb30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-05T19:28:30.316 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:30.310+0000 7effb2d2ae40 1 -- 172.21.15.167:0/2729011653 shutdown_connections 2023-10-05T19:28:30.316 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:30.310+0000 7effb2d2ae40 1 -- 172.21.15.167:0/2729011653 wait complete. 2023-10-05T19:28:30.317 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:30.310+0000 7effb2d2ae40 1 Processor -- start 2023-10-05T19:28:30.317 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:30.310+0000 7effb2d2ae40 1 -- start start 2023-10-05T19:28:30.317 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:30.310+0000 7effb2d2ae40 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55e121841d00 0x55e121863d40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-05T19:28:30.318 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:30.310+0000 7effb2d2ae40 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e121710e30 con 0x55e121841d00 2023-10-05T19:28:30.318 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:30.310+0000 7effb085f700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55e121841d00 0x55e121863d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-05T19:28:30.318 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:30.310+0000 7effb085f700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55e121841d00 0x55e121863d40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.118:3300/0 says I am v2:172.21.15.167:58078/0 (socket says 172.21.15.167:58078) 2023-10-05T19:28:30.318 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:30.310+0000 7effb085f700 1 -- 172.21.15.167:0/107004743 learned_addr learned my addr 172.21.15.167:0/107004743 (peer_addr_for_me v2:172.21.15.167:0/0) 2023-10-05T19:28:30.318 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:30.310+0000 7effb085f700 1 -- 172.21.15.167:0/107004743 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55e121864310 con 0x55e121841d00 2023-10-05T19:28:30.318 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:30.310+0000 7effb085f700 1 --2- 172.21.15.167:0/107004743 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55e121841d00 0x55e121863d40 secure :-1 s=READY pgs=310 cs=0 l=1 rev1=1 crypto rx=0x7effa400b5d0 tx=0x7effa4017000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e55ef62be56f287c server_cookie=0 in_seq=0 out_seq=0 2023-10-05T19:28:30.318 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:30.310+0000 7effaa7fc700 1 -- 172.21.15.167:0/107004743 <== mon.0 v2:172.21.15.118:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7effa400b7e0 con 0x55e121841d00 2023-10-05T19:28:30.318 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:30.310+0000 7effaa7fc700 1 -- 172.21.15.167:0/107004743 <== mon.0 v2:172.21.15.118:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7effa4001430 con 0x55e121841d00 2023-10-05T19:28:30.318 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:30.310+0000 7effaa7fc700 1 -- 172.21.15.167:0/107004743 <== mon.0 v2:172.21.15.118:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7effa400bab0 con 0x55e121841d00 2023-10-05T19:28:30.319 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:30.314+0000 7effb2d2ae40 1 -- 172.21.15.167:0/107004743 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55e121841d00 msgr2=0x55e121863d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-05T19:28:30.320 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:30.314+0000 7effb2d2ae40 1 --2- 172.21.15.167:0/107004743 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55e121841d00 0x55e121863d40 secure :-1 s=READY pgs=310 cs=0 l=1 rev1=1 crypto rx=0x7effa400b5d0 tx=0x7effa4017000 comp rx=0 tx=0).stop 2023-10-05T19:28:30.320 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:30.314+0000 7effb2d2ae40 1 -- 172.21.15.167:0/107004743 shutdown_connections 2023-10-05T19:28:30.320 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:30.314+0000 7effb2d2ae40 1 --2- 172.21.15.167:0/107004743 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55e121841d00 0x55e121863d40 unknown :-1 s=CLOSED pgs=310 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-05T19:28:30.320 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:30.314+0000 7effb2d2ae40 1 -- 172.21.15.167:0/107004743 >> 172.21.15.167:0/107004743 conn(0x55e12183d720 msgr2=0x55e12185b4f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-05T19:28:30.320 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:30.314+0000 7effb2d2ae40 1 -- 172.21.15.167:0/107004743 shutdown_connections 2023-10-05T19:28:30.320 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:30.314+0000 7effb2d2ae40 1 -- 172.21.15.167:0/107004743 wait complete. 2023-10-05T19:28:30.320 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:30.314+0000 7effb2d2ae40 1 Processor -- start 2023-10-05T19:28:30.320 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:30.314+0000 7effb2d2ae40 1 -- start start 2023-10-05T19:28:30.320 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:30.314+0000 7effb2d2ae40 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55e121841d00 0x55e12186ab30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-05T19:28:30.320 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:30.314+0000 7effb2d2ae40 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e1216e3ed0 con 0x55e121841d00 2023-10-05T19:28:30.321 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:30.314+0000 7effb085f700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55e121841d00 0x55e12186ab30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-05T19:28:30.322 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:30.314+0000 7effb085f700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55e121841d00 0x55e12186ab30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.118:3300/0 says I am v2:172.21.15.167:58094/0 (socket says 172.21.15.167:58094) 2023-10-05T19:28:30.322 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:30.314+0000 7effb085f700 1 -- 172.21.15.167:0/1048063379 learned_addr learned my addr 172.21.15.167:0/1048063379 (peer_addr_for_me v2:172.21.15.167:0/0) 2023-10-05T19:28:30.322 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:30.314+0000 7effb085f700 1 -- 172.21.15.167:0/1048063379 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7effa0001e10 con 0x55e121841d00 2023-10-05T19:28:30.322 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:30.314+0000 7effb085f700 1 --2- 172.21.15.167:0/1048063379 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55e121841d00 0x55e12186ab30 secure :-1 s=READY pgs=311 cs=0 l=1 rev1=1 crypto rx=0x7effa4001b70 tx=0x7effa4020000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-05T19:28:30.322 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:30.314+0000 7effa8ff9700 1 -- 172.21.15.167:0/1048063379 <== mon.0 v2:172.21.15.118:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7effa402c070 con 0x55e121841d00 2023-10-05T19:28:30.322 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:30.314+0000 7effa8ff9700 1 -- 172.21.15.167:0/1048063379 <== mon.0 v2:172.21.15.118:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7effa400e680 con 0x55e121841d00 2023-10-05T19:28:30.322 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:30.314+0000 7effb2d2ae40 1 -- 172.21.15.167:0/1048063379 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x55e12186b070 con 0x55e121841d00 2023-10-05T19:28:30.322 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:30.314+0000 7effa8ff9700 1 -- 172.21.15.167:0/1048063379 <== mon.0 v2:172.21.15.118:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7effa400e830 con 0x55e121841d00 2023-10-05T19:28:30.323 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:30.314+0000 7effb2d2ae40 1 -- 172.21.15.167:0/1048063379 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55e12186b510 con 0x55e121841d00 2023-10-05T19:28:30.323 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:30.318+0000 7effa8ff9700 1 -- 172.21.15.167:0/1048063379 <== mon.0 v2:172.21.15.118:3300/0 4 ==== mgrmap(e 5) v1 ==== 40329+0+0 (secure 0 0 0) 0x7effa400e990 con 0x55e121841d00 2023-10-05T19:28:30.324 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:30.318+0000 7effa8ff9700 1 --2- 172.21.15.167:0/1048063379 >> [v2:172.21.15.167:6824/29889,v1:172.21.15.167:6825/29889] conn(0x7eff94030ca0 0x7eff94033150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-05T19:28:30.324 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:30.318+0000 7effa8ff9700 1 -- 172.21.15.167:0/1048063379 <== mon.0 v2:172.21.15.118:3300/0 5 ==== osd_map(30..30 src has 1..30) v4 ==== 6057+0+0 (secure 0 0 0) 0x7effa4047670 con 0x55e121841d00 2023-10-05T19:28:30.324 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:30.318+0000 7effb2d2ae40 20 rados->read ofs=0 len=0 2023-10-05T19:28:30.324 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:30.318+0000 7effabfff700 1 --2- 172.21.15.167:0/1048063379 >> [v2:172.21.15.167:6824/29889,v1:172.21.15.167:6825/29889] conn(0x7eff94030ca0 0x7eff94033150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-05T19:28:30.324 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:30.318+0000 7effb2d2ae40 1 --2- 172.21.15.167:0/1048063379 >> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] conn(0x55e1218462e0 0x55e1218764e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-05T19:28:30.324 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:30.318+0000 7effb2d2ae40 1 -- 172.21.15.167:0/1048063379 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:1 5.12 5:49953fa1:::default.realm:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x55e121876be0 con 0x55e1218462e0 2023-10-05T19:28:30.324 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:30.318+0000 7effb1060700 1 --2- 172.21.15.167:0/1048063379 >> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] conn(0x55e1218462e0 0x55e1218764e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-05T19:28:30.324 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:30.318+0000 7effabfff700 1 --2- 172.21.15.167:0/1048063379 >> [v2:172.21.15.167:6824/29889,v1:172.21.15.167:6825/29889] conn(0x7eff94030ca0 0x7eff94033150 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7eff9c009830 tx=0x7eff9c00c040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-05T19:28:30.325 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:30.318+0000 7effb1060700 1 --2- 172.21.15.167:0/1048063379 >> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] conn(0x55e1218462e0 0x55e1218764e0 crc :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-05T19:28:30.325 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:30.318+0000 7effb1060700 1 -- 172.21.15.167:0/1048063379 <== osd.1 v2:172.21.15.167:6800/30035 1 ==== osd_op_reply(1 default.realm [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 ==== 157+0+0 (crc 0 0 0) 0x7effac00a5a0 con 0x55e1218462e0 2023-10-05T19:28:30.325 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:30.318+0000 7effb2d2ae40 20 rados_obj.operate() r=-2 bl.length=0 2023-10-05T19:28:30.325 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:30.318+0000 7effb2d2ae40 20 realm 2023-10-05T19:28:30.325 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:30.318+0000 7effb2d2ae40 20 rados->read ofs=0 len=0 2023-10-05T19:28:30.326 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:30.318+0000 7effb2d2ae40 1 -- 172.21.15.167:0/1048063379 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:2 5.12 5:49953fa1:::default.realm:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x55e121873b40 con 0x55e1218462e0 2023-10-05T19:28:30.326 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:30.318+0000 7effb1060700 1 -- 172.21.15.167:0/1048063379 <== osd.1 v2:172.21.15.167:6800/30035 2 ==== osd_op_reply(2 default.realm [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 ==== 157+0+0 (crc 0 0 0) 0x7effac00a5a0 con 0x55e1218462e0 2023-10-05T19:28:30.327 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:30.318+0000 7effb2d2ae40 20 rados_obj.operate() r=-2 bl.length=0 2023-10-05T19:28:30.327 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:30.318+0000 7effb2d2ae40 4 RGWPeriod::init failed to init realm id : (2) No such file or directory 2023-10-05T19:28:30.327 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:30.318+0000 7effb2d2ae40 20 rados->read ofs=0 len=0 2023-10-05T19:28:30.327 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:30.318+0000 7effb2d2ae40 1 -- 172.21.15.167:0/1048063379 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:3 5.12 5:49953fa1:::default.realm:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x55e121877710 con 0x55e1218462e0 2023-10-05T19:28:30.327 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:30.322+0000 7effb1060700 1 -- 172.21.15.167:0/1048063379 <== osd.1 v2:172.21.15.167:6800/30035 3 ==== osd_op_reply(3 default.realm [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 ==== 157+0+0 (crc 0 0 0) 0x7effac00a5a0 con 0x55e1218462e0 2023-10-05T19:28:30.327 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:30.322+0000 7effb2d2ae40 20 rados_obj.operate() r=-2 bl.length=0 2023-10-05T19:28:30.327 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:30.322+0000 7effb2d2ae40 20 rados->read ofs=0 len=0 2023-10-05T19:28:30.327 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:30.322+0000 7effb2d2ae40 1 -- 172.21.15.167:0/1048063379 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:4 5.f 5:f43b6ece:::zone_names.default:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x55e1218772f0 con 0x55e1218462e0 2023-10-05T19:28:30.328 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:30.322+0000 7effb1060700 1 -- 172.21.15.167:0/1048063379 <== osd.1 v2:172.21.15.167:6800/30035 4 ==== osd_op_reply(4 zone_names.default [read 0~46 out=46b] v0'0 uv1 ondisk = 0) v8 ==== 162+0+46 (crc 0 0 0) 0x7effac00a5a0 con 0x55e1218462e0 2023-10-05T19:28:30.329 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:30.322+0000 7effb2d2ae40 20 rados_obj.operate() r=0 bl.length=46 2023-10-05T19:28:30.329 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:30.322+0000 7effb2d2ae40 20 rados->read ofs=0 len=0 2023-10-05T19:28:30.329 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:30.322+0000 7effb2d2ae40 1 -- 172.21.15.167:0/1048063379 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:5 5.1d 5:b8374eb4:::zone_info.9a9df965-6612-45d4-a45f-6e3c889bacc2:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x55e121878180 con 0x55e1218462e0 2023-10-05T19:28:30.329 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:30.322+0000 7effb1060700 1 -- 172.21.15.167:0/1048063379 <== osd.1 v2:172.21.15.167:6800/30035 5 ==== osd_op_reply(5 zone_info.9a9df965-6612-45d4-a45f-6e3c889bacc2 [read 0~874 out=874b] v0'0 uv2 ondisk = 0) v8 ==== 190+0+874 (crc 0 0 0) 0x7effac00a5a0 con 0x55e1218462e0 2023-10-05T19:28:30.329 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:30.322+0000 7effb2d2ae40 20 rados_obj.operate() r=0 bl.length=874 2023-10-05T19:28:30.329 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:30.322+0000 7effb2d2ae40 20 searching for the correct realm 2023-10-05T19:28:30.329 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:30.322+0000 7effb2d2ae40 1 --2- 172.21.15.167:0/1048063379 >> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] conn(0x7effa0002300 0x55e12187bfe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-05T19:28:30.329 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:30.322+0000 7effb2d2ae40 1 -- 172.21.15.167:0/1048063379 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:6 5.0 5:00000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x55e12187c520 con 0x7effa0002300 2023-10-05T19:28:30.329 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:30.322+0000 7effb085f700 1 --2- 172.21.15.167:0/1048063379 >> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] conn(0x7effa0002300 0x55e12187bfe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-05T19:28:30.330 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:30.322+0000 7effb085f700 1 --2- 172.21.15.167:0/1048063379 >> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] conn(0x7effa0002300 0x55e12187bfe0 crc :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-05T19:28:30.331 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:30.326+0000 7effb085f700 1 -- 172.21.15.167:0/1048063379 <== osd.3 v2:172.21.15.167:6808/30036 1 ==== osd_op_reply(6 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7effa40473a0 con 0x7effa0002300 2023-10-05T19:28:30.331 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:30.326+0000 7effb085f700 1 --2- 172.21.15.167:0/1048063379 >> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] conn(0x7effa400d750 0x7effa403a610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-05T19:28:30.331 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:30.326+0000 7effb085f700 1 -- 172.21.15.167:0/1048063379 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:7 5.10 5:08000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x55e12187c520 con 0x7effa400d750 2023-10-05T19:28:30.331 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:30.326+0000 7effabfff700 1 --2- 172.21.15.167:0/1048063379 >> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] conn(0x7effa400d750 0x7effa403a610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-05T19:28:30.332 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:30.326+0000 7effabfff700 1 --2- 172.21.15.167:0/1048063379 >> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] conn(0x7effa400d750 0x7effa403a610 crc :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-05T19:28:30.332 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:30.326+0000 7effabfff700 1 -- 172.21.15.167:0/1048063379 <== osd.2 v2:172.21.15.167:6816/30037 1 ==== osd_op_reply(7 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7eff9c01d040 con 0x7effa400d750 2023-10-05T19:28:30.333 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:30.326+0000 7effabfff700 1 --2- 172.21.15.167:0/1048063379 >> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] conn(0x7eff9c015920 0x7eff9c014370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-05T19:28:30.333 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:30.326+0000 7effabfff700 1 -- 172.21.15.167:0/1048063379 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:8 5.8 5:10000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x55e12187c520 con 0x7eff9c015920 2023-10-05T19:28:30.333 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:30.326+0000 7effb1060700 1 --2- 172.21.15.167:0/1048063379 >> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] conn(0x7eff9c015920 0x7eff9c014370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-05T19:28:30.333 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:30.326+0000 7effb1060700 1 --2- 172.21.15.167:0/1048063379 >> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] conn(0x7eff9c015920 0x7eff9c014370 crc :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-05T19:28:30.334 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:30.330+0000 7effb1060700 1 -- 172.21.15.167:0/1048063379 <== osd.0 v2:172.21.15.118:6800/27690 1 ==== osd_op_reply(8 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7effac00a5a0 con 0x7eff9c015920 2023-10-05T19:28:30.334 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:30.330+0000 7effb1060700 1 -- 172.21.15.167:0/1048063379 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:9 5.18 5:18000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x55e12187c520 con 0x55e1218462e0 2023-10-05T19:28:30.335 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:30.330+0000 7effb1060700 1 -- 172.21.15.167:0/1048063379 <== osd.1 v2:172.21.15.167:6800/30035 6 ==== osd_op_reply(9 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7effac00a5a0 con 0x55e1218462e0 2023-10-05T19:28:30.335 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:30.330+0000 7effb1060700 1 -- 172.21.15.167:0/1048063379 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:10 5.4 5:20000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x55e12187c520 con 0x55e1218462e0 2023-10-05T19:28:30.336 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:30.330+0000 7effb1060700 1 -- 172.21.15.167:0/1048063379 <== osd.1 v2:172.21.15.167:6800/30035 7 ==== osd_op_reply(10 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7effac00a5a0 con 0x55e1218462e0 2023-10-05T19:28:30.336 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:30.330+0000 7effb1060700 1 -- 172.21.15.167:0/1048063379 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:11 5.14 5:28000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x55e12187c520 con 0x7eff9c015920 2023-10-05T19:28:30.337 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:30.330+0000 7effb1060700 1 -- 172.21.15.167:0/1048063379 <== osd.0 v2:172.21.15.118:6800/27690 2 ==== osd_op_reply(11 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7effac00a5a0 con 0x7eff9c015920 2023-10-05T19:28:30.337 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:30.330+0000 7effb1060700 1 -- 172.21.15.167:0/1048063379 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:12 5.c 5:30000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x55e12187c520 con 0x7eff9c015920 2023-10-05T19:28:30.337 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:30.330+0000 7effb1060700 1 -- 172.21.15.167:0/1048063379 <== osd.0 v2:172.21.15.118:6800/27690 3 ==== osd_op_reply(12 [pgnls start_epoch 30 out=112b] v23'1 uv1 ondisk = 1) v8 ==== 144+0+112 (crc 0 0 0) 0x7effac00a5a0 con 0x7eff9c015920 2023-10-05T19:28:30.338 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:30.330+0000 7effb1060700 1 -- 172.21.15.167:0/1048063379 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:13 5.1c 5:38000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x55e12187c520 con 0x7effa400d750 2023-10-05T19:28:30.338 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:30.334+0000 7effabfff700 1 -- 172.21.15.167:0/1048063379 <== osd.2 v2:172.21.15.167:6816/30037 2 ==== osd_op_reply(13 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7eff9c01d040 con 0x7effa400d750 2023-10-05T19:28:30.338 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:30.334+0000 7effabfff700 1 -- 172.21.15.167:0/1048063379 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:14 5.2 5:40000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x55e12187c520 con 0x7effa0002300 2023-10-05T19:28:30.339 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:30.334+0000 7effb085f700 1 -- 172.21.15.167:0/1048063379 <== osd.3 v2:172.21.15.167:6808/30036 2 ==== osd_op_reply(14 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7effa40473a0 con 0x7effa0002300 2023-10-05T19:28:30.339 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:30.334+0000 7effb085f700 1 -- 172.21.15.167:0/1048063379 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:15 5.12 5:48000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x55e12187c520 con 0x55e1218462e0 2023-10-05T19:28:30.340 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:30.334+0000 7effb1060700 1 -- 172.21.15.167:0/1048063379 <== osd.1 v2:172.21.15.167:6800/30035 8 ==== osd_op_reply(15 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7effac00a5a0 con 0x55e1218462e0 2023-10-05T19:28:30.340 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:30.334+0000 7effb1060700 1 -- 172.21.15.167:0/1048063379 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:16 5.a 5:50000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x55e12187c520 con 0x7eff9c015920 2023-10-05T19:28:30.340 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:30.334+0000 7effb1060700 1 -- 172.21.15.167:0/1048063379 <== osd.0 v2:172.21.15.118:6800/27690 4 ==== osd_op_reply(16 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7effac00a5a0 con 0x7eff9c015920 2023-10-05T19:28:30.341 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:30.334+0000 7effb1060700 1 -- 172.21.15.167:0/1048063379 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:17 5.1a 5:58000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x55e12187c520 con 0x55e1218462e0 2023-10-05T19:28:30.341 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:30.334+0000 7effb1060700 1 -- 172.21.15.167:0/1048063379 <== osd.1 v2:172.21.15.167:6800/30035 9 ==== osd_op_reply(17 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7effac00a5a0 con 0x55e1218462e0 2023-10-05T19:28:30.341 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:30.334+0000 7effb1060700 1 -- 172.21.15.167:0/1048063379 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:18 5.6 5:60000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x55e12187c520 con 0x7effa400d750 2023-10-05T19:28:30.342 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:30.338+0000 7effabfff700 1 -- 172.21.15.167:0/1048063379 <== osd.2 v2:172.21.15.167:6816/30037 3 ==== osd_op_reply(18 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7eff9c01d040 con 0x7effa400d750 2023-10-05T19:28:30.342 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:30.338+0000 7effabfff700 1 -- 172.21.15.167:0/1048063379 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:19 5.16 5:68000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x55e12187c520 con 0x7effa0002300 2023-10-05T19:28:30.343 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:30.338+0000 7effb085f700 1 -- 172.21.15.167:0/1048063379 <== osd.3 v2:172.21.15.167:6808/30036 3 ==== osd_op_reply(19 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7effa40473a0 con 0x7effa0002300 2023-10-05T19:28:30.343 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:30.338+0000 7effb085f700 1 -- 172.21.15.167:0/1048063379 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:20 5.e 5:70000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x55e12187c520 con 0x7effa0002300 2023-10-05T19:28:30.344 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:30.338+0000 7effb085f700 1 -- 172.21.15.167:0/1048063379 <== osd.3 v2:172.21.15.167:6808/30036 4 ==== osd_op_reply(20 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7effa40473a0 con 0x7effa0002300 2023-10-05T19:28:30.344 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:30.338+0000 7effb085f700 1 -- 172.21.15.167:0/1048063379 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:21 5.1e 5:78000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x55e12187c520 con 0x7eff9c015920 2023-10-05T19:28:30.344 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:30.338+0000 7effb1060700 1 -- 172.21.15.167:0/1048063379 <== osd.0 v2:172.21.15.118:6800/27690 5 ==== osd_op_reply(21 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7effac00a5a0 con 0x7eff9c015920 2023-10-05T19:28:30.345 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:30.338+0000 7effb1060700 1 -- 172.21.15.167:0/1048063379 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:22 5.1 5:80000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x55e12187c520 con 0x7effa0002300 2023-10-05T19:28:30.345 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:30.338+0000 7effb085f700 1 -- 172.21.15.167:0/1048063379 <== osd.3 v2:172.21.15.167:6808/30036 5 ==== osd_op_reply(22 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7effa40473a0 con 0x7effa0002300 2023-10-05T19:28:30.345 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:30.338+0000 7effb085f700 1 -- 172.21.15.167:0/1048063379 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:23 5.11 5:88000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x55e12187c520 con 0x7effa0002300 2023-10-05T19:28:30.346 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:30.342+0000 7effb085f700 1 -- 172.21.15.167:0/1048063379 <== osd.3 v2:172.21.15.167:6808/30036 6 ==== osd_op_reply(23 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7effa40473a0 con 0x7effa0002300 2023-10-05T19:28:30.346 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:30.342+0000 7effb085f700 1 -- 172.21.15.167:0/1048063379 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:24 5.9 5:90000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x55e12187c520 con 0x55e1218462e0 2023-10-05T19:28:30.347 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:30.342+0000 7effb1060700 1 -- 172.21.15.167:0/1048063379 <== osd.1 v2:172.21.15.167:6800/30035 10 ==== osd_op_reply(24 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7effac00a5a0 con 0x55e1218462e0 2023-10-05T19:28:30.347 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:30.342+0000 7effb1060700 1 -- 172.21.15.167:0/1048063379 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:25 5.19 5:98000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x55e12187c520 con 0x55e1218462e0 2023-10-05T19:28:30.348 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:30.342+0000 7effb1060700 1 -- 172.21.15.167:0/1048063379 <== osd.1 v2:172.21.15.167:6800/30035 11 ==== osd_op_reply(25 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7effac00a5a0 con 0x55e1218462e0 2023-10-05T19:28:30.348 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:30.342+0000 7effb1060700 1 -- 172.21.15.167:0/1048063379 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:26 5.5 5:a0000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x55e12187c520 con 0x7eff9c015920 2023-10-05T19:28:30.349 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:30.342+0000 7effb1060700 1 -- 172.21.15.167:0/1048063379 <== osd.0 v2:172.21.15.118:6800/27690 6 ==== osd_op_reply(26 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7effac00a5a0 con 0x7eff9c015920 2023-10-05T19:28:30.349 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:30.342+0000 7effb1060700 1 -- 172.21.15.167:0/1048063379 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:27 5.15 5:a8000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x55e12187c520 con 0x7effa0002300 2023-10-05T19:28:30.349 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:30.342+0000 7effb085f700 1 -- 172.21.15.167:0/1048063379 <== osd.3 v2:172.21.15.167:6808/30036 7 ==== osd_op_reply(27 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7effa40473a0 con 0x7effa0002300 2023-10-05T19:28:30.350 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:30.342+0000 7effb085f700 1 -- 172.21.15.167:0/1048063379 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:28 5.d 5:b0000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x55e12187c520 con 0x7effa400d750 2023-10-05T19:28:30.350 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:30.346+0000 7effabfff700 1 -- 172.21.15.167:0/1048063379 <== osd.2 v2:172.21.15.167:6816/30037 4 ==== osd_op_reply(28 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7eff9c01d040 con 0x7effa400d750 2023-10-05T19:28:30.350 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:30.346+0000 7effabfff700 1 -- 172.21.15.167:0/1048063379 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:29 5.1d 5:b8000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x55e12187c520 con 0x55e1218462e0 2023-10-05T19:28:30.351 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:30.346+0000 7effb1060700 1 -- 172.21.15.167:0/1048063379 <== osd.1 v2:172.21.15.167:6800/30035 12 ==== osd_op_reply(29 [pgnls start_epoch 30 out=107b] v30'2 uv2 ondisk = 1) v8 ==== 144+0+107 (crc 0 0 0) 0x7effac00a5a0 con 0x55e1218462e0 2023-10-05T19:28:30.351 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:30.346+0000 7effb1060700 1 -- 172.21.15.167:0/1048063379 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:30 5.3 5:c0000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x55e12187c520 con 0x7eff9c015920 2023-10-05T19:28:30.352 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:30.346+0000 7effb1060700 1 -- 172.21.15.167:0/1048063379 <== osd.0 v2:172.21.15.118:6800/27690 7 ==== osd_op_reply(30 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7effac00a5a0 con 0x7eff9c015920 2023-10-05T19:28:30.352 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:30.346+0000 7effb1060700 1 -- 172.21.15.167:0/1048063379 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:31 5.13 5:c8000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x55e12187c520 con 0x7effa0002300 2023-10-05T19:28:30.353 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:30.346+0000 7effb085f700 1 -- 172.21.15.167:0/1048063379 <== osd.3 v2:172.21.15.167:6808/30036 8 ==== osd_op_reply(31 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7effa40473a0 con 0x7effa0002300 2023-10-05T19:28:30.353 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:30.346+0000 7effb085f700 1 -- 172.21.15.167:0/1048063379 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:32 5.b 5:d0000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x55e12187c520 con 0x7eff9c015920 2023-10-05T19:28:30.354 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:30.346+0000 7effb1060700 1 -- 172.21.15.167:0/1048063379 <== osd.0 v2:172.21.15.118:6800/27690 8 ==== osd_op_reply(32 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7effac00a5a0 con 0x7eff9c015920 2023-10-05T19:28:30.354 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:30.346+0000 7effb1060700 1 -- 172.21.15.167:0/1048063379 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:33 5.1b 5:d8000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x55e12187c520 con 0x7effa400d750 2023-10-05T19:28:30.354 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:30.350+0000 7effabfff700 1 -- 172.21.15.167:0/1048063379 <== osd.2 v2:172.21.15.167:6816/30037 5 ==== osd_op_reply(33 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7eff9c01d040 con 0x7effa400d750 2023-10-05T19:28:30.354 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:30.350+0000 7effabfff700 1 -- 172.21.15.167:0/1048063379 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:34 5.7 5:e0000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x55e12187c520 con 0x7effa0002300 2023-10-05T19:28:30.355 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:30.350+0000 7effb085f700 1 -- 172.21.15.167:0/1048063379 <== osd.3 v2:172.21.15.167:6808/30036 9 ==== osd_op_reply(34 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7effa40473a0 con 0x7effa0002300 2023-10-05T19:28:30.355 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:30.350+0000 7effb085f700 1 -- 172.21.15.167:0/1048063379 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:35 5.17 5:e8000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x55e12187c520 con 0x7eff9c015920 2023-10-05T19:28:30.356 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:30.350+0000 7effb1060700 1 -- 172.21.15.167:0/1048063379 <== osd.0 v2:172.21.15.118:6800/27690 9 ==== osd_op_reply(35 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7effac00a5a0 con 0x7eff9c015920 2023-10-05T19:28:30.357 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:30.350+0000 7effb1060700 1 -- 172.21.15.167:0/1048063379 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:36 5.f 5:f0000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x55e12187c520 con 0x55e1218462e0 2023-10-05T19:28:30.357 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:30.350+0000 7effb1060700 1 -- 172.21.15.167:0/1048063379 <== osd.1 v2:172.21.15.167:6800/30035 13 ==== osd_op_reply(36 [pgnls start_epoch 30 out=115b] v23'2 uv2 ondisk = 1) v8 ==== 144+0+115 (crc 0 0 0) 0x7effac00a5a0 con 0x55e1218462e0 2023-10-05T19:28:30.358 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:30.350+0000 7effb1060700 1 -- 172.21.15.167:0/1048063379 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:37 5.1f 5:f8000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x55e12187c520 con 0x7effa400d750 2023-10-05T19:28:30.359 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:30.354+0000 7effabfff700 1 -- 172.21.15.167:0/1048063379 <== osd.2 v2:172.21.15.167:6816/30037 6 ==== osd_op_reply(37 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7eff9c01d040 con 0x7effa400d750 2023-10-05T19:28:30.359 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:30.354+0000 7effb2d2ae40 20 RGWRados::pool_iterate: got zonegroup_info.4f1f719d-9648-417b-b10c-e5d939825419 2023-10-05T19:28:30.359 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:30.354+0000 7effb2d2ae40 20 RGWRados::pool_iterate: got zone_info.9a9df965-6612-45d4-a45f-6e3c889bacc2 2023-10-05T19:28:30.359 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:30.354+0000 7effb2d2ae40 20 RGWRados::pool_iterate: got zone_names.default 2023-10-05T19:28:30.359 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:30.354+0000 7effb2d2ae40 20 RGWRados::pool_iterate: got zonegroups_names.default 2023-10-05T19:28:30.359 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:30.354+0000 7effb2d2ae40 20 rados->read ofs=0 len=0 2023-10-05T19:28:30.359 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:30.354+0000 7effb2d2ae40 1 -- 172.21.15.167:0/1048063379 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:38 5.12 5:49953fa1:::default.realm:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x55e121878d00 con 0x55e1218462e0 2023-10-05T19:28:30.360 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:30.354+0000 7effb1060700 1 -- 172.21.15.167:0/1048063379 <== osd.1 v2:172.21.15.167:6800/30035 14 ==== osd_op_reply(38 default.realm [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 ==== 157+0+0 (crc 0 0 0) 0x7effac00a5a0 con 0x55e1218462e0 2023-10-05T19:28:30.360 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:30.354+0000 7effb2d2ae40 20 rados_obj.operate() r=-2 bl.length=0 2023-10-05T19:28:30.360 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:30.354+0000 7effb2d2ae40 20 rados->read ofs=0 len=0 2023-10-05T19:28:30.360 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:30.354+0000 7effb2d2ae40 1 -- 172.21.15.167:0/1048063379 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:39 5.f 5:f4c53578:::zonegroups_names.default:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x55e121878900 con 0x55e1218462e0 2023-10-05T19:28:30.360 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:30.354+0000 7effb1060700 1 -- 172.21.15.167:0/1048063379 <== osd.1 v2:172.21.15.167:6800/30035 15 ==== osd_op_reply(39 zonegroups_names.default [read 0~46 out=46b] v0'0 uv2 ondisk = 0) v8 ==== 168+0+46 (crc 0 0 0) 0x7effac00a5a0 con 0x55e1218462e0 2023-10-05T19:28:30.361 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:30.354+0000 7effb2d2ae40 20 rados_obj.operate() r=0 bl.length=46 2023-10-05T19:28:30.361 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:30.354+0000 7effb2d2ae40 20 rados->read ofs=0 len=0 2023-10-05T19:28:30.361 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:30.354+0000 7effb2d2ae40 1 -- 172.21.15.167:0/1048063379 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:40 5.c 5:30d299a5:::zonegroup_info.4f1f719d-9648-417b-b10c-e5d939825419:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x55e1218788d0 con 0x7eff9c015920 2023-10-05T19:28:30.362 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:30.354+0000 7effb1060700 1 -- 172.21.15.167:0/1048063379 <== osd.0 v2:172.21.15.118:6800/27690 10 ==== osd_op_reply(40 zonegroup_info.4f1f719d-9648-417b-b10c-e5d939825419 [read 0~362 out=362b] v0'0 uv1 ondisk = 0) v8 ==== 195+0+362 (crc 0 0 0) 0x7effac00a5a0 con 0x7eff9c015920 2023-10-05T19:28:30.362 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:30.354+0000 7effb2d2ae40 20 rados_obj.operate() r=0 bl.length=362 2023-10-05T19:28:30.362 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:30.354+0000 7effb2d2ae40 20 rados->read ofs=0 len=0 2023-10-05T19:28:30.362 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:30.354+0000 7effb2d2ae40 1 -- 172.21.15.167:0/1048063379 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:41 5.13 5:cbbf6c60:::converted:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x55e121879690 con 0x7effa0002300 2023-10-05T19:28:30.362 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:30.354+0000 7effb085f700 1 -- 172.21.15.167:0/1048063379 <== osd.3 v2:172.21.15.167:6808/30036 10 ==== osd_op_reply(41 converted [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 ==== 153+0+0 (crc 0 0 0) 0x7effa40473a0 con 0x7effa0002300 2023-10-05T19:28:30.362 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:30.354+0000 7effb2d2ae40 20 rados_obj.operate() r=-2 bl.length=0 2023-10-05T19:28:30.362 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:30.354+0000 7effb2d2ae40 20 rados->read ofs=0 len=0 2023-10-05T19:28:30.362 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:30.354+0000 7effb2d2ae40 1 -- 172.21.15.167:0/1048063379 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:42 5.12 5:49953fa1:::default.realm:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x55e12187c960 con 0x55e1218462e0 2023-10-05T19:28:30.363 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:30.358+0000 7effb1060700 1 -- 172.21.15.167:0/1048063379 <== osd.1 v2:172.21.15.167:6800/30035 16 ==== osd_op_reply(42 default.realm [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 ==== 157+0+0 (crc 0 0 0) 0x7effac00a5a0 con 0x55e1218462e0 2023-10-05T19:28:30.363 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:30.358+0000 7effb2d2ae40 20 rados_obj.operate() r=-2 bl.length=0 2023-10-05T19:28:30.363 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:30.358+0000 7effb2d2ae40 20 rados->read ofs=0 len=0 2023-10-05T19:28:30.363 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:30.358+0000 7effb2d2ae40 1 -- 172.21.15.167:0/1048063379 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:43 5.f 5:f4c53578:::zonegroups_names.default:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x55e12187c960 con 0x55e1218462e0 2023-10-05T19:28:30.364 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:30.358+0000 7effb1060700 1 -- 172.21.15.167:0/1048063379 <== osd.1 v2:172.21.15.167:6800/30035 17 ==== osd_op_reply(43 zonegroups_names.default [read 0~46 out=46b] v0'0 uv2 ondisk = 0) v8 ==== 168+0+46 (crc 0 0 0) 0x7effac00a5a0 con 0x55e1218462e0 2023-10-05T19:28:30.364 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:30.358+0000 7effb2d2ae40 20 rados_obj.operate() r=0 bl.length=46 2023-10-05T19:28:30.364 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:30.358+0000 7effb2d2ae40 1 -- 172.21.15.167:0/1048063379 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:44 5.0 5:00000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x55e12187d970 con 0x7effa0002300 2023-10-05T19:28:30.366 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:30.358+0000 7effb085f700 1 -- 172.21.15.167:0/1048063379 <== osd.3 v2:172.21.15.167:6808/30036 11 ==== osd_op_reply(44 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7effa40473a0 con 0x7effa0002300 2023-10-05T19:28:30.366 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:30.362+0000 7effb085f700 1 -- 172.21.15.167:0/1048063379 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:45 5.10 5:08000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x55e12187d970 con 0x7effa400d750 2023-10-05T19:28:30.366 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:30.362+0000 7effabfff700 1 -- 172.21.15.167:0/1048063379 <== osd.2 v2:172.21.15.167:6816/30037 7 ==== osd_op_reply(45 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7eff9c01d040 con 0x7effa400d750 2023-10-05T19:28:30.367 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:30.362+0000 7effabfff700 1 -- 172.21.15.167:0/1048063379 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:46 5.8 5:10000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x55e12187d970 con 0x7eff9c015920 2023-10-05T19:28:30.367 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:30.362+0000 7effb1060700 1 -- 172.21.15.167:0/1048063379 <== osd.0 v2:172.21.15.118:6800/27690 11 ==== osd_op_reply(46 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7effac00a5a0 con 0x7eff9c015920 2023-10-05T19:28:30.368 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:30.362+0000 7effb1060700 1 -- 172.21.15.167:0/1048063379 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:47 5.18 5:18000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x55e121878d00 con 0x55e1218462e0 2023-10-05T19:28:30.368 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:30.362+0000 7effb1060700 1 -- 172.21.15.167:0/1048063379 <== osd.1 v2:172.21.15.167:6800/30035 18 ==== osd_op_reply(47 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7effac00a5a0 con 0x55e1218462e0 2023-10-05T19:28:30.369 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:30.362+0000 7effb1060700 1 -- 172.21.15.167:0/1048063379 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:48 5.4 5:20000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x55e121878d00 con 0x55e1218462e0 2023-10-05T19:28:30.369 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:30.362+0000 7effb1060700 1 -- 172.21.15.167:0/1048063379 <== osd.1 v2:172.21.15.167:6800/30035 19 ==== osd_op_reply(48 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7effac00a5a0 con 0x55e1218462e0 2023-10-05T19:28:30.369 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:30.362+0000 7effb1060700 1 -- 172.21.15.167:0/1048063379 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:49 5.14 5:28000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x55e121878d00 con 0x7eff9c015920 2023-10-05T19:28:30.370 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:30.366+0000 7effb1060700 1 -- 172.21.15.167:0/1048063379 <== osd.0 v2:172.21.15.118:6800/27690 12 ==== osd_op_reply(49 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7effac00a5a0 con 0x7eff9c015920 2023-10-05T19:28:30.370 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:30.366+0000 7effb1060700 1 -- 172.21.15.167:0/1048063379 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:50 5.c 5:30000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x55e121878d00 con 0x7eff9c015920 2023-10-05T19:28:30.372 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:30.366+0000 7effb1060700 1 -- 172.21.15.167:0/1048063379 <== osd.0 v2:172.21.15.118:6800/27690 13 ==== osd_op_reply(50 [pgnls start_epoch 30 out=112b] v23'1 uv1 ondisk = 1) v8 ==== 144+0+112 (crc 0 0 0) 0x7effac00a5a0 con 0x7eff9c015920 2023-10-05T19:28:30.372 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:30.366+0000 7effb1060700 1 -- 172.21.15.167:0/1048063379 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:51 5.1c 5:38000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x55e121878d00 con 0x7effa400d750 2023-10-05T19:28:30.372 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:30.366+0000 7effabfff700 1 -- 172.21.15.167:0/1048063379 <== osd.2 v2:172.21.15.167:6816/30037 8 ==== osd_op_reply(51 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7eff9c01d040 con 0x7effa400d750 2023-10-05T19:28:30.373 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:30.366+0000 7effabfff700 1 -- 172.21.15.167:0/1048063379 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:52 5.2 5:40000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x55e121878d00 con 0x7effa0002300 2023-10-05T19:28:30.373 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:30.366+0000 7effb085f700 1 -- 172.21.15.167:0/1048063379 <== osd.3 v2:172.21.15.167:6808/30036 12 ==== osd_op_reply(52 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7effa40473a0 con 0x7effa0002300 2023-10-05T19:28:30.373 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:30.366+0000 7effb085f700 1 -- 172.21.15.167:0/1048063379 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:53 5.12 5:48000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x55e121878d00 con 0x55e1218462e0 2023-10-05T19:28:30.374 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:30.370+0000 7effb1060700 1 -- 172.21.15.167:0/1048063379 <== osd.1 v2:172.21.15.167:6800/30035 20 ==== osd_op_reply(53 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7effac00a5a0 con 0x55e1218462e0 2023-10-05T19:28:30.374 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:30.370+0000 7effb1060700 1 -- 172.21.15.167:0/1048063379 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:54 5.a 5:50000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x55e121878d00 con 0x7eff9c015920 2023-10-05T19:28:30.375 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:30.370+0000 7effb1060700 1 -- 172.21.15.167:0/1048063379 <== osd.0 v2:172.21.15.118:6800/27690 14 ==== osd_op_reply(54 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7effac00a5a0 con 0x7eff9c015920 2023-10-05T19:28:30.375 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:30.370+0000 7effb1060700 1 -- 172.21.15.167:0/1048063379 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:55 5.1a 5:58000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x55e121878d00 con 0x55e1218462e0 2023-10-05T19:28:30.376 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:30.370+0000 7effb1060700 1 -- 172.21.15.167:0/1048063379 <== osd.1 v2:172.21.15.167:6800/30035 21 ==== osd_op_reply(55 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7effac00a5a0 con 0x55e1218462e0 2023-10-05T19:28:30.376 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:30.370+0000 7effb1060700 1 -- 172.21.15.167:0/1048063379 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:56 5.6 5:60000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x55e121878d00 con 0x7effa400d750 2023-10-05T19:28:30.377 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:30.370+0000 7effabfff700 1 -- 172.21.15.167:0/1048063379 <== osd.2 v2:172.21.15.167:6816/30037 9 ==== osd_op_reply(56 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7eff9c01d040 con 0x7effa400d750 2023-10-05T19:28:30.377 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:30.370+0000 7effabfff700 1 -- 172.21.15.167:0/1048063379 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:57 5.16 5:68000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x55e121878d00 con 0x7effa0002300 2023-10-05T19:28:30.378 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:30.370+0000 7effb085f700 1 -- 172.21.15.167:0/1048063379 <== osd.3 v2:172.21.15.167:6808/30036 13 ==== osd_op_reply(57 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7effa40473a0 con 0x7effa0002300 2023-10-05T19:28:30.378 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:30.374+0000 7effb085f700 1 -- 172.21.15.167:0/1048063379 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:58 5.e 5:70000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x55e121878d00 con 0x7effa0002300 2023-10-05T19:28:30.378 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:30.374+0000 7effb085f700 1 -- 172.21.15.167:0/1048063379 <== osd.3 v2:172.21.15.167:6808/30036 14 ==== osd_op_reply(58 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7effa40473a0 con 0x7effa0002300 2023-10-05T19:28:30.379 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:30.374+0000 7effb085f700 1 -- 172.21.15.167:0/1048063379 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:59 5.1e 5:78000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x55e121878d00 con 0x7eff9c015920 2023-10-05T19:28:30.379 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:30.374+0000 7effb1060700 1 -- 172.21.15.167:0/1048063379 <== osd.0 v2:172.21.15.118:6800/27690 15 ==== osd_op_reply(59 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7effac03f040 con 0x7eff9c015920 2023-10-05T19:28:30.380 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:30.374+0000 7effb1060700 1 -- 172.21.15.167:0/1048063379 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:60 5.1 5:80000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x55e121878d00 con 0x7effa0002300 2023-10-05T19:28:30.380 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:30.374+0000 7effb085f700 1 -- 172.21.15.167:0/1048063379 <== osd.3 v2:172.21.15.167:6808/30036 15 ==== osd_op_reply(60 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7effa40473a0 con 0x7effa0002300 2023-10-05T19:28:30.380 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:30.374+0000 7effb085f700 1 -- 172.21.15.167:0/1048063379 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:61 5.11 5:88000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x55e121878d00 con 0x7effa0002300 2023-10-05T19:28:30.381 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:30.374+0000 7effb085f700 1 -- 172.21.15.167:0/1048063379 <== osd.3 v2:172.21.15.167:6808/30036 16 ==== osd_op_reply(61 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7effa40473a0 con 0x7effa0002300 2023-10-05T19:28:30.381 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:30.374+0000 7effb085f700 1 -- 172.21.15.167:0/1048063379 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:62 5.9 5:90000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x55e121878d00 con 0x55e1218462e0 2023-10-05T19:28:30.382 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:30.378+0000 7effb1060700 1 -- 172.21.15.167:0/1048063379 <== osd.1 v2:172.21.15.167:6800/30035 22 ==== osd_op_reply(62 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7effac03f040 con 0x55e1218462e0 2023-10-05T19:28:30.382 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:30.378+0000 7effb1060700 1 -- 172.21.15.167:0/1048063379 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:63 5.19 5:98000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x55e121878d00 con 0x55e1218462e0 2023-10-05T19:28:30.382 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:30.378+0000 7effb1060700 1 -- 172.21.15.167:0/1048063379 <== osd.1 v2:172.21.15.167:6800/30035 23 ==== osd_op_reply(63 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7effac03f040 con 0x55e1218462e0 2023-10-05T19:28:30.383 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:30.378+0000 7effb1060700 1 -- 172.21.15.167:0/1048063379 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:64 5.5 5:a0000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x55e121878d00 con 0x7eff9c015920 2023-10-05T19:28:30.383 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:30.378+0000 7effb1060700 1 -- 172.21.15.167:0/1048063379 <== osd.0 v2:172.21.15.118:6800/27690 16 ==== osd_op_reply(64 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7effac03f040 con 0x7eff9c015920 2023-10-05T19:28:30.383 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:30.378+0000 7effb1060700 1 -- 172.21.15.167:0/1048063379 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:65 5.15 5:a8000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x55e121878d00 con 0x7effa0002300 2023-10-05T19:28:30.384 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:30.378+0000 7effb085f700 1 -- 172.21.15.167:0/1048063379 <== osd.3 v2:172.21.15.167:6808/30036 17 ==== osd_op_reply(65 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7effa40473a0 con 0x7effa0002300 2023-10-05T19:28:30.384 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:30.378+0000 7effb085f700 1 -- 172.21.15.167:0/1048063379 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:66 5.d 5:b0000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x55e121878d00 con 0x7effa400d750 2023-10-05T19:28:30.385 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:30.378+0000 7effabfff700 1 -- 172.21.15.167:0/1048063379 <== osd.2 v2:172.21.15.167:6816/30037 10 ==== osd_op_reply(66 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7eff9c01d040 con 0x7effa400d750 2023-10-05T19:28:30.385 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:30.378+0000 7effabfff700 1 -- 172.21.15.167:0/1048063379 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:67 5.1d 5:b8000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x55e121878d00 con 0x55e1218462e0 2023-10-05T19:28:30.386 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:30.378+0000 7effb1060700 1 -- 172.21.15.167:0/1048063379 <== osd.1 v2:172.21.15.167:6800/30035 24 ==== osd_op_reply(67 [pgnls start_epoch 30 out=107b] v30'2 uv2 ondisk = 1) v8 ==== 144+0+107 (crc 0 0 0) 0x7effac03f040 con 0x55e1218462e0 2023-10-05T19:28:30.386 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:30.378+0000 7effb1060700 1 -- 172.21.15.167:0/1048063379 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:68 5.3 5:c0000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x55e121878d00 con 0x7eff9c015920 2023-10-05T19:28:30.386 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:30.382+0000 7effb1060700 1 -- 172.21.15.167:0/1048063379 <== osd.0 v2:172.21.15.118:6800/27690 17 ==== osd_op_reply(68 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7effac03f040 con 0x7eff9c015920 2023-10-05T19:28:30.387 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:30.382+0000 7effb1060700 1 -- 172.21.15.167:0/1048063379 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:69 5.13 5:c8000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x55e121878d00 con 0x7effa0002300 2023-10-05T19:28:30.387 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:30.382+0000 7effb085f700 1 -- 172.21.15.167:0/1048063379 <== osd.3 v2:172.21.15.167:6808/30036 18 ==== osd_op_reply(69 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7effa40473a0 con 0x7effa0002300 2023-10-05T19:28:30.387 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:30.382+0000 7effb085f700 1 -- 172.21.15.167:0/1048063379 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:70 5.b 5:d0000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x55e121878d00 con 0x7eff9c015920 2023-10-05T19:28:30.388 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:30.382+0000 7effb1060700 1 -- 172.21.15.167:0/1048063379 <== osd.0 v2:172.21.15.118:6800/27690 18 ==== osd_op_reply(70 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7effac03f040 con 0x7eff9c015920 2023-10-05T19:28:30.388 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:30.382+0000 7effb1060700 1 -- 172.21.15.167:0/1048063379 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:71 5.1b 5:d8000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x55e121878d00 con 0x7effa400d750 2023-10-05T19:28:30.389 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:30.382+0000 7effabfff700 1 -- 172.21.15.167:0/1048063379 <== osd.2 v2:172.21.15.167:6816/30037 11 ==== osd_op_reply(71 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7eff9c01d040 con 0x7effa400d750 2023-10-05T19:28:30.389 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:30.382+0000 7effabfff700 1 -- 172.21.15.167:0/1048063379 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:72 5.7 5:e0000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x55e121878d00 con 0x7effa0002300 2023-10-05T19:28:30.390 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:30.386+0000 7effb085f700 1 -- 172.21.15.167:0/1048063379 <== osd.3 v2:172.21.15.167:6808/30036 19 ==== osd_op_reply(72 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7effa40473a0 con 0x7effa0002300 2023-10-05T19:28:30.390 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:30.386+0000 7effb085f700 1 -- 172.21.15.167:0/1048063379 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:73 5.17 5:e8000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x55e121878d00 con 0x7eff9c015920 2023-10-05T19:28:30.391 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:30.386+0000 7effb1060700 1 -- 172.21.15.167:0/1048063379 <== osd.0 v2:172.21.15.118:6800/27690 19 ==== osd_op_reply(73 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7effac03f040 con 0x7eff9c015920 2023-10-05T19:28:30.391 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:30.386+0000 7effb1060700 1 -- 172.21.15.167:0/1048063379 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:74 5.f 5:f0000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x55e121878d00 con 0x55e1218462e0 2023-10-05T19:28:30.392 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:30.386+0000 7effb1060700 1 -- 172.21.15.167:0/1048063379 <== osd.1 v2:172.21.15.167:6800/30035 25 ==== osd_op_reply(74 [pgnls start_epoch 30 out=115b] v23'2 uv2 ondisk = 1) v8 ==== 144+0+115 (crc 0 0 0) 0x7effac03f040 con 0x55e1218462e0 2023-10-05T19:28:30.392 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:30.386+0000 7effb1060700 1 -- 172.21.15.167:0/1048063379 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:75 5.1f 5:f8000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x55e121878d00 con 0x7effa400d750 2023-10-05T19:28:30.393 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:30.386+0000 7effabfff700 1 -- 172.21.15.167:0/1048063379 <== osd.2 v2:172.21.15.167:6816/30037 12 ==== osd_op_reply(75 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7eff9c01d040 con 0x7effa400d750 2023-10-05T19:28:30.393 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:30.386+0000 7effb2d2ae40 20 RGWRados::pool_iterate: got zonegroup_info.4f1f719d-9648-417b-b10c-e5d939825419 2023-10-05T19:28:30.393 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:30.386+0000 7effb2d2ae40 20 RGWRados::pool_iterate: got zone_info.9a9df965-6612-45d4-a45f-6e3c889bacc2 2023-10-05T19:28:30.393 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:30.386+0000 7effb2d2ae40 20 RGWRados::pool_iterate: got zone_names.default 2023-10-05T19:28:30.393 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:30.386+0000 7effb2d2ae40 20 RGWRados::pool_iterate: got zonegroups_names.default 2023-10-05T19:28:30.394 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:30.386+0000 7effb2d2ae40 20 rados->read ofs=0 len=0 2023-10-05T19:28:30.394 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:30.386+0000 7effb2d2ae40 1 -- 172.21.15.167:0/1048063379 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:76 5.f 5:f43b6ece:::zone_names.default:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x55e12187c960 con 0x55e1218462e0 2023-10-05T19:28:30.395 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:30.390+0000 7effb1060700 1 -- 172.21.15.167:0/1048063379 <== osd.1 v2:172.21.15.167:6800/30035 26 ==== osd_op_reply(76 zone_names.default [read 0~46 out=46b] v0'0 uv1 ondisk = 0) v8 ==== 162+0+46 (crc 0 0 0) 0x7effac03f040 con 0x55e1218462e0 2023-10-05T19:28:30.395 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:30.390+0000 7effb2d2ae40 20 rados_obj.operate() r=0 bl.length=46 2023-10-05T19:28:30.395 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:30.390+0000 7effb2d2ae40 20 rados->read ofs=0 len=0 2023-10-05T19:28:30.395 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:30.390+0000 7effb2d2ae40 1 -- 172.21.15.167:0/1048063379 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:77 5.1d 5:b8374eb4:::zone_info.9a9df965-6612-45d4-a45f-6e3c889bacc2:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x55e12187cd10 con 0x55e1218462e0 2023-10-05T19:28:30.395 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:30.390+0000 7effb1060700 1 -- 172.21.15.167:0/1048063379 <== osd.1 v2:172.21.15.167:6800/30035 27 ==== osd_op_reply(77 zone_info.9a9df965-6612-45d4-a45f-6e3c889bacc2 [read 0~874 out=874b] v0'0 uv2 ondisk = 0) v8 ==== 190+0+874 (crc 0 0 0) 0x7effac03f040 con 0x55e1218462e0 2023-10-05T19:28:30.395 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:30.390+0000 7effb2d2ae40 20 rados_obj.operate() r=0 bl.length=874 2023-10-05T19:28:30.395 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:30.390+0000 7effb2d2ae40 20 rados->read ofs=0 len=0 2023-10-05T19:28:30.395 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:30.390+0000 7effb2d2ae40 1 -- 172.21.15.167:0/1048063379 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:78 5.f 5:f4c53578:::zonegroups_names.default:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x55e12187d200 con 0x55e1218462e0 2023-10-05T19:28:30.396 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:30.390+0000 7effb1060700 1 -- 172.21.15.167:0/1048063379 <== osd.1 v2:172.21.15.167:6800/30035 28 ==== osd_op_reply(78 zonegroups_names.default [read 0~46 out=46b] v0'0 uv2 ondisk = 0) v8 ==== 168+0+46 (crc 0 0 0) 0x7effac03f040 con 0x55e1218462e0 2023-10-05T19:28:30.396 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:30.390+0000 7effb2d2ae40 20 rados_obj.operate() r=0 bl.length=46 2023-10-05T19:28:30.396 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:30.390+0000 7effb2d2ae40 20 rados->read ofs=0 len=0 2023-10-05T19:28:30.396 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:30.390+0000 7effb2d2ae40 1 -- 172.21.15.167:0/1048063379 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:79 5.c 5:30d299a5:::zonegroup_info.4f1f719d-9648-417b-b10c-e5d939825419:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x55e12187d200 con 0x7eff9c015920 2023-10-05T19:28:30.397 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:30.390+0000 7effb1060700 1 -- 172.21.15.167:0/1048063379 <== osd.0 v2:172.21.15.118:6800/27690 20 ==== osd_op_reply(79 zonegroup_info.4f1f719d-9648-417b-b10c-e5d939825419 [read 0~362 out=362b] v0'0 uv1 ondisk = 0) v8 ==== 195+0+362 (crc 0 0 0) 0x7effac052040 con 0x7eff9c015920 2023-10-05T19:28:30.397 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:30.390+0000 7effb2d2ae40 20 rados_obj.operate() r=0 bl.length=362 2023-10-05T19:28:30.397 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:30.390+0000 7effb2d2ae40 20 rados->read ofs=0 len=0 2023-10-05T19:28:30.397 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:30.390+0000 7effb2d2ae40 1 -- 172.21.15.167:0/1048063379 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:80 5.3 5:c0ad8cbd:::region_map:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x55e12187d300 con 0x7eff9c015920 2023-10-05T19:28:30.398 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:30.390+0000 7effb1060700 1 -- 172.21.15.167:0/1048063379 <== osd.0 v2:172.21.15.118:6800/27690 21 ==== osd_op_reply(80 region_map [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 ==== 154+0+0 (crc 0 0 0) 0x7effac052040 con 0x7eff9c015920 2023-10-05T19:28:30.398 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:30.390+0000 7effb2d2ae40 20 rados_obj.operate() r=-2 bl.length=0 2023-10-05T19:28:30.398 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:30.390+0000 7effb2d2ae40 20 zone default found 2023-10-05T19:28:30.398 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:30.390+0000 7effb2d2ae40 4 Realm: () 2023-10-05T19:28:30.398 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:30.390+0000 7effb2d2ae40 4 ZoneGroup: default (4f1f719d-9648-417b-b10c-e5d939825419) 2023-10-05T19:28:30.398 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:30.390+0000 7effb2d2ae40 4 Zone: default (9a9df965-6612-45d4-a45f-6e3c889bacc2) 2023-10-05T19:28:30.398 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:30.390+0000 7effb2d2ae40 10 cannot find current period zonegroup using local zonegroup configuration 2023-10-05T19:28:30.398 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:30.390+0000 7effb2d2ae40 20 zonegroup default 2023-10-05T19:28:30.399 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:30.390+0000 7effb2d2ae40 20 rados->read ofs=0 len=0 2023-10-05T19:28:30.399 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:30.390+0000 7effb2d2ae40 1 -- 172.21.15.167:0/1048063379 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:81 5.3 5:c52100b6:::period_config.default:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x55e12187d300 con 0x7eff9c015920 2023-10-05T19:28:30.402 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:30.394+0000 7effb1060700 1 -- 172.21.15.167:0/1048063379 <== osd.0 v2:172.21.15.118:6800/27690 22 ==== osd_op_reply(81 period_config.default [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 ==== 165+0+0 (crc 0 0 0) 0x7effac052040 con 0x7eff9c015920 2023-10-05T19:28:30.402 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:30.394+0000 7effb2d2ae40 20 rados_obj.operate() r=-2 bl.length=0 2023-10-05T19:28:30.402 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:30.394+0000 7effb2d2ae40 20 rados->read ofs=0 len=0 2023-10-05T19:28:30.402 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:30.394+0000 7effb2d2ae40 1 -- 172.21.15.167:0/1048063379 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:82 6.2 6:4347d321:::bucket.sync-source-hints.:head [call version.read in=11b,read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x55e12187ec80 con 0x7effa0002300 2023-10-05T19:28:30.402 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:30.394+0000 7effb085f700 1 -- 172.21.15.167:0/1048063379 <== osd.3 v2:172.21.15.167:6808/30036 20 ==== osd_op_reply(82 bucket.sync-source-hints. [call,read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 ==== 211+0+0 (crc 0 0 0) 0x7effa40473a0 con 0x7effa0002300 2023-10-05T19:28:30.402 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:30.394+0000 7effb2d2ae40 20 rados_obj.operate() r=-2 bl.length=0 2023-10-05T19:28:30.402 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:30.394+0000 7effb2d2ae40 20 rados->read ofs=0 len=0 2023-10-05T19:28:30.402 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:30.394+0000 7effb2d2ae40 1 -- 172.21.15.167:0/1048063379 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:83 6.b 6:d467b91b:::bucket.sync-target-hints.:head [call version.read in=11b,read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x55e12187f030 con 0x7effa0002300 2023-10-05T19:28:30.402 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:30.394+0000 7effb085f700 1 -- 172.21.15.167:0/1048063379 <== osd.3 v2:172.21.15.167:6808/30036 21 ==== osd_op_reply(83 bucket.sync-target-hints. [call,read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 ==== 211+0+0 (crc 0 0 0) 0x7effa40473a0 con 0x7effa0002300 2023-10-05T19:28:30.402 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:30.394+0000 7effb2d2ae40 20 rados_obj.operate() r=-2 bl.length=0 2023-10-05T19:28:30.403 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:30.394+0000 7effb2d2ae40 20 started sync module instance, tier type = 2023-10-05T19:28:30.403 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:30.394+0000 7effb2d2ae40 20 started zone id=9a9df965-6612-45d4-a45f-6e3c889bacc2 (name=default) with tier type = 2023-10-05T19:28:30.403 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:30.394+0000 7effb2d2ae40 1 -- 172.21.15.167:0/1048063379 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:84 7.1f 7:f95f44c2:::notify.0:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x55e12187e340 con 0x7effa400d750 2023-10-05T19:28:30.403 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:30.398+0000 7effabfff700 1 -- 172.21.15.167:0/1048063379 <== osd.2 v2:172.21.15.167:6816/30037 13 ==== osd_op_reply(84 notify.0 [create] v30'87 uv87 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7eff9c01d040 con 0x7effa400d750 2023-10-05T19:28:30.403 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:30.398+0000 7effb2d2ae40 1 -- 172.21.15.167:0/1048063379 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:85 7.1f 7:f95f44c2:::notify.0:head [watch watch cookie 94425418557248] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x55e12187df70 con 0x7effa400d750 2023-10-05T19:28:30.404 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:30.398+0000 7effb2d2ae40 1 -- 172.21.15.167:0/1048063379 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:86 7.0 7:05bf5b68:::notify.1:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x55e12187fe00 con 0x55e1218462e0 2023-10-05T19:28:30.406 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:30.402+0000 7effb1060700 1 -- 172.21.15.167:0/1048063379 <== osd.1 v2:172.21.15.167:6800/30035 29 ==== osd_op_reply(86 notify.1 [create] v30'87 uv87 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7effac052040 con 0x55e1218462e0 2023-10-05T19:28:30.406 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:30.402+0000 7effabfff700 1 -- 172.21.15.167:0/1048063379 <== osd.2 v2:172.21.15.167:6816/30037 14 ==== osd_op_reply(85 notify.0 [watch watch cookie 94425418557248] v30'88 uv87 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7eff9c01d040 con 0x7effa400d750 2023-10-05T19:28:30.406 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:30.402+0000 7effb2d2ae40 1 -- 172.21.15.167:0/1048063379 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:87 7.0 7:05bf5b68:::notify.1:head [watch watch cookie 94425418561744] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x55e1218806f0 con 0x55e1218462e0 2023-10-05T19:28:30.407 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:30.402+0000 7effb2d2ae40 1 -- 172.21.15.167:0/1048063379 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:88 7.15 7:a93a5511:::notify.2:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x55e12187f3e0 con 0x7effa0002300 2023-10-05T19:28:30.409 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:30.402+0000 7effb1060700 1 -- 172.21.15.167:0/1048063379 <== osd.1 v2:172.21.15.167:6800/30035 30 ==== osd_op_reply(87 notify.1 [watch watch cookie 94425418561744] v30'88 uv87 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7effac052040 con 0x55e1218462e0 2023-10-05T19:28:30.410 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:30.402+0000 7effb085f700 1 -- 172.21.15.167:0/1048063379 <== osd.3 v2:172.21.15.167:6808/30036 22 ==== osd_op_reply(88 notify.2 [create] v30'87 uv87 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7effa40473a0 con 0x7effa0002300 2023-10-05T19:28:30.410 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:30.402+0000 7effb2d2ae40 1 -- 172.21.15.167:0/1048063379 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:89 7.15 7:a93a5511:::notify.2:head [watch watch cookie 94425418564384] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x55e121880c60 con 0x7effa0002300 2023-10-05T19:28:30.410 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:30.402+0000 7effb2d2ae40 1 -- 172.21.15.167:0/1048063379 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:90 7.e 7:7759931f:::notify.3:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x55e121881670 con 0x7effa400d750 2023-10-05T19:28:30.412 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:30.406+0000 7effabfff700 1 -- 172.21.15.167:0/1048063379 <== osd.2 v2:172.21.15.167:6816/30037 15 ==== osd_op_reply(90 notify.3 [create] v30'87 uv87 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7eff9c01d040 con 0x7effa400d750 2023-10-05T19:28:30.413 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:30.406+0000 7effb2d2ae40 1 -- 172.21.15.167:0/1048063379 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:91 7.e 7:7759931f:::notify.3:head [watch watch cookie 94425418570272] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x55e121882360 con 0x7effa400d750 2023-10-05T19:28:30.413 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:30.406+0000 7effb2d2ae40 1 -- 172.21.15.167:0/1048063379 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:92 7.d 7:b4812045:::notify.4:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x55e121882d70 con 0x55e1218462e0 2023-10-05T19:28:30.413 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:30.406+0000 7effb085f700 1 -- 172.21.15.167:0/1048063379 <== osd.3 v2:172.21.15.167:6808/30036 23 ==== osd_op_reply(89 notify.2 [watch watch cookie 94425418564384] v30'88 uv87 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7effa40473a0 con 0x7effa0002300 2023-10-05T19:28:30.415 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:30.410+0000 7effb1060700 1 -- 172.21.15.167:0/1048063379 <== osd.1 v2:172.21.15.167:6800/30035 31 ==== osd_op_reply(92 notify.4 [create] v30'87 uv87 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7effac052040 con 0x55e1218462e0 2023-10-05T19:28:30.415 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:30.410+0000 7effb2d2ae40 1 -- 172.21.15.167:0/1048063379 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:93 7.d 7:b4812045:::notify.4:head [watch watch cookie 94425418573952] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x55e1218831c0 con 0x55e1218462e0 2023-10-05T19:28:30.415 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:30.410+0000 7effb2d2ae40 1 -- 172.21.15.167:0/1048063379 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:94 7.3 7:c609908c:::notify.5:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x55e121883a60 con 0x7eff9c015920 2023-10-05T19:28:30.416 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:30.410+0000 7effabfff700 1 -- 172.21.15.167:0/1048063379 <== osd.2 v2:172.21.15.167:6816/30037 16 ==== osd_op_reply(91 notify.3 [watch watch cookie 94425418570272] v30'88 uv87 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7eff9c01d040 con 0x7effa400d750 2023-10-05T19:28:30.418 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:30.410+0000 7effb1060700 1 -- 172.21.15.167:0/1048063379 <== osd.0 v2:172.21.15.118:6800/27690 23 ==== osd_op_reply(94 notify.5 [create] v30'87 uv87 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7effac052040 con 0x7eff9c015920 2023-10-05T19:28:30.418 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:30.410+0000 7effb2d2ae40 1 -- 172.21.15.167:0/1048063379 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:95 7.3 7:c609908c:::notify.5:head [watch watch cookie 94425418576320] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x55e121883b00 con 0x7eff9c015920 2023-10-05T19:28:30.418 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:30.410+0000 7effb2d2ae40 1 -- 172.21.15.167:0/1048063379 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:96 7.14 7:2b04a3e9:::notify.6:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x55e1218843e0 con 0x7eff9c015920 2023-10-05T19:28:30.418 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:30.414+0000 7effb1060700 1 -- 172.21.15.167:0/1048063379 <== osd.1 v2:172.21.15.167:6800/30035 32 ==== osd_op_reply(93 notify.4 [watch watch cookie 94425418573952] v30'88 uv87 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7effac052040 con 0x55e1218462e0 2023-10-05T19:28:30.420 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:30.414+0000 7effb1060700 1 -- 172.21.15.167:0/1048063379 <== osd.0 v2:172.21.15.118:6800/27690 24 ==== osd_op_reply(95 notify.5 [watch watch cookie 94425418576320] v30'88 uv87 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7effac052040 con 0x7eff9c015920 2023-10-05T19:28:30.421 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:30.414+0000 7effb1060700 1 -- 172.21.15.167:0/1048063379 <== osd.0 v2:172.21.15.118:6800/27690 25 ==== osd_op_reply(96 notify.6 [create] v30'87 uv87 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7effac052040 con 0x7eff9c015920 2023-10-05T19:28:30.421 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:30.414+0000 7effb2d2ae40 1 -- 172.21.15.167:0/1048063379 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:97 7.14 7:2b04a3e9:::notify.6:head [watch watch cookie 94425418575056] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x55e121883b00 con 0x7eff9c015920 2023-10-05T19:28:30.421 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:30.414+0000 7effb2d2ae40 1 -- 172.21.15.167:0/1048063379 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:98 7.9 7:93e5b521:::notify.7:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x55e121884cd0 con 0x7eff9c015920 2023-10-05T19:28:30.423 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:30.418+0000 7effb1060700 1 -- 172.21.15.167:0/1048063379 <== osd.0 v2:172.21.15.118:6800/27690 26 ==== osd_op_reply(97 notify.6 [watch watch cookie 94425418575056] v30'88 uv87 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7effac052040 con 0x7eff9c015920 2023-10-05T19:28:30.424 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:30.418+0000 7effb1060700 1 -- 172.21.15.167:0/1048063379 <== osd.0 v2:172.21.15.118:6800/27690 27 ==== osd_op_reply(98 notify.7 [create] v30'87 uv87 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7effac052040 con 0x7eff9c015920 2023-10-05T19:28:30.424 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:30.418+0000 7effb2d2ae40 1 -- 172.21.15.167:0/1048063379 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:99 7.9 7:93e5b521:::notify.7:head [watch watch cookie 94425418577424] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x55e121883a60 con 0x7eff9c015920 2023-10-05T19:28:30.424 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:30.418+0000 7effb2d2ae40 20 add_watcher() i=0 2023-10-05T19:28:30.424 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:30.418+0000 7effb2d2ae40 20 add_watcher() i=1 2023-10-05T19:28:30.424 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:30.418+0000 7effb2d2ae40 20 add_watcher() i=2 2023-10-05T19:28:30.424 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:30.418+0000 7effb2d2ae40 20 add_watcher() i=3 2023-10-05T19:28:30.424 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:30.418+0000 7effb2d2ae40 20 add_watcher() i=4 2023-10-05T19:28:30.424 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:30.418+0000 7effb2d2ae40 20 add_watcher() i=5 2023-10-05T19:28:30.424 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:30.418+0000 7effb2d2ae40 20 add_watcher() i=6 2023-10-05T19:28:30.426 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:30.418+0000 7effb1060700 1 -- 172.21.15.167:0/1048063379 <== osd.0 v2:172.21.15.118:6800/27690 28 ==== osd_op_reply(99 notify.7 [watch watch cookie 94425418577424] v30'88 uv87 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7effac052040 con 0x7eff9c015920 2023-10-05T19:28:30.426 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:30.418+0000 7effb2d2ae40 20 add_watcher() i=7 2023-10-05T19:28:30.426 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:30.418+0000 7effb2d2ae40 2 all 8 watchers are set, enabling cache 2023-10-05T19:28:30.426 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:30.418+0000 7effb2d2ae40 1 -- 172.21.15.167:0/1048063379 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:100 6.4 6:22d26bf9:::data_loggenerations_metadata:head [call version.check_conds in=74b,call version.read in=11b,read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x55e121887c90 con 0x55e1218462e0 2023-10-05T19:28:30.427 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:30.422+0000 7effb1060700 1 -- 172.21.15.167:0/1048063379 <== osd.1 v2:172.21.15.167:6800/30035 33 ==== osd_op_reply(100 data_loggenerations_metadata [call,call out=48b,read 0~28 out=28b] v0'0 uv1 ondisk = 0) v8 ==== 256+0+76 (crc 0 0 0) 0x7effac052040 con 0x55e1218462e0 2023-10-05T19:28:30.427 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:30.422+0000 7effb2d2ae40 1 -- 172.21.15.167:0/1048063379 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:101 6.4 6:22d26bf9:::data_loggenerations_metadata:head [watch watch cookie 94425418581872] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x55e1218850b0 con 0x55e1218462e0 2023-10-05T19:28:30.429 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:30.422+0000 7effb1060700 1 -- 172.21.15.167:0/1048063379 <== osd.1 v2:172.21.15.167:6800/30035 34 ==== osd_op_reply(101 data_loggenerations_metadata [watch watch cookie 94425418581872] v30'92 uv1 ondisk = 0) v8 ==== 172+0+0 (crc 0 0 0) 0x7effac052040 con 0x55e1218462e0 2023-10-05T19:28:30.429 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:30.422+0000 7eff81fe3700 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2023-10-05T19:28:30.429 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:30.422+0000 7effb2d2ae40 20 check_secure_mon_conn(): auth registy supported: methods=[2] modes=[2,1] 2023-10-05T19:28:30.429 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:30.422+0000 7effb2d2ae40 20 check_secure_mon_conn(): mode 1 is insecure 2023-10-05T19:28:30.429 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:30.422+0000 7effb2d2ae40 1 Processor -- start 2023-10-05T19:28:30.430 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:30.422+0000 7effb2d2ae40 1 -- start start 2023-10-05T19:28:30.430 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:30.422+0000 7effb2d2ae40 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55e121891f80 0x55e121892350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-05T19:28:30.430 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:30.422+0000 7effb2d2ae40 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e121892890 con 0x55e121891f80 2023-10-05T19:28:30.431 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:30.426+0000 7effabfff700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55e121891f80 0x55e121892350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-05T19:28:30.431 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:30.426+0000 7effabfff700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55e121891f80 0x55e121892350 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.118:3300/0 says I am v2:172.21.15.167:58108/0 (socket says 172.21.15.167:58108) 2023-10-05T19:28:30.431 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:30.426+0000 7effabfff700 1 -- 172.21.15.167:0/3466161022 learned_addr learned my addr 172.21.15.167:0/3466161022 (peer_addr_for_me v2:172.21.15.167:0/0) 2023-10-05T19:28:30.431 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:30.426+0000 7effabfff700 1 -- 172.21.15.167:0/3466161022 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55e1218929d0 con 0x55e121891f80 2023-10-05T19:28:30.432 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:30.426+0000 7effabfff700 1 --2- 172.21.15.167:0/3466161022 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55e121891f80 0x55e121892350 secure :-1 s=READY pgs=312 cs=0 l=1 rev1=1 crypto rx=0x7eff9c0086e0 tx=0x7eff9c017040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=27421fe1af6e4740 server_cookie=0 in_seq=0 out_seq=0 2023-10-05T19:28:30.432 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:30.426+0000 7eff817e2700 1 -- 172.21.15.167:0/3466161022 <== mon.0 v2:172.21.15.118:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7eff9c00f040 con 0x55e121891f80 2023-10-05T19:28:30.432 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:30.426+0000 7eff817e2700 1 -- 172.21.15.167:0/3466161022 <== mon.0 v2:172.21.15.118:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7eff9c02c710 con 0x55e121891f80 2023-10-05T19:28:30.432 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:30.426+0000 7eff817e2700 1 -- 172.21.15.167:0/3466161022 <== mon.0 v2:172.21.15.118:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7eff9c008a30 con 0x55e121891f80 2023-10-05T19:28:30.433 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:30.426+0000 7effb2d2ae40 1 -- 172.21.15.167:0/3466161022 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55e121891f80 msgr2=0x55e121892350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-05T19:28:30.433 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:30.426+0000 7effb2d2ae40 1 --2- 172.21.15.167:0/3466161022 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55e121891f80 0x55e121892350 secure :-1 s=READY pgs=312 cs=0 l=1 rev1=1 crypto rx=0x7eff9c0086e0 tx=0x7eff9c017040 comp rx=0 tx=0).stop 2023-10-05T19:28:30.433 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:30.426+0000 7effb2d2ae40 1 -- 172.21.15.167:0/3466161022 shutdown_connections 2023-10-05T19:28:30.433 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:30.426+0000 7effb2d2ae40 1 --2- 172.21.15.167:0/3466161022 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55e121891f80 0x55e121892350 unknown :-1 s=CLOSED pgs=312 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-05T19:28:30.433 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:30.426+0000 7effb2d2ae40 1 -- 172.21.15.167:0/3466161022 >> 172.21.15.167:0/3466161022 conn(0x55e12188d340 msgr2=0x55e12188b4f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-05T19:28:30.433 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:30.426+0000 7effb2d2ae40 1 -- 172.21.15.167:0/3466161022 shutdown_connections 2023-10-05T19:28:30.433 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:30.426+0000 7effb2d2ae40 1 -- 172.21.15.167:0/3466161022 wait complete. 2023-10-05T19:28:30.434 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:30.426+0000 7effb2d2ae40 1 Processor -- start 2023-10-05T19:28:30.434 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:30.426+0000 7effb2d2ae40 1 -- start start 2023-10-05T19:28:30.435 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:30.426+0000 7effb2d2ae40 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55e121891f80 0x55e121897ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-05T19:28:30.435 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:30.426+0000 7effb2d2ae40 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e121898ae0 con 0x55e121891f80 2023-10-05T19:28:30.435 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:30.430+0000 7effabfff700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55e121891f80 0x55e121897ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-05T19:28:30.435 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:30.430+0000 7effabfff700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55e121891f80 0x55e121897ae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.118:3300/0 says I am v2:172.21.15.167:58120/0 (socket says 172.21.15.167:58120) 2023-10-05T19:28:30.435 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:30.430+0000 7effabfff700 1 -- 172.21.15.167:0/1066262387 learned_addr learned my addr 172.21.15.167:0/1066262387 (peer_addr_for_me v2:172.21.15.167:0/0) 2023-10-05T19:28:30.435 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:30.430+0000 7effabfff700 1 -- 172.21.15.167:0/1066262387 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7eff9c008b90 con 0x55e121891f80 2023-10-05T19:28:30.436 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:30.430+0000 7effabfff700 1 --2- 172.21.15.167:0/1066262387 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55e121891f80 0x55e121897ae0 secure :-1 s=READY pgs=313 cs=0 l=1 rev1=1 crypto rx=0x7eff9c005df0 tx=0x7eff9c02e030 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-05T19:28:30.436 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:30.430+0000 7efefb7fe700 1 -- 172.21.15.167:0/1066262387 <== mon.0 v2:172.21.15.118:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7eff9c00f050 con 0x55e121891f80 2023-10-05T19:28:30.436 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:30.430+0000 7efefb7fe700 1 -- 172.21.15.167:0/1066262387 <== mon.0 v2:172.21.15.118:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7eff9c040430 con 0x55e121891f80 2023-10-05T19:28:30.436 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:30.430+0000 7effb2d2ae40 1 -- 172.21.15.167:0/1066262387 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x55e1218980e0 con 0x55e121891f80 2023-10-05T19:28:30.436 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:30.430+0000 7efefb7fe700 1 -- 172.21.15.167:0/1066262387 <== mon.0 v2:172.21.15.118:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7eff9c008a30 con 0x55e121891f80 2023-10-05T19:28:30.436 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:30.430+0000 7effb2d2ae40 1 -- 172.21.15.167:0/1066262387 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55e1218984c0 con 0x55e121891f80 2023-10-05T19:28:30.437 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:30.430+0000 7efefb7fe700 1 -- 172.21.15.167:0/1066262387 <== mon.0 v2:172.21.15.118:3300/0 4 ==== mgrmap(e 5) v1 ==== 40329+0+0 (secure 0 0 0) 0x7eff9c049450 con 0x55e121891f80 2023-10-05T19:28:30.437 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:30.430+0000 7efefb7fe700 1 --2- 172.21.15.167:0/1066262387 >> [v2:172.21.15.167:6824/29889,v1:172.21.15.167:6825/29889] conn(0x7efef0030b70 0x7efef0033020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-05T19:28:30.437 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:30.430+0000 7efefb7fe700 1 -- 172.21.15.167:0/1066262387 <== mon.0 v2:172.21.15.118:3300/0 5 ==== osd_map(30..30 src has 1..30) v4 ==== 6057+0+0 (secure 0 0 0) 0x7eff9c027040 con 0x55e121891f80 2023-10-05T19:28:30.437 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:30.430+0000 7effb2d2ae40 5 note: GC not initialized 2023-10-05T19:28:30.438 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:30.430+0000 7effb1060700 1 --2- 172.21.15.167:0/1066262387 >> [v2:172.21.15.167:6824/29889,v1:172.21.15.167:6825/29889] conn(0x7efef0030b70 0x7efef0033020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-05T19:28:30.438 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:30.434+0000 7effb1060700 1 --2- 172.21.15.167:0/1066262387 >> [v2:172.21.15.167:6824/29889,v1:172.21.15.167:6825/29889] conn(0x7efef0030b70 0x7efef0033020 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x55e121883cf0 tx=0x7effac00ae60 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-05T19:28:30.440 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:30.434+0000 7efef8ff9700 20 reqs_thread_entry: start 2023-10-05T19:28:30.441 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:30.434+0000 7efeee7fc700 20 reqs_thread_entry: start 2023-10-05T19:28:30.441 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:30.434+0000 7effb2d2ae40 20 init_complete bucket index max shards: 11 2023-10-05T19:28:30.441 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:30.434+0000 7effb2d2ae40 10 rgw notify: Started notification manager with: 1 workers 2023-10-05T19:28:30.442 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:30.434+0000 7efeecff9700 20 rgw notify: INFO: next queues processing will happen at: Thu Oct 5 19:29:00 2023 2023-10-05T19:28:30.442 INFO:teuthology.orchestra.run.smithi167.stderr: 2023-10-05T19:28:30.442 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:30.434+0000 7efed3fff700 20 reqs_thread_entry: start 2023-10-05T19:28:30.442 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:30.434+0000 7effb2d2ae40 20 get_system_obj_state: rctx=0x7ffcd41faef8 obj=default.rgw.meta:users.uid:foo state=0x55e12171c0b0 s->prefetch_data=0 2023-10-05T19:28:30.442 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:30.434+0000 7effb2d2ae40 1 -- 172.21.15.167:0/1048063379 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:102 8.18 8:1e6ca5ed:users.uid::foo:head [call version.read in=11b,getxattrs,stat] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x55e1219001d0 con 0x55e1218462e0 2023-10-05T19:28:30.443 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:30.438+0000 7effb1060700 1 -- 172.21.15.167:0/1048063379 <== osd.1 v2:172.21.15.167:6800/30035 35 ==== osd_op_reply(102 foo [call out=48b,getxattrs out=75b,stat out=16b] v0'0 uv11 ondisk = 0) v8 ==== 231+0+139 (crc 0 0 0) 0x7effac052040 con 0x55e1218462e0 2023-10-05T19:28:30.443 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:30.438+0000 7effb2d2ae40 20 get_system_obj_state: s->obj_tag was set empty 2023-10-05T19:28:30.443 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:30.438+0000 7effb2d2ae40 20 Read xattr: user.rgw.idtag 2023-10-05T19:28:30.443 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:30.438+0000 7effb2d2ae40 20 rados->read ofs=0 len=0 2023-10-05T19:28:30.443 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:30.438+0000 7effb2d2ae40 1 -- 172.21.15.167:0/1048063379 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:103 8.18 8:1e6ca5ed:users.uid::foo:head [call version.check_conds in=122b,call version.read in=11b,read 0~0,getxattrs] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x55e1219001d0 con 0x55e1218462e0 2023-10-05T19:28:30.444 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:30.438+0000 7effb1060700 1 -- 172.21.15.167:0/1048063379 <== osd.1 v2:172.21.15.167:6800/30035 36 ==== osd_op_reply(103 foo [call,call out=48b,read 0~357 out=357b,getxattrs out=75b] v0'0 uv11 ondisk = 0) v8 ==== 273+0+480 (crc 0 0 0) 0x7effac052040 con 0x55e1218462e0 2023-10-05T19:28:30.444 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:30.438+0000 7effb2d2ae40 20 rados_obj.operate() r=0 bl.length=357 2023-10-05T19:28:30.444 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:30.438+0000 7effb2d2ae40 1 -- 172.21.15.167:0/1048063379 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:104 8.9 8:96d98480:users.uid::foo.buckets:head [call user.list_buckets in=34b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x55e1219001d0 con 0x7effa400d750 2023-10-05T19:28:30.444 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:30.438+0000 7effabfff700 1 -- 172.21.15.167:0/1048063379 <== osd.2 v2:172.21.15.167:6816/30037 17 ==== osd_op_reply(104 foo.buckets [call out=183b] v0'0 uv2 ondisk = 0) v8 ==== 155+0+183 (crc 0 0 0) 0x7eff9c046070 con 0x7effa400d750 2023-10-05T19:28:30.445 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:30.438+0000 7effb2d2ae40 1 -- 172.21.15.167:0/1048063379 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:105 7.1f 7:f95f44c2:::notify.0:head [watch unwatch cookie 94425418557248] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x55e1219001d0 con 0x7effa400d750 2023-10-05T19:28:30.447 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:30.442+0000 7effabfff700 1 -- 172.21.15.167:0/1048063379 <== osd.2 v2:172.21.15.167:6816/30037 18 ==== osd_op_reply(105 notify.0 [watch unwatch cookie 94425418557248] v30'89 uv87 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7eff9c046070 con 0x7effa400d750 2023-10-05T19:28:30.447 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:30.442+0000 7effb2d2ae40 20 remove_watcher() i=0 2023-10-05T19:28:30.448 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:30.442+0000 7effb2d2ae40 2 removed watcher, disabling cache 2023-10-05T19:28:30.448 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:30.442+0000 7effb2d2ae40 1 -- 172.21.15.167:0/1048063379 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:106 7.0 7:05bf5b68:::notify.1:head [watch unwatch cookie 94425418561744] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x55e1219001d0 con 0x55e1218462e0 2023-10-05T19:28:30.450 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:30.442+0000 7effb1060700 1 -- 172.21.15.167:0/1048063379 <== osd.1 v2:172.21.15.167:6800/30035 37 ==== osd_op_reply(106 notify.1 [watch unwatch cookie 94425418561744] v30'89 uv87 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7effac052040 con 0x55e1218462e0 2023-10-05T19:28:30.450 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:30.446+0000 7effb2d2ae40 20 remove_watcher() i=1 2023-10-05T19:28:30.450 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:30.446+0000 7effb2d2ae40 1 -- 172.21.15.167:0/1048063379 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:107 7.15 7:a93a5511:::notify.2:head [watch unwatch cookie 94425418564384] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x55e1219001d0 con 0x7effa0002300 2023-10-05T19:28:30.452 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:30.446+0000 7effb085f700 1 -- 172.21.15.167:0/1048063379 <== osd.3 v2:172.21.15.167:6808/30036 24 ==== osd_op_reply(107 notify.2 [watch unwatch cookie 94425418564384] v30'89 uv87 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7effa40473a0 con 0x7effa0002300 2023-10-05T19:28:30.452 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:30.446+0000 7effb2d2ae40 20 remove_watcher() i=2 2023-10-05T19:28:30.453 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:30.446+0000 7effb2d2ae40 1 -- 172.21.15.167:0/1048063379 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:108 7.e 7:7759931f:::notify.3:head [watch unwatch cookie 94425418570272] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x55e12187f8d0 con 0x7effa400d750 2023-10-05T19:28:30.455 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:30.450+0000 7effabfff700 1 -- 172.21.15.167:0/1048063379 <== osd.2 v2:172.21.15.167:6816/30037 19 ==== osd_op_reply(108 notify.3 [watch unwatch cookie 94425418570272] v30'89 uv87 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7eff9c046070 con 0x7effa400d750 2023-10-05T19:28:30.455 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:30.450+0000 7effb2d2ae40 20 remove_watcher() i=3 2023-10-05T19:28:30.455 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:30.450+0000 7effb2d2ae40 1 -- 172.21.15.167:0/1048063379 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:109 7.d 7:b4812045:::notify.4:head [watch unwatch cookie 94425418573952] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x55e121880320 con 0x55e1218462e0 2023-10-05T19:28:30.458 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:30.450+0000 7effb1060700 1 -- 172.21.15.167:0/1048063379 <== osd.1 v2:172.21.15.167:6800/30035 38 ==== osd_op_reply(109 notify.4 [watch unwatch cookie 94425418573952] v30'89 uv87 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7effac052040 con 0x55e1218462e0 2023-10-05T19:28:30.458 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:30.450+0000 7effb2d2ae40 20 remove_watcher() i=4 2023-10-05T19:28:30.458 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:30.454+0000 7effb2d2ae40 1 -- 172.21.15.167:0/1048063379 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:110 7.3 7:c609908c:::notify.5:head [watch unwatch cookie 94425418576320] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x55e121881a20 con 0x7eff9c015920 2023-10-05T19:28:30.460 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:30.454+0000 7effb1060700 1 -- 172.21.15.167:0/1048063379 <== osd.0 v2:172.21.15.118:6800/27690 29 ==== osd_op_reply(110 notify.5 [watch unwatch cookie 94425418576320] v30'89 uv87 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7effac052040 con 0x7eff9c015920 2023-10-05T19:28:30.461 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:30.454+0000 7effb2d2ae40 20 remove_watcher() i=5 2023-10-05T19:28:30.461 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:30.454+0000 7effb2d2ae40 1 -- 172.21.15.167:0/1048063379 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:111 7.14 7:2b04a3e9:::notify.6:head [watch unwatch cookie 94425418575056] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x55e12187f8d0 con 0x7eff9c015920 2023-10-05T19:28:30.463 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:30.458+0000 7effb1060700 1 -- 172.21.15.167:0/1048063379 <== osd.0 v2:172.21.15.118:6800/27690 30 ==== osd_op_reply(111 notify.6 [watch unwatch cookie 94425418575056] v30'89 uv87 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7effac052040 con 0x7eff9c015920 2023-10-05T19:28:30.463 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:30.458+0000 7effb2d2ae40 20 remove_watcher() i=6 2023-10-05T19:28:30.463 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:30.458+0000 7effb2d2ae40 1 -- 172.21.15.167:0/1048063379 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:112 7.9 7:93e5b521:::notify.7:head [watch unwatch cookie 94425418577424] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x55e121880320 con 0x7eff9c015920 2023-10-05T19:28:30.466 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:30.462+0000 7effb1060700 1 -- 172.21.15.167:0/1048063379 <== osd.0 v2:172.21.15.118:6800/27690 31 ==== osd_op_reply(112 notify.7 [watch unwatch cookie 94425418577424] v30'89 uv87 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7effac052040 con 0x7eff9c015920 2023-10-05T19:28:30.466 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:30.462+0000 7effb2d2ae40 20 remove_watcher() i=7 2023-10-05T19:28:30.468 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:30.462+0000 7eff81fe3700 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2023-10-05T19:28:30.468 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:30.462+0000 7effb2d2ae40 1 -- 172.21.15.167:0/1048063379 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:113 6.4 6:22d26bf9:::data_loggenerations_metadata:head [watch unwatch cookie 94425418581872] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x55e121881a20 con 0x55e1218462e0 2023-10-05T19:28:30.471 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:30.466+0000 7effb1060700 1 -- 172.21.15.167:0/1048063379 <== osd.1 v2:172.21.15.167:6800/30035 39 ==== osd_op_reply(113 data_loggenerations_metadata [watch unwatch cookie 94425418581872] v30'93 uv1 ondisk = 0) v8 ==== 172+0+0 (crc 0 0 0) 0x7effac052040 con 0x55e1218462e0 2023-10-05T19:28:30.472 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:30.466+0000 7effb2d2ae40 1 -- 172.21.15.167:0/1048063379 >> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] conn(0x7eff9c015920 msgr2=0x7eff9c014370 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-05T19:28:30.472 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:30.466+0000 7effb2d2ae40 1 --2- 172.21.15.167:0/1048063379 >> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] conn(0x7eff9c015920 0x7eff9c014370 crc :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-05T19:28:30.472 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:30.466+0000 7effb2d2ae40 1 -- 172.21.15.167:0/1048063379 >> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] conn(0x55e1218462e0 msgr2=0x55e1218764e0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-05T19:28:30.472 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:30.466+0000 7effb2d2ae40 1 --2- 172.21.15.167:0/1048063379 >> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] conn(0x55e1218462e0 0x55e1218764e0 crc :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-05T19:28:30.472 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:30.466+0000 7effb2d2ae40 1 -- 172.21.15.167:0/1048063379 >> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] conn(0x7effa400d750 msgr2=0x7effa403a610 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-05T19:28:30.472 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:30.466+0000 7effb2d2ae40 1 --2- 172.21.15.167:0/1048063379 >> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] conn(0x7effa400d750 0x7effa403a610 crc :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-05T19:28:30.472 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:30.466+0000 7effb2d2ae40 1 -- 172.21.15.167:0/1048063379 >> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] conn(0x7effa0002300 msgr2=0x55e12187bfe0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-05T19:28:30.472 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:30.466+0000 7effb2d2ae40 1 --2- 172.21.15.167:0/1048063379 >> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] conn(0x7effa0002300 0x55e12187bfe0 crc :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-05T19:28:30.472 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:30.466+0000 7effb2d2ae40 1 -- 172.21.15.167:0/1048063379 >> [v2:172.21.15.167:6824/29889,v1:172.21.15.167:6825/29889] conn(0x7eff94030ca0 msgr2=0x7eff94033150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-05T19:28:30.473 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:30.466+0000 7effb2d2ae40 1 --2- 172.21.15.167:0/1048063379 >> [v2:172.21.15.167:6824/29889,v1:172.21.15.167:6825/29889] conn(0x7eff94030ca0 0x7eff94033150 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7eff9c009830 tx=0x7eff9c00c040 comp rx=0 tx=0).stop 2023-10-05T19:28:30.473 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:30.466+0000 7effb1060700 1 -- 172.21.15.167:0/1048063379 reap_dead start 2023-10-05T19:28:30.473 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:30.466+0000 7effb2d2ae40 1 -- 172.21.15.167:0/1048063379 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55e121841d00 msgr2=0x55e12186ab30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-05T19:28:30.473 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:30.466+0000 7effb2d2ae40 1 --2- 172.21.15.167:0/1048063379 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55e121841d00 0x55e12186ab30 secure :-1 s=READY pgs=311 cs=0 l=1 rev1=1 crypto rx=0x7effa4001b70 tx=0x7effa4020000 comp rx=0 tx=0).stop 2023-10-05T19:28:30.473 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:30.466+0000 7effb2d2ae40 1 -- 172.21.15.167:0/1048063379 shutdown_connections 2023-10-05T19:28:30.473 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:30.466+0000 7effb2d2ae40 1 --2- 172.21.15.167:0/1048063379 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55e121841d00 0x55e12186ab30 unknown :-1 s=CLOSED pgs=311 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-05T19:28:30.473 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:30.466+0000 7effb2d2ae40 1 -- 172.21.15.167:0/1048063379 >> 172.21.15.167:0/1048063379 conn(0x55e12183d720 msgr2=0x55e121859f70 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-05T19:28:30.473 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:30.466+0000 7effb2d2ae40 1 -- 172.21.15.167:0/1048063379 shutdown_connections 2023-10-05T19:28:30.473 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:30.466+0000 7effb2d2ae40 1 -- 172.21.15.167:0/1048063379 wait complete. 2023-10-05T19:28:30.474 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:30.466+0000 7effb2d2ae40 1 -- 172.21.15.167:0/1066262387 >> [v2:172.21.15.167:6824/29889,v1:172.21.15.167:6825/29889] conn(0x7efef0030b70 msgr2=0x7efef0033020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-05T19:28:30.474 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:30.466+0000 7effb2d2ae40 1 --2- 172.21.15.167:0/1066262387 >> [v2:172.21.15.167:6824/29889,v1:172.21.15.167:6825/29889] conn(0x7efef0030b70 0x7efef0033020 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x55e121883cf0 tx=0x7effac00ae60 comp rx=0 tx=0).stop 2023-10-05T19:28:30.474 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:30.466+0000 7effb2d2ae40 1 -- 172.21.15.167:0/1066262387 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55e121891f80 msgr2=0x55e121897ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-05T19:28:30.474 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:30.466+0000 7effb2d2ae40 1 --2- 172.21.15.167:0/1066262387 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55e121891f80 0x55e121897ae0 secure :-1 s=READY pgs=313 cs=0 l=1 rev1=1 crypto rx=0x7eff9c005df0 tx=0x7eff9c02e030 comp rx=0 tx=0).stop 2023-10-05T19:28:30.475 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:30.466+0000 7effb2d2ae40 1 -- 172.21.15.167:0/1066262387 shutdown_connections 2023-10-05T19:28:30.475 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:30.466+0000 7effb2d2ae40 1 --2- 172.21.15.167:0/1066262387 >> [v2:172.21.15.167:6824/29889,v1:172.21.15.167:6825/29889] conn(0x7efef0030b70 0x7efef0033020 unknown :-1 s=CLOSED pgs=111 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-05T19:28:30.475 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:30.466+0000 7effb2d2ae40 1 --2- 172.21.15.167:0/1066262387 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55e121891f80 0x55e121897ae0 unknown :-1 s=CLOSED pgs=313 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-05T19:28:30.475 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:30.466+0000 7effb2d2ae40 1 -- 172.21.15.167:0/1066262387 >> 172.21.15.167:0/1066262387 conn(0x55e12188d340 msgr2=0x55e12188b440 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-05T19:28:30.475 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:30.466+0000 7effb2d2ae40 1 -- 172.21.15.167:0/1066262387 shutdown_connections 2023-10-05T19:28:30.475 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:30.466+0000 7effb2d2ae40 1 -- 172.21.15.167:0/1066262387 wait complete. 2023-10-05T19:28:30.475 INFO:teuthology.orchestra.run.smithi167.stdout:[ 2023-10-05T19:28:30.475 INFO:teuthology.orchestra.run.smithi167.stdout: "myfoo" 2023-10-05T19:28:30.476 INFO:teuthology.orchestra.run.smithi167.stdout:] 2023-10-05T19:28:30.478 DEBUG:tasks.util.rgw: json result: ['myfoo'] 2023-10-05T19:28:30.551 INFO:tasks.radosgw_admin:RL: list_buckets foo bytes_out=0 bytes_in=0 failed=False 2023-10-05T19:28:30.552 INFO:tasks.util.rgw:rgwadmin: client.0 : ['bucket', 'list'] 2023-10-05T19:28:30.552 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'ceph', 'bucket', 'list'] 2023-10-05T19:28:30.552 DEBUG:teuthology.orchestra.run.smithi167:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster ceph bucket list 2023-10-05T19:28:30.591 INFO:teuthology.orchestra.run.smithi167.stderr:ignoring --setuser ceph since I am not root 2023-10-05T19:28:30.592 INFO:teuthology.orchestra.run.smithi167.stderr:ignoring --setgroup ceph since I am not root 2023-10-05T19:28:30.592 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:30.586+0000 7feca74d6e40 1 Processor -- start 2023-10-05T19:28:30.592 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:30.586+0000 7feca74d6e40 1 -- start start 2023-10-05T19:28:30.593 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:30.586+0000 7feca74d6e40 1 --2- >> v2:172.21.15.118:3300/0 conn(0x5565578505d0 0x5565578529b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-05T19:28:30.593 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:30.586+0000 7feca74d6e40 1 -- --> v1:172.21.15.118:6789/0 -- auth(proto 0 26 bytes epoch 0) v1 -- 0x556557796c00 con 0x55655784bff0 2023-10-05T19:28:30.593 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:30.586+0000 7feca74d6e40 1 -- --> v2:172.21.15.118:3300/0 -- mon_getmap magic: 0 v1 -- 0x556557753ee0 con 0x5565578505d0 2023-10-05T19:28:30.594 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:30.586+0000 7feca480a700 1 --2- >> v2:172.21.15.118:3300/0 conn(0x5565578505d0 0x5565578529b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-05T19:28:30.594 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:30.586+0000 7feca500b700 1 --1- >> v1:172.21.15.118:6789/0 conn(0x55655784bff0 0x55655784c3c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.118:6789/0 says I am v1:172.21.15.167:56566/0 (socket says 172.21.15.167:56566) 2023-10-05T19:28:30.594 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:30.586+0000 7feca500b700 1 -- 172.21.15.167:0/3696000937 learned_addr learned my addr 172.21.15.167:0/3696000937 (peer_addr_for_me v1:172.21.15.167:0/0) 2023-10-05T19:28:30.594 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:30.586+0000 7fec9ffff700 1 -- 172.21.15.167:0/3696000937 <== mon.0 v1:172.21.15.118:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 566540897 0 0) 0x556557796c00 con 0x55655784bff0 2023-10-05T19:28:30.594 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:30.586+0000 7fec9ffff700 1 -- 172.21.15.167:0/3696000937 --> v1:172.21.15.118:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fec94001e10 con 0x55655784bff0 2023-10-05T19:28:30.594 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:30.586+0000 7feca480a700 1 -- 172.21.15.167:0/3696000937 >> v1:172.21.15.118:6789/0 conn(0x55655784bff0 legacy=0x55655784c3c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-05T19:28:30.594 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:30.590+0000 7feca480a700 1 -- 172.21.15.167:0/3696000937 --> v2:172.21.15.118:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55655784cc00 con 0x5565578505d0 2023-10-05T19:28:30.596 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:30.590+0000 7feca480a700 1 --2- 172.21.15.167:0/3696000937 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5565578505d0 0x5565578529b0 secure :-1 s=READY pgs=315 cs=0 l=1 rev1=1 crypto rx=0x7fec9000a5d0 tx=0x7fec90005a50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5961d5d6e7d0e server_cookie=0 in_seq=0 out_seq=0 2023-10-05T19:28:30.596 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:30.590+0000 7fec9ffff700 1 -- 172.21.15.167:0/3696000937 <== mon.0 v2:172.21.15.118:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fec90007610 con 0x5565578505d0 2023-10-05T19:28:30.596 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:30.590+0000 7fec9ffff700 1 -- 172.21.15.167:0/3696000937 <== mon.0 v2:172.21.15.118:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fec90007770 con 0x5565578505d0 2023-10-05T19:28:30.596 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:30.590+0000 7fec9ffff700 1 -- 172.21.15.167:0/3696000937 <== mon.0 v2:172.21.15.118:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fec90007a20 con 0x5565578505d0 2023-10-05T19:28:30.596 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:30.590+0000 7feca74d6e40 1 -- 172.21.15.167:0/3696000937 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5565578505d0 msgr2=0x5565578529b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-05T19:28:30.596 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:30.590+0000 7feca74d6e40 1 --2- 172.21.15.167:0/3696000937 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5565578505d0 0x5565578529b0 secure :-1 s=READY pgs=315 cs=0 l=1 rev1=1 crypto rx=0x7fec9000a5d0 tx=0x7fec90005a50 comp rx=0 tx=0).stop 2023-10-05T19:28:30.596 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:30.590+0000 7feca74d6e40 1 -- 172.21.15.167:0/3696000937 shutdown_connections 2023-10-05T19:28:30.596 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:30.590+0000 7feca74d6e40 1 --2- 172.21.15.167:0/3696000937 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5565578505d0 0x5565578529b0 unknown :-1 s=CLOSED pgs=315 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-05T19:28:30.597 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:30.590+0000 7feca74d6e40 1 -- 172.21.15.167:0/3696000937 >> 172.21.15.167:0/3696000937 conn(0x5565577bf840 msgr2=0x55655784a4e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-05T19:28:30.597 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:30.590+0000 7feca74d6e40 1 -- 172.21.15.167:0/3696000937 shutdown_connections 2023-10-05T19:28:30.597 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:30.590+0000 7feca74d6e40 1 -- 172.21.15.167:0/3696000937 wait complete. 2023-10-05T19:28:30.597 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:30.590+0000 7feca74d6e40 1 Processor -- start 2023-10-05T19:28:30.597 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:30.590+0000 7feca74d6e40 1 -- start start 2023-10-05T19:28:30.598 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:30.594+0000 7feca74d6e40 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55655784bff0 0x55655786af10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-05T19:28:30.598 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:30.594+0000 7feca74d6e40 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x556557717e30 con 0x55655784bff0 2023-10-05T19:28:30.599 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:30.594+0000 7feca500b700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55655784bff0 0x55655786af10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-05T19:28:30.599 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:30.594+0000 7feca500b700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55655784bff0 0x55655786af10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.118:3300/0 says I am v2:172.21.15.167:58140/0 (socket says 172.21.15.167:58140) 2023-10-05T19:28:30.599 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:30.594+0000 7feca500b700 1 -- 172.21.15.167:0/2037764927 learned_addr learned my addr 172.21.15.167:0/2037764927 (peer_addr_for_me v2:172.21.15.167:0/0) 2023-10-05T19:28:30.599 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:30.594+0000 7feca500b700 1 -- 172.21.15.167:0/2037764927 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fec90005cf0 con 0x55655784bff0 2023-10-05T19:28:30.599 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:30.594+0000 7feca500b700 1 --2- 172.21.15.167:0/2037764927 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55655784bff0 0x55655786af10 secure :-1 s=READY pgs=316 cs=0 l=1 rev1=1 crypto rx=0x7fec98006e30 tx=0x7fec98006a10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=dfe152f9849b6067 server_cookie=0 in_seq=0 out_seq=0 2023-10-05T19:28:30.600 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:30.594+0000 7fec9effd700 1 -- 172.21.15.167:0/2037764927 <== mon.0 v2:172.21.15.118:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fec98006110 con 0x55655784bff0 2023-10-05T19:28:30.600 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:30.594+0000 7fec9effd700 1 -- 172.21.15.167:0/2037764927 <== mon.0 v2:172.21.15.118:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fec98006290 con 0x55655784bff0 2023-10-05T19:28:30.600 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:30.594+0000 7fec9effd700 1 -- 172.21.15.167:0/2037764927 <== mon.0 v2:172.21.15.118:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fec98004520 con 0x55655784bff0 2023-10-05T19:28:30.601 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:30.594+0000 7feca74d6e40 1 -- 172.21.15.167:0/2037764927 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55655784bff0 msgr2=0x55655786af10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-05T19:28:30.602 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:30.594+0000 7feca74d6e40 1 --2- 172.21.15.167:0/2037764927 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55655784bff0 0x55655786af10 secure :-1 s=READY pgs=316 cs=0 l=1 rev1=1 crypto rx=0x7fec98006e30 tx=0x7fec98006a10 comp rx=0 tx=0).stop 2023-10-05T19:28:30.602 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:30.594+0000 7feca74d6e40 1 -- 172.21.15.167:0/2037764927 shutdown_connections 2023-10-05T19:28:30.602 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:30.594+0000 7feca74d6e40 1 --2- 172.21.15.167:0/2037764927 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55655784bff0 0x55655786af10 unknown :-1 s=CLOSED pgs=316 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-05T19:28:30.602 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:30.594+0000 7feca74d6e40 1 -- 172.21.15.167:0/2037764927 >> 172.21.15.167:0/2037764927 conn(0x5565577bf840 msgr2=0x556557866ad0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-05T19:28:30.602 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:30.594+0000 7feca74d6e40 1 -- 172.21.15.167:0/2037764927 shutdown_connections 2023-10-05T19:28:30.602 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:30.594+0000 7feca74d6e40 1 -- 172.21.15.167:0/2037764927 wait complete. 2023-10-05T19:28:30.603 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:30.598+0000 7feca74d6e40 1 Processor -- start 2023-10-05T19:28:30.603 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:30.598+0000 7feca74d6e40 1 -- start start 2023-10-05T19:28:30.603 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:30.598+0000 7feca74d6e40 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55655784bff0 0x556557871cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-05T19:28:30.604 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:30.598+0000 7feca74d6e40 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x5565576eaed0 con 0x55655784bff0 2023-10-05T19:28:30.604 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:30.598+0000 7feca500b700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55655784bff0 0x556557871cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-05T19:28:30.604 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:30.598+0000 7feca500b700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55655784bff0 0x556557871cf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.118:3300/0 says I am v2:172.21.15.167:58142/0 (socket says 172.21.15.167:58142) 2023-10-05T19:28:30.604 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:30.598+0000 7feca500b700 1 -- 172.21.15.167:0/1420689918 learned_addr learned my addr 172.21.15.167:0/1420689918 (peer_addr_for_me v2:172.21.15.167:0/0) 2023-10-05T19:28:30.605 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:30.598+0000 7feca500b700 1 -- 172.21.15.167:0/1420689918 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fec94001e10 con 0x55655784bff0 2023-10-05T19:28:30.605 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:30.598+0000 7feca500b700 1 --2- 172.21.15.167:0/1420689918 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55655784bff0 0x556557871cf0 secure :-1 s=READY pgs=317 cs=0 l=1 rev1=1 crypto rx=0x7fec9800f000 tx=0x7fec98010000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-05T19:28:30.605 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:30.598+0000 7fec9d7fa700 1 -- 172.21.15.167:0/1420689918 <== mon.0 v2:172.21.15.118:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fec98006110 con 0x55655784bff0 2023-10-05T19:28:30.605 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:30.598+0000 7fec9d7fa700 1 -- 172.21.15.167:0/1420689918 <== mon.0 v2:172.21.15.118:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fec980044f0 con 0x55655784bff0 2023-10-05T19:28:30.606 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:30.598+0000 7feca74d6e40 1 -- 172.21.15.167:0/1420689918 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x556557872230 con 0x55655784bff0 2023-10-05T19:28:30.606 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:30.598+0000 7fec9d7fa700 1 -- 172.21.15.167:0/1420689918 <== mon.0 v2:172.21.15.118:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fec98014bd0 con 0x55655784bff0 2023-10-05T19:28:30.606 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:30.598+0000 7feca74d6e40 1 -- 172.21.15.167:0/1420689918 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x556557872670 con 0x55655784bff0 2023-10-05T19:28:30.606 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:30.602+0000 7fec9d7fa700 1 -- 172.21.15.167:0/1420689918 <== mon.0 v2:172.21.15.118:3300/0 4 ==== mgrmap(e 5) v1 ==== 40329+0+0 (secure 0 0 0) 0x7fec98028460 con 0x55655784bff0 2023-10-05T19:28:30.607 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:30.602+0000 7feca74d6e40 20 rados->read ofs=0 len=0 2023-10-05T19:28:30.607 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:30.602+0000 7fec9d7fa700 1 --2- 172.21.15.167:0/1420689918 >> [v2:172.21.15.167:6824/29889,v1:172.21.15.167:6825/29889] conn(0x7fec88030c50 0x7fec88033100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-05T19:28:30.607 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:30.602+0000 7fec9d7fa700 1 -- 172.21.15.167:0/1420689918 <== mon.0 v2:172.21.15.118:3300/0 5 ==== osd_map(30..30 src has 1..30) v4 ==== 6057+0+0 (secure 0 0 0) 0x7fec98045120 con 0x55655784bff0 2023-10-05T19:28:30.607 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:30.602+0000 7feca480a700 1 --2- 172.21.15.167:0/1420689918 >> [v2:172.21.15.167:6824/29889,v1:172.21.15.167:6825/29889] conn(0x7fec88030c50 0x7fec88033100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-05T19:28:30.608 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:30.602+0000 7feca74d6e40 1 --2- 172.21.15.167:0/1420689918 >> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] conn(0x5565578505d0 0x55655787d780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-05T19:28:30.608 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:30.602+0000 7feca74d6e40 1 -- 172.21.15.167:0/1420689918 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:1 5.12 5:49953fa1:::default.realm:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x55655787de80 con 0x5565578505d0 2023-10-05T19:28:30.608 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:30.602+0000 7feca580c700 1 --2- 172.21.15.167:0/1420689918 >> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] conn(0x5565578505d0 0x55655787d780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-05T19:28:30.609 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:30.602+0000 7feca480a700 1 --2- 172.21.15.167:0/1420689918 >> [v2:172.21.15.167:6824/29889,v1:172.21.15.167:6825/29889] conn(0x7fec88030c50 0x7fec88033100 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7fec900030c0 tx=0x7fec90014000 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-05T19:28:30.609 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:30.602+0000 7feca580c700 1 --2- 172.21.15.167:0/1420689918 >> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] conn(0x5565578505d0 0x55655787d780 crc :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-05T19:28:30.610 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:30.602+0000 7feca580c700 1 -- 172.21.15.167:0/1420689918 <== osd.1 v2:172.21.15.167:6800/30035 1 ==== osd_op_reply(1 default.realm [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 ==== 157+0+0 (crc 0 0 0) 0x7feca000a500 con 0x5565578505d0 2023-10-05T19:28:30.610 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:30.602+0000 7feca74d6e40 20 rados_obj.operate() r=-2 bl.length=0 2023-10-05T19:28:30.610 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:30.602+0000 7feca74d6e40 20 realm 2023-10-05T19:28:30.610 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:30.602+0000 7feca74d6e40 20 rados->read ofs=0 len=0 2023-10-05T19:28:30.610 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:30.602+0000 7feca74d6e40 1 -- 172.21.15.167:0/1420689918 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:2 5.12 5:49953fa1:::default.realm:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x55655787ada0 con 0x5565578505d0 2023-10-05T19:28:30.612 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:30.606+0000 7feca580c700 1 -- 172.21.15.167:0/1420689918 <== osd.1 v2:172.21.15.167:6800/30035 2 ==== osd_op_reply(2 default.realm [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 ==== 157+0+0 (crc 0 0 0) 0x7feca000a500 con 0x5565578505d0 2023-10-05T19:28:30.613 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:30.606+0000 7feca74d6e40 20 rados_obj.operate() r=-2 bl.length=0 2023-10-05T19:28:30.613 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:30.606+0000 7feca74d6e40 4 RGWPeriod::init failed to init realm id : (2) No such file or directory 2023-10-05T19:28:30.613 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:30.606+0000 7feca74d6e40 20 rados->read ofs=0 len=0 2023-10-05T19:28:30.613 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:30.606+0000 7feca74d6e40 1 -- 172.21.15.167:0/1420689918 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:3 5.12 5:49953fa1:::default.realm:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x55655787e960 con 0x5565578505d0 2023-10-05T19:28:30.613 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:30.606+0000 7feca580c700 1 -- 172.21.15.167:0/1420689918 <== osd.1 v2:172.21.15.167:6800/30035 3 ==== osd_op_reply(3 default.realm [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 ==== 157+0+0 (crc 0 0 0) 0x7feca000a500 con 0x5565578505d0 2023-10-05T19:28:30.613 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:30.606+0000 7feca74d6e40 20 rados_obj.operate() r=-2 bl.length=0 2023-10-05T19:28:30.613 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:30.606+0000 7feca74d6e40 20 rados->read ofs=0 len=0 2023-10-05T19:28:30.613 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:30.606+0000 7feca74d6e40 1 -- 172.21.15.167:0/1420689918 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:4 5.f 5:f43b6ece:::zone_names.default:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x55655787e510 con 0x5565578505d0 2023-10-05T19:28:30.613 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:30.606+0000 7feca580c700 1 -- 172.21.15.167:0/1420689918 <== osd.1 v2:172.21.15.167:6800/30035 4 ==== osd_op_reply(4 zone_names.default [read 0~46 out=46b] v0'0 uv1 ondisk = 0) v8 ==== 162+0+46 (crc 0 0 0) 0x7feca000a500 con 0x5565578505d0 2023-10-05T19:28:30.614 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:30.606+0000 7feca74d6e40 20 rados_obj.operate() r=0 bl.length=46 2023-10-05T19:28:30.614 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:30.606+0000 7feca74d6e40 20 rados->read ofs=0 len=0 2023-10-05T19:28:30.614 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:30.606+0000 7feca74d6e40 1 -- 172.21.15.167:0/1420689918 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:5 5.1d 5:b8374eb4:::zone_info.9a9df965-6612-45d4-a45f-6e3c889bacc2:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x55655787f390 con 0x5565578505d0 2023-10-05T19:28:30.614 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:30.606+0000 7feca580c700 1 -- 172.21.15.167:0/1420689918 <== osd.1 v2:172.21.15.167:6800/30035 5 ==== osd_op_reply(5 zone_info.9a9df965-6612-45d4-a45f-6e3c889bacc2 [read 0~874 out=874b] v0'0 uv2 ondisk = 0) v8 ==== 190+0+874 (crc 0 0 0) 0x7feca000a500 con 0x5565578505d0 2023-10-05T19:28:30.614 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:30.606+0000 7feca74d6e40 20 rados_obj.operate() r=0 bl.length=874 2023-10-05T19:28:30.614 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:30.606+0000 7feca74d6e40 20 searching for the correct realm 2023-10-05T19:28:30.614 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:30.606+0000 7feca74d6e40 1 --2- 172.21.15.167:0/1420689918 >> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] conn(0x5565578811e0 0x5565578835c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-05T19:28:30.614 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:30.606+0000 7feca74d6e40 1 -- 172.21.15.167:0/1420689918 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:6 5.0 5:00000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x556557883b00 con 0x5565578811e0 2023-10-05T19:28:30.615 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:30.606+0000 7feca500b700 1 --2- 172.21.15.167:0/1420689918 >> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] conn(0x5565578811e0 0x5565578835c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-05T19:28:30.615 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:30.606+0000 7feca500b700 1 --2- 172.21.15.167:0/1420689918 >> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] conn(0x5565578811e0 0x5565578835c0 crc :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-05T19:28:30.616 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:30.610+0000 7feca500b700 1 -- 172.21.15.167:0/1420689918 <== osd.3 v2:172.21.15.167:6808/30036 1 ==== osd_op_reply(6 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fec98024020 con 0x5565578811e0 2023-10-05T19:28:30.616 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:30.610+0000 7feca500b700 1 --2- 172.21.15.167:0/1420689918 >> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] conn(0x7fec980123c0 0x7fec98028710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-05T19:28:30.616 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:30.610+0000 7feca500b700 1 -- 172.21.15.167:0/1420689918 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:7 5.10 5:08000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x556557883b00 con 0x7fec980123c0 2023-10-05T19:28:30.616 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:30.610+0000 7feca480a700 1 --2- 172.21.15.167:0/1420689918 >> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] conn(0x7fec980123c0 0x7fec98028710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-05T19:28:30.616 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:30.610+0000 7feca480a700 1 --2- 172.21.15.167:0/1420689918 >> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] conn(0x7fec980123c0 0x7fec98028710 crc :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-05T19:28:30.617 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:30.610+0000 7feca480a700 1 -- 172.21.15.167:0/1420689918 <== osd.2 v2:172.21.15.167:6816/30037 1 ==== osd_op_reply(7 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fec900079f0 con 0x7fec980123c0 2023-10-05T19:28:30.617 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:30.610+0000 7feca480a700 1 --2- 172.21.15.167:0/1420689918 >> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] conn(0x7fec90007030 0x7fec9001d430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-05T19:28:30.617 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:30.610+0000 7feca480a700 1 -- 172.21.15.167:0/1420689918 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:8 5.8 5:10000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x556557883b00 con 0x7fec90007030 2023-10-05T19:28:30.617 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:30.610+0000 7feca580c700 1 --2- 172.21.15.167:0/1420689918 >> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] conn(0x7fec90007030 0x7fec9001d430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-05T19:28:30.618 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:30.610+0000 7feca580c700 1 --2- 172.21.15.167:0/1420689918 >> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] conn(0x7fec90007030 0x7fec9001d430 crc :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-05T19:28:30.618 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:30.614+0000 7feca580c700 1 -- 172.21.15.167:0/1420689918 <== osd.0 v2:172.21.15.118:6800/27690 1 ==== osd_op_reply(8 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7feca0014040 con 0x7fec90007030 2023-10-05T19:28:30.618 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:30.614+0000 7feca580c700 1 -- 172.21.15.167:0/1420689918 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:9 5.18 5:18000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x556557883b00 con 0x5565578505d0 2023-10-05T19:28:30.619 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:30.614+0000 7feca580c700 1 -- 172.21.15.167:0/1420689918 <== osd.1 v2:172.21.15.167:6800/30035 6 ==== osd_op_reply(9 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7feca0014040 con 0x5565578505d0 2023-10-05T19:28:30.619 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:30.614+0000 7feca580c700 1 -- 172.21.15.167:0/1420689918 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:10 5.4 5:20000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x556557883b00 con 0x5565578505d0 2023-10-05T19:28:30.620 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:30.614+0000 7feca580c700 1 -- 172.21.15.167:0/1420689918 <== osd.1 v2:172.21.15.167:6800/30035 7 ==== osd_op_reply(10 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7feca0014040 con 0x5565578505d0 2023-10-05T19:28:30.620 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:30.614+0000 7feca580c700 1 -- 172.21.15.167:0/1420689918 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:11 5.14 5:28000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x556557883b00 con 0x7fec90007030 2023-10-05T19:28:30.620 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:30.614+0000 7feca580c700 1 -- 172.21.15.167:0/1420689918 <== osd.0 v2:172.21.15.118:6800/27690 2 ==== osd_op_reply(11 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7feca0014040 con 0x7fec90007030 2023-10-05T19:28:30.621 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:30.614+0000 7feca580c700 1 -- 172.21.15.167:0/1420689918 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:12 5.c 5:30000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x556557883b00 con 0x7fec90007030 2023-10-05T19:28:30.621 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:30.614+0000 7feca580c700 1 -- 172.21.15.167:0/1420689918 <== osd.0 v2:172.21.15.118:6800/27690 3 ==== osd_op_reply(12 [pgnls start_epoch 30 out=112b] v23'1 uv1 ondisk = 1) v8 ==== 144+0+112 (crc 0 0 0) 0x7feca0014040 con 0x7fec90007030 2023-10-05T19:28:30.621 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:30.614+0000 7feca580c700 1 -- 172.21.15.167:0/1420689918 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:13 5.1c 5:38000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x556557883b00 con 0x7fec980123c0 2023-10-05T19:28:30.622 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:30.618+0000 7feca480a700 1 -- 172.21.15.167:0/1420689918 <== osd.2 v2:172.21.15.167:6816/30037 2 ==== osd_op_reply(13 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fec90029040 con 0x7fec980123c0 2023-10-05T19:28:30.622 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:30.618+0000 7feca480a700 1 -- 172.21.15.167:0/1420689918 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:14 5.2 5:40000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x556557883b00 con 0x5565578811e0 2023-10-05T19:28:30.623 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:30.618+0000 7feca500b700 1 -- 172.21.15.167:0/1420689918 <== osd.3 v2:172.21.15.167:6808/30036 2 ==== osd_op_reply(14 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fec98024020 con 0x5565578811e0 2023-10-05T19:28:30.623 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:30.618+0000 7feca500b700 1 -- 172.21.15.167:0/1420689918 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:15 5.12 5:48000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x556557883b00 con 0x5565578505d0 2023-10-05T19:28:30.624 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:30.618+0000 7feca580c700 1 -- 172.21.15.167:0/1420689918 <== osd.1 v2:172.21.15.167:6800/30035 8 ==== osd_op_reply(15 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7feca0014040 con 0x5565578505d0 2023-10-05T19:28:30.624 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:30.618+0000 7feca580c700 1 -- 172.21.15.167:0/1420689918 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:16 5.a 5:50000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x556557883b00 con 0x7fec90007030 2023-10-05T19:28:30.625 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:30.618+0000 7feca580c700 1 -- 172.21.15.167:0/1420689918 <== osd.0 v2:172.21.15.118:6800/27690 4 ==== osd_op_reply(16 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7feca0022040 con 0x7fec90007030 2023-10-05T19:28:30.625 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:30.618+0000 7feca580c700 1 -- 172.21.15.167:0/1420689918 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:17 5.1a 5:58000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x556557883b00 con 0x5565578505d0 2023-10-05T19:28:30.625 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:30.618+0000 7feca580c700 1 -- 172.21.15.167:0/1420689918 <== osd.1 v2:172.21.15.167:6800/30035 9 ==== osd_op_reply(17 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7feca0022040 con 0x5565578505d0 2023-10-05T19:28:30.626 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:30.618+0000 7feca580c700 1 -- 172.21.15.167:0/1420689918 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:18 5.6 5:60000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x556557883b00 con 0x7fec980123c0 2023-10-05T19:28:30.626 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:30.622+0000 7feca480a700 1 -- 172.21.15.167:0/1420689918 <== osd.2 v2:172.21.15.167:6816/30037 3 ==== osd_op_reply(18 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fec90029040 con 0x7fec980123c0 2023-10-05T19:28:30.626 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:30.622+0000 7feca480a700 1 -- 172.21.15.167:0/1420689918 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:19 5.16 5:68000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x556557883b00 con 0x5565578811e0 2023-10-05T19:28:30.627 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:30.622+0000 7feca500b700 1 -- 172.21.15.167:0/1420689918 <== osd.3 v2:172.21.15.167:6808/30036 3 ==== osd_op_reply(19 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fec98024020 con 0x5565578811e0 2023-10-05T19:28:30.627 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:30.622+0000 7feca500b700 1 -- 172.21.15.167:0/1420689918 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:20 5.e 5:70000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x556557883b00 con 0x5565578811e0 2023-10-05T19:28:30.627 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:30.622+0000 7feca500b700 1 -- 172.21.15.167:0/1420689918 <== osd.3 v2:172.21.15.167:6808/30036 4 ==== osd_op_reply(20 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fec98024020 con 0x5565578811e0 2023-10-05T19:28:30.628 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:30.622+0000 7feca500b700 1 -- 172.21.15.167:0/1420689918 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:21 5.1e 5:78000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x556557883b00 con 0x7fec90007030 2023-10-05T19:28:30.628 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:30.622+0000 7feca580c700 1 -- 172.21.15.167:0/1420689918 <== osd.0 v2:172.21.15.118:6800/27690 5 ==== osd_op_reply(21 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7feca0022040 con 0x7fec90007030 2023-10-05T19:28:30.628 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:30.622+0000 7feca580c700 1 -- 172.21.15.167:0/1420689918 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:22 5.1 5:80000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x556557883b00 con 0x5565578811e0 2023-10-05T19:28:30.629 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:30.622+0000 7feca500b700 1 -- 172.21.15.167:0/1420689918 <== osd.3 v2:172.21.15.167:6808/30036 5 ==== osd_op_reply(22 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fec98024020 con 0x5565578811e0 2023-10-05T19:28:30.629 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:30.622+0000 7feca500b700 1 -- 172.21.15.167:0/1420689918 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:23 5.11 5:88000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x556557883b00 con 0x5565578811e0 2023-10-05T19:28:30.630 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:30.622+0000 7feca500b700 1 -- 172.21.15.167:0/1420689918 <== osd.3 v2:172.21.15.167:6808/30036 6 ==== osd_op_reply(23 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fec98024020 con 0x5565578811e0 2023-10-05T19:28:30.630 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:30.626+0000 7feca500b700 1 -- 172.21.15.167:0/1420689918 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:24 5.9 5:90000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x556557883b00 con 0x5565578505d0 2023-10-05T19:28:30.630 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:30.626+0000 7feca580c700 1 -- 172.21.15.167:0/1420689918 <== osd.1 v2:172.21.15.167:6800/30035 10 ==== osd_op_reply(24 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7feca0022040 con 0x5565578505d0 2023-10-05T19:28:30.631 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:30.626+0000 7feca580c700 1 -- 172.21.15.167:0/1420689918 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:25 5.19 5:98000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x556557883b00 con 0x5565578505d0 2023-10-05T19:28:30.631 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:30.626+0000 7feca580c700 1 -- 172.21.15.167:0/1420689918 <== osd.1 v2:172.21.15.167:6800/30035 11 ==== osd_op_reply(25 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7feca0022040 con 0x5565578505d0 2023-10-05T19:28:30.631 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:30.626+0000 7feca580c700 1 -- 172.21.15.167:0/1420689918 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:26 5.5 5:a0000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x556557883b00 con 0x7fec90007030 2023-10-05T19:28:30.632 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:30.626+0000 7feca580c700 1 -- 172.21.15.167:0/1420689918 <== osd.0 v2:172.21.15.118:6800/27690 6 ==== osd_op_reply(26 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7feca0022040 con 0x7fec90007030 2023-10-05T19:28:30.632 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:30.626+0000 7feca580c700 1 -- 172.21.15.167:0/1420689918 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:27 5.15 5:a8000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x556557883b00 con 0x5565578811e0 2023-10-05T19:28:30.633 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:30.626+0000 7feca500b700 1 -- 172.21.15.167:0/1420689918 <== osd.3 v2:172.21.15.167:6808/30036 7 ==== osd_op_reply(27 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fec98024020 con 0x5565578811e0 2023-10-05T19:28:30.633 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:30.626+0000 7feca500b700 1 -- 172.21.15.167:0/1420689918 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:28 5.d 5:b0000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x556557883b00 con 0x7fec980123c0 2023-10-05T19:28:30.634 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:30.626+0000 7feca480a700 1 -- 172.21.15.167:0/1420689918 <== osd.2 v2:172.21.15.167:6816/30037 4 ==== osd_op_reply(28 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fec90029040 con 0x7fec980123c0 2023-10-05T19:28:30.634 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:30.630+0000 7feca480a700 1 -- 172.21.15.167:0/1420689918 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:29 5.1d 5:b8000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x556557883b00 con 0x5565578505d0 2023-10-05T19:28:30.634 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:30.630+0000 7feca580c700 1 -- 172.21.15.167:0/1420689918 <== osd.1 v2:172.21.15.167:6800/30035 12 ==== osd_op_reply(29 [pgnls start_epoch 30 out=107b] v30'2 uv2 ondisk = 1) v8 ==== 144+0+107 (crc 0 0 0) 0x7feca0022040 con 0x5565578505d0 2023-10-05T19:28:30.635 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:30.630+0000 7feca580c700 1 -- 172.21.15.167:0/1420689918 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:30 5.3 5:c0000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x556557883b00 con 0x7fec90007030 2023-10-05T19:28:30.636 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:30.630+0000 7feca580c700 1 -- 172.21.15.167:0/1420689918 <== osd.0 v2:172.21.15.118:6800/27690 7 ==== osd_op_reply(30 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7feca0022040 con 0x7fec90007030 2023-10-05T19:28:30.636 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:30.630+0000 7feca580c700 1 -- 172.21.15.167:0/1420689918 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:31 5.13 5:c8000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x556557883b00 con 0x5565578811e0 2023-10-05T19:28:30.636 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:30.630+0000 7feca500b700 1 -- 172.21.15.167:0/1420689918 <== osd.3 v2:172.21.15.167:6808/30036 8 ==== osd_op_reply(31 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fec98024020 con 0x5565578811e0 2023-10-05T19:28:30.636 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:30.630+0000 7feca500b700 1 -- 172.21.15.167:0/1420689918 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:32 5.b 5:d0000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x556557883b00 con 0x7fec90007030 2023-10-05T19:28:30.637 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:30.630+0000 7feca580c700 1 -- 172.21.15.167:0/1420689918 <== osd.0 v2:172.21.15.118:6800/27690 8 ==== osd_op_reply(32 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7feca0022040 con 0x7fec90007030 2023-10-05T19:28:30.637 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:30.630+0000 7feca580c700 1 -- 172.21.15.167:0/1420689918 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:33 5.1b 5:d8000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x556557883b00 con 0x7fec980123c0 2023-10-05T19:28:30.638 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:30.634+0000 7feca480a700 1 -- 172.21.15.167:0/1420689918 <== osd.2 v2:172.21.15.167:6816/30037 5 ==== osd_op_reply(33 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fec90029040 con 0x7fec980123c0 2023-10-05T19:28:30.638 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:30.634+0000 7feca480a700 1 -- 172.21.15.167:0/1420689918 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:34 5.7 5:e0000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x556557883b00 con 0x5565578811e0 2023-10-05T19:28:30.639 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:30.634+0000 7feca500b700 1 -- 172.21.15.167:0/1420689918 <== osd.3 v2:172.21.15.167:6808/30036 9 ==== osd_op_reply(34 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fec9803d040 con 0x5565578811e0 2023-10-05T19:28:30.639 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:30.634+0000 7feca500b700 1 -- 172.21.15.167:0/1420689918 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:35 5.17 5:e8000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x556557883b00 con 0x7fec90007030 2023-10-05T19:28:30.640 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:30.634+0000 7feca580c700 1 -- 172.21.15.167:0/1420689918 <== osd.0 v2:172.21.15.118:6800/27690 9 ==== osd_op_reply(35 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7feca0022040 con 0x7fec90007030 2023-10-05T19:28:30.640 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:30.634+0000 7feca580c700 1 -- 172.21.15.167:0/1420689918 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:36 5.f 5:f0000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x556557883b00 con 0x5565578505d0 2023-10-05T19:28:30.641 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:30.634+0000 7feca580c700 1 -- 172.21.15.167:0/1420689918 <== osd.1 v2:172.21.15.167:6800/30035 13 ==== osd_op_reply(36 [pgnls start_epoch 30 out=115b] v23'2 uv2 ondisk = 1) v8 ==== 144+0+115 (crc 0 0 0) 0x7feca002f080 con 0x5565578505d0 2023-10-05T19:28:30.641 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:30.634+0000 7feca580c700 1 -- 172.21.15.167:0/1420689918 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:37 5.1f 5:f8000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x556557883b00 con 0x7fec980123c0 2023-10-05T19:28:30.642 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:30.634+0000 7feca480a700 1 -- 172.21.15.167:0/1420689918 <== osd.2 v2:172.21.15.167:6816/30037 6 ==== osd_op_reply(37 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fec90029040 con 0x7fec980123c0 2023-10-05T19:28:30.642 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:30.634+0000 7feca74d6e40 20 RGWRados::pool_iterate: got zonegroup_info.4f1f719d-9648-417b-b10c-e5d939825419 2023-10-05T19:28:30.642 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:30.634+0000 7feca74d6e40 20 RGWRados::pool_iterate: got zone_info.9a9df965-6612-45d4-a45f-6e3c889bacc2 2023-10-05T19:28:30.642 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:30.634+0000 7feca74d6e40 20 RGWRados::pool_iterate: got zone_names.default 2023-10-05T19:28:30.642 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:30.634+0000 7feca74d6e40 20 RGWRados::pool_iterate: got zonegroups_names.default 2023-10-05T19:28:30.642 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:30.634+0000 7feca74d6e40 20 rados->read ofs=0 len=0 2023-10-05T19:28:30.643 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:30.634+0000 7feca74d6e40 1 -- 172.21.15.167:0/1420689918 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:38 5.12 5:49953fa1:::default.realm:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x55655787ff30 con 0x5565578505d0 2023-10-05T19:28:30.643 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:30.638+0000 7feca580c700 1 -- 172.21.15.167:0/1420689918 <== osd.1 v2:172.21.15.167:6800/30035 14 ==== osd_op_reply(38 default.realm [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 ==== 157+0+0 (crc 0 0 0) 0x7feca002f080 con 0x5565578505d0 2023-10-05T19:28:30.644 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:30.638+0000 7feca74d6e40 20 rados_obj.operate() r=-2 bl.length=0 2023-10-05T19:28:30.644 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:30.638+0000 7feca74d6e40 20 rados->read ofs=0 len=0 2023-10-05T19:28:30.644 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:30.638+0000 7feca74d6e40 1 -- 172.21.15.167:0/1420689918 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:39 5.f 5:f4c53578:::zonegroups_names.default:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x55655787fb30 con 0x5565578505d0 2023-10-05T19:28:30.644 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:30.638+0000 7feca580c700 1 -- 172.21.15.167:0/1420689918 <== osd.1 v2:172.21.15.167:6800/30035 15 ==== osd_op_reply(39 zonegroups_names.default [read 0~46 out=46b] v0'0 uv2 ondisk = 0) v8 ==== 168+0+46 (crc 0 0 0) 0x7feca002f080 con 0x5565578505d0 2023-10-05T19:28:30.644 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:30.638+0000 7feca74d6e40 20 rados_obj.operate() r=0 bl.length=46 2023-10-05T19:28:30.644 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:30.638+0000 7feca74d6e40 20 rados->read ofs=0 len=0 2023-10-05T19:28:30.644 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:30.638+0000 7feca74d6e40 1 -- 172.21.15.167:0/1420689918 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:40 5.c 5:30d299a5:::zonegroup_info.4f1f719d-9648-417b-b10c-e5d939825419:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x556557880870 con 0x7fec90007030 2023-10-05T19:28:30.646 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:30.638+0000 7feca580c700 1 -- 172.21.15.167:0/1420689918 <== osd.0 v2:172.21.15.118:6800/27690 10 ==== osd_op_reply(40 zonegroup_info.4f1f719d-9648-417b-b10c-e5d939825419 [read 0~362 out=362b] v0'0 uv1 ondisk = 0) v8 ==== 195+0+362 (crc 0 0 0) 0x7feca002f080 con 0x7fec90007030 2023-10-05T19:28:30.646 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:30.638+0000 7feca74d6e40 20 rados_obj.operate() r=0 bl.length=362 2023-10-05T19:28:30.646 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:30.638+0000 7feca74d6e40 20 rados->read ofs=0 len=0 2023-10-05T19:28:30.646 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:30.638+0000 7feca74d6e40 1 -- 172.21.15.167:0/1420689918 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:41 5.13 5:cbbf6c60:::converted:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x556557883eb0 con 0x5565578811e0 2023-10-05T19:28:30.646 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:30.638+0000 7feca500b700 1 -- 172.21.15.167:0/1420689918 <== osd.3 v2:172.21.15.167:6808/30036 10 ==== osd_op_reply(41 converted [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 ==== 153+0+0 (crc 0 0 0) 0x7fec9803d040 con 0x5565578811e0 2023-10-05T19:28:30.646 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:30.638+0000 7feca74d6e40 20 rados_obj.operate() r=-2 bl.length=0 2023-10-05T19:28:30.646 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:30.638+0000 7feca74d6e40 20 rados->read ofs=0 len=0 2023-10-05T19:28:30.646 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:30.638+0000 7feca74d6e40 1 -- 172.21.15.167:0/1420689918 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:42 5.12 5:49953fa1:::default.realm:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x556557884260 con 0x5565578505d0 2023-10-05T19:28:30.647 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:30.638+0000 7feca580c700 1 -- 172.21.15.167:0/1420689918 <== osd.1 v2:172.21.15.167:6800/30035 16 ==== osd_op_reply(42 default.realm [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 ==== 157+0+0 (crc 0 0 0) 0x7feca002f080 con 0x5565578505d0 2023-10-05T19:28:30.647 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:30.638+0000 7feca74d6e40 20 rados_obj.operate() r=-2 bl.length=0 2023-10-05T19:28:30.647 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:30.638+0000 7feca74d6e40 20 rados->read ofs=0 len=0 2023-10-05T19:28:30.647 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:30.638+0000 7feca74d6e40 1 -- 172.21.15.167:0/1420689918 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:43 5.f 5:f4c53578:::zonegroups_names.default:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x556557884260 con 0x5565578505d0 2023-10-05T19:28:30.647 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:30.642+0000 7feca580c700 1 -- 172.21.15.167:0/1420689918 <== osd.1 v2:172.21.15.167:6800/30035 17 ==== osd_op_reply(43 zonegroups_names.default [read 0~46 out=46b] v0'0 uv2 ondisk = 0) v8 ==== 168+0+46 (crc 0 0 0) 0x7feca002f080 con 0x5565578505d0 2023-10-05T19:28:30.647 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:30.642+0000 7feca74d6e40 20 rados_obj.operate() r=0 bl.length=46 2023-10-05T19:28:30.648 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:30.642+0000 7feca74d6e40 1 -- 172.21.15.167:0/1420689918 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:44 5.0 5:00000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x556557885270 con 0x5565578811e0 2023-10-05T19:28:30.649 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:30.642+0000 7feca500b700 1 -- 172.21.15.167:0/1420689918 <== osd.3 v2:172.21.15.167:6808/30036 11 ==== osd_op_reply(44 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fec9803d040 con 0x5565578811e0 2023-10-05T19:28:30.649 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:30.642+0000 7feca500b700 1 -- 172.21.15.167:0/1420689918 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:45 5.10 5:08000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x556557885270 con 0x7fec980123c0 2023-10-05T19:28:30.650 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:30.642+0000 7feca480a700 1 -- 172.21.15.167:0/1420689918 <== osd.2 v2:172.21.15.167:6816/30037 7 ==== osd_op_reply(45 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fec90029040 con 0x7fec980123c0 2023-10-05T19:28:30.650 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:30.642+0000 7feca480a700 1 -- 172.21.15.167:0/1420689918 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:46 5.8 5:10000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x556557885270 con 0x7fec90007030 2023-10-05T19:28:30.651 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:30.646+0000 7feca580c700 1 -- 172.21.15.167:0/1420689918 <== osd.0 v2:172.21.15.118:6800/27690 11 ==== osd_op_reply(46 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7feca002f080 con 0x7fec90007030 2023-10-05T19:28:30.651 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:30.646+0000 7feca580c700 1 -- 172.21.15.167:0/1420689918 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:47 5.18 5:18000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x55655787fb30 con 0x5565578505d0 2023-10-05T19:28:30.652 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:30.646+0000 7feca580c700 1 -- 172.21.15.167:0/1420689918 <== osd.1 v2:172.21.15.167:6800/30035 18 ==== osd_op_reply(47 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7feca002f080 con 0x5565578505d0 2023-10-05T19:28:30.652 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:30.646+0000 7feca580c700 1 -- 172.21.15.167:0/1420689918 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:48 5.4 5:20000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x55655787fb30 con 0x5565578505d0 2023-10-05T19:28:30.652 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:30.646+0000 7feca580c700 1 -- 172.21.15.167:0/1420689918 <== osd.1 v2:172.21.15.167:6800/30035 19 ==== osd_op_reply(48 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7feca002f080 con 0x5565578505d0 2023-10-05T19:28:30.652 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:30.646+0000 7feca580c700 1 -- 172.21.15.167:0/1420689918 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:49 5.14 5:28000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x55655787fb30 con 0x7fec90007030 2023-10-05T19:28:30.653 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:30.646+0000 7feca580c700 1 -- 172.21.15.167:0/1420689918 <== osd.0 v2:172.21.15.118:6800/27690 12 ==== osd_op_reply(49 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7feca002f080 con 0x7fec90007030 2023-10-05T19:28:30.653 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:30.646+0000 7feca580c700 1 -- 172.21.15.167:0/1420689918 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:50 5.c 5:30000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x55655787fb30 con 0x7fec90007030 2023-10-05T19:28:30.654 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:30.650+0000 7feca580c700 1 -- 172.21.15.167:0/1420689918 <== osd.0 v2:172.21.15.118:6800/27690 13 ==== osd_op_reply(50 [pgnls start_epoch 30 out=112b] v23'1 uv1 ondisk = 1) v8 ==== 144+0+112 (crc 0 0 0) 0x7feca002f080 con 0x7fec90007030 2023-10-05T19:28:30.654 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:30.650+0000 7feca580c700 1 -- 172.21.15.167:0/1420689918 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:51 5.1c 5:38000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x55655787fb30 con 0x7fec980123c0 2023-10-05T19:28:30.655 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:30.650+0000 7feca480a700 1 -- 172.21.15.167:0/1420689918 <== osd.2 v2:172.21.15.167:6816/30037 8 ==== osd_op_reply(51 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fec90029040 con 0x7fec980123c0 2023-10-05T19:28:30.655 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:30.650+0000 7feca480a700 1 -- 172.21.15.167:0/1420689918 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:52 5.2 5:40000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x55655787fb30 con 0x5565578811e0 2023-10-05T19:28:30.656 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:30.650+0000 7feca500b700 1 -- 172.21.15.167:0/1420689918 <== osd.3 v2:172.21.15.167:6808/30036 12 ==== osd_op_reply(52 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fec9803d040 con 0x5565578811e0 2023-10-05T19:28:30.656 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:30.650+0000 7feca500b700 1 -- 172.21.15.167:0/1420689918 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:53 5.12 5:48000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x55655787fb30 con 0x5565578505d0 2023-10-05T19:28:30.657 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:30.650+0000 7feca580c700 1 -- 172.21.15.167:0/1420689918 <== osd.1 v2:172.21.15.167:6800/30035 20 ==== osd_op_reply(53 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7feca002f080 con 0x5565578505d0 2023-10-05T19:28:30.657 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:30.650+0000 7feca580c700 1 -- 172.21.15.167:0/1420689918 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:54 5.a 5:50000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x55655787fb30 con 0x7fec90007030 2023-10-05T19:28:30.658 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:30.654+0000 7feca580c700 1 -- 172.21.15.167:0/1420689918 <== osd.0 v2:172.21.15.118:6800/27690 14 ==== osd_op_reply(54 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7feca002f080 con 0x7fec90007030 2023-10-05T19:28:30.658 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:30.654+0000 7feca580c700 1 -- 172.21.15.167:0/1420689918 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:55 5.1a 5:58000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x55655787fb30 con 0x5565578505d0 2023-10-05T19:28:30.659 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:30.654+0000 7feca580c700 1 -- 172.21.15.167:0/1420689918 <== osd.1 v2:172.21.15.167:6800/30035 21 ==== osd_op_reply(55 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7feca002f080 con 0x5565578505d0 2023-10-05T19:28:30.659 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:30.654+0000 7feca580c700 1 -- 172.21.15.167:0/1420689918 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:56 5.6 5:60000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x55655787fb30 con 0x7fec980123c0 2023-10-05T19:28:30.660 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:30.654+0000 7feca480a700 1 -- 172.21.15.167:0/1420689918 <== osd.2 v2:172.21.15.167:6816/30037 9 ==== osd_op_reply(56 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fec90029040 con 0x7fec980123c0 2023-10-05T19:28:30.660 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:30.654+0000 7feca480a700 1 -- 172.21.15.167:0/1420689918 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:57 5.16 5:68000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x55655787fb30 con 0x5565578811e0 2023-10-05T19:28:30.660 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:30.654+0000 7feca500b700 1 -- 172.21.15.167:0/1420689918 <== osd.3 v2:172.21.15.167:6808/30036 13 ==== osd_op_reply(57 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fec9803d040 con 0x5565578811e0 2023-10-05T19:28:30.661 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:30.654+0000 7feca500b700 1 -- 172.21.15.167:0/1420689918 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:58 5.e 5:70000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x55655787fb30 con 0x5565578811e0 2023-10-05T19:28:30.661 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:30.654+0000 7feca500b700 1 -- 172.21.15.167:0/1420689918 <== osd.3 v2:172.21.15.167:6808/30036 14 ==== osd_op_reply(58 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fec9803d040 con 0x5565578811e0 2023-10-05T19:28:30.661 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:30.654+0000 7feca500b700 1 -- 172.21.15.167:0/1420689918 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:59 5.1e 5:78000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x55655787fb30 con 0x7fec90007030 2023-10-05T19:28:30.662 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:30.658+0000 7feca580c700 1 -- 172.21.15.167:0/1420689918 <== osd.0 v2:172.21.15.118:6800/27690 15 ==== osd_op_reply(59 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7feca002f080 con 0x7fec90007030 2023-10-05T19:28:30.662 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:30.658+0000 7feca580c700 1 -- 172.21.15.167:0/1420689918 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:60 5.1 5:80000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x55655787fb30 con 0x5565578811e0 2023-10-05T19:28:30.663 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:30.658+0000 7feca500b700 1 -- 172.21.15.167:0/1420689918 <== osd.3 v2:172.21.15.167:6808/30036 15 ==== osd_op_reply(60 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fec9803d040 con 0x5565578811e0 2023-10-05T19:28:30.663 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:30.658+0000 7feca500b700 1 -- 172.21.15.167:0/1420689918 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:61 5.11 5:88000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x55655787fb30 con 0x5565578811e0 2023-10-05T19:28:30.664 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:30.658+0000 7feca500b700 1 -- 172.21.15.167:0/1420689918 <== osd.3 v2:172.21.15.167:6808/30036 16 ==== osd_op_reply(61 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fec9803d040 con 0x5565578811e0 2023-10-05T19:28:30.664 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:30.658+0000 7feca500b700 1 -- 172.21.15.167:0/1420689918 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:62 5.9 5:90000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x55655787fb30 con 0x5565578505d0 2023-10-05T19:28:30.665 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:30.658+0000 7feca580c700 1 -- 172.21.15.167:0/1420689918 <== osd.1 v2:172.21.15.167:6800/30035 22 ==== osd_op_reply(62 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7feca002f080 con 0x5565578505d0 2023-10-05T19:28:30.665 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:30.658+0000 7feca580c700 1 -- 172.21.15.167:0/1420689918 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:63 5.19 5:98000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x55655787fb30 con 0x5565578505d0 2023-10-05T19:28:30.666 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:30.658+0000 7feca580c700 1 -- 172.21.15.167:0/1420689918 <== osd.1 v2:172.21.15.167:6800/30035 23 ==== osd_op_reply(63 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7feca002f080 con 0x5565578505d0 2023-10-05T19:28:30.666 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:30.658+0000 7feca580c700 1 -- 172.21.15.167:0/1420689918 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:64 5.5 5:a0000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x55655787fb30 con 0x7fec90007030 2023-10-05T19:28:30.667 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:30.662+0000 7feca580c700 1 -- 172.21.15.167:0/1420689918 <== osd.0 v2:172.21.15.118:6800/27690 16 ==== osd_op_reply(64 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7feca002f080 con 0x7fec90007030 2023-10-05T19:28:30.667 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:30.662+0000 7feca580c700 1 -- 172.21.15.167:0/1420689918 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:65 5.15 5:a8000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x55655787fb30 con 0x5565578811e0 2023-10-05T19:28:30.668 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:30.662+0000 7feca500b700 1 -- 172.21.15.167:0/1420689918 <== osd.3 v2:172.21.15.167:6808/30036 17 ==== osd_op_reply(65 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fec9803d040 con 0x5565578811e0 2023-10-05T19:28:30.668 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:30.662+0000 7feca500b700 1 -- 172.21.15.167:0/1420689918 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:66 5.d 5:b0000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x55655787fb30 con 0x7fec980123c0 2023-10-05T19:28:30.668 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:30.662+0000 7feca480a700 1 -- 172.21.15.167:0/1420689918 <== osd.2 v2:172.21.15.167:6816/30037 10 ==== osd_op_reply(66 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fec90029040 con 0x7fec980123c0 2023-10-05T19:28:30.668 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:30.662+0000 7feca480a700 1 -- 172.21.15.167:0/1420689918 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:67 5.1d 5:b8000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x55655787fb30 con 0x5565578505d0 2023-10-05T19:28:30.669 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:30.662+0000 7feca580c700 1 -- 172.21.15.167:0/1420689918 <== osd.1 v2:172.21.15.167:6800/30035 24 ==== osd_op_reply(67 [pgnls start_epoch 30 out=107b] v30'2 uv2 ondisk = 1) v8 ==== 144+0+107 (crc 0 0 0) 0x7feca002f080 con 0x5565578505d0 2023-10-05T19:28:30.669 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:30.662+0000 7feca580c700 1 -- 172.21.15.167:0/1420689918 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:68 5.3 5:c0000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x55655787fb30 con 0x7fec90007030 2023-10-05T19:28:30.670 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:30.666+0000 7feca580c700 1 -- 172.21.15.167:0/1420689918 <== osd.0 v2:172.21.15.118:6800/27690 17 ==== osd_op_reply(68 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7feca002f080 con 0x7fec90007030 2023-10-05T19:28:30.670 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:30.666+0000 7feca580c700 1 -- 172.21.15.167:0/1420689918 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:69 5.13 5:c8000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x55655787fb30 con 0x5565578811e0 2023-10-05T19:28:30.671 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:30.666+0000 7feca500b700 1 -- 172.21.15.167:0/1420689918 <== osd.3 v2:172.21.15.167:6808/30036 18 ==== osd_op_reply(69 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fec9803d040 con 0x5565578811e0 2023-10-05T19:28:30.671 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:30.666+0000 7feca500b700 1 -- 172.21.15.167:0/1420689918 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:70 5.b 5:d0000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x55655787fb30 con 0x7fec90007030 2023-10-05T19:28:30.672 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:30.666+0000 7feca580c700 1 -- 172.21.15.167:0/1420689918 <== osd.0 v2:172.21.15.118:6800/27690 18 ==== osd_op_reply(70 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7feca002f080 con 0x7fec90007030 2023-10-05T19:28:30.672 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:30.666+0000 7feca580c700 1 -- 172.21.15.167:0/1420689918 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:71 5.1b 5:d8000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x55655787fb30 con 0x7fec980123c0 2023-10-05T19:28:30.673 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:30.666+0000 7feca480a700 1 -- 172.21.15.167:0/1420689918 <== osd.2 v2:172.21.15.167:6816/30037 11 ==== osd_op_reply(71 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fec90029040 con 0x7fec980123c0 2023-10-05T19:28:30.673 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:30.666+0000 7feca480a700 1 -- 172.21.15.167:0/1420689918 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:72 5.7 5:e0000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x55655787fb30 con 0x5565578811e0 2023-10-05T19:28:30.674 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:30.670+0000 7feca500b700 1 -- 172.21.15.167:0/1420689918 <== osd.3 v2:172.21.15.167:6808/30036 19 ==== osd_op_reply(72 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fec9803d040 con 0x5565578811e0 2023-10-05T19:28:30.674 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:30.670+0000 7feca500b700 1 -- 172.21.15.167:0/1420689918 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:73 5.17 5:e8000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x55655787fb30 con 0x7fec90007030 2023-10-05T19:28:30.675 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:30.670+0000 7feca580c700 1 -- 172.21.15.167:0/1420689918 <== osd.0 v2:172.21.15.118:6800/27690 19 ==== osd_op_reply(73 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7feca002f080 con 0x7fec90007030 2023-10-05T19:28:30.675 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:30.670+0000 7feca580c700 1 -- 172.21.15.167:0/1420689918 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:74 5.f 5:f0000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x55655787fb30 con 0x5565578505d0 2023-10-05T19:28:30.676 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:30.670+0000 7feca580c700 1 -- 172.21.15.167:0/1420689918 <== osd.1 v2:172.21.15.167:6800/30035 25 ==== osd_op_reply(74 [pgnls start_epoch 30 out=115b] v23'2 uv2 ondisk = 1) v8 ==== 144+0+115 (crc 0 0 0) 0x7feca002f080 con 0x5565578505d0 2023-10-05T19:28:30.676 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:30.670+0000 7feca580c700 1 -- 172.21.15.167:0/1420689918 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:75 5.1f 5:f8000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x55655787fb30 con 0x7fec980123c0 2023-10-05T19:28:30.677 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:30.670+0000 7feca480a700 1 -- 172.21.15.167:0/1420689918 <== osd.2 v2:172.21.15.167:6816/30037 12 ==== osd_op_reply(75 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fec90029040 con 0x7fec980123c0 2023-10-05T19:28:30.677 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:30.670+0000 7feca74d6e40 20 RGWRados::pool_iterate: got zonegroup_info.4f1f719d-9648-417b-b10c-e5d939825419 2023-10-05T19:28:30.677 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:30.670+0000 7feca74d6e40 20 RGWRados::pool_iterate: got zone_info.9a9df965-6612-45d4-a45f-6e3c889bacc2 2023-10-05T19:28:30.677 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:30.670+0000 7feca74d6e40 20 RGWRados::pool_iterate: got zone_names.default 2023-10-05T19:28:30.678 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:30.670+0000 7feca74d6e40 20 RGWRados::pool_iterate: got zonegroups_names.default 2023-10-05T19:28:30.678 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:30.670+0000 7feca74d6e40 20 rados->read ofs=0 len=0 2023-10-05T19:28:30.678 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:30.670+0000 7feca74d6e40 1 -- 172.21.15.167:0/1420689918 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:76 5.f 5:f43b6ece:::zone_names.default:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x556557884260 con 0x5565578505d0 2023-10-05T19:28:30.679 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:30.670+0000 7feca580c700 1 -- 172.21.15.167:0/1420689918 <== osd.1 v2:172.21.15.167:6800/30035 26 ==== osd_op_reply(76 zone_names.default [read 0~46 out=46b] v0'0 uv1 ondisk = 0) v8 ==== 162+0+46 (crc 0 0 0) 0x7feca002f080 con 0x5565578505d0 2023-10-05T19:28:30.679 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:30.670+0000 7feca74d6e40 20 rados_obj.operate() r=0 bl.length=46 2023-10-05T19:28:30.679 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:30.670+0000 7feca74d6e40 20 rados->read ofs=0 len=0 2023-10-05T19:28:30.679 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:30.670+0000 7feca74d6e40 1 -- 172.21.15.167:0/1420689918 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:77 5.1d 5:b8374eb4:::zone_info.9a9df965-6612-45d4-a45f-6e3c889bacc2:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x556557884610 con 0x5565578505d0 2023-10-05T19:28:30.679 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:30.674+0000 7feca580c700 1 -- 172.21.15.167:0/1420689918 <== osd.1 v2:172.21.15.167:6800/30035 27 ==== osd_op_reply(77 zone_info.9a9df965-6612-45d4-a45f-6e3c889bacc2 [read 0~874 out=874b] v0'0 uv2 ondisk = 0) v8 ==== 190+0+874 (crc 0 0 0) 0x7feca002f080 con 0x5565578505d0 2023-10-05T19:28:30.679 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:30.674+0000 7feca74d6e40 20 rados_obj.operate() r=0 bl.length=874 2023-10-05T19:28:30.679 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:30.674+0000 7feca74d6e40 20 rados->read ofs=0 len=0 2023-10-05T19:28:30.679 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:30.674+0000 7feca74d6e40 1 -- 172.21.15.167:0/1420689918 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:78 5.f 5:f4c53578:::zonegroups_names.default:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x556557884b00 con 0x5565578505d0 2023-10-05T19:28:30.680 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:30.674+0000 7feca580c700 1 -- 172.21.15.167:0/1420689918 <== osd.1 v2:172.21.15.167:6800/30035 28 ==== osd_op_reply(78 zonegroups_names.default [read 0~46 out=46b] v0'0 uv2 ondisk = 0) v8 ==== 168+0+46 (crc 0 0 0) 0x7feca002f080 con 0x5565578505d0 2023-10-05T19:28:30.680 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:30.674+0000 7feca74d6e40 20 rados_obj.operate() r=0 bl.length=46 2023-10-05T19:28:30.681 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:30.674+0000 7feca74d6e40 20 rados->read ofs=0 len=0 2023-10-05T19:28:30.681 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:30.674+0000 7feca74d6e40 1 -- 172.21.15.167:0/1420689918 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:79 5.c 5:30d299a5:::zonegroup_info.4f1f719d-9648-417b-b10c-e5d939825419:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x556557884b00 con 0x7fec90007030 2023-10-05T19:28:30.681 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:30.674+0000 7feca580c700 1 -- 172.21.15.167:0/1420689918 <== osd.0 v2:172.21.15.118:6800/27690 20 ==== osd_op_reply(79 zonegroup_info.4f1f719d-9648-417b-b10c-e5d939825419 [read 0~362 out=362b] v0'0 uv1 ondisk = 0) v8 ==== 195+0+362 (crc 0 0 0) 0x7feca002f080 con 0x7fec90007030 2023-10-05T19:28:30.681 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:30.674+0000 7feca74d6e40 20 rados_obj.operate() r=0 bl.length=362 2023-10-05T19:28:30.681 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:30.674+0000 7feca74d6e40 20 rados->read ofs=0 len=0 2023-10-05T19:28:30.681 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:30.674+0000 7feca74d6e40 1 -- 172.21.15.167:0/1420689918 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:80 5.3 5:c0ad8cbd:::region_map:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x556557884b60 con 0x7fec90007030 2023-10-05T19:28:30.682 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:30.674+0000 7feca580c700 1 -- 172.21.15.167:0/1420689918 <== osd.0 v2:172.21.15.118:6800/27690 21 ==== osd_op_reply(80 region_map [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 ==== 154+0+0 (crc 0 0 0) 0x7feca0050040 con 0x7fec90007030 2023-10-05T19:28:30.682 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:30.674+0000 7feca74d6e40 20 rados_obj.operate() r=-2 bl.length=0 2023-10-05T19:28:30.682 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:30.674+0000 7feca74d6e40 20 zone default found 2023-10-05T19:28:30.682 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:30.674+0000 7feca74d6e40 4 Realm: () 2023-10-05T19:28:30.682 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:30.674+0000 7feca74d6e40 4 ZoneGroup: default (4f1f719d-9648-417b-b10c-e5d939825419) 2023-10-05T19:28:30.682 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:30.674+0000 7feca74d6e40 4 Zone: default (9a9df965-6612-45d4-a45f-6e3c889bacc2) 2023-10-05T19:28:30.682 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:30.674+0000 7feca74d6e40 10 cannot find current period zonegroup using local zonegroup configuration 2023-10-05T19:28:30.682 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:30.674+0000 7feca74d6e40 20 zonegroup default 2023-10-05T19:28:30.682 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:30.674+0000 7feca74d6e40 20 rados->read ofs=0 len=0 2023-10-05T19:28:30.683 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:30.674+0000 7feca74d6e40 1 -- 172.21.15.167:0/1420689918 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:81 5.3 5:c52100b6:::period_config.default:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x556557884b60 con 0x7fec90007030 2023-10-05T19:28:30.683 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:30.678+0000 7feca580c700 1 -- 172.21.15.167:0/1420689918 <== osd.0 v2:172.21.15.118:6800/27690 22 ==== osd_op_reply(81 period_config.default [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 ==== 165+0+0 (crc 0 0 0) 0x7feca0050040 con 0x7fec90007030 2023-10-05T19:28:30.684 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:30.678+0000 7feca74d6e40 20 rados_obj.operate() r=-2 bl.length=0 2023-10-05T19:28:30.684 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:30.678+0000 7feca74d6e40 20 rados->read ofs=0 len=0 2023-10-05T19:28:30.684 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:30.678+0000 7feca74d6e40 1 -- 172.21.15.167:0/1420689918 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:82 6.2 6:4347d321:::bucket.sync-source-hints.:head [call version.read in=11b,read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x5565578864e0 con 0x5565578811e0 2023-10-05T19:28:30.684 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:30.678+0000 7feca500b700 1 -- 172.21.15.167:0/1420689918 <== osd.3 v2:172.21.15.167:6808/30036 20 ==== osd_op_reply(82 bucket.sync-source-hints. [call,read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 ==== 211+0+0 (crc 0 0 0) 0x7fec9803d040 con 0x5565578811e0 2023-10-05T19:28:30.684 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:30.678+0000 7feca74d6e40 20 rados_obj.operate() r=-2 bl.length=0 2023-10-05T19:28:30.684 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:30.678+0000 7feca74d6e40 20 rados->read ofs=0 len=0 2023-10-05T19:28:30.684 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:30.678+0000 7feca74d6e40 1 -- 172.21.15.167:0/1420689918 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:83 6.b 6:d467b91b:::bucket.sync-target-hints.:head [call version.read in=11b,read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x556557886890 con 0x5565578811e0 2023-10-05T19:28:30.685 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:30.678+0000 7feca500b700 1 -- 172.21.15.167:0/1420689918 <== osd.3 v2:172.21.15.167:6808/30036 21 ==== osd_op_reply(83 bucket.sync-target-hints. [call,read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 ==== 211+0+0 (crc 0 0 0) 0x7fec9803d040 con 0x5565578811e0 2023-10-05T19:28:30.685 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:30.678+0000 7feca74d6e40 20 rados_obj.operate() r=-2 bl.length=0 2023-10-05T19:28:30.685 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:30.678+0000 7feca74d6e40 20 started sync module instance, tier type = 2023-10-05T19:28:30.685 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:30.678+0000 7feca74d6e40 20 started zone id=9a9df965-6612-45d4-a45f-6e3c889bacc2 (name=default) with tier type = 2023-10-05T19:28:30.685 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:30.678+0000 7feca74d6e40 1 -- 172.21.15.167:0/1420689918 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:84 7.1f 7:f95f44c2:::notify.0:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x556557885ba0 con 0x7fec980123c0 2023-10-05T19:28:30.687 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:30.682+0000 7feca480a700 1 -- 172.21.15.167:0/1420689918 <== osd.2 v2:172.21.15.167:6816/30037 13 ==== osd_op_reply(84 notify.0 [create] v30'90 uv90 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7fec90029040 con 0x7fec980123c0 2023-10-05T19:28:30.687 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:30.682+0000 7feca74d6e40 1 -- 172.21.15.167:0/1420689918 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:85 7.1f 7:f95f44c2:::notify.0:head [watch watch cookie 93893748613024] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x5565578857d0 con 0x7fec980123c0 2023-10-05T19:28:30.687 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:30.682+0000 7feca74d6e40 1 -- 172.21.15.167:0/1420689918 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:86 7.0 7:05bf5b68:::notify.1:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x556557887620 con 0x5565578505d0 2023-10-05T19:28:30.690 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:30.686+0000 7feca580c700 1 -- 172.21.15.167:0/1420689918 <== osd.1 v2:172.21.15.167:6800/30035 29 ==== osd_op_reply(86 notify.1 [create] v30'90 uv90 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7feca0050040 con 0x5565578505d0 2023-10-05T19:28:30.690 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:30.686+0000 7feca74d6e40 1 -- 172.21.15.167:0/1420689918 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:87 7.0 7:05bf5b68:::notify.1:head [watch watch cookie 93893748617808] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x556557887f10 con 0x5565578505d0 2023-10-05T19:28:30.690 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:30.686+0000 7feca74d6e40 1 -- 172.21.15.167:0/1420689918 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:88 7.15 7:a93a5511:::notify.2:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x5565578887b0 con 0x5565578811e0 2023-10-05T19:28:30.690 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:30.686+0000 7feca480a700 1 -- 172.21.15.167:0/1420689918 <== osd.2 v2:172.21.15.167:6816/30037 14 ==== osd_op_reply(85 notify.0 [watch watch cookie 93893748613024] v30'91 uv90 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7fec90029040 con 0x7fec980123c0 2023-10-05T19:28:30.693 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:30.686+0000 7feca580c700 1 -- 172.21.15.167:0/1420689918 <== osd.1 v2:172.21.15.167:6800/30035 30 ==== osd_op_reply(87 notify.1 [watch watch cookie 93893748617808] v30'91 uv90 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7feca0050040 con 0x5565578505d0 2023-10-05T19:28:30.694 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:30.686+0000 7feca500b700 1 -- 172.21.15.167:0/1420689918 <== osd.3 v2:172.21.15.167:6808/30036 22 ==== osd_op_reply(88 notify.2 [create] v30'90 uv90 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7fec9803d040 con 0x5565578811e0 2023-10-05T19:28:30.694 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:30.686+0000 7feca74d6e40 1 -- 172.21.15.167:0/1420689918 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:89 7.15 7:a93a5511:::notify.2:head [watch watch cookie 93893748619808] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x556557886d60 con 0x5565578811e0 2023-10-05T19:28:30.694 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:30.690+0000 7feca74d6e40 1 -- 172.21.15.167:0/1420689918 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:90 7.e 7:7759931f:::notify.3:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x556557889050 con 0x7fec980123c0 2023-10-05T19:28:30.696 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:30.690+0000 7feca480a700 1 -- 172.21.15.167:0/1420689918 <== osd.2 v2:172.21.15.167:6816/30037 15 ==== osd_op_reply(90 notify.3 [create] v30'90 uv90 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7fec90029040 con 0x7fec980123c0 2023-10-05T19:28:30.696 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:30.690+0000 7feca74d6e40 1 -- 172.21.15.167:0/1420689918 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:91 7.e 7:7759931f:::notify.3:head [watch watch cookie 93893748624224] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x556557889970 con 0x7fec980123c0 2023-10-05T19:28:30.697 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:30.690+0000 7feca74d6e40 1 -- 172.21.15.167:0/1420689918 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:92 7.d 7:b4812045:::notify.4:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x55655788a4d0 con 0x5565578505d0 2023-10-05T19:28:30.697 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:30.690+0000 7feca500b700 1 -- 172.21.15.167:0/1420689918 <== osd.3 v2:172.21.15.167:6808/30036 23 ==== osd_op_reply(89 notify.2 [watch watch cookie 93893748619808] v30'91 uv90 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7fec9803d040 con 0x5565578811e0 2023-10-05T19:28:30.699 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:30.694+0000 7feca480a700 1 -- 172.21.15.167:0/1420689918 <== osd.2 v2:172.21.15.167:6816/30037 16 ==== osd_op_reply(91 notify.3 [watch watch cookie 93893748624224] v30'91 uv90 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7fec90029040 con 0x7fec980123c0 2023-10-05T19:28:30.699 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:30.694+0000 7feca580c700 1 -- 172.21.15.167:0/1420689918 <== osd.1 v2:172.21.15.167:6800/30035 31 ==== osd_op_reply(92 notify.4 [create] v30'90 uv90 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7feca0050040 con 0x5565578505d0 2023-10-05T19:28:30.700 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:30.694+0000 7feca74d6e40 1 -- 172.21.15.167:0/1420689918 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:93 7.d 7:b4812045:::notify.4:head [watch watch cookie 93893748620912] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x55655788a490 con 0x5565578505d0 2023-10-05T19:28:30.700 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:30.694+0000 7feca74d6e40 1 -- 172.21.15.167:0/1420689918 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:94 7.3 7:c609908c:::notify.5:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x55655788ad30 con 0x7fec90007030 2023-10-05T19:28:30.702 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:30.698+0000 7feca580c700 1 -- 172.21.15.167:0/1420689918 <== osd.1 v2:172.21.15.167:6800/30035 32 ==== osd_op_reply(93 notify.4 [watch watch cookie 93893748620912] v30'91 uv90 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7feca0050040 con 0x5565578505d0 2023-10-05T19:28:30.702 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:30.698+0000 7feca580c700 1 -- 172.21.15.167:0/1420689918 <== osd.0 v2:172.21.15.118:6800/27690 23 ==== osd_op_reply(94 notify.5 [create] v30'90 uv90 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7feca0050040 con 0x7fec90007030 2023-10-05T19:28:30.703 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:30.698+0000 7feca74d6e40 1 -- 172.21.15.167:0/1420689918 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:95 7.3 7:c609908c:::notify.5:head [watch watch cookie 93893748629968] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x55655788ab10 con 0x7fec90007030 2023-10-05T19:28:30.703 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:30.698+0000 7feca74d6e40 1 -- 172.21.15.167:0/1420689918 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:96 7.14 7:2b04a3e9:::notify.6:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x55655788b790 con 0x7fec90007030 2023-10-05T19:28:30.705 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:30.698+0000 7feca580c700 1 -- 172.21.15.167:0/1420689918 <== osd.0 v2:172.21.15.118:6800/27690 24 ==== osd_op_reply(96 notify.6 [create] v30'90 uv90 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7feca0050040 con 0x7fec90007030 2023-10-05T19:28:30.705 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:30.698+0000 7feca74d6e40 1 -- 172.21.15.167:0/1420689918 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:97 7.14 7:2b04a3e9:::notify.6:head [watch watch cookie 93893748634272] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x55655788bbe0 con 0x7fec90007030 2023-10-05T19:28:30.706 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:30.698+0000 7feca580c700 1 -- 172.21.15.167:0/1420689918 <== osd.0 v2:172.21.15.118:6800/27690 25 ==== osd_op_reply(95 notify.5 [watch watch cookie 93893748629968] v30'91 uv90 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7feca0050040 con 0x7fec90007030 2023-10-05T19:28:30.706 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:30.698+0000 7feca74d6e40 1 -- 172.21.15.167:0/1420689918 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:98 7.9 7:93e5b521:::notify.7:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x55655788c480 con 0x7fec90007030 2023-10-05T19:28:30.707 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:30.702+0000 7feca580c700 1 -- 172.21.15.167:0/1420689918 <== osd.0 v2:172.21.15.118:6800/27690 26 ==== osd_op_reply(97 notify.6 [watch watch cookie 93893748634272] v30'91 uv90 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7feca0050040 con 0x7fec90007030 2023-10-05T19:28:30.708 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:30.702+0000 7feca580c700 1 -- 172.21.15.167:0/1420689918 <== osd.0 v2:172.21.15.118:6800/27690 27 ==== osd_op_reply(98 notify.7 [create] v30'90 uv90 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7feca0050040 con 0x7fec90007030 2023-10-05T19:28:30.708 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:30.702+0000 7feca74d6e40 1 -- 172.21.15.167:0/1420689918 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:99 7.9 7:93e5b521:::notify.7:head [watch watch cookie 93893748635376] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x55655788c030 con 0x7fec90007030 2023-10-05T19:28:30.708 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:30.702+0000 7feca74d6e40 20 add_watcher() i=0 2023-10-05T19:28:30.708 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:30.702+0000 7feca74d6e40 20 add_watcher() i=1 2023-10-05T19:28:30.709 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:30.702+0000 7feca74d6e40 20 add_watcher() i=2 2023-10-05T19:28:30.709 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:30.702+0000 7feca74d6e40 20 add_watcher() i=3 2023-10-05T19:28:30.709 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:30.702+0000 7feca74d6e40 20 add_watcher() i=4 2023-10-05T19:28:30.709 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:30.702+0000 7feca74d6e40 20 add_watcher() i=5 2023-10-05T19:28:30.709 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:30.702+0000 7feca74d6e40 20 add_watcher() i=6 2023-10-05T19:28:30.710 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:30.706+0000 7feca580c700 1 -- 172.21.15.167:0/1420689918 <== osd.0 v2:172.21.15.118:6800/27690 28 ==== osd_op_reply(99 notify.7 [watch watch cookie 93893748635376] v30'91 uv90 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7feca0050040 con 0x7fec90007030 2023-10-05T19:28:30.710 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:30.706+0000 7feca74d6e40 20 add_watcher() i=7 2023-10-05T19:28:30.710 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:30.706+0000 7feca74d6e40 2 all 8 watchers are set, enabling cache 2023-10-05T19:28:30.710 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:30.706+0000 7feca74d6e40 1 -- 172.21.15.167:0/1420689918 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:100 6.4 6:22d26bf9:::data_loggenerations_metadata:head [call version.check_conds in=74b,call version.read in=11b,read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x55655788f240 con 0x5565578505d0 2023-10-05T19:28:30.711 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:30.706+0000 7feca580c700 1 -- 172.21.15.167:0/1420689918 <== osd.1 v2:172.21.15.167:6800/30035 33 ==== osd_op_reply(100 data_loggenerations_metadata [call,call out=48b,read 0~28 out=28b] v0'0 uv1 ondisk = 0) v8 ==== 256+0+76 (crc 0 0 0) 0x7feca0050040 con 0x5565578505d0 2023-10-05T19:28:30.711 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:30.706+0000 7feca74d6e40 1 -- 172.21.15.167:0/1420689918 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:101 6.4 6:22d26bf9:::data_loggenerations_metadata:head [watch watch cookie 93893748636848] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x55655788c5f0 con 0x5565578505d0 2023-10-05T19:28:30.713 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:30.706+0000 7feca580c700 1 -- 172.21.15.167:0/1420689918 <== osd.1 v2:172.21.15.167:6800/30035 34 ==== osd_op_reply(101 data_loggenerations_metadata [watch watch cookie 93893748636848] v30'94 uv1 ondisk = 0) v8 ==== 172+0+0 (crc 0 0 0) 0x7feca0050040 con 0x5565578505d0 2023-10-05T19:28:30.713 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:30.706+0000 7fec767e4700 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2023-10-05T19:28:30.713 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:30.706+0000 7feca74d6e40 20 check_secure_mon_conn(): auth registy supported: methods=[2] modes=[2,1] 2023-10-05T19:28:30.713 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:30.706+0000 7feca74d6e40 20 check_secure_mon_conn(): mode 1 is insecure 2023-10-05T19:28:30.715 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:30.706+0000 7feca74d6e40 1 Processor -- start 2023-10-05T19:28:30.715 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:30.710+0000 7feca74d6e40 1 -- start start 2023-10-05T19:28:30.715 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:30.710+0000 7feca74d6e40 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5565578996f0 0x556557899ac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-05T19:28:30.715 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:30.710+0000 7feca74d6e40 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x55655789a000 con 0x5565578996f0 2023-10-05T19:28:30.715 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:30.710+0000 7feca480a700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5565578996f0 0x556557899ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-05T19:28:30.715 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:30.710+0000 7feca480a700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5565578996f0 0x556557899ac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.118:3300/0 says I am v2:172.21.15.167:58146/0 (socket says 172.21.15.167:58146) 2023-10-05T19:28:30.715 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:30.710+0000 7feca480a700 1 -- 172.21.15.167:0/1542783811 learned_addr learned my addr 172.21.15.167:0/1542783811 (peer_addr_for_me v2:172.21.15.167:0/0) 2023-10-05T19:28:30.716 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:30.710+0000 7feca480a700 1 -- 172.21.15.167:0/1542783811 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55655789a140 con 0x5565578996f0 2023-10-05T19:28:30.716 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:30.710+0000 7feca480a700 1 --2- 172.21.15.167:0/1542783811 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5565578996f0 0x556557899ac0 secure :-1 s=READY pgs=318 cs=0 l=1 rev1=1 crypto rx=0x55655784d070 tx=0x7fec90030040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=44e49f2ce78b15a0 server_cookie=0 in_seq=0 out_seq=0 2023-10-05T19:28:30.716 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:30.710+0000 7fec75fe3700 1 -- 172.21.15.167:0/1542783811 <== mon.0 v2:172.21.15.118:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fec90025040 con 0x5565578996f0 2023-10-05T19:28:30.716 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:30.710+0000 7fec75fe3700 1 -- 172.21.15.167:0/1542783811 <== mon.0 v2:172.21.15.118:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fec90034050 con 0x5565578996f0 2023-10-05T19:28:30.717 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:30.710+0000 7fec75fe3700 1 -- 172.21.15.167:0/1542783811 <== mon.0 v2:172.21.15.118:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fec9002e560 con 0x5565578996f0 2023-10-05T19:28:30.717 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:30.710+0000 7feca74d6e40 1 -- 172.21.15.167:0/1542783811 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5565578996f0 msgr2=0x556557899ac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-05T19:28:30.718 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:30.710+0000 7feca74d6e40 1 --2- 172.21.15.167:0/1542783811 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5565578996f0 0x556557899ac0 secure :-1 s=READY pgs=318 cs=0 l=1 rev1=1 crypto rx=0x55655784d070 tx=0x7fec90030040 comp rx=0 tx=0).stop 2023-10-05T19:28:30.718 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:30.710+0000 7feca74d6e40 1 -- 172.21.15.167:0/1542783811 shutdown_connections 2023-10-05T19:28:30.718 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:30.710+0000 7feca74d6e40 1 --2- 172.21.15.167:0/1542783811 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5565578996f0 0x556557899ac0 unknown :-1 s=CLOSED pgs=318 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-05T19:28:30.718 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:30.710+0000 7feca74d6e40 1 -- 172.21.15.167:0/1542783811 >> 172.21.15.167:0/1542783811 conn(0x556557894ab0 msgr2=0x556557892c60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-05T19:28:30.718 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:30.710+0000 7feca74d6e40 1 -- 172.21.15.167:0/1542783811 shutdown_connections 2023-10-05T19:28:30.718 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:30.710+0000 7feca74d6e40 1 -- 172.21.15.167:0/1542783811 wait complete. 2023-10-05T19:28:30.722 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:30.714+0000 7feca74d6e40 1 Processor -- start 2023-10-05T19:28:30.722 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:30.714+0000 7feca74d6e40 1 -- start start 2023-10-05T19:28:30.722 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:30.714+0000 7feca74d6e40 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5565578996f0 0x55655789f1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-05T19:28:30.722 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:30.714+0000 7feca74d6e40 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x5565578a01a0 con 0x5565578996f0 2023-10-05T19:28:30.722 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:30.714+0000 7feca480a700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5565578996f0 0x55655789f1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-05T19:28:30.722 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:30.714+0000 7feca480a700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5565578996f0 0x55655789f1a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.118:3300/0 says I am v2:172.21.15.167:58150/0 (socket says 172.21.15.167:58150) 2023-10-05T19:28:30.722 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:30.714+0000 7feca480a700 1 -- 172.21.15.167:0/2215951926 learned_addr learned my addr 172.21.15.167:0/2215951926 (peer_addr_for_me v2:172.21.15.167:0/0) 2023-10-05T19:28:30.722 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:30.714+0000 7feca480a700 1 -- 172.21.15.167:0/2215951926 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55655784cc00 con 0x5565578996f0 2023-10-05T19:28:30.722 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:30.714+0000 7feca480a700 1 --2- 172.21.15.167:0/2215951926 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5565578996f0 0x55655789f1a0 secure :-1 s=READY pgs=319 cs=0 l=1 rev1=1 crypto rx=0x5565578937d0 tx=0x7fec90038000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-05T19:28:30.722 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:30.714+0000 7febe7fff700 1 -- 172.21.15.167:0/2215951926 <== mon.0 v2:172.21.15.118:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fec90025040 con 0x5565578996f0 2023-10-05T19:28:30.723 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:30.714+0000 7febe7fff700 1 -- 172.21.15.167:0/2215951926 <== mon.0 v2:172.21.15.118:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fec900399a0 con 0x5565578996f0 2023-10-05T19:28:30.723 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:30.714+0000 7feca74d6e40 1 -- 172.21.15.167:0/2215951926 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x55655789f7a0 con 0x5565578996f0 2023-10-05T19:28:30.723 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:30.714+0000 7febe7fff700 1 -- 172.21.15.167:0/2215951926 <== mon.0 v2:172.21.15.118:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fec90039b30 con 0x5565578996f0 2023-10-05T19:28:30.723 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:30.714+0000 7feca74d6e40 1 -- 172.21.15.167:0/2215951926 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55655789fc40 con 0x5565578996f0 2023-10-05T19:28:30.723 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:30.718+0000 7febe7fff700 1 -- 172.21.15.167:0/2215951926 <== mon.0 v2:172.21.15.118:3300/0 4 ==== mgrmap(e 5) v1 ==== 40329+0+0 (secure 0 0 0) 0x7fec90012400 con 0x5565578996f0 2023-10-05T19:28:30.724 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:30.718+0000 7febe7fff700 1 --2- 172.21.15.167:0/2215951926 >> [v2:172.21.15.167:6824/29889,v1:172.21.15.167:6825/29889] conn(0x7febec030c50 0x7febec033100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-05T19:28:30.724 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:30.718+0000 7febe7fff700 1 -- 172.21.15.167:0/2215951926 <== mon.0 v2:172.21.15.118:3300/0 5 ==== osd_map(30..30 src has 1..30) v4 ==== 6057+0+0 (secure 0 0 0) 0x7fec9004dc20 con 0x5565578996f0 2023-10-05T19:28:30.724 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:30.718+0000 7feca74d6e40 5 note: GC not initialized 2023-10-05T19:28:30.724 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:30.718+0000 7feca580c700 1 --2- 172.21.15.167:0/2215951926 >> [v2:172.21.15.167:6824/29889,v1:172.21.15.167:6825/29889] conn(0x7febec030c50 0x7febec033100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-05T19:28:30.724 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:30.718+0000 7feca580c700 1 --2- 172.21.15.167:0/2215951926 >> [v2:172.21.15.167:6824/29889,v1:172.21.15.167:6825/29889] conn(0x7febec030c50 0x7febec033100 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x55655788f2f0 tx=0x7feca0000f90 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-05T19:28:30.725 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:30.718+0000 7febe57fa700 20 reqs_thread_entry: start 2023-10-05T19:28:30.725 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:30.718+0000 7febdeffd700 20 reqs_thread_entry: start 2023-10-05T19:28:30.726 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:30.718+0000 7feca74d6e40 20 init_complete bucket index max shards: 11 2023-10-05T19:28:30.726 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:30.722+0000 7feca74d6e40 10 rgw notify: Started notification manager with: 1 workers 2023-10-05T19:28:30.726 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:30.722+0000 7febdd7fa700 20 rgw notify: INFO: next queues processing will happen at: Thu Oct 5 19:29:01 2023 2023-10-05T19:28:30.726 INFO:teuthology.orchestra.run.smithi167.stderr: 2023-10-05T19:28:30.727 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:30.722+0000 7febdcff9700 20 reqs_thread_entry: start 2023-10-05T19:28:30.727 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:30.722+0000 7feca74d6e40 1 -- 172.21.15.167:0/1420689918 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:102 8.0 8:00000000:root:::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x556557907440 con 0x5565578811e0 2023-10-05T19:28:30.728 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:30.722+0000 7feca500b700 1 -- 172.21.15.167:0/1420689918 <== osd.3 v2:172.21.15.167:6808/30036 24 ==== osd_op_reply(102 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fec9803d040 con 0x5565578811e0 2023-10-05T19:28:30.729 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:30.722+0000 7feca500b700 1 -- 172.21.15.167:0/1420689918 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:103 8.10 8:08000000:root:::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x556557907440 con 0x5565578811e0 2023-10-05T19:28:30.729 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:30.722+0000 7feca500b700 1 -- 172.21.15.167:0/1420689918 <== osd.3 v2:172.21.15.167:6808/30036 25 ==== osd_op_reply(103 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fec9803d040 con 0x5565578811e0 2023-10-05T19:28:30.729 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:30.722+0000 7feca500b700 1 -- 172.21.15.167:0/1420689918 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:104 8.8 8:10000000:root:::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x556557907440 con 0x5565578505d0 2023-10-05T19:28:30.730 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:30.726+0000 7feca580c700 1 -- 172.21.15.167:0/1420689918 <== osd.1 v2:172.21.15.167:6800/30035 35 ==== osd_op_reply(104 [pgnls start_epoch 30 out=49b] v30'2 uv2 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7feca0050040 con 0x5565578505d0 2023-10-05T19:28:30.730 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:30.726+0000 7feca580c700 1 -- 172.21.15.167:0/1420689918 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:105 8.18 8:18000000:root:::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x556557884260 con 0x5565578505d0 2023-10-05T19:28:30.731 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:30.726+0000 7feca580c700 1 -- 172.21.15.167:0/1420689918 <== osd.1 v2:172.21.15.167:6800/30035 36 ==== osd_op_reply(105 [pgnls start_epoch 30 out=49b] v30'11 uv11 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7feca0050040 con 0x5565578505d0 2023-10-05T19:28:30.731 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:30.726+0000 7feca580c700 1 -- 172.21.15.167:0/1420689918 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:106 8.4 8:20000000:root:::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x556557884260 con 0x7fec980123c0 2023-10-05T19:28:30.732 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:30.726+0000 7feca480a700 1 -- 172.21.15.167:0/1420689918 <== osd.2 v2:172.21.15.167:6816/30037 17 ==== osd_op_reply(106 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fec9003f020 con 0x7fec980123c0 2023-10-05T19:28:30.732 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:30.726+0000 7feca480a700 1 -- 172.21.15.167:0/1420689918 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:107 8.14 8:28000000:root:::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x7fec90042040 con 0x5565578811e0 2023-10-05T19:28:30.733 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:30.726+0000 7feca500b700 1 -- 172.21.15.167:0/1420689918 <== osd.3 v2:172.21.15.167:6808/30036 26 ==== osd_op_reply(107 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fec9803d040 con 0x5565578811e0 2023-10-05T19:28:30.733 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:30.726+0000 7feca500b700 1 -- 172.21.15.167:0/1420689918 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:108 8.c 8:30000000:root:::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x7fec90042040 con 0x5565578505d0 2023-10-05T19:28:30.733 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:30.726+0000 7feca580c700 1 -- 172.21.15.167:0/1420689918 <== osd.1 v2:172.21.15.167:6800/30035 37 ==== osd_op_reply(108 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7feca0050040 con 0x5565578505d0 2023-10-05T19:28:30.733 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:30.726+0000 7feca580c700 1 -- 172.21.15.167:0/1420689918 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:109 8.1c 8:38000000:root:::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x7fec90042040 con 0x5565578811e0 2023-10-05T19:28:30.734 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:30.730+0000 7feca500b700 1 -- 172.21.15.167:0/1420689918 <== osd.3 v2:172.21.15.167:6808/30036 27 ==== osd_op_reply(109 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fec9803d040 con 0x5565578811e0 2023-10-05T19:28:30.734 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:30.730+0000 7feca500b700 1 -- 172.21.15.167:0/1420689918 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:110 8.2 8:40000000:root:::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x7fec90042040 con 0x7fec980123c0 2023-10-05T19:28:30.735 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:30.730+0000 7feca480a700 1 -- 172.21.15.167:0/1420689918 <== osd.2 v2:172.21.15.167:6816/30037 18 ==== osd_op_reply(110 [pgnls start_epoch 30 out=49b] v30'2 uv2 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fec9003f020 con 0x7fec980123c0 2023-10-05T19:28:30.735 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:30.730+0000 7feca480a700 1 -- 172.21.15.167:0/1420689918 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:111 8.12 8:48000000:root:::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x7fec90042040 con 0x5565578811e0 2023-10-05T19:28:30.736 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:30.730+0000 7feca500b700 1 -- 172.21.15.167:0/1420689918 <== osd.3 v2:172.21.15.167:6808/30036 28 ==== osd_op_reply(111 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fec9804d040 con 0x5565578811e0 2023-10-05T19:28:30.736 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:30.730+0000 7feca500b700 1 -- 172.21.15.167:0/1420689918 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:112 8.a 8:50000000:root:::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x7fec90042040 con 0x5565578505d0 2023-10-05T19:28:30.736 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:30.730+0000 7feca580c700 1 -- 172.21.15.167:0/1420689918 <== osd.1 v2:172.21.15.167:6800/30035 38 ==== osd_op_reply(112 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7feca0050040 con 0x5565578505d0 2023-10-05T19:28:30.736 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:30.730+0000 7feca580c700 1 -- 172.21.15.167:0/1420689918 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:113 8.1a 8:58000000:root:::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x7fec90042040 con 0x7fec90007030 2023-10-05T19:28:30.737 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:30.730+0000 7feca580c700 1 -- 172.21.15.167:0/1420689918 <== osd.0 v2:172.21.15.118:6800/27690 29 ==== osd_op_reply(113 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7feca0050040 con 0x7fec90007030 2023-10-05T19:28:30.737 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:30.730+0000 7feca580c700 1 -- 172.21.15.167:0/1420689918 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:114 8.6 8:60000000:root:::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x7fec90042040 con 0x7fec90007030 2023-10-05T19:28:30.738 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:30.734+0000 7feca580c700 1 -- 172.21.15.167:0/1420689918 <== osd.0 v2:172.21.15.118:6800/27690 30 ==== osd_op_reply(114 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7feca0050040 con 0x7fec90007030 2023-10-05T19:28:30.738 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:30.734+0000 7feca580c700 1 -- 172.21.15.167:0/1420689918 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:115 8.16 8:68000000:root:::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x7fec90042040 con 0x5565578811e0 2023-10-05T19:28:30.739 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:30.734+0000 7feca500b700 1 -- 172.21.15.167:0/1420689918 <== osd.3 v2:172.21.15.167:6808/30036 29 ==== osd_op_reply(115 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fec9804d040 con 0x5565578811e0 2023-10-05T19:28:30.739 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:30.734+0000 7feca500b700 1 -- 172.21.15.167:0/1420689918 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:116 8.e 8:70000000:root:::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x7fec90042040 con 0x7fec90007030 2023-10-05T19:28:30.740 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:30.734+0000 7feca580c700 1 -- 172.21.15.167:0/1420689918 <== osd.0 v2:172.21.15.118:6800/27690 31 ==== osd_op_reply(116 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7feca0050040 con 0x7fec90007030 2023-10-05T19:28:30.740 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:30.734+0000 7feca580c700 1 -- 172.21.15.167:0/1420689918 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:117 8.1e 8:78000000:root:::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x7fec90042040 con 0x7fec90007030 2023-10-05T19:28:30.741 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:30.734+0000 7feca580c700 1 -- 172.21.15.167:0/1420689918 <== osd.0 v2:172.21.15.118:6800/27690 32 ==== osd_op_reply(117 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7feca0050040 con 0x7fec90007030 2023-10-05T19:28:30.741 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:30.734+0000 7feca580c700 1 -- 172.21.15.167:0/1420689918 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:118 8.1 8:80000000:root:::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x7fec90042040 con 0x7fec90007030 2023-10-05T19:28:30.742 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:30.738+0000 7feca580c700 1 -- 172.21.15.167:0/1420689918 <== osd.0 v2:172.21.15.118:6800/27690 33 ==== osd_op_reply(118 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7feca0050040 con 0x7fec90007030 2023-10-05T19:28:30.742 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:30.738+0000 7feca580c700 1 -- 172.21.15.167:0/1420689918 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:119 8.11 8:88000000:root:::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x7fec90042040 con 0x5565578811e0 2023-10-05T19:28:30.743 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:30.738+0000 7feca500b700 1 -- 172.21.15.167:0/1420689918 <== osd.3 v2:172.21.15.167:6808/30036 30 ==== osd_op_reply(119 [pgnls start_epoch 30 out=127b] v30'1 uv1 ondisk = 1) v8 ==== 144+0+127 (crc 0 0 0) 0x7fec9804d040 con 0x5565578811e0 2023-10-05T19:28:30.743 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:30.738+0000 7feca500b700 1 -- 172.21.15.167:0/1420689918 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:120 8.9 8:90000000:root:::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x7fec90042040 con 0x7fec980123c0 2023-10-05T19:28:30.744 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:30.738+0000 7feca480a700 1 -- 172.21.15.167:0/1420689918 <== osd.2 v2:172.21.15.167:6816/30037 19 ==== osd_op_reply(120 [pgnls start_epoch 30 out=70b] v30'2 uv2 ondisk = 1) v8 ==== 144+0+70 (crc 0 0 0) 0x7fec9003f020 con 0x7fec980123c0 2023-10-05T19:28:30.744 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:30.738+0000 7feca480a700 1 -- 172.21.15.167:0/1420689918 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:121 8.19 8:98000000:root:::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x7fec90042040 con 0x5565578505d0 2023-10-05T19:28:30.744 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:30.738+0000 7feca580c700 1 -- 172.21.15.167:0/1420689918 <== osd.1 v2:172.21.15.167:6800/30035 39 ==== osd_op_reply(121 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7feca0050040 con 0x5565578505d0 2023-10-05T19:28:30.745 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:30.738+0000 7feca580c700 1 -- 172.21.15.167:0/1420689918 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:122 8.5 8:a0000000:root:::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x7fec90042040 con 0x5565578505d0 2023-10-05T19:28:30.745 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:30.738+0000 7feca580c700 1 -- 172.21.15.167:0/1420689918 <== osd.1 v2:172.21.15.167:6800/30035 40 ==== osd_op_reply(122 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7feca0050040 con 0x5565578505d0 2023-10-05T19:28:30.745 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:30.738+0000 7feca580c700 1 -- 172.21.15.167:0/1420689918 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:123 8.15 8:a8000000:root:::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x7fec90042040 con 0x5565578811e0 2023-10-05T19:28:30.746 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:30.742+0000 7feca500b700 1 -- 172.21.15.167:0/1420689918 <== osd.3 v2:172.21.15.167:6808/30036 31 ==== osd_op_reply(123 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fec9804d040 con 0x5565578811e0 2023-10-05T19:28:30.746 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:30.742+0000 7feca500b700 1 -- 172.21.15.167:0/1420689918 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:124 8.d 8:b0000000:root:::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x7fec90042040 con 0x7fec980123c0 2023-10-05T19:28:30.747 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:30.742+0000 7feca480a700 1 -- 172.21.15.167:0/1420689918 <== osd.2 v2:172.21.15.167:6816/30037 20 ==== osd_op_reply(124 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fec9003f020 con 0x7fec980123c0 2023-10-05T19:28:30.747 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:30.742+0000 7feca480a700 1 -- 172.21.15.167:0/1420689918 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:125 8.1d 8:b8000000:root:::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x7fec90042040 con 0x7fec90007030 2023-10-05T19:28:30.748 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:30.742+0000 7feca580c700 1 -- 172.21.15.167:0/1420689918 <== osd.0 v2:172.21.15.118:6800/27690 34 ==== osd_op_reply(125 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7feca0050040 con 0x7fec90007030 2023-10-05T19:28:30.748 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:30.742+0000 7feca580c700 1 -- 172.21.15.167:0/1420689918 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:126 8.3 8:c0000000:root:::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x7fec90042040 con 0x5565578505d0 2023-10-05T19:28:30.749 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:30.742+0000 7feca580c700 1 -- 172.21.15.167:0/1420689918 <== osd.1 v2:172.21.15.167:6800/30035 41 ==== osd_op_reply(126 [pgnls start_epoch 30 out=49b] v30'1 uv1 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7feca0050040 con 0x5565578505d0 2023-10-05T19:28:30.749 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:30.742+0000 7feca580c700 1 -- 172.21.15.167:0/1420689918 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:127 8.13 8:c8000000:root:::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x7fec90042040 con 0x7fec90007030 2023-10-05T19:28:30.750 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:30.746+0000 7feca580c700 1 -- 172.21.15.167:0/1420689918 <== osd.0 v2:172.21.15.118:6800/27690 35 ==== osd_op_reply(127 [pgnls start_epoch 30 out=49b] v30'2 uv2 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7feca0050040 con 0x7fec90007030 2023-10-05T19:28:30.750 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:30.746+0000 7feca580c700 1 -- 172.21.15.167:0/1420689918 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:128 8.b 8:d0000000:root:::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x7fec90042040 con 0x5565578811e0 2023-10-05T19:28:30.751 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:30.746+0000 7feca500b700 1 -- 172.21.15.167:0/1420689918 <== osd.3 v2:172.21.15.167:6808/30036 32 ==== osd_op_reply(128 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fec9804d040 con 0x5565578811e0 2023-10-05T19:28:30.751 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:30.746+0000 7feca500b700 1 -- 172.21.15.167:0/1420689918 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:129 8.1b 8:d8000000:root:::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x7fec90042040 con 0x7fec980123c0 2023-10-05T19:28:30.752 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:30.746+0000 7feca480a700 1 -- 172.21.15.167:0/1420689918 <== osd.2 v2:172.21.15.167:6816/30037 21 ==== osd_op_reply(129 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fec9003f020 con 0x7fec980123c0 2023-10-05T19:28:30.752 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:30.746+0000 7feca480a700 1 -- 172.21.15.167:0/1420689918 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:130 8.7 8:e0000000:root:::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x7fec90042040 con 0x7fec90007030 2023-10-05T19:28:30.753 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:30.746+0000 7feca580c700 1 -- 172.21.15.167:0/1420689918 <== osd.0 v2:172.21.15.118:6800/27690 36 ==== osd_op_reply(130 [pgnls start_epoch 30 out=49b] v30'1 uv1 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7feca0050040 con 0x7fec90007030 2023-10-05T19:28:30.753 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:30.746+0000 7feca580c700 1 -- 172.21.15.167:0/1420689918 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:131 8.17 8:e8000000:root:::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x7fec90042040 con 0x5565578505d0 2023-10-05T19:28:30.753 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:30.746+0000 7feca580c700 1 -- 172.21.15.167:0/1420689918 <== osd.1 v2:172.21.15.167:6800/30035 42 ==== osd_op_reply(131 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7feca0050040 con 0x5565578505d0 2023-10-05T19:28:30.753 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:30.746+0000 7feca580c700 1 -- 172.21.15.167:0/1420689918 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:132 8.f 8:f0000000:root:::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x7fec90042040 con 0x5565578811e0 2023-10-05T19:28:30.755 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:30.750+0000 7feca500b700 1 -- 172.21.15.167:0/1420689918 <== osd.3 v2:172.21.15.167:6808/30036 33 ==== osd_op_reply(132 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fec9804d040 con 0x5565578811e0 2023-10-05T19:28:30.755 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:30.750+0000 7feca500b700 1 -- 172.21.15.167:0/1420689918 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:133 8.1f 8:f8000000:root:::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x7fec90042040 con 0x7fec980123c0 2023-10-05T19:28:30.755 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:30.750+0000 7feca480a700 1 -- 172.21.15.167:0/1420689918 <== osd.2 v2:172.21.15.167:6816/30037 22 ==== osd_op_reply(133 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fec9003f020 con 0x7fec980123c0 2023-10-05T19:28:30.756 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:30.750+0000 7feca74d6e40 20 RGWRados::pool_iterate: got .bucket.meta.myfoo:9a9df965-6612-45d4-a45f-6e3c889bacc2.4223.1 2023-10-05T19:28:30.756 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:30.750+0000 7feca74d6e40 20 RGWRados::pool_iterate: got myfoo 2023-10-05T19:28:30.756 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:30.750+0000 7feca74d6e40 1 -- 172.21.15.167:0/1420689918 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:134 7.1f 7:f95f44c2:::notify.0:head [watch unwatch cookie 93893748613024] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x5565578c5da0 con 0x7fec980123c0 2023-10-05T19:28:30.759 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:30.754+0000 7feca480a700 1 -- 172.21.15.167:0/1420689918 <== osd.2 v2:172.21.15.167:6816/30037 23 ==== osd_op_reply(134 notify.0 [watch unwatch cookie 93893748613024] v30'92 uv90 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7fec9003f020 con 0x7fec980123c0 2023-10-05T19:28:30.759 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:30.754+0000 7feca74d6e40 20 remove_watcher() i=0 2023-10-05T19:28:30.759 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:30.754+0000 7feca74d6e40 2 removed watcher, disabling cache 2023-10-05T19:28:30.759 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:30.754+0000 7feca74d6e40 1 -- 172.21.15.167:0/1420689918 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:135 7.0 7:05bf5b68:::notify.1:head [watch unwatch cookie 93893748617808] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x556557885fa0 con 0x5565578505d0 2023-10-05T19:28:30.762 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:30.758+0000 7feca580c700 1 -- 172.21.15.167:0/1420689918 <== osd.1 v2:172.21.15.167:6800/30035 43 ==== osd_op_reply(135 notify.1 [watch unwatch cookie 93893748617808] v30'92 uv90 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7feca0050040 con 0x5565578505d0 2023-10-05T19:28:30.762 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:30.758+0000 7feca74d6e40 20 remove_watcher() i=1 2023-10-05T19:28:30.762 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:30.758+0000 7feca74d6e40 1 -- 172.21.15.167:0/1420689918 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:136 7.15 7:a93a5511:::notify.2:head [watch unwatch cookie 93893748619808] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x556557887250 con 0x5565578811e0 2023-10-05T19:28:30.765 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:30.758+0000 7feca500b700 1 -- 172.21.15.167:0/1420689918 <== osd.3 v2:172.21.15.167:6808/30036 34 ==== osd_op_reply(136 notify.2 [watch unwatch cookie 93893748619808] v30'92 uv90 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7fec9804d040 con 0x5565578811e0 2023-10-05T19:28:30.765 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:30.758+0000 7feca74d6e40 20 remove_watcher() i=2 2023-10-05T19:28:30.765 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:30.758+0000 7feca74d6e40 1 -- 172.21.15.167:0/1420689918 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:137 7.e 7:7759931f:::notify.3:head [watch unwatch cookie 93893748624224] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x556557887a20 con 0x7fec980123c0 2023-10-05T19:28:30.767 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:30.762+0000 7feca480a700 1 -- 172.21.15.167:0/1420689918 <== osd.2 v2:172.21.15.167:6816/30037 24 ==== osd_op_reply(137 notify.3 [watch unwatch cookie 93893748624224] v30'92 uv90 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7fec9003f020 con 0x7fec980123c0 2023-10-05T19:28:30.767 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:30.762+0000 7feca74d6e40 20 remove_watcher() i=3 2023-10-05T19:28:30.768 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:30.762+0000 7feca74d6e40 1 -- 172.21.15.167:0/1420689918 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:138 7.d 7:b4812045:::notify.4:head [watch unwatch cookie 93893748620912] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x556557887a20 con 0x5565578505d0 2023-10-05T19:28:30.770 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:30.762+0000 7feca580c700 1 -- 172.21.15.167:0/1420689918 <== osd.1 v2:172.21.15.167:6800/30035 44 ==== osd_op_reply(138 notify.4 [watch unwatch cookie 93893748620912] v30'92 uv90 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7feca0050040 con 0x5565578505d0 2023-10-05T19:28:30.770 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:30.766+0000 7feca74d6e40 20 remove_watcher() i=4 2023-10-05T19:28:30.770 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:30.766+0000 7feca74d6e40 1 -- 172.21.15.167:0/1420689918 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:139 7.3 7:c609908c:::notify.5:head [watch unwatch cookie 93893748629968] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x556557887f10 con 0x7fec90007030 2023-10-05T19:28:30.773 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:30.766+0000 7feca580c700 1 -- 172.21.15.167:0/1420689918 <== osd.0 v2:172.21.15.118:6800/27690 37 ==== osd_op_reply(139 notify.5 [watch unwatch cookie 93893748629968] v30'92 uv90 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7feca0050040 con 0x7fec90007030 2023-10-05T19:28:30.773 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:30.766+0000 7feca74d6e40 20 remove_watcher() i=5 2023-10-05T19:28:30.773 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:30.766+0000 7feca74d6e40 1 -- 172.21.15.167:0/1420689918 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:140 7.14 7:2b04a3e9:::notify.6:head [watch unwatch cookie 93893748634272] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x556557887f10 con 0x7fec90007030 2023-10-05T19:28:30.776 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:30.770+0000 7feca580c700 1 -- 172.21.15.167:0/1420689918 <== osd.0 v2:172.21.15.118:6800/27690 38 ==== osd_op_reply(140 notify.6 [watch unwatch cookie 93893748634272] v30'92 uv90 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7feca0050040 con 0x7fec90007030 2023-10-05T19:28:30.776 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:30.770+0000 7feca74d6e40 20 remove_watcher() i=6 2023-10-05T19:28:30.777 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:30.770+0000 7feca74d6e40 1 -- 172.21.15.167:0/1420689918 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:141 7.9 7:93e5b521:::notify.7:head [watch unwatch cookie 93893748635376] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x556557887f10 con 0x7fec90007030 2023-10-05T19:28:30.779 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:30.774+0000 7feca580c700 1 -- 172.21.15.167:0/1420689918 <== osd.0 v2:172.21.15.118:6800/27690 39 ==== osd_op_reply(141 notify.7 [watch unwatch cookie 93893748635376] v30'92 uv90 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7feca0050040 con 0x7fec90007030 2023-10-05T19:28:30.780 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:30.774+0000 7feca74d6e40 20 remove_watcher() i=7 2023-10-05T19:28:30.781 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:30.774+0000 7fec767e4700 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2023-10-05T19:28:30.781 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:30.774+0000 7feca74d6e40 1 -- 172.21.15.167:0/1420689918 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:142 6.4 6:22d26bf9:::data_loggenerations_metadata:head [watch unwatch cookie 93893748636848] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x556557887f10 con 0x5565578505d0 2023-10-05T19:28:30.785 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:30.778+0000 7feca580c700 1 -- 172.21.15.167:0/1420689918 <== osd.1 v2:172.21.15.167:6800/30035 45 ==== osd_op_reply(142 data_loggenerations_metadata [watch unwatch cookie 93893748636848] v30'95 uv1 ondisk = 0) v8 ==== 172+0+0 (crc 0 0 0) 0x7feca0050040 con 0x5565578505d0 2023-10-05T19:28:30.785 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:30.778+0000 7feca74d6e40 1 -- 172.21.15.167:0/1420689918 >> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] conn(0x7fec90007030 msgr2=0x7fec9001d430 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-05T19:28:30.785 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:30.778+0000 7feca74d6e40 1 --2- 172.21.15.167:0/1420689918 >> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] conn(0x7fec90007030 0x7fec9001d430 crc :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-05T19:28:30.786 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:30.778+0000 7feca74d6e40 1 -- 172.21.15.167:0/1420689918 >> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] conn(0x5565578505d0 msgr2=0x55655787d780 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-05T19:28:30.786 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:30.778+0000 7feca74d6e40 1 --2- 172.21.15.167:0/1420689918 >> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] conn(0x5565578505d0 0x55655787d780 crc :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-05T19:28:30.786 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:30.778+0000 7feca74d6e40 1 -- 172.21.15.167:0/1420689918 >> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] conn(0x7fec980123c0 msgr2=0x7fec98028710 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-05T19:28:30.786 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:30.778+0000 7feca74d6e40 1 --2- 172.21.15.167:0/1420689918 >> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] conn(0x7fec980123c0 0x7fec98028710 crc :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-05T19:28:30.786 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:30.778+0000 7feca74d6e40 1 -- 172.21.15.167:0/1420689918 >> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] conn(0x5565578811e0 msgr2=0x5565578835c0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-05T19:28:30.786 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:30.778+0000 7feca74d6e40 1 --2- 172.21.15.167:0/1420689918 >> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] conn(0x5565578811e0 0x5565578835c0 crc :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-05T19:28:30.786 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:30.778+0000 7feca74d6e40 1 -- 172.21.15.167:0/1420689918 >> [v2:172.21.15.167:6824/29889,v1:172.21.15.167:6825/29889] conn(0x7fec88030c50 msgr2=0x7fec88033100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-05T19:28:30.786 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:30.778+0000 7feca74d6e40 1 --2- 172.21.15.167:0/1420689918 >> [v2:172.21.15.167:6824/29889,v1:172.21.15.167:6825/29889] conn(0x7fec88030c50 0x7fec88033100 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7fec900030c0 tx=0x7fec90014000 comp rx=0 tx=0).stop 2023-10-05T19:28:30.786 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:30.778+0000 7feca74d6e40 1 -- 172.21.15.167:0/1420689918 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55655784bff0 msgr2=0x556557871cf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-05T19:28:30.787 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:30.778+0000 7feca74d6e40 1 --2- 172.21.15.167:0/1420689918 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55655784bff0 0x556557871cf0 secure :-1 s=READY pgs=317 cs=0 l=1 rev1=1 crypto rx=0x7fec9800f000 tx=0x7fec98010000 comp rx=0 tx=0).stop 2023-10-05T19:28:30.787 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:30.778+0000 7feca580c700 1 -- 172.21.15.167:0/1420689918 reap_dead start 2023-10-05T19:28:30.787 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:30.778+0000 7feca74d6e40 1 -- 172.21.15.167:0/1420689918 shutdown_connections 2023-10-05T19:28:30.787 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:30.778+0000 7feca74d6e40 1 -- 172.21.15.167:0/1420689918 >> 172.21.15.167:0/1420689918 conn(0x5565577bf840 msgr2=0x556557865550 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-05T19:28:30.787 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:30.778+0000 7feca74d6e40 1 -- 172.21.15.167:0/1420689918 shutdown_connections 2023-10-05T19:28:30.787 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:30.778+0000 7feca74d6e40 1 -- 172.21.15.167:0/1420689918 wait complete. 2023-10-05T19:28:30.788 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:30.778+0000 7feca74d6e40 1 -- 172.21.15.167:0/2215951926 >> [v2:172.21.15.167:6824/29889,v1:172.21.15.167:6825/29889] conn(0x7febec030c50 msgr2=0x7febec033100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-05T19:28:30.788 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:30.778+0000 7feca74d6e40 1 --2- 172.21.15.167:0/2215951926 >> [v2:172.21.15.167:6824/29889,v1:172.21.15.167:6825/29889] conn(0x7febec030c50 0x7febec033100 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x55655788f2f0 tx=0x7feca0000f90 comp rx=0 tx=0).stop 2023-10-05T19:28:30.788 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:30.778+0000 7feca74d6e40 1 -- 172.21.15.167:0/2215951926 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5565578996f0 msgr2=0x55655789f1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-05T19:28:30.788 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:30.778+0000 7feca74d6e40 1 --2- 172.21.15.167:0/2215951926 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5565578996f0 0x55655789f1a0 secure :-1 s=READY pgs=319 cs=0 l=1 rev1=1 crypto rx=0x5565578937d0 tx=0x7fec90038000 comp rx=0 tx=0).stop 2023-10-05T19:28:30.788 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:30.782+0000 7feca74d6e40 1 -- 172.21.15.167:0/2215951926 shutdown_connections 2023-10-05T19:28:30.788 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:30.782+0000 7feca74d6e40 1 --2- 172.21.15.167:0/2215951926 >> [v2:172.21.15.167:6824/29889,v1:172.21.15.167:6825/29889] conn(0x7febec030c50 0x7febec033100 unknown :-1 s=CLOSED pgs=113 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-05T19:28:30.788 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:30.782+0000 7feca74d6e40 1 --2- 172.21.15.167:0/2215951926 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5565578996f0 0x55655789f1a0 unknown :-1 s=CLOSED pgs=319 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-05T19:28:30.789 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:30.782+0000 7feca74d6e40 1 -- 172.21.15.167:0/2215951926 >> 172.21.15.167:0/2215951926 conn(0x556557894ab0 msgr2=0x556557892bb0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-05T19:28:30.789 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:30.782+0000 7feca74d6e40 1 -- 172.21.15.167:0/2215951926 shutdown_connections 2023-10-05T19:28:30.789 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:30.782+0000 7feca74d6e40 1 -- 172.21.15.167:0/2215951926 wait complete. 2023-10-05T19:28:30.789 INFO:teuthology.orchestra.run.smithi167.stdout:[ 2023-10-05T19:28:30.790 INFO:teuthology.orchestra.run.smithi167.stdout: "myfoo" 2023-10-05T19:28:30.790 INFO:teuthology.orchestra.run.smithi167.stdout:] 2023-10-05T19:28:30.792 DEBUG:tasks.util.rgw: json result: ['myfoo'] 2023-10-05T19:28:30.813 INFO:tasks.radosgw_admin:RL: create_bucket myfoo2 foo bytes_out=0 bytes_in=0 failed=False 2023-10-05T19:28:30.828 INFO:tasks.radosgw_admin:RL: create_bucket myfoo3 foo bytes_out=0 bytes_in=0 failed=False 2023-10-05T19:28:30.846 INFO:tasks.radosgw_admin:RL: create_bucket myfoo4 foo bytes_out=0 bytes_in=0 failed=False 2023-10-05T19:28:30.895 INFO:tasks.radosgw_admin:RL: create_bucket myfoo5 foo bytes_out=0 bytes_in=218 failed=True 2023-10-05T19:28:30.972 INFO:tasks.radosgw_admin:RL: delete_bucket myfoo2 foo bytes_out=0 bytes_in=0 failed=False 2023-10-05T19:28:31.042 INFO:tasks.radosgw_admin:RL: delete_bucket myfoo3 foo bytes_out=0 bytes_in=0 failed=False 2023-10-05T19:28:31.120 INFO:tasks.radosgw_admin:RL: delete_bucket myfoo4 foo bytes_out=0 bytes_in=0 failed=False 2023-10-05T19:28:31.120 INFO:tasks.util.rgw:rgwadmin: client.0 : ['bucket', 'stats', '--bucket', 'myfoo'] 2023-10-05T19:28:31.121 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'ceph', 'bucket', 'stats', '--bucket', 'myfoo'] 2023-10-05T19:28:31.121 DEBUG:teuthology.orchestra.run.smithi167:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster ceph bucket stats --bucket myfoo 2023-10-05T19:28:31.161 INFO:teuthology.orchestra.run.smithi167.stderr:ignoring --setuser ceph since I am not root 2023-10-05T19:28:31.161 INFO:teuthology.orchestra.run.smithi167.stderr:ignoring --setgroup ceph since I am not root 2023-10-05T19:28:31.162 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.154+0000 7f3d0d3cae40 1 Processor -- start 2023-10-05T19:28:31.162 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.154+0000 7f3d0d3cae40 1 -- start start 2023-10-05T19:28:31.162 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.154+0000 7f3d0d3cae40 1 --2- >> v2:172.21.15.118:3300/0 conn(0x56427130e410 0x56427130cfc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-05T19:28:31.162 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.158+0000 7f3d0d3cae40 1 -- --> v1:172.21.15.118:6789/0 -- auth(proto 0 26 bytes epoch 0) v1 -- 0x56427125abd0 con 0x56427130e7e0 2023-10-05T19:28:31.162 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.158+0000 7f3d0d3cae40 1 -- --> v2:172.21.15.118:3300/0 -- mon_getmap magic: 0 v1 -- 0x564271217ee0 con 0x56427130e410 2023-10-05T19:28:31.163 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.158+0000 7f3d0aeff700 1 --2- >> v2:172.21.15.118:3300/0 conn(0x56427130e410 0x56427130cfc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-05T19:28:31.163 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.158+0000 7f3d0a6fe700 1 --1- >> v1:172.21.15.118:6789/0 conn(0x56427130e7e0 0x56427130d500 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.118:6789/0 says I am v1:172.21.15.167:56568/0 (socket says 172.21.15.167:56568) 2023-10-05T19:28:31.163 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.158+0000 7f3d0a6fe700 1 -- 172.21.15.167:0/1645974084 learned_addr learned my addr 172.21.15.167:0/1645974084 (peer_addr_for_me v1:172.21.15.167:0/0) 2023-10-05T19:28:31.163 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.158+0000 7f3d09efd700 1 -- 172.21.15.167:0/1645974084 <== mon.0 v1:172.21.15.118:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 323508066 0 0) 0x56427125abd0 con 0x56427130e7e0 2023-10-05T19:28:31.164 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.158+0000 7f3d09efd700 1 -- 172.21.15.167:0/1645974084 --> v1:172.21.15.118:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3cf8001e10 con 0x56427130e7e0 2023-10-05T19:28:31.164 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.158+0000 7f3d0aeff700 1 -- 172.21.15.167:0/1645974084 >> v1:172.21.15.118:6789/0 conn(0x56427130e7e0 legacy=0x56427130d500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-05T19:28:31.164 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.158+0000 7f3d0aeff700 1 -- 172.21.15.167:0/1645974084 --> v2:172.21.15.118:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x56427130dcd0 con 0x56427130e410 2023-10-05T19:28:31.165 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.158+0000 7f3d0aeff700 1 --2- 172.21.15.167:0/1645974084 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x56427130e410 0x56427130cfc0 secure :-1 s=READY pgs=321 cs=0 l=1 rev1=1 crypto rx=0x7f3cfc00a520 tx=0x7f3cfc005d40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=823c88c39f7de425 server_cookie=0 in_seq=0 out_seq=0 2023-10-05T19:28:31.165 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.158+0000 7f3d09efd700 1 -- 172.21.15.167:0/1645974084 <== mon.0 v2:172.21.15.118:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f3cfc00f590 con 0x56427130e410 2023-10-05T19:28:31.165 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.158+0000 7f3d09efd700 1 -- 172.21.15.167:0/1645974084 <== mon.0 v2:172.21.15.118:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3cfc00f6f0 con 0x56427130e410 2023-10-05T19:28:31.165 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.158+0000 7f3d09efd700 1 -- 172.21.15.167:0/1645974084 <== mon.0 v2:172.21.15.118:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f3cfc00f860 con 0x56427130e410 2023-10-05T19:28:31.165 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.158+0000 7f3d0d3cae40 1 -- 172.21.15.167:0/1645974084 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x56427130e410 msgr2=0x56427130cfc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-05T19:28:31.165 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.158+0000 7f3d0d3cae40 1 --2- 172.21.15.167:0/1645974084 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x56427130e410 0x56427130cfc0 secure :-1 s=READY pgs=321 cs=0 l=1 rev1=1 crypto rx=0x7f3cfc00a520 tx=0x7f3cfc005d40 comp rx=0 tx=0).stop 2023-10-05T19:28:31.165 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.158+0000 7f3d0d3cae40 1 -- 172.21.15.167:0/1645974084 shutdown_connections 2023-10-05T19:28:31.166 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.158+0000 7f3d0d3cae40 1 --2- 172.21.15.167:0/1645974084 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x56427130e410 0x56427130cfc0 unknown :-1 s=CLOSED pgs=321 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-05T19:28:31.166 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.158+0000 7f3d0d3cae40 1 -- 172.21.15.167:0/1645974084 >> 172.21.15.167:0/1645974084 conn(0x564271308720 msgr2=0x56427130ab30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-05T19:28:31.166 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.158+0000 7f3d0d3cae40 1 -- 172.21.15.167:0/1645974084 shutdown_connections 2023-10-05T19:28:31.166 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.158+0000 7f3d0d3cae40 1 -- 172.21.15.167:0/1645974084 wait complete. 2023-10-05T19:28:31.167 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.158+0000 7f3d0d3cae40 1 Processor -- start 2023-10-05T19:28:31.167 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.158+0000 7f3d0d3cae40 1 -- start start 2023-10-05T19:28:31.167 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.158+0000 7f3d0d3cae40 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x56427130e410 0x56427132ef10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-05T19:28:31.167 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.158+0000 7f3d0d3cae40 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x5642711dbe30 con 0x56427130e410 2023-10-05T19:28:31.167 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.162+0000 7f3d0aeff700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x56427130e410 0x56427132ef10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-05T19:28:31.167 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.162+0000 7f3d0aeff700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x56427130e410 0x56427132ef10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.118:3300/0 says I am v2:172.21.15.167:58158/0 (socket says 172.21.15.167:58158) 2023-10-05T19:28:31.168 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.162+0000 7f3d0aeff700 1 -- 172.21.15.167:0/1618850792 learned_addr learned my addr 172.21.15.167:0/1618850792 (peer_addr_for_me v2:172.21.15.167:0/0) 2023-10-05T19:28:31.168 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.162+0000 7f3d0aeff700 1 -- 172.21.15.167:0/1618850792 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3cfc00a0e0 con 0x56427130e410 2023-10-05T19:28:31.168 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.162+0000 7f3d0aeff700 1 --2- 172.21.15.167:0/1618850792 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x56427130e410 0x56427132ef10 secure :-1 s=READY pgs=322 cs=0 l=1 rev1=1 crypto rx=0x7f3cfc00a470 tx=0x7f3cfc014000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=654b6accdf7534c2 server_cookie=0 in_seq=0 out_seq=0 2023-10-05T19:28:31.169 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.162+0000 7f3d08efb700 1 -- 172.21.15.167:0/1618850792 <== mon.0 v2:172.21.15.118:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f3cfc00f860 con 0x56427130e410 2023-10-05T19:28:31.169 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.162+0000 7f3d08efb700 1 -- 172.21.15.167:0/1618850792 <== mon.0 v2:172.21.15.118:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3cfc00f9c0 con 0x56427130e410 2023-10-05T19:28:31.170 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.162+0000 7f3d08efb700 1 -- 172.21.15.167:0/1618850792 <== mon.0 v2:172.21.15.118:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f3cfc00fb30 con 0x56427130e410 2023-10-05T19:28:31.170 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.162+0000 7f3d0d3cae40 1 -- 172.21.15.167:0/1618850792 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x56427130e410 msgr2=0x56427132ef10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-05T19:28:31.170 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.162+0000 7f3d0d3cae40 1 --2- 172.21.15.167:0/1618850792 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x56427130e410 0x56427132ef10 secure :-1 s=READY pgs=322 cs=0 l=1 rev1=1 crypto rx=0x7f3cfc00a470 tx=0x7f3cfc014000 comp rx=0 tx=0).stop 2023-10-05T19:28:31.170 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.162+0000 7f3d0d3cae40 1 -- 172.21.15.167:0/1618850792 shutdown_connections 2023-10-05T19:28:31.170 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.162+0000 7f3d0d3cae40 1 --2- 172.21.15.167:0/1618850792 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x56427130e410 0x56427132ef10 unknown :-1 s=CLOSED pgs=322 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-05T19:28:31.170 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.162+0000 7f3d0d3cae40 1 -- 172.21.15.167:0/1618850792 >> 172.21.15.167:0/1618850792 conn(0x564271308720 msgr2=0x56427132aad0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-05T19:28:31.170 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.162+0000 7f3d0d3cae40 1 -- 172.21.15.167:0/1618850792 shutdown_connections 2023-10-05T19:28:31.170 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.162+0000 7f3d0d3cae40 1 -- 172.21.15.167:0/1618850792 wait complete. 2023-10-05T19:28:31.170 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.162+0000 7f3d0d3cae40 1 Processor -- start 2023-10-05T19:28:31.170 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.162+0000 7f3d0d3cae40 1 -- start start 2023-10-05T19:28:31.171 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.162+0000 7f3d0d3cae40 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x56427130e410 0x564271335bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-05T19:28:31.171 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.162+0000 7f3d0d3cae40 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x5642711aeed0 con 0x56427130e410 2023-10-05T19:28:31.171 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.162+0000 7f3d0aeff700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x56427130e410 0x564271335bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-05T19:28:31.171 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.162+0000 7f3d0aeff700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x56427130e410 0x564271335bf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.118:3300/0 says I am v2:172.21.15.167:58164/0 (socket says 172.21.15.167:58164) 2023-10-05T19:28:31.171 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.162+0000 7f3d0aeff700 1 -- 172.21.15.167:0/3739225997 learned_addr learned my addr 172.21.15.167:0/3739225997 (peer_addr_for_me v2:172.21.15.167:0/0) 2023-10-05T19:28:31.172 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.166+0000 7f3d0aeff700 1 -- 172.21.15.167:0/3739225997 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x56427130dcd0 con 0x56427130e410 2023-10-05T19:28:31.172 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.166+0000 7f3d0aeff700 1 --2- 172.21.15.167:0/3739225997 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x56427130e410 0x564271335bf0 secure :-1 s=READY pgs=323 cs=0 l=1 rev1=1 crypto rx=0x7f3cfc00a520 tx=0x7f3cfc01d000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-05T19:28:31.172 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.166+0000 7f3d02ffd700 1 -- 172.21.15.167:0/3739225997 <== mon.0 v2:172.21.15.118:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f3cfc0032f0 con 0x56427130e410 2023-10-05T19:28:31.172 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.166+0000 7f3d02ffd700 1 -- 172.21.15.167:0/3739225997 <== mon.0 v2:172.21.15.118:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3cfc010420 con 0x56427130e410 2023-10-05T19:28:31.172 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.166+0000 7f3d0d3cae40 1 -- 172.21.15.167:0/3739225997 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x564271336130 con 0x56427130e410 2023-10-05T19:28:31.172 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.166+0000 7f3d02ffd700 1 -- 172.21.15.167:0/3739225997 <== mon.0 v2:172.21.15.118:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f3cfc0105f0 con 0x56427130e410 2023-10-05T19:28:31.173 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.166+0000 7f3d0d3cae40 1 -- 172.21.15.167:0/3739225997 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x5642713365d0 con 0x56427130e410 2023-10-05T19:28:31.173 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.166+0000 7f3d0d3cae40 20 rados->read ofs=0 len=0 2023-10-05T19:28:31.174 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.166+0000 7f3d02ffd700 1 -- 172.21.15.167:0/3739225997 <== mon.0 v2:172.21.15.118:3300/0 4 ==== mgrmap(e 5) v1 ==== 40329+0+0 (secure 0 0 0) 0x7f3cfc010750 con 0x56427130e410 2023-10-05T19:28:31.174 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.166+0000 7f3d02ffd700 1 --2- 172.21.15.167:0/3739225997 >> [v2:172.21.15.167:6824/29889,v1:172.21.15.167:6825/29889] conn(0x7f3cec030c50 0x7f3cec033100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-05T19:28:31.174 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.166+0000 7f3d02ffd700 1 -- 172.21.15.167:0/3739225997 <== mon.0 v2:172.21.15.118:3300/0 5 ==== osd_map(30..30 src has 1..30) v4 ==== 6057+0+0 (secure 0 0 0) 0x7f3cfc02b070 con 0x56427130e410 2023-10-05T19:28:31.174 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.166+0000 7f3d0a6fe700 1 --2- 172.21.15.167:0/3739225997 >> [v2:172.21.15.167:6824/29889,v1:172.21.15.167:6825/29889] conn(0x7f3cec030c50 0x7f3cec033100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-05T19:28:31.174 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.166+0000 7f3d0d3cae40 1 --2- 172.21.15.167:0/3739225997 >> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] conn(0x56427130e7e0 0x5642713416b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-05T19:28:31.174 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.166+0000 7f3d0d3cae40 1 -- 172.21.15.167:0/3739225997 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:1 5.12 5:49953fa1:::default.realm:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x564271341cd0 con 0x56427130e7e0 2023-10-05T19:28:31.174 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.166+0000 7f3d0b700700 1 --2- 172.21.15.167:0/3739225997 >> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] conn(0x56427130e7e0 0x5642713416b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-05T19:28:31.174 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.170+0000 7f3d0a6fe700 1 --2- 172.21.15.167:0/3739225997 >> [v2:172.21.15.167:6824/29889,v1:172.21.15.167:6825/29889] conn(0x7f3cec030c50 0x7f3cec033100 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7f3cf4004170 tx=0x7f3cf4001400 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-05T19:28:31.175 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.170+0000 7f3d0b700700 1 --2- 172.21.15.167:0/3739225997 >> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] conn(0x56427130e7e0 0x5642713416b0 crc :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-05T19:28:31.176 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.170+0000 7f3d0b700700 1 -- 172.21.15.167:0/3739225997 <== osd.1 v2:172.21.15.167:6800/30035 1 ==== osd_op_reply(1 default.realm [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 ==== 157+0+0 (crc 0 0 0) 0x7f3d0400a500 con 0x56427130e7e0 2023-10-05T19:28:31.176 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.170+0000 7f3d0d3cae40 20 rados_obj.operate() r=-2 bl.length=0 2023-10-05T19:28:31.176 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.170+0000 7f3d0d3cae40 20 realm 2023-10-05T19:28:31.176 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.170+0000 7f3d0d3cae40 20 rados->read ofs=0 len=0 2023-10-05T19:28:31.176 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.170+0000 7f3d0d3cae40 1 -- 172.21.15.167:0/3739225997 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:2 5.12 5:49953fa1:::default.realm:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x56427133ed10 con 0x56427130e7e0 2023-10-05T19:28:31.176 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.170+0000 7f3d0b700700 1 -- 172.21.15.167:0/3739225997 <== osd.1 v2:172.21.15.167:6800/30035 2 ==== osd_op_reply(2 default.realm [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 ==== 157+0+0 (crc 0 0 0) 0x7f3d0400a500 con 0x56427130e7e0 2023-10-05T19:28:31.176 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.170+0000 7f3d0d3cae40 20 rados_obj.operate() r=-2 bl.length=0 2023-10-05T19:28:31.176 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.170+0000 7f3d0d3cae40 4 RGWPeriod::init failed to init realm id : (2) No such file or directory 2023-10-05T19:28:31.176 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.170+0000 7f3d0d3cae40 20 rados->read ofs=0 len=0 2023-10-05T19:28:31.177 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.170+0000 7f3d0d3cae40 1 -- 172.21.15.167:0/3739225997 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:3 5.12 5:49953fa1:::default.realm:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x5642713427b0 con 0x56427130e7e0 2023-10-05T19:28:31.178 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.170+0000 7f3d0b700700 1 -- 172.21.15.167:0/3739225997 <== osd.1 v2:172.21.15.167:6800/30035 3 ==== osd_op_reply(3 default.realm [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 ==== 157+0+0 (crc 0 0 0) 0x7f3d0400a500 con 0x56427130e7e0 2023-10-05T19:28:31.178 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.170+0000 7f3d0d3cae40 20 rados_obj.operate() r=-2 bl.length=0 2023-10-05T19:28:31.178 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.170+0000 7f3d0d3cae40 20 rados->read ofs=0 len=0 2023-10-05T19:28:31.179 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.170+0000 7f3d0d3cae40 1 -- 172.21.15.167:0/3739225997 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:4 5.f 5:f43b6ece:::zone_names.default:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x564271342390 con 0x56427130e7e0 2023-10-05T19:28:31.179 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.170+0000 7f3d0b700700 1 -- 172.21.15.167:0/3739225997 <== osd.1 v2:172.21.15.167:6800/30035 4 ==== osd_op_reply(4 zone_names.default [read 0~46 out=46b] v0'0 uv1 ondisk = 0) v8 ==== 162+0+46 (crc 0 0 0) 0x7f3d0400a500 con 0x56427130e7e0 2023-10-05T19:28:31.179 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.170+0000 7f3d0d3cae40 20 rados_obj.operate() r=0 bl.length=46 2023-10-05T19:28:31.179 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.170+0000 7f3d0d3cae40 20 rados->read ofs=0 len=0 2023-10-05T19:28:31.179 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.170+0000 7f3d0d3cae40 1 -- 172.21.15.167:0/3739225997 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:5 5.1d 5:b8374eb4:::zone_info.9a9df965-6612-45d4-a45f-6e3c889bacc2:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x564271343220 con 0x56427130e7e0 2023-10-05T19:28:31.179 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.170+0000 7f3d0b700700 1 -- 172.21.15.167:0/3739225997 <== osd.1 v2:172.21.15.167:6800/30035 5 ==== osd_op_reply(5 zone_info.9a9df965-6612-45d4-a45f-6e3c889bacc2 [read 0~874 out=874b] v0'0 uv2 ondisk = 0) v8 ==== 190+0+874 (crc 0 0 0) 0x7f3d0400a500 con 0x56427130e7e0 2023-10-05T19:28:31.179 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.170+0000 7f3d0d3cae40 20 rados_obj.operate() r=0 bl.length=874 2023-10-05T19:28:31.179 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.170+0000 7f3d0d3cae40 20 searching for the correct realm 2023-10-05T19:28:31.179 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.170+0000 7f3d0d3cae40 1 --2- 172.21.15.167:0/3739225997 >> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] conn(0x564271345070 0x564271347450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-05T19:28:31.179 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.170+0000 7f3d0d3cae40 1 -- 172.21.15.167:0/3739225997 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:6 5.0 5:00000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x564271347990 con 0x564271345070 2023-10-05T19:28:31.179 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.170+0000 7f3d0aeff700 1 --2- 172.21.15.167:0/3739225997 >> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] conn(0x564271345070 0x564271347450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-05T19:28:31.180 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.174+0000 7f3d0aeff700 1 --2- 172.21.15.167:0/3739225997 >> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] conn(0x564271345070 0x564271347450 crc :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-05T19:28:31.181 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.174+0000 7f3d0aeff700 1 -- 172.21.15.167:0/3739225997 <== osd.3 v2:172.21.15.167:6808/30036 1 ==== osd_op_reply(6 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f3cfc0377b0 con 0x564271345070 2023-10-05T19:28:31.181 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.174+0000 7f3d0aeff700 1 --2- 172.21.15.167:0/3739225997 >> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] conn(0x7f3cfc0373b0 0x7f3cfc037a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-05T19:28:31.181 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.174+0000 7f3d0aeff700 1 -- 172.21.15.167:0/3739225997 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:7 5.10 5:08000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x564271347990 con 0x7f3cfc0373b0 2023-10-05T19:28:31.181 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.174+0000 7f3d0a6fe700 1 --2- 172.21.15.167:0/3739225997 >> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] conn(0x7f3cfc0373b0 0x7f3cfc037a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-05T19:28:31.182 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.174+0000 7f3d0a6fe700 1 --2- 172.21.15.167:0/3739225997 >> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] conn(0x7f3cfc0373b0 0x7f3cfc037a70 crc :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-05T19:28:31.182 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.178+0000 7f3d0a6fe700 1 -- 172.21.15.167:0/3739225997 <== osd.2 v2:172.21.15.167:6816/30037 1 ==== osd_op_reply(7 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f3cf4006070 con 0x7f3cfc0373b0 2023-10-05T19:28:31.183 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.178+0000 7f3d0a6fe700 1 --2- 172.21.15.167:0/3739225997 >> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] conn(0x7f3cf400ebe0 0x7f3cf400d8e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-05T19:28:31.183 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.178+0000 7f3d0a6fe700 1 -- 172.21.15.167:0/3739225997 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:8 5.8 5:10000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x564271347990 con 0x7f3cf400ebe0 2023-10-05T19:28:31.183 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.178+0000 7f3d0b700700 1 --2- 172.21.15.167:0/3739225997 >> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] conn(0x7f3cf400ebe0 0x7f3cf400d8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-05T19:28:31.184 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.178+0000 7f3d0b700700 1 --2- 172.21.15.167:0/3739225997 >> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] conn(0x7f3cf400ebe0 0x7f3cf400d8e0 crc :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-05T19:28:31.185 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.178+0000 7f3d0b700700 1 -- 172.21.15.167:0/3739225997 <== osd.0 v2:172.21.15.118:6800/27690 1 ==== osd_op_reply(8 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f3d04014040 con 0x7f3cf400ebe0 2023-10-05T19:28:31.185 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.178+0000 7f3d0b700700 1 -- 172.21.15.167:0/3739225997 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:9 5.18 5:18000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x564271347990 con 0x56427130e7e0 2023-10-05T19:28:31.185 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.178+0000 7f3d0b700700 1 -- 172.21.15.167:0/3739225997 <== osd.1 v2:172.21.15.167:6800/30035 6 ==== osd_op_reply(9 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f3d04014040 con 0x56427130e7e0 2023-10-05T19:28:31.185 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.178+0000 7f3d0b700700 1 -- 172.21.15.167:0/3739225997 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:10 5.4 5:20000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x564271347990 con 0x56427130e7e0 2023-10-05T19:28:31.186 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.182+0000 7f3d0b700700 1 -- 172.21.15.167:0/3739225997 <== osd.1 v2:172.21.15.167:6800/30035 7 ==== osd_op_reply(10 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f3d04014040 con 0x56427130e7e0 2023-10-05T19:28:31.186 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.182+0000 7f3d0b700700 1 -- 172.21.15.167:0/3739225997 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:11 5.14 5:28000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x564271347990 con 0x7f3cf400ebe0 2023-10-05T19:28:31.187 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.182+0000 7f3d0b700700 1 -- 172.21.15.167:0/3739225997 <== osd.0 v2:172.21.15.118:6800/27690 2 ==== osd_op_reply(11 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f3d04014040 con 0x7f3cf400ebe0 2023-10-05T19:28:31.187 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.182+0000 7f3d0b700700 1 -- 172.21.15.167:0/3739225997 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:12 5.c 5:30000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x564271347990 con 0x7f3cf400ebe0 2023-10-05T19:28:31.188 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.182+0000 7f3d0b700700 1 -- 172.21.15.167:0/3739225997 <== osd.0 v2:172.21.15.118:6800/27690 3 ==== osd_op_reply(12 [pgnls start_epoch 30 out=112b] v23'1 uv1 ondisk = 1) v8 ==== 144+0+112 (crc 0 0 0) 0x7f3d04014040 con 0x7f3cf400ebe0 2023-10-05T19:28:31.188 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.182+0000 7f3d0b700700 1 -- 172.21.15.167:0/3739225997 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:13 5.1c 5:38000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x564271347990 con 0x7f3cfc0373b0 2023-10-05T19:28:31.188 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.182+0000 7f3d0a6fe700 1 -- 172.21.15.167:0/3739225997 <== osd.2 v2:172.21.15.167:6816/30037 2 ==== osd_op_reply(13 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f3cf4006070 con 0x7f3cfc0373b0 2023-10-05T19:28:31.189 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.182+0000 7f3d0a6fe700 1 -- 172.21.15.167:0/3739225997 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:14 5.2 5:40000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x564271347990 con 0x564271345070 2023-10-05T19:28:31.189 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.182+0000 7f3d0aeff700 1 -- 172.21.15.167:0/3739225997 <== osd.3 v2:172.21.15.167:6808/30036 2 ==== osd_op_reply(14 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f3cfc0377b0 con 0x564271345070 2023-10-05T19:28:31.189 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.182+0000 7f3d0aeff700 1 -- 172.21.15.167:0/3739225997 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:15 5.12 5:48000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x564271347990 con 0x56427130e7e0 2023-10-05T19:28:31.190 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.186+0000 7f3d0b700700 1 -- 172.21.15.167:0/3739225997 <== osd.1 v2:172.21.15.167:6800/30035 8 ==== osd_op_reply(15 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f3d04014040 con 0x56427130e7e0 2023-10-05T19:28:31.190 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.186+0000 7f3d0b700700 1 -- 172.21.15.167:0/3739225997 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:16 5.a 5:50000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x564271347990 con 0x7f3cf400ebe0 2023-10-05T19:28:31.191 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.186+0000 7f3d0b700700 1 -- 172.21.15.167:0/3739225997 <== osd.0 v2:172.21.15.118:6800/27690 4 ==== osd_op_reply(16 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f3d04022040 con 0x7f3cf400ebe0 2023-10-05T19:28:31.191 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.186+0000 7f3d0b700700 1 -- 172.21.15.167:0/3739225997 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:17 5.1a 5:58000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x564271347990 con 0x56427130e7e0 2023-10-05T19:28:31.191 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.186+0000 7f3d0b700700 1 -- 172.21.15.167:0/3739225997 <== osd.1 v2:172.21.15.167:6800/30035 9 ==== osd_op_reply(17 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f3d04022040 con 0x56427130e7e0 2023-10-05T19:28:31.192 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.186+0000 7f3d0b700700 1 -- 172.21.15.167:0/3739225997 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:18 5.6 5:60000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x564271347990 con 0x7f3cfc0373b0 2023-10-05T19:28:31.192 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.186+0000 7f3d0a6fe700 1 -- 172.21.15.167:0/3739225997 <== osd.2 v2:172.21.15.167:6816/30037 3 ==== osd_op_reply(18 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f3cf4006070 con 0x7f3cfc0373b0 2023-10-05T19:28:31.192 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.186+0000 7f3d0a6fe700 1 -- 172.21.15.167:0/3739225997 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:19 5.16 5:68000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x564271347990 con 0x564271345070 2023-10-05T19:28:31.193 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.186+0000 7f3d0aeff700 1 -- 172.21.15.167:0/3739225997 <== osd.3 v2:172.21.15.167:6808/30036 3 ==== osd_op_reply(19 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f3cfc0377b0 con 0x564271345070 2023-10-05T19:28:31.193 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.186+0000 7f3d0aeff700 1 -- 172.21.15.167:0/3739225997 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:20 5.e 5:70000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x564271347990 con 0x564271345070 2023-10-05T19:28:31.194 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.190+0000 7f3d0aeff700 1 -- 172.21.15.167:0/3739225997 <== osd.3 v2:172.21.15.167:6808/30036 4 ==== osd_op_reply(20 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f3cfc0377b0 con 0x564271345070 2023-10-05T19:28:31.194 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.190+0000 7f3d0aeff700 1 -- 172.21.15.167:0/3739225997 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:21 5.1e 5:78000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x564271347990 con 0x7f3cf400ebe0 2023-10-05T19:28:31.195 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.190+0000 7f3d0b700700 1 -- 172.21.15.167:0/3739225997 <== osd.0 v2:172.21.15.118:6800/27690 5 ==== osd_op_reply(21 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f3d04022040 con 0x7f3cf400ebe0 2023-10-05T19:28:31.195 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.190+0000 7f3d0b700700 1 -- 172.21.15.167:0/3739225997 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:22 5.1 5:80000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x564271347990 con 0x564271345070 2023-10-05T19:28:31.196 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.190+0000 7f3d0aeff700 1 -- 172.21.15.167:0/3739225997 <== osd.3 v2:172.21.15.167:6808/30036 5 ==== osd_op_reply(22 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f3cfc0377b0 con 0x564271345070 2023-10-05T19:28:31.196 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.190+0000 7f3d0aeff700 1 -- 172.21.15.167:0/3739225997 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:23 5.11 5:88000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x564271347990 con 0x564271345070 2023-10-05T19:28:31.196 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.190+0000 7f3d0aeff700 1 -- 172.21.15.167:0/3739225997 <== osd.3 v2:172.21.15.167:6808/30036 6 ==== osd_op_reply(23 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f3cfc0377b0 con 0x564271345070 2023-10-05T19:28:31.197 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.190+0000 7f3d0aeff700 1 -- 172.21.15.167:0/3739225997 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:24 5.9 5:90000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x564271347990 con 0x56427130e7e0 2023-10-05T19:28:31.197 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.190+0000 7f3d0b700700 1 -- 172.21.15.167:0/3739225997 <== osd.1 v2:172.21.15.167:6800/30035 10 ==== osd_op_reply(24 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f3d04022040 con 0x56427130e7e0 2023-10-05T19:28:31.197 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.190+0000 7f3d0b700700 1 -- 172.21.15.167:0/3739225997 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:25 5.19 5:98000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x564271347990 con 0x56427130e7e0 2023-10-05T19:28:31.198 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.194+0000 7f3d0b700700 1 -- 172.21.15.167:0/3739225997 <== osd.1 v2:172.21.15.167:6800/30035 11 ==== osd_op_reply(25 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f3d04022040 con 0x56427130e7e0 2023-10-05T19:28:31.198 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.194+0000 7f3d0b700700 1 -- 172.21.15.167:0/3739225997 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:26 5.5 5:a0000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x564271347990 con 0x7f3cf400ebe0 2023-10-05T19:28:31.199 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.194+0000 7f3d0b700700 1 -- 172.21.15.167:0/3739225997 <== osd.0 v2:172.21.15.118:6800/27690 6 ==== osd_op_reply(26 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f3d04022040 con 0x7f3cf400ebe0 2023-10-05T19:28:31.199 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.194+0000 7f3d0b700700 1 -- 172.21.15.167:0/3739225997 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:27 5.15 5:a8000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x564271347990 con 0x564271345070 2023-10-05T19:28:31.200 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.194+0000 7f3d0aeff700 1 -- 172.21.15.167:0/3739225997 <== osd.3 v2:172.21.15.167:6808/30036 7 ==== osd_op_reply(27 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f3cfc0377b0 con 0x564271345070 2023-10-05T19:28:31.200 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.194+0000 7f3d0aeff700 1 -- 172.21.15.167:0/3739225997 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:28 5.d 5:b0000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x564271347990 con 0x7f3cfc0373b0 2023-10-05T19:28:31.201 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.194+0000 7f3d0a6fe700 1 -- 172.21.15.167:0/3739225997 <== osd.2 v2:172.21.15.167:6816/30037 4 ==== osd_op_reply(28 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f3cf4006070 con 0x7f3cfc0373b0 2023-10-05T19:28:31.201 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.194+0000 7f3d0a6fe700 1 -- 172.21.15.167:0/3739225997 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:29 5.1d 5:b8000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x564271347990 con 0x56427130e7e0 2023-10-05T19:28:31.202 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.194+0000 7f3d0b700700 1 -- 172.21.15.167:0/3739225997 <== osd.1 v2:172.21.15.167:6800/30035 12 ==== osd_op_reply(29 [pgnls start_epoch 30 out=107b] v30'2 uv2 ondisk = 1) v8 ==== 144+0+107 (crc 0 0 0) 0x7f3d04022040 con 0x56427130e7e0 2023-10-05T19:28:31.202 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.194+0000 7f3d0b700700 1 -- 172.21.15.167:0/3739225997 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:30 5.3 5:c0000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x564271347990 con 0x7f3cf400ebe0 2023-10-05T19:28:31.203 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.198+0000 7f3d0b700700 1 -- 172.21.15.167:0/3739225997 <== osd.0 v2:172.21.15.118:6800/27690 7 ==== osd_op_reply(30 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f3d04022040 con 0x7f3cf400ebe0 2023-10-05T19:28:31.203 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.198+0000 7f3d0b700700 1 -- 172.21.15.167:0/3739225997 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:31 5.13 5:c8000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x564271347990 con 0x564271345070 2023-10-05T19:28:31.204 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.198+0000 7f3d0aeff700 1 -- 172.21.15.167:0/3739225997 <== osd.3 v2:172.21.15.167:6808/30036 8 ==== osd_op_reply(31 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f3cfc0377b0 con 0x564271345070 2023-10-05T19:28:31.204 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.198+0000 7f3d0aeff700 1 -- 172.21.15.167:0/3739225997 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:32 5.b 5:d0000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x564271347990 con 0x7f3cf400ebe0 2023-10-05T19:28:31.205 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.198+0000 7f3d0b700700 1 -- 172.21.15.167:0/3739225997 <== osd.0 v2:172.21.15.118:6800/27690 8 ==== osd_op_reply(32 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f3d04022040 con 0x7f3cf400ebe0 2023-10-05T19:28:31.205 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.198+0000 7f3d0b700700 1 -- 172.21.15.167:0/3739225997 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:33 5.1b 5:d8000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x564271347990 con 0x7f3cfc0373b0 2023-10-05T19:28:31.205 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.198+0000 7f3d0a6fe700 1 -- 172.21.15.167:0/3739225997 <== osd.2 v2:172.21.15.167:6816/30037 5 ==== osd_op_reply(33 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f3cf4006070 con 0x7f3cfc0373b0 2023-10-05T19:28:31.206 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.198+0000 7f3d0a6fe700 1 -- 172.21.15.167:0/3739225997 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:34 5.7 5:e0000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x564271347990 con 0x564271345070 2023-10-05T19:28:31.206 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.202+0000 7f3d0aeff700 1 -- 172.21.15.167:0/3739225997 <== osd.3 v2:172.21.15.167:6808/30036 9 ==== osd_op_reply(34 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f3cfc0377b0 con 0x564271345070 2023-10-05T19:28:31.206 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.202+0000 7f3d0aeff700 1 -- 172.21.15.167:0/3739225997 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:35 5.17 5:e8000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x564271347990 con 0x7f3cf400ebe0 2023-10-05T19:28:31.207 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.202+0000 7f3d0b700700 1 -- 172.21.15.167:0/3739225997 <== osd.0 v2:172.21.15.118:6800/27690 9 ==== osd_op_reply(35 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f3d04022040 con 0x7f3cf400ebe0 2023-10-05T19:28:31.207 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.202+0000 7f3d0b700700 1 -- 172.21.15.167:0/3739225997 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:36 5.f 5:f0000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x564271347990 con 0x56427130e7e0 2023-10-05T19:28:31.208 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.202+0000 7f3d0b700700 1 -- 172.21.15.167:0/3739225997 <== osd.1 v2:172.21.15.167:6800/30035 13 ==== osd_op_reply(36 [pgnls start_epoch 30 out=115b] v23'2 uv2 ondisk = 1) v8 ==== 144+0+115 (crc 0 0 0) 0x7f3d0402f080 con 0x56427130e7e0 2023-10-05T19:28:31.208 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.202+0000 7f3d0b700700 1 -- 172.21.15.167:0/3739225997 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:37 5.1f 5:f8000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x564271347990 con 0x7f3cfc0373b0 2023-10-05T19:28:31.209 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.202+0000 7f3d0a6fe700 1 -- 172.21.15.167:0/3739225997 <== osd.2 v2:172.21.15.167:6816/30037 6 ==== osd_op_reply(37 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f3cf4006070 con 0x7f3cfc0373b0 2023-10-05T19:28:31.209 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.202+0000 7f3d0d3cae40 20 RGWRados::pool_iterate: got zonegroup_info.4f1f719d-9648-417b-b10c-e5d939825419 2023-10-05T19:28:31.209 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.202+0000 7f3d0d3cae40 20 RGWRados::pool_iterate: got zone_info.9a9df965-6612-45d4-a45f-6e3c889bacc2 2023-10-05T19:28:31.209 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.202+0000 7f3d0d3cae40 20 RGWRados::pool_iterate: got zone_names.default 2023-10-05T19:28:31.210 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.202+0000 7f3d0d3cae40 20 RGWRados::pool_iterate: got zonegroups_names.default 2023-10-05T19:28:31.210 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.202+0000 7f3d0d3cae40 20 rados->read ofs=0 len=0 2023-10-05T19:28:31.210 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.202+0000 7f3d0d3cae40 1 -- 172.21.15.167:0/3739225997 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:38 5.12 5:49953fa1:::default.realm:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x564271343e20 con 0x56427130e7e0 2023-10-05T19:28:31.211 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.202+0000 7f3d0b700700 1 -- 172.21.15.167:0/3739225997 <== osd.1 v2:172.21.15.167:6800/30035 14 ==== osd_op_reply(38 default.realm [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 ==== 157+0+0 (crc 0 0 0) 0x7f3d0402f080 con 0x56427130e7e0 2023-10-05T19:28:31.211 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.202+0000 7f3d0d3cae40 20 rados_obj.operate() r=-2 bl.length=0 2023-10-05T19:28:31.211 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.206+0000 7f3d0d3cae40 20 rados->read ofs=0 len=0 2023-10-05T19:28:31.211 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.206+0000 7f3d0d3cae40 1 -- 172.21.15.167:0/3739225997 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:39 5.f 5:f4c53578:::zonegroups_names.default:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x5642713447a0 con 0x56427130e7e0 2023-10-05T19:28:31.211 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.206+0000 7f3d0b700700 1 -- 172.21.15.167:0/3739225997 <== osd.1 v2:172.21.15.167:6800/30035 15 ==== osd_op_reply(39 zonegroups_names.default [read 0~46 out=46b] v0'0 uv2 ondisk = 0) v8 ==== 168+0+46 (crc 0 0 0) 0x7f3d0402f080 con 0x56427130e7e0 2023-10-05T19:28:31.211 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.206+0000 7f3d0d3cae40 20 rados_obj.operate() r=0 bl.length=46 2023-10-05T19:28:31.212 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.206+0000 7f3d0d3cae40 20 rados->read ofs=0 len=0 2023-10-05T19:28:31.212 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.206+0000 7f3d0d3cae40 1 -- 172.21.15.167:0/3739225997 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:40 5.c 5:30d299a5:::zonegroup_info.4f1f719d-9648-417b-b10c-e5d939825419:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x564271343a10 con 0x7f3cf400ebe0 2023-10-05T19:28:31.213 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.206+0000 7f3d0b700700 1 -- 172.21.15.167:0/3739225997 <== osd.0 v2:172.21.15.118:6800/27690 10 ==== osd_op_reply(40 zonegroup_info.4f1f719d-9648-417b-b10c-e5d939825419 [read 0~362 out=362b] v0'0 uv1 ondisk = 0) v8 ==== 195+0+362 (crc 0 0 0) 0x7f3d0402f080 con 0x7f3cf400ebe0 2023-10-05T19:28:31.213 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.206+0000 7f3d0d3cae40 20 rados_obj.operate() r=0 bl.length=362 2023-10-05T19:28:31.213 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.206+0000 7f3d0d3cae40 20 rados->read ofs=0 len=0 2023-10-05T19:28:31.213 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.206+0000 7f3d0d3cae40 1 -- 172.21.15.167:0/3739225997 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:41 5.13 5:cbbf6c60:::converted:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x564271348280 con 0x564271345070 2023-10-05T19:28:31.214 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.206+0000 7f3d0aeff700 1 -- 172.21.15.167:0/3739225997 <== osd.3 v2:172.21.15.167:6808/30036 10 ==== osd_op_reply(41 converted [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 ==== 153+0+0 (crc 0 0 0) 0x7f3cfc053040 con 0x564271345070 2023-10-05T19:28:31.214 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.206+0000 7f3d0d3cae40 20 rados_obj.operate() r=-2 bl.length=0 2023-10-05T19:28:31.214 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.206+0000 7f3d0d3cae40 20 rados->read ofs=0 len=0 2023-10-05T19:28:31.214 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.206+0000 7f3d0d3cae40 1 -- 172.21.15.167:0/3739225997 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:42 5.12 5:49953fa1:::default.realm:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x564271348630 con 0x56427130e7e0 2023-10-05T19:28:31.214 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.206+0000 7f3d0b700700 1 -- 172.21.15.167:0/3739225997 <== osd.1 v2:172.21.15.167:6800/30035 16 ==== osd_op_reply(42 default.realm [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 ==== 157+0+0 (crc 0 0 0) 0x7f3d0402f080 con 0x56427130e7e0 2023-10-05T19:28:31.214 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.206+0000 7f3d0d3cae40 20 rados_obj.operate() r=-2 bl.length=0 2023-10-05T19:28:31.217 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.206+0000 7f3d0d3cae40 20 rados->read ofs=0 len=0 2023-10-05T19:28:31.217 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.206+0000 7f3d0d3cae40 1 -- 172.21.15.167:0/3739225997 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:43 5.f 5:f4c53578:::zonegroups_names.default:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x564271348630 con 0x56427130e7e0 2023-10-05T19:28:31.218 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.206+0000 7f3d0b700700 1 -- 172.21.15.167:0/3739225997 <== osd.1 v2:172.21.15.167:6800/30035 17 ==== osd_op_reply(43 zonegroups_names.default [read 0~46 out=46b] v0'0 uv2 ondisk = 0) v8 ==== 168+0+46 (crc 0 0 0) 0x7f3d0402f080 con 0x56427130e7e0 2023-10-05T19:28:31.218 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.206+0000 7f3d0d3cae40 20 rados_obj.operate() r=0 bl.length=46 2023-10-05T19:28:31.219 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.206+0000 7f3d0d3cae40 1 -- 172.21.15.167:0/3739225997 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:44 5.0 5:00000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x564271349640 con 0x564271345070 2023-10-05T19:28:31.219 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.210+0000 7f3d0aeff700 1 -- 172.21.15.167:0/3739225997 <== osd.3 v2:172.21.15.167:6808/30036 11 ==== osd_op_reply(44 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f3cfc053040 con 0x564271345070 2023-10-05T19:28:31.219 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.210+0000 7f3d0aeff700 1 -- 172.21.15.167:0/3739225997 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:45 5.10 5:08000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x564271349640 con 0x7f3cfc0373b0 2023-10-05T19:28:31.219 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.210+0000 7f3d0a6fe700 1 -- 172.21.15.167:0/3739225997 <== osd.2 v2:172.21.15.167:6816/30037 7 ==== osd_op_reply(45 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f3cf4006070 con 0x7f3cfc0373b0 2023-10-05T19:28:31.219 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.210+0000 7f3d0a6fe700 1 -- 172.21.15.167:0/3739225997 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:46 5.8 5:10000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x564271349640 con 0x7f3cf400ebe0 2023-10-05T19:28:31.219 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.214+0000 7f3d0b700700 1 -- 172.21.15.167:0/3739225997 <== osd.0 v2:172.21.15.118:6800/27690 11 ==== osd_op_reply(46 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f3d0402f080 con 0x7f3cf400ebe0 2023-10-05T19:28:31.219 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.214+0000 7f3d0b700700 1 -- 172.21.15.167:0/3739225997 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:47 5.18 5:18000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x564271343a10 con 0x56427130e7e0 2023-10-05T19:28:31.220 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.214+0000 7f3d0b700700 1 -- 172.21.15.167:0/3739225997 <== osd.1 v2:172.21.15.167:6800/30035 18 ==== osd_op_reply(47 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f3d0402f080 con 0x56427130e7e0 2023-10-05T19:28:31.220 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.214+0000 7f3d0b700700 1 -- 172.21.15.167:0/3739225997 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:48 5.4 5:20000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x564271343a10 con 0x56427130e7e0 2023-10-05T19:28:31.220 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.214+0000 7f3d0b700700 1 -- 172.21.15.167:0/3739225997 <== osd.1 v2:172.21.15.167:6800/30035 19 ==== osd_op_reply(48 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f3d0402f080 con 0x56427130e7e0 2023-10-05T19:28:31.220 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.214+0000 7f3d0b700700 1 -- 172.21.15.167:0/3739225997 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:49 5.14 5:28000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x564271343a10 con 0x7f3cf400ebe0 2023-10-05T19:28:31.221 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.214+0000 7f3d0b700700 1 -- 172.21.15.167:0/3739225997 <== osd.0 v2:172.21.15.118:6800/27690 12 ==== osd_op_reply(49 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f3d0402f080 con 0x7f3cf400ebe0 2023-10-05T19:28:31.221 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.214+0000 7f3d0b700700 1 -- 172.21.15.167:0/3739225997 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:50 5.c 5:30000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x564271343a10 con 0x7f3cf400ebe0 2023-10-05T19:28:31.222 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.214+0000 7f3d0b700700 1 -- 172.21.15.167:0/3739225997 <== osd.0 v2:172.21.15.118:6800/27690 13 ==== osd_op_reply(50 [pgnls start_epoch 30 out=112b] v23'1 uv1 ondisk = 1) v8 ==== 144+0+112 (crc 0 0 0) 0x7f3d0402f080 con 0x7f3cf400ebe0 2023-10-05T19:28:31.222 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.214+0000 7f3d0b700700 1 -- 172.21.15.167:0/3739225997 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:51 5.1c 5:38000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x564271343a10 con 0x7f3cfc0373b0 2023-10-05T19:28:31.222 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.218+0000 7f3d0a6fe700 1 -- 172.21.15.167:0/3739225997 <== osd.2 v2:172.21.15.167:6816/30037 8 ==== osd_op_reply(51 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f3cf4006070 con 0x7f3cfc0373b0 2023-10-05T19:28:31.223 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.218+0000 7f3d0a6fe700 1 -- 172.21.15.167:0/3739225997 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:52 5.2 5:40000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x564271343a10 con 0x564271345070 2023-10-05T19:28:31.223 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.218+0000 7f3d0aeff700 1 -- 172.21.15.167:0/3739225997 <== osd.3 v2:172.21.15.167:6808/30036 12 ==== osd_op_reply(52 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f3cfc053040 con 0x564271345070 2023-10-05T19:28:31.223 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.218+0000 7f3d0aeff700 1 -- 172.21.15.167:0/3739225997 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:53 5.12 5:48000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x564271343a10 con 0x56427130e7e0 2023-10-05T19:28:31.224 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.218+0000 7f3d0b700700 1 -- 172.21.15.167:0/3739225997 <== osd.1 v2:172.21.15.167:6800/30035 20 ==== osd_op_reply(53 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f3d0402f080 con 0x56427130e7e0 2023-10-05T19:28:31.224 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.218+0000 7f3d0b700700 1 -- 172.21.15.167:0/3739225997 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:54 5.a 5:50000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x564271343a10 con 0x7f3cf400ebe0 2023-10-05T19:28:31.225 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.218+0000 7f3d0b700700 1 -- 172.21.15.167:0/3739225997 <== osd.0 v2:172.21.15.118:6800/27690 14 ==== osd_op_reply(54 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f3d0402f080 con 0x7f3cf400ebe0 2023-10-05T19:28:31.225 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.218+0000 7f3d0b700700 1 -- 172.21.15.167:0/3739225997 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:55 5.1a 5:58000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x564271343a10 con 0x56427130e7e0 2023-10-05T19:28:31.226 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.222+0000 7f3d0b700700 1 -- 172.21.15.167:0/3739225997 <== osd.1 v2:172.21.15.167:6800/30035 21 ==== osd_op_reply(55 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f3d0402f080 con 0x56427130e7e0 2023-10-05T19:28:31.226 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.222+0000 7f3d0b700700 1 -- 172.21.15.167:0/3739225997 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:56 5.6 5:60000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x564271343a10 con 0x7f3cfc0373b0 2023-10-05T19:28:31.227 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.222+0000 7f3d0a6fe700 1 -- 172.21.15.167:0/3739225997 <== osd.2 v2:172.21.15.167:6816/30037 9 ==== osd_op_reply(56 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f3cf4006070 con 0x7f3cfc0373b0 2023-10-05T19:28:31.227 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.222+0000 7f3d0a6fe700 1 -- 172.21.15.167:0/3739225997 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:57 5.16 5:68000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x564271343a10 con 0x564271345070 2023-10-05T19:28:31.228 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.222+0000 7f3d0aeff700 1 -- 172.21.15.167:0/3739225997 <== osd.3 v2:172.21.15.167:6808/30036 13 ==== osd_op_reply(57 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f3cfc053040 con 0x564271345070 2023-10-05T19:28:31.228 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.222+0000 7f3d0aeff700 1 -- 172.21.15.167:0/3739225997 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:58 5.e 5:70000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x564271343a10 con 0x564271345070 2023-10-05T19:28:31.228 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.222+0000 7f3d0aeff700 1 -- 172.21.15.167:0/3739225997 <== osd.3 v2:172.21.15.167:6808/30036 14 ==== osd_op_reply(58 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f3cfc053040 con 0x564271345070 2023-10-05T19:28:31.228 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.222+0000 7f3d0aeff700 1 -- 172.21.15.167:0/3739225997 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:59 5.1e 5:78000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x564271343a10 con 0x7f3cf400ebe0 2023-10-05T19:28:31.230 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.222+0000 7f3d0b700700 1 -- 172.21.15.167:0/3739225997 <== osd.0 v2:172.21.15.118:6800/27690 15 ==== osd_op_reply(59 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f3d0402f080 con 0x7f3cf400ebe0 2023-10-05T19:28:31.230 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.222+0000 7f3d0b700700 1 -- 172.21.15.167:0/3739225997 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:60 5.1 5:80000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x564271343a10 con 0x564271345070 2023-10-05T19:28:31.230 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.226+0000 7f3d0aeff700 1 -- 172.21.15.167:0/3739225997 <== osd.3 v2:172.21.15.167:6808/30036 15 ==== osd_op_reply(60 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f3cfc053040 con 0x564271345070 2023-10-05T19:28:31.230 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.226+0000 7f3d0aeff700 1 -- 172.21.15.167:0/3739225997 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:61 5.11 5:88000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x564271343a10 con 0x564271345070 2023-10-05T19:28:31.231 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.226+0000 7f3d0aeff700 1 -- 172.21.15.167:0/3739225997 <== osd.3 v2:172.21.15.167:6808/30036 16 ==== osd_op_reply(61 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f3cfc053040 con 0x564271345070 2023-10-05T19:28:31.231 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.226+0000 7f3d0aeff700 1 -- 172.21.15.167:0/3739225997 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:62 5.9 5:90000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x564271343a10 con 0x56427130e7e0 2023-10-05T19:28:31.232 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.226+0000 7f3d0b700700 1 -- 172.21.15.167:0/3739225997 <== osd.1 v2:172.21.15.167:6800/30035 22 ==== osd_op_reply(62 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f3d0402f080 con 0x56427130e7e0 2023-10-05T19:28:31.232 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.226+0000 7f3d0b700700 1 -- 172.21.15.167:0/3739225997 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:63 5.19 5:98000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x564271343a10 con 0x56427130e7e0 2023-10-05T19:28:31.233 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.226+0000 7f3d0b700700 1 -- 172.21.15.167:0/3739225997 <== osd.1 v2:172.21.15.167:6800/30035 23 ==== osd_op_reply(63 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f3d0402f080 con 0x56427130e7e0 2023-10-05T19:28:31.233 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.226+0000 7f3d0b700700 1 -- 172.21.15.167:0/3739225997 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:64 5.5 5:a0000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x564271343a10 con 0x7f3cf400ebe0 2023-10-05T19:28:31.234 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.226+0000 7f3d0b700700 1 -- 172.21.15.167:0/3739225997 <== osd.0 v2:172.21.15.118:6800/27690 16 ==== osd_op_reply(64 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f3d0402f080 con 0x7f3cf400ebe0 2023-10-05T19:28:31.234 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.226+0000 7f3d0b700700 1 -- 172.21.15.167:0/3739225997 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:65 5.15 5:a8000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x564271343a10 con 0x564271345070 2023-10-05T19:28:31.234 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.230+0000 7f3d0aeff700 1 -- 172.21.15.167:0/3739225997 <== osd.3 v2:172.21.15.167:6808/30036 17 ==== osd_op_reply(65 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f3cfc053040 con 0x564271345070 2023-10-05T19:28:31.234 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.230+0000 7f3d0aeff700 1 -- 172.21.15.167:0/3739225997 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:66 5.d 5:b0000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x564271343a10 con 0x7f3cfc0373b0 2023-10-05T19:28:31.235 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.230+0000 7f3d0a6fe700 1 -- 172.21.15.167:0/3739225997 <== osd.2 v2:172.21.15.167:6816/30037 10 ==== osd_op_reply(66 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f3cf4006070 con 0x7f3cfc0373b0 2023-10-05T19:28:31.235 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.230+0000 7f3d0a6fe700 1 -- 172.21.15.167:0/3739225997 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:67 5.1d 5:b8000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x564271343a10 con 0x56427130e7e0 2023-10-05T19:28:31.236 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.230+0000 7f3d0b700700 1 -- 172.21.15.167:0/3739225997 <== osd.1 v2:172.21.15.167:6800/30035 24 ==== osd_op_reply(67 [pgnls start_epoch 30 out=107b] v30'2 uv2 ondisk = 1) v8 ==== 144+0+107 (crc 0 0 0) 0x7f3d0402f080 con 0x56427130e7e0 2023-10-05T19:28:31.236 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.230+0000 7f3d0b700700 1 -- 172.21.15.167:0/3739225997 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:68 5.3 5:c0000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x564271343a10 con 0x7f3cf400ebe0 2023-10-05T19:28:31.237 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.230+0000 7f3d0b700700 1 -- 172.21.15.167:0/3739225997 <== osd.0 v2:172.21.15.118:6800/27690 17 ==== osd_op_reply(68 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f3d0402f080 con 0x7f3cf400ebe0 2023-10-05T19:28:31.237 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.230+0000 7f3d0b700700 1 -- 172.21.15.167:0/3739225997 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:69 5.13 5:c8000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x564271343a10 con 0x564271345070 2023-10-05T19:28:31.238 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.234+0000 7f3d0aeff700 1 -- 172.21.15.167:0/3739225997 <== osd.3 v2:172.21.15.167:6808/30036 18 ==== osd_op_reply(69 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f3cfc053040 con 0x564271345070 2023-10-05T19:28:31.238 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.234+0000 7f3d0aeff700 1 -- 172.21.15.167:0/3739225997 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:70 5.b 5:d0000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x564271343a10 con 0x7f3cf400ebe0 2023-10-05T19:28:31.239 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.234+0000 7f3d0b700700 1 -- 172.21.15.167:0/3739225997 <== osd.0 v2:172.21.15.118:6800/27690 18 ==== osd_op_reply(70 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f3d0402f080 con 0x7f3cf400ebe0 2023-10-05T19:28:31.239 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.234+0000 7f3d0b700700 1 -- 172.21.15.167:0/3739225997 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:71 5.1b 5:d8000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x564271343a10 con 0x7f3cfc0373b0 2023-10-05T19:28:31.240 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.234+0000 7f3d0a6fe700 1 -- 172.21.15.167:0/3739225997 <== osd.2 v2:172.21.15.167:6816/30037 11 ==== osd_op_reply(71 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f3cf4006070 con 0x7f3cfc0373b0 2023-10-05T19:28:31.240 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.234+0000 7f3d0a6fe700 1 -- 172.21.15.167:0/3739225997 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:72 5.7 5:e0000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x564271343a10 con 0x564271345070 2023-10-05T19:28:31.240 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.234+0000 7f3d0aeff700 1 -- 172.21.15.167:0/3739225997 <== osd.3 v2:172.21.15.167:6808/30036 19 ==== osd_op_reply(72 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f3cfc053040 con 0x564271345070 2023-10-05T19:28:31.240 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.234+0000 7f3d0aeff700 1 -- 172.21.15.167:0/3739225997 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:73 5.17 5:e8000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x564271343a10 con 0x7f3cf400ebe0 2023-10-05T19:28:31.241 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.234+0000 7f3d0b700700 1 -- 172.21.15.167:0/3739225997 <== osd.0 v2:172.21.15.118:6800/27690 19 ==== osd_op_reply(73 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f3d0402f080 con 0x7f3cf400ebe0 2023-10-05T19:28:31.241 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.234+0000 7f3d0b700700 1 -- 172.21.15.167:0/3739225997 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:74 5.f 5:f0000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x564271343a10 con 0x56427130e7e0 2023-10-05T19:28:31.242 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.238+0000 7f3d0b700700 1 -- 172.21.15.167:0/3739225997 <== osd.1 v2:172.21.15.167:6800/30035 25 ==== osd_op_reply(74 [pgnls start_epoch 30 out=115b] v23'2 uv2 ondisk = 1) v8 ==== 144+0+115 (crc 0 0 0) 0x7f3d0402f080 con 0x56427130e7e0 2023-10-05T19:28:31.242 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.238+0000 7f3d0b700700 1 -- 172.21.15.167:0/3739225997 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:75 5.1f 5:f8000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x564271343a10 con 0x7f3cfc0373b0 2023-10-05T19:28:31.243 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.238+0000 7f3d0a6fe700 1 -- 172.21.15.167:0/3739225997 <== osd.2 v2:172.21.15.167:6816/30037 12 ==== osd_op_reply(75 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f3cf4006070 con 0x7f3cfc0373b0 2023-10-05T19:28:31.244 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.238+0000 7f3d0d3cae40 20 RGWRados::pool_iterate: got zonegroup_info.4f1f719d-9648-417b-b10c-e5d939825419 2023-10-05T19:28:31.244 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.238+0000 7f3d0d3cae40 20 RGWRados::pool_iterate: got zone_info.9a9df965-6612-45d4-a45f-6e3c889bacc2 2023-10-05T19:28:31.244 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.238+0000 7f3d0d3cae40 20 RGWRados::pool_iterate: got zone_names.default 2023-10-05T19:28:31.244 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.238+0000 7f3d0d3cae40 20 RGWRados::pool_iterate: got zonegroups_names.default 2023-10-05T19:28:31.244 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.238+0000 7f3d0d3cae40 20 rados->read ofs=0 len=0 2023-10-05T19:28:31.244 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.238+0000 7f3d0d3cae40 1 -- 172.21.15.167:0/3739225997 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:76 5.f 5:f43b6ece:::zone_names.default:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x564271348630 con 0x56427130e7e0 2023-10-05T19:28:31.245 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.238+0000 7f3d0b700700 1 -- 172.21.15.167:0/3739225997 <== osd.1 v2:172.21.15.167:6800/30035 26 ==== osd_op_reply(76 zone_names.default [read 0~46 out=46b] v0'0 uv1 ondisk = 0) v8 ==== 162+0+46 (crc 0 0 0) 0x7f3d0402f080 con 0x56427130e7e0 2023-10-05T19:28:31.245 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.238+0000 7f3d0d3cae40 20 rados_obj.operate() r=0 bl.length=46 2023-10-05T19:28:31.245 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.238+0000 7f3d0d3cae40 20 rados->read ofs=0 len=0 2023-10-05T19:28:31.245 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.238+0000 7f3d0d3cae40 1 -- 172.21.15.167:0/3739225997 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:77 5.1d 5:b8374eb4:::zone_info.9a9df965-6612-45d4-a45f-6e3c889bacc2:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x5642713489e0 con 0x56427130e7e0 2023-10-05T19:28:31.245 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.238+0000 7f3d0b700700 1 -- 172.21.15.167:0/3739225997 <== osd.1 v2:172.21.15.167:6800/30035 27 ==== osd_op_reply(77 zone_info.9a9df965-6612-45d4-a45f-6e3c889bacc2 [read 0~874 out=874b] v0'0 uv2 ondisk = 0) v8 ==== 190+0+874 (crc 0 0 0) 0x7f3d0402f080 con 0x56427130e7e0 2023-10-05T19:28:31.245 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.238+0000 7f3d0d3cae40 20 rados_obj.operate() r=0 bl.length=874 2023-10-05T19:28:31.245 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.238+0000 7f3d0d3cae40 20 rados->read ofs=0 len=0 2023-10-05T19:28:31.245 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.238+0000 7f3d0d3cae40 1 -- 172.21.15.167:0/3739225997 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:78 5.f 5:f4c53578:::zonegroups_names.default:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x564271348ed0 con 0x56427130e7e0 2023-10-05T19:28:31.246 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.238+0000 7f3d0b700700 1 -- 172.21.15.167:0/3739225997 <== osd.1 v2:172.21.15.167:6800/30035 28 ==== osd_op_reply(78 zonegroups_names.default [read 0~46 out=46b] v0'0 uv2 ondisk = 0) v8 ==== 168+0+46 (crc 0 0 0) 0x7f3d0402f080 con 0x56427130e7e0 2023-10-05T19:28:31.246 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.238+0000 7f3d0d3cae40 20 rados_obj.operate() r=0 bl.length=46 2023-10-05T19:28:31.246 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.238+0000 7f3d0d3cae40 20 rados->read ofs=0 len=0 2023-10-05T19:28:31.246 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.238+0000 7f3d0d3cae40 1 -- 172.21.15.167:0/3739225997 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:79 5.c 5:30d299a5:::zonegroup_info.4f1f719d-9648-417b-b10c-e5d939825419:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x564271348ed0 con 0x7f3cf400ebe0 2023-10-05T19:28:31.247 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.242+0000 7f3d0b700700 1 -- 172.21.15.167:0/3739225997 <== osd.0 v2:172.21.15.118:6800/27690 20 ==== osd_op_reply(79 zonegroup_info.4f1f719d-9648-417b-b10c-e5d939825419 [read 0~362 out=362b] v0'0 uv1 ondisk = 0) v8 ==== 195+0+362 (crc 0 0 0) 0x7f3d0402f080 con 0x7f3cf400ebe0 2023-10-05T19:28:31.247 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.242+0000 7f3d0d3cae40 20 rados_obj.operate() r=0 bl.length=362 2023-10-05T19:28:31.247 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.242+0000 7f3d0d3cae40 20 rados->read ofs=0 len=0 2023-10-05T19:28:31.247 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.242+0000 7f3d0d3cae40 1 -- 172.21.15.167:0/3739225997 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:80 5.3 5:c0ad8cbd:::region_map:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x564271348fd0 con 0x7f3cf400ebe0 2023-10-05T19:28:31.248 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.242+0000 7f3d0b700700 1 -- 172.21.15.167:0/3739225997 <== osd.0 v2:172.21.15.118:6800/27690 21 ==== osd_op_reply(80 region_map [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 ==== 154+0+0 (crc 0 0 0) 0x7f3d04050040 con 0x7f3cf400ebe0 2023-10-05T19:28:31.248 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.242+0000 7f3d0d3cae40 20 rados_obj.operate() r=-2 bl.length=0 2023-10-05T19:28:31.248 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.242+0000 7f3d0d3cae40 20 zone default found 2023-10-05T19:28:31.248 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.242+0000 7f3d0d3cae40 4 Realm: () 2023-10-05T19:28:31.248 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.242+0000 7f3d0d3cae40 4 ZoneGroup: default (4f1f719d-9648-417b-b10c-e5d939825419) 2023-10-05T19:28:31.248 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.242+0000 7f3d0d3cae40 4 Zone: default (9a9df965-6612-45d4-a45f-6e3c889bacc2) 2023-10-05T19:28:31.248 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.242+0000 7f3d0d3cae40 10 cannot find current period zonegroup using local zonegroup configuration 2023-10-05T19:28:31.248 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.242+0000 7f3d0d3cae40 20 zonegroup default 2023-10-05T19:28:31.249 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.242+0000 7f3d0d3cae40 20 rados->read ofs=0 len=0 2023-10-05T19:28:31.249 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.242+0000 7f3d0d3cae40 1 -- 172.21.15.167:0/3739225997 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:81 5.3 5:c52100b6:::period_config.default:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x564271348fd0 con 0x7f3cf400ebe0 2023-10-05T19:28:31.250 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.242+0000 7f3d0b700700 1 -- 172.21.15.167:0/3739225997 <== osd.0 v2:172.21.15.118:6800/27690 22 ==== osd_op_reply(81 period_config.default [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 ==== 165+0+0 (crc 0 0 0) 0x7f3d04050040 con 0x7f3cf400ebe0 2023-10-05T19:28:31.250 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.242+0000 7f3d0d3cae40 20 rados_obj.operate() r=-2 bl.length=0 2023-10-05T19:28:31.250 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.242+0000 7f3d0d3cae40 20 rados->read ofs=0 len=0 2023-10-05T19:28:31.250 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.242+0000 7f3d0d3cae40 1 -- 172.21.15.167:0/3739225997 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:82 6.2 6:4347d321:::bucket.sync-source-hints.:head [call version.read in=11b,read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x56427134a950 con 0x564271345070 2023-10-05T19:28:31.250 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.242+0000 7f3d0aeff700 1 -- 172.21.15.167:0/3739225997 <== osd.3 v2:172.21.15.167:6808/30036 20 ==== osd_op_reply(82 bucket.sync-source-hints. [call,read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 ==== 211+0+0 (crc 0 0 0) 0x7f3cfc053040 con 0x564271345070 2023-10-05T19:28:31.250 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.242+0000 7f3d0d3cae40 20 rados_obj.operate() r=-2 bl.length=0 2023-10-05T19:28:31.250 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.242+0000 7f3d0d3cae40 20 rados->read ofs=0 len=0 2023-10-05T19:28:31.250 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.242+0000 7f3d0d3cae40 1 -- 172.21.15.167:0/3739225997 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:83 6.b 6:d467b91b:::bucket.sync-target-hints.:head [call version.read in=11b,read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x56427134ad00 con 0x564271345070 2023-10-05T19:28:31.251 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.246+0000 7f3d0aeff700 1 -- 172.21.15.167:0/3739225997 <== osd.3 v2:172.21.15.167:6808/30036 21 ==== osd_op_reply(83 bucket.sync-target-hints. [call,read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 ==== 211+0+0 (crc 0 0 0) 0x7f3cfc053040 con 0x564271345070 2023-10-05T19:28:31.251 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.246+0000 7f3d0d3cae40 20 rados_obj.operate() r=-2 bl.length=0 2023-10-05T19:28:31.251 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.246+0000 7f3d0d3cae40 20 started sync module instance, tier type = 2023-10-05T19:28:31.251 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.246+0000 7f3d0d3cae40 20 started zone id=9a9df965-6612-45d4-a45f-6e3c889bacc2 (name=default) with tier type = 2023-10-05T19:28:31.251 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.246+0000 7f3d0d3cae40 1 -- 172.21.15.167:0/3739225997 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:84 7.1f 7:f95f44c2:::notify.0:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x56427134a010 con 0x7f3cfc0373b0 2023-10-05T19:28:31.253 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.246+0000 7f3d0a6fe700 1 -- 172.21.15.167:0/3739225997 <== osd.2 v2:172.21.15.167:6816/30037 13 ==== osd_op_reply(84 notify.0 [create] v30'93 uv93 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f3cf4006070 con 0x7f3cfc0373b0 2023-10-05T19:28:31.253 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.246+0000 7f3d0d3cae40 1 -- 172.21.15.167:0/3739225997 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:85 7.1f 7:f95f44c2:::notify.0:head [watch watch cookie 94843367105552] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x564271349c40 con 0x7f3cfc0373b0 2023-10-05T19:28:31.254 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.246+0000 7f3d0d3cae40 1 -- 172.21.15.167:0/3739225997 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:86 7.0 7:05bf5b68:::notify.1:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x56427134ba90 con 0x56427130e7e0 2023-10-05T19:28:31.256 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.250+0000 7f3d0a6fe700 1 -- 172.21.15.167:0/3739225997 <== osd.2 v2:172.21.15.167:6816/30037 14 ==== osd_op_reply(85 notify.0 [watch watch cookie 94843367105552] v30'94 uv93 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f3cf4006070 con 0x7f3cfc0373b0 2023-10-05T19:28:31.256 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.250+0000 7f3d0b700700 1 -- 172.21.15.167:0/3739225997 <== osd.1 v2:172.21.15.167:6800/30035 29 ==== osd_op_reply(86 notify.1 [create] v30'93 uv93 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f3d04050040 con 0x56427130e7e0 2023-10-05T19:28:31.257 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.250+0000 7f3d0d3cae40 1 -- 172.21.15.167:0/3739225997 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:87 7.0 7:05bf5b68:::notify.1:head [watch watch cookie 94843367109072] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x56427134be90 con 0x56427130e7e0 2023-10-05T19:28:31.257 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.250+0000 7f3d0d3cae40 1 -- 172.21.15.167:0/3739225997 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:88 7.15 7:a93a5511:::notify.2:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x56427134c730 con 0x564271345070 2023-10-05T19:28:31.260 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.254+0000 7f3d0aeff700 1 -- 172.21.15.167:0/3739225997 <== osd.3 v2:172.21.15.167:6808/30036 22 ==== osd_op_reply(88 notify.2 [create] v30'93 uv93 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f3cfc053040 con 0x564271345070 2023-10-05T19:28:31.260 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.254+0000 7f3d0d3cae40 1 -- 172.21.15.167:0/3739225997 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:89 7.15 7:a93a5511:::notify.2:head [watch watch cookie 94843367112336] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x56427134c360 con 0x564271345070 2023-10-05T19:28:31.260 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.254+0000 7f3d0d3cae40 1 -- 172.21.15.167:0/3739225997 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:90 7.e 7:7759931f:::notify.3:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x56427134d7e0 con 0x7f3cfc0373b0 2023-10-05T19:28:31.260 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.254+0000 7f3d0b700700 1 -- 172.21.15.167:0/3739225997 <== osd.1 v2:172.21.15.167:6800/30035 30 ==== osd_op_reply(87 notify.1 [watch watch cookie 94843367109072] v30'94 uv93 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f3d04050040 con 0x56427130e7e0 2023-10-05T19:28:31.262 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.258+0000 7f3d0aeff700 1 -- 172.21.15.167:0/3739225997 <== osd.3 v2:172.21.15.167:6808/30036 23 ==== osd_op_reply(89 notify.2 [watch watch cookie 94843367112336] v30'94 uv93 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f3cfc053040 con 0x564271345070 2023-10-05T19:28:31.262 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.258+0000 7f3d0a6fe700 1 -- 172.21.15.167:0/3739225997 <== osd.2 v2:172.21.15.167:6816/30037 15 ==== osd_op_reply(90 notify.3 [create] v30'93 uv93 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f3cf4006070 con 0x7f3cfc0373b0 2023-10-05T19:28:31.263 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.258+0000 7f3d0d3cae40 1 -- 172.21.15.167:0/3739225997 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:91 7.e 7:7759931f:::notify.3:head [watch watch cookie 94843367110176] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x56427134cc00 con 0x7f3cfc0373b0 2023-10-05T19:28:31.263 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.258+0000 7f3d0d3cae40 1 -- 172.21.15.167:0/3739225997 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:92 7.d 7:b4812045:::notify.4:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x56427134e2b0 con 0x56427130e7e0 2023-10-05T19:28:31.264 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.258+0000 7f3d0a6fe700 1 -- 172.21.15.167:0/3739225997 <== osd.2 v2:172.21.15.167:6816/30037 16 ==== osd_op_reply(91 notify.3 [watch watch cookie 94843367110176] v30'94 uv93 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f3cf4006070 con 0x7f3cfc0373b0 2023-10-05T19:28:31.265 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.258+0000 7f3d0b700700 1 -- 172.21.15.167:0/3739225997 <== osd.1 v2:172.21.15.167:6800/30035 31 ==== osd_op_reply(92 notify.4 [create] v30'93 uv93 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f3d04050040 con 0x56427130e7e0 2023-10-05T19:28:31.265 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.258+0000 7f3d0d3cae40 1 -- 172.21.15.167:0/3739225997 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:93 7.d 7:b4812045:::notify.4:head [watch watch cookie 94843367117488] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x56427134e6b0 con 0x56427130e7e0 2023-10-05T19:28:31.265 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.258+0000 7f3d0d3cae40 1 -- 172.21.15.167:0/3739225997 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:94 7.3 7:c609908c:::notify.5:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x56427134f0c0 con 0x7f3cf400ebe0 2023-10-05T19:28:31.268 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.262+0000 7f3d0b700700 1 -- 172.21.15.167:0/3739225997 <== osd.1 v2:172.21.15.167:6800/30035 32 ==== osd_op_reply(93 notify.4 [watch watch cookie 94843367117488] v30'94 uv93 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f3d04050040 con 0x56427130e7e0 2023-10-05T19:28:31.268 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.262+0000 7f3d0b700700 1 -- 172.21.15.167:0/3739225997 <== osd.0 v2:172.21.15.118:6800/27690 23 ==== osd_op_reply(94 notify.5 [create] v30'93 uv93 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f3d04050040 con 0x7f3cf400ebe0 2023-10-05T19:28:31.268 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.262+0000 7f3d0d3cae40 1 -- 172.21.15.167:0/3739225997 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:95 7.3 7:c609908c:::notify.5:head [watch watch cookie 94843367120256] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x56427134f0c0 con 0x7f3cf400ebe0 2023-10-05T19:28:31.268 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.262+0000 7f3d0d3cae40 1 -- 172.21.15.167:0/3739225997 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:96 7.14 7:2b04a3e9:::notify.6:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x56427134fad0 con 0x7f3cf400ebe0 2023-10-05T19:28:31.271 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.266+0000 7f3d0b700700 1 -- 172.21.15.167:0/3739225997 <== osd.0 v2:172.21.15.118:6800/27690 24 ==== osd_op_reply(96 notify.6 [create] v30'93 uv93 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f3d04050040 con 0x7f3cf400ebe0 2023-10-05T19:28:31.271 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.266+0000 7f3d0d3cae40 1 -- 172.21.15.167:0/3739225997 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:97 7.14 7:2b04a3e9:::notify.6:head [watch watch cookie 94843367126496] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x56427134ff20 con 0x7f3cf400ebe0 2023-10-05T19:28:31.271 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.266+0000 7f3d0d3cae40 1 -- 172.21.15.167:0/3739225997 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:98 7.9 7:93e5b521:::notify.7:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x564271350930 con 0x7f3cf400ebe0 2023-10-05T19:28:31.271 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.266+0000 7f3d0b700700 1 -- 172.21.15.167:0/3739225997 <== osd.0 v2:172.21.15.118:6800/27690 25 ==== osd_op_reply(95 notify.5 [watch watch cookie 94843367120256] v30'94 uv93 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f3d04050040 con 0x7f3cf400ebe0 2023-10-05T19:28:31.274 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.266+0000 7f3d0b700700 1 -- 172.21.15.167:0/3739225997 <== osd.0 v2:172.21.15.118:6800/27690 26 ==== osd_op_reply(98 notify.7 [create] v30'93 uv93 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f3d04050040 con 0x7f3cf400ebe0 2023-10-05T19:28:31.274 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.266+0000 7f3d0d3cae40 1 -- 172.21.15.167:0/3739225997 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:99 7.9 7:93e5b521:::notify.7:head [watch watch cookie 94843367123920] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x564271350930 con 0x7f3cf400ebe0 2023-10-05T19:28:31.274 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.266+0000 7f3d0b700700 1 -- 172.21.15.167:0/3739225997 <== osd.0 v2:172.21.15.118:6800/27690 27 ==== osd_op_reply(97 notify.6 [watch watch cookie 94843367126496] v30'94 uv93 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f3d04050040 con 0x7f3cf400ebe0 2023-10-05T19:28:31.274 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.266+0000 7f3d0d3cae40 20 add_watcher() i=0 2023-10-05T19:28:31.274 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.266+0000 7f3d0d3cae40 20 add_watcher() i=1 2023-10-05T19:28:31.274 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.266+0000 7f3d0d3cae40 20 add_watcher() i=2 2023-10-05T19:28:31.274 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.266+0000 7f3d0d3cae40 20 add_watcher() i=3 2023-10-05T19:28:31.274 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.266+0000 7f3d0d3cae40 20 add_watcher() i=4 2023-10-05T19:28:31.274 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.266+0000 7f3d0d3cae40 20 add_watcher() i=5 2023-10-05T19:28:31.275 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.266+0000 7f3d0d3cae40 20 add_watcher() i=6 2023-10-05T19:28:31.276 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.270+0000 7f3d0b700700 1 -- 172.21.15.167:0/3739225997 <== osd.0 v2:172.21.15.118:6800/27690 28 ==== osd_op_reply(99 notify.7 [watch watch cookie 94843367123920] v30'94 uv93 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f3d04050040 con 0x7f3cf400ebe0 2023-10-05T19:28:31.276 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.270+0000 7f3d0d3cae40 20 add_watcher() i=7 2023-10-05T19:28:31.276 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.270+0000 7f3d0d3cae40 2 all 8 watchers are set, enabling cache 2023-10-05T19:28:31.276 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.270+0000 7f3d0d3cae40 1 -- 172.21.15.167:0/3739225997 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:100 6.4 6:22d26bf9:::data_loggenerations_metadata:head [call version.check_conds in=74b,call version.read in=11b,read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x564271353960 con 0x56427130e7e0 2023-10-05T19:28:31.277 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.270+0000 7f3d0b700700 1 -- 172.21.15.167:0/3739225997 <== osd.1 v2:172.21.15.167:6800/30035 33 ==== osd_op_reply(100 data_loggenerations_metadata [call,call out=48b,read 0~28 out=28b] v0'0 uv1 ondisk = 0) v8 ==== 256+0+76 (crc 0 0 0) 0x7f3d04050040 con 0x56427130e7e0 2023-10-05T19:28:31.277 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.270+0000 7f3d0d3cae40 1 -- 172.21.15.167:0/3739225997 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:101 6.4 6:22d26bf9:::data_loggenerations_metadata:head [watch watch cookie 94843367130176] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x564271350d80 con 0x56427130e7e0 2023-10-05T19:28:31.279 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.274+0000 7f3d0b700700 1 -- 172.21.15.167:0/3739225997 <== osd.1 v2:172.21.15.167:6800/30035 34 ==== osd_op_reply(101 data_loggenerations_metadata [watch watch cookie 94843367130176] v30'97 uv1 ondisk = 0) v8 ==== 172+0+0 (crc 0 0 0) 0x7f3d04050040 con 0x56427130e7e0 2023-10-05T19:28:31.279 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.274+0000 7f3c57fff700 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2023-10-05T19:28:31.279 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.274+0000 7f3d0d3cae40 20 check_secure_mon_conn(): auth registy supported: methods=[2] modes=[2,1] 2023-10-05T19:28:31.279 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.274+0000 7f3d0d3cae40 20 check_secure_mon_conn(): mode 1 is insecure 2023-10-05T19:28:31.280 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.274+0000 7f3d0d3cae40 1 Processor -- start 2023-10-05T19:28:31.280 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.274+0000 7f3d0d3cae40 1 -- start start 2023-10-05T19:28:31.280 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.274+0000 7f3d0d3cae40 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x56427135dae0 0x56427135deb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-05T19:28:31.280 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.274+0000 7f3d0d3cae40 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x56427135e3f0 con 0x56427135dae0 2023-10-05T19:28:31.281 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.274+0000 7f3d0a6fe700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x56427135dae0 0x56427135deb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-05T19:28:31.281 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.274+0000 7f3d0a6fe700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x56427135dae0 0x56427135deb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.118:3300/0 says I am v2:172.21.15.167:58170/0 (socket says 172.21.15.167:58170) 2023-10-05T19:28:31.281 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.274+0000 7f3d0a6fe700 1 -- 172.21.15.167:0/1909434049 learned_addr learned my addr 172.21.15.167:0/1909434049 (peer_addr_for_me v2:172.21.15.167:0/0) 2023-10-05T19:28:31.281 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.274+0000 7f3d0a6fe700 1 -- 172.21.15.167:0/1909434049 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x56427135e530 con 0x56427135dae0 2023-10-05T19:28:31.281 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.274+0000 7f3d0a6fe700 1 --2- 172.21.15.167:0/1909434049 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x56427135dae0 0x56427135deb0 secure :-1 s=READY pgs=324 cs=0 l=1 rev1=1 crypto rx=0x564271342b60 tx=0x7f3cf4006000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=22c5312575270c4e server_cookie=0 in_seq=0 out_seq=0 2023-10-05T19:28:31.282 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.274+0000 7f3c577fe700 1 -- 172.21.15.167:0/1909434049 <== mon.0 v2:172.21.15.118:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f3cf4023040 con 0x56427135dae0 2023-10-05T19:28:31.282 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.274+0000 7f3c577fe700 1 -- 172.21.15.167:0/1909434049 <== mon.0 v2:172.21.15.118:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3cf4024040 con 0x56427135dae0 2023-10-05T19:28:31.282 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.274+0000 7f3c577fe700 1 -- 172.21.15.167:0/1909434049 <== mon.0 v2:172.21.15.118:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f3cf4013040 con 0x56427135dae0 2023-10-05T19:28:31.283 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.278+0000 7f3d0d3cae40 1 -- 172.21.15.167:0/1909434049 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x56427135dae0 msgr2=0x56427135deb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-05T19:28:31.283 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.278+0000 7f3d0d3cae40 1 --2- 172.21.15.167:0/1909434049 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x56427135dae0 0x56427135deb0 secure :-1 s=READY pgs=324 cs=0 l=1 rev1=1 crypto rx=0x564271342b60 tx=0x7f3cf4006000 comp rx=0 tx=0).stop 2023-10-05T19:28:31.283 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.278+0000 7f3d0d3cae40 1 -- 172.21.15.167:0/1909434049 shutdown_connections 2023-10-05T19:28:31.283 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.278+0000 7f3d0d3cae40 1 --2- 172.21.15.167:0/1909434049 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x56427135dae0 0x56427135deb0 unknown :-1 s=CLOSED pgs=324 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-05T19:28:31.283 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.278+0000 7f3d0d3cae40 1 -- 172.21.15.167:0/1909434049 >> 172.21.15.167:0/1909434049 conn(0x564271358ea0 msgr2=0x564271357050 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-05T19:28:31.284 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.278+0000 7f3d0d3cae40 1 -- 172.21.15.167:0/1909434049 shutdown_connections 2023-10-05T19:28:31.284 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.278+0000 7f3d0d3cae40 1 -- 172.21.15.167:0/1909434049 wait complete. 2023-10-05T19:28:31.285 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.278+0000 7f3d0d3cae40 1 Processor -- start 2023-10-05T19:28:31.285 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.278+0000 7f3d0d3cae40 1 -- start start 2023-10-05T19:28:31.285 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.278+0000 7f3d0d3cae40 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x56427135dae0 0x564271363650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-05T19:28:31.285 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.278+0000 7f3d0d3cae40 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x564271364650 con 0x56427135dae0 2023-10-05T19:28:31.285 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.278+0000 7f3d0a6fe700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x56427135dae0 0x564271363650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-05T19:28:31.285 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.278+0000 7f3d0a6fe700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x56427135dae0 0x564271363650 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.118:3300/0 says I am v2:172.21.15.167:58178/0 (socket says 172.21.15.167:58178) 2023-10-05T19:28:31.285 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.278+0000 7f3d0a6fe700 1 -- 172.21.15.167:0/1107188043 learned_addr learned my addr 172.21.15.167:0/1107188043 (peer_addr_for_me v2:172.21.15.167:0/0) 2023-10-05T19:28:31.285 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.278+0000 7f3d0a6fe700 1 -- 172.21.15.167:0/1107188043 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3cf8001e10 con 0x56427135dae0 2023-10-05T19:28:31.286 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.278+0000 7f3d0a6fe700 1 --2- 172.21.15.167:0/1107188043 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x56427135dae0 0x564271363650 secure :-1 s=READY pgs=325 cs=0 l=1 rev1=1 crypto rx=0x564271357bc0 tx=0x7f3cf4036000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-05T19:28:31.287 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.278+0000 7f3c55ffb700 1 -- 172.21.15.167:0/1107188043 <== mon.0 v2:172.21.15.118:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f3cf4023070 con 0x56427135dae0 2023-10-05T19:28:31.287 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.278+0000 7f3c55ffb700 1 -- 172.21.15.167:0/1107188043 <== mon.0 v2:172.21.15.118:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3cf401e630 con 0x56427135dae0 2023-10-05T19:28:31.287 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.278+0000 7f3d0d3cae40 1 -- 172.21.15.167:0/1107188043 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x564271363c50 con 0x56427135dae0 2023-10-05T19:28:31.287 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.278+0000 7f3c55ffb700 1 -- 172.21.15.167:0/1107188043 <== mon.0 v2:172.21.15.118:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f3cf401e7c0 con 0x56427135dae0 2023-10-05T19:28:31.287 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.278+0000 7f3d0d3cae40 1 -- 172.21.15.167:0/1107188043 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x564271364090 con 0x56427135dae0 2023-10-05T19:28:31.288 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.282+0000 7f3c55ffb700 1 -- 172.21.15.167:0/1107188043 <== mon.0 v2:172.21.15.118:3300/0 4 ==== mgrmap(e 5) v1 ==== 40329+0+0 (secure 0 0 0) 0x7f3cf403f040 con 0x56427135dae0 2023-10-05T19:28:31.288 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.282+0000 7f3c55ffb700 1 --2- 172.21.15.167:0/1107188043 >> [v2:172.21.15.167:6824/29889,v1:172.21.15.167:6825/29889] conn(0x7f3c48030c00 0x7f3c480330b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-05T19:28:31.288 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.282+0000 7f3c55ffb700 1 -- 172.21.15.167:0/1107188043 <== mon.0 v2:172.21.15.118:3300/0 5 ==== osd_map(30..30 src has 1..30) v4 ==== 6057+0+0 (secure 0 0 0) 0x7f3cf4042c10 con 0x56427135dae0 2023-10-05T19:28:31.288 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.282+0000 7f3d0b700700 1 --2- 172.21.15.167:0/1107188043 >> [v2:172.21.15.167:6824/29889,v1:172.21.15.167:6825/29889] conn(0x7f3c48030c00 0x7f3c480330b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-05T19:28:31.288 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.282+0000 7f3d0d3cae40 5 note: GC not initialized 2023-10-05T19:28:31.289 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.282+0000 7f3d0b700700 1 --2- 172.21.15.167:0/1107188043 >> [v2:172.21.15.167:6824/29889,v1:172.21.15.167:6825/29889] conn(0x7f3c48030c00 0x7f3c480330b0 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x56427134fc30 tx=0x7f3d04000f90 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-05T19:28:31.290 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.286+0000 7f3c4effd700 20 reqs_thread_entry: start 2023-10-05T19:28:31.291 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.286+0000 7f3c4cff9700 20 reqs_thread_entry: start 2023-10-05T19:28:31.291 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.286+0000 7f3d0d3cae40 20 init_complete bucket index max shards: 11 2023-10-05T19:28:31.292 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.286+0000 7f3d0d3cae40 10 rgw notify: Started notification manager with: 1 workers 2023-10-05T19:28:31.292 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.286+0000 7f3c3a7fc700 20 reqs_thread_entry: start 2023-10-05T19:28:31.292 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.286+0000 7f3d0d3cae40 20 get_system_obj_state: rctx=0x7ffe61d1e560 obj=default.rgw.meta:root:myfoo state=0x5642711e70b0 s->prefetch_data=0 2023-10-05T19:28:31.292 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.286+0000 7f3d0d3cae40 1 -- 172.21.15.167:0/3739225997 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:102 8.9 8:91b71841:root::myfoo:head [call version.read in=11b,getxattrs,stat] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x5642713cc5e0 con 0x7f3cfc0373b0 2023-10-05T19:28:31.292 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.286+0000 7f3c3affd700 20 rgw notify: INFO: next queues processing will happen at: Thu Oct 5 19:29:01 2023 2023-10-05T19:28:31.292 INFO:teuthology.orchestra.run.smithi167.stderr: 2023-10-05T19:28:31.293 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.286+0000 7f3d0a6fe700 1 -- 172.21.15.167:0/3739225997 <== osd.2 v2:172.21.15.167:6816/30037 17 ==== osd_op_reply(102 myfoo [call out=48b,getxattrs out=75b,stat out=16b] v0'0 uv1 ondisk = 0) v8 ==== 233+0+139 (crc 0 0 0) 0x7f3cf4006070 con 0x7f3cfc0373b0 2023-10-05T19:28:31.293 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.286+0000 7f3d0d3cae40 20 get_system_obj_state: s->obj_tag was set empty 2023-10-05T19:28:31.293 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.286+0000 7f3d0d3cae40 20 rados->read ofs=0 len=0 2023-10-05T19:28:31.293 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.286+0000 7f3d0d3cae40 1 -- 172.21.15.167:0/3739225997 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:103 8.9 8:91b71841:root::myfoo:head [call version.check_conds in=122b,call version.read in=11b,read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x5642713cf070 con 0x7f3cfc0373b0 2023-10-05T19:28:31.294 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.286+0000 7f3d0a6fe700 1 -- 172.21.15.167:0/3739225997 <== osd.2 v2:172.21.15.167:6816/30037 18 ==== osd_op_reply(103 myfoo [call,call out=48b,read 0~165 out=165b] v0'0 uv1 ondisk = 0) v8 ==== 233+0+213 (crc 0 0 0) 0x7f3cf4006070 con 0x7f3cfc0373b0 2023-10-05T19:28:31.294 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.286+0000 7f3d0d3cae40 20 rados_obj.operate() r=0 bl.length=165 2023-10-05T19:28:31.294 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.286+0000 7f3d0d3cae40 20 get_system_obj_state: rctx=0x7ffe61d1e560 obj=default.rgw.meta:root:.bucket.meta.myfoo:9a9df965-6612-45d4-a45f-6e3c889bacc2.4223.1 state=0x5642711bd2b0 s->prefetch_data=0 2023-10-05T19:28:31.295 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.286+0000 7f3d0d3cae40 1 -- 172.21.15.167:0/3739225997 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:104 8.11 8:8bb4ed75:root::.bucket.meta.myfoo%3a9a9df965-6612-45d4-a45f-6e3c889bacc2.4223.1:head [call version.read in=11b,getxattrs,stat] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x5642713cd270 con 0x564271345070 2023-10-05T19:28:31.295 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.290+0000 7f3d0aeff700 1 -- 172.21.15.167:0/3739225997 <== osd.3 v2:172.21.15.167:6808/30036 24 ==== osd_op_reply(104 .bucket.meta.myfoo:9a9df965-6612-45d4-a45f-6e3c889bacc2.4223.1 [call out=48b,getxattrs out=218b,stat out=16b] v0'0 uv1 ondisk = 0) v8 ==== 290+0+282 (crc 0 0 0) 0x7f3cfc053040 con 0x564271345070 2023-10-05T19:28:31.295 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.290+0000 7f3d0d3cae40 20 get_system_obj_state: s->obj_tag was set empty 2023-10-05T19:28:31.295 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.290+0000 7f3d0d3cae40 20 Read xattr: user.rgw.acl 2023-10-05T19:28:31.295 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.290+0000 7f3d0d3cae40 20 Read xattr: user.rgw.idtag 2023-10-05T19:28:31.295 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.290+0000 7f3d0d3cae40 20 rados->read ofs=0 len=0 2023-10-05T19:28:31.295 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.290+0000 7f3d0d3cae40 1 -- 172.21.15.167:0/3739225997 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:105 8.11 8:8bb4ed75:root::.bucket.meta.myfoo%3a9a9df965-6612-45d4-a45f-6e3c889bacc2.4223.1:head [call version.check_conds in=122b,call version.read in=11b,read 0~0,getxattrs] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x5642713cee10 con 0x564271345070 2023-10-05T19:28:31.297 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.290+0000 7f3d0aeff700 1 -- 172.21.15.167:0/3739225997 <== osd.3 v2:172.21.15.167:6808/30036 25 ==== osd_op_reply(105 .bucket.meta.myfoo:9a9df965-6612-45d4-a45f-6e3c889bacc2.4223.1 [call,call out=48b,read 0~352 out=352b,getxattrs out=218b] v0'0 uv1 ondisk = 0) v8 ==== 332+0+618 (crc 0 0 0) 0x7f3cfc053040 con 0x564271345070 2023-10-05T19:28:31.297 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.290+0000 7f3d0d3cae40 20 rados_obj.operate() r=0 bl.length=352 2023-10-05T19:28:31.297 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.290+0000 7f3d0d3cae40 20 couldn't put binfo cache entry, might have raced with data changes 2023-10-05T19:28:31.297 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.290+0000 7f3d0d3cae40 20 get_system_obj_state: rctx=0x7ffe61d1e4c0 obj=default.rgw.meta:root:myfoo state=0x5642711bd2b0 s->prefetch_data=0 2023-10-05T19:28:31.297 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.290+0000 7f3d0d3cae40 1 -- 172.21.15.167:0/3739225997 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:106 8.9 8:91b71841:root::myfoo:head [call version.read in=11b,getxattrs,stat] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x5642713cfe10 con 0x7f3cfc0373b0 2023-10-05T19:28:31.297 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.290+0000 7f3d0a6fe700 1 -- 172.21.15.167:0/3739225997 <== osd.2 v2:172.21.15.167:6816/30037 19 ==== osd_op_reply(106 myfoo [call out=48b,getxattrs out=75b,stat out=16b] v0'0 uv1 ondisk = 0) v8 ==== 233+0+139 (crc 0 0 0) 0x7f3cf4006070 con 0x7f3cfc0373b0 2023-10-05T19:28:31.297 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.290+0000 7f3d0d3cae40 20 get_system_obj_state: s->obj_tag was set empty 2023-10-05T19:28:31.297 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.290+0000 7f3d0d3cae40 20 rados->read ofs=0 len=0 2023-10-05T19:28:31.298 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.290+0000 7f3d0d3cae40 1 -- 172.21.15.167:0/3739225997 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:107 8.9 8:91b71841:root::myfoo:head [call version.check_conds in=122b,call version.read in=11b,read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x5642713cfe10 con 0x7f3cfc0373b0 2023-10-05T19:28:31.298 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.290+0000 7f3d0a6fe700 1 -- 172.21.15.167:0/3739225997 <== osd.2 v2:172.21.15.167:6816/30037 20 ==== osd_op_reply(107 myfoo [call,call out=48b,read 0~165 out=165b] v0'0 uv1 ondisk = 0) v8 ==== 233+0+213 (crc 0 0 0) 0x7f3cf4006070 con 0x7f3cfc0373b0 2023-10-05T19:28:31.298 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.290+0000 7f3d0d3cae40 20 rados_obj.operate() r=0 bl.length=165 2023-10-05T19:28:31.298 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.290+0000 7f3d0d3cae40 20 get_system_obj_state: rctx=0x7ffe61d1e4c0 obj=default.rgw.meta:root:.bucket.meta.myfoo:9a9df965-6612-45d4-a45f-6e3c889bacc2.4223.1 state=0x5642711e70b0 s->prefetch_data=0 2023-10-05T19:28:31.298 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.290+0000 7f3d0d3cae40 1 -- 172.21.15.167:0/3739225997 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:108 8.11 8:8bb4ed75:root::.bucket.meta.myfoo%3a9a9df965-6612-45d4-a45f-6e3c889bacc2.4223.1:head [call version.read in=11b,getxattrs,stat] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x5642713cfe10 con 0x564271345070 2023-10-05T19:28:31.302 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.290+0000 7f3d0aeff700 1 -- 172.21.15.167:0/3739225997 <== osd.3 v2:172.21.15.167:6808/30036 26 ==== osd_op_reply(108 .bucket.meta.myfoo:9a9df965-6612-45d4-a45f-6e3c889bacc2.4223.1 [call out=48b,getxattrs out=218b,stat out=16b] v0'0 uv1 ondisk = 0) v8 ==== 290+0+282 (crc 0 0 0) 0x7f3cfc053040 con 0x564271345070 2023-10-05T19:28:31.302 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.290+0000 7f3d0d3cae40 20 get_system_obj_state: s->obj_tag was set empty 2023-10-05T19:28:31.302 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.290+0000 7f3d0d3cae40 20 Read xattr: user.rgw.acl 2023-10-05T19:28:31.302 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.290+0000 7f3d0d3cae40 20 Read xattr: user.rgw.idtag 2023-10-05T19:28:31.302 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.290+0000 7f3d0d3cae40 20 rados->read ofs=0 len=0 2023-10-05T19:28:31.302 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.290+0000 7f3d0d3cae40 1 -- 172.21.15.167:0/3739225997 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:109 8.11 8:8bb4ed75:root::.bucket.meta.myfoo%3a9a9df965-6612-45d4-a45f-6e3c889bacc2.4223.1:head [call version.check_conds in=122b,call version.read in=11b,read 0~0,getxattrs] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x5642713cee10 con 0x564271345070 2023-10-05T19:28:31.302 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.294+0000 7f3d0aeff700 1 -- 172.21.15.167:0/3739225997 <== osd.3 v2:172.21.15.167:6808/30036 27 ==== osd_op_reply(109 .bucket.meta.myfoo:9a9df965-6612-45d4-a45f-6e3c889bacc2.4223.1 [call,call out=48b,read 0~352 out=352b,getxattrs out=218b] v0'0 uv1 ondisk = 0) v8 ==== 332+0+618 (crc 0 0 0) 0x7f3cfc053040 con 0x564271345070 2023-10-05T19:28:31.302 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.294+0000 7f3d0d3cae40 20 rados_obj.operate() r=0 bl.length=352 2023-10-05T19:28:31.302 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.294+0000 7f3d0d3cae40 20 couldn't put binfo cache entry, might have raced with data changes 2023-10-05T19:28:31.303 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.294+0000 7f3d0d3cae40 1 -- 172.21.15.167:0/3739225997 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:110 4.3d 4:bd5dd36d:::.dir.9a9df965-6612-45d4-a45f-6e3c889bacc2.4223.1.0:head [call rgw.bucket_list in=47b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x5642713d1880 con 0x564271345070 2023-10-05T19:28:31.303 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.294+0000 7f3d0d3cae40 1 -- 172.21.15.167:0/3739225997 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:111 4.29 4:955f623b:::.dir.9a9df965-6612-45d4-a45f-6e3c889bacc2.4223.1.1:head [call rgw.bucket_list in=47b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x5642713d2490 con 0x564271345070 2023-10-05T19:28:31.303 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.294+0000 7f3d0d3cae40 1 -- 172.21.15.167:0/3739225997 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:112 4.38 4:1d83ede1:::.dir.9a9df965-6612-45d4-a45f-6e3c889bacc2.4223.1.2:head [call rgw.bucket_list in=47b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x5642713d7400 con 0x56427130e7e0 2023-10-05T19:28:31.303 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.294+0000 7f3d0d3cae40 1 -- 172.21.15.167:0/3739225997 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:113 4.1e 4:7939a9ac:::.dir.9a9df965-6612-45d4-a45f-6e3c889bacc2.4223.1.3:head [call rgw.bucket_list in=47b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x5642713d9d20 con 0x7f3cf400ebe0 2023-10-05T19:28:31.303 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.294+0000 7f3d0d3cae40 1 -- 172.21.15.167:0/3739225997 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:114 4.e 4:719498d8:::.dir.9a9df965-6612-45d4-a45f-6e3c889bacc2.4223.1.4:head [call rgw.bucket_list in=47b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x5642713dc760 con 0x7f3cfc0373b0 2023-10-05T19:28:31.303 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.294+0000 7f3d0d3cae40 1 -- 172.21.15.167:0/3739225997 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:115 4.32 4:4f38608c:::.dir.9a9df965-6612-45d4-a45f-6e3c889bacc2.4223.1.5:head [call rgw.bucket_list in=47b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x5642713df3e0 con 0x56427130e7e0 2023-10-05T19:28:31.303 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.294+0000 7f3d0d3cae40 1 -- 172.21.15.167:0/3739225997 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:116 4.38 4:1f2a6458:::.dir.9a9df965-6612-45d4-a45f-6e3c889bacc2.4223.1.6:head [call rgw.bucket_list in=47b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x5642713e1de0 con 0x56427130e7e0 2023-10-05T19:28:31.303 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.294+0000 7f3d0d3cae40 1 -- 172.21.15.167:0/3739225997 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:117 4.2 4:41a3e149:::.dir.9a9df965-6612-45d4-a45f-6e3c889bacc2.4223.1.7:head [call rgw.bucket_list in=47b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x5642713e4ae0 con 0x56427130e7e0 2023-10-05T19:28:31.303 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.294+0000 7f3d0d3cae40 1 -- 172.21.15.167:0/3739225997 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:118 4.17 4:ea8a113e:::.dir.9a9df965-6612-45d4-a45f-6e3c889bacc2.4223.1.8:head [call rgw.bucket_list in=47b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x5642713e79d0 con 0x7f3cf400ebe0 2023-10-05T19:28:31.303 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.294+0000 7f3d0d3cae40 1 -- 172.21.15.167:0/3739225997 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:119 4.1e 4:7a3e9338:::.dir.9a9df965-6612-45d4-a45f-6e3c889bacc2.4223.1.9:head [call rgw.bucket_list in=47b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x5642713ea880 con 0x7f3cf400ebe0 2023-10-05T19:28:31.304 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.294+0000 7f3d0d3cae40 1 -- 172.21.15.167:0/3739225997 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:120 4.29 4:94432b39:::.dir.9a9df965-6612-45d4-a45f-6e3c889bacc2.4223.1.10:head [call rgw.bucket_list in=47b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x5642713ed780 con 0x564271345070 2023-10-05T19:28:31.304 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.294+0000 7f3d0b700700 1 -- 172.21.15.167:0/3739225997 <== osd.1 v2:172.21.15.167:6800/30035 35 ==== osd_op_reply(112 .dir.9a9df965-6612-45d4-a45f-6e3c889bacc2.4223.1.2 [call out=85b] v0'0 uv1 ondisk = 0) v8 ==== 194+0+85 (crc 0 0 0) 0x7f3d04050040 con 0x56427130e7e0 2023-10-05T19:28:31.304 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.294+0000 7f3d0aeff700 1 -- 172.21.15.167:0/3739225997 <== osd.3 v2:172.21.15.167:6808/30036 28 ==== osd_op_reply(110 .dir.9a9df965-6612-45d4-a45f-6e3c889bacc2.4223.1.0 [call out=85b] v0'0 uv1 ondisk = 0) v8 ==== 194+0+85 (crc 0 0 0) 0x7f3cfc053040 con 0x564271345070 2023-10-05T19:28:31.304 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.294+0000 7f3d0b700700 1 -- 172.21.15.167:0/3739225997 <== osd.0 v2:172.21.15.118:6800/27690 29 ==== osd_op_reply(113 .dir.9a9df965-6612-45d4-a45f-6e3c889bacc2.4223.1.3 [call out=85b] v0'0 uv1 ondisk = 0) v8 ==== 194+0+85 (crc 0 0 0) 0x7f3d04050040 con 0x7f3cf400ebe0 2023-10-05T19:28:31.304 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.294+0000 7f3d0aeff700 1 -- 172.21.15.167:0/3739225997 <== osd.3 v2:172.21.15.167:6808/30036 29 ==== osd_op_reply(111 .dir.9a9df965-6612-45d4-a45f-6e3c889bacc2.4223.1.1 [call out=85b] v0'0 uv1 ondisk = 0) v8 ==== 194+0+85 (crc 0 0 0) 0x7f3cfc053040 con 0x564271345070 2023-10-05T19:28:31.304 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.294+0000 7f3d0a6fe700 1 -- 172.21.15.167:0/3739225997 <== osd.2 v2:172.21.15.167:6816/30037 21 ==== osd_op_reply(114 .dir.9a9df965-6612-45d4-a45f-6e3c889bacc2.4223.1.4 [call out=85b] v0'0 uv1 ondisk = 0) v8 ==== 194+0+85 (crc 0 0 0) 0x7f3cf4006070 con 0x7f3cfc0373b0 2023-10-05T19:28:31.304 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.294+0000 7f3d0b700700 1 -- 172.21.15.167:0/3739225997 <== osd.0 v2:172.21.15.118:6800/27690 30 ==== osd_op_reply(118 .dir.9a9df965-6612-45d4-a45f-6e3c889bacc2.4223.1.8 [call out=85b] v0'0 uv1 ondisk = 0) v8 ==== 194+0+85 (crc 0 0 0) 0x7f3d04050040 con 0x7f3cf400ebe0 2023-10-05T19:28:31.304 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.294+0000 7f3d0aeff700 1 -- 172.21.15.167:0/3739225997 <== osd.3 v2:172.21.15.167:6808/30036 30 ==== osd_op_reply(120 .dir.9a9df965-6612-45d4-a45f-6e3c889bacc2.4223.1.10 [call out=85b] v0'0 uv2 ondisk = 0) v8 ==== 195+0+85 (crc 0 0 0) 0x7f3cfc053040 con 0x564271345070 2023-10-05T19:28:31.304 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.294+0000 7f3d0b700700 1 -- 172.21.15.167:0/3739225997 <== osd.0 v2:172.21.15.118:6800/27690 31 ==== osd_op_reply(119 .dir.9a9df965-6612-45d4-a45f-6e3c889bacc2.4223.1.9 [call out=85b] v0'0 uv2 ondisk = 0) v8 ==== 194+0+85 (crc 0 0 0) 0x7f3d04050040 con 0x7f3cf400ebe0 2023-10-05T19:28:31.304 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.294+0000 7f3d0b700700 1 -- 172.21.15.167:0/3739225997 <== osd.1 v2:172.21.15.167:6800/30035 36 ==== osd_op_reply(115 .dir.9a9df965-6612-45d4-a45f-6e3c889bacc2.4223.1.5 [call out=85b] v0'0 uv1 ondisk = 0) v8 ==== 194+0+85 (crc 0 0 0) 0x7f3d04050040 con 0x56427130e7e0 2023-10-05T19:28:31.304 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.294+0000 7f3d0b700700 1 -- 172.21.15.167:0/3739225997 <== osd.1 v2:172.21.15.167:6800/30035 37 ==== osd_op_reply(116 .dir.9a9df965-6612-45d4-a45f-6e3c889bacc2.4223.1.6 [call out=85b] v0'0 uv2 ondisk = 0) v8 ==== 194+0+85 (crc 0 0 0) 0x7f3d04050040 con 0x56427130e7e0 2023-10-05T19:28:31.305 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.294+0000 7f3d0b700700 1 -- 172.21.15.167:0/3739225997 <== osd.1 v2:172.21.15.167:6800/30035 38 ==== osd_op_reply(117 .dir.9a9df965-6612-45d4-a45f-6e3c889bacc2.4223.1.7 [call out=85b] v0'0 uv1 ondisk = 0) v8 ==== 194+0+85 (crc 0 0 0) 0x7f3d04050040 con 0x56427130e7e0 2023-10-05T19:28:31.305 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.294+0000 7f3d0d3cae40 1 -- 172.21.15.167:0/3739225997 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:121 7.1f 7:f95f44c2:::notify.0:head [watch unwatch cookie 94843367105552] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x5642713257f0 con 0x7f3cfc0373b0 2023-10-05T19:28:31.305 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.298+0000 7f3d0a6fe700 1 -- 172.21.15.167:0/3739225997 <== osd.2 v2:172.21.15.167:6816/30037 22 ==== osd_op_reply(121 notify.0 [watch unwatch cookie 94843367105552] v30'95 uv93 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f3cf4006070 con 0x7f3cfc0373b0 2023-10-05T19:28:31.306 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.298+0000 7f3d0d3cae40 20 remove_watcher() i=0 2023-10-05T19:28:31.306 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.298+0000 7f3d0d3cae40 2 removed watcher, disabling cache 2023-10-05T19:28:31.306 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.298+0000 7f3d0d3cae40 1 -- 172.21.15.167:0/3739225997 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:122 7.0 7:05bf5b68:::notify.1:head [watch unwatch cookie 94843367109072] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x5642713257f0 con 0x56427130e7e0 2023-10-05T19:28:31.306 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.298+0000 7f3d0b700700 1 -- 172.21.15.167:0/3739225997 <== osd.1 v2:172.21.15.167:6800/30035 39 ==== osd_op_reply(122 notify.1 [watch unwatch cookie 94843367109072] v30'95 uv93 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f3d04050040 con 0x56427130e7e0 2023-10-05T19:28:31.306 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.298+0000 7f3d0d3cae40 20 remove_watcher() i=1 2023-10-05T19:28:31.306 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.298+0000 7f3d0d3cae40 1 -- 172.21.15.167:0/3739225997 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:123 7.15 7:a93a5511:::notify.2:head [watch unwatch cookie 94843367112336] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x5642713257f0 con 0x564271345070 2023-10-05T19:28:31.308 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.302+0000 7f3d0aeff700 1 -- 172.21.15.167:0/3739225997 <== osd.3 v2:172.21.15.167:6808/30036 31 ==== osd_op_reply(123 notify.2 [watch unwatch cookie 94843367112336] v30'95 uv93 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f3cfc053040 con 0x564271345070 2023-10-05T19:28:31.308 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.302+0000 7f3d0d3cae40 20 remove_watcher() i=2 2023-10-05T19:28:31.308 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.302+0000 7f3d0d3cae40 1 -- 172.21.15.167:0/3739225997 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:124 7.e 7:7759931f:::notify.3:head [watch unwatch cookie 94843367110176] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x5642713257f0 con 0x7f3cfc0373b0 2023-10-05T19:28:31.310 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.306+0000 7f3d0a6fe700 1 -- 172.21.15.167:0/3739225997 <== osd.2 v2:172.21.15.167:6816/30037 23 ==== osd_op_reply(124 notify.3 [watch unwatch cookie 94843367110176] v30'95 uv93 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f3cf4006070 con 0x7f3cfc0373b0 2023-10-05T19:28:31.310 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.306+0000 7f3d0d3cae40 20 remove_watcher() i=3 2023-10-05T19:28:31.310 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.306+0000 7f3d0d3cae40 1 -- 172.21.15.167:0/3739225997 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:125 7.d 7:b4812045:::notify.4:head [watch unwatch cookie 94843367117488] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x5642713257f0 con 0x56427130e7e0 2023-10-05T19:28:31.313 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.306+0000 7f3d0b700700 1 -- 172.21.15.167:0/3739225997 <== osd.1 v2:172.21.15.167:6800/30035 40 ==== osd_op_reply(125 notify.4 [watch unwatch cookie 94843367117488] v30'95 uv93 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f3d04050040 con 0x56427130e7e0 2023-10-05T19:28:31.313 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.306+0000 7f3d0d3cae40 20 remove_watcher() i=4 2023-10-05T19:28:31.313 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.306+0000 7f3d0d3cae40 1 -- 172.21.15.167:0/3739225997 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:126 7.3 7:c609908c:::notify.5:head [watch unwatch cookie 94843367120256] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x5642713257f0 con 0x7f3cf400ebe0 2023-10-05T19:28:31.315 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.310+0000 7f3d0b700700 1 -- 172.21.15.167:0/3739225997 <== osd.0 v2:172.21.15.118:6800/27690 32 ==== osd_op_reply(126 notify.5 [watch unwatch cookie 94843367120256] v30'95 uv93 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f3d04050040 con 0x7f3cf400ebe0 2023-10-05T19:28:31.315 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.310+0000 7f3d0d3cae40 20 remove_watcher() i=5 2023-10-05T19:28:31.315 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.310+0000 7f3d0d3cae40 1 -- 172.21.15.167:0/3739225997 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:127 7.14 7:2b04a3e9:::notify.6:head [watch unwatch cookie 94843367126496] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x5642713257f0 con 0x7f3cf400ebe0 2023-10-05T19:28:31.318 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.314+0000 7f3d0b700700 1 -- 172.21.15.167:0/3739225997 <== osd.0 v2:172.21.15.118:6800/27690 33 ==== osd_op_reply(127 notify.6 [watch unwatch cookie 94843367126496] v30'95 uv93 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f3d04050040 con 0x7f3cf400ebe0 2023-10-05T19:28:31.318 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.314+0000 7f3d0d3cae40 20 remove_watcher() i=6 2023-10-05T19:28:31.318 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.314+0000 7f3d0d3cae40 1 -- 172.21.15.167:0/3739225997 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:128 7.9 7:93e5b521:::notify.7:head [watch unwatch cookie 94843367123920] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x5642713257f0 con 0x7f3cf400ebe0 2023-10-05T19:28:31.321 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.314+0000 7f3d0b700700 1 -- 172.21.15.167:0/3739225997 <== osd.0 v2:172.21.15.118:6800/27690 34 ==== osd_op_reply(128 notify.7 [watch unwatch cookie 94843367123920] v30'95 uv93 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f3d04050040 con 0x7f3cf400ebe0 2023-10-05T19:28:31.321 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.314+0000 7f3d0d3cae40 20 remove_watcher() i=7 2023-10-05T19:28:31.322 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.318+0000 7f3c57fff700 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2023-10-05T19:28:31.322 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.318+0000 7f3d0d3cae40 1 -- 172.21.15.167:0/3739225997 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:129 6.4 6:22d26bf9:::data_loggenerations_metadata:head [watch unwatch cookie 94843367130176] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x5642713257f0 con 0x56427130e7e0 2023-10-05T19:28:31.325 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.318+0000 7f3d0b700700 1 -- 172.21.15.167:0/3739225997 <== osd.1 v2:172.21.15.167:6800/30035 41 ==== osd_op_reply(129 data_loggenerations_metadata [watch unwatch cookie 94843367130176] v30'98 uv1 ondisk = 0) v8 ==== 172+0+0 (crc 0 0 0) 0x7f3d04050040 con 0x56427130e7e0 2023-10-05T19:28:31.326 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.318+0000 7f3d0d3cae40 1 -- 172.21.15.167:0/3739225997 >> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] conn(0x7f3cf400ebe0 msgr2=0x7f3cf400d8e0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-05T19:28:31.326 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.318+0000 7f3d0d3cae40 1 --2- 172.21.15.167:0/3739225997 >> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] conn(0x7f3cf400ebe0 0x7f3cf400d8e0 crc :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-05T19:28:31.326 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.318+0000 7f3d0d3cae40 1 -- 172.21.15.167:0/3739225997 >> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] conn(0x56427130e7e0 msgr2=0x5642713416b0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-05T19:28:31.326 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.318+0000 7f3d0d3cae40 1 --2- 172.21.15.167:0/3739225997 >> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] conn(0x56427130e7e0 0x5642713416b0 crc :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-05T19:28:31.326 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.318+0000 7f3d0d3cae40 1 -- 172.21.15.167:0/3739225997 >> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] conn(0x7f3cfc0373b0 msgr2=0x7f3cfc037a70 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-05T19:28:31.326 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.318+0000 7f3d0d3cae40 1 --2- 172.21.15.167:0/3739225997 >> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] conn(0x7f3cfc0373b0 0x7f3cfc037a70 crc :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-05T19:28:31.326 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.318+0000 7f3d0d3cae40 1 -- 172.21.15.167:0/3739225997 >> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] conn(0x564271345070 msgr2=0x564271347450 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-05T19:28:31.326 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.318+0000 7f3d0d3cae40 1 --2- 172.21.15.167:0/3739225997 >> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] conn(0x564271345070 0x564271347450 crc :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-05T19:28:31.326 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.318+0000 7f3d0d3cae40 1 -- 172.21.15.167:0/3739225997 >> [v2:172.21.15.167:6824/29889,v1:172.21.15.167:6825/29889] conn(0x7f3cec030c50 msgr2=0x7f3cec033100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-05T19:28:31.326 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.318+0000 7f3d0d3cae40 1 --2- 172.21.15.167:0/3739225997 >> [v2:172.21.15.167:6824/29889,v1:172.21.15.167:6825/29889] conn(0x7f3cec030c50 0x7f3cec033100 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7f3cf4004170 tx=0x7f3cf4001400 comp rx=0 tx=0).stop 2023-10-05T19:28:31.326 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.318+0000 7f3d0b700700 1 -- 172.21.15.167:0/3739225997 reap_dead start 2023-10-05T19:28:31.327 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.318+0000 7f3d0d3cae40 1 -- 172.21.15.167:0/3739225997 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x56427130e410 msgr2=0x564271335bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-05T19:28:31.327 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.318+0000 7f3d0d3cae40 1 --2- 172.21.15.167:0/3739225997 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x56427130e410 0x564271335bf0 secure :-1 s=READY pgs=323 cs=0 l=1 rev1=1 crypto rx=0x7f3cfc00a520 tx=0x7f3cfc01d000 comp rx=0 tx=0).stop 2023-10-05T19:28:31.327 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.318+0000 7f3d0d3cae40 1 -- 172.21.15.167:0/3739225997 shutdown_connections 2023-10-05T19:28:31.327 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.318+0000 7f3d0d3cae40 1 --2- 172.21.15.167:0/3739225997 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x56427130e410 0x564271335bf0 unknown :-1 s=CLOSED pgs=323 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-05T19:28:31.327 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.318+0000 7f3d0d3cae40 1 -- 172.21.15.167:0/3739225997 >> 172.21.15.167:0/3739225997 conn(0x564271308720 msgr2=0x564271329550 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-05T19:28:31.327 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.318+0000 7f3d0d3cae40 1 -- 172.21.15.167:0/3739225997 shutdown_connections 2023-10-05T19:28:31.327 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.318+0000 7f3d0d3cae40 1 -- 172.21.15.167:0/3739225997 wait complete. 2023-10-05T19:28:31.328 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.322+0000 7f3d0d3cae40 1 -- 172.21.15.167:0/1107188043 >> [v2:172.21.15.167:6824/29889,v1:172.21.15.167:6825/29889] conn(0x7f3c48030c00 msgr2=0x7f3c480330b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-05T19:28:31.328 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.322+0000 7f3d0d3cae40 1 --2- 172.21.15.167:0/1107188043 >> [v2:172.21.15.167:6824/29889,v1:172.21.15.167:6825/29889] conn(0x7f3c48030c00 0x7f3c480330b0 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x56427134fc30 tx=0x7f3d04000f90 comp rx=0 tx=0).stop 2023-10-05T19:28:31.328 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.322+0000 7f3d0d3cae40 1 -- 172.21.15.167:0/1107188043 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x56427135dae0 msgr2=0x564271363650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-05T19:28:31.328 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.322+0000 7f3d0d3cae40 1 --2- 172.21.15.167:0/1107188043 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x56427135dae0 0x564271363650 secure :-1 s=READY pgs=325 cs=0 l=1 rev1=1 crypto rx=0x564271357bc0 tx=0x7f3cf4036000 comp rx=0 tx=0).stop 2023-10-05T19:28:31.328 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.322+0000 7f3d0d3cae40 1 -- 172.21.15.167:0/1107188043 shutdown_connections 2023-10-05T19:28:31.328 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.322+0000 7f3d0d3cae40 1 --2- 172.21.15.167:0/1107188043 >> [v2:172.21.15.167:6824/29889,v1:172.21.15.167:6825/29889] conn(0x7f3c48030c00 0x7f3c480330b0 unknown :-1 s=CLOSED pgs=115 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-05T19:28:31.329 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.322+0000 7f3d0d3cae40 1 --2- 172.21.15.167:0/1107188043 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x56427135dae0 0x564271363650 unknown :-1 s=CLOSED pgs=325 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-05T19:28:31.329 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.322+0000 7f3d0d3cae40 1 -- 172.21.15.167:0/1107188043 >> 172.21.15.167:0/1107188043 conn(0x564271358ea0 msgr2=0x564271356fa0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-05T19:28:31.329 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.322+0000 7f3d0d3cae40 1 -- 172.21.15.167:0/1107188043 shutdown_connections 2023-10-05T19:28:31.329 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.322+0000 7f3d0d3cae40 1 -- 172.21.15.167:0/1107188043 wait complete. 2023-10-05T19:28:31.330 INFO:teuthology.orchestra.run.smithi167.stdout:{ 2023-10-05T19:28:31.330 INFO:teuthology.orchestra.run.smithi167.stdout: "bucket": "myfoo", 2023-10-05T19:28:31.330 INFO:teuthology.orchestra.run.smithi167.stdout: "num_shards": 11, 2023-10-05T19:28:31.330 INFO:teuthology.orchestra.run.smithi167.stdout: "tenant": "", 2023-10-05T19:28:31.330 INFO:teuthology.orchestra.run.smithi167.stdout: "zonegroup": "4f1f719d-9648-417b-b10c-e5d939825419", 2023-10-05T19:28:31.330 INFO:teuthology.orchestra.run.smithi167.stdout: "placement_rule": "default-placement", 2023-10-05T19:28:31.330 INFO:teuthology.orchestra.run.smithi167.stdout: "explicit_placement": { 2023-10-05T19:28:31.330 INFO:teuthology.orchestra.run.smithi167.stdout: "data_pool": "", 2023-10-05T19:28:31.331 INFO:teuthology.orchestra.run.smithi167.stdout: "data_extra_pool": "", 2023-10-05T19:28:31.331 INFO:teuthology.orchestra.run.smithi167.stdout: "index_pool": "" 2023-10-05T19:28:31.331 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2023-10-05T19:28:31.331 INFO:teuthology.orchestra.run.smithi167.stdout: "id": "9a9df965-6612-45d4-a45f-6e3c889bacc2.4223.1", 2023-10-05T19:28:31.331 INFO:teuthology.orchestra.run.smithi167.stdout: "marker": "9a9df965-6612-45d4-a45f-6e3c889bacc2.4223.1", 2023-10-05T19:28:31.331 INFO:teuthology.orchestra.run.smithi167.stdout: "index_type": "Normal", 2023-10-05T19:28:31.331 INFO:teuthology.orchestra.run.smithi167.stdout: "owner": "foo", 2023-10-05T19:28:31.331 INFO:teuthology.orchestra.run.smithi167.stdout: "ver": "0#1,1#1,2#1,3#1,4#1,5#1,6#1,7#1,8#1,9#1,10#1", 2023-10-05T19:28:31.331 INFO:teuthology.orchestra.run.smithi167.stdout: "master_ver": "0#0,1#0,2#0,3#0,4#0,5#0,6#0,7#0,8#0,9#0,10#0", 2023-10-05T19:28:31.331 INFO:teuthology.orchestra.run.smithi167.stdout: "mtime": "2023-10-05T19:28:30.258873Z", 2023-10-05T19:28:31.331 INFO:teuthology.orchestra.run.smithi167.stdout: "creation_time": "2023-10-05T19:28:30.250836Z", 2023-10-05T19:28:31.332 INFO:teuthology.orchestra.run.smithi167.stdout: "max_marker": "0#,1#,2#,3#,4#,5#,6#,7#,8#,9#,10#", 2023-10-05T19:28:31.332 INFO:teuthology.orchestra.run.smithi167.stdout: "usage": {}, 2023-10-05T19:28:31.332 INFO:teuthology.orchestra.run.smithi167.stdout: "bucket_quota": { 2023-10-05T19:28:31.332 INFO:teuthology.orchestra.run.smithi167.stdout: "enabled": false, 2023-10-05T19:28:31.332 INFO:teuthology.orchestra.run.smithi167.stdout: "check_on_raw": false, 2023-10-05T19:28:31.332 INFO:teuthology.orchestra.run.smithi167.stdout: "max_size": -1, 2023-10-05T19:28:31.332 INFO:teuthology.orchestra.run.smithi167.stdout: "max_size_kb": 0, 2023-10-05T19:28:31.332 INFO:teuthology.orchestra.run.smithi167.stdout: "max_objects": -1 2023-10-05T19:28:31.332 INFO:teuthology.orchestra.run.smithi167.stdout: } 2023-10-05T19:28:31.332 INFO:teuthology.orchestra.run.smithi167.stdout:} 2023-10-05T19:28:31.333 DEBUG:tasks.util.rgw: json result: {'bucket': 'myfoo', 'num_shards': 11, 'tenant': '', 'zonegroup': '4f1f719d-9648-417b-b10c-e5d939825419', 'placement_rule': 'default-placement', 'explicit_placement': {'data_pool': '', 'data_extra_pool': '', 'index_pool': ''}, 'id': '9a9df965-6612-45d4-a45f-6e3c889bacc2.4223.1', 'marker': '9a9df965-6612-45d4-a45f-6e3c889bacc2.4223.1', 'index_type': 'Normal', 'owner': 'foo', 'ver': '0#1,1#1,2#1,3#1,4#1,5#1,6#1,7#1,8#1,9#1,10#1', 'master_ver': '0#0,1#0,2#0,3#0,4#0,5#0,6#0,7#0,8#0,9#0,10#0', 'mtime': '2023-10-05T19:28:30.258873Z', 'creation_time': '2023-10-05T19:28:30.250836Z', 'max_marker': '0#,1#,2#,3#,4#,5#,6#,7#,8#,9#,10#', 'usage': {}, 'bucket_quota': {'enabled': False, 'check_on_raw': False, 'max_size': -1, 'max_size_kb': 0, 'max_objects': -1}} 2023-10-05T19:28:31.333 INFO:tasks.util.rgw:rgwadmin: client.0 : ['bucket', 'stats', '--uid', 'foo'] 2023-10-05T19:28:31.334 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'ceph', 'bucket', 'stats', '--uid', 'foo'] 2023-10-05T19:28:31.334 DEBUG:teuthology.orchestra.run.smithi167:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster ceph bucket stats --uid foo 2023-10-05T19:28:31.373 INFO:teuthology.orchestra.run.smithi167.stderr:ignoring --setuser ceph since I am not root 2023-10-05T19:28:31.373 INFO:teuthology.orchestra.run.smithi167.stderr:ignoring --setgroup ceph since I am not root 2023-10-05T19:28:31.374 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.370+0000 7fc9166f8e40 1 Processor -- start 2023-10-05T19:28:31.374 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.370+0000 7fc9166f8e40 1 -- start start 2023-10-05T19:28:31.374 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.370+0000 7fc9166f8e40 1 --2- >> v2:172.21.15.118:3300/0 conn(0x55c8b6e017e0 0x55c8b6e006d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-05T19:28:31.374 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.370+0000 7fc9166f8e40 1 -- --> v1:172.21.15.118:6789/0 -- auth(proto 0 26 bytes epoch 0) v1 -- 0x55c8b6d4dbd0 con 0x55c8b6e01410 2023-10-05T19:28:31.375 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.370+0000 7fc9166f8e40 1 -- --> v2:172.21.15.118:3300/0 -- mon_getmap magic: 0 v1 -- 0x55c8b6d0aee0 con 0x55c8b6e017e0 2023-10-05T19:28:31.375 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.370+0000 7fc90bfff700 1 --2- >> v2:172.21.15.118:3300/0 conn(0x55c8b6e017e0 0x55c8b6e006d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-05T19:28:31.375 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.370+0000 7fc91422d700 1 --1- >> v1:172.21.15.118:6789/0 conn(0x55c8b6e01410 0x55c8b6dfffc0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.118:6789/0 says I am v1:172.21.15.167:56584/0 (socket says 172.21.15.167:56584) 2023-10-05T19:28:31.376 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.370+0000 7fc91422d700 1 -- 172.21.15.167:0/2544621096 learned_addr learned my addr 172.21.15.167:0/2544621096 (peer_addr_for_me v1:172.21.15.167:0/0) 2023-10-05T19:28:31.376 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.370+0000 7fc90bfff700 1 -- 172.21.15.167:0/2544621096 >> v1:172.21.15.118:6789/0 conn(0x55c8b6e01410 legacy=0x55c8b6dfffc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-05T19:28:31.376 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.370+0000 7fc90bfff700 1 -- 172.21.15.167:0/2544621096 --> v2:172.21.15.118:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55c8b6e00cd0 con 0x55c8b6e017e0 2023-10-05T19:28:31.377 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.370+0000 7fc90bfff700 1 --2- 172.21.15.167:0/2544621096 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55c8b6e017e0 0x55c8b6e006d0 secure :-1 s=READY pgs=327 cs=0 l=1 rev1=1 crypto rx=0x7fc8fc00a5d0 tx=0x7fc8fc005a50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bfb498d05469af9a server_cookie=0 in_seq=0 out_seq=0 2023-10-05T19:28:31.377 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.370+0000 7fc913a2c700 1 -- 172.21.15.167:0/2544621096 <== mon.0 v2:172.21.15.118:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fc8fc0086a0 con 0x55c8b6e017e0 2023-10-05T19:28:31.377 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.370+0000 7fc913a2c700 1 -- 172.21.15.167:0/2544621096 <== mon.0 v2:172.21.15.118:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc8fc008800 con 0x55c8b6e017e0 2023-10-05T19:28:31.377 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.370+0000 7fc913a2c700 1 -- 172.21.15.167:0/2544621096 <== mon.0 v2:172.21.15.118:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fc8fc008ad0 con 0x55c8b6e017e0 2023-10-05T19:28:31.378 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.370+0000 7fc9166f8e40 1 -- 172.21.15.167:0/2544621096 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55c8b6e017e0 msgr2=0x55c8b6e006d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-05T19:28:31.378 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.370+0000 7fc9166f8e40 1 --2- 172.21.15.167:0/2544621096 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55c8b6e017e0 0x55c8b6e006d0 secure :-1 s=READY pgs=327 cs=0 l=1 rev1=1 crypto rx=0x7fc8fc00a5d0 tx=0x7fc8fc005a50 comp rx=0 tx=0).stop 2023-10-05T19:28:31.378 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.374+0000 7fc9166f8e40 1 -- 172.21.15.167:0/2544621096 shutdown_connections 2023-10-05T19:28:31.378 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.374+0000 7fc9166f8e40 1 --2- 172.21.15.167:0/2544621096 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55c8b6e017e0 0x55c8b6e006d0 unknown :-1 s=CLOSED pgs=327 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-05T19:28:31.378 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.374+0000 7fc9166f8e40 1 -- 172.21.15.167:0/2544621096 >> 172.21.15.167:0/2544621096 conn(0x55c8b6dfb720 msgr2=0x55c8b6dfdb30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-05T19:28:31.379 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.374+0000 7fc9166f8e40 1 -- 172.21.15.167:0/2544621096 shutdown_connections 2023-10-05T19:28:31.379 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.374+0000 7fc9166f8e40 1 -- 172.21.15.167:0/2544621096 wait complete. 2023-10-05T19:28:31.380 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.374+0000 7fc9166f8e40 1 Processor -- start 2023-10-05T19:28:31.380 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.374+0000 7fc9166f8e40 1 -- start start 2023-10-05T19:28:31.380 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.374+0000 7fc9166f8e40 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55c8b6e01410 0x55c8b6e22160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-05T19:28:31.380 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.374+0000 7fc9166f8e40 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c8b6ccee30 con 0x55c8b6e01410 2023-10-05T19:28:31.381 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.374+0000 7fc91422d700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55c8b6e01410 0x55c8b6e22160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-05T19:28:31.381 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.374+0000 7fc91422d700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55c8b6e01410 0x55c8b6e22160 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.118:3300/0 says I am v2:172.21.15.167:58184/0 (socket says 172.21.15.167:58184) 2023-10-05T19:28:31.381 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.374+0000 7fc91422d700 1 -- 172.21.15.167:0/1671850004 learned_addr learned my addr 172.21.15.167:0/1671850004 (peer_addr_for_me v2:172.21.15.167:0/0) 2023-10-05T19:28:31.382 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.374+0000 7fc91422d700 1 -- 172.21.15.167:0/1671850004 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc8fc005cf0 con 0x55c8b6e01410 2023-10-05T19:28:31.382 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.374+0000 7fc91422d700 1 --2- 172.21.15.167:0/1671850004 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55c8b6e01410 0x55c8b6e22160 secure :-1 s=READY pgs=328 cs=0 l=1 rev1=1 crypto rx=0x7fc904001c30 tx=0x7fc9040017e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b2cc8232d5c08056 server_cookie=0 in_seq=0 out_seq=0 2023-10-05T19:28:31.383 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.378+0000 7fc912a2a700 1 -- 172.21.15.167:0/1671850004 <== mon.0 v2:172.21.15.118:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fc90400bbf0 con 0x55c8b6e01410 2023-10-05T19:28:31.383 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.378+0000 7fc912a2a700 1 -- 172.21.15.167:0/1671850004 <== mon.0 v2:172.21.15.118:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc90400bd50 con 0x55c8b6e01410 2023-10-05T19:28:31.383 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.378+0000 7fc912a2a700 1 -- 172.21.15.167:0/1671850004 <== mon.0 v2:172.21.15.118:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fc90400e470 con 0x55c8b6e01410 2023-10-05T19:28:31.383 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.378+0000 7fc9166f8e40 1 -- 172.21.15.167:0/1671850004 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55c8b6e01410 msgr2=0x55c8b6e22160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-05T19:28:31.383 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.378+0000 7fc9166f8e40 1 --2- 172.21.15.167:0/1671850004 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55c8b6e01410 0x55c8b6e22160 secure :-1 s=READY pgs=328 cs=0 l=1 rev1=1 crypto rx=0x7fc904001c30 tx=0x7fc9040017e0 comp rx=0 tx=0).stop 2023-10-05T19:28:31.383 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.378+0000 7fc9166f8e40 1 -- 172.21.15.167:0/1671850004 shutdown_connections 2023-10-05T19:28:31.383 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.378+0000 7fc9166f8e40 1 --2- 172.21.15.167:0/1671850004 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55c8b6e01410 0x55c8b6e22160 unknown :-1 s=CLOSED pgs=328 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-05T19:28:31.384 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.378+0000 7fc9166f8e40 1 -- 172.21.15.167:0/1671850004 >> 172.21.15.167:0/1671850004 conn(0x55c8b6dfb720 msgr2=0x55c8b6e1dd20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-05T19:28:31.384 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.378+0000 7fc9166f8e40 1 -- 172.21.15.167:0/1671850004 shutdown_connections 2023-10-05T19:28:31.384 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.378+0000 7fc9166f8e40 1 -- 172.21.15.167:0/1671850004 wait complete. 2023-10-05T19:28:31.385 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.378+0000 7fc9166f8e40 1 Processor -- start 2023-10-05T19:28:31.385 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.378+0000 7fc9166f8e40 1 -- start start 2023-10-05T19:28:31.385 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.378+0000 7fc9166f8e40 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55c8b6e01410 0x55c8b6e28f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-05T19:28:31.385 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.378+0000 7fc9166f8e40 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c8b6ca1ed0 con 0x55c8b6e01410 2023-10-05T19:28:31.385 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.378+0000 7fc91422d700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55c8b6e01410 0x55c8b6e28f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-05T19:28:31.385 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.378+0000 7fc91422d700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55c8b6e01410 0x55c8b6e28f30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.118:3300/0 says I am v2:172.21.15.167:58200/0 (socket says 172.21.15.167:58200) 2023-10-05T19:28:31.385 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.378+0000 7fc91422d700 1 -- 172.21.15.167:0/3055819727 learned_addr learned my addr 172.21.15.167:0/3055819727 (peer_addr_for_me v2:172.21.15.167:0/0) 2023-10-05T19:28:31.385 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.378+0000 7fc91422d700 1 -- 172.21.15.167:0/3055819727 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55c8b6d4dbd0 con 0x55c8b6e01410 2023-10-05T19:28:31.386 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.378+0000 7fc91422d700 1 --2- 172.21.15.167:0/3055819727 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55c8b6e01410 0x55c8b6e28f30 secure :-1 s=READY pgs=329 cs=0 l=1 rev1=1 crypto rx=0x7fc904000c00 tx=0x7fc904016000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-05T19:28:31.386 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.378+0000 7fc911227700 1 -- 172.21.15.167:0/3055819727 <== mon.0 v2:172.21.15.118:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fc90401f070 con 0x55c8b6e01410 2023-10-05T19:28:31.386 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.378+0000 7fc911227700 1 -- 172.21.15.167:0/3055819727 <== mon.0 v2:172.21.15.118:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc904019690 con 0x55c8b6e01410 2023-10-05T19:28:31.387 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.378+0000 7fc9166f8e40 1 -- 172.21.15.167:0/3055819727 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x55c8b6e29470 con 0x55c8b6e01410 2023-10-05T19:28:31.387 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.378+0000 7fc911227700 1 -- 172.21.15.167:0/3055819727 <== mon.0 v2:172.21.15.118:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fc904019840 con 0x55c8b6e01410 2023-10-05T19:28:31.387 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.378+0000 7fc9166f8e40 1 -- 172.21.15.167:0/3055819727 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55c8b6e298b0 con 0x55c8b6e01410 2023-10-05T19:28:31.387 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.382+0000 7fc9166f8e40 20 rados->read ofs=0 len=0 2023-10-05T19:28:31.387 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.382+0000 7fc911227700 1 -- 172.21.15.167:0/3055819727 <== mon.0 v2:172.21.15.118:3300/0 4 ==== mgrmap(e 5) v1 ==== 40329+0+0 (secure 0 0 0) 0x7fc904022020 con 0x55c8b6e01410 2023-10-05T19:28:31.388 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.382+0000 7fc911227700 1 --2- 172.21.15.167:0/3055819727 >> [v2:172.21.15.167:6824/29889,v1:172.21.15.167:6825/29889] conn(0x7fc8f4030c40 0x7fc8f40330f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-05T19:28:31.388 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.382+0000 7fc911227700 1 -- 172.21.15.167:0/3055819727 <== mon.0 v2:172.21.15.118:3300/0 5 ==== osd_map(30..30 src has 1..30) v4 ==== 6057+0+0 (secure 0 0 0) 0x7fc904044e30 con 0x55c8b6e01410 2023-10-05T19:28:31.388 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.382+0000 7fc90bfff700 1 --2- 172.21.15.167:0/3055819727 >> [v2:172.21.15.167:6824/29889,v1:172.21.15.167:6825/29889] conn(0x7fc8f4030c40 0x7fc8f40330f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-05T19:28:31.388 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.382+0000 7fc9166f8e40 1 --2- 172.21.15.167:0/3055819727 >> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] conn(0x55c8b6e017e0 0x55c8b6e349c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-05T19:28:31.388 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.382+0000 7fc9166f8e40 1 -- 172.21.15.167:0/3055819727 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:1 5.12 5:49953fa1:::default.realm:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x55c8b6e35070 con 0x55c8b6e017e0 2023-10-05T19:28:31.388 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.382+0000 7fc914a2e700 1 --2- 172.21.15.167:0/3055819727 >> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] conn(0x55c8b6e017e0 0x55c8b6e349c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-05T19:28:31.389 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.382+0000 7fc90bfff700 1 --2- 172.21.15.167:0/3055819727 >> [v2:172.21.15.167:6824/29889,v1:172.21.15.167:6825/29889] conn(0x7fc8f4030c40 0x7fc8f40330f0 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7fc8fc005a90 tx=0x7fc8fc015000 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-05T19:28:31.389 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.382+0000 7fc914a2e700 1 --2- 172.21.15.167:0/3055819727 >> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] conn(0x55c8b6e017e0 0x55c8b6e349c0 crc :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-05T19:28:31.389 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.382+0000 7fc914a2e700 1 -- 172.21.15.167:0/3055819727 <== osd.1 v2:172.21.15.167:6800/30035 1 ==== osd_op_reply(1 default.realm [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 ==== 157+0+0 (crc 0 0 0) 0x7fc90c00a5a0 con 0x55c8b6e017e0 2023-10-05T19:28:31.389 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.382+0000 7fc9166f8e40 20 rados_obj.operate() r=-2 bl.length=0 2023-10-05T19:28:31.389 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.382+0000 7fc9166f8e40 20 realm 2023-10-05T19:28:31.390 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.382+0000 7fc9166f8e40 20 rados->read ofs=0 len=0 2023-10-05T19:28:31.390 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.382+0000 7fc9166f8e40 1 -- 172.21.15.167:0/3055819727 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:2 5.12 5:49953fa1:::default.realm:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x55c8b6e32020 con 0x55c8b6e017e0 2023-10-05T19:28:31.390 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.382+0000 7fc914a2e700 1 -- 172.21.15.167:0/3055819727 <== osd.1 v2:172.21.15.167:6800/30035 2 ==== osd_op_reply(2 default.realm [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 ==== 157+0+0 (crc 0 0 0) 0x7fc90c00a5a0 con 0x55c8b6e017e0 2023-10-05T19:28:31.390 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.382+0000 7fc9166f8e40 20 rados_obj.operate() r=-2 bl.length=0 2023-10-05T19:28:31.391 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.382+0000 7fc9166f8e40 4 RGWPeriod::init failed to init realm id : (2) No such file or directory 2023-10-05T19:28:31.391 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.382+0000 7fc9166f8e40 20 rados->read ofs=0 len=0 2023-10-05T19:28:31.391 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.382+0000 7fc9166f8e40 1 -- 172.21.15.167:0/3055819727 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:3 5.12 5:49953fa1:::default.realm:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x55c8b6e35b50 con 0x55c8b6e017e0 2023-10-05T19:28:31.391 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.386+0000 7fc914a2e700 1 -- 172.21.15.167:0/3055819727 <== osd.1 v2:172.21.15.167:6800/30035 3 ==== osd_op_reply(3 default.realm [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 ==== 157+0+0 (crc 0 0 0) 0x7fc90c00a5a0 con 0x55c8b6e017e0 2023-10-05T19:28:31.391 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.386+0000 7fc9166f8e40 20 rados_obj.operate() r=-2 bl.length=0 2023-10-05T19:28:31.391 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.386+0000 7fc9166f8e40 20 rados->read ofs=0 len=0 2023-10-05T19:28:31.391 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.386+0000 7fc9166f8e40 1 -- 172.21.15.167:0/3055819727 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:4 5.f 5:f43b6ece:::zone_names.default:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x55c8b6e35730 con 0x55c8b6e017e0 2023-10-05T19:28:31.392 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.386+0000 7fc914a2e700 1 -- 172.21.15.167:0/3055819727 <== osd.1 v2:172.21.15.167:6800/30035 4 ==== osd_op_reply(4 zone_names.default [read 0~46 out=46b] v0'0 uv1 ondisk = 0) v8 ==== 162+0+46 (crc 0 0 0) 0x7fc90c00a5a0 con 0x55c8b6e017e0 2023-10-05T19:28:31.392 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.386+0000 7fc9166f8e40 20 rados_obj.operate() r=0 bl.length=46 2023-10-05T19:28:31.392 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.386+0000 7fc9166f8e40 20 rados->read ofs=0 len=0 2023-10-05T19:28:31.392 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.386+0000 7fc9166f8e40 1 -- 172.21.15.167:0/3055819727 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:5 5.1d 5:b8374eb4:::zone_info.9a9df965-6612-45d4-a45f-6e3c889bacc2:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x55c8b6e365c0 con 0x55c8b6e017e0 2023-10-05T19:28:31.392 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.386+0000 7fc914a2e700 1 -- 172.21.15.167:0/3055819727 <== osd.1 v2:172.21.15.167:6800/30035 5 ==== osd_op_reply(5 zone_info.9a9df965-6612-45d4-a45f-6e3c889bacc2 [read 0~874 out=874b] v0'0 uv2 ondisk = 0) v8 ==== 190+0+874 (crc 0 0 0) 0x7fc90c00a5a0 con 0x55c8b6e017e0 2023-10-05T19:28:31.393 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.386+0000 7fc9166f8e40 20 rados_obj.operate() r=0 bl.length=874 2023-10-05T19:28:31.393 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.386+0000 7fc9166f8e40 20 searching for the correct realm 2023-10-05T19:28:31.393 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.386+0000 7fc9166f8e40 1 --2- 172.21.15.167:0/3055819727 >> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] conn(0x55c8b6e38410 0x55c8b6e3a7f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-05T19:28:31.393 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.386+0000 7fc9166f8e40 1 -- 172.21.15.167:0/3055819727 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:6 5.0 5:00000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x55c8b6e3ad30 con 0x55c8b6e38410 2023-10-05T19:28:31.393 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.386+0000 7fc91422d700 1 --2- 172.21.15.167:0/3055819727 >> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] conn(0x55c8b6e38410 0x55c8b6e3a7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-05T19:28:31.393 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.386+0000 7fc91422d700 1 --2- 172.21.15.167:0/3055819727 >> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] conn(0x55c8b6e38410 0x55c8b6e3a7f0 crc :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-05T19:28:31.395 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.390+0000 7fc91422d700 1 -- 172.21.15.167:0/3055819727 <== osd.3 v2:172.21.15.167:6808/30036 1 ==== osd_op_reply(6 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fc90400d7f0 con 0x55c8b6e38410 2023-10-05T19:28:31.396 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.390+0000 7fc91422d700 1 --2- 172.21.15.167:0/3055819727 >> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] conn(0x7fc904044040 0x7fc9040420e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-05T19:28:31.396 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.390+0000 7fc91422d700 1 -- 172.21.15.167:0/3055819727 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:7 5.10 5:08000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x55c8b6e3ad30 con 0x7fc904044040 2023-10-05T19:28:31.396 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.390+0000 7fc90bfff700 1 --2- 172.21.15.167:0/3055819727 >> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] conn(0x7fc904044040 0x7fc9040420e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-05T19:28:31.396 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.390+0000 7fc90bfff700 1 --2- 172.21.15.167:0/3055819727 >> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] conn(0x7fc904044040 0x7fc9040420e0 crc :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-05T19:28:31.397 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.390+0000 7fc90bfff700 1 -- 172.21.15.167:0/3055819727 <== osd.2 v2:172.21.15.167:6816/30037 1 ==== osd_op_reply(7 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fc8fc008ae0 con 0x7fc904044040 2023-10-05T19:28:31.397 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.390+0000 7fc90bfff700 1 --2- 172.21.15.167:0/3055819727 >> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] conn(0x7fc8fc007030 0x7fc8fc01c920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-05T19:28:31.397 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.390+0000 7fc90bfff700 1 -- 172.21.15.167:0/3055819727 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:8 5.8 5:10000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x55c8b6e3ad30 con 0x7fc8fc007030 2023-10-05T19:28:31.398 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.390+0000 7fc914a2e700 1 --2- 172.21.15.167:0/3055819727 >> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] conn(0x7fc8fc007030 0x7fc8fc01c920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-05T19:28:31.398 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.394+0000 7fc914a2e700 1 --2- 172.21.15.167:0/3055819727 >> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] conn(0x7fc8fc007030 0x7fc8fc01c920 crc :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-05T19:28:31.399 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.394+0000 7fc914a2e700 1 -- 172.21.15.167:0/3055819727 <== osd.0 v2:172.21.15.118:6800/27690 1 ==== osd_op_reply(8 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fc90c00a5a0 con 0x7fc8fc007030 2023-10-05T19:28:31.399 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.394+0000 7fc914a2e700 1 -- 172.21.15.167:0/3055819727 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:9 5.18 5:18000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x55c8b6e3ad30 con 0x55c8b6e017e0 2023-10-05T19:28:31.400 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.394+0000 7fc914a2e700 1 -- 172.21.15.167:0/3055819727 <== osd.1 v2:172.21.15.167:6800/30035 6 ==== osd_op_reply(9 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fc90c00a5a0 con 0x55c8b6e017e0 2023-10-05T19:28:31.400 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.394+0000 7fc914a2e700 1 -- 172.21.15.167:0/3055819727 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:10 5.4 5:20000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x55c8b6e3ad30 con 0x55c8b6e017e0 2023-10-05T19:28:31.401 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.394+0000 7fc914a2e700 1 -- 172.21.15.167:0/3055819727 <== osd.1 v2:172.21.15.167:6800/30035 7 ==== osd_op_reply(10 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fc90c00a5a0 con 0x55c8b6e017e0 2023-10-05T19:28:31.401 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.394+0000 7fc914a2e700 1 -- 172.21.15.167:0/3055819727 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:11 5.14 5:28000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x55c8b6e3ad30 con 0x7fc8fc007030 2023-10-05T19:28:31.402 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.398+0000 7fc914a2e700 1 -- 172.21.15.167:0/3055819727 <== osd.0 v2:172.21.15.118:6800/27690 2 ==== osd_op_reply(11 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fc90c00a5a0 con 0x7fc8fc007030 2023-10-05T19:28:31.402 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.398+0000 7fc914a2e700 1 -- 172.21.15.167:0/3055819727 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:12 5.c 5:30000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x55c8b6e3ad30 con 0x7fc8fc007030 2023-10-05T19:28:31.403 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.398+0000 7fc914a2e700 1 -- 172.21.15.167:0/3055819727 <== osd.0 v2:172.21.15.118:6800/27690 3 ==== osd_op_reply(12 [pgnls start_epoch 30 out=112b] v23'1 uv1 ondisk = 1) v8 ==== 144+0+112 (crc 0 0 0) 0x7fc90c00a5a0 con 0x7fc8fc007030 2023-10-05T19:28:31.403 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.398+0000 7fc914a2e700 1 -- 172.21.15.167:0/3055819727 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:13 5.1c 5:38000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x55c8b6e3ad30 con 0x7fc904044040 2023-10-05T19:28:31.404 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.398+0000 7fc90bfff700 1 -- 172.21.15.167:0/3055819727 <== osd.2 v2:172.21.15.167:6816/30037 2 ==== osd_op_reply(13 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fc8fc008ae0 con 0x7fc904044040 2023-10-05T19:28:31.404 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.398+0000 7fc90bfff700 1 -- 172.21.15.167:0/3055819727 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:14 5.2 5:40000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x55c8b6e3ad30 con 0x55c8b6e38410 2023-10-05T19:28:31.404 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.398+0000 7fc91422d700 1 -- 172.21.15.167:0/3055819727 <== osd.3 v2:172.21.15.167:6808/30036 2 ==== osd_op_reply(14 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fc90400d7f0 con 0x55c8b6e38410 2023-10-05T19:28:31.404 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.398+0000 7fc91422d700 1 -- 172.21.15.167:0/3055819727 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:15 5.12 5:48000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x55c8b6e3ad30 con 0x55c8b6e017e0 2023-10-05T19:28:31.405 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.398+0000 7fc914a2e700 1 -- 172.21.15.167:0/3055819727 <== osd.1 v2:172.21.15.167:6800/30035 8 ==== osd_op_reply(15 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fc90c00a5a0 con 0x55c8b6e017e0 2023-10-05T19:28:31.405 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.398+0000 7fc914a2e700 1 -- 172.21.15.167:0/3055819727 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:16 5.a 5:50000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x55c8b6e3ad30 con 0x7fc8fc007030 2023-10-05T19:28:31.406 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.402+0000 7fc914a2e700 1 -- 172.21.15.167:0/3055819727 <== osd.0 v2:172.21.15.118:6800/27690 4 ==== osd_op_reply(16 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fc90c00a5a0 con 0x7fc8fc007030 2023-10-05T19:28:31.407 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.402+0000 7fc914a2e700 1 -- 172.21.15.167:0/3055819727 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:17 5.1a 5:58000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x55c8b6e3ad30 con 0x55c8b6e017e0 2023-10-05T19:28:31.407 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.402+0000 7fc914a2e700 1 -- 172.21.15.167:0/3055819727 <== osd.1 v2:172.21.15.167:6800/30035 9 ==== osd_op_reply(17 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fc90c00a5a0 con 0x55c8b6e017e0 2023-10-05T19:28:31.407 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.402+0000 7fc914a2e700 1 -- 172.21.15.167:0/3055819727 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:18 5.6 5:60000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x55c8b6e3ad30 con 0x7fc904044040 2023-10-05T19:28:31.408 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.402+0000 7fc90bfff700 1 -- 172.21.15.167:0/3055819727 <== osd.2 v2:172.21.15.167:6816/30037 3 ==== osd_op_reply(18 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fc8fc008ae0 con 0x7fc904044040 2023-10-05T19:28:31.408 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.402+0000 7fc90bfff700 1 -- 172.21.15.167:0/3055819727 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:19 5.16 5:68000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x55c8b6e3ad30 con 0x55c8b6e38410 2023-10-05T19:28:31.409 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.402+0000 7fc91422d700 1 -- 172.21.15.167:0/3055819727 <== osd.3 v2:172.21.15.167:6808/30036 3 ==== osd_op_reply(19 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fc90400d7f0 con 0x55c8b6e38410 2023-10-05T19:28:31.409 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.402+0000 7fc91422d700 1 -- 172.21.15.167:0/3055819727 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:20 5.e 5:70000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x55c8b6e3ad30 con 0x55c8b6e38410 2023-10-05T19:28:31.409 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.402+0000 7fc91422d700 1 -- 172.21.15.167:0/3055819727 <== osd.3 v2:172.21.15.167:6808/30036 4 ==== osd_op_reply(20 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fc90400d7f0 con 0x55c8b6e38410 2023-10-05T19:28:31.410 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.402+0000 7fc91422d700 1 -- 172.21.15.167:0/3055819727 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:21 5.1e 5:78000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x55c8b6e3ad30 con 0x7fc8fc007030 2023-10-05T19:28:31.410 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.406+0000 7fc914a2e700 1 -- 172.21.15.167:0/3055819727 <== osd.0 v2:172.21.15.118:6800/27690 5 ==== osd_op_reply(21 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fc90c00a5a0 con 0x7fc8fc007030 2023-10-05T19:28:31.411 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.406+0000 7fc914a2e700 1 -- 172.21.15.167:0/3055819727 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:22 5.1 5:80000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x55c8b6e3ad30 con 0x55c8b6e38410 2023-10-05T19:28:31.411 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.406+0000 7fc91422d700 1 -- 172.21.15.167:0/3055819727 <== osd.3 v2:172.21.15.167:6808/30036 5 ==== osd_op_reply(22 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fc90400d7f0 con 0x55c8b6e38410 2023-10-05T19:28:31.412 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.406+0000 7fc91422d700 1 -- 172.21.15.167:0/3055819727 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:23 5.11 5:88000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x55c8b6e3ad30 con 0x55c8b6e38410 2023-10-05T19:28:31.412 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.406+0000 7fc91422d700 1 -- 172.21.15.167:0/3055819727 <== osd.3 v2:172.21.15.167:6808/30036 6 ==== osd_op_reply(23 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fc90400d7f0 con 0x55c8b6e38410 2023-10-05T19:28:31.412 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.406+0000 7fc91422d700 1 -- 172.21.15.167:0/3055819727 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:24 5.9 5:90000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x55c8b6e3ad30 con 0x55c8b6e017e0 2023-10-05T19:28:31.413 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.406+0000 7fc914a2e700 1 -- 172.21.15.167:0/3055819727 <== osd.1 v2:172.21.15.167:6800/30035 10 ==== osd_op_reply(24 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fc90c00a5a0 con 0x55c8b6e017e0 2023-10-05T19:28:31.413 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.406+0000 7fc914a2e700 1 -- 172.21.15.167:0/3055819727 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:25 5.19 5:98000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x55c8b6e3ad30 con 0x55c8b6e017e0 2023-10-05T19:28:31.414 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.410+0000 7fc914a2e700 1 -- 172.21.15.167:0/3055819727 <== osd.1 v2:172.21.15.167:6800/30035 11 ==== osd_op_reply(25 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fc90c00a5a0 con 0x55c8b6e017e0 2023-10-05T19:28:31.414 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.410+0000 7fc914a2e700 1 -- 172.21.15.167:0/3055819727 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:26 5.5 5:a0000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x55c8b6e3ad30 con 0x7fc8fc007030 2023-10-05T19:28:31.415 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.410+0000 7fc914a2e700 1 -- 172.21.15.167:0/3055819727 <== osd.0 v2:172.21.15.118:6800/27690 6 ==== osd_op_reply(26 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fc90c00a5a0 con 0x7fc8fc007030 2023-10-05T19:28:31.415 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.410+0000 7fc914a2e700 1 -- 172.21.15.167:0/3055819727 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:27 5.15 5:a8000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x55c8b6e3ad30 con 0x55c8b6e38410 2023-10-05T19:28:31.416 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.410+0000 7fc91422d700 1 -- 172.21.15.167:0/3055819727 <== osd.3 v2:172.21.15.167:6808/30036 7 ==== osd_op_reply(27 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fc90400d7f0 con 0x55c8b6e38410 2023-10-05T19:28:31.416 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.410+0000 7fc91422d700 1 -- 172.21.15.167:0/3055819727 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:28 5.d 5:b0000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x55c8b6e3ad30 con 0x7fc904044040 2023-10-05T19:28:31.417 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.410+0000 7fc90bfff700 1 -- 172.21.15.167:0/3055819727 <== osd.2 v2:172.21.15.167:6816/30037 4 ==== osd_op_reply(28 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fc8fc008ae0 con 0x7fc904044040 2023-10-05T19:28:31.417 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.410+0000 7fc90bfff700 1 -- 172.21.15.167:0/3055819727 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:29 5.1d 5:b8000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x55c8b6e3ad30 con 0x55c8b6e017e0 2023-10-05T19:28:31.417 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.410+0000 7fc914a2e700 1 -- 172.21.15.167:0/3055819727 <== osd.1 v2:172.21.15.167:6800/30035 12 ==== osd_op_reply(29 [pgnls start_epoch 30 out=107b] v30'2 uv2 ondisk = 1) v8 ==== 144+0+107 (crc 0 0 0) 0x7fc90c00a5a0 con 0x55c8b6e017e0 2023-10-05T19:28:31.417 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.410+0000 7fc914a2e700 1 -- 172.21.15.167:0/3055819727 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:30 5.3 5:c0000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x55c8b6e3ad30 con 0x7fc8fc007030 2023-10-05T19:28:31.418 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.414+0000 7fc914a2e700 1 -- 172.21.15.167:0/3055819727 <== osd.0 v2:172.21.15.118:6800/27690 7 ==== osd_op_reply(30 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fc90c00a5a0 con 0x7fc8fc007030 2023-10-05T19:28:31.418 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.414+0000 7fc914a2e700 1 -- 172.21.15.167:0/3055819727 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:31 5.13 5:c8000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x55c8b6e3ad30 con 0x55c8b6e38410 2023-10-05T19:28:31.421 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.414+0000 7fc91422d700 1 -- 172.21.15.167:0/3055819727 <== osd.3 v2:172.21.15.167:6808/30036 8 ==== osd_op_reply(31 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fc90400d7f0 con 0x55c8b6e38410 2023-10-05T19:28:31.421 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.414+0000 7fc91422d700 1 -- 172.21.15.167:0/3055819727 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:32 5.b 5:d0000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x55c8b6e3ad30 con 0x7fc8fc007030 2023-10-05T19:28:31.422 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.414+0000 7fc914a2e700 1 -- 172.21.15.167:0/3055819727 <== osd.0 v2:172.21.15.118:6800/27690 8 ==== osd_op_reply(32 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fc90c00a5a0 con 0x7fc8fc007030 2023-10-05T19:28:31.422 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.414+0000 7fc914a2e700 1 -- 172.21.15.167:0/3055819727 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:33 5.1b 5:d8000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x55c8b6e3ad30 con 0x7fc904044040 2023-10-05T19:28:31.422 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.414+0000 7fc90bfff700 1 -- 172.21.15.167:0/3055819727 <== osd.2 v2:172.21.15.167:6816/30037 5 ==== osd_op_reply(33 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fc8fc008ae0 con 0x7fc904044040 2023-10-05T19:28:31.422 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.414+0000 7fc90bfff700 1 -- 172.21.15.167:0/3055819727 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:34 5.7 5:e0000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x55c8b6e3ad30 con 0x55c8b6e38410 2023-10-05T19:28:31.422 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.418+0000 7fc91422d700 1 -- 172.21.15.167:0/3055819727 <== osd.3 v2:172.21.15.167:6808/30036 9 ==== osd_op_reply(34 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fc90400d7f0 con 0x55c8b6e38410 2023-10-05T19:28:31.422 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.418+0000 7fc91422d700 1 -- 172.21.15.167:0/3055819727 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:35 5.17 5:e8000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x55c8b6e3ad30 con 0x7fc8fc007030 2023-10-05T19:28:31.423 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.418+0000 7fc914a2e700 1 -- 172.21.15.167:0/3055819727 <== osd.0 v2:172.21.15.118:6800/27690 9 ==== osd_op_reply(35 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fc90c00a5a0 con 0x7fc8fc007030 2023-10-05T19:28:31.423 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.418+0000 7fc914a2e700 1 -- 172.21.15.167:0/3055819727 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:36 5.f 5:f0000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x55c8b6e3ad30 con 0x55c8b6e017e0 2023-10-05T19:28:31.424 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.418+0000 7fc914a2e700 1 -- 172.21.15.167:0/3055819727 <== osd.1 v2:172.21.15.167:6800/30035 13 ==== osd_op_reply(36 [pgnls start_epoch 30 out=115b] v23'2 uv2 ondisk = 1) v8 ==== 144+0+115 (crc 0 0 0) 0x7fc90c00a5a0 con 0x55c8b6e017e0 2023-10-05T19:28:31.424 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.418+0000 7fc914a2e700 1 -- 172.21.15.167:0/3055819727 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:37 5.1f 5:f8000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x55c8b6e3ad30 con 0x7fc904044040 2023-10-05T19:28:31.425 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.418+0000 7fc90bfff700 1 -- 172.21.15.167:0/3055819727 <== osd.2 v2:172.21.15.167:6816/30037 6 ==== osd_op_reply(37 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fc8fc008ae0 con 0x7fc904044040 2023-10-05T19:28:31.425 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.418+0000 7fc9166f8e40 20 RGWRados::pool_iterate: got zonegroup_info.4f1f719d-9648-417b-b10c-e5d939825419 2023-10-05T19:28:31.425 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.418+0000 7fc9166f8e40 20 RGWRados::pool_iterate: got zone_info.9a9df965-6612-45d4-a45f-6e3c889bacc2 2023-10-05T19:28:31.425 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.418+0000 7fc9166f8e40 20 RGWRados::pool_iterate: got zone_names.default 2023-10-05T19:28:31.425 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.418+0000 7fc9166f8e40 20 RGWRados::pool_iterate: got zonegroups_names.default 2023-10-05T19:28:31.426 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.418+0000 7fc9166f8e40 20 rados->read ofs=0 len=0 2023-10-05T19:28:31.426 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.418+0000 7fc9166f8e40 1 -- 172.21.15.167:0/3055819727 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:38 5.12 5:49953fa1:::default.realm:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x55c8b6e370e0 con 0x55c8b6e017e0 2023-10-05T19:28:31.426 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.418+0000 7fc914a2e700 1 -- 172.21.15.167:0/3055819727 <== osd.1 v2:172.21.15.167:6800/30035 14 ==== osd_op_reply(38 default.realm [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 ==== 157+0+0 (crc 0 0 0) 0x7fc90c00a5a0 con 0x55c8b6e017e0 2023-10-05T19:28:31.427 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.418+0000 7fc9166f8e40 20 rados_obj.operate() r=-2 bl.length=0 2023-10-05T19:28:31.427 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.418+0000 7fc9166f8e40 20 rados->read ofs=0 len=0 2023-10-05T19:28:31.427 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.418+0000 7fc9166f8e40 1 -- 172.21.15.167:0/3055819727 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:39 5.f 5:f4c53578:::zonegroups_names.default:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x55c8b6e36ce0 con 0x55c8b6e017e0 2023-10-05T19:28:31.427 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.422+0000 7fc914a2e700 1 -- 172.21.15.167:0/3055819727 <== osd.1 v2:172.21.15.167:6800/30035 15 ==== osd_op_reply(39 zonegroups_names.default [read 0~46 out=46b] v0'0 uv2 ondisk = 0) v8 ==== 168+0+46 (crc 0 0 0) 0x7fc90c00a5a0 con 0x55c8b6e017e0 2023-10-05T19:28:31.427 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.422+0000 7fc9166f8e40 20 rados_obj.operate() r=0 bl.length=46 2023-10-05T19:28:31.427 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.422+0000 7fc9166f8e40 20 rados->read ofs=0 len=0 2023-10-05T19:28:31.427 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.422+0000 7fc9166f8e40 1 -- 172.21.15.167:0/3055819727 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:40 5.c 5:30d299a5:::zonegroup_info.4f1f719d-9648-417b-b10c-e5d939825419:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x55c8b6e36cb0 con 0x7fc8fc007030 2023-10-05T19:28:31.428 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.422+0000 7fc914a2e700 1 -- 172.21.15.167:0/3055819727 <== osd.0 v2:172.21.15.118:6800/27690 10 ==== osd_op_reply(40 zonegroup_info.4f1f719d-9648-417b-b10c-e5d939825419 [read 0~362 out=362b] v0'0 uv1 ondisk = 0) v8 ==== 195+0+362 (crc 0 0 0) 0x7fc90c00a5a0 con 0x7fc8fc007030 2023-10-05T19:28:31.428 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.422+0000 7fc9166f8e40 20 rados_obj.operate() r=0 bl.length=362 2023-10-05T19:28:31.428 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.422+0000 7fc9166f8e40 20 rados->read ofs=0 len=0 2023-10-05T19:28:31.429 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.422+0000 7fc9166f8e40 1 -- 172.21.15.167:0/3055819727 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:41 5.13 5:cbbf6c60:::converted:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x55c8b6e37a70 con 0x55c8b6e38410 2023-10-05T19:28:31.429 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.422+0000 7fc91422d700 1 -- 172.21.15.167:0/3055819727 <== osd.3 v2:172.21.15.167:6808/30036 10 ==== osd_op_reply(41 converted [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 ==== 153+0+0 (crc 0 0 0) 0x7fc90400d7f0 con 0x55c8b6e38410 2023-10-05T19:28:31.429 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.422+0000 7fc9166f8e40 20 rados_obj.operate() r=-2 bl.length=0 2023-10-05T19:28:31.429 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.422+0000 7fc9166f8e40 20 rados->read ofs=0 len=0 2023-10-05T19:28:31.429 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.422+0000 7fc9166f8e40 1 -- 172.21.15.167:0/3055819727 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:42 5.12 5:49953fa1:::default.realm:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x55c8b6e3b170 con 0x55c8b6e017e0 2023-10-05T19:28:31.430 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.422+0000 7fc914a2e700 1 -- 172.21.15.167:0/3055819727 <== osd.1 v2:172.21.15.167:6800/30035 16 ==== osd_op_reply(42 default.realm [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 ==== 157+0+0 (crc 0 0 0) 0x7fc90c00a5a0 con 0x55c8b6e017e0 2023-10-05T19:28:31.430 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.422+0000 7fc9166f8e40 20 rados_obj.operate() r=-2 bl.length=0 2023-10-05T19:28:31.430 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.422+0000 7fc9166f8e40 20 rados->read ofs=0 len=0 2023-10-05T19:28:31.430 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.422+0000 7fc9166f8e40 1 -- 172.21.15.167:0/3055819727 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:43 5.f 5:f4c53578:::zonegroups_names.default:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x55c8b6e3b170 con 0x55c8b6e017e0 2023-10-05T19:28:31.430 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.422+0000 7fc914a2e700 1 -- 172.21.15.167:0/3055819727 <== osd.1 v2:172.21.15.167:6800/30035 17 ==== osd_op_reply(43 zonegroups_names.default [read 0~46 out=46b] v0'0 uv2 ondisk = 0) v8 ==== 168+0+46 (crc 0 0 0) 0x7fc90c00a5a0 con 0x55c8b6e017e0 2023-10-05T19:28:31.430 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.422+0000 7fc9166f8e40 20 rados_obj.operate() r=0 bl.length=46 2023-10-05T19:28:31.430 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.422+0000 7fc9166f8e40 1 -- 172.21.15.167:0/3055819727 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:44 5.0 5:00000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x55c8b6e3c180 con 0x55c8b6e38410 2023-10-05T19:28:31.432 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.426+0000 7fc91422d700 1 -- 172.21.15.167:0/3055819727 <== osd.3 v2:172.21.15.167:6808/30036 11 ==== osd_op_reply(44 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fc90400d7f0 con 0x55c8b6e38410 2023-10-05T19:28:31.432 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.426+0000 7fc91422d700 1 -- 172.21.15.167:0/3055819727 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:45 5.10 5:08000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x55c8b6e3c180 con 0x7fc904044040 2023-10-05T19:28:31.433 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.426+0000 7fc90bfff700 1 -- 172.21.15.167:0/3055819727 <== osd.2 v2:172.21.15.167:6816/30037 7 ==== osd_op_reply(45 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fc8fc008ae0 con 0x7fc904044040 2023-10-05T19:28:31.433 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.426+0000 7fc90bfff700 1 -- 172.21.15.167:0/3055819727 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:46 5.8 5:10000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x55c8b6e3c180 con 0x7fc8fc007030 2023-10-05T19:28:31.434 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.426+0000 7fc914a2e700 1 -- 172.21.15.167:0/3055819727 <== osd.0 v2:172.21.15.118:6800/27690 11 ==== osd_op_reply(46 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fc90c00a5a0 con 0x7fc8fc007030 2023-10-05T19:28:31.434 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.426+0000 7fc914a2e700 1 -- 172.21.15.167:0/3055819727 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:47 5.18 5:18000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x55c8b6e370e0 con 0x55c8b6e017e0 2023-10-05T19:28:31.435 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.430+0000 7fc914a2e700 1 -- 172.21.15.167:0/3055819727 <== osd.1 v2:172.21.15.167:6800/30035 18 ==== osd_op_reply(47 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fc90c00a5a0 con 0x55c8b6e017e0 2023-10-05T19:28:31.435 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.430+0000 7fc914a2e700 1 -- 172.21.15.167:0/3055819727 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:48 5.4 5:20000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x55c8b6e370e0 con 0x55c8b6e017e0 2023-10-05T19:28:31.436 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.430+0000 7fc914a2e700 1 -- 172.21.15.167:0/3055819727 <== osd.1 v2:172.21.15.167:6800/30035 19 ==== osd_op_reply(48 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fc90c00a5a0 con 0x55c8b6e017e0 2023-10-05T19:28:31.436 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.430+0000 7fc914a2e700 1 -- 172.21.15.167:0/3055819727 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:49 5.14 5:28000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x55c8b6e370e0 con 0x7fc8fc007030 2023-10-05T19:28:31.436 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.430+0000 7fc914a2e700 1 -- 172.21.15.167:0/3055819727 <== osd.0 v2:172.21.15.118:6800/27690 12 ==== osd_op_reply(49 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fc90c00a5a0 con 0x7fc8fc007030 2023-10-05T19:28:31.436 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.430+0000 7fc914a2e700 1 -- 172.21.15.167:0/3055819727 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:50 5.c 5:30000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x55c8b6e370e0 con 0x7fc8fc007030 2023-10-05T19:28:31.437 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.430+0000 7fc914a2e700 1 -- 172.21.15.167:0/3055819727 <== osd.0 v2:172.21.15.118:6800/27690 13 ==== osd_op_reply(50 [pgnls start_epoch 30 out=112b] v23'1 uv1 ondisk = 1) v8 ==== 144+0+112 (crc 0 0 0) 0x7fc90c00a5a0 con 0x7fc8fc007030 2023-10-05T19:28:31.437 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.430+0000 7fc914a2e700 1 -- 172.21.15.167:0/3055819727 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:51 5.1c 5:38000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x55c8b6e370e0 con 0x7fc904044040 2023-10-05T19:28:31.438 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.434+0000 7fc90bfff700 1 -- 172.21.15.167:0/3055819727 <== osd.2 v2:172.21.15.167:6816/30037 8 ==== osd_op_reply(51 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fc8fc008ae0 con 0x7fc904044040 2023-10-05T19:28:31.438 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.434+0000 7fc90bfff700 1 -- 172.21.15.167:0/3055819727 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:52 5.2 5:40000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x55c8b6e370e0 con 0x55c8b6e38410 2023-10-05T19:28:31.438 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.434+0000 7fc91422d700 1 -- 172.21.15.167:0/3055819727 <== osd.3 v2:172.21.15.167:6808/30036 12 ==== osd_op_reply(52 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fc90400d7f0 con 0x55c8b6e38410 2023-10-05T19:28:31.439 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.434+0000 7fc91422d700 1 -- 172.21.15.167:0/3055819727 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:53 5.12 5:48000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x55c8b6e370e0 con 0x55c8b6e017e0 2023-10-05T19:28:31.439 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.434+0000 7fc914a2e700 1 -- 172.21.15.167:0/3055819727 <== osd.1 v2:172.21.15.167:6800/30035 20 ==== osd_op_reply(53 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fc90c00a5a0 con 0x55c8b6e017e0 2023-10-05T19:28:31.440 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.434+0000 7fc914a2e700 1 -- 172.21.15.167:0/3055819727 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:54 5.a 5:50000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x55c8b6e370e0 con 0x7fc8fc007030 2023-10-05T19:28:31.440 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.434+0000 7fc914a2e700 1 -- 172.21.15.167:0/3055819727 <== osd.0 v2:172.21.15.118:6800/27690 14 ==== osd_op_reply(54 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fc90c00a5a0 con 0x7fc8fc007030 2023-10-05T19:28:31.440 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.434+0000 7fc914a2e700 1 -- 172.21.15.167:0/3055819727 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:55 5.1a 5:58000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x55c8b6e370e0 con 0x55c8b6e017e0 2023-10-05T19:28:31.441 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.434+0000 7fc914a2e700 1 -- 172.21.15.167:0/3055819727 <== osd.1 v2:172.21.15.167:6800/30035 21 ==== osd_op_reply(55 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fc90c00a5a0 con 0x55c8b6e017e0 2023-10-05T19:28:31.441 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.434+0000 7fc914a2e700 1 -- 172.21.15.167:0/3055819727 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:56 5.6 5:60000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x55c8b6e370e0 con 0x7fc904044040 2023-10-05T19:28:31.442 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.434+0000 7fc90bfff700 1 -- 172.21.15.167:0/3055819727 <== osd.2 v2:172.21.15.167:6816/30037 9 ==== osd_op_reply(56 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fc8fc008ae0 con 0x7fc904044040 2023-10-05T19:28:31.442 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.434+0000 7fc90bfff700 1 -- 172.21.15.167:0/3055819727 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:57 5.16 5:68000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x55c8b6e370e0 con 0x55c8b6e38410 2023-10-05T19:28:31.442 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.438+0000 7fc91422d700 1 -- 172.21.15.167:0/3055819727 <== osd.3 v2:172.21.15.167:6808/30036 13 ==== osd_op_reply(57 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fc90400d7f0 con 0x55c8b6e38410 2023-10-05T19:28:31.442 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.438+0000 7fc91422d700 1 -- 172.21.15.167:0/3055819727 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:58 5.e 5:70000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x55c8b6e370e0 con 0x55c8b6e38410 2023-10-05T19:28:31.443 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.438+0000 7fc91422d700 1 -- 172.21.15.167:0/3055819727 <== osd.3 v2:172.21.15.167:6808/30036 14 ==== osd_op_reply(58 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fc90400d7f0 con 0x55c8b6e38410 2023-10-05T19:28:31.443 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.438+0000 7fc91422d700 1 -- 172.21.15.167:0/3055819727 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:59 5.1e 5:78000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x55c8b6e370e0 con 0x7fc8fc007030 2023-10-05T19:28:31.444 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.438+0000 7fc914a2e700 1 -- 172.21.15.167:0/3055819727 <== osd.0 v2:172.21.15.118:6800/27690 15 ==== osd_op_reply(59 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fc90c03f040 con 0x7fc8fc007030 2023-10-05T19:28:31.444 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.438+0000 7fc914a2e700 1 -- 172.21.15.167:0/3055819727 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:60 5.1 5:80000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x55c8b6e370e0 con 0x55c8b6e38410 2023-10-05T19:28:31.445 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.438+0000 7fc91422d700 1 -- 172.21.15.167:0/3055819727 <== osd.3 v2:172.21.15.167:6808/30036 15 ==== osd_op_reply(60 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fc90400d7f0 con 0x55c8b6e38410 2023-10-05T19:28:31.445 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.438+0000 7fc91422d700 1 -- 172.21.15.167:0/3055819727 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:61 5.11 5:88000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x55c8b6e370e0 con 0x55c8b6e38410 2023-10-05T19:28:31.445 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.438+0000 7fc91422d700 1 -- 172.21.15.167:0/3055819727 <== osd.3 v2:172.21.15.167:6808/30036 16 ==== osd_op_reply(61 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fc90400d7f0 con 0x55c8b6e38410 2023-10-05T19:28:31.445 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.438+0000 7fc91422d700 1 -- 172.21.15.167:0/3055819727 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:62 5.9 5:90000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x55c8b6e370e0 con 0x55c8b6e017e0 2023-10-05T19:28:31.446 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.442+0000 7fc914a2e700 1 -- 172.21.15.167:0/3055819727 <== osd.1 v2:172.21.15.167:6800/30035 22 ==== osd_op_reply(62 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fc90c03f040 con 0x55c8b6e017e0 2023-10-05T19:28:31.446 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.442+0000 7fc914a2e700 1 -- 172.21.15.167:0/3055819727 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:63 5.19 5:98000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x55c8b6e370e0 con 0x55c8b6e017e0 2023-10-05T19:28:31.447 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.442+0000 7fc914a2e700 1 -- 172.21.15.167:0/3055819727 <== osd.1 v2:172.21.15.167:6800/30035 23 ==== osd_op_reply(63 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fc90c03f040 con 0x55c8b6e017e0 2023-10-05T19:28:31.447 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.442+0000 7fc914a2e700 1 -- 172.21.15.167:0/3055819727 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:64 5.5 5:a0000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x55c8b6e370e0 con 0x7fc8fc007030 2023-10-05T19:28:31.448 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.442+0000 7fc914a2e700 1 -- 172.21.15.167:0/3055819727 <== osd.0 v2:172.21.15.118:6800/27690 16 ==== osd_op_reply(64 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fc90c03f040 con 0x7fc8fc007030 2023-10-05T19:28:31.448 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.442+0000 7fc914a2e700 1 -- 172.21.15.167:0/3055819727 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:65 5.15 5:a8000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x55c8b6e370e0 con 0x55c8b6e38410 2023-10-05T19:28:31.449 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.442+0000 7fc91422d700 1 -- 172.21.15.167:0/3055819727 <== osd.3 v2:172.21.15.167:6808/30036 17 ==== osd_op_reply(65 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fc90400d7f0 con 0x55c8b6e38410 2023-10-05T19:28:31.449 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.442+0000 7fc91422d700 1 -- 172.21.15.167:0/3055819727 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:66 5.d 5:b0000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x55c8b6e370e0 con 0x7fc904044040 2023-10-05T19:28:31.449 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.442+0000 7fc90bfff700 1 -- 172.21.15.167:0/3055819727 <== osd.2 v2:172.21.15.167:6816/30037 10 ==== osd_op_reply(66 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fc8fc008ae0 con 0x7fc904044040 2023-10-05T19:28:31.450 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.442+0000 7fc90bfff700 1 -- 172.21.15.167:0/3055819727 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:67 5.1d 5:b8000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x55c8b6e370e0 con 0x55c8b6e017e0 2023-10-05T19:28:31.451 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.446+0000 7fc914a2e700 1 -- 172.21.15.167:0/3055819727 <== osd.1 v2:172.21.15.167:6800/30035 24 ==== osd_op_reply(67 [pgnls start_epoch 30 out=107b] v30'2 uv2 ondisk = 1) v8 ==== 144+0+107 (crc 0 0 0) 0x7fc90c03f040 con 0x55c8b6e017e0 2023-10-05T19:28:31.451 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.446+0000 7fc914a2e700 1 -- 172.21.15.167:0/3055819727 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:68 5.3 5:c0000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x55c8b6e370e0 con 0x7fc8fc007030 2023-10-05T19:28:31.451 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.446+0000 7fc914a2e700 1 -- 172.21.15.167:0/3055819727 <== osd.0 v2:172.21.15.118:6800/27690 17 ==== osd_op_reply(68 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fc90c03f040 con 0x7fc8fc007030 2023-10-05T19:28:31.452 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.446+0000 7fc914a2e700 1 -- 172.21.15.167:0/3055819727 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:69 5.13 5:c8000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x55c8b6e370e0 con 0x55c8b6e38410 2023-10-05T19:28:31.452 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.446+0000 7fc91422d700 1 -- 172.21.15.167:0/3055819727 <== osd.3 v2:172.21.15.167:6808/30036 18 ==== osd_op_reply(69 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fc90400d7f0 con 0x55c8b6e38410 2023-10-05T19:28:31.452 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.446+0000 7fc91422d700 1 -- 172.21.15.167:0/3055819727 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:70 5.b 5:d0000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x55c8b6e370e0 con 0x7fc8fc007030 2023-10-05T19:28:31.453 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.446+0000 7fc914a2e700 1 -- 172.21.15.167:0/3055819727 <== osd.0 v2:172.21.15.118:6800/27690 18 ==== osd_op_reply(70 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fc90c03f040 con 0x7fc8fc007030 2023-10-05T19:28:31.453 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.446+0000 7fc914a2e700 1 -- 172.21.15.167:0/3055819727 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:71 5.1b 5:d8000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x55c8b6e370e0 con 0x7fc904044040 2023-10-05T19:28:31.454 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.450+0000 7fc90bfff700 1 -- 172.21.15.167:0/3055819727 <== osd.2 v2:172.21.15.167:6816/30037 11 ==== osd_op_reply(71 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fc8fc008ae0 con 0x7fc904044040 2023-10-05T19:28:31.454 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.450+0000 7fc90bfff700 1 -- 172.21.15.167:0/3055819727 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:72 5.7 5:e0000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x55c8b6e370e0 con 0x55c8b6e38410 2023-10-05T19:28:31.455 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.450+0000 7fc91422d700 1 -- 172.21.15.167:0/3055819727 <== osd.3 v2:172.21.15.167:6808/30036 19 ==== osd_op_reply(72 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fc90400d7f0 con 0x55c8b6e38410 2023-10-05T19:28:31.455 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.450+0000 7fc91422d700 1 -- 172.21.15.167:0/3055819727 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:73 5.17 5:e8000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x55c8b6e370e0 con 0x7fc8fc007030 2023-10-05T19:28:31.456 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.450+0000 7fc914a2e700 1 -- 172.21.15.167:0/3055819727 <== osd.0 v2:172.21.15.118:6800/27690 19 ==== osd_op_reply(73 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fc90c03f040 con 0x7fc8fc007030 2023-10-05T19:28:31.456 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.450+0000 7fc914a2e700 1 -- 172.21.15.167:0/3055819727 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:74 5.f 5:f0000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x55c8b6e370e0 con 0x55c8b6e017e0 2023-10-05T19:28:31.457 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.450+0000 7fc914a2e700 1 -- 172.21.15.167:0/3055819727 <== osd.1 v2:172.21.15.167:6800/30035 25 ==== osd_op_reply(74 [pgnls start_epoch 30 out=115b] v23'2 uv2 ondisk = 1) v8 ==== 144+0+115 (crc 0 0 0) 0x7fc90c03f040 con 0x55c8b6e017e0 2023-10-05T19:28:31.457 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.450+0000 7fc914a2e700 1 -- 172.21.15.167:0/3055819727 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:75 5.1f 5:f8000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x55c8b6e370e0 con 0x7fc904044040 2023-10-05T19:28:31.458 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.450+0000 7fc90bfff700 1 -- 172.21.15.167:0/3055819727 <== osd.2 v2:172.21.15.167:6816/30037 12 ==== osd_op_reply(75 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fc8fc008ae0 con 0x7fc904044040 2023-10-05T19:28:31.458 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.450+0000 7fc9166f8e40 20 RGWRados::pool_iterate: got zonegroup_info.4f1f719d-9648-417b-b10c-e5d939825419 2023-10-05T19:28:31.458 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.450+0000 7fc9166f8e40 20 RGWRados::pool_iterate: got zone_info.9a9df965-6612-45d4-a45f-6e3c889bacc2 2023-10-05T19:28:31.458 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.450+0000 7fc9166f8e40 20 RGWRados::pool_iterate: got zone_names.default 2023-10-05T19:28:31.458 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.450+0000 7fc9166f8e40 20 RGWRados::pool_iterate: got zonegroups_names.default 2023-10-05T19:28:31.458 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.450+0000 7fc9166f8e40 20 rados->read ofs=0 len=0 2023-10-05T19:28:31.458 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.450+0000 7fc9166f8e40 1 -- 172.21.15.167:0/3055819727 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:76 5.f 5:f43b6ece:::zone_names.default:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x55c8b6e3b170 con 0x55c8b6e017e0 2023-10-05T19:28:31.459 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.454+0000 7fc914a2e700 1 -- 172.21.15.167:0/3055819727 <== osd.1 v2:172.21.15.167:6800/30035 26 ==== osd_op_reply(76 zone_names.default [read 0~46 out=46b] v0'0 uv1 ondisk = 0) v8 ==== 162+0+46 (crc 0 0 0) 0x7fc90c03f040 con 0x55c8b6e017e0 2023-10-05T19:28:31.459 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.454+0000 7fc9166f8e40 20 rados_obj.operate() r=0 bl.length=46 2023-10-05T19:28:31.459 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.454+0000 7fc9166f8e40 20 rados->read ofs=0 len=0 2023-10-05T19:28:31.459 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.454+0000 7fc9166f8e40 1 -- 172.21.15.167:0/3055819727 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:77 5.1d 5:b8374eb4:::zone_info.9a9df965-6612-45d4-a45f-6e3c889bacc2:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x55c8b6e3b520 con 0x55c8b6e017e0 2023-10-05T19:28:31.460 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.454+0000 7fc914a2e700 1 -- 172.21.15.167:0/3055819727 <== osd.1 v2:172.21.15.167:6800/30035 27 ==== osd_op_reply(77 zone_info.9a9df965-6612-45d4-a45f-6e3c889bacc2 [read 0~874 out=874b] v0'0 uv2 ondisk = 0) v8 ==== 190+0+874 (crc 0 0 0) 0x7fc90c03f040 con 0x55c8b6e017e0 2023-10-05T19:28:31.460 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.454+0000 7fc9166f8e40 20 rados_obj.operate() r=0 bl.length=874 2023-10-05T19:28:31.460 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.454+0000 7fc9166f8e40 20 rados->read ofs=0 len=0 2023-10-05T19:28:31.460 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.454+0000 7fc9166f8e40 1 -- 172.21.15.167:0/3055819727 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:78 5.f 5:f4c53578:::zonegroups_names.default:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x55c8b6e3ba10 con 0x55c8b6e017e0 2023-10-05T19:28:31.461 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.454+0000 7fc914a2e700 1 -- 172.21.15.167:0/3055819727 <== osd.1 v2:172.21.15.167:6800/30035 28 ==== osd_op_reply(78 zonegroups_names.default [read 0~46 out=46b] v0'0 uv2 ondisk = 0) v8 ==== 168+0+46 (crc 0 0 0) 0x7fc90c03f040 con 0x55c8b6e017e0 2023-10-05T19:28:31.461 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.454+0000 7fc9166f8e40 20 rados_obj.operate() r=0 bl.length=46 2023-10-05T19:28:31.461 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.454+0000 7fc9166f8e40 20 rados->read ofs=0 len=0 2023-10-05T19:28:31.461 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.454+0000 7fc9166f8e40 1 -- 172.21.15.167:0/3055819727 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:79 5.c 5:30d299a5:::zonegroup_info.4f1f719d-9648-417b-b10c-e5d939825419:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x55c8b6e3ba10 con 0x7fc8fc007030 2023-10-05T19:28:31.462 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.454+0000 7fc914a2e700 1 -- 172.21.15.167:0/3055819727 <== osd.0 v2:172.21.15.118:6800/27690 20 ==== osd_op_reply(79 zonegroup_info.4f1f719d-9648-417b-b10c-e5d939825419 [read 0~362 out=362b] v0'0 uv1 ondisk = 0) v8 ==== 195+0+362 (crc 0 0 0) 0x7fc90c052040 con 0x7fc8fc007030 2023-10-05T19:28:31.462 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.454+0000 7fc9166f8e40 20 rados_obj.operate() r=0 bl.length=362 2023-10-05T19:28:31.462 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.454+0000 7fc9166f8e40 20 rados->read ofs=0 len=0 2023-10-05T19:28:31.462 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.454+0000 7fc9166f8e40 1 -- 172.21.15.167:0/3055819727 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:80 5.3 5:c0ad8cbd:::region_map:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x55c8b6e3ba10 con 0x7fc8fc007030 2023-10-05T19:28:31.462 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.458+0000 7fc914a2e700 1 -- 172.21.15.167:0/3055819727 <== osd.0 v2:172.21.15.118:6800/27690 21 ==== osd_op_reply(80 region_map [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 ==== 154+0+0 (crc 0 0 0) 0x7fc90c052040 con 0x7fc8fc007030 2023-10-05T19:28:31.463 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.458+0000 7fc9166f8e40 20 rados_obj.operate() r=-2 bl.length=0 2023-10-05T19:28:31.463 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.458+0000 7fc9166f8e40 20 zone default found 2023-10-05T19:28:31.463 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.458+0000 7fc9166f8e40 4 Realm: () 2023-10-05T19:28:31.463 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.458+0000 7fc9166f8e40 4 ZoneGroup: default (4f1f719d-9648-417b-b10c-e5d939825419) 2023-10-05T19:28:31.463 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.458+0000 7fc9166f8e40 4 Zone: default (9a9df965-6612-45d4-a45f-6e3c889bacc2) 2023-10-05T19:28:31.463 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.458+0000 7fc9166f8e40 10 cannot find current period zonegroup using local zonegroup configuration 2023-10-05T19:28:31.463 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.458+0000 7fc9166f8e40 20 zonegroup default 2023-10-05T19:28:31.463 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.458+0000 7fc9166f8e40 20 rados->read ofs=0 len=0 2023-10-05T19:28:31.463 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.458+0000 7fc9166f8e40 1 -- 172.21.15.167:0/3055819727 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:81 5.3 5:c52100b6:::period_config.default:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x55c8b6e3ba10 con 0x7fc8fc007030 2023-10-05T19:28:31.464 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.458+0000 7fc914a2e700 1 -- 172.21.15.167:0/3055819727 <== osd.0 v2:172.21.15.118:6800/27690 22 ==== osd_op_reply(81 period_config.default [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 ==== 165+0+0 (crc 0 0 0) 0x7fc90c052040 con 0x7fc8fc007030 2023-10-05T19:28:31.464 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.458+0000 7fc9166f8e40 20 rados_obj.operate() r=-2 bl.length=0 2023-10-05T19:28:31.464 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.458+0000 7fc9166f8e40 20 rados->read ofs=0 len=0 2023-10-05T19:28:31.464 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.458+0000 7fc9166f8e40 1 -- 172.21.15.167:0/3055819727 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:82 6.2 6:4347d321:::bucket.sync-source-hints.:head [call version.read in=11b,read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x55c8b6e3d390 con 0x55c8b6e38410 2023-10-05T19:28:31.465 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.458+0000 7fc91422d700 1 -- 172.21.15.167:0/3055819727 <== osd.3 v2:172.21.15.167:6808/30036 20 ==== osd_op_reply(82 bucket.sync-source-hints. [call,read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 ==== 211+0+0 (crc 0 0 0) 0x7fc90400d7f0 con 0x55c8b6e38410 2023-10-05T19:28:31.465 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.458+0000 7fc9166f8e40 20 rados_obj.operate() r=-2 bl.length=0 2023-10-05T19:28:31.465 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.458+0000 7fc9166f8e40 20 rados->read ofs=0 len=0 2023-10-05T19:28:31.465 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.458+0000 7fc9166f8e40 1 -- 172.21.15.167:0/3055819727 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:83 6.b 6:d467b91b:::bucket.sync-target-hints.:head [call version.read in=11b,read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x55c8b6e3d740 con 0x55c8b6e38410 2023-10-05T19:28:31.465 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.458+0000 7fc91422d700 1 -- 172.21.15.167:0/3055819727 <== osd.3 v2:172.21.15.167:6808/30036 21 ==== osd_op_reply(83 bucket.sync-target-hints. [call,read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 ==== 211+0+0 (crc 0 0 0) 0x7fc90400d7f0 con 0x55c8b6e38410 2023-10-05T19:28:31.465 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.458+0000 7fc9166f8e40 20 rados_obj.operate() r=-2 bl.length=0 2023-10-05T19:28:31.466 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.458+0000 7fc9166f8e40 20 started sync module instance, tier type = 2023-10-05T19:28:31.466 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.458+0000 7fc9166f8e40 20 started zone id=9a9df965-6612-45d4-a45f-6e3c889bacc2 (name=default) with tier type = 2023-10-05T19:28:31.466 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.458+0000 7fc9166f8e40 1 -- 172.21.15.167:0/3055819727 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:84 7.1f 7:f95f44c2:::notify.0:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x55c8b6e3ca50 con 0x7fc904044040 2023-10-05T19:28:31.468 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.462+0000 7fc90bfff700 1 -- 172.21.15.167:0/3055819727 <== osd.2 v2:172.21.15.167:6816/30037 13 ==== osd_op_reply(84 notify.0 [create] v30'96 uv96 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7fc8fc008ae0 con 0x7fc904044040 2023-10-05T19:28:31.468 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.462+0000 7fc9166f8e40 1 -- 172.21.15.167:0/3055819727 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:85 7.1f 7:f95f44c2:::notify.0:head [watch watch cookie 94320550202960] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x55c8b6e3c680 con 0x7fc904044040 2023-10-05T19:28:31.468 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.462+0000 7fc9166f8e40 1 -- 172.21.15.167:0/3055819727 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:86 7.0 7:05bf5b68:::notify.1:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x55c8b6e3e530 con 0x55c8b6e017e0 2023-10-05T19:28:31.471 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.466+0000 7fc914a2e700 1 -- 172.21.15.167:0/3055819727 <== osd.1 v2:172.21.15.167:6800/30035 29 ==== osd_op_reply(86 notify.1 [create] v30'96 uv96 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7fc90c052040 con 0x55c8b6e017e0 2023-10-05T19:28:31.471 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.466+0000 7fc9166f8e40 1 -- 172.21.15.167:0/3055819727 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:87 7.0 7:05bf5b68:::notify.1:head [watch watch cookie 94320550207840] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x55c8b6e3ee20 con 0x55c8b6e017e0 2023-10-05T19:28:31.471 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.466+0000 7fc9166f8e40 1 -- 172.21.15.167:0/3055819727 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:88 7.15 7:a93a5511:::notify.2:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x55c8b6e3f6c0 con 0x55c8b6e38410 2023-10-05T19:28:31.471 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.466+0000 7fc90bfff700 1 -- 172.21.15.167:0/3055819727 <== osd.2 v2:172.21.15.167:6816/30037 14 ==== osd_op_reply(85 notify.0 [watch watch cookie 94320550202960] v30'97 uv96 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7fc8fc008ae0 con 0x7fc904044040 2023-10-05T19:28:31.474 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.466+0000 7fc914a2e700 1 -- 172.21.15.167:0/3055819727 <== osd.1 v2:172.21.15.167:6800/30035 30 ==== osd_op_reply(87 notify.1 [watch watch cookie 94320550207840] v30'97 uv96 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7fc90c052040 con 0x55c8b6e017e0 2023-10-05T19:28:31.475 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.470+0000 7fc91422d700 1 -- 172.21.15.167:0/3055819727 <== osd.3 v2:172.21.15.167:6808/30036 22 ==== osd_op_reply(88 notify.2 [create] v30'96 uv96 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7fc90400d7f0 con 0x55c8b6e38410 2023-10-05T19:28:31.475 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.470+0000 7fc9166f8e40 1 -- 172.21.15.167:0/3055819727 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:89 7.15 7:a93a5511:::notify.2:head [watch watch cookie 94320550209840] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x55c8b6e3dc10 con 0x55c8b6e38410 2023-10-05T19:28:31.475 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.470+0000 7fc9166f8e40 1 -- 172.21.15.167:0/3055819727 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:90 7.e 7:7759931f:::notify.3:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x55c8b6e3ff60 con 0x7fc904044040 2023-10-05T19:28:31.477 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.470+0000 7fc90bfff700 1 -- 172.21.15.167:0/3055819727 <== osd.2 v2:172.21.15.167:6816/30037 15 ==== osd_op_reply(90 notify.3 [create] v30'96 uv96 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7fc8fc008ae0 con 0x7fc904044040 2023-10-05T19:28:31.477 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.470+0000 7fc9166f8e40 1 -- 172.21.15.167:0/3055819727 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:91 7.e 7:7759931f:::notify.3:head [watch watch cookie 94320550214256] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x55c8b6e40800 con 0x7fc904044040 2023-10-05T19:28:31.478 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.470+0000 7fc9166f8e40 1 -- 172.21.15.167:0/3055819727 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:92 7.d 7:b4812045:::notify.4:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x55c8b6e413c0 con 0x55c8b6e017e0 2023-10-05T19:28:31.478 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.470+0000 7fc91422d700 1 -- 172.21.15.167:0/3055819727 <== osd.3 v2:172.21.15.167:6808/30036 23 ==== osd_op_reply(89 notify.2 [watch watch cookie 94320550209840] v30'97 uv96 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7fc90400d7f0 con 0x55c8b6e38410 2023-10-05T19:28:31.480 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.474+0000 7fc90bfff700 1 -- 172.21.15.167:0/3055819727 <== osd.2 v2:172.21.15.167:6816/30037 16 ==== osd_op_reply(91 notify.3 [watch watch cookie 94320550214256] v30'97 uv96 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7fc8fc008ae0 con 0x7fc904044040 2023-10-05T19:28:31.480 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.474+0000 7fc914a2e700 1 -- 172.21.15.167:0/3055819727 <== osd.1 v2:172.21.15.167:6800/30035 31 ==== osd_op_reply(92 notify.4 [create] v30'96 uv96 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7fc90c052040 con 0x55c8b6e017e0 2023-10-05T19:28:31.480 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.474+0000 7fc9166f8e40 1 -- 172.21.15.167:0/3055819727 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:93 7.d 7:b4812045:::notify.4:head [watch watch cookie 94320550216464] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x55c8b6e40e90 con 0x55c8b6e017e0 2023-10-05T19:28:31.480 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.474+0000 7fc9166f8e40 1 -- 172.21.15.167:0/3055819727 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:94 7.3 7:c609908c:::notify.5:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x55c8b6e41cb0 con 0x7fc8fc007030 2023-10-05T19:28:31.482 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.478+0000 7fc914a2e700 1 -- 172.21.15.167:0/3055819727 <== osd.1 v2:172.21.15.167:6800/30035 32 ==== osd_op_reply(93 notify.4 [watch watch cookie 94320550216464] v30'97 uv96 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7fc90c052040 con 0x55c8b6e017e0 2023-10-05T19:28:31.483 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.478+0000 7fc914a2e700 1 -- 172.21.15.167:0/3055819727 <== osd.0 v2:172.21.15.118:6800/27690 23 ==== osd_op_reply(94 notify.5 [create] v30'96 uv96 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7fc90c052040 con 0x7fc8fc007030 2023-10-05T19:28:31.483 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.478+0000 7fc9166f8e40 1 -- 172.21.15.167:0/3055819727 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:95 7.3 7:c609908c:::notify.5:head [watch watch cookie 94320550210944] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x55c8b6e41c60 con 0x7fc8fc007030 2023-10-05T19:28:31.483 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.478+0000 7fc9166f8e40 1 -- 172.21.15.167:0/3055819727 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:96 7.14 7:2b04a3e9:::notify.6:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x55c8b6e42670 con 0x7fc8fc007030 2023-10-05T19:28:31.486 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.478+0000 7fc914a2e700 1 -- 172.21.15.167:0/3055819727 <== osd.0 v2:172.21.15.118:6800/27690 24 ==== osd_op_reply(95 notify.5 [watch watch cookie 94320550210944] v30'97 uv96 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7fc90c052040 con 0x7fc8fc007030 2023-10-05T19:28:31.486 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.478+0000 7fc914a2e700 1 -- 172.21.15.167:0/3055819727 <== osd.0 v2:172.21.15.118:6800/27690 25 ==== osd_op_reply(96 notify.6 [create] v30'96 uv96 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7fc90c052040 con 0x7fc8fc007030 2023-10-05T19:28:31.486 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.482+0000 7fc9166f8e40 1 -- 172.21.15.167:0/3055819727 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:97 7.14 7:2b04a3e9:::notify.6:head [watch watch cookie 94320550222240] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x55c8b6e42670 con 0x7fc8fc007030 2023-10-05T19:28:31.486 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.482+0000 7fc9166f8e40 1 -- 172.21.15.167:0/3055819727 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:98 7.9 7:93e5b521:::notify.7:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x55c8b6e43080 con 0x7fc8fc007030 2023-10-05T19:28:31.489 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.482+0000 7fc914a2e700 1 -- 172.21.15.167:0/3055819727 <== osd.0 v2:172.21.15.118:6800/27690 26 ==== osd_op_reply(97 notify.6 [watch watch cookie 94320550222240] v30'97 uv96 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7fc90c052040 con 0x7fc8fc007030 2023-10-05T19:28:31.489 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.482+0000 7fc914a2e700 1 -- 172.21.15.167:0/3055819727 <== osd.0 v2:172.21.15.118:6800/27690 27 ==== osd_op_reply(98 notify.7 [create] v30'96 uv96 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7fc90c052040 con 0x7fc8fc007030 2023-10-05T19:28:31.489 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.482+0000 7fc9166f8e40 1 -- 172.21.15.167:0/3055819727 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:99 7.9 7:93e5b521:::notify.7:head [watch watch cookie 94320550224816] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x55c8b6e43080 con 0x7fc8fc007030 2023-10-05T19:28:31.489 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.482+0000 7fc9166f8e40 20 add_watcher() i=0 2023-10-05T19:28:31.489 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.482+0000 7fc9166f8e40 20 add_watcher() i=1 2023-10-05T19:28:31.489 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.482+0000 7fc9166f8e40 20 add_watcher() i=2 2023-10-05T19:28:31.490 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.482+0000 7fc9166f8e40 20 add_watcher() i=3 2023-10-05T19:28:31.490 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.482+0000 7fc9166f8e40 20 add_watcher() i=4 2023-10-05T19:28:31.490 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.482+0000 7fc9166f8e40 20 add_watcher() i=5 2023-10-05T19:28:31.490 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.482+0000 7fc9166f8e40 20 add_watcher() i=6 2023-10-05T19:28:31.491 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.486+0000 7fc914a2e700 1 -- 172.21.15.167:0/3055819727 <== osd.0 v2:172.21.15.118:6800/27690 28 ==== osd_op_reply(99 notify.7 [watch watch cookie 94320550224816] v30'97 uv96 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7fc90c052040 con 0x7fc8fc007030 2023-10-05T19:28:31.491 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.486+0000 7fc9166f8e40 20 add_watcher() i=7 2023-10-05T19:28:31.491 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.486+0000 7fc9166f8e40 2 all 8 watchers are set, enabling cache 2023-10-05T19:28:31.491 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.486+0000 7fc9166f8e40 1 -- 172.21.15.167:0/3055819727 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:100 6.4 6:22d26bf9:::data_loggenerations_metadata:head [call version.check_conds in=74b,call version.read in=11b,read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x55c8b6e46220 con 0x55c8b6e017e0 2023-10-05T19:28:31.492 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.486+0000 7fc914a2e700 1 -- 172.21.15.167:0/3055819727 <== osd.1 v2:172.21.15.167:6800/30035 33 ==== osd_op_reply(100 data_loggenerations_metadata [call,call out=48b,read 0~28 out=28b] v0'0 uv1 ondisk = 0) v8 ==== 256+0+76 (crc 0 0 0) 0x7fc90c052040 con 0x55c8b6e017e0 2023-10-05T19:28:31.492 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.486+0000 7fc9166f8e40 1 -- 172.21.15.167:0/3055819727 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:101 6.4 6:22d26bf9:::data_loggenerations_metadata:head [watch watch cookie 94320550227200] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x55c8b6e437a0 con 0x55c8b6e017e0 2023-10-05T19:28:31.494 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.490+0000 7fc914a2e700 1 -- 172.21.15.167:0/3055819727 <== osd.1 v2:172.21.15.167:6800/30035 34 ==== osd_op_reply(101 data_loggenerations_metadata [watch watch cookie 94320550227200] v30'99 uv1 ondisk = 0) v8 ==== 172+0+0 (crc 0 0 0) 0x7fc90c052040 con 0x55c8b6e017e0 2023-10-05T19:28:31.494 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.490+0000 7fc8e5fe3700 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2023-10-05T19:28:31.494 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.490+0000 7fc9166f8e40 20 check_secure_mon_conn(): auth registy supported: methods=[2] modes=[2,1] 2023-10-05T19:28:31.494 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.490+0000 7fc9166f8e40 20 check_secure_mon_conn(): mode 1 is insecure 2023-10-05T19:28:31.495 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.490+0000 7fc9166f8e40 1 Processor -- start 2023-10-05T19:28:31.495 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.490+0000 7fc9166f8e40 1 -- start start 2023-10-05T19:28:31.495 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.490+0000 7fc9166f8e40 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55c8b6e506d0 0x55c8b6e50aa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-05T19:28:31.495 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.490+0000 7fc9166f8e40 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c8b6e50fe0 con 0x55c8b6e506d0 2023-10-05T19:28:31.496 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.490+0000 7fc90bfff700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55c8b6e506d0 0x55c8b6e50aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-05T19:28:31.496 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.490+0000 7fc90bfff700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55c8b6e506d0 0x55c8b6e50aa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.118:3300/0 says I am v2:172.21.15.167:58208/0 (socket says 172.21.15.167:58208) 2023-10-05T19:28:31.496 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.490+0000 7fc90bfff700 1 -- 172.21.15.167:0/2693005653 learned_addr learned my addr 172.21.15.167:0/2693005653 (peer_addr_for_me v2:172.21.15.167:0/0) 2023-10-05T19:28:31.496 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.490+0000 7fc90bfff700 1 -- 172.21.15.167:0/2693005653 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55c8b6e51120 con 0x55c8b6e506d0 2023-10-05T19:28:31.497 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.490+0000 7fc90bfff700 1 --2- 172.21.15.167:0/2693005653 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55c8b6e506d0 0x55c8b6e50aa0 secure :-1 s=READY pgs=330 cs=0 l=1 rev1=1 crypto rx=0x55c8b6e00ca0 tx=0x7fc8fc012000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c312094a9dfb695 server_cookie=0 in_seq=0 out_seq=0 2023-10-05T19:28:31.497 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.490+0000 7fc8e57e2700 1 -- 172.21.15.167:0/2693005653 <== mon.0 v2:172.21.15.118:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fc8fc012c90 con 0x55c8b6e506d0 2023-10-05T19:28:31.497 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.490+0000 7fc8e57e2700 1 -- 172.21.15.167:0/2693005653 <== mon.0 v2:172.21.15.118:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc8fc012e10 con 0x55c8b6e506d0 2023-10-05T19:28:31.497 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.490+0000 7fc8e57e2700 1 -- 172.21.15.167:0/2693005653 <== mon.0 v2:172.21.15.118:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fc8fc02a8a0 con 0x55c8b6e506d0 2023-10-05T19:28:31.498 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.494+0000 7fc9166f8e40 1 -- 172.21.15.167:0/2693005653 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55c8b6e506d0 msgr2=0x55c8b6e50aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-05T19:28:31.498 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.494+0000 7fc9166f8e40 1 --2- 172.21.15.167:0/2693005653 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55c8b6e506d0 0x55c8b6e50aa0 secure :-1 s=READY pgs=330 cs=0 l=1 rev1=1 crypto rx=0x55c8b6e00ca0 tx=0x7fc8fc012000 comp rx=0 tx=0).stop 2023-10-05T19:28:31.499 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.494+0000 7fc9166f8e40 1 -- 172.21.15.167:0/2693005653 shutdown_connections 2023-10-05T19:28:31.499 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.494+0000 7fc9166f8e40 1 --2- 172.21.15.167:0/2693005653 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55c8b6e506d0 0x55c8b6e50aa0 unknown :-1 s=CLOSED pgs=330 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-05T19:28:31.499 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.494+0000 7fc9166f8e40 1 -- 172.21.15.167:0/2693005653 >> 172.21.15.167:0/2693005653 conn(0x55c8b6e4ba90 msgr2=0x55c8b6e49c40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-05T19:28:31.499 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.494+0000 7fc9166f8e40 1 -- 172.21.15.167:0/2693005653 shutdown_connections 2023-10-05T19:28:31.499 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.494+0000 7fc9166f8e40 1 -- 172.21.15.167:0/2693005653 wait complete. 2023-10-05T19:28:31.500 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.494+0000 7fc9166f8e40 1 Processor -- start 2023-10-05T19:28:31.500 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.494+0000 7fc9166f8e40 1 -- start start 2023-10-05T19:28:31.500 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.494+0000 7fc9166f8e40 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55c8b6e506d0 0x55c8b6e56180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-05T19:28:31.500 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.494+0000 7fc9166f8e40 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c8b6e57180 con 0x55c8b6e506d0 2023-10-05T19:28:31.500 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.494+0000 7fc90bfff700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55c8b6e506d0 0x55c8b6e56180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-05T19:28:31.500 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.494+0000 7fc90bfff700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55c8b6e506d0 0x55c8b6e56180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.118:3300/0 says I am v2:172.21.15.167:58212/0 (socket says 172.21.15.167:58212) 2023-10-05T19:28:31.500 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.494+0000 7fc90bfff700 1 -- 172.21.15.167:0/39568890 learned_addr learned my addr 172.21.15.167:0/39568890 (peer_addr_for_me v2:172.21.15.167:0/0) 2023-10-05T19:28:31.501 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.494+0000 7fc90bfff700 1 -- 172.21.15.167:0/39568890 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55c8b6e00cd0 con 0x55c8b6e506d0 2023-10-05T19:28:31.501 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.494+0000 7fc90bfff700 1 --2- 172.21.15.167:0/39568890 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55c8b6e506d0 0x55c8b6e56180 secure :-1 s=READY pgs=331 cs=0 l=1 rev1=1 crypto rx=0x55c8b6e4a7b0 tx=0x7fc8fc02f000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-05T19:28:31.501 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.494+0000 7fc85f7fe700 1 -- 172.21.15.167:0/39568890 <== mon.0 v2:172.21.15.118:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fc8fc03e950 con 0x55c8b6e506d0 2023-10-05T19:28:31.502 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.494+0000 7fc85f7fe700 1 -- 172.21.15.167:0/39568890 <== mon.0 v2:172.21.15.118:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc8fc03eab0 con 0x55c8b6e506d0 2023-10-05T19:28:31.502 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.494+0000 7fc9166f8e40 1 -- 172.21.15.167:0/39568890 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x55c8b6e56780 con 0x55c8b6e506d0 2023-10-05T19:28:31.502 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.494+0000 7fc85f7fe700 1 -- 172.21.15.167:0/39568890 <== mon.0 v2:172.21.15.118:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fc8fc03ec60 con 0x55c8b6e506d0 2023-10-05T19:28:31.502 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.494+0000 7fc9166f8e40 1 -- 172.21.15.167:0/39568890 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55c8b6e56c20 con 0x55c8b6e506d0 2023-10-05T19:28:31.502 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.498+0000 7fc85f7fe700 1 -- 172.21.15.167:0/39568890 <== mon.0 v2:172.21.15.118:3300/0 4 ==== mgrmap(e 5) v1 ==== 40329+0+0 (secure 0 0 0) 0x7fc8fc046070 con 0x55c8b6e506d0 2023-10-05T19:28:31.503 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.498+0000 7fc85f7fe700 1 --2- 172.21.15.167:0/39568890 >> [v2:172.21.15.167:6824/29889,v1:172.21.15.167:6825/29889] conn(0x7fc854030c00 0x7fc8540330b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-05T19:28:31.503 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.498+0000 7fc85f7fe700 1 -- 172.21.15.167:0/39568890 <== mon.0 v2:172.21.15.118:3300/0 5 ==== osd_map(30..30 src has 1..30) v4 ==== 6057+0+0 (secure 0 0 0) 0x7fc8fc02f070 con 0x55c8b6e506d0 2023-10-05T19:28:31.503 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.498+0000 7fc914a2e700 1 --2- 172.21.15.167:0/39568890 >> [v2:172.21.15.167:6824/29889,v1:172.21.15.167:6825/29889] conn(0x7fc854030c00 0x7fc8540330b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-05T19:28:31.503 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.498+0000 7fc9166f8e40 5 note: GC not initialized 2023-10-05T19:28:31.504 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.498+0000 7fc914a2e700 1 --2- 172.21.15.167:0/39568890 >> [v2:172.21.15.167:6824/29889,v1:172.21.15.167:6825/29889] conn(0x7fc854030c00 0x7fc8540330b0 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x55c8b6e43700 tx=0x7fc90c00aec0 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-05T19:28:31.505 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.498+0000 7fc85cff9700 20 reqs_thread_entry: start 2023-10-05T19:28:31.506 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.502+0000 7fc9166f8e40 20 init_complete bucket index max shards: 11 2023-10-05T19:28:31.506 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.502+0000 7fc8527fc700 20 reqs_thread_entry: start 2023-10-05T19:28:31.507 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.502+0000 7fc9166f8e40 10 rgw notify: Started notification manager with: 1 workers 2023-10-05T19:28:31.507 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.502+0000 7fc850ff9700 20 rgw notify: INFO: next queues processing will happen at: Thu Oct 5 19:29:01 2023 2023-10-05T19:28:31.507 INFO:teuthology.orchestra.run.smithi167.stderr: 2023-10-05T19:28:31.507 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.502+0000 7fc9166f8e40 20 get_system_obj_state: rctx=0x7ffde9a9be58 obj=default.rgw.meta:users.uid:foo state=0x55c8b6cda0b0 s->prefetch_data=0 2023-10-05T19:28:31.507 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.502+0000 7fc9166f8e40 1 -- 172.21.15.167:0/3055819727 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:102 8.18 8:1e6ca5ed:users.uid::foo:head [call version.read in=11b,getxattrs,stat] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x55c8b6ebe8f0 con 0x55c8b6e017e0 2023-10-05T19:28:31.507 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.502+0000 7fc837fff700 20 reqs_thread_entry: start 2023-10-05T19:28:31.508 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.502+0000 7fc914a2e700 1 -- 172.21.15.167:0/3055819727 <== osd.1 v2:172.21.15.167:6800/30035 35 ==== osd_op_reply(102 foo [call out=48b,getxattrs out=75b,stat out=16b] v0'0 uv11 ondisk = 0) v8 ==== 231+0+139 (crc 0 0 0) 0x7fc90c052040 con 0x55c8b6e017e0 2023-10-05T19:28:31.508 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.502+0000 7fc9166f8e40 20 get_system_obj_state: s->obj_tag was set empty 2023-10-05T19:28:31.508 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.502+0000 7fc9166f8e40 20 Read xattr: user.rgw.idtag 2023-10-05T19:28:31.508 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.502+0000 7fc9166f8e40 20 rados->read ofs=0 len=0 2023-10-05T19:28:31.508 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.502+0000 7fc9166f8e40 1 -- 172.21.15.167:0/3055819727 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:103 8.18 8:1e6ca5ed:users.uid::foo:head [call version.check_conds in=122b,call version.read in=11b,read 0~0,getxattrs] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x55c8b6ebe8f0 con 0x55c8b6e017e0 2023-10-05T19:28:31.509 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.502+0000 7fc914a2e700 1 -- 172.21.15.167:0/3055819727 <== osd.1 v2:172.21.15.167:6800/30035 36 ==== osd_op_reply(103 foo [call,call out=48b,read 0~357 out=357b,getxattrs out=75b] v0'0 uv11 ondisk = 0) v8 ==== 273+0+480 (crc 0 0 0) 0x7fc90c052040 con 0x55c8b6e017e0 2023-10-05T19:28:31.509 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.502+0000 7fc9166f8e40 20 rados_obj.operate() r=0 bl.length=357 2023-10-05T19:28:31.509 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.502+0000 7fc9166f8e40 1 -- 172.21.15.167:0/3055819727 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:104 8.9 8:96d98480:users.uid::foo.buckets:head [call user.list_buckets in=34b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x55c8b6ebe8f0 con 0x7fc904044040 2023-10-05T19:28:31.509 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.502+0000 7fc90bfff700 1 -- 172.21.15.167:0/3055819727 <== osd.2 v2:172.21.15.167:6816/30037 17 ==== osd_op_reply(104 foo.buckets [call out=183b] v0'0 uv14 ondisk = 0) v8 ==== 155+0+183 (crc 0 0 0) 0x7fc8fc032040 con 0x7fc904044040 2023-10-05T19:28:31.510 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.502+0000 7fc9166f8e40 20 get_system_obj_state: rctx=0x7ffde9a9bd10 obj=default.rgw.meta:root:myfoo state=0x55c8b6cda0b0 s->prefetch_data=0 2023-10-05T19:28:31.510 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.502+0000 7fc9166f8e40 1 -- 172.21.15.167:0/3055819727 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:105 8.9 8:91b71841:root::myfoo:head [call version.read in=11b,getxattrs,stat] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x55c8b6ebe8f0 con 0x7fc904044040 2023-10-05T19:28:31.510 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.506+0000 7fc90bfff700 1 -- 172.21.15.167:0/3055819727 <== osd.2 v2:172.21.15.167:6816/30037 18 ==== osd_op_reply(105 myfoo [call out=48b,getxattrs out=75b,stat out=16b] v0'0 uv1 ondisk = 0) v8 ==== 233+0+139 (crc 0 0 0) 0x7fc8fc032040 con 0x7fc904044040 2023-10-05T19:28:31.510 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.506+0000 7fc9166f8e40 20 get_system_obj_state: s->obj_tag was set empty 2023-10-05T19:28:31.510 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.506+0000 7fc9166f8e40 20 rados->read ofs=0 len=0 2023-10-05T19:28:31.511 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.506+0000 7fc9166f8e40 1 -- 172.21.15.167:0/3055819727 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:106 8.9 8:91b71841:root::myfoo:head [call version.check_conds in=122b,call version.read in=11b,read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x55c8b6ec06a0 con 0x7fc904044040 2023-10-05T19:28:31.511 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.506+0000 7fc90bfff700 1 -- 172.21.15.167:0/3055819727 <== osd.2 v2:172.21.15.167:6816/30037 19 ==== osd_op_reply(106 myfoo [call,call out=48b,read 0~165 out=165b] v0'0 uv1 ondisk = 0) v8 ==== 233+0+213 (crc 0 0 0) 0x7fc8fc0580e0 con 0x7fc904044040 2023-10-05T19:28:31.511 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.506+0000 7fc9166f8e40 20 rados_obj.operate() r=0 bl.length=165 2023-10-05T19:28:31.511 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.506+0000 7fc9166f8e40 20 get_system_obj_state: rctx=0x7ffde9a9bd10 obj=default.rgw.meta:root:.bucket.meta.myfoo:9a9df965-6612-45d4-a45f-6e3c889bacc2.4223.1 state=0x55c8b6cb02b0 s->prefetch_data=0 2023-10-05T19:28:31.511 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.506+0000 7fc9166f8e40 1 -- 172.21.15.167:0/3055819727 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:107 8.11 8:8bb4ed75:root::.bucket.meta.myfoo%3a9a9df965-6612-45d4-a45f-6e3c889bacc2.4223.1:head [call version.read in=11b,getxattrs,stat] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x55c8b6ec0ca0 con 0x55c8b6e38410 2023-10-05T19:28:31.512 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.506+0000 7fc91422d700 1 -- 172.21.15.167:0/3055819727 <== osd.3 v2:172.21.15.167:6808/30036 24 ==== osd_op_reply(107 .bucket.meta.myfoo:9a9df965-6612-45d4-a45f-6e3c889bacc2.4223.1 [call out=48b,getxattrs out=218b,stat out=16b] v0'0 uv1 ondisk = 0) v8 ==== 290+0+282 (crc 0 0 0) 0x7fc90400d7f0 con 0x55c8b6e38410 2023-10-05T19:28:31.512 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.506+0000 7fc9166f8e40 20 get_system_obj_state: s->obj_tag was set empty 2023-10-05T19:28:31.512 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.506+0000 7fc9166f8e40 20 Read xattr: user.rgw.acl 2023-10-05T19:28:31.512 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.506+0000 7fc9166f8e40 20 Read xattr: user.rgw.idtag 2023-10-05T19:28:31.512 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.506+0000 7fc9166f8e40 20 rados->read ofs=0 len=0 2023-10-05T19:28:31.513 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.506+0000 7fc9166f8e40 1 -- 172.21.15.167:0/3055819727 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:108 8.11 8:8bb4ed75:root::.bucket.meta.myfoo%3a9a9df965-6612-45d4-a45f-6e3c889bacc2.4223.1:head [call version.check_conds in=122b,call version.read in=11b,read 0~0,getxattrs] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x55c8b6ec3b00 con 0x55c8b6e38410 2023-10-05T19:28:31.514 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.506+0000 7fc91422d700 1 -- 172.21.15.167:0/3055819727 <== osd.3 v2:172.21.15.167:6808/30036 25 ==== osd_op_reply(108 .bucket.meta.myfoo:9a9df965-6612-45d4-a45f-6e3c889bacc2.4223.1 [call,call out=48b,read 0~352 out=352b,getxattrs out=218b] v0'0 uv1 ondisk = 0) v8 ==== 332+0+618 (crc 0 0 0) 0x7fc90400d7f0 con 0x55c8b6e38410 2023-10-05T19:28:31.514 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.506+0000 7fc9166f8e40 20 rados_obj.operate() r=0 bl.length=352 2023-10-05T19:28:31.514 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.506+0000 7fc9166f8e40 20 couldn't put binfo cache entry, might have raced with data changes 2023-10-05T19:28:31.514 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.506+0000 7fc9166f8e40 1 -- 172.21.15.167:0/3055819727 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:109 4.3d 4:bd5dd36d:::.dir.9a9df965-6612-45d4-a45f-6e3c889bacc2.4223.1.0:head [call rgw.bucket_list in=47b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x55c8b6ebfe00 con 0x55c8b6e38410 2023-10-05T19:28:31.514 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.506+0000 7fc9166f8e40 1 -- 172.21.15.167:0/3055819727 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:110 4.29 4:955f623b:::.dir.9a9df965-6612-45d4-a45f-6e3c889bacc2.4223.1.1:head [call rgw.bucket_list in=47b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x55c8b6ec6630 con 0x55c8b6e38410 2023-10-05T19:28:31.514 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.506+0000 7fc9166f8e40 1 -- 172.21.15.167:0/3055819727 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:111 4.38 4:1d83ede1:::.dir.9a9df965-6612-45d4-a45f-6e3c889bacc2.4223.1.2:head [call rgw.bucket_list in=47b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x55c8b6ec9150 con 0x55c8b6e017e0 2023-10-05T19:28:31.515 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.506+0000 7fc9166f8e40 1 -- 172.21.15.167:0/3055819727 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:112 4.1e 4:7939a9ac:::.dir.9a9df965-6612-45d4-a45f-6e3c889bacc2.4223.1.3:head [call rgw.bucket_list in=47b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x55c8b6ecba10 con 0x7fc8fc007030 2023-10-05T19:28:31.515 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.506+0000 7fc9166f8e40 1 -- 172.21.15.167:0/3055819727 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:113 4.e 4:719498d8:::.dir.9a9df965-6612-45d4-a45f-6e3c889bacc2.4223.1.4:head [call rgw.bucket_list in=47b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x55c8b6ece4e0 con 0x7fc904044040 2023-10-05T19:28:31.515 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.506+0000 7fc9166f8e40 1 -- 172.21.15.167:0/3055819727 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:114 4.32 4:4f38608c:::.dir.9a9df965-6612-45d4-a45f-6e3c889bacc2.4223.1.5:head [call rgw.bucket_list in=47b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x55c8b6ed0f00 con 0x55c8b6e017e0 2023-10-05T19:28:31.515 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.506+0000 7fc9166f8e40 1 -- 172.21.15.167:0/3055819727 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:115 4.38 4:1f2a6458:::.dir.9a9df965-6612-45d4-a45f-6e3c889bacc2.4223.1.6:head [call rgw.bucket_list in=47b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x55c8b6ed3cf0 con 0x55c8b6e017e0 2023-10-05T19:28:31.515 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.506+0000 7fc9166f8e40 1 -- 172.21.15.167:0/3055819727 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:116 4.2 4:41a3e149:::.dir.9a9df965-6612-45d4-a45f-6e3c889bacc2.4223.1.7:head [call rgw.bucket_list in=47b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x55c8b6ed6b90 con 0x55c8b6e017e0 2023-10-05T19:28:31.515 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.506+0000 7fc9166f8e40 1 -- 172.21.15.167:0/3055819727 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:117 4.17 4:ea8a113e:::.dir.9a9df965-6612-45d4-a45f-6e3c889bacc2.4223.1.8:head [call rgw.bucket_list in=47b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x55c8b6ed9a80 con 0x7fc8fc007030 2023-10-05T19:28:31.515 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.506+0000 7fc9166f8e40 1 -- 172.21.15.167:0/3055819727 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:118 4.1e 4:7a3e9338:::.dir.9a9df965-6612-45d4-a45f-6e3c889bacc2.4223.1.9:head [call rgw.bucket_list in=47b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x55c8b6edc930 con 0x7fc8fc007030 2023-10-05T19:28:31.515 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.506+0000 7fc9166f8e40 1 -- 172.21.15.167:0/3055819727 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:119 4.29 4:94432b39:::.dir.9a9df965-6612-45d4-a45f-6e3c889bacc2.4223.1.10:head [call rgw.bucket_list in=47b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x55c8b6edf830 con 0x55c8b6e38410 2023-10-05T19:28:31.515 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.510+0000 7fc914a2e700 1 -- 172.21.15.167:0/3055819727 <== osd.1 v2:172.21.15.167:6800/30035 37 ==== osd_op_reply(111 .dir.9a9df965-6612-45d4-a45f-6e3c889bacc2.4223.1.2 [call out=85b] v0'0 uv1 ondisk = 0) v8 ==== 194+0+85 (crc 0 0 0) 0x7fc90c052040 con 0x55c8b6e017e0 2023-10-05T19:28:31.515 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.510+0000 7fc914a2e700 1 -- 172.21.15.167:0/3055819727 <== osd.0 v2:172.21.15.118:6800/27690 29 ==== osd_op_reply(112 .dir.9a9df965-6612-45d4-a45f-6e3c889bacc2.4223.1.3 [call out=85b] v0'0 uv1 ondisk = 0) v8 ==== 194+0+85 (crc 0 0 0) 0x7fc90c052040 con 0x7fc8fc007030 2023-10-05T19:28:31.516 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.510+0000 7fc90bfff700 1 -- 172.21.15.167:0/3055819727 <== osd.2 v2:172.21.15.167:6816/30037 20 ==== osd_op_reply(113 .dir.9a9df965-6612-45d4-a45f-6e3c889bacc2.4223.1.4 [call out=85b] v0'0 uv1 ondisk = 0) v8 ==== 194+0+85 (crc 0 0 0) 0x7fc8fc0580e0 con 0x7fc904044040 2023-10-05T19:28:31.516 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.510+0000 7fc914a2e700 1 -- 172.21.15.167:0/3055819727 <== osd.0 v2:172.21.15.118:6800/27690 30 ==== osd_op_reply(117 .dir.9a9df965-6612-45d4-a45f-6e3c889bacc2.4223.1.8 [call out=85b] v0'0 uv1 ondisk = 0) v8 ==== 194+0+85 (crc 0 0 0) 0x7fc90c052040 con 0x7fc8fc007030 2023-10-05T19:28:31.516 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.510+0000 7fc91422d700 1 -- 172.21.15.167:0/3055819727 <== osd.3 v2:172.21.15.167:6808/30036 26 ==== osd_op_reply(109 .dir.9a9df965-6612-45d4-a45f-6e3c889bacc2.4223.1.0 [call out=85b] v0'0 uv1 ondisk = 0) v8 ==== 194+0+85 (crc 0 0 0) 0x7fc90400d7f0 con 0x55c8b6e38410 2023-10-05T19:28:31.516 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.510+0000 7fc914a2e700 1 -- 172.21.15.167:0/3055819727 <== osd.0 v2:172.21.15.118:6800/27690 31 ==== osd_op_reply(118 .dir.9a9df965-6612-45d4-a45f-6e3c889bacc2.4223.1.9 [call out=85b] v0'0 uv2 ondisk = 0) v8 ==== 194+0+85 (crc 0 0 0) 0x7fc90c052040 con 0x7fc8fc007030 2023-10-05T19:28:31.516 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.510+0000 7fc91422d700 1 -- 172.21.15.167:0/3055819727 <== osd.3 v2:172.21.15.167:6808/30036 27 ==== osd_op_reply(110 .dir.9a9df965-6612-45d4-a45f-6e3c889bacc2.4223.1.1 [call out=85b] v0'0 uv1 ondisk = 0) v8 ==== 194+0+85 (crc 0 0 0) 0x7fc90400d7f0 con 0x55c8b6e38410 2023-10-05T19:28:31.517 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.510+0000 7fc914a2e700 1 -- 172.21.15.167:0/3055819727 <== osd.1 v2:172.21.15.167:6800/30035 38 ==== osd_op_reply(115 .dir.9a9df965-6612-45d4-a45f-6e3c889bacc2.4223.1.6 [call out=85b] v0'0 uv2 ondisk = 0) v8 ==== 194+0+85 (crc 0 0 0) 0x7fc90c052040 con 0x55c8b6e017e0 2023-10-05T19:28:31.517 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.510+0000 7fc914a2e700 1 -- 172.21.15.167:0/3055819727 <== osd.1 v2:172.21.15.167:6800/30035 39 ==== osd_op_reply(114 .dir.9a9df965-6612-45d4-a45f-6e3c889bacc2.4223.1.5 [call out=85b] v0'0 uv1 ondisk = 0) v8 ==== 194+0+85 (crc 0 0 0) 0x7fc90c052040 con 0x55c8b6e017e0 2023-10-05T19:28:31.517 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.510+0000 7fc91422d700 1 -- 172.21.15.167:0/3055819727 <== osd.3 v2:172.21.15.167:6808/30036 28 ==== osd_op_reply(119 .dir.9a9df965-6612-45d4-a45f-6e3c889bacc2.4223.1.10 [call out=85b] v0'0 uv2 ondisk = 0) v8 ==== 195+0+85 (crc 0 0 0) 0x7fc90400d7f0 con 0x55c8b6e38410 2023-10-05T19:28:31.517 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.510+0000 7fc914a2e700 1 -- 172.21.15.167:0/3055819727 <== osd.1 v2:172.21.15.167:6800/30035 40 ==== osd_op_reply(116 .dir.9a9df965-6612-45d4-a45f-6e3c889bacc2.4223.1.7 [call out=85b] v0'0 uv1 ondisk = 0) v8 ==== 194+0+85 (crc 0 0 0) 0x7fc90c052040 con 0x55c8b6e017e0 2023-10-05T19:28:31.517 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.510+0000 7fc9166f8e40 1 -- 172.21.15.167:0/3055819727 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:120 7.1f 7:f95f44c2:::notify.0:head [watch unwatch cookie 94320550202960] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x55c8b6ece4e0 con 0x7fc904044040 2023-10-05T19:28:31.518 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.514+0000 7fc90bfff700 1 -- 172.21.15.167:0/3055819727 <== osd.2 v2:172.21.15.167:6816/30037 21 ==== osd_op_reply(120 notify.0 [watch unwatch cookie 94320550202960] v30'98 uv96 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7fc8fc0580e0 con 0x7fc904044040 2023-10-05T19:28:31.519 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.514+0000 7fc9166f8e40 20 remove_watcher() i=0 2023-10-05T19:28:31.519 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.514+0000 7fc9166f8e40 2 removed watcher, disabling cache 2023-10-05T19:28:31.519 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.514+0000 7fc9166f8e40 1 -- 172.21.15.167:0/3055819727 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:121 7.0 7:05bf5b68:::notify.1:head [watch unwatch cookie 94320550207840] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x55c8b6ece4e0 con 0x55c8b6e017e0 2023-10-05T19:28:31.521 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.514+0000 7fc914a2e700 1 -- 172.21.15.167:0/3055819727 <== osd.1 v2:172.21.15.167:6800/30035 41 ==== osd_op_reply(121 notify.1 [watch unwatch cookie 94320550207840] v30'98 uv96 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7fc90c052040 con 0x55c8b6e017e0 2023-10-05T19:28:31.522 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.514+0000 7fc9166f8e40 20 remove_watcher() i=1 2023-10-05T19:28:31.522 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.514+0000 7fc9166f8e40 1 -- 172.21.15.167:0/3055819727 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:122 7.15 7:a93a5511:::notify.2:head [watch unwatch cookie 94320550209840] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x55c8b6ece4e0 con 0x55c8b6e38410 2023-10-05T19:28:31.524 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.518+0000 7fc91422d700 1 -- 172.21.15.167:0/3055819727 <== osd.3 v2:172.21.15.167:6808/30036 29 ==== osd_op_reply(122 notify.2 [watch unwatch cookie 94320550209840] v30'98 uv96 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7fc90400d7f0 con 0x55c8b6e38410 2023-10-05T19:28:31.524 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.518+0000 7fc9166f8e40 20 remove_watcher() i=2 2023-10-05T19:28:31.524 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.518+0000 7fc9166f8e40 1 -- 172.21.15.167:0/3055819727 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:123 7.e 7:7759931f:::notify.3:head [watch unwatch cookie 94320550214256] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x55c8b6ece4e0 con 0x7fc904044040 2023-10-05T19:28:31.526 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.522+0000 7fc90bfff700 1 -- 172.21.15.167:0/3055819727 <== osd.2 v2:172.21.15.167:6816/30037 22 ==== osd_op_reply(123 notify.3 [watch unwatch cookie 94320550214256] v30'98 uv96 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7fc8fc0580e0 con 0x7fc904044040 2023-10-05T19:28:31.526 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.522+0000 7fc9166f8e40 20 remove_watcher() i=3 2023-10-05T19:28:31.526 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.522+0000 7fc9166f8e40 1 -- 172.21.15.167:0/3055819727 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:124 7.d 7:b4812045:::notify.4:head [watch unwatch cookie 94320550216464] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x55c8b6ece4e0 con 0x55c8b6e017e0 2023-10-05T19:28:31.529 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.522+0000 7fc914a2e700 1 -- 172.21.15.167:0/3055819727 <== osd.1 v2:172.21.15.167:6800/30035 42 ==== osd_op_reply(124 notify.4 [watch unwatch cookie 94320550216464] v30'98 uv96 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7fc90c052040 con 0x55c8b6e017e0 2023-10-05T19:28:31.529 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.522+0000 7fc9166f8e40 20 remove_watcher() i=4 2023-10-05T19:28:31.529 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.522+0000 7fc9166f8e40 1 -- 172.21.15.167:0/3055819727 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:125 7.3 7:c609908c:::notify.5:head [watch unwatch cookie 94320550210944] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x55c8b6ece4e0 con 0x7fc8fc007030 2023-10-05T19:28:31.532 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.526+0000 7fc914a2e700 1 -- 172.21.15.167:0/3055819727 <== osd.0 v2:172.21.15.118:6800/27690 32 ==== osd_op_reply(125 notify.5 [watch unwatch cookie 94320550210944] v30'98 uv96 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7fc90c052040 con 0x7fc8fc007030 2023-10-05T19:28:31.532 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.526+0000 7fc9166f8e40 20 remove_watcher() i=5 2023-10-05T19:28:31.532 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.526+0000 7fc9166f8e40 1 -- 172.21.15.167:0/3055819727 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:126 7.14 7:2b04a3e9:::notify.6:head [watch unwatch cookie 94320550222240] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x55c8b6ece4e0 con 0x7fc8fc007030 2023-10-05T19:28:31.535 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.530+0000 7fc914a2e700 1 -- 172.21.15.167:0/3055819727 <== osd.0 v2:172.21.15.118:6800/27690 33 ==== osd_op_reply(126 notify.6 [watch unwatch cookie 94320550222240] v30'98 uv96 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7fc90c052040 con 0x7fc8fc007030 2023-10-05T19:28:31.535 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.530+0000 7fc9166f8e40 20 remove_watcher() i=6 2023-10-05T19:28:31.535 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.530+0000 7fc9166f8e40 1 -- 172.21.15.167:0/3055819727 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:127 7.9 7:93e5b521:::notify.7:head [watch unwatch cookie 94320550224816] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x55c8b6ece4e0 con 0x7fc8fc007030 2023-10-05T19:28:31.538 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.534+0000 7fc914a2e700 1 -- 172.21.15.167:0/3055819727 <== osd.0 v2:172.21.15.118:6800/27690 34 ==== osd_op_reply(127 notify.7 [watch unwatch cookie 94320550224816] v30'98 uv96 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7fc90c052040 con 0x7fc8fc007030 2023-10-05T19:28:31.538 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.534+0000 7fc9166f8e40 20 remove_watcher() i=7 2023-10-05T19:28:31.540 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.534+0000 7fc8e5fe3700 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2023-10-05T19:28:31.540 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.534+0000 7fc9166f8e40 1 -- 172.21.15.167:0/3055819727 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:128 6.4 6:22d26bf9:::data_loggenerations_metadata:head [watch unwatch cookie 94320550227200] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x55c8b6ece4e0 con 0x55c8b6e017e0 2023-10-05T19:28:31.544 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.538+0000 7fc914a2e700 1 -- 172.21.15.167:0/3055819727 <== osd.1 v2:172.21.15.167:6800/30035 43 ==== osd_op_reply(128 data_loggenerations_metadata [watch unwatch cookie 94320550227200] v30'100 uv1 ondisk = 0) v8 ==== 172+0+0 (crc 0 0 0) 0x7fc90c052040 con 0x55c8b6e017e0 2023-10-05T19:28:31.544 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.538+0000 7fc9166f8e40 1 -- 172.21.15.167:0/3055819727 >> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] conn(0x7fc8fc007030 msgr2=0x7fc8fc01c920 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-05T19:28:31.544 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.538+0000 7fc9166f8e40 1 --2- 172.21.15.167:0/3055819727 >> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] conn(0x7fc8fc007030 0x7fc8fc01c920 crc :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-05T19:28:31.544 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.538+0000 7fc9166f8e40 1 -- 172.21.15.167:0/3055819727 >> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] conn(0x55c8b6e017e0 msgr2=0x55c8b6e349c0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-05T19:28:31.544 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.538+0000 7fc9166f8e40 1 --2- 172.21.15.167:0/3055819727 >> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] conn(0x55c8b6e017e0 0x55c8b6e349c0 crc :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-05T19:28:31.544 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.538+0000 7fc9166f8e40 1 -- 172.21.15.167:0/3055819727 >> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] conn(0x7fc904044040 msgr2=0x7fc9040420e0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-05T19:28:31.544 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.538+0000 7fc9166f8e40 1 --2- 172.21.15.167:0/3055819727 >> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] conn(0x7fc904044040 0x7fc9040420e0 crc :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-05T19:28:31.544 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.538+0000 7fc9166f8e40 1 -- 172.21.15.167:0/3055819727 >> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] conn(0x55c8b6e38410 msgr2=0x55c8b6e3a7f0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-05T19:28:31.545 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.538+0000 7fc9166f8e40 1 --2- 172.21.15.167:0/3055819727 >> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] conn(0x55c8b6e38410 0x55c8b6e3a7f0 crc :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-05T19:28:31.545 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.538+0000 7fc9166f8e40 1 -- 172.21.15.167:0/3055819727 >> [v2:172.21.15.167:6824/29889,v1:172.21.15.167:6825/29889] conn(0x7fc8f4030c40 msgr2=0x7fc8f40330f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-05T19:28:31.545 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.538+0000 7fc9166f8e40 1 --2- 172.21.15.167:0/3055819727 >> [v2:172.21.15.167:6824/29889,v1:172.21.15.167:6825/29889] conn(0x7fc8f4030c40 0x7fc8f40330f0 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7fc8fc005a90 tx=0x7fc8fc015000 comp rx=0 tx=0).stop 2023-10-05T19:28:31.545 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.538+0000 7fc9166f8e40 1 -- 172.21.15.167:0/3055819727 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55c8b6e01410 msgr2=0x55c8b6e28f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-05T19:28:31.545 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.538+0000 7fc9166f8e40 1 --2- 172.21.15.167:0/3055819727 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55c8b6e01410 0x55c8b6e28f30 secure :-1 s=READY pgs=329 cs=0 l=1 rev1=1 crypto rx=0x7fc904000c00 tx=0x7fc904016000 comp rx=0 tx=0).stop 2023-10-05T19:28:31.545 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.538+0000 7fc914a2e700 1 -- 172.21.15.167:0/3055819727 reap_dead start 2023-10-05T19:28:31.545 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.538+0000 7fc9166f8e40 1 -- 172.21.15.167:0/3055819727 shutdown_connections 2023-10-05T19:28:31.545 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.538+0000 7fc9166f8e40 1 --2- 172.21.15.167:0/3055819727 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55c8b6e01410 0x55c8b6e28f30 unknown :-1 s=CLOSED pgs=329 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-05T19:28:31.545 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.538+0000 7fc9166f8e40 1 -- 172.21.15.167:0/3055819727 >> 172.21.15.167:0/3055819727 conn(0x55c8b6dfb720 msgr2=0x55c8b6e1c7a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-05T19:28:31.545 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.538+0000 7fc9166f8e40 1 -- 172.21.15.167:0/3055819727 shutdown_connections 2023-10-05T19:28:31.545 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.538+0000 7fc9166f8e40 1 -- 172.21.15.167:0/3055819727 wait complete. 2023-10-05T19:28:31.546 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.538+0000 7fc9166f8e40 1 -- 172.21.15.167:0/39568890 >> [v2:172.21.15.167:6824/29889,v1:172.21.15.167:6825/29889] conn(0x7fc854030c00 msgr2=0x7fc8540330b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-05T19:28:31.546 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.538+0000 7fc9166f8e40 1 --2- 172.21.15.167:0/39568890 >> [v2:172.21.15.167:6824/29889,v1:172.21.15.167:6825/29889] conn(0x7fc854030c00 0x7fc8540330b0 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x55c8b6e43700 tx=0x7fc90c00aec0 comp rx=0 tx=0).stop 2023-10-05T19:28:31.547 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.538+0000 7fc9166f8e40 1 -- 172.21.15.167:0/39568890 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55c8b6e506d0 msgr2=0x55c8b6e56180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-05T19:28:31.547 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.538+0000 7fc9166f8e40 1 --2- 172.21.15.167:0/39568890 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55c8b6e506d0 0x55c8b6e56180 secure :-1 s=READY pgs=331 cs=0 l=1 rev1=1 crypto rx=0x55c8b6e4a7b0 tx=0x7fc8fc02f000 comp rx=0 tx=0).stop 2023-10-05T19:28:31.547 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.538+0000 7fc9166f8e40 1 -- 172.21.15.167:0/39568890 shutdown_connections 2023-10-05T19:28:31.547 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.538+0000 7fc9166f8e40 1 --2- 172.21.15.167:0/39568890 >> [v2:172.21.15.167:6824/29889,v1:172.21.15.167:6825/29889] conn(0x7fc854030c00 0x7fc8540330b0 unknown :-1 s=CLOSED pgs=117 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-05T19:28:31.547 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.538+0000 7fc9166f8e40 1 --2- 172.21.15.167:0/39568890 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55c8b6e506d0 0x55c8b6e56180 unknown :-1 s=CLOSED pgs=331 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-05T19:28:31.547 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.538+0000 7fc9166f8e40 1 -- 172.21.15.167:0/39568890 >> 172.21.15.167:0/39568890 conn(0x55c8b6e4ba90 msgr2=0x55c8b6e49b90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-05T19:28:31.547 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.538+0000 7fc9166f8e40 1 -- 172.21.15.167:0/39568890 shutdown_connections 2023-10-05T19:28:31.547 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.538+0000 7fc9166f8e40 1 -- 172.21.15.167:0/39568890 wait complete. 2023-10-05T19:28:31.548 INFO:teuthology.orchestra.run.smithi167.stdout:[ 2023-10-05T19:28:31.548 INFO:teuthology.orchestra.run.smithi167.stdout: { 2023-10-05T19:28:31.548 INFO:teuthology.orchestra.run.smithi167.stdout: "bucket": "myfoo", 2023-10-05T19:28:31.548 INFO:teuthology.orchestra.run.smithi167.stdout: "num_shards": 11, 2023-10-05T19:28:31.548 INFO:teuthology.orchestra.run.smithi167.stdout: "tenant": "", 2023-10-05T19:28:31.548 INFO:teuthology.orchestra.run.smithi167.stdout: "zonegroup": "4f1f719d-9648-417b-b10c-e5d939825419", 2023-10-05T19:28:31.548 INFO:teuthology.orchestra.run.smithi167.stdout: "placement_rule": "default-placement", 2023-10-05T19:28:31.548 INFO:teuthology.orchestra.run.smithi167.stdout: "explicit_placement": { 2023-10-05T19:28:31.548 INFO:teuthology.orchestra.run.smithi167.stdout: "data_pool": "", 2023-10-05T19:28:31.548 INFO:teuthology.orchestra.run.smithi167.stdout: "data_extra_pool": "", 2023-10-05T19:28:31.548 INFO:teuthology.orchestra.run.smithi167.stdout: "index_pool": "" 2023-10-05T19:28:31.549 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2023-10-05T19:28:31.549 INFO:teuthology.orchestra.run.smithi167.stdout: "id": "9a9df965-6612-45d4-a45f-6e3c889bacc2.4223.1", 2023-10-05T19:28:31.549 INFO:teuthology.orchestra.run.smithi167.stdout: "marker": "9a9df965-6612-45d4-a45f-6e3c889bacc2.4223.1", 2023-10-05T19:28:31.549 INFO:teuthology.orchestra.run.smithi167.stdout: "index_type": "Normal", 2023-10-05T19:28:31.549 INFO:teuthology.orchestra.run.smithi167.stdout: "owner": "foo", 2023-10-05T19:28:31.549 INFO:teuthology.orchestra.run.smithi167.stdout: "ver": "0#1,1#1,2#1,3#1,4#1,5#1,6#1,7#1,8#1,9#1,10#1", 2023-10-05T19:28:31.549 INFO:teuthology.orchestra.run.smithi167.stdout: "master_ver": "0#0,1#0,2#0,3#0,4#0,5#0,6#0,7#0,8#0,9#0,10#0", 2023-10-05T19:28:31.549 INFO:teuthology.orchestra.run.smithi167.stdout: "mtime": "2023-10-05T19:28:30.258873Z", 2023-10-05T19:28:31.549 INFO:teuthology.orchestra.run.smithi167.stdout: "creation_time": "2023-10-05T19:28:30.250836Z", 2023-10-05T19:28:31.549 INFO:teuthology.orchestra.run.smithi167.stdout: "max_marker": "0#,1#,2#,3#,4#,5#,6#,7#,8#,9#,10#", 2023-10-05T19:28:31.549 INFO:teuthology.orchestra.run.smithi167.stdout: "usage": {}, 2023-10-05T19:28:31.549 INFO:teuthology.orchestra.run.smithi167.stdout: "bucket_quota": { 2023-10-05T19:28:31.550 INFO:teuthology.orchestra.run.smithi167.stdout: "enabled": false, 2023-10-05T19:28:31.550 INFO:teuthology.orchestra.run.smithi167.stdout: "check_on_raw": false, 2023-10-05T19:28:31.550 INFO:teuthology.orchestra.run.smithi167.stdout: "max_size": -1, 2023-10-05T19:28:31.550 INFO:teuthology.orchestra.run.smithi167.stdout: "max_size_kb": 0, 2023-10-05T19:28:31.550 INFO:teuthology.orchestra.run.smithi167.stdout: "max_objects": -1 2023-10-05T19:28:31.550 INFO:teuthology.orchestra.run.smithi167.stdout: } 2023-10-05T19:28:31.550 INFO:teuthology.orchestra.run.smithi167.stdout: } 2023-10-05T19:28:31.550 INFO:teuthology.orchestra.run.smithi167.stdout:] 2023-10-05T19:28:31.551 DEBUG:tasks.util.rgw: json result: [{'bucket': 'myfoo', 'num_shards': 11, 'tenant': '', 'zonegroup': '4f1f719d-9648-417b-b10c-e5d939825419', 'placement_rule': 'default-placement', 'explicit_placement': {'data_pool': '', 'data_extra_pool': '', 'index_pool': ''}, 'id': '9a9df965-6612-45d4-a45f-6e3c889bacc2.4223.1', 'marker': '9a9df965-6612-45d4-a45f-6e3c889bacc2.4223.1', 'index_type': 'Normal', 'owner': 'foo', 'ver': '0#1,1#1,2#1,3#1,4#1,5#1,6#1,7#1,8#1,9#1,10#1', 'master_ver': '0#0,1#0,2#0,3#0,4#0,5#0,6#0,7#0,8#0,9#0,10#0', 'mtime': '2023-10-05T19:28:30.258873Z', 'creation_time': '2023-10-05T19:28:30.250836Z', 'max_marker': '0#,1#,2#,3#,4#,5#,6#,7#,8#,9#,10#', 'usage': {}, 'bucket_quota': {'enabled': False, 'check_on_raw': False, 'max_size': -1, 'max_size_kb': 0, 'max_objects': -1}}] 2023-10-05T19:28:31.565 INFO:tasks.radosgw_admin:RL: put_obj myfoo foo bytes_out=3 bytes_in=0 failed=False 2023-10-05T19:28:31.565 INFO:tasks.util.rgw:rgwadmin: client.0 : ['bucket', 'stats', '--bucket', 'myfoo'] 2023-10-05T19:28:31.565 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'ceph', 'bucket', 'stats', '--bucket', 'myfoo'] 2023-10-05T19:28:31.566 DEBUG:teuthology.orchestra.run.smithi167:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster ceph bucket stats --bucket myfoo 2023-10-05T19:28:31.634 INFO:teuthology.orchestra.run.smithi167.stderr:ignoring --setuser ceph since I am not root 2023-10-05T19:28:31.709 INFO:teuthology.orchestra.run.smithi167.stderr:ignoring --setgroup ceph since I am not root 2023-10-05T19:28:31.732 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.630+0000 7fe5089a5e40 1 Processor -- start 2023-10-05T19:28:31.732 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.630+0000 7fe5089a5e40 1 -- start start 2023-10-05T19:28:31.732 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.630+0000 7fe5089a5e40 1 --2- >> v2:172.21.15.118:3300/0 conn(0x565386e76600 0x565386e789e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-05T19:28:31.732 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.630+0000 7fe5089a5e40 1 -- --> v1:172.21.15.118:6789/0 -- auth(proto 0 26 bytes epoch 0) v1 -- 0x565386dbcbd0 con 0x565386e72020 2023-10-05T19:28:31.732 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.630+0000 7fe5089a5e40 1 -- --> v2:172.21.15.118:3300/0 -- mon_getmap magic: 0 v1 -- 0x565386d79ee0 con 0x565386e76600 2023-10-05T19:28:31.732 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.630+0000 7fe505cd9700 1 --2- >> v2:172.21.15.118:3300/0 conn(0x565386e76600 0x565386e789e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-05T19:28:31.733 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.630+0000 7fe5064da700 1 --1- >> v1:172.21.15.118:6789/0 conn(0x565386e72020 0x565386e723f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.118:6789/0 says I am v1:172.21.15.167:56594/0 (socket says 172.21.15.167:56594) 2023-10-05T19:28:31.733 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.630+0000 7fe5064da700 1 -- 172.21.15.167:0/2906185900 learned_addr learned my addr 172.21.15.167:0/2906185900 (peer_addr_for_me v1:172.21.15.167:0/0) 2023-10-05T19:28:31.733 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.630+0000 7fe505cd9700 1 --2- >> v2:172.21.15.118:3300/0 conn(0x565386e76600 0x565386e789e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.118:3300/0 says I am v2:172.21.15.167:58228/0 (socket says 172.21.15.167:58228) 2023-10-05T19:28:31.733 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.630+0000 7fe5054d8700 1 -- 172.21.15.167:0/2906185900 <== mon.0 v1:172.21.15.118:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1114901763 0 0) 0x565386dbcbd0 con 0x565386e72020 2023-10-05T19:28:31.733 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.630+0000 7fe5054d8700 1 -- 172.21.15.167:0/2906185900 --> v1:172.21.15.118:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe4e8001e10 con 0x565386e72020 2023-10-05T19:28:31.733 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.630+0000 7fe505cd9700 1 -- 172.21.15.167:0/2906185900 >> v1:172.21.15.118:6789/0 conn(0x565386e72020 legacy=0x565386e723f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-05T19:28:31.733 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.630+0000 7fe505cd9700 1 -- 172.21.15.167:0/2906185900 --> v2:172.21.15.118:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x565386e72c30 con 0x565386e76600 2023-10-05T19:28:31.733 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.630+0000 7fe505cd9700 1 --2- 172.21.15.167:0/2906185900 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x565386e76600 0x565386e789e0 secure :-1 s=READY pgs=333 cs=0 l=1 rev1=1 crypto rx=0x7fe4f000a5d0 tx=0x7fe4f0005a50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ee621d709d70daad server_cookie=0 in_seq=0 out_seq=0 2023-10-05T19:28:31.733 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.630+0000 7fe5054d8700 1 -- 172.21.15.167:0/2906185900 <== mon.0 v2:172.21.15.118:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fe4f0007610 con 0x565386e76600 2023-10-05T19:28:31.734 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.630+0000 7fe5054d8700 1 -- 172.21.15.167:0/2906185900 <== mon.0 v2:172.21.15.118:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe4f0007770 con 0x565386e76600 2023-10-05T19:28:31.734 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.630+0000 7fe5054d8700 1 -- 172.21.15.167:0/2906185900 <== mon.0 v2:172.21.15.118:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fe4f0007a20 con 0x565386e76600 2023-10-05T19:28:31.734 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.630+0000 7fe5089a5e40 1 -- 172.21.15.167:0/2906185900 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x565386e76600 msgr2=0x565386e789e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-05T19:28:31.734 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.630+0000 7fe5089a5e40 1 --2- 172.21.15.167:0/2906185900 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x565386e76600 0x565386e789e0 secure :-1 s=READY pgs=333 cs=0 l=1 rev1=1 crypto rx=0x7fe4f000a5d0 tx=0x7fe4f0005a50 comp rx=0 tx=0).stop 2023-10-05T19:28:31.734 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.630+0000 7fe5089a5e40 1 -- 172.21.15.167:0/2906185900 shutdown_connections 2023-10-05T19:28:31.734 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.630+0000 7fe5089a5e40 1 --2- 172.21.15.167:0/2906185900 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x565386e76600 0x565386e789e0 unknown :-1 s=CLOSED pgs=333 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-05T19:28:31.734 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.630+0000 7fe5089a5e40 1 -- 172.21.15.167:0/2906185900 >> 172.21.15.167:0/2906185900 conn(0x565386de5870 msgr2=0x565386e70510 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-05T19:28:31.734 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.630+0000 7fe5089a5e40 1 -- 172.21.15.167:0/2906185900 shutdown_connections 2023-10-05T19:28:31.734 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.634+0000 7fe5089a5e40 1 -- 172.21.15.167:0/2906185900 wait complete. 2023-10-05T19:28:31.734 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.634+0000 7fe5089a5e40 1 Processor -- start 2023-10-05T19:28:31.735 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.634+0000 7fe5089a5e40 1 -- start start 2023-10-05T19:28:31.735 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.634+0000 7fe5089a5e40 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x565386e72020 0x565386e90f40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-05T19:28:31.735 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.634+0000 7fe5089a5e40 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x565386d3de30 con 0x565386e72020 2023-10-05T19:28:31.735 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.634+0000 7fe5064da700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x565386e72020 0x565386e90f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-05T19:28:31.735 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.634+0000 7fe5064da700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x565386e72020 0x565386e90f40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.118:3300/0 says I am v2:172.21.15.167:58234/0 (socket says 172.21.15.167:58234) 2023-10-05T19:28:31.735 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.634+0000 7fe5064da700 1 -- 172.21.15.167:0/3478554177 learned_addr learned my addr 172.21.15.167:0/3478554177 (peer_addr_for_me v2:172.21.15.167:0/0) 2023-10-05T19:28:31.735 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.634+0000 7fe5064da700 1 -- 172.21.15.167:0/3478554177 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe4f0005cf0 con 0x565386e72020 2023-10-05T19:28:31.735 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.634+0000 7fe5064da700 1 --2- 172.21.15.167:0/3478554177 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x565386e72020 0x565386e90f40 secure :-1 s=READY pgs=334 cs=0 l=1 rev1=1 crypto rx=0x7fe500001c30 tx=0x7fe500001810 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4766d2a0c18253bb server_cookie=0 in_seq=0 out_seq=0 2023-10-05T19:28:31.735 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.634+0000 7fe4fffff700 1 -- 172.21.15.167:0/3478554177 <== mon.0 v2:172.21.15.118:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fe50000bbf0 con 0x565386e72020 2023-10-05T19:28:31.736 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.634+0000 7fe4fffff700 1 -- 172.21.15.167:0/3478554177 <== mon.0 v2:172.21.15.118:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe50000bd50 con 0x565386e72020 2023-10-05T19:28:31.736 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.634+0000 7fe4fffff700 1 -- 172.21.15.167:0/3478554177 <== mon.0 v2:172.21.15.118:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fe50000e470 con 0x565386e72020 2023-10-05T19:28:31.736 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.634+0000 7fe5089a5e40 1 -- 172.21.15.167:0/3478554177 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x565386e72020 msgr2=0x565386e90f40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-05T19:28:31.736 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.634+0000 7fe5089a5e40 1 --2- 172.21.15.167:0/3478554177 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x565386e72020 0x565386e90f40 secure :-1 s=READY pgs=334 cs=0 l=1 rev1=1 crypto rx=0x7fe500001c30 tx=0x7fe500001810 comp rx=0 tx=0).stop 2023-10-05T19:28:31.736 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.634+0000 7fe5089a5e40 1 -- 172.21.15.167:0/3478554177 shutdown_connections 2023-10-05T19:28:31.736 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.634+0000 7fe5089a5e40 1 --2- 172.21.15.167:0/3478554177 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x565386e72020 0x565386e90f40 unknown :-1 s=CLOSED pgs=334 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-05T19:28:31.736 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.634+0000 7fe5089a5e40 1 -- 172.21.15.167:0/3478554177 >> 172.21.15.167:0/3478554177 conn(0x565386de5870 msgr2=0x565386e8cb00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-05T19:28:31.736 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.634+0000 7fe5089a5e40 1 -- 172.21.15.167:0/3478554177 shutdown_connections 2023-10-05T19:28:31.736 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.634+0000 7fe5089a5e40 1 -- 172.21.15.167:0/3478554177 wait complete. 2023-10-05T19:28:31.737 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.634+0000 7fe5089a5e40 1 Processor -- start 2023-10-05T19:28:31.737 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.638+0000 7fe5089a5e40 1 -- start start 2023-10-05T19:28:31.737 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.638+0000 7fe5089a5e40 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x565386e72020 0x565386e97cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-05T19:28:31.737 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.638+0000 7fe5089a5e40 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x565386d10ed0 con 0x565386e72020 2023-10-05T19:28:31.737 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.638+0000 7fe5064da700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x565386e72020 0x565386e97cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-05T19:28:31.737 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.638+0000 7fe5064da700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x565386e72020 0x565386e97cd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.118:3300/0 says I am v2:172.21.15.167:58240/0 (socket says 172.21.15.167:58240) 2023-10-05T19:28:31.737 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.638+0000 7fe5064da700 1 -- 172.21.15.167:0/3045448955 learned_addr learned my addr 172.21.15.167:0/3045448955 (peer_addr_for_me v2:172.21.15.167:0/0) 2023-10-05T19:28:31.737 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.638+0000 7fe5064da700 1 -- 172.21.15.167:0/3045448955 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe4e8001e10 con 0x565386e72020 2023-10-05T19:28:31.737 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.638+0000 7fe5064da700 1 --2- 172.21.15.167:0/3045448955 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x565386e72020 0x565386e97cd0 secure :-1 s=READY pgs=335 cs=0 l=1 rev1=1 crypto rx=0x7fe50000b960 tx=0x7fe500014000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-05T19:28:31.738 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.638+0000 7fe4fe7fc700 1 -- 172.21.15.167:0/3045448955 <== mon.0 v2:172.21.15.118:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fe50001f070 con 0x565386e72020 2023-10-05T19:28:31.738 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.638+0000 7fe4fe7fc700 1 -- 172.21.15.167:0/3045448955 <== mon.0 v2:172.21.15.118:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe50001a2e0 con 0x565386e72020 2023-10-05T19:28:31.738 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.638+0000 7fe5089a5e40 1 -- 172.21.15.167:0/3045448955 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x565386e98210 con 0x565386e72020 2023-10-05T19:28:31.738 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.638+0000 7fe4fe7fc700 1 -- 172.21.15.167:0/3045448955 <== mon.0 v2:172.21.15.118:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fe50000e470 con 0x565386e72020 2023-10-05T19:28:31.738 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.638+0000 7fe5089a5e40 1 -- 172.21.15.167:0/3045448955 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x565386e98650 con 0x565386e72020 2023-10-05T19:28:31.738 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.638+0000 7fe4fe7fc700 1 -- 172.21.15.167:0/3045448955 <== mon.0 v2:172.21.15.118:3300/0 4 ==== mgrmap(e 5) v1 ==== 40329+0+0 (secure 0 0 0) 0x7fe50001aa10 con 0x565386e72020 2023-10-05T19:28:31.738 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.638+0000 7fe5089a5e40 20 rados->read ofs=0 len=0 2023-10-05T19:28:31.738 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.638+0000 7fe4fe7fc700 1 --2- 172.21.15.167:0/3045448955 >> [v2:172.21.15.167:6824/29889,v1:172.21.15.167:6825/29889] conn(0x7fe4ec030bf0 0x7fe4ec0330a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-05T19:28:31.738 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.642+0000 7fe4fe7fc700 1 -- 172.21.15.167:0/3045448955 <== mon.0 v2:172.21.15.118:3300/0 5 ==== osd_map(30..30 src has 1..30) v4 ==== 6057+0+0 (secure 0 0 0) 0x7fe500045300 con 0x565386e72020 2023-10-05T19:28:31.738 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.642+0000 7fe505cd9700 1 --2- 172.21.15.167:0/3045448955 >> [v2:172.21.15.167:6824/29889,v1:172.21.15.167:6825/29889] conn(0x7fe4ec030bf0 0x7fe4ec0330a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-05T19:28:31.739 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.642+0000 7fe5089a5e40 1 --2- 172.21.15.167:0/3045448955 >> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] conn(0x565386e76600 0x565386ea3730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-05T19:28:31.739 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.642+0000 7fe5089a5e40 1 -- 172.21.15.167:0/3045448955 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:1 5.12 5:49953fa1:::default.realm:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x565386ea3de0 con 0x565386e76600 2023-10-05T19:28:31.739 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.642+0000 7fe506cdb700 1 --2- 172.21.15.167:0/3045448955 >> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] conn(0x565386e76600 0x565386ea3730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-05T19:28:31.739 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.642+0000 7fe505cd9700 1 --2- 172.21.15.167:0/3045448955 >> [v2:172.21.15.167:6824/29889,v1:172.21.15.167:6825/29889] conn(0x7fe4ec030bf0 0x7fe4ec0330a0 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7fe4f00030c0 tx=0x7fe4f0014000 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-05T19:28:31.739 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.642+0000 7fe506cdb700 1 --2- 172.21.15.167:0/3045448955 >> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] conn(0x565386e76600 0x565386ea3730 crc :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-05T19:28:31.739 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.642+0000 7fe506cdb700 1 -- 172.21.15.167:0/3045448955 <== osd.1 v2:172.21.15.167:6800/30035 1 ==== osd_op_reply(1 default.realm [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 ==== 157+0+0 (crc 0 0 0) 0x7fe4f800a5a0 con 0x565386e76600 2023-10-05T19:28:31.739 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.642+0000 7fe5089a5e40 20 rados_obj.operate() r=-2 bl.length=0 2023-10-05T19:28:31.739 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.642+0000 7fe5089a5e40 20 realm 2023-10-05T19:28:31.739 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.642+0000 7fe5089a5e40 20 rados->read ofs=0 len=0 2023-10-05T19:28:31.739 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.642+0000 7fe5089a5e40 1 -- 172.21.15.167:0/3045448955 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:2 5.12 5:49953fa1:::default.realm:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x565386ea0d50 con 0x565386e76600 2023-10-05T19:28:31.740 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.642+0000 7fe506cdb700 1 -- 172.21.15.167:0/3045448955 <== osd.1 v2:172.21.15.167:6800/30035 2 ==== osd_op_reply(2 default.realm [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 ==== 157+0+0 (crc 0 0 0) 0x7fe4f800a5a0 con 0x565386e76600 2023-10-05T19:28:31.740 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.642+0000 7fe5089a5e40 20 rados_obj.operate() r=-2 bl.length=0 2023-10-05T19:28:31.740 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.642+0000 7fe5089a5e40 4 RGWPeriod::init failed to init realm id : (2) No such file or directory 2023-10-05T19:28:31.740 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.642+0000 7fe5089a5e40 20 rados->read ofs=0 len=0 2023-10-05T19:28:31.740 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.642+0000 7fe5089a5e40 1 -- 172.21.15.167:0/3045448955 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:3 5.12 5:49953fa1:::default.realm:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x565386ea48c0 con 0x565386e76600 2023-10-05T19:28:31.740 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.642+0000 7fe506cdb700 1 -- 172.21.15.167:0/3045448955 <== osd.1 v2:172.21.15.167:6800/30035 3 ==== osd_op_reply(3 default.realm [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 ==== 157+0+0 (crc 0 0 0) 0x7fe4f800a5a0 con 0x565386e76600 2023-10-05T19:28:31.740 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.642+0000 7fe5089a5e40 20 rados_obj.operate() r=-2 bl.length=0 2023-10-05T19:28:31.740 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.642+0000 7fe5089a5e40 20 rados->read ofs=0 len=0 2023-10-05T19:28:31.740 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.642+0000 7fe5089a5e40 1 -- 172.21.15.167:0/3045448955 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:4 5.f 5:f43b6ece:::zone_names.default:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x565386ea4470 con 0x565386e76600 2023-10-05T19:28:31.741 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.646+0000 7fe506cdb700 1 -- 172.21.15.167:0/3045448955 <== osd.1 v2:172.21.15.167:6800/30035 4 ==== osd_op_reply(4 zone_names.default [read 0~46 out=46b] v0'0 uv1 ondisk = 0) v8 ==== 162+0+46 (crc 0 0 0) 0x7fe4f800a5a0 con 0x565386e76600 2023-10-05T19:28:31.741 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.646+0000 7fe5089a5e40 20 rados_obj.operate() r=0 bl.length=46 2023-10-05T19:28:31.741 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.646+0000 7fe5089a5e40 20 rados->read ofs=0 len=0 2023-10-05T19:28:31.741 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.646+0000 7fe5089a5e40 1 -- 172.21.15.167:0/3045448955 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:5 5.1d 5:b8374eb4:::zone_info.9a9df965-6612-45d4-a45f-6e3c889bacc2:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x565386ea52f0 con 0x565386e76600 2023-10-05T19:28:31.741 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.646+0000 7fe506cdb700 1 -- 172.21.15.167:0/3045448955 <== osd.1 v2:172.21.15.167:6800/30035 5 ==== osd_op_reply(5 zone_info.9a9df965-6612-45d4-a45f-6e3c889bacc2 [read 0~874 out=874b] v0'0 uv2 ondisk = 0) v8 ==== 190+0+874 (crc 0 0 0) 0x7fe4f800a5a0 con 0x565386e76600 2023-10-05T19:28:31.741 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.646+0000 7fe5089a5e40 20 rados_obj.operate() r=0 bl.length=874 2023-10-05T19:28:31.741 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.646+0000 7fe5089a5e40 20 searching for the correct realm 2023-10-05T19:28:31.741 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.646+0000 7fe5089a5e40 1 --2- 172.21.15.167:0/3045448955 >> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] conn(0x565386ea7140 0x565386ea9520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-05T19:28:31.741 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.646+0000 7fe5089a5e40 1 -- 172.21.15.167:0/3045448955 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:6 5.0 5:00000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x565386ea9a60 con 0x565386ea7140 2023-10-05T19:28:31.742 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.646+0000 7fe5064da700 1 --2- 172.21.15.167:0/3045448955 >> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] conn(0x565386ea7140 0x565386ea9520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-05T19:28:31.742 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.646+0000 7fe5064da700 1 --2- 172.21.15.167:0/3045448955 >> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] conn(0x565386ea7140 0x565386ea9520 crc :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-05T19:28:31.742 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.646+0000 7fe5064da700 1 -- 172.21.15.167:0/3045448955 <== osd.3 v2:172.21.15.167:6808/30036 1 ==== osd_op_reply(6 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fe500014070 con 0x565386ea7140 2023-10-05T19:28:31.742 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.646+0000 7fe5064da700 1 --2- 172.21.15.167:0/3045448955 >> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] conn(0x7fe500016730 0x7fe500044a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-05T19:28:31.742 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.646+0000 7fe5064da700 1 -- 172.21.15.167:0/3045448955 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:7 5.10 5:08000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x565386ea9a60 con 0x7fe500016730 2023-10-05T19:28:31.742 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.646+0000 7fe505cd9700 1 --2- 172.21.15.167:0/3045448955 >> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] conn(0x7fe500016730 0x7fe500044a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-05T19:28:31.742 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.646+0000 7fe505cd9700 1 --2- 172.21.15.167:0/3045448955 >> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] conn(0x7fe500016730 0x7fe500044a90 crc :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-05T19:28:31.742 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.650+0000 7fe505cd9700 1 -- 172.21.15.167:0/3045448955 <== osd.2 v2:172.21.15.167:6816/30037 1 ==== osd_op_reply(7 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fe4f00079f0 con 0x7fe500016730 2023-10-05T19:28:31.742 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.650+0000 7fe505cd9700 1 --2- 172.21.15.167:0/3045448955 >> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] conn(0x7fe4f0007030 0x7fe4f001d430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-05T19:28:31.742 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.650+0000 7fe505cd9700 1 -- 172.21.15.167:0/3045448955 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:8 5.8 5:10000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x565386ea9a60 con 0x7fe4f0007030 2023-10-05T19:28:31.743 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.650+0000 7fe506cdb700 1 --2- 172.21.15.167:0/3045448955 >> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] conn(0x7fe4f0007030 0x7fe4f001d430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-05T19:28:31.743 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.650+0000 7fe506cdb700 1 --2- 172.21.15.167:0/3045448955 >> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] conn(0x7fe4f0007030 0x7fe4f001d430 crc :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-05T19:28:31.743 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.650+0000 7fe506cdb700 1 -- 172.21.15.167:0/3045448955 <== osd.0 v2:172.21.15.118:6800/27690 1 ==== osd_op_reply(8 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fe4f800a5a0 con 0x7fe4f0007030 2023-10-05T19:28:31.743 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.650+0000 7fe506cdb700 1 -- 172.21.15.167:0/3045448955 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:9 5.18 5:18000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x565386ea9a60 con 0x565386e76600 2023-10-05T19:28:31.743 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.650+0000 7fe506cdb700 1 -- 172.21.15.167:0/3045448955 <== osd.1 v2:172.21.15.167:6800/30035 6 ==== osd_op_reply(9 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fe4f800a5a0 con 0x565386e76600 2023-10-05T19:28:31.743 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.650+0000 7fe506cdb700 1 -- 172.21.15.167:0/3045448955 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:10 5.4 5:20000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x565386ea9a60 con 0x565386e76600 2023-10-05T19:28:31.743 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.650+0000 7fe506cdb700 1 -- 172.21.15.167:0/3045448955 <== osd.1 v2:172.21.15.167:6800/30035 7 ==== osd_op_reply(10 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fe4f800a5a0 con 0x565386e76600 2023-10-05T19:28:31.743 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.650+0000 7fe506cdb700 1 -- 172.21.15.167:0/3045448955 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:11 5.14 5:28000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x565386ea9a60 con 0x7fe4f0007030 2023-10-05T19:28:31.743 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.654+0000 7fe506cdb700 1 -- 172.21.15.167:0/3045448955 <== osd.0 v2:172.21.15.118:6800/27690 2 ==== osd_op_reply(11 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fe4f800a5a0 con 0x7fe4f0007030 2023-10-05T19:28:31.743 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.654+0000 7fe506cdb700 1 -- 172.21.15.167:0/3045448955 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:12 5.c 5:30000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x565386ea9a60 con 0x7fe4f0007030 2023-10-05T19:28:31.743 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.654+0000 7fe506cdb700 1 -- 172.21.15.167:0/3045448955 <== osd.0 v2:172.21.15.118:6800/27690 3 ==== osd_op_reply(12 [pgnls start_epoch 30 out=112b] v23'1 uv1 ondisk = 1) v8 ==== 144+0+112 (crc 0 0 0) 0x7fe4f800a5a0 con 0x7fe4f0007030 2023-10-05T19:28:31.744 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.654+0000 7fe506cdb700 1 -- 172.21.15.167:0/3045448955 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:13 5.1c 5:38000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x565386ea9a60 con 0x7fe500016730 2023-10-05T19:28:31.744 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.654+0000 7fe505cd9700 1 -- 172.21.15.167:0/3045448955 <== osd.2 v2:172.21.15.167:6816/30037 2 ==== osd_op_reply(13 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fe4f0029040 con 0x7fe500016730 2023-10-05T19:28:31.744 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.654+0000 7fe505cd9700 1 -- 172.21.15.167:0/3045448955 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:14 5.2 5:40000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x565386ea9a60 con 0x565386ea7140 2023-10-05T19:28:31.744 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.654+0000 7fe5064da700 1 -- 172.21.15.167:0/3045448955 <== osd.3 v2:172.21.15.167:6808/30036 2 ==== osd_op_reply(14 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fe500014070 con 0x565386ea7140 2023-10-05T19:28:31.744 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.654+0000 7fe5064da700 1 -- 172.21.15.167:0/3045448955 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:15 5.12 5:48000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x565386ea9a60 con 0x565386e76600 2023-10-05T19:28:31.744 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.658+0000 7fe506cdb700 1 -- 172.21.15.167:0/3045448955 <== osd.1 v2:172.21.15.167:6800/30035 8 ==== osd_op_reply(15 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fe4f800a5a0 con 0x565386e76600 2023-10-05T19:28:31.744 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.658+0000 7fe506cdb700 1 -- 172.21.15.167:0/3045448955 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:16 5.a 5:50000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x565386ea9a60 con 0x7fe4f0007030 2023-10-05T19:28:31.744 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.658+0000 7fe506cdb700 1 -- 172.21.15.167:0/3045448955 <== osd.0 v2:172.21.15.118:6800/27690 4 ==== osd_op_reply(16 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fe4f800a5a0 con 0x7fe4f0007030 2023-10-05T19:28:31.744 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.658+0000 7fe506cdb700 1 -- 172.21.15.167:0/3045448955 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:17 5.1a 5:58000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x565386ea9a60 con 0x565386e76600 2023-10-05T19:28:31.744 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.658+0000 7fe506cdb700 1 -- 172.21.15.167:0/3045448955 <== osd.1 v2:172.21.15.167:6800/30035 9 ==== osd_op_reply(17 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fe4f800a5a0 con 0x565386e76600 2023-10-05T19:28:31.744 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.658+0000 7fe506cdb700 1 -- 172.21.15.167:0/3045448955 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:18 5.6 5:60000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x565386ea9a60 con 0x7fe500016730 2023-10-05T19:28:31.745 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.658+0000 7fe505cd9700 1 -- 172.21.15.167:0/3045448955 <== osd.2 v2:172.21.15.167:6816/30037 3 ==== osd_op_reply(18 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fe4f0029040 con 0x7fe500016730 2023-10-05T19:28:31.745 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.658+0000 7fe505cd9700 1 -- 172.21.15.167:0/3045448955 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:19 5.16 5:68000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x565386ea9a60 con 0x565386ea7140 2023-10-05T19:28:31.745 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.658+0000 7fe5064da700 1 -- 172.21.15.167:0/3045448955 <== osd.3 v2:172.21.15.167:6808/30036 3 ==== osd_op_reply(19 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fe500014070 con 0x565386ea7140 2023-10-05T19:28:31.745 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.658+0000 7fe5064da700 1 -- 172.21.15.167:0/3045448955 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:20 5.e 5:70000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x565386ea9a60 con 0x565386ea7140 2023-10-05T19:28:31.745 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.662+0000 7fe5064da700 1 -- 172.21.15.167:0/3045448955 <== osd.3 v2:172.21.15.167:6808/30036 4 ==== osd_op_reply(20 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fe500014070 con 0x565386ea7140 2023-10-05T19:28:31.745 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.662+0000 7fe5064da700 1 -- 172.21.15.167:0/3045448955 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:21 5.1e 5:78000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x565386ea9a60 con 0x7fe4f0007030 2023-10-05T19:28:31.745 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.662+0000 7fe506cdb700 1 -- 172.21.15.167:0/3045448955 <== osd.0 v2:172.21.15.118:6800/27690 5 ==== osd_op_reply(21 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fe4f800a5a0 con 0x7fe4f0007030 2023-10-05T19:28:31.745 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.662+0000 7fe506cdb700 1 -- 172.21.15.167:0/3045448955 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:22 5.1 5:80000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x565386ea9a60 con 0x565386ea7140 2023-10-05T19:28:31.745 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.662+0000 7fe5064da700 1 -- 172.21.15.167:0/3045448955 <== osd.3 v2:172.21.15.167:6808/30036 5 ==== osd_op_reply(22 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fe500014070 con 0x565386ea7140 2023-10-05T19:28:31.745 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.662+0000 7fe5064da700 1 -- 172.21.15.167:0/3045448955 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:23 5.11 5:88000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x565386ea9a60 con 0x565386ea7140 2023-10-05T19:28:31.745 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.662+0000 7fe5064da700 1 -- 172.21.15.167:0/3045448955 <== osd.3 v2:172.21.15.167:6808/30036 6 ==== osd_op_reply(23 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fe500014070 con 0x565386ea7140 2023-10-05T19:28:31.745 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.662+0000 7fe5064da700 1 -- 172.21.15.167:0/3045448955 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:24 5.9 5:90000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x565386ea9a60 con 0x565386e76600 2023-10-05T19:28:31.746 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.662+0000 7fe506cdb700 1 -- 172.21.15.167:0/3045448955 <== osd.1 v2:172.21.15.167:6800/30035 10 ==== osd_op_reply(24 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fe4f800a5a0 con 0x565386e76600 2023-10-05T19:28:31.746 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.662+0000 7fe506cdb700 1 -- 172.21.15.167:0/3045448955 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:25 5.19 5:98000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x565386ea9a60 con 0x565386e76600 2023-10-05T19:28:31.746 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.666+0000 7fe506cdb700 1 -- 172.21.15.167:0/3045448955 <== osd.1 v2:172.21.15.167:6800/30035 11 ==== osd_op_reply(25 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fe4f800a5a0 con 0x565386e76600 2023-10-05T19:28:31.746 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.666+0000 7fe506cdb700 1 -- 172.21.15.167:0/3045448955 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:26 5.5 5:a0000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x565386ea9a60 con 0x7fe4f0007030 2023-10-05T19:28:31.746 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.666+0000 7fe506cdb700 1 -- 172.21.15.167:0/3045448955 <== osd.0 v2:172.21.15.118:6800/27690 6 ==== osd_op_reply(26 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fe4f800a5a0 con 0x7fe4f0007030 2023-10-05T19:28:31.746 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.666+0000 7fe506cdb700 1 -- 172.21.15.167:0/3045448955 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:27 5.15 5:a8000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x565386ea9a60 con 0x565386ea7140 2023-10-05T19:28:31.746 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.666+0000 7fe5064da700 1 -- 172.21.15.167:0/3045448955 <== osd.3 v2:172.21.15.167:6808/30036 7 ==== osd_op_reply(27 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fe500014070 con 0x565386ea7140 2023-10-05T19:28:31.746 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.666+0000 7fe5064da700 1 -- 172.21.15.167:0/3045448955 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:28 5.d 5:b0000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x565386ea9a60 con 0x7fe500016730 2023-10-05T19:28:31.746 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.666+0000 7fe505cd9700 1 -- 172.21.15.167:0/3045448955 <== osd.2 v2:172.21.15.167:6816/30037 4 ==== osd_op_reply(28 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fe4f0029040 con 0x7fe500016730 2023-10-05T19:28:31.746 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.666+0000 7fe505cd9700 1 -- 172.21.15.167:0/3045448955 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:29 5.1d 5:b8000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x565386ea9a60 con 0x565386e76600 2023-10-05T19:28:31.747 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.670+0000 7fe506cdb700 1 -- 172.21.15.167:0/3045448955 <== osd.1 v2:172.21.15.167:6800/30035 12 ==== osd_op_reply(29 [pgnls start_epoch 30 out=107b] v30'2 uv2 ondisk = 1) v8 ==== 144+0+107 (crc 0 0 0) 0x7fe4f800a5a0 con 0x565386e76600 2023-10-05T19:28:31.747 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.670+0000 7fe506cdb700 1 -- 172.21.15.167:0/3045448955 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:30 5.3 5:c0000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x565386ea9a60 con 0x7fe4f0007030 2023-10-05T19:28:31.747 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.670+0000 7fe506cdb700 1 -- 172.21.15.167:0/3045448955 <== osd.0 v2:172.21.15.118:6800/27690 7 ==== osd_op_reply(30 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fe4f800a5a0 con 0x7fe4f0007030 2023-10-05T19:28:31.747 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.670+0000 7fe506cdb700 1 -- 172.21.15.167:0/3045448955 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:31 5.13 5:c8000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x565386ea9a60 con 0x565386ea7140 2023-10-05T19:28:31.747 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.670+0000 7fe5064da700 1 -- 172.21.15.167:0/3045448955 <== osd.3 v2:172.21.15.167:6808/30036 8 ==== osd_op_reply(31 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fe500014070 con 0x565386ea7140 2023-10-05T19:28:31.747 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.670+0000 7fe5064da700 1 -- 172.21.15.167:0/3045448955 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:32 5.b 5:d0000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x565386ea9a60 con 0x7fe4f0007030 2023-10-05T19:28:31.747 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.670+0000 7fe506cdb700 1 -- 172.21.15.167:0/3045448955 <== osd.0 v2:172.21.15.118:6800/27690 8 ==== osd_op_reply(32 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fe4f800a5a0 con 0x7fe4f0007030 2023-10-05T19:28:31.747 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.670+0000 7fe506cdb700 1 -- 172.21.15.167:0/3045448955 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:33 5.1b 5:d8000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x565386ea9a60 con 0x7fe500016730 2023-10-05T19:28:31.747 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.674+0000 7fe505cd9700 1 -- 172.21.15.167:0/3045448955 <== osd.2 v2:172.21.15.167:6816/30037 5 ==== osd_op_reply(33 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fe4f0029040 con 0x7fe500016730 2023-10-05T19:28:31.747 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.674+0000 7fe505cd9700 1 -- 172.21.15.167:0/3045448955 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:34 5.7 5:e0000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x565386ea9a60 con 0x565386ea7140 2023-10-05T19:28:31.747 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.674+0000 7fe5064da700 1 -- 172.21.15.167:0/3045448955 <== osd.3 v2:172.21.15.167:6808/30036 9 ==== osd_op_reply(34 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fe500014070 con 0x565386ea7140 2023-10-05T19:28:31.747 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.674+0000 7fe5064da700 1 -- 172.21.15.167:0/3045448955 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:35 5.17 5:e8000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x565386ea9a60 con 0x7fe4f0007030 2023-10-05T19:28:31.748 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.674+0000 7fe506cdb700 1 -- 172.21.15.167:0/3045448955 <== osd.0 v2:172.21.15.118:6800/27690 9 ==== osd_op_reply(35 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fe4f800a5a0 con 0x7fe4f0007030 2023-10-05T19:28:31.748 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.674+0000 7fe506cdb700 1 -- 172.21.15.167:0/3045448955 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:36 5.f 5:f0000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x565386ea9a60 con 0x565386e76600 2023-10-05T19:28:31.748 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.674+0000 7fe506cdb700 1 -- 172.21.15.167:0/3045448955 <== osd.1 v2:172.21.15.167:6800/30035 13 ==== osd_op_reply(36 [pgnls start_epoch 30 out=115b] v23'2 uv2 ondisk = 1) v8 ==== 144+0+115 (crc 0 0 0) 0x7fe4f800a5a0 con 0x565386e76600 2023-10-05T19:28:31.748 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.674+0000 7fe506cdb700 1 -- 172.21.15.167:0/3045448955 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:37 5.1f 5:f8000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x565386ea9a60 con 0x7fe500016730 2023-10-05T19:28:31.748 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.674+0000 7fe505cd9700 1 -- 172.21.15.167:0/3045448955 <== osd.2 v2:172.21.15.167:6816/30037 6 ==== osd_op_reply(37 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fe4f0029040 con 0x7fe500016730 2023-10-05T19:28:31.748 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.674+0000 7fe5089a5e40 20 RGWRados::pool_iterate: got zonegroup_info.4f1f719d-9648-417b-b10c-e5d939825419 2023-10-05T19:28:31.748 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.674+0000 7fe5089a5e40 20 RGWRados::pool_iterate: got zone_info.9a9df965-6612-45d4-a45f-6e3c889bacc2 2023-10-05T19:28:31.748 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.674+0000 7fe5089a5e40 20 RGWRados::pool_iterate: got zone_names.default 2023-10-05T19:28:31.748 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.674+0000 7fe5089a5e40 20 RGWRados::pool_iterate: got zonegroups_names.default 2023-10-05T19:28:31.748 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.678+0000 7fe5089a5e40 20 rados->read ofs=0 len=0 2023-10-05T19:28:31.748 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.678+0000 7fe5089a5e40 1 -- 172.21.15.167:0/3045448955 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:38 5.12 5:49953fa1:::default.realm:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x565386ea5ed0 con 0x565386e76600 2023-10-05T19:28:31.748 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.678+0000 7fe506cdb700 1 -- 172.21.15.167:0/3045448955 <== osd.1 v2:172.21.15.167:6800/30035 14 ==== osd_op_reply(38 default.realm [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 ==== 157+0+0 (crc 0 0 0) 0x7fe4f800a5a0 con 0x565386e76600 2023-10-05T19:28:31.749 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.678+0000 7fe5089a5e40 20 rados_obj.operate() r=-2 bl.length=0 2023-10-05T19:28:31.749 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.678+0000 7fe5089a5e40 20 rados->read ofs=0 len=0 2023-10-05T19:28:31.749 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.678+0000 7fe5089a5e40 1 -- 172.21.15.167:0/3045448955 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:39 5.f 5:f4c53578:::zonegroups_names.default:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x565386ea6850 con 0x565386e76600 2023-10-05T19:28:31.749 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.678+0000 7fe506cdb700 1 -- 172.21.15.167:0/3045448955 <== osd.1 v2:172.21.15.167:6800/30035 15 ==== osd_op_reply(39 zonegroups_names.default [read 0~46 out=46b] v0'0 uv2 ondisk = 0) v8 ==== 168+0+46 (crc 0 0 0) 0x7fe4f800a5a0 con 0x565386e76600 2023-10-05T19:28:31.749 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.678+0000 7fe5089a5e40 20 rados_obj.operate() r=0 bl.length=46 2023-10-05T19:28:31.749 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.678+0000 7fe5089a5e40 20 rados->read ofs=0 len=0 2023-10-05T19:28:31.749 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.678+0000 7fe5089a5e40 1 -- 172.21.15.167:0/3045448955 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:40 5.c 5:30d299a5:::zonegroup_info.4f1f719d-9648-417b-b10c-e5d939825419:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x565386ea6860 con 0x7fe4f0007030 2023-10-05T19:28:31.749 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.678+0000 7fe506cdb700 1 -- 172.21.15.167:0/3045448955 <== osd.0 v2:172.21.15.118:6800/27690 10 ==== osd_op_reply(40 zonegroup_info.4f1f719d-9648-417b-b10c-e5d939825419 [read 0~362 out=362b] v0'0 uv1 ondisk = 0) v8 ==== 195+0+362 (crc 0 0 0) 0x7fe4f800a5a0 con 0x7fe4f0007030 2023-10-05T19:28:31.749 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.678+0000 7fe5089a5e40 20 rados_obj.operate() r=0 bl.length=362 2023-10-05T19:28:31.749 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.678+0000 7fe5089a5e40 20 rados->read ofs=0 len=0 2023-10-05T19:28:31.749 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.678+0000 7fe5089a5e40 1 -- 172.21.15.167:0/3045448955 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:41 5.13 5:cbbf6c60:::converted:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x565386ea6860 con 0x565386ea7140 2023-10-05T19:28:31.750 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.678+0000 7fe5064da700 1 -- 172.21.15.167:0/3045448955 <== osd.3 v2:172.21.15.167:6808/30036 10 ==== osd_op_reply(41 converted [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 ==== 153+0+0 (crc 0 0 0) 0x7fe500014070 con 0x565386ea7140 2023-10-05T19:28:31.750 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.678+0000 7fe5089a5e40 20 rados_obj.operate() r=-2 bl.length=0 2023-10-05T19:28:31.750 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.678+0000 7fe5089a5e40 20 rados->read ofs=0 len=0 2023-10-05T19:28:31.750 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.678+0000 7fe5089a5e40 1 -- 172.21.15.167:0/3045448955 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:42 5.12 5:49953fa1:::default.realm:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x565386ea9e10 con 0x565386e76600 2023-10-05T19:28:31.750 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.678+0000 7fe506cdb700 1 -- 172.21.15.167:0/3045448955 <== osd.1 v2:172.21.15.167:6800/30035 16 ==== osd_op_reply(42 default.realm [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 ==== 157+0+0 (crc 0 0 0) 0x7fe4f800a5a0 con 0x565386e76600 2023-10-05T19:28:31.750 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.678+0000 7fe5089a5e40 20 rados_obj.operate() r=-2 bl.length=0 2023-10-05T19:28:31.750 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.678+0000 7fe5089a5e40 20 rados->read ofs=0 len=0 2023-10-05T19:28:31.750 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.678+0000 7fe5089a5e40 1 -- 172.21.15.167:0/3045448955 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:43 5.f 5:f4c53578:::zonegroups_names.default:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x565386ea9e10 con 0x565386e76600 2023-10-05T19:28:31.750 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.682+0000 7fe506cdb700 1 -- 172.21.15.167:0/3045448955 <== osd.1 v2:172.21.15.167:6800/30035 17 ==== osd_op_reply(43 zonegroups_names.default [read 0~46 out=46b] v0'0 uv2 ondisk = 0) v8 ==== 168+0+46 (crc 0 0 0) 0x7fe4f800a5a0 con 0x565386e76600 2023-10-05T19:28:31.750 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.682+0000 7fe5089a5e40 20 rados_obj.operate() r=0 bl.length=46 2023-10-05T19:28:31.750 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.682+0000 7fe5089a5e40 1 -- 172.21.15.167:0/3045448955 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:44 5.0 5:00000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x565386ea9e10 con 0x565386ea7140 2023-10-05T19:28:31.751 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.682+0000 7fe5064da700 1 -- 172.21.15.167:0/3045448955 <== osd.3 v2:172.21.15.167:6808/30036 11 ==== osd_op_reply(44 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fe500014070 con 0x565386ea7140 2023-10-05T19:28:31.751 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.682+0000 7fe5064da700 1 -- 172.21.15.167:0/3045448955 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:45 5.10 5:08000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x565386ea9e10 con 0x7fe500016730 2023-10-05T19:28:31.751 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.682+0000 7fe505cd9700 1 -- 172.21.15.167:0/3045448955 <== osd.2 v2:172.21.15.167:6816/30037 7 ==== osd_op_reply(45 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fe4f0029040 con 0x7fe500016730 2023-10-05T19:28:31.751 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.682+0000 7fe505cd9700 1 -- 172.21.15.167:0/3045448955 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:46 5.8 5:10000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x565386ea9e10 con 0x7fe4f0007030 2023-10-05T19:28:31.751 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.686+0000 7fe506cdb700 1 -- 172.21.15.167:0/3045448955 <== osd.0 v2:172.21.15.118:6800/27690 11 ==== osd_op_reply(46 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fe4f800a5a0 con 0x7fe4f0007030 2023-10-05T19:28:31.751 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.686+0000 7fe506cdb700 1 -- 172.21.15.167:0/3045448955 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:47 5.18 5:18000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x565386ea5ed0 con 0x565386e76600 2023-10-05T19:28:31.751 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.686+0000 7fe506cdb700 1 -- 172.21.15.167:0/3045448955 <== osd.1 v2:172.21.15.167:6800/30035 18 ==== osd_op_reply(47 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fe4f800a5a0 con 0x565386e76600 2023-10-05T19:28:31.751 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.686+0000 7fe506cdb700 1 -- 172.21.15.167:0/3045448955 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:48 5.4 5:20000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x565386ea5ed0 con 0x565386e76600 2023-10-05T19:28:31.751 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.686+0000 7fe506cdb700 1 -- 172.21.15.167:0/3045448955 <== osd.1 v2:172.21.15.167:6800/30035 19 ==== osd_op_reply(48 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fe4f800a5a0 con 0x565386e76600 2023-10-05T19:28:31.751 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.686+0000 7fe506cdb700 1 -- 172.21.15.167:0/3045448955 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:49 5.14 5:28000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x565386ea5ed0 con 0x7fe4f0007030 2023-10-05T19:28:31.751 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.686+0000 7fe506cdb700 1 -- 172.21.15.167:0/3045448955 <== osd.0 v2:172.21.15.118:6800/27690 12 ==== osd_op_reply(49 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fe4f800a5a0 con 0x7fe4f0007030 2023-10-05T19:28:31.751 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.686+0000 7fe506cdb700 1 -- 172.21.15.167:0/3045448955 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:50 5.c 5:30000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x565386ea5ed0 con 0x7fe4f0007030 2023-10-05T19:28:31.752 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.690+0000 7fe506cdb700 1 -- 172.21.15.167:0/3045448955 <== osd.0 v2:172.21.15.118:6800/27690 13 ==== osd_op_reply(50 [pgnls start_epoch 30 out=112b] v23'1 uv1 ondisk = 1) v8 ==== 144+0+112 (crc 0 0 0) 0x7fe4f800a5a0 con 0x7fe4f0007030 2023-10-05T19:28:31.752 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.690+0000 7fe506cdb700 1 -- 172.21.15.167:0/3045448955 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:51 5.1c 5:38000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x565386ea5ed0 con 0x7fe500016730 2023-10-05T19:28:31.752 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.690+0000 7fe505cd9700 1 -- 172.21.15.167:0/3045448955 <== osd.2 v2:172.21.15.167:6816/30037 8 ==== osd_op_reply(51 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fe4f0029040 con 0x7fe500016730 2023-10-05T19:28:31.752 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.690+0000 7fe505cd9700 1 -- 172.21.15.167:0/3045448955 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:52 5.2 5:40000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x565386ea5ed0 con 0x565386ea7140 2023-10-05T19:28:31.752 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.690+0000 7fe5064da700 1 -- 172.21.15.167:0/3045448955 <== osd.3 v2:172.21.15.167:6808/30036 12 ==== osd_op_reply(52 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fe500014070 con 0x565386ea7140 2023-10-05T19:28:31.752 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.690+0000 7fe5064da700 1 -- 172.21.15.167:0/3045448955 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:53 5.12 5:48000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x565386ea5ed0 con 0x565386e76600 2023-10-05T19:28:31.752 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.690+0000 7fe506cdb700 1 -- 172.21.15.167:0/3045448955 <== osd.1 v2:172.21.15.167:6800/30035 20 ==== osd_op_reply(53 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fe4f800a5a0 con 0x565386e76600 2023-10-05T19:28:31.752 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.690+0000 7fe506cdb700 1 -- 172.21.15.167:0/3045448955 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:54 5.a 5:50000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x565386ea5ed0 con 0x7fe4f0007030 2023-10-05T19:28:31.752 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.690+0000 7fe506cdb700 1 -- 172.21.15.167:0/3045448955 <== osd.0 v2:172.21.15.118:6800/27690 14 ==== osd_op_reply(54 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fe4f800a5a0 con 0x7fe4f0007030 2023-10-05T19:28:31.752 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.690+0000 7fe506cdb700 1 -- 172.21.15.167:0/3045448955 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:55 5.1a 5:58000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x565386ea5ed0 con 0x565386e76600 2023-10-05T19:28:31.752 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.694+0000 7fe506cdb700 1 -- 172.21.15.167:0/3045448955 <== osd.1 v2:172.21.15.167:6800/30035 21 ==== osd_op_reply(55 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fe4f800a5a0 con 0x565386e76600 2023-10-05T19:28:31.753 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.694+0000 7fe506cdb700 1 -- 172.21.15.167:0/3045448955 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:56 5.6 5:60000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x565386ea5ed0 con 0x7fe500016730 2023-10-05T19:28:31.753 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.694+0000 7fe505cd9700 1 -- 172.21.15.167:0/3045448955 <== osd.2 v2:172.21.15.167:6816/30037 9 ==== osd_op_reply(56 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fe4f0029040 con 0x7fe500016730 2023-10-05T19:28:31.753 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.694+0000 7fe505cd9700 1 -- 172.21.15.167:0/3045448955 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:57 5.16 5:68000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x565386ea5ed0 con 0x565386ea7140 2023-10-05T19:28:31.753 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.694+0000 7fe5064da700 1 -- 172.21.15.167:0/3045448955 <== osd.3 v2:172.21.15.167:6808/30036 13 ==== osd_op_reply(57 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fe500014070 con 0x565386ea7140 2023-10-05T19:28:31.753 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.694+0000 7fe5064da700 1 -- 172.21.15.167:0/3045448955 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:58 5.e 5:70000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x565386ea5ed0 con 0x565386ea7140 2023-10-05T19:28:31.753 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.694+0000 7fe5064da700 1 -- 172.21.15.167:0/3045448955 <== osd.3 v2:172.21.15.167:6808/30036 14 ==== osd_op_reply(58 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fe50004d040 con 0x565386ea7140 2023-10-05T19:28:31.753 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.694+0000 7fe5064da700 1 -- 172.21.15.167:0/3045448955 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:59 5.1e 5:78000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x565386ea5ed0 con 0x7fe4f0007030 2023-10-05T19:28:31.753 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.698+0000 7fe506cdb700 1 -- 172.21.15.167:0/3045448955 <== osd.0 v2:172.21.15.118:6800/27690 15 ==== osd_op_reply(59 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fe4f803f040 con 0x7fe4f0007030 2023-10-05T19:28:31.753 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.698+0000 7fe506cdb700 1 -- 172.21.15.167:0/3045448955 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:60 5.1 5:80000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x565386ea5ed0 con 0x565386ea7140 2023-10-05T19:28:31.753 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.698+0000 7fe5064da700 1 -- 172.21.15.167:0/3045448955 <== osd.3 v2:172.21.15.167:6808/30036 15 ==== osd_op_reply(60 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fe50004d040 con 0x565386ea7140 2023-10-05T19:28:31.753 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.698+0000 7fe5064da700 1 -- 172.21.15.167:0/3045448955 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:61 5.11 5:88000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x565386ea5ed0 con 0x565386ea7140 2023-10-05T19:28:31.753 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.698+0000 7fe5064da700 1 -- 172.21.15.167:0/3045448955 <== osd.3 v2:172.21.15.167:6808/30036 16 ==== osd_op_reply(61 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fe50004d040 con 0x565386ea7140 2023-10-05T19:28:31.754 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.698+0000 7fe5064da700 1 -- 172.21.15.167:0/3045448955 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:62 5.9 5:90000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x565386ea5ed0 con 0x565386e76600 2023-10-05T19:28:31.754 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.698+0000 7fe506cdb700 1 -- 172.21.15.167:0/3045448955 <== osd.1 v2:172.21.15.167:6800/30035 22 ==== osd_op_reply(62 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fe4f803f040 con 0x565386e76600 2023-10-05T19:28:31.754 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.698+0000 7fe506cdb700 1 -- 172.21.15.167:0/3045448955 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:63 5.19 5:98000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x565386ea5ed0 con 0x565386e76600 2023-10-05T19:28:31.754 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.698+0000 7fe506cdb700 1 -- 172.21.15.167:0/3045448955 <== osd.1 v2:172.21.15.167:6800/30035 23 ==== osd_op_reply(63 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fe4f803f040 con 0x565386e76600 2023-10-05T19:28:31.754 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.698+0000 7fe506cdb700 1 -- 172.21.15.167:0/3045448955 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:64 5.5 5:a0000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x565386ea5ed0 con 0x7fe4f0007030 2023-10-05T19:28:31.754 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.702+0000 7fe506cdb700 1 -- 172.21.15.167:0/3045448955 <== osd.0 v2:172.21.15.118:6800/27690 16 ==== osd_op_reply(64 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fe4f803f040 con 0x7fe4f0007030 2023-10-05T19:28:31.754 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.702+0000 7fe506cdb700 1 -- 172.21.15.167:0/3045448955 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:65 5.15 5:a8000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x565386ea5ed0 con 0x565386ea7140 2023-10-05T19:28:31.754 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.702+0000 7fe5064da700 1 -- 172.21.15.167:0/3045448955 <== osd.3 v2:172.21.15.167:6808/30036 17 ==== osd_op_reply(65 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fe50004d040 con 0x565386ea7140 2023-10-05T19:28:31.754 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.702+0000 7fe5064da700 1 -- 172.21.15.167:0/3045448955 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:66 5.d 5:b0000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x565386ea5ed0 con 0x7fe500016730 2023-10-05T19:28:31.754 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.702+0000 7fe505cd9700 1 -- 172.21.15.167:0/3045448955 <== osd.2 v2:172.21.15.167:6816/30037 10 ==== osd_op_reply(66 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fe4f0029040 con 0x7fe500016730 2023-10-05T19:28:31.754 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.702+0000 7fe505cd9700 1 -- 172.21.15.167:0/3045448955 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:67 5.1d 5:b8000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x565386ea5ed0 con 0x565386e76600 2023-10-05T19:28:31.755 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.702+0000 7fe506cdb700 1 -- 172.21.15.167:0/3045448955 <== osd.1 v2:172.21.15.167:6800/30035 24 ==== osd_op_reply(67 [pgnls start_epoch 30 out=107b] v30'2 uv2 ondisk = 1) v8 ==== 144+0+107 (crc 0 0 0) 0x7fe4f803f040 con 0x565386e76600 2023-10-05T19:28:31.755 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.702+0000 7fe506cdb700 1 -- 172.21.15.167:0/3045448955 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:68 5.3 5:c0000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x565386ea5ed0 con 0x7fe4f0007030 2023-10-05T19:28:31.755 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.706+0000 7fe506cdb700 1 -- 172.21.15.167:0/3045448955 <== osd.0 v2:172.21.15.118:6800/27690 17 ==== osd_op_reply(68 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fe4f803f040 con 0x7fe4f0007030 2023-10-05T19:28:31.755 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.706+0000 7fe506cdb700 1 -- 172.21.15.167:0/3045448955 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:69 5.13 5:c8000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x565386ea5ed0 con 0x565386ea7140 2023-10-05T19:28:31.755 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.706+0000 7fe5064da700 1 -- 172.21.15.167:0/3045448955 <== osd.3 v2:172.21.15.167:6808/30036 18 ==== osd_op_reply(69 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fe50004d040 con 0x565386ea7140 2023-10-05T19:28:31.755 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.706+0000 7fe5064da700 1 -- 172.21.15.167:0/3045448955 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:70 5.b 5:d0000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x7fe500051040 con 0x7fe4f0007030 2023-10-05T19:28:31.755 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.706+0000 7fe506cdb700 1 -- 172.21.15.167:0/3045448955 <== osd.0 v2:172.21.15.118:6800/27690 18 ==== osd_op_reply(70 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fe4f803f040 con 0x7fe4f0007030 2023-10-05T19:28:31.755 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.706+0000 7fe506cdb700 1 -- 172.21.15.167:0/3045448955 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:71 5.1b 5:d8000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x7fe500051040 con 0x7fe500016730 2023-10-05T19:28:31.755 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.706+0000 7fe505cd9700 1 -- 172.21.15.167:0/3045448955 <== osd.2 v2:172.21.15.167:6816/30037 11 ==== osd_op_reply(71 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fe4f0029040 con 0x7fe500016730 2023-10-05T19:28:31.755 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.706+0000 7fe505cd9700 1 -- 172.21.15.167:0/3045448955 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:72 5.7 5:e0000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x7fe500051040 con 0x565386ea7140 2023-10-05T19:28:31.755 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.706+0000 7fe5064da700 1 -- 172.21.15.167:0/3045448955 <== osd.3 v2:172.21.15.167:6808/30036 19 ==== osd_op_reply(72 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fe50004d040 con 0x565386ea7140 2023-10-05T19:28:31.756 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.706+0000 7fe5064da700 1 -- 172.21.15.167:0/3045448955 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:73 5.17 5:e8000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x7fe500051040 con 0x7fe4f0007030 2023-10-05T19:28:31.756 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.710+0000 7fe506cdb700 1 -- 172.21.15.167:0/3045448955 <== osd.0 v2:172.21.15.118:6800/27690 19 ==== osd_op_reply(73 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fe4f803f040 con 0x7fe4f0007030 2023-10-05T19:28:31.756 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.710+0000 7fe506cdb700 1 -- 172.21.15.167:0/3045448955 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:74 5.f 5:f0000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x7fe500051040 con 0x565386e76600 2023-10-05T19:28:31.756 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.710+0000 7fe506cdb700 1 -- 172.21.15.167:0/3045448955 <== osd.1 v2:172.21.15.167:6800/30035 25 ==== osd_op_reply(74 [pgnls start_epoch 30 out=115b] v23'2 uv2 ondisk = 1) v8 ==== 144+0+115 (crc 0 0 0) 0x7fe4f803f040 con 0x565386e76600 2023-10-05T19:28:31.756 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.710+0000 7fe506cdb700 1 -- 172.21.15.167:0/3045448955 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:75 5.1f 5:f8000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x7fe500051040 con 0x7fe500016730 2023-10-05T19:28:31.756 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.710+0000 7fe505cd9700 1 -- 172.21.15.167:0/3045448955 <== osd.2 v2:172.21.15.167:6816/30037 12 ==== osd_op_reply(75 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fe4f0029040 con 0x7fe500016730 2023-10-05T19:28:31.756 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.710+0000 7fe5089a5e40 20 RGWRados::pool_iterate: got zonegroup_info.4f1f719d-9648-417b-b10c-e5d939825419 2023-10-05T19:28:31.756 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.710+0000 7fe5089a5e40 20 RGWRados::pool_iterate: got zone_info.9a9df965-6612-45d4-a45f-6e3c889bacc2 2023-10-05T19:28:31.756 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.710+0000 7fe5089a5e40 20 RGWRados::pool_iterate: got zone_names.default 2023-10-05T19:28:31.756 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.710+0000 7fe5089a5e40 20 RGWRados::pool_iterate: got zonegroups_names.default 2023-10-05T19:28:31.756 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.710+0000 7fe5089a5e40 20 rados->read ofs=0 len=0 2023-10-05T19:28:31.756 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.710+0000 7fe5089a5e40 1 -- 172.21.15.167:0/3045448955 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:76 5.f 5:f43b6ece:::zone_names.default:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x565386ea9e10 con 0x565386e76600 2023-10-05T19:28:31.757 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.710+0000 7fe506cdb700 1 -- 172.21.15.167:0/3045448955 <== osd.1 v2:172.21.15.167:6800/30035 26 ==== osd_op_reply(76 zone_names.default [read 0~46 out=46b] v0'0 uv1 ondisk = 0) v8 ==== 162+0+46 (crc 0 0 0) 0x7fe4f803f040 con 0x565386e76600 2023-10-05T19:28:31.757 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.710+0000 7fe5089a5e40 20 rados_obj.operate() r=0 bl.length=46 2023-10-05T19:28:31.757 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.710+0000 7fe5089a5e40 20 rados->read ofs=0 len=0 2023-10-05T19:28:31.757 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.710+0000 7fe5089a5e40 1 -- 172.21.15.167:0/3045448955 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:77 5.1d 5:b8374eb4:::zone_info.9a9df965-6612-45d4-a45f-6e3c889bacc2:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x565386eaa210 con 0x565386e76600 2023-10-05T19:28:31.757 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.710+0000 7fe506cdb700 1 -- 172.21.15.167:0/3045448955 <== osd.1 v2:172.21.15.167:6800/30035 27 ==== osd_op_reply(77 zone_info.9a9df965-6612-45d4-a45f-6e3c889bacc2 [read 0~874 out=874b] v0'0 uv2 ondisk = 0) v8 ==== 190+0+874 (crc 0 0 0) 0x7fe4f803f040 con 0x565386e76600 2023-10-05T19:28:31.757 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.710+0000 7fe5089a5e40 20 rados_obj.operate() r=0 bl.length=874 2023-10-05T19:28:31.757 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.714+0000 7fe5089a5e40 20 rados->read ofs=0 len=0 2023-10-05T19:28:31.757 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.714+0000 7fe5089a5e40 1 -- 172.21.15.167:0/3045448955 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:78 5.f 5:f4c53578:::zonegroups_names.default:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x565386eaa650 con 0x565386e76600 2023-10-05T19:28:31.757 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.714+0000 7fe506cdb700 1 -- 172.21.15.167:0/3045448955 <== osd.1 v2:172.21.15.167:6800/30035 28 ==== osd_op_reply(78 zonegroups_names.default [read 0~46 out=46b] v0'0 uv2 ondisk = 0) v8 ==== 168+0+46 (crc 0 0 0) 0x7fe4f803f040 con 0x565386e76600 2023-10-05T19:28:31.757 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.714+0000 7fe5089a5e40 20 rados_obj.operate() r=0 bl.length=46 2023-10-05T19:28:31.757 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.714+0000 7fe5089a5e40 20 rados->read ofs=0 len=0 2023-10-05T19:28:31.758 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.714+0000 7fe5089a5e40 1 -- 172.21.15.167:0/3045448955 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:79 5.c 5:30d299a5:::zonegroup_info.4f1f719d-9648-417b-b10c-e5d939825419:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x565386eaa650 con 0x7fe4f0007030 2023-10-05T19:28:31.758 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.714+0000 7fe506cdb700 1 -- 172.21.15.167:0/3045448955 <== osd.0 v2:172.21.15.118:6800/27690 20 ==== osd_op_reply(79 zonegroup_info.4f1f719d-9648-417b-b10c-e5d939825419 [read 0~362 out=362b] v0'0 uv1 ondisk = 0) v8 ==== 195+0+362 (crc 0 0 0) 0x7fe4f8052040 con 0x7fe4f0007030 2023-10-05T19:28:31.758 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.714+0000 7fe5089a5e40 20 rados_obj.operate() r=0 bl.length=362 2023-10-05T19:28:31.758 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.714+0000 7fe5089a5e40 20 rados->read ofs=0 len=0 2023-10-05T19:28:31.758 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.714+0000 7fe5089a5e40 1 -- 172.21.15.167:0/3045448955 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:80 5.3 5:c0ad8cbd:::region_map:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x565386eaaa80 con 0x7fe4f0007030 2023-10-05T19:28:31.758 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.714+0000 7fe506cdb700 1 -- 172.21.15.167:0/3045448955 <== osd.0 v2:172.21.15.118:6800/27690 21 ==== osd_op_reply(80 region_map [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 ==== 154+0+0 (crc 0 0 0) 0x7fe4f8052040 con 0x7fe4f0007030 2023-10-05T19:28:31.758 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.714+0000 7fe5089a5e40 20 rados_obj.operate() r=-2 bl.length=0 2023-10-05T19:28:31.758 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.714+0000 7fe5089a5e40 20 zone default found 2023-10-05T19:28:31.758 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.714+0000 7fe5089a5e40 4 Realm: () 2023-10-05T19:28:31.758 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.714+0000 7fe5089a5e40 4 ZoneGroup: default (4f1f719d-9648-417b-b10c-e5d939825419) 2023-10-05T19:28:31.758 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.714+0000 7fe5089a5e40 4 Zone: default (9a9df965-6612-45d4-a45f-6e3c889bacc2) 2023-10-05T19:28:31.759 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.714+0000 7fe5089a5e40 10 cannot find current period zonegroup using local zonegroup configuration 2023-10-05T19:28:31.759 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.714+0000 7fe5089a5e40 20 zonegroup default 2023-10-05T19:28:31.759 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.714+0000 7fe5089a5e40 20 rados->read ofs=0 len=0 2023-10-05T19:28:31.759 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.714+0000 7fe5089a5e40 1 -- 172.21.15.167:0/3045448955 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:81 5.3 5:c52100b6:::period_config.default:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x565386eaa7b0 con 0x7fe4f0007030 2023-10-05T19:28:31.759 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.714+0000 7fe506cdb700 1 -- 172.21.15.167:0/3045448955 <== osd.0 v2:172.21.15.118:6800/27690 22 ==== osd_op_reply(81 period_config.default [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 ==== 165+0+0 (crc 0 0 0) 0x7fe4f8052040 con 0x7fe4f0007030 2023-10-05T19:28:31.759 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.714+0000 7fe5089a5e40 20 rados_obj.operate() r=-2 bl.length=0 2023-10-05T19:28:31.759 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.718+0000 7fe5089a5e40 20 rados->read ofs=0 len=0 2023-10-05T19:28:31.759 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.718+0000 7fe5089a5e40 1 -- 172.21.15.167:0/3045448955 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:82 6.2 6:4347d321:::bucket.sync-source-hints.:head [call version.read in=11b,read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x565386eac130 con 0x565386ea7140 2023-10-05T19:28:31.759 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.718+0000 7fe5064da700 1 -- 172.21.15.167:0/3045448955 <== osd.3 v2:172.21.15.167:6808/30036 20 ==== osd_op_reply(82 bucket.sync-source-hints. [call,read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 ==== 211+0+0 (crc 0 0 0) 0x7fe50004d040 con 0x565386ea7140 2023-10-05T19:28:31.759 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.718+0000 7fe5089a5e40 20 rados_obj.operate() r=-2 bl.length=0 2023-10-05T19:28:31.759 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.718+0000 7fe5089a5e40 20 rados->read ofs=0 len=0 2023-10-05T19:28:31.759 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.718+0000 7fe5089a5e40 1 -- 172.21.15.167:0/3045448955 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:83 6.b 6:d467b91b:::bucket.sync-target-hints.:head [call version.read in=11b,read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x565386eac4e0 con 0x565386ea7140 2023-10-05T19:28:31.760 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.718+0000 7fe5064da700 1 -- 172.21.15.167:0/3045448955 <== osd.3 v2:172.21.15.167:6808/30036 21 ==== osd_op_reply(83 bucket.sync-target-hints. [call,read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 ==== 211+0+0 (crc 0 0 0) 0x7fe50004d040 con 0x565386ea7140 2023-10-05T19:28:31.760 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.718+0000 7fe5089a5e40 20 rados_obj.operate() r=-2 bl.length=0 2023-10-05T19:28:31.760 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.718+0000 7fe5089a5e40 20 started sync module instance, tier type = 2023-10-05T19:28:31.760 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.718+0000 7fe5089a5e40 20 started zone id=9a9df965-6612-45d4-a45f-6e3c889bacc2 (name=default) with tier type = 2023-10-05T19:28:31.760 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.718+0000 7fe5089a5e40 1 -- 172.21.15.167:0/3045448955 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:84 7.1f 7:f95f44c2:::notify.0:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x565386eab7f0 con 0x7fe500016730 2023-10-05T19:28:31.760 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.722+0000 7fe505cd9700 1 -- 172.21.15.167:0/3045448955 <== osd.2 v2:172.21.15.167:6816/30037 13 ==== osd_op_reply(84 notify.0 [create] v30'99 uv99 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7fe4f0029040 con 0x7fe500016730 2023-10-05T19:28:31.760 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.722+0000 7fe5089a5e40 1 -- 172.21.15.167:0/3045448955 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:85 7.1f 7:f95f44c2:::notify.0:head [watch watch cookie 94916745804784] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x565386ea63f0 con 0x7fe500016730 2023-10-05T19:28:31.760 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.722+0000 7fe5089a5e40 1 -- 172.21.15.167:0/3045448955 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:86 7.0 7:05bf5b68:::notify.1:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x565386ead2e0 con 0x565386e76600 2023-10-05T19:28:31.760 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.726+0000 7fe506cdb700 1 -- 172.21.15.167:0/3045448955 <== osd.1 v2:172.21.15.167:6800/30035 29 ==== osd_op_reply(86 notify.1 [create] v30'99 uv99 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7fe4f8052040 con 0x565386e76600 2023-10-05T19:28:31.760 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.726+0000 7fe5089a5e40 1 -- 172.21.15.167:0/3045448955 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:87 7.0 7:05bf5b68:::notify.1:head [watch watch cookie 94916745809328] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x565386eadbd0 con 0x565386e76600 2023-10-05T19:28:31.760 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.726+0000 7fe505cd9700 1 -- 172.21.15.167:0/3045448955 <== osd.2 v2:172.21.15.167:6816/30037 14 ==== osd_op_reply(85 notify.0 [watch watch cookie 94916745804784] v30'100 uv99 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7fe4f0029040 con 0x7fe500016730 2023-10-05T19:28:31.761 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.726+0000 7fe5089a5e40 1 -- 172.21.15.167:0/3045448955 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:88 7.15 7:a93a5511:::notify.2:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x565386eadf80 con 0x565386ea7140 2023-10-05T19:28:31.767 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.726+0000 7fe506cdb700 1 -- 172.21.15.167:0/3045448955 <== osd.1 v2:172.21.15.167:6800/30035 30 ==== osd_op_reply(87 notify.1 [watch watch cookie 94916745809328] v30'100 uv99 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7fe4f8052040 con 0x565386e76600 2023-10-05T19:28:31.767 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.730+0000 7fe5064da700 1 -- 172.21.15.167:0/3045448955 <== osd.3 v2:172.21.15.167:6808/30036 22 ==== osd_op_reply(88 notify.2 [create] v30'99 uv99 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7fe50004d040 con 0x565386ea7140 2023-10-05T19:28:31.767 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.730+0000 7fe5089a5e40 1 -- 172.21.15.167:0/3045448955 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:89 7.15 7:a93a5511:::notify.2:head [watch watch cookie 94916745811680] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x565386eac9b0 con 0x565386ea7140 2023-10-05T19:28:31.767 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.730+0000 7fe5089a5e40 1 -- 172.21.15.167:0/3045448955 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:90 7.e 7:7759931f:::notify.3:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x565386eaed10 con 0x7fe500016730 2023-10-05T19:28:31.767 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.730+0000 7fe505cd9700 1 -- 172.21.15.167:0/3045448955 <== osd.2 v2:172.21.15.167:6816/30037 15 ==== osd_op_reply(90 notify.3 [create] v30'99 uv99 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7fe4f0029040 con 0x7fe500016730 2023-10-05T19:28:31.767 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.730+0000 7fe5089a5e40 1 -- 172.21.15.167:0/3045448955 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:91 7.e 7:7759931f:::notify.3:head [watch watch cookie 94916745816096] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x565386eaf5b0 con 0x7fe500016730 2023-10-05T19:28:31.767 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.730+0000 7fe5089a5e40 1 -- 172.21.15.167:0/3045448955 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:92 7.d 7:b4812045:::notify.4:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x565386eb0090 con 0x565386e76600 2023-10-05T19:28:31.767 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.730+0000 7fe5064da700 1 -- 172.21.15.167:0/3045448955 <== osd.3 v2:172.21.15.167:6808/30036 23 ==== osd_op_reply(89 notify.2 [watch watch cookie 94916745811680] v30'100 uv99 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7fe50004d040 con 0x565386ea7140 2023-10-05T19:28:31.767 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.734+0000 7fe505cd9700 1 -- 172.21.15.167:0/3045448955 <== osd.2 v2:172.21.15.167:6816/30037 16 ==== osd_op_reply(91 notify.3 [watch watch cookie 94916745816096] v30'100 uv99 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7fe4f0029040 con 0x7fe500016730 2023-10-05T19:28:31.768 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.734+0000 7fe506cdb700 1 -- 172.21.15.167:0/3045448955 <== osd.1 v2:172.21.15.167:6800/30035 31 ==== osd_op_reply(92 notify.4 [create] v30'99 uv99 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7fe4f8052040 con 0x565386e76600 2023-10-05T19:28:31.768 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.734+0000 7fe5089a5e40 1 -- 172.21.15.167:0/3045448955 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:93 7.d 7:b4812045:::notify.4:head [watch watch cookie 94916745818304] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x565386eafb60 con 0x565386e76600 2023-10-05T19:28:31.768 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.734+0000 7fe5089a5e40 1 -- 172.21.15.167:0/3045448955 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:94 7.3 7:c609908c:::notify.5:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x565386eb0980 con 0x7fe4f0007030 2023-10-05T19:28:31.768 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.734+0000 7fe506cdb700 1 -- 172.21.15.167:0/3045448955 <== osd.0 v2:172.21.15.118:6800/27690 23 ==== osd_op_reply(94 notify.5 [create] v30'99 uv99 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7fe4f8052040 con 0x7fe4f0007030 2023-10-05T19:28:31.768 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.734+0000 7fe5089a5e40 1 -- 172.21.15.167:0/3045448955 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:95 7.3 7:c609908c:::notify.5:head [watch watch cookie 94916745823296] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x565386eb0d80 con 0x7fe4f0007030 2023-10-05T19:28:31.768 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.734+0000 7fe5089a5e40 1 -- 172.21.15.167:0/3045448955 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:96 7.14 7:2b04a3e9:::notify.6:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x565386eb1790 con 0x7fe4f0007030 2023-10-05T19:28:31.768 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.738+0000 7fe506cdb700 1 -- 172.21.15.167:0/3045448955 <== osd.1 v2:172.21.15.167:6800/30035 32 ==== osd_op_reply(93 notify.4 [watch watch cookie 94916745818304] v30'100 uv99 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7fe4f8052040 con 0x565386e76600 2023-10-05T19:28:31.768 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.738+0000 7fe506cdb700 1 -- 172.21.15.167:0/3045448955 <== osd.0 v2:172.21.15.118:6800/27690 24 ==== osd_op_reply(95 notify.5 [watch watch cookie 94916745823296] v30'100 uv99 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7fe4f8052040 con 0x7fe4f0007030 2023-10-05T19:28:31.768 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.738+0000 7fe506cdb700 1 -- 172.21.15.167:0/3045448955 <== osd.0 v2:172.21.15.118:6800/27690 25 ==== osd_op_reply(96 notify.6 [create] v30'99 uv99 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7fe4f8052040 con 0x7fe4f0007030 2023-10-05T19:28:31.768 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.738+0000 7fe5089a5e40 1 -- 172.21.15.167:0/3045448955 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:97 7.14 7:2b04a3e9:::notify.6:head [watch watch cookie 94916745824400] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x565386eb0ce0 con 0x7fe4f0007030 2023-10-05T19:28:31.768 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.738+0000 7fe5089a5e40 1 -- 172.21.15.167:0/3045448955 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:98 7.9 7:93e5b521:::notify.7:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x565386eb1c80 con 0x7fe4f0007030 2023-10-05T19:28:31.769 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.742+0000 7fe506cdb700 1 -- 172.21.15.167:0/3045448955 <== osd.0 v2:172.21.15.118:6800/27690 26 ==== osd_op_reply(97 notify.6 [watch watch cookie 94916745824400] v30'100 uv99 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7fe4f8052040 con 0x7fe4f0007030 2023-10-05T19:28:31.769 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.742+0000 7fe506cdb700 1 -- 172.21.15.167:0/3045448955 <== osd.0 v2:172.21.15.118:6800/27690 27 ==== osd_op_reply(98 notify.7 [create] v30'99 uv99 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7fe4f8052040 con 0x7fe4f0007030 2023-10-05T19:28:31.769 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.742+0000 7fe5089a5e40 1 -- 172.21.15.167:0/3045448955 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:99 7.9 7:93e5b521:::notify.7:head [watch watch cookie 94916745826976] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x565386eb1be0 con 0x7fe4f0007030 2023-10-05T19:28:31.769 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.742+0000 7fe5089a5e40 20 add_watcher() i=0 2023-10-05T19:28:31.769 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.742+0000 7fe5089a5e40 20 add_watcher() i=1 2023-10-05T19:28:31.769 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.742+0000 7fe5089a5e40 20 add_watcher() i=2 2023-10-05T19:28:31.769 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.742+0000 7fe5089a5e40 20 add_watcher() i=3 2023-10-05T19:28:31.769 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.742+0000 7fe5089a5e40 20 add_watcher() i=4 2023-10-05T19:28:31.769 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.742+0000 7fe5089a5e40 20 add_watcher() i=5 2023-10-05T19:28:31.769 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.742+0000 7fe5089a5e40 20 add_watcher() i=6 2023-10-05T19:28:31.769 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.742+0000 7fe506cdb700 1 -- 172.21.15.167:0/3045448955 <== osd.0 v2:172.21.15.118:6800/27690 28 ==== osd_op_reply(99 notify.7 [watch watch cookie 94916745826976] v30'100 uv99 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7fe4f8052040 con 0x7fe4f0007030 2023-10-05T19:28:31.769 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.742+0000 7fe5089a5e40 20 add_watcher() i=7 2023-10-05T19:28:31.770 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.742+0000 7fe5089a5e40 2 all 8 watchers are set, enabling cache 2023-10-05T19:28:31.770 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.742+0000 7fe5089a5e40 1 -- 172.21.15.167:0/3045448955 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:100 6.4 6:22d26bf9:::data_loggenerations_metadata:head [call version.check_conds in=74b,call version.read in=11b,read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x565386eb4d80 con 0x565386e76600 2023-10-05T19:28:31.770 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.746+0000 7fe506cdb700 1 -- 172.21.15.167:0/3045448955 <== osd.1 v2:172.21.15.167:6800/30035 33 ==== osd_op_reply(100 data_loggenerations_metadata [call,call out=48b,read 0~28 out=28b] v0'0 uv1 ondisk = 0) v8 ==== 256+0+76 (crc 0 0 0) 0x7fe4f8052040 con 0x565386e76600 2023-10-05T19:28:31.770 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.746+0000 7fe5089a5e40 1 -- 172.21.15.167:0/3045448955 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:101 6.4 6:22d26bf9:::data_loggenerations_metadata:head [watch watch cookie 94916745828448] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x565386eb22c0 con 0x565386e76600 2023-10-05T19:28:31.770 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.746+0000 7fe506cdb700 1 -- 172.21.15.167:0/3045448955 <== osd.1 v2:172.21.15.167:6800/30035 34 ==== osd_op_reply(101 data_loggenerations_metadata [watch watch cookie 94916745828448] v30'101 uv1 ondisk = 0) v8 ==== 172+0+0 (crc 0 0 0) 0x7fe4f8052040 con 0x565386e76600 2023-10-05T19:28:31.770 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.746+0000 7fe4d6fed700 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2023-10-05T19:28:31.770 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.746+0000 7fe5089a5e40 20 check_secure_mon_conn(): auth registy supported: methods=[2] modes=[2,1] 2023-10-05T19:28:31.770 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.746+0000 7fe5089a5e40 20 check_secure_mon_conn(): mode 1 is insecure 2023-10-05T19:28:31.770 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.746+0000 7fe5089a5e40 1 Processor -- start 2023-10-05T19:28:31.770 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.746+0000 7fe5089a5e40 1 -- start start 2023-10-05T19:28:31.770 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.750+0000 7fe5089a5e40 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x565386ebf1d0 0x565386ebf5a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-05T19:28:31.771 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.750+0000 7fe5089a5e40 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x565386ebfae0 con 0x565386ebf1d0 2023-10-05T19:28:31.771 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.750+0000 7fe505cd9700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x565386ebf1d0 0x565386ebf5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-05T19:28:31.771 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.750+0000 7fe505cd9700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x565386ebf1d0 0x565386ebf5a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.118:3300/0 says I am v2:172.21.15.167:58250/0 (socket says 172.21.15.167:58250) 2023-10-05T19:28:31.771 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.750+0000 7fe505cd9700 1 -- 172.21.15.167:0/1252226894 learned_addr learned my addr 172.21.15.167:0/1252226894 (peer_addr_for_me v2:172.21.15.167:0/0) 2023-10-05T19:28:31.771 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.750+0000 7fe505cd9700 1 -- 172.21.15.167:0/1252226894 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x565386ebfc20 con 0x565386ebf1d0 2023-10-05T19:28:31.771 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.750+0000 7fe505cd9700 1 --2- 172.21.15.167:0/1252226894 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x565386ebf1d0 0x565386ebf5a0 secure :-1 s=READY pgs=336 cs=0 l=1 rev1=1 crypto rx=0x565386e730a0 tx=0x7fe4f0030040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d6b3baa8887b9aed server_cookie=0 in_seq=0 out_seq=0 2023-10-05T19:28:31.771 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.750+0000 7fe4d67ec700 1 -- 172.21.15.167:0/1252226894 <== mon.0 v2:172.21.15.118:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fe4f0025040 con 0x565386ebf1d0 2023-10-05T19:28:31.771 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.750+0000 7fe4d67ec700 1 -- 172.21.15.167:0/1252226894 <== mon.0 v2:172.21.15.118:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe4f0034050 con 0x565386ebf1d0 2023-10-05T19:28:31.771 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.754+0000 7fe4d67ec700 1 -- 172.21.15.167:0/1252226894 <== mon.0 v2:172.21.15.118:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fe4f002e4e0 con 0x565386ebf1d0 2023-10-05T19:28:31.771 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.754+0000 7fe5089a5e40 1 -- 172.21.15.167:0/1252226894 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x565386ebf1d0 msgr2=0x565386ebf5a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-05T19:28:31.771 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.754+0000 7fe5089a5e40 1 --2- 172.21.15.167:0/1252226894 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x565386ebf1d0 0x565386ebf5a0 secure :-1 s=READY pgs=336 cs=0 l=1 rev1=1 crypto rx=0x565386e730a0 tx=0x7fe4f0030040 comp rx=0 tx=0).stop 2023-10-05T19:28:31.771 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.754+0000 7fe5089a5e40 1 -- 172.21.15.167:0/1252226894 shutdown_connections 2023-10-05T19:28:31.772 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.754+0000 7fe5089a5e40 1 --2- 172.21.15.167:0/1252226894 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x565386ebf1d0 0x565386ebf5a0 unknown :-1 s=CLOSED pgs=336 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-05T19:28:31.772 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.754+0000 7fe5089a5e40 1 -- 172.21.15.167:0/1252226894 >> 172.21.15.167:0/1252226894 conn(0x565386eba590 msgr2=0x565386eb8740 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-05T19:28:31.772 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.754+0000 7fe5089a5e40 1 -- 172.21.15.167:0/1252226894 shutdown_connections 2023-10-05T19:28:31.772 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.754+0000 7fe5089a5e40 1 -- 172.21.15.167:0/1252226894 wait complete. 2023-10-05T19:28:31.772 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.754+0000 7fe5089a5e40 1 Processor -- start 2023-10-05T19:28:31.772 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.754+0000 7fe5089a5e40 1 -- start start 2023-10-05T19:28:31.772 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.754+0000 7fe5089a5e40 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x565386ebf1d0 0x565386ec4c30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-05T19:28:31.772 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.754+0000 7fe5089a5e40 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x565386ec5c30 con 0x565386ebf1d0 2023-10-05T19:28:31.772 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.754+0000 7fe505cd9700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x565386ebf1d0 0x565386ec4c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-05T19:28:31.772 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.754+0000 7fe505cd9700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x565386ebf1d0 0x565386ec4c30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.118:3300/0 says I am v2:172.21.15.167:58262/0 (socket says 172.21.15.167:58262) 2023-10-05T19:28:31.772 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.754+0000 7fe505cd9700 1 -- 172.21.15.167:0/2509385734 learned_addr learned my addr 172.21.15.167:0/2509385734 (peer_addr_for_me v2:172.21.15.167:0/0) 2023-10-05T19:28:31.773 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.754+0000 7fe505cd9700 1 -- 172.21.15.167:0/2509385734 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x565386e72c30 con 0x565386ebf1d0 2023-10-05T19:28:31.773 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.754+0000 7fe505cd9700 1 --2- 172.21.15.167:0/2509385734 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x565386ebf1d0 0x565386ec4c30 secure :-1 s=READY pgs=337 cs=0 l=1 rev1=1 crypto rx=0x565386eb92b0 tx=0x7fe4f0038000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-05T19:28:31.773 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.758+0000 7fe4d4fe9700 1 -- 172.21.15.167:0/2509385734 <== mon.0 v2:172.21.15.118:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fe4f0025040 con 0x565386ebf1d0 2023-10-05T19:28:31.773 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.758+0000 7fe4d4fe9700 1 -- 172.21.15.167:0/2509385734 <== mon.0 v2:172.21.15.118:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe4f0039960 con 0x565386ebf1d0 2023-10-05T19:28:31.773 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.758+0000 7fe4d4fe9700 1 -- 172.21.15.167:0/2509385734 <== mon.0 v2:172.21.15.118:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fe4f0039b30 con 0x565386ebf1d0 2023-10-05T19:28:31.773 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.758+0000 7fe5089a5e40 1 -- 172.21.15.167:0/2509385734 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x565386ec5230 con 0x565386ebf1d0 2023-10-05T19:28:31.773 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.758+0000 7fe5089a5e40 1 -- 172.21.15.167:0/2509385734 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x565386ec56d0 con 0x565386ebf1d0 2023-10-05T19:28:31.773 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.758+0000 7fe4d4fe9700 1 -- 172.21.15.167:0/2509385734 <== mon.0 v2:172.21.15.118:3300/0 4 ==== mgrmap(e 5) v1 ==== 40329+0+0 (secure 0 0 0) 0x7fe4f0039c90 con 0x565386ebf1d0 2023-10-05T19:28:31.773 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.758+0000 7fe4d4fe9700 1 --2- 172.21.15.167:0/2509385734 >> [v2:172.21.15.167:6824/29889,v1:172.21.15.167:6825/29889] conn(0x7fe44c030c50 0x7fe44c033100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-05T19:28:31.773 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.758+0000 7fe4d4fe9700 1 -- 172.21.15.167:0/2509385734 <== mon.0 v2:172.21.15.118:3300/0 5 ==== osd_map(30..30 src has 1..30) v4 ==== 6057+0+0 (secure 0 0 0) 0x7fe4f004dc80 con 0x565386ebf1d0 2023-10-05T19:28:31.773 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.758+0000 7fe506cdb700 1 --2- 172.21.15.167:0/2509385734 >> [v2:172.21.15.167:6824/29889,v1:172.21.15.167:6825/29889] conn(0x7fe44c030c50 0x7fe44c033100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-05T19:28:31.774 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.758+0000 7fe5089a5e40 5 note: GC not initialized 2023-10-05T19:28:31.774 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.758+0000 7fe506cdb700 1 --2- 172.21.15.167:0/2509385734 >> [v2:172.21.15.167:6824/29889,v1:172.21.15.167:6825/29889] conn(0x7fe44c030c50 0x7fe44c033100 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x565386eb2220 tx=0x7fe4f800aec0 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-05T19:28:31.779 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.762+0000 7fe445ffb700 20 reqs_thread_entry: start 2023-10-05T19:28:31.779 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.762+0000 7fe5089a5e40 20 init_complete bucket index max shards: 11 2023-10-05T19:28:31.779 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.762+0000 7fe4377fe700 20 reqs_thread_entry: start 2023-10-05T19:28:31.779 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.762+0000 7fe5089a5e40 10 rgw notify: Started notification manager with: 1 workers 2023-10-05T19:28:31.779 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.762+0000 7fe435ffb700 20 rgw notify: INFO: next queues processing will happen at: Thu Oct 5 19:29:01 2023 2023-10-05T19:28:31.779 INFO:teuthology.orchestra.run.smithi167.stderr: 2023-10-05T19:28:31.779 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.762+0000 7fe4357fa700 20 reqs_thread_entry: start 2023-10-05T19:28:31.779 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.762+0000 7fe5089a5e40 20 get_system_obj_state: rctx=0x7ffc011d0770 obj=default.rgw.meta:root:myfoo state=0x565386d490b0 s->prefetch_data=0 2023-10-05T19:28:31.780 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.762+0000 7fe5089a5e40 1 -- 172.21.15.167:0/3045448955 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:102 8.9 8:91b71841:root::myfoo:head [call version.read in=11b,getxattrs,stat] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x565386f2dd80 con 0x7fe500016730 2023-10-05T19:28:31.780 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.762+0000 7fe505cd9700 1 -- 172.21.15.167:0/3045448955 <== osd.2 v2:172.21.15.167:6816/30037 17 ==== osd_op_reply(102 myfoo [call out=48b,getxattrs out=75b,stat out=16b] v0'0 uv1 ondisk = 0) v8 ==== 233+0+139 (crc 0 0 0) 0x7fe4f003f020 con 0x7fe500016730 2023-10-05T19:28:31.780 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.762+0000 7fe5089a5e40 20 get_system_obj_state: s->obj_tag was set empty 2023-10-05T19:28:31.780 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.762+0000 7fe5089a5e40 20 rados->read ofs=0 len=0 2023-10-05T19:28:31.780 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.762+0000 7fe5089a5e40 1 -- 172.21.15.167:0/3045448955 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:103 8.9 8:91b71841:root::myfoo:head [call version.check_conds in=122b,call version.read in=11b,read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x565386f30810 con 0x7fe500016730 2023-10-05T19:28:31.780 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.766+0000 7fe505cd9700 1 -- 172.21.15.167:0/3045448955 <== osd.2 v2:172.21.15.167:6816/30037 18 ==== osd_op_reply(103 myfoo [call,call out=48b,read 0~165 out=165b] v0'0 uv1 ondisk = 0) v8 ==== 233+0+213 (crc 0 0 0) 0x7fe4f003f020 con 0x7fe500016730 2023-10-05T19:28:31.780 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.766+0000 7fe5089a5e40 20 rados_obj.operate() r=0 bl.length=165 2023-10-05T19:28:31.780 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.766+0000 7fe5089a5e40 20 get_system_obj_state: rctx=0x7ffc011d0770 obj=default.rgw.meta:root:.bucket.meta.myfoo:9a9df965-6612-45d4-a45f-6e3c889bacc2.4223.1 state=0x565386d1f2b0 s->prefetch_data=0 2023-10-05T19:28:31.780 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.766+0000 7fe5089a5e40 1 -- 172.21.15.167:0/3045448955 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:104 8.11 8:8bb4ed75:root::.bucket.meta.myfoo%3a9a9df965-6612-45d4-a45f-6e3c889bacc2.4223.1:head [call version.read in=11b,getxattrs,stat] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x565386f2ea10 con 0x565386ea7140 2023-10-05T19:28:31.780 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.766+0000 7fe5064da700 1 -- 172.21.15.167:0/3045448955 <== osd.3 v2:172.21.15.167:6808/30036 24 ==== osd_op_reply(104 .bucket.meta.myfoo:9a9df965-6612-45d4-a45f-6e3c889bacc2.4223.1 [call out=48b,getxattrs out=218b,stat out=16b] v0'0 uv1 ondisk = 0) v8 ==== 290+0+282 (crc 0 0 0) 0x7fe50004d040 con 0x565386ea7140 2023-10-05T19:28:31.780 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.766+0000 7fe5089a5e40 20 get_system_obj_state: s->obj_tag was set empty 2023-10-05T19:28:31.781 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.766+0000 7fe5089a5e40 20 Read xattr: user.rgw.acl 2023-10-05T19:28:31.781 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.766+0000 7fe5089a5e40 20 Read xattr: user.rgw.idtag 2023-10-05T19:28:31.781 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.766+0000 7fe5089a5e40 20 rados->read ofs=0 len=0 2023-10-05T19:28:31.781 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.766+0000 7fe5089a5e40 1 -- 172.21.15.167:0/3045448955 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:105 8.11 8:8bb4ed75:root::.bucket.meta.myfoo%3a9a9df965-6612-45d4-a45f-6e3c889bacc2.4223.1:head [call version.check_conds in=122b,call version.read in=11b,read 0~0,getxattrs] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x565386f305e0 con 0x565386ea7140 2023-10-05T19:28:31.781 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.766+0000 7fe5064da700 1 -- 172.21.15.167:0/3045448955 <== osd.3 v2:172.21.15.167:6808/30036 25 ==== osd_op_reply(105 .bucket.meta.myfoo:9a9df965-6612-45d4-a45f-6e3c889bacc2.4223.1 [call,call out=48b,read 0~352 out=352b,getxattrs out=218b] v0'0 uv1 ondisk = 0) v8 ==== 332+0+618 (crc 0 0 0) 0x7fe50004d040 con 0x565386ea7140 2023-10-05T19:28:31.781 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.766+0000 7fe5089a5e40 20 rados_obj.operate() r=0 bl.length=352 2023-10-05T19:28:31.781 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.766+0000 7fe5089a5e40 20 couldn't put binfo cache entry, might have raced with data changes 2023-10-05T19:28:31.781 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.766+0000 7fe5089a5e40 20 get_system_obj_state: rctx=0x7ffc011d06d0 obj=default.rgw.meta:root:myfoo state=0x565386d1f2b0 s->prefetch_data=0 2023-10-05T19:28:31.781 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.766+0000 7fe5089a5e40 1 -- 172.21.15.167:0/3045448955 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:106 8.9 8:91b71841:root::myfoo:head [call version.read in=11b,getxattrs,stat] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x565386f315b0 con 0x7fe500016730 2023-10-05T19:28:31.781 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.766+0000 7fe505cd9700 1 -- 172.21.15.167:0/3045448955 <== osd.2 v2:172.21.15.167:6816/30037 19 ==== osd_op_reply(106 myfoo [call out=48b,getxattrs out=75b,stat out=16b] v0'0 uv1 ondisk = 0) v8 ==== 233+0+139 (crc 0 0 0) 0x7fe4f003f020 con 0x7fe500016730 2023-10-05T19:28:31.781 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.766+0000 7fe5089a5e40 20 get_system_obj_state: s->obj_tag was set empty 2023-10-05T19:28:31.781 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.766+0000 7fe5089a5e40 20 rados->read ofs=0 len=0 2023-10-05T19:28:31.782 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.766+0000 7fe5089a5e40 1 -- 172.21.15.167:0/3045448955 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:107 8.9 8:91b71841:root::myfoo:head [call version.check_conds in=122b,call version.read in=11b,read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x565386f315b0 con 0x7fe500016730 2023-10-05T19:28:31.782 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.766+0000 7fe505cd9700 1 -- 172.21.15.167:0/3045448955 <== osd.2 v2:172.21.15.167:6816/30037 20 ==== osd_op_reply(107 myfoo [call,call out=48b,read 0~165 out=165b] v0'0 uv1 ondisk = 0) v8 ==== 233+0+213 (crc 0 0 0) 0x7fe4f003f020 con 0x7fe500016730 2023-10-05T19:28:31.782 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.766+0000 7fe5089a5e40 20 rados_obj.operate() r=0 bl.length=165 2023-10-05T19:28:31.782 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.766+0000 7fe5089a5e40 20 get_system_obj_state: rctx=0x7ffc011d06d0 obj=default.rgw.meta:root:.bucket.meta.myfoo:9a9df965-6612-45d4-a45f-6e3c889bacc2.4223.1 state=0x565386d490b0 s->prefetch_data=0 2023-10-05T19:28:31.782 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.766+0000 7fe5089a5e40 1 -- 172.21.15.167:0/3045448955 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:108 8.11 8:8bb4ed75:root::.bucket.meta.myfoo%3a9a9df965-6612-45d4-a45f-6e3c889bacc2.4223.1:head [call version.read in=11b,getxattrs,stat] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x565386f315b0 con 0x565386ea7140 2023-10-05T19:28:31.782 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.770+0000 7fe5064da700 1 -- 172.21.15.167:0/3045448955 <== osd.3 v2:172.21.15.167:6808/30036 26 ==== osd_op_reply(108 .bucket.meta.myfoo:9a9df965-6612-45d4-a45f-6e3c889bacc2.4223.1 [call out=48b,getxattrs out=218b,stat out=16b] v0'0 uv1 ondisk = 0) v8 ==== 290+0+282 (crc 0 0 0) 0x7fe50004d040 con 0x565386ea7140 2023-10-05T19:28:31.782 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.770+0000 7fe5089a5e40 20 get_system_obj_state: s->obj_tag was set empty 2023-10-05T19:28:31.782 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.770+0000 7fe5089a5e40 20 Read xattr: user.rgw.acl 2023-10-05T19:28:31.782 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.770+0000 7fe5089a5e40 20 Read xattr: user.rgw.idtag 2023-10-05T19:28:31.782 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.770+0000 7fe5089a5e40 20 rados->read ofs=0 len=0 2023-10-05T19:28:31.782 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.770+0000 7fe5089a5e40 1 -- 172.21.15.167:0/3045448955 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:109 8.11 8:8bb4ed75:root::.bucket.meta.myfoo%3a9a9df965-6612-45d4-a45f-6e3c889bacc2.4223.1:head [call version.check_conds in=122b,call version.read in=11b,read 0~0,getxattrs] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x565386f305c0 con 0x565386ea7140 2023-10-05T19:28:31.782 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.770+0000 7fe5064da700 1 -- 172.21.15.167:0/3045448955 <== osd.3 v2:172.21.15.167:6808/30036 27 ==== osd_op_reply(109 .bucket.meta.myfoo:9a9df965-6612-45d4-a45f-6e3c889bacc2.4223.1 [call,call out=48b,read 0~352 out=352b,getxattrs out=218b] v0'0 uv1 ondisk = 0) v8 ==== 332+0+618 (crc 0 0 0) 0x7fe50004d040 con 0x565386ea7140 2023-10-05T19:28:31.783 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.770+0000 7fe5089a5e40 20 rados_obj.operate() r=0 bl.length=352 2023-10-05T19:28:31.783 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.770+0000 7fe5089a5e40 20 couldn't put binfo cache entry, might have raced with data changes 2023-10-05T19:28:31.783 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.770+0000 7fe5089a5e40 1 -- 172.21.15.167:0/3045448955 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:110 4.3d 4:bd5dd36d:::.dir.9a9df965-6612-45d4-a45f-6e3c889bacc2.4223.1.0:head [call rgw.bucket_list in=47b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x565386f33020 con 0x565386ea7140 2023-10-05T19:28:31.783 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.770+0000 7fe5089a5e40 1 -- 172.21.15.167:0/3045448955 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:111 4.29 4:955f623b:::.dir.9a9df965-6612-45d4-a45f-6e3c889bacc2.4223.1.1:head [call rgw.bucket_list in=47b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x565386f33c60 con 0x565386ea7140 2023-10-05T19:28:31.783 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.770+0000 7fe5089a5e40 1 -- 172.21.15.167:0/3045448955 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:112 4.38 4:1d83ede1:::.dir.9a9df965-6612-45d4-a45f-6e3c889bacc2.4223.1.2:head [call rgw.bucket_list in=47b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x565386f33c60 con 0x565386e76600 2023-10-05T19:28:31.783 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.770+0000 7fe5089a5e40 1 -- 172.21.15.167:0/3045448955 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:113 4.1e 4:7939a9ac:::.dir.9a9df965-6612-45d4-a45f-6e3c889bacc2.4223.1.3:head [call rgw.bucket_list in=47b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x565386f3aff0 con 0x7fe4f0007030 2023-10-05T19:28:31.783 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.770+0000 7fe5089a5e40 1 -- 172.21.15.167:0/3045448955 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:114 4.e 4:719498d8:::.dir.9a9df965-6612-45d4-a45f-6e3c889bacc2.4223.1.4:head [call rgw.bucket_list in=47b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x565386f3de70 con 0x7fe500016730 2023-10-05T19:28:31.783 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.770+0000 7fe5089a5e40 1 -- 172.21.15.167:0/3045448955 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:115 4.32 4:4f38608c:::.dir.9a9df965-6612-45d4-a45f-6e3c889bacc2.4223.1.5:head [call rgw.bucket_list in=47b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x565386f40980 con 0x565386e76600 2023-10-05T19:28:31.783 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.770+0000 7fe5089a5e40 1 -- 172.21.15.167:0/3045448955 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:116 4.38 4:1f2a6458:::.dir.9a9df965-6612-45d4-a45f-6e3c889bacc2.4223.1.6:head [call rgw.bucket_list in=47b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x565386f432c0 con 0x565386e76600 2023-10-05T19:28:31.783 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.770+0000 7fe5089a5e40 1 -- 172.21.15.167:0/3045448955 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:117 4.2 4:41a3e149:::.dir.9a9df965-6612-45d4-a45f-6e3c889bacc2.4223.1.7:head [call rgw.bucket_list in=47b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x565386f45e50 con 0x565386e76600 2023-10-05T19:28:31.783 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.770+0000 7fe5089a5e40 1 -- 172.21.15.167:0/3045448955 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:118 4.17 4:ea8a113e:::.dir.9a9df965-6612-45d4-a45f-6e3c889bacc2.4223.1.8:head [call rgw.bucket_list in=47b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x565386f48d40 con 0x7fe4f0007030 2023-10-05T19:28:31.784 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.770+0000 7fe5089a5e40 1 -- 172.21.15.167:0/3045448955 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:119 4.1e 4:7a3e9338:::.dir.9a9df965-6612-45d4-a45f-6e3c889bacc2.4223.1.9:head [call rgw.bucket_list in=47b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x565386f4bbf0 con 0x7fe4f0007030 2023-10-05T19:28:31.784 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.770+0000 7fe5089a5e40 1 -- 172.21.15.167:0/3045448955 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:120 4.29 4:94432b39:::.dir.9a9df965-6612-45d4-a45f-6e3c889bacc2.4223.1.10:head [call rgw.bucket_list in=47b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x565386f4eaf0 con 0x565386ea7140 2023-10-05T19:28:31.784 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.770+0000 7fe5064da700 1 -- 172.21.15.167:0/3045448955 <== osd.3 v2:172.21.15.167:6808/30036 28 ==== osd_op_reply(110 .dir.9a9df965-6612-45d4-a45f-6e3c889bacc2.4223.1.0 [call out=85b] v0'0 uv1 ondisk = 0) v8 ==== 194+0+85 (crc 0 0 0) 0x7fe50004d040 con 0x565386ea7140 2023-10-05T19:28:31.784 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.770+0000 7fe5064da700 1 -- 172.21.15.167:0/3045448955 <== osd.3 v2:172.21.15.167:6808/30036 29 ==== osd_op_reply(111 .dir.9a9df965-6612-45d4-a45f-6e3c889bacc2.4223.1.1 [call out=85b] v0'0 uv1 ondisk = 0) v8 ==== 194+0+85 (crc 0 0 0) 0x7fe50004d040 con 0x565386ea7140 2023-10-05T19:28:31.784 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.770+0000 7fe505cd9700 1 -- 172.21.15.167:0/3045448955 <== osd.2 v2:172.21.15.167:6816/30037 21 ==== osd_op_reply(114 .dir.9a9df965-6612-45d4-a45f-6e3c889bacc2.4223.1.4 [call out=85b] v0'0 uv1 ondisk = 0) v8 ==== 194+0+85 (crc 0 0 0) 0x7fe4f003f020 con 0x7fe500016730 2023-10-05T19:28:31.784 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.770+0000 7fe506cdb700 1 -- 172.21.15.167:0/3045448955 <== osd.1 v2:172.21.15.167:6800/30035 35 ==== osd_op_reply(112 .dir.9a9df965-6612-45d4-a45f-6e3c889bacc2.4223.1.2 [call out=85b] v0'0 uv1 ondisk = 0) v8 ==== 194+0+85 (crc 0 0 0) 0x7fe4f8052040 con 0x565386e76600 2023-10-05T19:28:31.784 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.770+0000 7fe506cdb700 1 -- 172.21.15.167:0/3045448955 <== osd.0 v2:172.21.15.118:6800/27690 29 ==== osd_op_reply(113 .dir.9a9df965-6612-45d4-a45f-6e3c889bacc2.4223.1.3 [call out=85b] v0'0 uv1 ondisk = 0) v8 ==== 194+0+85 (crc 0 0 0) 0x7fe4f8052040 con 0x7fe4f0007030 2023-10-05T19:28:31.784 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.770+0000 7fe5064da700 1 -- 172.21.15.167:0/3045448955 <== osd.3 v2:172.21.15.167:6808/30036 30 ==== osd_op_reply(120 .dir.9a9df965-6612-45d4-a45f-6e3c889bacc2.4223.1.10 [call out=85b] v0'0 uv2 ondisk = 0) v8 ==== 195+0+85 (crc 0 0 0) 0x7fe50004d040 con 0x565386ea7140 2023-10-05T19:28:31.784 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.770+0000 7fe506cdb700 1 -- 172.21.15.167:0/3045448955 <== osd.1 v2:172.21.15.167:6800/30035 36 ==== osd_op_reply(115 .dir.9a9df965-6612-45d4-a45f-6e3c889bacc2.4223.1.5 [call out=124b] v0'0 uv3 ondisk = 0) v8 ==== 194+0+124 (crc 0 0 0) 0x7fe4f8052040 con 0x565386e76600 2023-10-05T19:28:31.784 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.770+0000 7fe506cdb700 1 -- 172.21.15.167:0/3045448955 <== osd.1 v2:172.21.15.167:6800/30035 37 ==== osd_op_reply(116 .dir.9a9df965-6612-45d4-a45f-6e3c889bacc2.4223.1.6 [call out=85b] v0'0 uv2 ondisk = 0) v8 ==== 194+0+85 (crc 0 0 0) 0x7fe4f8052040 con 0x565386e76600 2023-10-05T19:28:31.784 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.770+0000 7fe506cdb700 1 -- 172.21.15.167:0/3045448955 <== osd.0 v2:172.21.15.118:6800/27690 30 ==== osd_op_reply(118 .dir.9a9df965-6612-45d4-a45f-6e3c889bacc2.4223.1.8 [call out=85b] v0'0 uv1 ondisk = 0) v8 ==== 194+0+85 (crc 0 0 0) 0x7fe4f8052040 con 0x7fe4f0007030 2023-10-05T19:28:31.785 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.770+0000 7fe506cdb700 1 -- 172.21.15.167:0/3045448955 <== osd.1 v2:172.21.15.167:6800/30035 38 ==== osd_op_reply(117 .dir.9a9df965-6612-45d4-a45f-6e3c889bacc2.4223.1.7 [call out=85b] v0'0 uv1 ondisk = 0) v8 ==== 194+0+85 (crc 0 0 0) 0x7fe4f8052040 con 0x565386e76600 2023-10-05T19:28:31.785 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.770+0000 7fe506cdb700 1 -- 172.21.15.167:0/3045448955 <== osd.0 v2:172.21.15.118:6800/27690 31 ==== osd_op_reply(119 .dir.9a9df965-6612-45d4-a45f-6e3c889bacc2.4223.1.9 [call out=85b] v0'0 uv2 ondisk = 0) v8 ==== 194+0+85 (crc 0 0 0) 0x7fe4f8052040 con 0x7fe4f0007030 2023-10-05T19:28:31.785 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.774+0000 7fe5089a5e40 1 -- 172.21.15.167:0/3045448955 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:121 7.1f 7:f95f44c2:::notify.0:head [watch unwatch cookie 94916745804784] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x565386f4bbf0 con 0x7fe500016730 2023-10-05T19:28:31.785 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.774+0000 7fe505cd9700 1 -- 172.21.15.167:0/3045448955 <== osd.2 v2:172.21.15.167:6816/30037 22 ==== osd_op_reply(121 notify.0 [watch unwatch cookie 94916745804784] v30'101 uv99 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7fe4f003f020 con 0x7fe500016730 2023-10-05T19:28:31.786 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.774+0000 7fe5089a5e40 20 remove_watcher() i=0 2023-10-05T19:28:31.786 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.774+0000 7fe5089a5e40 2 removed watcher, disabling cache 2023-10-05T19:28:31.786 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.774+0000 7fe5089a5e40 1 -- 172.21.15.167:0/3045448955 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:122 7.0 7:05bf5b68:::notify.1:head [watch unwatch cookie 94916745809328] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x565386f4bbf0 con 0x565386e76600 2023-10-05T19:28:31.786 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.778+0000 7fe506cdb700 1 -- 172.21.15.167:0/3045448955 <== osd.1 v2:172.21.15.167:6800/30035 39 ==== osd_op_reply(122 notify.1 [watch unwatch cookie 94916745809328] v30'101 uv99 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7fe4f8052040 con 0x565386e76600 2023-10-05T19:28:31.786 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.778+0000 7fe5089a5e40 20 remove_watcher() i=1 2023-10-05T19:28:31.786 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.778+0000 7fe5089a5e40 1 -- 172.21.15.167:0/3045448955 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:123 7.15 7:a93a5511:::notify.2:head [watch unwatch cookie 94916745811680] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x565386f4bbf0 con 0x565386ea7140 2023-10-05T19:28:31.787 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.782+0000 7fe5064da700 1 -- 172.21.15.167:0/3045448955 <== osd.3 v2:172.21.15.167:6808/30036 31 ==== osd_op_reply(123 notify.2 [watch unwatch cookie 94916745811680] v30'101 uv99 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7fe50004d040 con 0x565386ea7140 2023-10-05T19:28:31.787 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.782+0000 7fe5089a5e40 20 remove_watcher() i=2 2023-10-05T19:28:31.787 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.782+0000 7fe5089a5e40 1 -- 172.21.15.167:0/3045448955 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:124 7.e 7:7759931f:::notify.3:head [watch unwatch cookie 94916745816096] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x565386f4bbf0 con 0x7fe500016730 2023-10-05T19:28:31.789 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.782+0000 7fe505cd9700 1 -- 172.21.15.167:0/3045448955 <== osd.2 v2:172.21.15.167:6816/30037 23 ==== osd_op_reply(124 notify.3 [watch unwatch cookie 94916745816096] v30'101 uv99 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7fe4f003f020 con 0x7fe500016730 2023-10-05T19:28:31.789 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.782+0000 7fe5089a5e40 20 remove_watcher() i=3 2023-10-05T19:28:31.789 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.782+0000 7fe5089a5e40 1 -- 172.21.15.167:0/3045448955 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:125 7.d 7:b4812045:::notify.4:head [watch unwatch cookie 94916745818304] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x565386f4bbf0 con 0x565386e76600 2023-10-05T19:28:31.792 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.786+0000 7fe506cdb700 1 -- 172.21.15.167:0/3045448955 <== osd.1 v2:172.21.15.167:6800/30035 40 ==== osd_op_reply(125 notify.4 [watch unwatch cookie 94916745818304] v30'101 uv99 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7fe4f8052040 con 0x565386e76600 2023-10-05T19:28:31.792 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.786+0000 7fe5089a5e40 20 remove_watcher() i=4 2023-10-05T19:28:31.792 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.786+0000 7fe5089a5e40 1 -- 172.21.15.167:0/3045448955 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:126 7.3 7:c609908c:::notify.5:head [watch unwatch cookie 94916745823296] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x565386f4bbf0 con 0x7fe4f0007030 2023-10-05T19:28:31.795 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.790+0000 7fe506cdb700 1 -- 172.21.15.167:0/3045448955 <== osd.0 v2:172.21.15.118:6800/27690 32 ==== osd_op_reply(126 notify.5 [watch unwatch cookie 94916745823296] v30'101 uv99 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7fe4f8052040 con 0x7fe4f0007030 2023-10-05T19:28:31.795 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.790+0000 7fe5089a5e40 20 remove_watcher() i=5 2023-10-05T19:28:31.795 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.790+0000 7fe5089a5e40 1 -- 172.21.15.167:0/3045448955 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:127 7.14 7:2b04a3e9:::notify.6:head [watch unwatch cookie 94916745824400] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x565386f4bbf0 con 0x7fe4f0007030 2023-10-05T19:28:31.797 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.790+0000 7fe506cdb700 1 -- 172.21.15.167:0/3045448955 <== osd.0 v2:172.21.15.118:6800/27690 33 ==== osd_op_reply(127 notify.6 [watch unwatch cookie 94916745824400] v30'101 uv99 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7fe4f8052040 con 0x7fe4f0007030 2023-10-05T19:28:31.798 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.790+0000 7fe5089a5e40 20 remove_watcher() i=6 2023-10-05T19:28:31.798 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.790+0000 7fe5089a5e40 1 -- 172.21.15.167:0/3045448955 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:128 7.9 7:93e5b521:::notify.7:head [watch unwatch cookie 94916745826976] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x565386f4bbf0 con 0x7fe4f0007030 2023-10-05T19:28:31.801 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.794+0000 7fe506cdb700 1 -- 172.21.15.167:0/3045448955 <== osd.0 v2:172.21.15.118:6800/27690 34 ==== osd_op_reply(128 notify.7 [watch unwatch cookie 94916745826976] v30'101 uv99 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7fe4f8052040 con 0x7fe4f0007030 2023-10-05T19:28:31.801 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.794+0000 7fe5089a5e40 20 remove_watcher() i=7 2023-10-05T19:28:31.802 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.798+0000 7fe4d6fed700 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2023-10-05T19:28:31.803 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.798+0000 7fe5089a5e40 1 -- 172.21.15.167:0/3045448955 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:129 6.4 6:22d26bf9:::data_loggenerations_metadata:head [watch unwatch cookie 94916745828448] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x565386f4bbf0 con 0x565386e76600 2023-10-05T19:28:31.806 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.798+0000 7fe506cdb700 1 -- 172.21.15.167:0/3045448955 <== osd.1 v2:172.21.15.167:6800/30035 41 ==== osd_op_reply(129 data_loggenerations_metadata [watch unwatch cookie 94916745828448] v30'102 uv1 ondisk = 0) v8 ==== 172+0+0 (crc 0 0 0) 0x7fe4f8052040 con 0x565386e76600 2023-10-05T19:28:31.806 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.798+0000 7fe5089a5e40 1 -- 172.21.15.167:0/3045448955 >> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] conn(0x7fe4f0007030 msgr2=0x7fe4f001d430 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-05T19:28:31.806 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.798+0000 7fe5089a5e40 1 --2- 172.21.15.167:0/3045448955 >> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] conn(0x7fe4f0007030 0x7fe4f001d430 crc :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-05T19:28:31.806 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.798+0000 7fe5089a5e40 1 -- 172.21.15.167:0/3045448955 >> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] conn(0x565386e76600 msgr2=0x565386ea3730 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-05T19:28:31.807 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.798+0000 7fe5089a5e40 1 --2- 172.21.15.167:0/3045448955 >> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] conn(0x565386e76600 0x565386ea3730 crc :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-05T19:28:31.807 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.798+0000 7fe5089a5e40 1 -- 172.21.15.167:0/3045448955 >> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] conn(0x7fe500016730 msgr2=0x7fe500044a90 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-05T19:28:31.807 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.798+0000 7fe5089a5e40 1 --2- 172.21.15.167:0/3045448955 >> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] conn(0x7fe500016730 0x7fe500044a90 crc :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-05T19:28:31.807 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.798+0000 7fe5089a5e40 1 -- 172.21.15.167:0/3045448955 >> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] conn(0x565386ea7140 msgr2=0x565386ea9520 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-05T19:28:31.807 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.798+0000 7fe5089a5e40 1 --2- 172.21.15.167:0/3045448955 >> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] conn(0x565386ea7140 0x565386ea9520 crc :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-05T19:28:31.807 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.798+0000 7fe5089a5e40 1 -- 172.21.15.167:0/3045448955 >> [v2:172.21.15.167:6824/29889,v1:172.21.15.167:6825/29889] conn(0x7fe4ec030bf0 msgr2=0x7fe4ec0330a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-05T19:28:31.807 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.798+0000 7fe5089a5e40 1 --2- 172.21.15.167:0/3045448955 >> [v2:172.21.15.167:6824/29889,v1:172.21.15.167:6825/29889] conn(0x7fe4ec030bf0 0x7fe4ec0330a0 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7fe4f00030c0 tx=0x7fe4f0014000 comp rx=0 tx=0).stop 2023-10-05T19:28:31.807 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.802+0000 7fe5089a5e40 1 -- 172.21.15.167:0/3045448955 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x565386e72020 msgr2=0x565386e97cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-05T19:28:31.807 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.802+0000 7fe5089a5e40 1 --2- 172.21.15.167:0/3045448955 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x565386e72020 0x565386e97cd0 secure :-1 s=READY pgs=335 cs=0 l=1 rev1=1 crypto rx=0x7fe50000b960 tx=0x7fe500014000 comp rx=0 tx=0).stop 2023-10-05T19:28:31.807 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.802+0000 7fe506cdb700 1 -- 172.21.15.167:0/3045448955 reap_dead start 2023-10-05T19:28:31.807 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.802+0000 7fe5089a5e40 1 -- 172.21.15.167:0/3045448955 shutdown_connections 2023-10-05T19:28:31.808 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.802+0000 7fe5089a5e40 1 -- 172.21.15.167:0/3045448955 >> 172.21.15.167:0/3045448955 conn(0x565386de5870 msgr2=0x565386e8b580 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-05T19:28:31.808 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.802+0000 7fe5089a5e40 1 -- 172.21.15.167:0/3045448955 shutdown_connections 2023-10-05T19:28:31.808 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.802+0000 7fe5089a5e40 1 -- 172.21.15.167:0/3045448955 wait complete. 2023-10-05T19:28:31.809 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.802+0000 7fe5089a5e40 1 -- 172.21.15.167:0/2509385734 >> [v2:172.21.15.167:6824/29889,v1:172.21.15.167:6825/29889] conn(0x7fe44c030c50 msgr2=0x7fe44c033100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-05T19:28:31.809 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.802+0000 7fe5089a5e40 1 --2- 172.21.15.167:0/2509385734 >> [v2:172.21.15.167:6824/29889,v1:172.21.15.167:6825/29889] conn(0x7fe44c030c50 0x7fe44c033100 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x565386eb2220 tx=0x7fe4f800aec0 comp rx=0 tx=0).stop 2023-10-05T19:28:31.809 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.802+0000 7fe5089a5e40 1 -- 172.21.15.167:0/2509385734 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x565386ebf1d0 msgr2=0x565386ec4c30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-05T19:28:31.809 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.802+0000 7fe5089a5e40 1 --2- 172.21.15.167:0/2509385734 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x565386ebf1d0 0x565386ec4c30 secure :-1 s=READY pgs=337 cs=0 l=1 rev1=1 crypto rx=0x565386eb92b0 tx=0x7fe4f0038000 comp rx=0 tx=0).stop 2023-10-05T19:28:31.809 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.802+0000 7fe5089a5e40 1 -- 172.21.15.167:0/2509385734 shutdown_connections 2023-10-05T19:28:31.809 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.802+0000 7fe5089a5e40 1 --2- 172.21.15.167:0/2509385734 >> [v2:172.21.15.167:6824/29889,v1:172.21.15.167:6825/29889] conn(0x7fe44c030c50 0x7fe44c033100 unknown :-1 s=CLOSED pgs=119 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-05T19:28:31.809 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.802+0000 7fe5089a5e40 1 --2- 172.21.15.167:0/2509385734 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x565386ebf1d0 0x565386ec4c30 unknown :-1 s=CLOSED pgs=337 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-05T19:28:31.809 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.802+0000 7fe5089a5e40 1 -- 172.21.15.167:0/2509385734 >> 172.21.15.167:0/2509385734 conn(0x565386eba590 msgr2=0x565386eb8690 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-05T19:28:31.809 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.802+0000 7fe5089a5e40 1 -- 172.21.15.167:0/2509385734 shutdown_connections 2023-10-05T19:28:31.809 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.802+0000 7fe5089a5e40 1 -- 172.21.15.167:0/2509385734 wait complete. 2023-10-05T19:28:31.810 INFO:teuthology.orchestra.run.smithi167.stdout:{ 2023-10-05T19:28:31.810 INFO:teuthology.orchestra.run.smithi167.stdout: "bucket": "myfoo", 2023-10-05T19:28:31.810 INFO:teuthology.orchestra.run.smithi167.stdout: "num_shards": 11, 2023-10-05T19:28:31.810 INFO:teuthology.orchestra.run.smithi167.stdout: "tenant": "", 2023-10-05T19:28:31.810 INFO:teuthology.orchestra.run.smithi167.stdout: "zonegroup": "4f1f719d-9648-417b-b10c-e5d939825419", 2023-10-05T19:28:31.810 INFO:teuthology.orchestra.run.smithi167.stdout: "placement_rule": "default-placement", 2023-10-05T19:28:31.811 INFO:teuthology.orchestra.run.smithi167.stdout: "explicit_placement": { 2023-10-05T19:28:31.811 INFO:teuthology.orchestra.run.smithi167.stdout: "data_pool": "", 2023-10-05T19:28:31.811 INFO:teuthology.orchestra.run.smithi167.stdout: "data_extra_pool": "", 2023-10-05T19:28:31.811 INFO:teuthology.orchestra.run.smithi167.stdout: "index_pool": "" 2023-10-05T19:28:31.811 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2023-10-05T19:28:31.811 INFO:teuthology.orchestra.run.smithi167.stdout: "id": "9a9df965-6612-45d4-a45f-6e3c889bacc2.4223.1", 2023-10-05T19:28:31.811 INFO:teuthology.orchestra.run.smithi167.stdout: "marker": "9a9df965-6612-45d4-a45f-6e3c889bacc2.4223.1", 2023-10-05T19:28:31.811 INFO:teuthology.orchestra.run.smithi167.stdout: "index_type": "Normal", 2023-10-05T19:28:31.811 INFO:teuthology.orchestra.run.smithi167.stdout: "owner": "foo", 2023-10-05T19:28:31.811 INFO:teuthology.orchestra.run.smithi167.stdout: "ver": "0#1,1#1,2#1,3#1,4#1,5#2,6#1,7#1,8#1,9#1,10#1", 2023-10-05T19:28:31.811 INFO:teuthology.orchestra.run.smithi167.stdout: "master_ver": "0#0,1#0,2#0,3#0,4#0,5#0,6#0,7#0,8#0,9#0,10#0", 2023-10-05T19:28:31.812 INFO:teuthology.orchestra.run.smithi167.stdout: "mtime": "2023-10-05T19:28:30.258873Z", 2023-10-05T19:28:31.812 INFO:teuthology.orchestra.run.smithi167.stdout: "creation_time": "2023-10-05T19:28:30.250836Z", 2023-10-05T19:28:31.812 INFO:teuthology.orchestra.run.smithi167.stdout: "max_marker": "0#,1#,2#,3#,4#,5#,6#,7#,8#,9#,10#", 2023-10-05T19:28:31.812 INFO:teuthology.orchestra.run.smithi167.stdout: "usage": { 2023-10-05T19:28:31.812 INFO:teuthology.orchestra.run.smithi167.stdout: "rgw.main": { 2023-10-05T19:28:31.812 INFO:teuthology.orchestra.run.smithi167.stdout: "size": 3, 2023-10-05T19:28:31.812 INFO:teuthology.orchestra.run.smithi167.stdout: "size_actual": 4096, 2023-10-05T19:28:31.812 INFO:teuthology.orchestra.run.smithi167.stdout: "size_utilized": 3, 2023-10-05T19:28:31.812 INFO:teuthology.orchestra.run.smithi167.stdout: "size_kb": 1, 2023-10-05T19:28:31.812 INFO:teuthology.orchestra.run.smithi167.stdout: "size_kb_actual": 4, 2023-10-05T19:28:31.812 INFO:teuthology.orchestra.run.smithi167.stdout: "size_kb_utilized": 1, 2023-10-05T19:28:31.812 INFO:teuthology.orchestra.run.smithi167.stdout: "num_objects": 1 2023-10-05T19:28:31.813 INFO:teuthology.orchestra.run.smithi167.stdout: } 2023-10-05T19:28:31.813 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2023-10-05T19:28:31.813 INFO:teuthology.orchestra.run.smithi167.stdout: "bucket_quota": { 2023-10-05T19:28:31.813 INFO:teuthology.orchestra.run.smithi167.stdout: "enabled": false, 2023-10-05T19:28:31.813 INFO:teuthology.orchestra.run.smithi167.stdout: "check_on_raw": false, 2023-10-05T19:28:31.813 INFO:teuthology.orchestra.run.smithi167.stdout: "max_size": -1, 2023-10-05T19:28:31.813 INFO:teuthology.orchestra.run.smithi167.stdout: "max_size_kb": 0, 2023-10-05T19:28:31.813 INFO:teuthology.orchestra.run.smithi167.stdout: "max_objects": -1 2023-10-05T19:28:31.813 INFO:teuthology.orchestra.run.smithi167.stdout: } 2023-10-05T19:28:31.813 INFO:teuthology.orchestra.run.smithi167.stdout:} 2023-10-05T19:28:31.814 DEBUG:tasks.util.rgw: json result: {'bucket': 'myfoo', 'num_shards': 11, 'tenant': '', 'zonegroup': '4f1f719d-9648-417b-b10c-e5d939825419', 'placement_rule': 'default-placement', 'explicit_placement': {'data_pool': '', 'data_extra_pool': '', 'index_pool': ''}, 'id': '9a9df965-6612-45d4-a45f-6e3c889bacc2.4223.1', 'marker': '9a9df965-6612-45d4-a45f-6e3c889bacc2.4223.1', 'index_type': 'Normal', 'owner': 'foo', 'ver': '0#1,1#1,2#1,3#1,4#1,5#2,6#1,7#1,8#1,9#1,10#1', 'master_ver': '0#0,1#0,2#0,3#0,4#0,5#0,6#0,7#0,8#0,9#0,10#0', 'mtime': '2023-10-05T19:28:30.258873Z', 'creation_time': '2023-10-05T19:28:30.250836Z', 'max_marker': '0#,1#,2#,3#,4#,5#,6#,7#,8#,9#,10#', 'usage': {'rgw.main': {'size': 3, 'size_actual': 4096, 'size_utilized': 3, 'size_kb': 1, 'size_kb_actual': 4, 'size_kb_utilized': 1, 'num_objects': 1}}, 'bucket_quota': {'enabled': False, 'check_on_raw': False, 'max_size': -1, 'max_size_kb': 0, 'max_objects': -1}} 2023-10-05T19:28:31.814 INFO:tasks.util.rgw:rgwadmin: client.0 : ['user', 'stats', '--uid', 'foo', '--sync-stats'] 2023-10-05T19:28:31.815 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'ceph', 'user', 'stats', '--uid', 'foo', '--sync-stats'] 2023-10-05T19:28:31.815 DEBUG:teuthology.orchestra.run.smithi167:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster ceph user stats --uid foo --sync-stats 2023-10-05T19:28:31.851 INFO:teuthology.orchestra.run.smithi167.stderr:ignoring --setuser ceph since I am not root 2023-10-05T19:28:31.852 INFO:teuthology.orchestra.run.smithi167.stderr:ignoring --setgroup ceph since I am not root 2023-10-05T19:28:31.852 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.846+0000 7f183bc7ce40 1 Processor -- start 2023-10-05T19:28:31.853 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.846+0000 7f183bc7ce40 1 -- start start 2023-10-05T19:28:31.853 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.846+0000 7f183bc7ce40 1 --2- >> v2:172.21.15.118:3300/0 conn(0x5641e8a77030 0x5641e8a77400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-05T19:28:31.853 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.846+0000 7f183bc7ce40 1 -- --> v1:172.21.15.118:6789/0 -- auth(proto 0 26 bytes epoch 0) v1 -- 0x5641e89c1bd0 con 0x5641e8a77940 2023-10-05T19:28:31.853 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.846+0000 7f183bc7ce40 1 -- --> v2:172.21.15.118:3300/0 -- mon_getmap magic: 0 v1 -- 0x5641e897eee0 con 0x5641e8a77030 2023-10-05T19:28:31.854 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.846+0000 7f18397b1700 1 --2- >> v2:172.21.15.118:3300/0 conn(0x5641e8a77030 0x5641e8a77400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-05T19:28:31.854 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.846+0000 7f1838fb0700 1 --1- >> v1:172.21.15.118:6789/0 conn(0x5641e8a77940 0x5641e8a783e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.118:6789/0 says I am v1:172.21.15.167:56596/0 (socket says 172.21.15.167:56596) 2023-10-05T19:28:31.854 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.846+0000 7f1838fb0700 1 -- 172.21.15.167:0/3825181075 learned_addr learned my addr 172.21.15.167:0/3825181075 (peer_addr_for_me v1:172.21.15.167:0/0) 2023-10-05T19:28:31.854 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.846+0000 7f18397b1700 1 --2- >> v2:172.21.15.118:3300/0 conn(0x5641e8a77030 0x5641e8a77400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.118:3300/0 says I am v2:172.21.15.167:58272/0 (socket says 172.21.15.167:58272) 2023-10-05T19:28:31.854 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.846+0000 7f1833fff700 1 -- 172.21.15.167:0/3825181075 <== mon.0 v1:172.21.15.118:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 383510971 0 0) 0x5641e89c1bd0 con 0x5641e8a77940 2023-10-05T19:28:31.854 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.846+0000 7f1833fff700 1 -- 172.21.15.167:0/3825181075 --> v1:172.21.15.118:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f181c001e10 con 0x5641e8a77940 2023-10-05T19:28:31.854 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.850+0000 7f18397b1700 1 -- 172.21.15.167:0/3825181075 >> v1:172.21.15.118:6789/0 conn(0x5641e8a77940 legacy=0x5641e8a783e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-05T19:28:31.855 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.850+0000 7f18397b1700 1 -- 172.21.15.167:0/3825181075 --> v2:172.21.15.118:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5641e8a78af0 con 0x5641e8a77030 2023-10-05T19:28:31.856 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.850+0000 7f18397b1700 1 --2- 172.21.15.167:0/3825181075 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5641e8a77030 0x5641e8a77400 secure :-1 s=READY pgs=339 cs=0 l=1 rev1=1 crypto rx=0x7f182c00a520 tx=0x7f182c005d40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a93fde559b3ec003 server_cookie=0 in_seq=0 out_seq=0 2023-10-05T19:28:31.856 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.850+0000 7f1833fff700 1 -- 172.21.15.167:0/3825181075 <== mon.0 v2:172.21.15.118:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f182c00f590 con 0x5641e8a77030 2023-10-05T19:28:31.856 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.850+0000 7f1833fff700 1 -- 172.21.15.167:0/3825181075 <== mon.0 v2:172.21.15.118:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f182c00f6f0 con 0x5641e8a77030 2023-10-05T19:28:31.856 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.850+0000 7f1833fff700 1 -- 172.21.15.167:0/3825181075 <== mon.0 v2:172.21.15.118:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f182c00f880 con 0x5641e8a77030 2023-10-05T19:28:31.856 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.850+0000 7f183bc7ce40 1 -- 172.21.15.167:0/3825181075 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5641e8a77030 msgr2=0x5641e8a77400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-05T19:28:31.856 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.850+0000 7f183bc7ce40 1 --2- 172.21.15.167:0/3825181075 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5641e8a77030 0x5641e8a77400 secure :-1 s=READY pgs=339 cs=0 l=1 rev1=1 crypto rx=0x7f182c00a520 tx=0x7f182c005d40 comp rx=0 tx=0).stop 2023-10-05T19:28:31.856 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.850+0000 7f183bc7ce40 1 -- 172.21.15.167:0/3825181075 shutdown_connections 2023-10-05T19:28:31.856 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.850+0000 7f183bc7ce40 1 --2- 172.21.15.167:0/3825181075 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5641e8a77030 0x5641e8a77400 unknown :-1 s=CLOSED pgs=339 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-05T19:28:31.856 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.850+0000 7f183bc7ce40 1 -- 172.21.15.167:0/3825181075 >> 172.21.15.167:0/3825181075 conn(0x5641e89ea880 msgr2=0x5641e8a75520 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-05T19:28:31.856 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.850+0000 7f183bc7ce40 1 -- 172.21.15.167:0/3825181075 shutdown_connections 2023-10-05T19:28:31.857 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.850+0000 7f183bc7ce40 1 -- 172.21.15.167:0/3825181075 wait complete. 2023-10-05T19:28:31.857 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.850+0000 7f183bc7ce40 1 Processor -- start 2023-10-05T19:28:31.857 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.850+0000 7f183bc7ce40 1 -- start start 2023-10-05T19:28:31.857 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.850+0000 7f183bc7ce40 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5641e8a77940 0x5641e8a95ff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-05T19:28:31.858 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.850+0000 7f183bc7ce40 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x5641e8942e30 con 0x5641e8a77940 2023-10-05T19:28:31.858 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.850+0000 7f18397b1700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5641e8a77940 0x5641e8a95ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-05T19:28:31.858 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.850+0000 7f18397b1700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5641e8a77940 0x5641e8a95ff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.118:3300/0 says I am v2:172.21.15.167:58286/0 (socket says 172.21.15.167:58286) 2023-10-05T19:28:31.858 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.850+0000 7f18397b1700 1 -- 172.21.15.167:0/2782362980 learned_addr learned my addr 172.21.15.167:0/2782362980 (peer_addr_for_me v2:172.21.15.167:0/0) 2023-10-05T19:28:31.859 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.850+0000 7f18397b1700 1 -- 172.21.15.167:0/2782362980 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f182c00a0e0 con 0x5641e8a77940 2023-10-05T19:28:31.859 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.854+0000 7f18397b1700 1 --2- 172.21.15.167:0/2782362980 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5641e8a77940 0x5641e8a95ff0 secure :-1 s=READY pgs=340 cs=0 l=1 rev1=1 crypto rx=0x7f182c005ad0 tx=0x7f182c014000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=265f5d75e465942 server_cookie=0 in_seq=0 out_seq=0 2023-10-05T19:28:31.859 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.854+0000 7f1832ffd700 1 -- 172.21.15.167:0/2782362980 <== mon.0 v2:172.21.15.118:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f182c00f860 con 0x5641e8a77940 2023-10-05T19:28:31.859 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.854+0000 7f1832ffd700 1 -- 172.21.15.167:0/2782362980 <== mon.0 v2:172.21.15.118:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f182c00f9c0 con 0x5641e8a77940 2023-10-05T19:28:31.859 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.854+0000 7f1832ffd700 1 -- 172.21.15.167:0/2782362980 <== mon.0 v2:172.21.15.118:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f182c00fb50 con 0x5641e8a77940 2023-10-05T19:28:31.860 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.854+0000 7f183bc7ce40 1 -- 172.21.15.167:0/2782362980 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5641e8a77940 msgr2=0x5641e8a95ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-05T19:28:31.860 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.854+0000 7f183bc7ce40 1 --2- 172.21.15.167:0/2782362980 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5641e8a77940 0x5641e8a95ff0 secure :-1 s=READY pgs=340 cs=0 l=1 rev1=1 crypto rx=0x7f182c005ad0 tx=0x7f182c014000 comp rx=0 tx=0).stop 2023-10-05T19:28:31.860 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.854+0000 7f183bc7ce40 1 -- 172.21.15.167:0/2782362980 shutdown_connections 2023-10-05T19:28:31.860 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.854+0000 7f183bc7ce40 1 --2- 172.21.15.167:0/2782362980 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5641e8a77940 0x5641e8a95ff0 unknown :-1 s=CLOSED pgs=340 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-05T19:28:31.861 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.854+0000 7f183bc7ce40 1 -- 172.21.15.167:0/2782362980 >> 172.21.15.167:0/2782362980 conn(0x5641e89ea880 msgr2=0x5641e8a91bb0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-05T19:28:31.861 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.854+0000 7f183bc7ce40 1 -- 172.21.15.167:0/2782362980 shutdown_connections 2023-10-05T19:28:31.861 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.854+0000 7f183bc7ce40 1 -- 172.21.15.167:0/2782362980 wait complete. 2023-10-05T19:28:31.861 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.854+0000 7f183bc7ce40 1 Processor -- start 2023-10-05T19:28:31.861 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.854+0000 7f183bc7ce40 1 -- start start 2023-10-05T19:28:31.861 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.854+0000 7f183bc7ce40 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5641e8a9cd90 0x5641e8a9d160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-05T19:28:31.861 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.854+0000 7f183bc7ce40 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x5641e8915ed0 con 0x5641e8a9cd90 2023-10-05T19:28:31.861 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.854+0000 7f18397b1700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5641e8a9cd90 0x5641e8a9d160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-05T19:28:31.861 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.854+0000 7f18397b1700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5641e8a9cd90 0x5641e8a9d160 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.118:3300/0 says I am v2:172.21.15.167:58296/0 (socket says 172.21.15.167:58296) 2023-10-05T19:28:31.861 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.854+0000 7f18397b1700 1 -- 172.21.15.167:0/3125065775 learned_addr learned my addr 172.21.15.167:0/3125065775 (peer_addr_for_me v2:172.21.15.167:0/0) 2023-10-05T19:28:31.862 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.854+0000 7f18397b1700 1 -- 172.21.15.167:0/3125065775 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5641e8a78af0 con 0x5641e8a9cd90 2023-10-05T19:28:31.862 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.854+0000 7f18397b1700 1 --2- 172.21.15.167:0/3125065775 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5641e8a9cd90 0x5641e8a9d160 secure :-1 s=READY pgs=341 cs=0 l=1 rev1=1 crypto rx=0x7f182c0058e0 tx=0x7f182c01d000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-05T19:28:31.863 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.858+0000 7f18317fa700 1 -- 172.21.15.167:0/3125065775 <== mon.0 v2:172.21.15.118:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f182c01ce10 con 0x5641e8a9cd90 2023-10-05T19:28:31.863 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.858+0000 7f18317fa700 1 -- 172.21.15.167:0/3125065775 <== mon.0 v2:172.21.15.118:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f182c02f430 con 0x5641e8a9cd90 2023-10-05T19:28:31.863 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.858+0000 7f183bc7ce40 1 -- 172.21.15.167:0/3125065775 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x5641e8a9d6a0 con 0x5641e8a9cd90 2023-10-05T19:28:31.863 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.858+0000 7f18317fa700 1 -- 172.21.15.167:0/3125065775 <== mon.0 v2:172.21.15.118:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f182c02f5e0 con 0x5641e8a9cd90 2023-10-05T19:28:31.863 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.858+0000 7f183bc7ce40 1 -- 172.21.15.167:0/3125065775 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x5641e8aa0260 con 0x5641e8a9cd90 2023-10-05T19:28:31.863 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.858+0000 7f183bc7ce40 20 rados->read ofs=0 len=0 2023-10-05T19:28:31.864 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.858+0000 7f18317fa700 1 -- 172.21.15.167:0/3125065775 <== mon.0 v2:172.21.15.118:3300/0 4 ==== mgrmap(e 5) v1 ==== 40329+0+0 (secure 0 0 0) 0x7f182c010420 con 0x5641e8a9cd90 2023-10-05T19:28:31.864 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.858+0000 7f18317fa700 1 --2- 172.21.15.167:0/3125065775 >> [v2:172.21.15.167:6824/29889,v1:172.21.15.167:6825/29889] conn(0x7f1820030c50 0x7f1820033100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-05T19:28:31.864 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.858+0000 7f18317fa700 1 -- 172.21.15.167:0/3125065775 <== mon.0 v2:172.21.15.118:3300/0 5 ==== osd_map(30..30 src has 1..30) v4 ==== 6057+0+0 (secure 0 0 0) 0x7f182c02b070 con 0x5641e8a9cd90 2023-10-05T19:28:31.865 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.858+0000 7f1838fb0700 1 --2- 172.21.15.167:0/3125065775 >> [v2:172.21.15.167:6824/29889,v1:172.21.15.167:6825/29889] conn(0x7f1820030c50 0x7f1820033100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-05T19:28:31.865 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.858+0000 7f183bc7ce40 1 --2- 172.21.15.167:0/3125065775 >> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] conn(0x5641e8aa6ca0 0x5641e8aa9080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-05T19:28:31.865 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.858+0000 7f183bc7ce40 1 -- 172.21.15.167:0/3125065775 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:1 5.12 5:49953fa1:::default.realm:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x5641e8aa9690 con 0x5641e8aa6ca0 2023-10-05T19:28:31.865 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.858+0000 7f1839fb2700 1 --2- 172.21.15.167:0/3125065775 >> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] conn(0x5641e8aa6ca0 0x5641e8aa9080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-05T19:28:31.866 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.858+0000 7f1838fb0700 1 --2- 172.21.15.167:0/3125065775 >> [v2:172.21.15.167:6824/29889,v1:172.21.15.167:6825/29889] conn(0x7f1820030c50 0x7f1820033100 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7f1824004170 tx=0x7f1824001400 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-05T19:28:31.866 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.862+0000 7f1839fb2700 1 --2- 172.21.15.167:0/3125065775 >> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] conn(0x5641e8aa6ca0 0x5641e8aa9080 crc :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-05T19:28:31.867 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.862+0000 7f1839fb2700 1 -- 172.21.15.167:0/3125065775 <== osd.1 v2:172.21.15.167:6800/30035 1 ==== osd_op_reply(1 default.realm [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 ==== 157+0+0 (crc 0 0 0) 0x7f183400a500 con 0x5641e8aa6ca0 2023-10-05T19:28:31.867 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.862+0000 7f183bc7ce40 20 rados_obj.operate() r=-2 bl.length=0 2023-10-05T19:28:31.867 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.862+0000 7f183bc7ce40 20 realm 2023-10-05T19:28:31.867 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.862+0000 7f183bc7ce40 20 rados->read ofs=0 len=0 2023-10-05T19:28:31.868 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.862+0000 7f183bc7ce40 1 -- 172.21.15.167:0/3125065775 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:2 5.12 5:49953fa1:::default.realm:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x5641e8aa6310 con 0x5641e8aa6ca0 2023-10-05T19:28:31.868 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.862+0000 7f1839fb2700 1 -- 172.21.15.167:0/3125065775 <== osd.1 v2:172.21.15.167:6800/30035 2 ==== osd_op_reply(2 default.realm [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 ==== 157+0+0 (crc 0 0 0) 0x7f183400a500 con 0x5641e8aa6ca0 2023-10-05T19:28:31.868 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.862+0000 7f183bc7ce40 20 rados_obj.operate() r=-2 bl.length=0 2023-10-05T19:28:31.868 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.862+0000 7f183bc7ce40 4 RGWPeriod::init failed to init realm id : (2) No such file or directory 2023-10-05T19:28:31.868 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.862+0000 7f183bc7ce40 20 rados->read ofs=0 len=0 2023-10-05T19:28:31.868 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.862+0000 7f183bc7ce40 1 -- 172.21.15.167:0/3125065775 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:3 5.12 5:49953fa1:::default.realm:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x5641e8aaa190 con 0x5641e8aa6ca0 2023-10-05T19:28:31.868 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.862+0000 7f1839fb2700 1 -- 172.21.15.167:0/3125065775 <== osd.1 v2:172.21.15.167:6800/30035 3 ==== osd_op_reply(3 default.realm [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 ==== 157+0+0 (crc 0 0 0) 0x7f183400a500 con 0x5641e8aa6ca0 2023-10-05T19:28:31.868 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.862+0000 7f183bc7ce40 20 rados_obj.operate() r=-2 bl.length=0 2023-10-05T19:28:31.868 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.862+0000 7f183bc7ce40 20 rados->read ofs=0 len=0 2023-10-05T19:28:31.868 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.862+0000 7f183bc7ce40 1 -- 172.21.15.167:0/3125065775 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:4 5.f 5:f43b6ece:::zone_names.default:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x5641e8aa9d70 con 0x5641e8aa6ca0 2023-10-05T19:28:31.870 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.862+0000 7f1839fb2700 1 -- 172.21.15.167:0/3125065775 <== osd.1 v2:172.21.15.167:6800/30035 4 ==== osd_op_reply(4 zone_names.default [read 0~46 out=46b] v0'0 uv1 ondisk = 0) v8 ==== 162+0+46 (crc 0 0 0) 0x7f183400a500 con 0x5641e8aa6ca0 2023-10-05T19:28:31.870 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.862+0000 7f183bc7ce40 20 rados_obj.operate() r=0 bl.length=46 2023-10-05T19:28:31.870 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.862+0000 7f183bc7ce40 20 rados->read ofs=0 len=0 2023-10-05T19:28:31.870 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.862+0000 7f183bc7ce40 1 -- 172.21.15.167:0/3125065775 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:5 5.1d 5:b8374eb4:::zone_info.9a9df965-6612-45d4-a45f-6e3c889bacc2:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x5641e8aaac00 con 0x5641e8aa6ca0 2023-10-05T19:28:31.870 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.862+0000 7f1839fb2700 1 -- 172.21.15.167:0/3125065775 <== osd.1 v2:172.21.15.167:6800/30035 5 ==== osd_op_reply(5 zone_info.9a9df965-6612-45d4-a45f-6e3c889bacc2 [read 0~874 out=874b] v0'0 uv2 ondisk = 0) v8 ==== 190+0+874 (crc 0 0 0) 0x7f183400a500 con 0x5641e8aa6ca0 2023-10-05T19:28:31.870 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.862+0000 7f183bc7ce40 20 rados_obj.operate() r=0 bl.length=874 2023-10-05T19:28:31.870 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.862+0000 7f183bc7ce40 20 searching for the correct realm 2023-10-05T19:28:31.870 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.862+0000 7f183bc7ce40 1 --2- 172.21.15.167:0/3125065775 >> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] conn(0x5641e8aaca50 0x5641e8aaee30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-05T19:28:31.870 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.862+0000 7f183bc7ce40 1 -- 172.21.15.167:0/3125065775 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:6 5.0 5:00000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x5641e8aaf370 con 0x5641e8aaca50 2023-10-05T19:28:31.870 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.862+0000 7f18397b1700 1 --2- 172.21.15.167:0/3125065775 >> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] conn(0x5641e8aaca50 0x5641e8aaee30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-05T19:28:31.871 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.862+0000 7f18397b1700 1 --2- 172.21.15.167:0/3125065775 >> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] conn(0x5641e8aaca50 0x5641e8aaee30 crc :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-05T19:28:31.871 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.866+0000 7f18397b1700 1 -- 172.21.15.167:0/3125065775 <== osd.3 v2:172.21.15.167:6808/30036 1 ==== osd_op_reply(6 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f182c0387d0 con 0x5641e8aaca50 2023-10-05T19:28:31.871 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.866+0000 7f18397b1700 1 --2- 172.21.15.167:0/3125065775 >> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] conn(0x5641e8a77940 0x7f182c0289f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-05T19:28:31.871 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.866+0000 7f18397b1700 1 -- 172.21.15.167:0/3125065775 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:7 5.10 5:08000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x5641e8aaf370 con 0x5641e8a77940 2023-10-05T19:28:31.872 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.866+0000 7f1838fb0700 1 --2- 172.21.15.167:0/3125065775 >> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] conn(0x5641e8a77940 0x7f182c0289f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-05T19:28:31.872 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.866+0000 7f1838fb0700 1 --2- 172.21.15.167:0/3125065775 >> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] conn(0x5641e8a77940 0x7f182c0289f0 crc :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-05T19:28:31.873 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.866+0000 7f1838fb0700 1 -- 172.21.15.167:0/3125065775 <== osd.2 v2:172.21.15.167:6816/30037 1 ==== osd_op_reply(7 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f1824006070 con 0x5641e8a77940 2023-10-05T19:28:31.873 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.866+0000 7f1838fb0700 1 --2- 172.21.15.167:0/3125065775 >> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] conn(0x7f182400eb60 0x7f182400d8e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-05T19:28:31.873 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.866+0000 7f1838fb0700 1 -- 172.21.15.167:0/3125065775 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:8 5.8 5:10000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x5641e8aaf370 con 0x7f182400eb60 2023-10-05T19:28:31.873 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.866+0000 7f1839fb2700 1 --2- 172.21.15.167:0/3125065775 >> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] conn(0x7f182400eb60 0x7f182400d8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-05T19:28:31.874 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.870+0000 7f1839fb2700 1 --2- 172.21.15.167:0/3125065775 >> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] conn(0x7f182400eb60 0x7f182400d8e0 crc :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-05T19:28:31.875 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.870+0000 7f1839fb2700 1 -- 172.21.15.167:0/3125065775 <== osd.0 v2:172.21.15.118:6800/27690 1 ==== osd_op_reply(8 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f1834014040 con 0x7f182400eb60 2023-10-05T19:28:31.875 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.870+0000 7f1839fb2700 1 -- 172.21.15.167:0/3125065775 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:9 5.18 5:18000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x5641e8aaf370 con 0x5641e8aa6ca0 2023-10-05T19:28:31.875 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.870+0000 7f1839fb2700 1 -- 172.21.15.167:0/3125065775 <== osd.1 v2:172.21.15.167:6800/30035 6 ==== osd_op_reply(9 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f1834014040 con 0x5641e8aa6ca0 2023-10-05T19:28:31.875 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.870+0000 7f1839fb2700 1 -- 172.21.15.167:0/3125065775 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:10 5.4 5:20000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x5641e8aaf370 con 0x5641e8aa6ca0 2023-10-05T19:28:31.876 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.870+0000 7f1839fb2700 1 -- 172.21.15.167:0/3125065775 <== osd.1 v2:172.21.15.167:6800/30035 7 ==== osd_op_reply(10 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f1834014040 con 0x5641e8aa6ca0 2023-10-05T19:28:31.876 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.870+0000 7f1839fb2700 1 -- 172.21.15.167:0/3125065775 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:11 5.14 5:28000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x5641e8aaf370 con 0x7f182400eb60 2023-10-05T19:28:31.877 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.870+0000 7f1839fb2700 1 -- 172.21.15.167:0/3125065775 <== osd.0 v2:172.21.15.118:6800/27690 2 ==== osd_op_reply(11 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f1834014040 con 0x7f182400eb60 2023-10-05T19:28:31.877 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.870+0000 7f1839fb2700 1 -- 172.21.15.167:0/3125065775 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:12 5.c 5:30000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x5641e8aaf370 con 0x7f182400eb60 2023-10-05T19:28:31.878 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.874+0000 7f1839fb2700 1 -- 172.21.15.167:0/3125065775 <== osd.0 v2:172.21.15.118:6800/27690 3 ==== osd_op_reply(12 [pgnls start_epoch 30 out=112b] v23'1 uv1 ondisk = 1) v8 ==== 144+0+112 (crc 0 0 0) 0x7f1834014040 con 0x7f182400eb60 2023-10-05T19:28:31.878 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.874+0000 7f1839fb2700 1 -- 172.21.15.167:0/3125065775 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:13 5.1c 5:38000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x5641e8aaf370 con 0x5641e8a77940 2023-10-05T19:28:31.878 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.874+0000 7f1838fb0700 1 -- 172.21.15.167:0/3125065775 <== osd.2 v2:172.21.15.167:6816/30037 2 ==== osd_op_reply(13 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f1824006070 con 0x5641e8a77940 2023-10-05T19:28:31.879 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.874+0000 7f1838fb0700 1 -- 172.21.15.167:0/3125065775 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:14 5.2 5:40000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x5641e8aaf370 con 0x5641e8aaca50 2023-10-05T19:28:31.879 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.874+0000 7f18397b1700 1 -- 172.21.15.167:0/3125065775 <== osd.3 v2:172.21.15.167:6808/30036 2 ==== osd_op_reply(14 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f182c0387d0 con 0x5641e8aaca50 2023-10-05T19:28:31.879 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.874+0000 7f18397b1700 1 -- 172.21.15.167:0/3125065775 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:15 5.12 5:48000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x5641e8aaf370 con 0x5641e8aa6ca0 2023-10-05T19:28:31.880 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.874+0000 7f1839fb2700 1 -- 172.21.15.167:0/3125065775 <== osd.1 v2:172.21.15.167:6800/30035 8 ==== osd_op_reply(15 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f1834014040 con 0x5641e8aa6ca0 2023-10-05T19:28:31.880 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.874+0000 7f1839fb2700 1 -- 172.21.15.167:0/3125065775 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:16 5.a 5:50000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x5641e8aaf370 con 0x7f182400eb60 2023-10-05T19:28:31.881 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.874+0000 7f1839fb2700 1 -- 172.21.15.167:0/3125065775 <== osd.0 v2:172.21.15.118:6800/27690 4 ==== osd_op_reply(16 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f1834022040 con 0x7f182400eb60 2023-10-05T19:28:31.881 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.874+0000 7f1839fb2700 1 -- 172.21.15.167:0/3125065775 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:17 5.1a 5:58000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x5641e8aaf370 con 0x5641e8aa6ca0 2023-10-05T19:28:31.881 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.874+0000 7f1839fb2700 1 -- 172.21.15.167:0/3125065775 <== osd.1 v2:172.21.15.167:6800/30035 9 ==== osd_op_reply(17 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f1834022040 con 0x5641e8aa6ca0 2023-10-05T19:28:31.882 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.874+0000 7f1839fb2700 1 -- 172.21.15.167:0/3125065775 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:18 5.6 5:60000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x5641e8aaf370 con 0x5641e8a77940 2023-10-05T19:28:31.882 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.878+0000 7f1838fb0700 1 -- 172.21.15.167:0/3125065775 <== osd.2 v2:172.21.15.167:6816/30037 3 ==== osd_op_reply(18 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f1824006070 con 0x5641e8a77940 2023-10-05T19:28:31.882 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.878+0000 7f1838fb0700 1 -- 172.21.15.167:0/3125065775 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:19 5.16 5:68000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x5641e8aaf370 con 0x5641e8aaca50 2023-10-05T19:28:31.883 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.878+0000 7f18397b1700 1 -- 172.21.15.167:0/3125065775 <== osd.3 v2:172.21.15.167:6808/30036 3 ==== osd_op_reply(19 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f182c0387d0 con 0x5641e8aaca50 2023-10-05T19:28:31.883 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.878+0000 7f18397b1700 1 -- 172.21.15.167:0/3125065775 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:20 5.e 5:70000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x5641e8aaf370 con 0x5641e8aaca50 2023-10-05T19:28:31.883 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.878+0000 7f18397b1700 1 -- 172.21.15.167:0/3125065775 <== osd.3 v2:172.21.15.167:6808/30036 4 ==== osd_op_reply(20 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f182c0387d0 con 0x5641e8aaca50 2023-10-05T19:28:31.884 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.878+0000 7f18397b1700 1 -- 172.21.15.167:0/3125065775 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:21 5.1e 5:78000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x5641e8aaf370 con 0x7f182400eb60 2023-10-05T19:28:31.884 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.878+0000 7f1839fb2700 1 -- 172.21.15.167:0/3125065775 <== osd.0 v2:172.21.15.118:6800/27690 5 ==== osd_op_reply(21 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f1834022040 con 0x7f182400eb60 2023-10-05T19:28:31.885 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.878+0000 7f1839fb2700 1 -- 172.21.15.167:0/3125065775 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:22 5.1 5:80000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x5641e8aaf370 con 0x5641e8aaca50 2023-10-05T19:28:31.885 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.878+0000 7f18397b1700 1 -- 172.21.15.167:0/3125065775 <== osd.3 v2:172.21.15.167:6808/30036 5 ==== osd_op_reply(22 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f182c0387d0 con 0x5641e8aaca50 2023-10-05T19:28:31.885 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.878+0000 7f18397b1700 1 -- 172.21.15.167:0/3125065775 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:23 5.11 5:88000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x5641e8aaf370 con 0x5641e8aaca50 2023-10-05T19:28:31.886 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.882+0000 7f18397b1700 1 -- 172.21.15.167:0/3125065775 <== osd.3 v2:172.21.15.167:6808/30036 6 ==== osd_op_reply(23 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f182c0387d0 con 0x5641e8aaca50 2023-10-05T19:28:31.886 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.882+0000 7f18397b1700 1 -- 172.21.15.167:0/3125065775 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:24 5.9 5:90000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x5641e8aaf370 con 0x5641e8aa6ca0 2023-10-05T19:28:31.887 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.882+0000 7f1839fb2700 1 -- 172.21.15.167:0/3125065775 <== osd.1 v2:172.21.15.167:6800/30035 10 ==== osd_op_reply(24 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f1834022040 con 0x5641e8aa6ca0 2023-10-05T19:28:31.887 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.882+0000 7f1839fb2700 1 -- 172.21.15.167:0/3125065775 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:25 5.19 5:98000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x5641e8aaf370 con 0x5641e8aa6ca0 2023-10-05T19:28:31.887 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.882+0000 7f1839fb2700 1 -- 172.21.15.167:0/3125065775 <== osd.1 v2:172.21.15.167:6800/30035 11 ==== osd_op_reply(25 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f1834022040 con 0x5641e8aa6ca0 2023-10-05T19:28:31.887 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.882+0000 7f1839fb2700 1 -- 172.21.15.167:0/3125065775 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:26 5.5 5:a0000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x5641e8aaf370 con 0x7f182400eb60 2023-10-05T19:28:31.888 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.882+0000 7f1839fb2700 1 -- 172.21.15.167:0/3125065775 <== osd.0 v2:172.21.15.118:6800/27690 6 ==== osd_op_reply(26 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f1834022040 con 0x7f182400eb60 2023-10-05T19:28:31.889 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.882+0000 7f1839fb2700 1 -- 172.21.15.167:0/3125065775 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:27 5.15 5:a8000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x5641e8aaf370 con 0x5641e8aaca50 2023-10-05T19:28:31.889 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.882+0000 7f18397b1700 1 -- 172.21.15.167:0/3125065775 <== osd.3 v2:172.21.15.167:6808/30036 7 ==== osd_op_reply(27 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f182c0387d0 con 0x5641e8aaca50 2023-10-05T19:28:31.889 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.882+0000 7f18397b1700 1 -- 172.21.15.167:0/3125065775 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:28 5.d 5:b0000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x5641e8aaf370 con 0x5641e8a77940 2023-10-05T19:28:31.890 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.886+0000 7f1838fb0700 1 -- 172.21.15.167:0/3125065775 <== osd.2 v2:172.21.15.167:6816/30037 4 ==== osd_op_reply(28 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f1824006070 con 0x5641e8a77940 2023-10-05T19:28:31.890 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.886+0000 7f1838fb0700 1 -- 172.21.15.167:0/3125065775 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:29 5.1d 5:b8000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x5641e8aaf370 con 0x5641e8aa6ca0 2023-10-05T19:28:31.891 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.886+0000 7f1839fb2700 1 -- 172.21.15.167:0/3125065775 <== osd.1 v2:172.21.15.167:6800/30035 12 ==== osd_op_reply(29 [pgnls start_epoch 30 out=107b] v30'2 uv2 ondisk = 1) v8 ==== 144+0+107 (crc 0 0 0) 0x7f1834022040 con 0x5641e8aa6ca0 2023-10-05T19:28:31.891 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.886+0000 7f1839fb2700 1 -- 172.21.15.167:0/3125065775 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:30 5.3 5:c0000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x5641e8aaf370 con 0x7f182400eb60 2023-10-05T19:28:31.892 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.886+0000 7f1839fb2700 1 -- 172.21.15.167:0/3125065775 <== osd.0 v2:172.21.15.118:6800/27690 7 ==== osd_op_reply(30 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f1834022040 con 0x7f182400eb60 2023-10-05T19:28:31.892 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.886+0000 7f1839fb2700 1 -- 172.21.15.167:0/3125065775 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:31 5.13 5:c8000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x5641e8aaf370 con 0x5641e8aaca50 2023-10-05T19:28:31.893 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.886+0000 7f18397b1700 1 -- 172.21.15.167:0/3125065775 <== osd.3 v2:172.21.15.167:6808/30036 8 ==== osd_op_reply(31 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f182c04e040 con 0x5641e8aaca50 2023-10-05T19:28:31.893 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.886+0000 7f18397b1700 1 -- 172.21.15.167:0/3125065775 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:32 5.b 5:d0000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x5641e8aaf370 con 0x7f182400eb60 2023-10-05T19:28:31.894 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.890+0000 7f1839fb2700 1 -- 172.21.15.167:0/3125065775 <== osd.0 v2:172.21.15.118:6800/27690 8 ==== osd_op_reply(32 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f1834022040 con 0x7f182400eb60 2023-10-05T19:28:31.894 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.890+0000 7f1839fb2700 1 -- 172.21.15.167:0/3125065775 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:33 5.1b 5:d8000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x5641e8aaf370 con 0x5641e8a77940 2023-10-05T19:28:31.894 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.890+0000 7f1838fb0700 1 -- 172.21.15.167:0/3125065775 <== osd.2 v2:172.21.15.167:6816/30037 5 ==== osd_op_reply(33 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f1824006070 con 0x5641e8a77940 2023-10-05T19:28:31.895 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.890+0000 7f1838fb0700 1 -- 172.21.15.167:0/3125065775 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:34 5.7 5:e0000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x5641e8aaf370 con 0x5641e8aaca50 2023-10-05T19:28:31.895 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.890+0000 7f18397b1700 1 -- 172.21.15.167:0/3125065775 <== osd.3 v2:172.21.15.167:6808/30036 9 ==== osd_op_reply(34 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f182c04e040 con 0x5641e8aaca50 2023-10-05T19:28:31.895 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.890+0000 7f18397b1700 1 -- 172.21.15.167:0/3125065775 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:35 5.17 5:e8000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x5641e8aaf370 con 0x7f182400eb60 2023-10-05T19:28:31.896 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.890+0000 7f1839fb2700 1 -- 172.21.15.167:0/3125065775 <== osd.0 v2:172.21.15.118:6800/27690 9 ==== osd_op_reply(35 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f1834022040 con 0x7f182400eb60 2023-10-05T19:28:31.896 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.890+0000 7f1839fb2700 1 -- 172.21.15.167:0/3125065775 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:36 5.f 5:f0000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x5641e8aaf370 con 0x5641e8aa6ca0 2023-10-05T19:28:31.897 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.890+0000 7f1839fb2700 1 -- 172.21.15.167:0/3125065775 <== osd.1 v2:172.21.15.167:6800/30035 13 ==== osd_op_reply(36 [pgnls start_epoch 30 out=115b] v23'2 uv2 ondisk = 1) v8 ==== 144+0+115 (crc 0 0 0) 0x7f183402f080 con 0x5641e8aa6ca0 2023-10-05T19:28:31.897 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.890+0000 7f1839fb2700 1 -- 172.21.15.167:0/3125065775 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:37 5.1f 5:f8000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x5641e8aaf370 con 0x5641e8a77940 2023-10-05T19:28:31.898 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.894+0000 7f1838fb0700 1 -- 172.21.15.167:0/3125065775 <== osd.2 v2:172.21.15.167:6816/30037 6 ==== osd_op_reply(37 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f1824006070 con 0x5641e8a77940 2023-10-05T19:28:31.898 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.894+0000 7f183bc7ce40 20 RGWRados::pool_iterate: got zonegroup_info.4f1f719d-9648-417b-b10c-e5d939825419 2023-10-05T19:28:31.898 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.894+0000 7f183bc7ce40 20 RGWRados::pool_iterate: got zone_info.9a9df965-6612-45d4-a45f-6e3c889bacc2 2023-10-05T19:28:31.898 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.894+0000 7f183bc7ce40 20 RGWRados::pool_iterate: got zone_names.default 2023-10-05T19:28:31.899 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.894+0000 7f183bc7ce40 20 RGWRados::pool_iterate: got zonegroups_names.default 2023-10-05T19:28:31.899 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.894+0000 7f183bc7ce40 20 rados->read ofs=0 len=0 2023-10-05T19:28:31.899 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.894+0000 7f183bc7ce40 1 -- 172.21.15.167:0/3125065775 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:38 5.12 5:49953fa1:::default.realm:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x5641e8aab800 con 0x5641e8aa6ca0 2023-10-05T19:28:31.899 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.894+0000 7f1839fb2700 1 -- 172.21.15.167:0/3125065775 <== osd.1 v2:172.21.15.167:6800/30035 14 ==== osd_op_reply(38 default.realm [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 ==== 157+0+0 (crc 0 0 0) 0x7f183402f080 con 0x5641e8aa6ca0 2023-10-05T19:28:31.900 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.894+0000 7f183bc7ce40 20 rados_obj.operate() r=-2 bl.length=0 2023-10-05T19:28:31.900 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.894+0000 7f183bc7ce40 20 rados->read ofs=0 len=0 2023-10-05T19:28:31.900 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.894+0000 7f183bc7ce40 1 -- 172.21.15.167:0/3125065775 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:39 5.f 5:f4c53578:::zonegroups_names.default:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x5641e8aac180 con 0x5641e8aa6ca0 2023-10-05T19:28:31.900 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.894+0000 7f1839fb2700 1 -- 172.21.15.167:0/3125065775 <== osd.1 v2:172.21.15.167:6800/30035 15 ==== osd_op_reply(39 zonegroups_names.default [read 0~46 out=46b] v0'0 uv2 ondisk = 0) v8 ==== 168+0+46 (crc 0 0 0) 0x7f183402f080 con 0x5641e8aa6ca0 2023-10-05T19:28:31.900 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.894+0000 7f183bc7ce40 20 rados_obj.operate() r=0 bl.length=46 2023-10-05T19:28:31.900 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.894+0000 7f183bc7ce40 20 rados->read ofs=0 len=0 2023-10-05T19:28:31.900 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.894+0000 7f183bc7ce40 1 -- 172.21.15.167:0/3125065775 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:40 5.c 5:30d299a5:::zonegroup_info.4f1f719d-9648-417b-b10c-e5d939825419:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x5641e8aab3f0 con 0x7f182400eb60 2023-10-05T19:28:31.901 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.894+0000 7f1839fb2700 1 -- 172.21.15.167:0/3125065775 <== osd.0 v2:172.21.15.118:6800/27690 10 ==== osd_op_reply(40 zonegroup_info.4f1f719d-9648-417b-b10c-e5d939825419 [read 0~362 out=362b] v0'0 uv1 ondisk = 0) v8 ==== 195+0+362 (crc 0 0 0) 0x7f183402f080 con 0x7f182400eb60 2023-10-05T19:28:31.901 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.894+0000 7f183bc7ce40 20 rados_obj.operate() r=0 bl.length=362 2023-10-05T19:28:31.901 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.894+0000 7f183bc7ce40 20 rados->read ofs=0 len=0 2023-10-05T19:28:31.901 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.894+0000 7f183bc7ce40 1 -- 172.21.15.167:0/3125065775 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:41 5.13 5:cbbf6c60:::converted:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x5641e8aafc60 con 0x5641e8aaca50 2023-10-05T19:28:31.902 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.894+0000 7f18397b1700 1 -- 172.21.15.167:0/3125065775 <== osd.3 v2:172.21.15.167:6808/30036 10 ==== osd_op_reply(41 converted [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 ==== 153+0+0 (crc 0 0 0) 0x7f182c04e040 con 0x5641e8aaca50 2023-10-05T19:28:31.902 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.894+0000 7f183bc7ce40 20 rados_obj.operate() r=-2 bl.length=0 2023-10-05T19:28:31.902 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.894+0000 7f183bc7ce40 20 rados->read ofs=0 len=0 2023-10-05T19:28:31.902 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.894+0000 7f183bc7ce40 1 -- 172.21.15.167:0/3125065775 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:42 5.12 5:49953fa1:::default.realm:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x5641e8ab0010 con 0x5641e8aa6ca0 2023-10-05T19:28:31.903 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.898+0000 7f1839fb2700 1 -- 172.21.15.167:0/3125065775 <== osd.1 v2:172.21.15.167:6800/30035 16 ==== osd_op_reply(42 default.realm [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 ==== 157+0+0 (crc 0 0 0) 0x7f183402f080 con 0x5641e8aa6ca0 2023-10-05T19:28:31.903 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.898+0000 7f183bc7ce40 20 rados_obj.operate() r=-2 bl.length=0 2023-10-05T19:28:31.903 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.898+0000 7f183bc7ce40 20 rados->read ofs=0 len=0 2023-10-05T19:28:31.903 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.898+0000 7f183bc7ce40 1 -- 172.21.15.167:0/3125065775 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:43 5.f 5:f4c53578:::zonegroups_names.default:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x5641e8ab0010 con 0x5641e8aa6ca0 2023-10-05T19:28:31.903 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.898+0000 7f1839fb2700 1 -- 172.21.15.167:0/3125065775 <== osd.1 v2:172.21.15.167:6800/30035 17 ==== osd_op_reply(43 zonegroups_names.default [read 0~46 out=46b] v0'0 uv2 ondisk = 0) v8 ==== 168+0+46 (crc 0 0 0) 0x7f183402f080 con 0x5641e8aa6ca0 2023-10-05T19:28:31.903 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.898+0000 7f183bc7ce40 20 rados_obj.operate() r=0 bl.length=46 2023-10-05T19:28:31.903 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.898+0000 7f183bc7ce40 1 -- 172.21.15.167:0/3125065775 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:44 5.0 5:00000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x5641e8ab1020 con 0x5641e8aaca50 2023-10-05T19:28:31.905 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.898+0000 7f18397b1700 1 -- 172.21.15.167:0/3125065775 <== osd.3 v2:172.21.15.167:6808/30036 11 ==== osd_op_reply(44 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f182c04e040 con 0x5641e8aaca50 2023-10-05T19:28:31.905 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.898+0000 7f18397b1700 1 -- 172.21.15.167:0/3125065775 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:45 5.10 5:08000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x5641e8ab1020 con 0x5641e8a77940 2023-10-05T19:28:31.906 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.902+0000 7f1838fb0700 1 -- 172.21.15.167:0/3125065775 <== osd.2 v2:172.21.15.167:6816/30037 7 ==== osd_op_reply(45 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f1824006070 con 0x5641e8a77940 2023-10-05T19:28:31.906 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.902+0000 7f1838fb0700 1 -- 172.21.15.167:0/3125065775 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:46 5.8 5:10000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x5641e8ab1020 con 0x7f182400eb60 2023-10-05T19:28:31.907 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.902+0000 7f1839fb2700 1 -- 172.21.15.167:0/3125065775 <== osd.0 v2:172.21.15.118:6800/27690 11 ==== osd_op_reply(46 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f183402f080 con 0x7f182400eb60 2023-10-05T19:28:31.907 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.902+0000 7f1839fb2700 1 -- 172.21.15.167:0/3125065775 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:47 5.18 5:18000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x5641e8aab3f0 con 0x5641e8aa6ca0 2023-10-05T19:28:31.907 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.902+0000 7f1839fb2700 1 -- 172.21.15.167:0/3125065775 <== osd.1 v2:172.21.15.167:6800/30035 18 ==== osd_op_reply(47 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f183402f080 con 0x5641e8aa6ca0 2023-10-05T19:28:31.908 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.902+0000 7f1839fb2700 1 -- 172.21.15.167:0/3125065775 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:48 5.4 5:20000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x5641e8aab3f0 con 0x5641e8aa6ca0 2023-10-05T19:28:31.908 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.902+0000 7f1839fb2700 1 -- 172.21.15.167:0/3125065775 <== osd.1 v2:172.21.15.167:6800/30035 19 ==== osd_op_reply(48 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f183402f080 con 0x5641e8aa6ca0 2023-10-05T19:28:31.908 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.902+0000 7f1839fb2700 1 -- 172.21.15.167:0/3125065775 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:49 5.14 5:28000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x5641e8aab3f0 con 0x7f182400eb60 2023-10-05T19:28:31.909 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.902+0000 7f1839fb2700 1 -- 172.21.15.167:0/3125065775 <== osd.0 v2:172.21.15.118:6800/27690 12 ==== osd_op_reply(49 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f183402f080 con 0x7f182400eb60 2023-10-05T19:28:31.909 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.902+0000 7f1839fb2700 1 -- 172.21.15.167:0/3125065775 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:50 5.c 5:30000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x5641e8aab3f0 con 0x7f182400eb60 2023-10-05T19:28:31.910 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.906+0000 7f1839fb2700 1 -- 172.21.15.167:0/3125065775 <== osd.0 v2:172.21.15.118:6800/27690 13 ==== osd_op_reply(50 [pgnls start_epoch 30 out=112b] v23'1 uv1 ondisk = 1) v8 ==== 144+0+112 (crc 0 0 0) 0x7f183402f080 con 0x7f182400eb60 2023-10-05T19:28:31.910 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.906+0000 7f1839fb2700 1 -- 172.21.15.167:0/3125065775 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:51 5.1c 5:38000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x5641e8aab3f0 con 0x5641e8a77940 2023-10-05T19:28:31.911 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.906+0000 7f1838fb0700 1 -- 172.21.15.167:0/3125065775 <== osd.2 v2:172.21.15.167:6816/30037 8 ==== osd_op_reply(51 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f1824006070 con 0x5641e8a77940 2023-10-05T19:28:31.911 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.906+0000 7f1838fb0700 1 -- 172.21.15.167:0/3125065775 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:52 5.2 5:40000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x5641e8aab3f0 con 0x5641e8aaca50 2023-10-05T19:28:31.912 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.906+0000 7f18397b1700 1 -- 172.21.15.167:0/3125065775 <== osd.3 v2:172.21.15.167:6808/30036 12 ==== osd_op_reply(52 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f182c04e040 con 0x5641e8aaca50 2023-10-05T19:28:31.912 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.906+0000 7f18397b1700 1 -- 172.21.15.167:0/3125065775 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:53 5.12 5:48000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x5641e8aab3f0 con 0x5641e8aa6ca0 2023-10-05T19:28:31.913 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.906+0000 7f1839fb2700 1 -- 172.21.15.167:0/3125065775 <== osd.1 v2:172.21.15.167:6800/30035 20 ==== osd_op_reply(53 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f183402f080 con 0x5641e8aa6ca0 2023-10-05T19:28:31.913 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.906+0000 7f1839fb2700 1 -- 172.21.15.167:0/3125065775 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:54 5.a 5:50000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x5641e8aab3f0 con 0x7f182400eb60 2023-10-05T19:28:31.914 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.906+0000 7f1839fb2700 1 -- 172.21.15.167:0/3125065775 <== osd.0 v2:172.21.15.118:6800/27690 14 ==== osd_op_reply(54 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f183402f080 con 0x7f182400eb60 2023-10-05T19:28:31.914 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.906+0000 7f1839fb2700 1 -- 172.21.15.167:0/3125065775 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:55 5.1a 5:58000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x5641e8aab3f0 con 0x5641e8aa6ca0 2023-10-05T19:28:31.914 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.910+0000 7f1839fb2700 1 -- 172.21.15.167:0/3125065775 <== osd.1 v2:172.21.15.167:6800/30035 21 ==== osd_op_reply(55 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f183402f080 con 0x5641e8aa6ca0 2023-10-05T19:28:31.914 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.910+0000 7f1839fb2700 1 -- 172.21.15.167:0/3125065775 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:56 5.6 5:60000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x5641e8aab3f0 con 0x5641e8a77940 2023-10-05T19:28:31.915 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.910+0000 7f1838fb0700 1 -- 172.21.15.167:0/3125065775 <== osd.2 v2:172.21.15.167:6816/30037 9 ==== osd_op_reply(56 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f1824006070 con 0x5641e8a77940 2023-10-05T19:28:31.915 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.910+0000 7f1838fb0700 1 -- 172.21.15.167:0/3125065775 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:57 5.16 5:68000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x5641e8aab3f0 con 0x5641e8aaca50 2023-10-05T19:28:31.916 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.910+0000 7f18397b1700 1 -- 172.21.15.167:0/3125065775 <== osd.3 v2:172.21.15.167:6808/30036 13 ==== osd_op_reply(57 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f182c04e040 con 0x5641e8aaca50 2023-10-05T19:28:31.916 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.910+0000 7f18397b1700 1 -- 172.21.15.167:0/3125065775 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:58 5.e 5:70000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x5641e8aab3f0 con 0x5641e8aaca50 2023-10-05T19:28:31.917 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.910+0000 7f18397b1700 1 -- 172.21.15.167:0/3125065775 <== osd.3 v2:172.21.15.167:6808/30036 14 ==== osd_op_reply(58 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f182c04e040 con 0x5641e8aaca50 2023-10-05T19:28:31.917 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.910+0000 7f18397b1700 1 -- 172.21.15.167:0/3125065775 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:59 5.1e 5:78000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x5641e8aab3f0 con 0x7f182400eb60 2023-10-05T19:28:31.918 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.914+0000 7f1839fb2700 1 -- 172.21.15.167:0/3125065775 <== osd.0 v2:172.21.15.118:6800/27690 15 ==== osd_op_reply(59 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f183402f080 con 0x7f182400eb60 2023-10-05T19:28:31.918 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.914+0000 7f1839fb2700 1 -- 172.21.15.167:0/3125065775 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:60 5.1 5:80000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x5641e8aab3f0 con 0x5641e8aaca50 2023-10-05T19:28:31.919 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.914+0000 7f18397b1700 1 -- 172.21.15.167:0/3125065775 <== osd.3 v2:172.21.15.167:6808/30036 15 ==== osd_op_reply(60 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f182c04e040 con 0x5641e8aaca50 2023-10-05T19:28:31.919 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.914+0000 7f18397b1700 1 -- 172.21.15.167:0/3125065775 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:61 5.11 5:88000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x5641e8aab3f0 con 0x5641e8aaca50 2023-10-05T19:28:31.920 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.914+0000 7f18397b1700 1 -- 172.21.15.167:0/3125065775 <== osd.3 v2:172.21.15.167:6808/30036 16 ==== osd_op_reply(61 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f182c04e040 con 0x5641e8aaca50 2023-10-05T19:28:31.920 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.914+0000 7f18397b1700 1 -- 172.21.15.167:0/3125065775 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:62 5.9 5:90000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x5641e8aab3f0 con 0x5641e8aa6ca0 2023-10-05T19:28:31.921 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.914+0000 7f1839fb2700 1 -- 172.21.15.167:0/3125065775 <== osd.1 v2:172.21.15.167:6800/30035 22 ==== osd_op_reply(62 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f183402f080 con 0x5641e8aa6ca0 2023-10-05T19:28:31.921 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.914+0000 7f1839fb2700 1 -- 172.21.15.167:0/3125065775 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:63 5.19 5:98000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x5641e8aab3f0 con 0x5641e8aa6ca0 2023-10-05T19:28:31.921 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.914+0000 7f1839fb2700 1 -- 172.21.15.167:0/3125065775 <== osd.1 v2:172.21.15.167:6800/30035 23 ==== osd_op_reply(63 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f183402f080 con 0x5641e8aa6ca0 2023-10-05T19:28:31.922 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.914+0000 7f1839fb2700 1 -- 172.21.15.167:0/3125065775 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:64 5.5 5:a0000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x5641e8aab3f0 con 0x7f182400eb60 2023-10-05T19:28:31.923 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.918+0000 7f1839fb2700 1 -- 172.21.15.167:0/3125065775 <== osd.0 v2:172.21.15.118:6800/27690 16 ==== osd_op_reply(64 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f183402f080 con 0x7f182400eb60 2023-10-05T19:28:31.923 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.918+0000 7f1839fb2700 1 -- 172.21.15.167:0/3125065775 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:65 5.15 5:a8000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x5641e8aab3f0 con 0x5641e8aaca50 2023-10-05T19:28:31.924 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.918+0000 7f18397b1700 1 -- 172.21.15.167:0/3125065775 <== osd.3 v2:172.21.15.167:6808/30036 17 ==== osd_op_reply(65 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f182c04e040 con 0x5641e8aaca50 2023-10-05T19:28:31.924 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.918+0000 7f18397b1700 1 -- 172.21.15.167:0/3125065775 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:66 5.d 5:b0000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x5641e8aab3f0 con 0x5641e8a77940 2023-10-05T19:28:31.924 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.918+0000 7f1838fb0700 1 -- 172.21.15.167:0/3125065775 <== osd.2 v2:172.21.15.167:6816/30037 10 ==== osd_op_reply(66 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f1824006070 con 0x5641e8a77940 2023-10-05T19:28:31.924 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.918+0000 7f1838fb0700 1 -- 172.21.15.167:0/3125065775 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:67 5.1d 5:b8000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x5641e8aab3f0 con 0x5641e8aa6ca0 2023-10-05T19:28:31.925 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.918+0000 7f1839fb2700 1 -- 172.21.15.167:0/3125065775 <== osd.1 v2:172.21.15.167:6800/30035 24 ==== osd_op_reply(67 [pgnls start_epoch 30 out=107b] v30'2 uv2 ondisk = 1) v8 ==== 144+0+107 (crc 0 0 0) 0x7f183402f080 con 0x5641e8aa6ca0 2023-10-05T19:28:31.925 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.918+0000 7f1839fb2700 1 -- 172.21.15.167:0/3125065775 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:68 5.3 5:c0000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x5641e8aab3f0 con 0x7f182400eb60 2023-10-05T19:28:31.926 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.922+0000 7f1839fb2700 1 -- 172.21.15.167:0/3125065775 <== osd.0 v2:172.21.15.118:6800/27690 17 ==== osd_op_reply(68 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f183402f080 con 0x7f182400eb60 2023-10-05T19:28:31.926 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.922+0000 7f1839fb2700 1 -- 172.21.15.167:0/3125065775 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:69 5.13 5:c8000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x5641e8aab3f0 con 0x5641e8aaca50 2023-10-05T19:28:31.927 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.922+0000 7f18397b1700 1 -- 172.21.15.167:0/3125065775 <== osd.3 v2:172.21.15.167:6808/30036 18 ==== osd_op_reply(69 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f182c04e040 con 0x5641e8aaca50 2023-10-05T19:28:31.927 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.922+0000 7f18397b1700 1 -- 172.21.15.167:0/3125065775 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:70 5.b 5:d0000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x5641e8aab3f0 con 0x7f182400eb60 2023-10-05T19:28:31.928 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.922+0000 7f1839fb2700 1 -- 172.21.15.167:0/3125065775 <== osd.0 v2:172.21.15.118:6800/27690 18 ==== osd_op_reply(70 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f183402f080 con 0x7f182400eb60 2023-10-05T19:28:31.928 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.922+0000 7f1839fb2700 1 -- 172.21.15.167:0/3125065775 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:71 5.1b 5:d8000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x5641e8aab3f0 con 0x5641e8a77940 2023-10-05T19:28:31.929 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.922+0000 7f1838fb0700 1 -- 172.21.15.167:0/3125065775 <== osd.2 v2:172.21.15.167:6816/30037 11 ==== osd_op_reply(71 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f1824006070 con 0x5641e8a77940 2023-10-05T19:28:31.929 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.922+0000 7f1838fb0700 1 -- 172.21.15.167:0/3125065775 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:72 5.7 5:e0000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x5641e8aab3f0 con 0x5641e8aaca50 2023-10-05T19:28:31.929 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.922+0000 7f18397b1700 1 -- 172.21.15.167:0/3125065775 <== osd.3 v2:172.21.15.167:6808/30036 19 ==== osd_op_reply(72 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f182c04e040 con 0x5641e8aaca50 2023-10-05T19:28:31.930 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.922+0000 7f18397b1700 1 -- 172.21.15.167:0/3125065775 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:73 5.17 5:e8000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x5641e8aab3f0 con 0x7f182400eb60 2023-10-05T19:28:31.931 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.926+0000 7f1839fb2700 1 -- 172.21.15.167:0/3125065775 <== osd.0 v2:172.21.15.118:6800/27690 19 ==== osd_op_reply(73 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f183402f080 con 0x7f182400eb60 2023-10-05T19:28:31.931 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.926+0000 7f1839fb2700 1 -- 172.21.15.167:0/3125065775 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:74 5.f 5:f0000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x5641e8aab3f0 con 0x5641e8aa6ca0 2023-10-05T19:28:31.931 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.926+0000 7f1839fb2700 1 -- 172.21.15.167:0/3125065775 <== osd.1 v2:172.21.15.167:6800/30035 25 ==== osd_op_reply(74 [pgnls start_epoch 30 out=115b] v23'2 uv2 ondisk = 1) v8 ==== 144+0+115 (crc 0 0 0) 0x7f183402f080 con 0x5641e8aa6ca0 2023-10-05T19:28:31.931 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.926+0000 7f1839fb2700 1 -- 172.21.15.167:0/3125065775 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:75 5.1f 5:f8000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x5641e8aab3f0 con 0x5641e8a77940 2023-10-05T19:28:31.932 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.926+0000 7f1838fb0700 1 -- 172.21.15.167:0/3125065775 <== osd.2 v2:172.21.15.167:6816/30037 12 ==== osd_op_reply(75 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f1824006070 con 0x5641e8a77940 2023-10-05T19:28:31.933 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.926+0000 7f183bc7ce40 20 RGWRados::pool_iterate: got zonegroup_info.4f1f719d-9648-417b-b10c-e5d939825419 2023-10-05T19:28:31.933 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.926+0000 7f183bc7ce40 20 RGWRados::pool_iterate: got zone_info.9a9df965-6612-45d4-a45f-6e3c889bacc2 2023-10-05T19:28:31.933 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.926+0000 7f183bc7ce40 20 RGWRados::pool_iterate: got zone_names.default 2023-10-05T19:28:31.933 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.926+0000 7f183bc7ce40 20 RGWRados::pool_iterate: got zonegroups_names.default 2023-10-05T19:28:31.933 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.926+0000 7f183bc7ce40 20 rados->read ofs=0 len=0 2023-10-05T19:28:31.933 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.926+0000 7f183bc7ce40 1 -- 172.21.15.167:0/3125065775 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:76 5.f 5:f43b6ece:::zone_names.default:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x5641e8ab0010 con 0x5641e8aa6ca0 2023-10-05T19:28:31.934 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.926+0000 7f1839fb2700 1 -- 172.21.15.167:0/3125065775 <== osd.1 v2:172.21.15.167:6800/30035 26 ==== osd_op_reply(76 zone_names.default [read 0~46 out=46b] v0'0 uv1 ondisk = 0) v8 ==== 162+0+46 (crc 0 0 0) 0x7f183402f080 con 0x5641e8aa6ca0 2023-10-05T19:28:31.935 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.926+0000 7f183bc7ce40 20 rados_obj.operate() r=0 bl.length=46 2023-10-05T19:28:31.935 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.926+0000 7f183bc7ce40 20 rados->read ofs=0 len=0 2023-10-05T19:28:31.935 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.926+0000 7f183bc7ce40 1 -- 172.21.15.167:0/3125065775 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:77 5.1d 5:b8374eb4:::zone_info.9a9df965-6612-45d4-a45f-6e3c889bacc2:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x5641e8ab03c0 con 0x5641e8aa6ca0 2023-10-05T19:28:31.935 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.926+0000 7f1839fb2700 1 -- 172.21.15.167:0/3125065775 <== osd.1 v2:172.21.15.167:6800/30035 27 ==== osd_op_reply(77 zone_info.9a9df965-6612-45d4-a45f-6e3c889bacc2 [read 0~874 out=874b] v0'0 uv2 ondisk = 0) v8 ==== 190+0+874 (crc 0 0 0) 0x7f183402f080 con 0x5641e8aa6ca0 2023-10-05T19:28:31.935 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.926+0000 7f183bc7ce40 20 rados_obj.operate() r=0 bl.length=874 2023-10-05T19:28:31.935 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.926+0000 7f183bc7ce40 20 rados->read ofs=0 len=0 2023-10-05T19:28:31.935 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.930+0000 7f183bc7ce40 1 -- 172.21.15.167:0/3125065775 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:78 5.f 5:f4c53578:::zonegroups_names.default:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x5641e8ab08b0 con 0x5641e8aa6ca0 2023-10-05T19:28:31.935 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.930+0000 7f1839fb2700 1 -- 172.21.15.167:0/3125065775 <== osd.1 v2:172.21.15.167:6800/30035 28 ==== osd_op_reply(78 zonegroups_names.default [read 0~46 out=46b] v0'0 uv2 ondisk = 0) v8 ==== 168+0+46 (crc 0 0 0) 0x7f183402f080 con 0x5641e8aa6ca0 2023-10-05T19:28:31.935 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.930+0000 7f183bc7ce40 20 rados_obj.operate() r=0 bl.length=46 2023-10-05T19:28:31.935 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.930+0000 7f183bc7ce40 20 rados->read ofs=0 len=0 2023-10-05T19:28:31.936 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.930+0000 7f183bc7ce40 1 -- 172.21.15.167:0/3125065775 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:79 5.c 5:30d299a5:::zonegroup_info.4f1f719d-9648-417b-b10c-e5d939825419:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x5641e8ab08b0 con 0x7f182400eb60 2023-10-05T19:28:31.937 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.930+0000 7f1839fb2700 1 -- 172.21.15.167:0/3125065775 <== osd.0 v2:172.21.15.118:6800/27690 20 ==== osd_op_reply(79 zonegroup_info.4f1f719d-9648-417b-b10c-e5d939825419 [read 0~362 out=362b] v0'0 uv1 ondisk = 0) v8 ==== 195+0+362 (crc 0 0 0) 0x7f183402f080 con 0x7f182400eb60 2023-10-05T19:28:31.937 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.930+0000 7f183bc7ce40 20 rados_obj.operate() r=0 bl.length=362 2023-10-05T19:28:31.937 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.930+0000 7f183bc7ce40 20 rados->read ofs=0 len=0 2023-10-05T19:28:31.937 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.930+0000 7f183bc7ce40 1 -- 172.21.15.167:0/3125065775 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:80 5.3 5:c0ad8cbd:::region_map:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x5641e8ab09b0 con 0x7f182400eb60 2023-10-05T19:28:31.937 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.930+0000 7f1839fb2700 1 -- 172.21.15.167:0/3125065775 <== osd.0 v2:172.21.15.118:6800/27690 21 ==== osd_op_reply(80 region_map [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 ==== 154+0+0 (crc 0 0 0) 0x7f1834050040 con 0x7f182400eb60 2023-10-05T19:28:31.937 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.930+0000 7f183bc7ce40 20 rados_obj.operate() r=-2 bl.length=0 2023-10-05T19:28:31.937 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.930+0000 7f183bc7ce40 20 zone default found 2023-10-05T19:28:31.937 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.930+0000 7f183bc7ce40 4 Realm: () 2023-10-05T19:28:31.937 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.930+0000 7f183bc7ce40 4 ZoneGroup: default (4f1f719d-9648-417b-b10c-e5d939825419) 2023-10-05T19:28:31.938 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.930+0000 7f183bc7ce40 4 Zone: default (9a9df965-6612-45d4-a45f-6e3c889bacc2) 2023-10-05T19:28:31.938 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.930+0000 7f183bc7ce40 10 cannot find current period zonegroup using local zonegroup configuration 2023-10-05T19:28:31.938 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.930+0000 7f183bc7ce40 20 zonegroup default 2023-10-05T19:28:31.938 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.930+0000 7f183bc7ce40 20 rados->read ofs=0 len=0 2023-10-05T19:28:31.938 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.930+0000 7f183bc7ce40 1 -- 172.21.15.167:0/3125065775 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:81 5.3 5:c52100b6:::period_config.default:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x5641e8ab09b0 con 0x7f182400eb60 2023-10-05T19:28:31.939 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.930+0000 7f1839fb2700 1 -- 172.21.15.167:0/3125065775 <== osd.0 v2:172.21.15.118:6800/27690 22 ==== osd_op_reply(81 period_config.default [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 ==== 165+0+0 (crc 0 0 0) 0x7f1834050040 con 0x7f182400eb60 2023-10-05T19:28:31.939 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.930+0000 7f183bc7ce40 20 rados_obj.operate() r=-2 bl.length=0 2023-10-05T19:28:31.939 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.930+0000 7f183bc7ce40 20 rados->read ofs=0 len=0 2023-10-05T19:28:31.940 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.930+0000 7f183bc7ce40 1 -- 172.21.15.167:0/3125065775 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:82 6.2 6:4347d321:::bucket.sync-source-hints.:head [call version.read in=11b,read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x5641e8ab2330 con 0x5641e8aaca50 2023-10-05T19:28:31.940 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.930+0000 7f18397b1700 1 -- 172.21.15.167:0/3125065775 <== osd.3 v2:172.21.15.167:6808/30036 20 ==== osd_op_reply(82 bucket.sync-source-hints. [call,read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 ==== 211+0+0 (crc 0 0 0) 0x7f182c04e040 con 0x5641e8aaca50 2023-10-05T19:28:31.940 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.934+0000 7f183bc7ce40 20 rados_obj.operate() r=-2 bl.length=0 2023-10-05T19:28:31.940 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.934+0000 7f183bc7ce40 20 rados->read ofs=0 len=0 2023-10-05T19:28:31.940 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.934+0000 7f183bc7ce40 1 -- 172.21.15.167:0/3125065775 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:83 6.b 6:d467b91b:::bucket.sync-target-hints.:head [call version.read in=11b,read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x5641e8ab26e0 con 0x5641e8aaca50 2023-10-05T19:28:31.940 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.934+0000 7f18397b1700 1 -- 172.21.15.167:0/3125065775 <== osd.3 v2:172.21.15.167:6808/30036 21 ==== osd_op_reply(83 bucket.sync-target-hints. [call,read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 ==== 211+0+0 (crc 0 0 0) 0x7f182c04e040 con 0x5641e8aaca50 2023-10-05T19:28:31.940 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.934+0000 7f183bc7ce40 20 rados_obj.operate() r=-2 bl.length=0 2023-10-05T19:28:31.940 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.934+0000 7f183bc7ce40 20 started sync module instance, tier type = 2023-10-05T19:28:31.940 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.934+0000 7f183bc7ce40 20 started zone id=9a9df965-6612-45d4-a45f-6e3c889bacc2 (name=default) with tier type = 2023-10-05T19:28:31.940 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.934+0000 7f183bc7ce40 1 -- 172.21.15.167:0/3125065775 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:84 7.1f 7:f95f44c2:::notify.0:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x5641e8ab19f0 con 0x5641e8a77940 2023-10-05T19:28:31.941 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.934+0000 7f1838fb0700 1 -- 172.21.15.167:0/3125065775 <== osd.2 v2:172.21.15.167:6816/30037 13 ==== osd_op_reply(84 notify.0 [create] v30'102 uv102 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f1824006070 con 0x5641e8a77940 2023-10-05T19:28:31.941 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.934+0000 7f183bc7ce40 1 -- 172.21.15.167:0/3125065775 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:85 7.1f 7:f95f44c2:::notify.0:head [watch watch cookie 94841076391328] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x5641e8ab1620 con 0x5641e8a77940 2023-10-05T19:28:31.941 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.934+0000 7f183bc7ce40 1 -- 172.21.15.167:0/3125065775 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:86 7.0 7:05bf5b68:::notify.1:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x5641e8ab3470 con 0x5641e8aa6ca0 2023-10-05T19:28:31.944 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.938+0000 7f1838fb0700 1 -- 172.21.15.167:0/3125065775 <== osd.2 v2:172.21.15.167:6816/30037 14 ==== osd_op_reply(85 notify.0 [watch watch cookie 94841076391328] v30'103 uv102 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f1824006070 con 0x5641e8a77940 2023-10-05T19:28:31.945 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.938+0000 7f1839fb2700 1 -- 172.21.15.167:0/3125065775 <== osd.1 v2:172.21.15.167:6800/30035 29 ==== osd_op_reply(86 notify.1 [create] v30'102 uv102 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f1834050040 con 0x5641e8aa6ca0 2023-10-05T19:28:31.945 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.938+0000 7f183bc7ce40 1 -- 172.21.15.167:0/3125065775 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:87 7.0 7:05bf5b68:::notify.1:head [watch watch cookie 94841076394928] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x5641e8ab3530 con 0x5641e8aa6ca0 2023-10-05T19:28:31.945 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.938+0000 7f183bc7ce40 1 -- 172.21.15.167:0/3125065775 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:88 7.15 7:a93a5511:::notify.2:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x5641e8ab3f50 con 0x5641e8aaca50 2023-10-05T19:28:31.949 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.942+0000 7f18397b1700 1 -- 172.21.15.167:0/3125065775 <== osd.3 v2:172.21.15.167:6808/30036 22 ==== osd_op_reply(88 notify.2 [create] v30'102 uv102 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f182c04e040 con 0x5641e8aaca50 2023-10-05T19:28:31.949 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.942+0000 7f183bc7ce40 1 -- 172.21.15.167:0/3125065775 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:89 7.15 7:a93a5511:::notify.2:head [watch watch cookie 94841076398624] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x5641e8ab4840 con 0x5641e8aaca50 2023-10-05T19:28:31.949 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.942+0000 7f183bc7ce40 1 -- 172.21.15.167:0/3125065775 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:90 7.e 7:7759931f:::notify.3:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x5641e8ab5250 con 0x5641e8a77940 2023-10-05T19:28:31.949 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.942+0000 7f1839fb2700 1 -- 172.21.15.167:0/3125065775 <== osd.1 v2:172.21.15.167:6800/30035 30 ==== osd_op_reply(87 notify.1 [watch watch cookie 94841076394928] v30'103 uv102 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f1834050040 con 0x5641e8aa6ca0 2023-10-05T19:28:31.952 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.946+0000 7f1838fb0700 1 -- 172.21.15.167:0/3125065775 <== osd.2 v2:172.21.15.167:6816/30037 15 ==== osd_op_reply(90 notify.3 [create] v30'102 uv102 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f1824006070 con 0x5641e8a77940 2023-10-05T19:28:31.952 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.946+0000 7f183bc7ce40 1 -- 172.21.15.167:0/3125065775 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:91 7.e 7:7759931f:::notify.3:head [watch watch cookie 94841076396032] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x5641e8ab4e80 con 0x5641e8a77940 2023-10-05T19:28:31.952 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.946+0000 7f183bc7ce40 1 -- 172.21.15.167:0/3125065775 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:92 7.d 7:b4812045:::notify.4:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x5641e8ab61f0 con 0x5641e8aa6ca0 2023-10-05T19:28:31.952 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.946+0000 7f18397b1700 1 -- 172.21.15.167:0/3125065775 <== osd.3 v2:172.21.15.167:6808/30036 23 ==== osd_op_reply(89 notify.2 [watch watch cookie 94841076398624] v30'103 uv102 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f182c04e040 con 0x5641e8aaca50 2023-10-05T19:28:31.953 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.946+0000 7f1838fb0700 1 -- 172.21.15.167:0/3125065775 <== osd.2 v2:172.21.15.167:6816/30037 16 ==== osd_op_reply(91 notify.3 [watch watch cookie 94841076396032] v30'103 uv102 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f1824006070 con 0x5641e8a77940 2023-10-05T19:28:31.954 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.950+0000 7f1839fb2700 1 -- 172.21.15.167:0/3125065775 <== osd.1 v2:172.21.15.167:6800/30035 31 ==== osd_op_reply(92 notify.4 [create] v30'102 uv102 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f1834050040 con 0x5641e8aa6ca0 2023-10-05T19:28:31.954 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.950+0000 7f183bc7ce40 1 -- 172.21.15.167:0/3125065775 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:93 7.d 7:b4812045:::notify.4:head [watch watch cookie 94841076405760] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x5641e8ab5cc0 con 0x5641e8aa6ca0 2023-10-05T19:28:31.955 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.950+0000 7f183bc7ce40 1 -- 172.21.15.167:0/3125065775 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:94 7.3 7:c609908c:::notify.5:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x5641e8ab6ae0 con 0x7f182400eb60 2023-10-05T19:28:31.957 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.950+0000 7f1839fb2700 1 -- 172.21.15.167:0/3125065775 <== osd.1 v2:172.21.15.167:6800/30035 32 ==== osd_op_reply(93 notify.4 [watch watch cookie 94841076405760] v30'103 uv102 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f1834050040 con 0x5641e8aa6ca0 2023-10-05T19:28:31.958 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.954+0000 7f1839fb2700 1 -- 172.21.15.167:0/3125065775 <== osd.0 v2:172.21.15.118:6800/27690 23 ==== osd_op_reply(94 notify.5 [create] v30'102 uv102 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f1834050040 con 0x7f182400eb60 2023-10-05T19:28:31.958 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.954+0000 7f183bc7ce40 1 -- 172.21.15.167:0/3125065775 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:95 7.3 7:c609908c:::notify.5:head [watch watch cookie 94841076400976] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x5641e8ab6a90 con 0x7f182400eb60 2023-10-05T19:28:31.958 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.954+0000 7f183bc7ce40 1 -- 172.21.15.167:0/3125065775 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:96 7.14 7:2b04a3e9:::notify.6:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x5641e8ab74a0 con 0x7f182400eb60 2023-10-05T19:28:31.961 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.954+0000 7f1839fb2700 1 -- 172.21.15.167:0/3125065775 <== osd.0 v2:172.21.15.118:6800/27690 24 ==== osd_op_reply(95 notify.5 [watch watch cookie 94841076400976] v30'103 uv102 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f1834050040 con 0x7f182400eb60 2023-10-05T19:28:31.961 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.954+0000 7f1839fb2700 1 -- 172.21.15.167:0/3125065775 <== osd.0 v2:172.21.15.118:6800/27690 25 ==== osd_op_reply(96 notify.6 [create] v30'102 uv102 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f1834050040 con 0x7f182400eb60 2023-10-05T19:28:31.961 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.954+0000 7f183bc7ce40 1 -- 172.21.15.167:0/3125065775 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:97 7.14 7:2b04a3e9:::notify.6:head [watch watch cookie 94841076410320] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x5641e8ab74a0 con 0x7f182400eb60 2023-10-05T19:28:31.961 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.954+0000 7f183bc7ce40 1 -- 172.21.15.167:0/3125065775 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:98 7.9 7:93e5b521:::notify.7:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x5641e8ab7d40 con 0x7f182400eb60 2023-10-05T19:28:31.964 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.958+0000 7f1839fb2700 1 -- 172.21.15.167:0/3125065775 <== osd.0 v2:172.21.15.118:6800/27690 26 ==== osd_op_reply(98 notify.7 [create] v30'102 uv102 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f1834050040 con 0x7f182400eb60 2023-10-05T19:28:31.964 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.958+0000 7f183bc7ce40 1 -- 172.21.15.167:0/3125065775 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:99 7.9 7:93e5b521:::notify.7:head [watch watch cookie 94841076414160] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x5641e8ab8120 con 0x7f182400eb60 2023-10-05T19:28:31.964 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.958+0000 7f1839fb2700 1 -- 172.21.15.167:0/3125065775 <== osd.0 v2:172.21.15.118:6800/27690 27 ==== osd_op_reply(97 notify.6 [watch watch cookie 94841076410320] v30'103 uv102 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f1834050040 con 0x7f182400eb60 2023-10-05T19:28:31.964 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.958+0000 7f183bc7ce40 20 add_watcher() i=0 2023-10-05T19:28:31.964 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.958+0000 7f183bc7ce40 20 add_watcher() i=1 2023-10-05T19:28:31.964 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.958+0000 7f183bc7ce40 20 add_watcher() i=2 2023-10-05T19:28:31.964 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.958+0000 7f183bc7ce40 20 add_watcher() i=3 2023-10-05T19:28:31.964 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.958+0000 7f183bc7ce40 20 add_watcher() i=4 2023-10-05T19:28:31.965 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.958+0000 7f183bc7ce40 20 add_watcher() i=5 2023-10-05T19:28:31.965 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.958+0000 7f183bc7ce40 20 add_watcher() i=6 2023-10-05T19:28:31.966 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.958+0000 7f1839fb2700 1 -- 172.21.15.167:0/3125065775 <== osd.0 v2:172.21.15.118:6800/27690 28 ==== osd_op_reply(99 notify.7 [watch watch cookie 94841076414160] v30'103 uv102 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f1834050040 con 0x7f182400eb60 2023-10-05T19:28:31.966 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.962+0000 7f183bc7ce40 20 add_watcher() i=7 2023-10-05T19:28:31.966 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.962+0000 7f183bc7ce40 2 all 8 watchers are set, enabling cache 2023-10-05T19:28:31.966 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.962+0000 7f183bc7ce40 1 -- 172.21.15.167:0/3125065775 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:100 6.4 6:22d26bf9:::data_loggenerations_metadata:head [call version.check_conds in=74b,call version.read in=11b,read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x5641e8ab70d0 con 0x5641e8aa6ca0 2023-10-05T19:28:31.967 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.962+0000 7f1839fb2700 1 -- 172.21.15.167:0/3125065775 <== osd.1 v2:172.21.15.167:6800/30035 33 ==== osd_op_reply(100 data_loggenerations_metadata [call,call out=48b,read 0~28 out=28b] v0'0 uv1 ondisk = 0) v8 ==== 256+0+76 (crc 0 0 0) 0x7f1834050040 con 0x5641e8aa6ca0 2023-10-05T19:28:31.967 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.962+0000 7f183bc7ce40 1 -- 172.21.15.167:0/3125065775 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:101 6.4 6:22d26bf9:::data_loggenerations_metadata:head [watch watch cookie 94841076415472] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x5641e8ab70d0 con 0x5641e8aa6ca0 2023-10-05T19:28:31.969 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.962+0000 7f1839fb2700 1 -- 172.21.15.167:0/3125065775 <== osd.1 v2:172.21.15.167:6800/30035 34 ==== osd_op_reply(101 data_loggenerations_metadata [watch watch cookie 94841076415472] v30'103 uv1 ondisk = 0) v8 ==== 172+0+0 (crc 0 0 0) 0x7f1834050040 con 0x5641e8aa6ca0 2023-10-05T19:28:31.970 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.962+0000 7f1809feb700 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2023-10-05T19:28:31.970 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.962+0000 7f183bc7ce40 20 check_secure_mon_conn(): auth registy supported: methods=[2] modes=[2,1] 2023-10-05T19:28:31.970 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.962+0000 7f183bc7ce40 20 check_secure_mon_conn(): mode 1 is insecure 2023-10-05T19:28:31.970 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.966+0000 7f183bc7ce40 1 Processor -- start 2023-10-05T19:28:31.971 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.966+0000 7f183bc7ce40 1 -- start start 2023-10-05T19:28:31.971 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.966+0000 7f183bc7ce40 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5641e8ac5440 0x5641e8ac5810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-05T19:28:31.971 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.966+0000 7f183bc7ce40 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x5641e8ac5d50 con 0x5641e8ac5440 2023-10-05T19:28:31.971 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.966+0000 7f1838fb0700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5641e8ac5440 0x5641e8ac5810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-05T19:28:31.971 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.966+0000 7f1838fb0700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5641e8ac5440 0x5641e8ac5810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.118:3300/0 says I am v2:172.21.15.167:58310/0 (socket says 172.21.15.167:58310) 2023-10-05T19:28:31.971 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.966+0000 7f1838fb0700 1 -- 172.21.15.167:0/1572564379 learned_addr learned my addr 172.21.15.167:0/1572564379 (peer_addr_for_me v2:172.21.15.167:0/0) 2023-10-05T19:28:31.972 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.966+0000 7f1838fb0700 1 -- 172.21.15.167:0/1572564379 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5641e8ac5e90 con 0x5641e8ac5440 2023-10-05T19:28:31.972 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.966+0000 7f1838fb0700 1 --2- 172.21.15.167:0/1572564379 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5641e8ac5440 0x5641e8ac5810 secure :-1 s=READY pgs=342 cs=0 l=1 rev1=1 crypto rx=0x5641e8aaa540 tx=0x7f1824006000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c9857a4f8e2f68ae server_cookie=0 in_seq=0 out_seq=0 2023-10-05T19:28:31.973 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.966+0000 7f18097ea700 1 -- 172.21.15.167:0/1572564379 <== mon.0 v2:172.21.15.118:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f1824023040 con 0x5641e8ac5440 2023-10-05T19:28:31.973 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.966+0000 7f18097ea700 1 -- 172.21.15.167:0/1572564379 <== mon.0 v2:172.21.15.118:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1824024040 con 0x5641e8ac5440 2023-10-05T19:28:31.973 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.966+0000 7f18097ea700 1 -- 172.21.15.167:0/1572564379 <== mon.0 v2:172.21.15.118:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f1824013040 con 0x5641e8ac5440 2023-10-05T19:28:31.974 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.966+0000 7f183bc7ce40 1 -- 172.21.15.167:0/1572564379 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5641e8ac5440 msgr2=0x5641e8ac5810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-05T19:28:31.974 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.966+0000 7f183bc7ce40 1 --2- 172.21.15.167:0/1572564379 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5641e8ac5440 0x5641e8ac5810 secure :-1 s=READY pgs=342 cs=0 l=1 rev1=1 crypto rx=0x5641e8aaa540 tx=0x7f1824006000 comp rx=0 tx=0).stop 2023-10-05T19:28:31.974 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.970+0000 7f183bc7ce40 1 -- 172.21.15.167:0/1572564379 shutdown_connections 2023-10-05T19:28:31.974 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.970+0000 7f183bc7ce40 1 --2- 172.21.15.167:0/1572564379 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5641e8ac5440 0x5641e8ac5810 unknown :-1 s=CLOSED pgs=342 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-05T19:28:31.975 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.970+0000 7f183bc7ce40 1 -- 172.21.15.167:0/1572564379 >> 172.21.15.167:0/1572564379 conn(0x5641e8ac0800 msgr2=0x5641e8abe9b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-05T19:28:31.975 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.970+0000 7f183bc7ce40 1 -- 172.21.15.167:0/1572564379 shutdown_connections 2023-10-05T19:28:31.975 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.970+0000 7f183bc7ce40 1 -- 172.21.15.167:0/1572564379 wait complete. 2023-10-05T19:28:31.976 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.970+0000 7f183bc7ce40 1 Processor -- start 2023-10-05T19:28:31.976 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.970+0000 7f183bc7ce40 1 -- start start 2023-10-05T19:28:31.976 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.970+0000 7f183bc7ce40 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5641e8ac5440 0x5641e8acaf70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-05T19:28:31.976 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.970+0000 7f183bc7ce40 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x5641e8acbf70 con 0x5641e8ac5440 2023-10-05T19:28:31.976 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.970+0000 7f1838fb0700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5641e8ac5440 0x5641e8acaf70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-05T19:28:31.976 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.970+0000 7f1838fb0700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5641e8ac5440 0x5641e8acaf70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.118:3300/0 says I am v2:172.21.15.167:58322/0 (socket says 172.21.15.167:58322) 2023-10-05T19:28:31.976 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.970+0000 7f1838fb0700 1 -- 172.21.15.167:0/1478011711 learned_addr learned my addr 172.21.15.167:0/1478011711 (peer_addr_for_me v2:172.21.15.167:0/0) 2023-10-05T19:28:31.977 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.970+0000 7f1838fb0700 1 -- 172.21.15.167:0/1478011711 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f181c001e10 con 0x5641e8ac5440 2023-10-05T19:28:31.977 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.970+0000 7f1838fb0700 1 --2- 172.21.15.167:0/1478011711 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5641e8ac5440 0x5641e8acaf70 secure :-1 s=READY pgs=343 cs=0 l=1 rev1=1 crypto rx=0x7f182400dfd0 tx=0x7f182402f030 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-05T19:28:31.978 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.970+0000 7f17837fe700 1 -- 172.21.15.167:0/1478011711 <== mon.0 v2:172.21.15.118:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f1824023070 con 0x5641e8ac5440 2023-10-05T19:28:31.978 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.970+0000 7f17837fe700 1 -- 172.21.15.167:0/1478011711 <== mon.0 v2:172.21.15.118:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f182401e700 con 0x5641e8ac5440 2023-10-05T19:28:31.978 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.970+0000 7f183bc7ce40 1 -- 172.21.15.167:0/1478011711 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x5641e8acb570 con 0x5641e8ac5440 2023-10-05T19:28:31.978 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.970+0000 7f17837fe700 1 -- 172.21.15.167:0/1478011711 <== mon.0 v2:172.21.15.118:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f182401e8b0 con 0x5641e8ac5440 2023-10-05T19:28:31.978 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.970+0000 7f183bc7ce40 1 -- 172.21.15.167:0/1478011711 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x5641e8acb9b0 con 0x5641e8ac5440 2023-10-05T19:28:31.979 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.974+0000 7f17837fe700 1 -- 172.21.15.167:0/1478011711 <== mon.0 v2:172.21.15.118:3300/0 4 ==== mgrmap(e 5) v1 ==== 40329+0+0 (secure 0 0 0) 0x7f182401ea10 con 0x5641e8ac5440 2023-10-05T19:28:31.979 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.974+0000 7f17837fe700 1 --2- 172.21.15.167:0/1478011711 >> [v2:172.21.15.167:6824/29889,v1:172.21.15.167:6825/29889] conn(0x7f1778030c50 0x7f1778033100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-05T19:28:31.979 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.974+0000 7f17837fe700 1 -- 172.21.15.167:0/1478011711 <== mon.0 v2:172.21.15.118:3300/0 5 ==== osd_map(30..30 src has 1..30) v4 ==== 6057+0+0 (secure 0 0 0) 0x7f1824043c20 con 0x5641e8ac5440 2023-10-05T19:28:31.980 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.974+0000 7f1839fb2700 1 --2- 172.21.15.167:0/1478011711 >> [v2:172.21.15.167:6824/29889,v1:172.21.15.167:6825/29889] conn(0x7f1778030c50 0x7f1778033100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-05T19:28:31.980 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.974+0000 7f183bc7ce40 5 note: GC not initialized 2023-10-05T19:28:31.980 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.974+0000 7f1839fb2700 1 --2- 172.21.15.167:0/1478011711 >> [v2:172.21.15.167:6824/29889,v1:172.21.15.167:6825/29889] conn(0x7f1778030c50 0x7f1778033100 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x5641e8ab8080 tx=0x7f1834000f90 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-05T19:28:31.982 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.978+0000 7f1780ff9700 20 reqs_thread_entry: start 2023-10-05T19:28:31.983 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.978+0000 7f17767fc700 20 reqs_thread_entry: start 2023-10-05T19:28:31.983 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.978+0000 7f183bc7ce40 20 init_complete bucket index max shards: 11 2023-10-05T19:28:31.983 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.978+0000 7f183bc7ce40 10 rgw notify: Started notification manager with: 1 workers 2023-10-05T19:28:31.984 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.978+0000 7f1774ff9700 20 rgw notify: INFO: next queues processing will happen at: Thu Oct 5 19:29:02 2023 2023-10-05T19:28:31.984 INFO:teuthology.orchestra.run.smithi167.stderr: 2023-10-05T19:28:31.984 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.978+0000 7f175bfff700 20 reqs_thread_entry: start 2023-10-05T19:28:31.984 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.978+0000 7f183bc7ce40 20 get_system_obj_state: rctx=0x7ffcf2e78bd8 obj=default.rgw.meta:users.uid:foo state=0x5641e894e0b0 s->prefetch_data=0 2023-10-05T19:28:31.984 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.978+0000 7f183bc7ce40 1 -- 172.21.15.167:0/3125065775 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:102 8.18 8:1e6ca5ed:users.uid::foo:head [call version.read in=11b,getxattrs,stat] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x5641e8b336d0 con 0x5641e8aa6ca0 2023-10-05T19:28:31.985 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.978+0000 7f1839fb2700 1 -- 172.21.15.167:0/3125065775 <== osd.1 v2:172.21.15.167:6800/30035 35 ==== osd_op_reply(102 foo [call out=48b,getxattrs out=75b,stat out=16b] v0'0 uv11 ondisk = 0) v8 ==== 231+0+139 (crc 0 0 0) 0x7f1834050040 con 0x5641e8aa6ca0 2023-10-05T19:28:31.985 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.978+0000 7f183bc7ce40 20 get_system_obj_state: s->obj_tag was set empty 2023-10-05T19:28:31.985 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.978+0000 7f183bc7ce40 20 Read xattr: user.rgw.idtag 2023-10-05T19:28:31.985 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.978+0000 7f183bc7ce40 20 rados->read ofs=0 len=0 2023-10-05T19:28:31.985 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.978+0000 7f183bc7ce40 1 -- 172.21.15.167:0/3125065775 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:103 8.18 8:1e6ca5ed:users.uid::foo:head [call version.check_conds in=122b,call version.read in=11b,read 0~0,getxattrs] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x5641e8b33650 con 0x5641e8aa6ca0 2023-10-05T19:28:31.986 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.978+0000 7f1839fb2700 1 -- 172.21.15.167:0/3125065775 <== osd.1 v2:172.21.15.167:6800/30035 36 ==== osd_op_reply(103 foo [call,call out=48b,read 0~357 out=357b,getxattrs out=75b] v0'0 uv11 ondisk = 0) v8 ==== 273+0+480 (crc 0 0 0) 0x7f1834050040 con 0x5641e8aa6ca0 2023-10-05T19:28:31.986 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.978+0000 7f183bc7ce40 20 rados_obj.operate() r=0 bl.length=357 2023-10-05T19:28:31.986 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.978+0000 7f183bc7ce40 1 -- 172.21.15.167:0/3125065775 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:104 8.9 8:96d98480:users.uid::foo.buckets:head [call user.list_buckets in=34b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x5641e8b33650 con 0x5641e8a77940 2023-10-05T19:28:31.987 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.982+0000 7f1838fb0700 1 -- 172.21.15.167:0/3125065775 <== osd.2 v2:172.21.15.167:6816/30037 17 ==== osd_op_reply(104 foo.buckets [call out=183b] v0'0 uv14 ondisk = 0) v8 ==== 155+0+183 (crc 0 0 0) 0x7f182403f040 con 0x5641e8a77940 2023-10-05T19:28:31.987 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.982+0000 7f183bc7ce40 20 get_system_obj_state: rctx=0x7ffcf2e78f30 obj=default.rgw.meta:root:.bucket.meta.myfoo:9a9df965-6612-45d4-a45f-6e3c889bacc2.4223.1 state=0x5641e894e0b0 s->prefetch_data=0 2023-10-05T19:28:31.987 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.982+0000 7f183bc7ce40 1 -- 172.21.15.167:0/3125065775 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:105 8.11 8:8bb4ed75:root::.bucket.meta.myfoo%3a9a9df965-6612-45d4-a45f-6e3c889bacc2.4223.1:head [call version.read in=11b,getxattrs,stat] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x5641e8b34c60 con 0x5641e8aaca50 2023-10-05T19:28:31.988 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.982+0000 7f18397b1700 1 -- 172.21.15.167:0/3125065775 <== osd.3 v2:172.21.15.167:6808/30036 24 ==== osd_op_reply(105 .bucket.meta.myfoo:9a9df965-6612-45d4-a45f-6e3c889bacc2.4223.1 [call out=48b,getxattrs out=218b,stat out=16b] v0'0 uv1 ondisk = 0) v8 ==== 290+0+282 (crc 0 0 0) 0x7f182c04e040 con 0x5641e8aaca50 2023-10-05T19:28:31.988 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.982+0000 7f183bc7ce40 20 get_system_obj_state: s->obj_tag was set empty 2023-10-05T19:28:31.988 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.982+0000 7f183bc7ce40 20 Read xattr: user.rgw.acl 2023-10-05T19:28:31.988 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.982+0000 7f183bc7ce40 20 Read xattr: user.rgw.idtag 2023-10-05T19:28:31.988 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.982+0000 7f183bc7ce40 20 rados->read ofs=0 len=0 2023-10-05T19:28:31.988 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.982+0000 7f183bc7ce40 1 -- 172.21.15.167:0/3125065775 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:106 8.11 8:8bb4ed75:root::.bucket.meta.myfoo%3a9a9df965-6612-45d4-a45f-6e3c889bacc2.4223.1:head [call version.check_conds in=122b,call version.read in=11b,read 0~0,getxattrs] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x5641e8b35790 con 0x5641e8aaca50 2023-10-05T19:28:31.989 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.982+0000 7f18397b1700 1 -- 172.21.15.167:0/3125065775 <== osd.3 v2:172.21.15.167:6808/30036 25 ==== osd_op_reply(106 .bucket.meta.myfoo:9a9df965-6612-45d4-a45f-6e3c889bacc2.4223.1 [call,call out=48b,read 0~352 out=352b,getxattrs out=218b] v0'0 uv1 ondisk = 0) v8 ==== 332+0+618 (crc 0 0 0) 0x7f182c04e040 con 0x5641e8aaca50 2023-10-05T19:28:31.990 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.982+0000 7f183bc7ce40 20 rados_obj.operate() r=0 bl.length=352 2023-10-05T19:28:31.990 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.982+0000 7f183bc7ce40 20 couldn't put binfo cache entry, might have raced with data changes 2023-10-05T19:28:31.990 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.982+0000 7f183bc7ce40 1 -- 172.21.15.167:0/3125065775 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:107 4.3d 4:bd5dd36d:::.dir.9a9df965-6612-45d4-a45f-6e3c889bacc2.4223.1.0:head [call rgw.bucket_list in=47b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x5641e8b343c0 con 0x5641e8aaca50 2023-10-05T19:28:31.990 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.982+0000 7f183bc7ce40 1 -- 172.21.15.167:0/3125065775 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:108 4.29 4:955f623b:::.dir.9a9df965-6612-45d4-a45f-6e3c889bacc2.4223.1.1:head [call rgw.bucket_list in=47b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x5641e8b34870 con 0x5641e8aaca50 2023-10-05T19:28:31.990 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.982+0000 7f183bc7ce40 1 -- 172.21.15.167:0/3125065775 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:109 4.38 4:1d83ede1:::.dir.9a9df965-6612-45d4-a45f-6e3c889bacc2.4223.1.2:head [call rgw.bucket_list in=47b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x5641e8b3d140 con 0x5641e8aa6ca0 2023-10-05T19:28:31.990 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.982+0000 7f183bc7ce40 1 -- 172.21.15.167:0/3125065775 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:110 4.1e 4:7939a9ac:::.dir.9a9df965-6612-45d4-a45f-6e3c889bacc2.4223.1.3:head [call rgw.bucket_list in=47b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x5641e8b3faf0 con 0x7f182400eb60 2023-10-05T19:28:31.990 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.982+0000 7f183bc7ce40 1 -- 172.21.15.167:0/3125065775 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:111 4.e 4:719498d8:::.dir.9a9df965-6612-45d4-a45f-6e3c889bacc2.4223.1.4:head [call rgw.bucket_list in=47b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x5641e8b423b0 con 0x5641e8a77940 2023-10-05T19:28:31.990 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.982+0000 7f183bc7ce40 1 -- 172.21.15.167:0/3125065775 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:112 4.32 4:4f38608c:::.dir.9a9df965-6612-45d4-a45f-6e3c889bacc2.4223.1.5:head [call rgw.bucket_list in=47b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x5641e8b44d80 con 0x5641e8aa6ca0 2023-10-05T19:28:31.991 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.982+0000 7f183bc7ce40 1 -- 172.21.15.167:0/3125065775 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:113 4.38 4:1f2a6458:::.dir.9a9df965-6612-45d4-a45f-6e3c889bacc2.4223.1.6:head [call rgw.bucket_list in=47b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x5641e8b423b0 con 0x5641e8aa6ca0 2023-10-05T19:28:31.991 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.982+0000 7f183bc7ce40 1 -- 172.21.15.167:0/3125065775 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:114 4.2 4:41a3e149:::.dir.9a9df965-6612-45d4-a45f-6e3c889bacc2.4223.1.7:head [call rgw.bucket_list in=47b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x5641e8b423b0 con 0x5641e8aa6ca0 2023-10-05T19:28:31.992 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.982+0000 7f183bc7ce40 1 -- 172.21.15.167:0/3125065775 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:115 4.17 4:ea8a113e:::.dir.9a9df965-6612-45d4-a45f-6e3c889bacc2.4223.1.8:head [call rgw.bucket_list in=47b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x5641e8b4c290 con 0x7f182400eb60 2023-10-05T19:28:31.992 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.982+0000 7f183bc7ce40 1 -- 172.21.15.167:0/3125065775 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:116 4.1e 4:7a3e9338:::.dir.9a9df965-6612-45d4-a45f-6e3c889bacc2.4223.1.9:head [call rgw.bucket_list in=47b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x5641e8b4ec70 con 0x7f182400eb60 2023-10-05T19:28:31.992 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.982+0000 7f183bc7ce40 1 -- 172.21.15.167:0/3125065775 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:117 4.29 4:94432b39:::.dir.9a9df965-6612-45d4-a45f-6e3c889bacc2.4223.1.10:head [call rgw.bucket_list in=47b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x5641e8b51530 con 0x5641e8aaca50 2023-10-05T19:28:31.992 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.986+0000 7f1839fb2700 1 -- 172.21.15.167:0/3125065775 <== osd.0 v2:172.21.15.118:6800/27690 29 ==== osd_op_reply(115 .dir.9a9df965-6612-45d4-a45f-6e3c889bacc2.4223.1.8 [call out=85b] v0'0 uv1 ondisk = 0) v8 ==== 194+0+85 (crc 0 0 0) 0x7f1834050040 con 0x7f182400eb60 2023-10-05T19:28:31.992 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.986+0000 7f1838fb0700 1 -- 172.21.15.167:0/3125065775 <== osd.2 v2:172.21.15.167:6816/30037 18 ==== osd_op_reply(111 .dir.9a9df965-6612-45d4-a45f-6e3c889bacc2.4223.1.4 [call out=85b] v0'0 uv1 ondisk = 0) v8 ==== 194+0+85 (crc 0 0 0) 0x7f182403f040 con 0x5641e8a77940 2023-10-05T19:28:31.992 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.986+0000 7f1839fb2700 1 -- 172.21.15.167:0/3125065775 <== osd.0 v2:172.21.15.118:6800/27690 30 ==== osd_op_reply(110 .dir.9a9df965-6612-45d4-a45f-6e3c889bacc2.4223.1.3 [call out=85b] v0'0 uv1 ondisk = 0) v8 ==== 194+0+85 (crc 0 0 0) 0x7f1834050040 con 0x7f182400eb60 2023-10-05T19:28:31.992 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.986+0000 7f18397b1700 1 -- 172.21.15.167:0/3125065775 <== osd.3 v2:172.21.15.167:6808/30036 26 ==== osd_op_reply(107 .dir.9a9df965-6612-45d4-a45f-6e3c889bacc2.4223.1.0 [call out=85b] v0'0 uv1 ondisk = 0) v8 ==== 194+0+85 (crc 0 0 0) 0x7f182c04e040 con 0x5641e8aaca50 2023-10-05T19:28:31.992 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.986+0000 7f1839fb2700 1 -- 172.21.15.167:0/3125065775 <== osd.0 v2:172.21.15.118:6800/27690 31 ==== osd_op_reply(116 .dir.9a9df965-6612-45d4-a45f-6e3c889bacc2.4223.1.9 [call out=85b] v0'0 uv2 ondisk = 0) v8 ==== 194+0+85 (crc 0 0 0) 0x7f1834050040 con 0x7f182400eb60 2023-10-05T19:28:31.992 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.986+0000 7f1839fb2700 1 -- 172.21.15.167:0/3125065775 <== osd.1 v2:172.21.15.167:6800/30035 37 ==== osd_op_reply(112 .dir.9a9df965-6612-45d4-a45f-6e3c889bacc2.4223.1.5 [call out=124b] v0'0 uv3 ondisk = 0) v8 ==== 194+0+124 (crc 0 0 0) 0x7f1834050040 con 0x5641e8aa6ca0 2023-10-05T19:28:31.993 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.986+0000 7f18397b1700 1 -- 172.21.15.167:0/3125065775 <== osd.3 v2:172.21.15.167:6808/30036 27 ==== osd_op_reply(108 .dir.9a9df965-6612-45d4-a45f-6e3c889bacc2.4223.1.1 [call out=85b] v0'0 uv1 ondisk = 0) v8 ==== 194+0+85 (crc 0 0 0) 0x7f182c04e040 con 0x5641e8aaca50 2023-10-05T19:28:31.993 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.986+0000 7f1839fb2700 1 -- 172.21.15.167:0/3125065775 <== osd.1 v2:172.21.15.167:6800/30035 38 ==== osd_op_reply(109 .dir.9a9df965-6612-45d4-a45f-6e3c889bacc2.4223.1.2 [call out=85b] v0'0 uv1 ondisk = 0) v8 ==== 194+0+85 (crc 0 0 0) 0x7f1834050040 con 0x5641e8aa6ca0 2023-10-05T19:28:31.993 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.986+0000 7f18397b1700 1 -- 172.21.15.167:0/3125065775 <== osd.3 v2:172.21.15.167:6808/30036 28 ==== osd_op_reply(117 .dir.9a9df965-6612-45d4-a45f-6e3c889bacc2.4223.1.10 [call out=85b] v0'0 uv2 ondisk = 0) v8 ==== 195+0+85 (crc 0 0 0) 0x7f182c04e040 con 0x5641e8aaca50 2023-10-05T19:28:31.993 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.986+0000 7f1839fb2700 1 -- 172.21.15.167:0/3125065775 <== osd.1 v2:172.21.15.167:6800/30035 39 ==== osd_op_reply(114 .dir.9a9df965-6612-45d4-a45f-6e3c889bacc2.4223.1.7 [call out=85b] v0'0 uv1 ondisk = 0) v8 ==== 194+0+85 (crc 0 0 0) 0x7f1834050040 con 0x5641e8aa6ca0 2023-10-05T19:28:31.993 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.986+0000 7f1839fb2700 1 -- 172.21.15.167:0/3125065775 <== osd.1 v2:172.21.15.167:6800/30035 40 ==== osd_op_reply(113 .dir.9a9df965-6612-45d4-a45f-6e3c889bacc2.4223.1.6 [call out=85b] v0'0 uv2 ondisk = 0) v8 ==== 194+0+85 (crc 0 0 0) 0x7f1834050040 con 0x5641e8aa6ca0 2023-10-05T19:28:31.993 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.986+0000 7f183bc7ce40 1 -- 172.21.15.167:0/3125065775 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:118 8.9 8:96d98480:users.uid::foo.buckets:head [call user.set_buckets_info in=207b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x5641e8b44890 con 0x5641e8a77940 2023-10-05T19:28:31.994 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.990+0000 7f1838fb0700 1 -- 172.21.15.167:0/3125065775 <== osd.2 v2:172.21.15.167:6816/30037 19 ==== osd_op_reply(118 foo.buckets [call] v30'15 uv15 ondisk = 0) v8 ==== 155+0+0 (crc 0 0 0) 0x7f182403f040 con 0x5641e8a77940 2023-10-05T19:28:31.994 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.990+0000 7f183bc7ce40 1 -- 172.21.15.167:0/3125065775 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:119 8.9 8:96d98480:users.uid::foo.buckets:head [call user.complete_stats_sync in=37b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x5641e8b44890 con 0x5641e8a77940 2023-10-05T19:28:31.997 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.990+0000 7f1838fb0700 1 -- 172.21.15.167:0/3125065775 <== osd.2 v2:172.21.15.167:6816/30037 20 ==== osd_op_reply(119 foo.buckets [call] v30'16 uv16 ondisk = 0) v8 ==== 155+0+0 (crc 0 0 0) 0x7f182403f040 con 0x5641e8a77940 2023-10-05T19:28:31.998 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.990+0000 7f183bc7ce40 20 get_system_obj_state: rctx=0x7ffcf2e78e18 obj=default.rgw.meta:users.uid:foo state=0x5641e894e0b0 s->prefetch_data=0 2023-10-05T19:28:31.998 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.990+0000 7f183bc7ce40 1 -- 172.21.15.167:0/3125065775 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:120 8.18 8:1e6ca5ed:users.uid::foo:head [getxattrs,stat] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x5641e8b44890 con 0x5641e8aa6ca0 2023-10-05T19:28:31.998 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.990+0000 7f1839fb2700 1 -- 172.21.15.167:0/3125065775 <== osd.1 v2:172.21.15.167:6800/30035 41 ==== osd_op_reply(120 foo [getxattrs out=75b,stat out=16b] v0'0 uv11 ondisk = 0) v8 ==== 189+0+91 (crc 0 0 0) 0x7f1834050040 con 0x5641e8aa6ca0 2023-10-05T19:28:31.998 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.990+0000 7f183bc7ce40 20 get_system_obj_state: s->obj_tag was set empty 2023-10-05T19:28:31.998 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.990+0000 7f183bc7ce40 20 rados->read ofs=0 len=0 2023-10-05T19:28:31.998 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.990+0000 7f183bc7ce40 1 -- 172.21.15.167:0/3125065775 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:121 8.18 8:1e6ca5ed:users.uid::foo:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x5641e8b44890 con 0x5641e8aa6ca0 2023-10-05T19:28:31.999 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.994+0000 7f1839fb2700 1 -- 172.21.15.167:0/3125065775 <== osd.1 v2:172.21.15.167:6800/30035 42 ==== osd_op_reply(121 foo [read 0~357 out=357b] v0'0 uv11 ondisk = 0) v8 ==== 147+0+357 (crc 0 0 0) 0x7f1834050040 con 0x5641e8aa6ca0 2023-10-05T19:28:31.999 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.994+0000 7f183bc7ce40 20 rados_obj.operate() r=0 bl.length=357 2023-10-05T19:28:31.999 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.994+0000 7f183bc7ce40 1 -- 172.21.15.167:0/3125065775 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:122 8.9 8:96d98480:users.uid::foo.buckets:head [call user.get_header in=20b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x5641e8b44890 con 0x5641e8a77940 2023-10-05T19:28:31.999 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.994+0000 7f1838fb0700 1 -- 172.21.15.167:0/3125065775 <== osd.2 v2:172.21.15.167:6816/30037 21 ==== osd_op_reply(122 foo.buckets [call out=58b] v0'0 uv16 ondisk = 0) v8 ==== 155+0+58 (crc 0 0 0) 0x7f1824052040 con 0x5641e8a77940 2023-10-05T19:28:32.000 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.994+0000 7f183bc7ce40 1 -- 172.21.15.167:0/3125065775 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:123 7.1f 7:f95f44c2:::notify.0:head [watch unwatch cookie 94841076391328] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x5641e8b44890 con 0x5641e8a77940 2023-10-05T19:28:32.002 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.994+0000 7f1838fb0700 1 -- 172.21.15.167:0/3125065775 <== osd.2 v2:172.21.15.167:6816/30037 22 ==== osd_op_reply(123 notify.0 [watch unwatch cookie 94841076391328] v30'104 uv102 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f1824052040 con 0x5641e8a77940 2023-10-05T19:28:32.002 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.994+0000 7f183bc7ce40 20 remove_watcher() i=0 2023-10-05T19:28:32.002 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.994+0000 7f183bc7ce40 2 removed watcher, disabling cache 2023-10-05T19:28:32.002 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.994+0000 7f183bc7ce40 1 -- 172.21.15.167:0/3125065775 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:124 7.0 7:05bf5b68:::notify.1:head [watch unwatch cookie 94841076394928] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x5641e8a8c7f0 con 0x5641e8aa6ca0 2023-10-05T19:28:32.005 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.998+0000 7f1839fb2700 1 -- 172.21.15.167:0/3125065775 <== osd.1 v2:172.21.15.167:6800/30035 43 ==== osd_op_reply(124 notify.1 [watch unwatch cookie 94841076394928] v30'104 uv102 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f1834050040 con 0x5641e8aa6ca0 2023-10-05T19:28:32.005 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.998+0000 7f183bc7ce40 20 remove_watcher() i=1 2023-10-05T19:28:32.005 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:31.998+0000 7f183bc7ce40 1 -- 172.21.15.167:0/3125065775 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:125 7.15 7:a93a5511:::notify.2:head [watch unwatch cookie 94841076398624] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x5641e8ab2bb0 con 0x5641e8aaca50 2023-10-05T19:28:32.008 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.002+0000 7f18397b1700 1 -- 172.21.15.167:0/3125065775 <== osd.3 v2:172.21.15.167:6808/30036 29 ==== osd_op_reply(125 notify.2 [watch unwatch cookie 94841076398624] v30'104 uv102 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f182c04e040 con 0x5641e8aaca50 2023-10-05T19:28:32.008 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.002+0000 7f183bc7ce40 20 remove_watcher() i=2 2023-10-05T19:28:32.008 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.002+0000 7f183bc7ce40 1 -- 172.21.15.167:0/3125065775 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:126 7.e 7:7759931f:::notify.3:head [watch unwatch cookie 94841076396032] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x5641e8ab2bb0 con 0x5641e8a77940 2023-10-05T19:28:32.010 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.006+0000 7f1838fb0700 1 -- 172.21.15.167:0/3125065775 <== osd.2 v2:172.21.15.167:6816/30037 23 ==== osd_op_reply(126 notify.3 [watch unwatch cookie 94841076396032] v30'104 uv102 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f1824052040 con 0x5641e8a77940 2023-10-05T19:28:32.010 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.006+0000 7f183bc7ce40 20 remove_watcher() i=3 2023-10-05T19:28:32.011 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.006+0000 7f183bc7ce40 1 -- 172.21.15.167:0/3125065775 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:127 7.d 7:b4812045:::notify.4:head [watch unwatch cookie 94841076405760] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x5641e8ab3a20 con 0x5641e8aa6ca0 2023-10-05T19:28:32.014 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.006+0000 7f1839fb2700 1 -- 172.21.15.167:0/3125065775 <== osd.1 v2:172.21.15.167:6800/30035 44 ==== osd_op_reply(127 notify.4 [watch unwatch cookie 94841076405760] v30'104 uv102 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f1834050040 con 0x5641e8aa6ca0 2023-10-05T19:28:32.014 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.006+0000 7f183bc7ce40 20 remove_watcher() i=4 2023-10-05T19:28:32.014 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.006+0000 7f183bc7ce40 1 -- 172.21.15.167:0/3125065775 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:128 7.3 7:c609908c:::notify.5:head [watch unwatch cookie 94841076400976] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x5641e8ab3a20 con 0x7f182400eb60 2023-10-05T19:28:32.016 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.010+0000 7f1839fb2700 1 -- 172.21.15.167:0/3125065775 <== osd.0 v2:172.21.15.118:6800/27690 32 ==== osd_op_reply(128 notify.5 [watch unwatch cookie 94841076400976] v30'104 uv102 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f1834050040 con 0x7f182400eb60 2023-10-05T19:28:32.016 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.010+0000 7f183bc7ce40 20 remove_watcher() i=5 2023-10-05T19:28:32.016 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.010+0000 7f183bc7ce40 1 -- 172.21.15.167:0/3125065775 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:129 7.14 7:2b04a3e9:::notify.6:head [watch unwatch cookie 94841076410320] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x5641e8ab3a20 con 0x7f182400eb60 2023-10-05T19:28:32.020 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.014+0000 7f1839fb2700 1 -- 172.21.15.167:0/3125065775 <== osd.0 v2:172.21.15.118:6800/27690 33 ==== osd_op_reply(129 notify.6 [watch unwatch cookie 94841076410320] v30'104 uv102 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f1834050040 con 0x7f182400eb60 2023-10-05T19:28:32.020 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.014+0000 7f183bc7ce40 20 remove_watcher() i=6 2023-10-05T19:28:32.020 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.014+0000 7f183bc7ce40 1 -- 172.21.15.167:0/3125065775 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:130 7.9 7:93e5b521:::notify.7:head [watch unwatch cookie 94841076414160] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x5641e8ab3a20 con 0x7f182400eb60 2023-10-05T19:28:32.023 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.018+0000 7f1839fb2700 1 -- 172.21.15.167:0/3125065775 <== osd.0 v2:172.21.15.118:6800/27690 34 ==== osd_op_reply(130 notify.7 [watch unwatch cookie 94841076414160] v30'104 uv102 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f1834050040 con 0x7f182400eb60 2023-10-05T19:28:32.023 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.018+0000 7f183bc7ce40 20 remove_watcher() i=7 2023-10-05T19:28:32.024 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.018+0000 7f1809feb700 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2023-10-05T19:28:32.025 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.018+0000 7f183bc7ce40 1 -- 172.21.15.167:0/3125065775 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:131 6.4 6:22d26bf9:::data_loggenerations_metadata:head [watch unwatch cookie 94841076415472] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x5641e8ab3a20 con 0x5641e8aa6ca0 2023-10-05T19:28:32.028 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.022+0000 7f1839fb2700 1 -- 172.21.15.167:0/3125065775 <== osd.1 v2:172.21.15.167:6800/30035 45 ==== osd_op_reply(131 data_loggenerations_metadata [watch unwatch cookie 94841076415472] v30'104 uv1 ondisk = 0) v8 ==== 172+0+0 (crc 0 0 0) 0x7f1834050040 con 0x5641e8aa6ca0 2023-10-05T19:28:32.028 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.022+0000 7f183bc7ce40 1 -- 172.21.15.167:0/3125065775 >> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] conn(0x7f182400eb60 msgr2=0x7f182400d8e0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-05T19:28:32.028 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.022+0000 7f183bc7ce40 1 --2- 172.21.15.167:0/3125065775 >> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] conn(0x7f182400eb60 0x7f182400d8e0 crc :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-05T19:28:32.029 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.022+0000 7f183bc7ce40 1 -- 172.21.15.167:0/3125065775 >> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] conn(0x5641e8aa6ca0 msgr2=0x5641e8aa9080 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-05T19:28:32.029 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.022+0000 7f183bc7ce40 1 --2- 172.21.15.167:0/3125065775 >> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] conn(0x5641e8aa6ca0 0x5641e8aa9080 crc :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-05T19:28:32.029 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.022+0000 7f183bc7ce40 1 -- 172.21.15.167:0/3125065775 >> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] conn(0x5641e8a77940 msgr2=0x7f182c0289f0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-05T19:28:32.029 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.022+0000 7f183bc7ce40 1 --2- 172.21.15.167:0/3125065775 >> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] conn(0x5641e8a77940 0x7f182c0289f0 crc :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-05T19:28:32.029 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.022+0000 7f183bc7ce40 1 -- 172.21.15.167:0/3125065775 >> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] conn(0x5641e8aaca50 msgr2=0x5641e8aaee30 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-05T19:28:32.029 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.022+0000 7f183bc7ce40 1 --2- 172.21.15.167:0/3125065775 >> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] conn(0x5641e8aaca50 0x5641e8aaee30 crc :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-05T19:28:32.029 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.022+0000 7f183bc7ce40 1 -- 172.21.15.167:0/3125065775 >> [v2:172.21.15.167:6824/29889,v1:172.21.15.167:6825/29889] conn(0x7f1820030c50 msgr2=0x7f1820033100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-05T19:28:32.029 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.022+0000 7f183bc7ce40 1 --2- 172.21.15.167:0/3125065775 >> [v2:172.21.15.167:6824/29889,v1:172.21.15.167:6825/29889] conn(0x7f1820030c50 0x7f1820033100 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7f1824004170 tx=0x7f1824001400 comp rx=0 tx=0).stop 2023-10-05T19:28:32.029 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.022+0000 7f1839fb2700 1 -- 172.21.15.167:0/3125065775 reap_dead start 2023-10-05T19:28:32.029 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.022+0000 7f183bc7ce40 1 -- 172.21.15.167:0/3125065775 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5641e8a9cd90 msgr2=0x5641e8a9d160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-05T19:28:32.029 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.022+0000 7f183bc7ce40 1 --2- 172.21.15.167:0/3125065775 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5641e8a9cd90 0x5641e8a9d160 secure :-1 s=READY pgs=341 cs=0 l=1 rev1=1 crypto rx=0x7f182c0058e0 tx=0x7f182c01d000 comp rx=0 tx=0).stop 2023-10-05T19:28:32.030 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.022+0000 7f183bc7ce40 1 -- 172.21.15.167:0/3125065775 shutdown_connections 2023-10-05T19:28:32.030 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.022+0000 7f183bc7ce40 1 --2- 172.21.15.167:0/3125065775 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5641e8a9cd90 0x5641e8a9d160 unknown :-1 s=CLOSED pgs=341 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-05T19:28:32.030 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.022+0000 7f183bc7ce40 1 -- 172.21.15.167:0/3125065775 >> 172.21.15.167:0/3125065775 conn(0x5641e89ea880 msgr2=0x5641e8a90630 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-05T19:28:32.030 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.022+0000 7f183bc7ce40 1 -- 172.21.15.167:0/3125065775 shutdown_connections 2023-10-05T19:28:32.030 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.022+0000 7f183bc7ce40 1 -- 172.21.15.167:0/3125065775 wait complete. 2023-10-05T19:28:32.033 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.022+0000 7f183bc7ce40 1 -- 172.21.15.167:0/1478011711 >> [v2:172.21.15.167:6824/29889,v1:172.21.15.167:6825/29889] conn(0x7f1778030c50 msgr2=0x7f1778033100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-05T19:28:32.033 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.022+0000 7f183bc7ce40 1 --2- 172.21.15.167:0/1478011711 >> [v2:172.21.15.167:6824/29889,v1:172.21.15.167:6825/29889] conn(0x7f1778030c50 0x7f1778033100 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x5641e8ab8080 tx=0x7f1834000f90 comp rx=0 tx=0).stop 2023-10-05T19:28:32.034 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.022+0000 7f183bc7ce40 1 -- 172.21.15.167:0/1478011711 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5641e8ac5440 msgr2=0x5641e8acaf70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-05T19:28:32.034 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.022+0000 7f183bc7ce40 1 --2- 172.21.15.167:0/1478011711 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5641e8ac5440 0x5641e8acaf70 secure :-1 s=READY pgs=343 cs=0 l=1 rev1=1 crypto rx=0x7f182400dfd0 tx=0x7f182402f030 comp rx=0 tx=0).stop 2023-10-05T19:28:32.034 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.022+0000 7f183bc7ce40 1 -- 172.21.15.167:0/1478011711 shutdown_connections 2023-10-05T19:28:32.034 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.022+0000 7f183bc7ce40 1 --2- 172.21.15.167:0/1478011711 >> [v2:172.21.15.167:6824/29889,v1:172.21.15.167:6825/29889] conn(0x7f1778030c50 0x7f1778033100 unknown :-1 s=CLOSED pgs=121 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-05T19:28:32.034 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.022+0000 7f183bc7ce40 1 --2- 172.21.15.167:0/1478011711 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5641e8ac5440 0x5641e8acaf70 unknown :-1 s=CLOSED pgs=343 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-05T19:28:32.034 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.022+0000 7f183bc7ce40 1 -- 172.21.15.167:0/1478011711 >> 172.21.15.167:0/1478011711 conn(0x5641e8ac0800 msgr2=0x5641e8abe900 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-05T19:28:32.034 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.022+0000 7f183bc7ce40 1 -- 172.21.15.167:0/1478011711 shutdown_connections 2023-10-05T19:28:32.034 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.022+0000 7f183bc7ce40 1 -- 172.21.15.167:0/1478011711 wait complete. 2023-10-05T19:28:32.034 INFO:teuthology.orchestra.run.smithi167.stdout:{ 2023-10-05T19:28:32.035 INFO:teuthology.orchestra.run.smithi167.stdout: "stats": { 2023-10-05T19:28:32.035 INFO:teuthology.orchestra.run.smithi167.stdout: "size": 3, 2023-10-05T19:28:32.035 INFO:teuthology.orchestra.run.smithi167.stdout: "size_actual": 4096, 2023-10-05T19:28:32.035 INFO:teuthology.orchestra.run.smithi167.stdout: "size_kb": 1, 2023-10-05T19:28:32.035 INFO:teuthology.orchestra.run.smithi167.stdout: "size_kb_actual": 4, 2023-10-05T19:28:32.035 INFO:teuthology.orchestra.run.smithi167.stdout: "num_objects": 1 2023-10-05T19:28:32.035 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2023-10-05T19:28:32.035 INFO:teuthology.orchestra.run.smithi167.stdout: "last_stats_sync": "2023-10-05T19:28:31.991988Z", 2023-10-05T19:28:32.035 INFO:teuthology.orchestra.run.smithi167.stdout: "last_stats_update": "2023-10-05T19:28:31.989034Z" 2023-10-05T19:28:32.035 INFO:teuthology.orchestra.run.smithi167.stdout:} 2023-10-05T19:28:32.036 DEBUG:tasks.util.rgw: json result: {'stats': {'size': 3, 'size_actual': 4096, 'size_kb': 1, 'size_kb_actual': 4, 'num_objects': 1}, 'last_stats_sync': '2023-10-05T19:28:31.991988Z', 'last_stats_update': '2023-10-05T19:28:31.989034Z'} 2023-10-05T19:28:32.047 INFO:tasks.radosgw_admin:RL: delete_obj myfoo foo bytes_out=0 bytes_in=0 failed=False 2023-10-05T19:28:32.047 INFO:tasks.util.rgw:rgwadmin: client.0 : ['bucket', 'unlink', '--uid', 'foo', '--bucket', 'myfoo'] 2023-10-05T19:28:32.047 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'ceph', 'bucket', 'unlink', '--uid', 'foo', '--bucket', 'myfoo'] 2023-10-05T19:28:32.047 DEBUG:teuthology.orchestra.run.smithi167:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster ceph bucket unlink --uid foo --bucket myfoo 2023-10-05T19:28:32.085 INFO:teuthology.orchestra.run.smithi167.stderr:ignoring --setuser ceph since I am not root 2023-10-05T19:28:32.085 INFO:teuthology.orchestra.run.smithi167.stderr:ignoring --setgroup ceph since I am not root 2023-10-05T19:28:32.086 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.078+0000 7f106fea0e40 1 Processor -- start 2023-10-05T19:28:32.086 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.082+0000 7f106fea0e40 1 -- start start 2023-10-05T19:28:32.086 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.082+0000 7f106fea0e40 1 --2- >> v2:172.21.15.118:3300/0 conn(0x55797b084420 0x55797b082fd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-05T19:28:32.086 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.082+0000 7f106fea0e40 1 -- --> v1:172.21.15.118:6789/0 -- auth(proto 0 26 bytes epoch 0) v1 -- 0x55797afd0bd0 con 0x55797b0847f0 2023-10-05T19:28:32.086 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.082+0000 7f106fea0e40 1 -- --> v2:172.21.15.118:3300/0 -- mon_getmap magic: 0 v1 -- 0x55797af8dee0 con 0x55797b084420 2023-10-05T19:28:32.087 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.082+0000 7f106d9d5700 1 --2- >> v2:172.21.15.118:3300/0 conn(0x55797b084420 0x55797b082fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-05T19:28:32.087 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.082+0000 7f106d9d5700 1 --2- >> v2:172.21.15.118:3300/0 conn(0x55797b084420 0x55797b082fd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.118:3300/0 says I am v2:172.21.15.167:58330/0 (socket says 172.21.15.167:58330) 2023-10-05T19:28:32.087 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.082+0000 7f10651d4700 1 --1- 172.21.15.167:0/63832043 >> v1:172.21.15.118:6789/0 conn(0x55797b0847f0 0x55797b083510 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.118:6789/0 says I am v1:172.21.15.167:56600/0 (socket says 172.21.15.167:56600) 2023-10-05T19:28:32.088 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.082+0000 7f106d9d5700 1 -- 172.21.15.167:0/63832043 learned_addr learned my addr 172.21.15.167:0/63832043 (peer_addr_for_me v2:172.21.15.167:0/0) 2023-10-05T19:28:32.088 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.082+0000 7f106d9d5700 1 -- 172.21.15.167:0/63832043 >> v1:172.21.15.118:6789/0 conn(0x55797b0847f0 legacy=0x55797b083510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-05T19:28:32.088 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.082+0000 7f106d9d5700 1 -- 172.21.15.167:0/63832043 --> v2:172.21.15.118:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55797b083ce0 con 0x55797b084420 2023-10-05T19:28:32.089 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.082+0000 7f106d9d5700 1 --2- 172.21.15.167:0/63832043 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55797b084420 0x55797b082fd0 secure :-1 s=READY pgs=345 cs=0 l=1 rev1=1 crypto rx=0x7f106000a5c0 tx=0x7f1060005e00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=65d02f2cf6535ab0 server_cookie=0 in_seq=0 out_seq=0 2023-10-05T19:28:32.089 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.082+0000 7f106d1d4700 1 -- 172.21.15.167:0/63832043 <== mon.0 v2:172.21.15.118:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f106000e510 con 0x55797b084420 2023-10-05T19:28:32.089 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.082+0000 7f106d1d4700 1 -- 172.21.15.167:0/63832043 <== mon.0 v2:172.21.15.118:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f106000e670 con 0x55797b084420 2023-10-05T19:28:32.089 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.082+0000 7f106d1d4700 1 -- 172.21.15.167:0/63832043 <== mon.0 v2:172.21.15.118:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f106000e920 con 0x55797b084420 2023-10-05T19:28:32.089 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.082+0000 7f106fea0e40 1 -- 172.21.15.167:0/63832043 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55797b084420 msgr2=0x55797b082fd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-05T19:28:32.089 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.082+0000 7f106fea0e40 1 --2- 172.21.15.167:0/63832043 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55797b084420 0x55797b082fd0 secure :-1 s=READY pgs=345 cs=0 l=1 rev1=1 crypto rx=0x7f106000a5c0 tx=0x7f1060005e00 comp rx=0 tx=0).stop 2023-10-05T19:28:32.089 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.082+0000 7f106fea0e40 1 -- 172.21.15.167:0/63832043 shutdown_connections 2023-10-05T19:28:32.089 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.082+0000 7f106fea0e40 1 --2- 172.21.15.167:0/63832043 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55797b084420 0x55797b082fd0 unknown :-1 s=CLOSED pgs=345 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-05T19:28:32.089 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.082+0000 7f106fea0e40 1 -- 172.21.15.167:0/63832043 >> 172.21.15.167:0/63832043 conn(0x55797b07e730 msgr2=0x55797b080b40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-05T19:28:32.090 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.082+0000 7f106fea0e40 1 -- 172.21.15.167:0/63832043 shutdown_connections 2023-10-05T19:28:32.090 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.082+0000 7f106fea0e40 1 -- 172.21.15.167:0/63832043 wait complete. 2023-10-05T19:28:32.091 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.082+0000 7f106fea0e40 1 Processor -- start 2023-10-05T19:28:32.091 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.082+0000 7f106fea0e40 1 -- start start 2023-10-05T19:28:32.091 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.082+0000 7f106fea0e40 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55797b084420 0x55797b0a50f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-05T19:28:32.091 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.082+0000 7f106fea0e40 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x55797af51e30 con 0x55797b084420 2023-10-05T19:28:32.091 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.086+0000 7f106d9d5700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55797b084420 0x55797b0a50f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-05T19:28:32.091 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.086+0000 7f106d9d5700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55797b084420 0x55797b0a50f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.118:3300/0 says I am v2:172.21.15.167:58338/0 (socket says 172.21.15.167:58338) 2023-10-05T19:28:32.091 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.086+0000 7f106d9d5700 1 -- 172.21.15.167:0/1163334907 learned_addr learned my addr 172.21.15.167:0/1163334907 (peer_addr_for_me v2:172.21.15.167:0/0) 2023-10-05T19:28:32.091 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.086+0000 7f106d9d5700 1 -- 172.21.15.167:0/1163334907 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f106000a180 con 0x55797b084420 2023-10-05T19:28:32.092 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.086+0000 7f106d9d5700 1 --2- 172.21.15.167:0/1163334907 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55797b084420 0x55797b0a50f0 secure :-1 s=READY pgs=346 cs=0 l=1 rev1=1 crypto rx=0x7f1060003000 tx=0x7f1060013000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c5d30ab62d7f9604 server_cookie=0 in_seq=0 out_seq=0 2023-10-05T19:28:32.093 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.086+0000 7f1067fff700 1 -- 172.21.15.167:0/1163334907 <== mon.0 v2:172.21.15.118:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f1060005870 con 0x55797b084420 2023-10-05T19:28:32.093 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.086+0000 7f1067fff700 1 -- 172.21.15.167:0/1163334907 <== mon.0 v2:172.21.15.118:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f106000e7e0 con 0x55797b084420 2023-10-05T19:28:32.093 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.086+0000 7f1067fff700 1 -- 172.21.15.167:0/1163334907 <== mon.0 v2:172.21.15.118:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f106000e970 con 0x55797b084420 2023-10-05T19:28:32.093 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.086+0000 7f106fea0e40 1 -- 172.21.15.167:0/1163334907 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55797b084420 msgr2=0x55797b0a50f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-05T19:28:32.093 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.086+0000 7f106fea0e40 1 --2- 172.21.15.167:0/1163334907 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55797b084420 0x55797b0a50f0 secure :-1 s=READY pgs=346 cs=0 l=1 rev1=1 crypto rx=0x7f1060003000 tx=0x7f1060013000 comp rx=0 tx=0).stop 2023-10-05T19:28:32.093 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.086+0000 7f106fea0e40 1 -- 172.21.15.167:0/1163334907 shutdown_connections 2023-10-05T19:28:32.093 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.086+0000 7f106fea0e40 1 --2- 172.21.15.167:0/1163334907 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55797b084420 0x55797b0a50f0 unknown :-1 s=CLOSED pgs=346 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-05T19:28:32.093 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.086+0000 7f106fea0e40 1 -- 172.21.15.167:0/1163334907 >> 172.21.15.167:0/1163334907 conn(0x55797b07e730 msgr2=0x55797b0a0cb0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-05T19:28:32.093 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.086+0000 7f106fea0e40 1 -- 172.21.15.167:0/1163334907 shutdown_connections 2023-10-05T19:28:32.093 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.086+0000 7f106fea0e40 1 -- 172.21.15.167:0/1163334907 wait complete. 2023-10-05T19:28:32.094 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.086+0000 7f106fea0e40 1 Processor -- start 2023-10-05T19:28:32.094 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.086+0000 7f106fea0e40 1 -- start start 2023-10-05T19:28:32.094 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.086+0000 7f106fea0e40 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55797b084420 0x55797b0abd50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-05T19:28:32.094 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.086+0000 7f106fea0e40 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x55797af24ed0 con 0x55797b084420 2023-10-05T19:28:32.095 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.090+0000 7f106d9d5700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55797b084420 0x55797b0abd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-05T19:28:32.095 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.090+0000 7f106d9d5700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55797b084420 0x55797b0abd50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.118:3300/0 says I am v2:172.21.15.167:58340/0 (socket says 172.21.15.167:58340) 2023-10-05T19:28:32.095 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.090+0000 7f106d9d5700 1 -- 172.21.15.167:0/3268466528 learned_addr learned my addr 172.21.15.167:0/3268466528 (peer_addr_for_me v2:172.21.15.167:0/0) 2023-10-05T19:28:32.095 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.090+0000 7f106d9d5700 1 -- 172.21.15.167:0/3268466528 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55797b083ce0 con 0x55797b084420 2023-10-05T19:28:32.095 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.090+0000 7f106d9d5700 1 --2- 172.21.15.167:0/3268466528 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55797b084420 0x55797b0abd50 secure :-1 s=READY pgs=347 cs=0 l=1 rev1=1 crypto rx=0x7f106000a5c0 tx=0x7f106001d000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-05T19:28:32.096 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.090+0000 7f10667fc700 1 -- 172.21.15.167:0/3268466528 <== mon.0 v2:172.21.15.118:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f106002c070 con 0x55797b084420 2023-10-05T19:28:32.096 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.090+0000 7f10667fc700 1 -- 172.21.15.167:0/3268466528 <== mon.0 v2:172.21.15.118:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f10600275f0 con 0x55797b084420 2023-10-05T19:28:32.096 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.090+0000 7f10667fc700 1 -- 172.21.15.167:0/3268466528 <== mon.0 v2:172.21.15.118:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f10600278c0 con 0x55797b084420 2023-10-05T19:28:32.096 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.090+0000 7f106fea0e40 1 -- 172.21.15.167:0/3268466528 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x55797b0ac290 con 0x55797b084420 2023-10-05T19:28:32.096 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.090+0000 7f106fea0e40 1 -- 172.21.15.167:0/3268466528 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55797b0ac670 con 0x55797b084420 2023-10-05T19:28:32.097 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.090+0000 7f10667fc700 1 -- 172.21.15.167:0/3268466528 <== mon.0 v2:172.21.15.118:3300/0 4 ==== mgrmap(e 5) v1 ==== 40329+0+0 (secure 0 0 0) 0x7f1060017030 con 0x55797b084420 2023-10-05T19:28:32.098 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.094+0000 7f106fea0e40 20 rados->read ofs=0 len=0 2023-10-05T19:28:32.098 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.094+0000 7f10667fc700 1 --2- 172.21.15.167:0/3268466528 >> [v2:172.21.15.167:6824/29889,v1:172.21.15.167:6825/29889] conn(0x7f1050030c50 0x7f1050033100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-05T19:28:32.098 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.094+0000 7f10667fc700 1 -- 172.21.15.167:0/3268466528 <== mon.0 v2:172.21.15.118:3300/0 5 ==== osd_map(30..30 src has 1..30) v4 ==== 6057+0+0 (secure 0 0 0) 0x7f1060046670 con 0x55797b084420 2023-10-05T19:28:32.098 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.094+0000 7f10651d4700 1 --2- 172.21.15.167:0/3268466528 >> [v2:172.21.15.167:6824/29889,v1:172.21.15.167:6825/29889] conn(0x7f1050030c50 0x7f1050033100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-05T19:28:32.099 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.094+0000 7f106fea0e40 1 --2- 172.21.15.167:0/3268466528 >> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] conn(0x55797b0847f0 0x55797b0b7810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-05T19:28:32.099 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.094+0000 7f106fea0e40 1 -- 172.21.15.167:0/3268466528 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:1 5.12 5:49953fa1:::default.realm:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x55797b0b7e80 con 0x55797b0847f0 2023-10-05T19:28:32.099 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.094+0000 7f106e1d6700 1 --2- 172.21.15.167:0/3268466528 >> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] conn(0x55797b0847f0 0x55797b0b7810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-05T19:28:32.099 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.094+0000 7f10651d4700 1 --2- 172.21.15.167:0/3268466528 >> [v2:172.21.15.167:6824/29889,v1:172.21.15.167:6825/29889] conn(0x7f1050030c50 0x7f1050033100 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7f1058001eb0 tx=0x7f1058001580 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-05T19:28:32.100 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.094+0000 7f106e1d6700 1 --2- 172.21.15.167:0/3268466528 >> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] conn(0x55797b0847f0 0x55797b0b7810 crc :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-05T19:28:32.101 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.094+0000 7f106e1d6700 1 -- 172.21.15.167:0/3268466528 <== osd.1 v2:172.21.15.167:6800/30035 1 ==== osd_op_reply(1 default.realm [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 ==== 157+0+0 (crc 0 0 0) 0x7f106800a500 con 0x55797b0847f0 2023-10-05T19:28:32.101 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.094+0000 7f106fea0e40 20 rados_obj.operate() r=-2 bl.length=0 2023-10-05T19:28:32.101 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.094+0000 7f106fea0e40 20 realm 2023-10-05T19:28:32.101 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.094+0000 7f106fea0e40 20 rados->read ofs=0 len=0 2023-10-05T19:28:32.101 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.094+0000 7f106fea0e40 1 -- 172.21.15.167:0/3268466528 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:2 5.12 5:49953fa1:::default.realm:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x55797b0b4e70 con 0x55797b0847f0 2023-10-05T19:28:32.101 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.094+0000 7f106e1d6700 1 -- 172.21.15.167:0/3268466528 <== osd.1 v2:172.21.15.167:6800/30035 2 ==== osd_op_reply(2 default.realm [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 ==== 157+0+0 (crc 0 0 0) 0x7f106800a500 con 0x55797b0847f0 2023-10-05T19:28:32.101 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.094+0000 7f106fea0e40 20 rados_obj.operate() r=-2 bl.length=0 2023-10-05T19:28:32.102 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.094+0000 7f106fea0e40 4 RGWPeriod::init failed to init realm id : (2) No such file or directory 2023-10-05T19:28:32.102 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.094+0000 7f106fea0e40 20 rados->read ofs=0 len=0 2023-10-05T19:28:32.102 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.094+0000 7f106fea0e40 1 -- 172.21.15.167:0/3268466528 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:3 5.12 5:49953fa1:::default.realm:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x55797b0b8960 con 0x55797b0847f0 2023-10-05T19:28:32.103 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.094+0000 7f106e1d6700 1 -- 172.21.15.167:0/3268466528 <== osd.1 v2:172.21.15.167:6800/30035 3 ==== osd_op_reply(3 default.realm [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 ==== 157+0+0 (crc 0 0 0) 0x7f106800a500 con 0x55797b0847f0 2023-10-05T19:28:32.103 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.094+0000 7f106fea0e40 20 rados_obj.operate() r=-2 bl.length=0 2023-10-05T19:28:32.103 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.094+0000 7f106fea0e40 20 rados->read ofs=0 len=0 2023-10-05T19:28:32.103 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.094+0000 7f106fea0e40 1 -- 172.21.15.167:0/3268466528 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:4 5.f 5:f43b6ece:::zone_names.default:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x55797b0b8570 con 0x55797b0847f0 2023-10-05T19:28:32.104 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.098+0000 7f106e1d6700 1 -- 172.21.15.167:0/3268466528 <== osd.1 v2:172.21.15.167:6800/30035 4 ==== osd_op_reply(4 zone_names.default [read 0~46 out=46b] v0'0 uv1 ondisk = 0) v8 ==== 162+0+46 (crc 0 0 0) 0x7f106800a500 con 0x55797b0847f0 2023-10-05T19:28:32.104 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.098+0000 7f106fea0e40 20 rados_obj.operate() r=0 bl.length=46 2023-10-05T19:28:32.104 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.098+0000 7f106fea0e40 20 rados->read ofs=0 len=0 2023-10-05T19:28:32.104 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.098+0000 7f106fea0e40 1 -- 172.21.15.167:0/3268466528 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:5 5.1d 5:b8374eb4:::zone_info.9a9df965-6612-45d4-a45f-6e3c889bacc2:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x55797b0b97f0 con 0x55797b0847f0 2023-10-05T19:28:32.104 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.098+0000 7f106e1d6700 1 -- 172.21.15.167:0/3268466528 <== osd.1 v2:172.21.15.167:6800/30035 5 ==== osd_op_reply(5 zone_info.9a9df965-6612-45d4-a45f-6e3c889bacc2 [read 0~874 out=874b] v0'0 uv2 ondisk = 0) v8 ==== 190+0+874 (crc 0 0 0) 0x7f106800a500 con 0x55797b0847f0 2023-10-05T19:28:32.104 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.098+0000 7f106fea0e40 20 rados_obj.operate() r=0 bl.length=874 2023-10-05T19:28:32.104 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.098+0000 7f106fea0e40 20 searching for the correct realm 2023-10-05T19:28:32.104 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.098+0000 7f106fea0e40 1 --2- 172.21.15.167:0/3268466528 >> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] conn(0x55797b0bb5b0 0x55797b0bd990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-05T19:28:32.104 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.098+0000 7f106fea0e40 1 -- 172.21.15.167:0/3268466528 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:6 5.0 5:00000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x55797b0bded0 con 0x55797b0bb5b0 2023-10-05T19:28:32.104 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.098+0000 7f106d9d5700 1 --2- 172.21.15.167:0/3268466528 >> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] conn(0x55797b0bb5b0 0x55797b0bd990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-05T19:28:32.105 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.098+0000 7f106d9d5700 1 --2- 172.21.15.167:0/3268466528 >> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] conn(0x55797b0bb5b0 0x55797b0bd990 crc :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-05T19:28:32.106 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.098+0000 7f106d9d5700 1 -- 172.21.15.167:0/3268466528 <== osd.3 v2:172.21.15.167:6808/30036 1 ==== osd_op_reply(6 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f1060025450 con 0x55797b0bb5b0 2023-10-05T19:28:32.106 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.098+0000 7f106d9d5700 1 --2- 172.21.15.167:0/3268466528 >> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] conn(0x7f1060010070 0x7f10600187e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-05T19:28:32.106 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.098+0000 7f106d9d5700 1 -- 172.21.15.167:0/3268466528 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:7 5.10 5:08000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x55797b0bded0 con 0x7f1060010070 2023-10-05T19:28:32.106 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.102+0000 7f10651d4700 1 --2- 172.21.15.167:0/3268466528 >> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] conn(0x7f1060010070 0x7f10600187e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-05T19:28:32.106 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.102+0000 7f10651d4700 1 --2- 172.21.15.167:0/3268466528 >> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] conn(0x7f1060010070 0x7f10600187e0 crc :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-05T19:28:32.107 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.102+0000 7f10651d4700 1 -- 172.21.15.167:0/3268466528 <== osd.2 v2:172.21.15.167:6816/30037 1 ==== osd_op_reply(7 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f105800f7c0 con 0x7f1060010070 2023-10-05T19:28:32.107 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.102+0000 7f10651d4700 1 --2- 172.21.15.167:0/3268466528 >> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] conn(0x7f105800ea30 0x7f105800d310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-05T19:28:32.107 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.102+0000 7f10651d4700 1 -- 172.21.15.167:0/3268466528 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:8 5.8 5:10000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x55797b0bded0 con 0x7f105800ea30 2023-10-05T19:28:32.108 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.102+0000 7f106e1d6700 1 --2- 172.21.15.167:0/3268466528 >> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] conn(0x7f105800ea30 0x7f105800d310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-05T19:28:32.108 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.102+0000 7f106e1d6700 1 --2- 172.21.15.167:0/3268466528 >> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] conn(0x7f105800ea30 0x7f105800d310 crc :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-05T19:28:32.109 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.102+0000 7f106e1d6700 1 -- 172.21.15.167:0/3268466528 <== osd.0 v2:172.21.15.118:6800/27690 1 ==== osd_op_reply(8 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f1068014040 con 0x7f105800ea30 2023-10-05T19:28:32.109 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.102+0000 7f106e1d6700 1 -- 172.21.15.167:0/3268466528 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:9 5.18 5:18000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x55797b0bded0 con 0x55797b0847f0 2023-10-05T19:28:32.110 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.102+0000 7f106e1d6700 1 -- 172.21.15.167:0/3268466528 <== osd.1 v2:172.21.15.167:6800/30035 6 ==== osd_op_reply(9 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f1068014040 con 0x55797b0847f0 2023-10-05T19:28:32.110 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.106+0000 7f106e1d6700 1 -- 172.21.15.167:0/3268466528 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:10 5.4 5:20000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x55797b0bded0 con 0x55797b0847f0 2023-10-05T19:28:32.110 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.106+0000 7f106e1d6700 1 -- 172.21.15.167:0/3268466528 <== osd.1 v2:172.21.15.167:6800/30035 7 ==== osd_op_reply(10 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f1068014040 con 0x55797b0847f0 2023-10-05T19:28:32.111 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.106+0000 7f106e1d6700 1 -- 172.21.15.167:0/3268466528 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:11 5.14 5:28000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x55797b0bded0 con 0x7f105800ea30 2023-10-05T19:28:32.111 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.106+0000 7f106e1d6700 1 -- 172.21.15.167:0/3268466528 <== osd.0 v2:172.21.15.118:6800/27690 2 ==== osd_op_reply(11 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f1068014040 con 0x7f105800ea30 2023-10-05T19:28:32.111 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.106+0000 7f106e1d6700 1 -- 172.21.15.167:0/3268466528 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:12 5.c 5:30000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x55797b0bded0 con 0x7f105800ea30 2023-10-05T19:28:32.112 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.106+0000 7f106e1d6700 1 -- 172.21.15.167:0/3268466528 <== osd.0 v2:172.21.15.118:6800/27690 3 ==== osd_op_reply(12 [pgnls start_epoch 30 out=112b] v23'1 uv1 ondisk = 1) v8 ==== 144+0+112 (crc 0 0 0) 0x7f1068014040 con 0x7f105800ea30 2023-10-05T19:28:32.112 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.106+0000 7f106e1d6700 1 -- 172.21.15.167:0/3268466528 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:13 5.1c 5:38000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x55797b0bded0 con 0x7f1060010070 2023-10-05T19:28:32.113 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.106+0000 7f10651d4700 1 -- 172.21.15.167:0/3268466528 <== osd.2 v2:172.21.15.167:6816/30037 2 ==== osd_op_reply(13 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f105800f7c0 con 0x7f1060010070 2023-10-05T19:28:32.113 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.106+0000 7f10651d4700 1 -- 172.21.15.167:0/3268466528 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:14 5.2 5:40000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x55797b0bded0 con 0x55797b0bb5b0 2023-10-05T19:28:32.114 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.110+0000 7f106d9d5700 1 -- 172.21.15.167:0/3268466528 <== osd.3 v2:172.21.15.167:6808/30036 2 ==== osd_op_reply(14 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f1060025450 con 0x55797b0bb5b0 2023-10-05T19:28:32.114 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.110+0000 7f106d9d5700 1 -- 172.21.15.167:0/3268466528 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:15 5.12 5:48000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x55797b0bded0 con 0x55797b0847f0 2023-10-05T19:28:32.115 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.110+0000 7f106e1d6700 1 -- 172.21.15.167:0/3268466528 <== osd.1 v2:172.21.15.167:6800/30035 8 ==== osd_op_reply(15 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f1068014040 con 0x55797b0847f0 2023-10-05T19:28:32.115 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.110+0000 7f106e1d6700 1 -- 172.21.15.167:0/3268466528 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:16 5.a 5:50000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x55797b0bded0 con 0x7f105800ea30 2023-10-05T19:28:32.116 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.110+0000 7f106e1d6700 1 -- 172.21.15.167:0/3268466528 <== osd.0 v2:172.21.15.118:6800/27690 4 ==== osd_op_reply(16 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f1068014040 con 0x7f105800ea30 2023-10-05T19:28:32.116 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.110+0000 7f106e1d6700 1 -- 172.21.15.167:0/3268466528 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:17 5.1a 5:58000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x55797b0bded0 con 0x55797b0847f0 2023-10-05T19:28:32.116 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.110+0000 7f106e1d6700 1 -- 172.21.15.167:0/3268466528 <== osd.1 v2:172.21.15.167:6800/30035 9 ==== osd_op_reply(17 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f1068014040 con 0x55797b0847f0 2023-10-05T19:28:32.116 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.110+0000 7f106e1d6700 1 -- 172.21.15.167:0/3268466528 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:18 5.6 5:60000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x55797b0bded0 con 0x7f1060010070 2023-10-05T19:28:32.117 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.110+0000 7f10651d4700 1 -- 172.21.15.167:0/3268466528 <== osd.2 v2:172.21.15.167:6816/30037 3 ==== osd_op_reply(18 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f105800f7c0 con 0x7f1060010070 2023-10-05T19:28:32.117 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.110+0000 7f10651d4700 1 -- 172.21.15.167:0/3268466528 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:19 5.16 5:68000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x7f105801f040 con 0x55797b0bb5b0 2023-10-05T19:28:32.118 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.110+0000 7f106d9d5700 1 -- 172.21.15.167:0/3268466528 <== osd.3 v2:172.21.15.167:6808/30036 3 ==== osd_op_reply(19 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f1060025450 con 0x55797b0bb5b0 2023-10-05T19:28:32.118 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.114+0000 7f106d9d5700 1 -- 172.21.15.167:0/3268466528 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:20 5.e 5:70000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x7f105801f040 con 0x55797b0bb5b0 2023-10-05T19:28:32.119 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.114+0000 7f106d9d5700 1 -- 172.21.15.167:0/3268466528 <== osd.3 v2:172.21.15.167:6808/30036 4 ==== osd_op_reply(20 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f1060025450 con 0x55797b0bb5b0 2023-10-05T19:28:32.119 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.114+0000 7f106d9d5700 1 -- 172.21.15.167:0/3268466528 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:21 5.1e 5:78000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x7f105801f040 con 0x7f105800ea30 2023-10-05T19:28:32.120 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.114+0000 7f106e1d6700 1 -- 172.21.15.167:0/3268466528 <== osd.0 v2:172.21.15.118:6800/27690 5 ==== osd_op_reply(21 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f1068022040 con 0x7f105800ea30 2023-10-05T19:28:32.120 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.114+0000 7f106e1d6700 1 -- 172.21.15.167:0/3268466528 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:22 5.1 5:80000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x7f105801f040 con 0x55797b0bb5b0 2023-10-05T19:28:32.121 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.114+0000 7f106d9d5700 1 -- 172.21.15.167:0/3268466528 <== osd.3 v2:172.21.15.167:6808/30036 5 ==== osd_op_reply(22 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f1060025450 con 0x55797b0bb5b0 2023-10-05T19:28:32.121 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.114+0000 7f106d9d5700 1 -- 172.21.15.167:0/3268466528 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:23 5.11 5:88000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x7f105801f040 con 0x55797b0bb5b0 2023-10-05T19:28:32.121 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.114+0000 7f106d9d5700 1 -- 172.21.15.167:0/3268466528 <== osd.3 v2:172.21.15.167:6808/30036 6 ==== osd_op_reply(23 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f1060025450 con 0x55797b0bb5b0 2023-10-05T19:28:32.121 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.114+0000 7f106d9d5700 1 -- 172.21.15.167:0/3268466528 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:24 5.9 5:90000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x7f105801f040 con 0x55797b0847f0 2023-10-05T19:28:32.122 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.118+0000 7f106e1d6700 1 -- 172.21.15.167:0/3268466528 <== osd.1 v2:172.21.15.167:6800/30035 10 ==== osd_op_reply(24 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f1068022040 con 0x55797b0847f0 2023-10-05T19:28:32.122 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.118+0000 7f106e1d6700 1 -- 172.21.15.167:0/3268466528 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:25 5.19 5:98000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x7f105801f040 con 0x55797b0847f0 2023-10-05T19:28:32.123 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.118+0000 7f106e1d6700 1 -- 172.21.15.167:0/3268466528 <== osd.1 v2:172.21.15.167:6800/30035 11 ==== osd_op_reply(25 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f1068022040 con 0x55797b0847f0 2023-10-05T19:28:32.123 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.118+0000 7f106e1d6700 1 -- 172.21.15.167:0/3268466528 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:26 5.5 5:a0000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x7f105801f040 con 0x7f105800ea30 2023-10-05T19:28:32.124 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.118+0000 7f106e1d6700 1 -- 172.21.15.167:0/3268466528 <== osd.0 v2:172.21.15.118:6800/27690 6 ==== osd_op_reply(26 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f1068022040 con 0x7f105800ea30 2023-10-05T19:28:32.124 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.118+0000 7f106e1d6700 1 -- 172.21.15.167:0/3268466528 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:27 5.15 5:a8000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x7f105801f040 con 0x55797b0bb5b0 2023-10-05T19:28:32.125 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.118+0000 7f106d9d5700 1 -- 172.21.15.167:0/3268466528 <== osd.3 v2:172.21.15.167:6808/30036 7 ==== osd_op_reply(27 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f1060025450 con 0x55797b0bb5b0 2023-10-05T19:28:32.125 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.118+0000 7f106d9d5700 1 -- 172.21.15.167:0/3268466528 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:28 5.d 5:b0000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x7f105801f040 con 0x7f1060010070 2023-10-05T19:28:32.125 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.118+0000 7f10651d4700 1 -- 172.21.15.167:0/3268466528 <== osd.2 v2:172.21.15.167:6816/30037 4 ==== osd_op_reply(28 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f105800f7c0 con 0x7f1060010070 2023-10-05T19:28:32.126 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.118+0000 7f10651d4700 1 -- 172.21.15.167:0/3268466528 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:29 5.1d 5:b8000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x7f105801f040 con 0x55797b0847f0 2023-10-05T19:28:32.126 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.122+0000 7f106e1d6700 1 -- 172.21.15.167:0/3268466528 <== osd.1 v2:172.21.15.167:6800/30035 12 ==== osd_op_reply(29 [pgnls start_epoch 30 out=107b] v30'2 uv2 ondisk = 1) v8 ==== 144+0+107 (crc 0 0 0) 0x7f1068022040 con 0x55797b0847f0 2023-10-05T19:28:32.127 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.122+0000 7f106e1d6700 1 -- 172.21.15.167:0/3268466528 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:30 5.3 5:c0000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x7f105801f040 con 0x7f105800ea30 2023-10-05T19:28:32.127 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.122+0000 7f106e1d6700 1 -- 172.21.15.167:0/3268466528 <== osd.0 v2:172.21.15.118:6800/27690 7 ==== osd_op_reply(30 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f1068022040 con 0x7f105800ea30 2023-10-05T19:28:32.127 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.122+0000 7f106e1d6700 1 -- 172.21.15.167:0/3268466528 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:31 5.13 5:c8000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x7f105801f040 con 0x55797b0bb5b0 2023-10-05T19:28:32.128 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.122+0000 7f106d9d5700 1 -- 172.21.15.167:0/3268466528 <== osd.3 v2:172.21.15.167:6808/30036 8 ==== osd_op_reply(31 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f1060025450 con 0x55797b0bb5b0 2023-10-05T19:28:32.128 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.122+0000 7f106d9d5700 1 -- 172.21.15.167:0/3268466528 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:32 5.b 5:d0000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x7f105801f040 con 0x7f105800ea30 2023-10-05T19:28:32.129 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.122+0000 7f106e1d6700 1 -- 172.21.15.167:0/3268466528 <== osd.0 v2:172.21.15.118:6800/27690 8 ==== osd_op_reply(32 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f1068022040 con 0x7f105800ea30 2023-10-05T19:28:32.129 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.122+0000 7f106e1d6700 1 -- 172.21.15.167:0/3268466528 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:33 5.1b 5:d8000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x7f105801f040 con 0x7f1060010070 2023-10-05T19:28:32.129 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.122+0000 7f10651d4700 1 -- 172.21.15.167:0/3268466528 <== osd.2 v2:172.21.15.167:6816/30037 5 ==== osd_op_reply(33 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f105800f7c0 con 0x7f1060010070 2023-10-05T19:28:32.130 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.122+0000 7f10651d4700 1 -- 172.21.15.167:0/3268466528 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:34 5.7 5:e0000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x7f105801f040 con 0x55797b0bb5b0 2023-10-05T19:28:32.131 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.126+0000 7f106d9d5700 1 -- 172.21.15.167:0/3268466528 <== osd.3 v2:172.21.15.167:6808/30036 9 ==== osd_op_reply(34 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f1060025450 con 0x55797b0bb5b0 2023-10-05T19:28:32.131 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.126+0000 7f106d9d5700 1 -- 172.21.15.167:0/3268466528 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:35 5.17 5:e8000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x7f105801f040 con 0x7f105800ea30 2023-10-05T19:28:32.131 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.126+0000 7f106e1d6700 1 -- 172.21.15.167:0/3268466528 <== osd.0 v2:172.21.15.118:6800/27690 9 ==== osd_op_reply(35 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f1068022040 con 0x7f105800ea30 2023-10-05T19:28:32.131 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.126+0000 7f106e1d6700 1 -- 172.21.15.167:0/3268466528 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:36 5.f 5:f0000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x7f105801f040 con 0x55797b0847f0 2023-10-05T19:28:32.132 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.126+0000 7f106e1d6700 1 -- 172.21.15.167:0/3268466528 <== osd.1 v2:172.21.15.167:6800/30035 13 ==== osd_op_reply(36 [pgnls start_epoch 30 out=115b] v23'2 uv2 ondisk = 1) v8 ==== 144+0+115 (crc 0 0 0) 0x7f1068022040 con 0x55797b0847f0 2023-10-05T19:28:32.132 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.126+0000 7f106e1d6700 1 -- 172.21.15.167:0/3268466528 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:37 5.1f 5:f8000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x7f105801f040 con 0x7f1060010070 2023-10-05T19:28:32.133 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.126+0000 7f10651d4700 1 -- 172.21.15.167:0/3268466528 <== osd.2 v2:172.21.15.167:6816/30037 6 ==== osd_op_reply(37 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f105800f7c0 con 0x7f1060010070 2023-10-05T19:28:32.133 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.126+0000 7f106fea0e40 20 RGWRados::pool_iterate: got zonegroup_info.4f1f719d-9648-417b-b10c-e5d939825419 2023-10-05T19:28:32.134 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.126+0000 7f106fea0e40 20 RGWRados::pool_iterate: got zone_info.9a9df965-6612-45d4-a45f-6e3c889bacc2 2023-10-05T19:28:32.134 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.126+0000 7f106fea0e40 20 RGWRados::pool_iterate: got zone_names.default 2023-10-05T19:28:32.134 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.126+0000 7f106fea0e40 20 RGWRados::pool_iterate: got zonegroups_names.default 2023-10-05T19:28:32.134 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.126+0000 7f106fea0e40 20 rados->read ofs=0 len=0 2023-10-05T19:28:32.134 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.126+0000 7f106fea0e40 1 -- 172.21.15.167:0/3268466528 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:38 5.12 5:49953fa1:::default.realm:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x55797b0ba360 con 0x55797b0847f0 2023-10-05T19:28:32.135 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.130+0000 7f106e1d6700 1 -- 172.21.15.167:0/3268466528 <== osd.1 v2:172.21.15.167:6800/30035 14 ==== osd_op_reply(38 default.realm [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 ==== 157+0+0 (crc 0 0 0) 0x7f1068022040 con 0x55797b0847f0 2023-10-05T19:28:32.135 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.130+0000 7f106fea0e40 20 rados_obj.operate() r=-2 bl.length=0 2023-10-05T19:28:32.135 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.130+0000 7f106fea0e40 20 rados->read ofs=0 len=0 2023-10-05T19:28:32.135 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.130+0000 7f106fea0e40 1 -- 172.21.15.167:0/3268466528 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:39 5.f 5:f4c53578:::zonegroups_names.default:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x55797b0bac50 con 0x55797b0847f0 2023-10-05T19:28:32.135 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.130+0000 7f106e1d6700 1 -- 172.21.15.167:0/3268466528 <== osd.1 v2:172.21.15.167:6800/30035 15 ==== osd_op_reply(39 zonegroups_names.default [read 0~46 out=46b] v0'0 uv2 ondisk = 0) v8 ==== 168+0+46 (crc 0 0 0) 0x7f1068022040 con 0x55797b0847f0 2023-10-05T19:28:32.135 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.130+0000 7f106fea0e40 20 rados_obj.operate() r=0 bl.length=46 2023-10-05T19:28:32.135 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.130+0000 7f106fea0e40 20 rados->read ofs=0 len=0 2023-10-05T19:28:32.135 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.130+0000 7f106fea0e40 1 -- 172.21.15.167:0/3268466528 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:40 5.c 5:30d299a5:::zonegroup_info.4f1f719d-9648-417b-b10c-e5d939825419:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x55797b0be310 con 0x7f105800ea30 2023-10-05T19:28:32.136 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.130+0000 7f106e1d6700 1 -- 172.21.15.167:0/3268466528 <== osd.0 v2:172.21.15.118:6800/27690 10 ==== osd_op_reply(40 zonegroup_info.4f1f719d-9648-417b-b10c-e5d939825419 [read 0~362 out=362b] v0'0 uv1 ondisk = 0) v8 ==== 195+0+362 (crc 0 0 0) 0x7f1068022040 con 0x7f105800ea30 2023-10-05T19:28:32.136 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.130+0000 7f106fea0e40 20 rados_obj.operate() r=0 bl.length=362 2023-10-05T19:28:32.136 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.130+0000 7f106fea0e40 20 rados->read ofs=0 len=0 2023-10-05T19:28:32.136 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.130+0000 7f106fea0e40 1 -- 172.21.15.167:0/3268466528 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:41 5.13 5:cbbf6c60:::converted:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x55797b0be310 con 0x55797b0bb5b0 2023-10-05T19:28:32.137 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.130+0000 7f106d9d5700 1 -- 172.21.15.167:0/3268466528 <== osd.3 v2:172.21.15.167:6808/30036 10 ==== osd_op_reply(41 converted [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 ==== 153+0+0 (crc 0 0 0) 0x7f1060025450 con 0x55797b0bb5b0 2023-10-05T19:28:32.137 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.130+0000 7f106fea0e40 20 rados_obj.operate() r=-2 bl.length=0 2023-10-05T19:28:32.137 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.130+0000 7f106fea0e40 20 rados->read ofs=0 len=0 2023-10-05T19:28:32.137 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.130+0000 7f106fea0e40 1 -- 172.21.15.167:0/3268466528 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:42 5.12 5:49953fa1:::default.realm:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x55797b0be6c0 con 0x55797b0847f0 2023-10-05T19:28:32.137 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.130+0000 7f106e1d6700 1 -- 172.21.15.167:0/3268466528 <== osd.1 v2:172.21.15.167:6800/30035 16 ==== osd_op_reply(42 default.realm [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 ==== 157+0+0 (crc 0 0 0) 0x7f1068022040 con 0x55797b0847f0 2023-10-05T19:28:32.138 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.130+0000 7f106fea0e40 20 rados_obj.operate() r=-2 bl.length=0 2023-10-05T19:28:32.138 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.130+0000 7f106fea0e40 20 rados->read ofs=0 len=0 2023-10-05T19:28:32.138 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.130+0000 7f106fea0e40 1 -- 172.21.15.167:0/3268466528 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:43 5.f 5:f4c53578:::zonegroups_names.default:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x55797b0ba880 con 0x55797b0847f0 2023-10-05T19:28:32.139 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.130+0000 7f106e1d6700 1 -- 172.21.15.167:0/3268466528 <== osd.1 v2:172.21.15.167:6800/30035 17 ==== osd_op_reply(43 zonegroups_names.default [read 0~46 out=46b] v0'0 uv2 ondisk = 0) v8 ==== 168+0+46 (crc 0 0 0) 0x7f1068022040 con 0x55797b0847f0 2023-10-05T19:28:32.139 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.134+0000 7f106fea0e40 20 rados_obj.operate() r=0 bl.length=46 2023-10-05T19:28:32.139 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.134+0000 7f106fea0e40 1 -- 172.21.15.167:0/3268466528 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:44 5.0 5:00000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x55797b0ba850 con 0x55797b0bb5b0 2023-10-05T19:28:32.140 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.134+0000 7f106d9d5700 1 -- 172.21.15.167:0/3268466528 <== osd.3 v2:172.21.15.167:6808/30036 11 ==== osd_op_reply(44 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f1060025450 con 0x55797b0bb5b0 2023-10-05T19:28:32.141 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.134+0000 7f106d9d5700 1 -- 172.21.15.167:0/3268466528 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:45 5.10 5:08000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x55797b0ba850 con 0x7f1060010070 2023-10-05T19:28:32.141 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.134+0000 7f10651d4700 1 -- 172.21.15.167:0/3268466528 <== osd.2 v2:172.21.15.167:6816/30037 7 ==== osd_op_reply(45 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f105800f7c0 con 0x7f1060010070 2023-10-05T19:28:32.142 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.134+0000 7f10651d4700 1 -- 172.21.15.167:0/3268466528 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:46 5.8 5:10000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x55797b0ba850 con 0x7f105800ea30 2023-10-05T19:28:32.142 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.138+0000 7f106e1d6700 1 -- 172.21.15.167:0/3268466528 <== osd.0 v2:172.21.15.118:6800/27690 11 ==== osd_op_reply(46 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f1068022040 con 0x7f105800ea30 2023-10-05T19:28:32.142 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.138+0000 7f106e1d6700 1 -- 172.21.15.167:0/3268466528 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:47 5.18 5:18000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x55797b0bac50 con 0x55797b0847f0 2023-10-05T19:28:32.143 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.138+0000 7f106e1d6700 1 -- 172.21.15.167:0/3268466528 <== osd.1 v2:172.21.15.167:6800/30035 18 ==== osd_op_reply(47 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f1068022040 con 0x55797b0847f0 2023-10-05T19:28:32.143 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.138+0000 7f106e1d6700 1 -- 172.21.15.167:0/3268466528 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:48 5.4 5:20000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x55797b0bac50 con 0x55797b0847f0 2023-10-05T19:28:32.144 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.138+0000 7f106e1d6700 1 -- 172.21.15.167:0/3268466528 <== osd.1 v2:172.21.15.167:6800/30035 19 ==== osd_op_reply(48 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f1068022040 con 0x55797b0847f0 2023-10-05T19:28:32.144 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.138+0000 7f106e1d6700 1 -- 172.21.15.167:0/3268466528 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:49 5.14 5:28000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x55797b0bac50 con 0x7f105800ea30 2023-10-05T19:28:32.145 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.138+0000 7f106e1d6700 1 -- 172.21.15.167:0/3268466528 <== osd.0 v2:172.21.15.118:6800/27690 12 ==== osd_op_reply(49 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f1068022040 con 0x7f105800ea30 2023-10-05T19:28:32.145 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.138+0000 7f106e1d6700 1 -- 172.21.15.167:0/3268466528 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:50 5.c 5:30000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x55797b0bac50 con 0x7f105800ea30 2023-10-05T19:28:32.146 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.142+0000 7f106e1d6700 1 -- 172.21.15.167:0/3268466528 <== osd.0 v2:172.21.15.118:6800/27690 13 ==== osd_op_reply(50 [pgnls start_epoch 30 out=112b] v23'1 uv1 ondisk = 1) v8 ==== 144+0+112 (crc 0 0 0) 0x7f1068022040 con 0x7f105800ea30 2023-10-05T19:28:32.146 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.142+0000 7f106e1d6700 1 -- 172.21.15.167:0/3268466528 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:51 5.1c 5:38000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x55797b0bac50 con 0x7f1060010070 2023-10-05T19:28:32.147 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.142+0000 7f10651d4700 1 -- 172.21.15.167:0/3268466528 <== osd.2 v2:172.21.15.167:6816/30037 8 ==== osd_op_reply(51 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f105800f7c0 con 0x7f1060010070 2023-10-05T19:28:32.147 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.142+0000 7f10651d4700 1 -- 172.21.15.167:0/3268466528 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:52 5.2 5:40000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x55797b0bac50 con 0x55797b0bb5b0 2023-10-05T19:28:32.147 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.142+0000 7f106d9d5700 1 -- 172.21.15.167:0/3268466528 <== osd.3 v2:172.21.15.167:6808/30036 12 ==== osd_op_reply(52 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f1060025450 con 0x55797b0bb5b0 2023-10-05T19:28:32.148 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.142+0000 7f106d9d5700 1 -- 172.21.15.167:0/3268466528 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:53 5.12 5:48000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x55797b0bac50 con 0x55797b0847f0 2023-10-05T19:28:32.148 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.142+0000 7f106e1d6700 1 -- 172.21.15.167:0/3268466528 <== osd.1 v2:172.21.15.167:6800/30035 20 ==== osd_op_reply(53 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f1068022040 con 0x55797b0847f0 2023-10-05T19:28:32.148 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.142+0000 7f106e1d6700 1 -- 172.21.15.167:0/3268466528 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:54 5.a 5:50000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x55797b0bac50 con 0x7f105800ea30 2023-10-05T19:28:32.149 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.142+0000 7f106e1d6700 1 -- 172.21.15.167:0/3268466528 <== osd.0 v2:172.21.15.118:6800/27690 14 ==== osd_op_reply(54 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f1068022040 con 0x7f105800ea30 2023-10-05T19:28:32.149 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.142+0000 7f106e1d6700 1 -- 172.21.15.167:0/3268466528 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:55 5.1a 5:58000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x55797b0bac50 con 0x55797b0847f0 2023-10-05T19:28:32.150 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.146+0000 7f106e1d6700 1 -- 172.21.15.167:0/3268466528 <== osd.1 v2:172.21.15.167:6800/30035 21 ==== osd_op_reply(55 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f1068022040 con 0x55797b0847f0 2023-10-05T19:28:32.150 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.146+0000 7f106e1d6700 1 -- 172.21.15.167:0/3268466528 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:56 5.6 5:60000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x55797b0bac50 con 0x7f1060010070 2023-10-05T19:28:32.151 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.146+0000 7f10651d4700 1 -- 172.21.15.167:0/3268466528 <== osd.2 v2:172.21.15.167:6816/30037 9 ==== osd_op_reply(56 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f105800f7c0 con 0x7f1060010070 2023-10-05T19:28:32.151 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.146+0000 7f10651d4700 1 -- 172.21.15.167:0/3268466528 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:57 5.16 5:68000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x55797b0bac50 con 0x55797b0bb5b0 2023-10-05T19:28:32.151 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.146+0000 7f106d9d5700 1 -- 172.21.15.167:0/3268466528 <== osd.3 v2:172.21.15.167:6808/30036 13 ==== osd_op_reply(57 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f1060025450 con 0x55797b0bb5b0 2023-10-05T19:28:32.152 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.146+0000 7f106d9d5700 1 -- 172.21.15.167:0/3268466528 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:58 5.e 5:70000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x55797b0bac50 con 0x55797b0bb5b0 2023-10-05T19:28:32.152 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.146+0000 7f106d9d5700 1 -- 172.21.15.167:0/3268466528 <== osd.3 v2:172.21.15.167:6808/30036 14 ==== osd_op_reply(58 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f1060025450 con 0x55797b0bb5b0 2023-10-05T19:28:32.153 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.146+0000 7f106d9d5700 1 -- 172.21.15.167:0/3268466528 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:59 5.1e 5:78000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x55797b0bac50 con 0x7f105800ea30 2023-10-05T19:28:32.153 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.146+0000 7f106e1d6700 1 -- 172.21.15.167:0/3268466528 <== osd.0 v2:172.21.15.118:6800/27690 15 ==== osd_op_reply(59 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f1068022040 con 0x7f105800ea30 2023-10-05T19:28:32.154 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.146+0000 7f106e1d6700 1 -- 172.21.15.167:0/3268466528 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:60 5.1 5:80000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x55797b0bac50 con 0x55797b0bb5b0 2023-10-05T19:28:32.154 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.150+0000 7f106d9d5700 1 -- 172.21.15.167:0/3268466528 <== osd.3 v2:172.21.15.167:6808/30036 15 ==== osd_op_reply(60 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f1060025450 con 0x55797b0bb5b0 2023-10-05T19:28:32.154 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.150+0000 7f106d9d5700 1 -- 172.21.15.167:0/3268466528 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:61 5.11 5:88000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x55797b0bac50 con 0x55797b0bb5b0 2023-10-05T19:28:32.155 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.150+0000 7f106d9d5700 1 -- 172.21.15.167:0/3268466528 <== osd.3 v2:172.21.15.167:6808/30036 16 ==== osd_op_reply(61 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f1060025450 con 0x55797b0bb5b0 2023-10-05T19:28:32.155 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.150+0000 7f106d9d5700 1 -- 172.21.15.167:0/3268466528 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:62 5.9 5:90000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x55797b0bac50 con 0x55797b0847f0 2023-10-05T19:28:32.156 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.150+0000 7f106e1d6700 1 -- 172.21.15.167:0/3268466528 <== osd.1 v2:172.21.15.167:6800/30035 22 ==== osd_op_reply(62 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f1068022040 con 0x55797b0847f0 2023-10-05T19:28:32.156 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.150+0000 7f106e1d6700 1 -- 172.21.15.167:0/3268466528 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:63 5.19 5:98000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x55797b0bac50 con 0x55797b0847f0 2023-10-05T19:28:32.156 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.150+0000 7f106e1d6700 1 -- 172.21.15.167:0/3268466528 <== osd.1 v2:172.21.15.167:6800/30035 23 ==== osd_op_reply(63 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f1068022040 con 0x55797b0847f0 2023-10-05T19:28:32.156 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.150+0000 7f106e1d6700 1 -- 172.21.15.167:0/3268466528 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:64 5.5 5:a0000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x55797b0bac50 con 0x7f105800ea30 2023-10-05T19:28:32.158 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.150+0000 7f106e1d6700 1 -- 172.21.15.167:0/3268466528 <== osd.0 v2:172.21.15.118:6800/27690 16 ==== osd_op_reply(64 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f1068022040 con 0x7f105800ea30 2023-10-05T19:28:32.158 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.150+0000 7f106e1d6700 1 -- 172.21.15.167:0/3268466528 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:65 5.15 5:a8000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x55797b0bac50 con 0x55797b0bb5b0 2023-10-05T19:28:32.158 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.154+0000 7f106d9d5700 1 -- 172.21.15.167:0/3268466528 <== osd.3 v2:172.21.15.167:6808/30036 17 ==== osd_op_reply(65 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f1060025450 con 0x55797b0bb5b0 2023-10-05T19:28:32.158 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.154+0000 7f106d9d5700 1 -- 172.21.15.167:0/3268466528 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:66 5.d 5:b0000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x55797b0bac50 con 0x7f1060010070 2023-10-05T19:28:32.159 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.154+0000 7f10651d4700 1 -- 172.21.15.167:0/3268466528 <== osd.2 v2:172.21.15.167:6816/30037 10 ==== osd_op_reply(66 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f105800f7c0 con 0x7f1060010070 2023-10-05T19:28:32.159 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.154+0000 7f10651d4700 1 -- 172.21.15.167:0/3268466528 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:67 5.1d 5:b8000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x55797b0bac50 con 0x55797b0847f0 2023-10-05T19:28:32.160 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.154+0000 7f106e1d6700 1 -- 172.21.15.167:0/3268466528 <== osd.1 v2:172.21.15.167:6800/30035 24 ==== osd_op_reply(67 [pgnls start_epoch 30 out=107b] v30'2 uv2 ondisk = 1) v8 ==== 144+0+107 (crc 0 0 0) 0x7f1068022040 con 0x55797b0847f0 2023-10-05T19:28:32.160 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.154+0000 7f106e1d6700 1 -- 172.21.15.167:0/3268466528 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:68 5.3 5:c0000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x55797b0bac50 con 0x7f105800ea30 2023-10-05T19:28:32.161 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.154+0000 7f106e1d6700 1 -- 172.21.15.167:0/3268466528 <== osd.0 v2:172.21.15.118:6800/27690 17 ==== osd_op_reply(68 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f1068022040 con 0x7f105800ea30 2023-10-05T19:28:32.161 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.154+0000 7f106e1d6700 1 -- 172.21.15.167:0/3268466528 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:69 5.13 5:c8000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x55797b0bac50 con 0x55797b0bb5b0 2023-10-05T19:28:32.162 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.158+0000 7f106d9d5700 1 -- 172.21.15.167:0/3268466528 <== osd.3 v2:172.21.15.167:6808/30036 18 ==== osd_op_reply(69 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f1060025450 con 0x55797b0bb5b0 2023-10-05T19:28:32.162 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.158+0000 7f106d9d5700 1 -- 172.21.15.167:0/3268466528 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:70 5.b 5:d0000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x55797b0bac50 con 0x7f105800ea30 2023-10-05T19:28:32.163 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.158+0000 7f106e1d6700 1 -- 172.21.15.167:0/3268466528 <== osd.0 v2:172.21.15.118:6800/27690 18 ==== osd_op_reply(70 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f1068022040 con 0x7f105800ea30 2023-10-05T19:28:32.163 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.158+0000 7f106e1d6700 1 -- 172.21.15.167:0/3268466528 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:71 5.1b 5:d8000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x55797b0bac50 con 0x7f1060010070 2023-10-05T19:28:32.164 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.158+0000 7f10651d4700 1 -- 172.21.15.167:0/3268466528 <== osd.2 v2:172.21.15.167:6816/30037 11 ==== osd_op_reply(71 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f105800f7c0 con 0x7f1060010070 2023-10-05T19:28:32.164 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.158+0000 7f10651d4700 1 -- 172.21.15.167:0/3268466528 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:72 5.7 5:e0000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x55797b0bac50 con 0x55797b0bb5b0 2023-10-05T19:28:32.165 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.158+0000 7f106d9d5700 1 -- 172.21.15.167:0/3268466528 <== osd.3 v2:172.21.15.167:6808/30036 19 ==== osd_op_reply(72 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f1060025450 con 0x55797b0bb5b0 2023-10-05T19:28:32.165 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.158+0000 7f106d9d5700 1 -- 172.21.15.167:0/3268466528 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:73 5.17 5:e8000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x55797b0bac50 con 0x7f105800ea30 2023-10-05T19:28:32.165 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.158+0000 7f106e1d6700 1 -- 172.21.15.167:0/3268466528 <== osd.0 v2:172.21.15.118:6800/27690 19 ==== osd_op_reply(73 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f1068022040 con 0x7f105800ea30 2023-10-05T19:28:32.166 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.158+0000 7f106e1d6700 1 -- 172.21.15.167:0/3268466528 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:74 5.f 5:f0000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x55797b0bac50 con 0x55797b0847f0 2023-10-05T19:28:32.166 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.162+0000 7f106e1d6700 1 -- 172.21.15.167:0/3268466528 <== osd.1 v2:172.21.15.167:6800/30035 25 ==== osd_op_reply(74 [pgnls start_epoch 30 out=115b] v23'2 uv2 ondisk = 1) v8 ==== 144+0+115 (crc 0 0 0) 0x7f1068022040 con 0x55797b0847f0 2023-10-05T19:28:32.167 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.162+0000 7f106e1d6700 1 -- 172.21.15.167:0/3268466528 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:75 5.1f 5:f8000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x55797b0bac50 con 0x7f1060010070 2023-10-05T19:28:32.168 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.162+0000 7f10651d4700 1 -- 172.21.15.167:0/3268466528 <== osd.2 v2:172.21.15.167:6816/30037 12 ==== osd_op_reply(75 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f105800f7c0 con 0x7f1060010070 2023-10-05T19:28:32.168 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.162+0000 7f106fea0e40 20 RGWRados::pool_iterate: got zonegroup_info.4f1f719d-9648-417b-b10c-e5d939825419 2023-10-05T19:28:32.168 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.162+0000 7f106fea0e40 20 RGWRados::pool_iterate: got zone_info.9a9df965-6612-45d4-a45f-6e3c889bacc2 2023-10-05T19:28:32.168 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.162+0000 7f106fea0e40 20 RGWRados::pool_iterate: got zone_names.default 2023-10-05T19:28:32.168 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.162+0000 7f106fea0e40 20 RGWRados::pool_iterate: got zonegroups_names.default 2023-10-05T19:28:32.168 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.162+0000 7f106fea0e40 20 rados->read ofs=0 len=0 2023-10-05T19:28:32.168 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.162+0000 7f106fea0e40 1 -- 172.21.15.167:0/3268466528 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:76 5.f 5:f43b6ece:::zone_names.default:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x55797b0ba850 con 0x55797b0847f0 2023-10-05T19:28:32.170 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.162+0000 7f106e1d6700 1 -- 172.21.15.167:0/3268466528 <== osd.1 v2:172.21.15.167:6800/30035 26 ==== osd_op_reply(76 zone_names.default [read 0~46 out=46b] v0'0 uv1 ondisk = 0) v8 ==== 162+0+46 (crc 0 0 0) 0x7f1068022040 con 0x55797b0847f0 2023-10-05T19:28:32.170 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.162+0000 7f106fea0e40 20 rados_obj.operate() r=0 bl.length=46 2023-10-05T19:28:32.170 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.162+0000 7f106fea0e40 20 rados->read ofs=0 len=0 2023-10-05T19:28:32.170 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.162+0000 7f106fea0e40 1 -- 172.21.15.167:0/3268466528 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:77 5.1d 5:b8374eb4:::zone_info.9a9df965-6612-45d4-a45f-6e3c889bacc2:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x55797b0bec30 con 0x55797b0847f0 2023-10-05T19:28:32.170 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.162+0000 7f106e1d6700 1 -- 172.21.15.167:0/3268466528 <== osd.1 v2:172.21.15.167:6800/30035 27 ==== osd_op_reply(77 zone_info.9a9df965-6612-45d4-a45f-6e3c889bacc2 [read 0~874 out=874b] v0'0 uv2 ondisk = 0) v8 ==== 190+0+874 (crc 0 0 0) 0x7f1068022040 con 0x55797b0847f0 2023-10-05T19:28:32.170 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.162+0000 7f106fea0e40 20 rados_obj.operate() r=0 bl.length=874 2023-10-05T19:28:32.170 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.162+0000 7f106fea0e40 20 rados->read ofs=0 len=0 2023-10-05T19:28:32.170 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.162+0000 7f106fea0e40 1 -- 172.21.15.167:0/3268466528 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:78 5.f 5:f4c53578:::zonegroups_names.default:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x55797b0bf120 con 0x55797b0847f0 2023-10-05T19:28:32.171 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.162+0000 7f106e1d6700 1 -- 172.21.15.167:0/3268466528 <== osd.1 v2:172.21.15.167:6800/30035 28 ==== osd_op_reply(78 zonegroups_names.default [read 0~46 out=46b] v0'0 uv2 ondisk = 0) v8 ==== 168+0+46 (crc 0 0 0) 0x7f1068022040 con 0x55797b0847f0 2023-10-05T19:28:32.171 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.162+0000 7f106fea0e40 20 rados_obj.operate() r=0 bl.length=46 2023-10-05T19:28:32.171 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.162+0000 7f106fea0e40 20 rados->read ofs=0 len=0 2023-10-05T19:28:32.171 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.162+0000 7f106fea0e40 1 -- 172.21.15.167:0/3268466528 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:79 5.c 5:30d299a5:::zonegroup_info.4f1f719d-9648-417b-b10c-e5d939825419:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x55797b0bf120 con 0x7f105800ea30 2023-10-05T19:28:32.172 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.166+0000 7f106e1d6700 1 -- 172.21.15.167:0/3268466528 <== osd.0 v2:172.21.15.118:6800/27690 20 ==== osd_op_reply(79 zonegroup_info.4f1f719d-9648-417b-b10c-e5d939825419 [read 0~362 out=362b] v0'0 uv1 ondisk = 0) v8 ==== 195+0+362 (crc 0 0 0) 0x7f1068022040 con 0x7f105800ea30 2023-10-05T19:28:32.172 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.166+0000 7f106fea0e40 20 rados_obj.operate() r=0 bl.length=362 2023-10-05T19:28:32.172 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.166+0000 7f106fea0e40 20 rados->read ofs=0 len=0 2023-10-05T19:28:32.172 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.166+0000 7f106fea0e40 1 -- 172.21.15.167:0/3268466528 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:80 5.3 5:c0ad8cbd:::region_map:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x55797b0bf220 con 0x7f105800ea30 2023-10-05T19:28:32.172 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.166+0000 7f106e1d6700 1 -- 172.21.15.167:0/3268466528 <== osd.0 v2:172.21.15.118:6800/27690 21 ==== osd_op_reply(80 region_map [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 ==== 154+0+0 (crc 0 0 0) 0x7f1068022040 con 0x7f105800ea30 2023-10-05T19:28:32.172 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.166+0000 7f106fea0e40 20 rados_obj.operate() r=-2 bl.length=0 2023-10-05T19:28:32.173 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.166+0000 7f106fea0e40 20 zone default found 2023-10-05T19:28:32.173 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.166+0000 7f106fea0e40 4 Realm: () 2023-10-05T19:28:32.173 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.166+0000 7f106fea0e40 4 ZoneGroup: default (4f1f719d-9648-417b-b10c-e5d939825419) 2023-10-05T19:28:32.173 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.166+0000 7f106fea0e40 4 Zone: default (9a9df965-6612-45d4-a45f-6e3c889bacc2) 2023-10-05T19:28:32.173 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.166+0000 7f106fea0e40 10 cannot find current period zonegroup using local zonegroup configuration 2023-10-05T19:28:32.173 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.166+0000 7f106fea0e40 20 zonegroup default 2023-10-05T19:28:32.173 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.166+0000 7f106fea0e40 20 rados->read ofs=0 len=0 2023-10-05T19:28:32.173 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.166+0000 7f106fea0e40 1 -- 172.21.15.167:0/3268466528 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:81 5.3 5:c52100b6:::period_config.default:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x55797b0bf220 con 0x7f105800ea30 2023-10-05T19:28:32.175 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.166+0000 7f106e1d6700 1 -- 172.21.15.167:0/3268466528 <== osd.0 v2:172.21.15.118:6800/27690 22 ==== osd_op_reply(81 period_config.default [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 ==== 165+0+0 (crc 0 0 0) 0x7f1068045040 con 0x7f105800ea30 2023-10-05T19:28:32.175 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.166+0000 7f106fea0e40 20 rados_obj.operate() r=-2 bl.length=0 2023-10-05T19:28:32.175 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.166+0000 7f106fea0e40 20 rados->read ofs=0 len=0 2023-10-05T19:28:32.175 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.166+0000 7f106fea0e40 1 -- 172.21.15.167:0/3268466528 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:82 6.2 6:4347d321:::bucket.sync-source-hints.:head [call version.read in=11b,read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x55797b0c0ba0 con 0x55797b0bb5b0 2023-10-05T19:28:32.175 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.166+0000 7f106d9d5700 1 -- 172.21.15.167:0/3268466528 <== osd.3 v2:172.21.15.167:6808/30036 20 ==== osd_op_reply(82 bucket.sync-source-hints. [call,read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 ==== 211+0+0 (crc 0 0 0) 0x7f1060025450 con 0x55797b0bb5b0 2023-10-05T19:28:32.175 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.166+0000 7f106fea0e40 20 rados_obj.operate() r=-2 bl.length=0 2023-10-05T19:28:32.175 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.166+0000 7f106fea0e40 20 rados->read ofs=0 len=0 2023-10-05T19:28:32.176 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.166+0000 7f106fea0e40 1 -- 172.21.15.167:0/3268466528 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:83 6.b 6:d467b91b:::bucket.sync-target-hints.:head [call version.read in=11b,read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x55797b0c0f50 con 0x55797b0bb5b0 2023-10-05T19:28:32.176 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.170+0000 7f106d9d5700 1 -- 172.21.15.167:0/3268466528 <== osd.3 v2:172.21.15.167:6808/30036 21 ==== osd_op_reply(83 bucket.sync-target-hints. [call,read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 ==== 211+0+0 (crc 0 0 0) 0x7f1060025450 con 0x55797b0bb5b0 2023-10-05T19:28:32.176 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.170+0000 7f106fea0e40 20 rados_obj.operate() r=-2 bl.length=0 2023-10-05T19:28:32.176 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.170+0000 7f106fea0e40 20 started sync module instance, tier type = 2023-10-05T19:28:32.176 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.170+0000 7f106fea0e40 20 started zone id=9a9df965-6612-45d4-a45f-6e3c889bacc2 (name=default) with tier type = 2023-10-05T19:28:32.176 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.170+0000 7f106fea0e40 1 -- 172.21.15.167:0/3268466528 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:84 7.1f 7:f95f44c2:::notify.0:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x55797b0c0260 con 0x7f1060010070 2023-10-05T19:28:32.177 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.170+0000 7f10651d4700 1 -- 172.21.15.167:0/3268466528 <== osd.2 v2:172.21.15.167:6816/30037 13 ==== osd_op_reply(84 notify.0 [create] v30'105 uv105 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f105800f7c0 con 0x7f1060010070 2023-10-05T19:28:32.177 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.170+0000 7f106fea0e40 1 -- 172.21.15.167:0/3268466528 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:85 7.1f 7:f95f44c2:::notify.0:head [watch watch cookie 93980243789328] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x55797b0bfe90 con 0x7f1060010070 2023-10-05T19:28:32.177 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.170+0000 7f106fea0e40 1 -- 172.21.15.167:0/3268466528 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:86 7.0 7:05bf5b68:::notify.1:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x55797b0c1ce0 con 0x55797b0847f0 2023-10-05T19:28:32.180 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.174+0000 7f10651d4700 1 -- 172.21.15.167:0/3268466528 <== osd.2 v2:172.21.15.167:6816/30037 14 ==== osd_op_reply(85 notify.0 [watch watch cookie 93980243789328] v30'106 uv105 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f105800f7c0 con 0x7f1060010070 2023-10-05T19:28:32.181 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.174+0000 7f106e1d6700 1 -- 172.21.15.167:0/3268466528 <== osd.1 v2:172.21.15.167:6800/30035 29 ==== osd_op_reply(86 notify.1 [create] v30'105 uv105 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f1068045040 con 0x55797b0847f0 2023-10-05T19:28:32.181 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.174+0000 7f106fea0e40 1 -- 172.21.15.167:0/3268466528 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:87 7.0 7:05bf5b68:::notify.1:head [watch watch cookie 93980243792928] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x55797b0c1d60 con 0x55797b0847f0 2023-10-05T19:28:32.181 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.174+0000 7f106fea0e40 1 -- 172.21.15.167:0/3268466528 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:88 7.15 7:a93a5511:::notify.2:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x55797b0c27b0 con 0x55797b0bb5b0 2023-10-05T19:28:32.183 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.178+0000 7f106e1d6700 1 -- 172.21.15.167:0/3268466528 <== osd.1 v2:172.21.15.167:6800/30035 30 ==== osd_op_reply(87 notify.1 [watch watch cookie 93980243792928] v30'106 uv105 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f1068045040 con 0x55797b0847f0 2023-10-05T19:28:32.184 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.178+0000 7f106d9d5700 1 -- 172.21.15.167:0/3268466528 <== osd.3 v2:172.21.15.167:6808/30036 22 ==== osd_op_reply(88 notify.2 [create] v30'105 uv105 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f1060025450 con 0x55797b0bb5b0 2023-10-05T19:28:32.184 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.178+0000 7f106fea0e40 1 -- 172.21.15.167:0/3268466528 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:89 7.15 7:a93a5511:::notify.2:head [watch watch cookie 93980243794032] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x55797b0c1cc0 con 0x55797b0bb5b0 2023-10-05T19:28:32.184 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.178+0000 7f106fea0e40 1 -- 172.21.15.167:0/3268466528 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:90 7.e 7:7759931f:::notify.3:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x55797b0c3590 con 0x7f1060010070 2023-10-05T19:28:32.186 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.182+0000 7f10651d4700 1 -- 172.21.15.167:0/3268466528 <== osd.2 v2:172.21.15.167:6816/30037 15 ==== osd_op_reply(90 notify.3 [create] v30'105 uv105 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f105800f7c0 con 0x7f1060010070 2023-10-05T19:28:32.187 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.182+0000 7f106fea0e40 1 -- 172.21.15.167:0/3268466528 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:91 7.e 7:7759931f:::notify.3:head [watch watch cookie 93980243802432] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x55797b0c31c0 con 0x7f1060010070 2023-10-05T19:28:32.187 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.182+0000 7f106fea0e40 1 -- 172.21.15.167:0/3268466528 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:92 7.d 7:b4812045:::notify.4:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x55797b0c4a60 con 0x55797b0847f0 2023-10-05T19:28:32.187 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.182+0000 7f106d9d5700 1 -- 172.21.15.167:0/3268466528 <== osd.3 v2:172.21.15.167:6808/30036 23 ==== osd_op_reply(89 notify.2 [watch watch cookie 93980243794032] v30'106 uv105 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f1060025450 con 0x55797b0bb5b0 2023-10-05T19:28:32.188 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.182+0000 7f10651d4700 1 -- 172.21.15.167:0/3268466528 <== osd.2 v2:172.21.15.167:6816/30037 16 ==== osd_op_reply(91 notify.3 [watch watch cookie 93980243802432] v30'106 uv105 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f105800f7c0 con 0x7f1060010070 2023-10-05T19:28:32.189 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.182+0000 7f106e1d6700 1 -- 172.21.15.167:0/3268466528 <== osd.1 v2:172.21.15.167:6800/30035 31 ==== osd_op_reply(92 notify.4 [create] v30'105 uv105 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f1068045040 con 0x55797b0847f0 2023-10-05T19:28:32.189 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.182+0000 7f106fea0e40 1 -- 172.21.15.167:0/3268466528 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:93 7.d 7:b4812045:::notify.4:head [watch watch cookie 93980243803536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x55797b0c4530 con 0x55797b0847f0 2023-10-05T19:28:32.189 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.182+0000 7f106fea0e40 1 -- 172.21.15.167:0/3268466528 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:94 7.3 7:c609908c:::notify.5:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x55797b0c5350 con 0x7f105800ea30 2023-10-05T19:28:32.191 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.186+0000 7f106e1d6700 1 -- 172.21.15.167:0/3268466528 <== osd.1 v2:172.21.15.167:6800/30035 32 ==== osd_op_reply(93 notify.4 [watch watch cookie 93980243803536] v30'106 uv105 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f1068045040 con 0x55797b0847f0 2023-10-05T19:28:32.192 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.186+0000 7f106e1d6700 1 -- 172.21.15.167:0/3268466528 <== osd.0 v2:172.21.15.118:6800/27690 23 ==== osd_op_reply(94 notify.5 [create] v30'105 uv105 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f1068045040 con 0x7f105800ea30 2023-10-05T19:28:32.192 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.186+0000 7f106fea0e40 1 -- 172.21.15.167:0/3268466528 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:95 7.3 7:c609908c:::notify.5:head [watch watch cookie 93980243798960] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x55797b0c5300 con 0x7f105800ea30 2023-10-05T19:28:32.192 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.186+0000 7f106fea0e40 1 -- 172.21.15.167:0/3268466528 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:96 7.14 7:2b04a3e9:::notify.6:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x55797b0c5d10 con 0x7f105800ea30 2023-10-05T19:28:32.195 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.190+0000 7f106e1d6700 1 -- 172.21.15.167:0/3268466528 <== osd.0 v2:172.21.15.118:6800/27690 24 ==== osd_op_reply(95 notify.5 [watch watch cookie 93980243798960] v30'106 uv105 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f1068045040 con 0x7f105800ea30 2023-10-05T19:28:32.195 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.190+0000 7f106e1d6700 1 -- 172.21.15.167:0/3268466528 <== osd.0 v2:172.21.15.118:6800/27690 25 ==== osd_op_reply(96 notify.6 [create] v30'105 uv105 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f1068045040 con 0x7f105800ea30 2023-10-05T19:28:32.195 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.190+0000 7f106fea0e40 1 -- 172.21.15.167:0/3268466528 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:97 7.14 7:2b04a3e9:::notify.6:head [watch watch cookie 93980243808032] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x55797b0c5d10 con 0x7f105800ea30 2023-10-05T19:28:32.195 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.190+0000 7f106fea0e40 1 -- 172.21.15.167:0/3268466528 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:98 7.9 7:93e5b521:::notify.7:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x55797b0c6720 con 0x7f105800ea30 2023-10-05T19:28:32.198 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.194+0000 7f106e1d6700 1 -- 172.21.15.167:0/3268466528 <== osd.0 v2:172.21.15.118:6800/27690 26 ==== osd_op_reply(98 notify.7 [create] v30'105 uv105 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f1068045040 con 0x7f105800ea30 2023-10-05T19:28:32.198 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.194+0000 7f106e1d6700 1 -- 172.21.15.167:0/3268466528 <== osd.0 v2:172.21.15.118:6800/27690 27 ==== osd_op_reply(97 notify.6 [watch watch cookie 93980243808032] v30'106 uv105 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f1068045040 con 0x7f105800ea30 2023-10-05T19:28:32.198 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.194+0000 7f106fea0e40 1 -- 172.21.15.167:0/3268466528 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:99 7.9 7:93e5b521:::notify.7:head [watch watch cookie 93980243812912] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x55797b0c6680 con 0x7f105800ea30 2023-10-05T19:28:32.198 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.194+0000 7f106fea0e40 20 add_watcher() i=0 2023-10-05T19:28:32.199 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.194+0000 7f106fea0e40 20 add_watcher() i=1 2023-10-05T19:28:32.199 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.194+0000 7f106fea0e40 20 add_watcher() i=2 2023-10-05T19:28:32.199 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.194+0000 7f106fea0e40 20 add_watcher() i=3 2023-10-05T19:28:32.199 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.194+0000 7f106fea0e40 20 add_watcher() i=4 2023-10-05T19:28:32.199 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.194+0000 7f106fea0e40 20 add_watcher() i=5 2023-10-05T19:28:32.199 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.194+0000 7f106fea0e40 20 add_watcher() i=6 2023-10-05T19:28:32.201 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.194+0000 7f106e1d6700 1 -- 172.21.15.167:0/3268466528 <== osd.0 v2:172.21.15.118:6800/27690 28 ==== osd_op_reply(99 notify.7 [watch watch cookie 93980243812912] v30'106 uv105 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f1068045040 con 0x7f105800ea30 2023-10-05T19:28:32.201 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.194+0000 7f106fea0e40 20 add_watcher() i=7 2023-10-05T19:28:32.201 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.194+0000 7f106fea0e40 2 all 8 watchers are set, enabling cache 2023-10-05T19:28:32.201 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.194+0000 7f106fea0e40 1 -- 172.21.15.167:0/3268466528 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:100 6.4 6:22d26bf9:::data_loggenerations_metadata:head [call version.check_conds in=74b,call version.read in=11b,read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x55797b0c9ba0 con 0x55797b0847f0 2023-10-05T19:28:32.201 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.194+0000 7f106e1d6700 1 -- 172.21.15.167:0/3268466528 <== osd.1 v2:172.21.15.167:6800/30035 33 ==== osd_op_reply(100 data_loggenerations_metadata [call,call out=48b,read 0~28 out=28b] v0'0 uv1 ondisk = 0) v8 ==== 256+0+76 (crc 0 0 0) 0x7f1068045040 con 0x55797b0847f0 2023-10-05T19:28:32.202 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.194+0000 7f106fea0e40 1 -- 172.21.15.167:0/3268466528 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:101 6.4 6:22d26bf9:::data_loggenerations_metadata:head [watch watch cookie 93980243814016] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x55797b0c6fc0 con 0x55797b0847f0 2023-10-05T19:28:32.203 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.198+0000 7f106e1d6700 1 -- 172.21.15.167:0/3268466528 <== osd.1 v2:172.21.15.167:6800/30035 34 ==== osd_op_reply(101 data_loggenerations_metadata [watch watch cookie 93980243814016] v30'105 uv1 ondisk = 0) v8 ==== 172+0+0 (crc 0 0 0) 0x7f1068045040 con 0x55797b0847f0 2023-10-05T19:28:32.204 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.198+0000 7f103f7de700 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2023-10-05T19:28:32.204 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.198+0000 7f106fea0e40 20 check_secure_mon_conn(): auth registy supported: methods=[2] modes=[2,1] 2023-10-05T19:28:32.204 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.198+0000 7f106fea0e40 20 check_secure_mon_conn(): mode 1 is insecure 2023-10-05T19:28:32.205 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.198+0000 7f106fea0e40 1 Processor -- start 2023-10-05T19:28:32.205 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.198+0000 7f106fea0e40 1 -- start start 2023-10-05T19:28:32.205 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.198+0000 7f106fea0e40 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55797b0d4000 0x55797b0d43d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-05T19:28:32.205 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.198+0000 7f106fea0e40 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x55797b0d4910 con 0x55797b0d4000 2023-10-05T19:28:32.205 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.198+0000 7f10651d4700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55797b0d4000 0x55797b0d43d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-05T19:28:32.205 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.198+0000 7f10651d4700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55797b0d4000 0x55797b0d43d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.118:3300/0 says I am v2:172.21.15.167:58356/0 (socket says 172.21.15.167:58356) 2023-10-05T19:28:32.205 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.198+0000 7f10651d4700 1 -- 172.21.15.167:0/1945314659 learned_addr learned my addr 172.21.15.167:0/1945314659 (peer_addr_for_me v2:172.21.15.167:0/0) 2023-10-05T19:28:32.206 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.198+0000 7f10651d4700 1 -- 172.21.15.167:0/1945314659 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55797b0d4a50 con 0x55797b0d4000 2023-10-05T19:28:32.206 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.202+0000 7f10651d4700 1 --2- 172.21.15.167:0/1945314659 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55797b0d4000 0x55797b0d43d0 secure :-1 s=READY pgs=348 cs=0 l=1 rev1=1 crypto rx=0x7f105800f610 tx=0x7f1058006000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e9529c50b7e328e server_cookie=0 in_seq=0 out_seq=0 2023-10-05T19:28:32.206 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.202+0000 7f103efdd700 1 -- 172.21.15.167:0/1945314659 <== mon.0 v2:172.21.15.118:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f105800dd10 con 0x55797b0d4000 2023-10-05T19:28:32.207 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.202+0000 7f103efdd700 1 -- 172.21.15.167:0/1945314659 <== mon.0 v2:172.21.15.118:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f105800de70 con 0x55797b0d4000 2023-10-05T19:28:32.207 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.202+0000 7f103efdd700 1 -- 172.21.15.167:0/1945314659 <== mon.0 v2:172.21.15.118:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f105803c420 con 0x55797b0d4000 2023-10-05T19:28:32.208 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.202+0000 7f106fea0e40 1 -- 172.21.15.167:0/1945314659 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55797b0d4000 msgr2=0x55797b0d43d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-05T19:28:32.208 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.202+0000 7f106fea0e40 1 --2- 172.21.15.167:0/1945314659 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55797b0d4000 0x55797b0d43d0 secure :-1 s=READY pgs=348 cs=0 l=1 rev1=1 crypto rx=0x7f105800f610 tx=0x7f1058006000 comp rx=0 tx=0).stop 2023-10-05T19:28:32.208 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.202+0000 7f106fea0e40 1 -- 172.21.15.167:0/1945314659 shutdown_connections 2023-10-05T19:28:32.208 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.202+0000 7f106fea0e40 1 --2- 172.21.15.167:0/1945314659 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55797b0d4000 0x55797b0d43d0 unknown :-1 s=CLOSED pgs=348 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-05T19:28:32.208 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.202+0000 7f106fea0e40 1 -- 172.21.15.167:0/1945314659 >> 172.21.15.167:0/1945314659 conn(0x55797b0cf3c0 msgr2=0x55797b0cd510 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-05T19:28:32.208 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.202+0000 7f106fea0e40 1 -- 172.21.15.167:0/1945314659 shutdown_connections 2023-10-05T19:28:32.208 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.202+0000 7f106fea0e40 1 -- 172.21.15.167:0/1945314659 wait complete. 2023-10-05T19:28:32.209 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.202+0000 7f106fea0e40 1 Processor -- start 2023-10-05T19:28:32.209 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.202+0000 7f106fea0e40 1 -- start start 2023-10-05T19:28:32.209 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.202+0000 7f106fea0e40 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55797b0d4000 0x55797b0d9ba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-05T19:28:32.210 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.202+0000 7f106fea0e40 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x55797b0daba0 con 0x55797b0d4000 2023-10-05T19:28:32.210 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.202+0000 7f10651d4700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55797b0d4000 0x55797b0d9ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-05T19:28:32.210 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.202+0000 7f10651d4700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55797b0d4000 0x55797b0d9ba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.118:3300/0 says I am v2:172.21.15.167:58368/0 (socket says 172.21.15.167:58368) 2023-10-05T19:28:32.210 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.202+0000 7f10651d4700 1 -- 172.21.15.167:0/4045791114 learned_addr learned my addr 172.21.15.167:0/4045791114 (peer_addr_for_me v2:172.21.15.167:0/0) 2023-10-05T19:28:32.211 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.206+0000 7f10651d4700 1 -- 172.21.15.167:0/4045791114 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1058006070 con 0x55797b0d4000 2023-10-05T19:28:32.211 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.206+0000 7f10651d4700 1 --2- 172.21.15.167:0/4045791114 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55797b0d4000 0x55797b0d9ba0 secure :-1 s=READY pgs=349 cs=0 l=1 rev1=1 crypto rx=0x7f105800f610 tx=0x7f1058031000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-05T19:28:32.211 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.206+0000 7f103d7da700 1 -- 172.21.15.167:0/4045791114 <== mon.0 v2:172.21.15.118:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f105803dcc0 con 0x55797b0d4000 2023-10-05T19:28:32.211 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.206+0000 7f103d7da700 1 -- 172.21.15.167:0/4045791114 <== mon.0 v2:172.21.15.118:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f105803de20 con 0x55797b0d4000 2023-10-05T19:28:32.211 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.206+0000 7f103d7da700 1 -- 172.21.15.167:0/4045791114 <== mon.0 v2:172.21.15.118:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f1058047470 con 0x55797b0d4000 2023-10-05T19:28:32.212 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.206+0000 7f106fea0e40 1 -- 172.21.15.167:0/4045791114 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x55797b0da1a0 con 0x55797b0d4000 2023-10-05T19:28:32.212 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.206+0000 7f106fea0e40 1 -- 172.21.15.167:0/4045791114 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55797b0da5e0 con 0x55797b0d4000 2023-10-05T19:28:32.213 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.206+0000 7f103d7da700 1 -- 172.21.15.167:0/4045791114 <== mon.0 v2:172.21.15.118:3300/0 4 ==== mgrmap(e 5) v1 ==== 40329+0+0 (secure 0 0 0) 0x7f105801a040 con 0x55797b0d4000 2023-10-05T19:28:32.213 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.206+0000 7f103d7da700 1 --2- 172.21.15.167:0/4045791114 >> [v2:172.21.15.167:6824/29889,v1:172.21.15.167:6825/29889] conn(0x7f0fb4030c50 0x7f0fb4033100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-05T19:28:32.213 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.206+0000 7f103d7da700 1 -- 172.21.15.167:0/4045791114 <== mon.0 v2:172.21.15.118:3300/0 5 ==== osd_map(30..30 src has 1..30) v4 ==== 6057+0+0 (secure 0 0 0) 0x7f1058048df0 con 0x55797b0d4000 2023-10-05T19:28:32.213 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.206+0000 7f106fea0e40 5 note: GC not initialized 2023-10-05T19:28:32.213 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.206+0000 7f106e1d6700 1 --2- 172.21.15.167:0/4045791114 >> [v2:172.21.15.167:6824/29889,v1:172.21.15.167:6825/29889] conn(0x7f0fb4030c50 0x7f0fb4033100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-05T19:28:32.213 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.206+0000 7f106e1d6700 1 --2- 172.21.15.167:0/4045791114 >> [v2:172.21.15.167:6824/29889,v1:172.21.15.167:6825/29889] conn(0x7f0fb4030c50 0x7f0fb4033100 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x55797b0c5b50 tx=0x7f1068000f90 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-05T19:28:32.214 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.210+0000 7f0fae7fc700 20 reqs_thread_entry: start 2023-10-05T19:28:32.215 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.210+0000 7f106fea0e40 20 init_complete bucket index max shards: 11 2023-10-05T19:28:32.215 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.210+0000 7f0f9ffff700 20 reqs_thread_entry: start 2023-10-05T19:28:32.216 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.210+0000 7f106fea0e40 10 rgw notify: Started notification manager with: 1 workers 2023-10-05T19:28:32.216 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.210+0000 7f0f9e7fc700 20 rgw notify: INFO: next queues processing will happen at: Thu Oct 5 19:29:02 2023 2023-10-05T19:28:32.216 INFO:teuthology.orchestra.run.smithi167.stderr: 2023-10-05T19:28:32.216 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.210+0000 7f0f9dffb700 20 reqs_thread_entry: start 2023-10-05T19:28:32.216 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.210+0000 7f106fea0e40 20 get_system_obj_state: rctx=0x7fff0d2f9938 obj=default.rgw.meta:users.uid:foo state=0x55797af5d0b0 s->prefetch_data=0 2023-10-05T19:28:32.216 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.210+0000 7f106fea0e40 10 cache get: name=default.rgw.meta+users.uid+foo : miss 2023-10-05T19:28:32.217 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.210+0000 7f106fea0e40 1 -- 172.21.15.167:0/3268466528 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:102 8.18 8:1e6ca5ed:users.uid::foo:head [call version.read in=11b,getxattrs,stat] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x55797b1423f0 con 0x55797b0847f0 2023-10-05T19:28:32.218 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.210+0000 7f106e1d6700 1 -- 172.21.15.167:0/3268466528 <== osd.1 v2:172.21.15.167:6800/30035 35 ==== osd_op_reply(102 foo [call out=48b,getxattrs out=75b,stat out=16b] v0'0 uv11 ondisk = 0) v8 ==== 231+0+139 (crc 0 0 0) 0x7f1068045040 con 0x55797b0847f0 2023-10-05T19:28:32.218 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.210+0000 7f106fea0e40 10 cache put: name=default.rgw.meta+users.uid+foo info.flags=0x16 2023-10-05T19:28:32.218 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.210+0000 7f106fea0e40 10 adding default.rgw.meta+users.uid+foo to cache LRU end 2023-10-05T19:28:32.218 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.210+0000 7f106fea0e40 10 updating xattr: name=ceph.objclass.version bl.length()=42 2023-10-05T19:28:32.219 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.210+0000 7f106fea0e40 20 get_system_obj_state: s->obj_tag was set empty 2023-10-05T19:28:32.219 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.210+0000 7f106fea0e40 20 Read xattr: user.rgw.idtag 2023-10-05T19:28:32.219 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.210+0000 7f106fea0e40 10 cache get: name=default.rgw.meta+users.uid+foo : type miss (requested=0x13, cached=0x16) 2023-10-05T19:28:32.219 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.210+0000 7f106fea0e40 20 rados->read ofs=0 len=0 2023-10-05T19:28:32.219 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.210+0000 7f106fea0e40 1 -- 172.21.15.167:0/3268466528 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:103 8.18 8:1e6ca5ed:users.uid::foo:head [call version.check_conds in=122b,call version.read in=11b,read 0~0,getxattrs] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x55797b144e80 con 0x55797b0847f0 2023-10-05T19:28:32.219 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.210+0000 7f106e1d6700 1 -- 172.21.15.167:0/3268466528 <== osd.1 v2:172.21.15.167:6800/30035 36 ==== osd_op_reply(103 foo [call,call out=48b,read 0~357 out=357b,getxattrs out=75b] v0'0 uv11 ondisk = 0) v8 ==== 273+0+480 (crc 0 0 0) 0x7f1068045040 con 0x55797b0847f0 2023-10-05T19:28:32.219 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.210+0000 7f106fea0e40 20 rados_obj.operate() r=0 bl.length=357 2023-10-05T19:28:32.219 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.210+0000 7f106fea0e40 10 cache put: name=default.rgw.meta+users.uid+foo info.flags=0x13 2023-10-05T19:28:32.219 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.210+0000 7f106fea0e40 10 moving default.rgw.meta+users.uid+foo to cache LRU end 2023-10-05T19:28:32.219 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.210+0000 7f106fea0e40 10 updating xattr: name=ceph.objclass.version bl.length()=42 2023-10-05T19:28:32.220 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.210+0000 7f106fea0e40 20 get_system_obj_state: rctx=0x7fff0d2f9980 obj=default.rgw.meta:root:myfoo state=0x55797af5d0b0 s->prefetch_data=0 2023-10-05T19:28:32.220 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.210+0000 7f106fea0e40 10 cache get: name=default.rgw.meta+root+myfoo : miss 2023-10-05T19:28:32.220 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.210+0000 7f106fea0e40 1 -- 172.21.15.167:0/3268466528 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:104 8.9 8:91b71841:root::myfoo:head [call version.read in=11b,getxattrs,stat] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x55797b144690 con 0x7f1060010070 2023-10-05T19:28:32.222 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.214+0000 7f10651d4700 1 -- 172.21.15.167:0/3268466528 <== osd.2 v2:172.21.15.167:6816/30037 17 ==== osd_op_reply(104 myfoo [call out=48b,getxattrs out=75b,stat out=16b] v0'0 uv1 ondisk = 0) v8 ==== 233+0+139 (crc 0 0 0) 0x7f105800f7c0 con 0x7f1060010070 2023-10-05T19:28:32.222 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.214+0000 7f106fea0e40 10 cache put: name=default.rgw.meta+root+myfoo info.flags=0x16 2023-10-05T19:28:32.222 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.214+0000 7f106fea0e40 10 adding default.rgw.meta+root+myfoo to cache LRU end 2023-10-05T19:28:32.222 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.214+0000 7f106fea0e40 10 updating xattr: name=ceph.objclass.version bl.length()=42 2023-10-05T19:28:32.223 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.214+0000 7f106fea0e40 20 get_system_obj_state: s->obj_tag was set empty 2023-10-05T19:28:32.223 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.214+0000 7f106fea0e40 10 cache get: name=default.rgw.meta+root+myfoo : type miss (requested=0x11, cached=0x16) 2023-10-05T19:28:32.223 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.214+0000 7f106fea0e40 20 rados->read ofs=0 len=0 2023-10-05T19:28:32.223 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.214+0000 7f106fea0e40 1 -- 172.21.15.167:0/3268466528 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:105 8.9 8:91b71841:root::myfoo:head [call version.check_conds in=122b,call version.read in=11b,read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x55797b146c30 con 0x7f1060010070 2023-10-05T19:28:32.223 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.214+0000 7f10651d4700 1 -- 172.21.15.167:0/3268466528 <== osd.2 v2:172.21.15.167:6816/30037 18 ==== osd_op_reply(105 myfoo [call,call out=48b,read 0~165 out=165b] v0'0 uv1 ondisk = 0) v8 ==== 233+0+213 (crc 0 0 0) 0x7f105800f7c0 con 0x7f1060010070 2023-10-05T19:28:32.223 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.214+0000 7f106fea0e40 20 rados_obj.operate() r=0 bl.length=165 2023-10-05T19:28:32.223 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.214+0000 7f106fea0e40 10 cache put: name=default.rgw.meta+root+myfoo info.flags=0x11 2023-10-05T19:28:32.223 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.214+0000 7f106fea0e40 10 moving default.rgw.meta+root+myfoo to cache LRU end 2023-10-05T19:28:32.223 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.214+0000 7f106fea0e40 20 get_system_obj_state: rctx=0x7fff0d2f9980 obj=default.rgw.meta:root:.bucket.meta.myfoo:9a9df965-6612-45d4-a45f-6e3c889bacc2.4223.1 state=0x55797af332b0 s->prefetch_data=0 2023-10-05T19:28:32.224 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.214+0000 7f106fea0e40 10 cache get: name=default.rgw.meta+root+.bucket.meta.myfoo:9a9df965-6612-45d4-a45f-6e3c889bacc2.4223.1 : miss 2023-10-05T19:28:32.224 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.214+0000 7f106fea0e40 1 -- 172.21.15.167:0/3268466528 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:106 8.11 8:8bb4ed75:root::.bucket.meta.myfoo%3a9a9df965-6612-45d4-a45f-6e3c889bacc2.4223.1:head [call version.read in=11b,getxattrs,stat] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x55797b1462a0 con 0x55797b0bb5b0 2023-10-05T19:28:32.224 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.214+0000 7f106d9d5700 1 -- 172.21.15.167:0/3268466528 <== osd.3 v2:172.21.15.167:6808/30036 24 ==== osd_op_reply(106 .bucket.meta.myfoo:9a9df965-6612-45d4-a45f-6e3c889bacc2.4223.1 [call out=48b,getxattrs out=218b,stat out=16b] v0'0 uv1 ondisk = 0) v8 ==== 290+0+282 (crc 0 0 0) 0x7f1060025450 con 0x55797b0bb5b0 2023-10-05T19:28:32.224 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.214+0000 7f106fea0e40 10 cache put: name=default.rgw.meta+root+.bucket.meta.myfoo:9a9df965-6612-45d4-a45f-6e3c889bacc2.4223.1 info.flags=0x16 2023-10-05T19:28:32.224 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.214+0000 7f106fea0e40 10 adding default.rgw.meta+root+.bucket.meta.myfoo:9a9df965-6612-45d4-a45f-6e3c889bacc2.4223.1 to cache LRU end 2023-10-05T19:28:32.224 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.214+0000 7f106fea0e40 10 updating xattr: name=ceph.objclass.version bl.length()=42 2023-10-05T19:28:32.224 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.214+0000 7f106fea0e40 10 updating xattr: name=user.rgw.acl bl.length()=123 2023-10-05T19:28:32.224 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.214+0000 7f106fea0e40 20 get_system_obj_state: s->obj_tag was set empty 2023-10-05T19:28:32.224 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.214+0000 7f106fea0e40 20 Read xattr: user.rgw.acl 2023-10-05T19:28:32.224 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.214+0000 7f106fea0e40 20 Read xattr: user.rgw.idtag 2023-10-05T19:28:32.225 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.214+0000 7f106fea0e40 10 cache get: name=default.rgw.meta+root+.bucket.meta.myfoo:9a9df965-6612-45d4-a45f-6e3c889bacc2.4223.1 : type miss (requested=0x13, cached=0x16) 2023-10-05T19:28:32.225 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.214+0000 7f106fea0e40 20 rados->read ofs=0 len=0 2023-10-05T19:28:32.225 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.214+0000 7f106fea0e40 1 -- 172.21.15.167:0/3268466528 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:107 8.11 8:8bb4ed75:root::.bucket.meta.myfoo%3a9a9df965-6612-45d4-a45f-6e3c889bacc2.4223.1:head [call version.check_conds in=122b,call version.read in=11b,read 0~0,getxattrs] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x55797b1496c0 con 0x55797b0bb5b0 2023-10-05T19:28:32.225 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.214+0000 7f106d9d5700 1 -- 172.21.15.167:0/3268466528 <== osd.3 v2:172.21.15.167:6808/30036 25 ==== osd_op_reply(107 .bucket.meta.myfoo:9a9df965-6612-45d4-a45f-6e3c889bacc2.4223.1 [call,call out=48b,read 0~352 out=352b,getxattrs out=218b] v0'0 uv1 ondisk = 0) v8 ==== 332+0+618 (crc 0 0 0) 0x7f1060025450 con 0x55797b0bb5b0 2023-10-05T19:28:32.225 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.214+0000 7f106fea0e40 20 rados_obj.operate() r=0 bl.length=352 2023-10-05T19:28:32.225 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.214+0000 7f106fea0e40 10 cache put: name=default.rgw.meta+root+.bucket.meta.myfoo:9a9df965-6612-45d4-a45f-6e3c889bacc2.4223.1 info.flags=0x13 2023-10-05T19:28:32.225 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.214+0000 7f106fea0e40 10 moving default.rgw.meta+root+.bucket.meta.myfoo:9a9df965-6612-45d4-a45f-6e3c889bacc2.4223.1 to cache LRU end 2023-10-05T19:28:32.225 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.214+0000 7f106fea0e40 10 updating xattr: name=ceph.objclass.version bl.length()=42 2023-10-05T19:28:32.226 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.214+0000 7f106fea0e40 10 updating xattr: name=user.rgw.acl bl.length()=123 2023-10-05T19:28:32.226 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.214+0000 7f106fea0e40 10 chain_cache_entry: cache_locator=default.rgw.meta+root+.bucket.meta.myfoo:9a9df965-6612-45d4-a45f-6e3c889bacc2.4223.1 2023-10-05T19:28:32.226 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.214+0000 7f106fea0e40 20 get_system_obj_state: rctx=0x7fff0d2f9918 obj=default.rgw.meta:users.uid:foo state=0x55797af332b0 s->prefetch_data=0 2023-10-05T19:28:32.226 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.214+0000 7f106fea0e40 10 cache get: name=default.rgw.meta+users.uid+foo : type miss (requested=0x16, cached=0x13) 2023-10-05T19:28:32.226 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.218+0000 7f106fea0e40 1 -- 172.21.15.167:0/3268466528 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:108 8.18 8:1e6ca5ed:users.uid::foo:head [call version.read in=11b,getxattrs,stat] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x55797b149ce0 con 0x55797b0847f0 2023-10-05T19:28:32.227 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.218+0000 7f106e1d6700 1 -- 172.21.15.167:0/3268466528 <== osd.1 v2:172.21.15.167:6800/30035 37 ==== osd_op_reply(108 foo [call out=48b,getxattrs out=75b,stat out=16b] v0'0 uv11 ondisk = 0) v8 ==== 231+0+139 (crc 0 0 0) 0x7f1068045040 con 0x55797b0847f0 2023-10-05T19:28:32.227 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.218+0000 7f106fea0e40 10 cache put: name=default.rgw.meta+users.uid+foo info.flags=0x16 2023-10-05T19:28:32.228 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.218+0000 7f106fea0e40 10 moving default.rgw.meta+users.uid+foo to cache LRU end 2023-10-05T19:28:32.228 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.218+0000 7f106fea0e40 10 updating xattr: name=ceph.objclass.version bl.length()=42 2023-10-05T19:28:32.228 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.218+0000 7f106fea0e40 20 get_system_obj_state: s->obj_tag was set empty 2023-10-05T19:28:32.228 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.218+0000 7f106fea0e40 20 Read xattr: user.rgw.idtag 2023-10-05T19:28:32.228 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.218+0000 7f106fea0e40 10 cache get: name=default.rgw.meta+users.uid+foo : hit (requested=0x13, cached=0x17) 2023-10-05T19:28:32.228 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.218+0000 7f106fea0e40 1 -- 172.21.15.167:0/3268466528 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:109 8.9 8:96d98480:users.uid::foo.buckets:head [call user.remove_bucket in=58b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x55797b14acf0 con 0x7f1060010070 2023-10-05T19:28:32.228 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.222+0000 7f10651d4700 1 -- 172.21.15.167:0/3268466528 <== osd.2 v2:172.21.15.167:6816/30037 19 ==== osd_op_reply(109 foo.buckets [call] v30'17 uv17 ondisk = 0) v8 ==== 155+0+0 (crc 0 0 0) 0x7f105800f7c0 con 0x7f1060010070 2023-10-05T19:28:32.228 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.222+0000 7f106fea0e40 20 get_system_obj_state: rctx=0x7fff0d2f9a58 obj=default.rgw.meta:root:myfoo state=0x55797af332b0 s->prefetch_data=0 2023-10-05T19:28:32.228 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.222+0000 7f106fea0e40 10 cache get: name=default.rgw.meta+root+myfoo : type miss (requested=0x16, cached=0x13) 2023-10-05T19:28:32.228 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.222+0000 7f106fea0e40 1 -- 172.21.15.167:0/3268466528 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:110 8.9 8:91b71841:root::myfoo:head [call version.read in=11b,getxattrs,stat] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x55797b14a1d0 con 0x7f1060010070 2023-10-05T19:28:32.229 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.222+0000 7f10651d4700 1 -- 172.21.15.167:0/3268466528 <== osd.2 v2:172.21.15.167:6816/30037 20 ==== osd_op_reply(110 myfoo [call out=48b,getxattrs out=75b,stat out=16b] v0'0 uv1 ondisk = 0) v8 ==== 233+0+139 (crc 0 0 0) 0x7f1058057040 con 0x7f1060010070 2023-10-05T19:28:32.229 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.222+0000 7f106fea0e40 10 cache put: name=default.rgw.meta+root+myfoo info.flags=0x16 2023-10-05T19:28:32.229 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.222+0000 7f106fea0e40 10 moving default.rgw.meta+root+myfoo to cache LRU end 2023-10-05T19:28:32.229 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.222+0000 7f106fea0e40 10 updating xattr: name=ceph.objclass.version bl.length()=42 2023-10-05T19:28:32.229 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.222+0000 7f106fea0e40 20 get_system_obj_state: s->obj_tag was set empty 2023-10-05T19:28:32.229 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.222+0000 7f106fea0e40 20 Read xattr: user.rgw.idtag 2023-10-05T19:28:32.229 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.222+0000 7f106fea0e40 10 cache get: name=default.rgw.meta+root+myfoo : hit (requested=0x13, cached=0x17) 2023-10-05T19:28:32.229 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.222+0000 7f106fea0e40 1 -- 172.21.15.167:0/3268466528 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:111 8.9 8:91b71841:root::myfoo:head [delete,create,call version.check_conds in=122b,call version.set in=58b,writefull 0~165 in=165b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x55797b14e9f0 con 0x7f1060010070 2023-10-05T19:28:32.230 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.226+0000 7f10651d4700 1 -- 172.21.15.167:0/3268466528 <== osd.2 v2:172.21.15.167:6816/30037 21 ==== osd_op_reply(111 myfoo [delete,create,call,call,writefull 0~165] v30'18 uv18 ondisk = 0) v8 ==== 317+0+0 (crc 0 0 0) 0x7f1058057040 con 0x7f1060010070 2023-10-05T19:28:32.230 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.226+0000 7f106fea0e40 10 cache put: name=default.rgw.meta+root+myfoo info.flags=0x17 2023-10-05T19:28:32.231 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.226+0000 7f106fea0e40 10 moving default.rgw.meta+root+myfoo to cache LRU end 2023-10-05T19:28:32.231 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.226+0000 7f106fea0e40 10 distributing notification oid=default.rgw.control:notify.6 cni=[op: 0, obj: default.rgw.meta:root:myfoo, ofs0, ns] 2023-10-05T19:28:32.231 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.226+0000 7f106fea0e40 1 -- 172.21.15.167:0/3268466528 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:112 7.14 7:2b04a3e9:::notify.6:head [notify cookie 93980244370688 in=350b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x55797b14d640 con 0x7f105800ea30 2023-10-05T19:28:32.232 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.226+0000 7f106e1d6700 1 -- 172.21.15.167:0/3268466528 <== osd.0 v2:172.21.15.118:6800/27690 29 ==== watch-notify(notify (1) cookie 93980243808032 notify 128849018882 ret 0) v3 ==== 380+0+0 (crc 0 0 0) 0x7f1068031040 con 0x7f105800ea30 2023-10-05T19:28:32.232 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.226+0000 7f1067fff700 10 rgw watcher librados: RGWWatcher::handle_notify() notify_id 128849018882 cookie 93980243808032 notifier 4437 bl.length()=338 2023-10-05T19:28:32.232 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.226+0000 7f106e1d6700 1 -- 172.21.15.167:0/3268466528 <== osd.0 v2:172.21.15.118:6800/27690 30 ==== osd_op_reply(112 notify.6 [notify cookie 93980244370688 out=8b] v0'0 uv105 ondisk = 0) v8 ==== 152+0+8 (crc 0 0 0) 0x7f1068045040 con 0x7f105800ea30 2023-10-05T19:28:32.232 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.226+0000 7f1067fff700 10 rgw watcher librados: cache put: name=default.rgw.meta+root+myfoo info.flags=0x17 2023-10-05T19:28:32.232 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.226+0000 7f1067fff700 10 rgw watcher librados: moving default.rgw.meta+root+myfoo to cache LRU end 2023-10-05T19:28:32.232 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.226+0000 7f1067fff700 1 -- 172.21.15.167:0/3268466528 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:113 7.14 7:2b04a3e9:::notify.6:head [notify-ack in=20b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x7f1054003d90 con 0x7f105800ea30 2023-10-05T19:28:32.233 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.226+0000 7f106e1d6700 1 -- 172.21.15.167:0/3268466528 <== osd.0 v2:172.21.15.118:6800/27690 31 ==== watch-notify(notify_complete (2) cookie 93980244370688 notify 128849018882 ret 0) v3 ==== 42+0+48 (crc 0 0 0) 0x7f1068002590 con 0x7f105800ea30 2023-10-05T19:28:32.233 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.226+0000 7f106e1d6700 1 -- 172.21.15.167:0/3268466528 <== osd.0 v2:172.21.15.118:6800/27690 32 ==== osd_op_reply(113 notify.6 [notify-ack] v0'0 uv105 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f1068045040 con 0x7f105800ea30 2023-10-05T19:28:32.233 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.226+0000 7f106fea0e40 1 -- 172.21.15.167:0/3268466528 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:114 7.1f 7:f95f44c2:::notify.0:head [watch unwatch cookie 93980243789328] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x55797b09b9d0 con 0x7f1060010070 2023-10-05T19:28:32.237 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.230+0000 7f10651d4700 1 -- 172.21.15.167:0/3268466528 <== osd.2 v2:172.21.15.167:6816/30037 22 ==== osd_op_reply(114 notify.0 [watch unwatch cookie 93980243789328] v30'107 uv105 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f1058057040 con 0x7f1060010070 2023-10-05T19:28:32.237 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.230+0000 7f106fea0e40 20 remove_watcher() i=0 2023-10-05T19:28:32.237 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.230+0000 7f106fea0e40 2 removed watcher, disabling cache 2023-10-05T19:28:32.237 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.230+0000 7f106fea0e40 1 -- 172.21.15.167:0/3268466528 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:115 7.0 7:05bf5b68:::notify.1:head [watch unwatch cookie 93980243792928] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x55797b09b9d0 con 0x55797b0847f0 2023-10-05T19:28:32.238 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.230+0000 7f106e1d6700 1 -- 172.21.15.167:0/3268466528 <== osd.1 v2:172.21.15.167:6800/30035 38 ==== osd_op_reply(115 notify.1 [watch unwatch cookie 93980243792928] v30'107 uv105 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f1068045040 con 0x55797b0847f0 2023-10-05T19:28:32.238 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.230+0000 7f106fea0e40 20 remove_watcher() i=1 2023-10-05T19:28:32.238 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.230+0000 7f106fea0e40 1 -- 172.21.15.167:0/3268466528 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:116 7.15 7:a93a5511:::notify.2:head [watch unwatch cookie 93980243794032] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x55797b09b9d0 con 0x55797b0bb5b0 2023-10-05T19:28:32.240 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.234+0000 7f106d9d5700 1 -- 172.21.15.167:0/3268466528 <== osd.3 v2:172.21.15.167:6808/30036 26 ==== osd_op_reply(116 notify.2 [watch unwatch cookie 93980243794032] v30'107 uv105 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f1060025450 con 0x55797b0bb5b0 2023-10-05T19:28:32.240 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.234+0000 7f106fea0e40 20 remove_watcher() i=2 2023-10-05T19:28:32.240 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.234+0000 7f106fea0e40 1 -- 172.21.15.167:0/3268466528 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:117 7.e 7:7759931f:::notify.3:head [watch unwatch cookie 93980243802432] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x55797b09b9d0 con 0x7f1060010070 2023-10-05T19:28:32.243 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.238+0000 7f10651d4700 1 -- 172.21.15.167:0/3268466528 <== osd.2 v2:172.21.15.167:6816/30037 23 ==== osd_op_reply(117 notify.3 [watch unwatch cookie 93980243802432] v30'107 uv105 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f1058057040 con 0x7f1060010070 2023-10-05T19:28:32.243 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.238+0000 7f106fea0e40 20 remove_watcher() i=3 2023-10-05T19:28:32.243 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.238+0000 7f106fea0e40 1 -- 172.21.15.167:0/3268466528 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:118 7.d 7:b4812045:::notify.4:head [watch unwatch cookie 93980243803536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x55797b09b9d0 con 0x55797b0847f0 2023-10-05T19:28:32.246 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.238+0000 7f106e1d6700 1 -- 172.21.15.167:0/3268466528 <== osd.1 v2:172.21.15.167:6800/30035 39 ==== osd_op_reply(118 notify.4 [watch unwatch cookie 93980243803536] v30'107 uv105 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f106805e040 con 0x55797b0847f0 2023-10-05T19:28:32.246 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.238+0000 7f106fea0e40 20 remove_watcher() i=4 2023-10-05T19:28:32.246 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.238+0000 7f106fea0e40 1 -- 172.21.15.167:0/3268466528 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:119 7.3 7:c609908c:::notify.5:head [watch unwatch cookie 93980243798960] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x55797b09b9d0 con 0x7f105800ea30 2023-10-05T19:28:32.249 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.242+0000 7f106e1d6700 1 -- 172.21.15.167:0/3268466528 <== osd.0 v2:172.21.15.118:6800/27690 33 ==== osd_op_reply(119 notify.5 [watch unwatch cookie 93980243798960] v30'107 uv105 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f106805e040 con 0x7f105800ea30 2023-10-05T19:28:32.249 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.242+0000 7f106fea0e40 20 remove_watcher() i=5 2023-10-05T19:28:32.249 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.242+0000 7f106fea0e40 1 -- 172.21.15.167:0/3268466528 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:120 7.14 7:2b04a3e9:::notify.6:head [watch unwatch cookie 93980243808032] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x55797b09b9d0 con 0x7f105800ea30 2023-10-05T19:28:32.251 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.246+0000 7f106e1d6700 1 -- 172.21.15.167:0/3268466528 <== osd.0 v2:172.21.15.118:6800/27690 34 ==== osd_op_reply(120 notify.6 [watch unwatch cookie 93980243808032] v30'107 uv105 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f106805e040 con 0x7f105800ea30 2023-10-05T19:28:32.252 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.246+0000 7f106fea0e40 20 remove_watcher() i=6 2023-10-05T19:28:32.252 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.246+0000 7f106fea0e40 1 -- 172.21.15.167:0/3268466528 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:121 7.9 7:93e5b521:::notify.7:head [watch unwatch cookie 93980243812912] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x55797b09b9d0 con 0x7f105800ea30 2023-10-05T19:28:32.254 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.250+0000 7f106e1d6700 1 -- 172.21.15.167:0/3268466528 <== osd.0 v2:172.21.15.118:6800/27690 35 ==== osd_op_reply(121 notify.7 [watch unwatch cookie 93980243812912] v30'107 uv105 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f106805e040 con 0x7f105800ea30 2023-10-05T19:28:32.254 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.250+0000 7f106fea0e40 20 remove_watcher() i=7 2023-10-05T19:28:32.256 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.250+0000 7f103f7de700 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2023-10-05T19:28:32.256 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.250+0000 7f106fea0e40 1 -- 172.21.15.167:0/3268466528 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:122 6.4 6:22d26bf9:::data_loggenerations_metadata:head [watch unwatch cookie 93980243814016] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x55797b09b9d0 con 0x55797b0847f0 2023-10-05T19:28:32.260 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.254+0000 7f106e1d6700 1 -- 172.21.15.167:0/3268466528 <== osd.1 v2:172.21.15.167:6800/30035 40 ==== osd_op_reply(122 data_loggenerations_metadata [watch unwatch cookie 93980243814016] v30'106 uv1 ondisk = 0) v8 ==== 172+0+0 (crc 0 0 0) 0x7f106805e040 con 0x55797b0847f0 2023-10-05T19:28:32.260 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.254+0000 7f106fea0e40 1 -- 172.21.15.167:0/3268466528 >> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] conn(0x7f105800ea30 msgr2=0x7f105800d310 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-05T19:28:32.261 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.254+0000 7f106fea0e40 1 --2- 172.21.15.167:0/3268466528 >> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] conn(0x7f105800ea30 0x7f105800d310 crc :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-05T19:28:32.261 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.254+0000 7f106fea0e40 1 -- 172.21.15.167:0/3268466528 >> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] conn(0x55797b0847f0 msgr2=0x55797b0b7810 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-05T19:28:32.261 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.254+0000 7f106fea0e40 1 --2- 172.21.15.167:0/3268466528 >> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] conn(0x55797b0847f0 0x55797b0b7810 crc :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-05T19:28:32.261 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.254+0000 7f106fea0e40 1 -- 172.21.15.167:0/3268466528 >> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] conn(0x7f1060010070 msgr2=0x7f10600187e0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-05T19:28:32.261 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.254+0000 7f106fea0e40 1 --2- 172.21.15.167:0/3268466528 >> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] conn(0x7f1060010070 0x7f10600187e0 crc :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-05T19:28:32.261 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.254+0000 7f106fea0e40 1 -- 172.21.15.167:0/3268466528 >> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] conn(0x55797b0bb5b0 msgr2=0x55797b0bd990 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-05T19:28:32.262 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.254+0000 7f106fea0e40 1 --2- 172.21.15.167:0/3268466528 >> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] conn(0x55797b0bb5b0 0x55797b0bd990 crc :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-05T19:28:32.262 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.254+0000 7f106fea0e40 1 -- 172.21.15.167:0/3268466528 >> [v2:172.21.15.167:6824/29889,v1:172.21.15.167:6825/29889] conn(0x7f1050030c50 msgr2=0x7f1050033100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-05T19:28:32.262 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.254+0000 7f106fea0e40 1 --2- 172.21.15.167:0/3268466528 >> [v2:172.21.15.167:6824/29889,v1:172.21.15.167:6825/29889] conn(0x7f1050030c50 0x7f1050033100 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7f1058001eb0 tx=0x7f1058001580 comp rx=0 tx=0).stop 2023-10-05T19:28:32.262 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.254+0000 7f106fea0e40 1 -- 172.21.15.167:0/3268466528 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55797b084420 msgr2=0x55797b0abd50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-05T19:28:32.262 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.254+0000 7f106fea0e40 1 --2- 172.21.15.167:0/3268466528 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55797b084420 0x55797b0abd50 secure :-1 s=READY pgs=347 cs=0 l=1 rev1=1 crypto rx=0x7f106000a5c0 tx=0x7f106001d000 comp rx=0 tx=0).stop 2023-10-05T19:28:32.262 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.254+0000 7f106e1d6700 1 -- 172.21.15.167:0/3268466528 reap_dead start 2023-10-05T19:28:32.262 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.254+0000 7f106fea0e40 1 -- 172.21.15.167:0/3268466528 shutdown_connections 2023-10-05T19:28:32.262 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.254+0000 7f106fea0e40 1 --2- 172.21.15.167:0/3268466528 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55797b084420 0x55797b0abd50 unknown :-1 s=CLOSED pgs=347 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-05T19:28:32.263 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.254+0000 7f106fea0e40 1 -- 172.21.15.167:0/3268466528 >> 172.21.15.167:0/3268466528 conn(0x55797b07e730 msgr2=0x55797b09f730 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-05T19:28:32.263 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.254+0000 7f106fea0e40 1 -- 172.21.15.167:0/3268466528 shutdown_connections 2023-10-05T19:28:32.263 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.254+0000 7f106fea0e40 1 -- 172.21.15.167:0/3268466528 wait complete. 2023-10-05T19:28:32.264 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.254+0000 7f106fea0e40 1 -- 172.21.15.167:0/4045791114 >> [v2:172.21.15.167:6824/29889,v1:172.21.15.167:6825/29889] conn(0x7f0fb4030c50 msgr2=0x7f0fb4033100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-05T19:28:32.264 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.254+0000 7f106fea0e40 1 --2- 172.21.15.167:0/4045791114 >> [v2:172.21.15.167:6824/29889,v1:172.21.15.167:6825/29889] conn(0x7f0fb4030c50 0x7f0fb4033100 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x55797b0c5b50 tx=0x7f1068000f90 comp rx=0 tx=0).stop 2023-10-05T19:28:32.264 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.254+0000 7f106fea0e40 1 -- 172.21.15.167:0/4045791114 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55797b0d4000 msgr2=0x55797b0d9ba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-05T19:28:32.264 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.254+0000 7f106fea0e40 1 --2- 172.21.15.167:0/4045791114 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55797b0d4000 0x55797b0d9ba0 secure :-1 s=READY pgs=349 cs=0 l=1 rev1=1 crypto rx=0x7f105800f610 tx=0x7f1058031000 comp rx=0 tx=0).stop 2023-10-05T19:28:32.264 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.254+0000 7f106fea0e40 1 -- 172.21.15.167:0/4045791114 shutdown_connections 2023-10-05T19:28:32.264 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.254+0000 7f106fea0e40 1 --2- 172.21.15.167:0/4045791114 >> [v2:172.21.15.167:6824/29889,v1:172.21.15.167:6825/29889] conn(0x7f0fb4030c50 0x7f0fb4033100 unknown :-1 s=CLOSED pgs=123 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-05T19:28:32.264 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.254+0000 7f106fea0e40 1 --2- 172.21.15.167:0/4045791114 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55797b0d4000 0x55797b0d9ba0 unknown :-1 s=CLOSED pgs=349 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-05T19:28:32.264 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.254+0000 7f106fea0e40 1 -- 172.21.15.167:0/4045791114 >> 172.21.15.167:0/4045791114 conn(0x55797b0cf3c0 msgr2=0x55797b0cd460 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-05T19:28:32.264 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.254+0000 7f106fea0e40 1 -- 172.21.15.167:0/4045791114 shutdown_connections 2023-10-05T19:28:32.264 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.254+0000 7f106fea0e40 1 -- 172.21.15.167:0/4045791114 wait complete. 2023-10-05T19:28:32.267 INFO:tasks.util.rgw:rgwadmin: client.0 : ['user', 'create', '--uid', 'fud', '--display-name', 'Fud', '--access-key', 'p5YnriCv1nAtykxBrupQ', '--secret', 'Q8Tk6Q/27hfbFSYdSkPtUqhqx1GgzvpXa4WARozh', '--max-buckets', '1'] 2023-10-05T19:28:32.267 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'ceph', 'user', 'create', '--uid', 'fud', '--display-name', 'Fud', '--access-key', 'p5YnriCv1nAtykxBrupQ', '--secret', 'Q8Tk6Q/27hfbFSYdSkPtUqhqx1GgzvpXa4WARozh', '--max-buckets', '1'] 2023-10-05T19:28:32.267 DEBUG:teuthology.orchestra.run.smithi167:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster ceph user create --uid fud --display-name Fud --access-key p5YnriCv1nAtykxBrupQ --secret Q8Tk6Q/27hfbFSYdSkPtUqhqx1GgzvpXa4WARozh --max-buckets 1 2023-10-05T19:28:32.358 INFO:teuthology.orchestra.run.smithi167.stderr:ignoring --setuser ceph since I am not root 2023-10-05T19:28:32.358 INFO:teuthology.orchestra.run.smithi167.stderr:ignoring --setgroup ceph since I am not root 2023-10-05T19:28:32.359 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.354+0000 7fd937d99e40 1 Processor -- start 2023-10-05T19:28:32.359 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.354+0000 7fd937d99e40 1 -- start start 2023-10-05T19:28:32.359 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.354+0000 7fd937d99e40 1 --2- >> v2:172.21.15.118:3300/0 conn(0x555cd0400d60 0x555cd0401130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-05T19:28:32.359 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.354+0000 7fd937d99e40 1 -- --> v1:172.21.15.118:6789/0 -- auth(proto 0 26 bytes epoch 0) v1 -- 0x555cd034ebd0 con 0x555cd0401670 2023-10-05T19:28:32.359 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.354+0000 7fd937d99e40 1 -- --> v2:172.21.15.118:3300/0 -- mon_getmap magic: 0 v1 -- 0x555cd030bee0 con 0x555cd0400d60 2023-10-05T19:28:32.359 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.354+0000 7fd9358ce700 1 --2- >> v2:172.21.15.118:3300/0 conn(0x555cd0400d60 0x555cd0401130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-05T19:28:32.360 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.354+0000 7fd9350cd700 1 --1- >> v1:172.21.15.118:6789/0 conn(0x555cd0401670 0x555cd0402110 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.118:6789/0 says I am v1:172.21.15.167:56608/0 (socket says 172.21.15.167:56608) 2023-10-05T19:28:32.360 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.354+0000 7fd9350cd700 1 -- 172.21.15.167:0/734479950 learned_addr learned my addr 172.21.15.167:0/734479950 (peer_addr_for_me v1:172.21.15.167:0/0) 2023-10-05T19:28:32.360 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.354+0000 7fd9358ce700 1 --2- >> v2:172.21.15.118:3300/0 conn(0x555cd0400d60 0x555cd0401130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.118:3300/0 says I am v2:172.21.15.167:58382/0 (socket says 172.21.15.167:58382) 2023-10-05T19:28:32.360 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.354+0000 7fd9358ce700 1 -- 172.21.15.167:0/734479950 >> v1:172.21.15.118:6789/0 conn(0x555cd0401670 legacy=0x555cd0402110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-05T19:28:32.360 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.354+0000 7fd9358ce700 1 -- 172.21.15.167:0/734479950 --> v2:172.21.15.118:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x555cd0402820 con 0x555cd0400d60 2023-10-05T19:28:32.361 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.354+0000 7fd9358ce700 1 --2- 172.21.15.167:0/734479950 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x555cd0400d60 0x555cd0401130 secure :-1 s=READY pgs=351 cs=0 l=1 rev1=1 crypto rx=0x7fd92000a5d0 tx=0x7fd920005a50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8b8fc8d46b5725a0 server_cookie=0 in_seq=0 out_seq=0 2023-10-05T19:28:32.361 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.354+0000 7fd9348cc700 1 -- 172.21.15.167:0/734479950 <== mon.0 v2:172.21.15.118:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fd9200086a0 con 0x555cd0400d60 2023-10-05T19:28:32.362 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.354+0000 7fd9348cc700 1 -- 172.21.15.167:0/734479950 <== mon.0 v2:172.21.15.118:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd920008800 con 0x555cd0400d60 2023-10-05T19:28:32.362 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.354+0000 7fd9348cc700 1 -- 172.21.15.167:0/734479950 <== mon.0 v2:172.21.15.118:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fd920008ad0 con 0x555cd0400d60 2023-10-05T19:28:32.362 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.354+0000 7fd937d99e40 1 -- 172.21.15.167:0/734479950 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x555cd0400d60 msgr2=0x555cd0401130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-05T19:28:32.362 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.354+0000 7fd937d99e40 1 --2- 172.21.15.167:0/734479950 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x555cd0400d60 0x555cd0401130 secure :-1 s=READY pgs=351 cs=0 l=1 rev1=1 crypto rx=0x7fd92000a5d0 tx=0x7fd920005a50 comp rx=0 tx=0).stop 2023-10-05T19:28:32.362 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.354+0000 7fd937d99e40 1 -- 172.21.15.167:0/734479950 shutdown_connections 2023-10-05T19:28:32.362 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.354+0000 7fd937d99e40 1 --2- 172.21.15.167:0/734479950 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x555cd0400d60 0x555cd0401130 unknown :-1 s=CLOSED pgs=351 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-05T19:28:32.362 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.354+0000 7fd937d99e40 1 -- 172.21.15.167:0/734479950 >> 172.21.15.167:0/734479950 conn(0x555cd03fc780 msgr2=0x555cd03feb90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-05T19:28:32.362 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.354+0000 7fd937d99e40 1 -- 172.21.15.167:0/734479950 shutdown_connections 2023-10-05T19:28:32.362 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.354+0000 7fd937d99e40 1 -- 172.21.15.167:0/734479950 wait complete. 2023-10-05T19:28:32.363 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.358+0000 7fd937d99e40 1 Processor -- start 2023-10-05T19:28:32.363 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.358+0000 7fd937d99e40 1 -- start start 2023-10-05T19:28:32.363 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.358+0000 7fd937d99e40 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x555cd0400d60 0x555cd04210c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-05T19:28:32.363 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.358+0000 7fd937d99e40 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x555cd02cfe30 con 0x555cd0400d60 2023-10-05T19:28:32.364 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.358+0000 7fd9358ce700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x555cd0400d60 0x555cd04210c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-05T19:28:32.364 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.358+0000 7fd9358ce700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x555cd0400d60 0x555cd04210c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.118:3300/0 says I am v2:172.21.15.167:58396/0 (socket says 172.21.15.167:58396) 2023-10-05T19:28:32.364 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.358+0000 7fd9358ce700 1 -- 172.21.15.167:0/2927795329 learned_addr learned my addr 172.21.15.167:0/2927795329 (peer_addr_for_me v2:172.21.15.167:0/0) 2023-10-05T19:28:32.364 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.358+0000 7fd9358ce700 1 -- 172.21.15.167:0/2927795329 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd920005cf0 con 0x555cd0400d60 2023-10-05T19:28:32.365 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.358+0000 7fd9358ce700 1 --2- 172.21.15.167:0/2927795329 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x555cd0400d60 0x555cd04210c0 secure :-1 s=READY pgs=352 cs=0 l=1 rev1=1 crypto rx=0x7fd920000c00 tx=0x7fd920015000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2cfbc5f657150e32 server_cookie=0 in_seq=0 out_seq=0 2023-10-05T19:28:32.365 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.358+0000 7fd92f7fe700 1 -- 172.21.15.167:0/2927795329 <== mon.0 v2:172.21.15.118:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fd920008970 con 0x555cd0400d60 2023-10-05T19:28:32.365 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.358+0000 7fd92f7fe700 1 -- 172.21.15.167:0/2927795329 <== mon.0 v2:172.21.15.118:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd920008af0 con 0x555cd0400d60 2023-10-05T19:28:32.365 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.358+0000 7fd92f7fe700 1 -- 172.21.15.167:0/2927795329 <== mon.0 v2:172.21.15.118:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fd920008de0 con 0x555cd0400d60 2023-10-05T19:28:32.366 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.358+0000 7fd937d99e40 1 -- 172.21.15.167:0/2927795329 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x555cd0400d60 msgr2=0x555cd04210c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-05T19:28:32.366 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.358+0000 7fd937d99e40 1 --2- 172.21.15.167:0/2927795329 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x555cd0400d60 0x555cd04210c0 secure :-1 s=READY pgs=352 cs=0 l=1 rev1=1 crypto rx=0x7fd920000c00 tx=0x7fd920015000 comp rx=0 tx=0).stop 2023-10-05T19:28:32.366 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.358+0000 7fd937d99e40 1 -- 172.21.15.167:0/2927795329 shutdown_connections 2023-10-05T19:28:32.366 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.362+0000 7fd937d99e40 1 --2- 172.21.15.167:0/2927795329 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x555cd0400d60 0x555cd04210c0 unknown :-1 s=CLOSED pgs=352 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-05T19:28:32.366 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.362+0000 7fd937d99e40 1 -- 172.21.15.167:0/2927795329 >> 172.21.15.167:0/2927795329 conn(0x555cd03fc780 msgr2=0x555cd03755b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-05T19:28:32.366 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.362+0000 7fd937d99e40 1 -- 172.21.15.167:0/2927795329 shutdown_connections 2023-10-05T19:28:32.366 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.362+0000 7fd937d99e40 1 -- 172.21.15.167:0/2927795329 wait complete. 2023-10-05T19:28:32.367 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.362+0000 7fd937d99e40 1 Processor -- start 2023-10-05T19:28:32.367 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.362+0000 7fd937d99e40 1 -- start start 2023-10-05T19:28:32.368 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.362+0000 7fd937d99e40 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x555cd0400d60 0x555cd0429e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-05T19:28:32.368 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.362+0000 7fd937d99e40 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x555cd02a2ed0 con 0x555cd0400d60 2023-10-05T19:28:32.368 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.362+0000 7fd9358ce700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x555cd0400d60 0x555cd0429e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-05T19:28:32.368 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.362+0000 7fd9358ce700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x555cd0400d60 0x555cd0429e30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.118:3300/0 says I am v2:172.21.15.167:58410/0 (socket says 172.21.15.167:58410) 2023-10-05T19:28:32.368 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.362+0000 7fd9358ce700 1 -- 172.21.15.167:0/3309876569 learned_addr learned my addr 172.21.15.167:0/3309876569 (peer_addr_for_me v2:172.21.15.167:0/0) 2023-10-05T19:28:32.368 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.362+0000 7fd9358ce700 1 -- 172.21.15.167:0/3309876569 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x555cd0402820 con 0x555cd0400d60 2023-10-05T19:28:32.369 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.362+0000 7fd9358ce700 1 --2- 172.21.15.167:0/3309876569 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x555cd0400d60 0x555cd0429e30 secure :-1 s=READY pgs=353 cs=0 l=1 rev1=1 crypto rx=0x7fd92001e040 tx=0x7fd920003050 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-05T19:28:32.369 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.362+0000 7fd92dffb700 1 -- 172.21.15.167:0/3309876569 <== mon.0 v2:172.21.15.118:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fd920012d20 con 0x555cd0400d60 2023-10-05T19:28:32.369 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.362+0000 7fd937d99e40 1 -- 172.21.15.167:0/3309876569 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x555cd042a370 con 0x555cd0400d60 2023-10-05T19:28:32.369 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.362+0000 7fd92dffb700 1 -- 172.21.15.167:0/3309876569 <== mon.0 v2:172.21.15.118:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd920012e80 con 0x555cd0400d60 2023-10-05T19:28:32.369 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.362+0000 7fd92dffb700 1 -- 172.21.15.167:0/3309876569 <== mon.0 v2:172.21.15.118:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fd920030550 con 0x555cd0400d60 2023-10-05T19:28:32.370 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.362+0000 7fd937d99e40 1 -- 172.21.15.167:0/3309876569 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x555cd042a810 con 0x555cd0400d60 2023-10-05T19:28:32.370 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.366+0000 7fd937d99e40 20 rados->read ofs=0 len=0 2023-10-05T19:28:32.371 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.366+0000 7fd92dffb700 1 -- 172.21.15.167:0/3309876569 <== mon.0 v2:172.21.15.118:3300/0 4 ==== mgrmap(e 5) v1 ==== 40329+0+0 (secure 0 0 0) 0x7fd920019070 con 0x555cd0400d60 2023-10-05T19:28:32.371 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.366+0000 7fd92dffb700 1 --2- 172.21.15.167:0/3309876569 >> [v2:172.21.15.167:6824/29889,v1:172.21.15.167:6825/29889] conn(0x7fd918030c50 0x7fd918033100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-05T19:28:32.371 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.366+0000 7fd92dffb700 1 -- 172.21.15.167:0/3309876569 <== mon.0 v2:172.21.15.118:3300/0 5 ==== osd_map(30..30 src has 1..30) v4 ==== 6057+0+0 (secure 0 0 0) 0x7fd920046670 con 0x555cd0400d60 2023-10-05T19:28:32.371 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.366+0000 7fd9350cd700 1 --2- 172.21.15.167:0/3309876569 >> [v2:172.21.15.167:6824/29889,v1:172.21.15.167:6825/29889] conn(0x7fd918030c50 0x7fd918033100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-05T19:28:32.372 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.366+0000 7fd937d99e40 1 --2- 172.21.15.167:0/3309876569 >> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] conn(0x555cd0401670 0x555cd0435950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-05T19:28:32.372 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.366+0000 7fd937d99e40 1 -- 172.21.15.167:0/3309876569 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:1 5.12 5:49953fa1:::default.realm:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x555cd0435fc0 con 0x555cd0401670 2023-10-05T19:28:32.372 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.366+0000 7fd9360cf700 1 --2- 172.21.15.167:0/3309876569 >> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] conn(0x555cd0401670 0x555cd0435950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-05T19:28:32.372 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.366+0000 7fd9350cd700 1 --2- 172.21.15.167:0/3309876569 >> [v2:172.21.15.167:6824/29889,v1:172.21.15.167:6825/29889] conn(0x7fd918030c50 0x7fd918033100 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7fd9300044f0 tx=0x7fd9300067c0 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-05T19:28:32.373 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.366+0000 7fd9360cf700 1 --2- 172.21.15.167:0/3309876569 >> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] conn(0x555cd0401670 0x555cd0435950 crc :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-05T19:28:32.373 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.366+0000 7fd9360cf700 1 -- 172.21.15.167:0/3309876569 <== osd.1 v2:172.21.15.167:6800/30035 1 ==== osd_op_reply(1 default.realm [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 ==== 157+0+0 (crc 0 0 0) 0x7fd92800a500 con 0x555cd0401670 2023-10-05T19:28:32.373 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.366+0000 7fd937d99e40 20 rados_obj.operate() r=-2 bl.length=0 2023-10-05T19:28:32.374 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.366+0000 7fd937d99e40 20 realm 2023-10-05T19:28:32.374 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.366+0000 7fd937d99e40 20 rados->read ofs=0 len=0 2023-10-05T19:28:32.374 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.366+0000 7fd937d99e40 1 -- 172.21.15.167:0/3309876569 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:2 5.12 5:49953fa1:::default.realm:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x555cd0432fb0 con 0x555cd0401670 2023-10-05T19:28:32.375 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.370+0000 7fd9360cf700 1 -- 172.21.15.167:0/3309876569 <== osd.1 v2:172.21.15.167:6800/30035 2 ==== osd_op_reply(2 default.realm [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 ==== 157+0+0 (crc 0 0 0) 0x7fd92800a500 con 0x555cd0401670 2023-10-05T19:28:32.375 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.370+0000 7fd937d99e40 20 rados_obj.operate() r=-2 bl.length=0 2023-10-05T19:28:32.375 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.370+0000 7fd937d99e40 4 RGWPeriod::init failed to init realm id : (2) No such file or directory 2023-10-05T19:28:32.375 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.370+0000 7fd937d99e40 20 rados->read ofs=0 len=0 2023-10-05T19:28:32.375 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.370+0000 7fd937d99e40 1 -- 172.21.15.167:0/3309876569 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:3 5.12 5:49953fa1:::default.realm:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x555cd0436aa0 con 0x555cd0401670 2023-10-05T19:28:32.375 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.370+0000 7fd9360cf700 1 -- 172.21.15.167:0/3309876569 <== osd.1 v2:172.21.15.167:6800/30035 3 ==== osd_op_reply(3 default.realm [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 ==== 157+0+0 (crc 0 0 0) 0x7fd92800a500 con 0x555cd0401670 2023-10-05T19:28:32.375 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.370+0000 7fd937d99e40 20 rados_obj.operate() r=-2 bl.length=0 2023-10-05T19:28:32.375 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.370+0000 7fd937d99e40 20 rados->read ofs=0 len=0 2023-10-05T19:28:32.376 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.370+0000 7fd937d99e40 1 -- 172.21.15.167:0/3309876569 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:4 5.f 5:f43b6ece:::zone_names.default:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x555cd04366b0 con 0x555cd0401670 2023-10-05T19:28:32.377 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.370+0000 7fd9360cf700 1 -- 172.21.15.167:0/3309876569 <== osd.1 v2:172.21.15.167:6800/30035 4 ==== osd_op_reply(4 zone_names.default [read 0~46 out=46b] v0'0 uv1 ondisk = 0) v8 ==== 162+0+46 (crc 0 0 0) 0x7fd92800a500 con 0x555cd0401670 2023-10-05T19:28:32.377 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.370+0000 7fd937d99e40 20 rados_obj.operate() r=0 bl.length=46 2023-10-05T19:28:32.377 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.370+0000 7fd937d99e40 20 rados->read ofs=0 len=0 2023-10-05T19:28:32.377 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.370+0000 7fd937d99e40 1 -- 172.21.15.167:0/3309876569 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:5 5.1d 5:b8374eb4:::zone_info.9a9df965-6612-45d4-a45f-6e3c889bacc2:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x555cd0437930 con 0x555cd0401670 2023-10-05T19:28:32.377 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.370+0000 7fd9360cf700 1 -- 172.21.15.167:0/3309876569 <== osd.1 v2:172.21.15.167:6800/30035 5 ==== osd_op_reply(5 zone_info.9a9df965-6612-45d4-a45f-6e3c889bacc2 [read 0~874 out=874b] v0'0 uv2 ondisk = 0) v8 ==== 190+0+874 (crc 0 0 0) 0x7fd92800a500 con 0x555cd0401670 2023-10-05T19:28:32.377 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.370+0000 7fd937d99e40 20 rados_obj.operate() r=0 bl.length=874 2023-10-05T19:28:32.377 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.370+0000 7fd937d99e40 20 searching for the correct realm 2023-10-05T19:28:32.377 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.370+0000 7fd937d99e40 1 --2- 172.21.15.167:0/3309876569 >> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] conn(0x555cd04396f0 0x555cd043bad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-05T19:28:32.378 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.370+0000 7fd937d99e40 1 -- 172.21.15.167:0/3309876569 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:6 5.0 5:00000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x555cd043c010 con 0x555cd04396f0 2023-10-05T19:28:32.378 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.370+0000 7fd9358ce700 1 --2- 172.21.15.167:0/3309876569 >> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] conn(0x555cd04396f0 0x555cd043bad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-05T19:28:32.378 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.370+0000 7fd9358ce700 1 --2- 172.21.15.167:0/3309876569 >> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] conn(0x555cd04396f0 0x555cd043bad0 crc :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-05T19:28:32.379 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.374+0000 7fd9358ce700 1 -- 172.21.15.167:0/3309876569 <== osd.3 v2:172.21.15.167:6808/30036 1 ==== osd_op_reply(6 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fd92002d070 con 0x555cd04396f0 2023-10-05T19:28:32.379 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.374+0000 7fd9358ce700 1 --2- 172.21.15.167:0/3309876569 >> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] conn(0x7fd920007030 0x7fd92002aa00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-05T19:28:32.379 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.374+0000 7fd9358ce700 1 -- 172.21.15.167:0/3309876569 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:7 5.10 5:08000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x555cd043c010 con 0x7fd920007030 2023-10-05T19:28:32.379 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.374+0000 7fd9350cd700 1 --2- 172.21.15.167:0/3309876569 >> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] conn(0x7fd920007030 0x7fd92002aa00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-05T19:28:32.380 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.374+0000 7fd9350cd700 1 --2- 172.21.15.167:0/3309876569 >> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] conn(0x7fd920007030 0x7fd92002aa00 crc :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-05T19:28:32.380 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.374+0000 7fd9350cd700 1 -- 172.21.15.167:0/3309876569 <== osd.2 v2:172.21.15.167:6816/30037 1 ==== osd_op_reply(7 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fd930004520 con 0x7fd920007030 2023-10-05T19:28:32.380 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.374+0000 7fd9350cd700 1 --2- 172.21.15.167:0/3309876569 >> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] conn(0x7fd930011040 0x7fd930014950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-05T19:28:32.380 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.374+0000 7fd9350cd700 1 -- 172.21.15.167:0/3309876569 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:8 5.8 5:10000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x555cd043c010 con 0x7fd930011040 2023-10-05T19:28:32.380 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.374+0000 7fd9360cf700 1 --2- 172.21.15.167:0/3309876569 >> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] conn(0x7fd930011040 0x7fd930014950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-05T19:28:32.381 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.374+0000 7fd9360cf700 1 --2- 172.21.15.167:0/3309876569 >> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] conn(0x7fd930011040 0x7fd930014950 crc :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-05T19:28:32.381 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.374+0000 7fd9360cf700 1 -- 172.21.15.167:0/3309876569 <== osd.0 v2:172.21.15.118:6800/27690 1 ==== osd_op_reply(8 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fd928014040 con 0x7fd930011040 2023-10-05T19:28:32.382 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.374+0000 7fd9360cf700 1 -- 172.21.15.167:0/3309876569 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:9 5.18 5:18000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x555cd043c010 con 0x555cd0401670 2023-10-05T19:28:32.382 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.378+0000 7fd9360cf700 1 -- 172.21.15.167:0/3309876569 <== osd.1 v2:172.21.15.167:6800/30035 6 ==== osd_op_reply(9 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fd928014040 con 0x555cd0401670 2023-10-05T19:28:32.382 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.378+0000 7fd9360cf700 1 -- 172.21.15.167:0/3309876569 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:10 5.4 5:20000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x555cd043c010 con 0x555cd0401670 2023-10-05T19:28:32.383 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.378+0000 7fd9360cf700 1 -- 172.21.15.167:0/3309876569 <== osd.1 v2:172.21.15.167:6800/30035 7 ==== osd_op_reply(10 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fd928014040 con 0x555cd0401670 2023-10-05T19:28:32.383 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.378+0000 7fd9360cf700 1 -- 172.21.15.167:0/3309876569 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:11 5.14 5:28000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x555cd043c010 con 0x7fd930011040 2023-10-05T19:28:32.384 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.378+0000 7fd9360cf700 1 -- 172.21.15.167:0/3309876569 <== osd.0 v2:172.21.15.118:6800/27690 2 ==== osd_op_reply(11 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fd928014040 con 0x7fd930011040 2023-10-05T19:28:32.384 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.378+0000 7fd9360cf700 1 -- 172.21.15.167:0/3309876569 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:12 5.c 5:30000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x555cd043c010 con 0x7fd930011040 2023-10-05T19:28:32.384 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.378+0000 7fd9360cf700 1 -- 172.21.15.167:0/3309876569 <== osd.0 v2:172.21.15.118:6800/27690 3 ==== osd_op_reply(12 [pgnls start_epoch 30 out=112b] v23'1 uv1 ondisk = 1) v8 ==== 144+0+112 (crc 0 0 0) 0x7fd928014040 con 0x7fd930011040 2023-10-05T19:28:32.384 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.378+0000 7fd9360cf700 1 -- 172.21.15.167:0/3309876569 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:13 5.1c 5:38000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x555cd043c010 con 0x7fd920007030 2023-10-05T19:28:32.385 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.378+0000 7fd9350cd700 1 -- 172.21.15.167:0/3309876569 <== osd.2 v2:172.21.15.167:6816/30037 2 ==== osd_op_reply(13 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fd930004520 con 0x7fd920007030 2023-10-05T19:28:32.385 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.378+0000 7fd9350cd700 1 -- 172.21.15.167:0/3309876569 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:14 5.2 5:40000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x555cd043c010 con 0x555cd04396f0 2023-10-05T19:28:32.386 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.378+0000 7fd9358ce700 1 -- 172.21.15.167:0/3309876569 <== osd.3 v2:172.21.15.167:6808/30036 2 ==== osd_op_reply(14 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fd92002d070 con 0x555cd04396f0 2023-10-05T19:28:32.386 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.378+0000 7fd9358ce700 1 -- 172.21.15.167:0/3309876569 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:15 5.12 5:48000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x555cd043c010 con 0x555cd0401670 2023-10-05T19:28:32.386 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.382+0000 7fd9360cf700 1 -- 172.21.15.167:0/3309876569 <== osd.1 v2:172.21.15.167:6800/30035 8 ==== osd_op_reply(15 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fd928014040 con 0x555cd0401670 2023-10-05T19:28:32.386 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.382+0000 7fd9360cf700 1 -- 172.21.15.167:0/3309876569 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:16 5.a 5:50000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x555cd043c010 con 0x7fd930011040 2023-10-05T19:28:32.387 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.382+0000 7fd9360cf700 1 -- 172.21.15.167:0/3309876569 <== osd.0 v2:172.21.15.118:6800/27690 4 ==== osd_op_reply(16 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fd928022040 con 0x7fd930011040 2023-10-05T19:28:32.388 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.382+0000 7fd9360cf700 1 -- 172.21.15.167:0/3309876569 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:17 5.1a 5:58000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x555cd043c010 con 0x555cd0401670 2023-10-05T19:28:32.388 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.382+0000 7fd9360cf700 1 -- 172.21.15.167:0/3309876569 <== osd.1 v2:172.21.15.167:6800/30035 9 ==== osd_op_reply(17 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fd928022040 con 0x555cd0401670 2023-10-05T19:28:32.388 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.382+0000 7fd9360cf700 1 -- 172.21.15.167:0/3309876569 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:18 5.6 5:60000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x555cd043c010 con 0x7fd920007030 2023-10-05T19:28:32.389 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.382+0000 7fd9350cd700 1 -- 172.21.15.167:0/3309876569 <== osd.2 v2:172.21.15.167:6816/30037 3 ==== osd_op_reply(18 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fd930004520 con 0x7fd920007030 2023-10-05T19:28:32.389 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.382+0000 7fd9350cd700 1 -- 172.21.15.167:0/3309876569 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:19 5.16 5:68000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x555cd043c010 con 0x555cd04396f0 2023-10-05T19:28:32.389 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.382+0000 7fd9358ce700 1 -- 172.21.15.167:0/3309876569 <== osd.3 v2:172.21.15.167:6808/30036 3 ==== osd_op_reply(19 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fd92002d070 con 0x555cd04396f0 2023-10-05T19:28:32.390 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.382+0000 7fd9358ce700 1 -- 172.21.15.167:0/3309876569 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:20 5.e 5:70000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x555cd043c010 con 0x555cd04396f0 2023-10-05T19:28:32.390 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.386+0000 7fd9358ce700 1 -- 172.21.15.167:0/3309876569 <== osd.3 v2:172.21.15.167:6808/30036 4 ==== osd_op_reply(20 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fd92002d070 con 0x555cd04396f0 2023-10-05T19:28:32.390 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.386+0000 7fd9358ce700 1 -- 172.21.15.167:0/3309876569 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:21 5.1e 5:78000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x555cd043c010 con 0x7fd930011040 2023-10-05T19:28:32.391 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.386+0000 7fd9360cf700 1 -- 172.21.15.167:0/3309876569 <== osd.0 v2:172.21.15.118:6800/27690 5 ==== osd_op_reply(21 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fd928022040 con 0x7fd930011040 2023-10-05T19:28:32.391 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.386+0000 7fd9360cf700 1 -- 172.21.15.167:0/3309876569 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:22 5.1 5:80000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x555cd043c010 con 0x555cd04396f0 2023-10-05T19:28:32.392 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.386+0000 7fd9358ce700 1 -- 172.21.15.167:0/3309876569 <== osd.3 v2:172.21.15.167:6808/30036 5 ==== osd_op_reply(22 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fd92002d070 con 0x555cd04396f0 2023-10-05T19:28:32.392 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.386+0000 7fd9358ce700 1 -- 172.21.15.167:0/3309876569 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:23 5.11 5:88000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x555cd043c010 con 0x555cd04396f0 2023-10-05T19:28:32.393 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.386+0000 7fd9358ce700 1 -- 172.21.15.167:0/3309876569 <== osd.3 v2:172.21.15.167:6808/30036 6 ==== osd_op_reply(23 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fd92002d070 con 0x555cd04396f0 2023-10-05T19:28:32.393 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.386+0000 7fd9358ce700 1 -- 172.21.15.167:0/3309876569 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:24 5.9 5:90000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x555cd043c010 con 0x555cd0401670 2023-10-05T19:28:32.393 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.386+0000 7fd9360cf700 1 -- 172.21.15.167:0/3309876569 <== osd.1 v2:172.21.15.167:6800/30035 10 ==== osd_op_reply(24 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fd928022040 con 0x555cd0401670 2023-10-05T19:28:32.393 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.386+0000 7fd9360cf700 1 -- 172.21.15.167:0/3309876569 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:25 5.19 5:98000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x555cd043c010 con 0x555cd0401670 2023-10-05T19:28:32.394 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.390+0000 7fd9360cf700 1 -- 172.21.15.167:0/3309876569 <== osd.1 v2:172.21.15.167:6800/30035 11 ==== osd_op_reply(25 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fd928022040 con 0x555cd0401670 2023-10-05T19:28:32.394 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.390+0000 7fd9360cf700 1 -- 172.21.15.167:0/3309876569 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:26 5.5 5:a0000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x555cd043c010 con 0x7fd930011040 2023-10-05T19:28:32.395 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.390+0000 7fd9360cf700 1 -- 172.21.15.167:0/3309876569 <== osd.0 v2:172.21.15.118:6800/27690 6 ==== osd_op_reply(26 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fd928022040 con 0x7fd930011040 2023-10-05T19:28:32.395 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.390+0000 7fd9360cf700 1 -- 172.21.15.167:0/3309876569 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:27 5.15 5:a8000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x555cd043c010 con 0x555cd04396f0 2023-10-05T19:28:32.396 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.390+0000 7fd9358ce700 1 -- 172.21.15.167:0/3309876569 <== osd.3 v2:172.21.15.167:6808/30036 7 ==== osd_op_reply(27 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fd92002d070 con 0x555cd04396f0 2023-10-05T19:28:32.396 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.390+0000 7fd9358ce700 1 -- 172.21.15.167:0/3309876569 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:28 5.d 5:b0000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x555cd043c010 con 0x7fd920007030 2023-10-05T19:28:32.396 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.390+0000 7fd9350cd700 1 -- 172.21.15.167:0/3309876569 <== osd.2 v2:172.21.15.167:6816/30037 4 ==== osd_op_reply(28 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fd930004520 con 0x7fd920007030 2023-10-05T19:28:32.396 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.390+0000 7fd9350cd700 1 -- 172.21.15.167:0/3309876569 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:29 5.1d 5:b8000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x555cd043c010 con 0x555cd0401670 2023-10-05T19:28:32.397 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.390+0000 7fd9360cf700 1 -- 172.21.15.167:0/3309876569 <== osd.1 v2:172.21.15.167:6800/30035 12 ==== osd_op_reply(29 [pgnls start_epoch 30 out=107b] v30'2 uv2 ondisk = 1) v8 ==== 144+0+107 (crc 0 0 0) 0x7fd928022040 con 0x555cd0401670 2023-10-05T19:28:32.397 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.390+0000 7fd9360cf700 1 -- 172.21.15.167:0/3309876569 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:30 5.3 5:c0000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x555cd043c010 con 0x7fd930011040 2023-10-05T19:28:32.398 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.394+0000 7fd9360cf700 1 -- 172.21.15.167:0/3309876569 <== osd.0 v2:172.21.15.118:6800/27690 7 ==== osd_op_reply(30 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fd928022040 con 0x7fd930011040 2023-10-05T19:28:32.398 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.394+0000 7fd9360cf700 1 -- 172.21.15.167:0/3309876569 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:31 5.13 5:c8000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x555cd043c010 con 0x555cd04396f0 2023-10-05T19:28:32.399 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.394+0000 7fd9358ce700 1 -- 172.21.15.167:0/3309876569 <== osd.3 v2:172.21.15.167:6808/30036 8 ==== osd_op_reply(31 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fd92002d070 con 0x555cd04396f0 2023-10-05T19:28:32.399 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.394+0000 7fd9358ce700 1 -- 172.21.15.167:0/3309876569 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:32 5.b 5:d0000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x555cd043c010 con 0x7fd930011040 2023-10-05T19:28:32.400 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.394+0000 7fd9360cf700 1 -- 172.21.15.167:0/3309876569 <== osd.0 v2:172.21.15.118:6800/27690 8 ==== osd_op_reply(32 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fd928022040 con 0x7fd930011040 2023-10-05T19:28:32.400 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.394+0000 7fd9360cf700 1 -- 172.21.15.167:0/3309876569 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:33 5.1b 5:d8000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x555cd043c010 con 0x7fd920007030 2023-10-05T19:28:32.401 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.394+0000 7fd9350cd700 1 -- 172.21.15.167:0/3309876569 <== osd.2 v2:172.21.15.167:6816/30037 5 ==== osd_op_reply(33 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fd930004520 con 0x7fd920007030 2023-10-05T19:28:32.401 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.394+0000 7fd9350cd700 1 -- 172.21.15.167:0/3309876569 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:34 5.7 5:e0000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x555cd043c010 con 0x555cd04396f0 2023-10-05T19:28:32.401 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.394+0000 7fd9358ce700 1 -- 172.21.15.167:0/3309876569 <== osd.3 v2:172.21.15.167:6808/30036 9 ==== osd_op_reply(34 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fd92002d070 con 0x555cd04396f0 2023-10-05T19:28:32.401 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.394+0000 7fd9358ce700 1 -- 172.21.15.167:0/3309876569 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:35 5.17 5:e8000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x555cd043c010 con 0x7fd930011040 2023-10-05T19:28:32.402 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.398+0000 7fd9360cf700 1 -- 172.21.15.167:0/3309876569 <== osd.0 v2:172.21.15.118:6800/27690 9 ==== osd_op_reply(35 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fd928022040 con 0x7fd930011040 2023-10-05T19:28:32.402 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.398+0000 7fd9360cf700 1 -- 172.21.15.167:0/3309876569 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:36 5.f 5:f0000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x555cd043c010 con 0x555cd0401670 2023-10-05T19:28:32.403 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.398+0000 7fd9360cf700 1 -- 172.21.15.167:0/3309876569 <== osd.1 v2:172.21.15.167:6800/30035 13 ==== osd_op_reply(36 [pgnls start_epoch 30 out=115b] v23'2 uv2 ondisk = 1) v8 ==== 144+0+115 (crc 0 0 0) 0x7fd92802f080 con 0x555cd0401670 2023-10-05T19:28:32.403 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.398+0000 7fd9360cf700 1 -- 172.21.15.167:0/3309876569 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:37 5.1f 5:f8000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x555cd043c010 con 0x7fd920007030 2023-10-05T19:28:32.404 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.398+0000 7fd9350cd700 1 -- 172.21.15.167:0/3309876569 <== osd.2 v2:172.21.15.167:6816/30037 6 ==== osd_op_reply(37 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fd930004520 con 0x7fd920007030 2023-10-05T19:28:32.404 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.398+0000 7fd937d99e40 20 RGWRados::pool_iterate: got zonegroup_info.4f1f719d-9648-417b-b10c-e5d939825419 2023-10-05T19:28:32.404 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.398+0000 7fd937d99e40 20 RGWRados::pool_iterate: got zone_info.9a9df965-6612-45d4-a45f-6e3c889bacc2 2023-10-05T19:28:32.405 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.398+0000 7fd937d99e40 20 RGWRados::pool_iterate: got zone_names.default 2023-10-05T19:28:32.405 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.398+0000 7fd937d99e40 20 RGWRados::pool_iterate: got zonegroups_names.default 2023-10-05T19:28:32.405 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.398+0000 7fd937d99e40 20 rados->read ofs=0 len=0 2023-10-05T19:28:32.405 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.398+0000 7fd937d99e40 1 -- 172.21.15.167:0/3309876569 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:38 5.12 5:49953fa1:::default.realm:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x555cd04384a0 con 0x555cd0401670 2023-10-05T19:28:32.406 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.398+0000 7fd9360cf700 1 -- 172.21.15.167:0/3309876569 <== osd.1 v2:172.21.15.167:6800/30035 14 ==== osd_op_reply(38 default.realm [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 ==== 157+0+0 (crc 0 0 0) 0x7fd92802f080 con 0x555cd0401670 2023-10-05T19:28:32.406 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.398+0000 7fd937d99e40 20 rados_obj.operate() r=-2 bl.length=0 2023-10-05T19:28:32.406 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.398+0000 7fd937d99e40 20 rados->read ofs=0 len=0 2023-10-05T19:28:32.406 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.398+0000 7fd937d99e40 1 -- 172.21.15.167:0/3309876569 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:39 5.f 5:f4c53578:::zonegroups_names.default:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x555cd0438d90 con 0x555cd0401670 2023-10-05T19:28:32.406 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.398+0000 7fd9360cf700 1 -- 172.21.15.167:0/3309876569 <== osd.1 v2:172.21.15.167:6800/30035 15 ==== osd_op_reply(39 zonegroups_names.default [read 0~46 out=46b] v0'0 uv2 ondisk = 0) v8 ==== 168+0+46 (crc 0 0 0) 0x7fd92802f080 con 0x555cd0401670 2023-10-05T19:28:32.406 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.398+0000 7fd937d99e40 20 rados_obj.operate() r=0 bl.length=46 2023-10-05T19:28:32.406 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.398+0000 7fd937d99e40 20 rados->read ofs=0 len=0 2023-10-05T19:28:32.406 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.398+0000 7fd937d99e40 1 -- 172.21.15.167:0/3309876569 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:40 5.c 5:30d299a5:::zonegroup_info.4f1f719d-9648-417b-b10c-e5d939825419:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x555cd043c480 con 0x7fd930011040 2023-10-05T19:28:32.407 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.402+0000 7fd9360cf700 1 -- 172.21.15.167:0/3309876569 <== osd.0 v2:172.21.15.118:6800/27690 10 ==== osd_op_reply(40 zonegroup_info.4f1f719d-9648-417b-b10c-e5d939825419 [read 0~362 out=362b] v0'0 uv1 ondisk = 0) v8 ==== 195+0+362 (crc 0 0 0) 0x7fd92802f080 con 0x7fd930011040 2023-10-05T19:28:32.407 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.402+0000 7fd937d99e40 20 rados_obj.operate() r=0 bl.length=362 2023-10-05T19:28:32.407 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.402+0000 7fd937d99e40 20 rados->read ofs=0 len=0 2023-10-05T19:28:32.408 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.402+0000 7fd937d99e40 1 -- 172.21.15.167:0/3309876569 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:41 5.13 5:cbbf6c60:::converted:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x555cd043c480 con 0x555cd04396f0 2023-10-05T19:28:32.408 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.402+0000 7fd9358ce700 1 -- 172.21.15.167:0/3309876569 <== osd.3 v2:172.21.15.167:6808/30036 10 ==== osd_op_reply(41 converted [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 ==== 153+0+0 (crc 0 0 0) 0x7fd92002d070 con 0x555cd04396f0 2023-10-05T19:28:32.408 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.402+0000 7fd937d99e40 20 rados_obj.operate() r=-2 bl.length=0 2023-10-05T19:28:32.408 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.402+0000 7fd937d99e40 20 rados->read ofs=0 len=0 2023-10-05T19:28:32.408 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.402+0000 7fd937d99e40 1 -- 172.21.15.167:0/3309876569 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:42 5.12 5:49953fa1:::default.realm:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x555cd043c830 con 0x555cd0401670 2023-10-05T19:28:32.409 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.402+0000 7fd9360cf700 1 -- 172.21.15.167:0/3309876569 <== osd.1 v2:172.21.15.167:6800/30035 16 ==== osd_op_reply(42 default.realm [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 ==== 157+0+0 (crc 0 0 0) 0x7fd92802f080 con 0x555cd0401670 2023-10-05T19:28:32.409 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.402+0000 7fd937d99e40 20 rados_obj.operate() r=-2 bl.length=0 2023-10-05T19:28:32.409 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.402+0000 7fd937d99e40 20 rados->read ofs=0 len=0 2023-10-05T19:28:32.409 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.402+0000 7fd937d99e40 1 -- 172.21.15.167:0/3309876569 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:43 5.f 5:f4c53578:::zonegroups_names.default:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x555cd043cc30 con 0x555cd0401670 2023-10-05T19:28:32.409 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.402+0000 7fd9360cf700 1 -- 172.21.15.167:0/3309876569 <== osd.1 v2:172.21.15.167:6800/30035 17 ==== osd_op_reply(43 zonegroups_names.default [read 0~46 out=46b] v0'0 uv2 ondisk = 0) v8 ==== 168+0+46 (crc 0 0 0) 0x7fd92802f080 con 0x555cd0401670 2023-10-05T19:28:32.409 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.402+0000 7fd937d99e40 20 rados_obj.operate() r=0 bl.length=46 2023-10-05T19:28:32.409 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.402+0000 7fd937d99e40 1 -- 172.21.15.167:0/3309876569 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:44 5.0 5:00000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x555cd043dc40 con 0x555cd04396f0 2023-10-05T19:28:32.411 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.406+0000 7fd9358ce700 1 -- 172.21.15.167:0/3309876569 <== osd.3 v2:172.21.15.167:6808/30036 11 ==== osd_op_reply(44 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fd92002d070 con 0x555cd04396f0 2023-10-05T19:28:32.411 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.406+0000 7fd9358ce700 1 -- 172.21.15.167:0/3309876569 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:45 5.10 5:08000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x555cd043dc40 con 0x7fd920007030 2023-10-05T19:28:32.412 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.406+0000 7fd9350cd700 1 -- 172.21.15.167:0/3309876569 <== osd.2 v2:172.21.15.167:6816/30037 7 ==== osd_op_reply(45 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fd930004520 con 0x7fd920007030 2023-10-05T19:28:32.412 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.406+0000 7fd9350cd700 1 -- 172.21.15.167:0/3309876569 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:46 5.8 5:10000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x555cd043dc40 con 0x7fd930011040 2023-10-05T19:28:32.413 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.406+0000 7fd9360cf700 1 -- 172.21.15.167:0/3309876569 <== osd.0 v2:172.21.15.118:6800/27690 11 ==== osd_op_reply(46 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fd92802f080 con 0x7fd930011040 2023-10-05T19:28:32.413 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.406+0000 7fd9360cf700 1 -- 172.21.15.167:0/3309876569 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:47 5.18 5:18000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x555cd0438d90 con 0x555cd0401670 2023-10-05T19:28:32.414 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.410+0000 7fd9360cf700 1 -- 172.21.15.167:0/3309876569 <== osd.1 v2:172.21.15.167:6800/30035 18 ==== osd_op_reply(47 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fd92802f080 con 0x555cd0401670 2023-10-05T19:28:32.414 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.410+0000 7fd9360cf700 1 -- 172.21.15.167:0/3309876569 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:48 5.4 5:20000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x555cd0438d90 con 0x555cd0401670 2023-10-05T19:28:32.415 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.410+0000 7fd9360cf700 1 -- 172.21.15.167:0/3309876569 <== osd.1 v2:172.21.15.167:6800/30035 19 ==== osd_op_reply(48 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fd92802f080 con 0x555cd0401670 2023-10-05T19:28:32.415 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.410+0000 7fd9360cf700 1 -- 172.21.15.167:0/3309876569 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:49 5.14 5:28000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x555cd0438d90 con 0x7fd930011040 2023-10-05T19:28:32.415 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.410+0000 7fd9360cf700 1 -- 172.21.15.167:0/3309876569 <== osd.0 v2:172.21.15.118:6800/27690 12 ==== osd_op_reply(49 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fd92802f080 con 0x7fd930011040 2023-10-05T19:28:32.416 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.410+0000 7fd9360cf700 1 -- 172.21.15.167:0/3309876569 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:50 5.c 5:30000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x555cd0438d90 con 0x7fd930011040 2023-10-05T19:28:32.416 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.410+0000 7fd9360cf700 1 -- 172.21.15.167:0/3309876569 <== osd.0 v2:172.21.15.118:6800/27690 13 ==== osd_op_reply(50 [pgnls start_epoch 30 out=112b] v23'1 uv1 ondisk = 1) v8 ==== 144+0+112 (crc 0 0 0) 0x7fd92802f080 con 0x7fd930011040 2023-10-05T19:28:32.416 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.410+0000 7fd9360cf700 1 -- 172.21.15.167:0/3309876569 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:51 5.1c 5:38000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x555cd0438d90 con 0x7fd920007030 2023-10-05T19:28:32.417 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.410+0000 7fd9350cd700 1 -- 172.21.15.167:0/3309876569 <== osd.2 v2:172.21.15.167:6816/30037 8 ==== osd_op_reply(51 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fd930004520 con 0x7fd920007030 2023-10-05T19:28:32.417 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.410+0000 7fd9350cd700 1 -- 172.21.15.167:0/3309876569 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:52 5.2 5:40000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x555cd0438d90 con 0x555cd04396f0 2023-10-05T19:28:32.418 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.414+0000 7fd9358ce700 1 -- 172.21.15.167:0/3309876569 <== osd.3 v2:172.21.15.167:6808/30036 12 ==== osd_op_reply(52 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fd92002d070 con 0x555cd04396f0 2023-10-05T19:28:32.418 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.414+0000 7fd9358ce700 1 -- 172.21.15.167:0/3309876569 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:53 5.12 5:48000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x555cd0438d90 con 0x555cd0401670 2023-10-05T19:28:32.419 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.414+0000 7fd9360cf700 1 -- 172.21.15.167:0/3309876569 <== osd.1 v2:172.21.15.167:6800/30035 20 ==== osd_op_reply(53 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fd92802f080 con 0x555cd0401670 2023-10-05T19:28:32.419 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.414+0000 7fd9360cf700 1 -- 172.21.15.167:0/3309876569 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:54 5.a 5:50000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x555cd0438d90 con 0x7fd930011040 2023-10-05T19:28:32.420 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.414+0000 7fd9360cf700 1 -- 172.21.15.167:0/3309876569 <== osd.0 v2:172.21.15.118:6800/27690 14 ==== osd_op_reply(54 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fd92802f080 con 0x7fd930011040 2023-10-05T19:28:32.420 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.414+0000 7fd9360cf700 1 -- 172.21.15.167:0/3309876569 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:55 5.1a 5:58000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x555cd0438d90 con 0x555cd0401670 2023-10-05T19:28:32.420 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.414+0000 7fd9360cf700 1 -- 172.21.15.167:0/3309876569 <== osd.1 v2:172.21.15.167:6800/30035 21 ==== osd_op_reply(55 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fd92802f080 con 0x555cd0401670 2023-10-05T19:28:32.420 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.414+0000 7fd9360cf700 1 -- 172.21.15.167:0/3309876569 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:56 5.6 5:60000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x555cd0438d90 con 0x7fd920007030 2023-10-05T19:28:32.422 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.414+0000 7fd9350cd700 1 -- 172.21.15.167:0/3309876569 <== osd.2 v2:172.21.15.167:6816/30037 9 ==== osd_op_reply(56 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fd930004520 con 0x7fd920007030 2023-10-05T19:28:32.422 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.414+0000 7fd9350cd700 1 -- 172.21.15.167:0/3309876569 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:57 5.16 5:68000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x555cd0438d90 con 0x555cd04396f0 2023-10-05T19:28:32.422 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.418+0000 7fd9358ce700 1 -- 172.21.15.167:0/3309876569 <== osd.3 v2:172.21.15.167:6808/30036 13 ==== osd_op_reply(57 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fd92002d070 con 0x555cd04396f0 2023-10-05T19:28:32.422 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.418+0000 7fd9358ce700 1 -- 172.21.15.167:0/3309876569 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:58 5.e 5:70000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x555cd0438d90 con 0x555cd04396f0 2023-10-05T19:28:32.423 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.418+0000 7fd9358ce700 1 -- 172.21.15.167:0/3309876569 <== osd.3 v2:172.21.15.167:6808/30036 14 ==== osd_op_reply(58 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fd92002d070 con 0x555cd04396f0 2023-10-05T19:28:32.423 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.418+0000 7fd9358ce700 1 -- 172.21.15.167:0/3309876569 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:59 5.1e 5:78000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x555cd0438d90 con 0x7fd930011040 2023-10-05T19:28:32.424 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.418+0000 7fd9360cf700 1 -- 172.21.15.167:0/3309876569 <== osd.0 v2:172.21.15.118:6800/27690 15 ==== osd_op_reply(59 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fd92802f080 con 0x7fd930011040 2023-10-05T19:28:32.424 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.418+0000 7fd9360cf700 1 -- 172.21.15.167:0/3309876569 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:60 5.1 5:80000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x555cd0438d90 con 0x555cd04396f0 2023-10-05T19:28:32.424 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.418+0000 7fd9358ce700 1 -- 172.21.15.167:0/3309876569 <== osd.3 v2:172.21.15.167:6808/30036 15 ==== osd_op_reply(60 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fd92002d070 con 0x555cd04396f0 2023-10-05T19:28:32.424 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.418+0000 7fd9358ce700 1 -- 172.21.15.167:0/3309876569 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:61 5.11 5:88000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x555cd0438d90 con 0x555cd04396f0 2023-10-05T19:28:32.425 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.418+0000 7fd9358ce700 1 -- 172.21.15.167:0/3309876569 <== osd.3 v2:172.21.15.167:6808/30036 16 ==== osd_op_reply(61 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fd92002d070 con 0x555cd04396f0 2023-10-05T19:28:32.425 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.418+0000 7fd9358ce700 1 -- 172.21.15.167:0/3309876569 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:62 5.9 5:90000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x555cd0438d90 con 0x555cd0401670 2023-10-05T19:28:32.426 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.422+0000 7fd9360cf700 1 -- 172.21.15.167:0/3309876569 <== osd.1 v2:172.21.15.167:6800/30035 22 ==== osd_op_reply(62 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fd92802f080 con 0x555cd0401670 2023-10-05T19:28:32.426 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.422+0000 7fd9360cf700 1 -- 172.21.15.167:0/3309876569 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:63 5.19 5:98000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x555cd0438d90 con 0x555cd0401670 2023-10-05T19:28:32.427 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.422+0000 7fd9360cf700 1 -- 172.21.15.167:0/3309876569 <== osd.1 v2:172.21.15.167:6800/30035 23 ==== osd_op_reply(63 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fd92802f080 con 0x555cd0401670 2023-10-05T19:28:32.427 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.422+0000 7fd9360cf700 1 -- 172.21.15.167:0/3309876569 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:64 5.5 5:a0000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x555cd0438d90 con 0x7fd930011040 2023-10-05T19:28:32.427 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.422+0000 7fd9360cf700 1 -- 172.21.15.167:0/3309876569 <== osd.0 v2:172.21.15.118:6800/27690 16 ==== osd_op_reply(64 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fd92802f080 con 0x7fd930011040 2023-10-05T19:28:32.428 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.422+0000 7fd9360cf700 1 -- 172.21.15.167:0/3309876569 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:65 5.15 5:a8000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x555cd0438d90 con 0x555cd04396f0 2023-10-05T19:28:32.428 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.422+0000 7fd9358ce700 1 -- 172.21.15.167:0/3309876569 <== osd.3 v2:172.21.15.167:6808/30036 17 ==== osd_op_reply(65 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fd92002d070 con 0x555cd04396f0 2023-10-05T19:28:32.428 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.422+0000 7fd9358ce700 1 -- 172.21.15.167:0/3309876569 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:66 5.d 5:b0000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x555cd0438d90 con 0x7fd920007030 2023-10-05T19:28:32.429 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.422+0000 7fd9350cd700 1 -- 172.21.15.167:0/3309876569 <== osd.2 v2:172.21.15.167:6816/30037 10 ==== osd_op_reply(66 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fd930004520 con 0x7fd920007030 2023-10-05T19:28:32.429 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.422+0000 7fd9350cd700 1 -- 172.21.15.167:0/3309876569 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:67 5.1d 5:b8000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x555cd0438d90 con 0x555cd0401670 2023-10-05T19:28:32.430 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.426+0000 7fd9360cf700 1 -- 172.21.15.167:0/3309876569 <== osd.1 v2:172.21.15.167:6800/30035 24 ==== osd_op_reply(67 [pgnls start_epoch 30 out=107b] v30'2 uv2 ondisk = 1) v8 ==== 144+0+107 (crc 0 0 0) 0x7fd92802f080 con 0x555cd0401670 2023-10-05T19:28:32.430 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.426+0000 7fd9360cf700 1 -- 172.21.15.167:0/3309876569 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:68 5.3 5:c0000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x555cd0438d90 con 0x7fd930011040 2023-10-05T19:28:32.431 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.426+0000 7fd9360cf700 1 -- 172.21.15.167:0/3309876569 <== osd.0 v2:172.21.15.118:6800/27690 17 ==== osd_op_reply(68 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fd92802f080 con 0x7fd930011040 2023-10-05T19:28:32.431 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.426+0000 7fd9360cf700 1 -- 172.21.15.167:0/3309876569 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:69 5.13 5:c8000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x555cd0438d90 con 0x555cd04396f0 2023-10-05T19:28:32.432 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.426+0000 7fd9358ce700 1 -- 172.21.15.167:0/3309876569 <== osd.3 v2:172.21.15.167:6808/30036 18 ==== osd_op_reply(69 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fd92002d070 con 0x555cd04396f0 2023-10-05T19:28:32.432 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.426+0000 7fd9358ce700 1 -- 172.21.15.167:0/3309876569 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:70 5.b 5:d0000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x555cd0438d90 con 0x7fd930011040 2023-10-05T19:28:32.432 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.426+0000 7fd9360cf700 1 -- 172.21.15.167:0/3309876569 <== osd.0 v2:172.21.15.118:6800/27690 18 ==== osd_op_reply(70 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fd92802f080 con 0x7fd930011040 2023-10-05T19:28:32.432 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.426+0000 7fd9360cf700 1 -- 172.21.15.167:0/3309876569 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:71 5.1b 5:d8000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x555cd0438d90 con 0x7fd920007030 2023-10-05T19:28:32.433 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.426+0000 7fd9350cd700 1 -- 172.21.15.167:0/3309876569 <== osd.2 v2:172.21.15.167:6816/30037 11 ==== osd_op_reply(71 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fd930004520 con 0x7fd920007030 2023-10-05T19:28:32.433 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.426+0000 7fd9350cd700 1 -- 172.21.15.167:0/3309876569 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:72 5.7 5:e0000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x555cd0438d90 con 0x555cd04396f0 2023-10-05T19:28:32.434 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.430+0000 7fd9358ce700 1 -- 172.21.15.167:0/3309876569 <== osd.3 v2:172.21.15.167:6808/30036 19 ==== osd_op_reply(72 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fd92002d070 con 0x555cd04396f0 2023-10-05T19:28:32.434 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.430+0000 7fd9358ce700 1 -- 172.21.15.167:0/3309876569 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:73 5.17 5:e8000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x555cd0438d90 con 0x7fd930011040 2023-10-05T19:28:32.435 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.430+0000 7fd9360cf700 1 -- 172.21.15.167:0/3309876569 <== osd.0 v2:172.21.15.118:6800/27690 19 ==== osd_op_reply(73 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fd92802f080 con 0x7fd930011040 2023-10-05T19:28:32.435 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.430+0000 7fd9360cf700 1 -- 172.21.15.167:0/3309876569 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:74 5.f 5:f0000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x555cd0438d90 con 0x555cd0401670 2023-10-05T19:28:32.436 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.430+0000 7fd9360cf700 1 -- 172.21.15.167:0/3309876569 <== osd.1 v2:172.21.15.167:6800/30035 25 ==== osd_op_reply(74 [pgnls start_epoch 30 out=115b] v23'2 uv2 ondisk = 1) v8 ==== 144+0+115 (crc 0 0 0) 0x7fd92802f080 con 0x555cd0401670 2023-10-05T19:28:32.436 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.430+0000 7fd9360cf700 1 -- 172.21.15.167:0/3309876569 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:75 5.1f 5:f8000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x555cd0438d90 con 0x7fd920007030 2023-10-05T19:28:32.437 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.430+0000 7fd9350cd700 1 -- 172.21.15.167:0/3309876569 <== osd.2 v2:172.21.15.167:6816/30037 12 ==== osd_op_reply(75 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fd930004520 con 0x7fd920007030 2023-10-05T19:28:32.437 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.430+0000 7fd937d99e40 20 RGWRados::pool_iterate: got zonegroup_info.4f1f719d-9648-417b-b10c-e5d939825419 2023-10-05T19:28:32.437 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.430+0000 7fd937d99e40 20 RGWRados::pool_iterate: got zone_info.9a9df965-6612-45d4-a45f-6e3c889bacc2 2023-10-05T19:28:32.437 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.430+0000 7fd937d99e40 20 RGWRados::pool_iterate: got zone_names.default 2023-10-05T19:28:32.437 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.430+0000 7fd937d99e40 20 RGWRados::pool_iterate: got zonegroups_names.default 2023-10-05T19:28:32.437 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.430+0000 7fd937d99e40 20 rados->read ofs=0 len=0 2023-10-05T19:28:32.437 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.430+0000 7fd937d99e40 1 -- 172.21.15.167:0/3309876569 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:76 5.f 5:f43b6ece:::zone_names.default:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x555cd043cc30 con 0x555cd0401670 2023-10-05T19:28:32.438 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.430+0000 7fd9360cf700 1 -- 172.21.15.167:0/3309876569 <== osd.1 v2:172.21.15.167:6800/30035 26 ==== osd_op_reply(76 zone_names.default [read 0~46 out=46b] v0'0 uv1 ondisk = 0) v8 ==== 162+0+46 (crc 0 0 0) 0x7fd92802f080 con 0x555cd0401670 2023-10-05T19:28:32.438 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.430+0000 7fd937d99e40 20 rados_obj.operate() r=0 bl.length=46 2023-10-05T19:28:32.438 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.430+0000 7fd937d99e40 20 rados->read ofs=0 len=0 2023-10-05T19:28:32.438 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.430+0000 7fd937d99e40 1 -- 172.21.15.167:0/3309876569 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:77 5.1d 5:b8374eb4:::zone_info.9a9df965-6612-45d4-a45f-6e3c889bacc2:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x555cd043cfe0 con 0x555cd0401670 2023-10-05T19:28:32.439 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.434+0000 7fd9360cf700 1 -- 172.21.15.167:0/3309876569 <== osd.1 v2:172.21.15.167:6800/30035 27 ==== osd_op_reply(77 zone_info.9a9df965-6612-45d4-a45f-6e3c889bacc2 [read 0~874 out=874b] v0'0 uv2 ondisk = 0) v8 ==== 190+0+874 (crc 0 0 0) 0x7fd92802f080 con 0x555cd0401670 2023-10-05T19:28:32.439 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.434+0000 7fd937d99e40 20 rados_obj.operate() r=0 bl.length=874 2023-10-05T19:28:32.439 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.434+0000 7fd937d99e40 20 rados->read ofs=0 len=0 2023-10-05T19:28:32.441 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.434+0000 7fd937d99e40 1 -- 172.21.15.167:0/3309876569 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:78 5.f 5:f4c53578:::zonegroups_names.default:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x555cd043cfe0 con 0x555cd0401670 2023-10-05T19:28:32.444 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.434+0000 7fd9360cf700 1 -- 172.21.15.167:0/3309876569 <== osd.1 v2:172.21.15.167:6800/30035 28 ==== osd_op_reply(78 zonegroups_names.default [read 0~46 out=46b] v0'0 uv2 ondisk = 0) v8 ==== 168+0+46 (crc 0 0 0) 0x7fd92802f080 con 0x555cd0401670 2023-10-05T19:28:32.444 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.434+0000 7fd937d99e40 20 rados_obj.operate() r=0 bl.length=46 2023-10-05T19:28:32.444 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.434+0000 7fd937d99e40 20 rados->read ofs=0 len=0 2023-10-05T19:28:32.444 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.434+0000 7fd937d99e40 1 -- 172.21.15.167:0/3309876569 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:79 5.c 5:30d299a5:::zonegroup_info.4f1f719d-9648-417b-b10c-e5d939825419:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x555cd043d410 con 0x7fd930011040 2023-10-05T19:28:32.444 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.434+0000 7fd9360cf700 1 -- 172.21.15.167:0/3309876569 <== osd.0 v2:172.21.15.118:6800/27690 20 ==== osd_op_reply(79 zonegroup_info.4f1f719d-9648-417b-b10c-e5d939825419 [read 0~362 out=362b] v0'0 uv1 ondisk = 0) v8 ==== 195+0+362 (crc 0 0 0) 0x7fd92802f080 con 0x7fd930011040 2023-10-05T19:28:32.444 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.434+0000 7fd937d99e40 20 rados_obj.operate() r=0 bl.length=362 2023-10-05T19:28:32.444 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.434+0000 7fd937d99e40 20 rados->read ofs=0 len=0 2023-10-05T19:28:32.445 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.434+0000 7fd937d99e40 1 -- 172.21.15.167:0/3309876569 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:80 5.3 5:c0ad8cbd:::region_map:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x555cd043d410 con 0x7fd930011040 2023-10-05T19:28:32.445 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.434+0000 7fd9360cf700 1 -- 172.21.15.167:0/3309876569 <== osd.0 v2:172.21.15.118:6800/27690 21 ==== osd_op_reply(80 region_map [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 ==== 154+0+0 (crc 0 0 0) 0x7fd928050040 con 0x7fd930011040 2023-10-05T19:28:32.445 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.434+0000 7fd937d99e40 20 rados_obj.operate() r=-2 bl.length=0 2023-10-05T19:28:32.445 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.434+0000 7fd937d99e40 20 zone default found 2023-10-05T19:28:32.445 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.434+0000 7fd937d99e40 4 Realm: () 2023-10-05T19:28:32.445 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.434+0000 7fd937d99e40 4 ZoneGroup: default (4f1f719d-9648-417b-b10c-e5d939825419) 2023-10-05T19:28:32.445 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.434+0000 7fd937d99e40 4 Zone: default (9a9df965-6612-45d4-a45f-6e3c889bacc2) 2023-10-05T19:28:32.445 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.434+0000 7fd937d99e40 10 cannot find current period zonegroup using local zonegroup configuration 2023-10-05T19:28:32.445 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.434+0000 7fd937d99e40 20 zonegroup default 2023-10-05T19:28:32.445 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.434+0000 7fd937d99e40 20 rados->read ofs=0 len=0 2023-10-05T19:28:32.445 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.434+0000 7fd937d99e40 1 -- 172.21.15.167:0/3309876569 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:81 5.3 5:c52100b6:::period_config.default:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x555cd043d2d0 con 0x7fd930011040 2023-10-05T19:28:32.446 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.434+0000 7fd9360cf700 1 -- 172.21.15.167:0/3309876569 <== osd.0 v2:172.21.15.118:6800/27690 22 ==== osd_op_reply(81 period_config.default [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 ==== 165+0+0 (crc 0 0 0) 0x7fd928050040 con 0x7fd930011040 2023-10-05T19:28:32.446 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.434+0000 7fd937d99e40 20 rados_obj.operate() r=-2 bl.length=0 2023-10-05T19:28:32.446 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.434+0000 7fd937d99e40 20 rados->read ofs=0 len=0 2023-10-05T19:28:32.446 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.434+0000 7fd937d99e40 1 -- 172.21.15.167:0/3309876569 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:82 6.2 6:4347d321:::bucket.sync-source-hints.:head [call version.read in=11b,read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x555cd043ec50 con 0x555cd04396f0 2023-10-05T19:28:32.446 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.438+0000 7fd9358ce700 1 -- 172.21.15.167:0/3309876569 <== osd.3 v2:172.21.15.167:6808/30036 20 ==== osd_op_reply(82 bucket.sync-source-hints. [call,read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 ==== 211+0+0 (crc 0 0 0) 0x7fd92002d070 con 0x555cd04396f0 2023-10-05T19:28:32.446 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.438+0000 7fd937d99e40 20 rados_obj.operate() r=-2 bl.length=0 2023-10-05T19:28:32.446 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.438+0000 7fd937d99e40 20 rados->read ofs=0 len=0 2023-10-05T19:28:32.446 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.438+0000 7fd937d99e40 1 -- 172.21.15.167:0/3309876569 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:83 6.b 6:d467b91b:::bucket.sync-target-hints.:head [call version.read in=11b,read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x555cd043f000 con 0x555cd04396f0 2023-10-05T19:28:32.446 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.438+0000 7fd9358ce700 1 -- 172.21.15.167:0/3309876569 <== osd.3 v2:172.21.15.167:6808/30036 21 ==== osd_op_reply(83 bucket.sync-target-hints. [call,read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 ==== 211+0+0 (crc 0 0 0) 0x7fd92002d070 con 0x555cd04396f0 2023-10-05T19:28:32.446 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.438+0000 7fd937d99e40 20 rados_obj.operate() r=-2 bl.length=0 2023-10-05T19:28:32.446 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.438+0000 7fd937d99e40 20 started sync module instance, tier type = 2023-10-05T19:28:32.447 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.438+0000 7fd937d99e40 20 started zone id=9a9df965-6612-45d4-a45f-6e3c889bacc2 (name=default) with tier type = 2023-10-05T19:28:32.447 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.438+0000 7fd937d99e40 1 -- 172.21.15.167:0/3309876569 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:84 7.1f 7:f95f44c2:::notify.0:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x555cd043e310 con 0x7fd920007030 2023-10-05T19:28:32.447 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.438+0000 7fd9350cd700 1 -- 172.21.15.167:0/3309876569 <== osd.2 v2:172.21.15.167:6816/30037 13 ==== osd_op_reply(84 notify.0 [create] v30'108 uv108 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7fd93002f040 con 0x7fd920007030 2023-10-05T19:28:32.447 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.442+0000 7fd937d99e40 1 -- 172.21.15.167:0/3309876569 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:85 7.1f 7:f95f44c2:::notify.0:head [watch watch cookie 93857119463184] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x555cd04389c0 con 0x7fd920007030 2023-10-05T19:28:32.447 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.442+0000 7fd937d99e40 1 -- 172.21.15.167:0/3309876569 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:86 7.0 7:05bf5b68:::notify.1:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x555cd043fdd0 con 0x555cd0401670 2023-10-05T19:28:32.449 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.442+0000 7fd9350cd700 1 -- 172.21.15.167:0/3309876569 <== osd.2 v2:172.21.15.167:6816/30037 14 ==== osd_op_reply(85 notify.0 [watch watch cookie 93857119463184] v30'109 uv108 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7fd93002f040 con 0x7fd920007030 2023-10-05T19:28:32.449 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.442+0000 7fd9360cf700 1 -- 172.21.15.167:0/3309876569 <== osd.1 v2:172.21.15.167:6800/30035 29 ==== osd_op_reply(86 notify.1 [create] v30'108 uv108 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7fd928050040 con 0x555cd0401670 2023-10-05T19:28:32.449 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.442+0000 7fd937d99e40 1 -- 172.21.15.167:0/3309876569 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:87 7.0 7:05bf5b68:::notify.1:head [watch watch cookie 93857119466416] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x555cd04401d0 con 0x555cd0401670 2023-10-05T19:28:32.449 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.442+0000 7fd937d99e40 1 -- 172.21.15.167:0/3309876569 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:88 7.15 7:a93a5511:::notify.2:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x555cd0440a70 con 0x555cd04396f0 2023-10-05T19:28:32.452 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.446+0000 7fd9360cf700 1 -- 172.21.15.167:0/3309876569 <== osd.1 v2:172.21.15.167:6800/30035 30 ==== osd_op_reply(87 notify.1 [watch watch cookie 93857119466416] v30'109 uv108 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7fd928050040 con 0x555cd0401670 2023-10-05T19:28:32.453 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.446+0000 7fd9358ce700 1 -- 172.21.15.167:0/3309876569 <== osd.3 v2:172.21.15.167:6808/30036 22 ==== osd_op_reply(88 notify.2 [create] v30'108 uv108 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7fd92002d070 con 0x555cd04396f0 2023-10-05T19:28:32.453 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.446+0000 7fd937d99e40 1 -- 172.21.15.167:0/3309876569 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:89 7.15 7:a93a5511:::notify.2:head [watch watch cookie 93857119470032] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x555cd043f920 con 0x555cd04396f0 2023-10-05T19:28:32.453 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.446+0000 7fd937d99e40 1 -- 172.21.15.167:0/3309876569 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:90 7.e 7:7759931f:::notify.3:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x555cd0441800 con 0x7fd920007030 2023-10-05T19:28:32.455 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.450+0000 7fd9350cd700 1 -- 172.21.15.167:0/3309876569 <== osd.2 v2:172.21.15.167:6816/30037 15 ==== osd_op_reply(90 notify.3 [create] v30'108 uv108 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7fd93002f040 con 0x7fd920007030 2023-10-05T19:28:32.455 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.450+0000 7fd937d99e40 1 -- 172.21.15.167:0/3309876569 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:91 7.e 7:7759931f:::notify.3:head [watch watch cookie 93857119474448] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x555cd04420a0 con 0x7fd920007030 2023-10-05T19:28:32.455 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.450+0000 7fd9358ce700 1 -- 172.21.15.167:0/3309876569 <== osd.3 v2:172.21.15.167:6808/30036 23 ==== osd_op_reply(89 notify.2 [watch watch cookie 93857119470032] v30'109 uv108 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7fd92002d070 con 0x555cd04396f0 2023-10-05T19:28:32.455 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.450+0000 7fd937d99e40 1 -- 172.21.15.167:0/3309876569 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:92 7.d 7:b4812045:::notify.4:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x555cd0442c60 con 0x555cd0401670 2023-10-05T19:28:32.457 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.450+0000 7fd9350cd700 1 -- 172.21.15.167:0/3309876569 <== osd.2 v2:172.21.15.167:6816/30037 16 ==== osd_op_reply(91 notify.3 [watch watch cookie 93857119474448] v30'109 uv108 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7fd93002f040 con 0x7fd920007030 2023-10-05T19:28:32.458 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.454+0000 7fd9360cf700 1 -- 172.21.15.167:0/3309876569 <== osd.1 v2:172.21.15.167:6800/30035 31 ==== osd_op_reply(92 notify.4 [create] v30'108 uv108 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7fd928050040 con 0x555cd0401670 2023-10-05T19:28:32.458 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.454+0000 7fd937d99e40 1 -- 172.21.15.167:0/3309876569 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:93 7.d 7:b4812045:::notify.4:head [watch watch cookie 93857119476656] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x555cd0442730 con 0x555cd0401670 2023-10-05T19:28:32.458 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.454+0000 7fd937d99e40 1 -- 172.21.15.167:0/3309876569 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:94 7.3 7:c609908c:::notify.5:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x555cd0443550 con 0x7fd930011040 2023-10-05T19:28:32.461 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.454+0000 7fd9360cf700 1 -- 172.21.15.167:0/3309876569 <== osd.1 v2:172.21.15.167:6800/30035 32 ==== osd_op_reply(93 notify.4 [watch watch cookie 93857119476656] v30'109 uv108 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7fd928050040 con 0x555cd0401670 2023-10-05T19:28:32.461 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.454+0000 7fd9360cf700 1 -- 172.21.15.167:0/3309876569 <== osd.0 v2:172.21.15.118:6800/27690 23 ==== osd_op_reply(94 notify.5 [create] v30'108 uv108 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7fd928050040 con 0x7fd930011040 2023-10-05T19:28:32.461 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.454+0000 7fd937d99e40 1 -- 172.21.15.167:0/3309876569 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:95 7.3 7:c609908c:::notify.5:head [watch watch cookie 93857119473184] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x555cd0443500 con 0x7fd930011040 2023-10-05T19:28:32.461 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.454+0000 7fd937d99e40 1 -- 172.21.15.167:0/3309876569 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:96 7.14 7:2b04a3e9:::notify.6:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x555cd0443f10 con 0x7fd930011040 2023-10-05T19:28:32.464 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.458+0000 7fd9360cf700 1 -- 172.21.15.167:0/3309876569 <== osd.0 v2:172.21.15.118:6800/27690 24 ==== osd_op_reply(96 notify.6 [create] v30'108 uv108 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7fd928050040 con 0x7fd930011040 2023-10-05T19:28:32.464 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.458+0000 7fd937d99e40 1 -- 172.21.15.167:0/3309876569 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:97 7.14 7:2b04a3e9:::notify.6:head [watch watch cookie 93857119484448] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x555cd0444360 con 0x7fd930011040 2023-10-05T19:28:32.464 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.458+0000 7fd937d99e40 1 -- 172.21.15.167:0/3309876569 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:98 7.9 7:93e5b521:::notify.7:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x555cd0444d70 con 0x7fd930011040 2023-10-05T19:28:32.464 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.458+0000 7fd9360cf700 1 -- 172.21.15.167:0/3309876569 <== osd.0 v2:172.21.15.118:6800/27690 25 ==== osd_op_reply(95 notify.5 [watch watch cookie 93857119473184] v30'109 uv108 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7fd928050040 con 0x7fd930011040 2023-10-05T19:28:32.466 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.462+0000 7fd9360cf700 1 -- 172.21.15.167:0/3309876569 <== osd.0 v2:172.21.15.118:6800/27690 26 ==== osd_op_reply(98 notify.7 [create] v30'108 uv108 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7fd928050040 con 0x7fd930011040 2023-10-05T19:28:32.466 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.462+0000 7fd9360cf700 1 -- 172.21.15.167:0/3309876569 <== osd.0 v2:172.21.15.118:6800/27690 27 ==== osd_op_reply(97 notify.6 [watch watch cookie 93857119484448] v30'109 uv108 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7fd928050040 con 0x7fd930011040 2023-10-05T19:28:32.466 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.462+0000 7fd937d99e40 1 -- 172.21.15.167:0/3309876569 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:99 7.9 7:93e5b521:::notify.7:head [watch watch cookie 93857119481872] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x555cd0444880 con 0x7fd930011040 2023-10-05T19:28:32.467 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.462+0000 7fd937d99e40 20 add_watcher() i=0 2023-10-05T19:28:32.467 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.462+0000 7fd937d99e40 20 add_watcher() i=1 2023-10-05T19:28:32.467 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.462+0000 7fd937d99e40 20 add_watcher() i=2 2023-10-05T19:28:32.467 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.462+0000 7fd937d99e40 20 add_watcher() i=3 2023-10-05T19:28:32.467 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.462+0000 7fd937d99e40 20 add_watcher() i=4 2023-10-05T19:28:32.467 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.462+0000 7fd937d99e40 20 add_watcher() i=5 2023-10-05T19:28:32.467 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.462+0000 7fd937d99e40 20 add_watcher() i=6 2023-10-05T19:28:32.468 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.462+0000 7fd9360cf700 1 -- 172.21.15.167:0/3309876569 <== osd.0 v2:172.21.15.118:6800/27690 28 ==== osd_op_reply(99 notify.7 [watch watch cookie 93857119481872] v30'109 uv108 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7fd928050040 con 0x7fd930011040 2023-10-05T19:28:32.469 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.462+0000 7fd937d99e40 20 add_watcher() i=7 2023-10-05T19:28:32.469 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.462+0000 7fd937d99e40 2 all 8 watchers are set, enabling cache 2023-10-05T19:28:32.469 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.462+0000 7fd937d99e40 1 -- 172.21.15.167:0/3309876569 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:100 6.4 6:22d26bf9:::data_loggenerations_metadata:head [call version.check_conds in=74b,call version.read in=11b,read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x555cd0447da0 con 0x555cd0401670 2023-10-05T19:28:32.469 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.462+0000 7fd9360cf700 1 -- 172.21.15.167:0/3309876569 <== osd.1 v2:172.21.15.167:6800/30035 33 ==== osd_op_reply(100 data_loggenerations_metadata [call,call out=48b,read 0~28 out=28b] v0'0 uv1 ondisk = 0) v8 ==== 256+0+76 (crc 0 0 0) 0x7fd928050040 con 0x555cd0401670 2023-10-05T19:28:32.469 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.462+0000 7fd937d99e40 1 -- 172.21.15.167:0/3309876569 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:101 6.4 6:22d26bf9:::data_loggenerations_metadata:head [watch watch cookie 93857119488128] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x555cd04451c0 con 0x555cd0401670 2023-10-05T19:28:32.471 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.466+0000 7fd9360cf700 1 -- 172.21.15.167:0/3309876569 <== osd.1 v2:172.21.15.167:6800/30035 34 ==== osd_op_reply(101 data_loggenerations_metadata [watch watch cookie 93857119488128] v30'107 uv1 ondisk = 0) v8 ==== 172+0+0 (crc 0 0 0) 0x7fd928050040 con 0x555cd0401670 2023-10-05T19:28:32.472 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.466+0000 7fd906fe5700 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2023-10-05T19:28:32.472 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.466+0000 7fd937d99e40 20 check_secure_mon_conn(): auth registy supported: methods=[2] modes=[2,1] 2023-10-05T19:28:32.472 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.466+0000 7fd937d99e40 20 check_secure_mon_conn(): mode 1 is insecure 2023-10-05T19:28:32.472 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.466+0000 7fd937d99e40 1 Processor -- start 2023-10-05T19:28:32.473 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.466+0000 7fd937d99e40 1 -- start start 2023-10-05T19:28:32.473 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.466+0000 7fd937d99e40 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x555cd0452280 0x555cd0452650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-05T19:28:32.473 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.466+0000 7fd937d99e40 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x555cd0452b90 con 0x555cd0452280 2023-10-05T19:28:32.474 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.466+0000 7fd9350cd700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x555cd0452280 0x555cd0452650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-05T19:28:32.474 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.466+0000 7fd9350cd700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x555cd0452280 0x555cd0452650 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.118:3300/0 says I am v2:172.21.15.167:58414/0 (socket says 172.21.15.167:58414) 2023-10-05T19:28:32.474 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.466+0000 7fd9350cd700 1 -- 172.21.15.167:0/71677823 learned_addr learned my addr 172.21.15.167:0/71677823 (peer_addr_for_me v2:172.21.15.167:0/0) 2023-10-05T19:28:32.474 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.466+0000 7fd9350cd700 1 -- 172.21.15.167:0/71677823 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x555cd0452cd0 con 0x555cd0452280 2023-10-05T19:28:32.475 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.470+0000 7fd9350cd700 1 --2- 172.21.15.167:0/71677823 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x555cd0452280 0x555cd0452650 secure :-1 s=READY pgs=354 cs=0 l=1 rev1=1 crypto rx=0x7fd930001d40 tx=0x7fd930006000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6eab81b540f4d715 server_cookie=0 in_seq=0 out_seq=0 2023-10-05T19:28:32.475 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.470+0000 7fd9067e4700 1 -- 172.21.15.167:0/71677823 <== mon.0 v2:172.21.15.118:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fd93002c040 con 0x555cd0452280 2023-10-05T19:28:32.475 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.470+0000 7fd9067e4700 1 -- 172.21.15.167:0/71677823 <== mon.0 v2:172.21.15.118:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd93002d040 con 0x555cd0452280 2023-10-05T19:28:32.475 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.470+0000 7fd9067e4700 1 -- 172.21.15.167:0/71677823 <== mon.0 v2:172.21.15.118:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fd930028b40 con 0x555cd0452280 2023-10-05T19:28:32.476 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.470+0000 7fd937d99e40 1 -- 172.21.15.167:0/71677823 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x555cd0452280 msgr2=0x555cd0452650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-05T19:28:32.476 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.470+0000 7fd937d99e40 1 --2- 172.21.15.167:0/71677823 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x555cd0452280 0x555cd0452650 secure :-1 s=READY pgs=354 cs=0 l=1 rev1=1 crypto rx=0x7fd930001d40 tx=0x7fd930006000 comp rx=0 tx=0).stop 2023-10-05T19:28:32.476 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.470+0000 7fd937d99e40 1 -- 172.21.15.167:0/71677823 shutdown_connections 2023-10-05T19:28:32.476 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.470+0000 7fd937d99e40 1 --2- 172.21.15.167:0/71677823 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x555cd0452280 0x555cd0452650 unknown :-1 s=CLOSED pgs=354 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-05T19:28:32.476 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.470+0000 7fd937d99e40 1 -- 172.21.15.167:0/71677823 >> 172.21.15.167:0/71677823 conn(0x555cd044d640 msgr2=0x555cd044b790 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-05T19:28:32.476 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.470+0000 7fd937d99e40 1 -- 172.21.15.167:0/71677823 shutdown_connections 2023-10-05T19:28:32.476 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.470+0000 7fd937d99e40 1 -- 172.21.15.167:0/71677823 wait complete. 2023-10-05T19:28:32.477 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.470+0000 7fd937d99e40 1 Processor -- start 2023-10-05T19:28:32.477 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.470+0000 7fd937d99e40 1 -- start start 2023-10-05T19:28:32.477 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.470+0000 7fd937d99e40 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x555cd0452280 0x555cd0457e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-05T19:28:32.477 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.470+0000 7fd937d99e40 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x555cd0458e70 con 0x555cd0452280 2023-10-05T19:28:32.477 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.470+0000 7fd9350cd700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x555cd0452280 0x555cd0457e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-05T19:28:32.478 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.470+0000 7fd9350cd700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x555cd0452280 0x555cd0457e70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.118:3300/0 says I am v2:172.21.15.167:58426/0 (socket says 172.21.15.167:58426) 2023-10-05T19:28:32.478 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.470+0000 7fd9350cd700 1 -- 172.21.15.167:0/4265118016 learned_addr learned my addr 172.21.15.167:0/4265118016 (peer_addr_for_me v2:172.21.15.167:0/0) 2023-10-05T19:28:32.478 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.470+0000 7fd9350cd700 1 -- 172.21.15.167:0/4265118016 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x555cd034ebd0 con 0x555cd0452280 2023-10-05T19:28:32.478 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.470+0000 7fd9350cd700 1 --2- 172.21.15.167:0/4265118016 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x555cd0452280 0x555cd0457e70 secure :-1 s=READY pgs=355 cs=0 l=1 rev1=1 crypto rx=0x7fd930006040 tx=0x7fd93003c000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-05T19:28:32.479 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.474+0000 7fd904fe1700 1 -- 172.21.15.167:0/4265118016 <== mon.0 v2:172.21.15.118:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fd93002c070 con 0x555cd0452280 2023-10-05T19:28:32.479 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.474+0000 7fd904fe1700 1 -- 172.21.15.167:0/4265118016 <== mon.0 v2:172.21.15.118:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd93001c040 con 0x555cd0452280 2023-10-05T19:28:32.479 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.474+0000 7fd937d99e40 1 -- 172.21.15.167:0/4265118016 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x555cd0458470 con 0x555cd0452280 2023-10-05T19:28:32.479 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.474+0000 7fd904fe1700 1 -- 172.21.15.167:0/4265118016 <== mon.0 v2:172.21.15.118:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fd930028ec0 con 0x555cd0452280 2023-10-05T19:28:32.479 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.474+0000 7fd937d99e40 1 -- 172.21.15.167:0/4265118016 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x555cd04588b0 con 0x555cd0452280 2023-10-05T19:28:32.479 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.474+0000 7fd904fe1700 1 -- 172.21.15.167:0/4265118016 <== mon.0 v2:172.21.15.118:3300/0 4 ==== mgrmap(e 5) v1 ==== 40329+0+0 (secure 0 0 0) 0x7fd930045040 con 0x555cd0452280 2023-10-05T19:28:32.480 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.474+0000 7fd904fe1700 1 --2- 172.21.15.167:0/4265118016 >> [v2:172.21.15.167:6824/29889,v1:172.21.15.167:6825/29889] conn(0x7fd87c030c00 0x7fd87c0330b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-05T19:28:32.480 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.474+0000 7fd904fe1700 1 -- 172.21.15.167:0/4265118016 <== mon.0 v2:172.21.15.118:3300/0 5 ==== osd_map(30..30 src has 1..30) v4 ==== 6057+0+0 (secure 0 0 0) 0x7fd930051c20 con 0x555cd0452280 2023-10-05T19:28:32.480 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.474+0000 7fd9360cf700 1 --2- 172.21.15.167:0/4265118016 >> [v2:172.21.15.167:6824/29889,v1:172.21.15.167:6825/29889] conn(0x7fd87c030c00 0x7fd87c0330b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-05T19:28:32.480 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.474+0000 7fd937d99e40 5 note: GC not initialized 2023-10-05T19:28:32.481 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.474+0000 7fd9360cf700 1 --2- 172.21.15.167:0/4265118016 >> [v2:172.21.15.167:6824/29889,v1:172.21.15.167:6825/29889] conn(0x7fd87c030c00 0x7fd87c0330b0 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x555cd0444140 tx=0x7fd928000f90 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-05T19:28:32.483 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.478+0000 7fd875ffb700 20 reqs_thread_entry: start 2023-10-05T19:28:32.483 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.478+0000 7fd8677fe700 20 reqs_thread_entry: start 2023-10-05T19:28:32.483 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.478+0000 7fd937d99e40 20 init_complete bucket index max shards: 11 2023-10-05T19:28:32.484 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.478+0000 7fd937d99e40 10 rgw notify: Started notification manager with: 1 workers 2023-10-05T19:28:32.484 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.478+0000 7fd865ffb700 20 rgw notify: INFO: next queues processing will happen at: Thu Oct 5 19:29:02 2023 2023-10-05T19:28:32.484 INFO:teuthology.orchestra.run.smithi167.stderr: 2023-10-05T19:28:32.484 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.478+0000 7fd8657fa700 20 reqs_thread_entry: start 2023-10-05T19:28:32.484 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.478+0000 7fd937d99e40 20 get_system_obj_state: rctx=0x7ffd252369a8 obj=default.rgw.meta:users.uid:fud state=0x555cd02db0b0 s->prefetch_data=0 2023-10-05T19:28:32.484 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.478+0000 7fd937d99e40 10 cache get: name=default.rgw.meta+users.uid+fud : miss 2023-10-05T19:28:32.484 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.478+0000 7fd937d99e40 1 -- 172.21.15.167:0/3309876569 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:102 8.e 8:75f48285:users.uid::fud:head [call version.read in=11b,getxattrs,stat] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x555cd04c06e0 con 0x7fd930011040 2023-10-05T19:28:32.485 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.478+0000 7fd9360cf700 1 -- 172.21.15.167:0/3309876569 <== osd.0 v2:172.21.15.118:6800/27690 29 ==== osd_op_reply(102 fud [call,getxattrs,stat] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 ==== 231+0+0 (crc 0 0 0) 0x7fd928050040 con 0x7fd930011040 2023-10-05T19:28:32.485 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.478+0000 7fd937d99e40 10 cache put: name=default.rgw.meta+users.uid+fud info.flags=0x0 2023-10-05T19:28:32.485 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.478+0000 7fd937d99e40 10 adding default.rgw.meta+users.uid+fud to cache LRU end 2023-10-05T19:28:32.485 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.478+0000 7fd937d99e40 20 get_system_obj_state: rctx=0x7ffd25236698 obj=default.rgw.meta:users.keys:p5YnriCv1nAtykxBrupQ state=0x555cd02db0b0 s->prefetch_data=0 2023-10-05T19:28:32.485 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.478+0000 7fd937d99e40 10 cache get: name=default.rgw.meta+users.keys+p5YnriCv1nAtykxBrupQ : miss 2023-10-05T19:28:32.485 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.478+0000 7fd937d99e40 1 -- 172.21.15.167:0/3309876569 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:103 8.13 8:cc695d75:users.keys::p5YnriCv1nAtykxBrupQ:head [getxattrs,stat] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x555cd04c06e0 con 0x7fd930011040 2023-10-05T19:28:32.486 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.478+0000 7fd9360cf700 1 -- 172.21.15.167:0/3309876569 <== osd.0 v2:172.21.15.118:6800/27690 30 ==== osd_op_reply(103 p5YnriCv1nAtykxBrupQ [getxattrs,stat] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 ==== 206+0+0 (crc 0 0 0) 0x7fd928050040 con 0x7fd930011040 2023-10-05T19:28:32.486 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.478+0000 7fd937d99e40 10 cache put: name=default.rgw.meta+users.keys+p5YnriCv1nAtykxBrupQ info.flags=0x0 2023-10-05T19:28:32.486 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.478+0000 7fd937d99e40 10 adding default.rgw.meta+users.keys+p5YnriCv1nAtykxBrupQ to cache LRU end 2023-10-05T19:28:32.486 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.478+0000 7fd937d99e40 20 get_system_obj_state: rctx=0x7ffd25235f38 obj=default.rgw.meta:users.keys:p5YnriCv1nAtykxBrupQ state=0x555cd02db0b0 s->prefetch_data=0 2023-10-05T19:28:32.487 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.478+0000 7fd937d99e40 10 cache get: name=default.rgw.meta+users.keys+p5YnriCv1nAtykxBrupQ : hit (negative entry) 2023-10-05T19:28:32.487 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.478+0000 7fd937d99e40 20 get_system_obj_state: rctx=0x7ffd252365d8 obj=default.rgw.meta:users.keys:p5YnriCv1nAtykxBrupQ state=0x555cd02db0b0 s->prefetch_data=0 2023-10-05T19:28:32.487 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.478+0000 7fd937d99e40 10 cache get: name=default.rgw.meta+users.keys+p5YnriCv1nAtykxBrupQ : hit (negative entry) 2023-10-05T19:28:32.487 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.478+0000 7fd937d99e40 1 -- 172.21.15.167:0/3309876569 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:104 8.e 8:75f48285:users.uid::fud:head [delete,create,call version.set in=58b,writefull 0~346 in=346b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x555cd04c2c80 con 0x7fd930011040 2023-10-05T19:28:32.489 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.482+0000 7fd9360cf700 1 -- 172.21.15.167:0/3309876569 <== osd.0 v2:172.21.15.118:6800/27690 31 ==== osd_op_reply(104 fud [delete,create,call,writefull 0~346] v30'1 uv1 ondisk = 0) v8 ==== 273+0+0 (crc 0 0 0) 0x7fd928050040 con 0x7fd930011040 2023-10-05T19:28:32.489 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.482+0000 7fd937d99e40 10 cache put: name=default.rgw.meta+users.uid+fud info.flags=0x17 2023-10-05T19:28:32.489 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.482+0000 7fd937d99e40 10 moving default.rgw.meta+users.uid+fud to cache LRU end 2023-10-05T19:28:32.490 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.482+0000 7fd937d99e40 10 distributing notification oid=default.rgw.control:notify.5 cni=[op: 0, obj: default.rgw.meta:users.uid:fud, ofs0, ns] 2023-10-05T19:28:32.490 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.482+0000 7fd937d99e40 1 -- 172.21.15.167:0/3309876569 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:105 7.3 7:c609908c:::notify.5:head [notify cookie 93857120001136 in=534b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x555cd04c2650 con 0x7fd930011040 2023-10-05T19:28:32.490 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.486+0000 7fd9360cf700 1 -- 172.21.15.167:0/3309876569 <== osd.0 v2:172.21.15.118:6800/27690 32 ==== watch-notify(notify (1) cookie 93857119473184 notify 128849018883 ret 0) v3 ==== 564+0+0 (crc 0 0 0) 0x7fd92803d040 con 0x7fd930011040 2023-10-05T19:28:32.490 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.486+0000 7fd9360cf700 1 -- 172.21.15.167:0/3309876569 <== osd.0 v2:172.21.15.118:6800/27690 33 ==== osd_op_reply(105 notify.5 [notify cookie 93857120001136 out=8b] v0'0 uv108 ondisk = 0) v8 ==== 152+0+8 (crc 0 0 0) 0x7fd928050040 con 0x7fd930011040 2023-10-05T19:28:32.491 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.486+0000 7fd92f7fe700 10 rgw watcher librados: RGWWatcher::handle_notify() notify_id 128849018883 cookie 93857119473184 notifier 4443 bl.length()=522 2023-10-05T19:28:32.491 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.486+0000 7fd92f7fe700 10 rgw watcher librados: cache put: name=default.rgw.meta+users.uid+fud info.flags=0x17 2023-10-05T19:28:32.491 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.486+0000 7fd92f7fe700 10 rgw watcher librados: moving default.rgw.meta+users.uid+fud to cache LRU end 2023-10-05T19:28:32.491 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.486+0000 7fd92f7fe700 1 -- 172.21.15.167:0/3309876569 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:106 7.3 7:c609908c:::notify.5:head [notify-ack in=20b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x7fd914003d90 con 0x7fd930011040 2023-10-05T19:28:32.491 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.486+0000 7fd9360cf700 1 -- 172.21.15.167:0/3309876569 <== osd.0 v2:172.21.15.118:6800/27690 34 ==== osd_op_reply(106 notify.5 [notify-ack] v0'0 uv108 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7fd928050040 con 0x7fd930011040 2023-10-05T19:28:32.492 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.486+0000 7fd9360cf700 1 -- 172.21.15.167:0/3309876569 <== osd.0 v2:172.21.15.118:6800/27690 35 ==== watch-notify(notify_complete (2) cookie 93857120001136 notify 128849018883 ret 0) v3 ==== 42+0+48 (crc 0 0 0) 0x7fd928002c90 con 0x7fd930011040 2023-10-05T19:28:32.492 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.486+0000 7fd937d99e40 1 -- 172.21.15.167:0/3309876569 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:107 8.13 8:cc695d75:users.keys::p5YnriCv1nAtykxBrupQ:head [delete,create,writefull 0~7 in=7b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x555cd04c0c60 con 0x7fd930011040 2023-10-05T19:28:32.495 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.490+0000 7fd9360cf700 1 -- 172.21.15.167:0/3309876569 <== osd.0 v2:172.21.15.118:6800/27690 36 ==== osd_op_reply(107 p5YnriCv1nAtykxBrupQ [delete,create,writefull 0~7] v30'6 uv6 ondisk = 0) v8 ==== 248+0+0 (crc 0 0 0) 0x7fd928050040 con 0x7fd930011040 2023-10-05T19:28:32.495 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.490+0000 7fd937d99e40 10 cache put: name=default.rgw.meta+users.keys+p5YnriCv1nAtykxBrupQ info.flags=0x7 2023-10-05T19:28:32.495 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.490+0000 7fd937d99e40 10 moving default.rgw.meta+users.keys+p5YnriCv1nAtykxBrupQ to cache LRU end 2023-10-05T19:28:32.496 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.490+0000 7fd937d99e40 10 distributing notification oid=default.rgw.control:notify.2 cni=[op: 0, obj: default.rgw.meta:users.keys:p5YnriCv1nAtykxBrupQ, ofs0, ns] 2023-10-05T19:28:32.496 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.490+0000 7fd937d99e40 1 -- 172.21.15.167:0/3309876569 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:108 7.15 7:a93a5511:::notify.2:head [notify cookie 93857120004752 in=189b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x555cd04c0c60 con 0x555cd04396f0 2023-10-05T19:28:32.496 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.490+0000 7fd9358ce700 1 -- 172.21.15.167:0/3309876569 <== osd.3 v2:172.21.15.167:6808/30036 24 ==== watch-notify(notify (1) cookie 93857119470032 notify 128849018891 ret 0) v3 ==== 219+0+0 (crc 0 0 0) 0x555cd042a810 con 0x555cd04396f0 2023-10-05T19:28:32.496 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.490+0000 7fd92f7fe700 10 rgw watcher librados: RGWWatcher::handle_notify() notify_id 128849018891 cookie 93857119470032 notifier 4443 bl.length()=177 2023-10-05T19:28:32.496 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.490+0000 7fd92f7fe700 10 rgw watcher librados: cache put: name=default.rgw.meta+users.keys+p5YnriCv1nAtykxBrupQ info.flags=0x7 2023-10-05T19:28:32.497 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.490+0000 7fd92f7fe700 10 rgw watcher librados: moving default.rgw.meta+users.keys+p5YnriCv1nAtykxBrupQ to cache LRU end 2023-10-05T19:28:32.497 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.490+0000 7fd9358ce700 1 -- 172.21.15.167:0/3309876569 <== osd.3 v2:172.21.15.167:6808/30036 25 ==== osd_op_reply(108 notify.2 [notify cookie 93857120004752 out=8b] v0'0 uv108 ondisk = 0) v8 ==== 152+0+8 (crc 0 0 0) 0x7fd92002d070 con 0x555cd04396f0 2023-10-05T19:28:32.497 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.490+0000 7fd92f7fe700 1 -- 172.21.15.167:0/3309876569 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:109 7.15 7:a93a5511:::notify.2:head [notify-ack in=20b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x7fd914003d90 con 0x555cd04396f0 2023-10-05T19:28:32.497 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.490+0000 7fd9358ce700 1 -- 172.21.15.167:0/3309876569 <== osd.3 v2:172.21.15.167:6808/30036 26 ==== watch-notify(notify_complete (2) cookie 93857120004752 notify 128849018891 ret 0) v3 ==== 42+0+48 (crc 0 0 0) 0x555cd042a370 con 0x555cd04396f0 2023-10-05T19:28:32.497 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.490+0000 7fd9358ce700 1 -- 172.21.15.167:0/3309876569 <== osd.3 v2:172.21.15.167:6808/30036 27 ==== osd_op_reply(109 notify.2 [notify-ack] v0'0 uv108 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7fd92002d070 con 0x555cd04396f0 2023-10-05T19:28:32.498 INFO:teuthology.orchestra.run.smithi167.stdout:{ 2023-10-05T19:28:32.498 INFO:teuthology.orchestra.run.smithi167.stdout: "user_id": "fud", 2023-10-05T19:28:32.498 INFO:teuthology.orchestra.run.smithi167.stdout: "display_name": "Fud", 2023-10-05T19:28:32.498 INFO:teuthology.orchestra.run.smithi167.stdout: "email": "", 2023-10-05T19:28:32.498 INFO:teuthology.orchestra.run.smithi167.stdout: "suspended": 0, 2023-10-05T19:28:32.498 INFO:teuthology.orchestra.run.smithi167.stdout: "max_buckets": 1, 2023-10-05T19:28:32.498 INFO:teuthology.orchestra.run.smithi167.stdout: "subusers": [], 2023-10-05T19:28:32.498 INFO:teuthology.orchestra.run.smithi167.stdout: "keys": [ 2023-10-05T19:28:32.498 INFO:teuthology.orchestra.run.smithi167.stdout: { 2023-10-05T19:28:32.498 INFO:teuthology.orchestra.run.smithi167.stdout: "user": "fud", 2023-10-05T19:28:32.499 INFO:teuthology.orchestra.run.smithi167.stdout: "access_key": "p5YnriCv1nAtykxBrupQ", 2023-10-05T19:28:32.499 INFO:teuthology.orchestra.run.smithi167.stdout: "secret_key": "Q8Tk6Q/27hfbFSYdSkPtUqhqx1GgzvpXa4WARozh" 2023-10-05T19:28:32.499 INFO:teuthology.orchestra.run.smithi167.stdout: } 2023-10-05T19:28:32.499 INFO:teuthology.orchestra.run.smithi167.stdout: ], 2023-10-05T19:28:32.499 INFO:teuthology.orchestra.run.smithi167.stdout: "swift_keys": [], 2023-10-05T19:28:32.499 INFO:teuthology.orchestra.run.smithi167.stdout: "caps": [], 2023-10-05T19:28:32.499 INFO:teuthology.orchestra.run.smithi167.stdout: "op_mask": "read, write, delete", 2023-10-05T19:28:32.499 INFO:teuthology.orchestra.run.smithi167.stdout: "default_placement": "", 2023-10-05T19:28:32.499 INFO:teuthology.orchestra.run.smithi167.stdout: "default_storage_class": "", 2023-10-05T19:28:32.499 INFO:teuthology.orchestra.run.smithi167.stdout: "placement_tags": [], 2023-10-05T19:28:32.499 INFO:teuthology.orchestra.run.smithi167.stdout: "bucket_quota": { 2023-10-05T19:28:32.500 INFO:teuthology.orchestra.run.smithi167.stdout: "enabled": false, 2023-10-05T19:28:32.500 INFO:teuthology.orchestra.run.smithi167.stdout: "check_on_raw": false, 2023-10-05T19:28:32.500 INFO:teuthology.orchestra.run.smithi167.stdout: "max_size": -1, 2023-10-05T19:28:32.500 INFO:teuthology.orchestra.run.smithi167.stdout: "max_size_kb": 0, 2023-10-05T19:28:32.500 INFO:teuthology.orchestra.run.smithi167.stdout: "max_objects": -1 2023-10-05T19:28:32.500 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2023-10-05T19:28:32.500 INFO:teuthology.orchestra.run.smithi167.stdout: "user_quota": { 2023-10-05T19:28:32.500 INFO:teuthology.orchestra.run.smithi167.stdout: "enabled": false, 2023-10-05T19:28:32.500 INFO:teuthology.orchestra.run.smithi167.stdout: "check_on_raw": false, 2023-10-05T19:28:32.500 INFO:teuthology.orchestra.run.smithi167.stdout: "max_size": -1, 2023-10-05T19:28:32.500 INFO:teuthology.orchestra.run.smithi167.stdout: "max_size_kb": 0, 2023-10-05T19:28:32.500 INFO:teuthology.orchestra.run.smithi167.stdout: "max_objects": -1 2023-10-05T19:28:32.500 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2023-10-05T19:28:32.501 INFO:teuthology.orchestra.run.smithi167.stdout: "temp_url_keys": [], 2023-10-05T19:28:32.501 INFO:teuthology.orchestra.run.smithi167.stdout: "type": "rgw", 2023-10-05T19:28:32.501 INFO:teuthology.orchestra.run.smithi167.stdout: "mfa_ids": [] 2023-10-05T19:28:32.501 INFO:teuthology.orchestra.run.smithi167.stdout:} 2023-10-05T19:28:32.501 INFO:teuthology.orchestra.run.smithi167.stdout: 2023-10-05T19:28:32.501 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.490+0000 7fd937d99e40 1 -- 172.21.15.167:0/3309876569 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:110 7.1f 7:f95f44c2:::notify.0:head [watch unwatch cookie 93857119463184] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x555cd0416760 con 0x7fd920007030 2023-10-05T19:28:32.502 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.494+0000 7fd9350cd700 1 -- 172.21.15.167:0/3309876569 <== osd.2 v2:172.21.15.167:6816/30037 17 ==== osd_op_reply(110 notify.0 [watch unwatch cookie 93857119463184] v30'110 uv108 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7fd93002f040 con 0x7fd920007030 2023-10-05T19:28:32.502 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.494+0000 7fd937d99e40 20 remove_watcher() i=0 2023-10-05T19:28:32.502 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.494+0000 7fd937d99e40 2 removed watcher, disabling cache 2023-10-05T19:28:32.502 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.494+0000 7fd937d99e40 1 -- 172.21.15.167:0/3309876569 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:111 7.0 7:05bf5b68:::notify.1:head [watch unwatch cookie 93857119466416] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x555cd043e710 con 0x555cd0401670 2023-10-05T19:28:32.503 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.498+0000 7fd9360cf700 1 -- 172.21.15.167:0/3309876569 <== osd.1 v2:172.21.15.167:6800/30035 35 ==== osd_op_reply(111 notify.1 [watch unwatch cookie 93857119466416] v30'110 uv108 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7fd928050040 con 0x555cd0401670 2023-10-05T19:28:32.504 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.498+0000 7fd937d99e40 20 remove_watcher() i=1 2023-10-05T19:28:32.504 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.498+0000 7fd937d99e40 1 -- 172.21.15.167:0/3309876569 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:112 7.15 7:a93a5511:::notify.2:head [watch unwatch cookie 93857119470032] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x555cd043e710 con 0x555cd04396f0 2023-10-05T19:28:32.507 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.502+0000 7fd9358ce700 1 -- 172.21.15.167:0/3309876569 <== osd.3 v2:172.21.15.167:6808/30036 28 ==== osd_op_reply(112 notify.2 [watch unwatch cookie 93857119470032] v30'110 uv108 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7fd92002d070 con 0x555cd04396f0 2023-10-05T19:28:32.507 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.502+0000 7fd937d99e40 20 remove_watcher() i=2 2023-10-05T19:28:32.507 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.502+0000 7fd937d99e40 1 -- 172.21.15.167:0/3309876569 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:113 7.e 7:7759931f:::notify.3:head [watch unwatch cookie 93857119474448] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x555cd04401d0 con 0x7fd920007030 2023-10-05T19:28:32.510 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.502+0000 7fd9350cd700 1 -- 172.21.15.167:0/3309876569 <== osd.2 v2:172.21.15.167:6816/30037 18 ==== osd_op_reply(113 notify.3 [watch unwatch cookie 93857119474448] v30'110 uv108 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7fd93002f040 con 0x7fd920007030 2023-10-05T19:28:32.510 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.502+0000 7fd937d99e40 20 remove_watcher() i=3 2023-10-05T19:28:32.510 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.502+0000 7fd937d99e40 1 -- 172.21.15.167:0/3309876569 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:114 7.d 7:b4812045:::notify.4:head [watch unwatch cookie 93857119476656] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x555cd04401d0 con 0x555cd0401670 2023-10-05T19:28:32.512 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.506+0000 7fd9360cf700 1 -- 172.21.15.167:0/3309876569 <== osd.1 v2:172.21.15.167:6800/30035 36 ==== osd_op_reply(114 notify.4 [watch unwatch cookie 93857119476656] v30'110 uv108 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7fd928050040 con 0x555cd0401670 2023-10-05T19:28:32.512 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.506+0000 7fd937d99e40 20 remove_watcher() i=4 2023-10-05T19:28:32.512 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.506+0000 7fd937d99e40 1 -- 172.21.15.167:0/3309876569 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:115 7.3 7:c609908c:::notify.5:head [watch unwatch cookie 93857119473184] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x555cd0441bb0 con 0x7fd930011040 2023-10-05T19:28:32.515 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.510+0000 7fd9360cf700 1 -- 172.21.15.167:0/3309876569 <== osd.0 v2:172.21.15.118:6800/27690 37 ==== osd_op_reply(115 notify.5 [watch unwatch cookie 93857119473184] v30'110 uv108 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7fd928050040 con 0x7fd930011040 2023-10-05T19:28:32.515 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.510+0000 7fd937d99e40 20 remove_watcher() i=5 2023-10-05T19:28:32.515 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.510+0000 7fd937d99e40 1 -- 172.21.15.167:0/3309876569 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:116 7.14 7:2b04a3e9:::notify.6:head [watch unwatch cookie 93857119484448] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x555cd0440e20 con 0x7fd930011040 2023-10-05T19:28:32.518 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.514+0000 7fd9360cf700 1 -- 172.21.15.167:0/3309876569 <== osd.0 v2:172.21.15.118:6800/27690 38 ==== osd_op_reply(116 notify.6 [watch unwatch cookie 93857119484448] v30'110 uv108 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7fd928050040 con 0x7fd930011040 2023-10-05T19:28:32.518 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.514+0000 7fd937d99e40 20 remove_watcher() i=6 2023-10-05T19:28:32.518 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.514+0000 7fd937d99e40 1 -- 172.21.15.167:0/3309876569 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:117 7.9 7:93e5b521:::notify.7:head [watch unwatch cookie 93857119481872] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x555cd0441bb0 con 0x7fd930011040 2023-10-05T19:28:32.522 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.514+0000 7fd9360cf700 1 -- 172.21.15.167:0/3309876569 <== osd.0 v2:172.21.15.118:6800/27690 39 ==== osd_op_reply(117 notify.7 [watch unwatch cookie 93857119481872] v30'110 uv108 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7fd928050040 con 0x7fd930011040 2023-10-05T19:28:32.522 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.514+0000 7fd937d99e40 20 remove_watcher() i=7 2023-10-05T19:28:32.523 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.518+0000 7fd906fe5700 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2023-10-05T19:28:32.523 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.518+0000 7fd937d99e40 1 -- 172.21.15.167:0/3309876569 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:118 6.4 6:22d26bf9:::data_loggenerations_metadata:head [watch unwatch cookie 93857119488128] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x555cd0443a20 con 0x555cd0401670 2023-10-05T19:28:32.527 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.522+0000 7fd9360cf700 1 -- 172.21.15.167:0/3309876569 <== osd.1 v2:172.21.15.167:6800/30035 37 ==== osd_op_reply(118 data_loggenerations_metadata [watch unwatch cookie 93857119488128] v30'108 uv1 ondisk = 0) v8 ==== 172+0+0 (crc 0 0 0) 0x7fd928050040 con 0x555cd0401670 2023-10-05T19:28:32.527 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.522+0000 7fd937d99e40 1 -- 172.21.15.167:0/3309876569 >> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] conn(0x7fd930011040 msgr2=0x7fd930014950 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-05T19:28:32.527 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.522+0000 7fd937d99e40 1 --2- 172.21.15.167:0/3309876569 >> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] conn(0x7fd930011040 0x7fd930014950 crc :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-05T19:28:32.527 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.522+0000 7fd937d99e40 1 -- 172.21.15.167:0/3309876569 >> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] conn(0x555cd0401670 msgr2=0x555cd0435950 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-05T19:28:32.527 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.522+0000 7fd937d99e40 1 --2- 172.21.15.167:0/3309876569 >> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] conn(0x555cd0401670 0x555cd0435950 crc :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-05T19:28:32.527 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.522+0000 7fd937d99e40 1 -- 172.21.15.167:0/3309876569 >> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] conn(0x7fd920007030 msgr2=0x7fd92002aa00 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-05T19:28:32.527 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.522+0000 7fd937d99e40 1 --2- 172.21.15.167:0/3309876569 >> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] conn(0x7fd920007030 0x7fd92002aa00 crc :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-05T19:28:32.527 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.522+0000 7fd937d99e40 1 -- 172.21.15.167:0/3309876569 >> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] conn(0x555cd04396f0 msgr2=0x555cd043bad0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-05T19:28:32.528 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.522+0000 7fd937d99e40 1 --2- 172.21.15.167:0/3309876569 >> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] conn(0x555cd04396f0 0x555cd043bad0 crc :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-05T19:28:32.528 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.522+0000 7fd937d99e40 1 -- 172.21.15.167:0/3309876569 >> [v2:172.21.15.167:6824/29889,v1:172.21.15.167:6825/29889] conn(0x7fd918030c50 msgr2=0x7fd918033100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-05T19:28:32.528 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.522+0000 7fd937d99e40 1 --2- 172.21.15.167:0/3309876569 >> [v2:172.21.15.167:6824/29889,v1:172.21.15.167:6825/29889] conn(0x7fd918030c50 0x7fd918033100 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7fd9300044f0 tx=0x7fd9300067c0 comp rx=0 tx=0).stop 2023-10-05T19:28:32.528 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.522+0000 7fd9360cf700 1 -- 172.21.15.167:0/3309876569 reap_dead start 2023-10-05T19:28:32.528 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.522+0000 7fd937d99e40 1 -- 172.21.15.167:0/3309876569 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x555cd0400d60 msgr2=0x555cd0429e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-05T19:28:32.528 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.522+0000 7fd937d99e40 1 --2- 172.21.15.167:0/3309876569 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x555cd0400d60 0x555cd0429e30 secure :-1 s=READY pgs=353 cs=0 l=1 rev1=1 crypto rx=0x7fd92001e040 tx=0x7fd920003050 comp rx=0 tx=0).stop 2023-10-05T19:28:32.529 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.522+0000 7fd937d99e40 1 -- 172.21.15.167:0/3309876569 shutdown_connections 2023-10-05T19:28:32.529 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.522+0000 7fd937d99e40 1 --2- 172.21.15.167:0/3309876569 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x555cd0400d60 0x555cd0429e30 unknown :-1 s=CLOSED pgs=353 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-05T19:28:32.529 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.522+0000 7fd937d99e40 1 -- 172.21.15.167:0/3309876569 >> 172.21.15.167:0/3309876569 conn(0x555cd03fc780 msgr2=0x555cd041a500 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-05T19:28:32.529 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.522+0000 7fd937d99e40 1 -- 172.21.15.167:0/3309876569 shutdown_connections 2023-10-05T19:28:32.529 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.522+0000 7fd937d99e40 1 -- 172.21.15.167:0/3309876569 wait complete. 2023-10-05T19:28:32.530 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.522+0000 7fd937d99e40 1 -- 172.21.15.167:0/4265118016 >> [v2:172.21.15.167:6824/29889,v1:172.21.15.167:6825/29889] conn(0x7fd87c030c00 msgr2=0x7fd87c0330b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-05T19:28:32.530 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.522+0000 7fd937d99e40 1 --2- 172.21.15.167:0/4265118016 >> [v2:172.21.15.167:6824/29889,v1:172.21.15.167:6825/29889] conn(0x7fd87c030c00 0x7fd87c0330b0 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x555cd0444140 tx=0x7fd928000f90 comp rx=0 tx=0).stop 2023-10-05T19:28:32.530 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.522+0000 7fd937d99e40 1 -- 172.21.15.167:0/4265118016 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x555cd0452280 msgr2=0x555cd0457e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-05T19:28:32.530 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.522+0000 7fd937d99e40 1 --2- 172.21.15.167:0/4265118016 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x555cd0452280 0x555cd0457e70 secure :-1 s=READY pgs=355 cs=0 l=1 rev1=1 crypto rx=0x7fd930006040 tx=0x7fd93003c000 comp rx=0 tx=0).stop 2023-10-05T19:28:32.530 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.522+0000 7fd937d99e40 1 -- 172.21.15.167:0/4265118016 shutdown_connections 2023-10-05T19:28:32.530 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.522+0000 7fd937d99e40 1 --2- 172.21.15.167:0/4265118016 >> [v2:172.21.15.167:6824/29889,v1:172.21.15.167:6825/29889] conn(0x7fd87c030c00 0x7fd87c0330b0 unknown :-1 s=CLOSED pgs=125 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-05T19:28:32.530 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.522+0000 7fd937d99e40 1 --2- 172.21.15.167:0/4265118016 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x555cd0452280 0x555cd0457e70 unknown :-1 s=CLOSED pgs=355 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-05T19:28:32.530 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.522+0000 7fd937d99e40 1 -- 172.21.15.167:0/4265118016 >> 172.21.15.167:0/4265118016 conn(0x555cd044d640 msgr2=0x555cd044b6e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-05T19:28:32.530 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.522+0000 7fd937d99e40 1 -- 172.21.15.167:0/4265118016 shutdown_connections 2023-10-05T19:28:32.530 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.522+0000 7fd937d99e40 1 -- 172.21.15.167:0/4265118016 wait complete. 2023-10-05T19:28:32.534 DEBUG:tasks.util.rgw: json result: {'user_id': 'fud', 'display_name': 'Fud', 'email': '', 'suspended': 0, 'max_buckets': 1, 'subusers': [], 'keys': [{'user': 'fud', 'access_key': 'p5YnriCv1nAtykxBrupQ', 'secret_key': 'Q8Tk6Q/27hfbFSYdSkPtUqhqx1GgzvpXa4WARozh'}], 'swift_keys': [], 'caps': [], 'op_mask': 'read, write, delete', 'default_placement': '', 'default_storage_class': '', 'placement_tags': [], 'bucket_quota': {'enabled': False, 'check_on_raw': False, 'max_size': -1, 'max_size_kb': 0, 'max_objects': -1}, 'user_quota': {'enabled': False, 'check_on_raw': False, 'max_size': -1, 'max_size_kb': 0, 'max_objects': -1}, 'temp_url_keys': [], 'type': 'rgw', 'mfa_ids': []} 2023-10-05T19:28:32.545 INFO:tasks.radosgw_admin:RL: put_obj myfoo foo bytes_out=3 bytes_in=0 failed=False 2023-10-05T19:28:32.555 INFO:tasks.radosgw_admin:RL: delete_obj myfoo foo bytes_out=0 bytes_in=0 failed=False 2023-10-05T19:28:32.555 INFO:tasks.util.rgw:rgwadmin: client.0 : ['metadata', 'get', 'bucket:myfoo'] 2023-10-05T19:28:32.555 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'ceph', 'metadata', 'get', 'bucket:myfoo'] 2023-10-05T19:28:32.555 DEBUG:teuthology.orchestra.run.smithi167:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster ceph metadata get bucket:myfoo 2023-10-05T19:28:32.590 INFO:teuthology.orchestra.run.smithi167.stderr:ignoring --setuser ceph since I am not root 2023-10-05T19:28:32.590 INFO:teuthology.orchestra.run.smithi167.stderr:ignoring --setgroup ceph since I am not root 2023-10-05T19:28:32.591 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.586+0000 7f1f21db5e40 1 Processor -- start 2023-10-05T19:28:32.591 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.586+0000 7f1f21db5e40 1 -- start start 2023-10-05T19:28:32.591 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.586+0000 7f1f21db5e40 1 --2- >> v2:172.21.15.118:3300/0 conn(0x55574cb3ae30 0x55574cb3b200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-05T19:28:32.591 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.586+0000 7f1f21db5e40 1 -- --> v1:172.21.15.118:6789/0 -- auth(proto 0 26 bytes epoch 0) v1 -- 0x55574ca87bd0 con 0x55574cb3b740 2023-10-05T19:28:32.591 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.586+0000 7f1f21db5e40 1 -- --> v2:172.21.15.118:3300/0 -- mon_getmap magic: 0 v1 -- 0x55574ca44ee0 con 0x55574cb3ae30 2023-10-05T19:28:32.592 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.586+0000 7f1f1f8ea700 1 --2- >> v2:172.21.15.118:3300/0 conn(0x55574cb3ae30 0x55574cb3b200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-05T19:28:32.592 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.586+0000 7f1f1f0e9700 1 --1- >> v1:172.21.15.118:6789/0 conn(0x55574cb3b740 0x55574cb3c1e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.118:6789/0 says I am v1:172.21.15.167:56622/0 (socket says 172.21.15.167:56622) 2023-10-05T19:28:32.592 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.586+0000 7f1f1f8ea700 1 --2- >> v2:172.21.15.118:3300/0 conn(0x55574cb3ae30 0x55574cb3b200 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.118:3300/0 says I am v2:172.21.15.167:58428/0 (socket says 172.21.15.167:58428) 2023-10-05T19:28:32.592 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.586+0000 7f1f1f0e9700 1 -- 172.21.15.167:0/4205167955 learned_addr learned my addr 172.21.15.167:0/4205167955 (peer_addr_for_me v1:172.21.15.167:0/0) 2023-10-05T19:28:32.593 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.586+0000 7f1f1f8ea700 1 -- 172.21.15.167:0/4205167955 >> v1:172.21.15.118:6789/0 conn(0x55574cb3b740 legacy=0x55574cb3c1e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-05T19:28:32.593 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.586+0000 7f1f1f8ea700 1 -- 172.21.15.167:0/4205167955 --> v2:172.21.15.118:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55574cb3c8f0 con 0x55574cb3ae30 2023-10-05T19:28:32.593 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.586+0000 7f1f1f8ea700 1 --2- 172.21.15.167:0/4205167955 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55574cb3ae30 0x55574cb3b200 secure :-1 s=READY pgs=357 cs=0 l=1 rev1=1 crypto rx=0x7f1f1000a5d0 tx=0x7f1f10005a50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=578d3947132343ed server_cookie=0 in_seq=0 out_seq=0 2023-10-05T19:28:32.594 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.586+0000 7f1f1e8e8700 1 -- 172.21.15.167:0/4205167955 <== mon.0 v2:172.21.15.118:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f1f100086a0 con 0x55574cb3ae30 2023-10-05T19:28:32.594 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.586+0000 7f1f1e8e8700 1 -- 172.21.15.167:0/4205167955 <== mon.0 v2:172.21.15.118:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1f10008800 con 0x55574cb3ae30 2023-10-05T19:28:32.594 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.586+0000 7f1f1e8e8700 1 -- 172.21.15.167:0/4205167955 <== mon.0 v2:172.21.15.118:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f1f10008ad0 con 0x55574cb3ae30 2023-10-05T19:28:32.594 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.586+0000 7f1f21db5e40 1 -- 172.21.15.167:0/4205167955 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55574cb3ae30 msgr2=0x55574cb3b200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-05T19:28:32.594 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.586+0000 7f1f21db5e40 1 --2- 172.21.15.167:0/4205167955 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55574cb3ae30 0x55574cb3b200 secure :-1 s=READY pgs=357 cs=0 l=1 rev1=1 crypto rx=0x7f1f1000a5d0 tx=0x7f1f10005a50 comp rx=0 tx=0).stop 2023-10-05T19:28:32.594 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.586+0000 7f1f21db5e40 1 -- 172.21.15.167:0/4205167955 shutdown_connections 2023-10-05T19:28:32.595 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.586+0000 7f1f21db5e40 1 --2- 172.21.15.167:0/4205167955 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55574cb3ae30 0x55574cb3b200 unknown :-1 s=CLOSED pgs=357 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-05T19:28:32.595 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.586+0000 7f1f21db5e40 1 -- 172.21.15.167:0/4205167955 >> 172.21.15.167:0/4205167955 conn(0x55574cb36850 msgr2=0x55574cb38c60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-05T19:28:32.595 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.586+0000 7f1f21db5e40 1 -- 172.21.15.167:0/4205167955 shutdown_connections 2023-10-05T19:28:32.595 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.590+0000 7f1f21db5e40 1 -- 172.21.15.167:0/4205167955 wait complete. 2023-10-05T19:28:32.596 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.590+0000 7f1f21db5e40 1 Processor -- start 2023-10-05T19:28:32.596 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.590+0000 7f1f21db5e40 1 -- start start 2023-10-05T19:28:32.596 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.590+0000 7f1f21db5e40 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55574cb3ae30 0x55574cb59f30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-05T19:28:32.596 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.590+0000 7f1f21db5e40 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x55574ca08e30 con 0x55574cb3ae30 2023-10-05T19:28:32.596 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.590+0000 7f1f1f8ea700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55574cb3ae30 0x55574cb59f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-05T19:28:32.597 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.590+0000 7f1f1f8ea700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55574cb3ae30 0x55574cb59f30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.118:3300/0 says I am v2:172.21.15.167:58444/0 (socket says 172.21.15.167:58444) 2023-10-05T19:28:32.597 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.590+0000 7f1f1f8ea700 1 -- 172.21.15.167:0/3124030624 learned_addr learned my addr 172.21.15.167:0/3124030624 (peer_addr_for_me v2:172.21.15.167:0/0) 2023-10-05T19:28:32.597 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.590+0000 7f1f1f8ea700 1 -- 172.21.15.167:0/3124030624 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1f10005cf0 con 0x55574cb3ae30 2023-10-05T19:28:32.597 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.590+0000 7f1f1f8ea700 1 --2- 172.21.15.167:0/3124030624 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55574cb3ae30 0x55574cb59f30 secure :-1 s=READY pgs=358 cs=0 l=1 rev1=1 crypto rx=0x7f1f10000c00 tx=0x7f1f10015000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=54def3671f016e6d server_cookie=0 in_seq=0 out_seq=0 2023-10-05T19:28:32.598 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.590+0000 7f1f1d8e6700 1 -- 172.21.15.167:0/3124030624 <== mon.0 v2:172.21.15.118:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f1f10008970 con 0x55574cb3ae30 2023-10-05T19:28:32.598 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.590+0000 7f1f1d8e6700 1 -- 172.21.15.167:0/3124030624 <== mon.0 v2:172.21.15.118:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1f10008af0 con 0x55574cb3ae30 2023-10-05T19:28:32.598 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.590+0000 7f1f1d8e6700 1 -- 172.21.15.167:0/3124030624 <== mon.0 v2:172.21.15.118:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f1f10008de0 con 0x55574cb3ae30 2023-10-05T19:28:32.599 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.590+0000 7f1f21db5e40 1 -- 172.21.15.167:0/3124030624 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55574cb3ae30 msgr2=0x55574cb59f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-05T19:28:32.599 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.590+0000 7f1f21db5e40 1 --2- 172.21.15.167:0/3124030624 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55574cb3ae30 0x55574cb59f30 secure :-1 s=READY pgs=358 cs=0 l=1 rev1=1 crypto rx=0x7f1f10000c00 tx=0x7f1f10015000 comp rx=0 tx=0).stop 2023-10-05T19:28:32.599 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.590+0000 7f1f21db5e40 1 -- 172.21.15.167:0/3124030624 shutdown_connections 2023-10-05T19:28:32.599 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.590+0000 7f1f21db5e40 1 --2- 172.21.15.167:0/3124030624 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55574cb3ae30 0x55574cb59f30 unknown :-1 s=CLOSED pgs=358 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-05T19:28:32.599 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.590+0000 7f1f21db5e40 1 -- 172.21.15.167:0/3124030624 >> 172.21.15.167:0/3124030624 conn(0x55574cb36850 msgr2=0x55574cb55af0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-05T19:28:32.599 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.590+0000 7f1f21db5e40 1 -- 172.21.15.167:0/3124030624 shutdown_connections 2023-10-05T19:28:32.599 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.590+0000 7f1f21db5e40 1 -- 172.21.15.167:0/3124030624 wait complete. 2023-10-05T19:28:32.599 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.590+0000 7f1f21db5e40 1 Processor -- start 2023-10-05T19:28:32.599 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.590+0000 7f1f21db5e40 1 -- start start 2023-10-05T19:28:32.599 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.594+0000 7f1f21db5e40 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55574cb3ae30 0x55574cab0af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-05T19:28:32.599 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.594+0000 7f1f21db5e40 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x55574c9dbed0 con 0x55574cb3ae30 2023-10-05T19:28:32.601 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.594+0000 7f1f1f8ea700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55574cb3ae30 0x55574cab0af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-05T19:28:32.601 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.594+0000 7f1f1f8ea700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55574cb3ae30 0x55574cab0af0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.118:3300/0 says I am v2:172.21.15.167:58460/0 (socket says 172.21.15.167:58460) 2023-10-05T19:28:32.601 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.594+0000 7f1f1f8ea700 1 -- 172.21.15.167:0/369902812 learned_addr learned my addr 172.21.15.167:0/369902812 (peer_addr_for_me v2:172.21.15.167:0/0) 2023-10-05T19:28:32.601 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.594+0000 7f1f1f8ea700 1 -- 172.21.15.167:0/369902812 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55574cb3c8f0 con 0x55574cb3ae30 2023-10-05T19:28:32.601 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.594+0000 7f1f1f8ea700 1 --2- 172.21.15.167:0/369902812 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55574cb3ae30 0x55574cab0af0 secure :-1 s=READY pgs=359 cs=0 l=1 rev1=1 crypto rx=0x7f1f1000c010 tx=0x7f1f10003050 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-05T19:28:32.601 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.594+0000 7f1f17fff700 1 -- 172.21.15.167:0/369902812 <== mon.0 v2:172.21.15.118:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f1f10012ba0 con 0x55574cb3ae30 2023-10-05T19:28:32.601 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.594+0000 7f1f17fff700 1 -- 172.21.15.167:0/369902812 <== mon.0 v2:172.21.15.118:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1f1002a070 con 0x55574cb3ae30 2023-10-05T19:28:32.601 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.594+0000 7f1f17fff700 1 -- 172.21.15.167:0/369902812 <== mon.0 v2:172.21.15.118:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f1f10012ea0 con 0x55574cb3ae30 2023-10-05T19:28:32.601 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.594+0000 7f1f21db5e40 1 -- 172.21.15.167:0/369902812 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x55574cab1030 con 0x55574cb3ae30 2023-10-05T19:28:32.601 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.594+0000 7f1f21db5e40 1 -- 172.21.15.167:0/369902812 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55574caaf3f0 con 0x55574cb3ae30 2023-10-05T19:28:32.602 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.598+0000 7f1f17fff700 1 -- 172.21.15.167:0/369902812 <== mon.0 v2:172.21.15.118:3300/0 4 ==== mgrmap(e 5) v1 ==== 40329+0+0 (secure 0 0 0) 0x7f1f10019070 con 0x55574cb3ae30 2023-10-05T19:28:32.603 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.598+0000 7f1f21db5e40 20 rados->read ofs=0 len=0 2023-10-05T19:28:32.603 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.598+0000 7f1f17fff700 1 --2- 172.21.15.167:0/369902812 >> [v2:172.21.15.167:6824/29889,v1:172.21.15.167:6825/29889] conn(0x7f1f00030bf0 0x7f1f000330a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-05T19:28:32.603 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.598+0000 7f1f17fff700 1 -- 172.21.15.167:0/369902812 <== mon.0 v2:172.21.15.118:3300/0 5 ==== osd_map(30..30 src has 1..30) v4 ==== 6057+0+0 (secure 0 0 0) 0x7f1f1002c070 con 0x55574cb3ae30 2023-10-05T19:28:32.603 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.598+0000 7f1f1f0e9700 1 --2- 172.21.15.167:0/369902812 >> [v2:172.21.15.167:6824/29889,v1:172.21.15.167:6825/29889] conn(0x7f1f00030bf0 0x7f1f000330a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-05T19:28:32.604 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.598+0000 7f1f21db5e40 1 --2- 172.21.15.167:0/369902812 >> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] conn(0x55574cb3b740 0x55574cb6e820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-05T19:28:32.604 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.598+0000 7f1f21db5e40 1 -- 172.21.15.167:0/369902812 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:1 5.12 5:49953fa1:::default.realm:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x55574cb6eed0 con 0x55574cb3b740 2023-10-05T19:28:32.604 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.598+0000 7f1f200eb700 1 --2- 172.21.15.167:0/369902812 >> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] conn(0x55574cb3b740 0x55574cb6e820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-05T19:28:32.604 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.598+0000 7f1f1f0e9700 1 --2- 172.21.15.167:0/369902812 >> [v2:172.21.15.167:6824/29889,v1:172.21.15.167:6825/29889] conn(0x7f1f00030bf0 0x7f1f000330a0 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7f1f08004170 tx=0x7f1f08001380 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-05T19:28:32.605 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.598+0000 7f1f200eb700 1 --2- 172.21.15.167:0/369902812 >> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] conn(0x55574cb3b740 0x55574cb6e820 crc :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-05T19:28:32.605 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.598+0000 7f1f200eb700 1 -- 172.21.15.167:0/369902812 <== osd.1 v2:172.21.15.167:6800/30035 1 ==== osd_op_reply(1 default.realm [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 ==== 157+0+0 (crc 0 0 0) 0x7f1f1800a5a0 con 0x55574cb3b740 2023-10-05T19:28:32.605 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.598+0000 7f1f21db5e40 20 rados_obj.operate() r=-2 bl.length=0 2023-10-05T19:28:32.605 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.598+0000 7f1f21db5e40 20 realm 2023-10-05T19:28:32.605 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.598+0000 7f1f21db5e40 20 rados->read ofs=0 len=0 2023-10-05T19:28:32.606 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.598+0000 7f1f21db5e40 1 -- 172.21.15.167:0/369902812 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:2 5.12 5:49953fa1:::default.realm:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x55574cb6be80 con 0x55574cb3b740 2023-10-05T19:28:32.606 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.598+0000 7f1f200eb700 1 -- 172.21.15.167:0/369902812 <== osd.1 v2:172.21.15.167:6800/30035 2 ==== osd_op_reply(2 default.realm [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 ==== 157+0+0 (crc 0 0 0) 0x7f1f1800a5a0 con 0x55574cb3b740 2023-10-05T19:28:32.606 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.598+0000 7f1f21db5e40 20 rados_obj.operate() r=-2 bl.length=0 2023-10-05T19:28:32.606 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.598+0000 7f1f21db5e40 4 RGWPeriod::init failed to init realm id : (2) No such file or directory 2023-10-05T19:28:32.606 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.598+0000 7f1f21db5e40 20 rados->read ofs=0 len=0 2023-10-05T19:28:32.606 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.598+0000 7f1f21db5e40 1 -- 172.21.15.167:0/369902812 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:3 5.12 5:49953fa1:::default.realm:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x55574cb6f980 con 0x55574cb3b740 2023-10-05T19:28:32.607 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.598+0000 7f1f200eb700 1 -- 172.21.15.167:0/369902812 <== osd.1 v2:172.21.15.167:6800/30035 3 ==== osd_op_reply(3 default.realm [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 ==== 157+0+0 (crc 0 0 0) 0x7f1f1800a5a0 con 0x55574cb3b740 2023-10-05T19:28:32.608 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.598+0000 7f1f21db5e40 20 rados_obj.operate() r=-2 bl.length=0 2023-10-05T19:28:32.608 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.598+0000 7f1f21db5e40 20 rados->read ofs=0 len=0 2023-10-05T19:28:32.608 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.598+0000 7f1f21db5e40 1 -- 172.21.15.167:0/369902812 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:4 5.f 5:f43b6ece:::zone_names.default:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x55574cb6f560 con 0x55574cb3b740 2023-10-05T19:28:32.608 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.602+0000 7f1f200eb700 1 -- 172.21.15.167:0/369902812 <== osd.1 v2:172.21.15.167:6800/30035 4 ==== osd_op_reply(4 zone_names.default [read 0~46 out=46b] v0'0 uv1 ondisk = 0) v8 ==== 162+0+46 (crc 0 0 0) 0x7f1f1800a5a0 con 0x55574cb3b740 2023-10-05T19:28:32.608 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.602+0000 7f1f21db5e40 20 rados_obj.operate() r=0 bl.length=46 2023-10-05T19:28:32.608 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.602+0000 7f1f21db5e40 20 rados->read ofs=0 len=0 2023-10-05T19:28:32.608 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.602+0000 7f1f21db5e40 1 -- 172.21.15.167:0/369902812 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:5 5.1d 5:b8374eb4:::zone_info.9a9df965-6612-45d4-a45f-6e3c889bacc2:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x55574cb703f0 con 0x55574cb3b740 2023-10-05T19:28:32.608 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.602+0000 7f1f200eb700 1 -- 172.21.15.167:0/369902812 <== osd.1 v2:172.21.15.167:6800/30035 5 ==== osd_op_reply(5 zone_info.9a9df965-6612-45d4-a45f-6e3c889bacc2 [read 0~874 out=874b] v0'0 uv2 ondisk = 0) v8 ==== 190+0+874 (crc 0 0 0) 0x7f1f1800a5a0 con 0x55574cb3b740 2023-10-05T19:28:32.609 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.602+0000 7f1f21db5e40 20 rados_obj.operate() r=0 bl.length=874 2023-10-05T19:28:32.609 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.602+0000 7f1f21db5e40 20 searching for the correct realm 2023-10-05T19:28:32.609 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.602+0000 7f1f21db5e40 1 --2- 172.21.15.167:0/369902812 >> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] conn(0x55574cb72240 0x55574cb74620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-05T19:28:32.609 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.602+0000 7f1f21db5e40 1 -- 172.21.15.167:0/369902812 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:6 5.0 5:00000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x55574cb74b60 con 0x55574cb72240 2023-10-05T19:28:32.609 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.602+0000 7f1f1f8ea700 1 --2- 172.21.15.167:0/369902812 >> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] conn(0x55574cb72240 0x55574cb74620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-05T19:28:32.609 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.602+0000 7f1f1f8ea700 1 --2- 172.21.15.167:0/369902812 >> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] conn(0x55574cb72240 0x55574cb74620 crc :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-05T19:28:32.611 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.606+0000 7f1f1f8ea700 1 -- 172.21.15.167:0/369902812 <== osd.3 v2:172.21.15.167:6808/30036 1 ==== osd_op_reply(6 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f1f10022020 con 0x55574cb72240 2023-10-05T19:28:32.611 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.606+0000 7f1f1f8ea700 1 --2- 172.21.15.167:0/369902812 >> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] conn(0x7f1f10007030 0x7f1f100249c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-05T19:28:32.611 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.606+0000 7f1f1f8ea700 1 -- 172.21.15.167:0/369902812 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:7 5.10 5:08000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x55574cb74b60 con 0x7f1f10007030 2023-10-05T19:28:32.611 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.606+0000 7f1f1f0e9700 1 --2- 172.21.15.167:0/369902812 >> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] conn(0x7f1f10007030 0x7f1f100249c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-05T19:28:32.611 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.606+0000 7f1f1f0e9700 1 --2- 172.21.15.167:0/369902812 >> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] conn(0x7f1f10007030 0x7f1f100249c0 crc :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-05T19:28:32.612 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.606+0000 7f1f1f0e9700 1 -- 172.21.15.167:0/369902812 <== osd.2 v2:172.21.15.167:6816/30037 1 ==== osd_op_reply(7 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f1f08006070 con 0x7f1f10007030 2023-10-05T19:28:32.612 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.606+0000 7f1f1f0e9700 1 --2- 172.21.15.167:0/369902812 >> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] conn(0x7f1f0800d3d0 0x7f1f0800d820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-05T19:28:32.612 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.606+0000 7f1f1f0e9700 1 -- 172.21.15.167:0/369902812 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:8 5.8 5:10000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x55574cb74b60 con 0x7f1f0800d3d0 2023-10-05T19:28:32.613 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.606+0000 7f1f200eb700 1 --2- 172.21.15.167:0/369902812 >> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] conn(0x7f1f0800d3d0 0x7f1f0800d820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-05T19:28:32.613 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.606+0000 7f1f200eb700 1 --2- 172.21.15.167:0/369902812 >> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] conn(0x7f1f0800d3d0 0x7f1f0800d820 crc :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-05T19:28:32.614 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.610+0000 7f1f200eb700 1 -- 172.21.15.167:0/369902812 <== osd.0 v2:172.21.15.118:6800/27690 1 ==== osd_op_reply(8 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f1f1800a5a0 con 0x7f1f0800d3d0 2023-10-05T19:28:32.614 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.610+0000 7f1f200eb700 1 -- 172.21.15.167:0/369902812 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:9 5.18 5:18000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x55574cb74b60 con 0x55574cb3b740 2023-10-05T19:28:32.615 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.610+0000 7f1f200eb700 1 -- 172.21.15.167:0/369902812 <== osd.1 v2:172.21.15.167:6800/30035 6 ==== osd_op_reply(9 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f1f1800a5a0 con 0x55574cb3b740 2023-10-05T19:28:32.615 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.610+0000 7f1f200eb700 1 -- 172.21.15.167:0/369902812 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:10 5.4 5:20000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x55574cb74b60 con 0x55574cb3b740 2023-10-05T19:28:32.615 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.610+0000 7f1f200eb700 1 -- 172.21.15.167:0/369902812 <== osd.1 v2:172.21.15.167:6800/30035 7 ==== osd_op_reply(10 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f1f1800a5a0 con 0x55574cb3b740 2023-10-05T19:28:32.615 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.610+0000 7f1f200eb700 1 -- 172.21.15.167:0/369902812 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:11 5.14 5:28000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x55574cb74b60 con 0x7f1f0800d3d0 2023-10-05T19:28:32.616 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.610+0000 7f1f200eb700 1 -- 172.21.15.167:0/369902812 <== osd.0 v2:172.21.15.118:6800/27690 2 ==== osd_op_reply(11 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f1f1800a5a0 con 0x7f1f0800d3d0 2023-10-05T19:28:32.616 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.610+0000 7f1f200eb700 1 -- 172.21.15.167:0/369902812 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:12 5.c 5:30000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x55574cb74b60 con 0x7f1f0800d3d0 2023-10-05T19:28:32.617 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.610+0000 7f1f200eb700 1 -- 172.21.15.167:0/369902812 <== osd.0 v2:172.21.15.118:6800/27690 3 ==== osd_op_reply(12 [pgnls start_epoch 30 out=112b] v23'1 uv1 ondisk = 1) v8 ==== 144+0+112 (crc 0 0 0) 0x7f1f1800a5a0 con 0x7f1f0800d3d0 2023-10-05T19:28:32.617 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.610+0000 7f1f200eb700 1 -- 172.21.15.167:0/369902812 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:13 5.1c 5:38000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x55574cb74b60 con 0x7f1f10007030 2023-10-05T19:28:32.617 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.610+0000 7f1f1f0e9700 1 -- 172.21.15.167:0/369902812 <== osd.2 v2:172.21.15.167:6816/30037 2 ==== osd_op_reply(13 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f1f08006070 con 0x7f1f10007030 2023-10-05T19:28:32.618 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.610+0000 7f1f1f0e9700 1 -- 172.21.15.167:0/369902812 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:14 5.2 5:40000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x55574cb74b60 con 0x55574cb72240 2023-10-05T19:28:32.618 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.614+0000 7f1f1f8ea700 1 -- 172.21.15.167:0/369902812 <== osd.3 v2:172.21.15.167:6808/30036 2 ==== osd_op_reply(14 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f1f10022020 con 0x55574cb72240 2023-10-05T19:28:32.618 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.614+0000 7f1f1f8ea700 1 -- 172.21.15.167:0/369902812 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:15 5.12 5:48000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x55574cb74b60 con 0x55574cb3b740 2023-10-05T19:28:32.619 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.614+0000 7f1f200eb700 1 -- 172.21.15.167:0/369902812 <== osd.1 v2:172.21.15.167:6800/30035 8 ==== osd_op_reply(15 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f1f1800a5a0 con 0x55574cb3b740 2023-10-05T19:28:32.619 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.614+0000 7f1f200eb700 1 -- 172.21.15.167:0/369902812 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:16 5.a 5:50000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x55574cb74b60 con 0x7f1f0800d3d0 2023-10-05T19:28:32.620 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.614+0000 7f1f200eb700 1 -- 172.21.15.167:0/369902812 <== osd.0 v2:172.21.15.118:6800/27690 4 ==== osd_op_reply(16 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f1f1800a5a0 con 0x7f1f0800d3d0 2023-10-05T19:28:32.620 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.614+0000 7f1f200eb700 1 -- 172.21.15.167:0/369902812 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:17 5.1a 5:58000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x55574cb74b60 con 0x55574cb3b740 2023-10-05T19:28:32.620 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.614+0000 7f1f200eb700 1 -- 172.21.15.167:0/369902812 <== osd.1 v2:172.21.15.167:6800/30035 9 ==== osd_op_reply(17 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f1f1800a5a0 con 0x55574cb3b740 2023-10-05T19:28:32.620 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.614+0000 7f1f200eb700 1 -- 172.21.15.167:0/369902812 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:18 5.6 5:60000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x55574cb74b60 con 0x7f1f10007030 2023-10-05T19:28:32.621 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.614+0000 7f1f1f0e9700 1 -- 172.21.15.167:0/369902812 <== osd.2 v2:172.21.15.167:6816/30037 3 ==== osd_op_reply(18 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f1f08006070 con 0x7f1f10007030 2023-10-05T19:28:32.621 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.614+0000 7f1f1f0e9700 1 -- 172.21.15.167:0/369902812 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:19 5.16 5:68000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x55574cb74b60 con 0x55574cb72240 2023-10-05T19:28:32.622 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.614+0000 7f1f1f8ea700 1 -- 172.21.15.167:0/369902812 <== osd.3 v2:172.21.15.167:6808/30036 3 ==== osd_op_reply(19 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f1f10022020 con 0x55574cb72240 2023-10-05T19:28:32.622 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.614+0000 7f1f1f8ea700 1 -- 172.21.15.167:0/369902812 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:20 5.e 5:70000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x55574cb74b60 con 0x55574cb72240 2023-10-05T19:28:32.622 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.618+0000 7f1f1f8ea700 1 -- 172.21.15.167:0/369902812 <== osd.3 v2:172.21.15.167:6808/30036 4 ==== osd_op_reply(20 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f1f10022020 con 0x55574cb72240 2023-10-05T19:28:32.623 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.618+0000 7f1f1f8ea700 1 -- 172.21.15.167:0/369902812 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:21 5.1e 5:78000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x55574cb74b60 con 0x7f1f0800d3d0 2023-10-05T19:28:32.623 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.618+0000 7f1f200eb700 1 -- 172.21.15.167:0/369902812 <== osd.0 v2:172.21.15.118:6800/27690 5 ==== osd_op_reply(21 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f1f1800a5a0 con 0x7f1f0800d3d0 2023-10-05T19:28:32.623 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.618+0000 7f1f200eb700 1 -- 172.21.15.167:0/369902812 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:22 5.1 5:80000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x55574cb74b60 con 0x55574cb72240 2023-10-05T19:28:32.624 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.618+0000 7f1f1f8ea700 1 -- 172.21.15.167:0/369902812 <== osd.3 v2:172.21.15.167:6808/30036 5 ==== osd_op_reply(22 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f1f10022020 con 0x55574cb72240 2023-10-05T19:28:32.624 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.618+0000 7f1f1f8ea700 1 -- 172.21.15.167:0/369902812 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:23 5.11 5:88000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x55574cb74b60 con 0x55574cb72240 2023-10-05T19:28:32.624 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.618+0000 7f1f1f8ea700 1 -- 172.21.15.167:0/369902812 <== osd.3 v2:172.21.15.167:6808/30036 6 ==== osd_op_reply(23 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f1f10022020 con 0x55574cb72240 2023-10-05T19:28:32.624 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.618+0000 7f1f1f8ea700 1 -- 172.21.15.167:0/369902812 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:24 5.9 5:90000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x55574cb74b60 con 0x55574cb3b740 2023-10-05T19:28:32.625 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.618+0000 7f1f200eb700 1 -- 172.21.15.167:0/369902812 <== osd.1 v2:172.21.15.167:6800/30035 10 ==== osd_op_reply(24 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f1f1800a5a0 con 0x55574cb3b740 2023-10-05T19:28:32.625 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.618+0000 7f1f200eb700 1 -- 172.21.15.167:0/369902812 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:25 5.19 5:98000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x55574cb74b60 con 0x55574cb3b740 2023-10-05T19:28:32.626 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.618+0000 7f1f200eb700 1 -- 172.21.15.167:0/369902812 <== osd.1 v2:172.21.15.167:6800/30035 11 ==== osd_op_reply(25 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f1f1800a5a0 con 0x55574cb3b740 2023-10-05T19:28:32.626 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.618+0000 7f1f200eb700 1 -- 172.21.15.167:0/369902812 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:26 5.5 5:a0000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x55574cb74b60 con 0x7f1f0800d3d0 2023-10-05T19:28:32.627 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.622+0000 7f1f200eb700 1 -- 172.21.15.167:0/369902812 <== osd.0 v2:172.21.15.118:6800/27690 6 ==== osd_op_reply(26 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f1f1800a5a0 con 0x7f1f0800d3d0 2023-10-05T19:28:32.627 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.622+0000 7f1f200eb700 1 -- 172.21.15.167:0/369902812 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:27 5.15 5:a8000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x55574cb74b60 con 0x55574cb72240 2023-10-05T19:28:32.627 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.622+0000 7f1f1f8ea700 1 -- 172.21.15.167:0/369902812 <== osd.3 v2:172.21.15.167:6808/30036 7 ==== osd_op_reply(27 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f1f10022020 con 0x55574cb72240 2023-10-05T19:28:32.628 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.622+0000 7f1f1f8ea700 1 -- 172.21.15.167:0/369902812 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:28 5.d 5:b0000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x55574cb74b60 con 0x7f1f10007030 2023-10-05T19:28:32.628 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.622+0000 7f1f1f0e9700 1 -- 172.21.15.167:0/369902812 <== osd.2 v2:172.21.15.167:6816/30037 4 ==== osd_op_reply(28 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f1f08006070 con 0x7f1f10007030 2023-10-05T19:28:32.628 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.622+0000 7f1f1f0e9700 1 -- 172.21.15.167:0/369902812 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:29 5.1d 5:b8000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x55574cb74b60 con 0x55574cb3b740 2023-10-05T19:28:32.629 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.622+0000 7f1f200eb700 1 -- 172.21.15.167:0/369902812 <== osd.1 v2:172.21.15.167:6800/30035 12 ==== osd_op_reply(29 [pgnls start_epoch 30 out=107b] v30'2 uv2 ondisk = 1) v8 ==== 144+0+107 (crc 0 0 0) 0x7f1f1800a5a0 con 0x55574cb3b740 2023-10-05T19:28:32.629 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.622+0000 7f1f200eb700 1 -- 172.21.15.167:0/369902812 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:30 5.3 5:c0000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x55574cb74b60 con 0x7f1f0800d3d0 2023-10-05T19:28:32.630 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.626+0000 7f1f200eb700 1 -- 172.21.15.167:0/369902812 <== osd.0 v2:172.21.15.118:6800/27690 7 ==== osd_op_reply(30 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f1f1800a5a0 con 0x7f1f0800d3d0 2023-10-05T19:28:32.630 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.626+0000 7f1f200eb700 1 -- 172.21.15.167:0/369902812 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:31 5.13 5:c8000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x55574cb74b60 con 0x55574cb72240 2023-10-05T19:28:32.631 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.626+0000 7f1f1f8ea700 1 -- 172.21.15.167:0/369902812 <== osd.3 v2:172.21.15.167:6808/30036 8 ==== osd_op_reply(31 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f1f10022020 con 0x55574cb72240 2023-10-05T19:28:32.631 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.626+0000 7f1f1f8ea700 1 -- 172.21.15.167:0/369902812 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:32 5.b 5:d0000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x55574cb74b60 con 0x7f1f0800d3d0 2023-10-05T19:28:32.632 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.626+0000 7f1f200eb700 1 -- 172.21.15.167:0/369902812 <== osd.0 v2:172.21.15.118:6800/27690 8 ==== osd_op_reply(32 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f1f1800a5a0 con 0x7f1f0800d3d0 2023-10-05T19:28:32.632 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.626+0000 7f1f200eb700 1 -- 172.21.15.167:0/369902812 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:33 5.1b 5:d8000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x55574cb74b60 con 0x7f1f10007030 2023-10-05T19:28:32.633 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.626+0000 7f1f1f0e9700 1 -- 172.21.15.167:0/369902812 <== osd.2 v2:172.21.15.167:6816/30037 5 ==== osd_op_reply(33 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f1f08006070 con 0x7f1f10007030 2023-10-05T19:28:32.633 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.626+0000 7f1f1f0e9700 1 -- 172.21.15.167:0/369902812 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:34 5.7 5:e0000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x55574cb74b60 con 0x55574cb72240 2023-10-05T19:28:32.633 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.626+0000 7f1f1f8ea700 1 -- 172.21.15.167:0/369902812 <== osd.3 v2:172.21.15.167:6808/30036 9 ==== osd_op_reply(34 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f1f10022020 con 0x55574cb72240 2023-10-05T19:28:32.633 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.626+0000 7f1f1f8ea700 1 -- 172.21.15.167:0/369902812 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:35 5.17 5:e8000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x55574cb74b60 con 0x7f1f0800d3d0 2023-10-05T19:28:32.634 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.630+0000 7f1f200eb700 1 -- 172.21.15.167:0/369902812 <== osd.0 v2:172.21.15.118:6800/27690 9 ==== osd_op_reply(35 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f1f1800a5a0 con 0x7f1f0800d3d0 2023-10-05T19:28:32.634 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.630+0000 7f1f200eb700 1 -- 172.21.15.167:0/369902812 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:36 5.f 5:f0000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x55574cb74b60 con 0x55574cb3b740 2023-10-05T19:28:32.635 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.630+0000 7f1f200eb700 1 -- 172.21.15.167:0/369902812 <== osd.1 v2:172.21.15.167:6800/30035 13 ==== osd_op_reply(36 [pgnls start_epoch 30 out=115b] v23'2 uv2 ondisk = 1) v8 ==== 144+0+115 (crc 0 0 0) 0x7f1f1800a5a0 con 0x55574cb3b740 2023-10-05T19:28:32.635 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.630+0000 7f1f200eb700 1 -- 172.21.15.167:0/369902812 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:37 5.1f 5:f8000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x55574cb74b60 con 0x7f1f10007030 2023-10-05T19:28:32.636 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.630+0000 7f1f1f0e9700 1 -- 172.21.15.167:0/369902812 <== osd.2 v2:172.21.15.167:6816/30037 6 ==== osd_op_reply(37 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f1f08006070 con 0x7f1f10007030 2023-10-05T19:28:32.636 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.630+0000 7f1f21db5e40 20 RGWRados::pool_iterate: got zonegroup_info.4f1f719d-9648-417b-b10c-e5d939825419 2023-10-05T19:28:32.637 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.630+0000 7f1f21db5e40 20 RGWRados::pool_iterate: got zone_info.9a9df965-6612-45d4-a45f-6e3c889bacc2 2023-10-05T19:28:32.637 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.630+0000 7f1f21db5e40 20 RGWRados::pool_iterate: got zone_names.default 2023-10-05T19:28:32.637 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.630+0000 7f1f21db5e40 20 RGWRados::pool_iterate: got zonegroups_names.default 2023-10-05T19:28:32.637 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.630+0000 7f1f21db5e40 20 rados->read ofs=0 len=0 2023-10-05T19:28:32.637 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.630+0000 7f1f21db5e40 1 -- 172.21.15.167:0/369902812 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:38 5.12 5:49953fa1:::default.realm:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x55574cb70f70 con 0x55574cb3b740 2023-10-05T19:28:32.638 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.630+0000 7f1f200eb700 1 -- 172.21.15.167:0/369902812 <== osd.1 v2:172.21.15.167:6800/30035 14 ==== osd_op_reply(38 default.realm [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 ==== 157+0+0 (crc 0 0 0) 0x7f1f1800a5a0 con 0x55574cb3b740 2023-10-05T19:28:32.638 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.630+0000 7f1f21db5e40 20 rados_obj.operate() r=-2 bl.length=0 2023-10-05T19:28:32.638 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.630+0000 7f1f21db5e40 20 rados->read ofs=0 len=0 2023-10-05T19:28:32.638 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.630+0000 7f1f21db5e40 1 -- 172.21.15.167:0/369902812 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:39 5.f 5:f4c53578:::zonegroups_names.default:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x55574cb70b70 con 0x55574cb3b740 2023-10-05T19:28:32.638 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.630+0000 7f1f200eb700 1 -- 172.21.15.167:0/369902812 <== osd.1 v2:172.21.15.167:6800/30035 15 ==== osd_op_reply(39 zonegroups_names.default [read 0~46 out=46b] v0'0 uv2 ondisk = 0) v8 ==== 168+0+46 (crc 0 0 0) 0x7f1f1800a5a0 con 0x55574cb3b740 2023-10-05T19:28:32.638 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.630+0000 7f1f21db5e40 20 rados_obj.operate() r=0 bl.length=46 2023-10-05T19:28:32.639 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.630+0000 7f1f21db5e40 20 rados->read ofs=0 len=0 2023-10-05T19:28:32.639 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.630+0000 7f1f21db5e40 1 -- 172.21.15.167:0/369902812 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:40 5.c 5:30d299a5:::zonegroup_info.4f1f719d-9648-417b-b10c-e5d939825419:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x55574cb70b40 con 0x7f1f0800d3d0 2023-10-05T19:28:32.640 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.634+0000 7f1f200eb700 1 -- 172.21.15.167:0/369902812 <== osd.0 v2:172.21.15.118:6800/27690 10 ==== osd_op_reply(40 zonegroup_info.4f1f719d-9648-417b-b10c-e5d939825419 [read 0~362 out=362b] v0'0 uv1 ondisk = 0) v8 ==== 195+0+362 (crc 0 0 0) 0x7f1f1800a5a0 con 0x7f1f0800d3d0 2023-10-05T19:28:32.640 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.634+0000 7f1f21db5e40 20 rados_obj.operate() r=0 bl.length=362 2023-10-05T19:28:32.640 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.634+0000 7f1f21db5e40 20 rados->read ofs=0 len=0 2023-10-05T19:28:32.640 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.634+0000 7f1f21db5e40 1 -- 172.21.15.167:0/369902812 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:41 5.13 5:cbbf6c60:::converted:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x55574cb71900 con 0x55574cb72240 2023-10-05T19:28:32.640 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.634+0000 7f1f1f8ea700 1 -- 172.21.15.167:0/369902812 <== osd.3 v2:172.21.15.167:6808/30036 10 ==== osd_op_reply(41 converted [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 ==== 153+0+0 (crc 0 0 0) 0x7f1f10022020 con 0x55574cb72240 2023-10-05T19:28:32.640 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.634+0000 7f1f21db5e40 20 rados_obj.operate() r=-2 bl.length=0 2023-10-05T19:28:32.640 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.634+0000 7f1f21db5e40 20 rados->read ofs=0 len=0 2023-10-05T19:28:32.640 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.634+0000 7f1f21db5e40 1 -- 172.21.15.167:0/369902812 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:42 5.12 5:49953fa1:::default.realm:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x55574cb74fa0 con 0x55574cb3b740 2023-10-05T19:28:32.641 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.634+0000 7f1f200eb700 1 -- 172.21.15.167:0/369902812 <== osd.1 v2:172.21.15.167:6800/30035 16 ==== osd_op_reply(42 default.realm [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 ==== 157+0+0 (crc 0 0 0) 0x7f1f1800a5a0 con 0x55574cb3b740 2023-10-05T19:28:32.641 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.634+0000 7f1f21db5e40 20 rados_obj.operate() r=-2 bl.length=0 2023-10-05T19:28:32.641 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.634+0000 7f1f21db5e40 20 rados->read ofs=0 len=0 2023-10-05T19:28:32.641 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.634+0000 7f1f21db5e40 1 -- 172.21.15.167:0/369902812 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:43 5.f 5:f4c53578:::zonegroups_names.default:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x55574cb74fa0 con 0x55574cb3b740 2023-10-05T19:28:32.642 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.634+0000 7f1f200eb700 1 -- 172.21.15.167:0/369902812 <== osd.1 v2:172.21.15.167:6800/30035 17 ==== osd_op_reply(43 zonegroups_names.default [read 0~46 out=46b] v0'0 uv2 ondisk = 0) v8 ==== 168+0+46 (crc 0 0 0) 0x7f1f1800a5a0 con 0x55574cb3b740 2023-10-05T19:28:32.642 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.634+0000 7f1f21db5e40 20 rados_obj.operate() r=0 bl.length=46 2023-10-05T19:28:32.642 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.634+0000 7f1f21db5e40 1 -- 172.21.15.167:0/369902812 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:44 5.0 5:00000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x55574cb75fb0 con 0x55574cb72240 2023-10-05T19:28:32.644 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.638+0000 7f1f1f8ea700 1 -- 172.21.15.167:0/369902812 <== osd.3 v2:172.21.15.167:6808/30036 11 ==== osd_op_reply(44 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f1f10022020 con 0x55574cb72240 2023-10-05T19:28:32.644 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.638+0000 7f1f1f8ea700 1 -- 172.21.15.167:0/369902812 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:45 5.10 5:08000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x55574cb75fb0 con 0x7f1f10007030 2023-10-05T19:28:32.645 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.638+0000 7f1f1f0e9700 1 -- 172.21.15.167:0/369902812 <== osd.2 v2:172.21.15.167:6816/30037 7 ==== osd_op_reply(45 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f1f08006070 con 0x7f1f10007030 2023-10-05T19:28:32.645 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.638+0000 7f1f1f0e9700 1 -- 172.21.15.167:0/369902812 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:46 5.8 5:10000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x55574cb75fb0 con 0x7f1f0800d3d0 2023-10-05T19:28:32.646 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.638+0000 7f1f200eb700 1 -- 172.21.15.167:0/369902812 <== osd.0 v2:172.21.15.118:6800/27690 11 ==== osd_op_reply(46 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f1f1800a5a0 con 0x7f1f0800d3d0 2023-10-05T19:28:32.646 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.638+0000 7f1f200eb700 1 -- 172.21.15.167:0/369902812 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:47 5.18 5:18000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x55574cb70f70 con 0x55574cb3b740 2023-10-05T19:28:32.646 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.642+0000 7f1f200eb700 1 -- 172.21.15.167:0/369902812 <== osd.1 v2:172.21.15.167:6800/30035 18 ==== osd_op_reply(47 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f1f1800a5a0 con 0x55574cb3b740 2023-10-05T19:28:32.647 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.642+0000 7f1f200eb700 1 -- 172.21.15.167:0/369902812 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:48 5.4 5:20000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x55574cb70f70 con 0x55574cb3b740 2023-10-05T19:28:32.647 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.642+0000 7f1f200eb700 1 -- 172.21.15.167:0/369902812 <== osd.1 v2:172.21.15.167:6800/30035 19 ==== osd_op_reply(48 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f1f1800a5a0 con 0x55574cb3b740 2023-10-05T19:28:32.647 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.642+0000 7f1f200eb700 1 -- 172.21.15.167:0/369902812 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:49 5.14 5:28000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x55574cb70f70 con 0x7f1f0800d3d0 2023-10-05T19:28:32.648 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.642+0000 7f1f200eb700 1 -- 172.21.15.167:0/369902812 <== osd.0 v2:172.21.15.118:6800/27690 12 ==== osd_op_reply(49 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f1f1800a5a0 con 0x7f1f0800d3d0 2023-10-05T19:28:32.648 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.642+0000 7f1f200eb700 1 -- 172.21.15.167:0/369902812 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:50 5.c 5:30000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x55574cb70f70 con 0x7f1f0800d3d0 2023-10-05T19:28:32.649 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.642+0000 7f1f200eb700 1 -- 172.21.15.167:0/369902812 <== osd.0 v2:172.21.15.118:6800/27690 13 ==== osd_op_reply(50 [pgnls start_epoch 30 out=112b] v23'1 uv1 ondisk = 1) v8 ==== 144+0+112 (crc 0 0 0) 0x7f1f1800a5a0 con 0x7f1f0800d3d0 2023-10-05T19:28:32.649 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.642+0000 7f1f200eb700 1 -- 172.21.15.167:0/369902812 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:51 5.1c 5:38000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x55574cb70f70 con 0x7f1f10007030 2023-10-05T19:28:32.650 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.642+0000 7f1f1f0e9700 1 -- 172.21.15.167:0/369902812 <== osd.2 v2:172.21.15.167:6816/30037 8 ==== osd_op_reply(51 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f1f08006070 con 0x7f1f10007030 2023-10-05T19:28:32.650 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.646+0000 7f1f1f0e9700 1 -- 172.21.15.167:0/369902812 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:52 5.2 5:40000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x55574cb70f70 con 0x55574cb72240 2023-10-05T19:28:32.651 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.646+0000 7f1f1f8ea700 1 -- 172.21.15.167:0/369902812 <== osd.3 v2:172.21.15.167:6808/30036 12 ==== osd_op_reply(52 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f1f10022020 con 0x55574cb72240 2023-10-05T19:28:32.651 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.646+0000 7f1f1f8ea700 1 -- 172.21.15.167:0/369902812 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:53 5.12 5:48000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x55574cb70f70 con 0x55574cb3b740 2023-10-05T19:28:32.652 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.646+0000 7f1f200eb700 1 -- 172.21.15.167:0/369902812 <== osd.1 v2:172.21.15.167:6800/30035 20 ==== osd_op_reply(53 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f1f1800a5a0 con 0x55574cb3b740 2023-10-05T19:28:32.652 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.646+0000 7f1f200eb700 1 -- 172.21.15.167:0/369902812 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:54 5.a 5:50000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x55574cb70f70 con 0x7f1f0800d3d0 2023-10-05T19:28:32.653 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.646+0000 7f1f200eb700 1 -- 172.21.15.167:0/369902812 <== osd.0 v2:172.21.15.118:6800/27690 14 ==== osd_op_reply(54 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f1f1800a5a0 con 0x7f1f0800d3d0 2023-10-05T19:28:32.653 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.646+0000 7f1f200eb700 1 -- 172.21.15.167:0/369902812 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:55 5.1a 5:58000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x55574cb70f70 con 0x55574cb3b740 2023-10-05T19:28:32.654 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.646+0000 7f1f200eb700 1 -- 172.21.15.167:0/369902812 <== osd.1 v2:172.21.15.167:6800/30035 21 ==== osd_op_reply(55 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f1f1800a5a0 con 0x55574cb3b740 2023-10-05T19:28:32.654 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.650+0000 7f1f200eb700 1 -- 172.21.15.167:0/369902812 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:56 5.6 5:60000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x55574cb70f70 con 0x7f1f10007030 2023-10-05T19:28:32.655 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.650+0000 7f1f1f0e9700 1 -- 172.21.15.167:0/369902812 <== osd.2 v2:172.21.15.167:6816/30037 9 ==== osd_op_reply(56 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f1f08006070 con 0x7f1f10007030 2023-10-05T19:28:32.655 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.650+0000 7f1f1f0e9700 1 -- 172.21.15.167:0/369902812 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:57 5.16 5:68000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x55574cb70f70 con 0x55574cb72240 2023-10-05T19:28:32.655 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.650+0000 7f1f1f8ea700 1 -- 172.21.15.167:0/369902812 <== osd.3 v2:172.21.15.167:6808/30036 13 ==== osd_op_reply(57 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f1f10022020 con 0x55574cb72240 2023-10-05T19:28:32.655 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.650+0000 7f1f1f8ea700 1 -- 172.21.15.167:0/369902812 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:58 5.e 5:70000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x55574cb70f70 con 0x55574cb72240 2023-10-05T19:28:32.656 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.650+0000 7f1f1f8ea700 1 -- 172.21.15.167:0/369902812 <== osd.3 v2:172.21.15.167:6808/30036 14 ==== osd_op_reply(58 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f1f10022020 con 0x55574cb72240 2023-10-05T19:28:32.656 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.650+0000 7f1f1f8ea700 1 -- 172.21.15.167:0/369902812 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:59 5.1e 5:78000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x55574cb70f70 con 0x7f1f0800d3d0 2023-10-05T19:28:32.657 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.650+0000 7f1f200eb700 1 -- 172.21.15.167:0/369902812 <== osd.0 v2:172.21.15.118:6800/27690 15 ==== osd_op_reply(59 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f1f1803f040 con 0x7f1f0800d3d0 2023-10-05T19:28:32.657 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.650+0000 7f1f200eb700 1 -- 172.21.15.167:0/369902812 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:60 5.1 5:80000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x55574cb70f70 con 0x55574cb72240 2023-10-05T19:28:32.658 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.654+0000 7f1f1f8ea700 1 -- 172.21.15.167:0/369902812 <== osd.3 v2:172.21.15.167:6808/30036 15 ==== osd_op_reply(60 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f1f10022020 con 0x55574cb72240 2023-10-05T19:28:32.658 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.654+0000 7f1f1f8ea700 1 -- 172.21.15.167:0/369902812 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:61 5.11 5:88000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x55574cb70f70 con 0x55574cb72240 2023-10-05T19:28:32.659 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.654+0000 7f1f1f8ea700 1 -- 172.21.15.167:0/369902812 <== osd.3 v2:172.21.15.167:6808/30036 16 ==== osd_op_reply(61 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f1f10022020 con 0x55574cb72240 2023-10-05T19:28:32.659 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.654+0000 7f1f1f8ea700 1 -- 172.21.15.167:0/369902812 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:62 5.9 5:90000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x55574cb70f70 con 0x55574cb3b740 2023-10-05T19:28:32.660 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.654+0000 7f1f200eb700 1 -- 172.21.15.167:0/369902812 <== osd.1 v2:172.21.15.167:6800/30035 22 ==== osd_op_reply(62 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f1f1803f040 con 0x55574cb3b740 2023-10-05T19:28:32.660 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.654+0000 7f1f200eb700 1 -- 172.21.15.167:0/369902812 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:63 5.19 5:98000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x55574cb70f70 con 0x55574cb3b740 2023-10-05T19:28:32.661 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.654+0000 7f1f200eb700 1 -- 172.21.15.167:0/369902812 <== osd.1 v2:172.21.15.167:6800/30035 23 ==== osd_op_reply(63 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f1f1803f040 con 0x55574cb3b740 2023-10-05T19:28:32.661 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.654+0000 7f1f200eb700 1 -- 172.21.15.167:0/369902812 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:64 5.5 5:a0000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x55574cb70f70 con 0x7f1f0800d3d0 2023-10-05T19:28:32.662 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.654+0000 7f1f200eb700 1 -- 172.21.15.167:0/369902812 <== osd.0 v2:172.21.15.118:6800/27690 16 ==== osd_op_reply(64 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f1f1803f040 con 0x7f1f0800d3d0 2023-10-05T19:28:32.662 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.654+0000 7f1f200eb700 1 -- 172.21.15.167:0/369902812 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:65 5.15 5:a8000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x55574cb70f70 con 0x55574cb72240 2023-10-05T19:28:32.662 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.658+0000 7f1f1f8ea700 1 -- 172.21.15.167:0/369902812 <== osd.3 v2:172.21.15.167:6808/30036 17 ==== osd_op_reply(65 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f1f10022020 con 0x55574cb72240 2023-10-05T19:28:32.663 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.658+0000 7f1f1f8ea700 1 -- 172.21.15.167:0/369902812 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:66 5.d 5:b0000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x55574cb70f70 con 0x7f1f10007030 2023-10-05T19:28:32.663 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.658+0000 7f1f1f0e9700 1 -- 172.21.15.167:0/369902812 <== osd.2 v2:172.21.15.167:6816/30037 10 ==== osd_op_reply(66 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f1f08006070 con 0x7f1f10007030 2023-10-05T19:28:32.663 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.658+0000 7f1f1f0e9700 1 -- 172.21.15.167:0/369902812 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:67 5.1d 5:b8000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x55574cb70f70 con 0x55574cb3b740 2023-10-05T19:28:32.664 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.658+0000 7f1f200eb700 1 -- 172.21.15.167:0/369902812 <== osd.1 v2:172.21.15.167:6800/30035 24 ==== osd_op_reply(67 [pgnls start_epoch 30 out=107b] v30'2 uv2 ondisk = 1) v8 ==== 144+0+107 (crc 0 0 0) 0x7f1f1803f040 con 0x55574cb3b740 2023-10-05T19:28:32.664 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.658+0000 7f1f200eb700 1 -- 172.21.15.167:0/369902812 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:68 5.3 5:c0000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x55574cb70f70 con 0x7f1f0800d3d0 2023-10-05T19:28:32.665 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.658+0000 7f1f200eb700 1 -- 172.21.15.167:0/369902812 <== osd.0 v2:172.21.15.118:6800/27690 17 ==== osd_op_reply(68 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f1f1803f040 con 0x7f1f0800d3d0 2023-10-05T19:28:32.666 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.658+0000 7f1f200eb700 1 -- 172.21.15.167:0/369902812 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:69 5.13 5:c8000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x55574cb70f70 con 0x55574cb72240 2023-10-05T19:28:32.666 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.662+0000 7f1f1f8ea700 1 -- 172.21.15.167:0/369902812 <== osd.3 v2:172.21.15.167:6808/30036 18 ==== osd_op_reply(69 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f1f10022020 con 0x55574cb72240 2023-10-05T19:28:32.667 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.662+0000 7f1f1f8ea700 1 -- 172.21.15.167:0/369902812 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:70 5.b 5:d0000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x55574cb70f70 con 0x7f1f0800d3d0 2023-10-05T19:28:32.667 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.662+0000 7f1f200eb700 1 -- 172.21.15.167:0/369902812 <== osd.0 v2:172.21.15.118:6800/27690 18 ==== osd_op_reply(70 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f1f1803f040 con 0x7f1f0800d3d0 2023-10-05T19:28:32.668 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.662+0000 7f1f200eb700 1 -- 172.21.15.167:0/369902812 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:71 5.1b 5:d8000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x55574cb70f70 con 0x7f1f10007030 2023-10-05T19:28:32.668 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.662+0000 7f1f1f0e9700 1 -- 172.21.15.167:0/369902812 <== osd.2 v2:172.21.15.167:6816/30037 11 ==== osd_op_reply(71 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f1f08006070 con 0x7f1f10007030 2023-10-05T19:28:32.668 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.662+0000 7f1f1f0e9700 1 -- 172.21.15.167:0/369902812 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:72 5.7 5:e0000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x55574cb70f70 con 0x55574cb72240 2023-10-05T19:28:32.669 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.662+0000 7f1f1f8ea700 1 -- 172.21.15.167:0/369902812 <== osd.3 v2:172.21.15.167:6808/30036 19 ==== osd_op_reply(72 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f1f10022020 con 0x55574cb72240 2023-10-05T19:28:32.669 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.662+0000 7f1f1f8ea700 1 -- 172.21.15.167:0/369902812 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:73 5.17 5:e8000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x55574cb70f70 con 0x7f1f0800d3d0 2023-10-05T19:28:32.670 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.666+0000 7f1f200eb700 1 -- 172.21.15.167:0/369902812 <== osd.0 v2:172.21.15.118:6800/27690 19 ==== osd_op_reply(73 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f1f1803f040 con 0x7f1f0800d3d0 2023-10-05T19:28:32.670 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.666+0000 7f1f200eb700 1 -- 172.21.15.167:0/369902812 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:74 5.f 5:f0000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x55574cb70f70 con 0x55574cb3b740 2023-10-05T19:28:32.671 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.666+0000 7f1f200eb700 1 -- 172.21.15.167:0/369902812 <== osd.1 v2:172.21.15.167:6800/30035 25 ==== osd_op_reply(74 [pgnls start_epoch 30 out=115b] v23'2 uv2 ondisk = 1) v8 ==== 144+0+115 (crc 0 0 0) 0x7f1f1803f040 con 0x55574cb3b740 2023-10-05T19:28:32.671 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.666+0000 7f1f200eb700 1 -- 172.21.15.167:0/369902812 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:75 5.1f 5:f8000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x55574cb70f70 con 0x7f1f10007030 2023-10-05T19:28:32.672 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.666+0000 7f1f1f0e9700 1 -- 172.21.15.167:0/369902812 <== osd.2 v2:172.21.15.167:6816/30037 12 ==== osd_op_reply(75 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f1f08006070 con 0x7f1f10007030 2023-10-05T19:28:32.672 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.666+0000 7f1f21db5e40 20 RGWRados::pool_iterate: got zonegroup_info.4f1f719d-9648-417b-b10c-e5d939825419 2023-10-05T19:28:32.673 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.666+0000 7f1f21db5e40 20 RGWRados::pool_iterate: got zone_info.9a9df965-6612-45d4-a45f-6e3c889bacc2 2023-10-05T19:28:32.673 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.666+0000 7f1f21db5e40 20 RGWRados::pool_iterate: got zone_names.default 2023-10-05T19:28:32.673 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.666+0000 7f1f21db5e40 20 RGWRados::pool_iterate: got zonegroups_names.default 2023-10-05T19:28:32.673 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.666+0000 7f1f21db5e40 20 rados->read ofs=0 len=0 2023-10-05T19:28:32.673 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.666+0000 7f1f21db5e40 1 -- 172.21.15.167:0/369902812 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:76 5.f 5:f43b6ece:::zone_names.default:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x55574cb74fa0 con 0x55574cb3b740 2023-10-05T19:28:32.674 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.666+0000 7f1f200eb700 1 -- 172.21.15.167:0/369902812 <== osd.1 v2:172.21.15.167:6800/30035 26 ==== osd_op_reply(76 zone_names.default [read 0~46 out=46b] v0'0 uv1 ondisk = 0) v8 ==== 162+0+46 (crc 0 0 0) 0x7f1f1803f040 con 0x55574cb3b740 2023-10-05T19:28:32.675 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.666+0000 7f1f21db5e40 20 rados_obj.operate() r=0 bl.length=46 2023-10-05T19:28:32.675 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.666+0000 7f1f21db5e40 20 rados->read ofs=0 len=0 2023-10-05T19:28:32.675 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.666+0000 7f1f21db5e40 1 -- 172.21.15.167:0/369902812 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:77 5.1d 5:b8374eb4:::zone_info.9a9df965-6612-45d4-a45f-6e3c889bacc2:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x55574cb75350 con 0x55574cb3b740 2023-10-05T19:28:32.675 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.670+0000 7f1f200eb700 1 -- 172.21.15.167:0/369902812 <== osd.1 v2:172.21.15.167:6800/30035 27 ==== osd_op_reply(77 zone_info.9a9df965-6612-45d4-a45f-6e3c889bacc2 [read 0~874 out=874b] v0'0 uv2 ondisk = 0) v8 ==== 190+0+874 (crc 0 0 0) 0x7f1f1803f040 con 0x55574cb3b740 2023-10-05T19:28:32.675 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.670+0000 7f1f21db5e40 20 rados_obj.operate() r=0 bl.length=874 2023-10-05T19:28:32.675 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.670+0000 7f1f21db5e40 20 rados->read ofs=0 len=0 2023-10-05T19:28:32.675 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.670+0000 7f1f21db5e40 1 -- 172.21.15.167:0/369902812 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:78 5.f 5:f4c53578:::zonegroups_names.default:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x55574cb75840 con 0x55574cb3b740 2023-10-05T19:28:32.677 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.670+0000 7f1f200eb700 1 -- 172.21.15.167:0/369902812 <== osd.1 v2:172.21.15.167:6800/30035 28 ==== osd_op_reply(78 zonegroups_names.default [read 0~46 out=46b] v0'0 uv2 ondisk = 0) v8 ==== 168+0+46 (crc 0 0 0) 0x7f1f1803f040 con 0x55574cb3b740 2023-10-05T19:28:32.677 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.670+0000 7f1f21db5e40 20 rados_obj.operate() r=0 bl.length=46 2023-10-05T19:28:32.677 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.670+0000 7f1f21db5e40 20 rados->read ofs=0 len=0 2023-10-05T19:28:32.677 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.670+0000 7f1f21db5e40 1 -- 172.21.15.167:0/369902812 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:79 5.c 5:30d299a5:::zonegroup_info.4f1f719d-9648-417b-b10c-e5d939825419:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x55574cb75840 con 0x7f1f0800d3d0 2023-10-05T19:28:32.678 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.670+0000 7f1f200eb700 1 -- 172.21.15.167:0/369902812 <== osd.0 v2:172.21.15.118:6800/27690 20 ==== osd_op_reply(79 zonegroup_info.4f1f719d-9648-417b-b10c-e5d939825419 [read 0~362 out=362b] v0'0 uv1 ondisk = 0) v8 ==== 195+0+362 (crc 0 0 0) 0x7f1f18052040 con 0x7f1f0800d3d0 2023-10-05T19:28:32.678 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.670+0000 7f1f21db5e40 20 rados_obj.operate() r=0 bl.length=362 2023-10-05T19:28:32.678 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.670+0000 7f1f21db5e40 20 rados->read ofs=0 len=0 2023-10-05T19:28:32.678 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.670+0000 7f1f21db5e40 1 -- 172.21.15.167:0/369902812 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:80 5.3 5:c0ad8cbd:::region_map:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x55574cb75940 con 0x7f1f0800d3d0 2023-10-05T19:28:32.678 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.670+0000 7f1f200eb700 1 -- 172.21.15.167:0/369902812 <== osd.0 v2:172.21.15.118:6800/27690 21 ==== osd_op_reply(80 region_map [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 ==== 154+0+0 (crc 0 0 0) 0x7f1f18052040 con 0x7f1f0800d3d0 2023-10-05T19:28:32.678 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.670+0000 7f1f21db5e40 20 rados_obj.operate() r=-2 bl.length=0 2023-10-05T19:28:32.678 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.670+0000 7f1f21db5e40 20 zone default found 2023-10-05T19:28:32.678 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.670+0000 7f1f21db5e40 4 Realm: () 2023-10-05T19:28:32.678 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.670+0000 7f1f21db5e40 4 ZoneGroup: default (4f1f719d-9648-417b-b10c-e5d939825419) 2023-10-05T19:28:32.678 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.670+0000 7f1f21db5e40 4 Zone: default (9a9df965-6612-45d4-a45f-6e3c889bacc2) 2023-10-05T19:28:32.679 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.670+0000 7f1f21db5e40 10 cannot find current period zonegroup using local zonegroup configuration 2023-10-05T19:28:32.679 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.670+0000 7f1f21db5e40 20 zonegroup default 2023-10-05T19:28:32.679 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.670+0000 7f1f21db5e40 20 rados->read ofs=0 len=0 2023-10-05T19:28:32.679 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.670+0000 7f1f21db5e40 1 -- 172.21.15.167:0/369902812 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:81 5.3 5:c52100b6:::period_config.default:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x55574cb75940 con 0x7f1f0800d3d0 2023-10-05T19:28:32.680 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.674+0000 7f1f200eb700 1 -- 172.21.15.167:0/369902812 <== osd.0 v2:172.21.15.118:6800/27690 22 ==== osd_op_reply(81 period_config.default [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 ==== 165+0+0 (crc 0 0 0) 0x7f1f18052040 con 0x7f1f0800d3d0 2023-10-05T19:28:32.680 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.674+0000 7f1f21db5e40 20 rados_obj.operate() r=-2 bl.length=0 2023-10-05T19:28:32.680 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.674+0000 7f1f21db5e40 20 rados->read ofs=0 len=0 2023-10-05T19:28:32.681 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.674+0000 7f1f21db5e40 1 -- 172.21.15.167:0/369902812 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:82 6.2 6:4347d321:::bucket.sync-source-hints.:head [call version.read in=11b,read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x55574cb772c0 con 0x55574cb72240 2023-10-05T19:28:32.681 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.674+0000 7f1f1f8ea700 1 -- 172.21.15.167:0/369902812 <== osd.3 v2:172.21.15.167:6808/30036 20 ==== osd_op_reply(82 bucket.sync-source-hints. [call,read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 ==== 211+0+0 (crc 0 0 0) 0x7f1f10022020 con 0x55574cb72240 2023-10-05T19:28:32.681 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.674+0000 7f1f21db5e40 20 rados_obj.operate() r=-2 bl.length=0 2023-10-05T19:28:32.681 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.674+0000 7f1f21db5e40 20 rados->read ofs=0 len=0 2023-10-05T19:28:32.681 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.674+0000 7f1f21db5e40 1 -- 172.21.15.167:0/369902812 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:83 6.b 6:d467b91b:::bucket.sync-target-hints.:head [call version.read in=11b,read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x55574cb77670 con 0x55574cb72240 2023-10-05T19:28:32.681 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.674+0000 7f1f1f8ea700 1 -- 172.21.15.167:0/369902812 <== osd.3 v2:172.21.15.167:6808/30036 21 ==== osd_op_reply(83 bucket.sync-target-hints. [call,read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 ==== 211+0+0 (crc 0 0 0) 0x7f1f10022020 con 0x55574cb72240 2023-10-05T19:28:32.681 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.674+0000 7f1f21db5e40 20 rados_obj.operate() r=-2 bl.length=0 2023-10-05T19:28:32.681 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.674+0000 7f1f21db5e40 20 started sync module instance, tier type = 2023-10-05T19:28:32.681 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.674+0000 7f1f21db5e40 20 started zone id=9a9df965-6612-45d4-a45f-6e3c889bacc2 (name=default) with tier type = 2023-10-05T19:28:32.681 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.674+0000 7f1f21db5e40 1 -- 172.21.15.167:0/369902812 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:84 7.1f 7:f95f44c2:::notify.0:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x55574cb76980 con 0x7f1f10007030 2023-10-05T19:28:32.683 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.678+0000 7f1f1f0e9700 1 -- 172.21.15.167:0/369902812 <== osd.2 v2:172.21.15.167:6816/30037 13 ==== osd_op_reply(84 notify.0 [create] v30'111 uv111 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f1f08006070 con 0x7f1f10007030 2023-10-05T19:28:32.683 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.678+0000 7f1f21db5e40 1 -- 172.21.15.167:0/369902812 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:85 7.1f 7:f95f44c2:::notify.0:head [watch watch cookie 93833437605248] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x55574cb765b0 con 0x7f1f10007030 2023-10-05T19:28:32.683 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.678+0000 7f1f21db5e40 1 -- 172.21.15.167:0/369902812 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:86 7.0 7:05bf5b68:::notify.1:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x55574cb784a0 con 0x55574cb3b740 2023-10-05T19:28:32.686 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.678+0000 7f1f1f0e9700 1 -- 172.21.15.167:0/369902812 <== osd.2 v2:172.21.15.167:6816/30037 14 ==== osd_op_reply(85 notify.0 [watch watch cookie 93833437605248] v30'112 uv111 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f1f08006070 con 0x7f1f10007030 2023-10-05T19:28:32.686 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.682+0000 7f1f200eb700 1 -- 172.21.15.167:0/369902812 <== osd.1 v2:172.21.15.167:6800/30035 29 ==== osd_op_reply(86 notify.1 [create] v30'111 uv111 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f1f18052040 con 0x55574cb3b740 2023-10-05T19:28:32.686 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.682+0000 7f1f21db5e40 1 -- 172.21.15.167:0/369902812 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:87 7.0 7:05bf5b68:::notify.1:head [watch watch cookie 93833437608480] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x55574cb788a0 con 0x55574cb3b740 2023-10-05T19:28:32.686 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.682+0000 7f1f21db5e40 1 -- 172.21.15.167:0/369902812 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:88 7.15 7:a93a5511:::notify.2:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x55574cb79140 con 0x55574cb72240 2023-10-05T19:28:32.689 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.682+0000 7f1f200eb700 1 -- 172.21.15.167:0/369902812 <== osd.1 v2:172.21.15.167:6800/30035 30 ==== osd_op_reply(87 notify.1 [watch watch cookie 93833437608480] v30'112 uv111 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f1f18052040 con 0x55574cb3b740 2023-10-05T19:28:32.690 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.682+0000 7f1f1f8ea700 1 -- 172.21.15.167:0/369902812 <== osd.3 v2:172.21.15.167:6808/30036 22 ==== osd_op_reply(88 notify.2 [create] v30'111 uv111 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f1f10022020 con 0x55574cb72240 2023-10-05T19:28:32.690 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.686+0000 7f1f21db5e40 1 -- 172.21.15.167:0/369902812 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:89 7.15 7:a93a5511:::notify.2:head [watch watch cookie 93833437612192] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x55574cb78090 con 0x55574cb72240 2023-10-05T19:28:32.690 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.686+0000 7f1f21db5e40 1 -- 172.21.15.167:0/369902812 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:90 7.e 7:7759931f:::notify.3:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x55574cb79ed0 con 0x7f1f10007030 2023-10-05T19:28:32.693 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.686+0000 7f1f1f0e9700 1 -- 172.21.15.167:0/369902812 <== osd.2 v2:172.21.15.167:6816/30037 15 ==== osd_op_reply(90 notify.3 [create] v30'111 uv111 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f1f08006070 con 0x7f1f10007030 2023-10-05T19:28:32.693 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.686+0000 7f1f1f8ea700 1 -- 172.21.15.167:0/369902812 <== osd.3 v2:172.21.15.167:6808/30036 23 ==== osd_op_reply(89 notify.2 [watch watch cookie 93833437612192] v30'112 uv111 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f1f10022020 con 0x55574cb72240 2023-10-05T19:28:32.693 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.686+0000 7f1f21db5e40 1 -- 172.21.15.167:0/369902812 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:91 7.e 7:7759931f:::notify.3:head [watch watch cookie 93833437616608] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x55574cb7a770 con 0x7f1f10007030 2023-10-05T19:28:32.693 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.686+0000 7f1f21db5e40 1 -- 172.21.15.167:0/369902812 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:92 7.d 7:b4812045:::notify.4:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x55574cb7ad70 con 0x55574cb3b740 2023-10-05T19:28:32.695 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.690+0000 7f1f1f0e9700 1 -- 172.21.15.167:0/369902812 <== osd.2 v2:172.21.15.167:6816/30037 16 ==== osd_op_reply(91 notify.3 [watch watch cookie 93833437616608] v30'112 uv111 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f1f08006070 con 0x7f1f10007030 2023-10-05T19:28:32.695 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.690+0000 7f1f200eb700 1 -- 172.21.15.167:0/369902812 <== osd.1 v2:172.21.15.167:6800/30035 31 ==== osd_op_reply(92 notify.4 [create] v30'111 uv111 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f1f18052040 con 0x55574cb3b740 2023-10-05T19:28:32.695 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.690+0000 7f1f21db5e40 1 -- 172.21.15.167:0/369902812 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:93 7.d 7:b4812045:::notify.4:head [watch watch cookie 93833437618816] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x55574cb7b260 con 0x55574cb3b740 2023-10-05T19:28:32.695 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.690+0000 7f1f21db5e40 1 -- 172.21.15.167:0/369902812 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:94 7.3 7:c609908c:::notify.5:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x55574cb7bb00 con 0x7f1f0800d3d0 2023-10-05T19:28:32.698 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.690+0000 7f1f200eb700 1 -- 172.21.15.167:0/369902812 <== osd.0 v2:172.21.15.118:6800/27690 23 ==== osd_op_reply(94 notify.5 [create] v30'111 uv111 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f1f18052040 con 0x7f1f0800d3d0 2023-10-05T19:28:32.698 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.690+0000 7f1f21db5e40 1 -- 172.21.15.167:0/369902812 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:95 7.3 7:c609908c:::notify.5:head [watch watch cookie 93833437623440] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x55574cb7be50 con 0x7f1f0800d3d0 2023-10-05T19:28:32.698 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.690+0000 7f1f21db5e40 1 -- 172.21.15.167:0/369902812 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:96 7.14 7:2b04a3e9:::notify.6:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x55574cb7ca10 con 0x7f1f0800d3d0 2023-10-05T19:28:32.698 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.694+0000 7f1f200eb700 1 -- 172.21.15.167:0/369902812 <== osd.1 v2:172.21.15.167:6800/30035 32 ==== osd_op_reply(93 notify.4 [watch watch cookie 93833437618816] v30'112 uv111 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f1f18052040 con 0x55574cb3b740 2023-10-05T19:28:32.700 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.694+0000 7f1f200eb700 1 -- 172.21.15.167:0/369902812 <== osd.0 v2:172.21.15.118:6800/27690 24 ==== osd_op_reply(96 notify.6 [create] v30'111 uv111 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f1f18052040 con 0x7f1f0800d3d0 2023-10-05T19:28:32.700 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.694+0000 7f1f21db5e40 1 -- 172.21.15.167:0/369902812 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:97 7.14 7:2b04a3e9:::notify.6:head [watch watch cookie 93833437621616] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x55574cb7c9d0 con 0x7f1f0800d3d0 2023-10-05T19:28:32.700 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.694+0000 7f1f21db5e40 1 -- 172.21.15.167:0/369902812 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:98 7.9 7:93e5b521:::notify.7:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x55574cb7c9d0 con 0x7f1f0800d3d0 2023-10-05T19:28:32.701 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.694+0000 7f1f200eb700 1 -- 172.21.15.167:0/369902812 <== osd.0 v2:172.21.15.118:6800/27690 25 ==== osd_op_reply(95 notify.5 [watch watch cookie 93833437623440] v30'112 uv111 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f1f18052040 con 0x7f1f0800d3d0 2023-10-05T19:28:32.703 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.698+0000 7f1f200eb700 1 -- 172.21.15.167:0/369902812 <== osd.0 v2:172.21.15.118:6800/27690 26 ==== osd_op_reply(98 notify.7 [create] v30'111 uv111 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f1f18052040 con 0x7f1f0800d3d0 2023-10-05T19:28:32.703 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.698+0000 7f1f21db5e40 1 -- 172.21.15.167:0/369902812 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:99 7.9 7:93e5b521:::notify.7:head [watch watch cookie 93833437624544] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x55574cb7d3e0 con 0x7f1f0800d3d0 2023-10-05T19:28:32.703 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.698+0000 7f1f21db5e40 20 add_watcher() i=0 2023-10-05T19:28:32.703 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.698+0000 7f1f21db5e40 20 add_watcher() i=1 2023-10-05T19:28:32.703 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.698+0000 7f1f21db5e40 20 add_watcher() i=2 2023-10-05T19:28:32.703 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.698+0000 7f1f21db5e40 20 add_watcher() i=3 2023-10-05T19:28:32.703 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.698+0000 7f1f21db5e40 20 add_watcher() i=4 2023-10-05T19:28:32.703 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.698+0000 7f1f21db5e40 20 add_watcher() i=5 2023-10-05T19:28:32.703 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.698+0000 7f1f200eb700 1 -- 172.21.15.167:0/369902812 <== osd.0 v2:172.21.15.118:6800/27690 27 ==== osd_op_reply(97 notify.6 [watch watch cookie 93833437621616] v30'112 uv111 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f1f18052040 con 0x7f1f0800d3d0 2023-10-05T19:28:32.704 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.698+0000 7f1f21db5e40 20 add_watcher() i=6 2023-10-05T19:28:32.705 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.698+0000 7f1f200eb700 1 -- 172.21.15.167:0/369902812 <== osd.0 v2:172.21.15.118:6800/27690 28 ==== osd_op_reply(99 notify.7 [watch watch cookie 93833437624544] v30'112 uv111 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f1f18052040 con 0x7f1f0800d3d0 2023-10-05T19:28:32.705 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.698+0000 7f1f21db5e40 20 add_watcher() i=7 2023-10-05T19:28:32.705 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.698+0000 7f1f21db5e40 2 all 8 watchers are set, enabling cache 2023-10-05T19:28:32.705 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.698+0000 7f1f21db5e40 1 -- 172.21.15.167:0/369902812 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:100 6.4 6:22d26bf9:::data_loggenerations_metadata:head [call version.check_conds in=74b,call version.read in=11b,read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x55574cb80410 con 0x55574cb3b740 2023-10-05T19:28:32.706 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.698+0000 7f1f200eb700 1 -- 172.21.15.167:0/369902812 <== osd.1 v2:172.21.15.167:6800/30035 33 ==== osd_op_reply(100 data_loggenerations_metadata [call,call out=48b,read 0~28 out=28b] v0'0 uv1 ondisk = 0) v8 ==== 256+0+76 (crc 0 0 0) 0x7f1f18052040 con 0x55574cb3b740 2023-10-05T19:28:32.706 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.698+0000 7f1f21db5e40 1 -- 172.21.15.167:0/369902812 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:101 6.4 6:22d26bf9:::data_loggenerations_metadata:head [watch watch cookie 93833437630192] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x55574cb7d830 con 0x55574cb3b740 2023-10-05T19:28:32.708 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.702+0000 7f1f200eb700 1 -- 172.21.15.167:0/369902812 <== osd.1 v2:172.21.15.167:6800/30035 34 ==== osd_op_reply(101 data_loggenerations_metadata [watch watch cookie 93833437630192] v30'109 uv1 ondisk = 0) v8 ==== 172+0+0 (crc 0 0 0) 0x7f1f18052040 con 0x55574cb3b740 2023-10-05T19:28:32.708 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.702+0000 7f1ef17e2700 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2023-10-05T19:28:32.708 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.702+0000 7f1f21db5e40 20 check_secure_mon_conn(): auth registy supported: methods=[2] modes=[2,1] 2023-10-05T19:28:32.708 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.702+0000 7f1f21db5e40 20 check_secure_mon_conn(): mode 1 is insecure 2023-10-05T19:28:32.709 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.702+0000 7f1f21db5e40 1 Processor -- start 2023-10-05T19:28:32.709 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.702+0000 7f1f21db5e40 1 -- start start 2023-10-05T19:28:32.709 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.702+0000 7f1f21db5e40 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55574cb8a590 0x55574cb8a960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-05T19:28:32.709 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.702+0000 7f1f21db5e40 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x55574cb8aea0 con 0x55574cb8a590 2023-10-05T19:28:32.709 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.702+0000 7f1f1f0e9700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55574cb8a590 0x55574cb8a960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-05T19:28:32.710 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.702+0000 7f1f1f0e9700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55574cb8a590 0x55574cb8a960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.118:3300/0 says I am v2:172.21.15.167:58466/0 (socket says 172.21.15.167:58466) 2023-10-05T19:28:32.710 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.702+0000 7f1f1f0e9700 1 -- 172.21.15.167:0/3477290603 learned_addr learned my addr 172.21.15.167:0/3477290603 (peer_addr_for_me v2:172.21.15.167:0/0) 2023-10-05T19:28:32.710 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.702+0000 7f1f1f0e9700 1 -- 172.21.15.167:0/3477290603 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55574cb8afe0 con 0x55574cb8a590 2023-10-05T19:28:32.710 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.706+0000 7f1f1f0e9700 1 --2- 172.21.15.167:0/3477290603 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55574cb8a590 0x55574cb8a960 secure :-1 s=READY pgs=360 cs=0 l=1 rev1=1 crypto rx=0x55574cb6fd30 tx=0x7f1f08006000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8c97048ceafdd383 server_cookie=0 in_seq=0 out_seq=0 2023-10-05T19:28:32.710 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.706+0000 7f1ef0fe1700 1 -- 172.21.15.167:0/3477290603 <== mon.0 v2:172.21.15.118:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f1f08023040 con 0x55574cb8a590 2023-10-05T19:28:32.710 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.706+0000 7f1ef0fe1700 1 -- 172.21.15.167:0/3477290603 <== mon.0 v2:172.21.15.118:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1f08024040 con 0x55574cb8a590 2023-10-05T19:28:32.711 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.706+0000 7f1ef0fe1700 1 -- 172.21.15.167:0/3477290603 <== mon.0 v2:172.21.15.118:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f1f08013040 con 0x55574cb8a590 2023-10-05T19:28:32.712 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.706+0000 7f1f21db5e40 1 -- 172.21.15.167:0/3477290603 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55574cb8a590 msgr2=0x55574cb8a960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-05T19:28:32.712 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.706+0000 7f1f21db5e40 1 --2- 172.21.15.167:0/3477290603 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55574cb8a590 0x55574cb8a960 secure :-1 s=READY pgs=360 cs=0 l=1 rev1=1 crypto rx=0x55574cb6fd30 tx=0x7f1f08006000 comp rx=0 tx=0).stop 2023-10-05T19:28:32.712 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.706+0000 7f1f21db5e40 1 -- 172.21.15.167:0/3477290603 shutdown_connections 2023-10-05T19:28:32.712 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.706+0000 7f1f21db5e40 1 --2- 172.21.15.167:0/3477290603 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55574cb8a590 0x55574cb8a960 unknown :-1 s=CLOSED pgs=360 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-05T19:28:32.712 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.706+0000 7f1f21db5e40 1 -- 172.21.15.167:0/3477290603 >> 172.21.15.167:0/3477290603 conn(0x55574cb85950 msgr2=0x55574cb83b00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-05T19:28:32.712 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.706+0000 7f1f21db5e40 1 -- 172.21.15.167:0/3477290603 shutdown_connections 2023-10-05T19:28:32.712 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.706+0000 7f1f21db5e40 1 -- 172.21.15.167:0/3477290603 wait complete. 2023-10-05T19:28:32.713 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.706+0000 7f1f21db5e40 1 Processor -- start 2023-10-05T19:28:32.713 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.706+0000 7f1f21db5e40 1 -- start start 2023-10-05T19:28:32.713 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.706+0000 7f1f21db5e40 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55574cb8a590 0x55574cb90060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-05T19:28:32.713 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.706+0000 7f1f21db5e40 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x55574cb91060 con 0x55574cb8a590 2023-10-05T19:28:32.713 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.706+0000 7f1f1f0e9700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55574cb8a590 0x55574cb90060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-05T19:28:32.714 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.706+0000 7f1f1f0e9700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55574cb8a590 0x55574cb90060 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.118:3300/0 says I am v2:172.21.15.167:58476/0 (socket says 172.21.15.167:58476) 2023-10-05T19:28:32.714 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.706+0000 7f1f1f0e9700 1 -- 172.21.15.167:0/1910236425 learned_addr learned my addr 172.21.15.167:0/1910236425 (peer_addr_for_me v2:172.21.15.167:0/0) 2023-10-05T19:28:32.715 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.706+0000 7f1f1f0e9700 1 -- 172.21.15.167:0/1910236425 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55574ca87bd0 con 0x55574cb8a590 2023-10-05T19:28:32.715 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.710+0000 7f1f1f0e9700 1 --2- 172.21.15.167:0/1910236425 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55574cb8a590 0x55574cb90060 secure :-1 s=READY pgs=361 cs=0 l=1 rev1=1 crypto rx=0x7f1f0800b960 tx=0x7f1f08030000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-05T19:28:32.715 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.710+0000 7f1e6affd700 1 -- 172.21.15.167:0/1910236425 <== mon.0 v2:172.21.15.118:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f1f08023040 con 0x55574cb8a590 2023-10-05T19:28:32.715 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.710+0000 7f1e6affd700 1 -- 172.21.15.167:0/1910236425 <== mon.0 v2:172.21.15.118:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1f0803be30 con 0x55574cb8a590 2023-10-05T19:28:32.715 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.710+0000 7f1e6affd700 1 -- 172.21.15.167:0/1910236425 <== mon.0 v2:172.21.15.118:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f1f0801e420 con 0x55574cb8a590 2023-10-05T19:28:32.715 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.710+0000 7f1f21db5e40 1 -- 172.21.15.167:0/1910236425 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x55574cb90660 con 0x55574cb8a590 2023-10-05T19:28:32.715 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.710+0000 7f1f21db5e40 1 -- 172.21.15.167:0/1910236425 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55574cb90aa0 con 0x55574cb8a590 2023-10-05T19:28:32.716 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.710+0000 7f1e6affd700 1 -- 172.21.15.167:0/1910236425 <== mon.0 v2:172.21.15.118:3300/0 4 ==== mgrmap(e 5) v1 ==== 40329+0+0 (secure 0 0 0) 0x7f1f0801e580 con 0x55574cb8a590 2023-10-05T19:28:32.716 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.710+0000 7f1e6affd700 1 --2- 172.21.15.167:0/1910236425 >> [v2:172.21.15.167:6824/29889,v1:172.21.15.167:6825/29889] conn(0x7f1e60030bb0 0x7f1e60033060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-05T19:28:32.717 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.710+0000 7f1e6affd700 1 -- 172.21.15.167:0/1910236425 <== mon.0 v2:172.21.15.118:3300/0 5 ==== osd_map(30..30 src has 1..30) v4 ==== 6057+0+0 (secure 0 0 0) 0x7f1f0804fc60 con 0x55574cb8a590 2023-10-05T19:28:32.717 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.710+0000 7f1f200eb700 1 --2- 172.21.15.167:0/1910236425 >> [v2:172.21.15.167:6824/29889,v1:172.21.15.167:6825/29889] conn(0x7f1e60030bb0 0x7f1e60033060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-05T19:28:32.717 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.710+0000 7f1f21db5e40 5 note: GC not initialized 2023-10-05T19:28:32.717 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.710+0000 7f1f200eb700 1 --2- 172.21.15.167:0/1910236425 >> [v2:172.21.15.167:6824/29889,v1:172.21.15.167:6825/29889] conn(0x7f1e60030bb0 0x7f1e60033060 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x55574cb7c720 tx=0x7f1f1800aec0 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-05T19:28:32.719 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.714+0000 7f1e5ffff700 20 reqs_thread_entry: start 2023-10-05T19:28:32.720 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.714+0000 7f1e5dffb700 20 reqs_thread_entry: start 2023-10-05T19:28:32.720 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.714+0000 7f1f21db5e40 20 init_complete bucket index max shards: 11 2023-10-05T19:28:32.720 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.714+0000 7f1f21db5e40 10 rgw notify: Started notification manager with: 1 workers 2023-10-05T19:28:32.721 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.714+0000 7f1e4bfff700 20 rgw notify: INFO: next queues processing will happen at: Thu Oct 5 19:29:03 2023 2023-10-05T19:28:32.721 INFO:teuthology.orchestra.run.smithi167.stderr: 2023-10-05T19:28:32.721 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.714+0000 7f1e4b7fe700 20 reqs_thread_entry: start 2023-10-05T19:28:32.721 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.714+0000 7f1f21db5e40 20 get_system_obj_state: rctx=0x7ffffa46f318 obj=default.rgw.meta:root:myfoo state=0x55574ca140b0 s->prefetch_data=0 2023-10-05T19:28:32.721 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.714+0000 7f1f21db5e40 1 -- 172.21.15.167:0/369902812 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:102 8.9 8:91b71841:root::myfoo:head [call version.read in=11b,getxattrs,stat] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x55574cbf84d0 con 0x7f1f10007030 2023-10-05T19:28:32.722 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.714+0000 7f1f1f0e9700 1 -- 172.21.15.167:0/369902812 <== osd.2 v2:172.21.15.167:6816/30037 17 ==== osd_op_reply(102 myfoo [call out=48b,getxattrs out=75b,stat out=16b] v0'0 uv18 ondisk = 0) v8 ==== 233+0+139 (crc 0 0 0) 0x7f1f08006070 con 0x7f1f10007030 2023-10-05T19:28:32.722 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.714+0000 7f1f21db5e40 20 get_system_obj_state: s->obj_tag was set empty 2023-10-05T19:28:32.722 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.714+0000 7f1f21db5e40 20 Read xattr: user.rgw.idtag 2023-10-05T19:28:32.722 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.714+0000 7f1f21db5e40 20 rados->read ofs=0 len=0 2023-10-05T19:28:32.722 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.714+0000 7f1f21db5e40 1 -- 172.21.15.167:0/369902812 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:103 8.9 8:91b71841:root::myfoo:head [call version.check_conds in=122b,call version.read in=11b,read 0~0,getxattrs] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x55574cbfaf60 con 0x7f1f10007030 2023-10-05T19:28:32.723 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.718+0000 7f1f1f0e9700 1 -- 172.21.15.167:0/369902812 <== osd.2 v2:172.21.15.167:6816/30037 18 ==== osd_op_reply(103 myfoo [call,call out=48b,read 0~165 out=165b,getxattrs out=75b] v0'0 uv18 ondisk = 0) v8 ==== 275+0+288 (crc 0 0 0) 0x7f1f08006070 con 0x7f1f10007030 2023-10-05T19:28:32.723 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.718+0000 7f1f21db5e40 20 rados_obj.operate() r=0 bl.length=165 2023-10-05T19:28:32.723 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.718+0000 7f1f21db5e40 1 -- 172.21.15.167:0/369902812 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:104 7.1f 7:f95f44c2:::notify.0:head [watch unwatch cookie 93833437605248] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x55574cb507a0 con 0x7f1f10007030 2023-10-05T19:28:32.725 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.718+0000 7f1f1f0e9700 1 -- 172.21.15.167:0/369902812 <== osd.2 v2:172.21.15.167:6816/30037 19 ==== osd_op_reply(104 notify.0 [watch unwatch cookie 93833437605248] v30'113 uv111 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f1f08006070 con 0x7f1f10007030 2023-10-05T19:28:32.726 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.718+0000 7f1f21db5e40 20 remove_watcher() i=0 2023-10-05T19:28:32.726 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.718+0000 7f1f21db5e40 2 removed watcher, disabling cache 2023-10-05T19:28:32.726 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.718+0000 7f1f21db5e40 1 -- 172.21.15.167:0/369902812 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:105 7.0 7:05bf5b68:::notify.1:head [watch unwatch cookie 93833437608480] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x55574cb76d80 con 0x55574cb3b740 2023-10-05T19:28:32.728 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.722+0000 7f1f200eb700 1 -- 172.21.15.167:0/369902812 <== osd.1 v2:172.21.15.167:6800/30035 35 ==== osd_op_reply(105 notify.1 [watch unwatch cookie 93833437608480] v30'113 uv111 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f1f18052040 con 0x55574cb3b740 2023-10-05T19:28:32.729 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.722+0000 7f1f21db5e40 20 remove_watcher() i=1 2023-10-05T19:28:32.729 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.722+0000 7f1f21db5e40 1 -- 172.21.15.167:0/369902812 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:106 7.15 7:a93a5511:::notify.2:head [watch unwatch cookie 93833437612192] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x55574cb76d80 con 0x55574cb72240 2023-10-05T19:28:32.734 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.726+0000 7f1f1f8ea700 1 -- 172.21.15.167:0/369902812 <== osd.3 v2:172.21.15.167:6808/30036 24 ==== osd_op_reply(106 notify.2 [watch unwatch cookie 93833437612192] v30'113 uv111 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f1f10022020 con 0x55574cb72240 2023-10-05T19:28:32.735 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.726+0000 7f1f21db5e40 20 remove_watcher() i=2 2023-10-05T19:28:32.735 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.726+0000 7f1f21db5e40 1 -- 172.21.15.167:0/369902812 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:107 7.e 7:7759931f:::notify.3:head [watch unwatch cookie 93833437616608] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x55574cb788a0 con 0x7f1f10007030 2023-10-05T19:28:32.736 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.726+0000 7f1f1f0e9700 1 -- 172.21.15.167:0/369902812 <== osd.2 v2:172.21.15.167:6816/30037 20 ==== osd_op_reply(107 notify.3 [watch unwatch cookie 93833437616608] v30'113 uv111 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f1f08006070 con 0x7f1f10007030 2023-10-05T19:28:32.736 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.726+0000 7f1f21db5e40 20 remove_watcher() i=3 2023-10-05T19:28:32.736 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.726+0000 7f1f21db5e40 1 -- 172.21.15.167:0/369902812 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:108 7.d 7:b4812045:::notify.4:head [watch unwatch cookie 93833437618816] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x55574cb788a0 con 0x55574cb3b740 2023-10-05T19:28:32.737 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.730+0000 7f1f200eb700 1 -- 172.21.15.167:0/369902812 <== osd.1 v2:172.21.15.167:6800/30035 36 ==== osd_op_reply(108 notify.4 [watch unwatch cookie 93833437618816] v30'113 uv111 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f1f18052040 con 0x55574cb3b740 2023-10-05T19:28:32.737 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.730+0000 7f1f21db5e40 20 remove_watcher() i=4 2023-10-05T19:28:32.737 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.730+0000 7f1f21db5e40 1 -- 172.21.15.167:0/369902812 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:109 7.3 7:c609908c:::notify.5:head [watch unwatch cookie 93833437623440] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x55574cb7a280 con 0x7f1f0800d3d0 2023-10-05T19:28:32.739 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.734+0000 7f1f200eb700 1 -- 172.21.15.167:0/369902812 <== osd.0 v2:172.21.15.118:6800/27690 29 ==== osd_op_reply(109 notify.5 [watch unwatch cookie 93833437623440] v30'113 uv111 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f1f18052040 con 0x7f1f0800d3d0 2023-10-05T19:28:32.740 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.734+0000 7f1f21db5e40 20 remove_watcher() i=5 2023-10-05T19:28:32.740 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.734+0000 7f1f21db5e40 1 -- 172.21.15.167:0/369902812 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:110 7.14 7:2b04a3e9:::notify.6:head [watch unwatch cookie 93833437621616] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x55574cb788a0 con 0x7f1f0800d3d0 2023-10-05T19:28:32.742 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.738+0000 7f1f200eb700 1 -- 172.21.15.167:0/369902812 <== osd.0 v2:172.21.15.118:6800/27690 30 ==== osd_op_reply(110 notify.6 [watch unwatch cookie 93833437621616] v30'113 uv111 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f1f18052040 con 0x7f1f0800d3d0 2023-10-05T19:28:32.742 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.738+0000 7f1f21db5e40 20 remove_watcher() i=6 2023-10-05T19:28:32.743 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.738+0000 7f1f21db5e40 1 -- 172.21.15.167:0/369902812 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:111 7.9 7:93e5b521:::notify.7:head [watch unwatch cookie 93833437624544] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x55574cb788a0 con 0x7f1f0800d3d0 2023-10-05T19:28:32.745 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.738+0000 7f1f200eb700 1 -- 172.21.15.167:0/369902812 <== osd.0 v2:172.21.15.118:6800/27690 31 ==== osd_op_reply(111 notify.7 [watch unwatch cookie 93833437624544] v30'113 uv111 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f1f18052040 con 0x7f1f0800d3d0 2023-10-05T19:28:32.745 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.738+0000 7f1f21db5e40 20 remove_watcher() i=7 2023-10-05T19:28:32.747 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.742+0000 7f1ef17e2700 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2023-10-05T19:28:32.747 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.742+0000 7f1f21db5e40 1 -- 172.21.15.167:0/369902812 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:112 6.4 6:22d26bf9:::data_loggenerations_metadata:head [watch unwatch cookie 93833437630192] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x55574cb788a0 con 0x55574cb3b740 2023-10-05T19:28:32.751 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.742+0000 7f1f200eb700 1 -- 172.21.15.167:0/369902812 <== osd.1 v2:172.21.15.167:6800/30035 37 ==== osd_op_reply(112 data_loggenerations_metadata [watch unwatch cookie 93833437630192] v30'110 uv1 ondisk = 0) v8 ==== 172+0+0 (crc 0 0 0) 0x7f1f18052040 con 0x55574cb3b740 2023-10-05T19:28:32.751 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.746+0000 7f1f21db5e40 1 -- 172.21.15.167:0/369902812 >> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] conn(0x7f1f0800d3d0 msgr2=0x7f1f0800d820 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-05T19:28:32.751 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.746+0000 7f1f21db5e40 1 --2- 172.21.15.167:0/369902812 >> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] conn(0x7f1f0800d3d0 0x7f1f0800d820 crc :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-05T19:28:32.751 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.746+0000 7f1f21db5e40 1 -- 172.21.15.167:0/369902812 >> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] conn(0x55574cb3b740 msgr2=0x55574cb6e820 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-05T19:28:32.751 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.746+0000 7f1f21db5e40 1 --2- 172.21.15.167:0/369902812 >> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] conn(0x55574cb3b740 0x55574cb6e820 crc :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-05T19:28:32.751 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.746+0000 7f1f21db5e40 1 -- 172.21.15.167:0/369902812 >> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] conn(0x7f1f10007030 msgr2=0x7f1f100249c0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-05T19:28:32.752 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.746+0000 7f1f21db5e40 1 --2- 172.21.15.167:0/369902812 >> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] conn(0x7f1f10007030 0x7f1f100249c0 crc :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-05T19:28:32.752 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.746+0000 7f1f21db5e40 1 -- 172.21.15.167:0/369902812 >> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] conn(0x55574cb72240 msgr2=0x55574cb74620 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-05T19:28:32.752 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.746+0000 7f1f21db5e40 1 --2- 172.21.15.167:0/369902812 >> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] conn(0x55574cb72240 0x55574cb74620 crc :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-05T19:28:32.752 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.746+0000 7f1f21db5e40 1 -- 172.21.15.167:0/369902812 >> [v2:172.21.15.167:6824/29889,v1:172.21.15.167:6825/29889] conn(0x7f1f00030bf0 msgr2=0x7f1f000330a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-05T19:28:32.752 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.746+0000 7f1f21db5e40 1 --2- 172.21.15.167:0/369902812 >> [v2:172.21.15.167:6824/29889,v1:172.21.15.167:6825/29889] conn(0x7f1f00030bf0 0x7f1f000330a0 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7f1f08004170 tx=0x7f1f08001380 comp rx=0 tx=0).stop 2023-10-05T19:28:32.752 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.746+0000 7f1f21db5e40 1 -- 172.21.15.167:0/369902812 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55574cb3ae30 msgr2=0x55574cab0af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-05T19:28:32.752 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.746+0000 7f1f200eb700 1 -- 172.21.15.167:0/369902812 reap_dead start 2023-10-05T19:28:32.752 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.746+0000 7f1f21db5e40 1 --2- 172.21.15.167:0/369902812 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55574cb3ae30 0x55574cab0af0 secure :-1 s=READY pgs=359 cs=0 l=1 rev1=1 crypto rx=0x7f1f1000c010 tx=0x7f1f10003050 comp rx=0 tx=0).stop 2023-10-05T19:28:32.752 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.746+0000 7f1f21db5e40 1 -- 172.21.15.167:0/369902812 shutdown_connections 2023-10-05T19:28:32.752 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.746+0000 7f1f21db5e40 1 --2- 172.21.15.167:0/369902812 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55574cb3ae30 0x55574cab0af0 unknown :-1 s=CLOSED pgs=359 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-05T19:28:32.752 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.746+0000 7f1f21db5e40 1 -- 172.21.15.167:0/369902812 >> 172.21.15.167:0/369902812 conn(0x55574cb36850 msgr2=0x55574cb54570 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-05T19:28:32.753 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.746+0000 7f1f21db5e40 1 -- 172.21.15.167:0/369902812 shutdown_connections 2023-10-05T19:28:32.753 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.746+0000 7f1f21db5e40 1 -- 172.21.15.167:0/369902812 wait complete. 2023-10-05T19:28:32.754 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.746+0000 7f1f21db5e40 1 -- 172.21.15.167:0/1910236425 >> [v2:172.21.15.167:6824/29889,v1:172.21.15.167:6825/29889] conn(0x7f1e60030bb0 msgr2=0x7f1e60033060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-05T19:28:32.754 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.746+0000 7f1f21db5e40 1 --2- 172.21.15.167:0/1910236425 >> [v2:172.21.15.167:6824/29889,v1:172.21.15.167:6825/29889] conn(0x7f1e60030bb0 0x7f1e60033060 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x55574cb7c720 tx=0x7f1f1800aec0 comp rx=0 tx=0).stop 2023-10-05T19:28:32.754 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.746+0000 7f1f21db5e40 1 -- 172.21.15.167:0/1910236425 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55574cb8a590 msgr2=0x55574cb90060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-05T19:28:32.754 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.746+0000 7f1f21db5e40 1 --2- 172.21.15.167:0/1910236425 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55574cb8a590 0x55574cb90060 secure :-1 s=READY pgs=361 cs=0 l=1 rev1=1 crypto rx=0x7f1f0800b960 tx=0x7f1f08030000 comp rx=0 tx=0).stop 2023-10-05T19:28:32.754 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.746+0000 7f1f21db5e40 1 -- 172.21.15.167:0/1910236425 shutdown_connections 2023-10-05T19:28:32.755 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.746+0000 7f1f21db5e40 1 --2- 172.21.15.167:0/1910236425 >> [v2:172.21.15.167:6824/29889,v1:172.21.15.167:6825/29889] conn(0x7f1e60030bb0 0x7f1e60033060 unknown :-1 s=CLOSED pgs=127 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-05T19:28:32.755 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.746+0000 7f1f21db5e40 1 --2- 172.21.15.167:0/1910236425 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55574cb8a590 0x55574cb90060 unknown :-1 s=CLOSED pgs=361 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-05T19:28:32.755 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.746+0000 7f1f21db5e40 1 -- 172.21.15.167:0/1910236425 >> 172.21.15.167:0/1910236425 conn(0x55574cb85950 msgr2=0x55574cb83a50 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-05T19:28:32.755 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.746+0000 7f1f21db5e40 1 -- 172.21.15.167:0/1910236425 shutdown_connections 2023-10-05T19:28:32.755 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.746+0000 7f1f21db5e40 1 -- 172.21.15.167:0/1910236425 wait complete. 2023-10-05T19:28:32.755 INFO:teuthology.orchestra.run.smithi167.stdout:{ 2023-10-05T19:28:32.755 INFO:teuthology.orchestra.run.smithi167.stdout: "key": "bucket:myfoo", 2023-10-05T19:28:32.756 INFO:teuthology.orchestra.run.smithi167.stdout: "ver": { 2023-10-05T19:28:32.756 INFO:teuthology.orchestra.run.smithi167.stdout: "tag": "_wX2K8GKkMUD1AYqfrUWFEmF", 2023-10-05T19:28:32.756 INFO:teuthology.orchestra.run.smithi167.stdout: "ver": 2 2023-10-05T19:28:32.756 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2023-10-05T19:28:32.756 INFO:teuthology.orchestra.run.smithi167.stdout: "mtime": "2023-10-05T19:28:32.224781Z", 2023-10-05T19:28:32.756 INFO:teuthology.orchestra.run.smithi167.stdout: "data": { 2023-10-05T19:28:32.756 INFO:teuthology.orchestra.run.smithi167.stdout: "bucket": { 2023-10-05T19:28:32.756 INFO:teuthology.orchestra.run.smithi167.stdout: "name": "myfoo", 2023-10-05T19:28:32.757 INFO:teuthology.orchestra.run.smithi167.stdout: "marker": "9a9df965-6612-45d4-a45f-6e3c889bacc2.4223.1", 2023-10-05T19:28:32.757 INFO:teuthology.orchestra.run.smithi167.stdout: "bucket_id": "9a9df965-6612-45d4-a45f-6e3c889bacc2.4223.1", 2023-10-05T19:28:32.757 INFO:teuthology.orchestra.run.smithi167.stdout: "tenant": "", 2023-10-05T19:28:32.757 INFO:teuthology.orchestra.run.smithi167.stdout: "explicit_placement": { 2023-10-05T19:28:32.757 INFO:teuthology.orchestra.run.smithi167.stdout: "data_pool": "", 2023-10-05T19:28:32.757 INFO:teuthology.orchestra.run.smithi167.stdout: "data_extra_pool": "", 2023-10-05T19:28:32.757 INFO:teuthology.orchestra.run.smithi167.stdout: "index_pool": "" 2023-10-05T19:28:32.757 INFO:teuthology.orchestra.run.smithi167.stdout: } 2023-10-05T19:28:32.758 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2023-10-05T19:28:32.758 INFO:teuthology.orchestra.run.smithi167.stdout: "owner": "foo", 2023-10-05T19:28:32.758 INFO:teuthology.orchestra.run.smithi167.stdout: "creation_time": "2023-10-05T19:28:30.250836Z", 2023-10-05T19:28:32.758 INFO:teuthology.orchestra.run.smithi167.stdout: "linked": "false", 2023-10-05T19:28:32.758 INFO:teuthology.orchestra.run.smithi167.stdout: "has_bucket_info": "false" 2023-10-05T19:28:32.758 INFO:teuthology.orchestra.run.smithi167.stdout: } 2023-10-05T19:28:32.758 INFO:teuthology.orchestra.run.smithi167.stdout:} 2023-10-05T19:28:32.759 DEBUG:tasks.util.rgw: json result: {'key': 'bucket:myfoo', 'ver': {'tag': '_wX2K8GKkMUD1AYqfrUWFEmF', 'ver': 2}, 'mtime': '2023-10-05T19:28:32.224781Z', 'data': {'bucket': {'name': 'myfoo', 'marker': '9a9df965-6612-45d4-a45f-6e3c889bacc2.4223.1', 'bucket_id': '9a9df965-6612-45d4-a45f-6e3c889bacc2.4223.1', 'tenant': '', 'explicit_placement': {'data_pool': '', 'data_extra_pool': '', 'index_pool': ''}}, 'owner': 'foo', 'creation_time': '2023-10-05T19:28:30.250836Z', 'linked': 'false', 'has_bucket_info': 'false'}} 2023-10-05T19:28:32.759 INFO:tasks.util.rgw:rgwadmin: client.0 : ['bucket', 'link', '--uid', 'fud', '--bucket', 'myfoo', '--bucket-id', '9a9df965-6612-45d4-a45f-6e3c889bacc2.4223.1'] 2023-10-05T19:28:32.759 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'ceph', 'bucket', 'link', '--uid', 'fud', '--bucket', 'myfoo', '--bucket-id', '9a9df965-6612-45d4-a45f-6e3c889bacc2.4223.1'] 2023-10-05T19:28:32.760 DEBUG:teuthology.orchestra.run.smithi167:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster ceph bucket link --uid fud --bucket myfoo --bucket-id 9a9df965-6612-45d4-a45f-6e3c889bacc2.4223.1 2023-10-05T19:28:32.845 INFO:teuthology.orchestra.run.smithi167.stderr:ignoring --setuser ceph since I am not root 2023-10-05T19:28:32.845 INFO:teuthology.orchestra.run.smithi167.stderr:ignoring --setgroup ceph since I am not root 2023-10-05T19:28:32.846 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.838+0000 7f3999e0de40 1 Processor -- start 2023-10-05T19:28:32.846 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.838+0000 7f3999e0de40 1 -- start start 2023-10-05T19:28:32.846 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.842+0000 7f3999e0de40 1 --2- >> v2:172.21.15.118:3300/0 conn(0x557e6294c8b0 0x557e629e0f00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-05T19:28:32.846 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.842+0000 7f3999e0de40 1 -- --> v1:172.21.15.118:6789/0 -- auth(proto 0 26 bytes epoch 0) v1 -- 0x557e62924bd0 con 0x557e6294dbc0 2023-10-05T19:28:32.847 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.842+0000 7f3999e0de40 1 -- --> v2:172.21.15.118:3300/0 -- mon_getmap magic: 0 v1 -- 0x557e628e1ee0 con 0x557e6294c8b0 2023-10-05T19:28:32.851 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.842+0000 7f3997141700 1 --2- >> v2:172.21.15.118:3300/0 conn(0x557e6294c8b0 0x557e629e0f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-05T19:28:32.851 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.842+0000 7f3997942700 1 --1- >> v1:172.21.15.118:6789/0 conn(0x557e6294dbc0 0x557e6294c1a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.118:6789/0 says I am v1:172.21.15.167:56638/0 (socket says 172.21.15.167:56638) 2023-10-05T19:28:32.852 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.842+0000 7f3997141700 1 --2- >> v2:172.21.15.118:3300/0 conn(0x557e6294c8b0 0x557e629e0f00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.118:3300/0 says I am v2:172.21.15.167:58482/0 (socket says 172.21.15.167:58482) 2023-10-05T19:28:32.852 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.842+0000 7f3997141700 1 -- 172.21.15.167:0/1014754196 learned_addr learned my addr 172.21.15.167:0/1014754196 (peer_addr_for_me v2:172.21.15.167:0/0) 2023-10-05T19:28:32.852 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.842+0000 7f3997141700 1 -- 172.21.15.167:0/1014754196 >> v1:172.21.15.118:6789/0 conn(0x557e6294dbc0 legacy=0x557e6294c1a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-05T19:28:32.852 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.842+0000 7f3996940700 1 -- 172.21.15.167:0/1014754196 <== mon.0 v1:172.21.15.118:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3204146930 0 0) 0x557e62924bd0 con 0x557e6294dbc0 2023-10-05T19:28:32.852 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.842+0000 7f3997141700 1 -- 172.21.15.167:0/1014754196 --> v2:172.21.15.118:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x557e6294e050 con 0x557e6294c8b0 2023-10-05T19:28:32.852 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.842+0000 7f3997141700 1 --2- 172.21.15.167:0/1014754196 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x557e6294c8b0 0x557e629e0f00 secure :-1 s=READY pgs=363 cs=0 l=1 rev1=1 crypto rx=0x7f398000a650 tx=0x7f3980005b10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=754392037941c00a server_cookie=0 in_seq=0 out_seq=0 2023-10-05T19:28:32.852 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.842+0000 7f3996940700 1 -- 172.21.15.167:0/1014754196 <== mon.0 v2:172.21.15.118:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f3980008760 con 0x557e6294c8b0 2023-10-05T19:28:32.853 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.842+0000 7f3996940700 1 -- 172.21.15.167:0/1014754196 <== mon.0 v2:172.21.15.118:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f39800088c0 con 0x557e6294c8b0 2023-10-05T19:28:32.853 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.842+0000 7f3996940700 1 -- 172.21.15.167:0/1014754196 <== mon.0 v2:172.21.15.118:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f3980008bb0 con 0x557e6294c8b0 2023-10-05T19:28:32.853 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.842+0000 7f3999e0de40 1 -- 172.21.15.167:0/1014754196 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x557e6294c8b0 msgr2=0x557e629e0f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-05T19:28:32.853 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.842+0000 7f3999e0de40 1 --2- 172.21.15.167:0/1014754196 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x557e6294c8b0 0x557e629e0f00 secure :-1 s=READY pgs=363 cs=0 l=1 rev1=1 crypto rx=0x7f398000a650 tx=0x7f3980005b10 comp rx=0 tx=0).stop 2023-10-05T19:28:32.853 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.842+0000 7f3999e0de40 1 -- 172.21.15.167:0/1014754196 shutdown_connections 2023-10-05T19:28:32.853 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.842+0000 7f3999e0de40 1 --2- 172.21.15.167:0/1014754196 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x557e6294c8b0 0x557e629e0f00 unknown :-1 s=CLOSED pgs=363 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-05T19:28:32.853 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.842+0000 7f3999e0de40 1 -- 172.21.15.167:0/1014754196 >> 172.21.15.167:0/1014754196 conn(0x557e629d37f0 msgr2=0x557e629d5c00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-05T19:28:32.853 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.842+0000 7f3999e0de40 1 -- 172.21.15.167:0/1014754196 shutdown_connections 2023-10-05T19:28:32.853 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.842+0000 7f3999e0de40 1 -- 172.21.15.167:0/1014754196 wait complete. 2023-10-05T19:28:32.854 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.842+0000 7f3999e0de40 1 Processor -- start 2023-10-05T19:28:32.854 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.846+0000 7f3999e0de40 1 -- start start 2023-10-05T19:28:32.854 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.846+0000 7f3999e0de40 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x557e6294c8b0 0x557e629f8fb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-05T19:28:32.854 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.846+0000 7f3999e0de40 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x557e628a5e30 con 0x557e6294c8b0 2023-10-05T19:28:32.854 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.846+0000 7f3997942700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x557e6294c8b0 0x557e629f8fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-05T19:28:32.854 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.846+0000 7f3997942700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x557e6294c8b0 0x557e629f8fb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.118:3300/0 says I am v2:172.21.15.167:58498/0 (socket says 172.21.15.167:58498) 2023-10-05T19:28:32.854 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.846+0000 7f3997942700 1 -- 172.21.15.167:0/2041238904 learned_addr learned my addr 172.21.15.167:0/2041238904 (peer_addr_for_me v2:172.21.15.167:0/0) 2023-10-05T19:28:32.854 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.846+0000 7f3997942700 1 -- 172.21.15.167:0/2041238904 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3980005db0 con 0x557e6294c8b0 2023-10-05T19:28:32.854 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.846+0000 7f3997942700 1 --2- 172.21.15.167:0/2041238904 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x557e6294c8b0 0x557e629f8fb0 secure :-1 s=READY pgs=364 cs=0 l=1 rev1=1 crypto rx=0x7f3988001630 tx=0x7f39880069b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e548a90a7584f9dc server_cookie=0 in_seq=0 out_seq=0 2023-10-05T19:28:32.858 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.846+0000 7f399593e700 1 -- 172.21.15.167:0/2041238904 <== mon.0 v2:172.21.15.118:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f3988006320 con 0x557e6294c8b0 2023-10-05T19:28:32.858 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.846+0000 7f399593e700 1 -- 172.21.15.167:0/2041238904 <== mon.0 v2:172.21.15.118:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3988013420 con 0x557e6294c8b0 2023-10-05T19:28:32.858 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.846+0000 7f399593e700 1 -- 172.21.15.167:0/2041238904 <== mon.0 v2:172.21.15.118:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f39880135d0 con 0x557e6294c8b0 2023-10-05T19:28:32.858 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.846+0000 7f3999e0de40 1 -- 172.21.15.167:0/2041238904 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x557e6294c8b0 msgr2=0x557e629f8fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-05T19:28:32.858 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.846+0000 7f3999e0de40 1 --2- 172.21.15.167:0/2041238904 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x557e6294c8b0 0x557e629f8fb0 secure :-1 s=READY pgs=364 cs=0 l=1 rev1=1 crypto rx=0x7f3988001630 tx=0x7f39880069b0 comp rx=0 tx=0).stop 2023-10-05T19:28:32.858 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.846+0000 7f3999e0de40 1 -- 172.21.15.167:0/2041238904 shutdown_connections 2023-10-05T19:28:32.858 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.846+0000 7f3999e0de40 1 --2- 172.21.15.167:0/2041238904 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x557e6294c8b0 0x557e629f8fb0 unknown :-1 s=CLOSED pgs=364 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-05T19:28:32.859 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.846+0000 7f3999e0de40 1 -- 172.21.15.167:0/2041238904 >> 172.21.15.167:0/2041238904 conn(0x557e629d37f0 msgr2=0x557e629f4af0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-05T19:28:32.859 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.846+0000 7f3999e0de40 1 -- 172.21.15.167:0/2041238904 shutdown_connections 2023-10-05T19:28:32.859 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.846+0000 7f3999e0de40 1 -- 172.21.15.167:0/2041238904 wait complete. 2023-10-05T19:28:32.859 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.846+0000 7f3999e0de40 1 Processor -- start 2023-10-05T19:28:32.859 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.846+0000 7f3999e0de40 1 -- start start 2023-10-05T19:28:32.859 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.846+0000 7f3999e0de40 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x557e6294c8b0 0x557e629ffd90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-05T19:28:32.859 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.846+0000 7f3999e0de40 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x557e62878ed0 con 0x557e6294c8b0 2023-10-05T19:28:32.859 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.846+0000 7f3997942700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x557e6294c8b0 0x557e629ffd90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-05T19:28:32.859 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.846+0000 7f3997942700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x557e6294c8b0 0x557e629ffd90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.118:3300/0 says I am v2:172.21.15.167:58510/0 (socket says 172.21.15.167:58510) 2023-10-05T19:28:32.860 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.846+0000 7f3997942700 1 -- 172.21.15.167:0/906877954 learned_addr learned my addr 172.21.15.167:0/906877954 (peer_addr_for_me v2:172.21.15.167:0/0) 2023-10-05T19:28:32.860 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.850+0000 7f3997942700 1 -- 172.21.15.167:0/906877954 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3988006c90 con 0x557e6294c8b0 2023-10-05T19:28:32.860 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.850+0000 7f3997942700 1 --2- 172.21.15.167:0/906877954 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x557e6294c8b0 0x557e629ffd90 secure :-1 s=READY pgs=365 cs=0 l=1 rev1=1 crypto rx=0x7f3988000f80 tx=0x7f3988014000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-05T19:28:32.860 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.850+0000 7f398ffff700 1 -- 172.21.15.167:0/906877954 <== mon.0 v2:172.21.15.118:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f3988006320 con 0x557e6294c8b0 2023-10-05T19:28:32.860 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.850+0000 7f398ffff700 1 -- 172.21.15.167:0/906877954 <== mon.0 v2:172.21.15.118:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3988013590 con 0x557e6294c8b0 2023-10-05T19:28:32.860 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.850+0000 7f3999e0de40 1 -- 172.21.15.167:0/906877954 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x557e62a002d0 con 0x557e6294c8b0 2023-10-05T19:28:32.860 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.850+0000 7f398ffff700 1 -- 172.21.15.167:0/906877954 <== mon.0 v2:172.21.15.118:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f3988013e00 con 0x557e6294c8b0 2023-10-05T19:28:32.860 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.850+0000 7f3999e0de40 1 -- 172.21.15.167:0/906877954 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x557e62a00710 con 0x557e6294c8b0 2023-10-05T19:28:32.860 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.850+0000 7f3999e0de40 20 rados->read ofs=0 len=0 2023-10-05T19:28:32.861 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.850+0000 7f398ffff700 1 -- 172.21.15.167:0/906877954 <== mon.0 v2:172.21.15.118:3300/0 4 ==== mgrmap(e 5) v1 ==== 40329+0+0 (secure 0 0 0) 0x7f3988028460 con 0x557e6294c8b0 2023-10-05T19:28:32.861 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.850+0000 7f398ffff700 1 --2- 172.21.15.167:0/906877954 >> [v2:172.21.15.167:6824/29889,v1:172.21.15.167:6825/29889] conn(0x7f3978030c50 0x7f3978033100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-05T19:28:32.861 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.850+0000 7f398ffff700 1 -- 172.21.15.167:0/906877954 <== mon.0 v2:172.21.15.118:3300/0 5 ==== osd_map(30..30 src has 1..30) v4 ==== 6057+0+0 (secure 0 0 0) 0x7f39880452e0 con 0x557e6294c8b0 2023-10-05T19:28:32.863 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.854+0000 7f3997141700 1 --2- 172.21.15.167:0/906877954 >> [v2:172.21.15.167:6824/29889,v1:172.21.15.167:6825/29889] conn(0x7f3978030c50 0x7f3978033100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-05T19:28:32.863 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.854+0000 7f3999e0de40 1 --2- 172.21.15.167:0/906877954 >> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] conn(0x557e6294dbc0 0x557e62a0b910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-05T19:28:32.863 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.854+0000 7f3999e0de40 1 -- 172.21.15.167:0/906877954 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:1 5.12 5:49953fa1:::default.realm:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x557e62a0bf30 con 0x557e6294dbc0 2023-10-05T19:28:32.864 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.854+0000 7f3998143700 1 --2- 172.21.15.167:0/906877954 >> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] conn(0x557e6294dbc0 0x557e62a0b910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-05T19:28:32.864 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.854+0000 7f3997141700 1 --2- 172.21.15.167:0/906877954 >> [v2:172.21.15.167:6824/29889,v1:172.21.15.167:6825/29889] conn(0x7f3978030c50 0x7f3978033100 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7f39800030a0 tx=0x7f3980015000 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-05T19:28:32.864 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.854+0000 7f3998143700 1 --2- 172.21.15.167:0/906877954 >> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] conn(0x557e6294dbc0 0x557e62a0b910 crc :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-05T19:28:32.864 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.854+0000 7f3998143700 1 -- 172.21.15.167:0/906877954 <== osd.1 v2:172.21.15.167:6800/30035 1 ==== osd_op_reply(1 default.realm [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 ==== 157+0+0 (crc 0 0 0) 0x7f399000a500 con 0x557e6294dbc0 2023-10-05T19:28:32.864 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.854+0000 7f3999e0de40 20 rados_obj.operate() r=-2 bl.length=0 2023-10-05T19:28:32.864 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.854+0000 7f3999e0de40 20 realm 2023-10-05T19:28:32.864 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.854+0000 7f3999e0de40 20 rados->read ofs=0 len=0 2023-10-05T19:28:32.865 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.854+0000 7f3999e0de40 1 -- 172.21.15.167:0/906877954 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:2 5.12 5:49953fa1:::default.realm:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x557e62a08f30 con 0x557e6294dbc0 2023-10-05T19:28:32.865 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.854+0000 7f3998143700 1 -- 172.21.15.167:0/906877954 <== osd.1 v2:172.21.15.167:6800/30035 2 ==== osd_op_reply(2 default.realm [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 ==== 157+0+0 (crc 0 0 0) 0x7f399000a500 con 0x557e6294dbc0 2023-10-05T19:28:32.865 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.854+0000 7f3999e0de40 20 rados_obj.operate() r=-2 bl.length=0 2023-10-05T19:28:32.865 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.854+0000 7f3999e0de40 4 RGWPeriod::init failed to init realm id : (2) No such file or directory 2023-10-05T19:28:32.865 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.854+0000 7f3999e0de40 20 rados->read ofs=0 len=0 2023-10-05T19:28:32.865 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.854+0000 7f3999e0de40 1 -- 172.21.15.167:0/906877954 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:3 5.12 5:49953fa1:::default.realm:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x557e62a0ca10 con 0x557e6294dbc0 2023-10-05T19:28:32.866 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.854+0000 7f3998143700 1 -- 172.21.15.167:0/906877954 <== osd.1 v2:172.21.15.167:6800/30035 3 ==== osd_op_reply(3 default.realm [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 ==== 157+0+0 (crc 0 0 0) 0x7f399000a500 con 0x557e6294dbc0 2023-10-05T19:28:32.866 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.854+0000 7f3999e0de40 20 rados_obj.operate() r=-2 bl.length=0 2023-10-05T19:28:32.866 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.854+0000 7f3999e0de40 20 rados->read ofs=0 len=0 2023-10-05T19:28:32.866 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.854+0000 7f3999e0de40 1 -- 172.21.15.167:0/906877954 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:4 5.f 5:f43b6ece:::zone_names.default:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x557e62a0c5f0 con 0x557e6294dbc0 2023-10-05T19:28:32.866 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.854+0000 7f3998143700 1 -- 172.21.15.167:0/906877954 <== osd.1 v2:172.21.15.167:6800/30035 4 ==== osd_op_reply(4 zone_names.default [read 0~46 out=46b] v0'0 uv1 ondisk = 0) v8 ==== 162+0+46 (crc 0 0 0) 0x7f399000a500 con 0x557e6294dbc0 2023-10-05T19:28:32.867 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.854+0000 7f3999e0de40 20 rados_obj.operate() r=0 bl.length=46 2023-10-05T19:28:32.867 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.854+0000 7f3999e0de40 20 rados->read ofs=0 len=0 2023-10-05T19:28:32.867 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.854+0000 7f3999e0de40 1 -- 172.21.15.167:0/906877954 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:5 5.1d 5:b8374eb4:::zone_info.9a9df965-6612-45d4-a45f-6e3c889bacc2:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x557e62a0d5b0 con 0x557e6294dbc0 2023-10-05T19:28:32.867 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.858+0000 7f3998143700 1 -- 172.21.15.167:0/906877954 <== osd.1 v2:172.21.15.167:6800/30035 5 ==== osd_op_reply(5 zone_info.9a9df965-6612-45d4-a45f-6e3c889bacc2 [read 0~874 out=874b] v0'0 uv2 ondisk = 0) v8 ==== 190+0+874 (crc 0 0 0) 0x7f399000a500 con 0x557e6294dbc0 2023-10-05T19:28:32.867 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.858+0000 7f3999e0de40 20 rados_obj.operate() r=0 bl.length=874 2023-10-05T19:28:32.867 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.858+0000 7f3999e0de40 20 searching for the correct realm 2023-10-05T19:28:32.867 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.858+0000 7f3999e0de40 1 --2- 172.21.15.167:0/906877954 >> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] conn(0x557e62a0f4c0 0x557e62a118d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-05T19:28:32.868 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.858+0000 7f3999e0de40 1 -- 172.21.15.167:0/906877954 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:6 5.0 5:00000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x557e62a11ee0 con 0x557e62a0f4c0 2023-10-05T19:28:32.868 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.858+0000 7f3997942700 1 --2- 172.21.15.167:0/906877954 >> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] conn(0x557e62a0f4c0 0x557e62a118d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-05T19:28:32.868 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.858+0000 7f3997942700 1 --2- 172.21.15.167:0/906877954 >> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] conn(0x557e62a0f4c0 0x557e62a118d0 crc :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-05T19:28:32.870 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.858+0000 7f3997942700 1 -- 172.21.15.167:0/906877954 <== osd.3 v2:172.21.15.167:6808/30036 1 ==== osd_op_reply(6 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f3988024020 con 0x557e62a0f4c0 2023-10-05T19:28:32.870 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.858+0000 7f3997942700 1 --2- 172.21.15.167:0/906877954 >> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] conn(0x7f398801d350 0x7f3988043850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-05T19:28:32.871 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.858+0000 7f3997942700 1 -- 172.21.15.167:0/906877954 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:7 5.10 5:08000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x557e62a11ee0 con 0x7f398801d350 2023-10-05T19:28:32.871 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.858+0000 7f3997141700 1 --2- 172.21.15.167:0/906877954 >> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] conn(0x7f398801d350 0x7f3988043850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-05T19:28:32.871 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.862+0000 7f3997141700 1 --2- 172.21.15.167:0/906877954 >> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] conn(0x7f398801d350 0x7f3988043850 crc :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-05T19:28:32.871 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.862+0000 7f3997141700 1 -- 172.21.15.167:0/906877954 <== osd.2 v2:172.21.15.167:6816/30037 1 ==== osd_op_reply(7 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f3980012450 con 0x7f398801d350 2023-10-05T19:28:32.871 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.862+0000 7f3997141700 1 --2- 172.21.15.167:0/906877954 >> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] conn(0x7f3980007030 0x7f398001c920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-05T19:28:32.871 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.862+0000 7f3997141700 1 -- 172.21.15.167:0/906877954 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:8 5.8 5:10000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x557e62a11ee0 con 0x7f3980007030 2023-10-05T19:28:32.871 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.862+0000 7f3998143700 1 --2- 172.21.15.167:0/906877954 >> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] conn(0x7f3980007030 0x7f398001c920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-05T19:28:32.871 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.862+0000 7f3998143700 1 --2- 172.21.15.167:0/906877954 >> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] conn(0x7f3980007030 0x7f398001c920 crc :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-05T19:28:32.872 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.862+0000 7f3998143700 1 -- 172.21.15.167:0/906877954 <== osd.0 v2:172.21.15.118:6800/27690 1 ==== osd_op_reply(8 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f3990014040 con 0x7f3980007030 2023-10-05T19:28:32.872 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.862+0000 7f3998143700 1 -- 172.21.15.167:0/906877954 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:9 5.18 5:18000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x557e62a11ee0 con 0x557e6294dbc0 2023-10-05T19:28:32.872 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.862+0000 7f3998143700 1 -- 172.21.15.167:0/906877954 <== osd.1 v2:172.21.15.167:6800/30035 6 ==== osd_op_reply(9 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f3990014040 con 0x557e6294dbc0 2023-10-05T19:28:32.872 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.862+0000 7f3998143700 1 -- 172.21.15.167:0/906877954 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:10 5.4 5:20000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x557e62a11ee0 con 0x557e6294dbc0 2023-10-05T19:28:32.872 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.866+0000 7f3998143700 1 -- 172.21.15.167:0/906877954 <== osd.1 v2:172.21.15.167:6800/30035 7 ==== osd_op_reply(10 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f3990014040 con 0x557e6294dbc0 2023-10-05T19:28:32.872 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.866+0000 7f3998143700 1 -- 172.21.15.167:0/906877954 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:11 5.14 5:28000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x557e62a11ee0 con 0x7f3980007030 2023-10-05T19:28:32.873 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.866+0000 7f3998143700 1 -- 172.21.15.167:0/906877954 <== osd.0 v2:172.21.15.118:6800/27690 2 ==== osd_op_reply(11 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f3990014040 con 0x7f3980007030 2023-10-05T19:28:32.874 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.866+0000 7f3998143700 1 -- 172.21.15.167:0/906877954 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:12 5.c 5:30000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x557e62a11ee0 con 0x7f3980007030 2023-10-05T19:28:32.874 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.866+0000 7f3998143700 1 -- 172.21.15.167:0/906877954 <== osd.0 v2:172.21.15.118:6800/27690 3 ==== osd_op_reply(12 [pgnls start_epoch 30 out=112b] v23'1 uv1 ondisk = 1) v8 ==== 144+0+112 (crc 0 0 0) 0x7f3990014040 con 0x7f3980007030 2023-10-05T19:28:32.874 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.866+0000 7f3998143700 1 -- 172.21.15.167:0/906877954 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:13 5.1c 5:38000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x557e62a11ee0 con 0x7f398801d350 2023-10-05T19:28:32.874 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.866+0000 7f3997141700 1 -- 172.21.15.167:0/906877954 <== osd.2 v2:172.21.15.167:6816/30037 2 ==== osd_op_reply(13 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f3980012450 con 0x7f398801d350 2023-10-05T19:28:32.874 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.866+0000 7f3997141700 1 -- 172.21.15.167:0/906877954 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:14 5.2 5:40000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x557e62a11ee0 con 0x557e62a0f4c0 2023-10-05T19:28:32.874 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.866+0000 7f3997942700 1 -- 172.21.15.167:0/906877954 <== osd.3 v2:172.21.15.167:6808/30036 2 ==== osd_op_reply(14 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f3988024020 con 0x557e62a0f4c0 2023-10-05T19:28:32.874 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.866+0000 7f3997942700 1 -- 172.21.15.167:0/906877954 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:15 5.12 5:48000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x557e62a11ee0 con 0x557e6294dbc0 2023-10-05T19:28:32.875 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.870+0000 7f3998143700 1 -- 172.21.15.167:0/906877954 <== osd.1 v2:172.21.15.167:6800/30035 8 ==== osd_op_reply(15 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f3990014040 con 0x557e6294dbc0 2023-10-05T19:28:32.875 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.870+0000 7f3998143700 1 -- 172.21.15.167:0/906877954 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:16 5.a 5:50000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x557e62a11ee0 con 0x7f3980007030 2023-10-05T19:28:32.876 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.870+0000 7f3998143700 1 -- 172.21.15.167:0/906877954 <== osd.0 v2:172.21.15.118:6800/27690 4 ==== osd_op_reply(16 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f3990022040 con 0x7f3980007030 2023-10-05T19:28:32.876 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.870+0000 7f3998143700 1 -- 172.21.15.167:0/906877954 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:17 5.1a 5:58000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x557e62a11ee0 con 0x557e6294dbc0 2023-10-05T19:28:32.876 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.870+0000 7f3998143700 1 -- 172.21.15.167:0/906877954 <== osd.1 v2:172.21.15.167:6800/30035 9 ==== osd_op_reply(17 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f3990022040 con 0x557e6294dbc0 2023-10-05T19:28:32.877 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.870+0000 7f3998143700 1 -- 172.21.15.167:0/906877954 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:18 5.6 5:60000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x557e62a11ee0 con 0x7f398801d350 2023-10-05T19:28:32.877 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.870+0000 7f3997141700 1 -- 172.21.15.167:0/906877954 <== osd.2 v2:172.21.15.167:6816/30037 3 ==== osd_op_reply(18 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f3980012450 con 0x7f398801d350 2023-10-05T19:28:32.877 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.870+0000 7f3997141700 1 -- 172.21.15.167:0/906877954 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:19 5.16 5:68000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x557e62a11ee0 con 0x557e62a0f4c0 2023-10-05T19:28:32.878 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.874+0000 7f3997942700 1 -- 172.21.15.167:0/906877954 <== osd.3 v2:172.21.15.167:6808/30036 3 ==== osd_op_reply(19 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f3988024020 con 0x557e62a0f4c0 2023-10-05T19:28:32.878 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.874+0000 7f3997942700 1 -- 172.21.15.167:0/906877954 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:20 5.e 5:70000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x557e62a11ee0 con 0x557e62a0f4c0 2023-10-05T19:28:32.879 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.874+0000 7f3997942700 1 -- 172.21.15.167:0/906877954 <== osd.3 v2:172.21.15.167:6808/30036 4 ==== osd_op_reply(20 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f3988024020 con 0x557e62a0f4c0 2023-10-05T19:28:32.879 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.874+0000 7f3997942700 1 -- 172.21.15.167:0/906877954 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:21 5.1e 5:78000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x557e62a11ee0 con 0x7f3980007030 2023-10-05T19:28:32.880 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.874+0000 7f3998143700 1 -- 172.21.15.167:0/906877954 <== osd.0 v2:172.21.15.118:6800/27690 5 ==== osd_op_reply(21 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f3990022040 con 0x7f3980007030 2023-10-05T19:28:32.880 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.874+0000 7f3998143700 1 -- 172.21.15.167:0/906877954 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:22 5.1 5:80000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x557e62a11ee0 con 0x557e62a0f4c0 2023-10-05T19:28:32.881 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.874+0000 7f3997942700 1 -- 172.21.15.167:0/906877954 <== osd.3 v2:172.21.15.167:6808/30036 5 ==== osd_op_reply(22 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f3988024020 con 0x557e62a0f4c0 2023-10-05T19:28:32.881 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.874+0000 7f3997942700 1 -- 172.21.15.167:0/906877954 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:23 5.11 5:88000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x557e62a11ee0 con 0x557e62a0f4c0 2023-10-05T19:28:32.881 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.874+0000 7f3997942700 1 -- 172.21.15.167:0/906877954 <== osd.3 v2:172.21.15.167:6808/30036 6 ==== osd_op_reply(23 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f3988024020 con 0x557e62a0f4c0 2023-10-05T19:28:32.881 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.874+0000 7f3997942700 1 -- 172.21.15.167:0/906877954 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:24 5.9 5:90000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x557e62a11ee0 con 0x557e6294dbc0 2023-10-05T19:28:32.882 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.878+0000 7f3998143700 1 -- 172.21.15.167:0/906877954 <== osd.1 v2:172.21.15.167:6800/30035 10 ==== osd_op_reply(24 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f3990022040 con 0x557e6294dbc0 2023-10-05T19:28:32.882 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.878+0000 7f3998143700 1 -- 172.21.15.167:0/906877954 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:25 5.19 5:98000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x557e62a11ee0 con 0x557e6294dbc0 2023-10-05T19:28:32.883 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.878+0000 7f3998143700 1 -- 172.21.15.167:0/906877954 <== osd.1 v2:172.21.15.167:6800/30035 11 ==== osd_op_reply(25 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f3990022040 con 0x557e6294dbc0 2023-10-05T19:28:32.883 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.878+0000 7f3998143700 1 -- 172.21.15.167:0/906877954 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:26 5.5 5:a0000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x557e62a11ee0 con 0x7f3980007030 2023-10-05T19:28:32.884 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.878+0000 7f3998143700 1 -- 172.21.15.167:0/906877954 <== osd.0 v2:172.21.15.118:6800/27690 6 ==== osd_op_reply(26 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f3990022040 con 0x7f3980007030 2023-10-05T19:28:32.884 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.878+0000 7f3998143700 1 -- 172.21.15.167:0/906877954 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:27 5.15 5:a8000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x557e62a11ee0 con 0x557e62a0f4c0 2023-10-05T19:28:32.885 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.878+0000 7f3997942700 1 -- 172.21.15.167:0/906877954 <== osd.3 v2:172.21.15.167:6808/30036 7 ==== osd_op_reply(27 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f3988024020 con 0x557e62a0f4c0 2023-10-05T19:28:32.885 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.878+0000 7f3997942700 1 -- 172.21.15.167:0/906877954 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:28 5.d 5:b0000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x557e62a11ee0 con 0x7f398801d350 2023-10-05T19:28:32.886 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.878+0000 7f3997141700 1 -- 172.21.15.167:0/906877954 <== osd.2 v2:172.21.15.167:6816/30037 4 ==== osd_op_reply(28 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f3980012450 con 0x7f398801d350 2023-10-05T19:28:32.886 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.878+0000 7f3997141700 1 -- 172.21.15.167:0/906877954 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:29 5.1d 5:b8000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x557e62a11ee0 con 0x557e6294dbc0 2023-10-05T19:28:32.887 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.882+0000 7f3998143700 1 -- 172.21.15.167:0/906877954 <== osd.1 v2:172.21.15.167:6800/30035 12 ==== osd_op_reply(29 [pgnls start_epoch 30 out=107b] v30'2 uv2 ondisk = 1) v8 ==== 144+0+107 (crc 0 0 0) 0x7f3990022040 con 0x557e6294dbc0 2023-10-05T19:28:32.887 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.882+0000 7f3998143700 1 -- 172.21.15.167:0/906877954 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:30 5.3 5:c0000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x557e62a11ee0 con 0x7f3980007030 2023-10-05T19:28:32.888 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.882+0000 7f3998143700 1 -- 172.21.15.167:0/906877954 <== osd.0 v2:172.21.15.118:6800/27690 7 ==== osd_op_reply(30 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f3990022040 con 0x7f3980007030 2023-10-05T19:28:32.888 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.882+0000 7f3998143700 1 -- 172.21.15.167:0/906877954 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:31 5.13 5:c8000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x557e62a11ee0 con 0x557e62a0f4c0 2023-10-05T19:28:32.889 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.882+0000 7f3997942700 1 -- 172.21.15.167:0/906877954 <== osd.3 v2:172.21.15.167:6808/30036 8 ==== osd_op_reply(31 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f3988024020 con 0x557e62a0f4c0 2023-10-05T19:28:32.889 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.882+0000 7f3997942700 1 -- 172.21.15.167:0/906877954 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:32 5.b 5:d0000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x557e62a11ee0 con 0x7f3980007030 2023-10-05T19:28:32.890 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.882+0000 7f3998143700 1 -- 172.21.15.167:0/906877954 <== osd.0 v2:172.21.15.118:6800/27690 8 ==== osd_op_reply(32 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f3990022040 con 0x7f3980007030 2023-10-05T19:28:32.890 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.882+0000 7f3998143700 1 -- 172.21.15.167:0/906877954 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:33 5.1b 5:d8000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x557e62a11ee0 con 0x7f398801d350 2023-10-05T19:28:32.890 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.886+0000 7f3997141700 1 -- 172.21.15.167:0/906877954 <== osd.2 v2:172.21.15.167:6816/30037 5 ==== osd_op_reply(33 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f3980012450 con 0x7f398801d350 2023-10-05T19:28:32.891 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.886+0000 7f3997141700 1 -- 172.21.15.167:0/906877954 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:34 5.7 5:e0000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x557e62a11ee0 con 0x557e62a0f4c0 2023-10-05T19:28:32.891 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.886+0000 7f3997942700 1 -- 172.21.15.167:0/906877954 <== osd.3 v2:172.21.15.167:6808/30036 9 ==== osd_op_reply(34 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f3988024020 con 0x557e62a0f4c0 2023-10-05T19:28:32.891 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.886+0000 7f3997942700 1 -- 172.21.15.167:0/906877954 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:35 5.17 5:e8000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x557e62a11ee0 con 0x7f3980007030 2023-10-05T19:28:32.892 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.886+0000 7f3998143700 1 -- 172.21.15.167:0/906877954 <== osd.0 v2:172.21.15.118:6800/27690 9 ==== osd_op_reply(35 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f3990022040 con 0x7f3980007030 2023-10-05T19:28:32.892 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.886+0000 7f3998143700 1 -- 172.21.15.167:0/906877954 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:36 5.f 5:f0000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x557e62a11ee0 con 0x557e6294dbc0 2023-10-05T19:28:32.893 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.886+0000 7f3998143700 1 -- 172.21.15.167:0/906877954 <== osd.1 v2:172.21.15.167:6800/30035 13 ==== osd_op_reply(36 [pgnls start_epoch 30 out=115b] v23'2 uv2 ondisk = 1) v8 ==== 144+0+115 (crc 0 0 0) 0x7f399002f080 con 0x557e6294dbc0 2023-10-05T19:28:32.893 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.886+0000 7f3998143700 1 -- 172.21.15.167:0/906877954 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:37 5.1f 5:f8000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x557e62a11ee0 con 0x7f398801d350 2023-10-05T19:28:32.894 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.890+0000 7f3997141700 1 -- 172.21.15.167:0/906877954 <== osd.2 v2:172.21.15.167:6816/30037 6 ==== osd_op_reply(37 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f3980012450 con 0x7f398801d350 2023-10-05T19:28:32.894 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.890+0000 7f3999e0de40 20 RGWRados::pool_iterate: got zonegroup_info.4f1f719d-9648-417b-b10c-e5d939825419 2023-10-05T19:28:32.894 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.890+0000 7f3999e0de40 20 RGWRados::pool_iterate: got zone_info.9a9df965-6612-45d4-a45f-6e3c889bacc2 2023-10-05T19:28:32.895 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.890+0000 7f3999e0de40 20 RGWRados::pool_iterate: got zone_names.default 2023-10-05T19:28:32.895 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.890+0000 7f3999e0de40 20 RGWRados::pool_iterate: got zonegroups_names.default 2023-10-05T19:28:32.895 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.890+0000 7f3999e0de40 20 rados->read ofs=0 len=0 2023-10-05T19:28:32.895 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.890+0000 7f3999e0de40 1 -- 172.21.15.167:0/906877954 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:38 5.12 5:49953fa1:::default.realm:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x557e62a0df70 con 0x557e6294dbc0 2023-10-05T19:28:32.896 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.890+0000 7f3998143700 1 -- 172.21.15.167:0/906877954 <== osd.1 v2:172.21.15.167:6800/30035 14 ==== osd_op_reply(38 default.realm [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 ==== 157+0+0 (crc 0 0 0) 0x7f399002f080 con 0x557e6294dbc0 2023-10-05T19:28:32.896 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.890+0000 7f3999e0de40 20 rados_obj.operate() r=-2 bl.length=0 2023-10-05T19:28:32.896 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.890+0000 7f3999e0de40 20 rados->read ofs=0 len=0 2023-10-05T19:28:32.896 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.890+0000 7f3999e0de40 1 -- 172.21.15.167:0/906877954 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:39 5.f 5:f4c53578:::zonegroups_names.default:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x557e62a0e370 con 0x557e6294dbc0 2023-10-05T19:28:32.897 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.890+0000 7f3998143700 1 -- 172.21.15.167:0/906877954 <== osd.1 v2:172.21.15.167:6800/30035 15 ==== osd_op_reply(39 zonegroups_names.default [read 0~46 out=46b] v0'0 uv2 ondisk = 0) v8 ==== 168+0+46 (crc 0 0 0) 0x7f399002f080 con 0x557e6294dbc0 2023-10-05T19:28:32.897 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.890+0000 7f3999e0de40 20 rados_obj.operate() r=0 bl.length=46 2023-10-05T19:28:32.897 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.890+0000 7f3999e0de40 20 rados->read ofs=0 len=0 2023-10-05T19:28:32.897 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.890+0000 7f3999e0de40 1 -- 172.21.15.167:0/906877954 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:40 5.c 5:30d299a5:::zonegroup_info.4f1f719d-9648-417b-b10c-e5d939825419:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x557e62a12350 con 0x7f3980007030 2023-10-05T19:28:32.898 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.890+0000 7f3998143700 1 -- 172.21.15.167:0/906877954 <== osd.0 v2:172.21.15.118:6800/27690 10 ==== osd_op_reply(40 zonegroup_info.4f1f719d-9648-417b-b10c-e5d939825419 [read 0~362 out=362b] v0'0 uv1 ondisk = 0) v8 ==== 195+0+362 (crc 0 0 0) 0x7f399002f080 con 0x7f3980007030 2023-10-05T19:28:32.898 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.890+0000 7f3999e0de40 20 rados_obj.operate() r=0 bl.length=362 2023-10-05T19:28:32.899 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.890+0000 7f3999e0de40 20 rados->read ofs=0 len=0 2023-10-05T19:28:32.899 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.890+0000 7f3999e0de40 1 -- 172.21.15.167:0/906877954 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:41 5.13 5:cbbf6c60:::converted:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x557e62a12350 con 0x557e62a0f4c0 2023-10-05T19:28:32.899 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.890+0000 7f3997942700 1 -- 172.21.15.167:0/906877954 <== osd.3 v2:172.21.15.167:6808/30036 10 ==== osd_op_reply(41 converted [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 ==== 153+0+0 (crc 0 0 0) 0x7f3988024020 con 0x557e62a0f4c0 2023-10-05T19:28:32.899 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.890+0000 7f3999e0de40 20 rados_obj.operate() r=-2 bl.length=0 2023-10-05T19:28:32.899 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.890+0000 7f3999e0de40 20 rados->read ofs=0 len=0 2023-10-05T19:28:32.899 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.890+0000 7f3999e0de40 1 -- 172.21.15.167:0/906877954 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:42 5.12 5:49953fa1:::default.realm:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x557e62a12700 con 0x557e6294dbc0 2023-10-05T19:28:32.899 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.894+0000 7f3998143700 1 -- 172.21.15.167:0/906877954 <== osd.1 v2:172.21.15.167:6800/30035 16 ==== osd_op_reply(42 default.realm [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 ==== 157+0+0 (crc 0 0 0) 0x7f399002f080 con 0x557e6294dbc0 2023-10-05T19:28:32.899 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.894+0000 7f3999e0de40 20 rados_obj.operate() r=-2 bl.length=0 2023-10-05T19:28:32.899 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.894+0000 7f3999e0de40 20 rados->read ofs=0 len=0 2023-10-05T19:28:32.899 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.894+0000 7f3999e0de40 1 -- 172.21.15.167:0/906877954 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:43 5.f 5:f4c53578:::zonegroups_names.default:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x557e62a12ae0 con 0x557e6294dbc0 2023-10-05T19:28:32.900 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.894+0000 7f3998143700 1 -- 172.21.15.167:0/906877954 <== osd.1 v2:172.21.15.167:6800/30035 17 ==== osd_op_reply(43 zonegroups_names.default [read 0~46 out=46b] v0'0 uv2 ondisk = 0) v8 ==== 168+0+46 (crc 0 0 0) 0x7f399002f080 con 0x557e6294dbc0 2023-10-05T19:28:32.900 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.894+0000 7f3999e0de40 20 rados_obj.operate() r=0 bl.length=46 2023-10-05T19:28:32.900 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.894+0000 7f3999e0de40 1 -- 172.21.15.167:0/906877954 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:44 5.0 5:00000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x557e62a13af0 con 0x557e62a0f4c0 2023-10-05T19:28:32.901 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.894+0000 7f3997942700 1 -- 172.21.15.167:0/906877954 <== osd.3 v2:172.21.15.167:6808/30036 11 ==== osd_op_reply(44 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f3988024020 con 0x557e62a0f4c0 2023-10-05T19:28:32.902 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.894+0000 7f3997942700 1 -- 172.21.15.167:0/906877954 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:45 5.10 5:08000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x557e62a13af0 con 0x7f398801d350 2023-10-05T19:28:32.902 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.898+0000 7f3997141700 1 -- 172.21.15.167:0/906877954 <== osd.2 v2:172.21.15.167:6816/30037 7 ==== osd_op_reply(45 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f3980012450 con 0x7f398801d350 2023-10-05T19:28:32.902 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.898+0000 7f3997141700 1 -- 172.21.15.167:0/906877954 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:46 5.8 5:10000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x557e62a13af0 con 0x7f3980007030 2023-10-05T19:28:32.903 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.898+0000 7f3998143700 1 -- 172.21.15.167:0/906877954 <== osd.0 v2:172.21.15.118:6800/27690 11 ==== osd_op_reply(46 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f399002f080 con 0x7f3980007030 2023-10-05T19:28:32.903 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.898+0000 7f3998143700 1 -- 172.21.15.167:0/906877954 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:47 5.18 5:18000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x557e62a0e370 con 0x557e6294dbc0 2023-10-05T19:28:32.904 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.898+0000 7f3998143700 1 -- 172.21.15.167:0/906877954 <== osd.1 v2:172.21.15.167:6800/30035 18 ==== osd_op_reply(47 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f399002f080 con 0x557e6294dbc0 2023-10-05T19:28:32.904 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.898+0000 7f3998143700 1 -- 172.21.15.167:0/906877954 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:48 5.4 5:20000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x557e62a0e370 con 0x557e6294dbc0 2023-10-05T19:28:32.905 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.898+0000 7f3998143700 1 -- 172.21.15.167:0/906877954 <== osd.1 v2:172.21.15.167:6800/30035 19 ==== osd_op_reply(48 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f399002f080 con 0x557e6294dbc0 2023-10-05T19:28:32.905 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.898+0000 7f3998143700 1 -- 172.21.15.167:0/906877954 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:49 5.14 5:28000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x557e62a0e370 con 0x7f3980007030 2023-10-05T19:28:32.906 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.902+0000 7f3998143700 1 -- 172.21.15.167:0/906877954 <== osd.0 v2:172.21.15.118:6800/27690 12 ==== osd_op_reply(49 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f399002f080 con 0x7f3980007030 2023-10-05T19:28:32.906 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.902+0000 7f3998143700 1 -- 172.21.15.167:0/906877954 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:50 5.c 5:30000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x557e62a0e370 con 0x7f3980007030 2023-10-05T19:28:32.907 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.902+0000 7f3998143700 1 -- 172.21.15.167:0/906877954 <== osd.0 v2:172.21.15.118:6800/27690 13 ==== osd_op_reply(50 [pgnls start_epoch 30 out=112b] v23'1 uv1 ondisk = 1) v8 ==== 144+0+112 (crc 0 0 0) 0x7f399002f080 con 0x7f3980007030 2023-10-05T19:28:32.907 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.902+0000 7f3998143700 1 -- 172.21.15.167:0/906877954 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:51 5.1c 5:38000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x557e62a0e370 con 0x7f398801d350 2023-10-05T19:28:32.908 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.902+0000 7f3997141700 1 -- 172.21.15.167:0/906877954 <== osd.2 v2:172.21.15.167:6816/30037 8 ==== osd_op_reply(51 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f3980012450 con 0x7f398801d350 2023-10-05T19:28:32.908 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.902+0000 7f3997141700 1 -- 172.21.15.167:0/906877954 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:52 5.2 5:40000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x557e62a0e370 con 0x557e62a0f4c0 2023-10-05T19:28:32.909 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.902+0000 7f3997942700 1 -- 172.21.15.167:0/906877954 <== osd.3 v2:172.21.15.167:6808/30036 12 ==== osd_op_reply(52 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f3988024020 con 0x557e62a0f4c0 2023-10-05T19:28:32.909 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.902+0000 7f3997942700 1 -- 172.21.15.167:0/906877954 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:53 5.12 5:48000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x557e62a0e370 con 0x557e6294dbc0 2023-10-05T19:28:32.910 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.906+0000 7f3998143700 1 -- 172.21.15.167:0/906877954 <== osd.1 v2:172.21.15.167:6800/30035 20 ==== osd_op_reply(53 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f399002f080 con 0x557e6294dbc0 2023-10-05T19:28:32.910 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.906+0000 7f3998143700 1 -- 172.21.15.167:0/906877954 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:54 5.a 5:50000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x557e62a0e370 con 0x7f3980007030 2023-10-05T19:28:32.911 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.906+0000 7f3998143700 1 -- 172.21.15.167:0/906877954 <== osd.0 v2:172.21.15.118:6800/27690 14 ==== osd_op_reply(54 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f399002f080 con 0x7f3980007030 2023-10-05T19:28:32.911 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.906+0000 7f3998143700 1 -- 172.21.15.167:0/906877954 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:55 5.1a 5:58000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x557e62a0e370 con 0x557e6294dbc0 2023-10-05T19:28:32.912 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.906+0000 7f3998143700 1 -- 172.21.15.167:0/906877954 <== osd.1 v2:172.21.15.167:6800/30035 21 ==== osd_op_reply(55 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f399002f080 con 0x557e6294dbc0 2023-10-05T19:28:32.912 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.906+0000 7f3998143700 1 -- 172.21.15.167:0/906877954 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:56 5.6 5:60000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x557e62a0e370 con 0x7f398801d350 2023-10-05T19:28:32.913 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.906+0000 7f3997141700 1 -- 172.21.15.167:0/906877954 <== osd.2 v2:172.21.15.167:6816/30037 9 ==== osd_op_reply(56 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f3980012450 con 0x7f398801d350 2023-10-05T19:28:32.913 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.906+0000 7f3997141700 1 -- 172.21.15.167:0/906877954 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:57 5.16 5:68000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x557e62a0e370 con 0x557e62a0f4c0 2023-10-05T19:28:32.913 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.906+0000 7f3997942700 1 -- 172.21.15.167:0/906877954 <== osd.3 v2:172.21.15.167:6808/30036 13 ==== osd_op_reply(57 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f3988024020 con 0x557e62a0f4c0 2023-10-05T19:28:32.914 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.906+0000 7f3997942700 1 -- 172.21.15.167:0/906877954 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:58 5.e 5:70000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x557e62a0e370 con 0x557e62a0f4c0 2023-10-05T19:28:32.914 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.910+0000 7f3997942700 1 -- 172.21.15.167:0/906877954 <== osd.3 v2:172.21.15.167:6808/30036 14 ==== osd_op_reply(58 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f3988024020 con 0x557e62a0f4c0 2023-10-05T19:28:32.914 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.910+0000 7f3997942700 1 -- 172.21.15.167:0/906877954 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:59 5.1e 5:78000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x557e62a0e370 con 0x7f3980007030 2023-10-05T19:28:32.915 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.910+0000 7f3998143700 1 -- 172.21.15.167:0/906877954 <== osd.0 v2:172.21.15.118:6800/27690 15 ==== osd_op_reply(59 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f399002f080 con 0x7f3980007030 2023-10-05T19:28:32.916 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.910+0000 7f3998143700 1 -- 172.21.15.167:0/906877954 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:60 5.1 5:80000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x557e62a0e370 con 0x557e62a0f4c0 2023-10-05T19:28:32.916 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.910+0000 7f3997942700 1 -- 172.21.15.167:0/906877954 <== osd.3 v2:172.21.15.167:6808/30036 15 ==== osd_op_reply(60 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f3988024020 con 0x557e62a0f4c0 2023-10-05T19:28:32.916 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.910+0000 7f3997942700 1 -- 172.21.15.167:0/906877954 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:61 5.11 5:88000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x557e62a0e370 con 0x557e62a0f4c0 2023-10-05T19:28:32.917 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.910+0000 7f3997942700 1 -- 172.21.15.167:0/906877954 <== osd.3 v2:172.21.15.167:6808/30036 16 ==== osd_op_reply(61 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f3988024020 con 0x557e62a0f4c0 2023-10-05T19:28:32.917 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.910+0000 7f3997942700 1 -- 172.21.15.167:0/906877954 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:62 5.9 5:90000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x557e62a0e370 con 0x557e6294dbc0 2023-10-05T19:28:32.918 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.910+0000 7f3998143700 1 -- 172.21.15.167:0/906877954 <== osd.1 v2:172.21.15.167:6800/30035 22 ==== osd_op_reply(62 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f399002f080 con 0x557e6294dbc0 2023-10-05T19:28:32.918 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.910+0000 7f3998143700 1 -- 172.21.15.167:0/906877954 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:63 5.19 5:98000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x557e62a0e370 con 0x557e6294dbc0 2023-10-05T19:28:32.919 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.914+0000 7f3998143700 1 -- 172.21.15.167:0/906877954 <== osd.1 v2:172.21.15.167:6800/30035 23 ==== osd_op_reply(63 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f399002f080 con 0x557e6294dbc0 2023-10-05T19:28:32.919 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.914+0000 7f3998143700 1 -- 172.21.15.167:0/906877954 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:64 5.5 5:a0000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x557e62a0e370 con 0x7f3980007030 2023-10-05T19:28:32.919 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.914+0000 7f3998143700 1 -- 172.21.15.167:0/906877954 <== osd.0 v2:172.21.15.118:6800/27690 16 ==== osd_op_reply(64 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f399002f080 con 0x7f3980007030 2023-10-05T19:28:32.920 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.914+0000 7f3998143700 1 -- 172.21.15.167:0/906877954 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:65 5.15 5:a8000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x557e62a0e370 con 0x557e62a0f4c0 2023-10-05T19:28:32.920 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.914+0000 7f3997942700 1 -- 172.21.15.167:0/906877954 <== osd.3 v2:172.21.15.167:6808/30036 17 ==== osd_op_reply(65 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f3988024020 con 0x557e62a0f4c0 2023-10-05T19:28:32.920 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.914+0000 7f3997942700 1 -- 172.21.15.167:0/906877954 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:66 5.d 5:b0000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x557e62a0e370 con 0x7f398801d350 2023-10-05T19:28:32.921 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.914+0000 7f3997141700 1 -- 172.21.15.167:0/906877954 <== osd.2 v2:172.21.15.167:6816/30037 10 ==== osd_op_reply(66 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f3980012450 con 0x7f398801d350 2023-10-05T19:28:32.921 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.914+0000 7f3997141700 1 -- 172.21.15.167:0/906877954 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:67 5.1d 5:b8000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x557e62a0e370 con 0x557e6294dbc0 2023-10-05T19:28:32.922 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.918+0000 7f3998143700 1 -- 172.21.15.167:0/906877954 <== osd.1 v2:172.21.15.167:6800/30035 24 ==== osd_op_reply(67 [pgnls start_epoch 30 out=107b] v30'2 uv2 ondisk = 1) v8 ==== 144+0+107 (crc 0 0 0) 0x7f399002f080 con 0x557e6294dbc0 2023-10-05T19:28:32.922 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.918+0000 7f3998143700 1 -- 172.21.15.167:0/906877954 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:68 5.3 5:c0000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x557e62a0e370 con 0x7f3980007030 2023-10-05T19:28:32.923 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.918+0000 7f3998143700 1 -- 172.21.15.167:0/906877954 <== osd.0 v2:172.21.15.118:6800/27690 17 ==== osd_op_reply(68 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f399002f080 con 0x7f3980007030 2023-10-05T19:28:32.923 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.918+0000 7f3998143700 1 -- 172.21.15.167:0/906877954 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:69 5.13 5:c8000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x557e62a0e370 con 0x557e62a0f4c0 2023-10-05T19:28:32.924 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.918+0000 7f3997942700 1 -- 172.21.15.167:0/906877954 <== osd.3 v2:172.21.15.167:6808/30036 18 ==== osd_op_reply(69 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f3988024020 con 0x557e62a0f4c0 2023-10-05T19:28:32.924 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.918+0000 7f3997942700 1 -- 172.21.15.167:0/906877954 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:70 5.b 5:d0000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x557e62a0e370 con 0x7f3980007030 2023-10-05T19:28:32.925 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.918+0000 7f3998143700 1 -- 172.21.15.167:0/906877954 <== osd.0 v2:172.21.15.118:6800/27690 18 ==== osd_op_reply(70 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f399002f080 con 0x7f3980007030 2023-10-05T19:28:32.925 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.918+0000 7f3998143700 1 -- 172.21.15.167:0/906877954 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:71 5.1b 5:d8000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x557e62a0e370 con 0x7f398801d350 2023-10-05T19:28:32.926 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.922+0000 7f3997141700 1 -- 172.21.15.167:0/906877954 <== osd.2 v2:172.21.15.167:6816/30037 11 ==== osd_op_reply(71 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f3980012450 con 0x7f398801d350 2023-10-05T19:28:32.926 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.922+0000 7f3997141700 1 -- 172.21.15.167:0/906877954 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:72 5.7 5:e0000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x557e62a0e370 con 0x557e62a0f4c0 2023-10-05T19:28:32.927 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.922+0000 7f3997942700 1 -- 172.21.15.167:0/906877954 <== osd.3 v2:172.21.15.167:6808/30036 19 ==== osd_op_reply(72 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f3988024020 con 0x557e62a0f4c0 2023-10-05T19:28:32.927 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.922+0000 7f3997942700 1 -- 172.21.15.167:0/906877954 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:73 5.17 5:e8000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x557e62a0e370 con 0x7f3980007030 2023-10-05T19:28:32.928 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.922+0000 7f3998143700 1 -- 172.21.15.167:0/906877954 <== osd.0 v2:172.21.15.118:6800/27690 19 ==== osd_op_reply(73 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f399002f080 con 0x7f3980007030 2023-10-05T19:28:32.928 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.922+0000 7f3998143700 1 -- 172.21.15.167:0/906877954 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:74 5.f 5:f0000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x557e62a0e370 con 0x557e6294dbc0 2023-10-05T19:28:32.929 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.922+0000 7f3998143700 1 -- 172.21.15.167:0/906877954 <== osd.1 v2:172.21.15.167:6800/30035 25 ==== osd_op_reply(74 [pgnls start_epoch 30 out=115b] v23'2 uv2 ondisk = 1) v8 ==== 144+0+115 (crc 0 0 0) 0x7f399002f080 con 0x557e6294dbc0 2023-10-05T19:28:32.929 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.922+0000 7f3998143700 1 -- 172.21.15.167:0/906877954 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:75 5.1f 5:f8000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x557e62a0e370 con 0x7f398801d350 2023-10-05T19:28:32.930 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.922+0000 7f3997141700 1 -- 172.21.15.167:0/906877954 <== osd.2 v2:172.21.15.167:6816/30037 12 ==== osd_op_reply(75 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f3980012450 con 0x7f398801d350 2023-10-05T19:28:32.930 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.926+0000 7f3999e0de40 20 RGWRados::pool_iterate: got zonegroup_info.4f1f719d-9648-417b-b10c-e5d939825419 2023-10-05T19:28:32.930 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.926+0000 7f3999e0de40 20 RGWRados::pool_iterate: got zone_info.9a9df965-6612-45d4-a45f-6e3c889bacc2 2023-10-05T19:28:32.930 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.926+0000 7f3999e0de40 20 RGWRados::pool_iterate: got zone_names.default 2023-10-05T19:28:32.930 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.926+0000 7f3999e0de40 20 RGWRados::pool_iterate: got zonegroups_names.default 2023-10-05T19:28:32.931 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.926+0000 7f3999e0de40 20 rados->read ofs=0 len=0 2023-10-05T19:28:32.931 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.926+0000 7f3999e0de40 1 -- 172.21.15.167:0/906877954 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:76 5.f 5:f43b6ece:::zone_names.default:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x557e62a12ae0 con 0x557e6294dbc0 2023-10-05T19:28:32.931 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.926+0000 7f3998143700 1 -- 172.21.15.167:0/906877954 <== osd.1 v2:172.21.15.167:6800/30035 26 ==== osd_op_reply(76 zone_names.default [read 0~46 out=46b] v0'0 uv1 ondisk = 0) v8 ==== 162+0+46 (crc 0 0 0) 0x7f399002f080 con 0x557e6294dbc0 2023-10-05T19:28:32.931 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.926+0000 7f3999e0de40 20 rados_obj.operate() r=0 bl.length=46 2023-10-05T19:28:32.931 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.926+0000 7f3999e0de40 20 rados->read ofs=0 len=0 2023-10-05T19:28:32.931 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.926+0000 7f3999e0de40 1 -- 172.21.15.167:0/906877954 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:77 5.1d 5:b8374eb4:::zone_info.9a9df965-6612-45d4-a45f-6e3c889bacc2:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x557e62a12e90 con 0x557e6294dbc0 2023-10-05T19:28:32.932 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.926+0000 7f3998143700 1 -- 172.21.15.167:0/906877954 <== osd.1 v2:172.21.15.167:6800/30035 27 ==== osd_op_reply(77 zone_info.9a9df965-6612-45d4-a45f-6e3c889bacc2 [read 0~874 out=874b] v0'0 uv2 ondisk = 0) v8 ==== 190+0+874 (crc 0 0 0) 0x7f399002f080 con 0x557e6294dbc0 2023-10-05T19:28:32.932 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.926+0000 7f3999e0de40 20 rados_obj.operate() r=0 bl.length=874 2023-10-05T19:28:32.933 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.926+0000 7f3999e0de40 20 rados->read ofs=0 len=0 2023-10-05T19:28:32.933 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.926+0000 7f3999e0de40 1 -- 172.21.15.167:0/906877954 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:78 5.f 5:f4c53578:::zonegroups_names.default:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x557e62a12e90 con 0x557e6294dbc0 2023-10-05T19:28:32.933 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.926+0000 7f3998143700 1 -- 172.21.15.167:0/906877954 <== osd.1 v2:172.21.15.167:6800/30035 28 ==== osd_op_reply(78 zonegroups_names.default [read 0~46 out=46b] v0'0 uv2 ondisk = 0) v8 ==== 168+0+46 (crc 0 0 0) 0x7f399002f080 con 0x557e6294dbc0 2023-10-05T19:28:32.933 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.926+0000 7f3999e0de40 20 rados_obj.operate() r=0 bl.length=46 2023-10-05T19:28:32.934 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.926+0000 7f3999e0de40 20 rados->read ofs=0 len=0 2023-10-05T19:28:32.934 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.926+0000 7f3999e0de40 1 -- 172.21.15.167:0/906877954 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:79 5.c 5:30d299a5:::zonegroup_info.4f1f719d-9648-417b-b10c-e5d939825419:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x557e62a12ed0 con 0x7f3980007030 2023-10-05T19:28:32.936 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.926+0000 7f3998143700 1 -- 172.21.15.167:0/906877954 <== osd.0 v2:172.21.15.118:6800/27690 20 ==== osd_op_reply(79 zonegroup_info.4f1f719d-9648-417b-b10c-e5d939825419 [read 0~362 out=362b] v0'0 uv1 ondisk = 0) v8 ==== 195+0+362 (crc 0 0 0) 0x7f399002f080 con 0x7f3980007030 2023-10-05T19:28:32.936 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.926+0000 7f3999e0de40 20 rados_obj.operate() r=0 bl.length=362 2023-10-05T19:28:32.936 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.926+0000 7f3999e0de40 20 rados->read ofs=0 len=0 2023-10-05T19:28:32.936 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.926+0000 7f3999e0de40 1 -- 172.21.15.167:0/906877954 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:80 5.3 5:c0ad8cbd:::region_map:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x557e62a13310 con 0x7f3980007030 2023-10-05T19:28:32.936 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.930+0000 7f3998143700 1 -- 172.21.15.167:0/906877954 <== osd.0 v2:172.21.15.118:6800/27690 21 ==== osd_op_reply(80 region_map [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 ==== 154+0+0 (crc 0 0 0) 0x7f3990050040 con 0x7f3980007030 2023-10-05T19:28:32.936 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.930+0000 7f3999e0de40 20 rados_obj.operate() r=-2 bl.length=0 2023-10-05T19:28:32.937 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.930+0000 7f3999e0de40 20 zone default found 2023-10-05T19:28:32.937 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.930+0000 7f3999e0de40 4 Realm: () 2023-10-05T19:28:32.937 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.930+0000 7f3999e0de40 4 ZoneGroup: default (4f1f719d-9648-417b-b10c-e5d939825419) 2023-10-05T19:28:32.937 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.930+0000 7f3999e0de40 4 Zone: default (9a9df965-6612-45d4-a45f-6e3c889bacc2) 2023-10-05T19:28:32.937 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.930+0000 7f3999e0de40 10 cannot find current period zonegroup using local zonegroup configuration 2023-10-05T19:28:32.937 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.930+0000 7f3999e0de40 20 zonegroup default 2023-10-05T19:28:32.937 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.930+0000 7f3999e0de40 20 rados->read ofs=0 len=0 2023-10-05T19:28:32.937 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.930+0000 7f3999e0de40 1 -- 172.21.15.167:0/906877954 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:81 5.3 5:c52100b6:::period_config.default:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x557e62a13310 con 0x7f3980007030 2023-10-05T19:28:32.937 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.930+0000 7f3998143700 1 -- 172.21.15.167:0/906877954 <== osd.0 v2:172.21.15.118:6800/27690 22 ==== osd_op_reply(81 period_config.default [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 ==== 165+0+0 (crc 0 0 0) 0x7f3990050040 con 0x7f3980007030 2023-10-05T19:28:32.937 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.930+0000 7f3999e0de40 20 rados_obj.operate() r=-2 bl.length=0 2023-10-05T19:28:32.937 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.930+0000 7f3999e0de40 20 rados->read ofs=0 len=0 2023-10-05T19:28:32.938 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.930+0000 7f3999e0de40 1 -- 172.21.15.167:0/906877954 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:82 6.2 6:4347d321:::bucket.sync-source-hints.:head [call version.read in=11b,read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x557e62a14a70 con 0x557e62a0f4c0 2023-10-05T19:28:32.938 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.930+0000 7f3997942700 1 -- 172.21.15.167:0/906877954 <== osd.3 v2:172.21.15.167:6808/30036 20 ==== osd_op_reply(82 bucket.sync-source-hints. [call,read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 ==== 211+0+0 (crc 0 0 0) 0x7f3988024020 con 0x557e62a0f4c0 2023-10-05T19:28:32.938 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.930+0000 7f3999e0de40 20 rados_obj.operate() r=-2 bl.length=0 2023-10-05T19:28:32.938 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.930+0000 7f3999e0de40 20 rados->read ofs=0 len=0 2023-10-05T19:28:32.938 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.930+0000 7f3999e0de40 1 -- 172.21.15.167:0/906877954 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:83 6.b 6:d467b91b:::bucket.sync-target-hints.:head [call version.read in=11b,read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x557e62a14e20 con 0x557e62a0f4c0 2023-10-05T19:28:32.939 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.930+0000 7f3997942700 1 -- 172.21.15.167:0/906877954 <== osd.3 v2:172.21.15.167:6808/30036 21 ==== osd_op_reply(83 bucket.sync-target-hints. [call,read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 ==== 211+0+0 (crc 0 0 0) 0x7f3988024020 con 0x557e62a0f4c0 2023-10-05T19:28:32.939 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.930+0000 7f3999e0de40 20 rados_obj.operate() r=-2 bl.length=0 2023-10-05T19:28:32.939 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.930+0000 7f3999e0de40 20 started sync module instance, tier type = 2023-10-05T19:28:32.939 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.930+0000 7f3999e0de40 20 started zone id=9a9df965-6612-45d4-a45f-6e3c889bacc2 (name=default) with tier type = 2023-10-05T19:28:32.939 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.930+0000 7f3999e0de40 1 -- 172.21.15.167:0/906877954 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:84 7.1f 7:f95f44c2:::notify.0:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x557e62a14130 con 0x7f398801d350 2023-10-05T19:28:32.940 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.934+0000 7f3997141700 1 -- 172.21.15.167:0/906877954 <== osd.2 v2:172.21.15.167:6816/30037 13 ==== osd_op_reply(84 notify.0 [create] v30'114 uv114 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f3980012450 con 0x7f398801d350 2023-10-05T19:28:32.940 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.934+0000 7f3999e0de40 1 -- 172.21.15.167:0/906877954 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:85 7.1f 7:f95f44c2:::notify.0:head [watch watch cookie 94001308976352] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x557e62a0dba0 con 0x7f398801d350 2023-10-05T19:28:32.940 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.934+0000 7f3999e0de40 1 -- 172.21.15.167:0/906877954 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:86 7.0 7:05bf5b68:::notify.1:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x557e62a15c50 con 0x557e6294dbc0 2023-10-05T19:28:32.943 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.938+0000 7f3997141700 1 -- 172.21.15.167:0/906877954 <== osd.2 v2:172.21.15.167:6816/30037 14 ==== osd_op_reply(85 notify.0 [watch watch cookie 94001308976352] v30'115 uv114 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f3980012450 con 0x7f398801d350 2023-10-05T19:28:32.943 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.938+0000 7f3998143700 1 -- 172.21.15.167:0/906877954 <== osd.1 v2:172.21.15.167:6800/30035 29 ==== osd_op_reply(86 notify.1 [create] v30'114 uv114 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f3990050040 con 0x557e6294dbc0 2023-10-05T19:28:32.943 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.938+0000 7f3999e0de40 1 -- 172.21.15.167:0/906877954 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:87 7.0 7:05bf5b68:::notify.1:head [watch watch cookie 94001308979664] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x557e62a16050 con 0x557e6294dbc0 2023-10-05T19:28:32.943 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.938+0000 7f3999e0de40 1 -- 172.21.15.167:0/906877954 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:88 7.15 7:a93a5511:::notify.2:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x557e62a168f0 con 0x557e62a0f4c0 2023-10-05T19:28:32.947 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.942+0000 7f3997942700 1 -- 172.21.15.167:0/906877954 <== osd.3 v2:172.21.15.167:6808/30036 22 ==== osd_op_reply(88 notify.2 [create] v30'114 uv114 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f3988024020 con 0x557e62a0f4c0 2023-10-05T19:28:32.947 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.942+0000 7f3998143700 1 -- 172.21.15.167:0/906877954 <== osd.1 v2:172.21.15.167:6800/30035 30 ==== osd_op_reply(87 notify.1 [watch watch cookie 94001308979664] v30'115 uv114 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f3990050040 con 0x557e6294dbc0 2023-10-05T19:28:32.947 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.942+0000 7f3999e0de40 1 -- 172.21.15.167:0/906877954 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:89 7.15 7:a93a5511:::notify.2:head [watch watch cookie 94001308983376] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x557e62a15840 con 0x557e62a0f4c0 2023-10-05T19:28:32.947 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.942+0000 7f3999e0de40 1 -- 172.21.15.167:0/906877954 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:90 7.e 7:7759931f:::notify.3:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x557e62a17680 con 0x7f398801d350 2023-10-05T19:28:32.949 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.942+0000 7f3997141700 1 -- 172.21.15.167:0/906877954 <== osd.2 v2:172.21.15.167:6816/30037 15 ==== osd_op_reply(90 notify.3 [create] v30'114 uv114 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f3980012450 con 0x7f398801d350 2023-10-05T19:28:32.949 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.942+0000 7f3999e0de40 1 -- 172.21.15.167:0/906877954 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:91 7.e 7:7759931f:::notify.3:head [watch watch cookie 94001308987792] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x557e62a17f20 con 0x7f398801d350 2023-10-05T19:28:32.949 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.942+0000 7f3999e0de40 1 -- 172.21.15.167:0/906877954 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:92 7.d 7:b4812045:::notify.4:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x557e62a18ae0 con 0x557e6294dbc0 2023-10-05T19:28:32.949 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.942+0000 7f3997942700 1 -- 172.21.15.167:0/906877954 <== osd.3 v2:172.21.15.167:6808/30036 23 ==== osd_op_reply(89 notify.2 [watch watch cookie 94001308983376] v30'115 uv114 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f3988024020 con 0x557e62a0f4c0 2023-10-05T19:28:32.951 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.946+0000 7f3997141700 1 -- 172.21.15.167:0/906877954 <== osd.2 v2:172.21.15.167:6816/30037 16 ==== osd_op_reply(91 notify.3 [watch watch cookie 94001308987792] v30'115 uv114 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f3980012450 con 0x7f398801d350 2023-10-05T19:28:32.952 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.946+0000 7f3998143700 1 -- 172.21.15.167:0/906877954 <== osd.1 v2:172.21.15.167:6800/30035 31 ==== osd_op_reply(92 notify.4 [create] v30'114 uv114 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f3990050040 con 0x557e6294dbc0 2023-10-05T19:28:32.952 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.946+0000 7f3999e0de40 1 -- 172.21.15.167:0/906877954 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:93 7.d 7:b4812045:::notify.4:head [watch watch cookie 94001308990000] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x557e62a185b0 con 0x557e6294dbc0 2023-10-05T19:28:32.952 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.946+0000 7f3999e0de40 1 -- 172.21.15.167:0/906877954 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:94 7.3 7:c609908c:::notify.5:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x557e62a193d0 con 0x7f3980007030 2023-10-05T19:28:32.954 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.950+0000 7f3998143700 1 -- 172.21.15.167:0/906877954 <== osd.0 v2:172.21.15.118:6800/27690 23 ==== osd_op_reply(94 notify.5 [create] v30'114 uv114 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f3990050040 con 0x7f3980007030 2023-10-05T19:28:32.955 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.950+0000 7f3999e0de40 1 -- 172.21.15.167:0/906877954 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:95 7.3 7:c609908c:::notify.5:head [watch watch cookie 94001308995216] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x557e62a197d0 con 0x7f3980007030 2023-10-05T19:28:32.955 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.950+0000 7f3999e0de40 1 -- 172.21.15.167:0/906877954 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:96 7.14 7:2b04a3e9:::notify.6:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x557e62a1a220 con 0x7f3980007030 2023-10-05T19:28:32.955 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.950+0000 7f3998143700 1 -- 172.21.15.167:0/906877954 <== osd.1 v2:172.21.15.167:6800/30035 32 ==== osd_op_reply(93 notify.4 [watch watch cookie 94001308990000] v30'115 uv114 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f3990050040 con 0x557e6294dbc0 2023-10-05T19:28:32.957 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.950+0000 7f3998143700 1 -- 172.21.15.167:0/906877954 <== osd.0 v2:172.21.15.118:6800/27690 24 ==== osd_op_reply(95 notify.5 [watch watch cookie 94001308995216] v30'115 uv114 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f3990050040 con 0x7f3980007030 2023-10-05T19:28:32.958 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.950+0000 7f3998143700 1 -- 172.21.15.167:0/906877954 <== osd.0 v2:172.21.15.118:6800/27690 25 ==== osd_op_reply(96 notify.6 [create] v30'114 uv114 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f3990050040 con 0x7f3980007030 2023-10-05T19:28:32.958 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.950+0000 7f3999e0de40 1 -- 172.21.15.167:0/906877954 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:97 7.14 7:2b04a3e9:::notify.6:head [watch watch cookie 94001308986528] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x557e62a192e0 con 0x7f3980007030 2023-10-05T19:28:32.958 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.954+0000 7f3999e0de40 1 -- 172.21.15.167:0/906877954 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:98 7.9 7:93e5b521:::notify.7:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x557e62a1ab10 con 0x7f3980007030 2023-10-05T19:28:32.960 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.954+0000 7f3998143700 1 -- 172.21.15.167:0/906877954 <== osd.0 v2:172.21.15.118:6800/27690 26 ==== osd_op_reply(97 notify.6 [watch watch cookie 94001308986528] v30'115 uv114 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f3990050040 con 0x7f3980007030 2023-10-05T19:28:32.961 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.954+0000 7f3998143700 1 -- 172.21.15.167:0/906877954 <== osd.0 v2:172.21.15.118:6800/27690 27 ==== osd_op_reply(98 notify.7 [create] v30'114 uv114 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f3990050040 con 0x7f3980007030 2023-10-05T19:28:32.961 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.954+0000 7f3999e0de40 1 -- 172.21.15.167:0/906877954 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:99 7.9 7:93e5b521:::notify.7:head [watch watch cookie 94001308998896] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x557e62a1a940 con 0x7f3980007030 2023-10-05T19:28:32.961 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.954+0000 7f3999e0de40 20 add_watcher() i=0 2023-10-05T19:28:32.961 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.954+0000 7f3999e0de40 20 add_watcher() i=1 2023-10-05T19:28:32.961 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.954+0000 7f3999e0de40 20 add_watcher() i=2 2023-10-05T19:28:32.961 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.954+0000 7f3999e0de40 20 add_watcher() i=3 2023-10-05T19:28:32.961 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.954+0000 7f3999e0de40 20 add_watcher() i=4 2023-10-05T19:28:32.961 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.954+0000 7f3999e0de40 20 add_watcher() i=5 2023-10-05T19:28:32.961 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.954+0000 7f3999e0de40 20 add_watcher() i=6 2023-10-05T19:28:32.964 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.958+0000 7f3998143700 1 -- 172.21.15.167:0/906877954 <== osd.0 v2:172.21.15.118:6800/27690 28 ==== osd_op_reply(99 notify.7 [watch watch cookie 94001308998896] v30'115 uv114 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f3990050040 con 0x7f3980007030 2023-10-05T19:28:32.964 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.958+0000 7f3999e0de40 20 add_watcher() i=7 2023-10-05T19:28:32.964 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.958+0000 7f3999e0de40 2 all 8 watchers are set, enabling cache 2023-10-05T19:28:32.964 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.958+0000 7f3999e0de40 1 -- 172.21.15.167:0/906877954 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:100 6.4 6:22d26bf9:::data_loggenerations_metadata:head [call version.check_conds in=74b,call version.read in=11b,read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x557e62a1d970 con 0x557e6294dbc0 2023-10-05T19:28:32.965 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.958+0000 7f3998143700 1 -- 172.21.15.167:0/906877954 <== osd.1 v2:172.21.15.167:6800/30035 33 ==== osd_op_reply(100 data_loggenerations_metadata [call,call out=48b,read 0~28 out=28b] v0'0 uv1 ondisk = 0) v8 ==== 256+0+76 (crc 0 0 0) 0x7f3990050040 con 0x557e6294dbc0 2023-10-05T19:28:32.965 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.958+0000 7f3999e0de40 1 -- 172.21.15.167:0/906877954 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:101 6.4 6:22d26bf9:::data_loggenerations_metadata:head [watch watch cookie 94001309000784] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x557e62a1ae30 con 0x557e6294dbc0 2023-10-05T19:28:32.967 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.962+0000 7f3998143700 1 -- 172.21.15.167:0/906877954 <== osd.1 v2:172.21.15.167:6800/30035 34 ==== osd_op_reply(101 data_loggenerations_metadata [watch watch cookie 94001309000784] v30'111 uv1 ondisk = 0) v8 ==== 172+0+0 (crc 0 0 0) 0x7f3990050040 con 0x557e6294dbc0 2023-10-05T19:28:32.967 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.962+0000 7f39697e2700 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2023-10-05T19:28:32.967 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.962+0000 7f3999e0de40 20 check_secure_mon_conn(): auth registy supported: methods=[2] modes=[2,1] 2023-10-05T19:28:32.967 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.962+0000 7f3999e0de40 20 check_secure_mon_conn(): mode 1 is insecure 2023-10-05T19:28:32.968 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.962+0000 7f3999e0de40 1 Processor -- start 2023-10-05T19:28:32.968 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.962+0000 7f3999e0de40 1 -- start start 2023-10-05T19:28:32.968 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.962+0000 7f3999e0de40 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x557e62a280c0 0x557e62a28490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-05T19:28:32.968 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.962+0000 7f3999e0de40 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x557e62a289d0 con 0x557e62a280c0 2023-10-05T19:28:32.969 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.962+0000 7f3997141700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x557e62a280c0 0x557e62a28490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-05T19:28:32.969 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.962+0000 7f3997141700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x557e62a280c0 0x557e62a28490 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.118:3300/0 says I am v2:172.21.15.167:58512/0 (socket says 172.21.15.167:58512) 2023-10-05T19:28:32.969 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.962+0000 7f3997141700 1 -- 172.21.15.167:0/2484736955 learned_addr learned my addr 172.21.15.167:0/2484736955 (peer_addr_for_me v2:172.21.15.167:0/0) 2023-10-05T19:28:32.969 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.962+0000 7f3997141700 1 -- 172.21.15.167:0/2484736955 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x557e62a28b10 con 0x557e62a280c0 2023-10-05T19:28:32.970 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.962+0000 7f3997141700 1 --2- 172.21.15.167:0/2484736955 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x557e62a280c0 0x557e62a28490 secure :-1 s=READY pgs=366 cs=0 l=1 rev1=1 crypto rx=0x557e6294e020 tx=0x7f3980012000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=170e181360654970 server_cookie=0 in_seq=0 out_seq=0 2023-10-05T19:28:32.970 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.962+0000 7f3968fe1700 1 -- 172.21.15.167:0/2484736955 <== mon.0 v2:172.21.15.118:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f398001cee0 con 0x557e62a280c0 2023-10-05T19:28:32.970 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.962+0000 7f3968fe1700 1 -- 172.21.15.167:0/2484736955 <== mon.0 v2:172.21.15.118:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3980012da0 con 0x557e62a280c0 2023-10-05T19:28:32.970 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.962+0000 7f3968fe1700 1 -- 172.21.15.167:0/2484736955 <== mon.0 v2:172.21.15.118:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f398002a8a0 con 0x557e62a280c0 2023-10-05T19:28:32.971 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.966+0000 7f3999e0de40 1 -- 172.21.15.167:0/2484736955 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x557e62a280c0 msgr2=0x557e62a28490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-05T19:28:32.971 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.966+0000 7f3999e0de40 1 --2- 172.21.15.167:0/2484736955 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x557e62a280c0 0x557e62a28490 secure :-1 s=READY pgs=366 cs=0 l=1 rev1=1 crypto rx=0x557e6294e020 tx=0x7f3980012000 comp rx=0 tx=0).stop 2023-10-05T19:28:32.971 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.966+0000 7f3999e0de40 1 -- 172.21.15.167:0/2484736955 shutdown_connections 2023-10-05T19:28:32.971 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.966+0000 7f3999e0de40 1 --2- 172.21.15.167:0/2484736955 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x557e62a280c0 0x557e62a28490 unknown :-1 s=CLOSED pgs=366 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-05T19:28:32.972 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.966+0000 7f3999e0de40 1 -- 172.21.15.167:0/2484736955 >> 172.21.15.167:0/2484736955 conn(0x557e62a23480 msgr2=0x557e62a215d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-05T19:28:32.972 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.966+0000 7f3999e0de40 1 -- 172.21.15.167:0/2484736955 shutdown_connections 2023-10-05T19:28:32.972 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.966+0000 7f3999e0de40 1 -- 172.21.15.167:0/2484736955 wait complete. 2023-10-05T19:28:32.973 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.966+0000 7f3999e0de40 1 Processor -- start 2023-10-05T19:28:32.973 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.966+0000 7f3999e0de40 1 -- start start 2023-10-05T19:28:32.973 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.966+0000 7f3999e0de40 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x557e62a280c0 0x557e62a2dbf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-05T19:28:32.973 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.966+0000 7f3999e0de40 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x557e62a2ebf0 con 0x557e62a280c0 2023-10-05T19:28:32.973 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.966+0000 7f3997141700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x557e62a280c0 0x557e62a2dbf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-05T19:28:32.973 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.966+0000 7f3997141700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x557e62a280c0 0x557e62a2dbf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.118:3300/0 says I am v2:172.21.15.167:58526/0 (socket says 172.21.15.167:58526) 2023-10-05T19:28:32.973 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.966+0000 7f3997141700 1 -- 172.21.15.167:0/2402177238 learned_addr learned my addr 172.21.15.167:0/2402177238 (peer_addr_for_me v2:172.21.15.167:0/0) 2023-10-05T19:28:32.974 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.966+0000 7f3997141700 1 -- 172.21.15.167:0/2402177238 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x557e6294e050 con 0x557e62a280c0 2023-10-05T19:28:32.975 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.966+0000 7f3997141700 1 --2- 172.21.15.167:0/2402177238 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x557e62a280c0 0x557e62a2dbf0 secure :-1 s=READY pgs=367 cs=0 l=1 rev1=1 crypto rx=0x7f3980012fd0 tx=0x7f398002f000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-05T19:28:32.975 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.970+0000 7f38e2ffd700 1 -- 172.21.15.167:0/2402177238 <== mon.0 v2:172.21.15.118:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f3980009740 con 0x557e62a280c0 2023-10-05T19:28:32.975 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.970+0000 7f38e2ffd700 1 -- 172.21.15.167:0/2402177238 <== mon.0 v2:172.21.15.118:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f39800098e0 con 0x557e62a280c0 2023-10-05T19:28:32.975 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.970+0000 7f3999e0de40 1 -- 172.21.15.167:0/2402177238 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x557e62a2e1f0 con 0x557e62a280c0 2023-10-05T19:28:32.975 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.970+0000 7f38e2ffd700 1 -- 172.21.15.167:0/2402177238 <== mon.0 v2:172.21.15.118:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f3980009a90 con 0x557e62a280c0 2023-10-05T19:28:32.975 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.970+0000 7f3999e0de40 1 -- 172.21.15.167:0/2402177238 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x557e62a2e630 con 0x557e62a280c0 2023-10-05T19:28:32.976 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.970+0000 7f38e2ffd700 1 -- 172.21.15.167:0/2402177238 <== mon.0 v2:172.21.15.118:3300/0 4 ==== mgrmap(e 5) v1 ==== 40329+0+0 (secure 0 0 0) 0x7f3980009c80 con 0x557e62a280c0 2023-10-05T19:28:32.976 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.970+0000 7f38e2ffd700 1 --2- 172.21.15.167:0/2402177238 >> [v2:172.21.15.167:6824/29889,v1:172.21.15.167:6825/29889] conn(0x7f38d8030ca0 0x7f38d8033150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-05T19:28:32.976 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.970+0000 7f38e2ffd700 1 -- 172.21.15.167:0/2402177238 <== mon.0 v2:172.21.15.118:3300/0 5 ==== osd_map(30..30 src has 1..30) v4 ==== 6057+0+0 (secure 0 0 0) 0x7f398002f070 con 0x557e62a280c0 2023-10-05T19:28:32.977 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.970+0000 7f3998143700 1 --2- 172.21.15.167:0/2402177238 >> [v2:172.21.15.167:6824/29889,v1:172.21.15.167:6825/29889] conn(0x7f38d8030ca0 0x7f38d8033150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-05T19:28:32.977 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.970+0000 7f3999e0de40 5 note: GC not initialized 2023-10-05T19:28:32.977 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.970+0000 7f3998143700 1 --2- 172.21.15.167:0/2402177238 >> [v2:172.21.15.167:6824/29889,v1:172.21.15.167:6825/29889] conn(0x7f38d8030ca0 0x7f38d8033150 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x557e62a1ad90 tx=0x7f3990000f90 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-05T19:28:32.979 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.974+0000 7f38d7fff700 20 reqs_thread_entry: start 2023-10-05T19:28:32.979 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.974+0000 7f38d5ffb700 20 reqs_thread_entry: start 2023-10-05T19:28:32.979 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.974+0000 7f3999e0de40 20 init_complete bucket index max shards: 11 2023-10-05T19:28:32.980 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.974+0000 7f3999e0de40 10 rgw notify: Started notification manager with: 1 workers 2023-10-05T19:28:32.980 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.974+0000 7f38c3fff700 20 rgw notify: INFO: next queues processing will happen at: Thu Oct 5 19:29:03 2023 2023-10-05T19:28:32.980 INFO:teuthology.orchestra.run.smithi167.stderr: 2023-10-05T19:28:32.980 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.974+0000 7f38c37fe700 20 reqs_thread_entry: start 2023-10-05T19:28:32.980 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.974+0000 7f3999e0de40 20 get_system_obj_state: rctx=0x7ffed6917138 obj=default.rgw.meta:users.uid:fud state=0x557e628b10b0 s->prefetch_data=0 2023-10-05T19:28:32.980 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.974+0000 7f3999e0de40 10 cache get: name=default.rgw.meta+users.uid+fud : miss 2023-10-05T19:28:32.980 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.974+0000 7f3999e0de40 1 -- 172.21.15.167:0/906877954 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:102 8.e 8:75f48285:users.uid::fud:head [call version.read in=11b,getxattrs,stat] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x557e62a96400 con 0x7f3980007030 2023-10-05T19:28:32.981 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.974+0000 7f3998143700 1 -- 172.21.15.167:0/906877954 <== osd.0 v2:172.21.15.118:6800/27690 29 ==== osd_op_reply(102 fud [call out=48b,getxattrs out=75b,stat out=16b] v0'0 uv1 ondisk = 0) v8 ==== 231+0+139 (crc 0 0 0) 0x7f3990050040 con 0x7f3980007030 2023-10-05T19:28:32.981 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.974+0000 7f3999e0de40 10 cache put: name=default.rgw.meta+users.uid+fud info.flags=0x16 2023-10-05T19:28:32.981 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.974+0000 7f3999e0de40 10 adding default.rgw.meta+users.uid+fud to cache LRU end 2023-10-05T19:28:32.981 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.974+0000 7f3999e0de40 10 updating xattr: name=ceph.objclass.version bl.length()=42 2023-10-05T19:28:32.981 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.974+0000 7f3999e0de40 20 get_system_obj_state: s->obj_tag was set empty 2023-10-05T19:28:32.981 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.974+0000 7f3999e0de40 20 Read xattr: user.rgw.idtag 2023-10-05T19:28:32.981 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.974+0000 7f3999e0de40 10 cache get: name=default.rgw.meta+users.uid+fud : type miss (requested=0x13, cached=0x16) 2023-10-05T19:28:32.982 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.974+0000 7f3999e0de40 20 rados->read ofs=0 len=0 2023-10-05T19:28:32.982 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.974+0000 7f3999e0de40 1 -- 172.21.15.167:0/906877954 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:103 8.e 8:75f48285:users.uid::fud:head [call version.check_conds in=122b,call version.read in=11b,read 0~0,getxattrs] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x557e62a98e90 con 0x7f3980007030 2023-10-05T19:28:32.982 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.974+0000 7f3998143700 1 -- 172.21.15.167:0/906877954 <== osd.0 v2:172.21.15.118:6800/27690 30 ==== osd_op_reply(103 fud [call,call out=48b,read 0~346 out=346b,getxattrs out=75b] v0'0 uv1 ondisk = 0) v8 ==== 273+0+469 (crc 0 0 0) 0x7f3990050040 con 0x7f3980007030 2023-10-05T19:28:32.982 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.978+0000 7f3999e0de40 20 rados_obj.operate() r=0 bl.length=346 2023-10-05T19:28:32.983 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.978+0000 7f3999e0de40 10 cache put: name=default.rgw.meta+users.uid+fud info.flags=0x13 2023-10-05T19:28:32.983 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.978+0000 7f3999e0de40 10 moving default.rgw.meta+users.uid+fud to cache LRU end 2023-10-05T19:28:32.983 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.978+0000 7f3999e0de40 10 updating xattr: name=ceph.objclass.version bl.length()=42 2023-10-05T19:28:32.983 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.978+0000 7f3999e0de40 20 get_system_obj_state: rctx=0x7ffed69164e0 obj=default.rgw.meta:root:myfoo state=0x557e628b10b0 s->prefetch_data=0 2023-10-05T19:28:32.983 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.978+0000 7f3999e0de40 10 cache get: name=default.rgw.meta+root+myfoo : miss 2023-10-05T19:28:32.983 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.978+0000 7f3999e0de40 1 -- 172.21.15.167:0/906877954 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:104 8.9 8:91b71841:root::myfoo:head [call version.read in=11b,getxattrs,stat] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x557e62a986a0 con 0x7f398801d350 2023-10-05T19:28:32.984 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.978+0000 7f3997141700 1 -- 172.21.15.167:0/906877954 <== osd.2 v2:172.21.15.167:6816/30037 17 ==== osd_op_reply(104 myfoo [call out=48b,getxattrs out=75b,stat out=16b] v0'0 uv18 ondisk = 0) v8 ==== 233+0+139 (crc 0 0 0) 0x7f3980032040 con 0x7f398801d350 2023-10-05T19:28:32.984 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.978+0000 7f3999e0de40 10 cache put: name=default.rgw.meta+root+myfoo info.flags=0x16 2023-10-05T19:28:32.984 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.978+0000 7f3999e0de40 10 adding default.rgw.meta+root+myfoo to cache LRU end 2023-10-05T19:28:32.984 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.978+0000 7f3999e0de40 10 updating xattr: name=ceph.objclass.version bl.length()=42 2023-10-05T19:28:32.984 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.978+0000 7f3999e0de40 20 get_system_obj_state: s->obj_tag was set empty 2023-10-05T19:28:32.984 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.978+0000 7f3999e0de40 10 cache get: name=default.rgw.meta+root+myfoo : type miss (requested=0x11, cached=0x16) 2023-10-05T19:28:32.985 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.978+0000 7f3999e0de40 20 rados->read ofs=0 len=0 2023-10-05T19:28:32.985 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.978+0000 7f3999e0de40 1 -- 172.21.15.167:0/906877954 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:105 8.9 8:91b71841:root::myfoo:head [call version.check_conds in=122b,call version.read in=11b,read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x557e62a9b130 con 0x7f398801d350 2023-10-05T19:28:32.985 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.978+0000 7f3997141700 1 -- 172.21.15.167:0/906877954 <== osd.2 v2:172.21.15.167:6816/30037 18 ==== osd_op_reply(105 myfoo [call,call out=48b,read 0~165 out=165b] v0'0 uv18 ondisk = 0) v8 ==== 233+0+213 (crc 0 0 0) 0x7f3980032040 con 0x7f398801d350 2023-10-05T19:28:32.985 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.978+0000 7f3999e0de40 20 rados_obj.operate() r=0 bl.length=165 2023-10-05T19:28:32.985 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.978+0000 7f3999e0de40 10 cache put: name=default.rgw.meta+root+myfoo info.flags=0x11 2023-10-05T19:28:32.985 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.978+0000 7f3999e0de40 10 moving default.rgw.meta+root+myfoo to cache LRU end 2023-10-05T19:28:32.985 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.978+0000 7f3999e0de40 20 get_system_obj_state: rctx=0x7ffed69164e0 obj=default.rgw.meta:root:.bucket.meta.myfoo:9a9df965-6612-45d4-a45f-6e3c889bacc2.4223.1 state=0x557e628872b0 s->prefetch_data=0 2023-10-05T19:28:32.985 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.978+0000 7f3999e0de40 10 cache get: name=default.rgw.meta+root+.bucket.meta.myfoo:9a9df965-6612-45d4-a45f-6e3c889bacc2.4223.1 : miss 2023-10-05T19:28:32.985 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.978+0000 7f3999e0de40 1 -- 172.21.15.167:0/906877954 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:106 8.11 8:8bb4ed75:root::.bucket.meta.myfoo%3a9a9df965-6612-45d4-a45f-6e3c889bacc2.4223.1:head [call version.read in=11b,getxattrs,stat] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x557e62a9a9b0 con 0x557e62a0f4c0 2023-10-05T19:28:32.986 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.978+0000 7f3997942700 1 -- 172.21.15.167:0/906877954 <== osd.3 v2:172.21.15.167:6808/30036 24 ==== osd_op_reply(106 .bucket.meta.myfoo:9a9df965-6612-45d4-a45f-6e3c889bacc2.4223.1 [call out=48b,getxattrs out=218b,stat out=16b] v0'0 uv1 ondisk = 0) v8 ==== 290+0+282 (crc 0 0 0) 0x7f3988024020 con 0x557e62a0f4c0 2023-10-05T19:28:32.987 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.978+0000 7f3999e0de40 10 cache put: name=default.rgw.meta+root+.bucket.meta.myfoo:9a9df965-6612-45d4-a45f-6e3c889bacc2.4223.1 info.flags=0x16 2023-10-05T19:28:32.987 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.978+0000 7f3999e0de40 10 adding default.rgw.meta+root+.bucket.meta.myfoo:9a9df965-6612-45d4-a45f-6e3c889bacc2.4223.1 to cache LRU end 2023-10-05T19:28:32.987 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.978+0000 7f3999e0de40 10 updating xattr: name=ceph.objclass.version bl.length()=42 2023-10-05T19:28:32.987 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.978+0000 7f3999e0de40 10 updating xattr: name=user.rgw.acl bl.length()=123 2023-10-05T19:28:32.987 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.978+0000 7f3999e0de40 20 get_system_obj_state: s->obj_tag was set empty 2023-10-05T19:28:32.987 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.978+0000 7f3999e0de40 20 Read xattr: user.rgw.acl 2023-10-05T19:28:32.987 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.978+0000 7f3999e0de40 20 Read xattr: user.rgw.idtag 2023-10-05T19:28:32.987 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.978+0000 7f3999e0de40 10 cache get: name=default.rgw.meta+root+.bucket.meta.myfoo:9a9df965-6612-45d4-a45f-6e3c889bacc2.4223.1 : type miss (requested=0x13, cached=0x16) 2023-10-05T19:28:32.987 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.978+0000 7f3999e0de40 20 rados->read ofs=0 len=0 2023-10-05T19:28:32.987 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.978+0000 7f3999e0de40 1 -- 172.21.15.167:0/906877954 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:107 8.11 8:8bb4ed75:root::.bucket.meta.myfoo%3a9a9df965-6612-45d4-a45f-6e3c889bacc2.4223.1:head [call version.check_conds in=122b,call version.read in=11b,read 0~0,getxattrs] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x557e62a9da10 con 0x557e62a0f4c0 2023-10-05T19:28:32.988 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.982+0000 7f3997942700 1 -- 172.21.15.167:0/906877954 <== osd.3 v2:172.21.15.167:6808/30036 25 ==== osd_op_reply(107 .bucket.meta.myfoo:9a9df965-6612-45d4-a45f-6e3c889bacc2.4223.1 [call,call out=48b,read 0~352 out=352b,getxattrs out=218b] v0'0 uv1 ondisk = 0) v8 ==== 332+0+618 (crc 0 0 0) 0x7f3988024020 con 0x557e62a0f4c0 2023-10-05T19:28:32.988 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.982+0000 7f3999e0de40 20 rados_obj.operate() r=0 bl.length=352 2023-10-05T19:28:32.988 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.982+0000 7f3999e0de40 10 cache put: name=default.rgw.meta+root+.bucket.meta.myfoo:9a9df965-6612-45d4-a45f-6e3c889bacc2.4223.1 info.flags=0x13 2023-10-05T19:28:32.988 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.982+0000 7f3999e0de40 10 moving default.rgw.meta+root+.bucket.meta.myfoo:9a9df965-6612-45d4-a45f-6e3c889bacc2.4223.1 to cache LRU end 2023-10-05T19:28:32.988 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.982+0000 7f3999e0de40 10 updating xattr: name=ceph.objclass.version bl.length()=42 2023-10-05T19:28:32.988 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.982+0000 7f3999e0de40 10 updating xattr: name=user.rgw.acl bl.length()=123 2023-10-05T19:28:32.988 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.982+0000 7f3999e0de40 10 chain_cache_entry: cache_locator=default.rgw.meta+root+.bucket.meta.myfoo:9a9df965-6612-45d4-a45f-6e3c889bacc2.4223.1 2023-10-05T19:28:32.988 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.982+0000 7f3999e0de40 20 get_system_obj_state: rctx=0x7ffed6916478 obj=default.rgw.meta:users.uid:fud state=0x557e628872b0 s->prefetch_data=0 2023-10-05T19:28:32.988 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.982+0000 7f3999e0de40 10 cache get: name=default.rgw.meta+users.uid+fud : type miss (requested=0x16, cached=0x13) 2023-10-05T19:28:32.988 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.982+0000 7f3999e0de40 1 -- 172.21.15.167:0/906877954 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:108 8.e 8:75f48285:users.uid::fud:head [call version.read in=11b,getxattrs,stat] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x557e62a9e110 con 0x7f3980007030 2023-10-05T19:28:32.989 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.982+0000 7f3998143700 1 -- 172.21.15.167:0/906877954 <== osd.0 v2:172.21.15.118:6800/27690 31 ==== osd_op_reply(108 fud [call out=48b,getxattrs out=75b,stat out=16b] v0'0 uv1 ondisk = 0) v8 ==== 231+0+139 (crc 0 0 0) 0x7f3990050040 con 0x7f3980007030 2023-10-05T19:28:32.989 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.982+0000 7f3999e0de40 10 cache put: name=default.rgw.meta+users.uid+fud info.flags=0x16 2023-10-05T19:28:32.989 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.982+0000 7f3999e0de40 10 moving default.rgw.meta+users.uid+fud to cache LRU end 2023-10-05T19:28:32.989 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.982+0000 7f3999e0de40 10 updating xattr: name=ceph.objclass.version bl.length()=42 2023-10-05T19:28:32.989 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.982+0000 7f3999e0de40 20 get_system_obj_state: s->obj_tag was set empty 2023-10-05T19:28:32.990 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.982+0000 7f3999e0de40 20 Read xattr: user.rgw.idtag 2023-10-05T19:28:32.990 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.982+0000 7f3999e0de40 10 cache get: name=default.rgw.meta+users.uid+fud : hit (requested=0x13, cached=0x17) 2023-10-05T19:28:32.990 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.982+0000 7f3999e0de40 1 -- 172.21.15.167:0/906877954 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:109 8.9 8:96d98480:users.uid::foo.buckets:head [call user.remove_bucket in=58b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x557e62a9d510 con 0x7f398801d350 2023-10-05T19:28:32.990 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.982+0000 7f3997141700 1 -- 172.21.15.167:0/906877954 <== osd.2 v2:172.21.15.167:6816/30037 19 ==== osd_op_reply(109 foo.buckets [call] v30'19 uv17 ondisk = 0) v8 ==== 155+0+0 (crc 0 0 0) 0x7f3980032040 con 0x7f398801d350 2023-10-05T19:28:32.990 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.982+0000 7f3999e0de40 1 -- 172.21.15.167:0/906877954 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:110 8.11 8:8bb4ed75:root::.bucket.meta.myfoo%3a9a9df965-6612-45d4-a45f-6e3c889bacc2.4223.1:head [delete,create,call version.check_conds in=122b,call version.set in=58b,writefull 0~352 in=352b,setxattr user.rgw.acl (123) in=135b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x557e62aa6800 con 0x557e62a0f4c0 2023-10-05T19:28:32.993 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.986+0000 7f3997942700 1 -- 172.21.15.167:0/906877954 <== osd.3 v2:172.21.15.167:6808/30036 26 ==== osd_op_reply(110 .bucket.meta.myfoo:9a9df965-6612-45d4-a45f-6e3c889bacc2.4223.1 [delete,create,call,call,writefull 0~352,setxattr (123)] v30'2 uv2 ondisk = 0) v8 ==== 416+0+0 (crc 0 0 0) 0x7f3988024020 con 0x557e62a0f4c0 2023-10-05T19:28:32.993 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.986+0000 7f3999e0de40 10 cache put: name=default.rgw.meta+root+.bucket.meta.myfoo:9a9df965-6612-45d4-a45f-6e3c889bacc2.4223.1 info.flags=0x17 2023-10-05T19:28:32.993 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.986+0000 7f3999e0de40 10 moving default.rgw.meta+root+.bucket.meta.myfoo:9a9df965-6612-45d4-a45f-6e3c889bacc2.4223.1 to cache LRU end 2023-10-05T19:28:32.993 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.986+0000 7f3999e0de40 10 updating xattr: name=user.rgw.acl bl.length()=123 2023-10-05T19:28:32.993 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.986+0000 7f3999e0de40 10 distributing notification oid=default.rgw.control:notify.7 cni=[op: 0, obj: default.rgw.meta:root:.bucket.meta.myfoo:9a9df965-6612-45d4-a45f-6e3c889bacc2.4223.1, ofs0, ns] 2023-10-05T19:28:32.994 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.986+0000 7f3999e0de40 1 -- 172.21.15.167:0/906877954 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:111 7.9 7:93e5b521:::notify.7:head [notify cookie 94001309575024 in=737b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x557e62aa2990 con 0x7f3980007030 2023-10-05T19:28:32.994 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.990+0000 7f3998143700 1 -- 172.21.15.167:0/906877954 <== osd.0 v2:172.21.15.118:6800/27690 32 ==== watch-notify(notify (1) cookie 94001308998896 notify 128849018884 ret 0) v3 ==== 767+0+0 (crc 0 0 0) 0x7f399003d040 con 0x7f3980007030 2023-10-05T19:28:32.994 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.990+0000 7f399593e700 10 rgw watcher librados: RGWWatcher::handle_notify() notify_id 128849018884 cookie 94001308998896 notifier 4455 bl.length()=725 2023-10-05T19:28:32.995 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.990+0000 7f399593e700 10 rgw watcher librados: cache put: name=default.rgw.meta+root+.bucket.meta.myfoo:9a9df965-6612-45d4-a45f-6e3c889bacc2.4223.1 info.flags=0x17 2023-10-05T19:28:32.995 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.990+0000 7f399593e700 10 rgw watcher librados: moving default.rgw.meta+root+.bucket.meta.myfoo:9a9df965-6612-45d4-a45f-6e3c889bacc2.4223.1 to cache LRU end 2023-10-05T19:28:32.995 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.990+0000 7f399593e700 10 rgw watcher librados: updating xattr: name=user.rgw.acl bl.length()=123 2023-10-05T19:28:32.995 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.990+0000 7f3998143700 1 -- 172.21.15.167:0/906877954 <== osd.0 v2:172.21.15.118:6800/27690 33 ==== osd_op_reply(111 notify.7 [notify cookie 94001309575024 out=8b] v0'0 uv114 ondisk = 0) v8 ==== 152+0+8 (crc 0 0 0) 0x7f3990050040 con 0x7f3980007030 2023-10-05T19:28:32.995 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.990+0000 7f399593e700 1 -- 172.21.15.167:0/906877954 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:112 7.9 7:93e5b521:::notify.7:head [notify-ack in=20b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x7f397c003d90 con 0x7f3980007030 2023-10-05T19:28:32.996 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.990+0000 7f3998143700 1 -- 172.21.15.167:0/906877954 <== osd.0 v2:172.21.15.118:6800/27690 34 ==== osd_op_reply(112 notify.7 [notify-ack] v0'0 uv114 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f3990050040 con 0x7f3980007030 2023-10-05T19:28:32.996 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.990+0000 7f3998143700 1 -- 172.21.15.167:0/906877954 <== osd.0 v2:172.21.15.118:6800/27690 35 ==== watch-notify(notify_complete (2) cookie 94001309575024 notify 128849018884 ret 0) v3 ==== 42+0+48 (crc 0 0 0) 0x7f3990002bd0 con 0x7f3980007030 2023-10-05T19:28:32.996 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.990+0000 7f3999e0de40 20 handle_bi_update(): bucket=:myfoo[9a9df965-6612-45d4-a45f-6e3c889bacc2.4223.1]): orig_sources= new_sources= 2023-10-05T19:28:32.996 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.990+0000 7f3999e0de40 20 handle_bi_update(): bucket=:myfoo[9a9df965-6612-45d4-a45f-6e3c889bacc2.4223.1]): potential sources added=[] removed=[] 2023-10-05T19:28:32.996 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.990+0000 7f3999e0de40 20 handle_bi_update(): bucket=:myfoo[9a9df965-6612-45d4-a45f-6e3c889bacc2.4223.1]): orig_dests= new_dests= 2023-10-05T19:28:32.996 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.990+0000 7f3999e0de40 20 handle_bi_update(): bucket=:myfoo[9a9df965-6612-45d4-a45f-6e3c889bacc2.4223.1]): potential dests added=[] removed=[] 2023-10-05T19:28:32.996 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.990+0000 7f3999e0de40 1 -- 172.21.15.167:0/906877954 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:113 8.1 8:810902b1:users.uid::fud.buckets:head [call user.set_buckets_info in=207b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x557e62aa6730 con 0x7f3980007030 2023-10-05T19:28:32.999 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.994+0000 7f3998143700 1 -- 172.21.15.167:0/906877954 <== osd.0 v2:172.21.15.118:6800/27690 36 ==== osd_op_reply(113 fud.buckets [call] v30'1 uv1 ondisk = 0) v8 ==== 155+0+0 (crc 0 0 0) 0x7f3990050040 con 0x7f3980007030 2023-10-05T19:28:32.999 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.994+0000 7f3999e0de40 1 -- 172.21.15.167:0/906877954 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:114 8.9 8:91b71841:root::myfoo:head [delete,create,call version.inc in=38b,writefull 0~165 in=165b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x557e62aa6730 con 0x7f398801d350 2023-10-05T19:28:33.002 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.994+0000 7f3997141700 1 -- 172.21.15.167:0/906877954 <== osd.2 v2:172.21.15.167:6816/30037 20 ==== osd_op_reply(114 myfoo [delete,create,call,writefull 0~165] v30'20 uv20 ondisk = 0) v8 ==== 275+0+0 (crc 0 0 0) 0x7f3980032040 con 0x7f398801d350 2023-10-05T19:28:33.002 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.994+0000 7f3999e0de40 10 cache put: name=default.rgw.meta+root+myfoo info.flags=0x7 2023-10-05T19:28:33.002 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.994+0000 7f3999e0de40 10 moving default.rgw.meta+root+myfoo to cache LRU end 2023-10-05T19:28:33.002 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.994+0000 7f3999e0de40 10 distributing notification oid=default.rgw.control:notify.6 cni=[op: 0, obj: default.rgw.meta:root:myfoo, ofs0, ns] 2023-10-05T19:28:33.002 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.994+0000 7f3999e0de40 1 -- 172.21.15.167:0/906877954 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:115 7.14 7:2b04a3e9:::notify.6:head [notify cookie 94001309574960 in=326b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x557e62aa0ef0 con 0x7f3980007030 2023-10-05T19:28:33.003 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.998+0000 7f3998143700 1 -- 172.21.15.167:0/906877954 <== osd.0 v2:172.21.15.118:6800/27690 37 ==== watch-notify(notify (1) cookie 94001308986528 notify 128849018885 ret 0) v3 ==== 356+0+0 (crc 0 0 0) 0x7f3990002bd0 con 0x7f3980007030 2023-10-05T19:28:33.003 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.998+0000 7f399593e700 10 rgw watcher librados: RGWWatcher::handle_notify() notify_id 128849018885 cookie 94001308986528 notifier 4455 bl.length()=314 2023-10-05T19:28:33.003 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.998+0000 7f399593e700 10 rgw watcher librados: cache put: name=default.rgw.meta+root+myfoo info.flags=0x7 2023-10-05T19:28:33.003 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.998+0000 7f399593e700 10 rgw watcher librados: moving default.rgw.meta+root+myfoo to cache LRU end 2023-10-05T19:28:33.003 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.998+0000 7f3998143700 1 -- 172.21.15.167:0/906877954 <== osd.0 v2:172.21.15.118:6800/27690 38 ==== osd_op_reply(115 notify.6 [notify cookie 94001309574960 out=8b] v0'0 uv114 ondisk = 0) v8 ==== 152+0+8 (crc 0 0 0) 0x7f3990050040 con 0x7f3980007030 2023-10-05T19:28:33.003 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.998+0000 7f399593e700 1 -- 172.21.15.167:0/906877954 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:116 7.14 7:2b04a3e9:::notify.6:head [notify-ack in=20b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x7f397c003d90 con 0x7f3980007030 2023-10-05T19:28:33.004 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.998+0000 7f3998143700 1 -- 172.21.15.167:0/906877954 <== osd.0 v2:172.21.15.118:6800/27690 39 ==== osd_op_reply(116 notify.6 [notify-ack] v0'0 uv114 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f3990050040 con 0x7f3980007030 2023-10-05T19:28:33.004 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.998+0000 7f3998143700 1 -- 172.21.15.167:0/906877954 <== osd.0 v2:172.21.15.118:6800/27690 40 ==== watch-notify(notify_complete (2) cookie 94001309574960 notify 128849018885 ret 0) v3 ==== 42+0+48 (crc 0 0 0) 0x7f39900334d0 con 0x7f3980007030 2023-10-05T19:28:33.004 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:32.998+0000 7f3999e0de40 1 -- 172.21.15.167:0/906877954 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:117 7.1f 7:f95f44c2:::notify.0:head [watch unwatch cookie 94001308976352] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x557e629ef700 con 0x7f398801d350 2023-10-05T19:28:33.007 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.002+0000 7f3997141700 1 -- 172.21.15.167:0/906877954 <== osd.2 v2:172.21.15.167:6816/30037 21 ==== osd_op_reply(117 notify.0 [watch unwatch cookie 94001308976352] v30'116 uv114 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f3980032040 con 0x7f398801d350 2023-10-05T19:28:33.007 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.002+0000 7f3999e0de40 20 remove_watcher() i=0 2023-10-05T19:28:33.007 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.002+0000 7f3999e0de40 2 removed watcher, disabling cache 2023-10-05T19:28:33.007 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.002+0000 7f3999e0de40 1 -- 172.21.15.167:0/906877954 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:118 7.0 7:05bf5b68:::notify.1:head [watch unwatch cookie 94001308979664] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x557e62a144e0 con 0x557e6294dbc0 2023-10-05T19:28:33.010 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.006+0000 7f3998143700 1 -- 172.21.15.167:0/906877954 <== osd.1 v2:172.21.15.167:6800/30035 35 ==== osd_op_reply(118 notify.1 [watch unwatch cookie 94001308979664] v30'116 uv114 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f3990050040 con 0x557e6294dbc0 2023-10-05T19:28:33.011 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.006+0000 7f3999e0de40 20 remove_watcher() i=1 2023-10-05T19:28:33.011 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.006+0000 7f3999e0de40 1 -- 172.21.15.167:0/906877954 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:119 7.15 7:a93a5511:::notify.2:head [watch unwatch cookie 94001308983376] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x557e629ef700 con 0x557e62a0f4c0 2023-10-05T19:28:33.013 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.006+0000 7f3997942700 1 -- 172.21.15.167:0/906877954 <== osd.3 v2:172.21.15.167:6808/30036 27 ==== osd_op_reply(119 notify.2 [watch unwatch cookie 94001308983376] v30'116 uv114 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f3988024020 con 0x557e62a0f4c0 2023-10-05T19:28:33.013 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.006+0000 7f3999e0de40 20 remove_watcher() i=2 2023-10-05T19:28:33.013 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.006+0000 7f3999e0de40 1 -- 172.21.15.167:0/906877954 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:120 7.e 7:7759931f:::notify.3:head [watch unwatch cookie 94001308987792] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x557e62a16050 con 0x7f398801d350 2023-10-05T19:28:33.016 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.010+0000 7f3997141700 1 -- 172.21.15.167:0/906877954 <== osd.2 v2:172.21.15.167:6816/30037 22 ==== osd_op_reply(120 notify.3 [watch unwatch cookie 94001308987792] v30'116 uv114 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f3980032040 con 0x7f398801d350 2023-10-05T19:28:33.016 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.010+0000 7f3999e0de40 20 remove_watcher() i=3 2023-10-05T19:28:33.016 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.010+0000 7f3999e0de40 1 -- 172.21.15.167:0/906877954 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:121 7.d 7:b4812045:::notify.4:head [watch unwatch cookie 94001308990000] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x557e629ef700 con 0x557e6294dbc0 2023-10-05T19:28:33.019 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.014+0000 7f3998143700 1 -- 172.21.15.167:0/906877954 <== osd.1 v2:172.21.15.167:6800/30035 36 ==== osd_op_reply(121 notify.4 [watch unwatch cookie 94001308990000] v30'116 uv114 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f3990050040 con 0x557e6294dbc0 2023-10-05T19:28:33.020 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.014+0000 7f3999e0de40 20 remove_watcher() i=4 2023-10-05T19:28:33.020 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.014+0000 7f3999e0de40 1 -- 172.21.15.167:0/906877954 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:122 7.3 7:c609908c:::notify.5:head [watch unwatch cookie 94001308995216] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x557e62a17190 con 0x7f3980007030 2023-10-05T19:28:33.022 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.018+0000 7f3998143700 1 -- 172.21.15.167:0/906877954 <== osd.0 v2:172.21.15.118:6800/27690 41 ==== osd_op_reply(122 notify.5 [watch unwatch cookie 94001308995216] v30'116 uv114 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f3990050040 con 0x7f3980007030 2023-10-05T19:28:33.023 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.018+0000 7f3999e0de40 20 remove_watcher() i=5 2023-10-05T19:28:33.023 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.018+0000 7f3999e0de40 1 -- 172.21.15.167:0/906877954 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:123 7.14 7:2b04a3e9:::notify.6:head [watch unwatch cookie 94001308986528] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x557e629ef700 con 0x7f3980007030 2023-10-05T19:28:33.025 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.018+0000 7f3998143700 1 -- 172.21.15.167:0/906877954 <== osd.0 v2:172.21.15.118:6800/27690 42 ==== osd_op_reply(123 notify.6 [watch unwatch cookie 94001308986528] v30'116 uv114 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f3990050040 con 0x7f3980007030 2023-10-05T19:28:33.025 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.018+0000 7f3999e0de40 20 remove_watcher() i=6 2023-10-05T19:28:33.025 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.018+0000 7f3999e0de40 1 -- 172.21.15.167:0/906877954 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:124 7.9 7:93e5b521:::notify.7:head [watch unwatch cookie 94001308998896] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x557e62a18e90 con 0x7f3980007030 2023-10-05T19:28:33.029 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.022+0000 7f3998143700 1 -- 172.21.15.167:0/906877954 <== osd.0 v2:172.21.15.118:6800/27690 43 ==== osd_op_reply(124 notify.7 [watch unwatch cookie 94001308998896] v30'116 uv114 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f3990050040 con 0x7f3980007030 2023-10-05T19:28:33.029 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.022+0000 7f3999e0de40 20 remove_watcher() i=7 2023-10-05T19:28:33.030 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.026+0000 7f39697e2700 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2023-10-05T19:28:33.031 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.026+0000 7f3999e0de40 1 -- 172.21.15.167:0/906877954 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:125 6.4 6:22d26bf9:::data_loggenerations_metadata:head [watch unwatch cookie 94001309000784] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x557e629ef700 con 0x557e6294dbc0 2023-10-05T19:28:33.034 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.026+0000 7f3998143700 1 -- 172.21.15.167:0/906877954 <== osd.1 v2:172.21.15.167:6800/30035 37 ==== osd_op_reply(125 data_loggenerations_metadata [watch unwatch cookie 94001309000784] v30'112 uv1 ondisk = 0) v8 ==== 172+0+0 (crc 0 0 0) 0x7f3990050040 con 0x557e6294dbc0 2023-10-05T19:28:33.034 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.026+0000 7f3999e0de40 1 -- 172.21.15.167:0/906877954 >> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] conn(0x7f3980007030 msgr2=0x7f398001c920 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-05T19:28:33.035 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.026+0000 7f3999e0de40 1 --2- 172.21.15.167:0/906877954 >> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] conn(0x7f3980007030 0x7f398001c920 crc :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-05T19:28:33.035 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.026+0000 7f3999e0de40 1 -- 172.21.15.167:0/906877954 >> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] conn(0x557e6294dbc0 msgr2=0x557e62a0b910 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-05T19:28:33.035 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.026+0000 7f3999e0de40 1 --2- 172.21.15.167:0/906877954 >> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] conn(0x557e6294dbc0 0x557e62a0b910 crc :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-05T19:28:33.035 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.026+0000 7f3999e0de40 1 -- 172.21.15.167:0/906877954 >> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] conn(0x7f398801d350 msgr2=0x7f3988043850 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-05T19:28:33.035 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.026+0000 7f3999e0de40 1 --2- 172.21.15.167:0/906877954 >> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] conn(0x7f398801d350 0x7f3988043850 crc :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-05T19:28:33.035 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.026+0000 7f3999e0de40 1 -- 172.21.15.167:0/906877954 >> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] conn(0x557e62a0f4c0 msgr2=0x557e62a118d0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-05T19:28:33.035 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.026+0000 7f3999e0de40 1 --2- 172.21.15.167:0/906877954 >> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] conn(0x557e62a0f4c0 0x557e62a118d0 crc :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-05T19:28:33.035 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.030+0000 7f3999e0de40 1 -- 172.21.15.167:0/906877954 >> [v2:172.21.15.167:6824/29889,v1:172.21.15.167:6825/29889] conn(0x7f3978030c50 msgr2=0x7f3978033100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-05T19:28:33.035 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.030+0000 7f3999e0de40 1 --2- 172.21.15.167:0/906877954 >> [v2:172.21.15.167:6824/29889,v1:172.21.15.167:6825/29889] conn(0x7f3978030c50 0x7f3978033100 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7f39800030a0 tx=0x7f3980015000 comp rx=0 tx=0).stop 2023-10-05T19:28:33.035 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.030+0000 7f3998143700 1 -- 172.21.15.167:0/906877954 reap_dead start 2023-10-05T19:28:33.036 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.030+0000 7f3999e0de40 1 -- 172.21.15.167:0/906877954 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x557e6294c8b0 msgr2=0x557e629ffd90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-05T19:28:33.036 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.030+0000 7f3999e0de40 1 --2- 172.21.15.167:0/906877954 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x557e6294c8b0 0x557e629ffd90 secure :-1 s=READY pgs=365 cs=0 l=1 rev1=1 crypto rx=0x7f3988000f80 tx=0x7f3988014000 comp rx=0 tx=0).stop 2023-10-05T19:28:33.036 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.030+0000 7f3999e0de40 1 -- 172.21.15.167:0/906877954 shutdown_connections 2023-10-05T19:28:33.036 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.030+0000 7f3999e0de40 1 --2- 172.21.15.167:0/906877954 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x557e6294c8b0 0x557e629ffd90 unknown :-1 s=CLOSED pgs=365 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-05T19:28:33.036 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.030+0000 7f3999e0de40 1 -- 172.21.15.167:0/906877954 >> 172.21.15.167:0/906877954 conn(0x557e629d37f0 msgr2=0x557e629f3570 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-05T19:28:33.036 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.030+0000 7f3999e0de40 1 -- 172.21.15.167:0/906877954 shutdown_connections 2023-10-05T19:28:33.036 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.030+0000 7f3999e0de40 1 -- 172.21.15.167:0/906877954 wait complete. 2023-10-05T19:28:33.037 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.030+0000 7f3999e0de40 1 -- 172.21.15.167:0/2402177238 >> [v2:172.21.15.167:6824/29889,v1:172.21.15.167:6825/29889] conn(0x7f38d8030ca0 msgr2=0x7f38d8033150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-05T19:28:33.037 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.030+0000 7f3999e0de40 1 --2- 172.21.15.167:0/2402177238 >> [v2:172.21.15.167:6824/29889,v1:172.21.15.167:6825/29889] conn(0x7f38d8030ca0 0x7f38d8033150 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x557e62a1ad90 tx=0x7f3990000f90 comp rx=0 tx=0).stop 2023-10-05T19:28:33.037 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.030+0000 7f3999e0de40 1 -- 172.21.15.167:0/2402177238 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x557e62a280c0 msgr2=0x557e62a2dbf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-05T19:28:33.037 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.030+0000 7f3999e0de40 1 --2- 172.21.15.167:0/2402177238 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x557e62a280c0 0x557e62a2dbf0 secure :-1 s=READY pgs=367 cs=0 l=1 rev1=1 crypto rx=0x7f3980012fd0 tx=0x7f398002f000 comp rx=0 tx=0).stop 2023-10-05T19:28:33.038 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.030+0000 7f3999e0de40 1 -- 172.21.15.167:0/2402177238 shutdown_connections 2023-10-05T19:28:33.038 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.030+0000 7f3999e0de40 1 --2- 172.21.15.167:0/2402177238 >> [v2:172.21.15.167:6824/29889,v1:172.21.15.167:6825/29889] conn(0x7f38d8030ca0 0x7f38d8033150 unknown :-1 s=CLOSED pgs=129 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-05T19:28:33.038 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.030+0000 7f3999e0de40 1 --2- 172.21.15.167:0/2402177238 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x557e62a280c0 0x557e62a2dbf0 unknown :-1 s=CLOSED pgs=367 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-05T19:28:33.038 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.030+0000 7f3999e0de40 1 -- 172.21.15.167:0/2402177238 >> 172.21.15.167:0/2402177238 conn(0x557e62a23480 msgr2=0x557e62a21520 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-05T19:28:33.038 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.030+0000 7f3999e0de40 1 -- 172.21.15.167:0/2402177238 shutdown_connections 2023-10-05T19:28:33.038 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.030+0000 7f3999e0de40 1 -- 172.21.15.167:0/2402177238 wait complete. 2023-10-05T19:28:33.041 INFO:tasks.util.rgw:rgwadmin: client.0 : ['user', 'rm', '--uid', 'fud'] 2023-10-05T19:28:33.041 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'ceph', 'user', 'rm', '--uid', 'fud'] 2023-10-05T19:28:33.041 DEBUG:teuthology.orchestra.run.smithi167:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster ceph user rm --uid fud 2023-10-05T19:28:33.080 INFO:teuthology.orchestra.run.smithi167.stderr:ignoring --setuser ceph since I am not root 2023-10-05T19:28:33.080 INFO:teuthology.orchestra.run.smithi167.stderr:ignoring --setgroup ceph since I am not root 2023-10-05T19:28:33.081 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.074+0000 7f2a55661e40 1 Processor -- start 2023-10-05T19:28:33.081 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.074+0000 7f2a55661e40 1 -- start start 2023-10-05T19:28:33.081 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.074+0000 7f2a55661e40 1 --2- >> v2:172.21.15.118:3300/0 conn(0x5569c2464410 0x5569c2462fc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-05T19:28:33.081 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.074+0000 7f2a55661e40 1 -- --> v1:172.21.15.118:6789/0 -- auth(proto 0 26 bytes epoch 0) v1 -- 0x5569c23b0bd0 con 0x5569c24647e0 2023-10-05T19:28:33.081 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.074+0000 7f2a55661e40 1 -- --> v2:172.21.15.118:3300/0 -- mon_getmap magic: 0 v1 -- 0x5569c236dee0 con 0x5569c2464410 2023-10-05T19:28:33.082 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.074+0000 7f2a53196700 1 --2- >> v2:172.21.15.118:3300/0 conn(0x5569c2464410 0x5569c2462fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-05T19:28:33.082 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.074+0000 7f2a52995700 1 --1- >> v1:172.21.15.118:6789/0 conn(0x5569c24647e0 0x5569c2463500 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.118:6789/0 says I am v1:172.21.15.167:56646/0 (socket says 172.21.15.167:56646) 2023-10-05T19:28:33.082 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.074+0000 7f2a52995700 1 -- 172.21.15.167:0/2390720628 learned_addr learned my addr 172.21.15.167:0/2390720628 (peer_addr_for_me v1:172.21.15.167:0/0) 2023-10-05T19:28:33.083 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.078+0000 7f2a52194700 1 -- 172.21.15.167:0/2390720628 <== mon.0 v1:172.21.15.118:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1399385436 0 0) 0x5569c23b0bd0 con 0x5569c24647e0 2023-10-05T19:28:33.083 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.078+0000 7f2a52194700 1 -- 172.21.15.167:0/2390720628 --> v1:172.21.15.118:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2a40001e10 con 0x5569c24647e0 2023-10-05T19:28:33.083 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.078+0000 7f2a53196700 1 -- 172.21.15.167:0/2390720628 >> v1:172.21.15.118:6789/0 conn(0x5569c24647e0 legacy=0x5569c2463500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-05T19:28:33.083 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.078+0000 7f2a53196700 1 -- 172.21.15.167:0/2390720628 --> v2:172.21.15.118:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5569c2463cd0 con 0x5569c2464410 2023-10-05T19:28:33.083 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.078+0000 7f2a53196700 1 --2- 172.21.15.167:0/2390720628 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5569c2464410 0x5569c2462fc0 secure :-1 s=READY pgs=369 cs=0 l=1 rev1=1 crypto rx=0x7f2a4c00a520 tx=0x7f2a4c005d40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=87e275c829c009a server_cookie=0 in_seq=0 out_seq=0 2023-10-05T19:28:33.083 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.078+0000 7f2a52194700 1 -- 172.21.15.167:0/2390720628 <== mon.0 v2:172.21.15.118:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f2a4c00f590 con 0x5569c2464410 2023-10-05T19:28:33.083 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.078+0000 7f2a52194700 1 -- 172.21.15.167:0/2390720628 <== mon.0 v2:172.21.15.118:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2a4c00f6f0 con 0x5569c2464410 2023-10-05T19:28:33.084 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.078+0000 7f2a52194700 1 -- 172.21.15.167:0/2390720628 <== mon.0 v2:172.21.15.118:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f2a4c00f860 con 0x5569c2464410 2023-10-05T19:28:33.085 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.078+0000 7f2a55661e40 1 -- 172.21.15.167:0/2390720628 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5569c2464410 msgr2=0x5569c2462fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-05T19:28:33.085 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.078+0000 7f2a55661e40 1 --2- 172.21.15.167:0/2390720628 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5569c2464410 0x5569c2462fc0 secure :-1 s=READY pgs=369 cs=0 l=1 rev1=1 crypto rx=0x7f2a4c00a520 tx=0x7f2a4c005d40 comp rx=0 tx=0).stop 2023-10-05T19:28:33.085 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.078+0000 7f2a55661e40 1 -- 172.21.15.167:0/2390720628 shutdown_connections 2023-10-05T19:28:33.085 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.078+0000 7f2a55661e40 1 --2- 172.21.15.167:0/2390720628 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5569c2464410 0x5569c2462fc0 unknown :-1 s=CLOSED pgs=369 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-05T19:28:33.085 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.078+0000 7f2a55661e40 1 -- 172.21.15.167:0/2390720628 >> 172.21.15.167:0/2390720628 conn(0x5569c245e720 msgr2=0x5569c2460b30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-05T19:28:33.085 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.078+0000 7f2a55661e40 1 -- 172.21.15.167:0/2390720628 shutdown_connections 2023-10-05T19:28:33.085 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.078+0000 7f2a55661e40 1 -- 172.21.15.167:0/2390720628 wait complete. 2023-10-05T19:28:33.085 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.078+0000 7f2a55661e40 1 Processor -- start 2023-10-05T19:28:33.085 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.078+0000 7f2a55661e40 1 -- start start 2023-10-05T19:28:33.085 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.078+0000 7f2a55661e40 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5569c2464410 0x5569c2484f10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-05T19:28:33.086 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.078+0000 7f2a55661e40 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x5569c2331e30 con 0x5569c2464410 2023-10-05T19:28:33.087 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.078+0000 7f2a53196700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5569c2464410 0x5569c2484f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-05T19:28:33.087 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.078+0000 7f2a53196700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5569c2464410 0x5569c2484f10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.118:3300/0 says I am v2:172.21.15.167:58550/0 (socket says 172.21.15.167:58550) 2023-10-05T19:28:33.087 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.078+0000 7f2a53196700 1 -- 172.21.15.167:0/282257115 learned_addr learned my addr 172.21.15.167:0/282257115 (peer_addr_for_me v2:172.21.15.167:0/0) 2023-10-05T19:28:33.087 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.082+0000 7f2a53196700 1 -- 172.21.15.167:0/282257115 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2a4c00a0e0 con 0x5569c2464410 2023-10-05T19:28:33.087 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.082+0000 7f2a53196700 1 --2- 172.21.15.167:0/282257115 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5569c2464410 0x5569c2484f10 secure :-1 s=READY pgs=370 cs=0 l=1 rev1=1 crypto rx=0x7f2a4c00a470 tx=0x7f2a4c014000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=56824811ac8b6a8a server_cookie=0 in_seq=0 out_seq=0 2023-10-05T19:28:33.087 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.082+0000 7f2a51192700 1 -- 172.21.15.167:0/282257115 <== mon.0 v2:172.21.15.118:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f2a4c00f860 con 0x5569c2464410 2023-10-05T19:28:33.087 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.082+0000 7f2a51192700 1 -- 172.21.15.167:0/282257115 <== mon.0 v2:172.21.15.118:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2a4c00f9c0 con 0x5569c2464410 2023-10-05T19:28:33.087 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.082+0000 7f2a51192700 1 -- 172.21.15.167:0/282257115 <== mon.0 v2:172.21.15.118:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f2a4c00fb70 con 0x5569c2464410 2023-10-05T19:28:33.089 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.082+0000 7f2a55661e40 1 -- 172.21.15.167:0/282257115 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5569c2464410 msgr2=0x5569c2484f10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-05T19:28:33.089 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.082+0000 7f2a55661e40 1 --2- 172.21.15.167:0/282257115 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5569c2464410 0x5569c2484f10 secure :-1 s=READY pgs=370 cs=0 l=1 rev1=1 crypto rx=0x7f2a4c00a470 tx=0x7f2a4c014000 comp rx=0 tx=0).stop 2023-10-05T19:28:33.089 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.082+0000 7f2a55661e40 1 -- 172.21.15.167:0/282257115 shutdown_connections 2023-10-05T19:28:33.089 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.082+0000 7f2a55661e40 1 --2- 172.21.15.167:0/282257115 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5569c2464410 0x5569c2484f10 unknown :-1 s=CLOSED pgs=370 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-05T19:28:33.089 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.082+0000 7f2a55661e40 1 -- 172.21.15.167:0/282257115 >> 172.21.15.167:0/282257115 conn(0x5569c245e720 msgr2=0x5569c2480ad0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-05T19:28:33.089 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.082+0000 7f2a55661e40 1 -- 172.21.15.167:0/282257115 shutdown_connections 2023-10-05T19:28:33.089 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.082+0000 7f2a55661e40 1 -- 172.21.15.167:0/282257115 wait complete. 2023-10-05T19:28:33.089 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.082+0000 7f2a55661e40 1 Processor -- start 2023-10-05T19:28:33.089 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.082+0000 7f2a55661e40 1 -- start start 2023-10-05T19:28:33.090 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.082+0000 7f2a55661e40 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5569c24647e0 0x5569c248bce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-05T19:28:33.090 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.082+0000 7f2a55661e40 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x5569c2304ed0 con 0x5569c24647e0 2023-10-05T19:28:33.090 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.082+0000 7f2a53196700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5569c24647e0 0x5569c248bce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-05T19:28:33.090 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.082+0000 7f2a53196700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5569c24647e0 0x5569c248bce0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.118:3300/0 says I am v2:172.21.15.167:58562/0 (socket says 172.21.15.167:58562) 2023-10-05T19:28:33.090 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.082+0000 7f2a53196700 1 -- 172.21.15.167:0/1691752437 learned_addr learned my addr 172.21.15.167:0/1691752437 (peer_addr_for_me v2:172.21.15.167:0/0) 2023-10-05T19:28:33.091 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.082+0000 7f2a53196700 1 -- 172.21.15.167:0/1691752437 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5569c2463cd0 con 0x5569c24647e0 2023-10-05T19:28:33.091 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.082+0000 7f2a53196700 1 --2- 172.21.15.167:0/1691752437 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5569c24647e0 0x5569c248bce0 secure :-1 s=READY pgs=371 cs=0 l=1 rev1=1 crypto rx=0x7f2a4c000f80 tx=0x7f2a4c01d000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-05T19:28:33.091 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.086+0000 7f2a4b7fe700 1 -- 172.21.15.167:0/1691752437 <== mon.0 v2:172.21.15.118:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f2a4c02f430 con 0x5569c24647e0 2023-10-05T19:28:33.091 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.086+0000 7f2a4b7fe700 1 -- 172.21.15.167:0/1691752437 <== mon.0 v2:172.21.15.118:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2a4c02f5b0 con 0x5569c24647e0 2023-10-05T19:28:33.091 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.086+0000 7f2a55661e40 1 -- 172.21.15.167:0/1691752437 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x5569c248c220 con 0x5569c24647e0 2023-10-05T19:28:33.091 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.086+0000 7f2a4b7fe700 1 -- 172.21.15.167:0/1691752437 <== mon.0 v2:172.21.15.118:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f2a4c02f760 con 0x5569c24647e0 2023-10-05T19:28:33.091 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.086+0000 7f2a55661e40 1 -- 172.21.15.167:0/1691752437 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x5569c248c660 con 0x5569c24647e0 2023-10-05T19:28:33.092 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.086+0000 7f2a55661e40 20 rados->read ofs=0 len=0 2023-10-05T19:28:33.092 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.086+0000 7f2a4b7fe700 1 -- 172.21.15.167:0/1691752437 <== mon.0 v2:172.21.15.118:3300/0 4 ==== mgrmap(e 5) v1 ==== 40329+0+0 (secure 0 0 0) 0x7f2a4c010420 con 0x5569c24647e0 2023-10-05T19:28:33.092 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.086+0000 7f2a4b7fe700 1 --2- 172.21.15.167:0/1691752437 >> [v2:172.21.15.167:6824/29889,v1:172.21.15.167:6825/29889] conn(0x7f2a34030c50 0x7f2a34033100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-05T19:28:33.093 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.086+0000 7f2a4b7fe700 1 -- 172.21.15.167:0/1691752437 <== mon.0 v2:172.21.15.118:3300/0 5 ==== osd_map(30..30 src has 1..30) v4 ==== 6057+0+0 (secure 0 0 0) 0x7f2a4c02b070 con 0x5569c24647e0 2023-10-05T19:28:33.093 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.086+0000 7f2a52995700 1 --2- 172.21.15.167:0/1691752437 >> [v2:172.21.15.167:6824/29889,v1:172.21.15.167:6825/29889] conn(0x7f2a34030c50 0x7f2a34033100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-05T19:28:33.093 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.086+0000 7f2a55661e40 1 --2- 172.21.15.167:0/1691752437 >> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] conn(0x5569c2495710 0x5569c2497af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-05T19:28:33.093 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.086+0000 7f2a55661e40 1 -- 172.21.15.167:0/1691752437 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:1 5.12 5:49953fa1:::default.realm:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x5569c24981a0 con 0x5569c2495710 2023-10-05T19:28:33.093 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.086+0000 7f2a53997700 1 --2- 172.21.15.167:0/1691752437 >> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] conn(0x5569c2495710 0x5569c2497af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-05T19:28:33.094 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.086+0000 7f2a52995700 1 --2- 172.21.15.167:0/1691752437 >> [v2:172.21.15.167:6824/29889,v1:172.21.15.167:6825/29889] conn(0x7f2a34030c50 0x7f2a34033100 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7f2a3c004170 tx=0x7f2a3c001400 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-05T19:28:33.094 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.090+0000 7f2a53997700 1 --2- 172.21.15.167:0/1691752437 >> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] conn(0x5569c2495710 0x5569c2497af0 crc :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-05T19:28:33.095 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.090+0000 7f2a53997700 1 -- 172.21.15.167:0/1691752437 <== osd.1 v2:172.21.15.167:6800/30035 1 ==== osd_op_reply(1 default.realm [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 ==== 157+0+0 (crc 0 0 0) 0x7f2a4400a500 con 0x5569c2495710 2023-10-05T19:28:33.095 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.090+0000 7f2a55661e40 20 rados_obj.operate() r=-2 bl.length=0 2023-10-05T19:28:33.095 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.090+0000 7f2a55661e40 20 realm 2023-10-05T19:28:33.095 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.090+0000 7f2a55661e40 20 rados->read ofs=0 len=0 2023-10-05T19:28:33.095 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.090+0000 7f2a55661e40 1 -- 172.21.15.167:0/1691752437 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:2 5.12 5:49953fa1:::default.realm:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x5569c2494d80 con 0x5569c2495710 2023-10-05T19:28:33.096 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.090+0000 7f2a53997700 1 -- 172.21.15.167:0/1691752437 <== osd.1 v2:172.21.15.167:6800/30035 2 ==== osd_op_reply(2 default.realm [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 ==== 157+0+0 (crc 0 0 0) 0x7f2a4400a500 con 0x5569c2495710 2023-10-05T19:28:33.096 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.090+0000 7f2a55661e40 20 rados_obj.operate() r=-2 bl.length=0 2023-10-05T19:28:33.096 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.090+0000 7f2a55661e40 4 RGWPeriod::init failed to init realm id : (2) No such file or directory 2023-10-05T19:28:33.096 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.090+0000 7f2a55661e40 20 rados->read ofs=0 len=0 2023-10-05T19:28:33.096 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.090+0000 7f2a55661e40 1 -- 172.21.15.167:0/1691752437 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:3 5.12 5:49953fa1:::default.realm:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x5569c2498c80 con 0x5569c2495710 2023-10-05T19:28:33.096 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.090+0000 7f2a53997700 1 -- 172.21.15.167:0/1691752437 <== osd.1 v2:172.21.15.167:6800/30035 3 ==== osd_op_reply(3 default.realm [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 ==== 157+0+0 (crc 0 0 0) 0x7f2a4400a500 con 0x5569c2495710 2023-10-05T19:28:33.096 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.090+0000 7f2a55661e40 20 rados_obj.operate() r=-2 bl.length=0 2023-10-05T19:28:33.097 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.090+0000 7f2a55661e40 20 rados->read ofs=0 len=0 2023-10-05T19:28:33.097 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.090+0000 7f2a55661e40 1 -- 172.21.15.167:0/1691752437 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:4 5.f 5:f43b6ece:::zone_names.default:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x5569c2498890 con 0x5569c2495710 2023-10-05T19:28:33.098 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.090+0000 7f2a53997700 1 -- 172.21.15.167:0/1691752437 <== osd.1 v2:172.21.15.167:6800/30035 4 ==== osd_op_reply(4 zone_names.default [read 0~46 out=46b] v0'0 uv1 ondisk = 0) v8 ==== 162+0+46 (crc 0 0 0) 0x7f2a4400a500 con 0x5569c2495710 2023-10-05T19:28:33.098 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.090+0000 7f2a55661e40 20 rados_obj.operate() r=0 bl.length=46 2023-10-05T19:28:33.098 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.090+0000 7f2a55661e40 20 rados->read ofs=0 len=0 2023-10-05T19:28:33.098 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.090+0000 7f2a55661e40 1 -- 172.21.15.167:0/1691752437 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:5 5.1d 5:b8374eb4:::zone_info.9a9df965-6612-45d4-a45f-6e3c889bacc2:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x5569c2499b10 con 0x5569c2495710 2023-10-05T19:28:33.098 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.090+0000 7f2a53997700 1 -- 172.21.15.167:0/1691752437 <== osd.1 v2:172.21.15.167:6800/30035 5 ==== osd_op_reply(5 zone_info.9a9df965-6612-45d4-a45f-6e3c889bacc2 [read 0~874 out=874b] v0'0 uv2 ondisk = 0) v8 ==== 190+0+874 (crc 0 0 0) 0x7f2a4400a500 con 0x5569c2495710 2023-10-05T19:28:33.099 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.090+0000 7f2a55661e40 20 rados_obj.operate() r=0 bl.length=874 2023-10-05T19:28:33.099 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.090+0000 7f2a55661e40 20 searching for the correct realm 2023-10-05T19:28:33.099 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.090+0000 7f2a55661e40 1 --2- 172.21.15.167:0/1691752437 >> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] conn(0x5569c249b8d0 0x5569c249dcb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-05T19:28:33.099 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.090+0000 7f2a55661e40 1 -- 172.21.15.167:0/1691752437 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:6 5.0 5:00000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x5569c249e1f0 con 0x5569c249b8d0 2023-10-05T19:28:33.099 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.090+0000 7f2a53196700 1 --2- 172.21.15.167:0/1691752437 >> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] conn(0x5569c249b8d0 0x5569c249dcb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-05T19:28:33.099 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.094+0000 7f2a53196700 1 --2- 172.21.15.167:0/1691752437 >> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] conn(0x5569c249b8d0 0x5569c249dcb0 crc :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-05T19:28:33.101 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.094+0000 7f2a53196700 1 -- 172.21.15.167:0/1691752437 <== osd.3 v2:172.21.15.167:6808/30036 1 ==== osd_op_reply(6 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f2a4c035070 con 0x5569c249b8d0 2023-10-05T19:28:33.101 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.094+0000 7f2a53196700 1 --2- 172.21.15.167:0/1691752437 >> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] conn(0x7f2a4c038030 0x7f2a4c0079b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-05T19:28:33.101 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.094+0000 7f2a53196700 1 -- 172.21.15.167:0/1691752437 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:7 5.10 5:08000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x5569c249e1f0 con 0x7f2a4c038030 2023-10-05T19:28:33.101 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.094+0000 7f2a52995700 1 --2- 172.21.15.167:0/1691752437 >> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] conn(0x7f2a4c038030 0x7f2a4c0079b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-05T19:28:33.101 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.094+0000 7f2a52995700 1 --2- 172.21.15.167:0/1691752437 >> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] conn(0x7f2a4c038030 0x7f2a4c0079b0 crc :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-05T19:28:33.102 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.098+0000 7f2a52995700 1 -- 172.21.15.167:0/1691752437 <== osd.2 v2:172.21.15.167:6816/30037 1 ==== osd_op_reply(7 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f2a3c006070 con 0x7f2a4c038030 2023-10-05T19:28:33.102 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.098+0000 7f2a52995700 1 --2- 172.21.15.167:0/1691752437 >> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] conn(0x7f2a3c00ebe0 0x7f2a3c00d8e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-05T19:28:33.102 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.098+0000 7f2a52995700 1 -- 172.21.15.167:0/1691752437 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:8 5.8 5:10000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x5569c249e1f0 con 0x7f2a3c00ebe0 2023-10-05T19:28:33.102 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.098+0000 7f2a53997700 1 --2- 172.21.15.167:0/1691752437 >> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] conn(0x7f2a3c00ebe0 0x7f2a3c00d8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-05T19:28:33.103 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.098+0000 7f2a53997700 1 --2- 172.21.15.167:0/1691752437 >> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] conn(0x7f2a3c00ebe0 0x7f2a3c00d8e0 crc :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-05T19:28:33.104 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.098+0000 7f2a53997700 1 -- 172.21.15.167:0/1691752437 <== osd.0 v2:172.21.15.118:6800/27690 1 ==== osd_op_reply(8 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f2a44014040 con 0x7f2a3c00ebe0 2023-10-05T19:28:33.104 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.098+0000 7f2a53997700 1 -- 172.21.15.167:0/1691752437 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:9 5.18 5:18000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x5569c249e1f0 con 0x5569c2495710 2023-10-05T19:28:33.105 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.098+0000 7f2a53997700 1 -- 172.21.15.167:0/1691752437 <== osd.1 v2:172.21.15.167:6800/30035 6 ==== osd_op_reply(9 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f2a44014040 con 0x5569c2495710 2023-10-05T19:28:33.105 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.098+0000 7f2a53997700 1 -- 172.21.15.167:0/1691752437 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:10 5.4 5:20000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x5569c249e1f0 con 0x5569c2495710 2023-10-05T19:28:33.105 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.098+0000 7f2a53997700 1 -- 172.21.15.167:0/1691752437 <== osd.1 v2:172.21.15.167:6800/30035 7 ==== osd_op_reply(10 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f2a44014040 con 0x5569c2495710 2023-10-05T19:28:33.105 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.098+0000 7f2a53997700 1 -- 172.21.15.167:0/1691752437 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:11 5.14 5:28000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x5569c249e1f0 con 0x7f2a3c00ebe0 2023-10-05T19:28:33.106 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.102+0000 7f2a53997700 1 -- 172.21.15.167:0/1691752437 <== osd.0 v2:172.21.15.118:6800/27690 2 ==== osd_op_reply(11 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f2a44014040 con 0x7f2a3c00ebe0 2023-10-05T19:28:33.106 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.102+0000 7f2a53997700 1 -- 172.21.15.167:0/1691752437 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:12 5.c 5:30000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x5569c249e1f0 con 0x7f2a3c00ebe0 2023-10-05T19:28:33.107 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.102+0000 7f2a53997700 1 -- 172.21.15.167:0/1691752437 <== osd.0 v2:172.21.15.118:6800/27690 3 ==== osd_op_reply(12 [pgnls start_epoch 30 out=112b] v23'1 uv1 ondisk = 1) v8 ==== 144+0+112 (crc 0 0 0) 0x7f2a44014040 con 0x7f2a3c00ebe0 2023-10-05T19:28:33.107 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.102+0000 7f2a53997700 1 -- 172.21.15.167:0/1691752437 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:13 5.1c 5:38000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x5569c249e1f0 con 0x7f2a4c038030 2023-10-05T19:28:33.107 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.102+0000 7f2a52995700 1 -- 172.21.15.167:0/1691752437 <== osd.2 v2:172.21.15.167:6816/30037 2 ==== osd_op_reply(13 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f2a3c006070 con 0x7f2a4c038030 2023-10-05T19:28:33.108 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.102+0000 7f2a52995700 1 -- 172.21.15.167:0/1691752437 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:14 5.2 5:40000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x5569c249e1f0 con 0x5569c249b8d0 2023-10-05T19:28:33.108 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.102+0000 7f2a53196700 1 -- 172.21.15.167:0/1691752437 <== osd.3 v2:172.21.15.167:6808/30036 2 ==== osd_op_reply(14 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f2a4c035070 con 0x5569c249b8d0 2023-10-05T19:28:33.108 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.102+0000 7f2a53196700 1 -- 172.21.15.167:0/1691752437 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:15 5.12 5:48000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x5569c249e1f0 con 0x5569c2495710 2023-10-05T19:28:33.109 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.102+0000 7f2a53997700 1 -- 172.21.15.167:0/1691752437 <== osd.1 v2:172.21.15.167:6800/30035 8 ==== osd_op_reply(15 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f2a44014040 con 0x5569c2495710 2023-10-05T19:28:33.109 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.102+0000 7f2a53997700 1 -- 172.21.15.167:0/1691752437 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:16 5.a 5:50000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x5569c249e1f0 con 0x7f2a3c00ebe0 2023-10-05T19:28:33.110 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.106+0000 7f2a53997700 1 -- 172.21.15.167:0/1691752437 <== osd.0 v2:172.21.15.118:6800/27690 4 ==== osd_op_reply(16 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f2a44022040 con 0x7f2a3c00ebe0 2023-10-05T19:28:33.110 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.106+0000 7f2a53997700 1 -- 172.21.15.167:0/1691752437 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:17 5.1a 5:58000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x5569c249e1f0 con 0x5569c2495710 2023-10-05T19:28:33.111 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.106+0000 7f2a53997700 1 -- 172.21.15.167:0/1691752437 <== osd.1 v2:172.21.15.167:6800/30035 9 ==== osd_op_reply(17 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f2a44022040 con 0x5569c2495710 2023-10-05T19:28:33.111 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.106+0000 7f2a53997700 1 -- 172.21.15.167:0/1691752437 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:18 5.6 5:60000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x5569c249e1f0 con 0x7f2a4c038030 2023-10-05T19:28:33.111 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.106+0000 7f2a52995700 1 -- 172.21.15.167:0/1691752437 <== osd.2 v2:172.21.15.167:6816/30037 3 ==== osd_op_reply(18 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f2a3c006070 con 0x7f2a4c038030 2023-10-05T19:28:33.111 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.106+0000 7f2a52995700 1 -- 172.21.15.167:0/1691752437 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:19 5.16 5:68000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x5569c249e1f0 con 0x5569c249b8d0 2023-10-05T19:28:33.112 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.106+0000 7f2a53196700 1 -- 172.21.15.167:0/1691752437 <== osd.3 v2:172.21.15.167:6808/30036 3 ==== osd_op_reply(19 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f2a4c035070 con 0x5569c249b8d0 2023-10-05T19:28:33.112 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.106+0000 7f2a53196700 1 -- 172.21.15.167:0/1691752437 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:20 5.e 5:70000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x5569c249e1f0 con 0x5569c249b8d0 2023-10-05T19:28:33.113 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.106+0000 7f2a53196700 1 -- 172.21.15.167:0/1691752437 <== osd.3 v2:172.21.15.167:6808/30036 4 ==== osd_op_reply(20 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f2a4c035070 con 0x5569c249b8d0 2023-10-05T19:28:33.113 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.106+0000 7f2a53196700 1 -- 172.21.15.167:0/1691752437 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:21 5.1e 5:78000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x5569c249e1f0 con 0x7f2a3c00ebe0 2023-10-05T19:28:33.114 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.106+0000 7f2a53997700 1 -- 172.21.15.167:0/1691752437 <== osd.0 v2:172.21.15.118:6800/27690 5 ==== osd_op_reply(21 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f2a44022040 con 0x7f2a3c00ebe0 2023-10-05T19:28:33.114 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.106+0000 7f2a53997700 1 -- 172.21.15.167:0/1691752437 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:22 5.1 5:80000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x5569c249e1f0 con 0x5569c249b8d0 2023-10-05T19:28:33.115 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.110+0000 7f2a53196700 1 -- 172.21.15.167:0/1691752437 <== osd.3 v2:172.21.15.167:6808/30036 5 ==== osd_op_reply(22 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f2a4c035070 con 0x5569c249b8d0 2023-10-05T19:28:33.115 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.110+0000 7f2a53196700 1 -- 172.21.15.167:0/1691752437 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:23 5.11 5:88000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x5569c249e1f0 con 0x5569c249b8d0 2023-10-05T19:28:33.115 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.110+0000 7f2a53196700 1 -- 172.21.15.167:0/1691752437 <== osd.3 v2:172.21.15.167:6808/30036 6 ==== osd_op_reply(23 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f2a4c035070 con 0x5569c249b8d0 2023-10-05T19:28:33.115 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.110+0000 7f2a53196700 1 -- 172.21.15.167:0/1691752437 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:24 5.9 5:90000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x5569c249e1f0 con 0x5569c2495710 2023-10-05T19:28:33.116 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.110+0000 7f2a53997700 1 -- 172.21.15.167:0/1691752437 <== osd.1 v2:172.21.15.167:6800/30035 10 ==== osd_op_reply(24 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f2a44022040 con 0x5569c2495710 2023-10-05T19:28:33.116 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.110+0000 7f2a53997700 1 -- 172.21.15.167:0/1691752437 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:25 5.19 5:98000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x5569c249e1f0 con 0x5569c2495710 2023-10-05T19:28:33.117 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.110+0000 7f2a53997700 1 -- 172.21.15.167:0/1691752437 <== osd.1 v2:172.21.15.167:6800/30035 11 ==== osd_op_reply(25 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f2a44022040 con 0x5569c2495710 2023-10-05T19:28:33.117 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.110+0000 7f2a53997700 1 -- 172.21.15.167:0/1691752437 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:26 5.5 5:a0000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x5569c249e1f0 con 0x7f2a3c00ebe0 2023-10-05T19:28:33.118 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.114+0000 7f2a53997700 1 -- 172.21.15.167:0/1691752437 <== osd.0 v2:172.21.15.118:6800/27690 6 ==== osd_op_reply(26 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f2a44022040 con 0x7f2a3c00ebe0 2023-10-05T19:28:33.118 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.114+0000 7f2a53997700 1 -- 172.21.15.167:0/1691752437 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:27 5.15 5:a8000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x5569c249e1f0 con 0x5569c249b8d0 2023-10-05T19:28:33.119 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.114+0000 7f2a53196700 1 -- 172.21.15.167:0/1691752437 <== osd.3 v2:172.21.15.167:6808/30036 7 ==== osd_op_reply(27 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f2a4c035070 con 0x5569c249b8d0 2023-10-05T19:28:33.119 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.114+0000 7f2a53196700 1 -- 172.21.15.167:0/1691752437 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:28 5.d 5:b0000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x5569c249e1f0 con 0x7f2a4c038030 2023-10-05T19:28:33.119 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.114+0000 7f2a52995700 1 -- 172.21.15.167:0/1691752437 <== osd.2 v2:172.21.15.167:6816/30037 4 ==== osd_op_reply(28 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f2a3c006070 con 0x7f2a4c038030 2023-10-05T19:28:33.119 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.114+0000 7f2a52995700 1 -- 172.21.15.167:0/1691752437 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:29 5.1d 5:b8000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x5569c249e1f0 con 0x5569c2495710 2023-10-05T19:28:33.120 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.114+0000 7f2a53997700 1 -- 172.21.15.167:0/1691752437 <== osd.1 v2:172.21.15.167:6800/30035 12 ==== osd_op_reply(29 [pgnls start_epoch 30 out=107b] v30'2 uv2 ondisk = 1) v8 ==== 144+0+107 (crc 0 0 0) 0x7f2a44022040 con 0x5569c2495710 2023-10-05T19:28:33.120 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.114+0000 7f2a53997700 1 -- 172.21.15.167:0/1691752437 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:30 5.3 5:c0000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x5569c249e1f0 con 0x7f2a3c00ebe0 2023-10-05T19:28:33.121 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.114+0000 7f2a53997700 1 -- 172.21.15.167:0/1691752437 <== osd.0 v2:172.21.15.118:6800/27690 7 ==== osd_op_reply(30 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f2a44022040 con 0x7f2a3c00ebe0 2023-10-05T19:28:33.121 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.114+0000 7f2a53997700 1 -- 172.21.15.167:0/1691752437 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:31 5.13 5:c8000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x5569c249e1f0 con 0x5569c249b8d0 2023-10-05T19:28:33.121 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.114+0000 7f2a53196700 1 -- 172.21.15.167:0/1691752437 <== osd.3 v2:172.21.15.167:6808/30036 8 ==== osd_op_reply(31 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f2a4c035070 con 0x5569c249b8d0 2023-10-05T19:28:33.122 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.114+0000 7f2a53196700 1 -- 172.21.15.167:0/1691752437 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:32 5.b 5:d0000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x5569c249e1f0 con 0x7f2a3c00ebe0 2023-10-05T19:28:33.122 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.118+0000 7f2a53997700 1 -- 172.21.15.167:0/1691752437 <== osd.0 v2:172.21.15.118:6800/27690 8 ==== osd_op_reply(32 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f2a44022040 con 0x7f2a3c00ebe0 2023-10-05T19:28:33.123 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.118+0000 7f2a53997700 1 -- 172.21.15.167:0/1691752437 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:33 5.1b 5:d8000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x5569c249e1f0 con 0x7f2a4c038030 2023-10-05T19:28:33.123 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.118+0000 7f2a52995700 1 -- 172.21.15.167:0/1691752437 <== osd.2 v2:172.21.15.167:6816/30037 5 ==== osd_op_reply(33 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f2a3c006070 con 0x7f2a4c038030 2023-10-05T19:28:33.123 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.118+0000 7f2a52995700 1 -- 172.21.15.167:0/1691752437 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:34 5.7 5:e0000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x5569c249e1f0 con 0x5569c249b8d0 2023-10-05T19:28:33.124 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.118+0000 7f2a53196700 1 -- 172.21.15.167:0/1691752437 <== osd.3 v2:172.21.15.167:6808/30036 9 ==== osd_op_reply(34 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f2a4c035070 con 0x5569c249b8d0 2023-10-05T19:28:33.124 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.118+0000 7f2a53196700 1 -- 172.21.15.167:0/1691752437 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:35 5.17 5:e8000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x5569c249e1f0 con 0x7f2a3c00ebe0 2023-10-05T19:28:33.125 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.118+0000 7f2a53997700 1 -- 172.21.15.167:0/1691752437 <== osd.0 v2:172.21.15.118:6800/27690 9 ==== osd_op_reply(35 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f2a44022040 con 0x7f2a3c00ebe0 2023-10-05T19:28:33.125 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.118+0000 7f2a53997700 1 -- 172.21.15.167:0/1691752437 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:36 5.f 5:f0000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x5569c249e1f0 con 0x5569c2495710 2023-10-05T19:28:33.126 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.122+0000 7f2a53997700 1 -- 172.21.15.167:0/1691752437 <== osd.1 v2:172.21.15.167:6800/30035 13 ==== osd_op_reply(36 [pgnls start_epoch 30 out=115b] v23'2 uv2 ondisk = 1) v8 ==== 144+0+115 (crc 0 0 0) 0x7f2a4402f080 con 0x5569c2495710 2023-10-05T19:28:33.126 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.122+0000 7f2a53997700 1 -- 172.21.15.167:0/1691752437 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:37 5.1f 5:f8000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x5569c249e1f0 con 0x7f2a4c038030 2023-10-05T19:28:33.127 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.122+0000 7f2a52995700 1 -- 172.21.15.167:0/1691752437 <== osd.2 v2:172.21.15.167:6816/30037 6 ==== osd_op_reply(37 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f2a3c006070 con 0x7f2a4c038030 2023-10-05T19:28:33.127 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.122+0000 7f2a55661e40 20 RGWRados::pool_iterate: got zonegroup_info.4f1f719d-9648-417b-b10c-e5d939825419 2023-10-05T19:28:33.127 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.122+0000 7f2a55661e40 20 RGWRados::pool_iterate: got zone_info.9a9df965-6612-45d4-a45f-6e3c889bacc2 2023-10-05T19:28:33.127 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.122+0000 7f2a55661e40 20 RGWRados::pool_iterate: got zone_names.default 2023-10-05T19:28:33.127 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.122+0000 7f2a55661e40 20 RGWRados::pool_iterate: got zonegroups_names.default 2023-10-05T19:28:33.127 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.122+0000 7f2a55661e40 20 rados->read ofs=0 len=0 2023-10-05T19:28:33.127 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.122+0000 7f2a55661e40 1 -- 172.21.15.167:0/1691752437 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:38 5.12 5:49953fa1:::default.realm:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x5569c249a680 con 0x5569c2495710 2023-10-05T19:28:33.128 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.122+0000 7f2a53997700 1 -- 172.21.15.167:0/1691752437 <== osd.1 v2:172.21.15.167:6800/30035 14 ==== osd_op_reply(38 default.realm [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 ==== 157+0+0 (crc 0 0 0) 0x7f2a4402f080 con 0x5569c2495710 2023-10-05T19:28:33.128 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.122+0000 7f2a55661e40 20 rados_obj.operate() r=-2 bl.length=0 2023-10-05T19:28:33.128 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.122+0000 7f2a55661e40 20 rados->read ofs=0 len=0 2023-10-05T19:28:33.128 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.122+0000 7f2a55661e40 1 -- 172.21.15.167:0/1691752437 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:39 5.f 5:f4c53578:::zonegroups_names.default:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x5569c249af70 con 0x5569c2495710 2023-10-05T19:28:33.129 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.122+0000 7f2a53997700 1 -- 172.21.15.167:0/1691752437 <== osd.1 v2:172.21.15.167:6800/30035 15 ==== osd_op_reply(39 zonegroups_names.default [read 0~46 out=46b] v0'0 uv2 ondisk = 0) v8 ==== 168+0+46 (crc 0 0 0) 0x7f2a4402f080 con 0x5569c2495710 2023-10-05T19:28:33.129 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.122+0000 7f2a55661e40 20 rados_obj.operate() r=0 bl.length=46 2023-10-05T19:28:33.129 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.122+0000 7f2a55661e40 20 rados->read ofs=0 len=0 2023-10-05T19:28:33.129 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.122+0000 7f2a55661e40 1 -- 172.21.15.167:0/1691752437 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:40 5.c 5:30d299a5:::zonegroup_info.4f1f719d-9648-417b-b10c-e5d939825419:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x5569c249e630 con 0x7f2a3c00ebe0 2023-10-05T19:28:33.130 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.122+0000 7f2a53997700 1 -- 172.21.15.167:0/1691752437 <== osd.0 v2:172.21.15.118:6800/27690 10 ==== osd_op_reply(40 zonegroup_info.4f1f719d-9648-417b-b10c-e5d939825419 [read 0~362 out=362b] v0'0 uv1 ondisk = 0) v8 ==== 195+0+362 (crc 0 0 0) 0x7f2a4402f080 con 0x7f2a3c00ebe0 2023-10-05T19:28:33.130 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.122+0000 7f2a55661e40 20 rados_obj.operate() r=0 bl.length=362 2023-10-05T19:28:33.130 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.122+0000 7f2a55661e40 20 rados->read ofs=0 len=0 2023-10-05T19:28:33.130 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.122+0000 7f2a55661e40 1 -- 172.21.15.167:0/1691752437 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:41 5.13 5:cbbf6c60:::converted:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x5569c249e630 con 0x5569c249b8d0 2023-10-05T19:28:33.130 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.122+0000 7f2a53196700 1 -- 172.21.15.167:0/1691752437 <== osd.3 v2:172.21.15.167:6808/30036 10 ==== osd_op_reply(41 converted [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 ==== 153+0+0 (crc 0 0 0) 0x7f2a4c035070 con 0x5569c249b8d0 2023-10-05T19:28:33.131 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.126+0000 7f2a55661e40 20 rados_obj.operate() r=-2 bl.length=0 2023-10-05T19:28:33.131 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.126+0000 7f2a55661e40 20 rados->read ofs=0 len=0 2023-10-05T19:28:33.131 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.126+0000 7f2a55661e40 1 -- 172.21.15.167:0/1691752437 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:42 5.12 5:49953fa1:::default.realm:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x5569c249e9e0 con 0x5569c2495710 2023-10-05T19:28:33.132 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.126+0000 7f2a53997700 1 -- 172.21.15.167:0/1691752437 <== osd.1 v2:172.21.15.167:6800/30035 16 ==== osd_op_reply(42 default.realm [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 ==== 157+0+0 (crc 0 0 0) 0x7f2a4402f080 con 0x5569c2495710 2023-10-05T19:28:33.132 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.126+0000 7f2a55661e40 20 rados_obj.operate() r=-2 bl.length=0 2023-10-05T19:28:33.132 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.126+0000 7f2a55661e40 20 rados->read ofs=0 len=0 2023-10-05T19:28:33.132 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.126+0000 7f2a55661e40 1 -- 172.21.15.167:0/1691752437 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:43 5.f 5:f4c53578:::zonegroups_names.default:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x5569c249ede0 con 0x5569c2495710 2023-10-05T19:28:33.132 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.126+0000 7f2a53997700 1 -- 172.21.15.167:0/1691752437 <== osd.1 v2:172.21.15.167:6800/30035 17 ==== osd_op_reply(43 zonegroups_names.default [read 0~46 out=46b] v0'0 uv2 ondisk = 0) v8 ==== 168+0+46 (crc 0 0 0) 0x7f2a4402f080 con 0x5569c2495710 2023-10-05T19:28:33.132 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.126+0000 7f2a55661e40 20 rados_obj.operate() r=0 bl.length=46 2023-10-05T19:28:33.133 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.126+0000 7f2a55661e40 1 -- 172.21.15.167:0/1691752437 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:44 5.0 5:00000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x5569c249fdf0 con 0x5569c249b8d0 2023-10-05T19:28:33.134 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.126+0000 7f2a53196700 1 -- 172.21.15.167:0/1691752437 <== osd.3 v2:172.21.15.167:6808/30036 11 ==== osd_op_reply(44 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f2a4c035070 con 0x5569c249b8d0 2023-10-05T19:28:33.134 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.126+0000 7f2a53196700 1 -- 172.21.15.167:0/1691752437 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:45 5.10 5:08000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x5569c249fdf0 con 0x7f2a4c038030 2023-10-05T19:28:33.134 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.130+0000 7f2a52995700 1 -- 172.21.15.167:0/1691752437 <== osd.2 v2:172.21.15.167:6816/30037 7 ==== osd_op_reply(45 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f2a3c006070 con 0x7f2a4c038030 2023-10-05T19:28:33.134 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.130+0000 7f2a52995700 1 -- 172.21.15.167:0/1691752437 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:46 5.8 5:10000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x5569c249fdf0 con 0x7f2a3c00ebe0 2023-10-05T19:28:33.135 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.130+0000 7f2a53997700 1 -- 172.21.15.167:0/1691752437 <== osd.0 v2:172.21.15.118:6800/27690 11 ==== osd_op_reply(46 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f2a4402f080 con 0x7f2a3c00ebe0 2023-10-05T19:28:33.135 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.130+0000 7f2a53997700 1 -- 172.21.15.167:0/1691752437 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:47 5.18 5:18000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x5569c249af70 con 0x5569c2495710 2023-10-05T19:28:33.136 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.130+0000 7f2a53997700 1 -- 172.21.15.167:0/1691752437 <== osd.1 v2:172.21.15.167:6800/30035 18 ==== osd_op_reply(47 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f2a4402f080 con 0x5569c2495710 2023-10-05T19:28:33.136 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.130+0000 7f2a53997700 1 -- 172.21.15.167:0/1691752437 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:48 5.4 5:20000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x5569c249af70 con 0x5569c2495710 2023-10-05T19:28:33.137 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.130+0000 7f2a53997700 1 -- 172.21.15.167:0/1691752437 <== osd.1 v2:172.21.15.167:6800/30035 19 ==== osd_op_reply(48 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f2a4402f080 con 0x5569c2495710 2023-10-05T19:28:33.137 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.130+0000 7f2a53997700 1 -- 172.21.15.167:0/1691752437 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:49 5.14 5:28000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x5569c249af70 con 0x7f2a3c00ebe0 2023-10-05T19:28:33.138 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.130+0000 7f2a53997700 1 -- 172.21.15.167:0/1691752437 <== osd.0 v2:172.21.15.118:6800/27690 12 ==== osd_op_reply(49 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f2a4402f080 con 0x7f2a3c00ebe0 2023-10-05T19:28:33.138 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.134+0000 7f2a53997700 1 -- 172.21.15.167:0/1691752437 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:50 5.c 5:30000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x5569c249af70 con 0x7f2a3c00ebe0 2023-10-05T19:28:33.139 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.134+0000 7f2a53997700 1 -- 172.21.15.167:0/1691752437 <== osd.0 v2:172.21.15.118:6800/27690 13 ==== osd_op_reply(50 [pgnls start_epoch 30 out=112b] v23'1 uv1 ondisk = 1) v8 ==== 144+0+112 (crc 0 0 0) 0x7f2a4402f080 con 0x7f2a3c00ebe0 2023-10-05T19:28:33.139 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.134+0000 7f2a53997700 1 -- 172.21.15.167:0/1691752437 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:51 5.1c 5:38000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x5569c249af70 con 0x7f2a4c038030 2023-10-05T19:28:33.139 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.134+0000 7f2a52995700 1 -- 172.21.15.167:0/1691752437 <== osd.2 v2:172.21.15.167:6816/30037 8 ==== osd_op_reply(51 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f2a3c006070 con 0x7f2a4c038030 2023-10-05T19:28:33.140 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.134+0000 7f2a52995700 1 -- 172.21.15.167:0/1691752437 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:52 5.2 5:40000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x5569c249af70 con 0x5569c249b8d0 2023-10-05T19:28:33.140 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.134+0000 7f2a53196700 1 -- 172.21.15.167:0/1691752437 <== osd.3 v2:172.21.15.167:6808/30036 12 ==== osd_op_reply(52 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f2a4c035070 con 0x5569c249b8d0 2023-10-05T19:28:33.140 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.134+0000 7f2a53196700 1 -- 172.21.15.167:0/1691752437 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:53 5.12 5:48000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x5569c249af70 con 0x5569c2495710 2023-10-05T19:28:33.141 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.134+0000 7f2a53997700 1 -- 172.21.15.167:0/1691752437 <== osd.1 v2:172.21.15.167:6800/30035 20 ==== osd_op_reply(53 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f2a4402f080 con 0x5569c2495710 2023-10-05T19:28:33.141 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.134+0000 7f2a53997700 1 -- 172.21.15.167:0/1691752437 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:54 5.a 5:50000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x5569c249af70 con 0x7f2a3c00ebe0 2023-10-05T19:28:33.142 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.138+0000 7f2a53997700 1 -- 172.21.15.167:0/1691752437 <== osd.0 v2:172.21.15.118:6800/27690 14 ==== osd_op_reply(54 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f2a4402f080 con 0x7f2a3c00ebe0 2023-10-05T19:28:33.142 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.138+0000 7f2a53997700 1 -- 172.21.15.167:0/1691752437 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:55 5.1a 5:58000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x5569c249af70 con 0x5569c2495710 2023-10-05T19:28:33.143 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.138+0000 7f2a53997700 1 -- 172.21.15.167:0/1691752437 <== osd.1 v2:172.21.15.167:6800/30035 21 ==== osd_op_reply(55 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f2a4402f080 con 0x5569c2495710 2023-10-05T19:28:33.143 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.138+0000 7f2a53997700 1 -- 172.21.15.167:0/1691752437 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:56 5.6 5:60000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x5569c249af70 con 0x7f2a4c038030 2023-10-05T19:28:33.144 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.138+0000 7f2a52995700 1 -- 172.21.15.167:0/1691752437 <== osd.2 v2:172.21.15.167:6816/30037 9 ==== osd_op_reply(56 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f2a3c006070 con 0x7f2a4c038030 2023-10-05T19:28:33.144 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.138+0000 7f2a52995700 1 -- 172.21.15.167:0/1691752437 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:57 5.16 5:68000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x5569c249af70 con 0x5569c249b8d0 2023-10-05T19:28:33.145 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.138+0000 7f2a53196700 1 -- 172.21.15.167:0/1691752437 <== osd.3 v2:172.21.15.167:6808/30036 13 ==== osd_op_reply(57 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f2a4c035070 con 0x5569c249b8d0 2023-10-05T19:28:33.145 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.138+0000 7f2a53196700 1 -- 172.21.15.167:0/1691752437 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:58 5.e 5:70000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x5569c249af70 con 0x5569c249b8d0 2023-10-05T19:28:33.146 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.138+0000 7f2a53196700 1 -- 172.21.15.167:0/1691752437 <== osd.3 v2:172.21.15.167:6808/30036 14 ==== osd_op_reply(58 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f2a4c035070 con 0x5569c249b8d0 2023-10-05T19:28:33.146 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.138+0000 7f2a53196700 1 -- 172.21.15.167:0/1691752437 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:59 5.1e 5:78000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x5569c249af70 con 0x7f2a3c00ebe0 2023-10-05T19:28:33.146 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.142+0000 7f2a53997700 1 -- 172.21.15.167:0/1691752437 <== osd.0 v2:172.21.15.118:6800/27690 15 ==== osd_op_reply(59 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f2a4402f080 con 0x7f2a3c00ebe0 2023-10-05T19:28:33.146 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.142+0000 7f2a53997700 1 -- 172.21.15.167:0/1691752437 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:60 5.1 5:80000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x5569c249af70 con 0x5569c249b8d0 2023-10-05T19:28:33.147 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.142+0000 7f2a53196700 1 -- 172.21.15.167:0/1691752437 <== osd.3 v2:172.21.15.167:6808/30036 15 ==== osd_op_reply(60 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f2a4c035070 con 0x5569c249b8d0 2023-10-05T19:28:33.147 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.142+0000 7f2a53196700 1 -- 172.21.15.167:0/1691752437 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:61 5.11 5:88000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x5569c249af70 con 0x5569c249b8d0 2023-10-05T19:28:33.148 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.142+0000 7f2a53196700 1 -- 172.21.15.167:0/1691752437 <== osd.3 v2:172.21.15.167:6808/30036 16 ==== osd_op_reply(61 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f2a4c035070 con 0x5569c249b8d0 2023-10-05T19:28:33.148 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.142+0000 7f2a53196700 1 -- 172.21.15.167:0/1691752437 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:62 5.9 5:90000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x5569c249af70 con 0x5569c2495710 2023-10-05T19:28:33.149 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.142+0000 7f2a53997700 1 -- 172.21.15.167:0/1691752437 <== osd.1 v2:172.21.15.167:6800/30035 22 ==== osd_op_reply(62 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f2a4402f080 con 0x5569c2495710 2023-10-05T19:28:33.149 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.142+0000 7f2a53997700 1 -- 172.21.15.167:0/1691752437 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:63 5.19 5:98000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x5569c249af70 con 0x5569c2495710 2023-10-05T19:28:33.150 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.146+0000 7f2a53997700 1 -- 172.21.15.167:0/1691752437 <== osd.1 v2:172.21.15.167:6800/30035 23 ==== osd_op_reply(63 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f2a4402f080 con 0x5569c2495710 2023-10-05T19:28:33.150 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.146+0000 7f2a53997700 1 -- 172.21.15.167:0/1691752437 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:64 5.5 5:a0000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x5569c249af70 con 0x7f2a3c00ebe0 2023-10-05T19:28:33.151 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.146+0000 7f2a53997700 1 -- 172.21.15.167:0/1691752437 <== osd.0 v2:172.21.15.118:6800/27690 16 ==== osd_op_reply(64 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f2a4402f080 con 0x7f2a3c00ebe0 2023-10-05T19:28:33.151 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.146+0000 7f2a53997700 1 -- 172.21.15.167:0/1691752437 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:65 5.15 5:a8000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x5569c249af70 con 0x5569c249b8d0 2023-10-05T19:28:33.152 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.146+0000 7f2a53196700 1 -- 172.21.15.167:0/1691752437 <== osd.3 v2:172.21.15.167:6808/30036 17 ==== osd_op_reply(65 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f2a4c035070 con 0x5569c249b8d0 2023-10-05T19:28:33.152 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.146+0000 7f2a53196700 1 -- 172.21.15.167:0/1691752437 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:66 5.d 5:b0000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x5569c249af70 con 0x7f2a4c038030 2023-10-05T19:28:33.153 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.146+0000 7f2a52995700 1 -- 172.21.15.167:0/1691752437 <== osd.2 v2:172.21.15.167:6816/30037 10 ==== osd_op_reply(66 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f2a3c006070 con 0x7f2a4c038030 2023-10-05T19:28:33.153 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.146+0000 7f2a52995700 1 -- 172.21.15.167:0/1691752437 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:67 5.1d 5:b8000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x5569c249af70 con 0x5569c2495710 2023-10-05T19:28:33.154 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.146+0000 7f2a53997700 1 -- 172.21.15.167:0/1691752437 <== osd.1 v2:172.21.15.167:6800/30035 24 ==== osd_op_reply(67 [pgnls start_epoch 30 out=107b] v30'2 uv2 ondisk = 1) v8 ==== 144+0+107 (crc 0 0 0) 0x7f2a4402f080 con 0x5569c2495710 2023-10-05T19:28:33.154 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.150+0000 7f2a53997700 1 -- 172.21.15.167:0/1691752437 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:68 5.3 5:c0000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x5569c249af70 con 0x7f2a3c00ebe0 2023-10-05T19:28:33.155 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.150+0000 7f2a53997700 1 -- 172.21.15.167:0/1691752437 <== osd.0 v2:172.21.15.118:6800/27690 17 ==== osd_op_reply(68 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f2a4402f080 con 0x7f2a3c00ebe0 2023-10-05T19:28:33.155 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.150+0000 7f2a53997700 1 -- 172.21.15.167:0/1691752437 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:69 5.13 5:c8000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x5569c249af70 con 0x5569c249b8d0 2023-10-05T19:28:33.156 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.150+0000 7f2a53196700 1 -- 172.21.15.167:0/1691752437 <== osd.3 v2:172.21.15.167:6808/30036 18 ==== osd_op_reply(69 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f2a4c035070 con 0x5569c249b8d0 2023-10-05T19:28:33.156 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.150+0000 7f2a53196700 1 -- 172.21.15.167:0/1691752437 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:70 5.b 5:d0000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x5569c249af70 con 0x7f2a3c00ebe0 2023-10-05T19:28:33.156 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.150+0000 7f2a53997700 1 -- 172.21.15.167:0/1691752437 <== osd.0 v2:172.21.15.118:6800/27690 18 ==== osd_op_reply(70 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f2a4402f080 con 0x7f2a3c00ebe0 2023-10-05T19:28:33.157 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.150+0000 7f2a53997700 1 -- 172.21.15.167:0/1691752437 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:71 5.1b 5:d8000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x5569c249af70 con 0x7f2a4c038030 2023-10-05T19:28:33.157 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.150+0000 7f2a52995700 1 -- 172.21.15.167:0/1691752437 <== osd.2 v2:172.21.15.167:6816/30037 11 ==== osd_op_reply(71 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f2a3c006070 con 0x7f2a4c038030 2023-10-05T19:28:33.157 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.150+0000 7f2a52995700 1 -- 172.21.15.167:0/1691752437 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:72 5.7 5:e0000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x5569c249af70 con 0x5569c249b8d0 2023-10-05T19:28:33.158 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.154+0000 7f2a53196700 1 -- 172.21.15.167:0/1691752437 <== osd.3 v2:172.21.15.167:6808/30036 19 ==== osd_op_reply(72 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f2a4c035070 con 0x5569c249b8d0 2023-10-05T19:28:33.158 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.154+0000 7f2a53196700 1 -- 172.21.15.167:0/1691752437 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:73 5.17 5:e8000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x5569c249af70 con 0x7f2a3c00ebe0 2023-10-05T19:28:33.159 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.154+0000 7f2a53997700 1 -- 172.21.15.167:0/1691752437 <== osd.0 v2:172.21.15.118:6800/27690 19 ==== osd_op_reply(73 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f2a4402f080 con 0x7f2a3c00ebe0 2023-10-05T19:28:33.159 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.154+0000 7f2a53997700 1 -- 172.21.15.167:0/1691752437 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:74 5.f 5:f0000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x5569c249af70 con 0x5569c2495710 2023-10-05T19:28:33.160 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.154+0000 7f2a53997700 1 -- 172.21.15.167:0/1691752437 <== osd.1 v2:172.21.15.167:6800/30035 25 ==== osd_op_reply(74 [pgnls start_epoch 30 out=115b] v23'2 uv2 ondisk = 1) v8 ==== 144+0+115 (crc 0 0 0) 0x7f2a4402f080 con 0x5569c2495710 2023-10-05T19:28:33.160 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.154+0000 7f2a53997700 1 -- 172.21.15.167:0/1691752437 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:75 5.1f 5:f8000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x5569c249af70 con 0x7f2a4c038030 2023-10-05T19:28:33.161 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.154+0000 7f2a52995700 1 -- 172.21.15.167:0/1691752437 <== osd.2 v2:172.21.15.167:6816/30037 12 ==== osd_op_reply(75 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f2a3c006070 con 0x7f2a4c038030 2023-10-05T19:28:33.161 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.154+0000 7f2a55661e40 20 RGWRados::pool_iterate: got zonegroup_info.4f1f719d-9648-417b-b10c-e5d939825419 2023-10-05T19:28:33.161 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.154+0000 7f2a55661e40 20 RGWRados::pool_iterate: got zone_info.9a9df965-6612-45d4-a45f-6e3c889bacc2 2023-10-05T19:28:33.162 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.154+0000 7f2a55661e40 20 RGWRados::pool_iterate: got zone_names.default 2023-10-05T19:28:33.162 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.154+0000 7f2a55661e40 20 RGWRados::pool_iterate: got zonegroups_names.default 2023-10-05T19:28:33.162 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.154+0000 7f2a55661e40 20 rados->read ofs=0 len=0 2023-10-05T19:28:33.162 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.154+0000 7f2a55661e40 1 -- 172.21.15.167:0/1691752437 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:76 5.f 5:f43b6ece:::zone_names.default:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x5569c249ede0 con 0x5569c2495710 2023-10-05T19:28:33.163 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.158+0000 7f2a53997700 1 -- 172.21.15.167:0/1691752437 <== osd.1 v2:172.21.15.167:6800/30035 26 ==== osd_op_reply(76 zone_names.default [read 0~46 out=46b] v0'0 uv1 ondisk = 0) v8 ==== 162+0+46 (crc 0 0 0) 0x7f2a4402f080 con 0x5569c2495710 2023-10-05T19:28:33.163 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.158+0000 7f2a55661e40 20 rados_obj.operate() r=0 bl.length=46 2023-10-05T19:28:33.163 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.158+0000 7f2a55661e40 20 rados->read ofs=0 len=0 2023-10-05T19:28:33.163 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.158+0000 7f2a55661e40 1 -- 172.21.15.167:0/1691752437 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:77 5.1d 5:b8374eb4:::zone_info.9a9df965-6612-45d4-a45f-6e3c889bacc2:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x5569c249f190 con 0x5569c2495710 2023-10-05T19:28:33.163 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.158+0000 7f2a53997700 1 -- 172.21.15.167:0/1691752437 <== osd.1 v2:172.21.15.167:6800/30035 27 ==== osd_op_reply(77 zone_info.9a9df965-6612-45d4-a45f-6e3c889bacc2 [read 0~874 out=874b] v0'0 uv2 ondisk = 0) v8 ==== 190+0+874 (crc 0 0 0) 0x7f2a4402f080 con 0x5569c2495710 2023-10-05T19:28:33.163 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.158+0000 7f2a55661e40 20 rados_obj.operate() r=0 bl.length=874 2023-10-05T19:28:33.163 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.158+0000 7f2a55661e40 20 rados->read ofs=0 len=0 2023-10-05T19:28:33.164 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.158+0000 7f2a55661e40 1 -- 172.21.15.167:0/1691752437 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:78 5.f 5:f4c53578:::zonegroups_names.default:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x5569c249ab10 con 0x5569c2495710 2023-10-05T19:28:33.164 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.158+0000 7f2a53997700 1 -- 172.21.15.167:0/1691752437 <== osd.1 v2:172.21.15.167:6800/30035 28 ==== osd_op_reply(78 zonegroups_names.default [read 0~46 out=46b] v0'0 uv2 ondisk = 0) v8 ==== 168+0+46 (crc 0 0 0) 0x7f2a4402f080 con 0x5569c2495710 2023-10-05T19:28:33.165 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.158+0000 7f2a55661e40 20 rados_obj.operate() r=0 bl.length=46 2023-10-05T19:28:33.165 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.158+0000 7f2a55661e40 20 rados->read ofs=0 len=0 2023-10-05T19:28:33.165 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.158+0000 7f2a55661e40 1 -- 172.21.15.167:0/1691752437 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:79 5.c 5:30d299a5:::zonegroup_info.4f1f719d-9648-417b-b10c-e5d939825419:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x5569c249ab30 con 0x7f2a3c00ebe0 2023-10-05T19:28:33.165 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.158+0000 7f2a53997700 1 -- 172.21.15.167:0/1691752437 <== osd.0 v2:172.21.15.118:6800/27690 20 ==== osd_op_reply(79 zonegroup_info.4f1f719d-9648-417b-b10c-e5d939825419 [read 0~362 out=362b] v0'0 uv1 ondisk = 0) v8 ==== 195+0+362 (crc 0 0 0) 0x7f2a4402f080 con 0x7f2a3c00ebe0 2023-10-05T19:28:33.165 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.158+0000 7f2a55661e40 20 rados_obj.operate() r=0 bl.length=362 2023-10-05T19:28:33.165 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.158+0000 7f2a55661e40 20 rados->read ofs=0 len=0 2023-10-05T19:28:33.165 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.158+0000 7f2a55661e40 1 -- 172.21.15.167:0/1691752437 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:80 5.3 5:c0ad8cbd:::region_map:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x5569c249fc00 con 0x7f2a3c00ebe0 2023-10-05T19:28:33.166 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.158+0000 7f2a53997700 1 -- 172.21.15.167:0/1691752437 <== osd.0 v2:172.21.15.118:6800/27690 21 ==== osd_op_reply(80 region_map [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 ==== 154+0+0 (crc 0 0 0) 0x7f2a4402f080 con 0x7f2a3c00ebe0 2023-10-05T19:28:33.167 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.158+0000 7f2a55661e40 20 rados_obj.operate() r=-2 bl.length=0 2023-10-05T19:28:33.167 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.158+0000 7f2a55661e40 20 zone default found 2023-10-05T19:28:33.167 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.158+0000 7f2a55661e40 4 Realm: () 2023-10-05T19:28:33.167 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.158+0000 7f2a55661e40 4 ZoneGroup: default (4f1f719d-9648-417b-b10c-e5d939825419) 2023-10-05T19:28:33.167 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.158+0000 7f2a55661e40 4 Zone: default (9a9df965-6612-45d4-a45f-6e3c889bacc2) 2023-10-05T19:28:33.167 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.158+0000 7f2a55661e40 10 cannot find current period zonegroup using local zonegroup configuration 2023-10-05T19:28:33.167 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.158+0000 7f2a55661e40 20 zonegroup default 2023-10-05T19:28:33.167 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.158+0000 7f2a55661e40 20 rados->read ofs=0 len=0 2023-10-05T19:28:33.167 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.158+0000 7f2a55661e40 1 -- 172.21.15.167:0/1691752437 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:81 5.3 5:c52100b6:::period_config.default:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x5569c249fc00 con 0x7f2a3c00ebe0 2023-10-05T19:28:33.168 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.162+0000 7f2a53997700 1 -- 172.21.15.167:0/1691752437 <== osd.0 v2:172.21.15.118:6800/27690 22 ==== osd_op_reply(81 period_config.default [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 ==== 165+0+0 (crc 0 0 0) 0x7f2a4402f080 con 0x7f2a3c00ebe0 2023-10-05T19:28:33.168 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.162+0000 7f2a55661e40 20 rados_obj.operate() r=-2 bl.length=0 2023-10-05T19:28:33.168 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.162+0000 7f2a55661e40 20 rados->read ofs=0 len=0 2023-10-05T19:28:33.168 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.162+0000 7f2a55661e40 1 -- 172.21.15.167:0/1691752437 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:82 6.2 6:4347d321:::bucket.sync-source-hints.:head [call version.read in=11b,read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x5569c24a1220 con 0x5569c249b8d0 2023-10-05T19:28:33.169 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.162+0000 7f2a53196700 1 -- 172.21.15.167:0/1691752437 <== osd.3 v2:172.21.15.167:6808/30036 20 ==== osd_op_reply(82 bucket.sync-source-hints. [call,read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 ==== 211+0+0 (crc 0 0 0) 0x7f2a4c035070 con 0x5569c249b8d0 2023-10-05T19:28:33.169 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.162+0000 7f2a55661e40 20 rados_obj.operate() r=-2 bl.length=0 2023-10-05T19:28:33.169 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.162+0000 7f2a55661e40 20 rados->read ofs=0 len=0 2023-10-05T19:28:33.169 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.162+0000 7f2a55661e40 1 -- 172.21.15.167:0/1691752437 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:83 6.b 6:d467b91b:::bucket.sync-target-hints.:head [call version.read in=11b,read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x5569c24a15d0 con 0x5569c249b8d0 2023-10-05T19:28:33.169 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.162+0000 7f2a53196700 1 -- 172.21.15.167:0/1691752437 <== osd.3 v2:172.21.15.167:6808/30036 21 ==== osd_op_reply(83 bucket.sync-target-hints. [call,read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 ==== 211+0+0 (crc 0 0 0) 0x7f2a4c035070 con 0x5569c249b8d0 2023-10-05T19:28:33.169 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.162+0000 7f2a55661e40 20 rados_obj.operate() r=-2 bl.length=0 2023-10-05T19:28:33.169 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.162+0000 7f2a55661e40 20 started sync module instance, tier type = 2023-10-05T19:28:33.170 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.162+0000 7f2a55661e40 20 started zone id=9a9df965-6612-45d4-a45f-6e3c889bacc2 (name=default) with tier type = 2023-10-05T19:28:33.170 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.162+0000 7f2a55661e40 1 -- 172.21.15.167:0/1691752437 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:84 7.1f 7:f95f44c2:::notify.0:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x5569c24a08e0 con 0x7f2a4c038030 2023-10-05T19:28:33.171 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.166+0000 7f2a52995700 1 -- 172.21.15.167:0/1691752437 <== osd.2 v2:172.21.15.167:6816/30037 13 ==== osd_op_reply(84 notify.0 [create] v30'117 uv117 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f2a3c006070 con 0x7f2a4c038030 2023-10-05T19:28:33.171 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.166+0000 7f2a55661e40 1 -- 172.21.15.167:0/1691752437 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:85 7.1f 7:f95f44c2:::notify.0:head [watch watch cookie 93912719559824] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x5569c24a0510 con 0x7f2a4c038030 2023-10-05T19:28:33.171 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.166+0000 7f2a55661e40 1 -- 172.21.15.167:0/1691752437 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:86 7.0 7:05bf5b68:::notify.1:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x5569c24a2360 con 0x5569c2495710 2023-10-05T19:28:33.174 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.166+0000 7f2a52995700 1 -- 172.21.15.167:0/1691752437 <== osd.2 v2:172.21.15.167:6816/30037 14 ==== osd_op_reply(85 notify.0 [watch watch cookie 93912719559824] v30'118 uv117 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f2a3c006070 con 0x7f2a4c038030 2023-10-05T19:28:33.174 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.166+0000 7f2a53997700 1 -- 172.21.15.167:0/1691752437 <== osd.1 v2:172.21.15.167:6800/30035 29 ==== osd_op_reply(86 notify.1 [create] v30'117 uv117 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f2a4402f080 con 0x5569c2495710 2023-10-05T19:28:33.174 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.166+0000 7f2a55661e40 1 -- 172.21.15.167:0/1691752437 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:87 7.0 7:05bf5b68:::notify.1:head [watch watch cookie 93912719563424] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x5569c24a2760 con 0x5569c2495710 2023-10-05T19:28:33.174 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.170+0000 7f2a55661e40 1 -- 172.21.15.167:0/1691752437 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:88 7.15 7:a93a5511:::notify.2:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x5569c24a3000 con 0x5569c249b8d0 2023-10-05T19:28:33.177 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.170+0000 7f2a53196700 1 -- 172.21.15.167:0/1691752437 <== osd.3 v2:172.21.15.167:6808/30036 22 ==== osd_op_reply(88 notify.2 [create] v30'117 uv117 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f2a4c035070 con 0x5569c249b8d0 2023-10-05T19:28:33.177 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.170+0000 7f2a55661e40 1 -- 172.21.15.167:0/1691752437 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:89 7.15 7:a93a5511:::notify.2:head [watch watch cookie 93912719566976] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x5569c24a38a0 con 0x5569c249b8d0 2023-10-05T19:28:33.177 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.170+0000 7f2a55661e40 1 -- 172.21.15.167:0/1691752437 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:90 7.e 7:7759931f:::notify.3:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x5569c24a4140 con 0x7f2a4c038030 2023-10-05T19:28:33.178 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.170+0000 7f2a53997700 1 -- 172.21.15.167:0/1691752437 <== osd.1 v2:172.21.15.167:6800/30035 30 ==== osd_op_reply(87 notify.1 [watch watch cookie 93912719563424] v30'118 uv117 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f2a44050040 con 0x5569c2495710 2023-10-05T19:28:33.180 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.174+0000 7f2a52995700 1 -- 172.21.15.167:0/1691752437 <== osd.2 v2:172.21.15.167:6816/30037 15 ==== osd_op_reply(90 notify.3 [create] v30'117 uv117 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f2a3c006070 con 0x7f2a4c038030 2023-10-05T19:28:33.180 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.174+0000 7f2a55661e40 1 -- 172.21.15.167:0/1691752437 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:91 7.e 7:7759931f:::notify.3:head [watch watch cookie 93912719564528] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x5569c24a3d70 con 0x7f2a4c038030 2023-10-05T19:28:33.180 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.174+0000 7f2a55661e40 1 -- 172.21.15.167:0/1691752437 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:92 7.d 7:b4812045:::notify.4:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x5569c24a5160 con 0x5569c2495710 2023-10-05T19:28:33.180 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.174+0000 7f2a53196700 1 -- 172.21.15.167:0/1691752437 <== osd.3 v2:172.21.15.167:6808/30036 23 ==== osd_op_reply(89 notify.2 [watch watch cookie 93912719566976] v30'118 uv117 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f2a4c035070 con 0x5569c249b8d0 2023-10-05T19:28:33.182 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.178+0000 7f2a52995700 1 -- 172.21.15.167:0/1691752437 <== osd.2 v2:172.21.15.167:6816/30037 16 ==== osd_op_reply(91 notify.3 [watch watch cookie 93912719564528] v30'118 uv117 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f2a3c006070 con 0x7f2a4c038030 2023-10-05T19:28:33.183 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.178+0000 7f2a53997700 1 -- 172.21.15.167:0/1691752437 <== osd.1 v2:172.21.15.167:6800/30035 31 ==== osd_op_reply(92 notify.4 [create] v30'117 uv117 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f2a44050040 con 0x5569c2495710 2023-10-05T19:28:33.183 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.178+0000 7f2a55661e40 1 -- 172.21.15.167:0/1691752437 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:93 7.d 7:b4812045:::notify.4:head [watch watch cookie 93912719574336] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x5569c24a4c30 con 0x5569c2495710 2023-10-05T19:28:33.183 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.178+0000 7f2a55661e40 1 -- 172.21.15.167:0/1691752437 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:94 7.3 7:c609908c:::notify.5:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x5569c24a5a50 con 0x7f2a3c00ebe0 2023-10-05T19:28:33.185 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.178+0000 7f2a53997700 1 -- 172.21.15.167:0/1691752437 <== osd.1 v2:172.21.15.167:6800/30035 32 ==== osd_op_reply(93 notify.4 [watch watch cookie 93912719574336] v30'118 uv117 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f2a44050040 con 0x5569c2495710 2023-10-05T19:28:33.186 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.178+0000 7f2a53997700 1 -- 172.21.15.167:0/1691752437 <== osd.0 v2:172.21.15.118:6800/27690 23 ==== osd_op_reply(94 notify.5 [create] v30'117 uv117 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f2a44050040 con 0x7f2a3c00ebe0 2023-10-05T19:28:33.186 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.182+0000 7f2a55661e40 1 -- 172.21.15.167:0/1691752437 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:95 7.3 7:c609908c:::notify.5:head [watch watch cookie 93912719569840] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x5569c24a5a00 con 0x7f2a3c00ebe0 2023-10-05T19:28:33.186 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.182+0000 7f2a55661e40 1 -- 172.21.15.167:0/1691752437 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:96 7.14 7:2b04a3e9:::notify.6:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x5569c24a6410 con 0x7f2a3c00ebe0 2023-10-05T19:28:33.188 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.182+0000 7f2a53997700 1 -- 172.21.15.167:0/1691752437 <== osd.0 v2:172.21.15.118:6800/27690 24 ==== osd_op_reply(95 notify.5 [watch watch cookie 93912719569840] v30'118 uv117 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f2a44050040 con 0x7f2a3c00ebe0 2023-10-05T19:28:33.189 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.182+0000 7f2a53997700 1 -- 172.21.15.167:0/1691752437 <== osd.0 v2:172.21.15.118:6800/27690 25 ==== osd_op_reply(96 notify.6 [create] v30'117 uv117 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f2a44050040 con 0x7f2a3c00ebe0 2023-10-05T19:28:33.189 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.182+0000 7f2a55661e40 1 -- 172.21.15.167:0/1691752437 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:97 7.14 7:2b04a3e9:::notify.6:head [watch watch cookie 93912719578944] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x5569c24a6410 con 0x7f2a3c00ebe0 2023-10-05T19:28:33.189 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.182+0000 7f2a55661e40 1 -- 172.21.15.167:0/1691752437 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:98 7.9 7:93e5b521:::notify.7:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x5569c24a6e20 con 0x7f2a3c00ebe0 2023-10-05T19:28:33.191 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.186+0000 7f2a53997700 1 -- 172.21.15.167:0/1691752437 <== osd.0 v2:172.21.15.118:6800/27690 26 ==== osd_op_reply(98 notify.7 [create] v30'117 uv117 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f2a44050040 con 0x7f2a3c00ebe0 2023-10-05T19:28:33.191 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.186+0000 7f2a55661e40 1 -- 172.21.15.167:0/1691752437 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:99 7.9 7:93e5b521:::notify.7:head [watch watch cookie 93912719583536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x5569c24a7270 con 0x7f2a3c00ebe0 2023-10-05T19:28:33.192 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.186+0000 7f2a55661e40 20 add_watcher() i=0 2023-10-05T19:28:33.192 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.186+0000 7f2a55661e40 20 add_watcher() i=1 2023-10-05T19:28:33.192 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.186+0000 7f2a55661e40 20 add_watcher() i=2 2023-10-05T19:28:33.192 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.186+0000 7f2a55661e40 20 add_watcher() i=3 2023-10-05T19:28:33.192 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.186+0000 7f2a55661e40 20 add_watcher() i=4 2023-10-05T19:28:33.192 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.186+0000 7f2a55661e40 20 add_watcher() i=5 2023-10-05T19:28:33.192 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.186+0000 7f2a53997700 1 -- 172.21.15.167:0/1691752437 <== osd.0 v2:172.21.15.118:6800/27690 27 ==== osd_op_reply(97 notify.6 [watch watch cookie 93912719578944] v30'118 uv117 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f2a44050040 con 0x7f2a3c00ebe0 2023-10-05T19:28:33.192 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.186+0000 7f2a55661e40 20 add_watcher() i=6 2023-10-05T19:28:33.193 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.186+0000 7f2a53997700 1 -- 172.21.15.167:0/1691752437 <== osd.0 v2:172.21.15.118:6800/27690 28 ==== osd_op_reply(99 notify.7 [watch watch cookie 93912719583536] v30'118 uv117 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f2a44050040 con 0x7f2a3c00ebe0 2023-10-05T19:28:33.193 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.186+0000 7f2a55661e40 20 add_watcher() i=7 2023-10-05T19:28:33.194 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.186+0000 7f2a55661e40 2 all 8 watchers are set, enabling cache 2023-10-05T19:28:33.194 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.186+0000 7f2a55661e40 1 -- 172.21.15.167:0/1691752437 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:100 6.4 6:22d26bf9:::data_loggenerations_metadata:head [call version.check_conds in=74b,call version.read in=11b,read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x5569c24a5f20 con 0x5569c2495710 2023-10-05T19:28:33.194 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.190+0000 7f2a53997700 1 -- 172.21.15.167:0/1691752437 <== osd.1 v2:172.21.15.167:6800/30035 33 ==== osd_op_reply(100 data_loggenerations_metadata [call,call out=48b,read 0~28 out=28b] v0'0 uv1 ondisk = 0) v8 ==== 256+0+76 (crc 0 0 0) 0x7f2a44050040 con 0x5569c2495710 2023-10-05T19:28:33.194 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.190+0000 7f2a55661e40 1 -- 172.21.15.167:0/1691752437 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:101 6.4 6:22d26bf9:::data_loggenerations_metadata:head [watch watch cookie 93912719584640] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x5569c24a7680 con 0x5569c2495710 2023-10-05T19:28:33.196 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.190+0000 7f2a53997700 1 -- 172.21.15.167:0/1691752437 <== osd.1 v2:172.21.15.167:6800/30035 34 ==== osd_op_reply(101 data_loggenerations_metadata [watch watch cookie 93912719584640] v30'113 uv1 ondisk = 0) v8 ==== 172+0+0 (crc 0 0 0) 0x7f2a44050040 con 0x5569c2495710 2023-10-05T19:28:33.196 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.190+0000 7f2a24fe1700 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2023-10-05T19:28:33.196 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.190+0000 7f2a55661e40 20 check_secure_mon_conn(): auth registy supported: methods=[2] modes=[2,1] 2023-10-05T19:28:33.197 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.190+0000 7f2a55661e40 20 check_secure_mon_conn(): mode 1 is insecure 2023-10-05T19:28:33.197 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.190+0000 7f2a55661e40 1 Processor -- start 2023-10-05T19:28:33.197 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.190+0000 7f2a55661e40 1 -- start start 2023-10-05T19:28:33.197 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.190+0000 7f2a55661e40 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5569c24b47b0 0x5569c24b4b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-05T19:28:33.197 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.190+0000 7f2a55661e40 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x5569c24b50c0 con 0x5569c24b47b0 2023-10-05T19:28:33.198 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.190+0000 7f2a52995700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5569c24b47b0 0x5569c24b4b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-05T19:28:33.198 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.194+0000 7f2a52995700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5569c24b47b0 0x5569c24b4b80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.118:3300/0 says I am v2:172.21.15.167:58576/0 (socket says 172.21.15.167:58576) 2023-10-05T19:28:33.198 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.194+0000 7f2a52995700 1 -- 172.21.15.167:0/4127375761 learned_addr learned my addr 172.21.15.167:0/4127375761 (peer_addr_for_me v2:172.21.15.167:0/0) 2023-10-05T19:28:33.199 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.194+0000 7f2a52995700 1 -- 172.21.15.167:0/4127375761 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5569c24b5200 con 0x5569c24b47b0 2023-10-05T19:28:33.199 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.194+0000 7f2a52995700 1 --2- 172.21.15.167:0/4127375761 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5569c24b47b0 0x5569c24b4b80 secure :-1 s=READY pgs=372 cs=0 l=1 rev1=1 crypto rx=0x5569c24951b0 tx=0x7f2a3c006000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d8055fcbb196830b server_cookie=0 in_seq=0 out_seq=0 2023-10-05T19:28:33.199 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.194+0000 7f299ffff700 1 -- 172.21.15.167:0/4127375761 <== mon.0 v2:172.21.15.118:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f2a3c023040 con 0x5569c24b47b0 2023-10-05T19:28:33.199 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.194+0000 7f299ffff700 1 -- 172.21.15.167:0/4127375761 <== mon.0 v2:172.21.15.118:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2a3c024040 con 0x5569c24b47b0 2023-10-05T19:28:33.200 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.194+0000 7f299ffff700 1 -- 172.21.15.167:0/4127375761 <== mon.0 v2:172.21.15.118:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f2a3c013040 con 0x5569c24b47b0 2023-10-05T19:28:33.200 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.194+0000 7f2a55661e40 1 -- 172.21.15.167:0/4127375761 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5569c24b47b0 msgr2=0x5569c24b4b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-05T19:28:33.201 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.194+0000 7f2a55661e40 1 --2- 172.21.15.167:0/4127375761 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5569c24b47b0 0x5569c24b4b80 secure :-1 s=READY pgs=372 cs=0 l=1 rev1=1 crypto rx=0x5569c24951b0 tx=0x7f2a3c006000 comp rx=0 tx=0).stop 2023-10-05T19:28:33.201 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.194+0000 7f2a55661e40 1 -- 172.21.15.167:0/4127375761 shutdown_connections 2023-10-05T19:28:33.201 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.194+0000 7f2a55661e40 1 --2- 172.21.15.167:0/4127375761 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5569c24b47b0 0x5569c24b4b80 unknown :-1 s=CLOSED pgs=372 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-05T19:28:33.201 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.194+0000 7f2a55661e40 1 -- 172.21.15.167:0/4127375761 >> 172.21.15.167:0/4127375761 conn(0x5569c24afb70 msgr2=0x5569c24adcc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-05T19:28:33.201 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.194+0000 7f2a55661e40 1 -- 172.21.15.167:0/4127375761 shutdown_connections 2023-10-05T19:28:33.201 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.194+0000 7f2a55661e40 1 -- 172.21.15.167:0/4127375761 wait complete. 2023-10-05T19:28:33.202 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.194+0000 7f2a55661e40 1 Processor -- start 2023-10-05T19:28:33.202 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.194+0000 7f2a55661e40 1 -- start start 2023-10-05T19:28:33.202 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.194+0000 7f2a55661e40 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5569c24b47b0 0x5569c24ba390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-05T19:28:33.202 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.194+0000 7f2a55661e40 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x5569c24bb390 con 0x5569c24b47b0 2023-10-05T19:28:33.202 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.198+0000 7f2a52995700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5569c24b47b0 0x5569c24ba390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-05T19:28:33.203 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.198+0000 7f2a52995700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5569c24b47b0 0x5569c24ba390 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.118:3300/0 says I am v2:172.21.15.167:58588/0 (socket says 172.21.15.167:58588) 2023-10-05T19:28:33.203 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.198+0000 7f2a52995700 1 -- 172.21.15.167:0/3576704625 learned_addr learned my addr 172.21.15.167:0/3576704625 (peer_addr_for_me v2:172.21.15.167:0/0) 2023-10-05T19:28:33.204 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.198+0000 7f2a52995700 1 -- 172.21.15.167:0/3576704625 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2a40001e10 con 0x5569c24b47b0 2023-10-05T19:28:33.204 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.198+0000 7f2a52995700 1 --2- 172.21.15.167:0/3576704625 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5569c24b47b0 0x5569c24ba390 secure :-1 s=READY pgs=373 cs=0 l=1 rev1=1 crypto rx=0x5569c24ae830 tx=0x7f2a3c036000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-05T19:28:33.204 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.198+0000 7f299e7fc700 1 -- 172.21.15.167:0/3576704625 <== mon.0 v2:172.21.15.118:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f2a3c023070 con 0x5569c24b47b0 2023-10-05T19:28:33.204 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.198+0000 7f299e7fc700 1 -- 172.21.15.167:0/3576704625 <== mon.0 v2:172.21.15.118:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2a3c02de90 con 0x5569c24b47b0 2023-10-05T19:28:33.204 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.198+0000 7f299e7fc700 1 -- 172.21.15.167:0/3576704625 <== mon.0 v2:172.21.15.118:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f2a3c01e630 con 0x5569c24b47b0 2023-10-05T19:28:33.204 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.198+0000 7f2a55661e40 1 -- 172.21.15.167:0/3576704625 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x5569c24ba990 con 0x5569c24b47b0 2023-10-05T19:28:33.204 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.198+0000 7f2a55661e40 1 -- 172.21.15.167:0/3576704625 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x5569c24badd0 con 0x5569c24b47b0 2023-10-05T19:28:33.205 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.198+0000 7f299e7fc700 1 -- 172.21.15.167:0/3576704625 <== mon.0 v2:172.21.15.118:3300/0 4 ==== mgrmap(e 5) v1 ==== 40329+0+0 (secure 0 0 0) 0x7f2a3c01e790 con 0x5569c24b47b0 2023-10-05T19:28:33.205 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.198+0000 7f299e7fc700 1 --2- 172.21.15.167:0/3576704625 >> [v2:172.21.15.167:6824/29889,v1:172.21.15.167:6825/29889] conn(0x7f2994030c00 0x7f29940330b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-05T19:28:33.205 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.198+0000 7f299e7fc700 1 -- 172.21.15.167:0/3576704625 <== mon.0 v2:172.21.15.118:3300/0 5 ==== osd_map(30..30 src has 1..30) v4 ==== 6057+0+0 (secure 0 0 0) 0x7f2a3c04dcc0 con 0x5569c24b47b0 2023-10-05T19:28:33.205 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.198+0000 7f2a53997700 1 --2- 172.21.15.167:0/3576704625 >> [v2:172.21.15.167:6824/29889,v1:172.21.15.167:6825/29889] conn(0x7f2994030c00 0x7f29940330b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-05T19:28:33.206 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.198+0000 7f2a55661e40 1 --2- 172.21.15.167:0/3576704625 >> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] conn(0x5569c24c4f80 0x5569c24c7390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-05T19:28:33.206 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.198+0000 7f2a55661e40 1 -- 172.21.15.167:0/3576704625 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:1 6.19 6:9a6dd8c0:gc::gc.0:head [create,call version.check_conds in=74b,call rgw_gc.rgw_gc_queue_init in=45b,call version.set in=34b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x5569c24c7a30 con 0x5569c24c4f80 2023-10-05T19:28:33.206 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.202+0000 7f2a53196700 1 --2- 172.21.15.167:0/3576704625 >> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] conn(0x5569c24c4f80 0x5569c24c7390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-05T19:28:33.206 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.202+0000 7f2a53997700 1 --2- 172.21.15.167:0/3576704625 >> [v2:172.21.15.167:6824/29889,v1:172.21.15.167:6825/29889] conn(0x7f2994030c00 0x7f29940330b0 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x5569c24a7200 tx=0x7f2a44000f90 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-05T19:28:33.207 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.202+0000 7f2a53196700 1 --2- 172.21.15.167:0/3576704625 >> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] conn(0x5569c24c4f80 0x5569c24c7390 crc :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-05T19:28:33.209 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.202+0000 7f2a53196700 1 -- 172.21.15.167:0/3576704625 <== osd.1 v2:172.21.15.167:6800/30035 1 ==== osd_op_reply(1 gc.0 [create,call,call,call] v30'25 uv3 ondisk = -125 ((125) Operation canceled)) v8 ==== 274+0+0 (crc 0 0 0) 0x7f2a4c058040 con 0x5569c24c4f80 2023-10-05T19:28:33.209 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.202+0000 7f2a55661e40 1 --2- 172.21.15.167:0/3576704625 >> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] conn(0x5569c24c0bc0 0x5569c24cb0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-05T19:28:33.209 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.202+0000 7f2a55661e40 1 -- 172.21.15.167:0/3576704625 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:2 6.18 6:1e85421f:gc::gc.1:head [create,call version.check_conds in=74b,call rgw_gc.rgw_gc_queue_init in=45b,call version.set in=34b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x5569c24cb5e0 con 0x5569c24c0bc0 2023-10-05T19:28:33.209 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.202+0000 7f2a52995700 1 --2- 172.21.15.167:0/3576704625 >> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] conn(0x5569c24c0bc0 0x5569c24cb0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-05T19:28:33.210 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.206+0000 7f2a52995700 1 --2- 172.21.15.167:0/3576704625 >> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] conn(0x5569c24c0bc0 0x5569c24cb0a0 crc :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-05T19:28:33.213 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.206+0000 7f2a52995700 1 -- 172.21.15.167:0/3576704625 <== osd.0 v2:172.21.15.118:6800/27690 1 ==== osd_op_reply(2 gc.1 [create,call,call,call] v30'17 uv5 ondisk = -125 ((125) Operation canceled)) v8 ==== 274+0+0 (crc 0 0 0) 0x7f2a3c03f040 con 0x5569c24c0bc0 2023-10-05T19:28:33.213 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.206+0000 7f2a55661e40 1 --2- 172.21.15.167:0/3576704625 >> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] conn(0x5569c24c8ad0 0x5569c24cec50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-05T19:28:33.213 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.206+0000 7f2a55661e40 1 -- 172.21.15.167:0/3576704625 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:3 6.b 6:d242335b:gc::gc.2:head [create,call version.check_conds in=74b,call rgw_gc.rgw_gc_queue_init in=45b,call version.set in=34b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x5569c24cf1d0 con 0x5569c24c8ad0 2023-10-05T19:28:33.213 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.206+0000 7f2a53997700 1 --2- 172.21.15.167:0/3576704625 >> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] conn(0x5569c24c8ad0 0x5569c24cec50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-05T19:28:33.214 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.206+0000 7f2a53997700 1 --2- 172.21.15.167:0/3576704625 >> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] conn(0x5569c24c8ad0 0x5569c24cec50 crc :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-05T19:28:33.216 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.210+0000 7f2a53997700 1 -- 172.21.15.167:0/3576704625 <== osd.3 v2:172.21.15.167:6808/30036 1 ==== osd_op_reply(3 gc.2 [create,call,call,call] v30'14 uv3 ondisk = -125 ((125) Operation canceled)) v8 ==== 274+0+0 (crc 0 0 0) 0x7f2a44050040 con 0x5569c24c8ad0 2023-10-05T19:28:33.216 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.210+0000 7f2a55661e40 1 -- 172.21.15.167:0/3576704625 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:4 6.4 6:2166d8e1:gc::gc.3:head [create,call version.check_conds in=74b,call rgw_gc.rgw_gc_queue_init in=45b,call version.set in=34b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x5569c24cf820 con 0x5569c24c4f80 2023-10-05T19:28:33.219 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.214+0000 7f2a53196700 1 -- 172.21.15.167:0/3576704625 <== osd.1 v2:172.21.15.167:6800/30035 2 ==== osd_op_reply(4 gc.3 [create,call,call,call] v30'114 uv7 ondisk = -125 ((125) Operation canceled)) v8 ==== 274+0+0 (crc 0 0 0) 0x7f2a4c058040 con 0x5569c24c4f80 2023-10-05T19:28:33.219 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.214+0000 7f2a55661e40 1 --2- 172.21.15.167:0/3576704625 >> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] conn(0x5569c24d04c0 0x5569c24d28a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-05T19:28:33.219 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.214+0000 7f2a55661e40 1 -- 172.21.15.167:0/3576704625 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:5 6.3 6:c60da066:gc::gc.4:head [create,call version.check_conds in=74b,call rgw_gc.rgw_gc_queue_init in=45b,call version.set in=34b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x5569c24d2f50 con 0x5569c24d04c0 2023-10-05T19:28:33.219 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.214+0000 7f2a53196700 1 --2- 172.21.15.167:0/3576704625 >> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] conn(0x5569c24d04c0 0x5569c24d28a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-05T19:28:33.220 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.214+0000 7f2a53196700 1 --2- 172.21.15.167:0/3576704625 >> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] conn(0x5569c24d04c0 0x5569c24d28a0 crc :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-05T19:28:33.223 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.218+0000 7f2a53196700 1 -- 172.21.15.167:0/3576704625 <== osd.2 v2:172.21.15.167:6816/30037 1 ==== osd_op_reply(5 gc.4 [create,call,call,call] v30'32 uv7 ondisk = -125 ((125) Operation canceled)) v8 ==== 274+0+0 (crc 0 0 0) 0x7f2a4c058040 con 0x5569c24d04c0 2023-10-05T19:28:33.223 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.218+0000 7f2a55661e40 1 -- 172.21.15.167:0/3576704625 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:6 6.9 6:954be8d2:gc::gc.5:head [create,call version.check_conds in=74b,call rgw_gc.rgw_gc_queue_init in=45b,call version.set in=34b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x5569c24d3570 con 0x5569c24c8ad0 2023-10-05T19:28:33.225 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.218+0000 7f2a53997700 1 -- 172.21.15.167:0/3576704625 <== osd.3 v2:172.21.15.167:6808/30036 2 ==== osd_op_reply(6 gc.5 [create,call,call,call] v30'23 uv4 ondisk = -125 ((125) Operation canceled)) v8 ==== 274+0+0 (crc 0 0 0) 0x7f2a44050040 con 0x5569c24c8ad0 2023-10-05T19:28:33.225 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.218+0000 7f2a55661e40 1 -- 172.21.15.167:0/3576704625 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:7 6.e 6:748722ff:gc::gc.6:head [create,call version.check_conds in=74b,call rgw_gc.rgw_gc_queue_init in=45b,call version.set in=34b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x5569c24d3b90 con 0x5569c24c4f80 2023-10-05T19:28:33.228 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.222+0000 7f2a53196700 1 -- 172.21.15.167:0/3576704625 <== osd.1 v2:172.21.15.167:6800/30035 3 ==== osd_op_reply(7 gc.6 [create,call,call,call] v30'16 uv5 ondisk = -125 ((125) Operation canceled)) v8 ==== 274+0+0 (crc 0 0 0) 0x7f2a4c058040 con 0x5569c24c4f80 2023-10-05T19:28:33.228 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.222+0000 7f2a55661e40 1 -- 172.21.15.167:0/3576704625 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:8 6.1d 6:beaa4be4:gc::gc.7:head [create,call version.check_conds in=74b,call rgw_gc.rgw_gc_queue_init in=45b,call version.set in=34b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x5569c24d4080 con 0x5569c24c0bc0 2023-10-05T19:28:33.231 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.226+0000 7f2a52995700 1 -- 172.21.15.167:0/3576704625 <== osd.0 v2:172.21.15.118:6800/27690 2 ==== osd_op_reply(8 gc.7 [create,call,call,call] v30'21 uv4 ondisk = -125 ((125) Operation canceled)) v8 ==== 274+0+0 (crc 0 0 0) 0x7f2a3c03f040 con 0x5569c24c0bc0 2023-10-05T19:28:33.231 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.226+0000 7f2a55661e40 1 -- 172.21.15.167:0/3576704625 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:9 6.12 6:4f78e9a2:gc::gc.8:head [create,call version.check_conds in=74b,call rgw_gc.rgw_gc_queue_init in=45b,call version.set in=34b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x5569c24c3ad0 con 0x5569c24d04c0 2023-10-05T19:28:33.234 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.226+0000 7f2a53196700 1 -- 172.21.15.167:0/3576704625 <== osd.2 v2:172.21.15.167:6816/30037 2 ==== osd_op_reply(9 gc.8 [create,call,call,call] v30'14 uv3 ondisk = -125 ((125) Operation canceled)) v8 ==== 274+0+0 (crc 0 0 0) 0x7f2a4c058040 con 0x5569c24d04c0 2023-10-05T19:28:33.234 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.226+0000 7f2a55661e40 1 -- 172.21.15.167:0/3576704625 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:10 6.1c 6:3e426232:gc::gc.9:head [create,call version.check_conds in=74b,call rgw_gc.rgw_gc_queue_init in=45b,call version.set in=34b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x5569c24c3ad0 con 0x5569c24c8ad0 2023-10-05T19:28:33.236 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.230+0000 7f2a53997700 1 -- 172.21.15.167:0/3576704625 <== osd.3 v2:172.21.15.167:6808/30036 3 ==== osd_op_reply(10 gc.9 [create,call,call,call] v30'25 uv3 ondisk = -125 ((125) Operation canceled)) v8 ==== 274+0+0 (crc 0 0 0) 0x7f2a44050040 con 0x5569c24c8ad0 2023-10-05T19:28:33.237 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.230+0000 7f2a55661e40 1 -- 172.21.15.167:0/3576704625 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:11 6.3 6:c31f41d7:gc::gc.10:head [create,call version.check_conds in=74b,call rgw_gc.rgw_gc_queue_init in=45b,call version.set in=34b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x5569c24c3d40 con 0x5569c24d04c0 2023-10-05T19:28:33.239 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.234+0000 7f2a53196700 1 -- 172.21.15.167:0/3576704625 <== osd.2 v2:172.21.15.167:6816/30037 3 ==== osd_op_reply(11 gc.10 [create,call,call,call] v30'33 uv11 ondisk = -125 ((125) Operation canceled)) v8 ==== 275+0+0 (crc 0 0 0) 0x7f2a4c058040 con 0x5569c24d04c0 2023-10-05T19:28:33.239 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.234+0000 7f2a55661e40 1 -- 172.21.15.167:0/3576704625 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:12 6.13 6:c8333aab:gc::gc.11:head [create,call version.check_conds in=74b,call rgw_gc.rgw_gc_queue_init in=45b,call version.set in=34b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x5569c24c3d40 con 0x5569c24c0bc0 2023-10-05T19:28:33.242 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.234+0000 7f2a52995700 1 -- 172.21.15.167:0/3576704625 <== osd.0 v2:172.21.15.118:6800/27690 3 ==== osd_op_reply(12 gc.11 [create,call,call,call] v30'17 uv5 ondisk = -125 ((125) Operation canceled)) v8 ==== 275+0+0 (crc 0 0 0) 0x7f2a3c03f040 con 0x5569c24c0bc0 2023-10-05T19:28:33.242 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.234+0000 7f2a55661e40 1 -- 172.21.15.167:0/3576704625 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:13 6.15 6:ab569f81:gc::gc.12:head [create,call version.check_conds in=74b,call rgw_gc.rgw_gc_queue_init in=45b,call version.set in=34b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x5569c24d4e10 con 0x5569c24d04c0 2023-10-05T19:28:33.244 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.238+0000 7f2a53196700 1 -- 172.21.15.167:0/3576704625 <== osd.2 v2:172.21.15.167:6816/30037 4 ==== osd_op_reply(13 gc.12 [create,call,call,call] v30'14 uv3 ondisk = -125 ((125) Operation canceled)) v8 ==== 275+0+0 (crc 0 0 0) 0x7f2a4c058040 con 0x5569c24d04c0 2023-10-05T19:28:33.244 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.238+0000 7f2a55661e40 1 -- 172.21.15.167:0/3576704625 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:14 6.0 6:0667bb13:gc::gc.13:head [create,call version.check_conds in=74b,call rgw_gc.rgw_gc_queue_init in=45b,call version.set in=34b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x5569c24d4e10 con 0x5569c24c0bc0 2023-10-05T19:28:33.248 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.242+0000 7f2a52995700 1 -- 172.21.15.167:0/3576704625 <== osd.0 v2:172.21.15.118:6800/27690 4 ==== osd_op_reply(14 gc.13 [create,call,call,call] v30'27 uv6 ondisk = -125 ((125) Operation canceled)) v8 ==== 275+0+0 (crc 0 0 0) 0x7f2a3c03f040 con 0x5569c24c0bc0 2023-10-05T19:28:33.248 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.242+0000 7f2a55661e40 1 -- 172.21.15.167:0/3576704625 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:15 6.4 6:227f63b5:gc::gc.14:head [create,call version.check_conds in=74b,call rgw_gc.rgw_gc_queue_init in=45b,call version.set in=34b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x5569c24d5300 con 0x5569c24c4f80 2023-10-05T19:28:33.250 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.246+0000 7f2a53196700 1 -- 172.21.15.167:0/3576704625 <== osd.1 v2:172.21.15.167:6800/30035 4 ==== osd_op_reply(15 gc.14 [create,call,call,call] v30'115 uv13 ondisk = -125 ((125) Operation canceled)) v8 ==== 275+0+0 (crc 0 0 0) 0x7f2a4c058040 con 0x5569c24c4f80 2023-10-05T19:28:33.250 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.246+0000 7f2a55661e40 1 -- 172.21.15.167:0/3576704625 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:16 6.7 6:e07a6548:gc::gc.15:head [create,call version.check_conds in=74b,call rgw_gc.rgw_gc_queue_init in=45b,call version.set in=34b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x5569c24d5300 con 0x5569c24c8ad0 2023-10-05T19:28:33.253 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.246+0000 7f2a53997700 1 -- 172.21.15.167:0/3576704625 <== osd.3 v2:172.21.15.167:6808/30036 4 ==== osd_op_reply(16 gc.15 [create,call,call,call] v30'26 uv7 ondisk = -125 ((125) Operation canceled)) v8 ==== 275+0+0 (crc 0 0 0) 0x7f2a44050040 con 0x5569c24c8ad0 2023-10-05T19:28:33.253 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.246+0000 7f2a55661e40 1 -- 172.21.15.167:0/3576704625 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:17 6.3 6:c623b173:gc::gc.16:head [create,call version.check_conds in=74b,call rgw_gc.rgw_gc_queue_init in=45b,call version.set in=34b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x5569c24d5300 con 0x5569c24d04c0 2023-10-05T19:28:33.257 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.250+0000 7f2a53196700 1 -- 172.21.15.167:0/3576704625 <== osd.2 v2:172.21.15.167:6816/30037 5 ==== osd_op_reply(17 gc.16 [create,call,call,call] v30'34 uv13 ondisk = -125 ((125) Operation canceled)) v8 ==== 275+0+0 (crc 0 0 0) 0x7f2a4c058040 con 0x5569c24d04c0 2023-10-05T19:28:33.258 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.250+0000 7f2a55661e40 1 -- 172.21.15.167:0/3576704625 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:18 6.16 6:6849fd06:gc::gc.17:head [create,call version.check_conds in=74b,call rgw_gc.rgw_gc_queue_init in=45b,call version.set in=34b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x5569c24d5300 con 0x5569c24c8ad0 2023-10-05T19:28:33.258 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.254+0000 7f2a53997700 1 -- 172.21.15.167:0/3576704625 <== osd.3 v2:172.21.15.167:6808/30036 5 ==== osd_op_reply(18 gc.17 [create,call,call,call] v30'14 uv7 ondisk = -125 ((125) Operation canceled)) v8 ==== 275+0+0 (crc 0 0 0) 0x7f2a44050040 con 0x5569c24c8ad0 2023-10-05T19:28:33.259 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.254+0000 7f2a55661e40 1 -- 172.21.15.167:0/3576704625 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:19 6.d 6:b019ea83:gc::gc.18:head [create,call version.check_conds in=74b,call rgw_gc.rgw_gc_queue_init in=45b,call version.set in=34b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x5569c24d53f0 con 0x5569c24c4f80 2023-10-05T19:28:33.261 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.254+0000 7f2a53196700 1 -- 172.21.15.167:0/3576704625 <== osd.1 v2:172.21.15.167:6800/30035 5 ==== osd_op_reply(19 gc.18 [create,call,call,call] v30'22 uv5 ondisk = -125 ((125) Operation canceled)) v8 ==== 275+0+0 (crc 0 0 0) 0x7f2a4c058040 con 0x5569c24c4f80 2023-10-05T19:28:33.261 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.254+0000 7f2a55661e40 1 -- 172.21.15.167:0/3576704625 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:20 6.4 6:24c8e1ae:gc::gc.19:head [create,call version.check_conds in=74b,call rgw_gc.rgw_gc_queue_init in=45b,call version.set in=34b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x5569c24d53f0 con 0x5569c24c4f80 2023-10-05T19:28:33.263 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.258+0000 7f2a53196700 1 -- 172.21.15.167:0/3576704625 <== osd.1 v2:172.21.15.167:6800/30035 6 ==== osd_op_reply(20 gc.19 [create,call,call,call] v30'116 uv15 ondisk = -125 ((125) Operation canceled)) v8 ==== 275+0+0 (crc 0 0 0) 0x7f2a4c058040 con 0x5569c24c4f80 2023-10-05T19:28:33.263 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.258+0000 7f2a55661e40 1 -- 172.21.15.167:0/3576704625 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:21 6.7 6:e622f76a:gc::gc.20:head [create,call version.check_conds in=74b,call rgw_gc.rgw_gc_queue_init in=45b,call version.set in=34b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x5569c24d5390 con 0x5569c24c8ad0 2023-10-05T19:28:33.266 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.262+0000 7f2a53997700 1 -- 172.21.15.167:0/3576704625 <== osd.3 v2:172.21.15.167:6808/30036 6 ==== osd_op_reply(21 gc.20 [create,call,call,call] v30'27 uv9 ondisk = -125 ((125) Operation canceled)) v8 ==== 275+0+0 (crc 0 0 0) 0x7f2a44050040 con 0x5569c24c8ad0 2023-10-05T19:28:33.266 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.262+0000 7f2a55661e40 1 -- 172.21.15.167:0/3576704625 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:22 6.a 6:5127e642:gc::gc.21:head [create,call version.check_conds in=74b,call rgw_gc.rgw_gc_queue_init in=45b,call version.set in=34b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x5569c24d5450 con 0x5569c24d04c0 2023-10-05T19:28:33.268 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.262+0000 7f2a53196700 1 -- 172.21.15.167:0/3576704625 <== osd.2 v2:172.21.15.167:6816/30037 6 ==== osd_op_reply(22 gc.21 [create,call,call,call] v30'24 uv7 ondisk = -125 ((125) Operation canceled)) v8 ==== 275+0+0 (crc 0 0 0) 0x7f2a4c058040 con 0x5569c24d04c0 2023-10-05T19:28:33.269 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.262+0000 7f2a55661e40 1 -- 172.21.15.167:0/3576704625 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:23 6.11 6:8eb99c53:gc::gc.22:head [create,call version.check_conds in=74b,call rgw_gc.rgw_gc_queue_init in=45b,call version.set in=34b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x5569c24d5450 con 0x5569c24c8ad0 2023-10-05T19:28:33.271 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.266+0000 7f2a53997700 1 -- 172.21.15.167:0/3576704625 <== osd.3 v2:172.21.15.167:6808/30036 7 ==== osd_op_reply(23 gc.22 [create,call,call,call] v30'16 uv3 ondisk = -125 ((125) Operation canceled)) v8 ==== 275+0+0 (crc 0 0 0) 0x7f2a44050040 con 0x5569c24c8ad0 2023-10-05T19:28:33.271 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.266+0000 7f2a55661e40 1 -- 172.21.15.167:0/3576704625 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:24 6.8 6:17d7e8ed:gc::gc.23:head [create,call version.check_conds in=74b,call rgw_gc.rgw_gc_queue_init in=45b,call version.set in=34b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x5569c24d5420 con 0x5569c24c8ad0 2023-10-05T19:28:33.273 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.266+0000 7f2a53997700 1 -- 172.21.15.167:0/3576704625 <== osd.3 v2:172.21.15.167:6808/30036 8 ==== osd_op_reply(24 gc.23 [create,call,call,call] v30'25 uv5 ondisk = -125 ((125) Operation canceled)) v8 ==== 275+0+0 (crc 0 0 0) 0x7f2a44050040 con 0x5569c24c8ad0 2023-10-05T19:28:33.274 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.266+0000 7f2a55661e40 1 -- 172.21.15.167:0/3576704625 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:25 6.1b 6:db9d130d:gc::gc.24:head [create,call version.check_conds in=74b,call rgw_gc.rgw_gc_queue_init in=45b,call version.set in=34b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x5569c24d5450 con 0x5569c24c8ad0 2023-10-05T19:28:33.276 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.270+0000 7f2a53997700 1 -- 172.21.15.167:0/3576704625 <== osd.3 v2:172.21.15.167:6808/30036 9 ==== osd_op_reply(25 gc.24 [create,call,call,call] v30'10 uv3 ondisk = -125 ((125) Operation canceled)) v8 ==== 275+0+0 (crc 0 0 0) 0x7f2a44050040 con 0x5569c24c8ad0 2023-10-05T19:28:33.276 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.270+0000 7f2a55661e40 1 -- 172.21.15.167:0/3576704625 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:26 6.9 6:921ecdb1:gc::gc.25:head [create,call version.check_conds in=74b,call rgw_gc.rgw_gc_queue_init in=45b,call version.set in=34b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x5569c24d5450 con 0x5569c24c8ad0 2023-10-05T19:28:33.278 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.274+0000 7f2a53997700 1 -- 172.21.15.167:0/3576704625 <== osd.3 v2:172.21.15.167:6808/30036 10 ==== osd_op_reply(26 gc.25 [create,call,call,call] v30'24 uv10 ondisk = -125 ((125) Operation canceled)) v8 ==== 275+0+0 (crc 0 0 0) 0x7f2a44050040 con 0x5569c24c8ad0 2023-10-05T19:28:33.278 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.274+0000 7f2a55661e40 1 -- 172.21.15.167:0/3576704625 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:27 6.1f 6:fde4dd55:gc::gc.26:head [create,call version.check_conds in=74b,call rgw_gc.rgw_gc_queue_init in=45b,call version.set in=34b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x5569c24d5420 con 0x5569c24c0bc0 2023-10-05T19:28:33.281 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.274+0000 7f2a52995700 1 -- 172.21.15.167:0/3576704625 <== osd.0 v2:172.21.15.118:6800/27690 5 ==== osd_op_reply(27 gc.26 [create,call,call,call] v30'16 uv5 ondisk = -125 ((125) Operation canceled)) v8 ==== 275+0+0 (crc 0 0 0) 0x7f2a3c03f040 con 0x5569c24c0bc0 2023-10-05T19:28:33.281 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.274+0000 7f2a55661e40 1 -- 172.21.15.167:0/3576704625 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:28 6.1a 6:5909e671:gc::gc.27:head [create,call version.check_conds in=74b,call rgw_gc.rgw_gc_queue_init in=45b,call version.set in=34b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x5569c24d5420 con 0x5569c24c8ad0 2023-10-05T19:28:33.283 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.278+0000 7f2a53997700 1 -- 172.21.15.167:0/3576704625 <== osd.3 v2:172.21.15.167:6808/30036 11 ==== osd_op_reply(28 gc.27 [create,call,call,call] v30'14 uv7 ondisk = -125 ((125) Operation canceled)) v8 ==== 275+0+0 (crc 0 0 0) 0x7f2a44050040 con 0x5569c24c8ad0 2023-10-05T19:28:33.283 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.278+0000 7f2a55661e40 1 -- 172.21.15.167:0/3576704625 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:29 6.1d 6:b88e3271:gc::gc.28:head [create,call version.check_conds in=74b,call rgw_gc.rgw_gc_queue_init in=45b,call version.set in=34b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x5569c24d5450 con 0x5569c24c0bc0 2023-10-05T19:28:33.285 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.278+0000 7f2a52995700 1 -- 172.21.15.167:0/3576704625 <== osd.0 v2:172.21.15.118:6800/27690 6 ==== osd_op_reply(29 gc.28 [create,call,call,call] v30'22 uv8 ondisk = -125 ((125) Operation canceled)) v8 ==== 275+0+0 (crc 0 0 0) 0x7f2a3c03f040 con 0x5569c24c0bc0 2023-10-05T19:28:33.286 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.278+0000 7f2a55661e40 1 -- 172.21.15.167:0/3576704625 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:30 6.7 6:e47c9322:gc::gc.29:head [create,call version.check_conds in=74b,call rgw_gc.rgw_gc_queue_init in=45b,call version.set in=34b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x5569c24d5450 con 0x5569c24c8ad0 2023-10-05T19:28:33.288 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.282+0000 7f2a53997700 1 -- 172.21.15.167:0/3576704625 <== osd.3 v2:172.21.15.167:6808/30036 12 ==== osd_op_reply(30 gc.29 [create,call,call,call] v30'28 uv11 ondisk = -125 ((125) Operation canceled)) v8 ==== 275+0+0 (crc 0 0 0) 0x7f2a44050040 con 0x5569c24c8ad0 2023-10-05T19:28:33.288 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.282+0000 7f2a55661e40 1 -- 172.21.15.167:0/3576704625 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:31 6.0 6:013d6c33:gc::gc.30:head [create,call version.check_conds in=74b,call rgw_gc.rgw_gc_queue_init in=45b,call version.set in=34b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x5569c24d5420 con 0x5569c24c0bc0 2023-10-05T19:28:33.291 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.286+0000 7f2a52995700 1 -- 172.21.15.167:0/3576704625 <== osd.0 v2:172.21.15.118:6800/27690 7 ==== osd_op_reply(31 gc.30 [create,call,call,call] v30'28 uv12 ondisk = -125 ((125) Operation canceled)) v8 ==== 275+0+0 (crc 0 0 0) 0x7f2a3c03f040 con 0x5569c24c0bc0 2023-10-05T19:28:33.291 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.286+0000 7f2a55661e40 1 -- 172.21.15.167:0/3576704625 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:32 6.f 6:f24cb4a8:gc::gc.31:head [create,call version.check_conds in=74b,call rgw_gc.rgw_gc_queue_init in=45b,call version.set in=34b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x5569c24d5420 con 0x5569c24d04c0 2023-10-05T19:28:33.294 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.290+0000 7f2a53196700 1 -- 172.21.15.167:0/3576704625 <== osd.2 v2:172.21.15.167:6816/30037 7 ==== osd_op_reply(32 gc.31 [create,call,call,call] v30'20 uv5 ondisk = -125 ((125) Operation canceled)) v8 ==== 275+0+0 (crc 0 0 0) 0x7f2a4c058040 con 0x5569c24d04c0 2023-10-05T19:28:33.296 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.290+0000 7f29937fe700 20 reqs_thread_entry: start 2023-10-05T19:28:33.297 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.290+0000 7f2a55661e40 20 init_complete bucket index max shards: 11 2023-10-05T19:28:33.297 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.290+0000 7f29917fa700 20 reqs_thread_entry: start 2023-10-05T19:28:33.297 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.290+0000 7f2a55661e40 10 rgw notify: Started notification manager with: 1 workers 2023-10-05T19:28:33.298 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.290+0000 7f297f7fe700 20 rgw notify: INFO: next queues processing will happen at: Thu Oct 5 19:29:03 2023 2023-10-05T19:28:33.298 INFO:teuthology.orchestra.run.smithi167.stderr: 2023-10-05T19:28:33.298 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.290+0000 7f297effd700 20 reqs_thread_entry: start 2023-10-05T19:28:33.298 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.290+0000 7f2a55661e40 20 get_system_obj_state: rctx=0x7fff05a9f1e8 obj=default.rgw.meta:users.uid:fud state=0x5569c233d0b0 s->prefetch_data=0 2023-10-05T19:28:33.298 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.290+0000 7f2a55661e40 10 cache get: name=default.rgw.meta+users.uid+fud : miss 2023-10-05T19:28:33.298 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.290+0000 7f2a55661e40 1 -- 172.21.15.167:0/1691752437 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:102 8.e 8:75f48285:users.uid::fud:head [call version.read in=11b,getxattrs,stat] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x5569c2533e70 con 0x7f2a3c00ebe0 2023-10-05T19:28:33.299 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.294+0000 7f2a53997700 1 -- 172.21.15.167:0/1691752437 <== osd.0 v2:172.21.15.118:6800/27690 29 ==== osd_op_reply(102 fud [call out=48b,getxattrs out=75b,stat out=16b] v0'0 uv1 ondisk = 0) v8 ==== 231+0+139 (crc 0 0 0) 0x7f2a44050040 con 0x7f2a3c00ebe0 2023-10-05T19:28:33.299 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.294+0000 7f2a55661e40 10 cache put: name=default.rgw.meta+users.uid+fud info.flags=0x16 2023-10-05T19:28:33.299 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.294+0000 7f2a55661e40 10 adding default.rgw.meta+users.uid+fud to cache LRU end 2023-10-05T19:28:33.299 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.294+0000 7f2a55661e40 10 updating xattr: name=ceph.objclass.version bl.length()=42 2023-10-05T19:28:33.299 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.294+0000 7f2a55661e40 20 get_system_obj_state: s->obj_tag was set empty 2023-10-05T19:28:33.299 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.294+0000 7f2a55661e40 20 Read xattr: user.rgw.idtag 2023-10-05T19:28:33.299 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.294+0000 7f2a55661e40 10 cache get: name=default.rgw.meta+users.uid+fud : type miss (requested=0x13, cached=0x16) 2023-10-05T19:28:33.299 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.294+0000 7f2a55661e40 20 rados->read ofs=0 len=0 2023-10-05T19:28:33.299 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.294+0000 7f2a55661e40 1 -- 172.21.15.167:0/1691752437 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:103 8.e 8:75f48285:users.uid::fud:head [call version.check_conds in=122b,call version.read in=11b,read 0~0,getxattrs] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x5569c2536900 con 0x7f2a3c00ebe0 2023-10-05T19:28:33.300 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.294+0000 7f2a53997700 1 -- 172.21.15.167:0/1691752437 <== osd.0 v2:172.21.15.118:6800/27690 30 ==== osd_op_reply(103 fud [call,call out=48b,read 0~346 out=346b,getxattrs out=75b] v0'0 uv1 ondisk = 0) v8 ==== 273+0+469 (crc 0 0 0) 0x7f2a44050040 con 0x7f2a3c00ebe0 2023-10-05T19:28:33.300 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.294+0000 7f2a55661e40 20 rados_obj.operate() r=0 bl.length=346 2023-10-05T19:28:33.300 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.294+0000 7f2a55661e40 10 cache put: name=default.rgw.meta+users.uid+fud info.flags=0x13 2023-10-05T19:28:33.300 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.294+0000 7f2a55661e40 10 moving default.rgw.meta+users.uid+fud to cache LRU end 2023-10-05T19:28:33.300 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.294+0000 7f2a55661e40 10 updating xattr: name=ceph.objclass.version bl.length()=42 2023-10-05T19:28:33.300 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.294+0000 7f2a55661e40 1 -- 172.21.15.167:0/1691752437 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:104 8.1 8:810902b1:users.uid::fud.buckets:head [call user.list_buckets in=34b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x5569c2536900 con 0x7f2a3c00ebe0 2023-10-05T19:28:33.301 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.294+0000 7f2a53997700 1 -- 172.21.15.167:0/1691752437 <== osd.0 v2:172.21.15.118:6800/27690 31 ==== osd_op_reply(104 fud.buckets [call out=183b] v0'0 uv1 ondisk = 0) v8 ==== 155+0+183 (crc 0 0 0) 0x7f2a44050040 con 0x7f2a3c00ebe0 2023-10-05T19:28:33.301 INFO:teuthology.orchestra.run.smithi167.stderr:could not remove user: unable to remove user, must specify purge data to remove user with buckets 2023-10-05T19:28:33.301 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.294+0000 7f2a55661e40 1 -- 172.21.15.167:0/1691752437 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:105 7.1f 7:f95f44c2:::notify.0:head [watch unwatch cookie 93912719559824] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x5569c24f2420 con 0x7f2a4c038030 2023-10-05T19:28:33.305 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.298+0000 7f2a52995700 1 -- 172.21.15.167:0/1691752437 <== osd.2 v2:172.21.15.167:6816/30037 17 ==== osd_op_reply(105 notify.0 [watch unwatch cookie 93912719559824] v30'119 uv117 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f2a3c03f040 con 0x7f2a4c038030 2023-10-05T19:28:33.305 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.298+0000 7f2a55661e40 20 remove_watcher() i=0 2023-10-05T19:28:33.305 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.298+0000 7f2a55661e40 2 removed watcher, disabling cache 2023-10-05T19:28:33.305 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.298+0000 7f2a55661e40 1 -- 172.21.15.167:0/1691752437 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:106 7.0 7:05bf5b68:::notify.1:head [watch unwatch cookie 93912719563424] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x5569c247b7f0 con 0x5569c2495710 2023-10-05T19:28:33.307 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.302+0000 7f2a53997700 1 -- 172.21.15.167:0/1691752437 <== osd.1 v2:172.21.15.167:6800/30035 35 ==== osd_op_reply(106 notify.1 [watch unwatch cookie 93912719563424] v30'119 uv117 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f2a44050040 con 0x5569c2495710 2023-10-05T19:28:33.307 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.302+0000 7f2a55661e40 20 remove_watcher() i=1 2023-10-05T19:28:33.307 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.302+0000 7f2a55661e40 1 -- 172.21.15.167:0/1691752437 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:107 7.15 7:a93a5511:::notify.2:head [watch unwatch cookie 93912719566976] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x5569c247b7f0 con 0x5569c249b8d0 2023-10-05T19:28:33.310 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.306+0000 7f2a53196700 1 -- 172.21.15.167:0/1691752437 <== osd.3 v2:172.21.15.167:6808/30036 24 ==== osd_op_reply(107 notify.2 [watch unwatch cookie 93912719566976] v30'119 uv117 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f2a4c058040 con 0x5569c249b8d0 2023-10-05T19:28:33.310 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.306+0000 7f2a55661e40 20 remove_watcher() i=2 2023-10-05T19:28:33.310 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.306+0000 7f2a55661e40 1 -- 172.21.15.167:0/1691752437 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:108 7.e 7:7759931f:::notify.3:head [watch unwatch cookie 93912719564528] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x5569c247b7f0 con 0x7f2a4c038030 2023-10-05T19:28:33.313 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.306+0000 7f2a52995700 1 -- 172.21.15.167:0/1691752437 <== osd.2 v2:172.21.15.167:6816/30037 18 ==== osd_op_reply(108 notify.3 [watch unwatch cookie 93912719564528] v30'119 uv117 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f2a3c03f040 con 0x7f2a4c038030 2023-10-05T19:28:33.313 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.306+0000 7f2a55661e40 20 remove_watcher() i=3 2023-10-05T19:28:33.313 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.306+0000 7f2a55661e40 1 -- 172.21.15.167:0/1691752437 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:109 7.d 7:b4812045:::notify.4:head [watch unwatch cookie 93912719574336] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x5569c24a1f90 con 0x5569c2495710 2023-10-05T19:28:33.316 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.310+0000 7f2a53997700 1 -- 172.21.15.167:0/1691752437 <== osd.1 v2:172.21.15.167:6800/30035 36 ==== osd_op_reply(109 notify.4 [watch unwatch cookie 93912719574336] v30'119 uv117 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f2a44050040 con 0x5569c2495710 2023-10-05T19:28:33.316 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.310+0000 7f2a55661e40 20 remove_watcher() i=4 2023-10-05T19:28:33.316 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.310+0000 7f2a55661e40 1 -- 172.21.15.167:0/1691752437 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:110 7.3 7:c609908c:::notify.5:head [watch unwatch cookie 93912719569840] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x5569c24a1f90 con 0x7f2a3c00ebe0 2023-10-05T19:28:33.318 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.314+0000 7f2a53997700 1 -- 172.21.15.167:0/1691752437 <== osd.0 v2:172.21.15.118:6800/27690 32 ==== osd_op_reply(110 notify.5 [watch unwatch cookie 93912719569840] v30'119 uv117 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f2a44050040 con 0x7f2a3c00ebe0 2023-10-05T19:28:33.319 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.314+0000 7f2a55661e40 20 remove_watcher() i=5 2023-10-05T19:28:33.319 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.314+0000 7f2a55661e40 1 -- 172.21.15.167:0/1691752437 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:111 7.14 7:2b04a3e9:::notify.6:head [watch unwatch cookie 93912719578944] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x5569c24a1f90 con 0x7f2a3c00ebe0 2023-10-05T19:28:33.321 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.314+0000 7f2a53997700 1 -- 172.21.15.167:0/1691752437 <== osd.0 v2:172.21.15.118:6800/27690 33 ==== osd_op_reply(111 notify.6 [watch unwatch cookie 93912719578944] v30'119 uv117 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f2a44050040 con 0x7f2a3c00ebe0 2023-10-05T19:28:33.321 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.314+0000 7f2a55661e40 20 remove_watcher() i=6 2023-10-05T19:28:33.322 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.314+0000 7f2a55661e40 1 -- 172.21.15.167:0/1691752437 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:112 7.9 7:93e5b521:::notify.7:head [watch unwatch cookie 93912719583536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x5569c24a1f90 con 0x7f2a3c00ebe0 2023-10-05T19:28:33.324 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.318+0000 7f2a53997700 1 -- 172.21.15.167:0/1691752437 <== osd.0 v2:172.21.15.118:6800/27690 34 ==== osd_op_reply(112 notify.7 [watch unwatch cookie 93912719583536] v30'119 uv117 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f2a44050040 con 0x7f2a3c00ebe0 2023-10-05T19:28:33.325 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.318+0000 7f2a55661e40 20 remove_watcher() i=7 2023-10-05T19:28:33.326 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.322+0000 7f2a24fe1700 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2023-10-05T19:28:33.326 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.322+0000 7f2a55661e40 1 -- 172.21.15.167:0/1691752437 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:113 6.4 6:22d26bf9:::data_loggenerations_metadata:head [watch unwatch cookie 93912719584640] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x5569c24a1f90 con 0x5569c2495710 2023-10-05T19:28:33.331 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.322+0000 7f2a53997700 1 -- 172.21.15.167:0/1691752437 <== osd.1 v2:172.21.15.167:6800/30035 37 ==== osd_op_reply(113 data_loggenerations_metadata [watch unwatch cookie 93912719584640] v30'117 uv1 ondisk = 0) v8 ==== 172+0+0 (crc 0 0 0) 0x7f2a44050040 con 0x5569c2495710 2023-10-05T19:28:33.331 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.326+0000 7f2a55661e40 1 -- 172.21.15.167:0/1691752437 >> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] conn(0x7f2a3c00ebe0 msgr2=0x7f2a3c00d8e0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-05T19:28:33.331 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.326+0000 7f2a55661e40 1 --2- 172.21.15.167:0/1691752437 >> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] conn(0x7f2a3c00ebe0 0x7f2a3c00d8e0 crc :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-05T19:28:33.331 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.326+0000 7f2a55661e40 1 -- 172.21.15.167:0/1691752437 >> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] conn(0x5569c2495710 msgr2=0x5569c2497af0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-05T19:28:33.331 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.326+0000 7f2a55661e40 1 --2- 172.21.15.167:0/1691752437 >> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] conn(0x5569c2495710 0x5569c2497af0 crc :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-05T19:28:33.331 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.326+0000 7f2a55661e40 1 -- 172.21.15.167:0/1691752437 >> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] conn(0x7f2a4c038030 msgr2=0x7f2a4c0079b0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-05T19:28:33.331 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.326+0000 7f2a55661e40 1 --2- 172.21.15.167:0/1691752437 >> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] conn(0x7f2a4c038030 0x7f2a4c0079b0 crc :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-05T19:28:33.331 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.326+0000 7f2a55661e40 1 -- 172.21.15.167:0/1691752437 >> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] conn(0x5569c249b8d0 msgr2=0x5569c249dcb0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-05T19:28:33.332 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.326+0000 7f2a55661e40 1 --2- 172.21.15.167:0/1691752437 >> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] conn(0x5569c249b8d0 0x5569c249dcb0 crc :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-05T19:28:33.332 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.326+0000 7f2a55661e40 1 -- 172.21.15.167:0/1691752437 >> [v2:172.21.15.167:6824/29889,v1:172.21.15.167:6825/29889] conn(0x7f2a34030c50 msgr2=0x7f2a34033100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-05T19:28:33.332 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.326+0000 7f2a55661e40 1 --2- 172.21.15.167:0/1691752437 >> [v2:172.21.15.167:6824/29889,v1:172.21.15.167:6825/29889] conn(0x7f2a34030c50 0x7f2a34033100 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7f2a3c004170 tx=0x7f2a3c001400 comp rx=0 tx=0).stop 2023-10-05T19:28:33.332 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.326+0000 7f2a55661e40 1 -- 172.21.15.167:0/1691752437 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5569c24647e0 msgr2=0x5569c248bce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-05T19:28:33.332 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.326+0000 7f2a55661e40 1 --2- 172.21.15.167:0/1691752437 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5569c24647e0 0x5569c248bce0 secure :-1 s=READY pgs=371 cs=0 l=1 rev1=1 crypto rx=0x7f2a4c000f80 tx=0x7f2a4c01d000 comp rx=0 tx=0).stop 2023-10-05T19:28:33.332 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.326+0000 7f2a53997700 1 -- 172.21.15.167:0/1691752437 reap_dead start 2023-10-05T19:28:33.332 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.326+0000 7f2a55661e40 1 -- 172.21.15.167:0/1691752437 shutdown_connections 2023-10-05T19:28:33.332 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.326+0000 7f2a55661e40 1 --2- 172.21.15.167:0/1691752437 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5569c24647e0 0x5569c248bce0 unknown :-1 s=CLOSED pgs=371 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-05T19:28:33.332 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.326+0000 7f2a55661e40 1 -- 172.21.15.167:0/1691752437 >> 172.21.15.167:0/1691752437 conn(0x5569c245e720 msgr2=0x5569c247f550 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-05T19:28:33.332 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.326+0000 7f2a55661e40 1 -- 172.21.15.167:0/1691752437 shutdown_connections 2023-10-05T19:28:33.333 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.326+0000 7f2a55661e40 1 -- 172.21.15.167:0/1691752437 wait complete. 2023-10-05T19:28:33.334 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.326+0000 7f2a55661e40 1 -- 172.21.15.167:0/3576704625 >> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] conn(0x5569c24c0bc0 msgr2=0x5569c24cb0a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-05T19:28:33.334 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.326+0000 7f2a55661e40 1 --2- 172.21.15.167:0/3576704625 >> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] conn(0x5569c24c0bc0 0x5569c24cb0a0 crc :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-05T19:28:33.334 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.326+0000 7f2a55661e40 1 -- 172.21.15.167:0/3576704625 >> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] conn(0x5569c24c4f80 msgr2=0x5569c24c7390 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-05T19:28:33.334 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.326+0000 7f2a55661e40 1 --2- 172.21.15.167:0/3576704625 >> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] conn(0x5569c24c4f80 0x5569c24c7390 crc :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-05T19:28:33.334 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.326+0000 7f2a55661e40 1 -- 172.21.15.167:0/3576704625 >> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] conn(0x5569c24d04c0 msgr2=0x5569c24d28a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-05T19:28:33.334 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.326+0000 7f2a55661e40 1 --2- 172.21.15.167:0/3576704625 >> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] conn(0x5569c24d04c0 0x5569c24d28a0 crc :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-05T19:28:33.334 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.326+0000 7f2a55661e40 1 -- 172.21.15.167:0/3576704625 >> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] conn(0x5569c24c8ad0 msgr2=0x5569c24cec50 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-05T19:28:33.334 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.326+0000 7f2a55661e40 1 --2- 172.21.15.167:0/3576704625 >> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] conn(0x5569c24c8ad0 0x5569c24cec50 crc :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-05T19:28:33.335 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.326+0000 7f2a55661e40 1 -- 172.21.15.167:0/3576704625 >> [v2:172.21.15.167:6824/29889,v1:172.21.15.167:6825/29889] conn(0x7f2994030c00 msgr2=0x7f29940330b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-05T19:28:33.335 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.326+0000 7f2a55661e40 1 --2- 172.21.15.167:0/3576704625 >> [v2:172.21.15.167:6824/29889,v1:172.21.15.167:6825/29889] conn(0x7f2994030c00 0x7f29940330b0 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x5569c24a7200 tx=0x7f2a44000f90 comp rx=0 tx=0).stop 2023-10-05T19:28:33.335 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.326+0000 7f2a55661e40 1 -- 172.21.15.167:0/3576704625 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5569c24b47b0 msgr2=0x5569c24ba390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-05T19:28:33.335 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.326+0000 7f2a53196700 1 -- 172.21.15.167:0/3576704625 reap_dead start 2023-10-05T19:28:33.335 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.326+0000 7f2a55661e40 1 --2- 172.21.15.167:0/3576704625 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5569c24b47b0 0x5569c24ba390 secure :-1 s=READY pgs=373 cs=0 l=1 rev1=1 crypto rx=0x5569c24ae830 tx=0x7f2a3c036000 comp rx=0 tx=0).stop 2023-10-05T19:28:33.335 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.326+0000 7f2a55661e40 1 -- 172.21.15.167:0/3576704625 shutdown_connections 2023-10-05T19:28:33.335 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.326+0000 7f2a55661e40 1 --2- 172.21.15.167:0/3576704625 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5569c24b47b0 0x5569c24ba390 unknown :-1 s=CLOSED pgs=373 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-05T19:28:33.335 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.326+0000 7f2a55661e40 1 -- 172.21.15.167:0/3576704625 >> 172.21.15.167:0/3576704625 conn(0x5569c24afb70 msgr2=0x5569c24adc10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-05T19:28:33.335 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.326+0000 7f2a55661e40 1 -- 172.21.15.167:0/3576704625 shutdown_connections 2023-10-05T19:28:33.335 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.326+0000 7f2a55661e40 1 -- 172.21.15.167:0/3576704625 wait complete. 2023-10-05T19:28:33.337 DEBUG:teuthology.orchestra.run:got remote process result: 17 2023-10-05T19:28:33.338 INFO:tasks.util.rgw:rgwadmin: client.0 : ['bucket', 'unlink', '--uid', 'fud', '--bucket', 'myfoo'] 2023-10-05T19:28:33.338 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'ceph', 'bucket', 'unlink', '--uid', 'fud', '--bucket', 'myfoo'] 2023-10-05T19:28:33.338 DEBUG:teuthology.orchestra.run.smithi167:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster ceph bucket unlink --uid fud --bucket myfoo 2023-10-05T19:28:33.378 INFO:teuthology.orchestra.run.smithi167.stderr:ignoring --setuser ceph since I am not root 2023-10-05T19:28:33.378 INFO:teuthology.orchestra.run.smithi167.stderr:ignoring --setgroup ceph since I am not root 2023-10-05T19:28:33.379 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.374+0000 7fd27cb0ee40 1 Processor -- start 2023-10-05T19:28:33.379 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.374+0000 7fd27cb0ee40 1 -- start start 2023-10-05T19:28:33.379 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.374+0000 7fd27cb0ee40 1 --2- >> v2:172.21.15.118:3300/0 conn(0x56088e4fb7f0 0x56088e4fa6e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-05T19:28:33.379 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.374+0000 7fd27cb0ee40 1 -- --> v1:172.21.15.118:6789/0 -- auth(proto 0 26 bytes epoch 0) v1 -- 0x56088e447bd0 con 0x56088e4fb420 2023-10-05T19:28:33.379 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.374+0000 7fd27cb0ee40 1 -- --> v2:172.21.15.118:3300/0 -- mon_getmap magic: 0 v1 -- 0x56088e404ee0 con 0x56088e4fb7f0 2023-10-05T19:28:33.380 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.374+0000 7fd279e42700 1 --2- >> v2:172.21.15.118:3300/0 conn(0x56088e4fb7f0 0x56088e4fa6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-05T19:28:33.380 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.374+0000 7fd27a643700 1 --1- >> v1:172.21.15.118:6789/0 conn(0x56088e4fb420 0x56088e4f9fd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.118:6789/0 says I am v1:172.21.15.167:56658/0 (socket says 172.21.15.167:56658) 2023-10-05T19:28:33.380 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.374+0000 7fd27a643700 1 -- 172.21.15.167:0/3776626315 learned_addr learned my addr 172.21.15.167:0/3776626315 (peer_addr_for_me v1:172.21.15.167:0/0) 2023-10-05T19:28:33.381 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.374+0000 7fd279641700 1 -- 172.21.15.167:0/3776626315 <== mon.0 v1:172.21.15.118:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2255626769 0 0) 0x56088e447bd0 con 0x56088e4fb420 2023-10-05T19:28:33.381 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.374+0000 7fd279641700 1 -- 172.21.15.167:0/3776626315 --> v1:172.21.15.118:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd268001e10 con 0x56088e4fb420 2023-10-05T19:28:33.381 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.374+0000 7fd279e42700 1 -- 172.21.15.167:0/3776626315 >> v1:172.21.15.118:6789/0 conn(0x56088e4fb420 legacy=0x56088e4f9fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-05T19:28:33.381 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.374+0000 7fd279e42700 1 -- 172.21.15.167:0/3776626315 --> v2:172.21.15.118:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x56088e4face0 con 0x56088e4fb7f0 2023-10-05T19:28:33.381 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.374+0000 7fd279e42700 1 --2- 172.21.15.167:0/3776626315 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x56088e4fb7f0 0x56088e4fa6e0 secure :-1 s=READY pgs=375 cs=0 l=1 rev1=1 crypto rx=0x7fd26400a5d0 tx=0x7fd264005a50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=595c2294fa55a054 server_cookie=0 in_seq=0 out_seq=0 2023-10-05T19:28:33.382 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.374+0000 7fd279641700 1 -- 172.21.15.167:0/3776626315 <== mon.0 v2:172.21.15.118:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fd264007610 con 0x56088e4fb7f0 2023-10-05T19:28:33.382 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.374+0000 7fd279641700 1 -- 172.21.15.167:0/3776626315 <== mon.0 v2:172.21.15.118:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd264007770 con 0x56088e4fb7f0 2023-10-05T19:28:33.382 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.374+0000 7fd279641700 1 -- 172.21.15.167:0/3776626315 <== mon.0 v2:172.21.15.118:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fd264007a20 con 0x56088e4fb7f0 2023-10-05T19:28:33.382 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.378+0000 7fd27cb0ee40 1 -- 172.21.15.167:0/3776626315 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x56088e4fb7f0 msgr2=0x56088e4fa6e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-05T19:28:33.383 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.378+0000 7fd27cb0ee40 1 --2- 172.21.15.167:0/3776626315 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x56088e4fb7f0 0x56088e4fa6e0 secure :-1 s=READY pgs=375 cs=0 l=1 rev1=1 crypto rx=0x7fd26400a5d0 tx=0x7fd264005a50 comp rx=0 tx=0).stop 2023-10-05T19:28:33.383 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.378+0000 7fd27cb0ee40 1 -- 172.21.15.167:0/3776626315 shutdown_connections 2023-10-05T19:28:33.383 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.378+0000 7fd27cb0ee40 1 --2- 172.21.15.167:0/3776626315 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x56088e4fb7f0 0x56088e4fa6e0 unknown :-1 s=CLOSED pgs=375 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-05T19:28:33.383 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.378+0000 7fd27cb0ee40 1 -- 172.21.15.167:0/3776626315 >> 172.21.15.167:0/3776626315 conn(0x56088e4f5730 msgr2=0x56088e4f7b40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-05T19:28:33.383 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.378+0000 7fd27cb0ee40 1 -- 172.21.15.167:0/3776626315 shutdown_connections 2023-10-05T19:28:33.383 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.378+0000 7fd27cb0ee40 1 -- 172.21.15.167:0/3776626315 wait complete. 2023-10-05T19:28:33.384 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.378+0000 7fd27cb0ee40 1 Processor -- start 2023-10-05T19:28:33.384 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.378+0000 7fd27cb0ee40 1 -- start start 2023-10-05T19:28:33.384 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.378+0000 7fd27cb0ee40 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x56088e4fb420 0x56088e51bf90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-05T19:28:33.384 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.378+0000 7fd27cb0ee40 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x56088e3c8e30 con 0x56088e4fb420 2023-10-05T19:28:33.384 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.378+0000 7fd27a643700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x56088e4fb420 0x56088e51bf90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-05T19:28:33.384 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.378+0000 7fd27a643700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x56088e4fb420 0x56088e51bf90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.118:3300/0 says I am v2:172.21.15.167:58602/0 (socket says 172.21.15.167:58602) 2023-10-05T19:28:33.384 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.378+0000 7fd27a643700 1 -- 172.21.15.167:0/3148492127 learned_addr learned my addr 172.21.15.167:0/3148492127 (peer_addr_for_me v2:172.21.15.167:0/0) 2023-10-05T19:28:33.385 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.378+0000 7fd27a643700 1 -- 172.21.15.167:0/3148492127 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd264005cf0 con 0x56088e4fb420 2023-10-05T19:28:33.385 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.378+0000 7fd27a643700 1 --2- 172.21.15.167:0/3148492127 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x56088e4fb420 0x56088e51bf90 secure :-1 s=READY pgs=376 cs=0 l=1 rev1=1 crypto rx=0x7fd274006e30 tx=0x7fd274006a10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9b7bffbf65b956e3 server_cookie=0 in_seq=0 out_seq=0 2023-10-05T19:28:33.385 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.378+0000 7fd273fff700 1 -- 172.21.15.167:0/3148492127 <== mon.0 v2:172.21.15.118:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fd274006110 con 0x56088e4fb420 2023-10-05T19:28:33.385 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.378+0000 7fd273fff700 1 -- 172.21.15.167:0/3148492127 <== mon.0 v2:172.21.15.118:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd274006290 con 0x56088e4fb420 2023-10-05T19:28:33.385 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.378+0000 7fd273fff700 1 -- 172.21.15.167:0/3148492127 <== mon.0 v2:172.21.15.118:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fd274004520 con 0x56088e4fb420 2023-10-05T19:28:33.387 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.378+0000 7fd27cb0ee40 1 -- 172.21.15.167:0/3148492127 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x56088e4fb420 msgr2=0x56088e51bf90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-05T19:28:33.387 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.378+0000 7fd27cb0ee40 1 --2- 172.21.15.167:0/3148492127 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x56088e4fb420 0x56088e51bf90 secure :-1 s=READY pgs=376 cs=0 l=1 rev1=1 crypto rx=0x7fd274006e30 tx=0x7fd274006a10 comp rx=0 tx=0).stop 2023-10-05T19:28:33.387 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.378+0000 7fd27cb0ee40 1 -- 172.21.15.167:0/3148492127 shutdown_connections 2023-10-05T19:28:33.387 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.378+0000 7fd27cb0ee40 1 --2- 172.21.15.167:0/3148492127 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x56088e4fb420 0x56088e51bf90 unknown :-1 s=CLOSED pgs=376 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-05T19:28:33.387 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.378+0000 7fd27cb0ee40 1 -- 172.21.15.167:0/3148492127 >> 172.21.15.167:0/3148492127 conn(0x56088e4f5730 msgr2=0x56088e517b50 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-05T19:28:33.387 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.378+0000 7fd27cb0ee40 1 -- 172.21.15.167:0/3148492127 shutdown_connections 2023-10-05T19:28:33.387 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.382+0000 7fd27cb0ee40 1 -- 172.21.15.167:0/3148492127 wait complete. 2023-10-05T19:28:33.387 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.382+0000 7fd27cb0ee40 1 Processor -- start 2023-10-05T19:28:33.387 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.382+0000 7fd27cb0ee40 1 -- start start 2023-10-05T19:28:33.388 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.382+0000 7fd27cb0ee40 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x56088e4fb420 0x56088e522d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-05T19:28:33.388 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.382+0000 7fd27cb0ee40 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x56088e39bed0 con 0x56088e4fb420 2023-10-05T19:28:33.388 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.382+0000 7fd27a643700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x56088e4fb420 0x56088e522d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-05T19:28:33.389 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.382+0000 7fd27a643700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x56088e4fb420 0x56088e522d20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.118:3300/0 says I am v2:172.21.15.167:58604/0 (socket says 172.21.15.167:58604) 2023-10-05T19:28:33.389 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.382+0000 7fd27a643700 1 -- 172.21.15.167:0/403903602 learned_addr learned my addr 172.21.15.167:0/403903602 (peer_addr_for_me v2:172.21.15.167:0/0) 2023-10-05T19:28:33.389 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.382+0000 7fd27a643700 1 -- 172.21.15.167:0/403903602 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd268001e10 con 0x56088e4fb420 2023-10-05T19:28:33.389 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.382+0000 7fd27a643700 1 --2- 172.21.15.167:0/403903602 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x56088e4fb420 0x56088e522d20 secure :-1 s=READY pgs=377 cs=0 l=1 rev1=1 crypto rx=0x7fd274001530 tx=0x7fd274010000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-05T19:28:33.389 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.382+0000 7fd2727fc700 1 -- 172.21.15.167:0/403903602 <== mon.0 v2:172.21.15.118:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fd274006110 con 0x56088e4fb420 2023-10-05T19:28:33.390 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.382+0000 7fd2727fc700 1 -- 172.21.15.167:0/403903602 <== mon.0 v2:172.21.15.118:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd274006400 con 0x56088e4fb420 2023-10-05T19:28:33.390 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.382+0000 7fd27cb0ee40 1 -- 172.21.15.167:0/403903602 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x56088e523260 con 0x56088e4fb420 2023-10-05T19:28:33.390 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.382+0000 7fd2727fc700 1 -- 172.21.15.167:0/403903602 <== mon.0 v2:172.21.15.118:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fd274014bd0 con 0x56088e4fb420 2023-10-05T19:28:33.390 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.382+0000 7fd27cb0ee40 1 -- 172.21.15.167:0/403903602 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x56088e5236a0 con 0x56088e4fb420 2023-10-05T19:28:33.390 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.386+0000 7fd27cb0ee40 20 rados->read ofs=0 len=0 2023-10-05T19:28:33.391 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.386+0000 7fd2727fc700 1 -- 172.21.15.167:0/403903602 <== mon.0 v2:172.21.15.118:3300/0 4 ==== mgrmap(e 5) v1 ==== 40329+0+0 (secure 0 0 0) 0x7fd274028460 con 0x56088e4fb420 2023-10-05T19:28:33.391 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.386+0000 7fd2727fc700 1 --2- 172.21.15.167:0/403903602 >> [v2:172.21.15.167:6824/29889,v1:172.21.15.167:6825/29889] conn(0x7fd25c030c40 0x7fd25c0330f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-05T19:28:33.391 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.386+0000 7fd2727fc700 1 -- 172.21.15.167:0/403903602 <== mon.0 v2:172.21.15.118:3300/0 5 ==== osd_map(30..30 src has 1..30) v4 ==== 6057+0+0 (secure 0 0 0) 0x7fd274045130 con 0x56088e4fb420 2023-10-05T19:28:33.392 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.386+0000 7fd279e42700 1 --2- 172.21.15.167:0/403903602 >> [v2:172.21.15.167:6824/29889,v1:172.21.15.167:6825/29889] conn(0x7fd25c030c40 0x7fd25c0330f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-05T19:28:33.392 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.386+0000 7fd27cb0ee40 1 --2- 172.21.15.167:0/403903602 >> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] conn(0x56088e4fb7f0 0x56088e52e7e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-05T19:28:33.392 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.386+0000 7fd27cb0ee40 1 -- 172.21.15.167:0/403903602 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:1 5.12 5:49953fa1:::default.realm:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x56088e52eee0 con 0x56088e4fb7f0 2023-10-05T19:28:33.392 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.386+0000 7fd27ae44700 1 --2- 172.21.15.167:0/403903602 >> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] conn(0x56088e4fb7f0 0x56088e52e7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-05T19:28:33.393 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.386+0000 7fd279e42700 1 --2- 172.21.15.167:0/403903602 >> [v2:172.21.15.167:6824/29889,v1:172.21.15.167:6825/29889] conn(0x7fd25c030c40 0x7fd25c0330f0 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7fd2640030c0 tx=0x7fd264014000 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-05T19:28:33.393 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.386+0000 7fd27ae44700 1 --2- 172.21.15.167:0/403903602 >> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] conn(0x56088e4fb7f0 0x56088e52e7e0 crc :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-05T19:28:33.393 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.386+0000 7fd27ae44700 1 -- 172.21.15.167:0/403903602 <== osd.1 v2:172.21.15.167:6800/30035 1 ==== osd_op_reply(1 default.realm [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 ==== 157+0+0 (crc 0 0 0) 0x7fd26c00a500 con 0x56088e4fb7f0 2023-10-05T19:28:33.394 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.386+0000 7fd27cb0ee40 20 rados_obj.operate() r=-2 bl.length=0 2023-10-05T19:28:33.394 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.386+0000 7fd27cb0ee40 20 realm 2023-10-05T19:28:33.394 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.386+0000 7fd27cb0ee40 20 rados->read ofs=0 len=0 2023-10-05T19:28:33.394 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.386+0000 7fd27cb0ee40 1 -- 172.21.15.167:0/403903602 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:2 5.12 5:49953fa1:::default.realm:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x56088e52be00 con 0x56088e4fb7f0 2023-10-05T19:28:33.395 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.390+0000 7fd27ae44700 1 -- 172.21.15.167:0/403903602 <== osd.1 v2:172.21.15.167:6800/30035 2 ==== osd_op_reply(2 default.realm [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 ==== 157+0+0 (crc 0 0 0) 0x7fd26c00a500 con 0x56088e4fb7f0 2023-10-05T19:28:33.395 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.390+0000 7fd27cb0ee40 20 rados_obj.operate() r=-2 bl.length=0 2023-10-05T19:28:33.395 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.390+0000 7fd27cb0ee40 4 RGWPeriod::init failed to init realm id : (2) No such file or directory 2023-10-05T19:28:33.395 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.390+0000 7fd27cb0ee40 20 rados->read ofs=0 len=0 2023-10-05T19:28:33.395 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.390+0000 7fd27cb0ee40 1 -- 172.21.15.167:0/403903602 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:3 5.12 5:49953fa1:::default.realm:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x56088e52f9c0 con 0x56088e4fb7f0 2023-10-05T19:28:33.395 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.390+0000 7fd27ae44700 1 -- 172.21.15.167:0/403903602 <== osd.1 v2:172.21.15.167:6800/30035 3 ==== osd_op_reply(3 default.realm [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 ==== 157+0+0 (crc 0 0 0) 0x7fd26c00a500 con 0x56088e4fb7f0 2023-10-05T19:28:33.395 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.390+0000 7fd27cb0ee40 20 rados_obj.operate() r=-2 bl.length=0 2023-10-05T19:28:33.395 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.390+0000 7fd27cb0ee40 20 rados->read ofs=0 len=0 2023-10-05T19:28:33.395 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.390+0000 7fd27cb0ee40 1 -- 172.21.15.167:0/403903602 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:4 5.f 5:f43b6ece:::zone_names.default:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x56088e52f5a0 con 0x56088e4fb7f0 2023-10-05T19:28:33.396 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.390+0000 7fd27ae44700 1 -- 172.21.15.167:0/403903602 <== osd.1 v2:172.21.15.167:6800/30035 4 ==== osd_op_reply(4 zone_names.default [read 0~46 out=46b] v0'0 uv1 ondisk = 0) v8 ==== 162+0+46 (crc 0 0 0) 0x7fd26c00a500 con 0x56088e4fb7f0 2023-10-05T19:28:33.397 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.390+0000 7fd27cb0ee40 20 rados_obj.operate() r=0 bl.length=46 2023-10-05T19:28:33.397 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.390+0000 7fd27cb0ee40 20 rados->read ofs=0 len=0 2023-10-05T19:28:33.397 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.390+0000 7fd27cb0ee40 1 -- 172.21.15.167:0/403903602 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:5 5.1d 5:b8374eb4:::zone_info.9a9df965-6612-45d4-a45f-6e3c889bacc2:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x56088e530560 con 0x56088e4fb7f0 2023-10-05T19:28:33.397 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.390+0000 7fd27ae44700 1 -- 172.21.15.167:0/403903602 <== osd.1 v2:172.21.15.167:6800/30035 5 ==== osd_op_reply(5 zone_info.9a9df965-6612-45d4-a45f-6e3c889bacc2 [read 0~874 out=874b] v0'0 uv2 ondisk = 0) v8 ==== 190+0+874 (crc 0 0 0) 0x7fd26c00a500 con 0x56088e4fb7f0 2023-10-05T19:28:33.397 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.390+0000 7fd27cb0ee40 20 rados_obj.operate() r=0 bl.length=874 2023-10-05T19:28:33.397 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.390+0000 7fd27cb0ee40 20 searching for the correct realm 2023-10-05T19:28:33.397 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.390+0000 7fd27cb0ee40 1 --2- 172.21.15.167:0/403903602 >> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] conn(0x56088e532470 0x56088e534880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-05T19:28:33.397 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.390+0000 7fd27cb0ee40 1 -- 172.21.15.167:0/403903602 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:6 5.0 5:00000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x56088e534e90 con 0x56088e532470 2023-10-05T19:28:33.397 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.390+0000 7fd27a643700 1 --2- 172.21.15.167:0/403903602 >> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] conn(0x56088e532470 0x56088e534880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-05T19:28:33.397 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.390+0000 7fd27a643700 1 --2- 172.21.15.167:0/403903602 >> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] conn(0x56088e532470 0x56088e534880 crc :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-05T19:28:33.399 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.394+0000 7fd27a643700 1 -- 172.21.15.167:0/403903602 <== osd.3 v2:172.21.15.167:6808/30036 1 ==== osd_op_reply(6 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fd274024020 con 0x56088e532470 2023-10-05T19:28:33.399 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.394+0000 7fd27a643700 1 --2- 172.21.15.167:0/403903602 >> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] conn(0x7fd274027be0 0x7fd27401d7b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-05T19:28:33.399 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.394+0000 7fd27a643700 1 -- 172.21.15.167:0/403903602 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:7 5.10 5:08000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x56088e534e90 con 0x7fd274027be0 2023-10-05T19:28:33.399 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.394+0000 7fd279e42700 1 --2- 172.21.15.167:0/403903602 >> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] conn(0x7fd274027be0 0x7fd27401d7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-05T19:28:33.399 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.394+0000 7fd279e42700 1 --2- 172.21.15.167:0/403903602 >> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] conn(0x7fd274027be0 0x7fd27401d7b0 crc :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-05T19:28:33.400 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.394+0000 7fd279e42700 1 -- 172.21.15.167:0/403903602 <== osd.2 v2:172.21.15.167:6816/30037 1 ==== osd_op_reply(7 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fd26401f480 con 0x7fd274027be0 2023-10-05T19:28:33.400 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.394+0000 7fd279e42700 1 --2- 172.21.15.167:0/403903602 >> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] conn(0x7fd264007030 0x7fd26401d940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-05T19:28:33.400 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.394+0000 7fd279e42700 1 -- 172.21.15.167:0/403903602 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:8 5.8 5:10000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x56088e534e90 con 0x7fd264007030 2023-10-05T19:28:33.400 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.394+0000 7fd27ae44700 1 --2- 172.21.15.167:0/403903602 >> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] conn(0x7fd264007030 0x7fd26401d940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-05T19:28:33.401 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.394+0000 7fd27ae44700 1 --2- 172.21.15.167:0/403903602 >> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] conn(0x7fd264007030 0x7fd26401d940 crc :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-05T19:28:33.402 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.398+0000 7fd27ae44700 1 -- 172.21.15.167:0/403903602 <== osd.0 v2:172.21.15.118:6800/27690 1 ==== osd_op_reply(8 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fd26c014040 con 0x7fd264007030 2023-10-05T19:28:33.402 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.398+0000 7fd27ae44700 1 -- 172.21.15.167:0/403903602 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:9 5.18 5:18000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x56088e534e90 con 0x56088e4fb7f0 2023-10-05T19:28:33.402 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.398+0000 7fd27ae44700 1 -- 172.21.15.167:0/403903602 <== osd.1 v2:172.21.15.167:6800/30035 6 ==== osd_op_reply(9 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fd26c014040 con 0x56088e4fb7f0 2023-10-05T19:28:33.402 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.398+0000 7fd27ae44700 1 -- 172.21.15.167:0/403903602 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:10 5.4 5:20000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x56088e534e90 con 0x56088e4fb7f0 2023-10-05T19:28:33.403 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.398+0000 7fd27ae44700 1 -- 172.21.15.167:0/403903602 <== osd.1 v2:172.21.15.167:6800/30035 7 ==== osd_op_reply(10 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fd26c014040 con 0x56088e4fb7f0 2023-10-05T19:28:33.403 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.398+0000 7fd27ae44700 1 -- 172.21.15.167:0/403903602 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:11 5.14 5:28000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x56088e534e90 con 0x7fd264007030 2023-10-05T19:28:33.403 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.398+0000 7fd27ae44700 1 -- 172.21.15.167:0/403903602 <== osd.0 v2:172.21.15.118:6800/27690 2 ==== osd_op_reply(11 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fd26c014040 con 0x7fd264007030 2023-10-05T19:28:33.404 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.398+0000 7fd27ae44700 1 -- 172.21.15.167:0/403903602 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:12 5.c 5:30000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x56088e534e90 con 0x7fd264007030 2023-10-05T19:28:33.404 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.398+0000 7fd27ae44700 1 -- 172.21.15.167:0/403903602 <== osd.0 v2:172.21.15.118:6800/27690 3 ==== osd_op_reply(12 [pgnls start_epoch 30 out=112b] v23'1 uv1 ondisk = 1) v8 ==== 144+0+112 (crc 0 0 0) 0x7fd26c014040 con 0x7fd264007030 2023-10-05T19:28:33.404 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.398+0000 7fd27ae44700 1 -- 172.21.15.167:0/403903602 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:13 5.1c 5:38000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x56088e534e90 con 0x7fd274027be0 2023-10-05T19:28:33.405 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.398+0000 7fd279e42700 1 -- 172.21.15.167:0/403903602 <== osd.2 v2:172.21.15.167:6816/30037 2 ==== osd_op_reply(13 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fd264029040 con 0x7fd274027be0 2023-10-05T19:28:33.405 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.398+0000 7fd279e42700 1 -- 172.21.15.167:0/403903602 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:14 5.2 5:40000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x56088e534e90 con 0x56088e532470 2023-10-05T19:28:33.405 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.398+0000 7fd27a643700 1 -- 172.21.15.167:0/403903602 <== osd.3 v2:172.21.15.167:6808/30036 2 ==== osd_op_reply(14 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fd274024020 con 0x56088e532470 2023-10-05T19:28:33.405 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.398+0000 7fd27a643700 1 -- 172.21.15.167:0/403903602 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:15 5.12 5:48000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x56088e534e90 con 0x56088e4fb7f0 2023-10-05T19:28:33.406 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.402+0000 7fd27ae44700 1 -- 172.21.15.167:0/403903602 <== osd.1 v2:172.21.15.167:6800/30035 8 ==== osd_op_reply(15 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fd26c014040 con 0x56088e4fb7f0 2023-10-05T19:28:33.406 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.402+0000 7fd27ae44700 1 -- 172.21.15.167:0/403903602 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:16 5.a 5:50000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x56088e534e90 con 0x7fd264007030 2023-10-05T19:28:33.407 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.402+0000 7fd27ae44700 1 -- 172.21.15.167:0/403903602 <== osd.0 v2:172.21.15.118:6800/27690 4 ==== osd_op_reply(16 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fd26c022040 con 0x7fd264007030 2023-10-05T19:28:33.407 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.402+0000 7fd27ae44700 1 -- 172.21.15.167:0/403903602 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:17 5.1a 5:58000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x56088e534e90 con 0x56088e4fb7f0 2023-10-05T19:28:33.408 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.402+0000 7fd27ae44700 1 -- 172.21.15.167:0/403903602 <== osd.1 v2:172.21.15.167:6800/30035 9 ==== osd_op_reply(17 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fd26c022040 con 0x56088e4fb7f0 2023-10-05T19:28:33.408 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.402+0000 7fd27ae44700 1 -- 172.21.15.167:0/403903602 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:18 5.6 5:60000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x56088e534e90 con 0x7fd274027be0 2023-10-05T19:28:33.408 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.402+0000 7fd279e42700 1 -- 172.21.15.167:0/403903602 <== osd.2 v2:172.21.15.167:6816/30037 3 ==== osd_op_reply(18 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fd264029040 con 0x7fd274027be0 2023-10-05T19:28:33.408 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.402+0000 7fd279e42700 1 -- 172.21.15.167:0/403903602 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:19 5.16 5:68000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x56088e534e90 con 0x56088e532470 2023-10-05T19:28:33.409 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.402+0000 7fd27a643700 1 -- 172.21.15.167:0/403903602 <== osd.3 v2:172.21.15.167:6808/30036 3 ==== osd_op_reply(19 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fd274024020 con 0x56088e532470 2023-10-05T19:28:33.409 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.402+0000 7fd27a643700 1 -- 172.21.15.167:0/403903602 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:20 5.e 5:70000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x56088e534e90 con 0x56088e532470 2023-10-05T19:28:33.410 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.402+0000 7fd27a643700 1 -- 172.21.15.167:0/403903602 <== osd.3 v2:172.21.15.167:6808/30036 4 ==== osd_op_reply(20 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fd274024020 con 0x56088e532470 2023-10-05T19:28:33.410 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.402+0000 7fd27a643700 1 -- 172.21.15.167:0/403903602 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:21 5.1e 5:78000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x56088e534e90 con 0x7fd264007030 2023-10-05T19:28:33.411 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.406+0000 7fd27ae44700 1 -- 172.21.15.167:0/403903602 <== osd.0 v2:172.21.15.118:6800/27690 5 ==== osd_op_reply(21 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fd26c022040 con 0x7fd264007030 2023-10-05T19:28:33.411 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.406+0000 7fd27ae44700 1 -- 172.21.15.167:0/403903602 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:22 5.1 5:80000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x56088e534e90 con 0x56088e532470 2023-10-05T19:28:33.411 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.406+0000 7fd27a643700 1 -- 172.21.15.167:0/403903602 <== osd.3 v2:172.21.15.167:6808/30036 5 ==== osd_op_reply(22 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fd274024020 con 0x56088e532470 2023-10-05T19:28:33.411 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.406+0000 7fd27a643700 1 -- 172.21.15.167:0/403903602 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:23 5.11 5:88000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x56088e534e90 con 0x56088e532470 2023-10-05T19:28:33.412 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.406+0000 7fd27a643700 1 -- 172.21.15.167:0/403903602 <== osd.3 v2:172.21.15.167:6808/30036 6 ==== osd_op_reply(23 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fd274024020 con 0x56088e532470 2023-10-05T19:28:33.412 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.406+0000 7fd27a643700 1 -- 172.21.15.167:0/403903602 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:24 5.9 5:90000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x56088e534e90 con 0x56088e4fb7f0 2023-10-05T19:28:33.413 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.406+0000 7fd27ae44700 1 -- 172.21.15.167:0/403903602 <== osd.1 v2:172.21.15.167:6800/30035 10 ==== osd_op_reply(24 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fd26c022040 con 0x56088e4fb7f0 2023-10-05T19:28:33.413 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.406+0000 7fd27ae44700 1 -- 172.21.15.167:0/403903602 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:25 5.19 5:98000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x56088e534e90 con 0x56088e4fb7f0 2023-10-05T19:28:33.413 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.406+0000 7fd27ae44700 1 -- 172.21.15.167:0/403903602 <== osd.1 v2:172.21.15.167:6800/30035 11 ==== osd_op_reply(25 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fd26c022040 con 0x56088e4fb7f0 2023-10-05T19:28:33.413 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.406+0000 7fd27ae44700 1 -- 172.21.15.167:0/403903602 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:26 5.5 5:a0000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x56088e534e90 con 0x7fd264007030 2023-10-05T19:28:33.414 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.410+0000 7fd27ae44700 1 -- 172.21.15.167:0/403903602 <== osd.0 v2:172.21.15.118:6800/27690 6 ==== osd_op_reply(26 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fd26c022040 con 0x7fd264007030 2023-10-05T19:28:33.414 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.410+0000 7fd27ae44700 1 -- 172.21.15.167:0/403903602 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:27 5.15 5:a8000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x56088e534e90 con 0x56088e532470 2023-10-05T19:28:33.415 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.410+0000 7fd27a643700 1 -- 172.21.15.167:0/403903602 <== osd.3 v2:172.21.15.167:6808/30036 7 ==== osd_op_reply(27 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fd274024020 con 0x56088e532470 2023-10-05T19:28:33.415 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.410+0000 7fd27a643700 1 -- 172.21.15.167:0/403903602 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:28 5.d 5:b0000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x56088e534e90 con 0x7fd274027be0 2023-10-05T19:28:33.416 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.410+0000 7fd279e42700 1 -- 172.21.15.167:0/403903602 <== osd.2 v2:172.21.15.167:6816/30037 4 ==== osd_op_reply(28 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fd264029040 con 0x7fd274027be0 2023-10-05T19:28:33.416 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.410+0000 7fd279e42700 1 -- 172.21.15.167:0/403903602 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:29 5.1d 5:b8000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x56088e534e90 con 0x56088e4fb7f0 2023-10-05T19:28:33.417 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.410+0000 7fd27ae44700 1 -- 172.21.15.167:0/403903602 <== osd.1 v2:172.21.15.167:6800/30035 12 ==== osd_op_reply(29 [pgnls start_epoch 30 out=107b] v30'2 uv2 ondisk = 1) v8 ==== 144+0+107 (crc 0 0 0) 0x7fd26c022040 con 0x56088e4fb7f0 2023-10-05T19:28:33.417 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.410+0000 7fd27ae44700 1 -- 172.21.15.167:0/403903602 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:30 5.3 5:c0000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x56088e534e90 con 0x7fd264007030 2023-10-05T19:28:33.418 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.410+0000 7fd27ae44700 1 -- 172.21.15.167:0/403903602 <== osd.0 v2:172.21.15.118:6800/27690 7 ==== osd_op_reply(30 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fd26c022040 con 0x7fd264007030 2023-10-05T19:28:33.418 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.410+0000 7fd27ae44700 1 -- 172.21.15.167:0/403903602 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:31 5.13 5:c8000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x56088e534e90 con 0x56088e532470 2023-10-05T19:28:33.418 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.414+0000 7fd27a643700 1 -- 172.21.15.167:0/403903602 <== osd.3 v2:172.21.15.167:6808/30036 8 ==== osd_op_reply(31 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fd274024020 con 0x56088e532470 2023-10-05T19:28:33.418 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.414+0000 7fd27a643700 1 -- 172.21.15.167:0/403903602 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:32 5.b 5:d0000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x56088e534e90 con 0x7fd264007030 2023-10-05T19:28:33.419 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.414+0000 7fd27ae44700 1 -- 172.21.15.167:0/403903602 <== osd.0 v2:172.21.15.118:6800/27690 8 ==== osd_op_reply(32 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fd26c022040 con 0x7fd264007030 2023-10-05T19:28:33.419 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.414+0000 7fd27ae44700 1 -- 172.21.15.167:0/403903602 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:33 5.1b 5:d8000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x56088e534e90 con 0x7fd274027be0 2023-10-05T19:28:33.420 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.414+0000 7fd279e42700 1 -- 172.21.15.167:0/403903602 <== osd.2 v2:172.21.15.167:6816/30037 5 ==== osd_op_reply(33 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fd264029040 con 0x7fd274027be0 2023-10-05T19:28:33.420 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.414+0000 7fd279e42700 1 -- 172.21.15.167:0/403903602 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:34 5.7 5:e0000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x56088e534e90 con 0x56088e532470 2023-10-05T19:28:33.421 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.414+0000 7fd27a643700 1 -- 172.21.15.167:0/403903602 <== osd.3 v2:172.21.15.167:6808/30036 9 ==== osd_op_reply(34 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fd274024020 con 0x56088e532470 2023-10-05T19:28:33.421 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.414+0000 7fd27a643700 1 -- 172.21.15.167:0/403903602 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:35 5.17 5:e8000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x56088e534e90 con 0x7fd264007030 2023-10-05T19:28:33.422 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.414+0000 7fd27ae44700 1 -- 172.21.15.167:0/403903602 <== osd.0 v2:172.21.15.118:6800/27690 9 ==== osd_op_reply(35 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fd26c022040 con 0x7fd264007030 2023-10-05T19:28:33.422 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.414+0000 7fd27ae44700 1 -- 172.21.15.167:0/403903602 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:36 5.f 5:f0000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x56088e534e90 con 0x56088e4fb7f0 2023-10-05T19:28:33.422 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.418+0000 7fd27ae44700 1 -- 172.21.15.167:0/403903602 <== osd.1 v2:172.21.15.167:6800/30035 13 ==== osd_op_reply(36 [pgnls start_epoch 30 out=115b] v23'2 uv2 ondisk = 1) v8 ==== 144+0+115 (crc 0 0 0) 0x7fd26c02f080 con 0x56088e4fb7f0 2023-10-05T19:28:33.423 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.418+0000 7fd27ae44700 1 -- 172.21.15.167:0/403903602 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:37 5.1f 5:f8000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x56088e534e90 con 0x7fd274027be0 2023-10-05T19:28:33.423 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.418+0000 7fd279e42700 1 -- 172.21.15.167:0/403903602 <== osd.2 v2:172.21.15.167:6816/30037 6 ==== osd_op_reply(37 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fd264029040 con 0x7fd274027be0 2023-10-05T19:28:33.424 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.418+0000 7fd27cb0ee40 20 RGWRados::pool_iterate: got zonegroup_info.4f1f719d-9648-417b-b10c-e5d939825419 2023-10-05T19:28:33.424 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.418+0000 7fd27cb0ee40 20 RGWRados::pool_iterate: got zone_info.9a9df965-6612-45d4-a45f-6e3c889bacc2 2023-10-05T19:28:33.424 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.418+0000 7fd27cb0ee40 20 RGWRados::pool_iterate: got zone_names.default 2023-10-05T19:28:33.424 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.418+0000 7fd27cb0ee40 20 RGWRados::pool_iterate: got zonegroups_names.default 2023-10-05T19:28:33.424 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.418+0000 7fd27cb0ee40 20 rados->read ofs=0 len=0 2023-10-05T19:28:33.424 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.418+0000 7fd27cb0ee40 1 -- 172.21.15.167:0/403903602 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:38 5.12 5:49953fa1:::default.realm:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x56088e530f20 con 0x56088e4fb7f0 2023-10-05T19:28:33.425 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.418+0000 7fd27ae44700 1 -- 172.21.15.167:0/403903602 <== osd.1 v2:172.21.15.167:6800/30035 14 ==== osd_op_reply(38 default.realm [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 ==== 157+0+0 (crc 0 0 0) 0x7fd26c02f080 con 0x56088e4fb7f0 2023-10-05T19:28:33.425 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.418+0000 7fd27cb0ee40 20 rados_obj.operate() r=-2 bl.length=0 2023-10-05T19:28:33.425 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.418+0000 7fd27cb0ee40 20 rados->read ofs=0 len=0 2023-10-05T19:28:33.425 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.418+0000 7fd27cb0ee40 1 -- 172.21.15.167:0/403903602 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:39 5.f 5:f4c53578:::zonegroups_names.default:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x56088e531320 con 0x56088e4fb7f0 2023-10-05T19:28:33.425 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.418+0000 7fd27ae44700 1 -- 172.21.15.167:0/403903602 <== osd.1 v2:172.21.15.167:6800/30035 15 ==== osd_op_reply(39 zonegroups_names.default [read 0~46 out=46b] v0'0 uv2 ondisk = 0) v8 ==== 168+0+46 (crc 0 0 0) 0x7fd26c02f080 con 0x56088e4fb7f0 2023-10-05T19:28:33.426 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.418+0000 7fd27cb0ee40 20 rados_obj.operate() r=0 bl.length=46 2023-10-05T19:28:33.426 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.418+0000 7fd27cb0ee40 20 rados->read ofs=0 len=0 2023-10-05T19:28:33.426 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.418+0000 7fd27cb0ee40 1 -- 172.21.15.167:0/403903602 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:40 5.c 5:30d299a5:::zonegroup_info.4f1f719d-9648-417b-b10c-e5d939825419:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x56088e535300 con 0x7fd264007030 2023-10-05T19:28:33.427 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.418+0000 7fd27ae44700 1 -- 172.21.15.167:0/403903602 <== osd.0 v2:172.21.15.118:6800/27690 10 ==== osd_op_reply(40 zonegroup_info.4f1f719d-9648-417b-b10c-e5d939825419 [read 0~362 out=362b] v0'0 uv1 ondisk = 0) v8 ==== 195+0+362 (crc 0 0 0) 0x7fd26c02f080 con 0x7fd264007030 2023-10-05T19:28:33.427 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.418+0000 7fd27cb0ee40 20 rados_obj.operate() r=0 bl.length=362 2023-10-05T19:28:33.427 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.418+0000 7fd27cb0ee40 20 rados->read ofs=0 len=0 2023-10-05T19:28:33.427 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.418+0000 7fd27cb0ee40 1 -- 172.21.15.167:0/403903602 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:41 5.13 5:cbbf6c60:::converted:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x56088e535300 con 0x56088e532470 2023-10-05T19:28:33.427 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.422+0000 7fd27a643700 1 -- 172.21.15.167:0/403903602 <== osd.3 v2:172.21.15.167:6808/30036 10 ==== osd_op_reply(41 converted [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 ==== 153+0+0 (crc 0 0 0) 0x7fd274024020 con 0x56088e532470 2023-10-05T19:28:33.427 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.422+0000 7fd27cb0ee40 20 rados_obj.operate() r=-2 bl.length=0 2023-10-05T19:28:33.427 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.422+0000 7fd27cb0ee40 20 rados->read ofs=0 len=0 2023-10-05T19:28:33.427 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.422+0000 7fd27cb0ee40 1 -- 172.21.15.167:0/403903602 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:42 5.12 5:49953fa1:::default.realm:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x56088e5356b0 con 0x56088e4fb7f0 2023-10-05T19:28:33.428 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.422+0000 7fd27ae44700 1 -- 172.21.15.167:0/403903602 <== osd.1 v2:172.21.15.167:6800/30035 16 ==== osd_op_reply(42 default.realm [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 ==== 157+0+0 (crc 0 0 0) 0x7fd26c02f080 con 0x56088e4fb7f0 2023-10-05T19:28:33.428 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.422+0000 7fd27cb0ee40 20 rados_obj.operate() r=-2 bl.length=0 2023-10-05T19:28:33.428 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.422+0000 7fd27cb0ee40 20 rados->read ofs=0 len=0 2023-10-05T19:28:33.428 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.422+0000 7fd27cb0ee40 1 -- 172.21.15.167:0/403903602 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:43 5.f 5:f4c53578:::zonegroups_names.default:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x56088e530af0 con 0x56088e4fb7f0 2023-10-05T19:28:33.428 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.422+0000 7fd27ae44700 1 -- 172.21.15.167:0/403903602 <== osd.1 v2:172.21.15.167:6800/30035 17 ==== osd_op_reply(43 zonegroups_names.default [read 0~46 out=46b] v0'0 uv2 ondisk = 0) v8 ==== 168+0+46 (crc 0 0 0) 0x7fd26c02f080 con 0x56088e4fb7f0 2023-10-05T19:28:33.428 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.422+0000 7fd27cb0ee40 20 rados_obj.operate() r=0 bl.length=46 2023-10-05T19:28:33.429 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.422+0000 7fd27cb0ee40 1 -- 172.21.15.167:0/403903602 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:44 5.0 5:00000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x56088e530ac0 con 0x56088e532470 2023-10-05T19:28:33.430 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.426+0000 7fd27a643700 1 -- 172.21.15.167:0/403903602 <== osd.3 v2:172.21.15.167:6808/30036 11 ==== osd_op_reply(44 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fd274024020 con 0x56088e532470 2023-10-05T19:28:33.430 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.426+0000 7fd27a643700 1 -- 172.21.15.167:0/403903602 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:45 5.10 5:08000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x56088e530ac0 con 0x7fd274027be0 2023-10-05T19:28:33.430 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.426+0000 7fd279e42700 1 -- 172.21.15.167:0/403903602 <== osd.2 v2:172.21.15.167:6816/30037 7 ==== osd_op_reply(45 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fd264029040 con 0x7fd274027be0 2023-10-05T19:28:33.431 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.426+0000 7fd279e42700 1 -- 172.21.15.167:0/403903602 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:46 5.8 5:10000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x56088e530ac0 con 0x7fd264007030 2023-10-05T19:28:33.431 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.426+0000 7fd27ae44700 1 -- 172.21.15.167:0/403903602 <== osd.0 v2:172.21.15.118:6800/27690 11 ==== osd_op_reply(46 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fd26c02f080 con 0x7fd264007030 2023-10-05T19:28:33.432 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.426+0000 7fd27ae44700 1 -- 172.21.15.167:0/403903602 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:47 5.18 5:18000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x56088e531320 con 0x56088e4fb7f0 2023-10-05T19:28:33.432 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.426+0000 7fd27ae44700 1 -- 172.21.15.167:0/403903602 <== osd.1 v2:172.21.15.167:6800/30035 18 ==== osd_op_reply(47 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fd26c02f080 con 0x56088e4fb7f0 2023-10-05T19:28:33.433 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.426+0000 7fd27ae44700 1 -- 172.21.15.167:0/403903602 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:48 5.4 5:20000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x56088e531320 con 0x56088e4fb7f0 2023-10-05T19:28:33.433 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.426+0000 7fd27ae44700 1 -- 172.21.15.167:0/403903602 <== osd.1 v2:172.21.15.167:6800/30035 19 ==== osd_op_reply(48 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fd26c02f080 con 0x56088e4fb7f0 2023-10-05T19:28:33.433 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.426+0000 7fd27ae44700 1 -- 172.21.15.167:0/403903602 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:49 5.14 5:28000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x56088e531320 con 0x7fd264007030 2023-10-05T19:28:33.434 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.430+0000 7fd27ae44700 1 -- 172.21.15.167:0/403903602 <== osd.0 v2:172.21.15.118:6800/27690 12 ==== osd_op_reply(49 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fd26c02f080 con 0x7fd264007030 2023-10-05T19:28:33.434 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.430+0000 7fd27ae44700 1 -- 172.21.15.167:0/403903602 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:50 5.c 5:30000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x56088e531320 con 0x7fd264007030 2023-10-05T19:28:33.435 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.430+0000 7fd27ae44700 1 -- 172.21.15.167:0/403903602 <== osd.0 v2:172.21.15.118:6800/27690 13 ==== osd_op_reply(50 [pgnls start_epoch 30 out=112b] v23'1 uv1 ondisk = 1) v8 ==== 144+0+112 (crc 0 0 0) 0x7fd26c02f080 con 0x7fd264007030 2023-10-05T19:28:33.435 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.430+0000 7fd27ae44700 1 -- 172.21.15.167:0/403903602 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:51 5.1c 5:38000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x56088e531320 con 0x7fd274027be0 2023-10-05T19:28:33.436 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.430+0000 7fd279e42700 1 -- 172.21.15.167:0/403903602 <== osd.2 v2:172.21.15.167:6816/30037 8 ==== osd_op_reply(51 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fd264029040 con 0x7fd274027be0 2023-10-05T19:28:33.436 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.430+0000 7fd279e42700 1 -- 172.21.15.167:0/403903602 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:52 5.2 5:40000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x56088e531320 con 0x56088e532470 2023-10-05T19:28:33.437 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.430+0000 7fd27a643700 1 -- 172.21.15.167:0/403903602 <== osd.3 v2:172.21.15.167:6808/30036 12 ==== osd_op_reply(52 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fd274024020 con 0x56088e532470 2023-10-05T19:28:33.437 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.430+0000 7fd27a643700 1 -- 172.21.15.167:0/403903602 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:53 5.12 5:48000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x56088e531320 con 0x56088e4fb7f0 2023-10-05T19:28:33.437 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.430+0000 7fd27ae44700 1 -- 172.21.15.167:0/403903602 <== osd.1 v2:172.21.15.167:6800/30035 20 ==== osd_op_reply(53 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fd26c02f080 con 0x56088e4fb7f0 2023-10-05T19:28:33.437 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.430+0000 7fd27ae44700 1 -- 172.21.15.167:0/403903602 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:54 5.a 5:50000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x56088e531320 con 0x7fd264007030 2023-10-05T19:28:33.438 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.434+0000 7fd27ae44700 1 -- 172.21.15.167:0/403903602 <== osd.0 v2:172.21.15.118:6800/27690 14 ==== osd_op_reply(54 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fd26c02f080 con 0x7fd264007030 2023-10-05T19:28:33.439 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.434+0000 7fd27ae44700 1 -- 172.21.15.167:0/403903602 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:55 5.1a 5:58000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x56088e531320 con 0x56088e4fb7f0 2023-10-05T19:28:33.439 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.434+0000 7fd27ae44700 1 -- 172.21.15.167:0/403903602 <== osd.1 v2:172.21.15.167:6800/30035 21 ==== osd_op_reply(55 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fd26c02f080 con 0x56088e4fb7f0 2023-10-05T19:28:33.439 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.434+0000 7fd27ae44700 1 -- 172.21.15.167:0/403903602 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:56 5.6 5:60000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x56088e531320 con 0x7fd274027be0 2023-10-05T19:28:33.440 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.434+0000 7fd279e42700 1 -- 172.21.15.167:0/403903602 <== osd.2 v2:172.21.15.167:6816/30037 9 ==== osd_op_reply(56 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fd264029040 con 0x7fd274027be0 2023-10-05T19:28:33.440 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.434+0000 7fd279e42700 1 -- 172.21.15.167:0/403903602 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:57 5.16 5:68000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x56088e531320 con 0x56088e532470 2023-10-05T19:28:33.441 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.434+0000 7fd27a643700 1 -- 172.21.15.167:0/403903602 <== osd.3 v2:172.21.15.167:6808/30036 13 ==== osd_op_reply(57 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fd274024020 con 0x56088e532470 2023-10-05T19:28:33.441 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.434+0000 7fd27a643700 1 -- 172.21.15.167:0/403903602 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:58 5.e 5:70000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x56088e531320 con 0x56088e532470 2023-10-05T19:28:33.441 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.434+0000 7fd27a643700 1 -- 172.21.15.167:0/403903602 <== osd.3 v2:172.21.15.167:6808/30036 14 ==== osd_op_reply(58 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fd274024020 con 0x56088e532470 2023-10-05T19:28:33.442 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.434+0000 7fd27a643700 1 -- 172.21.15.167:0/403903602 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:59 5.1e 5:78000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x56088e531320 con 0x7fd264007030 2023-10-05T19:28:33.443 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.438+0000 7fd27ae44700 1 -- 172.21.15.167:0/403903602 <== osd.0 v2:172.21.15.118:6800/27690 15 ==== osd_op_reply(59 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fd26c02f080 con 0x7fd264007030 2023-10-05T19:28:33.443 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.438+0000 7fd27ae44700 1 -- 172.21.15.167:0/403903602 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:60 5.1 5:80000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x56088e531320 con 0x56088e532470 2023-10-05T19:28:33.443 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.438+0000 7fd27a643700 1 -- 172.21.15.167:0/403903602 <== osd.3 v2:172.21.15.167:6808/30036 15 ==== osd_op_reply(60 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fd274024020 con 0x56088e532470 2023-10-05T19:28:33.443 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.438+0000 7fd27a643700 1 -- 172.21.15.167:0/403903602 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:61 5.11 5:88000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x56088e531320 con 0x56088e532470 2023-10-05T19:28:33.444 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.438+0000 7fd27a643700 1 -- 172.21.15.167:0/403903602 <== osd.3 v2:172.21.15.167:6808/30036 16 ==== osd_op_reply(61 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fd274024020 con 0x56088e532470 2023-10-05T19:28:33.444 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.438+0000 7fd27a643700 1 -- 172.21.15.167:0/403903602 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:62 5.9 5:90000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x56088e531320 con 0x56088e4fb7f0 2023-10-05T19:28:33.445 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.438+0000 7fd27ae44700 1 -- 172.21.15.167:0/403903602 <== osd.1 v2:172.21.15.167:6800/30035 22 ==== osd_op_reply(62 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fd26c02f080 con 0x56088e4fb7f0 2023-10-05T19:28:33.445 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.438+0000 7fd27ae44700 1 -- 172.21.15.167:0/403903602 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:63 5.19 5:98000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x56088e531320 con 0x56088e4fb7f0 2023-10-05T19:28:33.446 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.442+0000 7fd27ae44700 1 -- 172.21.15.167:0/403903602 <== osd.1 v2:172.21.15.167:6800/30035 23 ==== osd_op_reply(63 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fd26c02f080 con 0x56088e4fb7f0 2023-10-05T19:28:33.446 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.442+0000 7fd27ae44700 1 -- 172.21.15.167:0/403903602 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:64 5.5 5:a0000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x56088e531320 con 0x7fd264007030 2023-10-05T19:28:33.447 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.442+0000 7fd27ae44700 1 -- 172.21.15.167:0/403903602 <== osd.0 v2:172.21.15.118:6800/27690 16 ==== osd_op_reply(64 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fd26c02f080 con 0x7fd264007030 2023-10-05T19:28:33.447 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.442+0000 7fd27ae44700 1 -- 172.21.15.167:0/403903602 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:65 5.15 5:a8000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x56088e531320 con 0x56088e532470 2023-10-05T19:28:33.447 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.442+0000 7fd27a643700 1 -- 172.21.15.167:0/403903602 <== osd.3 v2:172.21.15.167:6808/30036 17 ==== osd_op_reply(65 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fd274024020 con 0x56088e532470 2023-10-05T19:28:33.448 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.442+0000 7fd27a643700 1 -- 172.21.15.167:0/403903602 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:66 5.d 5:b0000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x56088e531320 con 0x7fd274027be0 2023-10-05T19:28:33.448 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.442+0000 7fd279e42700 1 -- 172.21.15.167:0/403903602 <== osd.2 v2:172.21.15.167:6816/30037 10 ==== osd_op_reply(66 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fd264029040 con 0x7fd274027be0 2023-10-05T19:28:33.448 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.442+0000 7fd279e42700 1 -- 172.21.15.167:0/403903602 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:67 5.1d 5:b8000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x56088e531320 con 0x56088e4fb7f0 2023-10-05T19:28:33.449 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.442+0000 7fd27ae44700 1 -- 172.21.15.167:0/403903602 <== osd.1 v2:172.21.15.167:6800/30035 24 ==== osd_op_reply(67 [pgnls start_epoch 30 out=107b] v30'2 uv2 ondisk = 1) v8 ==== 144+0+107 (crc 0 0 0) 0x7fd26c02f080 con 0x56088e4fb7f0 2023-10-05T19:28:33.449 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.442+0000 7fd27ae44700 1 -- 172.21.15.167:0/403903602 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:68 5.3 5:c0000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x56088e531320 con 0x7fd264007030 2023-10-05T19:28:33.450 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.446+0000 7fd27ae44700 1 -- 172.21.15.167:0/403903602 <== osd.0 v2:172.21.15.118:6800/27690 17 ==== osd_op_reply(68 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fd26c02f080 con 0x7fd264007030 2023-10-05T19:28:33.450 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.446+0000 7fd27ae44700 1 -- 172.21.15.167:0/403903602 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:69 5.13 5:c8000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x56088e531320 con 0x56088e532470 2023-10-05T19:28:33.451 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.446+0000 7fd27a643700 1 -- 172.21.15.167:0/403903602 <== osd.3 v2:172.21.15.167:6808/30036 18 ==== osd_op_reply(69 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fd274024020 con 0x56088e532470 2023-10-05T19:28:33.451 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.446+0000 7fd27a643700 1 -- 172.21.15.167:0/403903602 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:70 5.b 5:d0000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x56088e531320 con 0x7fd264007030 2023-10-05T19:28:33.452 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.446+0000 7fd27ae44700 1 -- 172.21.15.167:0/403903602 <== osd.0 v2:172.21.15.118:6800/27690 18 ==== osd_op_reply(70 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fd26c02f080 con 0x7fd264007030 2023-10-05T19:28:33.452 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.446+0000 7fd27ae44700 1 -- 172.21.15.167:0/403903602 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:71 5.1b 5:d8000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x56088e531320 con 0x7fd274027be0 2023-10-05T19:28:33.453 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.446+0000 7fd279e42700 1 -- 172.21.15.167:0/403903602 <== osd.2 v2:172.21.15.167:6816/30037 11 ==== osd_op_reply(71 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fd264029040 con 0x7fd274027be0 2023-10-05T19:28:33.453 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.446+0000 7fd279e42700 1 -- 172.21.15.167:0/403903602 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:72 5.7 5:e0000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x56088e531320 con 0x56088e532470 2023-10-05T19:28:33.454 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.450+0000 7fd27a643700 1 -- 172.21.15.167:0/403903602 <== osd.3 v2:172.21.15.167:6808/30036 19 ==== osd_op_reply(72 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fd274024020 con 0x56088e532470 2023-10-05T19:28:33.454 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.450+0000 7fd27a643700 1 -- 172.21.15.167:0/403903602 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:73 5.17 5:e8000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x56088e531320 con 0x7fd264007030 2023-10-05T19:28:33.455 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.450+0000 7fd27ae44700 1 -- 172.21.15.167:0/403903602 <== osd.0 v2:172.21.15.118:6800/27690 19 ==== osd_op_reply(73 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fd26c02f080 con 0x7fd264007030 2023-10-05T19:28:33.455 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.450+0000 7fd27ae44700 1 -- 172.21.15.167:0/403903602 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:74 5.f 5:f0000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x56088e531320 con 0x56088e4fb7f0 2023-10-05T19:28:33.456 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.450+0000 7fd27ae44700 1 -- 172.21.15.167:0/403903602 <== osd.1 v2:172.21.15.167:6800/30035 25 ==== osd_op_reply(74 [pgnls start_epoch 30 out=115b] v23'2 uv2 ondisk = 1) v8 ==== 144+0+115 (crc 0 0 0) 0x7fd26c02f080 con 0x56088e4fb7f0 2023-10-05T19:28:33.456 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.450+0000 7fd27ae44700 1 -- 172.21.15.167:0/403903602 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:75 5.1f 5:f8000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x56088e531320 con 0x7fd274027be0 2023-10-05T19:28:33.457 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.450+0000 7fd279e42700 1 -- 172.21.15.167:0/403903602 <== osd.2 v2:172.21.15.167:6816/30037 12 ==== osd_op_reply(75 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fd264029040 con 0x7fd274027be0 2023-10-05T19:28:33.457 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.450+0000 7fd27cb0ee40 20 RGWRados::pool_iterate: got zonegroup_info.4f1f719d-9648-417b-b10c-e5d939825419 2023-10-05T19:28:33.457 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.450+0000 7fd27cb0ee40 20 RGWRados::pool_iterate: got zone_info.9a9df965-6612-45d4-a45f-6e3c889bacc2 2023-10-05T19:28:33.457 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.450+0000 7fd27cb0ee40 20 RGWRados::pool_iterate: got zone_names.default 2023-10-05T19:28:33.458 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.450+0000 7fd27cb0ee40 20 RGWRados::pool_iterate: got zonegroups_names.default 2023-10-05T19:28:33.458 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.450+0000 7fd27cb0ee40 20 rados->read ofs=0 len=0 2023-10-05T19:28:33.458 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.450+0000 7fd27cb0ee40 1 -- 172.21.15.167:0/403903602 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:76 5.f 5:f43b6ece:::zone_names.default:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x56088e530ac0 con 0x56088e4fb7f0 2023-10-05T19:28:33.459 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.454+0000 7fd27ae44700 1 -- 172.21.15.167:0/403903602 <== osd.1 v2:172.21.15.167:6800/30035 26 ==== osd_op_reply(76 zone_names.default [read 0~46 out=46b] v0'0 uv1 ondisk = 0) v8 ==== 162+0+46 (crc 0 0 0) 0x7fd26c02f080 con 0x56088e4fb7f0 2023-10-05T19:28:33.459 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.454+0000 7fd27cb0ee40 20 rados_obj.operate() r=0 bl.length=46 2023-10-05T19:28:33.459 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.454+0000 7fd27cb0ee40 20 rados->read ofs=0 len=0 2023-10-05T19:28:33.459 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.454+0000 7fd27cb0ee40 1 -- 172.21.15.167:0/403903602 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:77 5.1d 5:b8374eb4:::zone_info.9a9df965-6612-45d4-a45f-6e3c889bacc2:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x56088e535ba0 con 0x56088e4fb7f0 2023-10-05T19:28:33.459 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.454+0000 7fd27ae44700 1 -- 172.21.15.167:0/403903602 <== osd.1 v2:172.21.15.167:6800/30035 27 ==== osd_op_reply(77 zone_info.9a9df965-6612-45d4-a45f-6e3c889bacc2 [read 0~874 out=874b] v0'0 uv2 ondisk = 0) v8 ==== 190+0+874 (crc 0 0 0) 0x7fd26c02f080 con 0x56088e4fb7f0 2023-10-05T19:28:33.459 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.454+0000 7fd27cb0ee40 20 rados_obj.operate() r=0 bl.length=874 2023-10-05T19:28:33.459 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.454+0000 7fd27cb0ee40 20 rados->read ofs=0 len=0 2023-10-05T19:28:33.460 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.454+0000 7fd27cb0ee40 1 -- 172.21.15.167:0/403903602 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:78 5.f 5:f4c53578:::zonegroups_names.default:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x56088e535db0 con 0x56088e4fb7f0 2023-10-05T19:28:33.460 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.454+0000 7fd27ae44700 1 -- 172.21.15.167:0/403903602 <== osd.1 v2:172.21.15.167:6800/30035 28 ==== osd_op_reply(78 zonegroups_names.default [read 0~46 out=46b] v0'0 uv2 ondisk = 0) v8 ==== 168+0+46 (crc 0 0 0) 0x7fd26c02f080 con 0x56088e4fb7f0 2023-10-05T19:28:33.460 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.454+0000 7fd27cb0ee40 20 rados_obj.operate() r=0 bl.length=46 2023-10-05T19:28:33.460 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.454+0000 7fd27cb0ee40 20 rados->read ofs=0 len=0 2023-10-05T19:28:33.460 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.454+0000 7fd27cb0ee40 1 -- 172.21.15.167:0/403903602 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:79 5.c 5:30d299a5:::zonegroup_info.4f1f719d-9648-417b-b10c-e5d939825419:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x56088e535dd0 con 0x7fd264007030 2023-10-05T19:28:33.461 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.454+0000 7fd27ae44700 1 -- 172.21.15.167:0/403903602 <== osd.0 v2:172.21.15.118:6800/27690 20 ==== osd_op_reply(79 zonegroup_info.4f1f719d-9648-417b-b10c-e5d939825419 [read 0~362 out=362b] v0'0 uv1 ondisk = 0) v8 ==== 195+0+362 (crc 0 0 0) 0x7fd26c02f080 con 0x7fd264007030 2023-10-05T19:28:33.461 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.454+0000 7fd27cb0ee40 20 rados_obj.operate() r=0 bl.length=362 2023-10-05T19:28:33.461 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.454+0000 7fd27cb0ee40 20 rados->read ofs=0 len=0 2023-10-05T19:28:33.461 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.454+0000 7fd27cb0ee40 1 -- 172.21.15.167:0/403903602 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:80 5.3 5:c0ad8cbd:::region_map:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x56088e536760 con 0x7fd264007030 2023-10-05T19:28:33.462 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.454+0000 7fd27ae44700 1 -- 172.21.15.167:0/403903602 <== osd.0 v2:172.21.15.118:6800/27690 21 ==== osd_op_reply(80 region_map [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 ==== 154+0+0 (crc 0 0 0) 0x7fd26c050040 con 0x7fd264007030 2023-10-05T19:28:33.462 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.454+0000 7fd27cb0ee40 20 rados_obj.operate() r=-2 bl.length=0 2023-10-05T19:28:33.462 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.454+0000 7fd27cb0ee40 20 zone default found 2023-10-05T19:28:33.462 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.454+0000 7fd27cb0ee40 4 Realm: () 2023-10-05T19:28:33.462 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.454+0000 7fd27cb0ee40 4 ZoneGroup: default (4f1f719d-9648-417b-b10c-e5d939825419) 2023-10-05T19:28:33.462 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.454+0000 7fd27cb0ee40 4 Zone: default (9a9df965-6612-45d4-a45f-6e3c889bacc2) 2023-10-05T19:28:33.462 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.454+0000 7fd27cb0ee40 10 cannot find current period zonegroup using local zonegroup configuration 2023-10-05T19:28:33.462 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.454+0000 7fd27cb0ee40 20 zonegroup default 2023-10-05T19:28:33.462 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.454+0000 7fd27cb0ee40 20 rados->read ofs=0 len=0 2023-10-05T19:28:33.463 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.454+0000 7fd27cb0ee40 1 -- 172.21.15.167:0/403903602 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:81 5.3 5:c52100b6:::period_config.default:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x56088e536760 con 0x7fd264007030 2023-10-05T19:28:33.466 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.458+0000 7fd27ae44700 1 -- 172.21.15.167:0/403903602 <== osd.0 v2:172.21.15.118:6800/27690 22 ==== osd_op_reply(81 period_config.default [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 ==== 165+0+0 (crc 0 0 0) 0x7fd26c050040 con 0x7fd264007030 2023-10-05T19:28:33.466 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.458+0000 7fd27cb0ee40 20 rados_obj.operate() r=-2 bl.length=0 2023-10-05T19:28:33.466 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.458+0000 7fd27cb0ee40 20 rados->read ofs=0 len=0 2023-10-05T19:28:33.466 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.458+0000 7fd27cb0ee40 1 -- 172.21.15.167:0/403903602 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:82 6.2 6:4347d321:::bucket.sync-source-hints.:head [call version.read in=11b,read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x56088e538000 con 0x56088e532470 2023-10-05T19:28:33.466 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.458+0000 7fd27a643700 1 -- 172.21.15.167:0/403903602 <== osd.3 v2:172.21.15.167:6808/30036 20 ==== osd_op_reply(82 bucket.sync-source-hints. [call,read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 ==== 211+0+0 (crc 0 0 0) 0x7fd274024020 con 0x56088e532470 2023-10-05T19:28:33.467 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.458+0000 7fd27cb0ee40 20 rados_obj.operate() r=-2 bl.length=0 2023-10-05T19:28:33.467 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.458+0000 7fd27cb0ee40 20 rados->read ofs=0 len=0 2023-10-05T19:28:33.467 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.458+0000 7fd27cb0ee40 1 -- 172.21.15.167:0/403903602 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:83 6.b 6:d467b91b:::bucket.sync-target-hints.:head [call version.read in=11b,read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x56088e5383b0 con 0x56088e532470 2023-10-05T19:28:33.467 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.458+0000 7fd27a643700 1 -- 172.21.15.167:0/403903602 <== osd.3 v2:172.21.15.167:6808/30036 21 ==== osd_op_reply(83 bucket.sync-target-hints. [call,read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 ==== 211+0+0 (crc 0 0 0) 0x7fd274024020 con 0x56088e532470 2023-10-05T19:28:33.467 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.458+0000 7fd27cb0ee40 20 rados_obj.operate() r=-2 bl.length=0 2023-10-05T19:28:33.467 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.458+0000 7fd27cb0ee40 20 started sync module instance, tier type = 2023-10-05T19:28:33.467 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.458+0000 7fd27cb0ee40 20 started zone id=9a9df965-6612-45d4-a45f-6e3c889bacc2 (name=default) with tier type = 2023-10-05T19:28:33.467 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.458+0000 7fd27cb0ee40 1 -- 172.21.15.167:0/403903602 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:84 7.1f 7:f95f44c2:::notify.0:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x56088e5376c0 con 0x7fd274027be0 2023-10-05T19:28:33.468 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.462+0000 7fd279e42700 1 -- 172.21.15.167:0/403903602 <== osd.2 v2:172.21.15.167:6816/30037 13 ==== osd_op_reply(84 notify.0 [create] v30'120 uv120 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7fd264029040 con 0x7fd274027be0 2023-10-05T19:28:33.468 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.462+0000 7fd27cb0ee40 1 -- 172.21.15.167:0/403903602 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:85 7.1f 7:f95f44c2:::notify.0:head [watch watch cookie 94594747560400] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x56088e538760 con 0x7fd274027be0 2023-10-05T19:28:33.468 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.462+0000 7fd27cb0ee40 1 -- 172.21.15.167:0/403903602 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:86 7.0 7:05bf5b68:::notify.1:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x56088e539000 con 0x56088e4fb7f0 2023-10-05T19:28:33.470 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.466+0000 7fd279e42700 1 -- 172.21.15.167:0/403903602 <== osd.2 v2:172.21.15.167:6816/30037 14 ==== osd_op_reply(85 notify.0 [watch watch cookie 94594747560400] v30'121 uv120 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7fd264029040 con 0x7fd274027be0 2023-10-05T19:28:33.470 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.466+0000 7fd27ae44700 1 -- 172.21.15.167:0/403903602 <== osd.1 v2:172.21.15.167:6800/30035 29 ==== osd_op_reply(86 notify.1 [create] v30'120 uv120 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7fd26c050040 con 0x56088e4fb7f0 2023-10-05T19:28:33.471 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.466+0000 7fd27cb0ee40 1 -- 172.21.15.167:0/403903602 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:87 7.0 7:05bf5b68:::notify.1:head [watch watch cookie 94594747566864] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x56088e537be0 con 0x56088e4fb7f0 2023-10-05T19:28:33.471 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.466+0000 7fd27cb0ee40 1 -- 172.21.15.167:0/403903602 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:88 7.15 7:a93a5511:::notify.2:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x56088e539a50 con 0x56088e532470 2023-10-05T19:28:33.473 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.466+0000 7fd27ae44700 1 -- 172.21.15.167:0/403903602 <== osd.1 v2:172.21.15.167:6800/30035 30 ==== osd_op_reply(87 notify.1 [watch watch cookie 94594747566864] v30'121 uv120 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7fd26c050040 con 0x56088e4fb7f0 2023-10-05T19:28:33.474 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.470+0000 7fd27a643700 1 -- 172.21.15.167:0/403903602 <== osd.3 v2:172.21.15.167:6808/30036 22 ==== osd_op_reply(88 notify.2 [create] v30'120 uv120 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7fd274024020 con 0x56088e532470 2023-10-05T19:28:33.474 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.470+0000 7fd27cb0ee40 1 -- 172.21.15.167:0/403903602 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:89 7.15 7:a93a5511:::notify.2:head [watch watch cookie 94594747569504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x56088e539e50 con 0x56088e532470 2023-10-05T19:28:33.474 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.470+0000 7fd27cb0ee40 1 -- 172.21.15.167:0/403903602 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:90 7.e 7:7759931f:::notify.3:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x56088e53a730 con 0x7fd274027be0 2023-10-05T19:28:33.477 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.470+0000 7fd27a643700 1 -- 172.21.15.167:0/403903602 <== osd.3 v2:172.21.15.167:6808/30036 23 ==== osd_op_reply(89 notify.2 [watch watch cookie 94594747569504] v30'121 uv120 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7fd274024020 con 0x56088e532470 2023-10-05T19:28:33.477 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.470+0000 7fd279e42700 1 -- 172.21.15.167:0/403903602 <== osd.2 v2:172.21.15.167:6816/30037 15 ==== osd_op_reply(90 notify.3 [create] v30'120 uv120 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7fd264029040 con 0x7fd274027be0 2023-10-05T19:28:33.477 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.470+0000 7fd27cb0ee40 1 -- 172.21.15.167:0/403903602 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:91 7.e 7:7759931f:::notify.3:head [watch watch cookie 94594747567968] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x56088e53a360 con 0x7fd274027be0 2023-10-05T19:28:33.477 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.470+0000 7fd27cb0ee40 1 -- 172.21.15.167:0/403903602 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:92 7.d 7:b4812045:::notify.4:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x56088e53b7a0 con 0x56088e4fb7f0 2023-10-05T19:28:33.478 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.474+0000 7fd279e42700 1 -- 172.21.15.167:0/403903602 <== osd.2 v2:172.21.15.167:6816/30037 16 ==== osd_op_reply(91 notify.3 [watch watch cookie 94594747567968] v30'121 uv120 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7fd264029040 con 0x7fd274027be0 2023-10-05T19:28:33.479 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.474+0000 7fd27ae44700 1 -- 172.21.15.167:0/403903602 <== osd.1 v2:172.21.15.167:6800/30035 31 ==== osd_op_reply(92 notify.4 [create] v30'120 uv120 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7fd26c050040 con 0x56088e4fb7f0 2023-10-05T19:28:33.479 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.474+0000 7fd27cb0ee40 1 -- 172.21.15.167:0/403903602 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:93 7.d 7:b4812045:::notify.4:head [watch watch cookie 94594747575088] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x56088e53bba0 con 0x56088e4fb7f0 2023-10-05T19:28:33.479 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.474+0000 7fd27cb0ee40 1 -- 172.21.15.167:0/403903602 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:94 7.3 7:c609908c:::notify.5:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x56088e53c5b0 con 0x7fd264007030 2023-10-05T19:28:33.482 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.478+0000 7fd27ae44700 1 -- 172.21.15.167:0/403903602 <== osd.1 v2:172.21.15.167:6800/30035 32 ==== osd_op_reply(93 notify.4 [watch watch cookie 94594747575088] v30'121 uv120 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7fd26c050040 con 0x56088e4fb7f0 2023-10-05T19:28:33.483 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.478+0000 7fd27ae44700 1 -- 172.21.15.167:0/403903602 <== osd.0 v2:172.21.15.118:6800/27690 23 ==== osd_op_reply(94 notify.5 [create] v30'120 uv120 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7fd26c050040 con 0x7fd264007030 2023-10-05T19:28:33.483 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.478+0000 7fd27cb0ee40 1 -- 172.21.15.167:0/403903602 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:95 7.3 7:c609908c:::notify.5:head [watch watch cookie 94594747576944] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x56088e53c5b0 con 0x7fd264007030 2023-10-05T19:28:33.483 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.478+0000 7fd27cb0ee40 1 -- 172.21.15.167:0/403903602 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:96 7.14 7:2b04a3e9:::notify.6:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x56088e53d010 con 0x7fd264007030 2023-10-05T19:28:33.485 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.478+0000 7fd27ae44700 1 -- 172.21.15.167:0/403903602 <== osd.0 v2:172.21.15.118:6800/27690 24 ==== osd_op_reply(96 notify.6 [create] v30'120 uv120 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7fd26c050040 con 0x7fd264007030 2023-10-05T19:28:33.485 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.478+0000 7fd27cb0ee40 1 -- 172.21.15.167:0/403903602 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:97 7.14 7:2b04a3e9:::notify.6:head [watch watch cookie 94594747583264] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x56088e53d460 con 0x7fd264007030 2023-10-05T19:28:33.485 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.478+0000 7fd27cb0ee40 1 -- 172.21.15.167:0/403903602 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:98 7.9 7:93e5b521:::notify.7:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x56088e53deb0 con 0x7fd264007030 2023-10-05T19:28:33.485 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.478+0000 7fd27ae44700 1 -- 172.21.15.167:0/403903602 <== osd.0 v2:172.21.15.118:6800/27690 25 ==== osd_op_reply(95 notify.5 [watch watch cookie 94594747576944] v30'121 uv120 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7fd26c050040 con 0x7fd264007030 2023-10-05T19:28:33.487 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.482+0000 7fd27ae44700 1 -- 172.21.15.167:0/403903602 <== osd.0 v2:172.21.15.118:6800/27690 26 ==== osd_op_reply(97 notify.6 [watch watch cookie 94594747583264] v30'121 uv120 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7fd26c050040 con 0x7fd264007030 2023-10-05T19:28:33.488 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.482+0000 7fd27ae44700 1 -- 172.21.15.167:0/403903602 <== osd.0 v2:172.21.15.118:6800/27690 27 ==== osd_op_reply(98 notify.7 [create] v30'120 uv120 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7fd26c050040 con 0x7fd264007030 2023-10-05T19:28:33.488 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.482+0000 7fd27cb0ee40 1 -- 172.21.15.167:0/403903602 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:99 7.9 7:93e5b521:::notify.7:head [watch watch cookie 94594747580608] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x56088e53cf70 con 0x7fd264007030 2023-10-05T19:28:33.488 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.482+0000 7fd27cb0ee40 20 add_watcher() i=0 2023-10-05T19:28:33.488 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.482+0000 7fd27cb0ee40 20 add_watcher() i=1 2023-10-05T19:28:33.488 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.482+0000 7fd27cb0ee40 20 add_watcher() i=2 2023-10-05T19:28:33.489 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.482+0000 7fd27cb0ee40 20 add_watcher() i=3 2023-10-05T19:28:33.489 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.482+0000 7fd27cb0ee40 20 add_watcher() i=4 2023-10-05T19:28:33.489 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.482+0000 7fd27cb0ee40 20 add_watcher() i=5 2023-10-05T19:28:33.489 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.482+0000 7fd27cb0ee40 20 add_watcher() i=6 2023-10-05T19:28:33.490 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.486+0000 7fd27ae44700 1 -- 172.21.15.167:0/403903602 <== osd.0 v2:172.21.15.118:6800/27690 28 ==== osd_op_reply(99 notify.7 [watch watch cookie 94594747580608] v30'121 uv120 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7fd26c050040 con 0x7fd264007030 2023-10-05T19:28:33.490 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.486+0000 7fd27cb0ee40 20 add_watcher() i=7 2023-10-05T19:28:33.490 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.486+0000 7fd27cb0ee40 2 all 8 watchers are set, enabling cache 2023-10-05T19:28:33.490 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.486+0000 7fd27cb0ee40 1 -- 172.21.15.167:0/403903602 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:100 6.4 6:22d26bf9:::data_loggenerations_metadata:head [call version.check_conds in=74b,call version.read in=11b,read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x56088e53cf70 con 0x56088e4fb7f0 2023-10-05T19:28:33.491 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.486+0000 7fd27ae44700 1 -- 172.21.15.167:0/403903602 <== osd.1 v2:172.21.15.167:6800/30035 33 ==== osd_op_reply(100 data_loggenerations_metadata [call,call out=48b,read 0~28 out=28b] v0'0 uv1 ondisk = 0) v8 ==== 256+0+76 (crc 0 0 0) 0x7fd26c050040 con 0x56088e4fb7f0 2023-10-05T19:28:33.491 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.486+0000 7fd27cb0ee40 1 -- 172.21.15.167:0/403903602 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:101 6.4 6:22d26bf9:::data_loggenerations_metadata:head [watch watch cookie 94594747586944] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x56088e53cf70 con 0x56088e4fb7f0 2023-10-05T19:28:33.493 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.486+0000 7fd27ae44700 1 -- 172.21.15.167:0/403903602 <== osd.1 v2:172.21.15.167:6800/30035 34 ==== osd_op_reply(101 data_loggenerations_metadata [watch watch cookie 94594747586944] v30'118 uv1 ondisk = 0) v8 ==== 172+0+0 (crc 0 0 0) 0x7fd26c050040 con 0x56088e4fb7f0 2023-10-05T19:28:33.493 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.486+0000 7fd24bfdf700 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2023-10-05T19:28:33.493 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.486+0000 7fd27cb0ee40 20 check_secure_mon_conn(): auth registy supported: methods=[2] modes=[2,1] 2023-10-05T19:28:33.493 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.486+0000 7fd27cb0ee40 20 check_secure_mon_conn(): mode 1 is insecure 2023-10-05T19:28:33.494 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.486+0000 7fd27cb0ee40 1 Processor -- start 2023-10-05T19:28:33.494 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.486+0000 7fd27cb0ee40 1 -- start start 2023-10-05T19:28:33.494 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.490+0000 7fd27cb0ee40 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x56088e54b370 0x56088e54b740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-05T19:28:33.494 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.490+0000 7fd27cb0ee40 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x56088e54bc80 con 0x56088e54b370 2023-10-05T19:28:33.495 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.490+0000 7fd279e42700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x56088e54b370 0x56088e54b740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-05T19:28:33.495 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.490+0000 7fd279e42700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x56088e54b370 0x56088e54b740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.118:3300/0 says I am v2:172.21.15.167:58620/0 (socket says 172.21.15.167:58620) 2023-10-05T19:28:33.495 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.490+0000 7fd279e42700 1 -- 172.21.15.167:0/3370130153 learned_addr learned my addr 172.21.15.167:0/3370130153 (peer_addr_for_me v2:172.21.15.167:0/0) 2023-10-05T19:28:33.495 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.490+0000 7fd279e42700 1 -- 172.21.15.167:0/3370130153 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x56088e54bdc0 con 0x56088e54b370 2023-10-05T19:28:33.496 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.490+0000 7fd279e42700 1 --2- 172.21.15.167:0/3370130153 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x56088e54b370 0x56088e54b740 secure :-1 s=READY pgs=378 cs=0 l=1 rev1=1 crypto rx=0x56088e4facb0 tx=0x7fd264030040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ec8a8c9045fe1664 server_cookie=0 in_seq=0 out_seq=0 2023-10-05T19:28:33.496 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.490+0000 7fd24b7de700 1 -- 172.21.15.167:0/3370130153 <== mon.0 v2:172.21.15.118:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fd264025040 con 0x56088e54b370 2023-10-05T19:28:33.496 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.490+0000 7fd24b7de700 1 -- 172.21.15.167:0/3370130153 <== mon.0 v2:172.21.15.118:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd264034050 con 0x56088e54b370 2023-10-05T19:28:33.496 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.490+0000 7fd24b7de700 1 -- 172.21.15.167:0/3370130153 <== mon.0 v2:172.21.15.118:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fd26402e4e0 con 0x56088e54b370 2023-10-05T19:28:33.497 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.490+0000 7fd27cb0ee40 1 -- 172.21.15.167:0/3370130153 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x56088e54b370 msgr2=0x56088e54b740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-05T19:28:33.497 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.490+0000 7fd27cb0ee40 1 --2- 172.21.15.167:0/3370130153 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x56088e54b370 0x56088e54b740 secure :-1 s=READY pgs=378 cs=0 l=1 rev1=1 crypto rx=0x56088e4facb0 tx=0x7fd264030040 comp rx=0 tx=0).stop 2023-10-05T19:28:33.497 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.490+0000 7fd27cb0ee40 1 -- 172.21.15.167:0/3370130153 shutdown_connections 2023-10-05T19:28:33.497 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.490+0000 7fd27cb0ee40 1 --2- 172.21.15.167:0/3370130153 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x56088e54b370 0x56088e54b740 unknown :-1 s=CLOSED pgs=378 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-05T19:28:33.497 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.490+0000 7fd27cb0ee40 1 -- 172.21.15.167:0/3370130153 >> 172.21.15.167:0/3370130153 conn(0x56088e546730 msgr2=0x56088e544880 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-05T19:28:33.498 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.490+0000 7fd27cb0ee40 1 -- 172.21.15.167:0/3370130153 shutdown_connections 2023-10-05T19:28:33.498 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.490+0000 7fd27cb0ee40 1 -- 172.21.15.167:0/3370130153 wait complete. 2023-10-05T19:28:33.498 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.490+0000 7fd27cb0ee40 1 Processor -- start 2023-10-05T19:28:33.498 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.494+0000 7fd27cb0ee40 1 -- start start 2023-10-05T19:28:33.498 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.494+0000 7fd27cb0ee40 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x56088e54b370 0x56088e550e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-05T19:28:33.499 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.494+0000 7fd27cb0ee40 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x56088e551e10 con 0x56088e54b370 2023-10-05T19:28:33.499 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.494+0000 7fd279e42700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x56088e54b370 0x56088e550e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-05T19:28:33.499 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.494+0000 7fd279e42700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x56088e54b370 0x56088e550e10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.118:3300/0 says I am v2:172.21.15.167:58632/0 (socket says 172.21.15.167:58632) 2023-10-05T19:28:33.499 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.494+0000 7fd279e42700 1 -- 172.21.15.167:0/3725797096 learned_addr learned my addr 172.21.15.167:0/3725797096 (peer_addr_for_me v2:172.21.15.167:0/0) 2023-10-05T19:28:33.500 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.494+0000 7fd279e42700 1 -- 172.21.15.167:0/3725797096 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x56088e4face0 con 0x56088e54b370 2023-10-05T19:28:33.500 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.494+0000 7fd279e42700 1 --2- 172.21.15.167:0/3725797096 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x56088e54b370 0x56088e550e10 secure :-1 s=READY pgs=379 cs=0 l=1 rev1=1 crypto rx=0x56088e5453f0 tx=0x7fd264038000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-05T19:28:33.500 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.494+0000 7fd249fdb700 1 -- 172.21.15.167:0/3725797096 <== mon.0 v2:172.21.15.118:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fd264039870 con 0x56088e54b370 2023-10-05T19:28:33.501 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.494+0000 7fd249fdb700 1 -- 172.21.15.167:0/3725797096 <== mon.0 v2:172.21.15.118:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd264025030 con 0x56088e54b370 2023-10-05T19:28:33.501 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.494+0000 7fd249fdb700 1 -- 172.21.15.167:0/3725797096 <== mon.0 v2:172.21.15.118:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fd264039a10 con 0x56088e54b370 2023-10-05T19:28:33.501 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.494+0000 7fd27cb0ee40 1 -- 172.21.15.167:0/3725797096 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x56088e551410 con 0x56088e54b370 2023-10-05T19:28:33.501 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.494+0000 7fd27cb0ee40 1 -- 172.21.15.167:0/3725797096 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x56088e551850 con 0x56088e54b370 2023-10-05T19:28:33.502 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.498+0000 7fd249fdb700 1 -- 172.21.15.167:0/3725797096 <== mon.0 v2:172.21.15.118:3300/0 4 ==== mgrmap(e 5) v1 ==== 40329+0+0 (secure 0 0 0) 0x7fd264039b70 con 0x56088e54b370 2023-10-05T19:28:33.503 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.498+0000 7fd249fdb700 1 --2- 172.21.15.167:0/3725797096 >> [v2:172.21.15.167:6824/29889,v1:172.21.15.167:6825/29889] conn(0x7fd1c0030c50 0x7fd1c0033100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-05T19:28:33.503 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.498+0000 7fd249fdb700 1 -- 172.21.15.167:0/3725797096 <== mon.0 v2:172.21.15.118:3300/0 5 ==== osd_map(30..30 src has 1..30) v4 ==== 6057+0+0 (secure 0 0 0) 0x7fd26404ec30 con 0x56088e54b370 2023-10-05T19:28:33.503 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.498+0000 7fd27ae44700 1 --2- 172.21.15.167:0/3725797096 >> [v2:172.21.15.167:6824/29889,v1:172.21.15.167:6825/29889] conn(0x7fd1c0030c50 0x7fd1c0033100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-05T19:28:33.503 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.498+0000 7fd27cb0ee40 5 note: GC not initialized 2023-10-05T19:28:33.504 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.498+0000 7fd27ae44700 1 --2- 172.21.15.167:0/3725797096 >> [v2:172.21.15.167:6824/29889,v1:172.21.15.167:6825/29889] conn(0x7fd1c0030c50 0x7fd1c0033100 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x56088e53d750 tx=0x7fd26c000f90 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-05T19:28:33.505 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.498+0000 7fd1baffd700 20 reqs_thread_entry: start 2023-10-05T19:28:33.506 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.502+0000 7fd1b8ff9700 20 reqs_thread_entry: start 2023-10-05T19:28:33.506 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.502+0000 7fd27cb0ee40 20 init_complete bucket index max shards: 11 2023-10-05T19:28:33.507 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.502+0000 7fd27cb0ee40 10 rgw notify: Started notification manager with: 1 workers 2023-10-05T19:28:33.507 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.502+0000 7fd1ae7fc700 20 reqs_thread_entry: start 2023-10-05T19:28:33.507 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.502+0000 7fd27cb0ee40 20 get_system_obj_state: rctx=0x7ffe844d58e8 obj=default.rgw.meta:users.uid:fud state=0x56088e3d40b0 s->prefetch_data=0 2023-10-05T19:28:33.507 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.502+0000 7fd27cb0ee40 10 cache get: name=default.rgw.meta+users.uid+fud : miss 2023-10-05T19:28:33.507 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.502+0000 7fd27cb0ee40 1 -- 172.21.15.167:0/403903602 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:102 8.e 8:75f48285:users.uid::fud:head [call version.read in=11b,getxattrs,stat] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x56088e5b96c0 con 0x7fd264007030 2023-10-05T19:28:33.507 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.502+0000 7fd1aeffd700 20 rgw notify: INFO: next queues processing will happen at: Thu Oct 5 19:29:03 2023 2023-10-05T19:28:33.507 INFO:teuthology.orchestra.run.smithi167.stderr: 2023-10-05T19:28:33.508 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.502+0000 7fd27ae44700 1 -- 172.21.15.167:0/403903602 <== osd.0 v2:172.21.15.118:6800/27690 29 ==== osd_op_reply(102 fud [call out=48b,getxattrs out=75b,stat out=16b] v0'0 uv1 ondisk = 0) v8 ==== 231+0+139 (crc 0 0 0) 0x7fd26c050040 con 0x7fd264007030 2023-10-05T19:28:33.508 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.502+0000 7fd27cb0ee40 10 cache put: name=default.rgw.meta+users.uid+fud info.flags=0x16 2023-10-05T19:28:33.508 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.502+0000 7fd27cb0ee40 10 adding default.rgw.meta+users.uid+fud to cache LRU end 2023-10-05T19:28:33.508 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.502+0000 7fd27cb0ee40 10 updating xattr: name=ceph.objclass.version bl.length()=42 2023-10-05T19:28:33.508 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.502+0000 7fd27cb0ee40 20 get_system_obj_state: s->obj_tag was set empty 2023-10-05T19:28:33.508 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.502+0000 7fd27cb0ee40 20 Read xattr: user.rgw.idtag 2023-10-05T19:28:33.508 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.502+0000 7fd27cb0ee40 10 cache get: name=default.rgw.meta+users.uid+fud : type miss (requested=0x13, cached=0x16) 2023-10-05T19:28:33.508 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.502+0000 7fd27cb0ee40 20 rados->read ofs=0 len=0 2023-10-05T19:28:33.508 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.502+0000 7fd27cb0ee40 1 -- 172.21.15.167:0/403903602 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:103 8.e 8:75f48285:users.uid::fud:head [call version.check_conds in=122b,call version.read in=11b,read 0~0,getxattrs] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x56088e5bc150 con 0x7fd264007030 2023-10-05T19:28:33.509 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.502+0000 7fd27ae44700 1 -- 172.21.15.167:0/403903602 <== osd.0 v2:172.21.15.118:6800/27690 30 ==== osd_op_reply(103 fud [call,call out=48b,read 0~346 out=346b,getxattrs out=75b] v0'0 uv1 ondisk = 0) v8 ==== 273+0+469 (crc 0 0 0) 0x7fd26c050040 con 0x7fd264007030 2023-10-05T19:28:33.509 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.502+0000 7fd27cb0ee40 20 rados_obj.operate() r=0 bl.length=346 2023-10-05T19:28:33.509 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.502+0000 7fd27cb0ee40 10 cache put: name=default.rgw.meta+users.uid+fud info.flags=0x13 2023-10-05T19:28:33.509 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.502+0000 7fd27cb0ee40 10 moving default.rgw.meta+users.uid+fud to cache LRU end 2023-10-05T19:28:33.509 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.502+0000 7fd27cb0ee40 10 updating xattr: name=ceph.objclass.version bl.length()=42 2023-10-05T19:28:33.510 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.502+0000 7fd27cb0ee40 20 get_system_obj_state: rctx=0x7ffe844d5930 obj=default.rgw.meta:root:myfoo state=0x56088e3d40b0 s->prefetch_data=0 2023-10-05T19:28:33.510 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.502+0000 7fd27cb0ee40 10 cache get: name=default.rgw.meta+root+myfoo : miss 2023-10-05T19:28:33.510 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.502+0000 7fd27cb0ee40 1 -- 172.21.15.167:0/403903602 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:104 8.9 8:91b71841:root::myfoo:head [call version.read in=11b,getxattrs,stat] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x56088e5bb960 con 0x7fd274027be0 2023-10-05T19:28:33.510 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.502+0000 7fd279e42700 1 -- 172.21.15.167:0/403903602 <== osd.2 v2:172.21.15.167:6816/30037 17 ==== osd_op_reply(104 myfoo [call out=48b,getxattrs out=75b,stat out=16b] v0'0 uv20 ondisk = 0) v8 ==== 233+0+139 (crc 0 0 0) 0x7fd26403f020 con 0x7fd274027be0 2023-10-05T19:28:33.510 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.502+0000 7fd27cb0ee40 10 cache put: name=default.rgw.meta+root+myfoo info.flags=0x16 2023-10-05T19:28:33.511 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.502+0000 7fd27cb0ee40 10 adding default.rgw.meta+root+myfoo to cache LRU end 2023-10-05T19:28:33.511 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.502+0000 7fd27cb0ee40 10 updating xattr: name=ceph.objclass.version bl.length()=42 2023-10-05T19:28:33.511 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.502+0000 7fd27cb0ee40 20 get_system_obj_state: s->obj_tag was set empty 2023-10-05T19:28:33.511 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.502+0000 7fd27cb0ee40 10 cache get: name=default.rgw.meta+root+myfoo : type miss (requested=0x11, cached=0x16) 2023-10-05T19:28:33.511 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.502+0000 7fd27cb0ee40 20 rados->read ofs=0 len=0 2023-10-05T19:28:33.511 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.506+0000 7fd27cb0ee40 1 -- 172.21.15.167:0/403903602 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:105 8.9 8:91b71841:root::myfoo:head [call version.check_conds in=122b,call version.read in=11b,read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x56088e5bdf00 con 0x7fd274027be0 2023-10-05T19:28:33.512 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.506+0000 7fd279e42700 1 -- 172.21.15.167:0/403903602 <== osd.2 v2:172.21.15.167:6816/30037 18 ==== osd_op_reply(105 myfoo [call,call out=48b,read 0~165 out=165b] v0'0 uv20 ondisk = 0) v8 ==== 233+0+213 (crc 0 0 0) 0x7fd26403f020 con 0x7fd274027be0 2023-10-05T19:28:33.512 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.506+0000 7fd27cb0ee40 20 rados_obj.operate() r=0 bl.length=165 2023-10-05T19:28:33.512 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.506+0000 7fd27cb0ee40 10 cache put: name=default.rgw.meta+root+myfoo info.flags=0x11 2023-10-05T19:28:33.512 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.506+0000 7fd27cb0ee40 10 moving default.rgw.meta+root+myfoo to cache LRU end 2023-10-05T19:28:33.512 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.506+0000 7fd27cb0ee40 20 get_system_obj_state: rctx=0x7ffe844d5930 obj=default.rgw.meta:root:.bucket.meta.myfoo:9a9df965-6612-45d4-a45f-6e3c889bacc2.4223.1 state=0x56088e3aa2b0 s->prefetch_data=0 2023-10-05T19:28:33.512 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.506+0000 7fd27cb0ee40 10 cache get: name=default.rgw.meta+root+.bucket.meta.myfoo:9a9df965-6612-45d4-a45f-6e3c889bacc2.4223.1 : miss 2023-10-05T19:28:33.513 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.506+0000 7fd27cb0ee40 1 -- 172.21.15.167:0/403903602 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:106 8.11 8:8bb4ed75:root::.bucket.meta.myfoo%3a9a9df965-6612-45d4-a45f-6e3c889bacc2.4223.1:head [call version.read in=11b,getxattrs,stat] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x56088e5bd690 con 0x56088e532470 2023-10-05T19:28:33.513 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.506+0000 7fd27a643700 1 -- 172.21.15.167:0/403903602 <== osd.3 v2:172.21.15.167:6808/30036 24 ==== osd_op_reply(106 .bucket.meta.myfoo:9a9df965-6612-45d4-a45f-6e3c889bacc2.4223.1 [call out=48b,getxattrs out=218b,stat out=16b] v0'0 uv2 ondisk = 0) v8 ==== 290+0+282 (crc 0 0 0) 0x7fd274024020 con 0x56088e532470 2023-10-05T19:28:33.513 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.506+0000 7fd27cb0ee40 10 cache put: name=default.rgw.meta+root+.bucket.meta.myfoo:9a9df965-6612-45d4-a45f-6e3c889bacc2.4223.1 info.flags=0x16 2023-10-05T19:28:33.513 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.506+0000 7fd27cb0ee40 10 adding default.rgw.meta+root+.bucket.meta.myfoo:9a9df965-6612-45d4-a45f-6e3c889bacc2.4223.1 to cache LRU end 2023-10-05T19:28:33.513 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.506+0000 7fd27cb0ee40 10 updating xattr: name=ceph.objclass.version bl.length()=42 2023-10-05T19:28:33.513 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.506+0000 7fd27cb0ee40 10 updating xattr: name=user.rgw.acl bl.length()=123 2023-10-05T19:28:33.513 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.506+0000 7fd27cb0ee40 20 get_system_obj_state: s->obj_tag was set empty 2023-10-05T19:28:33.513 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.506+0000 7fd27cb0ee40 20 Read xattr: user.rgw.acl 2023-10-05T19:28:33.513 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.506+0000 7fd27cb0ee40 20 Read xattr: user.rgw.idtag 2023-10-05T19:28:33.513 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.506+0000 7fd27cb0ee40 10 cache get: name=default.rgw.meta+root+.bucket.meta.myfoo:9a9df965-6612-45d4-a45f-6e3c889bacc2.4223.1 : type miss (requested=0x13, cached=0x16) 2023-10-05T19:28:33.513 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.506+0000 7fd27cb0ee40 20 rados->read ofs=0 len=0 2023-10-05T19:28:33.513 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.506+0000 7fd27cb0ee40 1 -- 172.21.15.167:0/403903602 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:107 8.11 8:8bb4ed75:root::.bucket.meta.myfoo%3a9a9df965-6612-45d4-a45f-6e3c889bacc2.4223.1:head [call version.check_conds in=122b,call version.read in=11b,read 0~0,getxattrs] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x56088e5c0990 con 0x56088e532470 2023-10-05T19:28:33.515 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.506+0000 7fd27a643700 1 -- 172.21.15.167:0/403903602 <== osd.3 v2:172.21.15.167:6808/30036 25 ==== osd_op_reply(107 .bucket.meta.myfoo:9a9df965-6612-45d4-a45f-6e3c889bacc2.4223.1 [call,call out=48b,read 0~352 out=352b,getxattrs out=218b] v0'0 uv2 ondisk = 0) v8 ==== 332+0+618 (crc 0 0 0) 0x7fd27403d280 con 0x56088e532470 2023-10-05T19:28:33.515 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.506+0000 7fd27cb0ee40 20 rados_obj.operate() r=0 bl.length=352 2023-10-05T19:28:33.515 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.506+0000 7fd27cb0ee40 10 cache put: name=default.rgw.meta+root+.bucket.meta.myfoo:9a9df965-6612-45d4-a45f-6e3c889bacc2.4223.1 info.flags=0x13 2023-10-05T19:28:33.515 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.506+0000 7fd27cb0ee40 10 moving default.rgw.meta+root+.bucket.meta.myfoo:9a9df965-6612-45d4-a45f-6e3c889bacc2.4223.1 to cache LRU end 2023-10-05T19:28:33.515 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.506+0000 7fd27cb0ee40 10 updating xattr: name=ceph.objclass.version bl.length()=42 2023-10-05T19:28:33.515 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.506+0000 7fd27cb0ee40 10 updating xattr: name=user.rgw.acl bl.length()=123 2023-10-05T19:28:33.515 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.506+0000 7fd27cb0ee40 10 chain_cache_entry: cache_locator=default.rgw.meta+root+.bucket.meta.myfoo:9a9df965-6612-45d4-a45f-6e3c889bacc2.4223.1 2023-10-05T19:28:33.515 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.506+0000 7fd27cb0ee40 20 get_system_obj_state: rctx=0x7ffe844d58c8 obj=default.rgw.meta:users.uid:fud state=0x56088e3aa2b0 s->prefetch_data=0 2023-10-05T19:28:33.515 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.506+0000 7fd27cb0ee40 10 cache get: name=default.rgw.meta+users.uid+fud : type miss (requested=0x16, cached=0x13) 2023-10-05T19:28:33.515 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.506+0000 7fd27cb0ee40 1 -- 172.21.15.167:0/403903602 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:108 8.e 8:75f48285:users.uid::fud:head [call version.read in=11b,getxattrs,stat] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x56088e5c0fd0 con 0x7fd264007030 2023-10-05T19:28:33.516 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.506+0000 7fd27ae44700 1 -- 172.21.15.167:0/403903602 <== osd.0 v2:172.21.15.118:6800/27690 31 ==== osd_op_reply(108 fud [call out=48b,getxattrs out=75b,stat out=16b] v0'0 uv1 ondisk = 0) v8 ==== 231+0+139 (crc 0 0 0) 0x7fd26c050040 con 0x7fd264007030 2023-10-05T19:28:33.516 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.506+0000 7fd27cb0ee40 10 cache put: name=default.rgw.meta+users.uid+fud info.flags=0x16 2023-10-05T19:28:33.516 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.506+0000 7fd27cb0ee40 10 moving default.rgw.meta+users.uid+fud to cache LRU end 2023-10-05T19:28:33.516 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.506+0000 7fd27cb0ee40 10 updating xattr: name=ceph.objclass.version bl.length()=42 2023-10-05T19:28:33.516 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.506+0000 7fd27cb0ee40 20 get_system_obj_state: s->obj_tag was set empty 2023-10-05T19:28:33.516 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.506+0000 7fd27cb0ee40 20 Read xattr: user.rgw.idtag 2023-10-05T19:28:33.516 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.506+0000 7fd27cb0ee40 10 cache get: name=default.rgw.meta+users.uid+fud : hit (requested=0x13, cached=0x17) 2023-10-05T19:28:33.516 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.506+0000 7fd27cb0ee40 1 -- 172.21.15.167:0/403903602 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:109 8.1 8:810902b1:users.uid::fud.buckets:head [call user.remove_bucket in=58b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x56088e5c0fd0 con 0x7fd264007030 2023-10-05T19:28:33.517 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.510+0000 7fd27ae44700 1 -- 172.21.15.167:0/403903602 <== osd.0 v2:172.21.15.118:6800/27690 32 ==== osd_op_reply(109 fud.buckets [call] v30'2 uv2 ondisk = 0) v8 ==== 155+0+0 (crc 0 0 0) 0x7fd26c050040 con 0x7fd264007030 2023-10-05T19:28:33.517 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.510+0000 7fd27cb0ee40 20 get_system_obj_state: rctx=0x7ffe844d5a08 obj=default.rgw.meta:root:myfoo state=0x56088e3aa2b0 s->prefetch_data=0 2023-10-05T19:28:33.517 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.510+0000 7fd27cb0ee40 10 cache get: name=default.rgw.meta+root+myfoo : type miss (requested=0x16, cached=0x13) 2023-10-05T19:28:33.517 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.510+0000 7fd27cb0ee40 1 -- 172.21.15.167:0/403903602 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:110 8.9 8:91b71841:root::myfoo:head [call version.read in=11b,getxattrs,stat] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x56088e5c1550 con 0x7fd274027be0 2023-10-05T19:28:33.518 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.510+0000 7fd279e42700 1 -- 172.21.15.167:0/403903602 <== osd.2 v2:172.21.15.167:6816/30037 19 ==== osd_op_reply(110 myfoo [call out=48b,getxattrs out=75b,stat out=16b] v0'0 uv20 ondisk = 0) v8 ==== 233+0+139 (crc 0 0 0) 0x7fd26403f020 con 0x7fd274027be0 2023-10-05T19:28:33.518 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.510+0000 7fd27cb0ee40 10 cache put: name=default.rgw.meta+root+myfoo info.flags=0x16 2023-10-05T19:28:33.518 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.510+0000 7fd27cb0ee40 10 moving default.rgw.meta+root+myfoo to cache LRU end 2023-10-05T19:28:33.518 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.510+0000 7fd27cb0ee40 10 updating xattr: name=ceph.objclass.version bl.length()=42 2023-10-05T19:28:33.518 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.510+0000 7fd27cb0ee40 20 get_system_obj_state: s->obj_tag was set empty 2023-10-05T19:28:33.519 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.510+0000 7fd27cb0ee40 20 Read xattr: user.rgw.idtag 2023-10-05T19:28:33.519 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.510+0000 7fd27cb0ee40 10 cache get: name=default.rgw.meta+root+myfoo : hit (requested=0x13, cached=0x17) 2023-10-05T19:28:33.519 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.510+0000 7fd27cb0ee40 1 -- 172.21.15.167:0/403903602 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:111 8.9 8:91b71841:root::myfoo:head [delete,create,call version.check_conds in=122b,call version.set in=58b,writefull 0~165 in=165b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x56088e5c63a0 con 0x7fd274027be0 2023-10-05T19:28:33.521 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.514+0000 7fd279e42700 1 -- 172.21.15.167:0/403903602 <== osd.2 v2:172.21.15.167:6816/30037 20 ==== osd_op_reply(111 myfoo [delete,create,call,call,writefull 0~165] v30'21 uv21 ondisk = 0) v8 ==== 317+0+0 (crc 0 0 0) 0x7fd26403f020 con 0x7fd274027be0 2023-10-05T19:28:33.521 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.514+0000 7fd27cb0ee40 10 cache put: name=default.rgw.meta+root+myfoo info.flags=0x17 2023-10-05T19:28:33.521 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.514+0000 7fd27cb0ee40 10 moving default.rgw.meta+root+myfoo to cache LRU end 2023-10-05T19:28:33.521 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.514+0000 7fd27cb0ee40 10 distributing notification oid=default.rgw.control:notify.6 cni=[op: 0, obj: default.rgw.meta:root:myfoo, ofs0, ns] 2023-10-05T19:28:33.521 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.514+0000 7fd27cb0ee40 1 -- 172.21.15.167:0/403903602 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:112 7.14 7:2b04a3e9:::notify.6:head [notify cookie 94594748145328 in=350b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x56088e5c6300 con 0x7fd264007030 2023-10-05T19:28:33.522 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.514+0000 7fd27ae44700 1 -- 172.21.15.167:0/403903602 <== osd.0 v2:172.21.15.118:6800/27690 33 ==== watch-notify(notify (1) cookie 94594747583264 notify 128849018886 ret 0) v3 ==== 380+0+0 (crc 0 0 0) 0x7fd26c03d040 con 0x7fd264007030 2023-10-05T19:28:33.522 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.514+0000 7fd273fff700 10 rgw watcher librados: RGWWatcher::handle_notify() notify_id 128849018886 cookie 94594747583264 notifier 4467 bl.length()=338 2023-10-05T19:28:33.522 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.518+0000 7fd273fff700 10 rgw watcher librados: cache put: name=default.rgw.meta+root+myfoo info.flags=0x17 2023-10-05T19:28:33.522 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.518+0000 7fd273fff700 10 rgw watcher librados: moving default.rgw.meta+root+myfoo to cache LRU end 2023-10-05T19:28:33.522 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.518+0000 7fd27ae44700 1 -- 172.21.15.167:0/403903602 <== osd.0 v2:172.21.15.118:6800/27690 34 ==== osd_op_reply(112 notify.6 [notify cookie 94594748145328 out=8b] v0'0 uv120 ondisk = 0) v8 ==== 152+0+8 (crc 0 0 0) 0x7fd26c050040 con 0x7fd264007030 2023-10-05T19:28:33.522 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.518+0000 7fd273fff700 1 -- 172.21.15.167:0/403903602 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:113 7.14 7:2b04a3e9:::notify.6:head [notify-ack in=20b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x7fd260003d90 con 0x7fd264007030 2023-10-05T19:28:33.523 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.518+0000 7fd27ae44700 1 -- 172.21.15.167:0/403903602 <== osd.0 v2:172.21.15.118:6800/27690 35 ==== osd_op_reply(113 notify.6 [notify-ack] v0'0 uv120 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7fd26c050040 con 0x7fd264007030 2023-10-05T19:28:33.523 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.518+0000 7fd27ae44700 1 -- 172.21.15.167:0/403903602 <== osd.0 v2:172.21.15.118:6800/27690 36 ==== watch-notify(notify_complete (2) cookie 94594748145328 notify 128849018886 ret 0) v3 ==== 42+0+48 (crc 0 0 0) 0x7fd26c002c30 con 0x7fd264007030 2023-10-05T19:28:33.523 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.518+0000 7fd27cb0ee40 1 -- 172.21.15.167:0/403903602 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:114 7.1f 7:f95f44c2:::notify.0:head [watch unwatch cookie 94594747560400] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x56088e5c63a0 con 0x7fd274027be0 2023-10-05T19:28:33.526 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.522+0000 7fd279e42700 1 -- 172.21.15.167:0/403903602 <== osd.2 v2:172.21.15.167:6816/30037 21 ==== osd_op_reply(114 notify.0 [watch unwatch cookie 94594747560400] v30'122 uv120 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7fd26403f020 con 0x7fd274027be0 2023-10-05T19:28:33.526 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.522+0000 7fd27cb0ee40 20 remove_watcher() i=0 2023-10-05T19:28:33.526 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.522+0000 7fd27cb0ee40 2 removed watcher, disabling cache 2023-10-05T19:28:33.526 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.522+0000 7fd27cb0ee40 1 -- 172.21.15.167:0/403903602 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:115 7.0 7:05bf5b68:::notify.1:head [watch unwatch cookie 94594747566864] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x56088e5128b0 con 0x56088e4fb7f0 2023-10-05T19:28:33.529 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.522+0000 7fd27ae44700 1 -- 172.21.15.167:0/403903602 <== osd.1 v2:172.21.15.167:6800/30035 35 ==== osd_op_reply(115 notify.1 [watch unwatch cookie 94594747566864] v30'122 uv120 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7fd26c050040 con 0x56088e4fb7f0 2023-10-05T19:28:33.529 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.522+0000 7fd27cb0ee40 20 remove_watcher() i=1 2023-10-05T19:28:33.529 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.522+0000 7fd27cb0ee40 1 -- 172.21.15.167:0/403903602 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:116 7.15 7:a93a5511:::notify.2:head [watch unwatch cookie 94594747569504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x56088e5128b0 con 0x56088e532470 2023-10-05T19:28:33.532 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.526+0000 7fd27a643700 1 -- 172.21.15.167:0/403903602 <== osd.3 v2:172.21.15.167:6808/30036 26 ==== osd_op_reply(116 notify.2 [watch unwatch cookie 94594747569504] v30'122 uv120 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7fd27403d280 con 0x56088e532470 2023-10-05T19:28:33.532 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.526+0000 7fd27cb0ee40 20 remove_watcher() i=2 2023-10-05T19:28:33.532 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.526+0000 7fd27cb0ee40 1 -- 172.21.15.167:0/403903602 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:117 7.e 7:7759931f:::notify.3:head [watch unwatch cookie 94594747567968] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x56088e5128b0 con 0x7fd274027be0 2023-10-05T19:28:33.534 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.530+0000 7fd279e42700 1 -- 172.21.15.167:0/403903602 <== osd.2 v2:172.21.15.167:6816/30037 22 ==== osd_op_reply(117 notify.3 [watch unwatch cookie 94594747567968] v30'122 uv120 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7fd26403f020 con 0x7fd274027be0 2023-10-05T19:28:33.534 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.530+0000 7fd27cb0ee40 20 remove_watcher() i=3 2023-10-05T19:28:33.535 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.530+0000 7fd27cb0ee40 1 -- 172.21.15.167:0/403903602 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:118 7.d 7:b4812045:::notify.4:head [watch unwatch cookie 94594747575088] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x56088e5128b0 con 0x56088e4fb7f0 2023-10-05T19:28:33.537 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.530+0000 7fd27ae44700 1 -- 172.21.15.167:0/403903602 <== osd.1 v2:172.21.15.167:6800/30035 36 ==== osd_op_reply(118 notify.4 [watch unwatch cookie 94594747575088] v30'122 uv120 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7fd26c050040 con 0x56088e4fb7f0 2023-10-05T19:28:33.538 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.530+0000 7fd27cb0ee40 20 remove_watcher() i=4 2023-10-05T19:28:33.538 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.530+0000 7fd27cb0ee40 1 -- 172.21.15.167:0/403903602 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:119 7.3 7:c609908c:::notify.5:head [watch unwatch cookie 94594747576944] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x56088e5128b0 con 0x7fd264007030 2023-10-05T19:28:33.541 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.534+0000 7fd27ae44700 1 -- 172.21.15.167:0/403903602 <== osd.0 v2:172.21.15.118:6800/27690 37 ==== osd_op_reply(119 notify.5 [watch unwatch cookie 94594747576944] v30'122 uv120 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7fd26c050040 con 0x7fd264007030 2023-10-05T19:28:33.541 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.534+0000 7fd27cb0ee40 20 remove_watcher() i=5 2023-10-05T19:28:33.541 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.534+0000 7fd27cb0ee40 1 -- 172.21.15.167:0/403903602 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:120 7.14 7:2b04a3e9:::notify.6:head [watch unwatch cookie 94594747583264] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x56088e5128b0 con 0x7fd264007030 2023-10-05T19:28:33.544 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.538+0000 7fd27ae44700 1 -- 172.21.15.167:0/403903602 <== osd.0 v2:172.21.15.118:6800/27690 38 ==== osd_op_reply(120 notify.6 [watch unwatch cookie 94594747583264] v30'122 uv120 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7fd26c050040 con 0x7fd264007030 2023-10-05T19:28:33.544 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.538+0000 7fd27cb0ee40 20 remove_watcher() i=6 2023-10-05T19:28:33.544 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.538+0000 7fd27cb0ee40 1 -- 172.21.15.167:0/403903602 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:121 7.9 7:93e5b521:::notify.7:head [watch unwatch cookie 94594747580608] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x56088e5128b0 con 0x7fd264007030 2023-10-05T19:28:33.547 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.542+0000 7fd27ae44700 1 -- 172.21.15.167:0/403903602 <== osd.0 v2:172.21.15.118:6800/27690 39 ==== osd_op_reply(121 notify.7 [watch unwatch cookie 94594747580608] v30'122 uv120 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7fd26c050040 con 0x7fd264007030 2023-10-05T19:28:33.547 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.542+0000 7fd27cb0ee40 20 remove_watcher() i=7 2023-10-05T19:28:33.549 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.542+0000 7fd24bfdf700 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2023-10-05T19:28:33.549 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.542+0000 7fd27cb0ee40 1 -- 172.21.15.167:0/403903602 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:122 6.4 6:22d26bf9:::data_loggenerations_metadata:head [watch unwatch cookie 94594747586944] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x56088e5128b0 con 0x56088e4fb7f0 2023-10-05T19:28:33.553 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.546+0000 7fd27ae44700 1 -- 172.21.15.167:0/403903602 <== osd.1 v2:172.21.15.167:6800/30035 37 ==== osd_op_reply(122 data_loggenerations_metadata [watch unwatch cookie 94594747586944] v30'119 uv1 ondisk = 0) v8 ==== 172+0+0 (crc 0 0 0) 0x7fd26c050040 con 0x56088e4fb7f0 2023-10-05T19:28:33.553 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.546+0000 7fd27cb0ee40 1 -- 172.21.15.167:0/403903602 >> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] conn(0x7fd264007030 msgr2=0x7fd26401d940 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-05T19:28:33.553 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.546+0000 7fd27cb0ee40 1 --2- 172.21.15.167:0/403903602 >> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] conn(0x7fd264007030 0x7fd26401d940 crc :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-05T19:28:33.553 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.546+0000 7fd27cb0ee40 1 -- 172.21.15.167:0/403903602 >> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] conn(0x56088e4fb7f0 msgr2=0x56088e52e7e0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-05T19:28:33.553 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.546+0000 7fd27cb0ee40 1 --2- 172.21.15.167:0/403903602 >> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] conn(0x56088e4fb7f0 0x56088e52e7e0 crc :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-05T19:28:33.553 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.546+0000 7fd27cb0ee40 1 -- 172.21.15.167:0/403903602 >> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] conn(0x7fd274027be0 msgr2=0x7fd27401d7b0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-05T19:28:33.553 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.546+0000 7fd27cb0ee40 1 --2- 172.21.15.167:0/403903602 >> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] conn(0x7fd274027be0 0x7fd27401d7b0 crc :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-05T19:28:33.553 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.546+0000 7fd27cb0ee40 1 -- 172.21.15.167:0/403903602 >> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] conn(0x56088e532470 msgr2=0x56088e534880 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-05T19:28:33.553 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.546+0000 7fd27cb0ee40 1 --2- 172.21.15.167:0/403903602 >> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] conn(0x56088e532470 0x56088e534880 crc :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-05T19:28:33.553 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.546+0000 7fd27cb0ee40 1 -- 172.21.15.167:0/403903602 >> [v2:172.21.15.167:6824/29889,v1:172.21.15.167:6825/29889] conn(0x7fd25c030c40 msgr2=0x7fd25c0330f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-05T19:28:33.554 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.546+0000 7fd27cb0ee40 1 --2- 172.21.15.167:0/403903602 >> [v2:172.21.15.167:6824/29889,v1:172.21.15.167:6825/29889] conn(0x7fd25c030c40 0x7fd25c0330f0 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7fd2640030c0 tx=0x7fd264014000 comp rx=0 tx=0).stop 2023-10-05T19:28:33.554 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.546+0000 7fd27cb0ee40 1 -- 172.21.15.167:0/403903602 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x56088e4fb420 msgr2=0x56088e522d20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-05T19:28:33.554 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.546+0000 7fd27ae44700 1 -- 172.21.15.167:0/403903602 reap_dead start 2023-10-05T19:28:33.554 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.546+0000 7fd27cb0ee40 1 --2- 172.21.15.167:0/403903602 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x56088e4fb420 0x56088e522d20 secure :-1 s=READY pgs=377 cs=0 l=1 rev1=1 crypto rx=0x7fd274001530 tx=0x7fd274010000 comp rx=0 tx=0).stop 2023-10-05T19:28:33.554 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.546+0000 7fd27cb0ee40 1 -- 172.21.15.167:0/403903602 shutdown_connections 2023-10-05T19:28:33.554 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.546+0000 7fd27cb0ee40 1 --2- 172.21.15.167:0/403903602 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x56088e4fb420 0x56088e522d20 unknown :-1 s=CLOSED pgs=377 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-05T19:28:33.554 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.546+0000 7fd27cb0ee40 1 -- 172.21.15.167:0/403903602 >> 172.21.15.167:0/403903602 conn(0x56088e4f5730 msgr2=0x56088e5165d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-05T19:28:33.554 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.546+0000 7fd27cb0ee40 1 -- 172.21.15.167:0/403903602 shutdown_connections 2023-10-05T19:28:33.555 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.546+0000 7fd27cb0ee40 1 -- 172.21.15.167:0/403903602 wait complete. 2023-10-05T19:28:33.555 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.546+0000 7fd27cb0ee40 1 -- 172.21.15.167:0/3725797096 >> [v2:172.21.15.167:6824/29889,v1:172.21.15.167:6825/29889] conn(0x7fd1c0030c50 msgr2=0x7fd1c0033100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-05T19:28:33.556 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.546+0000 7fd27cb0ee40 1 --2- 172.21.15.167:0/3725797096 >> [v2:172.21.15.167:6824/29889,v1:172.21.15.167:6825/29889] conn(0x7fd1c0030c50 0x7fd1c0033100 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x56088e53d750 tx=0x7fd26c000f90 comp rx=0 tx=0).stop 2023-10-05T19:28:33.556 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.546+0000 7fd27cb0ee40 1 -- 172.21.15.167:0/3725797096 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x56088e54b370 msgr2=0x56088e550e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-05T19:28:33.556 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.546+0000 7fd27cb0ee40 1 --2- 172.21.15.167:0/3725797096 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x56088e54b370 0x56088e550e10 secure :-1 s=READY pgs=379 cs=0 l=1 rev1=1 crypto rx=0x56088e5453f0 tx=0x7fd264038000 comp rx=0 tx=0).stop 2023-10-05T19:28:33.556 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.546+0000 7fd27cb0ee40 1 -- 172.21.15.167:0/3725797096 shutdown_connections 2023-10-05T19:28:33.556 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.546+0000 7fd27cb0ee40 1 --2- 172.21.15.167:0/3725797096 >> [v2:172.21.15.167:6824/29889,v1:172.21.15.167:6825/29889] conn(0x7fd1c0030c50 0x7fd1c0033100 unknown :-1 s=CLOSED pgs=133 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-05T19:28:33.556 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.546+0000 7fd27cb0ee40 1 --2- 172.21.15.167:0/3725797096 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x56088e54b370 0x56088e550e10 unknown :-1 s=CLOSED pgs=379 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-05T19:28:33.556 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.546+0000 7fd27cb0ee40 1 -- 172.21.15.167:0/3725797096 >> 172.21.15.167:0/3725797096 conn(0x56088e546730 msgr2=0x56088e5447d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-05T19:28:33.556 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.546+0000 7fd27cb0ee40 1 -- 172.21.15.167:0/3725797096 shutdown_connections 2023-10-05T19:28:33.556 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.550+0000 7fd27cb0ee40 1 -- 172.21.15.167:0/3725797096 wait complete. 2023-10-05T19:28:33.559 INFO:tasks.util.rgw:rgwadmin: client.0 : ['bucket', 'link', '--uid', 'foo', '--bucket', 'myfoo', '--bucket-id', '9a9df965-6612-45d4-a45f-6e3c889bacc2.4223.1'] 2023-10-05T19:28:33.559 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'ceph', 'bucket', 'link', '--uid', 'foo', '--bucket', 'myfoo', '--bucket-id', '9a9df965-6612-45d4-a45f-6e3c889bacc2.4223.1'] 2023-10-05T19:28:33.560 DEBUG:teuthology.orchestra.run.smithi167:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster ceph bucket link --uid foo --bucket myfoo --bucket-id 9a9df965-6612-45d4-a45f-6e3c889bacc2.4223.1 2023-10-05T19:28:33.644 INFO:teuthology.orchestra.run.smithi167.stderr:ignoring --setuser ceph since I am not root 2023-10-05T19:28:33.644 INFO:teuthology.orchestra.run.smithi167.stderr:ignoring --setgroup ceph since I am not root 2023-10-05T19:28:33.645 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.638+0000 7fce0a7cfe40 1 Processor -- start 2023-10-05T19:28:33.645 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.638+0000 7fce0a7cfe40 1 -- start start 2023-10-05T19:28:33.645 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.638+0000 7fce0a7cfe40 1 --2- >> v2:172.21.15.118:3300/0 conn(0x5638826162a0 0x563882618680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-05T19:28:33.645 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.638+0000 7fce0a7cfe40 1 -- --> v1:172.21.15.118:6789/0 -- auth(proto 0 26 bytes epoch 0) v1 -- 0x56388255fbd0 con 0x563882611cc0 2023-10-05T19:28:33.646 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.638+0000 7fce0a7cfe40 1 -- --> v2:172.21.15.118:3300/0 -- mon_getmap magic: 0 v1 -- 0x56388251cee0 con 0x5638826162a0 2023-10-05T19:28:33.647 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.638+0000 7fce07b03700 1 --2- >> v2:172.21.15.118:3300/0 conn(0x5638826162a0 0x563882618680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-05T19:28:33.647 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.638+0000 7fce08304700 1 --1- >> v1:172.21.15.118:6789/0 conn(0x563882611cc0 0x563882612090 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.118:6789/0 says I am v1:172.21.15.167:56674/0 (socket says 172.21.15.167:56674) 2023-10-05T19:28:33.647 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.642+0000 7fce08304700 1 -- 172.21.15.167:0/2353347329 learned_addr learned my addr 172.21.15.167:0/2353347329 (peer_addr_for_me v1:172.21.15.167:0/0) 2023-10-05T19:28:33.647 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.642+0000 7fce07302700 1 -- 172.21.15.167:0/2353347329 <== mon.0 v1:172.21.15.118:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4259275303 0 0) 0x56388255fbd0 con 0x563882611cc0 2023-10-05T19:28:33.647 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.642+0000 7fce07302700 1 -- 172.21.15.167:0/2353347329 --> v1:172.21.15.118:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fcdf0001e10 con 0x563882611cc0 2023-10-05T19:28:33.647 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.642+0000 7fce07b03700 1 -- 172.21.15.167:0/2353347329 >> v1:172.21.15.118:6789/0 conn(0x563882611cc0 legacy=0x563882612090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-05T19:28:33.647 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.642+0000 7fce07b03700 1 -- 172.21.15.167:0/2353347329 --> v2:172.21.15.118:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5638826128d0 con 0x5638826162a0 2023-10-05T19:28:33.648 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.642+0000 7fce07b03700 1 --2- 172.21.15.167:0/2353347329 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5638826162a0 0x563882618680 secure :-1 s=READY pgs=381 cs=0 l=1 rev1=1 crypto rx=0x7fcdfc00a5d0 tx=0x7fcdfc005a50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b160d7cbd46e881b server_cookie=0 in_seq=0 out_seq=0 2023-10-05T19:28:33.648 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.642+0000 7fce07302700 1 -- 172.21.15.167:0/2353347329 <== mon.0 v2:172.21.15.118:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fcdfc007610 con 0x5638826162a0 2023-10-05T19:28:33.648 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.642+0000 7fce07302700 1 -- 172.21.15.167:0/2353347329 <== mon.0 v2:172.21.15.118:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcdfc007770 con 0x5638826162a0 2023-10-05T19:28:33.649 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.642+0000 7fce07302700 1 -- 172.21.15.167:0/2353347329 <== mon.0 v2:172.21.15.118:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fcdfc007a40 con 0x5638826162a0 2023-10-05T19:28:33.649 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.642+0000 7fce0a7cfe40 1 -- 172.21.15.167:0/2353347329 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5638826162a0 msgr2=0x563882618680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-05T19:28:33.649 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.642+0000 7fce0a7cfe40 1 --2- 172.21.15.167:0/2353347329 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5638826162a0 0x563882618680 secure :-1 s=READY pgs=381 cs=0 l=1 rev1=1 crypto rx=0x7fcdfc00a5d0 tx=0x7fcdfc005a50 comp rx=0 tx=0).stop 2023-10-05T19:28:33.649 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.642+0000 7fce0a7cfe40 1 -- 172.21.15.167:0/2353347329 shutdown_connections 2023-10-05T19:28:33.649 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.642+0000 7fce0a7cfe40 1 --2- 172.21.15.167:0/2353347329 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5638826162a0 0x563882618680 unknown :-1 s=CLOSED pgs=381 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-05T19:28:33.649 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.642+0000 7fce0a7cfe40 1 -- 172.21.15.167:0/2353347329 >> 172.21.15.167:0/2353347329 conn(0x56388260d6e0 msgr2=0x56388260faf0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-05T19:28:33.649 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.642+0000 7fce0a7cfe40 1 -- 172.21.15.167:0/2353347329 shutdown_connections 2023-10-05T19:28:33.649 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.642+0000 7fce0a7cfe40 1 -- 172.21.15.167:0/2353347329 wait complete. 2023-10-05T19:28:33.651 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.642+0000 7fce0a7cfe40 1 Processor -- start 2023-10-05T19:28:33.651 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.642+0000 7fce0a7cfe40 1 -- start start 2023-10-05T19:28:33.651 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.642+0000 7fce0a7cfe40 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x563882611cc0 0x56388261b390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-05T19:28:33.651 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.642+0000 7fce0a7cfe40 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x5638824e0e30 con 0x563882611cc0 2023-10-05T19:28:33.652 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.642+0000 7fce08304700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x563882611cc0 0x56388261b390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-05T19:28:33.652 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.642+0000 7fce08304700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x563882611cc0 0x56388261b390 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.118:3300/0 says I am v2:172.21.15.167:58640/0 (socket says 172.21.15.167:58640) 2023-10-05T19:28:33.652 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.642+0000 7fce08304700 1 -- 172.21.15.167:0/927253452 learned_addr learned my addr 172.21.15.167:0/927253452 (peer_addr_for_me v2:172.21.15.167:0/0) 2023-10-05T19:28:33.652 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.646+0000 7fce08304700 1 -- 172.21.15.167:0/927253452 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcdfc005cf0 con 0x563882611cc0 2023-10-05T19:28:33.652 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.646+0000 7fce08304700 1 --2- 172.21.15.167:0/927253452 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x563882611cc0 0x56388261b390 secure :-1 s=READY pgs=382 cs=0 l=1 rev1=1 crypto rx=0x7fce00006e70 tx=0x7fce00006a20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9c9938e7ac6b973d server_cookie=0 in_seq=0 out_seq=0 2023-10-05T19:28:33.652 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.646+0000 7fce06300700 1 -- 172.21.15.167:0/927253452 <== mon.0 v2:172.21.15.118:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fce00006110 con 0x563882611cc0 2023-10-05T19:28:33.652 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.646+0000 7fce06300700 1 -- 172.21.15.167:0/927253452 <== mon.0 v2:172.21.15.118:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fce00006290 con 0x563882611cc0 2023-10-05T19:28:33.652 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.646+0000 7fce06300700 1 -- 172.21.15.167:0/927253452 <== mon.0 v2:172.21.15.118:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fce00004520 con 0x563882611cc0 2023-10-05T19:28:33.654 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.646+0000 7fce0a7cfe40 1 -- 172.21.15.167:0/927253452 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x563882611cc0 msgr2=0x56388261b390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-05T19:28:33.654 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.646+0000 7fce0a7cfe40 1 --2- 172.21.15.167:0/927253452 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x563882611cc0 0x56388261b390 secure :-1 s=READY pgs=382 cs=0 l=1 rev1=1 crypto rx=0x7fce00006e70 tx=0x7fce00006a20 comp rx=0 tx=0).stop 2023-10-05T19:28:33.654 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.646+0000 7fce0a7cfe40 1 -- 172.21.15.167:0/927253452 shutdown_connections 2023-10-05T19:28:33.655 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.646+0000 7fce0a7cfe40 1 --2- 172.21.15.167:0/927253452 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x563882611cc0 0x56388261b390 unknown :-1 s=CLOSED pgs=382 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-05T19:28:33.655 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.646+0000 7fce0a7cfe40 1 -- 172.21.15.167:0/927253452 >> 172.21.15.167:0/927253452 conn(0x56388260d6e0 msgr2=0x56388262d950 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-05T19:28:33.655 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.646+0000 7fce0a7cfe40 1 -- 172.21.15.167:0/927253452 shutdown_connections 2023-10-05T19:28:33.655 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.646+0000 7fce0a7cfe40 1 -- 172.21.15.167:0/927253452 wait complete. 2023-10-05T19:28:33.655 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.646+0000 7fce0a7cfe40 1 Processor -- start 2023-10-05T19:28:33.655 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.646+0000 7fce0a7cfe40 1 -- start start 2023-10-05T19:28:33.655 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.646+0000 7fce0a7cfe40 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x563882611cc0 0x56388263ae30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-05T19:28:33.655 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.646+0000 7fce0a7cfe40 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x5638824b3ed0 con 0x563882611cc0 2023-10-05T19:28:33.655 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.646+0000 7fce08304700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x563882611cc0 0x56388263ae30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-05T19:28:33.655 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.646+0000 7fce08304700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x563882611cc0 0x56388263ae30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.118:3300/0 says I am v2:172.21.15.167:58654/0 (socket says 172.21.15.167:58654) 2023-10-05T19:28:33.655 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.646+0000 7fce08304700 1 -- 172.21.15.167:0/2192230658 learned_addr learned my addr 172.21.15.167:0/2192230658 (peer_addr_for_me v2:172.21.15.167:0/0) 2023-10-05T19:28:33.655 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.646+0000 7fce08304700 1 -- 172.21.15.167:0/2192230658 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcdf0001e10 con 0x563882611cc0 2023-10-05T19:28:33.656 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.646+0000 7fce08304700 1 --2- 172.21.15.167:0/2192230658 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x563882611cc0 0x56388263ae30 secure :-1 s=READY pgs=383 cs=0 l=1 rev1=1 crypto rx=0x7fce00001350 tx=0x7fce00014000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-05T19:28:33.656 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.646+0000 7fce04afd700 1 -- 172.21.15.167:0/2192230658 <== mon.0 v2:172.21.15.118:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fce00006110 con 0x563882611cc0 2023-10-05T19:28:33.656 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.650+0000 7fce04afd700 1 -- 172.21.15.167:0/2192230658 <== mon.0 v2:172.21.15.118:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fce00006400 con 0x563882611cc0 2023-10-05T19:28:33.656 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.650+0000 7fce0a7cfe40 1 -- 172.21.15.167:0/2192230658 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x56388263b370 con 0x563882611cc0 2023-10-05T19:28:33.656 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.650+0000 7fce04afd700 1 -- 172.21.15.167:0/2192230658 <== mon.0 v2:172.21.15.118:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fce00013970 con 0x563882611cc0 2023-10-05T19:28:33.656 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.650+0000 7fce0a7cfe40 1 -- 172.21.15.167:0/2192230658 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x56388263b7b0 con 0x563882611cc0 2023-10-05T19:28:33.657 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.650+0000 7fce04afd700 1 -- 172.21.15.167:0/2192230658 <== mon.0 v2:172.21.15.118:3300/0 4 ==== mgrmap(e 5) v1 ==== 40329+0+0 (secure 0 0 0) 0x7fce0001a070 con 0x563882611cc0 2023-10-05T19:28:33.657 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.650+0000 7fce0a7cfe40 20 rados->read ofs=0 len=0 2023-10-05T19:28:33.657 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.650+0000 7fce04afd700 1 --2- 172.21.15.167:0/2192230658 >> [v2:172.21.15.167:6824/29889,v1:172.21.15.167:6825/29889] conn(0x7fcdf4030c50 0x7fcdf4033100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-05T19:28:33.657 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.650+0000 7fce04afd700 1 -- 172.21.15.167:0/2192230658 <== mon.0 v2:172.21.15.118:3300/0 5 ==== osd_map(30..30 src has 1..30) v4 ==== 6057+0+0 (secure 0 0 0) 0x7fce00044cc0 con 0x563882611cc0 2023-10-05T19:28:33.657 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.650+0000 7fce07b03700 1 --2- 172.21.15.167:0/2192230658 >> [v2:172.21.15.167:6824/29889,v1:172.21.15.167:6825/29889] conn(0x7fcdf4030c50 0x7fcdf4033100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-05T19:28:33.657 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.650+0000 7fce0a7cfe40 1 --2- 172.21.15.167:0/2192230658 >> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] conn(0x5638826162a0 0x5638826468f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-05T19:28:33.658 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.650+0000 7fce0a7cfe40 1 -- 172.21.15.167:0/2192230658 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:1 5.12 5:49953fa1:::default.realm:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x563882646ff0 con 0x5638826162a0 2023-10-05T19:28:33.658 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.650+0000 7fce08b05700 1 --2- 172.21.15.167:0/2192230658 >> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] conn(0x5638826162a0 0x5638826468f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-05T19:28:33.658 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.650+0000 7fce07b03700 1 --2- 172.21.15.167:0/2192230658 >> [v2:172.21.15.167:6824/29889,v1:172.21.15.167:6825/29889] conn(0x7fcdf4030c50 0x7fcdf4033100 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7fcdfc00afd0 tx=0x7fcdfc014000 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-05T19:28:33.659 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.650+0000 7fce08b05700 1 --2- 172.21.15.167:0/2192230658 >> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] conn(0x5638826162a0 0x5638826468f0 crc :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-05T19:28:33.659 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.654+0000 7fce08b05700 1 -- 172.21.15.167:0/2192230658 <== osd.1 v2:172.21.15.167:6800/30035 1 ==== osd_op_reply(1 default.realm [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 ==== 157+0+0 (crc 0 0 0) 0x7fcdf800a500 con 0x5638826162a0 2023-10-05T19:28:33.659 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.654+0000 7fce0a7cfe40 20 rados_obj.operate() r=-2 bl.length=0 2023-10-05T19:28:33.660 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.654+0000 7fce0a7cfe40 20 realm 2023-10-05T19:28:33.660 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.654+0000 7fce0a7cfe40 20 rados->read ofs=0 len=0 2023-10-05T19:28:33.660 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.654+0000 7fce0a7cfe40 1 -- 172.21.15.167:0/2192230658 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:2 5.12 5:49953fa1:::default.realm:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x563882643f50 con 0x5638826162a0 2023-10-05T19:28:33.660 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.654+0000 7fce08b05700 1 -- 172.21.15.167:0/2192230658 <== osd.1 v2:172.21.15.167:6800/30035 2 ==== osd_op_reply(2 default.realm [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 ==== 157+0+0 (crc 0 0 0) 0x7fcdf800a500 con 0x5638826162a0 2023-10-05T19:28:33.660 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.654+0000 7fce0a7cfe40 20 rados_obj.operate() r=-2 bl.length=0 2023-10-05T19:28:33.660 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.654+0000 7fce0a7cfe40 4 RGWPeriod::init failed to init realm id : (2) No such file or directory 2023-10-05T19:28:33.660 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.654+0000 7fce0a7cfe40 20 rados->read ofs=0 len=0 2023-10-05T19:28:33.660 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.654+0000 7fce0a7cfe40 1 -- 172.21.15.167:0/2192230658 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:3 5.12 5:49953fa1:::default.realm:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x563882647ad0 con 0x5638826162a0 2023-10-05T19:28:33.660 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.654+0000 7fce08b05700 1 -- 172.21.15.167:0/2192230658 <== osd.1 v2:172.21.15.167:6800/30035 3 ==== osd_op_reply(3 default.realm [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 ==== 157+0+0 (crc 0 0 0) 0x7fcdf800a500 con 0x5638826162a0 2023-10-05T19:28:33.660 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.654+0000 7fce0a7cfe40 20 rados_obj.operate() r=-2 bl.length=0 2023-10-05T19:28:33.660 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.654+0000 7fce0a7cfe40 20 rados->read ofs=0 len=0 2023-10-05T19:28:33.661 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.654+0000 7fce0a7cfe40 1 -- 172.21.15.167:0/2192230658 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:4 5.f 5:f43b6ece:::zone_names.default:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x5638826476e0 con 0x5638826162a0 2023-10-05T19:28:33.661 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.654+0000 7fce08b05700 1 -- 172.21.15.167:0/2192230658 <== osd.1 v2:172.21.15.167:6800/30035 4 ==== osd_op_reply(4 zone_names.default [read 0~46 out=46b] v0'0 uv1 ondisk = 0) v8 ==== 162+0+46 (crc 0 0 0) 0x7fcdf800a500 con 0x5638826162a0 2023-10-05T19:28:33.661 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.654+0000 7fce0a7cfe40 20 rados_obj.operate() r=0 bl.length=46 2023-10-05T19:28:33.661 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.654+0000 7fce0a7cfe40 20 rados->read ofs=0 len=0 2023-10-05T19:28:33.661 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.654+0000 7fce0a7cfe40 1 -- 172.21.15.167:0/2192230658 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:5 5.1d 5:b8374eb4:::zone_info.9a9df965-6612-45d4-a45f-6e3c889bacc2:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x563882648960 con 0x5638826162a0 2023-10-05T19:28:33.662 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.654+0000 7fce08b05700 1 -- 172.21.15.167:0/2192230658 <== osd.1 v2:172.21.15.167:6800/30035 5 ==== osd_op_reply(5 zone_info.9a9df965-6612-45d4-a45f-6e3c889bacc2 [read 0~874 out=874b] v0'0 uv2 ondisk = 0) v8 ==== 190+0+874 (crc 0 0 0) 0x7fcdf800a500 con 0x5638826162a0 2023-10-05T19:28:33.662 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.654+0000 7fce0a7cfe40 20 rados_obj.operate() r=0 bl.length=874 2023-10-05T19:28:33.662 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.654+0000 7fce0a7cfe40 20 searching for the correct realm 2023-10-05T19:28:33.662 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.654+0000 7fce0a7cfe40 1 --2- 172.21.15.167:0/2192230658 >> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] conn(0x56388264a720 0x56388264cb00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-05T19:28:33.662 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.654+0000 7fce0a7cfe40 1 -- 172.21.15.167:0/2192230658 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:6 5.0 5:00000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x56388264d040 con 0x56388264a720 2023-10-05T19:28:33.662 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.654+0000 7fce08304700 1 --2- 172.21.15.167:0/2192230658 >> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] conn(0x56388264a720 0x56388264cb00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-05T19:28:33.662 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.654+0000 7fce08304700 1 --2- 172.21.15.167:0/2192230658 >> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] conn(0x56388264a720 0x56388264cb00 crc :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-05T19:28:33.665 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.658+0000 7fce08304700 1 -- 172.21.15.167:0/2192230658 <== osd.3 v2:172.21.15.167:6808/30036 1 ==== osd_op_reply(6 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fce00025020 con 0x56388264a720 2023-10-05T19:28:33.666 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.658+0000 7fce08304700 1 --2- 172.21.15.167:0/2192230658 >> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] conn(0x7fce0001fc20 0x7fce0002a000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-05T19:28:33.666 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.658+0000 7fce08304700 1 -- 172.21.15.167:0/2192230658 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:7 5.10 5:08000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x56388264d040 con 0x7fce0001fc20 2023-10-05T19:28:33.666 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.658+0000 7fce07b03700 1 --2- 172.21.15.167:0/2192230658 >> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] conn(0x7fce0001fc20 0x7fce0002a000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-05T19:28:33.666 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.658+0000 7fce07b03700 1 --2- 172.21.15.167:0/2192230658 >> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] conn(0x7fce0001fc20 0x7fce0002a000 crc :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-05T19:28:33.666 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.658+0000 7fce07b03700 1 -- 172.21.15.167:0/2192230658 <== osd.2 v2:172.21.15.167:6816/30037 1 ==== osd_op_reply(7 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fcdfc007da0 con 0x7fce0001fc20 2023-10-05T19:28:33.666 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.658+0000 7fce07b03700 1 --2- 172.21.15.167:0/2192230658 >> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] conn(0x7fcdfc007030 0x7fcdfc01d430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-05T19:28:33.666 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.658+0000 7fce07b03700 1 -- 172.21.15.167:0/2192230658 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:8 5.8 5:10000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x56388264d040 con 0x7fcdfc007030 2023-10-05T19:28:33.666 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.658+0000 7fce08b05700 1 --2- 172.21.15.167:0/2192230658 >> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] conn(0x7fcdfc007030 0x7fcdfc01d430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-05T19:28:33.666 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.658+0000 7fce08b05700 1 --2- 172.21.15.167:0/2192230658 >> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] conn(0x7fcdfc007030 0x7fcdfc01d430 crc :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-05T19:28:33.667 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.662+0000 7fce08b05700 1 -- 172.21.15.167:0/2192230658 <== osd.0 v2:172.21.15.118:6800/27690 1 ==== osd_op_reply(8 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fcdf8014040 con 0x7fcdfc007030 2023-10-05T19:28:33.667 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.662+0000 7fce08b05700 1 -- 172.21.15.167:0/2192230658 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:9 5.18 5:18000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x56388264d040 con 0x5638826162a0 2023-10-05T19:28:33.667 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.662+0000 7fce08b05700 1 -- 172.21.15.167:0/2192230658 <== osd.1 v2:172.21.15.167:6800/30035 6 ==== osd_op_reply(9 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fcdf8014040 con 0x5638826162a0 2023-10-05T19:28:33.667 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.662+0000 7fce08b05700 1 -- 172.21.15.167:0/2192230658 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:10 5.4 5:20000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x56388264d040 con 0x5638826162a0 2023-10-05T19:28:33.668 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.662+0000 7fce08b05700 1 -- 172.21.15.167:0/2192230658 <== osd.1 v2:172.21.15.167:6800/30035 7 ==== osd_op_reply(10 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fcdf8014040 con 0x5638826162a0 2023-10-05T19:28:33.668 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.662+0000 7fce08b05700 1 -- 172.21.15.167:0/2192230658 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:11 5.14 5:28000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x56388264d040 con 0x7fcdfc007030 2023-10-05T19:28:33.668 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.662+0000 7fce08b05700 1 -- 172.21.15.167:0/2192230658 <== osd.0 v2:172.21.15.118:6800/27690 2 ==== osd_op_reply(11 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fcdf8014040 con 0x7fcdfc007030 2023-10-05T19:28:33.668 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.662+0000 7fce08b05700 1 -- 172.21.15.167:0/2192230658 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:12 5.c 5:30000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x56388264d040 con 0x7fcdfc007030 2023-10-05T19:28:33.669 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.662+0000 7fce08b05700 1 -- 172.21.15.167:0/2192230658 <== osd.0 v2:172.21.15.118:6800/27690 3 ==== osd_op_reply(12 [pgnls start_epoch 30 out=112b] v23'1 uv1 ondisk = 1) v8 ==== 144+0+112 (crc 0 0 0) 0x7fcdf8014040 con 0x7fcdfc007030 2023-10-05T19:28:33.669 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.662+0000 7fce08b05700 1 -- 172.21.15.167:0/2192230658 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:13 5.1c 5:38000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x56388264d040 con 0x7fce0001fc20 2023-10-05T19:28:33.669 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.662+0000 7fce07b03700 1 -- 172.21.15.167:0/2192230658 <== osd.2 v2:172.21.15.167:6816/30037 2 ==== osd_op_reply(13 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fcdfc029040 con 0x7fce0001fc20 2023-10-05T19:28:33.670 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.662+0000 7fce07b03700 1 -- 172.21.15.167:0/2192230658 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:14 5.2 5:40000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x56388264d040 con 0x56388264a720 2023-10-05T19:28:33.670 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.666+0000 7fce08304700 1 -- 172.21.15.167:0/2192230658 <== osd.3 v2:172.21.15.167:6808/30036 2 ==== osd_op_reply(14 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fce00025020 con 0x56388264a720 2023-10-05T19:28:33.670 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.666+0000 7fce08304700 1 -- 172.21.15.167:0/2192230658 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:15 5.12 5:48000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x56388264d040 con 0x5638826162a0 2023-10-05T19:28:33.671 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.666+0000 7fce08b05700 1 -- 172.21.15.167:0/2192230658 <== osd.1 v2:172.21.15.167:6800/30035 8 ==== osd_op_reply(15 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fcdf8014040 con 0x5638826162a0 2023-10-05T19:28:33.671 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.666+0000 7fce08b05700 1 -- 172.21.15.167:0/2192230658 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:16 5.a 5:50000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x56388264d040 con 0x7fcdfc007030 2023-10-05T19:28:33.672 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.666+0000 7fce08b05700 1 -- 172.21.15.167:0/2192230658 <== osd.0 v2:172.21.15.118:6800/27690 4 ==== osd_op_reply(16 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fcdf8022040 con 0x7fcdfc007030 2023-10-05T19:28:33.672 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.666+0000 7fce08b05700 1 -- 172.21.15.167:0/2192230658 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:17 5.1a 5:58000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x56388264d040 con 0x5638826162a0 2023-10-05T19:28:33.673 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.666+0000 7fce08b05700 1 -- 172.21.15.167:0/2192230658 <== osd.1 v2:172.21.15.167:6800/30035 9 ==== osd_op_reply(17 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fcdf8022040 con 0x5638826162a0 2023-10-05T19:28:33.673 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.666+0000 7fce08b05700 1 -- 172.21.15.167:0/2192230658 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:18 5.6 5:60000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x56388264d040 con 0x7fce0001fc20 2023-10-05T19:28:33.673 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.666+0000 7fce07b03700 1 -- 172.21.15.167:0/2192230658 <== osd.2 v2:172.21.15.167:6816/30037 3 ==== osd_op_reply(18 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fcdfc029040 con 0x7fce0001fc20 2023-10-05T19:28:33.673 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.666+0000 7fce07b03700 1 -- 172.21.15.167:0/2192230658 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:19 5.16 5:68000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x56388264d040 con 0x56388264a720 2023-10-05T19:28:33.674 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.670+0000 7fce08304700 1 -- 172.21.15.167:0/2192230658 <== osd.3 v2:172.21.15.167:6808/30036 3 ==== osd_op_reply(19 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fce00025020 con 0x56388264a720 2023-10-05T19:28:33.674 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.670+0000 7fce08304700 1 -- 172.21.15.167:0/2192230658 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:20 5.e 5:70000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x56388264d040 con 0x56388264a720 2023-10-05T19:28:33.675 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.670+0000 7fce08304700 1 -- 172.21.15.167:0/2192230658 <== osd.3 v2:172.21.15.167:6808/30036 4 ==== osd_op_reply(20 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fce00025020 con 0x56388264a720 2023-10-05T19:28:33.675 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.670+0000 7fce08304700 1 -- 172.21.15.167:0/2192230658 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:21 5.1e 5:78000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x56388264d040 con 0x7fcdfc007030 2023-10-05T19:28:33.676 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.670+0000 7fce08b05700 1 -- 172.21.15.167:0/2192230658 <== osd.0 v2:172.21.15.118:6800/27690 5 ==== osd_op_reply(21 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fcdf8022040 con 0x7fcdfc007030 2023-10-05T19:28:33.676 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.670+0000 7fce08b05700 1 -- 172.21.15.167:0/2192230658 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:22 5.1 5:80000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x56388264d040 con 0x56388264a720 2023-10-05T19:28:33.677 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.670+0000 7fce08304700 1 -- 172.21.15.167:0/2192230658 <== osd.3 v2:172.21.15.167:6808/30036 5 ==== osd_op_reply(22 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fce00025020 con 0x56388264a720 2023-10-05T19:28:33.677 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.670+0000 7fce08304700 1 -- 172.21.15.167:0/2192230658 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:23 5.11 5:88000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x56388264d040 con 0x56388264a720 2023-10-05T19:28:33.677 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.670+0000 7fce08304700 1 -- 172.21.15.167:0/2192230658 <== osd.3 v2:172.21.15.167:6808/30036 6 ==== osd_op_reply(23 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fce00025020 con 0x56388264a720 2023-10-05T19:28:33.677 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.670+0000 7fce08304700 1 -- 172.21.15.167:0/2192230658 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:24 5.9 5:90000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x56388264d040 con 0x5638826162a0 2023-10-05T19:28:33.678 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.674+0000 7fce08b05700 1 -- 172.21.15.167:0/2192230658 <== osd.1 v2:172.21.15.167:6800/30035 10 ==== osd_op_reply(24 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fcdf8022040 con 0x5638826162a0 2023-10-05T19:28:33.678 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.674+0000 7fce08b05700 1 -- 172.21.15.167:0/2192230658 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:25 5.19 5:98000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x56388264d040 con 0x5638826162a0 2023-10-05T19:28:33.679 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.674+0000 7fce08b05700 1 -- 172.21.15.167:0/2192230658 <== osd.1 v2:172.21.15.167:6800/30035 11 ==== osd_op_reply(25 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fcdf8022040 con 0x5638826162a0 2023-10-05T19:28:33.679 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.674+0000 7fce08b05700 1 -- 172.21.15.167:0/2192230658 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:26 5.5 5:a0000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x56388264d040 con 0x7fcdfc007030 2023-10-05T19:28:33.680 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.674+0000 7fce08b05700 1 -- 172.21.15.167:0/2192230658 <== osd.0 v2:172.21.15.118:6800/27690 6 ==== osd_op_reply(26 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fcdf8022040 con 0x7fcdfc007030 2023-10-05T19:28:33.680 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.674+0000 7fce08b05700 1 -- 172.21.15.167:0/2192230658 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:27 5.15 5:a8000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x56388264d040 con 0x56388264a720 2023-10-05T19:28:33.681 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.674+0000 7fce08304700 1 -- 172.21.15.167:0/2192230658 <== osd.3 v2:172.21.15.167:6808/30036 7 ==== osd_op_reply(27 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fce0004a040 con 0x56388264a720 2023-10-05T19:28:33.681 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.674+0000 7fce08304700 1 -- 172.21.15.167:0/2192230658 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:28 5.d 5:b0000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x56388264d040 con 0x7fce0001fc20 2023-10-05T19:28:33.682 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.674+0000 7fce07b03700 1 -- 172.21.15.167:0/2192230658 <== osd.2 v2:172.21.15.167:6816/30037 4 ==== osd_op_reply(28 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fcdfc029040 con 0x7fce0001fc20 2023-10-05T19:28:33.682 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.674+0000 7fce07b03700 1 -- 172.21.15.167:0/2192230658 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:29 5.1d 5:b8000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x56388264d040 con 0x5638826162a0 2023-10-05T19:28:33.682 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.678+0000 7fce08b05700 1 -- 172.21.15.167:0/2192230658 <== osd.1 v2:172.21.15.167:6800/30035 12 ==== osd_op_reply(29 [pgnls start_epoch 30 out=107b] v30'2 uv2 ondisk = 1) v8 ==== 144+0+107 (crc 0 0 0) 0x7fcdf8022040 con 0x5638826162a0 2023-10-05T19:28:33.682 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.678+0000 7fce08b05700 1 -- 172.21.15.167:0/2192230658 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:30 5.3 5:c0000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x56388264d040 con 0x7fcdfc007030 2023-10-05T19:28:33.684 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.678+0000 7fce08b05700 1 -- 172.21.15.167:0/2192230658 <== osd.0 v2:172.21.15.118:6800/27690 7 ==== osd_op_reply(30 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fcdf8022040 con 0x7fcdfc007030 2023-10-05T19:28:33.684 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.678+0000 7fce08b05700 1 -- 172.21.15.167:0/2192230658 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:31 5.13 5:c8000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x56388264d040 con 0x56388264a720 2023-10-05T19:28:33.684 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.678+0000 7fce08304700 1 -- 172.21.15.167:0/2192230658 <== osd.3 v2:172.21.15.167:6808/30036 8 ==== osd_op_reply(31 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fce0004a040 con 0x56388264a720 2023-10-05T19:28:33.684 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.678+0000 7fce08304700 1 -- 172.21.15.167:0/2192230658 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:32 5.b 5:d0000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x56388264d040 con 0x7fcdfc007030 2023-10-05T19:28:33.685 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.678+0000 7fce08b05700 1 -- 172.21.15.167:0/2192230658 <== osd.0 v2:172.21.15.118:6800/27690 8 ==== osd_op_reply(32 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fcdf8022040 con 0x7fcdfc007030 2023-10-05T19:28:33.686 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.678+0000 7fce08b05700 1 -- 172.21.15.167:0/2192230658 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:33 5.1b 5:d8000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x56388264d040 con 0x7fce0001fc20 2023-10-05T19:28:33.686 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.682+0000 7fce07b03700 1 -- 172.21.15.167:0/2192230658 <== osd.2 v2:172.21.15.167:6816/30037 5 ==== osd_op_reply(33 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fcdfc029040 con 0x7fce0001fc20 2023-10-05T19:28:33.686 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.682+0000 7fce07b03700 1 -- 172.21.15.167:0/2192230658 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:34 5.7 5:e0000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x56388264d040 con 0x56388264a720 2023-10-05T19:28:33.687 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.682+0000 7fce08304700 1 -- 172.21.15.167:0/2192230658 <== osd.3 v2:172.21.15.167:6808/30036 9 ==== osd_op_reply(34 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fce0004a040 con 0x56388264a720 2023-10-05T19:28:33.687 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.682+0000 7fce08304700 1 -- 172.21.15.167:0/2192230658 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:35 5.17 5:e8000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x56388264d040 con 0x7fcdfc007030 2023-10-05T19:28:33.688 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.682+0000 7fce08b05700 1 -- 172.21.15.167:0/2192230658 <== osd.0 v2:172.21.15.118:6800/27690 9 ==== osd_op_reply(35 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fcdf8022040 con 0x7fcdfc007030 2023-10-05T19:28:33.688 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.682+0000 7fce08b05700 1 -- 172.21.15.167:0/2192230658 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:36 5.f 5:f0000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x56388264d040 con 0x5638826162a0 2023-10-05T19:28:33.689 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.682+0000 7fce08b05700 1 -- 172.21.15.167:0/2192230658 <== osd.1 v2:172.21.15.167:6800/30035 13 ==== osd_op_reply(36 [pgnls start_epoch 30 out=115b] v23'2 uv2 ondisk = 1) v8 ==== 144+0+115 (crc 0 0 0) 0x7fcdf802f080 con 0x5638826162a0 2023-10-05T19:28:33.689 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.682+0000 7fce08b05700 1 -- 172.21.15.167:0/2192230658 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:37 5.1f 5:f8000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x56388264d040 con 0x7fce0001fc20 2023-10-05T19:28:33.690 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.686+0000 7fce07b03700 1 -- 172.21.15.167:0/2192230658 <== osd.2 v2:172.21.15.167:6816/30037 6 ==== osd_op_reply(37 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fcdfc029040 con 0x7fce0001fc20 2023-10-05T19:28:33.690 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.686+0000 7fce0a7cfe40 20 RGWRados::pool_iterate: got zonegroup_info.4f1f719d-9648-417b-b10c-e5d939825419 2023-10-05T19:28:33.690 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.686+0000 7fce0a7cfe40 20 RGWRados::pool_iterate: got zone_info.9a9df965-6612-45d4-a45f-6e3c889bacc2 2023-10-05T19:28:33.690 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.686+0000 7fce0a7cfe40 20 RGWRados::pool_iterate: got zone_names.default 2023-10-05T19:28:33.690 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.686+0000 7fce0a7cfe40 20 RGWRados::pool_iterate: got zonegroups_names.default 2023-10-05T19:28:33.690 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.686+0000 7fce0a7cfe40 20 rados->read ofs=0 len=0 2023-10-05T19:28:33.690 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.686+0000 7fce0a7cfe40 1 -- 172.21.15.167:0/2192230658 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:38 5.12 5:49953fa1:::default.realm:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x5638826494d0 con 0x5638826162a0 2023-10-05T19:28:33.691 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.686+0000 7fce08b05700 1 -- 172.21.15.167:0/2192230658 <== osd.1 v2:172.21.15.167:6800/30035 14 ==== osd_op_reply(38 default.realm [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 ==== 157+0+0 (crc 0 0 0) 0x7fcdf802f080 con 0x5638826162a0 2023-10-05T19:28:33.691 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.686+0000 7fce0a7cfe40 20 rados_obj.operate() r=-2 bl.length=0 2023-10-05T19:28:33.691 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.686+0000 7fce0a7cfe40 20 rados->read ofs=0 len=0 2023-10-05T19:28:33.692 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.686+0000 7fce0a7cfe40 1 -- 172.21.15.167:0/2192230658 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:39 5.f 5:f4c53578:::zonegroups_names.default:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x563882649dc0 con 0x5638826162a0 2023-10-05T19:28:33.692 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.686+0000 7fce08b05700 1 -- 172.21.15.167:0/2192230658 <== osd.1 v2:172.21.15.167:6800/30035 15 ==== osd_op_reply(39 zonegroups_names.default [read 0~46 out=46b] v0'0 uv2 ondisk = 0) v8 ==== 168+0+46 (crc 0 0 0) 0x7fcdf802f080 con 0x5638826162a0 2023-10-05T19:28:33.692 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.686+0000 7fce0a7cfe40 20 rados_obj.operate() r=0 bl.length=46 2023-10-05T19:28:33.692 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.686+0000 7fce0a7cfe40 20 rados->read ofs=0 len=0 2023-10-05T19:28:33.692 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.686+0000 7fce0a7cfe40 1 -- 172.21.15.167:0/2192230658 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:40 5.c 5:30d299a5:::zonegroup_info.4f1f719d-9648-417b-b10c-e5d939825419:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x56388264d480 con 0x7fcdfc007030 2023-10-05T19:28:33.692 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.686+0000 7fce08b05700 1 -- 172.21.15.167:0/2192230658 <== osd.0 v2:172.21.15.118:6800/27690 10 ==== osd_op_reply(40 zonegroup_info.4f1f719d-9648-417b-b10c-e5d939825419 [read 0~362 out=362b] v0'0 uv1 ondisk = 0) v8 ==== 195+0+362 (crc 0 0 0) 0x7fcdf802f080 con 0x7fcdfc007030 2023-10-05T19:28:33.692 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.686+0000 7fce0a7cfe40 20 rados_obj.operate() r=0 bl.length=362 2023-10-05T19:28:33.693 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.686+0000 7fce0a7cfe40 20 rados->read ofs=0 len=0 2023-10-05T19:28:33.693 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.686+0000 7fce0a7cfe40 1 -- 172.21.15.167:0/2192230658 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:41 5.13 5:cbbf6c60:::converted:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x56388264d480 con 0x56388264a720 2023-10-05T19:28:33.693 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.686+0000 7fce08304700 1 -- 172.21.15.167:0/2192230658 <== osd.3 v2:172.21.15.167:6808/30036 10 ==== osd_op_reply(41 converted [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 ==== 153+0+0 (crc 0 0 0) 0x7fce0004a040 con 0x56388264a720 2023-10-05T19:28:33.693 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.686+0000 7fce0a7cfe40 20 rados_obj.operate() r=-2 bl.length=0 2023-10-05T19:28:33.693 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.686+0000 7fce0a7cfe40 20 rados->read ofs=0 len=0 2023-10-05T19:28:33.693 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.686+0000 7fce0a7cfe40 1 -- 172.21.15.167:0/2192230658 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:42 5.12 5:49953fa1:::default.realm:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x56388264d830 con 0x5638826162a0 2023-10-05T19:28:33.694 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.686+0000 7fce08b05700 1 -- 172.21.15.167:0/2192230658 <== osd.1 v2:172.21.15.167:6800/30035 16 ==== osd_op_reply(42 default.realm [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 ==== 157+0+0 (crc 0 0 0) 0x7fcdf802f080 con 0x5638826162a0 2023-10-05T19:28:33.694 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.690+0000 7fce0a7cfe40 20 rados_obj.operate() r=-2 bl.length=0 2023-10-05T19:28:33.694 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.690+0000 7fce0a7cfe40 20 rados->read ofs=0 len=0 2023-10-05T19:28:33.694 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.690+0000 7fce0a7cfe40 1 -- 172.21.15.167:0/2192230658 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:43 5.f 5:f4c53578:::zonegroups_names.default:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x56388264dc60 con 0x5638826162a0 2023-10-05T19:28:33.695 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.690+0000 7fce08b05700 1 -- 172.21.15.167:0/2192230658 <== osd.1 v2:172.21.15.167:6800/30035 17 ==== osd_op_reply(43 zonegroups_names.default [read 0~46 out=46b] v0'0 uv2 ondisk = 0) v8 ==== 168+0+46 (crc 0 0 0) 0x7fcdf802f080 con 0x5638826162a0 2023-10-05T19:28:33.695 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.690+0000 7fce0a7cfe40 20 rados_obj.operate() r=0 bl.length=46 2023-10-05T19:28:33.695 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.690+0000 7fce0a7cfe40 1 -- 172.21.15.167:0/2192230658 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:44 5.0 5:00000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x56388264ec70 con 0x56388264a720 2023-10-05T19:28:33.697 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.690+0000 7fce08304700 1 -- 172.21.15.167:0/2192230658 <== osd.3 v2:172.21.15.167:6808/30036 11 ==== osd_op_reply(44 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fce0004a040 con 0x56388264a720 2023-10-05T19:28:33.697 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.690+0000 7fce08304700 1 -- 172.21.15.167:0/2192230658 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:45 5.10 5:08000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x56388264ec70 con 0x7fce0001fc20 2023-10-05T19:28:33.698 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.694+0000 7fce07b03700 1 -- 172.21.15.167:0/2192230658 <== osd.2 v2:172.21.15.167:6816/30037 7 ==== osd_op_reply(45 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fcdfc029040 con 0x7fce0001fc20 2023-10-05T19:28:33.698 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.694+0000 7fce07b03700 1 -- 172.21.15.167:0/2192230658 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:46 5.8 5:10000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x56388264ec70 con 0x7fcdfc007030 2023-10-05T19:28:33.699 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.694+0000 7fce08b05700 1 -- 172.21.15.167:0/2192230658 <== osd.0 v2:172.21.15.118:6800/27690 11 ==== osd_op_reply(46 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fcdf802f080 con 0x7fcdfc007030 2023-10-05T19:28:33.699 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.694+0000 7fce08b05700 1 -- 172.21.15.167:0/2192230658 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:47 5.18 5:18000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x563882649dc0 con 0x5638826162a0 2023-10-05T19:28:33.700 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.694+0000 7fce08b05700 1 -- 172.21.15.167:0/2192230658 <== osd.1 v2:172.21.15.167:6800/30035 18 ==== osd_op_reply(47 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fcdf802f080 con 0x5638826162a0 2023-10-05T19:28:33.700 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.694+0000 7fce08b05700 1 -- 172.21.15.167:0/2192230658 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:48 5.4 5:20000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x563882649dc0 con 0x5638826162a0 2023-10-05T19:28:33.701 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.694+0000 7fce08b05700 1 -- 172.21.15.167:0/2192230658 <== osd.1 v2:172.21.15.167:6800/30035 19 ==== osd_op_reply(48 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fcdf802f080 con 0x5638826162a0 2023-10-05T19:28:33.701 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.694+0000 7fce08b05700 1 -- 172.21.15.167:0/2192230658 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:49 5.14 5:28000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x563882649dc0 con 0x7fcdfc007030 2023-10-05T19:28:33.702 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.698+0000 7fce08b05700 1 -- 172.21.15.167:0/2192230658 <== osd.0 v2:172.21.15.118:6800/27690 12 ==== osd_op_reply(49 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fcdf802f080 con 0x7fcdfc007030 2023-10-05T19:28:33.702 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.698+0000 7fce08b05700 1 -- 172.21.15.167:0/2192230658 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:50 5.c 5:30000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x563882649dc0 con 0x7fcdfc007030 2023-10-05T19:28:33.703 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.698+0000 7fce08b05700 1 -- 172.21.15.167:0/2192230658 <== osd.0 v2:172.21.15.118:6800/27690 13 ==== osd_op_reply(50 [pgnls start_epoch 30 out=112b] v23'1 uv1 ondisk = 1) v8 ==== 144+0+112 (crc 0 0 0) 0x7fcdf802f080 con 0x7fcdfc007030 2023-10-05T19:28:33.703 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.698+0000 7fce08b05700 1 -- 172.21.15.167:0/2192230658 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:51 5.1c 5:38000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x563882649dc0 con 0x7fce0001fc20 2023-10-05T19:28:33.704 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.698+0000 7fce07b03700 1 -- 172.21.15.167:0/2192230658 <== osd.2 v2:172.21.15.167:6816/30037 8 ==== osd_op_reply(51 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fcdfc029040 con 0x7fce0001fc20 2023-10-05T19:28:33.704 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.698+0000 7fce07b03700 1 -- 172.21.15.167:0/2192230658 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:52 5.2 5:40000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x563882649dc0 con 0x56388264a720 2023-10-05T19:28:33.704 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.698+0000 7fce08304700 1 -- 172.21.15.167:0/2192230658 <== osd.3 v2:172.21.15.167:6808/30036 12 ==== osd_op_reply(52 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fce0004a040 con 0x56388264a720 2023-10-05T19:28:33.705 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.698+0000 7fce08304700 1 -- 172.21.15.167:0/2192230658 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:53 5.12 5:48000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x563882649dc0 con 0x5638826162a0 2023-10-05T19:28:33.705 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.698+0000 7fce08b05700 1 -- 172.21.15.167:0/2192230658 <== osd.1 v2:172.21.15.167:6800/30035 20 ==== osd_op_reply(53 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fcdf802f080 con 0x5638826162a0 2023-10-05T19:28:33.705 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.698+0000 7fce08b05700 1 -- 172.21.15.167:0/2192230658 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:54 5.a 5:50000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x563882649dc0 con 0x7fcdfc007030 2023-10-05T19:28:33.706 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.702+0000 7fce08b05700 1 -- 172.21.15.167:0/2192230658 <== osd.0 v2:172.21.15.118:6800/27690 14 ==== osd_op_reply(54 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fcdf802f080 con 0x7fcdfc007030 2023-10-05T19:28:33.707 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.702+0000 7fce08b05700 1 -- 172.21.15.167:0/2192230658 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:55 5.1a 5:58000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x563882649dc0 con 0x5638826162a0 2023-10-05T19:28:33.707 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.702+0000 7fce08b05700 1 -- 172.21.15.167:0/2192230658 <== osd.1 v2:172.21.15.167:6800/30035 21 ==== osd_op_reply(55 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fcdf802f080 con 0x5638826162a0 2023-10-05T19:28:33.707 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.702+0000 7fce08b05700 1 -- 172.21.15.167:0/2192230658 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:56 5.6 5:60000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x563882649dc0 con 0x7fce0001fc20 2023-10-05T19:28:33.708 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.702+0000 7fce07b03700 1 -- 172.21.15.167:0/2192230658 <== osd.2 v2:172.21.15.167:6816/30037 9 ==== osd_op_reply(56 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fcdfc029040 con 0x7fce0001fc20 2023-10-05T19:28:33.708 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.702+0000 7fce07b03700 1 -- 172.21.15.167:0/2192230658 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:57 5.16 5:68000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x563882649dc0 con 0x56388264a720 2023-10-05T19:28:33.709 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.702+0000 7fce08304700 1 -- 172.21.15.167:0/2192230658 <== osd.3 v2:172.21.15.167:6808/30036 13 ==== osd_op_reply(57 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fce0004a040 con 0x56388264a720 2023-10-05T19:28:33.709 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.702+0000 7fce08304700 1 -- 172.21.15.167:0/2192230658 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:58 5.e 5:70000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x563882649dc0 con 0x56388264a720 2023-10-05T19:28:33.710 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.706+0000 7fce08304700 1 -- 172.21.15.167:0/2192230658 <== osd.3 v2:172.21.15.167:6808/30036 14 ==== osd_op_reply(58 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fce0004a040 con 0x56388264a720 2023-10-05T19:28:33.710 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.706+0000 7fce08304700 1 -- 172.21.15.167:0/2192230658 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:59 5.1e 5:78000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x563882649dc0 con 0x7fcdfc007030 2023-10-05T19:28:33.711 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.706+0000 7fce08b05700 1 -- 172.21.15.167:0/2192230658 <== osd.0 v2:172.21.15.118:6800/27690 15 ==== osd_op_reply(59 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fcdf802f080 con 0x7fcdfc007030 2023-10-05T19:28:33.711 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.706+0000 7fce08b05700 1 -- 172.21.15.167:0/2192230658 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:60 5.1 5:80000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x563882649dc0 con 0x56388264a720 2023-10-05T19:28:33.712 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.706+0000 7fce08304700 1 -- 172.21.15.167:0/2192230658 <== osd.3 v2:172.21.15.167:6808/30036 15 ==== osd_op_reply(60 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fce0004a040 con 0x56388264a720 2023-10-05T19:28:33.712 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.706+0000 7fce08304700 1 -- 172.21.15.167:0/2192230658 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:61 5.11 5:88000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x563882649dc0 con 0x56388264a720 2023-10-05T19:28:33.712 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.706+0000 7fce08304700 1 -- 172.21.15.167:0/2192230658 <== osd.3 v2:172.21.15.167:6808/30036 16 ==== osd_op_reply(61 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fce0004a040 con 0x56388264a720 2023-10-05T19:28:33.713 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.706+0000 7fce08304700 1 -- 172.21.15.167:0/2192230658 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:62 5.9 5:90000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x563882649dc0 con 0x5638826162a0 2023-10-05T19:28:33.713 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.706+0000 7fce08b05700 1 -- 172.21.15.167:0/2192230658 <== osd.1 v2:172.21.15.167:6800/30035 22 ==== osd_op_reply(62 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fcdf802f080 con 0x5638826162a0 2023-10-05T19:28:33.714 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.706+0000 7fce08b05700 1 -- 172.21.15.167:0/2192230658 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:63 5.19 5:98000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x563882649dc0 con 0x5638826162a0 2023-10-05T19:28:33.714 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.710+0000 7fce08b05700 1 -- 172.21.15.167:0/2192230658 <== osd.1 v2:172.21.15.167:6800/30035 23 ==== osd_op_reply(63 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fcdf802f080 con 0x5638826162a0 2023-10-05T19:28:33.714 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.710+0000 7fce08b05700 1 -- 172.21.15.167:0/2192230658 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:64 5.5 5:a0000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x563882649dc0 con 0x7fcdfc007030 2023-10-05T19:28:33.715 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.710+0000 7fce08b05700 1 -- 172.21.15.167:0/2192230658 <== osd.0 v2:172.21.15.118:6800/27690 16 ==== osd_op_reply(64 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fcdf802f080 con 0x7fcdfc007030 2023-10-05T19:28:33.716 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.710+0000 7fce08b05700 1 -- 172.21.15.167:0/2192230658 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:65 5.15 5:a8000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x563882649dc0 con 0x56388264a720 2023-10-05T19:28:33.716 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.710+0000 7fce08304700 1 -- 172.21.15.167:0/2192230658 <== osd.3 v2:172.21.15.167:6808/30036 17 ==== osd_op_reply(65 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fce0004a040 con 0x56388264a720 2023-10-05T19:28:33.716 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.710+0000 7fce08304700 1 -- 172.21.15.167:0/2192230658 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:66 5.d 5:b0000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x563882649dc0 con 0x7fce0001fc20 2023-10-05T19:28:33.717 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.710+0000 7fce07b03700 1 -- 172.21.15.167:0/2192230658 <== osd.2 v2:172.21.15.167:6816/30037 10 ==== osd_op_reply(66 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fcdfc029040 con 0x7fce0001fc20 2023-10-05T19:28:33.717 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.710+0000 7fce07b03700 1 -- 172.21.15.167:0/2192230658 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:67 5.1d 5:b8000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x563882649dc0 con 0x5638826162a0 2023-10-05T19:28:33.718 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.714+0000 7fce08b05700 1 -- 172.21.15.167:0/2192230658 <== osd.1 v2:172.21.15.167:6800/30035 24 ==== osd_op_reply(67 [pgnls start_epoch 30 out=107b] v30'2 uv2 ondisk = 1) v8 ==== 144+0+107 (crc 0 0 0) 0x7fcdf802f080 con 0x5638826162a0 2023-10-05T19:28:33.718 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.714+0000 7fce08b05700 1 -- 172.21.15.167:0/2192230658 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:68 5.3 5:c0000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x563882649dc0 con 0x7fcdfc007030 2023-10-05T19:28:33.719 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.714+0000 7fce08b05700 1 -- 172.21.15.167:0/2192230658 <== osd.0 v2:172.21.15.118:6800/27690 17 ==== osd_op_reply(68 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fcdf802f080 con 0x7fcdfc007030 2023-10-05T19:28:33.719 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.714+0000 7fce08b05700 1 -- 172.21.15.167:0/2192230658 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:69 5.13 5:c8000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x563882649dc0 con 0x56388264a720 2023-10-05T19:28:33.720 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.714+0000 7fce08304700 1 -- 172.21.15.167:0/2192230658 <== osd.3 v2:172.21.15.167:6808/30036 18 ==== osd_op_reply(69 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fce0004a040 con 0x56388264a720 2023-10-05T19:28:33.720 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.714+0000 7fce08304700 1 -- 172.21.15.167:0/2192230658 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:70 5.b 5:d0000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x563882649dc0 con 0x7fcdfc007030 2023-10-05T19:28:33.721 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.714+0000 7fce08b05700 1 -- 172.21.15.167:0/2192230658 <== osd.0 v2:172.21.15.118:6800/27690 18 ==== osd_op_reply(70 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fcdf802f080 con 0x7fcdfc007030 2023-10-05T19:28:33.721 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.714+0000 7fce08b05700 1 -- 172.21.15.167:0/2192230658 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:71 5.1b 5:d8000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x563882649dc0 con 0x7fce0001fc20 2023-10-05T19:28:33.721 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.714+0000 7fce07b03700 1 -- 172.21.15.167:0/2192230658 <== osd.2 v2:172.21.15.167:6816/30037 11 ==== osd_op_reply(71 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fcdfc029040 con 0x7fce0001fc20 2023-10-05T19:28:33.722 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.714+0000 7fce07b03700 1 -- 172.21.15.167:0/2192230658 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:72 5.7 5:e0000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x563882649dc0 con 0x56388264a720 2023-10-05T19:28:33.722 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.718+0000 7fce08304700 1 -- 172.21.15.167:0/2192230658 <== osd.3 v2:172.21.15.167:6808/30036 19 ==== osd_op_reply(72 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fce0004a040 con 0x56388264a720 2023-10-05T19:28:33.722 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.718+0000 7fce08304700 1 -- 172.21.15.167:0/2192230658 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:73 5.17 5:e8000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x563882649dc0 con 0x7fcdfc007030 2023-10-05T19:28:33.723 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.718+0000 7fce08b05700 1 -- 172.21.15.167:0/2192230658 <== osd.0 v2:172.21.15.118:6800/27690 19 ==== osd_op_reply(73 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fcdf802f080 con 0x7fcdfc007030 2023-10-05T19:28:33.723 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.718+0000 7fce08b05700 1 -- 172.21.15.167:0/2192230658 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:74 5.f 5:f0000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x563882649dc0 con 0x5638826162a0 2023-10-05T19:28:33.724 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.718+0000 7fce08b05700 1 -- 172.21.15.167:0/2192230658 <== osd.1 v2:172.21.15.167:6800/30035 25 ==== osd_op_reply(74 [pgnls start_epoch 30 out=115b] v23'2 uv2 ondisk = 1) v8 ==== 144+0+115 (crc 0 0 0) 0x7fcdf802f080 con 0x5638826162a0 2023-10-05T19:28:33.724 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.718+0000 7fce08b05700 1 -- 172.21.15.167:0/2192230658 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:75 5.1f 5:f8000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x563882649dc0 con 0x7fce0001fc20 2023-10-05T19:28:33.725 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.718+0000 7fce07b03700 1 -- 172.21.15.167:0/2192230658 <== osd.2 v2:172.21.15.167:6816/30037 12 ==== osd_op_reply(75 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fcdfc029040 con 0x7fce0001fc20 2023-10-05T19:28:33.725 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.718+0000 7fce0a7cfe40 20 RGWRados::pool_iterate: got zonegroup_info.4f1f719d-9648-417b-b10c-e5d939825419 2023-10-05T19:28:33.725 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.718+0000 7fce0a7cfe40 20 RGWRados::pool_iterate: got zone_info.9a9df965-6612-45d4-a45f-6e3c889bacc2 2023-10-05T19:28:33.726 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.718+0000 7fce0a7cfe40 20 RGWRados::pool_iterate: got zone_names.default 2023-10-05T19:28:33.726 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.718+0000 7fce0a7cfe40 20 RGWRados::pool_iterate: got zonegroups_names.default 2023-10-05T19:28:33.726 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.718+0000 7fce0a7cfe40 20 rados->read ofs=0 len=0 2023-10-05T19:28:33.726 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.718+0000 7fce0a7cfe40 1 -- 172.21.15.167:0/2192230658 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:76 5.f 5:f43b6ece:::zone_names.default:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x56388264dc60 con 0x5638826162a0 2023-10-05T19:28:33.727 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.722+0000 7fce08b05700 1 -- 172.21.15.167:0/2192230658 <== osd.1 v2:172.21.15.167:6800/30035 26 ==== osd_op_reply(76 zone_names.default [read 0~46 out=46b] v0'0 uv1 ondisk = 0) v8 ==== 162+0+46 (crc 0 0 0) 0x7fcdf802f080 con 0x5638826162a0 2023-10-05T19:28:33.727 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.722+0000 7fce0a7cfe40 20 rados_obj.operate() r=0 bl.length=46 2023-10-05T19:28:33.727 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.722+0000 7fce0a7cfe40 20 rados->read ofs=0 len=0 2023-10-05T19:28:33.727 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.722+0000 7fce0a7cfe40 1 -- 172.21.15.167:0/2192230658 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:77 5.1d 5:b8374eb4:::zone_info.9a9df965-6612-45d4-a45f-6e3c889bacc2:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x56388264e010 con 0x5638826162a0 2023-10-05T19:28:33.727 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.722+0000 7fce08b05700 1 -- 172.21.15.167:0/2192230658 <== osd.1 v2:172.21.15.167:6800/30035 27 ==== osd_op_reply(77 zone_info.9a9df965-6612-45d4-a45f-6e3c889bacc2 [read 0~874 out=874b] v0'0 uv2 ondisk = 0) v8 ==== 190+0+874 (crc 0 0 0) 0x7fcdf802f080 con 0x5638826162a0 2023-10-05T19:28:33.727 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.722+0000 7fce0a7cfe40 20 rados_obj.operate() r=0 bl.length=874 2023-10-05T19:28:33.727 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.722+0000 7fce0a7cfe40 20 rados->read ofs=0 len=0 2023-10-05T19:28:33.727 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.722+0000 7fce0a7cfe40 1 -- 172.21.15.167:0/2192230658 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:78 5.f 5:f4c53578:::zonegroups_names.default:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x56388264e040 con 0x5638826162a0 2023-10-05T19:28:33.728 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.722+0000 7fce08b05700 1 -- 172.21.15.167:0/2192230658 <== osd.1 v2:172.21.15.167:6800/30035 28 ==== osd_op_reply(78 zonegroups_names.default [read 0~46 out=46b] v0'0 uv2 ondisk = 0) v8 ==== 168+0+46 (crc 0 0 0) 0x7fcdf802f080 con 0x5638826162a0 2023-10-05T19:28:33.728 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.722+0000 7fce0a7cfe40 20 rados_obj.operate() r=0 bl.length=46 2023-10-05T19:28:33.728 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.722+0000 7fce0a7cfe40 20 rados->read ofs=0 len=0 2023-10-05T19:28:33.728 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.722+0000 7fce0a7cfe40 1 -- 172.21.15.167:0/2192230658 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:79 5.c 5:30d299a5:::zonegroup_info.4f1f719d-9648-417b-b10c-e5d939825419:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x56388264e010 con 0x7fcdfc007030 2023-10-05T19:28:33.729 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.722+0000 7fce08b05700 1 -- 172.21.15.167:0/2192230658 <== osd.0 v2:172.21.15.118:6800/27690 20 ==== osd_op_reply(79 zonegroup_info.4f1f719d-9648-417b-b10c-e5d939825419 [read 0~362 out=362b] v0'0 uv1 ondisk = 0) v8 ==== 195+0+362 (crc 0 0 0) 0x7fcdf802f080 con 0x7fcdfc007030 2023-10-05T19:28:33.729 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.722+0000 7fce0a7cfe40 20 rados_obj.operate() r=0 bl.length=362 2023-10-05T19:28:33.729 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.722+0000 7fce0a7cfe40 20 rados->read ofs=0 len=0 2023-10-05T19:28:33.729 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.722+0000 7fce0a7cfe40 1 -- 172.21.15.167:0/2192230658 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:80 5.3 5:c0ad8cbd:::region_map:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x56388264e4c0 con 0x7fcdfc007030 2023-10-05T19:28:33.730 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.722+0000 7fce08b05700 1 -- 172.21.15.167:0/2192230658 <== osd.0 v2:172.21.15.118:6800/27690 21 ==== osd_op_reply(80 region_map [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 ==== 154+0+0 (crc 0 0 0) 0x7fcdf802f080 con 0x7fcdfc007030 2023-10-05T19:28:33.730 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.722+0000 7fce0a7cfe40 20 rados_obj.operate() r=-2 bl.length=0 2023-10-05T19:28:33.730 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.722+0000 7fce0a7cfe40 20 zone default found 2023-10-05T19:28:33.730 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.722+0000 7fce0a7cfe40 4 Realm: () 2023-10-05T19:28:33.730 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.722+0000 7fce0a7cfe40 4 ZoneGroup: default (4f1f719d-9648-417b-b10c-e5d939825419) 2023-10-05T19:28:33.730 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.722+0000 7fce0a7cfe40 4 Zone: default (9a9df965-6612-45d4-a45f-6e3c889bacc2) 2023-10-05T19:28:33.730 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.722+0000 7fce0a7cfe40 10 cannot find current period zonegroup using local zonegroup configuration 2023-10-05T19:28:33.730 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.722+0000 7fce0a7cfe40 20 zonegroup default 2023-10-05T19:28:33.731 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.722+0000 7fce0a7cfe40 20 rados->read ofs=0 len=0 2023-10-05T19:28:33.731 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.722+0000 7fce0a7cfe40 1 -- 172.21.15.167:0/2192230658 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:81 5.3 5:c52100b6:::period_config.default:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x56388264e4c0 con 0x7fcdfc007030 2023-10-05T19:28:33.731 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.726+0000 7fce08b05700 1 -- 172.21.15.167:0/2192230658 <== osd.0 v2:172.21.15.118:6800/27690 22 ==== osd_op_reply(81 period_config.default [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 ==== 165+0+0 (crc 0 0 0) 0x7fcdf8050040 con 0x7fcdfc007030 2023-10-05T19:28:33.732 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.726+0000 7fce0a7cfe40 20 rados_obj.operate() r=-2 bl.length=0 2023-10-05T19:28:33.732 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.726+0000 7fce0a7cfe40 20 rados->read ofs=0 len=0 2023-10-05T19:28:33.732 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.726+0000 7fce0a7cfe40 1 -- 172.21.15.167:0/2192230658 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:82 6.2 6:4347d321:::bucket.sync-source-hints.:head [call version.read in=11b,read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x56388264fbb0 con 0x56388264a720 2023-10-05T19:28:33.732 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.726+0000 7fce08304700 1 -- 172.21.15.167:0/2192230658 <== osd.3 v2:172.21.15.167:6808/30036 20 ==== osd_op_reply(82 bucket.sync-source-hints. [call,read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 ==== 211+0+0 (crc 0 0 0) 0x7fce0004a040 con 0x56388264a720 2023-10-05T19:28:33.732 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.726+0000 7fce0a7cfe40 20 rados_obj.operate() r=-2 bl.length=0 2023-10-05T19:28:33.732 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.726+0000 7fce0a7cfe40 20 rados->read ofs=0 len=0 2023-10-05T19:28:33.732 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.726+0000 7fce0a7cfe40 1 -- 172.21.15.167:0/2192230658 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:83 6.b 6:d467b91b:::bucket.sync-target-hints.:head [call version.read in=11b,read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x56388264ff60 con 0x56388264a720 2023-10-05T19:28:33.733 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.726+0000 7fce08304700 1 -- 172.21.15.167:0/2192230658 <== osd.3 v2:172.21.15.167:6808/30036 21 ==== osd_op_reply(83 bucket.sync-target-hints. [call,read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 ==== 211+0+0 (crc 0 0 0) 0x7fce0004a040 con 0x56388264a720 2023-10-05T19:28:33.733 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.726+0000 7fce0a7cfe40 20 rados_obj.operate() r=-2 bl.length=0 2023-10-05T19:28:33.733 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.726+0000 7fce0a7cfe40 20 started sync module instance, tier type = 2023-10-05T19:28:33.733 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.726+0000 7fce0a7cfe40 20 started zone id=9a9df965-6612-45d4-a45f-6e3c889bacc2 (name=default) with tier type = 2023-10-05T19:28:33.733 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.726+0000 7fce0a7cfe40 1 -- 172.21.15.167:0/2192230658 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:84 7.1f 7:f95f44c2:::notify.0:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x56388264f760 con 0x7fce0001fc20 2023-10-05T19:28:33.735 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.730+0000 7fce07b03700 1 -- 172.21.15.167:0/2192230658 <== osd.2 v2:172.21.15.167:6816/30037 13 ==== osd_op_reply(84 notify.0 [create] v30'123 uv123 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7fcdfc029040 con 0x7fce0001fc20 2023-10-05T19:28:33.735 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.730+0000 7fce0a7cfe40 1 -- 172.21.15.167:0/2192230658 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:85 7.1f 7:f95f44c2:::notify.0:head [watch watch cookie 94800705811056] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x563882650800 con 0x7fce0001fc20 2023-10-05T19:28:33.735 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.730+0000 7fce0a7cfe40 1 -- 172.21.15.167:0/2192230658 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:86 7.0 7:05bf5b68:::notify.1:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x5638826510a0 con 0x5638826162a0 2023-10-05T19:28:33.738 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.734+0000 7fce08b05700 1 -- 172.21.15.167:0/2192230658 <== osd.1 v2:172.21.15.167:6800/30035 29 ==== osd_op_reply(86 notify.1 [create] v30'123 uv123 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7fcdf8050040 con 0x5638826162a0 2023-10-05T19:28:33.738 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.734+0000 7fce0a7cfe40 1 -- 172.21.15.167:0/2192230658 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:87 7.0 7:05bf5b68:::notify.1:head [watch watch cookie 94800705817520] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x5638826514f0 con 0x5638826162a0 2023-10-05T19:28:33.738 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.734+0000 7fce0a7cfe40 1 -- 172.21.15.167:0/2192230658 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:88 7.15 7:a93a5511:::notify.2:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x563882651e20 con 0x56388264a720 2023-10-05T19:28:33.738 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.734+0000 7fce07b03700 1 -- 172.21.15.167:0/2192230658 <== osd.2 v2:172.21.15.167:6816/30037 14 ==== osd_op_reply(85 notify.0 [watch watch cookie 94800705811056] v30'124 uv123 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7fcdfc029040 con 0x7fce0001fc20 2023-10-05T19:28:33.741 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.734+0000 7fce08304700 1 -- 172.21.15.167:0/2192230658 <== osd.3 v2:172.21.15.167:6808/30036 22 ==== osd_op_reply(88 notify.2 [create] v30'123 uv123 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7fce0004a040 con 0x56388264a720 2023-10-05T19:28:33.741 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.734+0000 7fce0a7cfe40 1 -- 172.21.15.167:0/2192230658 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:89 7.15 7:a93a5511:::notify.2:head [watch watch cookie 94800705820976] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x563882652220 con 0x56388264a720 2023-10-05T19:28:33.741 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.734+0000 7fce0a7cfe40 1 -- 172.21.15.167:0/2192230658 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:90 7.e 7:7759931f:::notify.3:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x563882652c30 con 0x7fce0001fc20 2023-10-05T19:28:33.741 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.734+0000 7fce08b05700 1 -- 172.21.15.167:0/2192230658 <== osd.1 v2:172.21.15.167:6800/30035 30 ==== osd_op_reply(87 notify.1 [watch watch cookie 94800705817520] v30'124 uv123 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7fcdf8050040 con 0x5638826162a0 2023-10-05T19:28:33.743 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.738+0000 7fce07b03700 1 -- 172.21.15.167:0/2192230658 <== osd.2 v2:172.21.15.167:6816/30037 15 ==== osd_op_reply(90 notify.3 [create] v30'123 uv123 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7fcdfc029040 con 0x7fce0001fc20 2023-10-05T19:28:33.743 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.738+0000 7fce0a7cfe40 1 -- 172.21.15.167:0/2192230658 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:91 7.e 7:7759931f:::notify.3:head [watch watch cookie 94800705818624] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x563882652860 con 0x7fce0001fc20 2023-10-05T19:28:33.743 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.738+0000 7fce0a7cfe40 1 -- 172.21.15.167:0/2192230658 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:92 7.d 7:b4812045:::notify.4:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x563882653cb0 con 0x5638826162a0 2023-10-05T19:28:33.744 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.738+0000 7fce08304700 1 -- 172.21.15.167:0/2192230658 <== osd.3 v2:172.21.15.167:6808/30036 23 ==== osd_op_reply(89 notify.2 [watch watch cookie 94800705820976] v30'124 uv123 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7fce0004a040 con 0x56388264a720 2023-10-05T19:28:33.746 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.738+0000 7fce07b03700 1 -- 172.21.15.167:0/2192230658 <== osd.2 v2:172.21.15.167:6816/30037 16 ==== osd_op_reply(91 notify.3 [watch watch cookie 94800705818624] v30'124 uv123 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7fcdfc029040 con 0x7fce0001fc20 2023-10-05T19:28:33.746 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.738+0000 7fce08b05700 1 -- 172.21.15.167:0/2192230658 <== osd.1 v2:172.21.15.167:6800/30035 31 ==== osd_op_reply(92 notify.4 [create] v30'123 uv123 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7fcdf8050040 con 0x5638826162a0 2023-10-05T19:28:33.746 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.738+0000 7fce0a7cfe40 1 -- 172.21.15.167:0/2192230658 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:93 7.d 7:b4812045:::notify.4:head [watch watch cookie 94800705826784] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x563882653780 con 0x5638826162a0 2023-10-05T19:28:33.746 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.738+0000 7fce0a7cfe40 1 -- 172.21.15.167:0/2192230658 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:94 7.3 7:c609908c:::notify.5:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x5638826545a0 con 0x7fcdfc007030 2023-10-05T19:28:33.748 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.742+0000 7fce08b05700 1 -- 172.21.15.167:0/2192230658 <== osd.0 v2:172.21.15.118:6800/27690 23 ==== osd_op_reply(94 notify.5 [create] v30'123 uv123 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7fcdf8050040 con 0x7fcdfc007030 2023-10-05T19:28:33.748 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.742+0000 7fce0a7cfe40 1 -- 172.21.15.167:0/2192230658 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:95 7.3 7:c609908c:::notify.5:head [watch watch cookie 94800705831008] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x5638826549a0 con 0x7fcdfc007030 2023-10-05T19:28:33.748 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.742+0000 7fce0a7cfe40 1 -- 172.21.15.167:0/2192230658 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:96 7.14 7:2b04a3e9:::notify.6:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x5638826553b0 con 0x7fcdfc007030 2023-10-05T19:28:33.748 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.742+0000 7fce08b05700 1 -- 172.21.15.167:0/2192230658 <== osd.1 v2:172.21.15.167:6800/30035 32 ==== osd_op_reply(93 notify.4 [watch watch cookie 94800705826784] v30'124 uv123 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7fcdf8050040 con 0x5638826162a0 2023-10-05T19:28:33.751 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.746+0000 7fce08b05700 1 -- 172.21.15.167:0/2192230658 <== osd.0 v2:172.21.15.118:6800/27690 24 ==== osd_op_reply(95 notify.5 [watch watch cookie 94800705831008] v30'124 uv123 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7fcdf8050040 con 0x7fcdfc007030 2023-10-05T19:28:33.751 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.746+0000 7fce08b05700 1 -- 172.21.15.167:0/2192230658 <== osd.0 v2:172.21.15.118:6800/27690 25 ==== osd_op_reply(96 notify.6 [create] v30'123 uv123 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7fcdf8050040 con 0x7fcdfc007030 2023-10-05T19:28:33.751 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.746+0000 7fce0a7cfe40 1 -- 172.21.15.167:0/2192230658 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:97 7.14 7:2b04a3e9:::notify.6:head [watch watch cookie 94800705832112] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x563882654900 con 0x7fcdfc007030 2023-10-05T19:28:33.751 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.746+0000 7fce0a7cfe40 1 -- 172.21.15.167:0/2192230658 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:98 7.9 7:93e5b521:::notify.7:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x5638826558a0 con 0x7fcdfc007030 2023-10-05T19:28:33.753 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.746+0000 7fce08b05700 1 -- 172.21.15.167:0/2192230658 <== osd.0 v2:172.21.15.118:6800/27690 26 ==== osd_op_reply(97 notify.6 [watch watch cookie 94800705832112] v30'124 uv123 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7fcdf8050040 con 0x7fcdfc007030 2023-10-05T19:28:33.754 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.746+0000 7fce08b05700 1 -- 172.21.15.167:0/2192230658 <== osd.0 v2:172.21.15.118:6800/27690 27 ==== osd_op_reply(98 notify.7 [create] v30'123 uv123 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7fcdf8050040 con 0x7fcdfc007030 2023-10-05T19:28:33.754 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.746+0000 7fce0a7cfe40 1 -- 172.21.15.167:0/2192230658 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:99 7.9 7:93e5b521:::notify.7:head [watch watch cookie 94800705834688] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x563882655800 con 0x7fcdfc007030 2023-10-05T19:28:33.754 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.746+0000 7fce0a7cfe40 20 add_watcher() i=0 2023-10-05T19:28:33.754 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.746+0000 7fce0a7cfe40 20 add_watcher() i=1 2023-10-05T19:28:33.754 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.746+0000 7fce0a7cfe40 20 add_watcher() i=2 2023-10-05T19:28:33.754 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.746+0000 7fce0a7cfe40 20 add_watcher() i=3 2023-10-05T19:28:33.754 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.746+0000 7fce0a7cfe40 20 add_watcher() i=4 2023-10-05T19:28:33.754 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.746+0000 7fce0a7cfe40 20 add_watcher() i=5 2023-10-05T19:28:33.754 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.746+0000 7fce0a7cfe40 20 add_watcher() i=6 2023-10-05T19:28:33.756 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.750+0000 7fce08b05700 1 -- 172.21.15.167:0/2192230658 <== osd.0 v2:172.21.15.118:6800/27690 28 ==== osd_op_reply(99 notify.7 [watch watch cookie 94800705834688] v30'124 uv123 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7fcdf8050040 con 0x7fcdfc007030 2023-10-05T19:28:33.756 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.750+0000 7fce0a7cfe40 20 add_watcher() i=7 2023-10-05T19:28:33.756 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.750+0000 7fce0a7cfe40 2 all 8 watchers are set, enabling cache 2023-10-05T19:28:33.756 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.750+0000 7fce0a7cfe40 1 -- 172.21.15.167:0/2192230658 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:100 6.4 6:22d26bf9:::data_loggenerations_metadata:head [call version.check_conds in=74b,call version.read in=11b,read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x5638826589d0 con 0x5638826162a0 2023-10-05T19:28:33.756 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.750+0000 7fce08b05700 1 -- 172.21.15.167:0/2192230658 <== osd.1 v2:172.21.15.167:6800/30035 33 ==== osd_op_reply(100 data_loggenerations_metadata [call,call out=48b,read 0~28 out=28b] v0'0 uv1 ondisk = 0) v8 ==== 256+0+76 (crc 0 0 0) 0x7fcdf8050040 con 0x5638826162a0 2023-10-05T19:28:33.757 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.750+0000 7fce0a7cfe40 1 -- 172.21.15.167:0/2192230658 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:101 6.4 6:22d26bf9:::data_loggenerations_metadata:head [watch watch cookie 94800705836160] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x563882655dc0 con 0x5638826162a0 2023-10-05T19:28:33.759 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.754+0000 7fce08b05700 1 -- 172.21.15.167:0/2192230658 <== osd.1 v2:172.21.15.167:6800/30035 34 ==== osd_op_reply(101 data_loggenerations_metadata [watch watch cookie 94800705836160] v30'120 uv1 ondisk = 0) v8 ==== 172+0+0 (crc 0 0 0) 0x7fcdf8050040 con 0x5638826162a0 2023-10-05T19:28:33.759 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.754+0000 7fcdd9fe3700 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2023-10-05T19:28:33.759 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.754+0000 7fce0a7cfe40 20 check_secure_mon_conn(): auth registy supported: methods=[2] modes=[2,1] 2023-10-05T19:28:33.759 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.754+0000 7fce0a7cfe40 20 check_secure_mon_conn(): mode 1 is insecure 2023-10-05T19:28:33.760 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.754+0000 7fce0a7cfe40 1 Processor -- start 2023-10-05T19:28:33.760 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.754+0000 7fce0a7cfe40 1 -- start start 2023-10-05T19:28:33.761 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.754+0000 7fce0a7cfe40 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x563882663240 0x563882663610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-05T19:28:33.761 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.754+0000 7fce0a7cfe40 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x563882663b50 con 0x563882663240 2023-10-05T19:28:33.761 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.754+0000 7fce07b03700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x563882663240 0x563882663610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-05T19:28:33.761 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.754+0000 7fce07b03700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x563882663240 0x563882663610 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.118:3300/0 says I am v2:172.21.15.167:58668/0 (socket says 172.21.15.167:58668) 2023-10-05T19:28:33.761 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.754+0000 7fce07b03700 1 -- 172.21.15.167:0/2330975800 learned_addr learned my addr 172.21.15.167:0/2330975800 (peer_addr_for_me v2:172.21.15.167:0/0) 2023-10-05T19:28:33.761 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.754+0000 7fce07b03700 1 -- 172.21.15.167:0/2330975800 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x563882663c90 con 0x563882663240 2023-10-05T19:28:33.762 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.754+0000 7fce07b03700 1 --2- 172.21.15.167:0/2330975800 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x563882663240 0x563882663610 secure :-1 s=READY pgs=384 cs=0 l=1 rev1=1 crypto rx=0x563882612d40 tx=0x7fcdfc011000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b26e67b212fdda5 server_cookie=0 in_seq=0 out_seq=0 2023-10-05T19:28:33.762 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.758+0000 7fcdd97e2700 1 -- 172.21.15.167:0/2330975800 <== mon.0 v2:172.21.15.118:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fcdfc025040 con 0x563882663240 2023-10-05T19:28:33.762 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.758+0000 7fcdd97e2700 1 -- 172.21.15.167:0/2330975800 <== mon.0 v2:172.21.15.118:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcdfc039450 con 0x563882663240 2023-10-05T19:28:33.762 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.758+0000 7fcdd97e2700 1 -- 172.21.15.167:0/2330975800 <== mon.0 v2:172.21.15.118:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fcdfc039740 con 0x563882663240 2023-10-05T19:28:33.763 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.758+0000 7fce0a7cfe40 1 -- 172.21.15.167:0/2330975800 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x563882663240 msgr2=0x563882663610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-05T19:28:33.764 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.758+0000 7fce0a7cfe40 1 --2- 172.21.15.167:0/2330975800 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x563882663240 0x563882663610 secure :-1 s=READY pgs=384 cs=0 l=1 rev1=1 crypto rx=0x563882612d40 tx=0x7fcdfc011000 comp rx=0 tx=0).stop 2023-10-05T19:28:33.764 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.758+0000 7fce0a7cfe40 1 -- 172.21.15.167:0/2330975800 shutdown_connections 2023-10-05T19:28:33.764 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.758+0000 7fce0a7cfe40 1 --2- 172.21.15.167:0/2330975800 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x563882663240 0x563882663610 unknown :-1 s=CLOSED pgs=384 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-05T19:28:33.764 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.758+0000 7fce0a7cfe40 1 -- 172.21.15.167:0/2330975800 >> 172.21.15.167:0/2330975800 conn(0x56388265e600 msgr2=0x56388265c750 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-05T19:28:33.764 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.758+0000 7fce0a7cfe40 1 -- 172.21.15.167:0/2330975800 shutdown_connections 2023-10-05T19:28:33.764 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.758+0000 7fce0a7cfe40 1 -- 172.21.15.167:0/2330975800 wait complete. 2023-10-05T19:28:33.765 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.758+0000 7fce0a7cfe40 1 Processor -- start 2023-10-05T19:28:33.765 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.758+0000 7fce0a7cfe40 1 -- start start 2023-10-05T19:28:33.765 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.758+0000 7fce0a7cfe40 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x563882663240 0x563882668d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-05T19:28:33.765 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.758+0000 7fce0a7cfe40 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x563882669d80 con 0x563882663240 2023-10-05T19:28:33.766 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.758+0000 7fce07b03700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x563882663240 0x563882668d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-05T19:28:33.766 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.758+0000 7fce07b03700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x563882663240 0x563882668d80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.118:3300/0 says I am v2:172.21.15.167:58680/0 (socket says 172.21.15.167:58680) 2023-10-05T19:28:33.766 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.758+0000 7fce07b03700 1 -- 172.21.15.167:0/2262901231 learned_addr learned my addr 172.21.15.167:0/2262901231 (peer_addr_for_me v2:172.21.15.167:0/0) 2023-10-05T19:28:33.767 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.758+0000 7fce07b03700 1 -- 172.21.15.167:0/2262901231 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5638826128d0 con 0x563882663240 2023-10-05T19:28:33.767 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.762+0000 7fce07b03700 1 --2- 172.21.15.167:0/2262901231 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x563882663240 0x563882668d80 secure :-1 s=READY pgs=385 cs=0 l=1 rev1=1 crypto rx=0x56388265d2c0 tx=0x7fcdfc039000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-05T19:28:33.767 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.762+0000 7fcd537fe700 1 -- 172.21.15.167:0/2262901231 <== mon.0 v2:172.21.15.118:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fcdfc025040 con 0x563882663240 2023-10-05T19:28:33.767 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.762+0000 7fcd537fe700 1 -- 172.21.15.167:0/2262901231 <== mon.0 v2:172.21.15.118:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcdfc047070 con 0x563882663240 2023-10-05T19:28:33.767 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.762+0000 7fcd537fe700 1 -- 172.21.15.167:0/2262901231 <== mon.0 v2:172.21.15.118:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fcdfc02d6a0 con 0x563882663240 2023-10-05T19:28:33.767 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.762+0000 7fce0a7cfe40 1 -- 172.21.15.167:0/2262901231 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x563882669380 con 0x563882663240 2023-10-05T19:28:33.767 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.762+0000 7fce0a7cfe40 1 -- 172.21.15.167:0/2262901231 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x563882669820 con 0x563882663240 2023-10-05T19:28:33.768 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.762+0000 7fcd537fe700 1 -- 172.21.15.167:0/2262901231 <== mon.0 v2:172.21.15.118:3300/0 4 ==== mgrmap(e 5) v1 ==== 40329+0+0 (secure 0 0 0) 0x7fcdfc049070 con 0x563882663240 2023-10-05T19:28:33.768 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.762+0000 7fcd537fe700 1 --2- 172.21.15.167:0/2262901231 >> [v2:172.21.15.167:6824/29889,v1:172.21.15.167:6825/29889] conn(0x7fcd48030c50 0x7fcd48033100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-05T19:28:33.769 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.762+0000 7fcd537fe700 1 -- 172.21.15.167:0/2262901231 <== mon.0 v2:172.21.15.118:3300/0 5 ==== osd_map(30..30 src has 1..30) v4 ==== 6057+0+0 (secure 0 0 0) 0x7fcdfc032030 con 0x563882663240 2023-10-05T19:28:33.769 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.762+0000 7fce08b05700 1 --2- 172.21.15.167:0/2262901231 >> [v2:172.21.15.167:6824/29889,v1:172.21.15.167:6825/29889] conn(0x7fcd48030c50 0x7fcd48033100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-05T19:28:33.769 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.762+0000 7fce0a7cfe40 5 note: GC not initialized 2023-10-05T19:28:33.769 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.762+0000 7fce08b05700 1 --2- 172.21.15.167:0/2262901231 >> [v2:172.21.15.167:6824/29889,v1:172.21.15.167:6825/29889] conn(0x7fcd48030c50 0x7fcd48033100 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x563882658ad0 tx=0x7fcdf8000f90 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-05T19:28:33.771 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.766+0000 7fcd50ff9700 20 reqs_thread_entry: start 2023-10-05T19:28:33.772 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.766+0000 7fcd467fc700 20 reqs_thread_entry: start 2023-10-05T19:28:33.772 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.766+0000 7fce0a7cfe40 20 init_complete bucket index max shards: 11 2023-10-05T19:28:33.772 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.766+0000 7fce0a7cfe40 10 rgw notify: Started notification manager with: 1 workers 2023-10-05T19:28:33.772 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.766+0000 7fcd44ff9700 20 rgw notify: INFO: next queues processing will happen at: Thu Oct 5 19:29:03 2023 2023-10-05T19:28:33.773 INFO:teuthology.orchestra.run.smithi167.stderr: 2023-10-05T19:28:33.773 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.766+0000 7fcd2bfff700 20 reqs_thread_entry: start 2023-10-05T19:28:33.773 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.766+0000 7fce0a7cfe40 20 get_system_obj_state: rctx=0x7fff1dd77318 obj=default.rgw.meta:users.uid:foo state=0x5638824ec0b0 s->prefetch_data=0 2023-10-05T19:28:33.773 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.766+0000 7fce0a7cfe40 10 cache get: name=default.rgw.meta+users.uid+foo : miss 2023-10-05T19:28:33.773 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.766+0000 7fce0a7cfe40 1 -- 172.21.15.167:0/2192230658 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:102 8.18 8:1e6ca5ed:users.uid::foo:head [call version.read in=11b,getxattrs,stat] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x5638826d14d0 con 0x5638826162a0 2023-10-05T19:28:33.775 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.766+0000 7fce08b05700 1 -- 172.21.15.167:0/2192230658 <== osd.1 v2:172.21.15.167:6800/30035 35 ==== osd_op_reply(102 foo [call out=48b,getxattrs out=75b,stat out=16b] v0'0 uv11 ondisk = 0) v8 ==== 231+0+139 (crc 0 0 0) 0x7fcdf8050040 con 0x5638826162a0 2023-10-05T19:28:33.775 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.766+0000 7fce0a7cfe40 10 cache put: name=default.rgw.meta+users.uid+foo info.flags=0x16 2023-10-05T19:28:33.775 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.766+0000 7fce0a7cfe40 10 adding default.rgw.meta+users.uid+foo to cache LRU end 2023-10-05T19:28:33.775 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.766+0000 7fce0a7cfe40 10 updating xattr: name=ceph.objclass.version bl.length()=42 2023-10-05T19:28:33.775 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.766+0000 7fce0a7cfe40 20 get_system_obj_state: s->obj_tag was set empty 2023-10-05T19:28:33.775 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.766+0000 7fce0a7cfe40 20 Read xattr: user.rgw.idtag 2023-10-05T19:28:33.776 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.766+0000 7fce0a7cfe40 10 cache get: name=default.rgw.meta+users.uid+foo : type miss (requested=0x13, cached=0x16) 2023-10-05T19:28:33.776 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.766+0000 7fce0a7cfe40 20 rados->read ofs=0 len=0 2023-10-05T19:28:33.776 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.766+0000 7fce0a7cfe40 1 -- 172.21.15.167:0/2192230658 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:103 8.18 8:1e6ca5ed:users.uid::foo:head [call version.check_conds in=122b,call version.read in=11b,read 0~0,getxattrs] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x5638826d3f60 con 0x5638826162a0 2023-10-05T19:28:33.776 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.766+0000 7fce08b05700 1 -- 172.21.15.167:0/2192230658 <== osd.1 v2:172.21.15.167:6800/30035 36 ==== osd_op_reply(103 foo [call,call out=48b,read 0~357 out=357b,getxattrs out=75b] v0'0 uv11 ondisk = 0) v8 ==== 273+0+480 (crc 0 0 0) 0x7fcdf8050040 con 0x5638826162a0 2023-10-05T19:28:33.776 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.766+0000 7fce0a7cfe40 20 rados_obj.operate() r=0 bl.length=357 2023-10-05T19:28:33.777 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.766+0000 7fce0a7cfe40 10 cache put: name=default.rgw.meta+users.uid+foo info.flags=0x13 2023-10-05T19:28:33.777 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.766+0000 7fce0a7cfe40 10 moving default.rgw.meta+users.uid+foo to cache LRU end 2023-10-05T19:28:33.777 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.766+0000 7fce0a7cfe40 10 updating xattr: name=ceph.objclass.version bl.length()=42 2023-10-05T19:28:33.777 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.766+0000 7fce0a7cfe40 20 get_system_obj_state: rctx=0x7fff1dd766c0 obj=default.rgw.meta:root:myfoo state=0x5638824ec0b0 s->prefetch_data=0 2023-10-05T19:28:33.777 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.766+0000 7fce0a7cfe40 10 cache get: name=default.rgw.meta+root+myfoo : miss 2023-10-05T19:28:33.777 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.766+0000 7fce0a7cfe40 1 -- 172.21.15.167:0/2192230658 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:104 8.9 8:91b71841:root::myfoo:head [call version.read in=11b,getxattrs,stat] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x5638826d3770 con 0x7fce0001fc20 2023-10-05T19:28:33.778 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.770+0000 7fce07b03700 1 -- 172.21.15.167:0/2192230658 <== osd.2 v2:172.21.15.167:6816/30037 17 ==== osd_op_reply(104 myfoo [call out=48b,getxattrs out=75b,stat out=16b] v0'0 uv21 ondisk = 0) v8 ==== 233+0+139 (crc 0 0 0) 0x7fcdfc03f020 con 0x7fce0001fc20 2023-10-05T19:28:33.778 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.770+0000 7fce0a7cfe40 10 cache put: name=default.rgw.meta+root+myfoo info.flags=0x16 2023-10-05T19:28:33.778 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.770+0000 7fce0a7cfe40 10 adding default.rgw.meta+root+myfoo to cache LRU end 2023-10-05T19:28:33.778 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.770+0000 7fce0a7cfe40 10 updating xattr: name=ceph.objclass.version bl.length()=42 2023-10-05T19:28:33.778 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.770+0000 7fce0a7cfe40 20 get_system_obj_state: s->obj_tag was set empty 2023-10-05T19:28:33.778 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.770+0000 7fce0a7cfe40 10 cache get: name=default.rgw.meta+root+myfoo : type miss (requested=0x11, cached=0x16) 2023-10-05T19:28:33.779 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.770+0000 7fce0a7cfe40 20 rados->read ofs=0 len=0 2023-10-05T19:28:33.779 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.770+0000 7fce0a7cfe40 1 -- 172.21.15.167:0/2192230658 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:105 8.9 8:91b71841:root::myfoo:head [call version.check_conds in=122b,call version.read in=11b,read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x5638826d5d10 con 0x7fce0001fc20 2023-10-05T19:28:33.783 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.770+0000 7fce07b03700 1 -- 172.21.15.167:0/2192230658 <== osd.2 v2:172.21.15.167:6816/30037 18 ==== osd_op_reply(105 myfoo [call,call out=48b,read 0~165 out=165b] v0'0 uv21 ondisk = 0) v8 ==== 233+0+213 (crc 0 0 0) 0x7fcdfc03f020 con 0x7fce0001fc20 2023-10-05T19:28:33.783 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.770+0000 7fce0a7cfe40 20 rados_obj.operate() r=0 bl.length=165 2023-10-05T19:28:33.783 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.770+0000 7fce0a7cfe40 10 cache put: name=default.rgw.meta+root+myfoo info.flags=0x11 2023-10-05T19:28:33.784 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.770+0000 7fce0a7cfe40 10 moving default.rgw.meta+root+myfoo to cache LRU end 2023-10-05T19:28:33.784 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.770+0000 7fce0a7cfe40 20 get_system_obj_state: rctx=0x7fff1dd766c0 obj=default.rgw.meta:root:.bucket.meta.myfoo:9a9df965-6612-45d4-a45f-6e3c889bacc2.4223.1 state=0x5638824c22b0 s->prefetch_data=0 2023-10-05T19:28:33.784 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.770+0000 7fce0a7cfe40 10 cache get: name=default.rgw.meta+root+.bucket.meta.myfoo:9a9df965-6612-45d4-a45f-6e3c889bacc2.4223.1 : miss 2023-10-05T19:28:33.784 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.770+0000 7fce0a7cfe40 1 -- 172.21.15.167:0/2192230658 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:106 8.11 8:8bb4ed75:root::.bucket.meta.myfoo%3a9a9df965-6612-45d4-a45f-6e3c889bacc2.4223.1:head [call version.read in=11b,getxattrs,stat] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x5638826d53e0 con 0x56388264a720 2023-10-05T19:28:33.784 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.770+0000 7fce08304700 1 -- 172.21.15.167:0/2192230658 <== osd.3 v2:172.21.15.167:6808/30036 24 ==== osd_op_reply(106 .bucket.meta.myfoo:9a9df965-6612-45d4-a45f-6e3c889bacc2.4223.1 [call out=48b,getxattrs out=218b,stat out=16b] v0'0 uv2 ondisk = 0) v8 ==== 290+0+282 (crc 0 0 0) 0x7fce0004a040 con 0x56388264a720 2023-10-05T19:28:33.784 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.770+0000 7fce0a7cfe40 10 cache put: name=default.rgw.meta+root+.bucket.meta.myfoo:9a9df965-6612-45d4-a45f-6e3c889bacc2.4223.1 info.flags=0x16 2023-10-05T19:28:33.784 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.770+0000 7fce0a7cfe40 10 adding default.rgw.meta+root+.bucket.meta.myfoo:9a9df965-6612-45d4-a45f-6e3c889bacc2.4223.1 to cache LRU end 2023-10-05T19:28:33.784 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.770+0000 7fce0a7cfe40 10 updating xattr: name=ceph.objclass.version bl.length()=42 2023-10-05T19:28:33.784 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.770+0000 7fce0a7cfe40 10 updating xattr: name=user.rgw.acl bl.length()=123 2023-10-05T19:28:33.784 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.770+0000 7fce0a7cfe40 20 get_system_obj_state: s->obj_tag was set empty 2023-10-05T19:28:33.785 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.770+0000 7fce0a7cfe40 20 Read xattr: user.rgw.acl 2023-10-05T19:28:33.785 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.770+0000 7fce0a7cfe40 20 Read xattr: user.rgw.idtag 2023-10-05T19:28:33.785 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.770+0000 7fce0a7cfe40 10 cache get: name=default.rgw.meta+root+.bucket.meta.myfoo:9a9df965-6612-45d4-a45f-6e3c889bacc2.4223.1 : type miss (requested=0x13, cached=0x16) 2023-10-05T19:28:33.785 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.770+0000 7fce0a7cfe40 20 rados->read ofs=0 len=0 2023-10-05T19:28:33.785 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.770+0000 7fce0a7cfe40 1 -- 172.21.15.167:0/2192230658 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:107 8.11 8:8bb4ed75:root::.bucket.meta.myfoo%3a9a9df965-6612-45d4-a45f-6e3c889bacc2.4223.1:head [call version.check_conds in=122b,call version.read in=11b,read 0~0,getxattrs] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x5638826d87a0 con 0x56388264a720 2023-10-05T19:28:33.785 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.770+0000 7fce08304700 1 -- 172.21.15.167:0/2192230658 <== osd.3 v2:172.21.15.167:6808/30036 25 ==== osd_op_reply(107 .bucket.meta.myfoo:9a9df965-6612-45d4-a45f-6e3c889bacc2.4223.1 [call,call out=48b,read 0~352 out=352b,getxattrs out=218b] v0'0 uv2 ondisk = 0) v8 ==== 332+0+618 (crc 0 0 0) 0x7fce0004a040 con 0x56388264a720 2023-10-05T19:28:33.785 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.770+0000 7fce0a7cfe40 20 rados_obj.operate() r=0 bl.length=352 2023-10-05T19:28:33.786 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.770+0000 7fce0a7cfe40 10 cache put: name=default.rgw.meta+root+.bucket.meta.myfoo:9a9df965-6612-45d4-a45f-6e3c889bacc2.4223.1 info.flags=0x13 2023-10-05T19:28:33.786 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.770+0000 7fce0a7cfe40 10 moving default.rgw.meta+root+.bucket.meta.myfoo:9a9df965-6612-45d4-a45f-6e3c889bacc2.4223.1 to cache LRU end 2023-10-05T19:28:33.786 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.770+0000 7fce0a7cfe40 10 updating xattr: name=ceph.objclass.version bl.length()=42 2023-10-05T19:28:33.786 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.770+0000 7fce0a7cfe40 10 updating xattr: name=user.rgw.acl bl.length()=123 2023-10-05T19:28:33.786 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.770+0000 7fce0a7cfe40 10 chain_cache_entry: cache_locator=default.rgw.meta+root+.bucket.meta.myfoo:9a9df965-6612-45d4-a45f-6e3c889bacc2.4223.1 2023-10-05T19:28:33.786 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.770+0000 7fce0a7cfe40 20 get_system_obj_state: rctx=0x7fff1dd76658 obj=default.rgw.meta:users.uid:foo state=0x5638824c22b0 s->prefetch_data=0 2023-10-05T19:28:33.786 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.770+0000 7fce0a7cfe40 10 cache get: name=default.rgw.meta+users.uid+foo : type miss (requested=0x16, cached=0x13) 2023-10-05T19:28:33.786 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.770+0000 7fce0a7cfe40 1 -- 172.21.15.167:0/2192230658 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:108 8.18 8:1e6ca5ed:users.uid::foo:head [call version.read in=11b,getxattrs,stat] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x5638826d8600 con 0x5638826162a0 2023-10-05T19:28:33.787 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.770+0000 7fce08b05700 1 -- 172.21.15.167:0/2192230658 <== osd.1 v2:172.21.15.167:6800/30035 37 ==== osd_op_reply(108 foo [call out=48b,getxattrs out=75b,stat out=16b] v0'0 uv11 ondisk = 0) v8 ==== 231+0+139 (crc 0 0 0) 0x7fcdf8050040 con 0x5638826162a0 2023-10-05T19:28:33.787 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.770+0000 7fce0a7cfe40 10 cache put: name=default.rgw.meta+users.uid+foo info.flags=0x16 2023-10-05T19:28:33.787 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.770+0000 7fce0a7cfe40 10 moving default.rgw.meta+users.uid+foo to cache LRU end 2023-10-05T19:28:33.787 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.770+0000 7fce0a7cfe40 10 updating xattr: name=ceph.objclass.version bl.length()=42 2023-10-05T19:28:33.787 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.770+0000 7fce0a7cfe40 20 get_system_obj_state: s->obj_tag was set empty 2023-10-05T19:28:33.787 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.770+0000 7fce0a7cfe40 20 Read xattr: user.rgw.idtag 2023-10-05T19:28:33.787 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.770+0000 7fce0a7cfe40 10 cache get: name=default.rgw.meta+users.uid+foo : hit (requested=0x13, cached=0x17) 2023-10-05T19:28:33.787 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.770+0000 7fce0a7cfe40 1 -- 172.21.15.167:0/2192230658 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:109 8.1 8:810902b1:users.uid::fud.buckets:head [call user.remove_bucket in=58b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x5638826dafb0 con 0x7fcdfc007030 2023-10-05T19:28:33.787 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.774+0000 7fce08b05700 1 -- 172.21.15.167:0/2192230658 <== osd.0 v2:172.21.15.118:6800/27690 29 ==== osd_op_reply(109 fud.buckets [call] v30'3 uv2 ondisk = 0) v8 ==== 155+0+0 (crc 0 0 0) 0x7fcdf8050040 con 0x7fcdfc007030 2023-10-05T19:28:33.788 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.774+0000 7fce0a7cfe40 1 -- 172.21.15.167:0/2192230658 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:110 8.11 8:8bb4ed75:root::.bucket.meta.myfoo%3a9a9df965-6612-45d4-a45f-6e3c889bacc2.4223.1:head [delete,create,call version.check_conds in=122b,call version.set in=58b,writefull 0~352 in=352b,setxattr user.rgw.acl (123) in=135b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x5638826e0fe0 con 0x56388264a720 2023-10-05T19:28:33.788 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.778+0000 7fce08304700 1 -- 172.21.15.167:0/2192230658 <== osd.3 v2:172.21.15.167:6808/30036 26 ==== osd_op_reply(110 .bucket.meta.myfoo:9a9df965-6612-45d4-a45f-6e3c889bacc2.4223.1 [delete,create,call,call,writefull 0~352,setxattr (123)] v30'3 uv3 ondisk = 0) v8 ==== 416+0+0 (crc 0 0 0) 0x7fce0004a040 con 0x56388264a720 2023-10-05T19:28:33.788 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.778+0000 7fce0a7cfe40 10 cache put: name=default.rgw.meta+root+.bucket.meta.myfoo:9a9df965-6612-45d4-a45f-6e3c889bacc2.4223.1 info.flags=0x17 2023-10-05T19:28:33.788 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.778+0000 7fce0a7cfe40 10 moving default.rgw.meta+root+.bucket.meta.myfoo:9a9df965-6612-45d4-a45f-6e3c889bacc2.4223.1 to cache LRU end 2023-10-05T19:28:33.788 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.778+0000 7fce0a7cfe40 10 updating xattr: name=user.rgw.acl bl.length()=123 2023-10-05T19:28:33.788 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.778+0000 7fce0a7cfe40 10 distributing notification oid=default.rgw.control:notify.7 cni=[op: 0, obj: default.rgw.meta:root:.bucket.meta.myfoo:9a9df965-6612-45d4-a45f-6e3c889bacc2.4223.1, ofs0, ns] 2023-10-05T19:28:33.788 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.778+0000 7fce0a7cfe40 1 -- 172.21.15.167:0/2192230658 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:111 7.9 7:93e5b521:::notify.7:head [notify cookie 94800706408272 in=737b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x5638826d6410 con 0x7fcdfc007030 2023-10-05T19:28:33.790 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.778+0000 7fce08b05700 1 -- 172.21.15.167:0/2192230658 <== osd.0 v2:172.21.15.118:6800/27690 30 ==== watch-notify(notify (1) cookie 94800705834688 notify 128849018887 ret 0) v3 ==== 767+0+0 (crc 0 0 0) 0x7fcdf803d040 con 0x7fcdfc007030 2023-10-05T19:28:33.791 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.778+0000 7fce06300700 10 rgw watcher librados: RGWWatcher::handle_notify() notify_id 128849018887 cookie 94800705834688 notifier 4473 bl.length()=725 2023-10-05T19:28:33.791 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.778+0000 7fce08b05700 1 -- 172.21.15.167:0/2192230658 <== osd.0 v2:172.21.15.118:6800/27690 31 ==== osd_op_reply(111 notify.7 [notify cookie 94800706408272 out=8b] v0'0 uv123 ondisk = 0) v8 ==== 152+0+8 (crc 0 0 0) 0x7fcdf8050040 con 0x7fcdfc007030 2023-10-05T19:28:33.791 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.778+0000 7fce06300700 10 rgw watcher librados: cache put: name=default.rgw.meta+root+.bucket.meta.myfoo:9a9df965-6612-45d4-a45f-6e3c889bacc2.4223.1 info.flags=0x17 2023-10-05T19:28:33.791 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.778+0000 7fce06300700 10 rgw watcher librados: moving default.rgw.meta+root+.bucket.meta.myfoo:9a9df965-6612-45d4-a45f-6e3c889bacc2.4223.1 to cache LRU end 2023-10-05T19:28:33.791 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.778+0000 7fce06300700 10 rgw watcher librados: updating xattr: name=user.rgw.acl bl.length()=123 2023-10-05T19:28:33.791 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.778+0000 7fce06300700 1 -- 172.21.15.167:0/2192230658 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:112 7.9 7:93e5b521:::notify.7:head [notify-ack in=20b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x7fcde8003d90 con 0x7fcdfc007030 2023-10-05T19:28:33.791 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.778+0000 7fce08b05700 1 -- 172.21.15.167:0/2192230658 <== osd.0 v2:172.21.15.118:6800/27690 32 ==== osd_op_reply(112 notify.7 [notify-ack] v0'0 uv123 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7fcdf8050040 con 0x7fcdfc007030 2023-10-05T19:28:33.792 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.778+0000 7fce08b05700 1 -- 172.21.15.167:0/2192230658 <== osd.0 v2:172.21.15.118:6800/27690 33 ==== watch-notify(notify_complete (2) cookie 94800706408272 notify 128849018887 ret 0) v3 ==== 42+0+48 (crc 0 0 0) 0x7fcdf8002ab0 con 0x7fcdfc007030 2023-10-05T19:28:33.792 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.778+0000 7fce0a7cfe40 20 handle_bi_update(): bucket=:myfoo[9a9df965-6612-45d4-a45f-6e3c889bacc2.4223.1]): orig_sources= new_sources= 2023-10-05T19:28:33.792 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.778+0000 7fce0a7cfe40 20 handle_bi_update(): bucket=:myfoo[9a9df965-6612-45d4-a45f-6e3c889bacc2.4223.1]): potential sources added=[] removed=[] 2023-10-05T19:28:33.792 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.778+0000 7fce0a7cfe40 20 handle_bi_update(): bucket=:myfoo[9a9df965-6612-45d4-a45f-6e3c889bacc2.4223.1]): orig_dests= new_dests= 2023-10-05T19:28:33.792 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.778+0000 7fce0a7cfe40 20 handle_bi_update(): bucket=:myfoo[9a9df965-6612-45d4-a45f-6e3c889bacc2.4223.1]): potential dests added=[] removed=[] 2023-10-05T19:28:33.792 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.778+0000 7fce0a7cfe40 1 -- 172.21.15.167:0/2192230658 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:113 8.9 8:96d98480:users.uid::foo.buckets:head [call user.set_buckets_info in=207b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x5638826df7d0 con 0x7fce0001fc20 2023-10-05T19:28:33.792 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.782+0000 7fce07b03700 1 -- 172.21.15.167:0/2192230658 <== osd.2 v2:172.21.15.167:6816/30037 19 ==== osd_op_reply(113 foo.buckets [call] v30'22 uv22 ondisk = 0) v8 ==== 155+0+0 (crc 0 0 0) 0x7fcdfc03f020 con 0x7fce0001fc20 2023-10-05T19:28:33.793 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.782+0000 7fce0a7cfe40 1 -- 172.21.15.167:0/2192230658 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:114 8.9 8:91b71841:root::myfoo:head [delete,create,call version.inc in=38b,writefull 0~165 in=165b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x5638826e07e0 con 0x7fce0001fc20 2023-10-05T19:28:33.794 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.786+0000 7fce07b03700 1 -- 172.21.15.167:0/2192230658 <== osd.2 v2:172.21.15.167:6816/30037 20 ==== osd_op_reply(114 myfoo [delete,create,call,writefull 0~165] v30'23 uv23 ondisk = 0) v8 ==== 275+0+0 (crc 0 0 0) 0x7fcdfc03f020 con 0x7fce0001fc20 2023-10-05T19:28:33.795 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.786+0000 7fce0a7cfe40 10 cache put: name=default.rgw.meta+root+myfoo info.flags=0x7 2023-10-05T19:28:33.795 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.786+0000 7fce0a7cfe40 10 moving default.rgw.meta+root+myfoo to cache LRU end 2023-10-05T19:28:33.795 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.786+0000 7fce0a7cfe40 10 distributing notification oid=default.rgw.control:notify.6 cni=[op: 0, obj: default.rgw.meta:root:myfoo, ofs0, ns] 2023-10-05T19:28:33.795 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.786+0000 7fce0a7cfe40 1 -- 172.21.15.167:0/2192230658 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:115 7.14 7:2b04a3e9:::notify.6:head [notify cookie 94800706364048 in=326b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x5638826dfcc0 con 0x7fcdfc007030 2023-10-05T19:28:33.795 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.786+0000 7fce08b05700 1 -- 172.21.15.167:0/2192230658 <== osd.0 v2:172.21.15.118:6800/27690 34 ==== watch-notify(notify (1) cookie 94800705832112 notify 128849018888 ret 0) v3 ==== 356+0+0 (crc 0 0 0) 0x7fcdf8002ab0 con 0x7fcdfc007030 2023-10-05T19:28:33.795 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.786+0000 7fce06300700 10 rgw watcher librados: RGWWatcher::handle_notify() notify_id 128849018888 cookie 94800705832112 notifier 4473 bl.length()=314 2023-10-05T19:28:33.796 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.786+0000 7fce08b05700 1 -- 172.21.15.167:0/2192230658 <== osd.0 v2:172.21.15.118:6800/27690 35 ==== osd_op_reply(115 notify.6 [notify cookie 94800706364048 out=8b] v0'0 uv123 ondisk = 0) v8 ==== 152+0+8 (crc 0 0 0) 0x7fcdf8050040 con 0x7fcdfc007030 2023-10-05T19:28:33.796 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.786+0000 7fce06300700 10 rgw watcher librados: cache put: name=default.rgw.meta+root+myfoo info.flags=0x7 2023-10-05T19:28:33.796 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.786+0000 7fce06300700 10 rgw watcher librados: moving default.rgw.meta+root+myfoo to cache LRU end 2023-10-05T19:28:33.796 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.786+0000 7fce06300700 1 -- 172.21.15.167:0/2192230658 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:116 7.14 7:2b04a3e9:::notify.6:head [notify-ack in=20b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x7fcde8003d90 con 0x7fcdfc007030 2023-10-05T19:28:33.796 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.786+0000 7fce08b05700 1 -- 172.21.15.167:0/2192230658 <== osd.0 v2:172.21.15.118:6800/27690 36 ==== watch-notify(notify_complete (2) cookie 94800706364048 notify 128849018888 ret 0) v3 ==== 42+0+48 (crc 0 0 0) 0x7fcdf80334d0 con 0x7fcdfc007030 2023-10-05T19:28:33.796 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.786+0000 7fce08b05700 1 -- 172.21.15.167:0/2192230658 <== osd.0 v2:172.21.15.118:6800/27690 37 ==== osd_op_reply(116 notify.6 [notify-ack] v0'0 uv123 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7fcdf8050040 con 0x7fcdfc007030 2023-10-05T19:28:33.796 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.790+0000 7fce0a7cfe40 1 -- 172.21.15.167:0/2192230658 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:117 7.1f 7:f95f44c2:::notify.0:head [watch unwatch cookie 94800705811056] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x5638826285b0 con 0x7fce0001fc20 2023-10-05T19:28:33.797 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.790+0000 7fce07b03700 1 -- 172.21.15.167:0/2192230658 <== osd.2 v2:172.21.15.167:6816/30037 21 ==== osd_op_reply(117 notify.0 [watch unwatch cookie 94800705811056] v30'125 uv123 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7fcdfc03f020 con 0x7fce0001fc20 2023-10-05T19:28:33.797 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.790+0000 7fce0a7cfe40 20 remove_watcher() i=0 2023-10-05T19:28:33.797 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.790+0000 7fce0a7cfe40 2 removed watcher, disabling cache 2023-10-05T19:28:33.798 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.790+0000 7fce0a7cfe40 1 -- 172.21.15.167:0/2192230658 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:118 7.0 7:05bf5b68:::notify.1:head [watch unwatch cookie 94800705817520] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x5638826285b0 con 0x5638826162a0 2023-10-05T19:28:33.800 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.794+0000 7fce08b05700 1 -- 172.21.15.167:0/2192230658 <== osd.1 v2:172.21.15.167:6800/30035 38 ==== osd_op_reply(118 notify.1 [watch unwatch cookie 94800705817520] v30'125 uv123 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7fcdf8050040 con 0x5638826162a0 2023-10-05T19:28:33.800 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.794+0000 7fce0a7cfe40 20 remove_watcher() i=1 2023-10-05T19:28:33.801 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.794+0000 7fce0a7cfe40 1 -- 172.21.15.167:0/2192230658 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:119 7.15 7:a93a5511:::notify.2:head [watch unwatch cookie 94800705820976] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x5638826285b0 con 0x56388264a720 2023-10-05T19:28:33.803 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.798+0000 7fce08304700 1 -- 172.21.15.167:0/2192230658 <== osd.3 v2:172.21.15.167:6808/30036 27 ==== osd_op_reply(119 notify.2 [watch unwatch cookie 94800705820976] v30'125 uv123 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7fce0004a040 con 0x56388264a720 2023-10-05T19:28:33.803 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.798+0000 7fce0a7cfe40 20 remove_watcher() i=2 2023-10-05T19:28:33.803 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.798+0000 7fce0a7cfe40 1 -- 172.21.15.167:0/2192230658 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:120 7.e 7:7759931f:::notify.3:head [watch unwatch cookie 94800705818624] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x5638826285b0 con 0x7fce0001fc20 2023-10-05T19:28:33.806 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.798+0000 7fce07b03700 1 -- 172.21.15.167:0/2192230658 <== osd.2 v2:172.21.15.167:6816/30037 22 ==== osd_op_reply(120 notify.3 [watch unwatch cookie 94800705818624] v30'125 uv123 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7fcdfc03f020 con 0x7fce0001fc20 2023-10-05T19:28:33.806 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.798+0000 7fce0a7cfe40 20 remove_watcher() i=3 2023-10-05T19:28:33.806 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.798+0000 7fce0a7cfe40 1 -- 172.21.15.167:0/2192230658 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:121 7.d 7:b4812045:::notify.4:head [watch unwatch cookie 94800705826784] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x5638826285b0 con 0x5638826162a0 2023-10-05T19:28:33.808 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.802+0000 7fce08b05700 1 -- 172.21.15.167:0/2192230658 <== osd.1 v2:172.21.15.167:6800/30035 39 ==== osd_op_reply(121 notify.4 [watch unwatch cookie 94800705826784] v30'125 uv123 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7fcdf8050040 con 0x5638826162a0 2023-10-05T19:28:33.809 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.802+0000 7fce0a7cfe40 20 remove_watcher() i=4 2023-10-05T19:28:33.809 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.802+0000 7fce0a7cfe40 1 -- 172.21.15.167:0/2192230658 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:122 7.3 7:c609908c:::notify.5:head [watch unwatch cookie 94800705831008] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x5638826285b0 con 0x7fcdfc007030 2023-10-05T19:28:33.811 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.806+0000 7fce08b05700 1 -- 172.21.15.167:0/2192230658 <== osd.0 v2:172.21.15.118:6800/27690 38 ==== osd_op_reply(122 notify.5 [watch unwatch cookie 94800705831008] v30'125 uv123 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7fcdf8050040 con 0x7fcdfc007030 2023-10-05T19:28:33.811 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.806+0000 7fce0a7cfe40 20 remove_watcher() i=5 2023-10-05T19:28:33.811 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.806+0000 7fce0a7cfe40 1 -- 172.21.15.167:0/2192230658 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:123 7.14 7:2b04a3e9:::notify.6:head [watch unwatch cookie 94800705832112] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x5638826285b0 con 0x7fcdfc007030 2023-10-05T19:28:33.814 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.806+0000 7fce08b05700 1 -- 172.21.15.167:0/2192230658 <== osd.0 v2:172.21.15.118:6800/27690 39 ==== osd_op_reply(123 notify.6 [watch unwatch cookie 94800705832112] v30'125 uv123 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7fcdf8050040 con 0x7fcdfc007030 2023-10-05T19:28:33.814 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.810+0000 7fce0a7cfe40 20 remove_watcher() i=6 2023-10-05T19:28:33.814 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.810+0000 7fce0a7cfe40 1 -- 172.21.15.167:0/2192230658 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:124 7.9 7:93e5b521:::notify.7:head [watch unwatch cookie 94800705834688] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x5638826285b0 con 0x7fcdfc007030 2023-10-05T19:28:33.817 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.810+0000 7fce08b05700 1 -- 172.21.15.167:0/2192230658 <== osd.0 v2:172.21.15.118:6800/27690 40 ==== osd_op_reply(124 notify.7 [watch unwatch cookie 94800705834688] v30'125 uv123 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7fcdf8050040 con 0x7fcdfc007030 2023-10-05T19:28:33.818 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.810+0000 7fce0a7cfe40 20 remove_watcher() i=7 2023-10-05T19:28:33.819 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.814+0000 7fcdd9fe3700 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2023-10-05T19:28:33.819 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.814+0000 7fce0a7cfe40 1 -- 172.21.15.167:0/2192230658 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:125 6.4 6:22d26bf9:::data_loggenerations_metadata:head [watch unwatch cookie 94800705836160] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x5638826285b0 con 0x5638826162a0 2023-10-05T19:28:33.824 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.814+0000 7fce08b05700 1 -- 172.21.15.167:0/2192230658 <== osd.1 v2:172.21.15.167:6800/30035 40 ==== osd_op_reply(125 data_loggenerations_metadata [watch unwatch cookie 94800705836160] v30'121 uv1 ondisk = 0) v8 ==== 172+0+0 (crc 0 0 0) 0x7fcdf8050040 con 0x5638826162a0 2023-10-05T19:28:33.824 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.818+0000 7fce0a7cfe40 1 -- 172.21.15.167:0/2192230658 >> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] conn(0x7fcdfc007030 msgr2=0x7fcdfc01d430 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-05T19:28:33.824 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.818+0000 7fce0a7cfe40 1 --2- 172.21.15.167:0/2192230658 >> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] conn(0x7fcdfc007030 0x7fcdfc01d430 crc :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-05T19:28:33.824 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.818+0000 7fce0a7cfe40 1 -- 172.21.15.167:0/2192230658 >> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] conn(0x5638826162a0 msgr2=0x5638826468f0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-05T19:28:33.824 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.818+0000 7fce0a7cfe40 1 --2- 172.21.15.167:0/2192230658 >> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] conn(0x5638826162a0 0x5638826468f0 crc :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-05T19:28:33.824 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.818+0000 7fce0a7cfe40 1 -- 172.21.15.167:0/2192230658 >> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] conn(0x7fce0001fc20 msgr2=0x7fce0002a000 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-05T19:28:33.824 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.818+0000 7fce0a7cfe40 1 --2- 172.21.15.167:0/2192230658 >> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] conn(0x7fce0001fc20 0x7fce0002a000 crc :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-05T19:28:33.824 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.818+0000 7fce0a7cfe40 1 -- 172.21.15.167:0/2192230658 >> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] conn(0x56388264a720 msgr2=0x56388264cb00 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-05T19:28:33.824 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.818+0000 7fce0a7cfe40 1 --2- 172.21.15.167:0/2192230658 >> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] conn(0x56388264a720 0x56388264cb00 crc :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-05T19:28:33.824 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.818+0000 7fce0a7cfe40 1 -- 172.21.15.167:0/2192230658 >> [v2:172.21.15.167:6824/29889,v1:172.21.15.167:6825/29889] conn(0x7fcdf4030c50 msgr2=0x7fcdf4033100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-05T19:28:33.825 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.818+0000 7fce0a7cfe40 1 --2- 172.21.15.167:0/2192230658 >> [v2:172.21.15.167:6824/29889,v1:172.21.15.167:6825/29889] conn(0x7fcdf4030c50 0x7fcdf4033100 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7fcdfc00afd0 tx=0x7fcdfc014000 comp rx=0 tx=0).stop 2023-10-05T19:28:33.825 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.818+0000 7fce08b05700 1 -- 172.21.15.167:0/2192230658 reap_dead start 2023-10-05T19:28:33.825 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.818+0000 7fce0a7cfe40 1 -- 172.21.15.167:0/2192230658 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x563882611cc0 msgr2=0x56388263ae30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-05T19:28:33.825 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.818+0000 7fce0a7cfe40 1 --2- 172.21.15.167:0/2192230658 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x563882611cc0 0x56388263ae30 secure :-1 s=READY pgs=383 cs=0 l=1 rev1=1 crypto rx=0x7fce00001350 tx=0x7fce00014000 comp rx=0 tx=0).stop 2023-10-05T19:28:33.825 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.818+0000 7fce0a7cfe40 1 -- 172.21.15.167:0/2192230658 shutdown_connections 2023-10-05T19:28:33.825 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.818+0000 7fce0a7cfe40 1 --2- 172.21.15.167:0/2192230658 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x563882611cc0 0x56388263ae30 unknown :-1 s=CLOSED pgs=383 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-05T19:28:33.826 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.818+0000 7fce0a7cfe40 1 -- 172.21.15.167:0/2192230658 >> 172.21.15.167:0/2192230658 conn(0x56388260d6e0 msgr2=0x56388262c3d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-05T19:28:33.826 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.818+0000 7fce0a7cfe40 1 -- 172.21.15.167:0/2192230658 shutdown_connections 2023-10-05T19:28:33.826 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.818+0000 7fce0a7cfe40 1 -- 172.21.15.167:0/2192230658 wait complete. 2023-10-05T19:28:33.826 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.818+0000 7fce0a7cfe40 1 -- 172.21.15.167:0/2262901231 >> [v2:172.21.15.167:6824/29889,v1:172.21.15.167:6825/29889] conn(0x7fcd48030c50 msgr2=0x7fcd48033100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-05T19:28:33.826 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.818+0000 7fce0a7cfe40 1 --2- 172.21.15.167:0/2262901231 >> [v2:172.21.15.167:6824/29889,v1:172.21.15.167:6825/29889] conn(0x7fcd48030c50 0x7fcd48033100 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x563882658ad0 tx=0x7fcdf8000f90 comp rx=0 tx=0).stop 2023-10-05T19:28:33.826 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.818+0000 7fce0a7cfe40 1 -- 172.21.15.167:0/2262901231 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x563882663240 msgr2=0x563882668d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-05T19:28:33.827 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.818+0000 7fce0a7cfe40 1 --2- 172.21.15.167:0/2262901231 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x563882663240 0x563882668d80 secure :-1 s=READY pgs=385 cs=0 l=1 rev1=1 crypto rx=0x56388265d2c0 tx=0x7fcdfc039000 comp rx=0 tx=0).stop 2023-10-05T19:28:33.827 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.818+0000 7fce0a7cfe40 1 -- 172.21.15.167:0/2262901231 shutdown_connections 2023-10-05T19:28:33.827 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.818+0000 7fce0a7cfe40 1 --2- 172.21.15.167:0/2262901231 >> [v2:172.21.15.167:6824/29889,v1:172.21.15.167:6825/29889] conn(0x7fcd48030c50 0x7fcd48033100 unknown :-1 s=CLOSED pgs=135 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-05T19:28:33.827 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.818+0000 7fce0a7cfe40 1 --2- 172.21.15.167:0/2262901231 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x563882663240 0x563882668d80 unknown :-1 s=CLOSED pgs=385 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-05T19:28:33.827 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.818+0000 7fce0a7cfe40 1 -- 172.21.15.167:0/2262901231 >> 172.21.15.167:0/2262901231 conn(0x56388265e600 msgr2=0x56388265c6a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-05T19:28:33.827 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.818+0000 7fce0a7cfe40 1 -- 172.21.15.167:0/2262901231 shutdown_connections 2023-10-05T19:28:33.827 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.818+0000 7fce0a7cfe40 1 -- 172.21.15.167:0/2262901231 wait complete. 2023-10-05T19:28:33.830 INFO:tasks.util.rgw:rgwadmin: client.0 : ['user', 'rm', '--uid', 'fud'] 2023-10-05T19:28:33.830 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'ceph', 'user', 'rm', '--uid', 'fud'] 2023-10-05T19:28:33.830 DEBUG:teuthology.orchestra.run.smithi167:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster ceph user rm --uid fud 2023-10-05T19:28:33.876 INFO:teuthology.orchestra.run.smithi167.stderr:ignoring --setuser ceph since I am not root 2023-10-05T19:28:33.876 INFO:teuthology.orchestra.run.smithi167.stderr:ignoring --setgroup ceph since I am not root 2023-10-05T19:28:33.877 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.866+0000 7fe385185e40 1 Processor -- start 2023-10-05T19:28:33.877 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.866+0000 7fe385185e40 1 -- start start 2023-10-05T19:28:33.877 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.866+0000 7fe385185e40 1 --2- >> v2:172.21.15.118:3300/0 conn(0x560cbdf4ee30 0x560cbdf4f200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-05T19:28:33.877 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.866+0000 7fe385185e40 1 -- --> v1:172.21.15.118:6789/0 -- auth(proto 0 26 bytes epoch 0) v1 -- 0x560cbde9bbd0 con 0x560cbdf4f740 2023-10-05T19:28:33.877 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.866+0000 7fe385185e40 1 -- --> v2:172.21.15.118:3300/0 -- mon_getmap magic: 0 v1 -- 0x560cbde58ee0 con 0x560cbdf4ee30 2023-10-05T19:28:33.877 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.866+0000 7fe382cba700 1 --2- >> v2:172.21.15.118:3300/0 conn(0x560cbdf4ee30 0x560cbdf4f200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-05T19:28:33.878 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.866+0000 7fe3824b9700 1 --1- >> v1:172.21.15.118:6789/0 conn(0x560cbdf4f740 0x560cbdf501e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.118:6789/0 says I am v1:172.21.15.167:56680/0 (socket says 172.21.15.167:56680) 2023-10-05T19:28:33.878 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.866+0000 7fe3824b9700 1 -- 172.21.15.167:0/3175703175 learned_addr learned my addr 172.21.15.167:0/3175703175 (peer_addr_for_me v1:172.21.15.167:0/0) 2023-10-05T19:28:33.878 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.866+0000 7fe382cba700 1 -- 172.21.15.167:0/3175703175 >> v1:172.21.15.118:6789/0 conn(0x560cbdf4f740 legacy=0x560cbdf501e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-05T19:28:33.878 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.866+0000 7fe382cba700 1 -- 172.21.15.167:0/3175703175 --> v2:172.21.15.118:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x560cbdf508f0 con 0x560cbdf4ee30 2023-10-05T19:28:33.878 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.870+0000 7fe382cba700 1 --2- 172.21.15.167:0/3175703175 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x560cbdf4ee30 0x560cbdf4f200 secure :-1 s=READY pgs=387 cs=0 l=1 rev1=1 crypto rx=0x7fe37400a5d0 tx=0x7fe374005a50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b0f3283313de7a70 server_cookie=0 in_seq=0 out_seq=0 2023-10-05T19:28:33.879 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.870+0000 7fe381cb8700 1 -- 172.21.15.167:0/3175703175 <== mon.0 v2:172.21.15.118:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fe3740086a0 con 0x560cbdf4ee30 2023-10-05T19:28:33.879 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.870+0000 7fe381cb8700 1 -- 172.21.15.167:0/3175703175 <== mon.0 v2:172.21.15.118:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe374008800 con 0x560cbdf4ee30 2023-10-05T19:28:33.879 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.870+0000 7fe381cb8700 1 -- 172.21.15.167:0/3175703175 <== mon.0 v2:172.21.15.118:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fe374008ad0 con 0x560cbdf4ee30 2023-10-05T19:28:33.879 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.870+0000 7fe385185e40 1 -- 172.21.15.167:0/3175703175 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x560cbdf4ee30 msgr2=0x560cbdf4f200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-05T19:28:33.879 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.870+0000 7fe385185e40 1 --2- 172.21.15.167:0/3175703175 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x560cbdf4ee30 0x560cbdf4f200 secure :-1 s=READY pgs=387 cs=0 l=1 rev1=1 crypto rx=0x7fe37400a5d0 tx=0x7fe374005a50 comp rx=0 tx=0).stop 2023-10-05T19:28:33.879 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.870+0000 7fe385185e40 1 -- 172.21.15.167:0/3175703175 shutdown_connections 2023-10-05T19:28:33.880 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.870+0000 7fe385185e40 1 --2- 172.21.15.167:0/3175703175 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x560cbdf4ee30 0x560cbdf4f200 unknown :-1 s=CLOSED pgs=387 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-05T19:28:33.880 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.870+0000 7fe385185e40 1 -- 172.21.15.167:0/3175703175 >> 172.21.15.167:0/3175703175 conn(0x560cbdf4a850 msgr2=0x560cbdf4cc60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-05T19:28:33.880 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.870+0000 7fe385185e40 1 -- 172.21.15.167:0/3175703175 shutdown_connections 2023-10-05T19:28:33.880 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.870+0000 7fe385185e40 1 -- 172.21.15.167:0/3175703175 wait complete. 2023-10-05T19:28:33.886 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.870+0000 7fe385185e40 1 Processor -- start 2023-10-05T19:28:33.886 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.874+0000 7fe385185e40 1 -- start start 2023-10-05T19:28:33.886 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.874+0000 7fe385185e40 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x560cbdf4ee30 0x560cbdf6df00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-05T19:28:33.886 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.874+0000 7fe385185e40 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x560cbde1ce30 con 0x560cbdf4ee30 2023-10-05T19:28:33.886 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.874+0000 7fe382cba700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x560cbdf4ee30 0x560cbdf6df00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-05T19:28:33.887 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.874+0000 7fe382cba700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x560cbdf4ee30 0x560cbdf6df00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.118:3300/0 says I am v2:172.21.15.167:58694/0 (socket says 172.21.15.167:58694) 2023-10-05T19:28:33.887 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.874+0000 7fe382cba700 1 -- 172.21.15.167:0/846744261 learned_addr learned my addr 172.21.15.167:0/846744261 (peer_addr_for_me v2:172.21.15.167:0/0) 2023-10-05T19:28:33.887 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.874+0000 7fe382cba700 1 -- 172.21.15.167:0/846744261 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe374005cf0 con 0x560cbdf4ee30 2023-10-05T19:28:33.887 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.874+0000 7fe382cba700 1 --2- 172.21.15.167:0/846744261 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x560cbdf4ee30 0x560cbdf6df00 secure :-1 s=READY pgs=388 cs=0 l=1 rev1=1 crypto rx=0x7fe374000c00 tx=0x7fe374015000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=aa3eeb9768c15c1d server_cookie=0 in_seq=0 out_seq=0 2023-10-05T19:28:33.887 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.874+0000 7fe380cb6700 1 -- 172.21.15.167:0/846744261 <== mon.0 v2:172.21.15.118:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fe374008970 con 0x560cbdf4ee30 2023-10-05T19:28:33.887 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.874+0000 7fe380cb6700 1 -- 172.21.15.167:0/846744261 <== mon.0 v2:172.21.15.118:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe374008af0 con 0x560cbdf4ee30 2023-10-05T19:28:33.888 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.874+0000 7fe380cb6700 1 -- 172.21.15.167:0/846744261 <== mon.0 v2:172.21.15.118:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fe374008de0 con 0x560cbdf4ee30 2023-10-05T19:28:33.888 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.874+0000 7fe385185e40 1 -- 172.21.15.167:0/846744261 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x560cbdf4ee30 msgr2=0x560cbdf6df00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-05T19:28:33.888 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.874+0000 7fe385185e40 1 --2- 172.21.15.167:0/846744261 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x560cbdf4ee30 0x560cbdf6df00 secure :-1 s=READY pgs=388 cs=0 l=1 rev1=1 crypto rx=0x7fe374000c00 tx=0x7fe374015000 comp rx=0 tx=0).stop 2023-10-05T19:28:33.888 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.874+0000 7fe385185e40 1 -- 172.21.15.167:0/846744261 shutdown_connections 2023-10-05T19:28:33.888 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.874+0000 7fe385185e40 1 --2- 172.21.15.167:0/846744261 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x560cbdf4ee30 0x560cbdf6df00 unknown :-1 s=CLOSED pgs=388 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-05T19:28:33.888 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.874+0000 7fe385185e40 1 -- 172.21.15.167:0/846744261 >> 172.21.15.167:0/846744261 conn(0x560cbdf4a850 msgr2=0x560cbdf69ac0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-05T19:28:33.888 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.874+0000 7fe385185e40 1 -- 172.21.15.167:0/846744261 shutdown_connections 2023-10-05T19:28:33.888 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.878+0000 7fe385185e40 1 -- 172.21.15.167:0/846744261 wait complete. 2023-10-05T19:28:33.889 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.878+0000 7fe385185e40 1 Processor -- start 2023-10-05T19:28:33.889 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.878+0000 7fe385185e40 1 -- start start 2023-10-05T19:28:33.889 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.878+0000 7fe385185e40 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x560cbdf4ee30 0x560cbdf74b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-05T19:28:33.889 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.878+0000 7fe385185e40 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x560cbddefed0 con 0x560cbdf4ee30 2023-10-05T19:28:33.889 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.878+0000 7fe382cba700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x560cbdf4ee30 0x560cbdf74b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-05T19:28:33.889 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.878+0000 7fe382cba700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x560cbdf4ee30 0x560cbdf74b10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.118:3300/0 says I am v2:172.21.15.167:58700/0 (socket says 172.21.15.167:58700) 2023-10-05T19:28:33.889 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.878+0000 7fe382cba700 1 -- 172.21.15.167:0/1226698073 learned_addr learned my addr 172.21.15.167:0/1226698073 (peer_addr_for_me v2:172.21.15.167:0/0) 2023-10-05T19:28:33.890 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.878+0000 7fe382cba700 1 -- 172.21.15.167:0/1226698073 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x560cbdf508f0 con 0x560cbdf4ee30 2023-10-05T19:28:33.890 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.878+0000 7fe382cba700 1 --2- 172.21.15.167:0/1226698073 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x560cbdf4ee30 0x560cbdf74b10 secure :-1 s=READY pgs=389 cs=0 l=1 rev1=1 crypto rx=0x7fe37400c010 tx=0x7fe374003050 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-05T19:28:33.890 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.878+0000 7fe37affd700 1 -- 172.21.15.167:0/1226698073 <== mon.0 v2:172.21.15.118:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fe374012ba0 con 0x560cbdf4ee30 2023-10-05T19:28:33.890 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.878+0000 7fe37affd700 1 -- 172.21.15.167:0/1226698073 <== mon.0 v2:172.21.15.118:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe37402a070 con 0x560cbdf4ee30 2023-10-05T19:28:33.890 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.878+0000 7fe37affd700 1 -- 172.21.15.167:0/1226698073 <== mon.0 v2:172.21.15.118:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fe374012ea0 con 0x560cbdf4ee30 2023-10-05T19:28:33.890 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.878+0000 7fe385185e40 1 -- 172.21.15.167:0/1226698073 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x560cbdf75050 con 0x560cbdf4ee30 2023-10-05T19:28:33.890 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.878+0000 7fe385185e40 1 -- 172.21.15.167:0/1226698073 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x560cbdf754f0 con 0x560cbdf4ee30 2023-10-05T19:28:33.895 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.882+0000 7fe37affd700 1 -- 172.21.15.167:0/1226698073 <== mon.0 v2:172.21.15.118:3300/0 4 ==== mgrmap(e 5) v1 ==== 40329+0+0 (secure 0 0 0) 0x7fe374019070 con 0x560cbdf4ee30 2023-10-05T19:28:33.896 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.882+0000 7fe37affd700 1 --2- 172.21.15.167:0/1226698073 >> [v2:172.21.15.167:6824/29889,v1:172.21.15.167:6825/29889] conn(0x7fe364030c40 0x7fe3640330f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-05T19:28:33.896 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.882+0000 7fe37affd700 1 -- 172.21.15.167:0/1226698073 <== mon.0 v2:172.21.15.118:3300/0 5 ==== osd_map(30..30 src has 1..30) v4 ==== 6057+0+0 (secure 0 0 0) 0x7fe37402c070 con 0x560cbdf4ee30 2023-10-05T19:28:33.896 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.882+0000 7fe3824b9700 1 --2- 172.21.15.167:0/1226698073 >> [v2:172.21.15.167:6824/29889,v1:172.21.15.167:6825/29889] conn(0x7fe364030c40 0x7fe3640330f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-05T19:28:33.896 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.882+0000 7fe385185e40 20 rados->read ofs=0 len=0 2023-10-05T19:28:33.896 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.882+0000 7fe385185e40 1 --2- 172.21.15.167:0/1226698073 >> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] conn(0x560cbdf4f740 0x560cbdf80540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-05T19:28:33.896 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.882+0000 7fe385185e40 1 -- 172.21.15.167:0/1226698073 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:1 5.12 5:49953fa1:::default.realm:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x560cbdf80c70 con 0x560cbdf4f740 2023-10-05T19:28:33.896 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.882+0000 7fe3834bb700 1 --2- 172.21.15.167:0/1226698073 >> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] conn(0x560cbdf4f740 0x560cbdf80540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-05T19:28:33.897 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.882+0000 7fe3824b9700 1 --2- 172.21.15.167:0/1226698073 >> [v2:172.21.15.167:6824/29889,v1:172.21.15.167:6825/29889] conn(0x7fe364030c40 0x7fe3640330f0 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7fe36c004170 tx=0x7fe36c001380 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-05T19:28:33.897 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.882+0000 7fe3834bb700 1 --2- 172.21.15.167:0/1226698073 >> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] conn(0x560cbdf4f740 0x560cbdf80540 crc :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-05T19:28:33.897 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.882+0000 7fe3834bb700 1 -- 172.21.15.167:0/1226698073 <== osd.1 v2:172.21.15.167:6800/30035 1 ==== osd_op_reply(1 default.realm [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 ==== 157+0+0 (crc 0 0 0) 0x7fe37c00a5a0 con 0x560cbdf4f740 2023-10-05T19:28:33.897 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.882+0000 7fe385185e40 20 rados_obj.operate() r=-2 bl.length=0 2023-10-05T19:28:33.897 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.882+0000 7fe385185e40 20 realm 2023-10-05T19:28:33.897 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.882+0000 7fe385185e40 20 rados->read ofs=0 len=0 2023-10-05T19:28:33.897 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.882+0000 7fe385185e40 1 -- 172.21.15.167:0/1226698073 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:2 5.12 5:49953fa1:::default.realm:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x560cbdf7db70 con 0x560cbdf4f740 2023-10-05T19:28:33.897 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.886+0000 7fe3834bb700 1 -- 172.21.15.167:0/1226698073 <== osd.1 v2:172.21.15.167:6800/30035 2 ==== osd_op_reply(2 default.realm [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 ==== 157+0+0 (crc 0 0 0) 0x7fe37c00a5a0 con 0x560cbdf4f740 2023-10-05T19:28:33.898 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.886+0000 7fe385185e40 20 rados_obj.operate() r=-2 bl.length=0 2023-10-05T19:28:33.898 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.886+0000 7fe385185e40 4 RGWPeriod::init failed to init realm id : (2) No such file or directory 2023-10-05T19:28:33.898 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.886+0000 7fe385185e40 20 rados->read ofs=0 len=0 2023-10-05T19:28:33.898 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.886+0000 7fe385185e40 1 -- 172.21.15.167:0/1226698073 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:3 5.12 5:49953fa1:::default.realm:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x560cbdf817a0 con 0x560cbdf4f740 2023-10-05T19:28:33.898 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.886+0000 7fe3834bb700 1 -- 172.21.15.167:0/1226698073 <== osd.1 v2:172.21.15.167:6800/30035 3 ==== osd_op_reply(3 default.realm [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 ==== 157+0+0 (crc 0 0 0) 0x7fe37c00a5a0 con 0x560cbdf4f740 2023-10-05T19:28:33.898 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.886+0000 7fe385185e40 20 rados_obj.operate() r=-2 bl.length=0 2023-10-05T19:28:33.898 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.886+0000 7fe385185e40 20 rados->read ofs=0 len=0 2023-10-05T19:28:33.899 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.886+0000 7fe385185e40 1 -- 172.21.15.167:0/1226698073 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:4 5.f 5:f43b6ece:::zone_names.default:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x560cbdf813b0 con 0x560cbdf4f740 2023-10-05T19:28:33.899 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.886+0000 7fe3834bb700 1 -- 172.21.15.167:0/1226698073 <== osd.1 v2:172.21.15.167:6800/30035 4 ==== osd_op_reply(4 zone_names.default [read 0~46 out=46b] v0'0 uv1 ondisk = 0) v8 ==== 162+0+46 (crc 0 0 0) 0x7fe37c00a5a0 con 0x560cbdf4f740 2023-10-05T19:28:33.899 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.886+0000 7fe385185e40 20 rados_obj.operate() r=0 bl.length=46 2023-10-05T19:28:33.899 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.886+0000 7fe385185e40 20 rados->read ofs=0 len=0 2023-10-05T19:28:33.899 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.886+0000 7fe385185e40 1 -- 172.21.15.167:0/1226698073 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:5 5.1d 5:b8374eb4:::zone_info.9a9df965-6612-45d4-a45f-6e3c889bacc2:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x560cbdf82630 con 0x560cbdf4f740 2023-10-05T19:28:33.899 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.886+0000 7fe3834bb700 1 -- 172.21.15.167:0/1226698073 <== osd.1 v2:172.21.15.167:6800/30035 5 ==== osd_op_reply(5 zone_info.9a9df965-6612-45d4-a45f-6e3c889bacc2 [read 0~874 out=874b] v0'0 uv2 ondisk = 0) v8 ==== 190+0+874 (crc 0 0 0) 0x7fe37c00a5a0 con 0x560cbdf4f740 2023-10-05T19:28:33.900 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.886+0000 7fe385185e40 20 rados_obj.operate() r=0 bl.length=874 2023-10-05T19:28:33.900 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.886+0000 7fe385185e40 20 searching for the correct realm 2023-10-05T19:28:33.900 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.886+0000 7fe385185e40 1 --2- 172.21.15.167:0/1226698073 >> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] conn(0x560cbdf843f0 0x560cbdf867d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-05T19:28:33.900 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.886+0000 7fe385185e40 1 -- 172.21.15.167:0/1226698073 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:6 5.0 5:00000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x560cbdf86d10 con 0x560cbdf843f0 2023-10-05T19:28:33.900 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.886+0000 7fe382cba700 1 --2- 172.21.15.167:0/1226698073 >> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] conn(0x560cbdf843f0 0x560cbdf867d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-05T19:28:33.900 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.886+0000 7fe382cba700 1 --2- 172.21.15.167:0/1226698073 >> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] conn(0x560cbdf843f0 0x560cbdf867d0 crc :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-05T19:28:33.902 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.890+0000 7fe382cba700 1 -- 172.21.15.167:0/1226698073 <== osd.3 v2:172.21.15.167:6808/30036 1 ==== osd_op_reply(6 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fe374022020 con 0x560cbdf843f0 2023-10-05T19:28:33.902 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.890+0000 7fe382cba700 1 --2- 172.21.15.167:0/1226698073 >> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] conn(0x7fe374007030 0x7fe374001620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-05T19:28:33.902 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.890+0000 7fe382cba700 1 -- 172.21.15.167:0/1226698073 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:7 5.10 5:08000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x560cbdf86d10 con 0x7fe374007030 2023-10-05T19:28:33.902 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.890+0000 7fe3824b9700 1 --2- 172.21.15.167:0/1226698073 >> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] conn(0x7fe374007030 0x7fe374001620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-05T19:28:33.902 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.890+0000 7fe3824b9700 1 --2- 172.21.15.167:0/1226698073 >> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] conn(0x7fe374007030 0x7fe374001620 crc :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-05T19:28:33.903 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.890+0000 7fe3824b9700 1 -- 172.21.15.167:0/1226698073 <== osd.2 v2:172.21.15.167:6816/30037 1 ==== osd_op_reply(7 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fe36c006070 con 0x7fe374007030 2023-10-05T19:28:33.903 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.890+0000 7fe3824b9700 1 --2- 172.21.15.167:0/1226698073 >> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] conn(0x7fe36c00d3d0 0x7fe36c00d820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-05T19:28:33.903 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.890+0000 7fe3824b9700 1 -- 172.21.15.167:0/1226698073 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:8 5.8 5:10000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x560cbdf86d10 con 0x7fe36c00d3d0 2023-10-05T19:28:33.903 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.890+0000 7fe3834bb700 1 --2- 172.21.15.167:0/1226698073 >> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] conn(0x7fe36c00d3d0 0x7fe36c00d820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-05T19:28:33.903 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.894+0000 7fe3834bb700 1 --2- 172.21.15.167:0/1226698073 >> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] conn(0x7fe36c00d3d0 0x7fe36c00d820 crc :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-05T19:28:33.903 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.894+0000 7fe3834bb700 1 -- 172.21.15.167:0/1226698073 <== osd.0 v2:172.21.15.118:6800/27690 1 ==== osd_op_reply(8 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fe37c00a5a0 con 0x7fe36c00d3d0 2023-10-05T19:28:33.903 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.894+0000 7fe3834bb700 1 -- 172.21.15.167:0/1226698073 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:9 5.18 5:18000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x560cbdf86d10 con 0x560cbdf4f740 2023-10-05T19:28:33.903 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.894+0000 7fe3834bb700 1 -- 172.21.15.167:0/1226698073 <== osd.1 v2:172.21.15.167:6800/30035 6 ==== osd_op_reply(9 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fe37c00a5a0 con 0x560cbdf4f740 2023-10-05T19:28:33.903 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.894+0000 7fe3834bb700 1 -- 172.21.15.167:0/1226698073 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:10 5.4 5:20000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x560cbdf86d10 con 0x560cbdf4f740 2023-10-05T19:28:33.903 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.894+0000 7fe3834bb700 1 -- 172.21.15.167:0/1226698073 <== osd.1 v2:172.21.15.167:6800/30035 7 ==== osd_op_reply(10 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fe37c00a5a0 con 0x560cbdf4f740 2023-10-05T19:28:33.903 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.894+0000 7fe3834bb700 1 -- 172.21.15.167:0/1226698073 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:11 5.14 5:28000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x560cbdf86d10 con 0x7fe36c00d3d0 2023-10-05T19:28:33.904 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.894+0000 7fe3834bb700 1 -- 172.21.15.167:0/1226698073 <== osd.0 v2:172.21.15.118:6800/27690 2 ==== osd_op_reply(11 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fe37c00a5a0 con 0x7fe36c00d3d0 2023-10-05T19:28:33.904 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.894+0000 7fe3834bb700 1 -- 172.21.15.167:0/1226698073 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:12 5.c 5:30000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x560cbdf86d10 con 0x7fe36c00d3d0 2023-10-05T19:28:33.904 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.898+0000 7fe3834bb700 1 -- 172.21.15.167:0/1226698073 <== osd.0 v2:172.21.15.118:6800/27690 3 ==== osd_op_reply(12 [pgnls start_epoch 30 out=112b] v23'1 uv1 ondisk = 1) v8 ==== 144+0+112 (crc 0 0 0) 0x7fe37c00a5a0 con 0x7fe36c00d3d0 2023-10-05T19:28:33.905 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.898+0000 7fe3834bb700 1 -- 172.21.15.167:0/1226698073 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:13 5.1c 5:38000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x560cbdf86d10 con 0x7fe374007030 2023-10-05T19:28:33.905 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.898+0000 7fe3824b9700 1 -- 172.21.15.167:0/1226698073 <== osd.2 v2:172.21.15.167:6816/30037 2 ==== osd_op_reply(13 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fe36c006070 con 0x7fe374007030 2023-10-05T19:28:33.905 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.898+0000 7fe3824b9700 1 -- 172.21.15.167:0/1226698073 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:14 5.2 5:40000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x560cbdf86d10 con 0x560cbdf843f0 2023-10-05T19:28:33.905 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.898+0000 7fe382cba700 1 -- 172.21.15.167:0/1226698073 <== osd.3 v2:172.21.15.167:6808/30036 2 ==== osd_op_reply(14 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fe374022020 con 0x560cbdf843f0 2023-10-05T19:28:33.905 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.898+0000 7fe382cba700 1 -- 172.21.15.167:0/1226698073 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:15 5.12 5:48000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x560cbdf86d10 con 0x560cbdf4f740 2023-10-05T19:28:33.905 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.898+0000 7fe3834bb700 1 -- 172.21.15.167:0/1226698073 <== osd.1 v2:172.21.15.167:6800/30035 8 ==== osd_op_reply(15 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fe37c00a5a0 con 0x560cbdf4f740 2023-10-05T19:28:33.906 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.898+0000 7fe3834bb700 1 -- 172.21.15.167:0/1226698073 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:16 5.a 5:50000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x560cbdf86d10 con 0x7fe36c00d3d0 2023-10-05T19:28:33.906 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.902+0000 7fe3834bb700 1 -- 172.21.15.167:0/1226698073 <== osd.0 v2:172.21.15.118:6800/27690 4 ==== osd_op_reply(16 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fe37c00a5a0 con 0x7fe36c00d3d0 2023-10-05T19:28:33.906 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.902+0000 7fe3834bb700 1 -- 172.21.15.167:0/1226698073 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:17 5.1a 5:58000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x560cbdf86d10 con 0x560cbdf4f740 2023-10-05T19:28:33.907 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.902+0000 7fe3834bb700 1 -- 172.21.15.167:0/1226698073 <== osd.1 v2:172.21.15.167:6800/30035 9 ==== osd_op_reply(17 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fe37c00a5a0 con 0x560cbdf4f740 2023-10-05T19:28:33.907 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.902+0000 7fe3834bb700 1 -- 172.21.15.167:0/1226698073 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:18 5.6 5:60000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x560cbdf86d10 con 0x7fe374007030 2023-10-05T19:28:33.908 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.902+0000 7fe3824b9700 1 -- 172.21.15.167:0/1226698073 <== osd.2 v2:172.21.15.167:6816/30037 3 ==== osd_op_reply(18 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fe36c006070 con 0x7fe374007030 2023-10-05T19:28:33.908 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.902+0000 7fe3824b9700 1 -- 172.21.15.167:0/1226698073 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:19 5.16 5:68000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x560cbdf86d10 con 0x560cbdf843f0 2023-10-05T19:28:33.908 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.902+0000 7fe382cba700 1 -- 172.21.15.167:0/1226698073 <== osd.3 v2:172.21.15.167:6808/30036 3 ==== osd_op_reply(19 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fe374022020 con 0x560cbdf843f0 2023-10-05T19:28:33.908 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.902+0000 7fe382cba700 1 -- 172.21.15.167:0/1226698073 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:20 5.e 5:70000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x560cbdf86d10 con 0x560cbdf843f0 2023-10-05T19:28:33.909 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.902+0000 7fe382cba700 1 -- 172.21.15.167:0/1226698073 <== osd.3 v2:172.21.15.167:6808/30036 4 ==== osd_op_reply(20 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fe374022020 con 0x560cbdf843f0 2023-10-05T19:28:33.909 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.902+0000 7fe382cba700 1 -- 172.21.15.167:0/1226698073 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:21 5.1e 5:78000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x560cbdf86d10 con 0x7fe36c00d3d0 2023-10-05T19:28:33.910 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.902+0000 7fe3834bb700 1 -- 172.21.15.167:0/1226698073 <== osd.0 v2:172.21.15.118:6800/27690 5 ==== osd_op_reply(21 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fe37c00a5a0 con 0x7fe36c00d3d0 2023-10-05T19:28:33.910 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.906+0000 7fe3834bb700 1 -- 172.21.15.167:0/1226698073 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:22 5.1 5:80000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x560cbdf86d10 con 0x560cbdf843f0 2023-10-05T19:28:33.910 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.906+0000 7fe382cba700 1 -- 172.21.15.167:0/1226698073 <== osd.3 v2:172.21.15.167:6808/30036 5 ==== osd_op_reply(22 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fe374022020 con 0x560cbdf843f0 2023-10-05T19:28:33.910 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.906+0000 7fe382cba700 1 -- 172.21.15.167:0/1226698073 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:23 5.11 5:88000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x560cbdf86d10 con 0x560cbdf843f0 2023-10-05T19:28:33.911 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.906+0000 7fe382cba700 1 -- 172.21.15.167:0/1226698073 <== osd.3 v2:172.21.15.167:6808/30036 6 ==== osd_op_reply(23 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fe374022020 con 0x560cbdf843f0 2023-10-05T19:28:33.911 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.906+0000 7fe382cba700 1 -- 172.21.15.167:0/1226698073 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:24 5.9 5:90000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x560cbdf86d10 con 0x560cbdf4f740 2023-10-05T19:28:33.912 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.906+0000 7fe3834bb700 1 -- 172.21.15.167:0/1226698073 <== osd.1 v2:172.21.15.167:6800/30035 10 ==== osd_op_reply(24 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fe37c00a5a0 con 0x560cbdf4f740 2023-10-05T19:28:33.912 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.906+0000 7fe3834bb700 1 -- 172.21.15.167:0/1226698073 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:25 5.19 5:98000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x560cbdf86d10 con 0x560cbdf4f740 2023-10-05T19:28:33.913 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.906+0000 7fe3834bb700 1 -- 172.21.15.167:0/1226698073 <== osd.1 v2:172.21.15.167:6800/30035 11 ==== osd_op_reply(25 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fe37c00a5a0 con 0x560cbdf4f740 2023-10-05T19:28:33.913 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.906+0000 7fe3834bb700 1 -- 172.21.15.167:0/1226698073 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:26 5.5 5:a0000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x560cbdf86d10 con 0x7fe36c00d3d0 2023-10-05T19:28:33.914 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.906+0000 7fe3834bb700 1 -- 172.21.15.167:0/1226698073 <== osd.0 v2:172.21.15.118:6800/27690 6 ==== osd_op_reply(26 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fe37c00a5a0 con 0x7fe36c00d3d0 2023-10-05T19:28:33.914 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.906+0000 7fe3834bb700 1 -- 172.21.15.167:0/1226698073 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:27 5.15 5:a8000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x560cbdf86d10 con 0x560cbdf843f0 2023-10-05T19:28:33.914 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.910+0000 7fe382cba700 1 -- 172.21.15.167:0/1226698073 <== osd.3 v2:172.21.15.167:6808/30036 7 ==== osd_op_reply(27 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fe374022020 con 0x560cbdf843f0 2023-10-05T19:28:33.914 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.910+0000 7fe382cba700 1 -- 172.21.15.167:0/1226698073 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:28 5.d 5:b0000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x560cbdf86d10 con 0x7fe374007030 2023-10-05T19:28:33.915 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.910+0000 7fe3824b9700 1 -- 172.21.15.167:0/1226698073 <== osd.2 v2:172.21.15.167:6816/30037 4 ==== osd_op_reply(28 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fe36c006070 con 0x7fe374007030 2023-10-05T19:28:33.915 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.910+0000 7fe3824b9700 1 -- 172.21.15.167:0/1226698073 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:29 5.1d 5:b8000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x560cbdf86d10 con 0x560cbdf4f740 2023-10-05T19:28:33.916 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.910+0000 7fe3834bb700 1 -- 172.21.15.167:0/1226698073 <== osd.1 v2:172.21.15.167:6800/30035 12 ==== osd_op_reply(29 [pgnls start_epoch 30 out=107b] v30'2 uv2 ondisk = 1) v8 ==== 144+0+107 (crc 0 0 0) 0x7fe37c00a5a0 con 0x560cbdf4f740 2023-10-05T19:28:33.916 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.910+0000 7fe3834bb700 1 -- 172.21.15.167:0/1226698073 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:30 5.3 5:c0000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x560cbdf86d10 con 0x7fe36c00d3d0 2023-10-05T19:28:33.917 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.910+0000 7fe3834bb700 1 -- 172.21.15.167:0/1226698073 <== osd.0 v2:172.21.15.118:6800/27690 7 ==== osd_op_reply(30 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fe37c00a5a0 con 0x7fe36c00d3d0 2023-10-05T19:28:33.917 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.910+0000 7fe3834bb700 1 -- 172.21.15.167:0/1226698073 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:31 5.13 5:c8000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x560cbdf86d10 con 0x560cbdf843f0 2023-10-05T19:28:33.917 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.910+0000 7fe382cba700 1 -- 172.21.15.167:0/1226698073 <== osd.3 v2:172.21.15.167:6808/30036 8 ==== osd_op_reply(31 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fe374022020 con 0x560cbdf843f0 2023-10-05T19:28:33.917 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.910+0000 7fe382cba700 1 -- 172.21.15.167:0/1226698073 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:32 5.b 5:d0000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x560cbdf86d10 con 0x7fe36c00d3d0 2023-10-05T19:28:33.918 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.914+0000 7fe3834bb700 1 -- 172.21.15.167:0/1226698073 <== osd.0 v2:172.21.15.118:6800/27690 8 ==== osd_op_reply(32 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fe37c00a5a0 con 0x7fe36c00d3d0 2023-10-05T19:28:33.918 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.914+0000 7fe3834bb700 1 -- 172.21.15.167:0/1226698073 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:33 5.1b 5:d8000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x560cbdf86d10 con 0x7fe374007030 2023-10-05T19:28:33.919 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.914+0000 7fe3824b9700 1 -- 172.21.15.167:0/1226698073 <== osd.2 v2:172.21.15.167:6816/30037 5 ==== osd_op_reply(33 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fe36c006070 con 0x7fe374007030 2023-10-05T19:28:33.919 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.914+0000 7fe3824b9700 1 -- 172.21.15.167:0/1226698073 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:34 5.7 5:e0000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x560cbdf86d10 con 0x560cbdf843f0 2023-10-05T19:28:33.920 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.914+0000 7fe382cba700 1 -- 172.21.15.167:0/1226698073 <== osd.3 v2:172.21.15.167:6808/30036 9 ==== osd_op_reply(34 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fe374022020 con 0x560cbdf843f0 2023-10-05T19:28:33.920 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.914+0000 7fe382cba700 1 -- 172.21.15.167:0/1226698073 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:35 5.17 5:e8000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x560cbdf86d10 con 0x7fe36c00d3d0 2023-10-05T19:28:33.921 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.914+0000 7fe3834bb700 1 -- 172.21.15.167:0/1226698073 <== osd.0 v2:172.21.15.118:6800/27690 9 ==== osd_op_reply(35 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fe37c00a5a0 con 0x7fe36c00d3d0 2023-10-05T19:28:33.921 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.914+0000 7fe3834bb700 1 -- 172.21.15.167:0/1226698073 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:36 5.f 5:f0000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x560cbdf86d10 con 0x560cbdf4f740 2023-10-05T19:28:33.922 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.914+0000 7fe3834bb700 1 -- 172.21.15.167:0/1226698073 <== osd.1 v2:172.21.15.167:6800/30035 13 ==== osd_op_reply(36 [pgnls start_epoch 30 out=115b] v23'2 uv2 ondisk = 1) v8 ==== 144+0+115 (crc 0 0 0) 0x7fe37c00a5a0 con 0x560cbdf4f740 2023-10-05T19:28:33.922 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.918+0000 7fe3834bb700 1 -- 172.21.15.167:0/1226698073 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:37 5.1f 5:f8000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x560cbdf86d10 con 0x7fe374007030 2023-10-05T19:28:33.923 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.918+0000 7fe3824b9700 1 -- 172.21.15.167:0/1226698073 <== osd.2 v2:172.21.15.167:6816/30037 6 ==== osd_op_reply(37 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fe36c006070 con 0x7fe374007030 2023-10-05T19:28:33.923 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.918+0000 7fe385185e40 20 RGWRados::pool_iterate: got zonegroup_info.4f1f719d-9648-417b-b10c-e5d939825419 2023-10-05T19:28:33.923 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.918+0000 7fe385185e40 20 RGWRados::pool_iterate: got zone_info.9a9df965-6612-45d4-a45f-6e3c889bacc2 2023-10-05T19:28:33.923 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.918+0000 7fe385185e40 20 RGWRados::pool_iterate: got zone_names.default 2023-10-05T19:28:33.924 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.918+0000 7fe385185e40 20 RGWRados::pool_iterate: got zonegroups_names.default 2023-10-05T19:28:33.924 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.918+0000 7fe385185e40 20 rados->read ofs=0 len=0 2023-10-05T19:28:33.924 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.918+0000 7fe385185e40 1 -- 172.21.15.167:0/1226698073 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:38 5.12 5:49953fa1:::default.realm:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x560cbdf831a0 con 0x560cbdf4f740 2023-10-05T19:28:33.925 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.918+0000 7fe3834bb700 1 -- 172.21.15.167:0/1226698073 <== osd.1 v2:172.21.15.167:6800/30035 14 ==== osd_op_reply(38 default.realm [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 ==== 157+0+0 (crc 0 0 0) 0x7fe37c00a5a0 con 0x560cbdf4f740 2023-10-05T19:28:33.925 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.918+0000 7fe385185e40 20 rados_obj.operate() r=-2 bl.length=0 2023-10-05T19:28:33.925 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.918+0000 7fe385185e40 20 rados->read ofs=0 len=0 2023-10-05T19:28:33.925 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.918+0000 7fe385185e40 1 -- 172.21.15.167:0/1226698073 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:39 5.f 5:f4c53578:::zonegroups_names.default:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x560cbdf83a90 con 0x560cbdf4f740 2023-10-05T19:28:33.925 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.918+0000 7fe3834bb700 1 -- 172.21.15.167:0/1226698073 <== osd.1 v2:172.21.15.167:6800/30035 15 ==== osd_op_reply(39 zonegroups_names.default [read 0~46 out=46b] v0'0 uv2 ondisk = 0) v8 ==== 168+0+46 (crc 0 0 0) 0x7fe37c00a5a0 con 0x560cbdf4f740 2023-10-05T19:28:33.925 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.918+0000 7fe385185e40 20 rados_obj.operate() r=0 bl.length=46 2023-10-05T19:28:33.925 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.918+0000 7fe385185e40 20 rados->read ofs=0 len=0 2023-10-05T19:28:33.926 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.918+0000 7fe385185e40 1 -- 172.21.15.167:0/1226698073 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:40 5.c 5:30d299a5:::zonegroup_info.4f1f719d-9648-417b-b10c-e5d939825419:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x560cbdf83a90 con 0x7fe36c00d3d0 2023-10-05T19:28:33.926 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.918+0000 7fe3834bb700 1 -- 172.21.15.167:0/1226698073 <== osd.0 v2:172.21.15.118:6800/27690 10 ==== osd_op_reply(40 zonegroup_info.4f1f719d-9648-417b-b10c-e5d939825419 [read 0~362 out=362b] v0'0 uv1 ondisk = 0) v8 ==== 195+0+362 (crc 0 0 0) 0x7fe37c00a5a0 con 0x7fe36c00d3d0 2023-10-05T19:28:33.927 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.918+0000 7fe385185e40 20 rados_obj.operate() r=0 bl.length=362 2023-10-05T19:28:33.927 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.918+0000 7fe385185e40 20 rados->read ofs=0 len=0 2023-10-05T19:28:33.927 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.918+0000 7fe385185e40 1 -- 172.21.15.167:0/1226698073 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:41 5.13 5:cbbf6c60:::converted:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x560cbdf83a90 con 0x560cbdf843f0 2023-10-05T19:28:33.927 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.922+0000 7fe382cba700 1 -- 172.21.15.167:0/1226698073 <== osd.3 v2:172.21.15.167:6808/30036 10 ==== osd_op_reply(41 converted [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 ==== 153+0+0 (crc 0 0 0) 0x7fe374050040 con 0x560cbdf843f0 2023-10-05T19:28:33.927 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.922+0000 7fe385185e40 20 rados_obj.operate() r=-2 bl.length=0 2023-10-05T19:28:33.927 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.922+0000 7fe385185e40 20 rados->read ofs=0 len=0 2023-10-05T19:28:33.927 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.922+0000 7fe385185e40 1 -- 172.21.15.167:0/1226698073 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:42 5.12 5:49953fa1:::default.realm:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x560cbdf87180 con 0x560cbdf4f740 2023-10-05T19:28:33.928 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.922+0000 7fe3834bb700 1 -- 172.21.15.167:0/1226698073 <== osd.1 v2:172.21.15.167:6800/30035 16 ==== osd_op_reply(42 default.realm [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 ==== 157+0+0 (crc 0 0 0) 0x7fe37c00a5a0 con 0x560cbdf4f740 2023-10-05T19:28:33.928 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.922+0000 7fe385185e40 20 rados_obj.operate() r=-2 bl.length=0 2023-10-05T19:28:33.928 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.922+0000 7fe385185e40 20 rados->read ofs=0 len=0 2023-10-05T19:28:33.928 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.922+0000 7fe385185e40 1 -- 172.21.15.167:0/1226698073 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:43 5.f 5:f4c53578:::zonegroups_names.default:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x560cbdf836c0 con 0x560cbdf4f740 2023-10-05T19:28:33.928 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.922+0000 7fe3834bb700 1 -- 172.21.15.167:0/1226698073 <== osd.1 v2:172.21.15.167:6800/30035 17 ==== osd_op_reply(43 zonegroups_names.default [read 0~46 out=46b] v0'0 uv2 ondisk = 0) v8 ==== 168+0+46 (crc 0 0 0) 0x7fe37c00a5a0 con 0x560cbdf4f740 2023-10-05T19:28:33.929 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.922+0000 7fe385185e40 20 rados_obj.operate() r=0 bl.length=46 2023-10-05T19:28:33.929 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.922+0000 7fe385185e40 1 -- 172.21.15.167:0/1226698073 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:44 5.0 5:00000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x560cbdf836c0 con 0x560cbdf843f0 2023-10-05T19:28:33.931 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.926+0000 7fe382cba700 1 -- 172.21.15.167:0/1226698073 <== osd.3 v2:172.21.15.167:6808/30036 11 ==== osd_op_reply(44 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fe374050040 con 0x560cbdf843f0 2023-10-05T19:28:33.931 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.926+0000 7fe382cba700 1 -- 172.21.15.167:0/1226698073 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:45 5.10 5:08000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x560cbdf836c0 con 0x7fe374007030 2023-10-05T19:28:33.931 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.926+0000 7fe3824b9700 1 -- 172.21.15.167:0/1226698073 <== osd.2 v2:172.21.15.167:6816/30037 7 ==== osd_op_reply(45 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fe36c006070 con 0x7fe374007030 2023-10-05T19:28:33.932 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.926+0000 7fe3824b9700 1 -- 172.21.15.167:0/1226698073 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:46 5.8 5:10000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x560cbdf836c0 con 0x7fe36c00d3d0 2023-10-05T19:28:33.932 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.926+0000 7fe3834bb700 1 -- 172.21.15.167:0/1226698073 <== osd.0 v2:172.21.15.118:6800/27690 11 ==== osd_op_reply(46 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fe37c00a5a0 con 0x7fe36c00d3d0 2023-10-05T19:28:33.933 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.926+0000 7fe3834bb700 1 -- 172.21.15.167:0/1226698073 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:47 5.18 5:18000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x560cbdf831a0 con 0x560cbdf4f740 2023-10-05T19:28:33.933 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.926+0000 7fe3834bb700 1 -- 172.21.15.167:0/1226698073 <== osd.1 v2:172.21.15.167:6800/30035 18 ==== osd_op_reply(47 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fe37c00a5a0 con 0x560cbdf4f740 2023-10-05T19:28:33.933 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.926+0000 7fe3834bb700 1 -- 172.21.15.167:0/1226698073 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:48 5.4 5:20000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x560cbdf831a0 con 0x560cbdf4f740 2023-10-05T19:28:33.934 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.930+0000 7fe3834bb700 1 -- 172.21.15.167:0/1226698073 <== osd.1 v2:172.21.15.167:6800/30035 19 ==== osd_op_reply(48 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fe37c00a5a0 con 0x560cbdf4f740 2023-10-05T19:28:33.934 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.930+0000 7fe3834bb700 1 -- 172.21.15.167:0/1226698073 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:49 5.14 5:28000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x560cbdf831a0 con 0x7fe36c00d3d0 2023-10-05T19:28:33.935 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.930+0000 7fe3834bb700 1 -- 172.21.15.167:0/1226698073 <== osd.0 v2:172.21.15.118:6800/27690 12 ==== osd_op_reply(49 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fe37c00a5a0 con 0x7fe36c00d3d0 2023-10-05T19:28:33.936 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.930+0000 7fe3834bb700 1 -- 172.21.15.167:0/1226698073 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:50 5.c 5:30000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x560cbdf831a0 con 0x7fe36c00d3d0 2023-10-05T19:28:33.936 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.930+0000 7fe3834bb700 1 -- 172.21.15.167:0/1226698073 <== osd.0 v2:172.21.15.118:6800/27690 13 ==== osd_op_reply(50 [pgnls start_epoch 30 out=112b] v23'1 uv1 ondisk = 1) v8 ==== 144+0+112 (crc 0 0 0) 0x7fe37c00a5a0 con 0x7fe36c00d3d0 2023-10-05T19:28:33.936 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.930+0000 7fe3834bb700 1 -- 172.21.15.167:0/1226698073 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:51 5.1c 5:38000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x560cbdf831a0 con 0x7fe374007030 2023-10-05T19:28:33.937 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.930+0000 7fe3824b9700 1 -- 172.21.15.167:0/1226698073 <== osd.2 v2:172.21.15.167:6816/30037 8 ==== osd_op_reply(51 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fe36c006070 con 0x7fe374007030 2023-10-05T19:28:33.937 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.930+0000 7fe3824b9700 1 -- 172.21.15.167:0/1226698073 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:52 5.2 5:40000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x560cbdf831a0 con 0x560cbdf843f0 2023-10-05T19:28:33.938 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.934+0000 7fe382cba700 1 -- 172.21.15.167:0/1226698073 <== osd.3 v2:172.21.15.167:6808/30036 12 ==== osd_op_reply(52 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fe374050040 con 0x560cbdf843f0 2023-10-05T19:28:33.938 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.934+0000 7fe382cba700 1 -- 172.21.15.167:0/1226698073 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:53 5.12 5:48000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x560cbdf831a0 con 0x560cbdf4f740 2023-10-05T19:28:33.939 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.934+0000 7fe3834bb700 1 -- 172.21.15.167:0/1226698073 <== osd.1 v2:172.21.15.167:6800/30035 20 ==== osd_op_reply(53 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fe37c00a5a0 con 0x560cbdf4f740 2023-10-05T19:28:33.939 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.934+0000 7fe3834bb700 1 -- 172.21.15.167:0/1226698073 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:54 5.a 5:50000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x560cbdf831a0 con 0x7fe36c00d3d0 2023-10-05T19:28:33.940 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.934+0000 7fe3834bb700 1 -- 172.21.15.167:0/1226698073 <== osd.0 v2:172.21.15.118:6800/27690 14 ==== osd_op_reply(54 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fe37c00a5a0 con 0x7fe36c00d3d0 2023-10-05T19:28:33.940 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.934+0000 7fe3834bb700 1 -- 172.21.15.167:0/1226698073 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:55 5.1a 5:58000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x560cbdf831a0 con 0x560cbdf4f740 2023-10-05T19:28:33.941 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.934+0000 7fe3834bb700 1 -- 172.21.15.167:0/1226698073 <== osd.1 v2:172.21.15.167:6800/30035 21 ==== osd_op_reply(55 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fe37c00a5a0 con 0x560cbdf4f740 2023-10-05T19:28:33.941 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.934+0000 7fe3834bb700 1 -- 172.21.15.167:0/1226698073 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:56 5.6 5:60000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x560cbdf831a0 con 0x7fe374007030 2023-10-05T19:28:33.942 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.938+0000 7fe3824b9700 1 -- 172.21.15.167:0/1226698073 <== osd.2 v2:172.21.15.167:6816/30037 9 ==== osd_op_reply(56 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fe36c006070 con 0x7fe374007030 2023-10-05T19:28:33.942 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.938+0000 7fe3824b9700 1 -- 172.21.15.167:0/1226698073 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:57 5.16 5:68000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x560cbdf831a0 con 0x560cbdf843f0 2023-10-05T19:28:33.943 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.938+0000 7fe382cba700 1 -- 172.21.15.167:0/1226698073 <== osd.3 v2:172.21.15.167:6808/30036 13 ==== osd_op_reply(57 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fe374050040 con 0x560cbdf843f0 2023-10-05T19:28:33.943 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.938+0000 7fe382cba700 1 -- 172.21.15.167:0/1226698073 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:58 5.e 5:70000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x560cbdf831a0 con 0x560cbdf843f0 2023-10-05T19:28:33.944 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.938+0000 7fe382cba700 1 -- 172.21.15.167:0/1226698073 <== osd.3 v2:172.21.15.167:6808/30036 14 ==== osd_op_reply(58 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fe374050040 con 0x560cbdf843f0 2023-10-05T19:28:33.944 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.938+0000 7fe382cba700 1 -- 172.21.15.167:0/1226698073 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:59 5.1e 5:78000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x560cbdf831a0 con 0x7fe36c00d3d0 2023-10-05T19:28:33.945 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.938+0000 7fe3834bb700 1 -- 172.21.15.167:0/1226698073 <== osd.0 v2:172.21.15.118:6800/27690 15 ==== osd_op_reply(59 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fe37c03f040 con 0x7fe36c00d3d0 2023-10-05T19:28:33.945 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.938+0000 7fe3834bb700 1 -- 172.21.15.167:0/1226698073 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:60 5.1 5:80000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x560cbdf831a0 con 0x560cbdf843f0 2023-10-05T19:28:33.945 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.938+0000 7fe382cba700 1 -- 172.21.15.167:0/1226698073 <== osd.3 v2:172.21.15.167:6808/30036 15 ==== osd_op_reply(60 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fe374050040 con 0x560cbdf843f0 2023-10-05T19:28:33.945 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.938+0000 7fe382cba700 1 -- 172.21.15.167:0/1226698073 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:61 5.11 5:88000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x560cbdf831a0 con 0x560cbdf843f0 2023-10-05T19:28:33.946 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.942+0000 7fe382cba700 1 -- 172.21.15.167:0/1226698073 <== osd.3 v2:172.21.15.167:6808/30036 16 ==== osd_op_reply(61 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fe374050040 con 0x560cbdf843f0 2023-10-05T19:28:33.946 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.942+0000 7fe382cba700 1 -- 172.21.15.167:0/1226698073 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:62 5.9 5:90000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x560cbdf831a0 con 0x560cbdf4f740 2023-10-05T19:28:33.947 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.942+0000 7fe3834bb700 1 -- 172.21.15.167:0/1226698073 <== osd.1 v2:172.21.15.167:6800/30035 22 ==== osd_op_reply(62 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fe37c03f040 con 0x560cbdf4f740 2023-10-05T19:28:33.947 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.942+0000 7fe3834bb700 1 -- 172.21.15.167:0/1226698073 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:63 5.19 5:98000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x560cbdf831a0 con 0x560cbdf4f740 2023-10-05T19:28:33.948 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.942+0000 7fe3834bb700 1 -- 172.21.15.167:0/1226698073 <== osd.1 v2:172.21.15.167:6800/30035 23 ==== osd_op_reply(63 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fe37c03f040 con 0x560cbdf4f740 2023-10-05T19:28:33.948 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.942+0000 7fe3834bb700 1 -- 172.21.15.167:0/1226698073 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:64 5.5 5:a0000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x560cbdf831a0 con 0x7fe36c00d3d0 2023-10-05T19:28:33.949 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.942+0000 7fe3834bb700 1 -- 172.21.15.167:0/1226698073 <== osd.0 v2:172.21.15.118:6800/27690 16 ==== osd_op_reply(64 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fe37c03f040 con 0x7fe36c00d3d0 2023-10-05T19:28:33.949 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.942+0000 7fe3834bb700 1 -- 172.21.15.167:0/1226698073 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:65 5.15 5:a8000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x560cbdf831a0 con 0x560cbdf843f0 2023-10-05T19:28:33.950 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.946+0000 7fe382cba700 1 -- 172.21.15.167:0/1226698073 <== osd.3 v2:172.21.15.167:6808/30036 17 ==== osd_op_reply(65 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fe374050040 con 0x560cbdf843f0 2023-10-05T19:28:33.950 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.946+0000 7fe382cba700 1 -- 172.21.15.167:0/1226698073 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:66 5.d 5:b0000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x560cbdf831a0 con 0x7fe374007030 2023-10-05T19:28:33.951 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.946+0000 7fe3824b9700 1 -- 172.21.15.167:0/1226698073 <== osd.2 v2:172.21.15.167:6816/30037 10 ==== osd_op_reply(66 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fe36c006070 con 0x7fe374007030 2023-10-05T19:28:33.951 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.946+0000 7fe3824b9700 1 -- 172.21.15.167:0/1226698073 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:67 5.1d 5:b8000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x560cbdf831a0 con 0x560cbdf4f740 2023-10-05T19:28:33.952 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.946+0000 7fe3834bb700 1 -- 172.21.15.167:0/1226698073 <== osd.1 v2:172.21.15.167:6800/30035 24 ==== osd_op_reply(67 [pgnls start_epoch 30 out=107b] v30'2 uv2 ondisk = 1) v8 ==== 144+0+107 (crc 0 0 0) 0x7fe37c03f040 con 0x560cbdf4f740 2023-10-05T19:28:33.952 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.946+0000 7fe3834bb700 1 -- 172.21.15.167:0/1226698073 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:68 5.3 5:c0000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x560cbdf831a0 con 0x7fe36c00d3d0 2023-10-05T19:28:33.952 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.946+0000 7fe3834bb700 1 -- 172.21.15.167:0/1226698073 <== osd.0 v2:172.21.15.118:6800/27690 17 ==== osd_op_reply(68 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fe37c03f040 con 0x7fe36c00d3d0 2023-10-05T19:28:33.953 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.946+0000 7fe3834bb700 1 -- 172.21.15.167:0/1226698073 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:69 5.13 5:c8000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x560cbdf831a0 con 0x560cbdf843f0 2023-10-05T19:28:33.953 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.946+0000 7fe382cba700 1 -- 172.21.15.167:0/1226698073 <== osd.3 v2:172.21.15.167:6808/30036 18 ==== osd_op_reply(69 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fe374050040 con 0x560cbdf843f0 2023-10-05T19:28:33.953 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.946+0000 7fe382cba700 1 -- 172.21.15.167:0/1226698073 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:70 5.b 5:d0000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x560cbdf831a0 con 0x7fe36c00d3d0 2023-10-05T19:28:33.954 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.950+0000 7fe3834bb700 1 -- 172.21.15.167:0/1226698073 <== osd.0 v2:172.21.15.118:6800/27690 18 ==== osd_op_reply(70 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fe37c03f040 con 0x7fe36c00d3d0 2023-10-05T19:28:33.954 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.950+0000 7fe3834bb700 1 -- 172.21.15.167:0/1226698073 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:71 5.1b 5:d8000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x560cbdf831a0 con 0x7fe374007030 2023-10-05T19:28:33.955 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.950+0000 7fe3824b9700 1 -- 172.21.15.167:0/1226698073 <== osd.2 v2:172.21.15.167:6816/30037 11 ==== osd_op_reply(71 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fe36c006070 con 0x7fe374007030 2023-10-05T19:28:33.956 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.950+0000 7fe3824b9700 1 -- 172.21.15.167:0/1226698073 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:72 5.7 5:e0000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x560cbdf831a0 con 0x560cbdf843f0 2023-10-05T19:28:33.956 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.950+0000 7fe382cba700 1 -- 172.21.15.167:0/1226698073 <== osd.3 v2:172.21.15.167:6808/30036 19 ==== osd_op_reply(72 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fe374050040 con 0x560cbdf843f0 2023-10-05T19:28:33.956 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.950+0000 7fe382cba700 1 -- 172.21.15.167:0/1226698073 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:73 5.17 5:e8000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x560cbdf831a0 con 0x7fe36c00d3d0 2023-10-05T19:28:33.957 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.950+0000 7fe3834bb700 1 -- 172.21.15.167:0/1226698073 <== osd.0 v2:172.21.15.118:6800/27690 19 ==== osd_op_reply(73 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fe37c03f040 con 0x7fe36c00d3d0 2023-10-05T19:28:33.957 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.950+0000 7fe3834bb700 1 -- 172.21.15.167:0/1226698073 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:74 5.f 5:f0000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x560cbdf831a0 con 0x560cbdf4f740 2023-10-05T19:28:33.958 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.954+0000 7fe3834bb700 1 -- 172.21.15.167:0/1226698073 <== osd.1 v2:172.21.15.167:6800/30035 25 ==== osd_op_reply(74 [pgnls start_epoch 30 out=115b] v23'2 uv2 ondisk = 1) v8 ==== 144+0+115 (crc 0 0 0) 0x7fe37c03f040 con 0x560cbdf4f740 2023-10-05T19:28:33.958 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.954+0000 7fe3834bb700 1 -- 172.21.15.167:0/1226698073 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:75 5.1f 5:f8000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x560cbdf831a0 con 0x7fe374007030 2023-10-05T19:28:33.959 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.954+0000 7fe3824b9700 1 -- 172.21.15.167:0/1226698073 <== osd.2 v2:172.21.15.167:6816/30037 12 ==== osd_op_reply(75 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fe36c006070 con 0x7fe374007030 2023-10-05T19:28:33.959 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.954+0000 7fe385185e40 20 RGWRados::pool_iterate: got zonegroup_info.4f1f719d-9648-417b-b10c-e5d939825419 2023-10-05T19:28:33.960 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.954+0000 7fe385185e40 20 RGWRados::pool_iterate: got zone_info.9a9df965-6612-45d4-a45f-6e3c889bacc2 2023-10-05T19:28:33.960 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.954+0000 7fe385185e40 20 RGWRados::pool_iterate: got zone_names.default 2023-10-05T19:28:33.960 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.954+0000 7fe385185e40 20 RGWRados::pool_iterate: got zonegroups_names.default 2023-10-05T19:28:33.960 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.954+0000 7fe385185e40 20 rados->read ofs=0 len=0 2023-10-05T19:28:33.960 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.954+0000 7fe385185e40 1 -- 172.21.15.167:0/1226698073 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:76 5.f 5:f43b6ece:::zone_names.default:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x560cbdf836c0 con 0x560cbdf4f740 2023-10-05T19:28:33.960 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.954+0000 7fe3834bb700 1 -- 172.21.15.167:0/1226698073 <== osd.1 v2:172.21.15.167:6800/30035 26 ==== osd_op_reply(76 zone_names.default [read 0~46 out=46b] v0'0 uv1 ondisk = 0) v8 ==== 162+0+46 (crc 0 0 0) 0x7fe37c03f040 con 0x560cbdf4f740 2023-10-05T19:28:33.961 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.954+0000 7fe385185e40 20 rados_obj.operate() r=0 bl.length=46 2023-10-05T19:28:33.961 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.954+0000 7fe385185e40 20 rados->read ofs=0 len=0 2023-10-05T19:28:33.961 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.954+0000 7fe385185e40 1 -- 172.21.15.167:0/1226698073 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:77 5.1d 5:b8374eb4:::zone_info.9a9df965-6612-45d4-a45f-6e3c889bacc2:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x560cbdf87720 con 0x560cbdf4f740 2023-10-05T19:28:33.961 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.954+0000 7fe3834bb700 1 -- 172.21.15.167:0/1226698073 <== osd.1 v2:172.21.15.167:6800/30035 27 ==== osd_op_reply(77 zone_info.9a9df965-6612-45d4-a45f-6e3c889bacc2 [read 0~874 out=874b] v0'0 uv2 ondisk = 0) v8 ==== 190+0+874 (crc 0 0 0) 0x7fe37c03f040 con 0x560cbdf4f740 2023-10-05T19:28:33.961 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.954+0000 7fe385185e40 20 rados_obj.operate() r=0 bl.length=874 2023-10-05T19:28:33.961 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.954+0000 7fe385185e40 20 rados->read ofs=0 len=0 2023-10-05T19:28:33.962 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.954+0000 7fe385185e40 1 -- 172.21.15.167:0/1226698073 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:78 5.f 5:f4c53578:::zonegroups_names.default:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x560cbdf87c10 con 0x560cbdf4f740 2023-10-05T19:28:33.962 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.954+0000 7fe3834bb700 1 -- 172.21.15.167:0/1226698073 <== osd.1 v2:172.21.15.167:6800/30035 28 ==== osd_op_reply(78 zonegroups_names.default [read 0~46 out=46b] v0'0 uv2 ondisk = 0) v8 ==== 168+0+46 (crc 0 0 0) 0x7fe37c03f040 con 0x560cbdf4f740 2023-10-05T19:28:33.962 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.954+0000 7fe385185e40 20 rados_obj.operate() r=0 bl.length=46 2023-10-05T19:28:33.962 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.954+0000 7fe385185e40 20 rados->read ofs=0 len=0 2023-10-05T19:28:33.962 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.958+0000 7fe385185e40 1 -- 172.21.15.167:0/1226698073 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:79 5.c 5:30d299a5:::zonegroup_info.4f1f719d-9648-417b-b10c-e5d939825419:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x560cbdf87c10 con 0x7fe36c00d3d0 2023-10-05T19:28:33.963 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.958+0000 7fe3834bb700 1 -- 172.21.15.167:0/1226698073 <== osd.0 v2:172.21.15.118:6800/27690 20 ==== osd_op_reply(79 zonegroup_info.4f1f719d-9648-417b-b10c-e5d939825419 [read 0~362 out=362b] v0'0 uv1 ondisk = 0) v8 ==== 195+0+362 (crc 0 0 0) 0x7fe37c03f040 con 0x7fe36c00d3d0 2023-10-05T19:28:33.963 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.958+0000 7fe385185e40 20 rados_obj.operate() r=0 bl.length=362 2023-10-05T19:28:33.963 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.958+0000 7fe385185e40 20 rados->read ofs=0 len=0 2023-10-05T19:28:33.963 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.958+0000 7fe385185e40 1 -- 172.21.15.167:0/1226698073 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:80 5.3 5:c0ad8cbd:::region_map:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x560cbdf87d10 con 0x7fe36c00d3d0 2023-10-05T19:28:33.964 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.958+0000 7fe3834bb700 1 -- 172.21.15.167:0/1226698073 <== osd.0 v2:172.21.15.118:6800/27690 21 ==== osd_op_reply(80 region_map [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 ==== 154+0+0 (crc 0 0 0) 0x7fe37c03f040 con 0x7fe36c00d3d0 2023-10-05T19:28:33.964 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.958+0000 7fe385185e40 20 rados_obj.operate() r=-2 bl.length=0 2023-10-05T19:28:33.964 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.958+0000 7fe385185e40 20 zone default found 2023-10-05T19:28:33.964 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.958+0000 7fe385185e40 4 Realm: () 2023-10-05T19:28:33.964 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.958+0000 7fe385185e40 4 ZoneGroup: default (4f1f719d-9648-417b-b10c-e5d939825419) 2023-10-05T19:28:33.964 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.958+0000 7fe385185e40 4 Zone: default (9a9df965-6612-45d4-a45f-6e3c889bacc2) 2023-10-05T19:28:33.964 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.958+0000 7fe385185e40 10 cannot find current period zonegroup using local zonegroup configuration 2023-10-05T19:28:33.964 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.958+0000 7fe385185e40 20 zonegroup default 2023-10-05T19:28:33.965 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.958+0000 7fe385185e40 20 rados->read ofs=0 len=0 2023-10-05T19:28:33.965 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.958+0000 7fe385185e40 1 -- 172.21.15.167:0/1226698073 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:81 5.3 5:c52100b6:::period_config.default:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x560cbdf87d10 con 0x7fe36c00d3d0 2023-10-05T19:28:33.965 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.958+0000 7fe3834bb700 1 -- 172.21.15.167:0/1226698073 <== osd.0 v2:172.21.15.118:6800/27690 22 ==== osd_op_reply(81 period_config.default [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 ==== 165+0+0 (crc 0 0 0) 0x7fe37c03f040 con 0x7fe36c00d3d0 2023-10-05T19:28:33.965 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.958+0000 7fe385185e40 20 rados_obj.operate() r=-2 bl.length=0 2023-10-05T19:28:33.966 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.958+0000 7fe385185e40 20 rados->read ofs=0 len=0 2023-10-05T19:28:33.966 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.958+0000 7fe385185e40 1 -- 172.21.15.167:0/1226698073 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:82 6.2 6:4347d321:::bucket.sync-source-hints.:head [call version.read in=11b,read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x560cbdf89690 con 0x560cbdf843f0 2023-10-05T19:28:33.966 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.958+0000 7fe382cba700 1 -- 172.21.15.167:0/1226698073 <== osd.3 v2:172.21.15.167:6808/30036 20 ==== osd_op_reply(82 bucket.sync-source-hints. [call,read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 ==== 211+0+0 (crc 0 0 0) 0x7fe374050040 con 0x560cbdf843f0 2023-10-05T19:28:33.966 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.958+0000 7fe385185e40 20 rados_obj.operate() r=-2 bl.length=0 2023-10-05T19:28:33.966 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.958+0000 7fe385185e40 20 rados->read ofs=0 len=0 2023-10-05T19:28:33.966 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.958+0000 7fe385185e40 1 -- 172.21.15.167:0/1226698073 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:83 6.b 6:d467b91b:::bucket.sync-target-hints.:head [call version.read in=11b,read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x560cbdf89a40 con 0x560cbdf843f0 2023-10-05T19:28:33.967 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.962+0000 7fe382cba700 1 -- 172.21.15.167:0/1226698073 <== osd.3 v2:172.21.15.167:6808/30036 21 ==== osd_op_reply(83 bucket.sync-target-hints. [call,read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 ==== 211+0+0 (crc 0 0 0) 0x7fe374050040 con 0x560cbdf843f0 2023-10-05T19:28:33.967 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.962+0000 7fe385185e40 20 rados_obj.operate() r=-2 bl.length=0 2023-10-05T19:28:33.967 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.962+0000 7fe385185e40 20 started sync module instance, tier type = 2023-10-05T19:28:33.967 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.962+0000 7fe385185e40 20 started zone id=9a9df965-6612-45d4-a45f-6e3c889bacc2 (name=default) with tier type = 2023-10-05T19:28:33.967 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.962+0000 7fe385185e40 1 -- 172.21.15.167:0/1226698073 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:84 7.1f 7:f95f44c2:::notify.0:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x560cbdf88d50 con 0x7fe374007030 2023-10-05T19:28:33.969 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.962+0000 7fe3824b9700 1 -- 172.21.15.167:0/1226698073 <== osd.2 v2:172.21.15.167:6816/30037 13 ==== osd_op_reply(84 notify.0 [create] v30'126 uv126 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7fe36c006070 con 0x7fe374007030 2023-10-05T19:28:33.969 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.962+0000 7fe385185e40 1 -- 172.21.15.167:0/1226698073 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:85 7.1f 7:f95f44c2:::notify.0:head [watch watch cookie 94612726780240] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x560cbdf88980 con 0x7fe374007030 2023-10-05T19:28:33.969 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.962+0000 7fe385185e40 1 -- 172.21.15.167:0/1226698073 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:86 7.0 7:05bf5b68:::notify.1:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x560cbdf8a7d0 con 0x560cbdf4f740 2023-10-05T19:28:33.972 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.966+0000 7fe3824b9700 1 -- 172.21.15.167:0/1226698073 <== osd.2 v2:172.21.15.167:6816/30037 14 ==== osd_op_reply(85 notify.0 [watch watch cookie 94612726780240] v30'127 uv126 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7fe36c006070 con 0x7fe374007030 2023-10-05T19:28:33.972 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.966+0000 7fe3834bb700 1 -- 172.21.15.167:0/1226698073 <== osd.1 v2:172.21.15.167:6800/30035 29 ==== osd_op_reply(86 notify.1 [create] v30'126 uv126 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7fe37c03f040 con 0x560cbdf4f740 2023-10-05T19:28:33.973 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.966+0000 7fe385185e40 1 -- 172.21.15.167:0/1226698073 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:87 7.0 7:05bf5b68:::notify.1:head [watch watch cookie 94612726783760] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x560cbdf8abd0 con 0x560cbdf4f740 2023-10-05T19:28:33.973 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.966+0000 7fe385185e40 1 -- 172.21.15.167:0/1226698073 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:88 7.15 7:a93a5511:::notify.2:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x560cbdf8b470 con 0x560cbdf843f0 2023-10-05T19:28:33.975 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.970+0000 7fe3834bb700 1 -- 172.21.15.167:0/1226698073 <== osd.1 v2:172.21.15.167:6800/30035 30 ==== osd_op_reply(87 notify.1 [watch watch cookie 94612726783760] v30'127 uv126 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7fe37c03f040 con 0x560cbdf4f740 2023-10-05T19:28:33.976 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.970+0000 7fe382cba700 1 -- 172.21.15.167:0/1226698073 <== osd.3 v2:172.21.15.167:6808/30036 22 ==== osd_op_reply(88 notify.2 [create] v30'126 uv126 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7fe374050040 con 0x560cbdf843f0 2023-10-05T19:28:33.976 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.970+0000 7fe385185e40 1 -- 172.21.15.167:0/1226698073 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:89 7.15 7:a93a5511:::notify.2:head [watch watch cookie 94612726784864] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x560cbdf8b0c0 con 0x560cbdf843f0 2023-10-05T19:28:33.976 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.970+0000 7fe385185e40 1 -- 172.21.15.167:0/1226698073 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:90 7.e 7:7759931f:::notify.3:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x560cbdf8bfc0 con 0x7fe374007030 2023-10-05T19:28:33.979 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.974+0000 7fe3824b9700 1 -- 172.21.15.167:0/1226698073 <== osd.2 v2:172.21.15.167:6816/30037 15 ==== osd_op_reply(90 notify.3 [create] v30'126 uv126 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7fe36c006070 con 0x7fe374007030 2023-10-05T19:28:33.979 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.974+0000 7fe382cba700 1 -- 172.21.15.167:0/1226698073 <== osd.3 v2:172.21.15.167:6808/30036 23 ==== osd_op_reply(89 notify.2 [watch watch cookie 94612726784864] v30'127 uv126 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7fe374050040 con 0x560cbdf843f0 2023-10-05T19:28:33.979 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.974+0000 7fe385185e40 1 -- 172.21.15.167:0/1226698073 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:91 7.e 7:7759931f:::notify.3:head [watch watch cookie 94612726790864] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x560cbdf8c860 con 0x7fe374007030 2023-10-05T19:28:33.979 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.974+0000 7fe385185e40 1 -- 172.21.15.167:0/1226698073 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:92 7.d 7:b4812045:::notify.4:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x560cbdf8d100 con 0x560cbdf4f740 2023-10-05T19:28:33.981 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.974+0000 7fe3834bb700 1 -- 172.21.15.167:0/1226698073 <== osd.1 v2:172.21.15.167:6800/30035 31 ==== osd_op_reply(92 notify.4 [create] v30'126 uv126 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7fe37c03f040 con 0x560cbdf4f740 2023-10-05T19:28:33.981 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.974+0000 7fe385185e40 1 -- 172.21.15.167:0/1226698073 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:93 7.d 7:b4812045:::notify.4:head [watch watch cookie 94612726795840] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x560cbdf8d7c0 con 0x560cbdf4f740 2023-10-05T19:28:33.981 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.974+0000 7fe385185e40 1 -- 172.21.15.167:0/1226698073 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:94 7.3 7:c609908c:::notify.5:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x560cbdf8e320 con 0x7fe36c00d3d0 2023-10-05T19:28:33.982 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.974+0000 7fe3824b9700 1 -- 172.21.15.167:0/1226698073 <== osd.2 v2:172.21.15.167:6816/30037 16 ==== osd_op_reply(91 notify.3 [watch watch cookie 94612726790864] v30'127 uv126 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7fe36c006070 con 0x7fe374007030 2023-10-05T19:28:33.983 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.978+0000 7fe3834bb700 1 -- 172.21.15.167:0/1226698073 <== osd.1 v2:172.21.15.167:6800/30035 32 ==== osd_op_reply(93 notify.4 [watch watch cookie 94612726795840] v30'127 uv126 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7fe37c03f040 con 0x560cbdf4f740 2023-10-05T19:28:33.984 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.978+0000 7fe3834bb700 1 -- 172.21.15.167:0/1226698073 <== osd.0 v2:172.21.15.118:6800/27690 23 ==== osd_op_reply(94 notify.5 [create] v30'126 uv126 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7fe37c03f040 con 0x7fe36c00d3d0 2023-10-05T19:28:33.984 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.978+0000 7fe385185e40 1 -- 172.21.15.167:0/1226698073 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:95 7.3 7:c609908c:::notify.5:head [watch watch cookie 94612726793072] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x560cbdf8ddf0 con 0x7fe36c00d3d0 2023-10-05T19:28:33.984 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.978+0000 7fe385185e40 1 -- 172.21.15.167:0/1226698073 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:96 7.14 7:2b04a3e9:::notify.6:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x560cbdf8ec10 con 0x7fe36c00d3d0 2023-10-05T19:28:33.987 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.982+0000 7fe3834bb700 1 -- 172.21.15.167:0/1226698073 <== osd.0 v2:172.21.15.118:6800/27690 24 ==== osd_op_reply(95 notify.5 [watch watch cookie 94612726793072] v30'127 uv126 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7fe37c03f040 con 0x7fe36c00d3d0 2023-10-05T19:28:33.987 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.982+0000 7fe3834bb700 1 -- 172.21.15.167:0/1226698073 <== osd.0 v2:172.21.15.118:6800/27690 25 ==== osd_op_reply(96 notify.6 [create] v30'126 uv126 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7fe37c03f040 con 0x7fe36c00d3d0 2023-10-05T19:28:33.987 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.982+0000 7fe385185e40 1 -- 172.21.15.167:0/1226698073 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:97 7.14 7:2b04a3e9:::notify.6:head [watch watch cookie 94612726796944] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x560cbdf8ddf0 con 0x7fe36c00d3d0 2023-10-05T19:28:33.987 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.982+0000 7fe385185e40 1 -- 172.21.15.167:0/1226698073 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:98 7.9 7:93e5b521:::notify.7:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x560cbdf8ddf0 con 0x7fe36c00d3d0 2023-10-05T19:28:33.989 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.982+0000 7fe3834bb700 1 -- 172.21.15.167:0/1226698073 <== osd.0 v2:172.21.15.118:6800/27690 26 ==== osd_op_reply(97 notify.6 [watch watch cookie 94612726796944] v30'127 uv126 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7fe37c03f040 con 0x7fe36c00d3d0 2023-10-05T19:28:33.991 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.986+0000 7fe3834bb700 1 -- 172.21.15.167:0/1226698073 <== osd.0 v2:172.21.15.118:6800/27690 27 ==== osd_op_reply(98 notify.7 [create] v30'126 uv126 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7fe37c03f040 con 0x7fe36c00d3d0 2023-10-05T19:28:33.991 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.986+0000 7fe385185e40 1 -- 172.21.15.167:0/1226698073 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:99 7.9 7:93e5b521:::notify.7:head [watch watch cookie 94612726802208] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x560cbdf8f060 con 0x7fe36c00d3d0 2023-10-05T19:28:33.991 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.986+0000 7fe385185e40 20 add_watcher() i=0 2023-10-05T19:28:33.991 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.986+0000 7fe385185e40 20 add_watcher() i=1 2023-10-05T19:28:33.991 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.986+0000 7fe385185e40 20 add_watcher() i=2 2023-10-05T19:28:33.991 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.986+0000 7fe385185e40 20 add_watcher() i=3 2023-10-05T19:28:33.991 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.986+0000 7fe385185e40 20 add_watcher() i=4 2023-10-05T19:28:33.991 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.986+0000 7fe385185e40 20 add_watcher() i=5 2023-10-05T19:28:33.991 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.986+0000 7fe385185e40 20 add_watcher() i=6 2023-10-05T19:28:33.993 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.986+0000 7fe3834bb700 1 -- 172.21.15.167:0/1226698073 <== osd.0 v2:172.21.15.118:6800/27690 28 ==== osd_op_reply(99 notify.7 [watch watch cookie 94612726802208] v30'127 uv126 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7fe37c03f040 con 0x7fe36c00d3d0 2023-10-05T19:28:33.993 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.986+0000 7fe385185e40 20 add_watcher() i=7 2023-10-05T19:28:33.993 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.986+0000 7fe385185e40 2 all 8 watchers are set, enabling cache 2023-10-05T19:28:33.993 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.986+0000 7fe385185e40 1 -- 172.21.15.167:0/1226698073 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:100 6.4 6:22d26bf9:::data_loggenerations_metadata:head [call version.check_conds in=74b,call version.read in=11b,read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x560cbdf922e0 con 0x560cbdf4f740 2023-10-05T19:28:33.994 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.986+0000 7fe3834bb700 1 -- 172.21.15.167:0/1226698073 <== osd.1 v2:172.21.15.167:6800/30035 33 ==== osd_op_reply(100 data_loggenerations_metadata [call,call out=48b,read 0~28 out=28b] v0'0 uv1 ondisk = 0) v8 ==== 256+0+76 (crc 0 0 0) 0x7fe37c03f040 con 0x560cbdf4f740 2023-10-05T19:28:33.994 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.986+0000 7fe385185e40 1 -- 172.21.15.167:0/1226698073 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:101 6.4 6:22d26bf9:::data_loggenerations_metadata:head [watch watch cookie 94612726803680] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x560cbdf8f620 con 0x560cbdf4f740 2023-10-05T19:28:33.996 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.990+0000 7fe3834bb700 1 -- 172.21.15.167:0/1226698073 <== osd.1 v2:172.21.15.167:6800/30035 34 ==== osd_op_reply(101 data_loggenerations_metadata [watch watch cookie 94612726803680] v30'122 uv1 ondisk = 0) v8 ==== 172+0+0 (crc 0 0 0) 0x7fe37c03f040 con 0x560cbdf4f740 2023-10-05T19:28:33.996 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.990+0000 7fe2cffff700 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2023-10-05T19:28:33.996 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.990+0000 7fe385185e40 20 check_secure_mon_conn(): auth registy supported: methods=[2] modes=[2,1] 2023-10-05T19:28:33.997 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.990+0000 7fe385185e40 20 check_secure_mon_conn(): mode 1 is insecure 2023-10-05T19:28:33.997 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.990+0000 7fe385185e40 1 Processor -- start 2023-10-05T19:28:33.997 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.990+0000 7fe385185e40 1 -- start start 2023-10-05T19:28:33.997 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.990+0000 7fe385185e40 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x560cbdf9cab0 0x560cbdf9ce80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-05T19:28:33.997 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.990+0000 7fe385185e40 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x560cbdf9d3c0 con 0x560cbdf9cab0 2023-10-05T19:28:33.998 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.990+0000 7fe3824b9700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x560cbdf9cab0 0x560cbdf9ce80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-05T19:28:33.998 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.990+0000 7fe3824b9700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x560cbdf9cab0 0x560cbdf9ce80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.118:3300/0 says I am v2:172.21.15.167:58708/0 (socket says 172.21.15.167:58708) 2023-10-05T19:28:33.998 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.990+0000 7fe3824b9700 1 -- 172.21.15.167:0/2901489984 learned_addr learned my addr 172.21.15.167:0/2901489984 (peer_addr_for_me v2:172.21.15.167:0/0) 2023-10-05T19:28:33.998 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.994+0000 7fe3824b9700 1 -- 172.21.15.167:0/2901489984 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x560cbdf9d500 con 0x560cbdf9cab0 2023-10-05T19:28:33.999 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.994+0000 7fe3824b9700 1 --2- 172.21.15.167:0/2901489984 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x560cbdf9cab0 0x560cbdf9ce80 secure :-1 s=READY pgs=390 cs=0 l=1 rev1=1 crypto rx=0x560cbdf7dfa0 tx=0x7fe36c006000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=525b603c90aae71 server_cookie=0 in_seq=0 out_seq=0 2023-10-05T19:28:33.999 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.994+0000 7fe2cf7fe700 1 -- 172.21.15.167:0/2901489984 <== mon.0 v2:172.21.15.118:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fe36c023040 con 0x560cbdf9cab0 2023-10-05T19:28:33.999 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.994+0000 7fe2cf7fe700 1 -- 172.21.15.167:0/2901489984 <== mon.0 v2:172.21.15.118:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe36c024040 con 0x560cbdf9cab0 2023-10-05T19:28:33.999 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.994+0000 7fe2cf7fe700 1 -- 172.21.15.167:0/2901489984 <== mon.0 v2:172.21.15.118:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fe36c013040 con 0x560cbdf9cab0 2023-10-05T19:28:34.000 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.994+0000 7fe385185e40 1 -- 172.21.15.167:0/2901489984 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x560cbdf9cab0 msgr2=0x560cbdf9ce80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-05T19:28:34.000 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.994+0000 7fe385185e40 1 --2- 172.21.15.167:0/2901489984 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x560cbdf9cab0 0x560cbdf9ce80 secure :-1 s=READY pgs=390 cs=0 l=1 rev1=1 crypto rx=0x560cbdf7dfa0 tx=0x7fe36c006000 comp rx=0 tx=0).stop 2023-10-05T19:28:34.000 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.994+0000 7fe385185e40 1 -- 172.21.15.167:0/2901489984 shutdown_connections 2023-10-05T19:28:34.000 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.994+0000 7fe385185e40 1 --2- 172.21.15.167:0/2901489984 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x560cbdf9cab0 0x560cbdf9ce80 unknown :-1 s=CLOSED pgs=390 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-05T19:28:34.001 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.994+0000 7fe385185e40 1 -- 172.21.15.167:0/2901489984 >> 172.21.15.167:0/2901489984 conn(0x560cbdf97e70 msgr2=0x560cbdf95fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-05T19:28:34.001 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.994+0000 7fe385185e40 1 -- 172.21.15.167:0/2901489984 shutdown_connections 2023-10-05T19:28:34.001 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.994+0000 7fe385185e40 1 -- 172.21.15.167:0/2901489984 wait complete. 2023-10-05T19:28:34.002 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.994+0000 7fe385185e40 1 Processor -- start 2023-10-05T19:28:34.002 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.994+0000 7fe385185e40 1 -- start start 2023-10-05T19:28:34.002 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.994+0000 7fe385185e40 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x560cbdf9cab0 0x560cbdfa2600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-05T19:28:34.002 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.994+0000 7fe385185e40 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x560cbdfa3600 con 0x560cbdf9cab0 2023-10-05T19:28:34.002 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.994+0000 7fe3824b9700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x560cbdf9cab0 0x560cbdfa2600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-05T19:28:34.002 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.994+0000 7fe3824b9700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x560cbdf9cab0 0x560cbdfa2600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.118:3300/0 says I am v2:172.21.15.167:58710/0 (socket says 172.21.15.167:58710) 2023-10-05T19:28:34.002 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.994+0000 7fe3824b9700 1 -- 172.21.15.167:0/1395711003 learned_addr learned my addr 172.21.15.167:0/1395711003 (peer_addr_for_me v2:172.21.15.167:0/0) 2023-10-05T19:28:34.003 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.998+0000 7fe3824b9700 1 -- 172.21.15.167:0/1395711003 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x560cbde9bbd0 con 0x560cbdf9cab0 2023-10-05T19:28:34.003 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.998+0000 7fe3824b9700 1 --2- 172.21.15.167:0/1395711003 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x560cbdf9cab0 0x560cbdfa2600 secure :-1 s=READY pgs=391 cs=0 l=1 rev1=1 crypto rx=0x560cbdf96b30 tx=0x7fe36c02f000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-05T19:28:34.003 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.998+0000 7fe2cdffb700 1 -- 172.21.15.167:0/1395711003 <== mon.0 v2:172.21.15.118:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fe36c023040 con 0x560cbdf9cab0 2023-10-05T19:28:34.004 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.998+0000 7fe2cdffb700 1 -- 172.21.15.167:0/1395711003 <== mon.0 v2:172.21.15.118:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe36c01e420 con 0x560cbdf9cab0 2023-10-05T19:28:34.004 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.998+0000 7fe385185e40 1 -- 172.21.15.167:0/1395711003 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x560cbdfa2c00 con 0x560cbdf9cab0 2023-10-05T19:28:34.004 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.998+0000 7fe2cdffb700 1 -- 172.21.15.167:0/1395711003 <== mon.0 v2:172.21.15.118:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fe36c01e5f0 con 0x560cbdf9cab0 2023-10-05T19:28:34.004 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.998+0000 7fe385185e40 1 -- 172.21.15.167:0/1395711003 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x560cbdfa2fe0 con 0x560cbdf9cab0 2023-10-05T19:28:34.005 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.998+0000 7fe2cdffb700 1 -- 172.21.15.167:0/1395711003 <== mon.0 v2:172.21.15.118:3300/0 4 ==== mgrmap(e 5) v1 ==== 40329+0+0 (secure 0 0 0) 0x7fe36c01e860 con 0x560cbdf9cab0 2023-10-05T19:28:34.005 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.998+0000 7fe2cdffb700 1 --2- 172.21.15.167:0/1395711003 >> [v2:172.21.15.167:6824/29889,v1:172.21.15.167:6825/29889] conn(0x7fe2c0030c00 0x7fe2c00330b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-05T19:28:34.005 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.998+0000 7fe2cdffb700 1 -- 172.21.15.167:0/1395711003 <== mon.0 v2:172.21.15.118:3300/0 5 ==== osd_map(30..30 src has 1..30) v4 ==== 6057+0+0 (secure 0 0 0) 0x7fe36c04ec10 con 0x560cbdf9cab0 2023-10-05T19:28:34.005 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.998+0000 7fe3834bb700 1 --2- 172.21.15.167:0/1395711003 >> [v2:172.21.15.167:6824/29889,v1:172.21.15.167:6825/29889] conn(0x7fe2c0030c00 0x7fe2c00330b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-05T19:28:34.006 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.998+0000 7fe385185e40 1 --2- 172.21.15.167:0/1395711003 >> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] conn(0x560cbdfad1f0 0x560cbdfaf600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-05T19:28:34.006 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:33.998+0000 7fe385185e40 1 -- 172.21.15.167:0/1395711003 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:1 6.19 6:9a6dd8c0:gc::gc.0:head [create,call version.check_conds in=74b,call rgw_gc.rgw_gc_queue_init in=45b,call version.set in=34b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x560cbdfafca0 con 0x560cbdfad1f0 2023-10-05T19:28:34.006 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.002+0000 7fe382cba700 1 --2- 172.21.15.167:0/1395711003 >> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] conn(0x560cbdfad1f0 0x560cbdfaf600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-05T19:28:34.006 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.002+0000 7fe3834bb700 1 --2- 172.21.15.167:0/1395711003 >> [v2:172.21.15.167:6824/29889,v1:172.21.15.167:6825/29889] conn(0x7fe2c0030c00 0x7fe2c00330b0 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x560cbdf92360 tx=0x7fe37c00aec0 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-05T19:28:34.007 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.002+0000 7fe382cba700 1 --2- 172.21.15.167:0/1395711003 >> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] conn(0x560cbdfad1f0 0x560cbdfaf600 crc :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-05T19:28:34.009 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.002+0000 7fe382cba700 1 -- 172.21.15.167:0/1395711003 <== osd.1 v2:172.21.15.167:6800/30035 1 ==== osd_op_reply(1 gc.0 [create,call,call,call] v30'26 uv3 ondisk = -125 ((125) Operation canceled)) v8 ==== 274+0+0 (crc 0 0 0) 0x7fe374050040 con 0x560cbdfad1f0 2023-10-05T19:28:34.009 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.002+0000 7fe385185e40 1 --2- 172.21.15.167:0/1395711003 >> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] conn(0x560cbdfb0970 0x560cbdfb2d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-05T19:28:34.009 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.002+0000 7fe385185e40 1 -- 172.21.15.167:0/1395711003 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:2 6.18 6:1e85421f:gc::gc.1:head [create,call version.check_conds in=74b,call rgw_gc.rgw_gc_queue_init in=45b,call version.set in=34b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x560cbdfb3480 con 0x560cbdfb0970 2023-10-05T19:28:34.010 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.002+0000 7fe3824b9700 1 --2- 172.21.15.167:0/1395711003 >> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] conn(0x560cbdfb0970 0x560cbdfb2d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-05T19:28:34.010 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.006+0000 7fe3824b9700 1 --2- 172.21.15.167:0/1395711003 >> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] conn(0x560cbdfb0970 0x560cbdfb2d50 crc :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-05T19:28:34.013 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.006+0000 7fe3824b9700 1 -- 172.21.15.167:0/1395711003 <== osd.0 v2:172.21.15.118:6800/27690 1 ==== osd_op_reply(2 gc.1 [create,call,call,call] v30'18 uv5 ondisk = -125 ((125) Operation canceled)) v8 ==== 274+0+0 (crc 0 0 0) 0x7fe36c04a070 con 0x560cbdfb0970 2023-10-05T19:28:34.014 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.006+0000 7fe385185e40 1 --2- 172.21.15.167:0/1395711003 >> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] conn(0x560cbdfa8e30 0x560cbdfb6af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-05T19:28:34.014 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.006+0000 7fe385185e40 1 -- 172.21.15.167:0/1395711003 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:3 6.b 6:d242335b:gc::gc.2:head [create,call version.check_conds in=74b,call rgw_gc.rgw_gc_queue_init in=45b,call version.set in=34b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x560cbdfb7070 con 0x560cbdfa8e30 2023-10-05T19:28:34.014 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.006+0000 7fe3834bb700 1 --2- 172.21.15.167:0/1395711003 >> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] conn(0x560cbdfa8e30 0x560cbdfb6af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-05T19:28:34.014 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.010+0000 7fe3834bb700 1 --2- 172.21.15.167:0/1395711003 >> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] conn(0x560cbdfa8e30 0x560cbdfb6af0 crc :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-05T19:28:34.016 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.010+0000 7fe3834bb700 1 -- 172.21.15.167:0/1395711003 <== osd.3 v2:172.21.15.167:6808/30036 1 ==== osd_op_reply(3 gc.2 [create,call,call,call] v30'15 uv3 ondisk = -125 ((125) Operation canceled)) v8 ==== 274+0+0 (crc 0 0 0) 0x7fe37c03f040 con 0x560cbdfa8e30 2023-10-05T19:28:34.016 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.010+0000 7fe385185e40 1 -- 172.21.15.167:0/1395711003 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:4 6.4 6:2166d8e1:gc::gc.3:head [create,call version.check_conds in=74b,call rgw_gc.rgw_gc_queue_init in=45b,call version.set in=34b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x560cbdfb76c0 con 0x560cbdfad1f0 2023-10-05T19:28:34.019 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.014+0000 7fe382cba700 1 -- 172.21.15.167:0/1395711003 <== osd.1 v2:172.21.15.167:6800/30035 2 ==== osd_op_reply(4 gc.3 [create,call,call,call] v30'123 uv7 ondisk = -125 ((125) Operation canceled)) v8 ==== 274+0+0 (crc 0 0 0) 0x7fe374050040 con 0x560cbdfad1f0 2023-10-05T19:28:34.019 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.014+0000 7fe385185e40 1 --2- 172.21.15.167:0/1395711003 >> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] conn(0x560cbdfb8360 0x560cbdfba740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-05T19:28:34.019 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.014+0000 7fe385185e40 1 -- 172.21.15.167:0/1395711003 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:5 6.3 6:c60da066:gc::gc.4:head [create,call version.check_conds in=74b,call rgw_gc.rgw_gc_queue_init in=45b,call version.set in=34b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x560cbdfbadf0 con 0x560cbdfb8360 2023-10-05T19:28:34.019 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.014+0000 7fe382cba700 1 --2- 172.21.15.167:0/1395711003 >> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] conn(0x560cbdfb8360 0x560cbdfba740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-05T19:28:34.020 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.014+0000 7fe382cba700 1 --2- 172.21.15.167:0/1395711003 >> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] conn(0x560cbdfb8360 0x560cbdfba740 crc :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-05T19:28:34.022 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.018+0000 7fe382cba700 1 -- 172.21.15.167:0/1395711003 <== osd.2 v2:172.21.15.167:6816/30037 1 ==== osd_op_reply(5 gc.4 [create,call,call,call] v30'35 uv7 ondisk = -125 ((125) Operation canceled)) v8 ==== 274+0+0 (crc 0 0 0) 0x7fe374050040 con 0x560cbdfb8360 2023-10-05T19:28:34.022 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.018+0000 7fe385185e40 1 -- 172.21.15.167:0/1395711003 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:6 6.9 6:954be8d2:gc::gc.5:head [create,call version.check_conds in=74b,call rgw_gc.rgw_gc_queue_init in=45b,call version.set in=34b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x560cbdfbb410 con 0x560cbdfa8e30 2023-10-05T19:28:34.024 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.018+0000 7fe3834bb700 1 -- 172.21.15.167:0/1395711003 <== osd.3 v2:172.21.15.167:6808/30036 2 ==== osd_op_reply(6 gc.5 [create,call,call,call] v30'25 uv4 ondisk = -125 ((125) Operation canceled)) v8 ==== 274+0+0 (crc 0 0 0) 0x7fe37c03f040 con 0x560cbdfa8e30 2023-10-05T19:28:34.025 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.018+0000 7fe385185e40 1 -- 172.21.15.167:0/1395711003 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:7 6.e 6:748722ff:gc::gc.6:head [create,call version.check_conds in=74b,call rgw_gc.rgw_gc_queue_init in=45b,call version.set in=34b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x560cbdfbba30 con 0x560cbdfad1f0 2023-10-05T19:28:34.027 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.022+0000 7fe382cba700 1 -- 172.21.15.167:0/1395711003 <== osd.1 v2:172.21.15.167:6800/30035 3 ==== osd_op_reply(7 gc.6 [create,call,call,call] v30'17 uv5 ondisk = -125 ((125) Operation canceled)) v8 ==== 274+0+0 (crc 0 0 0) 0x7fe374050040 con 0x560cbdfad1f0 2023-10-05T19:28:34.027 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.022+0000 7fe385185e40 1 -- 172.21.15.167:0/1395711003 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:8 6.1d 6:beaa4be4:gc::gc.7:head [create,call version.check_conds in=74b,call rgw_gc.rgw_gc_queue_init in=45b,call version.set in=34b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x560cbdfbba30 con 0x560cbdfb0970 2023-10-05T19:28:34.030 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.022+0000 7fe3824b9700 1 -- 172.21.15.167:0/1395711003 <== osd.0 v2:172.21.15.118:6800/27690 2 ==== osd_op_reply(8 gc.7 [create,call,call,call] v30'23 uv4 ondisk = -125 ((125) Operation canceled)) v8 ==== 274+0+0 (crc 0 0 0) 0x7fe36c04a070 con 0x560cbdfb0970 2023-10-05T19:28:34.030 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.022+0000 7fe385185e40 1 -- 172.21.15.167:0/1395711003 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:9 6.12 6:4f78e9a2:gc::gc.8:head [create,call version.check_conds in=74b,call rgw_gc.rgw_gc_queue_init in=45b,call version.set in=34b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x560cbdfbc050 con 0x560cbdfb8360 2023-10-05T19:28:34.032 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.026+0000 7fe382cba700 1 -- 172.21.15.167:0/1395711003 <== osd.2 v2:172.21.15.167:6816/30037 2 ==== osd_op_reply(9 gc.8 [create,call,call,call] v30'15 uv3 ondisk = -125 ((125) Operation canceled)) v8 ==== 274+0+0 (crc 0 0 0) 0x7fe374050040 con 0x560cbdfb8360 2023-10-05T19:28:34.032 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.026+0000 7fe385185e40 1 -- 172.21.15.167:0/1395711003 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:10 6.1c 6:3e426232:gc::gc.9:head [create,call version.check_conds in=74b,call rgw_gc.rgw_gc_queue_init in=45b,call version.set in=34b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x560cbdfbc050 con 0x560cbdfa8e30 2023-10-05T19:28:34.034 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.030+0000 7fe3834bb700 1 -- 172.21.15.167:0/1395711003 <== osd.3 v2:172.21.15.167:6808/30036 3 ==== osd_op_reply(10 gc.9 [create,call,call,call] v30'26 uv3 ondisk = -125 ((125) Operation canceled)) v8 ==== 274+0+0 (crc 0 0 0) 0x7fe37c03f040 con 0x560cbdfa8e30 2023-10-05T19:28:34.034 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.030+0000 7fe385185e40 1 -- 172.21.15.167:0/1395711003 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:11 6.3 6:c31f41d7:gc::gc.10:head [create,call version.check_conds in=74b,call rgw_gc.rgw_gc_queue_init in=45b,call version.set in=34b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x560cbdfbc050 con 0x560cbdfb8360 2023-10-05T19:28:34.037 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.030+0000 7fe382cba700 1 -- 172.21.15.167:0/1395711003 <== osd.2 v2:172.21.15.167:6816/30037 3 ==== osd_op_reply(11 gc.10 [create,call,call,call] v30'36 uv11 ondisk = -125 ((125) Operation canceled)) v8 ==== 275+0+0 (crc 0 0 0) 0x7fe374050040 con 0x560cbdfb8360 2023-10-05T19:28:34.037 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.030+0000 7fe385185e40 1 -- 172.21.15.167:0/1395711003 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:12 6.13 6:c8333aab:gc::gc.11:head [create,call version.check_conds in=74b,call rgw_gc.rgw_gc_queue_init in=45b,call version.set in=34b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x560cbdfbc540 con 0x560cbdfb0970 2023-10-05T19:28:34.040 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.034+0000 7fe3824b9700 1 -- 172.21.15.167:0/1395711003 <== osd.0 v2:172.21.15.118:6800/27690 3 ==== osd_op_reply(12 gc.11 [create,call,call,call] v30'18 uv5 ondisk = -125 ((125) Operation canceled)) v8 ==== 275+0+0 (crc 0 0 0) 0x7fe36c04a070 con 0x560cbdfb0970 2023-10-05T19:28:34.040 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.034+0000 7fe385185e40 1 -- 172.21.15.167:0/1395711003 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:13 6.15 6:ab569f81:gc::gc.12:head [create,call version.check_conds in=74b,call rgw_gc.rgw_gc_queue_init in=45b,call version.set in=34b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x560cbdfbc540 con 0x560cbdfb8360 2023-10-05T19:28:34.042 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.038+0000 7fe382cba700 1 -- 172.21.15.167:0/1395711003 <== osd.2 v2:172.21.15.167:6816/30037 4 ==== osd_op_reply(13 gc.12 [create,call,call,call] v30'15 uv3 ondisk = -125 ((125) Operation canceled)) v8 ==== 275+0+0 (crc 0 0 0) 0x7fe374050040 con 0x560cbdfb8360 2023-10-05T19:28:34.042 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.038+0000 7fe385185e40 1 -- 172.21.15.167:0/1395711003 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:14 6.0 6:0667bb13:gc::gc.13:head [create,call version.check_conds in=74b,call rgw_gc.rgw_gc_queue_init in=45b,call version.set in=34b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x560cbdfbc540 con 0x560cbdfb0970 2023-10-05T19:28:34.046 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.038+0000 7fe3824b9700 1 -- 172.21.15.167:0/1395711003 <== osd.0 v2:172.21.15.118:6800/27690 4 ==== osd_op_reply(14 gc.13 [create,call,call,call] v30'29 uv6 ondisk = -125 ((125) Operation canceled)) v8 ==== 275+0+0 (crc 0 0 0) 0x7fe36c04a070 con 0x560cbdfb0970 2023-10-05T19:28:34.046 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.042+0000 7fe385185e40 1 -- 172.21.15.167:0/1395711003 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:15 6.4 6:227f63b5:gc::gc.14:head [create,call version.check_conds in=74b,call rgw_gc.rgw_gc_queue_init in=45b,call version.set in=34b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x560cbdfbc7b0 con 0x560cbdfad1f0 2023-10-05T19:28:34.049 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.042+0000 7fe382cba700 1 -- 172.21.15.167:0/1395711003 <== osd.1 v2:172.21.15.167:6800/30035 4 ==== osd_op_reply(15 gc.14 [create,call,call,call] v30'124 uv13 ondisk = -125 ((125) Operation canceled)) v8 ==== 275+0+0 (crc 0 0 0) 0x7fe374050040 con 0x560cbdfad1f0 2023-10-05T19:28:34.049 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.042+0000 7fe385185e40 1 -- 172.21.15.167:0/1395711003 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:16 6.7 6:e07a6548:gc::gc.15:head [create,call version.check_conds in=74b,call rgw_gc.rgw_gc_queue_init in=45b,call version.set in=34b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x560cbdfbc7b0 con 0x560cbdfa8e30 2023-10-05T19:28:34.052 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.046+0000 7fe3834bb700 1 -- 172.21.15.167:0/1395711003 <== osd.3 v2:172.21.15.167:6808/30036 4 ==== osd_op_reply(16 gc.15 [create,call,call,call] v30'29 uv7 ondisk = -125 ((125) Operation canceled)) v8 ==== 275+0+0 (crc 0 0 0) 0x7fe37c03f040 con 0x560cbdfa8e30 2023-10-05T19:28:34.052 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.046+0000 7fe385185e40 1 -- 172.21.15.167:0/1395711003 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:17 6.3 6:c623b173:gc::gc.16:head [create,call version.check_conds in=74b,call rgw_gc.rgw_gc_queue_init in=45b,call version.set in=34b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x560cbdfbc7b0 con 0x560cbdfb8360 2023-10-05T19:28:34.054 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.050+0000 7fe382cba700 1 -- 172.21.15.167:0/1395711003 <== osd.2 v2:172.21.15.167:6816/30037 5 ==== osd_op_reply(17 gc.16 [create,call,call,call] v30'37 uv13 ondisk = -125 ((125) Operation canceled)) v8 ==== 275+0+0 (crc 0 0 0) 0x7fe374050040 con 0x560cbdfb8360 2023-10-05T19:28:34.055 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.050+0000 7fe385185e40 1 -- 172.21.15.167:0/1395711003 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:18 6.16 6:6849fd06:gc::gc.17:head [create,call version.check_conds in=74b,call rgw_gc.rgw_gc_queue_init in=45b,call version.set in=34b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x560cbdfbc7b0 con 0x560cbdfa8e30 2023-10-05T19:28:34.057 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.050+0000 7fe3834bb700 1 -- 172.21.15.167:0/1395711003 <== osd.3 v2:172.21.15.167:6808/30036 5 ==== osd_op_reply(18 gc.17 [create,call,call,call] v30'15 uv7 ondisk = -125 ((125) Operation canceled)) v8 ==== 275+0+0 (crc 0 0 0) 0x7fe37c03f040 con 0x560cbdfa8e30 2023-10-05T19:28:34.057 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.050+0000 7fe385185e40 1 -- 172.21.15.167:0/1395711003 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:19 6.d 6:b019ea83:gc::gc.18:head [create,call version.check_conds in=74b,call rgw_gc.rgw_gc_queue_init in=45b,call version.set in=34b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x560cbdfbc7b0 con 0x560cbdfad1f0 2023-10-05T19:28:34.059 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.054+0000 7fe382cba700 1 -- 172.21.15.167:0/1395711003 <== osd.1 v2:172.21.15.167:6800/30035 5 ==== osd_op_reply(19 gc.18 [create,call,call,call] v30'23 uv5 ondisk = -125 ((125) Operation canceled)) v8 ==== 275+0+0 (crc 0 0 0) 0x7fe374050040 con 0x560cbdfad1f0 2023-10-05T19:28:34.060 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.054+0000 7fe385185e40 1 -- 172.21.15.167:0/1395711003 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:20 6.4 6:24c8e1ae:gc::gc.19:head [create,call version.check_conds in=74b,call rgw_gc.rgw_gc_queue_init in=45b,call version.set in=34b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x560cbdfbc7b0 con 0x560cbdfad1f0 2023-10-05T19:28:34.062 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.058+0000 7fe382cba700 1 -- 172.21.15.167:0/1395711003 <== osd.1 v2:172.21.15.167:6800/30035 6 ==== osd_op_reply(20 gc.19 [create,call,call,call] v30'125 uv15 ondisk = -125 ((125) Operation canceled)) v8 ==== 275+0+0 (crc 0 0 0) 0x7fe374050040 con 0x560cbdfad1f0 2023-10-05T19:28:34.062 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.058+0000 7fe385185e40 1 -- 172.21.15.167:0/1395711003 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:21 6.7 6:e622f76a:gc::gc.20:head [create,call version.check_conds in=74b,call rgw_gc.rgw_gc_queue_init in=45b,call version.set in=34b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x560cbdfbc7b0 con 0x560cbdfa8e30 2023-10-05T19:28:34.064 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.058+0000 7fe3834bb700 1 -- 172.21.15.167:0/1395711003 <== osd.3 v2:172.21.15.167:6808/30036 6 ==== osd_op_reply(21 gc.20 [create,call,call,call] v30'30 uv9 ondisk = -125 ((125) Operation canceled)) v8 ==== 275+0+0 (crc 0 0 0) 0x7fe37c03f040 con 0x560cbdfa8e30 2023-10-05T19:28:34.065 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.058+0000 7fe385185e40 1 -- 172.21.15.167:0/1395711003 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:22 6.a 6:5127e642:gc::gc.21:head [create,call version.check_conds in=74b,call rgw_gc.rgw_gc_queue_init in=45b,call version.set in=34b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x560cbdfbc7b0 con 0x560cbdfb8360 2023-10-05T19:28:34.067 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.062+0000 7fe382cba700 1 -- 172.21.15.167:0/1395711003 <== osd.2 v2:172.21.15.167:6816/30037 6 ==== osd_op_reply(22 gc.21 [create,call,call,call] v30'25 uv7 ondisk = -125 ((125) Operation canceled)) v8 ==== 275+0+0 (crc 0 0 0) 0x7fe374050040 con 0x560cbdfb8360 2023-10-05T19:28:34.067 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.062+0000 7fe385185e40 1 -- 172.21.15.167:0/1395711003 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:23 6.11 6:8eb99c53:gc::gc.22:head [create,call version.check_conds in=74b,call rgw_gc.rgw_gc_queue_init in=45b,call version.set in=34b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x560cbdfbc7b0 con 0x560cbdfa8e30 2023-10-05T19:28:34.069 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.062+0000 7fe3834bb700 1 -- 172.21.15.167:0/1395711003 <== osd.3 v2:172.21.15.167:6808/30036 7 ==== osd_op_reply(23 gc.22 [create,call,call,call] v30'17 uv3 ondisk = -125 ((125) Operation canceled)) v8 ==== 275+0+0 (crc 0 0 0) 0x7fe37c03f040 con 0x560cbdfa8e30 2023-10-05T19:28:34.069 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.062+0000 7fe385185e40 1 -- 172.21.15.167:0/1395711003 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:24 6.8 6:17d7e8ed:gc::gc.23:head [create,call version.check_conds in=74b,call rgw_gc.rgw_gc_queue_init in=45b,call version.set in=34b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x560cbdfbc7b0 con 0x560cbdfa8e30 2023-10-05T19:28:34.072 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.066+0000 7fe3834bb700 1 -- 172.21.15.167:0/1395711003 <== osd.3 v2:172.21.15.167:6808/30036 8 ==== osd_op_reply(24 gc.23 [create,call,call,call] v30'26 uv5 ondisk = -125 ((125) Operation canceled)) v8 ==== 275+0+0 (crc 0 0 0) 0x7fe37c03f040 con 0x560cbdfa8e30 2023-10-05T19:28:34.072 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.066+0000 7fe385185e40 1 -- 172.21.15.167:0/1395711003 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:25 6.1b 6:db9d130d:gc::gc.24:head [create,call version.check_conds in=74b,call rgw_gc.rgw_gc_queue_init in=45b,call version.set in=34b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x560cbdfbc7b0 con 0x560cbdfa8e30 2023-10-05T19:28:34.074 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.070+0000 7fe3834bb700 1 -- 172.21.15.167:0/1395711003 <== osd.3 v2:172.21.15.167:6808/30036 9 ==== osd_op_reply(25 gc.24 [create,call,call,call] v30'11 uv3 ondisk = -125 ((125) Operation canceled)) v8 ==== 275+0+0 (crc 0 0 0) 0x7fe37c03f040 con 0x560cbdfa8e30 2023-10-05T19:28:34.074 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.070+0000 7fe385185e40 1 -- 172.21.15.167:0/1395711003 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:26 6.9 6:921ecdb1:gc::gc.25:head [create,call version.check_conds in=74b,call rgw_gc.rgw_gc_queue_init in=45b,call version.set in=34b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x560cbdfbc7b0 con 0x560cbdfa8e30 2023-10-05T19:28:34.077 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.070+0000 7fe3834bb700 1 -- 172.21.15.167:0/1395711003 <== osd.3 v2:172.21.15.167:6808/30036 10 ==== osd_op_reply(26 gc.25 [create,call,call,call] v30'26 uv10 ondisk = -125 ((125) Operation canceled)) v8 ==== 275+0+0 (crc 0 0 0) 0x7fe37c03f040 con 0x560cbdfa8e30 2023-10-05T19:28:34.077 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.070+0000 7fe385185e40 1 -- 172.21.15.167:0/1395711003 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:27 6.1f 6:fde4dd55:gc::gc.26:head [create,call version.check_conds in=74b,call rgw_gc.rgw_gc_queue_init in=45b,call version.set in=34b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x560cbdfbc7b0 con 0x560cbdfb0970 2023-10-05T19:28:34.079 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.074+0000 7fe3824b9700 1 -- 172.21.15.167:0/1395711003 <== osd.0 v2:172.21.15.118:6800/27690 5 ==== osd_op_reply(27 gc.26 [create,call,call,call] v30'17 uv5 ondisk = -125 ((125) Operation canceled)) v8 ==== 275+0+0 (crc 0 0 0) 0x7fe36c04a070 con 0x560cbdfb0970 2023-10-05T19:28:34.079 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.074+0000 7fe385185e40 1 -- 172.21.15.167:0/1395711003 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:28 6.1a 6:5909e671:gc::gc.27:head [create,call version.check_conds in=74b,call rgw_gc.rgw_gc_queue_init in=45b,call version.set in=34b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x560cbdfbc7b0 con 0x560cbdfa8e30 2023-10-05T19:28:34.082 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.074+0000 7fe3834bb700 1 -- 172.21.15.167:0/1395711003 <== osd.3 v2:172.21.15.167:6808/30036 11 ==== osd_op_reply(28 gc.27 [create,call,call,call] v30'15 uv7 ondisk = -125 ((125) Operation canceled)) v8 ==== 275+0+0 (crc 0 0 0) 0x7fe37c03f040 con 0x560cbdfa8e30 2023-10-05T19:28:34.082 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.074+0000 7fe385185e40 1 -- 172.21.15.167:0/1395711003 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:29 6.1d 6:b88e3271:gc::gc.28:head [create,call version.check_conds in=74b,call rgw_gc.rgw_gc_queue_init in=45b,call version.set in=34b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x560cbdfbc7b0 con 0x560cbdfb0970 2023-10-05T19:28:34.084 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.078+0000 7fe3824b9700 1 -- 172.21.15.167:0/1395711003 <== osd.0 v2:172.21.15.118:6800/27690 6 ==== osd_op_reply(29 gc.28 [create,call,call,call] v30'24 uv8 ondisk = -125 ((125) Operation canceled)) v8 ==== 275+0+0 (crc 0 0 0) 0x7fe36c04a070 con 0x560cbdfb0970 2023-10-05T19:28:34.084 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.078+0000 7fe385185e40 1 -- 172.21.15.167:0/1395711003 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:30 6.7 6:e47c9322:gc::gc.29:head [create,call version.check_conds in=74b,call rgw_gc.rgw_gc_queue_init in=45b,call version.set in=34b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x560cbdfbc7b0 con 0x560cbdfa8e30 2023-10-05T19:28:34.087 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.082+0000 7fe3834bb700 1 -- 172.21.15.167:0/1395711003 <== osd.3 v2:172.21.15.167:6808/30036 12 ==== osd_op_reply(30 gc.29 [create,call,call,call] v30'31 uv11 ondisk = -125 ((125) Operation canceled)) v8 ==== 275+0+0 (crc 0 0 0) 0x7fe37c03f040 con 0x560cbdfa8e30 2023-10-05T19:28:34.087 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.082+0000 7fe385185e40 1 -- 172.21.15.167:0/1395711003 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:31 6.0 6:013d6c33:gc::gc.30:head [create,call version.check_conds in=74b,call rgw_gc.rgw_gc_queue_init in=45b,call version.set in=34b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x560cbdfbc7b0 con 0x560cbdfb0970 2023-10-05T19:28:34.090 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.082+0000 7fe3824b9700 1 -- 172.21.15.167:0/1395711003 <== osd.0 v2:172.21.15.118:6800/27690 7 ==== osd_op_reply(31 gc.30 [create,call,call,call] v30'30 uv12 ondisk = -125 ((125) Operation canceled)) v8 ==== 275+0+0 (crc 0 0 0) 0x7fe36c04a070 con 0x560cbdfb0970 2023-10-05T19:28:34.090 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.086+0000 7fe385185e40 1 -- 172.21.15.167:0/1395711003 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:32 6.f 6:f24cb4a8:gc::gc.31:head [create,call version.check_conds in=74b,call rgw_gc.rgw_gc_queue_init in=45b,call version.set in=34b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x560cbdfbc7b0 con 0x560cbdfb8360 2023-10-05T19:28:34.092 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.086+0000 7fe382cba700 1 -- 172.21.15.167:0/1395711003 <== osd.2 v2:172.21.15.167:6816/30037 7 ==== osd_op_reply(32 gc.31 [create,call,call,call] v30'21 uv5 ondisk = -125 ((125) Operation canceled)) v8 ==== 275+0+0 (crc 0 0 0) 0x7fe374050040 con 0x560cbdfb8360 2023-10-05T19:28:34.095 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.090+0000 7fe2c6ffd700 20 reqs_thread_entry: start 2023-10-05T19:28:34.096 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.090+0000 7fe2c4ff9700 20 reqs_thread_entry: start 2023-10-05T19:28:34.096 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.090+0000 7fe385185e40 20 init_complete bucket index max shards: 11 2023-10-05T19:28:34.096 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.090+0000 7fe385185e40 10 rgw notify: Started notification manager with: 1 workers 2023-10-05T19:28:34.097 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.090+0000 7fe2b2ffd700 20 rgw notify: INFO: next queues processing will happen at: Thu Oct 5 19:29:04 2023 2023-10-05T19:28:34.097 INFO:teuthology.orchestra.run.smithi167.stderr: 2023-10-05T19:28:34.097 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.090+0000 7fe2b27fc700 20 reqs_thread_entry: start 2023-10-05T19:28:34.097 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.090+0000 7fe385185e40 20 get_system_obj_state: rctx=0x7ffc1ffde058 obj=default.rgw.meta:users.uid:fud state=0x560cbde280b0 s->prefetch_data=0 2023-10-05T19:28:34.097 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.090+0000 7fe385185e40 10 cache get: name=default.rgw.meta+users.uid+fud : miss 2023-10-05T19:28:34.097 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.090+0000 7fe385185e40 1 -- 172.21.15.167:0/1226698073 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:102 8.e 8:75f48285:users.uid::fud:head [call version.read in=11b,getxattrs,stat] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x560cbe01b120 con 0x7fe36c00d3d0 2023-10-05T19:28:34.098 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.090+0000 7fe3834bb700 1 -- 172.21.15.167:0/1226698073 <== osd.0 v2:172.21.15.118:6800/27690 29 ==== osd_op_reply(102 fud [call out=48b,getxattrs out=75b,stat out=16b] v0'0 uv1 ondisk = 0) v8 ==== 231+0+139 (crc 0 0 0) 0x7fe37c03f040 con 0x7fe36c00d3d0 2023-10-05T19:28:34.098 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.090+0000 7fe385185e40 10 cache put: name=default.rgw.meta+users.uid+fud info.flags=0x16 2023-10-05T19:28:34.098 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.090+0000 7fe385185e40 10 adding default.rgw.meta+users.uid+fud to cache LRU end 2023-10-05T19:28:34.098 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.090+0000 7fe385185e40 10 updating xattr: name=ceph.objclass.version bl.length()=42 2023-10-05T19:28:34.098 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.090+0000 7fe385185e40 20 get_system_obj_state: s->obj_tag was set empty 2023-10-05T19:28:34.098 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.090+0000 7fe385185e40 20 Read xattr: user.rgw.idtag 2023-10-05T19:28:34.098 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.090+0000 7fe385185e40 10 cache get: name=default.rgw.meta+users.uid+fud : type miss (requested=0x13, cached=0x16) 2023-10-05T19:28:34.099 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.090+0000 7fe385185e40 20 rados->read ofs=0 len=0 2023-10-05T19:28:34.099 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.090+0000 7fe385185e40 1 -- 172.21.15.167:0/1226698073 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:103 8.e 8:75f48285:users.uid::fud:head [call version.check_conds in=122b,call version.read in=11b,read 0~0,getxattrs] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x560cbe01dbb0 con 0x7fe36c00d3d0 2023-10-05T19:28:34.099 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.094+0000 7fe3834bb700 1 -- 172.21.15.167:0/1226698073 <== osd.0 v2:172.21.15.118:6800/27690 30 ==== osd_op_reply(103 fud [call,call out=48b,read 0~346 out=346b,getxattrs out=75b] v0'0 uv1 ondisk = 0) v8 ==== 273+0+469 (crc 0 0 0) 0x7fe37c03f040 con 0x7fe36c00d3d0 2023-10-05T19:28:34.099 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.094+0000 7fe385185e40 20 rados_obj.operate() r=0 bl.length=346 2023-10-05T19:28:34.099 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.094+0000 7fe385185e40 10 cache put: name=default.rgw.meta+users.uid+fud info.flags=0x13 2023-10-05T19:28:34.100 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.094+0000 7fe385185e40 10 moving default.rgw.meta+users.uid+fud to cache LRU end 2023-10-05T19:28:34.100 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.094+0000 7fe385185e40 10 updating xattr: name=ceph.objclass.version bl.length()=42 2023-10-05T19:28:34.100 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.094+0000 7fe385185e40 1 -- 172.21.15.167:0/1226698073 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:104 8.1 8:810902b1:users.uid::fud.buckets:head [call user.list_buckets in=34b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x560cbe01dbb0 con 0x7fe36c00d3d0 2023-10-05T19:28:34.100 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.094+0000 7fe3834bb700 1 -- 172.21.15.167:0/1226698073 <== osd.0 v2:172.21.15.118:6800/27690 31 ==== osd_op_reply(104 fud.buckets [call out=15b] v0'0 uv2 ondisk = 0) v8 ==== 155+0+15 (crc 0 0 0) 0x7fe37c03f040 con 0x7fe36c00d3d0 2023-10-05T19:28:34.100 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.094+0000 7fe385185e40 10 removing key index: p5YnriCv1nAtykxBrupQ 2023-10-05T19:28:34.100 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.094+0000 7fe385185e40 10 distributing notification oid=default.rgw.control:notify.2 cni=[op: 1, obj: default.rgw.meta:users.keys:p5YnriCv1nAtykxBrupQ, ofs0, ns] 2023-10-05T19:28:34.101 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.094+0000 7fe385185e40 1 -- 172.21.15.167:0/1226698073 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:105 7.15 7:a93a5511:::notify.2:head [notify cookie 94612727383728 in=182b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x560cbe01e000 con 0x560cbdf843f0 2023-10-05T19:28:34.101 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.094+0000 7fe382cba700 1 -- 172.21.15.167:0/1226698073 <== osd.3 v2:172.21.15.167:6808/30036 24 ==== watch-notify(notify (1) cookie 94612726784864 notify 128849018892 ret 0) v3 ==== 212+0+0 (crc 0 0 0) 0x560cbdf754f0 con 0x560cbdf843f0 2023-10-05T19:28:34.101 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.094+0000 7fe380cb6700 10 rgw watcher librados: RGWWatcher::handle_notify() notify_id 128849018892 cookie 94612726784864 notifier 4479 bl.length()=170 2023-10-05T19:28:34.101 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.094+0000 7fe382cba700 1 -- 172.21.15.167:0/1226698073 <== osd.3 v2:172.21.15.167:6808/30036 25 ==== osd_op_reply(105 notify.2 [notify cookie 94612727383728 out=8b] v0'0 uv126 ondisk = 0) v8 ==== 152+0+8 (crc 0 0 0) 0x7fe374050040 con 0x560cbdf843f0 2023-10-05T19:28:34.102 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.094+0000 7fe380cb6700 1 -- 172.21.15.167:0/1226698073 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:106 7.15 7:a93a5511:::notify.2:head [notify-ack in=20b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x7fe368003c50 con 0x560cbdf843f0 2023-10-05T19:28:34.102 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.094+0000 7fe382cba700 1 -- 172.21.15.167:0/1226698073 <== osd.3 v2:172.21.15.167:6808/30036 26 ==== osd_op_reply(106 notify.2 [notify-ack] v0'0 uv126 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7fe374050040 con 0x560cbdf843f0 2023-10-05T19:28:34.102 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.094+0000 7fe382cba700 1 -- 172.21.15.167:0/1226698073 <== osd.3 v2:172.21.15.167:6808/30036 27 ==== watch-notify(notify_complete (2) cookie 94612727383728 notify 128849018892 ret 0) v3 ==== 42+0+48 (crc 0 0 0) 0x560cbdf75050 con 0x560cbdf843f0 2023-10-05T19:28:34.102 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.094+0000 7fe385185e40 1 -- 172.21.15.167:0/1226698073 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:107 8.13 8:cc695d75:users.keys::p5YnriCv1nAtykxBrupQ:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x560cbe01bb90 con 0x7fe36c00d3d0 2023-10-05T19:28:34.105 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.098+0000 7fe3834bb700 1 -- 172.21.15.167:0/1226698073 <== osd.0 v2:172.21.15.118:6800/27690 32 ==== osd_op_reply(107 p5YnriCv1nAtykxBrupQ [delete] v30'7 uv7 ondisk = 0) v8 ==== 164+0+0 (crc 0 0 0) 0x7fe37c03f040 con 0x7fe36c00d3d0 2023-10-05T19:28:34.105 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.098+0000 7fe385185e40 10 removing email index: 2023-10-05T19:28:34.105 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.098+0000 7fe385185e40 10 removing user buckets index 2023-10-05T19:28:34.105 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.098+0000 7fe385185e40 10 distributing notification oid=default.rgw.control:notify.6 cni=[op: 1, obj: default.rgw.meta:users.uid:fud.buckets, ofs0, ns] 2023-10-05T19:28:34.105 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.098+0000 7fe385185e40 1 -- 172.21.15.167:0/1226698073 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:108 7.14 7:2b04a3e9:::notify.6:head [notify cookie 94612727383728 in=172b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x560cbe01e000 con 0x7fe36c00d3d0 2023-10-05T19:28:34.106 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.098+0000 7fe3834bb700 1 -- 172.21.15.167:0/1226698073 <== osd.0 v2:172.21.15.118:6800/27690 33 ==== watch-notify(notify (1) cookie 94612726796944 notify 128849018889 ret 0) v3 ==== 202+0+0 (crc 0 0 0) 0x7fe37c04e040 con 0x7fe36c00d3d0 2023-10-05T19:28:34.106 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.098+0000 7fe380cb6700 10 rgw watcher librados: RGWWatcher::handle_notify() notify_id 128849018889 cookie 94612726796944 notifier 4479 bl.length()=160 2023-10-05T19:28:34.106 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.098+0000 7fe380cb6700 1 -- 172.21.15.167:0/1226698073 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:109 7.14 7:2b04a3e9:::notify.6:head [notify-ack in=20b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x7fe368003c50 con 0x7fe36c00d3d0 2023-10-05T19:28:34.106 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.098+0000 7fe3834bb700 1 -- 172.21.15.167:0/1226698073 <== osd.0 v2:172.21.15.118:6800/27690 34 ==== osd_op_reply(108 notify.6 [notify cookie 94612727383728 out=8b] v0'0 uv126 ondisk = 0) v8 ==== 152+0+8 (crc 0 0 0) 0x7fe37c03f040 con 0x7fe36c00d3d0 2023-10-05T19:28:34.106 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.102+0000 7fe3834bb700 1 -- 172.21.15.167:0/1226698073 <== osd.0 v2:172.21.15.118:6800/27690 35 ==== osd_op_reply(109 notify.6 [notify-ack] v0'0 uv126 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7fe37c03f040 con 0x7fe36c00d3d0 2023-10-05T19:28:34.107 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.102+0000 7fe3834bb700 1 -- 172.21.15.167:0/1226698073 <== osd.0 v2:172.21.15.118:6800/27690 36 ==== watch-notify(notify_complete (2) cookie 94612727383728 notify 128849018889 ret 0) v3 ==== 42+0+48 (crc 0 0 0) 0x7fe37c036940 con 0x7fe36c00d3d0 2023-10-05T19:28:34.107 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.102+0000 7fe385185e40 1 -- 172.21.15.167:0/1226698073 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:110 8.1 8:810902b1:users.uid::fud.buckets:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x560cbe01bb90 con 0x7fe36c00d3d0 2023-10-05T19:28:34.110 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.102+0000 7fe3834bb700 1 -- 172.21.15.167:0/1226698073 <== osd.0 v2:172.21.15.118:6800/27690 37 ==== osd_op_reply(110 fud.buckets [delete] v30'4 uv4 ondisk = 0) v8 ==== 155+0+0 (crc 0 0 0) 0x7fe37c03f040 con 0x7fe36c00d3d0 2023-10-05T19:28:34.110 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.102+0000 7fe385185e40 10 removing user index: fud 2023-10-05T19:28:34.110 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.102+0000 7fe385185e40 10 removing default.rgw.meta+users.uid+fud from cache 2023-10-05T19:28:34.110 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.102+0000 7fe385185e40 10 distributing notification oid=default.rgw.control:notify.5 cni=[op: 1, obj: default.rgw.meta:users.uid:fud, ofs0, ns] 2023-10-05T19:28:34.110 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.106+0000 7fe385185e40 1 -- 172.21.15.167:0/1226698073 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:111 7.3 7:c609908c:::notify.5:head [notify cookie 94612727379616 in=164b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x560cbe01cff0 con 0x7fe36c00d3d0 2023-10-05T19:28:34.111 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.106+0000 7fe3834bb700 1 -- 172.21.15.167:0/1226698073 <== osd.0 v2:172.21.15.118:6800/27690 38 ==== watch-notify(notify (1) cookie 94612726793072 notify 128849018890 ret 0) v3 ==== 194+0+0 (crc 0 0 0) 0x7fe37c036940 con 0x7fe36c00d3d0 2023-10-05T19:28:34.111 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.106+0000 7fe37bfff700 10 rgw watcher librados: RGWWatcher::handle_notify() notify_id 128849018890 cookie 94612726793072 notifier 4479 bl.length()=152 2023-10-05T19:28:34.111 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.106+0000 7fe3834bb700 1 -- 172.21.15.167:0/1226698073 <== osd.0 v2:172.21.15.118:6800/27690 39 ==== osd_op_reply(111 notify.5 [notify cookie 94612727379616 out=8b] v0'0 uv126 ondisk = 0) v8 ==== 152+0+8 (crc 0 0 0) 0x7fe37c03f040 con 0x7fe36c00d3d0 2023-10-05T19:28:34.112 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.106+0000 7fe37bfff700 1 -- 172.21.15.167:0/1226698073 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:112 7.3 7:c609908c:::notify.5:head [notify-ack in=20b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x7fe2d4003270 con 0x7fe36c00d3d0 2023-10-05T19:28:34.113 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.106+0000 7fe3834bb700 1 -- 172.21.15.167:0/1226698073 <== osd.0 v2:172.21.15.118:6800/27690 40 ==== osd_op_reply(112 notify.5 [notify-ack] v0'0 uv126 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7fe37c03f040 con 0x7fe36c00d3d0 2023-10-05T19:28:34.113 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.106+0000 7fe3834bb700 1 -- 172.21.15.167:0/1226698073 <== osd.0 v2:172.21.15.118:6800/27690 41 ==== watch-notify(notify_complete (2) cookie 94612727379616 notify 128849018890 ret 0) v3 ==== 42+0+48 (crc 0 0 0) 0x7fe37c036b70 con 0x7fe36c00d3d0 2023-10-05T19:28:34.113 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.106+0000 7fe385185e40 1 -- 172.21.15.167:0/1226698073 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:113 8.e 8:75f48285:users.uid::fud:head [call version.inc in=38b,delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x560cbe01cba0 con 0x7fe36c00d3d0 2023-10-05T19:28:34.116 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.110+0000 7fe3834bb700 1 -- 172.21.15.167:0/1226698073 <== osd.0 v2:172.21.15.118:6800/27690 42 ==== osd_op_reply(113 fud [call,delete] v30'2 uv2 ondisk = 0) v8 ==== 189+0+0 (crc 0 0 0) 0x7fe37c03f040 con 0x7fe36c00d3d0 2023-10-05T19:28:34.117 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.110+0000 7fe385185e40 1 -- 172.21.15.167:0/1226698073 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:114 7.1f 7:f95f44c2:::notify.0:head [watch unwatch cookie 94612726780240] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x560cbdf647a0 con 0x7fe374007030 2023-10-05T19:28:34.120 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.114+0000 7fe3824b9700 1 -- 172.21.15.167:0/1226698073 <== osd.2 v2:172.21.15.167:6816/30037 17 ==== osd_op_reply(114 notify.0 [watch unwatch cookie 94612726780240] v30'128 uv126 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7fe36c04a070 con 0x7fe374007030 2023-10-05T19:28:34.120 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.114+0000 7fe385185e40 20 remove_watcher() i=0 2023-10-05T19:28:34.120 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.114+0000 7fe385185e40 2 removed watcher, disabling cache 2023-10-05T19:28:34.120 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.114+0000 7fe385185e40 1 -- 172.21.15.167:0/1226698073 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:115 7.0 7:05bf5b68:::notify.1:head [watch unwatch cookie 94612726783760] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x560cbdf647a0 con 0x560cbdf4f740 2023-10-05T19:28:34.123 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.118+0000 7fe3834bb700 1 -- 172.21.15.167:0/1226698073 <== osd.1 v2:172.21.15.167:6800/30035 35 ==== osd_op_reply(115 notify.1 [watch unwatch cookie 94612726783760] v30'128 uv126 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7fe37c03f040 con 0x560cbdf4f740 2023-10-05T19:28:34.123 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.118+0000 7fe385185e40 20 remove_watcher() i=1 2023-10-05T19:28:34.123 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.118+0000 7fe385185e40 1 -- 172.21.15.167:0/1226698073 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:116 7.15 7:a93a5511:::notify.2:head [watch unwatch cookie 94612726784864] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x560cbdf647a0 con 0x560cbdf843f0 2023-10-05T19:28:34.126 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.122+0000 7fe382cba700 1 -- 172.21.15.167:0/1226698073 <== osd.3 v2:172.21.15.167:6808/30036 28 ==== osd_op_reply(116 notify.2 [watch unwatch cookie 94612726784864] v30'128 uv126 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7fe374050040 con 0x560cbdf843f0 2023-10-05T19:28:34.126 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.122+0000 7fe385185e40 20 remove_watcher() i=2 2023-10-05T19:28:34.126 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.122+0000 7fe385185e40 1 -- 172.21.15.167:0/1226698073 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:117 7.e 7:7759931f:::notify.3:head [watch unwatch cookie 94612726790864] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x560cbdf8a400 con 0x7fe374007030 2023-10-05T19:28:34.129 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.122+0000 7fe3824b9700 1 -- 172.21.15.167:0/1226698073 <== osd.2 v2:172.21.15.167:6816/30037 18 ==== osd_op_reply(117 notify.3 [watch unwatch cookie 94612726790864] v30'128 uv126 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7fe36c04a070 con 0x7fe374007030 2023-10-05T19:28:34.129 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.122+0000 7fe385185e40 20 remove_watcher() i=3 2023-10-05T19:28:34.129 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.122+0000 7fe385185e40 1 -- 172.21.15.167:0/1226698073 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:118 7.d 7:b4812045:::notify.4:head [watch unwatch cookie 94612726795840] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x560cbdf8a400 con 0x560cbdf4f740 2023-10-05T19:28:34.132 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.126+0000 7fe3834bb700 1 -- 172.21.15.167:0/1226698073 <== osd.1 v2:172.21.15.167:6800/30035 36 ==== osd_op_reply(118 notify.4 [watch unwatch cookie 94612726795840] v30'128 uv126 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7fe37c03f040 con 0x560cbdf4f740 2023-10-05T19:28:34.132 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.126+0000 7fe385185e40 20 remove_watcher() i=4 2023-10-05T19:28:34.132 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.126+0000 7fe385185e40 1 -- 172.21.15.167:0/1226698073 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:119 7.3 7:c609908c:::notify.5:head [watch unwatch cookie 94612726793072] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x560cbdf8a400 con 0x7fe36c00d3d0 2023-10-05T19:28:34.135 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.130+0000 7fe3834bb700 1 -- 172.21.15.167:0/1226698073 <== osd.0 v2:172.21.15.118:6800/27690 43 ==== osd_op_reply(119 notify.5 [watch unwatch cookie 94612726793072] v30'128 uv126 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7fe37c03f040 con 0x7fe36c00d3d0 2023-10-05T19:28:34.135 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.130+0000 7fe385185e40 20 remove_watcher() i=5 2023-10-05T19:28:34.135 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.130+0000 7fe385185e40 1 -- 172.21.15.167:0/1226698073 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:120 7.14 7:2b04a3e9:::notify.6:head [watch unwatch cookie 94612726796944] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x560cbdf8a400 con 0x7fe36c00d3d0 2023-10-05T19:28:34.139 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.134+0000 7fe3834bb700 1 -- 172.21.15.167:0/1226698073 <== osd.0 v2:172.21.15.118:6800/27690 44 ==== osd_op_reply(120 notify.6 [watch unwatch cookie 94612726796944] v30'128 uv126 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7fe37c03f040 con 0x7fe36c00d3d0 2023-10-05T19:28:34.139 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.134+0000 7fe385185e40 20 remove_watcher() i=6 2023-10-05T19:28:34.139 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.134+0000 7fe385185e40 1 -- 172.21.15.167:0/1226698073 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:121 7.9 7:93e5b521:::notify.7:head [watch unwatch cookie 94612726802208] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x560cbdf8a400 con 0x7fe36c00d3d0 2023-10-05T19:28:34.142 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.134+0000 7fe3834bb700 1 -- 172.21.15.167:0/1226698073 <== osd.0 v2:172.21.15.118:6800/27690 45 ==== osd_op_reply(121 notify.7 [watch unwatch cookie 94612726802208] v30'128 uv126 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7fe37c03f040 con 0x7fe36c00d3d0 2023-10-05T19:28:34.142 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.134+0000 7fe385185e40 20 remove_watcher() i=7 2023-10-05T19:28:34.143 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.138+0000 7fe2cffff700 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2023-10-05T19:28:34.144 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.138+0000 7fe385185e40 1 -- 172.21.15.167:0/1226698073 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:122 6.4 6:22d26bf9:::data_loggenerations_metadata:head [watch unwatch cookie 94612726803680] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x560cbdf8a400 con 0x560cbdf4f740 2023-10-05T19:28:34.147 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.142+0000 7fe3834bb700 1 -- 172.21.15.167:0/1226698073 <== osd.1 v2:172.21.15.167:6800/30035 37 ==== osd_op_reply(122 data_loggenerations_metadata [watch unwatch cookie 94612726803680] v30'126 uv1 ondisk = 0) v8 ==== 172+0+0 (crc 0 0 0) 0x7fe37c03f040 con 0x560cbdf4f740 2023-10-05T19:28:34.147 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.142+0000 7fe385185e40 1 -- 172.21.15.167:0/1226698073 >> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] conn(0x7fe36c00d3d0 msgr2=0x7fe36c00d820 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-05T19:28:34.147 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.142+0000 7fe385185e40 1 --2- 172.21.15.167:0/1226698073 >> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] conn(0x7fe36c00d3d0 0x7fe36c00d820 crc :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-05T19:28:34.147 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.142+0000 7fe385185e40 1 -- 172.21.15.167:0/1226698073 >> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] conn(0x560cbdf4f740 msgr2=0x560cbdf80540 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-05T19:28:34.148 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.142+0000 7fe385185e40 1 --2- 172.21.15.167:0/1226698073 >> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] conn(0x560cbdf4f740 0x560cbdf80540 crc :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-05T19:28:34.148 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.142+0000 7fe385185e40 1 -- 172.21.15.167:0/1226698073 >> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] conn(0x7fe374007030 msgr2=0x7fe374001620 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-05T19:28:34.148 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.142+0000 7fe385185e40 1 --2- 172.21.15.167:0/1226698073 >> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] conn(0x7fe374007030 0x7fe374001620 crc :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-05T19:28:34.148 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.142+0000 7fe385185e40 1 -- 172.21.15.167:0/1226698073 >> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] conn(0x560cbdf843f0 msgr2=0x560cbdf867d0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-05T19:28:34.148 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.142+0000 7fe385185e40 1 --2- 172.21.15.167:0/1226698073 >> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] conn(0x560cbdf843f0 0x560cbdf867d0 crc :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-05T19:28:34.148 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.142+0000 7fe385185e40 1 -- 172.21.15.167:0/1226698073 >> [v2:172.21.15.167:6824/29889,v1:172.21.15.167:6825/29889] conn(0x7fe364030c40 msgr2=0x7fe3640330f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-05T19:28:34.148 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.142+0000 7fe385185e40 1 --2- 172.21.15.167:0/1226698073 >> [v2:172.21.15.167:6824/29889,v1:172.21.15.167:6825/29889] conn(0x7fe364030c40 0x7fe3640330f0 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7fe36c004170 tx=0x7fe36c001380 comp rx=0 tx=0).stop 2023-10-05T19:28:34.148 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.142+0000 7fe385185e40 1 -- 172.21.15.167:0/1226698073 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x560cbdf4ee30 msgr2=0x560cbdf74b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-05T19:28:34.148 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.142+0000 7fe3834bb700 1 -- 172.21.15.167:0/1226698073 reap_dead start 2023-10-05T19:28:34.148 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.142+0000 7fe385185e40 1 --2- 172.21.15.167:0/1226698073 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x560cbdf4ee30 0x560cbdf74b10 secure :-1 s=READY pgs=389 cs=0 l=1 rev1=1 crypto rx=0x7fe37400c010 tx=0x7fe374003050 comp rx=0 tx=0).stop 2023-10-05T19:28:34.148 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.142+0000 7fe385185e40 1 -- 172.21.15.167:0/1226698073 shutdown_connections 2023-10-05T19:28:34.149 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.142+0000 7fe385185e40 1 --2- 172.21.15.167:0/1226698073 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x560cbdf4ee30 0x560cbdf74b10 unknown :-1 s=CLOSED pgs=389 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-05T19:28:34.149 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.142+0000 7fe385185e40 1 -- 172.21.15.167:0/1226698073 >> 172.21.15.167:0/1226698073 conn(0x560cbdf4a850 msgr2=0x560cbdf68540 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-05T19:28:34.149 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.142+0000 7fe385185e40 1 -- 172.21.15.167:0/1226698073 shutdown_connections 2023-10-05T19:28:34.149 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.142+0000 7fe385185e40 1 -- 172.21.15.167:0/1226698073 wait complete. 2023-10-05T19:28:34.150 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.142+0000 7fe385185e40 1 -- 172.21.15.167:0/1395711003 >> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] conn(0x560cbdfb0970 msgr2=0x560cbdfb2d50 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-05T19:28:34.150 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.142+0000 7fe385185e40 1 --2- 172.21.15.167:0/1395711003 >> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] conn(0x560cbdfb0970 0x560cbdfb2d50 crc :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-05T19:28:34.150 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.142+0000 7fe385185e40 1 -- 172.21.15.167:0/1395711003 >> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] conn(0x560cbdfad1f0 msgr2=0x560cbdfaf600 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-05T19:28:34.150 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.142+0000 7fe385185e40 1 --2- 172.21.15.167:0/1395711003 >> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] conn(0x560cbdfad1f0 0x560cbdfaf600 crc :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-05T19:28:34.150 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.142+0000 7fe385185e40 1 -- 172.21.15.167:0/1395711003 >> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] conn(0x560cbdfb8360 msgr2=0x560cbdfba740 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-05T19:28:34.150 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.142+0000 7fe385185e40 1 --2- 172.21.15.167:0/1395711003 >> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] conn(0x560cbdfb8360 0x560cbdfba740 crc :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-05T19:28:34.151 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.142+0000 7fe385185e40 1 -- 172.21.15.167:0/1395711003 >> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] conn(0x560cbdfa8e30 msgr2=0x560cbdfb6af0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-05T19:28:34.151 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.142+0000 7fe385185e40 1 --2- 172.21.15.167:0/1395711003 >> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] conn(0x560cbdfa8e30 0x560cbdfb6af0 crc :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-05T19:28:34.151 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.142+0000 7fe385185e40 1 -- 172.21.15.167:0/1395711003 >> [v2:172.21.15.167:6824/29889,v1:172.21.15.167:6825/29889] conn(0x7fe2c0030c00 msgr2=0x7fe2c00330b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-05T19:28:34.151 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.142+0000 7fe385185e40 1 --2- 172.21.15.167:0/1395711003 >> [v2:172.21.15.167:6824/29889,v1:172.21.15.167:6825/29889] conn(0x7fe2c0030c00 0x7fe2c00330b0 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x560cbdf92360 tx=0x7fe37c00aec0 comp rx=0 tx=0).stop 2023-10-05T19:28:34.151 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.142+0000 7fe385185e40 1 -- 172.21.15.167:0/1395711003 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x560cbdf9cab0 msgr2=0x560cbdfa2600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-05T19:28:34.151 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.142+0000 7fe382cba700 1 -- 172.21.15.167:0/1395711003 reap_dead start 2023-10-05T19:28:34.151 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.142+0000 7fe385185e40 1 --2- 172.21.15.167:0/1395711003 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x560cbdf9cab0 0x560cbdfa2600 secure :-1 s=READY pgs=391 cs=0 l=1 rev1=1 crypto rx=0x560cbdf96b30 tx=0x7fe36c02f000 comp rx=0 tx=0).stop 2023-10-05T19:28:34.151 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.142+0000 7fe385185e40 1 -- 172.21.15.167:0/1395711003 shutdown_connections 2023-10-05T19:28:34.151 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.142+0000 7fe385185e40 1 --2- 172.21.15.167:0/1395711003 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x560cbdf9cab0 0x560cbdfa2600 unknown :-1 s=CLOSED pgs=391 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-05T19:28:34.151 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.142+0000 7fe385185e40 1 -- 172.21.15.167:0/1395711003 >> 172.21.15.167:0/1395711003 conn(0x560cbdf97e70 msgr2=0x560cbdf95f10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-05T19:28:34.151 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.142+0000 7fe385185e40 1 -- 172.21.15.167:0/1395711003 shutdown_connections 2023-10-05T19:28:34.151 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.142+0000 7fe385185e40 1 -- 172.21.15.167:0/1395711003 wait complete. 2023-10-05T19:28:34.153 INFO:tasks.util.rgw:rgwadmin: client.0 : ['user', 'create', '--tenant', 'testx', '--uid', 'tenanteduser', '--display-name', 'tenanted-user', '--access-key', 'p5YnriCv1nAtykxBrupQ', '--secret', 'Q8Tk6Q/27hfbFSYdSkPtUqhqx1GgzvpXa4WARozh', '--max-buckets', '1'] 2023-10-05T19:28:34.154 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'ceph', 'user', 'create', '--tenant', 'testx', '--uid', 'tenanteduser', '--display-name', 'tenanted-user', '--access-key', 'p5YnriCv1nAtykxBrupQ', '--secret', 'Q8Tk6Q/27hfbFSYdSkPtUqhqx1GgzvpXa4WARozh', '--max-buckets', '1'] 2023-10-05T19:28:34.154 DEBUG:teuthology.orchestra.run.smithi167:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster ceph user create --tenant testx --uid tenanteduser --display-name tenanted-user --access-key p5YnriCv1nAtykxBrupQ --secret Q8Tk6Q/27hfbFSYdSkPtUqhqx1GgzvpXa4WARozh --max-buckets 1 2023-10-05T19:28:34.250 INFO:teuthology.orchestra.run.smithi167.stderr:ignoring --setuser ceph since I am not root 2023-10-05T19:28:34.250 INFO:teuthology.orchestra.run.smithi167.stderr:ignoring --setgroup ceph since I am not root 2023-10-05T19:28:34.251 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.246+0000 7fce14492e40 1 Processor -- start 2023-10-05T19:28:34.251 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.246+0000 7fce14492e40 1 -- start start 2023-10-05T19:28:34.251 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.246+0000 7fce14492e40 1 --2- >> v2:172.21.15.118:3300/0 conn(0x56184d21dde0 0x56184d21e1b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-05T19:28:34.251 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.246+0000 7fce14492e40 1 -- --> v1:172.21.15.118:6789/0 -- auth(proto 0 26 bytes epoch 0) v1 -- 0x56184d16abd0 con 0x56184d21e6f0 2023-10-05T19:28:34.251 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.246+0000 7fce14492e40 1 -- --> v2:172.21.15.118:3300/0 -- mon_getmap magic: 0 v1 -- 0x56184d127ee0 con 0x56184d21dde0 2023-10-05T19:28:34.252 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.246+0000 7fce11fc7700 1 --2- >> v2:172.21.15.118:3300/0 conn(0x56184d21dde0 0x56184d21e1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-05T19:28:34.252 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.246+0000 7fce117c6700 1 --1- >> v1:172.21.15.118:6789/0 conn(0x56184d21e6f0 0x56184d21f190 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.118:6789/0 says I am v1:172.21.15.167:56684/0 (socket says 172.21.15.167:56684) 2023-10-05T19:28:34.252 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.246+0000 7fce117c6700 1 -- 172.21.15.167:0/3367022322 learned_addr learned my addr 172.21.15.167:0/3367022322 (peer_addr_for_me v1:172.21.15.167:0/0) 2023-10-05T19:28:34.253 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.246+0000 7fce11fc7700 1 -- 172.21.15.167:0/3367022322 >> v1:172.21.15.118:6789/0 conn(0x56184d21e6f0 legacy=0x56184d21f190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-05T19:28:34.253 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.246+0000 7fce11fc7700 1 -- 172.21.15.167:0/3367022322 --> v2:172.21.15.118:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x56184d21f8a0 con 0x56184d21dde0 2023-10-05T19:28:34.253 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.246+0000 7fce11fc7700 1 --2- 172.21.15.167:0/3367022322 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x56184d21dde0 0x56184d21e1b0 secure :-1 s=READY pgs=393 cs=0 l=1 rev1=1 crypto rx=0x7fcdfc00a520 tx=0x7fcdfc005d40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d71830790dcf5b19 server_cookie=0 in_seq=0 out_seq=0 2023-10-05T19:28:34.253 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.246+0000 7fce10fc5700 1 -- 172.21.15.167:0/3367022322 <== mon.0 v2:172.21.15.118:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fcdfc00f590 con 0x56184d21dde0 2023-10-05T19:28:34.253 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.246+0000 7fce10fc5700 1 -- 172.21.15.167:0/3367022322 <== mon.0 v2:172.21.15.118:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcdfc00f6f0 con 0x56184d21dde0 2023-10-05T19:28:34.254 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.246+0000 7fce10fc5700 1 -- 172.21.15.167:0/3367022322 <== mon.0 v2:172.21.15.118:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fcdfc00f880 con 0x56184d21dde0 2023-10-05T19:28:34.254 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.250+0000 7fce14492e40 1 -- 172.21.15.167:0/3367022322 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x56184d21dde0 msgr2=0x56184d21e1b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-05T19:28:34.254 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.250+0000 7fce14492e40 1 --2- 172.21.15.167:0/3367022322 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x56184d21dde0 0x56184d21e1b0 secure :-1 s=READY pgs=393 cs=0 l=1 rev1=1 crypto rx=0x7fcdfc00a520 tx=0x7fcdfc005d40 comp rx=0 tx=0).stop 2023-10-05T19:28:34.255 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.250+0000 7fce14492e40 1 -- 172.21.15.167:0/3367022322 shutdown_connections 2023-10-05T19:28:34.255 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.250+0000 7fce14492e40 1 --2- 172.21.15.167:0/3367022322 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x56184d21dde0 0x56184d21e1b0 unknown :-1 s=CLOSED pgs=393 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-05T19:28:34.255 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.250+0000 7fce14492e40 1 -- 172.21.15.167:0/3367022322 >> 172.21.15.167:0/3367022322 conn(0x56184d219800 msgr2=0x56184d21bc10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-05T19:28:34.255 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.250+0000 7fce14492e40 1 -- 172.21.15.167:0/3367022322 shutdown_connections 2023-10-05T19:28:34.255 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.250+0000 7fce14492e40 1 -- 172.21.15.167:0/3367022322 wait complete. 2023-10-05T19:28:34.256 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.250+0000 7fce14492e40 1 Processor -- start 2023-10-05T19:28:34.256 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.250+0000 7fce14492e40 1 -- start start 2023-10-05T19:28:34.257 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.250+0000 7fce14492e40 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x56184d21dde0 0x56184d1940f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-05T19:28:34.257 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.250+0000 7fce14492e40 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x56184d0ebe30 con 0x56184d21dde0 2023-10-05T19:28:34.257 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.250+0000 7fce11fc7700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x56184d21dde0 0x56184d1940f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-05T19:28:34.257 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.250+0000 7fce11fc7700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x56184d21dde0 0x56184d1940f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.118:3300/0 says I am v2:172.21.15.167:58714/0 (socket says 172.21.15.167:58714) 2023-10-05T19:28:34.257 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.250+0000 7fce11fc7700 1 -- 172.21.15.167:0/2927006811 learned_addr learned my addr 172.21.15.167:0/2927006811 (peer_addr_for_me v2:172.21.15.167:0/0) 2023-10-05T19:28:34.258 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.250+0000 7fce11fc7700 1 -- 172.21.15.167:0/2927006811 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcdfc00a0e0 con 0x56184d21dde0 2023-10-05T19:28:34.258 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.254+0000 7fce11fc7700 1 --2- 172.21.15.167:0/2927006811 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x56184d21dde0 0x56184d1940f0 secure :-1 s=READY pgs=394 cs=0 l=1 rev1=1 crypto rx=0x7fcdfc005b20 tx=0x7fcdfc014000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4a6fada8aa26bf28 server_cookie=0 in_seq=0 out_seq=0 2023-10-05T19:28:34.258 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.254+0000 7fce0b7fe700 1 -- 172.21.15.167:0/2927006811 <== mon.0 v2:172.21.15.118:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fcdfc00f860 con 0x56184d21dde0 2023-10-05T19:28:34.259 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.254+0000 7fce0b7fe700 1 -- 172.21.15.167:0/2927006811 <== mon.0 v2:172.21.15.118:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcdfc00f9c0 con 0x56184d21dde0 2023-10-05T19:28:34.259 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.254+0000 7fce0b7fe700 1 -- 172.21.15.167:0/2927006811 <== mon.0 v2:172.21.15.118:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fcdfc00fb30 con 0x56184d21dde0 2023-10-05T19:28:34.260 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.254+0000 7fce14492e40 1 -- 172.21.15.167:0/2927006811 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x56184d21dde0 msgr2=0x56184d1940f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-05T19:28:34.260 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.254+0000 7fce14492e40 1 --2- 172.21.15.167:0/2927006811 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x56184d21dde0 0x56184d1940f0 secure :-1 s=READY pgs=394 cs=0 l=1 rev1=1 crypto rx=0x7fcdfc005b20 tx=0x7fcdfc014000 comp rx=0 tx=0).stop 2023-10-05T19:28:34.260 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.254+0000 7fce14492e40 1 -- 172.21.15.167:0/2927006811 shutdown_connections 2023-10-05T19:28:34.260 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.254+0000 7fce14492e40 1 --2- 172.21.15.167:0/2927006811 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x56184d21dde0 0x56184d1940f0 unknown :-1 s=CLOSED pgs=394 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-05T19:28:34.260 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.254+0000 7fce14492e40 1 -- 172.21.15.167:0/2927006811 >> 172.21.15.167:0/2927006811 conn(0x56184d219800 msgr2=0x56184d238ab0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-05T19:28:34.260 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.254+0000 7fce14492e40 1 -- 172.21.15.167:0/2927006811 shutdown_connections 2023-10-05T19:28:34.260 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.254+0000 7fce14492e40 1 -- 172.21.15.167:0/2927006811 wait complete. 2023-10-05T19:28:34.261 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.254+0000 7fce14492e40 1 Processor -- start 2023-10-05T19:28:34.261 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.254+0000 7fce14492e40 1 -- start start 2023-10-05T19:28:34.261 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.254+0000 7fce14492e40 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x56184d21dde0 0x56184d245d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-05T19:28:34.261 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.254+0000 7fce14492e40 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x56184d0beed0 con 0x56184d21dde0 2023-10-05T19:28:34.262 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.254+0000 7fce11fc7700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x56184d21dde0 0x56184d245d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-05T19:28:34.262 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.254+0000 7fce11fc7700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x56184d21dde0 0x56184d245d30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.118:3300/0 says I am v2:172.21.15.167:58728/0 (socket says 172.21.15.167:58728) 2023-10-05T19:28:34.262 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.254+0000 7fce11fc7700 1 -- 172.21.15.167:0/3157424682 learned_addr learned my addr 172.21.15.167:0/3157424682 (peer_addr_for_me v2:172.21.15.167:0/0) 2023-10-05T19:28:34.262 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.254+0000 7fce11fc7700 1 -- 172.21.15.167:0/3157424682 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x56184d21f8a0 con 0x56184d21dde0 2023-10-05T19:28:34.262 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.254+0000 7fce11fc7700 1 --2- 172.21.15.167:0/3157424682 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x56184d21dde0 0x56184d245d30 secure :-1 s=READY pgs=395 cs=0 l=1 rev1=1 crypto rx=0x7fcdfc00a520 tx=0x7fcdfc01d000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-05T19:28:34.262 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.254+0000 7fce09ffb700 1 -- 172.21.15.167:0/3157424682 <== mon.0 v2:172.21.15.118:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fcdfc02a070 con 0x56184d21dde0 2023-10-05T19:28:34.262 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.254+0000 7fce09ffb700 1 -- 172.21.15.167:0/3157424682 <== mon.0 v2:172.21.15.118:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcdfc010420 con 0x56184d21dde0 2023-10-05T19:28:34.263 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.254+0000 7fce14492e40 1 -- 172.21.15.167:0/3157424682 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x56184d246270 con 0x56184d21dde0 2023-10-05T19:28:34.263 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.254+0000 7fce09ffb700 1 -- 172.21.15.167:0/3157424682 <== mon.0 v2:172.21.15.118:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fcdfc0105b0 con 0x56184d21dde0 2023-10-05T19:28:34.263 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.258+0000 7fce14492e40 1 -- 172.21.15.167:0/3157424682 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x56184d246650 con 0x56184d21dde0 2023-10-05T19:28:34.263 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.258+0000 7fce09ffb700 1 -- 172.21.15.167:0/3157424682 <== mon.0 v2:172.21.15.118:3300/0 4 ==== mgrmap(e 5) v1 ==== 40329+0+0 (secure 0 0 0) 0x7fcdfc010870 con 0x56184d21dde0 2023-10-05T19:28:34.264 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.258+0000 7fce14492e40 20 rados->read ofs=0 len=0 2023-10-05T19:28:34.264 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.258+0000 7fce09ffb700 1 --2- 172.21.15.167:0/3157424682 >> [v2:172.21.15.167:6824/29889,v1:172.21.15.167:6825/29889] conn(0x7fcdf4030bf0 0x7fcdf40330a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-05T19:28:34.264 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.258+0000 7fce09ffb700 1 -- 172.21.15.167:0/3157424682 <== mon.0 v2:172.21.15.118:3300/0 5 ==== osd_map(30..30 src has 1..30) v4 ==== 6057+0+0 (secure 0 0 0) 0x7fcdfc0466c0 con 0x56184d21dde0 2023-10-05T19:28:34.264 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.258+0000 7fce117c6700 1 --2- 172.21.15.167:0/3157424682 >> [v2:172.21.15.167:6824/29889,v1:172.21.15.167:6825/29889] conn(0x7fcdf4030bf0 0x7fcdf40330a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-05T19:28:34.264 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.258+0000 7fce14492e40 1 --2- 172.21.15.167:0/3157424682 >> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] conn(0x56184d21e6f0 0x56184d2517f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-05T19:28:34.265 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.258+0000 7fce14492e40 1 -- 172.21.15.167:0/3157424682 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:1 5.12 5:49953fa1:::default.realm:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x56184d251ef0 con 0x56184d21e6f0 2023-10-05T19:28:34.265 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.258+0000 7fce127c8700 1 --2- 172.21.15.167:0/3157424682 >> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] conn(0x56184d21e6f0 0x56184d2517f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-05T19:28:34.265 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.258+0000 7fce117c6700 1 --2- 172.21.15.167:0/3157424682 >> [v2:172.21.15.167:6824/29889,v1:172.21.15.167:6825/29889] conn(0x7fcdf4030bf0 0x7fcdf40330a0 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7fce0c004170 tx=0x7fce0c001380 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-05T19:28:34.265 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.258+0000 7fce127c8700 1 --2- 172.21.15.167:0/3157424682 >> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] conn(0x56184d21e6f0 0x56184d2517f0 crc :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-05T19:28:34.266 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.262+0000 7fce127c8700 1 -- 172.21.15.167:0/3157424682 <== osd.1 v2:172.21.15.167:6800/30035 1 ==== osd_op_reply(1 default.realm [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 ==== 157+0+0 (crc 0 0 0) 0x7fce0400a500 con 0x56184d21e6f0 2023-10-05T19:28:34.266 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.262+0000 7fce14492e40 20 rados_obj.operate() r=-2 bl.length=0 2023-10-05T19:28:34.266 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.262+0000 7fce14492e40 20 realm 2023-10-05T19:28:34.266 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.262+0000 7fce14492e40 20 rados->read ofs=0 len=0 2023-10-05T19:28:34.266 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.262+0000 7fce14492e40 1 -- 172.21.15.167:0/3157424682 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:2 5.12 5:49953fa1:::default.realm:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x56184d24ee50 con 0x56184d21e6f0 2023-10-05T19:28:34.268 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.262+0000 7fce127c8700 1 -- 172.21.15.167:0/3157424682 <== osd.1 v2:172.21.15.167:6800/30035 2 ==== osd_op_reply(2 default.realm [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 ==== 157+0+0 (crc 0 0 0) 0x7fce0400a500 con 0x56184d21e6f0 2023-10-05T19:28:34.268 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.262+0000 7fce14492e40 20 rados_obj.operate() r=-2 bl.length=0 2023-10-05T19:28:34.268 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.262+0000 7fce14492e40 4 RGWPeriod::init failed to init realm id : (2) No such file or directory 2023-10-05T19:28:34.268 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.262+0000 7fce14492e40 20 rados->read ofs=0 len=0 2023-10-05T19:28:34.268 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.262+0000 7fce14492e40 1 -- 172.21.15.167:0/3157424682 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:3 5.12 5:49953fa1:::default.realm:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x56184d2529d0 con 0x56184d21e6f0 2023-10-05T19:28:34.268 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.262+0000 7fce127c8700 1 -- 172.21.15.167:0/3157424682 <== osd.1 v2:172.21.15.167:6800/30035 3 ==== osd_op_reply(3 default.realm [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 ==== 157+0+0 (crc 0 0 0) 0x7fce0400a500 con 0x56184d21e6f0 2023-10-05T19:28:34.268 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.262+0000 7fce14492e40 20 rados_obj.operate() r=-2 bl.length=0 2023-10-05T19:28:34.268 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.262+0000 7fce14492e40 20 rados->read ofs=0 len=0 2023-10-05T19:28:34.268 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.262+0000 7fce14492e40 1 -- 172.21.15.167:0/3157424682 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:4 5.f 5:f43b6ece:::zone_names.default:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x56184d2525e0 con 0x56184d21e6f0 2023-10-05T19:28:34.268 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.262+0000 7fce127c8700 1 -- 172.21.15.167:0/3157424682 <== osd.1 v2:172.21.15.167:6800/30035 4 ==== osd_op_reply(4 zone_names.default [read 0~46 out=46b] v0'0 uv1 ondisk = 0) v8 ==== 162+0+46 (crc 0 0 0) 0x7fce0400a500 con 0x56184d21e6f0 2023-10-05T19:28:34.269 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.262+0000 7fce14492e40 20 rados_obj.operate() r=0 bl.length=46 2023-10-05T19:28:34.269 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.262+0000 7fce14492e40 20 rados->read ofs=0 len=0 2023-10-05T19:28:34.269 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.262+0000 7fce14492e40 1 -- 172.21.15.167:0/3157424682 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:5 5.1d 5:b8374eb4:::zone_info.9a9df965-6612-45d4-a45f-6e3c889bacc2:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x56184d253860 con 0x56184d21e6f0 2023-10-05T19:28:34.269 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.262+0000 7fce127c8700 1 -- 172.21.15.167:0/3157424682 <== osd.1 v2:172.21.15.167:6800/30035 5 ==== osd_op_reply(5 zone_info.9a9df965-6612-45d4-a45f-6e3c889bacc2 [read 0~874 out=874b] v0'0 uv2 ondisk = 0) v8 ==== 190+0+874 (crc 0 0 0) 0x7fce0400a500 con 0x56184d21e6f0 2023-10-05T19:28:34.270 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.262+0000 7fce14492e40 20 rados_obj.operate() r=0 bl.length=874 2023-10-05T19:28:34.270 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.262+0000 7fce14492e40 20 searching for the correct realm 2023-10-05T19:28:34.270 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.262+0000 7fce14492e40 1 --2- 172.21.15.167:0/3157424682 >> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] conn(0x56184d255620 0x56184d257a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-05T19:28:34.270 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.262+0000 7fce14492e40 1 -- 172.21.15.167:0/3157424682 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:6 5.0 5:00000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x56184d257f40 con 0x56184d255620 2023-10-05T19:28:34.270 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.262+0000 7fce11fc7700 1 --2- 172.21.15.167:0/3157424682 >> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] conn(0x56184d255620 0x56184d257a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-05T19:28:34.270 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.262+0000 7fce11fc7700 1 --2- 172.21.15.167:0/3157424682 >> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] conn(0x56184d255620 0x56184d257a00 crc :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-05T19:28:34.271 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.266+0000 7fce11fc7700 1 -- 172.21.15.167:0/3157424682 <== osd.3 v2:172.21.15.167:6808/30036 1 ==== osd_op_reply(6 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fcdfc034070 con 0x56184d255620 2023-10-05T19:28:34.271 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.266+0000 7fce11fc7700 1 --2- 172.21.15.167:0/3157424682 >> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] conn(0x7fcdfc01b740 0x7fcdfc0019a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-05T19:28:34.271 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.266+0000 7fce11fc7700 1 -- 172.21.15.167:0/3157424682 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:7 5.10 5:08000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x56184d257f40 con 0x7fcdfc01b740 2023-10-05T19:28:34.271 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.266+0000 7fce117c6700 1 --2- 172.21.15.167:0/3157424682 >> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] conn(0x7fcdfc01b740 0x7fcdfc0019a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-05T19:28:34.272 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.266+0000 7fce117c6700 1 --2- 172.21.15.167:0/3157424682 >> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] conn(0x7fcdfc01b740 0x7fcdfc0019a0 crc :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-05T19:28:34.272 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.266+0000 7fce117c6700 1 -- 172.21.15.167:0/3157424682 <== osd.2 v2:172.21.15.167:6816/30037 1 ==== osd_op_reply(7 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fce0c006070 con 0x7fcdfc01b740 2023-10-05T19:28:34.272 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.266+0000 7fce117c6700 1 --2- 172.21.15.167:0/3157424682 >> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] conn(0x7fce0c00d3d0 0x7fce0c00d820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-05T19:28:34.272 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.266+0000 7fce117c6700 1 -- 172.21.15.167:0/3157424682 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:8 5.8 5:10000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x56184d257f40 con 0x7fce0c00d3d0 2023-10-05T19:28:34.273 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.266+0000 7fce127c8700 1 --2- 172.21.15.167:0/3157424682 >> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] conn(0x7fce0c00d3d0 0x7fce0c00d820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-05T19:28:34.274 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.266+0000 7fce127c8700 1 --2- 172.21.15.167:0/3157424682 >> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] conn(0x7fce0c00d3d0 0x7fce0c00d820 crc :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-05T19:28:34.275 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.270+0000 7fce127c8700 1 -- 172.21.15.167:0/3157424682 <== osd.0 v2:172.21.15.118:6800/27690 1 ==== osd_op_reply(8 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fce04014040 con 0x7fce0c00d3d0 2023-10-05T19:28:34.275 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.270+0000 7fce127c8700 1 -- 172.21.15.167:0/3157424682 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:9 5.18 5:18000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x56184d257f40 con 0x56184d21e6f0 2023-10-05T19:28:34.275 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.270+0000 7fce127c8700 1 -- 172.21.15.167:0/3157424682 <== osd.1 v2:172.21.15.167:6800/30035 6 ==== osd_op_reply(9 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fce04014040 con 0x56184d21e6f0 2023-10-05T19:28:34.275 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.270+0000 7fce127c8700 1 -- 172.21.15.167:0/3157424682 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:10 5.4 5:20000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x56184d257f40 con 0x56184d21e6f0 2023-10-05T19:28:34.276 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.270+0000 7fce127c8700 1 -- 172.21.15.167:0/3157424682 <== osd.1 v2:172.21.15.167:6800/30035 7 ==== osd_op_reply(10 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fce04014040 con 0x56184d21e6f0 2023-10-05T19:28:34.276 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.270+0000 7fce127c8700 1 -- 172.21.15.167:0/3157424682 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:11 5.14 5:28000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x56184d257f40 con 0x7fce0c00d3d0 2023-10-05T19:28:34.276 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.270+0000 7fce127c8700 1 -- 172.21.15.167:0/3157424682 <== osd.0 v2:172.21.15.118:6800/27690 2 ==== osd_op_reply(11 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fce04014040 con 0x7fce0c00d3d0 2023-10-05T19:28:34.277 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.270+0000 7fce127c8700 1 -- 172.21.15.167:0/3157424682 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:12 5.c 5:30000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x56184d257f40 con 0x7fce0c00d3d0 2023-10-05T19:28:34.277 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.270+0000 7fce127c8700 1 -- 172.21.15.167:0/3157424682 <== osd.0 v2:172.21.15.118:6800/27690 3 ==== osd_op_reply(12 [pgnls start_epoch 30 out=112b] v23'1 uv1 ondisk = 1) v8 ==== 144+0+112 (crc 0 0 0) 0x7fce04014040 con 0x7fce0c00d3d0 2023-10-05T19:28:34.277 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.270+0000 7fce127c8700 1 -- 172.21.15.167:0/3157424682 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:13 5.1c 5:38000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x56184d257f40 con 0x7fcdfc01b740 2023-10-05T19:28:34.278 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.274+0000 7fce117c6700 1 -- 172.21.15.167:0/3157424682 <== osd.2 v2:172.21.15.167:6816/30037 2 ==== osd_op_reply(13 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fce0c006070 con 0x7fcdfc01b740 2023-10-05T19:28:34.278 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.274+0000 7fce117c6700 1 -- 172.21.15.167:0/3157424682 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:14 5.2 5:40000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x56184d257f40 con 0x56184d255620 2023-10-05T19:28:34.279 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.274+0000 7fce11fc7700 1 -- 172.21.15.167:0/3157424682 <== osd.3 v2:172.21.15.167:6808/30036 2 ==== osd_op_reply(14 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fcdfc034070 con 0x56184d255620 2023-10-05T19:28:34.279 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.274+0000 7fce11fc7700 1 -- 172.21.15.167:0/3157424682 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:15 5.12 5:48000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x56184d257f40 con 0x56184d21e6f0 2023-10-05T19:28:34.279 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.274+0000 7fce127c8700 1 -- 172.21.15.167:0/3157424682 <== osd.1 v2:172.21.15.167:6800/30035 8 ==== osd_op_reply(15 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fce04014040 con 0x56184d21e6f0 2023-10-05T19:28:34.279 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.274+0000 7fce127c8700 1 -- 172.21.15.167:0/3157424682 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:16 5.a 5:50000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x56184d257f40 con 0x7fce0c00d3d0 2023-10-05T19:28:34.280 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.274+0000 7fce127c8700 1 -- 172.21.15.167:0/3157424682 <== osd.0 v2:172.21.15.118:6800/27690 4 ==== osd_op_reply(16 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fce04022040 con 0x7fce0c00d3d0 2023-10-05T19:28:34.280 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.274+0000 7fce127c8700 1 -- 172.21.15.167:0/3157424682 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:17 5.1a 5:58000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x56184d257f40 con 0x56184d21e6f0 2023-10-05T19:28:34.281 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.274+0000 7fce127c8700 1 -- 172.21.15.167:0/3157424682 <== osd.1 v2:172.21.15.167:6800/30035 9 ==== osd_op_reply(17 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fce04022040 con 0x56184d21e6f0 2023-10-05T19:28:34.281 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.274+0000 7fce127c8700 1 -- 172.21.15.167:0/3157424682 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:18 5.6 5:60000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x56184d257f40 con 0x7fcdfc01b740 2023-10-05T19:28:34.281 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.274+0000 7fce117c6700 1 -- 172.21.15.167:0/3157424682 <== osd.2 v2:172.21.15.167:6816/30037 3 ==== osd_op_reply(18 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fce0c006070 con 0x7fcdfc01b740 2023-10-05T19:28:34.281 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.274+0000 7fce117c6700 1 -- 172.21.15.167:0/3157424682 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:19 5.16 5:68000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x56184d257f40 con 0x56184d255620 2023-10-05T19:28:34.282 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.278+0000 7fce11fc7700 1 -- 172.21.15.167:0/3157424682 <== osd.3 v2:172.21.15.167:6808/30036 3 ==== osd_op_reply(19 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fcdfc034070 con 0x56184d255620 2023-10-05T19:28:34.282 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.278+0000 7fce11fc7700 1 -- 172.21.15.167:0/3157424682 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:20 5.e 5:70000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x56184d257f40 con 0x56184d255620 2023-10-05T19:28:34.285 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.278+0000 7fce11fc7700 1 -- 172.21.15.167:0/3157424682 <== osd.3 v2:172.21.15.167:6808/30036 4 ==== osd_op_reply(20 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fcdfc034070 con 0x56184d255620 2023-10-05T19:28:34.286 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.278+0000 7fce11fc7700 1 -- 172.21.15.167:0/3157424682 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:21 5.1e 5:78000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x56184d257f40 con 0x7fce0c00d3d0 2023-10-05T19:28:34.286 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.278+0000 7fce127c8700 1 -- 172.21.15.167:0/3157424682 <== osd.0 v2:172.21.15.118:6800/27690 5 ==== osd_op_reply(21 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fce04022040 con 0x7fce0c00d3d0 2023-10-05T19:28:34.286 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.278+0000 7fce127c8700 1 -- 172.21.15.167:0/3157424682 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:22 5.1 5:80000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x56184d257f40 con 0x56184d255620 2023-10-05T19:28:34.286 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.278+0000 7fce11fc7700 1 -- 172.21.15.167:0/3157424682 <== osd.3 v2:172.21.15.167:6808/30036 5 ==== osd_op_reply(22 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fcdfc034070 con 0x56184d255620 2023-10-05T19:28:34.286 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.278+0000 7fce11fc7700 1 -- 172.21.15.167:0/3157424682 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:23 5.11 5:88000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x56184d257f40 con 0x56184d255620 2023-10-05T19:28:34.286 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.278+0000 7fce11fc7700 1 -- 172.21.15.167:0/3157424682 <== osd.3 v2:172.21.15.167:6808/30036 6 ==== osd_op_reply(23 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fcdfc034070 con 0x56184d255620 2023-10-05T19:28:34.286 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.278+0000 7fce11fc7700 1 -- 172.21.15.167:0/3157424682 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:24 5.9 5:90000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x56184d257f40 con 0x56184d21e6f0 2023-10-05T19:28:34.286 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.278+0000 7fce127c8700 1 -- 172.21.15.167:0/3157424682 <== osd.1 v2:172.21.15.167:6800/30035 10 ==== osd_op_reply(24 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fce04022040 con 0x56184d21e6f0 2023-10-05T19:28:34.286 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.278+0000 7fce127c8700 1 -- 172.21.15.167:0/3157424682 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:25 5.19 5:98000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x56184d257f40 con 0x56184d21e6f0 2023-10-05T19:28:34.287 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.282+0000 7fce127c8700 1 -- 172.21.15.167:0/3157424682 <== osd.1 v2:172.21.15.167:6800/30035 11 ==== osd_op_reply(25 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fce04022040 con 0x56184d21e6f0 2023-10-05T19:28:34.287 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.282+0000 7fce127c8700 1 -- 172.21.15.167:0/3157424682 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:26 5.5 5:a0000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x56184d257f40 con 0x7fce0c00d3d0 2023-10-05T19:28:34.287 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.282+0000 7fce127c8700 1 -- 172.21.15.167:0/3157424682 <== osd.0 v2:172.21.15.118:6800/27690 6 ==== osd_op_reply(26 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fce04022040 con 0x7fce0c00d3d0 2023-10-05T19:28:34.287 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.282+0000 7fce127c8700 1 -- 172.21.15.167:0/3157424682 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:27 5.15 5:a8000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x56184d257f40 con 0x56184d255620 2023-10-05T19:28:34.288 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.282+0000 7fce11fc7700 1 -- 172.21.15.167:0/3157424682 <== osd.3 v2:172.21.15.167:6808/30036 7 ==== osd_op_reply(27 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fcdfc034070 con 0x56184d255620 2023-10-05T19:28:34.288 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.282+0000 7fce11fc7700 1 -- 172.21.15.167:0/3157424682 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:28 5.d 5:b0000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x56184d257f40 con 0x7fcdfc01b740 2023-10-05T19:28:34.289 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.282+0000 7fce117c6700 1 -- 172.21.15.167:0/3157424682 <== osd.2 v2:172.21.15.167:6816/30037 4 ==== osd_op_reply(28 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fce0c006070 con 0x7fcdfc01b740 2023-10-05T19:28:34.289 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.282+0000 7fce117c6700 1 -- 172.21.15.167:0/3157424682 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:29 5.1d 5:b8000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x56184d257f40 con 0x56184d21e6f0 2023-10-05T19:28:34.289 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.282+0000 7fce127c8700 1 -- 172.21.15.167:0/3157424682 <== osd.1 v2:172.21.15.167:6800/30035 12 ==== osd_op_reply(29 [pgnls start_epoch 30 out=107b] v30'2 uv2 ondisk = 1) v8 ==== 144+0+107 (crc 0 0 0) 0x7fce04022040 con 0x56184d21e6f0 2023-10-05T19:28:34.290 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.282+0000 7fce127c8700 1 -- 172.21.15.167:0/3157424682 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:30 5.3 5:c0000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x56184d257f40 con 0x7fce0c00d3d0 2023-10-05T19:28:34.290 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.286+0000 7fce127c8700 1 -- 172.21.15.167:0/3157424682 <== osd.0 v2:172.21.15.118:6800/27690 7 ==== osd_op_reply(30 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fce04022040 con 0x7fce0c00d3d0 2023-10-05T19:28:34.291 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.286+0000 7fce127c8700 1 -- 172.21.15.167:0/3157424682 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:31 5.13 5:c8000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x56184d257f40 con 0x56184d255620 2023-10-05T19:28:34.291 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.286+0000 7fce11fc7700 1 -- 172.21.15.167:0/3157424682 <== osd.3 v2:172.21.15.167:6808/30036 8 ==== osd_op_reply(31 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fcdfc034070 con 0x56184d255620 2023-10-05T19:28:34.291 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.286+0000 7fce11fc7700 1 -- 172.21.15.167:0/3157424682 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:32 5.b 5:d0000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x56184d257f40 con 0x7fce0c00d3d0 2023-10-05T19:28:34.292 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.286+0000 7fce127c8700 1 -- 172.21.15.167:0/3157424682 <== osd.0 v2:172.21.15.118:6800/27690 8 ==== osd_op_reply(32 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fce04022040 con 0x7fce0c00d3d0 2023-10-05T19:28:34.292 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.286+0000 7fce127c8700 1 -- 172.21.15.167:0/3157424682 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:33 5.1b 5:d8000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x56184d257f40 con 0x7fcdfc01b740 2023-10-05T19:28:34.293 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.286+0000 7fce117c6700 1 -- 172.21.15.167:0/3157424682 <== osd.2 v2:172.21.15.167:6816/30037 5 ==== osd_op_reply(33 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fce0c006070 con 0x7fcdfc01b740 2023-10-05T19:28:34.293 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.286+0000 7fce117c6700 1 -- 172.21.15.167:0/3157424682 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:34 5.7 5:e0000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x56184d257f40 con 0x56184d255620 2023-10-05T19:28:34.294 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.290+0000 7fce11fc7700 1 -- 172.21.15.167:0/3157424682 <== osd.3 v2:172.21.15.167:6808/30036 9 ==== osd_op_reply(34 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fcdfc034070 con 0x56184d255620 2023-10-05T19:28:34.294 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.290+0000 7fce11fc7700 1 -- 172.21.15.167:0/3157424682 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:35 5.17 5:e8000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x56184d257f40 con 0x7fce0c00d3d0 2023-10-05T19:28:34.295 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.290+0000 7fce127c8700 1 -- 172.21.15.167:0/3157424682 <== osd.0 v2:172.21.15.118:6800/27690 9 ==== osd_op_reply(35 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fce04022040 con 0x7fce0c00d3d0 2023-10-05T19:28:34.295 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.290+0000 7fce127c8700 1 -- 172.21.15.167:0/3157424682 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:36 5.f 5:f0000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x56184d257f40 con 0x56184d21e6f0 2023-10-05T19:28:34.296 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.290+0000 7fce127c8700 1 -- 172.21.15.167:0/3157424682 <== osd.1 v2:172.21.15.167:6800/30035 13 ==== osd_op_reply(36 [pgnls start_epoch 30 out=115b] v23'2 uv2 ondisk = 1) v8 ==== 144+0+115 (crc 0 0 0) 0x7fce0402f080 con 0x56184d21e6f0 2023-10-05T19:28:34.296 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.290+0000 7fce127c8700 1 -- 172.21.15.167:0/3157424682 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:37 5.1f 5:f8000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x56184d257f40 con 0x7fcdfc01b740 2023-10-05T19:28:34.297 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.290+0000 7fce117c6700 1 -- 172.21.15.167:0/3157424682 <== osd.2 v2:172.21.15.167:6816/30037 6 ==== osd_op_reply(37 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fce0c006070 con 0x7fcdfc01b740 2023-10-05T19:28:34.297 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.290+0000 7fce14492e40 20 RGWRados::pool_iterate: got zonegroup_info.4f1f719d-9648-417b-b10c-e5d939825419 2023-10-05T19:28:34.297 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.290+0000 7fce14492e40 20 RGWRados::pool_iterate: got zone_info.9a9df965-6612-45d4-a45f-6e3c889bacc2 2023-10-05T19:28:34.298 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.290+0000 7fce14492e40 20 RGWRados::pool_iterate: got zone_names.default 2023-10-05T19:28:34.298 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.290+0000 7fce14492e40 20 RGWRados::pool_iterate: got zonegroups_names.default 2023-10-05T19:28:34.298 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.290+0000 7fce14492e40 20 rados->read ofs=0 len=0 2023-10-05T19:28:34.298 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.290+0000 7fce14492e40 1 -- 172.21.15.167:0/3157424682 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:38 5.12 5:49953fa1:::default.realm:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x56184d2543d0 con 0x56184d21e6f0 2023-10-05T19:28:34.298 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.290+0000 7fce127c8700 1 -- 172.21.15.167:0/3157424682 <== osd.1 v2:172.21.15.167:6800/30035 14 ==== osd_op_reply(38 default.realm [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 ==== 157+0+0 (crc 0 0 0) 0x7fce0402f080 con 0x56184d21e6f0 2023-10-05T19:28:34.298 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.290+0000 7fce14492e40 20 rados_obj.operate() r=-2 bl.length=0 2023-10-05T19:28:34.298 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.290+0000 7fce14492e40 20 rados->read ofs=0 len=0 2023-10-05T19:28:34.298 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.290+0000 7fce14492e40 1 -- 172.21.15.167:0/3157424682 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:39 5.f 5:f4c53578:::zonegroups_names.default:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x56184d254cc0 con 0x56184d21e6f0 2023-10-05T19:28:34.299 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.294+0000 7fce127c8700 1 -- 172.21.15.167:0/3157424682 <== osd.1 v2:172.21.15.167:6800/30035 15 ==== osd_op_reply(39 zonegroups_names.default [read 0~46 out=46b] v0'0 uv2 ondisk = 0) v8 ==== 168+0+46 (crc 0 0 0) 0x7fce0402f080 con 0x56184d21e6f0 2023-10-05T19:28:34.299 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.294+0000 7fce14492e40 20 rados_obj.operate() r=0 bl.length=46 2023-10-05T19:28:34.299 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.294+0000 7fce14492e40 20 rados->read ofs=0 len=0 2023-10-05T19:28:34.299 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.294+0000 7fce14492e40 1 -- 172.21.15.167:0/3157424682 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:40 5.c 5:30d299a5:::zonegroup_info.4f1f719d-9648-417b-b10c-e5d939825419:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x56184d258380 con 0x7fce0c00d3d0 2023-10-05T19:28:34.300 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.294+0000 7fce127c8700 1 -- 172.21.15.167:0/3157424682 <== osd.0 v2:172.21.15.118:6800/27690 10 ==== osd_op_reply(40 zonegroup_info.4f1f719d-9648-417b-b10c-e5d939825419 [read 0~362 out=362b] v0'0 uv1 ondisk = 0) v8 ==== 195+0+362 (crc 0 0 0) 0x7fce0402f080 con 0x7fce0c00d3d0 2023-10-05T19:28:34.300 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.294+0000 7fce14492e40 20 rados_obj.operate() r=0 bl.length=362 2023-10-05T19:28:34.300 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.294+0000 7fce14492e40 20 rados->read ofs=0 len=0 2023-10-05T19:28:34.300 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.294+0000 7fce14492e40 1 -- 172.21.15.167:0/3157424682 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:41 5.13 5:cbbf6c60:::converted:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x56184d258380 con 0x56184d255620 2023-10-05T19:28:34.301 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.294+0000 7fce11fc7700 1 -- 172.21.15.167:0/3157424682 <== osd.3 v2:172.21.15.167:6808/30036 10 ==== osd_op_reply(41 converted [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 ==== 153+0+0 (crc 0 0 0) 0x7fcdfc034070 con 0x56184d255620 2023-10-05T19:28:34.301 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.294+0000 7fce14492e40 20 rados_obj.operate() r=-2 bl.length=0 2023-10-05T19:28:34.301 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.294+0000 7fce14492e40 20 rados->read ofs=0 len=0 2023-10-05T19:28:34.301 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.294+0000 7fce14492e40 1 -- 172.21.15.167:0/3157424682 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:42 5.12 5:49953fa1:::default.realm:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x56184d258730 con 0x56184d21e6f0 2023-10-05T19:28:34.301 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.294+0000 7fce127c8700 1 -- 172.21.15.167:0/3157424682 <== osd.1 v2:172.21.15.167:6800/30035 16 ==== osd_op_reply(42 default.realm [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 ==== 157+0+0 (crc 0 0 0) 0x7fce0402f080 con 0x56184d21e6f0 2023-10-05T19:28:34.301 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.294+0000 7fce14492e40 20 rados_obj.operate() r=-2 bl.length=0 2023-10-05T19:28:34.301 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.294+0000 7fce14492e40 20 rados->read ofs=0 len=0 2023-10-05T19:28:34.302 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.294+0000 7fce14492e40 1 -- 172.21.15.167:0/3157424682 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:43 5.f 5:f4c53578:::zonegroups_names.default:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x56184d258b30 con 0x56184d21e6f0 2023-10-05T19:28:34.302 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.294+0000 7fce127c8700 1 -- 172.21.15.167:0/3157424682 <== osd.1 v2:172.21.15.167:6800/30035 17 ==== osd_op_reply(43 zonegroups_names.default [read 0~46 out=46b] v0'0 uv2 ondisk = 0) v8 ==== 168+0+46 (crc 0 0 0) 0x7fce0402f080 con 0x56184d21e6f0 2023-10-05T19:28:34.302 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.294+0000 7fce14492e40 20 rados_obj.operate() r=0 bl.length=46 2023-10-05T19:28:34.302 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.294+0000 7fce14492e40 1 -- 172.21.15.167:0/3157424682 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:44 5.0 5:00000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x56184d259b40 con 0x56184d255620 2023-10-05T19:28:34.304 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.298+0000 7fce11fc7700 1 -- 172.21.15.167:0/3157424682 <== osd.3 v2:172.21.15.167:6808/30036 11 ==== osd_op_reply(44 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fcdfc034070 con 0x56184d255620 2023-10-05T19:28:34.304 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.298+0000 7fce11fc7700 1 -- 172.21.15.167:0/3157424682 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:45 5.10 5:08000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x56184d259b40 con 0x7fcdfc01b740 2023-10-05T19:28:34.304 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.298+0000 7fce117c6700 1 -- 172.21.15.167:0/3157424682 <== osd.2 v2:172.21.15.167:6816/30037 7 ==== osd_op_reply(45 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fce0c006070 con 0x7fcdfc01b740 2023-10-05T19:28:34.305 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.298+0000 7fce117c6700 1 -- 172.21.15.167:0/3157424682 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:46 5.8 5:10000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x56184d259b40 con 0x7fce0c00d3d0 2023-10-05T19:28:34.306 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.298+0000 7fce127c8700 1 -- 172.21.15.167:0/3157424682 <== osd.0 v2:172.21.15.118:6800/27690 11 ==== osd_op_reply(46 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fce0402f080 con 0x7fce0c00d3d0 2023-10-05T19:28:34.306 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.298+0000 7fce127c8700 1 -- 172.21.15.167:0/3157424682 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:47 5.18 5:18000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x56184d254cc0 con 0x56184d21e6f0 2023-10-05T19:28:34.307 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.302+0000 7fce127c8700 1 -- 172.21.15.167:0/3157424682 <== osd.1 v2:172.21.15.167:6800/30035 18 ==== osd_op_reply(47 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fce0402f080 con 0x56184d21e6f0 2023-10-05T19:28:34.307 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.302+0000 7fce127c8700 1 -- 172.21.15.167:0/3157424682 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:48 5.4 5:20000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x56184d254cc0 con 0x56184d21e6f0 2023-10-05T19:28:34.307 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.302+0000 7fce127c8700 1 -- 172.21.15.167:0/3157424682 <== osd.1 v2:172.21.15.167:6800/30035 19 ==== osd_op_reply(48 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fce0402f080 con 0x56184d21e6f0 2023-10-05T19:28:34.307 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.302+0000 7fce127c8700 1 -- 172.21.15.167:0/3157424682 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:49 5.14 5:28000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x56184d254cc0 con 0x7fce0c00d3d0 2023-10-05T19:28:34.308 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.302+0000 7fce127c8700 1 -- 172.21.15.167:0/3157424682 <== osd.0 v2:172.21.15.118:6800/27690 12 ==== osd_op_reply(49 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fce0402f080 con 0x7fce0c00d3d0 2023-10-05T19:28:34.308 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.302+0000 7fce127c8700 1 -- 172.21.15.167:0/3157424682 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:50 5.c 5:30000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x56184d254cc0 con 0x7fce0c00d3d0 2023-10-05T19:28:34.309 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.302+0000 7fce127c8700 1 -- 172.21.15.167:0/3157424682 <== osd.0 v2:172.21.15.118:6800/27690 13 ==== osd_op_reply(50 [pgnls start_epoch 30 out=112b] v23'1 uv1 ondisk = 1) v8 ==== 144+0+112 (crc 0 0 0) 0x7fce0402f080 con 0x7fce0c00d3d0 2023-10-05T19:28:34.309 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.302+0000 7fce127c8700 1 -- 172.21.15.167:0/3157424682 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:51 5.1c 5:38000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x56184d254cc0 con 0x7fcdfc01b740 2023-10-05T19:28:34.310 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.306+0000 7fce117c6700 1 -- 172.21.15.167:0/3157424682 <== osd.2 v2:172.21.15.167:6816/30037 8 ==== osd_op_reply(51 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fce0c006070 con 0x7fcdfc01b740 2023-10-05T19:28:34.310 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.306+0000 7fce117c6700 1 -- 172.21.15.167:0/3157424682 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:52 5.2 5:40000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x56184d254cc0 con 0x56184d255620 2023-10-05T19:28:34.311 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.306+0000 7fce11fc7700 1 -- 172.21.15.167:0/3157424682 <== osd.3 v2:172.21.15.167:6808/30036 12 ==== osd_op_reply(52 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fcdfc034070 con 0x56184d255620 2023-10-05T19:28:34.311 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.306+0000 7fce11fc7700 1 -- 172.21.15.167:0/3157424682 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:53 5.12 5:48000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x56184d254cc0 con 0x56184d21e6f0 2023-10-05T19:28:34.311 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.306+0000 7fce127c8700 1 -- 172.21.15.167:0/3157424682 <== osd.1 v2:172.21.15.167:6800/30035 20 ==== osd_op_reply(53 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fce0402f080 con 0x56184d21e6f0 2023-10-05T19:28:34.312 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.306+0000 7fce127c8700 1 -- 172.21.15.167:0/3157424682 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:54 5.a 5:50000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x56184d254cc0 con 0x7fce0c00d3d0 2023-10-05T19:28:34.312 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.306+0000 7fce127c8700 1 -- 172.21.15.167:0/3157424682 <== osd.0 v2:172.21.15.118:6800/27690 14 ==== osd_op_reply(54 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fce0402f080 con 0x7fce0c00d3d0 2023-10-05T19:28:34.313 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.306+0000 7fce127c8700 1 -- 172.21.15.167:0/3157424682 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:55 5.1a 5:58000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x56184d254cc0 con 0x56184d21e6f0 2023-10-05T19:28:34.313 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.306+0000 7fce127c8700 1 -- 172.21.15.167:0/3157424682 <== osd.1 v2:172.21.15.167:6800/30035 21 ==== osd_op_reply(55 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fce0402f080 con 0x56184d21e6f0 2023-10-05T19:28:34.313 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.306+0000 7fce127c8700 1 -- 172.21.15.167:0/3157424682 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:56 5.6 5:60000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x56184d254cc0 con 0x7fcdfc01b740 2023-10-05T19:28:34.314 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.310+0000 7fce117c6700 1 -- 172.21.15.167:0/3157424682 <== osd.2 v2:172.21.15.167:6816/30037 9 ==== osd_op_reply(56 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fce0c006070 con 0x7fcdfc01b740 2023-10-05T19:28:34.314 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.310+0000 7fce117c6700 1 -- 172.21.15.167:0/3157424682 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:57 5.16 5:68000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x56184d254cc0 con 0x56184d255620 2023-10-05T19:28:34.315 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.310+0000 7fce11fc7700 1 -- 172.21.15.167:0/3157424682 <== osd.3 v2:172.21.15.167:6808/30036 13 ==== osd_op_reply(57 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fcdfc034070 con 0x56184d255620 2023-10-05T19:28:34.315 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.310+0000 7fce11fc7700 1 -- 172.21.15.167:0/3157424682 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:58 5.e 5:70000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x56184d254cc0 con 0x56184d255620 2023-10-05T19:28:34.316 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.310+0000 7fce11fc7700 1 -- 172.21.15.167:0/3157424682 <== osd.3 v2:172.21.15.167:6808/30036 14 ==== osd_op_reply(58 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fcdfc034070 con 0x56184d255620 2023-10-05T19:28:34.316 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.310+0000 7fce11fc7700 1 -- 172.21.15.167:0/3157424682 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:59 5.1e 5:78000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x56184d254cc0 con 0x7fce0c00d3d0 2023-10-05T19:28:34.317 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.310+0000 7fce127c8700 1 -- 172.21.15.167:0/3157424682 <== osd.0 v2:172.21.15.118:6800/27690 15 ==== osd_op_reply(59 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fce0402f080 con 0x7fce0c00d3d0 2023-10-05T19:28:34.317 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.310+0000 7fce127c8700 1 -- 172.21.15.167:0/3157424682 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:60 5.1 5:80000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x56184d254cc0 con 0x56184d255620 2023-10-05T19:28:34.317 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.310+0000 7fce11fc7700 1 -- 172.21.15.167:0/3157424682 <== osd.3 v2:172.21.15.167:6808/30036 15 ==== osd_op_reply(60 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fcdfc034070 con 0x56184d255620 2023-10-05T19:28:34.318 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.310+0000 7fce11fc7700 1 -- 172.21.15.167:0/3157424682 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:61 5.11 5:88000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x56184d254cc0 con 0x56184d255620 2023-10-05T19:28:34.318 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.314+0000 7fce11fc7700 1 -- 172.21.15.167:0/3157424682 <== osd.3 v2:172.21.15.167:6808/30036 16 ==== osd_op_reply(61 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fcdfc034070 con 0x56184d255620 2023-10-05T19:28:34.318 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.314+0000 7fce11fc7700 1 -- 172.21.15.167:0/3157424682 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:62 5.9 5:90000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x56184d254cc0 con 0x56184d21e6f0 2023-10-05T19:28:34.319 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.314+0000 7fce127c8700 1 -- 172.21.15.167:0/3157424682 <== osd.1 v2:172.21.15.167:6800/30035 22 ==== osd_op_reply(62 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fce0402f080 con 0x56184d21e6f0 2023-10-05T19:28:34.319 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.314+0000 7fce127c8700 1 -- 172.21.15.167:0/3157424682 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:63 5.19 5:98000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x56184d254cc0 con 0x56184d21e6f0 2023-10-05T19:28:34.320 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.314+0000 7fce127c8700 1 -- 172.21.15.167:0/3157424682 <== osd.1 v2:172.21.15.167:6800/30035 23 ==== osd_op_reply(63 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fce0402f080 con 0x56184d21e6f0 2023-10-05T19:28:34.320 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.314+0000 7fce127c8700 1 -- 172.21.15.167:0/3157424682 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:64 5.5 5:a0000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x56184d254cc0 con 0x7fce0c00d3d0 2023-10-05T19:28:34.321 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.314+0000 7fce127c8700 1 -- 172.21.15.167:0/3157424682 <== osd.0 v2:172.21.15.118:6800/27690 16 ==== osd_op_reply(64 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fce0402f080 con 0x7fce0c00d3d0 2023-10-05T19:28:34.321 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.314+0000 7fce127c8700 1 -- 172.21.15.167:0/3157424682 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:65 5.15 5:a8000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x56184d254cc0 con 0x56184d255620 2023-10-05T19:28:34.321 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.314+0000 7fce11fc7700 1 -- 172.21.15.167:0/3157424682 <== osd.3 v2:172.21.15.167:6808/30036 17 ==== osd_op_reply(65 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fcdfc034070 con 0x56184d255620 2023-10-05T19:28:34.322 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.314+0000 7fce11fc7700 1 -- 172.21.15.167:0/3157424682 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:66 5.d 5:b0000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x56184d254cc0 con 0x7fcdfc01b740 2023-10-05T19:28:34.322 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.318+0000 7fce117c6700 1 -- 172.21.15.167:0/3157424682 <== osd.2 v2:172.21.15.167:6816/30037 10 ==== osd_op_reply(66 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fce0c006070 con 0x7fcdfc01b740 2023-10-05T19:28:34.322 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.318+0000 7fce117c6700 1 -- 172.21.15.167:0/3157424682 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:67 5.1d 5:b8000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x56184d254cc0 con 0x56184d21e6f0 2023-10-05T19:28:34.323 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.318+0000 7fce127c8700 1 -- 172.21.15.167:0/3157424682 <== osd.1 v2:172.21.15.167:6800/30035 24 ==== osd_op_reply(67 [pgnls start_epoch 30 out=107b] v30'2 uv2 ondisk = 1) v8 ==== 144+0+107 (crc 0 0 0) 0x7fce0402f080 con 0x56184d21e6f0 2023-10-05T19:28:34.323 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.318+0000 7fce127c8700 1 -- 172.21.15.167:0/3157424682 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:68 5.3 5:c0000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x56184d254cc0 con 0x7fce0c00d3d0 2023-10-05T19:28:34.324 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.318+0000 7fce127c8700 1 -- 172.21.15.167:0/3157424682 <== osd.0 v2:172.21.15.118:6800/27690 17 ==== osd_op_reply(68 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fce0402f080 con 0x7fce0c00d3d0 2023-10-05T19:28:34.324 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.318+0000 7fce127c8700 1 -- 172.21.15.167:0/3157424682 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:69 5.13 5:c8000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x56184d254cc0 con 0x56184d255620 2023-10-05T19:28:34.325 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.318+0000 7fce11fc7700 1 -- 172.21.15.167:0/3157424682 <== osd.3 v2:172.21.15.167:6808/30036 18 ==== osd_op_reply(69 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fcdfc034070 con 0x56184d255620 2023-10-05T19:28:34.325 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.318+0000 7fce11fc7700 1 -- 172.21.15.167:0/3157424682 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:70 5.b 5:d0000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x56184d254cc0 con 0x7fce0c00d3d0 2023-10-05T19:28:34.326 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.322+0000 7fce127c8700 1 -- 172.21.15.167:0/3157424682 <== osd.0 v2:172.21.15.118:6800/27690 18 ==== osd_op_reply(70 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fce0402f080 con 0x7fce0c00d3d0 2023-10-05T19:28:34.326 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.322+0000 7fce127c8700 1 -- 172.21.15.167:0/3157424682 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:71 5.1b 5:d8000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x56184d254cc0 con 0x7fcdfc01b740 2023-10-05T19:28:34.327 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.322+0000 7fce117c6700 1 -- 172.21.15.167:0/3157424682 <== osd.2 v2:172.21.15.167:6816/30037 11 ==== osd_op_reply(71 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fce0c006070 con 0x7fcdfc01b740 2023-10-05T19:28:34.327 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.322+0000 7fce117c6700 1 -- 172.21.15.167:0/3157424682 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:72 5.7 5:e0000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x56184d254cc0 con 0x56184d255620 2023-10-05T19:28:34.328 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.322+0000 7fce11fc7700 1 -- 172.21.15.167:0/3157424682 <== osd.3 v2:172.21.15.167:6808/30036 19 ==== osd_op_reply(72 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fcdfc034070 con 0x56184d255620 2023-10-05T19:28:34.328 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.322+0000 7fce11fc7700 1 -- 172.21.15.167:0/3157424682 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:73 5.17 5:e8000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x56184d254cc0 con 0x7fce0c00d3d0 2023-10-05T19:28:34.328 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.322+0000 7fce127c8700 1 -- 172.21.15.167:0/3157424682 <== osd.0 v2:172.21.15.118:6800/27690 19 ==== osd_op_reply(73 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fce0402f080 con 0x7fce0c00d3d0 2023-10-05T19:28:34.329 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.322+0000 7fce127c8700 1 -- 172.21.15.167:0/3157424682 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:74 5.f 5:f0000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x56184d254cc0 con 0x56184d21e6f0 2023-10-05T19:28:34.329 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.322+0000 7fce127c8700 1 -- 172.21.15.167:0/3157424682 <== osd.1 v2:172.21.15.167:6800/30035 25 ==== osd_op_reply(74 [pgnls start_epoch 30 out=115b] v23'2 uv2 ondisk = 1) v8 ==== 144+0+115 (crc 0 0 0) 0x7fce0402f080 con 0x56184d21e6f0 2023-10-05T19:28:34.330 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.322+0000 7fce127c8700 1 -- 172.21.15.167:0/3157424682 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:75 5.1f 5:f8000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x56184d254cc0 con 0x7fcdfc01b740 2023-10-05T19:28:34.331 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.326+0000 7fce117c6700 1 -- 172.21.15.167:0/3157424682 <== osd.2 v2:172.21.15.167:6816/30037 12 ==== osd_op_reply(75 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fce0c006070 con 0x7fcdfc01b740 2023-10-05T19:28:34.331 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.326+0000 7fce14492e40 20 RGWRados::pool_iterate: got zonegroup_info.4f1f719d-9648-417b-b10c-e5d939825419 2023-10-05T19:28:34.331 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.326+0000 7fce14492e40 20 RGWRados::pool_iterate: got zone_info.9a9df965-6612-45d4-a45f-6e3c889bacc2 2023-10-05T19:28:34.331 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.326+0000 7fce14492e40 20 RGWRados::pool_iterate: got zone_names.default 2023-10-05T19:28:34.331 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.326+0000 7fce14492e40 20 RGWRados::pool_iterate: got zonegroups_names.default 2023-10-05T19:28:34.331 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.326+0000 7fce14492e40 20 rados->read ofs=0 len=0 2023-10-05T19:28:34.331 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.326+0000 7fce14492e40 1 -- 172.21.15.167:0/3157424682 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:76 5.f 5:f43b6ece:::zone_names.default:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x56184d258b30 con 0x56184d21e6f0 2023-10-05T19:28:34.332 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.326+0000 7fce127c8700 1 -- 172.21.15.167:0/3157424682 <== osd.1 v2:172.21.15.167:6800/30035 26 ==== osd_op_reply(76 zone_names.default [read 0~46 out=46b] v0'0 uv1 ondisk = 0) v8 ==== 162+0+46 (crc 0 0 0) 0x7fce0402f080 con 0x56184d21e6f0 2023-10-05T19:28:34.332 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.326+0000 7fce14492e40 20 rados_obj.operate() r=0 bl.length=46 2023-10-05T19:28:34.332 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.326+0000 7fce14492e40 20 rados->read ofs=0 len=0 2023-10-05T19:28:34.333 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.326+0000 7fce14492e40 1 -- 172.21.15.167:0/3157424682 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:77 5.1d 5:b8374eb4:::zone_info.9a9df965-6612-45d4-a45f-6e3c889bacc2:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x56184d258ee0 con 0x56184d21e6f0 2023-10-05T19:28:34.333 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.326+0000 7fce127c8700 1 -- 172.21.15.167:0/3157424682 <== osd.1 v2:172.21.15.167:6800/30035 27 ==== osd_op_reply(77 zone_info.9a9df965-6612-45d4-a45f-6e3c889bacc2 [read 0~874 out=874b] v0'0 uv2 ondisk = 0) v8 ==== 190+0+874 (crc 0 0 0) 0x7fce0402f080 con 0x56184d21e6f0 2023-10-05T19:28:34.333 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.326+0000 7fce14492e40 20 rados_obj.operate() r=0 bl.length=874 2023-10-05T19:28:34.333 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.326+0000 7fce14492e40 20 rados->read ofs=0 len=0 2023-10-05T19:28:34.333 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.326+0000 7fce14492e40 1 -- 172.21.15.167:0/3157424682 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:78 5.f 5:f4c53578:::zonegroups_names.default:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x56184d2548f0 con 0x56184d21e6f0 2023-10-05T19:28:34.334 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.326+0000 7fce127c8700 1 -- 172.21.15.167:0/3157424682 <== osd.1 v2:172.21.15.167:6800/30035 28 ==== osd_op_reply(78 zonegroups_names.default [read 0~46 out=46b] v0'0 uv2 ondisk = 0) v8 ==== 168+0+46 (crc 0 0 0) 0x7fce0402f080 con 0x56184d21e6f0 2023-10-05T19:28:34.334 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.326+0000 7fce14492e40 20 rados_obj.operate() r=0 bl.length=46 2023-10-05T19:28:34.334 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.326+0000 7fce14492e40 20 rados->read ofs=0 len=0 2023-10-05T19:28:34.334 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.326+0000 7fce14492e40 1 -- 172.21.15.167:0/3157424682 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:79 5.c 5:30d299a5:::zonegroup_info.4f1f719d-9648-417b-b10c-e5d939825419:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x56184d254900 con 0x7fce0c00d3d0 2023-10-05T19:28:34.335 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.326+0000 7fce127c8700 1 -- 172.21.15.167:0/3157424682 <== osd.0 v2:172.21.15.118:6800/27690 20 ==== osd_op_reply(79 zonegroup_info.4f1f719d-9648-417b-b10c-e5d939825419 [read 0~362 out=362b] v0'0 uv1 ondisk = 0) v8 ==== 195+0+362 (crc 0 0 0) 0x7fce0402f080 con 0x7fce0c00d3d0 2023-10-05T19:28:34.335 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.326+0000 7fce14492e40 20 rados_obj.operate() r=0 bl.length=362 2023-10-05T19:28:34.335 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.330+0000 7fce14492e40 20 rados->read ofs=0 len=0 2023-10-05T19:28:34.335 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.330+0000 7fce14492e40 1 -- 172.21.15.167:0/3157424682 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:80 5.3 5:c0ad8cbd:::region_map:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x56184d259950 con 0x7fce0c00d3d0 2023-10-05T19:28:34.336 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.330+0000 7fce127c8700 1 -- 172.21.15.167:0/3157424682 <== osd.0 v2:172.21.15.118:6800/27690 21 ==== osd_op_reply(80 region_map [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 ==== 154+0+0 (crc 0 0 0) 0x7fce0402f080 con 0x7fce0c00d3d0 2023-10-05T19:28:34.336 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.330+0000 7fce14492e40 20 rados_obj.operate() r=-2 bl.length=0 2023-10-05T19:28:34.336 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.330+0000 7fce14492e40 20 zone default found 2023-10-05T19:28:34.336 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.330+0000 7fce14492e40 4 Realm: () 2023-10-05T19:28:34.336 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.330+0000 7fce14492e40 4 ZoneGroup: default (4f1f719d-9648-417b-b10c-e5d939825419) 2023-10-05T19:28:34.336 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.330+0000 7fce14492e40 4 Zone: default (9a9df965-6612-45d4-a45f-6e3c889bacc2) 2023-10-05T19:28:34.337 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.330+0000 7fce14492e40 10 cannot find current period zonegroup using local zonegroup configuration 2023-10-05T19:28:34.337 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.330+0000 7fce14492e40 20 zonegroup default 2023-10-05T19:28:34.337 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.330+0000 7fce14492e40 20 rados->read ofs=0 len=0 2023-10-05T19:28:34.337 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.330+0000 7fce14492e40 1 -- 172.21.15.167:0/3157424682 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:81 5.3 5:c52100b6:::period_config.default:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x56184d259950 con 0x7fce0c00d3d0 2023-10-05T19:28:34.337 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.330+0000 7fce127c8700 1 -- 172.21.15.167:0/3157424682 <== osd.0 v2:172.21.15.118:6800/27690 22 ==== osd_op_reply(81 period_config.default [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 ==== 165+0+0 (crc 0 0 0) 0x7fce0402f080 con 0x7fce0c00d3d0 2023-10-05T19:28:34.337 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.330+0000 7fce14492e40 20 rados_obj.operate() r=-2 bl.length=0 2023-10-05T19:28:34.337 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.330+0000 7fce14492e40 20 rados->read ofs=0 len=0 2023-10-05T19:28:34.337 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.330+0000 7fce14492e40 1 -- 172.21.15.167:0/3157424682 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:82 6.2 6:4347d321:::bucket.sync-source-hints.:head [call version.read in=11b,read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x56184d25b2d0 con 0x56184d255620 2023-10-05T19:28:34.338 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.330+0000 7fce11fc7700 1 -- 172.21.15.167:0/3157424682 <== osd.3 v2:172.21.15.167:6808/30036 20 ==== osd_op_reply(82 bucket.sync-source-hints. [call,read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 ==== 211+0+0 (crc 0 0 0) 0x7fcdfc034070 con 0x56184d255620 2023-10-05T19:28:34.338 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.330+0000 7fce14492e40 20 rados_obj.operate() r=-2 bl.length=0 2023-10-05T19:28:34.338 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.330+0000 7fce14492e40 20 rados->read ofs=0 len=0 2023-10-05T19:28:34.339 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.330+0000 7fce14492e40 1 -- 172.21.15.167:0/3157424682 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:83 6.b 6:d467b91b:::bucket.sync-target-hints.:head [call version.read in=11b,read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x56184d25b680 con 0x56184d255620 2023-10-05T19:28:34.339 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.330+0000 7fce11fc7700 1 -- 172.21.15.167:0/3157424682 <== osd.3 v2:172.21.15.167:6808/30036 21 ==== osd_op_reply(83 bucket.sync-target-hints. [call,read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 ==== 211+0+0 (crc 0 0 0) 0x7fcdfc034070 con 0x56184d255620 2023-10-05T19:28:34.339 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.330+0000 7fce14492e40 20 rados_obj.operate() r=-2 bl.length=0 2023-10-05T19:28:34.339 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.330+0000 7fce14492e40 20 started sync module instance, tier type = 2023-10-05T19:28:34.339 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.330+0000 7fce14492e40 20 started zone id=9a9df965-6612-45d4-a45f-6e3c889bacc2 (name=default) with tier type = 2023-10-05T19:28:34.339 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.330+0000 7fce14492e40 1 -- 172.21.15.167:0/3157424682 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:84 7.1f 7:f95f44c2:::notify.0:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x56184d25a990 con 0x7fcdfc01b740 2023-10-05T19:28:34.340 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.334+0000 7fce117c6700 1 -- 172.21.15.167:0/3157424682 <== osd.2 v2:172.21.15.167:6816/30037 13 ==== osd_op_reply(84 notify.0 [create] v30'129 uv129 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7fce0c006070 con 0x7fcdfc01b740 2023-10-05T19:28:34.340 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.334+0000 7fce14492e40 1 -- 172.21.15.167:0/3157424682 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:85 7.1f 7:f95f44c2:::notify.0:head [watch watch cookie 94662373516448] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x56184d25ba30 con 0x7fcdfc01b740 2023-10-05T19:28:34.341 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.334+0000 7fce14492e40 1 -- 172.21.15.167:0/3157424682 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:86 7.0 7:05bf5b68:::notify.1:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x56184d25c2d0 con 0x56184d21e6f0 2023-10-05T19:28:34.344 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.338+0000 7fce127c8700 1 -- 172.21.15.167:0/3157424682 <== osd.1 v2:172.21.15.167:6800/30035 29 ==== osd_op_reply(86 notify.1 [create] v30'129 uv129 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7fce0402f080 con 0x56184d21e6f0 2023-10-05T19:28:34.344 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.338+0000 7fce117c6700 1 -- 172.21.15.167:0/3157424682 <== osd.2 v2:172.21.15.167:6816/30037 14 ==== osd_op_reply(85 notify.0 [watch watch cookie 94662373516448] v30'130 uv129 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7fce0c006070 con 0x7fcdfc01b740 2023-10-05T19:28:34.344 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.338+0000 7fce14492e40 1 -- 172.21.15.167:0/3157424682 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:87 7.0 7:05bf5b68:::notify.1:head [watch watch cookie 94662373522912] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x56184d25ae40 con 0x56184d21e6f0 2023-10-05T19:28:34.344 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.338+0000 7fce14492e40 1 -- 172.21.15.167:0/3157424682 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:88 7.15 7:a93a5511:::notify.2:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x56184d25ccc0 con 0x56184d255620 2023-10-05T19:28:34.347 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.342+0000 7fce11fc7700 1 -- 172.21.15.167:0/3157424682 <== osd.3 v2:172.21.15.167:6808/30036 22 ==== osd_op_reply(88 notify.2 [create] v30'129 uv129 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7fcdfc034070 con 0x56184d255620 2023-10-05T19:28:34.347 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.342+0000 7fce14492e40 1 -- 172.21.15.167:0/3157424682 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:89 7.15 7:a93a5511:::notify.2:head [watch watch cookie 94662373525456] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x56184d25d5b0 con 0x56184d255620 2023-10-05T19:28:34.347 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.342+0000 7fce14492e40 1 -- 172.21.15.167:0/3157424682 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:90 7.e 7:7759931f:::notify.3:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x56184d25dfc0 con 0x7fcdfc01b740 2023-10-05T19:28:34.347 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.342+0000 7fce127c8700 1 -- 172.21.15.167:0/3157424682 <== osd.1 v2:172.21.15.167:6800/30035 30 ==== osd_op_reply(87 notify.1 [watch watch cookie 94662373522912] v30'130 uv129 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7fce04050040 con 0x56184d21e6f0 2023-10-05T19:28:34.350 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.342+0000 7fce117c6700 1 -- 172.21.15.167:0/3157424682 <== osd.2 v2:172.21.15.167:6816/30037 15 ==== osd_op_reply(90 notify.3 [create] v30'129 uv129 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7fce0c006070 con 0x7fcdfc01b740 2023-10-05T19:28:34.350 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.342+0000 7fce14492e40 1 -- 172.21.15.167:0/3157424682 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:91 7.e 7:7759931f:::notify.3:head [watch watch cookie 94662373524016] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x56184d25dbf0 con 0x7fcdfc01b740 2023-10-05T19:28:34.350 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.342+0000 7fce14492e40 1 -- 172.21.15.167:0/3157424682 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:92 7.d 7:b4812045:::notify.4:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x56184d25efc0 con 0x56184d21e6f0 2023-10-05T19:28:34.350 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.342+0000 7fce11fc7700 1 -- 172.21.15.167:0/3157424682 <== osd.3 v2:172.21.15.167:6808/30036 23 ==== osd_op_reply(89 notify.2 [watch watch cookie 94662373525456] v30'130 uv129 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7fcdfc034070 con 0x56184d255620 2023-10-05T19:28:34.352 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.346+0000 7fce117c6700 1 -- 172.21.15.167:0/3157424682 <== osd.2 v2:172.21.15.167:6816/30037 16 ==== osd_op_reply(91 notify.3 [watch watch cookie 94662373524016] v30'130 uv129 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7fce0c006070 con 0x7fcdfc01b740 2023-10-05T19:28:34.352 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.346+0000 7fce127c8700 1 -- 172.21.15.167:0/3157424682 <== osd.1 v2:172.21.15.167:6800/30035 31 ==== osd_op_reply(92 notify.4 [create] v30'129 uv129 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7fce04050040 con 0x56184d21e6f0 2023-10-05T19:28:34.352 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.346+0000 7fce14492e40 1 -- 172.21.15.167:0/3157424682 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:93 7.d 7:b4812045:::notify.4:head [watch watch cookie 94662373527744] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x56184d25ea90 con 0x56184d21e6f0 2023-10-05T19:28:34.352 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.346+0000 7fce14492e40 1 -- 172.21.15.167:0/3157424682 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:94 7.3 7:c609908c:::notify.5:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x56184d25f8b0 con 0x7fce0c00d3d0 2023-10-05T19:28:34.354 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.350+0000 7fce127c8700 1 -- 172.21.15.167:0/3157424682 <== osd.1 v2:172.21.15.167:6800/30035 32 ==== osd_op_reply(93 notify.4 [watch watch cookie 94662373527744] v30'130 uv129 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7fce04050040 con 0x56184d21e6f0 2023-10-05T19:28:34.355 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.350+0000 7fce127c8700 1 -- 172.21.15.167:0/3157424682 <== osd.0 v2:172.21.15.118:6800/27690 23 ==== osd_op_reply(94 notify.5 [create] v30'129 uv129 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7fce04050040 con 0x7fce0c00d3d0 2023-10-05T19:28:34.355 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.350+0000 7fce14492e40 1 -- 172.21.15.167:0/3157424682 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:95 7.3 7:c609908c:::notify.5:head [watch watch cookie 94662373532528] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x56184d25f860 con 0x7fce0c00d3d0 2023-10-05T19:28:34.355 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.350+0000 7fce14492e40 1 -- 172.21.15.167:0/3157424682 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:96 7.14 7:2b04a3e9:::notify.6:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x56184d260270 con 0x7fce0c00d3d0 2023-10-05T19:28:34.358 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.354+0000 7fce127c8700 1 -- 172.21.15.167:0/3157424682 <== osd.0 v2:172.21.15.118:6800/27690 24 ==== osd_op_reply(96 notify.6 [create] v30'129 uv129 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7fce04050040 con 0x7fce0c00d3d0 2023-10-05T19:28:34.358 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.354+0000 7fce127c8700 1 -- 172.21.15.167:0/3157424682 <== osd.0 v2:172.21.15.118:6800/27690 25 ==== osd_op_reply(95 notify.5 [watch watch cookie 94662373532528] v30'130 uv129 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7fce04050040 con 0x7fce0c00d3d0 2023-10-05T19:28:34.358 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.354+0000 7fce14492e40 1 -- 172.21.15.167:0/3157424682 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:97 7.14 7:2b04a3e9:::notify.6:head [watch watch cookie 94662373539200] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x56184d2606c0 con 0x7fce0c00d3d0 2023-10-05T19:28:34.359 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.354+0000 7fce14492e40 1 -- 172.21.15.167:0/3157424682 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:98 7.9 7:93e5b521:::notify.7:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x56184d25f370 con 0x7fce0c00d3d0 2023-10-05T19:28:34.361 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.354+0000 7fce127c8700 1 -- 172.21.15.167:0/3157424682 <== osd.0 v2:172.21.15.118:6800/27690 26 ==== osd_op_reply(98 notify.7 [create] v30'129 uv129 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7fce04050040 con 0x7fce0c00d3d0 2023-10-05T19:28:34.361 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.354+0000 7fce14492e40 1 -- 172.21.15.167:0/3157424682 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:99 7.9 7:93e5b521:::notify.7:head [watch watch cookie 94662373536624] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x56184d2610d0 con 0x7fce0c00d3d0 2023-10-05T19:28:34.361 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.354+0000 7fce14492e40 20 add_watcher() i=0 2023-10-05T19:28:34.361 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.354+0000 7fce14492e40 20 add_watcher() i=1 2023-10-05T19:28:34.361 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.354+0000 7fce14492e40 20 add_watcher() i=2 2023-10-05T19:28:34.362 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.354+0000 7fce14492e40 20 add_watcher() i=3 2023-10-05T19:28:34.362 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.354+0000 7fce14492e40 20 add_watcher() i=4 2023-10-05T19:28:34.362 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.354+0000 7fce14492e40 20 add_watcher() i=5 2023-10-05T19:28:34.362 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.354+0000 7fce127c8700 1 -- 172.21.15.167:0/3157424682 <== osd.0 v2:172.21.15.118:6800/27690 27 ==== osd_op_reply(97 notify.6 [watch watch cookie 94662373539200] v30'130 uv129 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7fce04050040 con 0x7fce0c00d3d0 2023-10-05T19:28:34.362 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.354+0000 7fce14492e40 20 add_watcher() i=6 2023-10-05T19:28:34.364 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.358+0000 7fce127c8700 1 -- 172.21.15.167:0/3157424682 <== osd.0 v2:172.21.15.118:6800/27690 28 ==== osd_op_reply(99 notify.7 [watch watch cookie 94662373536624] v30'130 uv129 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7fce04050040 con 0x7fce0c00d3d0 2023-10-05T19:28:34.364 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.358+0000 7fce14492e40 20 add_watcher() i=7 2023-10-05T19:28:34.364 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.358+0000 7fce14492e40 2 all 8 watchers are set, enabling cache 2023-10-05T19:28:34.364 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.358+0000 7fce14492e40 1 -- 172.21.15.167:0/3157424682 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:100 6.4 6:22d26bf9:::data_loggenerations_metadata:head [call version.check_conds in=74b,call version.read in=11b,read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x56184d264100 con 0x56184d21e6f0 2023-10-05T19:28:34.364 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.358+0000 7fce127c8700 1 -- 172.21.15.167:0/3157424682 <== osd.1 v2:172.21.15.167:6800/30035 33 ==== osd_op_reply(100 data_loggenerations_metadata [call,call out=48b,read 0~28 out=28b] v0'0 uv1 ondisk = 0) v8 ==== 256+0+76 (crc 0 0 0) 0x7fce04050040 con 0x56184d21e6f0 2023-10-05T19:28:34.364 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.358+0000 7fce14492e40 1 -- 172.21.15.167:0/3157424682 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:101 6.4 6:22d26bf9:::data_loggenerations_metadata:head [watch watch cookie 94662373542880] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x56184d261520 con 0x56184d21e6f0 2023-10-05T19:28:34.367 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.362+0000 7fce127c8700 1 -- 172.21.15.167:0/3157424682 <== osd.1 v2:172.21.15.167:6800/30035 34 ==== osd_op_reply(101 data_loggenerations_metadata [watch watch cookie 94662373542880] v30'127 uv1 ondisk = 0) v8 ==== 172+0+0 (crc 0 0 0) 0x7fce04050040 con 0x56184d21e6f0 2023-10-05T19:28:34.367 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.362+0000 7fcde2fe5700 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2023-10-05T19:28:34.367 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.362+0000 7fce14492e40 20 check_secure_mon_conn(): auth registy supported: methods=[2] modes=[2,1] 2023-10-05T19:28:34.367 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.362+0000 7fce14492e40 20 check_secure_mon_conn(): mode 1 is insecure 2023-10-05T19:28:34.368 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.362+0000 7fce14492e40 1 Processor -- start 2023-10-05T19:28:34.368 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.362+0000 7fce14492e40 1 -- start start 2023-10-05T19:28:34.368 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.362+0000 7fce14492e40 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x56184d26e4e0 0x56184d26e8b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-05T19:28:34.368 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.362+0000 7fce14492e40 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x56184d26edf0 con 0x56184d26e4e0 2023-10-05T19:28:34.369 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.362+0000 7fce117c6700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x56184d26e4e0 0x56184d26e8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-05T19:28:34.369 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.362+0000 7fce117c6700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x56184d26e4e0 0x56184d26e8b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.118:3300/0 says I am v2:172.21.15.167:41418/0 (socket says 172.21.15.167:41418) 2023-10-05T19:28:34.369 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.362+0000 7fce117c6700 1 -- 172.21.15.167:0/1798521403 learned_addr learned my addr 172.21.15.167:0/1798521403 (peer_addr_for_me v2:172.21.15.167:0/0) 2023-10-05T19:28:34.369 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.362+0000 7fce117c6700 1 -- 172.21.15.167:0/1798521403 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x56184d26ef30 con 0x56184d26e4e0 2023-10-05T19:28:34.370 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.362+0000 7fce117c6700 1 --2- 172.21.15.167:0/1798521403 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x56184d26e4e0 0x56184d26e8b0 secure :-1 s=READY pgs=396 cs=0 l=1 rev1=1 crypto rx=0x56184d252d80 tx=0x7fce0c006000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=abd090a6e45202b4 server_cookie=0 in_seq=0 out_seq=0 2023-10-05T19:28:34.370 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.366+0000 7fcde27e4700 1 -- 172.21.15.167:0/1798521403 <== mon.0 v2:172.21.15.118:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fce0c023040 con 0x56184d26e4e0 2023-10-05T19:28:34.370 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.366+0000 7fcde27e4700 1 -- 172.21.15.167:0/1798521403 <== mon.0 v2:172.21.15.118:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fce0c024040 con 0x56184d26e4e0 2023-10-05T19:28:34.370 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.366+0000 7fcde27e4700 1 -- 172.21.15.167:0/1798521403 <== mon.0 v2:172.21.15.118:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fce0c013040 con 0x56184d26e4e0 2023-10-05T19:28:34.371 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.366+0000 7fce14492e40 1 -- 172.21.15.167:0/1798521403 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x56184d26e4e0 msgr2=0x56184d26e8b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-05T19:28:34.371 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.366+0000 7fce14492e40 1 --2- 172.21.15.167:0/1798521403 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x56184d26e4e0 0x56184d26e8b0 secure :-1 s=READY pgs=396 cs=0 l=1 rev1=1 crypto rx=0x56184d252d80 tx=0x7fce0c006000 comp rx=0 tx=0).stop 2023-10-05T19:28:34.372 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.366+0000 7fce14492e40 1 -- 172.21.15.167:0/1798521403 shutdown_connections 2023-10-05T19:28:34.372 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.366+0000 7fce14492e40 1 --2- 172.21.15.167:0/1798521403 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x56184d26e4e0 0x56184d26e8b0 unknown :-1 s=CLOSED pgs=396 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-05T19:28:34.372 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.366+0000 7fce14492e40 1 -- 172.21.15.167:0/1798521403 >> 172.21.15.167:0/1798521403 conn(0x56184d2698a0 msgr2=0x56184d2679f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-05T19:28:34.372 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.366+0000 7fce14492e40 1 -- 172.21.15.167:0/1798521403 shutdown_connections 2023-10-05T19:28:34.372 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.366+0000 7fce14492e40 1 -- 172.21.15.167:0/1798521403 wait complete. 2023-10-05T19:28:34.373 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.366+0000 7fce14492e40 1 Processor -- start 2023-10-05T19:28:34.373 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.366+0000 7fce14492e40 1 -- start start 2023-10-05T19:28:34.373 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.366+0000 7fce14492e40 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x56184d26e4e0 0x56184d273fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-05T19:28:34.373 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.366+0000 7fce14492e40 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x56184d274fd0 con 0x56184d26e4e0 2023-10-05T19:28:34.373 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.366+0000 7fce117c6700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x56184d26e4e0 0x56184d273fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-05T19:28:34.373 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.366+0000 7fce117c6700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x56184d26e4e0 0x56184d273fd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.118:3300/0 says I am v2:172.21.15.167:41432/0 (socket says 172.21.15.167:41432) 2023-10-05T19:28:34.374 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.366+0000 7fce117c6700 1 -- 172.21.15.167:0/1834806530 learned_addr learned my addr 172.21.15.167:0/1834806530 (peer_addr_for_me v2:172.21.15.167:0/0) 2023-10-05T19:28:34.375 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.366+0000 7fce117c6700 1 -- 172.21.15.167:0/1834806530 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x56184d16abd0 con 0x56184d26e4e0 2023-10-05T19:28:34.375 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.370+0000 7fce117c6700 1 --2- 172.21.15.167:0/1834806530 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x56184d26e4e0 0x56184d273fd0 secure :-1 s=READY pgs=397 cs=0 l=1 rev1=1 crypto rx=0x56184d24f280 tx=0x7fce0c030000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-05T19:28:34.375 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.370+0000 7fcde0fe1700 1 -- 172.21.15.167:0/1834806530 <== mon.0 v2:172.21.15.118:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fce0c023040 con 0x56184d26e4e0 2023-10-05T19:28:34.375 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.370+0000 7fce14492e40 1 -- 172.21.15.167:0/1834806530 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x56184d2745d0 con 0x56184d26e4e0 2023-10-05T19:28:34.375 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.370+0000 7fcde0fe1700 1 -- 172.21.15.167:0/1834806530 <== mon.0 v2:172.21.15.118:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fce0c01ee80 con 0x56184d26e4e0 2023-10-05T19:28:34.375 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.370+0000 7fcde0fe1700 1 -- 172.21.15.167:0/1834806530 <== mon.0 v2:172.21.15.118:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fce0c02c410 con 0x56184d26e4e0 2023-10-05T19:28:34.375 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.370+0000 7fce14492e40 1 -- 172.21.15.167:0/1834806530 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x56184d2749b0 con 0x56184d26e4e0 2023-10-05T19:28:34.376 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.370+0000 7fcde0fe1700 1 -- 172.21.15.167:0/1834806530 <== mon.0 v2:172.21.15.118:3300/0 4 ==== mgrmap(e 5) v1 ==== 40329+0+0 (secure 0 0 0) 0x7fce0c02c610 con 0x56184d26e4e0 2023-10-05T19:28:34.376 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.370+0000 7fcde0fe1700 1 --2- 172.21.15.167:0/1834806530 >> [v2:172.21.15.167:6824/29889,v1:172.21.15.167:6825/29889] conn(0x7fcd58030c00 0x7fcd580330b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-05T19:28:34.377 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.370+0000 7fcde0fe1700 1 -- 172.21.15.167:0/1834806530 <== mon.0 v2:172.21.15.118:3300/0 5 ==== osd_map(30..30 src has 1..30) v4 ==== 6057+0+0 (secure 0 0 0) 0x7fce0c03bc40 con 0x56184d26e4e0 2023-10-05T19:28:34.377 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.370+0000 7fce127c8700 1 --2- 172.21.15.167:0/1834806530 >> [v2:172.21.15.167:6824/29889,v1:172.21.15.167:6825/29889] conn(0x7fcd58030c00 0x7fcd580330b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-05T19:28:34.377 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.370+0000 7fce14492e40 5 note: GC not initialized 2023-10-05T19:28:34.377 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.370+0000 7fce127c8700 1 --2- 172.21.15.167:0/1834806530 >> [v2:172.21.15.167:6824/29889,v1:172.21.15.167:6825/29889] conn(0x7fcd58030c00 0x7fcd580330b0 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x56184d2603d0 tx=0x7fce04000f90 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-05T19:28:34.379 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.374+0000 7fcd51ffb700 20 reqs_thread_entry: start 2023-10-05T19:28:34.380 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.374+0000 7fce14492e40 20 init_complete bucket index max shards: 11 2023-10-05T19:28:34.380 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.374+0000 7fcd437fe700 20 reqs_thread_entry: start 2023-10-05T19:28:34.380 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.374+0000 7fce14492e40 10 rgw notify: Started notification manager with: 1 workers 2023-10-05T19:28:34.381 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.374+0000 7fcd41ffb700 20 rgw notify: INFO: next queues processing will happen at: Thu Oct 5 19:29:04 2023 2023-10-05T19:28:34.381 INFO:teuthology.orchestra.run.smithi167.stderr: 2023-10-05T19:28:34.381 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.374+0000 7fcd417fa700 20 reqs_thread_entry: start 2023-10-05T19:28:34.381 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.374+0000 7fce14492e40 20 get_system_obj_state: rctx=0x7ffea9b3ab68 obj=default.rgw.meta:users.uid:testx$tenanteduser state=0x56184d0f70b0 s->prefetch_data=0 2023-10-05T19:28:34.381 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.374+0000 7fce14492e40 10 cache get: name=default.rgw.meta+users.uid+testx$tenanteduser : miss 2023-10-05T19:28:34.381 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.374+0000 7fce14492e40 1 -- 172.21.15.167:0/3157424682 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:102 8.13 8:cdfb9947:users.uid::testx$tenanteduser:head [call version.read in=11b,getxattrs,stat] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x56184d2dc850 con 0x7fce0c00d3d0 2023-10-05T19:28:34.382 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.374+0000 7fce127c8700 1 -- 172.21.15.167:0/3157424682 <== osd.0 v2:172.21.15.118:6800/27690 29 ==== osd_op_reply(102 testx$tenanteduser [call,getxattrs,stat] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 ==== 246+0+0 (crc 0 0 0) 0x7fce04050040 con 0x7fce0c00d3d0 2023-10-05T19:28:34.382 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.374+0000 7fce14492e40 10 cache put: name=default.rgw.meta+users.uid+testx$tenanteduser info.flags=0x0 2023-10-05T19:28:34.382 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.374+0000 7fce14492e40 10 adding default.rgw.meta+users.uid+testx$tenanteduser to cache LRU end 2023-10-05T19:28:34.382 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.374+0000 7fce14492e40 20 get_system_obj_state: rctx=0x7ffea9b3a858 obj=default.rgw.meta:users.keys:p5YnriCv1nAtykxBrupQ state=0x56184d0f70b0 s->prefetch_data=0 2023-10-05T19:28:34.382 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.374+0000 7fce14492e40 10 cache get: name=default.rgw.meta+users.keys+p5YnriCv1nAtykxBrupQ : miss 2023-10-05T19:28:34.382 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.374+0000 7fce14492e40 1 -- 172.21.15.167:0/3157424682 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:103 8.13 8:cc695d75:users.keys::p5YnriCv1nAtykxBrupQ:head [getxattrs,stat] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x56184d2dc850 con 0x7fce0c00d3d0 2023-10-05T19:28:34.383 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.378+0000 7fce127c8700 1 -- 172.21.15.167:0/3157424682 <== osd.0 v2:172.21.15.118:6800/27690 30 ==== osd_op_reply(103 p5YnriCv1nAtykxBrupQ [getxattrs,stat] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 ==== 206+0+0 (crc 0 0 0) 0x7fce04050040 con 0x7fce0c00d3d0 2023-10-05T19:28:34.383 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.378+0000 7fce14492e40 10 cache put: name=default.rgw.meta+users.keys+p5YnriCv1nAtykxBrupQ info.flags=0x0 2023-10-05T19:28:34.383 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.378+0000 7fce14492e40 10 adding default.rgw.meta+users.keys+p5YnriCv1nAtykxBrupQ to cache LRU end 2023-10-05T19:28:34.384 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.378+0000 7fce14492e40 20 get_system_obj_state: rctx=0x7ffea9b3a0f8 obj=default.rgw.meta:users.keys:p5YnriCv1nAtykxBrupQ state=0x56184d0f70b0 s->prefetch_data=0 2023-10-05T19:28:34.384 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.378+0000 7fce14492e40 10 cache get: name=default.rgw.meta+users.keys+p5YnriCv1nAtykxBrupQ : hit (negative entry) 2023-10-05T19:28:34.384 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.378+0000 7fce14492e40 20 get_system_obj_state: rctx=0x7ffea9b3a798 obj=default.rgw.meta:users.keys:p5YnriCv1nAtykxBrupQ state=0x56184d0f70b0 s->prefetch_data=0 2023-10-05T19:28:34.384 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.378+0000 7fce14492e40 10 cache get: name=default.rgw.meta+users.keys+p5YnriCv1nAtykxBrupQ : hit (negative entry) 2023-10-05T19:28:34.384 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.378+0000 7fce14492e40 1 -- 172.21.15.167:0/3157424682 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:104 8.13 8:cdfb9947:users.uid::testx$tenanteduser:head [delete,create,call version.set in=58b,writefull 0~385 in=385b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x56184d2def60 con 0x7fce0c00d3d0 2023-10-05T19:28:34.386 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.378+0000 7fce127c8700 1 -- 172.21.15.167:0/3157424682 <== osd.0 v2:172.21.15.118:6800/27690 31 ==== osd_op_reply(104 testx$tenanteduser [delete,create,call,writefull 0~385] v30'8 uv8 ondisk = 0) v8 ==== 288+0+0 (crc 0 0 0) 0x7fce04050040 con 0x7fce0c00d3d0 2023-10-05T19:28:34.386 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.382+0000 7fce14492e40 10 cache put: name=default.rgw.meta+users.uid+testx$tenanteduser info.flags=0x17 2023-10-05T19:28:34.386 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.382+0000 7fce14492e40 10 moving default.rgw.meta+users.uid+testx$tenanteduser to cache LRU end 2023-10-05T19:28:34.386 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.382+0000 7fce14492e40 10 distributing notification oid=default.rgw.control:notify.3 cni=[op: 0, obj: default.rgw.meta:users.uid:testx$tenanteduser, ofs0, ns] 2023-10-05T19:28:34.386 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.382+0000 7fce14492e40 1 -- 172.21.15.167:0/3157424682 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:105 7.e 7:7759931f:::notify.3:head [notify cookie 94662374055392 in=588b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x56184d2de720 con 0x7fcdfc01b740 2023-10-05T19:28:34.387 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.382+0000 7fce117c6700 1 -- 172.21.15.167:0/3157424682 <== osd.2 v2:172.21.15.167:6816/30037 17 ==== watch-notify(notify (1) cookie 94662373524016 notify 128849018892 ret 0) v3 ==== 618+0+0 (crc 0 0 0) 0x56184d2749b0 con 0x7fcdfc01b740 2023-10-05T19:28:34.387 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.382+0000 7fce0b7fe700 10 rgw watcher librados: RGWWatcher::handle_notify() notify_id 128849018892 cookie 94662373524016 notifier 4485 bl.length()=576 2023-10-05T19:28:34.388 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.382+0000 7fce0b7fe700 10 rgw watcher librados: cache put: name=default.rgw.meta+users.uid+testx$tenanteduser info.flags=0x17 2023-10-05T19:28:34.388 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.382+0000 7fce0b7fe700 10 rgw watcher librados: moving default.rgw.meta+users.uid+testx$tenanteduser to cache LRU end 2023-10-05T19:28:34.388 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.382+0000 7fce117c6700 1 -- 172.21.15.167:0/3157424682 <== osd.2 v2:172.21.15.167:6816/30037 18 ==== osd_op_reply(105 notify.3 [notify cookie 94662374055392 out=8b] v0'0 uv129 ondisk = 0) v8 ==== 152+0+8 (crc 0 0 0) 0x7fce0c04a070 con 0x7fcdfc01b740 2023-10-05T19:28:34.388 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.382+0000 7fce0b7fe700 1 -- 172.21.15.167:0/3157424682 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:106 7.e 7:7759931f:::notify.3:head [notify-ack in=20b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x7fcdf0003d90 con 0x7fcdfc01b740 2023-10-05T19:28:34.388 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.382+0000 7fce117c6700 1 -- 172.21.15.167:0/3157424682 <== osd.2 v2:172.21.15.167:6816/30037 19 ==== osd_op_reply(106 notify.3 [notify-ack] v0'0 uv129 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7fce0c04a070 con 0x7fcdfc01b740 2023-10-05T19:28:34.388 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.382+0000 7fce117c6700 1 -- 172.21.15.167:0/3157424682 <== osd.2 v2:172.21.15.167:6816/30037 20 ==== watch-notify(notify_complete (2) cookie 94662374055392 notify 128849018892 ret 0) v3 ==== 42+0+48 (crc 0 0 0) 0x56184d2745d0 con 0x7fcdfc01b740 2023-10-05T19:28:34.388 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.382+0000 7fce14492e40 1 -- 172.21.15.167:0/3157424682 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:107 8.13 8:cc695d75:users.keys::p5YnriCv1nAtykxBrupQ:head [delete,create,writefull 0~22 in=22b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x56184d2dd2c0 con 0x7fce0c00d3d0 2023-10-05T19:28:34.390 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.386+0000 7fce127c8700 1 -- 172.21.15.167:0/3157424682 <== osd.0 v2:172.21.15.118:6800/27690 32 ==== osd_op_reply(107 p5YnriCv1nAtykxBrupQ [delete,create,writefull 0~22] v30'9 uv9 ondisk = 0) v8 ==== 248+0+0 (crc 0 0 0) 0x7fce04050040 con 0x7fce0c00d3d0 2023-10-05T19:28:34.390 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.386+0000 7fce14492e40 10 cache put: name=default.rgw.meta+users.keys+p5YnriCv1nAtykxBrupQ info.flags=0x7 2023-10-05T19:28:34.391 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.386+0000 7fce14492e40 10 moving default.rgw.meta+users.keys+p5YnriCv1nAtykxBrupQ to cache LRU end 2023-10-05T19:28:34.391 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.386+0000 7fce14492e40 10 distributing notification oid=default.rgw.control:notify.2 cni=[op: 0, obj: default.rgw.meta:users.keys:p5YnriCv1nAtykxBrupQ, ofs0, ns] 2023-10-05T19:28:34.391 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.386+0000 7fce14492e40 1 -- 172.21.15.167:0/3157424682 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:108 7.15 7:a93a5511:::notify.2:head [notify cookie 94662374051280 in=204b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x56184d2dd220 con 0x56184d255620 2023-10-05T19:28:34.391 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.386+0000 7fce11fc7700 1 -- 172.21.15.167:0/3157424682 <== osd.3 v2:172.21.15.167:6808/30036 24 ==== watch-notify(notify (1) cookie 94662373525456 notify 128849018893 ret 0) v3 ==== 234+0+0 (crc 0 0 0) 0x56184d246650 con 0x56184d255620 2023-10-05T19:28:34.391 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.386+0000 7fce0b7fe700 10 rgw watcher librados: RGWWatcher::handle_notify() notify_id 128849018893 cookie 94662373525456 notifier 4485 bl.length()=192 2023-10-05T19:28:34.392 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.386+0000 7fce0b7fe700 10 rgw watcher librados: cache put: name=default.rgw.meta+users.keys+p5YnriCv1nAtykxBrupQ info.flags=0x7 2023-10-05T19:28:34.392 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.386+0000 7fce0b7fe700 10 rgw watcher librados: moving default.rgw.meta+users.keys+p5YnriCv1nAtykxBrupQ to cache LRU end 2023-10-05T19:28:34.392 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.386+0000 7fce0b7fe700 1 -- 172.21.15.167:0/3157424682 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:109 7.15 7:a93a5511:::notify.2:head [notify-ack in=20b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x7fcdf0003d90 con 0x56184d255620 2023-10-05T19:28:34.392 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.386+0000 7fce11fc7700 1 -- 172.21.15.167:0/3157424682 <== osd.3 v2:172.21.15.167:6808/30036 25 ==== osd_op_reply(108 notify.2 [notify cookie 94662374051280 out=8b] v0'0 uv129 ondisk = 0) v8 ==== 152+0+8 (crc 0 0 0) 0x7fcdfc034070 con 0x56184d255620 2023-10-05T19:28:34.392 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.386+0000 7fce11fc7700 1 -- 172.21.15.167:0/3157424682 <== osd.3 v2:172.21.15.167:6808/30036 26 ==== watch-notify(notify_complete (2) cookie 94662374051280 notify 128849018893 ret 0) v3 ==== 42+0+48 (crc 0 0 0) 0x56184d246270 con 0x56184d255620 2023-10-05T19:28:34.392 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.386+0000 7fce11fc7700 1 -- 172.21.15.167:0/3157424682 <== osd.3 v2:172.21.15.167:6808/30036 27 ==== osd_op_reply(109 notify.2 [notify-ack] v0'0 uv129 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7fcdfc034070 con 0x56184d255620 2023-10-05T19:28:34.393 INFO:teuthology.orchestra.run.smithi167.stdout:{ 2023-10-05T19:28:34.393 INFO:teuthology.orchestra.run.smithi167.stdout: "user_id": "testx$tenanteduser", 2023-10-05T19:28:34.393 INFO:teuthology.orchestra.run.smithi167.stdout: "display_name": "tenanted-user", 2023-10-05T19:28:34.393 INFO:teuthology.orchestra.run.smithi167.stdout: "email": "", 2023-10-05T19:28:34.393 INFO:teuthology.orchestra.run.smithi167.stdout: "suspended": 0, 2023-10-05T19:28:34.393 INFO:teuthology.orchestra.run.smithi167.stdout: "max_buckets": 1, 2023-10-05T19:28:34.393 INFO:teuthology.orchestra.run.smithi167.stdout: "subusers": [], 2023-10-05T19:28:34.393 INFO:teuthology.orchestra.run.smithi167.stdout: "keys": [ 2023-10-05T19:28:34.394 INFO:teuthology.orchestra.run.smithi167.stdout: { 2023-10-05T19:28:34.394 INFO:teuthology.orchestra.run.smithi167.stdout: "user": "testx$tenanteduser", 2023-10-05T19:28:34.394 INFO:teuthology.orchestra.run.smithi167.stdout: "access_key": "p5YnriCv1nAtykxBrupQ", 2023-10-05T19:28:34.394 INFO:teuthology.orchestra.run.smithi167.stdout: "secret_key": "Q8Tk6Q/27hfbFSYdSkPtUqhqx1GgzvpXa4WARozh" 2023-10-05T19:28:34.394 INFO:teuthology.orchestra.run.smithi167.stdout: } 2023-10-05T19:28:34.394 INFO:teuthology.orchestra.run.smithi167.stdout: ], 2023-10-05T19:28:34.394 INFO:teuthology.orchestra.run.smithi167.stdout: "swift_keys": [], 2023-10-05T19:28:34.394 INFO:teuthology.orchestra.run.smithi167.stdout: "caps": [], 2023-10-05T19:28:34.394 INFO:teuthology.orchestra.run.smithi167.stdout: "op_mask": "read, write, delete", 2023-10-05T19:28:34.394 INFO:teuthology.orchestra.run.smithi167.stdout: "default_placement": "", 2023-10-05T19:28:34.394 INFO:teuthology.orchestra.run.smithi167.stdout: "default_storage_class": "", 2023-10-05T19:28:34.395 INFO:teuthology.orchestra.run.smithi167.stdout: "placement_tags": [], 2023-10-05T19:28:34.395 INFO:teuthology.orchestra.run.smithi167.stdout: "bucket_quota": { 2023-10-05T19:28:34.395 INFO:teuthology.orchestra.run.smithi167.stdout: "enabled": false, 2023-10-05T19:28:34.395 INFO:teuthology.orchestra.run.smithi167.stdout: "check_on_raw": false, 2023-10-05T19:28:34.395 INFO:teuthology.orchestra.run.smithi167.stdout: "max_size": -1, 2023-10-05T19:28:34.395 INFO:teuthology.orchestra.run.smithi167.stdout: "max_size_kb": 0, 2023-10-05T19:28:34.395 INFO:teuthology.orchestra.run.smithi167.stdout: "max_objects": -1 2023-10-05T19:28:34.395 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2023-10-05T19:28:34.395 INFO:teuthology.orchestra.run.smithi167.stdout: "user_quota": { 2023-10-05T19:28:34.395 INFO:teuthology.orchestra.run.smithi167.stdout: "enabled": false, 2023-10-05T19:28:34.395 INFO:teuthology.orchestra.run.smithi167.stdout: "check_on_raw": false, 2023-10-05T19:28:34.395 INFO:teuthology.orchestra.run.smithi167.stdout: "max_size": -1, 2023-10-05T19:28:34.396 INFO:teuthology.orchestra.run.smithi167.stdout: "max_size_kb": 0, 2023-10-05T19:28:34.396 INFO:teuthology.orchestra.run.smithi167.stdout: "max_objects": -1 2023-10-05T19:28:34.396 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2023-10-05T19:28:34.396 INFO:teuthology.orchestra.run.smithi167.stdout: "temp_url_keys": [], 2023-10-05T19:28:34.396 INFO:teuthology.orchestra.run.smithi167.stdout: "type": "rgw", 2023-10-05T19:28:34.396 INFO:teuthology.orchestra.run.smithi167.stdout: "mfa_ids": [] 2023-10-05T19:28:34.396 INFO:teuthology.orchestra.run.smithi167.stdout:} 2023-10-05T19:28:34.396 INFO:teuthology.orchestra.run.smithi167.stdout: 2023-10-05T19:28:34.397 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.386+0000 7fce14492e40 1 -- 172.21.15.167:0/3157424682 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:110 7.1f 7:f95f44c2:::notify.0:head [watch unwatch cookie 94662373516448] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x56184d233790 con 0x7fcdfc01b740 2023-10-05T19:28:34.397 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.390+0000 7fce117c6700 1 -- 172.21.15.167:0/3157424682 <== osd.2 v2:172.21.15.167:6816/30037 21 ==== osd_op_reply(110 notify.0 [watch unwatch cookie 94662373516448] v30'131 uv129 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7fce0c04a070 con 0x7fcdfc01b740 2023-10-05T19:28:34.397 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.390+0000 7fce14492e40 20 remove_watcher() i=0 2023-10-05T19:28:34.397 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.390+0000 7fce14492e40 2 removed watcher, disabling cache 2023-10-05T19:28:34.397 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.390+0000 7fce14492e40 1 -- 172.21.15.167:0/3157424682 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:111 7.0 7:05bf5b68:::notify.1:head [watch unwatch cookie 94662373522912] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x56184d233790 con 0x56184d21e6f0 2023-10-05T19:28:34.399 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.394+0000 7fce127c8700 1 -- 172.21.15.167:0/3157424682 <== osd.1 v2:172.21.15.167:6800/30035 35 ==== osd_op_reply(111 notify.1 [watch unwatch cookie 94662373522912] v30'131 uv129 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7fce04050040 con 0x56184d21e6f0 2023-10-05T19:28:34.399 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.394+0000 7fce14492e40 20 remove_watcher() i=1 2023-10-05T19:28:34.399 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.394+0000 7fce14492e40 1 -- 172.21.15.167:0/3157424682 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:112 7.15 7:a93a5511:::notify.2:head [watch unwatch cookie 94662373525456] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x56184d233790 con 0x56184d255620 2023-10-05T19:28:34.402 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.398+0000 7fce11fc7700 1 -- 172.21.15.167:0/3157424682 <== osd.3 v2:172.21.15.167:6808/30036 28 ==== osd_op_reply(112 notify.2 [watch unwatch cookie 94662373525456] v30'131 uv129 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7fcdfc034070 con 0x56184d255620 2023-10-05T19:28:34.402 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.398+0000 7fce14492e40 20 remove_watcher() i=2 2023-10-05T19:28:34.402 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.398+0000 7fce14492e40 1 -- 172.21.15.167:0/3157424682 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:113 7.e 7:7759931f:::notify.3:head [watch unwatch cookie 94662373524016] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x56184d233790 con 0x7fcdfc01b740 2023-10-05T19:28:34.404 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.398+0000 7fce117c6700 1 -- 172.21.15.167:0/3157424682 <== osd.2 v2:172.21.15.167:6816/30037 22 ==== osd_op_reply(113 notify.3 [watch unwatch cookie 94662373524016] v30'131 uv129 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7fce0c053040 con 0x7fcdfc01b740 2023-10-05T19:28:34.404 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.398+0000 7fce14492e40 20 remove_watcher() i=3 2023-10-05T19:28:34.404 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.398+0000 7fce14492e40 1 -- 172.21.15.167:0/3157424682 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:114 7.d 7:b4812045:::notify.4:head [watch unwatch cookie 94662373527744] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x56184d25c2d0 con 0x56184d21e6f0 2023-10-05T19:28:34.407 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.402+0000 7fce127c8700 1 -- 172.21.15.167:0/3157424682 <== osd.1 v2:172.21.15.167:6800/30035 36 ==== osd_op_reply(114 notify.4 [watch unwatch cookie 94662373527744] v30'131 uv129 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7fce04050040 con 0x56184d21e6f0 2023-10-05T19:28:34.407 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.402+0000 7fce14492e40 20 remove_watcher() i=4 2023-10-05T19:28:34.407 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.402+0000 7fce14492e40 1 -- 172.21.15.167:0/3157424682 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:115 7.3 7:c609908c:::notify.5:head [watch unwatch cookie 94662373532528] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x56184d25c2d0 con 0x7fce0c00d3d0 2023-10-05T19:28:34.410 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.402+0000 7fce127c8700 1 -- 172.21.15.167:0/3157424682 <== osd.0 v2:172.21.15.118:6800/27690 33 ==== osd_op_reply(115 notify.5 [watch unwatch cookie 94662373532528] v30'131 uv129 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7fce04050040 con 0x7fce0c00d3d0 2023-10-05T19:28:34.410 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.402+0000 7fce14492e40 20 remove_watcher() i=5 2023-10-05T19:28:34.410 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.402+0000 7fce14492e40 1 -- 172.21.15.167:0/3157424682 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:116 7.14 7:2b04a3e9:::notify.6:head [watch unwatch cookie 94662373539200] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x56184d25c2d0 con 0x7fce0c00d3d0 2023-10-05T19:28:34.413 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.406+0000 7fce127c8700 1 -- 172.21.15.167:0/3157424682 <== osd.0 v2:172.21.15.118:6800/27690 34 ==== osd_op_reply(116 notify.6 [watch unwatch cookie 94662373539200] v30'131 uv129 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7fce04050040 con 0x7fce0c00d3d0 2023-10-05T19:28:34.413 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.406+0000 7fce14492e40 20 remove_watcher() i=6 2023-10-05T19:28:34.413 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.406+0000 7fce14492e40 1 -- 172.21.15.167:0/3157424682 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:117 7.9 7:93e5b521:::notify.7:head [watch unwatch cookie 94662373536624] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x56184d233790 con 0x7fce0c00d3d0 2023-10-05T19:28:34.416 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.410+0000 7fce127c8700 1 -- 172.21.15.167:0/3157424682 <== osd.0 v2:172.21.15.118:6800/27690 35 ==== osd_op_reply(117 notify.7 [watch unwatch cookie 94662373536624] v30'131 uv129 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7fce04050040 con 0x7fce0c00d3d0 2023-10-05T19:28:34.416 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.410+0000 7fce14492e40 20 remove_watcher() i=7 2023-10-05T19:28:34.418 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.410+0000 7fcde2fe5700 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2023-10-05T19:28:34.418 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.410+0000 7fce14492e40 1 -- 172.21.15.167:0/3157424682 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:118 6.4 6:22d26bf9:::data_loggenerations_metadata:head [watch unwatch cookie 94662373542880] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x56184d233790 con 0x56184d21e6f0 2023-10-05T19:28:34.421 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.414+0000 7fce127c8700 1 -- 172.21.15.167:0/3157424682 <== osd.1 v2:172.21.15.167:6800/30035 37 ==== osd_op_reply(118 data_loggenerations_metadata [watch unwatch cookie 94662373542880] v30'128 uv1 ondisk = 0) v8 ==== 172+0+0 (crc 0 0 0) 0x7fce04050040 con 0x56184d21e6f0 2023-10-05T19:28:34.422 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.414+0000 7fce14492e40 1 -- 172.21.15.167:0/3157424682 >> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] conn(0x7fce0c00d3d0 msgr2=0x7fce0c00d820 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-05T19:28:34.422 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.414+0000 7fce14492e40 1 --2- 172.21.15.167:0/3157424682 >> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] conn(0x7fce0c00d3d0 0x7fce0c00d820 crc :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-05T19:28:34.422 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.414+0000 7fce14492e40 1 -- 172.21.15.167:0/3157424682 >> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] conn(0x56184d21e6f0 msgr2=0x56184d2517f0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-05T19:28:34.422 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.414+0000 7fce14492e40 1 --2- 172.21.15.167:0/3157424682 >> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] conn(0x56184d21e6f0 0x56184d2517f0 crc :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-05T19:28:34.422 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.414+0000 7fce14492e40 1 -- 172.21.15.167:0/3157424682 >> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] conn(0x7fcdfc01b740 msgr2=0x7fcdfc0019a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-05T19:28:34.422 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.414+0000 7fce14492e40 1 --2- 172.21.15.167:0/3157424682 >> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] conn(0x7fcdfc01b740 0x7fcdfc0019a0 crc :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-05T19:28:34.422 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.414+0000 7fce14492e40 1 -- 172.21.15.167:0/3157424682 >> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] conn(0x56184d255620 msgr2=0x56184d257a00 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-05T19:28:34.422 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.414+0000 7fce14492e40 1 --2- 172.21.15.167:0/3157424682 >> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] conn(0x56184d255620 0x56184d257a00 crc :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-05T19:28:34.422 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.414+0000 7fce14492e40 1 -- 172.21.15.167:0/3157424682 >> [v2:172.21.15.167:6824/29889,v1:172.21.15.167:6825/29889] conn(0x7fcdf4030bf0 msgr2=0x7fcdf40330a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-05T19:28:34.423 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.414+0000 7fce14492e40 1 --2- 172.21.15.167:0/3157424682 >> [v2:172.21.15.167:6824/29889,v1:172.21.15.167:6825/29889] conn(0x7fcdf4030bf0 0x7fcdf40330a0 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7fce0c004170 tx=0x7fce0c001380 comp rx=0 tx=0).stop 2023-10-05T19:28:34.423 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.414+0000 7fce14492e40 1 -- 172.21.15.167:0/3157424682 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x56184d21dde0 msgr2=0x56184d245d30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-05T19:28:34.423 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.414+0000 7fce127c8700 1 -- 172.21.15.167:0/3157424682 reap_dead start 2023-10-05T19:28:34.423 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.414+0000 7fce14492e40 1 --2- 172.21.15.167:0/3157424682 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x56184d21dde0 0x56184d245d30 secure :-1 s=READY pgs=395 cs=0 l=1 rev1=1 crypto rx=0x7fcdfc00a520 tx=0x7fcdfc01d000 comp rx=0 tx=0).stop 2023-10-05T19:28:34.423 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.414+0000 7fce14492e40 1 -- 172.21.15.167:0/3157424682 shutdown_connections 2023-10-05T19:28:34.423 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.414+0000 7fce14492e40 1 --2- 172.21.15.167:0/3157424682 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x56184d21dde0 0x56184d245d30 unknown :-1 s=CLOSED pgs=395 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-05T19:28:34.423 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.414+0000 7fce14492e40 1 -- 172.21.15.167:0/3157424682 >> 172.21.15.167:0/3157424682 conn(0x56184d219800 msgr2=0x56184d237530 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-05T19:28:34.423 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.414+0000 7fce14492e40 1 -- 172.21.15.167:0/3157424682 shutdown_connections 2023-10-05T19:28:34.423 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.414+0000 7fce14492e40 1 -- 172.21.15.167:0/3157424682 wait complete. 2023-10-05T19:28:34.424 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.418+0000 7fce14492e40 1 -- 172.21.15.167:0/1834806530 >> [v2:172.21.15.167:6824/29889,v1:172.21.15.167:6825/29889] conn(0x7fcd58030c00 msgr2=0x7fcd580330b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-05T19:28:34.424 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.418+0000 7fce14492e40 1 --2- 172.21.15.167:0/1834806530 >> [v2:172.21.15.167:6824/29889,v1:172.21.15.167:6825/29889] conn(0x7fcd58030c00 0x7fcd580330b0 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x56184d2603d0 tx=0x7fce04000f90 comp rx=0 tx=0).stop 2023-10-05T19:28:34.424 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.418+0000 7fce14492e40 1 -- 172.21.15.167:0/1834806530 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x56184d26e4e0 msgr2=0x56184d273fd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-05T19:28:34.424 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.418+0000 7fce14492e40 1 --2- 172.21.15.167:0/1834806530 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x56184d26e4e0 0x56184d273fd0 secure :-1 s=READY pgs=397 cs=0 l=1 rev1=1 crypto rx=0x56184d24f280 tx=0x7fce0c030000 comp rx=0 tx=0).stop 2023-10-05T19:28:34.425 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.418+0000 7fce14492e40 1 -- 172.21.15.167:0/1834806530 shutdown_connections 2023-10-05T19:28:34.425 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.418+0000 7fce14492e40 1 --2- 172.21.15.167:0/1834806530 >> [v2:172.21.15.167:6824/29889,v1:172.21.15.167:6825/29889] conn(0x7fcd58030c00 0x7fcd580330b0 unknown :-1 s=CLOSED pgs=139 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-05T19:28:34.425 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.418+0000 7fce14492e40 1 --2- 172.21.15.167:0/1834806530 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x56184d26e4e0 0x56184d273fd0 unknown :-1 s=CLOSED pgs=397 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-05T19:28:34.425 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.418+0000 7fce14492e40 1 -- 172.21.15.167:0/1834806530 >> 172.21.15.167:0/1834806530 conn(0x56184d2698a0 msgr2=0x56184d267940 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-05T19:28:34.425 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.418+0000 7fce14492e40 1 -- 172.21.15.167:0/1834806530 shutdown_connections 2023-10-05T19:28:34.425 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.418+0000 7fce14492e40 1 -- 172.21.15.167:0/1834806530 wait complete. 2023-10-05T19:28:34.429 DEBUG:tasks.util.rgw: json result: {'user_id': 'testx$tenanteduser', 'display_name': 'tenanted-user', 'email': '', 'suspended': 0, 'max_buckets': 1, 'subusers': [], 'keys': [{'user': 'testx$tenanteduser', 'access_key': 'p5YnriCv1nAtykxBrupQ', 'secret_key': 'Q8Tk6Q/27hfbFSYdSkPtUqhqx1GgzvpXa4WARozh'}], 'swift_keys': [], 'caps': [], 'op_mask': 'read, write, delete', 'default_placement': '', 'default_storage_class': '', 'placement_tags': [], 'bucket_quota': {'enabled': False, 'check_on_raw': False, 'max_size': -1, 'max_size_kb': 0, 'max_objects': -1}, 'user_quota': {'enabled': False, 'check_on_raw': False, 'max_size': -1, 'max_size_kb': 0, 'max_objects': -1}, 'temp_url_keys': [], 'type': 'rgw', 'mfa_ids': []} 2023-10-05T19:28:34.430 INFO:tasks.util.rgw:rgwadmin: client.0 : ['bucket', 'link', '--bucket', '/myfoo', '--tenant', 'testx', '--uid', 'tenanteduser'] 2023-10-05T19:28:34.430 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'ceph', 'bucket', 'link', '--bucket', '/myfoo', '--tenant', 'testx', '--uid', 'tenanteduser'] 2023-10-05T19:28:34.430 DEBUG:teuthology.orchestra.run.smithi167:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster ceph bucket link --bucket /myfoo --tenant testx --uid tenanteduser 2023-10-05T19:28:34.473 INFO:teuthology.orchestra.run.smithi167.stderr:ignoring --setuser ceph since I am not root 2023-10-05T19:28:34.473 INFO:teuthology.orchestra.run.smithi167.stderr:ignoring --setgroup ceph since I am not root 2023-10-05T19:28:34.474 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.470+0000 7fdd70cb6e40 1 Processor -- start 2023-10-05T19:28:34.474 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.470+0000 7fdd70cb6e40 1 -- start start 2023-10-05T19:28:34.474 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.470+0000 7fdd70cb6e40 1 --2- >> v2:172.21.15.118:3300/0 conn(0x557d096e0fb0 0x557d096e1380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-05T19:28:34.475 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.470+0000 7fdd70cb6e40 1 -- --> v1:172.21.15.118:6789/0 -- auth(proto 0 26 bytes epoch 0) v1 -- 0x557d0962bbd0 con 0x557d096e18c0 2023-10-05T19:28:34.475 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.470+0000 7fdd70cb6e40 1 -- --> v2:172.21.15.118:3300/0 -- mon_getmap magic: 0 v1 -- 0x557d095e8ee0 con 0x557d096e0fb0 2023-10-05T19:28:34.476 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.470+0000 7fdd6e7eb700 1 --2- >> v2:172.21.15.118:3300/0 conn(0x557d096e0fb0 0x557d096e1380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-05T19:28:34.476 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.470+0000 7fdd6dfea700 1 --1- >> v1:172.21.15.118:6789/0 conn(0x557d096e18c0 0x557d096e2360 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.118:6789/0 says I am v1:172.21.15.167:59260/0 (socket says 172.21.15.167:59260) 2023-10-05T19:28:34.476 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.470+0000 7fdd6dfea700 1 -- 172.21.15.167:0/52119595 learned_addr learned my addr 172.21.15.167:0/52119595 (peer_addr_for_me v1:172.21.15.167:0/0) 2023-10-05T19:28:34.476 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.470+0000 7fdd6d7e9700 1 -- 172.21.15.167:0/52119595 <== mon.0 v1:172.21.15.118:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1028860212 0 0) 0x557d0962bbd0 con 0x557d096e18c0 2023-10-05T19:28:34.476 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.470+0000 7fdd6d7e9700 1 -- 172.21.15.167:0/52119595 --> v1:172.21.15.118:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdd58001e10 con 0x557d096e18c0 2023-10-05T19:28:34.476 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.470+0000 7fdd6e7eb700 1 -- 172.21.15.167:0/52119595 >> v1:172.21.15.118:6789/0 conn(0x557d096e18c0 legacy=0x557d096e2360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-05T19:28:34.476 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.470+0000 7fdd6e7eb700 1 -- 172.21.15.167:0/52119595 --> v2:172.21.15.118:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x557d096e2a70 con 0x557d096e0fb0 2023-10-05T19:28:34.477 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.470+0000 7fdd6e7eb700 1 --2- 172.21.15.167:0/52119595 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x557d096e0fb0 0x557d096e1380 secure :-1 s=READY pgs=399 cs=0 l=1 rev1=1 crypto rx=0x7fdd6000a5d0 tx=0x7fdd60005a50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=35ec9404a621d169 server_cookie=0 in_seq=0 out_seq=0 2023-10-05T19:28:34.477 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.470+0000 7fdd6d7e9700 1 -- 172.21.15.167:0/52119595 <== mon.0 v2:172.21.15.118:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fdd60007610 con 0x557d096e0fb0 2023-10-05T19:28:34.478 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.470+0000 7fdd6d7e9700 1 -- 172.21.15.167:0/52119595 <== mon.0 v2:172.21.15.118:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdd60007770 con 0x557d096e0fb0 2023-10-05T19:28:34.478 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.470+0000 7fdd6d7e9700 1 -- 172.21.15.167:0/52119595 <== mon.0 v2:172.21.15.118:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fdd60007a40 con 0x557d096e0fb0 2023-10-05T19:28:34.478 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.470+0000 7fdd70cb6e40 1 -- 172.21.15.167:0/52119595 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x557d096e0fb0 msgr2=0x557d096e1380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-05T19:28:34.478 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.470+0000 7fdd70cb6e40 1 --2- 172.21.15.167:0/52119595 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x557d096e0fb0 0x557d096e1380 secure :-1 s=READY pgs=399 cs=0 l=1 rev1=1 crypto rx=0x7fdd6000a5d0 tx=0x7fdd60005a50 comp rx=0 tx=0).stop 2023-10-05T19:28:34.478 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.470+0000 7fdd70cb6e40 1 -- 172.21.15.167:0/52119595 shutdown_connections 2023-10-05T19:28:34.478 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.470+0000 7fdd70cb6e40 1 --2- 172.21.15.167:0/52119595 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x557d096e0fb0 0x557d096e1380 unknown :-1 s=CLOSED pgs=399 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-05T19:28:34.478 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.470+0000 7fdd70cb6e40 1 -- 172.21.15.167:0/52119595 >> 172.21.15.167:0/52119595 conn(0x557d09654800 msgr2=0x557d09654c00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-05T19:28:34.478 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.470+0000 7fdd70cb6e40 1 -- 172.21.15.167:0/52119595 shutdown_connections 2023-10-05T19:28:34.478 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.470+0000 7fdd70cb6e40 1 -- 172.21.15.167:0/52119595 wait complete. 2023-10-05T19:28:34.479 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.474+0000 7fdd70cb6e40 1 Processor -- start 2023-10-05T19:28:34.479 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.474+0000 7fdd70cb6e40 1 -- start start 2023-10-05T19:28:34.479 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.474+0000 7fdd70cb6e40 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x557d096e0fb0 0x557d096fff30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-05T19:28:34.480 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.474+0000 7fdd70cb6e40 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x557d095ace30 con 0x557d096e0fb0 2023-10-05T19:28:34.480 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.474+0000 7fdd6e7eb700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x557d096e0fb0 0x557d096fff30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-05T19:28:34.480 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.474+0000 7fdd6e7eb700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x557d096e0fb0 0x557d096fff30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.118:3300/0 says I am v2:172.21.15.167:41464/0 (socket says 172.21.15.167:41464) 2023-10-05T19:28:34.480 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.474+0000 7fdd6e7eb700 1 -- 172.21.15.167:0/1753627466 learned_addr learned my addr 172.21.15.167:0/1753627466 (peer_addr_for_me v2:172.21.15.167:0/0) 2023-10-05T19:28:34.480 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.474+0000 7fdd6e7eb700 1 -- 172.21.15.167:0/1753627466 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdd60005cf0 con 0x557d096e0fb0 2023-10-05T19:28:34.481 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.474+0000 7fdd6e7eb700 1 --2- 172.21.15.167:0/1753627466 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x557d096e0fb0 0x557d096fff30 secure :-1 s=READY pgs=400 cs=0 l=1 rev1=1 crypto rx=0x7fdd60000c00 tx=0x7fdd60014000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3875ffddb40b4d63 server_cookie=0 in_seq=0 out_seq=0 2023-10-05T19:28:34.482 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.474+0000 7fdd5ffff700 1 -- 172.21.15.167:0/1753627466 <== mon.0 v2:172.21.15.118:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fdd600078e0 con 0x557d096e0fb0 2023-10-05T19:28:34.482 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.474+0000 7fdd5ffff700 1 -- 172.21.15.167:0/1753627466 <== mon.0 v2:172.21.15.118:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdd60007a40 con 0x557d096e0fb0 2023-10-05T19:28:34.482 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.474+0000 7fdd5ffff700 1 -- 172.21.15.167:0/1753627466 <== mon.0 v2:172.21.15.118:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fdd60007bd0 con 0x557d096e0fb0 2023-10-05T19:28:34.482 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.474+0000 7fdd70cb6e40 1 -- 172.21.15.167:0/1753627466 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x557d096e0fb0 msgr2=0x557d096fff30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-05T19:28:34.482 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.474+0000 7fdd70cb6e40 1 --2- 172.21.15.167:0/1753627466 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x557d096e0fb0 0x557d096fff30 secure :-1 s=READY pgs=400 cs=0 l=1 rev1=1 crypto rx=0x7fdd60000c00 tx=0x7fdd60014000 comp rx=0 tx=0).stop 2023-10-05T19:28:34.482 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.474+0000 7fdd70cb6e40 1 -- 172.21.15.167:0/1753627466 shutdown_connections 2023-10-05T19:28:34.482 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.474+0000 7fdd70cb6e40 1 --2- 172.21.15.167:0/1753627466 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x557d096e0fb0 0x557d096fff30 unknown :-1 s=CLOSED pgs=400 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-05T19:28:34.482 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.474+0000 7fdd70cb6e40 1 -- 172.21.15.167:0/1753627466 >> 172.21.15.167:0/1753627466 conn(0x557d09654800 msgr2=0x557d096fba70 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-05T19:28:34.482 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.474+0000 7fdd70cb6e40 1 -- 172.21.15.167:0/1753627466 shutdown_connections 2023-10-05T19:28:34.482 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.474+0000 7fdd70cb6e40 1 -- 172.21.15.167:0/1753627466 wait complete. 2023-10-05T19:28:34.482 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.474+0000 7fdd70cb6e40 1 Processor -- start 2023-10-05T19:28:34.483 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.474+0000 7fdd70cb6e40 1 -- start start 2023-10-05T19:28:34.483 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.474+0000 7fdd70cb6e40 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x557d096e0fb0 0x557d09706c30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-05T19:28:34.483 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.474+0000 7fdd70cb6e40 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x557d0957fed0 con 0x557d096e0fb0 2023-10-05T19:28:34.484 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.478+0000 7fdd6e7eb700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x557d096e0fb0 0x557d09706c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-05T19:28:34.484 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.478+0000 7fdd6e7eb700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x557d096e0fb0 0x557d09706c30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.118:3300/0 says I am v2:172.21.15.167:41470/0 (socket says 172.21.15.167:41470) 2023-10-05T19:28:34.484 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.478+0000 7fdd6e7eb700 1 -- 172.21.15.167:0/4130379316 learned_addr learned my addr 172.21.15.167:0/4130379316 (peer_addr_for_me v2:172.21.15.167:0/0) 2023-10-05T19:28:34.484 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.478+0000 7fdd6e7eb700 1 -- 172.21.15.167:0/4130379316 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x557d096e2a70 con 0x557d096e0fb0 2023-10-05T19:28:34.484 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.478+0000 7fdd6e7eb700 1 --2- 172.21.15.167:0/4130379316 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x557d096e0fb0 0x557d09706c30 secure :-1 s=READY pgs=401 cs=0 l=1 rev1=1 crypto rx=0x7fdd6001f040 tx=0x7fdd6001c000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-05T19:28:34.484 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.478+0000 7fdd5e7fc700 1 -- 172.21.15.167:0/4130379316 <== mon.0 v2:172.21.15.118:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fdd60007e50 con 0x557d096e0fb0 2023-10-05T19:28:34.484 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.478+0000 7fdd70cb6e40 1 -- 172.21.15.167:0/4130379316 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x557d09707170 con 0x557d096e0fb0 2023-10-05T19:28:34.485 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.478+0000 7fdd5e7fc700 1 -- 172.21.15.167:0/4130379316 <== mon.0 v2:172.21.15.118:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdd60029070 con 0x557d096e0fb0 2023-10-05T19:28:34.485 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.478+0000 7fdd5e7fc700 1 -- 172.21.15.167:0/4130379316 <== mon.0 v2:172.21.15.118:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fdd60030480 con 0x557d096e0fb0 2023-10-05T19:28:34.485 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.478+0000 7fdd70cb6e40 1 -- 172.21.15.167:0/4130379316 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x557d097075b0 con 0x557d096e0fb0 2023-10-05T19:28:34.485 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.478+0000 7fdd70cb6e40 20 rados->read ofs=0 len=0 2023-10-05T19:28:34.485 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.478+0000 7fdd5e7fc700 1 -- 172.21.15.167:0/4130379316 <== mon.0 v2:172.21.15.118:3300/0 4 ==== mgrmap(e 5) v1 ==== 40329+0+0 (secure 0 0 0) 0x7fdd60012400 con 0x557d096e0fb0 2023-10-05T19:28:34.486 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.482+0000 7fdd5e7fc700 1 --2- 172.21.15.167:0/4130379316 >> [v2:172.21.15.167:6824/29889,v1:172.21.15.167:6825/29889] conn(0x7fdd50030bf0 0x7fdd500330a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-05T19:28:34.486 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.482+0000 7fdd5e7fc700 1 -- 172.21.15.167:0/4130379316 <== mon.0 v2:172.21.15.118:3300/0 5 ==== osd_map(30..30 src has 1..30) v4 ==== 6057+0+0 (secure 0 0 0) 0x7fdd6002b070 con 0x557d096e0fb0 2023-10-05T19:28:34.491 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.482+0000 7fdd6dfea700 1 --2- 172.21.15.167:0/4130379316 >> [v2:172.21.15.167:6824/29889,v1:172.21.15.167:6825/29889] conn(0x7fdd50030bf0 0x7fdd500330a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-05T19:28:34.491 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.482+0000 7fdd70cb6e40 1 --2- 172.21.15.167:0/4130379316 >> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] conn(0x557d096e18c0 0x557d097126f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-05T19:28:34.491 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.482+0000 7fdd70cb6e40 1 -- 172.21.15.167:0/4130379316 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:1 5.12 5:49953fa1:::default.realm:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x557d09712da0 con 0x557d096e18c0 2023-10-05T19:28:34.491 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.482+0000 7fdd6efec700 1 --2- 172.21.15.167:0/4130379316 >> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] conn(0x557d096e18c0 0x557d097126f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-05T19:28:34.491 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.482+0000 7fdd6dfea700 1 --2- 172.21.15.167:0/4130379316 >> [v2:172.21.15.167:6824/29889,v1:172.21.15.167:6825/29889] conn(0x7fdd50030bf0 0x7fdd500330a0 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7fdd64004170 tx=0x7fdd64001400 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-05T19:28:34.491 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.482+0000 7fdd6efec700 1 --2- 172.21.15.167:0/4130379316 >> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] conn(0x557d096e18c0 0x557d097126f0 crc :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-05T19:28:34.491 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.482+0000 7fdd6efec700 1 -- 172.21.15.167:0/4130379316 <== osd.1 v2:172.21.15.167:6800/30035 1 ==== osd_op_reply(1 default.realm [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 ==== 157+0+0 (crc 0 0 0) 0x7fdd6800a5a0 con 0x557d096e18c0 2023-10-05T19:28:34.491 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.482+0000 7fdd70cb6e40 20 rados_obj.operate() r=-2 bl.length=0 2023-10-05T19:28:34.491 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.482+0000 7fdd70cb6e40 20 realm 2023-10-05T19:28:34.491 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.482+0000 7fdd70cb6e40 20 rados->read ofs=0 len=0 2023-10-05T19:28:34.492 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.482+0000 7fdd70cb6e40 1 -- 172.21.15.167:0/4130379316 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:2 5.12 5:49953fa1:::default.realm:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x557d0970fd50 con 0x557d096e18c0 2023-10-05T19:28:34.492 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.482+0000 7fdd6efec700 1 -- 172.21.15.167:0/4130379316 <== osd.1 v2:172.21.15.167:6800/30035 2 ==== osd_op_reply(2 default.realm [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 ==== 157+0+0 (crc 0 0 0) 0x7fdd6800a5a0 con 0x557d096e18c0 2023-10-05T19:28:34.492 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.482+0000 7fdd70cb6e40 20 rados_obj.operate() r=-2 bl.length=0 2023-10-05T19:28:34.492 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.482+0000 7fdd70cb6e40 4 RGWPeriod::init failed to init realm id : (2) No such file or directory 2023-10-05T19:28:34.492 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.482+0000 7fdd70cb6e40 20 rados->read ofs=0 len=0 2023-10-05T19:28:34.492 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.482+0000 7fdd70cb6e40 1 -- 172.21.15.167:0/4130379316 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:3 5.12 5:49953fa1:::default.realm:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x557d09713880 con 0x557d096e18c0 2023-10-05T19:28:34.492 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.482+0000 7fdd6efec700 1 -- 172.21.15.167:0/4130379316 <== osd.1 v2:172.21.15.167:6800/30035 3 ==== osd_op_reply(3 default.realm [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 ==== 157+0+0 (crc 0 0 0) 0x7fdd6800a5a0 con 0x557d096e18c0 2023-10-05T19:28:34.492 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.482+0000 7fdd70cb6e40 20 rados_obj.operate() r=-2 bl.length=0 2023-10-05T19:28:34.492 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.482+0000 7fdd70cb6e40 20 rados->read ofs=0 len=0 2023-10-05T19:28:34.492 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.482+0000 7fdd70cb6e40 1 -- 172.21.15.167:0/4130379316 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:4 5.f 5:f43b6ece:::zone_names.default:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x557d09713490 con 0x557d096e18c0 2023-10-05T19:28:34.492 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.486+0000 7fdd6efec700 1 -- 172.21.15.167:0/4130379316 <== osd.1 v2:172.21.15.167:6800/30035 4 ==== osd_op_reply(4 zone_names.default [read 0~46 out=46b] v0'0 uv1 ondisk = 0) v8 ==== 162+0+46 (crc 0 0 0) 0x7fdd6800a5a0 con 0x557d096e18c0 2023-10-05T19:28:34.493 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.486+0000 7fdd70cb6e40 20 rados_obj.operate() r=0 bl.length=46 2023-10-05T19:28:34.493 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.486+0000 7fdd70cb6e40 20 rados->read ofs=0 len=0 2023-10-05T19:28:34.493 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.486+0000 7fdd70cb6e40 1 -- 172.21.15.167:0/4130379316 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:5 5.1d 5:b8374eb4:::zone_info.9a9df965-6612-45d4-a45f-6e3c889bacc2:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x557d09714710 con 0x557d096e18c0 2023-10-05T19:28:34.494 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.486+0000 7fdd6efec700 1 -- 172.21.15.167:0/4130379316 <== osd.1 v2:172.21.15.167:6800/30035 5 ==== osd_op_reply(5 zone_info.9a9df965-6612-45d4-a45f-6e3c889bacc2 [read 0~874 out=874b] v0'0 uv2 ondisk = 0) v8 ==== 190+0+874 (crc 0 0 0) 0x7fdd6800a5a0 con 0x557d096e18c0 2023-10-05T19:28:34.494 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.486+0000 7fdd70cb6e40 20 rados_obj.operate() r=0 bl.length=874 2023-10-05T19:28:34.494 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.486+0000 7fdd70cb6e40 20 searching for the correct realm 2023-10-05T19:28:34.494 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.486+0000 7fdd70cb6e40 1 --2- 172.21.15.167:0/4130379316 >> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] conn(0x557d097164d0 0x557d097188b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-05T19:28:34.494 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.486+0000 7fdd70cb6e40 1 -- 172.21.15.167:0/4130379316 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:6 5.0 5:00000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x557d09718df0 con 0x557d097164d0 2023-10-05T19:28:34.494 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.486+0000 7fdd6e7eb700 1 --2- 172.21.15.167:0/4130379316 >> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] conn(0x557d097164d0 0x557d097188b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-05T19:28:34.494 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.486+0000 7fdd6e7eb700 1 --2- 172.21.15.167:0/4130379316 >> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] conn(0x557d097164d0 0x557d097188b0 crc :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-05T19:28:34.495 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.490+0000 7fdd6e7eb700 1 -- 172.21.15.167:0/4130379316 <== osd.3 v2:172.21.15.167:6808/30036 1 ==== osd_op_reply(6 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fdd6001c7f0 con 0x557d097164d0 2023-10-05T19:28:34.495 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.490+0000 7fdd6e7eb700 1 --2- 172.21.15.167:0/4130379316 >> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] conn(0x7fdd60007030 0x7fdd6000c9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-05T19:28:34.495 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.490+0000 7fdd6e7eb700 1 -- 172.21.15.167:0/4130379316 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:7 5.10 5:08000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x557d09718df0 con 0x7fdd60007030 2023-10-05T19:28:34.495 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.490+0000 7fdd6dfea700 1 --2- 172.21.15.167:0/4130379316 >> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] conn(0x7fdd60007030 0x7fdd6000c9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-05T19:28:34.496 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.490+0000 7fdd6dfea700 1 --2- 172.21.15.167:0/4130379316 >> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] conn(0x7fdd60007030 0x7fdd6000c9c0 crc :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-05T19:28:34.496 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.490+0000 7fdd6dfea700 1 -- 172.21.15.167:0/4130379316 <== osd.2 v2:172.21.15.167:6816/30037 1 ==== osd_op_reply(7 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fdd64006070 con 0x7fdd60007030 2023-10-05T19:28:34.496 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.490+0000 7fdd6dfea700 1 --2- 172.21.15.167:0/4130379316 >> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] conn(0x7fdd6400eb60 0x7fdd6400d8e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-05T19:28:34.496 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.490+0000 7fdd6dfea700 1 -- 172.21.15.167:0/4130379316 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:8 5.8 5:10000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x557d09718df0 con 0x7fdd6400eb60 2023-10-05T19:28:34.496 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.490+0000 7fdd6efec700 1 --2- 172.21.15.167:0/4130379316 >> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] conn(0x7fdd6400eb60 0x7fdd6400d8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-05T19:28:34.496 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.490+0000 7fdd6efec700 1 --2- 172.21.15.167:0/4130379316 >> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] conn(0x7fdd6400eb60 0x7fdd6400d8e0 crc :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-05T19:28:34.497 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.490+0000 7fdd6efec700 1 -- 172.21.15.167:0/4130379316 <== osd.0 v2:172.21.15.118:6800/27690 1 ==== osd_op_reply(8 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fdd6800a5a0 con 0x7fdd6400eb60 2023-10-05T19:28:34.497 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.490+0000 7fdd6efec700 1 -- 172.21.15.167:0/4130379316 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:9 5.18 5:18000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x557d09718df0 con 0x557d096e18c0 2023-10-05T19:28:34.498 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.494+0000 7fdd6efec700 1 -- 172.21.15.167:0/4130379316 <== osd.1 v2:172.21.15.167:6800/30035 6 ==== osd_op_reply(9 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fdd6800a5a0 con 0x557d096e18c0 2023-10-05T19:28:34.498 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.494+0000 7fdd6efec700 1 -- 172.21.15.167:0/4130379316 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:10 5.4 5:20000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x557d09718df0 con 0x557d096e18c0 2023-10-05T19:28:34.499 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.494+0000 7fdd6efec700 1 -- 172.21.15.167:0/4130379316 <== osd.1 v2:172.21.15.167:6800/30035 7 ==== osd_op_reply(10 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fdd6800a5a0 con 0x557d096e18c0 2023-10-05T19:28:34.499 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.494+0000 7fdd6efec700 1 -- 172.21.15.167:0/4130379316 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:11 5.14 5:28000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x557d09718df0 con 0x7fdd6400eb60 2023-10-05T19:28:34.499 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.494+0000 7fdd6efec700 1 -- 172.21.15.167:0/4130379316 <== osd.0 v2:172.21.15.118:6800/27690 2 ==== osd_op_reply(11 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fdd6800a5a0 con 0x7fdd6400eb60 2023-10-05T19:28:34.499 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.494+0000 7fdd6efec700 1 -- 172.21.15.167:0/4130379316 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:12 5.c 5:30000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x557d09718df0 con 0x7fdd6400eb60 2023-10-05T19:28:34.500 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.494+0000 7fdd6efec700 1 -- 172.21.15.167:0/4130379316 <== osd.0 v2:172.21.15.118:6800/27690 3 ==== osd_op_reply(12 [pgnls start_epoch 30 out=112b] v23'1 uv1 ondisk = 1) v8 ==== 144+0+112 (crc 0 0 0) 0x7fdd6800a5a0 con 0x7fdd6400eb60 2023-10-05T19:28:34.500 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.494+0000 7fdd6efec700 1 -- 172.21.15.167:0/4130379316 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:13 5.1c 5:38000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x557d09718df0 con 0x7fdd60007030 2023-10-05T19:28:34.501 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.494+0000 7fdd6dfea700 1 -- 172.21.15.167:0/4130379316 <== osd.2 v2:172.21.15.167:6816/30037 2 ==== osd_op_reply(13 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fdd64006070 con 0x7fdd60007030 2023-10-05T19:28:34.501 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.494+0000 7fdd6dfea700 1 -- 172.21.15.167:0/4130379316 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:14 5.2 5:40000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x557d09718df0 con 0x557d097164d0 2023-10-05T19:28:34.502 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.494+0000 7fdd6e7eb700 1 -- 172.21.15.167:0/4130379316 <== osd.3 v2:172.21.15.167:6808/30036 2 ==== osd_op_reply(14 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fdd6001c7f0 con 0x557d097164d0 2023-10-05T19:28:34.502 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.494+0000 7fdd6e7eb700 1 -- 172.21.15.167:0/4130379316 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:15 5.12 5:48000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x557d09718df0 con 0x557d096e18c0 2023-10-05T19:28:34.502 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.498+0000 7fdd6efec700 1 -- 172.21.15.167:0/4130379316 <== osd.1 v2:172.21.15.167:6800/30035 8 ==== osd_op_reply(15 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fdd6800a5a0 con 0x557d096e18c0 2023-10-05T19:28:34.502 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.498+0000 7fdd6efec700 1 -- 172.21.15.167:0/4130379316 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:16 5.a 5:50000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x557d09718df0 con 0x7fdd6400eb60 2023-10-05T19:28:34.503 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.498+0000 7fdd6efec700 1 -- 172.21.15.167:0/4130379316 <== osd.0 v2:172.21.15.118:6800/27690 4 ==== osd_op_reply(16 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fdd6800a5a0 con 0x7fdd6400eb60 2023-10-05T19:28:34.503 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.498+0000 7fdd6efec700 1 -- 172.21.15.167:0/4130379316 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:17 5.1a 5:58000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x557d09718df0 con 0x557d096e18c0 2023-10-05T19:28:34.504 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.498+0000 7fdd6efec700 1 -- 172.21.15.167:0/4130379316 <== osd.1 v2:172.21.15.167:6800/30035 9 ==== osd_op_reply(17 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fdd6800a5a0 con 0x557d096e18c0 2023-10-05T19:28:34.504 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.498+0000 7fdd6efec700 1 -- 172.21.15.167:0/4130379316 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:18 5.6 5:60000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x557d09718df0 con 0x7fdd60007030 2023-10-05T19:28:34.504 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.498+0000 7fdd6dfea700 1 -- 172.21.15.167:0/4130379316 <== osd.2 v2:172.21.15.167:6816/30037 3 ==== osd_op_reply(18 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fdd64006070 con 0x7fdd60007030 2023-10-05T19:28:34.505 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.498+0000 7fdd6dfea700 1 -- 172.21.15.167:0/4130379316 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:19 5.16 5:68000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x557d09718df0 con 0x557d097164d0 2023-10-05T19:28:34.505 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.498+0000 7fdd6e7eb700 1 -- 172.21.15.167:0/4130379316 <== osd.3 v2:172.21.15.167:6808/30036 3 ==== osd_op_reply(19 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fdd6001c7f0 con 0x557d097164d0 2023-10-05T19:28:34.505 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.498+0000 7fdd6e7eb700 1 -- 172.21.15.167:0/4130379316 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:20 5.e 5:70000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x557d09718df0 con 0x557d097164d0 2023-10-05T19:28:34.506 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.502+0000 7fdd6e7eb700 1 -- 172.21.15.167:0/4130379316 <== osd.3 v2:172.21.15.167:6808/30036 4 ==== osd_op_reply(20 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fdd6001c7f0 con 0x557d097164d0 2023-10-05T19:28:34.506 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.502+0000 7fdd6e7eb700 1 -- 172.21.15.167:0/4130379316 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:21 5.1e 5:78000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x557d09718df0 con 0x7fdd6400eb60 2023-10-05T19:28:34.507 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.502+0000 7fdd6efec700 1 -- 172.21.15.167:0/4130379316 <== osd.0 v2:172.21.15.118:6800/27690 5 ==== osd_op_reply(21 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fdd6800a5a0 con 0x7fdd6400eb60 2023-10-05T19:28:34.507 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.502+0000 7fdd6efec700 1 -- 172.21.15.167:0/4130379316 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:22 5.1 5:80000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x557d09718df0 con 0x557d097164d0 2023-10-05T19:28:34.507 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.502+0000 7fdd6e7eb700 1 -- 172.21.15.167:0/4130379316 <== osd.3 v2:172.21.15.167:6808/30036 5 ==== osd_op_reply(22 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fdd6001c7f0 con 0x557d097164d0 2023-10-05T19:28:34.507 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.502+0000 7fdd6e7eb700 1 -- 172.21.15.167:0/4130379316 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:23 5.11 5:88000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x557d09718df0 con 0x557d097164d0 2023-10-05T19:28:34.508 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.502+0000 7fdd6e7eb700 1 -- 172.21.15.167:0/4130379316 <== osd.3 v2:172.21.15.167:6808/30036 6 ==== osd_op_reply(23 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fdd6001c7f0 con 0x557d097164d0 2023-10-05T19:28:34.508 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.502+0000 7fdd6e7eb700 1 -- 172.21.15.167:0/4130379316 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:24 5.9 5:90000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x557d09718df0 con 0x557d096e18c0 2023-10-05T19:28:34.509 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.502+0000 7fdd6efec700 1 -- 172.21.15.167:0/4130379316 <== osd.1 v2:172.21.15.167:6800/30035 10 ==== osd_op_reply(24 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fdd6800a5a0 con 0x557d096e18c0 2023-10-05T19:28:34.509 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.502+0000 7fdd6efec700 1 -- 172.21.15.167:0/4130379316 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:25 5.19 5:98000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x557d09718df0 con 0x557d096e18c0 2023-10-05T19:28:34.510 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.506+0000 7fdd6efec700 1 -- 172.21.15.167:0/4130379316 <== osd.1 v2:172.21.15.167:6800/30035 11 ==== osd_op_reply(25 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fdd6800a5a0 con 0x557d096e18c0 2023-10-05T19:28:34.510 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.506+0000 7fdd6efec700 1 -- 172.21.15.167:0/4130379316 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:26 5.5 5:a0000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x557d09718df0 con 0x7fdd6400eb60 2023-10-05T19:28:34.511 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.506+0000 7fdd6efec700 1 -- 172.21.15.167:0/4130379316 <== osd.0 v2:172.21.15.118:6800/27690 6 ==== osd_op_reply(26 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fdd6800a5a0 con 0x7fdd6400eb60 2023-10-05T19:28:34.511 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.506+0000 7fdd6efec700 1 -- 172.21.15.167:0/4130379316 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:27 5.15 5:a8000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x557d09718df0 con 0x557d097164d0 2023-10-05T19:28:34.511 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.506+0000 7fdd6e7eb700 1 -- 172.21.15.167:0/4130379316 <== osd.3 v2:172.21.15.167:6808/30036 7 ==== osd_op_reply(27 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fdd6001c7f0 con 0x557d097164d0 2023-10-05T19:28:34.511 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.506+0000 7fdd6e7eb700 1 -- 172.21.15.167:0/4130379316 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:28 5.d 5:b0000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x557d09718df0 con 0x7fdd60007030 2023-10-05T19:28:34.512 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.506+0000 7fdd6dfea700 1 -- 172.21.15.167:0/4130379316 <== osd.2 v2:172.21.15.167:6816/30037 4 ==== osd_op_reply(28 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fdd64006070 con 0x7fdd60007030 2023-10-05T19:28:34.512 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.506+0000 7fdd6dfea700 1 -- 172.21.15.167:0/4130379316 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:29 5.1d 5:b8000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x557d09718df0 con 0x557d096e18c0 2023-10-05T19:28:34.513 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.506+0000 7fdd6efec700 1 -- 172.21.15.167:0/4130379316 <== osd.1 v2:172.21.15.167:6800/30035 12 ==== osd_op_reply(29 [pgnls start_epoch 30 out=107b] v30'2 uv2 ondisk = 1) v8 ==== 144+0+107 (crc 0 0 0) 0x7fdd6800a5a0 con 0x557d096e18c0 2023-10-05T19:28:34.513 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.506+0000 7fdd6efec700 1 -- 172.21.15.167:0/4130379316 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:30 5.3 5:c0000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x557d09718df0 con 0x7fdd6400eb60 2023-10-05T19:28:34.514 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.510+0000 7fdd6efec700 1 -- 172.21.15.167:0/4130379316 <== osd.0 v2:172.21.15.118:6800/27690 7 ==== osd_op_reply(30 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fdd6800a5a0 con 0x7fdd6400eb60 2023-10-05T19:28:34.514 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.510+0000 7fdd6efec700 1 -- 172.21.15.167:0/4130379316 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:31 5.13 5:c8000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x557d09718df0 con 0x557d097164d0 2023-10-05T19:28:34.515 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.510+0000 7fdd6e7eb700 1 -- 172.21.15.167:0/4130379316 <== osd.3 v2:172.21.15.167:6808/30036 8 ==== osd_op_reply(31 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fdd6001c7f0 con 0x557d097164d0 2023-10-05T19:28:34.515 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.510+0000 7fdd6e7eb700 1 -- 172.21.15.167:0/4130379316 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:32 5.b 5:d0000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x557d09718df0 con 0x7fdd6400eb60 2023-10-05T19:28:34.516 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.510+0000 7fdd6efec700 1 -- 172.21.15.167:0/4130379316 <== osd.0 v2:172.21.15.118:6800/27690 8 ==== osd_op_reply(32 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fdd6800a5a0 con 0x7fdd6400eb60 2023-10-05T19:28:34.516 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.510+0000 7fdd6efec700 1 -- 172.21.15.167:0/4130379316 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:33 5.1b 5:d8000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x557d09718df0 con 0x7fdd60007030 2023-10-05T19:28:34.517 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.510+0000 7fdd6dfea700 1 -- 172.21.15.167:0/4130379316 <== osd.2 v2:172.21.15.167:6816/30037 5 ==== osd_op_reply(33 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fdd64006070 con 0x7fdd60007030 2023-10-05T19:28:34.517 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.510+0000 7fdd6dfea700 1 -- 172.21.15.167:0/4130379316 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:34 5.7 5:e0000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x557d09718df0 con 0x557d097164d0 2023-10-05T19:28:34.517 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.510+0000 7fdd6e7eb700 1 -- 172.21.15.167:0/4130379316 <== osd.3 v2:172.21.15.167:6808/30036 9 ==== osd_op_reply(34 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fdd6001c7f0 con 0x557d097164d0 2023-10-05T19:28:34.518 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.510+0000 7fdd6e7eb700 1 -- 172.21.15.167:0/4130379316 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:35 5.17 5:e8000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x557d09718df0 con 0x7fdd6400eb60 2023-10-05T19:28:34.519 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.514+0000 7fdd6efec700 1 -- 172.21.15.167:0/4130379316 <== osd.0 v2:172.21.15.118:6800/27690 9 ==== osd_op_reply(35 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fdd6800a5a0 con 0x7fdd6400eb60 2023-10-05T19:28:34.519 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.514+0000 7fdd6efec700 1 -- 172.21.15.167:0/4130379316 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:36 5.f 5:f0000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x557d09718df0 con 0x557d096e18c0 2023-10-05T19:28:34.519 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.514+0000 7fdd6efec700 1 -- 172.21.15.167:0/4130379316 <== osd.1 v2:172.21.15.167:6800/30035 13 ==== osd_op_reply(36 [pgnls start_epoch 30 out=115b] v23'2 uv2 ondisk = 1) v8 ==== 144+0+115 (crc 0 0 0) 0x7fdd6800a5a0 con 0x557d096e18c0 2023-10-05T19:28:34.520 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.514+0000 7fdd6efec700 1 -- 172.21.15.167:0/4130379316 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:37 5.1f 5:f8000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x557d09718df0 con 0x7fdd60007030 2023-10-05T19:28:34.520 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.514+0000 7fdd6dfea700 1 -- 172.21.15.167:0/4130379316 <== osd.2 v2:172.21.15.167:6816/30037 6 ==== osd_op_reply(37 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fdd64006070 con 0x7fdd60007030 2023-10-05T19:28:34.520 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.514+0000 7fdd70cb6e40 20 RGWRados::pool_iterate: got zonegroup_info.4f1f719d-9648-417b-b10c-e5d939825419 2023-10-05T19:28:34.521 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.514+0000 7fdd70cb6e40 20 RGWRados::pool_iterate: got zone_info.9a9df965-6612-45d4-a45f-6e3c889bacc2 2023-10-05T19:28:34.521 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.514+0000 7fdd70cb6e40 20 RGWRados::pool_iterate: got zone_names.default 2023-10-05T19:28:34.521 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.514+0000 7fdd70cb6e40 20 RGWRados::pool_iterate: got zonegroups_names.default 2023-10-05T19:28:34.521 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.514+0000 7fdd70cb6e40 20 rados->read ofs=0 len=0 2023-10-05T19:28:34.521 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.514+0000 7fdd70cb6e40 1 -- 172.21.15.167:0/4130379316 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:38 5.12 5:49953fa1:::default.realm:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x557d09715280 con 0x557d096e18c0 2023-10-05T19:28:34.522 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.514+0000 7fdd6efec700 1 -- 172.21.15.167:0/4130379316 <== osd.1 v2:172.21.15.167:6800/30035 14 ==== osd_op_reply(38 default.realm [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 ==== 157+0+0 (crc 0 0 0) 0x7fdd6800a5a0 con 0x557d096e18c0 2023-10-05T19:28:34.522 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.514+0000 7fdd70cb6e40 20 rados_obj.operate() r=-2 bl.length=0 2023-10-05T19:28:34.522 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.514+0000 7fdd70cb6e40 20 rados->read ofs=0 len=0 2023-10-05T19:28:34.522 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.514+0000 7fdd70cb6e40 1 -- 172.21.15.167:0/4130379316 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:39 5.f 5:f4c53578:::zonegroups_names.default:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x557d09715b70 con 0x557d096e18c0 2023-10-05T19:28:34.522 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.514+0000 7fdd6efec700 1 -- 172.21.15.167:0/4130379316 <== osd.1 v2:172.21.15.167:6800/30035 15 ==== osd_op_reply(39 zonegroups_names.default [read 0~46 out=46b] v0'0 uv2 ondisk = 0) v8 ==== 168+0+46 (crc 0 0 0) 0x7fdd6800a5a0 con 0x557d096e18c0 2023-10-05T19:28:34.522 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.514+0000 7fdd70cb6e40 20 rados_obj.operate() r=0 bl.length=46 2023-10-05T19:28:34.522 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.518+0000 7fdd70cb6e40 20 rados->read ofs=0 len=0 2023-10-05T19:28:34.522 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.518+0000 7fdd70cb6e40 1 -- 172.21.15.167:0/4130379316 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:40 5.c 5:30d299a5:::zonegroup_info.4f1f719d-9648-417b-b10c-e5d939825419:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x557d09715b70 con 0x7fdd6400eb60 2023-10-05T19:28:34.523 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.518+0000 7fdd6efec700 1 -- 172.21.15.167:0/4130379316 <== osd.0 v2:172.21.15.118:6800/27690 10 ==== osd_op_reply(40 zonegroup_info.4f1f719d-9648-417b-b10c-e5d939825419 [read 0~362 out=362b] v0'0 uv1 ondisk = 0) v8 ==== 195+0+362 (crc 0 0 0) 0x7fdd6800a5a0 con 0x7fdd6400eb60 2023-10-05T19:28:34.524 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.518+0000 7fdd70cb6e40 20 rados_obj.operate() r=0 bl.length=362 2023-10-05T19:28:34.524 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.518+0000 7fdd70cb6e40 20 rados->read ofs=0 len=0 2023-10-05T19:28:34.524 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.518+0000 7fdd70cb6e40 1 -- 172.21.15.167:0/4130379316 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:41 5.13 5:cbbf6c60:::converted:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x557d09715b70 con 0x557d097164d0 2023-10-05T19:28:34.524 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.518+0000 7fdd6e7eb700 1 -- 172.21.15.167:0/4130379316 <== osd.3 v2:172.21.15.167:6808/30036 10 ==== osd_op_reply(41 converted [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 ==== 153+0+0 (crc 0 0 0) 0x7fdd6001c7f0 con 0x557d097164d0 2023-10-05T19:28:34.524 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.518+0000 7fdd70cb6e40 20 rados_obj.operate() r=-2 bl.length=0 2023-10-05T19:28:34.524 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.518+0000 7fdd70cb6e40 20 rados->read ofs=0 len=0 2023-10-05T19:28:34.524 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.518+0000 7fdd70cb6e40 1 -- 172.21.15.167:0/4130379316 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:42 5.12 5:49953fa1:::default.realm:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x557d09719260 con 0x557d096e18c0 2023-10-05T19:28:34.525 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.518+0000 7fdd6efec700 1 -- 172.21.15.167:0/4130379316 <== osd.1 v2:172.21.15.167:6800/30035 16 ==== osd_op_reply(42 default.realm [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 ==== 157+0+0 (crc 0 0 0) 0x7fdd6800a5a0 con 0x557d096e18c0 2023-10-05T19:28:34.525 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.518+0000 7fdd70cb6e40 20 rados_obj.operate() r=-2 bl.length=0 2023-10-05T19:28:34.525 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.518+0000 7fdd70cb6e40 20 rados->read ofs=0 len=0 2023-10-05T19:28:34.525 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.518+0000 7fdd70cb6e40 1 -- 172.21.15.167:0/4130379316 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:43 5.f 5:f4c53578:::zonegroups_names.default:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x557d09719690 con 0x557d096e18c0 2023-10-05T19:28:34.525 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.518+0000 7fdd6efec700 1 -- 172.21.15.167:0/4130379316 <== osd.1 v2:172.21.15.167:6800/30035 17 ==== osd_op_reply(43 zonegroups_names.default [read 0~46 out=46b] v0'0 uv2 ondisk = 0) v8 ==== 168+0+46 (crc 0 0 0) 0x7fdd6800a5a0 con 0x557d096e18c0 2023-10-05T19:28:34.525 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.518+0000 7fdd70cb6e40 20 rados_obj.operate() r=0 bl.length=46 2023-10-05T19:28:34.525 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.518+0000 7fdd70cb6e40 1 -- 172.21.15.167:0/4130379316 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:44 5.0 5:00000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x557d0971a6a0 con 0x557d097164d0 2023-10-05T19:28:34.527 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.522+0000 7fdd6e7eb700 1 -- 172.21.15.167:0/4130379316 <== osd.3 v2:172.21.15.167:6808/30036 11 ==== osd_op_reply(44 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fdd6001c7f0 con 0x557d097164d0 2023-10-05T19:28:34.527 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.522+0000 7fdd6e7eb700 1 -- 172.21.15.167:0/4130379316 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:45 5.10 5:08000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x557d0971a6a0 con 0x7fdd60007030 2023-10-05T19:28:34.528 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.522+0000 7fdd6dfea700 1 -- 172.21.15.167:0/4130379316 <== osd.2 v2:172.21.15.167:6816/30037 7 ==== osd_op_reply(45 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fdd64006070 con 0x7fdd60007030 2023-10-05T19:28:34.528 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.522+0000 7fdd6dfea700 1 -- 172.21.15.167:0/4130379316 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:46 5.8 5:10000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x557d0971a6a0 con 0x7fdd6400eb60 2023-10-05T19:28:34.529 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.522+0000 7fdd6efec700 1 -- 172.21.15.167:0/4130379316 <== osd.0 v2:172.21.15.118:6800/27690 11 ==== osd_op_reply(46 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fdd6800a5a0 con 0x7fdd6400eb60 2023-10-05T19:28:34.529 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.522+0000 7fdd6efec700 1 -- 172.21.15.167:0/4130379316 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:47 5.18 5:18000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x557d09715280 con 0x557d096e18c0 2023-10-05T19:28:34.529 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.522+0000 7fdd6efec700 1 -- 172.21.15.167:0/4130379316 <== osd.1 v2:172.21.15.167:6800/30035 18 ==== osd_op_reply(47 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fdd6800a5a0 con 0x557d096e18c0 2023-10-05T19:28:34.529 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.522+0000 7fdd6efec700 1 -- 172.21.15.167:0/4130379316 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:48 5.4 5:20000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x557d09715280 con 0x557d096e18c0 2023-10-05T19:28:34.530 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.526+0000 7fdd6efec700 1 -- 172.21.15.167:0/4130379316 <== osd.1 v2:172.21.15.167:6800/30035 19 ==== osd_op_reply(48 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fdd6800a5a0 con 0x557d096e18c0 2023-10-05T19:28:34.530 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.526+0000 7fdd6efec700 1 -- 172.21.15.167:0/4130379316 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:49 5.14 5:28000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x557d09715280 con 0x7fdd6400eb60 2023-10-05T19:28:34.531 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.526+0000 7fdd6efec700 1 -- 172.21.15.167:0/4130379316 <== osd.0 v2:172.21.15.118:6800/27690 12 ==== osd_op_reply(49 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fdd6800a5a0 con 0x7fdd6400eb60 2023-10-05T19:28:34.531 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.526+0000 7fdd6efec700 1 -- 172.21.15.167:0/4130379316 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:50 5.c 5:30000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x557d09715280 con 0x7fdd6400eb60 2023-10-05T19:28:34.532 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.526+0000 7fdd6efec700 1 -- 172.21.15.167:0/4130379316 <== osd.0 v2:172.21.15.118:6800/27690 13 ==== osd_op_reply(50 [pgnls start_epoch 30 out=112b] v23'1 uv1 ondisk = 1) v8 ==== 144+0+112 (crc 0 0 0) 0x7fdd6800a5a0 con 0x7fdd6400eb60 2023-10-05T19:28:34.532 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.526+0000 7fdd6efec700 1 -- 172.21.15.167:0/4130379316 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:51 5.1c 5:38000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x557d09715280 con 0x7fdd60007030 2023-10-05T19:28:34.533 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.526+0000 7fdd6dfea700 1 -- 172.21.15.167:0/4130379316 <== osd.2 v2:172.21.15.167:6816/30037 8 ==== osd_op_reply(51 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fdd64006070 con 0x7fdd60007030 2023-10-05T19:28:34.533 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.526+0000 7fdd6dfea700 1 -- 172.21.15.167:0/4130379316 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:52 5.2 5:40000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x557d09715280 con 0x557d097164d0 2023-10-05T19:28:34.533 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.526+0000 7fdd6e7eb700 1 -- 172.21.15.167:0/4130379316 <== osd.3 v2:172.21.15.167:6808/30036 12 ==== osd_op_reply(52 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fdd6001c7f0 con 0x557d097164d0 2023-10-05T19:28:34.533 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.526+0000 7fdd6e7eb700 1 -- 172.21.15.167:0/4130379316 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:53 5.12 5:48000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x557d09715280 con 0x557d096e18c0 2023-10-05T19:28:34.534 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.530+0000 7fdd6efec700 1 -- 172.21.15.167:0/4130379316 <== osd.1 v2:172.21.15.167:6800/30035 20 ==== osd_op_reply(53 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fdd6800a5a0 con 0x557d096e18c0 2023-10-05T19:28:34.534 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.530+0000 7fdd6efec700 1 -- 172.21.15.167:0/4130379316 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:54 5.a 5:50000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x557d09715280 con 0x7fdd6400eb60 2023-10-05T19:28:34.535 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.530+0000 7fdd6efec700 1 -- 172.21.15.167:0/4130379316 <== osd.0 v2:172.21.15.118:6800/27690 14 ==== osd_op_reply(54 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fdd6800a5a0 con 0x7fdd6400eb60 2023-10-05T19:28:34.535 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.530+0000 7fdd6efec700 1 -- 172.21.15.167:0/4130379316 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:55 5.1a 5:58000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x557d09715280 con 0x557d096e18c0 2023-10-05T19:28:34.536 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.530+0000 7fdd6efec700 1 -- 172.21.15.167:0/4130379316 <== osd.1 v2:172.21.15.167:6800/30035 21 ==== osd_op_reply(55 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fdd6800a5a0 con 0x557d096e18c0 2023-10-05T19:28:34.536 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.530+0000 7fdd6efec700 1 -- 172.21.15.167:0/4130379316 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:56 5.6 5:60000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x557d09715280 con 0x7fdd60007030 2023-10-05T19:28:34.537 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.530+0000 7fdd6dfea700 1 -- 172.21.15.167:0/4130379316 <== osd.2 v2:172.21.15.167:6816/30037 9 ==== osd_op_reply(56 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fdd64006070 con 0x7fdd60007030 2023-10-05T19:28:34.537 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.530+0000 7fdd6dfea700 1 -- 172.21.15.167:0/4130379316 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:57 5.16 5:68000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x557d09715280 con 0x557d097164d0 2023-10-05T19:28:34.537 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.530+0000 7fdd6e7eb700 1 -- 172.21.15.167:0/4130379316 <== osd.3 v2:172.21.15.167:6808/30036 13 ==== osd_op_reply(57 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fdd6001c7f0 con 0x557d097164d0 2023-10-05T19:28:34.537 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.530+0000 7fdd6e7eb700 1 -- 172.21.15.167:0/4130379316 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:58 5.e 5:70000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x557d09715280 con 0x557d097164d0 2023-10-05T19:28:34.538 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.534+0000 7fdd6e7eb700 1 -- 172.21.15.167:0/4130379316 <== osd.3 v2:172.21.15.167:6808/30036 14 ==== osd_op_reply(58 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fdd6001c7f0 con 0x557d097164d0 2023-10-05T19:28:34.538 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.534+0000 7fdd6e7eb700 1 -- 172.21.15.167:0/4130379316 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:59 5.1e 5:78000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x557d09715280 con 0x7fdd6400eb60 2023-10-05T19:28:34.539 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.534+0000 7fdd6efec700 1 -- 172.21.15.167:0/4130379316 <== osd.0 v2:172.21.15.118:6800/27690 15 ==== osd_op_reply(59 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fdd6803f040 con 0x7fdd6400eb60 2023-10-05T19:28:34.539 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.534+0000 7fdd6efec700 1 -- 172.21.15.167:0/4130379316 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:60 5.1 5:80000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x557d09715280 con 0x557d097164d0 2023-10-05T19:28:34.540 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.534+0000 7fdd6e7eb700 1 -- 172.21.15.167:0/4130379316 <== osd.3 v2:172.21.15.167:6808/30036 15 ==== osd_op_reply(60 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fdd6001c7f0 con 0x557d097164d0 2023-10-05T19:28:34.540 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.534+0000 7fdd6e7eb700 1 -- 172.21.15.167:0/4130379316 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:61 5.11 5:88000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x557d09715280 con 0x557d097164d0 2023-10-05T19:28:34.541 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.534+0000 7fdd6e7eb700 1 -- 172.21.15.167:0/4130379316 <== osd.3 v2:172.21.15.167:6808/30036 16 ==== osd_op_reply(61 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fdd6001c7f0 con 0x557d097164d0 2023-10-05T19:28:34.541 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.534+0000 7fdd6e7eb700 1 -- 172.21.15.167:0/4130379316 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:62 5.9 5:90000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x557d09715280 con 0x557d096e18c0 2023-10-05T19:28:34.542 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.534+0000 7fdd6efec700 1 -- 172.21.15.167:0/4130379316 <== osd.1 v2:172.21.15.167:6800/30035 22 ==== osd_op_reply(62 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fdd6803f040 con 0x557d096e18c0 2023-10-05T19:28:34.542 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.534+0000 7fdd6efec700 1 -- 172.21.15.167:0/4130379316 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:63 5.19 5:98000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x557d09715280 con 0x557d096e18c0 2023-10-05T19:28:34.542 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.538+0000 7fdd6efec700 1 -- 172.21.15.167:0/4130379316 <== osd.1 v2:172.21.15.167:6800/30035 23 ==== osd_op_reply(63 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fdd6803f040 con 0x557d096e18c0 2023-10-05T19:28:34.542 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.538+0000 7fdd6efec700 1 -- 172.21.15.167:0/4130379316 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:64 5.5 5:a0000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x557d09715280 con 0x7fdd6400eb60 2023-10-05T19:28:34.544 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.538+0000 7fdd6efec700 1 -- 172.21.15.167:0/4130379316 <== osd.0 v2:172.21.15.118:6800/27690 16 ==== osd_op_reply(64 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fdd6803f040 con 0x7fdd6400eb60 2023-10-05T19:28:34.544 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.538+0000 7fdd6efec700 1 -- 172.21.15.167:0/4130379316 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:65 5.15 5:a8000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x557d09715280 con 0x557d097164d0 2023-10-05T19:28:34.544 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.538+0000 7fdd6e7eb700 1 -- 172.21.15.167:0/4130379316 <== osd.3 v2:172.21.15.167:6808/30036 17 ==== osd_op_reply(65 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fdd6001c7f0 con 0x557d097164d0 2023-10-05T19:28:34.544 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.538+0000 7fdd6e7eb700 1 -- 172.21.15.167:0/4130379316 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:66 5.d 5:b0000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x557d09715280 con 0x7fdd60007030 2023-10-05T19:28:34.545 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.538+0000 7fdd6dfea700 1 -- 172.21.15.167:0/4130379316 <== osd.2 v2:172.21.15.167:6816/30037 10 ==== osd_op_reply(66 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fdd64006070 con 0x7fdd60007030 2023-10-05T19:28:34.545 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.538+0000 7fdd6dfea700 1 -- 172.21.15.167:0/4130379316 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:67 5.1d 5:b8000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x557d09715280 con 0x557d096e18c0 2023-10-05T19:28:34.546 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.542+0000 7fdd6efec700 1 -- 172.21.15.167:0/4130379316 <== osd.1 v2:172.21.15.167:6800/30035 24 ==== osd_op_reply(67 [pgnls start_epoch 30 out=107b] v30'2 uv2 ondisk = 1) v8 ==== 144+0+107 (crc 0 0 0) 0x7fdd6803f040 con 0x557d096e18c0 2023-10-05T19:28:34.546 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.542+0000 7fdd6efec700 1 -- 172.21.15.167:0/4130379316 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:68 5.3 5:c0000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x557d09715280 con 0x7fdd6400eb60 2023-10-05T19:28:34.547 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.542+0000 7fdd6efec700 1 -- 172.21.15.167:0/4130379316 <== osd.0 v2:172.21.15.118:6800/27690 17 ==== osd_op_reply(68 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fdd6803f040 con 0x7fdd6400eb60 2023-10-05T19:28:34.547 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.542+0000 7fdd6efec700 1 -- 172.21.15.167:0/4130379316 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:69 5.13 5:c8000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x557d09715280 con 0x557d097164d0 2023-10-05T19:28:34.547 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.542+0000 7fdd6e7eb700 1 -- 172.21.15.167:0/4130379316 <== osd.3 v2:172.21.15.167:6808/30036 18 ==== osd_op_reply(69 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fdd6001c7f0 con 0x557d097164d0 2023-10-05T19:28:34.548 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.542+0000 7fdd6e7eb700 1 -- 172.21.15.167:0/4130379316 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:70 5.b 5:d0000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x557d09715280 con 0x7fdd6400eb60 2023-10-05T19:28:34.548 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.542+0000 7fdd6efec700 1 -- 172.21.15.167:0/4130379316 <== osd.0 v2:172.21.15.118:6800/27690 18 ==== osd_op_reply(70 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fdd6803f040 con 0x7fdd6400eb60 2023-10-05T19:28:34.549 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.542+0000 7fdd6efec700 1 -- 172.21.15.167:0/4130379316 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:71 5.1b 5:d8000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x557d09715280 con 0x7fdd60007030 2023-10-05T19:28:34.549 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.542+0000 7fdd6dfea700 1 -- 172.21.15.167:0/4130379316 <== osd.2 v2:172.21.15.167:6816/30037 11 ==== osd_op_reply(71 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fdd64006070 con 0x7fdd60007030 2023-10-05T19:28:34.550 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.542+0000 7fdd6dfea700 1 -- 172.21.15.167:0/4130379316 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:72 5.7 5:e0000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x557d09715280 con 0x557d097164d0 2023-10-05T19:28:34.551 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.546+0000 7fdd6e7eb700 1 -- 172.21.15.167:0/4130379316 <== osd.3 v2:172.21.15.167:6808/30036 19 ==== osd_op_reply(72 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fdd6001c7f0 con 0x557d097164d0 2023-10-05T19:28:34.551 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.546+0000 7fdd6e7eb700 1 -- 172.21.15.167:0/4130379316 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:73 5.17 5:e8000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x557d09715280 con 0x7fdd6400eb60 2023-10-05T19:28:34.552 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.546+0000 7fdd6efec700 1 -- 172.21.15.167:0/4130379316 <== osd.0 v2:172.21.15.118:6800/27690 19 ==== osd_op_reply(73 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fdd6803f040 con 0x7fdd6400eb60 2023-10-05T19:28:34.552 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.546+0000 7fdd6efec700 1 -- 172.21.15.167:0/4130379316 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:74 5.f 5:f0000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x557d09715280 con 0x557d096e18c0 2023-10-05T19:28:34.553 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.546+0000 7fdd6efec700 1 -- 172.21.15.167:0/4130379316 <== osd.1 v2:172.21.15.167:6800/30035 25 ==== osd_op_reply(74 [pgnls start_epoch 30 out=115b] v23'2 uv2 ondisk = 1) v8 ==== 144+0+115 (crc 0 0 0) 0x7fdd6803f040 con 0x557d096e18c0 2023-10-05T19:28:34.553 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.546+0000 7fdd6efec700 1 -- 172.21.15.167:0/4130379316 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:75 5.1f 5:f8000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x557d09715280 con 0x7fdd60007030 2023-10-05T19:28:34.554 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.546+0000 7fdd6dfea700 1 -- 172.21.15.167:0/4130379316 <== osd.2 v2:172.21.15.167:6816/30037 12 ==== osd_op_reply(75 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fdd64006070 con 0x7fdd60007030 2023-10-05T19:28:34.554 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.546+0000 7fdd70cb6e40 20 RGWRados::pool_iterate: got zonegroup_info.4f1f719d-9648-417b-b10c-e5d939825419 2023-10-05T19:28:34.554 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.546+0000 7fdd70cb6e40 20 RGWRados::pool_iterate: got zone_info.9a9df965-6612-45d4-a45f-6e3c889bacc2 2023-10-05T19:28:34.554 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.546+0000 7fdd70cb6e40 20 RGWRados::pool_iterate: got zone_names.default 2023-10-05T19:28:34.554 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.546+0000 7fdd70cb6e40 20 RGWRados::pool_iterate: got zonegroups_names.default 2023-10-05T19:28:34.554 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.546+0000 7fdd70cb6e40 20 rados->read ofs=0 len=0 2023-10-05T19:28:34.554 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.546+0000 7fdd70cb6e40 1 -- 172.21.15.167:0/4130379316 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:76 5.f 5:f43b6ece:::zone_names.default:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x557d09719690 con 0x557d096e18c0 2023-10-05T19:28:34.555 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.550+0000 7fdd6efec700 1 -- 172.21.15.167:0/4130379316 <== osd.1 v2:172.21.15.167:6800/30035 26 ==== osd_op_reply(76 zone_names.default [read 0~46 out=46b] v0'0 uv1 ondisk = 0) v8 ==== 162+0+46 (crc 0 0 0) 0x7fdd6803f040 con 0x557d096e18c0 2023-10-05T19:28:34.555 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.550+0000 7fdd70cb6e40 20 rados_obj.operate() r=0 bl.length=46 2023-10-05T19:28:34.555 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.550+0000 7fdd70cb6e40 20 rados->read ofs=0 len=0 2023-10-05T19:28:34.555 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.550+0000 7fdd70cb6e40 1 -- 172.21.15.167:0/4130379316 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:77 5.1d 5:b8374eb4:::zone_info.9a9df965-6612-45d4-a45f-6e3c889bacc2:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x557d09719a40 con 0x557d096e18c0 2023-10-05T19:28:34.556 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.550+0000 7fdd6efec700 1 -- 172.21.15.167:0/4130379316 <== osd.1 v2:172.21.15.167:6800/30035 27 ==== osd_op_reply(77 zone_info.9a9df965-6612-45d4-a45f-6e3c889bacc2 [read 0~874 out=874b] v0'0 uv2 ondisk = 0) v8 ==== 190+0+874 (crc 0 0 0) 0x7fdd6803f040 con 0x557d096e18c0 2023-10-05T19:28:34.556 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.550+0000 7fdd70cb6e40 20 rados_obj.operate() r=0 bl.length=874 2023-10-05T19:28:34.556 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.550+0000 7fdd70cb6e40 20 rados->read ofs=0 len=0 2023-10-05T19:28:34.556 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.550+0000 7fdd70cb6e40 1 -- 172.21.15.167:0/4130379316 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:78 5.f 5:f4c53578:::zonegroups_names.default:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x557d097156e0 con 0x557d096e18c0 2023-10-05T19:28:34.556 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.550+0000 7fdd6efec700 1 -- 172.21.15.167:0/4130379316 <== osd.1 v2:172.21.15.167:6800/30035 28 ==== osd_op_reply(78 zonegroups_names.default [read 0~46 out=46b] v0'0 uv2 ondisk = 0) v8 ==== 168+0+46 (crc 0 0 0) 0x7fdd6803f040 con 0x557d096e18c0 2023-10-05T19:28:34.556 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.550+0000 7fdd70cb6e40 20 rados_obj.operate() r=0 bl.length=46 2023-10-05T19:28:34.557 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.550+0000 7fdd70cb6e40 20 rados->read ofs=0 len=0 2023-10-05T19:28:34.557 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.550+0000 7fdd70cb6e40 1 -- 172.21.15.167:0/4130379316 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:79 5.c 5:30d299a5:::zonegroup_info.4f1f719d-9648-417b-b10c-e5d939825419:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x557d097156c0 con 0x7fdd6400eb60 2023-10-05T19:28:34.557 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.550+0000 7fdd6efec700 1 -- 172.21.15.167:0/4130379316 <== osd.0 v2:172.21.15.118:6800/27690 20 ==== osd_op_reply(79 zonegroup_info.4f1f719d-9648-417b-b10c-e5d939825419 [read 0~362 out=362b] v0'0 uv1 ondisk = 0) v8 ==== 195+0+362 (crc 0 0 0) 0x7fdd6803f040 con 0x7fdd6400eb60 2023-10-05T19:28:34.557 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.550+0000 7fdd70cb6e40 20 rados_obj.operate() r=0 bl.length=362 2023-10-05T19:28:34.557 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.550+0000 7fdd70cb6e40 20 rados->read ofs=0 len=0 2023-10-05T19:28:34.557 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.550+0000 7fdd70cb6e40 1 -- 172.21.15.167:0/4130379316 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:80 5.3 5:c0ad8cbd:::region_map:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x557d09719f80 con 0x7fdd6400eb60 2023-10-05T19:28:34.558 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.550+0000 7fdd6efec700 1 -- 172.21.15.167:0/4130379316 <== osd.0 v2:172.21.15.118:6800/27690 21 ==== osd_op_reply(80 region_map [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 ==== 154+0+0 (crc 0 0 0) 0x7fdd68052040 con 0x7fdd6400eb60 2023-10-05T19:28:34.558 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.550+0000 7fdd70cb6e40 20 rados_obj.operate() r=-2 bl.length=0 2023-10-05T19:28:34.559 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.550+0000 7fdd70cb6e40 20 zone default found 2023-10-05T19:28:34.559 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.550+0000 7fdd70cb6e40 4 Realm: () 2023-10-05T19:28:34.559 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.550+0000 7fdd70cb6e40 4 ZoneGroup: default (4f1f719d-9648-417b-b10c-e5d939825419) 2023-10-05T19:28:34.559 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.550+0000 7fdd70cb6e40 4 Zone: default (9a9df965-6612-45d4-a45f-6e3c889bacc2) 2023-10-05T19:28:34.559 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.550+0000 7fdd70cb6e40 10 cannot find current period zonegroup using local zonegroup configuration 2023-10-05T19:28:34.559 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.550+0000 7fdd70cb6e40 20 zonegroup default 2023-10-05T19:28:34.559 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.550+0000 7fdd70cb6e40 20 rados->read ofs=0 len=0 2023-10-05T19:28:34.559 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.550+0000 7fdd70cb6e40 1 -- 172.21.15.167:0/4130379316 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:81 5.3 5:c52100b6:::period_config.default:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x557d09719f80 con 0x7fdd6400eb60 2023-10-05T19:28:34.559 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.554+0000 7fdd6efec700 1 -- 172.21.15.167:0/4130379316 <== osd.0 v2:172.21.15.118:6800/27690 22 ==== osd_op_reply(81 period_config.default [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 ==== 165+0+0 (crc 0 0 0) 0x7fdd68052040 con 0x7fdd6400eb60 2023-10-05T19:28:34.559 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.554+0000 7fdd70cb6e40 20 rados_obj.operate() r=-2 bl.length=0 2023-10-05T19:28:34.559 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.554+0000 7fdd70cb6e40 20 rados->read ofs=0 len=0 2023-10-05T19:28:34.560 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.554+0000 7fdd70cb6e40 1 -- 172.21.15.167:0/4130379316 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:82 6.2 6:4347d321:::bucket.sync-source-hints.:head [call version.read in=11b,read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x557d0971b8b0 con 0x557d097164d0 2023-10-05T19:28:34.561 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.554+0000 7fdd6e7eb700 1 -- 172.21.15.167:0/4130379316 <== osd.3 v2:172.21.15.167:6808/30036 20 ==== osd_op_reply(82 bucket.sync-source-hints. [call,read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 ==== 211+0+0 (crc 0 0 0) 0x7fdd6005d040 con 0x557d097164d0 2023-10-05T19:28:34.561 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.554+0000 7fdd70cb6e40 20 rados_obj.operate() r=-2 bl.length=0 2023-10-05T19:28:34.561 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.554+0000 7fdd70cb6e40 20 rados->read ofs=0 len=0 2023-10-05T19:28:34.561 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.554+0000 7fdd70cb6e40 1 -- 172.21.15.167:0/4130379316 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:83 6.b 6:d467b91b:::bucket.sync-target-hints.:head [call version.read in=11b,read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x557d0971bc60 con 0x557d097164d0 2023-10-05T19:28:34.561 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.554+0000 7fdd6e7eb700 1 -- 172.21.15.167:0/4130379316 <== osd.3 v2:172.21.15.167:6808/30036 21 ==== osd_op_reply(83 bucket.sync-target-hints. [call,read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 ==== 211+0+0 (crc 0 0 0) 0x7fdd6005d040 con 0x557d097164d0 2023-10-05T19:28:34.561 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.554+0000 7fdd70cb6e40 20 rados_obj.operate() r=-2 bl.length=0 2023-10-05T19:28:34.561 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.554+0000 7fdd70cb6e40 20 started sync module instance, tier type = 2023-10-05T19:28:34.561 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.554+0000 7fdd70cb6e40 20 started zone id=9a9df965-6612-45d4-a45f-6e3c889bacc2 (name=default) with tier type = 2023-10-05T19:28:34.561 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.554+0000 7fdd70cb6e40 1 -- 172.21.15.167:0/4130379316 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:84 7.1f 7:f95f44c2:::notify.0:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x557d0971af70 con 0x7fdd60007030 2023-10-05T19:28:34.563 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.558+0000 7fdd6dfea700 1 -- 172.21.15.167:0/4130379316 <== osd.2 v2:172.21.15.167:6816/30037 13 ==== osd_op_reply(84 notify.0 [create] v30'132 uv132 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7fdd64006070 con 0x7fdd60007030 2023-10-05T19:28:34.563 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.558+0000 7fdd70cb6e40 1 -- 172.21.15.167:0/4130379316 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:85 7.1f 7:f95f44c2:::notify.0:head [watch watch cookie 93995517717120] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x557d0971c010 con 0x7fdd60007030 2023-10-05T19:28:34.564 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.558+0000 7fdd70cb6e40 1 -- 172.21.15.167:0/4130379316 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:86 7.0 7:05bf5b68:::notify.1:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x557d0971c8b0 con 0x557d096e18c0 2023-10-05T19:28:34.567 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.562+0000 7fdd6dfea700 1 -- 172.21.15.167:0/4130379316 <== osd.2 v2:172.21.15.167:6816/30037 14 ==== osd_op_reply(85 notify.0 [watch watch cookie 93995517717120] v30'133 uv132 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7fdd64006070 con 0x7fdd60007030 2023-10-05T19:28:34.567 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.562+0000 7fdd6efec700 1 -- 172.21.15.167:0/4130379316 <== osd.1 v2:172.21.15.167:6800/30035 29 ==== osd_op_reply(86 notify.1 [create] v30'132 uv132 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7fdd68052040 con 0x557d096e18c0 2023-10-05T19:28:34.567 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.562+0000 7fdd70cb6e40 1 -- 172.21.15.167:0/4130379316 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:87 7.0 7:05bf5b68:::notify.1:head [watch watch cookie 93995517723584] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x557d0971cd00 con 0x557d096e18c0 2023-10-05T19:28:34.567 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.562+0000 7fdd70cb6e40 1 -- 172.21.15.167:0/4130379316 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:88 7.15 7:a93a5511:::notify.2:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x557d0971d5a0 con 0x557d097164d0 2023-10-05T19:28:34.570 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.566+0000 7fdd6efec700 1 -- 172.21.15.167:0/4130379316 <== osd.1 v2:172.21.15.167:6800/30035 30 ==== osd_op_reply(87 notify.1 [watch watch cookie 93995517723584] v30'133 uv132 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7fdd68052040 con 0x557d096e18c0 2023-10-05T19:28:34.570 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.566+0000 7fdd6e7eb700 1 -- 172.21.15.167:0/4130379316 <== osd.3 v2:172.21.15.167:6808/30036 22 ==== osd_op_reply(88 notify.2 [create] v30'132 uv132 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7fdd6005d040 con 0x557d097164d0 2023-10-05T19:28:34.571 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.566+0000 7fdd70cb6e40 1 -- 172.21.15.167:0/4130379316 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:89 7.15 7:a93a5511:::notify.2:head [watch watch cookie 93995517724688] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x557d0971d150 con 0x557d097164d0 2023-10-05T19:28:34.571 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.566+0000 7fdd70cb6e40 1 -- 172.21.15.167:0/4130379316 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:90 7.e 7:7759931f:::notify.3:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x557d0971e050 con 0x7fdd60007030 2023-10-05T19:28:34.573 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.566+0000 7fdd6dfea700 1 -- 172.21.15.167:0/4130379316 <== osd.2 v2:172.21.15.167:6816/30037 15 ==== osd_op_reply(90 notify.3 [create] v30'132 uv132 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7fdd64006070 con 0x7fdd60007030 2023-10-05T19:28:34.573 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.566+0000 7fdd70cb6e40 1 -- 172.21.15.167:0/4130379316 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:91 7.e 7:7759931f:::notify.3:head [watch watch cookie 93995517731920] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x557d0971ed90 con 0x7fdd60007030 2023-10-05T19:28:34.573 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.566+0000 7fdd70cb6e40 1 -- 172.21.15.167:0/4130379316 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:92 7.d 7:b4812045:::notify.4:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x557d0971f7a0 con 0x557d096e18c0 2023-10-05T19:28:34.574 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.566+0000 7fdd6e7eb700 1 -- 172.21.15.167:0/4130379316 <== osd.3 v2:172.21.15.167:6808/30036 23 ==== osd_op_reply(89 notify.2 [watch watch cookie 93995517724688] v30'133 uv132 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7fdd6005d040 con 0x557d097164d0 2023-10-05T19:28:34.576 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.570+0000 7fdd6dfea700 1 -- 172.21.15.167:0/4130379316 <== osd.2 v2:172.21.15.167:6816/30037 16 ==== osd_op_reply(91 notify.3 [watch watch cookie 93995517731920] v30'133 uv132 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7fdd64006070 con 0x7fdd60007030 2023-10-05T19:28:34.576 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.570+0000 7fdd6efec700 1 -- 172.21.15.167:0/4130379316 <== osd.1 v2:172.21.15.167:6800/30035 31 ==== osd_op_reply(92 notify.4 [create] v30'132 uv132 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7fdd68052040 con 0x557d096e18c0 2023-10-05T19:28:34.576 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.570+0000 7fdd70cb6e40 1 -- 172.21.15.167:0/4130379316 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:93 7.d 7:b4812045:::notify.4:head [watch watch cookie 93995517725792] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x557d0971f7a0 con 0x557d096e18c0 2023-10-05T19:28:34.576 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.570+0000 7fdd70cb6e40 1 -- 172.21.15.167:0/4130379316 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:94 7.3 7:c609908c:::notify.5:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x557d09720040 con 0x7fdd6400eb60 2023-10-05T19:28:34.578 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.574+0000 7fdd6efec700 1 -- 172.21.15.167:0/4130379316 <== osd.1 v2:172.21.15.167:6800/30035 32 ==== osd_op_reply(93 notify.4 [watch watch cookie 93995517725792] v30'133 uv132 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7fdd68052040 con 0x557d096e18c0 2023-10-05T19:28:34.579 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.574+0000 7fdd6efec700 1 -- 172.21.15.167:0/4130379316 <== osd.0 v2:172.21.15.118:6800/27690 23 ==== osd_op_reply(94 notify.5 [create] v30'132 uv132 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7fdd68052040 con 0x7fdd6400eb60 2023-10-05T19:28:34.579 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.574+0000 7fdd70cb6e40 1 -- 172.21.15.167:0/4130379316 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:95 7.3 7:c609908c:::notify.5:head [watch watch cookie 93995517735872] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x557d0971fd00 con 0x7fdd6400eb60 2023-10-05T19:28:34.579 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.574+0000 7fdd70cb6e40 1 -- 172.21.15.167:0/4130379316 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:96 7.14 7:2b04a3e9:::notify.6:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x557d097207e0 con 0x7fdd6400eb60 2023-10-05T19:28:34.582 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.574+0000 7fdd6efec700 1 -- 172.21.15.167:0/4130379316 <== osd.0 v2:172.21.15.118:6800/27690 24 ==== osd_op_reply(96 notify.6 [create] v30'132 uv132 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7fdd68052040 con 0x7fdd6400eb60 2023-10-05T19:28:34.582 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.574+0000 7fdd70cb6e40 1 -- 172.21.15.167:0/4130379316 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:97 7.14 7:2b04a3e9:::notify.6:head [watch watch cookie 93995517739696] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x557d09720bf0 con 0x7fdd6400eb60 2023-10-05T19:28:34.582 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.578+0000 7fdd70cb6e40 1 -- 172.21.15.167:0/4130379316 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:98 7.9 7:93e5b521:::notify.7:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x557d09721700 con 0x7fdd6400eb60 2023-10-05T19:28:34.582 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.578+0000 7fdd6efec700 1 -- 172.21.15.167:0/4130379316 <== osd.0 v2:172.21.15.118:6800/27690 25 ==== osd_op_reply(95 notify.5 [watch watch cookie 93995517735872] v30'133 uv132 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7fdd68052040 con 0x7fdd6400eb60 2023-10-05T19:28:34.584 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.578+0000 7fdd6efec700 1 -- 172.21.15.167:0/4130379316 <== osd.0 v2:172.21.15.118:6800/27690 26 ==== osd_op_reply(97 notify.6 [watch watch cookie 93995517739696] v30'133 uv132 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7fdd68052040 con 0x7fdd6400eb60 2023-10-05T19:28:34.585 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.578+0000 7fdd6efec700 1 -- 172.21.15.167:0/4130379316 <== osd.0 v2:172.21.15.118:6800/27690 27 ==== osd_op_reply(98 notify.7 [create] v30'132 uv132 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7fdd68052040 con 0x7fdd6400eb60 2023-10-05T19:28:34.585 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.578+0000 7fdd70cb6e40 1 -- 172.21.15.167:0/4130379316 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:99 7.9 7:93e5b521:::notify.7:head [watch watch cookie 93995517736976] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x557d09720700 con 0x7fdd6400eb60 2023-10-05T19:28:34.585 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.578+0000 7fdd70cb6e40 20 add_watcher() i=0 2023-10-05T19:28:34.585 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.578+0000 7fdd70cb6e40 20 add_watcher() i=1 2023-10-05T19:28:34.586 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.578+0000 7fdd70cb6e40 20 add_watcher() i=2 2023-10-05T19:28:34.586 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.578+0000 7fdd70cb6e40 20 add_watcher() i=3 2023-10-05T19:28:34.586 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.578+0000 7fdd70cb6e40 20 add_watcher() i=4 2023-10-05T19:28:34.586 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.578+0000 7fdd70cb6e40 20 add_watcher() i=5 2023-10-05T19:28:34.586 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.578+0000 7fdd70cb6e40 20 add_watcher() i=6 2023-10-05T19:28:34.587 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.582+0000 7fdd6efec700 1 -- 172.21.15.167:0/4130379316 <== osd.0 v2:172.21.15.118:6800/27690 28 ==== osd_op_reply(99 notify.7 [watch watch cookie 93995517736976] v30'133 uv132 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7fdd68052040 con 0x7fdd6400eb60 2023-10-05T19:28:34.588 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.582+0000 7fdd70cb6e40 20 add_watcher() i=7 2023-10-05T19:28:34.588 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.582+0000 7fdd70cb6e40 2 all 8 watchers are set, enabling cache 2023-10-05T19:28:34.588 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.582+0000 7fdd70cb6e40 1 -- 172.21.15.167:0/4130379316 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:100 6.4 6:22d26bf9:::data_loggenerations_metadata:head [call version.check_conds in=74b,call version.read in=11b,read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x557d09720c00 con 0x557d096e18c0 2023-10-05T19:28:34.588 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.582+0000 7fdd6efec700 1 -- 172.21.15.167:0/4130379316 <== osd.1 v2:172.21.15.167:6800/30035 33 ==== osd_op_reply(100 data_loggenerations_metadata [call,call out=48b,read 0~28 out=28b] v0'0 uv1 ondisk = 0) v8 ==== 256+0+76 (crc 0 0 0) 0x7fdd68052040 con 0x557d096e18c0 2023-10-05T19:28:34.588 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.582+0000 7fdd70cb6e40 1 -- 172.21.15.167:0/4130379316 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:101 6.4 6:22d26bf9:::data_loggenerations_metadata:head [watch watch cookie 93995517743568] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x557d09720c00 con 0x557d096e18c0 2023-10-05T19:28:34.590 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.586+0000 7fdd6efec700 1 -- 172.21.15.167:0/4130379316 <== osd.1 v2:172.21.15.167:6800/30035 34 ==== osd_op_reply(101 data_loggenerations_metadata [watch watch cookie 93995517743568] v30'129 uv1 ondisk = 0) v8 ==== 172+0+0 (crc 0 0 0) 0x7fdd68052040 con 0x557d096e18c0 2023-10-05T19:28:34.591 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.586+0000 7fdd3ffdf700 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2023-10-05T19:28:34.591 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.586+0000 7fdd70cb6e40 20 check_secure_mon_conn(): auth registy supported: methods=[2] modes=[2,1] 2023-10-05T19:28:34.591 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.586+0000 7fdd70cb6e40 20 check_secure_mon_conn(): mode 1 is insecure 2023-10-05T19:28:34.591 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.586+0000 7fdd70cb6e40 1 Processor -- start 2023-10-05T19:28:34.592 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.586+0000 7fdd70cb6e40 1 -- start start 2023-10-05T19:28:34.592 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.586+0000 7fdd70cb6e40 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x557d0972eb90 0x557d0972ef60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-05T19:28:34.592 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.586+0000 7fdd70cb6e40 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x557d0972f4a0 con 0x557d0972eb90 2023-10-05T19:28:34.592 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.586+0000 7fdd6dfea700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x557d0972eb90 0x557d0972ef60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-05T19:28:34.593 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.586+0000 7fdd6dfea700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x557d0972eb90 0x557d0972ef60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.118:3300/0 says I am v2:172.21.15.167:41486/0 (socket says 172.21.15.167:41486) 2023-10-05T19:28:34.593 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.586+0000 7fdd6dfea700 1 -- 172.21.15.167:0/1641098007 learned_addr learned my addr 172.21.15.167:0/1641098007 (peer_addr_for_me v2:172.21.15.167:0/0) 2023-10-05T19:28:34.593 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.586+0000 7fdd6dfea700 1 -- 172.21.15.167:0/1641098007 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x557d0972f5e0 con 0x557d0972eb90 2023-10-05T19:28:34.593 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.586+0000 7fdd6dfea700 1 --2- 172.21.15.167:0/1641098007 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x557d0972eb90 0x557d0972ef60 secure :-1 s=READY pgs=402 cs=0 l=1 rev1=1 crypto rx=0x557d09713c30 tx=0x7fdd64006000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=709eed9fe18b4c04 server_cookie=0 in_seq=0 out_seq=0 2023-10-05T19:28:34.594 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.586+0000 7fdd3f7de700 1 -- 172.21.15.167:0/1641098007 <== mon.0 v2:172.21.15.118:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fdd64023040 con 0x557d0972eb90 2023-10-05T19:28:34.594 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.586+0000 7fdd3f7de700 1 -- 172.21.15.167:0/1641098007 <== mon.0 v2:172.21.15.118:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdd64024040 con 0x557d0972eb90 2023-10-05T19:28:34.594 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.586+0000 7fdd3f7de700 1 -- 172.21.15.167:0/1641098007 <== mon.0 v2:172.21.15.118:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fdd64013040 con 0x557d0972eb90 2023-10-05T19:28:34.595 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.590+0000 7fdd70cb6e40 1 -- 172.21.15.167:0/1641098007 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x557d0972eb90 msgr2=0x557d0972ef60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-05T19:28:34.595 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.590+0000 7fdd70cb6e40 1 --2- 172.21.15.167:0/1641098007 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x557d0972eb90 0x557d0972ef60 secure :-1 s=READY pgs=402 cs=0 l=1 rev1=1 crypto rx=0x557d09713c30 tx=0x7fdd64006000 comp rx=0 tx=0).stop 2023-10-05T19:28:34.595 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.590+0000 7fdd70cb6e40 1 -- 172.21.15.167:0/1641098007 shutdown_connections 2023-10-05T19:28:34.595 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.590+0000 7fdd70cb6e40 1 --2- 172.21.15.167:0/1641098007 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x557d0972eb90 0x557d0972ef60 unknown :-1 s=CLOSED pgs=402 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-05T19:28:34.595 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.590+0000 7fdd70cb6e40 1 -- 172.21.15.167:0/1641098007 >> 172.21.15.167:0/1641098007 conn(0x557d09729f50 msgr2=0x557d097280a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-05T19:28:34.595 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.590+0000 7fdd70cb6e40 1 -- 172.21.15.167:0/1641098007 shutdown_connections 2023-10-05T19:28:34.595 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.590+0000 7fdd70cb6e40 1 -- 172.21.15.167:0/1641098007 wait complete. 2023-10-05T19:28:34.596 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.590+0000 7fdd70cb6e40 1 Processor -- start 2023-10-05T19:28:34.596 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.590+0000 7fdd70cb6e40 1 -- start start 2023-10-05T19:28:34.596 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.590+0000 7fdd70cb6e40 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x557d0972eb90 0x557d09734720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-05T19:28:34.597 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.590+0000 7fdd70cb6e40 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x557d09735720 con 0x557d0972eb90 2023-10-05T19:28:34.597 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.590+0000 7fdd6dfea700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x557d0972eb90 0x557d09734720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-05T19:28:34.597 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.590+0000 7fdd6dfea700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x557d0972eb90 0x557d09734720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.118:3300/0 says I am v2:172.21.15.167:41494/0 (socket says 172.21.15.167:41494) 2023-10-05T19:28:34.597 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.590+0000 7fdd6dfea700 1 -- 172.21.15.167:0/1620034584 learned_addr learned my addr 172.21.15.167:0/1620034584 (peer_addr_for_me v2:172.21.15.167:0/0) 2023-10-05T19:28:34.597 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.590+0000 7fdd6dfea700 1 -- 172.21.15.167:0/1620034584 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdd58001e10 con 0x557d0972eb90 2023-10-05T19:28:34.597 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.590+0000 7fdd6dfea700 1 --2- 172.21.15.167:0/1620034584 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x557d0972eb90 0x557d09734720 secure :-1 s=READY pgs=403 cs=0 l=1 rev1=1 crypto rx=0x7fdd6400bef0 tx=0x7fdd64030000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-05T19:28:34.598 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.590+0000 7fdd3dfdb700 1 -- 172.21.15.167:0/1620034584 <== mon.0 v2:172.21.15.118:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fdd64023070 con 0x557d0972eb90 2023-10-05T19:28:34.598 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.590+0000 7fdd3dfdb700 1 -- 172.21.15.167:0/1620034584 <== mon.0 v2:172.21.15.118:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdd6402d240 con 0x557d0972eb90 2023-10-05T19:28:34.598 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.594+0000 7fdd70cb6e40 1 -- 172.21.15.167:0/1620034584 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x557d09734d20 con 0x557d0972eb90 2023-10-05T19:28:34.598 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.594+0000 7fdd3dfdb700 1 -- 172.21.15.167:0/1620034584 <== mon.0 v2:172.21.15.118:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fdd64042440 con 0x557d0972eb90 2023-10-05T19:28:34.599 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.594+0000 7fdd70cb6e40 1 -- 172.21.15.167:0/1620034584 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x557d09735160 con 0x557d0972eb90 2023-10-05T19:28:34.599 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.594+0000 7fdd3dfdb700 1 -- 172.21.15.167:0/1620034584 <== mon.0 v2:172.21.15.118:3300/0 4 ==== mgrmap(e 5) v1 ==== 40329+0+0 (secure 0 0 0) 0x7fdd6403f040 con 0x557d0972eb90 2023-10-05T19:28:34.600 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.594+0000 7fdd3dfdb700 1 --2- 172.21.15.167:0/1620034584 >> [v2:172.21.15.167:6824/29889,v1:172.21.15.167:6825/29889] conn(0x7fdcb4030c00 0x7fdcb40330b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-05T19:28:34.600 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.594+0000 7fdd3dfdb700 1 -- 172.21.15.167:0/1620034584 <== mon.0 v2:172.21.15.118:3300/0 5 ==== osd_map(30..30 src has 1..30) v4 ==== 6057+0+0 (secure 0 0 0) 0x7fdd640200b0 con 0x557d0972eb90 2023-10-05T19:28:34.600 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.594+0000 7fdd6efec700 1 --2- 172.21.15.167:0/1620034584 >> [v2:172.21.15.167:6824/29889,v1:172.21.15.167:6825/29889] conn(0x7fdcb4030c00 0x7fdcb40330b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-05T19:28:34.600 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.594+0000 7fdd70cb6e40 5 note: GC not initialized 2023-10-05T19:28:34.601 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.594+0000 7fdd6efec700 1 --2- 172.21.15.167:0/1620034584 >> [v2:172.21.15.167:6824/29889,v1:172.21.15.167:6825/29889] conn(0x7fdcb4030c00 0x7fdcb40330b0 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x557d097209b0 tx=0x7fdd6800a280 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-05T19:28:34.603 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.598+0000 7fdcaeffd700 20 reqs_thread_entry: start 2023-10-05T19:28:34.603 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.598+0000 7fdd70cb6e40 20 init_complete bucket index max shards: 11 2023-10-05T19:28:34.603 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.598+0000 7fdcacff9700 20 reqs_thread_entry: start 2023-10-05T19:28:34.604 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.598+0000 7fdd70cb6e40 10 rgw notify: Started notification manager with: 1 workers 2023-10-05T19:28:34.604 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.598+0000 7fdc9effd700 20 rgw notify: INFO: next queues processing will happen at: Thu Oct 5 19:29:04 2023 2023-10-05T19:28:34.604 INFO:teuthology.orchestra.run.smithi167.stderr: 2023-10-05T19:28:34.604 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.598+0000 7fdc9e7fc700 20 reqs_thread_entry: start 2023-10-05T19:28:34.604 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.598+0000 7fdd70cb6e40 20 get_system_obj_state: rctx=0x7fffa3bc2208 obj=default.rgw.meta:users.uid:testx$tenanteduser state=0x557d095b80b0 s->prefetch_data=0 2023-10-05T19:28:34.605 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.598+0000 7fdd70cb6e40 10 cache get: name=default.rgw.meta+users.uid+testx$tenanteduser : miss 2023-10-05T19:28:34.605 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.598+0000 7fdd70cb6e40 1 -- 172.21.15.167:0/4130379316 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:102 8.13 8:cdfb9947:users.uid::testx$tenanteduser:head [call version.read in=11b,getxattrs,stat] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x557d0979cf10 con 0x7fdd6400eb60 2023-10-05T19:28:34.605 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.598+0000 7fdd6efec700 1 -- 172.21.15.167:0/4130379316 <== osd.0 v2:172.21.15.118:6800/27690 29 ==== osd_op_reply(102 testx$tenanteduser [call out=48b,getxattrs out=75b,stat out=16b] v0'0 uv8 ondisk = 0) v8 ==== 246+0+139 (crc 0 0 0) 0x7fdd68052040 con 0x7fdd6400eb60 2023-10-05T19:28:34.605 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.598+0000 7fdd70cb6e40 10 cache put: name=default.rgw.meta+users.uid+testx$tenanteduser info.flags=0x16 2023-10-05T19:28:34.606 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.598+0000 7fdd70cb6e40 10 adding default.rgw.meta+users.uid+testx$tenanteduser to cache LRU end 2023-10-05T19:28:34.606 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.598+0000 7fdd70cb6e40 10 updating xattr: name=ceph.objclass.version bl.length()=42 2023-10-05T19:28:34.606 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.598+0000 7fdd70cb6e40 20 get_system_obj_state: s->obj_tag was set empty 2023-10-05T19:28:34.606 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.598+0000 7fdd70cb6e40 20 Read xattr: user.rgw.idtag 2023-10-05T19:28:34.606 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.598+0000 7fdd70cb6e40 10 cache get: name=default.rgw.meta+users.uid+testx$tenanteduser : type miss (requested=0x13, cached=0x16) 2023-10-05T19:28:34.606 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.598+0000 7fdd70cb6e40 20 rados->read ofs=0 len=0 2023-10-05T19:28:34.606 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.598+0000 7fdd70cb6e40 1 -- 172.21.15.167:0/4130379316 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:103 8.13 8:cdfb9947:users.uid::testx$tenanteduser:head [call version.check_conds in=122b,call version.read in=11b,read 0~0,getxattrs] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x557d0979cee0 con 0x7fdd6400eb60 2023-10-05T19:28:34.607 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.602+0000 7fdd6efec700 1 -- 172.21.15.167:0/4130379316 <== osd.0 v2:172.21.15.118:6800/27690 30 ==== osd_op_reply(103 testx$tenanteduser [call,call out=48b,read 0~385 out=385b,getxattrs out=75b] v0'0 uv8 ondisk = 0) v8 ==== 288+0+508 (crc 0 0 0) 0x7fdd68052040 con 0x7fdd6400eb60 2023-10-05T19:28:34.607 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.602+0000 7fdd70cb6e40 20 rados_obj.operate() r=0 bl.length=385 2023-10-05T19:28:34.607 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.602+0000 7fdd70cb6e40 10 cache put: name=default.rgw.meta+users.uid+testx$tenanteduser info.flags=0x13 2023-10-05T19:28:34.607 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.602+0000 7fdd70cb6e40 10 moving default.rgw.meta+users.uid+testx$tenanteduser to cache LRU end 2023-10-05T19:28:34.607 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.602+0000 7fdd70cb6e40 10 updating xattr: name=ceph.objclass.version bl.length()=42 2023-10-05T19:28:34.607 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.602+0000 7fdd70cb6e40 20 get_system_obj_state: rctx=0x7fffa3bc15b0 obj=default.rgw.meta:root:myfoo state=0x557d095b80b0 s->prefetch_data=0 2023-10-05T19:28:34.607 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.602+0000 7fdd70cb6e40 10 cache get: name=default.rgw.meta+root+myfoo : miss 2023-10-05T19:28:34.607 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.602+0000 7fdd70cb6e40 1 -- 172.21.15.167:0/4130379316 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:104 8.9 8:91b71841:root::myfoo:head [call version.read in=11b,getxattrs,stat] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x557d0979cee0 con 0x7fdd60007030 2023-10-05T19:28:34.608 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.602+0000 7fdd6dfea700 1 -- 172.21.15.167:0/4130379316 <== osd.2 v2:172.21.15.167:6816/30037 17 ==== osd_op_reply(104 myfoo [call out=48b,getxattrs out=75b,stat out=16b] v0'0 uv23 ondisk = 0) v8 ==== 233+0+139 (crc 0 0 0) 0x7fdd64006070 con 0x7fdd60007030 2023-10-05T19:28:34.608 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.602+0000 7fdd70cb6e40 10 cache put: name=default.rgw.meta+root+myfoo info.flags=0x16 2023-10-05T19:28:34.609 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.602+0000 7fdd70cb6e40 10 adding default.rgw.meta+root+myfoo to cache LRU end 2023-10-05T19:28:34.609 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.602+0000 7fdd70cb6e40 10 updating xattr: name=ceph.objclass.version bl.length()=42 2023-10-05T19:28:34.609 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.602+0000 7fdd70cb6e40 20 get_system_obj_state: s->obj_tag was set empty 2023-10-05T19:28:34.609 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.602+0000 7fdd70cb6e40 10 cache get: name=default.rgw.meta+root+myfoo : type miss (requested=0x11, cached=0x16) 2023-10-05T19:28:34.609 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.602+0000 7fdd70cb6e40 20 rados->read ofs=0 len=0 2023-10-05T19:28:34.609 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.602+0000 7fdd70cb6e40 1 -- 172.21.15.167:0/4130379316 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:105 8.9 8:91b71841:root::myfoo:head [call version.check_conds in=122b,call version.read in=11b,read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x557d097a1c30 con 0x7fdd60007030 2023-10-05T19:28:34.609 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.602+0000 7fdd6dfea700 1 -- 172.21.15.167:0/4130379316 <== osd.2 v2:172.21.15.167:6816/30037 18 ==== osd_op_reply(105 myfoo [call,call out=48b,read 0~165 out=165b] v0'0 uv23 ondisk = 0) v8 ==== 233+0+213 (crc 0 0 0) 0x7fdd64006070 con 0x7fdd60007030 2023-10-05T19:28:34.609 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.602+0000 7fdd70cb6e40 20 rados_obj.operate() r=0 bl.length=165 2023-10-05T19:28:34.609 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.602+0000 7fdd70cb6e40 10 cache put: name=default.rgw.meta+root+myfoo info.flags=0x11 2023-10-05T19:28:34.609 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.602+0000 7fdd70cb6e40 10 moving default.rgw.meta+root+myfoo to cache LRU end 2023-10-05T19:28:34.609 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.602+0000 7fdd70cb6e40 20 get_system_obj_state: rctx=0x7fffa3bc15b0 obj=default.rgw.meta:root:.bucket.meta.myfoo:9a9df965-6612-45d4-a45f-6e3c889bacc2.4223.1 state=0x557d0958e2b0 s->prefetch_data=0 2023-10-05T19:28:34.610 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.602+0000 7fdd70cb6e40 10 cache get: name=default.rgw.meta+root+.bucket.meta.myfoo:9a9df965-6612-45d4-a45f-6e3c889bacc2.4223.1 : miss 2023-10-05T19:28:34.610 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.602+0000 7fdd70cb6e40 1 -- 172.21.15.167:0/4130379316 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:106 8.11 8:8bb4ed75:root::.bucket.meta.myfoo%3a9a9df965-6612-45d4-a45f-6e3c889bacc2.4223.1:head [call version.read in=11b,getxattrs,stat] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x557d097a13d0 con 0x557d097164d0 2023-10-05T19:28:34.610 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.602+0000 7fdd6e7eb700 1 -- 172.21.15.167:0/4130379316 <== osd.3 v2:172.21.15.167:6808/30036 24 ==== osd_op_reply(106 .bucket.meta.myfoo:9a9df965-6612-45d4-a45f-6e3c889bacc2.4223.1 [call out=48b,getxattrs out=218b,stat out=16b] v0'0 uv3 ondisk = 0) v8 ==== 290+0+282 (crc 0 0 0) 0x7fdd6005d040 con 0x557d097164d0 2023-10-05T19:28:34.610 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.602+0000 7fdd70cb6e40 10 cache put: name=default.rgw.meta+root+.bucket.meta.myfoo:9a9df965-6612-45d4-a45f-6e3c889bacc2.4223.1 info.flags=0x16 2023-10-05T19:28:34.611 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.602+0000 7fdd70cb6e40 10 adding default.rgw.meta+root+.bucket.meta.myfoo:9a9df965-6612-45d4-a45f-6e3c889bacc2.4223.1 to cache LRU end 2023-10-05T19:28:34.611 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.602+0000 7fdd70cb6e40 10 updating xattr: name=ceph.objclass.version bl.length()=42 2023-10-05T19:28:34.611 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.602+0000 7fdd70cb6e40 10 updating xattr: name=user.rgw.acl bl.length()=123 2023-10-05T19:28:34.611 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.602+0000 7fdd70cb6e40 20 get_system_obj_state: s->obj_tag was set empty 2023-10-05T19:28:34.611 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.602+0000 7fdd70cb6e40 20 Read xattr: user.rgw.acl 2023-10-05T19:28:34.611 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.602+0000 7fdd70cb6e40 20 Read xattr: user.rgw.idtag 2023-10-05T19:28:34.611 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.602+0000 7fdd70cb6e40 10 cache get: name=default.rgw.meta+root+.bucket.meta.myfoo:9a9df965-6612-45d4-a45f-6e3c889bacc2.4223.1 : type miss (requested=0x13, cached=0x16) 2023-10-05T19:28:34.611 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.602+0000 7fdd70cb6e40 20 rados->read ofs=0 len=0 2023-10-05T19:28:34.611 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.602+0000 7fdd70cb6e40 1 -- 172.21.15.167:0/4130379316 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:107 8.11 8:8bb4ed75:root::.bucket.meta.myfoo%3a9a9df965-6612-45d4-a45f-6e3c889bacc2.4223.1:head [call version.check_conds in=122b,call version.read in=11b,read 0~0,getxattrs] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x557d097a4330 con 0x557d097164d0 2023-10-05T19:28:34.613 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.606+0000 7fdd6e7eb700 1 -- 172.21.15.167:0/4130379316 <== osd.3 v2:172.21.15.167:6808/30036 25 ==== osd_op_reply(107 .bucket.meta.myfoo:9a9df965-6612-45d4-a45f-6e3c889bacc2.4223.1 [call,call out=48b,read 0~352 out=352b,getxattrs out=218b] v0'0 uv3 ondisk = 0) v8 ==== 332+0+618 (crc 0 0 0) 0x7fdd6005d040 con 0x557d097164d0 2023-10-05T19:28:34.613 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.606+0000 7fdd70cb6e40 20 rados_obj.operate() r=0 bl.length=352 2023-10-05T19:28:34.613 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.606+0000 7fdd70cb6e40 10 cache put: name=default.rgw.meta+root+.bucket.meta.myfoo:9a9df965-6612-45d4-a45f-6e3c889bacc2.4223.1 info.flags=0x13 2023-10-05T19:28:34.613 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.606+0000 7fdd70cb6e40 10 moving default.rgw.meta+root+.bucket.meta.myfoo:9a9df965-6612-45d4-a45f-6e3c889bacc2.4223.1 to cache LRU end 2023-10-05T19:28:34.613 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.606+0000 7fdd70cb6e40 10 updating xattr: name=ceph.objclass.version bl.length()=42 2023-10-05T19:28:34.613 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.606+0000 7fdd70cb6e40 10 updating xattr: name=user.rgw.acl bl.length()=123 2023-10-05T19:28:34.613 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.606+0000 7fdd70cb6e40 10 chain_cache_entry: cache_locator=default.rgw.meta+root+.bucket.meta.myfoo:9a9df965-6612-45d4-a45f-6e3c889bacc2.4223.1 2023-10-05T19:28:34.613 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.606+0000 7fdd70cb6e40 20 get_system_obj_state: rctx=0x7fffa3bc1548 obj=default.rgw.meta:users.uid:testx$tenanteduser state=0x557d0958e2b0 s->prefetch_data=0 2023-10-05T19:28:34.613 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.606+0000 7fdd70cb6e40 10 cache get: name=default.rgw.meta+users.uid+testx$tenanteduser : type miss (requested=0x16, cached=0x13) 2023-10-05T19:28:34.613 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.606+0000 7fdd70cb6e40 1 -- 172.21.15.167:0/4130379316 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:108 8.13 8:cdfb9947:users.uid::testx$tenanteduser:head [call version.read in=11b,getxattrs,stat] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x557d097a4aa0 con 0x7fdd6400eb60 2023-10-05T19:28:34.614 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.606+0000 7fdd6efec700 1 -- 172.21.15.167:0/4130379316 <== osd.0 v2:172.21.15.118:6800/27690 31 ==== osd_op_reply(108 testx$tenanteduser [call out=48b,getxattrs out=75b,stat out=16b] v0'0 uv8 ondisk = 0) v8 ==== 246+0+139 (crc 0 0 0) 0x7fdd68052040 con 0x7fdd6400eb60 2023-10-05T19:28:34.614 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.606+0000 7fdd70cb6e40 10 cache put: name=default.rgw.meta+users.uid+testx$tenanteduser info.flags=0x16 2023-10-05T19:28:34.614 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.606+0000 7fdd70cb6e40 10 moving default.rgw.meta+users.uid+testx$tenanteduser to cache LRU end 2023-10-05T19:28:34.614 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.606+0000 7fdd70cb6e40 10 updating xattr: name=ceph.objclass.version bl.length()=42 2023-10-05T19:28:34.614 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.606+0000 7fdd70cb6e40 20 get_system_obj_state: s->obj_tag was set empty 2023-10-05T19:28:34.614 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.606+0000 7fdd70cb6e40 20 Read xattr: user.rgw.idtag 2023-10-05T19:28:34.614 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.606+0000 7fdd70cb6e40 10 cache get: name=default.rgw.meta+users.uid+testx$tenanteduser : hit (requested=0x13, cached=0x17) 2023-10-05T19:28:34.614 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.606+0000 7fdd70cb6e40 1 -- 172.21.15.167:0/4130379316 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:109 8.9 8:96d98480:users.uid::foo.buckets:head [call user.remove_bucket in=58b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x557d097a2f50 con 0x7fdd60007030 2023-10-05T19:28:34.615 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.610+0000 7fdd6dfea700 1 -- 172.21.15.167:0/4130379316 <== osd.2 v2:172.21.15.167:6816/30037 19 ==== osd_op_reply(109 foo.buckets [call] v30'24 uv24 ondisk = 0) v8 ==== 155+0+0 (crc 0 0 0) 0x7fdd64006070 con 0x7fdd60007030 2023-10-05T19:28:34.615 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.610+0000 7fdd70cb6e40 1 -- 172.21.15.167:0/4130379316 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:110 8.6 8:6736df0c:root::.bucket.meta.testx%3amyfoo%3a9a9df965-6612-45d4-a45f-6e3c889bacc2.4223.1:head [create,call version.inc in=38b,writefull 0~371 in=371b,setxattr user.rgw.acl (203) in=215b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x557d097aa890 con 0x7fdd6400eb60 2023-10-05T19:28:34.619 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.614+0000 7fdd6efec700 1 -- 172.21.15.167:0/4130379316 <== osd.0 v2:172.21.15.118:6800/27690 32 ==== osd_op_reply(110 .bucket.meta.testx:myfoo:9a9df965-6612-45d4-a45f-6e3c889bacc2.4223.1 [create,call,writefull 0~371,setxattr (203)] v30'1 uv1 ondisk = 0) v8 ==== 338+0+0 (crc 0 0 0) 0x7fdd68052040 con 0x7fdd6400eb60 2023-10-05T19:28:34.619 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.614+0000 7fdd70cb6e40 10 cache put: name=default.rgw.meta+root+.bucket.meta.testx:myfoo:9a9df965-6612-45d4-a45f-6e3c889bacc2.4223.1 info.flags=0x7 2023-10-05T19:28:34.619 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.614+0000 7fdd70cb6e40 10 adding default.rgw.meta+root+.bucket.meta.testx:myfoo:9a9df965-6612-45d4-a45f-6e3c889bacc2.4223.1 to cache LRU end 2023-10-05T19:28:34.619 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.614+0000 7fdd70cb6e40 10 updating xattr: name=user.rgw.acl bl.length()=203 2023-10-05T19:28:34.619 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.614+0000 7fdd70cb6e40 10 distributing notification oid=default.rgw.control:notify.6 cni=[op: 0, obj: default.rgw.meta:root:.bucket.meta.testx:myfoo:9a9df965-6612-45d4-a45f-6e3c889bacc2.4223.1, ofs0, ns] 2023-10-05T19:28:34.620 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.614+0000 7fdd70cb6e40 1 -- 172.21.15.167:0/4130379316 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:111 7.14 7:2b04a3e9:::notify.6:head [notify cookie 93995518298224 in=818b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x557d097aa1c0 con 0x7fdd6400eb60 2023-10-05T19:28:34.620 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.614+0000 7fdd6efec700 1 -- 172.21.15.167:0/4130379316 <== osd.0 v2:172.21.15.118:6800/27690 33 ==== watch-notify(notify (1) cookie 93995517739696 notify 128849018891 ret 0) v3 ==== 848+0+0 (crc 0 0 0) 0x7fdd6804e380 con 0x7fdd6400eb60 2023-10-05T19:28:34.620 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.614+0000 7fdd5f7fe700 10 rgw watcher librados: RGWWatcher::handle_notify() notify_id 128849018891 cookie 93995517739696 notifier 4491 bl.length()=806 2023-10-05T19:28:34.620 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.614+0000 7fdd5f7fe700 10 rgw watcher librados: cache put: name=default.rgw.meta+root+.bucket.meta.testx:myfoo:9a9df965-6612-45d4-a45f-6e3c889bacc2.4223.1 info.flags=0x7 2023-10-05T19:28:34.620 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.614+0000 7fdd5f7fe700 10 rgw watcher librados: moving default.rgw.meta+root+.bucket.meta.testx:myfoo:9a9df965-6612-45d4-a45f-6e3c889bacc2.4223.1 to cache LRU end 2023-10-05T19:28:34.621 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.614+0000 7fdd5f7fe700 10 rgw watcher librados: updating xattr: name=user.rgw.acl bl.length()=203 2023-10-05T19:28:34.621 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.614+0000 7fdd5f7fe700 1 -- 172.21.15.167:0/4130379316 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:112 7.14 7:2b04a3e9:::notify.6:head [notify-ack in=20b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x7fdd54003d90 con 0x7fdd6400eb60 2023-10-05T19:28:34.621 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.614+0000 7fdd6efec700 1 -- 172.21.15.167:0/4130379316 <== osd.0 v2:172.21.15.118:6800/27690 34 ==== osd_op_reply(111 notify.6 [notify cookie 93995518298224 out=8b] v0'0 uv132 ondisk = 0) v8 ==== 152+0+8 (crc 0 0 0) 0x7fdd68052040 con 0x7fdd6400eb60 2023-10-05T19:28:34.621 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.614+0000 7fdd6efec700 1 -- 172.21.15.167:0/4130379316 <== osd.0 v2:172.21.15.118:6800/27690 35 ==== watch-notify(notify_complete (2) cookie 93995518298224 notify 128849018891 ret 0) v3 ==== 42+0+48 (crc 0 0 0) 0x7fdd68036b90 con 0x7fdd6400eb60 2023-10-05T19:28:34.621 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.614+0000 7fdd6efec700 1 -- 172.21.15.167:0/4130379316 <== osd.0 v2:172.21.15.118:6800/27690 36 ==== osd_op_reply(112 notify.6 [notify-ack] v0'0 uv132 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7fdd68052040 con 0x7fdd6400eb60 2023-10-05T19:28:34.621 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.614+0000 7fdd70cb6e40 20 handle_bi_update(): bucket=testx:myfoo[9a9df965-6612-45d4-a45f-6e3c889bacc2.4223.1]): orig_sources= new_sources= 2023-10-05T19:28:34.622 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.614+0000 7fdd70cb6e40 20 handle_bi_update(): bucket=testx:myfoo[9a9df965-6612-45d4-a45f-6e3c889bacc2.4223.1]): potential sources added=[] removed=[] 2023-10-05T19:28:34.622 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.614+0000 7fdd70cb6e40 20 handle_bi_update(): bucket=testx:myfoo[9a9df965-6612-45d4-a45f-6e3c889bacc2.4223.1]): orig_dests= new_dests= 2023-10-05T19:28:34.622 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.614+0000 7fdd70cb6e40 20 handle_bi_update(): bucket=testx:myfoo[9a9df965-6612-45d4-a45f-6e3c889bacc2.4223.1]): potential dests added=[] removed=[] 2023-10-05T19:28:34.622 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.614+0000 7fdd70cb6e40 1 -- 172.21.15.167:0/4130379316 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:113 8.c 8:37c7a57e:users.uid::testx$tenanteduser.buckets:head [call user.set_buckets_info in=207b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x557d097aa7e0 con 0x557d096e18c0 2023-10-05T19:28:34.624 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.618+0000 7fdd6efec700 1 -- 172.21.15.167:0/4130379316 <== osd.1 v2:172.21.15.167:6800/30035 35 ==== osd_op_reply(113 testx$tenanteduser.buckets [call] v30'1 uv1 ondisk = 0) v8 ==== 170+0+0 (crc 0 0 0) 0x7fdd6806c040 con 0x557d096e18c0 2023-10-05T19:28:34.624 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.618+0000 7fdd70cb6e40 1 -- 172.21.15.167:0/4130379316 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:114 8.5 8:a7a6c9d8:root::testx%2fmyfoo:head [delete,create,call version.inc in=38b,writefull 0~193 in=193b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x557d097a9d70 con 0x557d096e18c0 2023-10-05T19:28:34.628 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.622+0000 7fdd6efec700 1 -- 172.21.15.167:0/4130379316 <== osd.1 v2:172.21.15.167:6800/30035 36 ==== osd_op_reply(114 testx/myfoo [delete,create,call,writefull 0~193] v30'4 uv4 ondisk = 0) v8 ==== 281+0+0 (crc 0 0 0) 0x7fdd6806c040 con 0x557d096e18c0 2023-10-05T19:28:34.628 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.622+0000 7fdd70cb6e40 10 cache put: name=default.rgw.meta+root+testx/myfoo info.flags=0x7 2023-10-05T19:28:34.628 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.622+0000 7fdd70cb6e40 10 adding default.rgw.meta+root+testx/myfoo to cache LRU end 2023-10-05T19:28:34.628 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.622+0000 7fdd70cb6e40 10 distributing notification oid=default.rgw.control:notify.4 cni=[op: 0, obj: default.rgw.meta:root:testx/myfoo, ofs0, ns] 2023-10-05T19:28:34.629 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.622+0000 7fdd70cb6e40 1 -- 172.21.15.167:0/4130379316 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:115 7.d 7:b4812045:::notify.4:head [notify cookie 93995518298224 in=360b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x557d097ab010 con 0x557d096e18c0 2023-10-05T19:28:34.629 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.622+0000 7fdd6efec700 1 -- 172.21.15.167:0/4130379316 <== osd.1 v2:172.21.15.167:6800/30035 37 ==== watch-notify(notify (1) cookie 93995517725792 notify 128849018894 ret 0) v3 ==== 390+0+0 (crc 0 0 0) 0x7fdd68036b90 con 0x557d096e18c0 2023-10-05T19:28:34.629 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.622+0000 7fdd5ffff700 10 rgw watcher librados: RGWWatcher::handle_notify() notify_id 128849018894 cookie 93995517725792 notifier 4491 bl.length()=348 2023-10-05T19:28:34.629 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.622+0000 7fdd5ffff700 10 rgw watcher librados: cache put: name=default.rgw.meta+root+testx/myfoo info.flags=0x7 2023-10-05T19:28:34.629 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.622+0000 7fdd5ffff700 10 rgw watcher librados: moving default.rgw.meta+root+testx/myfoo to cache LRU end 2023-10-05T19:28:34.630 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.622+0000 7fdd5ffff700 1 -- 172.21.15.167:0/4130379316 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:116 7.d 7:b4812045:::notify.4:head [notify-ack in=20b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x7fdcbc003410 con 0x557d096e18c0 2023-10-05T19:28:34.630 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.622+0000 7fdd6efec700 1 -- 172.21.15.167:0/4130379316 <== osd.1 v2:172.21.15.167:6800/30035 38 ==== osd_op_reply(115 notify.4 [notify cookie 93995518298224 out=8b] v0'0 uv132 ondisk = 0) v8 ==== 152+0+8 (crc 0 0 0) 0x7fdd6806c040 con 0x557d096e18c0 2023-10-05T19:28:34.631 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.622+0000 7fdd6efec700 1 -- 172.21.15.167:0/4130379316 <== osd.1 v2:172.21.15.167:6800/30035 39 ==== watch-notify(notify_complete (2) cookie 93995518298224 notify 128849018894 ret 0) v3 ==== 42+0+48 (crc 0 0 0) 0x7fdd68036dc0 con 0x557d096e18c0 2023-10-05T19:28:34.631 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.622+0000 7fdd6efec700 1 -- 172.21.15.167:0/4130379316 <== osd.1 v2:172.21.15.167:6800/30035 40 ==== osd_op_reply(116 notify.4 [notify-ack] v0'0 uv132 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7fdd6806c040 con 0x557d096e18c0 2023-10-05T19:28:34.631 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.622+0000 7fdd70cb6e40 10 removing default.rgw.meta+root+myfoo from cache 2023-10-05T19:28:34.631 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.626+0000 7fdd70cb6e40 10 distributing notification oid=default.rgw.control:notify.6 cni=[op: 1, obj: default.rgw.meta:root:myfoo, ofs0, ns] 2023-10-05T19:28:34.631 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.626+0000 7fdd70cb6e40 1 -- 172.21.15.167:0/4130379316 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:117 7.14 7:2b04a3e9:::notify.6:head [notify cookie 93995518290000 in=161b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x557d097a81a0 con 0x7fdd6400eb60 2023-10-05T19:28:34.631 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.626+0000 7fdd6efec700 1 -- 172.21.15.167:0/4130379316 <== osd.0 v2:172.21.15.118:6800/27690 37 ==== watch-notify(notify (1) cookie 93995517739696 notify 128849018892 ret 0) v3 ==== 191+0+0 (crc 0 0 0) 0x7fdd68036dc0 con 0x7fdd6400eb60 2023-10-05T19:28:34.631 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.626+0000 7fdd5f7fe700 10 rgw watcher librados: RGWWatcher::handle_notify() notify_id 128849018892 cookie 93995517739696 notifier 4491 bl.length()=149 2023-10-05T19:28:34.631 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.626+0000 7fdd5f7fe700 1 -- 172.21.15.167:0/4130379316 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:118 7.14 7:2b04a3e9:::notify.6:head [notify-ack in=20b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x7fdd54003d90 con 0x7fdd6400eb60 2023-10-05T19:28:34.631 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.626+0000 7fdd6efec700 1 -- 172.21.15.167:0/4130379316 <== osd.0 v2:172.21.15.118:6800/27690 38 ==== osd_op_reply(117 notify.6 [notify cookie 93995518290000 out=8b] v0'0 uv132 ondisk = 0) v8 ==== 152+0+8 (crc 0 0 0) 0x7fdd6806c040 con 0x7fdd6400eb60 2023-10-05T19:28:34.632 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.626+0000 7fdd6efec700 1 -- 172.21.15.167:0/4130379316 <== osd.0 v2:172.21.15.118:6800/27690 39 ==== osd_op_reply(118 notify.6 [notify-ack] v0'0 uv132 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7fdd6806c040 con 0x7fdd6400eb60 2023-10-05T19:28:34.632 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.626+0000 7fdd6efec700 1 -- 172.21.15.167:0/4130379316 <== osd.0 v2:172.21.15.118:6800/27690 40 ==== watch-notify(notify_complete (2) cookie 93995518290000 notify 128849018892 ret 0) v3 ==== 42+0+48 (crc 0 0 0) 0x7fdd680635e0 con 0x7fdd6400eb60 2023-10-05T19:28:34.632 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.626+0000 7fdd70cb6e40 1 -- 172.21.15.167:0/4130379316 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:119 8.9 8:91b71841:root::myfoo:head [call version.inc in=38b,delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x557d097a7d50 con 0x7fdd60007030 2023-10-05T19:28:34.635 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.630+0000 7fdd6dfea700 1 -- 172.21.15.167:0/4130379316 <== osd.2 v2:172.21.15.167:6816/30037 20 ==== osd_op_reply(119 myfoo [call,delete] v30'25 uv25 ondisk = 0) v8 ==== 191+0+0 (crc 0 0 0) 0x7fdd64006070 con 0x7fdd60007030 2023-10-05T19:28:34.635 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.630+0000 7fdd70cb6e40 10 removing default.rgw.meta+root+.bucket.meta.myfoo:9a9df965-6612-45d4-a45f-6e3c889bacc2.4223.1 from cache 2023-10-05T19:28:34.635 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.630+0000 7fdd70cb6e40 10 distributing notification oid=default.rgw.control:notify.7 cni=[op: 1, obj: default.rgw.meta:root:.bucket.meta.myfoo:9a9df965-6612-45d4-a45f-6e3c889bacc2.4223.1, ofs0, ns] 2023-10-05T19:28:34.635 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.630+0000 7fdd70cb6e40 1 -- 172.21.15.167:0/4130379316 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:120 7.9 7:93e5b521:::notify.7:head [notify cookie 93995518269920 in=218b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x557d097a6d40 con 0x7fdd6400eb60 2023-10-05T19:28:34.635 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.630+0000 7fdd6efec700 1 -- 172.21.15.167:0/4130379316 <== osd.0 v2:172.21.15.118:6800/27690 41 ==== watch-notify(notify (1) cookie 93995517736976 notify 128849018893 ret 0) v3 ==== 248+0+0 (crc 0 0 0) 0x7fdd680635e0 con 0x7fdd6400eb60 2023-10-05T19:28:34.636 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.630+0000 7fdd5ffff700 10 rgw watcher librados: RGWWatcher::handle_notify() notify_id 128849018893 cookie 93995517736976 notifier 4491 bl.length()=206 2023-10-05T19:28:34.636 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.630+0000 7fdd6efec700 1 -- 172.21.15.167:0/4130379316 <== osd.0 v2:172.21.15.118:6800/27690 42 ==== osd_op_reply(120 notify.7 [notify cookie 93995518269920 out=8b] v0'0 uv132 ondisk = 0) v8 ==== 152+0+8 (crc 0 0 0) 0x7fdd6806c040 con 0x7fdd6400eb60 2023-10-05T19:28:34.636 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.630+0000 7fdd5ffff700 1 -- 172.21.15.167:0/4130379316 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:121 7.9 7:93e5b521:::notify.7:head [notify-ack in=20b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x7fdcbc003460 con 0x7fdd6400eb60 2023-10-05T19:28:34.636 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.630+0000 7fdd6efec700 1 -- 172.21.15.167:0/4130379316 <== osd.0 v2:172.21.15.118:6800/27690 43 ==== osd_op_reply(121 notify.7 [notify-ack] v0'0 uv132 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7fdd6806c040 con 0x7fdd6400eb60 2023-10-05T19:28:34.637 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.630+0000 7fdd6efec700 1 -- 172.21.15.167:0/4130379316 <== osd.0 v2:172.21.15.118:6800/27690 44 ==== watch-notify(notify_complete (2) cookie 93995518269920 notify 128849018893 ret 0) v3 ==== 42+0+48 (crc 0 0 0) 0x7fdd68063810 con 0x7fdd6400eb60 2023-10-05T19:28:34.637 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.630+0000 7fdd70cb6e40 1 -- 172.21.15.167:0/4130379316 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:122 8.11 8:8bb4ed75:root::.bucket.meta.myfoo%3a9a9df965-6612-45d4-a45f-6e3c889bacc2.4223.1:head [call version.inc in=38b,delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x557d097a6850 con 0x557d097164d0 2023-10-05T19:28:34.639 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.634+0000 7fdd6e7eb700 1 -- 172.21.15.167:0/4130379316 <== osd.3 v2:172.21.15.167:6808/30036 26 ==== osd_op_reply(122 .bucket.meta.myfoo:9a9df965-6612-45d4-a45f-6e3c889bacc2.4223.1 [call,delete] v30'4 uv4 ondisk = 0) v8 ==== 248+0+0 (crc 0 0 0) 0x7fdd6005d040 con 0x557d097164d0 2023-10-05T19:28:34.640 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.634+0000 7fdd70cb6e40 1 -- 172.21.15.167:0/4130379316 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:123 7.1f 7:f95f44c2:::notify.0:head [watch unwatch cookie 93995517717120] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x557d096f67d0 con 0x7fdd60007030 2023-10-05T19:28:34.643 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.638+0000 7fdd6dfea700 1 -- 172.21.15.167:0/4130379316 <== osd.2 v2:172.21.15.167:6816/30037 21 ==== osd_op_reply(123 notify.0 [watch unwatch cookie 93995517717120] v30'134 uv132 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7fdd64006070 con 0x7fdd60007030 2023-10-05T19:28:34.643 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.638+0000 7fdd70cb6e40 20 remove_watcher() i=0 2023-10-05T19:28:34.643 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.638+0000 7fdd70cb6e40 2 removed watcher, disabling cache 2023-10-05T19:28:34.643 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.638+0000 7fdd70cb6e40 1 -- 172.21.15.167:0/4130379316 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:124 7.0 7:05bf5b68:::notify.1:head [watch unwatch cookie 93995517723584] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x557d096f67d0 con 0x557d096e18c0 2023-10-05T19:28:34.645 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.638+0000 7fdd6efec700 1 -- 172.21.15.167:0/4130379316 <== osd.1 v2:172.21.15.167:6800/30035 41 ==== osd_op_reply(124 notify.1 [watch unwatch cookie 93995517723584] v30'134 uv132 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7fdd6806c040 con 0x557d096e18c0 2023-10-05T19:28:34.645 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.638+0000 7fdd70cb6e40 20 remove_watcher() i=1 2023-10-05T19:28:34.646 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.638+0000 7fdd70cb6e40 1 -- 172.21.15.167:0/4130379316 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:125 7.15 7:a93a5511:::notify.2:head [watch unwatch cookie 93995517724688] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x557d096f67d0 con 0x557d097164d0 2023-10-05T19:28:34.648 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.642+0000 7fdd6e7eb700 1 -- 172.21.15.167:0/4130379316 <== osd.3 v2:172.21.15.167:6808/30036 27 ==== osd_op_reply(125 notify.2 [watch unwatch cookie 93995517724688] v30'134 uv132 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7fdd6005d040 con 0x557d097164d0 2023-10-05T19:28:34.648 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.642+0000 7fdd70cb6e40 20 remove_watcher() i=2 2023-10-05T19:28:34.649 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.642+0000 7fdd70cb6e40 1 -- 172.21.15.167:0/4130379316 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:126 7.e 7:7759931f:::notify.3:head [watch unwatch cookie 93995517731920] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x557d096f67d0 con 0x7fdd60007030 2023-10-05T19:28:34.651 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.646+0000 7fdd6dfea700 1 -- 172.21.15.167:0/4130379316 <== osd.2 v2:172.21.15.167:6816/30037 22 ==== osd_op_reply(126 notify.3 [watch unwatch cookie 93995517731920] v30'134 uv132 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7fdd64006070 con 0x7fdd60007030 2023-10-05T19:28:34.651 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.646+0000 7fdd70cb6e40 20 remove_watcher() i=3 2023-10-05T19:28:34.651 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.646+0000 7fdd70cb6e40 1 -- 172.21.15.167:0/4130379316 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:127 7.d 7:b4812045:::notify.4:head [watch unwatch cookie 93995517725792] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x557d096f67d0 con 0x557d096e18c0 2023-10-05T19:28:34.654 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.646+0000 7fdd6efec700 1 -- 172.21.15.167:0/4130379316 <== osd.1 v2:172.21.15.167:6800/30035 42 ==== osd_op_reply(127 notify.4 [watch unwatch cookie 93995517725792] v30'134 uv132 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7fdd6806c040 con 0x557d096e18c0 2023-10-05T19:28:34.654 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.646+0000 7fdd70cb6e40 20 remove_watcher() i=4 2023-10-05T19:28:34.654 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.650+0000 7fdd70cb6e40 1 -- 172.21.15.167:0/4130379316 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:128 7.3 7:c609908c:::notify.5:head [watch unwatch cookie 93995517735872] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x557d096f67d0 con 0x7fdd6400eb60 2023-10-05T19:28:34.657 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.650+0000 7fdd6efec700 1 -- 172.21.15.167:0/4130379316 <== osd.0 v2:172.21.15.118:6800/27690 45 ==== osd_op_reply(128 notify.5 [watch unwatch cookie 93995517735872] v30'134 uv132 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7fdd6806c040 con 0x7fdd6400eb60 2023-10-05T19:28:34.657 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.650+0000 7fdd70cb6e40 20 remove_watcher() i=5 2023-10-05T19:28:34.657 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.650+0000 7fdd70cb6e40 1 -- 172.21.15.167:0/4130379316 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:129 7.14 7:2b04a3e9:::notify.6:head [watch unwatch cookie 93995517739696] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x557d096f67d0 con 0x7fdd6400eb60 2023-10-05T19:28:34.660 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.654+0000 7fdd6efec700 1 -- 172.21.15.167:0/4130379316 <== osd.0 v2:172.21.15.118:6800/27690 46 ==== osd_op_reply(129 notify.6 [watch unwatch cookie 93995517739696] v30'134 uv132 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7fdd6806c040 con 0x7fdd6400eb60 2023-10-05T19:28:34.660 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.654+0000 7fdd70cb6e40 20 remove_watcher() i=6 2023-10-05T19:28:34.660 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.654+0000 7fdd70cb6e40 1 -- 172.21.15.167:0/4130379316 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:130 7.9 7:93e5b521:::notify.7:head [watch unwatch cookie 93995517736976] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x557d096f67d0 con 0x7fdd6400eb60 2023-10-05T19:28:34.663 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.658+0000 7fdd6efec700 1 -- 172.21.15.167:0/4130379316 <== osd.0 v2:172.21.15.118:6800/27690 47 ==== osd_op_reply(130 notify.7 [watch unwatch cookie 93995517736976] v30'134 uv132 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7fdd6806c040 con 0x7fdd6400eb60 2023-10-05T19:28:34.663 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.658+0000 7fdd70cb6e40 20 remove_watcher() i=7 2023-10-05T19:28:34.665 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.658+0000 7fdd3ffdf700 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2023-10-05T19:28:34.665 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.658+0000 7fdd70cb6e40 1 -- 172.21.15.167:0/4130379316 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:131 6.4 6:22d26bf9:::data_loggenerations_metadata:head [watch unwatch cookie 93995517743568] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x557d096f67d0 con 0x557d096e18c0 2023-10-05T19:28:34.669 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.662+0000 7fdd6efec700 1 -- 172.21.15.167:0/4130379316 <== osd.1 v2:172.21.15.167:6800/30035 43 ==== osd_op_reply(131 data_loggenerations_metadata [watch unwatch cookie 93995517743568] v30'130 uv1 ondisk = 0) v8 ==== 172+0+0 (crc 0 0 0) 0x7fdd6806c040 con 0x557d096e18c0 2023-10-05T19:28:34.669 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.662+0000 7fdd70cb6e40 1 -- 172.21.15.167:0/4130379316 >> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] conn(0x7fdd6400eb60 msgr2=0x7fdd6400d8e0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-05T19:28:34.669 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.662+0000 7fdd70cb6e40 1 --2- 172.21.15.167:0/4130379316 >> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] conn(0x7fdd6400eb60 0x7fdd6400d8e0 crc :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-05T19:28:34.669 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.662+0000 7fdd70cb6e40 1 -- 172.21.15.167:0/4130379316 >> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] conn(0x557d096e18c0 msgr2=0x557d097126f0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-05T19:28:34.669 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.662+0000 7fdd70cb6e40 1 --2- 172.21.15.167:0/4130379316 >> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] conn(0x557d096e18c0 0x557d097126f0 crc :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-05T19:28:34.669 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.662+0000 7fdd70cb6e40 1 -- 172.21.15.167:0/4130379316 >> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] conn(0x7fdd60007030 msgr2=0x7fdd6000c9c0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-05T19:28:34.669 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.662+0000 7fdd70cb6e40 1 --2- 172.21.15.167:0/4130379316 >> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] conn(0x7fdd60007030 0x7fdd6000c9c0 crc :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-05T19:28:34.669 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.662+0000 7fdd70cb6e40 1 -- 172.21.15.167:0/4130379316 >> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] conn(0x557d097164d0 msgr2=0x557d097188b0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-05T19:28:34.669 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.662+0000 7fdd70cb6e40 1 --2- 172.21.15.167:0/4130379316 >> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] conn(0x557d097164d0 0x557d097188b0 crc :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-05T19:28:34.670 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.662+0000 7fdd70cb6e40 1 -- 172.21.15.167:0/4130379316 >> [v2:172.21.15.167:6824/29889,v1:172.21.15.167:6825/29889] conn(0x7fdd50030bf0 msgr2=0x7fdd500330a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-05T19:28:34.670 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.662+0000 7fdd70cb6e40 1 --2- 172.21.15.167:0/4130379316 >> [v2:172.21.15.167:6824/29889,v1:172.21.15.167:6825/29889] conn(0x7fdd50030bf0 0x7fdd500330a0 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7fdd64004170 tx=0x7fdd64001400 comp rx=0 tx=0).stop 2023-10-05T19:28:34.670 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.662+0000 7fdd6efec700 1 -- 172.21.15.167:0/4130379316 reap_dead start 2023-10-05T19:28:34.670 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.662+0000 7fdd70cb6e40 1 -- 172.21.15.167:0/4130379316 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x557d096e0fb0 msgr2=0x557d09706c30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-05T19:28:34.670 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.662+0000 7fdd70cb6e40 1 --2- 172.21.15.167:0/4130379316 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x557d096e0fb0 0x557d09706c30 secure :-1 s=READY pgs=401 cs=0 l=1 rev1=1 crypto rx=0x7fdd6001f040 tx=0x7fdd6001c000 comp rx=0 tx=0).stop 2023-10-05T19:28:34.670 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.662+0000 7fdd70cb6e40 1 -- 172.21.15.167:0/4130379316 shutdown_connections 2023-10-05T19:28:34.670 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.662+0000 7fdd70cb6e40 1 --2- 172.21.15.167:0/4130379316 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x557d096e0fb0 0x557d09706c30 unknown :-1 s=CLOSED pgs=401 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-05T19:28:34.670 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.662+0000 7fdd70cb6e40 1 -- 172.21.15.167:0/4130379316 >> 172.21.15.167:0/4130379316 conn(0x557d09654800 msgr2=0x557d096fa4f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-05T19:28:34.670 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.662+0000 7fdd70cb6e40 1 -- 172.21.15.167:0/4130379316 shutdown_connections 2023-10-05T19:28:34.670 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.662+0000 7fdd70cb6e40 1 -- 172.21.15.167:0/4130379316 wait complete. 2023-10-05T19:28:34.671 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.662+0000 7fdd70cb6e40 1 -- 172.21.15.167:0/1620034584 >> [v2:172.21.15.167:6824/29889,v1:172.21.15.167:6825/29889] conn(0x7fdcb4030c00 msgr2=0x7fdcb40330b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-05T19:28:34.671 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.662+0000 7fdd70cb6e40 1 --2- 172.21.15.167:0/1620034584 >> [v2:172.21.15.167:6824/29889,v1:172.21.15.167:6825/29889] conn(0x7fdcb4030c00 0x7fdcb40330b0 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x557d097209b0 tx=0x7fdd6800a280 comp rx=0 tx=0).stop 2023-10-05T19:28:34.671 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.662+0000 7fdd70cb6e40 1 -- 172.21.15.167:0/1620034584 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x557d0972eb90 msgr2=0x557d09734720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-05T19:28:34.672 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.662+0000 7fdd70cb6e40 1 --2- 172.21.15.167:0/1620034584 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x557d0972eb90 0x557d09734720 secure :-1 s=READY pgs=403 cs=0 l=1 rev1=1 crypto rx=0x7fdd6400bef0 tx=0x7fdd64030000 comp rx=0 tx=0).stop 2023-10-05T19:28:34.672 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.666+0000 7fdd70cb6e40 1 -- 172.21.15.167:0/1620034584 shutdown_connections 2023-10-05T19:28:34.672 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.666+0000 7fdd70cb6e40 1 --2- 172.21.15.167:0/1620034584 >> [v2:172.21.15.167:6824/29889,v1:172.21.15.167:6825/29889] conn(0x7fdcb4030c00 0x7fdcb40330b0 unknown :-1 s=CLOSED pgs=141 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-05T19:28:34.672 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.666+0000 7fdd70cb6e40 1 --2- 172.21.15.167:0/1620034584 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x557d0972eb90 0x557d09734720 unknown :-1 s=CLOSED pgs=403 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-05T19:28:34.672 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.666+0000 7fdd70cb6e40 1 -- 172.21.15.167:0/1620034584 >> 172.21.15.167:0/1620034584 conn(0x557d09729f50 msgr2=0x557d09727ff0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-05T19:28:34.672 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.666+0000 7fdd70cb6e40 1 -- 172.21.15.167:0/1620034584 shutdown_connections 2023-10-05T19:28:34.672 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.666+0000 7fdd70cb6e40 1 -- 172.21.15.167:0/1620034584 wait complete. 2023-10-05T19:28:34.676 INFO:tasks.util.rgw:rgwadmin: client.0 : ['metadata', 'get', 'bucket:testx/myfoo'] 2023-10-05T19:28:34.676 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'ceph', 'metadata', 'get', 'bucket:testx/myfoo'] 2023-10-05T19:28:34.676 DEBUG:teuthology.orchestra.run.smithi167:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster ceph metadata get bucket:testx/myfoo 2023-10-05T19:28:34.715 INFO:teuthology.orchestra.run.smithi167.stderr:ignoring --setuser ceph since I am not root 2023-10-05T19:28:34.715 INFO:teuthology.orchestra.run.smithi167.stderr:ignoring --setgroup ceph since I am not root 2023-10-05T19:28:34.716 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.710+0000 7f350de3de40 1 Processor -- start 2023-10-05T19:28:34.716 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.710+0000 7f350de3de40 1 -- start start 2023-10-05T19:28:34.716 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.710+0000 7f350de3de40 1 --2- >> v2:172.21.15.118:3300/0 conn(0x556db2466010 0x556db24663e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-05T19:28:34.716 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.710+0000 7f350de3de40 1 -- --> v1:172.21.15.118:6789/0 -- auth(proto 0 26 bytes epoch 0) v1 -- 0x556db23b0bd0 con 0x556db2466920 2023-10-05T19:28:34.716 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.710+0000 7f350de3de40 1 -- --> v2:172.21.15.118:3300/0 -- mon_getmap magic: 0 v1 -- 0x556db236dee0 con 0x556db2466010 2023-10-05T19:28:34.717 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.710+0000 7f350b171700 1 --1- >> v1:172.21.15.118:6789/0 conn(0x556db2466920 0x556db24673c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.118:6789/0 says I am v1:172.21.15.167:59262/0 (socket says 172.21.15.167:59262) 2023-10-05T19:28:34.717 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.710+0000 7f350b972700 1 --2- >> v2:172.21.15.118:3300/0 conn(0x556db2466010 0x556db24663e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-05T19:28:34.717 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.710+0000 7f350b171700 1 -- 172.21.15.167:0/1571246668 learned_addr learned my addr 172.21.15.167:0/1571246668 (peer_addr_for_me v1:172.21.15.167:0/0) 2023-10-05T19:28:34.718 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.710+0000 7f350a970700 1 -- 172.21.15.167:0/1571246668 <== mon.0 v1:172.21.15.118:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1927986931 0 0) 0x556db23b0bd0 con 0x556db2466920 2023-10-05T19:28:34.718 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.710+0000 7f350a970700 1 -- 172.21.15.167:0/1571246668 --> v1:172.21.15.118:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f34f8001e10 con 0x556db2466920 2023-10-05T19:28:34.718 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.710+0000 7f350b972700 1 -- 172.21.15.167:0/1571246668 >> v1:172.21.15.118:6789/0 conn(0x556db2466920 legacy=0x556db24673c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-05T19:28:34.718 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.710+0000 7f350b972700 1 -- 172.21.15.167:0/1571246668 --> v2:172.21.15.118:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x556db2467ad0 con 0x556db2466010 2023-10-05T19:28:34.719 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.714+0000 7f350b972700 1 --2- 172.21.15.167:0/1571246668 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x556db2466010 0x556db24663e0 secure :-1 s=READY pgs=405 cs=0 l=1 rev1=1 crypto rx=0x7f350400a5d0 tx=0x7f3504005a50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d61886576fd907e4 server_cookie=0 in_seq=0 out_seq=0 2023-10-05T19:28:34.719 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.714+0000 7f350a970700 1 -- 172.21.15.167:0/1571246668 <== mon.0 v2:172.21.15.118:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f3504007610 con 0x556db2466010 2023-10-05T19:28:34.719 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.714+0000 7f350a970700 1 -- 172.21.15.167:0/1571246668 <== mon.0 v2:172.21.15.118:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3504007770 con 0x556db2466010 2023-10-05T19:28:34.719 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.714+0000 7f350a970700 1 -- 172.21.15.167:0/1571246668 <== mon.0 v2:172.21.15.118:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f3504007a40 con 0x556db2466010 2023-10-05T19:28:34.720 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.714+0000 7f350de3de40 1 -- 172.21.15.167:0/1571246668 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x556db2466010 msgr2=0x556db24663e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-05T19:28:34.720 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.714+0000 7f350de3de40 1 --2- 172.21.15.167:0/1571246668 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x556db2466010 0x556db24663e0 secure :-1 s=READY pgs=405 cs=0 l=1 rev1=1 crypto rx=0x7f350400a5d0 tx=0x7f3504005a50 comp rx=0 tx=0).stop 2023-10-05T19:28:34.720 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.714+0000 7f350de3de40 1 -- 172.21.15.167:0/1571246668 shutdown_connections 2023-10-05T19:28:34.720 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.714+0000 7f350de3de40 1 --2- 172.21.15.167:0/1571246668 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x556db2466010 0x556db24663e0 unknown :-1 s=CLOSED pgs=405 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-05T19:28:34.720 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.714+0000 7f350de3de40 1 -- 172.21.15.167:0/1571246668 >> 172.21.15.167:0/1571246668 conn(0x556db23d9860 msgr2=0x556db23d9c60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-05T19:28:34.720 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.714+0000 7f350de3de40 1 -- 172.21.15.167:0/1571246668 shutdown_connections 2023-10-05T19:28:34.720 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.714+0000 7f350de3de40 1 -- 172.21.15.167:0/1571246668 wait complete. 2023-10-05T19:28:34.721 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.714+0000 7f350de3de40 1 Processor -- start 2023-10-05T19:28:34.721 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.714+0000 7f350de3de40 1 -- start start 2023-10-05T19:28:34.721 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.714+0000 7f350de3de40 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x556db2466010 0x556db2484f70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-05T19:28:34.721 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.714+0000 7f350de3de40 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x556db2331e30 con 0x556db2466010 2023-10-05T19:28:34.722 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.714+0000 7f350b972700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x556db2466010 0x556db2484f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-05T19:28:34.722 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.714+0000 7f350b972700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x556db2466010 0x556db2484f70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.118:3300/0 says I am v2:172.21.15.167:41520/0 (socket says 172.21.15.167:41520) 2023-10-05T19:28:34.722 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.714+0000 7f350b972700 1 -- 172.21.15.167:0/619292348 learned_addr learned my addr 172.21.15.167:0/619292348 (peer_addr_for_me v2:172.21.15.167:0/0) 2023-10-05T19:28:34.722 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.714+0000 7f350b972700 1 -- 172.21.15.167:0/619292348 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3504005cf0 con 0x556db2466010 2023-10-05T19:28:34.723 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.714+0000 7f350b972700 1 --2- 172.21.15.167:0/619292348 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x556db2466010 0x556db2484f70 secure :-1 s=READY pgs=406 cs=0 l=1 rev1=1 crypto rx=0x7f3504000c00 tx=0x7f3504014000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5e24a322761336a3 server_cookie=0 in_seq=0 out_seq=0 2023-10-05T19:28:34.723 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.718+0000 7f350996e700 1 -- 172.21.15.167:0/619292348 <== mon.0 v2:172.21.15.118:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f35040078e0 con 0x556db2466010 2023-10-05T19:28:34.723 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.718+0000 7f350996e700 1 -- 172.21.15.167:0/619292348 <== mon.0 v2:172.21.15.118:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3504007a40 con 0x556db2466010 2023-10-05T19:28:34.723 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.718+0000 7f350996e700 1 -- 172.21.15.167:0/619292348 <== mon.0 v2:172.21.15.118:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f3504007bd0 con 0x556db2466010 2023-10-05T19:28:34.723 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.718+0000 7f350de3de40 1 -- 172.21.15.167:0/619292348 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x556db2466010 msgr2=0x556db2484f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-05T19:28:34.723 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.718+0000 7f350de3de40 1 --2- 172.21.15.167:0/619292348 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x556db2466010 0x556db2484f70 secure :-1 s=READY pgs=406 cs=0 l=1 rev1=1 crypto rx=0x7f3504000c00 tx=0x7f3504014000 comp rx=0 tx=0).stop 2023-10-05T19:28:34.724 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.718+0000 7f350de3de40 1 -- 172.21.15.167:0/619292348 shutdown_connections 2023-10-05T19:28:34.724 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.718+0000 7f350de3de40 1 --2- 172.21.15.167:0/619292348 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x556db2466010 0x556db2484f70 unknown :-1 s=CLOSED pgs=406 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-05T19:28:34.724 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.718+0000 7f350de3de40 1 -- 172.21.15.167:0/619292348 >> 172.21.15.167:0/619292348 conn(0x556db23d9860 msgr2=0x556db2480b30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-05T19:28:34.724 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.718+0000 7f350de3de40 1 -- 172.21.15.167:0/619292348 shutdown_connections 2023-10-05T19:28:34.724 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.718+0000 7f350de3de40 1 -- 172.21.15.167:0/619292348 wait complete. 2023-10-05T19:28:34.725 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.718+0000 7f350de3de40 1 Processor -- start 2023-10-05T19:28:34.725 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.718+0000 7f350de3de40 1 -- start start 2023-10-05T19:28:34.725 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.718+0000 7f350de3de40 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x556db2466010 0x556db248bc30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-05T19:28:34.725 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.718+0000 7f350de3de40 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x556db2304ed0 con 0x556db2466010 2023-10-05T19:28:34.725 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.718+0000 7f350b972700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x556db2466010 0x556db248bc30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-05T19:28:34.725 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.718+0000 7f350b972700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x556db2466010 0x556db248bc30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.118:3300/0 says I am v2:172.21.15.167:41524/0 (socket says 172.21.15.167:41524) 2023-10-05T19:28:34.725 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.718+0000 7f350b972700 1 -- 172.21.15.167:0/3505225897 learned_addr learned my addr 172.21.15.167:0/3505225897 (peer_addr_for_me v2:172.21.15.167:0/0) 2023-10-05T19:28:34.726 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.718+0000 7f350b972700 1 -- 172.21.15.167:0/3505225897 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x556db2467ad0 con 0x556db2466010 2023-10-05T19:28:34.726 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.718+0000 7f350b972700 1 --2- 172.21.15.167:0/3505225897 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x556db2466010 0x556db248bc30 secure :-1 s=READY pgs=407 cs=0 l=1 rev1=1 crypto rx=0x7f350401f040 tx=0x7f350401c000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-05T19:28:34.726 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.718+0000 7f3503fff700 1 -- 172.21.15.167:0/3505225897 <== mon.0 v2:172.21.15.118:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f3504007e50 con 0x556db2466010 2023-10-05T19:28:34.727 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.718+0000 7f3503fff700 1 -- 172.21.15.167:0/3505225897 <== mon.0 v2:172.21.15.118:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3504029070 con 0x556db2466010 2023-10-05T19:28:34.727 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.718+0000 7f350de3de40 1 -- 172.21.15.167:0/3505225897 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x556db248c170 con 0x556db2466010 2023-10-05T19:28:34.727 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.718+0000 7f3503fff700 1 -- 172.21.15.167:0/3505225897 <== mon.0 v2:172.21.15.118:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f3504030480 con 0x556db2466010 2023-10-05T19:28:34.727 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.718+0000 7f350de3de40 1 -- 172.21.15.167:0/3505225897 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x556db248c5b0 con 0x556db2466010 2023-10-05T19:28:34.728 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.722+0000 7f350de3de40 20 rados->read ofs=0 len=0 2023-10-05T19:28:34.728 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.722+0000 7f3503fff700 1 -- 172.21.15.167:0/3505225897 <== mon.0 v2:172.21.15.118:3300/0 4 ==== mgrmap(e 5) v1 ==== 40329+0+0 (secure 0 0 0) 0x7f3504012400 con 0x556db2466010 2023-10-05T19:28:34.728 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.722+0000 7f3503fff700 1 --2- 172.21.15.167:0/3505225897 >> [v2:172.21.15.167:6824/29889,v1:172.21.15.167:6825/29889] conn(0x7f34ec030c40 0x7f34ec0330f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-05T19:28:34.728 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.722+0000 7f3503fff700 1 -- 172.21.15.167:0/3505225897 <== mon.0 v2:172.21.15.118:3300/0 5 ==== osd_map(30..30 src has 1..30) v4 ==== 6057+0+0 (secure 0 0 0) 0x7f350402b070 con 0x556db2466010 2023-10-05T19:28:34.729 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.722+0000 7f350b171700 1 --2- 172.21.15.167:0/3505225897 >> [v2:172.21.15.167:6824/29889,v1:172.21.15.167:6825/29889] conn(0x7f34ec030c40 0x7f34ec0330f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-05T19:28:34.729 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.722+0000 7f350de3de40 1 --2- 172.21.15.167:0/3505225897 >> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] conn(0x556db2466920 0x556db24976c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-05T19:28:34.729 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.722+0000 7f350de3de40 1 -- 172.21.15.167:0/3505225897 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:1 5.12 5:49953fa1:::default.realm:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x556db2497d30 con 0x556db2466920 2023-10-05T19:28:34.729 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.722+0000 7f350c173700 1 --2- 172.21.15.167:0/3505225897 >> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] conn(0x556db2466920 0x556db24976c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-05T19:28:34.729 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.722+0000 7f350c173700 1 --2- 172.21.15.167:0/3505225897 >> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] conn(0x556db2466920 0x556db24976c0 crc :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-05T19:28:34.731 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.722+0000 7f350b171700 1 --2- 172.21.15.167:0/3505225897 >> [v2:172.21.15.167:6824/29889,v1:172.21.15.167:6825/29889] conn(0x7f34ec030c40 0x7f34ec0330f0 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7f34f40044f0 tx=0x7f34f4006690 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-05T19:28:34.731 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.722+0000 7f350c173700 1 -- 172.21.15.167:0/3505225897 <== osd.1 v2:172.21.15.167:6800/30035 1 ==== osd_op_reply(1 default.realm [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 ==== 157+0+0 (crc 0 0 0) 0x7f34fc00a500 con 0x556db2466920 2023-10-05T19:28:34.731 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.722+0000 7f350de3de40 20 rados_obj.operate() r=-2 bl.length=0 2023-10-05T19:28:34.731 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.722+0000 7f350de3de40 20 realm 2023-10-05T19:28:34.731 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.722+0000 7f350de3de40 20 rados->read ofs=0 len=0 2023-10-05T19:28:34.731 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.722+0000 7f350de3de40 1 -- 172.21.15.167:0/3505225897 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:2 5.12 5:49953fa1:::default.realm:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x556db2494d20 con 0x556db2466920 2023-10-05T19:28:34.731 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.726+0000 7f350c173700 1 -- 172.21.15.167:0/3505225897 <== osd.1 v2:172.21.15.167:6800/30035 2 ==== osd_op_reply(2 default.realm [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 ==== 157+0+0 (crc 0 0 0) 0x7f34fc00a500 con 0x556db2466920 2023-10-05T19:28:34.731 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.726+0000 7f350de3de40 20 rados_obj.operate() r=-2 bl.length=0 2023-10-05T19:28:34.732 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.726+0000 7f350de3de40 4 RGWPeriod::init failed to init realm id : (2) No such file or directory 2023-10-05T19:28:34.732 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.726+0000 7f350de3de40 20 rados->read ofs=0 len=0 2023-10-05T19:28:34.732 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.726+0000 7f350de3de40 1 -- 172.21.15.167:0/3505225897 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:3 5.12 5:49953fa1:::default.realm:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x556db2498810 con 0x556db2466920 2023-10-05T19:28:34.732 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.726+0000 7f350c173700 1 -- 172.21.15.167:0/3505225897 <== osd.1 v2:172.21.15.167:6800/30035 3 ==== osd_op_reply(3 default.realm [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 ==== 157+0+0 (crc 0 0 0) 0x7f34fc00a500 con 0x556db2466920 2023-10-05T19:28:34.732 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.726+0000 7f350de3de40 20 rados_obj.operate() r=-2 bl.length=0 2023-10-05T19:28:34.732 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.726+0000 7f350de3de40 20 rados->read ofs=0 len=0 2023-10-05T19:28:34.732 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.726+0000 7f350de3de40 1 -- 172.21.15.167:0/3505225897 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:4 5.f 5:f43b6ece:::zone_names.default:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x556db24983f0 con 0x556db2466920 2023-10-05T19:28:34.732 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.726+0000 7f350c173700 1 -- 172.21.15.167:0/3505225897 <== osd.1 v2:172.21.15.167:6800/30035 4 ==== osd_op_reply(4 zone_names.default [read 0~46 out=46b] v0'0 uv1 ondisk = 0) v8 ==== 162+0+46 (crc 0 0 0) 0x7f34fc00a500 con 0x556db2466920 2023-10-05T19:28:34.732 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.726+0000 7f350de3de40 20 rados_obj.operate() r=0 bl.length=46 2023-10-05T19:28:34.732 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.726+0000 7f350de3de40 20 rados->read ofs=0 len=0 2023-10-05T19:28:34.733 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.726+0000 7f350de3de40 1 -- 172.21.15.167:0/3505225897 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:5 5.1d 5:b8374eb4:::zone_info.9a9df965-6612-45d4-a45f-6e3c889bacc2:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x556db2499280 con 0x556db2466920 2023-10-05T19:28:34.733 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.726+0000 7f350c173700 1 -- 172.21.15.167:0/3505225897 <== osd.1 v2:172.21.15.167:6800/30035 5 ==== osd_op_reply(5 zone_info.9a9df965-6612-45d4-a45f-6e3c889bacc2 [read 0~874 out=874b] v0'0 uv2 ondisk = 0) v8 ==== 190+0+874 (crc 0 0 0) 0x7f34fc00a500 con 0x556db2466920 2023-10-05T19:28:34.733 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.726+0000 7f350de3de40 20 rados_obj.operate() r=0 bl.length=874 2023-10-05T19:28:34.733 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.726+0000 7f350de3de40 20 searching for the correct realm 2023-10-05T19:28:34.734 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.726+0000 7f350de3de40 1 --2- 172.21.15.167:0/3505225897 >> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] conn(0x556db249b0d0 0x556db249d4b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-05T19:28:34.734 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.726+0000 7f350de3de40 1 -- 172.21.15.167:0/3505225897 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:6 5.0 5:00000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x556db249d9f0 con 0x556db249b0d0 2023-10-05T19:28:34.734 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.726+0000 7f350b972700 1 --2- 172.21.15.167:0/3505225897 >> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] conn(0x556db249b0d0 0x556db249d4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-05T19:28:34.734 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.726+0000 7f350b972700 1 --2- 172.21.15.167:0/3505225897 >> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] conn(0x556db249b0d0 0x556db249d4b0 crc :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-05T19:28:34.735 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.730+0000 7f350b972700 1 -- 172.21.15.167:0/3505225897 <== osd.3 v2:172.21.15.167:6808/30036 1 ==== osd_op_reply(6 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f350401c7f0 con 0x556db249b0d0 2023-10-05T19:28:34.735 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.730+0000 7f350b972700 1 --2- 172.21.15.167:0/3505225897 >> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] conn(0x7f3504007030 0x7f350400c9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-05T19:28:34.736 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.730+0000 7f350b972700 1 -- 172.21.15.167:0/3505225897 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:7 5.10 5:08000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x556db249d9f0 con 0x7f3504007030 2023-10-05T19:28:34.736 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.730+0000 7f350b171700 1 --2- 172.21.15.167:0/3505225897 >> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] conn(0x7f3504007030 0x7f350400c9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-05T19:28:34.736 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.730+0000 7f350b171700 1 --2- 172.21.15.167:0/3505225897 >> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] conn(0x7f3504007030 0x7f350400c9c0 crc :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-05T19:28:34.737 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.730+0000 7f350b171700 1 -- 172.21.15.167:0/3505225897 <== osd.2 v2:172.21.15.167:6816/30037 1 ==== osd_op_reply(7 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f34f4004520 con 0x7f3504007030 2023-10-05T19:28:34.737 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.730+0000 7f350b171700 1 --2- 172.21.15.167:0/3505225897 >> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] conn(0x7f34f4011040 0x7f34f4014950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-05T19:28:34.737 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.730+0000 7f350b171700 1 -- 172.21.15.167:0/3505225897 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:8 5.8 5:10000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x556db249d9f0 con 0x7f34f4011040 2023-10-05T19:28:34.737 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.730+0000 7f350c173700 1 --2- 172.21.15.167:0/3505225897 >> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] conn(0x7f34f4011040 0x7f34f4014950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-05T19:28:34.738 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.734+0000 7f350c173700 1 --2- 172.21.15.167:0/3505225897 >> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] conn(0x7f34f4011040 0x7f34f4014950 crc :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-05T19:28:34.739 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.734+0000 7f350c173700 1 -- 172.21.15.167:0/3505225897 <== osd.0 v2:172.21.15.118:6800/27690 1 ==== osd_op_reply(8 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f34fc014040 con 0x7f34f4011040 2023-10-05T19:28:34.739 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.734+0000 7f350c173700 1 -- 172.21.15.167:0/3505225897 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:9 5.18 5:18000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x556db249d9f0 con 0x556db2466920 2023-10-05T19:28:34.739 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.734+0000 7f350c173700 1 -- 172.21.15.167:0/3505225897 <== osd.1 v2:172.21.15.167:6800/30035 6 ==== osd_op_reply(9 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f34fc014040 con 0x556db2466920 2023-10-05T19:28:34.740 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.734+0000 7f350c173700 1 -- 172.21.15.167:0/3505225897 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:10 5.4 5:20000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x556db249d9f0 con 0x556db2466920 2023-10-05T19:28:34.740 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.734+0000 7f350c173700 1 -- 172.21.15.167:0/3505225897 <== osd.1 v2:172.21.15.167:6800/30035 7 ==== osd_op_reply(10 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f34fc014040 con 0x556db2466920 2023-10-05T19:28:34.740 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.734+0000 7f350c173700 1 -- 172.21.15.167:0/3505225897 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:11 5.14 5:28000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x556db249d9f0 con 0x7f34f4011040 2023-10-05T19:28:34.741 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.734+0000 7f350c173700 1 -- 172.21.15.167:0/3505225897 <== osd.0 v2:172.21.15.118:6800/27690 2 ==== osd_op_reply(11 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f34fc014040 con 0x7f34f4011040 2023-10-05T19:28:34.741 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.734+0000 7f350c173700 1 -- 172.21.15.167:0/3505225897 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:12 5.c 5:30000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x556db249d9f0 con 0x7f34f4011040 2023-10-05T19:28:34.742 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.738+0000 7f350c173700 1 -- 172.21.15.167:0/3505225897 <== osd.0 v2:172.21.15.118:6800/27690 3 ==== osd_op_reply(12 [pgnls start_epoch 30 out=112b] v23'1 uv1 ondisk = 1) v8 ==== 144+0+112 (crc 0 0 0) 0x7f34fc014040 con 0x7f34f4011040 2023-10-05T19:28:34.742 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.738+0000 7f350c173700 1 -- 172.21.15.167:0/3505225897 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:13 5.1c 5:38000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x556db249d9f0 con 0x7f3504007030 2023-10-05T19:28:34.743 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.738+0000 7f350b171700 1 -- 172.21.15.167:0/3505225897 <== osd.2 v2:172.21.15.167:6816/30037 2 ==== osd_op_reply(13 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f34f4004520 con 0x7f3504007030 2023-10-05T19:28:34.743 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.738+0000 7f350b171700 1 -- 172.21.15.167:0/3505225897 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:14 5.2 5:40000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x556db249d9f0 con 0x556db249b0d0 2023-10-05T19:28:34.744 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.738+0000 7f350b972700 1 -- 172.21.15.167:0/3505225897 <== osd.3 v2:172.21.15.167:6808/30036 2 ==== osd_op_reply(14 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f350401c7f0 con 0x556db249b0d0 2023-10-05T19:28:34.744 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.738+0000 7f350b972700 1 -- 172.21.15.167:0/3505225897 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:15 5.12 5:48000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x556db249d9f0 con 0x556db2466920 2023-10-05T19:28:34.744 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.738+0000 7f350c173700 1 -- 172.21.15.167:0/3505225897 <== osd.1 v2:172.21.15.167:6800/30035 8 ==== osd_op_reply(15 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f34fc014040 con 0x556db2466920 2023-10-05T19:28:34.744 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.738+0000 7f350c173700 1 -- 172.21.15.167:0/3505225897 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:16 5.a 5:50000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x556db249d9f0 con 0x7f34f4011040 2023-10-05T19:28:34.745 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.738+0000 7f350c173700 1 -- 172.21.15.167:0/3505225897 <== osd.0 v2:172.21.15.118:6800/27690 4 ==== osd_op_reply(16 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f34fc014040 con 0x7f34f4011040 2023-10-05T19:28:34.746 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.738+0000 7f350c173700 1 -- 172.21.15.167:0/3505225897 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:17 5.1a 5:58000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x556db249d9f0 con 0x556db2466920 2023-10-05T19:28:34.746 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.742+0000 7f350c173700 1 -- 172.21.15.167:0/3505225897 <== osd.1 v2:172.21.15.167:6800/30035 9 ==== osd_op_reply(17 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f34fc014040 con 0x556db2466920 2023-10-05T19:28:34.746 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.742+0000 7f350c173700 1 -- 172.21.15.167:0/3505225897 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:18 5.6 5:60000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x556db249d9f0 con 0x7f3504007030 2023-10-05T19:28:34.747 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.742+0000 7f350b171700 1 -- 172.21.15.167:0/3505225897 <== osd.2 v2:172.21.15.167:6816/30037 3 ==== osd_op_reply(18 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f34f4004520 con 0x7f3504007030 2023-10-05T19:28:34.747 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.742+0000 7f350b171700 1 -- 172.21.15.167:0/3505225897 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:19 5.16 5:68000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x556db249d9f0 con 0x556db249b0d0 2023-10-05T19:28:34.748 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.742+0000 7f350b972700 1 -- 172.21.15.167:0/3505225897 <== osd.3 v2:172.21.15.167:6808/30036 3 ==== osd_op_reply(19 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f350401c7f0 con 0x556db249b0d0 2023-10-05T19:28:34.748 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.742+0000 7f350b972700 1 -- 172.21.15.167:0/3505225897 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:20 5.e 5:70000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x556db249d9f0 con 0x556db249b0d0 2023-10-05T19:28:34.748 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.742+0000 7f350b972700 1 -- 172.21.15.167:0/3505225897 <== osd.3 v2:172.21.15.167:6808/30036 4 ==== osd_op_reply(20 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f350401c7f0 con 0x556db249b0d0 2023-10-05T19:28:34.748 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.742+0000 7f350b972700 1 -- 172.21.15.167:0/3505225897 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:21 5.1e 5:78000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x556db249d9f0 con 0x7f34f4011040 2023-10-05T19:28:34.750 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.742+0000 7f350c173700 1 -- 172.21.15.167:0/3505225897 <== osd.0 v2:172.21.15.118:6800/27690 5 ==== osd_op_reply(21 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f34fc022040 con 0x7f34f4011040 2023-10-05T19:28:34.750 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.742+0000 7f350c173700 1 -- 172.21.15.167:0/3505225897 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:22 5.1 5:80000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x556db249d9f0 con 0x556db249b0d0 2023-10-05T19:28:34.750 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.746+0000 7f350b972700 1 -- 172.21.15.167:0/3505225897 <== osd.3 v2:172.21.15.167:6808/30036 5 ==== osd_op_reply(22 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f350401c7f0 con 0x556db249b0d0 2023-10-05T19:28:34.750 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.746+0000 7f350b972700 1 -- 172.21.15.167:0/3505225897 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:23 5.11 5:88000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x556db249d9f0 con 0x556db249b0d0 2023-10-05T19:28:34.751 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.746+0000 7f350b972700 1 -- 172.21.15.167:0/3505225897 <== osd.3 v2:172.21.15.167:6808/30036 6 ==== osd_op_reply(23 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f350401c7f0 con 0x556db249b0d0 2023-10-05T19:28:34.751 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.746+0000 7f350b972700 1 -- 172.21.15.167:0/3505225897 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:24 5.9 5:90000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x556db249d9f0 con 0x556db2466920 2023-10-05T19:28:34.752 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.746+0000 7f350c173700 1 -- 172.21.15.167:0/3505225897 <== osd.1 v2:172.21.15.167:6800/30035 10 ==== osd_op_reply(24 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f34fc022040 con 0x556db2466920 2023-10-05T19:28:34.752 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.746+0000 7f350c173700 1 -- 172.21.15.167:0/3505225897 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:25 5.19 5:98000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x556db249d9f0 con 0x556db2466920 2023-10-05T19:28:34.753 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.746+0000 7f350c173700 1 -- 172.21.15.167:0/3505225897 <== osd.1 v2:172.21.15.167:6800/30035 11 ==== osd_op_reply(25 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f34fc022040 con 0x556db2466920 2023-10-05T19:28:34.753 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.746+0000 7f350c173700 1 -- 172.21.15.167:0/3505225897 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:26 5.5 5:a0000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x556db249d9f0 con 0x7f34f4011040 2023-10-05T19:28:34.754 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.750+0000 7f350c173700 1 -- 172.21.15.167:0/3505225897 <== osd.0 v2:172.21.15.118:6800/27690 6 ==== osd_op_reply(26 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f34fc022040 con 0x7f34f4011040 2023-10-05T19:28:34.754 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.750+0000 7f350c173700 1 -- 172.21.15.167:0/3505225897 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:27 5.15 5:a8000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x556db249d9f0 con 0x556db249b0d0 2023-10-05T19:28:34.755 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.750+0000 7f350b972700 1 -- 172.21.15.167:0/3505225897 <== osd.3 v2:172.21.15.167:6808/30036 7 ==== osd_op_reply(27 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f350401c7f0 con 0x556db249b0d0 2023-10-05T19:28:34.755 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.750+0000 7f350b972700 1 -- 172.21.15.167:0/3505225897 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:28 5.d 5:b0000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x556db249d9f0 con 0x7f3504007030 2023-10-05T19:28:34.756 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.750+0000 7f350b171700 1 -- 172.21.15.167:0/3505225897 <== osd.2 v2:172.21.15.167:6816/30037 4 ==== osd_op_reply(28 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f34f4004520 con 0x7f3504007030 2023-10-05T19:28:34.756 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.750+0000 7f350b171700 1 -- 172.21.15.167:0/3505225897 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:29 5.1d 5:b8000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x556db249d9f0 con 0x556db2466920 2023-10-05T19:28:34.757 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.750+0000 7f350c173700 1 -- 172.21.15.167:0/3505225897 <== osd.1 v2:172.21.15.167:6800/30035 12 ==== osd_op_reply(29 [pgnls start_epoch 30 out=107b] v30'2 uv2 ondisk = 1) v8 ==== 144+0+107 (crc 0 0 0) 0x7f34fc022040 con 0x556db2466920 2023-10-05T19:28:34.757 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.750+0000 7f350c173700 1 -- 172.21.15.167:0/3505225897 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:30 5.3 5:c0000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x556db249d9f0 con 0x7f34f4011040 2023-10-05T19:28:34.758 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.750+0000 7f350c173700 1 -- 172.21.15.167:0/3505225897 <== osd.0 v2:172.21.15.118:6800/27690 7 ==== osd_op_reply(30 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f34fc022040 con 0x7f34f4011040 2023-10-05T19:28:34.758 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.754+0000 7f350c173700 1 -- 172.21.15.167:0/3505225897 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:31 5.13 5:c8000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x556db249d9f0 con 0x556db249b0d0 2023-10-05T19:28:34.759 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.754+0000 7f350b972700 1 -- 172.21.15.167:0/3505225897 <== osd.3 v2:172.21.15.167:6808/30036 8 ==== osd_op_reply(31 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f350401c7f0 con 0x556db249b0d0 2023-10-05T19:28:34.759 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.754+0000 7f350b972700 1 -- 172.21.15.167:0/3505225897 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:32 5.b 5:d0000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x556db249d9f0 con 0x7f34f4011040 2023-10-05T19:28:34.760 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.754+0000 7f350c173700 1 -- 172.21.15.167:0/3505225897 <== osd.0 v2:172.21.15.118:6800/27690 8 ==== osd_op_reply(32 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f34fc022040 con 0x7f34f4011040 2023-10-05T19:28:34.760 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.754+0000 7f350c173700 1 -- 172.21.15.167:0/3505225897 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:33 5.1b 5:d8000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x556db249d9f0 con 0x7f3504007030 2023-10-05T19:28:34.761 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.754+0000 7f350b171700 1 -- 172.21.15.167:0/3505225897 <== osd.2 v2:172.21.15.167:6816/30037 5 ==== osd_op_reply(33 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f34f4004520 con 0x7f3504007030 2023-10-05T19:28:34.761 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.754+0000 7f350b171700 1 -- 172.21.15.167:0/3505225897 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:34 5.7 5:e0000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x556db249d9f0 con 0x556db249b0d0 2023-10-05T19:28:34.761 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.754+0000 7f350b972700 1 -- 172.21.15.167:0/3505225897 <== osd.3 v2:172.21.15.167:6808/30036 9 ==== osd_op_reply(34 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f350401c7f0 con 0x556db249b0d0 2023-10-05T19:28:34.761 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.754+0000 7f350b972700 1 -- 172.21.15.167:0/3505225897 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:35 5.17 5:e8000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x556db249d9f0 con 0x7f34f4011040 2023-10-05T19:28:34.762 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.758+0000 7f350c173700 1 -- 172.21.15.167:0/3505225897 <== osd.0 v2:172.21.15.118:6800/27690 9 ==== osd_op_reply(35 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f34fc022040 con 0x7f34f4011040 2023-10-05T19:28:34.762 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.758+0000 7f350c173700 1 -- 172.21.15.167:0/3505225897 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:36 5.f 5:f0000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x556db249d9f0 con 0x556db2466920 2023-10-05T19:28:34.763 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.758+0000 7f350c173700 1 -- 172.21.15.167:0/3505225897 <== osd.1 v2:172.21.15.167:6800/30035 13 ==== osd_op_reply(36 [pgnls start_epoch 30 out=115b] v23'2 uv2 ondisk = 1) v8 ==== 144+0+115 (crc 0 0 0) 0x7f34fc022040 con 0x556db2466920 2023-10-05T19:28:34.763 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.758+0000 7f350c173700 1 -- 172.21.15.167:0/3505225897 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:37 5.1f 5:f8000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x556db249d9f0 con 0x7f3504007030 2023-10-05T19:28:34.764 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.758+0000 7f350b171700 1 -- 172.21.15.167:0/3505225897 <== osd.2 v2:172.21.15.167:6816/30037 6 ==== osd_op_reply(37 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f34f4004520 con 0x7f3504007030 2023-10-05T19:28:34.765 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.758+0000 7f350de3de40 20 RGWRados::pool_iterate: got zonegroup_info.4f1f719d-9648-417b-b10c-e5d939825419 2023-10-05T19:28:34.765 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.758+0000 7f350de3de40 20 RGWRados::pool_iterate: got zone_info.9a9df965-6612-45d4-a45f-6e3c889bacc2 2023-10-05T19:28:34.765 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.758+0000 7f350de3de40 20 RGWRados::pool_iterate: got zone_names.default 2023-10-05T19:28:34.765 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.758+0000 7f350de3de40 20 RGWRados::pool_iterate: got zonegroups_names.default 2023-10-05T19:28:34.765 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.758+0000 7f350de3de40 20 rados->read ofs=0 len=0 2023-10-05T19:28:34.765 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.758+0000 7f350de3de40 1 -- 172.21.15.167:0/3505225897 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:38 5.12 5:49953fa1:::default.realm:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x556db2499e00 con 0x556db2466920 2023-10-05T19:28:34.766 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.758+0000 7f350c173700 1 -- 172.21.15.167:0/3505225897 <== osd.1 v2:172.21.15.167:6800/30035 14 ==== osd_op_reply(38 default.realm [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 ==== 157+0+0 (crc 0 0 0) 0x7f34fc022040 con 0x556db2466920 2023-10-05T19:28:34.766 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.758+0000 7f350de3de40 20 rados_obj.operate() r=-2 bl.length=0 2023-10-05T19:28:34.766 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.758+0000 7f350de3de40 20 rados->read ofs=0 len=0 2023-10-05T19:28:34.766 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.758+0000 7f350de3de40 1 -- 172.21.15.167:0/3505225897 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:39 5.f 5:f4c53578:::zonegroups_names.default:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x556db249a6f0 con 0x556db2466920 2023-10-05T19:28:34.766 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.758+0000 7f350c173700 1 -- 172.21.15.167:0/3505225897 <== osd.1 v2:172.21.15.167:6800/30035 15 ==== osd_op_reply(39 zonegroups_names.default [read 0~46 out=46b] v0'0 uv2 ondisk = 0) v8 ==== 168+0+46 (crc 0 0 0) 0x7f34fc022040 con 0x556db2466920 2023-10-05T19:28:34.767 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.762+0000 7f350de3de40 20 rados_obj.operate() r=0 bl.length=46 2023-10-05T19:28:34.767 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.762+0000 7f350de3de40 20 rados->read ofs=0 len=0 2023-10-05T19:28:34.767 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.762+0000 7f350de3de40 1 -- 172.21.15.167:0/3505225897 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:40 5.c 5:30d299a5:::zonegroup_info.4f1f719d-9648-417b-b10c-e5d939825419:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x556db2499a30 con 0x7f34f4011040 2023-10-05T19:28:34.768 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.762+0000 7f350c173700 1 -- 172.21.15.167:0/3505225897 <== osd.0 v2:172.21.15.118:6800/27690 10 ==== osd_op_reply(40 zonegroup_info.4f1f719d-9648-417b-b10c-e5d939825419 [read 0~362 out=362b] v0'0 uv1 ondisk = 0) v8 ==== 195+0+362 (crc 0 0 0) 0x7f34fc022040 con 0x7f34f4011040 2023-10-05T19:28:34.768 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.762+0000 7f350de3de40 20 rados_obj.operate() r=0 bl.length=362 2023-10-05T19:28:34.768 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.762+0000 7f350de3de40 20 rados->read ofs=0 len=0 2023-10-05T19:28:34.768 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.762+0000 7f350de3de40 1 -- 172.21.15.167:0/3505225897 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:41 5.13 5:cbbf6c60:::converted:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x556db249ddf0 con 0x556db249b0d0 2023-10-05T19:28:34.768 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.762+0000 7f350b972700 1 -- 172.21.15.167:0/3505225897 <== osd.3 v2:172.21.15.167:6808/30036 10 ==== osd_op_reply(41 converted [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 ==== 153+0+0 (crc 0 0 0) 0x7f350401c7f0 con 0x556db249b0d0 2023-10-05T19:28:34.768 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.762+0000 7f350de3de40 20 rados_obj.operate() r=-2 bl.length=0 2023-10-05T19:28:34.768 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.762+0000 7f350de3de40 20 rados->read ofs=0 len=0 2023-10-05T19:28:34.768 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.762+0000 7f350de3de40 1 -- 172.21.15.167:0/3505225897 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:42 5.12 5:49953fa1:::default.realm:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x556db249e1a0 con 0x556db2466920 2023-10-05T19:28:34.769 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.762+0000 7f350c173700 1 -- 172.21.15.167:0/3505225897 <== osd.1 v2:172.21.15.167:6800/30035 16 ==== osd_op_reply(42 default.realm [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 ==== 157+0+0 (crc 0 0 0) 0x7f34fc022040 con 0x556db2466920 2023-10-05T19:28:34.769 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.762+0000 7f350de3de40 20 rados_obj.operate() r=-2 bl.length=0 2023-10-05T19:28:34.770 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.762+0000 7f350de3de40 20 rados->read ofs=0 len=0 2023-10-05T19:28:34.770 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.762+0000 7f350de3de40 1 -- 172.21.15.167:0/3505225897 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:43 5.f 5:f4c53578:::zonegroups_names.default:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x556db249e1a0 con 0x556db2466920 2023-10-05T19:28:34.770 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.762+0000 7f350c173700 1 -- 172.21.15.167:0/3505225897 <== osd.1 v2:172.21.15.167:6800/30035 17 ==== osd_op_reply(43 zonegroups_names.default [read 0~46 out=46b] v0'0 uv2 ondisk = 0) v8 ==== 168+0+46 (crc 0 0 0) 0x7f34fc022040 con 0x556db2466920 2023-10-05T19:28:34.770 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.762+0000 7f350de3de40 20 rados_obj.operate() r=0 bl.length=46 2023-10-05T19:28:34.770 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.762+0000 7f350de3de40 1 -- 172.21.15.167:0/3505225897 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:44 5.0 5:00000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x556db249e1a0 con 0x556db249b0d0 2023-10-05T19:28:34.772 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.766+0000 7f350b972700 1 -- 172.21.15.167:0/3505225897 <== osd.3 v2:172.21.15.167:6808/30036 11 ==== osd_op_reply(44 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f350401c7f0 con 0x556db249b0d0 2023-10-05T19:28:34.772 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.766+0000 7f350b972700 1 -- 172.21.15.167:0/3505225897 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:45 5.10 5:08000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x556db249e1a0 con 0x7f3504007030 2023-10-05T19:28:34.773 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.766+0000 7f350b171700 1 -- 172.21.15.167:0/3505225897 <== osd.2 v2:172.21.15.167:6816/30037 7 ==== osd_op_reply(45 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f34f4004520 con 0x7f3504007030 2023-10-05T19:28:34.773 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.766+0000 7f350b171700 1 -- 172.21.15.167:0/3505225897 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:46 5.8 5:10000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x556db249e1a0 con 0x7f34f4011040 2023-10-05T19:28:34.774 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.766+0000 7f350c173700 1 -- 172.21.15.167:0/3505225897 <== osd.0 v2:172.21.15.118:6800/27690 11 ==== osd_op_reply(46 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f34fc022040 con 0x7f34f4011040 2023-10-05T19:28:34.774 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.770+0000 7f350c173700 1 -- 172.21.15.167:0/3505225897 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:47 5.18 5:18000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x556db249a6f0 con 0x556db2466920 2023-10-05T19:28:34.775 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.770+0000 7f350c173700 1 -- 172.21.15.167:0/3505225897 <== osd.1 v2:172.21.15.167:6800/30035 18 ==== osd_op_reply(47 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f34fc022040 con 0x556db2466920 2023-10-05T19:28:34.775 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.770+0000 7f350c173700 1 -- 172.21.15.167:0/3505225897 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:48 5.4 5:20000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x556db249a6f0 con 0x556db2466920 2023-10-05T19:28:34.775 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.770+0000 7f350c173700 1 -- 172.21.15.167:0/3505225897 <== osd.1 v2:172.21.15.167:6800/30035 19 ==== osd_op_reply(48 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f34fc022040 con 0x556db2466920 2023-10-05T19:28:34.775 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.770+0000 7f350c173700 1 -- 172.21.15.167:0/3505225897 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:49 5.14 5:28000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x556db249a6f0 con 0x7f34f4011040 2023-10-05T19:28:34.776 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.770+0000 7f350c173700 1 -- 172.21.15.167:0/3505225897 <== osd.0 v2:172.21.15.118:6800/27690 12 ==== osd_op_reply(49 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f34fc022040 con 0x7f34f4011040 2023-10-05T19:28:34.776 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.770+0000 7f350c173700 1 -- 172.21.15.167:0/3505225897 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:50 5.c 5:30000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x556db249a6f0 con 0x7f34f4011040 2023-10-05T19:28:34.777 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.770+0000 7f350c173700 1 -- 172.21.15.167:0/3505225897 <== osd.0 v2:172.21.15.118:6800/27690 13 ==== osd_op_reply(50 [pgnls start_epoch 30 out=112b] v23'1 uv1 ondisk = 1) v8 ==== 144+0+112 (crc 0 0 0) 0x7f34fc022040 con 0x7f34f4011040 2023-10-05T19:28:34.777 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.770+0000 7f350c173700 1 -- 172.21.15.167:0/3505225897 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:51 5.1c 5:38000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x556db249a6f0 con 0x7f3504007030 2023-10-05T19:28:34.778 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.774+0000 7f350b171700 1 -- 172.21.15.167:0/3505225897 <== osd.2 v2:172.21.15.167:6816/30037 8 ==== osd_op_reply(51 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f34f4004520 con 0x7f3504007030 2023-10-05T19:28:34.778 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.774+0000 7f350b171700 1 -- 172.21.15.167:0/3505225897 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:52 5.2 5:40000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x556db249a6f0 con 0x556db249b0d0 2023-10-05T19:28:34.779 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.774+0000 7f350b972700 1 -- 172.21.15.167:0/3505225897 <== osd.3 v2:172.21.15.167:6808/30036 12 ==== osd_op_reply(52 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f350401c7f0 con 0x556db249b0d0 2023-10-05T19:28:34.779 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.774+0000 7f350b972700 1 -- 172.21.15.167:0/3505225897 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:53 5.12 5:48000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x556db249a6f0 con 0x556db2466920 2023-10-05T19:28:34.780 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.774+0000 7f350c173700 1 -- 172.21.15.167:0/3505225897 <== osd.1 v2:172.21.15.167:6800/30035 20 ==== osd_op_reply(53 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f34fc022040 con 0x556db2466920 2023-10-05T19:28:34.780 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.774+0000 7f350c173700 1 -- 172.21.15.167:0/3505225897 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:54 5.a 5:50000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x556db249a6f0 con 0x7f34f4011040 2023-10-05T19:28:34.780 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.774+0000 7f350c173700 1 -- 172.21.15.167:0/3505225897 <== osd.0 v2:172.21.15.118:6800/27690 14 ==== osd_op_reply(54 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f34fc022040 con 0x7f34f4011040 2023-10-05T19:28:34.781 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.774+0000 7f350c173700 1 -- 172.21.15.167:0/3505225897 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:55 5.1a 5:58000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x556db249a6f0 con 0x556db2466920 2023-10-05T19:28:34.781 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.774+0000 7f350c173700 1 -- 172.21.15.167:0/3505225897 <== osd.1 v2:172.21.15.167:6800/30035 21 ==== osd_op_reply(55 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f34fc022040 con 0x556db2466920 2023-10-05T19:28:34.781 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.774+0000 7f350c173700 1 -- 172.21.15.167:0/3505225897 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:56 5.6 5:60000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x556db249a6f0 con 0x7f3504007030 2023-10-05T19:28:34.782 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.778+0000 7f350b171700 1 -- 172.21.15.167:0/3505225897 <== osd.2 v2:172.21.15.167:6816/30037 9 ==== osd_op_reply(56 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f34f4004520 con 0x7f3504007030 2023-10-05T19:28:34.782 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.778+0000 7f350b171700 1 -- 172.21.15.167:0/3505225897 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:57 5.16 5:68000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x556db249a6f0 con 0x556db249b0d0 2023-10-05T19:28:34.783 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.778+0000 7f350b972700 1 -- 172.21.15.167:0/3505225897 <== osd.3 v2:172.21.15.167:6808/30036 13 ==== osd_op_reply(57 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f350401c7f0 con 0x556db249b0d0 2023-10-05T19:28:34.783 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.778+0000 7f350b972700 1 -- 172.21.15.167:0/3505225897 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:58 5.e 5:70000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x556db249a6f0 con 0x556db249b0d0 2023-10-05T19:28:34.784 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.778+0000 7f350b972700 1 -- 172.21.15.167:0/3505225897 <== osd.3 v2:172.21.15.167:6808/30036 14 ==== osd_op_reply(58 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f350401c7f0 con 0x556db249b0d0 2023-10-05T19:28:34.784 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.778+0000 7f350b972700 1 -- 172.21.15.167:0/3505225897 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:59 5.1e 5:78000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x556db249a6f0 con 0x7f34f4011040 2023-10-05T19:28:34.785 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.778+0000 7f350c173700 1 -- 172.21.15.167:0/3505225897 <== osd.0 v2:172.21.15.118:6800/27690 15 ==== osd_op_reply(59 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f34fc022040 con 0x7f34f4011040 2023-10-05T19:28:34.785 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.778+0000 7f350c173700 1 -- 172.21.15.167:0/3505225897 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:60 5.1 5:80000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x556db249a6f0 con 0x556db249b0d0 2023-10-05T19:28:34.786 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.778+0000 7f350b972700 1 -- 172.21.15.167:0/3505225897 <== osd.3 v2:172.21.15.167:6808/30036 15 ==== osd_op_reply(60 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f350401c7f0 con 0x556db249b0d0 2023-10-05T19:28:34.786 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.778+0000 7f350b972700 1 -- 172.21.15.167:0/3505225897 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:61 5.11 5:88000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x556db249a6f0 con 0x556db249b0d0 2023-10-05T19:28:34.786 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.782+0000 7f350b972700 1 -- 172.21.15.167:0/3505225897 <== osd.3 v2:172.21.15.167:6808/30036 16 ==== osd_op_reply(61 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f350401c7f0 con 0x556db249b0d0 2023-10-05T19:28:34.787 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.782+0000 7f350b972700 1 -- 172.21.15.167:0/3505225897 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:62 5.9 5:90000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x556db249a6f0 con 0x556db2466920 2023-10-05T19:28:34.787 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.782+0000 7f350c173700 1 -- 172.21.15.167:0/3505225897 <== osd.1 v2:172.21.15.167:6800/30035 22 ==== osd_op_reply(62 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f34fc022040 con 0x556db2466920 2023-10-05T19:28:34.787 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.782+0000 7f350c173700 1 -- 172.21.15.167:0/3505225897 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:63 5.19 5:98000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x556db249a6f0 con 0x556db2466920 2023-10-05T19:28:34.788 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.782+0000 7f350c173700 1 -- 172.21.15.167:0/3505225897 <== osd.1 v2:172.21.15.167:6800/30035 23 ==== osd_op_reply(63 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f34fc022040 con 0x556db2466920 2023-10-05T19:28:34.788 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.782+0000 7f350c173700 1 -- 172.21.15.167:0/3505225897 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:64 5.5 5:a0000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x556db249a6f0 con 0x7f34f4011040 2023-10-05T19:28:34.789 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.782+0000 7f350c173700 1 -- 172.21.15.167:0/3505225897 <== osd.0 v2:172.21.15.118:6800/27690 16 ==== osd_op_reply(64 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f34fc022040 con 0x7f34f4011040 2023-10-05T19:28:34.789 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.782+0000 7f350c173700 1 -- 172.21.15.167:0/3505225897 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:65 5.15 5:a8000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x556db249a6f0 con 0x556db249b0d0 2023-10-05T19:28:34.789 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.782+0000 7f350b972700 1 -- 172.21.15.167:0/3505225897 <== osd.3 v2:172.21.15.167:6808/30036 17 ==== osd_op_reply(65 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f350401c7f0 con 0x556db249b0d0 2023-10-05T19:28:34.790 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.782+0000 7f350b972700 1 -- 172.21.15.167:0/3505225897 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:66 5.d 5:b0000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x556db249a6f0 con 0x7f3504007030 2023-10-05T19:28:34.791 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.786+0000 7f350b171700 1 -- 172.21.15.167:0/3505225897 <== osd.2 v2:172.21.15.167:6816/30037 10 ==== osd_op_reply(66 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f34f4004520 con 0x7f3504007030 2023-10-05T19:28:34.791 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.786+0000 7f350b171700 1 -- 172.21.15.167:0/3505225897 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:67 5.1d 5:b8000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x556db249a6f0 con 0x556db2466920 2023-10-05T19:28:34.792 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.786+0000 7f350c173700 1 -- 172.21.15.167:0/3505225897 <== osd.1 v2:172.21.15.167:6800/30035 24 ==== osd_op_reply(67 [pgnls start_epoch 30 out=107b] v30'2 uv2 ondisk = 1) v8 ==== 144+0+107 (crc 0 0 0) 0x7f34fc022040 con 0x556db2466920 2023-10-05T19:28:34.792 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.786+0000 7f350c173700 1 -- 172.21.15.167:0/3505225897 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:68 5.3 5:c0000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x556db249a6f0 con 0x7f34f4011040 2023-10-05T19:28:34.793 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.786+0000 7f350c173700 1 -- 172.21.15.167:0/3505225897 <== osd.0 v2:172.21.15.118:6800/27690 17 ==== osd_op_reply(68 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f34fc022040 con 0x7f34f4011040 2023-10-05T19:28:34.793 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.786+0000 7f350c173700 1 -- 172.21.15.167:0/3505225897 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:69 5.13 5:c8000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x556db249a6f0 con 0x556db249b0d0 2023-10-05T19:28:34.793 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.786+0000 7f350b972700 1 -- 172.21.15.167:0/3505225897 <== osd.3 v2:172.21.15.167:6808/30036 18 ==== osd_op_reply(69 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f350401c7f0 con 0x556db249b0d0 2023-10-05T19:28:34.794 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.786+0000 7f350b972700 1 -- 172.21.15.167:0/3505225897 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:70 5.b 5:d0000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x556db249a6f0 con 0x7f34f4011040 2023-10-05T19:28:34.794 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.790+0000 7f350c173700 1 -- 172.21.15.167:0/3505225897 <== osd.0 v2:172.21.15.118:6800/27690 18 ==== osd_op_reply(70 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f34fc022040 con 0x7f34f4011040 2023-10-05T19:28:34.794 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.790+0000 7f350c173700 1 -- 172.21.15.167:0/3505225897 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:71 5.1b 5:d8000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x556db249a6f0 con 0x7f3504007030 2023-10-05T19:28:34.795 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.790+0000 7f350b171700 1 -- 172.21.15.167:0/3505225897 <== osd.2 v2:172.21.15.167:6816/30037 11 ==== osd_op_reply(71 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f34f4004520 con 0x7f3504007030 2023-10-05T19:28:34.795 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.790+0000 7f350b171700 1 -- 172.21.15.167:0/3505225897 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:72 5.7 5:e0000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x556db249a6f0 con 0x556db249b0d0 2023-10-05T19:28:34.796 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.790+0000 7f350b972700 1 -- 172.21.15.167:0/3505225897 <== osd.3 v2:172.21.15.167:6808/30036 19 ==== osd_op_reply(72 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f350401c7f0 con 0x556db249b0d0 2023-10-05T19:28:34.797 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.790+0000 7f350b972700 1 -- 172.21.15.167:0/3505225897 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:73 5.17 5:e8000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x556db249a6f0 con 0x7f34f4011040 2023-10-05T19:28:34.797 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.790+0000 7f350c173700 1 -- 172.21.15.167:0/3505225897 <== osd.0 v2:172.21.15.118:6800/27690 19 ==== osd_op_reply(73 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f34fc022040 con 0x7f34f4011040 2023-10-05T19:28:34.798 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.790+0000 7f350c173700 1 -- 172.21.15.167:0/3505225897 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:74 5.f 5:f0000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x556db249a6f0 con 0x556db2466920 2023-10-05T19:28:34.799 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.794+0000 7f350c173700 1 -- 172.21.15.167:0/3505225897 <== osd.1 v2:172.21.15.167:6800/30035 25 ==== osd_op_reply(74 [pgnls start_epoch 30 out=115b] v23'2 uv2 ondisk = 1) v8 ==== 144+0+115 (crc 0 0 0) 0x7f34fc022040 con 0x556db2466920 2023-10-05T19:28:34.799 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.794+0000 7f350c173700 1 -- 172.21.15.167:0/3505225897 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:75 5.1f 5:f8000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x556db249a6f0 con 0x7f3504007030 2023-10-05T19:28:34.800 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.794+0000 7f350b171700 1 -- 172.21.15.167:0/3505225897 <== osd.2 v2:172.21.15.167:6816/30037 12 ==== osd_op_reply(75 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f34f4004520 con 0x7f3504007030 2023-10-05T19:28:34.800 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.794+0000 7f350de3de40 20 RGWRados::pool_iterate: got zonegroup_info.4f1f719d-9648-417b-b10c-e5d939825419 2023-10-05T19:28:34.800 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.794+0000 7f350de3de40 20 RGWRados::pool_iterate: got zone_info.9a9df965-6612-45d4-a45f-6e3c889bacc2 2023-10-05T19:28:34.800 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.794+0000 7f350de3de40 20 RGWRados::pool_iterate: got zone_names.default 2023-10-05T19:28:34.800 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.794+0000 7f350de3de40 20 RGWRados::pool_iterate: got zonegroups_names.default 2023-10-05T19:28:34.800 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.794+0000 7f350de3de40 20 rados->read ofs=0 len=0 2023-10-05T19:28:34.800 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.794+0000 7f350de3de40 1 -- 172.21.15.167:0/3505225897 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:76 5.f 5:f43b6ece:::zone_names.default:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x556db249e1a0 con 0x556db2466920 2023-10-05T19:28:34.801 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.794+0000 7f350c173700 1 -- 172.21.15.167:0/3505225897 <== osd.1 v2:172.21.15.167:6800/30035 26 ==== osd_op_reply(76 zone_names.default [read 0~46 out=46b] v0'0 uv1 ondisk = 0) v8 ==== 162+0+46 (crc 0 0 0) 0x7f34fc022040 con 0x556db2466920 2023-10-05T19:28:34.801 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.794+0000 7f350de3de40 20 rados_obj.operate() r=0 bl.length=46 2023-10-05T19:28:34.801 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.794+0000 7f350de3de40 20 rados->read ofs=0 len=0 2023-10-05T19:28:34.801 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.794+0000 7f350de3de40 1 -- 172.21.15.167:0/3505225897 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:77 5.1d 5:b8374eb4:::zone_info.9a9df965-6612-45d4-a45f-6e3c889bacc2:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x556db249e600 con 0x556db2466920 2023-10-05T19:28:34.801 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.794+0000 7f350c173700 1 -- 172.21.15.167:0/3505225897 <== osd.1 v2:172.21.15.167:6800/30035 27 ==== osd_op_reply(77 zone_info.9a9df965-6612-45d4-a45f-6e3c889bacc2 [read 0~874 out=874b] v0'0 uv2 ondisk = 0) v8 ==== 190+0+874 (crc 0 0 0) 0x7f34fc022040 con 0x556db2466920 2023-10-05T19:28:34.802 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.794+0000 7f350de3de40 20 rados_obj.operate() r=0 bl.length=874 2023-10-05T19:28:34.802 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.794+0000 7f350de3de40 20 rados->read ofs=0 len=0 2023-10-05T19:28:34.802 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.794+0000 7f350de3de40 1 -- 172.21.15.167:0/3505225897 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:78 5.f 5:f4c53578:::zonegroups_names.default:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x556db249eaf0 con 0x556db2466920 2023-10-05T19:28:34.802 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.798+0000 7f350c173700 1 -- 172.21.15.167:0/3505225897 <== osd.1 v2:172.21.15.167:6800/30035 28 ==== osd_op_reply(78 zonegroups_names.default [read 0~46 out=46b] v0'0 uv2 ondisk = 0) v8 ==== 168+0+46 (crc 0 0 0) 0x7f34fc045040 con 0x556db2466920 2023-10-05T19:28:34.802 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.798+0000 7f350de3de40 20 rados_obj.operate() r=0 bl.length=46 2023-10-05T19:28:34.802 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.798+0000 7f350de3de40 20 rados->read ofs=0 len=0 2023-10-05T19:28:34.803 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.798+0000 7f350de3de40 1 -- 172.21.15.167:0/3505225897 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:79 5.c 5:30d299a5:::zonegroup_info.4f1f719d-9648-417b-b10c-e5d939825419:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x556db249eaf0 con 0x7f34f4011040 2023-10-05T19:28:34.803 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.798+0000 7f350c173700 1 -- 172.21.15.167:0/3505225897 <== osd.0 v2:172.21.15.118:6800/27690 20 ==== osd_op_reply(79 zonegroup_info.4f1f719d-9648-417b-b10c-e5d939825419 [read 0~362 out=362b] v0'0 uv1 ondisk = 0) v8 ==== 195+0+362 (crc 0 0 0) 0x7f34fc045040 con 0x7f34f4011040 2023-10-05T19:28:34.803 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.798+0000 7f350de3de40 20 rados_obj.operate() r=0 bl.length=362 2023-10-05T19:28:34.803 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.798+0000 7f350de3de40 20 rados->read ofs=0 len=0 2023-10-05T19:28:34.803 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.798+0000 7f350de3de40 1 -- 172.21.15.167:0/3505225897 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:80 5.3 5:c0ad8cbd:::region_map:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x556db249eaf0 con 0x7f34f4011040 2023-10-05T19:28:34.804 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.798+0000 7f350c173700 1 -- 172.21.15.167:0/3505225897 <== osd.0 v2:172.21.15.118:6800/27690 21 ==== osd_op_reply(80 region_map [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 ==== 154+0+0 (crc 0 0 0) 0x7f34fc045040 con 0x7f34f4011040 2023-10-05T19:28:34.805 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.798+0000 7f350de3de40 20 rados_obj.operate() r=-2 bl.length=0 2023-10-05T19:28:34.805 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.798+0000 7f350de3de40 20 zone default found 2023-10-05T19:28:34.805 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.798+0000 7f350de3de40 4 Realm: () 2023-10-05T19:28:34.805 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.798+0000 7f350de3de40 4 ZoneGroup: default (4f1f719d-9648-417b-b10c-e5d939825419) 2023-10-05T19:28:34.805 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.798+0000 7f350de3de40 4 Zone: default (9a9df965-6612-45d4-a45f-6e3c889bacc2) 2023-10-05T19:28:34.805 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.798+0000 7f350de3de40 10 cannot find current period zonegroup using local zonegroup configuration 2023-10-05T19:28:34.805 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.798+0000 7f350de3de40 20 zonegroup default 2023-10-05T19:28:34.805 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.798+0000 7f350de3de40 20 rados->read ofs=0 len=0 2023-10-05T19:28:34.805 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.798+0000 7f350de3de40 1 -- 172.21.15.167:0/3505225897 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:81 5.3 5:c52100b6:::period_config.default:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x556db249eaf0 con 0x7f34f4011040 2023-10-05T19:28:34.805 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.798+0000 7f350c173700 1 -- 172.21.15.167:0/3505225897 <== osd.0 v2:172.21.15.118:6800/27690 22 ==== osd_op_reply(81 period_config.default [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 ==== 165+0+0 (crc 0 0 0) 0x7f34fc045040 con 0x7f34f4011040 2023-10-05T19:28:34.805 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.798+0000 7f350de3de40 20 rados_obj.operate() r=-2 bl.length=0 2023-10-05T19:28:34.806 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.798+0000 7f350de3de40 20 rados->read ofs=0 len=0 2023-10-05T19:28:34.806 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.798+0000 7f350de3de40 1 -- 172.21.15.167:0/3505225897 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:82 6.2 6:4347d321:::bucket.sync-source-hints.:head [call version.read in=11b,read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x556db24a0470 con 0x556db249b0d0 2023-10-05T19:28:34.807 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.798+0000 7f350b972700 1 -- 172.21.15.167:0/3505225897 <== osd.3 v2:172.21.15.167:6808/30036 20 ==== osd_op_reply(82 bucket.sync-source-hints. [call,read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 ==== 211+0+0 (crc 0 0 0) 0x7f350405f040 con 0x556db249b0d0 2023-10-05T19:28:34.807 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.798+0000 7f350de3de40 20 rados_obj.operate() r=-2 bl.length=0 2023-10-05T19:28:34.807 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.798+0000 7f350de3de40 20 rados->read ofs=0 len=0 2023-10-05T19:28:34.807 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.798+0000 7f350de3de40 1 -- 172.21.15.167:0/3505225897 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:83 6.b 6:d467b91b:::bucket.sync-target-hints.:head [call version.read in=11b,read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x556db24a0820 con 0x556db249b0d0 2023-10-05T19:28:34.807 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.802+0000 7f350b972700 1 -- 172.21.15.167:0/3505225897 <== osd.3 v2:172.21.15.167:6808/30036 21 ==== osd_op_reply(83 bucket.sync-target-hints. [call,read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 ==== 211+0+0 (crc 0 0 0) 0x7f350405f040 con 0x556db249b0d0 2023-10-05T19:28:34.807 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.802+0000 7f350de3de40 20 rados_obj.operate() r=-2 bl.length=0 2023-10-05T19:28:34.807 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.802+0000 7f350de3de40 20 started sync module instance, tier type = 2023-10-05T19:28:34.807 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.802+0000 7f350de3de40 20 started zone id=9a9df965-6612-45d4-a45f-6e3c889bacc2 (name=default) with tier type = 2023-10-05T19:28:34.807 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.802+0000 7f350de3de40 1 -- 172.21.15.167:0/3505225897 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:84 7.1f 7:f95f44c2:::notify.0:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x556db249fb30 con 0x7f3504007030 2023-10-05T19:28:34.809 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.802+0000 7f350b171700 1 -- 172.21.15.167:0/3505225897 <== osd.2 v2:172.21.15.167:6816/30037 13 ==== osd_op_reply(84 notify.0 [create] v30'135 uv135 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f34f402f040 con 0x7f3504007030 2023-10-05T19:28:34.810 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.802+0000 7f350de3de40 1 -- 172.21.15.167:0/3505225897 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:85 7.1f 7:f95f44c2:::notify.0:head [watch watch cookie 93929630990048] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x556db249f760 con 0x7f3504007030 2023-10-05T19:28:34.810 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.802+0000 7f350de3de40 1 -- 172.21.15.167:0/3505225897 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:86 7.0 7:05bf5b68:::notify.1:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x556db24a1680 con 0x556db2466920 2023-10-05T19:28:34.812 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.806+0000 7f350b171700 1 -- 172.21.15.167:0/3505225897 <== osd.2 v2:172.21.15.167:6816/30037 14 ==== osd_op_reply(85 notify.0 [watch watch cookie 93929630990048] v30'136 uv135 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f34f402f040 con 0x7f3504007030 2023-10-05T19:28:34.813 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.806+0000 7f350c173700 1 -- 172.21.15.167:0/3505225897 <== osd.1 v2:172.21.15.167:6800/30035 29 ==== osd_op_reply(86 notify.1 [create] v30'135 uv135 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f34fc045040 con 0x556db2466920 2023-10-05T19:28:34.813 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.806+0000 7f350de3de40 1 -- 172.21.15.167:0/3505225897 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:87 7.0 7:05bf5b68:::notify.1:head [watch watch cookie 93929630993360] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x556db24a1a80 con 0x556db2466920 2023-10-05T19:28:34.813 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.806+0000 7f350de3de40 1 -- 172.21.15.167:0/3505225897 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:88 7.15 7:a93a5511:::notify.2:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x556db24a2320 con 0x556db249b0d0 2023-10-05T19:28:34.816 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.810+0000 7f350b972700 1 -- 172.21.15.167:0/3505225897 <== osd.3 v2:172.21.15.167:6808/30036 22 ==== osd_op_reply(88 notify.2 [create] v30'135 uv135 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f350405f040 con 0x556db249b0d0 2023-10-05T19:28:34.816 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.810+0000 7f350de3de40 1 -- 172.21.15.167:0/3505225897 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:89 7.15 7:a93a5511:::notify.2:head [watch watch cookie 93929630997120] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x556db24a2bc0 con 0x556db249b0d0 2023-10-05T19:28:34.816 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.810+0000 7f350de3de40 1 -- 172.21.15.167:0/3505225897 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:90 7.e 7:7759931f:::notify.3:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x556db24a3460 con 0x7f3504007030 2023-10-05T19:28:34.816 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.810+0000 7f350c173700 1 -- 172.21.15.167:0/3505225897 <== osd.1 v2:172.21.15.167:6800/30035 30 ==== osd_op_reply(87 notify.1 [watch watch cookie 93929630993360] v30'136 uv135 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f34fc045040 con 0x556db2466920 2023-10-05T19:28:34.818 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.814+0000 7f350b972700 1 -- 172.21.15.167:0/3505225897 <== osd.3 v2:172.21.15.167:6808/30036 23 ==== osd_op_reply(89 notify.2 [watch watch cookie 93929630997120] v30'136 uv135 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f350405f040 con 0x556db249b0d0 2023-10-05T19:28:34.819 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.814+0000 7f350b171700 1 -- 172.21.15.167:0/3505225897 <== osd.2 v2:172.21.15.167:6816/30037 15 ==== osd_op_reply(90 notify.3 [create] v30'135 uv135 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f34f402f040 con 0x7f3504007030 2023-10-05T19:28:34.819 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.814+0000 7f350de3de40 1 -- 172.21.15.167:0/3505225897 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:91 7.e 7:7759931f:::notify.3:head [watch watch cookie 93929631002480] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x556db24a27f0 con 0x7f3504007030 2023-10-05T19:28:34.819 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.814+0000 7f350de3de40 1 -- 172.21.15.167:0/3505225897 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:92 7.d 7:b4812045:::notify.4:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x556db24a3e70 con 0x556db2466920 2023-10-05T19:28:34.822 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.814+0000 7f350b171700 1 -- 172.21.15.167:0/3505225897 <== osd.2 v2:172.21.15.167:6816/30037 16 ==== osd_op_reply(91 notify.3 [watch watch cookie 93929631002480] v30'136 uv135 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f34f402f040 con 0x7f3504007030 2023-10-05T19:28:34.822 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.814+0000 7f350c173700 1 -- 172.21.15.167:0/3505225897 <== osd.1 v2:172.21.15.167:6800/30035 31 ==== osd_op_reply(92 notify.4 [create] v30'135 uv135 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f34fc045040 con 0x556db2466920 2023-10-05T19:28:34.822 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.814+0000 7f350de3de40 1 -- 172.21.15.167:0/3505225897 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:93 7.d 7:b4812045:::notify.4:head [watch watch cookie 93929631005056] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x556db24a12b0 con 0x556db2466920 2023-10-05T19:28:34.822 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.814+0000 7f350de3de40 1 -- 172.21.15.167:0/3505225897 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:94 7.3 7:c609908c:::notify.5:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x556db24a4c50 con 0x7f34f4011040 2023-10-05T19:28:34.824 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.818+0000 7f350c173700 1 -- 172.21.15.167:0/3505225897 <== osd.1 v2:172.21.15.167:6800/30035 32 ==== osd_op_reply(93 notify.4 [watch watch cookie 93929631005056] v30'136 uv135 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f34fc045040 con 0x556db2466920 2023-10-05T19:28:34.824 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.818+0000 7f350c173700 1 -- 172.21.15.167:0/3505225897 <== osd.0 v2:172.21.15.118:6800/27690 23 ==== osd_op_reply(94 notify.5 [create] v30'135 uv135 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f34fc045040 con 0x7f34f4011040 2023-10-05T19:28:34.824 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.818+0000 7f350de3de40 1 -- 172.21.15.167:0/3505225897 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:95 7.3 7:c609908c:::notify.5:head [watch watch cookie 93929631007264] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x556db24a4b60 con 0x7f34f4011040 2023-10-05T19:28:34.824 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.818+0000 7f350de3de40 1 -- 172.21.15.167:0/3505225897 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:96 7.14 7:2b04a3e9:::notify.6:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x556db24a5570 con 0x7f34f4011040 2023-10-05T19:28:34.827 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.822+0000 7f350c173700 1 -- 172.21.15.167:0/3505225897 <== osd.0 v2:172.21.15.118:6800/27690 24 ==== osd_op_reply(96 notify.6 [create] v30'135 uv135 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f34fc045040 con 0x7f34f4011040 2023-10-05T19:28:34.827 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.822+0000 7f350c173700 1 -- 172.21.15.167:0/3505225897 <== osd.0 v2:172.21.15.118:6800/27690 25 ==== osd_op_reply(95 notify.5 [watch watch cookie 93929631007264] v30'136 uv135 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f34fc045040 con 0x7f34f4011040 2023-10-05T19:28:34.827 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.822+0000 7f350de3de40 1 -- 172.21.15.167:0/3505225897 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:97 7.14 7:2b04a3e9:::notify.6:head [watch watch cookie 93929631010944] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x556db24a59c0 con 0x7f34f4011040 2023-10-05T19:28:34.827 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.822+0000 7f350de3de40 1 -- 172.21.15.167:0/3505225897 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:98 7.9 7:93e5b521:::notify.7:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x556db24a5ee0 con 0x7f34f4011040 2023-10-05T19:28:34.830 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.822+0000 7f350c173700 1 -- 172.21.15.167:0/3505225897 <== osd.0 v2:172.21.15.118:6800/27690 26 ==== osd_op_reply(97 notify.6 [watch watch cookie 93929631010944] v30'136 uv135 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f34fc045040 con 0x7f34f4011040 2023-10-05T19:28:34.830 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.826+0000 7f350c173700 1 -- 172.21.15.167:0/3505225897 <== osd.0 v2:172.21.15.118:6800/27690 27 ==== osd_op_reply(98 notify.7 [create] v30'135 uv135 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f34fc045040 con 0x7f34f4011040 2023-10-05T19:28:34.830 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.826+0000 7f350de3de40 1 -- 172.21.15.167:0/3505225897 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:99 7.9 7:93e5b521:::notify.7:head [watch watch cookie 93929631012048] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x556db24a5920 con 0x7f34f4011040 2023-10-05T19:28:34.831 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.826+0000 7f350de3de40 20 add_watcher() i=0 2023-10-05T19:28:34.831 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.826+0000 7f350de3de40 20 add_watcher() i=1 2023-10-05T19:28:34.831 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.826+0000 7f350de3de40 20 add_watcher() i=2 2023-10-05T19:28:34.831 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.826+0000 7f350de3de40 20 add_watcher() i=3 2023-10-05T19:28:34.831 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.826+0000 7f350de3de40 20 add_watcher() i=4 2023-10-05T19:28:34.831 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.826+0000 7f350de3de40 20 add_watcher() i=5 2023-10-05T19:28:34.831 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.826+0000 7f350de3de40 20 add_watcher() i=6 2023-10-05T19:28:34.833 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.826+0000 7f350c173700 1 -- 172.21.15.167:0/3505225897 <== osd.0 v2:172.21.15.118:6800/27690 28 ==== osd_op_reply(99 notify.7 [watch watch cookie 93929631012048] v30'136 uv135 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f34fc045040 con 0x7f34f4011040 2023-10-05T19:28:34.833 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.826+0000 7f350de3de40 20 add_watcher() i=7 2023-10-05T19:28:34.833 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.826+0000 7f350de3de40 2 all 8 watchers are set, enabling cache 2023-10-05T19:28:34.833 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.826+0000 7f350de3de40 1 -- 172.21.15.167:0/3505225897 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:100 6.4 6:22d26bf9:::data_loggenerations_metadata:head [call version.check_conds in=74b,call version.read in=11b,read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x556db24a9400 con 0x556db2466920 2023-10-05T19:28:34.833 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.826+0000 7f350c173700 1 -- 172.21.15.167:0/3505225897 <== osd.1 v2:172.21.15.167:6800/30035 33 ==== osd_op_reply(100 data_loggenerations_metadata [call,call out=48b,read 0~28 out=28b] v0'0 uv1 ondisk = 0) v8 ==== 256+0+76 (crc 0 0 0) 0x7f34fc045040 con 0x556db2466920 2023-10-05T19:28:34.834 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.826+0000 7f350de3de40 1 -- 172.21.15.167:0/3505225897 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:101 6.4 6:22d26bf9:::data_loggenerations_metadata:head [watch watch cookie 93929631014624] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x556db24a6820 con 0x556db2466920 2023-10-05T19:28:34.835 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.830+0000 7f350c173700 1 -- 172.21.15.167:0/3505225897 <== osd.1 v2:172.21.15.167:6800/30035 34 ==== osd_op_reply(101 data_loggenerations_metadata [watch watch cookie 93929631014624] v30'131 uv1 ondisk = 0) v8 ==== 172+0+0 (crc 0 0 0) 0x7f34fc045040 con 0x556db2466920 2023-10-05T19:28:34.836 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.830+0000 7f350de3de40 20 check_secure_mon_conn(): auth registy supported: methods=[2] modes=[2,1] 2023-10-05T19:28:34.836 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.830+0000 7f350de3de40 20 check_secure_mon_conn(): mode 1 is insecure 2023-10-05T19:28:34.836 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.830+0000 7f350de3de40 1 Processor -- start 2023-10-05T19:28:34.836 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.830+0000 7f34dd7e2700 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2023-10-05T19:28:34.836 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.830+0000 7f350de3de40 1 -- start start 2023-10-05T19:28:34.837 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.830+0000 7f350de3de40 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x556db24b3690 0x556db24b3a60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-05T19:28:34.837 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.830+0000 7f350de3de40 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x556db24b3fa0 con 0x556db24b3690 2023-10-05T19:28:34.837 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.830+0000 7f350b171700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x556db24b3690 0x556db24b3a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-05T19:28:34.837 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.830+0000 7f350b171700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x556db24b3690 0x556db24b3a60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.118:3300/0 says I am v2:172.21.15.167:41536/0 (socket says 172.21.15.167:41536) 2023-10-05T19:28:34.838 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.830+0000 7f350b171700 1 -- 172.21.15.167:0/385203162 learned_addr learned my addr 172.21.15.167:0/385203162 (peer_addr_for_me v2:172.21.15.167:0/0) 2023-10-05T19:28:34.838 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.830+0000 7f350b171700 1 -- 172.21.15.167:0/385203162 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x556db24b40e0 con 0x556db24b3690 2023-10-05T19:28:34.838 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.830+0000 7f350b171700 1 --2- 172.21.15.167:0/385203162 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x556db24b3690 0x556db24b3a60 secure :-1 s=READY pgs=408 cs=0 l=1 rev1=1 crypto rx=0x7f350402fb90 tx=0x7f34f4006000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e9d29cd55d265832 server_cookie=0 in_seq=0 out_seq=0 2023-10-05T19:28:34.838 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.834+0000 7f34dcfe1700 1 -- 172.21.15.167:0/385203162 <== mon.0 v2:172.21.15.118:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f34f402c040 con 0x556db24b3690 2023-10-05T19:28:34.838 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.834+0000 7f34dcfe1700 1 -- 172.21.15.167:0/385203162 <== mon.0 v2:172.21.15.118:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f34f402d040 con 0x556db24b3690 2023-10-05T19:28:34.839 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.834+0000 7f34dcfe1700 1 -- 172.21.15.167:0/385203162 <== mon.0 v2:172.21.15.118:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f34f4028a20 con 0x556db24b3690 2023-10-05T19:28:34.840 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.834+0000 7f350de3de40 1 -- 172.21.15.167:0/385203162 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x556db24b3690 msgr2=0x556db24b3a60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-05T19:28:34.840 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.834+0000 7f350de3de40 1 --2- 172.21.15.167:0/385203162 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x556db24b3690 0x556db24b3a60 secure :-1 s=READY pgs=408 cs=0 l=1 rev1=1 crypto rx=0x7f350402fb90 tx=0x7f34f4006000 comp rx=0 tx=0).stop 2023-10-05T19:28:34.840 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.834+0000 7f350de3de40 1 -- 172.21.15.167:0/385203162 shutdown_connections 2023-10-05T19:28:34.840 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.834+0000 7f350de3de40 1 --2- 172.21.15.167:0/385203162 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x556db24b3690 0x556db24b3a60 unknown :-1 s=CLOSED pgs=408 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-05T19:28:34.840 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.834+0000 7f350de3de40 1 -- 172.21.15.167:0/385203162 >> 172.21.15.167:0/385203162 conn(0x556db24aea50 msgr2=0x556db24acc00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-05T19:28:34.840 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.834+0000 7f350de3de40 1 -- 172.21.15.167:0/385203162 shutdown_connections 2023-10-05T19:28:34.840 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.834+0000 7f350de3de40 1 -- 172.21.15.167:0/385203162 wait complete. 2023-10-05T19:28:34.840 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.834+0000 7f350de3de40 1 Processor -- start 2023-10-05T19:28:34.841 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.834+0000 7f350de3de40 1 -- start start 2023-10-05T19:28:34.841 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.834+0000 7f350de3de40 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x556db24b3690 0x556db24b90a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-05T19:28:34.841 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.834+0000 7f350de3de40 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x556db24ba0a0 con 0x556db24b3690 2023-10-05T19:28:34.841 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.834+0000 7f350b171700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x556db24b3690 0x556db24b90a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-05T19:28:34.842 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.834+0000 7f350b171700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x556db24b3690 0x556db24b90a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.118:3300/0 says I am v2:172.21.15.167:41552/0 (socket says 172.21.15.167:41552) 2023-10-05T19:28:34.842 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.834+0000 7f350b171700 1 -- 172.21.15.167:0/3865306600 learned_addr learned my addr 172.21.15.167:0/3865306600 (peer_addr_for_me v2:172.21.15.167:0/0) 2023-10-05T19:28:34.842 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.834+0000 7f350b171700 1 -- 172.21.15.167:0/3865306600 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f34f8001e10 con 0x556db24b3690 2023-10-05T19:28:34.842 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.834+0000 7f350b171700 1 --2- 172.21.15.167:0/3865306600 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x556db24b3690 0x556db24b90a0 secure :-1 s=READY pgs=409 cs=0 l=1 rev1=1 crypto rx=0x556db24ad770 tx=0x7f34f4036000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-05T19:28:34.843 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.838+0000 7f3456ffd700 1 -- 172.21.15.167:0/3865306600 <== mon.0 v2:172.21.15.118:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f34f402c030 con 0x556db24b3690 2023-10-05T19:28:34.843 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.838+0000 7f3456ffd700 1 -- 172.21.15.167:0/3865306600 <== mon.0 v2:172.21.15.118:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f34f401c040 con 0x556db24b3690 2023-10-05T19:28:34.843 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.838+0000 7f350de3de40 1 -- 172.21.15.167:0/3865306600 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x556db24b96a0 con 0x556db24b3690 2023-10-05T19:28:34.843 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.838+0000 7f3456ffd700 1 -- 172.21.15.167:0/3865306600 <== mon.0 v2:172.21.15.118:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f34f40041b0 con 0x556db24b3690 2023-10-05T19:28:34.843 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.838+0000 7f350de3de40 1 -- 172.21.15.167:0/3865306600 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x556db24b9b40 con 0x556db24b3690 2023-10-05T19:28:34.844 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.838+0000 7f3456ffd700 1 -- 172.21.15.167:0/3865306600 <== mon.0 v2:172.21.15.118:3300/0 4 ==== mgrmap(e 5) v1 ==== 40329+0+0 (secure 0 0 0) 0x7f34f4003450 con 0x556db24b3690 2023-10-05T19:28:34.844 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.838+0000 7f3456ffd700 1 --2- 172.21.15.167:0/3865306600 >> [v2:172.21.15.167:6824/29889,v1:172.21.15.167:6825/29889] conn(0x7f344c030c50 0x7f344c033100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-05T19:28:34.844 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.838+0000 7f3456ffd700 1 -- 172.21.15.167:0/3865306600 <== mon.0 v2:172.21.15.118:3300/0 5 ==== osd_map(30..30 src has 1..30) v4 ==== 6057+0+0 (secure 0 0 0) 0x7f34f4049c80 con 0x556db24b3690 2023-10-05T19:28:34.844 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.838+0000 7f350c173700 1 --2- 172.21.15.167:0/3865306600 >> [v2:172.21.15.167:6824/29889,v1:172.21.15.167:6825/29889] conn(0x7f344c030c50 0x7f344c033100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-05T19:28:34.844 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.838+0000 7f350de3de40 5 note: GC not initialized 2023-10-05T19:28:34.844 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.838+0000 7f350c173700 1 --2- 172.21.15.167:0/3865306600 >> [v2:172.21.15.167:6824/29889,v1:172.21.15.167:6825/29889] conn(0x7f344c030c50 0x7f344c033100 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x556db24a5aa0 tx=0x7f34fc000f90 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-05T19:28:34.845 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.838+0000 7f344bfff700 20 reqs_thread_entry: start 2023-10-05T19:28:34.846 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.842+0000 7f350de3de40 20 init_complete bucket index max shards: 11 2023-10-05T19:28:34.846 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.842+0000 7f3449ffb700 20 reqs_thread_entry: start 2023-10-05T19:28:34.847 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.842+0000 7f350de3de40 10 rgw notify: Started notification manager with: 1 workers 2023-10-05T19:28:34.847 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.842+0000 7f3437fff700 20 rgw notify: INFO: next queues processing will happen at: Thu Oct 5 19:29:05 2023 2023-10-05T19:28:34.847 INFO:teuthology.orchestra.run.smithi167.stderr: 2023-10-05T19:28:34.847 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.842+0000 7f34377fe700 20 reqs_thread_entry: start 2023-10-05T19:28:34.847 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.842+0000 7f350de3de40 20 get_system_obj_state: rctx=0x7ffee2986bf8 obj=default.rgw.meta:root:testx/myfoo state=0x556db233d0b0 s->prefetch_data=0 2023-10-05T19:28:34.847 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.842+0000 7f350de3de40 1 -- 172.21.15.167:0/3505225897 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:102 8.5 8:a7a6c9d8:root::testx%2fmyfoo:head [call version.read in=11b,getxattrs,stat] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x556db2521550 con 0x556db2466920 2023-10-05T19:28:34.848 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.842+0000 7f350c173700 1 -- 172.21.15.167:0/3505225897 <== osd.1 v2:172.21.15.167:6800/30035 35 ==== osd_op_reply(102 testx/myfoo [call out=48b,getxattrs out=75b,stat out=16b] v0'0 uv4 ondisk = 0) v8 ==== 239+0+139 (crc 0 0 0) 0x7f34fc045040 con 0x556db2466920 2023-10-05T19:28:34.848 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.842+0000 7f350de3de40 20 get_system_obj_state: s->obj_tag was set empty 2023-10-05T19:28:34.848 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.842+0000 7f350de3de40 20 Read xattr: user.rgw.idtag 2023-10-05T19:28:34.848 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.842+0000 7f350de3de40 20 rados->read ofs=0 len=0 2023-10-05T19:28:34.848 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.842+0000 7f350de3de40 1 -- 172.21.15.167:0/3505225897 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:103 8.5 8:a7a6c9d8:root::testx%2fmyfoo:head [call version.check_conds in=122b,call version.read in=11b,read 0~0,getxattrs] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x556db2521500 con 0x556db2466920 2023-10-05T19:28:34.849 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.842+0000 7f350c173700 1 -- 172.21.15.167:0/3505225897 <== osd.1 v2:172.21.15.167:6800/30035 36 ==== osd_op_reply(103 testx/myfoo [call,call out=48b,read 0~193 out=193b,getxattrs out=75b] v0'0 uv4 ondisk = 0) v8 ==== 281+0+316 (crc 0 0 0) 0x7f34fc045040 con 0x556db2466920 2023-10-05T19:28:34.849 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.842+0000 7f350de3de40 20 rados_obj.operate() r=0 bl.length=193 2023-10-05T19:28:34.849 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.842+0000 7f350de3de40 1 -- 172.21.15.167:0/3505225897 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:104 7.1f 7:f95f44c2:::notify.0:head [watch unwatch cookie 93929630990048] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x556db247b890 con 0x7f3504007030 2023-10-05T19:28:34.851 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.846+0000 7f350b171700 1 -- 172.21.15.167:0/3505225897 <== osd.2 v2:172.21.15.167:6816/30037 17 ==== osd_op_reply(104 notify.0 [watch unwatch cookie 93929630990048] v30'137 uv135 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f34f402f040 con 0x7f3504007030 2023-10-05T19:28:34.852 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.846+0000 7f350de3de40 20 remove_watcher() i=0 2023-10-05T19:28:34.852 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.846+0000 7f350de3de40 2 removed watcher, disabling cache 2023-10-05T19:28:34.852 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.846+0000 7f350de3de40 1 -- 172.21.15.167:0/3505225897 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:105 7.0 7:05bf5b68:::notify.1:head [watch unwatch cookie 93929630993360] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x556db249fee0 con 0x556db2466920 2023-10-05T19:28:34.854 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.850+0000 7f350c173700 1 -- 172.21.15.167:0/3505225897 <== osd.1 v2:172.21.15.167:6800/30035 37 ==== osd_op_reply(105 notify.1 [watch unwatch cookie 93929630993360] v30'137 uv135 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f34fc045040 con 0x556db2466920 2023-10-05T19:28:34.854 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.850+0000 7f350de3de40 20 remove_watcher() i=1 2023-10-05T19:28:34.854 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.850+0000 7f350de3de40 1 -- 172.21.15.167:0/3505225897 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:106 7.15 7:a93a5511:::notify.2:head [watch unwatch cookie 93929630997120] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x556db24a0bd0 con 0x556db249b0d0 2023-10-05T19:28:34.856 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.850+0000 7f350b972700 1 -- 172.21.15.167:0/3505225897 <== osd.3 v2:172.21.15.167:6808/30036 24 ==== osd_op_reply(106 notify.2 [watch unwatch cookie 93929630997120] v30'137 uv135 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f350405f040 con 0x556db249b0d0 2023-10-05T19:28:34.856 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.850+0000 7f350de3de40 20 remove_watcher() i=2 2023-10-05T19:28:34.857 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.850+0000 7f350de3de40 1 -- 172.21.15.167:0/3505225897 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:107 7.e 7:7759931f:::notify.3:head [watch unwatch cookie 93929631002480] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x556db24a1a80 con 0x7f3504007030 2023-10-05T19:28:34.859 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.854+0000 7f350b171700 1 -- 172.21.15.167:0/3505225897 <== osd.2 v2:172.21.15.167:6816/30037 18 ==== osd_op_reply(107 notify.3 [watch unwatch cookie 93929631002480] v30'137 uv135 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f34f402f040 con 0x7f3504007030 2023-10-05T19:28:34.859 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.854+0000 7f350de3de40 20 remove_watcher() i=3 2023-10-05T19:28:34.859 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.854+0000 7f350de3de40 1 -- 172.21.15.167:0/3505225897 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:108 7.d 7:b4812045:::notify.4:head [watch unwatch cookie 93929631005056] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x556db24a1a80 con 0x556db2466920 2023-10-05T19:28:34.862 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.854+0000 7f350c173700 1 -- 172.21.15.167:0/3505225897 <== osd.1 v2:172.21.15.167:6800/30035 38 ==== osd_op_reply(108 notify.4 [watch unwatch cookie 93929631005056] v30'137 uv135 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f34fc045040 con 0x556db2466920 2023-10-05T19:28:34.862 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.854+0000 7f350de3de40 20 remove_watcher() i=4 2023-10-05T19:28:34.862 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.854+0000 7f350de3de40 1 -- 172.21.15.167:0/3505225897 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:109 7.3 7:c609908c:::notify.5:head [watch unwatch cookie 93929631007264] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x556db24a3980 con 0x7f34f4011040 2023-10-05T19:28:34.865 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.858+0000 7f350c173700 1 -- 172.21.15.167:0/3505225897 <== osd.0 v2:172.21.15.118:6800/27690 29 ==== osd_op_reply(109 notify.5 [watch unwatch cookie 93929631007264] v30'137 uv135 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f34fc045040 con 0x7f34f4011040 2023-10-05T19:28:34.865 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.858+0000 7f350de3de40 20 remove_watcher() i=5 2023-10-05T19:28:34.865 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.858+0000 7f350de3de40 1 -- 172.21.15.167:0/3505225897 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:110 7.14 7:2b04a3e9:::notify.6:head [watch unwatch cookie 93929631010944] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x556db24a1a80 con 0x7f34f4011040 2023-10-05T19:28:34.867 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.862+0000 7f350c173700 1 -- 172.21.15.167:0/3505225897 <== osd.0 v2:172.21.15.118:6800/27690 30 ==== osd_op_reply(110 notify.6 [watch unwatch cookie 93929631010944] v30'137 uv135 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f34fc045040 con 0x7f34f4011040 2023-10-05T19:28:34.867 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.862+0000 7f350de3de40 20 remove_watcher() i=6 2023-10-05T19:28:34.867 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.862+0000 7f350de3de40 1 -- 172.21.15.167:0/3505225897 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:111 7.9 7:93e5b521:::notify.7:head [watch unwatch cookie 93929631012048] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x556db24a1a80 con 0x7f34f4011040 2023-10-05T19:28:34.870 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.866+0000 7f350c173700 1 -- 172.21.15.167:0/3505225897 <== osd.0 v2:172.21.15.118:6800/27690 31 ==== osd_op_reply(111 notify.7 [watch unwatch cookie 93929631012048] v30'137 uv135 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f34fc045040 con 0x7f34f4011040 2023-10-05T19:28:34.870 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.866+0000 7f350de3de40 20 remove_watcher() i=7 2023-10-05T19:28:34.872 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.866+0000 7f34dd7e2700 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2023-10-05T19:28:34.872 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.866+0000 7f350de3de40 1 -- 172.21.15.167:0/3505225897 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:112 6.4 6:22d26bf9:::data_loggenerations_metadata:head [watch unwatch cookie 93929631014624] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x556db24a1a80 con 0x556db2466920 2023-10-05T19:28:34.876 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.870+0000 7f350c173700 1 -- 172.21.15.167:0/3505225897 <== osd.1 v2:172.21.15.167:6800/30035 39 ==== osd_op_reply(112 data_loggenerations_metadata [watch unwatch cookie 93929631014624] v30'132 uv1 ondisk = 0) v8 ==== 172+0+0 (crc 0 0 0) 0x7f34fc045040 con 0x556db2466920 2023-10-05T19:28:34.876 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.870+0000 7f350de3de40 1 -- 172.21.15.167:0/3505225897 >> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] conn(0x7f34f4011040 msgr2=0x7f34f4014950 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-05T19:28:34.877 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.870+0000 7f350de3de40 1 --2- 172.21.15.167:0/3505225897 >> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] conn(0x7f34f4011040 0x7f34f4014950 crc :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-05T19:28:34.877 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.870+0000 7f350de3de40 1 -- 172.21.15.167:0/3505225897 >> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] conn(0x556db2466920 msgr2=0x556db24976c0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-05T19:28:34.877 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.870+0000 7f350de3de40 1 --2- 172.21.15.167:0/3505225897 >> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] conn(0x556db2466920 0x556db24976c0 crc :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-05T19:28:34.877 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.870+0000 7f350de3de40 1 -- 172.21.15.167:0/3505225897 >> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] conn(0x7f3504007030 msgr2=0x7f350400c9c0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-05T19:28:34.877 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.870+0000 7f350de3de40 1 --2- 172.21.15.167:0/3505225897 >> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] conn(0x7f3504007030 0x7f350400c9c0 crc :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-05T19:28:34.877 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.870+0000 7f350de3de40 1 -- 172.21.15.167:0/3505225897 >> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] conn(0x556db249b0d0 msgr2=0x556db249d4b0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-05T19:28:34.877 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.870+0000 7f350de3de40 1 --2- 172.21.15.167:0/3505225897 >> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] conn(0x556db249b0d0 0x556db249d4b0 crc :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-05T19:28:34.877 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.870+0000 7f350de3de40 1 -- 172.21.15.167:0/3505225897 >> [v2:172.21.15.167:6824/29889,v1:172.21.15.167:6825/29889] conn(0x7f34ec030c40 msgr2=0x7f34ec0330f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-05T19:28:34.877 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.870+0000 7f350de3de40 1 --2- 172.21.15.167:0/3505225897 >> [v2:172.21.15.167:6824/29889,v1:172.21.15.167:6825/29889] conn(0x7f34ec030c40 0x7f34ec0330f0 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7f34f40044f0 tx=0x7f34f4006690 comp rx=0 tx=0).stop 2023-10-05T19:28:34.877 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.870+0000 7f350de3de40 1 -- 172.21.15.167:0/3505225897 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x556db2466010 msgr2=0x556db248bc30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-05T19:28:34.877 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.870+0000 7f350c173700 1 -- 172.21.15.167:0/3505225897 reap_dead start 2023-10-05T19:28:34.878 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.870+0000 7f350de3de40 1 --2- 172.21.15.167:0/3505225897 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x556db2466010 0x556db248bc30 secure :-1 s=READY pgs=407 cs=0 l=1 rev1=1 crypto rx=0x7f350401f040 tx=0x7f350401c000 comp rx=0 tx=0).stop 2023-10-05T19:28:34.878 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.870+0000 7f350de3de40 1 -- 172.21.15.167:0/3505225897 shutdown_connections 2023-10-05T19:28:34.878 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.870+0000 7f350de3de40 1 --2- 172.21.15.167:0/3505225897 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x556db2466010 0x556db248bc30 unknown :-1 s=CLOSED pgs=407 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-05T19:28:34.878 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.870+0000 7f350de3de40 1 -- 172.21.15.167:0/3505225897 >> 172.21.15.167:0/3505225897 conn(0x556db23d9860 msgr2=0x556db247f5b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-05T19:28:34.878 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.870+0000 7f350de3de40 1 -- 172.21.15.167:0/3505225897 shutdown_connections 2023-10-05T19:28:34.878 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.870+0000 7f350de3de40 1 -- 172.21.15.167:0/3505225897 wait complete. 2023-10-05T19:28:34.879 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.870+0000 7f350de3de40 1 -- 172.21.15.167:0/3865306600 >> [v2:172.21.15.167:6824/29889,v1:172.21.15.167:6825/29889] conn(0x7f344c030c50 msgr2=0x7f344c033100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-05T19:28:34.879 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.870+0000 7f350de3de40 1 --2- 172.21.15.167:0/3865306600 >> [v2:172.21.15.167:6824/29889,v1:172.21.15.167:6825/29889] conn(0x7f344c030c50 0x7f344c033100 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x556db24a5aa0 tx=0x7f34fc000f90 comp rx=0 tx=0).stop 2023-10-05T19:28:34.879 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.870+0000 7f350de3de40 1 -- 172.21.15.167:0/3865306600 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x556db24b3690 msgr2=0x556db24b90a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-05T19:28:34.879 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.870+0000 7f350de3de40 1 --2- 172.21.15.167:0/3865306600 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x556db24b3690 0x556db24b90a0 secure :-1 s=READY pgs=409 cs=0 l=1 rev1=1 crypto rx=0x556db24ad770 tx=0x7f34f4036000 comp rx=0 tx=0).stop 2023-10-05T19:28:34.879 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.870+0000 7f350de3de40 1 -- 172.21.15.167:0/3865306600 shutdown_connections 2023-10-05T19:28:34.879 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.870+0000 7f350de3de40 1 --2- 172.21.15.167:0/3865306600 >> [v2:172.21.15.167:6824/29889,v1:172.21.15.167:6825/29889] conn(0x7f344c030c50 0x7f344c033100 unknown :-1 s=CLOSED pgs=143 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-05T19:28:34.879 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.870+0000 7f350de3de40 1 --2- 172.21.15.167:0/3865306600 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x556db24b3690 0x556db24b90a0 unknown :-1 s=CLOSED pgs=409 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-05T19:28:34.879 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.870+0000 7f350de3de40 1 -- 172.21.15.167:0/3865306600 >> 172.21.15.167:0/3865306600 conn(0x556db24aea50 msgr2=0x556db24acb50 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-05T19:28:34.880 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.870+0000 7f350de3de40 1 -- 172.21.15.167:0/3865306600 shutdown_connections 2023-10-05T19:28:34.880 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.870+0000 7f350de3de40 1 -- 172.21.15.167:0/3865306600 wait complete. 2023-10-05T19:28:34.881 INFO:teuthology.orchestra.run.smithi167.stdout:{ 2023-10-05T19:28:34.881 INFO:teuthology.orchestra.run.smithi167.stdout: "key": "bucket:testx/myfoo", 2023-10-05T19:28:34.881 INFO:teuthology.orchestra.run.smithi167.stdout: "ver": { 2023-10-05T19:28:34.881 INFO:teuthology.orchestra.run.smithi167.stdout: "tag": "KiBBBPEB4dyH64urzKgy/nLj", 2023-10-05T19:28:34.881 INFO:teuthology.orchestra.run.smithi167.stdout: "ver": 2 2023-10-05T19:28:34.881 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2023-10-05T19:28:34.881 INFO:teuthology.orchestra.run.smithi167.stdout: "mtime": "2023-10-05T19:28:34.622097Z", 2023-10-05T19:28:34.881 INFO:teuthology.orchestra.run.smithi167.stdout: "data": { 2023-10-05T19:28:34.881 INFO:teuthology.orchestra.run.smithi167.stdout: "bucket": { 2023-10-05T19:28:34.881 INFO:teuthology.orchestra.run.smithi167.stdout: "name": "myfoo", 2023-10-05T19:28:34.881 INFO:teuthology.orchestra.run.smithi167.stdout: "marker": "9a9df965-6612-45d4-a45f-6e3c889bacc2.4223.1", 2023-10-05T19:28:34.882 INFO:teuthology.orchestra.run.smithi167.stdout: "bucket_id": "9a9df965-6612-45d4-a45f-6e3c889bacc2.4223.1", 2023-10-05T19:28:34.882 INFO:teuthology.orchestra.run.smithi167.stdout: "tenant": "testx", 2023-10-05T19:28:34.882 INFO:teuthology.orchestra.run.smithi167.stdout: "explicit_placement": { 2023-10-05T19:28:34.882 INFO:teuthology.orchestra.run.smithi167.stdout: "data_pool": "", 2023-10-05T19:28:34.882 INFO:teuthology.orchestra.run.smithi167.stdout: "data_extra_pool": "", 2023-10-05T19:28:34.882 INFO:teuthology.orchestra.run.smithi167.stdout: "index_pool": "" 2023-10-05T19:28:34.882 INFO:teuthology.orchestra.run.smithi167.stdout: } 2023-10-05T19:28:34.882 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2023-10-05T19:28:34.882 INFO:teuthology.orchestra.run.smithi167.stdout: "owner": "testx$tenanteduser", 2023-10-05T19:28:34.882 INFO:teuthology.orchestra.run.smithi167.stdout: "creation_time": "2023-10-05T19:28:30.250836Z", 2023-10-05T19:28:34.882 INFO:teuthology.orchestra.run.smithi167.stdout: "linked": "true", 2023-10-05T19:28:34.882 INFO:teuthology.orchestra.run.smithi167.stdout: "has_bucket_info": "false" 2023-10-05T19:28:34.883 INFO:teuthology.orchestra.run.smithi167.stdout: } 2023-10-05T19:28:34.883 INFO:teuthology.orchestra.run.smithi167.stdout:} 2023-10-05T19:28:34.883 DEBUG:tasks.util.rgw: json result: {'key': 'bucket:testx/myfoo', 'ver': {'tag': 'KiBBBPEB4dyH64urzKgy/nLj', 'ver': 2}, 'mtime': '2023-10-05T19:28:34.622097Z', 'data': {'bucket': {'name': 'myfoo', 'marker': '9a9df965-6612-45d4-a45f-6e3c889bacc2.4223.1', 'bucket_id': '9a9df965-6612-45d4-a45f-6e3c889bacc2.4223.1', 'tenant': 'testx', 'explicit_placement': {'data_pool': '', 'data_extra_pool': '', 'index_pool': ''}}, 'owner': 'testx$tenanteduser', 'creation_time': '2023-10-05T19:28:30.250836Z', 'linked': 'true', 'has_bucket_info': 'false'}} 2023-10-05T19:28:34.884 INFO:tasks.util.rgw:rgwadmin: client.0 : ['bucket', 'link', '--bucket', 'testx/myfoo', '--uid', 'foo'] 2023-10-05T19:28:34.884 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'ceph', 'bucket', 'link', '--bucket', 'testx/myfoo', '--uid', 'foo'] 2023-10-05T19:28:34.884 DEBUG:teuthology.orchestra.run.smithi167:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster ceph bucket link --bucket testx/myfoo --uid foo 2023-10-05T19:28:34.925 INFO:teuthology.orchestra.run.smithi167.stderr:ignoring --setuser ceph since I am not root 2023-10-05T19:28:34.925 INFO:teuthology.orchestra.run.smithi167.stderr:ignoring --setgroup ceph since I am not root 2023-10-05T19:28:34.926 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.922+0000 7f32eba3ce40 1 Processor -- start 2023-10-05T19:28:34.926 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.922+0000 7f32eba3ce40 1 -- start start 2023-10-05T19:28:34.927 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.922+0000 7f32eba3ce40 1 --2- >> v2:172.21.15.118:3300/0 conn(0x560a572b97f0 0x560a572b86e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-05T19:28:34.927 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.922+0000 7f32eba3ce40 1 -- --> v1:172.21.15.118:6789/0 -- auth(proto 0 26 bytes epoch 0) v1 -- 0x560a57205bd0 con 0x560a572b9420 2023-10-05T19:28:34.927 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.922+0000 7f32eba3ce40 1 -- --> v2:172.21.15.118:3300/0 -- mon_getmap magic: 0 v1 -- 0x560a571c2ee0 con 0x560a572b97f0 2023-10-05T19:28:34.927 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.922+0000 7f32e8d70700 1 --2- >> v2:172.21.15.118:3300/0 conn(0x560a572b97f0 0x560a572b86e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-05T19:28:34.927 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.922+0000 7f32e9571700 1 --1- >> v1:172.21.15.118:6789/0 conn(0x560a572b9420 0x560a572b7fd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.118:6789/0 says I am v1:172.21.15.167:59276/0 (socket says 172.21.15.167:59276) 2023-10-05T19:28:34.927 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.922+0000 7f32e9571700 1 -- 172.21.15.167:0/834156390 learned_addr learned my addr 172.21.15.167:0/834156390 (peer_addr_for_me v1:172.21.15.167:0/0) 2023-10-05T19:28:34.927 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.922+0000 7f32e8d70700 1 --2- >> v2:172.21.15.118:3300/0 conn(0x560a572b97f0 0x560a572b86e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.118:3300/0 says I am v2:172.21.15.167:41558/0 (socket says 172.21.15.167:41558) 2023-10-05T19:28:34.929 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.922+0000 7f32e8d70700 1 -- 172.21.15.167:0/834156390 >> v1:172.21.15.118:6789/0 conn(0x560a572b9420 legacy=0x560a572b7fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-05T19:28:34.930 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.922+0000 7f32e8d70700 1 -- 172.21.15.167:0/834156390 --> v2:172.21.15.118:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x560a572b8ce0 con 0x560a572b97f0 2023-10-05T19:28:34.930 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.922+0000 7f32e8d70700 1 --2- 172.21.15.167:0/834156390 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x560a572b97f0 0x560a572b86e0 secure :-1 s=READY pgs=411 cs=0 l=1 rev1=1 crypto rx=0x7f32e000a5d0 tx=0x7f32e0005a50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=51e626167205e09c server_cookie=0 in_seq=0 out_seq=0 2023-10-05T19:28:34.930 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.922+0000 7f32dbfff700 1 -- 172.21.15.167:0/834156390 <== mon.0 v2:172.21.15.118:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f32e00086a0 con 0x560a572b97f0 2023-10-05T19:28:34.930 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.922+0000 7f32dbfff700 1 -- 172.21.15.167:0/834156390 <== mon.0 v2:172.21.15.118:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f32e0008800 con 0x560a572b97f0 2023-10-05T19:28:34.930 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.922+0000 7f32dbfff700 1 -- 172.21.15.167:0/834156390 <== mon.0 v2:172.21.15.118:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f32e0008ad0 con 0x560a572b97f0 2023-10-05T19:28:34.930 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.922+0000 7f32eba3ce40 1 -- 172.21.15.167:0/834156390 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x560a572b97f0 msgr2=0x560a572b86e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-05T19:28:34.930 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.922+0000 7f32eba3ce40 1 --2- 172.21.15.167:0/834156390 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x560a572b97f0 0x560a572b86e0 secure :-1 s=READY pgs=411 cs=0 l=1 rev1=1 crypto rx=0x7f32e000a5d0 tx=0x7f32e0005a50 comp rx=0 tx=0).stop 2023-10-05T19:28:34.930 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.922+0000 7f32eba3ce40 1 -- 172.21.15.167:0/834156390 shutdown_connections 2023-10-05T19:28:34.930 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.922+0000 7f32eba3ce40 1 --2- 172.21.15.167:0/834156390 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x560a572b97f0 0x560a572b86e0 unknown :-1 s=CLOSED pgs=411 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-05T19:28:34.930 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.922+0000 7f32eba3ce40 1 -- 172.21.15.167:0/834156390 >> 172.21.15.167:0/834156390 conn(0x560a572b3730 msgr2=0x560a572b5b40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-05T19:28:34.930 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.922+0000 7f32eba3ce40 1 -- 172.21.15.167:0/834156390 shutdown_connections 2023-10-05T19:28:34.931 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.922+0000 7f32eba3ce40 1 -- 172.21.15.167:0/834156390 wait complete. 2023-10-05T19:28:34.932 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.926+0000 7f32eba3ce40 1 Processor -- start 2023-10-05T19:28:34.932 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.926+0000 7f32eba3ce40 1 -- start start 2023-10-05T19:28:34.932 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.926+0000 7f32eba3ce40 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x560a572b9420 0x560a572da170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-05T19:28:34.933 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.926+0000 7f32eba3ce40 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x560a57186e30 con 0x560a572b9420 2023-10-05T19:28:34.933 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.926+0000 7f32e9571700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x560a572b9420 0x560a572da170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-05T19:28:34.933 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.926+0000 7f32e9571700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x560a572b9420 0x560a572da170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.118:3300/0 says I am v2:172.21.15.167:41560/0 (socket says 172.21.15.167:41560) 2023-10-05T19:28:34.933 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.926+0000 7f32e9571700 1 -- 172.21.15.167:0/1106678316 learned_addr learned my addr 172.21.15.167:0/1106678316 (peer_addr_for_me v2:172.21.15.167:0/0) 2023-10-05T19:28:34.933 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.926+0000 7f32e9571700 1 -- 172.21.15.167:0/1106678316 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f32e0005cf0 con 0x560a572b9420 2023-10-05T19:28:34.933 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.926+0000 7f32e9571700 1 --2- 172.21.15.167:0/1106678316 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x560a572b9420 0x560a572da170 secure :-1 s=READY pgs=412 cs=0 l=1 rev1=1 crypto rx=0x7f32e40014a0 tx=0x7f32e4006b90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=73cc567b0953af94 server_cookie=0 in_seq=0 out_seq=0 2023-10-05T19:28:34.933 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.926+0000 7f32daffd700 1 -- 172.21.15.167:0/1106678316 <== mon.0 v2:172.21.15.118:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f32e4006110 con 0x560a572b9420 2023-10-05T19:28:34.933 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.926+0000 7f32daffd700 1 -- 172.21.15.167:0/1106678316 <== mon.0 v2:172.21.15.118:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f32e4006270 con 0x560a572b9420 2023-10-05T19:28:34.933 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.926+0000 7f32daffd700 1 -- 172.21.15.167:0/1106678316 <== mon.0 v2:172.21.15.118:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f32e4004520 con 0x560a572b9420 2023-10-05T19:28:34.933 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.926+0000 7f32eba3ce40 1 -- 172.21.15.167:0/1106678316 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x560a572b9420 msgr2=0x560a572da170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-05T19:28:34.934 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.926+0000 7f32eba3ce40 1 --2- 172.21.15.167:0/1106678316 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x560a572b9420 0x560a572da170 secure :-1 s=READY pgs=412 cs=0 l=1 rev1=1 crypto rx=0x7f32e40014a0 tx=0x7f32e4006b90 comp rx=0 tx=0).stop 2023-10-05T19:28:34.934 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.926+0000 7f32eba3ce40 1 -- 172.21.15.167:0/1106678316 shutdown_connections 2023-10-05T19:28:34.934 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.926+0000 7f32eba3ce40 1 --2- 172.21.15.167:0/1106678316 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x560a572b9420 0x560a572da170 unknown :-1 s=CLOSED pgs=412 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-05T19:28:34.934 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.926+0000 7f32eba3ce40 1 -- 172.21.15.167:0/1106678316 >> 172.21.15.167:0/1106678316 conn(0x560a572b3730 msgr2=0x560a572d5d30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-05T19:28:34.934 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.926+0000 7f32eba3ce40 1 -- 172.21.15.167:0/1106678316 shutdown_connections 2023-10-05T19:28:34.934 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.926+0000 7f32eba3ce40 1 -- 172.21.15.167:0/1106678316 wait complete. 2023-10-05T19:28:34.935 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.926+0000 7f32eba3ce40 1 Processor -- start 2023-10-05T19:28:34.936 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.926+0000 7f32eba3ce40 1 -- start start 2023-10-05T19:28:34.936 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.926+0000 7f32eba3ce40 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x560a572b9420 0x560a572e0f40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-05T19:28:34.936 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.926+0000 7f32eba3ce40 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x560a57159ed0 con 0x560a572b9420 2023-10-05T19:28:34.936 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.926+0000 7f32e9571700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x560a572b9420 0x560a572e0f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-05T19:28:34.936 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.926+0000 7f32e9571700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x560a572b9420 0x560a572e0f40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.118:3300/0 says I am v2:172.21.15.167:41576/0 (socket says 172.21.15.167:41576) 2023-10-05T19:28:34.936 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.926+0000 7f32e9571700 1 -- 172.21.15.167:0/2262883315 learned_addr learned my addr 172.21.15.167:0/2262883315 (peer_addr_for_me v2:172.21.15.167:0/0) 2023-10-05T19:28:34.936 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.930+0000 7f32e9571700 1 -- 172.21.15.167:0/2262883315 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x560a57205bd0 con 0x560a572b9420 2023-10-05T19:28:34.936 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.930+0000 7f32e9571700 1 --2- 172.21.15.167:0/2262883315 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x560a572b9420 0x560a572e0f40 secure :-1 s=READY pgs=413 cs=0 l=1 rev1=1 crypto rx=0x7f32e4006bd0 tx=0x7f32e4014000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-05T19:28:34.936 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.930+0000 7f32d97fa700 1 -- 172.21.15.167:0/2262883315 <== mon.0 v2:172.21.15.118:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f32e4013750 con 0x560a572b9420 2023-10-05T19:28:34.936 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.930+0000 7f32d97fa700 1 -- 172.21.15.167:0/2262883315 <== mon.0 v2:172.21.15.118:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f32e40138b0 con 0x560a572b9420 2023-10-05T19:28:34.936 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.930+0000 7f32eba3ce40 1 -- 172.21.15.167:0/2262883315 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x560a572e1480 con 0x560a572b9420 2023-10-05T19:28:34.937 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.930+0000 7f32d97fa700 1 -- 172.21.15.167:0/2262883315 <== mon.0 v2:172.21.15.118:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f32e4013b80 con 0x560a572b9420 2023-10-05T19:28:34.937 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.930+0000 7f32eba3ce40 1 -- 172.21.15.167:0/2262883315 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x560a572e18c0 con 0x560a572b9420 2023-10-05T19:28:34.937 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.930+0000 7f32eba3ce40 20 rados->read ofs=0 len=0 2023-10-05T19:28:34.938 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.930+0000 7f32d97fa700 1 -- 172.21.15.167:0/2262883315 <== mon.0 v2:172.21.15.118:3300/0 4 ==== mgrmap(e 5) v1 ==== 40329+0+0 (secure 0 0 0) 0x7f32e4028460 con 0x560a572b9420 2023-10-05T19:28:34.938 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.930+0000 7f32d97fa700 1 --2- 172.21.15.167:0/2262883315 >> [v2:172.21.15.167:6824/29889,v1:172.21.15.167:6825/29889] conn(0x7f32cc030c00 0x7f32cc0330b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-05T19:28:34.938 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.930+0000 7f32d97fa700 1 -- 172.21.15.167:0/2262883315 <== mon.0 v2:172.21.15.118:3300/0 5 ==== osd_map(30..30 src has 1..30) v4 ==== 6057+0+0 (secure 0 0 0) 0x7f32e40451a0 con 0x560a572b9420 2023-10-05T19:28:34.938 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.930+0000 7f32e8d70700 1 --2- 172.21.15.167:0/2262883315 >> [v2:172.21.15.167:6824/29889,v1:172.21.15.167:6825/29889] conn(0x7f32cc030c00 0x7f32cc0330b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-05T19:28:34.939 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.930+0000 7f32eba3ce40 1 --2- 172.21.15.167:0/2262883315 >> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] conn(0x560a572b97f0 0x560a572eca30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-05T19:28:34.939 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.930+0000 7f32eba3ce40 1 -- 172.21.15.167:0/2262883315 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:1 5.12 5:49953fa1:::default.realm:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x560a572ed130 con 0x560a572b97f0 2023-10-05T19:28:34.939 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.934+0000 7f32e9d72700 1 --2- 172.21.15.167:0/2262883315 >> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] conn(0x560a572b97f0 0x560a572eca30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-05T19:28:34.939 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.934+0000 7f32e8d70700 1 --2- 172.21.15.167:0/2262883315 >> [v2:172.21.15.167:6824/29889,v1:172.21.15.167:6825/29889] conn(0x7f32cc030c00 0x7f32cc0330b0 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7f32e0005a90 tx=0x7f32e0015000 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-05T19:28:34.939 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.934+0000 7f32e9d72700 1 --2- 172.21.15.167:0/2262883315 >> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] conn(0x560a572b97f0 0x560a572eca30 crc :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-05T19:28:34.940 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.934+0000 7f32e9d72700 1 -- 172.21.15.167:0/2262883315 <== osd.1 v2:172.21.15.167:6800/30035 1 ==== osd_op_reply(1 default.realm [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 ==== 157+0+0 (crc 0 0 0) 0x7f32dc00a500 con 0x560a572b97f0 2023-10-05T19:28:34.940 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.934+0000 7f32eba3ce40 20 rados_obj.operate() r=-2 bl.length=0 2023-10-05T19:28:34.941 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.934+0000 7f32eba3ce40 20 realm 2023-10-05T19:28:34.941 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.934+0000 7f32eba3ce40 20 rados->read ofs=0 len=0 2023-10-05T19:28:34.941 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.934+0000 7f32eba3ce40 1 -- 172.21.15.167:0/2262883315 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:2 5.12 5:49953fa1:::default.realm:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x560a572ea090 con 0x560a572b97f0 2023-10-05T19:28:34.941 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.934+0000 7f32e9d72700 1 -- 172.21.15.167:0/2262883315 <== osd.1 v2:172.21.15.167:6800/30035 2 ==== osd_op_reply(2 default.realm [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 ==== 157+0+0 (crc 0 0 0) 0x7f32dc00a500 con 0x560a572b97f0 2023-10-05T19:28:34.941 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.934+0000 7f32eba3ce40 20 rados_obj.operate() r=-2 bl.length=0 2023-10-05T19:28:34.941 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.934+0000 7f32eba3ce40 4 RGWPeriod::init failed to init realm id : (2) No such file or directory 2023-10-05T19:28:34.941 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.934+0000 7f32eba3ce40 20 rados->read ofs=0 len=0 2023-10-05T19:28:34.941 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.934+0000 7f32eba3ce40 1 -- 172.21.15.167:0/2262883315 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:3 5.12 5:49953fa1:::default.realm:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x560a572edc10 con 0x560a572b97f0 2023-10-05T19:28:34.941 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.934+0000 7f32e9d72700 1 -- 172.21.15.167:0/2262883315 <== osd.1 v2:172.21.15.167:6800/30035 3 ==== osd_op_reply(3 default.realm [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 ==== 157+0+0 (crc 0 0 0) 0x7f32dc00a500 con 0x560a572b97f0 2023-10-05T19:28:34.941 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.934+0000 7f32eba3ce40 20 rados_obj.operate() r=-2 bl.length=0 2023-10-05T19:28:34.941 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.934+0000 7f32eba3ce40 20 rados->read ofs=0 len=0 2023-10-05T19:28:34.942 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.934+0000 7f32eba3ce40 1 -- 172.21.15.167:0/2262883315 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:4 5.f 5:f43b6ece:::zone_names.default:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x560a572ed820 con 0x560a572b97f0 2023-10-05T19:28:34.943 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.934+0000 7f32e9d72700 1 -- 172.21.15.167:0/2262883315 <== osd.1 v2:172.21.15.167:6800/30035 4 ==== osd_op_reply(4 zone_names.default [read 0~46 out=46b] v0'0 uv1 ondisk = 0) v8 ==== 162+0+46 (crc 0 0 0) 0x7f32dc00a500 con 0x560a572b97f0 2023-10-05T19:28:34.943 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.934+0000 7f32eba3ce40 20 rados_obj.operate() r=0 bl.length=46 2023-10-05T19:28:34.943 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.934+0000 7f32eba3ce40 20 rados->read ofs=0 len=0 2023-10-05T19:28:34.943 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.934+0000 7f32eba3ce40 1 -- 172.21.15.167:0/2262883315 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:5 5.1d 5:b8374eb4:::zone_info.9a9df965-6612-45d4-a45f-6e3c889bacc2:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x560a572eeaa0 con 0x560a572b97f0 2023-10-05T19:28:34.943 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.938+0000 7f32e9d72700 1 -- 172.21.15.167:0/2262883315 <== osd.1 v2:172.21.15.167:6800/30035 5 ==== osd_op_reply(5 zone_info.9a9df965-6612-45d4-a45f-6e3c889bacc2 [read 0~874 out=874b] v0'0 uv2 ondisk = 0) v8 ==== 190+0+874 (crc 0 0 0) 0x7f32dc00a500 con 0x560a572b97f0 2023-10-05T19:28:34.943 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.938+0000 7f32eba3ce40 20 rados_obj.operate() r=0 bl.length=874 2023-10-05T19:28:34.944 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.938+0000 7f32eba3ce40 20 searching for the correct realm 2023-10-05T19:28:34.944 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.938+0000 7f32eba3ce40 1 --2- 172.21.15.167:0/2262883315 >> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] conn(0x560a572f0860 0x560a572f2c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-05T19:28:34.944 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.938+0000 7f32eba3ce40 1 -- 172.21.15.167:0/2262883315 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:6 5.0 5:00000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x560a572f3180 con 0x560a572f0860 2023-10-05T19:28:34.944 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.938+0000 7f32e9571700 1 --2- 172.21.15.167:0/2262883315 >> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] conn(0x560a572f0860 0x560a572f2c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-05T19:28:34.944 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.938+0000 7f32e9571700 1 --2- 172.21.15.167:0/2262883315 >> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] conn(0x560a572f0860 0x560a572f2c40 crc :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-05T19:28:34.945 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.938+0000 7f32e9571700 1 -- 172.21.15.167:0/2262883315 <== osd.3 v2:172.21.15.167:6808/30036 1 ==== osd_op_reply(6 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f32e4024020 con 0x560a572f0860 2023-10-05T19:28:34.945 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.938+0000 7f32e9571700 1 --2- 172.21.15.167:0/2262883315 >> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] conn(0x7f32e4027be0 0x7f32e4028710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-05T19:28:34.945 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.938+0000 7f32e9571700 1 -- 172.21.15.167:0/2262883315 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:7 5.10 5:08000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x560a572f3180 con 0x7f32e4027be0 2023-10-05T19:28:34.946 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.938+0000 7f32e8d70700 1 --2- 172.21.15.167:0/2262883315 >> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] conn(0x7f32e4027be0 0x7f32e4028710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-05T19:28:34.946 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.938+0000 7f32e8d70700 1 --2- 172.21.15.167:0/2262883315 >> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] conn(0x7f32e4027be0 0x7f32e4028710 crc :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-05T19:28:34.946 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.942+0000 7f32e8d70700 1 -- 172.21.15.167:0/2262883315 <== osd.2 v2:172.21.15.167:6816/30037 1 ==== osd_op_reply(7 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f32e0008ae0 con 0x7f32e4027be0 2023-10-05T19:28:34.947 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.942+0000 7f32e8d70700 1 --2- 172.21.15.167:0/2262883315 >> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] conn(0x7f32e0007030 0x7f32e001c920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-05T19:28:34.947 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.942+0000 7f32e8d70700 1 -- 172.21.15.167:0/2262883315 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:8 5.8 5:10000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x560a572f3180 con 0x7f32e0007030 2023-10-05T19:28:34.947 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.942+0000 7f32e9d72700 1 --2- 172.21.15.167:0/2262883315 >> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] conn(0x7f32e0007030 0x7f32e001c920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-05T19:28:34.947 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.942+0000 7f32e9d72700 1 --2- 172.21.15.167:0/2262883315 >> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] conn(0x7f32e0007030 0x7f32e001c920 crc :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-05T19:28:34.948 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.942+0000 7f32e9d72700 1 -- 172.21.15.167:0/2262883315 <== osd.0 v2:172.21.15.118:6800/27690 1 ==== osd_op_reply(8 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f32dc014040 con 0x7f32e0007030 2023-10-05T19:28:34.948 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.942+0000 7f32e9d72700 1 -- 172.21.15.167:0/2262883315 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:9 5.18 5:18000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x560a572f3180 con 0x560a572b97f0 2023-10-05T19:28:34.949 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.942+0000 7f32e9d72700 1 -- 172.21.15.167:0/2262883315 <== osd.1 v2:172.21.15.167:6800/30035 6 ==== osd_op_reply(9 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f32dc014040 con 0x560a572b97f0 2023-10-05T19:28:34.949 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.942+0000 7f32e9d72700 1 -- 172.21.15.167:0/2262883315 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:10 5.4 5:20000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x560a572f3180 con 0x560a572b97f0 2023-10-05T19:28:34.949 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.942+0000 7f32e9d72700 1 -- 172.21.15.167:0/2262883315 <== osd.1 v2:172.21.15.167:6800/30035 7 ==== osd_op_reply(10 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f32dc014040 con 0x560a572b97f0 2023-10-05T19:28:34.950 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.942+0000 7f32e9d72700 1 -- 172.21.15.167:0/2262883315 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:11 5.14 5:28000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x560a572f3180 con 0x7f32e0007030 2023-10-05T19:28:34.950 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.946+0000 7f32e9d72700 1 -- 172.21.15.167:0/2262883315 <== osd.0 v2:172.21.15.118:6800/27690 2 ==== osd_op_reply(11 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f32dc014040 con 0x7f32e0007030 2023-10-05T19:28:34.950 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.946+0000 7f32e9d72700 1 -- 172.21.15.167:0/2262883315 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:12 5.c 5:30000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x560a572f3180 con 0x7f32e0007030 2023-10-05T19:28:34.951 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.946+0000 7f32e9d72700 1 -- 172.21.15.167:0/2262883315 <== osd.0 v2:172.21.15.118:6800/27690 3 ==== osd_op_reply(12 [pgnls start_epoch 30 out=112b] v23'1 uv1 ondisk = 1) v8 ==== 144+0+112 (crc 0 0 0) 0x7f32dc014040 con 0x7f32e0007030 2023-10-05T19:28:34.951 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.946+0000 7f32e9d72700 1 -- 172.21.15.167:0/2262883315 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:13 5.1c 5:38000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x560a572f3180 con 0x7f32e4027be0 2023-10-05T19:28:34.952 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.946+0000 7f32e8d70700 1 -- 172.21.15.167:0/2262883315 <== osd.2 v2:172.21.15.167:6816/30037 2 ==== osd_op_reply(13 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f32e0008ae0 con 0x7f32e4027be0 2023-10-05T19:28:34.952 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.946+0000 7f32e8d70700 1 -- 172.21.15.167:0/2262883315 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:14 5.2 5:40000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x560a572f3180 con 0x560a572f0860 2023-10-05T19:28:34.953 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.946+0000 7f32e9571700 1 -- 172.21.15.167:0/2262883315 <== osd.3 v2:172.21.15.167:6808/30036 2 ==== osd_op_reply(14 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f32e4024020 con 0x560a572f0860 2023-10-05T19:28:34.953 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.946+0000 7f32e9571700 1 -- 172.21.15.167:0/2262883315 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:15 5.12 5:48000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x560a572f3180 con 0x560a572b97f0 2023-10-05T19:28:34.953 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.946+0000 7f32e9d72700 1 -- 172.21.15.167:0/2262883315 <== osd.1 v2:172.21.15.167:6800/30035 8 ==== osd_op_reply(15 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f32dc014040 con 0x560a572b97f0 2023-10-05T19:28:34.954 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.946+0000 7f32e9d72700 1 -- 172.21.15.167:0/2262883315 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:16 5.a 5:50000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x560a572f3180 con 0x7f32e0007030 2023-10-05T19:28:34.954 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.950+0000 7f32e9d72700 1 -- 172.21.15.167:0/2262883315 <== osd.0 v2:172.21.15.118:6800/27690 4 ==== osd_op_reply(16 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f32dc022040 con 0x7f32e0007030 2023-10-05T19:28:34.954 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.950+0000 7f32e9d72700 1 -- 172.21.15.167:0/2262883315 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:17 5.1a 5:58000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x560a572f3180 con 0x560a572b97f0 2023-10-05T19:28:34.955 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.950+0000 7f32e9d72700 1 -- 172.21.15.167:0/2262883315 <== osd.1 v2:172.21.15.167:6800/30035 9 ==== osd_op_reply(17 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f32dc022040 con 0x560a572b97f0 2023-10-05T19:28:34.955 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.950+0000 7f32e9d72700 1 -- 172.21.15.167:0/2262883315 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:18 5.6 5:60000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x560a572f3180 con 0x7f32e4027be0 2023-10-05T19:28:34.956 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.950+0000 7f32e8d70700 1 -- 172.21.15.167:0/2262883315 <== osd.2 v2:172.21.15.167:6816/30037 3 ==== osd_op_reply(18 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f32e0008ae0 con 0x7f32e4027be0 2023-10-05T19:28:34.956 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.950+0000 7f32e8d70700 1 -- 172.21.15.167:0/2262883315 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:19 5.16 5:68000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x560a572f3180 con 0x560a572f0860 2023-10-05T19:28:34.957 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.950+0000 7f32e9571700 1 -- 172.21.15.167:0/2262883315 <== osd.3 v2:172.21.15.167:6808/30036 3 ==== osd_op_reply(19 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f32e4024020 con 0x560a572f0860 2023-10-05T19:28:34.957 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.950+0000 7f32e9571700 1 -- 172.21.15.167:0/2262883315 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:20 5.e 5:70000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x560a572f3180 con 0x560a572f0860 2023-10-05T19:28:34.958 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.950+0000 7f32e9571700 1 -- 172.21.15.167:0/2262883315 <== osd.3 v2:172.21.15.167:6808/30036 4 ==== osd_op_reply(20 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f32e4024020 con 0x560a572f0860 2023-10-05T19:28:34.958 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.950+0000 7f32e9571700 1 -- 172.21.15.167:0/2262883315 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:21 5.1e 5:78000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x560a572f3180 con 0x7f32e0007030 2023-10-05T19:28:34.959 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.954+0000 7f32e9d72700 1 -- 172.21.15.167:0/2262883315 <== osd.0 v2:172.21.15.118:6800/27690 5 ==== osd_op_reply(21 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f32dc022040 con 0x7f32e0007030 2023-10-05T19:28:34.959 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.954+0000 7f32e9d72700 1 -- 172.21.15.167:0/2262883315 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:22 5.1 5:80000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x560a572f3180 con 0x560a572f0860 2023-10-05T19:28:34.960 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.954+0000 7f32e9571700 1 -- 172.21.15.167:0/2262883315 <== osd.3 v2:172.21.15.167:6808/30036 5 ==== osd_op_reply(22 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f32e4024020 con 0x560a572f0860 2023-10-05T19:28:34.960 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.954+0000 7f32e9571700 1 -- 172.21.15.167:0/2262883315 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:23 5.11 5:88000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x560a572f3180 con 0x560a572f0860 2023-10-05T19:28:34.960 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.954+0000 7f32e9571700 1 -- 172.21.15.167:0/2262883315 <== osd.3 v2:172.21.15.167:6808/30036 6 ==== osd_op_reply(23 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f32e4024020 con 0x560a572f0860 2023-10-05T19:28:34.960 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.954+0000 7f32e9571700 1 -- 172.21.15.167:0/2262883315 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:24 5.9 5:90000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x560a572f3180 con 0x560a572b97f0 2023-10-05T19:28:34.961 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.954+0000 7f32e9d72700 1 -- 172.21.15.167:0/2262883315 <== osd.1 v2:172.21.15.167:6800/30035 10 ==== osd_op_reply(24 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f32dc022040 con 0x560a572b97f0 2023-10-05T19:28:34.961 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.954+0000 7f32e9d72700 1 -- 172.21.15.167:0/2262883315 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:25 5.19 5:98000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x560a572f3180 con 0x560a572b97f0 2023-10-05T19:28:34.962 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.958+0000 7f32e9d72700 1 -- 172.21.15.167:0/2262883315 <== osd.1 v2:172.21.15.167:6800/30035 11 ==== osd_op_reply(25 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f32dc022040 con 0x560a572b97f0 2023-10-05T19:28:34.962 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.958+0000 7f32e9d72700 1 -- 172.21.15.167:0/2262883315 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:26 5.5 5:a0000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x560a572f3180 con 0x7f32e0007030 2023-10-05T19:28:34.963 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.958+0000 7f32e9d72700 1 -- 172.21.15.167:0/2262883315 <== osd.0 v2:172.21.15.118:6800/27690 6 ==== osd_op_reply(26 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f32dc022040 con 0x7f32e0007030 2023-10-05T19:28:34.963 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.958+0000 7f32e9d72700 1 -- 172.21.15.167:0/2262883315 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:27 5.15 5:a8000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x560a572f3180 con 0x560a572f0860 2023-10-05T19:28:34.964 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.958+0000 7f32e9571700 1 -- 172.21.15.167:0/2262883315 <== osd.3 v2:172.21.15.167:6808/30036 7 ==== osd_op_reply(27 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f32e4024020 con 0x560a572f0860 2023-10-05T19:28:34.964 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.958+0000 7f32e9571700 1 -- 172.21.15.167:0/2262883315 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:28 5.d 5:b0000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x560a572f3180 con 0x7f32e4027be0 2023-10-05T19:28:34.965 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.958+0000 7f32e8d70700 1 -- 172.21.15.167:0/2262883315 <== osd.2 v2:172.21.15.167:6816/30037 4 ==== osd_op_reply(28 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f32e0008ae0 con 0x7f32e4027be0 2023-10-05T19:28:34.965 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.958+0000 7f32e8d70700 1 -- 172.21.15.167:0/2262883315 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:29 5.1d 5:b8000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x560a572f3180 con 0x560a572b97f0 2023-10-05T19:28:34.965 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.958+0000 7f32e9d72700 1 -- 172.21.15.167:0/2262883315 <== osd.1 v2:172.21.15.167:6800/30035 12 ==== osd_op_reply(29 [pgnls start_epoch 30 out=107b] v30'2 uv2 ondisk = 1) v8 ==== 144+0+107 (crc 0 0 0) 0x7f32dc022040 con 0x560a572b97f0 2023-10-05T19:28:34.966 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.958+0000 7f32e9d72700 1 -- 172.21.15.167:0/2262883315 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:30 5.3 5:c0000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x560a572f3180 con 0x7f32e0007030 2023-10-05T19:28:34.966 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.962+0000 7f32e9d72700 1 -- 172.21.15.167:0/2262883315 <== osd.0 v2:172.21.15.118:6800/27690 7 ==== osd_op_reply(30 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f32dc022040 con 0x7f32e0007030 2023-10-05T19:28:34.967 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.962+0000 7f32e9d72700 1 -- 172.21.15.167:0/2262883315 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:31 5.13 5:c8000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x560a572f3180 con 0x560a572f0860 2023-10-05T19:28:34.967 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.962+0000 7f32e9571700 1 -- 172.21.15.167:0/2262883315 <== osd.3 v2:172.21.15.167:6808/30036 8 ==== osd_op_reply(31 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f32e4024020 con 0x560a572f0860 2023-10-05T19:28:34.968 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.962+0000 7f32e9571700 1 -- 172.21.15.167:0/2262883315 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:32 5.b 5:d0000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x560a572f3180 con 0x7f32e0007030 2023-10-05T19:28:34.968 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.962+0000 7f32e9d72700 1 -- 172.21.15.167:0/2262883315 <== osd.0 v2:172.21.15.118:6800/27690 8 ==== osd_op_reply(32 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f32dc022040 con 0x7f32e0007030 2023-10-05T19:28:34.969 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.962+0000 7f32e9d72700 1 -- 172.21.15.167:0/2262883315 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:33 5.1b 5:d8000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x560a572f3180 con 0x7f32e4027be0 2023-10-05T19:28:34.969 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.962+0000 7f32e8d70700 1 -- 172.21.15.167:0/2262883315 <== osd.2 v2:172.21.15.167:6816/30037 5 ==== osd_op_reply(33 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f32e0008ae0 con 0x7f32e4027be0 2023-10-05T19:28:34.969 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.962+0000 7f32e8d70700 1 -- 172.21.15.167:0/2262883315 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:34 5.7 5:e0000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x560a572f3180 con 0x560a572f0860 2023-10-05T19:28:34.970 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.966+0000 7f32e9571700 1 -- 172.21.15.167:0/2262883315 <== osd.3 v2:172.21.15.167:6808/30036 9 ==== osd_op_reply(34 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f32e4024020 con 0x560a572f0860 2023-10-05T19:28:34.970 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.966+0000 7f32e9571700 1 -- 172.21.15.167:0/2262883315 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:35 5.17 5:e8000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x560a572f3180 con 0x7f32e0007030 2023-10-05T19:28:34.971 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.966+0000 7f32e9d72700 1 -- 172.21.15.167:0/2262883315 <== osd.0 v2:172.21.15.118:6800/27690 9 ==== osd_op_reply(35 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f32dc022040 con 0x7f32e0007030 2023-10-05T19:28:34.971 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.966+0000 7f32e9d72700 1 -- 172.21.15.167:0/2262883315 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:36 5.f 5:f0000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x560a572f3180 con 0x560a572b97f0 2023-10-05T19:28:34.972 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.966+0000 7f32e9d72700 1 -- 172.21.15.167:0/2262883315 <== osd.1 v2:172.21.15.167:6800/30035 13 ==== osd_op_reply(36 [pgnls start_epoch 30 out=115b] v23'2 uv2 ondisk = 1) v8 ==== 144+0+115 (crc 0 0 0) 0x7f32dc02f080 con 0x560a572b97f0 2023-10-05T19:28:34.972 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.966+0000 7f32e9d72700 1 -- 172.21.15.167:0/2262883315 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:37 5.1f 5:f8000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x560a572f3180 con 0x7f32e4027be0 2023-10-05T19:28:34.973 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.966+0000 7f32e8d70700 1 -- 172.21.15.167:0/2262883315 <== osd.2 v2:172.21.15.167:6816/30037 6 ==== osd_op_reply(37 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f32e0008ae0 con 0x7f32e4027be0 2023-10-05T19:28:34.973 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.966+0000 7f32eba3ce40 20 RGWRados::pool_iterate: got zonegroup_info.4f1f719d-9648-417b-b10c-e5d939825419 2023-10-05T19:28:34.973 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.966+0000 7f32eba3ce40 20 RGWRados::pool_iterate: got zone_info.9a9df965-6612-45d4-a45f-6e3c889bacc2 2023-10-05T19:28:34.974 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.966+0000 7f32eba3ce40 20 RGWRados::pool_iterate: got zone_names.default 2023-10-05T19:28:34.974 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.966+0000 7f32eba3ce40 20 RGWRados::pool_iterate: got zonegroups_names.default 2023-10-05T19:28:34.974 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.966+0000 7f32eba3ce40 20 rados->read ofs=0 len=0 2023-10-05T19:28:34.974 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.966+0000 7f32eba3ce40 1 -- 172.21.15.167:0/2262883315 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:38 5.12 5:49953fa1:::default.realm:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x560a572ef5e0 con 0x560a572b97f0 2023-10-05T19:28:34.975 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.970+0000 7f32e9d72700 1 -- 172.21.15.167:0/2262883315 <== osd.1 v2:172.21.15.167:6800/30035 14 ==== osd_op_reply(38 default.realm [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 ==== 157+0+0 (crc 0 0 0) 0x7f32dc02f080 con 0x560a572b97f0 2023-10-05T19:28:34.975 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.970+0000 7f32eba3ce40 20 rados_obj.operate() r=-2 bl.length=0 2023-10-05T19:28:34.975 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.970+0000 7f32eba3ce40 20 rados->read ofs=0 len=0 2023-10-05T19:28:34.975 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.970+0000 7f32eba3ce40 1 -- 172.21.15.167:0/2262883315 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:39 5.f 5:f4c53578:::zonegroups_names.default:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x560a572efed0 con 0x560a572b97f0 2023-10-05T19:28:34.975 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.970+0000 7f32e9d72700 1 -- 172.21.15.167:0/2262883315 <== osd.1 v2:172.21.15.167:6800/30035 15 ==== osd_op_reply(39 zonegroups_names.default [read 0~46 out=46b] v0'0 uv2 ondisk = 0) v8 ==== 168+0+46 (crc 0 0 0) 0x7f32dc02f080 con 0x560a572b97f0 2023-10-05T19:28:34.975 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.970+0000 7f32eba3ce40 20 rados_obj.operate() r=0 bl.length=46 2023-10-05T19:28:34.976 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.970+0000 7f32eba3ce40 20 rados->read ofs=0 len=0 2023-10-05T19:28:34.976 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.970+0000 7f32eba3ce40 1 -- 172.21.15.167:0/2262883315 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:40 5.c 5:30d299a5:::zonegroup_info.4f1f719d-9648-417b-b10c-e5d939825419:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x560a572f35c0 con 0x7f32e0007030 2023-10-05T19:28:34.976 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.970+0000 7f32e9d72700 1 -- 172.21.15.167:0/2262883315 <== osd.0 v2:172.21.15.118:6800/27690 10 ==== osd_op_reply(40 zonegroup_info.4f1f719d-9648-417b-b10c-e5d939825419 [read 0~362 out=362b] v0'0 uv1 ondisk = 0) v8 ==== 195+0+362 (crc 0 0 0) 0x7f32dc02f080 con 0x7f32e0007030 2023-10-05T19:28:34.977 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.970+0000 7f32eba3ce40 20 rados_obj.operate() r=0 bl.length=362 2023-10-05T19:28:34.977 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.970+0000 7f32eba3ce40 20 rados->read ofs=0 len=0 2023-10-05T19:28:34.977 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.970+0000 7f32eba3ce40 1 -- 172.21.15.167:0/2262883315 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:41 5.13 5:cbbf6c60:::converted:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x560a572f35c0 con 0x560a572f0860 2023-10-05T19:28:34.977 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.970+0000 7f32e9571700 1 -- 172.21.15.167:0/2262883315 <== osd.3 v2:172.21.15.167:6808/30036 10 ==== osd_op_reply(41 converted [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 ==== 153+0+0 (crc 0 0 0) 0x7f32e4024020 con 0x560a572f0860 2023-10-05T19:28:34.977 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.970+0000 7f32eba3ce40 20 rados_obj.operate() r=-2 bl.length=0 2023-10-05T19:28:34.977 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.970+0000 7f32eba3ce40 20 rados->read ofs=0 len=0 2023-10-05T19:28:34.977 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.970+0000 7f32eba3ce40 1 -- 172.21.15.167:0/2262883315 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:42 5.12 5:49953fa1:::default.realm:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x560a572f3970 con 0x560a572b97f0 2023-10-05T19:28:34.978 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.970+0000 7f32e9d72700 1 -- 172.21.15.167:0/2262883315 <== osd.1 v2:172.21.15.167:6800/30035 16 ==== osd_op_reply(42 default.realm [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 ==== 157+0+0 (crc 0 0 0) 0x7f32dc02f080 con 0x560a572b97f0 2023-10-05T19:28:34.978 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.970+0000 7f32eba3ce40 20 rados_obj.operate() r=-2 bl.length=0 2023-10-05T19:28:34.978 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.970+0000 7f32eba3ce40 20 rados->read ofs=0 len=0 2023-10-05T19:28:34.978 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.970+0000 7f32eba3ce40 1 -- 172.21.15.167:0/2262883315 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:43 5.f 5:f4c53578:::zonegroups_names.default:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x560a572efad0 con 0x560a572b97f0 2023-10-05T19:28:34.979 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.974+0000 7f32e9d72700 1 -- 172.21.15.167:0/2262883315 <== osd.1 v2:172.21.15.167:6800/30035 17 ==== osd_op_reply(43 zonegroups_names.default [read 0~46 out=46b] v0'0 uv2 ondisk = 0) v8 ==== 168+0+46 (crc 0 0 0) 0x7f32dc02f080 con 0x560a572b97f0 2023-10-05T19:28:34.979 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.974+0000 7f32eba3ce40 20 rados_obj.operate() r=0 bl.length=46 2023-10-05T19:28:34.979 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.974+0000 7f32eba3ce40 1 -- 172.21.15.167:0/2262883315 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:44 5.0 5:00000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x560a572efaa0 con 0x560a572f0860 2023-10-05T19:28:34.981 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.974+0000 7f32e9571700 1 -- 172.21.15.167:0/2262883315 <== osd.3 v2:172.21.15.167:6808/30036 11 ==== osd_op_reply(44 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f32e4024020 con 0x560a572f0860 2023-10-05T19:28:34.981 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.974+0000 7f32e9571700 1 -- 172.21.15.167:0/2262883315 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:45 5.10 5:08000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x560a572efaa0 con 0x7f32e4027be0 2023-10-05T19:28:34.982 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.978+0000 7f32e8d70700 1 -- 172.21.15.167:0/2262883315 <== osd.2 v2:172.21.15.167:6816/30037 7 ==== osd_op_reply(45 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f32e0008ae0 con 0x7f32e4027be0 2023-10-05T19:28:34.982 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.978+0000 7f32e8d70700 1 -- 172.21.15.167:0/2262883315 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:46 5.8 5:10000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x560a572efaa0 con 0x7f32e0007030 2023-10-05T19:28:34.983 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.978+0000 7f32e9d72700 1 -- 172.21.15.167:0/2262883315 <== osd.0 v2:172.21.15.118:6800/27690 11 ==== osd_op_reply(46 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f32dc02f080 con 0x7f32e0007030 2023-10-05T19:28:34.983 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.978+0000 7f32e9d72700 1 -- 172.21.15.167:0/2262883315 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:47 5.18 5:18000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x560a572efed0 con 0x560a572b97f0 2023-10-05T19:28:34.984 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.978+0000 7f32e9d72700 1 -- 172.21.15.167:0/2262883315 <== osd.1 v2:172.21.15.167:6800/30035 18 ==== osd_op_reply(47 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f32dc02f080 con 0x560a572b97f0 2023-10-05T19:28:34.984 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.978+0000 7f32e9d72700 1 -- 172.21.15.167:0/2262883315 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:48 5.4 5:20000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x560a572efed0 con 0x560a572b97f0 2023-10-05T19:28:34.984 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.978+0000 7f32e9d72700 1 -- 172.21.15.167:0/2262883315 <== osd.1 v2:172.21.15.167:6800/30035 19 ==== osd_op_reply(48 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f32dc02f080 con 0x560a572b97f0 2023-10-05T19:28:34.984 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.978+0000 7f32e9d72700 1 -- 172.21.15.167:0/2262883315 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:49 5.14 5:28000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x560a572efed0 con 0x7f32e0007030 2023-10-05T19:28:34.985 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.978+0000 7f32e9d72700 1 -- 172.21.15.167:0/2262883315 <== osd.0 v2:172.21.15.118:6800/27690 12 ==== osd_op_reply(49 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f32dc02f080 con 0x7f32e0007030 2023-10-05T19:28:34.985 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.978+0000 7f32e9d72700 1 -- 172.21.15.167:0/2262883315 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:50 5.c 5:30000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x560a572efed0 con 0x7f32e0007030 2023-10-05T19:28:34.986 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.982+0000 7f32e9d72700 1 -- 172.21.15.167:0/2262883315 <== osd.0 v2:172.21.15.118:6800/27690 13 ==== osd_op_reply(50 [pgnls start_epoch 30 out=112b] v23'1 uv1 ondisk = 1) v8 ==== 144+0+112 (crc 0 0 0) 0x7f32dc02f080 con 0x7f32e0007030 2023-10-05T19:28:34.986 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.982+0000 7f32e9d72700 1 -- 172.21.15.167:0/2262883315 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:51 5.1c 5:38000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x560a572efed0 con 0x7f32e4027be0 2023-10-05T19:28:34.987 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.982+0000 7f32e8d70700 1 -- 172.21.15.167:0/2262883315 <== osd.2 v2:172.21.15.167:6816/30037 8 ==== osd_op_reply(51 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f32e0008ae0 con 0x7f32e4027be0 2023-10-05T19:28:34.988 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.982+0000 7f32e8d70700 1 -- 172.21.15.167:0/2262883315 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:52 5.2 5:40000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x560a572efed0 con 0x560a572f0860 2023-10-05T19:28:34.988 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.982+0000 7f32e9571700 1 -- 172.21.15.167:0/2262883315 <== osd.3 v2:172.21.15.167:6808/30036 12 ==== osd_op_reply(52 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f32e4024020 con 0x560a572f0860 2023-10-05T19:28:34.988 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.982+0000 7f32e9571700 1 -- 172.21.15.167:0/2262883315 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:53 5.12 5:48000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x560a572efed0 con 0x560a572b97f0 2023-10-05T19:28:34.989 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.982+0000 7f32e9d72700 1 -- 172.21.15.167:0/2262883315 <== osd.1 v2:172.21.15.167:6800/30035 20 ==== osd_op_reply(53 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f32dc02f080 con 0x560a572b97f0 2023-10-05T19:28:34.989 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.982+0000 7f32e9d72700 1 -- 172.21.15.167:0/2262883315 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:54 5.a 5:50000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x560a572efed0 con 0x7f32e0007030 2023-10-05T19:28:34.990 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.986+0000 7f32e9d72700 1 -- 172.21.15.167:0/2262883315 <== osd.0 v2:172.21.15.118:6800/27690 14 ==== osd_op_reply(54 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f32dc02f080 con 0x7f32e0007030 2023-10-05T19:28:34.990 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.986+0000 7f32e9d72700 1 -- 172.21.15.167:0/2262883315 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:55 5.1a 5:58000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x560a572efed0 con 0x560a572b97f0 2023-10-05T19:28:34.991 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.986+0000 7f32e9d72700 1 -- 172.21.15.167:0/2262883315 <== osd.1 v2:172.21.15.167:6800/30035 21 ==== osd_op_reply(55 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f32dc02f080 con 0x560a572b97f0 2023-10-05T19:28:34.991 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.986+0000 7f32e9d72700 1 -- 172.21.15.167:0/2262883315 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:56 5.6 5:60000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x560a572efed0 con 0x7f32e4027be0 2023-10-05T19:28:34.992 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.986+0000 7f32e8d70700 1 -- 172.21.15.167:0/2262883315 <== osd.2 v2:172.21.15.167:6816/30037 9 ==== osd_op_reply(56 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f32e0008ae0 con 0x7f32e4027be0 2023-10-05T19:28:34.992 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.986+0000 7f32e8d70700 1 -- 172.21.15.167:0/2262883315 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:57 5.16 5:68000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x560a572efed0 con 0x560a572f0860 2023-10-05T19:28:34.992 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.986+0000 7f32e9571700 1 -- 172.21.15.167:0/2262883315 <== osd.3 v2:172.21.15.167:6808/30036 13 ==== osd_op_reply(57 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f32e4024020 con 0x560a572f0860 2023-10-05T19:28:34.992 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.986+0000 7f32e9571700 1 -- 172.21.15.167:0/2262883315 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:58 5.e 5:70000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x560a572efed0 con 0x560a572f0860 2023-10-05T19:28:34.993 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.986+0000 7f32e9571700 1 -- 172.21.15.167:0/2262883315 <== osd.3 v2:172.21.15.167:6808/30036 14 ==== osd_op_reply(58 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f32e4024020 con 0x560a572f0860 2023-10-05T19:28:34.993 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.986+0000 7f32e9571700 1 -- 172.21.15.167:0/2262883315 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:59 5.1e 5:78000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x560a572efed0 con 0x7f32e0007030 2023-10-05T19:28:34.994 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.990+0000 7f32e9d72700 1 -- 172.21.15.167:0/2262883315 <== osd.0 v2:172.21.15.118:6800/27690 15 ==== osd_op_reply(59 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f32dc02f080 con 0x7f32e0007030 2023-10-05T19:28:34.994 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.990+0000 7f32e9d72700 1 -- 172.21.15.167:0/2262883315 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:60 5.1 5:80000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x560a572efed0 con 0x560a572f0860 2023-10-05T19:28:34.995 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.990+0000 7f32e9571700 1 -- 172.21.15.167:0/2262883315 <== osd.3 v2:172.21.15.167:6808/30036 15 ==== osd_op_reply(60 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f32e4024020 con 0x560a572f0860 2023-10-05T19:28:34.995 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.990+0000 7f32e9571700 1 -- 172.21.15.167:0/2262883315 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:61 5.11 5:88000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x560a572efed0 con 0x560a572f0860 2023-10-05T19:28:34.995 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.990+0000 7f32e9571700 1 -- 172.21.15.167:0/2262883315 <== osd.3 v2:172.21.15.167:6808/30036 16 ==== osd_op_reply(61 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f32e4024020 con 0x560a572f0860 2023-10-05T19:28:34.995 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.990+0000 7f32e9571700 1 -- 172.21.15.167:0/2262883315 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:62 5.9 5:90000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x560a572efed0 con 0x560a572b97f0 2023-10-05T19:28:34.996 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.990+0000 7f32e9d72700 1 -- 172.21.15.167:0/2262883315 <== osd.1 v2:172.21.15.167:6800/30035 22 ==== osd_op_reply(62 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f32dc02f080 con 0x560a572b97f0 2023-10-05T19:28:34.997 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.990+0000 7f32e9d72700 1 -- 172.21.15.167:0/2262883315 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:63 5.19 5:98000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x560a572efed0 con 0x560a572b97f0 2023-10-05T19:28:34.997 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.990+0000 7f32e9d72700 1 -- 172.21.15.167:0/2262883315 <== osd.1 v2:172.21.15.167:6800/30035 23 ==== osd_op_reply(63 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f32dc02f080 con 0x560a572b97f0 2023-10-05T19:28:34.997 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.990+0000 7f32e9d72700 1 -- 172.21.15.167:0/2262883315 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:64 5.5 5:a0000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x560a572efed0 con 0x7f32e0007030 2023-10-05T19:28:34.998 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.994+0000 7f32e9d72700 1 -- 172.21.15.167:0/2262883315 <== osd.0 v2:172.21.15.118:6800/27690 16 ==== osd_op_reply(64 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f32dc02f080 con 0x7f32e0007030 2023-10-05T19:28:34.999 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.994+0000 7f32e9d72700 1 -- 172.21.15.167:0/2262883315 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:65 5.15 5:a8000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x560a572efed0 con 0x560a572f0860 2023-10-05T19:28:34.999 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.994+0000 7f32e9571700 1 -- 172.21.15.167:0/2262883315 <== osd.3 v2:172.21.15.167:6808/30036 17 ==== osd_op_reply(65 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f32e4024020 con 0x560a572f0860 2023-10-05T19:28:34.999 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.994+0000 7f32e9571700 1 -- 172.21.15.167:0/2262883315 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:66 5.d 5:b0000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x560a572efed0 con 0x7f32e4027be0 2023-10-05T19:28:35.000 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.994+0000 7f32e8d70700 1 -- 172.21.15.167:0/2262883315 <== osd.2 v2:172.21.15.167:6816/30037 10 ==== osd_op_reply(66 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f32e0008ae0 con 0x7f32e4027be0 2023-10-05T19:28:35.000 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.994+0000 7f32e8d70700 1 -- 172.21.15.167:0/2262883315 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:67 5.1d 5:b8000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x560a572efed0 con 0x560a572b97f0 2023-10-05T19:28:35.001 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.994+0000 7f32e9d72700 1 -- 172.21.15.167:0/2262883315 <== osd.1 v2:172.21.15.167:6800/30035 24 ==== osd_op_reply(67 [pgnls start_epoch 30 out=107b] v30'2 uv2 ondisk = 1) v8 ==== 144+0+107 (crc 0 0 0) 0x7f32dc02f080 con 0x560a572b97f0 2023-10-05T19:28:35.001 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.994+0000 7f32e9d72700 1 -- 172.21.15.167:0/2262883315 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:68 5.3 5:c0000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x560a572efed0 con 0x7f32e0007030 2023-10-05T19:28:35.002 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.998+0000 7f32e9d72700 1 -- 172.21.15.167:0/2262883315 <== osd.0 v2:172.21.15.118:6800/27690 17 ==== osd_op_reply(68 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f32dc02f080 con 0x7f32e0007030 2023-10-05T19:28:35.002 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.998+0000 7f32e9d72700 1 -- 172.21.15.167:0/2262883315 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:69 5.13 5:c8000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x560a572efed0 con 0x560a572f0860 2023-10-05T19:28:35.003 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.998+0000 7f32e9571700 1 -- 172.21.15.167:0/2262883315 <== osd.3 v2:172.21.15.167:6808/30036 18 ==== osd_op_reply(69 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f32e4024020 con 0x560a572f0860 2023-10-05T19:28:35.003 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.998+0000 7f32e9571700 1 -- 172.21.15.167:0/2262883315 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:70 5.b 5:d0000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x560a572efed0 con 0x7f32e0007030 2023-10-05T19:28:35.004 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.998+0000 7f32e9d72700 1 -- 172.21.15.167:0/2262883315 <== osd.0 v2:172.21.15.118:6800/27690 18 ==== osd_op_reply(70 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f32dc02f080 con 0x7f32e0007030 2023-10-05T19:28:35.004 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.998+0000 7f32e9d72700 1 -- 172.21.15.167:0/2262883315 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:71 5.1b 5:d8000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x560a572efed0 con 0x7f32e4027be0 2023-10-05T19:28:35.005 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.998+0000 7f32e8d70700 1 -- 172.21.15.167:0/2262883315 <== osd.2 v2:172.21.15.167:6816/30037 11 ==== osd_op_reply(71 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f32e0008ae0 con 0x7f32e4027be0 2023-10-05T19:28:35.005 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.998+0000 7f32e8d70700 1 -- 172.21.15.167:0/2262883315 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:72 5.7 5:e0000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x560a572efed0 con 0x560a572f0860 2023-10-05T19:28:35.005 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.998+0000 7f32e9571700 1 -- 172.21.15.167:0/2262883315 <== osd.3 v2:172.21.15.167:6808/30036 19 ==== osd_op_reply(72 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f32e4024020 con 0x560a572f0860 2023-10-05T19:28:35.006 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:34.998+0000 7f32e9571700 1 -- 172.21.15.167:0/2262883315 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:73 5.17 5:e8000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x560a572efed0 con 0x7f32e0007030 2023-10-05T19:28:35.006 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.002+0000 7f32e9d72700 1 -- 172.21.15.167:0/2262883315 <== osd.0 v2:172.21.15.118:6800/27690 19 ==== osd_op_reply(73 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f32dc02f080 con 0x7f32e0007030 2023-10-05T19:28:35.007 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.002+0000 7f32e9d72700 1 -- 172.21.15.167:0/2262883315 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:74 5.f 5:f0000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x560a572efed0 con 0x560a572b97f0 2023-10-05T19:28:35.007 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.002+0000 7f32e9d72700 1 -- 172.21.15.167:0/2262883315 <== osd.1 v2:172.21.15.167:6800/30035 25 ==== osd_op_reply(74 [pgnls start_epoch 30 out=115b] v23'2 uv2 ondisk = 1) v8 ==== 144+0+115 (crc 0 0 0) 0x7f32dc02f080 con 0x560a572b97f0 2023-10-05T19:28:35.007 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.002+0000 7f32e9d72700 1 -- 172.21.15.167:0/2262883315 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:75 5.1f 5:f8000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x560a572efed0 con 0x7f32e4027be0 2023-10-05T19:28:35.008 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.002+0000 7f32e8d70700 1 -- 172.21.15.167:0/2262883315 <== osd.2 v2:172.21.15.167:6816/30037 12 ==== osd_op_reply(75 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f32e0008ae0 con 0x7f32e4027be0 2023-10-05T19:28:35.009 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.002+0000 7f32eba3ce40 20 RGWRados::pool_iterate: got zonegroup_info.4f1f719d-9648-417b-b10c-e5d939825419 2023-10-05T19:28:35.009 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.002+0000 7f32eba3ce40 20 RGWRados::pool_iterate: got zone_info.9a9df965-6612-45d4-a45f-6e3c889bacc2 2023-10-05T19:28:35.009 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.002+0000 7f32eba3ce40 20 RGWRados::pool_iterate: got zone_names.default 2023-10-05T19:28:35.009 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.002+0000 7f32eba3ce40 20 RGWRados::pool_iterate: got zonegroups_names.default 2023-10-05T19:28:35.009 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.002+0000 7f32eba3ce40 20 rados->read ofs=0 len=0 2023-10-05T19:28:35.009 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.002+0000 7f32eba3ce40 1 -- 172.21.15.167:0/2262883315 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:76 5.f 5:f43b6ece:::zone_names.default:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x560a572efaa0 con 0x560a572b97f0 2023-10-05T19:28:35.010 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.002+0000 7f32e9d72700 1 -- 172.21.15.167:0/2262883315 <== osd.1 v2:172.21.15.167:6800/30035 26 ==== osd_op_reply(76 zone_names.default [read 0~46 out=46b] v0'0 uv1 ondisk = 0) v8 ==== 162+0+46 (crc 0 0 0) 0x7f32dc02f080 con 0x560a572b97f0 2023-10-05T19:28:35.010 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.002+0000 7f32eba3ce40 20 rados_obj.operate() r=0 bl.length=46 2023-10-05T19:28:35.010 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.002+0000 7f32eba3ce40 20 rados->read ofs=0 len=0 2023-10-05T19:28:35.011 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.002+0000 7f32eba3ce40 1 -- 172.21.15.167:0/2262883315 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:77 5.1d 5:b8374eb4:::zone_info.9a9df965-6612-45d4-a45f-6e3c889bacc2:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x560a572f3eb0 con 0x560a572b97f0 2023-10-05T19:28:35.011 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.006+0000 7f32e9d72700 1 -- 172.21.15.167:0/2262883315 <== osd.1 v2:172.21.15.167:6800/30035 27 ==== osd_op_reply(77 zone_info.9a9df965-6612-45d4-a45f-6e3c889bacc2 [read 0~874 out=874b] v0'0 uv2 ondisk = 0) v8 ==== 190+0+874 (crc 0 0 0) 0x7f32dc02f080 con 0x560a572b97f0 2023-10-05T19:28:35.011 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.006+0000 7f32eba3ce40 20 rados_obj.operate() r=0 bl.length=874 2023-10-05T19:28:35.011 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.006+0000 7f32eba3ce40 20 rados->read ofs=0 len=0 2023-10-05T19:28:35.011 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.006+0000 7f32eba3ce40 1 -- 172.21.15.167:0/2262883315 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:78 5.f 5:f4c53578:::zonegroups_names.default:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x560a572f43a0 con 0x560a572b97f0 2023-10-05T19:28:35.012 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.006+0000 7f32e9d72700 1 -- 172.21.15.167:0/2262883315 <== osd.1 v2:172.21.15.167:6800/30035 28 ==== osd_op_reply(78 zonegroups_names.default [read 0~46 out=46b] v0'0 uv2 ondisk = 0) v8 ==== 168+0+46 (crc 0 0 0) 0x7f32dc02f080 con 0x560a572b97f0 2023-10-05T19:28:35.012 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.006+0000 7f32eba3ce40 20 rados_obj.operate() r=0 bl.length=46 2023-10-05T19:28:35.012 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.006+0000 7f32eba3ce40 20 rados->read ofs=0 len=0 2023-10-05T19:28:35.013 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.006+0000 7f32eba3ce40 1 -- 172.21.15.167:0/2262883315 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:79 5.c 5:30d299a5:::zonegroup_info.4f1f719d-9648-417b-b10c-e5d939825419:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x560a572f43a0 con 0x7f32e0007030 2023-10-05T19:28:35.013 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.006+0000 7f32e9d72700 1 -- 172.21.15.167:0/2262883315 <== osd.0 v2:172.21.15.118:6800/27690 20 ==== osd_op_reply(79 zonegroup_info.4f1f719d-9648-417b-b10c-e5d939825419 [read 0~362 out=362b] v0'0 uv1 ondisk = 0) v8 ==== 195+0+362 (crc 0 0 0) 0x7f32dc02f080 con 0x7f32e0007030 2023-10-05T19:28:35.013 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.006+0000 7f32eba3ce40 20 rados_obj.operate() r=0 bl.length=362 2023-10-05T19:28:35.013 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.006+0000 7f32eba3ce40 20 rados->read ofs=0 len=0 2023-10-05T19:28:35.013 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.006+0000 7f32eba3ce40 1 -- 172.21.15.167:0/2262883315 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:80 5.3 5:c0ad8cbd:::region_map:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x560a572f43a0 con 0x7f32e0007030 2023-10-05T19:28:35.014 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.006+0000 7f32e9d72700 1 -- 172.21.15.167:0/2262883315 <== osd.0 v2:172.21.15.118:6800/27690 21 ==== osd_op_reply(80 region_map [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 ==== 154+0+0 (crc 0 0 0) 0x7f32dc02f080 con 0x7f32e0007030 2023-10-05T19:28:35.014 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.006+0000 7f32eba3ce40 20 rados_obj.operate() r=-2 bl.length=0 2023-10-05T19:28:35.014 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.006+0000 7f32eba3ce40 20 zone default found 2023-10-05T19:28:35.015 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.006+0000 7f32eba3ce40 4 Realm: () 2023-10-05T19:28:35.015 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.006+0000 7f32eba3ce40 4 ZoneGroup: default (4f1f719d-9648-417b-b10c-e5d939825419) 2023-10-05T19:28:35.015 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.006+0000 7f32eba3ce40 4 Zone: default (9a9df965-6612-45d4-a45f-6e3c889bacc2) 2023-10-05T19:28:35.015 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.006+0000 7f32eba3ce40 10 cannot find current period zonegroup using local zonegroup configuration 2023-10-05T19:28:35.015 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.006+0000 7f32eba3ce40 20 zonegroup default 2023-10-05T19:28:35.015 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.006+0000 7f32eba3ce40 20 rados->read ofs=0 len=0 2023-10-05T19:28:35.015 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.006+0000 7f32eba3ce40 1 -- 172.21.15.167:0/2262883315 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:81 5.3 5:c52100b6:::period_config.default:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x560a572f43a0 con 0x7f32e0007030 2023-10-05T19:28:35.015 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.010+0000 7f32e9d72700 1 -- 172.21.15.167:0/2262883315 <== osd.0 v2:172.21.15.118:6800/27690 22 ==== osd_op_reply(81 period_config.default [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 ==== 165+0+0 (crc 0 0 0) 0x7f32dc050040 con 0x7f32e0007030 2023-10-05T19:28:35.016 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.010+0000 7f32eba3ce40 20 rados_obj.operate() r=-2 bl.length=0 2023-10-05T19:28:35.016 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.010+0000 7f32eba3ce40 20 rados->read ofs=0 len=0 2023-10-05T19:28:35.016 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.010+0000 7f32eba3ce40 1 -- 172.21.15.167:0/2262883315 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:82 6.2 6:4347d321:::bucket.sync-source-hints.:head [call version.read in=11b,read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x560a572f5d20 con 0x560a572f0860 2023-10-05T19:28:35.017 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.010+0000 7f32e9571700 1 -- 172.21.15.167:0/2262883315 <== osd.3 v2:172.21.15.167:6808/30036 20 ==== osd_op_reply(82 bucket.sync-source-hints. [call,read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 ==== 211+0+0 (crc 0 0 0) 0x7f32e4024020 con 0x560a572f0860 2023-10-05T19:28:35.017 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.010+0000 7f32eba3ce40 20 rados_obj.operate() r=-2 bl.length=0 2023-10-05T19:28:35.017 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.010+0000 7f32eba3ce40 20 rados->read ofs=0 len=0 2023-10-05T19:28:35.017 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.010+0000 7f32eba3ce40 1 -- 172.21.15.167:0/2262883315 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:83 6.b 6:d467b91b:::bucket.sync-target-hints.:head [call version.read in=11b,read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x560a572f60d0 con 0x560a572f0860 2023-10-05T19:28:35.017 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.010+0000 7f32e9571700 1 -- 172.21.15.167:0/2262883315 <== osd.3 v2:172.21.15.167:6808/30036 21 ==== osd_op_reply(83 bucket.sync-target-hints. [call,read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 ==== 211+0+0 (crc 0 0 0) 0x7f32e4024020 con 0x560a572f0860 2023-10-05T19:28:35.017 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.010+0000 7f32eba3ce40 20 rados_obj.operate() r=-2 bl.length=0 2023-10-05T19:28:35.017 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.010+0000 7f32eba3ce40 20 started sync module instance, tier type = 2023-10-05T19:28:35.018 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.010+0000 7f32eba3ce40 20 started zone id=9a9df965-6612-45d4-a45f-6e3c889bacc2 (name=default) with tier type = 2023-10-05T19:28:35.018 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.010+0000 7f32eba3ce40 1 -- 172.21.15.167:0/2262883315 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:84 7.1f 7:f95f44c2:::notify.0:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x560a572f53e0 con 0x7f32e4027be0 2023-10-05T19:28:35.019 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.014+0000 7f32e8d70700 1 -- 172.21.15.167:0/2262883315 <== osd.2 v2:172.21.15.167:6816/30037 13 ==== osd_op_reply(84 notify.0 [create] v30'138 uv138 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f32e0008ae0 con 0x7f32e4027be0 2023-10-05T19:28:35.019 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.014+0000 7f32eba3ce40 1 -- 172.21.15.167:0/2262883315 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:85 7.1f 7:f95f44c2:::notify.0:head [watch watch cookie 94602412382176] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x560a572f5010 con 0x7f32e4027be0 2023-10-05T19:28:35.019 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.014+0000 7f32eba3ce40 1 -- 172.21.15.167:0/2262883315 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:86 7.0 7:05bf5b68:::notify.1:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x560a572f6e90 con 0x560a572b97f0 2023-10-05T19:28:35.022 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.014+0000 7f32e9d72700 1 -- 172.21.15.167:0/2262883315 <== osd.1 v2:172.21.15.167:6800/30035 29 ==== osd_op_reply(86 notify.1 [create] v30'138 uv138 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f32dc050040 con 0x560a572b97f0 2023-10-05T19:28:35.022 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.014+0000 7f32e8d70700 1 -- 172.21.15.167:0/2262883315 <== osd.2 v2:172.21.15.167:6816/30037 14 ==== osd_op_reply(85 notify.0 [watch watch cookie 94602412382176] v30'139 uv138 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f32e0008ae0 con 0x7f32e4027be0 2023-10-05T19:28:35.022 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.014+0000 7f32eba3ce40 1 -- 172.21.15.167:0/2262883315 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:87 7.0 7:05bf5b68:::notify.1:head [watch watch cookie 94602412387008] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x560a572f7780 con 0x560a572b97f0 2023-10-05T19:28:35.022 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.014+0000 7f32eba3ce40 1 -- 172.21.15.167:0/2262883315 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:88 7.15 7:a93a5511:::notify.2:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x560a572f7b30 con 0x560a572f0860 2023-10-05T19:28:35.025 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.018+0000 7f32e9571700 1 -- 172.21.15.167:0/2262883315 <== osd.3 v2:172.21.15.167:6808/30036 22 ==== osd_op_reply(88 notify.2 [create] v30'138 uv138 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f32e4024020 con 0x560a572f0860 2023-10-05T19:28:35.025 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.018+0000 7f32eba3ce40 1 -- 172.21.15.167:0/2262883315 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:89 7.15 7:a93a5511:::notify.2:head [watch watch cookie 94602412385408] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x560a572f83d0 con 0x560a572f0860 2023-10-05T19:28:35.025 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.018+0000 7f32eba3ce40 1 -- 172.21.15.167:0/2262883315 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:90 7.e 7:7759931f:::notify.3:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x560a572f8c70 con 0x7f32e4027be0 2023-10-05T19:28:35.026 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.018+0000 7f32e9d72700 1 -- 172.21.15.167:0/2262883315 <== osd.1 v2:172.21.15.167:6800/30035 30 ==== osd_op_reply(87 notify.1 [watch watch cookie 94602412387008] v30'139 uv138 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f32dc050040 con 0x560a572b97f0 2023-10-05T19:28:35.028 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.022+0000 7f32e8d70700 1 -- 172.21.15.167:0/2262883315 <== osd.2 v2:172.21.15.167:6816/30037 15 ==== osd_op_reply(90 notify.3 [create] v30'138 uv138 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f32e0008ae0 con 0x7f32e4027be0 2023-10-05T19:28:35.028 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.022+0000 7f32eba3ce40 1 -- 172.21.15.167:0/2262883315 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:91 7.e 7:7759931f:::notify.3:head [watch watch cookie 94602412389008] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x560a572f88a0 con 0x7f32e4027be0 2023-10-05T19:28:35.028 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.022+0000 7f32eba3ce40 1 -- 172.21.15.167:0/2262883315 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:92 7.d 7:b4812045:::notify.4:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x560a572f9c80 con 0x560a572b97f0 2023-10-05T19:28:35.028 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.022+0000 7f32e9571700 1 -- 172.21.15.167:0/2262883315 <== osd.3 v2:172.21.15.167:6808/30036 23 ==== osd_op_reply(89 notify.2 [watch watch cookie 94602412385408] v30'139 uv138 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f32e4024020 con 0x560a572f0860 2023-10-05T19:28:35.030 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.026+0000 7f32e8d70700 1 -- 172.21.15.167:0/2262883315 <== osd.2 v2:172.21.15.167:6816/30037 16 ==== osd_op_reply(91 notify.3 [watch watch cookie 94602412389008] v30'139 uv138 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f32e0008ae0 con 0x7f32e4027be0 2023-10-05T19:28:35.031 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.026+0000 7f32e9d72700 1 -- 172.21.15.167:0/2262883315 <== osd.1 v2:172.21.15.167:6800/30035 31 ==== osd_op_reply(92 notify.4 [create] v30'138 uv138 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f32dc050040 con 0x560a572b97f0 2023-10-05T19:28:35.032 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.026+0000 7f32eba3ce40 1 -- 172.21.15.167:0/2262883315 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:93 7.d 7:b4812045:::notify.4:head [watch watch cookie 94602412392160] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x560a572f9c80 con 0x560a572b97f0 2023-10-05T19:28:35.032 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.026+0000 7f32eba3ce40 1 -- 172.21.15.167:0/2262883315 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:94 7.3 7:c609908c:::notify.5:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x560a572fa570 con 0x7f32e0007030 2023-10-05T19:28:35.034 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.030+0000 7f32e9d72700 1 -- 172.21.15.167:0/2262883315 <== osd.1 v2:172.21.15.167:6800/30035 32 ==== osd_op_reply(93 notify.4 [watch watch cookie 94602412392160] v30'139 uv138 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f32dc050040 con 0x560a572b97f0 2023-10-05T19:28:35.034 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.030+0000 7f32e9d72700 1 -- 172.21.15.167:0/2262883315 <== osd.0 v2:172.21.15.118:6800/27690 23 ==== osd_op_reply(94 notify.5 [create] v30'138 uv138 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f32dc050040 con 0x7f32e0007030 2023-10-05T19:28:35.035 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.030+0000 7f32eba3ce40 1 -- 172.21.15.167:0/2262883315 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:95 7.3 7:c609908c:::notify.5:head [watch watch cookie 94602412399040] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x560a572fa300 con 0x7f32e0007030 2023-10-05T19:28:35.035 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.030+0000 7f32eba3ce40 1 -- 172.21.15.167:0/2262883315 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:96 7.14 7:2b04a3e9:::notify.6:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x560a572faec0 con 0x7f32e0007030 2023-10-05T19:28:35.037 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.030+0000 7f32e9d72700 1 -- 172.21.15.167:0/2262883315 <== osd.0 v2:172.21.15.118:6800/27690 24 ==== osd_op_reply(95 notify.5 [watch watch cookie 94602412399040] v30'139 uv138 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f32dc050040 con 0x7f32e0007030 2023-10-05T19:28:35.037 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.030+0000 7f32e9d72700 1 -- 172.21.15.167:0/2262883315 <== osd.0 v2:172.21.15.118:6800/27690 25 ==== osd_op_reply(96 notify.6 [create] v30'138 uv138 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f32dc050040 con 0x7f32e0007030 2023-10-05T19:28:35.037 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.030+0000 7f32eba3ce40 1 -- 172.21.15.167:0/2262883315 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:97 7.14 7:2b04a3e9:::notify.6:head [watch watch cookie 94602412403088] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x560a572fa300 con 0x7f32e0007030 2023-10-05T19:28:35.037 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.030+0000 7f32eba3ce40 1 -- 172.21.15.167:0/2262883315 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:98 7.9 7:93e5b521:::notify.7:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x560a572fb8a0 con 0x7f32e0007030 2023-10-05T19:28:35.040 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.034+0000 7f32e9d72700 1 -- 172.21.15.167:0/2262883315 <== osd.0 v2:172.21.15.118:6800/27690 26 ==== osd_op_reply(98 notify.7 [create] v30'138 uv138 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f32dc050040 con 0x7f32e0007030 2023-10-05T19:28:35.040 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.034+0000 7f32eba3ce40 1 -- 172.21.15.167:0/2262883315 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:99 7.9 7:93e5b521:::notify.7:head [watch watch cookie 94602412405616] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x560a572fbcb0 con 0x7f32e0007030 2023-10-05T19:28:35.040 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.034+0000 7f32eba3ce40 20 add_watcher() i=0 2023-10-05T19:28:35.040 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.034+0000 7f32eba3ce40 20 add_watcher() i=1 2023-10-05T19:28:35.040 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.034+0000 7f32eba3ce40 20 add_watcher() i=2 2023-10-05T19:28:35.040 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.034+0000 7f32eba3ce40 20 add_watcher() i=3 2023-10-05T19:28:35.040 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.034+0000 7f32eba3ce40 20 add_watcher() i=4 2023-10-05T19:28:35.040 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.034+0000 7f32eba3ce40 20 add_watcher() i=5 2023-10-05T19:28:35.041 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.034+0000 7f32e9d72700 1 -- 172.21.15.167:0/2262883315 <== osd.0 v2:172.21.15.118:6800/27690 27 ==== osd_op_reply(97 notify.6 [watch watch cookie 94602412403088] v30'139 uv138 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f32dc050040 con 0x7f32e0007030 2023-10-05T19:28:35.041 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.034+0000 7f32eba3ce40 20 add_watcher() i=6 2023-10-05T19:28:35.042 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.034+0000 7f32e9d72700 1 -- 172.21.15.167:0/2262883315 <== osd.0 v2:172.21.15.118:6800/27690 28 ==== osd_op_reply(99 notify.7 [watch watch cookie 94602412405616] v30'139 uv138 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f32dc050040 con 0x7f32e0007030 2023-10-05T19:28:35.042 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.034+0000 7f32eba3ce40 20 add_watcher() i=7 2023-10-05T19:28:35.042 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.034+0000 7f32eba3ce40 2 all 8 watchers are set, enabling cache 2023-10-05T19:28:35.042 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.034+0000 7f32eba3ce40 1 -- 172.21.15.167:0/2262883315 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:100 6.4 6:22d26bf9:::data_loggenerations_metadata:head [call version.check_conds in=74b,call version.read in=11b,read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x560a572f9e10 con 0x560a572b97f0 2023-10-05T19:28:35.042 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.038+0000 7f32e9d72700 1 -- 172.21.15.167:0/2262883315 <== osd.1 v2:172.21.15.167:6800/30035 33 ==== osd_op_reply(100 data_loggenerations_metadata [call,call out=48b,read 0~28 out=28b] v0'0 uv1 ondisk = 0) v8 ==== 256+0+76 (crc 0 0 0) 0x7f32dc050040 con 0x560a572b97f0 2023-10-05T19:28:35.042 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.038+0000 7f32eba3ce40 1 -- 172.21.15.167:0/2262883315 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:101 6.4 6:22d26bf9:::data_loggenerations_metadata:head [watch watch cookie 94602412406720] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x560a572fc1a0 con 0x560a572b97f0 2023-10-05T19:28:35.045 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.038+0000 7f32e9d72700 1 -- 172.21.15.167:0/2262883315 <== osd.1 v2:172.21.15.167:6800/30035 34 ==== osd_op_reply(101 data_loggenerations_metadata [watch watch cookie 94602412406720] v30'133 uv1 ondisk = 0) v8 ==== 172+0+0 (crc 0 0 0) 0x7f32dc050040 con 0x560a572b97f0 2023-10-05T19:28:35.045 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.038+0000 7f32ba7e4700 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2023-10-05T19:28:35.045 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.038+0000 7f32eba3ce40 20 check_secure_mon_conn(): auth registy supported: methods=[2] modes=[2,1] 2023-10-05T19:28:35.045 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.038+0000 7f32eba3ce40 20 check_secure_mon_conn(): mode 1 is insecure 2023-10-05T19:28:35.046 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.038+0000 7f32eba3ce40 1 Processor -- start 2023-10-05T19:28:35.046 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.038+0000 7f32eba3ce40 1 -- start start 2023-10-05T19:28:35.046 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.042+0000 7f32eba3ce40 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x560a57309310 0x560a573096e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-05T19:28:35.046 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.042+0000 7f32eba3ce40 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x560a57309c20 con 0x560a57309310 2023-10-05T19:28:35.047 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.042+0000 7f32e8d70700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x560a57309310 0x560a573096e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-05T19:28:35.047 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.042+0000 7f32e8d70700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x560a57309310 0x560a573096e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.118:3300/0 says I am v2:172.21.15.167:41584/0 (socket says 172.21.15.167:41584) 2023-10-05T19:28:35.047 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.042+0000 7f32e8d70700 1 -- 172.21.15.167:0/2787258923 learned_addr learned my addr 172.21.15.167:0/2787258923 (peer_addr_for_me v2:172.21.15.167:0/0) 2023-10-05T19:28:35.047 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.042+0000 7f32e8d70700 1 -- 172.21.15.167:0/2787258923 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x560a57309d60 con 0x560a57309310 2023-10-05T19:28:35.048 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.042+0000 7f32e8d70700 1 --2- 172.21.15.167:0/2787258923 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x560a57309310 0x560a573096e0 secure :-1 s=READY pgs=414 cs=0 l=1 rev1=1 crypto rx=0x7f32e0002000 tx=0x7f32e0012000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5cc05b3c3e4bbaba server_cookie=0 in_seq=0 out_seq=0 2023-10-05T19:28:35.048 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.042+0000 7f32b9fe3700 1 -- 172.21.15.167:0/2787258923 <== mon.0 v2:172.21.15.118:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f32e0012c90 con 0x560a57309310 2023-10-05T19:28:35.048 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.042+0000 7f32b9fe3700 1 -- 172.21.15.167:0/2787258923 <== mon.0 v2:172.21.15.118:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f32e0012df0 con 0x560a57309310 2023-10-05T19:28:35.048 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.042+0000 7f32b9fe3700 1 -- 172.21.15.167:0/2787258923 <== mon.0 v2:172.21.15.118:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f32e002a8a0 con 0x560a57309310 2023-10-05T19:28:35.049 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.042+0000 7f32eba3ce40 1 -- 172.21.15.167:0/2787258923 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x560a57309310 msgr2=0x560a573096e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-05T19:28:35.049 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.042+0000 7f32eba3ce40 1 --2- 172.21.15.167:0/2787258923 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x560a57309310 0x560a573096e0 secure :-1 s=READY pgs=414 cs=0 l=1 rev1=1 crypto rx=0x7f32e0002000 tx=0x7f32e0012000 comp rx=0 tx=0).stop 2023-10-05T19:28:35.049 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.042+0000 7f32eba3ce40 1 -- 172.21.15.167:0/2787258923 shutdown_connections 2023-10-05T19:28:35.049 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.042+0000 7f32eba3ce40 1 --2- 172.21.15.167:0/2787258923 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x560a57309310 0x560a573096e0 unknown :-1 s=CLOSED pgs=414 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-05T19:28:35.050 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.042+0000 7f32eba3ce40 1 -- 172.21.15.167:0/2787258923 >> 172.21.15.167:0/2787258923 conn(0x560a573046d0 msgr2=0x560a57302820 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-05T19:28:35.050 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.042+0000 7f32eba3ce40 1 -- 172.21.15.167:0/2787258923 shutdown_connections 2023-10-05T19:28:35.050 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.042+0000 7f32eba3ce40 1 -- 172.21.15.167:0/2787258923 wait complete. 2023-10-05T19:28:35.051 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.046+0000 7f32eba3ce40 1 Processor -- start 2023-10-05T19:28:35.051 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.046+0000 7f32eba3ce40 1 -- start start 2023-10-05T19:28:35.051 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.046+0000 7f32eba3ce40 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x560a57309310 0x560a5730ee40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-05T19:28:35.051 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.046+0000 7f32eba3ce40 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x560a5730fe40 con 0x560a57309310 2023-10-05T19:28:35.051 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.046+0000 7f32e8d70700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x560a57309310 0x560a5730ee40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-05T19:28:35.051 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.046+0000 7f32e8d70700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x560a57309310 0x560a5730ee40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.118:3300/0 says I am v2:172.21.15.167:41588/0 (socket says 172.21.15.167:41588) 2023-10-05T19:28:35.051 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.046+0000 7f32e8d70700 1 -- 172.21.15.167:0/3289891311 learned_addr learned my addr 172.21.15.167:0/3289891311 (peer_addr_for_me v2:172.21.15.167:0/0) 2023-10-05T19:28:35.052 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.046+0000 7f32e8d70700 1 -- 172.21.15.167:0/3289891311 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x560a572b8ce0 con 0x560a57309310 2023-10-05T19:28:35.052 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.046+0000 7f32e8d70700 1 --2- 172.21.15.167:0/3289891311 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x560a57309310 0x560a5730ee40 secure :-1 s=READY pgs=415 cs=0 l=1 rev1=1 crypto rx=0x560a57303390 tx=0x7f32e002f000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-05T19:28:35.052 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.046+0000 7f322bfff700 1 -- 172.21.15.167:0/3289891311 <== mon.0 v2:172.21.15.118:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f32e003e950 con 0x560a57309310 2023-10-05T19:28:35.052 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.046+0000 7f322bfff700 1 -- 172.21.15.167:0/3289891311 <== mon.0 v2:172.21.15.118:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f32e003ead0 con 0x560a57309310 2023-10-05T19:28:35.053 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.046+0000 7f32eba3ce40 1 -- 172.21.15.167:0/3289891311 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x560a5730f440 con 0x560a57309310 2023-10-05T19:28:35.053 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.046+0000 7f322bfff700 1 -- 172.21.15.167:0/3289891311 <== mon.0 v2:172.21.15.118:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f32e003ec80 con 0x560a57309310 2023-10-05T19:28:35.053 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.046+0000 7f32eba3ce40 1 -- 172.21.15.167:0/3289891311 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x560a5730f8e0 con 0x560a57309310 2023-10-05T19:28:35.054 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.050+0000 7f322bfff700 1 -- 172.21.15.167:0/3289891311 <== mon.0 v2:172.21.15.118:3300/0 4 ==== mgrmap(e 5) v1 ==== 40329+0+0 (secure 0 0 0) 0x7f32e0046070 con 0x560a57309310 2023-10-05T19:28:35.055 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.050+0000 7f322bfff700 1 --2- 172.21.15.167:0/3289891311 >> [v2:172.21.15.167:6824/29889,v1:172.21.15.167:6825/29889] conn(0x7f3230030c50 0x7f3230033100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-05T19:28:35.055 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.050+0000 7f322bfff700 1 -- 172.21.15.167:0/3289891311 <== mon.0 v2:172.21.15.118:3300/0 5 ==== osd_map(30..30 src has 1..30) v4 ==== 6057+0+0 (secure 0 0 0) 0x7f32e002f070 con 0x560a57309310 2023-10-05T19:28:35.055 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.050+0000 7f32e9d72700 1 --2- 172.21.15.167:0/3289891311 >> [v2:172.21.15.167:6824/29889,v1:172.21.15.167:6825/29889] conn(0x7f3230030c50 0x7f3230033100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-05T19:28:35.055 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.050+0000 7f32eba3ce40 5 note: GC not initialized 2023-10-05T19:28:35.055 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.050+0000 7f32e9d72700 1 --2- 172.21.15.167:0/3289891311 >> [v2:172.21.15.167:6824/29889,v1:172.21.15.167:6825/29889] conn(0x7f3230030c50 0x7f3230033100 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x560a572fae90 tx=0x7f32dc000f90 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-05T19:28:35.057 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.050+0000 7f3222ffd700 20 reqs_thread_entry: start 2023-10-05T19:28:35.058 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.050+0000 7f32eba3ce40 20 init_complete bucket index max shards: 11 2023-10-05T19:28:35.058 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.050+0000 7f32297fa700 20 reqs_thread_entry: start 2023-10-05T19:28:35.058 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.054+0000 7f32eba3ce40 10 rgw notify: Started notification manager with: 1 workers 2023-10-05T19:28:35.059 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.054+0000 7f32217fa700 20 rgw notify: INFO: next queues processing will happen at: Thu Oct 5 19:29:05 2023 2023-10-05T19:28:35.059 INFO:teuthology.orchestra.run.smithi167.stderr: 2023-10-05T19:28:35.059 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.054+0000 7f3220ff9700 20 reqs_thread_entry: start 2023-10-05T19:28:35.059 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.054+0000 7f32eba3ce40 20 get_system_obj_state: rctx=0x7ffd2fd6a5d8 obj=default.rgw.meta:users.uid:foo state=0x560a571920b0 s->prefetch_data=0 2023-10-05T19:28:35.059 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.054+0000 7f32eba3ce40 10 cache get: name=default.rgw.meta+users.uid+foo : miss 2023-10-05T19:28:35.059 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.054+0000 7f32eba3ce40 1 -- 172.21.15.167:0/2262883315 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:102 8.18 8:1e6ca5ed:users.uid::foo:head [call version.read in=11b,getxattrs,stat] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x560a57377670 con 0x560a572b97f0 2023-10-05T19:28:35.061 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.054+0000 7f32e9d72700 1 -- 172.21.15.167:0/2262883315 <== osd.1 v2:172.21.15.167:6800/30035 35 ==== osd_op_reply(102 foo [call out=48b,getxattrs out=75b,stat out=16b] v0'0 uv11 ondisk = 0) v8 ==== 231+0+139 (crc 0 0 0) 0x7f32dc050040 con 0x560a572b97f0 2023-10-05T19:28:35.061 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.054+0000 7f32eba3ce40 10 cache put: name=default.rgw.meta+users.uid+foo info.flags=0x16 2023-10-05T19:28:35.061 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.054+0000 7f32eba3ce40 10 adding default.rgw.meta+users.uid+foo to cache LRU end 2023-10-05T19:28:35.061 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.054+0000 7f32eba3ce40 10 updating xattr: name=ceph.objclass.version bl.length()=42 2023-10-05T19:28:35.061 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.054+0000 7f32eba3ce40 20 get_system_obj_state: s->obj_tag was set empty 2023-10-05T19:28:35.061 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.054+0000 7f32eba3ce40 20 Read xattr: user.rgw.idtag 2023-10-05T19:28:35.061 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.054+0000 7f32eba3ce40 10 cache get: name=default.rgw.meta+users.uid+foo : type miss (requested=0x13, cached=0x16) 2023-10-05T19:28:35.061 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.054+0000 7f32eba3ce40 20 rados->read ofs=0 len=0 2023-10-05T19:28:35.061 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.054+0000 7f32eba3ce40 1 -- 172.21.15.167:0/2262883315 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:103 8.18 8:1e6ca5ed:users.uid::foo:head [call version.check_conds in=122b,call version.read in=11b,read 0~0,getxattrs] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x560a5737a100 con 0x560a572b97f0 2023-10-05T19:28:35.061 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.054+0000 7f32e9d72700 1 -- 172.21.15.167:0/2262883315 <== osd.1 v2:172.21.15.167:6800/30035 36 ==== osd_op_reply(103 foo [call,call out=48b,read 0~357 out=357b,getxattrs out=75b] v0'0 uv11 ondisk = 0) v8 ==== 273+0+480 (crc 0 0 0) 0x7f32dc050040 con 0x560a572b97f0 2023-10-05T19:28:35.062 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.054+0000 7f32eba3ce40 20 rados_obj.operate() r=0 bl.length=357 2023-10-05T19:28:35.062 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.054+0000 7f32eba3ce40 10 cache put: name=default.rgw.meta+users.uid+foo info.flags=0x13 2023-10-05T19:28:35.062 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.054+0000 7f32eba3ce40 10 moving default.rgw.meta+users.uid+foo to cache LRU end 2023-10-05T19:28:35.062 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.054+0000 7f32eba3ce40 10 updating xattr: name=ceph.objclass.version bl.length()=42 2023-10-05T19:28:35.062 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.054+0000 7f32eba3ce40 20 get_system_obj_state: rctx=0x7ffd2fd69980 obj=default.rgw.meta:root:testx/myfoo state=0x560a571920b0 s->prefetch_data=0 2023-10-05T19:28:35.062 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.054+0000 7f32eba3ce40 10 cache get: name=default.rgw.meta+root+testx/myfoo : miss 2023-10-05T19:28:35.062 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.054+0000 7f32eba3ce40 1 -- 172.21.15.167:0/2262883315 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:104 8.5 8:a7a6c9d8:root::testx%2fmyfoo:head [call version.read in=11b,getxattrs,stat] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x560a57379910 con 0x560a572b97f0 2023-10-05T19:28:35.064 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.054+0000 7f32e9d72700 1 -- 172.21.15.167:0/2262883315 <== osd.1 v2:172.21.15.167:6800/30035 37 ==== osd_op_reply(104 testx/myfoo [call out=48b,getxattrs out=75b,stat out=16b] v0'0 uv4 ondisk = 0) v8 ==== 239+0+139 (crc 0 0 0) 0x7f32dc050040 con 0x560a572b97f0 2023-10-05T19:28:35.064 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.054+0000 7f32eba3ce40 10 cache put: name=default.rgw.meta+root+testx/myfoo info.flags=0x16 2023-10-05T19:28:35.064 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.054+0000 7f32eba3ce40 10 adding default.rgw.meta+root+testx/myfoo to cache LRU end 2023-10-05T19:28:35.064 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.054+0000 7f32eba3ce40 10 updating xattr: name=ceph.objclass.version bl.length()=42 2023-10-05T19:28:35.064 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.054+0000 7f32eba3ce40 20 get_system_obj_state: s->obj_tag was set empty 2023-10-05T19:28:35.064 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.054+0000 7f32eba3ce40 10 cache get: name=default.rgw.meta+root+testx/myfoo : type miss (requested=0x11, cached=0x16) 2023-10-05T19:28:35.065 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.054+0000 7f32eba3ce40 20 rados->read ofs=0 len=0 2023-10-05T19:28:35.065 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.054+0000 7f32eba3ce40 1 -- 172.21.15.167:0/2262883315 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:105 8.5 8:a7a6c9d8:root::testx%2fmyfoo:head [call version.check_conds in=122b,call version.read in=11b,read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x560a5737beb0 con 0x560a572b97f0 2023-10-05T19:28:35.065 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.058+0000 7f32e9d72700 1 -- 172.21.15.167:0/2262883315 <== osd.1 v2:172.21.15.167:6800/30035 38 ==== osd_op_reply(105 testx/myfoo [call,call out=48b,read 0~193 out=193b] v0'0 uv4 ondisk = 0) v8 ==== 239+0+241 (crc 0 0 0) 0x7f32dc050040 con 0x560a572b97f0 2023-10-05T19:28:35.065 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.058+0000 7f32eba3ce40 20 rados_obj.operate() r=0 bl.length=193 2023-10-05T19:28:35.065 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.058+0000 7f32eba3ce40 10 cache put: name=default.rgw.meta+root+testx/myfoo info.flags=0x11 2023-10-05T19:28:35.065 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.058+0000 7f32eba3ce40 10 moving default.rgw.meta+root+testx/myfoo to cache LRU end 2023-10-05T19:28:35.065 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.058+0000 7f32eba3ce40 20 get_system_obj_state: rctx=0x7ffd2fd69980 obj=default.rgw.meta:root:.bucket.meta.testx:myfoo:9a9df965-6612-45d4-a45f-6e3c889bacc2.4223.1 state=0x560a571682b0 s->prefetch_data=0 2023-10-05T19:28:35.065 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.058+0000 7f32eba3ce40 10 cache get: name=default.rgw.meta+root+.bucket.meta.testx:myfoo:9a9df965-6612-45d4-a45f-6e3c889bacc2.4223.1 : miss 2023-10-05T19:28:35.065 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.058+0000 7f32eba3ce40 1 -- 172.21.15.167:0/2262883315 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:106 8.6 8:6736df0c:root::.bucket.meta.testx%3amyfoo%3a9a9df965-6612-45d4-a45f-6e3c889bacc2.4223.1:head [call version.read in=11b,getxattrs,stat] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x560a5737b1e0 con 0x7f32e0007030 2023-10-05T19:28:35.065 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.058+0000 7f32e9d72700 1 -- 172.21.15.167:0/2262883315 <== osd.0 v2:172.21.15.118:6800/27690 29 ==== osd_op_reply(106 .bucket.meta.testx:myfoo:9a9df965-6612-45d4-a45f-6e3c889bacc2.4223.1 [call out=48b,getxattrs out=298b,stat out=16b] v0'0 uv1 ondisk = 0) v8 ==== 296+0+362 (crc 0 0 0) 0x7f32dc050040 con 0x7f32e0007030 2023-10-05T19:28:35.066 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.058+0000 7f32eba3ce40 10 cache put: name=default.rgw.meta+root+.bucket.meta.testx:myfoo:9a9df965-6612-45d4-a45f-6e3c889bacc2.4223.1 info.flags=0x16 2023-10-05T19:28:35.066 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.058+0000 7f32eba3ce40 10 adding default.rgw.meta+root+.bucket.meta.testx:myfoo:9a9df965-6612-45d4-a45f-6e3c889bacc2.4223.1 to cache LRU end 2023-10-05T19:28:35.066 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.058+0000 7f32eba3ce40 10 updating xattr: name=ceph.objclass.version bl.length()=42 2023-10-05T19:28:35.066 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.058+0000 7f32eba3ce40 10 updating xattr: name=user.rgw.acl bl.length()=203 2023-10-05T19:28:35.066 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.058+0000 7f32eba3ce40 20 get_system_obj_state: s->obj_tag was set empty 2023-10-05T19:28:35.066 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.058+0000 7f32eba3ce40 20 Read xattr: user.rgw.acl 2023-10-05T19:28:35.066 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.058+0000 7f32eba3ce40 20 Read xattr: user.rgw.idtag 2023-10-05T19:28:35.066 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.058+0000 7f32eba3ce40 10 cache get: name=default.rgw.meta+root+.bucket.meta.testx:myfoo:9a9df965-6612-45d4-a45f-6e3c889bacc2.4223.1 : type miss (requested=0x13, cached=0x16) 2023-10-05T19:28:35.066 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.058+0000 7f32eba3ce40 20 rados->read ofs=0 len=0 2023-10-05T19:28:35.066 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.058+0000 7f32eba3ce40 1 -- 172.21.15.167:0/2262883315 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:107 8.6 8:6736df0c:root::.bucket.meta.testx%3amyfoo%3a9a9df965-6612-45d4-a45f-6e3c889bacc2.4223.1:head [call version.check_conds in=122b,call version.read in=11b,read 0~0,getxattrs] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x560a5737dd30 con 0x7f32e0007030 2023-10-05T19:28:35.068 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.058+0000 7f32e9d72700 1 -- 172.21.15.167:0/2262883315 <== osd.0 v2:172.21.15.118:6800/27690 30 ==== osd_op_reply(107 .bucket.meta.testx:myfoo:9a9df965-6612-45d4-a45f-6e3c889bacc2.4223.1 [call,call out=48b,read 0~371 out=371b,getxattrs out=298b] v0'0 uv1 ondisk = 0) v8 ==== 338+0+717 (crc 0 0 0) 0x7f32dc050040 con 0x7f32e0007030 2023-10-05T19:28:35.068 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.058+0000 7f32eba3ce40 20 rados_obj.operate() r=0 bl.length=371 2023-10-05T19:28:35.068 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.058+0000 7f32eba3ce40 10 cache put: name=default.rgw.meta+root+.bucket.meta.testx:myfoo:9a9df965-6612-45d4-a45f-6e3c889bacc2.4223.1 info.flags=0x13 2023-10-05T19:28:35.068 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.058+0000 7f32eba3ce40 10 moving default.rgw.meta+root+.bucket.meta.testx:myfoo:9a9df965-6612-45d4-a45f-6e3c889bacc2.4223.1 to cache LRU end 2023-10-05T19:28:35.068 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.058+0000 7f32eba3ce40 10 updating xattr: name=ceph.objclass.version bl.length()=42 2023-10-05T19:28:35.068 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.058+0000 7f32eba3ce40 10 updating xattr: name=user.rgw.acl bl.length()=203 2023-10-05T19:28:35.069 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.058+0000 7f32eba3ce40 10 chain_cache_entry: cache_locator=default.rgw.meta+root+.bucket.meta.testx:myfoo:9a9df965-6612-45d4-a45f-6e3c889bacc2.4223.1 2023-10-05T19:28:35.069 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.058+0000 7f32eba3ce40 20 get_system_obj_state: rctx=0x7ffd2fd69918 obj=default.rgw.meta:users.uid:foo state=0x560a571682b0 s->prefetch_data=0 2023-10-05T19:28:35.069 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.058+0000 7f32eba3ce40 10 cache get: name=default.rgw.meta+users.uid+foo : type miss (requested=0x16, cached=0x13) 2023-10-05T19:28:35.069 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.058+0000 7f32eba3ce40 1 -- 172.21.15.167:0/2262883315 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:108 8.18 8:1e6ca5ed:users.uid::foo:head [call version.read in=11b,getxattrs,stat] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x560a5737daa0 con 0x560a572b97f0 2023-10-05T19:28:35.069 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.058+0000 7f32e9d72700 1 -- 172.21.15.167:0/2262883315 <== osd.1 v2:172.21.15.167:6800/30035 39 ==== osd_op_reply(108 foo [call out=48b,getxattrs out=75b,stat out=16b] v0'0 uv11 ondisk = 0) v8 ==== 231+0+139 (crc 0 0 0) 0x7f32dc050040 con 0x560a572b97f0 2023-10-05T19:28:35.069 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.058+0000 7f32eba3ce40 10 cache put: name=default.rgw.meta+users.uid+foo info.flags=0x16 2023-10-05T19:28:35.069 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.058+0000 7f32eba3ce40 10 moving default.rgw.meta+users.uid+foo to cache LRU end 2023-10-05T19:28:35.069 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.058+0000 7f32eba3ce40 10 updating xattr: name=ceph.objclass.version bl.length()=42 2023-10-05T19:28:35.069 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.058+0000 7f32eba3ce40 20 get_system_obj_state: s->obj_tag was set empty 2023-10-05T19:28:35.069 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.058+0000 7f32eba3ce40 20 Read xattr: user.rgw.idtag 2023-10-05T19:28:35.070 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.058+0000 7f32eba3ce40 10 cache get: name=default.rgw.meta+users.uid+foo : hit (requested=0x13, cached=0x17) 2023-10-05T19:28:35.070 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.058+0000 7f32eba3ce40 1 -- 172.21.15.167:0/2262883315 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:109 8.c 8:37c7a57e:users.uid::testx$tenanteduser.buckets:head [call user.remove_bucket in=58b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x560a57380370 con 0x560a572b97f0 2023-10-05T19:28:35.070 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.062+0000 7f32e9d72700 1 -- 172.21.15.167:0/2262883315 <== osd.1 v2:172.21.15.167:6800/30035 40 ==== osd_op_reply(109 testx$tenanteduser.buckets [call] v30'2 uv2 ondisk = 0) v8 ==== 170+0+0 (crc 0 0 0) 0x7f32dc050040 con 0x560a572b97f0 2023-10-05T19:28:35.070 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.062+0000 7f32eba3ce40 1 -- 172.21.15.167:0/2262883315 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:110 8.11 8:8bb4ed75:root::.bucket.meta.myfoo%3a9a9df965-6612-45d4-a45f-6e3c889bacc2.4223.1:head [create,call version.inc in=38b,writefull 0~352 in=352b,setxattr user.rgw.acl (123) in=135b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x560a57383b30 con 0x560a572f0860 2023-10-05T19:28:35.071 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.066+0000 7f32e9571700 1 -- 172.21.15.167:0/2262883315 <== osd.3 v2:172.21.15.167:6808/30036 24 ==== osd_op_reply(110 .bucket.meta.myfoo:9a9df965-6612-45d4-a45f-6e3c889bacc2.4223.1 [create,call,writefull 0~352,setxattr (123)] v30'5 uv5 ondisk = 0) v8 ==== 332+0+0 (crc 0 0 0) 0x7f32e4024020 con 0x560a572f0860 2023-10-05T19:28:35.071 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.066+0000 7f32eba3ce40 10 cache put: name=default.rgw.meta+root+.bucket.meta.myfoo:9a9df965-6612-45d4-a45f-6e3c889bacc2.4223.1 info.flags=0x7 2023-10-05T19:28:35.072 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.066+0000 7f32eba3ce40 10 adding default.rgw.meta+root+.bucket.meta.myfoo:9a9df965-6612-45d4-a45f-6e3c889bacc2.4223.1 to cache LRU end 2023-10-05T19:28:35.072 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.066+0000 7f32eba3ce40 10 updating xattr: name=user.rgw.acl bl.length()=123 2023-10-05T19:28:35.072 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.066+0000 7f32eba3ce40 10 distributing notification oid=default.rgw.control:notify.7 cni=[op: 0, obj: default.rgw.meta:root:.bucket.meta.myfoo:9a9df965-6612-45d4-a45f-6e3c889bacc2.4223.1, ofs0, ns] 2023-10-05T19:28:35.072 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.066+0000 7f32eba3ce40 1 -- 172.21.15.167:0/2262883315 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:111 7.9 7:93e5b521:::notify.7:head [notify cookie 94602412962800 in=713b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x560a57382b60 con 0x7f32e0007030 2023-10-05T19:28:35.073 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.066+0000 7f32e9d72700 1 -- 172.21.15.167:0/2262883315 <== osd.0 v2:172.21.15.118:6800/27690 31 ==== watch-notify(notify (1) cookie 94602412405616 notify 128849018894 ret 0) v3 ==== 743+0+0 (crc 0 0 0) 0x7f32dc03d040 con 0x7f32e0007030 2023-10-05T19:28:35.073 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.066+0000 7f32daffd700 10 rgw watcher librados: RGWWatcher::handle_notify() notify_id 128849018894 cookie 94602412405616 notifier 4503 bl.length()=701 2023-10-05T19:28:35.073 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.066+0000 7f32daffd700 10 rgw watcher librados: cache put: name=default.rgw.meta+root+.bucket.meta.myfoo:9a9df965-6612-45d4-a45f-6e3c889bacc2.4223.1 info.flags=0x7 2023-10-05T19:28:35.073 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.066+0000 7f32e9d72700 1 -- 172.21.15.167:0/2262883315 <== osd.0 v2:172.21.15.118:6800/27690 32 ==== osd_op_reply(111 notify.7 [notify cookie 94602412962800 out=8b] v0'0 uv138 ondisk = 0) v8 ==== 152+0+8 (crc 0 0 0) 0x7f32dc050040 con 0x7f32e0007030 2023-10-05T19:28:35.073 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.066+0000 7f32daffd700 10 rgw watcher librados: moving default.rgw.meta+root+.bucket.meta.myfoo:9a9df965-6612-45d4-a45f-6e3c889bacc2.4223.1 to cache LRU end 2023-10-05T19:28:35.073 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.066+0000 7f32daffd700 10 rgw watcher librados: updating xattr: name=user.rgw.acl bl.length()=123 2023-10-05T19:28:35.073 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.066+0000 7f32daffd700 1 -- 172.21.15.167:0/2262883315 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:112 7.9 7:93e5b521:::notify.7:head [notify-ack in=20b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x7f32c8003d90 con 0x7f32e0007030 2023-10-05T19:28:35.074 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.066+0000 7f32e9d72700 1 -- 172.21.15.167:0/2262883315 <== osd.0 v2:172.21.15.118:6800/27690 33 ==== watch-notify(notify_complete (2) cookie 94602412962800 notify 128849018894 ret 0) v3 ==== 42+0+48 (crc 0 0 0) 0x7f32dc002c40 con 0x7f32e0007030 2023-10-05T19:28:35.074 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.066+0000 7f32e9d72700 1 -- 172.21.15.167:0/2262883315 <== osd.0 v2:172.21.15.118:6800/27690 34 ==== osd_op_reply(112 notify.7 [notify-ack] v0'0 uv138 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f32dc050040 con 0x7f32e0007030 2023-10-05T19:28:35.074 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.066+0000 7f32eba3ce40 20 handle_bi_update(): bucket=:myfoo[9a9df965-6612-45d4-a45f-6e3c889bacc2.4223.1]): orig_sources= new_sources= 2023-10-05T19:28:35.074 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.066+0000 7f32eba3ce40 20 handle_bi_update(): bucket=:myfoo[9a9df965-6612-45d4-a45f-6e3c889bacc2.4223.1]): potential sources added=[] removed=[] 2023-10-05T19:28:35.074 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.066+0000 7f32eba3ce40 20 handle_bi_update(): bucket=:myfoo[9a9df965-6612-45d4-a45f-6e3c889bacc2.4223.1]): orig_dests= new_dests= 2023-10-05T19:28:35.075 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.066+0000 7f32eba3ce40 20 handle_bi_update(): bucket=:myfoo[9a9df965-6612-45d4-a45f-6e3c889bacc2.4223.1]): potential dests added=[] removed=[] 2023-10-05T19:28:35.075 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.066+0000 7f32eba3ce40 1 -- 172.21.15.167:0/2262883315 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:113 8.9 8:96d98480:users.uid::foo.buckets:head [call user.set_buckets_info in=207b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x560a573833f0 con 0x7f32e4027be0 2023-10-05T19:28:35.077 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.070+0000 7f32e8d70700 1 -- 172.21.15.167:0/2262883315 <== osd.2 v2:172.21.15.167:6816/30037 17 ==== osd_op_reply(113 foo.buckets [call] v30'26 uv26 ondisk = 0) v8 ==== 155+0+0 (crc 0 0 0) 0x7f32e0032040 con 0x7f32e4027be0 2023-10-05T19:28:35.077 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.070+0000 7f32eba3ce40 1 -- 172.21.15.167:0/2262883315 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:114 8.9 8:91b71841:root::myfoo:head [delete,create,call version.inc in=38b,writefull 0~165 in=165b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x560a57382a40 con 0x7f32e4027be0 2023-10-05T19:28:35.080 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.074+0000 7f32e8d70700 1 -- 172.21.15.167:0/2262883315 <== osd.2 v2:172.21.15.167:6816/30037 18 ==== osd_op_reply(114 myfoo [delete,create,call,writefull 0~165] v30'27 uv27 ondisk = 0) v8 ==== 275+0+0 (crc 0 0 0) 0x7f32e0032040 con 0x7f32e4027be0 2023-10-05T19:28:35.080 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.074+0000 7f32eba3ce40 10 cache put: name=default.rgw.meta+root+myfoo info.flags=0x7 2023-10-05T19:28:35.080 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.074+0000 7f32eba3ce40 10 adding default.rgw.meta+root+myfoo to cache LRU end 2023-10-05T19:28:35.080 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.074+0000 7f32eba3ce40 10 distributing notification oid=default.rgw.control:notify.6 cni=[op: 0, obj: default.rgw.meta:root:myfoo, ofs0, ns] 2023-10-05T19:28:35.080 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.074+0000 7f32eba3ce40 1 -- 172.21.15.167:0/2262883315 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:115 7.14 7:2b04a3e9:::notify.6:head [notify cookie 94602412959056 in=326b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x560a57384590 con 0x7f32e0007030 2023-10-05T19:28:35.081 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.074+0000 7f32e9d72700 1 -- 172.21.15.167:0/2262883315 <== osd.0 v2:172.21.15.118:6800/27690 35 ==== watch-notify(notify (1) cookie 94602412403088 notify 128849018895 ret 0) v3 ==== 356+0+0 (crc 0 0 0) 0x7f32dc002c40 con 0x7f32e0007030 2023-10-05T19:28:35.081 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.074+0000 7f32daffd700 10 rgw watcher librados: RGWWatcher::handle_notify() notify_id 128849018895 cookie 94602412403088 notifier 4503 bl.length()=314 2023-10-05T19:28:35.081 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.074+0000 7f32e9d72700 1 -- 172.21.15.167:0/2262883315 <== osd.0 v2:172.21.15.118:6800/27690 36 ==== osd_op_reply(115 notify.6 [notify cookie 94602412959056 out=8b] v0'0 uv138 ondisk = 0) v8 ==== 152+0+8 (crc 0 0 0) 0x7f32dc050040 con 0x7f32e0007030 2023-10-05T19:28:35.081 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.074+0000 7f32daffd700 10 rgw watcher librados: cache put: name=default.rgw.meta+root+myfoo info.flags=0x7 2023-10-05T19:28:35.082 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.074+0000 7f32daffd700 10 rgw watcher librados: moving default.rgw.meta+root+myfoo to cache LRU end 2023-10-05T19:28:35.082 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.074+0000 7f32daffd700 1 -- 172.21.15.167:0/2262883315 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:116 7.14 7:2b04a3e9:::notify.6:head [notify-ack in=20b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x7f32c8003d90 con 0x7f32e0007030 2023-10-05T19:28:35.083 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.074+0000 7f32e9d72700 1 -- 172.21.15.167:0/2262883315 <== osd.0 v2:172.21.15.118:6800/27690 37 ==== osd_op_reply(116 notify.6 [notify-ack] v0'0 uv138 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f32dc050040 con 0x7f32e0007030 2023-10-05T19:28:35.083 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.074+0000 7f32e9d72700 1 -- 172.21.15.167:0/2262883315 <== osd.0 v2:172.21.15.118:6800/27690 38 ==== watch-notify(notify_complete (2) cookie 94602412959056 notify 128849018895 ret 0) v3 ==== 42+0+48 (crc 0 0 0) 0x7f32dc002560 con 0x7f32e0007030 2023-10-05T19:28:35.083 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.078+0000 7f32eba3ce40 10 removing default.rgw.meta+root+testx/myfoo from cache 2023-10-05T19:28:35.083 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.078+0000 7f32eba3ce40 10 distributing notification oid=default.rgw.control:notify.4 cni=[op: 1, obj: default.rgw.meta:root:testx/myfoo, ofs0, ns] 2023-10-05T19:28:35.083 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.078+0000 7f32eba3ce40 1 -- 172.21.15.167:0/2262883315 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:117 7.d 7:b4812045:::notify.4:head [notify cookie 94602412959024 in=167b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x560a57380370 con 0x560a572b97f0 2023-10-05T19:28:35.083 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.078+0000 7f32e9d72700 1 -- 172.21.15.167:0/2262883315 <== osd.1 v2:172.21.15.167:6800/30035 41 ==== watch-notify(notify (1) cookie 94602412392160 notify 128849018895 ret 0) v3 ==== 197+0+0 (crc 0 0 0) 0x7f32dc002560 con 0x560a572b97f0 2023-10-05T19:28:35.083 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.078+0000 7f32daffd700 10 rgw watcher librados: RGWWatcher::handle_notify() notify_id 128849018895 cookie 94602412392160 notifier 4503 bl.length()=155 2023-10-05T19:28:35.083 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.078+0000 7f32daffd700 1 -- 172.21.15.167:0/2262883315 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:118 7.d 7:b4812045:::notify.4:head [notify-ack in=20b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x7f32c8003d90 con 0x560a572b97f0 2023-10-05T19:28:35.084 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.078+0000 7f32e9d72700 1 -- 172.21.15.167:0/2262883315 <== osd.1 v2:172.21.15.167:6800/30035 42 ==== osd_op_reply(117 notify.4 [notify cookie 94602412959024 out=8b] v0'0 uv138 ondisk = 0) v8 ==== 152+0+8 (crc 0 0 0) 0x7f32dc050040 con 0x560a572b97f0 2023-10-05T19:28:35.084 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.078+0000 7f32e9d72700 1 -- 172.21.15.167:0/2262883315 <== osd.1 v2:172.21.15.167:6800/30035 43 ==== watch-notify(notify_complete (2) cookie 94602412959024 notify 128849018895 ret 0) v3 ==== 42+0+48 (crc 0 0 0) 0x7f32dc065500 con 0x560a572b97f0 2023-10-05T19:28:35.084 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.078+0000 7f32eba3ce40 1 -- 172.21.15.167:0/2262883315 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:119 8.5 8:a7a6c9d8:root::testx%2fmyfoo:head [call version.inc in=38b,delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x560a57382a20 con 0x560a572b97f0 2023-10-05T19:28:35.084 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.078+0000 7f32e9d72700 1 -- 172.21.15.167:0/2262883315 <== osd.1 v2:172.21.15.167:6800/30035 44 ==== osd_op_reply(118 notify.4 [notify-ack] v0'0 uv138 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f32dc050040 con 0x560a572b97f0 2023-10-05T19:28:35.086 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.082+0000 7f32e9d72700 1 -- 172.21.15.167:0/2262883315 <== osd.1 v2:172.21.15.167:6800/30035 45 ==== osd_op_reply(119 testx/myfoo [call,delete] v30'5 uv5 ondisk = 0) v8 ==== 197+0+0 (crc 0 0 0) 0x7f32dc050040 con 0x560a572b97f0 2023-10-05T19:28:35.087 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.082+0000 7f32eba3ce40 10 removing default.rgw.meta+root+.bucket.meta.testx:myfoo:9a9df965-6612-45d4-a45f-6e3c889bacc2.4223.1 from cache 2023-10-05T19:28:35.087 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.082+0000 7f32eba3ce40 10 distributing notification oid=default.rgw.control:notify.6 cni=[op: 1, obj: default.rgw.meta:root:.bucket.meta.testx:myfoo:9a9df965-6612-45d4-a45f-6e3c889bacc2.4223.1, ofs0, ns] 2023-10-05T19:28:35.087 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.082+0000 7f32eba3ce40 1 -- 172.21.15.167:0/2262883315 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:120 7.14 7:2b04a3e9:::notify.6:head [notify cookie 94602412931104 in=224b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x560a57382a20 con 0x7f32e0007030 2023-10-05T19:28:35.088 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.082+0000 7f32e9d72700 1 -- 172.21.15.167:0/2262883315 <== osd.0 v2:172.21.15.118:6800/27690 39 ==== watch-notify(notify (1) cookie 94602412403088 notify 128849018896 ret 0) v3 ==== 254+0+0 (crc 0 0 0) 0x7f32dc065500 con 0x7f32e0007030 2023-10-05T19:28:35.088 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.082+0000 7f32da7fc700 10 rgw watcher librados: RGWWatcher::handle_notify() notify_id 128849018896 cookie 94602412403088 notifier 4503 bl.length()=212 2023-10-05T19:28:35.088 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.082+0000 7f32e9d72700 1 -- 172.21.15.167:0/2262883315 <== osd.0 v2:172.21.15.118:6800/27690 40 ==== osd_op_reply(120 notify.6 [notify cookie 94602412931104 out=8b] v0'0 uv138 ondisk = 0) v8 ==== 152+0+8 (crc 0 0 0) 0x7f32dc050040 con 0x7f32e0007030 2023-10-05T19:28:35.088 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.082+0000 7f32da7fc700 1 -- 172.21.15.167:0/2262883315 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:121 7.14 7:2b04a3e9:::notify.6:head [notify-ack in=20b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x7f32380032a0 con 0x7f32e0007030 2023-10-05T19:28:35.088 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.082+0000 7f32e9d72700 1 -- 172.21.15.167:0/2262883315 <== osd.0 v2:172.21.15.118:6800/27690 41 ==== osd_op_reply(121 notify.6 [notify-ack] v0'0 uv138 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f32dc050040 con 0x7f32e0007030 2023-10-05T19:28:35.089 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.082+0000 7f32e9d72700 1 -- 172.21.15.167:0/2262883315 <== osd.0 v2:172.21.15.118:6800/27690 42 ==== watch-notify(notify_complete (2) cookie 94602412931104 notify 128849018896 ret 0) v3 ==== 42+0+48 (crc 0 0 0) 0x7f32dc065a00 con 0x7f32e0007030 2023-10-05T19:28:35.089 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.082+0000 7f32eba3ce40 1 -- 172.21.15.167:0/2262883315 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:122 8.6 8:6736df0c:root::.bucket.meta.testx%3amyfoo%3a9a9df965-6612-45d4-a45f-6e3c889bacc2.4223.1:head [call version.inc in=38b,delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x560a5737b820 con 0x7f32e0007030 2023-10-05T19:28:35.091 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.086+0000 7f32e9d72700 1 -- 172.21.15.167:0/2262883315 <== osd.0 v2:172.21.15.118:6800/27690 43 ==== osd_op_reply(122 .bucket.meta.testx:myfoo:9a9df965-6612-45d4-a45f-6e3c889bacc2.4223.1 [call,delete] v30'2 uv2 ondisk = 0) v8 ==== 254+0+0 (crc 0 0 0) 0x7f32dc050040 con 0x7f32e0007030 2023-10-05T19:28:35.092 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.086+0000 7f32eba3ce40 1 -- 172.21.15.167:0/2262883315 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:123 7.1f 7:f95f44c2:::notify.0:head [watch unwatch cookie 94602412382176] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x560a572d0a90 con 0x7f32e4027be0 2023-10-05T19:28:35.095 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.090+0000 7f32e8d70700 1 -- 172.21.15.167:0/2262883315 <== osd.2 v2:172.21.15.167:6816/30037 19 ==== osd_op_reply(123 notify.0 [watch unwatch cookie 94602412382176] v30'140 uv138 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f32e0032040 con 0x7f32e4027be0 2023-10-05T19:28:35.095 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.090+0000 7f32eba3ce40 20 remove_watcher() i=0 2023-10-05T19:28:35.095 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.090+0000 7f32eba3ce40 2 removed watcher, disabling cache 2023-10-05T19:28:35.095 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.090+0000 7f32eba3ce40 1 -- 172.21.15.167:0/2262883315 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:124 7.0 7:05bf5b68:::notify.1:head [watch unwatch cookie 94602412387008] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x560a572d0a90 con 0x560a572b97f0 2023-10-05T19:28:35.098 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.090+0000 7f32e9d72700 1 -- 172.21.15.167:0/2262883315 <== osd.1 v2:172.21.15.167:6800/30035 46 ==== osd_op_reply(124 notify.1 [watch unwatch cookie 94602412387008] v30'140 uv138 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f32dc050040 con 0x560a572b97f0 2023-10-05T19:28:35.098 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.090+0000 7f32eba3ce40 20 remove_watcher() i=1 2023-10-05T19:28:35.098 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.090+0000 7f32eba3ce40 1 -- 172.21.15.167:0/2262883315 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:125 7.15 7:a93a5511:::notify.2:head [watch unwatch cookie 94602412385408] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x560a572d0a90 con 0x560a572f0860 2023-10-05T19:28:35.101 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.094+0000 7f32e9571700 1 -- 172.21.15.167:0/2262883315 <== osd.3 v2:172.21.15.167:6808/30036 25 ==== osd_op_reply(125 notify.2 [watch unwatch cookie 94602412385408] v30'140 uv138 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f32e4024020 con 0x560a572f0860 2023-10-05T19:28:35.101 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.094+0000 7f32eba3ce40 20 remove_watcher() i=2 2023-10-05T19:28:35.101 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.094+0000 7f32eba3ce40 1 -- 172.21.15.167:0/2262883315 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:126 7.e 7:7759931f:::notify.3:head [watch unwatch cookie 94602412389008] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x560a572f57e0 con 0x7f32e4027be0 2023-10-05T19:28:35.106 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.098+0000 7f32e8d70700 1 -- 172.21.15.167:0/2262883315 <== osd.2 v2:172.21.15.167:6816/30037 20 ==== osd_op_reply(126 notify.3 [watch unwatch cookie 94602412389008] v30'140 uv138 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f32e0032040 con 0x7f32e4027be0 2023-10-05T19:28:35.106 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.098+0000 7f32eba3ce40 20 remove_watcher() i=3 2023-10-05T19:28:35.107 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.098+0000 7f32eba3ce40 1 -- 172.21.15.167:0/2262883315 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:127 7.d 7:b4812045:::notify.4:head [watch unwatch cookie 94602412392160] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x560a572f7290 con 0x560a572b97f0 2023-10-05T19:28:35.107 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.102+0000 7f32e9d72700 1 -- 172.21.15.167:0/2262883315 <== osd.1 v2:172.21.15.167:6800/30035 47 ==== osd_op_reply(127 notify.4 [watch unwatch cookie 94602412392160] v30'140 uv138 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f32dc050040 con 0x560a572b97f0 2023-10-05T19:28:35.107 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.102+0000 7f32eba3ce40 20 remove_watcher() i=4 2023-10-05T19:28:35.107 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.102+0000 7f32eba3ce40 1 -- 172.21.15.167:0/2262883315 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:128 7.3 7:c609908c:::notify.5:head [watch unwatch cookie 94602412399040] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x560a572f57e0 con 0x7f32e0007030 2023-10-05T19:28:35.109 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.102+0000 7f32e9d72700 1 -- 172.21.15.167:0/2262883315 <== osd.0 v2:172.21.15.118:6800/27690 44 ==== osd_op_reply(128 notify.5 [watch unwatch cookie 94602412399040] v30'140 uv138 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f32dc050040 con 0x7f32e0007030 2023-10-05T19:28:35.109 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.102+0000 7f32eba3ce40 20 remove_watcher() i=5 2023-10-05T19:28:35.109 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.102+0000 7f32eba3ce40 1 -- 172.21.15.167:0/2262883315 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:129 7.14 7:2b04a3e9:::notify.6:head [watch unwatch cookie 94602412403088] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x560a572f7ee0 con 0x7f32e0007030 2023-10-05T19:28:35.113 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.106+0000 7f32e9d72700 1 -- 172.21.15.167:0/2262883315 <== osd.0 v2:172.21.15.118:6800/27690 45 ==== osd_op_reply(129 notify.6 [watch unwatch cookie 94602412403088] v30'140 uv138 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f32dc050040 con 0x7f32e0007030 2023-10-05T19:28:35.113 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.106+0000 7f32eba3ce40 20 remove_watcher() i=6 2023-10-05T19:28:35.113 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.106+0000 7f32eba3ce40 1 -- 172.21.15.167:0/2262883315 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:130 7.9 7:93e5b521:::notify.7:head [watch unwatch cookie 94602412405616] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x560a572f99c0 con 0x7f32e0007030 2023-10-05T19:28:35.116 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.110+0000 7f32e9d72700 1 -- 172.21.15.167:0/2262883315 <== osd.0 v2:172.21.15.118:6800/27690 46 ==== osd_op_reply(130 notify.7 [watch unwatch cookie 94602412405616] v30'140 uv138 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f32dc050040 con 0x7f32e0007030 2023-10-05T19:28:35.116 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.110+0000 7f32eba3ce40 20 remove_watcher() i=7 2023-10-05T19:28:35.118 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.110+0000 7f32ba7e4700 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2023-10-05T19:28:35.118 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.110+0000 7f32eba3ce40 1 -- 172.21.15.167:0/2262883315 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:131 6.4 6:22d26bf9:::data_loggenerations_metadata:head [watch unwatch cookie 94602412406720] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x560a572f57e0 con 0x560a572b97f0 2023-10-05T19:28:35.122 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.114+0000 7f32e9d72700 1 -- 172.21.15.167:0/2262883315 <== osd.1 v2:172.21.15.167:6800/30035 48 ==== osd_op_reply(131 data_loggenerations_metadata [watch unwatch cookie 94602412406720] v30'134 uv1 ondisk = 0) v8 ==== 172+0+0 (crc 0 0 0) 0x7f32dc050040 con 0x560a572b97f0 2023-10-05T19:28:35.122 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.114+0000 7f32eba3ce40 1 -- 172.21.15.167:0/2262883315 >> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] conn(0x7f32e0007030 msgr2=0x7f32e001c920 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-05T19:28:35.122 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.114+0000 7f32eba3ce40 1 --2- 172.21.15.167:0/2262883315 >> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] conn(0x7f32e0007030 0x7f32e001c920 crc :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-05T19:28:35.122 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.114+0000 7f32eba3ce40 1 -- 172.21.15.167:0/2262883315 >> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] conn(0x560a572b97f0 msgr2=0x560a572eca30 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-05T19:28:35.122 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.114+0000 7f32eba3ce40 1 --2- 172.21.15.167:0/2262883315 >> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] conn(0x560a572b97f0 0x560a572eca30 crc :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-05T19:28:35.122 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.114+0000 7f32eba3ce40 1 -- 172.21.15.167:0/2262883315 >> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] conn(0x7f32e4027be0 msgr2=0x7f32e4028710 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-05T19:28:35.122 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.114+0000 7f32eba3ce40 1 --2- 172.21.15.167:0/2262883315 >> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] conn(0x7f32e4027be0 0x7f32e4028710 crc :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-05T19:28:35.122 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.114+0000 7f32eba3ce40 1 -- 172.21.15.167:0/2262883315 >> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] conn(0x560a572f0860 msgr2=0x560a572f2c40 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-05T19:28:35.123 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.114+0000 7f32eba3ce40 1 --2- 172.21.15.167:0/2262883315 >> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] conn(0x560a572f0860 0x560a572f2c40 crc :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-05T19:28:35.123 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.114+0000 7f32eba3ce40 1 -- 172.21.15.167:0/2262883315 >> [v2:172.21.15.167:6824/29889,v1:172.21.15.167:6825/29889] conn(0x7f32cc030c00 msgr2=0x7f32cc0330b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-05T19:28:35.123 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.114+0000 7f32eba3ce40 1 --2- 172.21.15.167:0/2262883315 >> [v2:172.21.15.167:6824/29889,v1:172.21.15.167:6825/29889] conn(0x7f32cc030c00 0x7f32cc0330b0 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7f32e0005a90 tx=0x7f32e0015000 comp rx=0 tx=0).stop 2023-10-05T19:28:35.123 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.114+0000 7f32eba3ce40 1 -- 172.21.15.167:0/2262883315 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x560a572b9420 msgr2=0x560a572e0f40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-05T19:28:35.123 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.114+0000 7f32e9d72700 1 -- 172.21.15.167:0/2262883315 reap_dead start 2023-10-05T19:28:35.123 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.114+0000 7f32eba3ce40 1 --2- 172.21.15.167:0/2262883315 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x560a572b9420 0x560a572e0f40 secure :-1 s=READY pgs=413 cs=0 l=1 rev1=1 crypto rx=0x7f32e4006bd0 tx=0x7f32e4014000 comp rx=0 tx=0).stop 2023-10-05T19:28:35.123 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.114+0000 7f32eba3ce40 1 -- 172.21.15.167:0/2262883315 shutdown_connections 2023-10-05T19:28:35.123 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.114+0000 7f32eba3ce40 1 --2- 172.21.15.167:0/2262883315 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x560a572b9420 0x560a572e0f40 unknown :-1 s=CLOSED pgs=413 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-05T19:28:35.123 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.114+0000 7f32eba3ce40 1 -- 172.21.15.167:0/2262883315 >> 172.21.15.167:0/2262883315 conn(0x560a572b3730 msgr2=0x560a572d47b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-05T19:28:35.124 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.114+0000 7f32eba3ce40 1 -- 172.21.15.167:0/2262883315 shutdown_connections 2023-10-05T19:28:35.124 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.114+0000 7f32eba3ce40 1 -- 172.21.15.167:0/2262883315 wait complete. 2023-10-05T19:28:35.125 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.118+0000 7f32eba3ce40 1 -- 172.21.15.167:0/3289891311 >> [v2:172.21.15.167:6824/29889,v1:172.21.15.167:6825/29889] conn(0x7f3230030c50 msgr2=0x7f3230033100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-05T19:28:35.125 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.118+0000 7f32eba3ce40 1 --2- 172.21.15.167:0/3289891311 >> [v2:172.21.15.167:6824/29889,v1:172.21.15.167:6825/29889] conn(0x7f3230030c50 0x7f3230033100 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x560a572fae90 tx=0x7f32dc000f90 comp rx=0 tx=0).stop 2023-10-05T19:28:35.125 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.118+0000 7f32eba3ce40 1 -- 172.21.15.167:0/3289891311 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x560a57309310 msgr2=0x560a5730ee40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-05T19:28:35.125 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.118+0000 7f32eba3ce40 1 --2- 172.21.15.167:0/3289891311 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x560a57309310 0x560a5730ee40 secure :-1 s=READY pgs=415 cs=0 l=1 rev1=1 crypto rx=0x560a57303390 tx=0x7f32e002f000 comp rx=0 tx=0).stop 2023-10-05T19:28:35.125 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.118+0000 7f32eba3ce40 1 -- 172.21.15.167:0/3289891311 shutdown_connections 2023-10-05T19:28:35.125 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.118+0000 7f32eba3ce40 1 --2- 172.21.15.167:0/3289891311 >> [v2:172.21.15.167:6824/29889,v1:172.21.15.167:6825/29889] conn(0x7f3230030c50 0x7f3230033100 unknown :-1 s=CLOSED pgs=145 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-05T19:28:35.125 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.118+0000 7f32eba3ce40 1 --2- 172.21.15.167:0/3289891311 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x560a57309310 0x560a5730ee40 unknown :-1 s=CLOSED pgs=415 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-05T19:28:35.125 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.118+0000 7f32eba3ce40 1 -- 172.21.15.167:0/3289891311 >> 172.21.15.167:0/3289891311 conn(0x560a573046d0 msgr2=0x560a57302770 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-05T19:28:35.125 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.118+0000 7f32eba3ce40 1 -- 172.21.15.167:0/3289891311 shutdown_connections 2023-10-05T19:28:35.126 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.118+0000 7f32eba3ce40 1 -- 172.21.15.167:0/3289891311 wait complete. 2023-10-05T19:28:35.128 INFO:tasks.util.rgw:rgwadmin: client.0 : ['user', 'rm', '--tenant', 'testx', '--uid', 'tenanteduser'] 2023-10-05T19:28:35.128 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'ceph', 'user', 'rm', '--tenant', 'testx', '--uid', 'tenanteduser'] 2023-10-05T19:28:35.128 DEBUG:teuthology.orchestra.run.smithi167:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster ceph user rm --tenant testx --uid tenanteduser 2023-10-05T19:28:35.170 INFO:teuthology.orchestra.run.smithi167.stderr:ignoring --setuser ceph since I am not root 2023-10-05T19:28:35.170 INFO:teuthology.orchestra.run.smithi167.stderr:ignoring --setgroup ceph since I am not root 2023-10-05T19:28:35.171 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.166+0000 7f2db4522e40 1 Processor -- start 2023-10-05T19:28:35.171 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.166+0000 7f2db4522e40 1 -- start start 2023-10-05T19:28:35.171 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.166+0000 7f2db4522e40 1 --2- >> v2:172.21.15.118:3300/0 conn(0x560a8ac51d10 0x560a8ac520e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-05T19:28:35.171 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.166+0000 7f2db4522e40 1 -- --> v1:172.21.15.118:6789/0 -- auth(proto 0 26 bytes epoch 0) v1 -- 0x560a8ab9fbd0 con 0x560a8ac52620 2023-10-05T19:28:35.171 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.166+0000 7f2db4522e40 1 -- --> v2:172.21.15.118:3300/0 -- mon_getmap magic: 0 v1 -- 0x560a8ab5cee0 con 0x560a8ac51d10 2023-10-05T19:28:35.172 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.166+0000 7f2db2057700 1 --2- >> v2:172.21.15.118:3300/0 conn(0x560a8ac51d10 0x560a8ac520e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-05T19:28:35.172 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.166+0000 7f2db1856700 1 --1- >> v1:172.21.15.118:6789/0 conn(0x560a8ac52620 0x560a8ac530c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.118:6789/0 says I am v1:172.21.15.167:59278/0 (socket says 172.21.15.167:59278) 2023-10-05T19:28:35.172 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.166+0000 7f2db1856700 1 -- 172.21.15.167:0/2942239332 learned_addr learned my addr 172.21.15.167:0/2942239332 (peer_addr_for_me v1:172.21.15.167:0/0) 2023-10-05T19:28:35.172 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.166+0000 7f2db2057700 1 -- 172.21.15.167:0/2942239332 >> v1:172.21.15.118:6789/0 conn(0x560a8ac52620 legacy=0x560a8ac530c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-05T19:28:35.172 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.166+0000 7f2db2057700 1 -- 172.21.15.167:0/2942239332 --> v2:172.21.15.118:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x560a8ac537d0 con 0x560a8ac51d10 2023-10-05T19:28:35.172 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.166+0000 7f2db2057700 1 --2- 172.21.15.167:0/2942239332 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x560a8ac51d10 0x560a8ac520e0 secure :-1 s=READY pgs=417 cs=0 l=1 rev1=1 crypto rx=0x7f2dac00a520 tx=0x7f2dac005d40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a36085649e2bb153 server_cookie=0 in_seq=0 out_seq=0 2023-10-05T19:28:35.173 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.166+0000 7f2db1055700 1 -- 172.21.15.167:0/2942239332 <== mon.0 v2:172.21.15.118:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f2dac00f590 con 0x560a8ac51d10 2023-10-05T19:28:35.173 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.166+0000 7f2db1055700 1 -- 172.21.15.167:0/2942239332 <== mon.0 v2:172.21.15.118:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2dac00f6f0 con 0x560a8ac51d10 2023-10-05T19:28:35.174 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.166+0000 7f2db1055700 1 -- 172.21.15.167:0/2942239332 <== mon.0 v2:172.21.15.118:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f2dac00f880 con 0x560a8ac51d10 2023-10-05T19:28:35.174 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.166+0000 7f2db4522e40 1 -- 172.21.15.167:0/2942239332 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x560a8ac51d10 msgr2=0x560a8ac520e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-05T19:28:35.174 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.166+0000 7f2db4522e40 1 --2- 172.21.15.167:0/2942239332 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x560a8ac51d10 0x560a8ac520e0 secure :-1 s=READY pgs=417 cs=0 l=1 rev1=1 crypto rx=0x7f2dac00a520 tx=0x7f2dac005d40 comp rx=0 tx=0).stop 2023-10-05T19:28:35.174 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.166+0000 7f2db4522e40 1 -- 172.21.15.167:0/2942239332 shutdown_connections 2023-10-05T19:28:35.174 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.166+0000 7f2db4522e40 1 --2- 172.21.15.167:0/2942239332 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x560a8ac51d10 0x560a8ac520e0 unknown :-1 s=CLOSED pgs=417 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-05T19:28:35.174 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.166+0000 7f2db4522e40 1 -- 172.21.15.167:0/2942239332 >> 172.21.15.167:0/2942239332 conn(0x560a8ac4d730 msgr2=0x560a8ac4fb40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-05T19:28:35.174 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.166+0000 7f2db4522e40 1 -- 172.21.15.167:0/2942239332 shutdown_connections 2023-10-05T19:28:35.174 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.166+0000 7f2db4522e40 1 -- 172.21.15.167:0/2942239332 wait complete. 2023-10-05T19:28:35.177 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.170+0000 7f2db4522e40 1 Processor -- start 2023-10-05T19:28:35.177 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.170+0000 7f2db4522e40 1 -- start start 2023-10-05T19:28:35.177 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.170+0000 7f2db4522e40 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x560a8ac51d10 0x560a8ac6fce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-05T19:28:35.177 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.170+0000 7f2db4522e40 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x560a8ab20e30 con 0x560a8ac51d10 2023-10-05T19:28:35.177 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.170+0000 7f2db2057700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x560a8ac51d10 0x560a8ac6fce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-05T19:28:35.177 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.170+0000 7f2db2057700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x560a8ac51d10 0x560a8ac6fce0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.118:3300/0 says I am v2:172.21.15.167:41614/0 (socket says 172.21.15.167:41614) 2023-10-05T19:28:35.177 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.170+0000 7f2db2057700 1 -- 172.21.15.167:0/3024382932 learned_addr learned my addr 172.21.15.167:0/3024382932 (peer_addr_for_me v2:172.21.15.167:0/0) 2023-10-05T19:28:35.177 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.170+0000 7f2db2057700 1 -- 172.21.15.167:0/3024382932 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2dac00a0e0 con 0x560a8ac51d10 2023-10-05T19:28:35.177 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.170+0000 7f2db2057700 1 --2- 172.21.15.167:0/3024382932 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x560a8ac51d10 0x560a8ac6fce0 secure :-1 s=READY pgs=418 cs=0 l=1 rev1=1 crypto rx=0x7f2dac00a470 tx=0x7f2dac014000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=473c1396d564fcab server_cookie=0 in_seq=0 out_seq=0 2023-10-05T19:28:35.177 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.170+0000 7f2dabfff700 1 -- 172.21.15.167:0/3024382932 <== mon.0 v2:172.21.15.118:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f2dac00f860 con 0x560a8ac51d10 2023-10-05T19:28:35.178 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.170+0000 7f2dabfff700 1 -- 172.21.15.167:0/3024382932 <== mon.0 v2:172.21.15.118:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2dac00f9e0 con 0x560a8ac51d10 2023-10-05T19:28:35.178 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.170+0000 7f2dabfff700 1 -- 172.21.15.167:0/3024382932 <== mon.0 v2:172.21.15.118:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f2dac00fb90 con 0x560a8ac51d10 2023-10-05T19:28:35.178 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.170+0000 7f2db4522e40 1 -- 172.21.15.167:0/3024382932 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x560a8ac51d10 msgr2=0x560a8ac6fce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-05T19:28:35.178 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.170+0000 7f2db4522e40 1 --2- 172.21.15.167:0/3024382932 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x560a8ac51d10 0x560a8ac6fce0 secure :-1 s=READY pgs=418 cs=0 l=1 rev1=1 crypto rx=0x7f2dac00a470 tx=0x7f2dac014000 comp rx=0 tx=0).stop 2023-10-05T19:28:35.178 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.170+0000 7f2db4522e40 1 -- 172.21.15.167:0/3024382932 shutdown_connections 2023-10-05T19:28:35.178 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.170+0000 7f2db4522e40 1 --2- 172.21.15.167:0/3024382932 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x560a8ac51d10 0x560a8ac6fce0 unknown :-1 s=CLOSED pgs=418 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-05T19:28:35.178 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.170+0000 7f2db4522e40 1 -- 172.21.15.167:0/3024382932 >> 172.21.15.167:0/3024382932 conn(0x560a8ac4d730 msgr2=0x560a8ac6b8a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-05T19:28:35.178 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.170+0000 7f2db4522e40 1 -- 172.21.15.167:0/3024382932 shutdown_connections 2023-10-05T19:28:35.178 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.170+0000 7f2db4522e40 1 -- 172.21.15.167:0/3024382932 wait complete. 2023-10-05T19:28:35.179 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.170+0000 7f2db4522e40 1 Processor -- start 2023-10-05T19:28:35.179 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.170+0000 7f2db4522e40 1 -- start start 2023-10-05T19:28:35.179 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.170+0000 7f2db4522e40 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x560a8ac51d10 0x560a8ac76940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-05T19:28:35.179 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.170+0000 7f2db4522e40 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x560a8aaf3ed0 con 0x560a8ac51d10 2023-10-05T19:28:35.180 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.170+0000 7f2db2057700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x560a8ac51d10 0x560a8ac76940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-05T19:28:35.180 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.170+0000 7f2db2057700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x560a8ac51d10 0x560a8ac76940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.118:3300/0 says I am v2:172.21.15.167:41622/0 (socket says 172.21.15.167:41622) 2023-10-05T19:28:35.180 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.170+0000 7f2db2057700 1 -- 172.21.15.167:0/2665859004 learned_addr learned my addr 172.21.15.167:0/2665859004 (peer_addr_for_me v2:172.21.15.167:0/0) 2023-10-05T19:28:35.180 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.170+0000 7f2db2057700 1 -- 172.21.15.167:0/2665859004 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x560a8ac537d0 con 0x560a8ac51d10 2023-10-05T19:28:35.180 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.174+0000 7f2db2057700 1 --2- 172.21.15.167:0/2665859004 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x560a8ac51d10 0x560a8ac76940 secure :-1 s=READY pgs=419 cs=0 l=1 rev1=1 crypto rx=0x7f2dac00a470 tx=0x7f2dac01d000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-05T19:28:35.181 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.174+0000 7f2daa7fc700 1 -- 172.21.15.167:0/2665859004 <== mon.0 v2:172.21.15.118:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f2dac01ce00 con 0x560a8ac51d10 2023-10-05T19:28:35.181 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.174+0000 7f2daa7fc700 1 -- 172.21.15.167:0/2665859004 <== mon.0 v2:172.21.15.118:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2dac010630 con 0x560a8ac51d10 2023-10-05T19:28:35.181 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.174+0000 7f2db4522e40 1 -- 172.21.15.167:0/2665859004 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x560a8ac76e80 con 0x560a8ac51d10 2023-10-05T19:28:35.181 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.174+0000 7f2daa7fc700 1 -- 172.21.15.167:0/2665859004 <== mon.0 v2:172.21.15.118:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f2dac010820 con 0x560a8ac51d10 2023-10-05T19:28:35.181 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.174+0000 7f2db4522e40 1 -- 172.21.15.167:0/2665859004 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x560a8ac772c0 con 0x560a8ac51d10 2023-10-05T19:28:35.181 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.174+0000 7f2db4522e40 20 rados->read ofs=0 len=0 2023-10-05T19:28:35.181 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.174+0000 7f2daa7fc700 1 -- 172.21.15.167:0/2665859004 <== mon.0 v2:172.21.15.118:3300/0 4 ==== mgrmap(e 5) v1 ==== 40329+0+0 (secure 0 0 0) 0x7f2dac010980 con 0x560a8ac51d10 2023-10-05T19:28:35.182 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.174+0000 7f2daa7fc700 1 --2- 172.21.15.167:0/2665859004 >> [v2:172.21.15.167:6824/29889,v1:172.21.15.167:6825/29889] conn(0x7f2d94030c50 0x7f2d94033100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-05T19:28:35.183 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.174+0000 7f2daa7fc700 1 -- 172.21.15.167:0/2665859004 <== mon.0 v2:172.21.15.118:3300/0 5 ==== osd_map(30..30 src has 1..30) v4 ==== 6057+0+0 (secure 0 0 0) 0x7f2dac02b070 con 0x560a8ac51d10 2023-10-05T19:28:35.183 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.174+0000 7f2db1856700 1 --2- 172.21.15.167:0/2665859004 >> [v2:172.21.15.167:6824/29889,v1:172.21.15.167:6825/29889] conn(0x7f2d94030c50 0x7f2d94033100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-05T19:28:35.183 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.174+0000 7f2db4522e40 1 --2- 172.21.15.167:0/2665859004 >> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] conn(0x560a8ac52620 0x560a8ac823d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-05T19:28:35.183 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.174+0000 7f2db4522e40 1 -- 172.21.15.167:0/2665859004 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:1 5.12 5:49953fa1:::default.realm:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x560a8ac82a40 con 0x560a8ac52620 2023-10-05T19:28:35.183 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.174+0000 7f2db2858700 1 --2- 172.21.15.167:0/2665859004 >> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] conn(0x560a8ac52620 0x560a8ac823d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-05T19:28:35.183 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.174+0000 7f2db1856700 1 --2- 172.21.15.167:0/2665859004 >> [v2:172.21.15.167:6824/29889,v1:172.21.15.167:6825/29889] conn(0x7f2d94030c50 0x7f2d94033100 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7f2d9c004170 tx=0x7f2d9c001380 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-05T19:28:35.183 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.178+0000 7f2db2858700 1 --2- 172.21.15.167:0/2665859004 >> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] conn(0x560a8ac52620 0x560a8ac823d0 crc :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-05T19:28:35.183 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.178+0000 7f2db2858700 1 -- 172.21.15.167:0/2665859004 <== osd.1 v2:172.21.15.167:6800/30035 1 ==== osd_op_reply(1 default.realm [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 ==== 157+0+0 (crc 0 0 0) 0x7f2da400a500 con 0x560a8ac52620 2023-10-05T19:28:35.183 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.178+0000 7f2db4522e40 20 rados_obj.operate() r=-2 bl.length=0 2023-10-05T19:28:35.184 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.178+0000 7f2db4522e40 20 realm 2023-10-05T19:28:35.184 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.178+0000 7f2db4522e40 20 rados->read ofs=0 len=0 2023-10-05T19:28:35.184 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.178+0000 7f2db4522e40 1 -- 172.21.15.167:0/2665859004 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:2 5.12 5:49953fa1:::default.realm:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x560a8ac7fa30 con 0x560a8ac52620 2023-10-05T19:28:35.185 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.178+0000 7f2db2858700 1 -- 172.21.15.167:0/2665859004 <== osd.1 v2:172.21.15.167:6800/30035 2 ==== osd_op_reply(2 default.realm [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 ==== 157+0+0 (crc 0 0 0) 0x7f2da400a500 con 0x560a8ac52620 2023-10-05T19:28:35.185 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.178+0000 7f2db4522e40 20 rados_obj.operate() r=-2 bl.length=0 2023-10-05T19:28:35.185 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.178+0000 7f2db4522e40 4 RGWPeriod::init failed to init realm id : (2) No such file or directory 2023-10-05T19:28:35.186 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.178+0000 7f2db4522e40 20 rados->read ofs=0 len=0 2023-10-05T19:28:35.186 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.178+0000 7f2db4522e40 1 -- 172.21.15.167:0/2665859004 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:3 5.12 5:49953fa1:::default.realm:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x560a8ac83520 con 0x560a8ac52620 2023-10-05T19:28:35.186 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.178+0000 7f2db2858700 1 -- 172.21.15.167:0/2665859004 <== osd.1 v2:172.21.15.167:6800/30035 3 ==== osd_op_reply(3 default.realm [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 ==== 157+0+0 (crc 0 0 0) 0x7f2da400a500 con 0x560a8ac52620 2023-10-05T19:28:35.186 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.178+0000 7f2db4522e40 20 rados_obj.operate() r=-2 bl.length=0 2023-10-05T19:28:35.186 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.178+0000 7f2db4522e40 20 rados->read ofs=0 len=0 2023-10-05T19:28:35.186 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.178+0000 7f2db4522e40 1 -- 172.21.15.167:0/2665859004 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:4 5.f 5:f43b6ece:::zone_names.default:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x560a8ac83130 con 0x560a8ac52620 2023-10-05T19:28:35.186 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.178+0000 7f2db2858700 1 -- 172.21.15.167:0/2665859004 <== osd.1 v2:172.21.15.167:6800/30035 4 ==== osd_op_reply(4 zone_names.default [read 0~46 out=46b] v0'0 uv1 ondisk = 0) v8 ==== 162+0+46 (crc 0 0 0) 0x7f2da400a500 con 0x560a8ac52620 2023-10-05T19:28:35.186 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.178+0000 7f2db4522e40 20 rados_obj.operate() r=0 bl.length=46 2023-10-05T19:28:35.186 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.178+0000 7f2db4522e40 20 rados->read ofs=0 len=0 2023-10-05T19:28:35.186 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.178+0000 7f2db4522e40 1 -- 172.21.15.167:0/2665859004 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:5 5.1d 5:b8374eb4:::zone_info.9a9df965-6612-45d4-a45f-6e3c889bacc2:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x560a8ac843b0 con 0x560a8ac52620 2023-10-05T19:28:35.187 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.178+0000 7f2db2858700 1 -- 172.21.15.167:0/2665859004 <== osd.1 v2:172.21.15.167:6800/30035 5 ==== osd_op_reply(5 zone_info.9a9df965-6612-45d4-a45f-6e3c889bacc2 [read 0~874 out=874b] v0'0 uv2 ondisk = 0) v8 ==== 190+0+874 (crc 0 0 0) 0x7f2da400a500 con 0x560a8ac52620 2023-10-05T19:28:35.187 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.178+0000 7f2db4522e40 20 rados_obj.operate() r=0 bl.length=874 2023-10-05T19:28:35.187 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.178+0000 7f2db4522e40 20 searching for the correct realm 2023-10-05T19:28:35.187 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.178+0000 7f2db4522e40 1 --2- 172.21.15.167:0/2665859004 >> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] conn(0x560a8ac86170 0x560a8ac88550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-05T19:28:35.187 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.178+0000 7f2db4522e40 1 -- 172.21.15.167:0/2665859004 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:6 5.0 5:00000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x560a8ac88a90 con 0x560a8ac86170 2023-10-05T19:28:35.187 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.178+0000 7f2db2057700 1 --2- 172.21.15.167:0/2665859004 >> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] conn(0x560a8ac86170 0x560a8ac88550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-05T19:28:35.188 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.178+0000 7f2db2057700 1 --2- 172.21.15.167:0/2665859004 >> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] conn(0x560a8ac86170 0x560a8ac88550 crc :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-05T19:28:35.188 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.182+0000 7f2db2057700 1 -- 172.21.15.167:0/2665859004 <== osd.3 v2:172.21.15.167:6808/30036 1 ==== osd_op_reply(6 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f2dac024920 con 0x560a8ac86170 2023-10-05T19:28:35.188 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.182+0000 7f2db2057700 1 --2- 172.21.15.167:0/2665859004 >> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] conn(0x7f2dac037c00 0x7f2dac0395e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-05T19:28:35.188 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.182+0000 7f2db2057700 1 -- 172.21.15.167:0/2665859004 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:7 5.10 5:08000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x560a8ac88a90 con 0x7f2dac037c00 2023-10-05T19:28:35.188 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.182+0000 7f2db1856700 1 --2- 172.21.15.167:0/2665859004 >> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] conn(0x7f2dac037c00 0x7f2dac0395e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-05T19:28:35.188 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.182+0000 7f2db1856700 1 --2- 172.21.15.167:0/2665859004 >> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] conn(0x7f2dac037c00 0x7f2dac0395e0 crc :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-05T19:28:35.189 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.182+0000 7f2db1856700 1 -- 172.21.15.167:0/2665859004 <== osd.2 v2:172.21.15.167:6816/30037 1 ==== osd_op_reply(7 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f2d9c006070 con 0x7f2dac037c00 2023-10-05T19:28:35.189 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.182+0000 7f2db1856700 1 --2- 172.21.15.167:0/2665859004 >> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] conn(0x7f2d9c00d3d0 0x7f2d9c00d820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-05T19:28:35.189 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.182+0000 7f2db1856700 1 -- 172.21.15.167:0/2665859004 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:8 5.8 5:10000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x560a8ac88a90 con 0x7f2d9c00d3d0 2023-10-05T19:28:35.190 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.182+0000 7f2db2858700 1 --2- 172.21.15.167:0/2665859004 >> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] conn(0x7f2d9c00d3d0 0x7f2d9c00d820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-05T19:28:35.190 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.186+0000 7f2db2858700 1 --2- 172.21.15.167:0/2665859004 >> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] conn(0x7f2d9c00d3d0 0x7f2d9c00d820 crc :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-05T19:28:35.191 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.186+0000 7f2db2858700 1 -- 172.21.15.167:0/2665859004 <== osd.0 v2:172.21.15.118:6800/27690 1 ==== osd_op_reply(8 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f2da4014040 con 0x7f2d9c00d3d0 2023-10-05T19:28:35.191 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.186+0000 7f2db2858700 1 -- 172.21.15.167:0/2665859004 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:9 5.18 5:18000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x560a8ac88a90 con 0x560a8ac52620 2023-10-05T19:28:35.192 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.186+0000 7f2db2858700 1 -- 172.21.15.167:0/2665859004 <== osd.1 v2:172.21.15.167:6800/30035 6 ==== osd_op_reply(9 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f2da4014040 con 0x560a8ac52620 2023-10-05T19:28:35.192 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.186+0000 7f2db2858700 1 -- 172.21.15.167:0/2665859004 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:10 5.4 5:20000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x560a8ac88a90 con 0x560a8ac52620 2023-10-05T19:28:35.193 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.186+0000 7f2db2858700 1 -- 172.21.15.167:0/2665859004 <== osd.1 v2:172.21.15.167:6800/30035 7 ==== osd_op_reply(10 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f2da4014040 con 0x560a8ac52620 2023-10-05T19:28:35.193 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.186+0000 7f2db2858700 1 -- 172.21.15.167:0/2665859004 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:11 5.14 5:28000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x560a8ac88a90 con 0x7f2d9c00d3d0 2023-10-05T19:28:35.194 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.186+0000 7f2db2858700 1 -- 172.21.15.167:0/2665859004 <== osd.0 v2:172.21.15.118:6800/27690 2 ==== osd_op_reply(11 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f2da4014040 con 0x7f2d9c00d3d0 2023-10-05T19:28:35.194 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.186+0000 7f2db2858700 1 -- 172.21.15.167:0/2665859004 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:12 5.c 5:30000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x560a8ac88a90 con 0x7f2d9c00d3d0 2023-10-05T19:28:35.195 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.190+0000 7f2db2858700 1 -- 172.21.15.167:0/2665859004 <== osd.0 v2:172.21.15.118:6800/27690 3 ==== osd_op_reply(12 [pgnls start_epoch 30 out=112b] v23'1 uv1 ondisk = 1) v8 ==== 144+0+112 (crc 0 0 0) 0x7f2da4014040 con 0x7f2d9c00d3d0 2023-10-05T19:28:35.195 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.190+0000 7f2db2858700 1 -- 172.21.15.167:0/2665859004 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:13 5.1c 5:38000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x560a8ac88a90 con 0x7f2dac037c00 2023-10-05T19:28:35.195 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.190+0000 7f2db1856700 1 -- 172.21.15.167:0/2665859004 <== osd.2 v2:172.21.15.167:6816/30037 2 ==== osd_op_reply(13 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f2d9c006070 con 0x7f2dac037c00 2023-10-05T19:28:35.196 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.190+0000 7f2db1856700 1 -- 172.21.15.167:0/2665859004 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:14 5.2 5:40000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x560a8ac88a90 con 0x560a8ac86170 2023-10-05T19:28:35.196 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.190+0000 7f2db2057700 1 -- 172.21.15.167:0/2665859004 <== osd.3 v2:172.21.15.167:6808/30036 2 ==== osd_op_reply(14 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f2dac024920 con 0x560a8ac86170 2023-10-05T19:28:35.196 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.190+0000 7f2db2057700 1 -- 172.21.15.167:0/2665859004 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:15 5.12 5:48000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x560a8ac88a90 con 0x560a8ac52620 2023-10-05T19:28:35.197 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.190+0000 7f2db2858700 1 -- 172.21.15.167:0/2665859004 <== osd.1 v2:172.21.15.167:6800/30035 8 ==== osd_op_reply(15 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f2da4014040 con 0x560a8ac52620 2023-10-05T19:28:35.197 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.190+0000 7f2db2858700 1 -- 172.21.15.167:0/2665859004 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:16 5.a 5:50000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x560a8ac88a90 con 0x7f2d9c00d3d0 2023-10-05T19:28:35.198 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.190+0000 7f2db2858700 1 -- 172.21.15.167:0/2665859004 <== osd.0 v2:172.21.15.118:6800/27690 4 ==== osd_op_reply(16 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f2da4022040 con 0x7f2d9c00d3d0 2023-10-05T19:28:35.198 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.190+0000 7f2db2858700 1 -- 172.21.15.167:0/2665859004 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:17 5.1a 5:58000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x560a8ac88a90 con 0x560a8ac52620 2023-10-05T19:28:35.198 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.194+0000 7f2db2858700 1 -- 172.21.15.167:0/2665859004 <== osd.1 v2:172.21.15.167:6800/30035 9 ==== osd_op_reply(17 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f2da4022040 con 0x560a8ac52620 2023-10-05T19:28:35.198 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.194+0000 7f2db2858700 1 -- 172.21.15.167:0/2665859004 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:18 5.6 5:60000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x560a8ac88a90 con 0x7f2dac037c00 2023-10-05T19:28:35.199 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.194+0000 7f2db1856700 1 -- 172.21.15.167:0/2665859004 <== osd.2 v2:172.21.15.167:6816/30037 3 ==== osd_op_reply(18 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f2d9c006070 con 0x7f2dac037c00 2023-10-05T19:28:35.199 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.194+0000 7f2db1856700 1 -- 172.21.15.167:0/2665859004 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:19 5.16 5:68000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x560a8ac88a90 con 0x560a8ac86170 2023-10-05T19:28:35.199 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.194+0000 7f2db2057700 1 -- 172.21.15.167:0/2665859004 <== osd.3 v2:172.21.15.167:6808/30036 3 ==== osd_op_reply(19 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f2dac024920 con 0x560a8ac86170 2023-10-05T19:28:35.200 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.194+0000 7f2db2057700 1 -- 172.21.15.167:0/2665859004 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:20 5.e 5:70000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x560a8ac88a90 con 0x560a8ac86170 2023-10-05T19:28:35.200 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.194+0000 7f2db2057700 1 -- 172.21.15.167:0/2665859004 <== osd.3 v2:172.21.15.167:6808/30036 4 ==== osd_op_reply(20 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f2dac024920 con 0x560a8ac86170 2023-10-05T19:28:35.200 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.194+0000 7f2db2057700 1 -- 172.21.15.167:0/2665859004 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:21 5.1e 5:78000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x560a8ac88a90 con 0x7f2d9c00d3d0 2023-10-05T19:28:35.201 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.194+0000 7f2db2858700 1 -- 172.21.15.167:0/2665859004 <== osd.0 v2:172.21.15.118:6800/27690 5 ==== osd_op_reply(21 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f2da4022040 con 0x7f2d9c00d3d0 2023-10-05T19:28:35.201 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.194+0000 7f2db2858700 1 -- 172.21.15.167:0/2665859004 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:22 5.1 5:80000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x560a8ac88a90 con 0x560a8ac86170 2023-10-05T19:28:35.202 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.198+0000 7f2db2057700 1 -- 172.21.15.167:0/2665859004 <== osd.3 v2:172.21.15.167:6808/30036 5 ==== osd_op_reply(22 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f2dac024920 con 0x560a8ac86170 2023-10-05T19:28:35.202 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.198+0000 7f2db2057700 1 -- 172.21.15.167:0/2665859004 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:23 5.11 5:88000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x560a8ac88a90 con 0x560a8ac86170 2023-10-05T19:28:35.203 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.198+0000 7f2db2057700 1 -- 172.21.15.167:0/2665859004 <== osd.3 v2:172.21.15.167:6808/30036 6 ==== osd_op_reply(23 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f2dac024920 con 0x560a8ac86170 2023-10-05T19:28:35.203 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.198+0000 7f2db2057700 1 -- 172.21.15.167:0/2665859004 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:24 5.9 5:90000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x560a8ac88a90 con 0x560a8ac52620 2023-10-05T19:28:35.203 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.198+0000 7f2db2858700 1 -- 172.21.15.167:0/2665859004 <== osd.1 v2:172.21.15.167:6800/30035 10 ==== osd_op_reply(24 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f2da4022040 con 0x560a8ac52620 2023-10-05T19:28:35.204 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.198+0000 7f2db2858700 1 -- 172.21.15.167:0/2665859004 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:25 5.19 5:98000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x560a8ac88a90 con 0x560a8ac52620 2023-10-05T19:28:35.204 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.198+0000 7f2db2858700 1 -- 172.21.15.167:0/2665859004 <== osd.1 v2:172.21.15.167:6800/30035 11 ==== osd_op_reply(25 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f2da4022040 con 0x560a8ac52620 2023-10-05T19:28:35.204 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.198+0000 7f2db2858700 1 -- 172.21.15.167:0/2665859004 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:26 5.5 5:a0000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x560a8ac88a90 con 0x7f2d9c00d3d0 2023-10-05T19:28:35.205 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.198+0000 7f2db2858700 1 -- 172.21.15.167:0/2665859004 <== osd.0 v2:172.21.15.118:6800/27690 6 ==== osd_op_reply(26 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f2da4022040 con 0x7f2d9c00d3d0 2023-10-05T19:28:35.206 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.198+0000 7f2db2858700 1 -- 172.21.15.167:0/2665859004 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:27 5.15 5:a8000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x560a8ac88a90 con 0x560a8ac86170 2023-10-05T19:28:35.206 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.202+0000 7f2db2057700 1 -- 172.21.15.167:0/2665859004 <== osd.3 v2:172.21.15.167:6808/30036 7 ==== osd_op_reply(27 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f2dac024920 con 0x560a8ac86170 2023-10-05T19:28:35.206 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.202+0000 7f2db2057700 1 -- 172.21.15.167:0/2665859004 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:28 5.d 5:b0000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x560a8ac88a90 con 0x7f2dac037c00 2023-10-05T19:28:35.207 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.202+0000 7f2db1856700 1 -- 172.21.15.167:0/2665859004 <== osd.2 v2:172.21.15.167:6816/30037 4 ==== osd_op_reply(28 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f2d9c006070 con 0x7f2dac037c00 2023-10-05T19:28:35.207 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.202+0000 7f2db1856700 1 -- 172.21.15.167:0/2665859004 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:29 5.1d 5:b8000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x560a8ac88a90 con 0x560a8ac52620 2023-10-05T19:28:35.207 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.202+0000 7f2db2858700 1 -- 172.21.15.167:0/2665859004 <== osd.1 v2:172.21.15.167:6800/30035 12 ==== osd_op_reply(29 [pgnls start_epoch 30 out=107b] v30'2 uv2 ondisk = 1) v8 ==== 144+0+107 (crc 0 0 0) 0x7f2da4022040 con 0x560a8ac52620 2023-10-05T19:28:35.208 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.202+0000 7f2db2858700 1 -- 172.21.15.167:0/2665859004 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:30 5.3 5:c0000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x560a8ac88a90 con 0x7f2d9c00d3d0 2023-10-05T19:28:35.208 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.202+0000 7f2db2858700 1 -- 172.21.15.167:0/2665859004 <== osd.0 v2:172.21.15.118:6800/27690 7 ==== osd_op_reply(30 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f2da4022040 con 0x7f2d9c00d3d0 2023-10-05T19:28:35.209 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.202+0000 7f2db2858700 1 -- 172.21.15.167:0/2665859004 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:31 5.13 5:c8000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x560a8ac88a90 con 0x560a8ac86170 2023-10-05T19:28:35.209 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.202+0000 7f2db2057700 1 -- 172.21.15.167:0/2665859004 <== osd.3 v2:172.21.15.167:6808/30036 8 ==== osd_op_reply(31 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f2dac024920 con 0x560a8ac86170 2023-10-05T19:28:35.209 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.202+0000 7f2db2057700 1 -- 172.21.15.167:0/2665859004 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:32 5.b 5:d0000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x560a8ac88a90 con 0x7f2d9c00d3d0 2023-10-05T19:28:35.210 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.206+0000 7f2db2858700 1 -- 172.21.15.167:0/2665859004 <== osd.0 v2:172.21.15.118:6800/27690 8 ==== osd_op_reply(32 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f2da4022040 con 0x7f2d9c00d3d0 2023-10-05T19:28:35.210 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.206+0000 7f2db2858700 1 -- 172.21.15.167:0/2665859004 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:33 5.1b 5:d8000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x560a8ac88a90 con 0x7f2dac037c00 2023-10-05T19:28:35.211 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.206+0000 7f2db1856700 1 -- 172.21.15.167:0/2665859004 <== osd.2 v2:172.21.15.167:6816/30037 5 ==== osd_op_reply(33 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f2d9c006070 con 0x7f2dac037c00 2023-10-05T19:28:35.211 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.206+0000 7f2db1856700 1 -- 172.21.15.167:0/2665859004 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:34 5.7 5:e0000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x560a8ac88a90 con 0x560a8ac86170 2023-10-05T19:28:35.212 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.206+0000 7f2db2057700 1 -- 172.21.15.167:0/2665859004 <== osd.3 v2:172.21.15.167:6808/30036 9 ==== osd_op_reply(34 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f2dac024920 con 0x560a8ac86170 2023-10-05T19:28:35.212 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.206+0000 7f2db2057700 1 -- 172.21.15.167:0/2665859004 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:35 5.17 5:e8000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x560a8ac88a90 con 0x7f2d9c00d3d0 2023-10-05T19:28:35.213 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.206+0000 7f2db2858700 1 -- 172.21.15.167:0/2665859004 <== osd.0 v2:172.21.15.118:6800/27690 9 ==== osd_op_reply(35 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f2da4022040 con 0x7f2d9c00d3d0 2023-10-05T19:28:35.213 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.206+0000 7f2db2858700 1 -- 172.21.15.167:0/2665859004 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:36 5.f 5:f0000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x560a8ac88a90 con 0x560a8ac52620 2023-10-05T19:28:35.214 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.210+0000 7f2db2858700 1 -- 172.21.15.167:0/2665859004 <== osd.1 v2:172.21.15.167:6800/30035 13 ==== osd_op_reply(36 [pgnls start_epoch 30 out=115b] v23'2 uv2 ondisk = 1) v8 ==== 144+0+115 (crc 0 0 0) 0x7f2da402f080 con 0x560a8ac52620 2023-10-05T19:28:35.214 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.210+0000 7f2db2858700 1 -- 172.21.15.167:0/2665859004 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:37 5.1f 5:f8000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x560a8ac88a90 con 0x7f2dac037c00 2023-10-05T19:28:35.215 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.210+0000 7f2db1856700 1 -- 172.21.15.167:0/2665859004 <== osd.2 v2:172.21.15.167:6816/30037 6 ==== osd_op_reply(37 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f2d9c006070 con 0x7f2dac037c00 2023-10-05T19:28:35.215 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.210+0000 7f2db4522e40 20 RGWRados::pool_iterate: got zonegroup_info.4f1f719d-9648-417b-b10c-e5d939825419 2023-10-05T19:28:35.215 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.210+0000 7f2db4522e40 20 RGWRados::pool_iterate: got zone_info.9a9df965-6612-45d4-a45f-6e3c889bacc2 2023-10-05T19:28:35.215 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.210+0000 7f2db4522e40 20 RGWRados::pool_iterate: got zone_names.default 2023-10-05T19:28:35.215 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.210+0000 7f2db4522e40 20 RGWRados::pool_iterate: got zonegroups_names.default 2023-10-05T19:28:35.215 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.210+0000 7f2db4522e40 20 rados->read ofs=0 len=0 2023-10-05T19:28:35.215 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.210+0000 7f2db4522e40 1 -- 172.21.15.167:0/2665859004 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:38 5.12 5:49953fa1:::default.realm:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x560a8ac84f20 con 0x560a8ac52620 2023-10-05T19:28:35.216 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.210+0000 7f2db2858700 1 -- 172.21.15.167:0/2665859004 <== osd.1 v2:172.21.15.167:6800/30035 14 ==== osd_op_reply(38 default.realm [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 ==== 157+0+0 (crc 0 0 0) 0x7f2da402f080 con 0x560a8ac52620 2023-10-05T19:28:35.217 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.210+0000 7f2db4522e40 20 rados_obj.operate() r=-2 bl.length=0 2023-10-05T19:28:35.217 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.210+0000 7f2db4522e40 20 rados->read ofs=0 len=0 2023-10-05T19:28:35.217 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.210+0000 7f2db4522e40 1 -- 172.21.15.167:0/2665859004 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:39 5.f 5:f4c53578:::zonegroups_names.default:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x560a8ac85810 con 0x560a8ac52620 2023-10-05T19:28:35.217 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.210+0000 7f2db2858700 1 -- 172.21.15.167:0/2665859004 <== osd.1 v2:172.21.15.167:6800/30035 15 ==== osd_op_reply(39 zonegroups_names.default [read 0~46 out=46b] v0'0 uv2 ondisk = 0) v8 ==== 168+0+46 (crc 0 0 0) 0x7f2da402f080 con 0x560a8ac52620 2023-10-05T19:28:35.217 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.210+0000 7f2db4522e40 20 rados_obj.operate() r=0 bl.length=46 2023-10-05T19:28:35.217 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.210+0000 7f2db4522e40 20 rados->read ofs=0 len=0 2023-10-05T19:28:35.217 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.210+0000 7f2db4522e40 1 -- 172.21.15.167:0/2665859004 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:40 5.c 5:30d299a5:::zonegroup_info.4f1f719d-9648-417b-b10c-e5d939825419:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x560a8ac88ed0 con 0x7f2d9c00d3d0 2023-10-05T19:28:35.218 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.210+0000 7f2db2858700 1 -- 172.21.15.167:0/2665859004 <== osd.0 v2:172.21.15.118:6800/27690 10 ==== osd_op_reply(40 zonegroup_info.4f1f719d-9648-417b-b10c-e5d939825419 [read 0~362 out=362b] v0'0 uv1 ondisk = 0) v8 ==== 195+0+362 (crc 0 0 0) 0x7f2da402f080 con 0x7f2d9c00d3d0 2023-10-05T19:28:35.218 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.210+0000 7f2db4522e40 20 rados_obj.operate() r=0 bl.length=362 2023-10-05T19:28:35.219 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.210+0000 7f2db4522e40 20 rados->read ofs=0 len=0 2023-10-05T19:28:35.219 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.210+0000 7f2db4522e40 1 -- 172.21.15.167:0/2665859004 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:41 5.13 5:cbbf6c60:::converted:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x560a8ac88ed0 con 0x560a8ac86170 2023-10-05T19:28:35.219 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.210+0000 7f2db2057700 1 -- 172.21.15.167:0/2665859004 <== osd.3 v2:172.21.15.167:6808/30036 10 ==== osd_op_reply(41 converted [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 ==== 153+0+0 (crc 0 0 0) 0x7f2dac024920 con 0x560a8ac86170 2023-10-05T19:28:35.219 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.210+0000 7f2db4522e40 20 rados_obj.operate() r=-2 bl.length=0 2023-10-05T19:28:35.219 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.210+0000 7f2db4522e40 20 rados->read ofs=0 len=0 2023-10-05T19:28:35.219 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.210+0000 7f2db4522e40 1 -- 172.21.15.167:0/2665859004 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:42 5.12 5:49953fa1:::default.realm:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x560a8ac89280 con 0x560a8ac52620 2023-10-05T19:28:35.219 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.214+0000 7f2db2858700 1 -- 172.21.15.167:0/2665859004 <== osd.1 v2:172.21.15.167:6800/30035 16 ==== osd_op_reply(42 default.realm [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 ==== 157+0+0 (crc 0 0 0) 0x7f2da402f080 con 0x560a8ac52620 2023-10-05T19:28:35.219 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.214+0000 7f2db4522e40 20 rados_obj.operate() r=-2 bl.length=0 2023-10-05T19:28:35.219 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.214+0000 7f2db4522e40 20 rados->read ofs=0 len=0 2023-10-05T19:28:35.219 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.214+0000 7f2db4522e40 1 -- 172.21.15.167:0/2665859004 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:43 5.f 5:f4c53578:::zonegroups_names.default:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x560a8ac85440 con 0x560a8ac52620 2023-10-05T19:28:35.220 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.214+0000 7f2db2858700 1 -- 172.21.15.167:0/2665859004 <== osd.1 v2:172.21.15.167:6800/30035 17 ==== osd_op_reply(43 zonegroups_names.default [read 0~46 out=46b] v0'0 uv2 ondisk = 0) v8 ==== 168+0+46 (crc 0 0 0) 0x7f2da402f080 con 0x560a8ac52620 2023-10-05T19:28:35.220 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.214+0000 7f2db4522e40 20 rados_obj.operate() r=0 bl.length=46 2023-10-05T19:28:35.220 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.214+0000 7f2db4522e40 1 -- 172.21.15.167:0/2665859004 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:44 5.0 5:00000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x560a8ac85410 con 0x560a8ac86170 2023-10-05T19:28:35.222 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.218+0000 7f2db2057700 1 -- 172.21.15.167:0/2665859004 <== osd.3 v2:172.21.15.167:6808/30036 11 ==== osd_op_reply(44 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f2dac024920 con 0x560a8ac86170 2023-10-05T19:28:35.222 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.218+0000 7f2db2057700 1 -- 172.21.15.167:0/2665859004 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:45 5.10 5:08000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x560a8ac85410 con 0x7f2dac037c00 2023-10-05T19:28:35.222 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.218+0000 7f2db1856700 1 -- 172.21.15.167:0/2665859004 <== osd.2 v2:172.21.15.167:6816/30037 7 ==== osd_op_reply(45 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f2d9c006070 con 0x7f2dac037c00 2023-10-05T19:28:35.222 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.218+0000 7f2db1856700 1 -- 172.21.15.167:0/2665859004 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:46 5.8 5:10000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x560a8ac85410 con 0x7f2d9c00d3d0 2023-10-05T19:28:35.223 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.218+0000 7f2db2858700 1 -- 172.21.15.167:0/2665859004 <== osd.0 v2:172.21.15.118:6800/27690 11 ==== osd_op_reply(46 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f2da402f080 con 0x7f2d9c00d3d0 2023-10-05T19:28:35.224 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.218+0000 7f2db2858700 1 -- 172.21.15.167:0/2665859004 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:47 5.18 5:18000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x560a8ac85810 con 0x560a8ac52620 2023-10-05T19:28:35.224 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.218+0000 7f2db2858700 1 -- 172.21.15.167:0/2665859004 <== osd.1 v2:172.21.15.167:6800/30035 18 ==== osd_op_reply(47 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f2da402f080 con 0x560a8ac52620 2023-10-05T19:28:35.224 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.218+0000 7f2db2858700 1 -- 172.21.15.167:0/2665859004 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:48 5.4 5:20000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x560a8ac85810 con 0x560a8ac52620 2023-10-05T19:28:35.225 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.218+0000 7f2db2858700 1 -- 172.21.15.167:0/2665859004 <== osd.1 v2:172.21.15.167:6800/30035 19 ==== osd_op_reply(48 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f2da402f080 con 0x560a8ac52620 2023-10-05T19:28:35.225 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.218+0000 7f2db2858700 1 -- 172.21.15.167:0/2665859004 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:49 5.14 5:28000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x560a8ac85810 con 0x7f2d9c00d3d0 2023-10-05T19:28:35.226 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.222+0000 7f2db2858700 1 -- 172.21.15.167:0/2665859004 <== osd.0 v2:172.21.15.118:6800/27690 12 ==== osd_op_reply(49 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f2da402f080 con 0x7f2d9c00d3d0 2023-10-05T19:28:35.226 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.222+0000 7f2db2858700 1 -- 172.21.15.167:0/2665859004 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:50 5.c 5:30000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x560a8ac85810 con 0x7f2d9c00d3d0 2023-10-05T19:28:35.227 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.222+0000 7f2db2858700 1 -- 172.21.15.167:0/2665859004 <== osd.0 v2:172.21.15.118:6800/27690 13 ==== osd_op_reply(50 [pgnls start_epoch 30 out=112b] v23'1 uv1 ondisk = 1) v8 ==== 144+0+112 (crc 0 0 0) 0x7f2da402f080 con 0x7f2d9c00d3d0 2023-10-05T19:28:35.227 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.222+0000 7f2db2858700 1 -- 172.21.15.167:0/2665859004 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:51 5.1c 5:38000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x560a8ac85810 con 0x7f2dac037c00 2023-10-05T19:28:35.227 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.222+0000 7f2db1856700 1 -- 172.21.15.167:0/2665859004 <== osd.2 v2:172.21.15.167:6816/30037 8 ==== osd_op_reply(51 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f2d9c006070 con 0x7f2dac037c00 2023-10-05T19:28:35.228 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.222+0000 7f2db1856700 1 -- 172.21.15.167:0/2665859004 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:52 5.2 5:40000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x560a8ac85810 con 0x560a8ac86170 2023-10-05T19:28:35.228 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.222+0000 7f2db2057700 1 -- 172.21.15.167:0/2665859004 <== osd.3 v2:172.21.15.167:6808/30036 12 ==== osd_op_reply(52 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f2dac024920 con 0x560a8ac86170 2023-10-05T19:28:35.228 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.222+0000 7f2db2057700 1 -- 172.21.15.167:0/2665859004 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:53 5.12 5:48000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x560a8ac85810 con 0x560a8ac52620 2023-10-05T19:28:35.229 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.222+0000 7f2db2858700 1 -- 172.21.15.167:0/2665859004 <== osd.1 v2:172.21.15.167:6800/30035 20 ==== osd_op_reply(53 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f2da402f080 con 0x560a8ac52620 2023-10-05T19:28:35.229 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.222+0000 7f2db2858700 1 -- 172.21.15.167:0/2665859004 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:54 5.a 5:50000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x560a8ac85810 con 0x7f2d9c00d3d0 2023-10-05T19:28:35.230 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.226+0000 7f2db2858700 1 -- 172.21.15.167:0/2665859004 <== osd.0 v2:172.21.15.118:6800/27690 14 ==== osd_op_reply(54 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f2da402f080 con 0x7f2d9c00d3d0 2023-10-05T19:28:35.230 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.226+0000 7f2db2858700 1 -- 172.21.15.167:0/2665859004 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:55 5.1a 5:58000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x560a8ac85810 con 0x560a8ac52620 2023-10-05T19:28:35.231 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.226+0000 7f2db2858700 1 -- 172.21.15.167:0/2665859004 <== osd.1 v2:172.21.15.167:6800/30035 21 ==== osd_op_reply(55 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f2da402f080 con 0x560a8ac52620 2023-10-05T19:28:35.231 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.226+0000 7f2db2858700 1 -- 172.21.15.167:0/2665859004 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:56 5.6 5:60000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x560a8ac85810 con 0x7f2dac037c00 2023-10-05T19:28:35.232 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.226+0000 7f2db1856700 1 -- 172.21.15.167:0/2665859004 <== osd.2 v2:172.21.15.167:6816/30037 9 ==== osd_op_reply(56 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f2d9c006070 con 0x7f2dac037c00 2023-10-05T19:28:35.232 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.226+0000 7f2db1856700 1 -- 172.21.15.167:0/2665859004 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:57 5.16 5:68000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x560a8ac85810 con 0x560a8ac86170 2023-10-05T19:28:35.232 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.226+0000 7f2db2057700 1 -- 172.21.15.167:0/2665859004 <== osd.3 v2:172.21.15.167:6808/30036 13 ==== osd_op_reply(57 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f2dac024920 con 0x560a8ac86170 2023-10-05T19:28:35.233 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.226+0000 7f2db2057700 1 -- 172.21.15.167:0/2665859004 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:58 5.e 5:70000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x560a8ac85810 con 0x560a8ac86170 2023-10-05T19:28:35.233 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.226+0000 7f2db2057700 1 -- 172.21.15.167:0/2665859004 <== osd.3 v2:172.21.15.167:6808/30036 14 ==== osd_op_reply(58 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f2dac024920 con 0x560a8ac86170 2023-10-05T19:28:35.233 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.226+0000 7f2db2057700 1 -- 172.21.15.167:0/2665859004 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:59 5.1e 5:78000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x560a8ac85810 con 0x7f2d9c00d3d0 2023-10-05T19:28:35.235 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.230+0000 7f2db2858700 1 -- 172.21.15.167:0/2665859004 <== osd.0 v2:172.21.15.118:6800/27690 15 ==== osd_op_reply(59 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f2da402f080 con 0x7f2d9c00d3d0 2023-10-05T19:28:35.235 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.230+0000 7f2db2858700 1 -- 172.21.15.167:0/2665859004 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:60 5.1 5:80000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x560a8ac85810 con 0x560a8ac86170 2023-10-05T19:28:35.235 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.230+0000 7f2db2057700 1 -- 172.21.15.167:0/2665859004 <== osd.3 v2:172.21.15.167:6808/30036 15 ==== osd_op_reply(60 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f2dac024920 con 0x560a8ac86170 2023-10-05T19:28:35.236 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.230+0000 7f2db2057700 1 -- 172.21.15.167:0/2665859004 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:61 5.11 5:88000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x560a8ac85810 con 0x560a8ac86170 2023-10-05T19:28:35.236 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.230+0000 7f2db2057700 1 -- 172.21.15.167:0/2665859004 <== osd.3 v2:172.21.15.167:6808/30036 16 ==== osd_op_reply(61 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f2dac024920 con 0x560a8ac86170 2023-10-05T19:28:35.236 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.230+0000 7f2db2057700 1 -- 172.21.15.167:0/2665859004 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:62 5.9 5:90000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x560a8ac85810 con 0x560a8ac52620 2023-10-05T19:28:35.237 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.230+0000 7f2db2858700 1 -- 172.21.15.167:0/2665859004 <== osd.1 v2:172.21.15.167:6800/30035 22 ==== osd_op_reply(62 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f2da402f080 con 0x560a8ac52620 2023-10-05T19:28:35.237 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.230+0000 7f2db2858700 1 -- 172.21.15.167:0/2665859004 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:63 5.19 5:98000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x560a8ac85810 con 0x560a8ac52620 2023-10-05T19:28:35.238 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.230+0000 7f2db2858700 1 -- 172.21.15.167:0/2665859004 <== osd.1 v2:172.21.15.167:6800/30035 23 ==== osd_op_reply(63 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f2da402f080 con 0x560a8ac52620 2023-10-05T19:28:35.238 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.230+0000 7f2db2858700 1 -- 172.21.15.167:0/2665859004 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:64 5.5 5:a0000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x560a8ac85810 con 0x7f2d9c00d3d0 2023-10-05T19:28:35.238 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.234+0000 7f2db2858700 1 -- 172.21.15.167:0/2665859004 <== osd.0 v2:172.21.15.118:6800/27690 16 ==== osd_op_reply(64 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f2da402f080 con 0x7f2d9c00d3d0 2023-10-05T19:28:35.239 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.234+0000 7f2db2858700 1 -- 172.21.15.167:0/2665859004 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:65 5.15 5:a8000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x560a8ac85810 con 0x560a8ac86170 2023-10-05T19:28:35.240 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.234+0000 7f2db2057700 1 -- 172.21.15.167:0/2665859004 <== osd.3 v2:172.21.15.167:6808/30036 17 ==== osd_op_reply(65 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f2dac024920 con 0x560a8ac86170 2023-10-05T19:28:35.240 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.234+0000 7f2db2057700 1 -- 172.21.15.167:0/2665859004 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:66 5.d 5:b0000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x560a8ac85810 con 0x7f2dac037c00 2023-10-05T19:28:35.242 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.234+0000 7f2db1856700 1 -- 172.21.15.167:0/2665859004 <== osd.2 v2:172.21.15.167:6816/30037 10 ==== osd_op_reply(66 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f2d9c006070 con 0x7f2dac037c00 2023-10-05T19:28:35.242 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.234+0000 7f2db1856700 1 -- 172.21.15.167:0/2665859004 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:67 5.1d 5:b8000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x560a8ac85810 con 0x560a8ac52620 2023-10-05T19:28:35.243 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.234+0000 7f2db2858700 1 -- 172.21.15.167:0/2665859004 <== osd.1 v2:172.21.15.167:6800/30035 24 ==== osd_op_reply(67 [pgnls start_epoch 30 out=107b] v30'2 uv2 ondisk = 1) v8 ==== 144+0+107 (crc 0 0 0) 0x7f2da402f080 con 0x560a8ac52620 2023-10-05T19:28:35.243 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.234+0000 7f2db2858700 1 -- 172.21.15.167:0/2665859004 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:68 5.3 5:c0000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x560a8ac85810 con 0x7f2d9c00d3d0 2023-10-05T19:28:35.244 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.238+0000 7f2db2858700 1 -- 172.21.15.167:0/2665859004 <== osd.0 v2:172.21.15.118:6800/27690 17 ==== osd_op_reply(68 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f2da402f080 con 0x7f2d9c00d3d0 2023-10-05T19:28:35.244 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.238+0000 7f2db2858700 1 -- 172.21.15.167:0/2665859004 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:69 5.13 5:c8000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x560a8ac85810 con 0x560a8ac86170 2023-10-05T19:28:35.244 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.238+0000 7f2db2057700 1 -- 172.21.15.167:0/2665859004 <== osd.3 v2:172.21.15.167:6808/30036 18 ==== osd_op_reply(69 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f2dac024920 con 0x560a8ac86170 2023-10-05T19:28:35.244 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.238+0000 7f2db2057700 1 -- 172.21.15.167:0/2665859004 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:70 5.b 5:d0000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x560a8ac85810 con 0x7f2d9c00d3d0 2023-10-05T19:28:35.244 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.238+0000 7f2db2858700 1 -- 172.21.15.167:0/2665859004 <== osd.0 v2:172.21.15.118:6800/27690 18 ==== osd_op_reply(70 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f2da402f080 con 0x7f2d9c00d3d0 2023-10-05T19:28:35.244 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.238+0000 7f2db2858700 1 -- 172.21.15.167:0/2665859004 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:71 5.1b 5:d8000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x560a8ac85810 con 0x7f2dac037c00 2023-10-05T19:28:35.245 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.238+0000 7f2db1856700 1 -- 172.21.15.167:0/2665859004 <== osd.2 v2:172.21.15.167:6816/30037 11 ==== osd_op_reply(71 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f2d9c006070 con 0x7f2dac037c00 2023-10-05T19:28:35.245 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.238+0000 7f2db1856700 1 -- 172.21.15.167:0/2665859004 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:72 5.7 5:e0000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x560a8ac85810 con 0x560a8ac86170 2023-10-05T19:28:35.246 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.242+0000 7f2db2057700 1 -- 172.21.15.167:0/2665859004 <== osd.3 v2:172.21.15.167:6808/30036 19 ==== osd_op_reply(72 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f2dac024920 con 0x560a8ac86170 2023-10-05T19:28:35.246 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.242+0000 7f2db2057700 1 -- 172.21.15.167:0/2665859004 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:73 5.17 5:e8000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x560a8ac85810 con 0x7f2d9c00d3d0 2023-10-05T19:28:35.247 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.242+0000 7f2db2858700 1 -- 172.21.15.167:0/2665859004 <== osd.0 v2:172.21.15.118:6800/27690 19 ==== osd_op_reply(73 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f2da402f080 con 0x7f2d9c00d3d0 2023-10-05T19:28:35.247 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.242+0000 7f2db2858700 1 -- 172.21.15.167:0/2665859004 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:74 5.f 5:f0000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x560a8ac85810 con 0x560a8ac52620 2023-10-05T19:28:35.248 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.242+0000 7f2db2858700 1 -- 172.21.15.167:0/2665859004 <== osd.1 v2:172.21.15.167:6800/30035 25 ==== osd_op_reply(74 [pgnls start_epoch 30 out=115b] v23'2 uv2 ondisk = 1) v8 ==== 144+0+115 (crc 0 0 0) 0x7f2da402f080 con 0x560a8ac52620 2023-10-05T19:28:35.248 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.242+0000 7f2db2858700 1 -- 172.21.15.167:0/2665859004 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:75 5.1f 5:f8000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x560a8ac85810 con 0x7f2dac037c00 2023-10-05T19:28:35.249 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.242+0000 7f2db1856700 1 -- 172.21.15.167:0/2665859004 <== osd.2 v2:172.21.15.167:6816/30037 12 ==== osd_op_reply(75 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f2d9c006070 con 0x7f2dac037c00 2023-10-05T19:28:35.249 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.242+0000 7f2db4522e40 20 RGWRados::pool_iterate: got zonegroup_info.4f1f719d-9648-417b-b10c-e5d939825419 2023-10-05T19:28:35.249 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.242+0000 7f2db4522e40 20 RGWRados::pool_iterate: got zone_info.9a9df965-6612-45d4-a45f-6e3c889bacc2 2023-10-05T19:28:35.249 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.242+0000 7f2db4522e40 20 RGWRados::pool_iterate: got zone_names.default 2023-10-05T19:28:35.249 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.242+0000 7f2db4522e40 20 RGWRados::pool_iterate: got zonegroups_names.default 2023-10-05T19:28:35.250 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.242+0000 7f2db4522e40 20 rados->read ofs=0 len=0 2023-10-05T19:28:35.250 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.242+0000 7f2db4522e40 1 -- 172.21.15.167:0/2665859004 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:76 5.f 5:f43b6ece:::zone_names.default:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x560a8ac85410 con 0x560a8ac52620 2023-10-05T19:28:35.251 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.242+0000 7f2db2858700 1 -- 172.21.15.167:0/2665859004 <== osd.1 v2:172.21.15.167:6800/30035 26 ==== osd_op_reply(76 zone_names.default [read 0~46 out=46b] v0'0 uv1 ondisk = 0) v8 ==== 162+0+46 (crc 0 0 0) 0x7f2da402f080 con 0x560a8ac52620 2023-10-05T19:28:35.251 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.242+0000 7f2db4522e40 20 rados_obj.operate() r=0 bl.length=46 2023-10-05T19:28:35.251 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.242+0000 7f2db4522e40 20 rados->read ofs=0 len=0 2023-10-05T19:28:35.251 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.242+0000 7f2db4522e40 1 -- 172.21.15.167:0/2665859004 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:77 5.1d 5:b8374eb4:::zone_info.9a9df965-6612-45d4-a45f-6e3c889bacc2:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x560a8ac897f0 con 0x560a8ac52620 2023-10-05T19:28:35.251 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.246+0000 7f2db2858700 1 -- 172.21.15.167:0/2665859004 <== osd.1 v2:172.21.15.167:6800/30035 27 ==== osd_op_reply(77 zone_info.9a9df965-6612-45d4-a45f-6e3c889bacc2 [read 0~874 out=874b] v0'0 uv2 ondisk = 0) v8 ==== 190+0+874 (crc 0 0 0) 0x7f2da402f080 con 0x560a8ac52620 2023-10-05T19:28:35.251 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.246+0000 7f2db4522e40 20 rados_obj.operate() r=0 bl.length=874 2023-10-05T19:28:35.251 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.246+0000 7f2db4522e40 20 rados->read ofs=0 len=0 2023-10-05T19:28:35.251 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.246+0000 7f2db4522e40 1 -- 172.21.15.167:0/2665859004 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:78 5.f 5:f4c53578:::zonegroups_names.default:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x560a8ac89ce0 con 0x560a8ac52620 2023-10-05T19:28:35.252 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.246+0000 7f2db2858700 1 -- 172.21.15.167:0/2665859004 <== osd.1 v2:172.21.15.167:6800/30035 28 ==== osd_op_reply(78 zonegroups_names.default [read 0~46 out=46b] v0'0 uv2 ondisk = 0) v8 ==== 168+0+46 (crc 0 0 0) 0x7f2da402f080 con 0x560a8ac52620 2023-10-05T19:28:35.252 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.246+0000 7f2db4522e40 20 rados_obj.operate() r=0 bl.length=46 2023-10-05T19:28:35.252 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.246+0000 7f2db4522e40 20 rados->read ofs=0 len=0 2023-10-05T19:28:35.252 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.246+0000 7f2db4522e40 1 -- 172.21.15.167:0/2665859004 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:79 5.c 5:30d299a5:::zonegroup_info.4f1f719d-9648-417b-b10c-e5d939825419:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x560a8ac89ce0 con 0x7f2d9c00d3d0 2023-10-05T19:28:35.253 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.246+0000 7f2db2858700 1 -- 172.21.15.167:0/2665859004 <== osd.0 v2:172.21.15.118:6800/27690 20 ==== osd_op_reply(79 zonegroup_info.4f1f719d-9648-417b-b10c-e5d939825419 [read 0~362 out=362b] v0'0 uv1 ondisk = 0) v8 ==== 195+0+362 (crc 0 0 0) 0x7f2da402f080 con 0x7f2d9c00d3d0 2023-10-05T19:28:35.253 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.246+0000 7f2db4522e40 20 rados_obj.operate() r=0 bl.length=362 2023-10-05T19:28:35.253 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.246+0000 7f2db4522e40 20 rados->read ofs=0 len=0 2023-10-05T19:28:35.253 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.246+0000 7f2db4522e40 1 -- 172.21.15.167:0/2665859004 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:80 5.3 5:c0ad8cbd:::region_map:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x560a8ac89ce0 con 0x7f2d9c00d3d0 2023-10-05T19:28:35.254 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.246+0000 7f2db2858700 1 -- 172.21.15.167:0/2665859004 <== osd.0 v2:172.21.15.118:6800/27690 21 ==== osd_op_reply(80 region_map [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 ==== 154+0+0 (crc 0 0 0) 0x7f2da402f080 con 0x7f2d9c00d3d0 2023-10-05T19:28:35.254 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.246+0000 7f2db4522e40 20 rados_obj.operate() r=-2 bl.length=0 2023-10-05T19:28:35.254 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.246+0000 7f2db4522e40 20 zone default found 2023-10-05T19:28:35.254 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.246+0000 7f2db4522e40 4 Realm: () 2023-10-05T19:28:35.254 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.246+0000 7f2db4522e40 4 ZoneGroup: default (4f1f719d-9648-417b-b10c-e5d939825419) 2023-10-05T19:28:35.254 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.246+0000 7f2db4522e40 4 Zone: default (9a9df965-6612-45d4-a45f-6e3c889bacc2) 2023-10-05T19:28:35.254 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.246+0000 7f2db4522e40 10 cannot find current period zonegroup using local zonegroup configuration 2023-10-05T19:28:35.254 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.246+0000 7f2db4522e40 20 zonegroup default 2023-10-05T19:28:35.254 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.246+0000 7f2db4522e40 20 rados->read ofs=0 len=0 2023-10-05T19:28:35.254 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.246+0000 7f2db4522e40 1 -- 172.21.15.167:0/2665859004 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:81 5.3 5:c52100b6:::period_config.default:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x560a8ac89ce0 con 0x7f2d9c00d3d0 2023-10-05T19:28:35.255 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.250+0000 7f2db2858700 1 -- 172.21.15.167:0/2665859004 <== osd.0 v2:172.21.15.118:6800/27690 22 ==== osd_op_reply(81 period_config.default [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 ==== 165+0+0 (crc 0 0 0) 0x7f2da4050040 con 0x7f2d9c00d3d0 2023-10-05T19:28:35.255 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.250+0000 7f2db4522e40 20 rados_obj.operate() r=-2 bl.length=0 2023-10-05T19:28:35.255 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.250+0000 7f2db4522e40 20 rados->read ofs=0 len=0 2023-10-05T19:28:35.256 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.250+0000 7f2db4522e40 1 -- 172.21.15.167:0/2665859004 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:82 6.2 6:4347d321:::bucket.sync-source-hints.:head [call version.read in=11b,read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x560a8ac8b660 con 0x560a8ac86170 2023-10-05T19:28:35.256 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.250+0000 7f2db2057700 1 -- 172.21.15.167:0/2665859004 <== osd.3 v2:172.21.15.167:6808/30036 20 ==== osd_op_reply(82 bucket.sync-source-hints. [call,read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 ==== 211+0+0 (crc 0 0 0) 0x7f2dac024920 con 0x560a8ac86170 2023-10-05T19:28:35.256 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.250+0000 7f2db4522e40 20 rados_obj.operate() r=-2 bl.length=0 2023-10-05T19:28:35.256 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.250+0000 7f2db4522e40 20 rados->read ofs=0 len=0 2023-10-05T19:28:35.256 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.250+0000 7f2db4522e40 1 -- 172.21.15.167:0/2665859004 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:83 6.b 6:d467b91b:::bucket.sync-target-hints.:head [call version.read in=11b,read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x560a8ac8ba10 con 0x560a8ac86170 2023-10-05T19:28:35.256 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.250+0000 7f2db2057700 1 -- 172.21.15.167:0/2665859004 <== osd.3 v2:172.21.15.167:6808/30036 21 ==== osd_op_reply(83 bucket.sync-target-hints. [call,read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 ==== 211+0+0 (crc 0 0 0) 0x7f2dac05f040 con 0x560a8ac86170 2023-10-05T19:28:35.257 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.250+0000 7f2db4522e40 20 rados_obj.operate() r=-2 bl.length=0 2023-10-05T19:28:35.257 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.250+0000 7f2db4522e40 20 started sync module instance, tier type = 2023-10-05T19:28:35.257 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.250+0000 7f2db4522e40 20 started zone id=9a9df965-6612-45d4-a45f-6e3c889bacc2 (name=default) with tier type = 2023-10-05T19:28:35.257 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.250+0000 7f2db4522e40 1 -- 172.21.15.167:0/2665859004 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:84 7.1f 7:f95f44c2:::notify.0:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x560a8ac8ad20 con 0x7f2dac037c00 2023-10-05T19:28:35.259 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.254+0000 7f2db1856700 1 -- 172.21.15.167:0/2665859004 <== osd.2 v2:172.21.15.167:6816/30037 13 ==== osd_op_reply(84 notify.0 [create] v30'141 uv141 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f2d9c006070 con 0x7f2dac037c00 2023-10-05T19:28:35.259 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.254+0000 7f2db4522e40 1 -- 172.21.15.167:0/2665859004 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:85 7.1f 7:f95f44c2:::notify.0:head [watch watch cookie 94603278069968] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x560a8ac8a950 con 0x7f2dac037c00 2023-10-05T19:28:35.259 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.254+0000 7f2db4522e40 1 -- 172.21.15.167:0/2665859004 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:86 7.0 7:05bf5b68:::notify.1:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x560a8ac8c7e0 con 0x560a8ac52620 2023-10-05T19:28:35.262 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.258+0000 7f2db2858700 1 -- 172.21.15.167:0/2665859004 <== osd.1 v2:172.21.15.167:6800/30035 29 ==== osd_op_reply(86 notify.1 [create] v30'141 uv141 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f2da4050040 con 0x560a8ac52620 2023-10-05T19:28:35.262 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.258+0000 7f2db4522e40 1 -- 172.21.15.167:0/2665859004 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:87 7.0 7:05bf5b68:::notify.1:head [watch watch cookie 94603278074544] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x560a8ac8d0d0 con 0x560a8ac52620 2023-10-05T19:28:35.262 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.258+0000 7f2db1856700 1 -- 172.21.15.167:0/2665859004 <== osd.2 v2:172.21.15.167:6816/30037 14 ==== osd_op_reply(85 notify.0 [watch watch cookie 94603278069968] v30'142 uv141 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f2d9c006070 con 0x7f2dac037c00 2023-10-05T19:28:35.262 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.258+0000 7f2db4522e40 1 -- 172.21.15.167:0/2665859004 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:88 7.15 7:a93a5511:::notify.2:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x560a8ac8d970 con 0x560a8ac86170 2023-10-05T19:28:35.265 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.258+0000 7f2db2858700 1 -- 172.21.15.167:0/2665859004 <== osd.1 v2:172.21.15.167:6800/30035 30 ==== osd_op_reply(87 notify.1 [watch watch cookie 94603278074544] v30'142 uv141 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f2da4050040 con 0x560a8ac52620 2023-10-05T19:28:35.266 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.258+0000 7f2db2057700 1 -- 172.21.15.167:0/2665859004 <== osd.3 v2:172.21.15.167:6808/30036 22 ==== osd_op_reply(88 notify.2 [create] v30'141 uv141 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f2dac05f040 con 0x560a8ac86170 2023-10-05T19:28:35.266 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.258+0000 7f2db4522e40 1 -- 172.21.15.167:0/2665859004 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:89 7.15 7:a93a5511:::notify.2:head [watch watch cookie 94603278076896] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x560a8ac8bee0 con 0x560a8ac86170 2023-10-05T19:28:35.266 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.258+0000 7f2db4522e40 1 -- 172.21.15.167:0/2665859004 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:90 7.e 7:7759931f:::notify.3:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x560a8ac8e210 con 0x7f2dac037c00 2023-10-05T19:28:35.268 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.262+0000 7f2db1856700 1 -- 172.21.15.167:0/2665859004 <== osd.2 v2:172.21.15.167:6816/30037 15 ==== osd_op_reply(90 notify.3 [create] v30'141 uv141 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f2d9c006070 con 0x7f2dac037c00 2023-10-05T19:28:35.269 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.262+0000 7f2db4522e40 1 -- 172.21.15.167:0/2665859004 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:91 7.e 7:7759931f:::notify.3:head [watch watch cookie 94603278081312] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x560a8ac8eab0 con 0x7f2dac037c00 2023-10-05T19:28:35.269 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.262+0000 7f2db4522e40 1 -- 172.21.15.167:0/2665859004 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:92 7.d 7:b4812045:::notify.4:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x560a8ac8f670 con 0x560a8ac52620 2023-10-05T19:28:35.270 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.262+0000 7f2db2057700 1 -- 172.21.15.167:0/2665859004 <== osd.3 v2:172.21.15.167:6808/30036 23 ==== osd_op_reply(89 notify.2 [watch watch cookie 94603278076896] v30'142 uv141 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f2dac05f040 con 0x560a8ac86170 2023-10-05T19:28:35.272 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.266+0000 7f2db1856700 1 -- 172.21.15.167:0/2665859004 <== osd.2 v2:172.21.15.167:6816/30037 16 ==== osd_op_reply(91 notify.3 [watch watch cookie 94603278081312] v30'142 uv141 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f2d9c006070 con 0x7f2dac037c00 2023-10-05T19:28:35.272 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.266+0000 7f2db2858700 1 -- 172.21.15.167:0/2665859004 <== osd.1 v2:172.21.15.167:6800/30035 31 ==== osd_op_reply(92 notify.4 [create] v30'141 uv141 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f2da4050040 con 0x560a8ac52620 2023-10-05T19:28:35.272 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.266+0000 7f2db4522e40 1 -- 172.21.15.167:0/2665859004 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:93 7.d 7:b4812045:::notify.4:head [watch watch cookie 94603278083520] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x560a8ac8f140 con 0x560a8ac52620 2023-10-05T19:28:35.272 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.266+0000 7f2db4522e40 1 -- 172.21.15.167:0/2665859004 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:94 7.3 7:c609908c:::notify.5:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x560a8ac8ff60 con 0x7f2d9c00d3d0 2023-10-05T19:28:35.274 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.270+0000 7f2db2858700 1 -- 172.21.15.167:0/2665859004 <== osd.1 v2:172.21.15.167:6800/30035 32 ==== osd_op_reply(93 notify.4 [watch watch cookie 94603278083520] v30'142 uv141 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f2da4050040 con 0x560a8ac52620 2023-10-05T19:28:35.275 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.270+0000 7f2db2858700 1 -- 172.21.15.167:0/2665859004 <== osd.0 v2:172.21.15.118:6800/27690 23 ==== osd_op_reply(94 notify.5 [create] v30'141 uv141 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f2da4050040 con 0x7f2d9c00d3d0 2023-10-05T19:28:35.275 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.270+0000 7f2db4522e40 1 -- 172.21.15.167:0/2665859004 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:95 7.3 7:c609908c:::notify.5:head [watch watch cookie 94603278078000] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x560a8ac8ff10 con 0x7f2d9c00d3d0 2023-10-05T19:28:35.275 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.270+0000 7f2db4522e40 1 -- 172.21.15.167:0/2665859004 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:96 7.14 7:2b04a3e9:::notify.6:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x560a8ac907f0 con 0x7f2d9c00d3d0 2023-10-05T19:28:35.278 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.270+0000 7f2db2858700 1 -- 172.21.15.167:0/2665859004 <== osd.0 v2:172.21.15.118:6800/27690 24 ==== osd_op_reply(95 notify.5 [watch watch cookie 94603278078000] v30'142 uv141 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f2da4050040 con 0x7f2d9c00d3d0 2023-10-05T19:28:35.278 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.270+0000 7f2db2858700 1 -- 172.21.15.167:0/2665859004 <== osd.0 v2:172.21.15.118:6800/27690 25 ==== osd_op_reply(96 notify.6 [create] v30'141 uv141 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f2da4050040 con 0x7f2d9c00d3d0 2023-10-05T19:28:35.278 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.270+0000 7f2db4522e40 1 -- 172.21.15.167:0/2665859004 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:97 7.14 7:2b04a3e9:::notify.6:head [watch watch cookie 94603278088736] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x560a8ac903a0 con 0x7f2d9c00d3d0 2023-10-05T19:28:35.278 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.270+0000 7f2db4522e40 1 -- 172.21.15.167:0/2665859004 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:98 7.9 7:93e5b521:::notify.7:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x560a8ac91180 con 0x7f2d9c00d3d0 2023-10-05T19:28:35.281 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.274+0000 7f2db2858700 1 -- 172.21.15.167:0/2665859004 <== osd.0 v2:172.21.15.118:6800/27690 26 ==== osd_op_reply(98 notify.7 [create] v30'141 uv141 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f2da4050040 con 0x7f2d9c00d3d0 2023-10-05T19:28:35.281 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.274+0000 7f2db2858700 1 -- 172.21.15.167:0/2665859004 <== osd.0 v2:172.21.15.118:6800/27690 27 ==== osd_op_reply(97 notify.6 [watch watch cookie 94603278088736] v30'142 uv141 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f2da4050040 con 0x7f2d9c00d3d0 2023-10-05T19:28:35.281 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.274+0000 7f2db4522e40 1 -- 172.21.15.167:0/2665859004 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:99 7.9 7:93e5b521:::notify.7:head [watch watch cookie 94603278093392] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x560a8ac910a0 con 0x7f2d9c00d3d0 2023-10-05T19:28:35.281 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.274+0000 7f2db4522e40 20 add_watcher() i=0 2023-10-05T19:28:35.281 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.274+0000 7f2db4522e40 20 add_watcher() i=1 2023-10-05T19:28:35.281 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.274+0000 7f2db4522e40 20 add_watcher() i=2 2023-10-05T19:28:35.281 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.274+0000 7f2db4522e40 20 add_watcher() i=3 2023-10-05T19:28:35.281 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.274+0000 7f2db4522e40 20 add_watcher() i=4 2023-10-05T19:28:35.281 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.274+0000 7f2db4522e40 20 add_watcher() i=5 2023-10-05T19:28:35.281 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.274+0000 7f2db4522e40 20 add_watcher() i=6 2023-10-05T19:28:35.283 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.278+0000 7f2db2858700 1 -- 172.21.15.167:0/2665859004 <== osd.0 v2:172.21.15.118:6800/27690 28 ==== osd_op_reply(99 notify.7 [watch watch cookie 94603278093392] v30'142 uv141 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f2da4050040 con 0x7f2d9c00d3d0 2023-10-05T19:28:35.283 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.278+0000 7f2db4522e40 20 add_watcher() i=7 2023-10-05T19:28:35.283 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.278+0000 7f2db4522e40 2 all 8 watchers are set, enabling cache 2023-10-05T19:28:35.283 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.278+0000 7f2db4522e40 1 -- 172.21.15.167:0/2665859004 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:100 6.4 6:22d26bf9:::data_loggenerations_metadata:head [call version.check_conds in=74b,call version.read in=11b,read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x560a8ac945b0 con 0x560a8ac52620 2023-10-05T19:28:35.283 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.278+0000 7f2db2858700 1 -- 172.21.15.167:0/2665859004 <== osd.1 v2:172.21.15.167:6800/30035 33 ==== osd_op_reply(100 data_loggenerations_metadata [call,call out=48b,read 0~28 out=28b] v0'0 uv1 ondisk = 0) v8 ==== 256+0+76 (crc 0 0 0) 0x7f2da4050040 con 0x560a8ac52620 2023-10-05T19:28:35.284 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.278+0000 7f2db4522e40 1 -- 172.21.15.167:0/2665859004 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:101 6.4 6:22d26bf9:::data_loggenerations_metadata:head [watch watch cookie 94603278089840] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x560a8ac91a20 con 0x560a8ac52620 2023-10-05T19:28:35.285 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.278+0000 7f2db2858700 1 -- 172.21.15.167:0/2665859004 <== osd.1 v2:172.21.15.167:6800/30035 34 ==== osd_op_reply(101 data_loggenerations_metadata [watch watch cookie 94603278089840] v30'135 uv1 ondisk = 0) v8 ==== 172+0+0 (crc 0 0 0) 0x7f2da4050040 con 0x560a8ac52620 2023-10-05T19:28:35.286 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.282+0000 7f2d83fdf700 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2023-10-05T19:28:35.286 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.282+0000 7f2db4522e40 20 check_secure_mon_conn(): auth registy supported: methods=[2] modes=[2,1] 2023-10-05T19:28:35.286 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.282+0000 7f2db4522e40 20 check_secure_mon_conn(): mode 1 is insecure 2023-10-05T19:28:35.287 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.282+0000 7f2db4522e40 1 Processor -- start 2023-10-05T19:28:35.287 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.282+0000 7f2db4522e40 1 -- start start 2023-10-05T19:28:35.287 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.282+0000 7f2db4522e40 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x560a8ac9ebe0 0x560a8ac9efb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-05T19:28:35.287 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.282+0000 7f2db4522e40 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x560a8ac9f4f0 con 0x560a8ac9ebe0 2023-10-05T19:28:35.288 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.282+0000 7f2db1856700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x560a8ac9ebe0 0x560a8ac9efb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-05T19:28:35.288 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.282+0000 7f2db1856700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x560a8ac9ebe0 0x560a8ac9efb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.118:3300/0 says I am v2:172.21.15.167:41632/0 (socket says 172.21.15.167:41632) 2023-10-05T19:28:35.288 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.282+0000 7f2db1856700 1 -- 172.21.15.167:0/1334723546 learned_addr learned my addr 172.21.15.167:0/1334723546 (peer_addr_for_me v2:172.21.15.167:0/0) 2023-10-05T19:28:35.288 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.282+0000 7f2db1856700 1 -- 172.21.15.167:0/1334723546 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x560a8ac9f630 con 0x560a8ac9ebe0 2023-10-05T19:28:35.288 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.282+0000 7f2db1856700 1 --2- 172.21.15.167:0/1334723546 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x560a8ac9ebe0 0x560a8ac9efb0 secure :-1 s=READY pgs=420 cs=0 l=1 rev1=1 crypto rx=0x560a8ac838d0 tx=0x7f2d9c006000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=91bef5f2591090a4 server_cookie=0 in_seq=0 out_seq=0 2023-10-05T19:28:35.289 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.282+0000 7f2d837de700 1 -- 172.21.15.167:0/1334723546 <== mon.0 v2:172.21.15.118:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f2d9c023040 con 0x560a8ac9ebe0 2023-10-05T19:28:35.289 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.282+0000 7f2d837de700 1 -- 172.21.15.167:0/1334723546 <== mon.0 v2:172.21.15.118:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2d9c024040 con 0x560a8ac9ebe0 2023-10-05T19:28:35.289 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.282+0000 7f2d837de700 1 -- 172.21.15.167:0/1334723546 <== mon.0 v2:172.21.15.118:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f2d9c013040 con 0x560a8ac9ebe0 2023-10-05T19:28:35.290 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.286+0000 7f2db4522e40 1 -- 172.21.15.167:0/1334723546 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x560a8ac9ebe0 msgr2=0x560a8ac9efb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-05T19:28:35.290 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.286+0000 7f2db4522e40 1 --2- 172.21.15.167:0/1334723546 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x560a8ac9ebe0 0x560a8ac9efb0 secure :-1 s=READY pgs=420 cs=0 l=1 rev1=1 crypto rx=0x560a8ac838d0 tx=0x7f2d9c006000 comp rx=0 tx=0).stop 2023-10-05T19:28:35.290 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.286+0000 7f2db4522e40 1 -- 172.21.15.167:0/1334723546 shutdown_connections 2023-10-05T19:28:35.290 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.286+0000 7f2db4522e40 1 --2- 172.21.15.167:0/1334723546 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x560a8ac9ebe0 0x560a8ac9efb0 unknown :-1 s=CLOSED pgs=420 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-05T19:28:35.291 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.286+0000 7f2db4522e40 1 -- 172.21.15.167:0/1334723546 >> 172.21.15.167:0/1334723546 conn(0x560a8ac99fa0 msgr2=0x560a8ac980f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-05T19:28:35.291 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.286+0000 7f2db4522e40 1 -- 172.21.15.167:0/1334723546 shutdown_connections 2023-10-05T19:28:35.291 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.286+0000 7f2db4522e40 1 -- 172.21.15.167:0/1334723546 wait complete. 2023-10-05T19:28:35.292 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.286+0000 7f2db4522e40 1 Processor -- start 2023-10-05T19:28:35.292 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.286+0000 7f2db4522e40 1 -- start start 2023-10-05T19:28:35.292 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.286+0000 7f2db4522e40 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x560a8ac9ebe0 0x560a8aca46b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-05T19:28:35.292 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.286+0000 7f2db4522e40 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x560a8aca56b0 con 0x560a8ac9ebe0 2023-10-05T19:28:35.292 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.286+0000 7f2db1856700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x560a8ac9ebe0 0x560a8aca46b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-05T19:28:35.292 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.286+0000 7f2db1856700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x560a8ac9ebe0 0x560a8aca46b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.118:3300/0 says I am v2:172.21.15.167:41646/0 (socket says 172.21.15.167:41646) 2023-10-05T19:28:35.292 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.286+0000 7f2db1856700 1 -- 172.21.15.167:0/3232857559 learned_addr learned my addr 172.21.15.167:0/3232857559 (peer_addr_for_me v2:172.21.15.167:0/0) 2023-10-05T19:28:35.293 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.286+0000 7f2db1856700 1 -- 172.21.15.167:0/3232857559 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x560a8ab9fbd0 con 0x560a8ac9ebe0 2023-10-05T19:28:35.293 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.286+0000 7f2db1856700 1 --2- 172.21.15.167:0/3232857559 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x560a8ac9ebe0 0x560a8aca46b0 secure :-1 s=READY pgs=421 cs=0 l=1 rev1=1 crypto rx=0x7f2d9c00b960 tx=0x7f2d9c030000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-05T19:28:35.293 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.286+0000 7f2d81fdb700 1 -- 172.21.15.167:0/3232857559 <== mon.0 v2:172.21.15.118:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f2d9c023070 con 0x560a8ac9ebe0 2023-10-05T19:28:35.293 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.286+0000 7f2d81fdb700 1 -- 172.21.15.167:0/3232857559 <== mon.0 v2:172.21.15.118:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2d9c01e630 con 0x560a8ac9ebe0 2023-10-05T19:28:35.294 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.286+0000 7f2d81fdb700 1 -- 172.21.15.167:0/3232857559 <== mon.0 v2:172.21.15.118:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f2d9c01e7a0 con 0x560a8ac9ebe0 2023-10-05T19:28:35.294 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.286+0000 7f2db4522e40 1 -- 172.21.15.167:0/3232857559 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x560a8aca4cb0 con 0x560a8ac9ebe0 2023-10-05T19:28:35.294 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.286+0000 7f2db4522e40 1 -- 172.21.15.167:0/3232857559 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x560a8aca50f0 con 0x560a8ac9ebe0 2023-10-05T19:28:35.294 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.290+0000 7f2d81fdb700 1 -- 172.21.15.167:0/3232857559 <== mon.0 v2:172.21.15.118:3300/0 4 ==== mgrmap(e 5) v1 ==== 40329+0+0 (secure 0 0 0) 0x7f2d9c03f040 con 0x560a8ac9ebe0 2023-10-05T19:28:35.295 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.290+0000 7f2d81fdb700 1 --2- 172.21.15.167:0/3232857559 >> [v2:172.21.15.167:6824/29889,v1:172.21.15.167:6825/29889] conn(0x7f2cf8030c00 0x7f2cf80330b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-05T19:28:35.295 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.290+0000 7f2d81fdb700 1 -- 172.21.15.167:0/3232857559 <== mon.0 v2:172.21.15.118:3300/0 5 ==== osd_map(30..30 src has 1..30) v4 ==== 6057+0+0 (secure 0 0 0) 0x7f2d9c04cc60 con 0x560a8ac9ebe0 2023-10-05T19:28:35.295 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.290+0000 7f2db2858700 1 --2- 172.21.15.167:0/3232857559 >> [v2:172.21.15.167:6824/29889,v1:172.21.15.167:6825/29889] conn(0x7f2cf8030c00 0x7f2cf80330b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-05T19:28:35.295 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.290+0000 7f2db4522e40 1 --2- 172.21.15.167:0/3232857559 >> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] conn(0x560a8acaf330 0x560a8acb1740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-05T19:28:35.295 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.290+0000 7f2db4522e40 1 -- 172.21.15.167:0/3232857559 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:1 6.19 6:9a6dd8c0:gc::gc.0:head [create,call version.check_conds in=74b,call rgw_gc.rgw_gc_queue_init in=45b,call version.set in=34b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x560a8acb1de0 con 0x560a8acaf330 2023-10-05T19:28:35.296 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.290+0000 7f2db2057700 1 --2- 172.21.15.167:0/3232857559 >> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] conn(0x560a8acaf330 0x560a8acb1740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-05T19:28:35.296 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.290+0000 7f2db2858700 1 --2- 172.21.15.167:0/3232857559 >> [v2:172.21.15.167:6824/29889,v1:172.21.15.167:6825/29889] conn(0x7f2cf8030c00 0x7f2cf80330b0 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x560a8ac912e0 tx=0x7f2da4000f90 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-05T19:28:35.296 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.290+0000 7f2db2057700 1 --2- 172.21.15.167:0/3232857559 >> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] conn(0x560a8acaf330 0x560a8acb1740 crc :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-05T19:28:35.299 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.294+0000 7f2db2057700 1 -- 172.21.15.167:0/3232857559 <== osd.1 v2:172.21.15.167:6800/30035 1 ==== osd_op_reply(1 gc.0 [create,call,call,call] v30'27 uv3 ondisk = -125 ((125) Operation canceled)) v8 ==== 274+0+0 (crc 0 0 0) 0x7f2dac05f040 con 0x560a8acaf330 2023-10-05T19:28:35.299 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.294+0000 7f2db4522e40 1 --2- 172.21.15.167:0/3232857559 >> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] conn(0x560a8acb2ab0 0x560a8acb4e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-05T19:28:35.299 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.294+0000 7f2db4522e40 1 -- 172.21.15.167:0/3232857559 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:2 6.18 6:1e85421f:gc::gc.1:head [create,call version.check_conds in=74b,call rgw_gc.rgw_gc_queue_init in=45b,call version.set in=34b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x560a8acb55c0 con 0x560a8acb2ab0 2023-10-05T19:28:35.299 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.294+0000 7f2db1856700 1 --2- 172.21.15.167:0/3232857559 >> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] conn(0x560a8acb2ab0 0x560a8acb4e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-05T19:28:35.300 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.294+0000 7f2db1856700 1 --2- 172.21.15.167:0/3232857559 >> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] conn(0x560a8acb2ab0 0x560a8acb4e90 crc :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-05T19:28:35.302 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.298+0000 7f2db1856700 1 -- 172.21.15.167:0/3232857559 <== osd.0 v2:172.21.15.118:6800/27690 1 ==== osd_op_reply(2 gc.1 [create,call,call,call] v30'19 uv5 ondisk = -125 ((125) Operation canceled)) v8 ==== 274+0+0 (crc 0 0 0) 0x7f2d9c006070 con 0x560a8acb2ab0 2023-10-05T19:28:35.302 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.298+0000 7f2db4522e40 1 --2- 172.21.15.167:0/3232857559 >> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] conn(0x560a8acaaee0 0x560a8acb8c30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-05T19:28:35.303 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.298+0000 7f2db4522e40 1 -- 172.21.15.167:0/3232857559 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:3 6.b 6:d242335b:gc::gc.2:head [create,call version.check_conds in=74b,call rgw_gc.rgw_gc_queue_init in=45b,call version.set in=34b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x560a8acb9170 con 0x560a8acaaee0 2023-10-05T19:28:35.303 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.298+0000 7f2db2858700 1 --2- 172.21.15.167:0/3232857559 >> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] conn(0x560a8acaaee0 0x560a8acb8c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-05T19:28:35.303 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.298+0000 7f2db2858700 1 --2- 172.21.15.167:0/3232857559 >> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] conn(0x560a8acaaee0 0x560a8acb8c30 crc :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-05T19:28:35.306 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.298+0000 7f2db2858700 1 -- 172.21.15.167:0/3232857559 <== osd.3 v2:172.21.15.167:6808/30036 1 ==== osd_op_reply(3 gc.2 [create,call,call,call] v30'16 uv3 ondisk = -125 ((125) Operation canceled)) v8 ==== 274+0+0 (crc 0 0 0) 0x7f2da4050040 con 0x560a8acaaee0 2023-10-05T19:28:35.306 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.302+0000 7f2db4522e40 1 -- 172.21.15.167:0/3232857559 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:4 6.4 6:2166d8e1:gc::gc.3:head [create,call version.check_conds in=74b,call rgw_gc.rgw_gc_queue_init in=45b,call version.set in=34b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x560a8acb97c0 con 0x560a8acaf330 2023-10-05T19:28:35.309 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.302+0000 7f2db2057700 1 -- 172.21.15.167:0/3232857559 <== osd.1 v2:172.21.15.167:6800/30035 2 ==== osd_op_reply(4 gc.3 [create,call,call,call] v30'136 uv7 ondisk = -125 ((125) Operation canceled)) v8 ==== 274+0+0 (crc 0 0 0) 0x7f2dac05f040 con 0x560a8acaf330 2023-10-05T19:28:35.310 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.302+0000 7f2db4522e40 1 --2- 172.21.15.167:0/3232857559 >> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] conn(0x560a8acba460 0x560a8acbc840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-05T19:28:35.310 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.302+0000 7f2db4522e40 1 -- 172.21.15.167:0/3232857559 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:5 6.3 6:c60da066:gc::gc.4:head [create,call version.check_conds in=74b,call rgw_gc.rgw_gc_queue_init in=45b,call version.set in=34b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x560a8acbcef0 con 0x560a8acba460 2023-10-05T19:28:35.310 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.302+0000 7f2db2057700 1 --2- 172.21.15.167:0/3232857559 >> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] conn(0x560a8acba460 0x560a8acbc840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-05T19:28:35.310 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.306+0000 7f2db2057700 1 --2- 172.21.15.167:0/3232857559 >> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] conn(0x560a8acba460 0x560a8acbc840 crc :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-05T19:28:35.312 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.306+0000 7f2db2057700 1 -- 172.21.15.167:0/3232857559 <== osd.2 v2:172.21.15.167:6816/30037 1 ==== osd_op_reply(5 gc.4 [create,call,call,call] v30'38 uv7 ondisk = -125 ((125) Operation canceled)) v8 ==== 274+0+0 (crc 0 0 0) 0x7f2dac05f040 con 0x560a8acba460 2023-10-05T19:28:35.312 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.306+0000 7f2db4522e40 1 -- 172.21.15.167:0/3232857559 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:6 6.9 6:954be8d2:gc::gc.5:head [create,call version.check_conds in=74b,call rgw_gc.rgw_gc_queue_init in=45b,call version.set in=34b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x560a8acbd510 con 0x560a8acaaee0 2023-10-05T19:28:35.315 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.310+0000 7f2db2858700 1 -- 172.21.15.167:0/3232857559 <== osd.3 v2:172.21.15.167:6808/30036 2 ==== osd_op_reply(6 gc.5 [create,call,call,call] v30'27 uv4 ondisk = -125 ((125) Operation canceled)) v8 ==== 274+0+0 (crc 0 0 0) 0x7f2da4050040 con 0x560a8acaaee0 2023-10-05T19:28:35.315 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.310+0000 7f2db4522e40 1 -- 172.21.15.167:0/3232857559 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:7 6.e 6:748722ff:gc::gc.6:head [create,call version.check_conds in=74b,call rgw_gc.rgw_gc_queue_init in=45b,call version.set in=34b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x560a8acbdb30 con 0x560a8acaf330 2023-10-05T19:28:35.317 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.310+0000 7f2db2057700 1 -- 172.21.15.167:0/3232857559 <== osd.1 v2:172.21.15.167:6800/30035 3 ==== osd_op_reply(7 gc.6 [create,call,call,call] v30'18 uv5 ondisk = -125 ((125) Operation canceled)) v8 ==== 274+0+0 (crc 0 0 0) 0x7f2dac05f040 con 0x560a8acaf330 2023-10-05T19:28:35.318 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.310+0000 7f2db4522e40 1 -- 172.21.15.167:0/3232857559 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:8 6.1d 6:beaa4be4:gc::gc.7:head [create,call version.check_conds in=74b,call rgw_gc.rgw_gc_queue_init in=45b,call version.set in=34b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x560a8acbdb30 con 0x560a8acb2ab0 2023-10-05T19:28:35.320 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.314+0000 7f2db1856700 1 -- 172.21.15.167:0/3232857559 <== osd.0 v2:172.21.15.118:6800/27690 2 ==== osd_op_reply(8 gc.7 [create,call,call,call] v30'25 uv4 ondisk = -125 ((125) Operation canceled)) v8 ==== 274+0+0 (crc 0 0 0) 0x7f2d9c006070 con 0x560a8acb2ab0 2023-10-05T19:28:35.321 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.314+0000 7f2db4522e40 1 -- 172.21.15.167:0/3232857559 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:9 6.12 6:4f78e9a2:gc::gc.8:head [create,call version.check_conds in=74b,call rgw_gc.rgw_gc_queue_init in=45b,call version.set in=34b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x560a8acbe150 con 0x560a8acba460 2023-10-05T19:28:35.323 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.318+0000 7f2db2057700 1 -- 172.21.15.167:0/3232857559 <== osd.2 v2:172.21.15.167:6816/30037 2 ==== osd_op_reply(9 gc.8 [create,call,call,call] v30'16 uv3 ondisk = -125 ((125) Operation canceled)) v8 ==== 274+0+0 (crc 0 0 0) 0x7f2dac05f040 con 0x560a8acba460 2023-10-05T19:28:35.323 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.318+0000 7f2db4522e40 1 -- 172.21.15.167:0/3232857559 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:10 6.1c 6:3e426232:gc::gc.9:head [create,call version.check_conds in=74b,call rgw_gc.rgw_gc_queue_init in=45b,call version.set in=34b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x560a8acbe150 con 0x560a8acaaee0 2023-10-05T19:28:35.325 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.318+0000 7f2db2858700 1 -- 172.21.15.167:0/3232857559 <== osd.3 v2:172.21.15.167:6808/30036 3 ==== osd_op_reply(10 gc.9 [create,call,call,call] v30'27 uv3 ondisk = -125 ((125) Operation canceled)) v8 ==== 274+0+0 (crc 0 0 0) 0x7f2da4050040 con 0x560a8acaaee0 2023-10-05T19:28:35.325 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.318+0000 7f2db4522e40 1 -- 172.21.15.167:0/3232857559 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:11 6.3 6:c31f41d7:gc::gc.10:head [create,call version.check_conds in=74b,call rgw_gc.rgw_gc_queue_init in=45b,call version.set in=34b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x560a8acbe640 con 0x560a8acba460 2023-10-05T19:28:35.327 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.322+0000 7f2db2057700 1 -- 172.21.15.167:0/3232857559 <== osd.2 v2:172.21.15.167:6816/30037 3 ==== osd_op_reply(11 gc.10 [create,call,call,call] v30'39 uv11 ondisk = -125 ((125) Operation canceled)) v8 ==== 275+0+0 (crc 0 0 0) 0x7f2dac05f040 con 0x560a8acba460 2023-10-05T19:28:35.327 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.322+0000 7f2db4522e40 1 -- 172.21.15.167:0/3232857559 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:12 6.13 6:c8333aab:gc::gc.11:head [create,call version.check_conds in=74b,call rgw_gc.rgw_gc_queue_init in=45b,call version.set in=34b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x560a8acbea40 con 0x560a8acb2ab0 2023-10-05T19:28:35.330 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.326+0000 7f2db1856700 1 -- 172.21.15.167:0/3232857559 <== osd.0 v2:172.21.15.118:6800/27690 3 ==== osd_op_reply(12 gc.11 [create,call,call,call] v30'19 uv5 ondisk = -125 ((125) Operation canceled)) v8 ==== 275+0+0 (crc 0 0 0) 0x7f2d9c006070 con 0x560a8acb2ab0 2023-10-05T19:28:35.330 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.326+0000 7f2db4522e40 1 -- 172.21.15.167:0/3232857559 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:13 6.15 6:ab569f81:gc::gc.12:head [create,call version.check_conds in=74b,call rgw_gc.rgw_gc_queue_init in=45b,call version.set in=34b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x560a8acbef30 con 0x560a8acba460 2023-10-05T19:28:35.333 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.326+0000 7f2db2057700 1 -- 172.21.15.167:0/3232857559 <== osd.2 v2:172.21.15.167:6816/30037 4 ==== osd_op_reply(13 gc.12 [create,call,call,call] v30'16 uv3 ondisk = -125 ((125) Operation canceled)) v8 ==== 275+0+0 (crc 0 0 0) 0x7f2dac05f040 con 0x560a8acba460 2023-10-05T19:28:35.333 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.326+0000 7f2db4522e40 1 -- 172.21.15.167:0/3232857559 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:14 6.0 6:0667bb13:gc::gc.13:head [create,call version.check_conds in=74b,call rgw_gc.rgw_gc_queue_init in=45b,call version.set in=34b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x560a8acbef30 con 0x560a8acb2ab0 2023-10-05T19:28:35.335 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.330+0000 7f2db1856700 1 -- 172.21.15.167:0/3232857559 <== osd.0 v2:172.21.15.118:6800/27690 4 ==== osd_op_reply(14 gc.13 [create,call,call,call] v30'31 uv6 ondisk = -125 ((125) Operation canceled)) v8 ==== 275+0+0 (crc 0 0 0) 0x7f2d9c006070 con 0x560a8acb2ab0 2023-10-05T19:28:35.335 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.330+0000 7f2db4522e40 1 -- 172.21.15.167:0/3232857559 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:15 6.4 6:227f63b5:gc::gc.14:head [create,call version.check_conds in=74b,call rgw_gc.rgw_gc_queue_init in=45b,call version.set in=34b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x560a8acbef30 con 0x560a8acaf330 2023-10-05T19:28:35.338 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.330+0000 7f2db2057700 1 -- 172.21.15.167:0/3232857559 <== osd.1 v2:172.21.15.167:6800/30035 4 ==== osd_op_reply(15 gc.14 [create,call,call,call] v30'137 uv13 ondisk = -125 ((125) Operation canceled)) v8 ==== 275+0+0 (crc 0 0 0) 0x7f2dac05f040 con 0x560a8acaf330 2023-10-05T19:28:35.338 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.330+0000 7f2db4522e40 1 -- 172.21.15.167:0/3232857559 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:16 6.7 6:e07a6548:gc::gc.15:head [create,call version.check_conds in=74b,call rgw_gc.rgw_gc_queue_init in=45b,call version.set in=34b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x560a8acbef30 con 0x560a8acaaee0 2023-10-05T19:28:35.341 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.334+0000 7f2db2858700 1 -- 172.21.15.167:0/3232857559 <== osd.3 v2:172.21.15.167:6808/30036 4 ==== osd_op_reply(16 gc.15 [create,call,call,call] v30'32 uv7 ondisk = -125 ((125) Operation canceled)) v8 ==== 275+0+0 (crc 0 0 0) 0x7f2da4050040 con 0x560a8acaaee0 2023-10-05T19:28:35.341 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.334+0000 7f2db4522e40 1 -- 172.21.15.167:0/3232857559 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:17 6.3 6:c623b173:gc::gc.16:head [create,call version.check_conds in=74b,call rgw_gc.rgw_gc_queue_init in=45b,call version.set in=34b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x560a8acbf020 con 0x560a8acba460 2023-10-05T19:28:35.343 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.338+0000 7f2db2057700 1 -- 172.21.15.167:0/3232857559 <== osd.2 v2:172.21.15.167:6816/30037 5 ==== osd_op_reply(17 gc.16 [create,call,call,call] v30'40 uv13 ondisk = -125 ((125) Operation canceled)) v8 ==== 275+0+0 (crc 0 0 0) 0x7f2dac05f040 con 0x560a8acba460 2023-10-05T19:28:35.343 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.338+0000 7f2db4522e40 1 -- 172.21.15.167:0/3232857559 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:18 6.16 6:6849fd06:gc::gc.17:head [create,call version.check_conds in=74b,call rgw_gc.rgw_gc_queue_init in=45b,call version.set in=34b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x560a8acbf020 con 0x560a8acaaee0 2023-10-05T19:28:35.345 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.338+0000 7f2db2858700 1 -- 172.21.15.167:0/3232857559 <== osd.3 v2:172.21.15.167:6808/30036 5 ==== osd_op_reply(18 gc.17 [create,call,call,call] v30'16 uv7 ondisk = -125 ((125) Operation canceled)) v8 ==== 275+0+0 (crc 0 0 0) 0x7f2da4050040 con 0x560a8acaaee0 2023-10-05T19:28:35.346 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.338+0000 7f2db4522e40 1 -- 172.21.15.167:0/3232857559 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:19 6.d 6:b019ea83:gc::gc.18:head [create,call version.check_conds in=74b,call rgw_gc.rgw_gc_queue_init in=45b,call version.set in=34b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x560a8acbefc0 con 0x560a8acaf330 2023-10-05T19:28:35.348 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.342+0000 7f2db2057700 1 -- 172.21.15.167:0/3232857559 <== osd.1 v2:172.21.15.167:6800/30035 5 ==== osd_op_reply(19 gc.18 [create,call,call,call] v30'24 uv5 ondisk = -125 ((125) Operation canceled)) v8 ==== 275+0+0 (crc 0 0 0) 0x7f2dac05f040 con 0x560a8acaf330 2023-10-05T19:28:35.348 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.342+0000 7f2db4522e40 1 -- 172.21.15.167:0/3232857559 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:20 6.4 6:24c8e1ae:gc::gc.19:head [create,call version.check_conds in=74b,call rgw_gc.rgw_gc_queue_init in=45b,call version.set in=34b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x560a8acbefc0 con 0x560a8acaf330 2023-10-05T19:28:35.351 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.346+0000 7f2db2057700 1 -- 172.21.15.167:0/3232857559 <== osd.1 v2:172.21.15.167:6800/30035 6 ==== osd_op_reply(20 gc.19 [create,call,call,call] v30'138 uv15 ondisk = -125 ((125) Operation canceled)) v8 ==== 275+0+0 (crc 0 0 0) 0x7f2dac05f040 con 0x560a8acaf330 2023-10-05T19:28:35.351 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.346+0000 7f2db4522e40 1 -- 172.21.15.167:0/3232857559 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:21 6.7 6:e622f76a:gc::gc.20:head [create,call version.check_conds in=74b,call rgw_gc.rgw_gc_queue_init in=45b,call version.set in=34b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x560a8acbefc0 con 0x560a8acaaee0 2023-10-05T19:28:35.354 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.350+0000 7f2db2858700 1 -- 172.21.15.167:0/3232857559 <== osd.3 v2:172.21.15.167:6808/30036 6 ==== osd_op_reply(21 gc.20 [create,call,call,call] v30'33 uv9 ondisk = -125 ((125) Operation canceled)) v8 ==== 275+0+0 (crc 0 0 0) 0x7f2da4050040 con 0x560a8acaaee0 2023-10-05T19:28:35.354 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.350+0000 7f2db4522e40 1 -- 172.21.15.167:0/3232857559 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:22 6.a 6:5127e642:gc::gc.21:head [create,call version.check_conds in=74b,call rgw_gc.rgw_gc_queue_init in=45b,call version.set in=34b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x560a8acbef30 con 0x560a8acba460 2023-10-05T19:28:35.357 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.350+0000 7f2db2057700 1 -- 172.21.15.167:0/3232857559 <== osd.2 v2:172.21.15.167:6816/30037 6 ==== osd_op_reply(22 gc.21 [create,call,call,call] v30'26 uv7 ondisk = -125 ((125) Operation canceled)) v8 ==== 275+0+0 (crc 0 0 0) 0x7f2dac05f040 con 0x560a8acba460 2023-10-05T19:28:35.357 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.350+0000 7f2db4522e40 1 -- 172.21.15.167:0/3232857559 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:23 6.11 6:8eb99c53:gc::gc.22:head [create,call version.check_conds in=74b,call rgw_gc.rgw_gc_queue_init in=45b,call version.set in=34b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x560a8acbef30 con 0x560a8acaaee0 2023-10-05T19:28:35.359 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.354+0000 7f2db2858700 1 -- 172.21.15.167:0/3232857559 <== osd.3 v2:172.21.15.167:6808/30036 7 ==== osd_op_reply(23 gc.22 [create,call,call,call] v30'18 uv3 ondisk = -125 ((125) Operation canceled)) v8 ==== 275+0+0 (crc 0 0 0) 0x7f2da4050040 con 0x560a8acaaee0 2023-10-05T19:28:35.359 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.354+0000 7f2db4522e40 1 -- 172.21.15.167:0/3232857559 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:24 6.8 6:17d7e8ed:gc::gc.23:head [create,call version.check_conds in=74b,call rgw_gc.rgw_gc_queue_init in=45b,call version.set in=34b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x560a8acbef30 con 0x560a8acaaee0 2023-10-05T19:28:35.362 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.358+0000 7f2db2858700 1 -- 172.21.15.167:0/3232857559 <== osd.3 v2:172.21.15.167:6808/30036 8 ==== osd_op_reply(24 gc.23 [create,call,call,call] v30'27 uv5 ondisk = -125 ((125) Operation canceled)) v8 ==== 275+0+0 (crc 0 0 0) 0x7f2da4050040 con 0x560a8acaaee0 2023-10-05T19:28:35.362 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.358+0000 7f2db4522e40 1 -- 172.21.15.167:0/3232857559 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:25 6.1b 6:db9d130d:gc::gc.24:head [create,call version.check_conds in=74b,call rgw_gc.rgw_gc_queue_init in=45b,call version.set in=34b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x560a8acbef30 con 0x560a8acaaee0 2023-10-05T19:28:35.364 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.358+0000 7f2db2858700 1 -- 172.21.15.167:0/3232857559 <== osd.3 v2:172.21.15.167:6808/30036 9 ==== osd_op_reply(25 gc.24 [create,call,call,call] v30'12 uv3 ondisk = -125 ((125) Operation canceled)) v8 ==== 275+0+0 (crc 0 0 0) 0x7f2da4050040 con 0x560a8acaaee0 2023-10-05T19:28:35.364 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.358+0000 7f2db4522e40 1 -- 172.21.15.167:0/3232857559 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:26 6.9 6:921ecdb1:gc::gc.25:head [create,call version.check_conds in=74b,call rgw_gc.rgw_gc_queue_init in=45b,call version.set in=34b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x560a8acbef30 con 0x560a8acaaee0 2023-10-05T19:28:35.366 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.362+0000 7f2db2858700 1 -- 172.21.15.167:0/3232857559 <== osd.3 v2:172.21.15.167:6808/30036 10 ==== osd_op_reply(26 gc.25 [create,call,call,call] v30'28 uv10 ondisk = -125 ((125) Operation canceled)) v8 ==== 275+0+0 (crc 0 0 0) 0x7f2da4050040 con 0x560a8acaaee0 2023-10-05T19:28:35.367 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.362+0000 7f2db4522e40 1 -- 172.21.15.167:0/3232857559 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:27 6.1f 6:fde4dd55:gc::gc.26:head [create,call version.check_conds in=74b,call rgw_gc.rgw_gc_queue_init in=45b,call version.set in=34b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x560a8acbef30 con 0x560a8acb2ab0 2023-10-05T19:28:35.370 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.362+0000 7f2db1856700 1 -- 172.21.15.167:0/3232857559 <== osd.0 v2:172.21.15.118:6800/27690 5 ==== osd_op_reply(27 gc.26 [create,call,call,call] v30'18 uv5 ondisk = -125 ((125) Operation canceled)) v8 ==== 275+0+0 (crc 0 0 0) 0x7f2d9c006070 con 0x560a8acb2ab0 2023-10-05T19:28:35.370 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.362+0000 7f2db4522e40 1 -- 172.21.15.167:0/3232857559 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:28 6.1a 6:5909e671:gc::gc.27:head [create,call version.check_conds in=74b,call rgw_gc.rgw_gc_queue_init in=45b,call version.set in=34b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x560a8acbef30 con 0x560a8acaaee0 2023-10-05T19:28:35.372 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.366+0000 7f2db2858700 1 -- 172.21.15.167:0/3232857559 <== osd.3 v2:172.21.15.167:6808/30036 11 ==== osd_op_reply(28 gc.27 [create,call,call,call] v30'16 uv7 ondisk = -125 ((125) Operation canceled)) v8 ==== 275+0+0 (crc 0 0 0) 0x7f2da4050040 con 0x560a8acaaee0 2023-10-05T19:28:35.372 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.366+0000 7f2db4522e40 1 -- 172.21.15.167:0/3232857559 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:29 6.1d 6:b88e3271:gc::gc.28:head [create,call version.check_conds in=74b,call rgw_gc.rgw_gc_queue_init in=45b,call version.set in=34b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x560a8acbef30 con 0x560a8acb2ab0 2023-10-05T19:28:35.374 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.370+0000 7f2db1856700 1 -- 172.21.15.167:0/3232857559 <== osd.0 v2:172.21.15.118:6800/27690 6 ==== osd_op_reply(29 gc.28 [create,call,call,call] v30'26 uv8 ondisk = -125 ((125) Operation canceled)) v8 ==== 275+0+0 (crc 0 0 0) 0x7f2d9c006070 con 0x560a8acb2ab0 2023-10-05T19:28:35.375 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.370+0000 7f2db4522e40 1 -- 172.21.15.167:0/3232857559 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:30 6.7 6:e47c9322:gc::gc.29:head [create,call version.check_conds in=74b,call rgw_gc.rgw_gc_queue_init in=45b,call version.set in=34b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x560a8acbef30 con 0x560a8acaaee0 2023-10-05T19:28:35.377 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.370+0000 7f2db2858700 1 -- 172.21.15.167:0/3232857559 <== osd.3 v2:172.21.15.167:6808/30036 12 ==== osd_op_reply(30 gc.29 [create,call,call,call] v30'34 uv11 ondisk = -125 ((125) Operation canceled)) v8 ==== 275+0+0 (crc 0 0 0) 0x7f2da4050040 con 0x560a8acaaee0 2023-10-05T19:28:35.377 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.370+0000 7f2db4522e40 1 -- 172.21.15.167:0/3232857559 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:31 6.0 6:013d6c33:gc::gc.30:head [create,call version.check_conds in=74b,call rgw_gc.rgw_gc_queue_init in=45b,call version.set in=34b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x560a8acbef30 con 0x560a8acb2ab0 2023-10-05T19:28:35.380 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.374+0000 7f2db1856700 1 -- 172.21.15.167:0/3232857559 <== osd.0 v2:172.21.15.118:6800/27690 7 ==== osd_op_reply(31 gc.30 [create,call,call,call] v30'32 uv12 ondisk = -125 ((125) Operation canceled)) v8 ==== 275+0+0 (crc 0 0 0) 0x7f2d9c006070 con 0x560a8acb2ab0 2023-10-05T19:28:35.380 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.374+0000 7f2db4522e40 1 -- 172.21.15.167:0/3232857559 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:32 6.f 6:f24cb4a8:gc::gc.31:head [create,call version.check_conds in=74b,call rgw_gc.rgw_gc_queue_init in=45b,call version.set in=34b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x560a8acbef30 con 0x560a8acba460 2023-10-05T19:28:35.383 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.378+0000 7f2db2057700 1 -- 172.21.15.167:0/3232857559 <== osd.2 v2:172.21.15.167:6816/30037 7 ==== osd_op_reply(32 gc.31 [create,call,call,call] v30'22 uv5 ondisk = -125 ((125) Operation canceled)) v8 ==== 275+0+0 (crc 0 0 0) 0x7f2dac05f040 con 0x560a8acba460 2023-10-05T19:28:35.385 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.378+0000 7f2cf2ffd700 20 reqs_thread_entry: start 2023-10-05T19:28:35.386 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.382+0000 7f2db4522e40 20 init_complete bucket index max shards: 11 2023-10-05T19:28:35.386 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.382+0000 7f2cf0ff9700 20 reqs_thread_entry: start 2023-10-05T19:28:35.386 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.382+0000 7f2db4522e40 10 rgw notify: Started notification manager with: 1 workers 2023-10-05T19:28:35.387 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.382+0000 7f2ce67fc700 20 reqs_thread_entry: start 2023-10-05T19:28:35.387 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.382+0000 7f2db4522e40 20 get_system_obj_state: rctx=0x7ffd2378fc38 obj=default.rgw.meta:users.uid:testx$tenanteduser state=0x560a8ab2c0b0 s->prefetch_data=0 2023-10-05T19:28:35.387 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.382+0000 7f2db4522e40 10 cache get: name=default.rgw.meta+users.uid+testx$tenanteduser : miss 2023-10-05T19:28:35.388 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.382+0000 7f2db4522e40 1 -- 172.21.15.167:0/2665859004 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:102 8.13 8:cdfb9947:users.uid::testx$tenanteduser:head [call version.read in=11b,getxattrs,stat] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x560a8ad1da30 con 0x7f2d9c00d3d0 2023-10-05T19:28:35.388 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.382+0000 7f2ce6ffd700 20 rgw notify: INFO: next queues processing will happen at: Thu Oct 5 19:29:05 2023 2023-10-05T19:28:35.388 INFO:teuthology.orchestra.run.smithi167.stderr: 2023-10-05T19:28:35.389 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.382+0000 7f2db2858700 1 -- 172.21.15.167:0/2665859004 <== osd.0 v2:172.21.15.118:6800/27690 29 ==== osd_op_reply(102 testx$tenanteduser [call out=48b,getxattrs out=75b,stat out=16b] v0'0 uv8 ondisk = 0) v8 ==== 246+0+139 (crc 0 0 0) 0x7f2da4050040 con 0x7f2d9c00d3d0 2023-10-05T19:28:35.389 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.382+0000 7f2db4522e40 10 cache put: name=default.rgw.meta+users.uid+testx$tenanteduser info.flags=0x16 2023-10-05T19:28:35.389 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.382+0000 7f2db4522e40 10 adding default.rgw.meta+users.uid+testx$tenanteduser to cache LRU end 2023-10-05T19:28:35.389 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.382+0000 7f2db4522e40 10 updating xattr: name=ceph.objclass.version bl.length()=42 2023-10-05T19:28:35.389 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.382+0000 7f2db4522e40 20 get_system_obj_state: s->obj_tag was set empty 2023-10-05T19:28:35.389 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.382+0000 7f2db4522e40 20 Read xattr: user.rgw.idtag 2023-10-05T19:28:35.390 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.382+0000 7f2db4522e40 10 cache get: name=default.rgw.meta+users.uid+testx$tenanteduser : type miss (requested=0x13, cached=0x16) 2023-10-05T19:28:35.390 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.382+0000 7f2db4522e40 20 rados->read ofs=0 len=0 2023-10-05T19:28:35.390 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.382+0000 7f2db4522e40 1 -- 172.21.15.167:0/2665859004 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:103 8.13 8:cdfb9947:users.uid::testx$tenanteduser:head [call version.check_conds in=122b,call version.read in=11b,read 0~0,getxattrs] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x560a8ad204c0 con 0x7f2d9c00d3d0 2023-10-05T19:28:35.391 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.382+0000 7f2db2858700 1 -- 172.21.15.167:0/2665859004 <== osd.0 v2:172.21.15.118:6800/27690 30 ==== osd_op_reply(103 testx$tenanteduser [call,call out=48b,read 0~385 out=385b,getxattrs out=75b] v0'0 uv8 ondisk = 0) v8 ==== 288+0+508 (crc 0 0 0) 0x7f2da4050040 con 0x7f2d9c00d3d0 2023-10-05T19:28:35.391 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.382+0000 7f2db4522e40 20 rados_obj.operate() r=0 bl.length=385 2023-10-05T19:28:35.392 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.382+0000 7f2db4522e40 10 cache put: name=default.rgw.meta+users.uid+testx$tenanteduser info.flags=0x13 2023-10-05T19:28:35.392 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.382+0000 7f2db4522e40 10 moving default.rgw.meta+users.uid+testx$tenanteduser to cache LRU end 2023-10-05T19:28:35.392 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.382+0000 7f2db4522e40 10 updating xattr: name=ceph.objclass.version bl.length()=42 2023-10-05T19:28:35.392 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.382+0000 7f2db4522e40 1 -- 172.21.15.167:0/2665859004 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:104 8.c 8:37c7a57e:users.uid::testx$tenanteduser.buckets:head [call user.list_buckets in=34b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x560a8ad204c0 con 0x560a8ac52620 2023-10-05T19:28:35.392 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.386+0000 7f2db2858700 1 -- 172.21.15.167:0/2665859004 <== osd.1 v2:172.21.15.167:6800/30035 35 ==== osd_op_reply(104 testx$tenanteduser.buckets [call out=15b] v0'0 uv2 ondisk = 0) v8 ==== 170+0+15 (crc 0 0 0) 0x7f2da4050040 con 0x560a8ac52620 2023-10-05T19:28:35.392 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.386+0000 7f2db4522e40 10 removing key index: p5YnriCv1nAtykxBrupQ 2023-10-05T19:28:35.392 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.386+0000 7f2db4522e40 10 distributing notification oid=default.rgw.control:notify.2 cni=[op: 1, obj: default.rgw.meta:users.keys:p5YnriCv1nAtykxBrupQ, ofs0, ns] 2023-10-05T19:28:35.392 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.386+0000 7f2db4522e40 1 -- 172.21.15.167:0/2665859004 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:105 7.15 7:a93a5511:::notify.2:head [notify cookie 94603278675904 in=182b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x560a8ad20910 con 0x560a8ac86170 2023-10-05T19:28:35.392 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.386+0000 7f2db2057700 1 -- 172.21.15.167:0/2665859004 <== osd.3 v2:172.21.15.167:6808/30036 24 ==== watch-notify(notify (1) cookie 94603278076896 notify 128849018894 ret 0) v3 ==== 212+0+0 (crc 0 0 0) 0x7f2dac04f040 con 0x560a8ac86170 2023-10-05T19:28:35.393 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.386+0000 7f2dabfff700 10 rgw watcher librados: RGWWatcher::handle_notify() notify_id 128849018894 cookie 94603278076896 notifier 4509 bl.length()=170 2023-10-05T19:28:35.393 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.386+0000 7f2db2057700 1 -- 172.21.15.167:0/2665859004 <== osd.3 v2:172.21.15.167:6808/30036 25 ==== osd_op_reply(105 notify.2 [notify cookie 94603278675904 out=8b] v0'0 uv141 ondisk = 0) v8 ==== 152+0+8 (crc 0 0 0) 0x7f2dac05f040 con 0x560a8ac86170 2023-10-05T19:28:35.393 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.386+0000 7f2dabfff700 1 -- 172.21.15.167:0/2665859004 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:106 7.15 7:a93a5511:::notify.2:head [notify-ack in=20b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x7f2d98003c50 con 0x560a8ac86170 2023-10-05T19:28:35.393 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.386+0000 7f2db2057700 1 -- 172.21.15.167:0/2665859004 <== osd.3 v2:172.21.15.167:6808/30036 26 ==== osd_op_reply(106 notify.2 [notify-ack] v0'0 uv141 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f2dac05f040 con 0x560a8ac86170 2023-10-05T19:28:35.393 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.386+0000 7f2db2057700 1 -- 172.21.15.167:0/2665859004 <== osd.3 v2:172.21.15.167:6808/30036 27 ==== watch-notify(notify_complete (2) cookie 94603278675904 notify 128849018894 ret 0) v3 ==== 42+0+48 (crc 0 0 0) 0x560a8ac772c0 con 0x560a8ac86170 2023-10-05T19:28:35.394 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.386+0000 7f2db4522e40 1 -- 172.21.15.167:0/2665859004 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:107 8.13 8:cc695d75:users.keys::p5YnriCv1nAtykxBrupQ:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x560a8ad1e4a0 con 0x7f2d9c00d3d0 2023-10-05T19:28:35.395 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.390+0000 7f2db2858700 1 -- 172.21.15.167:0/2665859004 <== osd.0 v2:172.21.15.118:6800/27690 31 ==== osd_op_reply(107 p5YnriCv1nAtykxBrupQ [delete] v30'10 uv10 ondisk = 0) v8 ==== 164+0+0 (crc 0 0 0) 0x7f2da4050040 con 0x7f2d9c00d3d0 2023-10-05T19:28:35.395 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.390+0000 7f2db4522e40 10 removing email index: 2023-10-05T19:28:35.395 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.390+0000 7f2db4522e40 10 removing user buckets index 2023-10-05T19:28:35.395 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.390+0000 7f2db4522e40 10 distributing notification oid=default.rgw.control:notify.4 cni=[op: 1, obj: default.rgw.meta:users.uid:testx$tenanteduser.buckets, ofs0, ns] 2023-10-05T19:28:35.395 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.390+0000 7f2db4522e40 1 -- 172.21.15.167:0/2665859004 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:108 7.d 7:b4812045:::notify.4:head [notify cookie 94603278675904 in=187b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x560a8ad20910 con 0x560a8ac52620 2023-10-05T19:28:35.396 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.390+0000 7f2db2858700 1 -- 172.21.15.167:0/2665859004 <== osd.1 v2:172.21.15.167:6800/30035 36 ==== watch-notify(notify (1) cookie 94603278083520 notify 128849018896 ret 0) v3 ==== 217+0+0 (crc 0 0 0) 0x7f2da403d040 con 0x560a8ac52620 2023-10-05T19:28:35.396 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.390+0000 7f2dab7fe700 10 rgw watcher librados: RGWWatcher::handle_notify() notify_id 128849018896 cookie 94603278083520 notifier 4509 bl.length()=175 2023-10-05T19:28:35.396 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.390+0000 7f2dab7fe700 1 -- 172.21.15.167:0/2665859004 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:109 7.d 7:b4812045:::notify.4:head [notify-ack in=20b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x7f2d000032d0 con 0x560a8ac52620 2023-10-05T19:28:35.396 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.390+0000 7f2db2858700 1 -- 172.21.15.167:0/2665859004 <== osd.1 v2:172.21.15.167:6800/30035 37 ==== osd_op_reply(108 notify.4 [notify cookie 94603278675904 out=8b] v0'0 uv141 ondisk = 0) v8 ==== 152+0+8 (crc 0 0 0) 0x7f2da4050040 con 0x560a8ac52620 2023-10-05T19:28:35.397 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.390+0000 7f2db2858700 1 -- 172.21.15.167:0/2665859004 <== osd.1 v2:172.21.15.167:6800/30035 38 ==== watch-notify(notify_complete (2) cookie 94603278675904 notify 128849018896 ret 0) v3 ==== 42+0+48 (crc 0 0 0) 0x7f2da4002db0 con 0x560a8ac52620 2023-10-05T19:28:35.397 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.390+0000 7f2db2858700 1 -- 172.21.15.167:0/2665859004 <== osd.1 v2:172.21.15.167:6800/30035 39 ==== osd_op_reply(109 notify.4 [notify-ack] v0'0 uv141 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f2da4050040 con 0x560a8ac52620 2023-10-05T19:28:35.397 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.390+0000 7f2db4522e40 1 -- 172.21.15.167:0/2665859004 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:110 8.c 8:37c7a57e:users.uid::testx$tenanteduser.buckets:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x560a8ad1e4a0 con 0x560a8ac52620 2023-10-05T19:28:35.400 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.394+0000 7f2db2858700 1 -- 172.21.15.167:0/2665859004 <== osd.1 v2:172.21.15.167:6800/30035 40 ==== osd_op_reply(110 testx$tenanteduser.buckets [delete] v30'3 uv3 ondisk = 0) v8 ==== 170+0+0 (crc 0 0 0) 0x7f2da4050040 con 0x560a8ac52620 2023-10-05T19:28:35.400 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.394+0000 7f2db4522e40 10 removing user index: testx$tenanteduser 2023-10-05T19:28:35.400 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.394+0000 7f2db4522e40 10 removing default.rgw.meta+users.uid+testx$tenanteduser from cache 2023-10-05T19:28:35.400 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.394+0000 7f2db4522e40 10 distributing notification oid=default.rgw.control:notify.3 cni=[op: 1, obj: default.rgw.meta:users.uid:testx$tenanteduser, ofs0, ns] 2023-10-05T19:28:35.400 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.394+0000 7f2db4522e40 1 -- 172.21.15.167:0/2665859004 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:111 7.e 7:7759931f:::notify.3:head [notify cookie 94603278671792 in=179b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x560a8ad1f900 con 0x7f2dac037c00 2023-10-05T19:28:35.401 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.394+0000 7f2db1856700 1 -- 172.21.15.167:0/2665859004 <== osd.2 v2:172.21.15.167:6816/30037 17 ==== watch-notify(notify (1) cookie 94603278081312 notify 128849018893 ret 0) v3 ==== 209+0+0 (crc 0 0 0) 0x7f2d9c055040 con 0x7f2dac037c00 2023-10-05T19:28:35.401 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.394+0000 7f2dabfff700 10 rgw watcher librados: RGWWatcher::handle_notify() notify_id 128849018893 cookie 94603278081312 notifier 4509 bl.length()=167 2023-10-05T19:28:35.401 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.394+0000 7f2dabfff700 1 -- 172.21.15.167:0/2665859004 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:112 7.e 7:7759931f:::notify.3:head [notify-ack in=20b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x7f2d98003c50 con 0x7f2dac037c00 2023-10-05T19:28:35.401 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.394+0000 7f2db1856700 1 -- 172.21.15.167:0/2665859004 <== osd.2 v2:172.21.15.167:6816/30037 18 ==== osd_op_reply(111 notify.3 [notify cookie 94603278671792 out=8b] v0'0 uv141 ondisk = 0) v8 ==== 152+0+8 (crc 0 0 0) 0x7f2d9c006070 con 0x7f2dac037c00 2023-10-05T19:28:35.402 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.394+0000 7f2db1856700 1 -- 172.21.15.167:0/2665859004 <== osd.2 v2:172.21.15.167:6816/30037 19 ==== watch-notify(notify_complete (2) cookie 94603278671792 notify 128849018893 ret 0) v3 ==== 42+0+48 (crc 0 0 0) 0x560a8aca50f0 con 0x7f2dac037c00 2023-10-05T19:28:35.402 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.394+0000 7f2db1856700 1 -- 172.21.15.167:0/2665859004 <== osd.2 v2:172.21.15.167:6816/30037 20 ==== osd_op_reply(112 notify.3 [notify-ack] v0'0 uv141 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f2d9c006070 con 0x7f2dac037c00 2023-10-05T19:28:35.402 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.394+0000 7f2db4522e40 1 -- 172.21.15.167:0/2665859004 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:113 8.13 8:cdfb9947:users.uid::testx$tenanteduser:head [call version.inc in=38b,delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x560a8ad1f4b0 con 0x7f2d9c00d3d0 2023-10-05T19:28:35.405 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.398+0000 7f2db2858700 1 -- 172.21.15.167:0/2665859004 <== osd.0 v2:172.21.15.118:6800/27690 32 ==== osd_op_reply(113 testx$tenanteduser [call,delete] v30'11 uv11 ondisk = 0) v8 ==== 204+0+0 (crc 0 0 0) 0x7f2da4050040 con 0x7f2d9c00d3d0 2023-10-05T19:28:35.405 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.398+0000 7f2db4522e40 1 -- 172.21.15.167:0/2665859004 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:114 7.1f 7:f95f44c2:::notify.0:head [watch unwatch cookie 94603278069968] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x560a8ac66580 con 0x7f2dac037c00 2023-10-05T19:28:35.408 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.402+0000 7f2db1856700 1 -- 172.21.15.167:0/2665859004 <== osd.2 v2:172.21.15.167:6816/30037 21 ==== osd_op_reply(114 notify.0 [watch unwatch cookie 94603278069968] v30'143 uv141 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f2d9c006070 con 0x7f2dac037c00 2023-10-05T19:28:35.408 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.402+0000 7f2db4522e40 20 remove_watcher() i=0 2023-10-05T19:28:35.408 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.402+0000 7f2db4522e40 2 removed watcher, disabling cache 2023-10-05T19:28:35.408 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.402+0000 7f2db4522e40 1 -- 172.21.15.167:0/2665859004 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:115 7.0 7:05bf5b68:::notify.1:head [watch unwatch cookie 94603278074544] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x560a8ac66580 con 0x560a8ac52620 2023-10-05T19:28:35.410 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.406+0000 7f2db2858700 1 -- 172.21.15.167:0/2665859004 <== osd.1 v2:172.21.15.167:6800/30035 41 ==== osd_op_reply(115 notify.1 [watch unwatch cookie 94603278074544] v30'143 uv141 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f2da4050040 con 0x560a8ac52620 2023-10-05T19:28:35.411 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.406+0000 7f2db4522e40 20 remove_watcher() i=1 2023-10-05T19:28:35.411 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.406+0000 7f2db4522e40 1 -- 172.21.15.167:0/2665859004 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:116 7.15 7:a93a5511:::notify.2:head [watch unwatch cookie 94603278076896] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x560a8ac66580 con 0x560a8ac86170 2023-10-05T19:28:35.413 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.406+0000 7f2db2057700 1 -- 172.21.15.167:0/2665859004 <== osd.3 v2:172.21.15.167:6808/30036 28 ==== osd_op_reply(116 notify.2 [watch unwatch cookie 94603278076896] v30'143 uv141 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f2dac05f040 con 0x560a8ac86170 2023-10-05T19:28:35.413 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.406+0000 7f2db4522e40 20 remove_watcher() i=2 2023-10-05T19:28:35.413 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.406+0000 7f2db4522e40 1 -- 172.21.15.167:0/2665859004 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:117 7.e 7:7759931f:::notify.3:head [watch unwatch cookie 94603278081312] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x560a8ac66580 con 0x7f2dac037c00 2023-10-05T19:28:35.416 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.410+0000 7f2db1856700 1 -- 172.21.15.167:0/2665859004 <== osd.2 v2:172.21.15.167:6816/30037 22 ==== osd_op_reply(117 notify.3 [watch unwatch cookie 94603278081312] v30'143 uv141 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f2d9c006070 con 0x7f2dac037c00 2023-10-05T19:28:35.416 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.410+0000 7f2db4522e40 20 remove_watcher() i=3 2023-10-05T19:28:35.416 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.410+0000 7f2db4522e40 1 -- 172.21.15.167:0/2665859004 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:118 7.d 7:b4812045:::notify.4:head [watch unwatch cookie 94603278083520] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x560a8ac66580 con 0x560a8ac52620 2023-10-05T19:28:35.419 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.414+0000 7f2db2858700 1 -- 172.21.15.167:0/2665859004 <== osd.1 v2:172.21.15.167:6800/30035 42 ==== osd_op_reply(118 notify.4 [watch unwatch cookie 94603278083520] v30'143 uv141 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f2da4050040 con 0x560a8ac52620 2023-10-05T19:28:35.419 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.414+0000 7f2db4522e40 20 remove_watcher() i=4 2023-10-05T19:28:35.419 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.414+0000 7f2db4522e40 1 -- 172.21.15.167:0/2665859004 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:119 7.3 7:c609908c:::notify.5:head [watch unwatch cookie 94603278078000] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x560a8ac66580 con 0x7f2d9c00d3d0 2023-10-05T19:28:35.421 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.414+0000 7f2db2858700 1 -- 172.21.15.167:0/2665859004 <== osd.0 v2:172.21.15.118:6800/27690 33 ==== osd_op_reply(119 notify.5 [watch unwatch cookie 94603278078000] v30'143 uv141 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f2da4050040 con 0x7f2d9c00d3d0 2023-10-05T19:28:35.421 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.414+0000 7f2db4522e40 20 remove_watcher() i=5 2023-10-05T19:28:35.422 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.414+0000 7f2db4522e40 1 -- 172.21.15.167:0/2665859004 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:120 7.14 7:2b04a3e9:::notify.6:head [watch unwatch cookie 94603278088736] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x560a8ac8d0d0 con 0x7f2d9c00d3d0 2023-10-05T19:28:35.424 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.418+0000 7f2db2858700 1 -- 172.21.15.167:0/2665859004 <== osd.0 v2:172.21.15.118:6800/27690 34 ==== osd_op_reply(120 notify.6 [watch unwatch cookie 94603278088736] v30'143 uv141 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f2da4050040 con 0x7f2d9c00d3d0 2023-10-05T19:28:35.424 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.418+0000 7f2db4522e40 20 remove_watcher() i=6 2023-10-05T19:28:35.424 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.418+0000 7f2db4522e40 1 -- 172.21.15.167:0/2665859004 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:121 7.9 7:93e5b521:::notify.7:head [watch unwatch cookie 94603278093392] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x560a8ac8d0d0 con 0x7f2d9c00d3d0 2023-10-05T19:28:35.428 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.422+0000 7f2db2858700 1 -- 172.21.15.167:0/2665859004 <== osd.0 v2:172.21.15.118:6800/27690 35 ==== osd_op_reply(121 notify.7 [watch unwatch cookie 94603278093392] v30'143 uv141 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f2da4050040 con 0x7f2d9c00d3d0 2023-10-05T19:28:35.428 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.422+0000 7f2db4522e40 20 remove_watcher() i=7 2023-10-05T19:28:35.430 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.422+0000 7f2d83fdf700 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2023-10-05T19:28:35.430 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.426+0000 7f2db4522e40 1 -- 172.21.15.167:0/2665859004 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:122 6.4 6:22d26bf9:::data_loggenerations_metadata:head [watch unwatch cookie 94603278089840] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x560a8ac8d0d0 con 0x560a8ac52620 2023-10-05T19:28:35.433 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.426+0000 7f2db2858700 1 -- 172.21.15.167:0/2665859004 <== osd.1 v2:172.21.15.167:6800/30035 43 ==== osd_op_reply(122 data_loggenerations_metadata [watch unwatch cookie 94603278089840] v30'139 uv1 ondisk = 0) v8 ==== 172+0+0 (crc 0 0 0) 0x7f2da4050040 con 0x560a8ac52620 2023-10-05T19:28:35.433 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.426+0000 7f2db4522e40 1 -- 172.21.15.167:0/2665859004 >> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] conn(0x7f2d9c00d3d0 msgr2=0x7f2d9c00d820 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-05T19:28:35.433 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.426+0000 7f2db4522e40 1 --2- 172.21.15.167:0/2665859004 >> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] conn(0x7f2d9c00d3d0 0x7f2d9c00d820 crc :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-05T19:28:35.433 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.426+0000 7f2db4522e40 1 -- 172.21.15.167:0/2665859004 >> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] conn(0x560a8ac52620 msgr2=0x560a8ac823d0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-05T19:28:35.434 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.426+0000 7f2db4522e40 1 --2- 172.21.15.167:0/2665859004 >> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] conn(0x560a8ac52620 0x560a8ac823d0 crc :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-05T19:28:35.434 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.426+0000 7f2db4522e40 1 -- 172.21.15.167:0/2665859004 >> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] conn(0x7f2dac037c00 msgr2=0x7f2dac0395e0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-05T19:28:35.434 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.426+0000 7f2db4522e40 1 --2- 172.21.15.167:0/2665859004 >> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] conn(0x7f2dac037c00 0x7f2dac0395e0 crc :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-05T19:28:35.434 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.426+0000 7f2db4522e40 1 -- 172.21.15.167:0/2665859004 >> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] conn(0x560a8ac86170 msgr2=0x560a8ac88550 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-05T19:28:35.434 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.426+0000 7f2db4522e40 1 --2- 172.21.15.167:0/2665859004 >> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] conn(0x560a8ac86170 0x560a8ac88550 crc :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-05T19:28:35.436 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.426+0000 7f2db4522e40 1 -- 172.21.15.167:0/2665859004 >> [v2:172.21.15.167:6824/29889,v1:172.21.15.167:6825/29889] conn(0x7f2d94030c50 msgr2=0x7f2d94033100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-05T19:28:35.436 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.426+0000 7f2db4522e40 1 --2- 172.21.15.167:0/2665859004 >> [v2:172.21.15.167:6824/29889,v1:172.21.15.167:6825/29889] conn(0x7f2d94030c50 0x7f2d94033100 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7f2d9c004170 tx=0x7f2d9c001380 comp rx=0 tx=0).stop 2023-10-05T19:28:35.436 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.426+0000 7f2db2858700 1 -- 172.21.15.167:0/2665859004 reap_dead start 2023-10-05T19:28:35.436 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.426+0000 7f2db4522e40 1 -- 172.21.15.167:0/2665859004 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x560a8ac51d10 msgr2=0x560a8ac76940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-05T19:28:35.436 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.426+0000 7f2db4522e40 1 --2- 172.21.15.167:0/2665859004 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x560a8ac51d10 0x560a8ac76940 secure :-1 s=READY pgs=419 cs=0 l=1 rev1=1 crypto rx=0x7f2dac00a470 tx=0x7f2dac01d000 comp rx=0 tx=0).stop 2023-10-05T19:28:35.436 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.426+0000 7f2db4522e40 1 -- 172.21.15.167:0/2665859004 shutdown_connections 2023-10-05T19:28:35.436 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.426+0000 7f2db4522e40 1 --2- 172.21.15.167:0/2665859004 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x560a8ac51d10 0x560a8ac76940 unknown :-1 s=CLOSED pgs=419 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-05T19:28:35.436 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.430+0000 7f2db4522e40 1 -- 172.21.15.167:0/2665859004 >> 172.21.15.167:0/2665859004 conn(0x560a8ac4d730 msgr2=0x560a8ac6a320 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-05T19:28:35.436 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.430+0000 7f2db4522e40 1 -- 172.21.15.167:0/2665859004 shutdown_connections 2023-10-05T19:28:35.437 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.430+0000 7f2db4522e40 1 -- 172.21.15.167:0/2665859004 wait complete. 2023-10-05T19:28:35.437 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.430+0000 7f2db4522e40 1 -- 172.21.15.167:0/3232857559 >> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] conn(0x560a8acb2ab0 msgr2=0x560a8acb4e90 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-05T19:28:35.437 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.430+0000 7f2db4522e40 1 --2- 172.21.15.167:0/3232857559 >> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] conn(0x560a8acb2ab0 0x560a8acb4e90 crc :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-05T19:28:35.437 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.430+0000 7f2db4522e40 1 -- 172.21.15.167:0/3232857559 >> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] conn(0x560a8acaf330 msgr2=0x560a8acb1740 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-05T19:28:35.437 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.430+0000 7f2db4522e40 1 --2- 172.21.15.167:0/3232857559 >> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] conn(0x560a8acaf330 0x560a8acb1740 crc :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-05T19:28:35.437 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.430+0000 7f2db4522e40 1 -- 172.21.15.167:0/3232857559 >> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] conn(0x560a8acba460 msgr2=0x560a8acbc840 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-05T19:28:35.437 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.430+0000 7f2db4522e40 1 --2- 172.21.15.167:0/3232857559 >> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] conn(0x560a8acba460 0x560a8acbc840 crc :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-05T19:28:35.437 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.430+0000 7f2db4522e40 1 -- 172.21.15.167:0/3232857559 >> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] conn(0x560a8acaaee0 msgr2=0x560a8acb8c30 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-05T19:28:35.437 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.430+0000 7f2db4522e40 1 --2- 172.21.15.167:0/3232857559 >> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] conn(0x560a8acaaee0 0x560a8acb8c30 crc :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-05T19:28:35.437 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.430+0000 7f2db4522e40 1 -- 172.21.15.167:0/3232857559 >> [v2:172.21.15.167:6824/29889,v1:172.21.15.167:6825/29889] conn(0x7f2cf8030c00 msgr2=0x7f2cf80330b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-05T19:28:35.437 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.430+0000 7f2db4522e40 1 --2- 172.21.15.167:0/3232857559 >> [v2:172.21.15.167:6824/29889,v1:172.21.15.167:6825/29889] conn(0x7f2cf8030c00 0x7f2cf80330b0 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x560a8ac912e0 tx=0x7f2da4000f90 comp rx=0 tx=0).stop 2023-10-05T19:28:35.437 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.430+0000 7f2db4522e40 1 -- 172.21.15.167:0/3232857559 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x560a8ac9ebe0 msgr2=0x560a8aca46b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-05T19:28:35.438 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.430+0000 7f2db4522e40 1 --2- 172.21.15.167:0/3232857559 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x560a8ac9ebe0 0x560a8aca46b0 secure :-1 s=READY pgs=421 cs=0 l=1 rev1=1 crypto rx=0x7f2d9c00b960 tx=0x7f2d9c030000 comp rx=0 tx=0).stop 2023-10-05T19:28:35.438 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.430+0000 7f2db2057700 1 -- 172.21.15.167:0/3232857559 reap_dead start 2023-10-05T19:28:35.438 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.430+0000 7f2db4522e40 1 -- 172.21.15.167:0/3232857559 shutdown_connections 2023-10-05T19:28:35.438 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.430+0000 7f2db4522e40 1 -- 172.21.15.167:0/3232857559 >> 172.21.15.167:0/3232857559 conn(0x560a8ac99fa0 msgr2=0x560a8ac98040 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-05T19:28:35.438 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.430+0000 7f2db4522e40 1 -- 172.21.15.167:0/3232857559 shutdown_connections 2023-10-05T19:28:35.438 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.430+0000 7f2db4522e40 1 -- 172.21.15.167:0/3232857559 wait complete. 2023-10-05T19:28:35.453 INFO:tasks.radosgw_admin:RL: put_obj myfoo foo bytes_out=4 bytes_in=0 failed=False 2023-10-05T19:28:35.458 INFO:tasks.radosgw_admin:RL: get_obj myfoo foo bytes_out=0 bytes_in=4 failed=False 2023-10-05T19:28:35.507 INFO:tasks.radosgw_admin:RL: list_bucket myfoo foo bytes_out=0 bytes_in=0 failed=False 2023-10-05T19:28:35.507 INFO:tasks.util.rgw:rgwadmin: client.0 : ['object', 'rm', '--bucket', 'myfoo', '--object', 'four'] 2023-10-05T19:28:35.507 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'ceph', 'object', 'rm', '--bucket', 'myfoo', '--object', 'four'] 2023-10-05T19:28:35.507 DEBUG:teuthology.orchestra.run.smithi167:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster ceph object rm --bucket myfoo --object four 2023-10-05T19:28:35.549 INFO:teuthology.orchestra.run.smithi167.stderr:ignoring --setuser ceph since I am not root 2023-10-05T19:28:35.549 INFO:teuthology.orchestra.run.smithi167.stderr:ignoring --setgroup ceph since I am not root 2023-10-05T19:28:35.549 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.542+0000 7f783302ee40 1 Processor -- start 2023-10-05T19:28:35.550 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.542+0000 7f783302ee40 1 -- start start 2023-10-05T19:28:35.550 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.542+0000 7f783302ee40 1 --2- >> v2:172.21.15.118:3300/0 conn(0x5570fe6a5600 0x5570fe6a79e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-05T19:28:35.550 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.542+0000 7f783302ee40 1 -- --> v1:172.21.15.118:6789/0 -- auth(proto 0 26 bytes epoch 0) v1 -- 0x5570fe5ebbd0 con 0x5570fe6a1020 2023-10-05T19:28:35.550 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.542+0000 7f783302ee40 1 -- --> v2:172.21.15.118:3300/0 -- mon_getmap magic: 0 v1 -- 0x5570fe5a8ee0 con 0x5570fe6a5600 2023-10-05T19:28:35.550 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.546+0000 7f7830362700 1 --2- >> v2:172.21.15.118:3300/0 conn(0x5570fe6a5600 0x5570fe6a79e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-05T19:28:35.551 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.546+0000 7f7830b63700 1 --1- >> v1:172.21.15.118:6789/0 conn(0x5570fe6a1020 0x5570fe6a13f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.118:6789/0 says I am v1:172.21.15.167:59290/0 (socket says 172.21.15.167:59290) 2023-10-05T19:28:35.551 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.546+0000 7f7830362700 1 --2- >> v2:172.21.15.118:3300/0 conn(0x5570fe6a5600 0x5570fe6a79e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.118:3300/0 says I am v2:172.21.15.167:41656/0 (socket says 172.21.15.167:41656) 2023-10-05T19:28:35.551 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.546+0000 7f7830b63700 1 -- 172.21.15.167:0/1925071916 learned_addr learned my addr 172.21.15.167:0/1925071916 (peer_addr_for_me v1:172.21.15.167:0/0) 2023-10-05T19:28:35.551 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.546+0000 7f7830362700 1 -- 172.21.15.167:0/1925071916 >> v1:172.21.15.118:6789/0 conn(0x5570fe6a1020 legacy=0x5570fe6a13f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-05T19:28:35.551 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.546+0000 7f782fb61700 1 -- 172.21.15.167:0/1925071916 <== mon.0 v1:172.21.15.118:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4098826043 0 0) 0x5570fe5ebbd0 con 0x5570fe6a1020 2023-10-05T19:28:35.551 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.546+0000 7f7830362700 1 -- 172.21.15.167:0/1925071916 --> v2:172.21.15.118:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5570fe6a1c30 con 0x5570fe6a5600 2023-10-05T19:28:35.552 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.546+0000 7f7830362700 1 --2- 172.21.15.167:0/1925071916 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5570fe6a5600 0x5570fe6a79e0 secure :-1 s=READY pgs=423 cs=0 l=1 rev1=1 crypto rx=0x7f782400a5d0 tx=0x7f7824005a50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1fe928f10959ccb2 server_cookie=0 in_seq=0 out_seq=0 2023-10-05T19:28:35.553 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.546+0000 7f782fb61700 1 -- 172.21.15.167:0/1925071916 <== mon.0 v2:172.21.15.118:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f78240086a0 con 0x5570fe6a5600 2023-10-05T19:28:35.553 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.546+0000 7f782fb61700 1 -- 172.21.15.167:0/1925071916 <== mon.0 v2:172.21.15.118:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7824008800 con 0x5570fe6a5600 2023-10-05T19:28:35.553 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.546+0000 7f782fb61700 1 -- 172.21.15.167:0/1925071916 <== mon.0 v2:172.21.15.118:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f7824008ad0 con 0x5570fe6a5600 2023-10-05T19:28:35.553 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.546+0000 7f783302ee40 1 -- 172.21.15.167:0/1925071916 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5570fe6a5600 msgr2=0x5570fe6a79e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-05T19:28:35.553 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.546+0000 7f783302ee40 1 --2- 172.21.15.167:0/1925071916 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5570fe6a5600 0x5570fe6a79e0 secure :-1 s=READY pgs=423 cs=0 l=1 rev1=1 crypto rx=0x7f782400a5d0 tx=0x7f7824005a50 comp rx=0 tx=0).stop 2023-10-05T19:28:35.553 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.546+0000 7f783302ee40 1 -- 172.21.15.167:0/1925071916 shutdown_connections 2023-10-05T19:28:35.553 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.546+0000 7f783302ee40 1 --2- 172.21.15.167:0/1925071916 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5570fe6a5600 0x5570fe6a79e0 unknown :-1 s=CLOSED pgs=423 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-05T19:28:35.553 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.546+0000 7f783302ee40 1 -- 172.21.15.167:0/1925071916 >> 172.21.15.167:0/1925071916 conn(0x5570fe614870 msgr2=0x5570fe614c70 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-05T19:28:35.553 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.546+0000 7f783302ee40 1 -- 172.21.15.167:0/1925071916 shutdown_connections 2023-10-05T19:28:35.553 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.546+0000 7f783302ee40 1 -- 172.21.15.167:0/1925071916 wait complete. 2023-10-05T19:28:35.554 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.550+0000 7f783302ee40 1 Processor -- start 2023-10-05T19:28:35.554 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.550+0000 7f783302ee40 1 -- start start 2023-10-05T19:28:35.555 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.550+0000 7f783302ee40 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5570fe6a1020 0x5570fe6bff40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-05T19:28:35.555 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.550+0000 7f783302ee40 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x5570fe56ce30 con 0x5570fe6a1020 2023-10-05T19:28:35.556 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.550+0000 7f7830b63700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5570fe6a1020 0x5570fe6bff40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-05T19:28:35.556 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.550+0000 7f7830b63700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5570fe6a1020 0x5570fe6bff40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.118:3300/0 says I am v2:172.21.15.167:41660/0 (socket says 172.21.15.167:41660) 2023-10-05T19:28:35.556 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.550+0000 7f7830b63700 1 -- 172.21.15.167:0/2203600403 learned_addr learned my addr 172.21.15.167:0/2203600403 (peer_addr_for_me v2:172.21.15.167:0/0) 2023-10-05T19:28:35.556 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.550+0000 7f7830b63700 1 -- 172.21.15.167:0/2203600403 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7824005cf0 con 0x5570fe6a1020 2023-10-05T19:28:35.556 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.550+0000 7f7830b63700 1 --2- 172.21.15.167:0/2203600403 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5570fe6a1020 0x5570fe6bff40 secure :-1 s=READY pgs=424 cs=0 l=1 rev1=1 crypto rx=0x7f7828001630 tx=0x7f78280069b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1508a12b7ef8304d server_cookie=0 in_seq=0 out_seq=0 2023-10-05T19:28:35.557 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.550+0000 7f782eb5f700 1 -- 172.21.15.167:0/2203600403 <== mon.0 v2:172.21.15.118:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f7828006320 con 0x5570fe6a1020 2023-10-05T19:28:35.557 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.550+0000 7f782eb5f700 1 -- 172.21.15.167:0/2203600403 <== mon.0 v2:172.21.15.118:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f78280044c0 con 0x5570fe6a1020 2023-10-05T19:28:35.557 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.550+0000 7f782eb5f700 1 -- 172.21.15.167:0/2203600403 <== mon.0 v2:172.21.15.118:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f7828004650 con 0x5570fe6a1020 2023-10-05T19:28:35.558 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.550+0000 7f783302ee40 1 -- 172.21.15.167:0/2203600403 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5570fe6a1020 msgr2=0x5570fe6bff40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-05T19:28:35.558 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.550+0000 7f783302ee40 1 --2- 172.21.15.167:0/2203600403 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5570fe6a1020 0x5570fe6bff40 secure :-1 s=READY pgs=424 cs=0 l=1 rev1=1 crypto rx=0x7f7828001630 tx=0x7f78280069b0 comp rx=0 tx=0).stop 2023-10-05T19:28:35.558 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.550+0000 7f783302ee40 1 -- 172.21.15.167:0/2203600403 shutdown_connections 2023-10-05T19:28:35.558 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.550+0000 7f783302ee40 1 --2- 172.21.15.167:0/2203600403 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5570fe6a1020 0x5570fe6bff40 unknown :-1 s=CLOSED pgs=424 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-05T19:28:35.558 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.550+0000 7f783302ee40 1 -- 172.21.15.167:0/2203600403 >> 172.21.15.167:0/2203600403 conn(0x5570fe614870 msgr2=0x5570fe6bbb00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-05T19:28:35.558 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.554+0000 7f783302ee40 1 -- 172.21.15.167:0/2203600403 shutdown_connections 2023-10-05T19:28:35.559 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.554+0000 7f783302ee40 1 -- 172.21.15.167:0/2203600403 wait complete. 2023-10-05T19:28:35.559 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.554+0000 7f783302ee40 1 Processor -- start 2023-10-05T19:28:35.559 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.554+0000 7f783302ee40 1 -- start start 2023-10-05T19:28:35.560 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.554+0000 7f783302ee40 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5570fe6a1020 0x5570fe6c6d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-05T19:28:35.560 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.554+0000 7f783302ee40 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x5570fe53fed0 con 0x5570fe6a1020 2023-10-05T19:28:35.560 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.554+0000 7f7830b63700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5570fe6a1020 0x5570fe6c6d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-05T19:28:35.560 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.554+0000 7f7830b63700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5570fe6a1020 0x5570fe6c6d40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.118:3300/0 says I am v2:172.21.15.167:41672/0 (socket says 172.21.15.167:41672) 2023-10-05T19:28:35.560 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.554+0000 7f7830b63700 1 -- 172.21.15.167:0/2022643642 learned_addr learned my addr 172.21.15.167:0/2022643642 (peer_addr_for_me v2:172.21.15.167:0/0) 2023-10-05T19:28:35.561 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.554+0000 7f7830b63700 1 -- 172.21.15.167:0/2022643642 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7828006c90 con 0x5570fe6a1020 2023-10-05T19:28:35.561 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.554+0000 7f7830b63700 1 --2- 172.21.15.167:0/2022643642 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5570fe6a1020 0x5570fe6c6d40 secure :-1 s=READY pgs=425 cs=0 l=1 rev1=1 crypto rx=0x7f78280069f0 tx=0x7f7828014000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-05T19:28:35.561 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.554+0000 7f782d35c700 1 -- 172.21.15.167:0/2022643642 <== mon.0 v2:172.21.15.118:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f78280137a0 con 0x5570fe6a1020 2023-10-05T19:28:35.561 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.554+0000 7f783302ee40 1 -- 172.21.15.167:0/2022643642 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x5570fe6c7280 con 0x5570fe6a1020 2023-10-05T19:28:35.561 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.554+0000 7f782d35c700 1 -- 172.21.15.167:0/2022643642 <== mon.0 v2:172.21.15.118:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7828013920 con 0x5570fe6a1020 2023-10-05T19:28:35.561 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.554+0000 7f782d35c700 1 -- 172.21.15.167:0/2022643642 <== mon.0 v2:172.21.15.118:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f7828013c10 con 0x5570fe6a1020 2023-10-05T19:28:35.561 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.554+0000 7f783302ee40 1 -- 172.21.15.167:0/2022643642 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x5570fe6c76c0 con 0x5570fe6a1020 2023-10-05T19:28:35.562 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.558+0000 7f783302ee40 20 rados->read ofs=0 len=0 2023-10-05T19:28:35.563 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.558+0000 7f782d35c700 1 -- 172.21.15.167:0/2022643642 <== mon.0 v2:172.21.15.118:3300/0 4 ==== mgrmap(e 5) v1 ==== 40329+0+0 (secure 0 0 0) 0x7f782801a070 con 0x5570fe6a1020 2023-10-05T19:28:35.563 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.558+0000 7f782d35c700 1 --2- 172.21.15.167:0/2022643642 >> [v2:172.21.15.167:6824/29889,v1:172.21.15.167:6825/29889] conn(0x7f781c030c00 0x7f781c0330b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-05T19:28:35.564 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.558+0000 7f782d35c700 1 -- 172.21.15.167:0/2022643642 <== mon.0 v2:172.21.15.118:3300/0 5 ==== osd_map(30..30 src has 1..30) v4 ==== 6057+0+0 (secure 0 0 0) 0x7f7828044fa0 con 0x5570fe6a1020 2023-10-05T19:28:35.564 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.558+0000 7f783302ee40 1 --2- 172.21.15.167:0/2022643642 >> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] conn(0x5570fe6a5600 0x5570fe6d28f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-05T19:28:35.564 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.558+0000 7f783302ee40 1 -- 172.21.15.167:0/2022643642 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:1 5.12 5:49953fa1:::default.realm:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x5570fe6d2f10 con 0x5570fe6a5600 2023-10-05T19:28:35.564 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.558+0000 7f7830362700 1 --2- 172.21.15.167:0/2022643642 >> [v2:172.21.15.167:6824/29889,v1:172.21.15.167:6825/29889] conn(0x7f781c030c00 0x7f781c0330b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-05T19:28:35.564 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.558+0000 7f7831364700 1 --2- 172.21.15.167:0/2022643642 >> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] conn(0x5570fe6a5600 0x5570fe6d28f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-05T19:28:35.564 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.558+0000 7f7830362700 1 --2- 172.21.15.167:0/2022643642 >> [v2:172.21.15.167:6824/29889,v1:172.21.15.167:6825/29889] conn(0x7f781c030c00 0x7f781c0330b0 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7f7824005a90 tx=0x7f7824015000 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-05T19:28:35.565 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.558+0000 7f7831364700 1 --2- 172.21.15.167:0/2022643642 >> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] conn(0x5570fe6a5600 0x5570fe6d28f0 crc :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-05T19:28:35.565 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.558+0000 7f7831364700 1 -- 172.21.15.167:0/2022643642 <== osd.1 v2:172.21.15.167:6800/30035 1 ==== osd_op_reply(1 default.realm [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 ==== 157+0+0 (crc 0 0 0) 0x7f782000a500 con 0x5570fe6a5600 2023-10-05T19:28:35.565 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.558+0000 7f783302ee40 20 rados_obj.operate() r=-2 bl.length=0 2023-10-05T19:28:35.566 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.558+0000 7f783302ee40 20 realm 2023-10-05T19:28:35.566 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.558+0000 7f783302ee40 20 rados->read ofs=0 len=0 2023-10-05T19:28:35.566 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.558+0000 7f783302ee40 1 -- 172.21.15.167:0/2022643642 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:2 5.12 5:49953fa1:::default.realm:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x5570fe6cff50 con 0x5570fe6a5600 2023-10-05T19:28:35.567 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.558+0000 7f7831364700 1 -- 172.21.15.167:0/2022643642 <== osd.1 v2:172.21.15.167:6800/30035 2 ==== osd_op_reply(2 default.realm [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 ==== 157+0+0 (crc 0 0 0) 0x7f782000a500 con 0x5570fe6a5600 2023-10-05T19:28:35.567 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.558+0000 7f783302ee40 20 rados_obj.operate() r=-2 bl.length=0 2023-10-05T19:28:35.567 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.562+0000 7f783302ee40 4 RGWPeriod::init failed to init realm id : (2) No such file or directory 2023-10-05T19:28:35.567 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.562+0000 7f783302ee40 20 rados->read ofs=0 len=0 2023-10-05T19:28:35.567 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.562+0000 7f783302ee40 1 -- 172.21.15.167:0/2022643642 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:3 5.12 5:49953fa1:::default.realm:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x5570fe6d39f0 con 0x5570fe6a5600 2023-10-05T19:28:35.567 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.562+0000 7f7831364700 1 -- 172.21.15.167:0/2022643642 <== osd.1 v2:172.21.15.167:6800/30035 3 ==== osd_op_reply(3 default.realm [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 ==== 157+0+0 (crc 0 0 0) 0x7f782000a500 con 0x5570fe6a5600 2023-10-05T19:28:35.567 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.562+0000 7f783302ee40 20 rados_obj.operate() r=-2 bl.length=0 2023-10-05T19:28:35.567 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.562+0000 7f783302ee40 20 rados->read ofs=0 len=0 2023-10-05T19:28:35.567 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.562+0000 7f783302ee40 1 -- 172.21.15.167:0/2022643642 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:4 5.f 5:f43b6ece:::zone_names.default:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x5570fe6d3600 con 0x5570fe6a5600 2023-10-05T19:28:35.568 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.562+0000 7f7831364700 1 -- 172.21.15.167:0/2022643642 <== osd.1 v2:172.21.15.167:6800/30035 4 ==== osd_op_reply(4 zone_names.default [read 0~46 out=46b] v0'0 uv1 ondisk = 0) v8 ==== 162+0+46 (crc 0 0 0) 0x7f782000a500 con 0x5570fe6a5600 2023-10-05T19:28:35.569 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.562+0000 7f783302ee40 20 rados_obj.operate() r=0 bl.length=46 2023-10-05T19:28:35.569 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.562+0000 7f783302ee40 20 rados->read ofs=0 len=0 2023-10-05T19:28:35.569 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.562+0000 7f783302ee40 1 -- 172.21.15.167:0/2022643642 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:5 5.1d 5:b8374eb4:::zone_info.9a9df965-6612-45d4-a45f-6e3c889bacc2:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x5570fe6d4880 con 0x5570fe6a5600 2023-10-05T19:28:35.569 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.562+0000 7f7831364700 1 -- 172.21.15.167:0/2022643642 <== osd.1 v2:172.21.15.167:6800/30035 5 ==== osd_op_reply(5 zone_info.9a9df965-6612-45d4-a45f-6e3c889bacc2 [read 0~874 out=874b] v0'0 uv2 ondisk = 0) v8 ==== 190+0+874 (crc 0 0 0) 0x7f782000a500 con 0x5570fe6a5600 2023-10-05T19:28:35.569 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.562+0000 7f783302ee40 20 rados_obj.operate() r=0 bl.length=874 2023-10-05T19:28:35.569 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.562+0000 7f783302ee40 20 searching for the correct realm 2023-10-05T19:28:35.569 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.562+0000 7f783302ee40 1 --2- 172.21.15.167:0/2022643642 >> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] conn(0x5570fe6d6640 0x5570fe6d8a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-05T19:28:35.569 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.562+0000 7f783302ee40 1 -- 172.21.15.167:0/2022643642 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:6 5.0 5:00000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x5570fe6d8f60 con 0x5570fe6d6640 2023-10-05T19:28:35.569 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.562+0000 7f7830b63700 1 --2- 172.21.15.167:0/2022643642 >> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] conn(0x5570fe6d6640 0x5570fe6d8a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-05T19:28:35.570 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.562+0000 7f7830b63700 1 --2- 172.21.15.167:0/2022643642 >> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] conn(0x5570fe6d6640 0x5570fe6d8a20 crc :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-05T19:28:35.572 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.566+0000 7f7830b63700 1 -- 172.21.15.167:0/2022643642 <== osd.3 v2:172.21.15.167:6808/30036 1 ==== osd_op_reply(6 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f7828025020 con 0x5570fe6d6640 2023-10-05T19:28:35.572 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.566+0000 7f7830b63700 1 --2- 172.21.15.167:0/2022643642 >> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] conn(0x7f782801fc20 0x7f78280420c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-05T19:28:35.572 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.566+0000 7f7830b63700 1 -- 172.21.15.167:0/2022643642 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:7 5.10 5:08000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x5570fe6d8f60 con 0x7f782801fc20 2023-10-05T19:28:35.572 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.566+0000 7f7830362700 1 --2- 172.21.15.167:0/2022643642 >> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] conn(0x7f782801fc20 0x7f78280420c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-05T19:28:35.573 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.566+0000 7f7830362700 1 --2- 172.21.15.167:0/2022643642 >> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] conn(0x7f782801fc20 0x7f78280420c0 crc :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-05T19:28:35.574 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.566+0000 7f7830362700 1 -- 172.21.15.167:0/2022643642 <== osd.2 v2:172.21.15.167:6816/30037 1 ==== osd_op_reply(7 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f7824012450 con 0x7f782801fc20 2023-10-05T19:28:35.574 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.566+0000 7f7830362700 1 --2- 172.21.15.167:0/2022643642 >> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] conn(0x7f7824007030 0x7f782401c920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-05T19:28:35.574 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.566+0000 7f7830362700 1 -- 172.21.15.167:0/2022643642 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:8 5.8 5:10000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x5570fe6d8f60 con 0x7f7824007030 2023-10-05T19:28:35.574 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.570+0000 7f7831364700 1 --2- 172.21.15.167:0/2022643642 >> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] conn(0x7f7824007030 0x7f782401c920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-05T19:28:35.575 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.570+0000 7f7831364700 1 --2- 172.21.15.167:0/2022643642 >> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] conn(0x7f7824007030 0x7f782401c920 crc :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-05T19:28:35.576 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.570+0000 7f7831364700 1 -- 172.21.15.167:0/2022643642 <== osd.0 v2:172.21.15.118:6800/27690 1 ==== osd_op_reply(8 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f7820014040 con 0x7f7824007030 2023-10-05T19:28:35.576 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.570+0000 7f7831364700 1 -- 172.21.15.167:0/2022643642 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:9 5.18 5:18000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x5570fe6d8f60 con 0x5570fe6a5600 2023-10-05T19:28:35.577 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.570+0000 7f7831364700 1 -- 172.21.15.167:0/2022643642 <== osd.1 v2:172.21.15.167:6800/30035 6 ==== osd_op_reply(9 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f7820014040 con 0x5570fe6a5600 2023-10-05T19:28:35.577 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.570+0000 7f7831364700 1 -- 172.21.15.167:0/2022643642 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:10 5.4 5:20000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x5570fe6d8f60 con 0x5570fe6a5600 2023-10-05T19:28:35.577 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.570+0000 7f7831364700 1 -- 172.21.15.167:0/2022643642 <== osd.1 v2:172.21.15.167:6800/30035 7 ==== osd_op_reply(10 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f7820014040 con 0x5570fe6a5600 2023-10-05T19:28:35.578 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.570+0000 7f7831364700 1 -- 172.21.15.167:0/2022643642 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:11 5.14 5:28000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x5570fe6d8f60 con 0x7f7824007030 2023-10-05T19:28:35.578 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.574+0000 7f7831364700 1 -- 172.21.15.167:0/2022643642 <== osd.0 v2:172.21.15.118:6800/27690 2 ==== osd_op_reply(11 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f7820014040 con 0x7f7824007030 2023-10-05T19:28:35.579 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.574+0000 7f7831364700 1 -- 172.21.15.167:0/2022643642 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:12 5.c 5:30000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x5570fe6d8f60 con 0x7f7824007030 2023-10-05T19:28:35.579 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.574+0000 7f7831364700 1 -- 172.21.15.167:0/2022643642 <== osd.0 v2:172.21.15.118:6800/27690 3 ==== osd_op_reply(12 [pgnls start_epoch 30 out=112b] v23'1 uv1 ondisk = 1) v8 ==== 144+0+112 (crc 0 0 0) 0x7f7820014040 con 0x7f7824007030 2023-10-05T19:28:35.579 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.574+0000 7f7831364700 1 -- 172.21.15.167:0/2022643642 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:13 5.1c 5:38000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x5570fe6d8f60 con 0x7f782801fc20 2023-10-05T19:28:35.580 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.574+0000 7f7830362700 1 -- 172.21.15.167:0/2022643642 <== osd.2 v2:172.21.15.167:6816/30037 2 ==== osd_op_reply(13 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f7824012450 con 0x7f782801fc20 2023-10-05T19:28:35.580 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.574+0000 7f7830362700 1 -- 172.21.15.167:0/2022643642 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:14 5.2 5:40000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x5570fe6d8f60 con 0x5570fe6d6640 2023-10-05T19:28:35.581 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.574+0000 7f7830b63700 1 -- 172.21.15.167:0/2022643642 <== osd.3 v2:172.21.15.167:6808/30036 2 ==== osd_op_reply(14 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f7828025020 con 0x5570fe6d6640 2023-10-05T19:28:35.581 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.574+0000 7f7830b63700 1 -- 172.21.15.167:0/2022643642 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:15 5.12 5:48000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x5570fe6d8f60 con 0x5570fe6a5600 2023-10-05T19:28:35.582 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.574+0000 7f7831364700 1 -- 172.21.15.167:0/2022643642 <== osd.1 v2:172.21.15.167:6800/30035 8 ==== osd_op_reply(15 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f7820014040 con 0x5570fe6a5600 2023-10-05T19:28:35.582 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.574+0000 7f7831364700 1 -- 172.21.15.167:0/2022643642 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:16 5.a 5:50000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x5570fe6d8f60 con 0x7f7824007030 2023-10-05T19:28:35.582 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.578+0000 7f7831364700 1 -- 172.21.15.167:0/2022643642 <== osd.0 v2:172.21.15.118:6800/27690 4 ==== osd_op_reply(16 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f7820022040 con 0x7f7824007030 2023-10-05T19:28:35.583 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.578+0000 7f7831364700 1 -- 172.21.15.167:0/2022643642 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:17 5.1a 5:58000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x5570fe6d8f60 con 0x5570fe6a5600 2023-10-05T19:28:35.583 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.578+0000 7f7831364700 1 -- 172.21.15.167:0/2022643642 <== osd.1 v2:172.21.15.167:6800/30035 9 ==== osd_op_reply(17 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f7820022040 con 0x5570fe6a5600 2023-10-05T19:28:35.583 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.578+0000 7f7831364700 1 -- 172.21.15.167:0/2022643642 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:18 5.6 5:60000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x5570fe6d8f60 con 0x7f782801fc20 2023-10-05T19:28:35.584 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.578+0000 7f7830362700 1 -- 172.21.15.167:0/2022643642 <== osd.2 v2:172.21.15.167:6816/30037 3 ==== osd_op_reply(18 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f7824012450 con 0x7f782801fc20 2023-10-05T19:28:35.584 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.578+0000 7f7830362700 1 -- 172.21.15.167:0/2022643642 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:19 5.16 5:68000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x5570fe6d8f60 con 0x5570fe6d6640 2023-10-05T19:28:35.585 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.578+0000 7f7830b63700 1 -- 172.21.15.167:0/2022643642 <== osd.3 v2:172.21.15.167:6808/30036 3 ==== osd_op_reply(19 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f7828025020 con 0x5570fe6d6640 2023-10-05T19:28:35.585 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.578+0000 7f7830b63700 1 -- 172.21.15.167:0/2022643642 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:20 5.e 5:70000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x5570fe6d8f60 con 0x5570fe6d6640 2023-10-05T19:28:35.586 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.578+0000 7f7830b63700 1 -- 172.21.15.167:0/2022643642 <== osd.3 v2:172.21.15.167:6808/30036 4 ==== osd_op_reply(20 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f7828025020 con 0x5570fe6d6640 2023-10-05T19:28:35.586 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.578+0000 7f7830b63700 1 -- 172.21.15.167:0/2022643642 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:21 5.1e 5:78000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x5570fe6d8f60 con 0x7f7824007030 2023-10-05T19:28:35.587 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.582+0000 7f7831364700 1 -- 172.21.15.167:0/2022643642 <== osd.0 v2:172.21.15.118:6800/27690 5 ==== osd_op_reply(21 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f7820022040 con 0x7f7824007030 2023-10-05T19:28:35.587 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.582+0000 7f7831364700 1 -- 172.21.15.167:0/2022643642 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:22 5.1 5:80000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x5570fe6d8f60 con 0x5570fe6d6640 2023-10-05T19:28:35.587 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.582+0000 7f7830b63700 1 -- 172.21.15.167:0/2022643642 <== osd.3 v2:172.21.15.167:6808/30036 5 ==== osd_op_reply(22 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f7828025020 con 0x5570fe6d6640 2023-10-05T19:28:35.588 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.582+0000 7f7830b63700 1 -- 172.21.15.167:0/2022643642 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:23 5.11 5:88000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x5570fe6d8f60 con 0x5570fe6d6640 2023-10-05T19:28:35.588 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.582+0000 7f7830b63700 1 -- 172.21.15.167:0/2022643642 <== osd.3 v2:172.21.15.167:6808/30036 6 ==== osd_op_reply(23 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f7828025020 con 0x5570fe6d6640 2023-10-05T19:28:35.588 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.582+0000 7f7830b63700 1 -- 172.21.15.167:0/2022643642 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:24 5.9 5:90000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x5570fe6d8f60 con 0x5570fe6a5600 2023-10-05T19:28:35.589 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.582+0000 7f7831364700 1 -- 172.21.15.167:0/2022643642 <== osd.1 v2:172.21.15.167:6800/30035 10 ==== osd_op_reply(24 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f7820022040 con 0x5570fe6a5600 2023-10-05T19:28:35.589 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.582+0000 7f7831364700 1 -- 172.21.15.167:0/2022643642 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:25 5.19 5:98000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x5570fe6d8f60 con 0x5570fe6a5600 2023-10-05T19:28:35.590 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.582+0000 7f7831364700 1 -- 172.21.15.167:0/2022643642 <== osd.1 v2:172.21.15.167:6800/30035 11 ==== osd_op_reply(25 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f7820022040 con 0x5570fe6a5600 2023-10-05T19:28:35.590 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.582+0000 7f7831364700 1 -- 172.21.15.167:0/2022643642 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:26 5.5 5:a0000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x5570fe6d8f60 con 0x7f7824007030 2023-10-05T19:28:35.591 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.586+0000 7f7831364700 1 -- 172.21.15.167:0/2022643642 <== osd.0 v2:172.21.15.118:6800/27690 6 ==== osd_op_reply(26 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f7820022040 con 0x7f7824007030 2023-10-05T19:28:35.591 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.586+0000 7f7831364700 1 -- 172.21.15.167:0/2022643642 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:27 5.15 5:a8000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x5570fe6d8f60 con 0x5570fe6d6640 2023-10-05T19:28:35.592 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.586+0000 7f7830b63700 1 -- 172.21.15.167:0/2022643642 <== osd.3 v2:172.21.15.167:6808/30036 7 ==== osd_op_reply(27 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f7828025020 con 0x5570fe6d6640 2023-10-05T19:28:35.592 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.586+0000 7f7830b63700 1 -- 172.21.15.167:0/2022643642 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:28 5.d 5:b0000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x5570fe6d8f60 con 0x7f782801fc20 2023-10-05T19:28:35.592 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.586+0000 7f7830362700 1 -- 172.21.15.167:0/2022643642 <== osd.2 v2:172.21.15.167:6816/30037 4 ==== osd_op_reply(28 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f7824012450 con 0x7f782801fc20 2023-10-05T19:28:35.592 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.586+0000 7f7830362700 1 -- 172.21.15.167:0/2022643642 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:29 5.1d 5:b8000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x5570fe6d8f60 con 0x5570fe6a5600 2023-10-05T19:28:35.593 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.586+0000 7f7831364700 1 -- 172.21.15.167:0/2022643642 <== osd.1 v2:172.21.15.167:6800/30035 12 ==== osd_op_reply(29 [pgnls start_epoch 30 out=107b] v30'2 uv2 ondisk = 1) v8 ==== 144+0+107 (crc 0 0 0) 0x7f7820022040 con 0x5570fe6a5600 2023-10-05T19:28:35.593 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.586+0000 7f7831364700 1 -- 172.21.15.167:0/2022643642 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:30 5.3 5:c0000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x5570fe6d8f60 con 0x7f7824007030 2023-10-05T19:28:35.594 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.590+0000 7f7831364700 1 -- 172.21.15.167:0/2022643642 <== osd.0 v2:172.21.15.118:6800/27690 7 ==== osd_op_reply(30 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f7820022040 con 0x7f7824007030 2023-10-05T19:28:35.594 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.590+0000 7f7831364700 1 -- 172.21.15.167:0/2022643642 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:31 5.13 5:c8000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x5570fe6d8f60 con 0x5570fe6d6640 2023-10-05T19:28:35.595 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.590+0000 7f7830b63700 1 -- 172.21.15.167:0/2022643642 <== osd.3 v2:172.21.15.167:6808/30036 8 ==== osd_op_reply(31 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f7828025020 con 0x5570fe6d6640 2023-10-05T19:28:35.595 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.590+0000 7f7830b63700 1 -- 172.21.15.167:0/2022643642 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:32 5.b 5:d0000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x5570fe6d8f60 con 0x7f7824007030 2023-10-05T19:28:35.596 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.590+0000 7f7831364700 1 -- 172.21.15.167:0/2022643642 <== osd.0 v2:172.21.15.118:6800/27690 8 ==== osd_op_reply(32 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f7820022040 con 0x7f7824007030 2023-10-05T19:28:35.596 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.590+0000 7f7831364700 1 -- 172.21.15.167:0/2022643642 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:33 5.1b 5:d8000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x5570fe6d8f60 con 0x7f782801fc20 2023-10-05T19:28:35.596 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.590+0000 7f7830362700 1 -- 172.21.15.167:0/2022643642 <== osd.2 v2:172.21.15.167:6816/30037 5 ==== osd_op_reply(33 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f7824012450 con 0x7f782801fc20 2023-10-05T19:28:35.597 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.590+0000 7f7830362700 1 -- 172.21.15.167:0/2022643642 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:34 5.7 5:e0000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x5570fe6d8f60 con 0x5570fe6d6640 2023-10-05T19:28:35.597 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.590+0000 7f7830b63700 1 -- 172.21.15.167:0/2022643642 <== osd.3 v2:172.21.15.167:6808/30036 9 ==== osd_op_reply(34 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f7828025020 con 0x5570fe6d6640 2023-10-05T19:28:35.598 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.590+0000 7f7830b63700 1 -- 172.21.15.167:0/2022643642 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:35 5.17 5:e8000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x5570fe6d8f60 con 0x7f7824007030 2023-10-05T19:28:35.599 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.594+0000 7f7831364700 1 -- 172.21.15.167:0/2022643642 <== osd.0 v2:172.21.15.118:6800/27690 9 ==== osd_op_reply(35 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f7820022040 con 0x7f7824007030 2023-10-05T19:28:35.599 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.594+0000 7f7831364700 1 -- 172.21.15.167:0/2022643642 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:36 5.f 5:f0000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x5570fe6d8f60 con 0x5570fe6a5600 2023-10-05T19:28:35.600 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.594+0000 7f7831364700 1 -- 172.21.15.167:0/2022643642 <== osd.1 v2:172.21.15.167:6800/30035 13 ==== osd_op_reply(36 [pgnls start_epoch 30 out=115b] v23'2 uv2 ondisk = 1) v8 ==== 144+0+115 (crc 0 0 0) 0x7f782002f080 con 0x5570fe6a5600 2023-10-05T19:28:35.600 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.594+0000 7f7831364700 1 -- 172.21.15.167:0/2022643642 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:37 5.1f 5:f8000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x5570fe6d8f60 con 0x7f782801fc20 2023-10-05T19:28:35.601 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.594+0000 7f7830362700 1 -- 172.21.15.167:0/2022643642 <== osd.2 v2:172.21.15.167:6816/30037 6 ==== osd_op_reply(37 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f7824012450 con 0x7f782801fc20 2023-10-05T19:28:35.601 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.594+0000 7f783302ee40 20 RGWRados::pool_iterate: got zonegroup_info.4f1f719d-9648-417b-b10c-e5d939825419 2023-10-05T19:28:35.601 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.594+0000 7f783302ee40 20 RGWRados::pool_iterate: got zone_info.9a9df965-6612-45d4-a45f-6e3c889bacc2 2023-10-05T19:28:35.601 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.594+0000 7f783302ee40 20 RGWRados::pool_iterate: got zone_names.default 2023-10-05T19:28:35.602 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.594+0000 7f783302ee40 20 RGWRados::pool_iterate: got zonegroups_names.default 2023-10-05T19:28:35.602 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.594+0000 7f783302ee40 20 rados->read ofs=0 len=0 2023-10-05T19:28:35.602 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.594+0000 7f783302ee40 1 -- 172.21.15.167:0/2022643642 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:38 5.12 5:49953fa1:::default.realm:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x5570fe6d53f0 con 0x5570fe6a5600 2023-10-05T19:28:35.603 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.594+0000 7f7831364700 1 -- 172.21.15.167:0/2022643642 <== osd.1 v2:172.21.15.167:6800/30035 14 ==== osd_op_reply(38 default.realm [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 ==== 157+0+0 (crc 0 0 0) 0x7f782002f080 con 0x5570fe6a5600 2023-10-05T19:28:35.603 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.594+0000 7f783302ee40 20 rados_obj.operate() r=-2 bl.length=0 2023-10-05T19:28:35.604 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.594+0000 7f783302ee40 20 rados->read ofs=0 len=0 2023-10-05T19:28:35.604 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.594+0000 7f783302ee40 1 -- 172.21.15.167:0/2022643642 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:39 5.f 5:f4c53578:::zonegroups_names.default:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x5570fe6d5ce0 con 0x5570fe6a5600 2023-10-05T19:28:35.604 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.598+0000 7f7831364700 1 -- 172.21.15.167:0/2022643642 <== osd.1 v2:172.21.15.167:6800/30035 15 ==== osd_op_reply(39 zonegroups_names.default [read 0~46 out=46b] v0'0 uv2 ondisk = 0) v8 ==== 168+0+46 (crc 0 0 0) 0x7f782002f080 con 0x5570fe6a5600 2023-10-05T19:28:35.604 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.598+0000 7f783302ee40 20 rados_obj.operate() r=0 bl.length=46 2023-10-05T19:28:35.604 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.598+0000 7f783302ee40 20 rados->read ofs=0 len=0 2023-10-05T19:28:35.604 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.598+0000 7f783302ee40 1 -- 172.21.15.167:0/2022643642 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:40 5.c 5:30d299a5:::zonegroup_info.4f1f719d-9648-417b-b10c-e5d939825419:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x5570fe6d93a0 con 0x7f7824007030 2023-10-05T19:28:35.604 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.598+0000 7f7831364700 1 -- 172.21.15.167:0/2022643642 <== osd.0 v2:172.21.15.118:6800/27690 10 ==== osd_op_reply(40 zonegroup_info.4f1f719d-9648-417b-b10c-e5d939825419 [read 0~362 out=362b] v0'0 uv1 ondisk = 0) v8 ==== 195+0+362 (crc 0 0 0) 0x7f782002f080 con 0x7f7824007030 2023-10-05T19:28:35.604 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.598+0000 7f783302ee40 20 rados_obj.operate() r=0 bl.length=362 2023-10-05T19:28:35.605 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.598+0000 7f783302ee40 20 rados->read ofs=0 len=0 2023-10-05T19:28:35.605 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.598+0000 7f783302ee40 1 -- 172.21.15.167:0/2022643642 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:41 5.13 5:cbbf6c60:::converted:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x5570fe6d93a0 con 0x5570fe6d6640 2023-10-05T19:28:35.606 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.598+0000 7f7830b63700 1 -- 172.21.15.167:0/2022643642 <== osd.3 v2:172.21.15.167:6808/30036 10 ==== osd_op_reply(41 converted [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 ==== 153+0+0 (crc 0 0 0) 0x7f7828025020 con 0x5570fe6d6640 2023-10-05T19:28:35.606 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.598+0000 7f783302ee40 20 rados_obj.operate() r=-2 bl.length=0 2023-10-05T19:28:35.606 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.598+0000 7f783302ee40 20 rados->read ofs=0 len=0 2023-10-05T19:28:35.606 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.598+0000 7f783302ee40 1 -- 172.21.15.167:0/2022643642 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:42 5.12 5:49953fa1:::default.realm:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x5570fe6d9750 con 0x5570fe6a5600 2023-10-05T19:28:35.606 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.598+0000 7f7831364700 1 -- 172.21.15.167:0/2022643642 <== osd.1 v2:172.21.15.167:6800/30035 16 ==== osd_op_reply(42 default.realm [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 ==== 157+0+0 (crc 0 0 0) 0x7f782002f080 con 0x5570fe6a5600 2023-10-05T19:28:35.607 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.598+0000 7f783302ee40 20 rados_obj.operate() r=-2 bl.length=0 2023-10-05T19:28:35.607 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.598+0000 7f783302ee40 20 rados->read ofs=0 len=0 2023-10-05T19:28:35.607 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.598+0000 7f783302ee40 1 -- 172.21.15.167:0/2022643642 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:43 5.f 5:f4c53578:::zonegroups_names.default:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x5570fe6d9b50 con 0x5570fe6a5600 2023-10-05T19:28:35.607 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.598+0000 7f7831364700 1 -- 172.21.15.167:0/2022643642 <== osd.1 v2:172.21.15.167:6800/30035 17 ==== osd_op_reply(43 zonegroups_names.default [read 0~46 out=46b] v0'0 uv2 ondisk = 0) v8 ==== 168+0+46 (crc 0 0 0) 0x7f782002f080 con 0x5570fe6a5600 2023-10-05T19:28:35.607 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.598+0000 7f783302ee40 20 rados_obj.operate() r=0 bl.length=46 2023-10-05T19:28:35.607 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.598+0000 7f783302ee40 1 -- 172.21.15.167:0/2022643642 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:44 5.0 5:00000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x5570fe6dab60 con 0x5570fe6d6640 2023-10-05T19:28:35.608 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.602+0000 7f7830b63700 1 -- 172.21.15.167:0/2022643642 <== osd.3 v2:172.21.15.167:6808/30036 11 ==== osd_op_reply(44 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f7828025020 con 0x5570fe6d6640 2023-10-05T19:28:35.608 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.602+0000 7f7830b63700 1 -- 172.21.15.167:0/2022643642 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:45 5.10 5:08000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x5570fe6dab60 con 0x7f782801fc20 2023-10-05T19:28:35.609 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.602+0000 7f7830362700 1 -- 172.21.15.167:0/2022643642 <== osd.2 v2:172.21.15.167:6816/30037 7 ==== osd_op_reply(45 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f7824012450 con 0x7f782801fc20 2023-10-05T19:28:35.609 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.602+0000 7f7830362700 1 -- 172.21.15.167:0/2022643642 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:46 5.8 5:10000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x5570fe6dab60 con 0x7f7824007030 2023-10-05T19:28:35.610 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.606+0000 7f7831364700 1 -- 172.21.15.167:0/2022643642 <== osd.0 v2:172.21.15.118:6800/27690 11 ==== osd_op_reply(46 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f782002f080 con 0x7f7824007030 2023-10-05T19:28:35.610 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.606+0000 7f7831364700 1 -- 172.21.15.167:0/2022643642 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:47 5.18 5:18000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x5570fe6d5ce0 con 0x5570fe6a5600 2023-10-05T19:28:35.611 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.606+0000 7f7831364700 1 -- 172.21.15.167:0/2022643642 <== osd.1 v2:172.21.15.167:6800/30035 18 ==== osd_op_reply(47 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f782002f080 con 0x5570fe6a5600 2023-10-05T19:28:35.611 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.606+0000 7f7831364700 1 -- 172.21.15.167:0/2022643642 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:48 5.4 5:20000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x5570fe6d5ce0 con 0x5570fe6a5600 2023-10-05T19:28:35.612 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.606+0000 7f7831364700 1 -- 172.21.15.167:0/2022643642 <== osd.1 v2:172.21.15.167:6800/30035 19 ==== osd_op_reply(48 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f782002f080 con 0x5570fe6a5600 2023-10-05T19:28:35.612 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.606+0000 7f7831364700 1 -- 172.21.15.167:0/2022643642 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:49 5.14 5:28000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x5570fe6d5ce0 con 0x7f7824007030 2023-10-05T19:28:35.612 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.606+0000 7f7831364700 1 -- 172.21.15.167:0/2022643642 <== osd.0 v2:172.21.15.118:6800/27690 12 ==== osd_op_reply(49 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f782002f080 con 0x7f7824007030 2023-10-05T19:28:35.613 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.606+0000 7f7831364700 1 -- 172.21.15.167:0/2022643642 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:50 5.c 5:30000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x5570fe6d5ce0 con 0x7f7824007030 2023-10-05T19:28:35.613 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.606+0000 7f7831364700 1 -- 172.21.15.167:0/2022643642 <== osd.0 v2:172.21.15.118:6800/27690 13 ==== osd_op_reply(50 [pgnls start_epoch 30 out=112b] v23'1 uv1 ondisk = 1) v8 ==== 144+0+112 (crc 0 0 0) 0x7f782002f080 con 0x7f7824007030 2023-10-05T19:28:35.613 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.606+0000 7f7831364700 1 -- 172.21.15.167:0/2022643642 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:51 5.1c 5:38000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x5570fe6d5ce0 con 0x7f782801fc20 2023-10-05T19:28:35.614 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.610+0000 7f7830362700 1 -- 172.21.15.167:0/2022643642 <== osd.2 v2:172.21.15.167:6816/30037 8 ==== osd_op_reply(51 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f7824012450 con 0x7f782801fc20 2023-10-05T19:28:35.615 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.610+0000 7f7830362700 1 -- 172.21.15.167:0/2022643642 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:52 5.2 5:40000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x5570fe6d5ce0 con 0x5570fe6d6640 2023-10-05T19:28:35.615 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.610+0000 7f7830b63700 1 -- 172.21.15.167:0/2022643642 <== osd.3 v2:172.21.15.167:6808/30036 12 ==== osd_op_reply(52 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f7828025020 con 0x5570fe6d6640 2023-10-05T19:28:35.615 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.610+0000 7f7830b63700 1 -- 172.21.15.167:0/2022643642 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:53 5.12 5:48000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x5570fe6d5ce0 con 0x5570fe6a5600 2023-10-05T19:28:35.616 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.610+0000 7f7831364700 1 -- 172.21.15.167:0/2022643642 <== osd.1 v2:172.21.15.167:6800/30035 20 ==== osd_op_reply(53 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f782002f080 con 0x5570fe6a5600 2023-10-05T19:28:35.616 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.610+0000 7f7831364700 1 -- 172.21.15.167:0/2022643642 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:54 5.a 5:50000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x5570fe6d5ce0 con 0x7f7824007030 2023-10-05T19:28:35.617 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.610+0000 7f7831364700 1 -- 172.21.15.167:0/2022643642 <== osd.0 v2:172.21.15.118:6800/27690 14 ==== osd_op_reply(54 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f782002f080 con 0x7f7824007030 2023-10-05T19:28:35.617 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.610+0000 7f7831364700 1 -- 172.21.15.167:0/2022643642 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:55 5.1a 5:58000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x5570fe6d5ce0 con 0x5570fe6a5600 2023-10-05T19:28:35.618 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.614+0000 7f7831364700 1 -- 172.21.15.167:0/2022643642 <== osd.1 v2:172.21.15.167:6800/30035 21 ==== osd_op_reply(55 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f782002f080 con 0x5570fe6a5600 2023-10-05T19:28:35.618 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.614+0000 7f7831364700 1 -- 172.21.15.167:0/2022643642 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:56 5.6 5:60000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x5570fe6d5ce0 con 0x7f782801fc20 2023-10-05T19:28:35.619 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.614+0000 7f7830362700 1 -- 172.21.15.167:0/2022643642 <== osd.2 v2:172.21.15.167:6816/30037 9 ==== osd_op_reply(56 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f7824012450 con 0x7f782801fc20 2023-10-05T19:28:35.619 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.614+0000 7f7830362700 1 -- 172.21.15.167:0/2022643642 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:57 5.16 5:68000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x5570fe6d5ce0 con 0x5570fe6d6640 2023-10-05T19:28:35.620 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.614+0000 7f7830b63700 1 -- 172.21.15.167:0/2022643642 <== osd.3 v2:172.21.15.167:6808/30036 13 ==== osd_op_reply(57 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f7828025020 con 0x5570fe6d6640 2023-10-05T19:28:35.620 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.614+0000 7f7830b63700 1 -- 172.21.15.167:0/2022643642 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:58 5.e 5:70000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x5570fe6d5ce0 con 0x5570fe6d6640 2023-10-05T19:28:35.621 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.614+0000 7f7830b63700 1 -- 172.21.15.167:0/2022643642 <== osd.3 v2:172.21.15.167:6808/30036 14 ==== osd_op_reply(58 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f7828025020 con 0x5570fe6d6640 2023-10-05T19:28:35.621 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.614+0000 7f7830b63700 1 -- 172.21.15.167:0/2022643642 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:59 5.1e 5:78000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x5570fe6d5ce0 con 0x7f7824007030 2023-10-05T19:28:35.622 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.618+0000 7f7831364700 1 -- 172.21.15.167:0/2022643642 <== osd.0 v2:172.21.15.118:6800/27690 15 ==== osd_op_reply(59 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f782002f080 con 0x7f7824007030 2023-10-05T19:28:35.622 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.618+0000 7f7831364700 1 -- 172.21.15.167:0/2022643642 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:60 5.1 5:80000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x5570fe6d5ce0 con 0x5570fe6d6640 2023-10-05T19:28:35.623 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.618+0000 7f7830b63700 1 -- 172.21.15.167:0/2022643642 <== osd.3 v2:172.21.15.167:6808/30036 15 ==== osd_op_reply(60 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f7828025020 con 0x5570fe6d6640 2023-10-05T19:28:35.623 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.618+0000 7f7830b63700 1 -- 172.21.15.167:0/2022643642 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:61 5.11 5:88000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x5570fe6d5ce0 con 0x5570fe6d6640 2023-10-05T19:28:35.624 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.618+0000 7f7830b63700 1 -- 172.21.15.167:0/2022643642 <== osd.3 v2:172.21.15.167:6808/30036 16 ==== osd_op_reply(61 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f7828025020 con 0x5570fe6d6640 2023-10-05T19:28:35.624 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.618+0000 7f7830b63700 1 -- 172.21.15.167:0/2022643642 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:62 5.9 5:90000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x5570fe6d5ce0 con 0x5570fe6a5600 2023-10-05T19:28:35.624 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.618+0000 7f7831364700 1 -- 172.21.15.167:0/2022643642 <== osd.1 v2:172.21.15.167:6800/30035 22 ==== osd_op_reply(62 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f782002f080 con 0x5570fe6a5600 2023-10-05T19:28:35.625 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.618+0000 7f7831364700 1 -- 172.21.15.167:0/2022643642 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:63 5.19 5:98000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x5570fe6d5ce0 con 0x5570fe6a5600 2023-10-05T19:28:35.625 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.618+0000 7f7831364700 1 -- 172.21.15.167:0/2022643642 <== osd.1 v2:172.21.15.167:6800/30035 23 ==== osd_op_reply(63 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f782002f080 con 0x5570fe6a5600 2023-10-05T19:28:35.625 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.618+0000 7f7831364700 1 -- 172.21.15.167:0/2022643642 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:64 5.5 5:a0000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x5570fe6d5ce0 con 0x7f7824007030 2023-10-05T19:28:35.626 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.622+0000 7f7831364700 1 -- 172.21.15.167:0/2022643642 <== osd.0 v2:172.21.15.118:6800/27690 16 ==== osd_op_reply(64 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f782002f080 con 0x7f7824007030 2023-10-05T19:28:35.626 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.622+0000 7f7831364700 1 -- 172.21.15.167:0/2022643642 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:65 5.15 5:a8000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x5570fe6d5ce0 con 0x5570fe6d6640 2023-10-05T19:28:35.627 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.622+0000 7f7830b63700 1 -- 172.21.15.167:0/2022643642 <== osd.3 v2:172.21.15.167:6808/30036 17 ==== osd_op_reply(65 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f7828025020 con 0x5570fe6d6640 2023-10-05T19:28:35.627 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.622+0000 7f7830b63700 1 -- 172.21.15.167:0/2022643642 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:66 5.d 5:b0000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x5570fe6d5ce0 con 0x7f782801fc20 2023-10-05T19:28:35.628 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.622+0000 7f7830362700 1 -- 172.21.15.167:0/2022643642 <== osd.2 v2:172.21.15.167:6816/30037 10 ==== osd_op_reply(66 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f7824012450 con 0x7f782801fc20 2023-10-05T19:28:35.628 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.622+0000 7f7830362700 1 -- 172.21.15.167:0/2022643642 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:67 5.1d 5:b8000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x5570fe6d5ce0 con 0x5570fe6a5600 2023-10-05T19:28:35.629 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.622+0000 7f7831364700 1 -- 172.21.15.167:0/2022643642 <== osd.1 v2:172.21.15.167:6800/30035 24 ==== osd_op_reply(67 [pgnls start_epoch 30 out=107b] v30'2 uv2 ondisk = 1) v8 ==== 144+0+107 (crc 0 0 0) 0x7f782002f080 con 0x5570fe6a5600 2023-10-05T19:28:35.629 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.622+0000 7f7831364700 1 -- 172.21.15.167:0/2022643642 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:68 5.3 5:c0000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x5570fe6d5ce0 con 0x7f7824007030 2023-10-05T19:28:35.630 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.626+0000 7f7831364700 1 -- 172.21.15.167:0/2022643642 <== osd.0 v2:172.21.15.118:6800/27690 17 ==== osd_op_reply(68 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f782002f080 con 0x7f7824007030 2023-10-05T19:28:35.630 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.626+0000 7f7831364700 1 -- 172.21.15.167:0/2022643642 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:69 5.13 5:c8000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x5570fe6d5ce0 con 0x5570fe6d6640 2023-10-05T19:28:35.631 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.626+0000 7f7830b63700 1 -- 172.21.15.167:0/2022643642 <== osd.3 v2:172.21.15.167:6808/30036 18 ==== osd_op_reply(69 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f7828025020 con 0x5570fe6d6640 2023-10-05T19:28:35.631 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.626+0000 7f7830b63700 1 -- 172.21.15.167:0/2022643642 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:70 5.b 5:d0000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x5570fe6d5ce0 con 0x7f7824007030 2023-10-05T19:28:35.632 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.626+0000 7f7831364700 1 -- 172.21.15.167:0/2022643642 <== osd.0 v2:172.21.15.118:6800/27690 18 ==== osd_op_reply(70 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f782002f080 con 0x7f7824007030 2023-10-05T19:28:35.632 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.626+0000 7f7831364700 1 -- 172.21.15.167:0/2022643642 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:71 5.1b 5:d8000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x5570fe6d5ce0 con 0x7f782801fc20 2023-10-05T19:28:35.633 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.626+0000 7f7830362700 1 -- 172.21.15.167:0/2022643642 <== osd.2 v2:172.21.15.167:6816/30037 11 ==== osd_op_reply(71 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f7824012450 con 0x7f782801fc20 2023-10-05T19:28:35.633 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.626+0000 7f7830362700 1 -- 172.21.15.167:0/2022643642 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:72 5.7 5:e0000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x5570fe6d5ce0 con 0x5570fe6d6640 2023-10-05T19:28:35.634 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.626+0000 7f7830b63700 1 -- 172.21.15.167:0/2022643642 <== osd.3 v2:172.21.15.167:6808/30036 19 ==== osd_op_reply(72 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f7828025020 con 0x5570fe6d6640 2023-10-05T19:28:35.634 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.626+0000 7f7830b63700 1 -- 172.21.15.167:0/2022643642 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:73 5.17 5:e8000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x5570fe6d5ce0 con 0x7f7824007030 2023-10-05T19:28:35.635 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.630+0000 7f7831364700 1 -- 172.21.15.167:0/2022643642 <== osd.0 v2:172.21.15.118:6800/27690 19 ==== osd_op_reply(73 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f782002f080 con 0x7f7824007030 2023-10-05T19:28:35.635 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.630+0000 7f7831364700 1 -- 172.21.15.167:0/2022643642 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:74 5.f 5:f0000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x5570fe6d5ce0 con 0x5570fe6a5600 2023-10-05T19:28:35.636 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.630+0000 7f7831364700 1 -- 172.21.15.167:0/2022643642 <== osd.1 v2:172.21.15.167:6800/30035 25 ==== osd_op_reply(74 [pgnls start_epoch 30 out=115b] v23'2 uv2 ondisk = 1) v8 ==== 144+0+115 (crc 0 0 0) 0x7f782002f080 con 0x5570fe6a5600 2023-10-05T19:28:35.636 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.630+0000 7f7831364700 1 -- 172.21.15.167:0/2022643642 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:75 5.1f 5:f8000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x5570fe6d5ce0 con 0x7f782801fc20 2023-10-05T19:28:35.637 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.630+0000 7f7830362700 1 -- 172.21.15.167:0/2022643642 <== osd.2 v2:172.21.15.167:6816/30037 12 ==== osd_op_reply(75 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f7824012450 con 0x7f782801fc20 2023-10-05T19:28:35.637 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.630+0000 7f783302ee40 20 RGWRados::pool_iterate: got zonegroup_info.4f1f719d-9648-417b-b10c-e5d939825419 2023-10-05T19:28:35.637 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.630+0000 7f783302ee40 20 RGWRados::pool_iterate: got zone_info.9a9df965-6612-45d4-a45f-6e3c889bacc2 2023-10-05T19:28:35.637 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.630+0000 7f783302ee40 20 RGWRados::pool_iterate: got zone_names.default 2023-10-05T19:28:35.638 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.630+0000 7f783302ee40 20 RGWRados::pool_iterate: got zonegroups_names.default 2023-10-05T19:28:35.638 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.630+0000 7f783302ee40 20 rados->read ofs=0 len=0 2023-10-05T19:28:35.638 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.630+0000 7f783302ee40 1 -- 172.21.15.167:0/2022643642 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:76 5.f 5:f43b6ece:::zone_names.default:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x5570fe6d9b50 con 0x5570fe6a5600 2023-10-05T19:28:35.639 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.630+0000 7f7831364700 1 -- 172.21.15.167:0/2022643642 <== osd.1 v2:172.21.15.167:6800/30035 26 ==== osd_op_reply(76 zone_names.default [read 0~46 out=46b] v0'0 uv1 ondisk = 0) v8 ==== 162+0+46 (crc 0 0 0) 0x7f782002f080 con 0x5570fe6a5600 2023-10-05T19:28:35.639 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.630+0000 7f783302ee40 20 rados_obj.operate() r=0 bl.length=46 2023-10-05T19:28:35.639 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.634+0000 7f783302ee40 20 rados->read ofs=0 len=0 2023-10-05T19:28:35.639 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.634+0000 7f783302ee40 1 -- 172.21.15.167:0/2022643642 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:77 5.1d 5:b8374eb4:::zone_info.9a9df965-6612-45d4-a45f-6e3c889bacc2:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x5570fe6d9f00 con 0x5570fe6a5600 2023-10-05T19:28:35.639 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.634+0000 7f7831364700 1 -- 172.21.15.167:0/2022643642 <== osd.1 v2:172.21.15.167:6800/30035 27 ==== osd_op_reply(77 zone_info.9a9df965-6612-45d4-a45f-6e3c889bacc2 [read 0~874 out=874b] v0'0 uv2 ondisk = 0) v8 ==== 190+0+874 (crc 0 0 0) 0x7f782002f080 con 0x5570fe6a5600 2023-10-05T19:28:35.640 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.634+0000 7f783302ee40 20 rados_obj.operate() r=0 bl.length=874 2023-10-05T19:28:35.640 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.634+0000 7f783302ee40 20 rados->read ofs=0 len=0 2023-10-05T19:28:35.640 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.634+0000 7f783302ee40 1 -- 172.21.15.167:0/2022643642 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:78 5.f 5:f4c53578:::zonegroups_names.default:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x5570fe6d9f00 con 0x5570fe6a5600 2023-10-05T19:28:35.641 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.634+0000 7f7831364700 1 -- 172.21.15.167:0/2022643642 <== osd.1 v2:172.21.15.167:6800/30035 28 ==== osd_op_reply(78 zonegroups_names.default [read 0~46 out=46b] v0'0 uv2 ondisk = 0) v8 ==== 168+0+46 (crc 0 0 0) 0x7f782002f080 con 0x5570fe6a5600 2023-10-05T19:28:35.641 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.634+0000 7f783302ee40 20 rados_obj.operate() r=0 bl.length=46 2023-10-05T19:28:35.641 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.634+0000 7f783302ee40 20 rados->read ofs=0 len=0 2023-10-05T19:28:35.641 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.634+0000 7f783302ee40 1 -- 172.21.15.167:0/2022643642 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:79 5.c 5:30d299a5:::zonegroup_info.4f1f719d-9648-417b-b10c-e5d939825419:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x5570fe6d9f00 con 0x7f7824007030 2023-10-05T19:28:35.642 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.634+0000 7f7831364700 1 -- 172.21.15.167:0/2022643642 <== osd.0 v2:172.21.15.118:6800/27690 20 ==== osd_op_reply(79 zonegroup_info.4f1f719d-9648-417b-b10c-e5d939825419 [read 0~362 out=362b] v0'0 uv1 ondisk = 0) v8 ==== 195+0+362 (crc 0 0 0) 0x7f782002f080 con 0x7f7824007030 2023-10-05T19:28:35.642 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.634+0000 7f783302ee40 20 rados_obj.operate() r=0 bl.length=362 2023-10-05T19:28:35.642 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.634+0000 7f783302ee40 20 rados->read ofs=0 len=0 2023-10-05T19:28:35.642 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.634+0000 7f783302ee40 1 -- 172.21.15.167:0/2022643642 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:80 5.3 5:c0ad8cbd:::region_map:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x5570fe6da350 con 0x7f7824007030 2023-10-05T19:28:35.644 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.634+0000 7f7831364700 1 -- 172.21.15.167:0/2022643642 <== osd.0 v2:172.21.15.118:6800/27690 21 ==== osd_op_reply(80 region_map [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 ==== 154+0+0 (crc 0 0 0) 0x7f7820050040 con 0x7f7824007030 2023-10-05T19:28:35.644 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.634+0000 7f783302ee40 20 rados_obj.operate() r=-2 bl.length=0 2023-10-05T19:28:35.644 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.634+0000 7f783302ee40 20 zone default found 2023-10-05T19:28:35.644 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.634+0000 7f783302ee40 4 Realm: () 2023-10-05T19:28:35.644 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.634+0000 7f783302ee40 4 ZoneGroup: default (4f1f719d-9648-417b-b10c-e5d939825419) 2023-10-05T19:28:35.644 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.634+0000 7f783302ee40 4 Zone: default (9a9df965-6612-45d4-a45f-6e3c889bacc2) 2023-10-05T19:28:35.644 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.634+0000 7f783302ee40 10 cannot find current period zonegroup using local zonegroup configuration 2023-10-05T19:28:35.644 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.634+0000 7f783302ee40 20 zonegroup default 2023-10-05T19:28:35.644 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.634+0000 7f783302ee40 20 rados->read ofs=0 len=0 2023-10-05T19:28:35.645 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.634+0000 7f783302ee40 1 -- 172.21.15.167:0/2022643642 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:81 5.3 5:c52100b6:::period_config.default:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x5570fe6da350 con 0x7f7824007030 2023-10-05T19:28:35.645 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.638+0000 7f7831364700 1 -- 172.21.15.167:0/2022643642 <== osd.0 v2:172.21.15.118:6800/27690 22 ==== osd_op_reply(81 period_config.default [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 ==== 165+0+0 (crc 0 0 0) 0x7f7820050040 con 0x7f7824007030 2023-10-05T19:28:35.645 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.638+0000 7f783302ee40 20 rados_obj.operate() r=-2 bl.length=0 2023-10-05T19:28:35.645 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.638+0000 7f783302ee40 20 rados->read ofs=0 len=0 2023-10-05T19:28:35.645 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.638+0000 7f783302ee40 1 -- 172.21.15.167:0/2022643642 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:82 6.2 6:4347d321:::bucket.sync-source-hints.:head [call version.read in=11b,read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x5570fe6dbaa0 con 0x5570fe6d6640 2023-10-05T19:28:35.645 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.638+0000 7f7830b63700 1 -- 172.21.15.167:0/2022643642 <== osd.3 v2:172.21.15.167:6808/30036 20 ==== osd_op_reply(82 bucket.sync-source-hints. [call,read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 ==== 211+0+0 (crc 0 0 0) 0x7f7828025020 con 0x5570fe6d6640 2023-10-05T19:28:35.645 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.638+0000 7f783302ee40 20 rados_obj.operate() r=-2 bl.length=0 2023-10-05T19:28:35.645 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.638+0000 7f783302ee40 20 rados->read ofs=0 len=0 2023-10-05T19:28:35.645 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.638+0000 7f783302ee40 1 -- 172.21.15.167:0/2022643642 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:83 6.b 6:d467b91b:::bucket.sync-target-hints.:head [call version.read in=11b,read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x5570fe6dbe50 con 0x5570fe6d6640 2023-10-05T19:28:35.646 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.638+0000 7f7830b63700 1 -- 172.21.15.167:0/2022643642 <== osd.3 v2:172.21.15.167:6808/30036 21 ==== osd_op_reply(83 bucket.sync-target-hints. [call,read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 ==== 211+0+0 (crc 0 0 0) 0x7f7828025020 con 0x5570fe6d6640 2023-10-05T19:28:35.646 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.638+0000 7f783302ee40 20 rados_obj.operate() r=-2 bl.length=0 2023-10-05T19:28:35.646 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.638+0000 7f783302ee40 20 started sync module instance, tier type = 2023-10-05T19:28:35.647 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.638+0000 7f783302ee40 20 started zone id=9a9df965-6612-45d4-a45f-6e3c889bacc2 (name=default) with tier type = 2023-10-05T19:28:35.647 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.638+0000 7f783302ee40 1 -- 172.21.15.167:0/2022643642 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:84 7.1f 7:f95f44c2:::notify.0:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x5570fe6db160 con 0x7f782801fc20 2023-10-05T19:28:35.647 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.642+0000 7f7830362700 1 -- 172.21.15.167:0/2022643642 <== osd.2 v2:172.21.15.167:6816/30037 13 ==== osd_op_reply(84 notify.0 [create] v30'144 uv144 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f7824012450 con 0x7f782801fc20 2023-10-05T19:28:35.647 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.642+0000 7f783302ee40 1 -- 172.21.15.167:0/2022643642 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:85 7.1f 7:f95f44c2:::notify.0:head [watch watch cookie 93943793300832] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x5570fe6d5910 con 0x7f782801fc20 2023-10-05T19:28:35.648 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.642+0000 7f783302ee40 1 -- 172.21.15.167:0/2022643642 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:86 7.0 7:05bf5b68:::notify.1:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x5570fe6dcc70 con 0x5570fe6a5600 2023-10-05T19:28:35.650 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.646+0000 7f7830362700 1 -- 172.21.15.167:0/2022643642 <== osd.2 v2:172.21.15.167:6816/30037 14 ==== osd_op_reply(85 notify.0 [watch watch cookie 93943793300832] v30'145 uv144 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f7824012450 con 0x7f782801fc20 2023-10-05T19:28:35.651 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.646+0000 7f7831364700 1 -- 172.21.15.167:0/2022643642 <== osd.1 v2:172.21.15.167:6800/30035 29 ==== osd_op_reply(86 notify.1 [create] v30'144 uv144 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f7820050040 con 0x5570fe6a5600 2023-10-05T19:28:35.651 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.646+0000 7f783302ee40 1 -- 172.21.15.167:0/2022643642 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:87 7.0 7:05bf5b68:::notify.1:head [watch watch cookie 93943793305472] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x5570fe6dc360 con 0x5570fe6a5600 2023-10-05T19:28:35.651 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.646+0000 7f783302ee40 1 -- 172.21.15.167:0/2022643642 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:88 7.15 7:a93a5511:::notify.2:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x5570fe6dda50 con 0x5570fe6d6640 2023-10-05T19:28:35.653 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.646+0000 7f7831364700 1 -- 172.21.15.167:0/2022643642 <== osd.1 v2:172.21.15.167:6800/30035 30 ==== osd_op_reply(87 notify.1 [watch watch cookie 93943793305472] v30'145 uv144 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f7820050040 con 0x5570fe6a5600 2023-10-05T19:28:35.654 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.650+0000 7f7830b63700 1 -- 172.21.15.167:0/2022643642 <== osd.3 v2:172.21.15.167:6808/30036 22 ==== osd_op_reply(88 notify.2 [create] v30'144 uv144 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f7828025020 con 0x5570fe6d6640 2023-10-05T19:28:35.654 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.650+0000 7f783302ee40 1 -- 172.21.15.167:0/2022643642 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:89 7.15 7:a93a5511:::notify.2:head [watch watch cookie 93943793307760] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x5570fe6dde00 con 0x5570fe6d6640 2023-10-05T19:28:35.654 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.650+0000 7f783302ee40 1 -- 172.21.15.167:0/2022643642 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:90 7.e 7:7759931f:::notify.3:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x5570fe6de850 con 0x7f782801fc20 2023-10-05T19:28:35.656 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.650+0000 7f7830362700 1 -- 172.21.15.167:0/2022643642 <== osd.2 v2:172.21.15.167:6816/30037 15 ==== osd_op_reply(90 notify.3 [create] v30'144 uv144 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f7824012450 con 0x7f782801fc20 2023-10-05T19:28:35.657 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.650+0000 7f783302ee40 1 -- 172.21.15.167:0/2022643642 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:91 7.e 7:7759931f:::notify.3:head [watch watch cookie 93943793314896] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x5570fe6df590 con 0x7f782801fc20 2023-10-05T19:28:35.657 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.650+0000 7f783302ee40 1 -- 172.21.15.167:0/2022643642 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:92 7.d 7:b4812045:::notify.4:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x5570fe6dffa0 con 0x5570fe6a5600 2023-10-05T19:28:35.657 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.650+0000 7f7830b63700 1 -- 172.21.15.167:0/2022643642 <== osd.3 v2:172.21.15.167:6808/30036 23 ==== osd_op_reply(89 notify.2 [watch watch cookie 93943793307760] v30'145 uv144 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f7828025020 con 0x5570fe6d6640 2023-10-05T19:28:35.658 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.654+0000 7f7830362700 1 -- 172.21.15.167:0/2022643642 <== osd.2 v2:172.21.15.167:6816/30037 16 ==== osd_op_reply(91 notify.3 [watch watch cookie 93943793314896] v30'145 uv144 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f7824012450 con 0x7f782801fc20 2023-10-05T19:28:35.659 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.654+0000 7f7831364700 1 -- 172.21.15.167:0/2022643642 <== osd.1 v2:172.21.15.167:6800/30035 31 ==== osd_op_reply(92 notify.4 [create] v30'144 uv144 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f7820050040 con 0x5570fe6a5600 2023-10-05T19:28:35.660 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.654+0000 7f783302ee40 1 -- 172.21.15.167:0/2022643642 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:93 7.d 7:b4812045:::notify.4:head [watch watch cookie 93943793308864] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x5570fe6dffa0 con 0x5570fe6a5600 2023-10-05T19:28:35.660 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.654+0000 7f783302ee40 1 -- 172.21.15.167:0/2022643642 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:94 7.3 7:c609908c:::notify.5:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x5570fe6e0840 con 0x7f7824007030 2023-10-05T19:28:35.662 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.658+0000 7f7831364700 1 -- 172.21.15.167:0/2022643642 <== osd.1 v2:172.21.15.167:6800/30035 32 ==== osd_op_reply(93 notify.4 [watch watch cookie 93943793308864] v30'145 uv144 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f7820050040 con 0x5570fe6a5600 2023-10-05T19:28:35.663 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.658+0000 7f7831364700 1 -- 172.21.15.167:0/2022643642 <== osd.0 v2:172.21.15.118:6800/27690 23 ==== osd_op_reply(94 notify.5 [create] v30'144 uv144 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f7820050040 con 0x7f7824007030 2023-10-05T19:28:35.663 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.658+0000 7f783302ee40 1 -- 172.21.15.167:0/2022643642 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:95 7.3 7:c609908c:::notify.5:head [watch watch cookie 93943793318848] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x5570fe6e0500 con 0x7f7824007030 2023-10-05T19:28:35.663 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.658+0000 7f783302ee40 1 -- 172.21.15.167:0/2022643642 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:96 7.14 7:2b04a3e9:::notify.6:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x5570fe6e1010 con 0x7f7824007030 2023-10-05T19:28:35.666 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.662+0000 7f7831364700 1 -- 172.21.15.167:0/2022643642 <== osd.0 v2:172.21.15.118:6800/27690 24 ==== osd_op_reply(95 notify.5 [watch watch cookie 93943793318848] v30'145 uv144 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f7820050040 con 0x7f7824007030 2023-10-05T19:28:35.666 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.662+0000 7f7831364700 1 -- 172.21.15.167:0/2022643642 <== osd.0 v2:172.21.15.118:6800/27690 25 ==== osd_op_reply(96 notify.6 [create] v30'144 uv144 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f7820050040 con 0x7f7824007030 2023-10-05T19:28:35.666 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.662+0000 7f783302ee40 1 -- 172.21.15.167:0/2022643642 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:97 7.14 7:2b04a3e9:::notify.6:head [watch watch cookie 93943793322720] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x5570fe6e0500 con 0x7f7824007030 2023-10-05T19:28:35.667 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.662+0000 7f783302ee40 1 -- 172.21.15.167:0/2022643642 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:98 7.9 7:93e5b521:::notify.7:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x5570fe6e1a70 con 0x7f7824007030 2023-10-05T19:28:35.669 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.662+0000 7f7831364700 1 -- 172.21.15.167:0/2022643642 <== osd.0 v2:172.21.15.118:6800/27690 26 ==== osd_op_reply(98 notify.7 [create] v30'144 uv144 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f7820050040 con 0x7f7824007030 2023-10-05T19:28:35.669 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.662+0000 7f7831364700 1 -- 172.21.15.167:0/2022643642 <== osd.0 v2:172.21.15.118:6800/27690 27 ==== osd_op_reply(97 notify.6 [watch watch cookie 93943793322720] v30'145 uv144 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f7820050040 con 0x7f7824007030 2023-10-05T19:28:35.669 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.662+0000 7f783302ee40 1 -- 172.21.15.167:0/2022643642 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:99 7.9 7:93e5b521:::notify.7:head [watch watch cookie 93943793325440] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x5570fe6e1ec0 con 0x7f7824007030 2023-10-05T19:28:35.669 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.662+0000 7f783302ee40 20 add_watcher() i=0 2023-10-05T19:28:35.670 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.662+0000 7f783302ee40 20 add_watcher() i=1 2023-10-05T19:28:35.670 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.662+0000 7f783302ee40 20 add_watcher() i=2 2023-10-05T19:28:35.670 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.662+0000 7f783302ee40 20 add_watcher() i=3 2023-10-05T19:28:35.670 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.662+0000 7f783302ee40 20 add_watcher() i=4 2023-10-05T19:28:35.670 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.662+0000 7f783302ee40 20 add_watcher() i=5 2023-10-05T19:28:35.670 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.662+0000 7f783302ee40 20 add_watcher() i=6 2023-10-05T19:28:35.671 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.666+0000 7f7831364700 1 -- 172.21.15.167:0/2022643642 <== osd.0 v2:172.21.15.118:6800/27690 28 ==== osd_op_reply(99 notify.7 [watch watch cookie 93943793325440] v30'145 uv144 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f7820050040 con 0x7f7824007030 2023-10-05T19:28:35.671 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.666+0000 7f783302ee40 20 add_watcher() i=7 2023-10-05T19:28:35.671 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.666+0000 7f783302ee40 2 all 8 watchers are set, enabling cache 2023-10-05T19:28:35.671 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.666+0000 7f783302ee40 1 -- 172.21.15.167:0/2022643642 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:100 6.4 6:22d26bf9:::data_loggenerations_metadata:head [call version.check_conds in=74b,call version.read in=11b,read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x5570fe6e4ef0 con 0x5570fe6a5600 2023-10-05T19:28:35.672 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.666+0000 7f7831364700 1 -- 172.21.15.167:0/2022643642 <== osd.1 v2:172.21.15.167:6800/30035 33 ==== osd_op_reply(100 data_loggenerations_metadata [call,call out=48b,read 0~28 out=28b] v0'0 uv1 ondisk = 0) v8 ==== 256+0+76 (crc 0 0 0) 0x7f7820050040 con 0x5570fe6a5600 2023-10-05T19:28:35.672 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.666+0000 7f783302ee40 1 -- 172.21.15.167:0/2022643642 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:101 6.4 6:22d26bf9:::data_loggenerations_metadata:head [watch watch cookie 93943793326544] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x5570fe6e2310 con 0x5570fe6a5600 2023-10-05T19:28:35.675 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.670+0000 7f7831364700 1 -- 172.21.15.167:0/2022643642 <== osd.1 v2:172.21.15.167:6800/30035 34 ==== osd_op_reply(101 data_loggenerations_metadata [watch watch cookie 93943793326544] v30'140 uv1 ondisk = 0) v8 ==== 172+0+0 (crc 0 0 0) 0x7f7820050040 con 0x5570fe6a5600 2023-10-05T19:28:35.675 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.670+0000 7f78027e4700 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2023-10-05T19:28:35.675 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.670+0000 7f783302ee40 20 check_secure_mon_conn(): auth registy supported: methods=[2] modes=[2,1] 2023-10-05T19:28:35.675 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.670+0000 7f783302ee40 20 check_secure_mon_conn(): mode 1 is insecure 2023-10-05T19:28:35.676 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.670+0000 7f783302ee40 1 Processor -- start 2023-10-05T19:28:35.676 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.670+0000 7f783302ee40 1 -- start start 2023-10-05T19:28:35.676 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.670+0000 7f783302ee40 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5570fe6ef3d0 0x5570fe6ef7a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-05T19:28:35.676 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.670+0000 7f783302ee40 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x5570fe6efce0 con 0x5570fe6ef3d0 2023-10-05T19:28:35.677 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.670+0000 7f7830362700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5570fe6ef3d0 0x5570fe6ef7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-05T19:28:35.677 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.670+0000 7f7830362700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5570fe6ef3d0 0x5570fe6ef7a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.118:3300/0 says I am v2:172.21.15.167:41680/0 (socket says 172.21.15.167:41680) 2023-10-05T19:28:35.677 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.670+0000 7f7830362700 1 -- 172.21.15.167:0/4047708976 learned_addr learned my addr 172.21.15.167:0/4047708976 (peer_addr_for_me v2:172.21.15.167:0/0) 2023-10-05T19:28:35.677 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.670+0000 7f7830362700 1 -- 172.21.15.167:0/4047708976 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5570fe6efe20 con 0x5570fe6ef3d0 2023-10-05T19:28:35.678 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.670+0000 7f7830362700 1 --2- 172.21.15.167:0/4047708976 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5570fe6ef3d0 0x5570fe6ef7a0 secure :-1 s=READY pgs=426 cs=0 l=1 rev1=1 crypto rx=0x5570fe6a20a0 tx=0x7f7824012000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d4a0c0a3390ec45b server_cookie=0 in_seq=0 out_seq=0 2023-10-05T19:28:35.678 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.670+0000 7f7801fe3700 1 -- 172.21.15.167:0/4047708976 <== mon.0 v2:172.21.15.118:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f782402a540 con 0x5570fe6ef3d0 2023-10-05T19:28:35.678 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.670+0000 7f7801fe3700 1 -- 172.21.15.167:0/4047708976 <== mon.0 v2:172.21.15.118:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f782402a6a0 con 0x5570fe6ef3d0 2023-10-05T19:28:35.678 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.670+0000 7f7801fe3700 1 -- 172.21.15.167:0/4047708976 <== mon.0 v2:172.21.15.118:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f782402a850 con 0x5570fe6ef3d0 2023-10-05T19:28:35.679 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.674+0000 7f783302ee40 1 -- 172.21.15.167:0/4047708976 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5570fe6ef3d0 msgr2=0x5570fe6ef7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-05T19:28:35.679 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.674+0000 7f783302ee40 1 --2- 172.21.15.167:0/4047708976 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5570fe6ef3d0 0x5570fe6ef7a0 secure :-1 s=READY pgs=426 cs=0 l=1 rev1=1 crypto rx=0x5570fe6a20a0 tx=0x7f7824012000 comp rx=0 tx=0).stop 2023-10-05T19:28:35.679 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.674+0000 7f783302ee40 1 -- 172.21.15.167:0/4047708976 shutdown_connections 2023-10-05T19:28:35.680 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.674+0000 7f783302ee40 1 --2- 172.21.15.167:0/4047708976 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5570fe6ef3d0 0x5570fe6ef7a0 unknown :-1 s=CLOSED pgs=426 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-05T19:28:35.680 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.674+0000 7f783302ee40 1 -- 172.21.15.167:0/4047708976 >> 172.21.15.167:0/4047708976 conn(0x5570fe6ea790 msgr2=0x5570fe6e88e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-05T19:28:35.680 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.674+0000 7f783302ee40 1 -- 172.21.15.167:0/4047708976 shutdown_connections 2023-10-05T19:28:35.680 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.674+0000 7f783302ee40 1 -- 172.21.15.167:0/4047708976 wait complete. 2023-10-05T19:28:35.680 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.674+0000 7f783302ee40 1 Processor -- start 2023-10-05T19:28:35.680 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.674+0000 7f783302ee40 1 -- start start 2023-10-05T19:28:35.681 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.674+0000 7f783302ee40 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5570fe6ef3d0 0x5570fe6f4f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-05T19:28:35.681 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.674+0000 7f783302ee40 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x5570fe6f5f00 con 0x5570fe6ef3d0 2023-10-05T19:28:35.681 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.674+0000 7f7830362700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5570fe6ef3d0 0x5570fe6f4f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-05T19:28:35.682 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.674+0000 7f7830362700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5570fe6ef3d0 0x5570fe6f4f00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.118:3300/0 says I am v2:172.21.15.167:41692/0 (socket says 172.21.15.167:41692) 2023-10-05T19:28:35.682 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.674+0000 7f7830362700 1 -- 172.21.15.167:0/3592677366 learned_addr learned my addr 172.21.15.167:0/3592677366 (peer_addr_for_me v2:172.21.15.167:0/0) 2023-10-05T19:28:35.682 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.674+0000 7f7830362700 1 -- 172.21.15.167:0/3592677366 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5570fe6a1c30 con 0x5570fe6ef3d0 2023-10-05T19:28:35.682 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.674+0000 7f7830362700 1 --2- 172.21.15.167:0/3592677366 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5570fe6ef3d0 0x5570fe6f4f00 secure :-1 s=READY pgs=427 cs=0 l=1 rev1=1 crypto rx=0x5570fe6e9450 tx=0x7f782402f000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-05T19:28:35.683 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.674+0000 7f7773fff700 1 -- 172.21.15.167:0/3592677366 <== mon.0 v2:172.21.15.118:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f782403e910 con 0x5570fe6ef3d0 2023-10-05T19:28:35.683 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.678+0000 7f783302ee40 1 -- 172.21.15.167:0/3592677366 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x5570fe6f5500 con 0x5570fe6ef3d0 2023-10-05T19:28:35.683 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.678+0000 7f7773fff700 1 -- 172.21.15.167:0/3592677366 <== mon.0 v2:172.21.15.118:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f782403ea70 con 0x5570fe6ef3d0 2023-10-05T19:28:35.683 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.678+0000 7f7773fff700 1 -- 172.21.15.167:0/3592677366 <== mon.0 v2:172.21.15.118:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f782403ec60 con 0x5570fe6ef3d0 2023-10-05T19:28:35.683 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.678+0000 7f783302ee40 1 -- 172.21.15.167:0/3592677366 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x5570fe6f59a0 con 0x5570fe6ef3d0 2023-10-05T19:28:35.684 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.678+0000 7f7773fff700 1 -- 172.21.15.167:0/3592677366 <== mon.0 v2:172.21.15.118:3300/0 4 ==== mgrmap(e 5) v1 ==== 40329+0+0 (secure 0 0 0) 0x7f7824046070 con 0x5570fe6ef3d0 2023-10-05T19:28:35.684 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.678+0000 7f7773fff700 1 --2- 172.21.15.167:0/3592677366 >> [v2:172.21.15.167:6824/29889,v1:172.21.15.167:6825/29889] conn(0x7f7778030c00 0x7f77780330b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-05T19:28:35.685 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.678+0000 7f7773fff700 1 -- 172.21.15.167:0/3592677366 <== mon.0 v2:172.21.15.118:3300/0 5 ==== osd_map(30..30 src has 1..30) v4 ==== 6057+0+0 (secure 0 0 0) 0x7f782402f070 con 0x5570fe6ef3d0 2023-10-05T19:28:35.685 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.678+0000 7f7831364700 1 --2- 172.21.15.167:0/3592677366 >> [v2:172.21.15.167:6824/29889,v1:172.21.15.167:6825/29889] conn(0x7f7778030c00 0x7f77780330b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-05T19:28:35.685 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.678+0000 7f783302ee40 1 --2- 172.21.15.167:0/3592677366 >> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] conn(0x5570fe6ffaf0 0x5570fe701f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-05T19:28:35.685 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.678+0000 7f783302ee40 1 -- 172.21.15.167:0/3592677366 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:1 6.19 6:9a6dd8c0:gc::gc.0:head [create,call version.check_conds in=74b,call rgw_gc.rgw_gc_queue_init in=45b,call version.set in=34b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x5570fe7025a0 con 0x5570fe6ffaf0 2023-10-05T19:28:35.685 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.678+0000 7f7830b63700 1 --2- 172.21.15.167:0/3592677366 >> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] conn(0x5570fe6ffaf0 0x5570fe701f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-05T19:28:35.686 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.678+0000 7f7831364700 1 --2- 172.21.15.167:0/3592677366 >> [v2:172.21.15.167:6824/29889,v1:172.21.15.167:6825/29889] conn(0x7f7778030c00 0x7f77780330b0 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x5570fe6e02d0 tx=0x7f7820000f90 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-05T19:28:35.686 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.678+0000 7f7830b63700 1 --2- 172.21.15.167:0/3592677366 >> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] conn(0x5570fe6ffaf0 0x5570fe701f00 crc :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-05T19:28:35.688 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.682+0000 7f7830b63700 1 -- 172.21.15.167:0/3592677366 <== osd.1 v2:172.21.15.167:6800/30035 1 ==== osd_op_reply(1 gc.0 [create,call,call,call] v30'28 uv3 ondisk = -125 ((125) Operation canceled)) v8 ==== 274+0+0 (crc 0 0 0) 0x7f7828025020 con 0x5570fe6ffaf0 2023-10-05T19:28:35.688 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.682+0000 7f783302ee40 1 --2- 172.21.15.167:0/3592677366 >> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] conn(0x5570fe6fb730 0x5570fe705c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-05T19:28:35.688 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.682+0000 7f783302ee40 1 -- 172.21.15.167:0/3592677366 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:2 6.18 6:1e85421f:gc::gc.1:head [create,call version.check_conds in=74b,call rgw_gc.rgw_gc_queue_init in=45b,call version.set in=34b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x5570fe706150 con 0x5570fe6fb730 2023-10-05T19:28:35.688 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.682+0000 7f7830362700 1 --2- 172.21.15.167:0/3592677366 >> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] conn(0x5570fe6fb730 0x5570fe705c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-05T19:28:35.689 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.682+0000 7f7830362700 1 --2- 172.21.15.167:0/3592677366 >> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] conn(0x5570fe6fb730 0x5570fe705c10 crc :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-05T19:28:35.692 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.686+0000 7f7830362700 1 -- 172.21.15.167:0/3592677366 <== osd.0 v2:172.21.15.118:6800/27690 1 ==== osd_op_reply(2 gc.1 [create,call,call,call] v30'20 uv5 ondisk = -125 ((125) Operation canceled)) v8 ==== 274+0+0 (crc 0 0 0) 0x7f7824058040 con 0x5570fe6fb730 2023-10-05T19:28:35.692 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.686+0000 7f783302ee40 1 --2- 172.21.15.167:0/3592677366 >> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] conn(0x5570fe706e20 0x5570fe709230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-05T19:28:35.692 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.686+0000 7f783302ee40 1 -- 172.21.15.167:0/3592677366 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:3 6.b 6:d242335b:gc::gc.2:head [create,call version.check_conds in=74b,call rgw_gc.rgw_gc_queue_init in=45b,call version.set in=34b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x5570fe709960 con 0x5570fe706e20 2023-10-05T19:28:35.692 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.686+0000 7f7831364700 1 --2- 172.21.15.167:0/3592677366 >> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] conn(0x5570fe706e20 0x5570fe709230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-05T19:28:35.693 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.686+0000 7f7831364700 1 --2- 172.21.15.167:0/3592677366 >> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] conn(0x5570fe706e20 0x5570fe709230 crc :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-05T19:28:35.695 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.690+0000 7f7831364700 1 -- 172.21.15.167:0/3592677366 <== osd.3 v2:172.21.15.167:6808/30036 1 ==== osd_op_reply(3 gc.2 [create,call,call,call] v30'17 uv3 ondisk = -125 ((125) Operation canceled)) v8 ==== 274+0+0 (crc 0 0 0) 0x7f7820050040 con 0x5570fe706e20 2023-10-05T19:28:35.696 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.690+0000 7f783302ee40 1 -- 172.21.15.167:0/3592677366 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:4 6.4 6:2166d8e1:gc::gc.3:head [create,call version.check_conds in=74b,call rgw_gc.rgw_gc_queue_init in=45b,call version.set in=34b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x5570fe709fb0 con 0x5570fe6ffaf0 2023-10-05T19:28:35.698 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.694+0000 7f7830b63700 1 -- 172.21.15.167:0/3592677366 <== osd.1 v2:172.21.15.167:6800/30035 2 ==== osd_op_reply(4 gc.3 [create,call,call,call] v30'141 uv7 ondisk = -125 ((125) Operation canceled)) v8 ==== 274+0+0 (crc 0 0 0) 0x7f7828025020 con 0x5570fe6ffaf0 2023-10-05T19:28:35.698 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.694+0000 7f783302ee40 1 --2- 172.21.15.167:0/3592677366 >> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] conn(0x5570fe70afd0 0x5570fe70d3b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-05T19:28:35.699 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.694+0000 7f783302ee40 1 -- 172.21.15.167:0/3592677366 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:5 6.3 6:c60da066:gc::gc.4:head [create,call version.check_conds in=74b,call rgw_gc.rgw_gc_queue_init in=45b,call version.set in=34b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x5570fe70da60 con 0x5570fe70afd0 2023-10-05T19:28:35.699 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.694+0000 7f7830b63700 1 --2- 172.21.15.167:0/3592677366 >> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] conn(0x5570fe70afd0 0x5570fe70d3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-05T19:28:35.699 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.694+0000 7f7830b63700 1 --2- 172.21.15.167:0/3592677366 >> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] conn(0x5570fe70afd0 0x5570fe70d3b0 crc :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-05T19:28:35.701 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.694+0000 7f7830b63700 1 -- 172.21.15.167:0/3592677366 <== osd.2 v2:172.21.15.167:6816/30037 1 ==== osd_op_reply(5 gc.4 [create,call,call,call] v30'41 uv7 ondisk = -125 ((125) Operation canceled)) v8 ==== 274+0+0 (crc 0 0 0) 0x7f7828025020 con 0x5570fe70afd0 2023-10-05T19:28:35.701 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.694+0000 7f783302ee40 1 -- 172.21.15.167:0/3592677366 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:6 6.9 6:954be8d2:gc::gc.5:head [create,call version.check_conds in=74b,call rgw_gc.rgw_gc_queue_init in=45b,call version.set in=34b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x5570fe70e0b0 con 0x5570fe706e20 2023-10-05T19:28:35.704 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.698+0000 7f7831364700 1 -- 172.21.15.167:0/3592677366 <== osd.3 v2:172.21.15.167:6808/30036 2 ==== osd_op_reply(6 gc.5 [create,call,call,call] v30'29 uv4 ondisk = -125 ((125) Operation canceled)) v8 ==== 274+0+0 (crc 0 0 0) 0x7f7820050040 con 0x5570fe706e20 2023-10-05T19:28:35.704 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.698+0000 7f783302ee40 1 -- 172.21.15.167:0/3592677366 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:7 6.e 6:748722ff:gc::gc.6:head [create,call version.check_conds in=74b,call rgw_gc.rgw_gc_queue_init in=45b,call version.set in=34b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x5570fe70e6d0 con 0x5570fe6ffaf0 2023-10-05T19:28:35.706 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.702+0000 7f7830b63700 1 -- 172.21.15.167:0/3592677366 <== osd.1 v2:172.21.15.167:6800/30035 3 ==== osd_op_reply(7 gc.6 [create,call,call,call] v30'19 uv5 ondisk = -125 ((125) Operation canceled)) v8 ==== 274+0+0 (crc 0 0 0) 0x7f7828025020 con 0x5570fe6ffaf0 2023-10-05T19:28:35.707 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.702+0000 7f783302ee40 1 -- 172.21.15.167:0/3592677366 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:8 6.1d 6:beaa4be4:gc::gc.7:head [create,call version.check_conds in=74b,call rgw_gc.rgw_gc_queue_init in=45b,call version.set in=34b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x5570fe70ebc0 con 0x5570fe6fb730 2023-10-05T19:28:35.709 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.702+0000 7f7830362700 1 -- 172.21.15.167:0/3592677366 <== osd.0 v2:172.21.15.118:6800/27690 2 ==== osd_op_reply(8 gc.7 [create,call,call,call] v30'27 uv4 ondisk = -125 ((125) Operation canceled)) v8 ==== 274+0+0 (crc 0 0 0) 0x7f7824058040 con 0x5570fe6fb730 2023-10-05T19:28:35.709 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.702+0000 7f783302ee40 1 -- 172.21.15.167:0/3592677366 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:9 6.12 6:4f78e9a2:gc::gc.8:head [create,call version.check_conds in=74b,call rgw_gc.rgw_gc_queue_init in=45b,call version.set in=34b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x5570fe6fe640 con 0x5570fe70afd0 2023-10-05T19:28:35.712 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.706+0000 7f7830b63700 1 -- 172.21.15.167:0/3592677366 <== osd.2 v2:172.21.15.167:6816/30037 2 ==== osd_op_reply(9 gc.8 [create,call,call,call] v30'17 uv3 ondisk = -125 ((125) Operation canceled)) v8 ==== 274+0+0 (crc 0 0 0) 0x7f7828025020 con 0x5570fe70afd0 2023-10-05T19:28:35.712 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.706+0000 7f783302ee40 1 -- 172.21.15.167:0/3592677366 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:10 6.1c 6:3e426232:gc::gc.9:head [create,call version.check_conds in=74b,call rgw_gc.rgw_gc_queue_init in=45b,call version.set in=34b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x5570fe6fe640 con 0x5570fe706e20 2023-10-05T19:28:35.714 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.710+0000 7f7831364700 1 -- 172.21.15.167:0/3592677366 <== osd.3 v2:172.21.15.167:6808/30036 3 ==== osd_op_reply(10 gc.9 [create,call,call,call] v30'31 uv3 ondisk = -125 ((125) Operation canceled)) v8 ==== 274+0+0 (crc 0 0 0) 0x7f7820050040 con 0x5570fe706e20 2023-10-05T19:28:35.714 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.710+0000 7f783302ee40 1 -- 172.21.15.167:0/3592677366 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:11 6.3 6:c31f41d7:gc::gc.10:head [create,call version.check_conds in=74b,call rgw_gc.rgw_gc_queue_init in=45b,call version.set in=34b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x5570fe6fe8b0 con 0x5570fe70afd0 2023-10-05T19:28:35.717 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.710+0000 7f7830b63700 1 -- 172.21.15.167:0/3592677366 <== osd.2 v2:172.21.15.167:6816/30037 3 ==== osd_op_reply(11 gc.10 [create,call,call,call] v30'42 uv11 ondisk = -125 ((125) Operation canceled)) v8 ==== 275+0+0 (crc 0 0 0) 0x7f7828025020 con 0x5570fe70afd0 2023-10-05T19:28:35.717 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.710+0000 7f783302ee40 1 -- 172.21.15.167:0/3592677366 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:12 6.13 6:c8333aab:gc::gc.11:head [create,call version.check_conds in=74b,call rgw_gc.rgw_gc_queue_init in=45b,call version.set in=34b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x5570fe6fe8b0 con 0x5570fe6fb730 2023-10-05T19:28:35.719 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.714+0000 7f7830362700 1 -- 172.21.15.167:0/3592677366 <== osd.0 v2:172.21.15.118:6800/27690 3 ==== osd_op_reply(12 gc.11 [create,call,call,call] v30'20 uv5 ondisk = -125 ((125) Operation canceled)) v8 ==== 275+0+0 (crc 0 0 0) 0x7f7824058040 con 0x5570fe6fb730 2023-10-05T19:28:35.720 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.714+0000 7f783302ee40 1 -- 172.21.15.167:0/3592677366 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:13 6.15 6:ab569f81:gc::gc.12:head [create,call version.check_conds in=74b,call rgw_gc.rgw_gc_queue_init in=45b,call version.set in=34b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x5570fe70f950 con 0x5570fe70afd0 2023-10-05T19:28:35.722 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.718+0000 7f7830b63700 1 -- 172.21.15.167:0/3592677366 <== osd.2 v2:172.21.15.167:6816/30037 4 ==== osd_op_reply(13 gc.12 [create,call,call,call] v30'17 uv3 ondisk = -125 ((125) Operation canceled)) v8 ==== 275+0+0 (crc 0 0 0) 0x7f7828025020 con 0x5570fe70afd0 2023-10-05T19:28:35.722 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.718+0000 7f783302ee40 1 -- 172.21.15.167:0/3592677366 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:14 6.0 6:0667bb13:gc::gc.13:head [create,call version.check_conds in=74b,call rgw_gc.rgw_gc_queue_init in=45b,call version.set in=34b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x5570fe70f950 con 0x5570fe6fb730 2023-10-05T19:28:35.725 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.718+0000 7f7830362700 1 -- 172.21.15.167:0/3592677366 <== osd.0 v2:172.21.15.118:6800/27690 4 ==== osd_op_reply(14 gc.13 [create,call,call,call] v30'33 uv6 ondisk = -125 ((125) Operation canceled)) v8 ==== 275+0+0 (crc 0 0 0) 0x7f7824058040 con 0x5570fe6fb730 2023-10-05T19:28:35.725 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.718+0000 7f783302ee40 1 -- 172.21.15.167:0/3592677366 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:15 6.4 6:227f63b5:gc::gc.14:head [create,call version.check_conds in=74b,call rgw_gc.rgw_gc_queue_init in=45b,call version.set in=34b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x5570fe70ff70 con 0x5570fe6ffaf0 2023-10-05T19:28:35.728 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.722+0000 7f7830b63700 1 -- 172.21.15.167:0/3592677366 <== osd.1 v2:172.21.15.167:6800/30035 4 ==== osd_op_reply(15 gc.14 [create,call,call,call] v30'142 uv13 ondisk = -125 ((125) Operation canceled)) v8 ==== 275+0+0 (crc 0 0 0) 0x7f7828025020 con 0x5570fe6ffaf0 2023-10-05T19:28:35.728 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.722+0000 7f783302ee40 1 -- 172.21.15.167:0/3592677366 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:16 6.7 6:e07a6548:gc::gc.15:head [create,call version.check_conds in=74b,call rgw_gc.rgw_gc_queue_init in=45b,call version.set in=34b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x5570fe70ff70 con 0x5570fe706e20 2023-10-05T19:28:35.731 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.726+0000 7f7831364700 1 -- 172.21.15.167:0/3592677366 <== osd.3 v2:172.21.15.167:6808/30036 4 ==== osd_op_reply(16 gc.15 [create,call,call,call] v30'35 uv7 ondisk = -125 ((125) Operation canceled)) v8 ==== 275+0+0 (crc 0 0 0) 0x7f7820050040 con 0x5570fe706e20 2023-10-05T19:28:35.731 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.726+0000 7f783302ee40 1 -- 172.21.15.167:0/3592677366 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:17 6.3 6:c623b173:gc::gc.16:head [create,call version.check_conds in=74b,call rgw_gc.rgw_gc_queue_init in=45b,call version.set in=34b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x5570fe70ff70 con 0x5570fe70afd0 2023-10-05T19:28:35.734 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.726+0000 7f7830b63700 1 -- 172.21.15.167:0/3592677366 <== osd.2 v2:172.21.15.167:6816/30037 5 ==== osd_op_reply(17 gc.16 [create,call,call,call] v30'43 uv13 ondisk = -125 ((125) Operation canceled)) v8 ==== 275+0+0 (crc 0 0 0) 0x7f7828025020 con 0x5570fe70afd0 2023-10-05T19:28:35.734 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.730+0000 7f783302ee40 1 -- 172.21.15.167:0/3592677366 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:18 6.16 6:6849fd06:gc::gc.17:head [create,call version.check_conds in=74b,call rgw_gc.rgw_gc_queue_init in=45b,call version.set in=34b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x5570fe70ff70 con 0x5570fe706e20 2023-10-05T19:28:35.737 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.730+0000 7f7831364700 1 -- 172.21.15.167:0/3592677366 <== osd.3 v2:172.21.15.167:6808/30036 5 ==== osd_op_reply(18 gc.17 [create,call,call,call] v30'17 uv7 ondisk = -125 ((125) Operation canceled)) v8 ==== 275+0+0 (crc 0 0 0) 0x7f7820050040 con 0x5570fe706e20 2023-10-05T19:28:35.737 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.730+0000 7f783302ee40 1 -- 172.21.15.167:0/3592677366 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:19 6.d 6:b019ea83:gc::gc.18:head [create,call version.check_conds in=74b,call rgw_gc.rgw_gc_queue_init in=45b,call version.set in=34b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x5570fe70ff70 con 0x5570fe6ffaf0 2023-10-05T19:28:35.739 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.734+0000 7f7830b63700 1 -- 172.21.15.167:0/3592677366 <== osd.1 v2:172.21.15.167:6800/30035 5 ==== osd_op_reply(19 gc.18 [create,call,call,call] v30'25 uv5 ondisk = -125 ((125) Operation canceled)) v8 ==== 275+0+0 (crc 0 0 0) 0x7f7828025020 con 0x5570fe6ffaf0 2023-10-05T19:28:35.740 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.734+0000 7f783302ee40 1 -- 172.21.15.167:0/3592677366 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:20 6.4 6:24c8e1ae:gc::gc.19:head [create,call version.check_conds in=74b,call rgw_gc.rgw_gc_queue_init in=45b,call version.set in=34b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x5570fe70ff70 con 0x5570fe6ffaf0 2023-10-05T19:28:35.742 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.738+0000 7f7830b63700 1 -- 172.21.15.167:0/3592677366 <== osd.1 v2:172.21.15.167:6800/30035 6 ==== osd_op_reply(20 gc.19 [create,call,call,call] v30'143 uv15 ondisk = -125 ((125) Operation canceled)) v8 ==== 275+0+0 (crc 0 0 0) 0x7f7828025020 con 0x5570fe6ffaf0 2023-10-05T19:28:35.742 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.738+0000 7f783302ee40 1 -- 172.21.15.167:0/3592677366 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:21 6.7 6:e622f76a:gc::gc.20:head [create,call version.check_conds in=74b,call rgw_gc.rgw_gc_queue_init in=45b,call version.set in=34b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x5570fe70ff70 con 0x5570fe706e20 2023-10-05T19:28:35.745 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.738+0000 7f7831364700 1 -- 172.21.15.167:0/3592677366 <== osd.3 v2:172.21.15.167:6808/30036 6 ==== osd_op_reply(21 gc.20 [create,call,call,call] v30'36 uv9 ondisk = -125 ((125) Operation canceled)) v8 ==== 275+0+0 (crc 0 0 0) 0x7f7820050040 con 0x5570fe706e20 2023-10-05T19:28:35.745 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.738+0000 7f783302ee40 1 -- 172.21.15.167:0/3592677366 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:22 6.a 6:5127e642:gc::gc.21:head [create,call version.check_conds in=74b,call rgw_gc.rgw_gc_queue_init in=45b,call version.set in=34b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x5570fe70ff70 con 0x5570fe70afd0 2023-10-05T19:28:35.747 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.742+0000 7f7830b63700 1 -- 172.21.15.167:0/3592677366 <== osd.2 v2:172.21.15.167:6816/30037 6 ==== osd_op_reply(22 gc.21 [create,call,call,call] v30'27 uv7 ondisk = -125 ((125) Operation canceled)) v8 ==== 275+0+0 (crc 0 0 0) 0x7f7828025020 con 0x5570fe70afd0 2023-10-05T19:28:35.747 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.742+0000 7f783302ee40 1 -- 172.21.15.167:0/3592677366 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:23 6.11 6:8eb99c53:gc::gc.22:head [create,call version.check_conds in=74b,call rgw_gc.rgw_gc_queue_init in=45b,call version.set in=34b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x5570fe70ff70 con 0x5570fe706e20 2023-10-05T19:28:35.750 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.746+0000 7f7831364700 1 -- 172.21.15.167:0/3592677366 <== osd.3 v2:172.21.15.167:6808/30036 7 ==== osd_op_reply(23 gc.22 [create,call,call,call] v30'19 uv3 ondisk = -125 ((125) Operation canceled)) v8 ==== 275+0+0 (crc 0 0 0) 0x7f7820050040 con 0x5570fe706e20 2023-10-05T19:28:35.750 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.746+0000 7f783302ee40 1 -- 172.21.15.167:0/3592677366 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:24 6.8 6:17d7e8ed:gc::gc.23:head [create,call version.check_conds in=74b,call rgw_gc.rgw_gc_queue_init in=45b,call version.set in=34b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x5570fe710000 con 0x5570fe706e20 2023-10-05T19:28:35.752 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.746+0000 7f7831364700 1 -- 172.21.15.167:0/3592677366 <== osd.3 v2:172.21.15.167:6808/30036 8 ==== osd_op_reply(24 gc.23 [create,call,call,call] v30'28 uv5 ondisk = -125 ((125) Operation canceled)) v8 ==== 275+0+0 (crc 0 0 0) 0x7f7820050040 con 0x5570fe706e20 2023-10-05T19:28:35.752 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.746+0000 7f783302ee40 1 -- 172.21.15.167:0/3592677366 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:25 6.1b 6:db9d130d:gc::gc.24:head [create,call version.check_conds in=74b,call rgw_gc.rgw_gc_queue_init in=45b,call version.set in=34b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x5570fe710000 con 0x5570fe706e20 2023-10-05T19:28:35.754 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.750+0000 7f7831364700 1 -- 172.21.15.167:0/3592677366 <== osd.3 v2:172.21.15.167:6808/30036 9 ==== osd_op_reply(25 gc.24 [create,call,call,call] v30'13 uv3 ondisk = -125 ((125) Operation canceled)) v8 ==== 275+0+0 (crc 0 0 0) 0x7f7820050040 con 0x5570fe706e20 2023-10-05T19:28:35.754 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.750+0000 7f783302ee40 1 -- 172.21.15.167:0/3592677366 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:26 6.9 6:921ecdb1:gc::gc.25:head [create,call version.check_conds in=74b,call rgw_gc.rgw_gc_queue_init in=45b,call version.set in=34b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x5570fe710000 con 0x5570fe706e20 2023-10-05T19:28:35.756 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.750+0000 7f7831364700 1 -- 172.21.15.167:0/3592677366 <== osd.3 v2:172.21.15.167:6808/30036 10 ==== osd_op_reply(26 gc.25 [create,call,call,call] v30'30 uv10 ondisk = -125 ((125) Operation canceled)) v8 ==== 275+0+0 (crc 0 0 0) 0x7f7820050040 con 0x5570fe706e20 2023-10-05T19:28:35.756 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.750+0000 7f783302ee40 1 -- 172.21.15.167:0/3592677366 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:27 6.1f 6:fde4dd55:gc::gc.26:head [create,call version.check_conds in=74b,call rgw_gc.rgw_gc_queue_init in=45b,call version.set in=34b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x5570fe710000 con 0x5570fe6fb730 2023-10-05T19:28:35.759 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.754+0000 7f7830362700 1 -- 172.21.15.167:0/3592677366 <== osd.0 v2:172.21.15.118:6800/27690 5 ==== osd_op_reply(27 gc.26 [create,call,call,call] v30'19 uv5 ondisk = -125 ((125) Operation canceled)) v8 ==== 275+0+0 (crc 0 0 0) 0x7f7824058040 con 0x5570fe6fb730 2023-10-05T19:28:35.759 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.754+0000 7f783302ee40 1 -- 172.21.15.167:0/3592677366 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:28 6.1a 6:5909e671:gc::gc.27:head [create,call version.check_conds in=74b,call rgw_gc.rgw_gc_queue_init in=45b,call version.set in=34b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x5570fe710000 con 0x5570fe706e20 2023-10-05T19:28:35.762 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.754+0000 7f7831364700 1 -- 172.21.15.167:0/3592677366 <== osd.3 v2:172.21.15.167:6808/30036 11 ==== osd_op_reply(28 gc.27 [create,call,call,call] v30'17 uv7 ondisk = -125 ((125) Operation canceled)) v8 ==== 275+0+0 (crc 0 0 0) 0x7f7820050040 con 0x5570fe706e20 2023-10-05T19:28:35.762 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.758+0000 7f783302ee40 1 -- 172.21.15.167:0/3592677366 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:29 6.1d 6:b88e3271:gc::gc.28:head [create,call version.check_conds in=74b,call rgw_gc.rgw_gc_queue_init in=45b,call version.set in=34b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x5570fe710000 con 0x5570fe6fb730 2023-10-05T19:28:35.765 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.758+0000 7f7830362700 1 -- 172.21.15.167:0/3592677366 <== osd.0 v2:172.21.15.118:6800/27690 6 ==== osd_op_reply(29 gc.28 [create,call,call,call] v30'28 uv8 ondisk = -125 ((125) Operation canceled)) v8 ==== 275+0+0 (crc 0 0 0) 0x7f7824058040 con 0x5570fe6fb730 2023-10-05T19:28:35.765 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.758+0000 7f783302ee40 1 -- 172.21.15.167:0/3592677366 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:30 6.7 6:e47c9322:gc::gc.29:head [create,call version.check_conds in=74b,call rgw_gc.rgw_gc_queue_init in=45b,call version.set in=34b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x5570fe710000 con 0x5570fe706e20 2023-10-05T19:28:35.768 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.762+0000 7f7831364700 1 -- 172.21.15.167:0/3592677366 <== osd.3 v2:172.21.15.167:6808/30036 12 ==== osd_op_reply(30 gc.29 [create,call,call,call] v30'37 uv11 ondisk = -125 ((125) Operation canceled)) v8 ==== 275+0+0 (crc 0 0 0) 0x7f7820050040 con 0x5570fe706e20 2023-10-05T19:28:35.768 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.762+0000 7f783302ee40 1 -- 172.21.15.167:0/3592677366 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:31 6.0 6:013d6c33:gc::gc.30:head [create,call version.check_conds in=74b,call rgw_gc.rgw_gc_queue_init in=45b,call version.set in=34b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x5570fe710000 con 0x5570fe6fb730 2023-10-05T19:28:35.771 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.766+0000 7f7830362700 1 -- 172.21.15.167:0/3592677366 <== osd.0 v2:172.21.15.118:6800/27690 7 ==== osd_op_reply(31 gc.30 [create,call,call,call] v30'34 uv12 ondisk = -125 ((125) Operation canceled)) v8 ==== 275+0+0 (crc 0 0 0) 0x7f7824058040 con 0x5570fe6fb730 2023-10-05T19:28:35.771 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.766+0000 7f783302ee40 1 -- 172.21.15.167:0/3592677366 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:32 6.f 6:f24cb4a8:gc::gc.31:head [create,call version.check_conds in=74b,call rgw_gc.rgw_gc_queue_init in=45b,call version.set in=34b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x5570fe710000 con 0x5570fe70afd0 2023-10-05T19:28:35.773 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.766+0000 7f7830b63700 1 -- 172.21.15.167:0/3592677366 <== osd.2 v2:172.21.15.167:6816/30037 7 ==== osd_op_reply(32 gc.31 [create,call,call,call] v30'23 uv5 ondisk = -125 ((125) Operation canceled)) v8 ==== 275+0+0 (crc 0 0 0) 0x7f7828025020 con 0x5570fe70afd0 2023-10-05T19:28:35.776 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.770+0000 7f77717fa700 20 reqs_thread_entry: start 2023-10-05T19:28:35.776 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.770+0000 7f783302ee40 20 init_complete bucket index max shards: 11 2023-10-05T19:28:35.776 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.770+0000 7f776affd700 20 reqs_thread_entry: start 2023-10-05T19:28:35.777 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.770+0000 7f783302ee40 10 rgw notify: Started notification manager with: 1 workers 2023-10-05T19:28:35.777 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.770+0000 7f77697fa700 20 rgw notify: INFO: next queues processing will happen at: Thu Oct 5 19:29:05 2023 2023-10-05T19:28:35.778 INFO:teuthology.orchestra.run.smithi167.stderr: 2023-10-05T19:28:35.778 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.770+0000 7f7768ff9700 20 reqs_thread_entry: start 2023-10-05T19:28:35.778 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.770+0000 7f783302ee40 20 get_system_obj_state: rctx=0x7ffdb36ea3a0 obj=default.rgw.meta:root:myfoo state=0x5570fe5780b0 s->prefetch_data=0 2023-10-05T19:28:35.778 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.770+0000 7f783302ee40 10 cache get: name=default.rgw.meta+root+myfoo : miss 2023-10-05T19:28:35.778 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.770+0000 7f783302ee40 1 -- 172.21.15.167:0/2022643642 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:102 8.9 8:91b71841:root::myfoo:head [call version.read in=11b,getxattrs,stat] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x5570fe76ee60 con 0x7f782801fc20 2023-10-05T19:28:35.779 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.774+0000 7f7830362700 1 -- 172.21.15.167:0/2022643642 <== osd.2 v2:172.21.15.167:6816/30037 17 ==== osd_op_reply(102 myfoo [call out=48b,getxattrs out=75b,stat out=16b] v0'0 uv27 ondisk = 0) v8 ==== 233+0+139 (crc 0 0 0) 0x7f7824058040 con 0x7f782801fc20 2023-10-05T19:28:35.779 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.774+0000 7f783302ee40 10 cache put: name=default.rgw.meta+root+myfoo info.flags=0x16 2023-10-05T19:28:35.779 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.774+0000 7f783302ee40 10 adding default.rgw.meta+root+myfoo to cache LRU end 2023-10-05T19:28:35.779 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.774+0000 7f783302ee40 10 updating xattr: name=ceph.objclass.version bl.length()=42 2023-10-05T19:28:35.779 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.774+0000 7f783302ee40 20 get_system_obj_state: s->obj_tag was set empty 2023-10-05T19:28:35.779 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.774+0000 7f783302ee40 10 cache get: name=default.rgw.meta+root+myfoo : type miss (requested=0x11, cached=0x16) 2023-10-05T19:28:35.779 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.774+0000 7f783302ee40 20 rados->read ofs=0 len=0 2023-10-05T19:28:35.780 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.774+0000 7f783302ee40 1 -- 172.21.15.167:0/2022643642 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:103 8.9 8:91b71841:root::myfoo:head [call version.check_conds in=122b,call version.read in=11b,read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x5570fe76ed90 con 0x7f782801fc20 2023-10-05T19:28:35.781 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.774+0000 7f7830362700 1 -- 172.21.15.167:0/2022643642 <== osd.2 v2:172.21.15.167:6816/30037 18 ==== osd_op_reply(103 myfoo [call,call out=48b,read 0~165 out=165b] v0'0 uv27 ondisk = 0) v8 ==== 233+0+213 (crc 0 0 0) 0x7f7824058040 con 0x7f782801fc20 2023-10-05T19:28:35.781 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.774+0000 7f783302ee40 20 rados_obj.operate() r=0 bl.length=165 2023-10-05T19:28:35.781 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.774+0000 7f783302ee40 10 cache put: name=default.rgw.meta+root+myfoo info.flags=0x11 2023-10-05T19:28:35.782 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.774+0000 7f783302ee40 10 moving default.rgw.meta+root+myfoo to cache LRU end 2023-10-05T19:28:35.782 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.774+0000 7f783302ee40 20 get_system_obj_state: rctx=0x7ffdb36ea3a0 obj=default.rgw.meta:root:.bucket.meta.myfoo:9a9df965-6612-45d4-a45f-6e3c889bacc2.4223.1 state=0x5570fe54e2b0 s->prefetch_data=0 2023-10-05T19:28:35.782 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.774+0000 7f783302ee40 10 cache get: name=default.rgw.meta+root+.bucket.meta.myfoo:9a9df965-6612-45d4-a45f-6e3c889bacc2.4223.1 : miss 2023-10-05T19:28:35.782 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.774+0000 7f783302ee40 1 -- 172.21.15.167:0/2022643642 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:104 8.11 8:8bb4ed75:root::.bucket.meta.myfoo%3a9a9df965-6612-45d4-a45f-6e3c889bacc2.4223.1:head [call version.read in=11b,getxattrs,stat] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x5570fe7708e0 con 0x5570fe6d6640 2023-10-05T19:28:35.782 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.774+0000 7f7830b63700 1 -- 172.21.15.167:0/2022643642 <== osd.3 v2:172.21.15.167:6808/30036 24 ==== osd_op_reply(104 .bucket.meta.myfoo:9a9df965-6612-45d4-a45f-6e3c889bacc2.4223.1 [call out=48b,getxattrs out=218b,stat out=16b] v0'0 uv5 ondisk = 0) v8 ==== 290+0+282 (crc 0 0 0) 0x7f7828025020 con 0x5570fe6d6640 2023-10-05T19:28:35.782 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.774+0000 7f783302ee40 10 cache put: name=default.rgw.meta+root+.bucket.meta.myfoo:9a9df965-6612-45d4-a45f-6e3c889bacc2.4223.1 info.flags=0x16 2023-10-05T19:28:35.782 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.774+0000 7f783302ee40 10 adding default.rgw.meta+root+.bucket.meta.myfoo:9a9df965-6612-45d4-a45f-6e3c889bacc2.4223.1 to cache LRU end 2023-10-05T19:28:35.782 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.774+0000 7f783302ee40 10 updating xattr: name=ceph.objclass.version bl.length()=42 2023-10-05T19:28:35.782 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.774+0000 7f783302ee40 10 updating xattr: name=user.rgw.acl bl.length()=123 2023-10-05T19:28:35.783 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.774+0000 7f783302ee40 20 get_system_obj_state: s->obj_tag was set empty 2023-10-05T19:28:35.783 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.774+0000 7f783302ee40 20 Read xattr: user.rgw.acl 2023-10-05T19:28:35.783 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.774+0000 7f783302ee40 20 Read xattr: user.rgw.idtag 2023-10-05T19:28:35.783 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.774+0000 7f783302ee40 10 cache get: name=default.rgw.meta+root+.bucket.meta.myfoo:9a9df965-6612-45d4-a45f-6e3c889bacc2.4223.1 : type miss (requested=0x13, cached=0x16) 2023-10-05T19:28:35.783 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.774+0000 7f783302ee40 20 rados->read ofs=0 len=0 2023-10-05T19:28:35.783 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.774+0000 7f783302ee40 1 -- 172.21.15.167:0/2022643642 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:105 8.11 8:8bb4ed75:root::.bucket.meta.myfoo%3a9a9df965-6612-45d4-a45f-6e3c889bacc2.4223.1:head [call version.check_conds in=122b,call version.read in=11b,read 0~0,getxattrs] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x5570fe7714a0 con 0x5570fe6d6640 2023-10-05T19:28:35.784 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.774+0000 7f7830b63700 1 -- 172.21.15.167:0/2022643642 <== osd.3 v2:172.21.15.167:6808/30036 25 ==== osd_op_reply(105 .bucket.meta.myfoo:9a9df965-6612-45d4-a45f-6e3c889bacc2.4223.1 [call,call out=48b,read 0~352 out=352b,getxattrs out=218b] v0'0 uv5 ondisk = 0) v8 ==== 332+0+618 (crc 0 0 0) 0x7f7828025020 con 0x5570fe6d6640 2023-10-05T19:28:35.784 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.774+0000 7f783302ee40 20 rados_obj.operate() r=0 bl.length=352 2023-10-05T19:28:35.785 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.774+0000 7f783302ee40 10 cache put: name=default.rgw.meta+root+.bucket.meta.myfoo:9a9df965-6612-45d4-a45f-6e3c889bacc2.4223.1 info.flags=0x13 2023-10-05T19:28:35.785 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.774+0000 7f783302ee40 10 moving default.rgw.meta+root+.bucket.meta.myfoo:9a9df965-6612-45d4-a45f-6e3c889bacc2.4223.1 to cache LRU end 2023-10-05T19:28:35.785 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.774+0000 7f783302ee40 10 updating xattr: name=ceph.objclass.version bl.length()=42 2023-10-05T19:28:35.785 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.774+0000 7f783302ee40 10 updating xattr: name=user.rgw.acl bl.length()=123 2023-10-05T19:28:35.785 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.778+0000 7f783302ee40 10 chain_cache_entry: cache_locator=default.rgw.meta+root+.bucket.meta.myfoo:9a9df965-6612-45d4-a45f-6e3c889bacc2.4223.1 2023-10-05T19:28:35.785 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.778+0000 7f783302ee40 20 get_obj_state: rctx=0x7ffdb36ea660 obj=myfoo:four state=0x5570fe771ca8 s->prefetch_data=0 2023-10-05T19:28:35.785 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.778+0000 7f783302ee40 1 -- 172.21.15.167:0/2022643642 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:106 3.cs0 3:330f4ecf:::9a9df965-6612-45d4-a45f-6e3c889bacc2.4223.1_four:head [getxattrs,stat] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x5570fe772b00 con 0x5570fe6d6640 2023-10-05T19:28:35.785 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.778+0000 7f7830b63700 1 -- 172.21.15.167:0/2022643642 <== osd.3 v2:172.21.15.167:6808/30036 26 ==== osd_op_reply(106 9a9df965-6612-45d4-a45f-6e3c889bacc2.4223.1_four [getxattrs out=851b,stat out=16b] v0'0 uv1 ondisk = 0) v8 ==== 234+0+867 (crc 0 0 0) 0x7f7828025020 con 0x5570fe6d6640 2023-10-05T19:28:35.785 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.778+0000 7f783302ee40 10 manifest: total_size = 4 2023-10-05T19:28:35.785 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.778+0000 7f783302ee40 20 get_obj_state: setting s->obj_tag to 9a9df965-6612-45d4-a45f-6e3c889bacc2.4221.14690770643123788769 2023-10-05T19:28:35.785 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.778+0000 7f783302ee40 20 get_obj_state: rctx=0x7ffdb36ea660 obj=myfoo:four state=0x5570fe771ca8 s->prefetch_data=0 2023-10-05T19:28:35.786 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.778+0000 7f783302ee40 20 prepare_atomic_modification: state is not atomic. state=0x5570fe771ca8 2023-10-05T19:28:35.786 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.778+0000 7f783302ee40 20 bucket index object: default.rgw.buckets.index:.dir.9a9df965-6612-45d4-a45f-6e3c889bacc2.4223.1.6 2023-10-05T19:28:35.786 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.778+0000 7f783302ee40 1 -- 172.21.15.167:0/2022643642 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:107 4.38 4:1f2a6458:::.dir.9a9df965-6612-45d4-a45f-6e3c889bacc2.4223.1.6:head [stat,call rgw.guard_bucket_resharding in=36b,call rgw.bucket_prepare_op in=182b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x5570fe7733e0 con 0x5570fe6a5600 2023-10-05T19:28:35.786 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.778+0000 7f7831364700 1 -- 172.21.15.167:0/2022643642 <== osd.1 v2:172.21.15.167:6800/30035 35 ==== osd_op_reply(107 .dir.9a9df965-6612-45d4-a45f-6e3c889bacc2.4223.1.6 [stat,call,call] v30'7 uv7 ondisk = 0) v8 ==== 278+0+0 (crc 0 0 0) 0x7f7820050040 con 0x5570fe6a5600 2023-10-05T19:28:35.786 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.778+0000 7f783302ee40 1 -- 172.21.15.167:0/2022643642 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:108 3.cs0 3:330f4ecf:::9a9df965-6612-45d4-a45f-6e3c889bacc2.4223.1_four:head [call rgw.obj_remove in=40b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x5570fe7733e0 con 0x5570fe6d6640 2023-10-05T19:28:35.790 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.782+0000 7f7830b63700 1 -- 172.21.15.167:0/2022643642 <== osd.3 v2:172.21.15.167:6808/30036 27 ==== osd_op_reply(108 9a9df965-6612-45d4-a45f-6e3c889bacc2.4223.1_four [call] v30'2 uv2 ondisk = 0) v8 ==== 192+0+0 (crc 0 0 0) 0x7f7828025020 con 0x5570fe6d6640 2023-10-05T19:28:35.790 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.782+0000 7f783302ee40 1 -- 172.21.15.167:0/2022643642 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:109 4.38 4:1f2a6458:::.dir.9a9df965-6612-45d4-a45f-6e3c889bacc2.4223.1.6:head [stat,call rgw.guard_bucket_resharding in=36b,call rgw.bucket_complete_op in=259b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x5570fe7780c0 con 0x5570fe6a5600 2023-10-05T19:28:35.790 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.782+0000 7f783302ee40 10 cache get: name=default.rgw.log++bucket.sync-source-hints.myfoo : miss 2023-10-05T19:28:35.790 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.782+0000 7f783302ee40 20 rados->read ofs=0 len=0 2023-10-05T19:28:35.790 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.782+0000 7f783302ee40 1 -- 172.21.15.167:0/2022643642 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:110 6.18 6:1df25bda:::bucket.sync-source-hints.myfoo:head [call version.read in=11b,read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x5570fe77a690 con 0x7f7824007030 2023-10-05T19:28:35.791 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.786+0000 7f7831364700 1 -- 172.21.15.167:0/2022643642 <== osd.0 v2:172.21.15.118:6800/27690 29 ==== osd_op_reply(110 bucket.sync-source-hints.myfoo [call,read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 ==== 216+0+0 (crc 0 0 0) 0x7f7820050040 con 0x7f7824007030 2023-10-05T19:28:35.791 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.786+0000 7f783302ee40 20 rados_obj.operate() r=-2 bl.length=0 2023-10-05T19:28:35.791 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.786+0000 7f783302ee40 10 cache put: name=default.rgw.log++bucket.sync-source-hints.myfoo info.flags=0x0 2023-10-05T19:28:35.791 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.786+0000 7f783302ee40 10 adding default.rgw.log++bucket.sync-source-hints.myfoo to cache LRU end 2023-10-05T19:28:35.791 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.786+0000 7f783302ee40 10 cache get: name=default.rgw.log++bucket.sync-target-hints.myfoo : miss 2023-10-05T19:28:35.791 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.786+0000 7f783302ee40 20 rados->read ofs=0 len=0 2023-10-05T19:28:35.791 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.786+0000 7f783302ee40 1 -- 172.21.15.167:0/2022643642 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:111 6.4 6:26e4c4bc:::bucket.sync-target-hints.myfoo:head [call version.read in=11b,read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x5570fe77a690 con 0x5570fe6a5600 2023-10-05T19:28:35.792 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.786+0000 7f7831364700 1 -- 172.21.15.167:0/2022643642 <== osd.1 v2:172.21.15.167:6800/30035 36 ==== osd_op_reply(111 bucket.sync-target-hints.myfoo [call,read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 ==== 216+0+0 (crc 0 0 0) 0x7f7820050040 con 0x5570fe6a5600 2023-10-05T19:28:35.792 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.786+0000 7f783302ee40 20 rados_obj.operate() r=-2 bl.length=0 2023-10-05T19:28:35.793 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.786+0000 7f783302ee40 10 cache put: name=default.rgw.log++bucket.sync-target-hints.myfoo info.flags=0x0 2023-10-05T19:28:35.793 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.786+0000 7f783302ee40 10 adding default.rgw.log++bucket.sync-target-hints.myfoo to cache LRU end 2023-10-05T19:28:35.793 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.786+0000 7f783302ee40 10 chain_cache_entry: cache_locator= 2023-10-05T19:28:35.793 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.786+0000 7f783302ee40 20 chain_cache_entry: couldn't find cache locator 2023-10-05T19:28:35.793 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.786+0000 7f783302ee40 20 couldn't put bucket_sync_policy cache entry, might have raced with data changes 2023-10-05T19:28:35.793 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.786+0000 7f783302ee40 1 -- 172.21.15.167:0/2022643642 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:112 7.1f 7:f95f44c2:::notify.0:head [watch unwatch cookie 93943793300832] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x5570fe6b68a0 con 0x7f782801fc20 2023-10-05T19:28:35.794 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.786+0000 7f7831364700 1 -- 172.21.15.167:0/2022643642 <== osd.1 v2:172.21.15.167:6800/30035 37 ==== osd_op_reply(109 .dir.9a9df965-6612-45d4-a45f-6e3c889bacc2.4223.1.6 [stat,call,call] v30'8 uv8 ondisk = 0) v8 ==== 278+0+0 (crc 0 0 0) 0x7f7820050040 con 0x5570fe6a5600 2023-10-05T19:28:35.794 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.786+0000 7f782eb5f700 20 handle_completion(): completion ok for obj=four 2023-10-05T19:28:35.795 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.790+0000 7f7830362700 1 -- 172.21.15.167:0/2022643642 <== osd.2 v2:172.21.15.167:6816/30037 19 ==== osd_op_reply(112 notify.0 [watch unwatch cookie 93943793300832] v30'146 uv144 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f7824060040 con 0x7f782801fc20 2023-10-05T19:28:35.795 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.790+0000 7f783302ee40 20 remove_watcher() i=0 2023-10-05T19:28:35.795 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.790+0000 7f783302ee40 2 removed watcher, disabling cache 2023-10-05T19:28:35.795 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.790+0000 7f783302ee40 1 -- 172.21.15.167:0/2022643642 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:113 7.0 7:05bf5b68:::notify.1:head [watch unwatch cookie 93943793305472] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x5570fe6db560 con 0x5570fe6a5600 2023-10-05T19:28:35.797 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.790+0000 7f7831364700 1 -- 172.21.15.167:0/2022643642 <== osd.1 v2:172.21.15.167:6800/30035 38 ==== osd_op_reply(113 notify.1 [watch unwatch cookie 93943793305472] v30'146 uv144 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f7820050040 con 0x5570fe6a5600 2023-10-05T19:28:35.797 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.790+0000 7f783302ee40 20 remove_watcher() i=1 2023-10-05T19:28:35.798 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.790+0000 7f783302ee40 1 -- 172.21.15.167:0/2022643642 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:114 7.15 7:a93a5511:::notify.2:head [watch unwatch cookie 93943793307760] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x5570fe6db560 con 0x5570fe6d6640 2023-10-05T19:28:35.800 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.794+0000 7f7830b63700 1 -- 172.21.15.167:0/2022643642 <== osd.3 v2:172.21.15.167:6808/30036 28 ==== osd_op_reply(114 notify.2 [watch unwatch cookie 93943793307760] v30'146 uv144 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f7828025020 con 0x5570fe6d6640 2023-10-05T19:28:35.800 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.794+0000 7f783302ee40 20 remove_watcher() i=2 2023-10-05T19:28:35.800 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.794+0000 7f783302ee40 1 -- 172.21.15.167:0/2022643642 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:115 7.e 7:7759931f:::notify.3:head [watch unwatch cookie 93943793314896] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x5570fe6dd070 con 0x7f782801fc20 2023-10-05T19:28:35.802 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.798+0000 7f7830362700 1 -- 172.21.15.167:0/2022643642 <== osd.2 v2:172.21.15.167:6816/30037 20 ==== osd_op_reply(115 notify.3 [watch unwatch cookie 93943793314896] v30'146 uv144 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f7824060040 con 0x7f782801fc20 2023-10-05T19:28:35.802 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.798+0000 7f783302ee40 20 remove_watcher() i=3 2023-10-05T19:28:35.802 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.798+0000 7f783302ee40 1 -- 172.21.15.167:0/2022643642 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:116 7.d 7:b4812045:::notify.4:head [watch unwatch cookie 93943793308864] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x5570fe6dec50 con 0x5570fe6a5600 2023-10-05T19:28:35.805 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.798+0000 7f7831364700 1 -- 172.21.15.167:0/2022643642 <== osd.1 v2:172.21.15.167:6800/30035 39 ==== osd_op_reply(116 notify.4 [watch unwatch cookie 93943793308864] v30'146 uv144 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f7820050040 con 0x5570fe6a5600 2023-10-05T19:28:35.805 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.798+0000 7f783302ee40 20 remove_watcher() i=4 2023-10-05T19:28:35.805 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.798+0000 7f783302ee40 1 -- 172.21.15.167:0/2022643642 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:117 7.3 7:c609908c:::notify.5:head [watch unwatch cookie 93943793318848] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x5570fe6dec50 con 0x7f7824007030 2023-10-05T19:28:35.808 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.802+0000 7f7831364700 1 -- 172.21.15.167:0/2022643642 <== osd.0 v2:172.21.15.118:6800/27690 30 ==== osd_op_reply(117 notify.5 [watch unwatch cookie 93943793318848] v30'146 uv144 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f7820050040 con 0x7f7824007030 2023-10-05T19:28:35.808 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.802+0000 7f783302ee40 20 remove_watcher() i=5 2023-10-05T19:28:35.808 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.802+0000 7f783302ee40 1 -- 172.21.15.167:0/2022643642 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:118 7.14 7:2b04a3e9:::notify.6:head [watch unwatch cookie 93943793322720] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x5570fe6dfbc0 con 0x7f7824007030 2023-10-05T19:28:35.810 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.806+0000 7f7831364700 1 -- 172.21.15.167:0/2022643642 <== osd.0 v2:172.21.15.118:6800/27690 31 ==== osd_op_reply(118 notify.6 [watch unwatch cookie 93943793322720] v30'146 uv144 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f7820050040 con 0x7f7824007030 2023-10-05T19:28:35.810 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.806+0000 7f783302ee40 20 remove_watcher() i=6 2023-10-05T19:28:35.811 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.806+0000 7f783302ee40 1 -- 172.21.15.167:0/2022643642 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:119 7.9 7:93e5b521:::notify.7:head [watch unwatch cookie 93943793325440] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x5570fe6dec50 con 0x7f7824007030 2023-10-05T19:28:35.813 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.806+0000 7f7831364700 1 -- 172.21.15.167:0/2022643642 <== osd.0 v2:172.21.15.118:6800/27690 32 ==== osd_op_reply(119 notify.7 [watch unwatch cookie 93943793325440] v30'146 uv144 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f7820050040 con 0x7f7824007030 2023-10-05T19:28:35.813 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.806+0000 7f783302ee40 20 remove_watcher() i=7 2023-10-05T19:28:35.815 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.810+0000 7f78027e4700 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2023-10-05T19:28:35.815 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.810+0000 7f783302ee40 1 -- 172.21.15.167:0/2022643642 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:120 6.4 6:22d26bf9:::data_loggenerations_metadata:head [watch unwatch cookie 93943793326544] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x5570fe6e0ae0 con 0x5570fe6a5600 2023-10-05T19:28:35.820 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.810+0000 7f7831364700 1 -- 172.21.15.167:0/2022643642 <== osd.1 v2:172.21.15.167:6800/30035 40 ==== osd_op_reply(120 data_loggenerations_metadata [watch unwatch cookie 93943793326544] v30'144 uv1 ondisk = 0) v8 ==== 172+0+0 (crc 0 0 0) 0x7f7820050040 con 0x5570fe6a5600 2023-10-05T19:28:35.820 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.810+0000 7f783302ee40 1 -- 172.21.15.167:0/2022643642 >> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] conn(0x7f7824007030 msgr2=0x7f782401c920 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-05T19:28:35.820 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.810+0000 7f783302ee40 1 --2- 172.21.15.167:0/2022643642 >> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] conn(0x7f7824007030 0x7f782401c920 crc :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-05T19:28:35.820 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.810+0000 7f783302ee40 1 -- 172.21.15.167:0/2022643642 >> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] conn(0x5570fe6a5600 msgr2=0x5570fe6d28f0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-05T19:28:35.820 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.810+0000 7f783302ee40 1 --2- 172.21.15.167:0/2022643642 >> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] conn(0x5570fe6a5600 0x5570fe6d28f0 crc :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-05T19:28:35.820 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.810+0000 7f783302ee40 1 -- 172.21.15.167:0/2022643642 >> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] conn(0x7f782801fc20 msgr2=0x7f78280420c0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-05T19:28:35.820 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.810+0000 7f783302ee40 1 --2- 172.21.15.167:0/2022643642 >> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] conn(0x7f782801fc20 0x7f78280420c0 crc :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-05T19:28:35.820 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.810+0000 7f783302ee40 1 -- 172.21.15.167:0/2022643642 >> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] conn(0x5570fe6d6640 msgr2=0x5570fe6d8a20 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-05T19:28:35.820 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.810+0000 7f783302ee40 1 --2- 172.21.15.167:0/2022643642 >> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] conn(0x5570fe6d6640 0x5570fe6d8a20 crc :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-05T19:28:35.821 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.810+0000 7f783302ee40 1 -- 172.21.15.167:0/2022643642 >> [v2:172.21.15.167:6824/29889,v1:172.21.15.167:6825/29889] conn(0x7f781c030c00 msgr2=0x7f781c0330b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-05T19:28:35.821 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.810+0000 7f783302ee40 1 --2- 172.21.15.167:0/2022643642 >> [v2:172.21.15.167:6824/29889,v1:172.21.15.167:6825/29889] conn(0x7f781c030c00 0x7f781c0330b0 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7f7824005a90 tx=0x7f7824015000 comp rx=0 tx=0).stop 2023-10-05T19:28:35.821 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.814+0000 7f783302ee40 1 -- 172.21.15.167:0/2022643642 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5570fe6a1020 msgr2=0x5570fe6c6d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-05T19:28:35.821 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.814+0000 7f783302ee40 1 --2- 172.21.15.167:0/2022643642 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5570fe6a1020 0x5570fe6c6d40 secure :-1 s=READY pgs=425 cs=0 l=1 rev1=1 crypto rx=0x7f78280069f0 tx=0x7f7828014000 comp rx=0 tx=0).stop 2023-10-05T19:28:35.821 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.814+0000 7f7831364700 1 -- 172.21.15.167:0/2022643642 reap_dead start 2023-10-05T19:28:35.821 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.814+0000 7f783302ee40 1 -- 172.21.15.167:0/2022643642 shutdown_connections 2023-10-05T19:28:35.821 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.814+0000 7f783302ee40 1 --2- 172.21.15.167:0/2022643642 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5570fe6a1020 0x5570fe6c6d40 unknown :-1 s=CLOSED pgs=425 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-05T19:28:35.821 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.814+0000 7f783302ee40 1 -- 172.21.15.167:0/2022643642 >> 172.21.15.167:0/2022643642 conn(0x5570fe614870 msgr2=0x5570fe6ba580 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-05T19:28:35.821 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.814+0000 7f783302ee40 1 -- 172.21.15.167:0/2022643642 shutdown_connections 2023-10-05T19:28:35.821 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.814+0000 7f783302ee40 1 -- 172.21.15.167:0/2022643642 wait complete. 2023-10-05T19:28:35.821 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.814+0000 7f783302ee40 1 -- 172.21.15.167:0/3592677366 >> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] conn(0x5570fe6fb730 msgr2=0x5570fe705c10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-05T19:28:35.822 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.814+0000 7f783302ee40 1 --2- 172.21.15.167:0/3592677366 >> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] conn(0x5570fe6fb730 0x5570fe705c10 crc :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-05T19:28:35.822 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.814+0000 7f783302ee40 1 -- 172.21.15.167:0/3592677366 >> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] conn(0x5570fe6ffaf0 msgr2=0x5570fe701f00 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-05T19:28:35.822 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.814+0000 7f783302ee40 1 --2- 172.21.15.167:0/3592677366 >> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] conn(0x5570fe6ffaf0 0x5570fe701f00 crc :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-05T19:28:35.822 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.814+0000 7f783302ee40 1 -- 172.21.15.167:0/3592677366 >> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] conn(0x5570fe70afd0 msgr2=0x5570fe70d3b0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-05T19:28:35.822 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.814+0000 7f783302ee40 1 --2- 172.21.15.167:0/3592677366 >> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] conn(0x5570fe70afd0 0x5570fe70d3b0 crc :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-05T19:28:35.822 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.814+0000 7f783302ee40 1 -- 172.21.15.167:0/3592677366 >> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] conn(0x5570fe706e20 msgr2=0x5570fe709230 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-05T19:28:35.822 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.814+0000 7f783302ee40 1 --2- 172.21.15.167:0/3592677366 >> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] conn(0x5570fe706e20 0x5570fe709230 crc :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-05T19:28:35.822 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.814+0000 7f783302ee40 1 -- 172.21.15.167:0/3592677366 >> [v2:172.21.15.167:6824/29889,v1:172.21.15.167:6825/29889] conn(0x7f7778030c00 msgr2=0x7f77780330b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-05T19:28:35.822 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.814+0000 7f783302ee40 1 --2- 172.21.15.167:0/3592677366 >> [v2:172.21.15.167:6824/29889,v1:172.21.15.167:6825/29889] conn(0x7f7778030c00 0x7f77780330b0 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x5570fe6e02d0 tx=0x7f7820000f90 comp rx=0 tx=0).stop 2023-10-05T19:28:35.822 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.814+0000 7f783302ee40 1 -- 172.21.15.167:0/3592677366 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5570fe6ef3d0 msgr2=0x5570fe6f4f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-05T19:28:35.822 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.814+0000 7f783302ee40 1 --2- 172.21.15.167:0/3592677366 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5570fe6ef3d0 0x5570fe6f4f00 secure :-1 s=READY pgs=427 cs=0 l=1 rev1=1 crypto rx=0x5570fe6e9450 tx=0x7f782402f000 comp rx=0 tx=0).stop 2023-10-05T19:28:35.823 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.814+0000 7f7830b63700 1 -- 172.21.15.167:0/3592677366 reap_dead start 2023-10-05T19:28:35.823 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.814+0000 7f7830b63700 1 -- 172.21.15.167:0/3592677366 reap_dead start 2023-10-05T19:28:35.823 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.814+0000 7f783302ee40 1 -- 172.21.15.167:0/3592677366 shutdown_connections 2023-10-05T19:28:35.823 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.814+0000 7f783302ee40 1 -- 172.21.15.167:0/3592677366 >> 172.21.15.167:0/3592677366 conn(0x5570fe6ea790 msgr2=0x5570fe6e8830 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-05T19:28:35.823 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.814+0000 7f783302ee40 1 -- 172.21.15.167:0/3592677366 shutdown_connections 2023-10-05T19:28:35.823 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.814+0000 7f783302ee40 1 -- 172.21.15.167:0/3592677366 wait complete. 2023-10-05T19:28:35.825 INFO:tasks.util.rgw:rgwadmin: client.0 : ['bucket', 'stats', '--bucket', 'myfoo'] 2023-10-05T19:28:35.825 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'ceph', 'bucket', 'stats', '--bucket', 'myfoo'] 2023-10-05T19:28:35.825 DEBUG:teuthology.orchestra.run.smithi167:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster ceph bucket stats --bucket myfoo 2023-10-05T19:28:35.864 INFO:teuthology.orchestra.run.smithi167.stderr:ignoring --setuser ceph since I am not root 2023-10-05T19:28:35.864 INFO:teuthology.orchestra.run.smithi167.stderr:ignoring --setgroup ceph since I am not root 2023-10-05T19:28:35.865 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.858+0000 7f1428156e40 1 Processor -- start 2023-10-05T19:28:35.865 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.858+0000 7f1428156e40 1 -- start start 2023-10-05T19:28:35.865 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.858+0000 7f1428156e40 1 --2- >> v2:172.21.15.118:3300/0 conn(0x564a522ba5f0 0x564a522bc9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-05T19:28:35.865 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.858+0000 7f1428156e40 1 -- --> v1:172.21.15.118:6789/0 -- auth(proto 0 26 bytes epoch 0) v1 -- 0x564a52200bd0 con 0x564a522b6010 2023-10-05T19:28:35.865 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.858+0000 7f1428156e40 1 -- --> v2:172.21.15.118:3300/0 -- mon_getmap magic: 0 v1 -- 0x564a521bdee0 con 0x564a522ba5f0 2023-10-05T19:28:35.866 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.858+0000 7f142548a700 1 --2- >> v2:172.21.15.118:3300/0 conn(0x564a522ba5f0 0x564a522bc9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-05T19:28:35.866 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.858+0000 7f142548a700 1 --2- >> v2:172.21.15.118:3300/0 conn(0x564a522ba5f0 0x564a522bc9d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.118:3300/0 says I am v2:172.21.15.167:41694/0 (socket says 172.21.15.167:41694) 2023-10-05T19:28:35.866 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.858+0000 7f142548a700 1 -- 172.21.15.167:0/1786171337 learned_addr learned my addr 172.21.15.167:0/1786171337 (peer_addr_for_me v2:172.21.15.167:0/0) 2023-10-05T19:28:35.866 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.858+0000 7f1425c8b700 1 --1- 172.21.15.167:0/1786171337 >> v1:172.21.15.118:6789/0 conn(0x564a522b6010 0x564a522b63e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.118:6789/0 says I am v1:172.21.15.167:59298/0 (socket says 172.21.15.167:59298) 2023-10-05T19:28:35.866 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.862+0000 7f142548a700 1 -- 172.21.15.167:0/1786171337 >> v1:172.21.15.118:6789/0 conn(0x564a522b6010 legacy=0x564a522b63e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-05T19:28:35.866 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.862+0000 7f142548a700 1 -- 172.21.15.167:0/1786171337 --> v2:172.21.15.118:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x564a522b6c20 con 0x564a522ba5f0 2023-10-05T19:28:35.867 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.862+0000 7f142548a700 1 --2- 172.21.15.167:0/1786171337 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x564a522ba5f0 0x564a522bc9d0 secure :-1 s=READY pgs=429 cs=0 l=1 rev1=1 crypto rx=0x7f141000a5c0 tx=0x7f1410005e00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3241ee40c3fad8a4 server_cookie=0 in_seq=0 out_seq=0 2023-10-05T19:28:35.867 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.862+0000 7f1424c89700 1 -- 172.21.15.167:0/1786171337 <== mon.0 v2:172.21.15.118:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f141000e510 con 0x564a522ba5f0 2023-10-05T19:28:35.868 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.862+0000 7f1424c89700 1 -- 172.21.15.167:0/1786171337 <== mon.0 v2:172.21.15.118:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f141000e670 con 0x564a522ba5f0 2023-10-05T19:28:35.868 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.862+0000 7f1424c89700 1 -- 172.21.15.167:0/1786171337 <== mon.0 v2:172.21.15.118:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f141000e920 con 0x564a522ba5f0 2023-10-05T19:28:35.868 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.862+0000 7f1428156e40 1 -- 172.21.15.167:0/1786171337 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x564a522ba5f0 msgr2=0x564a522bc9d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-05T19:28:35.868 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.862+0000 7f1428156e40 1 --2- 172.21.15.167:0/1786171337 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x564a522ba5f0 0x564a522bc9d0 secure :-1 s=READY pgs=429 cs=0 l=1 rev1=1 crypto rx=0x7f141000a5c0 tx=0x7f1410005e00 comp rx=0 tx=0).stop 2023-10-05T19:28:35.868 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.862+0000 7f1428156e40 1 -- 172.21.15.167:0/1786171337 shutdown_connections 2023-10-05T19:28:35.868 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.862+0000 7f1428156e40 1 --2- 172.21.15.167:0/1786171337 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x564a522ba5f0 0x564a522bc9d0 unknown :-1 s=CLOSED pgs=429 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-05T19:28:35.868 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.862+0000 7f1428156e40 1 -- 172.21.15.167:0/1786171337 >> 172.21.15.167:0/1786171337 conn(0x564a52229860 msgr2=0x564a52229c60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-05T19:28:35.868 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.862+0000 7f1428156e40 1 -- 172.21.15.167:0/1786171337 shutdown_connections 2023-10-05T19:28:35.869 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.862+0000 7f1428156e40 1 -- 172.21.15.167:0/1786171337 wait complete. 2023-10-05T19:28:35.870 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.862+0000 7f1428156e40 1 Processor -- start 2023-10-05T19:28:35.870 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.862+0000 7f1428156e40 1 -- start start 2023-10-05T19:28:35.870 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.862+0000 7f1428156e40 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x564a522b6010 0x564a522d5190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-05T19:28:35.870 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.862+0000 7f1428156e40 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x564a52181e30 con 0x564a522b6010 2023-10-05T19:28:35.870 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.862+0000 7f1425c8b700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x564a522b6010 0x564a522d5190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-05T19:28:35.870 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.862+0000 7f1425c8b700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x564a522b6010 0x564a522d5190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.118:3300/0 says I am v2:172.21.15.167:41708/0 (socket says 172.21.15.167:41708) 2023-10-05T19:28:35.870 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.862+0000 7f1425c8b700 1 -- 172.21.15.167:0/2219134562 learned_addr learned my addr 172.21.15.167:0/2219134562 (peer_addr_for_me v2:172.21.15.167:0/0) 2023-10-05T19:28:35.870 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.862+0000 7f1425c8b700 1 -- 172.21.15.167:0/2219134562 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f141000a180 con 0x564a522b6010 2023-10-05T19:28:35.870 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.862+0000 7f1425c8b700 1 --2- 172.21.15.167:0/2219134562 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x564a522b6010 0x564a522d5190 secure :-1 s=READY pgs=430 cs=0 l=1 rev1=1 crypto rx=0x7f1420001cf0 tx=0x7f14200018a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=39825f57d67a55ba server_cookie=0 in_seq=0 out_seq=0 2023-10-05T19:28:35.871 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.866+0000 7f141f7fe700 1 -- 172.21.15.167:0/2219134562 <== mon.0 v2:172.21.15.118:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f1420014070 con 0x564a522b6010 2023-10-05T19:28:35.871 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.866+0000 7f141f7fe700 1 -- 172.21.15.167:0/2219134562 <== mon.0 v2:172.21.15.118:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f142000be10 con 0x564a522b6010 2023-10-05T19:28:35.871 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.866+0000 7f141f7fe700 1 -- 172.21.15.167:0/2219134562 <== mon.0 v2:172.21.15.118:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f142000f590 con 0x564a522b6010 2023-10-05T19:28:35.872 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.866+0000 7f1428156e40 1 -- 172.21.15.167:0/2219134562 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x564a522b6010 msgr2=0x564a522d5190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-05T19:28:35.872 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.866+0000 7f1428156e40 1 --2- 172.21.15.167:0/2219134562 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x564a522b6010 0x564a522d5190 secure :-1 s=READY pgs=430 cs=0 l=1 rev1=1 crypto rx=0x7f1420001cf0 tx=0x7f14200018a0 comp rx=0 tx=0).stop 2023-10-05T19:28:35.872 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.866+0000 7f1428156e40 1 -- 172.21.15.167:0/2219134562 shutdown_connections 2023-10-05T19:28:35.872 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.866+0000 7f1428156e40 1 --2- 172.21.15.167:0/2219134562 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x564a522b6010 0x564a522d5190 unknown :-1 s=CLOSED pgs=430 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-05T19:28:35.872 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.866+0000 7f1428156e40 1 -- 172.21.15.167:0/2219134562 >> 172.21.15.167:0/2219134562 conn(0x564a52229860 msgr2=0x564a522d0d50 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-05T19:28:35.872 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.866+0000 7f1428156e40 1 -- 172.21.15.167:0/2219134562 shutdown_connections 2023-10-05T19:28:35.872 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.866+0000 7f1428156e40 1 -- 172.21.15.167:0/2219134562 wait complete. 2023-10-05T19:28:35.873 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.866+0000 7f1428156e40 1 Processor -- start 2023-10-05T19:28:35.873 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.866+0000 7f1428156e40 1 -- start start 2023-10-05T19:28:35.873 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.866+0000 7f1428156e40 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x564a522b6010 0x564a522dbf60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-05T19:28:35.873 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.866+0000 7f1428156e40 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x564a52154ed0 con 0x564a522b6010 2023-10-05T19:28:35.874 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.866+0000 7f1425c8b700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x564a522b6010 0x564a522dbf60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-05T19:28:35.874 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.866+0000 7f1425c8b700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x564a522b6010 0x564a522dbf60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.118:3300/0 says I am v2:172.21.15.167:41718/0 (socket says 172.21.15.167:41718) 2023-10-05T19:28:35.874 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.866+0000 7f1425c8b700 1 -- 172.21.15.167:0/3011814457 learned_addr learned my addr 172.21.15.167:0/3011814457 (peer_addr_for_me v2:172.21.15.167:0/0) 2023-10-05T19:28:35.875 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.870+0000 7f1425c8b700 1 -- 172.21.15.167:0/3011814457 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x564a52200bd0 con 0x564a522b6010 2023-10-05T19:28:35.875 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.870+0000 7f1425c8b700 1 --2- 172.21.15.167:0/3011814457 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x564a522b6010 0x564a522dbf60 secure :-1 s=READY pgs=431 cs=0 l=1 rev1=1 crypto rx=0x7f14200018e0 tx=0x7f1420016000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-05T19:28:35.875 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.870+0000 7f141dffb700 1 -- 172.21.15.167:0/3011814457 <== mon.0 v2:172.21.15.118:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f142001c740 con 0x564a522b6010 2023-10-05T19:28:35.876 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.870+0000 7f141dffb700 1 -- 172.21.15.167:0/3011814457 <== mon.0 v2:172.21.15.118:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f142001c8a0 con 0x564a522b6010 2023-10-05T19:28:35.876 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.870+0000 7f1428156e40 1 -- 172.21.15.167:0/3011814457 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x564a522dc4a0 con 0x564a522b6010 2023-10-05T19:28:35.876 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.870+0000 7f141dffb700 1 -- 172.21.15.167:0/3011814457 <== mon.0 v2:172.21.15.118:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f142001cb90 con 0x564a522b6010 2023-10-05T19:28:35.876 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.870+0000 7f1428156e40 1 -- 172.21.15.167:0/3011814457 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x564a522dc8e0 con 0x564a522b6010 2023-10-05T19:28:35.877 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.870+0000 7f141dffb700 1 -- 172.21.15.167:0/3011814457 <== mon.0 v2:172.21.15.118:3300/0 4 ==== mgrmap(e 5) v1 ==== 40329+0+0 (secure 0 0 0) 0x7f1420023020 con 0x564a522b6010 2023-10-05T19:28:35.877 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.870+0000 7f1428156e40 20 rados->read ofs=0 len=0 2023-10-05T19:28:35.878 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.870+0000 7f141dffb700 1 --2- 172.21.15.167:0/3011814457 >> [v2:172.21.15.167:6824/29889,v1:172.21.15.167:6825/29889] conn(0x7f140c030c50 0x7f140c033100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-05T19:28:35.878 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.870+0000 7f141dffb700 1 -- 172.21.15.167:0/3011814457 <== mon.0 v2:172.21.15.118:3300/0 5 ==== osd_map(30..30 src has 1..30) v4 ==== 6057+0+0 (secure 0 0 0) 0x7f1420008030 con 0x564a522b6010 2023-10-05T19:28:35.878 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.874+0000 7f142548a700 1 --2- 172.21.15.167:0/3011814457 >> [v2:172.21.15.167:6824/29889,v1:172.21.15.167:6825/29889] conn(0x7f140c030c50 0x7f140c033100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-05T19:28:35.878 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.874+0000 7f1428156e40 1 --2- 172.21.15.167:0/3011814457 >> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] conn(0x564a522ba5f0 0x564a522e7a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-05T19:28:35.879 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.874+0000 7f1428156e40 1 -- 172.21.15.167:0/3011814457 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:1 5.12 5:49953fa1:::default.realm:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x564a522e8120 con 0x564a522ba5f0 2023-10-05T19:28:35.879 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.874+0000 7f142648c700 1 --2- 172.21.15.167:0/3011814457 >> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] conn(0x564a522ba5f0 0x564a522e7a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-05T19:28:35.879 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.874+0000 7f142548a700 1 --2- 172.21.15.167:0/3011814457 >> [v2:172.21.15.167:6824/29889,v1:172.21.15.167:6825/29889] conn(0x7f140c030c50 0x7f140c033100 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7f1410000f80 tx=0x7f1410013000 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-05T19:28:35.879 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.874+0000 7f142648c700 1 --2- 172.21.15.167:0/3011814457 >> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] conn(0x564a522ba5f0 0x564a522e7a20 crc :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-05T19:28:35.880 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.874+0000 7f142648c700 1 -- 172.21.15.167:0/3011814457 <== osd.1 v2:172.21.15.167:6800/30035 1 ==== osd_op_reply(1 default.realm [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 ==== 157+0+0 (crc 0 0 0) 0x7f141800a500 con 0x564a522ba5f0 2023-10-05T19:28:35.880 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.874+0000 7f1428156e40 20 rados_obj.operate() r=-2 bl.length=0 2023-10-05T19:28:35.880 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.874+0000 7f1428156e40 20 realm 2023-10-05T19:28:35.880 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.874+0000 7f1428156e40 20 rados->read ofs=0 len=0 2023-10-05T19:28:35.880 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.874+0000 7f1428156e40 1 -- 172.21.15.167:0/3011814457 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:2 5.12 5:49953fa1:::default.realm:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x564a522e5080 con 0x564a522ba5f0 2023-10-05T19:28:35.881 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.874+0000 7f142648c700 1 -- 172.21.15.167:0/3011814457 <== osd.1 v2:172.21.15.167:6800/30035 2 ==== osd_op_reply(2 default.realm [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 ==== 157+0+0 (crc 0 0 0) 0x7f141800a500 con 0x564a522ba5f0 2023-10-05T19:28:35.881 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.874+0000 7f1428156e40 20 rados_obj.operate() r=-2 bl.length=0 2023-10-05T19:28:35.881 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.874+0000 7f1428156e40 4 RGWPeriod::init failed to init realm id : (2) No such file or directory 2023-10-05T19:28:35.882 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.874+0000 7f1428156e40 20 rados->read ofs=0 len=0 2023-10-05T19:28:35.882 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.874+0000 7f1428156e40 1 -- 172.21.15.167:0/3011814457 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:3 5.12 5:49953fa1:::default.realm:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x564a522e8c00 con 0x564a522ba5f0 2023-10-05T19:28:35.882 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.874+0000 7f142648c700 1 -- 172.21.15.167:0/3011814457 <== osd.1 v2:172.21.15.167:6800/30035 3 ==== osd_op_reply(3 default.realm [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 ==== 157+0+0 (crc 0 0 0) 0x7f141800a500 con 0x564a522ba5f0 2023-10-05T19:28:35.882 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.874+0000 7f1428156e40 20 rados_obj.operate() r=-2 bl.length=0 2023-10-05T19:28:35.882 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.874+0000 7f1428156e40 20 rados->read ofs=0 len=0 2023-10-05T19:28:35.882 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.874+0000 7f1428156e40 1 -- 172.21.15.167:0/3011814457 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:4 5.f 5:f43b6ece:::zone_names.default:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x564a522e87e0 con 0x564a522ba5f0 2023-10-05T19:28:35.883 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.874+0000 7f142648c700 1 -- 172.21.15.167:0/3011814457 <== osd.1 v2:172.21.15.167:6800/30035 4 ==== osd_op_reply(4 zone_names.default [read 0~46 out=46b] v0'0 uv1 ondisk = 0) v8 ==== 162+0+46 (crc 0 0 0) 0x7f141800a500 con 0x564a522ba5f0 2023-10-05T19:28:35.883 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.874+0000 7f1428156e40 20 rados_obj.operate() r=0 bl.length=46 2023-10-05T19:28:35.883 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.874+0000 7f1428156e40 20 rados->read ofs=0 len=0 2023-10-05T19:28:35.883 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.874+0000 7f1428156e40 1 -- 172.21.15.167:0/3011814457 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:5 5.1d 5:b8374eb4:::zone_info.9a9df965-6612-45d4-a45f-6e3c889bacc2:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x564a522e9670 con 0x564a522ba5f0 2023-10-05T19:28:35.884 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.878+0000 7f142648c700 1 -- 172.21.15.167:0/3011814457 <== osd.1 v2:172.21.15.167:6800/30035 5 ==== osd_op_reply(5 zone_info.9a9df965-6612-45d4-a45f-6e3c889bacc2 [read 0~874 out=874b] v0'0 uv2 ondisk = 0) v8 ==== 190+0+874 (crc 0 0 0) 0x7f141800a500 con 0x564a522ba5f0 2023-10-05T19:28:35.884 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.878+0000 7f1428156e40 20 rados_obj.operate() r=0 bl.length=874 2023-10-05T19:28:35.884 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.878+0000 7f1428156e40 20 searching for the correct realm 2023-10-05T19:28:35.884 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.878+0000 7f1428156e40 1 --2- 172.21.15.167:0/3011814457 >> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] conn(0x564a522eb4c0 0x564a522ed8a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-05T19:28:35.884 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.878+0000 7f1428156e40 1 -- 172.21.15.167:0/3011814457 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:6 5.0 5:00000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x564a522edde0 con 0x564a522eb4c0 2023-10-05T19:28:35.884 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.878+0000 7f1425c8b700 1 --2- 172.21.15.167:0/3011814457 >> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] conn(0x564a522eb4c0 0x564a522ed8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-05T19:28:35.884 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.878+0000 7f1425c8b700 1 --2- 172.21.15.167:0/3011814457 >> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] conn(0x564a522eb4c0 0x564a522ed8a0 crc :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-05T19:28:35.886 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.878+0000 7f1425c8b700 1 -- 172.21.15.167:0/3011814457 <== osd.3 v2:172.21.15.167:6808/30036 1 ==== osd_op_reply(6 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f1420016070 con 0x564a522eb4c0 2023-10-05T19:28:35.886 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.878+0000 7f1425c8b700 1 --2- 172.21.15.167:0/3011814457 >> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] conn(0x7f142001a070 0x7f1420043730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-05T19:28:35.886 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.878+0000 7f1425c8b700 1 -- 172.21.15.167:0/3011814457 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:7 5.10 5:08000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x564a522edde0 con 0x7f142001a070 2023-10-05T19:28:35.886 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.878+0000 7f142548a700 1 --2- 172.21.15.167:0/3011814457 >> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] conn(0x7f142001a070 0x7f1420043730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-05T19:28:35.886 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.882+0000 7f142548a700 1 --2- 172.21.15.167:0/3011814457 >> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] conn(0x7f142001a070 0x7f1420043730 crc :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-05T19:28:35.887 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.882+0000 7f142548a700 1 -- 172.21.15.167:0/3011814457 <== osd.2 v2:172.21.15.167:6816/30037 1 ==== osd_op_reply(7 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f1410017050 con 0x7f142001a070 2023-10-05T19:28:35.887 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.882+0000 7f142548a700 1 --2- 172.21.15.167:0/3011814457 >> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] conn(0x7f1410010070 0x7f141001c430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-05T19:28:35.887 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.882+0000 7f142548a700 1 -- 172.21.15.167:0/3011814457 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:8 5.8 5:10000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x564a522edde0 con 0x7f1410010070 2023-10-05T19:28:35.887 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.882+0000 7f142648c700 1 --2- 172.21.15.167:0/3011814457 >> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] conn(0x7f1410010070 0x7f141001c430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-05T19:28:35.888 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.882+0000 7f142648c700 1 --2- 172.21.15.167:0/3011814457 >> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] conn(0x7f1410010070 0x7f141001c430 crc :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-05T19:28:35.889 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.882+0000 7f142648c700 1 -- 172.21.15.167:0/3011814457 <== osd.0 v2:172.21.15.118:6800/27690 1 ==== osd_op_reply(8 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f1418014040 con 0x7f1410010070 2023-10-05T19:28:35.889 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.882+0000 7f142648c700 1 -- 172.21.15.167:0/3011814457 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:9 5.18 5:18000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x564a522edde0 con 0x564a522ba5f0 2023-10-05T19:28:35.889 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.882+0000 7f142648c700 1 -- 172.21.15.167:0/3011814457 <== osd.1 v2:172.21.15.167:6800/30035 6 ==== osd_op_reply(9 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f1418014040 con 0x564a522ba5f0 2023-10-05T19:28:35.890 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.882+0000 7f142648c700 1 -- 172.21.15.167:0/3011814457 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:10 5.4 5:20000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x564a522edde0 con 0x564a522ba5f0 2023-10-05T19:28:35.890 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.886+0000 7f142648c700 1 -- 172.21.15.167:0/3011814457 <== osd.1 v2:172.21.15.167:6800/30035 7 ==== osd_op_reply(10 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f1418014040 con 0x564a522ba5f0 2023-10-05T19:28:35.890 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.886+0000 7f142648c700 1 -- 172.21.15.167:0/3011814457 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:11 5.14 5:28000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x564a522edde0 con 0x7f1410010070 2023-10-05T19:28:35.890 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.886+0000 7f142648c700 1 -- 172.21.15.167:0/3011814457 <== osd.0 v2:172.21.15.118:6800/27690 2 ==== osd_op_reply(11 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f1418014040 con 0x7f1410010070 2023-10-05T19:28:35.891 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.886+0000 7f142648c700 1 -- 172.21.15.167:0/3011814457 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:12 5.c 5:30000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x564a522edde0 con 0x7f1410010070 2023-10-05T19:28:35.891 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.886+0000 7f142648c700 1 -- 172.21.15.167:0/3011814457 <== osd.0 v2:172.21.15.118:6800/27690 3 ==== osd_op_reply(12 [pgnls start_epoch 30 out=112b] v23'1 uv1 ondisk = 1) v8 ==== 144+0+112 (crc 0 0 0) 0x7f1418014040 con 0x7f1410010070 2023-10-05T19:28:35.891 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.886+0000 7f142648c700 1 -- 172.21.15.167:0/3011814457 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:13 5.1c 5:38000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x564a522edde0 con 0x7f142001a070 2023-10-05T19:28:35.892 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.886+0000 7f142548a700 1 -- 172.21.15.167:0/3011814457 <== osd.2 v2:172.21.15.167:6816/30037 2 ==== osd_op_reply(13 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f1410017050 con 0x7f142001a070 2023-10-05T19:28:35.892 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.886+0000 7f142548a700 1 -- 172.21.15.167:0/3011814457 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:14 5.2 5:40000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x564a522edde0 con 0x564a522eb4c0 2023-10-05T19:28:35.892 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.886+0000 7f1425c8b700 1 -- 172.21.15.167:0/3011814457 <== osd.3 v2:172.21.15.167:6808/30036 2 ==== osd_op_reply(14 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f1420016070 con 0x564a522eb4c0 2023-10-05T19:28:35.893 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.886+0000 7f1425c8b700 1 -- 172.21.15.167:0/3011814457 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:15 5.12 5:48000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x564a522edde0 con 0x564a522ba5f0 2023-10-05T19:28:35.893 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.886+0000 7f142648c700 1 -- 172.21.15.167:0/3011814457 <== osd.1 v2:172.21.15.167:6800/30035 8 ==== osd_op_reply(15 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f1418014040 con 0x564a522ba5f0 2023-10-05T19:28:35.893 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.886+0000 7f142648c700 1 -- 172.21.15.167:0/3011814457 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:16 5.a 5:50000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x564a522edde0 con 0x7f1410010070 2023-10-05T19:28:35.894 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.890+0000 7f142648c700 1 -- 172.21.15.167:0/3011814457 <== osd.0 v2:172.21.15.118:6800/27690 4 ==== osd_op_reply(16 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f1418022040 con 0x7f1410010070 2023-10-05T19:28:35.895 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.890+0000 7f142648c700 1 -- 172.21.15.167:0/3011814457 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:17 5.1a 5:58000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x564a522edde0 con 0x564a522ba5f0 2023-10-05T19:28:35.895 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.890+0000 7f142648c700 1 -- 172.21.15.167:0/3011814457 <== osd.1 v2:172.21.15.167:6800/30035 9 ==== osd_op_reply(17 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f1418022040 con 0x564a522ba5f0 2023-10-05T19:28:35.895 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.890+0000 7f142648c700 1 -- 172.21.15.167:0/3011814457 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:18 5.6 5:60000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x564a522edde0 con 0x7f142001a070 2023-10-05T19:28:35.896 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.890+0000 7f142548a700 1 -- 172.21.15.167:0/3011814457 <== osd.2 v2:172.21.15.167:6816/30037 3 ==== osd_op_reply(18 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f1410017050 con 0x7f142001a070 2023-10-05T19:28:35.896 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.890+0000 7f142548a700 1 -- 172.21.15.167:0/3011814457 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:19 5.16 5:68000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x564a522edde0 con 0x564a522eb4c0 2023-10-05T19:28:35.896 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.890+0000 7f1425c8b700 1 -- 172.21.15.167:0/3011814457 <== osd.3 v2:172.21.15.167:6808/30036 3 ==== osd_op_reply(19 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f1420016070 con 0x564a522eb4c0 2023-10-05T19:28:35.897 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.890+0000 7f1425c8b700 1 -- 172.21.15.167:0/3011814457 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:20 5.e 5:70000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x564a522edde0 con 0x564a522eb4c0 2023-10-05T19:28:35.897 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.890+0000 7f1425c8b700 1 -- 172.21.15.167:0/3011814457 <== osd.3 v2:172.21.15.167:6808/30036 4 ==== osd_op_reply(20 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f1420016070 con 0x564a522eb4c0 2023-10-05T19:28:35.897 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.890+0000 7f1425c8b700 1 -- 172.21.15.167:0/3011814457 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:21 5.1e 5:78000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x564a522edde0 con 0x7f1410010070 2023-10-05T19:28:35.898 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.894+0000 7f142648c700 1 -- 172.21.15.167:0/3011814457 <== osd.0 v2:172.21.15.118:6800/27690 5 ==== osd_op_reply(21 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f1418022040 con 0x7f1410010070 2023-10-05T19:28:35.898 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.894+0000 7f142648c700 1 -- 172.21.15.167:0/3011814457 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:22 5.1 5:80000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x564a522edde0 con 0x564a522eb4c0 2023-10-05T19:28:35.899 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.894+0000 7f1425c8b700 1 -- 172.21.15.167:0/3011814457 <== osd.3 v2:172.21.15.167:6808/30036 5 ==== osd_op_reply(22 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f1420016070 con 0x564a522eb4c0 2023-10-05T19:28:35.899 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.894+0000 7f1425c8b700 1 -- 172.21.15.167:0/3011814457 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:23 5.11 5:88000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x564a522edde0 con 0x564a522eb4c0 2023-10-05T19:28:35.900 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.894+0000 7f1425c8b700 1 -- 172.21.15.167:0/3011814457 <== osd.3 v2:172.21.15.167:6808/30036 6 ==== osd_op_reply(23 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f1420016070 con 0x564a522eb4c0 2023-10-05T19:28:35.900 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.894+0000 7f1425c8b700 1 -- 172.21.15.167:0/3011814457 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:24 5.9 5:90000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x564a522edde0 con 0x564a522ba5f0 2023-10-05T19:28:35.901 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.894+0000 7f142648c700 1 -- 172.21.15.167:0/3011814457 <== osd.1 v2:172.21.15.167:6800/30035 10 ==== osd_op_reply(24 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f1418022040 con 0x564a522ba5f0 2023-10-05T19:28:35.901 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.894+0000 7f142648c700 1 -- 172.21.15.167:0/3011814457 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:25 5.19 5:98000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x564a522edde0 con 0x564a522ba5f0 2023-10-05T19:28:35.901 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.894+0000 7f142648c700 1 -- 172.21.15.167:0/3011814457 <== osd.1 v2:172.21.15.167:6800/30035 11 ==== osd_op_reply(25 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f1418022040 con 0x564a522ba5f0 2023-10-05T19:28:35.901 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.894+0000 7f142648c700 1 -- 172.21.15.167:0/3011814457 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:26 5.5 5:a0000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x564a522edde0 con 0x7f1410010070 2023-10-05T19:28:35.902 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.898+0000 7f142648c700 1 -- 172.21.15.167:0/3011814457 <== osd.0 v2:172.21.15.118:6800/27690 6 ==== osd_op_reply(26 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f1418022040 con 0x7f1410010070 2023-10-05T19:28:35.902 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.898+0000 7f142648c700 1 -- 172.21.15.167:0/3011814457 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:27 5.15 5:a8000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x564a522edde0 con 0x564a522eb4c0 2023-10-05T19:28:35.903 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.898+0000 7f1425c8b700 1 -- 172.21.15.167:0/3011814457 <== osd.3 v2:172.21.15.167:6808/30036 7 ==== osd_op_reply(27 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f1420016070 con 0x564a522eb4c0 2023-10-05T19:28:35.903 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.898+0000 7f1425c8b700 1 -- 172.21.15.167:0/3011814457 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:28 5.d 5:b0000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x564a522edde0 con 0x7f142001a070 2023-10-05T19:28:35.904 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.898+0000 7f142548a700 1 -- 172.21.15.167:0/3011814457 <== osd.2 v2:172.21.15.167:6816/30037 4 ==== osd_op_reply(28 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f1410017050 con 0x7f142001a070 2023-10-05T19:28:35.904 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.898+0000 7f142548a700 1 -- 172.21.15.167:0/3011814457 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:29 5.1d 5:b8000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x564a522edde0 con 0x564a522ba5f0 2023-10-05T19:28:35.905 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.898+0000 7f142648c700 1 -- 172.21.15.167:0/3011814457 <== osd.1 v2:172.21.15.167:6800/30035 12 ==== osd_op_reply(29 [pgnls start_epoch 30 out=107b] v30'2 uv2 ondisk = 1) v8 ==== 144+0+107 (crc 0 0 0) 0x7f1418022040 con 0x564a522ba5f0 2023-10-05T19:28:35.905 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.898+0000 7f142648c700 1 -- 172.21.15.167:0/3011814457 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:30 5.3 5:c0000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x564a522edde0 con 0x7f1410010070 2023-10-05T19:28:35.906 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.902+0000 7f142648c700 1 -- 172.21.15.167:0/3011814457 <== osd.0 v2:172.21.15.118:6800/27690 7 ==== osd_op_reply(30 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f1418022040 con 0x7f1410010070 2023-10-05T19:28:35.906 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.902+0000 7f142648c700 1 -- 172.21.15.167:0/3011814457 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:31 5.13 5:c8000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x564a522edde0 con 0x564a522eb4c0 2023-10-05T19:28:35.907 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.902+0000 7f1425c8b700 1 -- 172.21.15.167:0/3011814457 <== osd.3 v2:172.21.15.167:6808/30036 8 ==== osd_op_reply(31 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f1420016070 con 0x564a522eb4c0 2023-10-05T19:28:35.907 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.902+0000 7f1425c8b700 1 -- 172.21.15.167:0/3011814457 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:32 5.b 5:d0000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x564a522edde0 con 0x7f1410010070 2023-10-05T19:28:35.907 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.902+0000 7f142648c700 1 -- 172.21.15.167:0/3011814457 <== osd.0 v2:172.21.15.118:6800/27690 8 ==== osd_op_reply(32 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f1418022040 con 0x7f1410010070 2023-10-05T19:28:35.908 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.902+0000 7f142648c700 1 -- 172.21.15.167:0/3011814457 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:33 5.1b 5:d8000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x564a522edde0 con 0x7f142001a070 2023-10-05T19:28:35.908 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.902+0000 7f142548a700 1 -- 172.21.15.167:0/3011814457 <== osd.2 v2:172.21.15.167:6816/30037 5 ==== osd_op_reply(33 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f1410017050 con 0x7f142001a070 2023-10-05T19:28:35.908 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.902+0000 7f142548a700 1 -- 172.21.15.167:0/3011814457 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:34 5.7 5:e0000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x564a522edde0 con 0x564a522eb4c0 2023-10-05T19:28:35.909 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.902+0000 7f1425c8b700 1 -- 172.21.15.167:0/3011814457 <== osd.3 v2:172.21.15.167:6808/30036 9 ==== osd_op_reply(34 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f1420016070 con 0x564a522eb4c0 2023-10-05T19:28:35.909 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.902+0000 7f1425c8b700 1 -- 172.21.15.167:0/3011814457 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:35 5.17 5:e8000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x564a522edde0 con 0x7f1410010070 2023-10-05T19:28:35.910 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.906+0000 7f142648c700 1 -- 172.21.15.167:0/3011814457 <== osd.0 v2:172.21.15.118:6800/27690 9 ==== osd_op_reply(35 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f1418022040 con 0x7f1410010070 2023-10-05T19:28:35.910 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.906+0000 7f142648c700 1 -- 172.21.15.167:0/3011814457 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:36 5.f 5:f0000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x564a522edde0 con 0x564a522ba5f0 2023-10-05T19:28:35.911 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.906+0000 7f142648c700 1 -- 172.21.15.167:0/3011814457 <== osd.1 v2:172.21.15.167:6800/30035 13 ==== osd_op_reply(36 [pgnls start_epoch 30 out=115b] v23'2 uv2 ondisk = 1) v8 ==== 144+0+115 (crc 0 0 0) 0x7f141802f080 con 0x564a522ba5f0 2023-10-05T19:28:35.911 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.906+0000 7f142648c700 1 -- 172.21.15.167:0/3011814457 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:37 5.1f 5:f8000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x564a522edde0 con 0x7f142001a070 2023-10-05T19:28:35.912 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.906+0000 7f142548a700 1 -- 172.21.15.167:0/3011814457 <== osd.2 v2:172.21.15.167:6816/30037 6 ==== osd_op_reply(37 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f1410017050 con 0x7f142001a070 2023-10-05T19:28:35.912 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.906+0000 7f1428156e40 20 RGWRados::pool_iterate: got zonegroup_info.4f1f719d-9648-417b-b10c-e5d939825419 2023-10-05T19:28:35.912 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.906+0000 7f1428156e40 20 RGWRados::pool_iterate: got zone_info.9a9df965-6612-45d4-a45f-6e3c889bacc2 2023-10-05T19:28:35.912 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.906+0000 7f1428156e40 20 RGWRados::pool_iterate: got zone_names.default 2023-10-05T19:28:35.912 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.906+0000 7f1428156e40 20 RGWRados::pool_iterate: got zonegroups_names.default 2023-10-05T19:28:35.912 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.906+0000 7f1428156e40 20 rados->read ofs=0 len=0 2023-10-05T19:28:35.913 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.906+0000 7f1428156e40 1 -- 172.21.15.167:0/3011814457 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:38 5.12 5:49953fa1:::default.realm:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x564a522ea270 con 0x564a522ba5f0 2023-10-05T19:28:35.913 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.906+0000 7f142648c700 1 -- 172.21.15.167:0/3011814457 <== osd.1 v2:172.21.15.167:6800/30035 14 ==== osd_op_reply(38 default.realm [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 ==== 157+0+0 (crc 0 0 0) 0x7f141802f080 con 0x564a522ba5f0 2023-10-05T19:28:35.914 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.906+0000 7f1428156e40 20 rados_obj.operate() r=-2 bl.length=0 2023-10-05T19:28:35.914 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.906+0000 7f1428156e40 20 rados->read ofs=0 len=0 2023-10-05T19:28:35.914 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.906+0000 7f1428156e40 1 -- 172.21.15.167:0/3011814457 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:39 5.f 5:f4c53578:::zonegroups_names.default:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x564a522eabf0 con 0x564a522ba5f0 2023-10-05T19:28:35.914 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.906+0000 7f142648c700 1 -- 172.21.15.167:0/3011814457 <== osd.1 v2:172.21.15.167:6800/30035 15 ==== osd_op_reply(39 zonegroups_names.default [read 0~46 out=46b] v0'0 uv2 ondisk = 0) v8 ==== 168+0+46 (crc 0 0 0) 0x7f141802f080 con 0x564a522ba5f0 2023-10-05T19:28:35.914 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.906+0000 7f1428156e40 20 rados_obj.operate() r=0 bl.length=46 2023-10-05T19:28:35.914 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.906+0000 7f1428156e40 20 rados->read ofs=0 len=0 2023-10-05T19:28:35.914 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.906+0000 7f1428156e40 1 -- 172.21.15.167:0/3011814457 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:40 5.c 5:30d299a5:::zonegroup_info.4f1f719d-9648-417b-b10c-e5d939825419:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x564a522e9e60 con 0x7f1410010070 2023-10-05T19:28:35.915 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.910+0000 7f142648c700 1 -- 172.21.15.167:0/3011814457 <== osd.0 v2:172.21.15.118:6800/27690 10 ==== osd_op_reply(40 zonegroup_info.4f1f719d-9648-417b-b10c-e5d939825419 [read 0~362 out=362b] v0'0 uv1 ondisk = 0) v8 ==== 195+0+362 (crc 0 0 0) 0x7f141802f080 con 0x7f1410010070 2023-10-05T19:28:35.915 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.910+0000 7f1428156e40 20 rados_obj.operate() r=0 bl.length=362 2023-10-05T19:28:35.915 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.910+0000 7f1428156e40 20 rados->read ofs=0 len=0 2023-10-05T19:28:35.915 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.910+0000 7f1428156e40 1 -- 172.21.15.167:0/3011814457 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:41 5.13 5:cbbf6c60:::converted:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x564a522ee6d0 con 0x564a522eb4c0 2023-10-05T19:28:35.916 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.910+0000 7f1425c8b700 1 -- 172.21.15.167:0/3011814457 <== osd.3 v2:172.21.15.167:6808/30036 10 ==== osd_op_reply(41 converted [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 ==== 153+0+0 (crc 0 0 0) 0x7f1420016070 con 0x564a522eb4c0 2023-10-05T19:28:35.916 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.910+0000 7f1428156e40 20 rados_obj.operate() r=-2 bl.length=0 2023-10-05T19:28:35.916 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.910+0000 7f1428156e40 20 rados->read ofs=0 len=0 2023-10-05T19:28:35.916 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.910+0000 7f1428156e40 1 -- 172.21.15.167:0/3011814457 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:42 5.12 5:49953fa1:::default.realm:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x564a522eea80 con 0x564a522ba5f0 2023-10-05T19:28:35.916 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.910+0000 7f142648c700 1 -- 172.21.15.167:0/3011814457 <== osd.1 v2:172.21.15.167:6800/30035 16 ==== osd_op_reply(42 default.realm [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 ==== 157+0+0 (crc 0 0 0) 0x7f141802f080 con 0x564a522ba5f0 2023-10-05T19:28:35.916 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.910+0000 7f1428156e40 20 rados_obj.operate() r=-2 bl.length=0 2023-10-05T19:28:35.916 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.910+0000 7f1428156e40 20 rados->read ofs=0 len=0 2023-10-05T19:28:35.917 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.910+0000 7f1428156e40 1 -- 172.21.15.167:0/3011814457 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:43 5.f 5:f4c53578:::zonegroups_names.default:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x564a522eea80 con 0x564a522ba5f0 2023-10-05T19:28:35.917 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.910+0000 7f142648c700 1 -- 172.21.15.167:0/3011814457 <== osd.1 v2:172.21.15.167:6800/30035 17 ==== osd_op_reply(43 zonegroups_names.default [read 0~46 out=46b] v0'0 uv2 ondisk = 0) v8 ==== 168+0+46 (crc 0 0 0) 0x7f141802f080 con 0x564a522ba5f0 2023-10-05T19:28:35.917 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.910+0000 7f1428156e40 20 rados_obj.operate() r=0 bl.length=46 2023-10-05T19:28:35.917 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.910+0000 7f1428156e40 1 -- 172.21.15.167:0/3011814457 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:44 5.0 5:00000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x564a522efa90 con 0x564a522eb4c0 2023-10-05T19:28:35.922 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.914+0000 7f1425c8b700 1 -- 172.21.15.167:0/3011814457 <== osd.3 v2:172.21.15.167:6808/30036 11 ==== osd_op_reply(44 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f1420016070 con 0x564a522eb4c0 2023-10-05T19:28:35.922 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.914+0000 7f1425c8b700 1 -- 172.21.15.167:0/3011814457 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:45 5.10 5:08000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x564a522efa90 con 0x7f142001a070 2023-10-05T19:28:35.922 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.914+0000 7f142548a700 1 -- 172.21.15.167:0/3011814457 <== osd.2 v2:172.21.15.167:6816/30037 7 ==== osd_op_reply(45 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f1410017050 con 0x7f142001a070 2023-10-05T19:28:35.922 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.914+0000 7f142548a700 1 -- 172.21.15.167:0/3011814457 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:46 5.8 5:10000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x564a522efa90 con 0x7f1410010070 2023-10-05T19:28:35.923 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.914+0000 7f142648c700 1 -- 172.21.15.167:0/3011814457 <== osd.0 v2:172.21.15.118:6800/27690 11 ==== osd_op_reply(46 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f141802f080 con 0x7f1410010070 2023-10-05T19:28:35.923 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.914+0000 7f142648c700 1 -- 172.21.15.167:0/3011814457 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:47 5.18 5:18000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x564a522e9e60 con 0x564a522ba5f0 2023-10-05T19:28:35.923 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.918+0000 7f142648c700 1 -- 172.21.15.167:0/3011814457 <== osd.1 v2:172.21.15.167:6800/30035 18 ==== osd_op_reply(47 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f141802f080 con 0x564a522ba5f0 2023-10-05T19:28:35.923 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.918+0000 7f142648c700 1 -- 172.21.15.167:0/3011814457 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:48 5.4 5:20000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x564a522e9e60 con 0x564a522ba5f0 2023-10-05T19:28:35.923 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.918+0000 7f142648c700 1 -- 172.21.15.167:0/3011814457 <== osd.1 v2:172.21.15.167:6800/30035 19 ==== osd_op_reply(48 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f141802f080 con 0x564a522ba5f0 2023-10-05T19:28:35.923 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.918+0000 7f142648c700 1 -- 172.21.15.167:0/3011814457 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:49 5.14 5:28000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x564a522e9e60 con 0x7f1410010070 2023-10-05T19:28:35.924 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.918+0000 7f142648c700 1 -- 172.21.15.167:0/3011814457 <== osd.0 v2:172.21.15.118:6800/27690 12 ==== osd_op_reply(49 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f141802f080 con 0x7f1410010070 2023-10-05T19:28:35.924 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.918+0000 7f142648c700 1 -- 172.21.15.167:0/3011814457 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:50 5.c 5:30000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x564a522e9e60 con 0x7f1410010070 2023-10-05T19:28:35.925 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.918+0000 7f142648c700 1 -- 172.21.15.167:0/3011814457 <== osd.0 v2:172.21.15.118:6800/27690 13 ==== osd_op_reply(50 [pgnls start_epoch 30 out=112b] v23'1 uv1 ondisk = 1) v8 ==== 144+0+112 (crc 0 0 0) 0x7f141802f080 con 0x7f1410010070 2023-10-05T19:28:35.925 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.918+0000 7f142648c700 1 -- 172.21.15.167:0/3011814457 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:51 5.1c 5:38000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x564a522e9e60 con 0x7f142001a070 2023-10-05T19:28:35.926 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.918+0000 7f142548a700 1 -- 172.21.15.167:0/3011814457 <== osd.2 v2:172.21.15.167:6816/30037 8 ==== osd_op_reply(51 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f1410017050 con 0x7f142001a070 2023-10-05T19:28:35.926 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.918+0000 7f142548a700 1 -- 172.21.15.167:0/3011814457 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:52 5.2 5:40000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x564a522e9e60 con 0x564a522eb4c0 2023-10-05T19:28:35.926 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.922+0000 7f1425c8b700 1 -- 172.21.15.167:0/3011814457 <== osd.3 v2:172.21.15.167:6808/30036 12 ==== osd_op_reply(52 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f1420016070 con 0x564a522eb4c0 2023-10-05T19:28:35.926 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.922+0000 7f1425c8b700 1 -- 172.21.15.167:0/3011814457 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:53 5.12 5:48000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x564a522e9e60 con 0x564a522ba5f0 2023-10-05T19:28:35.927 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.922+0000 7f142648c700 1 -- 172.21.15.167:0/3011814457 <== osd.1 v2:172.21.15.167:6800/30035 20 ==== osd_op_reply(53 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f141802f080 con 0x564a522ba5f0 2023-10-05T19:28:35.927 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.922+0000 7f142648c700 1 -- 172.21.15.167:0/3011814457 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:54 5.a 5:50000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x564a522e9e60 con 0x7f1410010070 2023-10-05T19:28:35.928 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.922+0000 7f142648c700 1 -- 172.21.15.167:0/3011814457 <== osd.0 v2:172.21.15.118:6800/27690 14 ==== osd_op_reply(54 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f141802f080 con 0x7f1410010070 2023-10-05T19:28:35.928 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.922+0000 7f142648c700 1 -- 172.21.15.167:0/3011814457 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:55 5.1a 5:58000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x564a522e9e60 con 0x564a522ba5f0 2023-10-05T19:28:35.929 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.922+0000 7f142648c700 1 -- 172.21.15.167:0/3011814457 <== osd.1 v2:172.21.15.167:6800/30035 21 ==== osd_op_reply(55 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f141802f080 con 0x564a522ba5f0 2023-10-05T19:28:35.929 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.922+0000 7f142648c700 1 -- 172.21.15.167:0/3011814457 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:56 5.6 5:60000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x564a522e9e60 con 0x7f142001a070 2023-10-05T19:28:35.930 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.922+0000 7f142548a700 1 -- 172.21.15.167:0/3011814457 <== osd.2 v2:172.21.15.167:6816/30037 9 ==== osd_op_reply(56 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f1410017050 con 0x7f142001a070 2023-10-05T19:28:35.930 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.926+0000 7f142548a700 1 -- 172.21.15.167:0/3011814457 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:57 5.16 5:68000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x564a522e9e60 con 0x564a522eb4c0 2023-10-05T19:28:35.930 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.926+0000 7f1425c8b700 1 -- 172.21.15.167:0/3011814457 <== osd.3 v2:172.21.15.167:6808/30036 13 ==== osd_op_reply(57 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f1420016070 con 0x564a522eb4c0 2023-10-05T19:28:35.931 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.926+0000 7f1425c8b700 1 -- 172.21.15.167:0/3011814457 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:58 5.e 5:70000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x564a522e9e60 con 0x564a522eb4c0 2023-10-05T19:28:35.931 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.926+0000 7f1425c8b700 1 -- 172.21.15.167:0/3011814457 <== osd.3 v2:172.21.15.167:6808/30036 14 ==== osd_op_reply(58 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f1420016070 con 0x564a522eb4c0 2023-10-05T19:28:35.932 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.926+0000 7f1425c8b700 1 -- 172.21.15.167:0/3011814457 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:59 5.1e 5:78000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x564a522e9e60 con 0x7f1410010070 2023-10-05T19:28:35.932 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.926+0000 7f142648c700 1 -- 172.21.15.167:0/3011814457 <== osd.0 v2:172.21.15.118:6800/27690 15 ==== osd_op_reply(59 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f141802f080 con 0x7f1410010070 2023-10-05T19:28:35.933 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.926+0000 7f142648c700 1 -- 172.21.15.167:0/3011814457 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:60 5.1 5:80000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x564a522e9e60 con 0x564a522eb4c0 2023-10-05T19:28:35.933 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.926+0000 7f1425c8b700 1 -- 172.21.15.167:0/3011814457 <== osd.3 v2:172.21.15.167:6808/30036 15 ==== osd_op_reply(60 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f1420016070 con 0x564a522eb4c0 2023-10-05T19:28:35.933 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.926+0000 7f1425c8b700 1 -- 172.21.15.167:0/3011814457 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:61 5.11 5:88000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x564a522e9e60 con 0x564a522eb4c0 2023-10-05T19:28:35.934 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.930+0000 7f1425c8b700 1 -- 172.21.15.167:0/3011814457 <== osd.3 v2:172.21.15.167:6808/30036 16 ==== osd_op_reply(61 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f1420016070 con 0x564a522eb4c0 2023-10-05T19:28:35.934 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.930+0000 7f1425c8b700 1 -- 172.21.15.167:0/3011814457 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:62 5.9 5:90000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x564a522e9e60 con 0x564a522ba5f0 2023-10-05T19:28:35.935 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.930+0000 7f142648c700 1 -- 172.21.15.167:0/3011814457 <== osd.1 v2:172.21.15.167:6800/30035 22 ==== osd_op_reply(62 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f141802f080 con 0x564a522ba5f0 2023-10-05T19:28:35.935 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.930+0000 7f142648c700 1 -- 172.21.15.167:0/3011814457 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:63 5.19 5:98000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x564a522e9e60 con 0x564a522ba5f0 2023-10-05T19:28:35.936 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.930+0000 7f142648c700 1 -- 172.21.15.167:0/3011814457 <== osd.1 v2:172.21.15.167:6800/30035 23 ==== osd_op_reply(63 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f141802f080 con 0x564a522ba5f0 2023-10-05T19:28:35.936 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.930+0000 7f142648c700 1 -- 172.21.15.167:0/3011814457 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:64 5.5 5:a0000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x564a522e9e60 con 0x7f1410010070 2023-10-05T19:28:35.937 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.930+0000 7f142648c700 1 -- 172.21.15.167:0/3011814457 <== osd.0 v2:172.21.15.118:6800/27690 16 ==== osd_op_reply(64 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f141802f080 con 0x7f1410010070 2023-10-05T19:28:35.937 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.930+0000 7f142648c700 1 -- 172.21.15.167:0/3011814457 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:65 5.15 5:a8000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x564a522e9e60 con 0x564a522eb4c0 2023-10-05T19:28:35.938 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.934+0000 7f1425c8b700 1 -- 172.21.15.167:0/3011814457 <== osd.3 v2:172.21.15.167:6808/30036 17 ==== osd_op_reply(65 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f1420016070 con 0x564a522eb4c0 2023-10-05T19:28:35.938 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.934+0000 7f1425c8b700 1 -- 172.21.15.167:0/3011814457 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:66 5.d 5:b0000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x564a522e9e60 con 0x7f142001a070 2023-10-05T19:28:35.939 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.934+0000 7f142548a700 1 -- 172.21.15.167:0/3011814457 <== osd.2 v2:172.21.15.167:6816/30037 10 ==== osd_op_reply(66 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f1410017050 con 0x7f142001a070 2023-10-05T19:28:35.939 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.934+0000 7f142548a700 1 -- 172.21.15.167:0/3011814457 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:67 5.1d 5:b8000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x564a522e9e60 con 0x564a522ba5f0 2023-10-05T19:28:35.940 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.934+0000 7f142648c700 1 -- 172.21.15.167:0/3011814457 <== osd.1 v2:172.21.15.167:6800/30035 24 ==== osd_op_reply(67 [pgnls start_epoch 30 out=107b] v30'2 uv2 ondisk = 1) v8 ==== 144+0+107 (crc 0 0 0) 0x7f141802f080 con 0x564a522ba5f0 2023-10-05T19:28:35.940 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.934+0000 7f142648c700 1 -- 172.21.15.167:0/3011814457 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:68 5.3 5:c0000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x564a522e9e60 con 0x7f1410010070 2023-10-05T19:28:35.941 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.934+0000 7f142648c700 1 -- 172.21.15.167:0/3011814457 <== osd.0 v2:172.21.15.118:6800/27690 17 ==== osd_op_reply(68 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f141802f080 con 0x7f1410010070 2023-10-05T19:28:35.941 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.934+0000 7f142648c700 1 -- 172.21.15.167:0/3011814457 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:69 5.13 5:c8000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x564a522e9e60 con 0x564a522eb4c0 2023-10-05T19:28:35.942 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.934+0000 7f1425c8b700 1 -- 172.21.15.167:0/3011814457 <== osd.3 v2:172.21.15.167:6808/30036 18 ==== osd_op_reply(69 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f1420016070 con 0x564a522eb4c0 2023-10-05T19:28:35.942 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.934+0000 7f1425c8b700 1 -- 172.21.15.167:0/3011814457 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:70 5.b 5:d0000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x564a522e9e60 con 0x7f1410010070 2023-10-05T19:28:35.942 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.938+0000 7f142648c700 1 -- 172.21.15.167:0/3011814457 <== osd.0 v2:172.21.15.118:6800/27690 18 ==== osd_op_reply(70 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f141802f080 con 0x7f1410010070 2023-10-05T19:28:35.943 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.938+0000 7f142648c700 1 -- 172.21.15.167:0/3011814457 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:71 5.1b 5:d8000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x564a522e9e60 con 0x7f142001a070 2023-10-05T19:28:35.943 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.938+0000 7f142548a700 1 -- 172.21.15.167:0/3011814457 <== osd.2 v2:172.21.15.167:6816/30037 11 ==== osd_op_reply(71 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f1410017050 con 0x7f142001a070 2023-10-05T19:28:35.944 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.938+0000 7f142548a700 1 -- 172.21.15.167:0/3011814457 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:72 5.7 5:e0000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x564a522e9e60 con 0x564a522eb4c0 2023-10-05T19:28:35.944 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.938+0000 7f1425c8b700 1 -- 172.21.15.167:0/3011814457 <== osd.3 v2:172.21.15.167:6808/30036 19 ==== osd_op_reply(72 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f1420016070 con 0x564a522eb4c0 2023-10-05T19:28:35.944 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.938+0000 7f1425c8b700 1 -- 172.21.15.167:0/3011814457 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:73 5.17 5:e8000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x564a522e9e60 con 0x7f1410010070 2023-10-05T19:28:35.946 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.938+0000 7f142648c700 1 -- 172.21.15.167:0/3011814457 <== osd.0 v2:172.21.15.118:6800/27690 19 ==== osd_op_reply(73 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f141802f080 con 0x7f1410010070 2023-10-05T19:28:35.946 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.938+0000 7f142648c700 1 -- 172.21.15.167:0/3011814457 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:74 5.f 5:f0000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x564a522e9e60 con 0x564a522ba5f0 2023-10-05T19:28:35.947 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.942+0000 7f142648c700 1 -- 172.21.15.167:0/3011814457 <== osd.1 v2:172.21.15.167:6800/30035 25 ==== osd_op_reply(74 [pgnls start_epoch 30 out=115b] v23'2 uv2 ondisk = 1) v8 ==== 144+0+115 (crc 0 0 0) 0x7f141802f080 con 0x564a522ba5f0 2023-10-05T19:28:35.947 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.942+0000 7f142648c700 1 -- 172.21.15.167:0/3011814457 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:75 5.1f 5:f8000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x564a522e9e60 con 0x7f142001a070 2023-10-05T19:28:35.947 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.942+0000 7f142548a700 1 -- 172.21.15.167:0/3011814457 <== osd.2 v2:172.21.15.167:6816/30037 12 ==== osd_op_reply(75 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f1410017050 con 0x7f142001a070 2023-10-05T19:28:35.948 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.942+0000 7f1428156e40 20 RGWRados::pool_iterate: got zonegroup_info.4f1f719d-9648-417b-b10c-e5d939825419 2023-10-05T19:28:35.948 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.942+0000 7f1428156e40 20 RGWRados::pool_iterate: got zone_info.9a9df965-6612-45d4-a45f-6e3c889bacc2 2023-10-05T19:28:35.948 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.942+0000 7f1428156e40 20 RGWRados::pool_iterate: got zone_names.default 2023-10-05T19:28:35.948 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.942+0000 7f1428156e40 20 RGWRados::pool_iterate: got zonegroups_names.default 2023-10-05T19:28:35.948 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.942+0000 7f1428156e40 20 rados->read ofs=0 len=0 2023-10-05T19:28:35.948 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.942+0000 7f1428156e40 1 -- 172.21.15.167:0/3011814457 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:76 5.f 5:f43b6ece:::zone_names.default:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x564a522eea80 con 0x564a522ba5f0 2023-10-05T19:28:35.949 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.942+0000 7f142648c700 1 -- 172.21.15.167:0/3011814457 <== osd.1 v2:172.21.15.167:6800/30035 26 ==== osd_op_reply(76 zone_names.default [read 0~46 out=46b] v0'0 uv1 ondisk = 0) v8 ==== 162+0+46 (crc 0 0 0) 0x7f141802f080 con 0x564a522ba5f0 2023-10-05T19:28:35.949 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.942+0000 7f1428156e40 20 rados_obj.operate() r=0 bl.length=46 2023-10-05T19:28:35.949 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.942+0000 7f1428156e40 20 rados->read ofs=0 len=0 2023-10-05T19:28:35.949 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.942+0000 7f1428156e40 1 -- 172.21.15.167:0/3011814457 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:77 5.1d 5:b8374eb4:::zone_info.9a9df965-6612-45d4-a45f-6e3c889bacc2:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x564a522eee30 con 0x564a522ba5f0 2023-10-05T19:28:35.950 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.942+0000 7f142648c700 1 -- 172.21.15.167:0/3011814457 <== osd.1 v2:172.21.15.167:6800/30035 27 ==== osd_op_reply(77 zone_info.9a9df965-6612-45d4-a45f-6e3c889bacc2 [read 0~874 out=874b] v0'0 uv2 ondisk = 0) v8 ==== 190+0+874 (crc 0 0 0) 0x7f141802f080 con 0x564a522ba5f0 2023-10-05T19:28:35.950 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.942+0000 7f1428156e40 20 rados_obj.operate() r=0 bl.length=874 2023-10-05T19:28:35.950 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.942+0000 7f1428156e40 20 rados->read ofs=0 len=0 2023-10-05T19:28:35.950 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.942+0000 7f1428156e40 1 -- 172.21.15.167:0/3011814457 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:78 5.f 5:f4c53578:::zonegroups_names.default:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x564a522ef320 con 0x564a522ba5f0 2023-10-05T19:28:35.950 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.942+0000 7f142648c700 1 -- 172.21.15.167:0/3011814457 <== osd.1 v2:172.21.15.167:6800/30035 28 ==== osd_op_reply(78 zonegroups_names.default [read 0~46 out=46b] v0'0 uv2 ondisk = 0) v8 ==== 168+0+46 (crc 0 0 0) 0x7f141802f080 con 0x564a522ba5f0 2023-10-05T19:28:35.950 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.942+0000 7f1428156e40 20 rados_obj.operate() r=0 bl.length=46 2023-10-05T19:28:35.950 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.942+0000 7f1428156e40 20 rados->read ofs=0 len=0 2023-10-05T19:28:35.950 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.942+0000 7f1428156e40 1 -- 172.21.15.167:0/3011814457 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:79 5.c 5:30d299a5:::zonegroup_info.4f1f719d-9648-417b-b10c-e5d939825419:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x564a522ef320 con 0x7f1410010070 2023-10-05T19:28:35.951 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.946+0000 7f142648c700 1 -- 172.21.15.167:0/3011814457 <== osd.0 v2:172.21.15.118:6800/27690 20 ==== osd_op_reply(79 zonegroup_info.4f1f719d-9648-417b-b10c-e5d939825419 [read 0~362 out=362b] v0'0 uv1 ondisk = 0) v8 ==== 195+0+362 (crc 0 0 0) 0x7f141802f080 con 0x7f1410010070 2023-10-05T19:28:35.951 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.946+0000 7f1428156e40 20 rados_obj.operate() r=0 bl.length=362 2023-10-05T19:28:35.951 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.946+0000 7f1428156e40 20 rados->read ofs=0 len=0 2023-10-05T19:28:35.951 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.946+0000 7f1428156e40 1 -- 172.21.15.167:0/3011814457 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:80 5.3 5:c0ad8cbd:::region_map:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x564a522ef320 con 0x7f1410010070 2023-10-05T19:28:35.952 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.946+0000 7f142648c700 1 -- 172.21.15.167:0/3011814457 <== osd.0 v2:172.21.15.118:6800/27690 21 ==== osd_op_reply(80 region_map [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 ==== 154+0+0 (crc 0 0 0) 0x7f1418050040 con 0x7f1410010070 2023-10-05T19:28:35.952 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.946+0000 7f1428156e40 20 rados_obj.operate() r=-2 bl.length=0 2023-10-05T19:28:35.952 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.946+0000 7f1428156e40 20 zone default found 2023-10-05T19:28:35.952 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.946+0000 7f1428156e40 4 Realm: () 2023-10-05T19:28:35.952 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.946+0000 7f1428156e40 4 ZoneGroup: default (4f1f719d-9648-417b-b10c-e5d939825419) 2023-10-05T19:28:35.952 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.946+0000 7f1428156e40 4 Zone: default (9a9df965-6612-45d4-a45f-6e3c889bacc2) 2023-10-05T19:28:35.952 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.946+0000 7f1428156e40 10 cannot find current period zonegroup using local zonegroup configuration 2023-10-05T19:28:35.952 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.946+0000 7f1428156e40 20 zonegroup default 2023-10-05T19:28:35.953 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.946+0000 7f1428156e40 20 rados->read ofs=0 len=0 2023-10-05T19:28:35.953 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.946+0000 7f1428156e40 1 -- 172.21.15.167:0/3011814457 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:81 5.3 5:c52100b6:::period_config.default:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x564a522ef320 con 0x7f1410010070 2023-10-05T19:28:35.953 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.946+0000 7f142648c700 1 -- 172.21.15.167:0/3011814457 <== osd.0 v2:172.21.15.118:6800/27690 22 ==== osd_op_reply(81 period_config.default [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 ==== 165+0+0 (crc 0 0 0) 0x7f1418050040 con 0x7f1410010070 2023-10-05T19:28:35.953 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.946+0000 7f1428156e40 20 rados_obj.operate() r=-2 bl.length=0 2023-10-05T19:28:35.953 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.946+0000 7f1428156e40 20 rados->read ofs=0 len=0 2023-10-05T19:28:35.953 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.946+0000 7f1428156e40 1 -- 172.21.15.167:0/3011814457 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:82 6.2 6:4347d321:::bucket.sync-source-hints.:head [call version.read in=11b,read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x564a522f0ca0 con 0x564a522eb4c0 2023-10-05T19:28:35.954 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.946+0000 7f1425c8b700 1 -- 172.21.15.167:0/3011814457 <== osd.3 v2:172.21.15.167:6808/30036 20 ==== osd_op_reply(82 bucket.sync-source-hints. [call,read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 ==== 211+0+0 (crc 0 0 0) 0x7f1420016070 con 0x564a522eb4c0 2023-10-05T19:28:35.954 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.946+0000 7f1428156e40 20 rados_obj.operate() r=-2 bl.length=0 2023-10-05T19:28:35.954 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.946+0000 7f1428156e40 20 rados->read ofs=0 len=0 2023-10-05T19:28:35.954 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.946+0000 7f1428156e40 1 -- 172.21.15.167:0/3011814457 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:83 6.b 6:d467b91b:::bucket.sync-target-hints.:head [call version.read in=11b,read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x564a522f1050 con 0x564a522eb4c0 2023-10-05T19:28:35.955 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.950+0000 7f1425c8b700 1 -- 172.21.15.167:0/3011814457 <== osd.3 v2:172.21.15.167:6808/30036 21 ==== osd_op_reply(83 bucket.sync-target-hints. [call,read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 ==== 211+0+0 (crc 0 0 0) 0x7f1420016070 con 0x564a522eb4c0 2023-10-05T19:28:35.955 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.950+0000 7f1428156e40 20 rados_obj.operate() r=-2 bl.length=0 2023-10-05T19:28:35.955 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.950+0000 7f1428156e40 20 started sync module instance, tier type = 2023-10-05T19:28:35.955 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.950+0000 7f1428156e40 20 started zone id=9a9df965-6612-45d4-a45f-6e3c889bacc2 (name=default) with tier type = 2023-10-05T19:28:35.955 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.950+0000 7f1428156e40 1 -- 172.21.15.167:0/3011814457 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:84 7.1f 7:f95f44c2:::notify.0:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x564a522f0360 con 0x7f142001a070 2023-10-05T19:28:35.958 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.950+0000 7f142548a700 1 -- 172.21.15.167:0/3011814457 <== osd.2 v2:172.21.15.167:6816/30037 13 ==== osd_op_reply(84 notify.0 [create] v30'147 uv147 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f1410017050 con 0x7f142001a070 2023-10-05T19:28:35.958 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.950+0000 7f1428156e40 1 -- 172.21.15.167:0/3011814457 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:85 7.1f 7:f95f44c2:::notify.0:head [watch watch cookie 94877206382432] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x564a522eff90 con 0x7f142001a070 2023-10-05T19:28:35.958 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.950+0000 7f1428156e40 1 -- 172.21.15.167:0/3011814457 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:86 7.0 7:05bf5b68:::notify.1:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x564a522f1de0 con 0x564a522ba5f0 2023-10-05T19:28:35.961 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.954+0000 7f142648c700 1 -- 172.21.15.167:0/3011814457 <== osd.1 v2:172.21.15.167:6800/30035 29 ==== osd_op_reply(86 notify.1 [create] v30'147 uv147 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f1418050040 con 0x564a522ba5f0 2023-10-05T19:28:35.961 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.954+0000 7f1428156e40 1 -- 172.21.15.167:0/3011814457 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:87 7.0 7:05bf5b68:::notify.1:head [watch watch cookie 94877206387216] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x564a522f2460 con 0x564a522ba5f0 2023-10-05T19:28:35.961 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.954+0000 7f1428156e40 1 -- 172.21.15.167:0/3011814457 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:88 7.15 7:a93a5511:::notify.2:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x564a522f2e50 con 0x564a522eb4c0 2023-10-05T19:28:35.961 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.954+0000 7f142548a700 1 -- 172.21.15.167:0/3011814457 <== osd.2 v2:172.21.15.167:6816/30037 14 ==== osd_op_reply(85 notify.0 [watch watch cookie 94877206382432] v30'148 uv147 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f1410017050 con 0x7f142001a070 2023-10-05T19:28:35.963 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.958+0000 7f142648c700 1 -- 172.21.15.167:0/3011814457 <== osd.1 v2:172.21.15.167:6800/30035 30 ==== osd_op_reply(87 notify.1 [watch watch cookie 94877206387216] v30'148 uv147 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f1418050040 con 0x564a522ba5f0 2023-10-05T19:28:35.964 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.958+0000 7f1425c8b700 1 -- 172.21.15.167:0/3011814457 <== osd.3 v2:172.21.15.167:6808/30036 22 ==== osd_op_reply(88 notify.2 [create] v30'147 uv147 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f1420016070 con 0x564a522eb4c0 2023-10-05T19:28:35.964 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.958+0000 7f1428156e40 1 -- 172.21.15.167:0/3011814457 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:89 7.15 7:a93a5511:::notify.2:head [watch watch cookie 94877206385664] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x564a522f1ef0 con 0x564a522eb4c0 2023-10-05T19:28:35.964 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.958+0000 7f1428156e40 1 -- 172.21.15.167:0/3011814457 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:90 7.e 7:7759931f:::notify.3:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x564a522f3740 con 0x7f142001a070 2023-10-05T19:28:35.967 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.962+0000 7f1425c8b700 1 -- 172.21.15.167:0/3011814457 <== osd.3 v2:172.21.15.167:6808/30036 23 ==== osd_op_reply(89 notify.2 [watch watch cookie 94877206385664] v30'148 uv147 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f1420016070 con 0x564a522eb4c0 2023-10-05T19:28:35.967 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.962+0000 7f142548a700 1 -- 172.21.15.167:0/3011814457 <== osd.2 v2:172.21.15.167:6816/30037 15 ==== osd_op_reply(90 notify.3 [create] v30'147 uv147 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f1410017050 con 0x7f142001a070 2023-10-05T19:28:35.967 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.962+0000 7f1428156e40 1 -- 172.21.15.167:0/3011814457 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:91 7.e 7:7759931f:::notify.3:head [watch watch cookie 94877206393424] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x564a522f3af0 con 0x7f142001a070 2023-10-05T19:28:35.967 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.962+0000 7f1428156e40 1 -- 172.21.15.167:0/3011814457 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:92 7.d 7:b4812045:::notify.4:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x564a522f4650 con 0x564a522ba5f0 2023-10-05T19:28:35.969 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.962+0000 7f142548a700 1 -- 172.21.15.167:0/3011814457 <== osd.2 v2:172.21.15.167:6816/30037 16 ==== osd_op_reply(91 notify.3 [watch watch cookie 94877206393424] v30'148 uv147 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f1410017050 con 0x7f142001a070 2023-10-05T19:28:35.970 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.966+0000 7f142648c700 1 -- 172.21.15.167:0/3011814457 <== osd.1 v2:172.21.15.167:6800/30035 31 ==== osd_op_reply(92 notify.4 [create] v30'147 uv147 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f1418050040 con 0x564a522ba5f0 2023-10-05T19:28:35.970 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.966+0000 7f1428156e40 1 -- 172.21.15.167:0/3011814457 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:93 7.d 7:b4812045:::notify.4:head [watch watch cookie 94877206391072] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x564a522f4a50 con 0x564a522ba5f0 2023-10-05T19:28:35.970 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.966+0000 7f1428156e40 1 -- 172.21.15.167:0/3011814457 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:94 7.3 7:c609908c:::notify.5:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x564a522f5460 con 0x7f1410010070 2023-10-05T19:28:35.972 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.966+0000 7f142648c700 1 -- 172.21.15.167:0/3011814457 <== osd.1 v2:172.21.15.167:6800/30035 32 ==== osd_op_reply(93 notify.4 [watch watch cookie 94877206391072] v30'148 uv147 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f1418050040 con 0x564a522ba5f0 2023-10-05T19:28:35.973 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.966+0000 7f142648c700 1 -- 172.21.15.167:0/3011814457 <== osd.0 v2:172.21.15.118:6800/27690 23 ==== osd_op_reply(94 notify.5 [create] v30'147 uv147 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f1418050040 con 0x7f1410010070 2023-10-05T19:28:35.973 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.966+0000 7f1428156e40 1 -- 172.21.15.167:0/3011814457 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:95 7.3 7:c609908c:::notify.5:head [watch watch cookie 94877206397216] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x564a522f5460 con 0x7f1410010070 2023-10-05T19:28:35.973 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.966+0000 7f1428156e40 1 -- 172.21.15.167:0/3011814457 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:96 7.14 7:2b04a3e9:::notify.6:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x564a522f5e70 con 0x7f1410010070 2023-10-05T19:28:35.976 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.970+0000 7f142648c700 1 -- 172.21.15.167:0/3011814457 <== osd.0 v2:172.21.15.118:6800/27690 24 ==== osd_op_reply(96 notify.6 [create] v30'147 uv147 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f1418050040 con 0x7f1410010070 2023-10-05T19:28:35.976 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.970+0000 7f1428156e40 1 -- 172.21.15.167:0/3011814457 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:97 7.14 7:2b04a3e9:::notify.6:head [watch watch cookie 94877206403456] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x564a522f62c0 con 0x7f1410010070 2023-10-05T19:28:35.976 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.970+0000 7f1428156e40 1 -- 172.21.15.167:0/3011814457 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:98 7.9 7:93e5b521:::notify.7:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x564a522f6cd0 con 0x7f1410010070 2023-10-05T19:28:35.976 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.970+0000 7f142648c700 1 -- 172.21.15.167:0/3011814457 <== osd.0 v2:172.21.15.118:6800/27690 25 ==== osd_op_reply(95 notify.5 [watch watch cookie 94877206397216] v30'148 uv147 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f1418050040 con 0x7f1410010070 2023-10-05T19:28:35.978 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.970+0000 7f142648c700 1 -- 172.21.15.167:0/3011814457 <== osd.0 v2:172.21.15.118:6800/27690 26 ==== osd_op_reply(97 notify.6 [watch watch cookie 94877206403456] v30'148 uv147 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f1418050040 con 0x7f1410010070 2023-10-05T19:28:35.978 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.974+0000 7f142648c700 1 -- 172.21.15.167:0/3011814457 <== osd.0 v2:172.21.15.118:6800/27690 27 ==== osd_op_reply(98 notify.7 [create] v30'147 uv147 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f1418050040 con 0x7f1410010070 2023-10-05T19:28:35.979 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.974+0000 7f1428156e40 1 -- 172.21.15.167:0/3011814457 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:99 7.9 7:93e5b521:::notify.7:head [watch watch cookie 94877206404560] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x564a522f6220 con 0x7f1410010070 2023-10-05T19:28:35.979 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.974+0000 7f1428156e40 20 add_watcher() i=0 2023-10-05T19:28:35.979 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.974+0000 7f1428156e40 20 add_watcher() i=1 2023-10-05T19:28:35.979 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.974+0000 7f1428156e40 20 add_watcher() i=2 2023-10-05T19:28:35.979 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.974+0000 7f1428156e40 20 add_watcher() i=3 2023-10-05T19:28:35.979 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.974+0000 7f1428156e40 20 add_watcher() i=4 2023-10-05T19:28:35.979 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.974+0000 7f1428156e40 20 add_watcher() i=5 2023-10-05T19:28:35.979 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.974+0000 7f1428156e40 20 add_watcher() i=6 2023-10-05T19:28:35.981 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.974+0000 7f142648c700 1 -- 172.21.15.167:0/3011814457 <== osd.0 v2:172.21.15.118:6800/27690 28 ==== osd_op_reply(99 notify.7 [watch watch cookie 94877206404560] v30'148 uv147 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f1418050040 con 0x7f1410010070 2023-10-05T19:28:35.981 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.974+0000 7f1428156e40 20 add_watcher() i=7 2023-10-05T19:28:35.981 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.974+0000 7f1428156e40 2 all 8 watchers are set, enabling cache 2023-10-05T19:28:35.981 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.974+0000 7f1428156e40 1 -- 172.21.15.167:0/3011814457 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:100 6.4 6:22d26bf9:::data_loggenerations_metadata:head [call version.check_conds in=74b,call version.read in=11b,read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x564a522f6220 con 0x564a522ba5f0 2023-10-05T19:28:35.981 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.974+0000 7f142648c700 1 -- 172.21.15.167:0/3011814457 <== osd.1 v2:172.21.15.167:6800/30035 33 ==== osd_op_reply(100 data_loggenerations_metadata [call,call out=48b,read 0~28 out=28b] v0'0 uv1 ondisk = 0) v8 ==== 256+0+76 (crc 0 0 0) 0x7f1418050040 con 0x564a522ba5f0 2023-10-05T19:28:35.981 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.974+0000 7f1428156e40 1 -- 172.21.15.167:0/3011814457 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:101 6.4 6:22d26bf9:::data_loggenerations_metadata:head [watch watch cookie 94877206407136] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x564a522f7120 con 0x564a522ba5f0 2023-10-05T19:28:35.983 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.978+0000 7f142648c700 1 -- 172.21.15.167:0/3011814457 <== osd.1 v2:172.21.15.167:6800/30035 34 ==== osd_op_reply(101 data_loggenerations_metadata [watch watch cookie 94877206407136] v30'145 uv1 ondisk = 0) v8 ==== 172+0+0 (crc 0 0 0) 0x7f1418050040 con 0x564a522ba5f0 2023-10-05T19:28:35.984 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.978+0000 7f13f67ec700 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2023-10-05T19:28:35.984 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.978+0000 7f1428156e40 20 check_secure_mon_conn(): auth registy supported: methods=[2] modes=[2,1] 2023-10-05T19:28:35.984 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.978+0000 7f1428156e40 20 check_secure_mon_conn(): mode 1 is insecure 2023-10-05T19:28:35.985 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.978+0000 7f1428156e40 1 Processor -- start 2023-10-05T19:28:35.985 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.978+0000 7f1428156e40 1 -- start start 2023-10-05T19:28:35.985 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.978+0000 7f1428156e40 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x564a52303e00 0x564a523041d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-05T19:28:35.985 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.978+0000 7f1428156e40 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x564a52304710 con 0x564a52303e00 2023-10-05T19:28:35.985 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.978+0000 7f142548a700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x564a52303e00 0x564a523041d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-05T19:28:35.985 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.978+0000 7f142548a700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x564a52303e00 0x564a523041d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.118:3300/0 says I am v2:172.21.15.167:41720/0 (socket says 172.21.15.167:41720) 2023-10-05T19:28:35.985 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.978+0000 7f142548a700 1 -- 172.21.15.167:0/1411582029 learned_addr learned my addr 172.21.15.167:0/1411582029 (peer_addr_for_me v2:172.21.15.167:0/0) 2023-10-05T19:28:35.986 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.978+0000 7f142548a700 1 -- 172.21.15.167:0/1411582029 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x564a52304850 con 0x564a52303e00 2023-10-05T19:28:35.986 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.982+0000 7f142548a700 1 --2- 172.21.15.167:0/1411582029 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x564a52303e00 0x564a523041d0 secure :-1 s=READY pgs=432 cs=0 l=1 rev1=1 crypto rx=0x564a522b7090 tx=0x7f1410010000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=80f374a85b140060 server_cookie=0 in_seq=0 out_seq=0 2023-10-05T19:28:35.987 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.982+0000 7f13f5feb700 1 -- 172.21.15.167:0/1411582029 <== mon.0 v2:172.21.15.118:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f141001c9f0 con 0x564a52303e00 2023-10-05T19:28:35.987 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.982+0000 7f13f5feb700 1 -- 172.21.15.167:0/1411582029 <== mon.0 v2:172.21.15.118:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f141001cb50 con 0x564a52303e00 2023-10-05T19:28:35.987 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.982+0000 7f13f5feb700 1 -- 172.21.15.167:0/1411582029 <== mon.0 v2:172.21.15.118:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f141001cd40 con 0x564a52303e00 2023-10-05T19:28:35.988 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.982+0000 7f1428156e40 1 -- 172.21.15.167:0/1411582029 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x564a52303e00 msgr2=0x564a523041d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-05T19:28:35.988 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.982+0000 7f1428156e40 1 --2- 172.21.15.167:0/1411582029 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x564a52303e00 0x564a523041d0 secure :-1 s=READY pgs=432 cs=0 l=1 rev1=1 crypto rx=0x564a522b7090 tx=0x7f1410010000 comp rx=0 tx=0).stop 2023-10-05T19:28:35.988 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.982+0000 7f1428156e40 1 -- 172.21.15.167:0/1411582029 shutdown_connections 2023-10-05T19:28:35.988 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.982+0000 7f1428156e40 1 --2- 172.21.15.167:0/1411582029 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x564a52303e00 0x564a523041d0 unknown :-1 s=CLOSED pgs=432 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-05T19:28:35.988 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.982+0000 7f1428156e40 1 -- 172.21.15.167:0/1411582029 >> 172.21.15.167:0/1411582029 conn(0x564a522ff1c0 msgr2=0x564a522fd370 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-05T19:28:35.988 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.982+0000 7f1428156e40 1 -- 172.21.15.167:0/1411582029 shutdown_connections 2023-10-05T19:28:35.988 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.982+0000 7f1428156e40 1 -- 172.21.15.167:0/1411582029 wait complete. 2023-10-05T19:28:35.989 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.982+0000 7f1428156e40 1 Processor -- start 2023-10-05T19:28:35.989 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.982+0000 7f1428156e40 1 -- start start 2023-10-05T19:28:35.989 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.982+0000 7f1428156e40 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x564a52303e00 0x564a523098d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-05T19:28:35.990 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.982+0000 7f1428156e40 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x564a5230a8d0 con 0x564a52303e00 2023-10-05T19:28:35.990 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.982+0000 7f142548a700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x564a52303e00 0x564a523098d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-05T19:28:35.990 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.982+0000 7f142548a700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x564a52303e00 0x564a523098d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.118:3300/0 says I am v2:172.21.15.167:41734/0 (socket says 172.21.15.167:41734) 2023-10-05T19:28:35.990 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.982+0000 7f142548a700 1 -- 172.21.15.167:0/1155403084 learned_addr learned my addr 172.21.15.167:0/1155403084 (peer_addr_for_me v2:172.21.15.167:0/0) 2023-10-05T19:28:35.991 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.986+0000 7f142548a700 1 -- 172.21.15.167:0/1155403084 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x564a522b6c20 con 0x564a52303e00 2023-10-05T19:28:35.991 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.986+0000 7f142548a700 1 --2- 172.21.15.167:0/1155403084 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x564a52303e00 0x564a523098d0 secure :-1 s=READY pgs=433 cs=0 l=1 rev1=1 crypto rx=0x564a522fdee0 tx=0x7f141002f000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-05T19:28:35.991 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.986+0000 7f1367fff700 1 -- 172.21.15.167:0/1155403084 <== mon.0 v2:172.21.15.118:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f1410035400 con 0x564a52303e00 2023-10-05T19:28:35.991 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.986+0000 7f1367fff700 1 -- 172.21.15.167:0/1155403084 <== mon.0 v2:172.21.15.118:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1410035580 con 0x564a52303e00 2023-10-05T19:28:35.991 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.986+0000 7f1367fff700 1 -- 172.21.15.167:0/1155403084 <== mon.0 v2:172.21.15.118:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f1410035770 con 0x564a52303e00 2023-10-05T19:28:35.991 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.986+0000 7f1428156e40 1 -- 172.21.15.167:0/1155403084 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x564a52309ed0 con 0x564a52303e00 2023-10-05T19:28:35.991 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.986+0000 7f1428156e40 1 -- 172.21.15.167:0/1155403084 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x564a5230a310 con 0x564a52303e00 2023-10-05T19:28:35.992 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.986+0000 7f1367fff700 1 -- 172.21.15.167:0/1155403084 <== mon.0 v2:172.21.15.118:3300/0 4 ==== mgrmap(e 5) v1 ==== 40329+0+0 (secure 0 0 0) 0x7f14100358d0 con 0x564a52303e00 2023-10-05T19:28:35.992 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.986+0000 7f1367fff700 1 --2- 172.21.15.167:0/1155403084 >> [v2:172.21.15.167:6824/29889,v1:172.21.15.167:6825/29889] conn(0x7f136c030c50 0x7f136c033100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-05T19:28:35.992 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.986+0000 7f1367fff700 1 -- 172.21.15.167:0/1155403084 <== mon.0 v2:172.21.15.118:3300/0 5 ==== osd_map(30..30 src has 1..30) v4 ==== 6057+0+0 (secure 0 0 0) 0x7f141002f070 con 0x564a52303e00 2023-10-05T19:28:35.993 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.986+0000 7f142648c700 1 --2- 172.21.15.167:0/1155403084 >> [v2:172.21.15.167:6824/29889,v1:172.21.15.167:6825/29889] conn(0x7f136c030c50 0x7f136c033100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-05T19:28:35.993 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.986+0000 7f1428156e40 5 note: GC not initialized 2023-10-05T19:28:35.993 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.986+0000 7f142648c700 1 --2- 172.21.15.167:0/1155403084 >> [v2:172.21.15.167:6824/29889,v1:172.21.15.167:6825/29889] conn(0x7f136c030c50 0x7f136c033100 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x564a522f6220 tx=0x7f1418000f90 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-05T19:28:35.995 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.990+0000 7f13657fa700 20 reqs_thread_entry: start 2023-10-05T19:28:35.996 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.990+0000 7f135effd700 20 reqs_thread_entry: start 2023-10-05T19:28:35.996 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.990+0000 7f1428156e40 20 init_complete bucket index max shards: 11 2023-10-05T19:28:35.996 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.990+0000 7f1428156e40 10 rgw notify: Started notification manager with: 1 workers 2023-10-05T19:28:35.997 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.990+0000 7f135d7fa700 20 rgw notify: INFO: next queues processing will happen at: Thu Oct 5 19:29:06 2023 2023-10-05T19:28:35.997 INFO:teuthology.orchestra.run.smithi167.stderr: 2023-10-05T19:28:35.997 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.990+0000 7f135cff9700 20 reqs_thread_entry: start 2023-10-05T19:28:35.997 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.990+0000 7f1428156e40 20 get_system_obj_state: rctx=0x7ffdf81c9c10 obj=default.rgw.meta:root:myfoo state=0x564a5218d0b0 s->prefetch_data=0 2023-10-05T19:28:35.997 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.990+0000 7f1428156e40 1 -- 172.21.15.167:0/3011814457 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:102 8.9 8:91b71841:root::myfoo:head [call version.read in=11b,getxattrs,stat] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x564a523728c0 con 0x7f142001a070 2023-10-05T19:28:35.998 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.990+0000 7f142548a700 1 -- 172.21.15.167:0/3011814457 <== osd.2 v2:172.21.15.167:6816/30037 17 ==== osd_op_reply(102 myfoo [call out=48b,getxattrs out=75b,stat out=16b] v0'0 uv27 ondisk = 0) v8 ==== 233+0+139 (crc 0 0 0) 0x7f1410032040 con 0x7f142001a070 2023-10-05T19:28:35.998 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.990+0000 7f1428156e40 20 get_system_obj_state: s->obj_tag was set empty 2023-10-05T19:28:35.998 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.990+0000 7f1428156e40 20 rados->read ofs=0 len=0 2023-10-05T19:28:35.998 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.990+0000 7f1428156e40 1 -- 172.21.15.167:0/3011814457 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:103 8.9 8:91b71841:root::myfoo:head [call version.check_conds in=122b,call version.read in=11b,read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x564a52375350 con 0x7f142001a070 2023-10-05T19:28:35.999 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.994+0000 7f142548a700 1 -- 172.21.15.167:0/3011814457 <== osd.2 v2:172.21.15.167:6816/30037 18 ==== osd_op_reply(103 myfoo [call,call out=48b,read 0~165 out=165b] v0'0 uv27 ondisk = 0) v8 ==== 233+0+213 (crc 0 0 0) 0x7f1410032040 con 0x7f142001a070 2023-10-05T19:28:35.999 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.994+0000 7f1428156e40 20 rados_obj.operate() r=0 bl.length=165 2023-10-05T19:28:35.999 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.994+0000 7f1428156e40 20 get_system_obj_state: rctx=0x7ffdf81c9c10 obj=default.rgw.meta:root:.bucket.meta.myfoo:9a9df965-6612-45d4-a45f-6e3c889bacc2.4223.1 state=0x564a521632b0 s->prefetch_data=0 2023-10-05T19:28:35.999 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.994+0000 7f1428156e40 1 -- 172.21.15.167:0/3011814457 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:104 8.11 8:8bb4ed75:root::.bucket.meta.myfoo%3a9a9df965-6612-45d4-a45f-6e3c889bacc2.4223.1:head [call version.read in=11b,getxattrs,stat] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x564a52373550 con 0x564a522eb4c0 2023-10-05T19:28:36.000 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.994+0000 7f1425c8b700 1 -- 172.21.15.167:0/3011814457 <== osd.3 v2:172.21.15.167:6808/30036 24 ==== osd_op_reply(104 .bucket.meta.myfoo:9a9df965-6612-45d4-a45f-6e3c889bacc2.4223.1 [call out=48b,getxattrs out=218b,stat out=16b] v0'0 uv5 ondisk = 0) v8 ==== 290+0+282 (crc 0 0 0) 0x7f1420016070 con 0x564a522eb4c0 2023-10-05T19:28:36.000 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.994+0000 7f1428156e40 20 get_system_obj_state: s->obj_tag was set empty 2023-10-05T19:28:36.000 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.994+0000 7f1428156e40 20 Read xattr: user.rgw.acl 2023-10-05T19:28:36.000 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.994+0000 7f1428156e40 20 Read xattr: user.rgw.idtag 2023-10-05T19:28:36.000 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.994+0000 7f1428156e40 20 rados->read ofs=0 len=0 2023-10-05T19:28:36.000 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.994+0000 7f1428156e40 1 -- 172.21.15.167:0/3011814457 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:105 8.11 8:8bb4ed75:root::.bucket.meta.myfoo%3a9a9df965-6612-45d4-a45f-6e3c889bacc2.4223.1:head [call version.check_conds in=122b,call version.read in=11b,read 0~0,getxattrs] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x564a523750f0 con 0x564a522eb4c0 2023-10-05T19:28:36.001 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.994+0000 7f1425c8b700 1 -- 172.21.15.167:0/3011814457 <== osd.3 v2:172.21.15.167:6808/30036 25 ==== osd_op_reply(105 .bucket.meta.myfoo:9a9df965-6612-45d4-a45f-6e3c889bacc2.4223.1 [call,call out=48b,read 0~352 out=352b,getxattrs out=218b] v0'0 uv5 ondisk = 0) v8 ==== 332+0+618 (crc 0 0 0) 0x7f1420016070 con 0x564a522eb4c0 2023-10-05T19:28:36.001 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.994+0000 7f1428156e40 20 rados_obj.operate() r=0 bl.length=352 2023-10-05T19:28:36.001 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.994+0000 7f1428156e40 20 couldn't put binfo cache entry, might have raced with data changes 2023-10-05T19:28:36.001 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.994+0000 7f1428156e40 20 get_system_obj_state: rctx=0x7ffdf81c9b70 obj=default.rgw.meta:root:myfoo state=0x564a521632b0 s->prefetch_data=0 2023-10-05T19:28:36.002 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.994+0000 7f1428156e40 1 -- 172.21.15.167:0/3011814457 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:106 8.9 8:91b71841:root::myfoo:head [call version.read in=11b,getxattrs,stat] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x564a52371a10 con 0x7f142001a070 2023-10-05T19:28:36.002 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.994+0000 7f142548a700 1 -- 172.21.15.167:0/3011814457 <== osd.2 v2:172.21.15.167:6816/30037 19 ==== osd_op_reply(106 myfoo [call out=48b,getxattrs out=75b,stat out=16b] v0'0 uv27 ondisk = 0) v8 ==== 233+0+139 (crc 0 0 0) 0x7f1410032040 con 0x7f142001a070 2023-10-05T19:28:36.002 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.994+0000 7f1428156e40 20 get_system_obj_state: s->obj_tag was set empty 2023-10-05T19:28:36.002 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.994+0000 7f1428156e40 20 rados->read ofs=0 len=0 2023-10-05T19:28:36.002 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.994+0000 7f1428156e40 1 -- 172.21.15.167:0/3011814457 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:107 8.9 8:91b71841:root::myfoo:head [call version.check_conds in=122b,call version.read in=11b,read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x564a52371a10 con 0x7f142001a070 2023-10-05T19:28:36.004 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.998+0000 7f142548a700 1 -- 172.21.15.167:0/3011814457 <== osd.2 v2:172.21.15.167:6816/30037 20 ==== osd_op_reply(107 myfoo [call,call out=48b,read 0~165 out=165b] v0'0 uv27 ondisk = 0) v8 ==== 233+0+213 (crc 0 0 0) 0x7f1410032040 con 0x7f142001a070 2023-10-05T19:28:36.004 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.998+0000 7f1428156e40 20 rados_obj.operate() r=0 bl.length=165 2023-10-05T19:28:36.004 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.998+0000 7f1428156e40 20 get_system_obj_state: rctx=0x7ffdf81c9b70 obj=default.rgw.meta:root:.bucket.meta.myfoo:9a9df965-6612-45d4-a45f-6e3c889bacc2.4223.1 state=0x564a5218d0b0 s->prefetch_data=0 2023-10-05T19:28:36.004 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.998+0000 7f1428156e40 1 -- 172.21.15.167:0/3011814457 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:108 8.11 8:8bb4ed75:root::.bucket.meta.myfoo%3a9a9df965-6612-45d4-a45f-6e3c889bacc2.4223.1:head [call version.read in=11b,getxattrs,stat] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x564a52371a10 con 0x564a522eb4c0 2023-10-05T19:28:36.004 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.998+0000 7f1425c8b700 1 -- 172.21.15.167:0/3011814457 <== osd.3 v2:172.21.15.167:6808/30036 26 ==== osd_op_reply(108 .bucket.meta.myfoo:9a9df965-6612-45d4-a45f-6e3c889bacc2.4223.1 [call out=48b,getxattrs out=218b,stat out=16b] v0'0 uv5 ondisk = 0) v8 ==== 290+0+282 (crc 0 0 0) 0x7f1420016070 con 0x564a522eb4c0 2023-10-05T19:28:36.004 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.998+0000 7f1428156e40 20 get_system_obj_state: s->obj_tag was set empty 2023-10-05T19:28:36.004 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.998+0000 7f1428156e40 20 Read xattr: user.rgw.acl 2023-10-05T19:28:36.004 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.998+0000 7f1428156e40 20 Read xattr: user.rgw.idtag 2023-10-05T19:28:36.005 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.998+0000 7f1428156e40 20 rados->read ofs=0 len=0 2023-10-05T19:28:36.005 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.998+0000 7f1428156e40 1 -- 172.21.15.167:0/3011814457 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:109 8.11 8:8bb4ed75:root::.bucket.meta.myfoo%3a9a9df965-6612-45d4-a45f-6e3c889bacc2.4223.1:head [call version.check_conds in=122b,call version.read in=11b,read 0~0,getxattrs] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x564a523760f0 con 0x564a522eb4c0 2023-10-05T19:28:36.005 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.998+0000 7f1425c8b700 1 -- 172.21.15.167:0/3011814457 <== osd.3 v2:172.21.15.167:6808/30036 27 ==== osd_op_reply(109 .bucket.meta.myfoo:9a9df965-6612-45d4-a45f-6e3c889bacc2.4223.1 [call,call out=48b,read 0~352 out=352b,getxattrs out=218b] v0'0 uv5 ondisk = 0) v8 ==== 332+0+618 (crc 0 0 0) 0x7f1420016070 con 0x564a522eb4c0 2023-10-05T19:28:36.005 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.998+0000 7f1428156e40 20 rados_obj.operate() r=0 bl.length=352 2023-10-05T19:28:36.005 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.998+0000 7f1428156e40 20 couldn't put binfo cache entry, might have raced with data changes 2023-10-05T19:28:36.005 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.998+0000 7f1428156e40 1 -- 172.21.15.167:0/3011814457 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:110 4.3d 4:bd5dd36d:::.dir.9a9df965-6612-45d4-a45f-6e3c889bacc2.4223.1.0:head [call rgw.bucket_list in=47b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x564a52378050 con 0x564a522eb4c0 2023-10-05T19:28:36.005 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.998+0000 7f1428156e40 1 -- 172.21.15.167:0/3011814457 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:111 4.29 4:955f623b:::.dir.9a9df965-6612-45d4-a45f-6e3c889bacc2.4223.1.1:head [call rgw.bucket_list in=47b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x564a5237ab30 con 0x564a522eb4c0 2023-10-05T19:28:36.005 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.998+0000 7f1428156e40 1 -- 172.21.15.167:0/3011814457 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:112 4.38 4:1d83ede1:::.dir.9a9df965-6612-45d4-a45f-6e3c889bacc2.4223.1.2:head [call rgw.bucket_list in=47b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x564a5237d6c0 con 0x564a522ba5f0 2023-10-05T19:28:36.005 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.998+0000 7f1428156e40 1 -- 172.21.15.167:0/3011814457 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:113 4.1e 4:7939a9ac:::.dir.9a9df965-6612-45d4-a45f-6e3c889bacc2.4223.1.3:head [call rgw.bucket_list in=47b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x564a5237ff80 con 0x7f1410010070 2023-10-05T19:28:36.005 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.998+0000 7f1428156e40 1 -- 172.21.15.167:0/3011814457 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:114 4.e 4:719498d8:::.dir.9a9df965-6612-45d4-a45f-6e3c889bacc2.4223.1.4:head [call rgw.bucket_list in=47b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x564a52382b80 con 0x7f142001a070 2023-10-05T19:28:36.006 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.998+0000 7f1428156e40 1 -- 172.21.15.167:0/3011814457 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:115 4.32 4:4f38608c:::.dir.9a9df965-6612-45d4-a45f-6e3c889bacc2.4223.1.5:head [call rgw.bucket_list in=47b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x564a52385ac0 con 0x564a522ba5f0 2023-10-05T19:28:36.006 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.998+0000 7f1428156e40 1 -- 172.21.15.167:0/3011814457 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:116 4.38 4:1f2a6458:::.dir.9a9df965-6612-45d4-a45f-6e3c889bacc2.4223.1.6:head [call rgw.bucket_list in=47b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x564a52388880 con 0x564a522ba5f0 2023-10-05T19:28:36.006 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.998+0000 7f1428156e40 1 -- 172.21.15.167:0/3011814457 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:117 4.2 4:41a3e149:::.dir.9a9df965-6612-45d4-a45f-6e3c889bacc2.4223.1.7:head [call rgw.bucket_list in=47b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x564a5238b4e0 con 0x564a522ba5f0 2023-10-05T19:28:36.006 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.998+0000 7f1428156e40 1 -- 172.21.15.167:0/3011814457 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:118 4.17 4:ea8a113e:::.dir.9a9df965-6612-45d4-a45f-6e3c889bacc2.4223.1.8:head [call rgw.bucket_list in=47b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x564a5238e260 con 0x7f1410010070 2023-10-05T19:28:36.006 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.998+0000 7f1428156e40 1 -- 172.21.15.167:0/3011814457 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:119 4.1e 4:7a3e9338:::.dir.9a9df965-6612-45d4-a45f-6e3c889bacc2.4223.1.9:head [call rgw.bucket_list in=47b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x564a52390d40 con 0x7f1410010070 2023-10-05T19:28:36.006 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.998+0000 7f1428156e40 1 -- 172.21.15.167:0/3011814457 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:120 4.29 4:94432b39:::.dir.9a9df965-6612-45d4-a45f-6e3c889bacc2.4223.1.10:head [call rgw.bucket_list in=47b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x564a523938e0 con 0x564a522eb4c0 2023-10-05T19:28:36.007 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.998+0000 7f142648c700 1 -- 172.21.15.167:0/3011814457 <== osd.1 v2:172.21.15.167:6800/30035 35 ==== osd_op_reply(115 .dir.9a9df965-6612-45d4-a45f-6e3c889bacc2.4223.1.5 [call out=124b] v0'0 uv5 ondisk = 0) v8 ==== 194+0+124 (crc 0 0 0) 0x7f1418050040 con 0x564a522ba5f0 2023-10-05T19:28:36.007 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.998+0000 7f1425c8b700 1 -- 172.21.15.167:0/3011814457 <== osd.3 v2:172.21.15.167:6808/30036 28 ==== osd_op_reply(110 .dir.9a9df965-6612-45d4-a45f-6e3c889bacc2.4223.1.0 [call out=85b] v0'0 uv1 ondisk = 0) v8 ==== 194+0+85 (crc 0 0 0) 0x7f1420016070 con 0x564a522eb4c0 2023-10-05T19:28:36.007 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.998+0000 7f142648c700 1 -- 172.21.15.167:0/3011814457 <== osd.1 v2:172.21.15.167:6800/30035 36 ==== osd_op_reply(112 .dir.9a9df965-6612-45d4-a45f-6e3c889bacc2.4223.1.2 [call out=85b] v0'0 uv1 ondisk = 0) v8 ==== 194+0+85 (crc 0 0 0) 0x7f1418050040 con 0x564a522ba5f0 2023-10-05T19:28:36.007 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.998+0000 7f1425c8b700 1 -- 172.21.15.167:0/3011814457 <== osd.3 v2:172.21.15.167:6808/30036 29 ==== osd_op_reply(111 .dir.9a9df965-6612-45d4-a45f-6e3c889bacc2.4223.1.1 [call out=85b] v0'0 uv1 ondisk = 0) v8 ==== 194+0+85 (crc 0 0 0) 0x7f1420016070 con 0x564a522eb4c0 2023-10-05T19:28:36.007 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.998+0000 7f142648c700 1 -- 172.21.15.167:0/3011814457 <== osd.1 v2:172.21.15.167:6800/30035 37 ==== osd_op_reply(117 .dir.9a9df965-6612-45d4-a45f-6e3c889bacc2.4223.1.7 [call out=85b] v0'0 uv1 ondisk = 0) v8 ==== 194+0+85 (crc 0 0 0) 0x7f1418050040 con 0x564a522ba5f0 2023-10-05T19:28:36.008 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.998+0000 7f142548a700 1 -- 172.21.15.167:0/3011814457 <== osd.2 v2:172.21.15.167:6816/30037 21 ==== osd_op_reply(114 .dir.9a9df965-6612-45d4-a45f-6e3c889bacc2.4223.1.4 [call out=85b] v0'0 uv1 ondisk = 0) v8 ==== 194+0+85 (crc 0 0 0) 0x7f1410032040 con 0x7f142001a070 2023-10-05T19:28:36.008 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.998+0000 7f142648c700 1 -- 172.21.15.167:0/3011814457 <== osd.0 v2:172.21.15.118:6800/27690 29 ==== osd_op_reply(113 .dir.9a9df965-6612-45d4-a45f-6e3c889bacc2.4223.1.3 [call out=85b] v0'0 uv1 ondisk = 0) v8 ==== 194+0+85 (crc 0 0 0) 0x7f1418050040 con 0x7f1410010070 2023-10-05T19:28:36.008 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.998+0000 7f142648c700 1 -- 172.21.15.167:0/3011814457 <== osd.0 v2:172.21.15.118:6800/27690 30 ==== osd_op_reply(118 .dir.9a9df965-6612-45d4-a45f-6e3c889bacc2.4223.1.8 [call out=124b] v0'0 uv5 ondisk = 0) v8 ==== 194+0+124 (crc 0 0 0) 0x7f1418050040 con 0x7f1410010070 2023-10-05T19:28:36.008 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.998+0000 7f142648c700 1 -- 172.21.15.167:0/3011814457 <== osd.1 v2:172.21.15.167:6800/30035 38 ==== osd_op_reply(116 .dir.9a9df965-6612-45d4-a45f-6e3c889bacc2.4223.1.6 [call out=124b] v0'0 uv8 ondisk = 0) v8 ==== 194+0+124 (crc 0 0 0) 0x7f1418050040 con 0x564a522ba5f0 2023-10-05T19:28:36.008 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.998+0000 7f142648c700 1 -- 172.21.15.167:0/3011814457 <== osd.0 v2:172.21.15.118:6800/27690 31 ==== osd_op_reply(119 .dir.9a9df965-6612-45d4-a45f-6e3c889bacc2.4223.1.9 [call out=85b] v0'0 uv2 ondisk = 0) v8 ==== 194+0+85 (crc 0 0 0) 0x7f1418050040 con 0x7f1410010070 2023-10-05T19:28:36.008 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:35.998+0000 7f1425c8b700 1 -- 172.21.15.167:0/3011814457 <== osd.3 v2:172.21.15.167:6808/30036 30 ==== osd_op_reply(120 .dir.9a9df965-6612-45d4-a45f-6e3c889bacc2.4223.1.10 [call out=85b] v0'0 uv2 ondisk = 0) v8 ==== 195+0+85 (crc 0 0 0) 0x7f1420016070 con 0x564a522eb4c0 2023-10-05T19:28:36.008 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.002+0000 7f1428156e40 1 -- 172.21.15.167:0/3011814457 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:121 7.1f 7:f95f44c2:::notify.0:head [watch unwatch cookie 94877206382432] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x564a523938e0 con 0x7f142001a070 2023-10-05T19:28:36.009 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.002+0000 7f142548a700 1 -- 172.21.15.167:0/3011814457 <== osd.2 v2:172.21.15.167:6816/30037 22 ==== osd_op_reply(121 notify.0 [watch unwatch cookie 94877206382432] v30'149 uv147 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f1410032040 con 0x7f142001a070 2023-10-05T19:28:36.009 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.002+0000 7f1428156e40 20 remove_watcher() i=0 2023-10-05T19:28:36.009 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.002+0000 7f1428156e40 2 removed watcher, disabling cache 2023-10-05T19:28:36.009 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.002+0000 7f1428156e40 1 -- 172.21.15.167:0/3011814457 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:122 7.0 7:05bf5b68:::notify.1:head [watch unwatch cookie 94877206387216] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x564a523938e0 con 0x564a522ba5f0 2023-10-05T19:28:36.011 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.006+0000 7f142648c700 1 -- 172.21.15.167:0/3011814457 <== osd.1 v2:172.21.15.167:6800/30035 39 ==== osd_op_reply(122 notify.1 [watch unwatch cookie 94877206387216] v30'149 uv147 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f1418050040 con 0x564a522ba5f0 2023-10-05T19:28:36.011 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.006+0000 7f1428156e40 20 remove_watcher() i=1 2023-10-05T19:28:36.011 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.006+0000 7f1428156e40 1 -- 172.21.15.167:0/3011814457 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:123 7.15 7:a93a5511:::notify.2:head [watch unwatch cookie 94877206385664] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x564a523938e0 con 0x564a522eb4c0 2023-10-05T19:28:36.014 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.006+0000 7f1425c8b700 1 -- 172.21.15.167:0/3011814457 <== osd.3 v2:172.21.15.167:6808/30036 31 ==== osd_op_reply(123 notify.2 [watch unwatch cookie 94877206385664] v30'149 uv147 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f1420016070 con 0x564a522eb4c0 2023-10-05T19:28:36.014 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.006+0000 7f1428156e40 20 remove_watcher() i=2 2023-10-05T19:28:36.014 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.006+0000 7f1428156e40 1 -- 172.21.15.167:0/3011814457 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:124 7.e 7:7759931f:::notify.3:head [watch unwatch cookie 94877206393424] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x564a523938e0 con 0x7f142001a070 2023-10-05T19:28:36.016 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.010+0000 7f142548a700 1 -- 172.21.15.167:0/3011814457 <== osd.2 v2:172.21.15.167:6816/30037 23 ==== osd_op_reply(124 notify.3 [watch unwatch cookie 94877206393424] v30'149 uv147 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f1410032040 con 0x7f142001a070 2023-10-05T19:28:36.016 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.010+0000 7f1428156e40 20 remove_watcher() i=3 2023-10-05T19:28:36.016 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.010+0000 7f1428156e40 1 -- 172.21.15.167:0/3011814457 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:125 7.d 7:b4812045:::notify.4:head [watch unwatch cookie 94877206391072] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x564a523938e0 con 0x564a522ba5f0 2023-10-05T19:28:36.018 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.014+0000 7f142648c700 1 -- 172.21.15.167:0/3011814457 <== osd.1 v2:172.21.15.167:6800/30035 40 ==== osd_op_reply(125 notify.4 [watch unwatch cookie 94877206391072] v30'149 uv147 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f1418050040 con 0x564a522ba5f0 2023-10-05T19:28:36.018 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.014+0000 7f1428156e40 20 remove_watcher() i=4 2023-10-05T19:28:36.018 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.014+0000 7f1428156e40 1 -- 172.21.15.167:0/3011814457 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:126 7.3 7:c609908c:::notify.5:head [watch unwatch cookie 94877206397216] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x564a523938e0 con 0x7f1410010070 2023-10-05T19:28:36.021 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.014+0000 7f142648c700 1 -- 172.21.15.167:0/3011814457 <== osd.0 v2:172.21.15.118:6800/27690 32 ==== osd_op_reply(126 notify.5 [watch unwatch cookie 94877206397216] v30'149 uv147 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f1418050040 con 0x7f1410010070 2023-10-05T19:28:36.021 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.014+0000 7f1428156e40 20 remove_watcher() i=5 2023-10-05T19:28:36.021 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.014+0000 7f1428156e40 1 -- 172.21.15.167:0/3011814457 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:127 7.14 7:2b04a3e9:::notify.6:head [watch unwatch cookie 94877206403456] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x564a523938e0 con 0x7f1410010070 2023-10-05T19:28:36.024 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.018+0000 7f142648c700 1 -- 172.21.15.167:0/3011814457 <== osd.0 v2:172.21.15.118:6800/27690 33 ==== osd_op_reply(127 notify.6 [watch unwatch cookie 94877206403456] v30'149 uv147 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f1418050040 con 0x7f1410010070 2023-10-05T19:28:36.024 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.018+0000 7f1428156e40 20 remove_watcher() i=6 2023-10-05T19:28:36.025 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.018+0000 7f1428156e40 1 -- 172.21.15.167:0/3011814457 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:128 7.9 7:93e5b521:::notify.7:head [watch unwatch cookie 94877206404560] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x564a523938e0 con 0x7f1410010070 2023-10-05T19:28:36.027 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.022+0000 7f142648c700 1 -- 172.21.15.167:0/3011814457 <== osd.0 v2:172.21.15.118:6800/27690 34 ==== osd_op_reply(128 notify.7 [watch unwatch cookie 94877206404560] v30'149 uv147 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f1418050040 con 0x7f1410010070 2023-10-05T19:28:36.027 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.022+0000 7f1428156e40 20 remove_watcher() i=7 2023-10-05T19:28:36.029 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.022+0000 7f13f67ec700 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2023-10-05T19:28:36.029 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.022+0000 7f1428156e40 1 -- 172.21.15.167:0/3011814457 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:129 6.4 6:22d26bf9:::data_loggenerations_metadata:head [watch unwatch cookie 94877206407136] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x564a52331420 con 0x564a522ba5f0 2023-10-05T19:28:36.032 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.026+0000 7f142648c700 1 -- 172.21.15.167:0/3011814457 <== osd.1 v2:172.21.15.167:6800/30035 41 ==== osd_op_reply(129 data_loggenerations_metadata [watch unwatch cookie 94877206407136] v30'146 uv1 ondisk = 0) v8 ==== 172+0+0 (crc 0 0 0) 0x7f1418050040 con 0x564a522ba5f0 2023-10-05T19:28:36.032 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.026+0000 7f1428156e40 1 -- 172.21.15.167:0/3011814457 >> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] conn(0x7f1410010070 msgr2=0x7f141001c430 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-05T19:28:36.032 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.026+0000 7f1428156e40 1 --2- 172.21.15.167:0/3011814457 >> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] conn(0x7f1410010070 0x7f141001c430 crc :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-05T19:28:36.033 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.026+0000 7f1428156e40 1 -- 172.21.15.167:0/3011814457 >> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] conn(0x564a522ba5f0 msgr2=0x564a522e7a20 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-05T19:28:36.033 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.026+0000 7f1428156e40 1 --2- 172.21.15.167:0/3011814457 >> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] conn(0x564a522ba5f0 0x564a522e7a20 crc :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-05T19:28:36.033 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.026+0000 7f1428156e40 1 -- 172.21.15.167:0/3011814457 >> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] conn(0x7f142001a070 msgr2=0x7f1420043730 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-05T19:28:36.033 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.026+0000 7f1428156e40 1 --2- 172.21.15.167:0/3011814457 >> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] conn(0x7f142001a070 0x7f1420043730 crc :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-05T19:28:36.033 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.026+0000 7f1428156e40 1 -- 172.21.15.167:0/3011814457 >> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] conn(0x564a522eb4c0 msgr2=0x564a522ed8a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-05T19:28:36.033 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.026+0000 7f1428156e40 1 --2- 172.21.15.167:0/3011814457 >> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] conn(0x564a522eb4c0 0x564a522ed8a0 crc :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-05T19:28:36.033 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.026+0000 7f1428156e40 1 -- 172.21.15.167:0/3011814457 >> [v2:172.21.15.167:6824/29889,v1:172.21.15.167:6825/29889] conn(0x7f140c030c50 msgr2=0x7f140c033100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-05T19:28:36.033 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.026+0000 7f1428156e40 1 --2- 172.21.15.167:0/3011814457 >> [v2:172.21.15.167:6824/29889,v1:172.21.15.167:6825/29889] conn(0x7f140c030c50 0x7f140c033100 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7f1410000f80 tx=0x7f1410013000 comp rx=0 tx=0).stop 2023-10-05T19:28:36.033 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.026+0000 7f142648c700 1 -- 172.21.15.167:0/3011814457 reap_dead start 2023-10-05T19:28:36.033 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.026+0000 7f1428156e40 1 -- 172.21.15.167:0/3011814457 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x564a522b6010 msgr2=0x564a522dbf60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-05T19:28:36.034 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.026+0000 7f1428156e40 1 --2- 172.21.15.167:0/3011814457 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x564a522b6010 0x564a522dbf60 secure :-1 s=READY pgs=431 cs=0 l=1 rev1=1 crypto rx=0x7f14200018e0 tx=0x7f1420016000 comp rx=0 tx=0).stop 2023-10-05T19:28:36.034 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.026+0000 7f1428156e40 1 -- 172.21.15.167:0/3011814457 shutdown_connections 2023-10-05T19:28:36.034 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.026+0000 7f1428156e40 1 --2- 172.21.15.167:0/3011814457 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x564a522b6010 0x564a522dbf60 unknown :-1 s=CLOSED pgs=431 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-05T19:28:36.034 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.026+0000 7f1428156e40 1 -- 172.21.15.167:0/3011814457 >> 172.21.15.167:0/3011814457 conn(0x564a52229860 msgr2=0x564a522cf7d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-05T19:28:36.034 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.026+0000 7f1428156e40 1 -- 172.21.15.167:0/3011814457 shutdown_connections 2023-10-05T19:28:36.034 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.026+0000 7f1428156e40 1 -- 172.21.15.167:0/3011814457 wait complete. 2023-10-05T19:28:36.035 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.026+0000 7f1428156e40 1 -- 172.21.15.167:0/1155403084 >> [v2:172.21.15.167:6824/29889,v1:172.21.15.167:6825/29889] conn(0x7f136c030c50 msgr2=0x7f136c033100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-05T19:28:36.035 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.026+0000 7f1428156e40 1 --2- 172.21.15.167:0/1155403084 >> [v2:172.21.15.167:6824/29889,v1:172.21.15.167:6825/29889] conn(0x7f136c030c50 0x7f136c033100 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x564a522f6220 tx=0x7f1418000f90 comp rx=0 tx=0).stop 2023-10-05T19:28:36.035 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.026+0000 7f1428156e40 1 -- 172.21.15.167:0/1155403084 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x564a52303e00 msgr2=0x564a523098d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-05T19:28:36.035 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.026+0000 7f1428156e40 1 --2- 172.21.15.167:0/1155403084 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x564a52303e00 0x564a523098d0 secure :-1 s=READY pgs=433 cs=0 l=1 rev1=1 crypto rx=0x564a522fdee0 tx=0x7f141002f000 comp rx=0 tx=0).stop 2023-10-05T19:28:36.035 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.026+0000 7f1428156e40 1 -- 172.21.15.167:0/1155403084 shutdown_connections 2023-10-05T19:28:36.035 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.026+0000 7f1428156e40 1 --2- 172.21.15.167:0/1155403084 >> [v2:172.21.15.167:6824/29889,v1:172.21.15.167:6825/29889] conn(0x7f136c030c50 0x7f136c033100 unknown :-1 s=CLOSED pgs=151 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-05T19:28:36.035 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.026+0000 7f1428156e40 1 --2- 172.21.15.167:0/1155403084 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x564a52303e00 0x564a523098d0 unknown :-1 s=CLOSED pgs=433 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-05T19:28:36.036 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.026+0000 7f1428156e40 1 -- 172.21.15.167:0/1155403084 >> 172.21.15.167:0/1155403084 conn(0x564a522ff1c0 msgr2=0x564a522fd2c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-05T19:28:36.036 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.026+0000 7f1428156e40 1 -- 172.21.15.167:0/1155403084 shutdown_connections 2023-10-05T19:28:36.036 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.026+0000 7f1428156e40 1 -- 172.21.15.167:0/1155403084 wait complete. 2023-10-05T19:28:36.036 INFO:teuthology.orchestra.run.smithi167.stdout:{ 2023-10-05T19:28:36.036 INFO:teuthology.orchestra.run.smithi167.stdout: "bucket": "myfoo", 2023-10-05T19:28:36.036 INFO:teuthology.orchestra.run.smithi167.stdout: "num_shards": 11, 2023-10-05T19:28:36.036 INFO:teuthology.orchestra.run.smithi167.stdout: "tenant": "", 2023-10-05T19:28:36.036 INFO:teuthology.orchestra.run.smithi167.stdout: "zonegroup": "4f1f719d-9648-417b-b10c-e5d939825419", 2023-10-05T19:28:36.037 INFO:teuthology.orchestra.run.smithi167.stdout: "placement_rule": "default-placement", 2023-10-05T19:28:36.037 INFO:teuthology.orchestra.run.smithi167.stdout: "explicit_placement": { 2023-10-05T19:28:36.037 INFO:teuthology.orchestra.run.smithi167.stdout: "data_pool": "", 2023-10-05T19:28:36.037 INFO:teuthology.orchestra.run.smithi167.stdout: "data_extra_pool": "", 2023-10-05T19:28:36.037 INFO:teuthology.orchestra.run.smithi167.stdout: "index_pool": "" 2023-10-05T19:28:36.037 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2023-10-05T19:28:36.037 INFO:teuthology.orchestra.run.smithi167.stdout: "id": "9a9df965-6612-45d4-a45f-6e3c889bacc2.4223.1", 2023-10-05T19:28:36.037 INFO:teuthology.orchestra.run.smithi167.stdout: "marker": "9a9df965-6612-45d4-a45f-6e3c889bacc2.4223.1", 2023-10-05T19:28:36.037 INFO:teuthology.orchestra.run.smithi167.stdout: "index_type": "Normal", 2023-10-05T19:28:36.037 INFO:teuthology.orchestra.run.smithi167.stdout: "owner": "foo", 2023-10-05T19:28:36.037 INFO:teuthology.orchestra.run.smithi167.stdout: "ver": "0#1,1#1,2#1,3#1,4#1,5#3,6#3,7#1,8#3,9#1,10#1", 2023-10-05T19:28:36.037 INFO:teuthology.orchestra.run.smithi167.stdout: "master_ver": "0#0,1#0,2#0,3#0,4#0,5#0,6#0,7#0,8#0,9#0,10#0", 2023-10-05T19:28:36.038 INFO:teuthology.orchestra.run.smithi167.stdout: "mtime": "2023-10-05T19:28:35.066201Z", 2023-10-05T19:28:36.038 INFO:teuthology.orchestra.run.smithi167.stdout: "creation_time": "2023-10-05T19:28:30.250836Z", 2023-10-05T19:28:36.038 INFO:teuthology.orchestra.run.smithi167.stdout: "max_marker": "0#,1#,2#,3#,4#,5#,6#,7#,8#,9#,10#", 2023-10-05T19:28:36.038 INFO:teuthology.orchestra.run.smithi167.stdout: "usage": { 2023-10-05T19:28:36.038 INFO:teuthology.orchestra.run.smithi167.stdout: "rgw.main": { 2023-10-05T19:28:36.038 INFO:teuthology.orchestra.run.smithi167.stdout: "size": 0, 2023-10-05T19:28:36.038 INFO:teuthology.orchestra.run.smithi167.stdout: "size_actual": 0, 2023-10-05T19:28:36.038 INFO:teuthology.orchestra.run.smithi167.stdout: "size_utilized": 0, 2023-10-05T19:28:36.038 INFO:teuthology.orchestra.run.smithi167.stdout: "size_kb": 0, 2023-10-05T19:28:36.038 INFO:teuthology.orchestra.run.smithi167.stdout: "size_kb_actual": 0, 2023-10-05T19:28:36.038 INFO:teuthology.orchestra.run.smithi167.stdout: "size_kb_utilized": 0, 2023-10-05T19:28:36.039 INFO:teuthology.orchestra.run.smithi167.stdout: "num_objects": 0 2023-10-05T19:28:36.039 INFO:teuthology.orchestra.run.smithi167.stdout: } 2023-10-05T19:28:36.039 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2023-10-05T19:28:36.039 INFO:teuthology.orchestra.run.smithi167.stdout: "bucket_quota": { 2023-10-05T19:28:36.039 INFO:teuthology.orchestra.run.smithi167.stdout: "enabled": false, 2023-10-05T19:28:36.039 INFO:teuthology.orchestra.run.smithi167.stdout: "check_on_raw": false, 2023-10-05T19:28:36.039 INFO:teuthology.orchestra.run.smithi167.stdout: "max_size": -1, 2023-10-05T19:28:36.039 INFO:teuthology.orchestra.run.smithi167.stdout: "max_size_kb": 0, 2023-10-05T19:28:36.039 INFO:teuthology.orchestra.run.smithi167.stdout: "max_objects": -1 2023-10-05T19:28:36.039 INFO:teuthology.orchestra.run.smithi167.stdout: } 2023-10-05T19:28:36.039 INFO:teuthology.orchestra.run.smithi167.stdout:} 2023-10-05T19:28:36.040 DEBUG:tasks.util.rgw: json result: {'bucket': 'myfoo', 'num_shards': 11, 'tenant': '', 'zonegroup': '4f1f719d-9648-417b-b10c-e5d939825419', 'placement_rule': 'default-placement', 'explicit_placement': {'data_pool': '', 'data_extra_pool': '', 'index_pool': ''}, 'id': '9a9df965-6612-45d4-a45f-6e3c889bacc2.4223.1', 'marker': '9a9df965-6612-45d4-a45f-6e3c889bacc2.4223.1', 'index_type': 'Normal', 'owner': 'foo', 'ver': '0#1,1#1,2#1,3#1,4#1,5#3,6#3,7#1,8#3,9#1,10#1', 'master_ver': '0#0,1#0,2#0,3#0,4#0,5#0,6#0,7#0,8#0,9#0,10#0', 'mtime': '2023-10-05T19:28:35.066201Z', 'creation_time': '2023-10-05T19:28:30.250836Z', 'max_marker': '0#,1#,2#,3#,4#,5#,6#,7#,8#,9#,10#', 'usage': {'rgw.main': {'size': 0, 'size_actual': 0, 'size_utilized': 0, 'size_kb': 0, 'size_kb_actual': 0, 'size_kb_utilized': 0, 'num_objects': 0}}, 'bucket_quota': {'enabled': False, 'check_on_raw': False, 'max_size': -1, 'max_size_kb': 0, 'max_objects': -1}} 2023-10-05T19:28:36.041 INFO:tasks.util.rgw:rgwadmin: client.0 : ['log', 'list'] 2023-10-05T19:28:36.041 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'ceph', 'log', 'list'] 2023-10-05T19:28:36.041 DEBUG:teuthology.orchestra.run.smithi167:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster ceph log list 2023-10-05T19:28:36.078 INFO:teuthology.orchestra.run.smithi167.stderr:ignoring --setuser ceph since I am not root 2023-10-05T19:28:36.078 INFO:teuthology.orchestra.run.smithi167.stderr:ignoring --setgroup ceph since I am not root 2023-10-05T19:28:36.079 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.074+0000 7f536d8cee40 1 Processor -- start 2023-10-05T19:28:36.079 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.074+0000 7f536d8cee40 1 -- start start 2023-10-05T19:28:36.080 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.074+0000 7f536d8cee40 1 --2- >> v2:172.21.15.118:3300/0 conn(0x560cb900d3e0 0x560cb900f7c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-05T19:28:36.080 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.074+0000 7f536d8cee40 1 -- --> v1:172.21.15.118:6789/0 -- auth(proto 0 26 bytes epoch 0) v1 -- 0x560cb8f55c00 con 0x560cb9008e00 2023-10-05T19:28:36.080 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.074+0000 7f536d8cee40 1 -- --> v2:172.21.15.118:3300/0 -- mon_getmap magic: 0 v1 -- 0x560cb8f12ee0 con 0x560cb900d3e0 2023-10-05T19:28:36.080 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.074+0000 7f536ac02700 1 --2- >> v2:172.21.15.118:3300/0 conn(0x560cb900d3e0 0x560cb900f7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-05T19:28:36.080 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.074+0000 7f536b403700 1 --1- >> v1:172.21.15.118:6789/0 conn(0x560cb9008e00 0x560cb90091d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.118:6789/0 says I am v1:172.21.15.167:59310/0 (socket says 172.21.15.167:59310) 2023-10-05T19:28:36.080 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.074+0000 7f536ac02700 1 --2- >> v2:172.21.15.118:3300/0 conn(0x560cb900d3e0 0x560cb900f7c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.118:3300/0 says I am v2:172.21.15.167:41742/0 (socket says 172.21.15.167:41742) 2023-10-05T19:28:36.080 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.074+0000 7f536b403700 1 -- 172.21.15.167:0/2971878893 learned_addr learned my addr 172.21.15.167:0/2971878893 (peer_addr_for_me v1:172.21.15.167:0/0) 2023-10-05T19:28:36.081 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.074+0000 7f536ac02700 1 -- 172.21.15.167:0/2971878893 >> v1:172.21.15.118:6789/0 conn(0x560cb9008e00 legacy=0x560cb90091d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-05T19:28:36.081 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.074+0000 7f536ac02700 1 -- 172.21.15.167:0/2971878893 --> v2:172.21.15.118:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x560cb9009a10 con 0x560cb900d3e0 2023-10-05T19:28:36.082 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.074+0000 7f536ac02700 1 --2- 172.21.15.167:0/2971878893 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x560cb900d3e0 0x560cb900f7c0 secure :-1 s=READY pgs=435 cs=0 l=1 rev1=1 crypto rx=0x7f535400a5d0 tx=0x7f5354005a50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f0bf3bc172df1384 server_cookie=0 in_seq=0 out_seq=0 2023-10-05T19:28:36.082 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.074+0000 7f536a401700 1 -- 172.21.15.167:0/2971878893 <== mon.0 v2:172.21.15.118:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f53540086a0 con 0x560cb900d3e0 2023-10-05T19:28:36.082 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.074+0000 7f536a401700 1 -- 172.21.15.167:0/2971878893 <== mon.0 v2:172.21.15.118:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5354008800 con 0x560cb900d3e0 2023-10-05T19:28:36.082 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.074+0000 7f536a401700 1 -- 172.21.15.167:0/2971878893 <== mon.0 v2:172.21.15.118:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f5354008ad0 con 0x560cb900d3e0 2023-10-05T19:28:36.083 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.074+0000 7f536d8cee40 1 -- 172.21.15.167:0/2971878893 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x560cb900d3e0 msgr2=0x560cb900f7c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-05T19:28:36.083 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.074+0000 7f536d8cee40 1 --2- 172.21.15.167:0/2971878893 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x560cb900d3e0 0x560cb900f7c0 secure :-1 s=READY pgs=435 cs=0 l=1 rev1=1 crypto rx=0x7f535400a5d0 tx=0x7f5354005a50 comp rx=0 tx=0).stop 2023-10-05T19:28:36.084 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.074+0000 7f536d8cee40 1 -- 172.21.15.167:0/2971878893 shutdown_connections 2023-10-05T19:28:36.084 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.074+0000 7f536d8cee40 1 --2- 172.21.15.167:0/2971878893 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x560cb900d3e0 0x560cb900f7c0 unknown :-1 s=CLOSED pgs=435 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-05T19:28:36.084 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.074+0000 7f536d8cee40 1 -- 172.21.15.167:0/2971878893 >> 172.21.15.167:0/2971878893 conn(0x560cb9004820 msgr2=0x560cb9006c30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-05T19:28:36.084 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.074+0000 7f536d8cee40 1 -- 172.21.15.167:0/2971878893 shutdown_connections 2023-10-05T19:28:36.084 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.074+0000 7f536d8cee40 1 -- 172.21.15.167:0/2971878893 wait complete. 2023-10-05T19:28:36.084 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.078+0000 7f536d8cee40 1 Processor -- start 2023-10-05T19:28:36.084 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.078+0000 7f536d8cee40 1 -- start start 2023-10-05T19:28:36.084 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.078+0000 7f536d8cee40 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x560cb9008e00 0x560cb9027ed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-05T19:28:36.084 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.078+0000 7f536d8cee40 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x560cb8ed6e30 con 0x560cb9008e00 2023-10-05T19:28:36.084 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.078+0000 7f536b403700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x560cb9008e00 0x560cb9027ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-05T19:28:36.085 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.078+0000 7f536b403700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x560cb9008e00 0x560cb9027ed0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.118:3300/0 says I am v2:172.21.15.167:41750/0 (socket says 172.21.15.167:41750) 2023-10-05T19:28:36.085 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.078+0000 7f536b403700 1 -- 172.21.15.167:0/4109919618 learned_addr learned my addr 172.21.15.167:0/4109919618 (peer_addr_for_me v2:172.21.15.167:0/0) 2023-10-05T19:28:36.086 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.078+0000 7f536b403700 1 -- 172.21.15.167:0/4109919618 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5354005cf0 con 0x560cb9008e00 2023-10-05T19:28:36.087 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.078+0000 7f536b403700 1 --2- 172.21.15.167:0/4109919618 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x560cb9008e00 0x560cb9027ed0 secure :-1 s=READY pgs=436 cs=0 l=1 rev1=1 crypto rx=0x7f5364006fa0 tx=0x7f5364006b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e87f0171b2773fed server_cookie=0 in_seq=0 out_seq=0 2023-10-05T19:28:36.087 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.078+0000 7f53693ff700 1 -- 172.21.15.167:0/4109919618 <== mon.0 v2:172.21.15.118:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f5364006110 con 0x560cb9008e00 2023-10-05T19:28:36.087 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.078+0000 7f53693ff700 1 -- 172.21.15.167:0/4109919618 <== mon.0 v2:172.21.15.118:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5364006270 con 0x560cb9008e00 2023-10-05T19:28:36.087 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.078+0000 7f53693ff700 1 -- 172.21.15.167:0/4109919618 <== mon.0 v2:172.21.15.118:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f5364004520 con 0x560cb9008e00 2023-10-05T19:28:36.087 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.078+0000 7f536d8cee40 1 -- 172.21.15.167:0/4109919618 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x560cb9008e00 msgr2=0x560cb9027ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-05T19:28:36.087 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.078+0000 7f536d8cee40 1 --2- 172.21.15.167:0/4109919618 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x560cb9008e00 0x560cb9027ed0 secure :-1 s=READY pgs=436 cs=0 l=1 rev1=1 crypto rx=0x7f5364006fa0 tx=0x7f5364006b50 comp rx=0 tx=0).stop 2023-10-05T19:28:36.087 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.078+0000 7f536d8cee40 1 -- 172.21.15.167:0/4109919618 shutdown_connections 2023-10-05T19:28:36.088 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.078+0000 7f536d8cee40 1 --2- 172.21.15.167:0/4109919618 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x560cb9008e00 0x560cb9027ed0 unknown :-1 s=CLOSED pgs=436 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-05T19:28:36.088 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.078+0000 7f536d8cee40 1 -- 172.21.15.167:0/4109919618 >> 172.21.15.167:0/4109919618 conn(0x560cb9004820 msgr2=0x560cb9023a90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-05T19:28:36.088 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.078+0000 7f536d8cee40 1 -- 172.21.15.167:0/4109919618 shutdown_connections 2023-10-05T19:28:36.088 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.078+0000 7f536d8cee40 1 -- 172.21.15.167:0/4109919618 wait complete. 2023-10-05T19:28:36.088 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.082+0000 7f536d8cee40 1 Processor -- start 2023-10-05T19:28:36.088 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.082+0000 7f536d8cee40 1 -- start start 2023-10-05T19:28:36.088 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.082+0000 7f536d8cee40 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x560cb900d3e0 0x560cb902ee10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-05T19:28:36.088 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.082+0000 7f536d8cee40 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x560cb8ea9ed0 con 0x560cb900d3e0 2023-10-05T19:28:36.088 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.082+0000 7f536b403700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x560cb900d3e0 0x560cb902ee10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-05T19:28:36.088 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.082+0000 7f536b403700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x560cb900d3e0 0x560cb902ee10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.118:3300/0 says I am v2:172.21.15.167:41754/0 (socket says 172.21.15.167:41754) 2023-10-05T19:28:36.089 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.082+0000 7f536b403700 1 -- 172.21.15.167:0/187505431 learned_addr learned my addr 172.21.15.167:0/187505431 (peer_addr_for_me v2:172.21.15.167:0/0) 2023-10-05T19:28:36.089 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.082+0000 7f536b403700 1 -- 172.21.15.167:0/187505431 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x560cb8f55c00 con 0x560cb900d3e0 2023-10-05T19:28:36.090 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.082+0000 7f536b403700 1 --2- 172.21.15.167:0/187505431 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x560cb900d3e0 0x560cb902ee10 secure :-1 s=READY pgs=437 cs=0 l=1 rev1=1 crypto rx=0x7f5364000f80 tx=0x7f5364014000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-05T19:28:36.090 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.082+0000 7f53637fe700 1 -- 172.21.15.167:0/187505431 <== mon.0 v2:172.21.15.118:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f5364006110 con 0x560cb900d3e0 2023-10-05T19:28:36.090 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.082+0000 7f53637fe700 1 -- 172.21.15.167:0/187505431 <== mon.0 v2:172.21.15.118:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f53640063e0 con 0x560cb900d3e0 2023-10-05T19:28:36.090 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.082+0000 7f536d8cee40 1 -- 172.21.15.167:0/187505431 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x560cb902f350 con 0x560cb900d3e0 2023-10-05T19:28:36.090 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.082+0000 7f53637fe700 1 -- 172.21.15.167:0/187505431 <== mon.0 v2:172.21.15.118:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f5364013930 con 0x560cb900d3e0 2023-10-05T19:28:36.090 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.082+0000 7f536d8cee40 1 -- 172.21.15.167:0/187505431 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x560cb902f790 con 0x560cb900d3e0 2023-10-05T19:28:36.090 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.082+0000 7f536d8cee40 20 rados->read ofs=0 len=0 2023-10-05T19:28:36.091 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.086+0000 7f53637fe700 1 -- 172.21.15.167:0/187505431 <== mon.0 v2:172.21.15.118:3300/0 4 ==== mgrmap(e 5) v1 ==== 40329+0+0 (secure 0 0 0) 0x7f536401a070 con 0x560cb900d3e0 2023-10-05T19:28:36.091 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.086+0000 7f53637fe700 1 --2- 172.21.15.167:0/187505431 >> [v2:172.21.15.167:6824/29889,v1:172.21.15.167:6825/29889] conn(0x7f534c030c00 0x7f534c0330b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-05T19:28:36.091 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.086+0000 7f53637fe700 1 -- 172.21.15.167:0/187505431 <== mon.0 v2:172.21.15.118:3300/0 5 ==== osd_map(30..30 src has 1..30) v4 ==== 6057+0+0 (secure 0 0 0) 0x7f5364044d20 con 0x560cb900d3e0 2023-10-05T19:28:36.091 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.086+0000 7f536ac02700 1 --2- 172.21.15.167:0/187505431 >> [v2:172.21.15.167:6824/29889,v1:172.21.15.167:6825/29889] conn(0x7f534c030c00 0x7f534c0330b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-05T19:28:36.092 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.086+0000 7f536d8cee40 1 --2- 172.21.15.167:0/187505431 >> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] conn(0x560cb90388c0 0x560cb903aca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-05T19:28:36.092 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.086+0000 7f536d8cee40 1 -- 172.21.15.167:0/187505431 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:1 5.12 5:49953fa1:::default.realm:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x560cb903b3a0 con 0x560cb90388c0 2023-10-05T19:28:36.092 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.086+0000 7f536bc04700 1 --2- 172.21.15.167:0/187505431 >> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] conn(0x560cb90388c0 0x560cb903aca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-05T19:28:36.092 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.086+0000 7f536ac02700 1 --2- 172.21.15.167:0/187505431 >> [v2:172.21.15.167:6824/29889,v1:172.21.15.167:6825/29889] conn(0x7f534c030c00 0x7f534c0330b0 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7f5354005a90 tx=0x7f5354015000 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-05T19:28:36.092 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.086+0000 7f536bc04700 1 --2- 172.21.15.167:0/187505431 >> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] conn(0x560cb90388c0 0x560cb903aca0 crc :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-05T19:28:36.093 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.086+0000 7f536bc04700 1 -- 172.21.15.167:0/187505431 <== osd.1 v2:172.21.15.167:6800/30035 1 ==== osd_op_reply(1 default.realm [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 ==== 157+0+0 (crc 0 0 0) 0x7f535c00a500 con 0x560cb90388c0 2023-10-05T19:28:36.093 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.086+0000 7f536d8cee40 20 rados_obj.operate() r=-2 bl.length=0 2023-10-05T19:28:36.093 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.086+0000 7f536d8cee40 20 realm 2023-10-05T19:28:36.093 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.086+0000 7f536d8cee40 20 rados->read ofs=0 len=0 2023-10-05T19:28:36.093 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.086+0000 7f536d8cee40 1 -- 172.21.15.167:0/187505431 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:2 5.12 5:49953fa1:::default.realm:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x560cb9037f30 con 0x560cb90388c0 2023-10-05T19:28:36.094 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.086+0000 7f536bc04700 1 -- 172.21.15.167:0/187505431 <== osd.1 v2:172.21.15.167:6800/30035 2 ==== osd_op_reply(2 default.realm [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 ==== 157+0+0 (crc 0 0 0) 0x7f535c00a500 con 0x560cb90388c0 2023-10-05T19:28:36.094 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.086+0000 7f536d8cee40 20 rados_obj.operate() r=-2 bl.length=0 2023-10-05T19:28:36.094 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.086+0000 7f536d8cee40 4 RGWPeriod::init failed to init realm id : (2) No such file or directory 2023-10-05T19:28:36.094 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.086+0000 7f536d8cee40 20 rados->read ofs=0 len=0 2023-10-05T19:28:36.094 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.086+0000 7f536d8cee40 1 -- 172.21.15.167:0/187505431 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:3 5.12 5:49953fa1:::default.realm:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x560cb903be80 con 0x560cb90388c0 2023-10-05T19:28:36.095 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.090+0000 7f536bc04700 1 -- 172.21.15.167:0/187505431 <== osd.1 v2:172.21.15.167:6800/30035 3 ==== osd_op_reply(3 default.realm [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 ==== 157+0+0 (crc 0 0 0) 0x7f535c00a500 con 0x560cb90388c0 2023-10-05T19:28:36.095 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.090+0000 7f536d8cee40 20 rados_obj.operate() r=-2 bl.length=0 2023-10-05T19:28:36.095 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.090+0000 7f536d8cee40 20 rados->read ofs=0 len=0 2023-10-05T19:28:36.095 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.090+0000 7f536d8cee40 1 -- 172.21.15.167:0/187505431 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:4 5.f 5:f43b6ece:::zone_names.default:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x560cb903ba60 con 0x560cb90388c0 2023-10-05T19:28:36.096 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.090+0000 7f536bc04700 1 -- 172.21.15.167:0/187505431 <== osd.1 v2:172.21.15.167:6800/30035 4 ==== osd_op_reply(4 zone_names.default [read 0~46 out=46b] v0'0 uv1 ondisk = 0) v8 ==== 162+0+46 (crc 0 0 0) 0x7f535c00a500 con 0x560cb90388c0 2023-10-05T19:28:36.096 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.090+0000 7f536d8cee40 20 rados_obj.operate() r=0 bl.length=46 2023-10-05T19:28:36.096 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.090+0000 7f536d8cee40 20 rados->read ofs=0 len=0 2023-10-05T19:28:36.096 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.090+0000 7f536d8cee40 1 -- 172.21.15.167:0/187505431 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:5 5.1d 5:b8374eb4:::zone_info.9a9df965-6612-45d4-a45f-6e3c889bacc2:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x560cb903c8f0 con 0x560cb90388c0 2023-10-05T19:28:36.096 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.090+0000 7f536bc04700 1 -- 172.21.15.167:0/187505431 <== osd.1 v2:172.21.15.167:6800/30035 5 ==== osd_op_reply(5 zone_info.9a9df965-6612-45d4-a45f-6e3c889bacc2 [read 0~874 out=874b] v0'0 uv2 ondisk = 0) v8 ==== 190+0+874 (crc 0 0 0) 0x7f535c00a500 con 0x560cb90388c0 2023-10-05T19:28:36.096 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.090+0000 7f536d8cee40 20 rados_obj.operate() r=0 bl.length=874 2023-10-05T19:28:36.096 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.090+0000 7f536d8cee40 20 searching for the correct realm 2023-10-05T19:28:36.097 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.090+0000 7f536d8cee40 1 --2- 172.21.15.167:0/187505431 >> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] conn(0x560cb903e740 0x560cb9040b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-05T19:28:36.097 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.090+0000 7f536d8cee40 1 -- 172.21.15.167:0/187505431 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:6 5.0 5:00000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x560cb9041060 con 0x560cb903e740 2023-10-05T19:28:36.097 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.090+0000 7f536b403700 1 --2- 172.21.15.167:0/187505431 >> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] conn(0x560cb903e740 0x560cb9040b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-05T19:28:36.097 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.090+0000 7f536b403700 1 --2- 172.21.15.167:0/187505431 >> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] conn(0x560cb903e740 0x560cb9040b20 crc :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-05T19:28:36.099 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.094+0000 7f536b403700 1 -- 172.21.15.167:0/187505431 <== osd.3 v2:172.21.15.167:6808/30036 1 ==== osd_op_reply(6 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f5364025020 con 0x560cb903e740 2023-10-05T19:28:36.099 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.094+0000 7f536b403700 1 --2- 172.21.15.167:0/187505431 >> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] conn(0x560cb9008e00 0x7f53640113e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-05T19:28:36.099 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.094+0000 7f536b403700 1 -- 172.21.15.167:0/187505431 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:7 5.10 5:08000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x560cb9041060 con 0x560cb9008e00 2023-10-05T19:28:36.100 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.094+0000 7f536ac02700 1 --2- 172.21.15.167:0/187505431 >> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] conn(0x560cb9008e00 0x7f53640113e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-05T19:28:36.100 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.094+0000 7f536ac02700 1 --2- 172.21.15.167:0/187505431 >> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] conn(0x560cb9008e00 0x7f53640113e0 crc :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-05T19:28:36.100 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.094+0000 7f536ac02700 1 -- 172.21.15.167:0/187505431 <== osd.2 v2:172.21.15.167:6816/30037 1 ==== osd_op_reply(7 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f5354012450 con 0x560cb9008e00 2023-10-05T19:28:36.101 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.094+0000 7f536ac02700 1 --2- 172.21.15.167:0/187505431 >> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] conn(0x7f5354007030 0x7f535401c920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-05T19:28:36.101 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.094+0000 7f536ac02700 1 -- 172.21.15.167:0/187505431 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:8 5.8 5:10000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x560cb9041060 con 0x7f5354007030 2023-10-05T19:28:36.101 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.094+0000 7f536bc04700 1 --2- 172.21.15.167:0/187505431 >> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] conn(0x7f5354007030 0x7f535401c920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-05T19:28:36.101 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.094+0000 7f536bc04700 1 --2- 172.21.15.167:0/187505431 >> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] conn(0x7f5354007030 0x7f535401c920 crc :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-05T19:28:36.103 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.098+0000 7f536bc04700 1 -- 172.21.15.167:0/187505431 <== osd.0 v2:172.21.15.118:6800/27690 1 ==== osd_op_reply(8 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f535c014040 con 0x7f5354007030 2023-10-05T19:28:36.103 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.098+0000 7f536bc04700 1 -- 172.21.15.167:0/187505431 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:9 5.18 5:18000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x560cb9041060 con 0x560cb90388c0 2023-10-05T19:28:36.103 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.098+0000 7f536bc04700 1 -- 172.21.15.167:0/187505431 <== osd.1 v2:172.21.15.167:6800/30035 6 ==== osd_op_reply(9 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f535c014040 con 0x560cb90388c0 2023-10-05T19:28:36.103 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.098+0000 7f536bc04700 1 -- 172.21.15.167:0/187505431 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:10 5.4 5:20000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x560cb9041060 con 0x560cb90388c0 2023-10-05T19:28:36.103 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.098+0000 7f536bc04700 1 -- 172.21.15.167:0/187505431 <== osd.1 v2:172.21.15.167:6800/30035 7 ==== osd_op_reply(10 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f535c014040 con 0x560cb90388c0 2023-10-05T19:28:36.104 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.098+0000 7f536bc04700 1 -- 172.21.15.167:0/187505431 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:11 5.14 5:28000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x560cb9041060 con 0x7f5354007030 2023-10-05T19:28:36.105 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.098+0000 7f536bc04700 1 -- 172.21.15.167:0/187505431 <== osd.0 v2:172.21.15.118:6800/27690 2 ==== osd_op_reply(11 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f535c014040 con 0x7f5354007030 2023-10-05T19:28:36.105 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.098+0000 7f536bc04700 1 -- 172.21.15.167:0/187505431 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:12 5.c 5:30000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x560cb9041060 con 0x7f5354007030 2023-10-05T19:28:36.106 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.098+0000 7f536bc04700 1 -- 172.21.15.167:0/187505431 <== osd.0 v2:172.21.15.118:6800/27690 3 ==== osd_op_reply(12 [pgnls start_epoch 30 out=112b] v23'1 uv1 ondisk = 1) v8 ==== 144+0+112 (crc 0 0 0) 0x7f535c014040 con 0x7f5354007030 2023-10-05T19:28:36.106 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.098+0000 7f536bc04700 1 -- 172.21.15.167:0/187505431 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:13 5.1c 5:38000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x560cb9041060 con 0x560cb9008e00 2023-10-05T19:28:36.106 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.102+0000 7f536ac02700 1 -- 172.21.15.167:0/187505431 <== osd.2 v2:172.21.15.167:6816/30037 2 ==== osd_op_reply(13 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f5354012450 con 0x560cb9008e00 2023-10-05T19:28:36.106 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.102+0000 7f536ac02700 1 -- 172.21.15.167:0/187505431 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:14 5.2 5:40000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x560cb9041060 con 0x560cb903e740 2023-10-05T19:28:36.107 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.102+0000 7f536b403700 1 -- 172.21.15.167:0/187505431 <== osd.3 v2:172.21.15.167:6808/30036 2 ==== osd_op_reply(14 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f5364025020 con 0x560cb903e740 2023-10-05T19:28:36.107 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.102+0000 7f536b403700 1 -- 172.21.15.167:0/187505431 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:15 5.12 5:48000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x560cb9041060 con 0x560cb90388c0 2023-10-05T19:28:36.107 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.102+0000 7f536bc04700 1 -- 172.21.15.167:0/187505431 <== osd.1 v2:172.21.15.167:6800/30035 8 ==== osd_op_reply(15 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f535c014040 con 0x560cb90388c0 2023-10-05T19:28:36.108 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.102+0000 7f536bc04700 1 -- 172.21.15.167:0/187505431 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:16 5.a 5:50000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x560cb9041060 con 0x7f5354007030 2023-10-05T19:28:36.109 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.102+0000 7f536bc04700 1 -- 172.21.15.167:0/187505431 <== osd.0 v2:172.21.15.118:6800/27690 4 ==== osd_op_reply(16 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f535c022040 con 0x7f5354007030 2023-10-05T19:28:36.109 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.102+0000 7f536bc04700 1 -- 172.21.15.167:0/187505431 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:17 5.1a 5:58000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x560cb9041060 con 0x560cb90388c0 2023-10-05T19:28:36.109 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.102+0000 7f536bc04700 1 -- 172.21.15.167:0/187505431 <== osd.1 v2:172.21.15.167:6800/30035 9 ==== osd_op_reply(17 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f535c022040 con 0x560cb90388c0 2023-10-05T19:28:36.109 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.102+0000 7f536bc04700 1 -- 172.21.15.167:0/187505431 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:18 5.6 5:60000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x560cb9041060 con 0x560cb9008e00 2023-10-05T19:28:36.110 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.106+0000 7f536ac02700 1 -- 172.21.15.167:0/187505431 <== osd.2 v2:172.21.15.167:6816/30037 3 ==== osd_op_reply(18 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f5354012450 con 0x560cb9008e00 2023-10-05T19:28:36.110 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.106+0000 7f536ac02700 1 -- 172.21.15.167:0/187505431 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:19 5.16 5:68000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x560cb9041060 con 0x560cb903e740 2023-10-05T19:28:36.111 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.106+0000 7f536b403700 1 -- 172.21.15.167:0/187505431 <== osd.3 v2:172.21.15.167:6808/30036 3 ==== osd_op_reply(19 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f5364025020 con 0x560cb903e740 2023-10-05T19:28:36.111 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.106+0000 7f536b403700 1 -- 172.21.15.167:0/187505431 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:20 5.e 5:70000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x560cb9041060 con 0x560cb903e740 2023-10-05T19:28:36.112 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.106+0000 7f536b403700 1 -- 172.21.15.167:0/187505431 <== osd.3 v2:172.21.15.167:6808/30036 4 ==== osd_op_reply(20 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f5364025020 con 0x560cb903e740 2023-10-05T19:28:36.112 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.106+0000 7f536b403700 1 -- 172.21.15.167:0/187505431 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:21 5.1e 5:78000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x560cb9041060 con 0x7f5354007030 2023-10-05T19:28:36.113 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.106+0000 7f536bc04700 1 -- 172.21.15.167:0/187505431 <== osd.0 v2:172.21.15.118:6800/27690 5 ==== osd_op_reply(21 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f535c022040 con 0x7f5354007030 2023-10-05T19:28:36.113 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.106+0000 7f536bc04700 1 -- 172.21.15.167:0/187505431 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:22 5.1 5:80000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x560cb9041060 con 0x560cb903e740 2023-10-05T19:28:36.114 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.106+0000 7f536b403700 1 -- 172.21.15.167:0/187505431 <== osd.3 v2:172.21.15.167:6808/30036 5 ==== osd_op_reply(22 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f5364025020 con 0x560cb903e740 2023-10-05T19:28:36.114 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.106+0000 7f536b403700 1 -- 172.21.15.167:0/187505431 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:23 5.11 5:88000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x560cb9041060 con 0x560cb903e740 2023-10-05T19:28:36.114 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.110+0000 7f536b403700 1 -- 172.21.15.167:0/187505431 <== osd.3 v2:172.21.15.167:6808/30036 6 ==== osd_op_reply(23 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f5364025020 con 0x560cb903e740 2023-10-05T19:28:36.115 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.110+0000 7f536b403700 1 -- 172.21.15.167:0/187505431 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:24 5.9 5:90000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x560cb9041060 con 0x560cb90388c0 2023-10-05T19:28:36.115 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.110+0000 7f536bc04700 1 -- 172.21.15.167:0/187505431 <== osd.1 v2:172.21.15.167:6800/30035 10 ==== osd_op_reply(24 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f535c022040 con 0x560cb90388c0 2023-10-05T19:28:36.115 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.110+0000 7f536bc04700 1 -- 172.21.15.167:0/187505431 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:25 5.19 5:98000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x560cb9041060 con 0x560cb90388c0 2023-10-05T19:28:36.116 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.110+0000 7f536bc04700 1 -- 172.21.15.167:0/187505431 <== osd.1 v2:172.21.15.167:6800/30035 11 ==== osd_op_reply(25 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f535c022040 con 0x560cb90388c0 2023-10-05T19:28:36.116 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.110+0000 7f536bc04700 1 -- 172.21.15.167:0/187505431 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:26 5.5 5:a0000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x560cb9041060 con 0x7f5354007030 2023-10-05T19:28:36.117 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.110+0000 7f536bc04700 1 -- 172.21.15.167:0/187505431 <== osd.0 v2:172.21.15.118:6800/27690 6 ==== osd_op_reply(26 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f535c022040 con 0x7f5354007030 2023-10-05T19:28:36.117 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.110+0000 7f536bc04700 1 -- 172.21.15.167:0/187505431 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:27 5.15 5:a8000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x560cb9041060 con 0x560cb903e740 2023-10-05T19:28:36.118 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.110+0000 7f536b403700 1 -- 172.21.15.167:0/187505431 <== osd.3 v2:172.21.15.167:6808/30036 7 ==== osd_op_reply(27 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f5364025020 con 0x560cb903e740 2023-10-05T19:28:36.118 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.110+0000 7f536b403700 1 -- 172.21.15.167:0/187505431 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:28 5.d 5:b0000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x560cb9041060 con 0x560cb9008e00 2023-10-05T19:28:36.118 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.114+0000 7f536ac02700 1 -- 172.21.15.167:0/187505431 <== osd.2 v2:172.21.15.167:6816/30037 4 ==== osd_op_reply(28 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f5354012450 con 0x560cb9008e00 2023-10-05T19:28:36.119 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.114+0000 7f536ac02700 1 -- 172.21.15.167:0/187505431 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:29 5.1d 5:b8000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x560cb9041060 con 0x560cb90388c0 2023-10-05T19:28:36.119 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.114+0000 7f536bc04700 1 -- 172.21.15.167:0/187505431 <== osd.1 v2:172.21.15.167:6800/30035 12 ==== osd_op_reply(29 [pgnls start_epoch 30 out=107b] v30'2 uv2 ondisk = 1) v8 ==== 144+0+107 (crc 0 0 0) 0x7f535c022040 con 0x560cb90388c0 2023-10-05T19:28:36.119 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.114+0000 7f536bc04700 1 -- 172.21.15.167:0/187505431 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:30 5.3 5:c0000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x560cb9041060 con 0x7f5354007030 2023-10-05T19:28:36.120 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.114+0000 7f536bc04700 1 -- 172.21.15.167:0/187505431 <== osd.0 v2:172.21.15.118:6800/27690 7 ==== osd_op_reply(30 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f535c022040 con 0x7f5354007030 2023-10-05T19:28:36.120 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.114+0000 7f536bc04700 1 -- 172.21.15.167:0/187505431 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:31 5.13 5:c8000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x560cb9041060 con 0x560cb903e740 2023-10-05T19:28:36.121 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.114+0000 7f536b403700 1 -- 172.21.15.167:0/187505431 <== osd.3 v2:172.21.15.167:6808/30036 8 ==== osd_op_reply(31 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f5364025020 con 0x560cb903e740 2023-10-05T19:28:36.121 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.114+0000 7f536b403700 1 -- 172.21.15.167:0/187505431 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:32 5.b 5:d0000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x560cb9041060 con 0x7f5354007030 2023-10-05T19:28:36.122 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.118+0000 7f536bc04700 1 -- 172.21.15.167:0/187505431 <== osd.0 v2:172.21.15.118:6800/27690 8 ==== osd_op_reply(32 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f535c022040 con 0x7f5354007030 2023-10-05T19:28:36.122 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.118+0000 7f536bc04700 1 -- 172.21.15.167:0/187505431 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:33 5.1b 5:d8000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x560cb9041060 con 0x560cb9008e00 2023-10-05T19:28:36.126 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.118+0000 7f536ac02700 1 -- 172.21.15.167:0/187505431 <== osd.2 v2:172.21.15.167:6816/30037 5 ==== osd_op_reply(33 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f5354012450 con 0x560cb9008e00 2023-10-05T19:28:36.126 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.118+0000 7f536ac02700 1 -- 172.21.15.167:0/187505431 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:34 5.7 5:e0000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x560cb9041060 con 0x560cb903e740 2023-10-05T19:28:36.126 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.118+0000 7f536b403700 1 -- 172.21.15.167:0/187505431 <== osd.3 v2:172.21.15.167:6808/30036 9 ==== osd_op_reply(34 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f536403d040 con 0x560cb903e740 2023-10-05T19:28:36.126 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.118+0000 7f536b403700 1 -- 172.21.15.167:0/187505431 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:35 5.17 5:e8000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x560cb9041060 con 0x7f5354007030 2023-10-05T19:28:36.126 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.118+0000 7f536bc04700 1 -- 172.21.15.167:0/187505431 <== osd.0 v2:172.21.15.118:6800/27690 9 ==== osd_op_reply(35 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f535c022040 con 0x7f5354007030 2023-10-05T19:28:36.127 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.118+0000 7f536bc04700 1 -- 172.21.15.167:0/187505431 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:36 5.f 5:f0000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x560cb9041060 con 0x560cb90388c0 2023-10-05T19:28:36.127 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.118+0000 7f536bc04700 1 -- 172.21.15.167:0/187505431 <== osd.1 v2:172.21.15.167:6800/30035 13 ==== osd_op_reply(36 [pgnls start_epoch 30 out=115b] v23'2 uv2 ondisk = 1) v8 ==== 144+0+115 (crc 0 0 0) 0x7f535c02f080 con 0x560cb90388c0 2023-10-05T19:28:36.127 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.118+0000 7f536bc04700 1 -- 172.21.15.167:0/187505431 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:37 5.1f 5:f8000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x560cb9041060 con 0x560cb9008e00 2023-10-05T19:28:36.128 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.122+0000 7f536ac02700 1 -- 172.21.15.167:0/187505431 <== osd.2 v2:172.21.15.167:6816/30037 6 ==== osd_op_reply(37 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f5354012450 con 0x560cb9008e00 2023-10-05T19:28:36.128 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.122+0000 7f536d8cee40 20 RGWRados::pool_iterate: got zonegroup_info.4f1f719d-9648-417b-b10c-e5d939825419 2023-10-05T19:28:36.128 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.122+0000 7f536d8cee40 20 RGWRados::pool_iterate: got zone_info.9a9df965-6612-45d4-a45f-6e3c889bacc2 2023-10-05T19:28:36.128 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.122+0000 7f536d8cee40 20 RGWRados::pool_iterate: got zone_names.default 2023-10-05T19:28:36.128 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.122+0000 7f536d8cee40 20 RGWRados::pool_iterate: got zonegroups_names.default 2023-10-05T19:28:36.129 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.122+0000 7f536d8cee40 20 rados->read ofs=0 len=0 2023-10-05T19:28:36.129 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.122+0000 7f536d8cee40 1 -- 172.21.15.167:0/187505431 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:38 5.12 5:49953fa1:::default.realm:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x560cb903d4f0 con 0x560cb90388c0 2023-10-05T19:28:36.129 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.122+0000 7f536bc04700 1 -- 172.21.15.167:0/187505431 <== osd.1 v2:172.21.15.167:6800/30035 14 ==== osd_op_reply(38 default.realm [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 ==== 157+0+0 (crc 0 0 0) 0x7f535c02f080 con 0x560cb90388c0 2023-10-05T19:28:36.129 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.122+0000 7f536d8cee40 20 rados_obj.operate() r=-2 bl.length=0 2023-10-05T19:28:36.129 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.122+0000 7f536d8cee40 20 rados->read ofs=0 len=0 2023-10-05T19:28:36.129 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.122+0000 7f536d8cee40 1 -- 172.21.15.167:0/187505431 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:39 5.f 5:f4c53578:::zonegroups_names.default:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x560cb903de70 con 0x560cb90388c0 2023-10-05T19:28:36.129 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.122+0000 7f536bc04700 1 -- 172.21.15.167:0/187505431 <== osd.1 v2:172.21.15.167:6800/30035 15 ==== osd_op_reply(39 zonegroups_names.default [read 0~46 out=46b] v0'0 uv2 ondisk = 0) v8 ==== 168+0+46 (crc 0 0 0) 0x7f535c02f080 con 0x560cb90388c0 2023-10-05T19:28:36.129 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.122+0000 7f536d8cee40 20 rados_obj.operate() r=0 bl.length=46 2023-10-05T19:28:36.129 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.122+0000 7f536d8cee40 20 rados->read ofs=0 len=0 2023-10-05T19:28:36.129 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.122+0000 7f536d8cee40 1 -- 172.21.15.167:0/187505431 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:40 5.c 5:30d299a5:::zonegroup_info.4f1f719d-9648-417b-b10c-e5d939825419:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x560cb903d0e0 con 0x7f5354007030 2023-10-05T19:28:36.130 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.122+0000 7f536bc04700 1 -- 172.21.15.167:0/187505431 <== osd.0 v2:172.21.15.118:6800/27690 10 ==== osd_op_reply(40 zonegroup_info.4f1f719d-9648-417b-b10c-e5d939825419 [read 0~362 out=362b] v0'0 uv1 ondisk = 0) v8 ==== 195+0+362 (crc 0 0 0) 0x7f535c02f080 con 0x7f5354007030 2023-10-05T19:28:36.131 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.122+0000 7f536d8cee40 20 rados_obj.operate() r=0 bl.length=362 2023-10-05T19:28:36.131 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.122+0000 7f536d8cee40 20 rados->read ofs=0 len=0 2023-10-05T19:28:36.131 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.122+0000 7f536d8cee40 1 -- 172.21.15.167:0/187505431 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:41 5.13 5:cbbf6c60:::converted:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x560cb9041950 con 0x560cb903e740 2023-10-05T19:28:36.131 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.126+0000 7f536b403700 1 -- 172.21.15.167:0/187505431 <== osd.3 v2:172.21.15.167:6808/30036 10 ==== osd_op_reply(41 converted [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 ==== 153+0+0 (crc 0 0 0) 0x7f536403d040 con 0x560cb903e740 2023-10-05T19:28:36.131 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.126+0000 7f536d8cee40 20 rados_obj.operate() r=-2 bl.length=0 2023-10-05T19:28:36.131 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.126+0000 7f536d8cee40 20 rados->read ofs=0 len=0 2023-10-05T19:28:36.131 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.126+0000 7f536d8cee40 1 -- 172.21.15.167:0/187505431 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:42 5.12 5:49953fa1:::default.realm:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x560cb9041d00 con 0x560cb90388c0 2023-10-05T19:28:36.132 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.126+0000 7f536bc04700 1 -- 172.21.15.167:0/187505431 <== osd.1 v2:172.21.15.167:6800/30035 16 ==== osd_op_reply(42 default.realm [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 ==== 157+0+0 (crc 0 0 0) 0x7f535c02f080 con 0x560cb90388c0 2023-10-05T19:28:36.132 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.126+0000 7f536d8cee40 20 rados_obj.operate() r=-2 bl.length=0 2023-10-05T19:28:36.132 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.126+0000 7f536d8cee40 20 rados->read ofs=0 len=0 2023-10-05T19:28:36.132 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.126+0000 7f536d8cee40 1 -- 172.21.15.167:0/187505431 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:43 5.f 5:f4c53578:::zonegroups_names.default:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x560cb9041d00 con 0x560cb90388c0 2023-10-05T19:28:36.132 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.126+0000 7f536bc04700 1 -- 172.21.15.167:0/187505431 <== osd.1 v2:172.21.15.167:6800/30035 17 ==== osd_op_reply(43 zonegroups_names.default [read 0~46 out=46b] v0'0 uv2 ondisk = 0) v8 ==== 168+0+46 (crc 0 0 0) 0x7f535c02f080 con 0x560cb90388c0 2023-10-05T19:28:36.132 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.126+0000 7f536d8cee40 20 rados_obj.operate() r=0 bl.length=46 2023-10-05T19:28:36.132 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.126+0000 7f536d8cee40 1 -- 172.21.15.167:0/187505431 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:44 5.0 5:00000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x560cb9042d10 con 0x560cb903e740 2023-10-05T19:28:36.134 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.130+0000 7f536b403700 1 -- 172.21.15.167:0/187505431 <== osd.3 v2:172.21.15.167:6808/30036 11 ==== osd_op_reply(44 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f536403d040 con 0x560cb903e740 2023-10-05T19:28:36.134 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.130+0000 7f536b403700 1 -- 172.21.15.167:0/187505431 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:45 5.10 5:08000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x560cb9042d10 con 0x560cb9008e00 2023-10-05T19:28:36.135 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.130+0000 7f536ac02700 1 -- 172.21.15.167:0/187505431 <== osd.2 v2:172.21.15.167:6816/30037 7 ==== osd_op_reply(45 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f5354012450 con 0x560cb9008e00 2023-10-05T19:28:36.135 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.130+0000 7f536ac02700 1 -- 172.21.15.167:0/187505431 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:46 5.8 5:10000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x560cb9042d10 con 0x7f5354007030 2023-10-05T19:28:36.136 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.130+0000 7f536bc04700 1 -- 172.21.15.167:0/187505431 <== osd.0 v2:172.21.15.118:6800/27690 11 ==== osd_op_reply(46 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f535c02f080 con 0x7f5354007030 2023-10-05T19:28:36.136 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.130+0000 7f536bc04700 1 -- 172.21.15.167:0/187505431 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:47 5.18 5:18000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x560cb903d0e0 con 0x560cb90388c0 2023-10-05T19:28:36.137 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.130+0000 7f536bc04700 1 -- 172.21.15.167:0/187505431 <== osd.1 v2:172.21.15.167:6800/30035 18 ==== osd_op_reply(47 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f535c02f080 con 0x560cb90388c0 2023-10-05T19:28:36.137 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.130+0000 7f536bc04700 1 -- 172.21.15.167:0/187505431 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:48 5.4 5:20000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x560cb903d0e0 con 0x560cb90388c0 2023-10-05T19:28:36.137 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.130+0000 7f536bc04700 1 -- 172.21.15.167:0/187505431 <== osd.1 v2:172.21.15.167:6800/30035 19 ==== osd_op_reply(48 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f535c02f080 con 0x560cb90388c0 2023-10-05T19:28:36.138 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.130+0000 7f536bc04700 1 -- 172.21.15.167:0/187505431 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:49 5.14 5:28000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x560cb903d0e0 con 0x7f5354007030 2023-10-05T19:28:36.139 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.134+0000 7f536bc04700 1 -- 172.21.15.167:0/187505431 <== osd.0 v2:172.21.15.118:6800/27690 12 ==== osd_op_reply(49 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f535c02f080 con 0x7f5354007030 2023-10-05T19:28:36.139 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.134+0000 7f536bc04700 1 -- 172.21.15.167:0/187505431 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:50 5.c 5:30000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x560cb903d0e0 con 0x7f5354007030 2023-10-05T19:28:36.140 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.134+0000 7f536bc04700 1 -- 172.21.15.167:0/187505431 <== osd.0 v2:172.21.15.118:6800/27690 13 ==== osd_op_reply(50 [pgnls start_epoch 30 out=112b] v23'1 uv1 ondisk = 1) v8 ==== 144+0+112 (crc 0 0 0) 0x7f535c02f080 con 0x7f5354007030 2023-10-05T19:28:36.140 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.134+0000 7f536bc04700 1 -- 172.21.15.167:0/187505431 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:51 5.1c 5:38000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x560cb903d0e0 con 0x560cb9008e00 2023-10-05T19:28:36.141 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.134+0000 7f536ac02700 1 -- 172.21.15.167:0/187505431 <== osd.2 v2:172.21.15.167:6816/30037 8 ==== osd_op_reply(51 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f5354012450 con 0x560cb9008e00 2023-10-05T19:28:36.141 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.134+0000 7f536ac02700 1 -- 172.21.15.167:0/187505431 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:52 5.2 5:40000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x560cb903d0e0 con 0x560cb903e740 2023-10-05T19:28:36.141 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.134+0000 7f536b403700 1 -- 172.21.15.167:0/187505431 <== osd.3 v2:172.21.15.167:6808/30036 12 ==== osd_op_reply(52 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f536403d040 con 0x560cb903e740 2023-10-05T19:28:36.141 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.134+0000 7f536b403700 1 -- 172.21.15.167:0/187505431 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:53 5.12 5:48000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x560cb903d0e0 con 0x560cb90388c0 2023-10-05T19:28:36.142 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.138+0000 7f536bc04700 1 -- 172.21.15.167:0/187505431 <== osd.1 v2:172.21.15.167:6800/30035 20 ==== osd_op_reply(53 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f535c02f080 con 0x560cb90388c0 2023-10-05T19:28:36.142 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.138+0000 7f536bc04700 1 -- 172.21.15.167:0/187505431 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:54 5.a 5:50000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x560cb903d0e0 con 0x7f5354007030 2023-10-05T19:28:36.143 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.138+0000 7f536bc04700 1 -- 172.21.15.167:0/187505431 <== osd.0 v2:172.21.15.118:6800/27690 14 ==== osd_op_reply(54 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f535c02f080 con 0x7f5354007030 2023-10-05T19:28:36.143 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.138+0000 7f536bc04700 1 -- 172.21.15.167:0/187505431 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:55 5.1a 5:58000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x560cb903d0e0 con 0x560cb90388c0 2023-10-05T19:28:36.144 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.138+0000 7f536bc04700 1 -- 172.21.15.167:0/187505431 <== osd.1 v2:172.21.15.167:6800/30035 21 ==== osd_op_reply(55 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f535c02f080 con 0x560cb90388c0 2023-10-05T19:28:36.144 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.138+0000 7f536bc04700 1 -- 172.21.15.167:0/187505431 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:56 5.6 5:60000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x560cb903d0e0 con 0x560cb9008e00 2023-10-05T19:28:36.145 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.138+0000 7f536ac02700 1 -- 172.21.15.167:0/187505431 <== osd.2 v2:172.21.15.167:6816/30037 9 ==== osd_op_reply(56 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f5354012450 con 0x560cb9008e00 2023-10-05T19:28:36.145 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.138+0000 7f536ac02700 1 -- 172.21.15.167:0/187505431 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:57 5.16 5:68000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x560cb903d0e0 con 0x560cb903e740 2023-10-05T19:28:36.146 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.142+0000 7f536b403700 1 -- 172.21.15.167:0/187505431 <== osd.3 v2:172.21.15.167:6808/30036 13 ==== osd_op_reply(57 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f536403d040 con 0x560cb903e740 2023-10-05T19:28:36.146 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.142+0000 7f536b403700 1 -- 172.21.15.167:0/187505431 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:58 5.e 5:70000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x560cb903d0e0 con 0x560cb903e740 2023-10-05T19:28:36.147 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.142+0000 7f536b403700 1 -- 172.21.15.167:0/187505431 <== osd.3 v2:172.21.15.167:6808/30036 14 ==== osd_op_reply(58 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f536403d040 con 0x560cb903e740 2023-10-05T19:28:36.147 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.142+0000 7f536b403700 1 -- 172.21.15.167:0/187505431 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:59 5.1e 5:78000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x560cb903d0e0 con 0x7f5354007030 2023-10-05T19:28:36.148 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.142+0000 7f536bc04700 1 -- 172.21.15.167:0/187505431 <== osd.0 v2:172.21.15.118:6800/27690 15 ==== osd_op_reply(59 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f535c02f080 con 0x7f5354007030 2023-10-05T19:28:36.148 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.142+0000 7f536bc04700 1 -- 172.21.15.167:0/187505431 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:60 5.1 5:80000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x560cb903d0e0 con 0x560cb903e740 2023-10-05T19:28:36.149 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.142+0000 7f536b403700 1 -- 172.21.15.167:0/187505431 <== osd.3 v2:172.21.15.167:6808/30036 15 ==== osd_op_reply(60 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f536403d040 con 0x560cb903e740 2023-10-05T19:28:36.149 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.142+0000 7f536b403700 1 -- 172.21.15.167:0/187505431 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:61 5.11 5:88000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x560cb903d0e0 con 0x560cb903e740 2023-10-05T19:28:36.150 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.142+0000 7f536b403700 1 -- 172.21.15.167:0/187505431 <== osd.3 v2:172.21.15.167:6808/30036 16 ==== osd_op_reply(61 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f536403d040 con 0x560cb903e740 2023-10-05T19:28:36.150 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.142+0000 7f536b403700 1 -- 172.21.15.167:0/187505431 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:62 5.9 5:90000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x560cb903d0e0 con 0x560cb90388c0 2023-10-05T19:28:36.150 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.146+0000 7f536bc04700 1 -- 172.21.15.167:0/187505431 <== osd.1 v2:172.21.15.167:6800/30035 22 ==== osd_op_reply(62 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f535c02f080 con 0x560cb90388c0 2023-10-05T19:28:36.150 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.146+0000 7f536bc04700 1 -- 172.21.15.167:0/187505431 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:63 5.19 5:98000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x560cb903d0e0 con 0x560cb90388c0 2023-10-05T19:28:36.151 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.146+0000 7f536bc04700 1 -- 172.21.15.167:0/187505431 <== osd.1 v2:172.21.15.167:6800/30035 23 ==== osd_op_reply(63 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f535c02f080 con 0x560cb90388c0 2023-10-05T19:28:36.151 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.146+0000 7f536bc04700 1 -- 172.21.15.167:0/187505431 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:64 5.5 5:a0000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x560cb903d0e0 con 0x7f5354007030 2023-10-05T19:28:36.152 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.146+0000 7f536bc04700 1 -- 172.21.15.167:0/187505431 <== osd.0 v2:172.21.15.118:6800/27690 16 ==== osd_op_reply(64 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f535c02f080 con 0x7f5354007030 2023-10-05T19:28:36.152 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.146+0000 7f536bc04700 1 -- 172.21.15.167:0/187505431 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:65 5.15 5:a8000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x560cb903d0e0 con 0x560cb903e740 2023-10-05T19:28:36.153 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.146+0000 7f536b403700 1 -- 172.21.15.167:0/187505431 <== osd.3 v2:172.21.15.167:6808/30036 17 ==== osd_op_reply(65 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f536403d040 con 0x560cb903e740 2023-10-05T19:28:36.153 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.146+0000 7f536b403700 1 -- 172.21.15.167:0/187505431 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:66 5.d 5:b0000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x560cb903d0e0 con 0x560cb9008e00 2023-10-05T19:28:36.154 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.150+0000 7f536ac02700 1 -- 172.21.15.167:0/187505431 <== osd.2 v2:172.21.15.167:6816/30037 10 ==== osd_op_reply(66 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f5354012450 con 0x560cb9008e00 2023-10-05T19:28:36.154 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.150+0000 7f536ac02700 1 -- 172.21.15.167:0/187505431 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:67 5.1d 5:b8000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x560cb903d0e0 con 0x560cb90388c0 2023-10-05T19:28:36.155 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.150+0000 7f536bc04700 1 -- 172.21.15.167:0/187505431 <== osd.1 v2:172.21.15.167:6800/30035 24 ==== osd_op_reply(67 [pgnls start_epoch 30 out=107b] v30'2 uv2 ondisk = 1) v8 ==== 144+0+107 (crc 0 0 0) 0x7f535c02f080 con 0x560cb90388c0 2023-10-05T19:28:36.155 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.150+0000 7f536bc04700 1 -- 172.21.15.167:0/187505431 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:68 5.3 5:c0000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x560cb903d0e0 con 0x7f5354007030 2023-10-05T19:28:36.156 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.150+0000 7f536bc04700 1 -- 172.21.15.167:0/187505431 <== osd.0 v2:172.21.15.118:6800/27690 17 ==== osd_op_reply(68 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f535c02f080 con 0x7f5354007030 2023-10-05T19:28:36.156 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.150+0000 7f536bc04700 1 -- 172.21.15.167:0/187505431 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:69 5.13 5:c8000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x560cb903d0e0 con 0x560cb903e740 2023-10-05T19:28:36.157 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.150+0000 7f536b403700 1 -- 172.21.15.167:0/187505431 <== osd.3 v2:172.21.15.167:6808/30036 18 ==== osd_op_reply(69 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f536403d040 con 0x560cb903e740 2023-10-05T19:28:36.157 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.150+0000 7f536b403700 1 -- 172.21.15.167:0/187505431 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:70 5.b 5:d0000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x560cb903d0e0 con 0x7f5354007030 2023-10-05T19:28:36.158 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.154+0000 7f536bc04700 1 -- 172.21.15.167:0/187505431 <== osd.0 v2:172.21.15.118:6800/27690 18 ==== osd_op_reply(70 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f535c02f080 con 0x7f5354007030 2023-10-05T19:28:36.158 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.154+0000 7f536bc04700 1 -- 172.21.15.167:0/187505431 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:71 5.1b 5:d8000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x560cb903d0e0 con 0x560cb9008e00 2023-10-05T19:28:36.159 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.154+0000 7f536ac02700 1 -- 172.21.15.167:0/187505431 <== osd.2 v2:172.21.15.167:6816/30037 11 ==== osd_op_reply(71 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f5354012450 con 0x560cb9008e00 2023-10-05T19:28:36.159 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.154+0000 7f536ac02700 1 -- 172.21.15.167:0/187505431 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:72 5.7 5:e0000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x560cb903d0e0 con 0x560cb903e740 2023-10-05T19:28:36.160 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.154+0000 7f536b403700 1 -- 172.21.15.167:0/187505431 <== osd.3 v2:172.21.15.167:6808/30036 19 ==== osd_op_reply(72 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f536403d040 con 0x560cb903e740 2023-10-05T19:28:36.160 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.154+0000 7f536b403700 1 -- 172.21.15.167:0/187505431 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:73 5.17 5:e8000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x560cb903d0e0 con 0x7f5354007030 2023-10-05T19:28:36.161 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.154+0000 7f536bc04700 1 -- 172.21.15.167:0/187505431 <== osd.0 v2:172.21.15.118:6800/27690 19 ==== osd_op_reply(73 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f535c02f080 con 0x7f5354007030 2023-10-05T19:28:36.161 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.154+0000 7f536bc04700 1 -- 172.21.15.167:0/187505431 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:74 5.f 5:f0000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x560cb903d0e0 con 0x560cb90388c0 2023-10-05T19:28:36.162 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.158+0000 7f536bc04700 1 -- 172.21.15.167:0/187505431 <== osd.1 v2:172.21.15.167:6800/30035 25 ==== osd_op_reply(74 [pgnls start_epoch 30 out=115b] v23'2 uv2 ondisk = 1) v8 ==== 144+0+115 (crc 0 0 0) 0x7f535c02f080 con 0x560cb90388c0 2023-10-05T19:28:36.162 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.158+0000 7f536bc04700 1 -- 172.21.15.167:0/187505431 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:75 5.1f 5:f8000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x560cb903d0e0 con 0x560cb9008e00 2023-10-05T19:28:36.163 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.158+0000 7f536ac02700 1 -- 172.21.15.167:0/187505431 <== osd.2 v2:172.21.15.167:6816/30037 12 ==== osd_op_reply(75 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f5354012450 con 0x560cb9008e00 2023-10-05T19:28:36.163 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.158+0000 7f536d8cee40 20 RGWRados::pool_iterate: got zonegroup_info.4f1f719d-9648-417b-b10c-e5d939825419 2023-10-05T19:28:36.163 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.158+0000 7f536d8cee40 20 RGWRados::pool_iterate: got zone_info.9a9df965-6612-45d4-a45f-6e3c889bacc2 2023-10-05T19:28:36.163 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.158+0000 7f536d8cee40 20 RGWRados::pool_iterate: got zone_names.default 2023-10-05T19:28:36.163 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.158+0000 7f536d8cee40 20 RGWRados::pool_iterate: got zonegroups_names.default 2023-10-05T19:28:36.164 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.158+0000 7f536d8cee40 20 rados->read ofs=0 len=0 2023-10-05T19:28:36.164 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.158+0000 7f536d8cee40 1 -- 172.21.15.167:0/187505431 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:76 5.f 5:f43b6ece:::zone_names.default:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x560cb9041d00 con 0x560cb90388c0 2023-10-05T19:28:36.164 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.158+0000 7f536bc04700 1 -- 172.21.15.167:0/187505431 <== osd.1 v2:172.21.15.167:6800/30035 26 ==== osd_op_reply(76 zone_names.default [read 0~46 out=46b] v0'0 uv1 ondisk = 0) v8 ==== 162+0+46 (crc 0 0 0) 0x7f535c02f080 con 0x560cb90388c0 2023-10-05T19:28:36.164 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.158+0000 7f536d8cee40 20 rados_obj.operate() r=0 bl.length=46 2023-10-05T19:28:36.164 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.158+0000 7f536d8cee40 20 rados->read ofs=0 len=0 2023-10-05T19:28:36.164 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.158+0000 7f536d8cee40 1 -- 172.21.15.167:0/187505431 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:77 5.1d 5:b8374eb4:::zone_info.9a9df965-6612-45d4-a45f-6e3c889bacc2:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x560cb90420b0 con 0x560cb90388c0 2023-10-05T19:28:36.165 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.158+0000 7f536bc04700 1 -- 172.21.15.167:0/187505431 <== osd.1 v2:172.21.15.167:6800/30035 27 ==== osd_op_reply(77 zone_info.9a9df965-6612-45d4-a45f-6e3c889bacc2 [read 0~874 out=874b] v0'0 uv2 ondisk = 0) v8 ==== 190+0+874 (crc 0 0 0) 0x7f535c02f080 con 0x560cb90388c0 2023-10-05T19:28:36.165 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.158+0000 7f536d8cee40 20 rados_obj.operate() r=0 bl.length=874 2023-10-05T19:28:36.165 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.158+0000 7f536d8cee40 20 rados->read ofs=0 len=0 2023-10-05T19:28:36.165 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.158+0000 7f536d8cee40 1 -- 172.21.15.167:0/187505431 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:78 5.f 5:f4c53578:::zonegroups_names.default:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x560cb90414c0 con 0x560cb90388c0 2023-10-05T19:28:36.166 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.158+0000 7f536bc04700 1 -- 172.21.15.167:0/187505431 <== osd.1 v2:172.21.15.167:6800/30035 28 ==== osd_op_reply(78 zonegroups_names.default [read 0~46 out=46b] v0'0 uv2 ondisk = 0) v8 ==== 168+0+46 (crc 0 0 0) 0x7f535c02f080 con 0x560cb90388c0 2023-10-05T19:28:36.166 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.158+0000 7f536d8cee40 20 rados_obj.operate() r=0 bl.length=46 2023-10-05T19:28:36.166 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.158+0000 7f536d8cee40 20 rados->read ofs=0 len=0 2023-10-05T19:28:36.166 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.158+0000 7f536d8cee40 1 -- 172.21.15.167:0/187505431 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:79 5.c 5:30d299a5:::zonegroup_info.4f1f719d-9648-417b-b10c-e5d939825419:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x560cb90414e0 con 0x7f5354007030 2023-10-05T19:28:36.166 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.162+0000 7f536bc04700 1 -- 172.21.15.167:0/187505431 <== osd.0 v2:172.21.15.118:6800/27690 20 ==== osd_op_reply(79 zonegroup_info.4f1f719d-9648-417b-b10c-e5d939825419 [read 0~362 out=362b] v0'0 uv1 ondisk = 0) v8 ==== 195+0+362 (crc 0 0 0) 0x7f535c02f080 con 0x7f5354007030 2023-10-05T19:28:36.167 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.162+0000 7f536d8cee40 20 rados_obj.operate() r=0 bl.length=362 2023-10-05T19:28:36.167 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.162+0000 7f536d8cee40 20 rados->read ofs=0 len=0 2023-10-05T19:28:36.167 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.162+0000 7f536d8cee40 1 -- 172.21.15.167:0/187505431 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:80 5.3 5:c0ad8cbd:::region_map:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x560cb9042b20 con 0x7f5354007030 2023-10-05T19:28:36.167 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.162+0000 7f536bc04700 1 -- 172.21.15.167:0/187505431 <== osd.0 v2:172.21.15.118:6800/27690 21 ==== osd_op_reply(80 region_map [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 ==== 154+0+0 (crc 0 0 0) 0x7f535c050040 con 0x7f5354007030 2023-10-05T19:28:36.167 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.162+0000 7f536d8cee40 20 rados_obj.operate() r=-2 bl.length=0 2023-10-05T19:28:36.168 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.162+0000 7f536d8cee40 20 zone default found 2023-10-05T19:28:36.168 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.162+0000 7f536d8cee40 4 Realm: () 2023-10-05T19:28:36.168 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.162+0000 7f536d8cee40 4 ZoneGroup: default (4f1f719d-9648-417b-b10c-e5d939825419) 2023-10-05T19:28:36.168 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.162+0000 7f536d8cee40 4 Zone: default (9a9df965-6612-45d4-a45f-6e3c889bacc2) 2023-10-05T19:28:36.168 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.162+0000 7f536d8cee40 10 cannot find current period zonegroup using local zonegroup configuration 2023-10-05T19:28:36.168 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.162+0000 7f536d8cee40 20 zonegroup default 2023-10-05T19:28:36.168 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.162+0000 7f536d8cee40 20 rados->read ofs=0 len=0 2023-10-05T19:28:36.168 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.162+0000 7f536d8cee40 1 -- 172.21.15.167:0/187505431 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:81 5.3 5:c52100b6:::period_config.default:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x560cb9042b20 con 0x7f5354007030 2023-10-05T19:28:36.169 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.162+0000 7f536bc04700 1 -- 172.21.15.167:0/187505431 <== osd.0 v2:172.21.15.118:6800/27690 22 ==== osd_op_reply(81 period_config.default [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 ==== 165+0+0 (crc 0 0 0) 0x7f535c050040 con 0x7f5354007030 2023-10-05T19:28:36.169 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.162+0000 7f536d8cee40 20 rados_obj.operate() r=-2 bl.length=0 2023-10-05T19:28:36.169 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.162+0000 7f536d8cee40 20 rados->read ofs=0 len=0 2023-10-05T19:28:36.169 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.162+0000 7f536d8cee40 1 -- 172.21.15.167:0/187505431 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:82 6.2 6:4347d321:::bucket.sync-source-hints.:head [call version.read in=11b,read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x560cb9044170 con 0x560cb903e740 2023-10-05T19:28:36.169 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.162+0000 7f536b403700 1 -- 172.21.15.167:0/187505431 <== osd.3 v2:172.21.15.167:6808/30036 20 ==== osd_op_reply(82 bucket.sync-source-hints. [call,read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 ==== 211+0+0 (crc 0 0 0) 0x7f536403d040 con 0x560cb903e740 2023-10-05T19:28:36.170 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.162+0000 7f536d8cee40 20 rados_obj.operate() r=-2 bl.length=0 2023-10-05T19:28:36.170 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.162+0000 7f536d8cee40 20 rados->read ofs=0 len=0 2023-10-05T19:28:36.170 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.162+0000 7f536d8cee40 1 -- 172.21.15.167:0/187505431 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:83 6.b 6:d467b91b:::bucket.sync-target-hints.:head [call version.read in=11b,read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x560cb9044520 con 0x560cb903e740 2023-10-05T19:28:36.170 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.166+0000 7f536b403700 1 -- 172.21.15.167:0/187505431 <== osd.3 v2:172.21.15.167:6808/30036 21 ==== osd_op_reply(83 bucket.sync-target-hints. [call,read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 ==== 211+0+0 (crc 0 0 0) 0x7f536403d040 con 0x560cb903e740 2023-10-05T19:28:36.171 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.166+0000 7f536d8cee40 20 rados_obj.operate() r=-2 bl.length=0 2023-10-05T19:28:36.171 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.166+0000 7f536d8cee40 20 started sync module instance, tier type = 2023-10-05T19:28:36.171 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.166+0000 7f536d8cee40 20 started zone id=9a9df965-6612-45d4-a45f-6e3c889bacc2 (name=default) with tier type = 2023-10-05T19:28:36.171 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.166+0000 7f536d8cee40 1 -- 172.21.15.167:0/187505431 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:84 7.1f 7:f95f44c2:::notify.0:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x560cb9043830 con 0x560cb9008e00 2023-10-05T19:28:36.173 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.166+0000 7f536ac02700 1 -- 172.21.15.167:0/187505431 <== osd.2 v2:172.21.15.167:6816/30037 13 ==== osd_op_reply(84 notify.0 [create] v30'150 uv150 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f5354012450 con 0x560cb9008e00 2023-10-05T19:28:36.173 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.166+0000 7f536d8cee40 1 -- 172.21.15.167:0/187505431 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:85 7.1f 7:f95f44c2:::notify.0:head [watch watch cookie 94612643658800] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x560cb9043460 con 0x560cb9008e00 2023-10-05T19:28:36.173 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.166+0000 7f536d8cee40 1 -- 172.21.15.167:0/187505431 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:86 7.0 7:05bf5b68:::notify.1:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x560cb9045350 con 0x560cb90388c0 2023-10-05T19:28:36.176 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.170+0000 7f536bc04700 1 -- 172.21.15.167:0/187505431 <== osd.1 v2:172.21.15.167:6800/30035 29 ==== osd_op_reply(86 notify.1 [create] v30'150 uv150 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f535c050040 con 0x560cb90388c0 2023-10-05T19:28:36.176 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.170+0000 7f536d8cee40 1 -- 172.21.15.167:0/187505431 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:87 7.0 7:05bf5b68:::notify.1:head [watch watch cookie 94612643663392] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x560cb9045c40 con 0x560cb90388c0 2023-10-05T19:28:36.176 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.170+0000 7f536ac02700 1 -- 172.21.15.167:0/187505431 <== osd.2 v2:172.21.15.167:6816/30037 14 ==== osd_op_reply(85 notify.0 [watch watch cookie 94612643658800] v30'151 uv150 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f5354012450 con 0x560cb9008e00 2023-10-05T19:28:36.176 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.170+0000 7f536d8cee40 1 -- 172.21.15.167:0/187505431 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:88 7.15 7:a93a5511:::notify.2:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x560cb90464e0 con 0x560cb903e740 2023-10-05T19:28:36.179 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.174+0000 7f536bc04700 1 -- 172.21.15.167:0/187505431 <== osd.1 v2:172.21.15.167:6800/30035 30 ==== osd_op_reply(87 notify.1 [watch watch cookie 94612643663392] v30'151 uv150 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f535c050040 con 0x560cb90388c0 2023-10-05T19:28:36.180 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.174+0000 7f536b403700 1 -- 172.21.15.167:0/187505431 <== osd.3 v2:172.21.15.167:6808/30036 22 ==== osd_op_reply(88 notify.2 [create] v30'150 uv150 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f536403d040 con 0x560cb903e740 2023-10-05T19:28:36.180 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.174+0000 7f536d8cee40 1 -- 172.21.15.167:0/187505431 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:89 7.15 7:a93a5511:::notify.2:head [watch watch cookie 94612643665744] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x560cb90449f0 con 0x560cb903e740 2023-10-05T19:28:36.180 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.174+0000 7f536d8cee40 1 -- 172.21.15.167:0/187505431 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:90 7.e 7:7759931f:::notify.3:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x560cb9046d80 con 0x560cb9008e00 2023-10-05T19:28:36.183 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.178+0000 7f536ac02700 1 -- 172.21.15.167:0/187505431 <== osd.2 v2:172.21.15.167:6816/30037 15 ==== osd_op_reply(90 notify.3 [create] v30'150 uv150 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f5354012450 con 0x560cb9008e00 2023-10-05T19:28:36.183 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.178+0000 7f536d8cee40 1 -- 172.21.15.167:0/187505431 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:91 7.e 7:7759931f:::notify.3:head [watch watch cookie 94612643670160] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x560cb9047620 con 0x560cb9008e00 2023-10-05T19:28:36.183 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.178+0000 7f536d8cee40 1 -- 172.21.15.167:0/187505431 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:92 7.d 7:b4812045:::notify.4:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x560cb9048160 con 0x560cb90388c0 2023-10-05T19:28:36.183 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.178+0000 7f536b403700 1 -- 172.21.15.167:0/187505431 <== osd.3 v2:172.21.15.167:6808/30036 23 ==== osd_op_reply(89 notify.2 [watch watch cookie 94612643665744] v30'151 uv150 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f536403d040 con 0x560cb903e740 2023-10-05T19:28:36.185 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.178+0000 7f536ac02700 1 -- 172.21.15.167:0/187505431 <== osd.2 v2:172.21.15.167:6816/30037 16 ==== osd_op_reply(91 notify.3 [watch watch cookie 94612643670160] v30'151 uv150 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f5354012450 con 0x560cb9008e00 2023-10-05T19:28:36.185 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.178+0000 7f536bc04700 1 -- 172.21.15.167:0/187505431 <== osd.1 v2:172.21.15.167:6800/30035 31 ==== osd_op_reply(92 notify.4 [create] v30'150 uv150 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f535c050040 con 0x560cb90388c0 2023-10-05T19:28:36.185 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.178+0000 7f536d8cee40 1 -- 172.21.15.167:0/187505431 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:93 7.d 7:b4812045:::notify.4:head [watch watch cookie 94612643672368] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x560cb9047c30 con 0x560cb90388c0 2023-10-05T19:28:36.185 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.178+0000 7f536d8cee40 1 -- 172.21.15.167:0/187505431 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:94 7.3 7:c609908c:::notify.5:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x560cb9048a50 con 0x7f5354007030 2023-10-05T19:28:36.187 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.182+0000 7f536bc04700 1 -- 172.21.15.167:0/187505431 <== osd.1 v2:172.21.15.167:6800/30035 32 ==== osd_op_reply(93 notify.4 [watch watch cookie 94612643672368] v30'151 uv150 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f535c050040 con 0x560cb90388c0 2023-10-05T19:28:36.188 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.182+0000 7f536bc04700 1 -- 172.21.15.167:0/187505431 <== osd.0 v2:172.21.15.118:6800/27690 23 ==== osd_op_reply(94 notify.5 [create] v30'150 uv150 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f535c050040 con 0x7f5354007030 2023-10-05T19:28:36.188 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.182+0000 7f536d8cee40 1 -- 172.21.15.167:0/187505431 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:95 7.3 7:c609908c:::notify.5:head [watch watch cookie 94612643666848] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x560cb9048a00 con 0x7f5354007030 2023-10-05T19:28:36.188 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.182+0000 7f536d8cee40 1 -- 172.21.15.167:0/187505431 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:96 7.14 7:2b04a3e9:::notify.6:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x560cb9049410 con 0x7f5354007030 2023-10-05T19:28:36.190 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.186+0000 7f536bc04700 1 -- 172.21.15.167:0/187505431 <== osd.0 v2:172.21.15.118:6800/27690 24 ==== osd_op_reply(95 notify.5 [watch watch cookie 94612643666848] v30'151 uv150 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f535c050040 con 0x7f5354007030 2023-10-05T19:28:36.191 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.186+0000 7f536bc04700 1 -- 172.21.15.167:0/187505431 <== osd.0 v2:172.21.15.118:6800/27690 25 ==== osd_op_reply(96 notify.6 [create] v30'150 uv150 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f535c050040 con 0x7f5354007030 2023-10-05T19:28:36.191 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.186+0000 7f536d8cee40 1 -- 172.21.15.167:0/187505431 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:97 7.14 7:2b04a3e9:::notify.6:head [watch watch cookie 94612643677456] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x560cb9048960 con 0x7f5354007030 2023-10-05T19:28:36.191 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.186+0000 7f536d8cee40 1 -- 172.21.15.167:0/187505431 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:98 7.9 7:93e5b521:::notify.7:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x560cb9049da0 con 0x7f5354007030 2023-10-05T19:28:36.193 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.186+0000 7f536bc04700 1 -- 172.21.15.167:0/187505431 <== osd.0 v2:172.21.15.118:6800/27690 26 ==== osd_op_reply(97 notify.6 [watch watch cookie 94612643677456] v30'151 uv150 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f535c050040 con 0x7f5354007030 2023-10-05T19:28:36.194 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.190+0000 7f536bc04700 1 -- 172.21.15.167:0/187505431 <== osd.0 v2:172.21.15.118:6800/27690 27 ==== osd_op_reply(98 notify.7 [create] v30'150 uv150 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f535c050040 con 0x7f5354007030 2023-10-05T19:28:36.194 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.190+0000 7f536d8cee40 1 -- 172.21.15.167:0/187505431 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:99 7.9 7:93e5b521:::notify.7:head [watch watch cookie 94612643680592] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x560cb9049d60 con 0x7f5354007030 2023-10-05T19:28:36.194 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.190+0000 7f536d8cee40 20 add_watcher() i=0 2023-10-05T19:28:36.194 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.190+0000 7f536d8cee40 20 add_watcher() i=1 2023-10-05T19:28:36.195 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.190+0000 7f536d8cee40 20 add_watcher() i=2 2023-10-05T19:28:36.195 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.190+0000 7f536d8cee40 20 add_watcher() i=3 2023-10-05T19:28:36.195 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.190+0000 7f536d8cee40 20 add_watcher() i=4 2023-10-05T19:28:36.195 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.190+0000 7f536d8cee40 20 add_watcher() i=5 2023-10-05T19:28:36.195 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.190+0000 7f536d8cee40 20 add_watcher() i=6 2023-10-05T19:28:36.197 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.190+0000 7f536bc04700 1 -- 172.21.15.167:0/187505431 <== osd.0 v2:172.21.15.118:6800/27690 28 ==== osd_op_reply(99 notify.7 [watch watch cookie 94612643680592] v30'151 uv150 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f535c050040 con 0x7f5354007030 2023-10-05T19:28:36.197 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.190+0000 7f536d8cee40 20 add_watcher() i=7 2023-10-05T19:28:36.197 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.190+0000 7f536d8cee40 2 all 8 watchers are set, enabling cache 2023-10-05T19:28:36.197 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.190+0000 7f536d8cee40 1 -- 172.21.15.167:0/187505431 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:100 6.4 6:22d26bf9:::data_loggenerations_metadata:head [call version.check_conds in=74b,call version.read in=11b,read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x560cb904cfe0 con 0x560cb90388c0 2023-10-05T19:28:36.197 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.190+0000 7f536bc04700 1 -- 172.21.15.167:0/187505431 <== osd.1 v2:172.21.15.167:6800/30035 33 ==== osd_op_reply(100 data_loggenerations_metadata [call,call out=48b,read 0~28 out=28b] v0'0 uv1 ondisk = 0) v8 ==== 256+0+76 (crc 0 0 0) 0x7f535c050040 con 0x560cb90388c0 2023-10-05T19:28:36.198 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.190+0000 7f536d8cee40 1 -- 172.21.15.167:0/187505431 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:101 6.4 6:22d26bf9:::data_loggenerations_metadata:head [watch watch cookie 94612643682784] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x560cb904a320 con 0x560cb90388c0 2023-10-05T19:28:36.200 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.194+0000 7f536bc04700 1 -- 172.21.15.167:0/187505431 <== osd.1 v2:172.21.15.167:6800/30035 34 ==== osd_op_reply(101 data_loggenerations_metadata [watch watch cookie 94612643682784] v30'147 uv1 ondisk = 0) v8 ==== 172+0+0 (crc 0 0 0) 0x7f535c050040 con 0x560cb90388c0 2023-10-05T19:28:36.200 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.194+0000 7f533cfe1700 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2023-10-05T19:28:36.200 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.194+0000 7f536d8cee40 20 check_secure_mon_conn(): auth registy supported: methods=[2] modes=[2,1] 2023-10-05T19:28:36.200 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.194+0000 7f536d8cee40 20 check_secure_mon_conn(): mode 1 is insecure 2023-10-05T19:28:36.201 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.194+0000 7f536d8cee40 1 Processor -- start 2023-10-05T19:28:36.201 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.194+0000 7f536d8cee40 1 -- start start 2023-10-05T19:28:36.201 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.194+0000 7f536d8cee40 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x560cb90574c0 0x560cb9057890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-05T19:28:36.201 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.194+0000 7f536d8cee40 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x560cb9057dd0 con 0x560cb90574c0 2023-10-05T19:28:36.201 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.194+0000 7f536ac02700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x560cb90574c0 0x560cb9057890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-05T19:28:36.201 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.194+0000 7f536ac02700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x560cb90574c0 0x560cb9057890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.118:3300/0 says I am v2:172.21.15.167:41758/0 (socket says 172.21.15.167:41758) 2023-10-05T19:28:36.201 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.194+0000 7f536ac02700 1 -- 172.21.15.167:0/1457589456 learned_addr learned my addr 172.21.15.167:0/1457589456 (peer_addr_for_me v2:172.21.15.167:0/0) 2023-10-05T19:28:36.202 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.194+0000 7f536ac02700 1 -- 172.21.15.167:0/1457589456 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x560cb9057f10 con 0x560cb90574c0 2023-10-05T19:28:36.202 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.198+0000 7f536ac02700 1 --2- 172.21.15.167:0/1457589456 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x560cb90574c0 0x560cb9057890 secure :-1 s=READY pgs=438 cs=0 l=1 rev1=1 crypto rx=0x560cb9009e80 tx=0x7f5354012000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=53f65d8857153fae server_cookie=0 in_seq=0 out_seq=0 2023-10-05T19:28:36.203 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.198+0000 7f52b7fff700 1 -- 172.21.15.167:0/1457589456 <== mon.0 v2:172.21.15.118:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f535402a5c0 con 0x560cb90574c0 2023-10-05T19:28:36.203 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.198+0000 7f52b7fff700 1 -- 172.21.15.167:0/1457589456 <== mon.0 v2:172.21.15.118:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f535402a720 con 0x560cb90574c0 2023-10-05T19:28:36.203 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.198+0000 7f52b7fff700 1 -- 172.21.15.167:0/1457589456 <== mon.0 v2:172.21.15.118:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f535402a910 con 0x560cb90574c0 2023-10-05T19:28:36.204 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.198+0000 7f536d8cee40 1 -- 172.21.15.167:0/1457589456 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x560cb90574c0 msgr2=0x560cb9057890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-05T19:28:36.204 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.198+0000 7f536d8cee40 1 --2- 172.21.15.167:0/1457589456 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x560cb90574c0 0x560cb9057890 secure :-1 s=READY pgs=438 cs=0 l=1 rev1=1 crypto rx=0x560cb9009e80 tx=0x7f5354012000 comp rx=0 tx=0).stop 2023-10-05T19:28:36.204 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.198+0000 7f536d8cee40 1 -- 172.21.15.167:0/1457589456 shutdown_connections 2023-10-05T19:28:36.204 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.198+0000 7f536d8cee40 1 --2- 172.21.15.167:0/1457589456 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x560cb90574c0 0x560cb9057890 unknown :-1 s=CLOSED pgs=438 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-05T19:28:36.204 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.198+0000 7f536d8cee40 1 -- 172.21.15.167:0/1457589456 >> 172.21.15.167:0/1457589456 conn(0x560cb9052880 msgr2=0x560cb9050a30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-05T19:28:36.204 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.198+0000 7f536d8cee40 1 -- 172.21.15.167:0/1457589456 shutdown_connections 2023-10-05T19:28:36.204 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.198+0000 7f536d8cee40 1 -- 172.21.15.167:0/1457589456 wait complete. 2023-10-05T19:28:36.205 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.198+0000 7f536d8cee40 1 Processor -- start 2023-10-05T19:28:36.206 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.198+0000 7f536d8cee40 1 -- start start 2023-10-05T19:28:36.206 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.198+0000 7f536d8cee40 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x560cb90574c0 0x560cb905ced0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-05T19:28:36.206 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.198+0000 7f536d8cee40 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x560cb905ded0 con 0x560cb90574c0 2023-10-05T19:28:36.206 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.198+0000 7f536ac02700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x560cb90574c0 0x560cb905ced0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-05T19:28:36.206 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.198+0000 7f536ac02700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x560cb90574c0 0x560cb905ced0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.118:3300/0 says I am v2:172.21.15.167:41766/0 (socket says 172.21.15.167:41766) 2023-10-05T19:28:36.206 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.198+0000 7f536ac02700 1 -- 172.21.15.167:0/2614141872 learned_addr learned my addr 172.21.15.167:0/2614141872 (peer_addr_for_me v2:172.21.15.167:0/0) 2023-10-05T19:28:36.206 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.198+0000 7f536ac02700 1 -- 172.21.15.167:0/2614141872 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x560cb9009a10 con 0x560cb90574c0 2023-10-05T19:28:36.207 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.202+0000 7f536ac02700 1 --2- 172.21.15.167:0/2614141872 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x560cb90574c0 0x560cb905ced0 secure :-1 s=READY pgs=439 cs=0 l=1 rev1=1 crypto rx=0x560cb90515a0 tx=0x7f535402f000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-05T19:28:36.207 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.202+0000 7f52b67fc700 1 -- 172.21.15.167:0/2614141872 <== mon.0 v2:172.21.15.118:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f535403e950 con 0x560cb90574c0 2023-10-05T19:28:36.207 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.202+0000 7f52b67fc700 1 -- 172.21.15.167:0/2614141872 <== mon.0 v2:172.21.15.118:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f535403eab0 con 0x560cb90574c0 2023-10-05T19:28:36.207 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.202+0000 7f536d8cee40 1 -- 172.21.15.167:0/2614141872 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x560cb905d4d0 con 0x560cb90574c0 2023-10-05T19:28:36.208 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.202+0000 7f52b67fc700 1 -- 172.21.15.167:0/2614141872 <== mon.0 v2:172.21.15.118:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f535403ec80 con 0x560cb90574c0 2023-10-05T19:28:36.208 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.202+0000 7f536d8cee40 1 -- 172.21.15.167:0/2614141872 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x560cb905d910 con 0x560cb90574c0 2023-10-05T19:28:36.208 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.202+0000 7f52b67fc700 1 -- 172.21.15.167:0/2614141872 <== mon.0 v2:172.21.15.118:3300/0 4 ==== mgrmap(e 5) v1 ==== 40329+0+0 (secure 0 0 0) 0x7f5354046070 con 0x560cb90574c0 2023-10-05T19:28:36.208 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.202+0000 7f52b67fc700 1 --2- 172.21.15.167:0/2614141872 >> [v2:172.21.15.167:6824/29889,v1:172.21.15.167:6825/29889] conn(0x7f52ac030c50 0x7f52ac033100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-05T19:28:36.208 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.202+0000 7f52b67fc700 1 -- 172.21.15.167:0/2614141872 <== mon.0 v2:172.21.15.118:3300/0 5 ==== osd_map(30..30 src has 1..30) v4 ==== 6057+0+0 (secure 0 0 0) 0x7f535402f070 con 0x560cb90574c0 2023-10-05T19:28:36.209 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.202+0000 7f536d8cee40 5 note: GC not initialized 2023-10-05T19:28:36.209 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.202+0000 7f536bc04700 1 --2- 172.21.15.167:0/2614141872 >> [v2:172.21.15.167:6824/29889,v1:172.21.15.167:6825/29889] conn(0x7f52ac030c50 0x7f52ac033100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-05T19:28:36.209 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.202+0000 7f536bc04700 1 --2- 172.21.15.167:0/2614141872 >> [v2:172.21.15.167:6824/29889,v1:172.21.15.167:6825/29889] conn(0x7f52ac030c50 0x7f52ac033100 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x560cb904d0a0 tx=0x7f535c000f90 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-05T19:28:36.211 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.206+0000 7f52ab7fe700 20 reqs_thread_entry: start 2023-10-05T19:28:36.211 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.206+0000 7f52a97fa700 20 reqs_thread_entry: start 2023-10-05T19:28:36.211 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.206+0000 7f536d8cee40 20 init_complete bucket index max shards: 11 2023-10-05T19:28:36.211 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.206+0000 7f536d8cee40 10 rgw notify: Started notification manager with: 1 workers 2023-10-05T19:28:36.211 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.206+0000 7f52977fe700 20 rgw notify: INFO: next queues processing will happen at: Thu Oct 5 19:29:06 2023 2023-10-05T19:28:36.211 INFO:teuthology.orchestra.run.smithi167.stderr: 2023-10-05T19:28:36.212 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.206+0000 7f5296ffd700 20 reqs_thread_entry: start 2023-10-05T19:28:36.212 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.206+0000 7f536d8cee40 1 --2- 172.21.15.167:0/2614141872 >> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] conn(0x560cb90c5410 0x560cb90c77f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-05T19:28:36.212 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.206+0000 7f536d8cee40 1 -- 172.21.15.167:0/2614141872 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:1 6.0 6:00000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x560cb90c7d30 con 0x560cb90c5410 2023-10-05T19:28:36.212 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.206+0000 7f536b403700 1 --2- 172.21.15.167:0/2614141872 >> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] conn(0x560cb90c5410 0x560cb90c77f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-05T19:28:36.212 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.206+0000 7f536b403700 1 --2- 172.21.15.167:0/2614141872 >> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] conn(0x560cb90c5410 0x560cb90c77f0 crc :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-05T19:28:36.215 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.210+0000 7f536b403700 1 -- 172.21.15.167:0/2614141872 <== osd.0 v2:172.21.15.118:6800/27690 1 ==== osd_op_reply(1 [pgnls start_epoch 30 out=254b] v30'34 uv18 ondisk = 1) v8 ==== 144+0+254 (crc 0 0 0) 0x7f536403d040 con 0x560cb90c5410 2023-10-05T19:28:36.215 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.210+0000 7f536b403700 1 -- 172.21.15.167:0/2614141872 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:2 6.10 6:08000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x560cb90c7d30 con 0x560cb90c5410 2023-10-05T19:28:36.216 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.210+0000 7f536b403700 1 -- 172.21.15.167:0/2614141872 <== osd.0 v2:172.21.15.118:6800/27690 2 ==== osd_op_reply(2 [pgnls start_epoch 30 out=131b] v30'4 uv4 ondisk = 1) v8 ==== 144+0+131 (crc 0 0 0) 0x7f536403d040 con 0x560cb90c5410 2023-10-05T19:28:36.217 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.210+0000 7f536b403700 1 --2- 172.21.15.167:0/2614141872 >> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] conn(0x7f536403ebd0 0x7f53640123c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-05T19:28:36.217 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.210+0000 7f536b403700 1 -- 172.21.15.167:0/2614141872 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:3 6.8 6:10000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x560cb90c7d30 con 0x7f536403ebd0 2023-10-05T19:28:36.217 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.210+0000 7f536ac02700 1 --2- 172.21.15.167:0/2614141872 >> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] conn(0x7f536403ebd0 0x7f53640123c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-05T19:28:36.217 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.210+0000 7f536ac02700 1 --2- 172.21.15.167:0/2614141872 >> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] conn(0x7f536403ebd0 0x7f53640123c0 crc :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-05T19:28:36.218 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.214+0000 7f536ac02700 1 -- 172.21.15.167:0/2614141872 <== osd.3 v2:172.21.15.167:6808/30036 1 ==== osd_op_reply(3 [pgnls start_epoch 30 out=322b] v30'28 uv24 ondisk = 1) v8 ==== 144+0+322 (crc 0 0 0) 0x7f5354032040 con 0x7f536403ebd0 2023-10-05T19:28:36.218 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.214+0000 7f536ac02700 1 -- 172.21.15.167:0/2614141872 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:4 6.18 6:18000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x7f5354048070 con 0x560cb90c5410 2023-10-05T19:28:36.219 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.214+0000 7f536b403700 1 -- 172.21.15.167:0/2614141872 <== osd.0 v2:172.21.15.118:6800/27690 3 ==== osd_op_reply(4 [pgnls start_epoch 30 out=248b] v30'20 uv16 ondisk = 1) v8 ==== 144+0+248 (crc 0 0 0) 0x7f536403d040 con 0x560cb90c5410 2023-10-05T19:28:36.220 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.214+0000 7f536b403700 1 --2- 172.21.15.167:0/2614141872 >> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] conn(0x7f536403d7a0 0x7f5364042750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-05T19:28:36.220 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.214+0000 7f536b403700 1 -- 172.21.15.167:0/2614141872 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:5 6.4 6:20000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x7f5354048070 con 0x7f536403d7a0 2023-10-05T19:28:36.220 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.214+0000 7f536bc04700 1 --2- 172.21.15.167:0/2614141872 >> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] conn(0x7f536403d7a0 0x7f5364042750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-05T19:28:36.220 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.214+0000 7f536bc04700 1 --2- 172.21.15.167:0/2614141872 >> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] conn(0x7f536403d7a0 0x7f5364042750 crc :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-05T19:28:36.221 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.214+0000 7f536bc04700 1 -- 172.21.15.167:0/2614141872 <== osd.1 v2:172.21.15.167:6800/30035 1 ==== osd_op_reply(5 [pgnls start_epoch 30 out=335b] v30'147 uv23 ondisk = 1) v8 ==== 144+0+335 (crc 0 0 0) 0x7f535c050040 con 0x7f536403d7a0 2023-10-05T19:28:36.221 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.214+0000 7f536bc04700 1 --2- 172.21.15.167:0/2614141872 >> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] conn(0x7f535c060040 0x7f535c054960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-05T19:28:36.221 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.214+0000 7f536bc04700 1 -- 172.21.15.167:0/2614141872 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:6 6.14 6:28000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x560cb9041d00 con 0x7f535c060040 2023-10-05T19:28:36.222 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.214+0000 7f536b403700 1 --2- 172.21.15.167:0/2614141872 >> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] conn(0x7f535c060040 0x7f535c054960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-05T19:28:36.222 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.218+0000 7f536b403700 1 --2- 172.21.15.167:0/2614141872 >> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] conn(0x7f535c060040 0x7f535c054960 crc :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-05T19:28:36.223 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.218+0000 7f536b403700 1 -- 172.21.15.167:0/2614141872 <== osd.2 v2:172.21.15.167:6816/30037 1 ==== osd_op_reply(6 [pgnls start_epoch 30 out=213b] v30'13 uv13 ondisk = 1) v8 ==== 144+0+213 (crc 0 0 0) 0x7f536403d040 con 0x7f535c060040 2023-10-05T19:28:36.223 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.218+0000 7f536b403700 1 -- 172.21.15.167:0/2614141872 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:7 6.c 6:30000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x560cb9041d00 con 0x7f536403ebd0 2023-10-05T19:28:36.224 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.218+0000 7f536ac02700 1 -- 172.21.15.167:0/2614141872 <== osd.3 v2:172.21.15.167:6808/30036 2 ==== osd_op_reply(7 [pgnls start_epoch 30 out=254b] v30'10 uv10 ondisk = 1) v8 ==== 144+0+254 (crc 0 0 0) 0x7f5354032040 con 0x7f536403ebd0 2023-10-05T19:28:36.224 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.218+0000 7f536ac02700 1 -- 172.21.15.167:0/2614141872 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:8 6.1c 6:38000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x7f5354048040 con 0x7f536403ebd0 2023-10-05T19:28:36.225 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.218+0000 7f536ac02700 1 -- 172.21.15.167:0/2614141872 <== osd.3 v2:172.21.15.167:6808/30036 3 ==== osd_op_reply(8 [pgnls start_epoch 30 out=370b] v30'31 uv30 ondisk = 1) v8 ==== 144+0+370 (crc 0 0 0) 0x7f5354058040 con 0x7f536403ebd0 2023-10-05T19:28:36.225 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.218+0000 7f536ac02700 1 -- 172.21.15.167:0/2614141872 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:9 6.2 6:40000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x7f5354048040 con 0x7f536403ebd0 2023-10-05T19:28:36.226 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.222+0000 7f536ac02700 1 -- 172.21.15.167:0/2614141872 <== osd.3 v2:172.21.15.167:6808/30036 4 ==== osd_op_reply(9 [pgnls start_epoch 30 out=213b] v30'10 uv10 ondisk = 1) v8 ==== 144+0+213 (crc 0 0 0) 0x7f5354058040 con 0x7f536403ebd0 2023-10-05T19:28:36.226 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.222+0000 7f536ac02700 1 -- 172.21.15.167:0/2614141872 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:10 6.12 6:48000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x7f5354048040 con 0x7f535c060040 2023-10-05T19:28:36.227 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.222+0000 7f536b403700 1 -- 172.21.15.167:0/2614141872 <== osd.2 v2:172.21.15.167:6816/30037 2 ==== osd_op_reply(10 [pgnls start_epoch 30 out=172b] v30'17 uv9 ondisk = 1) v8 ==== 144+0+172 (crc 0 0 0) 0x7f536403d040 con 0x7f535c060040 2023-10-05T19:28:36.227 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.222+0000 7f536b403700 1 -- 172.21.15.167:0/2614141872 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:11 6.a 6:50000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x7f5354048040 con 0x7f535c060040 2023-10-05T19:28:36.228 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.222+0000 7f536b403700 1 -- 172.21.15.167:0/2614141872 <== osd.2 v2:172.21.15.167:6816/30037 3 ==== osd_op_reply(11 [pgnls start_epoch 30 out=336b] v30'27 uv19 ondisk = 1) v8 ==== 144+0+336 (crc 0 0 0) 0x7f536403d040 con 0x7f535c060040 2023-10-05T19:28:36.228 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.222+0000 7f536b403700 1 -- 172.21.15.167:0/2614141872 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:12 6.1a 6:58000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x7f5354048040 con 0x7f536403ebd0 2023-10-05T19:28:36.229 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.222+0000 7f536ac02700 1 -- 172.21.15.167:0/2614141872 <== osd.3 v2:172.21.15.167:6808/30036 5 ==== osd_op_reply(12 [pgnls start_epoch 30 out=131b] v30'17 uv9 ondisk = 1) v8 ==== 144+0+131 (crc 0 0 0) 0x7f5354058040 con 0x7f536403ebd0 2023-10-05T19:28:36.229 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.222+0000 7f536ac02700 1 -- 172.21.15.167:0/2614141872 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:13 6.6 6:60000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x7f5354048040 con 0x7f536403ebd0 2023-10-05T19:28:36.230 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.226+0000 7f536ac02700 1 -- 172.21.15.167:0/2614141872 <== osd.3 v2:172.21.15.167:6808/30036 6 ==== osd_op_reply(13 [pgnls start_epoch 30 out=254b] v30'12 uv12 ondisk = 1) v8 ==== 144+0+254 (crc 0 0 0) 0x7f5354058040 con 0x7f536403ebd0 2023-10-05T19:28:36.230 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.226+0000 7f536ac02700 1 -- 172.21.15.167:0/2614141872 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:14 6.16 6:68000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x7f5354048040 con 0x7f536403ebd0 2023-10-05T19:28:36.231 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.226+0000 7f536ac02700 1 -- 172.21.15.167:0/2614141872 <== osd.3 v2:172.21.15.167:6808/30036 7 ==== osd_op_reply(14 [pgnls start_epoch 30 out=90b] v30'17 uv9 ondisk = 1) v8 ==== 144+0+90 (crc 0 0 0) 0x7f5354058040 con 0x7f536403ebd0 2023-10-05T19:28:36.231 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.226+0000 7f536ac02700 1 -- 172.21.15.167:0/2614141872 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:15 6.e 6:70000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x7f5354048040 con 0x7f536403d7a0 2023-10-05T19:28:36.232 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.226+0000 7f536bc04700 1 -- 172.21.15.167:0/2614141872 <== osd.1 v2:172.21.15.167:6800/30035 2 ==== osd_op_reply(15 [pgnls start_epoch 30 out=213b] v30'19 uv11 ondisk = 1) v8 ==== 144+0+213 (crc 0 0 0) 0x7f535c050040 con 0x7f536403d7a0 2023-10-05T19:28:36.233 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.226+0000 7f536bc04700 1 -- 172.21.15.167:0/2614141872 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:16 6.1e 6:78000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x7f5354048040 con 0x7f536403ebd0 2023-10-05T19:28:36.233 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.226+0000 7f536ac02700 1 -- 172.21.15.167:0/2614141872 <== osd.3 v2:172.21.15.167:6808/30036 8 ==== osd_op_reply(16 [pgnls start_epoch 30 out=254b] v30'10 uv10 ondisk = 1) v8 ==== 144+0+254 (crc 0 0 0) 0x7f5354058040 con 0x7f536403ebd0 2023-10-05T19:28:36.233 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.226+0000 7f536ac02700 1 -- 172.21.15.167:0/2614141872 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:17 6.1 6:80000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x7f5354048040 con 0x7f536403d7a0 2023-10-05T19:28:36.235 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.230+0000 7f536bc04700 1 -- 172.21.15.167:0/2614141872 <== osd.1 v2:172.21.15.167:6800/30035 3 ==== osd_op_reply(17 [pgnls start_epoch 30 out=254b] v30'14 uv14 ondisk = 1) v8 ==== 144+0+254 (crc 0 0 0) 0x7f535c050040 con 0x7f536403d7a0 2023-10-05T19:28:36.235 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.230+0000 7f536bc04700 1 -- 172.21.15.167:0/2614141872 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:18 6.11 6:88000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x7f5354048040 con 0x7f536403ebd0 2023-10-05T19:28:36.236 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.230+0000 7f536ac02700 1 -- 172.21.15.167:0/2614141872 <== osd.3 v2:172.21.15.167:6808/30036 9 ==== osd_op_reply(18 [pgnls start_epoch 30 out=213b] v30'19 uv11 ondisk = 1) v8 ==== 144+0+213 (crc 0 0 0) 0x7f5354063040 con 0x7f536403ebd0 2023-10-05T19:28:36.236 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.230+0000 7f536ac02700 1 -- 172.21.15.167:0/2614141872 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:19 6.9 6:90000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x7f5354048040 con 0x7f536403ebd0 2023-10-05T19:28:36.237 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.230+0000 7f536ac02700 1 -- 172.21.15.167:0/2614141872 <== osd.3 v2:172.21.15.167:6808/30036 10 ==== osd_op_reply(19 [pgnls start_epoch 30 out=248b] v30'30 uv22 ondisk = 1) v8 ==== 144+0+248 (crc 0 0 0) 0x7f5354063040 con 0x7f536403ebd0 2023-10-05T19:28:36.237 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.230+0000 7f536ac02700 1 -- 172.21.15.167:0/2614141872 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:20 6.19 6:98000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x7f5354048040 con 0x7f536403d7a0 2023-10-05T19:28:36.238 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.234+0000 7f536bc04700 1 -- 172.21.15.167:0/2614141872 <== osd.1 v2:172.21.15.167:6800/30035 4 ==== osd_op_reply(20 [pgnls start_epoch 30 out=371b] v30'28 uv24 ondisk = 1) v8 ==== 144+0+371 (crc 0 0 0) 0x7f535c050040 con 0x7f536403d7a0 2023-10-05T19:28:36.238 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.234+0000 7f536bc04700 1 -- 172.21.15.167:0/2614141872 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:21 6.5 6:a0000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x7f5354048040 con 0x7f535c060040 2023-10-05T19:28:36.239 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.234+0000 7f536b403700 1 -- 172.21.15.167:0/2614141872 <== osd.2 v2:172.21.15.167:6816/30037 4 ==== osd_op_reply(21 [pgnls start_epoch 30 out=276b] v30'13 uv13 ondisk = 1) v8 ==== 144+0+276 (crc 0 0 0) 0x7f536403d040 con 0x7f535c060040 2023-10-05T19:28:36.239 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.234+0000 7f536b403700 1 -- 172.21.15.167:0/2614141872 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:22 6.15 6:a8000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x7f5354048040 con 0x7f535c060040 2023-10-05T19:28:36.240 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.234+0000 7f536b403700 1 -- 172.21.15.167:0/2614141872 <== osd.2 v2:172.21.15.167:6816/30037 5 ==== osd_op_reply(22 [pgnls start_epoch 30 out=172b] v30'17 uv9 ondisk = 1) v8 ==== 144+0+172 (crc 0 0 0) 0x7f536403d040 con 0x7f535c060040 2023-10-05T19:28:36.241 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.234+0000 7f536b403700 1 -- 172.21.15.167:0/2614141872 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:23 6.d 6:b0000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x7f5354048040 con 0x7f536403d7a0 2023-10-05T19:28:36.242 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.234+0000 7f536bc04700 1 -- 172.21.15.167:0/2614141872 <== osd.1 v2:172.21.15.167:6800/30035 5 ==== osd_op_reply(23 [pgnls start_epoch 30 out=295b] v30'25 uv17 ondisk = 1) v8 ==== 144+0+295 (crc 0 0 0) 0x7f535c050040 con 0x7f536403d7a0 2023-10-05T19:28:36.242 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.234+0000 7f536bc04700 1 -- 172.21.15.167:0/2614141872 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:24 6.1d 6:b8000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x7f5354048040 con 0x560cb90c5410 2023-10-05T19:28:36.243 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.238+0000 7f536b403700 1 -- 172.21.15.167:0/2614141872 <== osd.0 v2:172.21.15.118:6800/27690 4 ==== osd_op_reply(24 [pgnls start_epoch 30 out=172b] v30'28 uv12 ondisk = 1) v8 ==== 144+0+172 (crc 0 0 0) 0x7f536403d040 con 0x560cb90c5410 2023-10-05T19:28:36.243 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.238+0000 7f536b403700 1 -- 172.21.15.167:0/2614141872 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:25 6.3 6:c0000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x7f5354048040 con 0x7f535c060040 2023-10-05T19:28:36.244 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.238+0000 7f536b403700 1 -- 172.21.15.167:0/2614141872 <== osd.2 v2:172.21.15.167:6816/30037 6 ==== osd_op_reply(25 [pgnls start_epoch 30 out=213b] v30'43 uv19 ondisk = 1) v8 ==== 144+0+213 (crc 0 0 0) 0x7f536403d040 con 0x7f535c060040 2023-10-05T19:28:36.244 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.238+0000 7f536b403700 1 -- 172.21.15.167:0/2614141872 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:26 6.13 6:c8000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x7f5354048040 con 0x560cb90c5410 2023-10-05T19:28:36.245 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.238+0000 7f536b403700 1 -- 172.21.15.167:0/2614141872 <== osd.0 v2:172.21.15.118:6800/27690 5 ==== osd_op_reply(26 [pgnls start_epoch 30 out=172b] v30'20 uv11 ondisk = 1) v8 ==== 144+0+172 (crc 0 0 0) 0x7f536403d040 con 0x560cb90c5410 2023-10-05T19:28:36.245 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.238+0000 7f536b403700 1 -- 172.21.15.167:0/2614141872 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:27 6.b 6:d0000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x7f5354048040 con 0x7f536403ebd0 2023-10-05T19:28:36.246 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.242+0000 7f536ac02700 1 -- 172.21.15.167:0/2614141872 <== osd.3 v2:172.21.15.167:6808/30036 11 ==== osd_op_reply(27 [pgnls start_epoch 30 out=172b] v30'17 uv9 ondisk = 1) v8 ==== 144+0+172 (crc 0 0 0) 0x7f5354063040 con 0x7f536403ebd0 2023-10-05T19:28:36.246 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.242+0000 7f536ac02700 1 -- 172.21.15.167:0/2614141872 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:28 6.1b 6:d8000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x7f5354048040 con 0x7f536403ebd0 2023-10-05T19:28:36.247 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.242+0000 7f536ac02700 1 -- 172.21.15.167:0/2614141872 <== osd.3 v2:172.21.15.167:6808/30036 12 ==== osd_op_reply(28 [pgnls start_epoch 30 out=90b] v30'13 uv5 ondisk = 1) v8 ==== 144+0+90 (crc 0 0 0) 0x7f5354063040 con 0x7f536403ebd0 2023-10-05T19:28:36.247 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.242+0000 7f536ac02700 1 -- 172.21.15.167:0/2614141872 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:29 6.7 6:e0000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x7f5354048040 con 0x7f536403ebd0 2023-10-05T19:28:36.248 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.242+0000 7f536ac02700 1 -- 172.21.15.167:0/2614141872 <== osd.3 v2:172.21.15.167:6808/30036 13 ==== osd_op_reply(29 [pgnls start_epoch 30 out=90b] v30'37 uv13 ondisk = 1) v8 ==== 144+0+90 (crc 0 0 0) 0x7f5354063040 con 0x7f536403ebd0 2023-10-05T19:28:36.248 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.242+0000 7f536ac02700 1 -- 172.21.15.167:0/2614141872 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:30 6.17 6:e8000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x7f5354048040 con 0x7f536403d7a0 2023-10-05T19:28:36.249 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.242+0000 7f536bc04700 1 -- 172.21.15.167:0/2614141872 <== osd.1 v2:172.21.15.167:6800/30035 6 ==== osd_op_reply(30 [pgnls start_epoch 30 out=131b] v30'6 uv6 ondisk = 1) v8 ==== 144+0+131 (crc 0 0 0) 0x7f535c050040 con 0x7f536403d7a0 2023-10-05T19:28:36.249 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.242+0000 7f536bc04700 1 -- 172.21.15.167:0/2614141872 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:31 6.f 6:f0000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x7f5354048040 con 0x7f535c060040 2023-10-05T19:28:36.250 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.242+0000 7f536b403700 1 -- 172.21.15.167:0/2614141872 <== osd.2 v2:172.21.15.167:6816/30037 7 ==== osd_op_reply(31 [pgnls start_epoch 30 out=295b] v30'23 uv15 ondisk = 1) v8 ==== 144+0+295 (crc 0 0 0) 0x7f536403d040 con 0x7f535c060040 2023-10-05T19:28:36.250 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.242+0000 7f536b403700 1 -- 172.21.15.167:0/2614141872 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:32 6.1f 6:f8000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x7f5354048040 con 0x560cb90c5410 2023-10-05T19:28:36.251 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.246+0000 7f536b403700 1 -- 172.21.15.167:0/2614141872 <== osd.0 v2:172.21.15.118:6800/27690 6 ==== osd_op_reply(32 [pgnls start_epoch 30 out=213b] v30'19 uv11 ondisk = 1) v8 ==== 144+0+213 (crc 0 0 0) 0x7f536403d040 con 0x560cb90c5410 2023-10-05T19:28:36.253 INFO:teuthology.orchestra.run.smithi167.stdout:[ 2023-10-05T19:28:36.254 INFO:teuthology.orchestra.run.smithi167.stdout: "obj_delete_at_hint.0000000078", 2023-10-05T19:28:36.254 INFO:teuthology.orchestra.run.smithi167.stdout: "obj_delete_at_hint.0000000070", 2023-10-05T19:28:36.254 INFO:teuthology.orchestra.run.smithi167.stdout: "obj_delete_at_hint.0000000104", 2023-10-05T19:28:36.254 INFO:teuthology.orchestra.run.smithi167.stdout: "obj_delete_at_hint.0000000026", 2023-10-05T19:28:36.254 INFO:teuthology.orchestra.run.smithi167.stdout: "obj_delete_at_hint.0000000028", 2023-10-05T19:28:36.254 INFO:teuthology.orchestra.run.smithi167.stdout: "obj_delete_at_hint.0000000040", 2023-10-05T19:28:36.254 INFO:teuthology.orchestra.run.smithi167.stdout: "obj_delete_at_hint.0000000015", 2023-10-05T19:28:36.254 INFO:teuthology.orchestra.run.smithi167.stdout: "obj_delete_at_hint.0000000069", 2023-10-05T19:28:36.254 INFO:teuthology.orchestra.run.smithi167.stdout: "obj_delete_at_hint.0000000095", 2023-10-05T19:28:36.254 INFO:teuthology.orchestra.run.smithi167.stdout: "obj_delete_at_hint.0000000003", 2023-10-05T19:28:36.254 INFO:teuthology.orchestra.run.smithi167.stdout: "obj_delete_at_hint.0000000047", 2023-10-05T19:28:36.255 INFO:teuthology.orchestra.run.smithi167.stdout: "obj_delete_at_hint.0000000035", 2023-10-05T19:28:36.255 INFO:teuthology.orchestra.run.smithi167.stdout: "obj_delete_at_hint.0000000037", 2023-10-05T19:28:36.255 INFO:teuthology.orchestra.run.smithi167.stdout: "2023-10-05-19--", 2023-10-05T19:28:36.255 INFO:teuthology.orchestra.run.smithi167.stdout: "obj_delete_at_hint.0000000024", 2023-10-05T19:28:36.255 INFO:teuthology.orchestra.run.smithi167.stdout: "2023-10-05-19-9a9df965-6612-45d4-a45f-6e3c889bacc2.4223.4-myfoo4", 2023-10-05T19:28:36.255 INFO:teuthology.orchestra.run.smithi167.stdout: "obj_delete_at_hint.0000000000", 2023-10-05T19:28:36.255 INFO:teuthology.orchestra.run.smithi167.stdout: "obj_delete_at_hint.0000000031", 2023-10-05T19:28:36.255 INFO:teuthology.orchestra.run.smithi167.stdout: "data_loggenerations_metadata", 2023-10-05T19:28:36.255 INFO:teuthology.orchestra.run.smithi167.stdout: "obj_delete_at_hint.0000000076", 2023-10-05T19:28:36.255 INFO:teuthology.orchestra.run.smithi167.stdout: "obj_delete_at_hint.0000000113", 2023-10-05T19:28:36.255 INFO:teuthology.orchestra.run.smithi167.stdout: "obj_delete_at_hint.0000000005", 2023-10-05T19:28:36.255 INFO:teuthology.orchestra.run.smithi167.stdout: "obj_delete_at_hint.0000000011", 2023-10-05T19:28:36.256 INFO:teuthology.orchestra.run.smithi167.stdout: "obj_delete_at_hint.0000000058", 2023-10-05T19:28:36.256 INFO:teuthology.orchestra.run.smithi167.stdout: "obj_delete_at_hint.0000000052", 2023-10-05T19:28:36.256 INFO:teuthology.orchestra.run.smithi167.stdout: "obj_delete_at_hint.0000000088", 2023-10-05T19:28:36.256 INFO:teuthology.orchestra.run.smithi167.stdout: "obj_delete_at_hint.0000000080", 2023-10-05T19:28:36.256 INFO:teuthology.orchestra.run.smithi167.stdout: "obj_delete_at_hint.0000000090", 2023-10-05T19:28:36.256 INFO:teuthology.orchestra.run.smithi167.stdout: "obj_delete_at_hint.0000000110", 2023-10-05T19:28:36.256 INFO:teuthology.orchestra.run.smithi167.stdout: "obj_delete_at_hint.0000000096", 2023-10-05T19:28:36.256 INFO:teuthology.orchestra.run.smithi167.stdout: "obj_delete_at_hint.0000000087", 2023-10-05T19:28:36.256 INFO:teuthology.orchestra.run.smithi167.stdout: "obj_delete_at_hint.0000000008", 2023-10-05T19:28:36.256 INFO:teuthology.orchestra.run.smithi167.stdout: "obj_delete_at_hint.0000000006", 2023-10-05T19:28:36.256 INFO:teuthology.orchestra.run.smithi167.stdout: "obj_delete_at_hint.0000000029", 2023-10-05T19:28:36.256 INFO:teuthology.orchestra.run.smithi167.stdout: "obj_delete_at_hint.0000000089", 2023-10-05T19:28:36.257 INFO:teuthology.orchestra.run.smithi167.stdout: "obj_delete_at_hint.0000000022", 2023-10-05T19:28:36.257 INFO:teuthology.orchestra.run.smithi167.stdout: "2023-10-05-19-9a9df965-6612-45d4-a45f-6e3c889bacc2.4223.1-myfoo", 2023-10-05T19:28:36.257 INFO:teuthology.orchestra.run.smithi167.stdout: "obj_delete_at_hint.0000000016", 2023-10-05T19:28:36.257 INFO:teuthology.orchestra.run.smithi167.stdout: "obj_delete_at_hint.0000000041", 2023-10-05T19:28:36.257 INFO:teuthology.orchestra.run.smithi167.stdout: "obj_delete_at_hint.0000000018", 2023-10-05T19:28:36.257 INFO:teuthology.orchestra.run.smithi167.stdout: "obj_delete_at_hint.0000000092", 2023-10-05T19:28:36.257 INFO:teuthology.orchestra.run.smithi167.stdout: "obj_delete_at_hint.0000000014", 2023-10-05T19:28:36.257 INFO:teuthology.orchestra.run.smithi167.stdout: "obj_delete_at_hint.0000000112", 2023-10-05T19:28:36.257 INFO:teuthology.orchestra.run.smithi167.stdout: "obj_delete_at_hint.0000000007", 2023-10-05T19:28:36.257 INFO:teuthology.orchestra.run.smithi167.stdout: "obj_delete_at_hint.0000000021", 2023-10-05T19:28:36.257 INFO:teuthology.orchestra.run.smithi167.stdout: "obj_delete_at_hint.0000000064", 2023-10-05T19:28:36.257 INFO:teuthology.orchestra.run.smithi167.stdout: "obj_delete_at_hint.0000000071", 2023-10-05T19:28:36.258 INFO:teuthology.orchestra.run.smithi167.stdout: "obj_delete_at_hint.0000000074", 2023-10-05T19:28:36.258 INFO:teuthology.orchestra.run.smithi167.stdout: "obj_delete_at_hint.0000000081", 2023-10-05T19:28:36.258 INFO:teuthology.orchestra.run.smithi167.stdout: "obj_delete_at_hint.0000000009", 2023-10-05T19:28:36.258 INFO:teuthology.orchestra.run.smithi167.stdout: "obj_delete_at_hint.0000000121", 2023-10-05T19:28:36.258 INFO:teuthology.orchestra.run.smithi167.stdout: "obj_delete_at_hint.0000000125", 2023-10-05T19:28:36.258 INFO:teuthology.orchestra.run.smithi167.stdout: "obj_delete_at_hint.0000000082", 2023-10-05T19:28:36.258 INFO:teuthology.orchestra.run.smithi167.stdout: "obj_delete_at_hint.0000000105", 2023-10-05T19:28:36.258 INFO:teuthology.orchestra.run.smithi167.stdout: "obj_delete_at_hint.0000000059", 2023-10-05T19:28:36.258 INFO:teuthology.orchestra.run.smithi167.stdout: "obj_delete_at_hint.0000000077", 2023-10-05T19:28:36.258 INFO:teuthology.orchestra.run.smithi167.stdout: "obj_delete_at_hint.0000000032", 2023-10-05T19:28:36.258 INFO:teuthology.orchestra.run.smithi167.stdout: "obj_delete_at_hint.0000000053", 2023-10-05T19:28:36.258 INFO:teuthology.orchestra.run.smithi167.stdout: "obj_delete_at_hint.0000000091", 2023-10-05T19:28:36.259 INFO:teuthology.orchestra.run.smithi167.stdout: "obj_delete_at_hint.0000000065", 2023-10-05T19:28:36.259 INFO:teuthology.orchestra.run.smithi167.stdout: "obj_delete_at_hint.0000000083", 2023-10-05T19:28:36.259 INFO:teuthology.orchestra.run.smithi167.stdout: "obj_delete_at_hint.0000000010", 2023-10-05T19:28:36.259 INFO:teuthology.orchestra.run.smithi167.stdout: "obj_delete_at_hint.0000000045", 2023-10-05T19:28:36.259 INFO:teuthology.orchestra.run.smithi167.stdout: "obj_delete_at_hint.0000000002", 2023-10-05T19:28:36.259 INFO:teuthology.orchestra.run.smithi167.stdout: "obj_delete_at_hint.0000000116", 2023-10-05T19:28:36.259 INFO:teuthology.orchestra.run.smithi167.stdout: "obj_delete_at_hint.0000000034", 2023-10-05T19:28:36.259 INFO:teuthology.orchestra.run.smithi167.stdout: "obj_delete_at_hint.0000000101", 2023-10-05T19:28:36.259 INFO:teuthology.orchestra.run.smithi167.stdout: "obj_delete_at_hint.0000000079", 2023-10-05T19:28:36.259 INFO:teuthology.orchestra.run.smithi167.stdout: "obj_delete_at_hint.0000000049", 2023-10-05T19:28:36.259 INFO:teuthology.orchestra.run.smithi167.stdout: "obj_delete_at_hint.0000000117", 2023-10-05T19:28:36.259 INFO:teuthology.orchestra.run.smithi167.stdout: "obj_delete_at_hint.0000000044", 2023-10-05T19:28:36.259 INFO:teuthology.orchestra.run.smithi167.stdout: "obj_delete_at_hint.0000000066", 2023-10-05T19:28:36.260 INFO:teuthology.orchestra.run.smithi167.stdout: "obj_delete_at_hint.0000000068", 2023-10-05T19:28:36.260 INFO:teuthology.orchestra.run.smithi167.stdout: "obj_delete_at_hint.0000000085", 2023-10-05T19:28:36.260 INFO:teuthology.orchestra.run.smithi167.stdout: "obj_delete_at_hint.0000000073", 2023-10-05T19:28:36.260 INFO:teuthology.orchestra.run.smithi167.stdout: "obj_delete_at_hint.0000000038", 2023-10-05T19:28:36.260 INFO:teuthology.orchestra.run.smithi167.stdout: "obj_delete_at_hint.0000000118", 2023-10-05T19:28:36.260 INFO:teuthology.orchestra.run.smithi167.stdout: "obj_delete_at_hint.0000000036", 2023-10-05T19:28:36.260 INFO:teuthology.orchestra.run.smithi167.stdout: "obj_delete_at_hint.0000000103", 2023-10-05T19:28:36.260 INFO:teuthology.orchestra.run.smithi167.stdout: "obj_delete_at_hint.0000000119", 2023-10-05T19:28:36.260 INFO:teuthology.orchestra.run.smithi167.stdout: "obj_delete_at_hint.0000000098", 2023-10-05T19:28:36.260 INFO:teuthology.orchestra.run.smithi167.stdout: "obj_delete_at_hint.0000000027", 2023-10-05T19:28:36.260 INFO:teuthology.orchestra.run.smithi167.stdout: "obj_delete_at_hint.0000000019", 2023-10-05T19:28:36.260 INFO:teuthology.orchestra.run.smithi167.stdout: "2023-10-05-19-9a9df965-6612-45d4-a45f-6e3c889bacc2.4223.2-myfoo2", 2023-10-05T19:28:36.261 INFO:teuthology.orchestra.run.smithi167.stdout: "obj_delete_at_hint.0000000039", 2023-10-05T19:28:36.261 INFO:teuthology.orchestra.run.smithi167.stdout: "obj_delete_at_hint.0000000100", 2023-10-05T19:28:36.261 INFO:teuthology.orchestra.run.smithi167.stdout: "obj_delete_at_hint.0000000093", 2023-10-05T19:28:36.261 INFO:teuthology.orchestra.run.smithi167.stdout: "2023-10-05-19-9a9df965-6612-45d4-a45f-6e3c889bacc2.4223.3-myfoo3", 2023-10-05T19:28:36.261 INFO:teuthology.orchestra.run.smithi167.stdout: "obj_delete_at_hint.0000000004", 2023-10-05T19:28:36.261 INFO:teuthology.orchestra.run.smithi167.stdout: "obj_delete_at_hint.0000000063", 2023-10-05T19:28:36.261 INFO:teuthology.orchestra.run.smithi167.stdout: "obj_delete_at_hint.0000000122", 2023-10-05T19:28:36.261 INFO:teuthology.orchestra.run.smithi167.stdout: "obj_delete_at_hint.0000000057", 2023-10-05T19:28:36.261 INFO:teuthology.orchestra.run.smithi167.stdout: "obj_delete_at_hint.0000000054", 2023-10-05T19:28:36.261 INFO:teuthology.orchestra.run.smithi167.stdout: "obj_delete_at_hint.0000000114", 2023-10-05T19:28:36.261 INFO:teuthology.orchestra.run.smithi167.stdout: "data_log.0", 2023-10-05T19:28:36.261 INFO:teuthology.orchestra.run.smithi167.stdout: "obj_delete_at_hint.0000000012", 2023-10-05T19:28:36.261 INFO:teuthology.orchestra.run.smithi167.stdout: "obj_delete_at_hint.0000000084", 2023-10-05T19:28:36.262 INFO:teuthology.orchestra.run.smithi167.stdout: "obj_delete_at_hint.0000000043", 2023-10-05T19:28:36.262 INFO:teuthology.orchestra.run.smithi167.stdout: "obj_delete_at_hint.0000000111", 2023-10-05T19:28:36.262 INFO:teuthology.orchestra.run.smithi167.stdout: "obj_delete_at_hint.0000000048", 2023-10-05T19:28:36.262 INFO:teuthology.orchestra.run.smithi167.stdout: "obj_delete_at_hint.0000000020", 2023-10-05T19:28:36.262 INFO:teuthology.orchestra.run.smithi167.stdout: "obj_delete_at_hint.0000000099", 2023-10-05T19:28:36.262 INFO:teuthology.orchestra.run.smithi167.stdout: "obj_delete_at_hint.0000000056", 2023-10-05T19:28:36.262 INFO:teuthology.orchestra.run.smithi167.stdout: "obj_delete_at_hint.0000000072", 2023-10-05T19:28:36.262 INFO:teuthology.orchestra.run.smithi167.stdout: "obj_delete_at_hint.0000000062", 2023-10-05T19:28:36.262 INFO:teuthology.orchestra.run.smithi167.stdout: "obj_delete_at_hint.0000000115", 2023-10-05T19:28:36.262 INFO:teuthology.orchestra.run.smithi167.stdout: "obj_delete_at_hint.0000000033", 2023-10-05T19:28:36.262 INFO:teuthology.orchestra.run.smithi167.stdout: "obj_delete_at_hint.0000000025", 2023-10-05T19:28:36.262 INFO:teuthology.orchestra.run.smithi167.stdout: "obj_delete_at_hint.0000000123", 2023-10-05T19:28:36.263 INFO:teuthology.orchestra.run.smithi167.stdout: "obj_delete_at_hint.0000000108", 2023-10-05T19:28:36.263 INFO:teuthology.orchestra.run.smithi167.stdout: "obj_delete_at_hint.0000000094", 2023-10-05T19:28:36.263 INFO:teuthology.orchestra.run.smithi167.stdout: "obj_delete_at_hint.0000000060", 2023-10-05T19:28:36.263 INFO:teuthology.orchestra.run.smithi167.stdout: "obj_delete_at_hint.0000000109", 2023-10-05T19:28:36.263 INFO:teuthology.orchestra.run.smithi167.stdout: "obj_delete_at_hint.0000000013", 2023-10-05T19:28:36.263 INFO:teuthology.orchestra.run.smithi167.stdout: "obj_delete_at_hint.0000000051", 2023-10-05T19:28:36.263 INFO:teuthology.orchestra.run.smithi167.stdout: "obj_delete_at_hint.0000000106", 2023-10-05T19:28:36.263 INFO:teuthology.orchestra.run.smithi167.stdout: "obj_delete_at_hint.0000000017", 2023-10-05T19:28:36.263 INFO:teuthology.orchestra.run.smithi167.stdout: "obj_delete_at_hint.0000000023", 2023-10-05T19:28:36.263 INFO:teuthology.orchestra.run.smithi167.stdout: "obj_delete_at_hint.0000000107", 2023-10-05T19:28:36.263 INFO:teuthology.orchestra.run.smithi167.stdout: "obj_delete_at_hint.0000000046", 2023-10-05T19:28:36.263 INFO:teuthology.orchestra.run.smithi167.stdout: "obj_delete_at_hint.0000000067", 2023-10-05T19:28:36.263 INFO:teuthology.orchestra.run.smithi167.stdout: "obj_delete_at_hint.0000000086", 2023-10-05T19:28:36.264 INFO:teuthology.orchestra.run.smithi167.stdout: "obj_delete_at_hint.0000000050", 2023-10-05T19:28:36.264 INFO:teuthology.orchestra.run.smithi167.stdout: "obj_delete_at_hint.0000000102", 2023-10-05T19:28:36.264 INFO:teuthology.orchestra.run.smithi167.stdout: "obj_delete_at_hint.0000000124", 2023-10-05T19:28:36.264 INFO:teuthology.orchestra.run.smithi167.stdout: "obj_delete_at_hint.0000000001", 2023-10-05T19:28:36.264 INFO:teuthology.orchestra.run.smithi167.stdout: "obj_delete_at_hint.0000000126", 2023-10-05T19:28:36.264 INFO:teuthology.orchestra.run.smithi167.stdout: "obj_delete_at_hint.0000000055", 2023-10-05T19:28:36.264 INFO:teuthology.orchestra.run.smithi167.stdout: "obj_delete_at_hint.0000000061", 2023-10-05T19:28:36.264 INFO:teuthology.orchestra.run.smithi167.stdout: "obj_delete_at_hint.0000000097", 2023-10-05T19:28:36.264 INFO:teuthology.orchestra.run.smithi167.stdout: "obj_delete_at_hint.0000000042", 2023-10-05T19:28:36.264 INFO:teuthology.orchestra.run.smithi167.stdout: "obj_delete_at_hint.0000000075", 2023-10-05T19:28:36.264 INFO:teuthology.orchestra.run.smithi167.stdout: "obj_delete_at_hint.0000000120", 2023-10-05T19:28:36.264 INFO:teuthology.orchestra.run.smithi167.stdout: "obj_delete_at_hint.0000000030" 2023-10-05T19:28:36.265 INFO:teuthology.orchestra.run.smithi167.stdout:] 2023-10-05T19:28:36.265 INFO:teuthology.orchestra.run.smithi167.stdout: 2023-10-05T19:28:36.266 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.250+0000 7f536d8cee40 1 -- 172.21.15.167:0/187505431 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:102 7.1f 7:f95f44c2:::notify.0:head [watch unwatch cookie 94612643658800] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x560cb9041d00 con 0x560cb9008e00 2023-10-05T19:28:36.266 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.250+0000 7f536ac02700 1 -- 172.21.15.167:0/187505431 <== osd.2 v2:172.21.15.167:6816/30037 17 ==== osd_op_reply(102 notify.0 [watch unwatch cookie 94612643658800] v30'152 uv150 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f5354063040 con 0x560cb9008e00 2023-10-05T19:28:36.266 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.250+0000 7f536d8cee40 20 remove_watcher() i=0 2023-10-05T19:28:36.266 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.250+0000 7f536d8cee40 2 removed watcher, disabling cache 2023-10-05T19:28:36.266 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.250+0000 7f536d8cee40 1 -- 172.21.15.167:0/187505431 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:103 7.0 7:05bf5b68:::notify.1:head [watch unwatch cookie 94612643663392] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x560cb901e7b0 con 0x560cb90388c0 2023-10-05T19:28:36.266 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.254+0000 7f536bc04700 1 -- 172.21.15.167:0/187505431 <== osd.1 v2:172.21.15.167:6800/30035 35 ==== osd_op_reply(103 notify.1 [watch unwatch cookie 94612643663392] v30'152 uv150 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f535c050040 con 0x560cb90388c0 2023-10-05T19:28:36.266 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.254+0000 7f536d8cee40 20 remove_watcher() i=1 2023-10-05T19:28:36.266 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.254+0000 7f536d8cee40 1 -- 172.21.15.167:0/187505431 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:104 7.15 7:a93a5511:::notify.2:head [watch unwatch cookie 94612643665744] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x560cb9044e20 con 0x560cb903e740 2023-10-05T19:28:36.267 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.258+0000 7f536b403700 1 -- 172.21.15.167:0/187505431 <== osd.3 v2:172.21.15.167:6808/30036 24 ==== osd_op_reply(104 notify.2 [watch unwatch cookie 94612643665744] v30'152 uv150 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f536403d040 con 0x560cb903e740 2023-10-05T19:28:36.267 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.258+0000 7f536d8cee40 20 remove_watcher() i=2 2023-10-05T19:28:36.267 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.258+0000 7f536d8cee40 1 -- 172.21.15.167:0/187505431 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:105 7.e 7:7759931f:::notify.3:head [watch unwatch cookie 94612643670160] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x560cb9045750 con 0x560cb9008e00 2023-10-05T19:28:36.267 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.262+0000 7f536ac02700 1 -- 172.21.15.167:0/187505431 <== osd.2 v2:172.21.15.167:6816/30037 18 ==== osd_op_reply(105 notify.3 [watch unwatch cookie 94612643670160] v30'152 uv150 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f5354063040 con 0x560cb9008e00 2023-10-05T19:28:36.267 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.262+0000 7f536d8cee40 20 remove_watcher() i=3 2023-10-05T19:28:36.268 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.262+0000 7f536d8cee40 1 -- 172.21.15.167:0/187505431 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:106 7.d 7:b4812045:::notify.4:head [watch unwatch cookie 94612643672368] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x560cb9045750 con 0x560cb90388c0 2023-10-05T19:28:36.269 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.262+0000 7f536bc04700 1 -- 172.21.15.167:0/187505431 <== osd.1 v2:172.21.15.167:6800/30035 36 ==== osd_op_reply(106 notify.4 [watch unwatch cookie 94612643672368] v30'152 uv150 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f535c050040 con 0x560cb90388c0 2023-10-05T19:28:36.269 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.262+0000 7f536d8cee40 20 remove_watcher() i=4 2023-10-05T19:28:36.269 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.262+0000 7f536d8cee40 1 -- 172.21.15.167:0/187505431 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:107 7.3 7:c609908c:::notify.5:head [watch unwatch cookie 94612643666848] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x560cb9045750 con 0x7f5354007030 2023-10-05T19:28:36.272 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.266+0000 7f536bc04700 1 -- 172.21.15.167:0/187505431 <== osd.0 v2:172.21.15.118:6800/27690 29 ==== osd_op_reply(107 notify.5 [watch unwatch cookie 94612643666848] v30'152 uv150 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f535c050040 con 0x7f5354007030 2023-10-05T19:28:36.272 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.266+0000 7f536d8cee40 20 remove_watcher() i=5 2023-10-05T19:28:36.272 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.266+0000 7f536d8cee40 1 -- 172.21.15.167:0/187505431 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:108 7.14 7:2b04a3e9:::notify.6:head [watch unwatch cookie 94612643677456] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x560cb9047130 con 0x7f5354007030 2023-10-05T19:28:36.275 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.270+0000 7f536bc04700 1 -- 172.21.15.167:0/187505431 <== osd.0 v2:172.21.15.118:6800/27690 30 ==== osd_op_reply(108 notify.6 [watch unwatch cookie 94612643677456] v30'152 uv150 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f535c050040 con 0x7f5354007030 2023-10-05T19:28:36.275 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.270+0000 7f536d8cee40 20 remove_watcher() i=6 2023-10-05T19:28:36.275 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.270+0000 7f536d8cee40 1 -- 172.21.15.167:0/187505431 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:109 7.9 7:93e5b521:::notify.7:head [watch unwatch cookie 94612643680592] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x560cb9047130 con 0x7f5354007030 2023-10-05T19:28:36.278 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.270+0000 7f536bc04700 1 -- 172.21.15.167:0/187505431 <== osd.0 v2:172.21.15.118:6800/27690 31 ==== osd_op_reply(109 notify.7 [watch unwatch cookie 94612643680592] v30'152 uv150 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f535c050040 con 0x7f5354007030 2023-10-05T19:28:36.278 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.270+0000 7f536d8cee40 20 remove_watcher() i=7 2023-10-05T19:28:36.279 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.274+0000 7f533cfe1700 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2023-10-05T19:28:36.279 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.274+0000 7f536d8cee40 1 -- 172.21.15.167:0/187505431 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:110 6.4 6:22d26bf9:::data_loggenerations_metadata:head [watch unwatch cookie 94612643682784] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x560cb9047130 con 0x560cb90388c0 2023-10-05T19:28:36.282 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.278+0000 7f536bc04700 1 -- 172.21.15.167:0/187505431 <== osd.1 v2:172.21.15.167:6800/30035 37 ==== osd_op_reply(110 data_loggenerations_metadata [watch unwatch cookie 94612643682784] v30'148 uv1 ondisk = 0) v8 ==== 172+0+0 (crc 0 0 0) 0x7f535c050040 con 0x560cb90388c0 2023-10-05T19:28:36.283 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.278+0000 7f536d8cee40 1 -- 172.21.15.167:0/187505431 >> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] conn(0x7f5354007030 msgr2=0x7f535401c920 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-05T19:28:36.283 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.278+0000 7f536d8cee40 1 --2- 172.21.15.167:0/187505431 >> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] conn(0x7f5354007030 0x7f535401c920 crc :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-05T19:28:36.283 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.278+0000 7f536d8cee40 1 -- 172.21.15.167:0/187505431 >> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] conn(0x560cb90388c0 msgr2=0x560cb903aca0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-05T19:28:36.283 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.278+0000 7f536d8cee40 1 --2- 172.21.15.167:0/187505431 >> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] conn(0x560cb90388c0 0x560cb903aca0 crc :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-05T19:28:36.283 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.278+0000 7f536d8cee40 1 -- 172.21.15.167:0/187505431 >> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] conn(0x560cb9008e00 msgr2=0x7f53640113e0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-05T19:28:36.283 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.278+0000 7f536d8cee40 1 --2- 172.21.15.167:0/187505431 >> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] conn(0x560cb9008e00 0x7f53640113e0 crc :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-05T19:28:36.283 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.278+0000 7f536d8cee40 1 -- 172.21.15.167:0/187505431 >> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] conn(0x560cb903e740 msgr2=0x560cb9040b20 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-05T19:28:36.283 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.278+0000 7f536d8cee40 1 --2- 172.21.15.167:0/187505431 >> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] conn(0x560cb903e740 0x560cb9040b20 crc :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-05T19:28:36.283 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.278+0000 7f536d8cee40 1 -- 172.21.15.167:0/187505431 >> [v2:172.21.15.167:6824/29889,v1:172.21.15.167:6825/29889] conn(0x7f534c030c00 msgr2=0x7f534c0330b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-05T19:28:36.283 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.278+0000 7f536d8cee40 1 --2- 172.21.15.167:0/187505431 >> [v2:172.21.15.167:6824/29889,v1:172.21.15.167:6825/29889] conn(0x7f534c030c00 0x7f534c0330b0 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7f5354005a90 tx=0x7f5354015000 comp rx=0 tx=0).stop 2023-10-05T19:28:36.284 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.278+0000 7f536bc04700 1 -- 172.21.15.167:0/187505431 reap_dead start 2023-10-05T19:28:36.284 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.278+0000 7f536d8cee40 1 -- 172.21.15.167:0/187505431 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x560cb900d3e0 msgr2=0x560cb902ee10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-05T19:28:36.284 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.278+0000 7f536d8cee40 1 --2- 172.21.15.167:0/187505431 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x560cb900d3e0 0x560cb902ee10 secure :-1 s=READY pgs=437 cs=0 l=1 rev1=1 crypto rx=0x7f5364000f80 tx=0x7f5364014000 comp rx=0 tx=0).stop 2023-10-05T19:28:36.284 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.278+0000 7f536d8cee40 1 -- 172.21.15.167:0/187505431 shutdown_connections 2023-10-05T19:28:36.284 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.278+0000 7f536d8cee40 1 --2- 172.21.15.167:0/187505431 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x560cb900d3e0 0x560cb902ee10 unknown :-1 s=CLOSED pgs=437 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-05T19:28:36.284 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.278+0000 7f536d8cee40 1 -- 172.21.15.167:0/187505431 >> 172.21.15.167:0/187505431 conn(0x560cb9004820 msgr2=0x560cb9022510 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-05T19:28:36.285 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.278+0000 7f536d8cee40 1 -- 172.21.15.167:0/187505431 shutdown_connections 2023-10-05T19:28:36.286 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.278+0000 7f536d8cee40 1 -- 172.21.15.167:0/187505431 wait complete. 2023-10-05T19:28:36.286 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.278+0000 7f536d8cee40 1 -- 172.21.15.167:0/2614141872 >> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] conn(0x560cb90c5410 msgr2=0x560cb90c77f0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-05T19:28:36.286 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.278+0000 7f536d8cee40 1 --2- 172.21.15.167:0/2614141872 >> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] conn(0x560cb90c5410 0x560cb90c77f0 crc :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-05T19:28:36.286 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.278+0000 7f536d8cee40 1 -- 172.21.15.167:0/2614141872 >> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] conn(0x7f536403d7a0 msgr2=0x7f5364042750 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-05T19:28:36.286 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.278+0000 7f536d8cee40 1 --2- 172.21.15.167:0/2614141872 >> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] conn(0x7f536403d7a0 0x7f5364042750 crc :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-05T19:28:36.286 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.278+0000 7f536d8cee40 1 -- 172.21.15.167:0/2614141872 >> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] conn(0x7f535c060040 msgr2=0x7f535c054960 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-05T19:28:36.286 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.278+0000 7f536d8cee40 1 --2- 172.21.15.167:0/2614141872 >> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] conn(0x7f535c060040 0x7f535c054960 crc :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-05T19:28:36.286 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.278+0000 7f536d8cee40 1 -- 172.21.15.167:0/2614141872 >> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] conn(0x7f536403ebd0 msgr2=0x7f53640123c0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-05T19:28:36.286 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.278+0000 7f536d8cee40 1 --2- 172.21.15.167:0/2614141872 >> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] conn(0x7f536403ebd0 0x7f53640123c0 crc :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-05T19:28:36.286 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.278+0000 7f536d8cee40 1 -- 172.21.15.167:0/2614141872 >> [v2:172.21.15.167:6824/29889,v1:172.21.15.167:6825/29889] conn(0x7f52ac030c50 msgr2=0x7f52ac033100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-05T19:28:36.286 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.278+0000 7f536d8cee40 1 --2- 172.21.15.167:0/2614141872 >> [v2:172.21.15.167:6824/29889,v1:172.21.15.167:6825/29889] conn(0x7f52ac030c50 0x7f52ac033100 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x560cb904d0a0 tx=0x7f535c000f90 comp rx=0 tx=0).stop 2023-10-05T19:28:36.287 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.278+0000 7f536d8cee40 1 -- 172.21.15.167:0/2614141872 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x560cb90574c0 msgr2=0x560cb905ced0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-05T19:28:36.287 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.278+0000 7f536b403700 1 -- 172.21.15.167:0/2614141872 reap_dead start 2023-10-05T19:28:36.287 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.278+0000 7f536d8cee40 1 --2- 172.21.15.167:0/2614141872 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x560cb90574c0 0x560cb905ced0 secure :-1 s=READY pgs=439 cs=0 l=1 rev1=1 crypto rx=0x560cb90515a0 tx=0x7f535402f000 comp rx=0 tx=0).stop 2023-10-05T19:28:36.287 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.278+0000 7f536d8cee40 1 -- 172.21.15.167:0/2614141872 shutdown_connections 2023-10-05T19:28:36.287 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.278+0000 7f536d8cee40 1 --2- 172.21.15.167:0/2614141872 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x560cb90574c0 0x560cb905ced0 unknown :-1 s=CLOSED pgs=439 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-05T19:28:36.287 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.278+0000 7f536d8cee40 1 -- 172.21.15.167:0/2614141872 >> 172.21.15.167:0/2614141872 conn(0x560cb9052880 msgr2=0x560cb9050980 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-05T19:28:36.287 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.278+0000 7f536d8cee40 1 -- 172.21.15.167:0/2614141872 shutdown_connections 2023-10-05T19:28:36.287 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.278+0000 7f536d8cee40 1 -- 172.21.15.167:0/2614141872 wait complete. 2023-10-05T19:28:36.291 DEBUG:tasks.util.rgw: json result: ['obj_delete_at_hint.0000000078', 'obj_delete_at_hint.0000000070', 'obj_delete_at_hint.0000000104', 'obj_delete_at_hint.0000000026', 'obj_delete_at_hint.0000000028', 'obj_delete_at_hint.0000000040', 'obj_delete_at_hint.0000000015', 'obj_delete_at_hint.0000000069', 'obj_delete_at_hint.0000000095', 'obj_delete_at_hint.0000000003', 'obj_delete_at_hint.0000000047', 'obj_delete_at_hint.0000000035', 'obj_delete_at_hint.0000000037', '2023-10-05-19--', 'obj_delete_at_hint.0000000024', '2023-10-05-19-9a9df965-6612-45d4-a45f-6e3c889bacc2.4223.4-myfoo4', 'obj_delete_at_hint.0000000000', 'obj_delete_at_hint.0000000031', 'data_loggenerations_metadata', 'obj_delete_at_hint.0000000076', 'obj_delete_at_hint.0000000113', 'obj_delete_at_hint.0000000005', 'obj_delete_at_hint.0000000011', 'obj_delete_at_hint.0000000058', 'obj_delete_at_hint.0000000052', 'obj_delete_at_hint.0000000088', 'obj_delete_at_hint.0000000080', 'obj_delete_at_hint.0000000090', 'obj_delete_at_hint.0000000110', 'obj_delete_at_hint.0000000096', 'obj_delete_at_hint.0000000087', 'obj_delete_at_hint.0000000008', 'obj_delete_at_hint.0000000006', 'obj_delete_at_hint.0000000029', 'obj_delete_at_hint.0000000089', 'obj_delete_at_hint.0000000022', '2023-10-05-19-9a9df965-6612-45d4-a45f-6e3c889bacc2.4223.1-myfoo', 'obj_delete_at_hint.0000000016', 'obj_delete_at_hint.0000000041', 'obj_delete_at_hint.0000000018', 'obj_delete_at_hint.0000000092', 'obj_delete_at_hint.0000000014', 'obj_delete_at_hint.0000000112', 'obj_delete_at_hint.0000000007', 'obj_delete_at_hint.0000000021', 'obj_delete_at_hint.0000000064', 'obj_delete_at_hint.0000000071', 'obj_delete_at_hint.0000000074', 'obj_delete_at_hint.0000000081', 'obj_delete_at_hint.0000000009', 'obj_delete_at_hint.0000000121', 'obj_delete_at_hint.0000000125', 'obj_delete_at_hint.0000000082', 'obj_delete_at_hint.0000000105', 'obj_delete_at_hint.0000000059', 'obj_delete_at_hint.0000000077', 'obj_delete_at_hint.0000000032', 'obj_delete_at_hint.0000000053', 'obj_delete_at_hint.0000000091', 'obj_delete_at_hint.0000000065', 'obj_delete_at_hint.0000000083', 'obj_delete_at_hint.0000000010', 'obj_delete_at_hint.0000000045', 'obj_delete_at_hint.0000000002', 'obj_delete_at_hint.0000000116', 'obj_delete_at_hint.0000000034', 'obj_delete_at_hint.0000000101', 'obj_delete_at_hint.0000000079', 'obj_delete_at_hint.0000000049', 'obj_delete_at_hint.0000000117', 'obj_delete_at_hint.0000000044', 'obj_delete_at_hint.0000000066', 'obj_delete_at_hint.0000000068', 'obj_delete_at_hint.0000000085', 'obj_delete_at_hint.0000000073', 'obj_delete_at_hint.0000000038', 'obj_delete_at_hint.0000000118', 'obj_delete_at_hint.0000000036', 'obj_delete_at_hint.0000000103', 'obj_delete_at_hint.0000000119', 'obj_delete_at_hint.0000000098', 'obj_delete_at_hint.0000000027', 'obj_delete_at_hint.0000000019', '2023-10-05-19-9a9df965-6612-45d4-a45f-6e3c889bacc2.4223.2-myfoo2', 'obj_delete_at_hint.0000000039', 'obj_delete_at_hint.0000000100', 'obj_delete_at_hint.0000000093', '2023-10-05-19-9a9df965-6612-45d4-a45f-6e3c889bacc2.4223.3-myfoo3', 'obj_delete_at_hint.0000000004', 'obj_delete_at_hint.0000000063', 'obj_delete_at_hint.0000000122', 'obj_delete_at_hint.0000000057', 'obj_delete_at_hint.0000000054', 'obj_delete_at_hint.0000000114', 'data_log.0', 'obj_delete_at_hint.0000000012', 'obj_delete_at_hint.0000000084', 'obj_delete_at_hint.0000000043', 'obj_delete_at_hint.0000000111', 'obj_delete_at_hint.0000000048', 'obj_delete_at_hint.0000000020', 'obj_delete_at_hint.0000000099', 'obj_delete_at_hint.0000000056', 'obj_delete_at_hint.0000000072', 'obj_delete_at_hint.0000000062', 'obj_delete_at_hint.0000000115', 'obj_delete_at_hint.0000000033', 'obj_delete_at_hint.0000000025', 'obj_delete_at_hint.0000000123', 'obj_delete_at_hint.0000000108', 'obj_delete_at_hint.0000000094', 'obj_delete_at_hint.0000000060', 'obj_delete_at_hint.0000000109', 'obj_delete_at_hint.0000000013', 'obj_delete_at_hint.0000000051', 'obj_delete_at_hint.0000000106', 'obj_delete_at_hint.0000000017', 'obj_delete_at_hint.0000000023', 'obj_delete_at_hint.0000000107', 'obj_delete_at_hint.0000000046', 'obj_delete_at_hint.0000000067', 'obj_delete_at_hint.0000000086', 'obj_delete_at_hint.0000000050', 'obj_delete_at_hint.0000000102', 'obj_delete_at_hint.0000000124', 'obj_delete_at_hint.0000000001', 'obj_delete_at_hint.0000000126', 'obj_delete_at_hint.0000000055', 'obj_delete_at_hint.0000000061', 'obj_delete_at_hint.0000000097', 'obj_delete_at_hint.0000000042', 'obj_delete_at_hint.0000000075', 'obj_delete_at_hint.0000000120', 'obj_delete_at_hint.0000000030'] 2023-10-05T19:28:36.291 INFO:tasks.util.rgw:rgwadmin: client.0 : ['log', 'show', '--object', '2023-10-05-19--'] 2023-10-05T19:28:36.291 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'ceph', 'log', 'show', '--object', '2023-10-05-19--'] 2023-10-05T19:28:36.291 DEBUG:teuthology.orchestra.run.smithi167:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster ceph log show --object 2023-10-05-19-- 2023-10-05T19:28:36.332 INFO:teuthology.orchestra.run.smithi167.stderr:ignoring --setuser ceph since I am not root 2023-10-05T19:28:36.332 INFO:teuthology.orchestra.run.smithi167.stderr:ignoring --setgroup ceph since I am not root 2023-10-05T19:28:36.333 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.326+0000 7f805e5ebe40 1 Processor -- start 2023-10-05T19:28:36.333 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.326+0000 7f805e5ebe40 1 -- start start 2023-10-05T19:28:36.333 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.326+0000 7f805e5ebe40 1 --2- >> v2:172.21.15.118:3300/0 conn(0x559b75d51400 0x559b75d537e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-05T19:28:36.333 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.326+0000 7f805e5ebe40 1 -- --> v1:172.21.15.118:6789/0 -- auth(proto 0 26 bytes epoch 0) v1 -- 0x559b75c99bd0 con 0x559b75cc0c50 2023-10-05T19:28:36.334 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.326+0000 7f805e5ebe40 1 -- --> v2:172.21.15.118:3300/0 -- mon_getmap magic: 0 v1 -- 0x559b75c56ee0 con 0x559b75d51400 2023-10-05T19:28:36.334 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.326+0000 7f805b91f700 1 --2- >> v2:172.21.15.118:3300/0 conn(0x559b75d51400 0x559b75d537e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-05T19:28:36.334 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.326+0000 7f805c120700 1 --1- >> v1:172.21.15.118:6789/0 conn(0x559b75cc0c50 0x559b75cc1020 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.118:6789/0 says I am v1:172.21.15.167:59312/0 (socket says 172.21.15.167:59312) 2023-10-05T19:28:36.335 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.330+0000 7f805b91f700 1 --2- >> v2:172.21.15.118:3300/0 conn(0x559b75d51400 0x559b75d537e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.118:3300/0 says I am v2:172.21.15.167:41774/0 (socket says 172.21.15.167:41774) 2023-10-05T19:28:36.335 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.330+0000 7f805c120700 1 -- 172.21.15.167:0/1269273884 learned_addr learned my addr 172.21.15.167:0/1269273884 (peer_addr_for_me v1:172.21.15.167:0/0) 2023-10-05T19:28:36.335 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.330+0000 7f805b91f700 1 -- 172.21.15.167:0/1269273884 >> v1:172.21.15.118:6789/0 conn(0x559b75cc0c50 legacy=0x559b75cc1020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-05T19:28:36.335 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.330+0000 7f805b11e700 1 -- 172.21.15.167:0/1269273884 <== mon.0 v1:172.21.15.118:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1598982132 0 0) 0x559b75c99bd0 con 0x559b75cc0c50 2023-10-05T19:28:36.335 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.330+0000 7f805b91f700 1 -- 172.21.15.167:0/1269273884 --> v2:172.21.15.118:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x559b75cc1790 con 0x559b75d51400 2023-10-05T19:28:36.336 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.330+0000 7f805b91f700 1 --2- 172.21.15.167:0/1269273884 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x559b75d51400 0x559b75d537e0 secure :-1 s=READY pgs=441 cs=0 l=1 rev1=1 crypto rx=0x7f804400a5d0 tx=0x7f8044005a50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=baa4e2b0d3ace6ce server_cookie=0 in_seq=0 out_seq=0 2023-10-05T19:28:36.336 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.330+0000 7f805b11e700 1 -- 172.21.15.167:0/1269273884 <== mon.0 v2:172.21.15.118:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f80440086a0 con 0x559b75d51400 2023-10-05T19:28:36.336 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.330+0000 7f805b11e700 1 -- 172.21.15.167:0/1269273884 <== mon.0 v2:172.21.15.118:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8044008800 con 0x559b75d51400 2023-10-05T19:28:36.336 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.330+0000 7f805b11e700 1 -- 172.21.15.167:0/1269273884 <== mon.0 v2:172.21.15.118:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f8044008ad0 con 0x559b75d51400 2023-10-05T19:28:36.337 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.330+0000 7f805e5ebe40 1 -- 172.21.15.167:0/1269273884 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x559b75d51400 msgr2=0x559b75d537e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-05T19:28:36.337 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.330+0000 7f805e5ebe40 1 --2- 172.21.15.167:0/1269273884 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x559b75d51400 0x559b75d537e0 secure :-1 s=READY pgs=441 cs=0 l=1 rev1=1 crypto rx=0x7f804400a5d0 tx=0x7f8044005a50 comp rx=0 tx=0).stop 2023-10-05T19:28:36.337 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.330+0000 7f805e5ebe40 1 -- 172.21.15.167:0/1269273884 shutdown_connections 2023-10-05T19:28:36.337 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.330+0000 7f805e5ebe40 1 --2- 172.21.15.167:0/1269273884 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x559b75d51400 0x559b75d537e0 unknown :-1 s=CLOSED pgs=441 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-05T19:28:36.337 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.330+0000 7f805e5ebe40 1 -- 172.21.15.167:0/1269273884 >> 172.21.15.167:0/1269273884 conn(0x559b75d47720 msgr2=0x559b75d49b30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-05T19:28:36.337 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.330+0000 7f805e5ebe40 1 -- 172.21.15.167:0/1269273884 shutdown_connections 2023-10-05T19:28:36.337 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.330+0000 7f805e5ebe40 1 -- 172.21.15.167:0/1269273884 wait complete. 2023-10-05T19:28:36.339 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.334+0000 7f805e5ebe40 1 Processor -- start 2023-10-05T19:28:36.339 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.334+0000 7f805e5ebe40 1 -- start start 2023-10-05T19:28:36.339 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.334+0000 7f805e5ebe40 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x559b75cc0c50 0x559b75d6bd50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-05T19:28:36.339 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.334+0000 7f805e5ebe40 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x559b75c1ae30 con 0x559b75cc0c50 2023-10-05T19:28:36.340 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.334+0000 7f805c120700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x559b75cc0c50 0x559b75d6bd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-05T19:28:36.340 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.334+0000 7f805c120700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x559b75cc0c50 0x559b75d6bd50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.118:3300/0 says I am v2:172.21.15.167:41780/0 (socket says 172.21.15.167:41780) 2023-10-05T19:28:36.340 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.334+0000 7f805c120700 1 -- 172.21.15.167:0/3879955919 learned_addr learned my addr 172.21.15.167:0/3879955919 (peer_addr_for_me v2:172.21.15.167:0/0) 2023-10-05T19:28:36.340 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.334+0000 7f805c120700 1 -- 172.21.15.167:0/3879955919 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8044005cf0 con 0x559b75cc0c50 2023-10-05T19:28:36.341 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.334+0000 7f805c120700 1 --2- 172.21.15.167:0/3879955919 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x559b75cc0c50 0x559b75d6bd50 secure :-1 s=READY pgs=442 cs=0 l=1 rev1=1 crypto rx=0x7f8054001630 tx=0x7f80540069b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=116ba4131e137a03 server_cookie=0 in_seq=0 out_seq=0 2023-10-05T19:28:36.341 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.334+0000 7f805a11c700 1 -- 172.21.15.167:0/3879955919 <== mon.0 v2:172.21.15.118:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f8054006320 con 0x559b75cc0c50 2023-10-05T19:28:36.341 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.334+0000 7f805a11c700 1 -- 172.21.15.167:0/3879955919 <== mon.0 v2:172.21.15.118:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f80540044c0 con 0x559b75cc0c50 2023-10-05T19:28:36.341 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.334+0000 7f805a11c700 1 -- 172.21.15.167:0/3879955919 <== mon.0 v2:172.21.15.118:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f8054004650 con 0x559b75cc0c50 2023-10-05T19:28:36.342 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.338+0000 7f805e5ebe40 1 -- 172.21.15.167:0/3879955919 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x559b75cc0c50 msgr2=0x559b75d6bd50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-05T19:28:36.343 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.338+0000 7f805e5ebe40 1 --2- 172.21.15.167:0/3879955919 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x559b75cc0c50 0x559b75d6bd50 secure :-1 s=READY pgs=442 cs=0 l=1 rev1=1 crypto rx=0x7f8054001630 tx=0x7f80540069b0 comp rx=0 tx=0).stop 2023-10-05T19:28:36.343 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.338+0000 7f805e5ebe40 1 -- 172.21.15.167:0/3879955919 shutdown_connections 2023-10-05T19:28:36.343 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.338+0000 7f805e5ebe40 1 --2- 172.21.15.167:0/3879955919 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x559b75cc0c50 0x559b75d6bd50 unknown :-1 s=CLOSED pgs=442 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-05T19:28:36.343 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.338+0000 7f805e5ebe40 1 -- 172.21.15.167:0/3879955919 >> 172.21.15.167:0/3879955919 conn(0x559b75d47720 msgr2=0x559b75d67910 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-05T19:28:36.343 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.338+0000 7f805e5ebe40 1 -- 172.21.15.167:0/3879955919 shutdown_connections 2023-10-05T19:28:36.343 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.338+0000 7f805e5ebe40 1 -- 172.21.15.167:0/3879955919 wait complete. 2023-10-05T19:28:36.343 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.338+0000 7f805e5ebe40 1 Processor -- start 2023-10-05T19:28:36.344 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.338+0000 7f805e5ebe40 1 -- start start 2023-10-05T19:28:36.344 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.338+0000 7f805e5ebe40 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x559b75cc0c50 0x559b75d72b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-05T19:28:36.344 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.338+0000 7f805e5ebe40 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x559b75beded0 con 0x559b75cc0c50 2023-10-05T19:28:36.344 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.338+0000 7f805c120700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x559b75cc0c50 0x559b75d72b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-05T19:28:36.345 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.338+0000 7f805c120700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x559b75cc0c50 0x559b75d72b30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.118:3300/0 says I am v2:172.21.15.167:41796/0 (socket says 172.21.15.167:41796) 2023-10-05T19:28:36.345 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.338+0000 7f805c120700 1 -- 172.21.15.167:0/453671774 learned_addr learned my addr 172.21.15.167:0/453671774 (peer_addr_for_me v2:172.21.15.167:0/0) 2023-10-05T19:28:36.346 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.338+0000 7f805c120700 1 -- 172.21.15.167:0/453671774 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8054006c90 con 0x559b75cc0c50 2023-10-05T19:28:36.346 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.338+0000 7f805c120700 1 --2- 172.21.15.167:0/453671774 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x559b75cc0c50 0x559b75d72b30 secure :-1 s=READY pgs=443 cs=0 l=1 rev1=1 crypto rx=0x7f8054000f80 tx=0x7f8054014000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-05T19:28:36.346 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.338+0000 7f8058919700 1 -- 172.21.15.167:0/453671774 <== mon.0 v2:172.21.15.118:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f80540136c0 con 0x559b75cc0c50 2023-10-05T19:28:36.346 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.338+0000 7f8058919700 1 -- 172.21.15.167:0/453671774 <== mon.0 v2:172.21.15.118:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8054013820 con 0x559b75cc0c50 2023-10-05T19:28:36.346 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.338+0000 7f805e5ebe40 1 -- 172.21.15.167:0/453671774 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x559b75d73070 con 0x559b75cc0c50 2023-10-05T19:28:36.346 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.338+0000 7f8058919700 1 -- 172.21.15.167:0/453671774 <== mon.0 v2:172.21.15.118:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f8054013ad0 con 0x559b75cc0c50 2023-10-05T19:28:36.346 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.342+0000 7f805e5ebe40 1 -- 172.21.15.167:0/453671774 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x559b75d734b0 con 0x559b75cc0c50 2023-10-05T19:28:36.347 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.342+0000 7f8058919700 1 -- 172.21.15.167:0/453671774 <== mon.0 v2:172.21.15.118:3300/0 4 ==== mgrmap(e 5) v1 ==== 40329+0+0 (secure 0 0 0) 0x7f805401a070 con 0x559b75cc0c50 2023-10-05T19:28:36.347 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.342+0000 7f8058919700 1 --2- 172.21.15.167:0/453671774 >> [v2:172.21.15.167:6824/29889,v1:172.21.15.167:6825/29889] conn(0x7f803c030c50 0x7f803c033100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-05T19:28:36.347 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.342+0000 7f8058919700 1 -- 172.21.15.167:0/453671774 <== mon.0 v2:172.21.15.118:3300/0 5 ==== osd_map(30..30 src has 1..30) v4 ==== 6057+0+0 (secure 0 0 0) 0x7f8054044f00 con 0x559b75cc0c50 2023-10-05T19:28:36.348 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.342+0000 7f805b91f700 1 --2- 172.21.15.167:0/453671774 >> [v2:172.21.15.167:6824/29889,v1:172.21.15.167:6825/29889] conn(0x7f803c030c50 0x7f803c033100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-05T19:28:36.348 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.342+0000 7f805e5ebe40 20 rados->read ofs=0 len=0 2023-10-05T19:28:36.348 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.342+0000 7f805e5ebe40 1 --2- 172.21.15.167:0/453671774 >> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] conn(0x559b75d51400 0x559b75d7e580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-05T19:28:36.348 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.342+0000 7f805e5ebe40 1 -- 172.21.15.167:0/453671774 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:1 5.12 5:49953fa1:::default.realm:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x559b75d7ecb0 con 0x559b75d51400 2023-10-05T19:28:36.348 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.342+0000 7f805c921700 1 --2- 172.21.15.167:0/453671774 >> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] conn(0x559b75d51400 0x559b75d7e580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-05T19:28:36.349 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.342+0000 7f805b91f700 1 --2- 172.21.15.167:0/453671774 >> [v2:172.21.15.167:6824/29889,v1:172.21.15.167:6825/29889] conn(0x7f803c030c50 0x7f803c033100 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7f8044005a90 tx=0x7f8044015000 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-05T19:28:36.349 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.342+0000 7f805c921700 1 --2- 172.21.15.167:0/453671774 >> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] conn(0x559b75d51400 0x559b75d7e580 crc :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-05T19:28:36.350 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.342+0000 7f805c921700 1 -- 172.21.15.167:0/453671774 <== osd.1 v2:172.21.15.167:6800/30035 1 ==== osd_op_reply(1 default.realm [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 ==== 157+0+0 (crc 0 0 0) 0x7f804c00a500 con 0x559b75d51400 2023-10-05T19:28:36.350 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.342+0000 7f805e5ebe40 20 rados_obj.operate() r=-2 bl.length=0 2023-10-05T19:28:36.350 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.342+0000 7f805e5ebe40 20 realm 2023-10-05T19:28:36.350 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.342+0000 7f805e5ebe40 20 rados->read ofs=0 len=0 2023-10-05T19:28:36.350 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.342+0000 7f805e5ebe40 1 -- 172.21.15.167:0/453671774 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:2 5.12 5:49953fa1:::default.realm:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x559b75d7bb20 con 0x559b75d51400 2023-10-05T19:28:36.351 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.346+0000 7f805c921700 1 -- 172.21.15.167:0/453671774 <== osd.1 v2:172.21.15.167:6800/30035 2 ==== osd_op_reply(2 default.realm [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 ==== 157+0+0 (crc 0 0 0) 0x7f804c00a500 con 0x559b75d51400 2023-10-05T19:28:36.351 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.346+0000 7f805e5ebe40 20 rados_obj.operate() r=-2 bl.length=0 2023-10-05T19:28:36.351 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.346+0000 7f805e5ebe40 4 RGWPeriod::init failed to init realm id : (2) No such file or directory 2023-10-05T19:28:36.351 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.346+0000 7f805e5ebe40 20 rados->read ofs=0 len=0 2023-10-05T19:28:36.351 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.346+0000 7f805e5ebe40 1 -- 172.21.15.167:0/453671774 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:3 5.12 5:49953fa1:::default.realm:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x559b75d7f790 con 0x559b75d51400 2023-10-05T19:28:36.353 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.346+0000 7f805c921700 1 -- 172.21.15.167:0/453671774 <== osd.1 v2:172.21.15.167:6800/30035 3 ==== osd_op_reply(3 default.realm [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 ==== 157+0+0 (crc 0 0 0) 0x7f804c00a500 con 0x559b75d51400 2023-10-05T19:28:36.353 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.346+0000 7f805e5ebe40 20 rados_obj.operate() r=-2 bl.length=0 2023-10-05T19:28:36.353 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.346+0000 7f805e5ebe40 20 rados->read ofs=0 len=0 2023-10-05T19:28:36.353 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.346+0000 7f805e5ebe40 1 -- 172.21.15.167:0/453671774 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:4 5.f 5:f43b6ece:::zone_names.default:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x559b75d7f370 con 0x559b75d51400 2023-10-05T19:28:36.353 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.346+0000 7f805c921700 1 -- 172.21.15.167:0/453671774 <== osd.1 v2:172.21.15.167:6800/30035 4 ==== osd_op_reply(4 zone_names.default [read 0~46 out=46b] v0'0 uv1 ondisk = 0) v8 ==== 162+0+46 (crc 0 0 0) 0x7f804c00a500 con 0x559b75d51400 2023-10-05T19:28:36.353 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.346+0000 7f805e5ebe40 20 rados_obj.operate() r=0 bl.length=46 2023-10-05T19:28:36.353 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.346+0000 7f805e5ebe40 20 rados->read ofs=0 len=0 2023-10-05T19:28:36.353 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.346+0000 7f805e5ebe40 1 -- 172.21.15.167:0/453671774 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:5 5.1d 5:b8374eb4:::zone_info.9a9df965-6612-45d4-a45f-6e3c889bacc2:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x559b75d80200 con 0x559b75d51400 2023-10-05T19:28:36.353 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.346+0000 7f805c921700 1 -- 172.21.15.167:0/453671774 <== osd.1 v2:172.21.15.167:6800/30035 5 ==== osd_op_reply(5 zone_info.9a9df965-6612-45d4-a45f-6e3c889bacc2 [read 0~874 out=874b] v0'0 uv2 ondisk = 0) v8 ==== 190+0+874 (crc 0 0 0) 0x7f804c00a500 con 0x559b75d51400 2023-10-05T19:28:36.353 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.346+0000 7f805e5ebe40 20 rados_obj.operate() r=0 bl.length=874 2023-10-05T19:28:36.353 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.346+0000 7f805e5ebe40 20 searching for the correct realm 2023-10-05T19:28:36.354 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.346+0000 7f805e5ebe40 1 --2- 172.21.15.167:0/453671774 >> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] conn(0x559b75d82050 0x559b75d84430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-05T19:28:36.354 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.346+0000 7f805e5ebe40 1 -- 172.21.15.167:0/453671774 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:6 5.0 5:00000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x559b75d84970 con 0x559b75d82050 2023-10-05T19:28:36.354 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.346+0000 7f805c120700 1 --2- 172.21.15.167:0/453671774 >> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] conn(0x559b75d82050 0x559b75d84430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-05T19:28:36.354 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.346+0000 7f805c120700 1 --2- 172.21.15.167:0/453671774 >> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] conn(0x559b75d82050 0x559b75d84430 crc :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-05T19:28:36.355 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.350+0000 7f805c120700 1 -- 172.21.15.167:0/453671774 <== osd.3 v2:172.21.15.167:6808/30036 1 ==== osd_op_reply(6 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f8054025020 con 0x559b75d82050 2023-10-05T19:28:36.356 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.350+0000 7f805c120700 1 --2- 172.21.15.167:0/453671774 >> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] conn(0x7f80540106e0 0x7f805402a000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-05T19:28:36.356 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.350+0000 7f805c120700 1 -- 172.21.15.167:0/453671774 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:7 5.10 5:08000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x559b75d84970 con 0x7f80540106e0 2023-10-05T19:28:36.356 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.350+0000 7f805b91f700 1 --2- 172.21.15.167:0/453671774 >> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] conn(0x7f80540106e0 0x7f805402a000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-05T19:28:36.356 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.350+0000 7f805b91f700 1 --2- 172.21.15.167:0/453671774 >> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] conn(0x7f80540106e0 0x7f805402a000 crc :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-05T19:28:36.357 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.350+0000 7f805b91f700 1 -- 172.21.15.167:0/453671774 <== osd.2 v2:172.21.15.167:6816/30037 1 ==== osd_op_reply(7 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f8044012450 con 0x7f80540106e0 2023-10-05T19:28:36.357 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.350+0000 7f805b91f700 1 --2- 172.21.15.167:0/453671774 >> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] conn(0x7f8044007030 0x7f804401c920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-05T19:28:36.357 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.350+0000 7f805b91f700 1 -- 172.21.15.167:0/453671774 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:8 5.8 5:10000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x559b75d84970 con 0x7f8044007030 2023-10-05T19:28:36.357 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.350+0000 7f805c921700 1 --2- 172.21.15.167:0/453671774 >> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] conn(0x7f8044007030 0x7f804401c920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-05T19:28:36.358 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.354+0000 7f805c921700 1 --2- 172.21.15.167:0/453671774 >> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] conn(0x7f8044007030 0x7f804401c920 crc :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-05T19:28:36.359 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.354+0000 7f805c921700 1 -- 172.21.15.167:0/453671774 <== osd.0 v2:172.21.15.118:6800/27690 1 ==== osd_op_reply(8 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f804c014040 con 0x7f8044007030 2023-10-05T19:28:36.359 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.354+0000 7f805c921700 1 -- 172.21.15.167:0/453671774 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:9 5.18 5:18000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x559b75d84970 con 0x559b75d51400 2023-10-05T19:28:36.359 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.354+0000 7f805c921700 1 -- 172.21.15.167:0/453671774 <== osd.1 v2:172.21.15.167:6800/30035 6 ==== osd_op_reply(9 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f804c014040 con 0x559b75d51400 2023-10-05T19:28:36.360 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.354+0000 7f805c921700 1 -- 172.21.15.167:0/453671774 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:10 5.4 5:20000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x559b75d84970 con 0x559b75d51400 2023-10-05T19:28:36.360 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.354+0000 7f805c921700 1 -- 172.21.15.167:0/453671774 <== osd.1 v2:172.21.15.167:6800/30035 7 ==== osd_op_reply(10 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f804c014040 con 0x559b75d51400 2023-10-05T19:28:36.360 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.354+0000 7f805c921700 1 -- 172.21.15.167:0/453671774 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:11 5.14 5:28000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x559b75d84970 con 0x7f8044007030 2023-10-05T19:28:36.361 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.354+0000 7f805c921700 1 -- 172.21.15.167:0/453671774 <== osd.0 v2:172.21.15.118:6800/27690 2 ==== osd_op_reply(11 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f804c014040 con 0x7f8044007030 2023-10-05T19:28:36.361 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.354+0000 7f805c921700 1 -- 172.21.15.167:0/453671774 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:12 5.c 5:30000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x559b75d84970 con 0x7f8044007030 2023-10-05T19:28:36.362 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.354+0000 7f805c921700 1 -- 172.21.15.167:0/453671774 <== osd.0 v2:172.21.15.118:6800/27690 3 ==== osd_op_reply(12 [pgnls start_epoch 30 out=112b] v23'1 uv1 ondisk = 1) v8 ==== 144+0+112 (crc 0 0 0) 0x7f804c014040 con 0x7f8044007030 2023-10-05T19:28:36.362 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.354+0000 7f805c921700 1 -- 172.21.15.167:0/453671774 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:13 5.1c 5:38000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x559b75d84970 con 0x7f80540106e0 2023-10-05T19:28:36.362 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.358+0000 7f805b91f700 1 -- 172.21.15.167:0/453671774 <== osd.2 v2:172.21.15.167:6816/30037 2 ==== osd_op_reply(13 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f8044012450 con 0x7f80540106e0 2023-10-05T19:28:36.363 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.358+0000 7f805b91f700 1 -- 172.21.15.167:0/453671774 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:14 5.2 5:40000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x559b75d84970 con 0x559b75d82050 2023-10-05T19:28:36.363 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.358+0000 7f805c120700 1 -- 172.21.15.167:0/453671774 <== osd.3 v2:172.21.15.167:6808/30036 2 ==== osd_op_reply(14 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f8054025020 con 0x559b75d82050 2023-10-05T19:28:36.363 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.358+0000 7f805c120700 1 -- 172.21.15.167:0/453671774 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:15 5.12 5:48000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x559b75d84970 con 0x559b75d51400 2023-10-05T19:28:36.364 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.358+0000 7f805c921700 1 -- 172.21.15.167:0/453671774 <== osd.1 v2:172.21.15.167:6800/30035 8 ==== osd_op_reply(15 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f804c014040 con 0x559b75d51400 2023-10-05T19:28:36.364 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.358+0000 7f805c921700 1 -- 172.21.15.167:0/453671774 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:16 5.a 5:50000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x559b75d84970 con 0x7f8044007030 2023-10-05T19:28:36.365 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.358+0000 7f805c921700 1 -- 172.21.15.167:0/453671774 <== osd.0 v2:172.21.15.118:6800/27690 4 ==== osd_op_reply(16 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f804c022040 con 0x7f8044007030 2023-10-05T19:28:36.365 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.358+0000 7f805c921700 1 -- 172.21.15.167:0/453671774 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:17 5.1a 5:58000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x559b75d84970 con 0x559b75d51400 2023-10-05T19:28:36.365 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.358+0000 7f805c921700 1 -- 172.21.15.167:0/453671774 <== osd.1 v2:172.21.15.167:6800/30035 9 ==== osd_op_reply(17 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f804c022040 con 0x559b75d51400 2023-10-05T19:28:36.365 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.358+0000 7f805c921700 1 -- 172.21.15.167:0/453671774 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:18 5.6 5:60000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x559b75d84970 con 0x7f80540106e0 2023-10-05T19:28:36.366 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.358+0000 7f805b91f700 1 -- 172.21.15.167:0/453671774 <== osd.2 v2:172.21.15.167:6816/30037 3 ==== osd_op_reply(18 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f8044012450 con 0x7f80540106e0 2023-10-05T19:28:36.366 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.362+0000 7f805b91f700 1 -- 172.21.15.167:0/453671774 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:19 5.16 5:68000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x559b75d84970 con 0x559b75d82050 2023-10-05T19:28:36.366 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.362+0000 7f805c120700 1 -- 172.21.15.167:0/453671774 <== osd.3 v2:172.21.15.167:6808/30036 3 ==== osd_op_reply(19 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f8054025020 con 0x559b75d82050 2023-10-05T19:28:36.366 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.362+0000 7f805c120700 1 -- 172.21.15.167:0/453671774 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:20 5.e 5:70000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x559b75d84970 con 0x559b75d82050 2023-10-05T19:28:36.367 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.362+0000 7f805c120700 1 -- 172.21.15.167:0/453671774 <== osd.3 v2:172.21.15.167:6808/30036 4 ==== osd_op_reply(20 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f8054025020 con 0x559b75d82050 2023-10-05T19:28:36.367 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.362+0000 7f805c120700 1 -- 172.21.15.167:0/453671774 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:21 5.1e 5:78000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x559b75d84970 con 0x7f8044007030 2023-10-05T19:28:36.368 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.362+0000 7f805c921700 1 -- 172.21.15.167:0/453671774 <== osd.0 v2:172.21.15.118:6800/27690 5 ==== osd_op_reply(21 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f804c022040 con 0x7f8044007030 2023-10-05T19:28:36.368 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.362+0000 7f805c921700 1 -- 172.21.15.167:0/453671774 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:22 5.1 5:80000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x559b75d84970 con 0x559b75d82050 2023-10-05T19:28:36.368 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.362+0000 7f805c120700 1 -- 172.21.15.167:0/453671774 <== osd.3 v2:172.21.15.167:6808/30036 5 ==== osd_op_reply(22 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f8054025020 con 0x559b75d82050 2023-10-05T19:28:36.369 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.362+0000 7f805c120700 1 -- 172.21.15.167:0/453671774 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:23 5.11 5:88000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x559b75d84970 con 0x559b75d82050 2023-10-05T19:28:36.369 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.362+0000 7f805c120700 1 -- 172.21.15.167:0/453671774 <== osd.3 v2:172.21.15.167:6808/30036 6 ==== osd_op_reply(23 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f8054025020 con 0x559b75d82050 2023-10-05T19:28:36.369 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.362+0000 7f805c120700 1 -- 172.21.15.167:0/453671774 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:24 5.9 5:90000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x559b75d84970 con 0x559b75d51400 2023-10-05T19:28:36.370 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.366+0000 7f805c921700 1 -- 172.21.15.167:0/453671774 <== osd.1 v2:172.21.15.167:6800/30035 10 ==== osd_op_reply(24 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f804c022040 con 0x559b75d51400 2023-10-05T19:28:36.370 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.366+0000 7f805c921700 1 -- 172.21.15.167:0/453671774 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:25 5.19 5:98000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x559b75d84970 con 0x559b75d51400 2023-10-05T19:28:36.371 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.366+0000 7f805c921700 1 -- 172.21.15.167:0/453671774 <== osd.1 v2:172.21.15.167:6800/30035 11 ==== osd_op_reply(25 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f804c022040 con 0x559b75d51400 2023-10-05T19:28:36.371 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.366+0000 7f805c921700 1 -- 172.21.15.167:0/453671774 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:26 5.5 5:a0000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x559b75d84970 con 0x7f8044007030 2023-10-05T19:28:36.372 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.366+0000 7f805c921700 1 -- 172.21.15.167:0/453671774 <== osd.0 v2:172.21.15.118:6800/27690 6 ==== osd_op_reply(26 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f804c022040 con 0x7f8044007030 2023-10-05T19:28:36.372 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.366+0000 7f805c921700 1 -- 172.21.15.167:0/453671774 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:27 5.15 5:a8000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x559b75d84970 con 0x559b75d82050 2023-10-05T19:28:36.373 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.366+0000 7f805c120700 1 -- 172.21.15.167:0/453671774 <== osd.3 v2:172.21.15.167:6808/30036 7 ==== osd_op_reply(27 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f8054025020 con 0x559b75d82050 2023-10-05T19:28:36.373 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.366+0000 7f805c120700 1 -- 172.21.15.167:0/453671774 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:28 5.d 5:b0000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x559b75d84970 con 0x7f80540106e0 2023-10-05T19:28:36.373 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.366+0000 7f805b91f700 1 -- 172.21.15.167:0/453671774 <== osd.2 v2:172.21.15.167:6816/30037 4 ==== osd_op_reply(28 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f8044012450 con 0x7f80540106e0 2023-10-05T19:28:36.373 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.366+0000 7f805b91f700 1 -- 172.21.15.167:0/453671774 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:29 5.1d 5:b8000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x559b75d84970 con 0x559b75d51400 2023-10-05T19:28:36.374 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.370+0000 7f805c921700 1 -- 172.21.15.167:0/453671774 <== osd.1 v2:172.21.15.167:6800/30035 12 ==== osd_op_reply(29 [pgnls start_epoch 30 out=107b] v30'2 uv2 ondisk = 1) v8 ==== 144+0+107 (crc 0 0 0) 0x7f804c022040 con 0x559b75d51400 2023-10-05T19:28:36.374 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.370+0000 7f805c921700 1 -- 172.21.15.167:0/453671774 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:30 5.3 5:c0000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x559b75d84970 con 0x7f8044007030 2023-10-05T19:28:36.375 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.370+0000 7f805c921700 1 -- 172.21.15.167:0/453671774 <== osd.0 v2:172.21.15.118:6800/27690 7 ==== osd_op_reply(30 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f804c022040 con 0x7f8044007030 2023-10-05T19:28:36.375 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.370+0000 7f805c921700 1 -- 172.21.15.167:0/453671774 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:31 5.13 5:c8000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x559b75d84970 con 0x559b75d82050 2023-10-05T19:28:36.376 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.370+0000 7f805c120700 1 -- 172.21.15.167:0/453671774 <== osd.3 v2:172.21.15.167:6808/30036 8 ==== osd_op_reply(31 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f8054025020 con 0x559b75d82050 2023-10-05T19:28:36.376 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.370+0000 7f805c120700 1 -- 172.21.15.167:0/453671774 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:32 5.b 5:d0000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x559b75d84970 con 0x7f8044007030 2023-10-05T19:28:36.377 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.370+0000 7f805c921700 1 -- 172.21.15.167:0/453671774 <== osd.0 v2:172.21.15.118:6800/27690 8 ==== osd_op_reply(32 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f804c022040 con 0x7f8044007030 2023-10-05T19:28:36.377 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.370+0000 7f805c921700 1 -- 172.21.15.167:0/453671774 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:33 5.1b 5:d8000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x559b75d84970 con 0x7f80540106e0 2023-10-05T19:28:36.378 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.370+0000 7f805b91f700 1 -- 172.21.15.167:0/453671774 <== osd.2 v2:172.21.15.167:6816/30037 5 ==== osd_op_reply(33 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f8044012450 con 0x7f80540106e0 2023-10-05T19:28:36.378 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.370+0000 7f805b91f700 1 -- 172.21.15.167:0/453671774 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:34 5.7 5:e0000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x559b75d84970 con 0x559b75d82050 2023-10-05T19:28:36.378 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.374+0000 7f805c120700 1 -- 172.21.15.167:0/453671774 <== osd.3 v2:172.21.15.167:6808/30036 9 ==== osd_op_reply(34 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f8054025020 con 0x559b75d82050 2023-10-05T19:28:36.378 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.374+0000 7f805c120700 1 -- 172.21.15.167:0/453671774 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:35 5.17 5:e8000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x559b75d84970 con 0x7f8044007030 2023-10-05T19:28:36.379 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.374+0000 7f805c921700 1 -- 172.21.15.167:0/453671774 <== osd.0 v2:172.21.15.118:6800/27690 9 ==== osd_op_reply(35 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f804c022040 con 0x7f8044007030 2023-10-05T19:28:36.380 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.374+0000 7f805c921700 1 -- 172.21.15.167:0/453671774 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:36 5.f 5:f0000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x559b75d84970 con 0x559b75d51400 2023-10-05T19:28:36.380 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.374+0000 7f805c921700 1 -- 172.21.15.167:0/453671774 <== osd.1 v2:172.21.15.167:6800/30035 13 ==== osd_op_reply(36 [pgnls start_epoch 30 out=115b] v23'2 uv2 ondisk = 1) v8 ==== 144+0+115 (crc 0 0 0) 0x7f804c02f080 con 0x559b75d51400 2023-10-05T19:28:36.380 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.374+0000 7f805c921700 1 -- 172.21.15.167:0/453671774 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:37 5.1f 5:f8000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x559b75d84970 con 0x7f80540106e0 2023-10-05T19:28:36.381 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.374+0000 7f805b91f700 1 -- 172.21.15.167:0/453671774 <== osd.2 v2:172.21.15.167:6816/30037 6 ==== osd_op_reply(37 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f8044012450 con 0x7f80540106e0 2023-10-05T19:28:36.381 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.374+0000 7f805e5ebe40 20 RGWRados::pool_iterate: got zonegroup_info.4f1f719d-9648-417b-b10c-e5d939825419 2023-10-05T19:28:36.381 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.374+0000 7f805e5ebe40 20 RGWRados::pool_iterate: got zone_info.9a9df965-6612-45d4-a45f-6e3c889bacc2 2023-10-05T19:28:36.381 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.374+0000 7f805e5ebe40 20 RGWRados::pool_iterate: got zone_names.default 2023-10-05T19:28:36.381 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.374+0000 7f805e5ebe40 20 RGWRados::pool_iterate: got zonegroups_names.default 2023-10-05T19:28:36.382 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.374+0000 7f805e5ebe40 20 rados->read ofs=0 len=0 2023-10-05T19:28:36.382 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.374+0000 7f805e5ebe40 1 -- 172.21.15.167:0/453671774 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:38 5.12 5:49953fa1:::default.realm:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x559b75d80e00 con 0x559b75d51400 2023-10-05T19:28:36.382 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.374+0000 7f805c921700 1 -- 172.21.15.167:0/453671774 <== osd.1 v2:172.21.15.167:6800/30035 14 ==== osd_op_reply(38 default.realm [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 ==== 157+0+0 (crc 0 0 0) 0x7f804c02f080 con 0x559b75d51400 2023-10-05T19:28:36.383 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.374+0000 7f805e5ebe40 20 rados_obj.operate() r=-2 bl.length=0 2023-10-05T19:28:36.383 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.374+0000 7f805e5ebe40 20 rados->read ofs=0 len=0 2023-10-05T19:28:36.383 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.374+0000 7f805e5ebe40 1 -- 172.21.15.167:0/453671774 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:39 5.f 5:f4c53578:::zonegroups_names.default:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x559b75d81780 con 0x559b75d51400 2023-10-05T19:28:36.383 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.378+0000 7f805c921700 1 -- 172.21.15.167:0/453671774 <== osd.1 v2:172.21.15.167:6800/30035 15 ==== osd_op_reply(39 zonegroups_names.default [read 0~46 out=46b] v0'0 uv2 ondisk = 0) v8 ==== 168+0+46 (crc 0 0 0) 0x7f804c02f080 con 0x559b75d51400 2023-10-05T19:28:36.383 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.378+0000 7f805e5ebe40 20 rados_obj.operate() r=0 bl.length=46 2023-10-05T19:28:36.383 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.378+0000 7f805e5ebe40 20 rados->read ofs=0 len=0 2023-10-05T19:28:36.383 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.378+0000 7f805e5ebe40 1 -- 172.21.15.167:0/453671774 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:40 5.c 5:30d299a5:::zonegroup_info.4f1f719d-9648-417b-b10c-e5d939825419:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x559b75d809f0 con 0x7f8044007030 2023-10-05T19:28:36.384 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.378+0000 7f805c921700 1 -- 172.21.15.167:0/453671774 <== osd.0 v2:172.21.15.118:6800/27690 10 ==== osd_op_reply(40 zonegroup_info.4f1f719d-9648-417b-b10c-e5d939825419 [read 0~362 out=362b] v0'0 uv1 ondisk = 0) v8 ==== 195+0+362 (crc 0 0 0) 0x7f804c02f080 con 0x7f8044007030 2023-10-05T19:28:36.384 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.378+0000 7f805e5ebe40 20 rados_obj.operate() r=0 bl.length=362 2023-10-05T19:28:36.384 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.378+0000 7f805e5ebe40 20 rados->read ofs=0 len=0 2023-10-05T19:28:36.384 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.378+0000 7f805e5ebe40 1 -- 172.21.15.167:0/453671774 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:41 5.13 5:cbbf6c60:::converted:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x559b75d85260 con 0x559b75d82050 2023-10-05T19:28:36.384 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.378+0000 7f805c120700 1 -- 172.21.15.167:0/453671774 <== osd.3 v2:172.21.15.167:6808/30036 10 ==== osd_op_reply(41 converted [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 ==== 153+0+0 (crc 0 0 0) 0x7f8054025020 con 0x559b75d82050 2023-10-05T19:28:36.384 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.378+0000 7f805e5ebe40 20 rados_obj.operate() r=-2 bl.length=0 2023-10-05T19:28:36.385 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.378+0000 7f805e5ebe40 20 rados->read ofs=0 len=0 2023-10-05T19:28:36.385 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.378+0000 7f805e5ebe40 1 -- 172.21.15.167:0/453671774 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:42 5.12 5:49953fa1:::default.realm:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x559b75d85610 con 0x559b75d51400 2023-10-05T19:28:36.385 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.378+0000 7f805c921700 1 -- 172.21.15.167:0/453671774 <== osd.1 v2:172.21.15.167:6800/30035 16 ==== osd_op_reply(42 default.realm [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 ==== 157+0+0 (crc 0 0 0) 0x7f804c02f080 con 0x559b75d51400 2023-10-05T19:28:36.385 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.378+0000 7f805e5ebe40 20 rados_obj.operate() r=-2 bl.length=0 2023-10-05T19:28:36.386 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.378+0000 7f805e5ebe40 20 rados->read ofs=0 len=0 2023-10-05T19:28:36.386 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.378+0000 7f805e5ebe40 1 -- 172.21.15.167:0/453671774 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:43 5.f 5:f4c53578:::zonegroups_names.default:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x559b75d85610 con 0x559b75d51400 2023-10-05T19:28:36.386 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.378+0000 7f805c921700 1 -- 172.21.15.167:0/453671774 <== osd.1 v2:172.21.15.167:6800/30035 17 ==== osd_op_reply(43 zonegroups_names.default [read 0~46 out=46b] v0'0 uv2 ondisk = 0) v8 ==== 168+0+46 (crc 0 0 0) 0x7f804c02f080 con 0x559b75d51400 2023-10-05T19:28:36.386 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.378+0000 7f805e5ebe40 20 rados_obj.operate() r=0 bl.length=46 2023-10-05T19:28:36.386 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.378+0000 7f805e5ebe40 1 -- 172.21.15.167:0/453671774 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:44 5.0 5:00000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x559b75d86620 con 0x559b75d82050 2023-10-05T19:28:36.388 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.382+0000 7f805c120700 1 -- 172.21.15.167:0/453671774 <== osd.3 v2:172.21.15.167:6808/30036 11 ==== osd_op_reply(44 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f8054025020 con 0x559b75d82050 2023-10-05T19:28:36.388 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.382+0000 7f805c120700 1 -- 172.21.15.167:0/453671774 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:45 5.10 5:08000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x7f8054048040 con 0x7f80540106e0 2023-10-05T19:28:36.388 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.382+0000 7f805b91f700 1 -- 172.21.15.167:0/453671774 <== osd.2 v2:172.21.15.167:6816/30037 7 ==== osd_op_reply(45 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f8044012450 con 0x7f80540106e0 2023-10-05T19:28:36.388 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.382+0000 7f805b91f700 1 -- 172.21.15.167:0/453671774 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:46 5.8 5:10000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x7f8054048040 con 0x7f8044007030 2023-10-05T19:28:36.389 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.382+0000 7f805c921700 1 -- 172.21.15.167:0/453671774 <== osd.0 v2:172.21.15.118:6800/27690 11 ==== osd_op_reply(46 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f804c02f080 con 0x7f8044007030 2023-10-05T19:28:36.390 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.382+0000 7f805c921700 1 -- 172.21.15.167:0/453671774 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:47 5.18 5:18000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x559b75d809f0 con 0x559b75d51400 2023-10-05T19:28:36.390 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.386+0000 7f805c921700 1 -- 172.21.15.167:0/453671774 <== osd.1 v2:172.21.15.167:6800/30035 18 ==== osd_op_reply(47 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f804c02f080 con 0x559b75d51400 2023-10-05T19:28:36.390 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.386+0000 7f805c921700 1 -- 172.21.15.167:0/453671774 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:48 5.4 5:20000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x559b75d809f0 con 0x559b75d51400 2023-10-05T19:28:36.391 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.386+0000 7f805c921700 1 -- 172.21.15.167:0/453671774 <== osd.1 v2:172.21.15.167:6800/30035 19 ==== osd_op_reply(48 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f804c02f080 con 0x559b75d51400 2023-10-05T19:28:36.391 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.386+0000 7f805c921700 1 -- 172.21.15.167:0/453671774 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:49 5.14 5:28000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x559b75d809f0 con 0x7f8044007030 2023-10-05T19:28:36.392 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.386+0000 7f805c921700 1 -- 172.21.15.167:0/453671774 <== osd.0 v2:172.21.15.118:6800/27690 12 ==== osd_op_reply(49 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f804c02f080 con 0x7f8044007030 2023-10-05T19:28:36.392 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.386+0000 7f805c921700 1 -- 172.21.15.167:0/453671774 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:50 5.c 5:30000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x559b75d809f0 con 0x7f8044007030 2023-10-05T19:28:36.393 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.386+0000 7f805c921700 1 -- 172.21.15.167:0/453671774 <== osd.0 v2:172.21.15.118:6800/27690 13 ==== osd_op_reply(50 [pgnls start_epoch 30 out=112b] v23'1 uv1 ondisk = 1) v8 ==== 144+0+112 (crc 0 0 0) 0x7f804c02f080 con 0x7f8044007030 2023-10-05T19:28:36.393 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.386+0000 7f805c921700 1 -- 172.21.15.167:0/453671774 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:51 5.1c 5:38000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x559b75d809f0 con 0x7f80540106e0 2023-10-05T19:28:36.393 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.386+0000 7f805b91f700 1 -- 172.21.15.167:0/453671774 <== osd.2 v2:172.21.15.167:6816/30037 8 ==== osd_op_reply(51 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f8044012450 con 0x7f80540106e0 2023-10-05T19:28:36.394 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.386+0000 7f805b91f700 1 -- 172.21.15.167:0/453671774 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:52 5.2 5:40000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x559b75d809f0 con 0x559b75d82050 2023-10-05T19:28:36.394 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.390+0000 7f805c120700 1 -- 172.21.15.167:0/453671774 <== osd.3 v2:172.21.15.167:6808/30036 12 ==== osd_op_reply(52 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f8054025020 con 0x559b75d82050 2023-10-05T19:28:36.395 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.390+0000 7f805c120700 1 -- 172.21.15.167:0/453671774 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:53 5.12 5:48000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x559b75d809f0 con 0x559b75d51400 2023-10-05T19:28:36.395 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.390+0000 7f805c921700 1 -- 172.21.15.167:0/453671774 <== osd.1 v2:172.21.15.167:6800/30035 20 ==== osd_op_reply(53 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f804c02f080 con 0x559b75d51400 2023-10-05T19:28:36.395 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.390+0000 7f805c921700 1 -- 172.21.15.167:0/453671774 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:54 5.a 5:50000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x559b75d809f0 con 0x7f8044007030 2023-10-05T19:28:36.396 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.390+0000 7f805c921700 1 -- 172.21.15.167:0/453671774 <== osd.0 v2:172.21.15.118:6800/27690 14 ==== osd_op_reply(54 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f804c02f080 con 0x7f8044007030 2023-10-05T19:28:36.396 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.390+0000 7f805c921700 1 -- 172.21.15.167:0/453671774 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:55 5.1a 5:58000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x559b75d809f0 con 0x559b75d51400 2023-10-05T19:28:36.397 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.390+0000 7f805c921700 1 -- 172.21.15.167:0/453671774 <== osd.1 v2:172.21.15.167:6800/30035 21 ==== osd_op_reply(55 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f804c02f080 con 0x559b75d51400 2023-10-05T19:28:36.397 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.390+0000 7f805c921700 1 -- 172.21.15.167:0/453671774 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:56 5.6 5:60000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x559b75d809f0 con 0x7f80540106e0 2023-10-05T19:28:36.398 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.394+0000 7f805b91f700 1 -- 172.21.15.167:0/453671774 <== osd.2 v2:172.21.15.167:6816/30037 9 ==== osd_op_reply(56 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f8044012450 con 0x7f80540106e0 2023-10-05T19:28:36.398 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.394+0000 7f805b91f700 1 -- 172.21.15.167:0/453671774 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:57 5.16 5:68000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x559b75d809f0 con 0x559b75d82050 2023-10-05T19:28:36.399 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.394+0000 7f805c120700 1 -- 172.21.15.167:0/453671774 <== osd.3 v2:172.21.15.167:6808/30036 13 ==== osd_op_reply(57 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f8054025020 con 0x559b75d82050 2023-10-05T19:28:36.399 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.394+0000 7f805c120700 1 -- 172.21.15.167:0/453671774 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:58 5.e 5:70000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x559b75d809f0 con 0x559b75d82050 2023-10-05T19:28:36.399 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.394+0000 7f805c120700 1 -- 172.21.15.167:0/453671774 <== osd.3 v2:172.21.15.167:6808/30036 14 ==== osd_op_reply(58 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f8054025020 con 0x559b75d82050 2023-10-05T19:28:36.399 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.394+0000 7f805c120700 1 -- 172.21.15.167:0/453671774 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:59 5.1e 5:78000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x559b75d809f0 con 0x7f8044007030 2023-10-05T19:28:36.400 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.394+0000 7f805c921700 1 -- 172.21.15.167:0/453671774 <== osd.0 v2:172.21.15.118:6800/27690 15 ==== osd_op_reply(59 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f804c02f080 con 0x7f8044007030 2023-10-05T19:28:36.401 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.394+0000 7f805c921700 1 -- 172.21.15.167:0/453671774 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:60 5.1 5:80000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x559b75d809f0 con 0x559b75d82050 2023-10-05T19:28:36.401 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.394+0000 7f805c120700 1 -- 172.21.15.167:0/453671774 <== osd.3 v2:172.21.15.167:6808/30036 15 ==== osd_op_reply(60 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f8054025020 con 0x559b75d82050 2023-10-05T19:28:36.402 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.394+0000 7f805c120700 1 -- 172.21.15.167:0/453671774 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:61 5.11 5:88000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x559b75d809f0 con 0x559b75d82050 2023-10-05T19:28:36.402 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.398+0000 7f805c120700 1 -- 172.21.15.167:0/453671774 <== osd.3 v2:172.21.15.167:6808/30036 16 ==== osd_op_reply(61 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f8054025020 con 0x559b75d82050 2023-10-05T19:28:36.402 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.398+0000 7f805c120700 1 -- 172.21.15.167:0/453671774 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:62 5.9 5:90000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x559b75d809f0 con 0x559b75d51400 2023-10-05T19:28:36.403 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.398+0000 7f805c921700 1 -- 172.21.15.167:0/453671774 <== osd.1 v2:172.21.15.167:6800/30035 22 ==== osd_op_reply(62 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f804c02f080 con 0x559b75d51400 2023-10-05T19:28:36.403 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.398+0000 7f805c921700 1 -- 172.21.15.167:0/453671774 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:63 5.19 5:98000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x559b75d809f0 con 0x559b75d51400 2023-10-05T19:28:36.404 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.398+0000 7f805c921700 1 -- 172.21.15.167:0/453671774 <== osd.1 v2:172.21.15.167:6800/30035 23 ==== osd_op_reply(63 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f804c02f080 con 0x559b75d51400 2023-10-05T19:28:36.404 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.398+0000 7f805c921700 1 -- 172.21.15.167:0/453671774 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:64 5.5 5:a0000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x559b75d809f0 con 0x7f8044007030 2023-10-05T19:28:36.405 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.398+0000 7f805c921700 1 -- 172.21.15.167:0/453671774 <== osd.0 v2:172.21.15.118:6800/27690 16 ==== osd_op_reply(64 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f804c02f080 con 0x7f8044007030 2023-10-05T19:28:36.405 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.398+0000 7f805c921700 1 -- 172.21.15.167:0/453671774 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:65 5.15 5:a8000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x559b75d809f0 con 0x559b75d82050 2023-10-05T19:28:36.406 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.402+0000 7f805c120700 1 -- 172.21.15.167:0/453671774 <== osd.3 v2:172.21.15.167:6808/30036 17 ==== osd_op_reply(65 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f8054025020 con 0x559b75d82050 2023-10-05T19:28:36.406 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.402+0000 7f805c120700 1 -- 172.21.15.167:0/453671774 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:66 5.d 5:b0000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x559b75d809f0 con 0x7f80540106e0 2023-10-05T19:28:36.406 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.402+0000 7f805b91f700 1 -- 172.21.15.167:0/453671774 <== osd.2 v2:172.21.15.167:6816/30037 10 ==== osd_op_reply(66 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f8044012450 con 0x7f80540106e0 2023-10-05T19:28:36.407 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.402+0000 7f805b91f700 1 -- 172.21.15.167:0/453671774 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:67 5.1d 5:b8000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x559b75d809f0 con 0x559b75d51400 2023-10-05T19:28:36.407 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.402+0000 7f805c921700 1 -- 172.21.15.167:0/453671774 <== osd.1 v2:172.21.15.167:6800/30035 24 ==== osd_op_reply(67 [pgnls start_epoch 30 out=107b] v30'2 uv2 ondisk = 1) v8 ==== 144+0+107 (crc 0 0 0) 0x7f804c02f080 con 0x559b75d51400 2023-10-05T19:28:36.407 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.402+0000 7f805c921700 1 -- 172.21.15.167:0/453671774 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:68 5.3 5:c0000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x559b75d809f0 con 0x7f8044007030 2023-10-05T19:28:36.408 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.402+0000 7f805c921700 1 -- 172.21.15.167:0/453671774 <== osd.0 v2:172.21.15.118:6800/27690 17 ==== osd_op_reply(68 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f804c02f080 con 0x7f8044007030 2023-10-05T19:28:36.408 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.402+0000 7f805c921700 1 -- 172.21.15.167:0/453671774 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:69 5.13 5:c8000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x559b75d809f0 con 0x559b75d82050 2023-10-05T19:28:36.409 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.402+0000 7f805c120700 1 -- 172.21.15.167:0/453671774 <== osd.3 v2:172.21.15.167:6808/30036 18 ==== osd_op_reply(69 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f8054025020 con 0x559b75d82050 2023-10-05T19:28:36.409 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.402+0000 7f805c120700 1 -- 172.21.15.167:0/453671774 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:70 5.b 5:d0000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x559b75d809f0 con 0x7f8044007030 2023-10-05T19:28:36.410 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.406+0000 7f805c921700 1 -- 172.21.15.167:0/453671774 <== osd.0 v2:172.21.15.118:6800/27690 18 ==== osd_op_reply(70 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f804c02f080 con 0x7f8044007030 2023-10-05T19:28:36.410 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.406+0000 7f805c921700 1 -- 172.21.15.167:0/453671774 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:71 5.1b 5:d8000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x559b75d809f0 con 0x7f80540106e0 2023-10-05T19:28:36.411 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.406+0000 7f805b91f700 1 -- 172.21.15.167:0/453671774 <== osd.2 v2:172.21.15.167:6816/30037 11 ==== osd_op_reply(71 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f8044012450 con 0x7f80540106e0 2023-10-05T19:28:36.411 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.406+0000 7f805b91f700 1 -- 172.21.15.167:0/453671774 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:72 5.7 5:e0000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x559b75d809f0 con 0x559b75d82050 2023-10-05T19:28:36.412 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.406+0000 7f805c120700 1 -- 172.21.15.167:0/453671774 <== osd.3 v2:172.21.15.167:6808/30036 19 ==== osd_op_reply(72 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f8054025020 con 0x559b75d82050 2023-10-05T19:28:36.412 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.406+0000 7f805c120700 1 -- 172.21.15.167:0/453671774 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:73 5.17 5:e8000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x559b75d809f0 con 0x7f8044007030 2023-10-05T19:28:36.412 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.406+0000 7f805c921700 1 -- 172.21.15.167:0/453671774 <== osd.0 v2:172.21.15.118:6800/27690 19 ==== osd_op_reply(73 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f804c02f080 con 0x7f8044007030 2023-10-05T19:28:36.412 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.406+0000 7f805c921700 1 -- 172.21.15.167:0/453671774 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:74 5.f 5:f0000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x559b75d809f0 con 0x559b75d51400 2023-10-05T19:28:36.413 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.406+0000 7f805c921700 1 -- 172.21.15.167:0/453671774 <== osd.1 v2:172.21.15.167:6800/30035 25 ==== osd_op_reply(74 [pgnls start_epoch 30 out=115b] v23'2 uv2 ondisk = 1) v8 ==== 144+0+115 (crc 0 0 0) 0x7f804c02f080 con 0x559b75d51400 2023-10-05T19:28:36.413 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.406+0000 7f805c921700 1 -- 172.21.15.167:0/453671774 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:75 5.1f 5:f8000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x559b75d809f0 con 0x7f80540106e0 2023-10-05T19:28:36.414 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.410+0000 7f805b91f700 1 -- 172.21.15.167:0/453671774 <== osd.2 v2:172.21.15.167:6816/30037 12 ==== osd_op_reply(75 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f8044012450 con 0x7f80540106e0 2023-10-05T19:28:36.415 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.410+0000 7f805e5ebe40 20 RGWRados::pool_iterate: got zonegroup_info.4f1f719d-9648-417b-b10c-e5d939825419 2023-10-05T19:28:36.415 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.410+0000 7f805e5ebe40 20 RGWRados::pool_iterate: got zone_info.9a9df965-6612-45d4-a45f-6e3c889bacc2 2023-10-05T19:28:36.415 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.410+0000 7f805e5ebe40 20 RGWRados::pool_iterate: got zone_names.default 2023-10-05T19:28:36.415 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.410+0000 7f805e5ebe40 20 RGWRados::pool_iterate: got zonegroups_names.default 2023-10-05T19:28:36.415 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.410+0000 7f805e5ebe40 20 rados->read ofs=0 len=0 2023-10-05T19:28:36.415 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.410+0000 7f805e5ebe40 1 -- 172.21.15.167:0/453671774 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:76 5.f 5:f43b6ece:::zone_names.default:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x559b75d85610 con 0x559b75d51400 2023-10-05T19:28:36.416 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.410+0000 7f805c921700 1 -- 172.21.15.167:0/453671774 <== osd.1 v2:172.21.15.167:6800/30035 26 ==== osd_op_reply(76 zone_names.default [read 0~46 out=46b] v0'0 uv1 ondisk = 0) v8 ==== 162+0+46 (crc 0 0 0) 0x7f804c02f080 con 0x559b75d51400 2023-10-05T19:28:36.416 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.410+0000 7f805e5ebe40 20 rados_obj.operate() r=0 bl.length=46 2023-10-05T19:28:36.416 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.410+0000 7f805e5ebe40 20 rados->read ofs=0 len=0 2023-10-05T19:28:36.416 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.410+0000 7f805e5ebe40 1 -- 172.21.15.167:0/453671774 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:77 5.1d 5:b8374eb4:::zone_info.9a9df965-6612-45d4-a45f-6e3c889bacc2:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x559b75d859c0 con 0x559b75d51400 2023-10-05T19:28:36.416 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.410+0000 7f805c921700 1 -- 172.21.15.167:0/453671774 <== osd.1 v2:172.21.15.167:6800/30035 27 ==== osd_op_reply(77 zone_info.9a9df965-6612-45d4-a45f-6e3c889bacc2 [read 0~874 out=874b] v0'0 uv2 ondisk = 0) v8 ==== 190+0+874 (crc 0 0 0) 0x7f804c02f080 con 0x559b75d51400 2023-10-05T19:28:36.417 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.410+0000 7f805e5ebe40 20 rados_obj.operate() r=0 bl.length=874 2023-10-05T19:28:36.417 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.410+0000 7f805e5ebe40 20 rados->read ofs=0 len=0 2023-10-05T19:28:36.417 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.410+0000 7f805e5ebe40 1 -- 172.21.15.167:0/453671774 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:78 5.f 5:f4c53578:::zonegroups_names.default:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x559b75d85e00 con 0x559b75d51400 2023-10-05T19:28:36.418 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.410+0000 7f805c921700 1 -- 172.21.15.167:0/453671774 <== osd.1 v2:172.21.15.167:6800/30035 28 ==== osd_op_reply(78 zonegroups_names.default [read 0~46 out=46b] v0'0 uv2 ondisk = 0) v8 ==== 168+0+46 (crc 0 0 0) 0x7f804c02f080 con 0x559b75d51400 2023-10-05T19:28:36.418 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.410+0000 7f805e5ebe40 20 rados_obj.operate() r=0 bl.length=46 2023-10-05T19:28:36.418 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.410+0000 7f805e5ebe40 20 rados->read ofs=0 len=0 2023-10-05T19:28:36.418 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.410+0000 7f805e5ebe40 1 -- 172.21.15.167:0/453671774 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:79 5.c 5:30d299a5:::zonegroup_info.4f1f719d-9648-417b-b10c-e5d939825419:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x559b75d85e00 con 0x7f8044007030 2023-10-05T19:28:36.418 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.410+0000 7f805c921700 1 -- 172.21.15.167:0/453671774 <== osd.0 v2:172.21.15.118:6800/27690 20 ==== osd_op_reply(79 zonegroup_info.4f1f719d-9648-417b-b10c-e5d939825419 [read 0~362 out=362b] v0'0 uv1 ondisk = 0) v8 ==== 195+0+362 (crc 0 0 0) 0x7f804c02f080 con 0x7f8044007030 2023-10-05T19:28:36.418 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.410+0000 7f805e5ebe40 20 rados_obj.operate() r=0 bl.length=362 2023-10-05T19:28:36.418 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.410+0000 7f805e5ebe40 20 rados->read ofs=0 len=0 2023-10-05T19:28:36.419 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.410+0000 7f805e5ebe40 1 -- 172.21.15.167:0/453671774 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:80 5.3 5:c0ad8cbd:::region_map:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x559b75d85e30 con 0x7f8044007030 2023-10-05T19:28:36.419 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.414+0000 7f805c921700 1 -- 172.21.15.167:0/453671774 <== osd.0 v2:172.21.15.118:6800/27690 21 ==== osd_op_reply(80 region_map [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 ==== 154+0+0 (crc 0 0 0) 0x7f804c050040 con 0x7f8044007030 2023-10-05T19:28:36.420 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.414+0000 7f805e5ebe40 20 rados_obj.operate() r=-2 bl.length=0 2023-10-05T19:28:36.420 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.414+0000 7f805e5ebe40 20 zone default found 2023-10-05T19:28:36.420 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.414+0000 7f805e5ebe40 4 Realm: () 2023-10-05T19:28:36.420 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.414+0000 7f805e5ebe40 4 ZoneGroup: default (4f1f719d-9648-417b-b10c-e5d939825419) 2023-10-05T19:28:36.420 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.414+0000 7f805e5ebe40 4 Zone: default (9a9df965-6612-45d4-a45f-6e3c889bacc2) 2023-10-05T19:28:36.420 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.414+0000 7f805e5ebe40 10 cannot find current period zonegroup using local zonegroup configuration 2023-10-05T19:28:36.420 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.414+0000 7f805e5ebe40 20 zonegroup default 2023-10-05T19:28:36.420 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.414+0000 7f805e5ebe40 20 rados->read ofs=0 len=0 2023-10-05T19:28:36.420 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.414+0000 7f805e5ebe40 1 -- 172.21.15.167:0/453671774 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:81 5.3 5:c52100b6:::period_config.default:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x559b75d85e30 con 0x7f8044007030 2023-10-05T19:28:36.420 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.414+0000 7f805c921700 1 -- 172.21.15.167:0/453671774 <== osd.0 v2:172.21.15.118:6800/27690 22 ==== osd_op_reply(81 period_config.default [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 ==== 165+0+0 (crc 0 0 0) 0x7f804c050040 con 0x7f8044007030 2023-10-05T19:28:36.420 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.414+0000 7f805e5ebe40 20 rados_obj.operate() r=-2 bl.length=0 2023-10-05T19:28:36.420 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.414+0000 7f805e5ebe40 20 rados->read ofs=0 len=0 2023-10-05T19:28:36.421 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.414+0000 7f805e5ebe40 1 -- 172.21.15.167:0/453671774 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:82 6.2 6:4347d321:::bucket.sync-source-hints.:head [call version.read in=11b,read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x559b75d85e30 con 0x559b75d82050 2023-10-05T19:28:36.421 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.414+0000 7f805c120700 1 -- 172.21.15.167:0/453671774 <== osd.3 v2:172.21.15.167:6808/30036 20 ==== osd_op_reply(82 bucket.sync-source-hints. [call,read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 ==== 211+0+0 (crc 0 0 0) 0x7f8054025020 con 0x559b75d82050 2023-10-05T19:28:36.422 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.414+0000 7f805e5ebe40 20 rados_obj.operate() r=-2 bl.length=0 2023-10-05T19:28:36.422 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.414+0000 7f805e5ebe40 20 rados->read ofs=0 len=0 2023-10-05T19:28:36.422 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.414+0000 7f805e5ebe40 1 -- 172.21.15.167:0/453671774 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:83 6.b 6:d467b91b:::bucket.sync-target-hints.:head [call version.read in=11b,read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x559b75d86230 con 0x559b75d82050 2023-10-05T19:28:36.422 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.414+0000 7f805c120700 1 -- 172.21.15.167:0/453671774 <== osd.3 v2:172.21.15.167:6808/30036 21 ==== osd_op_reply(83 bucket.sync-target-hints. [call,read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 ==== 211+0+0 (crc 0 0 0) 0x7f8054025020 con 0x559b75d82050 2023-10-05T19:28:36.422 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.414+0000 7f805e5ebe40 20 rados_obj.operate() r=-2 bl.length=0 2023-10-05T19:28:36.422 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.414+0000 7f805e5ebe40 20 started sync module instance, tier type = 2023-10-05T19:28:36.422 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.414+0000 7f805e5ebe40 20 started zone id=9a9df965-6612-45d4-a45f-6e3c889bacc2 (name=default) with tier type = 2023-10-05T19:28:36.422 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.414+0000 7f805e5ebe40 1 -- 172.21.15.167:0/453671774 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:84 7.1f 7:f95f44c2:::notify.0:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x559b75d87a10 con 0x7f80540106e0 2023-10-05T19:28:36.424 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.418+0000 7f805b91f700 1 -- 172.21.15.167:0/453671774 <== osd.2 v2:172.21.15.167:6816/30037 13 ==== osd_op_reply(84 notify.0 [create] v30'153 uv153 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f8044012450 con 0x7f80540106e0 2023-10-05T19:28:36.424 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.418+0000 7f805e5ebe40 1 -- 172.21.15.167:0/453671774 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:85 7.1f 7:f95f44c2:::notify.0:head [watch watch cookie 94126185414080] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x559b75d84e90 con 0x7f80540106e0 2023-10-05T19:28:36.424 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.418+0000 7f805e5ebe40 1 -- 172.21.15.167:0/453671774 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:86 7.0 7:05bf5b68:::notify.1:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x559b75d88d60 con 0x559b75d51400 2023-10-05T19:28:36.427 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.422+0000 7f805c921700 1 -- 172.21.15.167:0/453671774 <== osd.1 v2:172.21.15.167:6800/30035 29 ==== osd_op_reply(86 notify.1 [create] v30'153 uv153 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f804c050040 con 0x559b75d51400 2023-10-05T19:28:36.427 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.422+0000 7f805e5ebe40 1 -- 172.21.15.167:0/453671774 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:87 7.0 7:05bf5b68:::notify.1:head [watch watch cookie 94126185417104] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x559b75d89650 con 0x559b75d51400 2023-10-05T19:28:36.427 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.422+0000 7f805e5ebe40 1 -- 172.21.15.167:0/453671774 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:88 7.15 7:a93a5511:::notify.2:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x559b75d89ef0 con 0x559b75d82050 2023-10-05T19:28:36.428 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.422+0000 7f805b91f700 1 -- 172.21.15.167:0/453671774 <== osd.2 v2:172.21.15.167:6816/30037 14 ==== osd_op_reply(85 notify.0 [watch watch cookie 94126185414080] v30'154 uv153 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f8044012450 con 0x7f80540106e0 2023-10-05T19:28:36.430 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.422+0000 7f805c120700 1 -- 172.21.15.167:0/453671774 <== osd.3 v2:172.21.15.167:6808/30036 22 ==== osd_op_reply(88 notify.2 [create] v30'153 uv153 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f8054025020 con 0x559b75d82050 2023-10-05T19:28:36.430 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.422+0000 7f805e5ebe40 1 -- 172.21.15.167:0/453671774 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:89 7.15 7:a93a5511:::notify.2:head [watch watch cookie 94126185420368] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x559b75d88330 con 0x559b75d82050 2023-10-05T19:28:36.430 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.422+0000 7f805e5ebe40 1 -- 172.21.15.167:0/453671774 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:90 7.e 7:7759931f:::notify.3:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x559b75d8ac80 con 0x7f80540106e0 2023-10-05T19:28:36.430 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.426+0000 7f805c921700 1 -- 172.21.15.167:0/453671774 <== osd.1 v2:172.21.15.167:6800/30035 30 ==== osd_op_reply(87 notify.1 [watch watch cookie 94126185417104] v30'154 uv153 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f804c050040 con 0x559b75d51400 2023-10-05T19:28:36.432 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.426+0000 7f805b91f700 1 -- 172.21.15.167:0/453671774 <== osd.2 v2:172.21.15.167:6816/30037 15 ==== osd_op_reply(90 notify.3 [create] v30'153 uv153 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f8044012450 con 0x7f80540106e0 2023-10-05T19:28:36.432 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.426+0000 7f805e5ebe40 1 -- 172.21.15.167:0/453671774 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:91 7.e 7:7759931f:::notify.3:head [watch watch cookie 94126185419104] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x559b75d8a8b0 con 0x7f80540106e0 2023-10-05T19:28:36.432 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.426+0000 7f805e5ebe40 1 -- 172.21.15.167:0/453671774 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:92 7.d 7:b4812045:::notify.4:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x559b75d8bbc0 con 0x559b75d51400 2023-10-05T19:28:36.433 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.426+0000 7f805c120700 1 -- 172.21.15.167:0/453671774 <== osd.3 v2:172.21.15.167:6808/30036 23 ==== osd_op_reply(89 notify.2 [watch watch cookie 94126185420368] v30'154 uv153 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f8054025020 con 0x559b75d82050 2023-10-05T19:28:36.434 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.430+0000 7f805b91f700 1 -- 172.21.15.167:0/453671774 <== osd.2 v2:172.21.15.167:6816/30037 16 ==== osd_op_reply(91 notify.3 [watch watch cookie 94126185419104] v30'154 uv153 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f8044012450 con 0x7f80540106e0 2023-10-05T19:28:36.435 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.430+0000 7f805c921700 1 -- 172.21.15.167:0/453671774 <== osd.1 v2:172.21.15.167:6800/30035 31 ==== osd_op_reply(92 notify.4 [create] v30'153 uv153 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f804c050040 con 0x559b75d51400 2023-10-05T19:28:36.435 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.430+0000 7f805e5ebe40 1 -- 172.21.15.167:0/453671774 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:93 7.d 7:b4812045:::notify.4:head [watch watch cookie 94126185426992] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x559b75d8b690 con 0x559b75d51400 2023-10-05T19:28:36.435 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.430+0000 7f805e5ebe40 1 -- 172.21.15.167:0/453671774 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:94 7.3 7:c609908c:::notify.5:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x559b75d8c4b0 con 0x7f8044007030 2023-10-05T19:28:36.438 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.430+0000 7f805c921700 1 -- 172.21.15.167:0/453671774 <== osd.0 v2:172.21.15.118:6800/27690 23 ==== osd_op_reply(94 notify.5 [create] v30'153 uv153 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f804c050040 con 0x7f8044007030 2023-10-05T19:28:36.438 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.430+0000 7f805e5ebe40 1 -- 172.21.15.167:0/453671774 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:95 7.3 7:c609908c:::notify.5:head [watch watch cookie 94126185430896] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x559b75d8c8b0 con 0x7f8044007030 2023-10-05T19:28:36.438 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.430+0000 7f805e5ebe40 1 -- 172.21.15.167:0/453671774 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:96 7.14 7:2b04a3e9:::notify.6:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x559b75d8d2c0 con 0x7f8044007030 2023-10-05T19:28:36.438 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.430+0000 7f805c921700 1 -- 172.21.15.167:0/453671774 <== osd.1 v2:172.21.15.167:6800/30035 32 ==== osd_op_reply(93 notify.4 [watch watch cookie 94126185426992] v30'154 uv153 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f804c050040 con 0x559b75d51400 2023-10-05T19:28:36.439 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.434+0000 7f805c921700 1 -- 172.21.15.167:0/453671774 <== osd.0 v2:172.21.15.118:6800/27690 24 ==== osd_op_reply(95 notify.5 [watch watch cookie 94126185430896] v30'154 uv153 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f804c050040 con 0x7f8044007030 2023-10-05T19:28:36.440 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.434+0000 7f805c921700 1 -- 172.21.15.167:0/453671774 <== osd.0 v2:172.21.15.118:6800/27690 25 ==== osd_op_reply(96 notify.6 [create] v30'153 uv153 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f804c050040 con 0x7f8044007030 2023-10-05T19:28:36.440 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.434+0000 7f805e5ebe40 1 -- 172.21.15.167:0/453671774 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:97 7.14 7:2b04a3e9:::notify.6:head [watch watch cookie 94126185432000] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x559b75d8c810 con 0x7f8044007030 2023-10-05T19:28:36.440 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.434+0000 7f805e5ebe40 1 -- 172.21.15.167:0/453671774 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:98 7.9 7:93e5b521:::notify.7:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x559b75d8c810 con 0x7f8044007030 2023-10-05T19:28:36.442 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.438+0000 7f805c921700 1 -- 172.21.15.167:0/453671774 <== osd.0 v2:172.21.15.118:6800/27690 26 ==== osd_op_reply(97 notify.6 [watch watch cookie 94126185432000] v30'154 uv153 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f804c050040 con 0x7f8044007030 2023-10-05T19:28:36.443 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.438+0000 7f805c921700 1 -- 172.21.15.167:0/453671774 <== osd.0 v2:172.21.15.118:6800/27690 27 ==== osd_op_reply(98 notify.7 [create] v30'153 uv153 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f804c050040 con 0x7f8044007030 2023-10-05T19:28:36.443 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.438+0000 7f805e5ebe40 1 -- 172.21.15.167:0/453671774 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:99 7.9 7:93e5b521:::notify.7:head [watch watch cookie 94126185434576] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x559b75d8d710 con 0x7f8044007030 2023-10-05T19:28:36.443 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.438+0000 7f805e5ebe40 20 add_watcher() i=0 2023-10-05T19:28:36.443 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.438+0000 7f805e5ebe40 20 add_watcher() i=1 2023-10-05T19:28:36.443 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.438+0000 7f805e5ebe40 20 add_watcher() i=2 2023-10-05T19:28:36.443 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.438+0000 7f805e5ebe40 20 add_watcher() i=3 2023-10-05T19:28:36.443 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.438+0000 7f805e5ebe40 20 add_watcher() i=4 2023-10-05T19:28:36.443 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.438+0000 7f805e5ebe40 20 add_watcher() i=5 2023-10-05T19:28:36.444 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.438+0000 7f805e5ebe40 20 add_watcher() i=6 2023-10-05T19:28:36.445 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.438+0000 7f805c921700 1 -- 172.21.15.167:0/453671774 <== osd.0 v2:172.21.15.118:6800/27690 28 ==== osd_op_reply(99 notify.7 [watch watch cookie 94126185434576] v30'154 uv153 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f804c050040 con 0x7f8044007030 2023-10-05T19:28:36.445 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.438+0000 7f805e5ebe40 20 add_watcher() i=7 2023-10-05T19:28:36.445 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.438+0000 7f805e5ebe40 2 all 8 watchers are set, enabling cache 2023-10-05T19:28:36.445 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.438+0000 7f805e5ebe40 1 -- 172.21.15.167:0/453671774 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:100 6.4 6:22d26bf9:::data_loggenerations_metadata:head [call version.check_conds in=74b,call version.read in=11b,read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x559b75d90990 con 0x559b75d51400 2023-10-05T19:28:36.446 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.438+0000 7f805c921700 1 -- 172.21.15.167:0/453671774 <== osd.1 v2:172.21.15.167:6800/30035 33 ==== osd_op_reply(100 data_loggenerations_metadata [call,call out=48b,read 0~28 out=28b] v0'0 uv1 ondisk = 0) v8 ==== 256+0+76 (crc 0 0 0) 0x7f804c050040 con 0x559b75d51400 2023-10-05T19:28:36.446 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.442+0000 7f805e5ebe40 1 -- 172.21.15.167:0/453671774 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:101 6.4 6:22d26bf9:::data_loggenerations_metadata:head [watch watch cookie 94126185436048] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x559b75d8dcd0 con 0x559b75d51400 2023-10-05T19:28:36.448 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.442+0000 7f805c921700 1 -- 172.21.15.167:0/453671774 <== osd.1 v2:172.21.15.167:6800/30035 34 ==== osd_op_reply(101 data_loggenerations_metadata [watch watch cookie 94126185436048] v30'149 uv1 ondisk = 0) v8 ==== 172+0+0 (crc 0 0 0) 0x7f804c050040 con 0x559b75d51400 2023-10-05T19:28:36.448 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.442+0000 7f802dfe3700 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2023-10-05T19:28:36.449 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.442+0000 7f805e5ebe40 20 check_secure_mon_conn(): auth registy supported: methods=[2] modes=[2,1] 2023-10-05T19:28:36.449 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.442+0000 7f805e5ebe40 20 check_secure_mon_conn(): mode 1 is insecure 2023-10-05T19:28:36.449 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.442+0000 7f805e5ebe40 1 Processor -- start 2023-10-05T19:28:36.449 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.442+0000 7f805e5ebe40 1 -- start start 2023-10-05T19:28:36.449 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.442+0000 7f805e5ebe40 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x559b75d9ba90 0x559b75d9be60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-05T19:28:36.450 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.442+0000 7f805e5ebe40 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x559b75d9c3a0 con 0x559b75d9ba90 2023-10-05T19:28:36.450 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.442+0000 7f805b91f700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x559b75d9ba90 0x559b75d9be60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-05T19:28:36.450 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.442+0000 7f805b91f700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x559b75d9ba90 0x559b75d9be60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.118:3300/0 says I am v2:172.21.15.167:41800/0 (socket says 172.21.15.167:41800) 2023-10-05T19:28:36.450 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.442+0000 7f805b91f700 1 -- 172.21.15.167:0/2975527127 learned_addr learned my addr 172.21.15.167:0/2975527127 (peer_addr_for_me v2:172.21.15.167:0/0) 2023-10-05T19:28:36.451 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.446+0000 7f805b91f700 1 -- 172.21.15.167:0/2975527127 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x559b75d9c4e0 con 0x559b75d9ba90 2023-10-05T19:28:36.451 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.446+0000 7f805b91f700 1 --2- 172.21.15.167:0/2975527127 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x559b75d9ba90 0x559b75d9be60 secure :-1 s=READY pgs=444 cs=0 l=1 rev1=1 crypto rx=0x559b75cc1760 tx=0x7f8044012000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b87fdeae43ec7462 server_cookie=0 in_seq=0 out_seq=0 2023-10-05T19:28:36.451 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.446+0000 7f802d7e2700 1 -- 172.21.15.167:0/2975527127 <== mon.0 v2:172.21.15.118:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f804402a5c0 con 0x559b75d9ba90 2023-10-05T19:28:36.451 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.446+0000 7f802d7e2700 1 -- 172.21.15.167:0/2975527127 <== mon.0 v2:172.21.15.118:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f804402a720 con 0x559b75d9ba90 2023-10-05T19:28:36.452 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.446+0000 7f802d7e2700 1 -- 172.21.15.167:0/2975527127 <== mon.0 v2:172.21.15.118:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f804402a910 con 0x559b75d9ba90 2023-10-05T19:28:36.453 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.446+0000 7f805e5ebe40 1 -- 172.21.15.167:0/2975527127 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x559b75d9ba90 msgr2=0x559b75d9be60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-05T19:28:36.453 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.446+0000 7f805e5ebe40 1 --2- 172.21.15.167:0/2975527127 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x559b75d9ba90 0x559b75d9be60 secure :-1 s=READY pgs=444 cs=0 l=1 rev1=1 crypto rx=0x559b75cc1760 tx=0x7f8044012000 comp rx=0 tx=0).stop 2023-10-05T19:28:36.453 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.446+0000 7f805e5ebe40 1 -- 172.21.15.167:0/2975527127 shutdown_connections 2023-10-05T19:28:36.453 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.446+0000 7f805e5ebe40 1 --2- 172.21.15.167:0/2975527127 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x559b75d9ba90 0x559b75d9be60 unknown :-1 s=CLOSED pgs=444 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-05T19:28:36.453 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.446+0000 7f805e5ebe40 1 -- 172.21.15.167:0/2975527127 >> 172.21.15.167:0/2975527127 conn(0x559b75d8f850 msgr2=0x559b75d99ea0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-05T19:28:36.453 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.446+0000 7f805e5ebe40 1 -- 172.21.15.167:0/2975527127 shutdown_connections 2023-10-05T19:28:36.453 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.446+0000 7f805e5ebe40 1 -- 172.21.15.167:0/2975527127 wait complete. 2023-10-05T19:28:36.453 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.446+0000 7f805e5ebe40 1 Processor -- start 2023-10-05T19:28:36.453 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.446+0000 7f805e5ebe40 1 -- start start 2023-10-05T19:28:36.454 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.446+0000 7f805e5ebe40 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x559b75d9ba90 0x559b75da2a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-05T19:28:36.454 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.446+0000 7f805e5ebe40 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x559b75da3a60 con 0x559b75d9ba90 2023-10-05T19:28:36.455 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.446+0000 7f805b91f700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x559b75d9ba90 0x559b75da2a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-05T19:28:36.455 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.446+0000 7f805b91f700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x559b75d9ba90 0x559b75da2a60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.118:3300/0 says I am v2:172.21.15.167:41812/0 (socket says 172.21.15.167:41812) 2023-10-05T19:28:36.455 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.446+0000 7f805b91f700 1 -- 172.21.15.167:0/3070112822 learned_addr learned my addr 172.21.15.167:0/3070112822 (peer_addr_for_me v2:172.21.15.167:0/0) 2023-10-05T19:28:36.455 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.450+0000 7f805b91f700 1 -- 172.21.15.167:0/3070112822 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x559b75cc1790 con 0x559b75d9ba90 2023-10-05T19:28:36.455 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.450+0000 7f805b91f700 1 --2- 172.21.15.167:0/3070112822 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x559b75d9ba90 0x559b75da2a60 secure :-1 s=READY pgs=445 cs=0 l=1 rev1=1 crypto rx=0x559b75d900a0 tx=0x7f804402f000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-05T19:28:36.456 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.450+0000 7f7fa77fe700 1 -- 172.21.15.167:0/3070112822 <== mon.0 v2:172.21.15.118:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f804402a540 con 0x559b75d9ba90 2023-10-05T19:28:36.456 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.450+0000 7f7fa77fe700 1 -- 172.21.15.167:0/3070112822 <== mon.0 v2:172.21.15.118:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f804403ecd0 con 0x559b75d9ba90 2023-10-05T19:28:36.456 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.450+0000 7f805e5ebe40 1 -- 172.21.15.167:0/3070112822 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x559b75da3060 con 0x559b75d9ba90 2023-10-05T19:28:36.456 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.450+0000 7f7fa77fe700 1 -- 172.21.15.167:0/3070112822 <== mon.0 v2:172.21.15.118:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f804403ee80 con 0x559b75d9ba90 2023-10-05T19:28:36.456 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.450+0000 7f805e5ebe40 1 -- 172.21.15.167:0/3070112822 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x559b75da34a0 con 0x559b75d9ba90 2023-10-05T19:28:36.457 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.450+0000 7f7fa77fe700 1 -- 172.21.15.167:0/3070112822 <== mon.0 v2:172.21.15.118:3300/0 4 ==== mgrmap(e 5) v1 ==== 40329+0+0 (secure 0 0 0) 0x7f8044046070 con 0x559b75d9ba90 2023-10-05T19:28:36.457 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.450+0000 7f7fa77fe700 1 --2- 172.21.15.167:0/3070112822 >> [v2:172.21.15.167:6824/29889,v1:172.21.15.167:6825/29889] conn(0x7f7f9c030c50 0x7f7f9c033100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-05T19:28:36.457 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.450+0000 7f7fa77fe700 1 -- 172.21.15.167:0/3070112822 <== mon.0 v2:172.21.15.118:3300/0 5 ==== osd_map(30..30 src has 1..30) v4 ==== 6057+0+0 (secure 0 0 0) 0x7f804402ae40 con 0x559b75d9ba90 2023-10-05T19:28:36.457 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.450+0000 7f805c921700 1 --2- 172.21.15.167:0/3070112822 >> [v2:172.21.15.167:6824/29889,v1:172.21.15.167:6825/29889] conn(0x7f7f9c030c50 0x7f7f9c033100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-05T19:28:36.457 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.450+0000 7f805e5ebe40 5 note: GC not initialized 2023-10-05T19:28:36.458 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.450+0000 7f805c921700 1 --2- 172.21.15.167:0/3070112822 >> [v2:172.21.15.167:6824/29889,v1:172.21.15.167:6825/29889] conn(0x7f7f9c030c50 0x7f7f9c033100 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x559b75d90a10 tx=0x7f804c000f90 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-05T19:28:36.459 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.454+0000 7f7fa4ff9700 20 reqs_thread_entry: start 2023-10-05T19:28:36.460 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.454+0000 7f805e5ebe40 20 init_complete bucket index max shards: 11 2023-10-05T19:28:36.460 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.454+0000 7f7f9a7fc700 20 reqs_thread_entry: start 2023-10-05T19:28:36.460 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.454+0000 7f805e5ebe40 10 rgw notify: Started notification manager with: 1 workers 2023-10-05T19:28:36.460 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.454+0000 7f7f98ff9700 20 rgw notify: INFO: next queues processing will happen at: Thu Oct 5 19:29:06 2023 2023-10-05T19:28:36.461 INFO:teuthology.orchestra.run.smithi167.stderr: 2023-10-05T19:28:36.461 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.454+0000 7f7f7ffff700 20 reqs_thread_entry: start 2023-10-05T19:28:36.461 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.454+0000 7f805e5ebe40 10 log_show_next pos 0 bl 0 off 0 eof 0 2023-10-05T19:28:36.461 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.454+0000 7f805e5ebe40 1 --2- 172.21.15.167:0/3070112822 >> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] conn(0x559b75e09f90 0x559b75e0c370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-05T19:28:36.461 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.454+0000 7f805e5ebe40 1 -- 172.21.15.167:0/3070112822 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:1 6.8 6:163360d8:::2023-10-05-19--:head [read 0~1048576] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x559b75e0c8b0 con 0x559b75e09f90 2023-10-05T19:28:36.461 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.454+0000 7f805c120700 1 --2- 172.21.15.167:0/3070112822 >> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] conn(0x559b75e09f90 0x559b75e0c370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-05T19:28:36.462 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.454+0000 7f805c120700 1 --2- 172.21.15.167:0/3070112822 >> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] conn(0x559b75e09f90 0x559b75e0c370 crc :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-05T19:28:36.462 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.458+0000 7f805c120700 1 -- 172.21.15.167:0/3070112822 <== osd.3 v2:172.21.15.167:6808/30036 1 ==== osd_op_reply(1 2023-10-05-19-- [read 0~658 out=658b] v0'0 uv21 ondisk = 0) v8 ==== 159+0+658 (crc 0 0 0) 0x7f8054025020 con 0x559b75e09f90 2023-10-05T19:28:36.462 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.458+0000 7f805e5ebe40 10 read 658 2023-10-05T19:28:36.463 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.458+0000 7f805e5ebe40 10 log_show_next pos 658 bl 658 off 304 eof 1 2023-10-05T19:28:36.463 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.458+0000 7f805e5ebe40 10 log_show_next pos 658 bl 658 off 658 eof 1 2023-10-05T19:28:36.463 INFO:teuthology.orchestra.run.smithi167.stdout:{ 2023-10-05T19:28:36.463 INFO:teuthology.orchestra.run.smithi167.stdout: "bucket_id": "", 2023-10-05T19:28:36.463 INFO:teuthology.orchestra.run.smithi167.stdout: "bucket_owner": "", 2023-10-05T19:28:36.463 INFO:teuthology.orchestra.run.smithi167.stdout: "bucket": "", 2023-10-05T19:28:36.463 INFO:teuthology.orchestra.run.smithi167.stdout: "log_entries": [ 2023-10-05T19:28:36.463 INFO:teuthology.orchestra.run.smithi167.stdout: { 2023-10-05T19:28:36.463 INFO:teuthology.orchestra.run.smithi167.stdout: "bucket": "", 2023-10-05T19:28:36.464 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2023-10-05T19:28:21.942481Z", 2023-10-05T19:28:36.464 INFO:teuthology.orchestra.run.smithi167.stdout: "time_local": "2023-10-05T19:28:21.942481+0000", 2023-10-05T19:28:36.464 INFO:teuthology.orchestra.run.smithi167.stdout: "remote_addr": "172.21.15.167", 2023-10-05T19:28:36.464 INFO:teuthology.orchestra.run.smithi167.stdout: "user": "anonymous", 2023-10-05T19:28:36.464 INFO:teuthology.orchestra.run.smithi167.stdout: "operation": "list_buckets", 2023-10-05T19:28:36.464 INFO:teuthology.orchestra.run.smithi167.stdout: "uri": "GET / HTTP/1.1", 2023-10-05T19:28:36.464 INFO:teuthology.orchestra.run.smithi167.stdout: "http_status": "200", 2023-10-05T19:28:36.464 INFO:teuthology.orchestra.run.smithi167.stdout: "error_code": "", 2023-10-05T19:28:36.464 INFO:teuthology.orchestra.run.smithi167.stdout: "bytes_sent": 231, 2023-10-05T19:28:36.464 INFO:teuthology.orchestra.run.smithi167.stdout: "bytes_received": 0, 2023-10-05T19:28:36.464 INFO:teuthology.orchestra.run.smithi167.stdout: "object_size": 0, 2023-10-05T19:28:36.464 INFO:teuthology.orchestra.run.smithi167.stdout: "total_time": 0, 2023-10-05T19:28:36.465 INFO:teuthology.orchestra.run.smithi167.stdout: "user_agent": "curl/7.68.0", 2023-10-05T19:28:36.465 INFO:teuthology.orchestra.run.smithi167.stdout: "referrer": "", 2023-10-05T19:28:36.465 INFO:teuthology.orchestra.run.smithi167.stdout: "trans_id": "tx00000f3d18f9ab713a009-00651f0e55-107d-default", 2023-10-05T19:28:36.465 INFO:teuthology.orchestra.run.smithi167.stdout: "authentication_type": "Local", 2023-10-05T19:28:36.465 INFO:teuthology.orchestra.run.smithi167.stdout: "temp_url": false 2023-10-05T19:28:36.465 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2023-10-05T19:28:36.465 INFO:teuthology.orchestra.run.smithi167.stdout: { 2023-10-05T19:28:36.465 INFO:teuthology.orchestra.run.smithi167.stdout: "bucket": "", 2023-10-05T19:28:36.465 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2023-10-05T19:28:30.474284Z", 2023-10-05T19:28:36.465 INFO:teuthology.orchestra.run.smithi167.stdout: "time_local": "2023-10-05T19:28:30.474284+0000", 2023-10-05T19:28:36.465 INFO:teuthology.orchestra.run.smithi167.stdout: "remote_addr": "172.21.0.51", 2023-10-05T19:28:36.466 INFO:teuthology.orchestra.run.smithi167.stdout: "user": "foo", 2023-10-05T19:28:36.466 INFO:teuthology.orchestra.run.smithi167.stdout: "operation": "list_buckets", 2023-10-05T19:28:36.466 INFO:teuthology.orchestra.run.smithi167.stdout: "uri": "GET / HTTP/1.1", 2023-10-05T19:28:36.466 INFO:teuthology.orchestra.run.smithi167.stdout: "http_status": "200", 2023-10-05T19:28:36.466 INFO:teuthology.orchestra.run.smithi167.stdout: "error_code": "", 2023-10-05T19:28:36.466 INFO:teuthology.orchestra.run.smithi167.stdout: "bytes_sent": 322, 2023-10-05T19:28:36.466 INFO:teuthology.orchestra.run.smithi167.stdout: "bytes_received": 0, 2023-10-05T19:28:36.466 INFO:teuthology.orchestra.run.smithi167.stdout: "object_size": 0, 2023-10-05T19:28:36.466 INFO:teuthology.orchestra.run.smithi167.stdout: "total_time": 0, 2023-10-05T19:28:36.466 INFO:teuthology.orchestra.run.smithi167.stdout: "user_agent": "Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic", 2023-10-05T19:28:36.466 INFO:teuthology.orchestra.run.smithi167.stdout: "referrer": "", 2023-10-05T19:28:36.466 INFO:teuthology.orchestra.run.smithi167.stdout: "trans_id": "tx00000a12db0c1f8aacef2-00651f0e5e-107d-default", 2023-10-05T19:28:36.467 INFO:teuthology.orchestra.run.smithi167.stdout: "authentication_type": "Local", 2023-10-05T19:28:36.467 INFO:teuthology.orchestra.run.smithi167.stdout: "access_key_id": "9te6NH5mcdcq0Tc5i8i1", 2023-10-05T19:28:36.467 INFO:teuthology.orchestra.run.smithi167.stdout: "temp_url": false 2023-10-05T19:28:36.467 INFO:teuthology.orchestra.run.smithi167.stdout: } 2023-10-05T19:28:36.467 INFO:teuthology.orchestra.run.smithi167.stdout: ], 2023-10-05T19:28:36.467 INFO:teuthology.orchestra.run.smithi167.stdout: "log_sum": { 2023-10-05T19:28:36.467 INFO:teuthology.orchestra.run.smithi167.stdout: "bytes_sent": 553, 2023-10-05T19:28:36.467 INFO:teuthology.orchestra.run.smithi167.stdout: "bytes_received": 0, 2023-10-05T19:28:36.467 INFO:teuthology.orchestra.run.smithi167.stdout: "total_time": 0, 2023-10-05T19:28:36.467 INFO:teuthology.orchestra.run.smithi167.stdout: "total_entries": 2 2023-10-05T19:28:36.467 INFO:teuthology.orchestra.run.smithi167.stdout: } 2023-10-05T19:28:36.467 INFO:teuthology.orchestra.run.smithi167.stdout:} 2023-10-05T19:28:36.467 INFO:teuthology.orchestra.run.smithi167.stdout: 2023-10-05T19:28:36.468 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.458+0000 7f805e5ebe40 1 -- 172.21.15.167:0/453671774 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:102 7.1f 7:f95f44c2:::notify.0:head [watch unwatch cookie 94126185414080] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x559b75dc95c0 con 0x7f80540106e0 2023-10-05T19:28:36.468 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.458+0000 7f805b91f700 1 -- 172.21.15.167:0/453671774 <== osd.2 v2:172.21.15.167:6816/30037 17 ==== osd_op_reply(102 notify.0 [watch unwatch cookie 94126185414080] v30'155 uv153 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f8044032040 con 0x7f80540106e0 2023-10-05T19:28:36.469 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.458+0000 7f805e5ebe40 20 remove_watcher() i=0 2023-10-05T19:28:36.469 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.458+0000 7f805e5ebe40 2 removed watcher, disabling cache 2023-10-05T19:28:36.469 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.458+0000 7f805e5ebe40 1 -- 172.21.15.167:0/453671774 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:103 7.0 7:05bf5b68:::notify.1:head [watch unwatch cookie 94126185417104] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x559b75d87dc0 con 0x559b75d51400 2023-10-05T19:28:36.469 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.462+0000 7f805c921700 1 -- 172.21.15.167:0/453671774 <== osd.1 v2:172.21.15.167:6800/30035 35 ==== osd_op_reply(103 notify.1 [watch unwatch cookie 94126185417104] v30'155 uv153 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f804c050040 con 0x559b75d51400 2023-10-05T19:28:36.469 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.462+0000 7f805e5ebe40 20 remove_watcher() i=1 2023-10-05T19:28:36.469 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.462+0000 7f805e5ebe40 1 -- 172.21.15.167:0/453671774 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:104 7.15 7:a93a5511:::notify.2:head [watch unwatch cookie 94126185420368] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x559b75d87dc0 con 0x559b75d82050 2023-10-05T19:28:36.471 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.466+0000 7f805c120700 1 -- 172.21.15.167:0/453671774 <== osd.3 v2:172.21.15.167:6808/30036 24 ==== osd_op_reply(104 notify.2 [watch unwatch cookie 94126185420368] v30'155 uv153 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f8054025020 con 0x559b75d82050 2023-10-05T19:28:36.471 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.466+0000 7f805e5ebe40 20 remove_watcher() i=2 2023-10-05T19:28:36.471 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.466+0000 7f805e5ebe40 1 -- 172.21.15.167:0/453671774 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:105 7.e 7:7759931f:::notify.3:head [watch unwatch cookie 94126185419104] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x559b75d87dc0 con 0x7f80540106e0 2023-10-05T19:28:36.473 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.466+0000 7f805b91f700 1 -- 172.21.15.167:0/453671774 <== osd.2 v2:172.21.15.167:6816/30037 18 ==== osd_op_reply(105 notify.3 [watch unwatch cookie 94126185419104] v30'155 uv153 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f8044032040 con 0x7f80540106e0 2023-10-05T19:28:36.474 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.466+0000 7f805e5ebe40 20 remove_watcher() i=3 2023-10-05T19:28:36.474 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.466+0000 7f805e5ebe40 1 -- 172.21.15.167:0/453671774 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:106 7.d 7:b4812045:::notify.4:head [watch unwatch cookie 94126185426992] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x559b75d89160 con 0x559b75d51400 2023-10-05T19:28:36.476 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.470+0000 7f805c921700 1 -- 172.21.15.167:0/453671774 <== osd.1 v2:172.21.15.167:6800/30035 36 ==== osd_op_reply(106 notify.4 [watch unwatch cookie 94126185426992] v30'155 uv153 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f804c050040 con 0x559b75d51400 2023-10-05T19:28:36.477 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.470+0000 7f805e5ebe40 20 remove_watcher() i=4 2023-10-05T19:28:36.477 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.470+0000 7f805e5ebe40 1 -- 172.21.15.167:0/453671774 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:107 7.3 7:c609908c:::notify.5:head [watch unwatch cookie 94126185430896] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x559b75d89160 con 0x7f8044007030 2023-10-05T19:28:36.479 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.474+0000 7f805c921700 1 -- 172.21.15.167:0/453671774 <== osd.0 v2:172.21.15.118:6800/27690 29 ==== osd_op_reply(107 notify.5 [watch unwatch cookie 94126185430896] v30'155 uv153 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f804c050040 con 0x7f8044007030 2023-10-05T19:28:36.480 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.474+0000 7f805e5ebe40 20 remove_watcher() i=5 2023-10-05T19:28:36.480 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.474+0000 7f805e5ebe40 1 -- 172.21.15.167:0/453671774 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:108 7.14 7:2b04a3e9:::notify.6:head [watch unwatch cookie 94126185432000] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x559b75d89160 con 0x7f8044007030 2023-10-05T19:28:36.482 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.478+0000 7f805c921700 1 -- 172.21.15.167:0/453671774 <== osd.0 v2:172.21.15.118:6800/27690 30 ==== osd_op_reply(108 notify.6 [watch unwatch cookie 94126185432000] v30'155 uv153 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f804c050040 con 0x7f8044007030 2023-10-05T19:28:36.483 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.478+0000 7f805e5ebe40 20 remove_watcher() i=6 2023-10-05T19:28:36.483 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.478+0000 7f805e5ebe40 1 -- 172.21.15.167:0/453671774 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:109 7.9 7:93e5b521:::notify.7:head [watch unwatch cookie 94126185434576] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x559b75d89160 con 0x7f8044007030 2023-10-05T19:28:36.486 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.478+0000 7f805c921700 1 -- 172.21.15.167:0/453671774 <== osd.0 v2:172.21.15.118:6800/27690 31 ==== osd_op_reply(109 notify.7 [watch unwatch cookie 94126185434576] v30'155 uv153 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f804c050040 con 0x7f8044007030 2023-10-05T19:28:36.486 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.478+0000 7f805e5ebe40 20 remove_watcher() i=7 2023-10-05T19:28:36.488 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.482+0000 7f802dfe3700 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2023-10-05T19:28:36.488 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.482+0000 7f805e5ebe40 1 -- 172.21.15.167:0/453671774 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:110 6.4 6:22d26bf9:::data_loggenerations_metadata:head [watch unwatch cookie 94126185436048] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x559b75d89160 con 0x559b75d51400 2023-10-05T19:28:36.491 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.486+0000 7f805c921700 1 -- 172.21.15.167:0/453671774 <== osd.1 v2:172.21.15.167:6800/30035 37 ==== osd_op_reply(110 data_loggenerations_metadata [watch unwatch cookie 94126185436048] v30'150 uv1 ondisk = 0) v8 ==== 172+0+0 (crc 0 0 0) 0x7f804c050040 con 0x559b75d51400 2023-10-05T19:28:36.491 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.486+0000 7f805e5ebe40 1 -- 172.21.15.167:0/453671774 >> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] conn(0x7f8044007030 msgr2=0x7f804401c920 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-05T19:28:36.492 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.486+0000 7f805e5ebe40 1 --2- 172.21.15.167:0/453671774 >> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] conn(0x7f8044007030 0x7f804401c920 crc :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-05T19:28:36.492 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.486+0000 7f805e5ebe40 1 -- 172.21.15.167:0/453671774 >> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] conn(0x559b75d51400 msgr2=0x559b75d7e580 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-05T19:28:36.492 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.486+0000 7f805e5ebe40 1 --2- 172.21.15.167:0/453671774 >> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] conn(0x559b75d51400 0x559b75d7e580 crc :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-05T19:28:36.492 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.486+0000 7f805e5ebe40 1 -- 172.21.15.167:0/453671774 >> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] conn(0x7f80540106e0 msgr2=0x7f805402a000 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-05T19:28:36.492 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.486+0000 7f805e5ebe40 1 --2- 172.21.15.167:0/453671774 >> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] conn(0x7f80540106e0 0x7f805402a000 crc :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-05T19:28:36.492 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.486+0000 7f805e5ebe40 1 -- 172.21.15.167:0/453671774 >> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] conn(0x559b75d82050 msgr2=0x559b75d84430 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-05T19:28:36.492 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.486+0000 7f805e5ebe40 1 --2- 172.21.15.167:0/453671774 >> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] conn(0x559b75d82050 0x559b75d84430 crc :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-05T19:28:36.492 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.486+0000 7f805e5ebe40 1 -- 172.21.15.167:0/453671774 >> [v2:172.21.15.167:6824/29889,v1:172.21.15.167:6825/29889] conn(0x7f803c030c50 msgr2=0x7f803c033100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-05T19:28:36.492 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.486+0000 7f805e5ebe40 1 --2- 172.21.15.167:0/453671774 >> [v2:172.21.15.167:6824/29889,v1:172.21.15.167:6825/29889] conn(0x7f803c030c50 0x7f803c033100 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7f8044005a90 tx=0x7f8044015000 comp rx=0 tx=0).stop 2023-10-05T19:28:36.492 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.486+0000 7f805e5ebe40 1 -- 172.21.15.167:0/453671774 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x559b75cc0c50 msgr2=0x559b75d72b30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-05T19:28:36.493 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.486+0000 7f805c921700 1 -- 172.21.15.167:0/453671774 reap_dead start 2023-10-05T19:28:36.493 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.486+0000 7f805e5ebe40 1 --2- 172.21.15.167:0/453671774 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x559b75cc0c50 0x559b75d72b30 secure :-1 s=READY pgs=443 cs=0 l=1 rev1=1 crypto rx=0x7f8054000f80 tx=0x7f8054014000 comp rx=0 tx=0).stop 2023-10-05T19:28:36.493 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.486+0000 7f805e5ebe40 1 -- 172.21.15.167:0/453671774 shutdown_connections 2023-10-05T19:28:36.493 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.486+0000 7f805e5ebe40 1 --2- 172.21.15.167:0/453671774 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x559b75cc0c50 0x559b75d72b30 secure :-1 s=CLOSED pgs=443 cs=0 l=1 rev1=1 crypto rx=0x7f8054000f80 tx=0x7f8054014000 comp rx=0 tx=0).stop 2023-10-05T19:28:36.493 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.486+0000 7f805e5ebe40 1 -- 172.21.15.167:0/453671774 >> 172.21.15.167:0/453671774 conn(0x559b75d47720 msgr2=0x559b75d66390 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-05T19:28:36.493 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.486+0000 7f805e5ebe40 1 -- 172.21.15.167:0/453671774 shutdown_connections 2023-10-05T19:28:36.493 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.486+0000 7f805e5ebe40 1 -- 172.21.15.167:0/453671774 wait complete. 2023-10-05T19:28:36.494 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.486+0000 7f805e5ebe40 1 -- 172.21.15.167:0/3070112822 >> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] conn(0x559b75e09f90 msgr2=0x559b75e0c370 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-05T19:28:36.494 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.486+0000 7f805e5ebe40 1 --2- 172.21.15.167:0/3070112822 >> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] conn(0x559b75e09f90 0x559b75e0c370 crc :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-05T19:28:36.494 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.486+0000 7f805e5ebe40 1 -- 172.21.15.167:0/3070112822 >> [v2:172.21.15.167:6824/29889,v1:172.21.15.167:6825/29889] conn(0x7f7f9c030c50 msgr2=0x7f7f9c033100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-05T19:28:36.494 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.486+0000 7f805e5ebe40 1 --2- 172.21.15.167:0/3070112822 >> [v2:172.21.15.167:6824/29889,v1:172.21.15.167:6825/29889] conn(0x7f7f9c030c50 0x7f7f9c033100 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x559b75d90a10 tx=0x7f804c000f90 comp rx=0 tx=0).stop 2023-10-05T19:28:36.495 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.486+0000 7f805e5ebe40 1 -- 172.21.15.167:0/3070112822 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x559b75d9ba90 msgr2=0x559b75da2a60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-05T19:28:36.495 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.486+0000 7f805e5ebe40 1 --2- 172.21.15.167:0/3070112822 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x559b75d9ba90 0x559b75da2a60 secure :-1 s=READY pgs=445 cs=0 l=1 rev1=1 crypto rx=0x559b75d900a0 tx=0x7f804402f000 comp rx=0 tx=0).stop 2023-10-05T19:28:36.495 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.486+0000 7f805e5ebe40 1 -- 172.21.15.167:0/3070112822 shutdown_connections 2023-10-05T19:28:36.495 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.486+0000 7f805e5ebe40 1 --2- 172.21.15.167:0/3070112822 >> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] conn(0x559b75e09f90 0x559b75e0c370 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-05T19:28:36.495 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.486+0000 7f805e5ebe40 1 --2- 172.21.15.167:0/3070112822 >> [v2:172.21.15.167:6824/29889,v1:172.21.15.167:6825/29889] conn(0x7f7f9c030c50 0x7f7f9c033100 unknown :-1 s=CLOSED pgs=155 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-05T19:28:36.495 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.486+0000 7f805e5ebe40 1 --2- 172.21.15.167:0/3070112822 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x559b75d9ba90 0x559b75da2a60 unknown :-1 s=CLOSED pgs=445 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-05T19:28:36.495 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.486+0000 7f805e5ebe40 1 -- 172.21.15.167:0/3070112822 >> 172.21.15.167:0/3070112822 conn(0x559b75d8f850 msgr2=0x559b75d9d0c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-05T19:28:36.495 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.486+0000 7f805e5ebe40 1 -- 172.21.15.167:0/3070112822 shutdown_connections 2023-10-05T19:28:36.495 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.486+0000 7f805e5ebe40 1 -- 172.21.15.167:0/3070112822 wait complete. 2023-10-05T19:28:36.498 DEBUG:tasks.util.rgw: json result: {'bucket_id': '', 'bucket_owner': '', 'bucket': '', 'log_entries': [{'bucket': '', 'time': '2023-10-05T19:28:21.942481Z', 'time_local': '2023-10-05T19:28:21.942481+0000', 'remote_addr': '172.21.15.167', 'user': 'anonymous', 'operation': 'list_buckets', 'uri': 'GET / HTTP/1.1', 'http_status': '200', 'error_code': '', 'bytes_sent': 231, 'bytes_received': 0, 'object_size': 0, 'total_time': 0, 'user_agent': 'curl/7.68.0', 'referrer': '', 'trans_id': 'tx00000f3d18f9ab713a009-00651f0e55-107d-default', 'authentication_type': 'Local', 'temp_url': False}, {'bucket': '', 'time': '2023-10-05T19:28:30.474284Z', 'time_local': '2023-10-05T19:28:30.474284+0000', 'remote_addr': '172.21.0.51', 'user': 'foo', 'operation': 'list_buckets', 'uri': 'GET / HTTP/1.1', 'http_status': '200', 'error_code': '', 'bytes_sent': 322, 'bytes_received': 0, 'object_size': 0, 'total_time': 0, 'user_agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'referrer': '', 'trans_id': 'tx00000a12db0c1f8aacef2-00651f0e5e-107d-default', 'authentication_type': 'Local', 'access_key_id': '9te6NH5mcdcq0Tc5i8i1', 'temp_url': False}], 'log_sum': {'bytes_sent': 553, 'bytes_received': 0, 'total_time': 0, 'total_entries': 2}} 2023-10-05T19:28:36.499 INFO:tasks.util.rgw:rgwadmin: client.0 : ['log', 'rm', '--object', '2023-10-05-19--'] 2023-10-05T19:28:36.499 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'ceph', 'log', 'rm', '--object', '2023-10-05-19--'] 2023-10-05T19:28:36.499 DEBUG:teuthology.orchestra.run.smithi167:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster ceph log rm --object 2023-10-05-19-- 2023-10-05T19:28:36.582 INFO:teuthology.orchestra.run.smithi167.stderr:ignoring --setuser ceph since I am not root 2023-10-05T19:28:36.583 INFO:teuthology.orchestra.run.smithi167.stderr:ignoring --setgroup ceph since I am not root 2023-10-05T19:28:36.583 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.578+0000 7f6402d38e40 1 Processor -- start 2023-10-05T19:28:36.583 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.578+0000 7f6402d38e40 1 -- start start 2023-10-05T19:28:36.584 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.578+0000 7f6402d38e40 1 --2- >> v2:172.21.15.118:3300/0 conn(0x5566f49b3410 0x5566f49b1fc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-05T19:28:36.584 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.578+0000 7f6402d38e40 1 -- --> v1:172.21.15.118:6789/0 -- auth(proto 0 26 bytes epoch 0) v1 -- 0x5566f48ffbd0 con 0x5566f49b37e0 2023-10-05T19:28:36.584 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.578+0000 7f6402d38e40 1 -- --> v2:172.21.15.118:3300/0 -- mon_getmap magic: 0 v1 -- 0x5566f48bcee0 con 0x5566f49b3410 2023-10-05T19:28:36.585 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.578+0000 7f640086d700 1 --2- >> v2:172.21.15.118:3300/0 conn(0x5566f49b3410 0x5566f49b1fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-05T19:28:36.585 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.578+0000 7f640006c700 1 --1- >> v1:172.21.15.118:6789/0 conn(0x5566f49b37e0 0x5566f49b2500 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.118:6789/0 says I am v1:172.21.15.167:59326/0 (socket says 172.21.15.167:59326) 2023-10-05T19:28:36.585 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.578+0000 7f640006c700 1 -- 172.21.15.167:0/1281366988 learned_addr learned my addr 172.21.15.167:0/1281366988 (peer_addr_for_me v1:172.21.15.167:0/0) 2023-10-05T19:28:36.585 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.578+0000 7f63ff86b700 1 -- 172.21.15.167:0/1281366988 <== mon.0 v1:172.21.15.118:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3321115183 0 0) 0x5566f48ffbd0 con 0x5566f49b37e0 2023-10-05T19:28:36.585 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.578+0000 7f63ff86b700 1 -- 172.21.15.167:0/1281366988 --> v1:172.21.15.118:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f63e0001e10 con 0x5566f49b37e0 2023-10-05T19:28:36.585 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.578+0000 7f640086d700 1 -- 172.21.15.167:0/1281366988 >> v1:172.21.15.118:6789/0 conn(0x5566f49b37e0 legacy=0x5566f49b2500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-05T19:28:36.585 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.578+0000 7f640086d700 1 -- 172.21.15.167:0/1281366988 --> v2:172.21.15.118:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5566f49b2cd0 con 0x5566f49b3410 2023-10-05T19:28:36.586 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.578+0000 7f640086d700 1 --2- 172.21.15.167:0/1281366988 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5566f49b3410 0x5566f49b1fc0 secure :-1 s=READY pgs=447 cs=0 l=1 rev1=1 crypto rx=0x7f63f000a5d0 tx=0x7f63f0005a50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9df066a62c074d2e server_cookie=0 in_seq=0 out_seq=0 2023-10-05T19:28:36.586 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.578+0000 7f63ff86b700 1 -- 172.21.15.167:0/1281366988 <== mon.0 v2:172.21.15.118:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f63f0007610 con 0x5566f49b3410 2023-10-05T19:28:36.586 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.578+0000 7f63ff86b700 1 -- 172.21.15.167:0/1281366988 <== mon.0 v2:172.21.15.118:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63f0007770 con 0x5566f49b3410 2023-10-05T19:28:36.586 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.582+0000 7f63ff86b700 1 -- 172.21.15.167:0/1281366988 <== mon.0 v2:172.21.15.118:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f63f0007a40 con 0x5566f49b3410 2023-10-05T19:28:36.587 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.582+0000 7f6402d38e40 1 -- 172.21.15.167:0/1281366988 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5566f49b3410 msgr2=0x5566f49b1fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-05T19:28:36.587 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.582+0000 7f6402d38e40 1 --2- 172.21.15.167:0/1281366988 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5566f49b3410 0x5566f49b1fc0 secure :-1 s=READY pgs=447 cs=0 l=1 rev1=1 crypto rx=0x7f63f000a5d0 tx=0x7f63f0005a50 comp rx=0 tx=0).stop 2023-10-05T19:28:36.587 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.582+0000 7f6402d38e40 1 -- 172.21.15.167:0/1281366988 shutdown_connections 2023-10-05T19:28:36.587 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.582+0000 7f6402d38e40 1 --2- 172.21.15.167:0/1281366988 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5566f49b3410 0x5566f49b1fc0 unknown :-1 s=CLOSED pgs=447 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-05T19:28:36.588 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.582+0000 7f6402d38e40 1 -- 172.21.15.167:0/1281366988 >> 172.21.15.167:0/1281366988 conn(0x5566f49ad720 msgr2=0x5566f49afb30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-05T19:28:36.588 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.582+0000 7f6402d38e40 1 -- 172.21.15.167:0/1281366988 shutdown_connections 2023-10-05T19:28:36.588 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.582+0000 7f6402d38e40 1 -- 172.21.15.167:0/1281366988 wait complete. 2023-10-05T19:28:36.588 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.582+0000 7f6402d38e40 1 Processor -- start 2023-10-05T19:28:36.588 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.582+0000 7f6402d38e40 1 -- start start 2023-10-05T19:28:36.589 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.582+0000 7f6402d38e40 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5566f49b37e0 0x5566f49d4030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-05T19:28:36.589 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.582+0000 7f6402d38e40 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x5566f4880e30 con 0x5566f49b37e0 2023-10-05T19:28:36.589 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.582+0000 7f640086d700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5566f49b37e0 0x5566f49d4030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-05T19:28:36.589 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.582+0000 7f640086d700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5566f49b37e0 0x5566f49d4030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.118:3300/0 says I am v2:172.21.15.167:41830/0 (socket says 172.21.15.167:41830) 2023-10-05T19:28:36.589 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.582+0000 7f640086d700 1 -- 172.21.15.167:0/2805920674 learned_addr learned my addr 172.21.15.167:0/2805920674 (peer_addr_for_me v2:172.21.15.167:0/0) 2023-10-05T19:28:36.589 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.582+0000 7f640086d700 1 -- 172.21.15.167:0/2805920674 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f63f0005cf0 con 0x5566f49b37e0 2023-10-05T19:28:36.591 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.582+0000 7f640086d700 1 --2- 172.21.15.167:0/2805920674 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5566f49b37e0 0x5566f49d4030 secure :-1 s=READY pgs=448 cs=0 l=1 rev1=1 crypto rx=0x7f63f0005a90 tx=0x7f63f0014000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fdce0476c9da05e0 server_cookie=0 in_seq=0 out_seq=0 2023-10-05T19:28:36.591 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.582+0000 7f63fe869700 1 -- 172.21.15.167:0/2805920674 <== mon.0 v2:172.21.15.118:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f63f0021070 con 0x5566f49b37e0 2023-10-05T19:28:36.591 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.582+0000 7f63fe869700 1 -- 172.21.15.167:0/2805920674 <== mon.0 v2:172.21.15.118:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63f0007960 con 0x5566f49b37e0 2023-10-05T19:28:36.591 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.582+0000 7f63fe869700 1 -- 172.21.15.167:0/2805920674 <== mon.0 v2:172.21.15.118:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f63f0007c50 con 0x5566f49b37e0 2023-10-05T19:28:36.591 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.586+0000 7f6402d38e40 1 -- 172.21.15.167:0/2805920674 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5566f49b37e0 msgr2=0x5566f49d4030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-05T19:28:36.591 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.586+0000 7f6402d38e40 1 --2- 172.21.15.167:0/2805920674 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5566f49b37e0 0x5566f49d4030 secure :-1 s=READY pgs=448 cs=0 l=1 rev1=1 crypto rx=0x7f63f0005a90 tx=0x7f63f0014000 comp rx=0 tx=0).stop 2023-10-05T19:28:36.591 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.586+0000 7f6402d38e40 1 -- 172.21.15.167:0/2805920674 shutdown_connections 2023-10-05T19:28:36.591 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.586+0000 7f6402d38e40 1 --2- 172.21.15.167:0/2805920674 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5566f49b37e0 0x5566f49d4030 unknown :-1 s=CLOSED pgs=448 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-05T19:28:36.592 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.586+0000 7f6402d38e40 1 -- 172.21.15.167:0/2805920674 >> 172.21.15.167:0/2805920674 conn(0x5566f49ad720 msgr2=0x5566f49cfbf0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-05T19:28:36.592 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.586+0000 7f6402d38e40 1 -- 172.21.15.167:0/2805920674 shutdown_connections 2023-10-05T19:28:36.592 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.586+0000 7f6402d38e40 1 -- 172.21.15.167:0/2805920674 wait complete. 2023-10-05T19:28:36.592 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.586+0000 7f6402d38e40 1 Processor -- start 2023-10-05T19:28:36.592 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.586+0000 7f6402d38e40 1 -- start start 2023-10-05T19:28:36.592 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.586+0000 7f6402d38e40 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5566f49b37e0 0x5566f49dab80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-05T19:28:36.592 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.586+0000 7f6402d38e40 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x5566f4853ed0 con 0x5566f49b37e0 2023-10-05T19:28:36.593 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.586+0000 7f640086d700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5566f49b37e0 0x5566f49dab80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-05T19:28:36.593 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.586+0000 7f640086d700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5566f49b37e0 0x5566f49dab80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.118:3300/0 says I am v2:172.21.15.167:41832/0 (socket says 172.21.15.167:41832) 2023-10-05T19:28:36.593 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.586+0000 7f640086d700 1 -- 172.21.15.167:0/1379194618 learned_addr learned my addr 172.21.15.167:0/1379194618 (peer_addr_for_me v2:172.21.15.167:0/0) 2023-10-05T19:28:36.593 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.586+0000 7f640086d700 1 -- 172.21.15.167:0/1379194618 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5566f49b2cd0 con 0x5566f49b37e0 2023-10-05T19:28:36.594 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.586+0000 7f640086d700 1 --2- 172.21.15.167:0/1379194618 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5566f49b37e0 0x5566f49dab80 secure :-1 s=READY pgs=449 cs=0 l=1 rev1=1 crypto rx=0x7f63f001f040 tx=0x7f63f001c000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-05T19:28:36.594 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.586+0000 7f63fd066700 1 -- 172.21.15.167:0/1379194618 <== mon.0 v2:172.21.15.118:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f63f0021040 con 0x5566f49b37e0 2023-10-05T19:28:36.594 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.586+0000 7f63fd066700 1 -- 172.21.15.167:0/1379194618 <== mon.0 v2:172.21.15.118:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63f0026400 con 0x5566f49b37e0 2023-10-05T19:28:36.594 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.586+0000 7f6402d38e40 1 -- 172.21.15.167:0/1379194618 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x5566f49db0c0 con 0x5566f49b37e0 2023-10-05T19:28:36.594 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.586+0000 7f63fd066700 1 -- 172.21.15.167:0/1379194618 <== mon.0 v2:172.21.15.118:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f63f00266f0 con 0x5566f49b37e0 2023-10-05T19:28:36.594 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.586+0000 7f6402d38e40 1 -- 172.21.15.167:0/1379194618 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x5566f49db500 con 0x5566f49b37e0 2023-10-05T19:28:36.595 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.590+0000 7f63fd066700 1 -- 172.21.15.167:0/1379194618 <== mon.0 v2:172.21.15.118:3300/0 4 ==== mgrmap(e 5) v1 ==== 40329+0+0 (secure 0 0 0) 0x7f63f0024040 con 0x5566f49b37e0 2023-10-05T19:28:36.596 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.590+0000 7f6402d38e40 20 rados->read ofs=0 len=0 2023-10-05T19:28:36.596 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.590+0000 7f63fd066700 1 --2- 172.21.15.167:0/1379194618 >> [v2:172.21.15.167:6824/29889,v1:172.21.15.167:6825/29889] conn(0x7f63e4030bf0 0x7f63e40330a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-05T19:28:36.596 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.590+0000 7f63fd066700 1 -- 172.21.15.167:0/1379194618 <== mon.0 v2:172.21.15.118:3300/0 5 ==== osd_map(30..30 src has 1..30) v4 ==== 6057+0+0 (secure 0 0 0) 0x7f63f0045c60 con 0x5566f49b37e0 2023-10-05T19:28:36.596 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.590+0000 7f640006c700 1 --2- 172.21.15.167:0/1379194618 >> [v2:172.21.15.167:6824/29889,v1:172.21.15.167:6825/29889] conn(0x7f63e4030bf0 0x7f63e40330a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-05T19:28:36.597 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.590+0000 7f6402d38e40 1 --2- 172.21.15.167:0/1379194618 >> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] conn(0x5566f49e4690 0x5566f49e6a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-05T19:28:36.597 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.590+0000 7f6402d38e40 1 -- 172.21.15.167:0/1379194618 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:1 5.12 5:49953fa1:::default.realm:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x5566f49e7120 con 0x5566f49e4690 2023-10-05T19:28:36.597 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.590+0000 7f640106e700 1 --2- 172.21.15.167:0/1379194618 >> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] conn(0x5566f49e4690 0x5566f49e6a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-05T19:28:36.597 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.590+0000 7f640006c700 1 --2- 172.21.15.167:0/1379194618 >> [v2:172.21.15.167:6824/29889,v1:172.21.15.167:6825/29889] conn(0x7f63e4030bf0 0x7f63e40330a0 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7f63e80044f0 tx=0x7f63e8006690 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-05T19:28:36.598 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.590+0000 7f640106e700 1 --2- 172.21.15.167:0/1379194618 >> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] conn(0x5566f49e4690 0x5566f49e6a70 crc :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-05T19:28:36.598 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.590+0000 7f640106e700 1 -- 172.21.15.167:0/1379194618 <== osd.1 v2:172.21.15.167:6800/30035 1 ==== osd_op_reply(1 default.realm [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 ==== 157+0+0 (crc 0 0 0) 0x7f63f800a500 con 0x5566f49e4690 2023-10-05T19:28:36.598 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.590+0000 7f6402d38e40 20 rados_obj.operate() r=-2 bl.length=0 2023-10-05T19:28:36.598 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.590+0000 7f6402d38e40 20 realm 2023-10-05T19:28:36.598 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.590+0000 7f6402d38e40 20 rados->read ofs=0 len=0 2023-10-05T19:28:36.598 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.590+0000 7f6402d38e40 1 -- 172.21.15.167:0/1379194618 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:2 5.12 5:49953fa1:::default.realm:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x5566f49e3d00 con 0x5566f49e4690 2023-10-05T19:28:36.599 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.594+0000 7f640106e700 1 -- 172.21.15.167:0/1379194618 <== osd.1 v2:172.21.15.167:6800/30035 2 ==== osd_op_reply(2 default.realm [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 ==== 157+0+0 (crc 0 0 0) 0x7f63f800a500 con 0x5566f49e4690 2023-10-05T19:28:36.600 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.594+0000 7f6402d38e40 20 rados_obj.operate() r=-2 bl.length=0 2023-10-05T19:28:36.600 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.594+0000 7f6402d38e40 4 RGWPeriod::init failed to init realm id : (2) No such file or directory 2023-10-05T19:28:36.600 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.594+0000 7f6402d38e40 20 rados->read ofs=0 len=0 2023-10-05T19:28:36.600 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.594+0000 7f6402d38e40 1 -- 172.21.15.167:0/1379194618 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:3 5.12 5:49953fa1:::default.realm:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x5566f49e7c00 con 0x5566f49e4690 2023-10-05T19:28:36.600 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.594+0000 7f640106e700 1 -- 172.21.15.167:0/1379194618 <== osd.1 v2:172.21.15.167:6800/30035 3 ==== osd_op_reply(3 default.realm [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 ==== 157+0+0 (crc 0 0 0) 0x7f63f800a500 con 0x5566f49e4690 2023-10-05T19:28:36.600 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.594+0000 7f6402d38e40 20 rados_obj.operate() r=-2 bl.length=0 2023-10-05T19:28:36.600 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.594+0000 7f6402d38e40 20 rados->read ofs=0 len=0 2023-10-05T19:28:36.600 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.594+0000 7f6402d38e40 1 -- 172.21.15.167:0/1379194618 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:4 5.f 5:f43b6ece:::zone_names.default:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x5566f49e7810 con 0x5566f49e4690 2023-10-05T19:28:36.600 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.594+0000 7f640106e700 1 -- 172.21.15.167:0/1379194618 <== osd.1 v2:172.21.15.167:6800/30035 4 ==== osd_op_reply(4 zone_names.default [read 0~46 out=46b] v0'0 uv1 ondisk = 0) v8 ==== 162+0+46 (crc 0 0 0) 0x7f63f800a500 con 0x5566f49e4690 2023-10-05T19:28:36.600 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.594+0000 7f6402d38e40 20 rados_obj.operate() r=0 bl.length=46 2023-10-05T19:28:36.601 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.594+0000 7f6402d38e40 20 rados->read ofs=0 len=0 2023-10-05T19:28:36.601 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.594+0000 7f6402d38e40 1 -- 172.21.15.167:0/1379194618 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:5 5.1d 5:b8374eb4:::zone_info.9a9df965-6612-45d4-a45f-6e3c889bacc2:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x5566f49e8a90 con 0x5566f49e4690 2023-10-05T19:28:36.601 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.594+0000 7f640106e700 1 -- 172.21.15.167:0/1379194618 <== osd.1 v2:172.21.15.167:6800/30035 5 ==== osd_op_reply(5 zone_info.9a9df965-6612-45d4-a45f-6e3c889bacc2 [read 0~874 out=874b] v0'0 uv2 ondisk = 0) v8 ==== 190+0+874 (crc 0 0 0) 0x7f63f800a500 con 0x5566f49e4690 2023-10-05T19:28:36.602 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.594+0000 7f6402d38e40 20 rados_obj.operate() r=0 bl.length=874 2023-10-05T19:28:36.602 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.594+0000 7f6402d38e40 20 searching for the correct realm 2023-10-05T19:28:36.602 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.594+0000 7f6402d38e40 1 --2- 172.21.15.167:0/1379194618 >> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] conn(0x5566f49ea850 0x5566f49ecc30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-05T19:28:36.602 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.594+0000 7f6402d38e40 1 -- 172.21.15.167:0/1379194618 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:6 5.0 5:00000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x5566f49ed170 con 0x5566f49ea850 2023-10-05T19:28:36.602 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.594+0000 7f640086d700 1 --2- 172.21.15.167:0/1379194618 >> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] conn(0x5566f49ea850 0x5566f49ecc30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-05T19:28:36.602 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.594+0000 7f640086d700 1 --2- 172.21.15.167:0/1379194618 >> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] conn(0x5566f49ea850 0x5566f49ecc30 crc :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-05T19:28:36.603 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.598+0000 7f640086d700 1 -- 172.21.15.167:0/1379194618 <== osd.3 v2:172.21.15.167:6808/30036 1 ==== osd_op_reply(6 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f63f0006560 con 0x5566f49ea850 2023-10-05T19:28:36.603 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.598+0000 7f640086d700 1 --2- 172.21.15.167:0/1379194618 >> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] conn(0x5566f49b3410 0x7f63f001ca10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-05T19:28:36.603 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.598+0000 7f640086d700 1 -- 172.21.15.167:0/1379194618 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:7 5.10 5:08000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x5566f49ed170 con 0x5566f49b3410 2023-10-05T19:28:36.603 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.598+0000 7f640006c700 1 --2- 172.21.15.167:0/1379194618 >> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] conn(0x5566f49b3410 0x7f63f001ca10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-05T19:28:36.604 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.598+0000 7f640006c700 1 --2- 172.21.15.167:0/1379194618 >> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] conn(0x5566f49b3410 0x7f63f001ca10 crc :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-05T19:28:36.605 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.598+0000 7f640006c700 1 -- 172.21.15.167:0/1379194618 <== osd.2 v2:172.21.15.167:6816/30037 1 ==== osd_op_reply(7 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f63e8004520 con 0x5566f49b3410 2023-10-05T19:28:36.605 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.598+0000 7f640006c700 1 --2- 172.21.15.167:0/1379194618 >> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] conn(0x7f63e8011040 0x7f63e8014950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-05T19:28:36.605 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.598+0000 7f640006c700 1 -- 172.21.15.167:0/1379194618 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:8 5.8 5:10000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x5566f49ed170 con 0x7f63e8011040 2023-10-05T19:28:36.605 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.598+0000 7f640106e700 1 --2- 172.21.15.167:0/1379194618 >> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] conn(0x7f63e8011040 0x7f63e8014950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-05T19:28:36.605 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.598+0000 7f640106e700 1 --2- 172.21.15.167:0/1379194618 >> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] conn(0x7f63e8011040 0x7f63e8014950 crc :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-05T19:28:36.606 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.602+0000 7f640106e700 1 -- 172.21.15.167:0/1379194618 <== osd.0 v2:172.21.15.118:6800/27690 1 ==== osd_op_reply(8 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f63f8014040 con 0x7f63e8011040 2023-10-05T19:28:36.607 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.602+0000 7f640106e700 1 -- 172.21.15.167:0/1379194618 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:9 5.18 5:18000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x5566f49ed170 con 0x5566f49e4690 2023-10-05T19:28:36.607 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.602+0000 7f640106e700 1 -- 172.21.15.167:0/1379194618 <== osd.1 v2:172.21.15.167:6800/30035 6 ==== osd_op_reply(9 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f63f8014040 con 0x5566f49e4690 2023-10-05T19:28:36.607 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.602+0000 7f640106e700 1 -- 172.21.15.167:0/1379194618 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:10 5.4 5:20000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x5566f49ed170 con 0x5566f49e4690 2023-10-05T19:28:36.608 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.602+0000 7f640106e700 1 -- 172.21.15.167:0/1379194618 <== osd.1 v2:172.21.15.167:6800/30035 7 ==== osd_op_reply(10 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f63f8014040 con 0x5566f49e4690 2023-10-05T19:28:36.608 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.602+0000 7f640106e700 1 -- 172.21.15.167:0/1379194618 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:11 5.14 5:28000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x5566f49ed170 con 0x7f63e8011040 2023-10-05T19:28:36.609 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.602+0000 7f640106e700 1 -- 172.21.15.167:0/1379194618 <== osd.0 v2:172.21.15.118:6800/27690 2 ==== osd_op_reply(11 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f63f8014040 con 0x7f63e8011040 2023-10-05T19:28:36.609 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.602+0000 7f640106e700 1 -- 172.21.15.167:0/1379194618 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:12 5.c 5:30000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x5566f49ed170 con 0x7f63e8011040 2023-10-05T19:28:36.610 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.606+0000 7f640106e700 1 -- 172.21.15.167:0/1379194618 <== osd.0 v2:172.21.15.118:6800/27690 3 ==== osd_op_reply(12 [pgnls start_epoch 30 out=112b] v23'1 uv1 ondisk = 1) v8 ==== 144+0+112 (crc 0 0 0) 0x7f63f8014040 con 0x7f63e8011040 2023-10-05T19:28:36.610 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.606+0000 7f640106e700 1 -- 172.21.15.167:0/1379194618 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:13 5.1c 5:38000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x5566f49ed170 con 0x5566f49b3410 2023-10-05T19:28:36.610 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.606+0000 7f640006c700 1 -- 172.21.15.167:0/1379194618 <== osd.2 v2:172.21.15.167:6816/30037 2 ==== osd_op_reply(13 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f63e8004520 con 0x5566f49b3410 2023-10-05T19:28:36.610 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.606+0000 7f640006c700 1 -- 172.21.15.167:0/1379194618 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:14 5.2 5:40000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x5566f49ed170 con 0x5566f49ea850 2023-10-05T19:28:36.611 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.606+0000 7f640086d700 1 -- 172.21.15.167:0/1379194618 <== osd.3 v2:172.21.15.167:6808/30036 2 ==== osd_op_reply(14 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f63f0006560 con 0x5566f49ea850 2023-10-05T19:28:36.611 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.606+0000 7f640086d700 1 -- 172.21.15.167:0/1379194618 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:15 5.12 5:48000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x5566f49ed170 con 0x5566f49e4690 2023-10-05T19:28:36.612 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.606+0000 7f640106e700 1 -- 172.21.15.167:0/1379194618 <== osd.1 v2:172.21.15.167:6800/30035 8 ==== osd_op_reply(15 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f63f8014040 con 0x5566f49e4690 2023-10-05T19:28:36.612 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.606+0000 7f640106e700 1 -- 172.21.15.167:0/1379194618 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:16 5.a 5:50000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x5566f49ed170 con 0x7f63e8011040 2023-10-05T19:28:36.613 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.606+0000 7f640106e700 1 -- 172.21.15.167:0/1379194618 <== osd.0 v2:172.21.15.118:6800/27690 4 ==== osd_op_reply(16 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f63f8014040 con 0x7f63e8011040 2023-10-05T19:28:36.613 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.606+0000 7f640106e700 1 -- 172.21.15.167:0/1379194618 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:17 5.1a 5:58000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x5566f49ed170 con 0x5566f49e4690 2023-10-05T19:28:36.613 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.606+0000 7f640106e700 1 -- 172.21.15.167:0/1379194618 <== osd.1 v2:172.21.15.167:6800/30035 9 ==== osd_op_reply(17 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f63f8014040 con 0x5566f49e4690 2023-10-05T19:28:36.613 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.606+0000 7f640106e700 1 -- 172.21.15.167:0/1379194618 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:18 5.6 5:60000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x5566f49ed170 con 0x5566f49b3410 2023-10-05T19:28:36.614 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.610+0000 7f640006c700 1 -- 172.21.15.167:0/1379194618 <== osd.2 v2:172.21.15.167:6816/30037 3 ==== osd_op_reply(18 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f63e8004520 con 0x5566f49b3410 2023-10-05T19:28:36.614 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.610+0000 7f640006c700 1 -- 172.21.15.167:0/1379194618 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:19 5.16 5:68000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x5566f49ed170 con 0x5566f49ea850 2023-10-05T19:28:36.615 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.610+0000 7f640086d700 1 -- 172.21.15.167:0/1379194618 <== osd.3 v2:172.21.15.167:6808/30036 3 ==== osd_op_reply(19 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f63f0006560 con 0x5566f49ea850 2023-10-05T19:28:36.615 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.610+0000 7f640086d700 1 -- 172.21.15.167:0/1379194618 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:20 5.e 5:70000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x5566f49ed170 con 0x5566f49ea850 2023-10-05T19:28:36.615 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.610+0000 7f640086d700 1 -- 172.21.15.167:0/1379194618 <== osd.3 v2:172.21.15.167:6808/30036 4 ==== osd_op_reply(20 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f63f0006560 con 0x5566f49ea850 2023-10-05T19:28:36.615 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.610+0000 7f640086d700 1 -- 172.21.15.167:0/1379194618 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:21 5.1e 5:78000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x5566f49ed170 con 0x7f63e8011040 2023-10-05T19:28:36.616 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.610+0000 7f640106e700 1 -- 172.21.15.167:0/1379194618 <== osd.0 v2:172.21.15.118:6800/27690 5 ==== osd_op_reply(21 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f63f8022040 con 0x7f63e8011040 2023-10-05T19:28:36.616 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.610+0000 7f640106e700 1 -- 172.21.15.167:0/1379194618 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:22 5.1 5:80000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x5566f49ed170 con 0x5566f49ea850 2023-10-05T19:28:36.617 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.610+0000 7f640086d700 1 -- 172.21.15.167:0/1379194618 <== osd.3 v2:172.21.15.167:6808/30036 5 ==== osd_op_reply(22 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f63f0006560 con 0x5566f49ea850 2023-10-05T19:28:36.617 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.610+0000 7f640086d700 1 -- 172.21.15.167:0/1379194618 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:23 5.11 5:88000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x5566f49ed170 con 0x5566f49ea850 2023-10-05T19:28:36.618 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.614+0000 7f640086d700 1 -- 172.21.15.167:0/1379194618 <== osd.3 v2:172.21.15.167:6808/30036 6 ==== osd_op_reply(23 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f63f0006560 con 0x5566f49ea850 2023-10-05T19:28:36.618 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.614+0000 7f640086d700 1 -- 172.21.15.167:0/1379194618 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:24 5.9 5:90000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x5566f49ed170 con 0x5566f49e4690 2023-10-05T19:28:36.618 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.614+0000 7f640106e700 1 -- 172.21.15.167:0/1379194618 <== osd.1 v2:172.21.15.167:6800/30035 10 ==== osd_op_reply(24 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f63f8022040 con 0x5566f49e4690 2023-10-05T19:28:36.619 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.614+0000 7f640106e700 1 -- 172.21.15.167:0/1379194618 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:25 5.19 5:98000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x5566f49ed170 con 0x5566f49e4690 2023-10-05T19:28:36.619 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.614+0000 7f640106e700 1 -- 172.21.15.167:0/1379194618 <== osd.1 v2:172.21.15.167:6800/30035 11 ==== osd_op_reply(25 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f63f8022040 con 0x5566f49e4690 2023-10-05T19:28:36.619 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.614+0000 7f640106e700 1 -- 172.21.15.167:0/1379194618 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:26 5.5 5:a0000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x5566f49ed170 con 0x7f63e8011040 2023-10-05T19:28:36.620 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.614+0000 7f640106e700 1 -- 172.21.15.167:0/1379194618 <== osd.0 v2:172.21.15.118:6800/27690 6 ==== osd_op_reply(26 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f63f8022040 con 0x7f63e8011040 2023-10-05T19:28:36.620 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.614+0000 7f640106e700 1 -- 172.21.15.167:0/1379194618 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:27 5.15 5:a8000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x5566f49ed170 con 0x5566f49ea850 2023-10-05T19:28:36.621 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.614+0000 7f640086d700 1 -- 172.21.15.167:0/1379194618 <== osd.3 v2:172.21.15.167:6808/30036 7 ==== osd_op_reply(27 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f63f0006560 con 0x5566f49ea850 2023-10-05T19:28:36.621 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.614+0000 7f640086d700 1 -- 172.21.15.167:0/1379194618 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:28 5.d 5:b0000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x5566f49ed170 con 0x5566f49b3410 2023-10-05T19:28:36.622 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.614+0000 7f640006c700 1 -- 172.21.15.167:0/1379194618 <== osd.2 v2:172.21.15.167:6816/30037 4 ==== osd_op_reply(28 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f63e8004520 con 0x5566f49b3410 2023-10-05T19:28:36.622 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.618+0000 7f640006c700 1 -- 172.21.15.167:0/1379194618 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:29 5.1d 5:b8000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x5566f49ed170 con 0x5566f49e4690 2023-10-05T19:28:36.622 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.618+0000 7f640106e700 1 -- 172.21.15.167:0/1379194618 <== osd.1 v2:172.21.15.167:6800/30035 12 ==== osd_op_reply(29 [pgnls start_epoch 30 out=107b] v30'2 uv2 ondisk = 1) v8 ==== 144+0+107 (crc 0 0 0) 0x7f63f8022040 con 0x5566f49e4690 2023-10-05T19:28:36.622 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.618+0000 7f640106e700 1 -- 172.21.15.167:0/1379194618 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:30 5.3 5:c0000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x5566f49ed170 con 0x7f63e8011040 2023-10-05T19:28:36.623 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.618+0000 7f640106e700 1 -- 172.21.15.167:0/1379194618 <== osd.0 v2:172.21.15.118:6800/27690 7 ==== osd_op_reply(30 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f63f8022040 con 0x7f63e8011040 2023-10-05T19:28:36.624 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.618+0000 7f640106e700 1 -- 172.21.15.167:0/1379194618 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:31 5.13 5:c8000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x5566f49ed170 con 0x5566f49ea850 2023-10-05T19:28:36.624 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.618+0000 7f640086d700 1 -- 172.21.15.167:0/1379194618 <== osd.3 v2:172.21.15.167:6808/30036 8 ==== osd_op_reply(31 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f63f0006560 con 0x5566f49ea850 2023-10-05T19:28:36.624 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.618+0000 7f640086d700 1 -- 172.21.15.167:0/1379194618 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:32 5.b 5:d0000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x5566f49ed170 con 0x7f63e8011040 2023-10-05T19:28:36.625 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.618+0000 7f640106e700 1 -- 172.21.15.167:0/1379194618 <== osd.0 v2:172.21.15.118:6800/27690 8 ==== osd_op_reply(32 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f63f8022040 con 0x7f63e8011040 2023-10-05T19:28:36.625 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.618+0000 7f640106e700 1 -- 172.21.15.167:0/1379194618 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:33 5.1b 5:d8000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x5566f49ed170 con 0x5566f49b3410 2023-10-05T19:28:36.626 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.622+0000 7f640006c700 1 -- 172.21.15.167:0/1379194618 <== osd.2 v2:172.21.15.167:6816/30037 5 ==== osd_op_reply(33 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f63e8004520 con 0x5566f49b3410 2023-10-05T19:28:36.626 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.622+0000 7f640006c700 1 -- 172.21.15.167:0/1379194618 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:34 5.7 5:e0000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x5566f49ed170 con 0x5566f49ea850 2023-10-05T19:28:36.627 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.622+0000 7f640086d700 1 -- 172.21.15.167:0/1379194618 <== osd.3 v2:172.21.15.167:6808/30036 9 ==== osd_op_reply(34 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f63f0006560 con 0x5566f49ea850 2023-10-05T19:28:36.627 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.622+0000 7f640086d700 1 -- 172.21.15.167:0/1379194618 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:35 5.17 5:e8000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x5566f49ed170 con 0x7f63e8011040 2023-10-05T19:28:36.628 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.622+0000 7f640106e700 1 -- 172.21.15.167:0/1379194618 <== osd.0 v2:172.21.15.118:6800/27690 9 ==== osd_op_reply(35 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f63f8022040 con 0x7f63e8011040 2023-10-05T19:28:36.628 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.622+0000 7f640106e700 1 -- 172.21.15.167:0/1379194618 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:36 5.f 5:f0000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x5566f49ed170 con 0x5566f49e4690 2023-10-05T19:28:36.629 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.622+0000 7f640106e700 1 -- 172.21.15.167:0/1379194618 <== osd.1 v2:172.21.15.167:6800/30035 13 ==== osd_op_reply(36 [pgnls start_epoch 30 out=115b] v23'2 uv2 ondisk = 1) v8 ==== 144+0+115 (crc 0 0 0) 0x7f63f8022040 con 0x5566f49e4690 2023-10-05T19:28:36.629 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.622+0000 7f640106e700 1 -- 172.21.15.167:0/1379194618 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:37 5.1f 5:f8000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x5566f49ed170 con 0x5566f49b3410 2023-10-05T19:28:36.630 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.622+0000 7f640006c700 1 -- 172.21.15.167:0/1379194618 <== osd.2 v2:172.21.15.167:6816/30037 6 ==== osd_op_reply(37 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f63e8004520 con 0x5566f49b3410 2023-10-05T19:28:36.630 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.622+0000 7f6402d38e40 20 RGWRados::pool_iterate: got zonegroup_info.4f1f719d-9648-417b-b10c-e5d939825419 2023-10-05T19:28:36.630 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.622+0000 7f6402d38e40 20 RGWRados::pool_iterate: got zone_info.9a9df965-6612-45d4-a45f-6e3c889bacc2 2023-10-05T19:28:36.630 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.622+0000 7f6402d38e40 20 RGWRados::pool_iterate: got zone_names.default 2023-10-05T19:28:36.630 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.622+0000 7f6402d38e40 20 RGWRados::pool_iterate: got zonegroups_names.default 2023-10-05T19:28:36.630 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.622+0000 7f6402d38e40 20 rados->read ofs=0 len=0 2023-10-05T19:28:36.630 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.622+0000 7f6402d38e40 1 -- 172.21.15.167:0/1379194618 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:38 5.12 5:49953fa1:::default.realm:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x5566f49e9600 con 0x5566f49e4690 2023-10-05T19:28:36.631 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.626+0000 7f640106e700 1 -- 172.21.15.167:0/1379194618 <== osd.1 v2:172.21.15.167:6800/30035 14 ==== osd_op_reply(38 default.realm [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 ==== 157+0+0 (crc 0 0 0) 0x7f63f8022040 con 0x5566f49e4690 2023-10-05T19:28:36.631 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.626+0000 7f6402d38e40 20 rados_obj.operate() r=-2 bl.length=0 2023-10-05T19:28:36.631 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.626+0000 7f6402d38e40 20 rados->read ofs=0 len=0 2023-10-05T19:28:36.631 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.626+0000 7f6402d38e40 1 -- 172.21.15.167:0/1379194618 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:39 5.f 5:f4c53578:::zonegroups_names.default:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x5566f49e9ef0 con 0x5566f49e4690 2023-10-05T19:28:36.631 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.626+0000 7f640106e700 1 -- 172.21.15.167:0/1379194618 <== osd.1 v2:172.21.15.167:6800/30035 15 ==== osd_op_reply(39 zonegroups_names.default [read 0~46 out=46b] v0'0 uv2 ondisk = 0) v8 ==== 168+0+46 (crc 0 0 0) 0x7f63f8022040 con 0x5566f49e4690 2023-10-05T19:28:36.632 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.626+0000 7f6402d38e40 20 rados_obj.operate() r=0 bl.length=46 2023-10-05T19:28:36.632 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.626+0000 7f6402d38e40 20 rados->read ofs=0 len=0 2023-10-05T19:28:36.632 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.626+0000 7f6402d38e40 1 -- 172.21.15.167:0/1379194618 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:40 5.c 5:30d299a5:::zonegroup_info.4f1f719d-9648-417b-b10c-e5d939825419:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x5566f49ed5b0 con 0x7f63e8011040 2023-10-05T19:28:36.632 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.626+0000 7f640106e700 1 -- 172.21.15.167:0/1379194618 <== osd.0 v2:172.21.15.118:6800/27690 10 ==== osd_op_reply(40 zonegroup_info.4f1f719d-9648-417b-b10c-e5d939825419 [read 0~362 out=362b] v0'0 uv1 ondisk = 0) v8 ==== 195+0+362 (crc 0 0 0) 0x7f63f8022040 con 0x7f63e8011040 2023-10-05T19:28:36.632 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.626+0000 7f6402d38e40 20 rados_obj.operate() r=0 bl.length=362 2023-10-05T19:28:36.633 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.626+0000 7f6402d38e40 20 rados->read ofs=0 len=0 2023-10-05T19:28:36.633 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.626+0000 7f6402d38e40 1 -- 172.21.15.167:0/1379194618 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:41 5.13 5:cbbf6c60:::converted:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x5566f49ed5b0 con 0x5566f49ea850 2023-10-05T19:28:36.634 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.626+0000 7f640086d700 1 -- 172.21.15.167:0/1379194618 <== osd.3 v2:172.21.15.167:6808/30036 10 ==== osd_op_reply(41 converted [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 ==== 153+0+0 (crc 0 0 0) 0x7f63f0006560 con 0x5566f49ea850 2023-10-05T19:28:36.634 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.626+0000 7f6402d38e40 20 rados_obj.operate() r=-2 bl.length=0 2023-10-05T19:28:36.634 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.626+0000 7f6402d38e40 20 rados->read ofs=0 len=0 2023-10-05T19:28:36.634 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.626+0000 7f6402d38e40 1 -- 172.21.15.167:0/1379194618 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:42 5.12 5:49953fa1:::default.realm:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x5566f49ed960 con 0x5566f49e4690 2023-10-05T19:28:36.634 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.626+0000 7f640106e700 1 -- 172.21.15.167:0/1379194618 <== osd.1 v2:172.21.15.167:6800/30035 16 ==== osd_op_reply(42 default.realm [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 ==== 157+0+0 (crc 0 0 0) 0x7f63f8022040 con 0x5566f49e4690 2023-10-05T19:28:36.634 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.626+0000 7f6402d38e40 20 rados_obj.operate() r=-2 bl.length=0 2023-10-05T19:28:36.634 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.626+0000 7f6402d38e40 20 rados->read ofs=0 len=0 2023-10-05T19:28:36.634 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.626+0000 7f6402d38e40 1 -- 172.21.15.167:0/1379194618 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:43 5.f 5:f4c53578:::zonegroups_names.default:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x5566f49edd60 con 0x5566f49e4690 2023-10-05T19:28:36.635 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.630+0000 7f640106e700 1 -- 172.21.15.167:0/1379194618 <== osd.1 v2:172.21.15.167:6800/30035 17 ==== osd_op_reply(43 zonegroups_names.default [read 0~46 out=46b] v0'0 uv2 ondisk = 0) v8 ==== 168+0+46 (crc 0 0 0) 0x7f63f8022040 con 0x5566f49e4690 2023-10-05T19:28:36.635 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.630+0000 7f6402d38e40 20 rados_obj.operate() r=0 bl.length=46 2023-10-05T19:28:36.635 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.630+0000 7f6402d38e40 1 -- 172.21.15.167:0/1379194618 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:44 5.0 5:00000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x5566f49eed70 con 0x5566f49ea850 2023-10-05T19:28:36.637 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.630+0000 7f640086d700 1 -- 172.21.15.167:0/1379194618 <== osd.3 v2:172.21.15.167:6808/30036 11 ==== osd_op_reply(44 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f63f0006560 con 0x5566f49ea850 2023-10-05T19:28:36.637 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.630+0000 7f640086d700 1 -- 172.21.15.167:0/1379194618 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:45 5.10 5:08000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x5566f49eed70 con 0x5566f49b3410 2023-10-05T19:28:36.638 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.630+0000 7f640006c700 1 -- 172.21.15.167:0/1379194618 <== osd.2 v2:172.21.15.167:6816/30037 7 ==== osd_op_reply(45 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f63e8004520 con 0x5566f49b3410 2023-10-05T19:28:36.638 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.630+0000 7f640006c700 1 -- 172.21.15.167:0/1379194618 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:46 5.8 5:10000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x5566f49eed70 con 0x7f63e8011040 2023-10-05T19:28:36.639 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.634+0000 7f640106e700 1 -- 172.21.15.167:0/1379194618 <== osd.0 v2:172.21.15.118:6800/27690 11 ==== osd_op_reply(46 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f63f8022040 con 0x7f63e8011040 2023-10-05T19:28:36.639 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.634+0000 7f640106e700 1 -- 172.21.15.167:0/1379194618 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:47 5.18 5:18000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x5566f49e9ef0 con 0x5566f49e4690 2023-10-05T19:28:36.639 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.634+0000 7f640106e700 1 -- 172.21.15.167:0/1379194618 <== osd.1 v2:172.21.15.167:6800/30035 18 ==== osd_op_reply(47 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f63f8022040 con 0x5566f49e4690 2023-10-05T19:28:36.640 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.634+0000 7f640106e700 1 -- 172.21.15.167:0/1379194618 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:48 5.4 5:20000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x5566f49e9ef0 con 0x5566f49e4690 2023-10-05T19:28:36.640 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.634+0000 7f640106e700 1 -- 172.21.15.167:0/1379194618 <== osd.1 v2:172.21.15.167:6800/30035 19 ==== osd_op_reply(48 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f63f8022040 con 0x5566f49e4690 2023-10-05T19:28:36.640 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.634+0000 7f640106e700 1 -- 172.21.15.167:0/1379194618 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:49 5.14 5:28000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x5566f49e9ef0 con 0x7f63e8011040 2023-10-05T19:28:36.641 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.634+0000 7f640106e700 1 -- 172.21.15.167:0/1379194618 <== osd.0 v2:172.21.15.118:6800/27690 12 ==== osd_op_reply(49 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f63f8022040 con 0x7f63e8011040 2023-10-05T19:28:36.641 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.634+0000 7f640106e700 1 -- 172.21.15.167:0/1379194618 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:50 5.c 5:30000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x5566f49e9ef0 con 0x7f63e8011040 2023-10-05T19:28:36.642 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.638+0000 7f640106e700 1 -- 172.21.15.167:0/1379194618 <== osd.0 v2:172.21.15.118:6800/27690 13 ==== osd_op_reply(50 [pgnls start_epoch 30 out=112b] v23'1 uv1 ondisk = 1) v8 ==== 144+0+112 (crc 0 0 0) 0x7f63f8022040 con 0x7f63e8011040 2023-10-05T19:28:36.642 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.638+0000 7f640106e700 1 -- 172.21.15.167:0/1379194618 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:51 5.1c 5:38000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x5566f49e9ef0 con 0x5566f49b3410 2023-10-05T19:28:36.643 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.638+0000 7f640006c700 1 -- 172.21.15.167:0/1379194618 <== osd.2 v2:172.21.15.167:6816/30037 8 ==== osd_op_reply(51 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f63e8004520 con 0x5566f49b3410 2023-10-05T19:28:36.643 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.638+0000 7f640006c700 1 -- 172.21.15.167:0/1379194618 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:52 5.2 5:40000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x5566f49e9ef0 con 0x5566f49ea850 2023-10-05T19:28:36.644 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.638+0000 7f640086d700 1 -- 172.21.15.167:0/1379194618 <== osd.3 v2:172.21.15.167:6808/30036 12 ==== osd_op_reply(52 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f63f0006560 con 0x5566f49ea850 2023-10-05T19:28:36.644 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.638+0000 7f640086d700 1 -- 172.21.15.167:0/1379194618 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:53 5.12 5:48000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x5566f49e9ef0 con 0x5566f49e4690 2023-10-05T19:28:36.644 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.638+0000 7f640106e700 1 -- 172.21.15.167:0/1379194618 <== osd.1 v2:172.21.15.167:6800/30035 20 ==== osd_op_reply(53 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f63f8022040 con 0x5566f49e4690 2023-10-05T19:28:36.644 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.638+0000 7f640106e700 1 -- 172.21.15.167:0/1379194618 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:54 5.a 5:50000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x5566f49e9ef0 con 0x7f63e8011040 2023-10-05T19:28:36.645 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.638+0000 7f640106e700 1 -- 172.21.15.167:0/1379194618 <== osd.0 v2:172.21.15.118:6800/27690 14 ==== osd_op_reply(54 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f63f8022040 con 0x7f63e8011040 2023-10-05T19:28:36.645 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.638+0000 7f640106e700 1 -- 172.21.15.167:0/1379194618 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:55 5.1a 5:58000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x5566f49e9ef0 con 0x5566f49e4690 2023-10-05T19:28:36.646 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.642+0000 7f640106e700 1 -- 172.21.15.167:0/1379194618 <== osd.1 v2:172.21.15.167:6800/30035 21 ==== osd_op_reply(55 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f63f8022040 con 0x5566f49e4690 2023-10-05T19:28:36.646 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.642+0000 7f640106e700 1 -- 172.21.15.167:0/1379194618 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:56 5.6 5:60000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x5566f49e9ef0 con 0x5566f49b3410 2023-10-05T19:28:36.647 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.642+0000 7f640006c700 1 -- 172.21.15.167:0/1379194618 <== osd.2 v2:172.21.15.167:6816/30037 9 ==== osd_op_reply(56 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f63e8004520 con 0x5566f49b3410 2023-10-05T19:28:36.647 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.642+0000 7f640006c700 1 -- 172.21.15.167:0/1379194618 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:57 5.16 5:68000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x5566f49e9ef0 con 0x5566f49ea850 2023-10-05T19:28:36.648 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.642+0000 7f640086d700 1 -- 172.21.15.167:0/1379194618 <== osd.3 v2:172.21.15.167:6808/30036 13 ==== osd_op_reply(57 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f63f0006560 con 0x5566f49ea850 2023-10-05T19:28:36.648 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.642+0000 7f640086d700 1 -- 172.21.15.167:0/1379194618 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:58 5.e 5:70000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x5566f49e9ef0 con 0x5566f49ea850 2023-10-05T19:28:36.648 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.642+0000 7f640086d700 1 -- 172.21.15.167:0/1379194618 <== osd.3 v2:172.21.15.167:6808/30036 14 ==== osd_op_reply(58 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f63f0006560 con 0x5566f49ea850 2023-10-05T19:28:36.649 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.642+0000 7f640086d700 1 -- 172.21.15.167:0/1379194618 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:59 5.1e 5:78000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x5566f49e9ef0 con 0x7f63e8011040 2023-10-05T19:28:36.649 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.642+0000 7f640106e700 1 -- 172.21.15.167:0/1379194618 <== osd.0 v2:172.21.15.118:6800/27690 15 ==== osd_op_reply(59 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f63f8022040 con 0x7f63e8011040 2023-10-05T19:28:36.650 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.642+0000 7f640106e700 1 -- 172.21.15.167:0/1379194618 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:60 5.1 5:80000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x5566f49e9ef0 con 0x5566f49ea850 2023-10-05T19:28:36.650 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.646+0000 7f640086d700 1 -- 172.21.15.167:0/1379194618 <== osd.3 v2:172.21.15.167:6808/30036 15 ==== osd_op_reply(60 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f63f0006560 con 0x5566f49ea850 2023-10-05T19:28:36.651 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.646+0000 7f640086d700 1 -- 172.21.15.167:0/1379194618 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:61 5.11 5:88000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x5566f49e9ef0 con 0x5566f49ea850 2023-10-05T19:28:36.651 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.646+0000 7f640086d700 1 -- 172.21.15.167:0/1379194618 <== osd.3 v2:172.21.15.167:6808/30036 16 ==== osd_op_reply(61 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f63f0006560 con 0x5566f49ea850 2023-10-05T19:28:36.651 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.646+0000 7f640086d700 1 -- 172.21.15.167:0/1379194618 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:62 5.9 5:90000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x5566f49e9ef0 con 0x5566f49e4690 2023-10-05T19:28:36.652 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.646+0000 7f640106e700 1 -- 172.21.15.167:0/1379194618 <== osd.1 v2:172.21.15.167:6800/30035 22 ==== osd_op_reply(62 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f63f8022040 con 0x5566f49e4690 2023-10-05T19:28:36.652 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.646+0000 7f640106e700 1 -- 172.21.15.167:0/1379194618 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:63 5.19 5:98000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x5566f49e9ef0 con 0x5566f49e4690 2023-10-05T19:28:36.653 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.646+0000 7f640106e700 1 -- 172.21.15.167:0/1379194618 <== osd.1 v2:172.21.15.167:6800/30035 23 ==== osd_op_reply(63 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f63f8022040 con 0x5566f49e4690 2023-10-05T19:28:36.653 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.646+0000 7f640106e700 1 -- 172.21.15.167:0/1379194618 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:64 5.5 5:a0000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x5566f49e9ef0 con 0x7f63e8011040 2023-10-05T19:28:36.654 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.646+0000 7f640106e700 1 -- 172.21.15.167:0/1379194618 <== osd.0 v2:172.21.15.118:6800/27690 16 ==== osd_op_reply(64 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f63f8022040 con 0x7f63e8011040 2023-10-05T19:28:36.654 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.646+0000 7f640106e700 1 -- 172.21.15.167:0/1379194618 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:65 5.15 5:a8000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x5566f49e9ef0 con 0x5566f49ea850 2023-10-05T19:28:36.655 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.650+0000 7f640086d700 1 -- 172.21.15.167:0/1379194618 <== osd.3 v2:172.21.15.167:6808/30036 17 ==== osd_op_reply(65 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f63f0006560 con 0x5566f49ea850 2023-10-05T19:28:36.655 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.650+0000 7f640086d700 1 -- 172.21.15.167:0/1379194618 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:66 5.d 5:b0000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x5566f49e9ef0 con 0x5566f49b3410 2023-10-05T19:28:36.656 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.650+0000 7f640006c700 1 -- 172.21.15.167:0/1379194618 <== osd.2 v2:172.21.15.167:6816/30037 10 ==== osd_op_reply(66 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f63e8004520 con 0x5566f49b3410 2023-10-05T19:28:36.656 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.650+0000 7f640006c700 1 -- 172.21.15.167:0/1379194618 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:67 5.1d 5:b8000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x5566f49e9ef0 con 0x5566f49e4690 2023-10-05T19:28:36.656 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.650+0000 7f640106e700 1 -- 172.21.15.167:0/1379194618 <== osd.1 v2:172.21.15.167:6800/30035 24 ==== osd_op_reply(67 [pgnls start_epoch 30 out=107b] v30'2 uv2 ondisk = 1) v8 ==== 144+0+107 (crc 0 0 0) 0x7f63f8022040 con 0x5566f49e4690 2023-10-05T19:28:36.657 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.650+0000 7f640106e700 1 -- 172.21.15.167:0/1379194618 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:68 5.3 5:c0000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x5566f49e9ef0 con 0x7f63e8011040 2023-10-05T19:28:36.657 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.650+0000 7f640106e700 1 -- 172.21.15.167:0/1379194618 <== osd.0 v2:172.21.15.118:6800/27690 17 ==== osd_op_reply(68 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f63f8022040 con 0x7f63e8011040 2023-10-05T19:28:36.657 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.650+0000 7f640106e700 1 -- 172.21.15.167:0/1379194618 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:69 5.13 5:c8000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x5566f49e9ef0 con 0x5566f49ea850 2023-10-05T19:28:36.658 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.654+0000 7f640086d700 1 -- 172.21.15.167:0/1379194618 <== osd.3 v2:172.21.15.167:6808/30036 18 ==== osd_op_reply(69 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f63f0006560 con 0x5566f49ea850 2023-10-05T19:28:36.659 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.654+0000 7f640086d700 1 -- 172.21.15.167:0/1379194618 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:70 5.b 5:d0000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x5566f49e9ef0 con 0x7f63e8011040 2023-10-05T19:28:36.659 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.654+0000 7f640106e700 1 -- 172.21.15.167:0/1379194618 <== osd.0 v2:172.21.15.118:6800/27690 18 ==== osd_op_reply(70 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f63f8022040 con 0x7f63e8011040 2023-10-05T19:28:36.660 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.654+0000 7f640106e700 1 -- 172.21.15.167:0/1379194618 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:71 5.1b 5:d8000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x5566f49e9ef0 con 0x5566f49b3410 2023-10-05T19:28:36.660 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.654+0000 7f640006c700 1 -- 172.21.15.167:0/1379194618 <== osd.2 v2:172.21.15.167:6816/30037 11 ==== osd_op_reply(71 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f63e8004520 con 0x5566f49b3410 2023-10-05T19:28:36.660 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.654+0000 7f640006c700 1 -- 172.21.15.167:0/1379194618 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:72 5.7 5:e0000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x5566f49e9ef0 con 0x5566f49ea850 2023-10-05T19:28:36.661 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.654+0000 7f640086d700 1 -- 172.21.15.167:0/1379194618 <== osd.3 v2:172.21.15.167:6808/30036 19 ==== osd_op_reply(72 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f63f0006560 con 0x5566f49ea850 2023-10-05T19:28:36.661 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.654+0000 7f640086d700 1 -- 172.21.15.167:0/1379194618 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:73 5.17 5:e8000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x5566f49e9ef0 con 0x7f63e8011040 2023-10-05T19:28:36.662 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.658+0000 7f640106e700 1 -- 172.21.15.167:0/1379194618 <== osd.0 v2:172.21.15.118:6800/27690 19 ==== osd_op_reply(73 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f63f8022040 con 0x7f63e8011040 2023-10-05T19:28:36.662 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.658+0000 7f640106e700 1 -- 172.21.15.167:0/1379194618 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:74 5.f 5:f0000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x5566f49e9ef0 con 0x5566f49e4690 2023-10-05T19:28:36.663 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.658+0000 7f640106e700 1 -- 172.21.15.167:0/1379194618 <== osd.1 v2:172.21.15.167:6800/30035 25 ==== osd_op_reply(74 [pgnls start_epoch 30 out=115b] v23'2 uv2 ondisk = 1) v8 ==== 144+0+115 (crc 0 0 0) 0x7f63f8022040 con 0x5566f49e4690 2023-10-05T19:28:36.663 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.658+0000 7f640106e700 1 -- 172.21.15.167:0/1379194618 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:75 5.1f 5:f8000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x5566f49e9ef0 con 0x5566f49b3410 2023-10-05T19:28:36.664 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.658+0000 7f640006c700 1 -- 172.21.15.167:0/1379194618 <== osd.2 v2:172.21.15.167:6816/30037 12 ==== osd_op_reply(75 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f63e8004520 con 0x5566f49b3410 2023-10-05T19:28:36.664 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.658+0000 7f6402d38e40 20 RGWRados::pool_iterate: got zonegroup_info.4f1f719d-9648-417b-b10c-e5d939825419 2023-10-05T19:28:36.664 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.658+0000 7f6402d38e40 20 RGWRados::pool_iterate: got zone_info.9a9df965-6612-45d4-a45f-6e3c889bacc2 2023-10-05T19:28:36.665 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.658+0000 7f6402d38e40 20 RGWRados::pool_iterate: got zone_names.default 2023-10-05T19:28:36.665 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.658+0000 7f6402d38e40 20 RGWRados::pool_iterate: got zonegroups_names.default 2023-10-05T19:28:36.665 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.658+0000 7f6402d38e40 20 rados->read ofs=0 len=0 2023-10-05T19:28:36.665 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.658+0000 7f6402d38e40 1 -- 172.21.15.167:0/1379194618 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:76 5.f 5:f43b6ece:::zone_names.default:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x5566f49edd60 con 0x5566f49e4690 2023-10-05T19:28:36.665 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.658+0000 7f640106e700 1 -- 172.21.15.167:0/1379194618 <== osd.1 v2:172.21.15.167:6800/30035 26 ==== osd_op_reply(76 zone_names.default [read 0~46 out=46b] v0'0 uv1 ondisk = 0) v8 ==== 162+0+46 (crc 0 0 0) 0x7f63f8022040 con 0x5566f49e4690 2023-10-05T19:28:36.666 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.658+0000 7f6402d38e40 20 rados_obj.operate() r=0 bl.length=46 2023-10-05T19:28:36.666 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.658+0000 7f6402d38e40 20 rados->read ofs=0 len=0 2023-10-05T19:28:36.666 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.658+0000 7f6402d38e40 1 -- 172.21.15.167:0/1379194618 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:77 5.1d 5:b8374eb4:::zone_info.9a9df965-6612-45d4-a45f-6e3c889bacc2:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x5566f49ee110 con 0x5566f49e4690 2023-10-05T19:28:36.667 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.658+0000 7f640106e700 1 -- 172.21.15.167:0/1379194618 <== osd.1 v2:172.21.15.167:6800/30035 27 ==== osd_op_reply(77 zone_info.9a9df965-6612-45d4-a45f-6e3c889bacc2 [read 0~874 out=874b] v0'0 uv2 ondisk = 0) v8 ==== 190+0+874 (crc 0 0 0) 0x7f63f8022040 con 0x5566f49e4690 2023-10-05T19:28:36.667 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.658+0000 7f6402d38e40 20 rados_obj.operate() r=0 bl.length=874 2023-10-05T19:28:36.667 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.662+0000 7f6402d38e40 20 rados->read ofs=0 len=0 2023-10-05T19:28:36.667 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.662+0000 7f6402d38e40 1 -- 172.21.15.167:0/1379194618 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:78 5.f 5:f4c53578:::zonegroups_names.default:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x5566f49e9a90 con 0x5566f49e4690 2023-10-05T19:28:36.667 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.662+0000 7f640106e700 1 -- 172.21.15.167:0/1379194618 <== osd.1 v2:172.21.15.167:6800/30035 28 ==== osd_op_reply(78 zonegroups_names.default [read 0~46 out=46b] v0'0 uv2 ondisk = 0) v8 ==== 168+0+46 (crc 0 0 0) 0x7f63f8022040 con 0x5566f49e4690 2023-10-05T19:28:36.667 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.662+0000 7f6402d38e40 20 rados_obj.operate() r=0 bl.length=46 2023-10-05T19:28:36.667 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.662+0000 7f6402d38e40 20 rados->read ofs=0 len=0 2023-10-05T19:28:36.667 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.662+0000 7f6402d38e40 1 -- 172.21.15.167:0/1379194618 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:79 5.c 5:30d299a5:::zonegroup_info.4f1f719d-9648-417b-b10c-e5d939825419:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x5566f49e9ab0 con 0x7f63e8011040 2023-10-05T19:28:36.668 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.662+0000 7f640106e700 1 -- 172.21.15.167:0/1379194618 <== osd.0 v2:172.21.15.118:6800/27690 20 ==== osd_op_reply(79 zonegroup_info.4f1f719d-9648-417b-b10c-e5d939825419 [read 0~362 out=362b] v0'0 uv1 ondisk = 0) v8 ==== 195+0+362 (crc 0 0 0) 0x7f63f8022040 con 0x7f63e8011040 2023-10-05T19:28:36.668 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.662+0000 7f6402d38e40 20 rados_obj.operate() r=0 bl.length=362 2023-10-05T19:28:36.668 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.662+0000 7f6402d38e40 20 rados->read ofs=0 len=0 2023-10-05T19:28:36.668 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.662+0000 7f6402d38e40 1 -- 172.21.15.167:0/1379194618 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:80 5.3 5:c0ad8cbd:::region_map:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x5566f49eeb80 con 0x7f63e8011040 2023-10-05T19:28:36.669 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.662+0000 7f640106e700 1 -- 172.21.15.167:0/1379194618 <== osd.0 v2:172.21.15.118:6800/27690 21 ==== osd_op_reply(80 region_map [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 ==== 154+0+0 (crc 0 0 0) 0x7f63f8022040 con 0x7f63e8011040 2023-10-05T19:28:36.669 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.662+0000 7f6402d38e40 20 rados_obj.operate() r=-2 bl.length=0 2023-10-05T19:28:36.669 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.662+0000 7f6402d38e40 20 zone default found 2023-10-05T19:28:36.669 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.662+0000 7f6402d38e40 4 Realm: () 2023-10-05T19:28:36.669 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.662+0000 7f6402d38e40 4 ZoneGroup: default (4f1f719d-9648-417b-b10c-e5d939825419) 2023-10-05T19:28:36.669 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.662+0000 7f6402d38e40 4 Zone: default (9a9df965-6612-45d4-a45f-6e3c889bacc2) 2023-10-05T19:28:36.670 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.662+0000 7f6402d38e40 10 cannot find current period zonegroup using local zonegroup configuration 2023-10-05T19:28:36.670 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.662+0000 7f6402d38e40 20 zonegroup default 2023-10-05T19:28:36.670 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.662+0000 7f6402d38e40 20 rados->read ofs=0 len=0 2023-10-05T19:28:36.670 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.662+0000 7f6402d38e40 1 -- 172.21.15.167:0/1379194618 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:81 5.3 5:c52100b6:::period_config.default:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x5566f49eeb80 con 0x7f63e8011040 2023-10-05T19:28:36.671 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.662+0000 7f640106e700 1 -- 172.21.15.167:0/1379194618 <== osd.0 v2:172.21.15.118:6800/27690 22 ==== osd_op_reply(81 period_config.default [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 ==== 165+0+0 (crc 0 0 0) 0x7f63f8045040 con 0x7f63e8011040 2023-10-05T19:28:36.671 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.662+0000 7f6402d38e40 20 rados_obj.operate() r=-2 bl.length=0 2023-10-05T19:28:36.671 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.662+0000 7f6402d38e40 20 rados->read ofs=0 len=0 2023-10-05T19:28:36.671 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.662+0000 7f6402d38e40 1 -- 172.21.15.167:0/1379194618 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:82 6.2 6:4347d321:::bucket.sync-source-hints.:head [call version.read in=11b,read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x5566f49f01a0 con 0x5566f49ea850 2023-10-05T19:28:36.671 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.666+0000 7f640086d700 1 -- 172.21.15.167:0/1379194618 <== osd.3 v2:172.21.15.167:6808/30036 20 ==== osd_op_reply(82 bucket.sync-source-hints. [call,read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 ==== 211+0+0 (crc 0 0 0) 0x7f63f0006560 con 0x5566f49ea850 2023-10-05T19:28:36.671 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.666+0000 7f6402d38e40 20 rados_obj.operate() r=-2 bl.length=0 2023-10-05T19:28:36.671 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.666+0000 7f6402d38e40 20 rados->read ofs=0 len=0 2023-10-05T19:28:36.671 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.666+0000 7f6402d38e40 1 -- 172.21.15.167:0/1379194618 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:83 6.b 6:d467b91b:::bucket.sync-target-hints.:head [call version.read in=11b,read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x5566f49f0550 con 0x5566f49ea850 2023-10-05T19:28:36.672 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.666+0000 7f640086d700 1 -- 172.21.15.167:0/1379194618 <== osd.3 v2:172.21.15.167:6808/30036 21 ==== osd_op_reply(83 bucket.sync-target-hints. [call,read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 ==== 211+0+0 (crc 0 0 0) 0x7f63f0006560 con 0x5566f49ea850 2023-10-05T19:28:36.672 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.666+0000 7f6402d38e40 20 rados_obj.operate() r=-2 bl.length=0 2023-10-05T19:28:36.672 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.666+0000 7f6402d38e40 20 started sync module instance, tier type = 2023-10-05T19:28:36.672 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.666+0000 7f6402d38e40 20 started zone id=9a9df965-6612-45d4-a45f-6e3c889bacc2 (name=default) with tier type = 2023-10-05T19:28:36.672 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.666+0000 7f6402d38e40 1 -- 172.21.15.167:0/1379194618 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:84 7.1f 7:f95f44c2:::notify.0:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x5566f49ef860 con 0x5566f49b3410 2023-10-05T19:28:36.675 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.670+0000 7f640006c700 1 -- 172.21.15.167:0/1379194618 <== osd.2 v2:172.21.15.167:6816/30037 13 ==== osd_op_reply(84 notify.0 [create] v30'156 uv156 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f63e802f040 con 0x5566f49b3410 2023-10-05T19:28:36.675 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.670+0000 7f6402d38e40 1 -- 172.21.15.167:0/1379194618 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:85 7.1f 7:f95f44c2:::notify.0:head [watch watch cookie 93900679085072] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x5566f49ef490 con 0x5566f49b3410 2023-10-05T19:28:36.675 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.670+0000 7f6402d38e40 1 -- 172.21.15.167:0/1379194618 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:86 7.0 7:05bf5b68:::notify.1:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x5566f49f12e0 con 0x5566f49e4690 2023-10-05T19:28:36.677 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.670+0000 7f640006c700 1 -- 172.21.15.167:0/1379194618 <== osd.2 v2:172.21.15.167:6816/30037 14 ==== osd_op_reply(85 notify.0 [watch watch cookie 93900679085072] v30'157 uv156 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f63e802f040 con 0x5566f49b3410 2023-10-05T19:28:36.678 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.674+0000 7f640106e700 1 -- 172.21.15.167:0/1379194618 <== osd.1 v2:172.21.15.167:6800/30035 29 ==== osd_op_reply(86 notify.1 [create] v30'156 uv156 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f63f8045040 con 0x5566f49e4690 2023-10-05T19:28:36.678 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.674+0000 7f6402d38e40 1 -- 172.21.15.167:0/1379194618 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:87 7.0 7:05bf5b68:::notify.1:head [watch watch cookie 93900679088672] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x5566f49f16e0 con 0x5566f49e4690 2023-10-05T19:28:36.678 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.674+0000 7f6402d38e40 1 -- 172.21.15.167:0/1379194618 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:88 7.15 7:a93a5511:::notify.2:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x5566f49f1f80 con 0x5566f49ea850 2023-10-05T19:28:36.681 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.674+0000 7f640106e700 1 -- 172.21.15.167:0/1379194618 <== osd.1 v2:172.21.15.167:6800/30035 30 ==== osd_op_reply(87 notify.1 [watch watch cookie 93900679088672] v30'157 uv156 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f63f8045040 con 0x5566f49e4690 2023-10-05T19:28:36.682 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.678+0000 7f640086d700 1 -- 172.21.15.167:0/1379194618 <== osd.3 v2:172.21.15.167:6808/30036 22 ==== osd_op_reply(88 notify.2 [create] v30'156 uv156 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f63f0006560 con 0x5566f49ea850 2023-10-05T19:28:36.682 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.678+0000 7f6402d38e40 1 -- 172.21.15.167:0/1379194618 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:89 7.15 7:a93a5511:::notify.2:head [watch watch cookie 93900679091936] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x5566f49f0f90 con 0x5566f49ea850 2023-10-05T19:28:36.682 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.678+0000 7f6402d38e40 1 -- 172.21.15.167:0/1379194618 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:90 7.e 7:7759931f:::notify.3:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x5566f49f2d10 con 0x5566f49b3410 2023-10-05T19:28:36.685 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.678+0000 7f640006c700 1 -- 172.21.15.167:0/1379194618 <== osd.2 v2:172.21.15.167:6816/30037 15 ==== osd_op_reply(90 notify.3 [create] v30'156 uv156 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f63e802f040 con 0x5566f49b3410 2023-10-05T19:28:36.685 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.678+0000 7f6402d38e40 1 -- 172.21.15.167:0/1379194618 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:91 7.e 7:7759931f:::notify.3:head [watch watch cookie 93900679096352] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x5566f49f35f0 con 0x5566f49b3410 2023-10-05T19:28:36.685 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.678+0000 7f6402d38e40 1 -- 172.21.15.167:0/1379194618 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:92 7.d 7:b4812045:::notify.4:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x5566f49f4150 con 0x5566f49e4690 2023-10-05T19:28:36.685 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.678+0000 7f640086d700 1 -- 172.21.15.167:0/1379194618 <== osd.3 v2:172.21.15.167:6808/30036 23 ==== osd_op_reply(89 notify.2 [watch watch cookie 93900679091936] v30'157 uv156 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f63f0006560 con 0x5566f49ea850 2023-10-05T19:28:36.687 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.682+0000 7f640006c700 1 -- 172.21.15.167:0/1379194618 <== osd.2 v2:172.21.15.167:6816/30037 16 ==== osd_op_reply(91 notify.3 [watch watch cookie 93900679096352] v30'157 uv156 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f63e802f040 con 0x5566f49b3410 2023-10-05T19:28:36.688 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.682+0000 7f640106e700 1 -- 172.21.15.167:0/1379194618 <== osd.1 v2:172.21.15.167:6800/30035 31 ==== osd_op_reply(92 notify.4 [create] v30'156 uv156 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f63f8045040 con 0x5566f49e4690 2023-10-05T19:28:36.688 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.682+0000 7f6402d38e40 1 -- 172.21.15.167:0/1379194618 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:93 7.d 7:b4812045:::notify.4:head [watch watch cookie 93900679095088] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x5566f49f3c20 con 0x5566f49e4690 2023-10-05T19:28:36.688 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.682+0000 7f6402d38e40 1 -- 172.21.15.167:0/1379194618 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:94 7.3 7:c609908c:::notify.5:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x5566f49f4a40 con 0x7f63e8011040 2023-10-05T19:28:36.690 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.686+0000 7f640106e700 1 -- 172.21.15.167:0/1379194618 <== osd.1 v2:172.21.15.167:6800/30035 32 ==== osd_op_reply(93 notify.4 [watch watch cookie 93900679095088] v30'157 uv156 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f63f8045040 con 0x5566f49e4690 2023-10-05T19:28:36.691 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.686+0000 7f640106e700 1 -- 172.21.15.167:0/1379194618 <== osd.0 v2:172.21.15.118:6800/27690 23 ==== osd_op_reply(94 notify.5 [create] v30'156 uv156 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f63f8045040 con 0x7f63e8011040 2023-10-05T19:28:36.691 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.686+0000 7f6402d38e40 1 -- 172.21.15.167:0/1379194618 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:95 7.3 7:c609908c:::notify.5:head [watch watch cookie 93900679098560] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x5566f49f49f0 con 0x7f63e8011040 2023-10-05T19:28:36.691 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.686+0000 7f6402d38e40 1 -- 172.21.15.167:0/1379194618 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:96 7.14 7:2b04a3e9:::notify.6:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x5566f49f54a0 con 0x7f63e8011040 2023-10-05T19:28:36.694 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.690+0000 7f640106e700 1 -- 172.21.15.167:0/1379194618 <== osd.0 v2:172.21.15.118:6800/27690 24 ==== osd_op_reply(95 notify.5 [watch watch cookie 93900679098560] v30'157 uv156 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f63f8045040 con 0x7f63e8011040 2023-10-05T19:28:36.694 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.690+0000 7f640106e700 1 -- 172.21.15.167:0/1379194618 <== osd.0 v2:172.21.15.118:6800/27690 25 ==== osd_op_reply(96 notify.6 [create] v30'156 uv156 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f63f8045040 con 0x7f63e8011040 2023-10-05T19:28:36.694 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.690+0000 7f6402d38e40 1 -- 172.21.15.167:0/1379194618 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:97 7.14 7:2b04a3e9:::notify.6:head [watch watch cookie 93900679104016] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x5566f49f54a0 con 0x7f63e8011040 2023-10-05T19:28:36.694 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.690+0000 7f6402d38e40 1 -- 172.21.15.167:0/1379194618 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:98 7.9 7:93e5b521:::notify.7:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x5566f49f5eb0 con 0x7f63e8011040 2023-10-05T19:28:36.697 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.690+0000 7f640106e700 1 -- 172.21.15.167:0/1379194618 <== osd.0 v2:172.21.15.118:6800/27690 26 ==== osd_op_reply(97 notify.6 [watch watch cookie 93900679104016] v30'157 uv156 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f63f8045040 con 0x7f63e8011040 2023-10-05T19:28:36.697 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.690+0000 7f640106e700 1 -- 172.21.15.167:0/1379194618 <== osd.0 v2:172.21.15.118:6800/27690 27 ==== osd_op_reply(98 notify.7 [create] v30'156 uv156 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f63f8045040 con 0x7f63e8011040 2023-10-05T19:28:36.697 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.690+0000 7f6402d38e40 1 -- 172.21.15.167:0/1379194618 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:99 7.9 7:93e5b521:::notify.7:head [watch watch cookie 93900679106768] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x5566f49f5eb0 con 0x7f63e8011040 2023-10-05T19:28:36.697 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.690+0000 7f6402d38e40 20 add_watcher() i=0 2023-10-05T19:28:36.697 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.690+0000 7f6402d38e40 20 add_watcher() i=1 2023-10-05T19:28:36.697 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.690+0000 7f6402d38e40 20 add_watcher() i=2 2023-10-05T19:28:36.698 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.690+0000 7f6402d38e40 20 add_watcher() i=3 2023-10-05T19:28:36.698 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.690+0000 7f6402d38e40 20 add_watcher() i=4 2023-10-05T19:28:36.698 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.690+0000 7f6402d38e40 20 add_watcher() i=5 2023-10-05T19:28:36.698 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.690+0000 7f6402d38e40 20 add_watcher() i=6 2023-10-05T19:28:36.700 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.694+0000 7f640106e700 1 -- 172.21.15.167:0/1379194618 <== osd.0 v2:172.21.15.118:6800/27690 28 ==== osd_op_reply(99 notify.7 [watch watch cookie 93900679106768] v30'157 uv156 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f63f8045040 con 0x7f63e8011040 2023-10-05T19:28:36.700 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.694+0000 7f6402d38e40 20 add_watcher() i=7 2023-10-05T19:28:36.700 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.694+0000 7f6402d38e40 2 all 8 watchers are set, enabling cache 2023-10-05T19:28:36.700 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.694+0000 7f6402d38e40 1 -- 172.21.15.167:0/1379194618 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:100 6.4 6:22d26bf9:::data_loggenerations_metadata:head [call version.check_conds in=74b,call version.read in=11b,read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x5566f49f9020 con 0x5566f49e4690 2023-10-05T19:28:36.701 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.694+0000 7f640106e700 1 -- 172.21.15.167:0/1379194618 <== osd.1 v2:172.21.15.167:6800/30035 33 ==== osd_op_reply(100 data_loggenerations_metadata [call,call out=48b,read 0~28 out=28b] v0'0 uv1 ondisk = 0) v8 ==== 256+0+76 (crc 0 0 0) 0x7f63f8045040 con 0x5566f49e4690 2023-10-05T19:28:36.701 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.694+0000 7f6402d38e40 1 -- 172.21.15.167:0/1379194618 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:101 6.4 6:22d26bf9:::data_loggenerations_metadata:head [watch watch cookie 93900679109056] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x5566f49f6300 con 0x5566f49e4690 2023-10-05T19:28:36.703 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.698+0000 7f640106e700 1 -- 172.21.15.167:0/1379194618 <== osd.1 v2:172.21.15.167:6800/30035 34 ==== osd_op_reply(101 data_loggenerations_metadata [watch watch cookie 93900679109056] v30'151 uv1 ondisk = 0) v8 ==== 172+0+0 (crc 0 0 0) 0x7f63f8045040 con 0x5566f49e4690 2023-10-05T19:28:36.703 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.698+0000 7f63d17f2700 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2023-10-05T19:28:36.703 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.698+0000 7f6402d38e40 20 check_secure_mon_conn(): auth registy supported: methods=[2] modes=[2,1] 2023-10-05T19:28:36.703 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.698+0000 7f6402d38e40 20 check_secure_mon_conn(): mode 1 is insecure 2023-10-05T19:28:36.704 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.698+0000 7f6402d38e40 1 Processor -- start 2023-10-05T19:28:36.704 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.698+0000 7f6402d38e40 1 -- start start 2023-10-05T19:28:36.704 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.698+0000 7f6402d38e40 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5566f4a03850 0x5566f4a03c20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-05T19:28:36.704 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.698+0000 7f6402d38e40 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x5566f4a04160 con 0x5566f4a03850 2023-10-05T19:28:36.705 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.698+0000 7f640006c700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5566f4a03850 0x5566f4a03c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-05T19:28:36.705 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.698+0000 7f640006c700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5566f4a03850 0x5566f4a03c20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.118:3300/0 says I am v2:172.21.15.167:41846/0 (socket says 172.21.15.167:41846) 2023-10-05T19:28:36.705 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.698+0000 7f640006c700 1 -- 172.21.15.167:0/2464221491 learned_addr learned my addr 172.21.15.167:0/2464221491 (peer_addr_for_me v2:172.21.15.167:0/0) 2023-10-05T19:28:36.705 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.698+0000 7f640006c700 1 -- 172.21.15.167:0/2464221491 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5566f4a042a0 con 0x5566f4a03850 2023-10-05T19:28:36.706 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.698+0000 7f640006c700 1 --2- 172.21.15.167:0/2464221491 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5566f4a03850 0x5566f4a03c20 secure :-1 s=READY pgs=450 cs=0 l=1 rev1=1 crypto rx=0x5566f49e82f0 tx=0x7f63e8006000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c2b1789898063dc7 server_cookie=0 in_seq=0 out_seq=0 2023-10-05T19:28:36.706 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.698+0000 7f63d0ff1700 1 -- 172.21.15.167:0/2464221491 <== mon.0 v2:172.21.15.118:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f63e802c040 con 0x5566f4a03850 2023-10-05T19:28:36.706 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.702+0000 7f63d0ff1700 1 -- 172.21.15.167:0/2464221491 <== mon.0 v2:172.21.15.118:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63e802d040 con 0x5566f4a03850 2023-10-05T19:28:36.706 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.702+0000 7f63d0ff1700 1 -- 172.21.15.167:0/2464221491 <== mon.0 v2:172.21.15.118:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f63e80289c0 con 0x5566f4a03850 2023-10-05T19:28:36.707 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.702+0000 7f6402d38e40 1 -- 172.21.15.167:0/2464221491 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5566f4a03850 msgr2=0x5566f4a03c20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-05T19:28:36.708 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.702+0000 7f6402d38e40 1 --2- 172.21.15.167:0/2464221491 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5566f4a03850 0x5566f4a03c20 secure :-1 s=READY pgs=450 cs=0 l=1 rev1=1 crypto rx=0x5566f49e82f0 tx=0x7f63e8006000 comp rx=0 tx=0).stop 2023-10-05T19:28:36.708 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.702+0000 7f6402d38e40 1 -- 172.21.15.167:0/2464221491 shutdown_connections 2023-10-05T19:28:36.708 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.702+0000 7f6402d38e40 1 --2- 172.21.15.167:0/2464221491 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5566f4a03850 0x5566f4a03c20 unknown :-1 s=CLOSED pgs=450 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-05T19:28:36.708 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.702+0000 7f6402d38e40 1 -- 172.21.15.167:0/2464221491 >> 172.21.15.167:0/2464221491 conn(0x5566f49fec10 msgr2=0x5566f49fcd60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-05T19:28:36.708 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.702+0000 7f6402d38e40 1 -- 172.21.15.167:0/2464221491 shutdown_connections 2023-10-05T19:28:36.708 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.702+0000 7f6402d38e40 1 -- 172.21.15.167:0/2464221491 wait complete. 2023-10-05T19:28:36.709 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.702+0000 7f6402d38e40 1 Processor -- start 2023-10-05T19:28:36.709 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.702+0000 7f6402d38e40 1 -- start start 2023-10-05T19:28:36.709 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.702+0000 7f6402d38e40 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5566f4a03850 0x5566f4a093e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-05T19:28:36.709 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.702+0000 7f6402d38e40 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x5566f4a0a3e0 con 0x5566f4a03850 2023-10-05T19:28:36.710 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.702+0000 7f640006c700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5566f4a03850 0x5566f4a093e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-05T19:28:36.710 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.702+0000 7f640006c700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5566f4a03850 0x5566f4a093e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.118:3300/0 says I am v2:172.21.15.167:41860/0 (socket says 172.21.15.167:41860) 2023-10-05T19:28:36.710 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.702+0000 7f640006c700 1 -- 172.21.15.167:0/142232446 learned_addr learned my addr 172.21.15.167:0/142232446 (peer_addr_for_me v2:172.21.15.167:0/0) 2023-10-05T19:28:36.710 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.702+0000 7f640006c700 1 -- 172.21.15.167:0/142232446 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f63e0001e10 con 0x5566f4a03850 2023-10-05T19:28:36.710 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.706+0000 7f640006c700 1 --2- 172.21.15.167:0/142232446 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5566f4a03850 0x5566f4a093e0 secure :-1 s=READY pgs=451 cs=0 l=1 rev1=1 crypto rx=0x5566f49fd8d0 tx=0x7f63e8036000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-05T19:28:36.711 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.706+0000 7f634affd700 1 -- 172.21.15.167:0/142232446 <== mon.0 v2:172.21.15.118:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f63e802c030 con 0x5566f4a03850 2023-10-05T19:28:36.711 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.706+0000 7f634affd700 1 -- 172.21.15.167:0/142232446 <== mon.0 v2:172.21.15.118:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63e801c040 con 0x5566f4a03850 2023-10-05T19:28:36.711 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.706+0000 7f6402d38e40 1 -- 172.21.15.167:0/142232446 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x5566f4a099e0 con 0x5566f4a03850 2023-10-05T19:28:36.711 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.706+0000 7f634affd700 1 -- 172.21.15.167:0/142232446 <== mon.0 v2:172.21.15.118:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f63e8004050 con 0x5566f4a03850 2023-10-05T19:28:36.712 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.706+0000 7f6402d38e40 1 -- 172.21.15.167:0/142232446 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x5566f4a09e20 con 0x5566f4a03850 2023-10-05T19:28:36.712 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.706+0000 7f634affd700 1 -- 172.21.15.167:0/142232446 <== mon.0 v2:172.21.15.118:3300/0 4 ==== mgrmap(e 5) v1 ==== 40329+0+0 (secure 0 0 0) 0x7f63e8003450 con 0x5566f4a03850 2023-10-05T19:28:36.713 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.706+0000 7f634affd700 1 --2- 172.21.15.167:0/142232446 >> [v2:172.21.15.167:6824/29889,v1:172.21.15.167:6825/29889] conn(0x7f6340030c00 0x7f63400330b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-05T19:28:36.713 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.706+0000 7f634affd700 1 -- 172.21.15.167:0/142232446 <== mon.0 v2:172.21.15.118:3300/0 5 ==== osd_map(30..30 src has 1..30) v4 ==== 6057+0+0 (secure 0 0 0) 0x7f63e8049ca0 con 0x5566f4a03850 2023-10-05T19:28:36.713 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.706+0000 7f640106e700 1 --2- 172.21.15.167:0/142232446 >> [v2:172.21.15.167:6824/29889,v1:172.21.15.167:6825/29889] conn(0x7f6340030c00 0x7f63400330b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-05T19:28:36.713 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.706+0000 7f6402d38e40 5 note: GC not initialized 2023-10-05T19:28:36.714 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.710+0000 7f640106e700 1 --2- 172.21.15.167:0/142232446 >> [v2:172.21.15.167:6824/29889,v1:172.21.15.167:6825/29889] conn(0x7f6340030c00 0x7f63400330b0 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x5566f49f90b0 tx=0x7f63f8000f90 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-05T19:28:36.716 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.710+0000 7f633ffff700 20 reqs_thread_entry: start 2023-10-05T19:28:36.716 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.710+0000 7f6402d38e40 20 init_complete bucket index max shards: 11 2023-10-05T19:28:36.716 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.710+0000 7f633dffb700 20 reqs_thread_entry: start 2023-10-05T19:28:36.717 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.710+0000 7f6402d38e40 10 rgw notify: Started notification manager with: 1 workers 2023-10-05T19:28:36.717 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.710+0000 7f632bfff700 20 rgw notify: INFO: next queues processing will happen at: Thu Oct 5 19:29:06 2023 2023-10-05T19:28:36.717 INFO:teuthology.orchestra.run.smithi167.stderr: 2023-10-05T19:28:36.718 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.710+0000 7f632b7fe700 20 reqs_thread_entry: start 2023-10-05T19:28:36.718 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.710+0000 7f6402d38e40 1 --2- 172.21.15.167:0/142232446 >> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] conn(0x5566f4a70950 0x5566f4a72d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-05T19:28:36.718 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.710+0000 7f6402d38e40 1 -- 172.21.15.167:0/142232446 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:1 6.8 6:163360d8:::2023-10-05-19--:head [delete] snapc 0=[] ondisk+write+peerstat_old+known_if_redirected+supports_pool_eio e30) v8 -- 0x5566f4a73270 con 0x5566f4a70950 2023-10-05T19:28:36.718 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.710+0000 7f640086d700 1 --2- 172.21.15.167:0/142232446 >> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] conn(0x5566f4a70950 0x5566f4a72d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-05T19:28:36.718 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.714+0000 7f640086d700 1 --2- 172.21.15.167:0/142232446 >> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] conn(0x5566f4a70950 0x5566f4a72d30 crc :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-05T19:28:36.721 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.714+0000 7f640086d700 1 -- 172.21.15.167:0/142232446 <== osd.3 v2:172.21.15.167:6808/30036 1 ==== osd_op_reply(1 2023-10-05-19-- [delete] v30'29 uv29 ondisk = 0) v8 ==== 159+0+0 (crc 0 0 0) 0x7f63f0006560 con 0x5566f4a70950 2023-10-05T19:28:36.721 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.714+0000 7f6402d38e40 1 -- 172.21.15.167:0/1379194618 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:102 7.1f 7:f95f44c2:::notify.0:head [watch unwatch cookie 93900679085072] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x5566f4a2fa90 con 0x5566f49b3410 2023-10-05T19:28:36.724 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.718+0000 7f640006c700 1 -- 172.21.15.167:0/1379194618 <== osd.2 v2:172.21.15.167:6816/30037 17 ==== osd_op_reply(102 notify.0 [watch unwatch cookie 93900679085072] v30'158 uv156 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f63e802f040 con 0x5566f49b3410 2023-10-05T19:28:36.724 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.718+0000 7f6402d38e40 20 remove_watcher() i=0 2023-10-05T19:28:36.724 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.718+0000 7f6402d38e40 2 removed watcher, disabling cache 2023-10-05T19:28:36.725 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.718+0000 7f6402d38e40 1 -- 172.21.15.167:0/1379194618 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:103 7.0 7:05bf5b68:::notify.1:head [watch unwatch cookie 93900679088672] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x5566f49efc10 con 0x5566f49e4690 2023-10-05T19:28:36.727 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.722+0000 7f640106e700 1 -- 172.21.15.167:0/1379194618 <== osd.1 v2:172.21.15.167:6800/30035 35 ==== osd_op_reply(103 notify.1 [watch unwatch cookie 93900679088672] v30'158 uv156 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f63f8045040 con 0x5566f49e4690 2023-10-05T19:28:36.728 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.722+0000 7f6402d38e40 20 remove_watcher() i=1 2023-10-05T19:28:36.728 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.722+0000 7f6402d38e40 1 -- 172.21.15.167:0/1379194618 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:104 7.15 7:a93a5511:::notify.2:head [watch unwatch cookie 93900679091936] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x5566f49efc10 con 0x5566f49ea850 2023-10-05T19:28:36.730 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.726+0000 7f640086d700 1 -- 172.21.15.167:0/1379194618 <== osd.3 v2:172.21.15.167:6808/30036 24 ==== osd_op_reply(104 notify.2 [watch unwatch cookie 93900679091936] v30'158 uv156 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f63f0006560 con 0x5566f49ea850 2023-10-05T19:28:36.730 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.726+0000 7f6402d38e40 20 remove_watcher() i=2 2023-10-05T19:28:36.730 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.726+0000 7f6402d38e40 1 -- 172.21.15.167:0/1379194618 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:105 7.e 7:7759931f:::notify.3:head [watch unwatch cookie 93900679096352] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x5566f49f16e0 con 0x5566f49b3410 2023-10-05T19:28:36.733 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.726+0000 7f640006c700 1 -- 172.21.15.167:0/1379194618 <== osd.2 v2:172.21.15.167:6816/30037 18 ==== osd_op_reply(105 notify.3 [watch unwatch cookie 93900679096352] v30'158 uv156 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f63e802f040 con 0x5566f49b3410 2023-10-05T19:28:36.733 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.726+0000 7f6402d38e40 20 remove_watcher() i=3 2023-10-05T19:28:36.733 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.726+0000 7f6402d38e40 1 -- 172.21.15.167:0/1379194618 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:106 7.d 7:b4812045:::notify.4:head [watch unwatch cookie 93900679095088] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x5566f49efc10 con 0x5566f49e4690 2023-10-05T19:28:36.735 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.730+0000 7f640106e700 1 -- 172.21.15.167:0/1379194618 <== osd.1 v2:172.21.15.167:6800/30035 36 ==== osd_op_reply(106 notify.4 [watch unwatch cookie 93900679095088] v30'158 uv156 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f63f8045040 con 0x5566f49e4690 2023-10-05T19:28:36.736 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.730+0000 7f6402d38e40 20 remove_watcher() i=4 2023-10-05T19:28:36.736 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.730+0000 7f6402d38e40 1 -- 172.21.15.167:0/1379194618 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:107 7.3 7:c609908c:::notify.5:head [watch unwatch cookie 93900679098560] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x5566f49f16e0 con 0x7f63e8011040 2023-10-05T19:28:36.738 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.734+0000 7f640106e700 1 -- 172.21.15.167:0/1379194618 <== osd.0 v2:172.21.15.118:6800/27690 29 ==== osd_op_reply(107 notify.5 [watch unwatch cookie 93900679098560] v30'158 uv156 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f63f8045040 con 0x7f63e8011040 2023-10-05T19:28:36.738 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.734+0000 7f6402d38e40 20 remove_watcher() i=5 2023-10-05T19:28:36.739 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.734+0000 7f6402d38e40 1 -- 172.21.15.167:0/1379194618 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:108 7.14 7:2b04a3e9:::notify.6:head [watch unwatch cookie 93900679104016] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x5566f49f16e0 con 0x7f63e8011040 2023-10-05T19:28:36.741 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.734+0000 7f640106e700 1 -- 172.21.15.167:0/1379194618 <== osd.0 v2:172.21.15.118:6800/27690 30 ==== osd_op_reply(108 notify.6 [watch unwatch cookie 93900679104016] v30'158 uv156 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f63f8045040 con 0x7f63e8011040 2023-10-05T19:28:36.742 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.734+0000 7f6402d38e40 20 remove_watcher() i=6 2023-10-05T19:28:36.742 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.734+0000 7f6402d38e40 1 -- 172.21.15.167:0/1379194618 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:109 7.9 7:93e5b521:::notify.7:head [watch unwatch cookie 93900679106768] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x5566f49f16e0 con 0x7f63e8011040 2023-10-05T19:28:36.744 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.738+0000 7f640106e700 1 -- 172.21.15.167:0/1379194618 <== osd.0 v2:172.21.15.118:6800/27690 31 ==== osd_op_reply(109 notify.7 [watch unwatch cookie 93900679106768] v30'158 uv156 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f63f8045040 con 0x7f63e8011040 2023-10-05T19:28:36.744 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.738+0000 7f6402d38e40 20 remove_watcher() i=7 2023-10-05T19:28:36.746 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.742+0000 7f63d17f2700 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2023-10-05T19:28:36.746 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.742+0000 7f6402d38e40 1 -- 172.21.15.167:0/1379194618 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:110 6.4 6:22d26bf9:::data_loggenerations_metadata:head [watch unwatch cookie 93900679109056] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x5566f49f16e0 con 0x5566f49e4690 2023-10-05T19:28:36.750 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.742+0000 7f640106e700 1 -- 172.21.15.167:0/1379194618 <== osd.1 v2:172.21.15.167:6800/30035 37 ==== osd_op_reply(110 data_loggenerations_metadata [watch unwatch cookie 93900679109056] v30'152 uv1 ondisk = 0) v8 ==== 172+0+0 (crc 0 0 0) 0x7f63f8045040 con 0x5566f49e4690 2023-10-05T19:28:36.750 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.742+0000 7f6402d38e40 1 -- 172.21.15.167:0/1379194618 >> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] conn(0x7f63e8011040 msgr2=0x7f63e8014950 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-05T19:28:36.750 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.742+0000 7f6402d38e40 1 --2- 172.21.15.167:0/1379194618 >> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] conn(0x7f63e8011040 0x7f63e8014950 crc :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-05T19:28:36.750 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.742+0000 7f6402d38e40 1 -- 172.21.15.167:0/1379194618 >> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] conn(0x5566f49e4690 msgr2=0x5566f49e6a70 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-05T19:28:36.750 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.742+0000 7f6402d38e40 1 --2- 172.21.15.167:0/1379194618 >> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] conn(0x5566f49e4690 0x5566f49e6a70 crc :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-05T19:28:36.750 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.742+0000 7f6402d38e40 1 -- 172.21.15.167:0/1379194618 >> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] conn(0x5566f49b3410 msgr2=0x7f63f001ca10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-05T19:28:36.750 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.742+0000 7f6402d38e40 1 --2- 172.21.15.167:0/1379194618 >> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] conn(0x5566f49b3410 0x7f63f001ca10 crc :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-05T19:28:36.750 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.742+0000 7f6402d38e40 1 -- 172.21.15.167:0/1379194618 >> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] conn(0x5566f49ea850 msgr2=0x5566f49ecc30 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-05T19:28:36.750 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.742+0000 7f6402d38e40 1 --2- 172.21.15.167:0/1379194618 >> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] conn(0x5566f49ea850 0x5566f49ecc30 crc :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-05T19:28:36.751 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.742+0000 7f6402d38e40 1 -- 172.21.15.167:0/1379194618 >> [v2:172.21.15.167:6824/29889,v1:172.21.15.167:6825/29889] conn(0x7f63e4030bf0 msgr2=0x7f63e40330a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-05T19:28:36.751 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.742+0000 7f6402d38e40 1 --2- 172.21.15.167:0/1379194618 >> [v2:172.21.15.167:6824/29889,v1:172.21.15.167:6825/29889] conn(0x7f63e4030bf0 0x7f63e40330a0 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7f63e80044f0 tx=0x7f63e8006690 comp rx=0 tx=0).stop 2023-10-05T19:28:36.751 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.742+0000 7f6402d38e40 1 -- 172.21.15.167:0/1379194618 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5566f49b37e0 msgr2=0x5566f49dab80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-05T19:28:36.751 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.742+0000 7f6402d38e40 1 --2- 172.21.15.167:0/1379194618 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5566f49b37e0 0x5566f49dab80 secure :-1 s=READY pgs=449 cs=0 l=1 rev1=1 crypto rx=0x7f63f001f040 tx=0x7f63f001c000 comp rx=0 tx=0).stop 2023-10-05T19:28:36.751 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.742+0000 7f640106e700 1 -- 172.21.15.167:0/1379194618 reap_dead start 2023-10-05T19:28:36.751 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.742+0000 7f640106e700 1 -- 172.21.15.167:0/1379194618 reap_dead start 2023-10-05T19:28:36.751 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.742+0000 7f6402d38e40 1 -- 172.21.15.167:0/1379194618 shutdown_connections 2023-10-05T19:28:36.751 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.742+0000 7f6402d38e40 1 -- 172.21.15.167:0/1379194618 >> 172.21.15.167:0/1379194618 conn(0x5566f49ad720 msgr2=0x5566f49ce670 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-05T19:28:36.751 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.742+0000 7f6402d38e40 1 -- 172.21.15.167:0/1379194618 shutdown_connections 2023-10-05T19:28:36.751 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.742+0000 7f6402d38e40 1 -- 172.21.15.167:0/1379194618 wait complete. 2023-10-05T19:28:36.752 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.746+0000 7f6402d38e40 1 -- 172.21.15.167:0/142232446 >> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] conn(0x5566f4a70950 msgr2=0x5566f4a72d30 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-05T19:28:36.753 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.746+0000 7f6402d38e40 1 --2- 172.21.15.167:0/142232446 >> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] conn(0x5566f4a70950 0x5566f4a72d30 crc :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-05T19:28:36.753 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.746+0000 7f6402d38e40 1 -- 172.21.15.167:0/142232446 >> [v2:172.21.15.167:6824/29889,v1:172.21.15.167:6825/29889] conn(0x7f6340030c00 msgr2=0x7f63400330b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-05T19:28:36.753 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.746+0000 7f6402d38e40 1 --2- 172.21.15.167:0/142232446 >> [v2:172.21.15.167:6824/29889,v1:172.21.15.167:6825/29889] conn(0x7f6340030c00 0x7f63400330b0 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x5566f49f90b0 tx=0x7f63f8000f90 comp rx=0 tx=0).stop 2023-10-05T19:28:36.753 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.746+0000 7f6402d38e40 1 -- 172.21.15.167:0/142232446 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5566f4a03850 msgr2=0x5566f4a093e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-05T19:28:36.753 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.746+0000 7f6402d38e40 1 --2- 172.21.15.167:0/142232446 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5566f4a03850 0x5566f4a093e0 secure :-1 s=READY pgs=451 cs=0 l=1 rev1=1 crypto rx=0x5566f49fd8d0 tx=0x7f63e8036000 comp rx=0 tx=0).stop 2023-10-05T19:28:36.753 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.746+0000 7f6402d38e40 1 -- 172.21.15.167:0/142232446 shutdown_connections 2023-10-05T19:28:36.753 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.746+0000 7f6402d38e40 1 --2- 172.21.15.167:0/142232446 >> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] conn(0x5566f4a70950 0x5566f4a72d30 unknown :-1 s=CLOSED pgs=73 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-05T19:28:36.753 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.746+0000 7f6402d38e40 1 --2- 172.21.15.167:0/142232446 >> [v2:172.21.15.167:6824/29889,v1:172.21.15.167:6825/29889] conn(0x7f6340030c00 0x7f63400330b0 unknown :-1 s=CLOSED pgs=157 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-05T19:28:36.753 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.746+0000 7f6402d38e40 1 --2- 172.21.15.167:0/142232446 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5566f4a03850 0x5566f4a093e0 unknown :-1 s=CLOSED pgs=451 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-05T19:28:36.753 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.746+0000 7f6402d38e40 1 -- 172.21.15.167:0/142232446 >> 172.21.15.167:0/142232446 conn(0x5566f49fec10 msgr2=0x5566f49fccb0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-05T19:28:36.753 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.746+0000 7f6402d38e40 1 -- 172.21.15.167:0/142232446 shutdown_connections 2023-10-05T19:28:36.754 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.746+0000 7f6402d38e40 1 -- 172.21.15.167:0/142232446 wait complete. 2023-10-05T19:28:36.756 INFO:tasks.util.rgw:rgwadmin: client.0 : ['log', 'show', '--object', '2023-10-05-19-9a9df965-6612-45d4-a45f-6e3c889bacc2.4223.4-myfoo4'] 2023-10-05T19:28:36.756 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'ceph', 'log', 'show', '--object', '2023-10-05-19-9a9df965-6612-45d4-a45f-6e3c889bacc2.4223.4-myfoo4'] 2023-10-05T19:28:36.757 DEBUG:teuthology.orchestra.run.smithi167:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster ceph log show --object 2023-10-05-19-9a9df965-6612-45d4-a45f-6e3c889bacc2.4223.4-myfoo4 2023-10-05T19:28:36.798 INFO:teuthology.orchestra.run.smithi167.stderr:ignoring --setuser ceph since I am not root 2023-10-05T19:28:36.798 INFO:teuthology.orchestra.run.smithi167.stderr:ignoring --setgroup ceph since I am not root 2023-10-05T19:28:36.798 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.794+0000 7fc7c4047e40 1 Processor -- start 2023-10-05T19:28:36.798 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.794+0000 7fc7c4047e40 1 -- start start 2023-10-05T19:28:36.799 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.794+0000 7fc7c4047e40 1 --2- >> v2:172.21.15.118:3300/0 conn(0x560a577dc5f0 0x560a577de9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-05T19:28:36.799 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.794+0000 7fc7c4047e40 1 -- --> v1:172.21.15.118:6789/0 -- auth(proto 0 26 bytes epoch 0) v1 -- 0x560a57722bd0 con 0x560a577d8010 2023-10-05T19:28:36.799 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.794+0000 7fc7c4047e40 1 -- --> v2:172.21.15.118:3300/0 -- mon_getmap magic: 0 v1 -- 0x560a576dfee0 con 0x560a577dc5f0 2023-10-05T19:28:36.800 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.794+0000 7fc7c137b700 1 --2- >> v2:172.21.15.118:3300/0 conn(0x560a577dc5f0 0x560a577de9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-05T19:28:36.800 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.794+0000 7fc7c1b7c700 1 --1- >> v1:172.21.15.118:6789/0 conn(0x560a577d8010 0x560a577d83e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.118:6789/0 says I am v1:172.21.15.167:59336/0 (socket says 172.21.15.167:59336) 2023-10-05T19:28:36.800 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.794+0000 7fc7c137b700 1 --2- >> v2:172.21.15.118:3300/0 conn(0x560a577dc5f0 0x560a577de9d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.118:3300/0 says I am v2:172.21.15.167:41870/0 (socket says 172.21.15.167:41870) 2023-10-05T19:28:36.800 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.794+0000 7fc7c1b7c700 1 -- 172.21.15.167:0/1778158049 learned_addr learned my addr 172.21.15.167:0/1778158049 (peer_addr_for_me v1:172.21.15.167:0/0) 2023-10-05T19:28:36.800 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.794+0000 7fc7c137b700 1 -- 172.21.15.167:0/1778158049 >> v1:172.21.15.118:6789/0 conn(0x560a577d8010 legacy=0x560a577d83e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-05T19:28:36.801 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.794+0000 7fc7c137b700 1 -- 172.21.15.167:0/1778158049 --> v2:172.21.15.118:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x560a577d8c20 con 0x560a577dc5f0 2023-10-05T19:28:36.801 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.794+0000 7fc7c137b700 1 --2- 172.21.15.167:0/1778158049 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x560a577dc5f0 0x560a577de9d0 secure :-1 s=READY pgs=453 cs=0 l=1 rev1=1 crypto rx=0x7fc7ac00a520 tx=0x7fc7ac005d40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bd6fbc95efd014dd server_cookie=0 in_seq=0 out_seq=0 2023-10-05T19:28:36.802 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.794+0000 7fc7c0b7a700 1 -- 172.21.15.167:0/1778158049 <== mon.0 v2:172.21.15.118:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fc7ac00f590 con 0x560a577dc5f0 2023-10-05T19:28:36.802 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.794+0000 7fc7c0b7a700 1 -- 172.21.15.167:0/1778158049 <== mon.0 v2:172.21.15.118:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc7ac00f6f0 con 0x560a577dc5f0 2023-10-05T19:28:36.802 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.794+0000 7fc7c0b7a700 1 -- 172.21.15.167:0/1778158049 <== mon.0 v2:172.21.15.118:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fc7ac00f880 con 0x560a577dc5f0 2023-10-05T19:28:36.802 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.794+0000 7fc7c4047e40 1 -- 172.21.15.167:0/1778158049 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x560a577dc5f0 msgr2=0x560a577de9d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-05T19:28:36.802 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.794+0000 7fc7c4047e40 1 --2- 172.21.15.167:0/1778158049 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x560a577dc5f0 0x560a577de9d0 secure :-1 s=READY pgs=453 cs=0 l=1 rev1=1 crypto rx=0x7fc7ac00a520 tx=0x7fc7ac005d40 comp rx=0 tx=0).stop 2023-10-05T19:28:36.802 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.794+0000 7fc7c4047e40 1 -- 172.21.15.167:0/1778158049 shutdown_connections 2023-10-05T19:28:36.802 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.794+0000 7fc7c4047e40 1 --2- 172.21.15.167:0/1778158049 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x560a577dc5f0 0x560a577de9d0 unknown :-1 s=CLOSED pgs=453 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-05T19:28:36.803 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.794+0000 7fc7c4047e40 1 -- 172.21.15.167:0/1778158049 >> 172.21.15.167:0/1778158049 conn(0x560a5774b860 msgr2=0x560a5774bc60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-05T19:28:36.803 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.794+0000 7fc7c4047e40 1 -- 172.21.15.167:0/1778158049 shutdown_connections 2023-10-05T19:28:36.803 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.794+0000 7fc7c4047e40 1 -- 172.21.15.167:0/1778158049 wait complete. 2023-10-05T19:28:36.803 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.798+0000 7fc7c4047e40 1 Processor -- start 2023-10-05T19:28:36.803 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.798+0000 7fc7c4047e40 1 -- start start 2023-10-05T19:28:36.804 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.798+0000 7fc7c4047e40 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x560a577d8010 0x560a577f71d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-05T19:28:36.804 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.798+0000 7fc7c4047e40 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x560a576a3e30 con 0x560a577d8010 2023-10-05T19:28:36.805 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.798+0000 7fc7c1b7c700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x560a577d8010 0x560a577f71d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-05T19:28:36.805 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.798+0000 7fc7c1b7c700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x560a577d8010 0x560a577f71d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.118:3300/0 says I am v2:172.21.15.167:41882/0 (socket says 172.21.15.167:41882) 2023-10-05T19:28:36.805 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.798+0000 7fc7c1b7c700 1 -- 172.21.15.167:0/2184261540 learned_addr learned my addr 172.21.15.167:0/2184261540 (peer_addr_for_me v2:172.21.15.167:0/0) 2023-10-05T19:28:36.805 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.798+0000 7fc7c1b7c700 1 -- 172.21.15.167:0/2184261540 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc7ac00a0e0 con 0x560a577d8010 2023-10-05T19:28:36.805 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.798+0000 7fc7c1b7c700 1 --2- 172.21.15.167:0/2184261540 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x560a577d8010 0x560a577f71d0 secure :-1 s=READY pgs=454 cs=0 l=1 rev1=1 crypto rx=0x7fc7b4001c30 tx=0x7fc7b40017e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ea940e8a124094a5 server_cookie=0 in_seq=0 out_seq=0 2023-10-05T19:28:36.806 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.798+0000 7fc7bb7fe700 1 -- 172.21.15.167:0/2184261540 <== mon.0 v2:172.21.15.118:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fc7b400bbf0 con 0x560a577d8010 2023-10-05T19:28:36.806 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.798+0000 7fc7bb7fe700 1 -- 172.21.15.167:0/2184261540 <== mon.0 v2:172.21.15.118:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc7b400bd50 con 0x560a577d8010 2023-10-05T19:28:36.806 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.798+0000 7fc7bb7fe700 1 -- 172.21.15.167:0/2184261540 <== mon.0 v2:172.21.15.118:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fc7b400e470 con 0x560a577d8010 2023-10-05T19:28:36.807 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.802+0000 7fc7c4047e40 1 -- 172.21.15.167:0/2184261540 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x560a577d8010 msgr2=0x560a577f71d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-05T19:28:36.808 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.802+0000 7fc7c4047e40 1 --2- 172.21.15.167:0/2184261540 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x560a577d8010 0x560a577f71d0 secure :-1 s=READY pgs=454 cs=0 l=1 rev1=1 crypto rx=0x7fc7b4001c30 tx=0x7fc7b40017e0 comp rx=0 tx=0).stop 2023-10-05T19:28:36.808 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.802+0000 7fc7c4047e40 1 -- 172.21.15.167:0/2184261540 shutdown_connections 2023-10-05T19:28:36.808 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.802+0000 7fc7c4047e40 1 --2- 172.21.15.167:0/2184261540 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x560a577d8010 0x560a577f71d0 unknown :-1 s=CLOSED pgs=454 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-05T19:28:36.808 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.802+0000 7fc7c4047e40 1 -- 172.21.15.167:0/2184261540 >> 172.21.15.167:0/2184261540 conn(0x560a5774b860 msgr2=0x560a577f2d90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-05T19:28:36.808 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.802+0000 7fc7c4047e40 1 -- 172.21.15.167:0/2184261540 shutdown_connections 2023-10-05T19:28:36.808 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.802+0000 7fc7c4047e40 1 -- 172.21.15.167:0/2184261540 wait complete. 2023-10-05T19:28:36.809 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.802+0000 7fc7c4047e40 1 Processor -- start 2023-10-05T19:28:36.809 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.802+0000 7fc7c4047e40 1 -- start start 2023-10-05T19:28:36.809 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.802+0000 7fc7c4047e40 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x560a577d8010 0x560a577fdf60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-05T19:28:36.810 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.802+0000 7fc7c4047e40 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x560a57676ed0 con 0x560a577d8010 2023-10-05T19:28:36.810 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.806+0000 7fc7c1b7c700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x560a577d8010 0x560a577fdf60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-05T19:28:36.810 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.806+0000 7fc7c1b7c700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x560a577d8010 0x560a577fdf60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.118:3300/0 says I am v2:172.21.15.167:41884/0 (socket says 172.21.15.167:41884) 2023-10-05T19:28:36.810 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.806+0000 7fc7c1b7c700 1 -- 172.21.15.167:0/287475553 learned_addr learned my addr 172.21.15.167:0/287475553 (peer_addr_for_me v2:172.21.15.167:0/0) 2023-10-05T19:28:36.811 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.806+0000 7fc7c1b7c700 1 -- 172.21.15.167:0/287475553 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x560a57722bd0 con 0x560a577d8010 2023-10-05T19:28:36.811 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.806+0000 7fc7c1b7c700 1 --2- 172.21.15.167:0/287475553 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x560a577d8010 0x560a577fdf60 secure :-1 s=READY pgs=455 cs=0 l=1 rev1=1 crypto rx=0x7fc7b4004000 tx=0x7fc7b4016000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-05T19:28:36.811 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.806+0000 7fc7b9ffb700 1 -- 172.21.15.167:0/287475553 <== mon.0 v2:172.21.15.118:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fc7b401f070 con 0x560a577d8010 2023-10-05T19:28:36.812 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.806+0000 7fc7b9ffb700 1 -- 172.21.15.167:0/287475553 <== mon.0 v2:172.21.15.118:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc7b40197b0 con 0x560a577d8010 2023-10-05T19:28:36.812 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.806+0000 7fc7c4047e40 1 -- 172.21.15.167:0/287475553 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x560a577fe4a0 con 0x560a577d8010 2023-10-05T19:28:36.812 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.806+0000 7fc7b9ffb700 1 -- 172.21.15.167:0/287475553 <== mon.0 v2:172.21.15.118:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fc7b4019980 con 0x560a577d8010 2023-10-05T19:28:36.812 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.806+0000 7fc7c4047e40 1 -- 172.21.15.167:0/287475553 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x560a577fe8e0 con 0x560a577d8010 2023-10-05T19:28:36.813 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.806+0000 7fc7b9ffb700 1 -- 172.21.15.167:0/287475553 <== mon.0 v2:172.21.15.118:3300/0 4 ==== mgrmap(e 5) v1 ==== 40329+0+0 (secure 0 0 0) 0x7fc7b4022020 con 0x560a577d8010 2023-10-05T19:28:36.814 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.806+0000 7fc7c4047e40 20 rados->read ofs=0 len=0 2023-10-05T19:28:36.814 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.806+0000 7fc7b9ffb700 1 --2- 172.21.15.167:0/287475553 >> [v2:172.21.15.167:6824/29889,v1:172.21.15.167:6825/29889] conn(0x7fc7a4030c50 0x7fc7a4033100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-05T19:28:36.814 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.806+0000 7fc7b9ffb700 1 -- 172.21.15.167:0/287475553 <== mon.0 v2:172.21.15.118:3300/0 5 ==== osd_map(30..30 src has 1..30) v4 ==== 6057+0+0 (secure 0 0 0) 0x7fc7b4044e10 con 0x560a577d8010 2023-10-05T19:28:36.814 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.806+0000 7fc7c137b700 1 --2- 172.21.15.167:0/287475553 >> [v2:172.21.15.167:6824/29889,v1:172.21.15.167:6825/29889] conn(0x7fc7a4030c50 0x7fc7a4033100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-05T19:28:36.814 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.810+0000 7fc7c4047e40 1 --2- 172.21.15.167:0/287475553 >> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] conn(0x560a577dc5f0 0x560a578099f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-05T19:28:36.814 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.810+0000 7fc7c4047e40 1 -- 172.21.15.167:0/287475553 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:1 5.12 5:49953fa1:::default.realm:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x560a5780a0f0 con 0x560a577dc5f0 2023-10-05T19:28:36.814 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.810+0000 7fc7c237d700 1 --2- 172.21.15.167:0/287475553 >> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] conn(0x560a577dc5f0 0x560a578099f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-05T19:28:36.815 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.810+0000 7fc7c137b700 1 --2- 172.21.15.167:0/287475553 >> [v2:172.21.15.167:6824/29889,v1:172.21.15.167:6825/29889] conn(0x7fc7a4030c50 0x7fc7a4033100 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7fc7ac005d80 tx=0x7fc7ac014000 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-05T19:28:36.815 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.810+0000 7fc7c237d700 1 --2- 172.21.15.167:0/287475553 >> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] conn(0x560a577dc5f0 0x560a578099f0 crc :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-05T19:28:36.816 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.810+0000 7fc7c237d700 1 -- 172.21.15.167:0/287475553 <== osd.1 v2:172.21.15.167:6800/30035 1 ==== osd_op_reply(1 default.realm [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 ==== 157+0+0 (crc 0 0 0) 0x7fc7bc00a500 con 0x560a577dc5f0 2023-10-05T19:28:36.816 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.810+0000 7fc7c4047e40 20 rados_obj.operate() r=-2 bl.length=0 2023-10-05T19:28:36.816 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.810+0000 7fc7c4047e40 20 realm 2023-10-05T19:28:36.816 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.810+0000 7fc7c4047e40 20 rados->read ofs=0 len=0 2023-10-05T19:28:36.816 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.810+0000 7fc7c4047e40 1 -- 172.21.15.167:0/287475553 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:2 5.12 5:49953fa1:::default.realm:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x560a57807050 con 0x560a577dc5f0 2023-10-05T19:28:36.817 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.810+0000 7fc7c237d700 1 -- 172.21.15.167:0/287475553 <== osd.1 v2:172.21.15.167:6800/30035 2 ==== osd_op_reply(2 default.realm [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 ==== 157+0+0 (crc 0 0 0) 0x7fc7bc00a500 con 0x560a577dc5f0 2023-10-05T19:28:36.817 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.810+0000 7fc7c4047e40 20 rados_obj.operate() r=-2 bl.length=0 2023-10-05T19:28:36.817 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.810+0000 7fc7c4047e40 4 RGWPeriod::init failed to init realm id : (2) No such file or directory 2023-10-05T19:28:36.817 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.810+0000 7fc7c4047e40 20 rados->read ofs=0 len=0 2023-10-05T19:28:36.818 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.810+0000 7fc7c4047e40 1 -- 172.21.15.167:0/287475553 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:3 5.12 5:49953fa1:::default.realm:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x560a5780abd0 con 0x560a577dc5f0 2023-10-05T19:28:36.818 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.810+0000 7fc7c237d700 1 -- 172.21.15.167:0/287475553 <== osd.1 v2:172.21.15.167:6800/30035 3 ==== osd_op_reply(3 default.realm [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 ==== 157+0+0 (crc 0 0 0) 0x7fc7bc00a500 con 0x560a577dc5f0 2023-10-05T19:28:36.818 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.810+0000 7fc7c4047e40 20 rados_obj.operate() r=-2 bl.length=0 2023-10-05T19:28:36.818 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.810+0000 7fc7c4047e40 20 rados->read ofs=0 len=0 2023-10-05T19:28:36.818 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.810+0000 7fc7c4047e40 1 -- 172.21.15.167:0/287475553 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:4 5.f 5:f43b6ece:::zone_names.default:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x560a5780a7b0 con 0x560a577dc5f0 2023-10-05T19:28:36.818 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.810+0000 7fc7c237d700 1 -- 172.21.15.167:0/287475553 <== osd.1 v2:172.21.15.167:6800/30035 4 ==== osd_op_reply(4 zone_names.default [read 0~46 out=46b] v0'0 uv1 ondisk = 0) v8 ==== 162+0+46 (crc 0 0 0) 0x7fc7bc00a500 con 0x560a577dc5f0 2023-10-05T19:28:36.818 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.810+0000 7fc7c4047e40 20 rados_obj.operate() r=0 bl.length=46 2023-10-05T19:28:36.818 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.810+0000 7fc7c4047e40 20 rados->read ofs=0 len=0 2023-10-05T19:28:36.818 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.810+0000 7fc7c4047e40 1 -- 172.21.15.167:0/287475553 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:5 5.1d 5:b8374eb4:::zone_info.9a9df965-6612-45d4-a45f-6e3c889bacc2:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x560a5780b640 con 0x560a577dc5f0 2023-10-05T19:28:36.819 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.810+0000 7fc7c237d700 1 -- 172.21.15.167:0/287475553 <== osd.1 v2:172.21.15.167:6800/30035 5 ==== osd_op_reply(5 zone_info.9a9df965-6612-45d4-a45f-6e3c889bacc2 [read 0~874 out=874b] v0'0 uv2 ondisk = 0) v8 ==== 190+0+874 (crc 0 0 0) 0x7fc7bc00a500 con 0x560a577dc5f0 2023-10-05T19:28:36.820 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.810+0000 7fc7c4047e40 20 rados_obj.operate() r=0 bl.length=874 2023-10-05T19:28:36.820 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.810+0000 7fc7c4047e40 20 searching for the correct realm 2023-10-05T19:28:36.820 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.814+0000 7fc7c4047e40 1 --2- 172.21.15.167:0/287475553 >> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] conn(0x560a5780d490 0x560a5780f870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-05T19:28:36.820 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.814+0000 7fc7c4047e40 1 -- 172.21.15.167:0/287475553 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:6 5.0 5:00000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x560a5780fdb0 con 0x560a5780d490 2023-10-05T19:28:36.820 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.814+0000 7fc7c1b7c700 1 --2- 172.21.15.167:0/287475553 >> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] conn(0x560a5780d490 0x560a5780f870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-05T19:28:36.820 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.814+0000 7fc7c1b7c700 1 --2- 172.21.15.167:0/287475553 >> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] conn(0x560a5780d490 0x560a5780f870 crc :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-05T19:28:36.821 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.814+0000 7fc7c1b7c700 1 -- 172.21.15.167:0/287475553 <== osd.3 v2:172.21.15.167:6808/30036 1 ==== osd_op_reply(6 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fc7b4019b30 con 0x560a5780d490 2023-10-05T19:28:36.821 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.814+0000 7fc7c1b7c700 1 --2- 172.21.15.167:0/287475553 >> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] conn(0x7fc7b40420d0 0x7fc7b403d620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-05T19:28:36.821 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.814+0000 7fc7c1b7c700 1 -- 172.21.15.167:0/287475553 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:7 5.10 5:08000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x560a5780fdb0 con 0x7fc7b40420d0 2023-10-05T19:28:36.821 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.814+0000 7fc7c137b700 1 --2- 172.21.15.167:0/287475553 >> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] conn(0x7fc7b40420d0 0x7fc7b403d620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-05T19:28:36.821 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.814+0000 7fc7c137b700 1 --2- 172.21.15.167:0/287475553 >> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] conn(0x7fc7b40420d0 0x7fc7b403d620 crc :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-05T19:28:36.822 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.814+0000 7fc7c137b700 1 -- 172.21.15.167:0/287475553 <== osd.2 v2:172.21.15.167:6816/30037 1 ==== osd_op_reply(7 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fc7ac00fca0 con 0x7fc7b40420d0 2023-10-05T19:28:36.822 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.814+0000 7fc7c137b700 1 --2- 172.21.15.167:0/287475553 >> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] conn(0x7fc7ac029420 0x7fc7ac029830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-05T19:28:36.822 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.814+0000 7fc7c137b700 1 -- 172.21.15.167:0/287475553 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:8 5.8 5:10000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x560a5780fdb0 con 0x7fc7ac029420 2023-10-05T19:28:36.822 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.818+0000 7fc7c237d700 1 --2- 172.21.15.167:0/287475553 >> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] conn(0x7fc7ac029420 0x7fc7ac029830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-05T19:28:36.823 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.818+0000 7fc7c237d700 1 --2- 172.21.15.167:0/287475553 >> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] conn(0x7fc7ac029420 0x7fc7ac029830 crc :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-05T19:28:36.824 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.818+0000 7fc7c237d700 1 -- 172.21.15.167:0/287475553 <== osd.0 v2:172.21.15.118:6800/27690 1 ==== osd_op_reply(8 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fc7bc014040 con 0x7fc7ac029420 2023-10-05T19:28:36.824 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.818+0000 7fc7c237d700 1 -- 172.21.15.167:0/287475553 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:9 5.18 5:18000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x560a5780fdb0 con 0x560a577dc5f0 2023-10-05T19:28:36.824 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.818+0000 7fc7c237d700 1 -- 172.21.15.167:0/287475553 <== osd.1 v2:172.21.15.167:6800/30035 6 ==== osd_op_reply(9 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fc7bc014040 con 0x560a577dc5f0 2023-10-05T19:28:36.824 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.818+0000 7fc7c237d700 1 -- 172.21.15.167:0/287475553 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:10 5.4 5:20000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x560a5780fdb0 con 0x560a577dc5f0 2023-10-05T19:28:36.825 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.818+0000 7fc7c237d700 1 -- 172.21.15.167:0/287475553 <== osd.1 v2:172.21.15.167:6800/30035 7 ==== osd_op_reply(10 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fc7bc014040 con 0x560a577dc5f0 2023-10-05T19:28:36.825 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.818+0000 7fc7c237d700 1 -- 172.21.15.167:0/287475553 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:11 5.14 5:28000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x560a5780fdb0 con 0x7fc7ac029420 2023-10-05T19:28:36.826 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.818+0000 7fc7c237d700 1 -- 172.21.15.167:0/287475553 <== osd.0 v2:172.21.15.118:6800/27690 2 ==== osd_op_reply(11 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fc7bc014040 con 0x7fc7ac029420 2023-10-05T19:28:36.826 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.818+0000 7fc7c237d700 1 -- 172.21.15.167:0/287475553 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:12 5.c 5:30000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x560a5780fdb0 con 0x7fc7ac029420 2023-10-05T19:28:36.827 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.822+0000 7fc7c237d700 1 -- 172.21.15.167:0/287475553 <== osd.0 v2:172.21.15.118:6800/27690 3 ==== osd_op_reply(12 [pgnls start_epoch 30 out=112b] v23'1 uv1 ondisk = 1) v8 ==== 144+0+112 (crc 0 0 0) 0x7fc7bc014040 con 0x7fc7ac029420 2023-10-05T19:28:36.827 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.822+0000 7fc7c237d700 1 -- 172.21.15.167:0/287475553 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:13 5.1c 5:38000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x560a5780fdb0 con 0x7fc7b40420d0 2023-10-05T19:28:36.827 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.822+0000 7fc7c137b700 1 -- 172.21.15.167:0/287475553 <== osd.2 v2:172.21.15.167:6816/30037 2 ==== osd_op_reply(13 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fc7ac00fca0 con 0x7fc7b40420d0 2023-10-05T19:28:36.827 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.822+0000 7fc7c137b700 1 -- 172.21.15.167:0/287475553 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:14 5.2 5:40000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x560a5780fdb0 con 0x560a5780d490 2023-10-05T19:28:36.828 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.822+0000 7fc7c1b7c700 1 -- 172.21.15.167:0/287475553 <== osd.3 v2:172.21.15.167:6808/30036 2 ==== osd_op_reply(14 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fc7b4019b30 con 0x560a5780d490 2023-10-05T19:28:36.828 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.822+0000 7fc7c1b7c700 1 -- 172.21.15.167:0/287475553 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:15 5.12 5:48000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x560a5780fdb0 con 0x560a577dc5f0 2023-10-05T19:28:36.829 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.822+0000 7fc7c237d700 1 -- 172.21.15.167:0/287475553 <== osd.1 v2:172.21.15.167:6800/30035 8 ==== osd_op_reply(15 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fc7bc014040 con 0x560a577dc5f0 2023-10-05T19:28:36.829 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.822+0000 7fc7c237d700 1 -- 172.21.15.167:0/287475553 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:16 5.a 5:50000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x560a5780fdb0 con 0x7fc7ac029420 2023-10-05T19:28:36.830 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.822+0000 7fc7c237d700 1 -- 172.21.15.167:0/287475553 <== osd.0 v2:172.21.15.118:6800/27690 4 ==== osd_op_reply(16 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fc7bc022040 con 0x7fc7ac029420 2023-10-05T19:28:36.830 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.822+0000 7fc7c237d700 1 -- 172.21.15.167:0/287475553 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:17 5.1a 5:58000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x560a5780fdb0 con 0x560a577dc5f0 2023-10-05T19:28:36.830 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.826+0000 7fc7c237d700 1 -- 172.21.15.167:0/287475553 <== osd.1 v2:172.21.15.167:6800/30035 9 ==== osd_op_reply(17 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fc7bc022040 con 0x560a577dc5f0 2023-10-05T19:28:36.831 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.826+0000 7fc7c237d700 1 -- 172.21.15.167:0/287475553 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:18 5.6 5:60000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x560a5780fdb0 con 0x7fc7b40420d0 2023-10-05T19:28:36.831 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.826+0000 7fc7c137b700 1 -- 172.21.15.167:0/287475553 <== osd.2 v2:172.21.15.167:6816/30037 3 ==== osd_op_reply(18 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fc7ac00fca0 con 0x7fc7b40420d0 2023-10-05T19:28:36.831 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.826+0000 7fc7c137b700 1 -- 172.21.15.167:0/287475553 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:19 5.16 5:68000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x560a5780fdb0 con 0x560a5780d490 2023-10-05T19:28:36.832 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.826+0000 7fc7c1b7c700 1 -- 172.21.15.167:0/287475553 <== osd.3 v2:172.21.15.167:6808/30036 3 ==== osd_op_reply(19 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fc7b4019b30 con 0x560a5780d490 2023-10-05T19:28:36.832 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.826+0000 7fc7c1b7c700 1 -- 172.21.15.167:0/287475553 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:20 5.e 5:70000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x560a5780fdb0 con 0x560a5780d490 2023-10-05T19:28:36.832 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.826+0000 7fc7c1b7c700 1 -- 172.21.15.167:0/287475553 <== osd.3 v2:172.21.15.167:6808/30036 4 ==== osd_op_reply(20 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fc7b4019b30 con 0x560a5780d490 2023-10-05T19:28:36.833 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.826+0000 7fc7c1b7c700 1 -- 172.21.15.167:0/287475553 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:21 5.1e 5:78000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x560a5780fdb0 con 0x7fc7ac029420 2023-10-05T19:28:36.833 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.826+0000 7fc7c237d700 1 -- 172.21.15.167:0/287475553 <== osd.0 v2:172.21.15.118:6800/27690 5 ==== osd_op_reply(21 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fc7bc022040 con 0x7fc7ac029420 2023-10-05T19:28:36.833 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.826+0000 7fc7c237d700 1 -- 172.21.15.167:0/287475553 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:22 5.1 5:80000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x560a5780fdb0 con 0x560a5780d490 2023-10-05T19:28:36.834 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.830+0000 7fc7c1b7c700 1 -- 172.21.15.167:0/287475553 <== osd.3 v2:172.21.15.167:6808/30036 5 ==== osd_op_reply(22 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fc7b4038040 con 0x560a5780d490 2023-10-05T19:28:36.834 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.830+0000 7fc7c1b7c700 1 -- 172.21.15.167:0/287475553 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:23 5.11 5:88000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x560a5780fdb0 con 0x560a5780d490 2023-10-05T19:28:36.835 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.830+0000 7fc7c1b7c700 1 -- 172.21.15.167:0/287475553 <== osd.3 v2:172.21.15.167:6808/30036 6 ==== osd_op_reply(23 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fc7b4038040 con 0x560a5780d490 2023-10-05T19:28:36.835 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.830+0000 7fc7c1b7c700 1 -- 172.21.15.167:0/287475553 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:24 5.9 5:90000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x560a5780fdb0 con 0x560a577dc5f0 2023-10-05T19:28:36.835 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.830+0000 7fc7c237d700 1 -- 172.21.15.167:0/287475553 <== osd.1 v2:172.21.15.167:6800/30035 10 ==== osd_op_reply(24 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fc7bc022040 con 0x560a577dc5f0 2023-10-05T19:28:36.835 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.830+0000 7fc7c237d700 1 -- 172.21.15.167:0/287475553 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:25 5.19 5:98000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x560a5780fdb0 con 0x560a577dc5f0 2023-10-05T19:28:36.836 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.830+0000 7fc7c237d700 1 -- 172.21.15.167:0/287475553 <== osd.1 v2:172.21.15.167:6800/30035 11 ==== osd_op_reply(25 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fc7bc022040 con 0x560a577dc5f0 2023-10-05T19:28:36.836 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.830+0000 7fc7c237d700 1 -- 172.21.15.167:0/287475553 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:26 5.5 5:a0000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x560a5780fdb0 con 0x7fc7ac029420 2023-10-05T19:28:36.836 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.830+0000 7fc7c237d700 1 -- 172.21.15.167:0/287475553 <== osd.0 v2:172.21.15.118:6800/27690 6 ==== osd_op_reply(26 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fc7bc022040 con 0x7fc7ac029420 2023-10-05T19:28:36.836 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.830+0000 7fc7c237d700 1 -- 172.21.15.167:0/287475553 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:27 5.15 5:a8000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x560a5780fdb0 con 0x560a5780d490 2023-10-05T19:28:36.837 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.830+0000 7fc7c1b7c700 1 -- 172.21.15.167:0/287475553 <== osd.3 v2:172.21.15.167:6808/30036 7 ==== osd_op_reply(27 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fc7b4038040 con 0x560a5780d490 2023-10-05T19:28:36.837 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.830+0000 7fc7c1b7c700 1 -- 172.21.15.167:0/287475553 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:28 5.d 5:b0000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x560a5780fdb0 con 0x7fc7b40420d0 2023-10-05T19:28:36.837 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.830+0000 7fc7c137b700 1 -- 172.21.15.167:0/287475553 <== osd.2 v2:172.21.15.167:6816/30037 4 ==== osd_op_reply(28 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fc7ac00fca0 con 0x7fc7b40420d0 2023-10-05T19:28:36.837 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.830+0000 7fc7c137b700 1 -- 172.21.15.167:0/287475553 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:29 5.1d 5:b8000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x560a5780fdb0 con 0x560a577dc5f0 2023-10-05T19:28:36.838 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.830+0000 7fc7c237d700 1 -- 172.21.15.167:0/287475553 <== osd.1 v2:172.21.15.167:6800/30035 12 ==== osd_op_reply(29 [pgnls start_epoch 30 out=107b] v30'2 uv2 ondisk = 1) v8 ==== 144+0+107 (crc 0 0 0) 0x7fc7bc022040 con 0x560a577dc5f0 2023-10-05T19:28:36.838 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.830+0000 7fc7c237d700 1 -- 172.21.15.167:0/287475553 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:30 5.3 5:c0000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x560a5780fdb0 con 0x7fc7ac029420 2023-10-05T19:28:36.838 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.834+0000 7fc7c237d700 1 -- 172.21.15.167:0/287475553 <== osd.0 v2:172.21.15.118:6800/27690 7 ==== osd_op_reply(30 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fc7bc022040 con 0x7fc7ac029420 2023-10-05T19:28:36.839 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.834+0000 7fc7c237d700 1 -- 172.21.15.167:0/287475553 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:31 5.13 5:c8000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x560a5780fdb0 con 0x560a5780d490 2023-10-05T19:28:36.839 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.834+0000 7fc7c1b7c700 1 -- 172.21.15.167:0/287475553 <== osd.3 v2:172.21.15.167:6808/30036 8 ==== osd_op_reply(31 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fc7b4047040 con 0x560a5780d490 2023-10-05T19:28:36.839 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.834+0000 7fc7c1b7c700 1 -- 172.21.15.167:0/287475553 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:32 5.b 5:d0000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x560a5780fdb0 con 0x7fc7ac029420 2023-10-05T19:28:36.839 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.834+0000 7fc7c237d700 1 -- 172.21.15.167:0/287475553 <== osd.0 v2:172.21.15.118:6800/27690 8 ==== osd_op_reply(32 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fc7bc022040 con 0x7fc7ac029420 2023-10-05T19:28:36.839 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.834+0000 7fc7c237d700 1 -- 172.21.15.167:0/287475553 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:33 5.1b 5:d8000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x560a5780fdb0 con 0x7fc7b40420d0 2023-10-05T19:28:36.841 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.834+0000 7fc7c137b700 1 -- 172.21.15.167:0/287475553 <== osd.2 v2:172.21.15.167:6816/30037 5 ==== osd_op_reply(33 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fc7ac00fca0 con 0x7fc7b40420d0 2023-10-05T19:28:36.841 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.834+0000 7fc7c137b700 1 -- 172.21.15.167:0/287475553 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:34 5.7 5:e0000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x560a5780fdb0 con 0x560a5780d490 2023-10-05T19:28:36.842 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.838+0000 7fc7c1b7c700 1 -- 172.21.15.167:0/287475553 <== osd.3 v2:172.21.15.167:6808/30036 9 ==== osd_op_reply(34 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fc7b4047040 con 0x560a5780d490 2023-10-05T19:28:36.842 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.838+0000 7fc7c1b7c700 1 -- 172.21.15.167:0/287475553 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:35 5.17 5:e8000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x560a5780fdb0 con 0x7fc7ac029420 2023-10-05T19:28:36.843 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.838+0000 7fc7c237d700 1 -- 172.21.15.167:0/287475553 <== osd.0 v2:172.21.15.118:6800/27690 9 ==== osd_op_reply(35 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fc7bc022040 con 0x7fc7ac029420 2023-10-05T19:28:36.843 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.838+0000 7fc7c237d700 1 -- 172.21.15.167:0/287475553 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:36 5.f 5:f0000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x560a5780fdb0 con 0x560a577dc5f0 2023-10-05T19:28:36.844 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.838+0000 7fc7c237d700 1 -- 172.21.15.167:0/287475553 <== osd.1 v2:172.21.15.167:6800/30035 13 ==== osd_op_reply(36 [pgnls start_epoch 30 out=115b] v23'2 uv2 ondisk = 1) v8 ==== 144+0+115 (crc 0 0 0) 0x7fc7bc02f080 con 0x560a577dc5f0 2023-10-05T19:28:36.844 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.838+0000 7fc7c237d700 1 -- 172.21.15.167:0/287475553 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:37 5.1f 5:f8000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x560a5780fdb0 con 0x7fc7b40420d0 2023-10-05T19:28:36.845 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.838+0000 7fc7c137b700 1 -- 172.21.15.167:0/287475553 <== osd.2 v2:172.21.15.167:6816/30037 6 ==== osd_op_reply(37 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fc7ac00fca0 con 0x7fc7b40420d0 2023-10-05T19:28:36.845 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.838+0000 7fc7c4047e40 20 RGWRados::pool_iterate: got zonegroup_info.4f1f719d-9648-417b-b10c-e5d939825419 2023-10-05T19:28:36.845 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.838+0000 7fc7c4047e40 20 RGWRados::pool_iterate: got zone_info.9a9df965-6612-45d4-a45f-6e3c889bacc2 2023-10-05T19:28:36.845 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.838+0000 7fc7c4047e40 20 RGWRados::pool_iterate: got zone_names.default 2023-10-05T19:28:36.845 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.838+0000 7fc7c4047e40 20 RGWRados::pool_iterate: got zonegroups_names.default 2023-10-05T19:28:36.845 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.838+0000 7fc7c4047e40 20 rados->read ofs=0 len=0 2023-10-05T19:28:36.846 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.838+0000 7fc7c4047e40 1 -- 172.21.15.167:0/287475553 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:38 5.12 5:49953fa1:::default.realm:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x560a5780c210 con 0x560a577dc5f0 2023-10-05T19:28:36.846 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.838+0000 7fc7c237d700 1 -- 172.21.15.167:0/287475553 <== osd.1 v2:172.21.15.167:6800/30035 14 ==== osd_op_reply(38 default.realm [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 ==== 157+0+0 (crc 0 0 0) 0x7fc7bc02f080 con 0x560a577dc5f0 2023-10-05T19:28:36.846 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.838+0000 7fc7c4047e40 20 rados_obj.operate() r=-2 bl.length=0 2023-10-05T19:28:36.846 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.838+0000 7fc7c4047e40 20 rados->read ofs=0 len=0 2023-10-05T19:28:36.846 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.838+0000 7fc7c4047e40 1 -- 172.21.15.167:0/287475553 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:39 5.f 5:f4c53578:::zonegroups_names.default:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x560a5780cb90 con 0x560a577dc5f0 2023-10-05T19:28:36.847 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.838+0000 7fc7c237d700 1 -- 172.21.15.167:0/287475553 <== osd.1 v2:172.21.15.167:6800/30035 15 ==== osd_op_reply(39 zonegroups_names.default [read 0~46 out=46b] v0'0 uv2 ondisk = 0) v8 ==== 168+0+46 (crc 0 0 0) 0x7fc7bc02f080 con 0x560a577dc5f0 2023-10-05T19:28:36.848 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.838+0000 7fc7c4047e40 20 rados_obj.operate() r=0 bl.length=46 2023-10-05T19:28:36.848 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.838+0000 7fc7c4047e40 20 rados->read ofs=0 len=0 2023-10-05T19:28:36.848 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.838+0000 7fc7c4047e40 1 -- 172.21.15.167:0/287475553 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:40 5.c 5:30d299a5:::zonegroup_info.4f1f719d-9648-417b-b10c-e5d939825419:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x560a5780bdd0 con 0x7fc7ac029420 2023-10-05T19:28:36.848 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.842+0000 7fc7c237d700 1 -- 172.21.15.167:0/287475553 <== osd.0 v2:172.21.15.118:6800/27690 10 ==== osd_op_reply(40 zonegroup_info.4f1f719d-9648-417b-b10c-e5d939825419 [read 0~362 out=362b] v0'0 uv1 ondisk = 0) v8 ==== 195+0+362 (crc 0 0 0) 0x7fc7bc02f080 con 0x7fc7ac029420 2023-10-05T19:28:36.848 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.842+0000 7fc7c4047e40 20 rados_obj.operate() r=0 bl.length=362 2023-10-05T19:28:36.848 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.842+0000 7fc7c4047e40 20 rados->read ofs=0 len=0 2023-10-05T19:28:36.848 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.842+0000 7fc7c4047e40 1 -- 172.21.15.167:0/287475553 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:41 5.13 5:cbbf6c60:::converted:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x560a57810160 con 0x560a5780d490 2023-10-05T19:28:36.848 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.842+0000 7fc7c1b7c700 1 -- 172.21.15.167:0/287475553 <== osd.3 v2:172.21.15.167:6808/30036 10 ==== osd_op_reply(41 converted [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 ==== 153+0+0 (crc 0 0 0) 0x7fc7b4047040 con 0x560a5780d490 2023-10-05T19:28:36.848 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.842+0000 7fc7c4047e40 20 rados_obj.operate() r=-2 bl.length=0 2023-10-05T19:28:36.848 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.842+0000 7fc7c4047e40 20 rados->read ofs=0 len=0 2023-10-05T19:28:36.849 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.842+0000 7fc7c4047e40 1 -- 172.21.15.167:0/287475553 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:42 5.12 5:49953fa1:::default.realm:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x560a57810510 con 0x560a577dc5f0 2023-10-05T19:28:36.849 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.842+0000 7fc7c237d700 1 -- 172.21.15.167:0/287475553 <== osd.1 v2:172.21.15.167:6800/30035 16 ==== osd_op_reply(42 default.realm [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 ==== 157+0+0 (crc 0 0 0) 0x7fc7bc02f080 con 0x560a577dc5f0 2023-10-05T19:28:36.849 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.842+0000 7fc7c4047e40 20 rados_obj.operate() r=-2 bl.length=0 2023-10-05T19:28:36.849 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.842+0000 7fc7c4047e40 20 rados->read ofs=0 len=0 2023-10-05T19:28:36.849 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.842+0000 7fc7c4047e40 1 -- 172.21.15.167:0/287475553 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:43 5.f 5:f4c53578:::zonegroups_names.default:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x560a57810510 con 0x560a577dc5f0 2023-10-05T19:28:36.849 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.842+0000 7fc7c237d700 1 -- 172.21.15.167:0/287475553 <== osd.1 v2:172.21.15.167:6800/30035 17 ==== osd_op_reply(43 zonegroups_names.default [read 0~46 out=46b] v0'0 uv2 ondisk = 0) v8 ==== 168+0+46 (crc 0 0 0) 0x7fc7bc02f080 con 0x560a577dc5f0 2023-10-05T19:28:36.849 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.842+0000 7fc7c4047e40 20 rados_obj.operate() r=0 bl.length=46 2023-10-05T19:28:36.850 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.842+0000 7fc7c4047e40 1 -- 172.21.15.167:0/287475553 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:44 5.0 5:00000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x560a57811520 con 0x560a5780d490 2023-10-05T19:28:36.850 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.846+0000 7fc7c1b7c700 1 -- 172.21.15.167:0/287475553 <== osd.3 v2:172.21.15.167:6808/30036 11 ==== osd_op_reply(44 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fc7b4047040 con 0x560a5780d490 2023-10-05T19:28:36.850 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.846+0000 7fc7c1b7c700 1 -- 172.21.15.167:0/287475553 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:45 5.10 5:08000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x560a57811520 con 0x7fc7b40420d0 2023-10-05T19:28:36.850 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.846+0000 7fc7c137b700 1 -- 172.21.15.167:0/287475553 <== osd.2 v2:172.21.15.167:6816/30037 7 ==== osd_op_reply(45 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fc7ac00fca0 con 0x7fc7b40420d0 2023-10-05T19:28:36.851 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.846+0000 7fc7c137b700 1 -- 172.21.15.167:0/287475553 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:46 5.8 5:10000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x560a57811520 con 0x7fc7ac029420 2023-10-05T19:28:36.851 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.846+0000 7fc7c237d700 1 -- 172.21.15.167:0/287475553 <== osd.0 v2:172.21.15.118:6800/27690 11 ==== osd_op_reply(46 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fc7bc02f080 con 0x7fc7ac029420 2023-10-05T19:28:36.851 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.846+0000 7fc7c237d700 1 -- 172.21.15.167:0/287475553 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:47 5.18 5:18000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x560a5780cb90 con 0x560a577dc5f0 2023-10-05T19:28:36.852 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.846+0000 7fc7c237d700 1 -- 172.21.15.167:0/287475553 <== osd.1 v2:172.21.15.167:6800/30035 18 ==== osd_op_reply(47 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fc7bc02f080 con 0x560a577dc5f0 2023-10-05T19:28:36.852 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.846+0000 7fc7c237d700 1 -- 172.21.15.167:0/287475553 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:48 5.4 5:20000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x560a5780cb90 con 0x560a577dc5f0 2023-10-05T19:28:36.852 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.846+0000 7fc7c237d700 1 -- 172.21.15.167:0/287475553 <== osd.1 v2:172.21.15.167:6800/30035 19 ==== osd_op_reply(48 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fc7bc02f080 con 0x560a577dc5f0 2023-10-05T19:28:36.852 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.846+0000 7fc7c237d700 1 -- 172.21.15.167:0/287475553 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:49 5.14 5:28000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x560a5780cb90 con 0x7fc7ac029420 2023-10-05T19:28:36.853 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.846+0000 7fc7c237d700 1 -- 172.21.15.167:0/287475553 <== osd.0 v2:172.21.15.118:6800/27690 12 ==== osd_op_reply(49 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fc7bc02f080 con 0x7fc7ac029420 2023-10-05T19:28:36.853 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.846+0000 7fc7c237d700 1 -- 172.21.15.167:0/287475553 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:50 5.c 5:30000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x560a5780cb90 con 0x7fc7ac029420 2023-10-05T19:28:36.854 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.850+0000 7fc7c237d700 1 -- 172.21.15.167:0/287475553 <== osd.0 v2:172.21.15.118:6800/27690 13 ==== osd_op_reply(50 [pgnls start_epoch 30 out=112b] v23'1 uv1 ondisk = 1) v8 ==== 144+0+112 (crc 0 0 0) 0x7fc7bc02f080 con 0x7fc7ac029420 2023-10-05T19:28:36.854 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.850+0000 7fc7c237d700 1 -- 172.21.15.167:0/287475553 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:51 5.1c 5:38000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x560a5780cb90 con 0x7fc7b40420d0 2023-10-05T19:28:36.855 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.850+0000 7fc7c137b700 1 -- 172.21.15.167:0/287475553 <== osd.2 v2:172.21.15.167:6816/30037 8 ==== osd_op_reply(51 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fc7ac00fca0 con 0x7fc7b40420d0 2023-10-05T19:28:36.855 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.850+0000 7fc7c137b700 1 -- 172.21.15.167:0/287475553 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:52 5.2 5:40000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x560a5780cb90 con 0x560a5780d490 2023-10-05T19:28:36.855 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.850+0000 7fc7c1b7c700 1 -- 172.21.15.167:0/287475553 <== osd.3 v2:172.21.15.167:6808/30036 12 ==== osd_op_reply(52 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fc7b4047040 con 0x560a5780d490 2023-10-05T19:28:36.855 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.850+0000 7fc7c1b7c700 1 -- 172.21.15.167:0/287475553 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:53 5.12 5:48000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x560a5780cb90 con 0x560a577dc5f0 2023-10-05T19:28:36.856 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.850+0000 7fc7c237d700 1 -- 172.21.15.167:0/287475553 <== osd.1 v2:172.21.15.167:6800/30035 20 ==== osd_op_reply(53 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fc7bc02f080 con 0x560a577dc5f0 2023-10-05T19:28:36.856 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.850+0000 7fc7c237d700 1 -- 172.21.15.167:0/287475553 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:54 5.a 5:50000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x560a5780cb90 con 0x7fc7ac029420 2023-10-05T19:28:36.857 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.850+0000 7fc7c237d700 1 -- 172.21.15.167:0/287475553 <== osd.0 v2:172.21.15.118:6800/27690 14 ==== osd_op_reply(54 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fc7bc02f080 con 0x7fc7ac029420 2023-10-05T19:28:36.857 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.850+0000 7fc7c237d700 1 -- 172.21.15.167:0/287475553 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:55 5.1a 5:58000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x560a5780cb90 con 0x560a577dc5f0 2023-10-05T19:28:36.857 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.850+0000 7fc7c237d700 1 -- 172.21.15.167:0/287475553 <== osd.1 v2:172.21.15.167:6800/30035 21 ==== osd_op_reply(55 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fc7bc02f080 con 0x560a577dc5f0 2023-10-05T19:28:36.857 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.850+0000 7fc7c237d700 1 -- 172.21.15.167:0/287475553 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:56 5.6 5:60000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x560a5780cb90 con 0x7fc7b40420d0 2023-10-05T19:28:36.857 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.850+0000 7fc7c137b700 1 -- 172.21.15.167:0/287475553 <== osd.2 v2:172.21.15.167:6816/30037 9 ==== osd_op_reply(56 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fc7ac00fca0 con 0x7fc7b40420d0 2023-10-05T19:28:36.858 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.850+0000 7fc7c137b700 1 -- 172.21.15.167:0/287475553 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:57 5.16 5:68000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x560a5780cb90 con 0x560a5780d490 2023-10-05T19:28:36.858 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.850+0000 7fc7c1b7c700 1 -- 172.21.15.167:0/287475553 <== osd.3 v2:172.21.15.167:6808/30036 13 ==== osd_op_reply(57 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fc7b4047040 con 0x560a5780d490 2023-10-05T19:28:36.858 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.850+0000 7fc7c1b7c700 1 -- 172.21.15.167:0/287475553 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:58 5.e 5:70000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x560a5780cb90 con 0x560a5780d490 2023-10-05T19:28:36.858 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.854+0000 7fc7c1b7c700 1 -- 172.21.15.167:0/287475553 <== osd.3 v2:172.21.15.167:6808/30036 14 ==== osd_op_reply(58 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fc7b4047040 con 0x560a5780d490 2023-10-05T19:28:36.858 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.854+0000 7fc7c1b7c700 1 -- 172.21.15.167:0/287475553 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:59 5.1e 5:78000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x560a5780cb90 con 0x7fc7ac029420 2023-10-05T19:28:36.859 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.854+0000 7fc7c237d700 1 -- 172.21.15.167:0/287475553 <== osd.0 v2:172.21.15.118:6800/27690 15 ==== osd_op_reply(59 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fc7bc02f080 con 0x7fc7ac029420 2023-10-05T19:28:36.859 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.854+0000 7fc7c237d700 1 -- 172.21.15.167:0/287475553 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:60 5.1 5:80000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x560a5780cb90 con 0x560a5780d490 2023-10-05T19:28:36.859 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.854+0000 7fc7c1b7c700 1 -- 172.21.15.167:0/287475553 <== osd.3 v2:172.21.15.167:6808/30036 15 ==== osd_op_reply(60 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fc7b4047040 con 0x560a5780d490 2023-10-05T19:28:36.859 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.854+0000 7fc7c1b7c700 1 -- 172.21.15.167:0/287475553 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:61 5.11 5:88000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x560a5780cb90 con 0x560a5780d490 2023-10-05T19:28:36.860 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.854+0000 7fc7c1b7c700 1 -- 172.21.15.167:0/287475553 <== osd.3 v2:172.21.15.167:6808/30036 16 ==== osd_op_reply(61 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fc7b4047040 con 0x560a5780d490 2023-10-05T19:28:36.860 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.854+0000 7fc7c1b7c700 1 -- 172.21.15.167:0/287475553 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:62 5.9 5:90000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x560a5780cb90 con 0x560a577dc5f0 2023-10-05T19:28:36.860 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.854+0000 7fc7c237d700 1 -- 172.21.15.167:0/287475553 <== osd.1 v2:172.21.15.167:6800/30035 22 ==== osd_op_reply(62 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fc7bc02f080 con 0x560a577dc5f0 2023-10-05T19:28:36.860 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.854+0000 7fc7c237d700 1 -- 172.21.15.167:0/287475553 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:63 5.19 5:98000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x560a5780cb90 con 0x560a577dc5f0 2023-10-05T19:28:36.861 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.854+0000 7fc7c237d700 1 -- 172.21.15.167:0/287475553 <== osd.1 v2:172.21.15.167:6800/30035 23 ==== osd_op_reply(63 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fc7bc02f080 con 0x560a577dc5f0 2023-10-05T19:28:36.861 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.854+0000 7fc7c237d700 1 -- 172.21.15.167:0/287475553 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:64 5.5 5:a0000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x560a5780cb90 con 0x7fc7ac029420 2023-10-05T19:28:36.862 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.854+0000 7fc7c237d700 1 -- 172.21.15.167:0/287475553 <== osd.0 v2:172.21.15.118:6800/27690 16 ==== osd_op_reply(64 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fc7bc02f080 con 0x7fc7ac029420 2023-10-05T19:28:36.862 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.854+0000 7fc7c237d700 1 -- 172.21.15.167:0/287475553 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:65 5.15 5:a8000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x560a5780cb90 con 0x560a5780d490 2023-10-05T19:28:36.862 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.858+0000 7fc7c1b7c700 1 -- 172.21.15.167:0/287475553 <== osd.3 v2:172.21.15.167:6808/30036 17 ==== osd_op_reply(65 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fc7b4047040 con 0x560a5780d490 2023-10-05T19:28:36.862 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.858+0000 7fc7c1b7c700 1 -- 172.21.15.167:0/287475553 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:66 5.d 5:b0000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x560a5780cb90 con 0x7fc7b40420d0 2023-10-05T19:28:36.863 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.858+0000 7fc7c137b700 1 -- 172.21.15.167:0/287475553 <== osd.2 v2:172.21.15.167:6816/30037 10 ==== osd_op_reply(66 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fc7ac00fca0 con 0x7fc7b40420d0 2023-10-05T19:28:36.863 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.858+0000 7fc7c137b700 1 -- 172.21.15.167:0/287475553 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:67 5.1d 5:b8000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x560a5780cb90 con 0x560a577dc5f0 2023-10-05T19:28:36.863 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.858+0000 7fc7c237d700 1 -- 172.21.15.167:0/287475553 <== osd.1 v2:172.21.15.167:6800/30035 24 ==== osd_op_reply(67 [pgnls start_epoch 30 out=107b] v30'2 uv2 ondisk = 1) v8 ==== 144+0+107 (crc 0 0 0) 0x7fc7bc02f080 con 0x560a577dc5f0 2023-10-05T19:28:36.863 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.858+0000 7fc7c237d700 1 -- 172.21.15.167:0/287475553 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:68 5.3 5:c0000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x560a5780cb90 con 0x7fc7ac029420 2023-10-05T19:28:36.864 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.858+0000 7fc7c237d700 1 -- 172.21.15.167:0/287475553 <== osd.0 v2:172.21.15.118:6800/27690 17 ==== osd_op_reply(68 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fc7bc02f080 con 0x7fc7ac029420 2023-10-05T19:28:36.864 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.858+0000 7fc7c237d700 1 -- 172.21.15.167:0/287475553 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:69 5.13 5:c8000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x560a5780cb90 con 0x560a5780d490 2023-10-05T19:28:36.864 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.858+0000 7fc7c1b7c700 1 -- 172.21.15.167:0/287475553 <== osd.3 v2:172.21.15.167:6808/30036 18 ==== osd_op_reply(69 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fc7b4047040 con 0x560a5780d490 2023-10-05T19:28:36.864 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.858+0000 7fc7c1b7c700 1 -- 172.21.15.167:0/287475553 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:70 5.b 5:d0000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x560a5780cb90 con 0x7fc7ac029420 2023-10-05T19:28:36.865 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.858+0000 7fc7c237d700 1 -- 172.21.15.167:0/287475553 <== osd.0 v2:172.21.15.118:6800/27690 18 ==== osd_op_reply(70 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fc7bc02f080 con 0x7fc7ac029420 2023-10-05T19:28:36.865 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.858+0000 7fc7c237d700 1 -- 172.21.15.167:0/287475553 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:71 5.1b 5:d8000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x560a5780cb90 con 0x7fc7b40420d0 2023-10-05T19:28:36.865 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.858+0000 7fc7c137b700 1 -- 172.21.15.167:0/287475553 <== osd.2 v2:172.21.15.167:6816/30037 11 ==== osd_op_reply(71 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fc7ac00fca0 con 0x7fc7b40420d0 2023-10-05T19:28:36.866 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.858+0000 7fc7c137b700 1 -- 172.21.15.167:0/287475553 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:72 5.7 5:e0000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x560a5780cb90 con 0x560a5780d490 2023-10-05T19:28:36.866 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.862+0000 7fc7c1b7c700 1 -- 172.21.15.167:0/287475553 <== osd.3 v2:172.21.15.167:6808/30036 19 ==== osd_op_reply(72 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fc7b4047040 con 0x560a5780d490 2023-10-05T19:28:36.866 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.862+0000 7fc7c1b7c700 1 -- 172.21.15.167:0/287475553 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:73 5.17 5:e8000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x560a5780cb90 con 0x7fc7ac029420 2023-10-05T19:28:36.867 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.862+0000 7fc7c237d700 1 -- 172.21.15.167:0/287475553 <== osd.0 v2:172.21.15.118:6800/27690 19 ==== osd_op_reply(73 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fc7bc02f080 con 0x7fc7ac029420 2023-10-05T19:28:36.867 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.862+0000 7fc7c237d700 1 -- 172.21.15.167:0/287475553 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:74 5.f 5:f0000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x560a5780cb90 con 0x560a577dc5f0 2023-10-05T19:28:36.867 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.862+0000 7fc7c237d700 1 -- 172.21.15.167:0/287475553 <== osd.1 v2:172.21.15.167:6800/30035 25 ==== osd_op_reply(74 [pgnls start_epoch 30 out=115b] v23'2 uv2 ondisk = 1) v8 ==== 144+0+115 (crc 0 0 0) 0x7fc7bc02f080 con 0x560a577dc5f0 2023-10-05T19:28:36.868 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.862+0000 7fc7c237d700 1 -- 172.21.15.167:0/287475553 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:75 5.1f 5:f8000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x560a5780cb90 con 0x7fc7b40420d0 2023-10-05T19:28:36.868 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.862+0000 7fc7c137b700 1 -- 172.21.15.167:0/287475553 <== osd.2 v2:172.21.15.167:6816/30037 12 ==== osd_op_reply(75 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fc7ac00fca0 con 0x7fc7b40420d0 2023-10-05T19:28:36.868 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.862+0000 7fc7c4047e40 20 RGWRados::pool_iterate: got zonegroup_info.4f1f719d-9648-417b-b10c-e5d939825419 2023-10-05T19:28:36.868 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.862+0000 7fc7c4047e40 20 RGWRados::pool_iterate: got zone_info.9a9df965-6612-45d4-a45f-6e3c889bacc2 2023-10-05T19:28:36.868 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.862+0000 7fc7c4047e40 20 RGWRados::pool_iterate: got zone_names.default 2023-10-05T19:28:36.868 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.862+0000 7fc7c4047e40 20 RGWRados::pool_iterate: got zonegroups_names.default 2023-10-05T19:28:36.869 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.862+0000 7fc7c4047e40 20 rados->read ofs=0 len=0 2023-10-05T19:28:36.869 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.862+0000 7fc7c4047e40 1 -- 172.21.15.167:0/287475553 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:76 5.f 5:f43b6ece:::zone_names.default:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x560a57810510 con 0x560a577dc5f0 2023-10-05T19:28:36.870 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.862+0000 7fc7c237d700 1 -- 172.21.15.167:0/287475553 <== osd.1 v2:172.21.15.167:6800/30035 26 ==== osd_op_reply(76 zone_names.default [read 0~46 out=46b] v0'0 uv1 ondisk = 0) v8 ==== 162+0+46 (crc 0 0 0) 0x7fc7bc02f080 con 0x560a577dc5f0 2023-10-05T19:28:36.870 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.862+0000 7fc7c4047e40 20 rados_obj.operate() r=0 bl.length=46 2023-10-05T19:28:36.870 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.862+0000 7fc7c4047e40 20 rados->read ofs=0 len=0 2023-10-05T19:28:36.870 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.862+0000 7fc7c4047e40 1 -- 172.21.15.167:0/287475553 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:77 5.1d 5:b8374eb4:::zone_info.9a9df965-6612-45d4-a45f-6e3c889bacc2:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x560a578108c0 con 0x560a577dc5f0 2023-10-05T19:28:36.870 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.862+0000 7fc7c237d700 1 -- 172.21.15.167:0/287475553 <== osd.1 v2:172.21.15.167:6800/30035 27 ==== osd_op_reply(77 zone_info.9a9df965-6612-45d4-a45f-6e3c889bacc2 [read 0~874 out=874b] v0'0 uv2 ondisk = 0) v8 ==== 190+0+874 (crc 0 0 0) 0x7fc7bc02f080 con 0x560a577dc5f0 2023-10-05T19:28:36.870 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.862+0000 7fc7c4047e40 20 rados_obj.operate() r=0 bl.length=874 2023-10-05T19:28:36.870 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.862+0000 7fc7c4047e40 20 rados->read ofs=0 len=0 2023-10-05T19:28:36.871 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.862+0000 7fc7c4047e40 1 -- 172.21.15.167:0/287475553 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:78 5.f 5:f4c53578:::zonegroups_names.default:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x560a57810d00 con 0x560a577dc5f0 2023-10-05T19:28:36.871 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.862+0000 7fc7c237d700 1 -- 172.21.15.167:0/287475553 <== osd.1 v2:172.21.15.167:6800/30035 28 ==== osd_op_reply(78 zonegroups_names.default [read 0~46 out=46b] v0'0 uv2 ondisk = 0) v8 ==== 168+0+46 (crc 0 0 0) 0x7fc7bc02f080 con 0x560a577dc5f0 2023-10-05T19:28:36.871 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.862+0000 7fc7c4047e40 20 rados_obj.operate() r=0 bl.length=46 2023-10-05T19:28:36.871 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.862+0000 7fc7c4047e40 20 rados->read ofs=0 len=0 2023-10-05T19:28:36.871 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.862+0000 7fc7c4047e40 1 -- 172.21.15.167:0/287475553 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:79 5.c 5:30d299a5:::zonegroup_info.4f1f719d-9648-417b-b10c-e5d939825419:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x560a57810d00 con 0x7fc7ac029420 2023-10-05T19:28:36.872 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.866+0000 7fc7c237d700 1 -- 172.21.15.167:0/287475553 <== osd.0 v2:172.21.15.118:6800/27690 20 ==== osd_op_reply(79 zonegroup_info.4f1f719d-9648-417b-b10c-e5d939825419 [read 0~362 out=362b] v0'0 uv1 ondisk = 0) v8 ==== 195+0+362 (crc 0 0 0) 0x7fc7bc02f080 con 0x7fc7ac029420 2023-10-05T19:28:36.872 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.866+0000 7fc7c4047e40 20 rados_obj.operate() r=0 bl.length=362 2023-10-05T19:28:36.872 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.866+0000 7fc7c4047e40 20 rados->read ofs=0 len=0 2023-10-05T19:28:36.872 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.866+0000 7fc7c4047e40 1 -- 172.21.15.167:0/287475553 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:80 5.3 5:c0ad8cbd:::region_map:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x560a57811130 con 0x7fc7ac029420 2023-10-05T19:28:36.872 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.866+0000 7fc7c237d700 1 -- 172.21.15.167:0/287475553 <== osd.0 v2:172.21.15.118:6800/27690 21 ==== osd_op_reply(80 region_map [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 ==== 154+0+0 (crc 0 0 0) 0x7fc7bc050040 con 0x7fc7ac029420 2023-10-05T19:28:36.872 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.866+0000 7fc7c4047e40 20 rados_obj.operate() r=-2 bl.length=0 2023-10-05T19:28:36.872 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.866+0000 7fc7c4047e40 20 zone default found 2023-10-05T19:28:36.872 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.866+0000 7fc7c4047e40 4 Realm: () 2023-10-05T19:28:36.872 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.866+0000 7fc7c4047e40 4 ZoneGroup: default (4f1f719d-9648-417b-b10c-e5d939825419) 2023-10-05T19:28:36.873 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.866+0000 7fc7c4047e40 4 Zone: default (9a9df965-6612-45d4-a45f-6e3c889bacc2) 2023-10-05T19:28:36.873 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.866+0000 7fc7c4047e40 10 cannot find current period zonegroup using local zonegroup configuration 2023-10-05T19:28:36.873 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.866+0000 7fc7c4047e40 20 zonegroup default 2023-10-05T19:28:36.873 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.866+0000 7fc7c4047e40 20 rados->read ofs=0 len=0 2023-10-05T19:28:36.873 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.866+0000 7fc7c4047e40 1 -- 172.21.15.167:0/287475553 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:81 5.3 5:c52100b6:::period_config.default:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x560a57810e60 con 0x7fc7ac029420 2023-10-05T19:28:36.874 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.866+0000 7fc7c237d700 1 -- 172.21.15.167:0/287475553 <== osd.0 v2:172.21.15.118:6800/27690 22 ==== osd_op_reply(81 period_config.default [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 ==== 165+0+0 (crc 0 0 0) 0x7fc7bc050040 con 0x7fc7ac029420 2023-10-05T19:28:36.874 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.866+0000 7fc7c4047e40 20 rados_obj.operate() r=-2 bl.length=0 2023-10-05T19:28:36.874 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.866+0000 7fc7c4047e40 20 rados->read ofs=0 len=0 2023-10-05T19:28:36.874 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.866+0000 7fc7c4047e40 1 -- 172.21.15.167:0/287475553 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:82 6.2 6:4347d321:::bucket.sync-source-hints.:head [call version.read in=11b,read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x560a578127e0 con 0x560a5780d490 2023-10-05T19:28:36.875 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.866+0000 7fc7c1b7c700 1 -- 172.21.15.167:0/287475553 <== osd.3 v2:172.21.15.167:6808/30036 20 ==== osd_op_reply(82 bucket.sync-source-hints. [call,read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 ==== 211+0+0 (crc 0 0 0) 0x7fc7b4047040 con 0x560a5780d490 2023-10-05T19:28:36.875 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.866+0000 7fc7c4047e40 20 rados_obj.operate() r=-2 bl.length=0 2023-10-05T19:28:36.875 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.866+0000 7fc7c4047e40 20 rados->read ofs=0 len=0 2023-10-05T19:28:36.875 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.866+0000 7fc7c4047e40 1 -- 172.21.15.167:0/287475553 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:83 6.b 6:d467b91b:::bucket.sync-target-hints.:head [call version.read in=11b,read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x560a57812b90 con 0x560a5780d490 2023-10-05T19:28:36.875 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.866+0000 7fc7c1b7c700 1 -- 172.21.15.167:0/287475553 <== osd.3 v2:172.21.15.167:6808/30036 21 ==== osd_op_reply(83 bucket.sync-target-hints. [call,read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 ==== 211+0+0 (crc 0 0 0) 0x7fc7b4047040 con 0x560a5780d490 2023-10-05T19:28:36.875 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.866+0000 7fc7c4047e40 20 rados_obj.operate() r=-2 bl.length=0 2023-10-05T19:28:36.875 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.866+0000 7fc7c4047e40 20 started sync module instance, tier type = 2023-10-05T19:28:36.875 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.866+0000 7fc7c4047e40 20 started zone id=9a9df965-6612-45d4-a45f-6e3c889bacc2 (name=default) with tier type = 2023-10-05T19:28:36.875 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.866+0000 7fc7c4047e40 1 -- 172.21.15.167:0/287475553 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:84 7.1f 7:f95f44c2:::notify.0:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x560a57811ea0 con 0x7fc7b40420d0 2023-10-05T19:28:36.876 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.870+0000 7fc7c137b700 1 -- 172.21.15.167:0/287475553 <== osd.2 v2:172.21.15.167:6816/30037 13 ==== osd_op_reply(84 notify.0 [create] v30'159 uv159 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7fc7ac00fca0 con 0x7fc7b40420d0 2023-10-05T19:28:36.876 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.870+0000 7fc7c4047e40 1 -- 172.21.15.167:0/287475553 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:85 7.1f 7:f95f44c2:::notify.0:head [watch watch cookie 94602417742416] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x560a5780c730 con 0x7fc7b40420d0 2023-10-05T19:28:36.876 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.870+0000 7fc7c4047e40 1 -- 172.21.15.167:0/287475553 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:86 7.0 7:05bf5b68:::notify.1:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x560a578139f0 con 0x560a577dc5f0 2023-10-05T19:28:36.878 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.874+0000 7fc7c137b700 1 -- 172.21.15.167:0/287475553 <== osd.2 v2:172.21.15.167:6816/30037 14 ==== osd_op_reply(85 notify.0 [watch watch cookie 94602417742416] v30'160 uv159 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7fc7ac00fca0 con 0x7fc7b40420d0 2023-10-05T19:28:36.879 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.874+0000 7fc7c237d700 1 -- 172.21.15.167:0/287475553 <== osd.1 v2:172.21.15.167:6800/30035 29 ==== osd_op_reply(86 notify.1 [create] v30'159 uv159 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7fc7bc050040 con 0x560a577dc5f0 2023-10-05T19:28:36.879 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.874+0000 7fc7c4047e40 1 -- 172.21.15.167:0/287475553 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:87 7.0 7:05bf5b68:::notify.1:head [watch watch cookie 94602417745728] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x560a57813df0 con 0x560a577dc5f0 2023-10-05T19:28:36.879 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.874+0000 7fc7c4047e40 1 -- 172.21.15.167:0/287475553 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:88 7.15 7:a93a5511:::notify.2:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x560a57814690 con 0x560a5780d490 2023-10-05T19:28:36.882 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.878+0000 7fc7c237d700 1 -- 172.21.15.167:0/287475553 <== osd.1 v2:172.21.15.167:6800/30035 30 ==== osd_op_reply(87 notify.1 [watch watch cookie 94602417745728] v30'160 uv159 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7fc7bc050040 con 0x560a577dc5f0 2023-10-05T19:28:36.882 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.878+0000 7fc7c1b7c700 1 -- 172.21.15.167:0/287475553 <== osd.3 v2:172.21.15.167:6808/30036 22 ==== osd_op_reply(88 notify.2 [create] v30'159 uv159 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7fc7b4047040 con 0x560a5780d490 2023-10-05T19:28:36.882 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.878+0000 7fc7c4047e40 1 -- 172.21.15.167:0/287475553 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:89 7.15 7:a93a5511:::notify.2:head [watch watch cookie 94602417749488] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x560a578135e0 con 0x560a5780d490 2023-10-05T19:28:36.882 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.878+0000 7fc7c4047e40 1 -- 172.21.15.167:0/287475553 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:90 7.e 7:7759931f:::notify.3:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x560a57815420 con 0x7fc7b40420d0 2023-10-05T19:28:36.885 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.878+0000 7fc7c1b7c700 1 -- 172.21.15.167:0/287475553 <== osd.3 v2:172.21.15.167:6808/30036 23 ==== osd_op_reply(89 notify.2 [watch watch cookie 94602417749488] v30'160 uv159 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7fc7b4047040 con 0x560a5780d490 2023-10-05T19:28:36.885 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.878+0000 7fc7c137b700 1 -- 172.21.15.167:0/287475553 <== osd.2 v2:172.21.15.167:6816/30037 15 ==== osd_op_reply(90 notify.3 [create] v30'159 uv159 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7fc7ac00fca0 con 0x7fc7b40420d0 2023-10-05T19:28:36.885 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.878+0000 7fc7c4047e40 1 -- 172.21.15.167:0/287475553 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:91 7.e 7:7759931f:::notify.3:head [watch watch cookie 94602417752640] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x560a578157d0 con 0x7fc7b40420d0 2023-10-05T19:28:36.885 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.878+0000 7fc7c4047e40 1 -- 172.21.15.167:0/287475553 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:92 7.d 7:b4812045:::notify.4:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x560a57816310 con 0x560a577dc5f0 2023-10-05T19:28:36.888 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.882+0000 7fc7c137b700 1 -- 172.21.15.167:0/287475553 <== osd.2 v2:172.21.15.167:6816/30037 16 ==== osd_op_reply(91 notify.3 [watch watch cookie 94602417752640] v30'160 uv159 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7fc7ac00fca0 con 0x7fc7b40420d0 2023-10-05T19:28:36.888 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.882+0000 7fc7c237d700 1 -- 172.21.15.167:0/287475553 <== osd.1 v2:172.21.15.167:6800/30035 31 ==== osd_op_reply(92 notify.4 [create] v30'159 uv159 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7fc7bc050040 con 0x560a577dc5f0 2023-10-05T19:28:36.888 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.882+0000 7fc7c4047e40 1 -- 172.21.15.167:0/287475553 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:93 7.d 7:b4812045:::notify.4:head [watch watch cookie 94602417753744] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x560a57816710 con 0x560a577dc5f0 2023-10-05T19:28:36.888 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.882+0000 7fc7c4047e40 1 -- 172.21.15.167:0/287475553 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:94 7.3 7:c609908c:::notify.5:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x560a57817120 con 0x7fc7ac029420 2023-10-05T19:28:36.891 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.886+0000 7fc7c237d700 1 -- 172.21.15.167:0/287475553 <== osd.0 v2:172.21.15.118:6800/27690 23 ==== osd_op_reply(94 notify.5 [create] v30'159 uv159 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7fc7bc050040 con 0x7fc7ac029420 2023-10-05T19:28:36.891 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.886+0000 7fc7c4047e40 1 -- 172.21.15.167:0/287475553 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:95 7.3 7:c609908c:::notify.5:head [watch watch cookie 94602417761328] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x560a57817570 con 0x7fc7ac029420 2023-10-05T19:28:36.891 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.886+0000 7fc7c4047e40 1 -- 172.21.15.167:0/287475553 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:96 7.14 7:2b04a3e9:::notify.6:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x560a57817fc0 con 0x7fc7ac029420 2023-10-05T19:28:36.891 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.886+0000 7fc7c237d700 1 -- 172.21.15.167:0/287475553 <== osd.1 v2:172.21.15.167:6800/30035 32 ==== osd_op_reply(93 notify.4 [watch watch cookie 94602417753744] v30'160 uv159 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7fc7bc050040 con 0x560a577dc5f0 2023-10-05T19:28:36.894 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.886+0000 7fc7c237d700 1 -- 172.21.15.167:0/287475553 <== osd.0 v2:172.21.15.118:6800/27690 24 ==== osd_op_reply(95 notify.5 [watch watch cookie 94602417761328] v30'160 uv159 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7fc7bc050040 con 0x7fc7ac029420 2023-10-05T19:28:36.894 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.886+0000 7fc7c237d700 1 -- 172.21.15.167:0/287475553 <== osd.0 v2:172.21.15.118:6800/27690 25 ==== osd_op_reply(96 notify.6 [create] v30'159 uv159 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7fc7bc050040 con 0x7fc7ac029420 2023-10-05T19:28:36.894 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.886+0000 7fc7c4047e40 1 -- 172.21.15.167:0/287475553 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:97 7.14 7:2b04a3e9:::notify.6:head [watch watch cookie 94602417757664] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x560a57817570 con 0x7fc7ac029420 2023-10-05T19:28:36.894 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.886+0000 7fc7c4047e40 1 -- 172.21.15.167:0/287475553 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:98 7.9 7:93e5b521:::notify.7:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x560a57818390 con 0x7fc7ac029420 2023-10-05T19:28:36.896 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.890+0000 7fc7c237d700 1 -- 172.21.15.167:0/287475553 <== osd.0 v2:172.21.15.118:6800/27690 26 ==== osd_op_reply(97 notify.6 [watch watch cookie 94602417757664] v30'160 uv159 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7fc7bc050040 con 0x7fc7ac029420 2023-10-05T19:28:36.897 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.890+0000 7fc7c237d700 1 -- 172.21.15.167:0/287475553 <== osd.0 v2:172.21.15.118:6800/27690 27 ==== osd_op_reply(98 notify.7 [create] v30'159 uv159 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7fc7bc050040 con 0x7fc7ac029420 2023-10-05T19:28:36.897 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.890+0000 7fc7c4047e40 1 -- 172.21.15.167:0/287475553 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:99 7.9 7:93e5b521:::notify.7:head [watch watch cookie 94602417765984] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x560a57817570 con 0x7fc7ac029420 2023-10-05T19:28:36.897 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.890+0000 7fc7c4047e40 20 add_watcher() i=0 2023-10-05T19:28:36.897 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.890+0000 7fc7c4047e40 20 add_watcher() i=1 2023-10-05T19:28:36.897 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.890+0000 7fc7c4047e40 20 add_watcher() i=2 2023-10-05T19:28:36.897 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.890+0000 7fc7c4047e40 20 add_watcher() i=3 2023-10-05T19:28:36.897 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.890+0000 7fc7c4047e40 20 add_watcher() i=4 2023-10-05T19:28:36.897 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.890+0000 7fc7c4047e40 20 add_watcher() i=5 2023-10-05T19:28:36.897 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.890+0000 7fc7c4047e40 20 add_watcher() i=6 2023-10-05T19:28:36.899 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.894+0000 7fc7c237d700 1 -- 172.21.15.167:0/287475553 <== osd.0 v2:172.21.15.118:6800/27690 28 ==== osd_op_reply(99 notify.7 [watch watch cookie 94602417765984] v30'160 uv159 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7fc7bc050040 con 0x7fc7ac029420 2023-10-05T19:28:36.899 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.894+0000 7fc7c4047e40 20 add_watcher() i=7 2023-10-05T19:28:36.899 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.894+0000 7fc7c4047e40 2 all 8 watchers are set, enabling cache 2023-10-05T19:28:36.899 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.894+0000 7fc7c4047e40 1 -- 172.21.15.167:0/287475553 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:100 6.4 6:22d26bf9:::data_loggenerations_metadata:head [call version.check_conds in=74b,call version.read in=11b,read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x560a5781b710 con 0x560a577dc5f0 2023-10-05T19:28:36.899 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.894+0000 7fc7c237d700 1 -- 172.21.15.167:0/287475553 <== osd.1 v2:172.21.15.167:6800/30035 33 ==== osd_op_reply(100 data_loggenerations_metadata [call,call out=48b,read 0~28 out=28b] v0'0 uv1 ondisk = 0) v8 ==== 256+0+76 (crc 0 0 0) 0x7fc7bc050040 con 0x560a577dc5f0 2023-10-05T19:28:36.900 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.894+0000 7fc7c4047e40 1 -- 172.21.15.167:0/287475553 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:101 6.4 6:22d26bf9:::data_loggenerations_metadata:head [watch watch cookie 94602417762896] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x560a57818bd0 con 0x560a577dc5f0 2023-10-05T19:28:36.902 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.894+0000 7fc7c237d700 1 -- 172.21.15.167:0/287475553 <== osd.1 v2:172.21.15.167:6800/30035 34 ==== osd_op_reply(101 data_loggenerations_metadata [watch watch cookie 94602417762896] v30'153 uv1 ondisk = 0) v8 ==== 172+0+0 (crc 0 0 0) 0x7fc7bc050040 con 0x560a577dc5f0 2023-10-05T19:28:36.902 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.898+0000 7fc792fe5700 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2023-10-05T19:28:36.902 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.898+0000 7fc7c4047e40 20 check_secure_mon_conn(): auth registy supported: methods=[2] modes=[2,1] 2023-10-05T19:28:36.902 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.898+0000 7fc7c4047e40 20 check_secure_mon_conn(): mode 1 is insecure 2023-10-05T19:28:36.903 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.898+0000 7fc7c4047e40 1 Processor -- start 2023-10-05T19:28:36.903 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.898+0000 7fc7c4047e40 1 -- start start 2023-10-05T19:28:36.904 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.898+0000 7fc7c4047e40 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x560a57825b10 0x560a57825ee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-05T19:28:36.904 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.898+0000 7fc7c4047e40 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x560a57826420 con 0x560a57825b10 2023-10-05T19:28:36.904 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.898+0000 7fc7c137b700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x560a57825b10 0x560a57825ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-05T19:28:36.904 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.898+0000 7fc7c137b700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x560a57825b10 0x560a57825ee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.118:3300/0 says I am v2:172.21.15.167:41898/0 (socket says 172.21.15.167:41898) 2023-10-05T19:28:36.904 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.898+0000 7fc7c137b700 1 -- 172.21.15.167:0/722735005 learned_addr learned my addr 172.21.15.167:0/722735005 (peer_addr_for_me v2:172.21.15.167:0/0) 2023-10-05T19:28:36.905 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.898+0000 7fc7c137b700 1 -- 172.21.15.167:0/722735005 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x560a57826560 con 0x560a57825b10 2023-10-05T19:28:36.905 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.898+0000 7fc7c137b700 1 --2- 172.21.15.167:0/722735005 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x560a57825b10 0x560a57825ee0 secure :-1 s=READY pgs=456 cs=0 l=1 rev1=1 crypto rx=0x7fc7ac00ffd0 tx=0x7fc7ac011000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=94da9630e20ce165 server_cookie=0 in_seq=0 out_seq=0 2023-10-05T19:28:36.905 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.898+0000 7fc7927e4700 1 -- 172.21.15.167:0/722735005 <== mon.0 v2:172.21.15.118:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fc7ac030040 con 0x560a57825b10 2023-10-05T19:28:36.905 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.898+0000 7fc7927e4700 1 -- 172.21.15.167:0/722735005 <== mon.0 v2:172.21.15.118:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc7ac036820 con 0x560a57825b10 2023-10-05T19:28:36.905 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.898+0000 7fc7927e4700 1 -- 172.21.15.167:0/722735005 <== mon.0 v2:172.21.15.118:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fc7ac0369f0 con 0x560a57825b10 2023-10-05T19:28:36.906 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.898+0000 7fc7c4047e40 1 -- 172.21.15.167:0/722735005 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x560a57825b10 msgr2=0x560a57825ee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-05T19:28:36.906 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.898+0000 7fc7c4047e40 1 --2- 172.21.15.167:0/722735005 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x560a57825b10 0x560a57825ee0 secure :-1 s=READY pgs=456 cs=0 l=1 rev1=1 crypto rx=0x7fc7ac00ffd0 tx=0x7fc7ac011000 comp rx=0 tx=0).stop 2023-10-05T19:28:36.906 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.902+0000 7fc7c4047e40 1 -- 172.21.15.167:0/722735005 shutdown_connections 2023-10-05T19:28:36.906 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.902+0000 7fc7c4047e40 1 --2- 172.21.15.167:0/722735005 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x560a57825b10 0x560a57825ee0 unknown :-1 s=CLOSED pgs=456 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-05T19:28:36.906 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.902+0000 7fc7c4047e40 1 -- 172.21.15.167:0/722735005 >> 172.21.15.167:0/722735005 conn(0x560a57820ed0 msgr2=0x560a5781f080 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-05T19:28:36.906 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.902+0000 7fc7c4047e40 1 -- 172.21.15.167:0/722735005 shutdown_connections 2023-10-05T19:28:36.907 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.902+0000 7fc7c4047e40 1 -- 172.21.15.167:0/722735005 wait complete. 2023-10-05T19:28:36.907 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.902+0000 7fc7c4047e40 1 Processor -- start 2023-10-05T19:28:36.907 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.902+0000 7fc7c4047e40 1 -- start start 2023-10-05T19:28:36.908 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.902+0000 7fc7c4047e40 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x560a57825b10 0x560a5782b5a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-05T19:28:36.908 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.902+0000 7fc7c4047e40 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x560a5782c5a0 con 0x560a57825b10 2023-10-05T19:28:36.908 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.902+0000 7fc7c137b700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x560a57825b10 0x560a5782b5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-05T19:28:36.908 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.902+0000 7fc7c137b700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x560a57825b10 0x560a5782b5a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.118:3300/0 says I am v2:172.21.15.167:41908/0 (socket says 172.21.15.167:41908) 2023-10-05T19:28:36.908 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.902+0000 7fc7c137b700 1 -- 172.21.15.167:0/1580933318 learned_addr learned my addr 172.21.15.167:0/1580933318 (peer_addr_for_me v2:172.21.15.167:0/0) 2023-10-05T19:28:36.909 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.902+0000 7fc7c137b700 1 -- 172.21.15.167:0/1580933318 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x560a577d8c20 con 0x560a57825b10 2023-10-05T19:28:36.909 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.902+0000 7fc7c137b700 1 --2- 172.21.15.167:0/1580933318 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x560a57825b10 0x560a5782b5a0 secure :-1 s=READY pgs=457 cs=0 l=1 rev1=1 crypto rx=0x7fc7ac00ffd0 tx=0x7fc7ac042000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-05T19:28:36.909 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.902+0000 7fc790fe1700 1 -- 172.21.15.167:0/1580933318 <== mon.0 v2:172.21.15.118:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fc7ac030050 con 0x560a57825b10 2023-10-05T19:28:36.909 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.902+0000 7fc790fe1700 1 -- 172.21.15.167:0/1580933318 <== mon.0 v2:172.21.15.118:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc7ac04e070 con 0x560a57825b10 2023-10-05T19:28:36.909 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.902+0000 7fc7c4047e40 1 -- 172.21.15.167:0/1580933318 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x560a5782bba0 con 0x560a57825b10 2023-10-05T19:28:36.909 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.902+0000 7fc790fe1700 1 -- 172.21.15.167:0/1580933318 <== mon.0 v2:172.21.15.118:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fc7ac042d60 con 0x560a57825b10 2023-10-05T19:28:36.909 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.902+0000 7fc7c4047e40 1 -- 172.21.15.167:0/1580933318 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x560a5782c040 con 0x560a57825b10 2023-10-05T19:28:36.910 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.906+0000 7fc790fe1700 1 -- 172.21.15.167:0/1580933318 <== mon.0 v2:172.21.15.118:3300/0 4 ==== mgrmap(e 5) v1 ==== 40329+0+0 (secure 0 0 0) 0x7fc7ac050070 con 0x560a57825b10 2023-10-05T19:28:36.911 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.906+0000 7fc790fe1700 1 --2- 172.21.15.167:0/1580933318 >> [v2:172.21.15.167:6824/29889,v1:172.21.15.167:6825/29889] conn(0x7fc708030c50 0x7fc708033100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-05T19:28:36.911 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.906+0000 7fc790fe1700 1 -- 172.21.15.167:0/1580933318 <== mon.0 v2:172.21.15.118:3300/0 5 ==== osd_map(30..30 src has 1..30) v4 ==== 6057+0+0 (secure 0 0 0) 0x7fc7ac03b030 con 0x560a57825b10 2023-10-05T19:28:36.911 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.906+0000 7fc7c4047e40 5 note: GC not initialized 2023-10-05T19:28:36.911 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.906+0000 7fc7c237d700 1 --2- 172.21.15.167:0/1580933318 >> [v2:172.21.15.167:6824/29889,v1:172.21.15.167:6825/29889] conn(0x7fc708030c50 0x7fc708033100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-05T19:28:36.912 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.906+0000 7fc7c237d700 1 --2- 172.21.15.167:0/1580933318 >> [v2:172.21.15.167:6824/29889,v1:172.21.15.167:6825/29889] conn(0x7fc708030c50 0x7fc708033100 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x560a578177e0 tx=0x7fc7bc000f90 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-05T19:28:36.914 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.906+0000 7fc701ffb700 20 reqs_thread_entry: start 2023-10-05T19:28:36.915 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.910+0000 7fc7c4047e40 20 init_complete bucket index max shards: 11 2023-10-05T19:28:36.915 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.910+0000 7fc6f37fe700 20 reqs_thread_entry: start 2023-10-05T19:28:36.915 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.910+0000 7fc7c4047e40 10 rgw notify: Started notification manager with: 1 workers 2023-10-05T19:28:36.915 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.910+0000 7fc6f1ffb700 20 rgw notify: INFO: next queues processing will happen at: Thu Oct 5 19:29:07 2023 2023-10-05T19:28:36.915 INFO:teuthology.orchestra.run.smithi167.stderr: 2023-10-05T19:28:36.915 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.910+0000 7fc6f17fa700 20 reqs_thread_entry: start 2023-10-05T19:28:36.915 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.910+0000 7fc7c4047e40 10 log_show_next pos 0 bl 0 off 0 eof 0 2023-10-05T19:28:36.916 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.910+0000 7fc7c4047e40 1 --2- 172.21.15.167:0/1580933318 >> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] conn(0x560a57892b10 0x560a57894ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-05T19:28:36.916 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.910+0000 7fc7c4047e40 1 -- 172.21.15.167:0/1580933318 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:1 6.18 6:19c98439:::2023-10-05-19-9a9df965-6612-45d4-a45f-6e3c889bacc2.4223.4-myfoo4:head [read 0~1048576] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x560a57895430 con 0x560a57892b10 2023-10-05T19:28:36.916 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.910+0000 7fc7c1b7c700 1 --2- 172.21.15.167:0/1580933318 >> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] conn(0x560a57892b10 0x560a57894ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-05T19:28:36.916 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.910+0000 7fc7c1b7c700 1 --2- 172.21.15.167:0/1580933318 >> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] conn(0x560a57892b10 0x560a57894ef0 crc :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-05T19:28:36.917 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.910+0000 7fc7c1b7c700 1 -- 172.21.15.167:0/1580933318 <== osd.0 v2:172.21.15.118:6800/27690 1 ==== osd_op_reply(1 2023-10-05-19-9a9df965-6612-45d4-a45f-6e3c889bacc2.4223.4-myfoo4 [read 0~850 out=850b] v0'0 uv16 ondisk = 0) v8 ==== 208+0+850 (crc 0 0 0) 0x7fc7b4047040 con 0x560a57892b10 2023-10-05T19:28:36.917 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.910+0000 7fc7c4047e40 10 read 850 2023-10-05T19:28:36.918 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.910+0000 7fc7c4047e40 10 log_show_next pos 850 bl 850 off 419 eof 1 2023-10-05T19:28:36.918 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.910+0000 7fc7c4047e40 10 log_show_next pos 850 bl 850 off 850 eof 1 2023-10-05T19:28:36.918 INFO:teuthology.orchestra.run.smithi167.stdout:{ 2023-10-05T19:28:36.918 INFO:teuthology.orchestra.run.smithi167.stdout: "bucket_id": "9a9df965-6612-45d4-a45f-6e3c889bacc2.4223.4", 2023-10-05T19:28:36.918 INFO:teuthology.orchestra.run.smithi167.stdout: "bucket_owner": "foo", 2023-10-05T19:28:36.918 INFO:teuthology.orchestra.run.smithi167.stdout: "bucket": "myfoo4", 2023-10-05T19:28:36.918 INFO:teuthology.orchestra.run.smithi167.stdout: "log_entries": [ 2023-10-05T19:28:36.919 INFO:teuthology.orchestra.run.smithi167.stdout: { 2023-10-05T19:28:36.919 INFO:teuthology.orchestra.run.smithi167.stdout: "bucket": "myfoo4", 2023-10-05T19:28:36.919 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2023-10-05T19:28:30.822276Z", 2023-10-05T19:28:36.919 INFO:teuthology.orchestra.run.smithi167.stdout: "time_local": "2023-10-05T19:28:30.822276+0000", 2023-10-05T19:28:36.919 INFO:teuthology.orchestra.run.smithi167.stdout: "remote_addr": "172.21.0.51", 2023-10-05T19:28:36.919 INFO:teuthology.orchestra.run.smithi167.stdout: "user": "foo", 2023-10-05T19:28:36.919 INFO:teuthology.orchestra.run.smithi167.stdout: "operation": "create_bucket", 2023-10-05T19:28:36.919 INFO:teuthology.orchestra.run.smithi167.stdout: "uri": "PUT /myfoo4/ HTTP/1.1", 2023-10-05T19:28:36.919 INFO:teuthology.orchestra.run.smithi167.stdout: "http_status": "200", 2023-10-05T19:28:36.919 INFO:teuthology.orchestra.run.smithi167.stdout: "error_code": "", 2023-10-05T19:28:36.919 INFO:teuthology.orchestra.run.smithi167.stdout: "bytes_sent": 0, 2023-10-05T19:28:36.919 INFO:teuthology.orchestra.run.smithi167.stdout: "bytes_received": 0, 2023-10-05T19:28:36.919 INFO:teuthology.orchestra.run.smithi167.stdout: "object_size": 0, 2023-10-05T19:28:36.920 INFO:teuthology.orchestra.run.smithi167.stdout: "total_time": 19, 2023-10-05T19:28:36.920 INFO:teuthology.orchestra.run.smithi167.stdout: "user_agent": "Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic", 2023-10-05T19:28:36.920 INFO:teuthology.orchestra.run.smithi167.stdout: "referrer": "", 2023-10-05T19:28:36.920 INFO:teuthology.orchestra.run.smithi167.stdout: "trans_id": "tx00000990e075e6bd5e55a-00651f0e5e-107d-default", 2023-10-05T19:28:36.920 INFO:teuthology.orchestra.run.smithi167.stdout: "authentication_type": "Local", 2023-10-05T19:28:36.920 INFO:teuthology.orchestra.run.smithi167.stdout: "access_key_id": "9te6NH5mcdcq0Tc5i8i1", 2023-10-05T19:28:36.920 INFO:teuthology.orchestra.run.smithi167.stdout: "temp_url": false 2023-10-05T19:28:36.920 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2023-10-05T19:28:36.920 INFO:teuthology.orchestra.run.smithi167.stdout: { 2023-10-05T19:28:36.920 INFO:teuthology.orchestra.run.smithi167.stdout: "bucket": "myfoo4", 2023-10-05T19:28:36.920 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2023-10-05T19:28:31.038271Z", 2023-10-05T19:28:36.920 INFO:teuthology.orchestra.run.smithi167.stdout: "time_local": "2023-10-05T19:28:31.038271+0000", 2023-10-05T19:28:36.921 INFO:teuthology.orchestra.run.smithi167.stdout: "remote_addr": "172.21.0.51", 2023-10-05T19:28:36.921 INFO:teuthology.orchestra.run.smithi167.stdout: "user": "foo", 2023-10-05T19:28:36.921 INFO:teuthology.orchestra.run.smithi167.stdout: "operation": "delete_bucket", 2023-10-05T19:28:36.921 INFO:teuthology.orchestra.run.smithi167.stdout: "uri": "DELETE /myfoo4/ HTTP/1.1", 2023-10-05T19:28:36.921 INFO:teuthology.orchestra.run.smithi167.stdout: "http_status": "204", 2023-10-05T19:28:36.921 INFO:teuthology.orchestra.run.smithi167.stdout: "error_code": "NoContent", 2023-10-05T19:28:36.921 INFO:teuthology.orchestra.run.smithi167.stdout: "bytes_sent": 0, 2023-10-05T19:28:36.921 INFO:teuthology.orchestra.run.smithi167.stdout: "bytes_received": 0, 2023-10-05T19:28:36.921 INFO:teuthology.orchestra.run.smithi167.stdout: "object_size": 0, 2023-10-05T19:28:36.921 INFO:teuthology.orchestra.run.smithi167.stdout: "total_time": 75, 2023-10-05T19:28:36.921 INFO:teuthology.orchestra.run.smithi167.stdout: "user_agent": "Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic", 2023-10-05T19:28:36.921 INFO:teuthology.orchestra.run.smithi167.stdout: "referrer": "", 2023-10-05T19:28:36.922 INFO:teuthology.orchestra.run.smithi167.stdout: "trans_id": "tx00000ca5b400930287da7-00651f0e5f-107d-default", 2023-10-05T19:28:36.922 INFO:teuthology.orchestra.run.smithi167.stdout: "authentication_type": "Local", 2023-10-05T19:28:36.922 INFO:teuthology.orchestra.run.smithi167.stdout: "access_key_id": "9te6NH5mcdcq0Tc5i8i1", 2023-10-05T19:28:36.922 INFO:teuthology.orchestra.run.smithi167.stdout: "temp_url": false 2023-10-05T19:28:36.922 INFO:teuthology.orchestra.run.smithi167.stdout: } 2023-10-05T19:28:36.922 INFO:teuthology.orchestra.run.smithi167.stdout: ], 2023-10-05T19:28:36.922 INFO:teuthology.orchestra.run.smithi167.stdout: "log_sum": { 2023-10-05T19:28:36.922 INFO:teuthology.orchestra.run.smithi167.stdout: "bytes_sent": 0, 2023-10-05T19:28:36.922 INFO:teuthology.orchestra.run.smithi167.stdout: "bytes_received": 0, 2023-10-05T19:28:36.922 INFO:teuthology.orchestra.run.smithi167.stdout: "total_time": 94, 2023-10-05T19:28:36.922 INFO:teuthology.orchestra.run.smithi167.stdout: "total_entries": 2 2023-10-05T19:28:36.922 INFO:teuthology.orchestra.run.smithi167.stdout: } 2023-10-05T19:28:36.923 INFO:teuthology.orchestra.run.smithi167.stdout:} 2023-10-05T19:28:36.923 INFO:teuthology.orchestra.run.smithi167.stdout: 2023-10-05T19:28:36.923 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.914+0000 7fc7c4047e40 1 -- 172.21.15.167:0/287475553 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:102 7.1f 7:f95f44c2:::notify.0:head [watch unwatch cookie 94602417742416] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x560a57852140 con 0x7fc7b40420d0 2023-10-05T19:28:36.924 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.914+0000 7fc7c137b700 1 -- 172.21.15.167:0/287475553 <== osd.2 v2:172.21.15.167:6816/30037 17 ==== osd_op_reply(102 notify.0 [watch unwatch cookie 94602417742416] v30'161 uv159 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7fc7ac048020 con 0x7fc7b40420d0 2023-10-05T19:28:36.924 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.914+0000 7fc7c4047e40 20 remove_watcher() i=0 2023-10-05T19:28:36.924 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.914+0000 7fc7c4047e40 2 removed watcher, disabling cache 2023-10-05T19:28:36.924 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.914+0000 7fc7c4047e40 1 -- 172.21.15.167:0/287475553 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:103 7.0 7:05bf5b68:::notify.1:head [watch unwatch cookie 94602417745728] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x560a57812250 con 0x560a577dc5f0 2023-10-05T19:28:36.924 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.918+0000 7fc7c237d700 1 -- 172.21.15.167:0/287475553 <== osd.1 v2:172.21.15.167:6800/30035 35 ==== osd_op_reply(103 notify.1 [watch unwatch cookie 94602417745728] v30'161 uv159 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7fc7bc050040 con 0x560a577dc5f0 2023-10-05T19:28:36.924 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.918+0000 7fc7c4047e40 20 remove_watcher() i=1 2023-10-05T19:28:36.924 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.918+0000 7fc7c4047e40 1 -- 172.21.15.167:0/287475553 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:104 7.15 7:a93a5511:::notify.2:head [watch unwatch cookie 94602417749488] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x560a57812f40 con 0x560a5780d490 2023-10-05T19:28:36.926 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.922+0000 7fc7c1b7c700 1 -- 172.21.15.167:0/287475553 <== osd.3 v2:172.21.15.167:6808/30036 24 ==== osd_op_reply(104 notify.2 [watch unwatch cookie 94602417749488] v30'161 uv159 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7fc7b4047040 con 0x560a5780d490 2023-10-05T19:28:36.926 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.922+0000 7fc7c4047e40 20 remove_watcher() i=2 2023-10-05T19:28:36.927 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.922+0000 7fc7c4047e40 1 -- 172.21.15.167:0/287475553 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:105 7.e 7:7759931f:::notify.3:head [watch unwatch cookie 94602417752640] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x560a57813df0 con 0x7fc7b40420d0 2023-10-05T19:28:36.929 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.922+0000 7fc7c137b700 1 -- 172.21.15.167:0/287475553 <== osd.2 v2:172.21.15.167:6816/30037 18 ==== osd_op_reply(105 notify.3 [watch unwatch cookie 94602417752640] v30'161 uv159 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7fc7ac048020 con 0x7fc7b40420d0 2023-10-05T19:28:36.929 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.922+0000 7fc7c4047e40 20 remove_watcher() i=3 2023-10-05T19:28:36.929 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.922+0000 7fc7c4047e40 1 -- 172.21.15.167:0/287475553 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:106 7.d 7:b4812045:::notify.4:head [watch unwatch cookie 94602417753744] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x560a57813df0 con 0x560a577dc5f0 2023-10-05T19:28:36.932 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.926+0000 7fc7c237d700 1 -- 172.21.15.167:0/287475553 <== osd.1 v2:172.21.15.167:6800/30035 36 ==== osd_op_reply(106 notify.4 [watch unwatch cookie 94602417753744] v30'161 uv159 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7fc7bc050040 con 0x560a577dc5f0 2023-10-05T19:28:36.932 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.926+0000 7fc7c4047e40 20 remove_watcher() i=4 2023-10-05T19:28:36.932 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.926+0000 7fc7c4047e40 1 -- 172.21.15.167:0/287475553 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:107 7.3 7:c609908c:::notify.5:head [watch unwatch cookie 94602417761328] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x560a57813df0 con 0x7fc7ac029420 2023-10-05T19:28:36.935 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.930+0000 7fc7c237d700 1 -- 172.21.15.167:0/287475553 <== osd.0 v2:172.21.15.118:6800/27690 29 ==== osd_op_reply(107 notify.5 [watch unwatch cookie 94602417761328] v30'161 uv159 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7fc7bc050040 con 0x7fc7ac029420 2023-10-05T19:28:36.935 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.930+0000 7fc7c4047e40 20 remove_watcher() i=5 2023-10-05T19:28:36.935 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.930+0000 7fc7c4047e40 1 -- 172.21.15.167:0/287475553 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:108 7.14 7:2b04a3e9:::notify.6:head [watch unwatch cookie 94602417757664] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x560a57813df0 con 0x7fc7ac029420 2023-10-05T19:28:36.938 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.934+0000 7fc7c237d700 1 -- 172.21.15.167:0/287475553 <== osd.0 v2:172.21.15.118:6800/27690 30 ==== osd_op_reply(108 notify.6 [watch unwatch cookie 94602417757664] v30'161 uv159 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7fc7bc050040 con 0x7fc7ac029420 2023-10-05T19:28:36.938 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.934+0000 7fc7c4047e40 20 remove_watcher() i=6 2023-10-05T19:28:36.939 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.934+0000 7fc7c4047e40 1 -- 172.21.15.167:0/287475553 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:109 7.9 7:93e5b521:::notify.7:head [watch unwatch cookie 94602417765984] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x560a57813df0 con 0x7fc7ac029420 2023-10-05T19:28:36.941 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.934+0000 7fc7c237d700 1 -- 172.21.15.167:0/287475553 <== osd.0 v2:172.21.15.118:6800/27690 31 ==== osd_op_reply(109 notify.7 [watch unwatch cookie 94602417765984] v30'161 uv159 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7fc7bc050040 con 0x7fc7ac029420 2023-10-05T19:28:36.941 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.934+0000 7fc7c4047e40 20 remove_watcher() i=7 2023-10-05T19:28:36.943 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.938+0000 7fc792fe5700 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2023-10-05T19:28:36.943 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.938+0000 7fc7c4047e40 1 -- 172.21.15.167:0/287475553 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:110 6.4 6:22d26bf9:::data_loggenerations_metadata:head [watch unwatch cookie 94602417762896] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x560a57813df0 con 0x560a577dc5f0 2023-10-05T19:28:36.946 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.938+0000 7fc7c237d700 1 -- 172.21.15.167:0/287475553 <== osd.1 v2:172.21.15.167:6800/30035 37 ==== osd_op_reply(110 data_loggenerations_metadata [watch unwatch cookie 94602417762896] v30'154 uv1 ondisk = 0) v8 ==== 172+0+0 (crc 0 0 0) 0x7fc7bc050040 con 0x560a577dc5f0 2023-10-05T19:28:36.946 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.938+0000 7fc7c4047e40 1 -- 172.21.15.167:0/287475553 >> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] conn(0x7fc7ac029420 msgr2=0x7fc7ac029830 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-05T19:28:36.946 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.938+0000 7fc7c4047e40 1 --2- 172.21.15.167:0/287475553 >> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] conn(0x7fc7ac029420 0x7fc7ac029830 crc :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-05T19:28:36.946 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.938+0000 7fc7c4047e40 1 -- 172.21.15.167:0/287475553 >> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] conn(0x560a577dc5f0 msgr2=0x560a578099f0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-05T19:28:36.946 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.938+0000 7fc7c4047e40 1 --2- 172.21.15.167:0/287475553 >> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] conn(0x560a577dc5f0 0x560a578099f0 crc :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-05T19:28:36.946 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.938+0000 7fc7c4047e40 1 -- 172.21.15.167:0/287475553 >> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] conn(0x7fc7b40420d0 msgr2=0x7fc7b403d620 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-05T19:28:36.947 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.938+0000 7fc7c4047e40 1 --2- 172.21.15.167:0/287475553 >> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] conn(0x7fc7b40420d0 0x7fc7b403d620 crc :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-05T19:28:36.947 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.938+0000 7fc7c4047e40 1 -- 172.21.15.167:0/287475553 >> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] conn(0x560a5780d490 msgr2=0x560a5780f870 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-05T19:28:36.947 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.938+0000 7fc7c4047e40 1 --2- 172.21.15.167:0/287475553 >> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] conn(0x560a5780d490 0x560a5780f870 crc :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-05T19:28:36.947 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.938+0000 7fc7c4047e40 1 -- 172.21.15.167:0/287475553 >> [v2:172.21.15.167:6824/29889,v1:172.21.15.167:6825/29889] conn(0x7fc7a4030c50 msgr2=0x7fc7a4033100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-05T19:28:36.947 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.938+0000 7fc7c4047e40 1 --2- 172.21.15.167:0/287475553 >> [v2:172.21.15.167:6824/29889,v1:172.21.15.167:6825/29889] conn(0x7fc7a4030c50 0x7fc7a4033100 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7fc7ac005d80 tx=0x7fc7ac014000 comp rx=0 tx=0).stop 2023-10-05T19:28:36.947 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.938+0000 7fc7c237d700 1 -- 172.21.15.167:0/287475553 reap_dead start 2023-10-05T19:28:36.947 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.938+0000 7fc7c4047e40 1 -- 172.21.15.167:0/287475553 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x560a577d8010 msgr2=0x560a577fdf60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-05T19:28:36.947 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.938+0000 7fc7c4047e40 1 --2- 172.21.15.167:0/287475553 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x560a577d8010 0x560a577fdf60 secure :-1 s=READY pgs=455 cs=0 l=1 rev1=1 crypto rx=0x7fc7b4004000 tx=0x7fc7b4016000 comp rx=0 tx=0).stop 2023-10-05T19:28:36.947 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.938+0000 7fc7c4047e40 1 -- 172.21.15.167:0/287475553 shutdown_connections 2023-10-05T19:28:36.947 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.938+0000 7fc7c4047e40 1 --2- 172.21.15.167:0/287475553 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x560a577d8010 0x560a577fdf60 unknown :-1 s=CLOSED pgs=455 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-05T19:28:36.947 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.938+0000 7fc7c4047e40 1 -- 172.21.15.167:0/287475553 >> 172.21.15.167:0/287475553 conn(0x560a5774b860 msgr2=0x560a577f1810 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-05T19:28:36.947 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.938+0000 7fc7c4047e40 1 -- 172.21.15.167:0/287475553 shutdown_connections 2023-10-05T19:28:36.948 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.938+0000 7fc7c4047e40 1 -- 172.21.15.167:0/287475553 wait complete. 2023-10-05T19:28:36.949 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.942+0000 7fc7c4047e40 1 -- 172.21.15.167:0/1580933318 >> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] conn(0x560a57892b10 msgr2=0x560a57894ef0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-05T19:28:36.949 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.942+0000 7fc7c4047e40 1 --2- 172.21.15.167:0/1580933318 >> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] conn(0x560a57892b10 0x560a57894ef0 crc :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-05T19:28:36.949 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.942+0000 7fc7c4047e40 1 -- 172.21.15.167:0/1580933318 >> [v2:172.21.15.167:6824/29889,v1:172.21.15.167:6825/29889] conn(0x7fc708030c50 msgr2=0x7fc708033100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-05T19:28:36.949 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.942+0000 7fc7c4047e40 1 --2- 172.21.15.167:0/1580933318 >> [v2:172.21.15.167:6824/29889,v1:172.21.15.167:6825/29889] conn(0x7fc708030c50 0x7fc708033100 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x560a578177e0 tx=0x7fc7bc000f90 comp rx=0 tx=0).stop 2023-10-05T19:28:36.949 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.942+0000 7fc7c4047e40 1 -- 172.21.15.167:0/1580933318 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x560a57825b10 msgr2=0x560a5782b5a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-05T19:28:36.949 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.942+0000 7fc7c4047e40 1 --2- 172.21.15.167:0/1580933318 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x560a57825b10 0x560a5782b5a0 secure :-1 s=READY pgs=457 cs=0 l=1 rev1=1 crypto rx=0x7fc7ac00ffd0 tx=0x7fc7ac042000 comp rx=0 tx=0).stop 2023-10-05T19:28:36.949 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.942+0000 7fc7c4047e40 1 -- 172.21.15.167:0/1580933318 shutdown_connections 2023-10-05T19:28:36.949 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.942+0000 7fc7c4047e40 1 --2- 172.21.15.167:0/1580933318 >> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] conn(0x560a57892b10 0x560a57894ef0 unknown :-1 s=CLOSED pgs=73 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-05T19:28:36.949 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.942+0000 7fc7c4047e40 1 --2- 172.21.15.167:0/1580933318 >> [v2:172.21.15.167:6824/29889,v1:172.21.15.167:6825/29889] conn(0x7fc708030c50 0x7fc708033100 unknown :-1 s=CLOSED pgs=159 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-05T19:28:36.949 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.942+0000 7fc7c4047e40 1 --2- 172.21.15.167:0/1580933318 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x560a57825b10 0x560a5782b5a0 unknown :-1 s=CLOSED pgs=457 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-05T19:28:36.950 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.942+0000 7fc7c4047e40 1 -- 172.21.15.167:0/1580933318 >> 172.21.15.167:0/1580933318 conn(0x560a57820ed0 msgr2=0x560a5781efd0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-05T19:28:36.950 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.942+0000 7fc7c4047e40 1 -- 172.21.15.167:0/1580933318 shutdown_connections 2023-10-05T19:28:36.950 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.942+0000 7fc7c4047e40 1 -- 172.21.15.167:0/1580933318 wait complete. 2023-10-05T19:28:36.953 DEBUG:tasks.util.rgw: json result: {'bucket_id': '9a9df965-6612-45d4-a45f-6e3c889bacc2.4223.4', 'bucket_owner': 'foo', 'bucket': 'myfoo4', 'log_entries': [{'bucket': 'myfoo4', 'time': '2023-10-05T19:28:30.822276Z', 'time_local': '2023-10-05T19:28:30.822276+0000', 'remote_addr': '172.21.0.51', 'user': 'foo', 'operation': 'create_bucket', 'uri': 'PUT /myfoo4/ HTTP/1.1', 'http_status': '200', 'error_code': '', 'bytes_sent': 0, 'bytes_received': 0, 'object_size': 0, 'total_time': 19, 'user_agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'referrer': '', 'trans_id': 'tx00000990e075e6bd5e55a-00651f0e5e-107d-default', 'authentication_type': 'Local', 'access_key_id': '9te6NH5mcdcq0Tc5i8i1', 'temp_url': False}, {'bucket': 'myfoo4', 'time': '2023-10-05T19:28:31.038271Z', 'time_local': '2023-10-05T19:28:31.038271+0000', 'remote_addr': '172.21.0.51', 'user': 'foo', 'operation': 'delete_bucket', 'uri': 'DELETE /myfoo4/ HTTP/1.1', 'http_status': '204', 'error_code': 'NoContent', 'bytes_sent': 0, 'bytes_received': 0, 'object_size': 0, 'total_time': 75, 'user_agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'referrer': '', 'trans_id': 'tx00000ca5b400930287da7-00651f0e5f-107d-default', 'authentication_type': 'Local', 'access_key_id': '9te6NH5mcdcq0Tc5i8i1', 'temp_url': False}], 'log_sum': {'bytes_sent': 0, 'bytes_received': 0, 'total_time': 94, 'total_entries': 2}} 2023-10-05T19:28:36.953 DEBUG:tasks.radosgw_admin:checking log entry: 2023-10-05T19:28:36.953 DEBUG:tasks.radosgw_admin:checking log entry: 2023-10-05T19:28:36.953 INFO:tasks.util.rgw:rgwadmin: client.0 : ['log', 'rm', '--object', '2023-10-05-19-9a9df965-6612-45d4-a45f-6e3c889bacc2.4223.4-myfoo4'] 2023-10-05T19:28:36.953 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'ceph', 'log', 'rm', '--object', '2023-10-05-19-9a9df965-6612-45d4-a45f-6e3c889bacc2.4223.4-myfoo4'] 2023-10-05T19:28:36.953 DEBUG:teuthology.orchestra.run.smithi167:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster ceph log rm --object 2023-10-05-19-9a9df965-6612-45d4-a45f-6e3c889bacc2.4223.4-myfoo4 2023-10-05T19:28:36.992 INFO:teuthology.orchestra.run.smithi167.stderr:ignoring --setuser ceph since I am not root 2023-10-05T19:28:36.992 INFO:teuthology.orchestra.run.smithi167.stderr:ignoring --setgroup ceph since I am not root 2023-10-05T19:28:36.993 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.986+0000 7f676640de40 1 Processor -- start 2023-10-05T19:28:36.993 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.986+0000 7f676640de40 1 -- start start 2023-10-05T19:28:36.993 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.986+0000 7f676640de40 1 --2- >> v2:172.21.15.118:3300/0 conn(0x55fba77ecd00 0x55fba77ed0d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-05T19:28:36.993 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.986+0000 7f676640de40 1 -- --> v1:172.21.15.118:6789/0 -- auth(proto 0 26 bytes epoch 0) v1 -- 0x55fba773abd0 con 0x55fba77ed610 2023-10-05T19:28:36.993 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.986+0000 7f676640de40 1 -- --> v2:172.21.15.118:3300/0 -- mon_getmap magic: 0 v1 -- 0x55fba76f7ee0 con 0x55fba77ecd00 2023-10-05T19:28:36.997 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.986+0000 7f6763f42700 1 --2- >> v2:172.21.15.118:3300/0 conn(0x55fba77ecd00 0x55fba77ed0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-05T19:28:36.997 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.986+0000 7f6763741700 1 --1- >> v1:172.21.15.118:6789/0 conn(0x55fba77ed610 0x55fba77ee0b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.118:6789/0 says I am v1:172.21.15.167:59350/0 (socket says 172.21.15.167:59350) 2023-10-05T19:28:36.997 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.986+0000 7f6763741700 1 -- 172.21.15.167:0/355376746 learned_addr learned my addr 172.21.15.167:0/355376746 (peer_addr_for_me v1:172.21.15.167:0/0) 2023-10-05T19:28:36.997 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.990+0000 7f6763f42700 1 -- 172.21.15.167:0/355376746 >> v1:172.21.15.118:6789/0 conn(0x55fba77ed610 legacy=0x55fba77ee0b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-05T19:28:36.997 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.990+0000 7f6762f40700 1 -- 172.21.15.167:0/355376746 <== mon.0 v1:172.21.15.118:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2497963486 0 0) 0x55fba773abd0 con 0x55fba77ed610 2023-10-05T19:28:36.997 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.990+0000 7f6763f42700 1 -- 172.21.15.167:0/355376746 --> v2:172.21.15.118:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55fba77ee7c0 con 0x55fba77ecd00 2023-10-05T19:28:36.997 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.990+0000 7f6763f42700 1 --2- 172.21.15.167:0/355376746 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55fba77ecd00 0x55fba77ed0d0 secure :-1 s=READY pgs=459 cs=0 l=1 rev1=1 crypto rx=0x7f675c00a6a0 tx=0x7f675c005a50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9fbe809628e09e68 server_cookie=0 in_seq=0 out_seq=0 2023-10-05T19:28:36.998 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.990+0000 7f6762f40700 1 -- 172.21.15.167:0/355376746 <== mon.0 v2:172.21.15.118:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f675c00f070 con 0x55fba77ecd00 2023-10-05T19:28:36.998 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.990+0000 7f6762f40700 1 -- 172.21.15.167:0/355376746 <== mon.0 v2:172.21.15.118:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f675c00a2e0 con 0x55fba77ecd00 2023-10-05T19:28:36.998 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.990+0000 7f6762f40700 1 -- 172.21.15.167:0/355376746 <== mon.0 v2:172.21.15.118:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f675c0085b0 con 0x55fba77ecd00 2023-10-05T19:28:36.998 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.990+0000 7f676640de40 1 -- 172.21.15.167:0/355376746 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55fba77ecd00 msgr2=0x55fba77ed0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-05T19:28:36.998 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.990+0000 7f676640de40 1 --2- 172.21.15.167:0/355376746 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55fba77ecd00 0x55fba77ed0d0 secure :-1 s=READY pgs=459 cs=0 l=1 rev1=1 crypto rx=0x7f675c00a6a0 tx=0x7f675c005a50 comp rx=0 tx=0).stop 2023-10-05T19:28:36.998 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.990+0000 7f676640de40 1 -- 172.21.15.167:0/355376746 shutdown_connections 2023-10-05T19:28:36.998 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.990+0000 7f676640de40 1 --2- 172.21.15.167:0/355376746 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55fba77ecd00 0x55fba77ed0d0 unknown :-1 s=CLOSED pgs=459 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-05T19:28:36.998 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.990+0000 7f676640de40 1 -- 172.21.15.167:0/355376746 >> 172.21.15.167:0/355376746 conn(0x55fba77e8720 msgr2=0x55fba77eab30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-05T19:28:36.998 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.990+0000 7f676640de40 1 -- 172.21.15.167:0/355376746 shutdown_connections 2023-10-05T19:28:36.998 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.990+0000 7f676640de40 1 -- 172.21.15.167:0/355376746 wait complete. 2023-10-05T19:28:36.999 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.990+0000 7f676640de40 1 Processor -- start 2023-10-05T19:28:36.999 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.990+0000 7f676640de40 1 -- start start 2023-10-05T19:28:36.999 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.990+0000 7f676640de40 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55fba77ecd00 0x55fba780efb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-05T19:28:36.999 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.990+0000 7f676640de40 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x55fba76bbe30 con 0x55fba77ecd00 2023-10-05T19:28:37.001 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.990+0000 7f6763f42700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55fba77ecd00 0x55fba780efb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-05T19:28:37.001 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.990+0000 7f6763f42700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55fba77ecd00 0x55fba780efb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.118:3300/0 says I am v2:172.21.15.167:41922/0 (socket says 172.21.15.167:41922) 2023-10-05T19:28:37.001 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.990+0000 7f6763f42700 1 -- 172.21.15.167:0/3949401317 learned_addr learned my addr 172.21.15.167:0/3949401317 (peer_addr_for_me v2:172.21.15.167:0/0) 2023-10-05T19:28:37.001 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.990+0000 7f6763f42700 1 -- 172.21.15.167:0/3949401317 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f675c005cf0 con 0x55fba77ecd00 2023-10-05T19:28:37.002 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.994+0000 7f6763f42700 1 --2- 172.21.15.167:0/3949401317 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55fba77ecd00 0x55fba780efb0 secure :-1 s=READY pgs=460 cs=0 l=1 rev1=1 crypto rx=0x7f675c005cc0 tx=0x7f675c018000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c4edd663a1783dc0 server_cookie=0 in_seq=0 out_seq=0 2023-10-05T19:28:37.002 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.994+0000 7f6761f3e700 1 -- 172.21.15.167:0/3949401317 <== mon.0 v2:172.21.15.118:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f675c007560 con 0x55fba77ecd00 2023-10-05T19:28:37.002 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.994+0000 7f6761f3e700 1 -- 172.21.15.167:0/3949401317 <== mon.0 v2:172.21.15.118:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f675c0076c0 con 0x55fba77ecd00 2023-10-05T19:28:37.002 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.994+0000 7f6761f3e700 1 -- 172.21.15.167:0/3949401317 <== mon.0 v2:172.21.15.118:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f675c0079b0 con 0x55fba77ecd00 2023-10-05T19:28:37.002 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.994+0000 7f676640de40 1 -- 172.21.15.167:0/3949401317 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55fba77ecd00 msgr2=0x55fba780efb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-05T19:28:37.002 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.994+0000 7f676640de40 1 --2- 172.21.15.167:0/3949401317 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55fba77ecd00 0x55fba780efb0 secure :-1 s=READY pgs=460 cs=0 l=1 rev1=1 crypto rx=0x7f675c005cc0 tx=0x7f675c018000 comp rx=0 tx=0).stop 2023-10-05T19:28:37.002 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.994+0000 7f676640de40 1 -- 172.21.15.167:0/3949401317 shutdown_connections 2023-10-05T19:28:37.002 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.994+0000 7f676640de40 1 --2- 172.21.15.167:0/3949401317 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55fba77ecd00 0x55fba780efb0 unknown :-1 s=CLOSED pgs=460 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-05T19:28:37.002 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.994+0000 7f676640de40 1 -- 172.21.15.167:0/3949401317 >> 172.21.15.167:0/3949401317 conn(0x55fba77e8720 msgr2=0x55fba7806760 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-05T19:28:37.002 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.994+0000 7f676640de40 1 -- 172.21.15.167:0/3949401317 shutdown_connections 2023-10-05T19:28:37.002 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.994+0000 7f676640de40 1 -- 172.21.15.167:0/3949401317 wait complete. 2023-10-05T19:28:37.003 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.994+0000 7f676640de40 1 Processor -- start 2023-10-05T19:28:37.003 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.994+0000 7f676640de40 1 -- start start 2023-10-05T19:28:37.003 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.994+0000 7f676640de40 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55fba77ecd00 0x55fba7815cb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-05T19:28:37.003 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.994+0000 7f676640de40 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x55fba768eed0 con 0x55fba77ecd00 2023-10-05T19:28:37.003 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.994+0000 7f6763f42700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55fba77ecd00 0x55fba7815cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-05T19:28:37.003 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.994+0000 7f6763f42700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55fba77ecd00 0x55fba7815cb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.118:3300/0 says I am v2:172.21.15.167:41938/0 (socket says 172.21.15.167:41938) 2023-10-05T19:28:37.003 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.994+0000 7f6763f42700 1 -- 172.21.15.167:0/639123382 learned_addr learned my addr 172.21.15.167:0/639123382 (peer_addr_for_me v2:172.21.15.167:0/0) 2023-10-05T19:28:37.003 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.994+0000 7f6763f42700 1 -- 172.21.15.167:0/639123382 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55fba77ee7c0 con 0x55fba77ecd00 2023-10-05T19:28:37.006 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.994+0000 7f6763f42700 1 --2- 172.21.15.167:0/639123382 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55fba77ecd00 0x55fba7815cb0 secure :-1 s=READY pgs=461 cs=0 l=1 rev1=1 crypto rx=0x7f675c003070 tx=0x7f675c01b000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-05T19:28:37.007 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.994+0000 7f675bfff700 1 -- 172.21.15.167:0/639123382 <== mon.0 v2:172.21.15.118:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f675c01c710 con 0x55fba77ecd00 2023-10-05T19:28:37.007 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.994+0000 7f676640de40 1 -- 172.21.15.167:0/639123382 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x55fba78161f0 con 0x55fba77ecd00 2023-10-05T19:28:37.007 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.994+0000 7f675bfff700 1 -- 172.21.15.167:0/639123382 <== mon.0 v2:172.21.15.118:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f675c01c870 con 0x55fba77ecd00 2023-10-05T19:28:37.007 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.998+0000 7f675bfff700 1 -- 172.21.15.167:0/639123382 <== mon.0 v2:172.21.15.118:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f675c01c9e0 con 0x55fba77ecd00 2023-10-05T19:28:37.007 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.998+0000 7f676640de40 1 -- 172.21.15.167:0/639123382 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55fba7816690 con 0x55fba77ecd00 2023-10-05T19:28:37.007 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.998+0000 7f675bfff700 1 -- 172.21.15.167:0/639123382 <== mon.0 v2:172.21.15.118:3300/0 4 ==== mgrmap(e 5) v1 ==== 40329+0+0 (secure 0 0 0) 0x7f675c023050 con 0x55fba77ecd00 2023-10-05T19:28:37.007 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.998+0000 7f675bfff700 1 --2- 172.21.15.167:0/639123382 >> [v2:172.21.15.167:6824/29889,v1:172.21.15.167:6825/29889] conn(0x7f6748030c00 0x7f67480330b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-05T19:28:37.007 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.998+0000 7f675bfff700 1 -- 172.21.15.167:0/639123382 <== mon.0 v2:172.21.15.118:3300/0 5 ==== osd_map(30..30 src has 1..30) v4 ==== 6057+0+0 (secure 0 0 0) 0x7f675c0450b0 con 0x55fba77ecd00 2023-10-05T19:28:37.007 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.998+0000 7f6763741700 1 --2- 172.21.15.167:0/639123382 >> [v2:172.21.15.167:6824/29889,v1:172.21.15.167:6825/29889] conn(0x7f6748030c00 0x7f67480330b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-05T19:28:37.008 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.998+0000 7f676640de40 20 rados->read ofs=0 len=0 2023-10-05T19:28:37.008 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.998+0000 7f676640de40 1 --2- 172.21.15.167:0/639123382 >> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] conn(0x55fba77ed610 0x55fba7821630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-05T19:28:37.008 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.998+0000 7f676640de40 1 -- 172.21.15.167:0/639123382 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:1 5.12 5:49953fa1:::default.realm:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x55fba7821d60 con 0x55fba77ed610 2023-10-05T19:28:37.008 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.998+0000 7f6764743700 1 --2- 172.21.15.167:0/639123382 >> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] conn(0x55fba77ed610 0x55fba7821630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-05T19:28:37.008 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.998+0000 7f6763741700 1 --2- 172.21.15.167:0/639123382 >> [v2:172.21.15.167:6824/29889,v1:172.21.15.167:6825/29889] conn(0x7f6748030c00 0x7f67480330b0 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7f674c001560 tx=0x7f674c006650 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-05T19:28:37.008 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.998+0000 7f6764743700 1 --2- 172.21.15.167:0/639123382 >> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] conn(0x55fba77ed610 0x55fba7821630 crc :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-05T19:28:37.008 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.998+0000 7f6764743700 1 -- 172.21.15.167:0/639123382 <== osd.1 v2:172.21.15.167:6800/30035 1 ==== osd_op_reply(1 default.realm [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 ==== 157+0+0 (crc 0 0 0) 0x7f675400a500 con 0x55fba77ed610 2023-10-05T19:28:37.008 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.998+0000 7f676640de40 20 rados_obj.operate() r=-2 bl.length=0 2023-10-05T19:28:37.008 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.998+0000 7f676640de40 20 realm 2023-10-05T19:28:37.008 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.998+0000 7f676640de40 20 rados->read ofs=0 len=0 2023-10-05T19:28:37.008 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:36.998+0000 7f676640de40 1 -- 172.21.15.167:0/639123382 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:2 5.12 5:49953fa1:::default.realm:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x55fba781ebd0 con 0x55fba77ed610 2023-10-05T19:28:37.009 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.002+0000 7f6764743700 1 -- 172.21.15.167:0/639123382 <== osd.1 v2:172.21.15.167:6800/30035 2 ==== osd_op_reply(2 default.realm [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 ==== 157+0+0 (crc 0 0 0) 0x7f675400a500 con 0x55fba77ed610 2023-10-05T19:28:37.009 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.002+0000 7f676640de40 20 rados_obj.operate() r=-2 bl.length=0 2023-10-05T19:28:37.009 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.002+0000 7f676640de40 4 RGWPeriod::init failed to init realm id : (2) No such file or directory 2023-10-05T19:28:37.009 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.002+0000 7f676640de40 20 rados->read ofs=0 len=0 2023-10-05T19:28:37.009 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.002+0000 7f676640de40 1 -- 172.21.15.167:0/639123382 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:3 5.12 5:49953fa1:::default.realm:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x55fba7822840 con 0x55fba77ed610 2023-10-05T19:28:37.009 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.002+0000 7f6764743700 1 -- 172.21.15.167:0/639123382 <== osd.1 v2:172.21.15.167:6800/30035 3 ==== osd_op_reply(3 default.realm [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 ==== 157+0+0 (crc 0 0 0) 0x7f675400a500 con 0x55fba77ed610 2023-10-05T19:28:37.009 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.002+0000 7f676640de40 20 rados_obj.operate() r=-2 bl.length=0 2023-10-05T19:28:37.009 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.002+0000 7f676640de40 20 rados->read ofs=0 len=0 2023-10-05T19:28:37.009 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.002+0000 7f676640de40 1 -- 172.21.15.167:0/639123382 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:4 5.f 5:f43b6ece:::zone_names.default:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x55fba7822450 con 0x55fba77ed610 2023-10-05T19:28:37.011 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.002+0000 7f6764743700 1 -- 172.21.15.167:0/639123382 <== osd.1 v2:172.21.15.167:6800/30035 4 ==== osd_op_reply(4 zone_names.default [read 0~46 out=46b] v0'0 uv1 ondisk = 0) v8 ==== 162+0+46 (crc 0 0 0) 0x7f675400a500 con 0x55fba77ed610 2023-10-05T19:28:37.011 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.002+0000 7f676640de40 20 rados_obj.operate() r=0 bl.length=46 2023-10-05T19:28:37.011 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.002+0000 7f676640de40 20 rados->read ofs=0 len=0 2023-10-05T19:28:37.011 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.002+0000 7f676640de40 1 -- 172.21.15.167:0/639123382 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:5 5.1d 5:b8374eb4:::zone_info.9a9df965-6612-45d4-a45f-6e3c889bacc2:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x55fba78236d0 con 0x55fba77ed610 2023-10-05T19:28:37.011 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.002+0000 7f6764743700 1 -- 172.21.15.167:0/639123382 <== osd.1 v2:172.21.15.167:6800/30035 5 ==== osd_op_reply(5 zone_info.9a9df965-6612-45d4-a45f-6e3c889bacc2 [read 0~874 out=874b] v0'0 uv2 ondisk = 0) v8 ==== 190+0+874 (crc 0 0 0) 0x7f675400a500 con 0x55fba77ed610 2023-10-05T19:28:37.011 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.002+0000 7f676640de40 20 rados_obj.operate() r=0 bl.length=874 2023-10-05T19:28:37.011 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.002+0000 7f676640de40 20 searching for the correct realm 2023-10-05T19:28:37.012 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.002+0000 7f676640de40 1 --2- 172.21.15.167:0/639123382 >> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] conn(0x55fba7825490 0x55fba7827870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-05T19:28:37.012 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.002+0000 7f676640de40 1 -- 172.21.15.167:0/639123382 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:6 5.0 5:00000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x55fba7827db0 con 0x55fba7825490 2023-10-05T19:28:37.012 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.002+0000 7f6763f42700 1 --2- 172.21.15.167:0/639123382 >> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] conn(0x55fba7825490 0x55fba7827870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-05T19:28:37.012 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.002+0000 7f6763f42700 1 --2- 172.21.15.167:0/639123382 >> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] conn(0x55fba7825490 0x55fba7827870 crc :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-05T19:28:37.012 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.006+0000 7f6763f42700 1 -- 172.21.15.167:0/639123382 <== osd.3 v2:172.21.15.167:6808/30036 1 ==== osd_op_reply(6 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f675c02e590 con 0x55fba7825490 2023-10-05T19:28:37.012 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.006+0000 7f6763f42700 1 --2- 172.21.15.167:0/639123382 >> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] conn(0x7f675c007070 0x7f675c043890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-05T19:28:37.012 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.006+0000 7f6763f42700 1 -- 172.21.15.167:0/639123382 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:7 5.10 5:08000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x55fba7827db0 con 0x7f675c007070 2023-10-05T19:28:37.012 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.006+0000 7f6763741700 1 --2- 172.21.15.167:0/639123382 >> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] conn(0x7f675c007070 0x7f675c043890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-05T19:28:37.013 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.006+0000 7f6763741700 1 --2- 172.21.15.167:0/639123382 >> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] conn(0x7f675c007070 0x7f675c043890 crc :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-05T19:28:37.013 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.006+0000 7f6763741700 1 -- 172.21.15.167:0/639123382 <== osd.2 v2:172.21.15.167:6816/30037 1 ==== osd_op_reply(7 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f674c004520 con 0x7f675c007070 2023-10-05T19:28:37.013 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.006+0000 7f6763741700 1 --2- 172.21.15.167:0/639123382 >> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] conn(0x7f674c011040 0x7f674c014950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-05T19:28:37.013 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.006+0000 7f6763741700 1 -- 172.21.15.167:0/639123382 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:8 5.8 5:10000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x55fba7827db0 con 0x7f674c011040 2023-10-05T19:28:37.013 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.006+0000 7f6764743700 1 --2- 172.21.15.167:0/639123382 >> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] conn(0x7f674c011040 0x7f674c014950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-05T19:28:37.014 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.006+0000 7f6764743700 1 --2- 172.21.15.167:0/639123382 >> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] conn(0x7f674c011040 0x7f674c014950 crc :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-05T19:28:37.014 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.010+0000 7f6764743700 1 -- 172.21.15.167:0/639123382 <== osd.0 v2:172.21.15.118:6800/27690 1 ==== osd_op_reply(8 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f6754014040 con 0x7f674c011040 2023-10-05T19:28:37.014 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.010+0000 7f6764743700 1 -- 172.21.15.167:0/639123382 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:9 5.18 5:18000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x55fba7827db0 con 0x55fba77ed610 2023-10-05T19:28:37.015 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.010+0000 7f6764743700 1 -- 172.21.15.167:0/639123382 <== osd.1 v2:172.21.15.167:6800/30035 6 ==== osd_op_reply(9 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f6754014040 con 0x55fba77ed610 2023-10-05T19:28:37.015 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.010+0000 7f6764743700 1 -- 172.21.15.167:0/639123382 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:10 5.4 5:20000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x55fba7827db0 con 0x55fba77ed610 2023-10-05T19:28:37.015 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.010+0000 7f6764743700 1 -- 172.21.15.167:0/639123382 <== osd.1 v2:172.21.15.167:6800/30035 7 ==== osd_op_reply(10 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f6754014040 con 0x55fba77ed610 2023-10-05T19:28:37.016 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.010+0000 7f6764743700 1 -- 172.21.15.167:0/639123382 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:11 5.14 5:28000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x55fba7827db0 con 0x7f674c011040 2023-10-05T19:28:37.016 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.010+0000 7f6764743700 1 -- 172.21.15.167:0/639123382 <== osd.0 v2:172.21.15.118:6800/27690 2 ==== osd_op_reply(11 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f6754014040 con 0x7f674c011040 2023-10-05T19:28:37.016 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.010+0000 7f6764743700 1 -- 172.21.15.167:0/639123382 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:12 5.c 5:30000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x55fba7827db0 con 0x7f674c011040 2023-10-05T19:28:37.017 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.010+0000 7f6764743700 1 -- 172.21.15.167:0/639123382 <== osd.0 v2:172.21.15.118:6800/27690 3 ==== osd_op_reply(12 [pgnls start_epoch 30 out=112b] v23'1 uv1 ondisk = 1) v8 ==== 144+0+112 (crc 0 0 0) 0x7f6754014040 con 0x7f674c011040 2023-10-05T19:28:37.017 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.010+0000 7f6764743700 1 -- 172.21.15.167:0/639123382 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:13 5.1c 5:38000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x55fba7827db0 con 0x7f675c007070 2023-10-05T19:28:37.018 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.014+0000 7f6763741700 1 -- 172.21.15.167:0/639123382 <== osd.2 v2:172.21.15.167:6816/30037 2 ==== osd_op_reply(13 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f674c004520 con 0x7f675c007070 2023-10-05T19:28:37.018 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.014+0000 7f6763741700 1 -- 172.21.15.167:0/639123382 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:14 5.2 5:40000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x55fba7827db0 con 0x55fba7825490 2023-10-05T19:28:37.019 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.014+0000 7f6763f42700 1 -- 172.21.15.167:0/639123382 <== osd.3 v2:172.21.15.167:6808/30036 2 ==== osd_op_reply(14 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f675c02e590 con 0x55fba7825490 2023-10-05T19:28:37.019 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.014+0000 7f6763f42700 1 -- 172.21.15.167:0/639123382 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:15 5.12 5:48000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x55fba7827db0 con 0x55fba77ed610 2023-10-05T19:28:37.019 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.014+0000 7f6764743700 1 -- 172.21.15.167:0/639123382 <== osd.1 v2:172.21.15.167:6800/30035 8 ==== osd_op_reply(15 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f6754014040 con 0x55fba77ed610 2023-10-05T19:28:37.020 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.014+0000 7f6764743700 1 -- 172.21.15.167:0/639123382 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:16 5.a 5:50000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x55fba7827db0 con 0x7f674c011040 2023-10-05T19:28:37.020 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.014+0000 7f6764743700 1 -- 172.21.15.167:0/639123382 <== osd.0 v2:172.21.15.118:6800/27690 4 ==== osd_op_reply(16 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f6754022040 con 0x7f674c011040 2023-10-05T19:28:37.021 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.014+0000 7f6764743700 1 -- 172.21.15.167:0/639123382 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:17 5.1a 5:58000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x55fba7827db0 con 0x55fba77ed610 2023-10-05T19:28:37.021 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.014+0000 7f6764743700 1 -- 172.21.15.167:0/639123382 <== osd.1 v2:172.21.15.167:6800/30035 9 ==== osd_op_reply(17 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f6754022040 con 0x55fba77ed610 2023-10-05T19:28:37.021 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.014+0000 7f6764743700 1 -- 172.21.15.167:0/639123382 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:18 5.6 5:60000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x55fba7827db0 con 0x7f675c007070 2023-10-05T19:28:37.022 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.014+0000 7f6763741700 1 -- 172.21.15.167:0/639123382 <== osd.2 v2:172.21.15.167:6816/30037 3 ==== osd_op_reply(18 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f674c004520 con 0x7f675c007070 2023-10-05T19:28:37.022 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.014+0000 7f6763741700 1 -- 172.21.15.167:0/639123382 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:19 5.16 5:68000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x55fba7827db0 con 0x55fba7825490 2023-10-05T19:28:37.022 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.018+0000 7f6763f42700 1 -- 172.21.15.167:0/639123382 <== osd.3 v2:172.21.15.167:6808/30036 3 ==== osd_op_reply(19 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f675c02e590 con 0x55fba7825490 2023-10-05T19:28:37.022 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.018+0000 7f6763f42700 1 -- 172.21.15.167:0/639123382 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:20 5.e 5:70000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x55fba7827db0 con 0x55fba7825490 2023-10-05T19:28:37.023 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.018+0000 7f6763f42700 1 -- 172.21.15.167:0/639123382 <== osd.3 v2:172.21.15.167:6808/30036 4 ==== osd_op_reply(20 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f675c02e590 con 0x55fba7825490 2023-10-05T19:28:37.023 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.018+0000 7f6763f42700 1 -- 172.21.15.167:0/639123382 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:21 5.1e 5:78000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x55fba7827db0 con 0x7f674c011040 2023-10-05T19:28:37.024 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.018+0000 7f6764743700 1 -- 172.21.15.167:0/639123382 <== osd.0 v2:172.21.15.118:6800/27690 5 ==== osd_op_reply(21 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f6754022040 con 0x7f674c011040 2023-10-05T19:28:37.025 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.018+0000 7f6764743700 1 -- 172.21.15.167:0/639123382 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:22 5.1 5:80000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x55fba7827db0 con 0x55fba7825490 2023-10-05T19:28:37.025 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.018+0000 7f6763f42700 1 -- 172.21.15.167:0/639123382 <== osd.3 v2:172.21.15.167:6808/30036 5 ==== osd_op_reply(22 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f675c02e590 con 0x55fba7825490 2023-10-05T19:28:37.025 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.018+0000 7f6763f42700 1 -- 172.21.15.167:0/639123382 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:23 5.11 5:88000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x55fba7827db0 con 0x55fba7825490 2023-10-05T19:28:37.026 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.022+0000 7f6763f42700 1 -- 172.21.15.167:0/639123382 <== osd.3 v2:172.21.15.167:6808/30036 6 ==== osd_op_reply(23 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f675c02e590 con 0x55fba7825490 2023-10-05T19:28:37.026 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.022+0000 7f6763f42700 1 -- 172.21.15.167:0/639123382 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:24 5.9 5:90000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x55fba7827db0 con 0x55fba77ed610 2023-10-05T19:28:37.027 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.022+0000 7f6764743700 1 -- 172.21.15.167:0/639123382 <== osd.1 v2:172.21.15.167:6800/30035 10 ==== osd_op_reply(24 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f6754022040 con 0x55fba77ed610 2023-10-05T19:28:37.027 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.022+0000 7f6764743700 1 -- 172.21.15.167:0/639123382 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:25 5.19 5:98000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x55fba7827db0 con 0x55fba77ed610 2023-10-05T19:28:37.027 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.022+0000 7f6764743700 1 -- 172.21.15.167:0/639123382 <== osd.1 v2:172.21.15.167:6800/30035 11 ==== osd_op_reply(25 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f6754022040 con 0x55fba77ed610 2023-10-05T19:28:37.028 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.022+0000 7f6764743700 1 -- 172.21.15.167:0/639123382 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:26 5.5 5:a0000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x55fba7827db0 con 0x7f674c011040 2023-10-05T19:28:37.028 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.022+0000 7f6764743700 1 -- 172.21.15.167:0/639123382 <== osd.0 v2:172.21.15.118:6800/27690 6 ==== osd_op_reply(26 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f6754022040 con 0x7f674c011040 2023-10-05T19:28:37.028 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.022+0000 7f6764743700 1 -- 172.21.15.167:0/639123382 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:27 5.15 5:a8000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x55fba7827db0 con 0x55fba7825490 2023-10-05T19:28:37.029 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.022+0000 7f6763f42700 1 -- 172.21.15.167:0/639123382 <== osd.3 v2:172.21.15.167:6808/30036 7 ==== osd_op_reply(27 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f675c02e590 con 0x55fba7825490 2023-10-05T19:28:37.029 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.022+0000 7f6763f42700 1 -- 172.21.15.167:0/639123382 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:28 5.d 5:b0000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x55fba7827db0 con 0x7f675c007070 2023-10-05T19:28:37.030 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.026+0000 7f6763741700 1 -- 172.21.15.167:0/639123382 <== osd.2 v2:172.21.15.167:6816/30037 4 ==== osd_op_reply(28 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f674c004520 con 0x7f675c007070 2023-10-05T19:28:37.030 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.026+0000 7f6763741700 1 -- 172.21.15.167:0/639123382 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:29 5.1d 5:b8000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x55fba7827db0 con 0x55fba77ed610 2023-10-05T19:28:37.031 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.026+0000 7f6764743700 1 -- 172.21.15.167:0/639123382 <== osd.1 v2:172.21.15.167:6800/30035 12 ==== osd_op_reply(29 [pgnls start_epoch 30 out=107b] v30'2 uv2 ondisk = 1) v8 ==== 144+0+107 (crc 0 0 0) 0x7f6754022040 con 0x55fba77ed610 2023-10-05T19:28:37.031 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.026+0000 7f6764743700 1 -- 172.21.15.167:0/639123382 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:30 5.3 5:c0000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x55fba7827db0 con 0x7f674c011040 2023-10-05T19:28:37.032 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.026+0000 7f6764743700 1 -- 172.21.15.167:0/639123382 <== osd.0 v2:172.21.15.118:6800/27690 7 ==== osd_op_reply(30 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f6754022040 con 0x7f674c011040 2023-10-05T19:28:37.032 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.026+0000 7f6764743700 1 -- 172.21.15.167:0/639123382 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:31 5.13 5:c8000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x55fba7827db0 con 0x55fba7825490 2023-10-05T19:28:37.033 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.026+0000 7f6763f42700 1 -- 172.21.15.167:0/639123382 <== osd.3 v2:172.21.15.167:6808/30036 8 ==== osd_op_reply(31 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f675c02e590 con 0x55fba7825490 2023-10-05T19:28:37.033 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.026+0000 7f6763f42700 1 -- 172.21.15.167:0/639123382 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:32 5.b 5:d0000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x55fba7827db0 con 0x7f674c011040 2023-10-05T19:28:37.034 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.030+0000 7f6764743700 1 -- 172.21.15.167:0/639123382 <== osd.0 v2:172.21.15.118:6800/27690 8 ==== osd_op_reply(32 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f6754022040 con 0x7f674c011040 2023-10-05T19:28:37.034 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.030+0000 7f6764743700 1 -- 172.21.15.167:0/639123382 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:33 5.1b 5:d8000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x55fba7827db0 con 0x7f675c007070 2023-10-05T19:28:37.035 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.030+0000 7f6763741700 1 -- 172.21.15.167:0/639123382 <== osd.2 v2:172.21.15.167:6816/30037 5 ==== osd_op_reply(33 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f674c004520 con 0x7f675c007070 2023-10-05T19:28:37.035 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.030+0000 7f6763741700 1 -- 172.21.15.167:0/639123382 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:34 5.7 5:e0000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x55fba7827db0 con 0x55fba7825490 2023-10-05T19:28:37.036 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.030+0000 7f6763f42700 1 -- 172.21.15.167:0/639123382 <== osd.3 v2:172.21.15.167:6808/30036 9 ==== osd_op_reply(34 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f675c02e590 con 0x55fba7825490 2023-10-05T19:28:37.036 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.030+0000 7f6763f42700 1 -- 172.21.15.167:0/639123382 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:35 5.17 5:e8000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x55fba7827db0 con 0x7f674c011040 2023-10-05T19:28:37.037 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.030+0000 7f6764743700 1 -- 172.21.15.167:0/639123382 <== osd.0 v2:172.21.15.118:6800/27690 9 ==== osd_op_reply(35 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f6754022040 con 0x7f674c011040 2023-10-05T19:28:37.037 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.030+0000 7f6764743700 1 -- 172.21.15.167:0/639123382 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:36 5.f 5:f0000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x55fba7827db0 con 0x55fba77ed610 2023-10-05T19:28:37.038 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.030+0000 7f6764743700 1 -- 172.21.15.167:0/639123382 <== osd.1 v2:172.21.15.167:6800/30035 13 ==== osd_op_reply(36 [pgnls start_epoch 30 out=115b] v23'2 uv2 ondisk = 1) v8 ==== 144+0+115 (crc 0 0 0) 0x7f675402f080 con 0x55fba77ed610 2023-10-05T19:28:37.038 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.030+0000 7f6764743700 1 -- 172.21.15.167:0/639123382 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:37 5.1f 5:f8000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x55fba7827db0 con 0x7f675c007070 2023-10-05T19:28:37.039 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.034+0000 7f6763741700 1 -- 172.21.15.167:0/639123382 <== osd.2 v2:172.21.15.167:6816/30037 6 ==== osd_op_reply(37 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f674c004520 con 0x7f675c007070 2023-10-05T19:28:37.039 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.034+0000 7f676640de40 20 RGWRados::pool_iterate: got zonegroup_info.4f1f719d-9648-417b-b10c-e5d939825419 2023-10-05T19:28:37.039 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.034+0000 7f676640de40 20 RGWRados::pool_iterate: got zone_info.9a9df965-6612-45d4-a45f-6e3c889bacc2 2023-10-05T19:28:37.039 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.034+0000 7f676640de40 20 RGWRados::pool_iterate: got zone_names.default 2023-10-05T19:28:37.039 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.034+0000 7f676640de40 20 RGWRados::pool_iterate: got zonegroups_names.default 2023-10-05T19:28:37.039 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.034+0000 7f676640de40 20 rados->read ofs=0 len=0 2023-10-05T19:28:37.039 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.034+0000 7f676640de40 1 -- 172.21.15.167:0/639123382 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:38 5.12 5:49953fa1:::default.realm:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x55fba7824240 con 0x55fba77ed610 2023-10-05T19:28:37.040 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.034+0000 7f6764743700 1 -- 172.21.15.167:0/639123382 <== osd.1 v2:172.21.15.167:6800/30035 14 ==== osd_op_reply(38 default.realm [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 ==== 157+0+0 (crc 0 0 0) 0x7f675402f080 con 0x55fba77ed610 2023-10-05T19:28:37.040 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.034+0000 7f676640de40 20 rados_obj.operate() r=-2 bl.length=0 2023-10-05T19:28:37.041 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.034+0000 7f676640de40 20 rados->read ofs=0 len=0 2023-10-05T19:28:37.041 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.034+0000 7f676640de40 1 -- 172.21.15.167:0/639123382 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:39 5.f 5:f4c53578:::zonegroups_names.default:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x55fba7824b30 con 0x55fba77ed610 2023-10-05T19:28:37.041 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.034+0000 7f6764743700 1 -- 172.21.15.167:0/639123382 <== osd.1 v2:172.21.15.167:6800/30035 15 ==== osd_op_reply(39 zonegroups_names.default [read 0~46 out=46b] v0'0 uv2 ondisk = 0) v8 ==== 168+0+46 (crc 0 0 0) 0x7f675402f080 con 0x55fba77ed610 2023-10-05T19:28:37.041 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.034+0000 7f676640de40 20 rados_obj.operate() r=0 bl.length=46 2023-10-05T19:28:37.041 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.034+0000 7f676640de40 20 rados->read ofs=0 len=0 2023-10-05T19:28:37.041 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.034+0000 7f676640de40 1 -- 172.21.15.167:0/639123382 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:40 5.c 5:30d299a5:::zonegroup_info.4f1f719d-9648-417b-b10c-e5d939825419:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x55fba78281f0 con 0x7f674c011040 2023-10-05T19:28:37.043 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.034+0000 7f6764743700 1 -- 172.21.15.167:0/639123382 <== osd.0 v2:172.21.15.118:6800/27690 10 ==== osd_op_reply(40 zonegroup_info.4f1f719d-9648-417b-b10c-e5d939825419 [read 0~362 out=362b] v0'0 uv1 ondisk = 0) v8 ==== 195+0+362 (crc 0 0 0) 0x7f675402f080 con 0x7f674c011040 2023-10-05T19:28:37.043 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.034+0000 7f676640de40 20 rados_obj.operate() r=0 bl.length=362 2023-10-05T19:28:37.043 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.034+0000 7f676640de40 20 rados->read ofs=0 len=0 2023-10-05T19:28:37.043 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.034+0000 7f676640de40 1 -- 172.21.15.167:0/639123382 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:41 5.13 5:cbbf6c60:::converted:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x55fba78281f0 con 0x55fba7825490 2023-10-05T19:28:37.043 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.034+0000 7f6763f42700 1 -- 172.21.15.167:0/639123382 <== osd.3 v2:172.21.15.167:6808/30036 10 ==== osd_op_reply(41 converted [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 ==== 153+0+0 (crc 0 0 0) 0x7f675c02e590 con 0x55fba7825490 2023-10-05T19:28:37.043 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.038+0000 7f676640de40 20 rados_obj.operate() r=-2 bl.length=0 2023-10-05T19:28:37.043 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.038+0000 7f676640de40 20 rados->read ofs=0 len=0 2023-10-05T19:28:37.043 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.038+0000 7f676640de40 1 -- 172.21.15.167:0/639123382 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:42 5.12 5:49953fa1:::default.realm:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x55fba78285a0 con 0x55fba77ed610 2023-10-05T19:28:37.043 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.038+0000 7f6764743700 1 -- 172.21.15.167:0/639123382 <== osd.1 v2:172.21.15.167:6800/30035 16 ==== osd_op_reply(42 default.realm [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 ==== 157+0+0 (crc 0 0 0) 0x7f675402f080 con 0x55fba77ed610 2023-10-05T19:28:37.043 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.038+0000 7f676640de40 20 rados_obj.operate() r=-2 bl.length=0 2023-10-05T19:28:37.044 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.038+0000 7f676640de40 20 rados->read ofs=0 len=0 2023-10-05T19:28:37.044 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.038+0000 7f676640de40 1 -- 172.21.15.167:0/639123382 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:43 5.f 5:f4c53578:::zonegroups_names.default:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x55fba78289d0 con 0x55fba77ed610 2023-10-05T19:28:37.044 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.038+0000 7f6764743700 1 -- 172.21.15.167:0/639123382 <== osd.1 v2:172.21.15.167:6800/30035 17 ==== osd_op_reply(43 zonegroups_names.default [read 0~46 out=46b] v0'0 uv2 ondisk = 0) v8 ==== 168+0+46 (crc 0 0 0) 0x7f675402f080 con 0x55fba77ed610 2023-10-05T19:28:37.044 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.038+0000 7f676640de40 20 rados_obj.operate() r=0 bl.length=46 2023-10-05T19:28:37.044 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.038+0000 7f676640de40 1 -- 172.21.15.167:0/639123382 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:44 5.0 5:00000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x55fba78299e0 con 0x55fba7825490 2023-10-05T19:28:37.046 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.042+0000 7f6763f42700 1 -- 172.21.15.167:0/639123382 <== osd.3 v2:172.21.15.167:6808/30036 11 ==== osd_op_reply(44 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f675c02e590 con 0x55fba7825490 2023-10-05T19:28:37.046 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.042+0000 7f6763f42700 1 -- 172.21.15.167:0/639123382 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:45 5.10 5:08000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x55fba78299e0 con 0x7f675c007070 2023-10-05T19:28:37.047 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.042+0000 7f6763741700 1 -- 172.21.15.167:0/639123382 <== osd.2 v2:172.21.15.167:6816/30037 7 ==== osd_op_reply(45 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f674c004520 con 0x7f675c007070 2023-10-05T19:28:37.047 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.042+0000 7f6763741700 1 -- 172.21.15.167:0/639123382 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:46 5.8 5:10000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x55fba78299e0 con 0x7f674c011040 2023-10-05T19:28:37.048 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.042+0000 7f6764743700 1 -- 172.21.15.167:0/639123382 <== osd.0 v2:172.21.15.118:6800/27690 11 ==== osd_op_reply(46 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f675402f080 con 0x7f674c011040 2023-10-05T19:28:37.048 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.042+0000 7f6764743700 1 -- 172.21.15.167:0/639123382 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:47 5.18 5:18000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x55fba7824b30 con 0x55fba77ed610 2023-10-05T19:28:37.049 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.042+0000 7f6764743700 1 -- 172.21.15.167:0/639123382 <== osd.1 v2:172.21.15.167:6800/30035 18 ==== osd_op_reply(47 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f675402f080 con 0x55fba77ed610 2023-10-05T19:28:37.049 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.042+0000 7f6764743700 1 -- 172.21.15.167:0/639123382 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:48 5.4 5:20000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x55fba7824b30 con 0x55fba77ed610 2023-10-05T19:28:37.050 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.042+0000 7f6764743700 1 -- 172.21.15.167:0/639123382 <== osd.1 v2:172.21.15.167:6800/30035 19 ==== osd_op_reply(48 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f675402f080 con 0x55fba77ed610 2023-10-05T19:28:37.050 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.042+0000 7f6764743700 1 -- 172.21.15.167:0/639123382 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:49 5.14 5:28000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x55fba7824b30 con 0x7f674c011040 2023-10-05T19:28:37.051 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.046+0000 7f6764743700 1 -- 172.21.15.167:0/639123382 <== osd.0 v2:172.21.15.118:6800/27690 12 ==== osd_op_reply(49 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f675402f080 con 0x7f674c011040 2023-10-05T19:28:37.051 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.046+0000 7f6764743700 1 -- 172.21.15.167:0/639123382 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:50 5.c 5:30000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x55fba7824b30 con 0x7f674c011040 2023-10-05T19:28:37.052 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.046+0000 7f6764743700 1 -- 172.21.15.167:0/639123382 <== osd.0 v2:172.21.15.118:6800/27690 13 ==== osd_op_reply(50 [pgnls start_epoch 30 out=112b] v23'1 uv1 ondisk = 1) v8 ==== 144+0+112 (crc 0 0 0) 0x7f675402f080 con 0x7f674c011040 2023-10-05T19:28:37.052 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.046+0000 7f6764743700 1 -- 172.21.15.167:0/639123382 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:51 5.1c 5:38000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x55fba7824b30 con 0x7f675c007070 2023-10-05T19:28:37.052 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.046+0000 7f6763741700 1 -- 172.21.15.167:0/639123382 <== osd.2 v2:172.21.15.167:6816/30037 8 ==== osd_op_reply(51 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f674c004520 con 0x7f675c007070 2023-10-05T19:28:37.053 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.046+0000 7f6763741700 1 -- 172.21.15.167:0/639123382 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:52 5.2 5:40000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x55fba7824b30 con 0x55fba7825490 2023-10-05T19:28:37.053 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.046+0000 7f6763f42700 1 -- 172.21.15.167:0/639123382 <== osd.3 v2:172.21.15.167:6808/30036 12 ==== osd_op_reply(52 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f675c02e590 con 0x55fba7825490 2023-10-05T19:28:37.053 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.046+0000 7f6763f42700 1 -- 172.21.15.167:0/639123382 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:53 5.12 5:48000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x55fba7824b30 con 0x55fba77ed610 2023-10-05T19:28:37.054 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.050+0000 7f6764743700 1 -- 172.21.15.167:0/639123382 <== osd.1 v2:172.21.15.167:6800/30035 20 ==== osd_op_reply(53 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f675402f080 con 0x55fba77ed610 2023-10-05T19:28:37.054 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.050+0000 7f6764743700 1 -- 172.21.15.167:0/639123382 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:54 5.a 5:50000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x55fba7824b30 con 0x7f674c011040 2023-10-05T19:28:37.055 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.050+0000 7f6764743700 1 -- 172.21.15.167:0/639123382 <== osd.0 v2:172.21.15.118:6800/27690 14 ==== osd_op_reply(54 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f675402f080 con 0x7f674c011040 2023-10-05T19:28:37.055 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.050+0000 7f6764743700 1 -- 172.21.15.167:0/639123382 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:55 5.1a 5:58000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x55fba7824b30 con 0x55fba77ed610 2023-10-05T19:28:37.056 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.050+0000 7f6764743700 1 -- 172.21.15.167:0/639123382 <== osd.1 v2:172.21.15.167:6800/30035 21 ==== osd_op_reply(55 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f675402f080 con 0x55fba77ed610 2023-10-05T19:28:37.056 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.050+0000 7f6764743700 1 -- 172.21.15.167:0/639123382 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:56 5.6 5:60000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x55fba7824b30 con 0x7f675c007070 2023-10-05T19:28:37.056 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.050+0000 7f6763741700 1 -- 172.21.15.167:0/639123382 <== osd.2 v2:172.21.15.167:6816/30037 9 ==== osd_op_reply(56 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f674c004520 con 0x7f675c007070 2023-10-05T19:28:37.057 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.050+0000 7f6763741700 1 -- 172.21.15.167:0/639123382 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:57 5.16 5:68000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x55fba7824b30 con 0x55fba7825490 2023-10-05T19:28:37.057 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.050+0000 7f6763f42700 1 -- 172.21.15.167:0/639123382 <== osd.3 v2:172.21.15.167:6808/30036 13 ==== osd_op_reply(57 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f675c02e590 con 0x55fba7825490 2023-10-05T19:28:37.057 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.050+0000 7f6763f42700 1 -- 172.21.15.167:0/639123382 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:58 5.e 5:70000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x55fba7824b30 con 0x55fba7825490 2023-10-05T19:28:37.058 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.054+0000 7f6763f42700 1 -- 172.21.15.167:0/639123382 <== osd.3 v2:172.21.15.167:6808/30036 14 ==== osd_op_reply(58 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f675c02e590 con 0x55fba7825490 2023-10-05T19:28:37.058 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.054+0000 7f6763f42700 1 -- 172.21.15.167:0/639123382 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:59 5.1e 5:78000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x55fba7824b30 con 0x7f674c011040 2023-10-05T19:28:37.059 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.054+0000 7f6764743700 1 -- 172.21.15.167:0/639123382 <== osd.0 v2:172.21.15.118:6800/27690 15 ==== osd_op_reply(59 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f675402f080 con 0x7f674c011040 2023-10-05T19:28:37.059 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.054+0000 7f6764743700 1 -- 172.21.15.167:0/639123382 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:60 5.1 5:80000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x55fba7824b30 con 0x55fba7825490 2023-10-05T19:28:37.060 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.054+0000 7f6763f42700 1 -- 172.21.15.167:0/639123382 <== osd.3 v2:172.21.15.167:6808/30036 15 ==== osd_op_reply(60 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f675c02e590 con 0x55fba7825490 2023-10-05T19:28:37.060 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.054+0000 7f6763f42700 1 -- 172.21.15.167:0/639123382 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:61 5.11 5:88000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x55fba7824b30 con 0x55fba7825490 2023-10-05T19:28:37.061 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.054+0000 7f6763f42700 1 -- 172.21.15.167:0/639123382 <== osd.3 v2:172.21.15.167:6808/30036 16 ==== osd_op_reply(61 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f675c02e590 con 0x55fba7825490 2023-10-05T19:28:37.061 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.054+0000 7f6763f42700 1 -- 172.21.15.167:0/639123382 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:62 5.9 5:90000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x55fba7824b30 con 0x55fba77ed610 2023-10-05T19:28:37.062 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.054+0000 7f6764743700 1 -- 172.21.15.167:0/639123382 <== osd.1 v2:172.21.15.167:6800/30035 22 ==== osd_op_reply(62 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f675402f080 con 0x55fba77ed610 2023-10-05T19:28:37.062 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.054+0000 7f6764743700 1 -- 172.21.15.167:0/639123382 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:63 5.19 5:98000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x55fba7824b30 con 0x55fba77ed610 2023-10-05T19:28:37.063 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.058+0000 7f6764743700 1 -- 172.21.15.167:0/639123382 <== osd.1 v2:172.21.15.167:6800/30035 23 ==== osd_op_reply(63 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f675402f080 con 0x55fba77ed610 2023-10-05T19:28:37.063 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.058+0000 7f6764743700 1 -- 172.21.15.167:0/639123382 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:64 5.5 5:a0000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x55fba7824b30 con 0x7f674c011040 2023-10-05T19:28:37.063 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.058+0000 7f6764743700 1 -- 172.21.15.167:0/639123382 <== osd.0 v2:172.21.15.118:6800/27690 16 ==== osd_op_reply(64 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f675402f080 con 0x7f674c011040 2023-10-05T19:28:37.064 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.058+0000 7f6764743700 1 -- 172.21.15.167:0/639123382 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:65 5.15 5:a8000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x55fba7824b30 con 0x55fba7825490 2023-10-05T19:28:37.064 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.058+0000 7f6763f42700 1 -- 172.21.15.167:0/639123382 <== osd.3 v2:172.21.15.167:6808/30036 17 ==== osd_op_reply(65 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f675c02e590 con 0x55fba7825490 2023-10-05T19:28:37.065 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.058+0000 7f6763f42700 1 -- 172.21.15.167:0/639123382 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:66 5.d 5:b0000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x55fba7824b30 con 0x7f675c007070 2023-10-05T19:28:37.066 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.058+0000 7f6763741700 1 -- 172.21.15.167:0/639123382 <== osd.2 v2:172.21.15.167:6816/30037 10 ==== osd_op_reply(66 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f674c004520 con 0x7f675c007070 2023-10-05T19:28:37.066 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.058+0000 7f6763741700 1 -- 172.21.15.167:0/639123382 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:67 5.1d 5:b8000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x55fba7824b30 con 0x55fba77ed610 2023-10-05T19:28:37.066 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.062+0000 7f6764743700 1 -- 172.21.15.167:0/639123382 <== osd.1 v2:172.21.15.167:6800/30035 24 ==== osd_op_reply(67 [pgnls start_epoch 30 out=107b] v30'2 uv2 ondisk = 1) v8 ==== 144+0+107 (crc 0 0 0) 0x7f675402f080 con 0x55fba77ed610 2023-10-05T19:28:37.067 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.062+0000 7f6764743700 1 -- 172.21.15.167:0/639123382 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:68 5.3 5:c0000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x55fba7824b30 con 0x7f674c011040 2023-10-05T19:28:37.068 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.062+0000 7f6764743700 1 -- 172.21.15.167:0/639123382 <== osd.0 v2:172.21.15.118:6800/27690 17 ==== osd_op_reply(68 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f675402f080 con 0x7f674c011040 2023-10-05T19:28:37.068 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.062+0000 7f6764743700 1 -- 172.21.15.167:0/639123382 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:69 5.13 5:c8000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x55fba7824b30 con 0x55fba7825490 2023-10-05T19:28:37.068 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.062+0000 7f6763f42700 1 -- 172.21.15.167:0/639123382 <== osd.3 v2:172.21.15.167:6808/30036 18 ==== osd_op_reply(69 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f675c02e590 con 0x55fba7825490 2023-10-05T19:28:37.068 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.062+0000 7f6763f42700 1 -- 172.21.15.167:0/639123382 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:70 5.b 5:d0000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x55fba7824b30 con 0x7f674c011040 2023-10-05T19:28:37.070 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.062+0000 7f6764743700 1 -- 172.21.15.167:0/639123382 <== osd.0 v2:172.21.15.118:6800/27690 18 ==== osd_op_reply(70 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f675402f080 con 0x7f674c011040 2023-10-05T19:28:37.070 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.062+0000 7f6764743700 1 -- 172.21.15.167:0/639123382 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:71 5.1b 5:d8000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x55fba7824b30 con 0x7f675c007070 2023-10-05T19:28:37.070 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.066+0000 7f6763741700 1 -- 172.21.15.167:0/639123382 <== osd.2 v2:172.21.15.167:6816/30037 11 ==== osd_op_reply(71 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f674c004520 con 0x7f675c007070 2023-10-05T19:28:37.070 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.066+0000 7f6763741700 1 -- 172.21.15.167:0/639123382 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:72 5.7 5:e0000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x55fba7824b30 con 0x55fba7825490 2023-10-05T19:28:37.071 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.066+0000 7f6763f42700 1 -- 172.21.15.167:0/639123382 <== osd.3 v2:172.21.15.167:6808/30036 19 ==== osd_op_reply(72 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f675c02e590 con 0x55fba7825490 2023-10-05T19:28:37.072 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.066+0000 7f6763f42700 1 -- 172.21.15.167:0/639123382 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:73 5.17 5:e8000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x55fba7824b30 con 0x7f674c011040 2023-10-05T19:28:37.072 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.066+0000 7f6764743700 1 -- 172.21.15.167:0/639123382 <== osd.0 v2:172.21.15.118:6800/27690 19 ==== osd_op_reply(73 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f675402f080 con 0x7f674c011040 2023-10-05T19:28:37.072 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.066+0000 7f6764743700 1 -- 172.21.15.167:0/639123382 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:74 5.f 5:f0000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x55fba7824b30 con 0x55fba77ed610 2023-10-05T19:28:37.073 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.066+0000 7f6764743700 1 -- 172.21.15.167:0/639123382 <== osd.1 v2:172.21.15.167:6800/30035 25 ==== osd_op_reply(74 [pgnls start_epoch 30 out=115b] v23'2 uv2 ondisk = 1) v8 ==== 144+0+115 (crc 0 0 0) 0x7f675402f080 con 0x55fba77ed610 2023-10-05T19:28:37.073 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.066+0000 7f6764743700 1 -- 172.21.15.167:0/639123382 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:75 5.1f 5:f8000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x55fba7824b30 con 0x7f675c007070 2023-10-05T19:28:37.074 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.070+0000 7f6763741700 1 -- 172.21.15.167:0/639123382 <== osd.2 v2:172.21.15.167:6816/30037 12 ==== osd_op_reply(75 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f674c004520 con 0x7f675c007070 2023-10-05T19:28:37.074 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.070+0000 7f676640de40 20 RGWRados::pool_iterate: got zonegroup_info.4f1f719d-9648-417b-b10c-e5d939825419 2023-10-05T19:28:37.075 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.070+0000 7f676640de40 20 RGWRados::pool_iterate: got zone_info.9a9df965-6612-45d4-a45f-6e3c889bacc2 2023-10-05T19:28:37.075 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.070+0000 7f676640de40 20 RGWRados::pool_iterate: got zone_names.default 2023-10-05T19:28:37.075 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.070+0000 7f676640de40 20 RGWRados::pool_iterate: got zonegroups_names.default 2023-10-05T19:28:37.075 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.070+0000 7f676640de40 20 rados->read ofs=0 len=0 2023-10-05T19:28:37.075 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.070+0000 7f676640de40 1 -- 172.21.15.167:0/639123382 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:76 5.f 5:f43b6ece:::zone_names.default:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x55fba78289d0 con 0x55fba77ed610 2023-10-05T19:28:37.076 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.070+0000 7f6764743700 1 -- 172.21.15.167:0/639123382 <== osd.1 v2:172.21.15.167:6800/30035 26 ==== osd_op_reply(76 zone_names.default [read 0~46 out=46b] v0'0 uv1 ondisk = 0) v8 ==== 162+0+46 (crc 0 0 0) 0x7f675402f080 con 0x55fba77ed610 2023-10-05T19:28:37.076 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.070+0000 7f676640de40 20 rados_obj.operate() r=0 bl.length=46 2023-10-05T19:28:37.076 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.070+0000 7f676640de40 20 rados->read ofs=0 len=0 2023-10-05T19:28:37.076 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.070+0000 7f676640de40 1 -- 172.21.15.167:0/639123382 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:77 5.1d 5:b8374eb4:::zone_info.9a9df965-6612-45d4-a45f-6e3c889bacc2:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x55fba7828d80 con 0x55fba77ed610 2023-10-05T19:28:37.077 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.070+0000 7f6764743700 1 -- 172.21.15.167:0/639123382 <== osd.1 v2:172.21.15.167:6800/30035 27 ==== osd_op_reply(77 zone_info.9a9df965-6612-45d4-a45f-6e3c889bacc2 [read 0~874 out=874b] v0'0 uv2 ondisk = 0) v8 ==== 190+0+874 (crc 0 0 0) 0x7f675402f080 con 0x55fba77ed610 2023-10-05T19:28:37.077 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.070+0000 7f676640de40 20 rados_obj.operate() r=0 bl.length=874 2023-10-05T19:28:37.077 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.070+0000 7f676640de40 20 rados->read ofs=0 len=0 2023-10-05T19:28:37.077 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.070+0000 7f676640de40 1 -- 172.21.15.167:0/639123382 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:78 5.f 5:f4c53578:::zonegroups_names.default:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x55fba7828db0 con 0x55fba77ed610 2023-10-05T19:28:37.077 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.070+0000 7f6764743700 1 -- 172.21.15.167:0/639123382 <== osd.1 v2:172.21.15.167:6800/30035 28 ==== osd_op_reply(78 zonegroups_names.default [read 0~46 out=46b] v0'0 uv2 ondisk = 0) v8 ==== 168+0+46 (crc 0 0 0) 0x7f675402f080 con 0x55fba77ed610 2023-10-05T19:28:37.077 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.070+0000 7f676640de40 20 rados_obj.operate() r=0 bl.length=46 2023-10-05T19:28:37.077 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.070+0000 7f676640de40 20 rados->read ofs=0 len=0 2023-10-05T19:28:37.077 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.070+0000 7f676640de40 1 -- 172.21.15.167:0/639123382 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:79 5.c 5:30d299a5:::zonegroup_info.4f1f719d-9648-417b-b10c-e5d939825419:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x55fba78291e0 con 0x7f674c011040 2023-10-05T19:28:37.078 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.074+0000 7f6764743700 1 -- 172.21.15.167:0/639123382 <== osd.0 v2:172.21.15.118:6800/27690 20 ==== osd_op_reply(79 zonegroup_info.4f1f719d-9648-417b-b10c-e5d939825419 [read 0~362 out=362b] v0'0 uv1 ondisk = 0) v8 ==== 195+0+362 (crc 0 0 0) 0x7f675402f080 con 0x7f674c011040 2023-10-05T19:28:37.078 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.074+0000 7f676640de40 20 rados_obj.operate() r=0 bl.length=362 2023-10-05T19:28:37.078 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.074+0000 7f676640de40 20 rados->read ofs=0 len=0 2023-10-05T19:28:37.078 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.074+0000 7f676640de40 1 -- 172.21.15.167:0/639123382 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:80 5.3 5:c0ad8cbd:::region_map:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x55fba78291e0 con 0x7f674c011040 2023-10-05T19:28:37.079 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.074+0000 7f6764743700 1 -- 172.21.15.167:0/639123382 <== osd.0 v2:172.21.15.118:6800/27690 21 ==== osd_op_reply(80 region_map [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 ==== 154+0+0 (crc 0 0 0) 0x7f675402f080 con 0x7f674c011040 2023-10-05T19:28:37.079 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.074+0000 7f676640de40 20 rados_obj.operate() r=-2 bl.length=0 2023-10-05T19:28:37.079 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.074+0000 7f676640de40 20 zone default found 2023-10-05T19:28:37.079 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.074+0000 7f676640de40 4 Realm: () 2023-10-05T19:28:37.079 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.074+0000 7f676640de40 4 ZoneGroup: default (4f1f719d-9648-417b-b10c-e5d939825419) 2023-10-05T19:28:37.080 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.074+0000 7f676640de40 4 Zone: default (9a9df965-6612-45d4-a45f-6e3c889bacc2) 2023-10-05T19:28:37.080 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.074+0000 7f676640de40 10 cannot find current period zonegroup using local zonegroup configuration 2023-10-05T19:28:37.080 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.074+0000 7f676640de40 20 zonegroup default 2023-10-05T19:28:37.080 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.074+0000 7f676640de40 20 rados->read ofs=0 len=0 2023-10-05T19:28:37.080 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.074+0000 7f676640de40 1 -- 172.21.15.167:0/639123382 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:81 5.3 5:c52100b6:::period_config.default:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x55fba78290b0 con 0x7f674c011040 2023-10-05T19:28:37.081 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.074+0000 7f6764743700 1 -- 172.21.15.167:0/639123382 <== osd.0 v2:172.21.15.118:6800/27690 22 ==== osd_op_reply(81 period_config.default [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 ==== 165+0+0 (crc 0 0 0) 0x7f6754050040 con 0x7f674c011040 2023-10-05T19:28:37.081 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.074+0000 7f676640de40 20 rados_obj.operate() r=-2 bl.length=0 2023-10-05T19:28:37.081 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.074+0000 7f676640de40 20 rados->read ofs=0 len=0 2023-10-05T19:28:37.081 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.074+0000 7f676640de40 1 -- 172.21.15.167:0/639123382 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:82 6.2 6:4347d321:::bucket.sync-source-hints.:head [call version.read in=11b,read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x55fba782aa30 con 0x55fba7825490 2023-10-05T19:28:37.081 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.074+0000 7f6763f42700 1 -- 172.21.15.167:0/639123382 <== osd.3 v2:172.21.15.167:6808/30036 20 ==== osd_op_reply(82 bucket.sync-source-hints. [call,read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 ==== 211+0+0 (crc 0 0 0) 0x7f675c02e590 con 0x55fba7825490 2023-10-05T19:28:37.081 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.074+0000 7f676640de40 20 rados_obj.operate() r=-2 bl.length=0 2023-10-05T19:28:37.082 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.074+0000 7f676640de40 20 rados->read ofs=0 len=0 2023-10-05T19:28:37.082 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.074+0000 7f676640de40 1 -- 172.21.15.167:0/639123382 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:83 6.b 6:d467b91b:::bucket.sync-target-hints.:head [call version.read in=11b,read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x55fba782ade0 con 0x55fba7825490 2023-10-05T19:28:37.082 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.074+0000 7f6763f42700 1 -- 172.21.15.167:0/639123382 <== osd.3 v2:172.21.15.167:6808/30036 21 ==== osd_op_reply(83 bucket.sync-target-hints. [call,read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 ==== 211+0+0 (crc 0 0 0) 0x7f675c02e590 con 0x55fba7825490 2023-10-05T19:28:37.082 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.074+0000 7f676640de40 20 rados_obj.operate() r=-2 bl.length=0 2023-10-05T19:28:37.083 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.078+0000 7f676640de40 20 started sync module instance, tier type = 2023-10-05T19:28:37.083 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.078+0000 7f676640de40 20 started zone id=9a9df965-6612-45d4-a45f-6e3c889bacc2 (name=default) with tier type = 2023-10-05T19:28:37.083 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.078+0000 7f676640de40 1 -- 172.21.15.167:0/639123382 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:84 7.1f 7:f95f44c2:::notify.0:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x55fba782a0f0 con 0x7f675c007070 2023-10-05T19:28:37.085 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.078+0000 7f6763741700 1 -- 172.21.15.167:0/639123382 <== osd.2 v2:172.21.15.167:6816/30037 13 ==== osd_op_reply(84 notify.0 [create] v30'162 uv162 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f674c02f040 con 0x7f675c007070 2023-10-05T19:28:37.085 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.078+0000 7f676640de40 1 -- 172.21.15.167:0/639123382 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:85 7.1f 7:f95f44c2:::notify.0:head [watch watch cookie 94539335509152] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x55fba7824760 con 0x7f675c007070 2023-10-05T19:28:37.085 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.078+0000 7f676640de40 1 -- 172.21.15.167:0/639123382 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:86 7.0 7:05bf5b68:::notify.1:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x55fba782bb70 con 0x55fba77ed610 2023-10-05T19:28:37.088 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.082+0000 7f6764743700 1 -- 172.21.15.167:0/639123382 <== osd.1 v2:172.21.15.167:6800/30035 29 ==== osd_op_reply(86 notify.1 [create] v30'162 uv162 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f6754050040 con 0x55fba77ed610 2023-10-05T19:28:37.089 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.082+0000 7f676640de40 1 -- 172.21.15.167:0/639123382 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:87 7.0 7:05bf5b68:::notify.1:head [watch watch cookie 94539335514016] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x55fba782c1a0 con 0x55fba77ed610 2023-10-05T19:28:37.089 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.082+0000 7f676640de40 1 -- 172.21.15.167:0/639123382 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:88 7.15 7:a93a5511:::notify.2:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x55fba782cb90 con 0x55fba7825490 2023-10-05T19:28:37.089 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.082+0000 7f6763741700 1 -- 172.21.15.167:0/639123382 <== osd.2 v2:172.21.15.167:6816/30037 14 ==== osd_op_reply(85 notify.0 [watch watch cookie 94539335509152] v30'163 uv162 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f674c02f040 con 0x7f675c007070 2023-10-05T19:28:37.092 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.086+0000 7f6763f42700 1 -- 172.21.15.167:0/639123382 <== osd.3 v2:172.21.15.167:6808/30036 22 ==== osd_op_reply(88 notify.2 [create] v30'162 uv162 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f675c02e590 con 0x55fba7825490 2023-10-05T19:28:37.092 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.086+0000 7f676640de40 1 -- 172.21.15.167:0/639123382 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:89 7.15 7:a93a5511:::notify.2:head [watch watch cookie 94539335512464] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x55fba782cf90 con 0x55fba7825490 2023-10-05T19:28:37.092 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.086+0000 7f676640de40 1 -- 172.21.15.167:0/639123382 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:90 7.e 7:7759931f:::notify.3:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x55fba782d830 con 0x7f675c007070 2023-10-05T19:28:37.092 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.086+0000 7f6764743700 1 -- 172.21.15.167:0/639123382 <== osd.1 v2:172.21.15.167:6800/30035 30 ==== osd_op_reply(87 notify.1 [watch watch cookie 94539335514016] v30'163 uv162 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f6754050040 con 0x55fba77ed610 2023-10-05T19:28:37.095 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.090+0000 7f6763f42700 1 -- 172.21.15.167:0/639123382 <== osd.3 v2:172.21.15.167:6808/30036 23 ==== osd_op_reply(89 notify.2 [watch watch cookie 94539335512464] v30'163 uv162 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f675c02e590 con 0x55fba7825490 2023-10-05T19:28:37.095 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.090+0000 7f6763741700 1 -- 172.21.15.167:0/639123382 <== osd.2 v2:172.21.15.167:6816/30037 15 ==== osd_op_reply(90 notify.3 [create] v30'162 uv162 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f674c02f040 con 0x7f675c007070 2023-10-05T19:28:37.095 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.090+0000 7f676640de40 1 -- 172.21.15.167:0/639123382 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:91 7.e 7:7759931f:::notify.3:head [watch watch cookie 94539335521088] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x55fba782c780 con 0x7f675c007070 2023-10-05T19:28:37.095 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.090+0000 7f676640de40 1 -- 172.21.15.167:0/639123382 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:92 7.d 7:b4812045:::notify.4:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x55fba782e360 con 0x55fba77ed610 2023-10-05T19:28:37.096 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.090+0000 7f6763741700 1 -- 172.21.15.167:0/639123382 <== osd.2 v2:172.21.15.167:6816/30037 16 ==== osd_op_reply(91 notify.3 [watch watch cookie 94539335521088] v30'163 uv162 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f674c02f040 con 0x7f675c007070 2023-10-05T19:28:37.097 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.090+0000 7f6764743700 1 -- 172.21.15.167:0/639123382 <== osd.1 v2:172.21.15.167:6800/30035 31 ==== osd_op_reply(92 notify.4 [create] v30'162 uv162 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f6754050040 con 0x55fba77ed610 2023-10-05T19:28:37.097 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.090+0000 7f676640de40 1 -- 172.21.15.167:0/639123382 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:93 7.d 7:b4812045:::notify.4:head [watch watch cookie 94539335515184] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x55fba782e760 con 0x55fba77ed610 2023-10-05T19:28:37.097 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.090+0000 7f676640de40 1 -- 172.21.15.167:0/639123382 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:94 7.3 7:c609908c:::notify.5:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x55fba782f170 con 0x7f674c011040 2023-10-05T19:28:37.099 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.094+0000 7f6764743700 1 -- 172.21.15.167:0/639123382 <== osd.1 v2:172.21.15.167:6800/30035 32 ==== osd_op_reply(93 notify.4 [watch watch cookie 94539335515184] v30'163 uv162 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f6754050040 con 0x55fba77ed610 2023-10-05T19:28:37.100 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.094+0000 7f6764743700 1 -- 172.21.15.167:0/639123382 <== osd.0 v2:172.21.15.118:6800/27690 23 ==== osd_op_reply(94 notify.5 [create] v30'162 uv162 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f6754050040 con 0x7f674c011040 2023-10-05T19:28:37.100 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.094+0000 7f676640de40 1 -- 172.21.15.167:0/639123382 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:95 7.3 7:c609908c:::notify.5:head [watch watch cookie 94539335523888] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x55fba782f170 con 0x7f674c011040 2023-10-05T19:28:37.100 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.094+0000 7f676640de40 1 -- 172.21.15.167:0/639123382 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:96 7.14 7:2b04a3e9:::notify.6:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x55fba782fa50 con 0x7f674c011040 2023-10-05T19:28:37.102 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.098+0000 7f6764743700 1 -- 172.21.15.167:0/639123382 <== osd.0 v2:172.21.15.118:6800/27690 24 ==== osd_op_reply(96 notify.6 [create] v30'162 uv162 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f6754050040 con 0x7f674c011040 2023-10-05T19:28:37.102 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.098+0000 7f676640de40 1 -- 172.21.15.167:0/639123382 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:97 7.14 7:2b04a3e9:::notify.6:head [watch watch cookie 94539335528816] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x55fba782f5c0 con 0x7f674c011040 2023-10-05T19:28:37.103 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.098+0000 7f6764743700 1 -- 172.21.15.167:0/639123382 <== osd.0 v2:172.21.15.118:6800/27690 25 ==== osd_op_reply(95 notify.5 [watch watch cookie 94539335523888] v30'163 uv162 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f6754050040 con 0x7f674c011040 2023-10-05T19:28:37.103 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.098+0000 7f676640de40 1 -- 172.21.15.167:0/639123382 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:98 7.9 7:93e5b521:::notify.7:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x55fba78309e0 con 0x7f674c011040 2023-10-05T19:28:37.105 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.098+0000 7f6764743700 1 -- 172.21.15.167:0/639123382 <== osd.0 v2:172.21.15.118:6800/27690 26 ==== osd_op_reply(98 notify.7 [create] v30'162 uv162 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f6754050040 con 0x7f674c011040 2023-10-05T19:28:37.105 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.098+0000 7f6764743700 1 -- 172.21.15.167:0/639123382 <== osd.0 v2:172.21.15.118:6800/27690 27 ==== osd_op_reply(97 notify.6 [watch watch cookie 94539335528816] v30'163 uv162 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f6754050040 con 0x7f674c011040 2023-10-05T19:28:37.106 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.098+0000 7f676640de40 1 -- 172.21.15.167:0/639123382 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:99 7.9 7:93e5b521:::notify.7:head [watch watch cookie 94539335527552] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x55fba78309a0 con 0x7f674c011040 2023-10-05T19:28:37.106 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.098+0000 7f676640de40 20 add_watcher() i=0 2023-10-05T19:28:37.106 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.098+0000 7f676640de40 20 add_watcher() i=1 2023-10-05T19:28:37.106 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.098+0000 7f676640de40 20 add_watcher() i=2 2023-10-05T19:28:37.106 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.098+0000 7f676640de40 20 add_watcher() i=3 2023-10-05T19:28:37.106 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.098+0000 7f676640de40 20 add_watcher() i=4 2023-10-05T19:28:37.106 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.098+0000 7f676640de40 20 add_watcher() i=5 2023-10-05T19:28:37.106 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.098+0000 7f676640de40 20 add_watcher() i=6 2023-10-05T19:28:37.108 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.102+0000 7f6764743700 1 -- 172.21.15.167:0/639123382 <== osd.0 v2:172.21.15.118:6800/27690 28 ==== osd_op_reply(99 notify.7 [watch watch cookie 94539335527552] v30'163 uv162 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f6754050040 con 0x7f674c011040 2023-10-05T19:28:37.108 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.102+0000 7f676640de40 20 add_watcher() i=7 2023-10-05T19:28:37.108 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.102+0000 7f676640de40 2 all 8 watchers are set, enabling cache 2023-10-05T19:28:37.108 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.102+0000 7f676640de40 1 -- 172.21.15.167:0/639123382 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:100 6.4 6:22d26bf9:::data_loggenerations_metadata:head [call version.check_conds in=74b,call version.read in=11b,read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x55fba78309a0 con 0x55fba77ed610 2023-10-05T19:28:37.108 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.102+0000 7f6764743700 1 -- 172.21.15.167:0/639123382 <== osd.1 v2:172.21.15.167:6800/30035 33 ==== osd_op_reply(100 data_loggenerations_metadata [call,call out=48b,read 0~28 out=28b] v0'0 uv1 ondisk = 0) v8 ==== 256+0+76 (crc 0 0 0) 0x7f6754050040 con 0x55fba77ed610 2023-10-05T19:28:37.109 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.102+0000 7f676640de40 1 -- 172.21.15.167:0/639123382 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:101 6.4 6:22d26bf9:::data_loggenerations_metadata:head [watch watch cookie 94539335533744] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x55fba7830900 con 0x55fba77ed610 2023-10-05T19:28:37.110 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.106+0000 7f6764743700 1 -- 172.21.15.167:0/639123382 <== osd.1 v2:172.21.15.167:6800/30035 34 ==== osd_op_reply(101 data_loggenerations_metadata [watch watch cookie 94539335533744] v30'155 uv1 ondisk = 0) v8 ==== 172+0+0 (crc 0 0 0) 0x7f6754050040 con 0x55fba77ed610 2023-10-05T19:28:37.110 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.106+0000 7f676640de40 20 check_secure_mon_conn(): auth registy supported: methods=[2] modes=[2,1] 2023-10-05T19:28:37.110 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.106+0000 7f676640de40 20 check_secure_mon_conn(): mode 1 is insecure 2023-10-05T19:28:37.111 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.106+0000 7f676640de40 1 Processor -- start 2023-10-05T19:28:37.111 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.106+0000 7f66affff700 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2023-10-05T19:28:37.112 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.106+0000 7f676640de40 1 -- start start 2023-10-05T19:28:37.112 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.106+0000 7f676640de40 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55fba783deb0 0x55fba783e280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-05T19:28:37.112 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.106+0000 7f676640de40 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x55fba783e7c0 con 0x55fba783deb0 2023-10-05T19:28:37.112 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.106+0000 7f6763741700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55fba783deb0 0x55fba783e280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-05T19:28:37.113 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.106+0000 7f6763741700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55fba783deb0 0x55fba783e280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.118:3300/0 says I am v2:172.21.15.167:41950/0 (socket says 172.21.15.167:41950) 2023-10-05T19:28:37.113 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.106+0000 7f6763741700 1 -- 172.21.15.167:0/955323930 learned_addr learned my addr 172.21.15.167:0/955323930 (peer_addr_for_me v2:172.21.15.167:0/0) 2023-10-05T19:28:37.113 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.106+0000 7f6763741700 1 -- 172.21.15.167:0/955323930 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55fba783e900 con 0x55fba783deb0 2023-10-05T19:28:37.113 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.106+0000 7f6763741700 1 --2- 172.21.15.167:0/955323930 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55fba783deb0 0x55fba783e280 secure :-1 s=READY pgs=462 cs=0 l=1 rev1=1 crypto rx=0x7f675c000c00 tx=0x7f674c006000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=687357f4375afbea server_cookie=0 in_seq=0 out_seq=0 2023-10-05T19:28:37.113 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.106+0000 7f66a7fff700 1 -- 172.21.15.167:0/955323930 <== mon.0 v2:172.21.15.118:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f674c02c040 con 0x55fba783deb0 2023-10-05T19:28:37.113 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.106+0000 7f66a7fff700 1 -- 172.21.15.167:0/955323930 <== mon.0 v2:172.21.15.118:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f674c02d040 con 0x55fba783deb0 2023-10-05T19:28:37.114 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.106+0000 7f66a7fff700 1 -- 172.21.15.167:0/955323930 <== mon.0 v2:172.21.15.118:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f674c004e30 con 0x55fba783deb0 2023-10-05T19:28:37.115 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.110+0000 7f676640de40 1 -- 172.21.15.167:0/955323930 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55fba783deb0 msgr2=0x55fba783e280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-05T19:28:37.115 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.110+0000 7f676640de40 1 --2- 172.21.15.167:0/955323930 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55fba783deb0 0x55fba783e280 secure :-1 s=READY pgs=462 cs=0 l=1 rev1=1 crypto rx=0x7f675c000c00 tx=0x7f674c006000 comp rx=0 tx=0).stop 2023-10-05T19:28:37.115 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.110+0000 7f676640de40 1 -- 172.21.15.167:0/955323930 shutdown_connections 2023-10-05T19:28:37.115 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.110+0000 7f676640de40 1 --2- 172.21.15.167:0/955323930 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55fba783deb0 0x55fba783e280 unknown :-1 s=CLOSED pgs=462 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-05T19:28:37.115 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.110+0000 7f676640de40 1 -- 172.21.15.167:0/955323930 >> 172.21.15.167:0/955323930 conn(0x55fba7839270 msgr2=0x55fba78373c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-05T19:28:37.115 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.110+0000 7f676640de40 1 -- 172.21.15.167:0/955323930 shutdown_connections 2023-10-05T19:28:37.115 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.110+0000 7f676640de40 1 -- 172.21.15.167:0/955323930 wait complete. 2023-10-05T19:28:37.116 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.110+0000 7f676640de40 1 Processor -- start 2023-10-05T19:28:37.116 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.110+0000 7f676640de40 1 -- start start 2023-10-05T19:28:37.116 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.110+0000 7f676640de40 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55fba783deb0 0x55fba7843aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-05T19:28:37.116 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.110+0000 7f676640de40 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x55fba7844aa0 con 0x55fba783deb0 2023-10-05T19:28:37.117 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.110+0000 7f6763741700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55fba783deb0 0x55fba7843aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-05T19:28:37.117 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.110+0000 7f6763741700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55fba783deb0 0x55fba7843aa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.118:3300/0 says I am v2:172.21.15.167:41966/0 (socket says 172.21.15.167:41966) 2023-10-05T19:28:37.117 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.110+0000 7f6763741700 1 -- 172.21.15.167:0/3633894867 learned_addr learned my addr 172.21.15.167:0/3633894867 (peer_addr_for_me v2:172.21.15.167:0/0) 2023-10-05T19:28:37.118 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.110+0000 7f6763741700 1 -- 172.21.15.167:0/3633894867 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f674c004b30 con 0x55fba783deb0 2023-10-05T19:28:37.118 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.110+0000 7f6763741700 1 --2- 172.21.15.167:0/3633894867 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55fba783deb0 0x55fba7843aa0 secure :-1 s=READY pgs=463 cs=0 l=1 rev1=1 crypto rx=0x7f674c041040 tx=0x7f674c03c000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-05T19:28:37.118 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.110+0000 7f66ae7fc700 1 -- 172.21.15.167:0/3633894867 <== mon.0 v2:172.21.15.118:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f674c02c070 con 0x55fba783deb0 2023-10-05T19:28:37.118 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.110+0000 7f66ae7fc700 1 -- 172.21.15.167:0/3633894867 <== mon.0 v2:172.21.15.118:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f674c01c040 con 0x55fba783deb0 2023-10-05T19:28:37.118 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.110+0000 7f676640de40 1 -- 172.21.15.167:0/3633894867 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x55fba78440a0 con 0x55fba783deb0 2023-10-05T19:28:37.118 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.110+0000 7f66ae7fc700 1 -- 172.21.15.167:0/3633894867 <== mon.0 v2:172.21.15.118:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f674c048410 con 0x55fba783deb0 2023-10-05T19:28:37.118 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.110+0000 7f676640de40 1 -- 172.21.15.167:0/3633894867 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55fba78444e0 con 0x55fba783deb0 2023-10-05T19:28:37.119 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.114+0000 7f66ae7fc700 1 -- 172.21.15.167:0/3633894867 <== mon.0 v2:172.21.15.118:3300/0 4 ==== mgrmap(e 5) v1 ==== 40329+0+0 (secure 0 0 0) 0x7f674c045050 con 0x55fba783deb0 2023-10-05T19:28:37.119 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.114+0000 7f66ae7fc700 1 --2- 172.21.15.167:0/3633894867 >> [v2:172.21.15.167:6824/29889,v1:172.21.15.167:6825/29889] conn(0x7f66a0030c50 0x7f66a0033100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-05T19:28:37.119 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.114+0000 7f66ae7fc700 1 -- 172.21.15.167:0/3633894867 <== mon.0 v2:172.21.15.118:3300/0 5 ==== osd_map(30..30 src has 1..30) v4 ==== 6057+0+0 (secure 0 0 0) 0x7f674c053260 con 0x55fba783deb0 2023-10-05T19:28:37.119 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.114+0000 7f6764743700 1 --2- 172.21.15.167:0/3633894867 >> [v2:172.21.15.167:6824/29889,v1:172.21.15.167:6825/29889] conn(0x7f66a0030c50 0x7f66a0033100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-05T19:28:37.119 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.114+0000 7f676640de40 5 note: GC not initialized 2023-10-05T19:28:37.120 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.114+0000 7f6764743700 1 --2- 172.21.15.167:0/3633894867 >> [v2:172.21.15.167:6824/29889,v1:172.21.15.167:6825/29889] conn(0x7f66a0030c50 0x7f66a0033100 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x55fba782f7c0 tx=0x7f6754000f90 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-05T19:28:37.122 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.118+0000 7f66a6ffd700 20 reqs_thread_entry: start 2023-10-05T19:28:37.122 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.118+0000 7f66a4ff9700 20 reqs_thread_entry: start 2023-10-05T19:28:37.122 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.118+0000 7f676640de40 20 init_complete bucket index max shards: 11 2023-10-05T19:28:37.123 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.118+0000 7f676640de40 10 rgw notify: Started notification manager with: 1 workers 2023-10-05T19:28:37.123 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.118+0000 7f6696ffd700 20 rgw notify: INFO: next queues processing will happen at: Thu Oct 5 19:29:07 2023 2023-10-05T19:28:37.124 INFO:teuthology.orchestra.run.smithi167.stderr: 2023-10-05T19:28:37.124 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.118+0000 7f66967fc700 20 reqs_thread_entry: start 2023-10-05T19:28:37.124 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.118+0000 7f676640de40 1 --2- 172.21.15.167:0/3633894867 >> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] conn(0x55fba78ab070 0x55fba78ad450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-05T19:28:37.124 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.118+0000 7f676640de40 1 -- 172.21.15.167:0/3633894867 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:1 6.18 6:19c98439:::2023-10-05-19-9a9df965-6612-45d4-a45f-6e3c889bacc2.4223.4-myfoo4:head [delete] snapc 0=[] ondisk+write+peerstat_old+known_if_redirected+supports_pool_eio e30) v8 -- 0x55fba78ad990 con 0x55fba78ab070 2023-10-05T19:28:37.124 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.118+0000 7f6763f42700 1 --2- 172.21.15.167:0/3633894867 >> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] conn(0x55fba78ab070 0x55fba78ad450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-05T19:28:37.124 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.118+0000 7f6763f42700 1 --2- 172.21.15.167:0/3633894867 >> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] conn(0x55fba78ab070 0x55fba78ad450 crc :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-05T19:28:37.127 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.122+0000 7f6763f42700 1 -- 172.21.15.167:0/3633894867 <== osd.0 v2:172.21.15.118:6800/27690 1 ==== osd_op_reply(1 2023-10-05-19-9a9df965-6612-45d4-a45f-6e3c889bacc2.4223.4-myfoo4 [delete] v30'21 uv21 ondisk = 0) v8 ==== 208+0+0 (crc 0 0 0) 0x7f675c02e590 con 0x55fba78ab070 2023-10-05T19:28:37.128 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.122+0000 7f676640de40 1 -- 172.21.15.167:0/639123382 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:102 7.1f 7:f95f44c2:::notify.0:head [watch unwatch cookie 94539335509152] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x55fba786a1b0 con 0x7f675c007070 2023-10-05T19:28:37.130 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.126+0000 7f6763741700 1 -- 172.21.15.167:0/639123382 <== osd.2 v2:172.21.15.167:6816/30037 17 ==== osd_op_reply(102 notify.0 [watch unwatch cookie 94539335509152] v30'164 uv162 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f674c02f040 con 0x7f675c007070 2023-10-05T19:28:37.130 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.126+0000 7f676640de40 20 remove_watcher() i=0 2023-10-05T19:28:37.130 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.126+0000 7f676640de40 2 removed watcher, disabling cache 2023-10-05T19:28:37.131 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.126+0000 7f676640de40 1 -- 172.21.15.167:0/639123382 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:103 7.0 7:05bf5b68:::notify.1:head [watch unwatch cookie 94539335514016] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x55fba782a4a0 con 0x55fba77ed610 2023-10-05T19:28:37.133 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.126+0000 7f6764743700 1 -- 172.21.15.167:0/639123382 <== osd.1 v2:172.21.15.167:6800/30035 35 ==== osd_op_reply(103 notify.1 [watch unwatch cookie 94539335514016] v30'164 uv162 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f6754050040 con 0x55fba77ed610 2023-10-05T19:28:37.133 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.126+0000 7f676640de40 20 remove_watcher() i=1 2023-10-05T19:28:37.133 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.126+0000 7f676640de40 1 -- 172.21.15.167:0/639123382 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:104 7.15 7:a93a5511:::notify.2:head [watch unwatch cookie 94539335512464] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x55fba782a4a0 con 0x55fba7825490 2023-10-05T19:28:37.136 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.130+0000 7f6763f42700 1 -- 172.21.15.167:0/639123382 <== osd.3 v2:172.21.15.167:6808/30036 24 ==== osd_op_reply(104 notify.2 [watch unwatch cookie 94539335512464] v30'164 uv162 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f675c02e590 con 0x55fba7825490 2023-10-05T19:28:37.136 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.130+0000 7f676640de40 20 remove_watcher() i=2 2023-10-05T19:28:37.136 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.130+0000 7f676640de40 1 -- 172.21.15.167:0/639123382 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:105 7.e 7:7759931f:::notify.3:head [watch unwatch cookie 94539335521088] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x55fba782b190 con 0x7f675c007070 2023-10-05T19:28:37.139 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.134+0000 7f6763741700 1 -- 172.21.15.167:0/639123382 <== osd.2 v2:172.21.15.167:6816/30037 18 ==== osd_op_reply(105 notify.3 [watch unwatch cookie 94539335521088] v30'164 uv162 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f674c02f040 con 0x7f675c007070 2023-10-05T19:28:37.139 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.134+0000 7f676640de40 20 remove_watcher() i=3 2023-10-05T19:28:37.139 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.134+0000 7f676640de40 1 -- 172.21.15.167:0/639123382 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:106 7.d 7:b4812045:::notify.4:head [watch unwatch cookie 94539335515184] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x55fba782b190 con 0x55fba77ed610 2023-10-05T19:28:37.142 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.138+0000 7f6764743700 1 -- 172.21.15.167:0/639123382 <== osd.1 v2:172.21.15.167:6800/30035 36 ==== osd_op_reply(106 notify.4 [watch unwatch cookie 94539335515184] v30'164 uv162 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f6754050040 con 0x55fba77ed610 2023-10-05T19:28:37.142 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.138+0000 7f676640de40 20 remove_watcher() i=4 2023-10-05T19:28:37.143 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.138+0000 7f676640de40 1 -- 172.21.15.167:0/639123382 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:107 7.3 7:c609908c:::notify.5:head [watch unwatch cookie 94539335523888] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x55fba782b190 con 0x7f674c011040 2023-10-05T19:28:37.145 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.138+0000 7f6764743700 1 -- 172.21.15.167:0/639123382 <== osd.0 v2:172.21.15.118:6800/27690 29 ==== osd_op_reply(107 notify.5 [watch unwatch cookie 94539335523888] v30'164 uv162 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f6754050040 con 0x7f674c011040 2023-10-05T19:28:37.145 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.138+0000 7f676640de40 20 remove_watcher() i=5 2023-10-05T19:28:37.145 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.138+0000 7f676640de40 1 -- 172.21.15.167:0/639123382 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:108 7.14 7:2b04a3e9:::notify.6:head [watch unwatch cookie 94539335528816] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x55fba782b190 con 0x7f674c011040 2023-10-05T19:28:37.148 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.142+0000 7f6764743700 1 -- 172.21.15.167:0/639123382 <== osd.0 v2:172.21.15.118:6800/27690 30 ==== osd_op_reply(108 notify.6 [watch unwatch cookie 94539335528816] v30'164 uv162 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f6754050040 con 0x7f674c011040 2023-10-05T19:28:37.148 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.142+0000 7f676640de40 20 remove_watcher() i=6 2023-10-05T19:28:37.149 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.142+0000 7f676640de40 1 -- 172.21.15.167:0/639123382 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:109 7.9 7:93e5b521:::notify.7:head [watch unwatch cookie 94539335527552] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x55fba782f170 con 0x7f674c011040 2023-10-05T19:28:37.151 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.146+0000 7f6764743700 1 -- 172.21.15.167:0/639123382 <== osd.0 v2:172.21.15.118:6800/27690 31 ==== osd_op_reply(109 notify.7 [watch unwatch cookie 94539335527552] v30'164 uv162 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f6754050040 con 0x7f674c011040 2023-10-05T19:28:37.151 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.146+0000 7f676640de40 20 remove_watcher() i=7 2023-10-05T19:28:37.154 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.146+0000 7f66affff700 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2023-10-05T19:28:37.154 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.146+0000 7f676640de40 1 -- 172.21.15.167:0/639123382 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:110 6.4 6:22d26bf9:::data_loggenerations_metadata:head [watch unwatch cookie 94539335533744] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x55fba786ce20 con 0x55fba77ed610 2023-10-05T19:28:37.157 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.150+0000 7f6764743700 1 -- 172.21.15.167:0/639123382 <== osd.1 v2:172.21.15.167:6800/30035 37 ==== osd_op_reply(110 data_loggenerations_metadata [watch unwatch cookie 94539335533744] v30'156 uv1 ondisk = 0) v8 ==== 172+0+0 (crc 0 0 0) 0x7f6754050040 con 0x55fba77ed610 2023-10-05T19:28:37.157 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.150+0000 7f676640de40 1 -- 172.21.15.167:0/639123382 >> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] conn(0x7f674c011040 msgr2=0x7f674c014950 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-05T19:28:37.157 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.150+0000 7f676640de40 1 --2- 172.21.15.167:0/639123382 >> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] conn(0x7f674c011040 0x7f674c014950 crc :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-05T19:28:37.158 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.150+0000 7f676640de40 1 -- 172.21.15.167:0/639123382 >> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] conn(0x55fba77ed610 msgr2=0x55fba7821630 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-05T19:28:37.158 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.150+0000 7f676640de40 1 --2- 172.21.15.167:0/639123382 >> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] conn(0x55fba77ed610 0x55fba7821630 crc :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-05T19:28:37.158 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.150+0000 7f676640de40 1 -- 172.21.15.167:0/639123382 >> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] conn(0x7f675c007070 msgr2=0x7f675c043890 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-05T19:28:37.158 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.150+0000 7f676640de40 1 --2- 172.21.15.167:0/639123382 >> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] conn(0x7f675c007070 0x7f675c043890 crc :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-05T19:28:37.158 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.150+0000 7f676640de40 1 -- 172.21.15.167:0/639123382 >> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] conn(0x55fba7825490 msgr2=0x55fba7827870 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-05T19:28:37.158 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.150+0000 7f676640de40 1 --2- 172.21.15.167:0/639123382 >> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] conn(0x55fba7825490 0x55fba7827870 crc :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-05T19:28:37.158 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.150+0000 7f676640de40 1 -- 172.21.15.167:0/639123382 >> [v2:172.21.15.167:6824/29889,v1:172.21.15.167:6825/29889] conn(0x7f6748030c00 msgr2=0x7f67480330b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-05T19:28:37.158 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.150+0000 7f676640de40 1 --2- 172.21.15.167:0/639123382 >> [v2:172.21.15.167:6824/29889,v1:172.21.15.167:6825/29889] conn(0x7f6748030c00 0x7f67480330b0 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7f674c001560 tx=0x7f674c006650 comp rx=0 tx=0).stop 2023-10-05T19:28:37.158 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.150+0000 7f676640de40 1 -- 172.21.15.167:0/639123382 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55fba77ecd00 msgr2=0x55fba7815cb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-05T19:28:37.158 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.150+0000 7f6764743700 1 -- 172.21.15.167:0/639123382 reap_dead start 2023-10-05T19:28:37.159 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.150+0000 7f676640de40 1 --2- 172.21.15.167:0/639123382 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55fba77ecd00 0x55fba7815cb0 secure :-1 s=READY pgs=461 cs=0 l=1 rev1=1 crypto rx=0x7f675c003070 tx=0x7f675c01b000 comp rx=0 tx=0).stop 2023-10-05T19:28:37.159 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.150+0000 7f676640de40 1 -- 172.21.15.167:0/639123382 shutdown_connections 2023-10-05T19:28:37.159 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.150+0000 7f676640de40 1 --2- 172.21.15.167:0/639123382 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55fba77ecd00 0x55fba7815cb0 unknown :-1 s=CLOSED pgs=461 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-05T19:28:37.159 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.150+0000 7f676640de40 1 -- 172.21.15.167:0/639123382 >> 172.21.15.167:0/639123382 conn(0x55fba77e8720 msgr2=0x55fba78051e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-05T19:28:37.159 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.150+0000 7f676640de40 1 -- 172.21.15.167:0/639123382 shutdown_connections 2023-10-05T19:28:37.159 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.150+0000 7f676640de40 1 -- 172.21.15.167:0/639123382 wait complete. 2023-10-05T19:28:37.160 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.154+0000 7f676640de40 1 -- 172.21.15.167:0/3633894867 >> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] conn(0x55fba78ab070 msgr2=0x55fba78ad450 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-05T19:28:37.160 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.154+0000 7f676640de40 1 --2- 172.21.15.167:0/3633894867 >> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] conn(0x55fba78ab070 0x55fba78ad450 crc :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-05T19:28:37.160 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.154+0000 7f676640de40 1 -- 172.21.15.167:0/3633894867 >> [v2:172.21.15.167:6824/29889,v1:172.21.15.167:6825/29889] conn(0x7f66a0030c50 msgr2=0x7f66a0033100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-05T19:28:37.160 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.154+0000 7f676640de40 1 --2- 172.21.15.167:0/3633894867 >> [v2:172.21.15.167:6824/29889,v1:172.21.15.167:6825/29889] conn(0x7f66a0030c50 0x7f66a0033100 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x55fba782f7c0 tx=0x7f6754000f90 comp rx=0 tx=0).stop 2023-10-05T19:28:37.160 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.154+0000 7f676640de40 1 -- 172.21.15.167:0/3633894867 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55fba783deb0 msgr2=0x55fba7843aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-05T19:28:37.161 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.154+0000 7f676640de40 1 --2- 172.21.15.167:0/3633894867 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55fba783deb0 0x55fba7843aa0 secure :-1 s=READY pgs=463 cs=0 l=1 rev1=1 crypto rx=0x7f674c041040 tx=0x7f674c03c000 comp rx=0 tx=0).stop 2023-10-05T19:28:37.161 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.154+0000 7f676640de40 1 -- 172.21.15.167:0/3633894867 shutdown_connections 2023-10-05T19:28:37.161 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.154+0000 7f676640de40 1 --2- 172.21.15.167:0/3633894867 >> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] conn(0x55fba78ab070 0x55fba78ad450 unknown :-1 s=CLOSED pgs=75 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-05T19:28:37.161 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.154+0000 7f676640de40 1 --2- 172.21.15.167:0/3633894867 >> [v2:172.21.15.167:6824/29889,v1:172.21.15.167:6825/29889] conn(0x7f66a0030c50 0x7f66a0033100 unknown :-1 s=CLOSED pgs=161 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-05T19:28:37.161 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.154+0000 7f676640de40 1 --2- 172.21.15.167:0/3633894867 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55fba783deb0 0x55fba7843aa0 unknown :-1 s=CLOSED pgs=463 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-05T19:28:37.161 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.154+0000 7f676640de40 1 -- 172.21.15.167:0/3633894867 >> 172.21.15.167:0/3633894867 conn(0x55fba7839270 msgr2=0x55fba7837310 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-05T19:28:37.161 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.154+0000 7f676640de40 1 -- 172.21.15.167:0/3633894867 shutdown_connections 2023-10-05T19:28:37.161 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.154+0000 7f676640de40 1 -- 172.21.15.167:0/3633894867 wait complete. 2023-10-05T19:28:37.164 INFO:tasks.util.rgw:rgwadmin: client.0 : ['log', 'show', '--object', '2023-10-05-19-9a9df965-6612-45d4-a45f-6e3c889bacc2.4223.1-myfoo'] 2023-10-05T19:28:37.164 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'ceph', 'log', 'show', '--object', '2023-10-05-19-9a9df965-6612-45d4-a45f-6e3c889bacc2.4223.1-myfoo'] 2023-10-05T19:28:37.164 DEBUG:teuthology.orchestra.run.smithi167:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster ceph log show --object 2023-10-05-19-9a9df965-6612-45d4-a45f-6e3c889bacc2.4223.1-myfoo 2023-10-05T19:28:37.247 INFO:teuthology.orchestra.run.smithi167.stderr:ignoring --setuser ceph since I am not root 2023-10-05T19:28:37.247 INFO:teuthology.orchestra.run.smithi167.stderr:ignoring --setgroup ceph since I am not root 2023-10-05T19:28:37.248 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.242+0000 7f0ff0633e40 1 Processor -- start 2023-10-05T19:28:37.248 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.242+0000 7f0ff0633e40 1 -- start start 2023-10-05T19:28:37.248 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.242+0000 7f0ff0633e40 1 --2- >> v2:172.21.15.118:3300/0 conn(0x55fe8787c410 0x55fe8787afc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-05T19:28:37.248 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.242+0000 7f0ff0633e40 1 -- --> v1:172.21.15.118:6789/0 -- auth(proto 0 26 bytes epoch 0) v1 -- 0x55fe877c8bd0 con 0x55fe8787c7e0 2023-10-05T19:28:37.248 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.242+0000 7f0ff0633e40 1 -- --> v2:172.21.15.118:3300/0 -- mon_getmap magic: 0 v1 -- 0x55fe87785ee0 con 0x55fe8787c410 2023-10-05T19:28:37.249 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.242+0000 7f0fee168700 1 --2- >> v2:172.21.15.118:3300/0 conn(0x55fe8787c410 0x55fe8787afc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-05T19:28:37.249 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.242+0000 7f0fed967700 1 --1- >> v1:172.21.15.118:6789/0 conn(0x55fe8787c7e0 0x55fe8787b500 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.118:6789/0 says I am v1:172.21.15.167:59356/0 (socket says 172.21.15.167:59356) 2023-10-05T19:28:37.249 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.242+0000 7f0fee168700 1 --2- >> v2:172.21.15.118:3300/0 conn(0x55fe8787c410 0x55fe8787afc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.118:3300/0 says I am v2:172.21.15.167:41972/0 (socket says 172.21.15.167:41972) 2023-10-05T19:28:37.250 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.242+0000 7f0fed967700 1 -- 172.21.15.167:0/771940351 learned_addr learned my addr 172.21.15.167:0/771940351 (peer_addr_for_me v1:172.21.15.167:0/0) 2023-10-05T19:28:37.250 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.242+0000 7f0fee168700 1 -- 172.21.15.167:0/771940351 >> v1:172.21.15.118:6789/0 conn(0x55fe8787c7e0 legacy=0x55fe8787b500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-05T19:28:37.250 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.242+0000 7f0fed166700 1 -- 172.21.15.167:0/771940351 <== mon.0 v1:172.21.15.118:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 34431524 0 0) 0x55fe877c8bd0 con 0x55fe8787c7e0 2023-10-05T19:28:37.250 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.242+0000 7f0fee168700 1 -- 172.21.15.167:0/771940351 --> v2:172.21.15.118:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55fe8787bcd0 con 0x55fe8787c410 2023-10-05T19:28:37.251 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.242+0000 7f0fee168700 1 --2- 172.21.15.167:0/771940351 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55fe8787c410 0x55fe8787afc0 secure :-1 s=READY pgs=465 cs=0 l=1 rev1=1 crypto rx=0x7f0fe800a520 tx=0x7f0fe8005d40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f5b13273320d7e0b server_cookie=0 in_seq=0 out_seq=0 2023-10-05T19:28:37.251 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.246+0000 7f0fed166700 1 -- 172.21.15.167:0/771940351 <== mon.0 v2:172.21.15.118:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f0fe800f590 con 0x55fe8787c410 2023-10-05T19:28:37.251 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.246+0000 7f0fed166700 1 -- 172.21.15.167:0/771940351 <== mon.0 v2:172.21.15.118:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0fe800f6f0 con 0x55fe8787c410 2023-10-05T19:28:37.251 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.246+0000 7f0fed166700 1 -- 172.21.15.167:0/771940351 <== mon.0 v2:172.21.15.118:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f0fe800f880 con 0x55fe8787c410 2023-10-05T19:28:37.252 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.246+0000 7f0ff0633e40 1 -- 172.21.15.167:0/771940351 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55fe8787c410 msgr2=0x55fe8787afc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-05T19:28:37.252 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.246+0000 7f0ff0633e40 1 --2- 172.21.15.167:0/771940351 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55fe8787c410 0x55fe8787afc0 secure :-1 s=READY pgs=465 cs=0 l=1 rev1=1 crypto rx=0x7f0fe800a520 tx=0x7f0fe8005d40 comp rx=0 tx=0).stop 2023-10-05T19:28:37.252 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.246+0000 7f0ff0633e40 1 -- 172.21.15.167:0/771940351 shutdown_connections 2023-10-05T19:28:37.252 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.246+0000 7f0ff0633e40 1 --2- 172.21.15.167:0/771940351 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55fe8787c410 0x55fe8787afc0 unknown :-1 s=CLOSED pgs=465 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-05T19:28:37.252 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.246+0000 7f0ff0633e40 1 -- 172.21.15.167:0/771940351 >> 172.21.15.167:0/771940351 conn(0x55fe87876720 msgr2=0x55fe87878b30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-05T19:28:37.252 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.246+0000 7f0ff0633e40 1 -- 172.21.15.167:0/771940351 shutdown_connections 2023-10-05T19:28:37.252 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.246+0000 7f0ff0633e40 1 -- 172.21.15.167:0/771940351 wait complete. 2023-10-05T19:28:37.253 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.246+0000 7f0ff0633e40 1 Processor -- start 2023-10-05T19:28:37.253 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.246+0000 7f0ff0633e40 1 -- start start 2023-10-05T19:28:37.253 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.246+0000 7f0ff0633e40 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55fe8787c410 0x55fe8789cfa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-05T19:28:37.253 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.246+0000 7f0ff0633e40 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x55fe87749e30 con 0x55fe8787c410 2023-10-05T19:28:37.253 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.246+0000 7f0fee168700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55fe8787c410 0x55fe8789cfa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-05T19:28:37.254 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.246+0000 7f0fee168700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55fe8787c410 0x55fe8789cfa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.118:3300/0 says I am v2:172.21.15.167:41978/0 (socket says 172.21.15.167:41978) 2023-10-05T19:28:37.254 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.246+0000 7f0fee168700 1 -- 172.21.15.167:0/689435658 learned_addr learned my addr 172.21.15.167:0/689435658 (peer_addr_for_me v2:172.21.15.167:0/0) 2023-10-05T19:28:37.254 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.246+0000 7f0fee168700 1 -- 172.21.15.167:0/689435658 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0fe800a0e0 con 0x55fe8787c410 2023-10-05T19:28:37.254 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.246+0000 7f0fee168700 1 --2- 172.21.15.167:0/689435658 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55fe8787c410 0x55fe8789cfa0 secure :-1 s=READY pgs=466 cs=0 l=1 rev1=1 crypto rx=0x7f0fe8005d80 tx=0x7f0fe8014000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=253fa29c549b7e99 server_cookie=0 in_seq=0 out_seq=0 2023-10-05T19:28:37.254 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.246+0000 7f0fe7fff700 1 -- 172.21.15.167:0/689435658 <== mon.0 v2:172.21.15.118:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f0fe800f860 con 0x55fe8787c410 2023-10-05T19:28:37.254 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.246+0000 7f0fe7fff700 1 -- 172.21.15.167:0/689435658 <== mon.0 v2:172.21.15.118:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0fe800f9c0 con 0x55fe8787c410 2023-10-05T19:28:37.254 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.246+0000 7f0fe7fff700 1 -- 172.21.15.167:0/689435658 <== mon.0 v2:172.21.15.118:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f0fe800fb70 con 0x55fe8787c410 2023-10-05T19:28:37.255 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.250+0000 7f0ff0633e40 1 -- 172.21.15.167:0/689435658 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55fe8787c410 msgr2=0x55fe8789cfa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-05T19:28:37.255 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.250+0000 7f0ff0633e40 1 --2- 172.21.15.167:0/689435658 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55fe8787c410 0x55fe8789cfa0 secure :-1 s=READY pgs=466 cs=0 l=1 rev1=1 crypto rx=0x7f0fe8005d80 tx=0x7f0fe8014000 comp rx=0 tx=0).stop 2023-10-05T19:28:37.255 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.250+0000 7f0ff0633e40 1 -- 172.21.15.167:0/689435658 shutdown_connections 2023-10-05T19:28:37.256 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.250+0000 7f0ff0633e40 1 --2- 172.21.15.167:0/689435658 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55fe8787c410 0x55fe8789cfa0 unknown :-1 s=CLOSED pgs=466 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-05T19:28:37.256 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.250+0000 7f0ff0633e40 1 -- 172.21.15.167:0/689435658 >> 172.21.15.167:0/689435658 conn(0x55fe87876720 msgr2=0x55fe87898b60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-05T19:28:37.256 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.250+0000 7f0ff0633e40 1 -- 172.21.15.167:0/689435658 shutdown_connections 2023-10-05T19:28:37.256 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.250+0000 7f0ff0633e40 1 -- 172.21.15.167:0/689435658 wait complete. 2023-10-05T19:28:37.256 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.250+0000 7f0ff0633e40 1 Processor -- start 2023-10-05T19:28:37.256 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.250+0000 7f0ff0633e40 1 -- start start 2023-10-05T19:28:37.257 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.250+0000 7f0ff0633e40 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55fe8787c410 0x55fe878a3c00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-05T19:28:37.257 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.250+0000 7f0ff0633e40 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x55fe8771ced0 con 0x55fe8787c410 2023-10-05T19:28:37.257 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.250+0000 7f0fee168700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55fe8787c410 0x55fe878a3c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-05T19:28:37.257 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.250+0000 7f0fee168700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55fe8787c410 0x55fe878a3c00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.118:3300/0 says I am v2:172.21.15.167:41986/0 (socket says 172.21.15.167:41986) 2023-10-05T19:28:37.257 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.250+0000 7f0fee168700 1 -- 172.21.15.167:0/1589630579 learned_addr learned my addr 172.21.15.167:0/1589630579 (peer_addr_for_me v2:172.21.15.167:0/0) 2023-10-05T19:28:37.258 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.250+0000 7f0fee168700 1 -- 172.21.15.167:0/1589630579 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55fe8787bcd0 con 0x55fe8787c410 2023-10-05T19:28:37.258 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.254+0000 7f0fee168700 1 --2- 172.21.15.167:0/1589630579 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55fe8787c410 0x55fe878a3c00 secure :-1 s=READY pgs=467 cs=0 l=1 rev1=1 crypto rx=0x7f0fe800a520 tx=0x7f0fe801d000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-05T19:28:37.259 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.254+0000 7f0fe67fc700 1 -- 172.21.15.167:0/1589630579 <== mon.0 v2:172.21.15.118:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f0fe802f430 con 0x55fe8787c410 2023-10-05T19:28:37.259 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.254+0000 7f0fe67fc700 1 -- 172.21.15.167:0/1589630579 <== mon.0 v2:172.21.15.118:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0fe802f590 con 0x55fe8787c410 2023-10-05T19:28:37.259 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.254+0000 7f0ff0633e40 1 -- 172.21.15.167:0/1589630579 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x55fe878a4140 con 0x55fe8787c410 2023-10-05T19:28:37.259 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.254+0000 7f0fe67fc700 1 -- 172.21.15.167:0/1589630579 <== mon.0 v2:172.21.15.118:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f0fe802f700 con 0x55fe8787c410 2023-10-05T19:28:37.259 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.254+0000 7f0ff0633e40 1 -- 172.21.15.167:0/1589630579 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55fe878a4580 con 0x55fe8787c410 2023-10-05T19:28:37.260 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.254+0000 7f0fe67fc700 1 -- 172.21.15.167:0/1589630579 <== mon.0 v2:172.21.15.118:3300/0 4 ==== mgrmap(e 5) v1 ==== 40329+0+0 (secure 0 0 0) 0x7f0fe8038450 con 0x55fe8787c410 2023-10-05T19:28:37.260 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.254+0000 7f0ff0633e40 20 rados->read ofs=0 len=0 2023-10-05T19:28:37.261 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.254+0000 7f0fe67fc700 1 --2- 172.21.15.167:0/1589630579 >> [v2:172.21.15.167:6824/29889,v1:172.21.15.167:6825/29889] conn(0x7f0fd4030c00 0x7f0fd40330b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-05T19:28:37.261 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.254+0000 7f0fe67fc700 1 -- 172.21.15.167:0/1589630579 <== mon.0 v2:172.21.15.118:3300/0 5 ==== osd_map(30..30 src has 1..30) v4 ==== 6057+0+0 (secure 0 0 0) 0x7f0fe802b070 con 0x55fe8787c410 2023-10-05T19:28:37.261 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.254+0000 7f0fed967700 1 --2- 172.21.15.167:0/1589630579 >> [v2:172.21.15.167:6824/29889,v1:172.21.15.167:6825/29889] conn(0x7f0fd4030c00 0x7f0fd40330b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-05T19:28:37.261 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.254+0000 7f0ff0633e40 1 --2- 172.21.15.167:0/1589630579 >> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] conn(0x55fe8787c7e0 0x55fe878af690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-05T19:28:37.261 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.254+0000 7f0ff0633e40 1 -- 172.21.15.167:0/1589630579 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:1 5.12 5:49953fa1:::default.realm:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x55fe878afd90 con 0x55fe8787c7e0 2023-10-05T19:28:37.262 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.254+0000 7f0fee969700 1 --2- 172.21.15.167:0/1589630579 >> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] conn(0x55fe8787c7e0 0x55fe878af690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-05T19:28:37.262 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.254+0000 7f0fed967700 1 --2- 172.21.15.167:0/1589630579 >> [v2:172.21.15.167:6824/29889,v1:172.21.15.167:6825/29889] conn(0x7f0fd4030c00 0x7f0fd40330b0 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7f0fd80011e0 tx=0x7f0fd80013a0 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-05T19:28:37.262 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.258+0000 7f0fee969700 1 --2- 172.21.15.167:0/1589630579 >> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] conn(0x55fe8787c7e0 0x55fe878af690 crc :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-05T19:28:37.263 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.258+0000 7f0fee969700 1 -- 172.21.15.167:0/1589630579 <== osd.1 v2:172.21.15.167:6800/30035 1 ==== osd_op_reply(1 default.realm [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 ==== 157+0+0 (crc 0 0 0) 0x7f0fe000a500 con 0x55fe8787c7e0 2023-10-05T19:28:37.263 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.258+0000 7f0ff0633e40 20 rados_obj.operate() r=-2 bl.length=0 2023-10-05T19:28:37.263 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.258+0000 7f0ff0633e40 20 realm 2023-10-05T19:28:37.263 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.258+0000 7f0ff0633e40 20 rados->read ofs=0 len=0 2023-10-05T19:28:37.263 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.258+0000 7f0ff0633e40 1 -- 172.21.15.167:0/1589630579 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:2 5.12 5:49953fa1:::default.realm:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x55fe878accf0 con 0x55fe8787c7e0 2023-10-05T19:28:37.265 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.258+0000 7f0fee969700 1 -- 172.21.15.167:0/1589630579 <== osd.1 v2:172.21.15.167:6800/30035 2 ==== osd_op_reply(2 default.realm [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 ==== 157+0+0 (crc 0 0 0) 0x7f0fe000a500 con 0x55fe8787c7e0 2023-10-05T19:28:37.265 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.258+0000 7f0ff0633e40 20 rados_obj.operate() r=-2 bl.length=0 2023-10-05T19:28:37.265 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.258+0000 7f0ff0633e40 4 RGWPeriod::init failed to init realm id : (2) No such file or directory 2023-10-05T19:28:37.265 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.258+0000 7f0ff0633e40 20 rados->read ofs=0 len=0 2023-10-05T19:28:37.265 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.258+0000 7f0ff0633e40 1 -- 172.21.15.167:0/1589630579 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:3 5.12 5:49953fa1:::default.realm:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x55fe878b0870 con 0x55fe8787c7e0 2023-10-05T19:28:37.265 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.258+0000 7f0fee969700 1 -- 172.21.15.167:0/1589630579 <== osd.1 v2:172.21.15.167:6800/30035 3 ==== osd_op_reply(3 default.realm [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 ==== 157+0+0 (crc 0 0 0) 0x7f0fe000a500 con 0x55fe8787c7e0 2023-10-05T19:28:37.265 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.258+0000 7f0ff0633e40 20 rados_obj.operate() r=-2 bl.length=0 2023-10-05T19:28:37.265 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.258+0000 7f0ff0633e40 20 rados->read ofs=0 len=0 2023-10-05T19:28:37.265 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.258+0000 7f0ff0633e40 1 -- 172.21.15.167:0/1589630579 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:4 5.f 5:f43b6ece:::zone_names.default:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x55fe878b0450 con 0x55fe8787c7e0 2023-10-05T19:28:37.265 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.258+0000 7f0fee969700 1 -- 172.21.15.167:0/1589630579 <== osd.1 v2:172.21.15.167:6800/30035 4 ==== osd_op_reply(4 zone_names.default [read 0~46 out=46b] v0'0 uv1 ondisk = 0) v8 ==== 162+0+46 (crc 0 0 0) 0x7f0fe000a500 con 0x55fe8787c7e0 2023-10-05T19:28:37.266 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.258+0000 7f0ff0633e40 20 rados_obj.operate() r=0 bl.length=46 2023-10-05T19:28:37.266 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.258+0000 7f0ff0633e40 20 rados->read ofs=0 len=0 2023-10-05T19:28:37.266 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.258+0000 7f0ff0633e40 1 -- 172.21.15.167:0/1589630579 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:5 5.1d 5:b8374eb4:::zone_info.9a9df965-6612-45d4-a45f-6e3c889bacc2:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x55fe878b12e0 con 0x55fe8787c7e0 2023-10-05T19:28:37.267 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.258+0000 7f0fee969700 1 -- 172.21.15.167:0/1589630579 <== osd.1 v2:172.21.15.167:6800/30035 5 ==== osd_op_reply(5 zone_info.9a9df965-6612-45d4-a45f-6e3c889bacc2 [read 0~874 out=874b] v0'0 uv2 ondisk = 0) v8 ==== 190+0+874 (crc 0 0 0) 0x7f0fe000a500 con 0x55fe8787c7e0 2023-10-05T19:28:37.267 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.258+0000 7f0ff0633e40 20 rados_obj.operate() r=0 bl.length=874 2023-10-05T19:28:37.267 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.258+0000 7f0ff0633e40 20 searching for the correct realm 2023-10-05T19:28:37.267 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.258+0000 7f0ff0633e40 1 --2- 172.21.15.167:0/1589630579 >> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] conn(0x55fe878b3130 0x55fe878b5510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-05T19:28:37.267 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.258+0000 7f0ff0633e40 1 -- 172.21.15.167:0/1589630579 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:6 5.0 5:00000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x55fe878b5a50 con 0x55fe878b3130 2023-10-05T19:28:37.267 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.258+0000 7f0fee168700 1 --2- 172.21.15.167:0/1589630579 >> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] conn(0x55fe878b3130 0x55fe878b5510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-05T19:28:37.267 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.262+0000 7f0fee168700 1 --2- 172.21.15.167:0/1589630579 >> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] conn(0x55fe878b3130 0x55fe878b5510 crc :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-05T19:28:37.268 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.262+0000 7f0fee168700 1 -- 172.21.15.167:0/1589630579 <== osd.3 v2:172.21.15.167:6808/30036 1 ==== osd_op_reply(6 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f0fe8035070 con 0x55fe878b3130 2023-10-05T19:28:37.268 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.262+0000 7f0fee168700 1 --2- 172.21.15.167:0/1589630579 >> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] conn(0x7f0fe8038700 0x7f0fe802f860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-05T19:28:37.268 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.262+0000 7f0fee168700 1 -- 172.21.15.167:0/1589630579 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:7 5.10 5:08000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x55fe878b5a50 con 0x7f0fe8038700 2023-10-05T19:28:37.268 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.262+0000 7f0fed967700 1 --2- 172.21.15.167:0/1589630579 >> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] conn(0x7f0fe8038700 0x7f0fe802f860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-05T19:28:37.269 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.262+0000 7f0fed967700 1 --2- 172.21.15.167:0/1589630579 >> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] conn(0x7f0fe8038700 0x7f0fe802f860 crc :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-05T19:28:37.269 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.262+0000 7f0fed967700 1 -- 172.21.15.167:0/1589630579 <== osd.2 v2:172.21.15.167:6816/30037 1 ==== osd_op_reply(7 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f0fd8006070 con 0x7f0fe8038700 2023-10-05T19:28:37.269 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.262+0000 7f0fed967700 1 --2- 172.21.15.167:0/1589630579 >> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] conn(0x7f0fd800d8e0 0x7f0fd800c360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-05T19:28:37.269 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.262+0000 7f0fed967700 1 -- 172.21.15.167:0/1589630579 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:8 5.8 5:10000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x55fe878b5a50 con 0x7f0fd800d8e0 2023-10-05T19:28:37.270 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.262+0000 7f0fee969700 1 --2- 172.21.15.167:0/1589630579 >> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] conn(0x7f0fd800d8e0 0x7f0fd800c360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-05T19:28:37.270 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.266+0000 7f0fee969700 1 --2- 172.21.15.167:0/1589630579 >> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] conn(0x7f0fd800d8e0 0x7f0fd800c360 crc :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-05T19:28:37.271 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.266+0000 7f0fee969700 1 -- 172.21.15.167:0/1589630579 <== osd.0 v2:172.21.15.118:6800/27690 1 ==== osd_op_reply(8 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f0fe0014040 con 0x7f0fd800d8e0 2023-10-05T19:28:37.272 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.266+0000 7f0fee969700 1 -- 172.21.15.167:0/1589630579 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:9 5.18 5:18000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x55fe878b5a50 con 0x55fe8787c7e0 2023-10-05T19:28:37.272 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.266+0000 7f0fee969700 1 -- 172.21.15.167:0/1589630579 <== osd.1 v2:172.21.15.167:6800/30035 6 ==== osd_op_reply(9 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f0fe0014040 con 0x55fe8787c7e0 2023-10-05T19:28:37.272 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.266+0000 7f0fee969700 1 -- 172.21.15.167:0/1589630579 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:10 5.4 5:20000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x55fe878b5a50 con 0x55fe8787c7e0 2023-10-05T19:28:37.273 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.266+0000 7f0fee969700 1 -- 172.21.15.167:0/1589630579 <== osd.1 v2:172.21.15.167:6800/30035 7 ==== osd_op_reply(10 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f0fe0014040 con 0x55fe8787c7e0 2023-10-05T19:28:37.273 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.266+0000 7f0fee969700 1 -- 172.21.15.167:0/1589630579 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:11 5.14 5:28000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x55fe878b5a50 con 0x7f0fd800d8e0 2023-10-05T19:28:37.273 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.266+0000 7f0fee969700 1 -- 172.21.15.167:0/1589630579 <== osd.0 v2:172.21.15.118:6800/27690 2 ==== osd_op_reply(11 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f0fe0014040 con 0x7f0fd800d8e0 2023-10-05T19:28:37.274 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.266+0000 7f0fee969700 1 -- 172.21.15.167:0/1589630579 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:12 5.c 5:30000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x55fe878b5a50 con 0x7f0fd800d8e0 2023-10-05T19:28:37.275 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.270+0000 7f0fee969700 1 -- 172.21.15.167:0/1589630579 <== osd.0 v2:172.21.15.118:6800/27690 3 ==== osd_op_reply(12 [pgnls start_epoch 30 out=112b] v23'1 uv1 ondisk = 1) v8 ==== 144+0+112 (crc 0 0 0) 0x7f0fe0014040 con 0x7f0fd800d8e0 2023-10-05T19:28:37.275 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.270+0000 7f0fee969700 1 -- 172.21.15.167:0/1589630579 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:13 5.1c 5:38000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x55fe878b5a50 con 0x7f0fe8038700 2023-10-05T19:28:37.275 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.270+0000 7f0fed967700 1 -- 172.21.15.167:0/1589630579 <== osd.2 v2:172.21.15.167:6816/30037 2 ==== osd_op_reply(13 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f0fd8006070 con 0x7f0fe8038700 2023-10-05T19:28:37.275 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.270+0000 7f0fed967700 1 -- 172.21.15.167:0/1589630579 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:14 5.2 5:40000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x55fe878b5a50 con 0x55fe878b3130 2023-10-05T19:28:37.276 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.270+0000 7f0fee168700 1 -- 172.21.15.167:0/1589630579 <== osd.3 v2:172.21.15.167:6808/30036 2 ==== osd_op_reply(14 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f0fe8035070 con 0x55fe878b3130 2023-10-05T19:28:37.276 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.270+0000 7f0fee168700 1 -- 172.21.15.167:0/1589630579 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:15 5.12 5:48000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x55fe878b5a50 con 0x55fe8787c7e0 2023-10-05T19:28:37.277 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.270+0000 7f0fee969700 1 -- 172.21.15.167:0/1589630579 <== osd.1 v2:172.21.15.167:6800/30035 8 ==== osd_op_reply(15 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f0fe0014040 con 0x55fe8787c7e0 2023-10-05T19:28:37.277 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.270+0000 7f0fee969700 1 -- 172.21.15.167:0/1589630579 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:16 5.a 5:50000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x55fe878b5a50 con 0x7f0fd800d8e0 2023-10-05T19:28:37.278 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.274+0000 7f0fee969700 1 -- 172.21.15.167:0/1589630579 <== osd.0 v2:172.21.15.118:6800/27690 4 ==== osd_op_reply(16 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f0fe0022040 con 0x7f0fd800d8e0 2023-10-05T19:28:37.278 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.274+0000 7f0fee969700 1 -- 172.21.15.167:0/1589630579 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:17 5.1a 5:58000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x55fe878b5a50 con 0x55fe8787c7e0 2023-10-05T19:28:37.279 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.274+0000 7f0fee969700 1 -- 172.21.15.167:0/1589630579 <== osd.1 v2:172.21.15.167:6800/30035 9 ==== osd_op_reply(17 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f0fe0022040 con 0x55fe8787c7e0 2023-10-05T19:28:37.279 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.274+0000 7f0fee969700 1 -- 172.21.15.167:0/1589630579 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:18 5.6 5:60000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x55fe878b5a50 con 0x7f0fe8038700 2023-10-05T19:28:37.279 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.274+0000 7f0fed967700 1 -- 172.21.15.167:0/1589630579 <== osd.2 v2:172.21.15.167:6816/30037 3 ==== osd_op_reply(18 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f0fd8006070 con 0x7f0fe8038700 2023-10-05T19:28:37.280 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.274+0000 7f0fed967700 1 -- 172.21.15.167:0/1589630579 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:19 5.16 5:68000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x55fe878b5a50 con 0x55fe878b3130 2023-10-05T19:28:37.280 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.274+0000 7f0fee168700 1 -- 172.21.15.167:0/1589630579 <== osd.3 v2:172.21.15.167:6808/30036 3 ==== osd_op_reply(19 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f0fe8035070 con 0x55fe878b3130 2023-10-05T19:28:37.281 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.274+0000 7f0fee168700 1 -- 172.21.15.167:0/1589630579 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:20 5.e 5:70000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x55fe878b5a50 con 0x55fe878b3130 2023-10-05T19:28:37.281 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.274+0000 7f0fee168700 1 -- 172.21.15.167:0/1589630579 <== osd.3 v2:172.21.15.167:6808/30036 4 ==== osd_op_reply(20 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f0fe8035070 con 0x55fe878b3130 2023-10-05T19:28:37.281 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.274+0000 7f0fee168700 1 -- 172.21.15.167:0/1589630579 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:21 5.1e 5:78000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x55fe878b5a50 con 0x7f0fd800d8e0 2023-10-05T19:28:37.282 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.278+0000 7f0fee969700 1 -- 172.21.15.167:0/1589630579 <== osd.0 v2:172.21.15.118:6800/27690 5 ==== osd_op_reply(21 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f0fe0022040 con 0x7f0fd800d8e0 2023-10-05T19:28:37.282 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.278+0000 7f0fee969700 1 -- 172.21.15.167:0/1589630579 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:22 5.1 5:80000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x55fe878b5a50 con 0x55fe878b3130 2023-10-05T19:28:37.283 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.278+0000 7f0fee168700 1 -- 172.21.15.167:0/1589630579 <== osd.3 v2:172.21.15.167:6808/30036 5 ==== osd_op_reply(22 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f0fe8035070 con 0x55fe878b3130 2023-10-05T19:28:37.283 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.278+0000 7f0fee168700 1 -- 172.21.15.167:0/1589630579 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:23 5.11 5:88000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x55fe878b5a50 con 0x55fe878b3130 2023-10-05T19:28:37.284 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.278+0000 7f0fee168700 1 -- 172.21.15.167:0/1589630579 <== osd.3 v2:172.21.15.167:6808/30036 6 ==== osd_op_reply(23 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f0fe8035070 con 0x55fe878b3130 2023-10-05T19:28:37.284 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.278+0000 7f0fee168700 1 -- 172.21.15.167:0/1589630579 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:24 5.9 5:90000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x55fe878b5a50 con 0x55fe8787c7e0 2023-10-05T19:28:37.285 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.278+0000 7f0fee969700 1 -- 172.21.15.167:0/1589630579 <== osd.1 v2:172.21.15.167:6800/30035 10 ==== osd_op_reply(24 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f0fe0022040 con 0x55fe8787c7e0 2023-10-05T19:28:37.285 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.278+0000 7f0fee969700 1 -- 172.21.15.167:0/1589630579 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:25 5.19 5:98000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x55fe878b5a50 con 0x55fe8787c7e0 2023-10-05T19:28:37.285 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.278+0000 7f0fee969700 1 -- 172.21.15.167:0/1589630579 <== osd.1 v2:172.21.15.167:6800/30035 11 ==== osd_op_reply(25 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f0fe0022040 con 0x55fe8787c7e0 2023-10-05T19:28:37.286 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.278+0000 7f0fee969700 1 -- 172.21.15.167:0/1589630579 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:26 5.5 5:a0000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x55fe878b5a50 con 0x7f0fd800d8e0 2023-10-05T19:28:37.287 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.282+0000 7f0fee969700 1 -- 172.21.15.167:0/1589630579 <== osd.0 v2:172.21.15.118:6800/27690 6 ==== osd_op_reply(26 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f0fe0022040 con 0x7f0fd800d8e0 2023-10-05T19:28:37.287 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.282+0000 7f0fee969700 1 -- 172.21.15.167:0/1589630579 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:27 5.15 5:a8000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x55fe878b5a50 con 0x55fe878b3130 2023-10-05T19:28:37.287 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.282+0000 7f0fee168700 1 -- 172.21.15.167:0/1589630579 <== osd.3 v2:172.21.15.167:6808/30036 7 ==== osd_op_reply(27 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f0fe8035070 con 0x55fe878b3130 2023-10-05T19:28:37.288 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.282+0000 7f0fee168700 1 -- 172.21.15.167:0/1589630579 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:28 5.d 5:b0000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x55fe878b5a50 con 0x7f0fe8038700 2023-10-05T19:28:37.288 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.282+0000 7f0fed967700 1 -- 172.21.15.167:0/1589630579 <== osd.2 v2:172.21.15.167:6816/30037 4 ==== osd_op_reply(28 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f0fd8006070 con 0x7f0fe8038700 2023-10-05T19:28:37.288 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.282+0000 7f0fed967700 1 -- 172.21.15.167:0/1589630579 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:29 5.1d 5:b8000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x55fe878b5a50 con 0x55fe8787c7e0 2023-10-05T19:28:37.289 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.282+0000 7f0fee969700 1 -- 172.21.15.167:0/1589630579 <== osd.1 v2:172.21.15.167:6800/30035 12 ==== osd_op_reply(29 [pgnls start_epoch 30 out=107b] v30'2 uv2 ondisk = 1) v8 ==== 144+0+107 (crc 0 0 0) 0x7f0fe0022040 con 0x55fe8787c7e0 2023-10-05T19:28:37.289 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.282+0000 7f0fee969700 1 -- 172.21.15.167:0/1589630579 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:30 5.3 5:c0000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x55fe878b5a50 con 0x7f0fd800d8e0 2023-10-05T19:28:37.290 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.286+0000 7f0fee969700 1 -- 172.21.15.167:0/1589630579 <== osd.0 v2:172.21.15.118:6800/27690 7 ==== osd_op_reply(30 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f0fe0022040 con 0x7f0fd800d8e0 2023-10-05T19:28:37.290 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.286+0000 7f0fee969700 1 -- 172.21.15.167:0/1589630579 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:31 5.13 5:c8000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x55fe878b5a50 con 0x55fe878b3130 2023-10-05T19:28:37.291 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.286+0000 7f0fee168700 1 -- 172.21.15.167:0/1589630579 <== osd.3 v2:172.21.15.167:6808/30036 8 ==== osd_op_reply(31 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f0fe804e040 con 0x55fe878b3130 2023-10-05T19:28:37.291 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.286+0000 7f0fee168700 1 -- 172.21.15.167:0/1589630579 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:32 5.b 5:d0000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x55fe878b5a50 con 0x7f0fd800d8e0 2023-10-05T19:28:37.292 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.286+0000 7f0fee969700 1 -- 172.21.15.167:0/1589630579 <== osd.0 v2:172.21.15.118:6800/27690 8 ==== osd_op_reply(32 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f0fe0022040 con 0x7f0fd800d8e0 2023-10-05T19:28:37.292 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.286+0000 7f0fee969700 1 -- 172.21.15.167:0/1589630579 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:33 5.1b 5:d8000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x55fe878b5a50 con 0x7f0fe8038700 2023-10-05T19:28:37.293 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.286+0000 7f0fed967700 1 -- 172.21.15.167:0/1589630579 <== osd.2 v2:172.21.15.167:6816/30037 5 ==== osd_op_reply(33 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f0fd8006070 con 0x7f0fe8038700 2023-10-05T19:28:37.293 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.286+0000 7f0fed967700 1 -- 172.21.15.167:0/1589630579 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:34 5.7 5:e0000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x55fe878b5a50 con 0x55fe878b3130 2023-10-05T19:28:37.294 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.290+0000 7f0fee168700 1 -- 172.21.15.167:0/1589630579 <== osd.3 v2:172.21.15.167:6808/30036 9 ==== osd_op_reply(34 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f0fe804e040 con 0x55fe878b3130 2023-10-05T19:28:37.294 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.290+0000 7f0fee168700 1 -- 172.21.15.167:0/1589630579 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:35 5.17 5:e8000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x55fe878b5a50 con 0x7f0fd800d8e0 2023-10-05T19:28:37.295 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.290+0000 7f0fee969700 1 -- 172.21.15.167:0/1589630579 <== osd.0 v2:172.21.15.118:6800/27690 9 ==== osd_op_reply(35 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f0fe0022040 con 0x7f0fd800d8e0 2023-10-05T19:28:37.295 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.290+0000 7f0fee969700 1 -- 172.21.15.167:0/1589630579 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:36 5.f 5:f0000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x55fe878b5a50 con 0x55fe8787c7e0 2023-10-05T19:28:37.296 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.290+0000 7f0fee969700 1 -- 172.21.15.167:0/1589630579 <== osd.1 v2:172.21.15.167:6800/30035 13 ==== osd_op_reply(36 [pgnls start_epoch 30 out=115b] v23'2 uv2 ondisk = 1) v8 ==== 144+0+115 (crc 0 0 0) 0x7f0fe002f080 con 0x55fe8787c7e0 2023-10-05T19:28:37.296 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.290+0000 7f0fee969700 1 -- 172.21.15.167:0/1589630579 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:37 5.1f 5:f8000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x55fe878b5a50 con 0x7f0fe8038700 2023-10-05T19:28:37.297 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.290+0000 7f0fed967700 1 -- 172.21.15.167:0/1589630579 <== osd.2 v2:172.21.15.167:6816/30037 6 ==== osd_op_reply(37 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f0fd8006070 con 0x7f0fe8038700 2023-10-05T19:28:37.297 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.290+0000 7f0ff0633e40 20 RGWRados::pool_iterate: got zonegroup_info.4f1f719d-9648-417b-b10c-e5d939825419 2023-10-05T19:28:37.298 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.290+0000 7f0ff0633e40 20 RGWRados::pool_iterate: got zone_info.9a9df965-6612-45d4-a45f-6e3c889bacc2 2023-10-05T19:28:37.298 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.290+0000 7f0ff0633e40 20 RGWRados::pool_iterate: got zone_names.default 2023-10-05T19:28:37.298 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.290+0000 7f0ff0633e40 20 RGWRados::pool_iterate: got zonegroups_names.default 2023-10-05T19:28:37.298 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.290+0000 7f0ff0633e40 20 rados->read ofs=0 len=0 2023-10-05T19:28:37.298 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.290+0000 7f0ff0633e40 1 -- 172.21.15.167:0/1589630579 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:38 5.12 5:49953fa1:::default.realm:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x55fe878b1ee0 con 0x55fe8787c7e0 2023-10-05T19:28:37.299 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.294+0000 7f0fee969700 1 -- 172.21.15.167:0/1589630579 <== osd.1 v2:172.21.15.167:6800/30035 14 ==== osd_op_reply(38 default.realm [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 ==== 157+0+0 (crc 0 0 0) 0x7f0fe002f080 con 0x55fe8787c7e0 2023-10-05T19:28:37.299 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.294+0000 7f0ff0633e40 20 rados_obj.operate() r=-2 bl.length=0 2023-10-05T19:28:37.299 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.294+0000 7f0ff0633e40 20 rados->read ofs=0 len=0 2023-10-05T19:28:37.299 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.294+0000 7f0ff0633e40 1 -- 172.21.15.167:0/1589630579 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:39 5.f 5:f4c53578:::zonegroups_names.default:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x55fe878b2860 con 0x55fe8787c7e0 2023-10-05T19:28:37.299 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.294+0000 7f0fee969700 1 -- 172.21.15.167:0/1589630579 <== osd.1 v2:172.21.15.167:6800/30035 15 ==== osd_op_reply(39 zonegroups_names.default [read 0~46 out=46b] v0'0 uv2 ondisk = 0) v8 ==== 168+0+46 (crc 0 0 0) 0x7f0fe002f080 con 0x55fe8787c7e0 2023-10-05T19:28:37.299 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.294+0000 7f0ff0633e40 20 rados_obj.operate() r=0 bl.length=46 2023-10-05T19:28:37.299 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.294+0000 7f0ff0633e40 20 rados->read ofs=0 len=0 2023-10-05T19:28:37.299 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.294+0000 7f0ff0633e40 1 -- 172.21.15.167:0/1589630579 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:40 5.c 5:30d299a5:::zonegroup_info.4f1f719d-9648-417b-b10c-e5d939825419:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x55fe878b1ad0 con 0x7f0fd800d8e0 2023-10-05T19:28:37.300 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.294+0000 7f0fee969700 1 -- 172.21.15.167:0/1589630579 <== osd.0 v2:172.21.15.118:6800/27690 10 ==== osd_op_reply(40 zonegroup_info.4f1f719d-9648-417b-b10c-e5d939825419 [read 0~362 out=362b] v0'0 uv1 ondisk = 0) v8 ==== 195+0+362 (crc 0 0 0) 0x7f0fe002f080 con 0x7f0fd800d8e0 2023-10-05T19:28:37.300 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.294+0000 7f0ff0633e40 20 rados_obj.operate() r=0 bl.length=362 2023-10-05T19:28:37.300 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.294+0000 7f0ff0633e40 20 rados->read ofs=0 len=0 2023-10-05T19:28:37.300 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.294+0000 7f0ff0633e40 1 -- 172.21.15.167:0/1589630579 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:41 5.13 5:cbbf6c60:::converted:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x55fe878b62f0 con 0x55fe878b3130 2023-10-05T19:28:37.301 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.294+0000 7f0fee168700 1 -- 172.21.15.167:0/1589630579 <== osd.3 v2:172.21.15.167:6808/30036 10 ==== osd_op_reply(41 converted [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 ==== 153+0+0 (crc 0 0 0) 0x7f0fe804e040 con 0x55fe878b3130 2023-10-05T19:28:37.301 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.294+0000 7f0ff0633e40 20 rados_obj.operate() r=-2 bl.length=0 2023-10-05T19:28:37.301 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.294+0000 7f0ff0633e40 20 rados->read ofs=0 len=0 2023-10-05T19:28:37.301 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.294+0000 7f0ff0633e40 1 -- 172.21.15.167:0/1589630579 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:42 5.12 5:49953fa1:::default.realm:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x55fe878b66a0 con 0x55fe8787c7e0 2023-10-05T19:28:37.302 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.294+0000 7f0fee969700 1 -- 172.21.15.167:0/1589630579 <== osd.1 v2:172.21.15.167:6800/30035 16 ==== osd_op_reply(42 default.realm [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 ==== 157+0+0 (crc 0 0 0) 0x7f0fe002f080 con 0x55fe8787c7e0 2023-10-05T19:28:37.302 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.294+0000 7f0ff0633e40 20 rados_obj.operate() r=-2 bl.length=0 2023-10-05T19:28:37.302 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.294+0000 7f0ff0633e40 20 rados->read ofs=0 len=0 2023-10-05T19:28:37.302 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.294+0000 7f0ff0633e40 1 -- 172.21.15.167:0/1589630579 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:43 5.f 5:f4c53578:::zonegroups_names.default:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x55fe878b66a0 con 0x55fe8787c7e0 2023-10-05T19:28:37.303 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.298+0000 7f0fee969700 1 -- 172.21.15.167:0/1589630579 <== osd.1 v2:172.21.15.167:6800/30035 17 ==== osd_op_reply(43 zonegroups_names.default [read 0~46 out=46b] v0'0 uv2 ondisk = 0) v8 ==== 168+0+46 (crc 0 0 0) 0x7f0fe002f080 con 0x55fe8787c7e0 2023-10-05T19:28:37.303 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.298+0000 7f0ff0633e40 20 rados_obj.operate() r=0 bl.length=46 2023-10-05T19:28:37.303 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.298+0000 7f0ff0633e40 1 -- 172.21.15.167:0/1589630579 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:44 5.0 5:00000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x55fe878b76b0 con 0x55fe878b3130 2023-10-05T19:28:37.305 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.298+0000 7f0fee168700 1 -- 172.21.15.167:0/1589630579 <== osd.3 v2:172.21.15.167:6808/30036 11 ==== osd_op_reply(44 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f0fe804e040 con 0x55fe878b3130 2023-10-05T19:28:37.305 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.298+0000 7f0fee168700 1 -- 172.21.15.167:0/1589630579 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:45 5.10 5:08000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x55fe878b76b0 con 0x7f0fe8038700 2023-10-05T19:28:37.306 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.298+0000 7f0fed967700 1 -- 172.21.15.167:0/1589630579 <== osd.2 v2:172.21.15.167:6816/30037 7 ==== osd_op_reply(45 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f0fd8006070 con 0x7f0fe8038700 2023-10-05T19:28:37.306 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.302+0000 7f0fed967700 1 -- 172.21.15.167:0/1589630579 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:46 5.8 5:10000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x55fe878b76b0 con 0x7f0fd800d8e0 2023-10-05T19:28:37.307 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.302+0000 7f0fee969700 1 -- 172.21.15.167:0/1589630579 <== osd.0 v2:172.21.15.118:6800/27690 11 ==== osd_op_reply(46 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f0fe002f080 con 0x7f0fd800d8e0 2023-10-05T19:28:37.307 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.302+0000 7f0fee969700 1 -- 172.21.15.167:0/1589630579 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:47 5.18 5:18000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x55fe878b1ad0 con 0x55fe8787c7e0 2023-10-05T19:28:37.308 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.302+0000 7f0fee969700 1 -- 172.21.15.167:0/1589630579 <== osd.1 v2:172.21.15.167:6800/30035 18 ==== osd_op_reply(47 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f0fe002f080 con 0x55fe8787c7e0 2023-10-05T19:28:37.308 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.302+0000 7f0fee969700 1 -- 172.21.15.167:0/1589630579 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:48 5.4 5:20000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x55fe878b1ad0 con 0x55fe8787c7e0 2023-10-05T19:28:37.308 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.302+0000 7f0fee969700 1 -- 172.21.15.167:0/1589630579 <== osd.1 v2:172.21.15.167:6800/30035 19 ==== osd_op_reply(48 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f0fe002f080 con 0x55fe8787c7e0 2023-10-05T19:28:37.309 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.302+0000 7f0fee969700 1 -- 172.21.15.167:0/1589630579 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:49 5.14 5:28000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x55fe878b1ad0 con 0x7f0fd800d8e0 2023-10-05T19:28:37.310 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.302+0000 7f0fee969700 1 -- 172.21.15.167:0/1589630579 <== osd.0 v2:172.21.15.118:6800/27690 12 ==== osd_op_reply(49 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f0fe002f080 con 0x7f0fd800d8e0 2023-10-05T19:28:37.310 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.302+0000 7f0fee969700 1 -- 172.21.15.167:0/1589630579 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:50 5.c 5:30000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x55fe878b1ad0 con 0x7f0fd800d8e0 2023-10-05T19:28:37.311 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.306+0000 7f0fee969700 1 -- 172.21.15.167:0/1589630579 <== osd.0 v2:172.21.15.118:6800/27690 13 ==== osd_op_reply(50 [pgnls start_epoch 30 out=112b] v23'1 uv1 ondisk = 1) v8 ==== 144+0+112 (crc 0 0 0) 0x7f0fe002f080 con 0x7f0fd800d8e0 2023-10-05T19:28:37.311 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.306+0000 7f0fee969700 1 -- 172.21.15.167:0/1589630579 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:51 5.1c 5:38000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x55fe878b1ad0 con 0x7f0fe8038700 2023-10-05T19:28:37.312 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.306+0000 7f0fed967700 1 -- 172.21.15.167:0/1589630579 <== osd.2 v2:172.21.15.167:6816/30037 8 ==== osd_op_reply(51 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f0fd8006070 con 0x7f0fe8038700 2023-10-05T19:28:37.312 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.306+0000 7f0fed967700 1 -- 172.21.15.167:0/1589630579 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:52 5.2 5:40000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x55fe878b1ad0 con 0x55fe878b3130 2023-10-05T19:28:37.312 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.306+0000 7f0fee168700 1 -- 172.21.15.167:0/1589630579 <== osd.3 v2:172.21.15.167:6808/30036 12 ==== osd_op_reply(52 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f0fe804e040 con 0x55fe878b3130 2023-10-05T19:28:37.312 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.306+0000 7f0fee168700 1 -- 172.21.15.167:0/1589630579 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:53 5.12 5:48000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x55fe878b1ad0 con 0x55fe8787c7e0 2023-10-05T19:28:37.313 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.306+0000 7f0fee969700 1 -- 172.21.15.167:0/1589630579 <== osd.1 v2:172.21.15.167:6800/30035 20 ==== osd_op_reply(53 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f0fe002f080 con 0x55fe8787c7e0 2023-10-05T19:28:37.313 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.306+0000 7f0fee969700 1 -- 172.21.15.167:0/1589630579 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:54 5.a 5:50000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x55fe878b1ad0 con 0x7f0fd800d8e0 2023-10-05T19:28:37.314 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.310+0000 7f0fee969700 1 -- 172.21.15.167:0/1589630579 <== osd.0 v2:172.21.15.118:6800/27690 14 ==== osd_op_reply(54 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f0fe002f080 con 0x7f0fd800d8e0 2023-10-05T19:28:37.314 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.310+0000 7f0fee969700 1 -- 172.21.15.167:0/1589630579 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:55 5.1a 5:58000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x55fe878b1ad0 con 0x55fe8787c7e0 2023-10-05T19:28:37.315 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.310+0000 7f0fee969700 1 -- 172.21.15.167:0/1589630579 <== osd.1 v2:172.21.15.167:6800/30035 21 ==== osd_op_reply(55 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f0fe002f080 con 0x55fe8787c7e0 2023-10-05T19:28:37.315 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.310+0000 7f0fee969700 1 -- 172.21.15.167:0/1589630579 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:56 5.6 5:60000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x55fe878b1ad0 con 0x7f0fe8038700 2023-10-05T19:28:37.315 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.310+0000 7f0fed967700 1 -- 172.21.15.167:0/1589630579 <== osd.2 v2:172.21.15.167:6816/30037 9 ==== osd_op_reply(56 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f0fd8006070 con 0x7f0fe8038700 2023-10-05T19:28:37.316 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.310+0000 7f0fed967700 1 -- 172.21.15.167:0/1589630579 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:57 5.16 5:68000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x55fe878b1ad0 con 0x55fe878b3130 2023-10-05T19:28:37.316 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.310+0000 7f0fee168700 1 -- 172.21.15.167:0/1589630579 <== osd.3 v2:172.21.15.167:6808/30036 13 ==== osd_op_reply(57 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f0fe804e040 con 0x55fe878b3130 2023-10-05T19:28:37.316 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.310+0000 7f0fee168700 1 -- 172.21.15.167:0/1589630579 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:58 5.e 5:70000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x55fe878b1ad0 con 0x55fe878b3130 2023-10-05T19:28:37.317 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.310+0000 7f0fee168700 1 -- 172.21.15.167:0/1589630579 <== osd.3 v2:172.21.15.167:6808/30036 14 ==== osd_op_reply(58 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f0fe804e040 con 0x55fe878b3130 2023-10-05T19:28:37.317 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.310+0000 7f0fee168700 1 -- 172.21.15.167:0/1589630579 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:59 5.1e 5:78000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x55fe878b1ad0 con 0x7f0fd800d8e0 2023-10-05T19:28:37.318 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.314+0000 7f0fee969700 1 -- 172.21.15.167:0/1589630579 <== osd.0 v2:172.21.15.118:6800/27690 15 ==== osd_op_reply(59 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f0fe002f080 con 0x7f0fd800d8e0 2023-10-05T19:28:37.318 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.314+0000 7f0fee969700 1 -- 172.21.15.167:0/1589630579 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:60 5.1 5:80000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x55fe878b1ad0 con 0x55fe878b3130 2023-10-05T19:28:37.319 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.314+0000 7f0fee168700 1 -- 172.21.15.167:0/1589630579 <== osd.3 v2:172.21.15.167:6808/30036 15 ==== osd_op_reply(60 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f0fe804e040 con 0x55fe878b3130 2023-10-05T19:28:37.319 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.314+0000 7f0fee168700 1 -- 172.21.15.167:0/1589630579 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:61 5.11 5:88000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x55fe878b1ad0 con 0x55fe878b3130 2023-10-05T19:28:37.319 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.314+0000 7f0fee168700 1 -- 172.21.15.167:0/1589630579 <== osd.3 v2:172.21.15.167:6808/30036 16 ==== osd_op_reply(61 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f0fe804e040 con 0x55fe878b3130 2023-10-05T19:28:37.319 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.314+0000 7f0fee168700 1 -- 172.21.15.167:0/1589630579 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:62 5.9 5:90000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x55fe878b1ad0 con 0x55fe8787c7e0 2023-10-05T19:28:37.320 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.314+0000 7f0fee969700 1 -- 172.21.15.167:0/1589630579 <== osd.1 v2:172.21.15.167:6800/30035 22 ==== osd_op_reply(62 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f0fe002f080 con 0x55fe8787c7e0 2023-10-05T19:28:37.320 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.314+0000 7f0fee969700 1 -- 172.21.15.167:0/1589630579 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:63 5.19 5:98000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x55fe878b1ad0 con 0x55fe8787c7e0 2023-10-05T19:28:37.321 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.314+0000 7f0fee969700 1 -- 172.21.15.167:0/1589630579 <== osd.1 v2:172.21.15.167:6800/30035 23 ==== osd_op_reply(63 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f0fe002f080 con 0x55fe8787c7e0 2023-10-05T19:28:37.321 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.314+0000 7f0fee969700 1 -- 172.21.15.167:0/1589630579 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:64 5.5 5:a0000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x55fe878b1ad0 con 0x7f0fd800d8e0 2023-10-05T19:28:37.322 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.314+0000 7f0fee969700 1 -- 172.21.15.167:0/1589630579 <== osd.0 v2:172.21.15.118:6800/27690 16 ==== osd_op_reply(64 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f0fe002f080 con 0x7f0fd800d8e0 2023-10-05T19:28:37.322 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.314+0000 7f0fee969700 1 -- 172.21.15.167:0/1589630579 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:65 5.15 5:a8000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x55fe878b1ad0 con 0x55fe878b3130 2023-10-05T19:28:37.322 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.318+0000 7f0fee168700 1 -- 172.21.15.167:0/1589630579 <== osd.3 v2:172.21.15.167:6808/30036 17 ==== osd_op_reply(65 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f0fe804e040 con 0x55fe878b3130 2023-10-05T19:28:37.322 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.318+0000 7f0fee168700 1 -- 172.21.15.167:0/1589630579 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:66 5.d 5:b0000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x55fe878b1ad0 con 0x7f0fe8038700 2023-10-05T19:28:37.323 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.318+0000 7f0fed967700 1 -- 172.21.15.167:0/1589630579 <== osd.2 v2:172.21.15.167:6816/30037 10 ==== osd_op_reply(66 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f0fd8006070 con 0x7f0fe8038700 2023-10-05T19:28:37.323 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.318+0000 7f0fed967700 1 -- 172.21.15.167:0/1589630579 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:67 5.1d 5:b8000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x55fe878b1ad0 con 0x55fe8787c7e0 2023-10-05T19:28:37.324 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.318+0000 7f0fee969700 1 -- 172.21.15.167:0/1589630579 <== osd.1 v2:172.21.15.167:6800/30035 24 ==== osd_op_reply(67 [pgnls start_epoch 30 out=107b] v30'2 uv2 ondisk = 1) v8 ==== 144+0+107 (crc 0 0 0) 0x7f0fe002f080 con 0x55fe8787c7e0 2023-10-05T19:28:37.324 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.318+0000 7f0fee969700 1 -- 172.21.15.167:0/1589630579 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:68 5.3 5:c0000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x55fe878b1ad0 con 0x7f0fd800d8e0 2023-10-05T19:28:37.325 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.318+0000 7f0fee969700 1 -- 172.21.15.167:0/1589630579 <== osd.0 v2:172.21.15.118:6800/27690 17 ==== osd_op_reply(68 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f0fe002f080 con 0x7f0fd800d8e0 2023-10-05T19:28:37.325 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.318+0000 7f0fee969700 1 -- 172.21.15.167:0/1589630579 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:69 5.13 5:c8000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x55fe878b1ad0 con 0x55fe878b3130 2023-10-05T19:28:37.326 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.318+0000 7f0fee168700 1 -- 172.21.15.167:0/1589630579 <== osd.3 v2:172.21.15.167:6808/30036 18 ==== osd_op_reply(69 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f0fe804e040 con 0x55fe878b3130 2023-10-05T19:28:37.326 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.318+0000 7f0fee168700 1 -- 172.21.15.167:0/1589630579 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:70 5.b 5:d0000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x55fe878b1ad0 con 0x7f0fd800d8e0 2023-10-05T19:28:37.327 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.322+0000 7f0fee969700 1 -- 172.21.15.167:0/1589630579 <== osd.0 v2:172.21.15.118:6800/27690 18 ==== osd_op_reply(70 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f0fe002f080 con 0x7f0fd800d8e0 2023-10-05T19:28:37.327 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.322+0000 7f0fee969700 1 -- 172.21.15.167:0/1589630579 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:71 5.1b 5:d8000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x55fe878b1ad0 con 0x7f0fe8038700 2023-10-05T19:28:37.328 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.322+0000 7f0fed967700 1 -- 172.21.15.167:0/1589630579 <== osd.2 v2:172.21.15.167:6816/30037 11 ==== osd_op_reply(71 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f0fd8006070 con 0x7f0fe8038700 2023-10-05T19:28:37.328 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.322+0000 7f0fed967700 1 -- 172.21.15.167:0/1589630579 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:72 5.7 5:e0000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x55fe878b1ad0 con 0x55fe878b3130 2023-10-05T19:28:37.328 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.322+0000 7f0fee168700 1 -- 172.21.15.167:0/1589630579 <== osd.3 v2:172.21.15.167:6808/30036 19 ==== osd_op_reply(72 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f0fe804e040 con 0x55fe878b3130 2023-10-05T19:28:37.328 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.322+0000 7f0fee168700 1 -- 172.21.15.167:0/1589630579 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:73 5.17 5:e8000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x55fe878b1ad0 con 0x7f0fd800d8e0 2023-10-05T19:28:37.329 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.322+0000 7f0fee969700 1 -- 172.21.15.167:0/1589630579 <== osd.0 v2:172.21.15.118:6800/27690 19 ==== osd_op_reply(73 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f0fe002f080 con 0x7f0fd800d8e0 2023-10-05T19:28:37.329 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.322+0000 7f0fee969700 1 -- 172.21.15.167:0/1589630579 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:74 5.f 5:f0000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x55fe878b1ad0 con 0x55fe8787c7e0 2023-10-05T19:28:37.330 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.326+0000 7f0fee969700 1 -- 172.21.15.167:0/1589630579 <== osd.1 v2:172.21.15.167:6800/30035 25 ==== osd_op_reply(74 [pgnls start_epoch 30 out=115b] v23'2 uv2 ondisk = 1) v8 ==== 144+0+115 (crc 0 0 0) 0x7f0fe002f080 con 0x55fe8787c7e0 2023-10-05T19:28:37.330 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.326+0000 7f0fee969700 1 -- 172.21.15.167:0/1589630579 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:75 5.1f 5:f8000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x55fe878b1ad0 con 0x7f0fe8038700 2023-10-05T19:28:37.331 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.326+0000 7f0fed967700 1 -- 172.21.15.167:0/1589630579 <== osd.2 v2:172.21.15.167:6816/30037 12 ==== osd_op_reply(75 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f0fd8006070 con 0x7f0fe8038700 2023-10-05T19:28:37.331 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.326+0000 7f0ff0633e40 20 RGWRados::pool_iterate: got zonegroup_info.4f1f719d-9648-417b-b10c-e5d939825419 2023-10-05T19:28:37.331 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.326+0000 7f0ff0633e40 20 RGWRados::pool_iterate: got zone_info.9a9df965-6612-45d4-a45f-6e3c889bacc2 2023-10-05T19:28:37.332 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.326+0000 7f0ff0633e40 20 RGWRados::pool_iterate: got zone_names.default 2023-10-05T19:28:37.332 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.326+0000 7f0ff0633e40 20 RGWRados::pool_iterate: got zonegroups_names.default 2023-10-05T19:28:37.332 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.326+0000 7f0ff0633e40 20 rados->read ofs=0 len=0 2023-10-05T19:28:37.332 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.326+0000 7f0ff0633e40 1 -- 172.21.15.167:0/1589630579 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:76 5.f 5:f43b6ece:::zone_names.default:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x55fe878b66a0 con 0x55fe8787c7e0 2023-10-05T19:28:37.333 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.326+0000 7f0fee969700 1 -- 172.21.15.167:0/1589630579 <== osd.1 v2:172.21.15.167:6800/30035 26 ==== osd_op_reply(76 zone_names.default [read 0~46 out=46b] v0'0 uv1 ondisk = 0) v8 ==== 162+0+46 (crc 0 0 0) 0x7f0fe002f080 con 0x55fe8787c7e0 2023-10-05T19:28:37.333 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.326+0000 7f0ff0633e40 20 rados_obj.operate() r=0 bl.length=46 2023-10-05T19:28:37.333 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.326+0000 7f0ff0633e40 20 rados->read ofs=0 len=0 2023-10-05T19:28:37.333 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.326+0000 7f0ff0633e40 1 -- 172.21.15.167:0/1589630579 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:77 5.1d 5:b8374eb4:::zone_info.9a9df965-6612-45d4-a45f-6e3c889bacc2:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x55fe878b6a50 con 0x55fe8787c7e0 2023-10-05T19:28:37.333 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.326+0000 7f0fee969700 1 -- 172.21.15.167:0/1589630579 <== osd.1 v2:172.21.15.167:6800/30035 27 ==== osd_op_reply(77 zone_info.9a9df965-6612-45d4-a45f-6e3c889bacc2 [read 0~874 out=874b] v0'0 uv2 ondisk = 0) v8 ==== 190+0+874 (crc 0 0 0) 0x7f0fe002f080 con 0x55fe8787c7e0 2023-10-05T19:28:37.333 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.326+0000 7f0ff0633e40 20 rados_obj.operate() r=0 bl.length=874 2023-10-05T19:28:37.333 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.326+0000 7f0ff0633e40 20 rados->read ofs=0 len=0 2023-10-05T19:28:37.334 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.326+0000 7f0ff0633e40 1 -- 172.21.15.167:0/1589630579 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:78 5.f 5:f4c53578:::zonegroups_names.default:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x55fe878b6f40 con 0x55fe8787c7e0 2023-10-05T19:28:37.334 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.326+0000 7f0fee969700 1 -- 172.21.15.167:0/1589630579 <== osd.1 v2:172.21.15.167:6800/30035 28 ==== osd_op_reply(78 zonegroups_names.default [read 0~46 out=46b] v0'0 uv2 ondisk = 0) v8 ==== 168+0+46 (crc 0 0 0) 0x7f0fe002f080 con 0x55fe8787c7e0 2023-10-05T19:28:37.334 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.326+0000 7f0ff0633e40 20 rados_obj.operate() r=0 bl.length=46 2023-10-05T19:28:37.334 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.326+0000 7f0ff0633e40 20 rados->read ofs=0 len=0 2023-10-05T19:28:37.334 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.326+0000 7f0ff0633e40 1 -- 172.21.15.167:0/1589630579 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:79 5.c 5:30d299a5:::zonegroup_info.4f1f719d-9648-417b-b10c-e5d939825419:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x55fe878b6f40 con 0x7f0fd800d8e0 2023-10-05T19:28:37.335 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.330+0000 7f0fee969700 1 -- 172.21.15.167:0/1589630579 <== osd.0 v2:172.21.15.118:6800/27690 20 ==== osd_op_reply(79 zonegroup_info.4f1f719d-9648-417b-b10c-e5d939825419 [read 0~362 out=362b] v0'0 uv1 ondisk = 0) v8 ==== 195+0+362 (crc 0 0 0) 0x7f0fe002f080 con 0x7f0fd800d8e0 2023-10-05T19:28:37.335 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.330+0000 7f0ff0633e40 20 rados_obj.operate() r=0 bl.length=362 2023-10-05T19:28:37.335 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.330+0000 7f0ff0633e40 20 rados->read ofs=0 len=0 2023-10-05T19:28:37.336 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.330+0000 7f0ff0633e40 1 -- 172.21.15.167:0/1589630579 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:80 5.3 5:c0ad8cbd:::region_map:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x55fe878b7040 con 0x7f0fd800d8e0 2023-10-05T19:28:37.337 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.330+0000 7f0fee969700 1 -- 172.21.15.167:0/1589630579 <== osd.0 v2:172.21.15.118:6800/27690 21 ==== osd_op_reply(80 region_map [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 ==== 154+0+0 (crc 0 0 0) 0x7f0fe0050040 con 0x7f0fd800d8e0 2023-10-05T19:28:37.337 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.330+0000 7f0ff0633e40 20 rados_obj.operate() r=-2 bl.length=0 2023-10-05T19:28:37.337 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.330+0000 7f0ff0633e40 20 zone default found 2023-10-05T19:28:37.337 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.330+0000 7f0ff0633e40 4 Realm: () 2023-10-05T19:28:37.337 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.330+0000 7f0ff0633e40 4 ZoneGroup: default (4f1f719d-9648-417b-b10c-e5d939825419) 2023-10-05T19:28:37.337 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.330+0000 7f0ff0633e40 4 Zone: default (9a9df965-6612-45d4-a45f-6e3c889bacc2) 2023-10-05T19:28:37.337 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.330+0000 7f0ff0633e40 10 cannot find current period zonegroup using local zonegroup configuration 2023-10-05T19:28:37.337 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.330+0000 7f0ff0633e40 20 zonegroup default 2023-10-05T19:28:37.338 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.330+0000 7f0ff0633e40 20 rados->read ofs=0 len=0 2023-10-05T19:28:37.338 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.330+0000 7f0ff0633e40 1 -- 172.21.15.167:0/1589630579 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:81 5.3 5:c52100b6:::period_config.default:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x55fe878b7040 con 0x7f0fd800d8e0 2023-10-05T19:28:37.338 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.330+0000 7f0fee969700 1 -- 172.21.15.167:0/1589630579 <== osd.0 v2:172.21.15.118:6800/27690 22 ==== osd_op_reply(81 period_config.default [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 ==== 165+0+0 (crc 0 0 0) 0x7f0fe0050040 con 0x7f0fd800d8e0 2023-10-05T19:28:37.338 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.330+0000 7f0ff0633e40 20 rados_obj.operate() r=-2 bl.length=0 2023-10-05T19:28:37.338 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.330+0000 7f0ff0633e40 20 rados->read ofs=0 len=0 2023-10-05T19:28:37.338 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.330+0000 7f0ff0633e40 1 -- 172.21.15.167:0/1589630579 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:82 6.2 6:4347d321:::bucket.sync-source-hints.:head [call version.read in=11b,read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x55fe878b89c0 con 0x55fe878b3130 2023-10-05T19:28:37.339 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.330+0000 7f0fee168700 1 -- 172.21.15.167:0/1589630579 <== osd.3 v2:172.21.15.167:6808/30036 20 ==== osd_op_reply(82 bucket.sync-source-hints. [call,read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 ==== 211+0+0 (crc 0 0 0) 0x7f0fe804e040 con 0x55fe878b3130 2023-10-05T19:28:37.339 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.330+0000 7f0ff0633e40 20 rados_obj.operate() r=-2 bl.length=0 2023-10-05T19:28:37.339 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.330+0000 7f0ff0633e40 20 rados->read ofs=0 len=0 2023-10-05T19:28:37.340 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.330+0000 7f0ff0633e40 1 -- 172.21.15.167:0/1589630579 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:83 6.b 6:d467b91b:::bucket.sync-target-hints.:head [call version.read in=11b,read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x55fe878b8d70 con 0x55fe878b3130 2023-10-05T19:28:37.340 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.334+0000 7f0fee168700 1 -- 172.21.15.167:0/1589630579 <== osd.3 v2:172.21.15.167:6808/30036 21 ==== osd_op_reply(83 bucket.sync-target-hints. [call,read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 ==== 211+0+0 (crc 0 0 0) 0x7f0fe804e040 con 0x55fe878b3130 2023-10-05T19:28:37.340 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.334+0000 7f0ff0633e40 20 rados_obj.operate() r=-2 bl.length=0 2023-10-05T19:28:37.340 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.334+0000 7f0ff0633e40 20 started sync module instance, tier type = 2023-10-05T19:28:37.340 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.334+0000 7f0ff0633e40 20 started zone id=9a9df965-6612-45d4-a45f-6e3c889bacc2 (name=default) with tier type = 2023-10-05T19:28:37.340 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.334+0000 7f0ff0633e40 1 -- 172.21.15.167:0/1589630579 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:84 7.1f 7:f95f44c2:::notify.0:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x55fe878b8080 con 0x7f0fe8038700 2023-10-05T19:28:37.341 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.334+0000 7f0fed967700 1 -- 172.21.15.167:0/1589630579 <== osd.2 v2:172.21.15.167:6816/30037 13 ==== osd_op_reply(84 notify.0 [create] v30'165 uv165 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f0fd8006070 con 0x7f0fe8038700 2023-10-05T19:28:37.341 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.334+0000 7f0ff0633e40 1 -- 172.21.15.167:0/1589630579 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:85 7.1f 7:f95f44c2:::notify.0:head [watch watch cookie 94551684121728] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x55fe878b7cb0 con 0x7f0fe8038700 2023-10-05T19:28:37.341 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.334+0000 7f0ff0633e40 1 -- 172.21.15.167:0/1589630579 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:86 7.0 7:05bf5b68:::notify.1:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x55fe878b9b60 con 0x55fe8787c7e0 2023-10-05T19:28:37.344 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.338+0000 7f0fee969700 1 -- 172.21.15.167:0/1589630579 <== osd.1 v2:172.21.15.167:6800/30035 29 ==== osd_op_reply(86 notify.1 [create] v30'165 uv165 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f0fe0050040 con 0x55fe8787c7e0 2023-10-05T19:28:37.344 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.338+0000 7f0ff0633e40 1 -- 172.21.15.167:0/1589630579 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:87 7.0 7:05bf5b68:::notify.1:head [watch watch cookie 94551684126608] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x55fe878ba450 con 0x55fe8787c7e0 2023-10-05T19:28:37.344 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.338+0000 7f0ff0633e40 1 -- 172.21.15.167:0/1589630579 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:88 7.15 7:a93a5511:::notify.2:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x55fe878bacf0 con 0x55fe878b3130 2023-10-05T19:28:37.344 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.338+0000 7f0fed967700 1 -- 172.21.15.167:0/1589630579 <== osd.2 v2:172.21.15.167:6816/30037 14 ==== osd_op_reply(85 notify.0 [watch watch cookie 94551684121728] v30'166 uv165 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f0fd8006070 con 0x7f0fe8038700 2023-10-05T19:28:37.347 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.342+0000 7f0fee969700 1 -- 172.21.15.167:0/1589630579 <== osd.1 v2:172.21.15.167:6800/30035 30 ==== osd_op_reply(87 notify.1 [watch watch cookie 94551684126608] v30'166 uv165 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f0fe0050040 con 0x55fe8787c7e0 2023-10-05T19:28:37.347 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.342+0000 7f0fee168700 1 -- 172.21.15.167:0/1589630579 <== osd.3 v2:172.21.15.167:6808/30036 22 ==== osd_op_reply(88 notify.2 [create] v30'165 uv165 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f0fe804e040 con 0x55fe878b3130 2023-10-05T19:28:37.348 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.342+0000 7f0ff0633e40 1 -- 172.21.15.167:0/1589630579 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:89 7.15 7:a93a5511:::notify.2:head [watch watch cookie 94551684128608] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x55fe878b9240 con 0x55fe878b3130 2023-10-05T19:28:37.348 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.342+0000 7f0ff0633e40 1 -- 172.21.15.167:0/1589630579 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:90 7.e 7:7759931f:::notify.3:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x55fe878bb590 con 0x7f0fe8038700 2023-10-05T19:28:37.350 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.342+0000 7f0fed967700 1 -- 172.21.15.167:0/1589630579 <== osd.2 v2:172.21.15.167:6816/30037 15 ==== osd_op_reply(90 notify.3 [create] v30'165 uv165 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f0fd8006070 con 0x7f0fe8038700 2023-10-05T19:28:37.350 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.342+0000 7f0ff0633e40 1 -- 172.21.15.167:0/1589630579 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:91 7.e 7:7759931f:::notify.3:head [watch watch cookie 94551684133024] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x55fe878bbe30 con 0x7f0fe8038700 2023-10-05T19:28:37.350 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.342+0000 7f0ff0633e40 1 -- 172.21.15.167:0/1589630579 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:92 7.d 7:b4812045:::notify.4:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x55fe878bc9f0 con 0x55fe8787c7e0 2023-10-05T19:28:37.350 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.346+0000 7f0fee168700 1 -- 172.21.15.167:0/1589630579 <== osd.3 v2:172.21.15.167:6808/30036 23 ==== osd_op_reply(89 notify.2 [watch watch cookie 94551684128608] v30'166 uv165 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f0fe804e040 con 0x55fe878b3130 2023-10-05T19:28:37.352 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.346+0000 7f0fed967700 1 -- 172.21.15.167:0/1589630579 <== osd.2 v2:172.21.15.167:6816/30037 16 ==== osd_op_reply(91 notify.3 [watch watch cookie 94551684133024] v30'166 uv165 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f0fd8006070 con 0x7f0fe8038700 2023-10-05T19:28:37.353 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.346+0000 7f0fee969700 1 -- 172.21.15.167:0/1589630579 <== osd.1 v2:172.21.15.167:6800/30035 31 ==== osd_op_reply(92 notify.4 [create] v30'165 uv165 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f0fe0050040 con 0x55fe8787c7e0 2023-10-05T19:28:37.353 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.346+0000 7f0ff0633e40 1 -- 172.21.15.167:0/1589630579 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:93 7.d 7:b4812045:::notify.4:head [watch watch cookie 94551684135232] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x55fe878bc4c0 con 0x55fe8787c7e0 2023-10-05T19:28:37.353 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.346+0000 7f0ff0633e40 1 -- 172.21.15.167:0/1589630579 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:94 7.3 7:c609908c:::notify.5:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x55fe878bd2e0 con 0x7f0fd800d8e0 2023-10-05T19:28:37.355 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.350+0000 7f0fee969700 1 -- 172.21.15.167:0/1589630579 <== osd.1 v2:172.21.15.167:6800/30035 32 ==== osd_op_reply(93 notify.4 [watch watch cookie 94551684135232] v30'166 uv165 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f0fe0050040 con 0x55fe8787c7e0 2023-10-05T19:28:37.356 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.350+0000 7f0fee969700 1 -- 172.21.15.167:0/1589630579 <== osd.0 v2:172.21.15.118:6800/27690 23 ==== osd_op_reply(94 notify.5 [create] v30'165 uv165 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f0fe0050040 con 0x7f0fd800d8e0 2023-10-05T19:28:37.356 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.350+0000 7f0ff0633e40 1 -- 172.21.15.167:0/1589630579 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:95 7.3 7:c609908c:::notify.5:head [watch watch cookie 94551684129712] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x55fe878bd290 con 0x7f0fd800d8e0 2023-10-05T19:28:37.356 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.350+0000 7f0ff0633e40 1 -- 172.21.15.167:0/1589630579 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:96 7.14 7:2b04a3e9:::notify.6:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x55fe878bdca0 con 0x7f0fd800d8e0 2023-10-05T19:28:37.358 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.354+0000 7f0fee969700 1 -- 172.21.15.167:0/1589630579 <== osd.0 v2:172.21.15.118:6800/27690 24 ==== osd_op_reply(95 notify.5 [watch watch cookie 94551684129712] v30'166 uv165 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f0fe0050040 con 0x7f0fd800d8e0 2023-10-05T19:28:37.359 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.354+0000 7f0fee969700 1 -- 172.21.15.167:0/1589630579 <== osd.0 v2:172.21.15.118:6800/27690 25 ==== osd_op_reply(96 notify.6 [create] v30'165 uv165 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f0fe0050040 con 0x7f0fd800d8e0 2023-10-05T19:28:37.359 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.354+0000 7f0ff0633e40 1 -- 172.21.15.167:0/1589630579 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:97 7.14 7:2b04a3e9:::notify.6:head [watch watch cookie 94551684140736] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x55fe878bdca0 con 0x7f0fd800d8e0 2023-10-05T19:28:37.359 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.354+0000 7f0ff0633e40 1 -- 172.21.15.167:0/1589630579 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:98 7.9 7:93e5b521:::notify.7:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x55fe878be6b0 con 0x7f0fd800d8e0 2023-10-05T19:28:37.362 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.354+0000 7f0fee969700 1 -- 172.21.15.167:0/1589630579 <== osd.0 v2:172.21.15.118:6800/27690 26 ==== osd_op_reply(98 notify.7 [create] v30'165 uv165 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f0fe0050040 con 0x7f0fd800d8e0 2023-10-05T19:28:37.362 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.354+0000 7f0ff0633e40 1 -- 172.21.15.167:0/1589630579 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:99 7.9 7:93e5b521:::notify.7:head [watch watch cookie 94551684145600] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x55fe878beb00 con 0x7f0fd800d8e0 2023-10-05T19:28:37.362 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.354+0000 7f0ff0633e40 20 add_watcher() i=0 2023-10-05T19:28:37.362 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.354+0000 7f0ff0633e40 20 add_watcher() i=1 2023-10-05T19:28:37.362 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.354+0000 7f0ff0633e40 20 add_watcher() i=2 2023-10-05T19:28:37.362 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.354+0000 7f0ff0633e40 20 add_watcher() i=3 2023-10-05T19:28:37.362 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.354+0000 7f0ff0633e40 20 add_watcher() i=4 2023-10-05T19:28:37.362 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.354+0000 7f0ff0633e40 20 add_watcher() i=5 2023-10-05T19:28:37.362 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.354+0000 7f0fee969700 1 -- 172.21.15.167:0/1589630579 <== osd.0 v2:172.21.15.118:6800/27690 27 ==== osd_op_reply(97 notify.6 [watch watch cookie 94551684140736] v30'166 uv165 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f0fe0050040 con 0x7f0fd800d8e0 2023-10-05T19:28:37.363 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.354+0000 7f0ff0633e40 20 add_watcher() i=6 2023-10-05T19:28:37.364 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.358+0000 7f0fee969700 1 -- 172.21.15.167:0/1589630579 <== osd.0 v2:172.21.15.118:6800/27690 28 ==== osd_op_reply(99 notify.7 [watch watch cookie 94551684145600] v30'166 uv165 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f0fe0050040 con 0x7f0fd800d8e0 2023-10-05T19:28:37.364 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.358+0000 7f0ff0633e40 20 add_watcher() i=7 2023-10-05T19:28:37.364 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.358+0000 7f0ff0633e40 2 all 8 watchers are set, enabling cache 2023-10-05T19:28:37.364 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.358+0000 7f0ff0633e40 1 -- 172.21.15.167:0/1589630579 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:100 6.4 6:22d26bf9:::data_loggenerations_metadata:head [call version.check_conds in=74b,call version.read in=11b,read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x55fe878c1b30 con 0x55fe8787c7e0 2023-10-05T19:28:37.365 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.358+0000 7f0fee969700 1 -- 172.21.15.167:0/1589630579 <== osd.1 v2:172.21.15.167:6800/30035 33 ==== osd_op_reply(100 data_loggenerations_metadata [call,call out=48b,read 0~28 out=28b] v0'0 uv1 ondisk = 0) v8 ==== 256+0+76 (crc 0 0 0) 0x7f0fe0050040 con 0x55fe8787c7e0 2023-10-05T19:28:37.365 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.358+0000 7f0ff0633e40 1 -- 172.21.15.167:0/1589630579 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:101 6.4 6:22d26bf9:::data_loggenerations_metadata:head [watch watch cookie 94551684146704] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x55fe878bef50 con 0x55fe8787c7e0 2023-10-05T19:28:37.367 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.362+0000 7f0fee969700 1 -- 172.21.15.167:0/1589630579 <== osd.1 v2:172.21.15.167:6800/30035 34 ==== osd_op_reply(101 data_loggenerations_metadata [watch watch cookie 94551684146704] v30'157 uv1 ondisk = 0) v8 ==== 172+0+0 (crc 0 0 0) 0x7f0fe0050040 con 0x55fe8787c7e0 2023-10-05T19:28:37.367 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.362+0000 7f0fbefed700 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2023-10-05T19:28:37.367 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.362+0000 7f0ff0633e40 20 check_secure_mon_conn(): auth registy supported: methods=[2] modes=[2,1] 2023-10-05T19:28:37.367 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.362+0000 7f0ff0633e40 20 check_secure_mon_conn(): mode 1 is insecure 2023-10-05T19:28:37.368 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.362+0000 7f0ff0633e40 1 Processor -- start 2023-10-05T19:28:37.368 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.362+0000 7f0ff0633e40 1 -- start start 2023-10-05T19:28:37.368 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.362+0000 7f0ff0633e40 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55fe878cbce0 0x55fe878cc0b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-05T19:28:37.368 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.362+0000 7f0ff0633e40 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x55fe878cc5f0 con 0x55fe878cbce0 2023-10-05T19:28:37.368 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.362+0000 7f0fed967700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55fe878cbce0 0x55fe878cc0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-05T19:28:37.369 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.362+0000 7f0fed967700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55fe878cbce0 0x55fe878cc0b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.118:3300/0 says I am v2:172.21.15.167:42002/0 (socket says 172.21.15.167:42002) 2023-10-05T19:28:37.369 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.362+0000 7f0fed967700 1 -- 172.21.15.167:0/1035715029 learned_addr learned my addr 172.21.15.167:0/1035715029 (peer_addr_for_me v2:172.21.15.167:0/0) 2023-10-05T19:28:37.369 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.362+0000 7f0fed967700 1 -- 172.21.15.167:0/1035715029 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55fe878cc730 con 0x55fe878cbce0 2023-10-05T19:28:37.369 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.362+0000 7f0fed967700 1 --2- 172.21.15.167:0/1035715029 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55fe878cbce0 0x55fe878cc0b0 secure :-1 s=READY pgs=468 cs=0 l=1 rev1=1 crypto rx=0x55fe878b0e80 tx=0x7f0fd8006000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3d2e05b7bc3b913e server_cookie=0 in_seq=0 out_seq=0 2023-10-05T19:28:37.370 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.362+0000 7f0fbe7ec700 1 -- 172.21.15.167:0/1035715029 <== mon.0 v2:172.21.15.118:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f0fd8023040 con 0x55fe878cbce0 2023-10-05T19:28:37.370 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.362+0000 7f0fbe7ec700 1 -- 172.21.15.167:0/1035715029 <== mon.0 v2:172.21.15.118:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0fd8024040 con 0x55fe878cbce0 2023-10-05T19:28:37.370 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.362+0000 7f0fbe7ec700 1 -- 172.21.15.167:0/1035715029 <== mon.0 v2:172.21.15.118:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f0fd8013040 con 0x55fe878cbce0 2023-10-05T19:28:37.371 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.366+0000 7f0ff0633e40 1 -- 172.21.15.167:0/1035715029 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55fe878cbce0 msgr2=0x55fe878cc0b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-05T19:28:37.371 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.366+0000 7f0ff0633e40 1 --2- 172.21.15.167:0/1035715029 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55fe878cbce0 0x55fe878cc0b0 secure :-1 s=READY pgs=468 cs=0 l=1 rev1=1 crypto rx=0x55fe878b0e80 tx=0x7f0fd8006000 comp rx=0 tx=0).stop 2023-10-05T19:28:37.371 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.366+0000 7f0ff0633e40 1 -- 172.21.15.167:0/1035715029 shutdown_connections 2023-10-05T19:28:37.372 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.366+0000 7f0ff0633e40 1 --2- 172.21.15.167:0/1035715029 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55fe878cbce0 0x55fe878cc0b0 unknown :-1 s=CLOSED pgs=468 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-05T19:28:37.372 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.366+0000 7f0ff0633e40 1 -- 172.21.15.167:0/1035715029 >> 172.21.15.167:0/1035715029 conn(0x55fe878c70a0 msgr2=0x55fe878c5250 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-05T19:28:37.372 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.366+0000 7f0ff0633e40 1 -- 172.21.15.167:0/1035715029 shutdown_connections 2023-10-05T19:28:37.372 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.366+0000 7f0ff0633e40 1 -- 172.21.15.167:0/1035715029 wait complete. 2023-10-05T19:28:37.373 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.366+0000 7f0ff0633e40 1 Processor -- start 2023-10-05T19:28:37.373 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.366+0000 7f0ff0633e40 1 -- start start 2023-10-05T19:28:37.373 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.366+0000 7f0ff0633e40 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55fe878cbce0 0x55fe878d17f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-05T19:28:37.373 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.366+0000 7f0ff0633e40 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x55fe878d27f0 con 0x55fe878cbce0 2023-10-05T19:28:37.373 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.366+0000 7f0fed967700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55fe878cbce0 0x55fe878d17f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-05T19:28:37.373 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.366+0000 7f0fed967700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55fe878cbce0 0x55fe878d17f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.118:3300/0 says I am v2:172.21.15.167:42010/0 (socket says 172.21.15.167:42010) 2023-10-05T19:28:37.373 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.366+0000 7f0fed967700 1 -- 172.21.15.167:0/3985092605 learned_addr learned my addr 172.21.15.167:0/3985092605 (peer_addr_for_me v2:172.21.15.167:0/0) 2023-10-05T19:28:37.374 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.366+0000 7f0fed967700 1 -- 172.21.15.167:0/3985092605 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0fd800ed40 con 0x55fe878cbce0 2023-10-05T19:28:37.374 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.366+0000 7f0fed967700 1 --2- 172.21.15.167:0/3985092605 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55fe878cbce0 0x55fe878d17f0 secure :-1 s=READY pgs=469 cs=0 l=1 rev1=1 crypto rx=0x55fe878b0c20 tx=0x7f0fd802f000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-05T19:28:37.374 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.370+0000 7f0fbcfe9700 1 -- 172.21.15.167:0/3985092605 <== mon.0 v2:172.21.15.118:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f0fd8023040 con 0x55fe878cbce0 2023-10-05T19:28:37.375 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.370+0000 7f0fbcfe9700 1 -- 172.21.15.167:0/3985092605 <== mon.0 v2:172.21.15.118:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0fd801e5f0 con 0x55fe878cbce0 2023-10-05T19:28:37.375 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.370+0000 7f0ff0633e40 1 -- 172.21.15.167:0/3985092605 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x55fe878d1df0 con 0x55fe878cbce0 2023-10-05T19:28:37.375 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.370+0000 7f0fbcfe9700 1 -- 172.21.15.167:0/3985092605 <== mon.0 v2:172.21.15.118:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f0fd801e760 con 0x55fe878cbce0 2023-10-05T19:28:37.375 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.370+0000 7f0ff0633e40 1 -- 172.21.15.167:0/3985092605 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55fe878d2230 con 0x55fe878cbce0 2023-10-05T19:28:37.375 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.370+0000 7f0fbcfe9700 1 -- 172.21.15.167:0/3985092605 <== mon.0 v2:172.21.15.118:3300/0 4 ==== mgrmap(e 5) v1 ==== 40329+0+0 (secure 0 0 0) 0x7f0fd801e8c0 con 0x55fe878cbce0 2023-10-05T19:28:37.376 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.370+0000 7f0fbcfe9700 1 --2- 172.21.15.167:0/3985092605 >> [v2:172.21.15.167:6824/29889,v1:172.21.15.167:6825/29889] conn(0x7f0f34030c50 0x7f0f34033100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-05T19:28:37.376 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.370+0000 7f0fbcfe9700 1 -- 172.21.15.167:0/3985092605 <== mon.0 v2:172.21.15.118:3300/0 5 ==== osd_map(30..30 src has 1..30) v4 ==== 6057+0+0 (secure 0 0 0) 0x7f0fd804fc60 con 0x55fe878cbce0 2023-10-05T19:28:37.376 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.370+0000 7f0ff0633e40 5 note: GC not initialized 2023-10-05T19:28:37.377 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.370+0000 7f0fee969700 1 --2- 172.21.15.167:0/3985092605 >> [v2:172.21.15.167:6824/29889,v1:172.21.15.167:6825/29889] conn(0x7f0f34030c50 0x7f0f34033100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-05T19:28:37.377 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.370+0000 7f0fee969700 1 --2- 172.21.15.167:0/3985092605 >> [v2:172.21.15.167:6824/29889,v1:172.21.15.167:6825/29889] conn(0x7f0f34030c50 0x7f0f34033100 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x55fe878bdb50 tx=0x7f0fe0000f90 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-05T19:28:37.379 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.374+0000 7f0f277fe700 20 reqs_thread_entry: start 2023-10-05T19:28:37.380 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.374+0000 7f0ff0633e40 20 init_complete bucket index max shards: 11 2023-10-05T19:28:37.380 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.374+0000 7f0f2dffb700 20 reqs_thread_entry: start 2023-10-05T19:28:37.380 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.374+0000 7f0ff0633e40 10 rgw notify: Started notification manager with: 1 workers 2023-10-05T19:28:37.380 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.374+0000 7f0f25ffb700 20 rgw notify: INFO: next queues processing will happen at: Thu Oct 5 19:29:07 2023 2023-10-05T19:28:37.380 INFO:teuthology.orchestra.run.smithi167.stderr: 2023-10-05T19:28:37.380 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.374+0000 7f0f257fa700 20 reqs_thread_entry: start 2023-10-05T19:28:37.380 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.374+0000 7f0ff0633e40 10 log_show_next pos 0 bl 0 off 0 eof 0 2023-10-05T19:28:37.380 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.374+0000 7f0ff0633e40 1 --2- 172.21.15.167:0/3985092605 >> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] conn(0x55fe87938d20 0x55fe8793b100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-05T19:28:37.380 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.374+0000 7f0ff0633e40 1 -- 172.21.15.167:0/3985092605 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:1 6.1c 6:38c02c58:::2023-10-05-19-9a9df965-6612-45d4-a45f-6e3c889bacc2.4223.1-myfoo:head [read 0~1048576] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x55fe8793b640 con 0x55fe87938d20 2023-10-05T19:28:37.381 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.374+0000 7f0fee168700 1 --2- 172.21.15.167:0/3985092605 >> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] conn(0x55fe87938d20 0x55fe8793b100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-05T19:28:37.381 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.374+0000 7f0fee168700 1 --2- 172.21.15.167:0/3985092605 >> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] conn(0x55fe87938d20 0x55fe8793b100 crc :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-05T19:28:37.382 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.374+0000 7f0fee168700 1 -- 172.21.15.167:0/3985092605 <== osd.3 v2:172.21.15.167:6808/30036 1 ==== osd_op_reply(1 2023-10-05-19-9a9df965-6612-45d4-a45f-6e3c889bacc2.4223.1-myfoo [read 0~3730 out=3730b] v0'0 uv30 ondisk = 0) v8 ==== 207+0+3730 (crc 0 0 0) 0x7f0fe804e040 con 0x55fe87938d20 2023-10-05T19:28:37.382 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.374+0000 7f0ff0633e40 10 read 3730 2023-10-05T19:28:37.382 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.374+0000 7f0ff0633e40 10 log_show_next pos 3730 bl 3730 off 417 eof 1 2023-10-05T19:28:37.382 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.374+0000 7f0ff0633e40 10 log_show_next pos 3730 bl 3730 off 922 eof 1 2023-10-05T19:28:37.382 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.374+0000 7f0ff0633e40 10 log_show_next pos 3730 bl 3730 off 1442 eof 1 2023-10-05T19:28:37.382 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.374+0000 7f0ff0633e40 10 log_show_next pos 3730 bl 3730 off 1947 eof 1 2023-10-05T19:28:37.382 INFO:teuthology.orchestra.run.smithi167.stdout:{ 2023-10-05T19:28:37.383 INFO:teuthology.orchestra.run.smithi167.stdout: "bucket_id": "9a9df965-6612-45d4-a45f-6e3c889bacc2.4223.1", 2023-10-05T19:28:37.383 INFO:teuthology.orchestra.run.smithi167.stdout: "bucket_owner": "foo", 2023-10-05T19:28:37.383 INFO:teuthology.orchestra.run.smithi167.stdout: "bucket": "myfoo", 2023-10-05T19:28:37.383 INFO:teuthology.orchestra.run.smithi167.stdout: "log_entries": [ 2023-10-05T19:28:37.383 INFO:teuthology.orchestra.run.smithi167.stdout: { 2023-10-05T19:28:37.383 INFO:teuthology.orchestra.run.smithi167.stdout: "bucket": "myfoo", 2023-10-05T19:28:37.383 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2023-10-05T19:28:30.246289Z", 2023-10-05T19:28:37.383 INFO:teuthology.orchestra.run.smithi167.stdout: "time_local": "2023-10-05T19:28:30.246289+0000", 2023-10-05T19:28:37.383 INFO:teuthology.orchestra.run.smithi167.stdout: "remote_addr": "172.21.0.51", 2023-10-05T19:28:37.383 INFO:teuthology.orchestra.run.smithi167.stdout: "user": "foo", 2023-10-05T19:28:37.383 INFO:teuthology.orchestra.run.smithi167.stdout: "operation": "create_bucket", 2023-10-05T19:28:37.384 INFO:teuthology.orchestra.run.smithi167.stdout: "uri": "PUT /myfoo/ HTTP/1.1", 2023-10-05T19:28:37.384 INFO:teuthology.orchestra.run.smithi167.stdout: "http_status": "200", 2023-10-05T19:28:37.384 INFO:teuthology.orchestra.run.smithi167.stdout: "error_code": "", 2023-10-05T19:28:37.384 INFO:teuthology.orchestra.run.smithi167.stdout: "bytes_sent": 0, 2023-10-05T19:28:37.384 INFO:teuthology.orchestra.run.smithi167.stdout: "bytes_received": 0, 2023-10-05T19:28:37.384 INFO:teuthology.orchestra.run.smithi167.stdout: "object_size": 0, 2023-10-05T19:28:37.384 INFO:teuthology.orchestra.run.smithi167.stdout: "total_time": 19, 2023-10-05T19:28:37.384 INFO:teuthology.orchestra.run.smithi167.stdout: "user_agent": "Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic", 2023-10-05T19:28:37.384 INFO:teuthology.orchestra.run.smithi167.stdout: "referrer": "", 2023-10-05T19:28:37.384 INFO:teuthology.orchestra.run.smithi167.stdout: "trans_id": "tx00000b440a193ad5ff537-00651f0e5e-107d-default", 2023-10-05T19:28:37.384 INFO:teuthology.orchestra.run.smithi167.stdout: "authentication_type": "Local", 2023-10-05T19:28:37.384 INFO:teuthology.orchestra.run.smithi167.stdout: "access_key_id": "9te6NH5mcdcq0Tc5i8i1", 2023-10-05T19:28:37.385 INFO:teuthology.orchestra.run.smithi167.stdout: "temp_url": false 2023-10-05T19:28:37.385 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2023-10-05T19:28:37.385 INFO:teuthology.orchestra.run.smithi167.stdout: { 2023-10-05T19:28:37.385 INFO:teuthology.orchestra.run.smithi167.stdout: "bucket": "myfoo", 2023-10-05T19:28:37.385 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2023-10-05T19:28:31.546259Z", 2023-10-05T19:28:37.385 INFO:teuthology.orchestra.run.smithi167.stdout: "time_local": "2023-10-05T19:28:31.546259+0000", 2023-10-05T19:28:37.385 INFO:teuthology.orchestra.run.smithi167.stdout: "remote_addr": "172.21.0.51", 2023-10-05T19:28:37.385 INFO:teuthology.orchestra.run.smithi167.stdout: "user": "foo", 2023-10-05T19:28:37.385 INFO:teuthology.orchestra.run.smithi167.stdout: "operation": "put_obj", 2023-10-05T19:28:37.385 INFO:teuthology.orchestra.run.smithi167.stdout: "uri": "PUT /myfoo/f97c5d29941bfb1b2fdab0874906ab82 HTTP/1.1", 2023-10-05T19:28:37.385 INFO:teuthology.orchestra.run.smithi167.stdout: "http_status": "200", 2023-10-05T19:28:37.385 INFO:teuthology.orchestra.run.smithi167.stdout: "error_code": "", 2023-10-05T19:28:37.386 INFO:teuthology.orchestra.run.smithi167.stdout: "bytes_sent": 0, 2023-10-05T19:28:37.386 INFO:teuthology.orchestra.run.smithi167.stdout: "bytes_received": 3, 2023-10-05T19:28:37.386 INFO:teuthology.orchestra.run.smithi167.stdout: "object_size": 3, 2023-10-05T19:28:37.386 INFO:teuthology.orchestra.run.smithi167.stdout: "total_time": 11, 2023-10-05T19:28:37.386 INFO:teuthology.orchestra.run.smithi167.stdout: "user_agent": "Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic", 2023-10-05T19:28:37.386 INFO:teuthology.orchestra.run.smithi167.stdout: "referrer": "", 2023-10-05T19:28:37.386 INFO:teuthology.orchestra.run.smithi167.stdout: "trans_id": "tx00000d5fe88a59b839c54-00651f0e5f-107d-default", 2023-10-05T19:28:37.386 INFO:teuthology.orchestra.run.smithi167.stdout: "authentication_type": "Local", 2023-10-05T19:28:37.386 INFO:teuthology.orchestra.run.smithi167.stdout: "access_key_id": "9te6NH5mcdcq0Tc5i8i1", 2023-10-05T19:28:37.386 INFO:teuthology.orchestra.run.smithi167.stdout: "temp_url": false 2023-10-05T19:28:37.386 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2023-10-05T19:28:37.386 INFO:teuthology.orchestra.run.smithi167.stdout: { 2023-10-05T19:28:37.387 INFO:teuthology.orchestra.run.smithi167.stdout: "bucket": "myfoo", 2023-10-05T19:28:37.387 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2023-10-05T19:28:32.030248Z", 2023-10-05T19:28:37.387 INFO:teuthology.orchestra.run.smithi167.stdout: "time_local": "2023-10-05T19:28:32.030248+0000", 2023-10-05T19:28:37.387 INFO:teuthology.orchestra.run.smithi167.stdout: "remote_addr": "172.21.0.51", 2023-10-05T19:28:37.387 INFO:teuthology.orchestra.run.smithi167.stdout: "user": "foo", 2023-10-05T19:28:37.387 INFO:teuthology.orchestra.run.smithi167.stdout: "operation": "delete_obj", 2023-10-05T19:28:37.387 INFO:teuthology.orchestra.run.smithi167.stdout: "uri": "DELETE /myfoo/f97c5d29941bfb1b2fdab0874906ab82 HTTP/1.1", 2023-10-05T19:28:37.387 INFO:teuthology.orchestra.run.smithi167.stdout: "http_status": "204", 2023-10-05T19:28:37.387 INFO:teuthology.orchestra.run.smithi167.stdout: "error_code": "NoContent", 2023-10-05T19:28:37.387 INFO:teuthology.orchestra.run.smithi167.stdout: "bytes_sent": 0, 2023-10-05T19:28:37.387 INFO:teuthology.orchestra.run.smithi167.stdout: "bytes_received": 0, 2023-10-05T19:28:37.387 INFO:teuthology.orchestra.run.smithi167.stdout: "object_size": 0, 2023-10-05T19:28:37.388 INFO:teuthology.orchestra.run.smithi167.stdout: "total_time": 11, 2023-10-05T19:28:37.388 INFO:teuthology.orchestra.run.smithi167.stdout: "user_agent": "Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic", 2023-10-05T19:28:37.388 INFO:teuthology.orchestra.run.smithi167.stdout: "referrer": "", 2023-10-05T19:28:37.388 INFO:teuthology.orchestra.run.smithi167.stdout: "trans_id": "tx000005389497466473575-00651f0e60-107d-default", 2023-10-05T19:28:37.388 INFO:teuthology.orchestra.run.smithi167.stdout: "authentication_type": "Local", 2023-10-05T19:28:37.388 INFO:teuthology.orchestra.run.smithi167.stdout: "access_key_id": "9te6NH5mcdcq0Tc5i8i1", 2023-10-05T19:28:37.388 INFO:teuthology.orchestra.run.smithi167.stdout: "temp_url": false 2023-10-05T19:28:37.388 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2023-10-05T19:28:37.388 INFO:teuthology.orchestra.run.smithi167.stdout: { 2023-10-05T19:28:37.388 INFO:teuthology.orchestra.run.smithi167.stdout: "bucket": "myfoo", 2023-10-05T19:28:37.388 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2023-10-05T19:28:32.530237Z", 2023-10-05T19:28:37.388 INFO:teuthology.orchestra.run.smithi167.stdout: "time_local": "2023-10-05T19:28:32.530237+0000", 2023-10-05T19:28:37.389 INFO:teuthology.orchestra.run.smithi167.stdout: "remote_addr": "172.21.0.51", 2023-10-05T19:28:37.389 INFO:teuthology.orchestra.run.smithi167.stdout: "user": "foo", 2023-10-05T19:28:37.389 INFO:teuthology.orchestra.run.smithi167.stdout: "operation": "put_obj", 2023-10-05T19:28:37.389 INFO:teuthology.orchestra.run.smithi167.stdout: "uri": "PUT /myfoo/b8a9f715dbb64fd5c56e7783c6820a61 HTTP/1.1", 2023-10-05T19:28:37.389 INFO:teuthology.orchestra.run.smithi167.stdout: "http_status": "200", 2023-10-05T19:28:37.389 INFO:teuthology.orchestra.run.smithi167.stdout: "error_code": "", 2023-10-05T19:28:37.389 INFO:teuthology.orchestra.run.smithi167.stdout: "bytes_sent": 0, 2023-10-05T19:28:37.389 INFO:teuthology.orchestra.run.smithi167.stdout: "bytes_received": 3, 2023-10-05T19:28:37.389 INFO:teuthology.orchestra.run.smithi167.stdout: "object_size": 3, 2023-10-05T19:28:37.389 INFO:teuthology.orchestra.run.smithi167.stdout: "total_time": 7, 2023-10-05T19:28:37.389 INFO:teuthology.orchestra.run.smithi167.stdout: "user_agent": "Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic", 2023-10-05T19:28:37.389 INFO:teuthology.orchestra.run.smithi167.stdout: "referrer": "", 2023-10-05T19:28:37.389 INFO:teuthology.orchestra.run.smithi167.stdout: "trans_id": "tx000002f93fed07c2d0dde-00651f0e60-107d-default", 2023-10-05T19:28:37.390 INFO:teuthology.orchestra.run.smithi167.stdout: "authentication_type": "Local", 2023-10-05T19:28:37.390 INFO:teuthology.orchestra.run.smithi167.stdout: "access_key_id": "9te6NH5mcdcq0Tc5i8i1", 2023-10-05T19:28:37.390 INFO:teuthology.orchestra.run.smithi167.stdout: "temp_url": false 2023-10-05T19:28:37.390 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2023-10-05T19:28:37.390 INFO:teuthology.orchestra.run.smithi167.stdout: { 2023-10-05T19:28:37.390 INFO:teuthology.orchestra.run.smithi167.stdout: "bucket": "myfoo", 2023-10-05T19:28:37.390 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2023-10-05T19:28:32.542236Z", 2023-10-05T19:28:37.390 INFO:teuthology.orchestra.run.smithi167.stdout: "time_local": "2023-10-05T19:28:32.542236+0000", 2023-10-05T19:28:37.390 INFO:teuthology.orchestra.run.smithi167.stdout: "remote_addr": "172.21.0.51", 2023-10-05T19:28:37.390 INFO:teuthology.orchestra.run.smithi167.stdout: "user": "foo", 2023-10-05T19:28:37.390 INFO:teuthology.orchestra.run.smithi167.stdout: "operation": "delete_obj", 2023-10-05T19:28:37.391 INFO:teuthology.orchestra.run.smithi167.stdout: "uri": "DELETE /myfoo/b8a9f715dbb64fd5c56e7783c6820a61 HTTP/1.1", 2023-10-05T19:28:37.391 INFO:teuthology.orchestra.run.smithi167.stdout: "http_status": "204", 2023-10-05T19:28:37.391 INFO:teuthology.orchestra.run.smithi167.stdout: "error_code": "NoContent", 2023-10-05T19:28:37.391 INFO:teuthology.orchestra.run.smithi167.stdout: "bytes_sent": 0, 2023-10-05T19:28:37.391 INFO:teuthology.orchestra.run.smithi167.stdout: "bytes_received": 0, 2023-10-05T19:28:37.391 INFO:teuthology.orchestra.run.smithi167.stdout: "object_size": 0, 2023-10-05T19:28:37.391 INFO:teuthology.orchestra.run.smithi167.stdout: "total_time": 7, 2023-10-05T19:28:37.391 INFO:teuthology.orchestra.run.smithi167.stdout: "user_agent": "Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic", 2023-10-05T19:28:37.391 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.374+0000 7f0ff0633e40 10 log_show_next pos 3730 bl 3730 off 2467 eof 1 2023-10-05T19:28:37.391 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.374+0000 7f0ff0633e40 10 log_show_next pos 3730 bl 3730 off 2888 eof 1 2023-10-05T19:28:37.392 INFO:teuthology.orchestra.run.smithi167.stdout: "referrer": "", 2023-10-05T19:28:37.392 INFO:teuthology.orchestra.run.smithi167.stdout: "trans_id": "tx00000487a097d662347b7-00651f0e60-107d-default", 2023-10-05T19:28:37.392 INFO:teuthology.orchestra.run.smithi167.stdout: "authentication_type": "Local", 2023-10-05T19:28:37.392 INFO:teuthology.orchestra.run.smithi167.stdout: "access_key_id": "9te6NH5mcdcq0Tc5i8i1", 2023-10-05T19:28:37.392 INFO:teuthology.orchestra.run.smithi167.stdout: "temp_url": false 2023-10-05T19:28:37.392 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2023-10-05T19:28:37.392 INFO:teuthology.orchestra.run.smithi167.stdout: { 2023-10-05T19:28:37.392 INFO:teuthology.orchestra.run.smithi167.stdout: "bucket": "myfoo", 2023-10-05T19:28:37.392 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2023-10-05T19:28:35.438170Z", 2023-10-05T19:28:37.393 INFO:teuthology.orchestra.run.smithi167.stdout: "time_local": "2023-10-05T19:28:35.438170+0000", 2023-10-05T19:28:37.393 INFO:teuthology.orchestra.run.smithi167.stdout: "remote_addr": "172.21.0.51", 2023-10-05T19:28:37.393 INFO:teuthology.orchestra.run.smithi167.stdout: "user": "foo", 2023-10-05T19:28:37.393 INFO:teuthology.orchestra.run.smithi167.stdout: "operation": "put_obj", 2023-10-05T19:28:37.393 INFO:teuthology.orchestra.run.smithi167.stdout: "uri": "PUT /myfoo/four HTTP/1.1", 2023-10-05T19:28:37.393 INFO:teuthology.orchestra.run.smithi167.stdout: "http_status": "200", 2023-10-05T19:28:37.393 INFO:teuthology.orchestra.run.smithi167.stdout: "error_code": "", 2023-10-05T19:28:37.393 INFO:teuthology.orchestra.run.smithi167.stdout: "bytes_sent": 0, 2023-10-05T19:28:37.393 INFO:teuthology.orchestra.run.smithi167.stdout: "bytes_received": 4, 2023-10-05T19:28:37.393 INFO:teuthology.orchestra.run.smithi167.stdout: "object_size": 4, 2023-10-05T19:28:37.393 INFO:teuthology.orchestra.run.smithi167.stdout: "total_time": 7, 2023-10-05T19:28:37.393 INFO:teuthology.orchestra.run.smithi167.stdout: "user_agent": "Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic", 2023-10-05T19:28:37.394 INFO:teuthology.orchestra.run.smithi167.stdout: "referrer": "", 2023-10-05T19:28:37.394 INFO:teuthology.orchestra.run.smithi167.stdout: "trans_id": "tx00000cbe01a109c215be1-00651f0e63-107d-default", 2023-10-05T19:28:37.394 INFO:teuthology.orchestra.run.smithi167.stdout: "authentication_type": "Local", 2023-10-05T19:28:37.394 INFO:teuthology.orchestra.run.smithi167.stdout: "access_key_id": "9te6NH5mcdcq0Tc5i8i1", 2023-10-05T19:28:37.394 INFO:teuthology.orchestra.run.smithi167.stdout: "temp_url": false 2023-10-05T19:28:37.394 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2023-10-05T19:28:37.394 INFO:teuthology.orchestra.run.smithi167.stdout: { 2023-10-05T19:28:37.394 INFO:teuthology.orchestra.run.smithi167.stdout: "bucket": "myfoo", 2023-10-05T19:28:37.394 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2023-10-05T19:28:35.450169Z", 2023-10-05T19:28:37.394 INFO:teuthology.orchestra.run.smithi167.stdout: "time_local": "2023-10-05T19:28:35.450169+0000", 2023-10-05T19:28:37.394 INFO:teuthology.orchestra.run.smithi167.stdout: "remote_addr": "172.21.0.51", 2023-10-05T19:28:37.394 INFO:teuthology.orchestra.run.smithi167.stdout: "object_owner": "foo", 2023-10-05T19:28:37.395 INFO:teuthology.orchestra.run.smithi167.stdout: "user": "foo", 2023-10-05T19:28:37.395 INFO:teuthology.orchestra.run.smithi167.stdout: "operation": "get_obj", 2023-10-05T19:28:37.395 INFO:teuthology.orchestra.run.smithi167.stdout: "uri": "GET /myfoo/four HTTP/1.1", 2023-10-05T19:28:37.395 INFO:teuthology.orchestra.run.smithi167.stdout: "http_status": "200", 2023-10-05T19:28:37.395 INFO:teuthology.orchestra.run.smithi167.stdout: "error_code": "", 2023-10-05T19:28:37.395 INFO:teuthology.orchestra.run.smithi167.stdout: "bytes_sent": 4, 2023-10-05T19:28:37.395 INFO:teuthology.orchestra.run.smithi167.stdout: "bytes_received": 0, 2023-10-05T19:28:37.395 INFO:teuthology.orchestra.run.smithi167.stdout: "object_size": 4, 2023-10-05T19:28:37.395 INFO:teuthology.orchestra.run.smithi167.stdout: "total_time": 0, 2023-10-05T19:28:37.395 INFO:teuthology.orchestra.run.smithi167.stdout: "user_agent": "Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic", 2023-10-05T19:28:37.395 INFO:teuthology.orchestra.run.smithi167.stdout: "referrer": "", 2023-10-05T19:28:37.395 INFO:teuthology.orchestra.run.smithi167.stdout: "trans_id": "tx00000a9c154c28044c309-00651f0e63-107d-default", 2023-10-05T19:28:37.395 INFO:teuthology.orchestra.run.smithi167.stdout: "authentication_type": "Local", 2023-10-05T19:28:37.396 INFO:teuthology.orchestra.run.smithi167.stdout: "access_key_id": "9te6NH5mcdcq0Tc5i8i1", 2023-10-05T19:28:37.396 INFO:teuthology.orchestra.run.smithi167.stdout: "temp_url": false 2023-10-05T19:28:37.396 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2023-10-05T19:28:37.396 INFO:teuthology.orchestra.run.smithi167.stdout: { 2023-10-05T19:28:37.396 INFO:teuthology.orchestra.run.smithi167.stdout: "bucket": "myfoo", 2023-10-05T19:28:37.396 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2023-10-05T19:28:35.454169Z", 2023-10-05T19:28:37.396 INFO:teuthology.orchestra.run.smithi167.stdout: "time_local": "2023-10-05T19:28:35.454169+0000", 2023-10-05T19:28:37.396 INFO:teuthology.orchestra.run.smithi167.stdout: "remote_addr": "172.21.0.51", 2023-10-05T19:28:37.396 INFO:teuthology.orchestra.run.smithi167.stdout: "user": "foo", 2023-10-05T19:28:37.396 INFO:teuthology.orchestra.run.smithi167.stdout: "operation": "list_bucket", 2023-10-05T19:28:37.396 INFO:teuthology.orchestra.run.smithi167.stdout: "uri": "GET /myfoo/ HTTP/1.1", 2023-10-05T19:28:37.396 INFO:teuthology.orchestra.run.smithi167.stdout: "http_status": "200", 2023-10-05T19:28:37.397 INFO:teuthology.orchestra.run.smithi167.stdout: "error_code": "", 2023-10-05T19:28:37.397 INFO:teuthology.orchestra.run.smithi167.stdout: "bytes_sent": 515, 2023-10-05T19:28:37.397 INFO:teuthology.orchestra.run.smithi167.stdout: "bytes_received": 0, 2023-10-05T19:28:37.397 INFO:teuthology.orchestra.run.smithi167.stdout: "object_size": 0, 2023-10-05T19:28:37.397 INFO:teuthology.orchestra.run.smithi167.stdout: "total_time": 3, 2023-10-05T19:28:37.397 INFO:teuthology.orchestra.run.smithi167.stdout: "user_agent": "Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic", 2023-10-05T19:28:37.397 INFO:teuthology.orchestra.run.smithi167.stdout: "referrer": "", 2023-10-05T19:28:37.397 INFO:teuthology.orchestra.run.smithi167.stdout: "trans_id": "tx000005252a93a187ef3cd-00651f0e63-107d-default", 2023-10-05T19:28:37.397 INFO:teuthology.orchestra.run.smithi167.stdout: "authentication_type": "Local", 2023-10-05T19:28:37.397 INFO:teuthology.orchestra.run.smithi167.stdout: "access_key_id": "9te6NH5mcdcq0Tc5i8i1", 2023-10-05T19:28:37.397 INFO:teuthology.orchestra.run.smithi167.stdout: "temp_url": false 2023-10-05T19:28:37.397 INFO:teuthology.orchestra.run.smithi167.stdout: } 2023-10-05T19:28:37.398 INFO:teuthology.orchestra.run.smithi167.stdout: ], 2023-10-05T19:28:37.398 INFO:teuthology.orchestra.run.smithi167.stdout: "log_sum": { 2023-10-05T19:28:37.398 INFO:teuthology.orchestra.run.smithi167.stdout: "bytes_sent": 519, 2023-10-05T19:28:37.398 INFO:teuthology.orchestra.run.smithi167.stdout: "bytes_received": 10, 2023-10-05T19:28:37.398 INFO:teuthology.orchestra.run.smithi167.stdout: "total_time": 65, 2023-10-05T19:28:37.398 INFO:teuthology.orchestra.run.smithi167.stdout: "total_entries": 8 2023-10-05T19:28:37.398 INFO:teuthology.orchestra.run.smithi167.stdout: } 2023-10-05T19:28:37.398 INFO:teuthology.orchestra.run.smithi167.stdout:} 2023-10-05T19:28:37.398 INFO:teuthology.orchestra.run.smithi167.stdout: 2023-10-05T19:28:37.400 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.374+0000 7f0ff0633e40 10 log_show_next pos 3730 bl 3730 off 3315 eof 1 2023-10-05T19:28:37.400 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.374+0000 7f0ff0633e40 10 log_show_next pos 3730 bl 3730 off 3730 eof 1 2023-10-05T19:28:37.400 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.378+0000 7f0ff0633e40 1 -- 172.21.15.167:0/1589630579 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:102 7.1f 7:f95f44c2:::notify.0:head [watch unwatch cookie 94551684121728] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x55fe878f8350 con 0x7f0fe8038700 2023-10-05T19:28:37.400 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.378+0000 7f0fed967700 1 -- 172.21.15.167:0/1589630579 <== osd.2 v2:172.21.15.167:6816/30037 17 ==== osd_op_reply(102 notify.0 [watch unwatch cookie 94551684121728] v30'167 uv165 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f0fd8006070 con 0x7f0fe8038700 2023-10-05T19:28:37.401 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.378+0000 7f0ff0633e40 20 remove_watcher() i=0 2023-10-05T19:28:37.401 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.378+0000 7f0ff0633e40 2 removed watcher, disabling cache 2023-10-05T19:28:37.401 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.378+0000 7f0ff0633e40 1 -- 172.21.15.167:0/1589630579 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:103 7.0 7:05bf5b68:::notify.1:head [watch unwatch cookie 94551684126608] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x55fe878b8480 con 0x55fe8787c7e0 2023-10-05T19:28:37.401 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.382+0000 7f0fee969700 1 -- 172.21.15.167:0/1589630579 <== osd.1 v2:172.21.15.167:6800/30035 35 ==== osd_op_reply(103 notify.1 [watch unwatch cookie 94551684126608] v30'167 uv165 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f0fe0050040 con 0x55fe8787c7e0 2023-10-05T19:28:37.401 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.382+0000 7f0ff0633e40 20 remove_watcher() i=1 2023-10-05T19:28:37.401 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.382+0000 7f0ff0633e40 1 -- 172.21.15.167:0/1589630579 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:104 7.15 7:a93a5511:::notify.2:head [watch unwatch cookie 94551684128608] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x55fe878b8480 con 0x55fe878b3130 2023-10-05T19:28:37.401 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.386+0000 7f0fee168700 1 -- 172.21.15.167:0/1589630579 <== osd.3 v2:172.21.15.167:6808/30036 24 ==== osd_op_reply(104 notify.2 [watch unwatch cookie 94551684128608] v30'167 uv165 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f0fe804e040 con 0x55fe878b3130 2023-10-05T19:28:37.401 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.386+0000 7f0ff0633e40 20 remove_watcher() i=2 2023-10-05T19:28:37.401 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.386+0000 7f0ff0633e40 1 -- 172.21.15.167:0/1589630579 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:105 7.e 7:7759931f:::notify.3:head [watch unwatch cookie 94551684133024] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x55fe878b9f60 con 0x7f0fe8038700 2023-10-05T19:28:37.401 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.386+0000 7f0fed967700 1 -- 172.21.15.167:0/1589630579 <== osd.2 v2:172.21.15.167:6816/30037 18 ==== osd_op_reply(105 notify.3 [watch unwatch cookie 94551684133024] v30'167 uv165 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f0fd8006070 con 0x7f0fe8038700 2023-10-05T19:28:37.401 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.386+0000 7f0ff0633e40 20 remove_watcher() i=3 2023-10-05T19:28:37.402 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.386+0000 7f0ff0633e40 1 -- 172.21.15.167:0/1589630579 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:106 7.d 7:b4812045:::notify.4:head [watch unwatch cookie 94551684135232] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x55fe878b8480 con 0x55fe8787c7e0 2023-10-05T19:28:37.402 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.390+0000 7f0fee969700 1 -- 172.21.15.167:0/1589630579 <== osd.1 v2:172.21.15.167:6800/30035 36 ==== osd_op_reply(106 notify.4 [watch unwatch cookie 94551684135232] v30'167 uv165 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f0fe0050040 con 0x55fe8787c7e0 2023-10-05T19:28:37.402 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.390+0000 7f0ff0633e40 20 remove_watcher() i=4 2023-10-05T19:28:37.402 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.390+0000 7f0ff0633e40 1 -- 172.21.15.167:0/1589630579 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:107 7.3 7:c609908c:::notify.5:head [watch unwatch cookie 94551684129712] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x55fe878bb0a0 con 0x7f0fd800d8e0 2023-10-05T19:28:37.402 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.394+0000 7f0fee969700 1 -- 172.21.15.167:0/1589630579 <== osd.0 v2:172.21.15.118:6800/27690 29 ==== osd_op_reply(107 notify.5 [watch unwatch cookie 94551684129712] v30'167 uv165 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f0fe0050040 con 0x7f0fd800d8e0 2023-10-05T19:28:37.402 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.394+0000 7f0ff0633e40 20 remove_watcher() i=5 2023-10-05T19:28:37.402 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.394+0000 7f0ff0633e40 1 -- 172.21.15.167:0/1589630579 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:108 7.14 7:2b04a3e9:::notify.6:head [watch unwatch cookie 94551684140736] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x55fe878bb940 con 0x7f0fd800d8e0 2023-10-05T19:28:37.403 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.398+0000 7f0fee969700 1 -- 172.21.15.167:0/1589630579 <== osd.0 v2:172.21.15.118:6800/27690 30 ==== osd_op_reply(108 notify.6 [watch unwatch cookie 94551684140736] v30'167 uv165 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f0fe0050040 con 0x7f0fd800d8e0 2023-10-05T19:28:37.403 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.398+0000 7f0ff0633e40 20 remove_watcher() i=6 2023-10-05T19:28:37.403 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.398+0000 7f0ff0633e40 1 -- 172.21.15.167:0/1589630579 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:109 7.9 7:93e5b521:::notify.7:head [watch unwatch cookie 94551684145600] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x55fe878b8480 con 0x7f0fd800d8e0 2023-10-05T19:28:37.406 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.398+0000 7f0fee969700 1 -- 172.21.15.167:0/1589630579 <== osd.0 v2:172.21.15.118:6800/27690 31 ==== osd_op_reply(109 notify.7 [watch unwatch cookie 94551684145600] v30'167 uv165 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f0fe0050040 con 0x7f0fd800d8e0 2023-10-05T19:28:37.406 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.398+0000 7f0ff0633e40 20 remove_watcher() i=7 2023-10-05T19:28:37.408 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.402+0000 7f0fbefed700 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2023-10-05T19:28:37.408 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.402+0000 7f0ff0633e40 1 -- 172.21.15.167:0/1589630579 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:110 6.4 6:22d26bf9:::data_loggenerations_metadata:head [watch unwatch cookie 94551684146704] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x55fe878faad0 con 0x55fe8787c7e0 2023-10-05T19:28:37.411 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.406+0000 7f0fee969700 1 -- 172.21.15.167:0/1589630579 <== osd.1 v2:172.21.15.167:6800/30035 37 ==== osd_op_reply(110 data_loggenerations_metadata [watch unwatch cookie 94551684146704] v30'158 uv1 ondisk = 0) v8 ==== 172+0+0 (crc 0 0 0) 0x7f0fe0050040 con 0x55fe8787c7e0 2023-10-05T19:28:37.411 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.406+0000 7f0ff0633e40 1 -- 172.21.15.167:0/1589630579 >> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] conn(0x7f0fd800d8e0 msgr2=0x7f0fd800c360 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-05T19:28:37.411 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.406+0000 7f0ff0633e40 1 --2- 172.21.15.167:0/1589630579 >> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] conn(0x7f0fd800d8e0 0x7f0fd800c360 crc :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-05T19:28:37.412 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.406+0000 7f0ff0633e40 1 -- 172.21.15.167:0/1589630579 >> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] conn(0x55fe8787c7e0 msgr2=0x55fe878af690 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-05T19:28:37.412 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.406+0000 7f0ff0633e40 1 --2- 172.21.15.167:0/1589630579 >> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] conn(0x55fe8787c7e0 0x55fe878af690 crc :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-05T19:28:37.412 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.406+0000 7f0ff0633e40 1 -- 172.21.15.167:0/1589630579 >> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] conn(0x7f0fe8038700 msgr2=0x7f0fe802f860 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-05T19:28:37.412 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.406+0000 7f0ff0633e40 1 --2- 172.21.15.167:0/1589630579 >> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] conn(0x7f0fe8038700 0x7f0fe802f860 crc :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-05T19:28:37.412 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.406+0000 7f0ff0633e40 1 -- 172.21.15.167:0/1589630579 >> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] conn(0x55fe878b3130 msgr2=0x55fe878b5510 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-05T19:28:37.412 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.406+0000 7f0ff0633e40 1 --2- 172.21.15.167:0/1589630579 >> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] conn(0x55fe878b3130 0x55fe878b5510 crc :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-05T19:28:37.412 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.406+0000 7f0ff0633e40 1 -- 172.21.15.167:0/1589630579 >> [v2:172.21.15.167:6824/29889,v1:172.21.15.167:6825/29889] conn(0x7f0fd4030c00 msgr2=0x7f0fd40330b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-05T19:28:37.412 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.406+0000 7f0ff0633e40 1 --2- 172.21.15.167:0/1589630579 >> [v2:172.21.15.167:6824/29889,v1:172.21.15.167:6825/29889] conn(0x7f0fd4030c00 0x7f0fd40330b0 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7f0fd80011e0 tx=0x7f0fd80013a0 comp rx=0 tx=0).stop 2023-10-05T19:28:37.412 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.406+0000 7f0fee969700 1 -- 172.21.15.167:0/1589630579 reap_dead start 2023-10-05T19:28:37.412 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.406+0000 7f0ff0633e40 1 -- 172.21.15.167:0/1589630579 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55fe8787c410 msgr2=0x55fe878a3c00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-05T19:28:37.412 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.406+0000 7f0ff0633e40 1 --2- 172.21.15.167:0/1589630579 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55fe8787c410 0x55fe878a3c00 secure :-1 s=READY pgs=467 cs=0 l=1 rev1=1 crypto rx=0x7f0fe800a520 tx=0x7f0fe801d000 comp rx=0 tx=0).stop 2023-10-05T19:28:37.413 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.406+0000 7f0ff0633e40 1 -- 172.21.15.167:0/1589630579 shutdown_connections 2023-10-05T19:28:37.413 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.406+0000 7f0ff0633e40 1 --2- 172.21.15.167:0/1589630579 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55fe8787c410 0x55fe878a3c00 unknown :-1 s=CLOSED pgs=467 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-05T19:28:37.413 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.406+0000 7f0ff0633e40 1 -- 172.21.15.167:0/1589630579 >> 172.21.15.167:0/1589630579 conn(0x55fe87876720 msgr2=0x55fe878975e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-05T19:28:37.413 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.406+0000 7f0ff0633e40 1 -- 172.21.15.167:0/1589630579 shutdown_connections 2023-10-05T19:28:37.413 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.406+0000 7f0ff0633e40 1 -- 172.21.15.167:0/1589630579 wait complete. 2023-10-05T19:28:37.414 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.406+0000 7f0ff0633e40 1 -- 172.21.15.167:0/3985092605 >> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] conn(0x55fe87938d20 msgr2=0x55fe8793b100 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-05T19:28:37.414 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.406+0000 7f0ff0633e40 1 --2- 172.21.15.167:0/3985092605 >> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] conn(0x55fe87938d20 0x55fe8793b100 crc :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-05T19:28:37.414 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.406+0000 7f0ff0633e40 1 -- 172.21.15.167:0/3985092605 >> [v2:172.21.15.167:6824/29889,v1:172.21.15.167:6825/29889] conn(0x7f0f34030c50 msgr2=0x7f0f34033100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-05T19:28:37.414 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.406+0000 7f0ff0633e40 1 --2- 172.21.15.167:0/3985092605 >> [v2:172.21.15.167:6824/29889,v1:172.21.15.167:6825/29889] conn(0x7f0f34030c50 0x7f0f34033100 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x55fe878bdb50 tx=0x7f0fe0000f90 comp rx=0 tx=0).stop 2023-10-05T19:28:37.414 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.406+0000 7f0ff0633e40 1 -- 172.21.15.167:0/3985092605 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55fe878cbce0 msgr2=0x55fe878d17f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-05T19:28:37.414 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.406+0000 7f0ff0633e40 1 --2- 172.21.15.167:0/3985092605 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55fe878cbce0 0x55fe878d17f0 secure :-1 s=READY pgs=469 cs=0 l=1 rev1=1 crypto rx=0x55fe878b0c20 tx=0x7f0fd802f000 comp rx=0 tx=0).stop 2023-10-05T19:28:37.415 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.406+0000 7f0ff0633e40 1 -- 172.21.15.167:0/3985092605 shutdown_connections 2023-10-05T19:28:37.415 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.406+0000 7f0ff0633e40 1 --2- 172.21.15.167:0/3985092605 >> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] conn(0x55fe87938d20 0x55fe8793b100 unknown :-1 s=CLOSED pgs=77 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-05T19:28:37.415 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.406+0000 7f0ff0633e40 1 --2- 172.21.15.167:0/3985092605 >> [v2:172.21.15.167:6824/29889,v1:172.21.15.167:6825/29889] conn(0x7f0f34030c50 0x7f0f34033100 unknown :-1 s=CLOSED pgs=163 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-05T19:28:37.415 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.406+0000 7f0ff0633e40 1 --2- 172.21.15.167:0/3985092605 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55fe878cbce0 0x55fe878d17f0 unknown :-1 s=CLOSED pgs=469 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-05T19:28:37.415 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.406+0000 7f0ff0633e40 1 -- 172.21.15.167:0/3985092605 >> 172.21.15.167:0/3985092605 conn(0x55fe878c70a0 msgr2=0x55fe878c51a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-05T19:28:37.415 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.406+0000 7f0ff0633e40 1 -- 172.21.15.167:0/3985092605 shutdown_connections 2023-10-05T19:28:37.415 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.406+0000 7f0ff0633e40 1 -- 172.21.15.167:0/3985092605 wait complete. 2023-10-05T19:28:37.418 DEBUG:tasks.util.rgw: json result: {'bucket_id': '9a9df965-6612-45d4-a45f-6e3c889bacc2.4223.1', 'bucket_owner': 'foo', 'bucket': 'myfoo', 'log_entries': [{'bucket': 'myfoo', 'time': '2023-10-05T19:28:30.246289Z', 'time_local': '2023-10-05T19:28:30.246289+0000', 'remote_addr': '172.21.0.51', 'user': 'foo', 'operation': 'create_bucket', 'uri': 'PUT /myfoo/ HTTP/1.1', 'http_status': '200', 'error_code': '', 'bytes_sent': 0, 'bytes_received': 0, 'object_size': 0, 'total_time': 19, 'user_agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'referrer': '', 'trans_id': 'tx00000b440a193ad5ff537-00651f0e5e-107d-default', 'authentication_type': 'Local', 'access_key_id': '9te6NH5mcdcq0Tc5i8i1', 'temp_url': False}, {'bucket': 'myfoo', 'time': '2023-10-05T19:28:31.546259Z', 'time_local': '2023-10-05T19:28:31.546259+0000', 'remote_addr': '172.21.0.51', 'user': 'foo', 'operation': 'put_obj', 'uri': 'PUT /myfoo/f97c5d29941bfb1b2fdab0874906ab82 HTTP/1.1', 'http_status': '200', 'error_code': '', 'bytes_sent': 0, 'bytes_received': 3, 'object_size': 3, 'total_time': 11, 'user_agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'referrer': '', 'trans_id': 'tx00000d5fe88a59b839c54-00651f0e5f-107d-default', 'authentication_type': 'Local', 'access_key_id': '9te6NH5mcdcq0Tc5i8i1', 'temp_url': False}, {'bucket': 'myfoo', 'time': '2023-10-05T19:28:32.030248Z', 'time_local': '2023-10-05T19:28:32.030248+0000', 'remote_addr': '172.21.0.51', 'user': 'foo', 'operation': 'delete_obj', 'uri': 'DELETE /myfoo/f97c5d29941bfb1b2fdab0874906ab82 HTTP/1.1', 'http_status': '204', 'error_code': 'NoContent', 'bytes_sent': 0, 'bytes_received': 0, 'object_size': 0, 'total_time': 11, 'user_agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'referrer': '', 'trans_id': 'tx000005389497466473575-00651f0e60-107d-default', 'authentication_type': 'Local', 'access_key_id': '9te6NH5mcdcq0Tc5i8i1', 'temp_url': False}, {'bucket': 'myfoo', 'time': '2023-10-05T19:28:32.530237Z', 'time_local': '2023-10-05T19:28:32.530237+0000', 'remote_addr': '172.21.0.51', 'user': 'foo', 'operation': 'put_obj', 'uri': 'PUT /myfoo/b8a9f715dbb64fd5c56e7783c6820a61 HTTP/1.1', 'http_status': '200', 'error_code': '', 'bytes_sent': 0, 'bytes_received': 3, 'object_size': 3, 'total_time': 7, 'user_agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'referrer': '', 'trans_id': 'tx000002f93fed07c2d0dde-00651f0e60-107d-default', 'authentication_type': 'Local', 'access_key_id': '9te6NH5mcdcq0Tc5i8i1', 'temp_url': False}, {'bucket': 'myfoo', 'time': '2023-10-05T19:28:32.542236Z', 'time_local': '2023-10-05T19:28:32.542236+0000', 'remote_addr': '172.21.0.51', 'user': 'foo', 'operation': 'delete_obj', 'uri': 'DELETE /myfoo/b8a9f715dbb64fd5c56e7783c6820a61 HTTP/1.1', 'http_status': '204', 'error_code': 'NoContent', 'bytes_sent': 0, 'bytes_received': 0, 'object_size': 0, 'total_time': 7, 'user_agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'referrer': '', 'trans_id': 'tx00000487a097d662347b7-00651f0e60-107d-default', 'authentication_type': 'Local', 'access_key_id': '9te6NH5mcdcq0Tc5i8i1', 'temp_url': False}, {'bucket': 'myfoo', 'time': '2023-10-05T19:28:35.438170Z', 'time_local': '2023-10-05T19:28:35.438170+0000', 'remote_addr': '172.21.0.51', 'user': 'foo', 'operation': 'put_obj', 'uri': 'PUT /myfoo/four HTTP/1.1', 'http_status': '200', 'error_code': '', 'bytes_sent': 0, 'bytes_received': 4, 'object_size': 4, 'total_time': 7, 'user_agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'referrer': '', 'trans_id': 'tx00000cbe01a109c215be1-00651f0e63-107d-default', 'authentication_type': 'Local', 'access_key_id': '9te6NH5mcdcq0Tc5i8i1', 'temp_url': False}, {'bucket': 'myfoo', 'time': '2023-10-05T19:28:35.450169Z', 'time_local': '2023-10-05T19:28:35.450169+0000', 'remote_addr': '172.21.0.51', 'object_owner': 'foo', 'user': 'foo', 'operation': 'get_obj', 'uri': 'GET /myfoo/four HTTP/1.1', 'http_status': '200', 'error_code': '', 'bytes_sent': 4, 'bytes_received': 0, 'object_size': 4, 'total_time': 0, 'user_agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'referrer': '', 'trans_id': 'tx00000a9c154c28044c309-00651f0e63-107d-default', 'authentication_type': 'Local', 'access_key_id': '9te6NH5mcdcq0Tc5i8i1', 'temp_url': False}, {'bucket': 'myfoo', 'time': '2023-10-05T19:28:35.454169Z', 'time_local': '2023-10-05T19:28:35.454169+0000', 'remote_addr': '172.21.0.51', 'user': 'foo', 'operation': 'list_bucket', 'uri': 'GET /myfoo/ HTTP/1.1', 'http_status': '200', 'error_code': '', 'bytes_sent': 515, 'bytes_received': 0, 'object_size': 0, 'total_time': 3, 'user_agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'referrer': '', 'trans_id': 'tx000005252a93a187ef3cd-00651f0e63-107d-default', 'authentication_type': 'Local', 'access_key_id': '9te6NH5mcdcq0Tc5i8i1', 'temp_url': False}], 'log_sum': {'bytes_sent': 519, 'bytes_received': 10, 'total_time': 65, 'total_entries': 8}} 2023-10-05T19:28:37.418 DEBUG:tasks.radosgw_admin:checking log entry: 2023-10-05T19:28:37.418 DEBUG:tasks.radosgw_admin:checking log entry: 2023-10-05T19:28:37.418 DEBUG:tasks.radosgw_admin:checking log entry: 2023-10-05T19:28:37.418 DEBUG:tasks.radosgw_admin:checking log entry: 2023-10-05T19:28:37.419 DEBUG:tasks.radosgw_admin:checking log entry: 2023-10-05T19:28:37.419 DEBUG:tasks.radosgw_admin:checking log entry: 2023-10-05T19:28:37.419 DEBUG:tasks.radosgw_admin:checking log entry: 2023-10-05T19:28:37.419 DEBUG:tasks.radosgw_admin:checking log entry: 2023-10-05T19:28:37.419 INFO:tasks.util.rgw:rgwadmin: client.0 : ['log', 'rm', '--object', '2023-10-05-19-9a9df965-6612-45d4-a45f-6e3c889bacc2.4223.1-myfoo'] 2023-10-05T19:28:37.419 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'ceph', 'log', 'rm', '--object', '2023-10-05-19-9a9df965-6612-45d4-a45f-6e3c889bacc2.4223.1-myfoo'] 2023-10-05T19:28:37.419 DEBUG:teuthology.orchestra.run.smithi167:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster ceph log rm --object 2023-10-05-19-9a9df965-6612-45d4-a45f-6e3c889bacc2.4223.1-myfoo 2023-10-05T19:28:37.455 INFO:teuthology.orchestra.run.smithi167.stderr:ignoring --setuser ceph since I am not root 2023-10-05T19:28:37.455 INFO:teuthology.orchestra.run.smithi167.stderr:ignoring --setgroup ceph since I am not root 2023-10-05T19:28:37.456 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.450+0000 7f0fa95f4e40 1 Processor -- start 2023-10-05T19:28:37.456 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.450+0000 7f0fa95f4e40 1 -- start start 2023-10-05T19:28:37.456 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.450+0000 7f0fa95f4e40 1 --2- >> v2:172.21.15.118:3300/0 conn(0x55bcbba67410 0x55bcbba697f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-05T19:28:37.456 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.450+0000 7f0fa95f4e40 1 -- --> v1:172.21.15.118:6789/0 -- auth(proto 0 26 bytes epoch 0) v1 -- 0x55bcbb9afbd0 con 0x55bcbba62e30 2023-10-05T19:28:37.456 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.450+0000 7f0fa95f4e40 1 -- --> v2:172.21.15.118:3300/0 -- mon_getmap magic: 0 v1 -- 0x55bcbb96cee0 con 0x55bcbba67410 2023-10-05T19:28:37.457 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.450+0000 7f0fa6928700 1 --2- >> v2:172.21.15.118:3300/0 conn(0x55bcbba67410 0x55bcbba697f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-05T19:28:37.457 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.450+0000 7f0fa7129700 1 --1- >> v1:172.21.15.118:6789/0 conn(0x55bcbba62e30 0x55bcbba63200 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.118:6789/0 says I am v1:172.21.15.167:59362/0 (socket says 172.21.15.167:59362) 2023-10-05T19:28:37.457 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.450+0000 7f0fa7129700 1 -- 172.21.15.167:0/4075958939 learned_addr learned my addr 172.21.15.167:0/4075958939 (peer_addr_for_me v1:172.21.15.167:0/0) 2023-10-05T19:28:37.457 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.450+0000 7f0fa6928700 1 -- 172.21.15.167:0/4075958939 >> v1:172.21.15.118:6789/0 conn(0x55bcbba62e30 legacy=0x55bcbba63200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-05T19:28:37.457 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.450+0000 7f0fa6928700 1 -- 172.21.15.167:0/4075958939 --> v2:172.21.15.118:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55bcbba63a40 con 0x55bcbba67410 2023-10-05T19:28:37.458 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.450+0000 7f0fa6928700 1 --2- 172.21.15.167:0/4075958939 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55bcbba67410 0x55bcbba697f0 secure :-1 s=READY pgs=471 cs=0 l=1 rev1=1 crypto rx=0x7f0f9000a520 tx=0x7f0f90005d40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4b7d0b33cee1eb6a server_cookie=0 in_seq=0 out_seq=0 2023-10-05T19:28:37.458 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.454+0000 7f0fa6127700 1 -- 172.21.15.167:0/4075958939 <== mon.0 v2:172.21.15.118:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f0f9000f590 con 0x55bcbba67410 2023-10-05T19:28:37.458 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.454+0000 7f0fa6127700 1 -- 172.21.15.167:0/4075958939 <== mon.0 v2:172.21.15.118:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0f9000f6f0 con 0x55bcbba67410 2023-10-05T19:28:37.459 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.454+0000 7f0fa6127700 1 -- 172.21.15.167:0/4075958939 <== mon.0 v2:172.21.15.118:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f0f9000f880 con 0x55bcbba67410 2023-10-05T19:28:37.459 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.454+0000 7f0fa95f4e40 1 -- 172.21.15.167:0/4075958939 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55bcbba67410 msgr2=0x55bcbba697f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-05T19:28:37.459 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.454+0000 7f0fa95f4e40 1 --2- 172.21.15.167:0/4075958939 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55bcbba67410 0x55bcbba697f0 secure :-1 s=READY pgs=471 cs=0 l=1 rev1=1 crypto rx=0x7f0f9000a520 tx=0x7f0f90005d40 comp rx=0 tx=0).stop 2023-10-05T19:28:37.459 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.454+0000 7f0fa95f4e40 1 -- 172.21.15.167:0/4075958939 shutdown_connections 2023-10-05T19:28:37.460 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.454+0000 7f0fa95f4e40 1 --2- 172.21.15.167:0/4075958939 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55bcbba67410 0x55bcbba697f0 unknown :-1 s=CLOSED pgs=471 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-05T19:28:37.460 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.454+0000 7f0fa95f4e40 1 -- 172.21.15.167:0/4075958939 >> 172.21.15.167:0/4075958939 conn(0x55bcbba5e850 msgr2=0x55bcbba60c60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-05T19:28:37.460 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.454+0000 7f0fa95f4e40 1 -- 172.21.15.167:0/4075958939 shutdown_connections 2023-10-05T19:28:37.460 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.454+0000 7f0fa95f4e40 1 -- 172.21.15.167:0/4075958939 wait complete. 2023-10-05T19:28:37.461 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.454+0000 7f0fa95f4e40 1 Processor -- start 2023-10-05T19:28:37.461 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.454+0000 7f0fa95f4e40 1 -- start start 2023-10-05T19:28:37.461 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.454+0000 7f0fa95f4e40 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55bcbba62e30 0x55bcbb9d88e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-05T19:28:37.461 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.454+0000 7f0fa95f4e40 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x55bcbb930e30 con 0x55bcbba62e30 2023-10-05T19:28:37.461 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.454+0000 7f0fa7129700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55bcbba62e30 0x55bcbb9d88e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-05T19:28:37.461 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.454+0000 7f0fa7129700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55bcbba62e30 0x55bcbb9d88e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.118:3300/0 says I am v2:172.21.15.167:42024/0 (socket says 172.21.15.167:42024) 2023-10-05T19:28:37.461 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.454+0000 7f0fa7129700 1 -- 172.21.15.167:0/2292618184 learned_addr learned my addr 172.21.15.167:0/2292618184 (peer_addr_for_me v2:172.21.15.167:0/0) 2023-10-05T19:28:37.462 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.454+0000 7f0fa7129700 1 -- 172.21.15.167:0/2292618184 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0f9000a0e0 con 0x55bcbba62e30 2023-10-05T19:28:37.462 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.454+0000 7f0fa7129700 1 --2- 172.21.15.167:0/2292618184 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55bcbba62e30 0x55bcbb9d88e0 secure :-1 s=READY pgs=472 cs=0 l=1 rev1=1 crypto rx=0x7f0f98001c30 tx=0x7f0f980017e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3a6c03b4a91f708c server_cookie=0 in_seq=0 out_seq=0 2023-10-05T19:28:37.462 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.458+0000 7f0fa5125700 1 -- 172.21.15.167:0/2292618184 <== mon.0 v2:172.21.15.118:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f0f9800bbf0 con 0x55bcbba62e30 2023-10-05T19:28:37.462 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.458+0000 7f0fa5125700 1 -- 172.21.15.167:0/2292618184 <== mon.0 v2:172.21.15.118:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0f9800bd50 con 0x55bcbba62e30 2023-10-05T19:28:37.462 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.458+0000 7f0fa5125700 1 -- 172.21.15.167:0/2292618184 <== mon.0 v2:172.21.15.118:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f0f9800e470 con 0x55bcbba62e30 2023-10-05T19:28:37.464 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.458+0000 7f0fa95f4e40 1 -- 172.21.15.167:0/2292618184 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55bcbba62e30 msgr2=0x55bcbb9d88e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-05T19:28:37.464 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.458+0000 7f0fa95f4e40 1 --2- 172.21.15.167:0/2292618184 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55bcbba62e30 0x55bcbb9d88e0 secure :-1 s=READY pgs=472 cs=0 l=1 rev1=1 crypto rx=0x7f0f98001c30 tx=0x7f0f980017e0 comp rx=0 tx=0).stop 2023-10-05T19:28:37.464 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.458+0000 7f0fa95f4e40 1 -- 172.21.15.167:0/2292618184 shutdown_connections 2023-10-05T19:28:37.464 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.458+0000 7f0fa95f4e40 1 --2- 172.21.15.167:0/2292618184 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55bcbba62e30 0x55bcbb9d88e0 unknown :-1 s=CLOSED pgs=472 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-05T19:28:37.464 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.458+0000 7f0fa95f4e40 1 -- 172.21.15.167:0/2292618184 >> 172.21.15.167:0/2292618184 conn(0x55bcbba5e850 msgr2=0x55bcbba7db50 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-05T19:28:37.464 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.458+0000 7f0fa95f4e40 1 -- 172.21.15.167:0/2292618184 shutdown_connections 2023-10-05T19:28:37.464 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.458+0000 7f0fa95f4e40 1 -- 172.21.15.167:0/2292618184 wait complete. 2023-10-05T19:28:37.464 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.458+0000 7f0fa95f4e40 1 Processor -- start 2023-10-05T19:28:37.464 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.458+0000 7f0fa95f4e40 1 -- start start 2023-10-05T19:28:37.464 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.458+0000 7f0fa95f4e40 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55bcbba62e30 0x55bcbba8af60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-05T19:28:37.465 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.458+0000 7f0fa95f4e40 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x55bcbb903ed0 con 0x55bcbba62e30 2023-10-05T19:28:37.465 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.458+0000 7f0fa7129700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55bcbba62e30 0x55bcbba8af60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-05T19:28:37.465 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.458+0000 7f0fa7129700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55bcbba62e30 0x55bcbba8af60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.118:3300/0 says I am v2:172.21.15.167:42032/0 (socket says 172.21.15.167:42032) 2023-10-05T19:28:37.465 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.458+0000 7f0fa7129700 1 -- 172.21.15.167:0/552831313 learned_addr learned my addr 172.21.15.167:0/552831313 (peer_addr_for_me v2:172.21.15.167:0/0) 2023-10-05T19:28:37.466 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.458+0000 7f0fa7129700 1 -- 172.21.15.167:0/552831313 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55bcbb9afbd0 con 0x55bcbba62e30 2023-10-05T19:28:37.466 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.458+0000 7f0fa7129700 1 --2- 172.21.15.167:0/552831313 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55bcbba62e30 0x55bcbba8af60 secure :-1 s=READY pgs=473 cs=0 l=1 rev1=1 crypto rx=0x7f0f98001820 tx=0x7f0f98016000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-05T19:28:37.466 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.458+0000 7f0f9f7fe700 1 -- 172.21.15.167:0/552831313 <== mon.0 v2:172.21.15.118:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f0f9801f070 con 0x55bcbba62e30 2023-10-05T19:28:37.466 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.458+0000 7f0f9f7fe700 1 -- 172.21.15.167:0/552831313 <== mon.0 v2:172.21.15.118:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0f980197b0 con 0x55bcbba62e30 2023-10-05T19:28:37.466 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.458+0000 7f0fa95f4e40 1 -- 172.21.15.167:0/552831313 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x55bcbba8b4a0 con 0x55bcbba62e30 2023-10-05T19:28:37.466 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.458+0000 7f0f9f7fe700 1 -- 172.21.15.167:0/552831313 <== mon.0 v2:172.21.15.118:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f0f98019960 con 0x55bcbba62e30 2023-10-05T19:28:37.466 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.458+0000 7f0fa95f4e40 1 -- 172.21.15.167:0/552831313 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55bcbba8b8e0 con 0x55bcbba62e30 2023-10-05T19:28:37.467 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.462+0000 7f0fa95f4e40 20 rados->read ofs=0 len=0 2023-10-05T19:28:37.467 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.462+0000 7f0f9f7fe700 1 -- 172.21.15.167:0/552831313 <== mon.0 v2:172.21.15.118:3300/0 4 ==== mgrmap(e 5) v1 ==== 40329+0+0 (secure 0 0 0) 0x7f0f98022020 con 0x55bcbba62e30 2023-10-05T19:28:37.467 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.462+0000 7f0f9f7fe700 1 --2- 172.21.15.167:0/552831313 >> [v2:172.21.15.167:6824/29889,v1:172.21.15.167:6825/29889] conn(0x7f0f88030c00 0x7f0f880330b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-05T19:28:37.468 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.462+0000 7f0f9f7fe700 1 -- 172.21.15.167:0/552831313 <== mon.0 v2:172.21.15.118:3300/0 5 ==== osd_map(30..30 src has 1..30) v4 ==== 6057+0+0 (secure 0 0 0) 0x7f0f98044e80 con 0x55bcbba62e30 2023-10-05T19:28:37.468 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.462+0000 7f0fa6928700 1 --2- 172.21.15.167:0/552831313 >> [v2:172.21.15.167:6824/29889,v1:172.21.15.167:6825/29889] conn(0x7f0f88030c00 0x7f0f880330b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-05T19:28:37.468 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.462+0000 7f0fa95f4e40 1 --2- 172.21.15.167:0/552831313 >> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] conn(0x55bcbba67410 0x55bcbba96a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-05T19:28:37.468 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.462+0000 7f0fa95f4e40 1 -- 172.21.15.167:0/552831313 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:1 5.12 5:49953fa1:::default.realm:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x55bcbba97120 con 0x55bcbba67410 2023-10-05T19:28:37.469 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.462+0000 7f0fa792a700 1 --2- 172.21.15.167:0/552831313 >> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] conn(0x55bcbba67410 0x55bcbba96a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-05T19:28:37.469 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.462+0000 7f0fa6928700 1 --2- 172.21.15.167:0/552831313 >> [v2:172.21.15.167:6824/29889,v1:172.21.15.167:6825/29889] conn(0x7f0f88030c00 0x7f0f880330b0 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7f0f90005d80 tx=0x7f0f90014000 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-05T19:28:37.469 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.462+0000 7f0fa792a700 1 --2- 172.21.15.167:0/552831313 >> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] conn(0x55bcbba67410 0x55bcbba96a20 crc :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-05T19:28:37.470 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.462+0000 7f0fa792a700 1 -- 172.21.15.167:0/552831313 <== osd.1 v2:172.21.15.167:6800/30035 1 ==== osd_op_reply(1 default.realm [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 ==== 157+0+0 (crc 0 0 0) 0x7f0fa000a500 con 0x55bcbba67410 2023-10-05T19:28:37.470 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.466+0000 7f0fa95f4e40 20 rados_obj.operate() r=-2 bl.length=0 2023-10-05T19:28:37.470 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.466+0000 7f0fa95f4e40 20 realm 2023-10-05T19:28:37.470 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.466+0000 7f0fa95f4e40 20 rados->read ofs=0 len=0 2023-10-05T19:28:37.470 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.466+0000 7f0fa95f4e40 1 -- 172.21.15.167:0/552831313 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:2 5.12 5:49953fa1:::default.realm:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x55bcbba94080 con 0x55bcbba67410 2023-10-05T19:28:37.471 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.466+0000 7f0fa792a700 1 -- 172.21.15.167:0/552831313 <== osd.1 v2:172.21.15.167:6800/30035 2 ==== osd_op_reply(2 default.realm [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 ==== 157+0+0 (crc 0 0 0) 0x7f0fa000a500 con 0x55bcbba67410 2023-10-05T19:28:37.471 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.466+0000 7f0fa95f4e40 20 rados_obj.operate() r=-2 bl.length=0 2023-10-05T19:28:37.471 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.466+0000 7f0fa95f4e40 4 RGWPeriod::init failed to init realm id : (2) No such file or directory 2023-10-05T19:28:37.471 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.466+0000 7f0fa95f4e40 20 rados->read ofs=0 len=0 2023-10-05T19:28:37.471 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.466+0000 7f0fa95f4e40 1 -- 172.21.15.167:0/552831313 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:3 5.12 5:49953fa1:::default.realm:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x55bcbba97c00 con 0x55bcbba67410 2023-10-05T19:28:37.472 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.466+0000 7f0fa792a700 1 -- 172.21.15.167:0/552831313 <== osd.1 v2:172.21.15.167:6800/30035 3 ==== osd_op_reply(3 default.realm [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 ==== 157+0+0 (crc 0 0 0) 0x7f0fa000a500 con 0x55bcbba67410 2023-10-05T19:28:37.472 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.466+0000 7f0fa95f4e40 20 rados_obj.operate() r=-2 bl.length=0 2023-10-05T19:28:37.472 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.466+0000 7f0fa95f4e40 20 rados->read ofs=0 len=0 2023-10-05T19:28:37.472 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.466+0000 7f0fa95f4e40 1 -- 172.21.15.167:0/552831313 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:4 5.f 5:f43b6ece:::zone_names.default:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x55bcbba97810 con 0x55bcbba67410 2023-10-05T19:28:37.472 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.466+0000 7f0fa792a700 1 -- 172.21.15.167:0/552831313 <== osd.1 v2:172.21.15.167:6800/30035 4 ==== osd_op_reply(4 zone_names.default [read 0~46 out=46b] v0'0 uv1 ondisk = 0) v8 ==== 162+0+46 (crc 0 0 0) 0x7f0fa000a500 con 0x55bcbba67410 2023-10-05T19:28:37.472 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.466+0000 7f0fa95f4e40 20 rados_obj.operate() r=0 bl.length=46 2023-10-05T19:28:37.472 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.466+0000 7f0fa95f4e40 20 rados->read ofs=0 len=0 2023-10-05T19:28:37.473 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.466+0000 7f0fa95f4e40 1 -- 172.21.15.167:0/552831313 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:5 5.1d 5:b8374eb4:::zone_info.9a9df965-6612-45d4-a45f-6e3c889bacc2:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x55bcbba98a90 con 0x55bcbba67410 2023-10-05T19:28:37.473 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.466+0000 7f0fa792a700 1 -- 172.21.15.167:0/552831313 <== osd.1 v2:172.21.15.167:6800/30035 5 ==== osd_op_reply(5 zone_info.9a9df965-6612-45d4-a45f-6e3c889bacc2 [read 0~874 out=874b] v0'0 uv2 ondisk = 0) v8 ==== 190+0+874 (crc 0 0 0) 0x7f0fa000a500 con 0x55bcbba67410 2023-10-05T19:28:37.473 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.466+0000 7f0fa95f4e40 20 rados_obj.operate() r=0 bl.length=874 2023-10-05T19:28:37.474 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.466+0000 7f0fa95f4e40 20 searching for the correct realm 2023-10-05T19:28:37.474 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.466+0000 7f0fa95f4e40 1 --2- 172.21.15.167:0/552831313 >> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] conn(0x55bcbba9a850 0x55bcbba9cc30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-05T19:28:37.474 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.466+0000 7f0fa95f4e40 1 -- 172.21.15.167:0/552831313 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:6 5.0 5:00000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x55bcbba9d170 con 0x55bcbba9a850 2023-10-05T19:28:37.474 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.466+0000 7f0fa7129700 1 --2- 172.21.15.167:0/552831313 >> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] conn(0x55bcbba9a850 0x55bcbba9cc30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-05T19:28:37.474 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.466+0000 7f0fa7129700 1 --2- 172.21.15.167:0/552831313 >> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] conn(0x55bcbba9a850 0x55bcbba9cc30 crc :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-05T19:28:37.475 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.470+0000 7f0fa7129700 1 -- 172.21.15.167:0/552831313 <== osd.3 v2:172.21.15.167:6808/30036 1 ==== osd_op_reply(6 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f0f98017440 con 0x55bcbba9a850 2023-10-05T19:28:37.476 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.470+0000 7f0fa7129700 1 --2- 172.21.15.167:0/552831313 >> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] conn(0x7f0f98016b40 0x7f0f980243d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-05T19:28:37.476 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.470+0000 7f0fa7129700 1 -- 172.21.15.167:0/552831313 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:7 5.10 5:08000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x55bcbba9d170 con 0x7f0f98016b40 2023-10-05T19:28:37.476 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.470+0000 7f0fa6928700 1 --2- 172.21.15.167:0/552831313 >> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] conn(0x7f0f98016b40 0x7f0f980243d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-05T19:28:37.476 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.470+0000 7f0fa6928700 1 --2- 172.21.15.167:0/552831313 >> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] conn(0x7f0f98016b40 0x7f0f980243d0 crc :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-05T19:28:37.477 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.470+0000 7f0fa6928700 1 -- 172.21.15.167:0/552831313 <== osd.2 v2:172.21.15.167:6816/30037 1 ==== osd_op_reply(7 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f0f9000fca0 con 0x7f0f98016b40 2023-10-05T19:28:37.477 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.470+0000 7f0fa6928700 1 --2- 172.21.15.167:0/552831313 >> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] conn(0x7f0f90029420 0x7f0f90029830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-05T19:28:37.477 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.470+0000 7f0fa6928700 1 -- 172.21.15.167:0/552831313 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:8 5.8 5:10000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x55bcbba9d170 con 0x7f0f90029420 2023-10-05T19:28:37.477 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.470+0000 7f0fa792a700 1 --2- 172.21.15.167:0/552831313 >> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] conn(0x7f0f90029420 0x7f0f90029830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-05T19:28:37.478 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.474+0000 7f0fa792a700 1 --2- 172.21.15.167:0/552831313 >> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] conn(0x7f0f90029420 0x7f0f90029830 crc :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-05T19:28:37.479 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.474+0000 7f0fa792a700 1 -- 172.21.15.167:0/552831313 <== osd.0 v2:172.21.15.118:6800/27690 1 ==== osd_op_reply(8 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f0fa0014040 con 0x7f0f90029420 2023-10-05T19:28:37.479 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.474+0000 7f0fa792a700 1 -- 172.21.15.167:0/552831313 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:9 5.18 5:18000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x55bcbba9d170 con 0x55bcbba67410 2023-10-05T19:28:37.480 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.474+0000 7f0fa792a700 1 -- 172.21.15.167:0/552831313 <== osd.1 v2:172.21.15.167:6800/30035 6 ==== osd_op_reply(9 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f0fa0014040 con 0x55bcbba67410 2023-10-05T19:28:37.480 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.474+0000 7f0fa792a700 1 -- 172.21.15.167:0/552831313 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:10 5.4 5:20000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x55bcbba9d170 con 0x55bcbba67410 2023-10-05T19:28:37.480 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.474+0000 7f0fa792a700 1 -- 172.21.15.167:0/552831313 <== osd.1 v2:172.21.15.167:6800/30035 7 ==== osd_op_reply(10 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f0fa0014040 con 0x55bcbba67410 2023-10-05T19:28:37.480 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.474+0000 7f0fa792a700 1 -- 172.21.15.167:0/552831313 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:11 5.14 5:28000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x55bcbba9d170 con 0x7f0f90029420 2023-10-05T19:28:37.481 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.474+0000 7f0fa792a700 1 -- 172.21.15.167:0/552831313 <== osd.0 v2:172.21.15.118:6800/27690 2 ==== osd_op_reply(11 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f0fa0014040 con 0x7f0f90029420 2023-10-05T19:28:37.481 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.474+0000 7f0fa792a700 1 -- 172.21.15.167:0/552831313 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:12 5.c 5:30000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x55bcbba9d170 con 0x7f0f90029420 2023-10-05T19:28:37.482 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.478+0000 7f0fa792a700 1 -- 172.21.15.167:0/552831313 <== osd.0 v2:172.21.15.118:6800/27690 3 ==== osd_op_reply(12 [pgnls start_epoch 30 out=112b] v23'1 uv1 ondisk = 1) v8 ==== 144+0+112 (crc 0 0 0) 0x7f0fa0014040 con 0x7f0f90029420 2023-10-05T19:28:37.482 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.478+0000 7f0fa792a700 1 -- 172.21.15.167:0/552831313 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:13 5.1c 5:38000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x55bcbba9d170 con 0x7f0f98016b40 2023-10-05T19:28:37.483 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.478+0000 7f0fa6928700 1 -- 172.21.15.167:0/552831313 <== osd.2 v2:172.21.15.167:6816/30037 2 ==== osd_op_reply(13 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f0f9000fca0 con 0x7f0f98016b40 2023-10-05T19:28:37.483 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.478+0000 7f0fa6928700 1 -- 172.21.15.167:0/552831313 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:14 5.2 5:40000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x55bcbba9d170 con 0x55bcbba9a850 2023-10-05T19:28:37.484 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.478+0000 7f0fa7129700 1 -- 172.21.15.167:0/552831313 <== osd.3 v2:172.21.15.167:6808/30036 2 ==== osd_op_reply(14 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f0f98017440 con 0x55bcbba9a850 2023-10-05T19:28:37.484 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.478+0000 7f0fa7129700 1 -- 172.21.15.167:0/552831313 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:15 5.12 5:48000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x55bcbba9d170 con 0x55bcbba67410 2023-10-05T19:28:37.484 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.478+0000 7f0fa792a700 1 -- 172.21.15.167:0/552831313 <== osd.1 v2:172.21.15.167:6800/30035 8 ==== osd_op_reply(15 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f0fa0014040 con 0x55bcbba67410 2023-10-05T19:28:37.485 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.478+0000 7f0fa792a700 1 -- 172.21.15.167:0/552831313 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:16 5.a 5:50000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x55bcbba9d170 con 0x7f0f90029420 2023-10-05T19:28:37.485 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.478+0000 7f0fa792a700 1 -- 172.21.15.167:0/552831313 <== osd.0 v2:172.21.15.118:6800/27690 4 ==== osd_op_reply(16 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f0fa0022040 con 0x7f0f90029420 2023-10-05T19:28:37.486 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.478+0000 7f0fa792a700 1 -- 172.21.15.167:0/552831313 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:17 5.1a 5:58000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x55bcbba9d170 con 0x55bcbba67410 2023-10-05T19:28:37.486 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.482+0000 7f0fa792a700 1 -- 172.21.15.167:0/552831313 <== osd.1 v2:172.21.15.167:6800/30035 9 ==== osd_op_reply(17 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f0fa0022040 con 0x55bcbba67410 2023-10-05T19:28:37.486 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.482+0000 7f0fa792a700 1 -- 172.21.15.167:0/552831313 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:18 5.6 5:60000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x55bcbba9d170 con 0x7f0f98016b40 2023-10-05T19:28:37.487 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.482+0000 7f0fa6928700 1 -- 172.21.15.167:0/552831313 <== osd.2 v2:172.21.15.167:6816/30037 3 ==== osd_op_reply(18 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f0f9000fca0 con 0x7f0f98016b40 2023-10-05T19:28:37.487 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.482+0000 7f0fa6928700 1 -- 172.21.15.167:0/552831313 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:19 5.16 5:68000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x55bcbba9d170 con 0x55bcbba9a850 2023-10-05T19:28:37.488 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.482+0000 7f0fa7129700 1 -- 172.21.15.167:0/552831313 <== osd.3 v2:172.21.15.167:6808/30036 3 ==== osd_op_reply(19 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f0f98017440 con 0x55bcbba9a850 2023-10-05T19:28:37.488 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.482+0000 7f0fa7129700 1 -- 172.21.15.167:0/552831313 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:20 5.e 5:70000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x55bcbba9d170 con 0x55bcbba9a850 2023-10-05T19:28:37.489 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.482+0000 7f0fa7129700 1 -- 172.21.15.167:0/552831313 <== osd.3 v2:172.21.15.167:6808/30036 4 ==== osd_op_reply(20 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f0f98017440 con 0x55bcbba9a850 2023-10-05T19:28:37.489 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.482+0000 7f0fa7129700 1 -- 172.21.15.167:0/552831313 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:21 5.1e 5:78000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x55bcbba9d170 con 0x7f0f90029420 2023-10-05T19:28:37.490 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.486+0000 7f0fa792a700 1 -- 172.21.15.167:0/552831313 <== osd.0 v2:172.21.15.118:6800/27690 5 ==== osd_op_reply(21 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f0fa0022040 con 0x7f0f90029420 2023-10-05T19:28:37.490 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.486+0000 7f0fa792a700 1 -- 172.21.15.167:0/552831313 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:22 5.1 5:80000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x55bcbba9d170 con 0x55bcbba9a850 2023-10-05T19:28:37.491 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.486+0000 7f0fa7129700 1 -- 172.21.15.167:0/552831313 <== osd.3 v2:172.21.15.167:6808/30036 5 ==== osd_op_reply(22 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f0f98017440 con 0x55bcbba9a850 2023-10-05T19:28:37.491 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.486+0000 7f0fa7129700 1 -- 172.21.15.167:0/552831313 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:23 5.11 5:88000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x55bcbba9d170 con 0x55bcbba9a850 2023-10-05T19:28:37.492 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.486+0000 7f0fa7129700 1 -- 172.21.15.167:0/552831313 <== osd.3 v2:172.21.15.167:6808/30036 6 ==== osd_op_reply(23 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f0f98017440 con 0x55bcbba9a850 2023-10-05T19:28:37.492 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.486+0000 7f0fa7129700 1 -- 172.21.15.167:0/552831313 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:24 5.9 5:90000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x55bcbba9d170 con 0x55bcbba67410 2023-10-05T19:28:37.493 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.486+0000 7f0fa792a700 1 -- 172.21.15.167:0/552831313 <== osd.1 v2:172.21.15.167:6800/30035 10 ==== osd_op_reply(24 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f0fa0022040 con 0x55bcbba67410 2023-10-05T19:28:37.493 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.486+0000 7f0fa792a700 1 -- 172.21.15.167:0/552831313 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:25 5.19 5:98000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x55bcbba9d170 con 0x55bcbba67410 2023-10-05T19:28:37.494 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.490+0000 7f0fa792a700 1 -- 172.21.15.167:0/552831313 <== osd.1 v2:172.21.15.167:6800/30035 11 ==== osd_op_reply(25 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f0fa0022040 con 0x55bcbba67410 2023-10-05T19:28:37.494 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.490+0000 7f0fa792a700 1 -- 172.21.15.167:0/552831313 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:26 5.5 5:a0000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x55bcbba9d170 con 0x7f0f90029420 2023-10-05T19:28:37.495 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.490+0000 7f0fa792a700 1 -- 172.21.15.167:0/552831313 <== osd.0 v2:172.21.15.118:6800/27690 6 ==== osd_op_reply(26 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f0fa0022040 con 0x7f0f90029420 2023-10-05T19:28:37.495 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.490+0000 7f0fa792a700 1 -- 172.21.15.167:0/552831313 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:27 5.15 5:a8000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x55bcbba9d170 con 0x55bcbba9a850 2023-10-05T19:28:37.496 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.490+0000 7f0fa7129700 1 -- 172.21.15.167:0/552831313 <== osd.3 v2:172.21.15.167:6808/30036 7 ==== osd_op_reply(27 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f0f98017440 con 0x55bcbba9a850 2023-10-05T19:28:37.496 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.490+0000 7f0fa7129700 1 -- 172.21.15.167:0/552831313 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:28 5.d 5:b0000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x55bcbba9d170 con 0x7f0f98016b40 2023-10-05T19:28:37.497 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.490+0000 7f0fa6928700 1 -- 172.21.15.167:0/552831313 <== osd.2 v2:172.21.15.167:6816/30037 4 ==== osd_op_reply(28 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f0f9000fca0 con 0x7f0f98016b40 2023-10-05T19:28:37.497 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.490+0000 7f0fa6928700 1 -- 172.21.15.167:0/552831313 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:29 5.1d 5:b8000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x55bcbba9d170 con 0x55bcbba67410 2023-10-05T19:28:37.498 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.490+0000 7f0fa792a700 1 -- 172.21.15.167:0/552831313 <== osd.1 v2:172.21.15.167:6800/30035 12 ==== osd_op_reply(29 [pgnls start_epoch 30 out=107b] v30'2 uv2 ondisk = 1) v8 ==== 144+0+107 (crc 0 0 0) 0x7f0fa0022040 con 0x55bcbba67410 2023-10-05T19:28:37.498 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.490+0000 7f0fa792a700 1 -- 172.21.15.167:0/552831313 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:30 5.3 5:c0000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x55bcbba9d170 con 0x7f0f90029420 2023-10-05T19:28:37.499 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.494+0000 7f0fa792a700 1 -- 172.21.15.167:0/552831313 <== osd.0 v2:172.21.15.118:6800/27690 7 ==== osd_op_reply(30 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f0fa0022040 con 0x7f0f90029420 2023-10-05T19:28:37.499 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.494+0000 7f0fa792a700 1 -- 172.21.15.167:0/552831313 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:31 5.13 5:c8000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x55bcbba9d170 con 0x55bcbba9a850 2023-10-05T19:28:37.500 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.494+0000 7f0fa7129700 1 -- 172.21.15.167:0/552831313 <== osd.3 v2:172.21.15.167:6808/30036 8 ==== osd_op_reply(31 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f0f98017440 con 0x55bcbba9a850 2023-10-05T19:28:37.500 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.494+0000 7f0fa7129700 1 -- 172.21.15.167:0/552831313 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:32 5.b 5:d0000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x55bcbba9d170 con 0x7f0f90029420 2023-10-05T19:28:37.501 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.494+0000 7f0fa792a700 1 -- 172.21.15.167:0/552831313 <== osd.0 v2:172.21.15.118:6800/27690 8 ==== osd_op_reply(32 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f0fa0022040 con 0x7f0f90029420 2023-10-05T19:28:37.501 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.494+0000 7f0fa792a700 1 -- 172.21.15.167:0/552831313 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:33 5.1b 5:d8000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x55bcbba9d170 con 0x7f0f98016b40 2023-10-05T19:28:37.501 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.494+0000 7f0fa6928700 1 -- 172.21.15.167:0/552831313 <== osd.2 v2:172.21.15.167:6816/30037 5 ==== osd_op_reply(33 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f0f9000fca0 con 0x7f0f98016b40 2023-10-05T19:28:37.501 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.494+0000 7f0fa6928700 1 -- 172.21.15.167:0/552831313 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:34 5.7 5:e0000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x55bcbba9d170 con 0x55bcbba9a850 2023-10-05T19:28:37.502 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.498+0000 7f0fa7129700 1 -- 172.21.15.167:0/552831313 <== osd.3 v2:172.21.15.167:6808/30036 9 ==== osd_op_reply(34 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f0f98046040 con 0x55bcbba9a850 2023-10-05T19:28:37.502 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.498+0000 7f0fa7129700 1 -- 172.21.15.167:0/552831313 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:35 5.17 5:e8000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x55bcbba9d170 con 0x7f0f90029420 2023-10-05T19:28:37.503 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.498+0000 7f0fa792a700 1 -- 172.21.15.167:0/552831313 <== osd.0 v2:172.21.15.118:6800/27690 9 ==== osd_op_reply(35 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f0fa0022040 con 0x7f0f90029420 2023-10-05T19:28:37.503 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.498+0000 7f0fa792a700 1 -- 172.21.15.167:0/552831313 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:36 5.f 5:f0000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x55bcbba9d170 con 0x55bcbba67410 2023-10-05T19:28:37.504 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.498+0000 7f0fa792a700 1 -- 172.21.15.167:0/552831313 <== osd.1 v2:172.21.15.167:6800/30035 13 ==== osd_op_reply(36 [pgnls start_epoch 30 out=115b] v23'2 uv2 ondisk = 1) v8 ==== 144+0+115 (crc 0 0 0) 0x7f0fa002f080 con 0x55bcbba67410 2023-10-05T19:28:37.504 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.498+0000 7f0fa792a700 1 -- 172.21.15.167:0/552831313 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:37 5.1f 5:f8000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x55bcbba9d170 con 0x7f0f98016b40 2023-10-05T19:28:37.505 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.498+0000 7f0fa6928700 1 -- 172.21.15.167:0/552831313 <== osd.2 v2:172.21.15.167:6816/30037 6 ==== osd_op_reply(37 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f0f9000fca0 con 0x7f0f98016b40 2023-10-05T19:28:37.505 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.498+0000 7f0fa95f4e40 20 RGWRados::pool_iterate: got zonegroup_info.4f1f719d-9648-417b-b10c-e5d939825419 2023-10-05T19:28:37.506 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.498+0000 7f0fa95f4e40 20 RGWRados::pool_iterate: got zone_info.9a9df965-6612-45d4-a45f-6e3c889bacc2 2023-10-05T19:28:37.506 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.498+0000 7f0fa95f4e40 20 RGWRados::pool_iterate: got zone_names.default 2023-10-05T19:28:37.506 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.498+0000 7f0fa95f4e40 20 RGWRados::pool_iterate: got zonegroups_names.default 2023-10-05T19:28:37.506 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.498+0000 7f0fa95f4e40 20 rados->read ofs=0 len=0 2023-10-05T19:28:37.506 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.498+0000 7f0fa95f4e40 1 -- 172.21.15.167:0/552831313 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:38 5.12 5:49953fa1:::default.realm:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x55bcbba99600 con 0x55bcbba67410 2023-10-05T19:28:37.507 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.502+0000 7f0fa792a700 1 -- 172.21.15.167:0/552831313 <== osd.1 v2:172.21.15.167:6800/30035 14 ==== osd_op_reply(38 default.realm [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 ==== 157+0+0 (crc 0 0 0) 0x7f0fa002f080 con 0x55bcbba67410 2023-10-05T19:28:37.507 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.502+0000 7f0fa95f4e40 20 rados_obj.operate() r=-2 bl.length=0 2023-10-05T19:28:37.507 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.502+0000 7f0fa95f4e40 20 rados->read ofs=0 len=0 2023-10-05T19:28:37.507 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.502+0000 7f0fa95f4e40 1 -- 172.21.15.167:0/552831313 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:39 5.f 5:f4c53578:::zonegroups_names.default:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x55bcbba99ef0 con 0x55bcbba67410 2023-10-05T19:28:37.507 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.502+0000 7f0fa792a700 1 -- 172.21.15.167:0/552831313 <== osd.1 v2:172.21.15.167:6800/30035 15 ==== osd_op_reply(39 zonegroups_names.default [read 0~46 out=46b] v0'0 uv2 ondisk = 0) v8 ==== 168+0+46 (crc 0 0 0) 0x7f0fa002f080 con 0x55bcbba67410 2023-10-05T19:28:37.507 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.502+0000 7f0fa95f4e40 20 rados_obj.operate() r=0 bl.length=46 2023-10-05T19:28:37.507 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.502+0000 7f0fa95f4e40 20 rados->read ofs=0 len=0 2023-10-05T19:28:37.507 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.502+0000 7f0fa95f4e40 1 -- 172.21.15.167:0/552831313 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:40 5.c 5:30d299a5:::zonegroup_info.4f1f719d-9648-417b-b10c-e5d939825419:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x55bcbba9d5b0 con 0x7f0f90029420 2023-10-05T19:28:37.508 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.502+0000 7f0fa792a700 1 -- 172.21.15.167:0/552831313 <== osd.0 v2:172.21.15.118:6800/27690 10 ==== osd_op_reply(40 zonegroup_info.4f1f719d-9648-417b-b10c-e5d939825419 [read 0~362 out=362b] v0'0 uv1 ondisk = 0) v8 ==== 195+0+362 (crc 0 0 0) 0x7f0fa002f080 con 0x7f0f90029420 2023-10-05T19:28:37.508 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.502+0000 7f0fa95f4e40 20 rados_obj.operate() r=0 bl.length=362 2023-10-05T19:28:37.508 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.502+0000 7f0fa95f4e40 20 rados->read ofs=0 len=0 2023-10-05T19:28:37.508 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.502+0000 7f0fa95f4e40 1 -- 172.21.15.167:0/552831313 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:41 5.13 5:cbbf6c60:::converted:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x55bcbba9d5b0 con 0x55bcbba9a850 2023-10-05T19:28:37.509 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.502+0000 7f0fa7129700 1 -- 172.21.15.167:0/552831313 <== osd.3 v2:172.21.15.167:6808/30036 10 ==== osd_op_reply(41 converted [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 ==== 153+0+0 (crc 0 0 0) 0x7f0f98046040 con 0x55bcbba9a850 2023-10-05T19:28:37.509 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.502+0000 7f0fa95f4e40 20 rados_obj.operate() r=-2 bl.length=0 2023-10-05T19:28:37.509 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.502+0000 7f0fa95f4e40 20 rados->read ofs=0 len=0 2023-10-05T19:28:37.509 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.502+0000 7f0fa95f4e40 1 -- 172.21.15.167:0/552831313 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:42 5.12 5:49953fa1:::default.realm:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x55bcbba9d960 con 0x55bcbba67410 2023-10-05T19:28:37.510 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.502+0000 7f0fa792a700 1 -- 172.21.15.167:0/552831313 <== osd.1 v2:172.21.15.167:6800/30035 16 ==== osd_op_reply(42 default.realm [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 ==== 157+0+0 (crc 0 0 0) 0x7f0fa002f080 con 0x55bcbba67410 2023-10-05T19:28:37.510 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.502+0000 7f0fa95f4e40 20 rados_obj.operate() r=-2 bl.length=0 2023-10-05T19:28:37.510 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.502+0000 7f0fa95f4e40 20 rados->read ofs=0 len=0 2023-10-05T19:28:37.510 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.502+0000 7f0fa95f4e40 1 -- 172.21.15.167:0/552831313 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:43 5.f 5:f4c53578:::zonegroups_names.default:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x55bcbba9dd90 con 0x55bcbba67410 2023-10-05T19:28:37.510 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.506+0000 7f0fa792a700 1 -- 172.21.15.167:0/552831313 <== osd.1 v2:172.21.15.167:6800/30035 17 ==== osd_op_reply(43 zonegroups_names.default [read 0~46 out=46b] v0'0 uv2 ondisk = 0) v8 ==== 168+0+46 (crc 0 0 0) 0x7f0fa002f080 con 0x55bcbba67410 2023-10-05T19:28:37.510 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.506+0000 7f0fa95f4e40 20 rados_obj.operate() r=0 bl.length=46 2023-10-05T19:28:37.510 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.506+0000 7f0fa95f4e40 1 -- 172.21.15.167:0/552831313 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:44 5.0 5:00000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x55bcbba9eda0 con 0x55bcbba9a850 2023-10-05T19:28:37.512 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.506+0000 7f0fa7129700 1 -- 172.21.15.167:0/552831313 <== osd.3 v2:172.21.15.167:6808/30036 11 ==== osd_op_reply(44 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f0f98046040 con 0x55bcbba9a850 2023-10-05T19:28:37.513 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.506+0000 7f0fa7129700 1 -- 172.21.15.167:0/552831313 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:45 5.10 5:08000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x55bcbba9eda0 con 0x7f0f98016b40 2023-10-05T19:28:37.513 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.506+0000 7f0fa6928700 1 -- 172.21.15.167:0/552831313 <== osd.2 v2:172.21.15.167:6816/30037 7 ==== osd_op_reply(45 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f0f9000fca0 con 0x7f0f98016b40 2023-10-05T19:28:37.514 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.506+0000 7f0fa6928700 1 -- 172.21.15.167:0/552831313 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:46 5.8 5:10000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x55bcbba9eda0 con 0x7f0f90029420 2023-10-05T19:28:37.515 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.510+0000 7f0fa792a700 1 -- 172.21.15.167:0/552831313 <== osd.0 v2:172.21.15.118:6800/27690 11 ==== osd_op_reply(46 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f0fa002f080 con 0x7f0f90029420 2023-10-05T19:28:37.515 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.510+0000 7f0fa792a700 1 -- 172.21.15.167:0/552831313 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:47 5.18 5:18000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x55bcbba99ef0 con 0x55bcbba67410 2023-10-05T19:28:37.515 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.510+0000 7f0fa792a700 1 -- 172.21.15.167:0/552831313 <== osd.1 v2:172.21.15.167:6800/30035 18 ==== osd_op_reply(47 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f0fa002f080 con 0x55bcbba67410 2023-10-05T19:28:37.516 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.510+0000 7f0fa792a700 1 -- 172.21.15.167:0/552831313 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:48 5.4 5:20000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x55bcbba99ef0 con 0x55bcbba67410 2023-10-05T19:28:37.516 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.510+0000 7f0fa792a700 1 -- 172.21.15.167:0/552831313 <== osd.1 v2:172.21.15.167:6800/30035 19 ==== osd_op_reply(48 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f0fa002f080 con 0x55bcbba67410 2023-10-05T19:28:37.516 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.510+0000 7f0fa792a700 1 -- 172.21.15.167:0/552831313 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:49 5.14 5:28000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x55bcbba99ef0 con 0x7f0f90029420 2023-10-05T19:28:37.517 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.510+0000 7f0fa792a700 1 -- 172.21.15.167:0/552831313 <== osd.0 v2:172.21.15.118:6800/27690 12 ==== osd_op_reply(49 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f0fa002f080 con 0x7f0f90029420 2023-10-05T19:28:37.517 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.510+0000 7f0fa792a700 1 -- 172.21.15.167:0/552831313 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:50 5.c 5:30000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x55bcbba99ef0 con 0x7f0f90029420 2023-10-05T19:28:37.518 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.514+0000 7f0fa792a700 1 -- 172.21.15.167:0/552831313 <== osd.0 v2:172.21.15.118:6800/27690 13 ==== osd_op_reply(50 [pgnls start_epoch 30 out=112b] v23'1 uv1 ondisk = 1) v8 ==== 144+0+112 (crc 0 0 0) 0x7f0fa002f080 con 0x7f0f90029420 2023-10-05T19:28:37.518 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.514+0000 7f0fa792a700 1 -- 172.21.15.167:0/552831313 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:51 5.1c 5:38000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x55bcbba99ef0 con 0x7f0f98016b40 2023-10-05T19:28:37.519 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.514+0000 7f0fa6928700 1 -- 172.21.15.167:0/552831313 <== osd.2 v2:172.21.15.167:6816/30037 8 ==== osd_op_reply(51 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f0f9000fca0 con 0x7f0f98016b40 2023-10-05T19:28:37.519 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.514+0000 7f0fa6928700 1 -- 172.21.15.167:0/552831313 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:52 5.2 5:40000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x55bcbba99ef0 con 0x55bcbba9a850 2023-10-05T19:28:37.520 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.514+0000 7f0fa7129700 1 -- 172.21.15.167:0/552831313 <== osd.3 v2:172.21.15.167:6808/30036 12 ==== osd_op_reply(52 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f0f98046040 con 0x55bcbba9a850 2023-10-05T19:28:37.520 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.514+0000 7f0fa7129700 1 -- 172.21.15.167:0/552831313 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:53 5.12 5:48000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x55bcbba99ef0 con 0x55bcbba67410 2023-10-05T19:28:37.521 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.514+0000 7f0fa792a700 1 -- 172.21.15.167:0/552831313 <== osd.1 v2:172.21.15.167:6800/30035 20 ==== osd_op_reply(53 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f0fa002f080 con 0x55bcbba67410 2023-10-05T19:28:37.521 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.514+0000 7f0fa792a700 1 -- 172.21.15.167:0/552831313 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:54 5.a 5:50000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x55bcbba99ef0 con 0x7f0f90029420 2023-10-05T19:28:37.522 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.518+0000 7f0fa792a700 1 -- 172.21.15.167:0/552831313 <== osd.0 v2:172.21.15.118:6800/27690 14 ==== osd_op_reply(54 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f0fa002f080 con 0x7f0f90029420 2023-10-05T19:28:37.522 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.518+0000 7f0fa792a700 1 -- 172.21.15.167:0/552831313 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:55 5.1a 5:58000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x55bcbba99ef0 con 0x55bcbba67410 2023-10-05T19:28:37.523 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.518+0000 7f0fa792a700 1 -- 172.21.15.167:0/552831313 <== osd.1 v2:172.21.15.167:6800/30035 21 ==== osd_op_reply(55 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f0fa002f080 con 0x55bcbba67410 2023-10-05T19:28:37.523 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.518+0000 7f0fa792a700 1 -- 172.21.15.167:0/552831313 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:56 5.6 5:60000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x55bcbba99ef0 con 0x7f0f98016b40 2023-10-05T19:28:37.523 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.518+0000 7f0fa6928700 1 -- 172.21.15.167:0/552831313 <== osd.2 v2:172.21.15.167:6816/30037 9 ==== osd_op_reply(56 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f0f9000fca0 con 0x7f0f98016b40 2023-10-05T19:28:37.523 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.518+0000 7f0fa6928700 1 -- 172.21.15.167:0/552831313 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:57 5.16 5:68000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x55bcbba99ef0 con 0x55bcbba9a850 2023-10-05T19:28:37.524 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.518+0000 7f0fa7129700 1 -- 172.21.15.167:0/552831313 <== osd.3 v2:172.21.15.167:6808/30036 13 ==== osd_op_reply(57 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f0f98046040 con 0x55bcbba9a850 2023-10-05T19:28:37.524 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.518+0000 7f0fa7129700 1 -- 172.21.15.167:0/552831313 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:58 5.e 5:70000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x55bcbba99ef0 con 0x55bcbba9a850 2023-10-05T19:28:37.525 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.518+0000 7f0fa7129700 1 -- 172.21.15.167:0/552831313 <== osd.3 v2:172.21.15.167:6808/30036 14 ==== osd_op_reply(58 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f0f98046040 con 0x55bcbba9a850 2023-10-05T19:28:37.525 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.518+0000 7f0fa7129700 1 -- 172.21.15.167:0/552831313 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:59 5.1e 5:78000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x55bcbba99ef0 con 0x7f0f90029420 2023-10-05T19:28:37.526 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.522+0000 7f0fa792a700 1 -- 172.21.15.167:0/552831313 <== osd.0 v2:172.21.15.118:6800/27690 15 ==== osd_op_reply(59 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f0fa002f080 con 0x7f0f90029420 2023-10-05T19:28:37.526 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.522+0000 7f0fa792a700 1 -- 172.21.15.167:0/552831313 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:60 5.1 5:80000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x55bcbba99ef0 con 0x55bcbba9a850 2023-10-05T19:28:37.526 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.522+0000 7f0fa7129700 1 -- 172.21.15.167:0/552831313 <== osd.3 v2:172.21.15.167:6808/30036 15 ==== osd_op_reply(60 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f0f98046040 con 0x55bcbba9a850 2023-10-05T19:28:37.527 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.522+0000 7f0fa7129700 1 -- 172.21.15.167:0/552831313 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:61 5.11 5:88000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x55bcbba99ef0 con 0x55bcbba9a850 2023-10-05T19:28:37.527 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.522+0000 7f0fa7129700 1 -- 172.21.15.167:0/552831313 <== osd.3 v2:172.21.15.167:6808/30036 16 ==== osd_op_reply(61 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f0f98046040 con 0x55bcbba9a850 2023-10-05T19:28:37.527 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.522+0000 7f0fa7129700 1 -- 172.21.15.167:0/552831313 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:62 5.9 5:90000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x55bcbba99ef0 con 0x55bcbba67410 2023-10-05T19:28:37.528 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.522+0000 7f0fa792a700 1 -- 172.21.15.167:0/552831313 <== osd.1 v2:172.21.15.167:6800/30035 22 ==== osd_op_reply(62 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f0fa002f080 con 0x55bcbba67410 2023-10-05T19:28:37.528 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.522+0000 7f0fa792a700 1 -- 172.21.15.167:0/552831313 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:63 5.19 5:98000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x55bcbba99ef0 con 0x55bcbba67410 2023-10-05T19:28:37.529 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.522+0000 7f0fa792a700 1 -- 172.21.15.167:0/552831313 <== osd.1 v2:172.21.15.167:6800/30035 23 ==== osd_op_reply(63 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f0fa002f080 con 0x55bcbba67410 2023-10-05T19:28:37.529 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.522+0000 7f0fa792a700 1 -- 172.21.15.167:0/552831313 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:64 5.5 5:a0000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x55bcbba99ef0 con 0x7f0f90029420 2023-10-05T19:28:37.530 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.526+0000 7f0fa792a700 1 -- 172.21.15.167:0/552831313 <== osd.0 v2:172.21.15.118:6800/27690 16 ==== osd_op_reply(64 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f0fa002f080 con 0x7f0f90029420 2023-10-05T19:28:37.530 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.526+0000 7f0fa792a700 1 -- 172.21.15.167:0/552831313 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:65 5.15 5:a8000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x55bcbba99ef0 con 0x55bcbba9a850 2023-10-05T19:28:37.531 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.526+0000 7f0fa7129700 1 -- 172.21.15.167:0/552831313 <== osd.3 v2:172.21.15.167:6808/30036 17 ==== osd_op_reply(65 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f0f98046040 con 0x55bcbba9a850 2023-10-05T19:28:37.531 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.526+0000 7f0fa7129700 1 -- 172.21.15.167:0/552831313 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:66 5.d 5:b0000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x55bcbba99ef0 con 0x7f0f98016b40 2023-10-05T19:28:37.532 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.526+0000 7f0fa6928700 1 -- 172.21.15.167:0/552831313 <== osd.2 v2:172.21.15.167:6816/30037 10 ==== osd_op_reply(66 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f0f9000fca0 con 0x7f0f98016b40 2023-10-05T19:28:37.532 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.526+0000 7f0fa6928700 1 -- 172.21.15.167:0/552831313 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:67 5.1d 5:b8000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x55bcbba99ef0 con 0x55bcbba67410 2023-10-05T19:28:37.533 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.526+0000 7f0fa792a700 1 -- 172.21.15.167:0/552831313 <== osd.1 v2:172.21.15.167:6800/30035 24 ==== osd_op_reply(67 [pgnls start_epoch 30 out=107b] v30'2 uv2 ondisk = 1) v8 ==== 144+0+107 (crc 0 0 0) 0x7f0fa002f080 con 0x55bcbba67410 2023-10-05T19:28:37.533 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.526+0000 7f0fa792a700 1 -- 172.21.15.167:0/552831313 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:68 5.3 5:c0000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x55bcbba99ef0 con 0x7f0f90029420 2023-10-05T19:28:37.534 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.530+0000 7f0fa792a700 1 -- 172.21.15.167:0/552831313 <== osd.0 v2:172.21.15.118:6800/27690 17 ==== osd_op_reply(68 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f0fa002f080 con 0x7f0f90029420 2023-10-05T19:28:37.534 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.530+0000 7f0fa792a700 1 -- 172.21.15.167:0/552831313 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:69 5.13 5:c8000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x55bcbba99ef0 con 0x55bcbba9a850 2023-10-05T19:28:37.535 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.530+0000 7f0fa7129700 1 -- 172.21.15.167:0/552831313 <== osd.3 v2:172.21.15.167:6808/30036 18 ==== osd_op_reply(69 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f0f98046040 con 0x55bcbba9a850 2023-10-05T19:28:37.535 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.530+0000 7f0fa7129700 1 -- 172.21.15.167:0/552831313 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:70 5.b 5:d0000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x55bcbba99ef0 con 0x7f0f90029420 2023-10-05T19:28:37.536 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.530+0000 7f0fa792a700 1 -- 172.21.15.167:0/552831313 <== osd.0 v2:172.21.15.118:6800/27690 18 ==== osd_op_reply(70 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f0fa002f080 con 0x7f0f90029420 2023-10-05T19:28:37.536 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.530+0000 7f0fa792a700 1 -- 172.21.15.167:0/552831313 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:71 5.1b 5:d8000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x55bcbba99ef0 con 0x7f0f98016b40 2023-10-05T19:28:37.537 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.530+0000 7f0fa6928700 1 -- 172.21.15.167:0/552831313 <== osd.2 v2:172.21.15.167:6816/30037 11 ==== osd_op_reply(71 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f0f9000fca0 con 0x7f0f98016b40 2023-10-05T19:28:37.537 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.530+0000 7f0fa6928700 1 -- 172.21.15.167:0/552831313 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:72 5.7 5:e0000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x55bcbba99ef0 con 0x55bcbba9a850 2023-10-05T19:28:37.538 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.530+0000 7f0fa7129700 1 -- 172.21.15.167:0/552831313 <== osd.3 v2:172.21.15.167:6808/30036 19 ==== osd_op_reply(72 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f0f98046040 con 0x55bcbba9a850 2023-10-05T19:28:37.538 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.534+0000 7f0fa7129700 1 -- 172.21.15.167:0/552831313 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:73 5.17 5:e8000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x55bcbba99ef0 con 0x7f0f90029420 2023-10-05T19:28:37.539 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.534+0000 7f0fa792a700 1 -- 172.21.15.167:0/552831313 <== osd.0 v2:172.21.15.118:6800/27690 19 ==== osd_op_reply(73 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f0fa002f080 con 0x7f0f90029420 2023-10-05T19:28:37.539 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.534+0000 7f0fa792a700 1 -- 172.21.15.167:0/552831313 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:74 5.f 5:f0000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x55bcbba99ef0 con 0x55bcbba67410 2023-10-05T19:28:37.540 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.534+0000 7f0fa792a700 1 -- 172.21.15.167:0/552831313 <== osd.1 v2:172.21.15.167:6800/30035 25 ==== osd_op_reply(74 [pgnls start_epoch 30 out=115b] v23'2 uv2 ondisk = 1) v8 ==== 144+0+115 (crc 0 0 0) 0x7f0fa002f080 con 0x55bcbba67410 2023-10-05T19:28:37.540 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.534+0000 7f0fa792a700 1 -- 172.21.15.167:0/552831313 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:75 5.1f 5:f8000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x55bcbba99ef0 con 0x7f0f98016b40 2023-10-05T19:28:37.541 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.534+0000 7f0fa6928700 1 -- 172.21.15.167:0/552831313 <== osd.2 v2:172.21.15.167:6816/30037 12 ==== osd_op_reply(75 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f0f9000fca0 con 0x7f0f98016b40 2023-10-05T19:28:37.541 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.534+0000 7f0fa95f4e40 20 RGWRados::pool_iterate: got zonegroup_info.4f1f719d-9648-417b-b10c-e5d939825419 2023-10-05T19:28:37.541 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.534+0000 7f0fa95f4e40 20 RGWRados::pool_iterate: got zone_info.9a9df965-6612-45d4-a45f-6e3c889bacc2 2023-10-05T19:28:37.541 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.534+0000 7f0fa95f4e40 20 RGWRados::pool_iterate: got zone_names.default 2023-10-05T19:28:37.541 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.534+0000 7f0fa95f4e40 20 RGWRados::pool_iterate: got zonegroups_names.default 2023-10-05T19:28:37.541 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.534+0000 7f0fa95f4e40 20 rados->read ofs=0 len=0 2023-10-05T19:28:37.541 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.534+0000 7f0fa95f4e40 1 -- 172.21.15.167:0/552831313 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:76 5.f 5:f43b6ece:::zone_names.default:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x55bcbba9dd90 con 0x55bcbba67410 2023-10-05T19:28:37.542 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.534+0000 7f0fa792a700 1 -- 172.21.15.167:0/552831313 <== osd.1 v2:172.21.15.167:6800/30035 26 ==== osd_op_reply(76 zone_names.default [read 0~46 out=46b] v0'0 uv1 ondisk = 0) v8 ==== 162+0+46 (crc 0 0 0) 0x7f0fa002f080 con 0x55bcbba67410 2023-10-05T19:28:37.542 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.534+0000 7f0fa95f4e40 20 rados_obj.operate() r=0 bl.length=46 2023-10-05T19:28:37.542 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.534+0000 7f0fa95f4e40 20 rados->read ofs=0 len=0 2023-10-05T19:28:37.543 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.534+0000 7f0fa95f4e40 1 -- 172.21.15.167:0/552831313 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:77 5.1d 5:b8374eb4:::zone_info.9a9df965-6612-45d4-a45f-6e3c889bacc2:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x55bcbba9e140 con 0x55bcbba67410 2023-10-05T19:28:37.543 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.538+0000 7f0fa792a700 1 -- 172.21.15.167:0/552831313 <== osd.1 v2:172.21.15.167:6800/30035 27 ==== osd_op_reply(77 zone_info.9a9df965-6612-45d4-a45f-6e3c889bacc2 [read 0~874 out=874b] v0'0 uv2 ondisk = 0) v8 ==== 190+0+874 (crc 0 0 0) 0x7f0fa002f080 con 0x55bcbba67410 2023-10-05T19:28:37.543 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.538+0000 7f0fa95f4e40 20 rados_obj.operate() r=0 bl.length=874 2023-10-05T19:28:37.543 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.538+0000 7f0fa95f4e40 20 rados->read ofs=0 len=0 2023-10-05T19:28:37.543 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.538+0000 7f0fa95f4e40 1 -- 172.21.15.167:0/552831313 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:78 5.f 5:f4c53578:::zonegroups_names.default:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x55bcbba9e140 con 0x55bcbba67410 2023-10-05T19:28:37.543 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.538+0000 7f0fa792a700 1 -- 172.21.15.167:0/552831313 <== osd.1 v2:172.21.15.167:6800/30035 28 ==== osd_op_reply(78 zonegroups_names.default [read 0~46 out=46b] v0'0 uv2 ondisk = 0) v8 ==== 168+0+46 (crc 0 0 0) 0x7f0fa002f080 con 0x55bcbba67410 2023-10-05T19:28:37.544 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.538+0000 7f0fa95f4e40 20 rados_obj.operate() r=0 bl.length=46 2023-10-05T19:28:37.544 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.538+0000 7f0fa95f4e40 20 rados->read ofs=0 len=0 2023-10-05T19:28:37.544 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.538+0000 7f0fa95f4e40 1 -- 172.21.15.167:0/552831313 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:79 5.c 5:30d299a5:::zonegroup_info.4f1f719d-9648-417b-b10c-e5d939825419:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x55bcbba9e140 con 0x7f0f90029420 2023-10-05T19:28:37.544 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.538+0000 7f0fa792a700 1 -- 172.21.15.167:0/552831313 <== osd.0 v2:172.21.15.118:6800/27690 20 ==== osd_op_reply(79 zonegroup_info.4f1f719d-9648-417b-b10c-e5d939825419 [read 0~362 out=362b] v0'0 uv1 ondisk = 0) v8 ==== 195+0+362 (crc 0 0 0) 0x7f0fa002f080 con 0x7f0f90029420 2023-10-05T19:28:37.544 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.538+0000 7f0fa95f4e40 20 rados_obj.operate() r=0 bl.length=362 2023-10-05T19:28:37.545 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.538+0000 7f0fa95f4e40 20 rados->read ofs=0 len=0 2023-10-05T19:28:37.545 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.538+0000 7f0fa95f4e40 1 -- 172.21.15.167:0/552831313 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:80 5.3 5:c0ad8cbd:::region_map:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x55bcbba9e590 con 0x7f0f90029420 2023-10-05T19:28:37.545 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.538+0000 7f0fa792a700 1 -- 172.21.15.167:0/552831313 <== osd.0 v2:172.21.15.118:6800/27690 21 ==== osd_op_reply(80 region_map [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 ==== 154+0+0 (crc 0 0 0) 0x7f0fa002f080 con 0x7f0f90029420 2023-10-05T19:28:37.545 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.538+0000 7f0fa95f4e40 20 rados_obj.operate() r=-2 bl.length=0 2023-10-05T19:28:37.545 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.538+0000 7f0fa95f4e40 20 zone default found 2023-10-05T19:28:37.545 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.538+0000 7f0fa95f4e40 4 Realm: () 2023-10-05T19:28:37.546 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.538+0000 7f0fa95f4e40 4 ZoneGroup: default (4f1f719d-9648-417b-b10c-e5d939825419) 2023-10-05T19:28:37.546 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.538+0000 7f0fa95f4e40 4 Zone: default (9a9df965-6612-45d4-a45f-6e3c889bacc2) 2023-10-05T19:28:37.546 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.538+0000 7f0fa95f4e40 10 cannot find current period zonegroup using local zonegroup configuration 2023-10-05T19:28:37.546 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.538+0000 7f0fa95f4e40 20 zonegroup default 2023-10-05T19:28:37.546 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.538+0000 7f0fa95f4e40 20 rados->read ofs=0 len=0 2023-10-05T19:28:37.546 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.538+0000 7f0fa95f4e40 1 -- 172.21.15.167:0/552831313 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:81 5.3 5:c52100b6:::period_config.default:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x55bcbba9e590 con 0x7f0f90029420 2023-10-05T19:28:37.547 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.538+0000 7f0fa792a700 1 -- 172.21.15.167:0/552831313 <== osd.0 v2:172.21.15.118:6800/27690 22 ==== osd_op_reply(81 period_config.default [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 ==== 165+0+0 (crc 0 0 0) 0x7f0fa0050040 con 0x7f0f90029420 2023-10-05T19:28:37.547 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.542+0000 7f0fa95f4e40 20 rados_obj.operate() r=-2 bl.length=0 2023-10-05T19:28:37.547 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.542+0000 7f0fa95f4e40 20 rados->read ofs=0 len=0 2023-10-05T19:28:37.547 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.542+0000 7f0fa95f4e40 1 -- 172.21.15.167:0/552831313 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:82 6.2 6:4347d321:::bucket.sync-source-hints.:head [call version.read in=11b,read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x55bcbba9fce0 con 0x55bcbba9a850 2023-10-05T19:28:37.547 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.542+0000 7f0fa7129700 1 -- 172.21.15.167:0/552831313 <== osd.3 v2:172.21.15.167:6808/30036 20 ==== osd_op_reply(82 bucket.sync-source-hints. [call,read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 ==== 211+0+0 (crc 0 0 0) 0x7f0f98046040 con 0x55bcbba9a850 2023-10-05T19:28:37.547 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.542+0000 7f0fa95f4e40 20 rados_obj.operate() r=-2 bl.length=0 2023-10-05T19:28:37.547 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.542+0000 7f0fa95f4e40 20 rados->read ofs=0 len=0 2023-10-05T19:28:37.548 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.542+0000 7f0fa95f4e40 1 -- 172.21.15.167:0/552831313 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:83 6.b 6:d467b91b:::bucket.sync-target-hints.:head [call version.read in=11b,read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x55bcbbaa0090 con 0x55bcbba9a850 2023-10-05T19:28:37.548 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.542+0000 7f0fa7129700 1 -- 172.21.15.167:0/552831313 <== osd.3 v2:172.21.15.167:6808/30036 21 ==== osd_op_reply(83 bucket.sync-target-hints. [call,read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 ==== 211+0+0 (crc 0 0 0) 0x7f0f98052040 con 0x55bcbba9a850 2023-10-05T19:28:37.548 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.542+0000 7f0fa95f4e40 20 rados_obj.operate() r=-2 bl.length=0 2023-10-05T19:28:37.548 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.542+0000 7f0fa95f4e40 20 started sync module instance, tier type = 2023-10-05T19:28:37.548 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.542+0000 7f0fa95f4e40 20 started zone id=9a9df965-6612-45d4-a45f-6e3c889bacc2 (name=default) with tier type = 2023-10-05T19:28:37.549 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.542+0000 7f0fa95f4e40 1 -- 172.21.15.167:0/552831313 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:84 7.1f 7:f95f44c2:::notify.0:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x55bcbba9f3a0 con 0x7f0f98016b40 2023-10-05T19:28:37.551 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.546+0000 7f0fa6928700 1 -- 172.21.15.167:0/552831313 <== osd.2 v2:172.21.15.167:6816/30037 13 ==== osd_op_reply(84 notify.0 [create] v30'168 uv168 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f0f9000fca0 con 0x7f0f98016b40 2023-10-05T19:28:37.551 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.546+0000 7f0fa95f4e40 1 -- 172.21.15.167:0/552831313 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:85 7.1f 7:f95f44c2:::notify.0:head [watch watch cookie 94269090690976] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x55bcbba99b20 con 0x7f0f98016b40 2023-10-05T19:28:37.551 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.546+0000 7f0fa95f4e40 1 -- 172.21.15.167:0/552831313 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:86 7.0 7:05bf5b68:::notify.1:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x55bcbbaa0e60 con 0x55bcbba67410 2023-10-05T19:28:37.554 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.546+0000 7f0fa792a700 1 -- 172.21.15.167:0/552831313 <== osd.1 v2:172.21.15.167:6800/30035 29 ==== osd_op_reply(86 notify.1 [create] v30'168 uv168 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f0fa0050040 con 0x55bcbba67410 2023-10-05T19:28:37.554 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.546+0000 7f0fa6928700 1 -- 172.21.15.167:0/552831313 <== osd.2 v2:172.21.15.167:6816/30037 14 ==== osd_op_reply(85 notify.0 [watch watch cookie 94269090690976] v30'169 uv168 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f0f9000fca0 con 0x7f0f98016b40 2023-10-05T19:28:37.554 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.546+0000 7f0fa95f4e40 1 -- 172.21.15.167:0/552831313 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:87 7.0 7:05bf5b68:::notify.1:head [watch watch cookie 94269090695472] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x55bcbbaa1750 con 0x55bcbba67410 2023-10-05T19:28:37.554 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.546+0000 7f0fa95f4e40 1 -- 172.21.15.167:0/552831313 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:88 7.15 7:a93a5511:::notify.2:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x55bcbbaa1b00 con 0x55bcbba9a850 2023-10-05T19:28:37.556 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.550+0000 7f0fa792a700 1 -- 172.21.15.167:0/552831313 <== osd.1 v2:172.21.15.167:6800/30035 30 ==== osd_op_reply(87 notify.1 [watch watch cookie 94269090695472] v30'169 uv168 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f0fa0050040 con 0x55bcbba67410 2023-10-05T19:28:37.557 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.550+0000 7f0fa7129700 1 -- 172.21.15.167:0/552831313 <== osd.3 v2:172.21.15.167:6808/30036 22 ==== osd_op_reply(88 notify.2 [create] v30'168 uv168 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f0f98052040 con 0x55bcbba9a850 2023-10-05T19:28:37.557 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.550+0000 7f0fa95f4e40 1 -- 172.21.15.167:0/552831313 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:89 7.15 7:a93a5511:::notify.2:head [watch watch cookie 94269090697824] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x55bcbbaa16b0 con 0x55bcbba9a850 2023-10-05T19:28:37.557 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.550+0000 7f0fa95f4e40 1 -- 172.21.15.167:0/552831313 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:90 7.e 7:7759931f:::notify.3:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x55bcbbaa2890 con 0x7f0f98016b40 2023-10-05T19:28:37.559 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.554+0000 7f0fa6928700 1 -- 172.21.15.167:0/552831313 <== osd.2 v2:172.21.15.167:6816/30037 15 ==== osd_op_reply(90 notify.3 [create] v30'168 uv168 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f0f9000fca0 con 0x7f0f98016b40 2023-10-05T19:28:37.560 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.554+0000 7f0fa95f4e40 1 -- 172.21.15.167:0/552831313 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:91 7.e 7:7759931f:::notify.3:head [watch watch cookie 94269090702240] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x55bcbbaa3130 con 0x7f0f98016b40 2023-10-05T19:28:37.560 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.554+0000 7f0fa95f4e40 1 -- 172.21.15.167:0/552831313 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:92 7.d 7:b4812045:::notify.4:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x55bcbbaa3cc0 con 0x55bcbba67410 2023-10-05T19:28:37.560 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.554+0000 7f0fa7129700 1 -- 172.21.15.167:0/552831313 <== osd.3 v2:172.21.15.167:6808/30036 23 ==== osd_op_reply(89 notify.2 [watch watch cookie 94269090697824] v30'169 uv168 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f0f98052040 con 0x55bcbba9a850 2023-10-05T19:28:37.561 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.554+0000 7f0fa6928700 1 -- 172.21.15.167:0/552831313 <== osd.2 v2:172.21.15.167:6816/30037 16 ==== osd_op_reply(91 notify.3 [watch watch cookie 94269090702240] v30'169 uv168 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f0f9000fca0 con 0x7f0f98016b40 2023-10-05T19:28:37.562 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.558+0000 7f0fa792a700 1 -- 172.21.15.167:0/552831313 <== osd.1 v2:172.21.15.167:6800/30035 31 ==== osd_op_reply(92 notify.4 [create] v30'168 uv168 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f0fa0050040 con 0x55bcbba67410 2023-10-05T19:28:37.562 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.558+0000 7f0fa95f4e40 1 -- 172.21.15.167:0/552831313 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:93 7.d 7:b4812045:::notify.4:head [watch watch cookie 94269090704448] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x55bcbbaa3790 con 0x55bcbba67410 2023-10-05T19:28:37.563 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.558+0000 7f0fa95f4e40 1 -- 172.21.15.167:0/552831313 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:94 7.3 7:c609908c:::notify.5:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x55bcbbaa45b0 con 0x7f0f90029420 2023-10-05T19:28:37.565 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.558+0000 7f0fa792a700 1 -- 172.21.15.167:0/552831313 <== osd.0 v2:172.21.15.118:6800/27690 23 ==== osd_op_reply(94 notify.5 [create] v30'168 uv168 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f0fa0050040 con 0x7f0f90029420 2023-10-05T19:28:37.565 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.558+0000 7f0fa95f4e40 1 -- 172.21.15.167:0/552831313 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:95 7.3 7:c609908c:::notify.5:head [watch watch cookie 94269090709616] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x55bcbbaa49b0 con 0x7f0f90029420 2023-10-05T19:28:37.565 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.558+0000 7f0fa95f4e40 1 -- 172.21.15.167:0/552831313 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:96 7.14 7:2b04a3e9:::notify.6:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x55bcbbaa53c0 con 0x7f0f90029420 2023-10-05T19:28:37.565 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.558+0000 7f0fa792a700 1 -- 172.21.15.167:0/552831313 <== osd.1 v2:172.21.15.167:6800/30035 32 ==== osd_op_reply(93 notify.4 [watch watch cookie 94269090704448] v30'169 uv168 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f0fa0050040 con 0x55bcbba67410 2023-10-05T19:28:37.567 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.562+0000 7f0fa792a700 1 -- 172.21.15.167:0/552831313 <== osd.0 v2:172.21.15.118:6800/27690 24 ==== osd_op_reply(95 notify.5 [watch watch cookie 94269090709616] v30'169 uv168 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f0fa0050040 con 0x7f0f90029420 2023-10-05T19:28:37.568 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.562+0000 7f0fa792a700 1 -- 172.21.15.167:0/552831313 <== osd.0 v2:172.21.15.118:6800/27690 25 ==== osd_op_reply(96 notify.6 [create] v30'168 uv168 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f0fa0050040 con 0x7f0f90029420 2023-10-05T19:28:37.568 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.562+0000 7f0fa95f4e40 1 -- 172.21.15.167:0/552831313 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:97 7.14 7:2b04a3e9:::notify.6:head [watch watch cookie 94269090710720] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x55bcbbaa4910 con 0x7f0f90029420 2023-10-05T19:28:37.568 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.562+0000 7f0fa95f4e40 1 -- 172.21.15.167:0/552831313 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:98 7.9 7:93e5b521:::notify.7:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x55bcbbaa58b0 con 0x7f0f90029420 2023-10-05T19:28:37.571 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.566+0000 7f0fa792a700 1 -- 172.21.15.167:0/552831313 <== osd.0 v2:172.21.15.118:6800/27690 26 ==== osd_op_reply(97 notify.6 [watch watch cookie 94269090710720] v30'169 uv168 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f0fa0050040 con 0x7f0f90029420 2023-10-05T19:28:37.571 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.566+0000 7f0fa792a700 1 -- 172.21.15.167:0/552831313 <== osd.0 v2:172.21.15.118:6800/27690 27 ==== osd_op_reply(98 notify.7 [create] v30'168 uv168 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f0fa0050040 con 0x7f0f90029420 2023-10-05T19:28:37.572 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.566+0000 7f0fa95f4e40 1 -- 172.21.15.167:0/552831313 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:99 7.9 7:93e5b521:::notify.7:head [watch watch cookie 94269090713296] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x55bcbbaa5810 con 0x7f0f90029420 2023-10-05T19:28:37.572 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.566+0000 7f0fa95f4e40 20 add_watcher() i=0 2023-10-05T19:28:37.572 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.566+0000 7f0fa95f4e40 20 add_watcher() i=1 2023-10-05T19:28:37.572 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.566+0000 7f0fa95f4e40 20 add_watcher() i=2 2023-10-05T19:28:37.572 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.566+0000 7f0fa95f4e40 20 add_watcher() i=3 2023-10-05T19:28:37.572 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.566+0000 7f0fa95f4e40 20 add_watcher() i=4 2023-10-05T19:28:37.572 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.566+0000 7f0fa95f4e40 20 add_watcher() i=5 2023-10-05T19:28:37.572 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.566+0000 7f0fa95f4e40 20 add_watcher() i=6 2023-10-05T19:28:37.574 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.566+0000 7f0fa792a700 1 -- 172.21.15.167:0/552831313 <== osd.0 v2:172.21.15.118:6800/27690 28 ==== osd_op_reply(99 notify.7 [watch watch cookie 94269090713296] v30'169 uv168 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f0fa0050040 con 0x7f0f90029420 2023-10-05T19:28:37.574 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.566+0000 7f0fa95f4e40 20 add_watcher() i=7 2023-10-05T19:28:37.574 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.566+0000 7f0fa95f4e40 2 all 8 watchers are set, enabling cache 2023-10-05T19:28:37.574 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.566+0000 7f0fa95f4e40 1 -- 172.21.15.167:0/552831313 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:100 6.4 6:22d26bf9:::data_loggenerations_metadata:head [call version.check_conds in=74b,call version.read in=11b,read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x55bcbbaa89b0 con 0x55bcbba67410 2023-10-05T19:28:37.574 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.570+0000 7f0fa792a700 1 -- 172.21.15.167:0/552831313 <== osd.1 v2:172.21.15.167:6800/30035 33 ==== osd_op_reply(100 data_loggenerations_metadata [call,call out=48b,read 0~28 out=28b] v0'0 uv1 ondisk = 0) v8 ==== 256+0+76 (crc 0 0 0) 0x7f0fa0050040 con 0x55bcbba67410 2023-10-05T19:28:37.575 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.570+0000 7f0fa95f4e40 1 -- 172.21.15.167:0/552831313 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:101 6.4 6:22d26bf9:::data_loggenerations_metadata:head [watch watch cookie 94269090714768] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x55bcbbaa5f20 con 0x55bcbba67410 2023-10-05T19:28:37.577 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.570+0000 7f0fa792a700 1 -- 172.21.15.167:0/552831313 <== osd.1 v2:172.21.15.167:6800/30035 34 ==== osd_op_reply(101 data_loggenerations_metadata [watch watch cookie 94269090714768] v30'159 uv1 ondisk = 0) v8 ==== 172+0+0 (crc 0 0 0) 0x7f0fa0050040 con 0x55bcbba67410 2023-10-05T19:28:37.577 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.570+0000 7f0f78fe1700 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2023-10-05T19:28:37.577 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.570+0000 7f0fa95f4e40 20 check_secure_mon_conn(): auth registy supported: methods=[2] modes=[2,1] 2023-10-05T19:28:37.577 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.570+0000 7f0fa95f4e40 20 check_secure_mon_conn(): mode 1 is insecure 2023-10-05T19:28:37.578 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.570+0000 7f0fa95f4e40 1 Processor -- start 2023-10-05T19:28:37.578 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.570+0000 7f0fa95f4e40 1 -- start start 2023-10-05T19:28:37.578 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.574+0000 7f0fa95f4e40 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55bcbbab3180 0x55bcbbab3550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-05T19:28:37.578 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.574+0000 7f0fa95f4e40 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x55bcbbab3a90 con 0x55bcbbab3180 2023-10-05T19:28:37.579 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.574+0000 7f0fa6928700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55bcbbab3180 0x55bcbbab3550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-05T19:28:37.579 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.574+0000 7f0fa6928700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55bcbbab3180 0x55bcbbab3550 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.118:3300/0 says I am v2:172.21.15.167:42034/0 (socket says 172.21.15.167:42034) 2023-10-05T19:28:37.579 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.574+0000 7f0fa6928700 1 -- 172.21.15.167:0/1815131207 learned_addr learned my addr 172.21.15.167:0/1815131207 (peer_addr_for_me v2:172.21.15.167:0/0) 2023-10-05T19:28:37.579 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.574+0000 7f0fa6928700 1 -- 172.21.15.167:0/1815131207 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55bcbbab3bd0 con 0x55bcbbab3180 2023-10-05T19:28:37.580 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.574+0000 7f0fa6928700 1 --2- 172.21.15.167:0/1815131207 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55bcbbab3180 0x55bcbbab3550 secure :-1 s=READY pgs=474 cs=0 l=1 rev1=1 crypto rx=0x55bcbba63eb0 tx=0x7f0f90011000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3e9097b6136f01bb server_cookie=0 in_seq=0 out_seq=0 2023-10-05T19:28:37.580 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.574+0000 7f0ef3fff700 1 -- 172.21.15.167:0/1815131207 <== mon.0 v2:172.21.15.118:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f0f90030040 con 0x55bcbbab3180 2023-10-05T19:28:37.580 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.574+0000 7f0ef3fff700 1 -- 172.21.15.167:0/1815131207 <== mon.0 v2:172.21.15.118:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0f90036820 con 0x55bcbbab3180 2023-10-05T19:28:37.580 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.574+0000 7f0ef3fff700 1 -- 172.21.15.167:0/1815131207 <== mon.0 v2:172.21.15.118:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f0f900369f0 con 0x55bcbbab3180 2023-10-05T19:28:37.581 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.574+0000 7f0fa95f4e40 1 -- 172.21.15.167:0/1815131207 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55bcbbab3180 msgr2=0x55bcbbab3550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-05T19:28:37.581 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.574+0000 7f0fa95f4e40 1 --2- 172.21.15.167:0/1815131207 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55bcbbab3180 0x55bcbbab3550 secure :-1 s=READY pgs=474 cs=0 l=1 rev1=1 crypto rx=0x55bcbba63eb0 tx=0x7f0f90011000 comp rx=0 tx=0).stop 2023-10-05T19:28:37.582 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.574+0000 7f0fa95f4e40 1 -- 172.21.15.167:0/1815131207 shutdown_connections 2023-10-05T19:28:37.582 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.574+0000 7f0fa95f4e40 1 --2- 172.21.15.167:0/1815131207 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55bcbbab3180 0x55bcbbab3550 unknown :-1 s=CLOSED pgs=474 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-05T19:28:37.582 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.574+0000 7f0fa95f4e40 1 -- 172.21.15.167:0/1815131207 >> 172.21.15.167:0/1815131207 conn(0x55bcbbaae540 msgr2=0x55bcbbaac690 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-05T19:28:37.582 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.574+0000 7f0fa95f4e40 1 -- 172.21.15.167:0/1815131207 shutdown_connections 2023-10-05T19:28:37.582 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.574+0000 7f0fa95f4e40 1 -- 172.21.15.167:0/1815131207 wait complete. 2023-10-05T19:28:37.583 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.578+0000 7f0fa95f4e40 1 Processor -- start 2023-10-05T19:28:37.583 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.578+0000 7f0fa95f4e40 1 -- start start 2023-10-05T19:28:37.583 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.578+0000 7f0fa95f4e40 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55bcbbab3180 0x55bcbbab8cb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-05T19:28:37.583 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.578+0000 7f0fa95f4e40 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x55bcbbab9cb0 con 0x55bcbbab3180 2023-10-05T19:28:37.583 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.578+0000 7f0fa6928700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55bcbbab3180 0x55bcbbab8cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-05T19:28:37.583 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.578+0000 7f0fa6928700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55bcbbab3180 0x55bcbbab8cb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.118:3300/0 says I am v2:172.21.15.167:42046/0 (socket says 172.21.15.167:42046) 2023-10-05T19:28:37.583 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.578+0000 7f0fa6928700 1 -- 172.21.15.167:0/3656086001 learned_addr learned my addr 172.21.15.167:0/3656086001 (peer_addr_for_me v2:172.21.15.167:0/0) 2023-10-05T19:28:37.584 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.578+0000 7f0fa6928700 1 -- 172.21.15.167:0/3656086001 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55bcbba63a40 con 0x55bcbbab3180 2023-10-05T19:28:37.585 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.578+0000 7f0fa6928700 1 --2- 172.21.15.167:0/3656086001 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55bcbbab3180 0x55bcbbab8cb0 secure :-1 s=READY pgs=475 cs=0 l=1 rev1=1 crypto rx=0x7f0f90044040 tx=0x7f0f90042000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-05T19:28:37.585 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.578+0000 7f0ef27fc700 1 -- 172.21.15.167:0/3656086001 <== mon.0 v2:172.21.15.118:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f0f90030050 con 0x55bcbbab3180 2023-10-05T19:28:37.585 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.578+0000 7f0ef27fc700 1 -- 172.21.15.167:0/3656086001 <== mon.0 v2:172.21.15.118:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0f9004e070 con 0x55bcbbab3180 2023-10-05T19:28:37.585 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.578+0000 7f0fa95f4e40 1 -- 172.21.15.167:0/3656086001 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x55bcbbab92b0 con 0x55bcbbab3180 2023-10-05T19:28:37.585 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.578+0000 7f0ef27fc700 1 -- 172.21.15.167:0/3656086001 <== mon.0 v2:172.21.15.118:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f0f9001c7c0 con 0x55bcbbab3180 2023-10-05T19:28:37.585 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.578+0000 7f0fa95f4e40 1 -- 172.21.15.167:0/3656086001 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55bcbbab9750 con 0x55bcbbab3180 2023-10-05T19:28:37.586 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.582+0000 7f0ef27fc700 1 -- 172.21.15.167:0/3656086001 <== mon.0 v2:172.21.15.118:3300/0 4 ==== mgrmap(e 5) v1 ==== 40329+0+0 (secure 0 0 0) 0x7f0f90050030 con 0x55bcbbab3180 2023-10-05T19:28:37.587 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.582+0000 7f0ef27fc700 1 --2- 172.21.15.167:0/3656086001 >> [v2:172.21.15.167:6824/29889,v1:172.21.15.167:6825/29889] conn(0x7f0ee8030bb0 0x7f0ee8033060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-05T19:28:37.587 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.582+0000 7f0ef27fc700 1 -- 172.21.15.167:0/3656086001 <== mon.0 v2:172.21.15.118:3300/0 5 ==== osd_map(30..30 src has 1..30) v4 ==== 6057+0+0 (secure 0 0 0) 0x7f0f9003b030 con 0x55bcbbab3180 2023-10-05T19:28:37.587 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.582+0000 7f0fa792a700 1 --2- 172.21.15.167:0/3656086001 >> [v2:172.21.15.167:6824/29889,v1:172.21.15.167:6825/29889] conn(0x7f0ee8030bb0 0x7f0ee8033060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-05T19:28:37.587 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.582+0000 7f0fa95f4e40 5 note: GC not initialized 2023-10-05T19:28:37.588 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.582+0000 7f0fa792a700 1 --2- 172.21.15.167:0/3656086001 >> [v2:172.21.15.167:6824/29889,v1:172.21.15.167:6825/29889] conn(0x7f0ee8030bb0 0x7f0ee8033060 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x55bcbbaa5e80 tx=0x7f0fa0000f90 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-05T19:28:37.589 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.582+0000 7f0ee77fe700 20 reqs_thread_entry: start 2023-10-05T19:28:37.590 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.586+0000 7f0fa95f4e40 20 init_complete bucket index max shards: 11 2023-10-05T19:28:37.590 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.586+0000 7f0ee57fa700 20 reqs_thread_entry: start 2023-10-05T19:28:37.591 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.586+0000 7f0fa95f4e40 10 rgw notify: Started notification manager with: 1 workers 2023-10-05T19:28:37.591 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.586+0000 7f0ed37fe700 20 rgw notify: INFO: next queues processing will happen at: Thu Oct 5 19:29:07 2023 2023-10-05T19:28:37.591 INFO:teuthology.orchestra.run.smithi167.stderr: 2023-10-05T19:28:37.591 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.586+0000 7f0ed2ffd700 20 reqs_thread_entry: start 2023-10-05T19:28:37.591 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.586+0000 7f0fa95f4e40 1 --2- 172.21.15.167:0/3656086001 >> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] conn(0x55bcbbb20300 0x55bcbbb226e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-05T19:28:37.592 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.586+0000 7f0fa95f4e40 1 -- 172.21.15.167:0/3656086001 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:1 6.1c 6:38c02c58:::2023-10-05-19-9a9df965-6612-45d4-a45f-6e3c889bacc2.4223.1-myfoo:head [delete] snapc 0=[] ondisk+write+peerstat_old+known_if_redirected+supports_pool_eio e30) v8 -- 0x55bcbbb22c20 con 0x55bcbbb20300 2023-10-05T19:28:37.592 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.586+0000 7f0fa7129700 1 --2- 172.21.15.167:0/3656086001 >> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] conn(0x55bcbbb20300 0x55bcbbb226e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-05T19:28:37.592 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.586+0000 7f0fa7129700 1 --2- 172.21.15.167:0/3656086001 >> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] conn(0x55bcbbb20300 0x55bcbbb226e0 crc :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-05T19:28:37.594 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.590+0000 7f0fa7129700 1 -- 172.21.15.167:0/3656086001 <== osd.3 v2:172.21.15.167:6808/30036 1 ==== osd_op_reply(1 2023-10-05-19-9a9df965-6612-45d4-a45f-6e3c889bacc2.4223.1-myfoo [delete] v30'32 uv32 ondisk = 0) v8 ==== 207+0+0 (crc 0 0 0) 0x7f0f98052040 con 0x55bcbbb20300 2023-10-05T19:28:37.594 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.590+0000 7f0fa95f4e40 1 -- 172.21.15.167:0/552831313 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:102 7.1f 7:f95f44c2:::notify.0:head [watch unwatch cookie 94269090690976] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x55bcbbadf440 con 0x7f0f98016b40 2023-10-05T19:28:37.597 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.590+0000 7f0fa6928700 1 -- 172.21.15.167:0/552831313 <== osd.2 v2:172.21.15.167:6816/30037 17 ==== osd_op_reply(102 notify.0 [watch unwatch cookie 94269090690976] v30'170 uv168 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f0f90048020 con 0x7f0f98016b40 2023-10-05T19:28:37.597 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.590+0000 7f0fa95f4e40 20 remove_watcher() i=0 2023-10-05T19:28:37.597 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.590+0000 7f0fa95f4e40 2 removed watcher, disabling cache 2023-10-05T19:28:37.597 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.590+0000 7f0fa95f4e40 1 -- 172.21.15.167:0/552831313 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:103 7.0 7:05bf5b68:::notify.1:head [watch unwatch cookie 94269090695472] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x55bcbba9f7a0 con 0x55bcbba67410 2023-10-05T19:28:37.600 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.594+0000 7f0fa792a700 1 -- 172.21.15.167:0/552831313 <== osd.1 v2:172.21.15.167:6800/30035 35 ==== osd_op_reply(103 notify.1 [watch unwatch cookie 94269090695472] v30'170 uv168 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f0fa0050040 con 0x55bcbba67410 2023-10-05T19:28:37.600 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.594+0000 7f0fa95f4e40 20 remove_watcher() i=1 2023-10-05T19:28:37.600 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.594+0000 7f0fa95f4e40 1 -- 172.21.15.167:0/552831313 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:104 7.15 7:a93a5511:::notify.2:head [watch unwatch cookie 94269090697824] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x55bcbbaa0930 con 0x55bcbba9a850 2023-10-05T19:28:37.603 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.598+0000 7f0fa7129700 1 -- 172.21.15.167:0/552831313 <== osd.3 v2:172.21.15.167:6808/30036 24 ==== osd_op_reply(104 notify.2 [watch unwatch cookie 94269090697824] v30'170 uv168 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f0f98052040 con 0x55bcbba9a850 2023-10-05T19:28:37.603 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.598+0000 7f0fa95f4e40 20 remove_watcher() i=2 2023-10-05T19:28:37.603 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.598+0000 7f0fa95f4e40 1 -- 172.21.15.167:0/552831313 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:105 7.e 7:7759931f:::notify.3:head [watch unwatch cookie 94269090702240] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x55bcbbaa0930 con 0x7f0f98016b40 2023-10-05T19:28:37.605 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.598+0000 7f0fa6928700 1 -- 172.21.15.167:0/552831313 <== osd.2 v2:172.21.15.167:6816/30037 18 ==== osd_op_reply(105 notify.3 [watch unwatch cookie 94269090702240] v30'170 uv168 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f0f90048020 con 0x7f0f98016b40 2023-10-05T19:28:37.605 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.598+0000 7f0fa95f4e40 20 remove_watcher() i=3 2023-10-05T19:28:37.605 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.598+0000 7f0fa95f4e40 1 -- 172.21.15.167:0/552831313 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:106 7.d 7:b4812045:::notify.4:head [watch unwatch cookie 94269090704448] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x55bcbbaa23a0 con 0x55bcbba67410 2023-10-05T19:28:37.608 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.602+0000 7f0fa792a700 1 -- 172.21.15.167:0/552831313 <== osd.1 v2:172.21.15.167:6800/30035 36 ==== osd_op_reply(106 notify.4 [watch unwatch cookie 94269090704448] v30'170 uv168 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f0fa0050040 con 0x55bcbba67410 2023-10-05T19:28:37.609 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.602+0000 7f0fa95f4e40 20 remove_watcher() i=4 2023-10-05T19:28:37.609 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.602+0000 7f0fa95f4e40 1 -- 172.21.15.167:0/552831313 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:107 7.3 7:c609908c:::notify.5:head [watch unwatch cookie 94269090709616] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x55bcbbaa0930 con 0x7f0f90029420 2023-10-05T19:28:37.611 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.606+0000 7f0fa792a700 1 -- 172.21.15.167:0/552831313 <== osd.0 v2:172.21.15.118:6800/27690 29 ==== osd_op_reply(107 notify.5 [watch unwatch cookie 94269090709616] v30'170 uv168 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f0fa0050040 con 0x7f0f90029420 2023-10-05T19:28:37.611 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.606+0000 7f0fa95f4e40 20 remove_watcher() i=5 2023-10-05T19:28:37.611 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.606+0000 7f0fa95f4e40 1 -- 172.21.15.167:0/552831313 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:108 7.14 7:2b04a3e9:::notify.6:head [watch unwatch cookie 94269090710720] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x55bcbbaa2c40 con 0x7f0f90029420 2023-10-05T19:28:37.614 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.610+0000 7f0fa792a700 1 -- 172.21.15.167:0/552831313 <== osd.0 v2:172.21.15.118:6800/27690 30 ==== osd_op_reply(108 notify.6 [watch unwatch cookie 94269090710720] v30'170 uv168 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f0fa0050040 con 0x7f0f90029420 2023-10-05T19:28:37.614 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.610+0000 7f0fa95f4e40 20 remove_watcher() i=6 2023-10-05T19:28:37.615 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.610+0000 7f0fa95f4e40 1 -- 172.21.15.167:0/552831313 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:109 7.9 7:93e5b521:::notify.7:head [watch unwatch cookie 94269090713296] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x55bcbbaa0930 con 0x7f0f90029420 2023-10-05T19:28:37.618 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.610+0000 7f0fa792a700 1 -- 172.21.15.167:0/552831313 <== osd.0 v2:172.21.15.118:6800/27690 31 ==== osd_op_reply(109 notify.7 [watch unwatch cookie 94269090713296] v30'170 uv168 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f0fa0050040 con 0x7f0f90029420 2023-10-05T19:28:37.618 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.610+0000 7f0fa95f4e40 20 remove_watcher() i=7 2023-10-05T19:28:37.619 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.614+0000 7f0f78fe1700 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2023-10-05T19:28:37.620 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.614+0000 7f0fa95f4e40 1 -- 172.21.15.167:0/552831313 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:110 6.4 6:22d26bf9:::data_loggenerations_metadata:head [watch unwatch cookie 94269090714768] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x55bcbbaa0930 con 0x55bcbba67410 2023-10-05T19:28:37.622 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.618+0000 7f0fa792a700 1 -- 172.21.15.167:0/552831313 <== osd.1 v2:172.21.15.167:6800/30035 37 ==== osd_op_reply(110 data_loggenerations_metadata [watch unwatch cookie 94269090714768] v30'160 uv1 ondisk = 0) v8 ==== 172+0+0 (crc 0 0 0) 0x7f0fa0050040 con 0x55bcbba67410 2023-10-05T19:28:37.623 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.618+0000 7f0fa95f4e40 1 -- 172.21.15.167:0/552831313 >> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] conn(0x7f0f90029420 msgr2=0x7f0f90029830 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-05T19:28:37.623 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.618+0000 7f0fa95f4e40 1 --2- 172.21.15.167:0/552831313 >> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] conn(0x7f0f90029420 0x7f0f90029830 crc :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-05T19:28:37.623 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.618+0000 7f0fa95f4e40 1 -- 172.21.15.167:0/552831313 >> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] conn(0x55bcbba67410 msgr2=0x55bcbba96a20 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-05T19:28:37.623 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.618+0000 7f0fa95f4e40 1 --2- 172.21.15.167:0/552831313 >> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] conn(0x55bcbba67410 0x55bcbba96a20 crc :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-05T19:28:37.623 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.618+0000 7f0fa95f4e40 1 -- 172.21.15.167:0/552831313 >> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] conn(0x7f0f98016b40 msgr2=0x7f0f980243d0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-05T19:28:37.623 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.618+0000 7f0fa95f4e40 1 --2- 172.21.15.167:0/552831313 >> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] conn(0x7f0f98016b40 0x7f0f980243d0 crc :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-05T19:28:37.623 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.618+0000 7f0fa95f4e40 1 -- 172.21.15.167:0/552831313 >> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] conn(0x55bcbba9a850 msgr2=0x55bcbba9cc30 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-05T19:28:37.623 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.618+0000 7f0fa95f4e40 1 --2- 172.21.15.167:0/552831313 >> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] conn(0x55bcbba9a850 0x55bcbba9cc30 crc :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-05T19:28:37.623 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.618+0000 7f0fa95f4e40 1 -- 172.21.15.167:0/552831313 >> [v2:172.21.15.167:6824/29889,v1:172.21.15.167:6825/29889] conn(0x7f0f88030c00 msgr2=0x7f0f880330b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-05T19:28:37.623 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.618+0000 7f0fa95f4e40 1 --2- 172.21.15.167:0/552831313 >> [v2:172.21.15.167:6824/29889,v1:172.21.15.167:6825/29889] conn(0x7f0f88030c00 0x7f0f880330b0 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7f0f90005d80 tx=0x7f0f90014000 comp rx=0 tx=0).stop 2023-10-05T19:28:37.623 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.618+0000 7f0fa792a700 1 -- 172.21.15.167:0/552831313 reap_dead start 2023-10-05T19:28:37.624 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.618+0000 7f0fa95f4e40 1 -- 172.21.15.167:0/552831313 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55bcbba62e30 msgr2=0x55bcbba8af60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-05T19:28:37.624 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.618+0000 7f0fa95f4e40 1 --2- 172.21.15.167:0/552831313 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55bcbba62e30 0x55bcbba8af60 secure :-1 s=READY pgs=473 cs=0 l=1 rev1=1 crypto rx=0x7f0f98001820 tx=0x7f0f98016000 comp rx=0 tx=0).stop 2023-10-05T19:28:37.625 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.618+0000 7f0fa95f4e40 1 -- 172.21.15.167:0/552831313 shutdown_connections 2023-10-05T19:28:37.625 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.618+0000 7f0fa95f4e40 1 --2- 172.21.15.167:0/552831313 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55bcbba62e30 0x55bcbba8af60 unknown :-1 s=CLOSED pgs=473 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-05T19:28:37.625 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.618+0000 7f0fa95f4e40 1 -- 172.21.15.167:0/552831313 >> 172.21.15.167:0/552831313 conn(0x55bcbba5e850 msgr2=0x55bcbba7c5d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-05T19:28:37.625 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.618+0000 7f0fa95f4e40 1 -- 172.21.15.167:0/552831313 shutdown_connections 2023-10-05T19:28:37.625 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.618+0000 7f0fa95f4e40 1 -- 172.21.15.167:0/552831313 wait complete. 2023-10-05T19:28:37.625 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.618+0000 7f0fa95f4e40 1 -- 172.21.15.167:0/3656086001 >> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] conn(0x55bcbbb20300 msgr2=0x55bcbbb226e0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-05T19:28:37.626 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.618+0000 7f0fa95f4e40 1 --2- 172.21.15.167:0/3656086001 >> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] conn(0x55bcbbb20300 0x55bcbbb226e0 crc :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-05T19:28:37.626 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.618+0000 7f0fa95f4e40 1 -- 172.21.15.167:0/3656086001 >> [v2:172.21.15.167:6824/29889,v1:172.21.15.167:6825/29889] conn(0x7f0ee8030bb0 msgr2=0x7f0ee8033060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-05T19:28:37.626 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.618+0000 7f0fa95f4e40 1 --2- 172.21.15.167:0/3656086001 >> [v2:172.21.15.167:6824/29889,v1:172.21.15.167:6825/29889] conn(0x7f0ee8030bb0 0x7f0ee8033060 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x55bcbbaa5e80 tx=0x7f0fa0000f90 comp rx=0 tx=0).stop 2023-10-05T19:28:37.626 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.618+0000 7f0fa95f4e40 1 -- 172.21.15.167:0/3656086001 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55bcbbab3180 msgr2=0x55bcbbab8cb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-05T19:28:37.626 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.618+0000 7f0fa95f4e40 1 --2- 172.21.15.167:0/3656086001 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55bcbbab3180 0x55bcbbab8cb0 secure :-1 s=READY pgs=475 cs=0 l=1 rev1=1 crypto rx=0x7f0f90044040 tx=0x7f0f90042000 comp rx=0 tx=0).stop 2023-10-05T19:28:37.626 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.618+0000 7f0fa95f4e40 1 -- 172.21.15.167:0/3656086001 shutdown_connections 2023-10-05T19:28:37.626 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.618+0000 7f0fa95f4e40 1 --2- 172.21.15.167:0/3656086001 >> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] conn(0x55bcbbb20300 0x55bcbbb226e0 unknown :-1 s=CLOSED pgs=79 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-05T19:28:37.626 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.618+0000 7f0fa95f4e40 1 --2- 172.21.15.167:0/3656086001 >> [v2:172.21.15.167:6824/29889,v1:172.21.15.167:6825/29889] conn(0x7f0ee8030bb0 0x7f0ee8033060 unknown :-1 s=CLOSED pgs=165 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-05T19:28:37.626 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.618+0000 7f0fa95f4e40 1 --2- 172.21.15.167:0/3656086001 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55bcbbab3180 0x55bcbbab8cb0 unknown :-1 s=CLOSED pgs=475 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-05T19:28:37.626 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.618+0000 7f0fa95f4e40 1 -- 172.21.15.167:0/3656086001 >> 172.21.15.167:0/3656086001 conn(0x55bcbbaae540 msgr2=0x55bcbbaac5e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-05T19:28:37.626 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.618+0000 7f0fa95f4e40 1 -- 172.21.15.167:0/3656086001 shutdown_connections 2023-10-05T19:28:37.627 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.618+0000 7f0fa95f4e40 1 -- 172.21.15.167:0/3656086001 wait complete. 2023-10-05T19:28:37.630 INFO:tasks.util.rgw:rgwadmin: client.0 : ['log', 'show', '--object', '2023-10-05-19-9a9df965-6612-45d4-a45f-6e3c889bacc2.4223.2-myfoo2'] 2023-10-05T19:28:37.631 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'ceph', 'log', 'show', '--object', '2023-10-05-19-9a9df965-6612-45d4-a45f-6e3c889bacc2.4223.2-myfoo2'] 2023-10-05T19:28:37.631 DEBUG:teuthology.orchestra.run.smithi167:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster ceph log show --object 2023-10-05-19-9a9df965-6612-45d4-a45f-6e3c889bacc2.4223.2-myfoo2 2023-10-05T19:28:37.712 INFO:teuthology.orchestra.run.smithi167.stderr:ignoring --setuser ceph since I am not root 2023-10-05T19:28:37.712 INFO:teuthology.orchestra.run.smithi167.stderr:ignoring --setgroup ceph since I am not root 2023-10-05T19:28:37.713 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.706+0000 7f8809ff3e40 1 Processor -- start 2023-10-05T19:28:37.713 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.706+0000 7f8809ff3e40 1 -- start start 2023-10-05T19:28:37.713 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.706+0000 7f8809ff3e40 1 --2- >> v2:172.21.15.118:3300/0 conn(0x5555a69f72f0 0x5555a69f96d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-05T19:28:37.713 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.706+0000 7f8809ff3e40 1 -- --> v1:172.21.15.118:6789/0 -- auth(proto 0 26 bytes epoch 0) v1 -- 0x5555a693fbd0 con 0x5555a69673d0 2023-10-05T19:28:37.714 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.706+0000 7f8809ff3e40 1 -- --> v2:172.21.15.118:3300/0 -- mon_getmap magic: 0 v1 -- 0x5555a68fcee0 con 0x5555a69f72f0 2023-10-05T19:28:37.714 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.706+0000 7f8807327700 1 --2- >> v2:172.21.15.118:3300/0 conn(0x5555a69f72f0 0x5555a69f96d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-05T19:28:37.714 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.706+0000 7f8807b28700 1 --1- >> v1:172.21.15.118:6789/0 conn(0x5555a69673d0 0x5555a69f5bd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.118:6789/0 says I am v1:172.21.15.167:59376/0 (socket says 172.21.15.167:59376) 2023-10-05T19:28:37.714 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.706+0000 7f8807b28700 1 -- 172.21.15.167:0/3034309534 learned_addr learned my addr 172.21.15.167:0/3034309534 (peer_addr_for_me v1:172.21.15.167:0/0) 2023-10-05T19:28:37.715 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.710+0000 7f8806b26700 1 -- 172.21.15.167:0/3034309534 <== mon.0 v1:172.21.15.118:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3960669325 0 0) 0x5555a693fbd0 con 0x5555a69673d0 2023-10-05T19:28:37.715 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.710+0000 7f8807327700 1 -- 172.21.15.167:0/3034309534 >> v1:172.21.15.118:6789/0 conn(0x5555a69673d0 legacy=0x5555a69f5bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-05T19:28:37.715 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.710+0000 7f8807327700 1 -- 172.21.15.167:0/3034309534 --> v2:172.21.15.118:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5555a69fac20 con 0x5555a69f72f0 2023-10-05T19:28:37.716 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.710+0000 7f8807327700 1 --2- 172.21.15.167:0/3034309534 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5555a69f72f0 0x5555a69f96d0 secure :-1 s=READY pgs=477 cs=0 l=1 rev1=1 crypto rx=0x7f87f000a5d0 tx=0x7f87f0005a50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7e6528a84d0dc41a server_cookie=0 in_seq=0 out_seq=0 2023-10-05T19:28:37.716 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.710+0000 7f8806b26700 1 -- 172.21.15.167:0/3034309534 <== mon.0 v2:172.21.15.118:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f87f00086a0 con 0x5555a69f72f0 2023-10-05T19:28:37.716 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.710+0000 7f8806b26700 1 -- 172.21.15.167:0/3034309534 <== mon.0 v2:172.21.15.118:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f87f0008800 con 0x5555a69f72f0 2023-10-05T19:28:37.716 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.710+0000 7f8806b26700 1 -- 172.21.15.167:0/3034309534 <== mon.0 v2:172.21.15.118:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f87f0008ad0 con 0x5555a69f72f0 2023-10-05T19:28:37.717 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.710+0000 7f8809ff3e40 1 -- 172.21.15.167:0/3034309534 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5555a69f72f0 msgr2=0x5555a69f96d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-05T19:28:37.717 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.710+0000 7f8809ff3e40 1 --2- 172.21.15.167:0/3034309534 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5555a69f72f0 0x5555a69f96d0 secure :-1 s=READY pgs=477 cs=0 l=1 rev1=1 crypto rx=0x7f87f000a5d0 tx=0x7f87f0005a50 comp rx=0 tx=0).stop 2023-10-05T19:28:37.717 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.710+0000 7f8809ff3e40 1 -- 172.21.15.167:0/3034309534 shutdown_connections 2023-10-05T19:28:37.717 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.710+0000 7f8809ff3e40 1 --2- 172.21.15.167:0/3034309534 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5555a69f72f0 0x5555a69f96d0 unknown :-1 s=CLOSED pgs=477 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-05T19:28:37.717 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.710+0000 7f8809ff3e40 1 -- 172.21.15.167:0/3034309534 >> 172.21.15.167:0/3034309534 conn(0x5555a69ed720 msgr2=0x5555a69efb30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-05T19:28:37.717 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.710+0000 7f8809ff3e40 1 -- 172.21.15.167:0/3034309534 shutdown_connections 2023-10-05T19:28:37.717 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.710+0000 7f8809ff3e40 1 -- 172.21.15.167:0/3034309534 wait complete. 2023-10-05T19:28:37.718 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.710+0000 7f8809ff3e40 1 Processor -- start 2023-10-05T19:28:37.718 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.710+0000 7f8809ff3e40 1 -- start start 2023-10-05T19:28:37.718 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.710+0000 7f8809ff3e40 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5555a69673d0 0x5555a6a11df0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-05T19:28:37.719 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.710+0000 7f8809ff3e40 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x5555a68c0e30 con 0x5555a69673d0 2023-10-05T19:28:37.719 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.710+0000 7f8807b28700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5555a69673d0 0x5555a6a11df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-05T19:28:37.719 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.714+0000 7f8807b28700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5555a69673d0 0x5555a6a11df0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.118:3300/0 says I am v2:172.21.15.167:42060/0 (socket says 172.21.15.167:42060) 2023-10-05T19:28:37.719 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.714+0000 7f8807b28700 1 -- 172.21.15.167:0/1596549503 learned_addr learned my addr 172.21.15.167:0/1596549503 (peer_addr_for_me v2:172.21.15.167:0/0) 2023-10-05T19:28:37.719 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.714+0000 7f8807b28700 1 -- 172.21.15.167:0/1596549503 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f87f0005cf0 con 0x5555a69673d0 2023-10-05T19:28:37.720 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.714+0000 7f8807b28700 1 --2- 172.21.15.167:0/1596549503 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5555a69673d0 0x5555a6a11df0 secure :-1 s=READY pgs=478 cs=0 l=1 rev1=1 crypto rx=0x7f87f8001dc0 tx=0x7f87f8001790 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5f9f31571a45727a server_cookie=0 in_seq=0 out_seq=0 2023-10-05T19:28:37.720 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.714+0000 7f8805b24700 1 -- 172.21.15.167:0/1596549503 <== mon.0 v2:172.21.15.118:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f87f800bce0 con 0x5555a69673d0 2023-10-05T19:28:37.720 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.714+0000 7f8805b24700 1 -- 172.21.15.167:0/1596549503 <== mon.0 v2:172.21.15.118:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f87f800be40 con 0x5555a69673d0 2023-10-05T19:28:37.720 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.714+0000 7f8805b24700 1 -- 172.21.15.167:0/1596549503 <== mon.0 v2:172.21.15.118:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f87f800e5d0 con 0x5555a69673d0 2023-10-05T19:28:37.720 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.714+0000 7f8809ff3e40 1 -- 172.21.15.167:0/1596549503 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5555a69673d0 msgr2=0x5555a6a11df0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-05T19:28:37.721 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.714+0000 7f8809ff3e40 1 --2- 172.21.15.167:0/1596549503 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5555a69673d0 0x5555a6a11df0 secure :-1 s=READY pgs=478 cs=0 l=1 rev1=1 crypto rx=0x7f87f8001dc0 tx=0x7f87f8001790 comp rx=0 tx=0).stop 2023-10-05T19:28:37.721 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.714+0000 7f8809ff3e40 1 -- 172.21.15.167:0/1596549503 shutdown_connections 2023-10-05T19:28:37.721 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.714+0000 7f8809ff3e40 1 --2- 172.21.15.167:0/1596549503 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5555a69673d0 0x5555a6a11df0 unknown :-1 s=CLOSED pgs=478 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-05T19:28:37.721 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.714+0000 7f8809ff3e40 1 -- 172.21.15.167:0/1596549503 >> 172.21.15.167:0/1596549503 conn(0x5555a69ed720 msgr2=0x5555a6a0d9b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-05T19:28:37.721 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.714+0000 7f8809ff3e40 1 -- 172.21.15.167:0/1596549503 shutdown_connections 2023-10-05T19:28:37.721 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.714+0000 7f8809ff3e40 1 -- 172.21.15.167:0/1596549503 wait complete. 2023-10-05T19:28:37.722 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.714+0000 7f8809ff3e40 1 Processor -- start 2023-10-05T19:28:37.722 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.714+0000 7f8809ff3e40 1 -- start start 2023-10-05T19:28:37.722 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.718+0000 7f8809ff3e40 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5555a69673d0 0x5555a6a18bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-05T19:28:37.723 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.718+0000 7f8809ff3e40 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x5555a6893ed0 con 0x5555a69673d0 2023-10-05T19:28:37.723 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.718+0000 7f8807b28700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5555a69673d0 0x5555a6a18bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-05T19:28:37.723 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.718+0000 7f8807b28700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5555a69673d0 0x5555a6a18bd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.118:3300/0 says I am v2:172.21.15.167:42062/0 (socket says 172.21.15.167:42062) 2023-10-05T19:28:37.723 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.718+0000 7f8807b28700 1 -- 172.21.15.167:0/3210337137 learned_addr learned my addr 172.21.15.167:0/3210337137 (peer_addr_for_me v2:172.21.15.167:0/0) 2023-10-05T19:28:37.723 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.718+0000 7f8807b28700 1 -- 172.21.15.167:0/3210337137 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f87f8001a70 con 0x5555a69673d0 2023-10-05T19:28:37.724 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.718+0000 7f8807b28700 1 --2- 172.21.15.167:0/3210337137 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5555a69673d0 0x5555a6a18bd0 secure :-1 s=READY pgs=479 cs=0 l=1 rev1=1 crypto rx=0x7f87f80017d0 tx=0x7f87f8016000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-05T19:28:37.724 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.718+0000 7f87fffff700 1 -- 172.21.15.167:0/3210337137 <== mon.0 v2:172.21.15.118:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f87f801f070 con 0x5555a69673d0 2023-10-05T19:28:37.724 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.718+0000 7f87fffff700 1 -- 172.21.15.167:0/3210337137 <== mon.0 v2:172.21.15.118:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f87f801a550 con 0x5555a69673d0 2023-10-05T19:28:37.724 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.718+0000 7f8809ff3e40 1 -- 172.21.15.167:0/3210337137 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x5555a6a19110 con 0x5555a69673d0 2023-10-05T19:28:37.725 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.718+0000 7f87fffff700 1 -- 172.21.15.167:0/3210337137 <== mon.0 v2:172.21.15.118:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f87f801a840 con 0x5555a69673d0 2023-10-05T19:28:37.725 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.718+0000 7f8809ff3e40 1 -- 172.21.15.167:0/3210337137 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x5555a6a19550 con 0x5555a69673d0 2023-10-05T19:28:37.725 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.718+0000 7f87fffff700 1 -- 172.21.15.167:0/3210337137 <== mon.0 v2:172.21.15.118:3300/0 4 ==== mgrmap(e 5) v1 ==== 40329+0+0 (secure 0 0 0) 0x7f87f801a9f0 con 0x5555a69673d0 2023-10-05T19:28:37.726 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.722+0000 7f87fffff700 1 --2- 172.21.15.167:0/3210337137 >> [v2:172.21.15.167:6824/29889,v1:172.21.15.167:6825/29889] conn(0x7f87e8030c00 0x7f87e80330b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-05T19:28:37.726 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.722+0000 7f87fffff700 1 -- 172.21.15.167:0/3210337137 <== mon.0 v2:172.21.15.118:3300/0 5 ==== osd_map(30..30 src has 1..30) v4 ==== 6057+0+0 (secure 0 0 0) 0x7f87f8045260 con 0x5555a69673d0 2023-10-05T19:28:37.727 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.722+0000 7f8809ff3e40 20 rados->read ofs=0 len=0 2023-10-05T19:28:37.727 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.722+0000 7f8807327700 1 --2- 172.21.15.167:0/3210337137 >> [v2:172.21.15.167:6824/29889,v1:172.21.15.167:6825/29889] conn(0x7f87e8030c00 0x7f87e80330b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-05T19:28:37.727 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.722+0000 7f8809ff3e40 1 --2- 172.21.15.167:0/3210337137 >> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] conn(0x5555a69f72f0 0x5555a6a245f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-05T19:28:37.727 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.722+0000 7f8809ff3e40 1 -- 172.21.15.167:0/3210337137 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:1 5.12 5:49953fa1:::default.realm:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x5555a6a24d20 con 0x5555a69f72f0 2023-10-05T19:28:37.727 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.722+0000 7f8808329700 1 --2- 172.21.15.167:0/3210337137 >> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] conn(0x5555a69f72f0 0x5555a6a245f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-05T19:28:37.727 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.722+0000 7f8807327700 1 --2- 172.21.15.167:0/3210337137 >> [v2:172.21.15.167:6824/29889,v1:172.21.15.167:6825/29889] conn(0x7f87e8030c00 0x7f87e80330b0 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7f87f0005a90 tx=0x7f87f0015000 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-05T19:28:37.728 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.722+0000 7f8808329700 1 --2- 172.21.15.167:0/3210337137 >> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] conn(0x5555a69f72f0 0x5555a6a245f0 crc :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-05T19:28:37.728 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.722+0000 7f8808329700 1 -- 172.21.15.167:0/3210337137 <== osd.1 v2:172.21.15.167:6800/30035 1 ==== osd_op_reply(1 default.realm [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 ==== 157+0+0 (crc 0 0 0) 0x7f880000a5a0 con 0x5555a69f72f0 2023-10-05T19:28:37.728 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.722+0000 7f8809ff3e40 20 rados_obj.operate() r=-2 bl.length=0 2023-10-05T19:28:37.728 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.722+0000 7f8809ff3e40 20 realm 2023-10-05T19:28:37.729 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.722+0000 7f8809ff3e40 20 rados->read ofs=0 len=0 2023-10-05T19:28:37.729 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.722+0000 7f8809ff3e40 1 -- 172.21.15.167:0/3210337137 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:2 5.12 5:49953fa1:::default.realm:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x5555a6a21c20 con 0x5555a69f72f0 2023-10-05T19:28:37.729 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.722+0000 7f8808329700 1 -- 172.21.15.167:0/3210337137 <== osd.1 v2:172.21.15.167:6800/30035 2 ==== osd_op_reply(2 default.realm [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 ==== 157+0+0 (crc 0 0 0) 0x7f880000a5a0 con 0x5555a69f72f0 2023-10-05T19:28:37.729 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.722+0000 7f8809ff3e40 20 rados_obj.operate() r=-2 bl.length=0 2023-10-05T19:28:37.729 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.722+0000 7f8809ff3e40 4 RGWPeriod::init failed to init realm id : (2) No such file or directory 2023-10-05T19:28:37.729 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.722+0000 7f8809ff3e40 20 rados->read ofs=0 len=0 2023-10-05T19:28:37.729 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.722+0000 7f8809ff3e40 1 -- 172.21.15.167:0/3210337137 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:3 5.12 5:49953fa1:::default.realm:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x5555a6a25850 con 0x5555a69f72f0 2023-10-05T19:28:37.730 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.722+0000 7f8808329700 1 -- 172.21.15.167:0/3210337137 <== osd.1 v2:172.21.15.167:6800/30035 3 ==== osd_op_reply(3 default.realm [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 ==== 157+0+0 (crc 0 0 0) 0x7f880000a5a0 con 0x5555a69f72f0 2023-10-05T19:28:37.730 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.722+0000 7f8809ff3e40 20 rados_obj.operate() r=-2 bl.length=0 2023-10-05T19:28:37.730 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.722+0000 7f8809ff3e40 20 rados->read ofs=0 len=0 2023-10-05T19:28:37.730 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.722+0000 7f8809ff3e40 1 -- 172.21.15.167:0/3210337137 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:4 5.f 5:f43b6ece:::zone_names.default:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x5555a6a25430 con 0x5555a69f72f0 2023-10-05T19:28:37.731 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.722+0000 7f8808329700 1 -- 172.21.15.167:0/3210337137 <== osd.1 v2:172.21.15.167:6800/30035 4 ==== osd_op_reply(4 zone_names.default [read 0~46 out=46b] v0'0 uv1 ondisk = 0) v8 ==== 162+0+46 (crc 0 0 0) 0x7f880000a5a0 con 0x5555a69f72f0 2023-10-05T19:28:37.731 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.722+0000 7f8809ff3e40 20 rados_obj.operate() r=0 bl.length=46 2023-10-05T19:28:37.731 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.722+0000 7f8809ff3e40 20 rados->read ofs=0 len=0 2023-10-05T19:28:37.731 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.722+0000 7f8809ff3e40 1 -- 172.21.15.167:0/3210337137 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:5 5.1d 5:b8374eb4:::zone_info.9a9df965-6612-45d4-a45f-6e3c889bacc2:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x5555a6a262c0 con 0x5555a69f72f0 2023-10-05T19:28:37.731 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.722+0000 7f8808329700 1 -- 172.21.15.167:0/3210337137 <== osd.1 v2:172.21.15.167:6800/30035 5 ==== osd_op_reply(5 zone_info.9a9df965-6612-45d4-a45f-6e3c889bacc2 [read 0~874 out=874b] v0'0 uv2 ondisk = 0) v8 ==== 190+0+874 (crc 0 0 0) 0x7f880000a5a0 con 0x5555a69f72f0 2023-10-05T19:28:37.731 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.722+0000 7f8809ff3e40 20 rados_obj.operate() r=0 bl.length=874 2023-10-05T19:28:37.731 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.722+0000 7f8809ff3e40 20 searching for the correct realm 2023-10-05T19:28:37.731 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.722+0000 7f8809ff3e40 1 --2- 172.21.15.167:0/3210337137 >> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] conn(0x5555a6a28110 0x5555a6a2a4f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-05T19:28:37.731 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.722+0000 7f8809ff3e40 1 -- 172.21.15.167:0/3210337137 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:6 5.0 5:00000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x5555a6a2aa30 con 0x5555a6a28110 2023-10-05T19:28:37.731 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.722+0000 7f8807b28700 1 --2- 172.21.15.167:0/3210337137 >> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] conn(0x5555a6a28110 0x5555a6a2a4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-05T19:28:37.731 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.726+0000 7f8807b28700 1 --2- 172.21.15.167:0/3210337137 >> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] conn(0x5555a6a28110 0x5555a6a2a4f0 crc :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-05T19:28:37.733 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.726+0000 7f8807b28700 1 -- 172.21.15.167:0/3210337137 <== osd.3 v2:172.21.15.167:6808/30036 1 ==== osd_op_reply(6 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f87f8014070 con 0x5555a6a28110 2023-10-05T19:28:37.733 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.726+0000 7f8807b28700 1 --2- 172.21.15.167:0/3210337137 >> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] conn(0x7f87f800d750 0x7f87f8042230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-05T19:28:37.733 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.726+0000 7f8807b28700 1 -- 172.21.15.167:0/3210337137 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:7 5.10 5:08000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x5555a6a2aa30 con 0x7f87f800d750 2023-10-05T19:28:37.733 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.726+0000 7f8807327700 1 --2- 172.21.15.167:0/3210337137 >> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] conn(0x7f87f800d750 0x7f87f8042230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-05T19:28:37.733 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.726+0000 7f8807327700 1 --2- 172.21.15.167:0/3210337137 >> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] conn(0x7f87f800d750 0x7f87f8042230 crc :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-05T19:28:37.734 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.726+0000 7f8807327700 1 -- 172.21.15.167:0/3210337137 <== osd.2 v2:172.21.15.167:6816/30037 1 ==== osd_op_reply(7 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f87f0012450 con 0x7f87f800d750 2023-10-05T19:28:37.734 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.726+0000 7f8807327700 1 --2- 172.21.15.167:0/3210337137 >> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] conn(0x7f87f0007030 0x7f87f001c920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-05T19:28:37.734 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.726+0000 7f8807327700 1 -- 172.21.15.167:0/3210337137 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:8 5.8 5:10000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x5555a6a2aa30 con 0x7f87f0007030 2023-10-05T19:28:37.734 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.730+0000 7f8808329700 1 --2- 172.21.15.167:0/3210337137 >> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] conn(0x7f87f0007030 0x7f87f001c920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-05T19:28:37.734 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.730+0000 7f8808329700 1 --2- 172.21.15.167:0/3210337137 >> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] conn(0x7f87f0007030 0x7f87f001c920 crc :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-05T19:28:37.735 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.730+0000 7f8808329700 1 -- 172.21.15.167:0/3210337137 <== osd.0 v2:172.21.15.118:6800/27690 1 ==== osd_op_reply(8 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f880000a5a0 con 0x7f87f0007030 2023-10-05T19:28:37.736 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.730+0000 7f8808329700 1 -- 172.21.15.167:0/3210337137 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:9 5.18 5:18000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x5555a6a2aa30 con 0x5555a69f72f0 2023-10-05T19:28:37.736 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.730+0000 7f8808329700 1 -- 172.21.15.167:0/3210337137 <== osd.1 v2:172.21.15.167:6800/30035 6 ==== osd_op_reply(9 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f880000a5a0 con 0x5555a69f72f0 2023-10-05T19:28:37.736 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.730+0000 7f8808329700 1 -- 172.21.15.167:0/3210337137 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:10 5.4 5:20000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x5555a6a2aa30 con 0x5555a69f72f0 2023-10-05T19:28:37.737 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.730+0000 7f8808329700 1 -- 172.21.15.167:0/3210337137 <== osd.1 v2:172.21.15.167:6800/30035 7 ==== osd_op_reply(10 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f880000a5a0 con 0x5555a69f72f0 2023-10-05T19:28:37.737 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.730+0000 7f8808329700 1 -- 172.21.15.167:0/3210337137 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:11 5.14 5:28000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x5555a6a2aa30 con 0x7f87f0007030 2023-10-05T19:28:37.738 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.730+0000 7f8808329700 1 -- 172.21.15.167:0/3210337137 <== osd.0 v2:172.21.15.118:6800/27690 2 ==== osd_op_reply(11 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f880000a5a0 con 0x7f87f0007030 2023-10-05T19:28:37.738 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.730+0000 7f8808329700 1 -- 172.21.15.167:0/3210337137 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:12 5.c 5:30000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x5555a6a2aa30 con 0x7f87f0007030 2023-10-05T19:28:37.739 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.734+0000 7f8808329700 1 -- 172.21.15.167:0/3210337137 <== osd.0 v2:172.21.15.118:6800/27690 3 ==== osd_op_reply(12 [pgnls start_epoch 30 out=112b] v23'1 uv1 ondisk = 1) v8 ==== 144+0+112 (crc 0 0 0) 0x7f880000a5a0 con 0x7f87f0007030 2023-10-05T19:28:37.739 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.734+0000 7f8808329700 1 -- 172.21.15.167:0/3210337137 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:13 5.1c 5:38000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x5555a6a2aa30 con 0x7f87f800d750 2023-10-05T19:28:37.740 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.734+0000 7f8807327700 1 -- 172.21.15.167:0/3210337137 <== osd.2 v2:172.21.15.167:6816/30037 2 ==== osd_op_reply(13 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f87f0012450 con 0x7f87f800d750 2023-10-05T19:28:37.740 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.734+0000 7f8807327700 1 -- 172.21.15.167:0/3210337137 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:14 5.2 5:40000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x5555a6a2aa30 con 0x5555a6a28110 2023-10-05T19:28:37.740 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.734+0000 7f8807b28700 1 -- 172.21.15.167:0/3210337137 <== osd.3 v2:172.21.15.167:6808/30036 2 ==== osd_op_reply(14 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f87f8014070 con 0x5555a6a28110 2023-10-05T19:28:37.740 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.734+0000 7f8807b28700 1 -- 172.21.15.167:0/3210337137 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:15 5.12 5:48000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x5555a6a2aa30 con 0x5555a69f72f0 2023-10-05T19:28:37.741 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.734+0000 7f8808329700 1 -- 172.21.15.167:0/3210337137 <== osd.1 v2:172.21.15.167:6800/30035 8 ==== osd_op_reply(15 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f880000a5a0 con 0x5555a69f72f0 2023-10-05T19:28:37.741 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.734+0000 7f8808329700 1 -- 172.21.15.167:0/3210337137 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:16 5.a 5:50000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x5555a6a2aa30 con 0x7f87f0007030 2023-10-05T19:28:37.742 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.738+0000 7f8808329700 1 -- 172.21.15.167:0/3210337137 <== osd.0 v2:172.21.15.118:6800/27690 4 ==== osd_op_reply(16 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f880000a5a0 con 0x7f87f0007030 2023-10-05T19:28:37.742 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.738+0000 7f8808329700 1 -- 172.21.15.167:0/3210337137 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:17 5.1a 5:58000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x5555a6a2aa30 con 0x5555a69f72f0 2023-10-05T19:28:37.743 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.738+0000 7f8808329700 1 -- 172.21.15.167:0/3210337137 <== osd.1 v2:172.21.15.167:6800/30035 9 ==== osd_op_reply(17 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f880000a5a0 con 0x5555a69f72f0 2023-10-05T19:28:37.743 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.738+0000 7f8808329700 1 -- 172.21.15.167:0/3210337137 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:18 5.6 5:60000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x5555a6a2aa30 con 0x7f87f800d750 2023-10-05T19:28:37.743 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.738+0000 7f8807327700 1 -- 172.21.15.167:0/3210337137 <== osd.2 v2:172.21.15.167:6816/30037 3 ==== osd_op_reply(18 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f87f0012450 con 0x7f87f800d750 2023-10-05T19:28:37.743 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.738+0000 7f8807327700 1 -- 172.21.15.167:0/3210337137 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:19 5.16 5:68000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x5555a6a2aa30 con 0x5555a6a28110 2023-10-05T19:28:37.744 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.738+0000 7f8807b28700 1 -- 172.21.15.167:0/3210337137 <== osd.3 v2:172.21.15.167:6808/30036 3 ==== osd_op_reply(19 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f87f8014070 con 0x5555a6a28110 2023-10-05T19:28:37.744 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.738+0000 7f8807b28700 1 -- 172.21.15.167:0/3210337137 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:20 5.e 5:70000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x5555a6a2aa30 con 0x5555a6a28110 2023-10-05T19:28:37.745 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.738+0000 7f8807b28700 1 -- 172.21.15.167:0/3210337137 <== osd.3 v2:172.21.15.167:6808/30036 4 ==== osd_op_reply(20 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f87f8014070 con 0x5555a6a28110 2023-10-05T19:28:37.745 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.738+0000 7f8807b28700 1 -- 172.21.15.167:0/3210337137 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:21 5.1e 5:78000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x5555a6a2aa30 con 0x7f87f0007030 2023-10-05T19:28:37.746 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.742+0000 7f8808329700 1 -- 172.21.15.167:0/3210337137 <== osd.0 v2:172.21.15.118:6800/27690 5 ==== osd_op_reply(21 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f880000a5a0 con 0x7f87f0007030 2023-10-05T19:28:37.746 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.742+0000 7f8808329700 1 -- 172.21.15.167:0/3210337137 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:22 5.1 5:80000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x5555a6a2aa30 con 0x5555a6a28110 2023-10-05T19:28:37.746 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.742+0000 7f8807b28700 1 -- 172.21.15.167:0/3210337137 <== osd.3 v2:172.21.15.167:6808/30036 5 ==== osd_op_reply(22 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f87f8014070 con 0x5555a6a28110 2023-10-05T19:28:37.747 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.742+0000 7f8807b28700 1 -- 172.21.15.167:0/3210337137 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:23 5.11 5:88000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x5555a6a2aa30 con 0x5555a6a28110 2023-10-05T19:28:37.747 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.742+0000 7f8807b28700 1 -- 172.21.15.167:0/3210337137 <== osd.3 v2:172.21.15.167:6808/30036 6 ==== osd_op_reply(23 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f87f8014070 con 0x5555a6a28110 2023-10-05T19:28:37.747 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.742+0000 7f8807b28700 1 -- 172.21.15.167:0/3210337137 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:24 5.9 5:90000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x5555a6a2aa30 con 0x5555a69f72f0 2023-10-05T19:28:37.748 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.742+0000 7f8808329700 1 -- 172.21.15.167:0/3210337137 <== osd.1 v2:172.21.15.167:6800/30035 10 ==== osd_op_reply(24 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f880000a5a0 con 0x5555a69f72f0 2023-10-05T19:28:37.748 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.742+0000 7f8808329700 1 -- 172.21.15.167:0/3210337137 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:25 5.19 5:98000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x5555a6a2aa30 con 0x5555a69f72f0 2023-10-05T19:28:37.748 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.742+0000 7f8808329700 1 -- 172.21.15.167:0/3210337137 <== osd.1 v2:172.21.15.167:6800/30035 11 ==== osd_op_reply(25 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f880000a5a0 con 0x5555a69f72f0 2023-10-05T19:28:37.749 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.742+0000 7f8808329700 1 -- 172.21.15.167:0/3210337137 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:26 5.5 5:a0000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x5555a6a2aa30 con 0x7f87f0007030 2023-10-05T19:28:37.749 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.742+0000 7f8808329700 1 -- 172.21.15.167:0/3210337137 <== osd.0 v2:172.21.15.118:6800/27690 6 ==== osd_op_reply(26 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f880000a5a0 con 0x7f87f0007030 2023-10-05T19:28:37.750 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.742+0000 7f8808329700 1 -- 172.21.15.167:0/3210337137 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:27 5.15 5:a8000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x5555a6a2aa30 con 0x5555a6a28110 2023-10-05T19:28:37.750 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.746+0000 7f8807b28700 1 -- 172.21.15.167:0/3210337137 <== osd.3 v2:172.21.15.167:6808/30036 7 ==== osd_op_reply(27 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f87f8014070 con 0x5555a6a28110 2023-10-05T19:28:37.750 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.746+0000 7f8807b28700 1 -- 172.21.15.167:0/3210337137 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:28 5.d 5:b0000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x5555a6a2aa30 con 0x7f87f800d750 2023-10-05T19:28:37.751 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.746+0000 7f8807327700 1 -- 172.21.15.167:0/3210337137 <== osd.2 v2:172.21.15.167:6816/30037 4 ==== osd_op_reply(28 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f87f0012450 con 0x7f87f800d750 2023-10-05T19:28:37.751 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.746+0000 7f8807327700 1 -- 172.21.15.167:0/3210337137 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:29 5.1d 5:b8000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x5555a6a2aa30 con 0x5555a69f72f0 2023-10-05T19:28:37.752 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.746+0000 7f8808329700 1 -- 172.21.15.167:0/3210337137 <== osd.1 v2:172.21.15.167:6800/30035 12 ==== osd_op_reply(29 [pgnls start_epoch 30 out=107b] v30'2 uv2 ondisk = 1) v8 ==== 144+0+107 (crc 0 0 0) 0x7f880000a5a0 con 0x5555a69f72f0 2023-10-05T19:28:37.752 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.746+0000 7f8808329700 1 -- 172.21.15.167:0/3210337137 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:30 5.3 5:c0000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x5555a6a2aa30 con 0x7f87f0007030 2023-10-05T19:28:37.753 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.746+0000 7f8808329700 1 -- 172.21.15.167:0/3210337137 <== osd.0 v2:172.21.15.118:6800/27690 7 ==== osd_op_reply(30 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f880000a5a0 con 0x7f87f0007030 2023-10-05T19:28:37.753 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.746+0000 7f8808329700 1 -- 172.21.15.167:0/3210337137 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:31 5.13 5:c8000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x5555a6a2aa30 con 0x5555a6a28110 2023-10-05T19:28:37.753 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.746+0000 7f8807b28700 1 -- 172.21.15.167:0/3210337137 <== osd.3 v2:172.21.15.167:6808/30036 8 ==== osd_op_reply(31 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f87f8014070 con 0x5555a6a28110 2023-10-05T19:28:37.753 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.746+0000 7f8807b28700 1 -- 172.21.15.167:0/3210337137 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:32 5.b 5:d0000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x5555a6a2aa30 con 0x7f87f0007030 2023-10-05T19:28:37.754 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.750+0000 7f8808329700 1 -- 172.21.15.167:0/3210337137 <== osd.0 v2:172.21.15.118:6800/27690 8 ==== osd_op_reply(32 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f880000a5a0 con 0x7f87f0007030 2023-10-05T19:28:37.754 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.750+0000 7f8808329700 1 -- 172.21.15.167:0/3210337137 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:33 5.1b 5:d8000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x5555a6a2aa30 con 0x7f87f800d750 2023-10-05T19:28:37.755 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.750+0000 7f8807327700 1 -- 172.21.15.167:0/3210337137 <== osd.2 v2:172.21.15.167:6816/30037 5 ==== osd_op_reply(33 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f87f0012450 con 0x7f87f800d750 2023-10-05T19:28:37.755 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.750+0000 7f8807327700 1 -- 172.21.15.167:0/3210337137 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:34 5.7 5:e0000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x5555a6a2aa30 con 0x5555a6a28110 2023-10-05T19:28:37.756 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.750+0000 7f8807b28700 1 -- 172.21.15.167:0/3210337137 <== osd.3 v2:172.21.15.167:6808/30036 9 ==== osd_op_reply(34 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f87f8014070 con 0x5555a6a28110 2023-10-05T19:28:37.756 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.750+0000 7f8807b28700 1 -- 172.21.15.167:0/3210337137 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:35 5.17 5:e8000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x5555a6a2aa30 con 0x7f87f0007030 2023-10-05T19:28:37.757 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.750+0000 7f8808329700 1 -- 172.21.15.167:0/3210337137 <== osd.0 v2:172.21.15.118:6800/27690 9 ==== osd_op_reply(35 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f880000a5a0 con 0x7f87f0007030 2023-10-05T19:28:37.757 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.750+0000 7f8808329700 1 -- 172.21.15.167:0/3210337137 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:36 5.f 5:f0000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x5555a6a2aa30 con 0x5555a69f72f0 2023-10-05T19:28:37.758 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.750+0000 7f8808329700 1 -- 172.21.15.167:0/3210337137 <== osd.1 v2:172.21.15.167:6800/30035 13 ==== osd_op_reply(36 [pgnls start_epoch 30 out=115b] v23'2 uv2 ondisk = 1) v8 ==== 144+0+115 (crc 0 0 0) 0x7f880000a5a0 con 0x5555a69f72f0 2023-10-05T19:28:37.758 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.754+0000 7f8808329700 1 -- 172.21.15.167:0/3210337137 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:37 5.1f 5:f8000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x5555a6a2aa30 con 0x7f87f800d750 2023-10-05T19:28:37.759 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.754+0000 7f8807327700 1 -- 172.21.15.167:0/3210337137 <== osd.2 v2:172.21.15.167:6816/30037 6 ==== osd_op_reply(37 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f87f0012450 con 0x7f87f800d750 2023-10-05T19:28:37.759 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.754+0000 7f8809ff3e40 20 RGWRados::pool_iterate: got zonegroup_info.4f1f719d-9648-417b-b10c-e5d939825419 2023-10-05T19:28:37.759 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.754+0000 7f8809ff3e40 20 RGWRados::pool_iterate: got zone_info.9a9df965-6612-45d4-a45f-6e3c889bacc2 2023-10-05T19:28:37.759 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.754+0000 7f8809ff3e40 20 RGWRados::pool_iterate: got zone_names.default 2023-10-05T19:28:37.759 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.754+0000 7f8809ff3e40 20 RGWRados::pool_iterate: got zonegroups_names.default 2023-10-05T19:28:37.759 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.754+0000 7f8809ff3e40 20 rados->read ofs=0 len=0 2023-10-05T19:28:37.759 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.754+0000 7f8809ff3e40 1 -- 172.21.15.167:0/3210337137 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:38 5.12 5:49953fa1:::default.realm:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x5555a6a26e10 con 0x5555a69f72f0 2023-10-05T19:28:37.760 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.754+0000 7f8808329700 1 -- 172.21.15.167:0/3210337137 <== osd.1 v2:172.21.15.167:6800/30035 14 ==== osd_op_reply(38 default.realm [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 ==== 157+0+0 (crc 0 0 0) 0x7f880000a5a0 con 0x5555a69f72f0 2023-10-05T19:28:37.760 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.754+0000 7f8809ff3e40 20 rados_obj.operate() r=-2 bl.length=0 2023-10-05T19:28:37.760 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.754+0000 7f8809ff3e40 20 rados->read ofs=0 len=0 2023-10-05T19:28:37.760 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.754+0000 7f8809ff3e40 1 -- 172.21.15.167:0/3210337137 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:39 5.f 5:f4c53578:::zonegroups_names.default:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x5555a6a26a10 con 0x5555a69f72f0 2023-10-05T19:28:37.760 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.754+0000 7f8808329700 1 -- 172.21.15.167:0/3210337137 <== osd.1 v2:172.21.15.167:6800/30035 15 ==== osd_op_reply(39 zonegroups_names.default [read 0~46 out=46b] v0'0 uv2 ondisk = 0) v8 ==== 168+0+46 (crc 0 0 0) 0x7f880000a5a0 con 0x5555a69f72f0 2023-10-05T19:28:37.761 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.754+0000 7f8809ff3e40 20 rados_obj.operate() r=0 bl.length=46 2023-10-05T19:28:37.761 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.754+0000 7f8809ff3e40 20 rados->read ofs=0 len=0 2023-10-05T19:28:37.761 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.754+0000 7f8809ff3e40 1 -- 172.21.15.167:0/3210337137 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:40 5.c 5:30d299a5:::zonegroup_info.4f1f719d-9648-417b-b10c-e5d939825419:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x5555a6a269e0 con 0x7f87f0007030 2023-10-05T19:28:37.762 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.754+0000 7f8808329700 1 -- 172.21.15.167:0/3210337137 <== osd.0 v2:172.21.15.118:6800/27690 10 ==== osd_op_reply(40 zonegroup_info.4f1f719d-9648-417b-b10c-e5d939825419 [read 0~362 out=362b] v0'0 uv1 ondisk = 0) v8 ==== 195+0+362 (crc 0 0 0) 0x7f880000a5a0 con 0x7f87f0007030 2023-10-05T19:28:37.762 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.754+0000 7f8809ff3e40 20 rados_obj.operate() r=0 bl.length=362 2023-10-05T19:28:37.762 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.754+0000 7f8809ff3e40 20 rados->read ofs=0 len=0 2023-10-05T19:28:37.762 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.754+0000 7f8809ff3e40 1 -- 172.21.15.167:0/3210337137 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:41 5.13 5:cbbf6c60:::converted:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x5555a6a277a0 con 0x5555a6a28110 2023-10-05T19:28:37.762 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.754+0000 7f8807b28700 1 -- 172.21.15.167:0/3210337137 <== osd.3 v2:172.21.15.167:6808/30036 10 ==== osd_op_reply(41 converted [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 ==== 153+0+0 (crc 0 0 0) 0x7f87f8014070 con 0x5555a6a28110 2023-10-05T19:28:37.762 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.754+0000 7f8809ff3e40 20 rados_obj.operate() r=-2 bl.length=0 2023-10-05T19:28:37.762 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.754+0000 7f8809ff3e40 20 rados->read ofs=0 len=0 2023-10-05T19:28:37.762 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.754+0000 7f8809ff3e40 1 -- 172.21.15.167:0/3210337137 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:42 5.12 5:49953fa1:::default.realm:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x5555a6a2ae70 con 0x5555a69f72f0 2023-10-05T19:28:37.763 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.758+0000 7f8808329700 1 -- 172.21.15.167:0/3210337137 <== osd.1 v2:172.21.15.167:6800/30035 16 ==== osd_op_reply(42 default.realm [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 ==== 157+0+0 (crc 0 0 0) 0x7f880000a5a0 con 0x5555a69f72f0 2023-10-05T19:28:37.763 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.758+0000 7f8809ff3e40 20 rados_obj.operate() r=-2 bl.length=0 2023-10-05T19:28:37.763 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.758+0000 7f8809ff3e40 20 rados->read ofs=0 len=0 2023-10-05T19:28:37.763 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.758+0000 7f8809ff3e40 1 -- 172.21.15.167:0/3210337137 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:43 5.f 5:f4c53578:::zonegroups_names.default:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x5555a6a2ae70 con 0x5555a69f72f0 2023-10-05T19:28:37.763 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.758+0000 7f8808329700 1 -- 172.21.15.167:0/3210337137 <== osd.1 v2:172.21.15.167:6800/30035 17 ==== osd_op_reply(43 zonegroups_names.default [read 0~46 out=46b] v0'0 uv2 ondisk = 0) v8 ==== 168+0+46 (crc 0 0 0) 0x7f880000a5a0 con 0x5555a69f72f0 2023-10-05T19:28:37.763 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.758+0000 7f8809ff3e40 20 rados_obj.operate() r=0 bl.length=46 2023-10-05T19:28:37.764 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.758+0000 7f8809ff3e40 1 -- 172.21.15.167:0/3210337137 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:44 5.0 5:00000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x5555a6a2be80 con 0x5555a6a28110 2023-10-05T19:28:37.766 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.762+0000 7f8807b28700 1 -- 172.21.15.167:0/3210337137 <== osd.3 v2:172.21.15.167:6808/30036 11 ==== osd_op_reply(44 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f87f8014070 con 0x5555a6a28110 2023-10-05T19:28:37.766 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.762+0000 7f8807b28700 1 -- 172.21.15.167:0/3210337137 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:45 5.10 5:08000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x5555a6a2be80 con 0x7f87f800d750 2023-10-05T19:28:37.770 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.762+0000 7f8807327700 1 -- 172.21.15.167:0/3210337137 <== osd.2 v2:172.21.15.167:6816/30037 7 ==== osd_op_reply(45 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f87f0012450 con 0x7f87f800d750 2023-10-05T19:28:37.770 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.762+0000 7f8807327700 1 -- 172.21.15.167:0/3210337137 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:46 5.8 5:10000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x5555a6a2be80 con 0x7f87f0007030 2023-10-05T19:28:37.770 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.762+0000 7f8808329700 1 -- 172.21.15.167:0/3210337137 <== osd.0 v2:172.21.15.118:6800/27690 11 ==== osd_op_reply(46 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f880000a5a0 con 0x7f87f0007030 2023-10-05T19:28:37.770 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.762+0000 7f8808329700 1 -- 172.21.15.167:0/3210337137 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:47 5.18 5:18000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x5555a6a26e10 con 0x5555a69f72f0 2023-10-05T19:28:37.770 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.762+0000 7f8808329700 1 -- 172.21.15.167:0/3210337137 <== osd.1 v2:172.21.15.167:6800/30035 18 ==== osd_op_reply(47 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f880000a5a0 con 0x5555a69f72f0 2023-10-05T19:28:37.771 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.762+0000 7f8808329700 1 -- 172.21.15.167:0/3210337137 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:48 5.4 5:20000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x5555a6a26e10 con 0x5555a69f72f0 2023-10-05T19:28:37.771 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.762+0000 7f8808329700 1 -- 172.21.15.167:0/3210337137 <== osd.1 v2:172.21.15.167:6800/30035 19 ==== osd_op_reply(48 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f880000a5a0 con 0x5555a69f72f0 2023-10-05T19:28:37.771 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.762+0000 7f8808329700 1 -- 172.21.15.167:0/3210337137 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:49 5.14 5:28000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x5555a6a26e10 con 0x7f87f0007030 2023-10-05T19:28:37.771 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.766+0000 7f8808329700 1 -- 172.21.15.167:0/3210337137 <== osd.0 v2:172.21.15.118:6800/27690 12 ==== osd_op_reply(49 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f880000a5a0 con 0x7f87f0007030 2023-10-05T19:28:37.771 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.766+0000 7f8808329700 1 -- 172.21.15.167:0/3210337137 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:50 5.c 5:30000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x5555a6a26e10 con 0x7f87f0007030 2023-10-05T19:28:37.772 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.766+0000 7f8808329700 1 -- 172.21.15.167:0/3210337137 <== osd.0 v2:172.21.15.118:6800/27690 13 ==== osd_op_reply(50 [pgnls start_epoch 30 out=112b] v23'1 uv1 ondisk = 1) v8 ==== 144+0+112 (crc 0 0 0) 0x7f880000a5a0 con 0x7f87f0007030 2023-10-05T19:28:37.772 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.766+0000 7f8808329700 1 -- 172.21.15.167:0/3210337137 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:51 5.1c 5:38000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x5555a6a26e10 con 0x7f87f800d750 2023-10-05T19:28:37.772 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.766+0000 7f8807327700 1 -- 172.21.15.167:0/3210337137 <== osd.2 v2:172.21.15.167:6816/30037 8 ==== osd_op_reply(51 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f87f0012450 con 0x7f87f800d750 2023-10-05T19:28:37.772 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.766+0000 7f8807327700 1 -- 172.21.15.167:0/3210337137 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:52 5.2 5:40000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x5555a6a26e10 con 0x5555a6a28110 2023-10-05T19:28:37.773 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.766+0000 7f8807b28700 1 -- 172.21.15.167:0/3210337137 <== osd.3 v2:172.21.15.167:6808/30036 12 ==== osd_op_reply(52 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f87f8014070 con 0x5555a6a28110 2023-10-05T19:28:37.774 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.766+0000 7f8807b28700 1 -- 172.21.15.167:0/3210337137 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:53 5.12 5:48000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x5555a6a26e10 con 0x5555a69f72f0 2023-10-05T19:28:37.774 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.770+0000 7f8808329700 1 -- 172.21.15.167:0/3210337137 <== osd.1 v2:172.21.15.167:6800/30035 20 ==== osd_op_reply(53 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f880000a5a0 con 0x5555a69f72f0 2023-10-05T19:28:37.774 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.770+0000 7f8808329700 1 -- 172.21.15.167:0/3210337137 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:54 5.a 5:50000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x5555a6a26e10 con 0x7f87f0007030 2023-10-05T19:28:37.775 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.770+0000 7f8808329700 1 -- 172.21.15.167:0/3210337137 <== osd.0 v2:172.21.15.118:6800/27690 14 ==== osd_op_reply(54 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f880000a5a0 con 0x7f87f0007030 2023-10-05T19:28:37.775 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.770+0000 7f8808329700 1 -- 172.21.15.167:0/3210337137 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:55 5.1a 5:58000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x5555a6a26e10 con 0x5555a69f72f0 2023-10-05T19:28:37.776 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.770+0000 7f8808329700 1 -- 172.21.15.167:0/3210337137 <== osd.1 v2:172.21.15.167:6800/30035 21 ==== osd_op_reply(55 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f880000a5a0 con 0x5555a69f72f0 2023-10-05T19:28:37.776 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.770+0000 7f8808329700 1 -- 172.21.15.167:0/3210337137 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:56 5.6 5:60000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x5555a6a26e10 con 0x7f87f800d750 2023-10-05T19:28:37.777 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.770+0000 7f8807327700 1 -- 172.21.15.167:0/3210337137 <== osd.2 v2:172.21.15.167:6816/30037 9 ==== osd_op_reply(56 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f87f0012450 con 0x7f87f800d750 2023-10-05T19:28:37.777 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.770+0000 7f8807327700 1 -- 172.21.15.167:0/3210337137 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:57 5.16 5:68000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x5555a6a26e10 con 0x5555a6a28110 2023-10-05T19:28:37.777 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.770+0000 7f8807b28700 1 -- 172.21.15.167:0/3210337137 <== osd.3 v2:172.21.15.167:6808/30036 13 ==== osd_op_reply(57 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f87f8014070 con 0x5555a6a28110 2023-10-05T19:28:37.778 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.770+0000 7f8807b28700 1 -- 172.21.15.167:0/3210337137 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:58 5.e 5:70000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x5555a6a26e10 con 0x5555a6a28110 2023-10-05T19:28:37.778 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.774+0000 7f8807b28700 1 -- 172.21.15.167:0/3210337137 <== osd.3 v2:172.21.15.167:6808/30036 14 ==== osd_op_reply(58 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f87f8014070 con 0x5555a6a28110 2023-10-05T19:28:37.778 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.774+0000 7f8807b28700 1 -- 172.21.15.167:0/3210337137 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:59 5.1e 5:78000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x5555a6a26e10 con 0x7f87f0007030 2023-10-05T19:28:37.779 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.774+0000 7f8808329700 1 -- 172.21.15.167:0/3210337137 <== osd.0 v2:172.21.15.118:6800/27690 15 ==== osd_op_reply(59 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f880003f040 con 0x7f87f0007030 2023-10-05T19:28:37.779 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.774+0000 7f8808329700 1 -- 172.21.15.167:0/3210337137 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:60 5.1 5:80000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x5555a6a26e10 con 0x5555a6a28110 2023-10-05T19:28:37.780 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.774+0000 7f8807b28700 1 -- 172.21.15.167:0/3210337137 <== osd.3 v2:172.21.15.167:6808/30036 15 ==== osd_op_reply(60 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f87f8014070 con 0x5555a6a28110 2023-10-05T19:28:37.780 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.774+0000 7f8807b28700 1 -- 172.21.15.167:0/3210337137 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:61 5.11 5:88000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x5555a6a26e10 con 0x5555a6a28110 2023-10-05T19:28:37.780 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.774+0000 7f8807b28700 1 -- 172.21.15.167:0/3210337137 <== osd.3 v2:172.21.15.167:6808/30036 16 ==== osd_op_reply(61 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f87f8014070 con 0x5555a6a28110 2023-10-05T19:28:37.781 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.774+0000 7f8807b28700 1 -- 172.21.15.167:0/3210337137 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:62 5.9 5:90000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x5555a6a26e10 con 0x5555a69f72f0 2023-10-05T19:28:37.781 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.774+0000 7f8808329700 1 -- 172.21.15.167:0/3210337137 <== osd.1 v2:172.21.15.167:6800/30035 22 ==== osd_op_reply(62 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f880003f040 con 0x5555a69f72f0 2023-10-05T19:28:37.781 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.774+0000 7f8808329700 1 -- 172.21.15.167:0/3210337137 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:63 5.19 5:98000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x5555a6a26e10 con 0x5555a69f72f0 2023-10-05T19:28:37.782 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.778+0000 7f8808329700 1 -- 172.21.15.167:0/3210337137 <== osd.1 v2:172.21.15.167:6800/30035 23 ==== osd_op_reply(63 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f880003f040 con 0x5555a69f72f0 2023-10-05T19:28:37.782 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.778+0000 7f8808329700 1 -- 172.21.15.167:0/3210337137 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:64 5.5 5:a0000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x5555a6a26e10 con 0x7f87f0007030 2023-10-05T19:28:37.783 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.778+0000 7f8808329700 1 -- 172.21.15.167:0/3210337137 <== osd.0 v2:172.21.15.118:6800/27690 16 ==== osd_op_reply(64 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f880003f040 con 0x7f87f0007030 2023-10-05T19:28:37.783 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.778+0000 7f8808329700 1 -- 172.21.15.167:0/3210337137 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:65 5.15 5:a8000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x5555a6a26e10 con 0x5555a6a28110 2023-10-05T19:28:37.784 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.778+0000 7f8807b28700 1 -- 172.21.15.167:0/3210337137 <== osd.3 v2:172.21.15.167:6808/30036 17 ==== osd_op_reply(65 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f87f8014070 con 0x5555a6a28110 2023-10-05T19:28:37.784 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.778+0000 7f8807b28700 1 -- 172.21.15.167:0/3210337137 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:66 5.d 5:b0000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x5555a6a26e10 con 0x7f87f800d750 2023-10-05T19:28:37.785 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.778+0000 7f8807327700 1 -- 172.21.15.167:0/3210337137 <== osd.2 v2:172.21.15.167:6816/30037 10 ==== osd_op_reply(66 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f87f0012450 con 0x7f87f800d750 2023-10-05T19:28:37.785 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.778+0000 7f8807327700 1 -- 172.21.15.167:0/3210337137 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:67 5.1d 5:b8000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x5555a6a26e10 con 0x5555a69f72f0 2023-10-05T19:28:37.786 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.778+0000 7f8808329700 1 -- 172.21.15.167:0/3210337137 <== osd.1 v2:172.21.15.167:6800/30035 24 ==== osd_op_reply(67 [pgnls start_epoch 30 out=107b] v30'2 uv2 ondisk = 1) v8 ==== 144+0+107 (crc 0 0 0) 0x7f880003f040 con 0x5555a69f72f0 2023-10-05T19:28:37.786 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.778+0000 7f8808329700 1 -- 172.21.15.167:0/3210337137 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:68 5.3 5:c0000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x5555a6a26e10 con 0x7f87f0007030 2023-10-05T19:28:37.787 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.782+0000 7f8808329700 1 -- 172.21.15.167:0/3210337137 <== osd.0 v2:172.21.15.118:6800/27690 17 ==== osd_op_reply(68 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f880003f040 con 0x7f87f0007030 2023-10-05T19:28:37.787 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.782+0000 7f8808329700 1 -- 172.21.15.167:0/3210337137 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:69 5.13 5:c8000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x5555a6a26e10 con 0x5555a6a28110 2023-10-05T19:28:37.787 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.782+0000 7f8807b28700 1 -- 172.21.15.167:0/3210337137 <== osd.3 v2:172.21.15.167:6808/30036 18 ==== osd_op_reply(69 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f87f8014070 con 0x5555a6a28110 2023-10-05T19:28:37.788 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.782+0000 7f8807b28700 1 -- 172.21.15.167:0/3210337137 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:70 5.b 5:d0000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x5555a6a26e10 con 0x7f87f0007030 2023-10-05T19:28:37.788 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.782+0000 7f8808329700 1 -- 172.21.15.167:0/3210337137 <== osd.0 v2:172.21.15.118:6800/27690 18 ==== osd_op_reply(70 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f880003f040 con 0x7f87f0007030 2023-10-05T19:28:37.789 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.782+0000 7f8808329700 1 -- 172.21.15.167:0/3210337137 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:71 5.1b 5:d8000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x5555a6a26e10 con 0x7f87f800d750 2023-10-05T19:28:37.789 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.782+0000 7f8807327700 1 -- 172.21.15.167:0/3210337137 <== osd.2 v2:172.21.15.167:6816/30037 11 ==== osd_op_reply(71 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f87f0012450 con 0x7f87f800d750 2023-10-05T19:28:37.789 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.782+0000 7f8807327700 1 -- 172.21.15.167:0/3210337137 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:72 5.7 5:e0000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x5555a6a26e10 con 0x5555a6a28110 2023-10-05T19:28:37.790 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.786+0000 7f8807b28700 1 -- 172.21.15.167:0/3210337137 <== osd.3 v2:172.21.15.167:6808/30036 19 ==== osd_op_reply(72 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f87f8014070 con 0x5555a6a28110 2023-10-05T19:28:37.790 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.786+0000 7f8807b28700 1 -- 172.21.15.167:0/3210337137 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:73 5.17 5:e8000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x5555a6a26e10 con 0x7f87f0007030 2023-10-05T19:28:37.791 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.786+0000 7f8808329700 1 -- 172.21.15.167:0/3210337137 <== osd.0 v2:172.21.15.118:6800/27690 19 ==== osd_op_reply(73 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f880003f040 con 0x7f87f0007030 2023-10-05T19:28:37.791 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.786+0000 7f8808329700 1 -- 172.21.15.167:0/3210337137 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:74 5.f 5:f0000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x5555a6a26e10 con 0x5555a69f72f0 2023-10-05T19:28:37.792 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.786+0000 7f8808329700 1 -- 172.21.15.167:0/3210337137 <== osd.1 v2:172.21.15.167:6800/30035 25 ==== osd_op_reply(74 [pgnls start_epoch 30 out=115b] v23'2 uv2 ondisk = 1) v8 ==== 144+0+115 (crc 0 0 0) 0x7f880003f040 con 0x5555a69f72f0 2023-10-05T19:28:37.792 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.786+0000 7f8808329700 1 -- 172.21.15.167:0/3210337137 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:75 5.1f 5:f8000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x5555a6a26e10 con 0x7f87f800d750 2023-10-05T19:28:37.793 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.786+0000 7f8807327700 1 -- 172.21.15.167:0/3210337137 <== osd.2 v2:172.21.15.167:6816/30037 12 ==== osd_op_reply(75 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f87f0012450 con 0x7f87f800d750 2023-10-05T19:28:37.793 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.786+0000 7f8809ff3e40 20 RGWRados::pool_iterate: got zonegroup_info.4f1f719d-9648-417b-b10c-e5d939825419 2023-10-05T19:28:37.793 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.786+0000 7f8809ff3e40 20 RGWRados::pool_iterate: got zone_info.9a9df965-6612-45d4-a45f-6e3c889bacc2 2023-10-05T19:28:37.794 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.786+0000 7f8809ff3e40 20 RGWRados::pool_iterate: got zone_names.default 2023-10-05T19:28:37.794 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.786+0000 7f8809ff3e40 20 RGWRados::pool_iterate: got zonegroups_names.default 2023-10-05T19:28:37.794 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.786+0000 7f8809ff3e40 20 rados->read ofs=0 len=0 2023-10-05T19:28:37.794 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.786+0000 7f8809ff3e40 1 -- 172.21.15.167:0/3210337137 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:76 5.f 5:f43b6ece:::zone_names.default:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x5555a6a2ae70 con 0x5555a69f72f0 2023-10-05T19:28:37.795 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.786+0000 7f8808329700 1 -- 172.21.15.167:0/3210337137 <== osd.1 v2:172.21.15.167:6800/30035 26 ==== osd_op_reply(76 zone_names.default [read 0~46 out=46b] v0'0 uv1 ondisk = 0) v8 ==== 162+0+46 (crc 0 0 0) 0x7f880003f040 con 0x5555a69f72f0 2023-10-05T19:28:37.795 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.786+0000 7f8809ff3e40 20 rados_obj.operate() r=0 bl.length=46 2023-10-05T19:28:37.795 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.790+0000 7f8809ff3e40 20 rados->read ofs=0 len=0 2023-10-05T19:28:37.795 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.790+0000 7f8809ff3e40 1 -- 172.21.15.167:0/3210337137 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:77 5.1d 5:b8374eb4:::zone_info.9a9df965-6612-45d4-a45f-6e3c889bacc2:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x5555a6a2b220 con 0x5555a69f72f0 2023-10-05T19:28:37.795 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.790+0000 7f8808329700 1 -- 172.21.15.167:0/3210337137 <== osd.1 v2:172.21.15.167:6800/30035 27 ==== osd_op_reply(77 zone_info.9a9df965-6612-45d4-a45f-6e3c889bacc2 [read 0~874 out=874b] v0'0 uv2 ondisk = 0) v8 ==== 190+0+874 (crc 0 0 0) 0x7f880003f040 con 0x5555a69f72f0 2023-10-05T19:28:37.795 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.790+0000 7f8809ff3e40 20 rados_obj.operate() r=0 bl.length=874 2023-10-05T19:28:37.796 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.790+0000 7f8809ff3e40 20 rados->read ofs=0 len=0 2023-10-05T19:28:37.796 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.790+0000 7f8809ff3e40 1 -- 172.21.15.167:0/3210337137 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:78 5.f 5:f4c53578:::zonegroups_names.default:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x5555a6a2b710 con 0x5555a69f72f0 2023-10-05T19:28:37.797 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.790+0000 7f8808329700 1 -- 172.21.15.167:0/3210337137 <== osd.1 v2:172.21.15.167:6800/30035 28 ==== osd_op_reply(78 zonegroups_names.default [read 0~46 out=46b] v0'0 uv2 ondisk = 0) v8 ==== 168+0+46 (crc 0 0 0) 0x7f880003f040 con 0x5555a69f72f0 2023-10-05T19:28:37.797 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.790+0000 7f8809ff3e40 20 rados_obj.operate() r=0 bl.length=46 2023-10-05T19:28:37.797 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.790+0000 7f8809ff3e40 20 rados->read ofs=0 len=0 2023-10-05T19:28:37.797 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.790+0000 7f8809ff3e40 1 -- 172.21.15.167:0/3210337137 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:79 5.c 5:30d299a5:::zonegroup_info.4f1f719d-9648-417b-b10c-e5d939825419:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x5555a6a2b710 con 0x7f87f0007030 2023-10-05T19:28:37.797 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.790+0000 7f8808329700 1 -- 172.21.15.167:0/3210337137 <== osd.0 v2:172.21.15.118:6800/27690 20 ==== osd_op_reply(79 zonegroup_info.4f1f719d-9648-417b-b10c-e5d939825419 [read 0~362 out=362b] v0'0 uv1 ondisk = 0) v8 ==== 195+0+362 (crc 0 0 0) 0x7f8800052040 con 0x7f87f0007030 2023-10-05T19:28:37.797 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.790+0000 7f8809ff3e40 20 rados_obj.operate() r=0 bl.length=362 2023-10-05T19:28:37.797 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.790+0000 7f8809ff3e40 20 rados->read ofs=0 len=0 2023-10-05T19:28:37.797 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.790+0000 7f8809ff3e40 1 -- 172.21.15.167:0/3210337137 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:80 5.3 5:c0ad8cbd:::region_map:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x5555a6a2b710 con 0x7f87f0007030 2023-10-05T19:28:37.798 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.790+0000 7f8808329700 1 -- 172.21.15.167:0/3210337137 <== osd.0 v2:172.21.15.118:6800/27690 21 ==== osd_op_reply(80 region_map [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 ==== 154+0+0 (crc 0 0 0) 0x7f8800052040 con 0x7f87f0007030 2023-10-05T19:28:37.798 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.790+0000 7f8809ff3e40 20 rados_obj.operate() r=-2 bl.length=0 2023-10-05T19:28:37.799 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.790+0000 7f8809ff3e40 20 zone default found 2023-10-05T19:28:37.799 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.790+0000 7f8809ff3e40 4 Realm: () 2023-10-05T19:28:37.799 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.790+0000 7f8809ff3e40 4 ZoneGroup: default (4f1f719d-9648-417b-b10c-e5d939825419) 2023-10-05T19:28:37.799 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.790+0000 7f8809ff3e40 4 Zone: default (9a9df965-6612-45d4-a45f-6e3c889bacc2) 2023-10-05T19:28:37.799 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.790+0000 7f8809ff3e40 10 cannot find current period zonegroup using local zonegroup configuration 2023-10-05T19:28:37.799 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.790+0000 7f8809ff3e40 20 zonegroup default 2023-10-05T19:28:37.799 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.790+0000 7f8809ff3e40 20 rados->read ofs=0 len=0 2023-10-05T19:28:37.799 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.790+0000 7f8809ff3e40 1 -- 172.21.15.167:0/3210337137 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:81 5.3 5:c52100b6:::period_config.default:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x5555a6a2b710 con 0x7f87f0007030 2023-10-05T19:28:37.799 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.794+0000 7f8808329700 1 -- 172.21.15.167:0/3210337137 <== osd.0 v2:172.21.15.118:6800/27690 22 ==== osd_op_reply(81 period_config.default [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 ==== 165+0+0 (crc 0 0 0) 0x7f8800052040 con 0x7f87f0007030 2023-10-05T19:28:37.799 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.794+0000 7f8809ff3e40 20 rados_obj.operate() r=-2 bl.length=0 2023-10-05T19:28:37.800 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.794+0000 7f8809ff3e40 20 rados->read ofs=0 len=0 2023-10-05T19:28:37.800 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.794+0000 7f8809ff3e40 1 -- 172.21.15.167:0/3210337137 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:82 6.2 6:4347d321:::bucket.sync-source-hints.:head [call version.read in=11b,read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x5555a6a2d090 con 0x5555a6a28110 2023-10-05T19:28:37.801 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.794+0000 7f8807b28700 1 -- 172.21.15.167:0/3210337137 <== osd.3 v2:172.21.15.167:6808/30036 20 ==== osd_op_reply(82 bucket.sync-source-hints. [call,read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 ==== 211+0+0 (crc 0 0 0) 0x7f87f8014070 con 0x5555a6a28110 2023-10-05T19:28:37.801 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.794+0000 7f8809ff3e40 20 rados_obj.operate() r=-2 bl.length=0 2023-10-05T19:28:37.801 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.794+0000 7f8809ff3e40 20 rados->read ofs=0 len=0 2023-10-05T19:28:37.801 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.794+0000 7f8809ff3e40 1 -- 172.21.15.167:0/3210337137 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:83 6.b 6:d467b91b:::bucket.sync-target-hints.:head [call version.read in=11b,read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x5555a6a2d440 con 0x5555a6a28110 2023-10-05T19:28:37.801 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.794+0000 7f8807b28700 1 -- 172.21.15.167:0/3210337137 <== osd.3 v2:172.21.15.167:6808/30036 21 ==== osd_op_reply(83 bucket.sync-target-hints. [call,read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 ==== 211+0+0 (crc 0 0 0) 0x7f87f8014070 con 0x5555a6a28110 2023-10-05T19:28:37.801 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.794+0000 7f8809ff3e40 20 rados_obj.operate() r=-2 bl.length=0 2023-10-05T19:28:37.801 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.794+0000 7f8809ff3e40 20 started sync module instance, tier type = 2023-10-05T19:28:37.801 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.794+0000 7f8809ff3e40 20 started zone id=9a9df965-6612-45d4-a45f-6e3c889bacc2 (name=default) with tier type = 2023-10-05T19:28:37.801 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.794+0000 7f8809ff3e40 1 -- 172.21.15.167:0/3210337137 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:84 7.1f 7:f95f44c2:::notify.0:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x5555a6a2c750 con 0x7f87f800d750 2023-10-05T19:28:37.803 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.798+0000 7f8807327700 1 -- 172.21.15.167:0/3210337137 <== osd.2 v2:172.21.15.167:6816/30037 13 ==== osd_op_reply(84 notify.0 [create] v30'171 uv171 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f87f0012450 con 0x7f87f800d750 2023-10-05T19:28:37.803 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.798+0000 7f8809ff3e40 1 -- 172.21.15.167:0/3210337137 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:85 7.1f 7:f95f44c2:::notify.0:head [watch watch cookie 93826356265568] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x5555a6a2d7f0 con 0x7f87f800d750 2023-10-05T19:28:37.803 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.798+0000 7f8809ff3e40 1 -- 172.21.15.167:0/3210337137 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:86 7.0 7:05bf5b68:::notify.1:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x5555a6a2e1c0 con 0x5555a69f72f0 2023-10-05T19:28:37.806 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.802+0000 7f8808329700 1 -- 172.21.15.167:0/3210337137 <== osd.1 v2:172.21.15.167:6800/30035 29 ==== osd_op_reply(86 notify.1 [create] v30'171 uv171 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f8800052040 con 0x5555a69f72f0 2023-10-05T19:28:37.807 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.802+0000 7f8809ff3e40 1 -- 172.21.15.167:0/3210337137 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:87 7.0 7:05bf5b68:::notify.1:head [watch watch cookie 93826356272272] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x5555a6a2eab0 con 0x5555a69f72f0 2023-10-05T19:28:37.807 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.802+0000 7f8809ff3e40 1 -- 172.21.15.167:0/3210337137 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:88 7.15 7:a93a5511:::notify.2:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x5555a6a2f350 con 0x5555a6a28110 2023-10-05T19:28:37.807 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.802+0000 7f8807327700 1 -- 172.21.15.167:0/3210337137 <== osd.2 v2:172.21.15.167:6816/30037 14 ==== osd_op_reply(85 notify.0 [watch watch cookie 93826356265568] v30'172 uv171 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f87f0012450 con 0x7f87f800d750 2023-10-05T19:28:37.809 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.802+0000 7f8807b28700 1 -- 172.21.15.167:0/3210337137 <== osd.3 v2:172.21.15.167:6808/30036 22 ==== osd_op_reply(88 notify.2 [create] v30'171 uv171 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f87f8014070 con 0x5555a6a28110 2023-10-05T19:28:37.809 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.802+0000 7f8809ff3e40 1 -- 172.21.15.167:0/3210337137 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:89 7.15 7:a93a5511:::notify.2:head [watch watch cookie 93826356275888] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x5555a6a2cc20 con 0x5555a6a28110 2023-10-05T19:28:37.810 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.802+0000 7f8809ff3e40 1 -- 172.21.15.167:0/3210337137 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:90 7.e 7:7759931f:::notify.3:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x5555a6a300e0 con 0x7f87f800d750 2023-10-05T19:28:37.810 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.806+0000 7f8808329700 1 -- 172.21.15.167:0/3210337137 <== osd.1 v2:172.21.15.167:6800/30035 30 ==== osd_op_reply(87 notify.1 [watch watch cookie 93826356272272] v30'172 uv171 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f8800052040 con 0x5555a69f72f0 2023-10-05T19:28:37.812 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.806+0000 7f8807327700 1 -- 172.21.15.167:0/3210337137 <== osd.2 v2:172.21.15.167:6816/30037 15 ==== osd_op_reply(90 notify.3 [create] v30'171 uv171 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f87f0012450 con 0x7f87f800d750 2023-10-05T19:28:37.812 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.806+0000 7f8807b28700 1 -- 172.21.15.167:0/3210337137 <== osd.3 v2:172.21.15.167:6808/30036 23 ==== osd_op_reply(89 notify.2 [watch watch cookie 93826356275888] v30'172 uv171 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f87f8014070 con 0x5555a6a28110 2023-10-05T19:28:37.812 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.806+0000 7f8809ff3e40 1 -- 172.21.15.167:0/3210337137 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:91 7.e 7:7759931f:::notify.3:head [watch watch cookie 93826356274624] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x5555a6a2fd10 con 0x7f87f800d750 2023-10-05T19:28:37.812 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.806+0000 7f8809ff3e40 1 -- 172.21.15.167:0/3210337137 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:92 7.d 7:b4812045:::notify.4:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x5555a6a30af0 con 0x5555a69f72f0 2023-10-05T19:28:37.814 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.810+0000 7f8807327700 1 -- 172.21.15.167:0/3210337137 <== osd.2 v2:172.21.15.167:6816/30037 16 ==== osd_op_reply(91 notify.3 [watch watch cookie 93826356274624] v30'172 uv171 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f87f0012450 con 0x7f87f800d750 2023-10-05T19:28:37.815 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.810+0000 7f8808329700 1 -- 172.21.15.167:0/3210337137 <== osd.1 v2:172.21.15.167:6800/30035 31 ==== osd_op_reply(92 notify.4 [create] v30'171 uv171 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f8800052040 con 0x5555a69f72f0 2023-10-05T19:28:37.815 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.810+0000 7f8809ff3e40 1 -- 172.21.15.167:0/3210337137 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:93 7.d 7:b4812045:::notify.4:head [watch watch cookie 93826356282880] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x5555a6a30a50 con 0x5555a69f72f0 2023-10-05T19:28:37.815 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.810+0000 7f8809ff3e40 1 -- 172.21.15.167:0/3210337137 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:94 7.3 7:c609908c:::notify.5:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x5555a6a318d0 con 0x7f87f0007030 2023-10-05T19:28:37.818 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.810+0000 7f8808329700 1 -- 172.21.15.167:0/3210337137 <== osd.0 v2:172.21.15.118:6800/27690 23 ==== osd_op_reply(94 notify.5 [create] v30'171 uv171 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f8800052040 con 0x7f87f0007030 2023-10-05T19:28:37.818 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.810+0000 7f8809ff3e40 1 -- 172.21.15.167:0/3210337137 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:95 7.3 7:c609908c:::notify.5:head [watch watch cookie 93826356286432] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x5555a6a31d20 con 0x7f87f0007030 2023-10-05T19:28:37.818 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.810+0000 7f8809ff3e40 1 -- 172.21.15.167:0/3210337137 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:96 7.14 7:2b04a3e9:::notify.6:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x5555a6a327a0 con 0x7f87f0007030 2023-10-05T19:28:37.818 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.814+0000 7f8808329700 1 -- 172.21.15.167:0/3210337137 <== osd.1 v2:172.21.15.167:6800/30035 32 ==== osd_op_reply(93 notify.4 [watch watch cookie 93826356282880] v30'172 uv171 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f8800052040 con 0x5555a69f72f0 2023-10-05T19:28:37.821 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.814+0000 7f8808329700 1 -- 172.21.15.167:0/3210337137 <== osd.0 v2:172.21.15.118:6800/27690 24 ==== osd_op_reply(95 notify.5 [watch watch cookie 93826356286432] v30'172 uv171 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f8800052040 con 0x7f87f0007030 2023-10-05T19:28:37.821 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.814+0000 7f8808329700 1 -- 172.21.15.167:0/3210337137 <== osd.0 v2:172.21.15.118:6800/27690 25 ==== osd_op_reply(96 notify.6 [create] v30'171 uv171 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f8800052040 con 0x7f87f0007030 2023-10-05T19:28:37.821 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.814+0000 7f8809ff3e40 1 -- 172.21.15.167:0/3210337137 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:97 7.14 7:2b04a3e9:::notify.6:head [watch watch cookie 93826356285168] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x5555a6a31830 con 0x7f87f0007030 2023-10-05T19:28:37.821 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.814+0000 7f8809ff3e40 1 -- 172.21.15.167:0/3210337137 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:98 7.9 7:93e5b521:::notify.7:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x5555a6a32ec0 con 0x7f87f0007030 2023-10-05T19:28:37.823 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.818+0000 7f8808329700 1 -- 172.21.15.167:0/3210337137 <== osd.0 v2:172.21.15.118:6800/27690 26 ==== osd_op_reply(97 notify.6 [watch watch cookie 93826356285168] v30'172 uv171 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f8800052040 con 0x7f87f0007030 2023-10-05T19:28:37.823 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.818+0000 7f8808329700 1 -- 172.21.15.167:0/3210337137 <== osd.0 v2:172.21.15.118:6800/27690 27 ==== osd_op_reply(98 notify.7 [create] v30'171 uv171 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f8800052040 con 0x7f87f0007030 2023-10-05T19:28:37.824 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.818+0000 7f8809ff3e40 1 -- 172.21.15.167:0/3210337137 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:99 7.9 7:93e5b521:::notify.7:head [watch watch cookie 93826356290160] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x5555a6a32ec0 con 0x7f87f0007030 2023-10-05T19:28:37.824 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.818+0000 7f8809ff3e40 20 add_watcher() i=0 2023-10-05T19:28:37.824 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.818+0000 7f8809ff3e40 20 add_watcher() i=1 2023-10-05T19:28:37.824 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.818+0000 7f8809ff3e40 20 add_watcher() i=2 2023-10-05T19:28:37.824 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.818+0000 7f8809ff3e40 20 add_watcher() i=3 2023-10-05T19:28:37.824 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.818+0000 7f8809ff3e40 20 add_watcher() i=4 2023-10-05T19:28:37.824 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.818+0000 7f8809ff3e40 20 add_watcher() i=5 2023-10-05T19:28:37.824 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.818+0000 7f8809ff3e40 20 add_watcher() i=6 2023-10-05T19:28:37.826 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.818+0000 7f8808329700 1 -- 172.21.15.167:0/3210337137 <== osd.0 v2:172.21.15.118:6800/27690 28 ==== osd_op_reply(99 notify.7 [watch watch cookie 93826356290160] v30'172 uv171 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f8800052040 con 0x7f87f0007030 2023-10-05T19:28:37.826 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.822+0000 7f8809ff3e40 20 add_watcher() i=7 2023-10-05T19:28:37.826 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.822+0000 7f8809ff3e40 2 all 8 watchers are set, enabling cache 2023-10-05T19:28:37.826 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.822+0000 7f8809ff3e40 1 -- 172.21.15.167:0/3210337137 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:100 6.4 6:22d26bf9:::data_loggenerations_metadata:head [call version.check_conds in=74b,call version.read in=11b,read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x5555a6a35ef0 con 0x5555a69f72f0 2023-10-05T19:28:37.827 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.822+0000 7f8808329700 1 -- 172.21.15.167:0/3210337137 <== osd.1 v2:172.21.15.167:6800/30035 33 ==== osd_op_reply(100 data_loggenerations_metadata [call,call out=48b,read 0~28 out=28b] v0'0 uv1 ondisk = 0) v8 ==== 256+0+76 (crc 0 0 0) 0x7f8800052040 con 0x5555a69f72f0 2023-10-05T19:28:37.827 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.822+0000 7f8809ff3e40 1 -- 172.21.15.167:0/3210337137 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:101 6.4 6:22d26bf9:::data_loggenerations_metadata:head [watch watch cookie 93826356292048] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x5555a6a333b0 con 0x5555a69f72f0 2023-10-05T19:28:37.829 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.822+0000 7f8808329700 1 -- 172.21.15.167:0/3210337137 <== osd.1 v2:172.21.15.167:6800/30035 34 ==== osd_op_reply(101 data_loggenerations_metadata [watch watch cookie 93826356292048] v30'161 uv1 ondisk = 0) v8 ==== 172+0+0 (crc 0 0 0) 0x7f8800052040 con 0x5555a69f72f0 2023-10-05T19:28:37.829 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.822+0000 7f87d97e2700 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2023-10-05T19:28:37.829 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.822+0000 7f8809ff3e40 20 check_secure_mon_conn(): auth registy supported: methods=[2] modes=[2,1] 2023-10-05T19:28:37.829 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.822+0000 7f8809ff3e40 20 check_secure_mon_conn(): mode 1 is insecure 2023-10-05T19:28:37.830 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.822+0000 7f8809ff3e40 1 Processor -- start 2023-10-05T19:28:37.830 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.822+0000 7f8809ff3e40 1 -- start start 2023-10-05T19:28:37.830 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.822+0000 7f8809ff3e40 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5555a6a40300 0x5555a6a406d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-05T19:28:37.831 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.822+0000 7f8809ff3e40 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x5555a6a40c10 con 0x5555a6a40300 2023-10-05T19:28:37.831 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.826+0000 7f8807327700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5555a6a40300 0x5555a6a406d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-05T19:28:37.831 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.826+0000 7f8807327700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5555a6a40300 0x5555a6a406d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.118:3300/0 says I am v2:172.21.15.167:42078/0 (socket says 172.21.15.167:42078) 2023-10-05T19:28:37.831 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.826+0000 7f8807327700 1 -- 172.21.15.167:0/2398539159 learned_addr learned my addr 172.21.15.167:0/2398539159 (peer_addr_for_me v2:172.21.15.167:0/0) 2023-10-05T19:28:37.832 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.826+0000 7f8807327700 1 -- 172.21.15.167:0/2398539159 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5555a6a40d50 con 0x5555a6a40300 2023-10-05T19:28:37.832 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.826+0000 7f8807327700 1 --2- 172.21.15.167:0/2398539159 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5555a6a40300 0x5555a6a406d0 secure :-1 s=READY pgs=480 cs=0 l=1 rev1=1 crypto rx=0x5555a6966c60 tx=0x7f87f0012000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5f917ed313f64c44 server_cookie=0 in_seq=0 out_seq=0 2023-10-05T19:28:37.832 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.826+0000 7f87d8fe1700 1 -- 172.21.15.167:0/2398539159 <== mon.0 v2:172.21.15.118:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f87f002a540 con 0x5555a6a40300 2023-10-05T19:28:37.832 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.826+0000 7f87d8fe1700 1 -- 172.21.15.167:0/2398539159 <== mon.0 v2:172.21.15.118:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f87f002a6c0 con 0x5555a6a40300 2023-10-05T19:28:37.832 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.826+0000 7f87d8fe1700 1 -- 172.21.15.167:0/2398539159 <== mon.0 v2:172.21.15.118:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f87f002a870 con 0x5555a6a40300 2023-10-05T19:28:37.833 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.826+0000 7f8809ff3e40 1 -- 172.21.15.167:0/2398539159 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5555a6a40300 msgr2=0x5555a6a406d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-05T19:28:37.833 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.826+0000 7f8809ff3e40 1 --2- 172.21.15.167:0/2398539159 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5555a6a40300 0x5555a6a406d0 secure :-1 s=READY pgs=480 cs=0 l=1 rev1=1 crypto rx=0x5555a6966c60 tx=0x7f87f0012000 comp rx=0 tx=0).stop 2023-10-05T19:28:37.834 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.826+0000 7f8809ff3e40 1 -- 172.21.15.167:0/2398539159 shutdown_connections 2023-10-05T19:28:37.834 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.826+0000 7f8809ff3e40 1 --2- 172.21.15.167:0/2398539159 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5555a6a40300 0x5555a6a406d0 unknown :-1 s=CLOSED pgs=480 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-05T19:28:37.834 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.826+0000 7f8809ff3e40 1 -- 172.21.15.167:0/2398539159 >> 172.21.15.167:0/2398539159 conn(0x5555a6a3b6c0 msgr2=0x5555a6a39870 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-05T19:28:37.834 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.826+0000 7f8809ff3e40 1 -- 172.21.15.167:0/2398539159 shutdown_connections 2023-10-05T19:28:37.834 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.826+0000 7f8809ff3e40 1 -- 172.21.15.167:0/2398539159 wait complete. 2023-10-05T19:28:37.834 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.826+0000 7f8809ff3e40 1 Processor -- start 2023-10-05T19:28:37.834 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.826+0000 7f8809ff3e40 1 -- start start 2023-10-05T19:28:37.836 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.826+0000 7f8809ff3e40 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5555a6a40300 0x5555a6a45dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-05T19:28:37.836 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.826+0000 7f8809ff3e40 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x5555a6a46dc0 con 0x5555a6a40300 2023-10-05T19:28:37.836 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.830+0000 7f8807327700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5555a6a40300 0x5555a6a45dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-05T19:28:37.836 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.830+0000 7f8807327700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5555a6a40300 0x5555a6a45dc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.118:3300/0 says I am v2:172.21.15.167:42082/0 (socket says 172.21.15.167:42082) 2023-10-05T19:28:37.836 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.830+0000 7f8807327700 1 -- 172.21.15.167:0/3649689687 learned_addr learned my addr 172.21.15.167:0/3649689687 (peer_addr_for_me v2:172.21.15.167:0/0) 2023-10-05T19:28:37.836 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.830+0000 7f8807327700 1 -- 172.21.15.167:0/3649689687 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5555a69fac20 con 0x5555a6a40300 2023-10-05T19:28:37.836 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.830+0000 7f8807327700 1 --2- 172.21.15.167:0/3649689687 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5555a6a40300 0x5555a6a45dc0 secure :-1 s=READY pgs=481 cs=0 l=1 rev1=1 crypto rx=0x5555a6a3a3e0 tx=0x7f87f002f000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-05T19:28:37.836 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.830+0000 7f8752ffd700 1 -- 172.21.15.167:0/3649689687 <== mon.0 v2:172.21.15.118:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f87f003eb20 con 0x5555a6a40300 2023-10-05T19:28:37.836 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.830+0000 7f8809ff3e40 1 -- 172.21.15.167:0/3649689687 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x5555a6a463c0 con 0x5555a6a40300 2023-10-05T19:28:37.836 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.830+0000 7f8809ff3e40 1 -- 172.21.15.167:0/3649689687 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x5555a6a46860 con 0x5555a6a40300 2023-10-05T19:28:37.837 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.830+0000 7f8752ffd700 1 -- 172.21.15.167:0/3649689687 <== mon.0 v2:172.21.15.118:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f87f003eca0 con 0x5555a6a40300 2023-10-05T19:28:37.837 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.830+0000 7f8752ffd700 1 -- 172.21.15.167:0/3649689687 <== mon.0 v2:172.21.15.118:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f87f003ee50 con 0x5555a6a40300 2023-10-05T19:28:37.837 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.830+0000 7f8752ffd700 1 -- 172.21.15.167:0/3649689687 <== mon.0 v2:172.21.15.118:3300/0 4 ==== mgrmap(e 5) v1 ==== 40329+0+0 (secure 0 0 0) 0x7f87f0051440 con 0x5555a6a40300 2023-10-05T19:28:37.838 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.830+0000 7f8752ffd700 1 --2- 172.21.15.167:0/3649689687 >> [v2:172.21.15.167:6824/29889,v1:172.21.15.167:6825/29889] conn(0x7f8748030bb0 0x7f8748033060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-05T19:28:37.838 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.830+0000 7f8752ffd700 1 -- 172.21.15.167:0/3649689687 <== mon.0 v2:172.21.15.118:3300/0 5 ==== osd_map(30..30 src has 1..30) v4 ==== 6057+0+0 (secure 0 0 0) 0x7f87f002f070 con 0x5555a6a40300 2023-10-05T19:28:37.838 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.834+0000 7f8808329700 1 --2- 172.21.15.167:0/3649689687 >> [v2:172.21.15.167:6824/29889,v1:172.21.15.167:6825/29889] conn(0x7f8748030bb0 0x7f8748033060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-05T19:28:37.838 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.834+0000 7f8809ff3e40 5 note: GC not initialized 2023-10-05T19:28:37.838 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.834+0000 7f8808329700 1 --2- 172.21.15.167:0/3649689687 >> [v2:172.21.15.167:6824/29889,v1:172.21.15.167:6825/29889] conn(0x7f8748030bb0 0x7f8748033060 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x5555a6a33310 tx=0x7f8800012e70 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-05T19:28:37.840 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.834+0000 7f8747fff700 20 reqs_thread_entry: start 2023-10-05T19:28:37.840 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.834+0000 7f8809ff3e40 20 init_complete bucket index max shards: 11 2023-10-05T19:28:37.840 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.834+0000 7f8745ffb700 20 reqs_thread_entry: start 2023-10-05T19:28:37.841 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.834+0000 7f8809ff3e40 10 rgw notify: Started notification manager with: 1 workers 2023-10-05T19:28:37.841 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.834+0000 7f8733fff700 20 rgw notify: INFO: next queues processing will happen at: Thu Oct 5 19:29:08 2023 2023-10-05T19:28:37.841 INFO:teuthology.orchestra.run.smithi167.stderr: 2023-10-05T19:28:37.842 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.834+0000 7f87337fe700 20 reqs_thread_entry: start 2023-10-05T19:28:37.842 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.834+0000 7f8809ff3e40 10 log_show_next pos 0 bl 0 off 0 eof 0 2023-10-05T19:28:37.842 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.834+0000 7f8809ff3e40 1 --2- 172.21.15.167:0/3649689687 >> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] conn(0x5555a6aad270 0x5555a6aaf650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-05T19:28:37.842 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.834+0000 7f8809ff3e40 1 -- 172.21.15.167:0/3649689687 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:1 6.9 6:926cbca3:::2023-10-05-19-9a9df965-6612-45d4-a45f-6e3c889bacc2.4223.2-myfoo2:head [read 0~1048576] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x5555a6aafb90 con 0x5555a6aad270 2023-10-05T19:28:37.842 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.834+0000 7f8807b28700 1 --2- 172.21.15.167:0/3649689687 >> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] conn(0x5555a6aad270 0x5555a6aaf650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-05T19:28:37.842 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.838+0000 7f8807b28700 1 --2- 172.21.15.167:0/3649689687 >> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] conn(0x5555a6aad270 0x5555a6aaf650 crc :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-05T19:28:37.843 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.838+0000 7f8807b28700 1 -- 172.21.15.167:0/3649689687 <== osd.3 v2:172.21.15.167:6808/30036 1 ==== osd_op_reply(1 2023-10-05-19-9a9df965-6612-45d4-a45f-6e3c889bacc2.4223.2-myfoo2 [read 0~850 out=850b] v0'0 uv22 ondisk = 0) v8 ==== 208+0+850 (crc 0 0 0) 0x7f87f8014070 con 0x5555a6aad270 2023-10-05T19:28:37.843 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.838+0000 7f8809ff3e40 10 read 850 2023-10-05T19:28:37.843 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.838+0000 7f8809ff3e40 10 log_show_next pos 850 bl 850 off 419 eof 1 2023-10-05T19:28:37.843 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.838+0000 7f8809ff3e40 10 log_show_next pos 850 bl 850 off 850 eof 1 2023-10-05T19:28:37.843 INFO:teuthology.orchestra.run.smithi167.stdout:{ 2023-10-05T19:28:37.844 INFO:teuthology.orchestra.run.smithi167.stdout: "bucket_id": "9a9df965-6612-45d4-a45f-6e3c889bacc2.4223.2", 2023-10-05T19:28:37.844 INFO:teuthology.orchestra.run.smithi167.stdout: "bucket_owner": "foo", 2023-10-05T19:28:37.844 INFO:teuthology.orchestra.run.smithi167.stdout: "bucket": "myfoo2", 2023-10-05T19:28:37.844 INFO:teuthology.orchestra.run.smithi167.stdout: "log_entries": [ 2023-10-05T19:28:37.844 INFO:teuthology.orchestra.run.smithi167.stdout: { 2023-10-05T19:28:37.844 INFO:teuthology.orchestra.run.smithi167.stdout: "bucket": "myfoo2", 2023-10-05T19:28:37.844 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2023-10-05T19:28:30.786277Z", 2023-10-05T19:28:37.844 INFO:teuthology.orchestra.run.smithi167.stdout: "time_local": "2023-10-05T19:28:30.786277+0000", 2023-10-05T19:28:37.844 INFO:teuthology.orchestra.run.smithi167.stdout: "remote_addr": "172.21.0.51", 2023-10-05T19:28:37.844 INFO:teuthology.orchestra.run.smithi167.stdout: "user": "foo", 2023-10-05T19:28:37.845 INFO:teuthology.orchestra.run.smithi167.stdout: "operation": "create_bucket", 2023-10-05T19:28:37.845 INFO:teuthology.orchestra.run.smithi167.stdout: "uri": "PUT /myfoo2/ HTTP/1.1", 2023-10-05T19:28:37.845 INFO:teuthology.orchestra.run.smithi167.stdout: "http_status": "200", 2023-10-05T19:28:37.845 INFO:teuthology.orchestra.run.smithi167.stdout: "error_code": "", 2023-10-05T19:28:37.845 INFO:teuthology.orchestra.run.smithi167.stdout: "bytes_sent": 0, 2023-10-05T19:28:37.845 INFO:teuthology.orchestra.run.smithi167.stdout: "bytes_received": 0, 2023-10-05T19:28:37.845 INFO:teuthology.orchestra.run.smithi167.stdout: "object_size": 0, 2023-10-05T19:28:37.845 INFO:teuthology.orchestra.run.smithi167.stdout: "total_time": 19, 2023-10-05T19:28:37.845 INFO:teuthology.orchestra.run.smithi167.stdout: "user_agent": "Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic", 2023-10-05T19:28:37.845 INFO:teuthology.orchestra.run.smithi167.stdout: "referrer": "", 2023-10-05T19:28:37.845 INFO:teuthology.orchestra.run.smithi167.stdout: "trans_id": "tx00000e49afb9a63904523-00651f0e5e-107d-default", 2023-10-05T19:28:37.845 INFO:teuthology.orchestra.run.smithi167.stdout: "authentication_type": "Local", 2023-10-05T19:28:37.846 INFO:teuthology.orchestra.run.smithi167.stdout: "access_key_id": "9te6NH5mcdcq0Tc5i8i1", 2023-10-05T19:28:37.846 INFO:teuthology.orchestra.run.smithi167.stdout: "temp_url": false 2023-10-05T19:28:37.846 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2023-10-05T19:28:37.846 INFO:teuthology.orchestra.run.smithi167.stdout: { 2023-10-05T19:28:37.846 INFO:teuthology.orchestra.run.smithi167.stdout: "bucket": "myfoo2", 2023-10-05T19:28:37.846 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2023-10-05T19:28:30.890274Z", 2023-10-05T19:28:37.846 INFO:teuthology.orchestra.run.smithi167.stdout: "time_local": "2023-10-05T19:28:30.890274+0000", 2023-10-05T19:28:37.846 INFO:teuthology.orchestra.run.smithi167.stdout: "remote_addr": "172.21.0.51", 2023-10-05T19:28:37.846 INFO:teuthology.orchestra.run.smithi167.stdout: "user": "foo", 2023-10-05T19:28:37.846 INFO:teuthology.orchestra.run.smithi167.stdout: "operation": "delete_bucket", 2023-10-05T19:28:37.847 INFO:teuthology.orchestra.run.smithi167.stdout: "uri": "DELETE /myfoo2/ HTTP/1.1", 2023-10-05T19:28:37.847 INFO:teuthology.orchestra.run.smithi167.stdout: "http_status": "204", 2023-10-05T19:28:37.847 INFO:teuthology.orchestra.run.smithi167.stdout: "error_code": "NoContent", 2023-10-05T19:28:37.847 INFO:teuthology.orchestra.run.smithi167.stdout: "bytes_sent": 0, 2023-10-05T19:28:37.847 INFO:teuthology.orchestra.run.smithi167.stdout: "bytes_received": 0, 2023-10-05T19:28:37.847 INFO:teuthology.orchestra.run.smithi167.stdout: "object_size": 0, 2023-10-05T19:28:37.847 INFO:teuthology.orchestra.run.smithi167.stdout: "total_time": 75, 2023-10-05T19:28:37.847 INFO:teuthology.orchestra.run.smithi167.stdout: "user_agent": "Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic", 2023-10-05T19:28:37.847 INFO:teuthology.orchestra.run.smithi167.stdout: "referrer": "", 2023-10-05T19:28:37.847 INFO:teuthology.orchestra.run.smithi167.stdout: "trans_id": "tx00000e3035719548cb337-00651f0e5e-107d-default", 2023-10-05T19:28:37.847 INFO:teuthology.orchestra.run.smithi167.stdout: "authentication_type": "Local", 2023-10-05T19:28:37.848 INFO:teuthology.orchestra.run.smithi167.stdout: "access_key_id": "9te6NH5mcdcq0Tc5i8i1", 2023-10-05T19:28:37.848 INFO:teuthology.orchestra.run.smithi167.stdout: "temp_url": false 2023-10-05T19:28:37.848 INFO:teuthology.orchestra.run.smithi167.stdout: } 2023-10-05T19:28:37.848 INFO:teuthology.orchestra.run.smithi167.stdout: ], 2023-10-05T19:28:37.848 INFO:teuthology.orchestra.run.smithi167.stdout: "log_sum": { 2023-10-05T19:28:37.848 INFO:teuthology.orchestra.run.smithi167.stdout: "bytes_sent": 0, 2023-10-05T19:28:37.848 INFO:teuthology.orchestra.run.smithi167.stdout: "bytes_received": 0, 2023-10-05T19:28:37.848 INFO:teuthology.orchestra.run.smithi167.stdout: "total_time": 94, 2023-10-05T19:28:37.848 INFO:teuthology.orchestra.run.smithi167.stdout: "total_entries": 2 2023-10-05T19:28:37.848 INFO:teuthology.orchestra.run.smithi167.stdout: } 2023-10-05T19:28:37.848 INFO:teuthology.orchestra.run.smithi167.stdout:} 2023-10-05T19:28:37.849 INFO:teuthology.orchestra.run.smithi167.stdout: 2023-10-05T19:28:37.849 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.838+0000 7f8809ff3e40 1 -- 172.21.15.167:0/3210337137 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:102 7.1f 7:f95f44c2:::notify.0:head [watch unwatch cookie 93826356265568] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x5555a6a6c8a0 con 0x7f87f800d750 2023-10-05T19:28:37.849 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.842+0000 7f8807327700 1 -- 172.21.15.167:0/3210337137 <== osd.2 v2:172.21.15.167:6816/30037 17 ==== osd_op_reply(102 notify.0 [watch unwatch cookie 93826356265568] v30'173 uv171 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f87f0032040 con 0x7f87f800d750 2023-10-05T19:28:37.850 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.842+0000 7f8809ff3e40 20 remove_watcher() i=0 2023-10-05T19:28:37.850 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.842+0000 7f8809ff3e40 2 removed watcher, disabling cache 2023-10-05T19:28:37.850 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.842+0000 7f8809ff3e40 1 -- 172.21.15.167:0/3210337137 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:103 7.0 7:05bf5b68:::notify.1:head [watch unwatch cookie 93826356272272] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x5555a6a2c260 con 0x5555a69f72f0 2023-10-05T19:28:37.850 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.842+0000 7f8808329700 1 -- 172.21.15.167:0/3210337137 <== osd.1 v2:172.21.15.167:6800/30035 35 ==== osd_op_reply(103 notify.1 [watch unwatch cookie 93826356272272] v30'173 uv171 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f8800052040 con 0x5555a69f72f0 2023-10-05T19:28:37.850 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.842+0000 7f8809ff3e40 20 remove_watcher() i=1 2023-10-05T19:28:37.850 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.842+0000 7f8809ff3e40 1 -- 172.21.15.167:0/3210337137 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:104 7.15 7:a93a5511:::notify.2:head [watch unwatch cookie 93826356275888] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x5555a6a2c260 con 0x5555a6a28110 2023-10-05T19:28:37.851 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.846+0000 7f8807b28700 1 -- 172.21.15.167:0/3210337137 <== osd.3 v2:172.21.15.167:6808/30036 24 ==== osd_op_reply(104 notify.2 [watch unwatch cookie 93826356275888] v30'173 uv171 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f87f8014070 con 0x5555a6a28110 2023-10-05T19:28:37.851 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.846+0000 7f8809ff3e40 20 remove_watcher() i=2 2023-10-05T19:28:37.852 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.846+0000 7f8809ff3e40 1 -- 172.21.15.167:0/3210337137 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:105 7.e 7:7759931f:::notify.3:head [watch unwatch cookie 93826356274624] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x5555a6a2eab0 con 0x7f87f800d750 2023-10-05T19:28:37.854 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.846+0000 7f8807327700 1 -- 172.21.15.167:0/3210337137 <== osd.2 v2:172.21.15.167:6816/30037 18 ==== osd_op_reply(105 notify.3 [watch unwatch cookie 93826356274624] v30'173 uv171 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f87f0032040 con 0x7f87f800d750 2023-10-05T19:28:37.854 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.846+0000 7f8809ff3e40 20 remove_watcher() i=3 2023-10-05T19:28:37.854 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.846+0000 7f8809ff3e40 1 -- 172.21.15.167:0/3210337137 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:106 7.d 7:b4812045:::notify.4:head [watch unwatch cookie 93826356282880] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x5555a6a2e5c0 con 0x5555a69f72f0 2023-10-05T19:28:37.857 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.850+0000 7f8808329700 1 -- 172.21.15.167:0/3210337137 <== osd.1 v2:172.21.15.167:6800/30035 36 ==== osd_op_reply(106 notify.4 [watch unwatch cookie 93826356282880] v30'173 uv171 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f8800052040 con 0x5555a69f72f0 2023-10-05T19:28:37.857 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.850+0000 7f8809ff3e40 20 remove_watcher() i=4 2023-10-05T19:28:37.857 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.850+0000 7f8809ff3e40 1 -- 172.21.15.167:0/3210337137 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:107 7.3 7:c609908c:::notify.5:head [watch unwatch cookie 93826356286432] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x5555a6a2eab0 con 0x7f87f0007030 2023-10-05T19:28:37.859 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.854+0000 7f8808329700 1 -- 172.21.15.167:0/3210337137 <== osd.0 v2:172.21.15.118:6800/27690 29 ==== osd_op_reply(107 notify.5 [watch unwatch cookie 93826356286432] v30'173 uv171 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f8800052040 con 0x7f87f0007030 2023-10-05T19:28:37.860 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.854+0000 7f8809ff3e40 20 remove_watcher() i=5 2023-10-05T19:28:37.860 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.854+0000 7f8809ff3e40 1 -- 172.21.15.167:0/3210337137 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:108 7.14 7:2b04a3e9:::notify.6:head [watch unwatch cookie 93826356285168] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x5555a6a30600 con 0x7f87f0007030 2023-10-05T19:28:37.862 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.858+0000 7f8808329700 1 -- 172.21.15.167:0/3210337137 <== osd.0 v2:172.21.15.118:6800/27690 30 ==== osd_op_reply(108 notify.6 [watch unwatch cookie 93826356285168] v30'173 uv171 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f8800052040 con 0x7f87f0007030 2023-10-05T19:28:37.862 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.858+0000 7f8809ff3e40 20 remove_watcher() i=6 2023-10-05T19:28:37.863 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.858+0000 7f8809ff3e40 1 -- 172.21.15.167:0/3210337137 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:109 7.9 7:93e5b521:::notify.7:head [watch unwatch cookie 93826356290160] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x5555a6a2e5c0 con 0x7f87f0007030 2023-10-05T19:28:37.866 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.858+0000 7f8808329700 1 -- 172.21.15.167:0/3210337137 <== osd.0 v2:172.21.15.118:6800/27690 31 ==== osd_op_reply(109 notify.7 [watch unwatch cookie 93826356290160] v30'173 uv171 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f8800052040 con 0x7f87f0007030 2023-10-05T19:28:37.866 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.858+0000 7f8809ff3e40 20 remove_watcher() i=7 2023-10-05T19:28:37.868 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.862+0000 7f87d97e2700 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2023-10-05T19:28:37.868 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.862+0000 7f8809ff3e40 1 -- 172.21.15.167:0/3210337137 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:110 6.4 6:22d26bf9:::data_loggenerations_metadata:head [watch unwatch cookie 93826356292048] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x5555a6a6f020 con 0x5555a69f72f0 2023-10-05T19:28:37.871 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.866+0000 7f8808329700 1 -- 172.21.15.167:0/3210337137 <== osd.1 v2:172.21.15.167:6800/30035 37 ==== osd_op_reply(110 data_loggenerations_metadata [watch unwatch cookie 93826356292048] v30'162 uv1 ondisk = 0) v8 ==== 172+0+0 (crc 0 0 0) 0x7f8800052040 con 0x5555a69f72f0 2023-10-05T19:28:37.871 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.866+0000 7f8809ff3e40 1 -- 172.21.15.167:0/3210337137 >> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] conn(0x7f87f0007030 msgr2=0x7f87f001c920 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-05T19:28:37.871 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.866+0000 7f8809ff3e40 1 --2- 172.21.15.167:0/3210337137 >> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] conn(0x7f87f0007030 0x7f87f001c920 crc :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-05T19:28:37.872 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.866+0000 7f8809ff3e40 1 -- 172.21.15.167:0/3210337137 >> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] conn(0x5555a69f72f0 msgr2=0x5555a6a245f0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-05T19:28:37.872 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.866+0000 7f8809ff3e40 1 --2- 172.21.15.167:0/3210337137 >> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] conn(0x5555a69f72f0 0x5555a6a245f0 crc :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-05T19:28:37.872 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.866+0000 7f8809ff3e40 1 -- 172.21.15.167:0/3210337137 >> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] conn(0x7f87f800d750 msgr2=0x7f87f8042230 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-05T19:28:37.872 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.866+0000 7f8809ff3e40 1 --2- 172.21.15.167:0/3210337137 >> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] conn(0x7f87f800d750 0x7f87f8042230 crc :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-05T19:28:37.872 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.866+0000 7f8809ff3e40 1 -- 172.21.15.167:0/3210337137 >> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] conn(0x5555a6a28110 msgr2=0x5555a6a2a4f0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-05T19:28:37.872 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.866+0000 7f8809ff3e40 1 --2- 172.21.15.167:0/3210337137 >> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] conn(0x5555a6a28110 0x5555a6a2a4f0 crc :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-05T19:28:37.872 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.866+0000 7f8809ff3e40 1 -- 172.21.15.167:0/3210337137 >> [v2:172.21.15.167:6824/29889,v1:172.21.15.167:6825/29889] conn(0x7f87e8030c00 msgr2=0x7f87e80330b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-05T19:28:37.872 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.866+0000 7f8809ff3e40 1 --2- 172.21.15.167:0/3210337137 >> [v2:172.21.15.167:6824/29889,v1:172.21.15.167:6825/29889] conn(0x7f87e8030c00 0x7f87e80330b0 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7f87f0005a90 tx=0x7f87f0015000 comp rx=0 tx=0).stop 2023-10-05T19:28:37.872 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.866+0000 7f8808329700 1 -- 172.21.15.167:0/3210337137 reap_dead start 2023-10-05T19:28:37.873 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.866+0000 7f8809ff3e40 1 -- 172.21.15.167:0/3210337137 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5555a69673d0 msgr2=0x5555a6a18bd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-05T19:28:37.873 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.866+0000 7f8809ff3e40 1 --2- 172.21.15.167:0/3210337137 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5555a69673d0 0x5555a6a18bd0 secure :-1 s=READY pgs=479 cs=0 l=1 rev1=1 crypto rx=0x7f87f80017d0 tx=0x7f87f8016000 comp rx=0 tx=0).stop 2023-10-05T19:28:37.873 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.866+0000 7f8809ff3e40 1 -- 172.21.15.167:0/3210337137 shutdown_connections 2023-10-05T19:28:37.873 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.866+0000 7f8809ff3e40 1 --2- 172.21.15.167:0/3210337137 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5555a69673d0 0x5555a6a18bd0 secure :-1 s=CLOSED pgs=479 cs=0 l=1 rev1=1 crypto rx=0x7f87f80017d0 tx=0x7f87f8016000 comp rx=0 tx=0).stop 2023-10-05T19:28:37.873 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.866+0000 7f8809ff3e40 1 -- 172.21.15.167:0/3210337137 >> 172.21.15.167:0/3210337137 conn(0x5555a69ed720 msgr2=0x5555a6a0c430 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-05T19:28:37.873 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.866+0000 7f8809ff3e40 1 -- 172.21.15.167:0/3210337137 shutdown_connections 2023-10-05T19:28:37.873 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.866+0000 7f8809ff3e40 1 -- 172.21.15.167:0/3210337137 wait complete. 2023-10-05T19:28:37.874 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.866+0000 7f8809ff3e40 1 -- 172.21.15.167:0/3649689687 >> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] conn(0x5555a6aad270 msgr2=0x5555a6aaf650 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-05T19:28:37.874 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.866+0000 7f8809ff3e40 1 --2- 172.21.15.167:0/3649689687 >> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] conn(0x5555a6aad270 0x5555a6aaf650 crc :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-05T19:28:37.874 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.866+0000 7f8809ff3e40 1 -- 172.21.15.167:0/3649689687 >> [v2:172.21.15.167:6824/29889,v1:172.21.15.167:6825/29889] conn(0x7f8748030bb0 msgr2=0x7f8748033060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-05T19:28:37.875 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.866+0000 7f8809ff3e40 1 --2- 172.21.15.167:0/3649689687 >> [v2:172.21.15.167:6824/29889,v1:172.21.15.167:6825/29889] conn(0x7f8748030bb0 0x7f8748033060 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x5555a6a33310 tx=0x7f8800012e70 comp rx=0 tx=0).stop 2023-10-05T19:28:37.875 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.866+0000 7f8809ff3e40 1 -- 172.21.15.167:0/3649689687 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5555a6a40300 msgr2=0x5555a6a45dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-05T19:28:37.875 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.866+0000 7f8809ff3e40 1 --2- 172.21.15.167:0/3649689687 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5555a6a40300 0x5555a6a45dc0 secure :-1 s=READY pgs=481 cs=0 l=1 rev1=1 crypto rx=0x5555a6a3a3e0 tx=0x7f87f002f000 comp rx=0 tx=0).stop 2023-10-05T19:28:37.875 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.866+0000 7f8809ff3e40 1 -- 172.21.15.167:0/3649689687 shutdown_connections 2023-10-05T19:28:37.875 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.866+0000 7f8809ff3e40 1 --2- 172.21.15.167:0/3649689687 >> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] conn(0x5555a6aad270 0x5555a6aaf650 unknown :-1 s=CLOSED pgs=81 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-05T19:28:37.875 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.866+0000 7f8809ff3e40 1 --2- 172.21.15.167:0/3649689687 >> [v2:172.21.15.167:6824/29889,v1:172.21.15.167:6825/29889] conn(0x7f8748030bb0 0x7f8748033060 unknown :-1 s=CLOSED pgs=167 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-05T19:28:37.875 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.866+0000 7f8809ff3e40 1 --2- 172.21.15.167:0/3649689687 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5555a6a40300 0x5555a6a45dc0 unknown :-1 s=CLOSED pgs=481 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-05T19:28:37.875 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.866+0000 7f8809ff3e40 1 -- 172.21.15.167:0/3649689687 >> 172.21.15.167:0/3649689687 conn(0x5555a6a3b6c0 msgr2=0x5555a6a397c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-05T19:28:37.875 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.866+0000 7f8809ff3e40 1 -- 172.21.15.167:0/3649689687 shutdown_connections 2023-10-05T19:28:37.875 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.866+0000 7f8809ff3e40 1 -- 172.21.15.167:0/3649689687 wait complete. 2023-10-05T19:28:37.878 DEBUG:tasks.util.rgw: json result: {'bucket_id': '9a9df965-6612-45d4-a45f-6e3c889bacc2.4223.2', 'bucket_owner': 'foo', 'bucket': 'myfoo2', 'log_entries': [{'bucket': 'myfoo2', 'time': '2023-10-05T19:28:30.786277Z', 'time_local': '2023-10-05T19:28:30.786277+0000', 'remote_addr': '172.21.0.51', 'user': 'foo', 'operation': 'create_bucket', 'uri': 'PUT /myfoo2/ HTTP/1.1', 'http_status': '200', 'error_code': '', 'bytes_sent': 0, 'bytes_received': 0, 'object_size': 0, 'total_time': 19, 'user_agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'referrer': '', 'trans_id': 'tx00000e49afb9a63904523-00651f0e5e-107d-default', 'authentication_type': 'Local', 'access_key_id': '9te6NH5mcdcq0Tc5i8i1', 'temp_url': False}, {'bucket': 'myfoo2', 'time': '2023-10-05T19:28:30.890274Z', 'time_local': '2023-10-05T19:28:30.890274+0000', 'remote_addr': '172.21.0.51', 'user': 'foo', 'operation': 'delete_bucket', 'uri': 'DELETE /myfoo2/ HTTP/1.1', 'http_status': '204', 'error_code': 'NoContent', 'bytes_sent': 0, 'bytes_received': 0, 'object_size': 0, 'total_time': 75, 'user_agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'referrer': '', 'trans_id': 'tx00000e3035719548cb337-00651f0e5e-107d-default', 'authentication_type': 'Local', 'access_key_id': '9te6NH5mcdcq0Tc5i8i1', 'temp_url': False}], 'log_sum': {'bytes_sent': 0, 'bytes_received': 0, 'total_time': 94, 'total_entries': 2}} 2023-10-05T19:28:37.878 DEBUG:tasks.radosgw_admin:checking log entry: 2023-10-05T19:28:37.878 DEBUG:tasks.radosgw_admin:checking log entry: 2023-10-05T19:28:37.878 INFO:tasks.util.rgw:rgwadmin: client.0 : ['log', 'rm', '--object', '2023-10-05-19-9a9df965-6612-45d4-a45f-6e3c889bacc2.4223.2-myfoo2'] 2023-10-05T19:28:37.878 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'ceph', 'log', 'rm', '--object', '2023-10-05-19-9a9df965-6612-45d4-a45f-6e3c889bacc2.4223.2-myfoo2'] 2023-10-05T19:28:37.879 DEBUG:teuthology.orchestra.run.smithi167:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster ceph log rm --object 2023-10-05-19-9a9df965-6612-45d4-a45f-6e3c889bacc2.4223.2-myfoo2 2023-10-05T19:28:37.919 INFO:teuthology.orchestra.run.smithi167.stderr:ignoring --setuser ceph since I am not root 2023-10-05T19:28:37.919 INFO:teuthology.orchestra.run.smithi167.stderr:ignoring --setgroup ceph since I am not root 2023-10-05T19:28:37.920 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.914+0000 7fe96d6f4e40 1 Processor -- start 2023-10-05T19:28:37.920 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.914+0000 7fe96d6f4e40 1 -- start start 2023-10-05T19:28:37.920 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.914+0000 7fe96d6f4e40 1 --2- >> v2:172.21.15.118:3300/0 conn(0x55ddca9133d0 0x55ddca9a3fd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-05T19:28:37.920 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.914+0000 7fe96d6f4e40 1 -- --> v1:172.21.15.118:6789/0 -- auth(proto 0 26 bytes epoch 0) v1 -- 0x55ddca8ebbd0 con 0x55ddca9a4510 2023-10-05T19:28:37.920 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.914+0000 7fe96d6f4e40 1 -- --> v2:172.21.15.118:3300/0 -- mon_getmap magic: 0 v1 -- 0x55ddca8a8ee0 con 0x55ddca9133d0 2023-10-05T19:28:37.921 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.914+0000 7fe96b229700 1 --2- >> v2:172.21.15.118:3300/0 conn(0x55ddca9133d0 0x55ddca9a3fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-05T19:28:37.921 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.914+0000 7fe96aa28700 1 --1- >> v1:172.21.15.118:6789/0 conn(0x55ddca9a4510 0x55ddca9a68f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.118:6789/0 says I am v1:172.21.15.167:59378/0 (socket says 172.21.15.167:59378) 2023-10-05T19:28:37.921 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.914+0000 7fe96aa28700 1 -- 172.21.15.167:0/678739325 learned_addr learned my addr 172.21.15.167:0/678739325 (peer_addr_for_me v1:172.21.15.167:0/0) 2023-10-05T19:28:37.921 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.914+0000 7fe96b229700 1 -- 172.21.15.167:0/678739325 >> v1:172.21.15.118:6789/0 conn(0x55ddca9a4510 legacy=0x55ddca9a68f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-05T19:28:37.921 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.914+0000 7fe96b229700 1 -- 172.21.15.167:0/678739325 --> v2:172.21.15.118:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55ddca9137a0 con 0x55ddca9133d0 2023-10-05T19:28:37.922 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.914+0000 7fe96b229700 1 --2- 172.21.15.167:0/678739325 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55ddca9133d0 0x55ddca9a3fd0 secure :-1 s=READY pgs=483 cs=0 l=1 rev1=1 crypto rx=0x7fe95c00a5d0 tx=0x7fe95c005a50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bface3b360817e30 server_cookie=0 in_seq=0 out_seq=0 2023-10-05T19:28:37.923 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.918+0000 7fe96a227700 1 -- 172.21.15.167:0/678739325 <== mon.0 v2:172.21.15.118:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fe95c0086a0 con 0x55ddca9133d0 2023-10-05T19:28:37.923 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.918+0000 7fe96a227700 1 -- 172.21.15.167:0/678739325 <== mon.0 v2:172.21.15.118:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe95c008800 con 0x55ddca9133d0 2023-10-05T19:28:37.923 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.918+0000 7fe96a227700 1 -- 172.21.15.167:0/678739325 <== mon.0 v2:172.21.15.118:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fe95c008ad0 con 0x55ddca9133d0 2023-10-05T19:28:37.923 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.918+0000 7fe96d6f4e40 1 -- 172.21.15.167:0/678739325 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55ddca9133d0 msgr2=0x55ddca9a3fd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-05T19:28:37.923 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.918+0000 7fe96d6f4e40 1 --2- 172.21.15.167:0/678739325 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55ddca9133d0 0x55ddca9a3fd0 secure :-1 s=READY pgs=483 cs=0 l=1 rev1=1 crypto rx=0x7fe95c00a5d0 tx=0x7fe95c005a50 comp rx=0 tx=0).stop 2023-10-05T19:28:37.923 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.918+0000 7fe96d6f4e40 1 -- 172.21.15.167:0/678739325 shutdown_connections 2023-10-05T19:28:37.923 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.918+0000 7fe96d6f4e40 1 --2- 172.21.15.167:0/678739325 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55ddca9133d0 0x55ddca9a3fd0 unknown :-1 s=CLOSED pgs=483 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-05T19:28:37.923 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.918+0000 7fe96d6f4e40 1 -- 172.21.15.167:0/678739325 >> 172.21.15.167:0/678739325 conn(0x55ddca999720 msgr2=0x55ddca99bb30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-05T19:28:37.923 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.918+0000 7fe96d6f4e40 1 -- 172.21.15.167:0/678739325 shutdown_connections 2023-10-05T19:28:37.924 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.918+0000 7fe96d6f4e40 1 -- 172.21.15.167:0/678739325 wait complete. 2023-10-05T19:28:37.924 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.918+0000 7fe96d6f4e40 1 Processor -- start 2023-10-05T19:28:37.925 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.918+0000 7fe96d6f4e40 1 -- start start 2023-10-05T19:28:37.925 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.918+0000 7fe96d6f4e40 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55ddca9133d0 0x55ddca9c0150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-05T19:28:37.925 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.918+0000 7fe96d6f4e40 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ddca86ce30 con 0x55ddca9133d0 2023-10-05T19:28:37.925 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.918+0000 7fe96b229700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55ddca9133d0 0x55ddca9c0150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-05T19:28:37.925 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.918+0000 7fe96b229700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55ddca9133d0 0x55ddca9c0150 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.118:3300/0 says I am v2:172.21.15.167:42112/0 (socket says 172.21.15.167:42112) 2023-10-05T19:28:37.925 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.918+0000 7fe96b229700 1 -- 172.21.15.167:0/1234884187 learned_addr learned my addr 172.21.15.167:0/1234884187 (peer_addr_for_me v2:172.21.15.167:0/0) 2023-10-05T19:28:37.925 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.918+0000 7fe96b229700 1 -- 172.21.15.167:0/1234884187 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe95c005cf0 con 0x55ddca9133d0 2023-10-05T19:28:37.926 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.918+0000 7fe96b229700 1 --2- 172.21.15.167:0/1234884187 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55ddca9133d0 0x55ddca9c0150 secure :-1 s=READY pgs=484 cs=0 l=1 rev1=1 crypto rx=0x7fe95c005a90 tx=0x7fe95c015000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cb413fdea48eb2cb server_cookie=0 in_seq=0 out_seq=0 2023-10-05T19:28:37.926 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.918+0000 7fe969225700 1 -- 172.21.15.167:0/1234884187 <== mon.0 v2:172.21.15.118:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fe95c008a40 con 0x55ddca9133d0 2023-10-05T19:28:37.926 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.918+0000 7fe969225700 1 -- 172.21.15.167:0/1234884187 <== mon.0 v2:172.21.15.118:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe95c008bc0 con 0x55ddca9133d0 2023-10-05T19:28:37.926 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.918+0000 7fe969225700 1 -- 172.21.15.167:0/1234884187 <== mon.0 v2:172.21.15.118:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fe95c012450 con 0x55ddca9133d0 2023-10-05T19:28:37.927 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.922+0000 7fe96d6f4e40 1 -- 172.21.15.167:0/1234884187 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55ddca9133d0 msgr2=0x55ddca9c0150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-05T19:28:37.927 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.922+0000 7fe96d6f4e40 1 --2- 172.21.15.167:0/1234884187 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55ddca9133d0 0x55ddca9c0150 secure :-1 s=READY pgs=484 cs=0 l=1 rev1=1 crypto rx=0x7fe95c005a90 tx=0x7fe95c015000 comp rx=0 tx=0).stop 2023-10-05T19:28:37.927 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.922+0000 7fe96d6f4e40 1 -- 172.21.15.167:0/1234884187 shutdown_connections 2023-10-05T19:28:37.927 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.922+0000 7fe96d6f4e40 1 --2- 172.21.15.167:0/1234884187 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55ddca9133d0 0x55ddca9c0150 unknown :-1 s=CLOSED pgs=484 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-05T19:28:37.927 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.922+0000 7fe96d6f4e40 1 -- 172.21.15.167:0/1234884187 >> 172.21.15.167:0/1234884187 conn(0x55ddca999720 msgr2=0x55ddca9bbd10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-05T19:28:37.927 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.922+0000 7fe96d6f4e40 1 -- 172.21.15.167:0/1234884187 shutdown_connections 2023-10-05T19:28:37.928 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.922+0000 7fe96d6f4e40 1 -- 172.21.15.167:0/1234884187 wait complete. 2023-10-05T19:28:37.928 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.922+0000 7fe96d6f4e40 1 Processor -- start 2023-10-05T19:28:37.928 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.922+0000 7fe96d6f4e40 1 -- start start 2023-10-05T19:28:37.928 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.922+0000 7fe96d6f4e40 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55ddca9133d0 0x55ddca9c6d60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-05T19:28:37.928 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.922+0000 7fe96d6f4e40 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ddca83fed0 con 0x55ddca9133d0 2023-10-05T19:28:37.929 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.922+0000 7fe96b229700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55ddca9133d0 0x55ddca9c6d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-05T19:28:37.929 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.922+0000 7fe96b229700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55ddca9133d0 0x55ddca9c6d60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.118:3300/0 says I am v2:172.21.15.167:42120/0 (socket says 172.21.15.167:42120) 2023-10-05T19:28:37.929 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.922+0000 7fe96b229700 1 -- 172.21.15.167:0/2790000739 learned_addr learned my addr 172.21.15.167:0/2790000739 (peer_addr_for_me v2:172.21.15.167:0/0) 2023-10-05T19:28:37.930 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.922+0000 7fe96b229700 1 -- 172.21.15.167:0/2790000739 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55ddca9137a0 con 0x55ddca9133d0 2023-10-05T19:28:37.930 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.926+0000 7fe96b229700 1 --2- 172.21.15.167:0/2790000739 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55ddca9133d0 0x55ddca9c6d60 secure :-1 s=READY pgs=485 cs=0 l=1 rev1=1 crypto rx=0x7fe95c00c010 tx=0x7fe95c01c000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-05T19:28:37.930 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.926+0000 7fe9637fe700 1 -- 172.21.15.167:0/2790000739 <== mon.0 v2:172.21.15.118:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fe95c008e50 con 0x55ddca9133d0 2023-10-05T19:28:37.931 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.926+0000 7fe9637fe700 1 -- 172.21.15.167:0/2790000739 <== mon.0 v2:172.21.15.118:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe95c025660 con 0x55ddca9133d0 2023-10-05T19:28:37.931 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.926+0000 7fe9637fe700 1 -- 172.21.15.167:0/2790000739 <== mon.0 v2:172.21.15.118:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fe95c025950 con 0x55ddca9133d0 2023-10-05T19:28:37.931 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.926+0000 7fe96d6f4e40 1 -- 172.21.15.167:0/2790000739 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x55ddca9c72a0 con 0x55ddca9133d0 2023-10-05T19:28:37.931 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.926+0000 7fe96d6f4e40 1 -- 172.21.15.167:0/2790000739 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55ddca9c7740 con 0x55ddca9133d0 2023-10-05T19:28:37.931 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.926+0000 7fe96d6f4e40 20 rados->read ofs=0 len=0 2023-10-05T19:28:37.932 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.926+0000 7fe9637fe700 1 -- 172.21.15.167:0/2790000739 <== mon.0 v2:172.21.15.118:3300/0 4 ==== mgrmap(e 5) v1 ==== 40329+0+0 (secure 0 0 0) 0x7fe95c019070 con 0x55ddca9133d0 2023-10-05T19:28:37.933 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.926+0000 7fe9637fe700 1 --2- 172.21.15.167:0/2790000739 >> [v2:172.21.15.167:6824/29889,v1:172.21.15.167:6825/29889] conn(0x7fe94c030c00 0x7fe94c0330b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-05T19:28:37.933 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.926+0000 7fe9637fe700 1 -- 172.21.15.167:0/2790000739 <== mon.0 v2:172.21.15.118:3300/0 5 ==== osd_map(30..30 src has 1..30) v4 ==== 6057+0+0 (secure 0 0 0) 0x7fe95c046770 con 0x55ddca9133d0 2023-10-05T19:28:37.933 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.926+0000 7fe96aa28700 1 --2- 172.21.15.167:0/2790000739 >> [v2:172.21.15.167:6824/29889,v1:172.21.15.167:6825/29889] conn(0x7fe94c030c00 0x7fe94c0330b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-05T19:28:37.933 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.926+0000 7fe96d6f4e40 1 --2- 172.21.15.167:0/2790000739 >> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] conn(0x55ddca9a4510 0x55ddca9d2880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-05T19:28:37.934 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.926+0000 7fe96d6f4e40 1 -- 172.21.15.167:0/2790000739 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:1 5.12 5:49953fa1:::default.realm:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x55ddca9d2f30 con 0x55ddca9a4510 2023-10-05T19:28:37.934 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.926+0000 7fe96ba2a700 1 --2- 172.21.15.167:0/2790000739 >> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] conn(0x55ddca9a4510 0x55ddca9d2880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-05T19:28:37.934 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.930+0000 7fe96aa28700 1 --2- 172.21.15.167:0/2790000739 >> [v2:172.21.15.167:6824/29889,v1:172.21.15.167:6825/29889] conn(0x7fe94c030c00 0x7fe94c0330b0 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7fe954001df0 tx=0x7fe954001310 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-05T19:28:37.934 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.930+0000 7fe96ba2a700 1 --2- 172.21.15.167:0/2790000739 >> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] conn(0x55ddca9a4510 0x55ddca9d2880 crc :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-05T19:28:37.935 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.930+0000 7fe96ba2a700 1 -- 172.21.15.167:0/2790000739 <== osd.1 v2:172.21.15.167:6800/30035 1 ==== osd_op_reply(1 default.realm [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 ==== 157+0+0 (crc 0 0 0) 0x7fe96400a5a0 con 0x55ddca9a4510 2023-10-05T19:28:37.935 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.930+0000 7fe96d6f4e40 20 rados_obj.operate() r=-2 bl.length=0 2023-10-05T19:28:37.935 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.930+0000 7fe96d6f4e40 20 realm 2023-10-05T19:28:37.935 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.930+0000 7fe96d6f4e40 20 rados->read ofs=0 len=0 2023-10-05T19:28:37.935 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.930+0000 7fe96d6f4e40 1 -- 172.21.15.167:0/2790000739 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:2 5.12 5:49953fa1:::default.realm:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x55ddca9cfee0 con 0x55ddca9a4510 2023-10-05T19:28:37.937 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.930+0000 7fe96ba2a700 1 -- 172.21.15.167:0/2790000739 <== osd.1 v2:172.21.15.167:6800/30035 2 ==== osd_op_reply(2 default.realm [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 ==== 157+0+0 (crc 0 0 0) 0x7fe96400a5a0 con 0x55ddca9a4510 2023-10-05T19:28:37.937 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.930+0000 7fe96d6f4e40 20 rados_obj.operate() r=-2 bl.length=0 2023-10-05T19:28:37.937 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.930+0000 7fe96d6f4e40 4 RGWPeriod::init failed to init realm id : (2) No such file or directory 2023-10-05T19:28:37.937 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.930+0000 7fe96d6f4e40 20 rados->read ofs=0 len=0 2023-10-05T19:28:37.937 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.930+0000 7fe96d6f4e40 1 -- 172.21.15.167:0/2790000739 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:3 5.12 5:49953fa1:::default.realm:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x55ddca9d3a10 con 0x55ddca9a4510 2023-10-05T19:28:37.937 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.930+0000 7fe96ba2a700 1 -- 172.21.15.167:0/2790000739 <== osd.1 v2:172.21.15.167:6800/30035 3 ==== osd_op_reply(3 default.realm [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 ==== 157+0+0 (crc 0 0 0) 0x7fe96400a5a0 con 0x55ddca9a4510 2023-10-05T19:28:37.937 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.930+0000 7fe96d6f4e40 20 rados_obj.operate() r=-2 bl.length=0 2023-10-05T19:28:37.937 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.930+0000 7fe96d6f4e40 20 rados->read ofs=0 len=0 2023-10-05T19:28:37.937 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.930+0000 7fe96d6f4e40 1 -- 172.21.15.167:0/2790000739 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:4 5.f 5:f43b6ece:::zone_names.default:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x55ddca9d3620 con 0x55ddca9a4510 2023-10-05T19:28:37.937 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.930+0000 7fe96ba2a700 1 -- 172.21.15.167:0/2790000739 <== osd.1 v2:172.21.15.167:6800/30035 4 ==== osd_op_reply(4 zone_names.default [read 0~46 out=46b] v0'0 uv1 ondisk = 0) v8 ==== 162+0+46 (crc 0 0 0) 0x7fe96400a5a0 con 0x55ddca9a4510 2023-10-05T19:28:37.938 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.930+0000 7fe96d6f4e40 20 rados_obj.operate() r=0 bl.length=46 2023-10-05T19:28:37.938 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.930+0000 7fe96d6f4e40 20 rados->read ofs=0 len=0 2023-10-05T19:28:37.938 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.930+0000 7fe96d6f4e40 1 -- 172.21.15.167:0/2790000739 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:5 5.1d 5:b8374eb4:::zone_info.9a9df965-6612-45d4-a45f-6e3c889bacc2:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x55ddca9d48a0 con 0x55ddca9a4510 2023-10-05T19:28:37.939 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.930+0000 7fe96ba2a700 1 -- 172.21.15.167:0/2790000739 <== osd.1 v2:172.21.15.167:6800/30035 5 ==== osd_op_reply(5 zone_info.9a9df965-6612-45d4-a45f-6e3c889bacc2 [read 0~874 out=874b] v0'0 uv2 ondisk = 0) v8 ==== 190+0+874 (crc 0 0 0) 0x7fe96400a5a0 con 0x55ddca9a4510 2023-10-05T19:28:37.939 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.930+0000 7fe96d6f4e40 20 rados_obj.operate() r=0 bl.length=874 2023-10-05T19:28:37.939 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.930+0000 7fe96d6f4e40 20 searching for the correct realm 2023-10-05T19:28:37.939 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.930+0000 7fe96d6f4e40 1 --2- 172.21.15.167:0/2790000739 >> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] conn(0x55ddca9d6660 0x55ddca9d8a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-05T19:28:37.939 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.930+0000 7fe96d6f4e40 1 -- 172.21.15.167:0/2790000739 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:6 5.0 5:00000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x55ddca9d8f80 con 0x55ddca9d6660 2023-10-05T19:28:37.939 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.930+0000 7fe96b229700 1 --2- 172.21.15.167:0/2790000739 >> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] conn(0x55ddca9d6660 0x55ddca9d8a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-05T19:28:37.939 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.934+0000 7fe96b229700 1 --2- 172.21.15.167:0/2790000739 >> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] conn(0x55ddca9d6660 0x55ddca9d8a40 crc :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-05T19:28:37.941 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.934+0000 7fe96b229700 1 -- 172.21.15.167:0/2790000739 <== osd.3 v2:172.21.15.167:6808/30036 1 ==== osd_op_reply(6 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fe95c011570 con 0x55ddca9d6660 2023-10-05T19:28:37.941 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.934+0000 7fe96b229700 1 --2- 172.21.15.167:0/2790000739 >> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] conn(0x7fe95c007030 0x7fe95c011830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-05T19:28:37.941 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.934+0000 7fe96b229700 1 -- 172.21.15.167:0/2790000739 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:7 5.10 5:08000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x55ddca9d8f80 con 0x7fe95c007030 2023-10-05T19:28:37.941 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.934+0000 7fe96aa28700 1 --2- 172.21.15.167:0/2790000739 >> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] conn(0x7fe95c007030 0x7fe95c011830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-05T19:28:37.942 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.934+0000 7fe96aa28700 1 --2- 172.21.15.167:0/2790000739 >> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] conn(0x7fe95c007030 0x7fe95c011830 crc :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-05T19:28:37.942 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.938+0000 7fe96aa28700 1 -- 172.21.15.167:0/2790000739 <== osd.2 v2:172.21.15.167:6816/30037 1 ==== osd_op_reply(7 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fe954006070 con 0x7fe95c007030 2023-10-05T19:28:37.942 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.938+0000 7fe96aa28700 1 --2- 172.21.15.167:0/2790000739 >> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] conn(0x7fe95400eb60 0x7fe95400d8e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-05T19:28:37.942 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.938+0000 7fe96aa28700 1 -- 172.21.15.167:0/2790000739 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:8 5.8 5:10000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x55ddca9d8f80 con 0x7fe95400eb60 2023-10-05T19:28:37.943 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.938+0000 7fe96ba2a700 1 --2- 172.21.15.167:0/2790000739 >> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] conn(0x7fe95400eb60 0x7fe95400d8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-05T19:28:37.943 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.938+0000 7fe96ba2a700 1 --2- 172.21.15.167:0/2790000739 >> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] conn(0x7fe95400eb60 0x7fe95400d8e0 crc :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-05T19:28:37.944 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.938+0000 7fe96ba2a700 1 -- 172.21.15.167:0/2790000739 <== osd.0 v2:172.21.15.118:6800/27690 1 ==== osd_op_reply(8 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fe96400a5a0 con 0x7fe95400eb60 2023-10-05T19:28:37.944 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.938+0000 7fe96ba2a700 1 -- 172.21.15.167:0/2790000739 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:9 5.18 5:18000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x55ddca9d8f80 con 0x55ddca9a4510 2023-10-05T19:28:37.944 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.938+0000 7fe96ba2a700 1 -- 172.21.15.167:0/2790000739 <== osd.1 v2:172.21.15.167:6800/30035 6 ==== osd_op_reply(9 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fe96400a5a0 con 0x55ddca9a4510 2023-10-05T19:28:37.944 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.938+0000 7fe96ba2a700 1 -- 172.21.15.167:0/2790000739 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:10 5.4 5:20000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x55ddca9d8f80 con 0x55ddca9a4510 2023-10-05T19:28:37.945 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.938+0000 7fe96ba2a700 1 -- 172.21.15.167:0/2790000739 <== osd.1 v2:172.21.15.167:6800/30035 7 ==== osd_op_reply(10 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fe96400a5a0 con 0x55ddca9a4510 2023-10-05T19:28:37.945 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.938+0000 7fe96ba2a700 1 -- 172.21.15.167:0/2790000739 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:11 5.14 5:28000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x55ddca9d8f80 con 0x7fe95400eb60 2023-10-05T19:28:37.946 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.942+0000 7fe96ba2a700 1 -- 172.21.15.167:0/2790000739 <== osd.0 v2:172.21.15.118:6800/27690 2 ==== osd_op_reply(11 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fe96400a5a0 con 0x7fe95400eb60 2023-10-05T19:28:37.946 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.942+0000 7fe96ba2a700 1 -- 172.21.15.167:0/2790000739 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:12 5.c 5:30000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x55ddca9d8f80 con 0x7fe95400eb60 2023-10-05T19:28:37.947 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.942+0000 7fe96ba2a700 1 -- 172.21.15.167:0/2790000739 <== osd.0 v2:172.21.15.118:6800/27690 3 ==== osd_op_reply(12 [pgnls start_epoch 30 out=112b] v23'1 uv1 ondisk = 1) v8 ==== 144+0+112 (crc 0 0 0) 0x7fe96400a5a0 con 0x7fe95400eb60 2023-10-05T19:28:37.947 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.942+0000 7fe96ba2a700 1 -- 172.21.15.167:0/2790000739 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:13 5.1c 5:38000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x55ddca9d8f80 con 0x7fe95c007030 2023-10-05T19:28:37.947 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.942+0000 7fe96aa28700 1 -- 172.21.15.167:0/2790000739 <== osd.2 v2:172.21.15.167:6816/30037 2 ==== osd_op_reply(13 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fe954006070 con 0x7fe95c007030 2023-10-05T19:28:37.947 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.942+0000 7fe96aa28700 1 -- 172.21.15.167:0/2790000739 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:14 5.2 5:40000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x55ddca9d8f80 con 0x55ddca9d6660 2023-10-05T19:28:37.948 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.942+0000 7fe96b229700 1 -- 172.21.15.167:0/2790000739 <== osd.3 v2:172.21.15.167:6808/30036 2 ==== osd_op_reply(14 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fe95c011570 con 0x55ddca9d6660 2023-10-05T19:28:37.948 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.942+0000 7fe96b229700 1 -- 172.21.15.167:0/2790000739 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:15 5.12 5:48000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x55ddca9d8f80 con 0x55ddca9a4510 2023-10-05T19:28:37.948 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.942+0000 7fe96ba2a700 1 -- 172.21.15.167:0/2790000739 <== osd.1 v2:172.21.15.167:6800/30035 8 ==== osd_op_reply(15 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fe96400a5a0 con 0x55ddca9a4510 2023-10-05T19:28:37.949 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.942+0000 7fe96ba2a700 1 -- 172.21.15.167:0/2790000739 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:16 5.a 5:50000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x55ddca9d8f80 con 0x7fe95400eb60 2023-10-05T19:28:37.949 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.942+0000 7fe96ba2a700 1 -- 172.21.15.167:0/2790000739 <== osd.0 v2:172.21.15.118:6800/27690 4 ==== osd_op_reply(16 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fe96400a5a0 con 0x7fe95400eb60 2023-10-05T19:28:37.949 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.942+0000 7fe96ba2a700 1 -- 172.21.15.167:0/2790000739 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:17 5.1a 5:58000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x55ddca9d8f80 con 0x55ddca9a4510 2023-10-05T19:28:37.950 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.946+0000 7fe96ba2a700 1 -- 172.21.15.167:0/2790000739 <== osd.1 v2:172.21.15.167:6800/30035 9 ==== osd_op_reply(17 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fe96400a5a0 con 0x55ddca9a4510 2023-10-05T19:28:37.950 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.946+0000 7fe96ba2a700 1 -- 172.21.15.167:0/2790000739 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:18 5.6 5:60000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x55ddca9d8f80 con 0x7fe95c007030 2023-10-05T19:28:37.951 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.946+0000 7fe96aa28700 1 -- 172.21.15.167:0/2790000739 <== osd.2 v2:172.21.15.167:6816/30037 3 ==== osd_op_reply(18 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fe954006070 con 0x7fe95c007030 2023-10-05T19:28:37.951 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.946+0000 7fe96aa28700 1 -- 172.21.15.167:0/2790000739 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:19 5.16 5:68000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x55ddca9d8f80 con 0x55ddca9d6660 2023-10-05T19:28:37.952 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.946+0000 7fe96b229700 1 -- 172.21.15.167:0/2790000739 <== osd.3 v2:172.21.15.167:6808/30036 3 ==== osd_op_reply(19 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fe95c011570 con 0x55ddca9d6660 2023-10-05T19:28:37.952 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.946+0000 7fe96b229700 1 -- 172.21.15.167:0/2790000739 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:20 5.e 5:70000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x55ddca9d8f80 con 0x55ddca9d6660 2023-10-05T19:28:37.952 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.946+0000 7fe96b229700 1 -- 172.21.15.167:0/2790000739 <== osd.3 v2:172.21.15.167:6808/30036 4 ==== osd_op_reply(20 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fe95c011570 con 0x55ddca9d6660 2023-10-05T19:28:37.952 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.946+0000 7fe96b229700 1 -- 172.21.15.167:0/2790000739 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:21 5.1e 5:78000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x55ddca9d8f80 con 0x7fe95400eb60 2023-10-05T19:28:37.953 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.946+0000 7fe96ba2a700 1 -- 172.21.15.167:0/2790000739 <== osd.0 v2:172.21.15.118:6800/27690 5 ==== osd_op_reply(21 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fe96400a5a0 con 0x7fe95400eb60 2023-10-05T19:28:37.953 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.946+0000 7fe96ba2a700 1 -- 172.21.15.167:0/2790000739 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:22 5.1 5:80000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x55ddca9d8f80 con 0x55ddca9d6660 2023-10-05T19:28:37.954 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.946+0000 7fe96b229700 1 -- 172.21.15.167:0/2790000739 <== osd.3 v2:172.21.15.167:6808/30036 5 ==== osd_op_reply(22 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fe95c011570 con 0x55ddca9d6660 2023-10-05T19:28:37.954 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.946+0000 7fe96b229700 1 -- 172.21.15.167:0/2790000739 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:23 5.11 5:88000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x55ddca9d8f80 con 0x55ddca9d6660 2023-10-05T19:28:37.954 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.950+0000 7fe96b229700 1 -- 172.21.15.167:0/2790000739 <== osd.3 v2:172.21.15.167:6808/30036 6 ==== osd_op_reply(23 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fe95c011570 con 0x55ddca9d6660 2023-10-05T19:28:37.955 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.950+0000 7fe96b229700 1 -- 172.21.15.167:0/2790000739 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:24 5.9 5:90000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x55ddca9d8f80 con 0x55ddca9a4510 2023-10-05T19:28:37.955 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.950+0000 7fe96ba2a700 1 -- 172.21.15.167:0/2790000739 <== osd.1 v2:172.21.15.167:6800/30035 10 ==== osd_op_reply(24 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fe96400a5a0 con 0x55ddca9a4510 2023-10-05T19:28:37.955 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.950+0000 7fe96ba2a700 1 -- 172.21.15.167:0/2790000739 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:25 5.19 5:98000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x55ddca9d8f80 con 0x55ddca9a4510 2023-10-05T19:28:37.956 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.950+0000 7fe96ba2a700 1 -- 172.21.15.167:0/2790000739 <== osd.1 v2:172.21.15.167:6800/30035 11 ==== osd_op_reply(25 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fe96400a5a0 con 0x55ddca9a4510 2023-10-05T19:28:37.956 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.950+0000 7fe96ba2a700 1 -- 172.21.15.167:0/2790000739 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:26 5.5 5:a0000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x55ddca9d8f80 con 0x7fe95400eb60 2023-10-05T19:28:37.957 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.950+0000 7fe96ba2a700 1 -- 172.21.15.167:0/2790000739 <== osd.0 v2:172.21.15.118:6800/27690 6 ==== osd_op_reply(26 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fe96400a5a0 con 0x7fe95400eb60 2023-10-05T19:28:37.957 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.950+0000 7fe96ba2a700 1 -- 172.21.15.167:0/2790000739 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:27 5.15 5:a8000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x55ddca9d8f80 con 0x55ddca9d6660 2023-10-05T19:28:37.958 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.954+0000 7fe96b229700 1 -- 172.21.15.167:0/2790000739 <== osd.3 v2:172.21.15.167:6808/30036 7 ==== osd_op_reply(27 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fe95c011570 con 0x55ddca9d6660 2023-10-05T19:28:37.958 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.954+0000 7fe96b229700 1 -- 172.21.15.167:0/2790000739 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:28 5.d 5:b0000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x55ddca9d8f80 con 0x7fe95c007030 2023-10-05T19:28:37.959 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.954+0000 7fe96aa28700 1 -- 172.21.15.167:0/2790000739 <== osd.2 v2:172.21.15.167:6816/30037 4 ==== osd_op_reply(28 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fe954006070 con 0x7fe95c007030 2023-10-05T19:28:37.959 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.954+0000 7fe96aa28700 1 -- 172.21.15.167:0/2790000739 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:29 5.1d 5:b8000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x55ddca9d8f80 con 0x55ddca9a4510 2023-10-05T19:28:37.959 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.954+0000 7fe96ba2a700 1 -- 172.21.15.167:0/2790000739 <== osd.1 v2:172.21.15.167:6800/30035 12 ==== osd_op_reply(29 [pgnls start_epoch 30 out=107b] v30'2 uv2 ondisk = 1) v8 ==== 144+0+107 (crc 0 0 0) 0x7fe96400a5a0 con 0x55ddca9a4510 2023-10-05T19:28:37.959 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.954+0000 7fe96ba2a700 1 -- 172.21.15.167:0/2790000739 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:30 5.3 5:c0000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x55ddca9d8f80 con 0x7fe95400eb60 2023-10-05T19:28:37.960 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.954+0000 7fe96ba2a700 1 -- 172.21.15.167:0/2790000739 <== osd.0 v2:172.21.15.118:6800/27690 7 ==== osd_op_reply(30 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fe96400a5a0 con 0x7fe95400eb60 2023-10-05T19:28:37.960 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.954+0000 7fe96ba2a700 1 -- 172.21.15.167:0/2790000739 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:31 5.13 5:c8000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x55ddca9d8f80 con 0x55ddca9d6660 2023-10-05T19:28:37.961 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.954+0000 7fe96b229700 1 -- 172.21.15.167:0/2790000739 <== osd.3 v2:172.21.15.167:6808/30036 8 ==== osd_op_reply(31 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fe95c011570 con 0x55ddca9d6660 2023-10-05T19:28:37.961 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.954+0000 7fe96b229700 1 -- 172.21.15.167:0/2790000739 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:32 5.b 5:d0000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x55ddca9d8f80 con 0x7fe95400eb60 2023-10-05T19:28:37.962 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.958+0000 7fe96ba2a700 1 -- 172.21.15.167:0/2790000739 <== osd.0 v2:172.21.15.118:6800/27690 8 ==== osd_op_reply(32 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fe96400a5a0 con 0x7fe95400eb60 2023-10-05T19:28:37.962 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.958+0000 7fe96ba2a700 1 -- 172.21.15.167:0/2790000739 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:33 5.1b 5:d8000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x55ddca9d8f80 con 0x7fe95c007030 2023-10-05T19:28:37.963 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.958+0000 7fe96aa28700 1 -- 172.21.15.167:0/2790000739 <== osd.2 v2:172.21.15.167:6816/30037 5 ==== osd_op_reply(33 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fe954006070 con 0x7fe95c007030 2023-10-05T19:28:37.963 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.958+0000 7fe96aa28700 1 -- 172.21.15.167:0/2790000739 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:34 5.7 5:e0000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x55ddca9d8f80 con 0x55ddca9d6660 2023-10-05T19:28:37.964 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.958+0000 7fe96b229700 1 -- 172.21.15.167:0/2790000739 <== osd.3 v2:172.21.15.167:6808/30036 9 ==== osd_op_reply(34 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fe95c011570 con 0x55ddca9d6660 2023-10-05T19:28:37.964 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.958+0000 7fe96b229700 1 -- 172.21.15.167:0/2790000739 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:35 5.17 5:e8000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x55ddca9d8f80 con 0x7fe95400eb60 2023-10-05T19:28:37.965 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.958+0000 7fe96ba2a700 1 -- 172.21.15.167:0/2790000739 <== osd.0 v2:172.21.15.118:6800/27690 9 ==== osd_op_reply(35 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fe96400a5a0 con 0x7fe95400eb60 2023-10-05T19:28:37.965 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.958+0000 7fe96ba2a700 1 -- 172.21.15.167:0/2790000739 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:36 5.f 5:f0000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x55ddca9d8f80 con 0x55ddca9a4510 2023-10-05T19:28:37.965 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.958+0000 7fe96ba2a700 1 -- 172.21.15.167:0/2790000739 <== osd.1 v2:172.21.15.167:6800/30035 13 ==== osd_op_reply(36 [pgnls start_epoch 30 out=115b] v23'2 uv2 ondisk = 1) v8 ==== 144+0+115 (crc 0 0 0) 0x7fe96400a5a0 con 0x55ddca9a4510 2023-10-05T19:28:37.966 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.958+0000 7fe96ba2a700 1 -- 172.21.15.167:0/2790000739 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:37 5.1f 5:f8000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x55ddca9d8f80 con 0x7fe95c007030 2023-10-05T19:28:37.966 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.962+0000 7fe96aa28700 1 -- 172.21.15.167:0/2790000739 <== osd.2 v2:172.21.15.167:6816/30037 6 ==== osd_op_reply(37 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fe954006070 con 0x7fe95c007030 2023-10-05T19:28:37.967 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.962+0000 7fe96d6f4e40 20 RGWRados::pool_iterate: got zonegroup_info.4f1f719d-9648-417b-b10c-e5d939825419 2023-10-05T19:28:37.967 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.962+0000 7fe96d6f4e40 20 RGWRados::pool_iterate: got zone_info.9a9df965-6612-45d4-a45f-6e3c889bacc2 2023-10-05T19:28:37.967 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.962+0000 7fe96d6f4e40 20 RGWRados::pool_iterate: got zone_names.default 2023-10-05T19:28:37.967 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.962+0000 7fe96d6f4e40 20 RGWRados::pool_iterate: got zonegroups_names.default 2023-10-05T19:28:37.967 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.962+0000 7fe96d6f4e40 20 rados->read ofs=0 len=0 2023-10-05T19:28:37.967 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.962+0000 7fe96d6f4e40 1 -- 172.21.15.167:0/2790000739 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:38 5.12 5:49953fa1:::default.realm:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x55ddca9d5390 con 0x55ddca9a4510 2023-10-05T19:28:37.968 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.962+0000 7fe96ba2a700 1 -- 172.21.15.167:0/2790000739 <== osd.1 v2:172.21.15.167:6800/30035 14 ==== osd_op_reply(38 default.realm [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 ==== 157+0+0 (crc 0 0 0) 0x7fe96400a5a0 con 0x55ddca9a4510 2023-10-05T19:28:37.968 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.962+0000 7fe96d6f4e40 20 rados_obj.operate() r=-2 bl.length=0 2023-10-05T19:28:37.968 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.962+0000 7fe96d6f4e40 20 rados->read ofs=0 len=0 2023-10-05T19:28:37.968 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.962+0000 7fe96d6f4e40 1 -- 172.21.15.167:0/2790000739 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:39 5.f 5:f4c53578:::zonegroups_names.default:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x55ddca9d5c80 con 0x55ddca9a4510 2023-10-05T19:28:37.968 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.962+0000 7fe96ba2a700 1 -- 172.21.15.167:0/2790000739 <== osd.1 v2:172.21.15.167:6800/30035 15 ==== osd_op_reply(39 zonegroups_names.default [read 0~46 out=46b] v0'0 uv2 ondisk = 0) v8 ==== 168+0+46 (crc 0 0 0) 0x7fe96400a5a0 con 0x55ddca9a4510 2023-10-05T19:28:37.968 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.962+0000 7fe96d6f4e40 20 rados_obj.operate() r=0 bl.length=46 2023-10-05T19:28:37.968 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.962+0000 7fe96d6f4e40 20 rados->read ofs=0 len=0 2023-10-05T19:28:37.969 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.962+0000 7fe96d6f4e40 1 -- 172.21.15.167:0/2790000739 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:40 5.c 5:30d299a5:::zonegroup_info.4f1f719d-9648-417b-b10c-e5d939825419:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x55ddca9d5c80 con 0x7fe95400eb60 2023-10-05T19:28:37.970 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.962+0000 7fe96ba2a700 1 -- 172.21.15.167:0/2790000739 <== osd.0 v2:172.21.15.118:6800/27690 10 ==== osd_op_reply(40 zonegroup_info.4f1f719d-9648-417b-b10c-e5d939825419 [read 0~362 out=362b] v0'0 uv1 ondisk = 0) v8 ==== 195+0+362 (crc 0 0 0) 0x7fe96400a5a0 con 0x7fe95400eb60 2023-10-05T19:28:37.970 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.962+0000 7fe96d6f4e40 20 rados_obj.operate() r=0 bl.length=362 2023-10-05T19:28:37.970 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.962+0000 7fe96d6f4e40 20 rados->read ofs=0 len=0 2023-10-05T19:28:37.970 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.962+0000 7fe96d6f4e40 1 -- 172.21.15.167:0/2790000739 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:41 5.13 5:cbbf6c60:::converted:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x55ddca9d5c80 con 0x55ddca9d6660 2023-10-05T19:28:37.970 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.962+0000 7fe96b229700 1 -- 172.21.15.167:0/2790000739 <== osd.3 v2:172.21.15.167:6808/30036 10 ==== osd_op_reply(41 converted [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 ==== 153+0+0 (crc 0 0 0) 0x7fe95c011570 con 0x55ddca9d6660 2023-10-05T19:28:37.970 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.962+0000 7fe96d6f4e40 20 rados_obj.operate() r=-2 bl.length=0 2023-10-05T19:28:37.970 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.962+0000 7fe96d6f4e40 20 rados->read ofs=0 len=0 2023-10-05T19:28:37.971 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.962+0000 7fe96d6f4e40 1 -- 172.21.15.167:0/2790000739 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:42 5.12 5:49953fa1:::default.realm:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x55ddca9d93f0 con 0x55ddca9a4510 2023-10-05T19:28:37.971 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.966+0000 7fe96ba2a700 1 -- 172.21.15.167:0/2790000739 <== osd.1 v2:172.21.15.167:6800/30035 16 ==== osd_op_reply(42 default.realm [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 ==== 157+0+0 (crc 0 0 0) 0x7fe96400a5a0 con 0x55ddca9a4510 2023-10-05T19:28:37.971 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.966+0000 7fe96d6f4e40 20 rados_obj.operate() r=-2 bl.length=0 2023-10-05T19:28:37.971 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.966+0000 7fe96d6f4e40 20 rados->read ofs=0 len=0 2023-10-05T19:28:37.971 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.966+0000 7fe96d6f4e40 1 -- 172.21.15.167:0/2790000739 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:43 5.f 5:f4c53578:::zonegroups_names.default:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x55ddca9d97d0 con 0x55ddca9a4510 2023-10-05T19:28:37.971 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.966+0000 7fe96ba2a700 1 -- 172.21.15.167:0/2790000739 <== osd.1 v2:172.21.15.167:6800/30035 17 ==== osd_op_reply(43 zonegroups_names.default [read 0~46 out=46b] v0'0 uv2 ondisk = 0) v8 ==== 168+0+46 (crc 0 0 0) 0x7fe96400a5a0 con 0x55ddca9a4510 2023-10-05T19:28:37.971 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.966+0000 7fe96d6f4e40 20 rados_obj.operate() r=0 bl.length=46 2023-10-05T19:28:37.972 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.966+0000 7fe96d6f4e40 1 -- 172.21.15.167:0/2790000739 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:44 5.0 5:00000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x55ddca9da7e0 con 0x55ddca9d6660 2023-10-05T19:28:37.973 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.966+0000 7fe96b229700 1 -- 172.21.15.167:0/2790000739 <== osd.3 v2:172.21.15.167:6808/30036 11 ==== osd_op_reply(44 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fe95c011570 con 0x55ddca9d6660 2023-10-05T19:28:37.973 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.966+0000 7fe96b229700 1 -- 172.21.15.167:0/2790000739 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:45 5.10 5:08000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x55ddca9da7e0 con 0x7fe95c007030 2023-10-05T19:28:37.974 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.966+0000 7fe96aa28700 1 -- 172.21.15.167:0/2790000739 <== osd.2 v2:172.21.15.167:6816/30037 7 ==== osd_op_reply(45 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fe954006070 con 0x7fe95c007030 2023-10-05T19:28:37.974 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.970+0000 7fe96aa28700 1 -- 172.21.15.167:0/2790000739 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:46 5.8 5:10000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x55ddca9da7e0 con 0x7fe95400eb60 2023-10-05T19:28:37.978 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.970+0000 7fe96ba2a700 1 -- 172.21.15.167:0/2790000739 <== osd.0 v2:172.21.15.118:6800/27690 11 ==== osd_op_reply(46 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fe96400a5a0 con 0x7fe95400eb60 2023-10-05T19:28:37.978 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.970+0000 7fe96ba2a700 1 -- 172.21.15.167:0/2790000739 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:47 5.18 5:18000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x55ddca9d5390 con 0x55ddca9a4510 2023-10-05T19:28:37.978 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.970+0000 7fe96ba2a700 1 -- 172.21.15.167:0/2790000739 <== osd.1 v2:172.21.15.167:6800/30035 18 ==== osd_op_reply(47 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fe96400a5a0 con 0x55ddca9a4510 2023-10-05T19:28:37.978 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.970+0000 7fe96ba2a700 1 -- 172.21.15.167:0/2790000739 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:48 5.4 5:20000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x55ddca9d5390 con 0x55ddca9a4510 2023-10-05T19:28:37.978 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.970+0000 7fe96ba2a700 1 -- 172.21.15.167:0/2790000739 <== osd.1 v2:172.21.15.167:6800/30035 19 ==== osd_op_reply(48 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fe96400a5a0 con 0x55ddca9a4510 2023-10-05T19:28:37.978 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.970+0000 7fe96ba2a700 1 -- 172.21.15.167:0/2790000739 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:49 5.14 5:28000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x55ddca9d5390 con 0x7fe95400eb60 2023-10-05T19:28:37.978 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.970+0000 7fe96ba2a700 1 -- 172.21.15.167:0/2790000739 <== osd.0 v2:172.21.15.118:6800/27690 12 ==== osd_op_reply(49 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fe96400a5a0 con 0x7fe95400eb60 2023-10-05T19:28:37.978 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.970+0000 7fe96ba2a700 1 -- 172.21.15.167:0/2790000739 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:50 5.c 5:30000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x55ddca9d5390 con 0x7fe95400eb60 2023-10-05T19:28:37.979 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.974+0000 7fe96ba2a700 1 -- 172.21.15.167:0/2790000739 <== osd.0 v2:172.21.15.118:6800/27690 13 ==== osd_op_reply(50 [pgnls start_epoch 30 out=112b] v23'1 uv1 ondisk = 1) v8 ==== 144+0+112 (crc 0 0 0) 0x7fe96400a5a0 con 0x7fe95400eb60 2023-10-05T19:28:37.979 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.974+0000 7fe96ba2a700 1 -- 172.21.15.167:0/2790000739 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:51 5.1c 5:38000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x55ddca9d5390 con 0x7fe95c007030 2023-10-05T19:28:37.979 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.974+0000 7fe96aa28700 1 -- 172.21.15.167:0/2790000739 <== osd.2 v2:172.21.15.167:6816/30037 8 ==== osd_op_reply(51 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fe954006070 con 0x7fe95c007030 2023-10-05T19:28:37.979 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.974+0000 7fe96aa28700 1 -- 172.21.15.167:0/2790000739 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:52 5.2 5:40000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x55ddca9d5390 con 0x55ddca9d6660 2023-10-05T19:28:37.979 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.974+0000 7fe96b229700 1 -- 172.21.15.167:0/2790000739 <== osd.3 v2:172.21.15.167:6808/30036 12 ==== osd_op_reply(52 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fe95c011570 con 0x55ddca9d6660 2023-10-05T19:28:37.980 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.974+0000 7fe96b229700 1 -- 172.21.15.167:0/2790000739 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:53 5.12 5:48000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x55ddca9d5390 con 0x55ddca9a4510 2023-10-05T19:28:37.980 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.974+0000 7fe96ba2a700 1 -- 172.21.15.167:0/2790000739 <== osd.1 v2:172.21.15.167:6800/30035 20 ==== osd_op_reply(53 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fe96400a5a0 con 0x55ddca9a4510 2023-10-05T19:28:37.980 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.974+0000 7fe96ba2a700 1 -- 172.21.15.167:0/2790000739 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:54 5.a 5:50000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x55ddca9d5390 con 0x7fe95400eb60 2023-10-05T19:28:37.981 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.974+0000 7fe96ba2a700 1 -- 172.21.15.167:0/2790000739 <== osd.0 v2:172.21.15.118:6800/27690 14 ==== osd_op_reply(54 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fe96400a5a0 con 0x7fe95400eb60 2023-10-05T19:28:37.981 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.974+0000 7fe96ba2a700 1 -- 172.21.15.167:0/2790000739 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:55 5.1a 5:58000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x55ddca9d5390 con 0x55ddca9a4510 2023-10-05T19:28:37.982 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.978+0000 7fe96ba2a700 1 -- 172.21.15.167:0/2790000739 <== osd.1 v2:172.21.15.167:6800/30035 21 ==== osd_op_reply(55 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fe96400a5a0 con 0x55ddca9a4510 2023-10-05T19:28:37.982 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.978+0000 7fe96ba2a700 1 -- 172.21.15.167:0/2790000739 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:56 5.6 5:60000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x55ddca9d5390 con 0x7fe95c007030 2023-10-05T19:28:37.983 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.978+0000 7fe96aa28700 1 -- 172.21.15.167:0/2790000739 <== osd.2 v2:172.21.15.167:6816/30037 9 ==== osd_op_reply(56 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fe954006070 con 0x7fe95c007030 2023-10-05T19:28:37.983 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.978+0000 7fe96aa28700 1 -- 172.21.15.167:0/2790000739 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:57 5.16 5:68000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x55ddca9d5390 con 0x55ddca9d6660 2023-10-05T19:28:37.983 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.978+0000 7fe96b229700 1 -- 172.21.15.167:0/2790000739 <== osd.3 v2:172.21.15.167:6808/30036 13 ==== osd_op_reply(57 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fe95c011570 con 0x55ddca9d6660 2023-10-05T19:28:37.983 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.978+0000 7fe96b229700 1 -- 172.21.15.167:0/2790000739 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:58 5.e 5:70000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x55ddca9d5390 con 0x55ddca9d6660 2023-10-05T19:28:37.984 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.978+0000 7fe96b229700 1 -- 172.21.15.167:0/2790000739 <== osd.3 v2:172.21.15.167:6808/30036 14 ==== osd_op_reply(58 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fe95c011570 con 0x55ddca9d6660 2023-10-05T19:28:37.984 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.978+0000 7fe96b229700 1 -- 172.21.15.167:0/2790000739 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:59 5.1e 5:78000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x55ddca9d5390 con 0x7fe95400eb60 2023-10-05T19:28:37.985 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.978+0000 7fe96ba2a700 1 -- 172.21.15.167:0/2790000739 <== osd.0 v2:172.21.15.118:6800/27690 15 ==== osd_op_reply(59 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fe96403f040 con 0x7fe95400eb60 2023-10-05T19:28:37.985 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.978+0000 7fe96ba2a700 1 -- 172.21.15.167:0/2790000739 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:60 5.1 5:80000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x55ddca9d5390 con 0x55ddca9d6660 2023-10-05T19:28:37.986 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.982+0000 7fe96b229700 1 -- 172.21.15.167:0/2790000739 <== osd.3 v2:172.21.15.167:6808/30036 15 ==== osd_op_reply(60 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fe95c011570 con 0x55ddca9d6660 2023-10-05T19:28:37.986 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.982+0000 7fe96b229700 1 -- 172.21.15.167:0/2790000739 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:61 5.11 5:88000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x55ddca9d5390 con 0x55ddca9d6660 2023-10-05T19:28:37.987 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.982+0000 7fe96b229700 1 -- 172.21.15.167:0/2790000739 <== osd.3 v2:172.21.15.167:6808/30036 16 ==== osd_op_reply(61 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fe95c011570 con 0x55ddca9d6660 2023-10-05T19:28:37.987 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.982+0000 7fe96b229700 1 -- 172.21.15.167:0/2790000739 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:62 5.9 5:90000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x55ddca9d5390 con 0x55ddca9a4510 2023-10-05T19:28:37.988 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.982+0000 7fe96ba2a700 1 -- 172.21.15.167:0/2790000739 <== osd.1 v2:172.21.15.167:6800/30035 22 ==== osd_op_reply(62 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fe96403f040 con 0x55ddca9a4510 2023-10-05T19:28:37.988 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.982+0000 7fe96ba2a700 1 -- 172.21.15.167:0/2790000739 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:63 5.19 5:98000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x55ddca9d5390 con 0x55ddca9a4510 2023-10-05T19:28:37.988 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.982+0000 7fe96ba2a700 1 -- 172.21.15.167:0/2790000739 <== osd.1 v2:172.21.15.167:6800/30035 23 ==== osd_op_reply(63 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fe96403f040 con 0x55ddca9a4510 2023-10-05T19:28:37.988 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.982+0000 7fe96ba2a700 1 -- 172.21.15.167:0/2790000739 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:64 5.5 5:a0000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x55ddca9d5390 con 0x7fe95400eb60 2023-10-05T19:28:37.989 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.982+0000 7fe96ba2a700 1 -- 172.21.15.167:0/2790000739 <== osd.0 v2:172.21.15.118:6800/27690 16 ==== osd_op_reply(64 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fe96403f040 con 0x7fe95400eb60 2023-10-05T19:28:37.989 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.982+0000 7fe96ba2a700 1 -- 172.21.15.167:0/2790000739 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:65 5.15 5:a8000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x55ddca9d5390 con 0x55ddca9d6660 2023-10-05T19:28:37.990 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.986+0000 7fe96b229700 1 -- 172.21.15.167:0/2790000739 <== osd.3 v2:172.21.15.167:6808/30036 17 ==== osd_op_reply(65 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fe95c011570 con 0x55ddca9d6660 2023-10-05T19:28:37.990 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.986+0000 7fe96b229700 1 -- 172.21.15.167:0/2790000739 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:66 5.d 5:b0000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x55ddca9d5390 con 0x7fe95c007030 2023-10-05T19:28:37.991 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.986+0000 7fe96aa28700 1 -- 172.21.15.167:0/2790000739 <== osd.2 v2:172.21.15.167:6816/30037 10 ==== osd_op_reply(66 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fe954006070 con 0x7fe95c007030 2023-10-05T19:28:37.991 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.986+0000 7fe96aa28700 1 -- 172.21.15.167:0/2790000739 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:67 5.1d 5:b8000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x55ddca9d5390 con 0x55ddca9a4510 2023-10-05T19:28:37.992 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.986+0000 7fe96ba2a700 1 -- 172.21.15.167:0/2790000739 <== osd.1 v2:172.21.15.167:6800/30035 24 ==== osd_op_reply(67 [pgnls start_epoch 30 out=107b] v30'2 uv2 ondisk = 1) v8 ==== 144+0+107 (crc 0 0 0) 0x7fe96403f040 con 0x55ddca9a4510 2023-10-05T19:28:37.992 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.986+0000 7fe96ba2a700 1 -- 172.21.15.167:0/2790000739 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:68 5.3 5:c0000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x55ddca9d5390 con 0x7fe95400eb60 2023-10-05T19:28:37.993 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.986+0000 7fe96ba2a700 1 -- 172.21.15.167:0/2790000739 <== osd.0 v2:172.21.15.118:6800/27690 17 ==== osd_op_reply(68 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fe96403f040 con 0x7fe95400eb60 2023-10-05T19:28:37.993 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.986+0000 7fe96ba2a700 1 -- 172.21.15.167:0/2790000739 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:69 5.13 5:c8000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x55ddca9d5390 con 0x55ddca9d6660 2023-10-05T19:28:37.994 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.990+0000 7fe96b229700 1 -- 172.21.15.167:0/2790000739 <== osd.3 v2:172.21.15.167:6808/30036 18 ==== osd_op_reply(69 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fe95c011570 con 0x55ddca9d6660 2023-10-05T19:28:37.994 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.990+0000 7fe96b229700 1 -- 172.21.15.167:0/2790000739 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:70 5.b 5:d0000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x55ddca9d5390 con 0x7fe95400eb60 2023-10-05T19:28:37.995 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.990+0000 7fe96ba2a700 1 -- 172.21.15.167:0/2790000739 <== osd.0 v2:172.21.15.118:6800/27690 18 ==== osd_op_reply(70 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fe96403f040 con 0x7fe95400eb60 2023-10-05T19:28:37.995 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.990+0000 7fe96ba2a700 1 -- 172.21.15.167:0/2790000739 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:71 5.1b 5:d8000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x55ddca9d5390 con 0x7fe95c007030 2023-10-05T19:28:37.996 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.990+0000 7fe96aa28700 1 -- 172.21.15.167:0/2790000739 <== osd.2 v2:172.21.15.167:6816/30037 11 ==== osd_op_reply(71 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fe954006070 con 0x7fe95c007030 2023-10-05T19:28:37.996 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.990+0000 7fe96aa28700 1 -- 172.21.15.167:0/2790000739 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:72 5.7 5:e0000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x55ddca9d5390 con 0x55ddca9d6660 2023-10-05T19:28:37.997 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.990+0000 7fe96b229700 1 -- 172.21.15.167:0/2790000739 <== osd.3 v2:172.21.15.167:6808/30036 19 ==== osd_op_reply(72 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fe95c011570 con 0x55ddca9d6660 2023-10-05T19:28:37.997 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.990+0000 7fe96b229700 1 -- 172.21.15.167:0/2790000739 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:73 5.17 5:e8000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x55ddca9d5390 con 0x7fe95400eb60 2023-10-05T19:28:37.998 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.994+0000 7fe96ba2a700 1 -- 172.21.15.167:0/2790000739 <== osd.0 v2:172.21.15.118:6800/27690 19 ==== osd_op_reply(73 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fe96403f040 con 0x7fe95400eb60 2023-10-05T19:28:37.998 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.994+0000 7fe96ba2a700 1 -- 172.21.15.167:0/2790000739 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:74 5.f 5:f0000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x55ddca9d5390 con 0x55ddca9a4510 2023-10-05T19:28:37.999 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.994+0000 7fe96ba2a700 1 -- 172.21.15.167:0/2790000739 <== osd.1 v2:172.21.15.167:6800/30035 25 ==== osd_op_reply(74 [pgnls start_epoch 30 out=115b] v23'2 uv2 ondisk = 1) v8 ==== 144+0+115 (crc 0 0 0) 0x7fe96403f040 con 0x55ddca9a4510 2023-10-05T19:28:37.999 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.994+0000 7fe96ba2a700 1 -- 172.21.15.167:0/2790000739 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:75 5.1f 5:f8000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x55ddca9d5390 con 0x7fe95c007030 2023-10-05T19:28:38.000 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.994+0000 7fe96aa28700 1 -- 172.21.15.167:0/2790000739 <== osd.2 v2:172.21.15.167:6816/30037 12 ==== osd_op_reply(75 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fe954006070 con 0x7fe95c007030 2023-10-05T19:28:38.000 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.994+0000 7fe96d6f4e40 20 RGWRados::pool_iterate: got zonegroup_info.4f1f719d-9648-417b-b10c-e5d939825419 2023-10-05T19:28:38.000 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.994+0000 7fe96d6f4e40 20 RGWRados::pool_iterate: got zone_info.9a9df965-6612-45d4-a45f-6e3c889bacc2 2023-10-05T19:28:38.000 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.994+0000 7fe96d6f4e40 20 RGWRados::pool_iterate: got zone_names.default 2023-10-05T19:28:38.000 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.994+0000 7fe96d6f4e40 20 RGWRados::pool_iterate: got zonegroups_names.default 2023-10-05T19:28:38.001 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.994+0000 7fe96d6f4e40 20 rados->read ofs=0 len=0 2023-10-05T19:28:38.001 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.994+0000 7fe96d6f4e40 1 -- 172.21.15.167:0/2790000739 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:76 5.f 5:f43b6ece:::zone_names.default:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x55ddca9d97d0 con 0x55ddca9a4510 2023-10-05T19:28:38.002 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.994+0000 7fe96ba2a700 1 -- 172.21.15.167:0/2790000739 <== osd.1 v2:172.21.15.167:6800/30035 26 ==== osd_op_reply(76 zone_names.default [read 0~46 out=46b] v0'0 uv1 ondisk = 0) v8 ==== 162+0+46 (crc 0 0 0) 0x7fe96403f040 con 0x55ddca9a4510 2023-10-05T19:28:38.002 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.994+0000 7fe96d6f4e40 20 rados_obj.operate() r=0 bl.length=46 2023-10-05T19:28:38.002 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.994+0000 7fe96d6f4e40 20 rados->read ofs=0 len=0 2023-10-05T19:28:38.002 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.994+0000 7fe96d6f4e40 1 -- 172.21.15.167:0/2790000739 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:77 5.1d 5:b8374eb4:::zone_info.9a9df965-6612-45d4-a45f-6e3c889bacc2:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x55ddca9d9b80 con 0x55ddca9a4510 2023-10-05T19:28:38.002 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.994+0000 7fe96ba2a700 1 -- 172.21.15.167:0/2790000739 <== osd.1 v2:172.21.15.167:6800/30035 27 ==== osd_op_reply(77 zone_info.9a9df965-6612-45d4-a45f-6e3c889bacc2 [read 0~874 out=874b] v0'0 uv2 ondisk = 0) v8 ==== 190+0+874 (crc 0 0 0) 0x7fe96403f040 con 0x55ddca9a4510 2023-10-05T19:28:38.002 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.994+0000 7fe96d6f4e40 20 rados_obj.operate() r=0 bl.length=874 2023-10-05T19:28:38.002 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.994+0000 7fe96d6f4e40 20 rados->read ofs=0 len=0 2023-10-05T19:28:38.002 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.994+0000 7fe96d6f4e40 1 -- 172.21.15.167:0/2790000739 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:78 5.f 5:f4c53578:::zonegroups_names.default:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x55ddca9d5880 con 0x55ddca9a4510 2023-10-05T19:28:38.003 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.998+0000 7fe96ba2a700 1 -- 172.21.15.167:0/2790000739 <== osd.1 v2:172.21.15.167:6800/30035 28 ==== osd_op_reply(78 zonegroups_names.default [read 0~46 out=46b] v0'0 uv2 ondisk = 0) v8 ==== 168+0+46 (crc 0 0 0) 0x7fe96403f040 con 0x55ddca9a4510 2023-10-05T19:28:38.004 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.998+0000 7fe96d6f4e40 20 rados_obj.operate() r=0 bl.length=46 2023-10-05T19:28:38.004 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.998+0000 7fe96d6f4e40 20 rados->read ofs=0 len=0 2023-10-05T19:28:38.004 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.998+0000 7fe96d6f4e40 1 -- 172.21.15.167:0/2790000739 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:79 5.c 5:30d299a5:::zonegroup_info.4f1f719d-9648-417b-b10c-e5d939825419:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x55ddca9d5860 con 0x7fe95400eb60 2023-10-05T19:28:38.004 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.998+0000 7fe96ba2a700 1 -- 172.21.15.167:0/2790000739 <== osd.0 v2:172.21.15.118:6800/27690 20 ==== osd_op_reply(79 zonegroup_info.4f1f719d-9648-417b-b10c-e5d939825419 [read 0~362 out=362b] v0'0 uv1 ondisk = 0) v8 ==== 195+0+362 (crc 0 0 0) 0x7fe96403f040 con 0x7fe95400eb60 2023-10-05T19:28:38.004 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.998+0000 7fe96d6f4e40 20 rados_obj.operate() r=0 bl.length=362 2023-10-05T19:28:38.004 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.998+0000 7fe96d6f4e40 20 rados->read ofs=0 len=0 2023-10-05T19:28:38.004 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.998+0000 7fe96d6f4e40 1 -- 172.21.15.167:0/2790000739 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:80 5.3 5:c0ad8cbd:::region_map:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x55ddca9da110 con 0x7fe95400eb60 2023-10-05T19:28:38.005 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.998+0000 7fe96ba2a700 1 -- 172.21.15.167:0/2790000739 <== osd.0 v2:172.21.15.118:6800/27690 21 ==== osd_op_reply(80 region_map [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 ==== 154+0+0 (crc 0 0 0) 0x7fe96403f040 con 0x7fe95400eb60 2023-10-05T19:28:38.006 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.998+0000 7fe96d6f4e40 20 rados_obj.operate() r=-2 bl.length=0 2023-10-05T19:28:38.006 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.998+0000 7fe96d6f4e40 20 zone default found 2023-10-05T19:28:38.006 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.998+0000 7fe96d6f4e40 4 Realm: () 2023-10-05T19:28:38.006 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.998+0000 7fe96d6f4e40 4 ZoneGroup: default (4f1f719d-9648-417b-b10c-e5d939825419) 2023-10-05T19:28:38.006 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.998+0000 7fe96d6f4e40 4 Zone: default (9a9df965-6612-45d4-a45f-6e3c889bacc2) 2023-10-05T19:28:38.006 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.998+0000 7fe96d6f4e40 10 cannot find current period zonegroup using local zonegroup configuration 2023-10-05T19:28:38.006 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.998+0000 7fe96d6f4e40 20 zonegroup default 2023-10-05T19:28:38.006 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.998+0000 7fe96d6f4e40 20 rados->read ofs=0 len=0 2023-10-05T19:28:38.007 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.998+0000 7fe96d6f4e40 1 -- 172.21.15.167:0/2790000739 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:81 5.3 5:c52100b6:::period_config.default:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x55ddca9da110 con 0x7fe95400eb60 2023-10-05T19:28:38.007 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.998+0000 7fe96ba2a700 1 -- 172.21.15.167:0/2790000739 <== osd.0 v2:172.21.15.118:6800/27690 22 ==== osd_op_reply(81 period_config.default [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 ==== 165+0+0 (crc 0 0 0) 0x7fe96403f040 con 0x7fe95400eb60 2023-10-05T19:28:38.007 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.998+0000 7fe96d6f4e40 20 rados_obj.operate() r=-2 bl.length=0 2023-10-05T19:28:38.007 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.998+0000 7fe96d6f4e40 20 rados->read ofs=0 len=0 2023-10-05T19:28:38.007 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:37.998+0000 7fe96d6f4e40 1 -- 172.21.15.167:0/2790000739 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:82 6.2 6:4347d321:::bucket.sync-source-hints.:head [call version.read in=11b,read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x55ddca9dba40 con 0x55ddca9d6660 2023-10-05T19:28:38.008 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.002+0000 7fe96b229700 1 -- 172.21.15.167:0/2790000739 <== osd.3 v2:172.21.15.167:6808/30036 20 ==== osd_op_reply(82 bucket.sync-source-hints. [call,read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 ==== 211+0+0 (crc 0 0 0) 0x7fe95c011570 con 0x55ddca9d6660 2023-10-05T19:28:38.008 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.002+0000 7fe96d6f4e40 20 rados_obj.operate() r=-2 bl.length=0 2023-10-05T19:28:38.008 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.002+0000 7fe96d6f4e40 20 rados->read ofs=0 len=0 2023-10-05T19:28:38.008 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.002+0000 7fe96d6f4e40 1 -- 172.21.15.167:0/2790000739 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:83 6.b 6:d467b91b:::bucket.sync-target-hints.:head [call version.read in=11b,read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x55ddca9dbdf0 con 0x55ddca9d6660 2023-10-05T19:28:38.008 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.002+0000 7fe96b229700 1 -- 172.21.15.167:0/2790000739 <== osd.3 v2:172.21.15.167:6808/30036 21 ==== osd_op_reply(83 bucket.sync-target-hints. [call,read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 ==== 211+0+0 (crc 0 0 0) 0x7fe95c011570 con 0x55ddca9d6660 2023-10-05T19:28:38.008 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.002+0000 7fe96d6f4e40 20 rados_obj.operate() r=-2 bl.length=0 2023-10-05T19:28:38.009 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.002+0000 7fe96d6f4e40 20 started sync module instance, tier type = 2023-10-05T19:28:38.009 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.002+0000 7fe96d6f4e40 20 started zone id=9a9df965-6612-45d4-a45f-6e3c889bacc2 (name=default) with tier type = 2023-10-05T19:28:38.009 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.002+0000 7fe96d6f4e40 1 -- 172.21.15.167:0/2790000739 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:84 7.1f 7:f95f44c2:::notify.0:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x55ddca9db100 con 0x7fe95c007030 2023-10-05T19:28:38.010 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.006+0000 7fe96aa28700 1 -- 172.21.15.167:0/2790000739 <== osd.2 v2:172.21.15.167:6816/30037 13 ==== osd_op_reply(84 notify.0 [create] v30'174 uv174 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7fe954006070 con 0x7fe95c007030 2023-10-05T19:28:38.010 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.006+0000 7fe96d6f4e40 1 -- 172.21.15.167:0/2790000739 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:85 7.1f 7:f95f44c2:::notify.0:head [watch watch cookie 94411075459968] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x55ddca9dc1a0 con 0x7fe95c007030 2023-10-05T19:28:38.010 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.006+0000 7fe96d6f4e40 1 -- 172.21.15.167:0/2790000739 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:86 7.0 7:05bf5b68:::notify.1:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x55ddca9dca40 con 0x55ddca9a4510 2023-10-05T19:28:38.013 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.006+0000 7fe96ba2a700 1 -- 172.21.15.167:0/2790000739 <== osd.1 v2:172.21.15.167:6800/30035 29 ==== osd_op_reply(86 notify.1 [create] v30'174 uv174 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7fe96403f040 con 0x55ddca9a4510 2023-10-05T19:28:38.013 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.006+0000 7fe96d6f4e40 1 -- 172.21.15.167:0/2790000739 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:87 7.0 7:05bf5b68:::notify.1:head [watch watch cookie 94411075470960] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x55ddca9dcff0 con 0x55ddca9a4510 2023-10-05T19:28:38.013 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.006+0000 7fe96d6f4e40 1 -- 172.21.15.167:0/2790000739 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:88 7.15 7:a93a5511:::notify.2:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x55ddca9dda40 con 0x55ddca9d6660 2023-10-05T19:28:38.014 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.006+0000 7fe96aa28700 1 -- 172.21.15.167:0/2790000739 <== osd.2 v2:172.21.15.167:6816/30037 14 ==== osd_op_reply(85 notify.0 [watch watch cookie 94411075459968] v30'175 uv174 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7fe954006070 con 0x7fe95c007030 2023-10-05T19:28:38.016 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.010+0000 7fe96b229700 1 -- 172.21.15.167:0/2790000739 <== osd.3 v2:172.21.15.167:6808/30036 22 ==== osd_op_reply(88 notify.2 [create] v30'174 uv174 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7fe95c011570 con 0x55ddca9d6660 2023-10-05T19:28:38.016 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.010+0000 7fe96d6f4e40 1 -- 172.21.15.167:0/2790000739 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:89 7.15 7:a93a5511:::notify.2:head [watch watch cookie 94411075466496] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x55ddca9dde40 con 0x55ddca9d6660 2023-10-05T19:28:38.016 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.010+0000 7fe96d6f4e40 1 -- 172.21.15.167:0/2790000739 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:90 7.e 7:7759931f:::notify.3:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x55ddca9de720 con 0x7fe95c007030 2023-10-05T19:28:38.017 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.010+0000 7fe96ba2a700 1 -- 172.21.15.167:0/2790000739 <== osd.1 v2:172.21.15.167:6800/30035 30 ==== osd_op_reply(87 notify.1 [watch watch cookie 94411075470960] v30'175 uv174 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7fe96403f040 con 0x55ddca9a4510 2023-10-05T19:28:38.018 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.014+0000 7fe96aa28700 1 -- 172.21.15.167:0/2790000739 <== osd.2 v2:172.21.15.167:6816/30037 15 ==== osd_op_reply(90 notify.3 [create] v30'174 uv174 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7fe954006070 con 0x7fe95c007030 2023-10-05T19:28:38.019 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.014+0000 7fe96d6f4e40 1 -- 172.21.15.167:0/2790000739 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:91 7.e 7:7759931f:::notify.3:head [watch watch cookie 94411075472064] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x55ddca9de310 con 0x7fe95c007030 2023-10-05T19:28:38.019 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.014+0000 7fe96d6f4e40 1 -- 172.21.15.167:0/2790000739 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:92 7.d 7:b4812045:::notify.4:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x55ddca9df7c0 con 0x55ddca9a4510 2023-10-05T19:28:38.019 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.014+0000 7fe96b229700 1 -- 172.21.15.167:0/2790000739 <== osd.3 v2:172.21.15.167:6808/30036 23 ==== osd_op_reply(89 notify.2 [watch watch cookie 94411075466496] v30'175 uv174 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7fe95c011570 con 0x55ddca9d6660 2023-10-05T19:28:38.021 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.014+0000 7fe96aa28700 1 -- 172.21.15.167:0/2790000739 <== osd.2 v2:172.21.15.167:6816/30037 16 ==== osd_op_reply(91 notify.3 [watch watch cookie 94411075472064] v30'175 uv174 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7fe954006070 con 0x7fe95c007030 2023-10-05T19:28:38.021 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.014+0000 7fe96ba2a700 1 -- 172.21.15.167:0/2790000739 <== osd.1 v2:172.21.15.167:6800/30035 31 ==== osd_op_reply(92 notify.4 [create] v30'174 uv174 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7fe96403f040 con 0x55ddca9a4510 2023-10-05T19:28:38.021 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.014+0000 7fe96d6f4e40 1 -- 172.21.15.167:0/2790000739 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:93 7.d 7:b4812045:::notify.4:head [watch watch cookie 94411075480352] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x55ddca9df290 con 0x55ddca9a4510 2023-10-05T19:28:38.022 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.014+0000 7fe96d6f4e40 1 -- 172.21.15.167:0/2790000739 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:94 7.3 7:c609908c:::notify.5:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x55ddca9e00b0 con 0x7fe95400eb60 2023-10-05T19:28:38.024 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.018+0000 7fe96ba2a700 1 -- 172.21.15.167:0/2790000739 <== osd.1 v2:172.21.15.167:6800/30035 32 ==== osd_op_reply(93 notify.4 [watch watch cookie 94411075480352] v30'175 uv174 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7fe96403f040 con 0x55ddca9a4510 2023-10-05T19:28:38.024 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.018+0000 7fe96ba2a700 1 -- 172.21.15.167:0/2790000739 <== osd.0 v2:172.21.15.118:6800/27690 23 ==== osd_op_reply(94 notify.5 [create] v30'174 uv174 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7fe96403f040 con 0x7fe95400eb60 2023-10-05T19:28:38.025 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.018+0000 7fe96d6f4e40 1 -- 172.21.15.167:0/2790000739 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:95 7.3 7:c609908c:::notify.5:head [watch watch cookie 94411075474704] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x55ddca9e0060 con 0x7fe95400eb60 2023-10-05T19:28:38.025 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.018+0000 7fe96d6f4e40 1 -- 172.21.15.167:0/2790000739 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:96 7.14 7:2b04a3e9:::notify.6:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x55ddca9e0a70 con 0x7fe95400eb60 2023-10-05T19:28:38.027 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.022+0000 7fe96ba2a700 1 -- 172.21.15.167:0/2790000739 <== osd.0 v2:172.21.15.118:6800/27690 24 ==== osd_op_reply(95 notify.5 [watch watch cookie 94411075474704] v30'175 uv174 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7fe96403f040 con 0x7fe95400eb60 2023-10-05T19:28:38.027 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.022+0000 7fe96ba2a700 1 -- 172.21.15.167:0/2790000739 <== osd.0 v2:172.21.15.118:6800/27690 25 ==== osd_op_reply(96 notify.6 [create] v30'174 uv174 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7fe96403f040 con 0x7fe95400eb60 2023-10-05T19:28:38.027 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.022+0000 7fe96d6f4e40 1 -- 172.21.15.167:0/2790000739 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:97 7.14 7:2b04a3e9:::notify.6:head [watch watch cookie 94411075485088] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x55ddca9e0a70 con 0x7fe95400eb60 2023-10-05T19:28:38.028 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.022+0000 7fe96d6f4e40 1 -- 172.21.15.167:0/2790000739 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:98 7.9 7:93e5b521:::notify.7:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x55ddca9e1480 con 0x7fe95400eb60 2023-10-05T19:28:38.030 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.022+0000 7fe96ba2a700 1 -- 172.21.15.167:0/2790000739 <== osd.0 v2:172.21.15.118:6800/27690 26 ==== osd_op_reply(97 notify.6 [watch watch cookie 94411075485088] v30'175 uv174 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7fe96403f040 con 0x7fe95400eb60 2023-10-05T19:28:38.030 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.022+0000 7fe96ba2a700 1 -- 172.21.15.167:0/2790000739 <== osd.0 v2:172.21.15.118:6800/27690 27 ==== osd_op_reply(98 notify.7 [create] v30'174 uv174 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7fe96403f040 con 0x7fe95400eb60 2023-10-05T19:28:38.030 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.022+0000 7fe96d6f4e40 1 -- 172.21.15.167:0/2790000739 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:99 7.9 7:93e5b521:::notify.7:head [watch watch cookie 94411075487376] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x55ddca9e1480 con 0x7fe95400eb60 2023-10-05T19:28:38.030 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.022+0000 7fe96d6f4e40 20 add_watcher() i=0 2023-10-05T19:28:38.030 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.022+0000 7fe96d6f4e40 20 add_watcher() i=1 2023-10-05T19:28:38.030 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.022+0000 7fe96d6f4e40 20 add_watcher() i=2 2023-10-05T19:28:38.030 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.022+0000 7fe96d6f4e40 20 add_watcher() i=3 2023-10-05T19:28:38.030 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.022+0000 7fe96d6f4e40 20 add_watcher() i=4 2023-10-05T19:28:38.031 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.022+0000 7fe96d6f4e40 20 add_watcher() i=5 2023-10-05T19:28:38.031 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.022+0000 7fe96d6f4e40 20 add_watcher() i=6 2023-10-05T19:28:38.032 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.026+0000 7fe96ba2a700 1 -- 172.21.15.167:0/2790000739 <== osd.0 v2:172.21.15.118:6800/27690 28 ==== osd_op_reply(99 notify.7 [watch watch cookie 94411075487376] v30'175 uv174 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7fe96403f040 con 0x7fe95400eb60 2023-10-05T19:28:38.032 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.026+0000 7fe96d6f4e40 20 add_watcher() i=7 2023-10-05T19:28:38.032 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.026+0000 7fe96d6f4e40 2 all 8 watchers are set, enabling cache 2023-10-05T19:28:38.032 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.026+0000 7fe96d6f4e40 1 -- 172.21.15.167:0/2790000739 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:100 6.4 6:22d26bf9:::data_loggenerations_metadata:head [call version.check_conds in=74b,call version.read in=11b,read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x55ddca9e4550 con 0x55ddca9a4510 2023-10-05T19:28:38.033 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.026+0000 7fe96ba2a700 1 -- 172.21.15.167:0/2790000739 <== osd.1 v2:172.21.15.167:6800/30035 33 ==== osd_op_reply(100 data_loggenerations_metadata [call,call out=48b,read 0~28 out=28b] v0'0 uv1 ondisk = 0) v8 ==== 256+0+76 (crc 0 0 0) 0x7fe96403f040 con 0x55ddca9a4510 2023-10-05T19:28:38.033 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.026+0000 7fe96d6f4e40 1 -- 172.21.15.167:0/2790000739 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:101 6.4 6:22d26bf9:::data_loggenerations_metadata:head [watch watch cookie 94411075489680] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x55ddca9e18d0 con 0x55ddca9a4510 2023-10-05T19:28:38.035 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.030+0000 7fe96ba2a700 1 -- 172.21.15.167:0/2790000739 <== osd.1 v2:172.21.15.167:6800/30035 34 ==== osd_op_reply(101 data_loggenerations_metadata [watch watch cookie 94411075489680] v30'163 uv1 ondisk = 0) v8 ==== 172+0+0 (crc 0 0 0) 0x7fe96403f040 con 0x55ddca9a4510 2023-10-05T19:28:38.035 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.030+0000 7fe93cfe1700 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2023-10-05T19:28:38.035 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.030+0000 7fe96d6f4e40 20 check_secure_mon_conn(): auth registy supported: methods=[2] modes=[2,1] 2023-10-05T19:28:38.035 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.030+0000 7fe96d6f4e40 20 check_secure_mon_conn(): mode 1 is insecure 2023-10-05T19:28:38.036 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.030+0000 7fe96d6f4e40 1 Processor -- start 2023-10-05T19:28:38.036 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.030+0000 7fe96d6f4e40 1 -- start start 2023-10-05T19:28:38.037 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.030+0000 7fe96d6f4e40 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55ddca9eed20 0x55ddca9ef0f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-05T19:28:38.037 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.030+0000 7fe96d6f4e40 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ddca9ef630 con 0x55ddca9eed20 2023-10-05T19:28:38.037 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.030+0000 7fe96aa28700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55ddca9eed20 0x55ddca9ef0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-05T19:28:38.037 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.030+0000 7fe96aa28700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55ddca9eed20 0x55ddca9ef0f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.118:3300/0 says I am v2:172.21.15.167:42134/0 (socket says 172.21.15.167:42134) 2023-10-05T19:28:38.037 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.030+0000 7fe96aa28700 1 -- 172.21.15.167:0/2778960918 learned_addr learned my addr 172.21.15.167:0/2778960918 (peer_addr_for_me v2:172.21.15.167:0/0) 2023-10-05T19:28:38.038 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.030+0000 7fe96aa28700 1 -- 172.21.15.167:0/2778960918 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55ddca9ef770 con 0x55ddca9eed20 2023-10-05T19:28:38.038 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.030+0000 7fe96aa28700 1 --2- 172.21.15.167:0/2778960918 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55ddca9eed20 0x55ddca9ef0f0 secure :-1 s=READY pgs=486 cs=0 l=1 rev1=1 crypto rx=0x55ddca9d3dc0 tx=0x7fe954006000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c93a7a45977c477d server_cookie=0 in_seq=0 out_seq=0 2023-10-05T19:28:38.038 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.030+0000 7fe8b7fff700 1 -- 172.21.15.167:0/2778960918 <== mon.0 v2:172.21.15.118:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fe954023040 con 0x55ddca9eed20 2023-10-05T19:28:38.038 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.030+0000 7fe8b7fff700 1 -- 172.21.15.167:0/2778960918 <== mon.0 v2:172.21.15.118:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe954024040 con 0x55ddca9eed20 2023-10-05T19:28:38.038 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.034+0000 7fe8b7fff700 1 -- 172.21.15.167:0/2778960918 <== mon.0 v2:172.21.15.118:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fe954013040 con 0x55ddca9eed20 2023-10-05T19:28:38.039 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.034+0000 7fe96d6f4e40 1 -- 172.21.15.167:0/2778960918 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55ddca9eed20 msgr2=0x55ddca9ef0f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-05T19:28:38.040 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.034+0000 7fe96d6f4e40 1 --2- 172.21.15.167:0/2778960918 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55ddca9eed20 0x55ddca9ef0f0 secure :-1 s=READY pgs=486 cs=0 l=1 rev1=1 crypto rx=0x55ddca9d3dc0 tx=0x7fe954006000 comp rx=0 tx=0).stop 2023-10-05T19:28:38.040 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.034+0000 7fe96d6f4e40 1 -- 172.21.15.167:0/2778960918 shutdown_connections 2023-10-05T19:28:38.040 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.034+0000 7fe96d6f4e40 1 --2- 172.21.15.167:0/2778960918 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55ddca9eed20 0x55ddca9ef0f0 unknown :-1 s=CLOSED pgs=486 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-05T19:28:38.040 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.034+0000 7fe96d6f4e40 1 -- 172.21.15.167:0/2778960918 >> 172.21.15.167:0/2778960918 conn(0x55ddca9ea0e0 msgr2=0x55ddca9e8230 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-05T19:28:38.040 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.034+0000 7fe96d6f4e40 1 -- 172.21.15.167:0/2778960918 shutdown_connections 2023-10-05T19:28:38.040 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.034+0000 7fe96d6f4e40 1 -- 172.21.15.167:0/2778960918 wait complete. 2023-10-05T19:28:38.041 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.034+0000 7fe96d6f4e40 1 Processor -- start 2023-10-05T19:28:38.041 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.034+0000 7fe96d6f4e40 1 -- start start 2023-10-05T19:28:38.041 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.034+0000 7fe96d6f4e40 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55ddca9eed20 0x55ddca9f4880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-05T19:28:38.041 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.034+0000 7fe96d6f4e40 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ddca9f5880 con 0x55ddca9eed20 2023-10-05T19:28:38.041 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.034+0000 7fe96aa28700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55ddca9eed20 0x55ddca9f4880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-05T19:28:38.041 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.034+0000 7fe96aa28700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55ddca9eed20 0x55ddca9f4880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.118:3300/0 says I am v2:172.21.15.167:42146/0 (socket says 172.21.15.167:42146) 2023-10-05T19:28:38.041 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.034+0000 7fe96aa28700 1 -- 172.21.15.167:0/1697315292 learned_addr learned my addr 172.21.15.167:0/1697315292 (peer_addr_for_me v2:172.21.15.167:0/0) 2023-10-05T19:28:38.042 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.034+0000 7fe96aa28700 1 -- 172.21.15.167:0/1697315292 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55ddca8ebbd0 con 0x55ddca9eed20 2023-10-05T19:28:38.042 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.034+0000 7fe96aa28700 1 --2- 172.21.15.167:0/1697315292 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55ddca9eed20 0x55ddca9f4880 secure :-1 s=READY pgs=487 cs=0 l=1 rev1=1 crypto rx=0x7fe95403b040 tx=0x7fe954036000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-05T19:28:38.042 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.038+0000 7fe8b67fc700 1 -- 172.21.15.167:0/1697315292 <== mon.0 v2:172.21.15.118:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fe954023070 con 0x55ddca9eed20 2023-10-05T19:28:38.043 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.038+0000 7fe8b67fc700 1 -- 172.21.15.167:0/1697315292 <== mon.0 v2:172.21.15.118:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe95401e700 con 0x55ddca9eed20 2023-10-05T19:28:38.043 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.038+0000 7fe8b67fc700 1 -- 172.21.15.167:0/1697315292 <== mon.0 v2:172.21.15.118:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fe95401e8d0 con 0x55ddca9eed20 2023-10-05T19:28:38.043 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.038+0000 7fe96d6f4e40 1 -- 172.21.15.167:0/1697315292 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x55ddca9f4e80 con 0x55ddca9eed20 2023-10-05T19:28:38.043 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.038+0000 7fe96d6f4e40 1 -- 172.21.15.167:0/1697315292 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55ddca9f52c0 con 0x55ddca9eed20 2023-10-05T19:28:38.044 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.038+0000 7fe8b67fc700 1 -- 172.21.15.167:0/1697315292 <== mon.0 v2:172.21.15.118:3300/0 4 ==== mgrmap(e 5) v1 ==== 40329+0+0 (secure 0 0 0) 0x7fe95401ea30 con 0x55ddca9eed20 2023-10-05T19:28:38.044 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.038+0000 7fe8b67fc700 1 --2- 172.21.15.167:0/1697315292 >> [v2:172.21.15.167:6824/29889,v1:172.21.15.167:6825/29889] conn(0x7fe8ac030c50 0x7fe8ac033100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-05T19:28:38.044 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.038+0000 7fe8b67fc700 1 -- 172.21.15.167:0/1697315292 <== mon.0 v2:172.21.15.118:3300/0 5 ==== osd_map(30..30 src has 1..30) v4 ==== 6057+0+0 (secure 0 0 0) 0x7fe954010110 con 0x55ddca9eed20 2023-10-05T19:28:38.045 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.038+0000 7fe96ba2a700 1 --2- 172.21.15.167:0/1697315292 >> [v2:172.21.15.167:6824/29889,v1:172.21.15.167:6825/29889] conn(0x7fe8ac030c50 0x7fe8ac033100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-05T19:28:38.045 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.038+0000 7fe96d6f4e40 5 note: GC not initialized 2023-10-05T19:28:38.045 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.038+0000 7fe96ba2a700 1 --2- 172.21.15.167:0/1697315292 >> [v2:172.21.15.167:6824/29889,v1:172.21.15.167:6825/29889] conn(0x7fe8ac030c50 0x7fe8ac033100 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x55ddca9e4580 tx=0x7fe964012740 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-05T19:28:38.047 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.042+0000 7fe8ab7fe700 20 reqs_thread_entry: start 2023-10-05T19:28:38.048 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.042+0000 7fe8a97fa700 20 reqs_thread_entry: start 2023-10-05T19:28:38.048 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.042+0000 7fe96d6f4e40 20 init_complete bucket index max shards: 11 2023-10-05T19:28:38.048 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.042+0000 7fe96d6f4e40 10 rgw notify: Started notification manager with: 1 workers 2023-10-05T19:28:38.048 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.042+0000 7fe8977fe700 20 rgw notify: INFO: next queues processing will happen at: Thu Oct 5 19:29:08 2023 2023-10-05T19:28:38.048 INFO:teuthology.orchestra.run.smithi167.stderr: 2023-10-05T19:28:38.048 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.042+0000 7fe896ffd700 20 reqs_thread_entry: start 2023-10-05T19:28:38.048 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.042+0000 7fe96d6f4e40 1 --2- 172.21.15.167:0/1697315292 >> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] conn(0x55ddcaa5be70 0x55ddcaa5e250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-05T19:28:38.048 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.042+0000 7fe96d6f4e40 1 -- 172.21.15.167:0/1697315292 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:1 6.9 6:926cbca3:::2023-10-05-19-9a9df965-6612-45d4-a45f-6e3c889bacc2.4223.2-myfoo2:head [delete] snapc 0=[] ondisk+write+peerstat_old+known_if_redirected+supports_pool_eio e30) v8 -- 0x55ddcaa5e790 con 0x55ddcaa5be70 2023-10-05T19:28:38.049 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.042+0000 7fe96b229700 1 --2- 172.21.15.167:0/1697315292 >> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] conn(0x55ddcaa5be70 0x55ddcaa5e250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-05T19:28:38.049 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.042+0000 7fe96b229700 1 --2- 172.21.15.167:0/1697315292 >> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] conn(0x55ddcaa5be70 0x55ddcaa5e250 crc :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-05T19:28:38.051 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.046+0000 7fe96b229700 1 -- 172.21.15.167:0/1697315292 <== osd.3 v2:172.21.15.167:6808/30036 1 ==== osd_op_reply(1 2023-10-05-19-9a9df965-6612-45d4-a45f-6e3c889bacc2.4223.2-myfoo2 [delete] v30'31 uv31 ondisk = 0) v8 ==== 208+0+0 (crc 0 0 0) 0x7fe95c011570 con 0x55ddcaa5be70 2023-10-05T19:28:38.051 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.046+0000 7fe96d6f4e40 1 -- 172.21.15.167:0/2790000739 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:102 7.1f 7:f95f44c2:::notify.0:head [watch unwatch cookie 94411075459968] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x55ddcaa1afb0 con 0x7fe95c007030 2023-10-05T19:28:38.054 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.046+0000 7fe96aa28700 1 -- 172.21.15.167:0/2790000739 <== osd.2 v2:172.21.15.167:6816/30037 17 ==== osd_op_reply(102 notify.0 [watch unwatch cookie 94411075459968] v30'176 uv174 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7fe954048020 con 0x7fe95c007030 2023-10-05T19:28:38.054 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.046+0000 7fe96d6f4e40 20 remove_watcher() i=0 2023-10-05T19:28:38.054 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.046+0000 7fe96d6f4e40 2 removed watcher, disabling cache 2023-10-05T19:28:38.054 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.046+0000 7fe96d6f4e40 1 -- 172.21.15.167:0/2790000739 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:103 7.0 7:05bf5b68:::notify.1:head [watch unwatch cookie 94411075470960] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x55ddca9d9b80 con 0x55ddca9a4510 2023-10-05T19:28:38.056 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.050+0000 7fe96ba2a700 1 -- 172.21.15.167:0/2790000739 <== osd.1 v2:172.21.15.167:6800/30035 35 ==== osd_op_reply(103 notify.1 [watch unwatch cookie 94411075470960] v30'176 uv174 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7fe96403f040 con 0x55ddca9a4510 2023-10-05T19:28:38.057 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.050+0000 7fe96d6f4e40 20 remove_watcher() i=1 2023-10-05T19:28:38.057 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.050+0000 7fe96d6f4e40 1 -- 172.21.15.167:0/2790000739 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:104 7.15 7:a93a5511:::notify.2:head [watch unwatch cookie 94411075466496] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x55ddca9d9b80 con 0x55ddca9d6660 2023-10-05T19:28:38.060 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.054+0000 7fe96b229700 1 -- 172.21.15.167:0/2790000739 <== osd.3 v2:172.21.15.167:6808/30036 24 ==== osd_op_reply(104 notify.2 [watch unwatch cookie 94411075466496] v30'176 uv174 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7fe95c011570 con 0x55ddca9d6660 2023-10-05T19:28:38.060 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.054+0000 7fe96d6f4e40 20 remove_watcher() i=2 2023-10-05T19:28:38.060 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.054+0000 7fe96d6f4e40 1 -- 172.21.15.167:0/2790000739 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:105 7.e 7:7759931f:::notify.3:head [watch unwatch cookie 94411075472064] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x55ddca9db500 con 0x7fe95c007030 2023-10-05T19:28:38.062 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.054+0000 7fe96aa28700 1 -- 172.21.15.167:0/2790000739 <== osd.2 v2:172.21.15.167:6816/30037 18 ==== osd_op_reply(105 notify.3 [watch unwatch cookie 94411075472064] v30'176 uv174 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7fe954048020 con 0x7fe95c007030 2023-10-05T19:28:38.062 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.054+0000 7fe96d6f4e40 20 remove_watcher() i=3 2023-10-05T19:28:38.062 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.054+0000 7fe96d6f4e40 1 -- 172.21.15.167:0/2790000739 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:106 7.d 7:b4812045:::notify.4:head [watch unwatch cookie 94411075480352] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x55ddca9db500 con 0x55ddca9a4510 2023-10-05T19:28:38.064 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.058+0000 7fe96ba2a700 1 -- 172.21.15.167:0/2790000739 <== osd.1 v2:172.21.15.167:6800/30035 36 ==== osd_op_reply(106 notify.4 [watch unwatch cookie 94411075480352] v30'176 uv174 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7fe96403f040 con 0x55ddca9a4510 2023-10-05T19:28:38.065 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.058+0000 7fe96d6f4e40 20 remove_watcher() i=4 2023-10-05T19:28:38.065 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.058+0000 7fe96d6f4e40 1 -- 172.21.15.167:0/2790000739 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:107 7.3 7:c609908c:::notify.5:head [watch unwatch cookie 94411075474704] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x55ddca9db500 con 0x7fe95400eb60 2023-10-05T19:28:38.067 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.062+0000 7fe96ba2a700 1 -- 172.21.15.167:0/2790000739 <== osd.0 v2:172.21.15.118:6800/27690 29 ==== osd_op_reply(107 notify.5 [watch unwatch cookie 94411075474704] v30'176 uv174 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7fe96403f040 con 0x7fe95400eb60 2023-10-05T19:28:38.067 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.062+0000 7fe96d6f4e40 20 remove_watcher() i=5 2023-10-05T19:28:38.067 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.062+0000 7fe96d6f4e40 1 -- 172.21.15.167:0/2790000739 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:108 7.14 7:2b04a3e9:::notify.6:head [watch unwatch cookie 94411075485088] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x55ddca9dd510 con 0x7fe95400eb60 2023-10-05T19:28:38.070 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.066+0000 7fe96ba2a700 1 -- 172.21.15.167:0/2790000739 <== osd.0 v2:172.21.15.118:6800/27690 30 ==== osd_op_reply(108 notify.6 [watch unwatch cookie 94411075485088] v30'176 uv174 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7fe96403f040 con 0x7fe95400eb60 2023-10-05T19:28:38.070 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.066+0000 7fe96d6f4e40 20 remove_watcher() i=6 2023-10-05T19:28:38.070 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.066+0000 7fe96d6f4e40 1 -- 172.21.15.167:0/2790000739 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:109 7.9 7:93e5b521:::notify.7:head [watch unwatch cookie 94411075487376] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x55ddca9dd510 con 0x7fe95400eb60 2023-10-05T19:28:38.073 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.066+0000 7fe96ba2a700 1 -- 172.21.15.167:0/2790000739 <== osd.0 v2:172.21.15.118:6800/27690 31 ==== osd_op_reply(109 notify.7 [watch unwatch cookie 94411075487376] v30'176 uv174 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7fe96403f040 con 0x7fe95400eb60 2023-10-05T19:28:38.073 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.066+0000 7fe96d6f4e40 20 remove_watcher() i=7 2023-10-05T19:28:38.075 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.070+0000 7fe93cfe1700 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2023-10-05T19:28:38.075 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.070+0000 7fe96d6f4e40 1 -- 172.21.15.167:0/2790000739 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:110 6.4 6:22d26bf9:::data_loggenerations_metadata:head [watch unwatch cookie 94411075489680] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x55ddca9dd510 con 0x55ddca9a4510 2023-10-05T19:28:38.078 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.070+0000 7fe96ba2a700 1 -- 172.21.15.167:0/2790000739 <== osd.1 v2:172.21.15.167:6800/30035 37 ==== osd_op_reply(110 data_loggenerations_metadata [watch unwatch cookie 94411075489680] v30'164 uv1 ondisk = 0) v8 ==== 172+0+0 (crc 0 0 0) 0x7fe96403f040 con 0x55ddca9a4510 2023-10-05T19:28:38.078 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.070+0000 7fe96d6f4e40 1 -- 172.21.15.167:0/2790000739 >> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] conn(0x7fe95400eb60 msgr2=0x7fe95400d8e0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-05T19:28:38.079 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.070+0000 7fe96d6f4e40 1 --2- 172.21.15.167:0/2790000739 >> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] conn(0x7fe95400eb60 0x7fe95400d8e0 crc :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-05T19:28:38.079 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.070+0000 7fe96d6f4e40 1 -- 172.21.15.167:0/2790000739 >> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] conn(0x55ddca9a4510 msgr2=0x55ddca9d2880 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-05T19:28:38.079 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.070+0000 7fe96d6f4e40 1 --2- 172.21.15.167:0/2790000739 >> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] conn(0x55ddca9a4510 0x55ddca9d2880 crc :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-05T19:28:38.079 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.070+0000 7fe96d6f4e40 1 -- 172.21.15.167:0/2790000739 >> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] conn(0x7fe95c007030 msgr2=0x7fe95c011830 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-05T19:28:38.079 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.070+0000 7fe96d6f4e40 1 --2- 172.21.15.167:0/2790000739 >> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] conn(0x7fe95c007030 0x7fe95c011830 crc :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-05T19:28:38.079 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.070+0000 7fe96d6f4e40 1 -- 172.21.15.167:0/2790000739 >> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] conn(0x55ddca9d6660 msgr2=0x55ddca9d8a40 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-05T19:28:38.079 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.070+0000 7fe96d6f4e40 1 --2- 172.21.15.167:0/2790000739 >> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] conn(0x55ddca9d6660 0x55ddca9d8a40 crc :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-05T19:28:38.079 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.070+0000 7fe96d6f4e40 1 -- 172.21.15.167:0/2790000739 >> [v2:172.21.15.167:6824/29889,v1:172.21.15.167:6825/29889] conn(0x7fe94c030c00 msgr2=0x7fe94c0330b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-05T19:28:38.079 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.074+0000 7fe96d6f4e40 1 --2- 172.21.15.167:0/2790000739 >> [v2:172.21.15.167:6824/29889,v1:172.21.15.167:6825/29889] conn(0x7fe94c030c00 0x7fe94c0330b0 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7fe954001df0 tx=0x7fe954001310 comp rx=0 tx=0).stop 2023-10-05T19:28:38.080 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.074+0000 7fe96d6f4e40 1 -- 172.21.15.167:0/2790000739 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55ddca9133d0 msgr2=0x55ddca9c6d60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-05T19:28:38.080 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.074+0000 7fe96d6f4e40 1 --2- 172.21.15.167:0/2790000739 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55ddca9133d0 0x55ddca9c6d60 secure :-1 s=READY pgs=485 cs=0 l=1 rev1=1 crypto rx=0x7fe95c00c010 tx=0x7fe95c01c000 comp rx=0 tx=0).stop 2023-10-05T19:28:38.080 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.074+0000 7fe96ba2a700 1 -- 172.21.15.167:0/2790000739 reap_dead start 2023-10-05T19:28:38.080 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.074+0000 7fe96d6f4e40 1 -- 172.21.15.167:0/2790000739 shutdown_connections 2023-10-05T19:28:38.080 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.074+0000 7fe96d6f4e40 1 -- 172.21.15.167:0/2790000739 >> 172.21.15.167:0/2790000739 conn(0x55ddca999720 msgr2=0x55ddca9ba790 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-05T19:28:38.080 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.074+0000 7fe96d6f4e40 1 -- 172.21.15.167:0/2790000739 shutdown_connections 2023-10-05T19:28:38.080 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.074+0000 7fe96d6f4e40 1 -- 172.21.15.167:0/2790000739 wait complete. 2023-10-05T19:28:38.081 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.074+0000 7fe96d6f4e40 1 -- 172.21.15.167:0/1697315292 >> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] conn(0x55ddcaa5be70 msgr2=0x55ddcaa5e250 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-05T19:28:38.081 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.074+0000 7fe96d6f4e40 1 --2- 172.21.15.167:0/1697315292 >> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] conn(0x55ddcaa5be70 0x55ddcaa5e250 crc :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-05T19:28:38.081 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.074+0000 7fe96d6f4e40 1 -- 172.21.15.167:0/1697315292 >> [v2:172.21.15.167:6824/29889,v1:172.21.15.167:6825/29889] conn(0x7fe8ac030c50 msgr2=0x7fe8ac033100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-05T19:28:38.082 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.074+0000 7fe96d6f4e40 1 --2- 172.21.15.167:0/1697315292 >> [v2:172.21.15.167:6824/29889,v1:172.21.15.167:6825/29889] conn(0x7fe8ac030c50 0x7fe8ac033100 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x55ddca9e4580 tx=0x7fe964012740 comp rx=0 tx=0).stop 2023-10-05T19:28:38.082 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.074+0000 7fe96d6f4e40 1 -- 172.21.15.167:0/1697315292 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55ddca9eed20 msgr2=0x55ddca9f4880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-05T19:28:38.082 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.074+0000 7fe96d6f4e40 1 --2- 172.21.15.167:0/1697315292 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55ddca9eed20 0x55ddca9f4880 secure :-1 s=READY pgs=487 cs=0 l=1 rev1=1 crypto rx=0x7fe95403b040 tx=0x7fe954036000 comp rx=0 tx=0).stop 2023-10-05T19:28:38.082 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.074+0000 7fe96d6f4e40 1 -- 172.21.15.167:0/1697315292 shutdown_connections 2023-10-05T19:28:38.082 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.074+0000 7fe96d6f4e40 1 --2- 172.21.15.167:0/1697315292 >> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] conn(0x55ddcaa5be70 0x55ddcaa5e250 unknown :-1 s=CLOSED pgs=83 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-05T19:28:38.082 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.074+0000 7fe96d6f4e40 1 --2- 172.21.15.167:0/1697315292 >> [v2:172.21.15.167:6824/29889,v1:172.21.15.167:6825/29889] conn(0x7fe8ac030c50 0x7fe8ac033100 unknown :-1 s=CLOSED pgs=169 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-05T19:28:38.082 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.074+0000 7fe96d6f4e40 1 --2- 172.21.15.167:0/1697315292 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55ddca9eed20 0x55ddca9f4880 unknown :-1 s=CLOSED pgs=487 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-05T19:28:38.082 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.074+0000 7fe96d6f4e40 1 -- 172.21.15.167:0/1697315292 >> 172.21.15.167:0/1697315292 conn(0x55ddca9ea0e0 msgr2=0x55ddca9e8180 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-05T19:28:38.082 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.074+0000 7fe96d6f4e40 1 -- 172.21.15.167:0/1697315292 shutdown_connections 2023-10-05T19:28:38.082 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.074+0000 7fe96d6f4e40 1 -- 172.21.15.167:0/1697315292 wait complete. 2023-10-05T19:28:38.085 INFO:tasks.util.rgw:rgwadmin: client.0 : ['log', 'show', '--object', '2023-10-05-19-9a9df965-6612-45d4-a45f-6e3c889bacc2.4223.3-myfoo3'] 2023-10-05T19:28:38.086 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'ceph', 'log', 'show', '--object', '2023-10-05-19-9a9df965-6612-45d4-a45f-6e3c889bacc2.4223.3-myfoo3'] 2023-10-05T19:28:38.086 DEBUG:teuthology.orchestra.run.smithi167:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster ceph log show --object 2023-10-05-19-9a9df965-6612-45d4-a45f-6e3c889bacc2.4223.3-myfoo3 2023-10-05T19:28:38.126 INFO:teuthology.orchestra.run.smithi167.stderr:ignoring --setuser ceph since I am not root 2023-10-05T19:28:38.126 INFO:teuthology.orchestra.run.smithi167.stderr:ignoring --setgroup ceph since I am not root 2023-10-05T19:28:38.127 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.122+0000 7f3e58524e40 1 Processor -- start 2023-10-05T19:28:38.127 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.122+0000 7f3e58524e40 1 -- start start 2023-10-05T19:28:38.127 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.122+0000 7f3e58524e40 1 --2- >> v2:172.21.15.118:3300/0 conn(0x55c236d1a2e0 0x55c236d1c6c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-05T19:28:38.127 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.122+0000 7f3e58524e40 1 -- --> v1:172.21.15.118:6789/0 -- auth(proto 0 26 bytes epoch 0) v1 -- 0x55c236c63bd0 con 0x55c236d15d00 2023-10-05T19:28:38.127 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.122+0000 7f3e58524e40 1 -- --> v2:172.21.15.118:3300/0 -- mon_getmap magic: 0 v1 -- 0x55c236c20ee0 con 0x55c236d1a2e0 2023-10-05T19:28:38.128 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.122+0000 7f3e55858700 1 --2- >> v2:172.21.15.118:3300/0 conn(0x55c236d1a2e0 0x55c236d1c6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-05T19:28:38.129 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.122+0000 7f3e55858700 1 --2- >> v2:172.21.15.118:3300/0 conn(0x55c236d1a2e0 0x55c236d1c6c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.118:3300/0 says I am v2:172.21.15.167:42148/0 (socket says 172.21.15.167:42148) 2023-10-05T19:28:38.129 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.122+0000 7f3e56059700 1 --1- >> v1:172.21.15.118:6789/0 conn(0x55c236d15d00 0x55c236d160d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.118:6789/0 says I am v1:172.21.15.167:59390/0 (socket says 172.21.15.167:59390) 2023-10-05T19:28:38.129 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.122+0000 7f3e55858700 1 -- 172.21.15.167:0/80163566 learned_addr learned my addr 172.21.15.167:0/80163566 (peer_addr_for_me v2:172.21.15.167:0/0) 2023-10-05T19:28:38.129 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.122+0000 7f3e55858700 1 -- 172.21.15.167:0/80163566 >> v1:172.21.15.118:6789/0 conn(0x55c236d15d00 legacy=0x55c236d160d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-05T19:28:38.129 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.122+0000 7f3e55858700 1 -- 172.21.15.167:0/80163566 --> v2:172.21.15.118:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55c236d16910 con 0x55c236d1a2e0 2023-10-05T19:28:38.129 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.122+0000 7f3e55858700 1 --2- 172.21.15.167:0/80163566 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55c236d1a2e0 0x55c236d1c6c0 secure :-1 s=READY pgs=489 cs=0 l=1 rev1=1 crypto rx=0x7f3e40005fd0 tx=0x7f3e40005e90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e447f767d06b36b6 server_cookie=0 in_seq=0 out_seq=0 2023-10-05T19:28:38.130 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.122+0000 7f3e55057700 1 -- 172.21.15.167:0/80163566 <== mon.0 v2:172.21.15.118:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f3e4000a960 con 0x55c236d1a2e0 2023-10-05T19:28:38.130 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.122+0000 7f3e55057700 1 -- 172.21.15.167:0/80163566 <== mon.0 v2:172.21.15.118:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3e4000aac0 con 0x55c236d1a2e0 2023-10-05T19:28:38.130 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.122+0000 7f3e55057700 1 -- 172.21.15.167:0/80163566 <== mon.0 v2:172.21.15.118:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f3e4000ac50 con 0x55c236d1a2e0 2023-10-05T19:28:38.130 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.122+0000 7f3e58524e40 1 -- 172.21.15.167:0/80163566 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55c236d1a2e0 msgr2=0x55c236d1c6c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-05T19:28:38.130 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.122+0000 7f3e58524e40 1 --2- 172.21.15.167:0/80163566 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55c236d1a2e0 0x55c236d1c6c0 secure :-1 s=READY pgs=489 cs=0 l=1 rev1=1 crypto rx=0x7f3e40005fd0 tx=0x7f3e40005e90 comp rx=0 tx=0).stop 2023-10-05T19:28:38.131 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.122+0000 7f3e58524e40 1 -- 172.21.15.167:0/80163566 shutdown_connections 2023-10-05T19:28:38.131 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.122+0000 7f3e58524e40 1 --2- 172.21.15.167:0/80163566 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55c236d1a2e0 0x55c236d1c6c0 unknown :-1 s=CLOSED pgs=489 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-05T19:28:38.131 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.122+0000 7f3e58524e40 1 -- 172.21.15.167:0/80163566 >> 172.21.15.167:0/80163566 conn(0x55c236d11720 msgr2=0x55c236d13b30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-05T19:28:38.131 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.122+0000 7f3e58524e40 1 -- 172.21.15.167:0/80163566 shutdown_connections 2023-10-05T19:28:38.131 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.122+0000 7f3e58524e40 1 -- 172.21.15.167:0/80163566 wait complete. 2023-10-05T19:28:38.132 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.126+0000 7f3e58524e40 1 Processor -- start 2023-10-05T19:28:38.132 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.126+0000 7f3e58524e40 1 -- start start 2023-10-05T19:28:38.132 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.126+0000 7f3e58524e40 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55c236d15d00 0x55c236d381d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-05T19:28:38.132 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.126+0000 7f3e58524e40 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c236be4e30 con 0x55c236d15d00 2023-10-05T19:28:38.132 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.126+0000 7f3e56059700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55c236d15d00 0x55c236d381d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-05T19:28:38.133 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.126+0000 7f3e56059700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55c236d15d00 0x55c236d381d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.118:3300/0 says I am v2:172.21.15.167:42150/0 (socket says 172.21.15.167:42150) 2023-10-05T19:28:38.133 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.126+0000 7f3e56059700 1 -- 172.21.15.167:0/3349386415 learned_addr learned my addr 172.21.15.167:0/3349386415 (peer_addr_for_me v2:172.21.15.167:0/0) 2023-10-05T19:28:38.133 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.126+0000 7f3e56059700 1 -- 172.21.15.167:0/3349386415 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3e4000c3d0 con 0x55c236d15d00 2023-10-05T19:28:38.133 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.126+0000 7f3e56059700 1 --2- 172.21.15.167:0/3349386415 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55c236d15d00 0x55c236d381d0 secure :-1 s=READY pgs=490 cs=0 l=1 rev1=1 crypto rx=0x7f3e50001cb0 tx=0x7f3e50001860 comp rx=0 tx=0).ready entity=mon.0 client_cookie=756124ea23fa2a49 server_cookie=0 in_seq=0 out_seq=0 2023-10-05T19:28:38.134 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.126+0000 7f3e4ffff700 1 -- 172.21.15.167:0/3349386415 <== mon.0 v2:172.21.15.118:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f3e50014070 con 0x55c236d15d00 2023-10-05T19:28:38.134 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.126+0000 7f3e4ffff700 1 -- 172.21.15.167:0/3349386415 <== mon.0 v2:172.21.15.118:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3e5000bd40 con 0x55c236d15d00 2023-10-05T19:28:38.134 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.126+0000 7f3e4ffff700 1 -- 172.21.15.167:0/3349386415 <== mon.0 v2:172.21.15.118:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f3e5000f450 con 0x55c236d15d00 2023-10-05T19:28:38.134 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.126+0000 7f3e58524e40 1 -- 172.21.15.167:0/3349386415 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55c236d15d00 msgr2=0x55c236d381d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-05T19:28:38.135 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.126+0000 7f3e58524e40 1 --2- 172.21.15.167:0/3349386415 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55c236d15d00 0x55c236d381d0 secure :-1 s=READY pgs=490 cs=0 l=1 rev1=1 crypto rx=0x7f3e50001cb0 tx=0x7f3e50001860 comp rx=0 tx=0).stop 2023-10-05T19:28:38.135 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.126+0000 7f3e58524e40 1 -- 172.21.15.167:0/3349386415 shutdown_connections 2023-10-05T19:28:38.135 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.126+0000 7f3e58524e40 1 --2- 172.21.15.167:0/3349386415 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55c236d15d00 0x55c236d381d0 unknown :-1 s=CLOSED pgs=490 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-05T19:28:38.135 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.126+0000 7f3e58524e40 1 -- 172.21.15.167:0/3349386415 >> 172.21.15.167:0/3349386415 conn(0x55c236d11720 msgr2=0x55c236d2f980 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-05T19:28:38.135 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.126+0000 7f3e58524e40 1 -- 172.21.15.167:0/3349386415 shutdown_connections 2023-10-05T19:28:38.135 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.126+0000 7f3e58524e40 1 -- 172.21.15.167:0/3349386415 wait complete. 2023-10-05T19:28:38.135 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.130+0000 7f3e58524e40 1 Processor -- start 2023-10-05T19:28:38.135 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.130+0000 7f3e58524e40 1 -- start start 2023-10-05T19:28:38.135 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.130+0000 7f3e58524e40 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55c236d15d00 0x55c236d3efa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-05T19:28:38.135 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.130+0000 7f3e58524e40 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c236bb7ed0 con 0x55c236d15d00 2023-10-05T19:28:38.137 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.130+0000 7f3e56059700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55c236d15d00 0x55c236d3efa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-05T19:28:38.137 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.130+0000 7f3e56059700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55c236d15d00 0x55c236d3efa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.118:3300/0 says I am v2:172.21.15.167:42160/0 (socket says 172.21.15.167:42160) 2023-10-05T19:28:38.137 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.130+0000 7f3e56059700 1 -- 172.21.15.167:0/4284071628 learned_addr learned my addr 172.21.15.167:0/4284071628 (peer_addr_for_me v2:172.21.15.167:0/0) 2023-10-05T19:28:38.137 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.130+0000 7f3e56059700 1 -- 172.21.15.167:0/4284071628 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55c236c63bd0 con 0x55c236d15d00 2023-10-05T19:28:38.137 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.130+0000 7f3e56059700 1 --2- 172.21.15.167:0/4284071628 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55c236d15d00 0x55c236d3efa0 secure :-1 s=READY pgs=491 cs=0 l=1 rev1=1 crypto rx=0x7f3e50000c00 tx=0x7f3e50016000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-05T19:28:38.137 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.130+0000 7f3e4e7fc700 1 -- 172.21.15.167:0/4284071628 <== mon.0 v2:172.21.15.118:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f3e5000f450 con 0x55c236d15d00 2023-10-05T19:28:38.137 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.130+0000 7f3e4e7fc700 1 -- 172.21.15.167:0/4284071628 <== mon.0 v2:172.21.15.118:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3e5000fe80 con 0x55c236d15d00 2023-10-05T19:28:38.137 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.130+0000 7f3e58524e40 1 -- 172.21.15.167:0/4284071628 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x55c236d3f4e0 con 0x55c236d15d00 2023-10-05T19:28:38.138 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.130+0000 7f3e4e7fc700 1 -- 172.21.15.167:0/4284071628 <== mon.0 v2:172.21.15.118:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f3e5001cb00 con 0x55c236d15d00 2023-10-05T19:28:38.138 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.130+0000 7f3e58524e40 1 -- 172.21.15.167:0/4284071628 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55c236d3f920 con 0x55c236d15d00 2023-10-05T19:28:38.138 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.130+0000 7f3e4e7fc700 1 -- 172.21.15.167:0/4284071628 <== mon.0 v2:172.21.15.118:3300/0 4 ==== mgrmap(e 5) v1 ==== 40329+0+0 (secure 0 0 0) 0x7f3e50023020 con 0x55c236d15d00 2023-10-05T19:28:38.138 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.134+0000 7f3e4e7fc700 1 --2- 172.21.15.167:0/4284071628 >> [v2:172.21.15.167:6824/29889,v1:172.21.15.167:6825/29889] conn(0x7f3e38030c00 0x7f3e380330b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-05T19:28:38.138 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.134+0000 7f3e4e7fc700 1 -- 172.21.15.167:0/4284071628 <== mon.0 v2:172.21.15.118:3300/0 5 ==== osd_map(30..30 src has 1..30) v4 ==== 6057+0+0 (secure 0 0 0) 0x7f3e50008030 con 0x55c236d15d00 2023-10-05T19:28:38.139 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.134+0000 7f3e55858700 1 --2- 172.21.15.167:0/4284071628 >> [v2:172.21.15.167:6824/29889,v1:172.21.15.167:6825/29889] conn(0x7f3e38030c00 0x7f3e380330b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-05T19:28:38.139 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.134+0000 7f3e58524e40 20 rados->read ofs=0 len=0 2023-10-05T19:28:38.139 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.134+0000 7f3e58524e40 1 --2- 172.21.15.167:0/4284071628 >> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] conn(0x55c236d1a2e0 0x55c236d4a980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-05T19:28:38.139 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.134+0000 7f3e58524e40 1 -- 172.21.15.167:0/4284071628 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:1 5.12 5:49953fa1:::default.realm:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x55c236d4b0b0 con 0x55c236d1a2e0 2023-10-05T19:28:38.140 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.134+0000 7f3e55858700 1 --2- 172.21.15.167:0/4284071628 >> [v2:172.21.15.167:6824/29889,v1:172.21.15.167:6825/29889] conn(0x7f3e38030c00 0x7f3e380330b0 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7f3e40005ed0 tx=0x7f3e40019000 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-05T19:28:38.140 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.134+0000 7f3e5685a700 1 --2- 172.21.15.167:0/4284071628 >> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] conn(0x55c236d1a2e0 0x55c236d4a980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-05T19:28:38.140 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.134+0000 7f3e5685a700 1 --2- 172.21.15.167:0/4284071628 >> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] conn(0x55c236d1a2e0 0x55c236d4a980 crc :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-05T19:28:38.140 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.134+0000 7f3e5685a700 1 -- 172.21.15.167:0/4284071628 <== osd.1 v2:172.21.15.167:6800/30035 1 ==== osd_op_reply(1 default.realm [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 ==== 157+0+0 (crc 0 0 0) 0x7f3e4800a500 con 0x55c236d1a2e0 2023-10-05T19:28:38.141 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.134+0000 7f3e58524e40 20 rados_obj.operate() r=-2 bl.length=0 2023-10-05T19:28:38.141 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.134+0000 7f3e58524e40 20 realm 2023-10-05T19:28:38.141 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.134+0000 7f3e58524e40 20 rados->read ofs=0 len=0 2023-10-05T19:28:38.141 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.134+0000 7f3e58524e40 1 -- 172.21.15.167:0/4284071628 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:2 5.12 5:49953fa1:::default.realm:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x55c236d47f60 con 0x55c236d1a2e0 2023-10-05T19:28:38.143 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.134+0000 7f3e5685a700 1 -- 172.21.15.167:0/4284071628 <== osd.1 v2:172.21.15.167:6800/30035 2 ==== osd_op_reply(2 default.realm [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 ==== 157+0+0 (crc 0 0 0) 0x7f3e4800a500 con 0x55c236d1a2e0 2023-10-05T19:28:38.143 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.134+0000 7f3e58524e40 20 rados_obj.operate() r=-2 bl.length=0 2023-10-05T19:28:38.143 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.134+0000 7f3e58524e40 4 RGWPeriod::init failed to init realm id : (2) No such file or directory 2023-10-05T19:28:38.143 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.134+0000 7f3e58524e40 20 rados->read ofs=0 len=0 2023-10-05T19:28:38.143 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.134+0000 7f3e58524e40 1 -- 172.21.15.167:0/4284071628 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:3 5.12 5:49953fa1:::default.realm:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x55c236d4bb90 con 0x55c236d1a2e0 2023-10-05T19:28:38.143 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.134+0000 7f3e5685a700 1 -- 172.21.15.167:0/4284071628 <== osd.1 v2:172.21.15.167:6800/30035 3 ==== osd_op_reply(3 default.realm [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 ==== 157+0+0 (crc 0 0 0) 0x7f3e4800a500 con 0x55c236d1a2e0 2023-10-05T19:28:38.143 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.134+0000 7f3e58524e40 20 rados_obj.operate() r=-2 bl.length=0 2023-10-05T19:28:38.143 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.134+0000 7f3e58524e40 20 rados->read ofs=0 len=0 2023-10-05T19:28:38.143 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.134+0000 7f3e58524e40 1 -- 172.21.15.167:0/4284071628 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:4 5.f 5:f43b6ece:::zone_names.default:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x55c236d4b770 con 0x55c236d1a2e0 2023-10-05T19:28:38.144 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.134+0000 7f3e5685a700 1 -- 172.21.15.167:0/4284071628 <== osd.1 v2:172.21.15.167:6800/30035 4 ==== osd_op_reply(4 zone_names.default [read 0~46 out=46b] v0'0 uv1 ondisk = 0) v8 ==== 162+0+46 (crc 0 0 0) 0x7f3e4800a500 con 0x55c236d1a2e0 2023-10-05T19:28:38.144 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.134+0000 7f3e58524e40 20 rados_obj.operate() r=0 bl.length=46 2023-10-05T19:28:38.144 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.134+0000 7f3e58524e40 20 rados->read ofs=0 len=0 2023-10-05T19:28:38.144 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.134+0000 7f3e58524e40 1 -- 172.21.15.167:0/4284071628 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:5 5.1d 5:b8374eb4:::zone_info.9a9df965-6612-45d4-a45f-6e3c889bacc2:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x55c236d4c600 con 0x55c236d1a2e0 2023-10-05T19:28:38.144 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.138+0000 7f3e5685a700 1 -- 172.21.15.167:0/4284071628 <== osd.1 v2:172.21.15.167:6800/30035 5 ==== osd_op_reply(5 zone_info.9a9df965-6612-45d4-a45f-6e3c889bacc2 [read 0~874 out=874b] v0'0 uv2 ondisk = 0) v8 ==== 190+0+874 (crc 0 0 0) 0x7f3e4800a500 con 0x55c236d1a2e0 2023-10-05T19:28:38.144 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.138+0000 7f3e58524e40 20 rados_obj.operate() r=0 bl.length=874 2023-10-05T19:28:38.144 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.138+0000 7f3e58524e40 20 searching for the correct realm 2023-10-05T19:28:38.144 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.138+0000 7f3e58524e40 1 --2- 172.21.15.167:0/4284071628 >> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] conn(0x55c236d4e450 0x55c236d50830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-05T19:28:38.144 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.138+0000 7f3e58524e40 1 -- 172.21.15.167:0/4284071628 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:6 5.0 5:00000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x55c236d50d70 con 0x55c236d4e450 2023-10-05T19:28:38.144 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.138+0000 7f3e56059700 1 --2- 172.21.15.167:0/4284071628 >> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] conn(0x55c236d4e450 0x55c236d50830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-05T19:28:38.145 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.138+0000 7f3e56059700 1 --2- 172.21.15.167:0/4284071628 >> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] conn(0x55c236d4e450 0x55c236d50830 crc :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-05T19:28:38.145 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.138+0000 7f3e56059700 1 -- 172.21.15.167:0/4284071628 <== osd.3 v2:172.21.15.167:6808/30036 1 ==== osd_op_reply(6 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f3e50016070 con 0x55c236d4e450 2023-10-05T19:28:38.145 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.138+0000 7f3e56059700 1 --2- 172.21.15.167:0/4284071628 >> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] conn(0x7f3e50030060 0x7f3e50043130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-05T19:28:38.145 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.138+0000 7f3e56059700 1 -- 172.21.15.167:0/4284071628 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:7 5.10 5:08000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x55c236d50d70 con 0x7f3e50030060 2023-10-05T19:28:38.146 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.138+0000 7f3e55858700 1 --2- 172.21.15.167:0/4284071628 >> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] conn(0x7f3e50030060 0x7f3e50043130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-05T19:28:38.146 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.138+0000 7f3e55858700 1 --2- 172.21.15.167:0/4284071628 >> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] conn(0x7f3e50030060 0x7f3e50043130 crc :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-05T19:28:38.146 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.142+0000 7f3e55858700 1 -- 172.21.15.167:0/4284071628 <== osd.2 v2:172.21.15.167:6816/30037 1 ==== osd_op_reply(7 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f3e4002a040 con 0x7f3e50030060 2023-10-05T19:28:38.146 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.142+0000 7f3e55858700 1 --2- 172.21.15.167:0/4284071628 >> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] conn(0x7f3e40022430 0x7f3e40022860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-05T19:28:38.146 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.142+0000 7f3e55858700 1 -- 172.21.15.167:0/4284071628 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:8 5.8 5:10000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x55c236d50d70 con 0x7f3e40022430 2023-10-05T19:28:38.147 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.142+0000 7f3e5685a700 1 --2- 172.21.15.167:0/4284071628 >> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] conn(0x7f3e40022430 0x7f3e40022860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-05T19:28:38.147 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.142+0000 7f3e5685a700 1 --2- 172.21.15.167:0/4284071628 >> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] conn(0x7f3e40022430 0x7f3e40022860 crc :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-05T19:28:38.148 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.142+0000 7f3e5685a700 1 -- 172.21.15.167:0/4284071628 <== osd.0 v2:172.21.15.118:6800/27690 1 ==== osd_op_reply(8 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f3e48014040 con 0x7f3e40022430 2023-10-05T19:28:38.148 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.142+0000 7f3e5685a700 1 -- 172.21.15.167:0/4284071628 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:9 5.18 5:18000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x55c236d50d70 con 0x55c236d1a2e0 2023-10-05T19:28:38.149 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.142+0000 7f3e5685a700 1 -- 172.21.15.167:0/4284071628 <== osd.1 v2:172.21.15.167:6800/30035 6 ==== osd_op_reply(9 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f3e48014040 con 0x55c236d1a2e0 2023-10-05T19:28:38.149 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.142+0000 7f3e5685a700 1 -- 172.21.15.167:0/4284071628 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:10 5.4 5:20000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x55c236d50d70 con 0x55c236d1a2e0 2023-10-05T19:28:38.150 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.142+0000 7f3e5685a700 1 -- 172.21.15.167:0/4284071628 <== osd.1 v2:172.21.15.167:6800/30035 7 ==== osd_op_reply(10 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f3e48014040 con 0x55c236d1a2e0 2023-10-05T19:28:38.150 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.142+0000 7f3e5685a700 1 -- 172.21.15.167:0/4284071628 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:11 5.14 5:28000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x55c236d50d70 con 0x7f3e40022430 2023-10-05T19:28:38.151 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.146+0000 7f3e5685a700 1 -- 172.21.15.167:0/4284071628 <== osd.0 v2:172.21.15.118:6800/27690 2 ==== osd_op_reply(11 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f3e48014040 con 0x7f3e40022430 2023-10-05T19:28:38.151 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.146+0000 7f3e5685a700 1 -- 172.21.15.167:0/4284071628 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:12 5.c 5:30000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x55c236d50d70 con 0x7f3e40022430 2023-10-05T19:28:38.152 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.146+0000 7f3e5685a700 1 -- 172.21.15.167:0/4284071628 <== osd.0 v2:172.21.15.118:6800/27690 3 ==== osd_op_reply(12 [pgnls start_epoch 30 out=112b] v23'1 uv1 ondisk = 1) v8 ==== 144+0+112 (crc 0 0 0) 0x7f3e48014040 con 0x7f3e40022430 2023-10-05T19:28:38.152 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.146+0000 7f3e5685a700 1 -- 172.21.15.167:0/4284071628 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:13 5.1c 5:38000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x55c236d50d70 con 0x7f3e50030060 2023-10-05T19:28:38.152 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.146+0000 7f3e55858700 1 -- 172.21.15.167:0/4284071628 <== osd.2 v2:172.21.15.167:6816/30037 2 ==== osd_op_reply(13 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f3e4002a040 con 0x7f3e50030060 2023-10-05T19:28:38.152 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.146+0000 7f3e55858700 1 -- 172.21.15.167:0/4284071628 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:14 5.2 5:40000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x55c236d50d70 con 0x55c236d4e450 2023-10-05T19:28:38.153 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.146+0000 7f3e56059700 1 -- 172.21.15.167:0/4284071628 <== osd.3 v2:172.21.15.167:6808/30036 2 ==== osd_op_reply(14 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f3e50016070 con 0x55c236d4e450 2023-10-05T19:28:38.153 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.146+0000 7f3e56059700 1 -- 172.21.15.167:0/4284071628 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:15 5.12 5:48000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x55c236d50d70 con 0x55c236d1a2e0 2023-10-05T19:28:38.154 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.146+0000 7f3e5685a700 1 -- 172.21.15.167:0/4284071628 <== osd.1 v2:172.21.15.167:6800/30035 8 ==== osd_op_reply(15 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f3e48014040 con 0x55c236d1a2e0 2023-10-05T19:28:38.154 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.150+0000 7f3e5685a700 1 -- 172.21.15.167:0/4284071628 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:16 5.a 5:50000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x55c236d50d70 con 0x7f3e40022430 2023-10-05T19:28:38.155 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.150+0000 7f3e5685a700 1 -- 172.21.15.167:0/4284071628 <== osd.0 v2:172.21.15.118:6800/27690 4 ==== osd_op_reply(16 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f3e48022040 con 0x7f3e40022430 2023-10-05T19:28:38.155 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.150+0000 7f3e5685a700 1 -- 172.21.15.167:0/4284071628 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:17 5.1a 5:58000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x55c236d50d70 con 0x55c236d1a2e0 2023-10-05T19:28:38.155 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.150+0000 7f3e5685a700 1 -- 172.21.15.167:0/4284071628 <== osd.1 v2:172.21.15.167:6800/30035 9 ==== osd_op_reply(17 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f3e48022040 con 0x55c236d1a2e0 2023-10-05T19:28:38.155 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.150+0000 7f3e5685a700 1 -- 172.21.15.167:0/4284071628 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:18 5.6 5:60000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x55c236d50d70 con 0x7f3e50030060 2023-10-05T19:28:38.156 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.150+0000 7f3e55858700 1 -- 172.21.15.167:0/4284071628 <== osd.2 v2:172.21.15.167:6816/30037 3 ==== osd_op_reply(18 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f3e4002a040 con 0x7f3e50030060 2023-10-05T19:28:38.156 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.150+0000 7f3e55858700 1 -- 172.21.15.167:0/4284071628 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:19 5.16 5:68000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x55c236d50d70 con 0x55c236d4e450 2023-10-05T19:28:38.157 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.150+0000 7f3e56059700 1 -- 172.21.15.167:0/4284071628 <== osd.3 v2:172.21.15.167:6808/30036 3 ==== osd_op_reply(19 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f3e50016070 con 0x55c236d4e450 2023-10-05T19:28:38.157 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.150+0000 7f3e56059700 1 -- 172.21.15.167:0/4284071628 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:20 5.e 5:70000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x55c236d50d70 con 0x55c236d4e450 2023-10-05T19:28:38.158 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.154+0000 7f3e56059700 1 -- 172.21.15.167:0/4284071628 <== osd.3 v2:172.21.15.167:6808/30036 4 ==== osd_op_reply(20 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f3e50016070 con 0x55c236d4e450 2023-10-05T19:28:38.158 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.154+0000 7f3e56059700 1 -- 172.21.15.167:0/4284071628 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:21 5.1e 5:78000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x55c236d50d70 con 0x7f3e40022430 2023-10-05T19:28:38.159 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.154+0000 7f3e5685a700 1 -- 172.21.15.167:0/4284071628 <== osd.0 v2:172.21.15.118:6800/27690 5 ==== osd_op_reply(21 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f3e48022040 con 0x7f3e40022430 2023-10-05T19:28:38.159 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.154+0000 7f3e5685a700 1 -- 172.21.15.167:0/4284071628 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:22 5.1 5:80000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x55c236d50d70 con 0x55c236d4e450 2023-10-05T19:28:38.160 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.154+0000 7f3e56059700 1 -- 172.21.15.167:0/4284071628 <== osd.3 v2:172.21.15.167:6808/30036 5 ==== osd_op_reply(22 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f3e50016070 con 0x55c236d4e450 2023-10-05T19:28:38.160 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.154+0000 7f3e56059700 1 -- 172.21.15.167:0/4284071628 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:23 5.11 5:88000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x55c236d50d70 con 0x55c236d4e450 2023-10-05T19:28:38.160 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.154+0000 7f3e56059700 1 -- 172.21.15.167:0/4284071628 <== osd.3 v2:172.21.15.167:6808/30036 6 ==== osd_op_reply(23 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f3e50016070 con 0x55c236d4e450 2023-10-05T19:28:38.160 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.154+0000 7f3e56059700 1 -- 172.21.15.167:0/4284071628 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:24 5.9 5:90000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x55c236d50d70 con 0x55c236d1a2e0 2023-10-05T19:28:38.161 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.154+0000 7f3e5685a700 1 -- 172.21.15.167:0/4284071628 <== osd.1 v2:172.21.15.167:6800/30035 10 ==== osd_op_reply(24 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f3e48022040 con 0x55c236d1a2e0 2023-10-05T19:28:38.161 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.154+0000 7f3e5685a700 1 -- 172.21.15.167:0/4284071628 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:25 5.19 5:98000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x55c236d50d70 con 0x55c236d1a2e0 2023-10-05T19:28:38.162 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.154+0000 7f3e5685a700 1 -- 172.21.15.167:0/4284071628 <== osd.1 v2:172.21.15.167:6800/30035 11 ==== osd_op_reply(25 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f3e48022040 con 0x55c236d1a2e0 2023-10-05T19:28:38.162 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.158+0000 7f3e5685a700 1 -- 172.21.15.167:0/4284071628 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:26 5.5 5:a0000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x55c236d50d70 con 0x7f3e40022430 2023-10-05T19:28:38.163 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.158+0000 7f3e5685a700 1 -- 172.21.15.167:0/4284071628 <== osd.0 v2:172.21.15.118:6800/27690 6 ==== osd_op_reply(26 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f3e48022040 con 0x7f3e40022430 2023-10-05T19:28:38.163 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.158+0000 7f3e5685a700 1 -- 172.21.15.167:0/4284071628 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:27 5.15 5:a8000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x55c236d50d70 con 0x55c236d4e450 2023-10-05T19:28:38.163 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.158+0000 7f3e56059700 1 -- 172.21.15.167:0/4284071628 <== osd.3 v2:172.21.15.167:6808/30036 7 ==== osd_op_reply(27 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f3e50016070 con 0x55c236d4e450 2023-10-05T19:28:38.163 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.158+0000 7f3e56059700 1 -- 172.21.15.167:0/4284071628 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:28 5.d 5:b0000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x55c236d50d70 con 0x7f3e50030060 2023-10-05T19:28:38.164 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.158+0000 7f3e55858700 1 -- 172.21.15.167:0/4284071628 <== osd.2 v2:172.21.15.167:6816/30037 4 ==== osd_op_reply(28 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f3e4002a040 con 0x7f3e50030060 2023-10-05T19:28:38.164 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.158+0000 7f3e55858700 1 -- 172.21.15.167:0/4284071628 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:29 5.1d 5:b8000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x55c236d50d70 con 0x55c236d1a2e0 2023-10-05T19:28:38.165 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.158+0000 7f3e5685a700 1 -- 172.21.15.167:0/4284071628 <== osd.1 v2:172.21.15.167:6800/30035 12 ==== osd_op_reply(29 [pgnls start_epoch 30 out=107b] v30'2 uv2 ondisk = 1) v8 ==== 144+0+107 (crc 0 0 0) 0x7f3e48022040 con 0x55c236d1a2e0 2023-10-05T19:28:38.165 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.158+0000 7f3e5685a700 1 -- 172.21.15.167:0/4284071628 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:30 5.3 5:c0000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x55c236d50d70 con 0x7f3e40022430 2023-10-05T19:28:38.166 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.162+0000 7f3e5685a700 1 -- 172.21.15.167:0/4284071628 <== osd.0 v2:172.21.15.118:6800/27690 7 ==== osd_op_reply(30 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f3e48022040 con 0x7f3e40022430 2023-10-05T19:28:38.166 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.162+0000 7f3e5685a700 1 -- 172.21.15.167:0/4284071628 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:31 5.13 5:c8000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x55c236d50d70 con 0x55c236d4e450 2023-10-05T19:28:38.167 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.162+0000 7f3e56059700 1 -- 172.21.15.167:0/4284071628 <== osd.3 v2:172.21.15.167:6808/30036 8 ==== osd_op_reply(31 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f3e50016070 con 0x55c236d4e450 2023-10-05T19:28:38.167 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.162+0000 7f3e56059700 1 -- 172.21.15.167:0/4284071628 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:32 5.b 5:d0000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x55c236d50d70 con 0x7f3e40022430 2023-10-05T19:28:38.168 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.162+0000 7f3e5685a700 1 -- 172.21.15.167:0/4284071628 <== osd.0 v2:172.21.15.118:6800/27690 8 ==== osd_op_reply(32 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f3e48022040 con 0x7f3e40022430 2023-10-05T19:28:38.168 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.162+0000 7f3e5685a700 1 -- 172.21.15.167:0/4284071628 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:33 5.1b 5:d8000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x55c236d50d70 con 0x7f3e50030060 2023-10-05T19:28:38.168 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.162+0000 7f3e55858700 1 -- 172.21.15.167:0/4284071628 <== osd.2 v2:172.21.15.167:6816/30037 5 ==== osd_op_reply(33 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f3e4002a040 con 0x7f3e50030060 2023-10-05T19:28:38.169 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.162+0000 7f3e55858700 1 -- 172.21.15.167:0/4284071628 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:34 5.7 5:e0000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x55c236d50d70 con 0x55c236d4e450 2023-10-05T19:28:38.169 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.162+0000 7f3e56059700 1 -- 172.21.15.167:0/4284071628 <== osd.3 v2:172.21.15.167:6808/30036 9 ==== osd_op_reply(34 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f3e50016070 con 0x55c236d4e450 2023-10-05T19:28:38.169 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.162+0000 7f3e56059700 1 -- 172.21.15.167:0/4284071628 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:35 5.17 5:e8000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x55c236d50d70 con 0x7f3e40022430 2023-10-05T19:28:38.170 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.166+0000 7f3e5685a700 1 -- 172.21.15.167:0/4284071628 <== osd.0 v2:172.21.15.118:6800/27690 9 ==== osd_op_reply(35 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f3e48022040 con 0x7f3e40022430 2023-10-05T19:28:38.170 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.166+0000 7f3e5685a700 1 -- 172.21.15.167:0/4284071628 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:36 5.f 5:f0000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x55c236d50d70 con 0x55c236d1a2e0 2023-10-05T19:28:38.171 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.166+0000 7f3e5685a700 1 -- 172.21.15.167:0/4284071628 <== osd.1 v2:172.21.15.167:6800/30035 13 ==== osd_op_reply(36 [pgnls start_epoch 30 out=115b] v23'2 uv2 ondisk = 1) v8 ==== 144+0+115 (crc 0 0 0) 0x7f3e4802f080 con 0x55c236d1a2e0 2023-10-05T19:28:38.171 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.166+0000 7f3e5685a700 1 -- 172.21.15.167:0/4284071628 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:37 5.1f 5:f8000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x55c236d50d70 con 0x7f3e50030060 2023-10-05T19:28:38.172 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.166+0000 7f3e55858700 1 -- 172.21.15.167:0/4284071628 <== osd.2 v2:172.21.15.167:6816/30037 6 ==== osd_op_reply(37 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f3e4002a040 con 0x7f3e50030060 2023-10-05T19:28:38.172 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.166+0000 7f3e58524e40 20 RGWRados::pool_iterate: got zonegroup_info.4f1f719d-9648-417b-b10c-e5d939825419 2023-10-05T19:28:38.172 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.166+0000 7f3e58524e40 20 RGWRados::pool_iterate: got zone_info.9a9df965-6612-45d4-a45f-6e3c889bacc2 2023-10-05T19:28:38.173 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.166+0000 7f3e58524e40 20 RGWRados::pool_iterate: got zone_names.default 2023-10-05T19:28:38.173 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.166+0000 7f3e58524e40 20 RGWRados::pool_iterate: got zonegroups_names.default 2023-10-05T19:28:38.173 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.166+0000 7f3e58524e40 20 rados->read ofs=0 len=0 2023-10-05T19:28:38.173 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.166+0000 7f3e58524e40 1 -- 172.21.15.167:0/4284071628 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:38 5.12 5:49953fa1:::default.realm:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x55c236d4d200 con 0x55c236d1a2e0 2023-10-05T19:28:38.174 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.166+0000 7f3e5685a700 1 -- 172.21.15.167:0/4284071628 <== osd.1 v2:172.21.15.167:6800/30035 14 ==== osd_op_reply(38 default.realm [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 ==== 157+0+0 (crc 0 0 0) 0x7f3e4802f080 con 0x55c236d1a2e0 2023-10-05T19:28:38.174 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.166+0000 7f3e58524e40 20 rados_obj.operate() r=-2 bl.length=0 2023-10-05T19:28:38.174 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.166+0000 7f3e58524e40 20 rados->read ofs=0 len=0 2023-10-05T19:28:38.174 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.166+0000 7f3e58524e40 1 -- 172.21.15.167:0/4284071628 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:39 5.f 5:f4c53578:::zonegroups_names.default:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x55c236d4db80 con 0x55c236d1a2e0 2023-10-05T19:28:38.174 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.166+0000 7f3e5685a700 1 -- 172.21.15.167:0/4284071628 <== osd.1 v2:172.21.15.167:6800/30035 15 ==== osd_op_reply(39 zonegroups_names.default [read 0~46 out=46b] v0'0 uv2 ondisk = 0) v8 ==== 168+0+46 (crc 0 0 0) 0x7f3e4802f080 con 0x55c236d1a2e0 2023-10-05T19:28:38.174 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.166+0000 7f3e58524e40 20 rados_obj.operate() r=0 bl.length=46 2023-10-05T19:28:38.174 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.166+0000 7f3e58524e40 20 rados->read ofs=0 len=0 2023-10-05T19:28:38.174 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.166+0000 7f3e58524e40 1 -- 172.21.15.167:0/4284071628 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:40 5.c 5:30d299a5:::zonegroup_info.4f1f719d-9648-417b-b10c-e5d939825419:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x55c236d4cdf0 con 0x7f3e40022430 2023-10-05T19:28:38.175 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.170+0000 7f3e5685a700 1 -- 172.21.15.167:0/4284071628 <== osd.0 v2:172.21.15.118:6800/27690 10 ==== osd_op_reply(40 zonegroup_info.4f1f719d-9648-417b-b10c-e5d939825419 [read 0~362 out=362b] v0'0 uv1 ondisk = 0) v8 ==== 195+0+362 (crc 0 0 0) 0x7f3e4802f080 con 0x7f3e40022430 2023-10-05T19:28:38.175 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.170+0000 7f3e58524e40 20 rados_obj.operate() r=0 bl.length=362 2023-10-05T19:28:38.175 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.170+0000 7f3e58524e40 20 rados->read ofs=0 len=0 2023-10-05T19:28:38.175 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.170+0000 7f3e58524e40 1 -- 172.21.15.167:0/4284071628 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:41 5.13 5:cbbf6c60:::converted:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x55c236d51660 con 0x55c236d4e450 2023-10-05T19:28:38.176 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.170+0000 7f3e56059700 1 -- 172.21.15.167:0/4284071628 <== osd.3 v2:172.21.15.167:6808/30036 10 ==== osd_op_reply(41 converted [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 ==== 153+0+0 (crc 0 0 0) 0x7f3e50016070 con 0x55c236d4e450 2023-10-05T19:28:38.176 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.170+0000 7f3e58524e40 20 rados_obj.operate() r=-2 bl.length=0 2023-10-05T19:28:38.176 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.170+0000 7f3e58524e40 20 rados->read ofs=0 len=0 2023-10-05T19:28:38.176 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.170+0000 7f3e58524e40 1 -- 172.21.15.167:0/4284071628 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:42 5.12 5:49953fa1:::default.realm:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x55c236d51a10 con 0x55c236d1a2e0 2023-10-05T19:28:38.177 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.170+0000 7f3e5685a700 1 -- 172.21.15.167:0/4284071628 <== osd.1 v2:172.21.15.167:6800/30035 16 ==== osd_op_reply(42 default.realm [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 ==== 157+0+0 (crc 0 0 0) 0x7f3e4802f080 con 0x55c236d1a2e0 2023-10-05T19:28:38.177 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.170+0000 7f3e58524e40 20 rados_obj.operate() r=-2 bl.length=0 2023-10-05T19:28:38.177 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.170+0000 7f3e58524e40 20 rados->read ofs=0 len=0 2023-10-05T19:28:38.177 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.170+0000 7f3e58524e40 1 -- 172.21.15.167:0/4284071628 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:43 5.f 5:f4c53578:::zonegroups_names.default:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x55c236d51a10 con 0x55c236d1a2e0 2023-10-05T19:28:38.177 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.170+0000 7f3e5685a700 1 -- 172.21.15.167:0/4284071628 <== osd.1 v2:172.21.15.167:6800/30035 17 ==== osd_op_reply(43 zonegroups_names.default [read 0~46 out=46b] v0'0 uv2 ondisk = 0) v8 ==== 168+0+46 (crc 0 0 0) 0x7f3e4802f080 con 0x55c236d1a2e0 2023-10-05T19:28:38.177 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.170+0000 7f3e58524e40 20 rados_obj.operate() r=0 bl.length=46 2023-10-05T19:28:38.178 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.170+0000 7f3e58524e40 1 -- 172.21.15.167:0/4284071628 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:44 5.0 5:00000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x55c236d52a20 con 0x55c236d4e450 2023-10-05T19:28:38.182 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.174+0000 7f3e56059700 1 -- 172.21.15.167:0/4284071628 <== osd.3 v2:172.21.15.167:6808/30036 11 ==== osd_op_reply(44 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f3e50016070 con 0x55c236d4e450 2023-10-05T19:28:38.182 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.174+0000 7f3e56059700 1 -- 172.21.15.167:0/4284071628 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:45 5.10 5:08000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x55c236d52a20 con 0x7f3e50030060 2023-10-05T19:28:38.182 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.174+0000 7f3e55858700 1 -- 172.21.15.167:0/4284071628 <== osd.2 v2:172.21.15.167:6816/30037 7 ==== osd_op_reply(45 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f3e4002a040 con 0x7f3e50030060 2023-10-05T19:28:38.182 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.174+0000 7f3e55858700 1 -- 172.21.15.167:0/4284071628 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:46 5.8 5:10000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x55c236d52a20 con 0x7f3e40022430 2023-10-05T19:28:38.182 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.174+0000 7f3e5685a700 1 -- 172.21.15.167:0/4284071628 <== osd.0 v2:172.21.15.118:6800/27690 11 ==== osd_op_reply(46 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f3e4802f080 con 0x7f3e40022430 2023-10-05T19:28:38.182 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.174+0000 7f3e5685a700 1 -- 172.21.15.167:0/4284071628 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:47 5.18 5:18000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x55c236d4cdf0 con 0x55c236d1a2e0 2023-10-05T19:28:38.183 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.178+0000 7f3e5685a700 1 -- 172.21.15.167:0/4284071628 <== osd.1 v2:172.21.15.167:6800/30035 18 ==== osd_op_reply(47 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f3e4802f080 con 0x55c236d1a2e0 2023-10-05T19:28:38.183 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.178+0000 7f3e5685a700 1 -- 172.21.15.167:0/4284071628 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:48 5.4 5:20000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x55c236d4cdf0 con 0x55c236d1a2e0 2023-10-05T19:28:38.183 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.178+0000 7f3e5685a700 1 -- 172.21.15.167:0/4284071628 <== osd.1 v2:172.21.15.167:6800/30035 19 ==== osd_op_reply(48 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f3e4802f080 con 0x55c236d1a2e0 2023-10-05T19:28:38.183 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.178+0000 7f3e5685a700 1 -- 172.21.15.167:0/4284071628 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:49 5.14 5:28000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x55c236d4cdf0 con 0x7f3e40022430 2023-10-05T19:28:38.184 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.178+0000 7f3e5685a700 1 -- 172.21.15.167:0/4284071628 <== osd.0 v2:172.21.15.118:6800/27690 12 ==== osd_op_reply(49 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f3e4802f080 con 0x7f3e40022430 2023-10-05T19:28:38.184 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.178+0000 7f3e5685a700 1 -- 172.21.15.167:0/4284071628 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:50 5.c 5:30000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x55c236d4cdf0 con 0x7f3e40022430 2023-10-05T19:28:38.185 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.178+0000 7f3e5685a700 1 -- 172.21.15.167:0/4284071628 <== osd.0 v2:172.21.15.118:6800/27690 13 ==== osd_op_reply(50 [pgnls start_epoch 30 out=112b] v23'1 uv1 ondisk = 1) v8 ==== 144+0+112 (crc 0 0 0) 0x7f3e4802f080 con 0x7f3e40022430 2023-10-05T19:28:38.185 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.178+0000 7f3e5685a700 1 -- 172.21.15.167:0/4284071628 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:51 5.1c 5:38000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x55c236d4cdf0 con 0x7f3e50030060 2023-10-05T19:28:38.186 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.182+0000 7f3e55858700 1 -- 172.21.15.167:0/4284071628 <== osd.2 v2:172.21.15.167:6816/30037 8 ==== osd_op_reply(51 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f3e4002a040 con 0x7f3e50030060 2023-10-05T19:28:38.186 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.182+0000 7f3e55858700 1 -- 172.21.15.167:0/4284071628 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:52 5.2 5:40000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x55c236d4cdf0 con 0x55c236d4e450 2023-10-05T19:28:38.187 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.182+0000 7f3e56059700 1 -- 172.21.15.167:0/4284071628 <== osd.3 v2:172.21.15.167:6808/30036 12 ==== osd_op_reply(52 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f3e50016070 con 0x55c236d4e450 2023-10-05T19:28:38.187 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.182+0000 7f3e56059700 1 -- 172.21.15.167:0/4284071628 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:53 5.12 5:48000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x55c236d4cdf0 con 0x55c236d1a2e0 2023-10-05T19:28:38.188 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.182+0000 7f3e5685a700 1 -- 172.21.15.167:0/4284071628 <== osd.1 v2:172.21.15.167:6800/30035 20 ==== osd_op_reply(53 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f3e4802f080 con 0x55c236d1a2e0 2023-10-05T19:28:38.188 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.182+0000 7f3e5685a700 1 -- 172.21.15.167:0/4284071628 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:54 5.a 5:50000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x55c236d4cdf0 con 0x7f3e40022430 2023-10-05T19:28:38.189 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.182+0000 7f3e5685a700 1 -- 172.21.15.167:0/4284071628 <== osd.0 v2:172.21.15.118:6800/27690 14 ==== osd_op_reply(54 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f3e4802f080 con 0x7f3e40022430 2023-10-05T19:28:38.189 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.182+0000 7f3e5685a700 1 -- 172.21.15.167:0/4284071628 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:55 5.1a 5:58000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x55c236d4cdf0 con 0x55c236d1a2e0 2023-10-05T19:28:38.190 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.182+0000 7f3e5685a700 1 -- 172.21.15.167:0/4284071628 <== osd.1 v2:172.21.15.167:6800/30035 21 ==== osd_op_reply(55 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f3e4802f080 con 0x55c236d1a2e0 2023-10-05T19:28:38.190 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.182+0000 7f3e5685a700 1 -- 172.21.15.167:0/4284071628 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:56 5.6 5:60000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x55c236d4cdf0 con 0x7f3e50030060 2023-10-05T19:28:38.190 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.186+0000 7f3e55858700 1 -- 172.21.15.167:0/4284071628 <== osd.2 v2:172.21.15.167:6816/30037 9 ==== osd_op_reply(56 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f3e4002a040 con 0x7f3e50030060 2023-10-05T19:28:38.190 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.186+0000 7f3e55858700 1 -- 172.21.15.167:0/4284071628 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:57 5.16 5:68000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x55c236d4cdf0 con 0x55c236d4e450 2023-10-05T19:28:38.191 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.186+0000 7f3e56059700 1 -- 172.21.15.167:0/4284071628 <== osd.3 v2:172.21.15.167:6808/30036 13 ==== osd_op_reply(57 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f3e50016070 con 0x55c236d4e450 2023-10-05T19:28:38.192 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.186+0000 7f3e56059700 1 -- 172.21.15.167:0/4284071628 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:58 5.e 5:70000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x55c236d4cdf0 con 0x55c236d4e450 2023-10-05T19:28:38.192 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.186+0000 7f3e56059700 1 -- 172.21.15.167:0/4284071628 <== osd.3 v2:172.21.15.167:6808/30036 14 ==== osd_op_reply(58 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f3e50016070 con 0x55c236d4e450 2023-10-05T19:28:38.192 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.186+0000 7f3e56059700 1 -- 172.21.15.167:0/4284071628 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:59 5.1e 5:78000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x55c236d4cdf0 con 0x7f3e40022430 2023-10-05T19:28:38.193 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.186+0000 7f3e5685a700 1 -- 172.21.15.167:0/4284071628 <== osd.0 v2:172.21.15.118:6800/27690 15 ==== osd_op_reply(59 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f3e4802f080 con 0x7f3e40022430 2023-10-05T19:28:38.194 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.186+0000 7f3e5685a700 1 -- 172.21.15.167:0/4284071628 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:60 5.1 5:80000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x55c236d4cdf0 con 0x55c236d4e450 2023-10-05T19:28:38.194 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.190+0000 7f3e56059700 1 -- 172.21.15.167:0/4284071628 <== osd.3 v2:172.21.15.167:6808/30036 15 ==== osd_op_reply(60 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f3e50016070 con 0x55c236d4e450 2023-10-05T19:28:38.194 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.190+0000 7f3e56059700 1 -- 172.21.15.167:0/4284071628 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:61 5.11 5:88000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x55c236d4cdf0 con 0x55c236d4e450 2023-10-05T19:28:38.195 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.190+0000 7f3e56059700 1 -- 172.21.15.167:0/4284071628 <== osd.3 v2:172.21.15.167:6808/30036 16 ==== osd_op_reply(61 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f3e50016070 con 0x55c236d4e450 2023-10-05T19:28:38.195 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.190+0000 7f3e56059700 1 -- 172.21.15.167:0/4284071628 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:62 5.9 5:90000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x55c236d4cdf0 con 0x55c236d1a2e0 2023-10-05T19:28:38.196 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.190+0000 7f3e5685a700 1 -- 172.21.15.167:0/4284071628 <== osd.1 v2:172.21.15.167:6800/30035 22 ==== osd_op_reply(62 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f3e4802f080 con 0x55c236d1a2e0 2023-10-05T19:28:38.196 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.190+0000 7f3e5685a700 1 -- 172.21.15.167:0/4284071628 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:63 5.19 5:98000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x55c236d4cdf0 con 0x55c236d1a2e0 2023-10-05T19:28:38.197 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.190+0000 7f3e5685a700 1 -- 172.21.15.167:0/4284071628 <== osd.1 v2:172.21.15.167:6800/30035 23 ==== osd_op_reply(63 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f3e4802f080 con 0x55c236d1a2e0 2023-10-05T19:28:38.197 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.190+0000 7f3e5685a700 1 -- 172.21.15.167:0/4284071628 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:64 5.5 5:a0000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x55c236d4cdf0 con 0x7f3e40022430 2023-10-05T19:28:38.198 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.194+0000 7f3e5685a700 1 -- 172.21.15.167:0/4284071628 <== osd.0 v2:172.21.15.118:6800/27690 16 ==== osd_op_reply(64 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f3e4802f080 con 0x7f3e40022430 2023-10-05T19:28:38.198 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.194+0000 7f3e5685a700 1 -- 172.21.15.167:0/4284071628 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:65 5.15 5:a8000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x55c236d4cdf0 con 0x55c236d4e450 2023-10-05T19:28:38.199 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.194+0000 7f3e56059700 1 -- 172.21.15.167:0/4284071628 <== osd.3 v2:172.21.15.167:6808/30036 17 ==== osd_op_reply(65 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f3e50016070 con 0x55c236d4e450 2023-10-05T19:28:38.199 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.194+0000 7f3e56059700 1 -- 172.21.15.167:0/4284071628 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:66 5.d 5:b0000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x55c236d4cdf0 con 0x7f3e50030060 2023-10-05T19:28:38.200 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.194+0000 7f3e55858700 1 -- 172.21.15.167:0/4284071628 <== osd.2 v2:172.21.15.167:6816/30037 10 ==== osd_op_reply(66 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f3e4002a040 con 0x7f3e50030060 2023-10-05T19:28:38.200 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.194+0000 7f3e55858700 1 -- 172.21.15.167:0/4284071628 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:67 5.1d 5:b8000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x55c236d4cdf0 con 0x55c236d1a2e0 2023-10-05T19:28:38.201 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.194+0000 7f3e5685a700 1 -- 172.21.15.167:0/4284071628 <== osd.1 v2:172.21.15.167:6800/30035 24 ==== osd_op_reply(67 [pgnls start_epoch 30 out=107b] v30'2 uv2 ondisk = 1) v8 ==== 144+0+107 (crc 0 0 0) 0x7f3e4802f080 con 0x55c236d1a2e0 2023-10-05T19:28:38.201 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.194+0000 7f3e5685a700 1 -- 172.21.15.167:0/4284071628 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:68 5.3 5:c0000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x55c236d4cdf0 con 0x7f3e40022430 2023-10-05T19:28:38.202 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.194+0000 7f3e5685a700 1 -- 172.21.15.167:0/4284071628 <== osd.0 v2:172.21.15.118:6800/27690 17 ==== osd_op_reply(68 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f3e4802f080 con 0x7f3e40022430 2023-10-05T19:28:38.202 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.194+0000 7f3e5685a700 1 -- 172.21.15.167:0/4284071628 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:69 5.13 5:c8000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x55c236d4cdf0 con 0x55c236d4e450 2023-10-05T19:28:38.203 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.198+0000 7f3e56059700 1 -- 172.21.15.167:0/4284071628 <== osd.3 v2:172.21.15.167:6808/30036 18 ==== osd_op_reply(69 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f3e50016070 con 0x55c236d4e450 2023-10-05T19:28:38.203 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.198+0000 7f3e56059700 1 -- 172.21.15.167:0/4284071628 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:70 5.b 5:d0000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x55c236d4cdf0 con 0x7f3e40022430 2023-10-05T19:28:38.204 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.198+0000 7f3e5685a700 1 -- 172.21.15.167:0/4284071628 <== osd.0 v2:172.21.15.118:6800/27690 18 ==== osd_op_reply(70 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f3e4802f080 con 0x7f3e40022430 2023-10-05T19:28:38.204 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.198+0000 7f3e5685a700 1 -- 172.21.15.167:0/4284071628 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:71 5.1b 5:d8000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x55c236d4cdf0 con 0x7f3e50030060 2023-10-05T19:28:38.205 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.198+0000 7f3e55858700 1 -- 172.21.15.167:0/4284071628 <== osd.2 v2:172.21.15.167:6816/30037 11 ==== osd_op_reply(71 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f3e4002a040 con 0x7f3e50030060 2023-10-05T19:28:38.205 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.198+0000 7f3e55858700 1 -- 172.21.15.167:0/4284071628 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:72 5.7 5:e0000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x55c236d4cdf0 con 0x55c236d4e450 2023-10-05T19:28:38.205 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.198+0000 7f3e56059700 1 -- 172.21.15.167:0/4284071628 <== osd.3 v2:172.21.15.167:6808/30036 19 ==== osd_op_reply(72 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f3e50016070 con 0x55c236d4e450 2023-10-05T19:28:38.205 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.198+0000 7f3e56059700 1 -- 172.21.15.167:0/4284071628 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:73 5.17 5:e8000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x55c236d4cdf0 con 0x7f3e40022430 2023-10-05T19:28:38.206 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.202+0000 7f3e5685a700 1 -- 172.21.15.167:0/4284071628 <== osd.0 v2:172.21.15.118:6800/27690 19 ==== osd_op_reply(73 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f3e4802f080 con 0x7f3e40022430 2023-10-05T19:28:38.206 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.202+0000 7f3e5685a700 1 -- 172.21.15.167:0/4284071628 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:74 5.f 5:f0000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x55c236d4cdf0 con 0x55c236d1a2e0 2023-10-05T19:28:38.207 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.202+0000 7f3e5685a700 1 -- 172.21.15.167:0/4284071628 <== osd.1 v2:172.21.15.167:6800/30035 25 ==== osd_op_reply(74 [pgnls start_epoch 30 out=115b] v23'2 uv2 ondisk = 1) v8 ==== 144+0+115 (crc 0 0 0) 0x7f3e4802f080 con 0x55c236d1a2e0 2023-10-05T19:28:38.207 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.202+0000 7f3e5685a700 1 -- 172.21.15.167:0/4284071628 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:75 5.1f 5:f8000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x55c236d4cdf0 con 0x7f3e50030060 2023-10-05T19:28:38.209 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.202+0000 7f3e55858700 1 -- 172.21.15.167:0/4284071628 <== osd.2 v2:172.21.15.167:6816/30037 12 ==== osd_op_reply(75 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f3e4002a040 con 0x7f3e50030060 2023-10-05T19:28:38.209 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.202+0000 7f3e58524e40 20 RGWRados::pool_iterate: got zonegroup_info.4f1f719d-9648-417b-b10c-e5d939825419 2023-10-05T19:28:38.209 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.202+0000 7f3e58524e40 20 RGWRados::pool_iterate: got zone_info.9a9df965-6612-45d4-a45f-6e3c889bacc2 2023-10-05T19:28:38.209 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.202+0000 7f3e58524e40 20 RGWRados::pool_iterate: got zone_names.default 2023-10-05T19:28:38.209 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.202+0000 7f3e58524e40 20 RGWRados::pool_iterate: got zonegroups_names.default 2023-10-05T19:28:38.209 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.202+0000 7f3e58524e40 20 rados->read ofs=0 len=0 2023-10-05T19:28:38.209 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.202+0000 7f3e58524e40 1 -- 172.21.15.167:0/4284071628 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:76 5.f 5:f43b6ece:::zone_names.default:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x55c236d51a10 con 0x55c236d1a2e0 2023-10-05T19:28:38.210 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.202+0000 7f3e5685a700 1 -- 172.21.15.167:0/4284071628 <== osd.1 v2:172.21.15.167:6800/30035 26 ==== osd_op_reply(76 zone_names.default [read 0~46 out=46b] v0'0 uv1 ondisk = 0) v8 ==== 162+0+46 (crc 0 0 0) 0x7f3e4802f080 con 0x55c236d1a2e0 2023-10-05T19:28:38.210 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.202+0000 7f3e58524e40 20 rados_obj.operate() r=0 bl.length=46 2023-10-05T19:28:38.210 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.202+0000 7f3e58524e40 20 rados->read ofs=0 len=0 2023-10-05T19:28:38.210 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.202+0000 7f3e58524e40 1 -- 172.21.15.167:0/4284071628 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:77 5.1d 5:b8374eb4:::zone_info.9a9df965-6612-45d4-a45f-6e3c889bacc2:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x55c236d51dc0 con 0x55c236d1a2e0 2023-10-05T19:28:38.211 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.206+0000 7f3e5685a700 1 -- 172.21.15.167:0/4284071628 <== osd.1 v2:172.21.15.167:6800/30035 27 ==== osd_op_reply(77 zone_info.9a9df965-6612-45d4-a45f-6e3c889bacc2 [read 0~874 out=874b] v0'0 uv2 ondisk = 0) v8 ==== 190+0+874 (crc 0 0 0) 0x7f3e4802f080 con 0x55c236d1a2e0 2023-10-05T19:28:38.211 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.206+0000 7f3e58524e40 20 rados_obj.operate() r=0 bl.length=874 2023-10-05T19:28:38.211 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.206+0000 7f3e58524e40 20 rados->read ofs=0 len=0 2023-10-05T19:28:38.211 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.206+0000 7f3e58524e40 1 -- 172.21.15.167:0/4284071628 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:78 5.f 5:f4c53578:::zonegroups_names.default:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x55c236d522b0 con 0x55c236d1a2e0 2023-10-05T19:28:38.211 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.206+0000 7f3e5685a700 1 -- 172.21.15.167:0/4284071628 <== osd.1 v2:172.21.15.167:6800/30035 28 ==== osd_op_reply(78 zonegroups_names.default [read 0~46 out=46b] v0'0 uv2 ondisk = 0) v8 ==== 168+0+46 (crc 0 0 0) 0x7f3e4802f080 con 0x55c236d1a2e0 2023-10-05T19:28:38.211 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.206+0000 7f3e58524e40 20 rados_obj.operate() r=0 bl.length=46 2023-10-05T19:28:38.212 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.206+0000 7f3e58524e40 20 rados->read ofs=0 len=0 2023-10-05T19:28:38.212 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.206+0000 7f3e58524e40 1 -- 172.21.15.167:0/4284071628 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:79 5.c 5:30d299a5:::zonegroup_info.4f1f719d-9648-417b-b10c-e5d939825419:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x55c236d522b0 con 0x7f3e40022430 2023-10-05T19:28:38.212 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.206+0000 7f3e5685a700 1 -- 172.21.15.167:0/4284071628 <== osd.0 v2:172.21.15.118:6800/27690 20 ==== osd_op_reply(79 zonegroup_info.4f1f719d-9648-417b-b10c-e5d939825419 [read 0~362 out=362b] v0'0 uv1 ondisk = 0) v8 ==== 195+0+362 (crc 0 0 0) 0x7f3e4802f080 con 0x7f3e40022430 2023-10-05T19:28:38.212 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.206+0000 7f3e58524e40 20 rados_obj.operate() r=0 bl.length=362 2023-10-05T19:28:38.212 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.206+0000 7f3e58524e40 20 rados->read ofs=0 len=0 2023-10-05T19:28:38.213 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.206+0000 7f3e58524e40 1 -- 172.21.15.167:0/4284071628 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:80 5.3 5:c0ad8cbd:::region_map:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x55c236d522b0 con 0x7f3e40022430 2023-10-05T19:28:38.213 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.206+0000 7f3e5685a700 1 -- 172.21.15.167:0/4284071628 <== osd.0 v2:172.21.15.118:6800/27690 21 ==== osd_op_reply(80 region_map [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 ==== 154+0+0 (crc 0 0 0) 0x7f3e48050040 con 0x7f3e40022430 2023-10-05T19:28:38.213 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.206+0000 7f3e58524e40 20 rados_obj.operate() r=-2 bl.length=0 2023-10-05T19:28:38.213 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.206+0000 7f3e58524e40 20 zone default found 2023-10-05T19:28:38.213 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.206+0000 7f3e58524e40 4 Realm: () 2023-10-05T19:28:38.213 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.206+0000 7f3e58524e40 4 ZoneGroup: default (4f1f719d-9648-417b-b10c-e5d939825419) 2023-10-05T19:28:38.214 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.206+0000 7f3e58524e40 4 Zone: default (9a9df965-6612-45d4-a45f-6e3c889bacc2) 2023-10-05T19:28:38.214 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.206+0000 7f3e58524e40 10 cannot find current period zonegroup using local zonegroup configuration 2023-10-05T19:28:38.214 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.206+0000 7f3e58524e40 20 zonegroup default 2023-10-05T19:28:38.214 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.206+0000 7f3e58524e40 20 rados->read ofs=0 len=0 2023-10-05T19:28:38.214 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.206+0000 7f3e58524e40 1 -- 172.21.15.167:0/4284071628 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:81 5.3 5:c52100b6:::period_config.default:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x55c236d522b0 con 0x7f3e40022430 2023-10-05T19:28:38.215 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.206+0000 7f3e5685a700 1 -- 172.21.15.167:0/4284071628 <== osd.0 v2:172.21.15.118:6800/27690 22 ==== osd_op_reply(81 period_config.default [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 ==== 165+0+0 (crc 0 0 0) 0x7f3e48050040 con 0x7f3e40022430 2023-10-05T19:28:38.215 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.206+0000 7f3e58524e40 20 rados_obj.operate() r=-2 bl.length=0 2023-10-05T19:28:38.215 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.206+0000 7f3e58524e40 20 rados->read ofs=0 len=0 2023-10-05T19:28:38.215 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.206+0000 7f3e58524e40 1 -- 172.21.15.167:0/4284071628 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:82 6.2 6:4347d321:::bucket.sync-source-hints.:head [call version.read in=11b,read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x55c236d53c30 con 0x55c236d4e450 2023-10-05T19:28:38.215 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.210+0000 7f3e56059700 1 -- 172.21.15.167:0/4284071628 <== osd.3 v2:172.21.15.167:6808/30036 20 ==== osd_op_reply(82 bucket.sync-source-hints. [call,read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 ==== 211+0+0 (crc 0 0 0) 0x7f3e50016070 con 0x55c236d4e450 2023-10-05T19:28:38.215 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.210+0000 7f3e58524e40 20 rados_obj.operate() r=-2 bl.length=0 2023-10-05T19:28:38.215 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.210+0000 7f3e58524e40 20 rados->read ofs=0 len=0 2023-10-05T19:28:38.215 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.210+0000 7f3e58524e40 1 -- 172.21.15.167:0/4284071628 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:83 6.b 6:d467b91b:::bucket.sync-target-hints.:head [call version.read in=11b,read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x55c236d53fe0 con 0x55c236d4e450 2023-10-05T19:28:38.216 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.210+0000 7f3e56059700 1 -- 172.21.15.167:0/4284071628 <== osd.3 v2:172.21.15.167:6808/30036 21 ==== osd_op_reply(83 bucket.sync-target-hints. [call,read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 ==== 211+0+0 (crc 0 0 0) 0x7f3e50016070 con 0x55c236d4e450 2023-10-05T19:28:38.216 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.210+0000 7f3e58524e40 20 rados_obj.operate() r=-2 bl.length=0 2023-10-05T19:28:38.216 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.210+0000 7f3e58524e40 20 started sync module instance, tier type = 2023-10-05T19:28:38.216 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.210+0000 7f3e58524e40 20 started zone id=9a9df965-6612-45d4-a45f-6e3c889bacc2 (name=default) with tier type = 2023-10-05T19:28:38.216 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.210+0000 7f3e58524e40 1 -- 172.21.15.167:0/4284071628 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:84 7.1f 7:f95f44c2:::notify.0:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x55c236d532f0 con 0x7f3e50030060 2023-10-05T19:28:38.218 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.214+0000 7f3e55858700 1 -- 172.21.15.167:0/4284071628 <== osd.2 v2:172.21.15.167:6816/30037 13 ==== osd_op_reply(84 notify.0 [create] v30'177 uv177 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f3e4002a040 con 0x7f3e50030060 2023-10-05T19:28:38.219 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.214+0000 7f3e58524e40 1 -- 172.21.15.167:0/4284071628 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:85 7.1f 7:f95f44c2:::notify.0:head [watch watch cookie 94292631959280] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x55c236d52f20 con 0x7f3e50030060 2023-10-05T19:28:38.219 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.214+0000 7f3e58524e40 1 -- 172.21.15.167:0/4284071628 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:86 7.0 7:05bf5b68:::notify.1:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x55c236d54d70 con 0x55c236d1a2e0 2023-10-05T19:28:38.222 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.214+0000 7f3e5685a700 1 -- 172.21.15.167:0/4284071628 <== osd.1 v2:172.21.15.167:6800/30035 29 ==== osd_op_reply(86 notify.1 [create] v30'177 uv177 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f3e48050040 con 0x55c236d1a2e0 2023-10-05T19:28:38.222 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.214+0000 7f3e58524e40 1 -- 172.21.15.167:0/4284071628 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:87 7.0 7:05bf5b68:::notify.1:head [watch watch cookie 94292631964064] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x55c236d553a0 con 0x55c236d1a2e0 2023-10-05T19:28:38.222 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.214+0000 7f3e55858700 1 -- 172.21.15.167:0/4284071628 <== osd.2 v2:172.21.15.167:6816/30037 14 ==== osd_op_reply(85 notify.0 [watch watch cookie 94292631959280] v30'178 uv177 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f3e4002a040 con 0x7f3e50030060 2023-10-05T19:28:38.222 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.214+0000 7f3e58524e40 1 -- 172.21.15.167:0/4284071628 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:88 7.15 7:a93a5511:::notify.2:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x55c236d55df0 con 0x55c236d4e450 2023-10-05T19:28:38.225 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.218+0000 7f3e56059700 1 -- 172.21.15.167:0/4284071628 <== osd.3 v2:172.21.15.167:6808/30036 22 ==== osd_op_reply(88 notify.2 [create] v30'177 uv177 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f3e50016070 con 0x55c236d4e450 2023-10-05T19:28:38.225 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.218+0000 7f3e58524e40 1 -- 172.21.15.167:0/4284071628 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:89 7.15 7:a93a5511:::notify.2:head [watch watch cookie 94292631962512] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x55c236d561f0 con 0x55c236d4e450 2023-10-05T19:28:38.225 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.218+0000 7f3e58524e40 1 -- 172.21.15.167:0/4284071628 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:90 7.e 7:7759931f:::notify.3:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x55c236d56ad0 con 0x7f3e50030060 2023-10-05T19:28:38.225 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.218+0000 7f3e5685a700 1 -- 172.21.15.167:0/4284071628 <== osd.1 v2:172.21.15.167:6800/30035 30 ==== osd_op_reply(87 notify.1 [watch watch cookie 94292631964064] v30'178 uv177 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f3e48050040 con 0x55c236d1a2e0 2023-10-05T19:28:38.227 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.222+0000 7f3e55858700 1 -- 172.21.15.167:0/4284071628 <== osd.2 v2:172.21.15.167:6816/30037 15 ==== osd_op_reply(90 notify.3 [create] v30'177 uv177 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f3e4002a040 con 0x7f3e50030060 2023-10-05T19:28:38.227 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.222+0000 7f3e58524e40 1 -- 172.21.15.167:0/4284071628 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:91 7.e 7:7759931f:::notify.3:head [watch watch cookie 94292631965232] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x55c236d566c0 con 0x7f3e50030060 2023-10-05T19:28:38.227 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.222+0000 7f3e58524e40 1 -- 172.21.15.167:0/4284071628 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:92 7.d 7:b4812045:::notify.4:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x55c236d57b70 con 0x55c236d1a2e0 2023-10-05T19:28:38.228 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.222+0000 7f3e56059700 1 -- 172.21.15.167:0/4284071628 <== osd.3 v2:172.21.15.167:6808/30036 23 ==== osd_op_reply(89 notify.2 [watch watch cookie 94292631962512] v30'178 uv177 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f3e50016070 con 0x55c236d4e450 2023-10-05T19:28:38.230 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.222+0000 7f3e55858700 1 -- 172.21.15.167:0/4284071628 <== osd.2 v2:172.21.15.167:6816/30037 16 ==== osd_op_reply(91 notify.3 [watch watch cookie 94292631965232] v30'178 uv177 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f3e4002a040 con 0x7f3e50030060 2023-10-05T19:28:38.230 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.226+0000 7f3e5685a700 1 -- 172.21.15.167:0/4284071628 <== osd.1 v2:172.21.15.167:6800/30035 31 ==== osd_op_reply(92 notify.4 [create] v30'177 uv177 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f3e48050040 con 0x55c236d1a2e0 2023-10-05T19:28:38.230 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.226+0000 7f3e58524e40 1 -- 172.21.15.167:0/4284071628 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:93 7.d 7:b4812045:::notify.4:head [watch watch cookie 94292631973584] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x55c236d57640 con 0x55c236d1a2e0 2023-10-05T19:28:38.230 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.226+0000 7f3e58524e40 1 -- 172.21.15.167:0/4284071628 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:94 7.3 7:c609908c:::notify.5:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x55c236d58460 con 0x7f3e40022430 2023-10-05T19:28:38.233 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.226+0000 7f3e5685a700 1 -- 172.21.15.167:0/4284071628 <== osd.0 v2:172.21.15.118:6800/27690 23 ==== osd_op_reply(94 notify.5 [create] v30'177 uv177 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f3e48050040 con 0x7f3e40022430 2023-10-05T19:28:38.233 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.226+0000 7f3e58524e40 1 -- 172.21.15.167:0/4284071628 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:95 7.3 7:c609908c:::notify.5:head [watch watch cookie 94292631977760] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x55c236d58860 con 0x7f3e40022430 2023-10-05T19:28:38.233 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.226+0000 7f3e58524e40 1 -- 172.21.15.167:0/4284071628 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:96 7.14 7:2b04a3e9:::notify.6:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x55c236d59310 con 0x7f3e40022430 2023-10-05T19:28:38.233 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.226+0000 7f3e5685a700 1 -- 172.21.15.167:0/4284071628 <== osd.1 v2:172.21.15.167:6800/30035 32 ==== osd_op_reply(93 notify.4 [watch watch cookie 94292631973584] v30'178 uv177 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f3e48050040 con 0x55c236d1a2e0 2023-10-05T19:28:38.235 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.230+0000 7f3e5685a700 1 -- 172.21.15.167:0/4284071628 <== osd.0 v2:172.21.15.118:6800/27690 24 ==== osd_op_reply(95 notify.5 [watch watch cookie 94292631977760] v30'178 uv177 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f3e48050040 con 0x7f3e40022430 2023-10-05T19:28:38.236 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.230+0000 7f3e5685a700 1 -- 172.21.15.167:0/4284071628 <== osd.0 v2:172.21.15.118:6800/27690 25 ==== osd_op_reply(96 notify.6 [create] v30'177 uv177 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f3e48050040 con 0x7f3e40022430 2023-10-05T19:28:38.236 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.230+0000 7f3e58524e40 1 -- 172.21.15.167:0/4284071628 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:97 7.14 7:2b04a3e9:::notify.6:head [watch watch cookie 94292631967936] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x55c236d58370 con 0x7f3e40022430 2023-10-05T19:28:38.236 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.230+0000 7f3e58524e40 1 -- 172.21.15.167:0/4284071628 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:98 7.9 7:93e5b521:::notify.7:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x55c236d59c00 con 0x7f3e40022430 2023-10-05T19:28:38.238 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.234+0000 7f3e5685a700 1 -- 172.21.15.167:0/4284071628 <== osd.0 v2:172.21.15.118:6800/27690 26 ==== osd_op_reply(98 notify.7 [create] v30'177 uv177 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f3e48050040 con 0x7f3e40022430 2023-10-05T19:28:38.239 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.234+0000 7f3e5685a700 1 -- 172.21.15.167:0/4284071628 <== osd.0 v2:172.21.15.118:6800/27690 27 ==== osd_op_reply(97 notify.6 [watch watch cookie 94292631967936] v30'178 uv177 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f3e48050040 con 0x7f3e40022430 2023-10-05T19:28:38.239 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.234+0000 7f3e58524e40 1 -- 172.21.15.167:0/4284071628 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:99 7.9 7:93e5b521:::notify.7:head [watch watch cookie 94292631981536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x55c236d58370 con 0x7f3e40022430 2023-10-05T19:28:38.239 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.234+0000 7f3e58524e40 20 add_watcher() i=0 2023-10-05T19:28:38.239 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.234+0000 7f3e58524e40 20 add_watcher() i=1 2023-10-05T19:28:38.239 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.234+0000 7f3e58524e40 20 add_watcher() i=2 2023-10-05T19:28:38.239 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.234+0000 7f3e58524e40 20 add_watcher() i=3 2023-10-05T19:28:38.239 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.234+0000 7f3e58524e40 20 add_watcher() i=4 2023-10-05T19:28:38.239 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.234+0000 7f3e58524e40 20 add_watcher() i=5 2023-10-05T19:28:38.239 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.234+0000 7f3e58524e40 20 add_watcher() i=6 2023-10-05T19:28:38.240 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.234+0000 7f3e5685a700 1 -- 172.21.15.167:0/4284071628 <== osd.0 v2:172.21.15.118:6800/27690 28 ==== osd_op_reply(99 notify.7 [watch watch cookie 94292631981536] v30'178 uv177 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f3e48050040 con 0x7f3e40022430 2023-10-05T19:28:38.240 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.234+0000 7f3e58524e40 20 add_watcher() i=7 2023-10-05T19:28:38.240 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.234+0000 7f3e58524e40 2 all 8 watchers are set, enabling cache 2023-10-05T19:28:38.240 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.234+0000 7f3e58524e40 1 -- 172.21.15.167:0/4284071628 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:100 6.4 6:22d26bf9:::data_loggenerations_metadata:head [call version.check_conds in=74b,call version.read in=11b,read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x55c236d5caf0 con 0x55c236d1a2e0 2023-10-05T19:28:38.241 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.234+0000 7f3e5685a700 1 -- 172.21.15.167:0/4284071628 <== osd.1 v2:172.21.15.167:6800/30035 33 ==== osd_op_reply(100 data_loggenerations_metadata [call,call out=48b,read 0~28 out=28b] v0'0 uv1 ondisk = 0) v8 ==== 256+0+76 (crc 0 0 0) 0x7f3e48050040 con 0x55c236d1a2e0 2023-10-05T19:28:38.241 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.234+0000 7f3e58524e40 1 -- 172.21.15.167:0/4284071628 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:101 6.4 6:22d26bf9:::data_loggenerations_metadata:head [watch watch cookie 94292631983424] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x55c236d5a020 con 0x55c236d1a2e0 2023-10-05T19:28:38.243 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.238+0000 7f3e5685a700 1 -- 172.21.15.167:0/4284071628 <== osd.1 v2:172.21.15.167:6800/30035 34 ==== osd_op_reply(101 data_loggenerations_metadata [watch watch cookie 94292631983424] v30'165 uv1 ondisk = 0) v8 ==== 172+0+0 (crc 0 0 0) 0x7f3e48050040 con 0x55c236d1a2e0 2023-10-05T19:28:38.244 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.238+0000 7f3e27fdf700 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2023-10-05T19:28:38.244 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.238+0000 7f3e58524e40 20 check_secure_mon_conn(): auth registy supported: methods=[2] modes=[2,1] 2023-10-05T19:28:38.244 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.238+0000 7f3e58524e40 20 check_secure_mon_conn(): mode 1 is insecure 2023-10-05T19:28:38.244 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.238+0000 7f3e58524e40 1 Processor -- start 2023-10-05T19:28:38.244 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.238+0000 7f3e58524e40 1 -- start start 2023-10-05T19:28:38.245 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.238+0000 7f3e58524e40 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55c236d66e40 0x55c236d67210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-05T19:28:38.245 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.238+0000 7f3e58524e40 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c236d67750 con 0x55c236d66e40 2023-10-05T19:28:38.245 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.238+0000 7f3e55858700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55c236d66e40 0x55c236d67210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-05T19:28:38.245 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.238+0000 7f3e55858700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55c236d66e40 0x55c236d67210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.118:3300/0 says I am v2:172.21.15.167:42170/0 (socket says 172.21.15.167:42170) 2023-10-05T19:28:38.245 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.238+0000 7f3e55858700 1 -- 172.21.15.167:0/3156547542 learned_addr learned my addr 172.21.15.167:0/3156547542 (peer_addr_for_me v2:172.21.15.167:0/0) 2023-10-05T19:28:38.246 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.238+0000 7f3e55858700 1 -- 172.21.15.167:0/3156547542 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55c236d67890 con 0x55c236d66e40 2023-10-05T19:28:38.246 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.242+0000 7f3e55858700 1 --2- 172.21.15.167:0/3156547542 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55c236d66e40 0x55c236d67210 secure :-1 s=READY pgs=492 cs=0 l=1 rev1=1 crypto rx=0x55c236d16d80 tx=0x7f3e40014000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bb68bb24c40c51e server_cookie=0 in_seq=0 out_seq=0 2023-10-05T19:28:38.247 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.242+0000 7f3e277de700 1 -- 172.21.15.167:0/3156547542 <== mon.0 v2:172.21.15.118:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f3e40017580 con 0x55c236d66e40 2023-10-05T19:28:38.247 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.242+0000 7f3e277de700 1 -- 172.21.15.167:0/3156547542 <== mon.0 v2:172.21.15.118:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3e400176e0 con 0x55c236d66e40 2023-10-05T19:28:38.247 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.242+0000 7f3e277de700 1 -- 172.21.15.167:0/3156547542 <== mon.0 v2:172.21.15.118:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f3e400179d0 con 0x55c236d66e40 2023-10-05T19:28:38.248 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.242+0000 7f3e58524e40 1 -- 172.21.15.167:0/3156547542 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55c236d66e40 msgr2=0x55c236d67210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-05T19:28:38.248 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.242+0000 7f3e58524e40 1 --2- 172.21.15.167:0/3156547542 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55c236d66e40 0x55c236d67210 secure :-1 s=READY pgs=492 cs=0 l=1 rev1=1 crypto rx=0x55c236d16d80 tx=0x7f3e40014000 comp rx=0 tx=0).stop 2023-10-05T19:28:38.248 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.242+0000 7f3e58524e40 1 -- 172.21.15.167:0/3156547542 shutdown_connections 2023-10-05T19:28:38.248 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.242+0000 7f3e58524e40 1 --2- 172.21.15.167:0/3156547542 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55c236d66e40 0x55c236d67210 unknown :-1 s=CLOSED pgs=492 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-05T19:28:38.248 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.242+0000 7f3e58524e40 1 -- 172.21.15.167:0/3156547542 >> 172.21.15.167:0/3156547542 conn(0x55c236d62200 msgr2=0x55c236d603b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-05T19:28:38.248 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.242+0000 7f3e58524e40 1 -- 172.21.15.167:0/3156547542 shutdown_connections 2023-10-05T19:28:38.249 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.242+0000 7f3e58524e40 1 -- 172.21.15.167:0/3156547542 wait complete. 2023-10-05T19:28:38.251 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.242+0000 7f3e58524e40 1 Processor -- start 2023-10-05T19:28:38.251 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.242+0000 7f3e58524e40 1 -- start start 2023-10-05T19:28:38.251 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.242+0000 7f3e58524e40 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55c236d66e40 0x55c236d6c8a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-05T19:28:38.251 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.242+0000 7f3e58524e40 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c236d6d8a0 con 0x55c236d66e40 2023-10-05T19:28:38.251 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.242+0000 7f3e55858700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55c236d66e40 0x55c236d6c8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-05T19:28:38.252 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.242+0000 7f3e55858700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55c236d66e40 0x55c236d6c8a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.118:3300/0 says I am v2:172.21.15.167:42172/0 (socket says 172.21.15.167:42172) 2023-10-05T19:28:38.252 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.242+0000 7f3e55858700 1 -- 172.21.15.167:0/3070013241 learned_addr learned my addr 172.21.15.167:0/3070013241 (peer_addr_for_me v2:172.21.15.167:0/0) 2023-10-05T19:28:38.252 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.246+0000 7f3e55858700 1 -- 172.21.15.167:0/3070013241 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55c236d16910 con 0x55c236d66e40 2023-10-05T19:28:38.252 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.246+0000 7f3e55858700 1 --2- 172.21.15.167:0/3070013241 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55c236d66e40 0x55c236d6c8a0 secure :-1 s=READY pgs=493 cs=0 l=1 rev1=1 crypto rx=0x7f3e4000f930 tx=0x7f3e40032030 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-05T19:28:38.252 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.246+0000 7f3e25fdb700 1 -- 172.21.15.167:0/3070013241 <== mon.0 v2:172.21.15.118:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f3e40030690 con 0x55c236d66e40 2023-10-05T19:28:38.252 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.246+0000 7f3e58524e40 1 -- 172.21.15.167:0/3070013241 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x55c236d6cea0 con 0x55c236d66e40 2023-10-05T19:28:38.252 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.246+0000 7f3e25fdb700 1 -- 172.21.15.167:0/3070013241 <== mon.0 v2:172.21.15.118:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3e400307f0 con 0x55c236d66e40 2023-10-05T19:28:38.252 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.246+0000 7f3e25fdb700 1 -- 172.21.15.167:0/3070013241 <== mon.0 v2:172.21.15.118:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f3e40030ae0 con 0x55c236d66e40 2023-10-05T19:28:38.252 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.246+0000 7f3e58524e40 1 -- 172.21.15.167:0/3070013241 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55c236d6d280 con 0x55c236d66e40 2023-10-05T19:28:38.253 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.246+0000 7f3e25fdb700 1 -- 172.21.15.167:0/3070013241 <== mon.0 v2:172.21.15.118:3300/0 4 ==== mgrmap(e 5) v1 ==== 40329+0+0 (secure 0 0 0) 0x7f3e40003040 con 0x55c236d66e40 2023-10-05T19:28:38.253 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.246+0000 7f3e25fdb700 1 --2- 172.21.15.167:0/3070013241 >> [v2:172.21.15.167:6824/29889,v1:172.21.15.167:6825/29889] conn(0x7f3d9c030c00 0x7f3d9c0330b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-05T19:28:38.253 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.246+0000 7f3e25fdb700 1 -- 172.21.15.167:0/3070013241 <== mon.0 v2:172.21.15.118:3300/0 5 ==== osd_map(30..30 src has 1..30) v4 ==== 6057+0+0 (secure 0 0 0) 0x7f3e40048c70 con 0x55c236d66e40 2023-10-05T19:28:38.253 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.246+0000 7f3e5685a700 1 --2- 172.21.15.167:0/3070013241 >> [v2:172.21.15.167:6824/29889,v1:172.21.15.167:6825/29889] conn(0x7f3d9c030c00 0x7f3d9c0330b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-05T19:28:38.253 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.246+0000 7f3e58524e40 5 note: GC not initialized 2023-10-05T19:28:38.254 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.246+0000 7f3e5685a700 1 --2- 172.21.15.167:0/3070013241 >> [v2:172.21.15.167:6824/29889,v1:172.21.15.167:6825/29889] conn(0x7f3d9c030c00 0x7f3d9c0330b0 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x55c236d59a00 tx=0x7f3e48000f90 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-05T19:28:38.255 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.250+0000 7f3d96ffd700 20 reqs_thread_entry: start 2023-10-05T19:28:38.255 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.250+0000 7f3d94ff9700 20 reqs_thread_entry: start 2023-10-05T19:28:38.255 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.250+0000 7f3e58524e40 20 init_complete bucket index max shards: 11 2023-10-05T19:28:38.256 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.250+0000 7f3e58524e40 10 rgw notify: Started notification manager with: 1 workers 2023-10-05T19:28:38.256 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.250+0000 7f3d86ffd700 20 rgw notify: INFO: next queues processing will happen at: Thu Oct 5 19:29:08 2023 2023-10-05T19:28:38.257 INFO:teuthology.orchestra.run.smithi167.stderr: 2023-10-05T19:28:38.257 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.250+0000 7f3e58524e40 10 log_show_next pos 0 bl 0 off 0 eof 0 2023-10-05T19:28:38.257 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.250+0000 7f3d867fc700 20 reqs_thread_entry: start 2023-10-05T19:28:38.257 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.250+0000 7f3e58524e40 1 --2- 172.21.15.167:0/3070013241 >> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] conn(0x55c236dd3dd0 0x55c236dd61b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-05T19:28:38.257 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.250+0000 7f3e58524e40 1 -- 172.21.15.167:0/3070013241 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:1 6.19 6:9cc6e5ad:::2023-10-05-19-9a9df965-6612-45d4-a45f-6e3c889bacc2.4223.3-myfoo3:head [read 0~1048576] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x55c236dd66f0 con 0x55c236dd3dd0 2023-10-05T19:28:38.257 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.250+0000 7f3e56059700 1 --2- 172.21.15.167:0/3070013241 >> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] conn(0x55c236dd3dd0 0x55c236dd61b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-05T19:28:38.257 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.250+0000 7f3e56059700 1 --2- 172.21.15.167:0/3070013241 >> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] conn(0x55c236dd3dd0 0x55c236dd61b0 crc :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-05T19:28:38.258 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.254+0000 7f3e56059700 1 -- 172.21.15.167:0/3070013241 <== osd.1 v2:172.21.15.167:6800/30035 1 ==== osd_op_reply(1 2023-10-05-19-9a9df965-6612-45d4-a45f-6e3c889bacc2.4223.3-myfoo3 [read 0~850 out=850b] v0'0 uv21 ondisk = 0) v8 ==== 208+0+850 (crc 0 0 0) 0x7f3e50016070 con 0x55c236dd3dd0 2023-10-05T19:28:38.258 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.254+0000 7f3e58524e40 10 read 850 2023-10-05T19:28:38.258 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.254+0000 7f3e58524e40 10 log_show_next pos 850 bl 850 off 419 eof 1 2023-10-05T19:28:38.258 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.254+0000 7f3e58524e40 10 log_show_next pos 850 bl 850 off 850 eof 1 2023-10-05T19:28:38.259 INFO:teuthology.orchestra.run.smithi167.stdout:{ 2023-10-05T19:28:38.259 INFO:teuthology.orchestra.run.smithi167.stdout: "bucket_id": "9a9df965-6612-45d4-a45f-6e3c889bacc2.4223.3", 2023-10-05T19:28:38.259 INFO:teuthology.orchestra.run.smithi167.stdout: "bucket_owner": "foo", 2023-10-05T19:28:38.259 INFO:teuthology.orchestra.run.smithi167.stdout: "bucket": "myfoo3", 2023-10-05T19:28:38.259 INFO:teuthology.orchestra.run.smithi167.stdout: "log_entries": [ 2023-10-05T19:28:38.259 INFO:teuthology.orchestra.run.smithi167.stdout: { 2023-10-05T19:28:38.259 INFO:teuthology.orchestra.run.smithi167.stdout: "bucket": "myfoo3", 2023-10-05T19:28:38.259 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2023-10-05T19:28:30.810276Z", 2023-10-05T19:28:38.260 INFO:teuthology.orchestra.run.smithi167.stdout: "time_local": "2023-10-05T19:28:30.810276+0000", 2023-10-05T19:28:38.260 INFO:teuthology.orchestra.run.smithi167.stdout: "remote_addr": "172.21.0.51", 2023-10-05T19:28:38.260 INFO:teuthology.orchestra.run.smithi167.stdout: "user": "foo", 2023-10-05T19:28:38.260 INFO:teuthology.orchestra.run.smithi167.stdout: "operation": "create_bucket", 2023-10-05T19:28:38.260 INFO:teuthology.orchestra.run.smithi167.stdout: "uri": "PUT /myfoo3/ HTTP/1.1", 2023-10-05T19:28:38.260 INFO:teuthology.orchestra.run.smithi167.stdout: "http_status": "200", 2023-10-05T19:28:38.260 INFO:teuthology.orchestra.run.smithi167.stdout: "error_code": "", 2023-10-05T19:28:38.260 INFO:teuthology.orchestra.run.smithi167.stdout: "bytes_sent": 0, 2023-10-05T19:28:38.260 INFO:teuthology.orchestra.run.smithi167.stdout: "bytes_received": 0, 2023-10-05T19:28:38.260 INFO:teuthology.orchestra.run.smithi167.stdout: "object_size": 0, 2023-10-05T19:28:38.260 INFO:teuthology.orchestra.run.smithi167.stdout: "total_time": 11, 2023-10-05T19:28:38.261 INFO:teuthology.orchestra.run.smithi167.stdout: "user_agent": "Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic", 2023-10-05T19:28:38.261 INFO:teuthology.orchestra.run.smithi167.stdout: "referrer": "", 2023-10-05T19:28:38.261 INFO:teuthology.orchestra.run.smithi167.stdout: "trans_id": "tx000002fafc9be3d0d3f21-00651f0e5e-107d-default", 2023-10-05T19:28:38.261 INFO:teuthology.orchestra.run.smithi167.stdout: "authentication_type": "Local", 2023-10-05T19:28:38.261 INFO:teuthology.orchestra.run.smithi167.stdout: "access_key_id": "9te6NH5mcdcq0Tc5i8i1", 2023-10-05T19:28:38.261 INFO:teuthology.orchestra.run.smithi167.stdout: "temp_url": false 2023-10-05T19:28:38.261 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2023-10-05T19:28:38.261 INFO:teuthology.orchestra.run.smithi167.stdout: { 2023-10-05T19:28:38.261 INFO:teuthology.orchestra.run.smithi167.stdout: "bucket": "myfoo3", 2023-10-05T19:28:38.261 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2023-10-05T19:28:30.966273Z", 2023-10-05T19:28:38.261 INFO:teuthology.orchestra.run.smithi167.stdout: "time_local": "2023-10-05T19:28:30.966273+0000", 2023-10-05T19:28:38.262 INFO:teuthology.orchestra.run.smithi167.stdout: "remote_addr": "172.21.0.51", 2023-10-05T19:28:38.262 INFO:teuthology.orchestra.run.smithi167.stdout: "user": "foo", 2023-10-05T19:28:38.262 INFO:teuthology.orchestra.run.smithi167.stdout: "operation": "delete_bucket", 2023-10-05T19:28:38.262 INFO:teuthology.orchestra.run.smithi167.stdout: "uri": "DELETE /myfoo3/ HTTP/1.1", 2023-10-05T19:28:38.262 INFO:teuthology.orchestra.run.smithi167.stdout: "http_status": "204", 2023-10-05T19:28:38.262 INFO:teuthology.orchestra.run.smithi167.stdout: "error_code": "NoContent", 2023-10-05T19:28:38.262 INFO:teuthology.orchestra.run.smithi167.stdout: "bytes_sent": 0, 2023-10-05T19:28:38.262 INFO:teuthology.orchestra.run.smithi167.stdout: "bytes_received": 0, 2023-10-05T19:28:38.262 INFO:teuthology.orchestra.run.smithi167.stdout: "object_size": 0, 2023-10-05T19:28:38.262 INFO:teuthology.orchestra.run.smithi167.stdout: "total_time": 67, 2023-10-05T19:28:38.262 INFO:teuthology.orchestra.run.smithi167.stdout: "user_agent": "Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic", 2023-10-05T19:28:38.262 INFO:teuthology.orchestra.run.smithi167.stdout: "referrer": "", 2023-10-05T19:28:38.263 INFO:teuthology.orchestra.run.smithi167.stdout: "trans_id": "tx0000007aaa9da0c3ff087-00651f0e5e-107d-default", 2023-10-05T19:28:38.263 INFO:teuthology.orchestra.run.smithi167.stdout: "authentication_type": "Local", 2023-10-05T19:28:38.263 INFO:teuthology.orchestra.run.smithi167.stdout: "access_key_id": "9te6NH5mcdcq0Tc5i8i1", 2023-10-05T19:28:38.263 INFO:teuthology.orchestra.run.smithi167.stdout: "temp_url": false 2023-10-05T19:28:38.263 INFO:teuthology.orchestra.run.smithi167.stdout: } 2023-10-05T19:28:38.263 INFO:teuthology.orchestra.run.smithi167.stdout: ], 2023-10-05T19:28:38.263 INFO:teuthology.orchestra.run.smithi167.stdout: "log_sum": { 2023-10-05T19:28:38.263 INFO:teuthology.orchestra.run.smithi167.stdout: "bytes_sent": 0, 2023-10-05T19:28:38.263 INFO:teuthology.orchestra.run.smithi167.stdout: "bytes_received": 0, 2023-10-05T19:28:38.263 INFO:teuthology.orchestra.run.smithi167.stdout: "total_time": 78, 2023-10-05T19:28:38.263 INFO:teuthology.orchestra.run.smithi167.stdout: "total_entries": 2 2023-10-05T19:28:38.263 INFO:teuthology.orchestra.run.smithi167.stdout: } 2023-10-05T19:28:38.264 INFO:teuthology.orchestra.run.smithi167.stdout:} 2023-10-05T19:28:38.264 INFO:teuthology.orchestra.run.smithi167.stdout: 2023-10-05T19:28:38.264 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.254+0000 7f3e58524e40 1 -- 172.21.15.167:0/4284071628 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:102 7.1f 7:f95f44c2:::notify.0:head [watch unwatch cookie 94292631959280] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x55c236d93400 con 0x7f3e50030060 2023-10-05T19:28:38.265 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.254+0000 7f3e55858700 1 -- 172.21.15.167:0/4284071628 <== osd.2 v2:172.21.15.167:6816/30037 17 ==== osd_op_reply(102 notify.0 [watch unwatch cookie 94292631959280] v30'179 uv177 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f3e40042040 con 0x7f3e50030060 2023-10-05T19:28:38.265 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.254+0000 7f3e58524e40 20 remove_watcher() i=0 2023-10-05T19:28:38.265 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.254+0000 7f3e58524e40 2 removed watcher, disabling cache 2023-10-05T19:28:38.265 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.254+0000 7f3e58524e40 1 -- 172.21.15.167:0/4284071628 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:103 7.0 7:05bf5b68:::notify.1:head [watch unwatch cookie 94292631964064] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x55c236d536f0 con 0x55c236d1a2e0 2023-10-05T19:28:38.265 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.258+0000 7f3e5685a700 1 -- 172.21.15.167:0/4284071628 <== osd.1 v2:172.21.15.167:6800/30035 35 ==== osd_op_reply(103 notify.1 [watch unwatch cookie 94292631964064] v30'179 uv177 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f3e48050040 con 0x55c236d1a2e0 2023-10-05T19:28:38.265 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.258+0000 7f3e58524e40 20 remove_watcher() i=1 2023-10-05T19:28:38.265 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.258+0000 7f3e58524e40 1 -- 172.21.15.167:0/4284071628 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:104 7.15 7:a93a5511:::notify.2:head [watch unwatch cookie 94292631962512] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x55c236d536f0 con 0x55c236d4e450 2023-10-05T19:28:38.267 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.262+0000 7f3e56059700 1 -- 172.21.15.167:0/4284071628 <== osd.3 v2:172.21.15.167:6808/30036 24 ==== osd_op_reply(104 notify.2 [watch unwatch cookie 94292631962512] v30'179 uv177 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f3e50016070 con 0x55c236d4e450 2023-10-05T19:28:38.267 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.262+0000 7f3e58524e40 20 remove_watcher() i=2 2023-10-05T19:28:38.267 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.262+0000 7f3e58524e40 1 -- 172.21.15.167:0/4284071628 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:105 7.e 7:7759931f:::notify.3:head [watch unwatch cookie 94292631965232] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x55c236d54390 con 0x7f3e50030060 2023-10-05T19:28:38.269 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.262+0000 7f3e55858700 1 -- 172.21.15.167:0/4284071628 <== osd.2 v2:172.21.15.167:6816/30037 18 ==== osd_op_reply(105 notify.3 [watch unwatch cookie 94292631965232] v30'179 uv177 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f3e40042040 con 0x7f3e50030060 2023-10-05T19:28:38.269 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.262+0000 7f3e58524e40 20 remove_watcher() i=3 2023-10-05T19:28:38.270 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.262+0000 7f3e58524e40 1 -- 172.21.15.167:0/4284071628 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:106 7.d 7:b4812045:::notify.4:head [watch unwatch cookie 94292631973584] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x55c236d54e90 con 0x55c236d1a2e0 2023-10-05T19:28:38.272 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.266+0000 7f3e5685a700 1 -- 172.21.15.167:0/4284071628 <== osd.1 v2:172.21.15.167:6800/30035 36 ==== osd_op_reply(106 notify.4 [watch unwatch cookie 94292631973584] v30'179 uv177 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f3e48050040 con 0x55c236d1a2e0 2023-10-05T19:28:38.272 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.266+0000 7f3e58524e40 20 remove_watcher() i=4 2023-10-05T19:28:38.272 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.266+0000 7f3e58524e40 1 -- 172.21.15.167:0/4284071628 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:107 7.3 7:c609908c:::notify.5:head [watch unwatch cookie 94292631977760] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x55c236d54e90 con 0x7f3e40022430 2023-10-05T19:28:38.275 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.270+0000 7f3e5685a700 1 -- 172.21.15.167:0/4284071628 <== osd.0 v2:172.21.15.118:6800/27690 29 ==== osd_op_reply(107 notify.5 [watch unwatch cookie 94292631977760] v30'179 uv177 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f3e48050040 con 0x7f3e40022430 2023-10-05T19:28:38.275 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.270+0000 7f3e58524e40 20 remove_watcher() i=5 2023-10-05T19:28:38.275 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.270+0000 7f3e58524e40 1 -- 172.21.15.167:0/4284071628 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:108 7.14 7:2b04a3e9:::notify.6:head [watch unwatch cookie 94292631967936] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x55c236d54e90 con 0x7f3e40022430 2023-10-05T19:28:38.278 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.270+0000 7f3e5685a700 1 -- 172.21.15.167:0/4284071628 <== osd.0 v2:172.21.15.118:6800/27690 30 ==== osd_op_reply(108 notify.6 [watch unwatch cookie 94292631967936] v30'179 uv177 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f3e48050040 con 0x7f3e40022430 2023-10-05T19:28:38.278 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.274+0000 7f3e58524e40 20 remove_watcher() i=6 2023-10-05T19:28:38.278 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.274+0000 7f3e58524e40 1 -- 172.21.15.167:0/4284071628 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:109 7.9 7:93e5b521:::notify.7:head [watch unwatch cookie 94292631981536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x55c236d54e90 con 0x7f3e40022430 2023-10-05T19:28:38.281 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.274+0000 7f3e5685a700 1 -- 172.21.15.167:0/4284071628 <== osd.0 v2:172.21.15.118:6800/27690 31 ==== osd_op_reply(109 notify.7 [watch unwatch cookie 94292631981536] v30'179 uv177 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f3e48050040 con 0x7f3e40022430 2023-10-05T19:28:38.281 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.274+0000 7f3e58524e40 20 remove_watcher() i=7 2023-10-05T19:28:38.283 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.278+0000 7f3e27fdf700 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2023-10-05T19:28:38.283 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.278+0000 7f3e58524e40 1 -- 172.21.15.167:0/4284071628 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:110 6.4 6:22d26bf9:::data_loggenerations_metadata:head [watch unwatch cookie 94292631983424] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x55c236d95b80 con 0x55c236d1a2e0 2023-10-05T19:28:38.286 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.278+0000 7f3e5685a700 1 -- 172.21.15.167:0/4284071628 <== osd.1 v2:172.21.15.167:6800/30035 37 ==== osd_op_reply(110 data_loggenerations_metadata [watch unwatch cookie 94292631983424] v30'166 uv1 ondisk = 0) v8 ==== 172+0+0 (crc 0 0 0) 0x7f3e48050040 con 0x55c236d1a2e0 2023-10-05T19:28:38.286 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.278+0000 7f3e58524e40 1 -- 172.21.15.167:0/4284071628 >> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] conn(0x7f3e40022430 msgr2=0x7f3e40022860 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-05T19:28:38.287 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.278+0000 7f3e58524e40 1 --2- 172.21.15.167:0/4284071628 >> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] conn(0x7f3e40022430 0x7f3e40022860 crc :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-05T19:28:38.287 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.278+0000 7f3e58524e40 1 -- 172.21.15.167:0/4284071628 >> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] conn(0x55c236d1a2e0 msgr2=0x55c236d4a980 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-05T19:28:38.287 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.278+0000 7f3e58524e40 1 --2- 172.21.15.167:0/4284071628 >> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] conn(0x55c236d1a2e0 0x55c236d4a980 crc :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-05T19:28:38.287 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.278+0000 7f3e58524e40 1 -- 172.21.15.167:0/4284071628 >> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] conn(0x7f3e50030060 msgr2=0x7f3e50043130 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-05T19:28:38.287 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.278+0000 7f3e58524e40 1 --2- 172.21.15.167:0/4284071628 >> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] conn(0x7f3e50030060 0x7f3e50043130 crc :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-05T19:28:38.287 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.278+0000 7f3e58524e40 1 -- 172.21.15.167:0/4284071628 >> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] conn(0x55c236d4e450 msgr2=0x55c236d50830 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-05T19:28:38.287 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.278+0000 7f3e58524e40 1 --2- 172.21.15.167:0/4284071628 >> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] conn(0x55c236d4e450 0x55c236d50830 crc :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-05T19:28:38.287 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.282+0000 7f3e58524e40 1 -- 172.21.15.167:0/4284071628 >> [v2:172.21.15.167:6824/29889,v1:172.21.15.167:6825/29889] conn(0x7f3e38030c00 msgr2=0x7f3e380330b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-05T19:28:38.287 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.282+0000 7f3e58524e40 1 --2- 172.21.15.167:0/4284071628 >> [v2:172.21.15.167:6824/29889,v1:172.21.15.167:6825/29889] conn(0x7f3e38030c00 0x7f3e380330b0 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7f3e40005ed0 tx=0x7f3e40019000 comp rx=0 tx=0).stop 2023-10-05T19:28:38.288 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.282+0000 7f3e5685a700 1 -- 172.21.15.167:0/4284071628 reap_dead start 2023-10-05T19:28:38.288 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.282+0000 7f3e58524e40 1 -- 172.21.15.167:0/4284071628 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55c236d15d00 msgr2=0x55c236d3efa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-05T19:28:38.288 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.282+0000 7f3e58524e40 1 --2- 172.21.15.167:0/4284071628 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55c236d15d00 0x55c236d3efa0 secure :-1 s=READY pgs=491 cs=0 l=1 rev1=1 crypto rx=0x7f3e50000c00 tx=0x7f3e50016000 comp rx=0 tx=0).stop 2023-10-05T19:28:38.288 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.282+0000 7f3e58524e40 1 -- 172.21.15.167:0/4284071628 shutdown_connections 2023-10-05T19:28:38.288 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.282+0000 7f3e58524e40 1 --2- 172.21.15.167:0/4284071628 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55c236d15d00 0x55c236d3efa0 unknown :-1 s=CLOSED pgs=491 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-05T19:28:38.288 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.282+0000 7f3e58524e40 1 -- 172.21.15.167:0/4284071628 >> 172.21.15.167:0/4284071628 conn(0x55c236d11720 msgr2=0x55c236d2e400 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-05T19:28:38.288 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.282+0000 7f3e58524e40 1 -- 172.21.15.167:0/4284071628 shutdown_connections 2023-10-05T19:28:38.288 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.282+0000 7f3e58524e40 1 -- 172.21.15.167:0/4284071628 wait complete. 2023-10-05T19:28:38.289 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.282+0000 7f3e58524e40 1 -- 172.21.15.167:0/3070013241 >> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] conn(0x55c236dd3dd0 msgr2=0x55c236dd61b0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-05T19:28:38.289 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.282+0000 7f3e58524e40 1 --2- 172.21.15.167:0/3070013241 >> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] conn(0x55c236dd3dd0 0x55c236dd61b0 crc :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-05T19:28:38.289 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.282+0000 7f3e58524e40 1 -- 172.21.15.167:0/3070013241 >> [v2:172.21.15.167:6824/29889,v1:172.21.15.167:6825/29889] conn(0x7f3d9c030c00 msgr2=0x7f3d9c0330b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-05T19:28:38.290 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.282+0000 7f3e58524e40 1 --2- 172.21.15.167:0/3070013241 >> [v2:172.21.15.167:6824/29889,v1:172.21.15.167:6825/29889] conn(0x7f3d9c030c00 0x7f3d9c0330b0 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x55c236d59a00 tx=0x7f3e48000f90 comp rx=0 tx=0).stop 2023-10-05T19:28:38.290 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.282+0000 7f3e58524e40 1 -- 172.21.15.167:0/3070013241 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55c236d66e40 msgr2=0x55c236d6c8a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-05T19:28:38.290 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.282+0000 7f3e58524e40 1 --2- 172.21.15.167:0/3070013241 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55c236d66e40 0x55c236d6c8a0 secure :-1 s=READY pgs=493 cs=0 l=1 rev1=1 crypto rx=0x7f3e4000f930 tx=0x7f3e40032030 comp rx=0 tx=0).stop 2023-10-05T19:28:38.290 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.282+0000 7f3e58524e40 1 -- 172.21.15.167:0/3070013241 shutdown_connections 2023-10-05T19:28:38.290 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.282+0000 7f3e58524e40 1 --2- 172.21.15.167:0/3070013241 >> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] conn(0x55c236dd3dd0 0x55c236dd61b0 unknown :-1 s=CLOSED pgs=81 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-05T19:28:38.290 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.282+0000 7f3e58524e40 1 --2- 172.21.15.167:0/3070013241 >> [v2:172.21.15.167:6824/29889,v1:172.21.15.167:6825/29889] conn(0x7f3d9c030c00 0x7f3d9c0330b0 unknown :-1 s=CLOSED pgs=171 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-05T19:28:38.290 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.282+0000 7f3e58524e40 1 --2- 172.21.15.167:0/3070013241 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55c236d66e40 0x55c236d6c8a0 unknown :-1 s=CLOSED pgs=493 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-05T19:28:38.290 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.282+0000 7f3e58524e40 1 -- 172.21.15.167:0/3070013241 >> 172.21.15.167:0/3070013241 conn(0x55c236d62200 msgr2=0x55c236d60300 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-05T19:28:38.290 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.282+0000 7f3e58524e40 1 -- 172.21.15.167:0/3070013241 shutdown_connections 2023-10-05T19:28:38.290 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.282+0000 7f3e58524e40 1 -- 172.21.15.167:0/3070013241 wait complete. 2023-10-05T19:28:38.293 DEBUG:tasks.util.rgw: json result: {'bucket_id': '9a9df965-6612-45d4-a45f-6e3c889bacc2.4223.3', 'bucket_owner': 'foo', 'bucket': 'myfoo3', 'log_entries': [{'bucket': 'myfoo3', 'time': '2023-10-05T19:28:30.810276Z', 'time_local': '2023-10-05T19:28:30.810276+0000', 'remote_addr': '172.21.0.51', 'user': 'foo', 'operation': 'create_bucket', 'uri': 'PUT /myfoo3/ HTTP/1.1', 'http_status': '200', 'error_code': '', 'bytes_sent': 0, 'bytes_received': 0, 'object_size': 0, 'total_time': 11, 'user_agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'referrer': '', 'trans_id': 'tx000002fafc9be3d0d3f21-00651f0e5e-107d-default', 'authentication_type': 'Local', 'access_key_id': '9te6NH5mcdcq0Tc5i8i1', 'temp_url': False}, {'bucket': 'myfoo3', 'time': '2023-10-05T19:28:30.966273Z', 'time_local': '2023-10-05T19:28:30.966273+0000', 'remote_addr': '172.21.0.51', 'user': 'foo', 'operation': 'delete_bucket', 'uri': 'DELETE /myfoo3/ HTTP/1.1', 'http_status': '204', 'error_code': 'NoContent', 'bytes_sent': 0, 'bytes_received': 0, 'object_size': 0, 'total_time': 67, 'user_agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'referrer': '', 'trans_id': 'tx0000007aaa9da0c3ff087-00651f0e5e-107d-default', 'authentication_type': 'Local', 'access_key_id': '9te6NH5mcdcq0Tc5i8i1', 'temp_url': False}], 'log_sum': {'bytes_sent': 0, 'bytes_received': 0, 'total_time': 78, 'total_entries': 2}} 2023-10-05T19:28:38.293 DEBUG:tasks.radosgw_admin:checking log entry: 2023-10-05T19:28:38.293 DEBUG:tasks.radosgw_admin:checking log entry: 2023-10-05T19:28:38.293 INFO:tasks.util.rgw:rgwadmin: client.0 : ['log', 'rm', '--object', '2023-10-05-19-9a9df965-6612-45d4-a45f-6e3c889bacc2.4223.3-myfoo3'] 2023-10-05T19:28:38.294 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'ceph', 'log', 'rm', '--object', '2023-10-05-19-9a9df965-6612-45d4-a45f-6e3c889bacc2.4223.3-myfoo3'] 2023-10-05T19:28:38.294 DEBUG:teuthology.orchestra.run.smithi167:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster ceph log rm --object 2023-10-05-19-9a9df965-6612-45d4-a45f-6e3c889bacc2.4223.3-myfoo3 2023-10-05T19:28:38.373 INFO:teuthology.orchestra.run.smithi167.stderr:ignoring --setuser ceph since I am not root 2023-10-05T19:28:38.373 INFO:teuthology.orchestra.run.smithi167.stderr:ignoring --setgroup ceph since I am not root 2023-10-05T19:28:38.374 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.366+0000 7f2486d1ce40 1 Processor -- start 2023-10-05T19:28:38.374 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.366+0000 7f2486d1ce40 1 -- start start 2023-10-05T19:28:38.374 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.366+0000 7f2486d1ce40 1 --2- >> v2:172.21.15.118:3300/0 conn(0x562220af6e30 0x562220af7200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-05T19:28:38.374 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.366+0000 7f2486d1ce40 1 -- --> v1:172.21.15.118:6789/0 -- auth(proto 0 26 bytes epoch 0) v1 -- 0x562220a43bd0 con 0x562220af7740 2023-10-05T19:28:38.374 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.370+0000 7f2486d1ce40 1 -- --> v2:172.21.15.118:3300/0 -- mon_getmap magic: 0 v1 -- 0x562220a00ee0 con 0x562220af6e30 2023-10-05T19:28:38.375 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.370+0000 7f2484851700 1 --2- >> v2:172.21.15.118:3300/0 conn(0x562220af6e30 0x562220af7200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-05T19:28:38.375 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.370+0000 7f2484050700 1 --1- >> v1:172.21.15.118:6789/0 conn(0x562220af7740 0x562220af81e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.118:6789/0 says I am v1:172.21.15.167:59396/0 (socket says 172.21.15.167:59396) 2023-10-05T19:28:38.375 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.370+0000 7f2484851700 1 --2- >> v2:172.21.15.118:3300/0 conn(0x562220af6e30 0x562220af7200 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.118:3300/0 says I am v2:172.21.15.167:42186/0 (socket says 172.21.15.167:42186) 2023-10-05T19:28:38.375 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.370+0000 7f2484050700 1 -- 172.21.15.167:0/920209733 learned_addr learned my addr 172.21.15.167:0/920209733 (peer_addr_for_me v1:172.21.15.167:0/0) 2023-10-05T19:28:38.375 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.370+0000 7f2484851700 1 -- 172.21.15.167:0/920209733 >> v1:172.21.15.118:6789/0 conn(0x562220af7740 legacy=0x562220af81e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-05T19:28:38.375 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.370+0000 7f2484851700 1 -- 172.21.15.167:0/920209733 --> v2:172.21.15.118:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x562220af88f0 con 0x562220af6e30 2023-10-05T19:28:38.376 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.370+0000 7f2484851700 1 --2- 172.21.15.167:0/920209733 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x562220af6e30 0x562220af7200 secure :-1 s=READY pgs=495 cs=0 l=1 rev1=1 crypto rx=0x7f247c00a5d0 tx=0x7f247c005a50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5379b80cea5f36fb server_cookie=0 in_seq=0 out_seq=0 2023-10-05T19:28:38.376 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.370+0000 7f248384f700 1 -- 172.21.15.167:0/920209733 <== mon.0 v2:172.21.15.118:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f247c0086a0 con 0x562220af6e30 2023-10-05T19:28:38.376 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.370+0000 7f248384f700 1 -- 172.21.15.167:0/920209733 <== mon.0 v2:172.21.15.118:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f247c008800 con 0x562220af6e30 2023-10-05T19:28:38.376 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.370+0000 7f248384f700 1 -- 172.21.15.167:0/920209733 <== mon.0 v2:172.21.15.118:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f247c008ad0 con 0x562220af6e30 2023-10-05T19:28:38.376 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.370+0000 7f2486d1ce40 1 -- 172.21.15.167:0/920209733 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x562220af6e30 msgr2=0x562220af7200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-05T19:28:38.377 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.370+0000 7f2486d1ce40 1 --2- 172.21.15.167:0/920209733 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x562220af6e30 0x562220af7200 secure :-1 s=READY pgs=495 cs=0 l=1 rev1=1 crypto rx=0x7f247c00a5d0 tx=0x7f247c005a50 comp rx=0 tx=0).stop 2023-10-05T19:28:38.377 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.370+0000 7f2486d1ce40 1 -- 172.21.15.167:0/920209733 shutdown_connections 2023-10-05T19:28:38.378 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.370+0000 7f2486d1ce40 1 --2- 172.21.15.167:0/920209733 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x562220af6e30 0x562220af7200 unknown :-1 s=CLOSED pgs=495 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-05T19:28:38.378 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.370+0000 7f2486d1ce40 1 -- 172.21.15.167:0/920209733 >> 172.21.15.167:0/920209733 conn(0x562220af2850 msgr2=0x562220af4c60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-05T19:28:38.378 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.370+0000 7f2486d1ce40 1 -- 172.21.15.167:0/920209733 shutdown_connections 2023-10-05T19:28:38.378 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.370+0000 7f2486d1ce40 1 -- 172.21.15.167:0/920209733 wait complete. 2023-10-05T19:28:38.378 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.370+0000 7f2486d1ce40 1 Processor -- start 2023-10-05T19:28:38.378 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.370+0000 7f2486d1ce40 1 -- start start 2023-10-05T19:28:38.378 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.370+0000 7f2486d1ce40 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x562220af6e30 0x562220b15ef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-05T19:28:38.378 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.370+0000 7f2486d1ce40 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x5622209c4e30 con 0x562220af6e30 2023-10-05T19:28:38.379 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.374+0000 7f2484851700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x562220af6e30 0x562220b15ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-05T19:28:38.379 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.374+0000 7f2484851700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x562220af6e30 0x562220b15ef0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.118:3300/0 says I am v2:172.21.15.167:42188/0 (socket says 172.21.15.167:42188) 2023-10-05T19:28:38.380 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.374+0000 7f2484851700 1 -- 172.21.15.167:0/2222356060 learned_addr learned my addr 172.21.15.167:0/2222356060 (peer_addr_for_me v2:172.21.15.167:0/0) 2023-10-05T19:28:38.380 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.374+0000 7f2484851700 1 -- 172.21.15.167:0/2222356060 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f247c005cf0 con 0x562220af6e30 2023-10-05T19:28:38.380 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.374+0000 7f2484851700 1 --2- 172.21.15.167:0/2222356060 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x562220af6e30 0x562220b15ef0 secure :-1 s=READY pgs=496 cs=0 l=1 rev1=1 crypto rx=0x7f247c00a520 tx=0x7f247c015000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2475aa5cc0313cef server_cookie=0 in_seq=0 out_seq=0 2023-10-05T19:28:38.380 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.374+0000 7f248284d700 1 -- 172.21.15.167:0/2222356060 <== mon.0 v2:172.21.15.118:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f247c008970 con 0x562220af6e30 2023-10-05T19:28:38.380 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.374+0000 7f248284d700 1 -- 172.21.15.167:0/2222356060 <== mon.0 v2:172.21.15.118:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f247c008ad0 con 0x562220af6e30 2023-10-05T19:28:38.380 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.374+0000 7f248284d700 1 -- 172.21.15.167:0/2222356060 <== mon.0 v2:172.21.15.118:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f247c008dc0 con 0x562220af6e30 2023-10-05T19:28:38.381 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.374+0000 7f2486d1ce40 1 -- 172.21.15.167:0/2222356060 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x562220af6e30 msgr2=0x562220b15ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-05T19:28:38.381 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.374+0000 7f2486d1ce40 1 --2- 172.21.15.167:0/2222356060 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x562220af6e30 0x562220b15ef0 secure :-1 s=READY pgs=496 cs=0 l=1 rev1=1 crypto rx=0x7f247c00a520 tx=0x7f247c015000 comp rx=0 tx=0).stop 2023-10-05T19:28:38.381 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.374+0000 7f2486d1ce40 1 -- 172.21.15.167:0/2222356060 shutdown_connections 2023-10-05T19:28:38.381 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.374+0000 7f2486d1ce40 1 --2- 172.21.15.167:0/2222356060 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x562220af6e30 0x562220b15ef0 unknown :-1 s=CLOSED pgs=496 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-05T19:28:38.381 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.374+0000 7f2486d1ce40 1 -- 172.21.15.167:0/2222356060 >> 172.21.15.167:0/2222356060 conn(0x562220af2850 msgr2=0x562220b11ab0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-05T19:28:38.381 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.374+0000 7f2486d1ce40 1 -- 172.21.15.167:0/2222356060 shutdown_connections 2023-10-05T19:28:38.382 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.374+0000 7f2486d1ce40 1 -- 172.21.15.167:0/2222356060 wait complete. 2023-10-05T19:28:38.382 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.374+0000 7f2486d1ce40 1 Processor -- start 2023-10-05T19:28:38.382 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.378+0000 7f2486d1ce40 1 -- start start 2023-10-05T19:28:38.382 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.378+0000 7f2486d1ce40 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x562220af6e30 0x562220b1cb00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-05T19:28:38.383 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.378+0000 7f2486d1ce40 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x562220997ed0 con 0x562220af6e30 2023-10-05T19:28:38.386 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.378+0000 7f2484851700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x562220af6e30 0x562220b1cb00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-05T19:28:38.386 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.378+0000 7f2484851700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x562220af6e30 0x562220b1cb00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.118:3300/0 says I am v2:172.21.15.167:42196/0 (socket says 172.21.15.167:42196) 2023-10-05T19:28:38.386 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.378+0000 7f2484851700 1 -- 172.21.15.167:0/2512362784 learned_addr learned my addr 172.21.15.167:0/2512362784 (peer_addr_for_me v2:172.21.15.167:0/0) 2023-10-05T19:28:38.387 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.378+0000 7f2484851700 1 -- 172.21.15.167:0/2512362784 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x562220af88f0 con 0x562220af6e30 2023-10-05T19:28:38.387 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.378+0000 7f2484851700 1 --2- 172.21.15.167:0/2512362784 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x562220af6e30 0x562220b1cb00 secure :-1 s=READY pgs=497 cs=0 l=1 rev1=1 crypto rx=0x7f247c005540 tx=0x7f247c003050 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-05T19:28:38.387 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.378+0000 7f248104a700 1 -- 172.21.15.167:0/2512362784 <== mon.0 v2:172.21.15.118:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f247c012bf0 con 0x562220af6e30 2023-10-05T19:28:38.387 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.378+0000 7f248104a700 1 -- 172.21.15.167:0/2512362784 <== mon.0 v2:172.21.15.118:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f247c02a070 con 0x562220af6e30 2023-10-05T19:28:38.387 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.378+0000 7f248104a700 1 -- 172.21.15.167:0/2512362784 <== mon.0 v2:172.21.15.118:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f247c012e90 con 0x562220af6e30 2023-10-05T19:28:38.387 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.378+0000 7f2486d1ce40 1 -- 172.21.15.167:0/2512362784 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x562220b1d040 con 0x562220af6e30 2023-10-05T19:28:38.387 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.378+0000 7f2486d1ce40 1 -- 172.21.15.167:0/2512362784 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x562220b1d4e0 con 0x562220af6e30 2023-10-05T19:28:38.387 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.382+0000 7f248104a700 1 -- 172.21.15.167:0/2512362784 <== mon.0 v2:172.21.15.118:3300/0 4 ==== mgrmap(e 5) v1 ==== 40329+0+0 (secure 0 0 0) 0x7f247c019070 con 0x562220af6e30 2023-10-05T19:28:38.387 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.382+0000 7f248104a700 1 --2- 172.21.15.167:0/2512362784 >> [v2:172.21.15.167:6824/29889,v1:172.21.15.167:6825/29889] conn(0x7f2464030c50 0x7f2464033100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-05T19:28:38.387 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.382+0000 7f248104a700 1 -- 172.21.15.167:0/2512362784 <== mon.0 v2:172.21.15.118:3300/0 5 ==== osd_map(30..30 src has 1..30) v4 ==== 6057+0+0 (secure 0 0 0) 0x7f247c02c070 con 0x562220af6e30 2023-10-05T19:28:38.389 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.382+0000 7f2486d1ce40 20 rados->read ofs=0 len=0 2023-10-05T19:28:38.389 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.382+0000 7f2484050700 1 --2- 172.21.15.167:0/2512362784 >> [v2:172.21.15.167:6824/29889,v1:172.21.15.167:6825/29889] conn(0x7f2464030c50 0x7f2464033100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-05T19:28:38.389 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.382+0000 7f2486d1ce40 1 --2- 172.21.15.167:0/2512362784 >> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] conn(0x562220af7740 0x562220b285d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-05T19:28:38.389 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.382+0000 7f2486d1ce40 1 -- 172.21.15.167:0/2512362784 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:1 5.12 5:49953fa1:::default.realm:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x562220b28c80 con 0x562220af7740 2023-10-05T19:28:38.389 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.382+0000 7f2485052700 1 --2- 172.21.15.167:0/2512362784 >> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] conn(0x562220af7740 0x562220b285d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-05T19:28:38.389 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.382+0000 7f2484050700 1 --2- 172.21.15.167:0/2512362784 >> [v2:172.21.15.167:6824/29889,v1:172.21.15.167:6825/29889] conn(0x7f2464030c50 0x7f2464033100 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7f246c0016f0 tx=0x7f246c0067f0 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-05T19:28:38.389 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.382+0000 7f2485052700 1 --2- 172.21.15.167:0/2512362784 >> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] conn(0x562220af7740 0x562220b285d0 crc :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-05T19:28:38.389 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.382+0000 7f2485052700 1 -- 172.21.15.167:0/2512362784 <== osd.1 v2:172.21.15.167:6800/30035 1 ==== osd_op_reply(1 default.realm [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 ==== 157+0+0 (crc 0 0 0) 0x7f247400a500 con 0x562220af7740 2023-10-05T19:28:38.389 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.382+0000 7f2486d1ce40 20 rados_obj.operate() r=-2 bl.length=0 2023-10-05T19:28:38.389 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.382+0000 7f2486d1ce40 20 realm 2023-10-05T19:28:38.390 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.382+0000 7f2486d1ce40 20 rados->read ofs=0 len=0 2023-10-05T19:28:38.390 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.382+0000 7f2486d1ce40 1 -- 172.21.15.167:0/2512362784 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:2 5.12 5:49953fa1:::default.realm:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x562220b25c30 con 0x562220af7740 2023-10-05T19:28:38.391 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.382+0000 7f2485052700 1 -- 172.21.15.167:0/2512362784 <== osd.1 v2:172.21.15.167:6800/30035 2 ==== osd_op_reply(2 default.realm [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 ==== 157+0+0 (crc 0 0 0) 0x7f247400a500 con 0x562220af7740 2023-10-05T19:28:38.392 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.382+0000 7f2486d1ce40 20 rados_obj.operate() r=-2 bl.length=0 2023-10-05T19:28:38.392 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.382+0000 7f2486d1ce40 4 RGWPeriod::init failed to init realm id : (2) No such file or directory 2023-10-05T19:28:38.392 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.382+0000 7f2486d1ce40 20 rados->read ofs=0 len=0 2023-10-05T19:28:38.392 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.382+0000 7f2486d1ce40 1 -- 172.21.15.167:0/2512362784 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:3 5.12 5:49953fa1:::default.realm:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x562220b29760 con 0x562220af7740 2023-10-05T19:28:38.392 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.382+0000 7f2485052700 1 -- 172.21.15.167:0/2512362784 <== osd.1 v2:172.21.15.167:6800/30035 3 ==== osd_op_reply(3 default.realm [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 ==== 157+0+0 (crc 0 0 0) 0x7f247400a500 con 0x562220af7740 2023-10-05T19:28:38.392 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.382+0000 7f2486d1ce40 20 rados_obj.operate() r=-2 bl.length=0 2023-10-05T19:28:38.392 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.382+0000 7f2486d1ce40 20 rados->read ofs=0 len=0 2023-10-05T19:28:38.392 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.382+0000 7f2486d1ce40 1 -- 172.21.15.167:0/2512362784 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:4 5.f 5:f43b6ece:::zone_names.default:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x562220b29370 con 0x562220af7740 2023-10-05T19:28:38.392 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.386+0000 7f2485052700 1 -- 172.21.15.167:0/2512362784 <== osd.1 v2:172.21.15.167:6800/30035 4 ==== osd_op_reply(4 zone_names.default [read 0~46 out=46b] v0'0 uv1 ondisk = 0) v8 ==== 162+0+46 (crc 0 0 0) 0x7f247400a500 con 0x562220af7740 2023-10-05T19:28:38.392 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.386+0000 7f2486d1ce40 20 rados_obj.operate() r=0 bl.length=46 2023-10-05T19:28:38.392 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.386+0000 7f2486d1ce40 20 rados->read ofs=0 len=0 2023-10-05T19:28:38.393 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.386+0000 7f2486d1ce40 1 -- 172.21.15.167:0/2512362784 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:5 5.1d 5:b8374eb4:::zone_info.9a9df965-6612-45d4-a45f-6e3c889bacc2:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x562220b2a5f0 con 0x562220af7740 2023-10-05T19:28:38.393 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.386+0000 7f2485052700 1 -- 172.21.15.167:0/2512362784 <== osd.1 v2:172.21.15.167:6800/30035 5 ==== osd_op_reply(5 zone_info.9a9df965-6612-45d4-a45f-6e3c889bacc2 [read 0~874 out=874b] v0'0 uv2 ondisk = 0) v8 ==== 190+0+874 (crc 0 0 0) 0x7f247400a500 con 0x562220af7740 2023-10-05T19:28:38.393 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.386+0000 7f2486d1ce40 20 rados_obj.operate() r=0 bl.length=874 2023-10-05T19:28:38.393 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.386+0000 7f2486d1ce40 20 searching for the correct realm 2023-10-05T19:28:38.393 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.386+0000 7f2486d1ce40 1 --2- 172.21.15.167:0/2512362784 >> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] conn(0x562220b2c3b0 0x562220b2e790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-05T19:28:38.393 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.386+0000 7f2486d1ce40 1 -- 172.21.15.167:0/2512362784 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:6 5.0 5:00000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x562220b2ecd0 con 0x562220b2c3b0 2023-10-05T19:28:38.393 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.386+0000 7f2484851700 1 --2- 172.21.15.167:0/2512362784 >> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] conn(0x562220b2c3b0 0x562220b2e790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-05T19:28:38.394 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.386+0000 7f2484851700 1 --2- 172.21.15.167:0/2512362784 >> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] conn(0x562220b2c3b0 0x562220b2e790 crc :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-05T19:28:38.394 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.386+0000 7f2484851700 1 -- 172.21.15.167:0/2512362784 <== osd.3 v2:172.21.15.167:6808/30036 1 ==== osd_op_reply(6 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f247c022020 con 0x562220b2c3b0 2023-10-05T19:28:38.394 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.390+0000 7f2484851700 1 --2- 172.21.15.167:0/2512362784 >> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] conn(0x7f247c007030 0x7f247c01c410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-05T19:28:38.394 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.390+0000 7f2484851700 1 -- 172.21.15.167:0/2512362784 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:7 5.10 5:08000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x562220b2ecd0 con 0x7f247c007030 2023-10-05T19:28:38.394 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.390+0000 7f2484050700 1 --2- 172.21.15.167:0/2512362784 >> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] conn(0x7f247c007030 0x7f247c01c410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-05T19:28:38.395 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.390+0000 7f2484050700 1 --2- 172.21.15.167:0/2512362784 >> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] conn(0x7f247c007030 0x7f247c01c410 crc :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-05T19:28:38.395 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.390+0000 7f2484050700 1 -- 172.21.15.167:0/2512362784 <== osd.2 v2:172.21.15.167:6816/30037 1 ==== osd_op_reply(7 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f246c0060d0 con 0x7f247c007030 2023-10-05T19:28:38.395 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.390+0000 7f2484050700 1 --2- 172.21.15.167:0/2512362784 >> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] conn(0x7f246c011040 0x7f246c014950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-05T19:28:38.395 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.390+0000 7f2484050700 1 -- 172.21.15.167:0/2512362784 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:8 5.8 5:10000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x562220b2ecd0 con 0x7f246c011040 2023-10-05T19:28:38.396 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.390+0000 7f2485052700 1 --2- 172.21.15.167:0/2512362784 >> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] conn(0x7f246c011040 0x7f246c014950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-05T19:28:38.396 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.390+0000 7f2485052700 1 --2- 172.21.15.167:0/2512362784 >> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] conn(0x7f246c011040 0x7f246c014950 crc :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-05T19:28:38.397 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.390+0000 7f2485052700 1 -- 172.21.15.167:0/2512362784 <== osd.0 v2:172.21.15.118:6800/27690 1 ==== osd_op_reply(8 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f2474014040 con 0x7f246c011040 2023-10-05T19:28:38.397 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.390+0000 7f2485052700 1 -- 172.21.15.167:0/2512362784 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:9 5.18 5:18000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x562220b2ecd0 con 0x562220af7740 2023-10-05T19:28:38.398 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.394+0000 7f2485052700 1 -- 172.21.15.167:0/2512362784 <== osd.1 v2:172.21.15.167:6800/30035 6 ==== osd_op_reply(9 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f2474014040 con 0x562220af7740 2023-10-05T19:28:38.398 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.394+0000 7f2485052700 1 -- 172.21.15.167:0/2512362784 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:10 5.4 5:20000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x562220b2ecd0 con 0x562220af7740 2023-10-05T19:28:38.399 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.394+0000 7f2485052700 1 -- 172.21.15.167:0/2512362784 <== osd.1 v2:172.21.15.167:6800/30035 7 ==== osd_op_reply(10 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f2474014040 con 0x562220af7740 2023-10-05T19:28:38.399 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.394+0000 7f2485052700 1 -- 172.21.15.167:0/2512362784 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:11 5.14 5:28000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x562220b2ecd0 con 0x7f246c011040 2023-10-05T19:28:38.400 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.394+0000 7f2485052700 1 -- 172.21.15.167:0/2512362784 <== osd.0 v2:172.21.15.118:6800/27690 2 ==== osd_op_reply(11 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f2474014040 con 0x7f246c011040 2023-10-05T19:28:38.400 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.394+0000 7f2485052700 1 -- 172.21.15.167:0/2512362784 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:12 5.c 5:30000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x562220b2ecd0 con 0x7f246c011040 2023-10-05T19:28:38.401 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.394+0000 7f2485052700 1 -- 172.21.15.167:0/2512362784 <== osd.0 v2:172.21.15.118:6800/27690 3 ==== osd_op_reply(12 [pgnls start_epoch 30 out=112b] v23'1 uv1 ondisk = 1) v8 ==== 144+0+112 (crc 0 0 0) 0x7f2474014040 con 0x7f246c011040 2023-10-05T19:28:38.401 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.394+0000 7f2485052700 1 -- 172.21.15.167:0/2512362784 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:13 5.1c 5:38000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x562220b2ecd0 con 0x7f247c007030 2023-10-05T19:28:38.401 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.394+0000 7f2484050700 1 -- 172.21.15.167:0/2512362784 <== osd.2 v2:172.21.15.167:6816/30037 2 ==== osd_op_reply(13 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f246c004bd0 con 0x7f247c007030 2023-10-05T19:28:38.401 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.394+0000 7f2484050700 1 -- 172.21.15.167:0/2512362784 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:14 5.2 5:40000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x562220b2ecd0 con 0x562220b2c3b0 2023-10-05T19:28:38.402 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.398+0000 7f2484851700 1 -- 172.21.15.167:0/2512362784 <== osd.3 v2:172.21.15.167:6808/30036 2 ==== osd_op_reply(14 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f247c022020 con 0x562220b2c3b0 2023-10-05T19:28:38.402 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.398+0000 7f2484851700 1 -- 172.21.15.167:0/2512362784 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:15 5.12 5:48000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x562220b2ecd0 con 0x562220af7740 2023-10-05T19:28:38.403 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.398+0000 7f2485052700 1 -- 172.21.15.167:0/2512362784 <== osd.1 v2:172.21.15.167:6800/30035 8 ==== osd_op_reply(15 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f2474014040 con 0x562220af7740 2023-10-05T19:28:38.403 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.398+0000 7f2485052700 1 -- 172.21.15.167:0/2512362784 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:16 5.a 5:50000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x562220b2ecd0 con 0x7f246c011040 2023-10-05T19:28:38.404 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.398+0000 7f2485052700 1 -- 172.21.15.167:0/2512362784 <== osd.0 v2:172.21.15.118:6800/27690 4 ==== osd_op_reply(16 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f2474022040 con 0x7f246c011040 2023-10-05T19:28:38.404 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.398+0000 7f2485052700 1 -- 172.21.15.167:0/2512362784 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:17 5.1a 5:58000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x562220b2ecd0 con 0x562220af7740 2023-10-05T19:28:38.404 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.398+0000 7f2485052700 1 -- 172.21.15.167:0/2512362784 <== osd.1 v2:172.21.15.167:6800/30035 9 ==== osd_op_reply(17 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f2474022040 con 0x562220af7740 2023-10-05T19:28:38.405 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.398+0000 7f2485052700 1 -- 172.21.15.167:0/2512362784 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:18 5.6 5:60000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x562220b2ecd0 con 0x7f247c007030 2023-10-05T19:28:38.405 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.398+0000 7f2484050700 1 -- 172.21.15.167:0/2512362784 <== osd.2 v2:172.21.15.167:6816/30037 3 ==== osd_op_reply(18 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f246c004bd0 con 0x7f247c007030 2023-10-05T19:28:38.405 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.398+0000 7f2484050700 1 -- 172.21.15.167:0/2512362784 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:19 5.16 5:68000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x562220b2ecd0 con 0x562220b2c3b0 2023-10-05T19:28:38.406 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.402+0000 7f2484851700 1 -- 172.21.15.167:0/2512362784 <== osd.3 v2:172.21.15.167:6808/30036 3 ==== osd_op_reply(19 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f247c022020 con 0x562220b2c3b0 2023-10-05T19:28:38.406 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.402+0000 7f2484851700 1 -- 172.21.15.167:0/2512362784 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:20 5.e 5:70000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x562220b2ecd0 con 0x562220b2c3b0 2023-10-05T19:28:38.407 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.402+0000 7f2484851700 1 -- 172.21.15.167:0/2512362784 <== osd.3 v2:172.21.15.167:6808/30036 4 ==== osd_op_reply(20 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f247c022020 con 0x562220b2c3b0 2023-10-05T19:28:38.407 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.402+0000 7f2484851700 1 -- 172.21.15.167:0/2512362784 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:21 5.1e 5:78000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x562220b2ecd0 con 0x7f246c011040 2023-10-05T19:28:38.408 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.402+0000 7f2485052700 1 -- 172.21.15.167:0/2512362784 <== osd.0 v2:172.21.15.118:6800/27690 5 ==== osd_op_reply(21 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f2474022040 con 0x7f246c011040 2023-10-05T19:28:38.408 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.402+0000 7f2485052700 1 -- 172.21.15.167:0/2512362784 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:22 5.1 5:80000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x562220b2ecd0 con 0x562220b2c3b0 2023-10-05T19:28:38.409 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.402+0000 7f2484851700 1 -- 172.21.15.167:0/2512362784 <== osd.3 v2:172.21.15.167:6808/30036 5 ==== osd_op_reply(22 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f247c022020 con 0x562220b2c3b0 2023-10-05T19:28:38.409 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.402+0000 7f2484851700 1 -- 172.21.15.167:0/2512362784 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:23 5.11 5:88000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x562220b2ecd0 con 0x562220b2c3b0 2023-10-05T19:28:38.410 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.402+0000 7f2484851700 1 -- 172.21.15.167:0/2512362784 <== osd.3 v2:172.21.15.167:6808/30036 6 ==== osd_op_reply(23 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f247c022020 con 0x562220b2c3b0 2023-10-05T19:28:38.410 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.402+0000 7f2484851700 1 -- 172.21.15.167:0/2512362784 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:24 5.9 5:90000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x562220b2ecd0 con 0x562220af7740 2023-10-05T19:28:38.411 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.406+0000 7f2485052700 1 -- 172.21.15.167:0/2512362784 <== osd.1 v2:172.21.15.167:6800/30035 10 ==== osd_op_reply(24 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f2474022040 con 0x562220af7740 2023-10-05T19:28:38.411 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.406+0000 7f2485052700 1 -- 172.21.15.167:0/2512362784 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:25 5.19 5:98000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x562220b2ecd0 con 0x562220af7740 2023-10-05T19:28:38.411 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.406+0000 7f2485052700 1 -- 172.21.15.167:0/2512362784 <== osd.1 v2:172.21.15.167:6800/30035 11 ==== osd_op_reply(25 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f2474022040 con 0x562220af7740 2023-10-05T19:28:38.411 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.406+0000 7f2485052700 1 -- 172.21.15.167:0/2512362784 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:26 5.5 5:a0000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x562220b2ecd0 con 0x7f246c011040 2023-10-05T19:28:38.413 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.406+0000 7f2485052700 1 -- 172.21.15.167:0/2512362784 <== osd.0 v2:172.21.15.118:6800/27690 6 ==== osd_op_reply(26 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f2474022040 con 0x7f246c011040 2023-10-05T19:28:38.413 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.406+0000 7f2485052700 1 -- 172.21.15.167:0/2512362784 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:27 5.15 5:a8000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x562220b2ecd0 con 0x562220b2c3b0 2023-10-05T19:28:38.413 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.406+0000 7f2484851700 1 -- 172.21.15.167:0/2512362784 <== osd.3 v2:172.21.15.167:6808/30036 7 ==== osd_op_reply(27 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f247c022020 con 0x562220b2c3b0 2023-10-05T19:28:38.413 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.406+0000 7f2484851700 1 -- 172.21.15.167:0/2512362784 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:28 5.d 5:b0000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x562220b2ecd0 con 0x7f247c007030 2023-10-05T19:28:38.414 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.410+0000 7f2484050700 1 -- 172.21.15.167:0/2512362784 <== osd.2 v2:172.21.15.167:6816/30037 4 ==== osd_op_reply(28 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f246c004bd0 con 0x7f247c007030 2023-10-05T19:28:38.414 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.410+0000 7f2484050700 1 -- 172.21.15.167:0/2512362784 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:29 5.1d 5:b8000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x562220b2ecd0 con 0x562220af7740 2023-10-05T19:28:38.415 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.410+0000 7f2485052700 1 -- 172.21.15.167:0/2512362784 <== osd.1 v2:172.21.15.167:6800/30035 12 ==== osd_op_reply(29 [pgnls start_epoch 30 out=107b] v30'2 uv2 ondisk = 1) v8 ==== 144+0+107 (crc 0 0 0) 0x7f2474022040 con 0x562220af7740 2023-10-05T19:28:38.415 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.410+0000 7f2485052700 1 -- 172.21.15.167:0/2512362784 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:30 5.3 5:c0000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x562220b2ecd0 con 0x7f246c011040 2023-10-05T19:28:38.416 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.410+0000 7f2485052700 1 -- 172.21.15.167:0/2512362784 <== osd.0 v2:172.21.15.118:6800/27690 7 ==== osd_op_reply(30 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f2474022040 con 0x7f246c011040 2023-10-05T19:28:38.416 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.410+0000 7f2485052700 1 -- 172.21.15.167:0/2512362784 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:31 5.13 5:c8000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x562220b2ecd0 con 0x562220b2c3b0 2023-10-05T19:28:38.417 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.410+0000 7f2484851700 1 -- 172.21.15.167:0/2512362784 <== osd.3 v2:172.21.15.167:6808/30036 8 ==== osd_op_reply(31 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f247c051040 con 0x562220b2c3b0 2023-10-05T19:28:38.417 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.410+0000 7f2484851700 1 -- 172.21.15.167:0/2512362784 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:32 5.b 5:d0000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x562220b2ecd0 con 0x7f246c011040 2023-10-05T19:28:38.418 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.414+0000 7f2485052700 1 -- 172.21.15.167:0/2512362784 <== osd.0 v2:172.21.15.118:6800/27690 8 ==== osd_op_reply(32 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f2474022040 con 0x7f246c011040 2023-10-05T19:28:38.418 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.414+0000 7f2485052700 1 -- 172.21.15.167:0/2512362784 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:33 5.1b 5:d8000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x562220b2ecd0 con 0x7f247c007030 2023-10-05T19:28:38.419 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.414+0000 7f2484050700 1 -- 172.21.15.167:0/2512362784 <== osd.2 v2:172.21.15.167:6816/30037 5 ==== osd_op_reply(33 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f246c004bd0 con 0x7f247c007030 2023-10-05T19:28:38.419 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.414+0000 7f2484050700 1 -- 172.21.15.167:0/2512362784 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:34 5.7 5:e0000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x562220b2ecd0 con 0x562220b2c3b0 2023-10-05T19:28:38.420 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.414+0000 7f2484851700 1 -- 172.21.15.167:0/2512362784 <== osd.3 v2:172.21.15.167:6808/30036 9 ==== osd_op_reply(34 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f247c051040 con 0x562220b2c3b0 2023-10-05T19:28:38.420 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.414+0000 7f2484851700 1 -- 172.21.15.167:0/2512362784 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:35 5.17 5:e8000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x562220b2ecd0 con 0x7f246c011040 2023-10-05T19:28:38.421 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.414+0000 7f2485052700 1 -- 172.21.15.167:0/2512362784 <== osd.0 v2:172.21.15.118:6800/27690 9 ==== osd_op_reply(35 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f2474022040 con 0x7f246c011040 2023-10-05T19:28:38.421 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.414+0000 7f2485052700 1 -- 172.21.15.167:0/2512362784 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:36 5.f 5:f0000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x562220b2ecd0 con 0x562220af7740 2023-10-05T19:28:38.422 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.414+0000 7f2485052700 1 -- 172.21.15.167:0/2512362784 <== osd.1 v2:172.21.15.167:6800/30035 13 ==== osd_op_reply(36 [pgnls start_epoch 30 out=115b] v23'2 uv2 ondisk = 1) v8 ==== 144+0+115 (crc 0 0 0) 0x7f247402f080 con 0x562220af7740 2023-10-05T19:28:38.422 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.414+0000 7f2485052700 1 -- 172.21.15.167:0/2512362784 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:37 5.1f 5:f8000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x562220b2ecd0 con 0x7f247c007030 2023-10-05T19:28:38.423 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.418+0000 7f2484050700 1 -- 172.21.15.167:0/2512362784 <== osd.2 v2:172.21.15.167:6816/30037 6 ==== osd_op_reply(37 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f246c004bd0 con 0x7f247c007030 2023-10-05T19:28:38.423 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.418+0000 7f2486d1ce40 20 RGWRados::pool_iterate: got zonegroup_info.4f1f719d-9648-417b-b10c-e5d939825419 2023-10-05T19:28:38.423 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.418+0000 7f2486d1ce40 20 RGWRados::pool_iterate: got zone_info.9a9df965-6612-45d4-a45f-6e3c889bacc2 2023-10-05T19:28:38.423 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.418+0000 7f2486d1ce40 20 RGWRados::pool_iterate: got zone_names.default 2023-10-05T19:28:38.423 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.418+0000 7f2486d1ce40 20 RGWRados::pool_iterate: got zonegroups_names.default 2023-10-05T19:28:38.423 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.418+0000 7f2486d1ce40 20 rados->read ofs=0 len=0 2023-10-05T19:28:38.423 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.418+0000 7f2486d1ce40 1 -- 172.21.15.167:0/2512362784 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:38 5.12 5:49953fa1:::default.realm:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x562220b2b160 con 0x562220af7740 2023-10-05T19:28:38.424 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.418+0000 7f2485052700 1 -- 172.21.15.167:0/2512362784 <== osd.1 v2:172.21.15.167:6800/30035 14 ==== osd_op_reply(38 default.realm [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 ==== 157+0+0 (crc 0 0 0) 0x7f247402f080 con 0x562220af7740 2023-10-05T19:28:38.424 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.418+0000 7f2486d1ce40 20 rados_obj.operate() r=-2 bl.length=0 2023-10-05T19:28:38.425 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.418+0000 7f2486d1ce40 20 rados->read ofs=0 len=0 2023-10-05T19:28:38.425 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.418+0000 7f2486d1ce40 1 -- 172.21.15.167:0/2512362784 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:39 5.f 5:f4c53578:::zonegroups_names.default:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x562220b2ba50 con 0x562220af7740 2023-10-05T19:28:38.425 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.418+0000 7f2485052700 1 -- 172.21.15.167:0/2512362784 <== osd.1 v2:172.21.15.167:6800/30035 15 ==== osd_op_reply(39 zonegroups_names.default [read 0~46 out=46b] v0'0 uv2 ondisk = 0) v8 ==== 168+0+46 (crc 0 0 0) 0x7f247402f080 con 0x562220af7740 2023-10-05T19:28:38.425 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.418+0000 7f2486d1ce40 20 rados_obj.operate() r=0 bl.length=46 2023-10-05T19:28:38.425 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.418+0000 7f2486d1ce40 20 rados->read ofs=0 len=0 2023-10-05T19:28:38.425 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.418+0000 7f2486d1ce40 1 -- 172.21.15.167:0/2512362784 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:40 5.c 5:30d299a5:::zonegroup_info.4f1f719d-9648-417b-b10c-e5d939825419:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x562220b2f110 con 0x7f246c011040 2023-10-05T19:28:38.426 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.418+0000 7f2485052700 1 -- 172.21.15.167:0/2512362784 <== osd.0 v2:172.21.15.118:6800/27690 10 ==== osd_op_reply(40 zonegroup_info.4f1f719d-9648-417b-b10c-e5d939825419 [read 0~362 out=362b] v0'0 uv1 ondisk = 0) v8 ==== 195+0+362 (crc 0 0 0) 0x7f247402f080 con 0x7f246c011040 2023-10-05T19:28:38.426 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.418+0000 7f2486d1ce40 20 rados_obj.operate() r=0 bl.length=362 2023-10-05T19:28:38.426 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.418+0000 7f2486d1ce40 20 rados->read ofs=0 len=0 2023-10-05T19:28:38.426 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.418+0000 7f2486d1ce40 1 -- 172.21.15.167:0/2512362784 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:41 5.13 5:cbbf6c60:::converted:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x562220b2f110 con 0x562220b2c3b0 2023-10-05T19:28:38.426 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.418+0000 7f2484851700 1 -- 172.21.15.167:0/2512362784 <== osd.3 v2:172.21.15.167:6808/30036 10 ==== osd_op_reply(41 converted [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 ==== 153+0+0 (crc 0 0 0) 0x7f247c051040 con 0x562220b2c3b0 2023-10-05T19:28:38.427 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.418+0000 7f2486d1ce40 20 rados_obj.operate() r=-2 bl.length=0 2023-10-05T19:28:38.427 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.418+0000 7f2486d1ce40 20 rados->read ofs=0 len=0 2023-10-05T19:28:38.427 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.422+0000 7f2486d1ce40 1 -- 172.21.15.167:0/2512362784 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:42 5.12 5:49953fa1:::default.realm:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x562220b2f4c0 con 0x562220af7740 2023-10-05T19:28:38.428 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.422+0000 7f2485052700 1 -- 172.21.15.167:0/2512362784 <== osd.1 v2:172.21.15.167:6800/30035 16 ==== osd_op_reply(42 default.realm [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 ==== 157+0+0 (crc 0 0 0) 0x7f247402f080 con 0x562220af7740 2023-10-05T19:28:38.428 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.422+0000 7f2486d1ce40 20 rados_obj.operate() r=-2 bl.length=0 2023-10-05T19:28:38.428 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.422+0000 7f2486d1ce40 20 rados->read ofs=0 len=0 2023-10-05T19:28:38.428 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.422+0000 7f2486d1ce40 1 -- 172.21.15.167:0/2512362784 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:43 5.f 5:f4c53578:::zonegroups_names.default:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x562220b2f8c0 con 0x562220af7740 2023-10-05T19:28:38.428 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.422+0000 7f2485052700 1 -- 172.21.15.167:0/2512362784 <== osd.1 v2:172.21.15.167:6800/30035 17 ==== osd_op_reply(43 zonegroups_names.default [read 0~46 out=46b] v0'0 uv2 ondisk = 0) v8 ==== 168+0+46 (crc 0 0 0) 0x7f247402f080 con 0x562220af7740 2023-10-05T19:28:38.428 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.422+0000 7f2486d1ce40 20 rados_obj.operate() r=0 bl.length=46 2023-10-05T19:28:38.428 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.422+0000 7f2486d1ce40 1 -- 172.21.15.167:0/2512362784 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:44 5.0 5:00000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x562220b308d0 con 0x562220b2c3b0 2023-10-05T19:28:38.430 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.426+0000 7f2484851700 1 -- 172.21.15.167:0/2512362784 <== osd.3 v2:172.21.15.167:6808/30036 11 ==== osd_op_reply(44 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f247c051040 con 0x562220b2c3b0 2023-10-05T19:28:38.431 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.426+0000 7f2484851700 1 -- 172.21.15.167:0/2512362784 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:45 5.10 5:08000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x562220b308d0 con 0x7f247c007030 2023-10-05T19:28:38.432 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.426+0000 7f2484050700 1 -- 172.21.15.167:0/2512362784 <== osd.2 v2:172.21.15.167:6816/30037 7 ==== osd_op_reply(45 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f246c004bd0 con 0x7f247c007030 2023-10-05T19:28:38.432 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.426+0000 7f2484050700 1 -- 172.21.15.167:0/2512362784 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:46 5.8 5:10000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x562220b308d0 con 0x7f246c011040 2023-10-05T19:28:38.432 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.426+0000 7f2485052700 1 -- 172.21.15.167:0/2512362784 <== osd.0 v2:172.21.15.118:6800/27690 11 ==== osd_op_reply(46 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f247402f080 con 0x7f246c011040 2023-10-05T19:28:38.433 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.426+0000 7f2485052700 1 -- 172.21.15.167:0/2512362784 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:47 5.18 5:18000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x562220b2ba50 con 0x562220af7740 2023-10-05T19:28:38.433 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.426+0000 7f2485052700 1 -- 172.21.15.167:0/2512362784 <== osd.1 v2:172.21.15.167:6800/30035 18 ==== osd_op_reply(47 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f247402f080 con 0x562220af7740 2023-10-05T19:28:38.433 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.426+0000 7f2485052700 1 -- 172.21.15.167:0/2512362784 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:48 5.4 5:20000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x562220b2ba50 con 0x562220af7740 2023-10-05T19:28:38.434 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.430+0000 7f2485052700 1 -- 172.21.15.167:0/2512362784 <== osd.1 v2:172.21.15.167:6800/30035 19 ==== osd_op_reply(48 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f247402f080 con 0x562220af7740 2023-10-05T19:28:38.434 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.430+0000 7f2485052700 1 -- 172.21.15.167:0/2512362784 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:49 5.14 5:28000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x562220b2ba50 con 0x7f246c011040 2023-10-05T19:28:38.435 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.430+0000 7f2485052700 1 -- 172.21.15.167:0/2512362784 <== osd.0 v2:172.21.15.118:6800/27690 12 ==== osd_op_reply(49 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f247402f080 con 0x7f246c011040 2023-10-05T19:28:38.435 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.430+0000 7f2485052700 1 -- 172.21.15.167:0/2512362784 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:50 5.c 5:30000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x562220b2ba50 con 0x7f246c011040 2023-10-05T19:28:38.436 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.430+0000 7f2485052700 1 -- 172.21.15.167:0/2512362784 <== osd.0 v2:172.21.15.118:6800/27690 13 ==== osd_op_reply(50 [pgnls start_epoch 30 out=112b] v23'1 uv1 ondisk = 1) v8 ==== 144+0+112 (crc 0 0 0) 0x7f247402f080 con 0x7f246c011040 2023-10-05T19:28:38.436 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.430+0000 7f2485052700 1 -- 172.21.15.167:0/2512362784 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:51 5.1c 5:38000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x562220b2ba50 con 0x7f247c007030 2023-10-05T19:28:38.437 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.430+0000 7f2484050700 1 -- 172.21.15.167:0/2512362784 <== osd.2 v2:172.21.15.167:6816/30037 8 ==== osd_op_reply(51 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f246c004bd0 con 0x7f247c007030 2023-10-05T19:28:38.437 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.430+0000 7f2484050700 1 -- 172.21.15.167:0/2512362784 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:52 5.2 5:40000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x562220b2ba50 con 0x562220b2c3b0 2023-10-05T19:28:38.438 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.430+0000 7f2484851700 1 -- 172.21.15.167:0/2512362784 <== osd.3 v2:172.21.15.167:6808/30036 12 ==== osd_op_reply(52 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f247c051040 con 0x562220b2c3b0 2023-10-05T19:28:38.438 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.430+0000 7f2484851700 1 -- 172.21.15.167:0/2512362784 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:53 5.12 5:48000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x562220b2ba50 con 0x562220af7740 2023-10-05T19:28:38.438 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.434+0000 7f2485052700 1 -- 172.21.15.167:0/2512362784 <== osd.1 v2:172.21.15.167:6800/30035 20 ==== osd_op_reply(53 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f247402f080 con 0x562220af7740 2023-10-05T19:28:38.438 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.434+0000 7f2485052700 1 -- 172.21.15.167:0/2512362784 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:54 5.a 5:50000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x562220b2ba50 con 0x7f246c011040 2023-10-05T19:28:38.439 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.434+0000 7f2485052700 1 -- 172.21.15.167:0/2512362784 <== osd.0 v2:172.21.15.118:6800/27690 14 ==== osd_op_reply(54 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f247402f080 con 0x7f246c011040 2023-10-05T19:28:38.440 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.434+0000 7f2485052700 1 -- 172.21.15.167:0/2512362784 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:55 5.1a 5:58000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x562220b2ba50 con 0x562220af7740 2023-10-05T19:28:38.440 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.434+0000 7f2485052700 1 -- 172.21.15.167:0/2512362784 <== osd.1 v2:172.21.15.167:6800/30035 21 ==== osd_op_reply(55 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f247402f080 con 0x562220af7740 2023-10-05T19:28:38.440 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.434+0000 7f2485052700 1 -- 172.21.15.167:0/2512362784 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:56 5.6 5:60000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x562220b2ba50 con 0x7f247c007030 2023-10-05T19:28:38.441 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.434+0000 7f2484050700 1 -- 172.21.15.167:0/2512362784 <== osd.2 v2:172.21.15.167:6816/30037 9 ==== osd_op_reply(56 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f246c004bd0 con 0x7f247c007030 2023-10-05T19:28:38.441 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.434+0000 7f2484050700 1 -- 172.21.15.167:0/2512362784 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:57 5.16 5:68000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x562220b2ba50 con 0x562220b2c3b0 2023-10-05T19:28:38.442 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.438+0000 7f2484851700 1 -- 172.21.15.167:0/2512362784 <== osd.3 v2:172.21.15.167:6808/30036 13 ==== osd_op_reply(57 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f247c051040 con 0x562220b2c3b0 2023-10-05T19:28:38.442 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.438+0000 7f2484851700 1 -- 172.21.15.167:0/2512362784 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:58 5.e 5:70000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x562220b2ba50 con 0x562220b2c3b0 2023-10-05T19:28:38.443 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.438+0000 7f2484851700 1 -- 172.21.15.167:0/2512362784 <== osd.3 v2:172.21.15.167:6808/30036 14 ==== osd_op_reply(58 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f247c051040 con 0x562220b2c3b0 2023-10-05T19:28:38.443 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.438+0000 7f2484851700 1 -- 172.21.15.167:0/2512362784 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:59 5.1e 5:78000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x562220b2ba50 con 0x7f246c011040 2023-10-05T19:28:38.444 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.438+0000 7f2485052700 1 -- 172.21.15.167:0/2512362784 <== osd.0 v2:172.21.15.118:6800/27690 15 ==== osd_op_reply(59 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f247402f080 con 0x7f246c011040 2023-10-05T19:28:38.444 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.438+0000 7f2485052700 1 -- 172.21.15.167:0/2512362784 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:60 5.1 5:80000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x562220b2ba50 con 0x562220b2c3b0 2023-10-05T19:28:38.445 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.438+0000 7f2484851700 1 -- 172.21.15.167:0/2512362784 <== osd.3 v2:172.21.15.167:6808/30036 15 ==== osd_op_reply(60 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f247c051040 con 0x562220b2c3b0 2023-10-05T19:28:38.445 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.438+0000 7f2484851700 1 -- 172.21.15.167:0/2512362784 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:61 5.11 5:88000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x562220b2ba50 con 0x562220b2c3b0 2023-10-05T19:28:38.446 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.442+0000 7f2484851700 1 -- 172.21.15.167:0/2512362784 <== osd.3 v2:172.21.15.167:6808/30036 16 ==== osd_op_reply(61 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f247c051040 con 0x562220b2c3b0 2023-10-05T19:28:38.446 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.442+0000 7f2484851700 1 -- 172.21.15.167:0/2512362784 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:62 5.9 5:90000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x562220b2ba50 con 0x562220af7740 2023-10-05T19:28:38.447 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.442+0000 7f2485052700 1 -- 172.21.15.167:0/2512362784 <== osd.1 v2:172.21.15.167:6800/30035 22 ==== osd_op_reply(62 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f247402f080 con 0x562220af7740 2023-10-05T19:28:38.447 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.442+0000 7f2485052700 1 -- 172.21.15.167:0/2512362784 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:63 5.19 5:98000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x562220b2ba50 con 0x562220af7740 2023-10-05T19:28:38.448 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.442+0000 7f2485052700 1 -- 172.21.15.167:0/2512362784 <== osd.1 v2:172.21.15.167:6800/30035 23 ==== osd_op_reply(63 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f247402f080 con 0x562220af7740 2023-10-05T19:28:38.448 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.442+0000 7f2485052700 1 -- 172.21.15.167:0/2512362784 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:64 5.5 5:a0000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x562220b2ba50 con 0x7f246c011040 2023-10-05T19:28:38.449 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.442+0000 7f2485052700 1 -- 172.21.15.167:0/2512362784 <== osd.0 v2:172.21.15.118:6800/27690 16 ==== osd_op_reply(64 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f247402f080 con 0x7f246c011040 2023-10-05T19:28:38.449 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.442+0000 7f2485052700 1 -- 172.21.15.167:0/2512362784 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:65 5.15 5:a8000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x562220b2ba50 con 0x562220b2c3b0 2023-10-05T19:28:38.450 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.442+0000 7f2484851700 1 -- 172.21.15.167:0/2512362784 <== osd.3 v2:172.21.15.167:6808/30036 17 ==== osd_op_reply(65 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f247c051040 con 0x562220b2c3b0 2023-10-05T19:28:38.450 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.442+0000 7f2484851700 1 -- 172.21.15.167:0/2512362784 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:66 5.d 5:b0000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x562220b2ba50 con 0x7f247c007030 2023-10-05T19:28:38.450 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.446+0000 7f2484050700 1 -- 172.21.15.167:0/2512362784 <== osd.2 v2:172.21.15.167:6816/30037 10 ==== osd_op_reply(66 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f246c004bd0 con 0x7f247c007030 2023-10-05T19:28:38.451 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.446+0000 7f2484050700 1 -- 172.21.15.167:0/2512362784 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:67 5.1d 5:b8000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x562220b2ba50 con 0x562220af7740 2023-10-05T19:28:38.451 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.446+0000 7f2485052700 1 -- 172.21.15.167:0/2512362784 <== osd.1 v2:172.21.15.167:6800/30035 24 ==== osd_op_reply(67 [pgnls start_epoch 30 out=107b] v30'2 uv2 ondisk = 1) v8 ==== 144+0+107 (crc 0 0 0) 0x7f247402f080 con 0x562220af7740 2023-10-05T19:28:38.451 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.446+0000 7f2485052700 1 -- 172.21.15.167:0/2512362784 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:68 5.3 5:c0000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x562220b2ba50 con 0x7f246c011040 2023-10-05T19:28:38.452 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.446+0000 7f2485052700 1 -- 172.21.15.167:0/2512362784 <== osd.0 v2:172.21.15.118:6800/27690 17 ==== osd_op_reply(68 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f247402f080 con 0x7f246c011040 2023-10-05T19:28:38.452 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.446+0000 7f2485052700 1 -- 172.21.15.167:0/2512362784 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:69 5.13 5:c8000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x562220b2ba50 con 0x562220b2c3b0 2023-10-05T19:28:38.453 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.446+0000 7f2484851700 1 -- 172.21.15.167:0/2512362784 <== osd.3 v2:172.21.15.167:6808/30036 18 ==== osd_op_reply(69 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f247c051040 con 0x562220b2c3b0 2023-10-05T19:28:38.453 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.446+0000 7f2484851700 1 -- 172.21.15.167:0/2512362784 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:70 5.b 5:d0000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x562220b2ba50 con 0x7f246c011040 2023-10-05T19:28:38.454 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.450+0000 7f2485052700 1 -- 172.21.15.167:0/2512362784 <== osd.0 v2:172.21.15.118:6800/27690 18 ==== osd_op_reply(70 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f247402f080 con 0x7f246c011040 2023-10-05T19:28:38.454 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.450+0000 7f2485052700 1 -- 172.21.15.167:0/2512362784 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:71 5.1b 5:d8000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x562220b2ba50 con 0x7f247c007030 2023-10-05T19:28:38.455 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.450+0000 7f2484050700 1 -- 172.21.15.167:0/2512362784 <== osd.2 v2:172.21.15.167:6816/30037 11 ==== osd_op_reply(71 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f246c004bd0 con 0x7f247c007030 2023-10-05T19:28:38.455 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.450+0000 7f2484050700 1 -- 172.21.15.167:0/2512362784 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:72 5.7 5:e0000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x562220b2ba50 con 0x562220b2c3b0 2023-10-05T19:28:38.455 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.450+0000 7f2484851700 1 -- 172.21.15.167:0/2512362784 <== osd.3 v2:172.21.15.167:6808/30036 19 ==== osd_op_reply(72 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f247c051040 con 0x562220b2c3b0 2023-10-05T19:28:38.456 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.450+0000 7f2484851700 1 -- 172.21.15.167:0/2512362784 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:73 5.17 5:e8000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x562220b2ba50 con 0x7f246c011040 2023-10-05T19:28:38.456 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.450+0000 7f2485052700 1 -- 172.21.15.167:0/2512362784 <== osd.0 v2:172.21.15.118:6800/27690 19 ==== osd_op_reply(73 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f247402f080 con 0x7f246c011040 2023-10-05T19:28:38.457 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.450+0000 7f2485052700 1 -- 172.21.15.167:0/2512362784 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:74 5.f 5:f0000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x562220b2ba50 con 0x562220af7740 2023-10-05T19:28:38.457 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.450+0000 7f2485052700 1 -- 172.21.15.167:0/2512362784 <== osd.1 v2:172.21.15.167:6800/30035 25 ==== osd_op_reply(74 [pgnls start_epoch 30 out=115b] v23'2 uv2 ondisk = 1) v8 ==== 144+0+115 (crc 0 0 0) 0x7f247402f080 con 0x562220af7740 2023-10-05T19:28:38.458 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.450+0000 7f2485052700 1 -- 172.21.15.167:0/2512362784 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:75 5.1f 5:f8000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x562220b2ba50 con 0x7f247c007030 2023-10-05T19:28:38.458 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.454+0000 7f2484050700 1 -- 172.21.15.167:0/2512362784 <== osd.2 v2:172.21.15.167:6816/30037 12 ==== osd_op_reply(75 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f246c004bd0 con 0x7f247c007030 2023-10-05T19:28:38.459 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.454+0000 7f2486d1ce40 20 RGWRados::pool_iterate: got zonegroup_info.4f1f719d-9648-417b-b10c-e5d939825419 2023-10-05T19:28:38.459 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.454+0000 7f2486d1ce40 20 RGWRados::pool_iterate: got zone_info.9a9df965-6612-45d4-a45f-6e3c889bacc2 2023-10-05T19:28:38.459 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.454+0000 7f2486d1ce40 20 RGWRados::pool_iterate: got zone_names.default 2023-10-05T19:28:38.459 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.454+0000 7f2486d1ce40 20 RGWRados::pool_iterate: got zonegroups_names.default 2023-10-05T19:28:38.459 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.454+0000 7f2486d1ce40 20 rados->read ofs=0 len=0 2023-10-05T19:28:38.459 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.454+0000 7f2486d1ce40 1 -- 172.21.15.167:0/2512362784 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:76 5.f 5:f43b6ece:::zone_names.default:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x562220b2f8c0 con 0x562220af7740 2023-10-05T19:28:38.460 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.454+0000 7f2485052700 1 -- 172.21.15.167:0/2512362784 <== osd.1 v2:172.21.15.167:6800/30035 26 ==== osd_op_reply(76 zone_names.default [read 0~46 out=46b] v0'0 uv1 ondisk = 0) v8 ==== 162+0+46 (crc 0 0 0) 0x7f247402f080 con 0x562220af7740 2023-10-05T19:28:38.460 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.454+0000 7f2486d1ce40 20 rados_obj.operate() r=0 bl.length=46 2023-10-05T19:28:38.460 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.454+0000 7f2486d1ce40 20 rados->read ofs=0 len=0 2023-10-05T19:28:38.460 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.454+0000 7f2486d1ce40 1 -- 172.21.15.167:0/2512362784 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:77 5.1d 5:b8374eb4:::zone_info.9a9df965-6612-45d4-a45f-6e3c889bacc2:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x562220b2fc70 con 0x562220af7740 2023-10-05T19:28:38.461 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.454+0000 7f2485052700 1 -- 172.21.15.167:0/2512362784 <== osd.1 v2:172.21.15.167:6800/30035 27 ==== osd_op_reply(77 zone_info.9a9df965-6612-45d4-a45f-6e3c889bacc2 [read 0~874 out=874b] v0'0 uv2 ondisk = 0) v8 ==== 190+0+874 (crc 0 0 0) 0x7f247402f080 con 0x562220af7740 2023-10-05T19:28:38.461 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.454+0000 7f2486d1ce40 20 rados_obj.operate() r=0 bl.length=874 2023-10-05T19:28:38.461 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.454+0000 7f2486d1ce40 20 rados->read ofs=0 len=0 2023-10-05T19:28:38.461 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.454+0000 7f2486d1ce40 1 -- 172.21.15.167:0/2512362784 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:78 5.f 5:f4c53578:::zonegroups_names.default:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x562220b2b5c0 con 0x562220af7740 2023-10-05T19:28:38.461 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.454+0000 7f2485052700 1 -- 172.21.15.167:0/2512362784 <== osd.1 v2:172.21.15.167:6800/30035 28 ==== osd_op_reply(78 zonegroups_names.default [read 0~46 out=46b] v0'0 uv2 ondisk = 0) v8 ==== 168+0+46 (crc 0 0 0) 0x7f247402f080 con 0x562220af7740 2023-10-05T19:28:38.461 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.454+0000 7f2486d1ce40 20 rados_obj.operate() r=0 bl.length=46 2023-10-05T19:28:38.461 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.454+0000 7f2486d1ce40 20 rados->read ofs=0 len=0 2023-10-05T19:28:38.461 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.454+0000 7f2486d1ce40 1 -- 172.21.15.167:0/2512362784 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:79 5.c 5:30d299a5:::zonegroup_info.4f1f719d-9648-417b-b10c-e5d939825419:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x562220b2b5a0 con 0x7f246c011040 2023-10-05T19:28:38.462 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.454+0000 7f2485052700 1 -- 172.21.15.167:0/2512362784 <== osd.0 v2:172.21.15.118:6800/27690 20 ==== osd_op_reply(79 zonegroup_info.4f1f719d-9648-417b-b10c-e5d939825419 [read 0~362 out=362b] v0'0 uv1 ondisk = 0) v8 ==== 195+0+362 (crc 0 0 0) 0x7f247402f080 con 0x7f246c011040 2023-10-05T19:28:38.462 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.454+0000 7f2486d1ce40 20 rados_obj.operate() r=0 bl.length=362 2023-10-05T19:28:38.462 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.454+0000 7f2486d1ce40 20 rados->read ofs=0 len=0 2023-10-05T19:28:38.462 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.454+0000 7f2486d1ce40 1 -- 172.21.15.167:0/2512362784 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:80 5.3 5:c0ad8cbd:::region_map:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x562220b301b0 con 0x7f246c011040 2023-10-05T19:28:38.463 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.458+0000 7f2485052700 1 -- 172.21.15.167:0/2512362784 <== osd.0 v2:172.21.15.118:6800/27690 21 ==== osd_op_reply(80 region_map [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 ==== 154+0+0 (crc 0 0 0) 0x7f247402f080 con 0x7f246c011040 2023-10-05T19:28:38.463 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.458+0000 7f2486d1ce40 20 rados_obj.operate() r=-2 bl.length=0 2023-10-05T19:28:38.463 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.458+0000 7f2486d1ce40 20 zone default found 2023-10-05T19:28:38.463 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.458+0000 7f2486d1ce40 4 Realm: () 2023-10-05T19:28:38.463 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.458+0000 7f2486d1ce40 4 ZoneGroup: default (4f1f719d-9648-417b-b10c-e5d939825419) 2023-10-05T19:28:38.463 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.458+0000 7f2486d1ce40 4 Zone: default (9a9df965-6612-45d4-a45f-6e3c889bacc2) 2023-10-05T19:28:38.463 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.458+0000 7f2486d1ce40 10 cannot find current period zonegroup using local zonegroup configuration 2023-10-05T19:28:38.463 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.458+0000 7f2486d1ce40 20 zonegroup default 2023-10-05T19:28:38.463 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.458+0000 7f2486d1ce40 20 rados->read ofs=0 len=0 2023-10-05T19:28:38.463 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.458+0000 7f2486d1ce40 1 -- 172.21.15.167:0/2512362784 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:81 5.3 5:c52100b6:::period_config.default:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x562220b301b0 con 0x7f246c011040 2023-10-05T19:28:38.464 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.458+0000 7f2485052700 1 -- 172.21.15.167:0/2512362784 <== osd.0 v2:172.21.15.118:6800/27690 22 ==== osd_op_reply(81 period_config.default [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 ==== 165+0+0 (crc 0 0 0) 0x7f247402f080 con 0x7f246c011040 2023-10-05T19:28:38.464 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.458+0000 7f2486d1ce40 20 rados_obj.operate() r=-2 bl.length=0 2023-10-05T19:28:38.464 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.458+0000 7f2486d1ce40 20 rados->read ofs=0 len=0 2023-10-05T19:28:38.465 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.458+0000 7f2486d1ce40 1 -- 172.21.15.167:0/2512362784 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:82 6.2 6:4347d321:::bucket.sync-source-hints.:head [call version.read in=11b,read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x562220b31b30 con 0x562220b2c3b0 2023-10-05T19:28:38.465 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.458+0000 7f2484851700 1 -- 172.21.15.167:0/2512362784 <== osd.3 v2:172.21.15.167:6808/30036 20 ==== osd_op_reply(82 bucket.sync-source-hints. [call,read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 ==== 211+0+0 (crc 0 0 0) 0x7f247c051040 con 0x562220b2c3b0 2023-10-05T19:28:38.465 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.458+0000 7f2486d1ce40 20 rados_obj.operate() r=-2 bl.length=0 2023-10-05T19:28:38.465 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.458+0000 7f2486d1ce40 20 rados->read ofs=0 len=0 2023-10-05T19:28:38.465 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.458+0000 7f2486d1ce40 1 -- 172.21.15.167:0/2512362784 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:83 6.b 6:d467b91b:::bucket.sync-target-hints.:head [call version.read in=11b,read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x562220b31ee0 con 0x562220b2c3b0 2023-10-05T19:28:38.465 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.458+0000 7f2484851700 1 -- 172.21.15.167:0/2512362784 <== osd.3 v2:172.21.15.167:6808/30036 21 ==== osd_op_reply(83 bucket.sync-target-hints. [call,read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 ==== 211+0+0 (crc 0 0 0) 0x7f247c051040 con 0x562220b2c3b0 2023-10-05T19:28:38.466 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.458+0000 7f2486d1ce40 20 rados_obj.operate() r=-2 bl.length=0 2023-10-05T19:28:38.466 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.458+0000 7f2486d1ce40 20 started sync module instance, tier type = 2023-10-05T19:28:38.466 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.458+0000 7f2486d1ce40 20 started zone id=9a9df965-6612-45d4-a45f-6e3c889bacc2 (name=default) with tier type = 2023-10-05T19:28:38.466 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.458+0000 7f2486d1ce40 1 -- 172.21.15.167:0/2512362784 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:84 7.1f 7:f95f44c2:::notify.0:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x562220b311f0 con 0x7f247c007030 2023-10-05T19:28:38.467 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.462+0000 7f2484050700 1 -- 172.21.15.167:0/2512362784 <== osd.2 v2:172.21.15.167:6816/30037 13 ==== osd_op_reply(84 notify.0 [create] v30'180 uv180 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f246c02f040 con 0x7f247c007030 2023-10-05T19:28:38.468 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.462+0000 7f2486d1ce40 1 -- 172.21.15.167:0/2512362784 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:85 7.1f 7:f95f44c2:::notify.0:head [watch watch cookie 94704577477744] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x562220b32290 con 0x7f247c007030 2023-10-05T19:28:38.468 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.462+0000 7f2486d1ce40 1 -- 172.21.15.167:0/2512362784 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:86 7.0 7:05bf5b68:::notify.1:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x562220b32b30 con 0x562220af7740 2023-10-05T19:28:38.471 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.466+0000 7f2484050700 1 -- 172.21.15.167:0/2512362784 <== osd.2 v2:172.21.15.167:6816/30037 14 ==== osd_op_reply(85 notify.0 [watch watch cookie 94704577477744] v30'181 uv180 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f246c02f040 con 0x7f247c007030 2023-10-05T19:28:38.471 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.466+0000 7f2485052700 1 -- 172.21.15.167:0/2512362784 <== osd.1 v2:172.21.15.167:6800/30035 29 ==== osd_op_reply(86 notify.1 [create] v30'180 uv180 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f247402f080 con 0x562220af7740 2023-10-05T19:28:38.471 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.466+0000 7f2486d1ce40 1 -- 172.21.15.167:0/2512362784 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:87 7.0 7:05bf5b68:::notify.1:head [watch watch cookie 94704577488448] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x562220b32f80 con 0x562220af7740 2023-10-05T19:28:38.471 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.466+0000 7f2486d1ce40 1 -- 172.21.15.167:0/2512362784 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:88 7.15 7:a93a5511:::notify.2:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x562220b33820 con 0x562220b2c3b0 2023-10-05T19:28:38.474 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.470+0000 7f2484851700 1 -- 172.21.15.167:0/2512362784 <== osd.3 v2:172.21.15.167:6808/30036 22 ==== osd_op_reply(88 notify.2 [create] v30'180 uv180 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f247c051040 con 0x562220b2c3b0 2023-10-05T19:28:38.474 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.470+0000 7f2486d1ce40 1 -- 172.21.15.167:0/2512362784 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:89 7.15 7:a93a5511:::notify.2:head [watch watch cookie 94704577490816] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x562220b340c0 con 0x562220b2c3b0 2023-10-05T19:28:38.474 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.470+0000 7f2485052700 1 -- 172.21.15.167:0/2512362784 <== osd.1 v2:172.21.15.167:6800/30035 30 ==== osd_op_reply(87 notify.1 [watch watch cookie 94704577488448] v30'181 uv180 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f2474050040 con 0x562220af7740 2023-10-05T19:28:38.474 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.470+0000 7f2486d1ce40 1 -- 172.21.15.167:0/2512362784 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:90 7.e 7:7759931f:::notify.3:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x562220b34470 con 0x7f247c007030 2023-10-05T19:28:38.477 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.470+0000 7f2484050700 1 -- 172.21.15.167:0/2512362784 <== osd.2 v2:172.21.15.167:6816/30037 15 ==== osd_op_reply(90 notify.3 [create] v30'180 uv180 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f246c02f040 con 0x7f247c007030 2023-10-05T19:28:38.477 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.470+0000 7f2486d1ce40 1 -- 172.21.15.167:0/2512362784 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:91 7.e 7:7759931f:::notify.3:head [watch watch cookie 94704577497120] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x562220b32bb0 con 0x7f247c007030 2023-10-05T19:28:38.477 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.470+0000 7f2486d1ce40 1 -- 172.21.15.167:0/2512362784 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:92 7.d 7:b4812045:::notify.4:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x562220b35910 con 0x562220af7740 2023-10-05T19:28:38.477 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.470+0000 7f2484851700 1 -- 172.21.15.167:0/2512362784 <== osd.3 v2:172.21.15.167:6808/30036 23 ==== osd_op_reply(89 notify.2 [watch watch cookie 94704577490816] v30'181 uv180 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f247c051040 con 0x562220b2c3b0 2023-10-05T19:28:38.479 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.474+0000 7f2484050700 1 -- 172.21.15.167:0/2512362784 <== osd.2 v2:172.21.15.167:6816/30037 16 ==== osd_op_reply(91 notify.3 [watch watch cookie 94704577497120] v30'181 uv180 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f246c02f040 con 0x7f247c007030 2023-10-05T19:28:38.480 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.474+0000 7f2485052700 1 -- 172.21.15.167:0/2512362784 <== osd.1 v2:172.21.15.167:6800/30035 31 ==== osd_op_reply(92 notify.4 [create] v30'180 uv180 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f2474050040 con 0x562220af7740 2023-10-05T19:28:38.480 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.474+0000 7f2486d1ce40 1 -- 172.21.15.167:0/2512362784 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:93 7.d 7:b4812045:::notify.4:head [watch watch cookie 94704577498224] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x562220b353e0 con 0x562220af7740 2023-10-05T19:28:38.480 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.474+0000 7f2486d1ce40 1 -- 172.21.15.167:0/2512362784 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:94 7.3 7:c609908c:::notify.5:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x562220b36200 con 0x7f246c011040 2023-10-05T19:28:38.482 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.478+0000 7f2485052700 1 -- 172.21.15.167:0/2512362784 <== osd.1 v2:172.21.15.167:6800/30035 32 ==== osd_op_reply(93 notify.4 [watch watch cookie 94704577498224] v30'181 uv180 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f2474050040 con 0x562220af7740 2023-10-05T19:28:38.483 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.478+0000 7f2485052700 1 -- 172.21.15.167:0/2512362784 <== osd.0 v2:172.21.15.118:6800/27690 23 ==== osd_op_reply(94 notify.5 [create] v30'180 uv180 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f2474050040 con 0x7f246c011040 2023-10-05T19:28:38.483 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.478+0000 7f2486d1ce40 1 -- 172.21.15.167:0/2512362784 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:95 7.3 7:c609908c:::notify.5:head [watch watch cookie 94704577493968] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x562220b361b0 con 0x7f246c011040 2023-10-05T19:28:38.483 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.478+0000 7f2486d1ce40 1 -- 172.21.15.167:0/2512362784 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:96 7.14 7:2b04a3e9:::notify.6:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x562220b36c60 con 0x7f246c011040 2023-10-05T19:28:38.485 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.478+0000 7f2485052700 1 -- 172.21.15.167:0/2512362784 <== osd.0 v2:172.21.15.118:6800/27690 24 ==== osd_op_reply(95 notify.5 [watch watch cookie 94704577493968] v30'181 uv180 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f2474050040 con 0x7f246c011040 2023-10-05T19:28:38.486 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.478+0000 7f2485052700 1 -- 172.21.15.167:0/2512362784 <== osd.0 v2:172.21.15.118:6800/27690 25 ==== osd_op_reply(96 notify.6 [create] v30'180 uv180 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f2474050040 con 0x7f246c011040 2023-10-05T19:28:38.486 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.478+0000 7f2486d1ce40 1 -- 172.21.15.167:0/2512362784 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:97 7.14 7:2b04a3e9:::notify.6:head [watch watch cookie 94704577502688] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x562220b36c60 con 0x7f246c011040 2023-10-05T19:28:38.486 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.478+0000 7f2486d1ce40 1 -- 172.21.15.167:0/2512362784 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:98 7.9 7:93e5b521:::notify.7:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x562220b37670 con 0x7f246c011040 2023-10-05T19:28:38.489 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.482+0000 7f2485052700 1 -- 172.21.15.167:0/2512362784 <== osd.0 v2:172.21.15.118:6800/27690 26 ==== osd_op_reply(97 notify.6 [watch watch cookie 94704577502688] v30'181 uv180 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f2474050040 con 0x7f246c011040 2023-10-05T19:28:38.489 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.482+0000 7f2485052700 1 -- 172.21.15.167:0/2512362784 <== osd.0 v2:172.21.15.118:6800/27690 27 ==== osd_op_reply(98 notify.7 [create] v30'180 uv180 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f2474050040 con 0x7f246c011040 2023-10-05T19:28:38.489 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.482+0000 7f2486d1ce40 1 -- 172.21.15.167:0/2512362784 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:99 7.9 7:93e5b521:::notify.7:head [watch watch cookie 94704577505408] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x562220b37670 con 0x7f246c011040 2023-10-05T19:28:38.489 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.482+0000 7f2486d1ce40 20 add_watcher() i=0 2023-10-05T19:28:38.489 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.482+0000 7f2486d1ce40 20 add_watcher() i=1 2023-10-05T19:28:38.489 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.482+0000 7f2486d1ce40 20 add_watcher() i=2 2023-10-05T19:28:38.489 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.482+0000 7f2486d1ce40 20 add_watcher() i=3 2023-10-05T19:28:38.489 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.482+0000 7f2486d1ce40 20 add_watcher() i=4 2023-10-05T19:28:38.490 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.482+0000 7f2486d1ce40 20 add_watcher() i=5 2023-10-05T19:28:38.490 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.482+0000 7f2486d1ce40 20 add_watcher() i=6 2023-10-05T19:28:38.491 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.486+0000 7f2485052700 1 -- 172.21.15.167:0/2512362784 <== osd.0 v2:172.21.15.118:6800/27690 28 ==== osd_op_reply(99 notify.7 [watch watch cookie 94704577505408] v30'181 uv180 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f2474050040 con 0x7f246c011040 2023-10-05T19:28:38.491 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.486+0000 7f2486d1ce40 20 add_watcher() i=7 2023-10-05T19:28:38.491 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.486+0000 7f2486d1ce40 2 all 8 watchers are set, enabling cache 2023-10-05T19:28:38.491 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.486+0000 7f2486d1ce40 1 -- 172.21.15.167:0/2512362784 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:100 6.4 6:22d26bf9:::data_loggenerations_metadata:head [call version.check_conds in=74b,call version.read in=11b,read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x562220b3a780 con 0x562220af7740 2023-10-05T19:28:38.492 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.486+0000 7f2485052700 1 -- 172.21.15.167:0/2512362784 <== osd.1 v2:172.21.15.167:6800/30035 33 ==== osd_op_reply(100 data_loggenerations_metadata [call,call out=48b,read 0~28 out=28b] v0'0 uv1 ondisk = 0) v8 ==== 256+0+76 (crc 0 0 0) 0x7f2474050040 con 0x562220af7740 2023-10-05T19:28:38.492 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.486+0000 7f2486d1ce40 1 -- 172.21.15.167:0/2512362784 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:101 6.4 6:22d26bf9:::data_loggenerations_metadata:head [watch watch cookie 94704577507712] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x562220b37ac0 con 0x562220af7740 2023-10-05T19:28:38.494 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.490+0000 7f2485052700 1 -- 172.21.15.167:0/2512362784 <== osd.1 v2:172.21.15.167:6800/30035 34 ==== osd_op_reply(101 data_loggenerations_metadata [watch watch cookie 94704577507712] v30'167 uv1 ondisk = 0) v8 ==== 172+0+0 (crc 0 0 0) 0x7f2474050040 con 0x562220af7740 2023-10-05T19:28:38.494 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.490+0000 7f24567e4700 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2023-10-05T19:28:38.495 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.490+0000 7f2486d1ce40 20 check_secure_mon_conn(): auth registy supported: methods=[2] modes=[2,1] 2023-10-05T19:28:38.495 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.490+0000 7f2486d1ce40 20 check_secure_mon_conn(): mode 1 is insecure 2023-10-05T19:28:38.495 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.490+0000 7f2486d1ce40 1 Processor -- start 2023-10-05T19:28:38.495 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.490+0000 7f2486d1ce40 1 -- start start 2023-10-05T19:28:38.496 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.490+0000 7f2486d1ce40 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x562220b44fa0 0x562220b45370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-05T19:28:38.496 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.490+0000 7f2486d1ce40 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x562220b458b0 con 0x562220b44fa0 2023-10-05T19:28:38.496 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.490+0000 7f2484050700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x562220b44fa0 0x562220b45370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-05T19:28:38.496 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.490+0000 7f2484050700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x562220b44fa0 0x562220b45370 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.118:3300/0 says I am v2:172.21.15.167:42210/0 (socket says 172.21.15.167:42210) 2023-10-05T19:28:38.496 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.490+0000 7f2484050700 1 -- 172.21.15.167:0/3357530832 learned_addr learned my addr 172.21.15.167:0/3357530832 (peer_addr_for_me v2:172.21.15.167:0/0) 2023-10-05T19:28:38.497 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.490+0000 7f2484050700 1 -- 172.21.15.167:0/3357530832 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x562220b459f0 con 0x562220b44fa0 2023-10-05T19:28:38.497 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.490+0000 7f2484050700 1 --2- 172.21.15.167:0/3357530832 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x562220b44fa0 0x562220b45370 secure :-1 s=READY pgs=498 cs=0 l=1 rev1=1 crypto rx=0x562220b29b10 tx=0x7f246c006000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=76acab6a29a2ce43 server_cookie=0 in_seq=0 out_seq=0 2023-10-05T19:28:38.497 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.490+0000 7f2455fe3700 1 -- 172.21.15.167:0/3357530832 <== mon.0 v2:172.21.15.118:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f246c02c040 con 0x562220b44fa0 2023-10-05T19:28:38.497 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.490+0000 7f2455fe3700 1 -- 172.21.15.167:0/3357530832 <== mon.0 v2:172.21.15.118:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f246c02d040 con 0x562220b44fa0 2023-10-05T19:28:38.497 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.490+0000 7f2455fe3700 1 -- 172.21.15.167:0/3357530832 <== mon.0 v2:172.21.15.118:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f246c028ca0 con 0x562220b44fa0 2023-10-05T19:28:38.498 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.494+0000 7f2486d1ce40 1 -- 172.21.15.167:0/3357530832 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x562220b44fa0 msgr2=0x562220b45370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-05T19:28:38.498 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.494+0000 7f2486d1ce40 1 --2- 172.21.15.167:0/3357530832 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x562220b44fa0 0x562220b45370 secure :-1 s=READY pgs=498 cs=0 l=1 rev1=1 crypto rx=0x562220b29b10 tx=0x7f246c006000 comp rx=0 tx=0).stop 2023-10-05T19:28:38.499 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.494+0000 7f2486d1ce40 1 -- 172.21.15.167:0/3357530832 shutdown_connections 2023-10-05T19:28:38.499 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.494+0000 7f2486d1ce40 1 --2- 172.21.15.167:0/3357530832 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x562220b44fa0 0x562220b45370 unknown :-1 s=CLOSED pgs=498 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-05T19:28:38.499 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.494+0000 7f2486d1ce40 1 -- 172.21.15.167:0/3357530832 >> 172.21.15.167:0/3357530832 conn(0x562220b40360 msgr2=0x562220b3e4b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-05T19:28:38.499 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.494+0000 7f2486d1ce40 1 -- 172.21.15.167:0/3357530832 shutdown_connections 2023-10-05T19:28:38.499 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.494+0000 7f2486d1ce40 1 -- 172.21.15.167:0/3357530832 wait complete. 2023-10-05T19:28:38.500 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.494+0000 7f2486d1ce40 1 Processor -- start 2023-10-05T19:28:38.500 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.494+0000 7f2486d1ce40 1 -- start start 2023-10-05T19:28:38.500 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.494+0000 7f2486d1ce40 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x562220b44fa0 0x562220b4aba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-05T19:28:38.500 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.494+0000 7f2486d1ce40 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x562220b4bba0 con 0x562220b44fa0 2023-10-05T19:28:38.500 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.494+0000 7f2484050700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x562220b44fa0 0x562220b4aba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-05T19:28:38.500 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.494+0000 7f2484050700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x562220b44fa0 0x562220b4aba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.118:3300/0 says I am v2:172.21.15.167:42224/0 (socket says 172.21.15.167:42224) 2023-10-05T19:28:38.500 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.494+0000 7f2484050700 1 -- 172.21.15.167:0/1153337789 learned_addr learned my addr 172.21.15.167:0/1153337789 (peer_addr_for_me v2:172.21.15.167:0/0) 2023-10-05T19:28:38.501 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.494+0000 7f2484050700 1 -- 172.21.15.167:0/1153337789 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x562220a43bd0 con 0x562220b44fa0 2023-10-05T19:28:38.501 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.494+0000 7f2484050700 1 --2- 172.21.15.167:0/1153337789 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x562220b44fa0 0x562220b4aba0 secure :-1 s=READY pgs=499 cs=0 l=1 rev1=1 crypto rx=0x7f246c015fd0 tx=0x7f246c03c000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-05T19:28:38.502 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.494+0000 7f23c7fff700 1 -- 172.21.15.167:0/1153337789 <== mon.0 v2:172.21.15.118:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f246c02c070 con 0x562220b44fa0 2023-10-05T19:28:38.502 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.494+0000 7f23c7fff700 1 -- 172.21.15.167:0/1153337789 <== mon.0 v2:172.21.15.118:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f246c01c040 con 0x562220b44fa0 2023-10-05T19:28:38.502 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.494+0000 7f2486d1ce40 1 -- 172.21.15.167:0/1153337789 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x562220b4b1a0 con 0x562220b44fa0 2023-10-05T19:28:38.502 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.494+0000 7f23c7fff700 1 -- 172.21.15.167:0/1153337789 <== mon.0 v2:172.21.15.118:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f246c04a5f0 con 0x562220b44fa0 2023-10-05T19:28:38.502 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.494+0000 7f2486d1ce40 1 -- 172.21.15.167:0/1153337789 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x562220b4b580 con 0x562220b44fa0 2023-10-05T19:28:38.502 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.498+0000 7f23c7fff700 1 -- 172.21.15.167:0/1153337789 <== mon.0 v2:172.21.15.118:3300/0 4 ==== mgrmap(e 5) v1 ==== 40329+0+0 (secure 0 0 0) 0x7f246c011410 con 0x562220b44fa0 2023-10-05T19:28:38.503 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.498+0000 7f23c7fff700 1 --2- 172.21.15.167:0/1153337789 >> [v2:172.21.15.167:6824/29889,v1:172.21.15.167:6825/29889] conn(0x7f23cc030c50 0x7f23cc033100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-05T19:28:38.503 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.498+0000 7f23c7fff700 1 -- 172.21.15.167:0/1153337789 <== mon.0 v2:172.21.15.118:3300/0 5 ==== osd_map(30..30 src has 1..30) v4 ==== 6057+0+0 (secure 0 0 0) 0x7f246c008110 con 0x562220b44fa0 2023-10-05T19:28:38.503 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.498+0000 7f2486d1ce40 5 note: GC not initialized 2023-10-05T19:28:38.503 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.498+0000 7f2485052700 1 --2- 172.21.15.167:0/1153337789 >> [v2:172.21.15.167:6824/29889,v1:172.21.15.167:6825/29889] conn(0x7f23cc030c50 0x7f23cc033100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-05T19:28:38.505 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.498+0000 7f2485052700 1 --2- 172.21.15.167:0/1153337789 >> [v2:172.21.15.167:6824/29889,v1:172.21.15.167:6825/29889] conn(0x7f23cc030c50 0x7f23cc033100 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x562220b3a840 tx=0x7f2474000f90 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-05T19:28:38.506 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.498+0000 7f23c57fa700 20 reqs_thread_entry: start 2023-10-05T19:28:38.507 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.502+0000 7f23beffd700 20 reqs_thread_entry: start 2023-10-05T19:28:38.507 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.502+0000 7f2486d1ce40 20 init_complete bucket index max shards: 11 2023-10-05T19:28:38.507 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.502+0000 7f2486d1ce40 10 rgw notify: Started notification manager with: 1 workers 2023-10-05T19:28:38.507 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.502+0000 7f23bd7fa700 20 rgw notify: INFO: next queues processing will happen at: Thu Oct 5 19:29:08 2023 2023-10-05T19:28:38.507 INFO:teuthology.orchestra.run.smithi167.stderr: 2023-10-05T19:28:38.507 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.502+0000 7f23bcff9700 20 reqs_thread_entry: start 2023-10-05T19:28:38.507 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.502+0000 7f2486d1ce40 1 --2- 172.21.15.167:0/1153337789 >> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] conn(0x562220bb21b0 0x562220bb4590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-05T19:28:38.507 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.502+0000 7f2486d1ce40 1 -- 172.21.15.167:0/1153337789 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:1 6.19 6:9cc6e5ad:::2023-10-05-19-9a9df965-6612-45d4-a45f-6e3c889bacc2.4223.3-myfoo3:head [delete] snapc 0=[] ondisk+write+peerstat_old+known_if_redirected+supports_pool_eio e30) v8 -- 0x562220bb4ad0 con 0x562220bb21b0 2023-10-05T19:28:38.508 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.502+0000 7f2484851700 1 --2- 172.21.15.167:0/1153337789 >> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] conn(0x562220bb21b0 0x562220bb4590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-05T19:28:38.508 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.502+0000 7f2484851700 1 --2- 172.21.15.167:0/1153337789 >> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] conn(0x562220bb21b0 0x562220bb4590 crc :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-05T19:28:38.510 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.506+0000 7f2484851700 1 -- 172.21.15.167:0/1153337789 <== osd.1 v2:172.21.15.167:6800/30035 1 ==== osd_op_reply(1 2023-10-05-19-9a9df965-6612-45d4-a45f-6e3c889bacc2.4223.3-myfoo3 [delete] v30'29 uv29 ondisk = 0) v8 ==== 208+0+0 (crc 0 0 0) 0x7f247c051040 con 0x562220bb21b0 2023-10-05T19:28:38.511 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.506+0000 7f2486d1ce40 1 -- 172.21.15.167:0/2512362784 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:102 7.1f 7:f95f44c2:::notify.0:head [watch unwatch cookie 94704577477744] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x562220b712f0 con 0x7f247c007030 2023-10-05T19:28:38.513 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.506+0000 7f2484050700 1 -- 172.21.15.167:0/2512362784 <== osd.2 v2:172.21.15.167:6816/30037 17 ==== osd_op_reply(102 notify.0 [watch unwatch cookie 94704577477744] v30'182 uv180 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f246c045040 con 0x7f247c007030 2023-10-05T19:28:38.514 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.506+0000 7f2486d1ce40 20 remove_watcher() i=0 2023-10-05T19:28:38.514 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.506+0000 7f2486d1ce40 2 removed watcher, disabling cache 2023-10-05T19:28:38.514 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.506+0000 7f2486d1ce40 1 -- 172.21.15.167:0/2512362784 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:103 7.0 7:05bf5b68:::notify.1:head [watch unwatch cookie 94704577488448] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x562220b2fc70 con 0x562220af7740 2023-10-05T19:28:38.516 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.510+0000 7f2485052700 1 -- 172.21.15.167:0/2512362784 <== osd.1 v2:172.21.15.167:6800/30035 35 ==== osd_op_reply(103 notify.1 [watch unwatch cookie 94704577488448] v30'182 uv180 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f2474050040 con 0x562220af7740 2023-10-05T19:28:38.517 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.510+0000 7f2486d1ce40 20 remove_watcher() i=1 2023-10-05T19:28:38.517 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.510+0000 7f2486d1ce40 1 -- 172.21.15.167:0/2512362784 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:104 7.15 7:a93a5511:::notify.2:head [watch unwatch cookie 94704577490816] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x562220b2fc70 con 0x562220b2c3b0 2023-10-05T19:28:38.520 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.514+0000 7f2484851700 1 -- 172.21.15.167:0/2512362784 <== osd.3 v2:172.21.15.167:6808/30036 24 ==== osd_op_reply(104 notify.2 [watch unwatch cookie 94704577490816] v30'182 uv180 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f247c051040 con 0x562220b2c3b0 2023-10-05T19:28:38.520 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.514+0000 7f2486d1ce40 20 remove_watcher() i=2 2023-10-05T19:28:38.520 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.514+0000 7f2486d1ce40 1 -- 172.21.15.167:0/2512362784 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:105 7.e 7:7759931f:::notify.3:head [watch unwatch cookie 94704577497120] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x562220b32f80 con 0x7f247c007030 2023-10-05T19:28:38.522 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.518+0000 7f2484050700 1 -- 172.21.15.167:0/2512362784 <== osd.2 v2:172.21.15.167:6816/30037 18 ==== osd_op_reply(105 notify.3 [watch unwatch cookie 94704577497120] v30'182 uv180 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f246c045040 con 0x7f247c007030 2023-10-05T19:28:38.522 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.518+0000 7f2486d1ce40 20 remove_watcher() i=3 2023-10-05T19:28:38.522 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.518+0000 7f2486d1ce40 1 -- 172.21.15.167:0/2512362784 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:106 7.d 7:b4812045:::notify.4:head [watch unwatch cookie 94704577498224] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x562220b2fc70 con 0x562220af7740 2023-10-05T19:28:38.525 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.518+0000 7f2485052700 1 -- 172.21.15.167:0/2512362784 <== osd.1 v2:172.21.15.167:6800/30035 36 ==== osd_op_reply(106 notify.4 [watch unwatch cookie 94704577498224] v30'182 uv180 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f2474050040 con 0x562220af7740 2023-10-05T19:28:38.525 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.518+0000 7f2486d1ce40 20 remove_watcher() i=4 2023-10-05T19:28:38.525 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.518+0000 7f2486d1ce40 1 -- 172.21.15.167:0/2512362784 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:107 7.3 7:c609908c:::notify.5:head [watch unwatch cookie 94704577493968] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x562220b32f80 con 0x7f246c011040 2023-10-05T19:28:38.528 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.522+0000 7f2485052700 1 -- 172.21.15.167:0/2512362784 <== osd.0 v2:172.21.15.118:6800/27690 29 ==== osd_op_reply(107 notify.5 [watch unwatch cookie 94704577493968] v30'182 uv180 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f2474050040 con 0x7f246c011040 2023-10-05T19:28:38.528 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.522+0000 7f2486d1ce40 20 remove_watcher() i=5 2023-10-05T19:28:38.528 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.522+0000 7f2486d1ce40 1 -- 172.21.15.167:0/2512362784 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:108 7.14 7:2b04a3e9:::notify.6:head [watch unwatch cookie 94704577502688] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x562220b2fc70 con 0x7f246c011040 2023-10-05T19:28:38.532 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.526+0000 7f2485052700 1 -- 172.21.15.167:0/2512362784 <== osd.0 v2:172.21.15.118:6800/27690 30 ==== osd_op_reply(108 notify.6 [watch unwatch cookie 94704577502688] v30'182 uv180 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f2474050040 con 0x7f246c011040 2023-10-05T19:28:38.532 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.526+0000 7f2486d1ce40 20 remove_watcher() i=6 2023-10-05T19:28:38.532 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.526+0000 7f2486d1ce40 1 -- 172.21.15.167:0/2512362784 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:109 7.9 7:93e5b521:::notify.7:head [watch unwatch cookie 94704577505408] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x562220b33bd0 con 0x7f246c011040 2023-10-05T19:28:38.535 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.530+0000 7f2485052700 1 -- 172.21.15.167:0/2512362784 <== osd.0 v2:172.21.15.118:6800/27690 31 ==== osd_op_reply(109 notify.7 [watch unwatch cookie 94704577505408] v30'182 uv180 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f2474050040 con 0x7f246c011040 2023-10-05T19:28:38.535 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.530+0000 7f2486d1ce40 20 remove_watcher() i=7 2023-10-05T19:28:38.537 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.530+0000 7f24567e4700 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2023-10-05T19:28:38.537 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.530+0000 7f2486d1ce40 1 -- 172.21.15.167:0/2512362784 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:110 6.4 6:22d26bf9:::data_loggenerations_metadata:head [watch unwatch cookie 94704577507712] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x562220b73f60 con 0x562220af7740 2023-10-05T19:28:38.541 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.534+0000 7f2485052700 1 -- 172.21.15.167:0/2512362784 <== osd.1 v2:172.21.15.167:6800/30035 37 ==== osd_op_reply(110 data_loggenerations_metadata [watch unwatch cookie 94704577507712] v30'168 uv1 ondisk = 0) v8 ==== 172+0+0 (crc 0 0 0) 0x7f2474050040 con 0x562220af7740 2023-10-05T19:28:38.541 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.534+0000 7f2486d1ce40 1 -- 172.21.15.167:0/2512362784 >> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] conn(0x7f246c011040 msgr2=0x7f246c014950 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-05T19:28:38.541 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.534+0000 7f2486d1ce40 1 --2- 172.21.15.167:0/2512362784 >> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] conn(0x7f246c011040 0x7f246c014950 crc :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-05T19:28:38.541 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.534+0000 7f2486d1ce40 1 -- 172.21.15.167:0/2512362784 >> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] conn(0x562220af7740 msgr2=0x562220b285d0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-05T19:28:38.541 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.534+0000 7f2486d1ce40 1 --2- 172.21.15.167:0/2512362784 >> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] conn(0x562220af7740 0x562220b285d0 crc :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-05T19:28:38.541 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.534+0000 7f2486d1ce40 1 -- 172.21.15.167:0/2512362784 >> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] conn(0x7f247c007030 msgr2=0x7f247c01c410 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-05T19:28:38.541 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.534+0000 7f2486d1ce40 1 --2- 172.21.15.167:0/2512362784 >> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] conn(0x7f247c007030 0x7f247c01c410 crc :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-05T19:28:38.541 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.534+0000 7f2486d1ce40 1 -- 172.21.15.167:0/2512362784 >> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] conn(0x562220b2c3b0 msgr2=0x562220b2e790 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-05T19:28:38.541 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.534+0000 7f2486d1ce40 1 --2- 172.21.15.167:0/2512362784 >> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] conn(0x562220b2c3b0 0x562220b2e790 crc :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-05T19:28:38.542 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.534+0000 7f2486d1ce40 1 -- 172.21.15.167:0/2512362784 >> [v2:172.21.15.167:6824/29889,v1:172.21.15.167:6825/29889] conn(0x7f2464030c50 msgr2=0x7f2464033100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-05T19:28:38.542 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.534+0000 7f2486d1ce40 1 --2- 172.21.15.167:0/2512362784 >> [v2:172.21.15.167:6824/29889,v1:172.21.15.167:6825/29889] conn(0x7f2464030c50 0x7f2464033100 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7f246c0016f0 tx=0x7f246c0067f0 comp rx=0 tx=0).stop 2023-10-05T19:28:38.542 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.534+0000 7f2485052700 1 -- 172.21.15.167:0/2512362784 reap_dead start 2023-10-05T19:28:38.542 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.534+0000 7f2486d1ce40 1 -- 172.21.15.167:0/2512362784 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x562220af6e30 msgr2=0x562220b1cb00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-05T19:28:38.542 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.534+0000 7f2486d1ce40 1 --2- 172.21.15.167:0/2512362784 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x562220af6e30 0x562220b1cb00 secure :-1 s=READY pgs=497 cs=0 l=1 rev1=1 crypto rx=0x7f247c005540 tx=0x7f247c003050 comp rx=0 tx=0).stop 2023-10-05T19:28:38.542 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.534+0000 7f2486d1ce40 1 -- 172.21.15.167:0/2512362784 shutdown_connections 2023-10-05T19:28:38.542 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.534+0000 7f2486d1ce40 1 --2- 172.21.15.167:0/2512362784 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x562220af6e30 0x562220b1cb00 unknown :-1 s=CLOSED pgs=497 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-05T19:28:38.542 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.534+0000 7f2486d1ce40 1 -- 172.21.15.167:0/2512362784 >> 172.21.15.167:0/2512362784 conn(0x562220af2850 msgr2=0x562220b10530 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-05T19:28:38.542 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.534+0000 7f2486d1ce40 1 -- 172.21.15.167:0/2512362784 shutdown_connections 2023-10-05T19:28:38.542 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.534+0000 7f2486d1ce40 1 -- 172.21.15.167:0/2512362784 wait complete. 2023-10-05T19:28:38.543 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.534+0000 7f2486d1ce40 1 -- 172.21.15.167:0/1153337789 >> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] conn(0x562220bb21b0 msgr2=0x562220bb4590 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-05T19:28:38.543 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.534+0000 7f2486d1ce40 1 --2- 172.21.15.167:0/1153337789 >> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] conn(0x562220bb21b0 0x562220bb4590 crc :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-05T19:28:38.544 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.534+0000 7f2486d1ce40 1 -- 172.21.15.167:0/1153337789 >> [v2:172.21.15.167:6824/29889,v1:172.21.15.167:6825/29889] conn(0x7f23cc030c50 msgr2=0x7f23cc033100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-05T19:28:38.544 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.534+0000 7f2486d1ce40 1 --2- 172.21.15.167:0/1153337789 >> [v2:172.21.15.167:6824/29889,v1:172.21.15.167:6825/29889] conn(0x7f23cc030c50 0x7f23cc033100 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x562220b3a840 tx=0x7f2474000f90 comp rx=0 tx=0).stop 2023-10-05T19:28:38.544 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.534+0000 7f2486d1ce40 1 -- 172.21.15.167:0/1153337789 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x562220b44fa0 msgr2=0x562220b4aba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-05T19:28:38.544 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.534+0000 7f2486d1ce40 1 --2- 172.21.15.167:0/1153337789 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x562220b44fa0 0x562220b4aba0 secure :-1 s=READY pgs=499 cs=0 l=1 rev1=1 crypto rx=0x7f246c015fd0 tx=0x7f246c03c000 comp rx=0 tx=0).stop 2023-10-05T19:28:38.544 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.534+0000 7f2486d1ce40 1 -- 172.21.15.167:0/1153337789 shutdown_connections 2023-10-05T19:28:38.544 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.534+0000 7f2486d1ce40 1 --2- 172.21.15.167:0/1153337789 >> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] conn(0x562220bb21b0 0x562220bb4590 unknown :-1 s=CLOSED pgs=83 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-05T19:28:38.544 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.534+0000 7f2486d1ce40 1 --2- 172.21.15.167:0/1153337789 >> [v2:172.21.15.167:6824/29889,v1:172.21.15.167:6825/29889] conn(0x7f23cc030c50 0x7f23cc033100 unknown :-1 s=CLOSED pgs=173 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-05T19:28:38.544 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.534+0000 7f2486d1ce40 1 --2- 172.21.15.167:0/1153337789 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x562220b44fa0 0x562220b4aba0 unknown :-1 s=CLOSED pgs=499 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-05T19:28:38.544 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.534+0000 7f2486d1ce40 1 -- 172.21.15.167:0/1153337789 >> 172.21.15.167:0/1153337789 conn(0x562220b40360 msgr2=0x562220b3e400 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-05T19:28:38.544 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.534+0000 7f2486d1ce40 1 -- 172.21.15.167:0/1153337789 shutdown_connections 2023-10-05T19:28:38.544 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.538+0000 7f2486d1ce40 1 -- 172.21.15.167:0/1153337789 wait complete. 2023-10-05T19:28:38.548 INFO:tasks.util.rgw:rgwadmin: client.0 : ['user', 'suspend', '--uid', 'foo'] 2023-10-05T19:28:38.548 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'ceph', 'user', 'suspend', '--uid', 'foo'] 2023-10-05T19:28:38.548 DEBUG:teuthology.orchestra.run.smithi167:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster ceph user suspend --uid foo 2023-10-05T19:28:38.582 INFO:teuthology.orchestra.run.smithi167.stderr:ignoring --setuser ceph since I am not root 2023-10-05T19:28:38.583 INFO:teuthology.orchestra.run.smithi167.stderr:ignoring --setgroup ceph since I am not root 2023-10-05T19:28:38.583 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.578+0000 7f3f612dde40 1 Processor -- start 2023-10-05T19:28:38.584 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.578+0000 7f3f612dde40 1 -- start start 2023-10-05T19:28:38.584 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.578+0000 7f3f612dde40 1 --2- >> v2:172.21.15.118:3300/0 conn(0x55fa9948ae30 0x55fa9948b200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-05T19:28:38.584 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.578+0000 7f3f612dde40 1 -- --> v1:172.21.15.118:6789/0 -- auth(proto 0 26 bytes epoch 0) v1 -- 0x55fa993d7bd0 con 0x55fa9948b740 2023-10-05T19:28:38.584 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.578+0000 7f3f612dde40 1 -- --> v2:172.21.15.118:3300/0 -- mon_getmap magic: 0 v1 -- 0x55fa99394ee0 con 0x55fa9948ae30 2023-10-05T19:28:38.584 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.578+0000 7f3f5ee12700 1 --2- >> v2:172.21.15.118:3300/0 conn(0x55fa9948ae30 0x55fa9948b200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-05T19:28:38.584 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.578+0000 7f3f5e611700 1 --1- >> v1:172.21.15.118:6789/0 conn(0x55fa9948b740 0x55fa9948c1e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.118:6789/0 says I am v1:172.21.15.167:59400/0 (socket says 172.21.15.167:59400) 2023-10-05T19:28:38.585 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.578+0000 7f3f5e611700 1 -- 172.21.15.167:0/2529755205 learned_addr learned my addr 172.21.15.167:0/2529755205 (peer_addr_for_me v1:172.21.15.167:0/0) 2023-10-05T19:28:38.585 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.578+0000 7f3f5ee12700 1 -- 172.21.15.167:0/2529755205 >> v1:172.21.15.118:6789/0 conn(0x55fa9948b740 legacy=0x55fa9948c1e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-05T19:28:38.585 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.578+0000 7f3f5ee12700 1 -- 172.21.15.167:0/2529755205 --> v2:172.21.15.118:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55fa9948c8f0 con 0x55fa9948ae30 2023-10-05T19:28:38.586 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.578+0000 7f3f5ee12700 1 --2- 172.21.15.167:0/2529755205 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55fa9948ae30 0x55fa9948b200 secure :-1 s=READY pgs=501 cs=0 l=1 rev1=1 crypto rx=0x7f3f4800a520 tx=0x7f3f48005d40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=168fb8cb911be61e server_cookie=0 in_seq=0 out_seq=0 2023-10-05T19:28:38.586 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.578+0000 7f3f5de10700 1 -- 172.21.15.167:0/2529755205 <== mon.0 v2:172.21.15.118:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f3f4800f590 con 0x55fa9948ae30 2023-10-05T19:28:38.586 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.578+0000 7f3f5de10700 1 -- 172.21.15.167:0/2529755205 <== mon.0 v2:172.21.15.118:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3f4800f6f0 con 0x55fa9948ae30 2023-10-05T19:28:38.586 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.582+0000 7f3f5de10700 1 -- 172.21.15.167:0/2529755205 <== mon.0 v2:172.21.15.118:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f3f4800f880 con 0x55fa9948ae30 2023-10-05T19:28:38.587 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.582+0000 7f3f612dde40 1 -- 172.21.15.167:0/2529755205 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55fa9948ae30 msgr2=0x55fa9948b200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-05T19:28:38.587 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.582+0000 7f3f612dde40 1 --2- 172.21.15.167:0/2529755205 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55fa9948ae30 0x55fa9948b200 secure :-1 s=READY pgs=501 cs=0 l=1 rev1=1 crypto rx=0x7f3f4800a520 tx=0x7f3f48005d40 comp rx=0 tx=0).stop 2023-10-05T19:28:38.587 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.582+0000 7f3f612dde40 1 -- 172.21.15.167:0/2529755205 shutdown_connections 2023-10-05T19:28:38.587 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.582+0000 7f3f612dde40 1 --2- 172.21.15.167:0/2529755205 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55fa9948ae30 0x55fa9948b200 unknown :-1 s=CLOSED pgs=501 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-05T19:28:38.587 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.582+0000 7f3f612dde40 1 -- 172.21.15.167:0/2529755205 >> 172.21.15.167:0/2529755205 conn(0x55fa99486850 msgr2=0x55fa99488c60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-05T19:28:38.587 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.582+0000 7f3f612dde40 1 -- 172.21.15.167:0/2529755205 shutdown_connections 2023-10-05T19:28:38.587 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.582+0000 7f3f612dde40 1 -- 172.21.15.167:0/2529755205 wait complete. 2023-10-05T19:28:38.588 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.582+0000 7f3f612dde40 1 Processor -- start 2023-10-05T19:28:38.588 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.582+0000 7f3f612dde40 1 -- start start 2023-10-05T19:28:38.589 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.582+0000 7f3f612dde40 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55fa9948ae30 0x55fa994a9e60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-05T19:28:38.589 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.582+0000 7f3f612dde40 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x55fa99358e30 con 0x55fa9948ae30 2023-10-05T19:28:38.589 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.582+0000 7f3f5ee12700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55fa9948ae30 0x55fa994a9e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-05T19:28:38.589 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.582+0000 7f3f5ee12700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55fa9948ae30 0x55fa994a9e60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.118:3300/0 says I am v2:172.21.15.167:42242/0 (socket says 172.21.15.167:42242) 2023-10-05T19:28:38.589 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.582+0000 7f3f5ee12700 1 -- 172.21.15.167:0/986446791 learned_addr learned my addr 172.21.15.167:0/986446791 (peer_addr_for_me v2:172.21.15.167:0/0) 2023-10-05T19:28:38.590 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.582+0000 7f3f5ee12700 1 -- 172.21.15.167:0/986446791 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3f4800a0e0 con 0x55fa9948ae30 2023-10-05T19:28:38.590 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.582+0000 7f3f5ee12700 1 --2- 172.21.15.167:0/986446791 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55fa9948ae30 0x55fa994a9e60 secure :-1 s=READY pgs=502 cs=0 l=1 rev1=1 crypto rx=0x7f3f48005b20 tx=0x7f3f48014000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6d5ebd161826a2ca server_cookie=0 in_seq=0 out_seq=0 2023-10-05T19:28:38.590 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.582+0000 7f3f5ce0e700 1 -- 172.21.15.167:0/986446791 <== mon.0 v2:172.21.15.118:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f3f4800f860 con 0x55fa9948ae30 2023-10-05T19:28:38.591 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.582+0000 7f3f5ce0e700 1 -- 172.21.15.167:0/986446791 <== mon.0 v2:172.21.15.118:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3f4800f9c0 con 0x55fa9948ae30 2023-10-05T19:28:38.591 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.582+0000 7f3f5ce0e700 1 -- 172.21.15.167:0/986446791 <== mon.0 v2:172.21.15.118:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f3f4800fb30 con 0x55fa9948ae30 2023-10-05T19:28:38.591 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.586+0000 7f3f612dde40 1 -- 172.21.15.167:0/986446791 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55fa9948ae30 msgr2=0x55fa994a9e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-05T19:28:38.591 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.586+0000 7f3f612dde40 1 --2- 172.21.15.167:0/986446791 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55fa9948ae30 0x55fa994a9e60 secure :-1 s=READY pgs=502 cs=0 l=1 rev1=1 crypto rx=0x7f3f48005b20 tx=0x7f3f48014000 comp rx=0 tx=0).stop 2023-10-05T19:28:38.591 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.586+0000 7f3f612dde40 1 -- 172.21.15.167:0/986446791 shutdown_connections 2023-10-05T19:28:38.591 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.586+0000 7f3f612dde40 1 --2- 172.21.15.167:0/986446791 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55fa9948ae30 0x55fa994a9e60 unknown :-1 s=CLOSED pgs=502 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-05T19:28:38.591 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.586+0000 7f3f612dde40 1 -- 172.21.15.167:0/986446791 >> 172.21.15.167:0/986446791 conn(0x55fa99486850 msgr2=0x55fa994a5a20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-05T19:28:38.591 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.586+0000 7f3f612dde40 1 -- 172.21.15.167:0/986446791 shutdown_connections 2023-10-05T19:28:38.591 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.586+0000 7f3f612dde40 1 -- 172.21.15.167:0/986446791 wait complete. 2023-10-05T19:28:38.592 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.586+0000 7f3f612dde40 1 Processor -- start 2023-10-05T19:28:38.593 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.586+0000 7f3f612dde40 1 -- start start 2023-10-05T19:28:38.593 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.586+0000 7f3f612dde40 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55fa9948b740 0x55fa994b0c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-05T19:28:38.593 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.586+0000 7f3f612dde40 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x55fa9932bed0 con 0x55fa9948b740 2023-10-05T19:28:38.593 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.586+0000 7f3f5ee12700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55fa9948b740 0x55fa994b0c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-05T19:28:38.593 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.586+0000 7f3f5ee12700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55fa9948b740 0x55fa994b0c90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.118:3300/0 says I am v2:172.21.15.167:42250/0 (socket says 172.21.15.167:42250) 2023-10-05T19:28:38.593 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.586+0000 7f3f5ee12700 1 -- 172.21.15.167:0/1639253938 learned_addr learned my addr 172.21.15.167:0/1639253938 (peer_addr_for_me v2:172.21.15.167:0/0) 2023-10-05T19:28:38.593 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.586+0000 7f3f5ee12700 1 -- 172.21.15.167:0/1639253938 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55fa9948c8f0 con 0x55fa9948b740 2023-10-05T19:28:38.593 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.586+0000 7f3f5ee12700 1 --2- 172.21.15.167:0/1639253938 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55fa9948b740 0x55fa994b0c90 secure :-1 s=READY pgs=503 cs=0 l=1 rev1=1 crypto rx=0x7f3f480058c0 tx=0x7f3f4801d000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-05T19:28:38.594 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.586+0000 7f3f56ffd700 1 -- 172.21.15.167:0/1639253938 <== mon.0 v2:172.21.15.118:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f3f4802a070 con 0x55fa9948b740 2023-10-05T19:28:38.594 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.586+0000 7f3f612dde40 1 -- 172.21.15.167:0/1639253938 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x55fa994b11d0 con 0x55fa9948b740 2023-10-05T19:28:38.594 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.586+0000 7f3f56ffd700 1 -- 172.21.15.167:0/1639253938 <== mon.0 v2:172.21.15.118:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3f48010420 con 0x55fa9948b740 2023-10-05T19:28:38.594 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.586+0000 7f3f56ffd700 1 -- 172.21.15.167:0/1639253938 <== mon.0 v2:172.21.15.118:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f3f480105d0 con 0x55fa9948b740 2023-10-05T19:28:38.594 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.586+0000 7f3f612dde40 1 -- 172.21.15.167:0/1639253938 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55fa994b15b0 con 0x55fa9948b740 2023-10-05T19:28:38.595 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.590+0000 7f3f612dde40 20 rados->read ofs=0 len=0 2023-10-05T19:28:38.595 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.590+0000 7f3f56ffd700 1 -- 172.21.15.167:0/1639253938 <== mon.0 v2:172.21.15.118:3300/0 4 ==== mgrmap(e 5) v1 ==== 40329+0+0 (secure 0 0 0) 0x7f3f48010800 con 0x55fa9948b740 2023-10-05T19:28:38.595 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.590+0000 7f3f56ffd700 1 --2- 172.21.15.167:0/1639253938 >> [v2:172.21.15.167:6824/29889,v1:172.21.15.167:6825/29889] conn(0x7f3f40030c50 0x7f3f40033100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-05T19:28:38.596 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.590+0000 7f3f56ffd700 1 -- 172.21.15.167:0/1639253938 <== mon.0 v2:172.21.15.118:3300/0 5 ==== osd_map(30..30 src has 1..30) v4 ==== 6057+0+0 (secure 0 0 0) 0x7f3f48046590 con 0x55fa9948b740 2023-10-05T19:28:38.596 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.590+0000 7f3f5e611700 1 --2- 172.21.15.167:0/1639253938 >> [v2:172.21.15.167:6824/29889,v1:172.21.15.167:6825/29889] conn(0x7f3f40030c50 0x7f3f40033100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-05T19:28:38.596 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.590+0000 7f3f612dde40 1 --2- 172.21.15.167:0/1639253938 >> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] conn(0x55fa994ba7a0 0x55fa994bcb80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-05T19:28:38.596 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.590+0000 7f3f612dde40 1 -- 172.21.15.167:0/1639253938 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:1 5.12 5:49953fa1:::default.realm:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x55fa994bd1f0 con 0x55fa994ba7a0 2023-10-05T19:28:38.596 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.590+0000 7f3f5f613700 1 --2- 172.21.15.167:0/1639253938 >> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] conn(0x55fa994ba7a0 0x55fa994bcb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-05T19:28:38.597 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.590+0000 7f3f5e611700 1 --2- 172.21.15.167:0/1639253938 >> [v2:172.21.15.167:6824/29889,v1:172.21.15.167:6825/29889] conn(0x7f3f40030c50 0x7f3f40033100 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7f3f50001e90 tx=0x7f3f500013b0 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-05T19:28:38.597 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.590+0000 7f3f5f613700 1 --2- 172.21.15.167:0/1639253938 >> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] conn(0x55fa994ba7a0 0x55fa994bcb80 crc :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-05T19:28:38.598 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.590+0000 7f3f5f613700 1 -- 172.21.15.167:0/1639253938 <== osd.1 v2:172.21.15.167:6800/30035 1 ==== osd_op_reply(1 default.realm [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 ==== 157+0+0 (crc 0 0 0) 0x7f3f5800a500 con 0x55fa994ba7a0 2023-10-05T19:28:38.598 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.590+0000 7f3f612dde40 20 rados_obj.operate() r=-2 bl.length=0 2023-10-05T19:28:38.598 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.590+0000 7f3f612dde40 20 realm 2023-10-05T19:28:38.598 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.590+0000 7f3f612dde40 20 rados->read ofs=0 len=0 2023-10-05T19:28:38.598 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.590+0000 7f3f612dde40 1 -- 172.21.15.167:0/1639253938 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:2 5.12 5:49953fa1:::default.realm:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x55fa994b9e10 con 0x55fa994ba7a0 2023-10-05T19:28:38.599 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.594+0000 7f3f5f613700 1 -- 172.21.15.167:0/1639253938 <== osd.1 v2:172.21.15.167:6800/30035 2 ==== osd_op_reply(2 default.realm [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 ==== 157+0+0 (crc 0 0 0) 0x7f3f5800a500 con 0x55fa994ba7a0 2023-10-05T19:28:38.599 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.594+0000 7f3f612dde40 20 rados_obj.operate() r=-2 bl.length=0 2023-10-05T19:28:38.599 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.594+0000 7f3f612dde40 4 RGWPeriod::init failed to init realm id : (2) No such file or directory 2023-10-05T19:28:38.599 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.594+0000 7f3f612dde40 20 rados->read ofs=0 len=0 2023-10-05T19:28:38.599 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.594+0000 7f3f612dde40 1 -- 172.21.15.167:0/1639253938 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:3 5.12 5:49953fa1:::default.realm:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x55fa994bdcd0 con 0x55fa994ba7a0 2023-10-05T19:28:38.599 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.594+0000 7f3f5f613700 1 -- 172.21.15.167:0/1639253938 <== osd.1 v2:172.21.15.167:6800/30035 3 ==== osd_op_reply(3 default.realm [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 ==== 157+0+0 (crc 0 0 0) 0x7f3f5800a500 con 0x55fa994ba7a0 2023-10-05T19:28:38.599 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.594+0000 7f3f612dde40 20 rados_obj.operate() r=-2 bl.length=0 2023-10-05T19:28:38.599 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.594+0000 7f3f612dde40 20 rados->read ofs=0 len=0 2023-10-05T19:28:38.600 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.594+0000 7f3f612dde40 1 -- 172.21.15.167:0/1639253938 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:4 5.f 5:f43b6ece:::zone_names.default:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x55fa994bd8e0 con 0x55fa994ba7a0 2023-10-05T19:28:38.601 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.594+0000 7f3f5f613700 1 -- 172.21.15.167:0/1639253938 <== osd.1 v2:172.21.15.167:6800/30035 4 ==== osd_op_reply(4 zone_names.default [read 0~46 out=46b] v0'0 uv1 ondisk = 0) v8 ==== 162+0+46 (crc 0 0 0) 0x7f3f5800a500 con 0x55fa994ba7a0 2023-10-05T19:28:38.601 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.594+0000 7f3f612dde40 20 rados_obj.operate() r=0 bl.length=46 2023-10-05T19:28:38.601 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.594+0000 7f3f612dde40 20 rados->read ofs=0 len=0 2023-10-05T19:28:38.601 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.594+0000 7f3f612dde40 1 -- 172.21.15.167:0/1639253938 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:5 5.1d 5:b8374eb4:::zone_info.9a9df965-6612-45d4-a45f-6e3c889bacc2:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x55fa994beb60 con 0x55fa994ba7a0 2023-10-05T19:28:38.601 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.594+0000 7f3f5f613700 1 -- 172.21.15.167:0/1639253938 <== osd.1 v2:172.21.15.167:6800/30035 5 ==== osd_op_reply(5 zone_info.9a9df965-6612-45d4-a45f-6e3c889bacc2 [read 0~874 out=874b] v0'0 uv2 ondisk = 0) v8 ==== 190+0+874 (crc 0 0 0) 0x7f3f5800a500 con 0x55fa994ba7a0 2023-10-05T19:28:38.601 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.594+0000 7f3f612dde40 20 rados_obj.operate() r=0 bl.length=874 2023-10-05T19:28:38.601 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.594+0000 7f3f612dde40 20 searching for the correct realm 2023-10-05T19:28:38.601 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.594+0000 7f3f612dde40 1 --2- 172.21.15.167:0/1639253938 >> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] conn(0x55fa994c0920 0x55fa994c2d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-05T19:28:38.601 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.594+0000 7f3f612dde40 1 -- 172.21.15.167:0/1639253938 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:6 5.0 5:00000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x55fa994c3240 con 0x55fa994c0920 2023-10-05T19:28:38.602 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.594+0000 7f3f5ee12700 1 --2- 172.21.15.167:0/1639253938 >> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] conn(0x55fa994c0920 0x55fa994c2d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-05T19:28:38.602 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.594+0000 7f3f5ee12700 1 --2- 172.21.15.167:0/1639253938 >> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] conn(0x55fa994c0920 0x55fa994c2d00 crc :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-05T19:28:38.604 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.598+0000 7f3f5ee12700 1 -- 172.21.15.167:0/1639253938 <== osd.3 v2:172.21.15.167:6808/30036 1 ==== osd_op_reply(6 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f3f48037960 con 0x55fa994c0920 2023-10-05T19:28:38.604 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.598+0000 7f3f5ee12700 1 --2- 172.21.15.167:0/1639253938 >> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] conn(0x7f3f48037070 0x7f3f480183b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-05T19:28:38.604 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.598+0000 7f3f5ee12700 1 -- 172.21.15.167:0/1639253938 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:7 5.10 5:08000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x55fa994c3240 con 0x7f3f48037070 2023-10-05T19:28:38.604 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.598+0000 7f3f5e611700 1 --2- 172.21.15.167:0/1639253938 >> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] conn(0x7f3f48037070 0x7f3f480183b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-05T19:28:38.605 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.598+0000 7f3f5e611700 1 --2- 172.21.15.167:0/1639253938 >> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] conn(0x7f3f48037070 0x7f3f480183b0 crc :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-05T19:28:38.606 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.598+0000 7f3f5e611700 1 -- 172.21.15.167:0/1639253938 <== osd.2 v2:172.21.15.167:6816/30037 1 ==== osd_op_reply(7 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f3f50006070 con 0x7f3f48037070 2023-10-05T19:28:38.606 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.598+0000 7f3f5e611700 1 --2- 172.21.15.167:0/1639253938 >> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] conn(0x7f3f5000d8e0 0x7f3f5000c360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-05T19:28:38.606 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.598+0000 7f3f5e611700 1 -- 172.21.15.167:0/1639253938 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:8 5.8 5:10000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x55fa994c3240 con 0x7f3f5000d8e0 2023-10-05T19:28:38.606 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.602+0000 7f3f5f613700 1 --2- 172.21.15.167:0/1639253938 >> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] conn(0x7f3f5000d8e0 0x7f3f5000c360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-05T19:28:38.607 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.602+0000 7f3f5f613700 1 --2- 172.21.15.167:0/1639253938 >> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] conn(0x7f3f5000d8e0 0x7f3f5000c360 crc :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-05T19:28:38.608 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.602+0000 7f3f5f613700 1 -- 172.21.15.167:0/1639253938 <== osd.0 v2:172.21.15.118:6800/27690 1 ==== osd_op_reply(8 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f3f58014040 con 0x7f3f5000d8e0 2023-10-05T19:28:38.608 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.602+0000 7f3f5f613700 1 -- 172.21.15.167:0/1639253938 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:9 5.18 5:18000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x55fa994c3240 con 0x55fa994ba7a0 2023-10-05T19:28:38.608 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.602+0000 7f3f5f613700 1 -- 172.21.15.167:0/1639253938 <== osd.1 v2:172.21.15.167:6800/30035 6 ==== osd_op_reply(9 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f3f58014040 con 0x55fa994ba7a0 2023-10-05T19:28:38.608 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.602+0000 7f3f5f613700 1 -- 172.21.15.167:0/1639253938 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:10 5.4 5:20000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x55fa994c3240 con 0x55fa994ba7a0 2023-10-05T19:28:38.609 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.602+0000 7f3f5f613700 1 -- 172.21.15.167:0/1639253938 <== osd.1 v2:172.21.15.167:6800/30035 7 ==== osd_op_reply(10 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f3f58014040 con 0x55fa994ba7a0 2023-10-05T19:28:38.609 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.602+0000 7f3f5f613700 1 -- 172.21.15.167:0/1639253938 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:11 5.14 5:28000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x55fa994c3240 con 0x7f3f5000d8e0 2023-10-05T19:28:38.610 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.606+0000 7f3f5f613700 1 -- 172.21.15.167:0/1639253938 <== osd.0 v2:172.21.15.118:6800/27690 2 ==== osd_op_reply(11 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f3f58014040 con 0x7f3f5000d8e0 2023-10-05T19:28:38.610 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.606+0000 7f3f5f613700 1 -- 172.21.15.167:0/1639253938 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:12 5.c 5:30000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x55fa994c3240 con 0x7f3f5000d8e0 2023-10-05T19:28:38.611 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.606+0000 7f3f5f613700 1 -- 172.21.15.167:0/1639253938 <== osd.0 v2:172.21.15.118:6800/27690 3 ==== osd_op_reply(12 [pgnls start_epoch 30 out=112b] v23'1 uv1 ondisk = 1) v8 ==== 144+0+112 (crc 0 0 0) 0x7f3f58014040 con 0x7f3f5000d8e0 2023-10-05T19:28:38.611 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.606+0000 7f3f5f613700 1 -- 172.21.15.167:0/1639253938 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:13 5.1c 5:38000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x55fa994c3240 con 0x7f3f48037070 2023-10-05T19:28:38.612 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.606+0000 7f3f5e611700 1 -- 172.21.15.167:0/1639253938 <== osd.2 v2:172.21.15.167:6816/30037 2 ==== osd_op_reply(13 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f3f50006070 con 0x7f3f48037070 2023-10-05T19:28:38.612 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.606+0000 7f3f5e611700 1 -- 172.21.15.167:0/1639253938 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:14 5.2 5:40000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x55fa994c3240 con 0x55fa994c0920 2023-10-05T19:28:38.612 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.606+0000 7f3f5ee12700 1 -- 172.21.15.167:0/1639253938 <== osd.3 v2:172.21.15.167:6808/30036 2 ==== osd_op_reply(14 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f3f48040040 con 0x55fa994c0920 2023-10-05T19:28:38.612 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.606+0000 7f3f5ee12700 1 -- 172.21.15.167:0/1639253938 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:15 5.12 5:48000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x55fa994c3240 con 0x55fa994ba7a0 2023-10-05T19:28:38.613 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.606+0000 7f3f5f613700 1 -- 172.21.15.167:0/1639253938 <== osd.1 v2:172.21.15.167:6800/30035 8 ==== osd_op_reply(15 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f3f58014040 con 0x55fa994ba7a0 2023-10-05T19:28:38.613 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.606+0000 7f3f5f613700 1 -- 172.21.15.167:0/1639253938 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:16 5.a 5:50000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x55fa994c3240 con 0x7f3f5000d8e0 2023-10-05T19:28:38.614 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.610+0000 7f3f5f613700 1 -- 172.21.15.167:0/1639253938 <== osd.0 v2:172.21.15.118:6800/27690 4 ==== osd_op_reply(16 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f3f58022040 con 0x7f3f5000d8e0 2023-10-05T19:28:38.614 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.610+0000 7f3f5f613700 1 -- 172.21.15.167:0/1639253938 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:17 5.1a 5:58000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x55fa994c3240 con 0x55fa994ba7a0 2023-10-05T19:28:38.615 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.610+0000 7f3f5f613700 1 -- 172.21.15.167:0/1639253938 <== osd.1 v2:172.21.15.167:6800/30035 9 ==== osd_op_reply(17 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f3f58022040 con 0x55fa994ba7a0 2023-10-05T19:28:38.615 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.610+0000 7f3f5f613700 1 -- 172.21.15.167:0/1639253938 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:18 5.6 5:60000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x55fa994c3240 con 0x7f3f48037070 2023-10-05T19:28:38.616 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.610+0000 7f3f5e611700 1 -- 172.21.15.167:0/1639253938 <== osd.2 v2:172.21.15.167:6816/30037 3 ==== osd_op_reply(18 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f3f50006070 con 0x7f3f48037070 2023-10-05T19:28:38.616 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.610+0000 7f3f5e611700 1 -- 172.21.15.167:0/1639253938 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:19 5.16 5:68000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x55fa994c3240 con 0x55fa994c0920 2023-10-05T19:28:38.616 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.610+0000 7f3f5ee12700 1 -- 172.21.15.167:0/1639253938 <== osd.3 v2:172.21.15.167:6808/30036 3 ==== osd_op_reply(19 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f3f48040040 con 0x55fa994c0920 2023-10-05T19:28:38.617 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.610+0000 7f3f5ee12700 1 -- 172.21.15.167:0/1639253938 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:20 5.e 5:70000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x55fa994c3240 con 0x55fa994c0920 2023-10-05T19:28:38.617 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.610+0000 7f3f5ee12700 1 -- 172.21.15.167:0/1639253938 <== osd.3 v2:172.21.15.167:6808/30036 4 ==== osd_op_reply(20 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f3f48040040 con 0x55fa994c0920 2023-10-05T19:28:38.617 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.610+0000 7f3f5ee12700 1 -- 172.21.15.167:0/1639253938 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:21 5.1e 5:78000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x55fa994c3240 con 0x7f3f5000d8e0 2023-10-05T19:28:38.618 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.614+0000 7f3f5f613700 1 -- 172.21.15.167:0/1639253938 <== osd.0 v2:172.21.15.118:6800/27690 5 ==== osd_op_reply(21 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f3f58022040 con 0x7f3f5000d8e0 2023-10-05T19:28:38.618 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.614+0000 7f3f5f613700 1 -- 172.21.15.167:0/1639253938 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:22 5.1 5:80000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x55fa994c3240 con 0x55fa994c0920 2023-10-05T19:28:38.619 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.614+0000 7f3f5ee12700 1 -- 172.21.15.167:0/1639253938 <== osd.3 v2:172.21.15.167:6808/30036 5 ==== osd_op_reply(22 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f3f48040040 con 0x55fa994c0920 2023-10-05T19:28:38.619 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.614+0000 7f3f5ee12700 1 -- 172.21.15.167:0/1639253938 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:23 5.11 5:88000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x55fa994c3240 con 0x55fa994c0920 2023-10-05T19:28:38.620 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.614+0000 7f3f5ee12700 1 -- 172.21.15.167:0/1639253938 <== osd.3 v2:172.21.15.167:6808/30036 6 ==== osd_op_reply(23 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f3f48040040 con 0x55fa994c0920 2023-10-05T19:28:38.620 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.614+0000 7f3f5ee12700 1 -- 172.21.15.167:0/1639253938 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:24 5.9 5:90000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x55fa994c3240 con 0x55fa994ba7a0 2023-10-05T19:28:38.621 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.614+0000 7f3f5f613700 1 -- 172.21.15.167:0/1639253938 <== osd.1 v2:172.21.15.167:6800/30035 10 ==== osd_op_reply(24 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f3f58022040 con 0x55fa994ba7a0 2023-10-05T19:28:38.621 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.614+0000 7f3f5f613700 1 -- 172.21.15.167:0/1639253938 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:25 5.19 5:98000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x55fa994c3240 con 0x55fa994ba7a0 2023-10-05T19:28:38.621 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.614+0000 7f3f5f613700 1 -- 172.21.15.167:0/1639253938 <== osd.1 v2:172.21.15.167:6800/30035 11 ==== osd_op_reply(25 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f3f58022040 con 0x55fa994ba7a0 2023-10-05T19:28:38.622 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.614+0000 7f3f5f613700 1 -- 172.21.15.167:0/1639253938 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:26 5.5 5:a0000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x55fa994c3240 con 0x7f3f5000d8e0 2023-10-05T19:28:38.622 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.618+0000 7f3f5f613700 1 -- 172.21.15.167:0/1639253938 <== osd.0 v2:172.21.15.118:6800/27690 6 ==== osd_op_reply(26 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f3f58022040 con 0x7f3f5000d8e0 2023-10-05T19:28:38.622 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.618+0000 7f3f5f613700 1 -- 172.21.15.167:0/1639253938 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:27 5.15 5:a8000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x55fa994c3240 con 0x55fa994c0920 2023-10-05T19:28:38.623 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.618+0000 7f3f5ee12700 1 -- 172.21.15.167:0/1639253938 <== osd.3 v2:172.21.15.167:6808/30036 7 ==== osd_op_reply(27 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f3f48040040 con 0x55fa994c0920 2023-10-05T19:28:38.623 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.618+0000 7f3f5ee12700 1 -- 172.21.15.167:0/1639253938 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:28 5.d 5:b0000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x55fa994c3240 con 0x7f3f48037070 2023-10-05T19:28:38.624 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.618+0000 7f3f5e611700 1 -- 172.21.15.167:0/1639253938 <== osd.2 v2:172.21.15.167:6816/30037 4 ==== osd_op_reply(28 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f3f50006070 con 0x7f3f48037070 2023-10-05T19:28:38.624 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.618+0000 7f3f5e611700 1 -- 172.21.15.167:0/1639253938 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:29 5.1d 5:b8000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x55fa994c3240 con 0x55fa994ba7a0 2023-10-05T19:28:38.625 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.618+0000 7f3f5f613700 1 -- 172.21.15.167:0/1639253938 <== osd.1 v2:172.21.15.167:6800/30035 12 ==== osd_op_reply(29 [pgnls start_epoch 30 out=107b] v30'2 uv2 ondisk = 1) v8 ==== 144+0+107 (crc 0 0 0) 0x7f3f58022040 con 0x55fa994ba7a0 2023-10-05T19:28:38.625 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.618+0000 7f3f5f613700 1 -- 172.21.15.167:0/1639253938 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:30 5.3 5:c0000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x55fa994c3240 con 0x7f3f5000d8e0 2023-10-05T19:28:38.626 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.622+0000 7f3f5f613700 1 -- 172.21.15.167:0/1639253938 <== osd.0 v2:172.21.15.118:6800/27690 7 ==== osd_op_reply(30 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f3f58022040 con 0x7f3f5000d8e0 2023-10-05T19:28:38.626 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.622+0000 7f3f5f613700 1 -- 172.21.15.167:0/1639253938 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:31 5.13 5:c8000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x55fa994c3240 con 0x55fa994c0920 2023-10-05T19:28:38.627 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.622+0000 7f3f5ee12700 1 -- 172.21.15.167:0/1639253938 <== osd.3 v2:172.21.15.167:6808/30036 8 ==== osd_op_reply(31 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f3f48040040 con 0x55fa994c0920 2023-10-05T19:28:38.627 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.622+0000 7f3f5ee12700 1 -- 172.21.15.167:0/1639253938 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:32 5.b 5:d0000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x55fa994c3240 con 0x7f3f5000d8e0 2023-10-05T19:28:38.628 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.622+0000 7f3f5f613700 1 -- 172.21.15.167:0/1639253938 <== osd.0 v2:172.21.15.118:6800/27690 8 ==== osd_op_reply(32 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f3f58022040 con 0x7f3f5000d8e0 2023-10-05T19:28:38.628 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.622+0000 7f3f5f613700 1 -- 172.21.15.167:0/1639253938 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:33 5.1b 5:d8000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x55fa994c3240 con 0x7f3f48037070 2023-10-05T19:28:38.629 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.622+0000 7f3f5e611700 1 -- 172.21.15.167:0/1639253938 <== osd.2 v2:172.21.15.167:6816/30037 5 ==== osd_op_reply(33 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f3f50006070 con 0x7f3f48037070 2023-10-05T19:28:38.629 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.622+0000 7f3f5e611700 1 -- 172.21.15.167:0/1639253938 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:34 5.7 5:e0000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x55fa994c3240 con 0x55fa994c0920 2023-10-05T19:28:38.629 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.622+0000 7f3f5ee12700 1 -- 172.21.15.167:0/1639253938 <== osd.3 v2:172.21.15.167:6808/30036 9 ==== osd_op_reply(34 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f3f48040040 con 0x55fa994c0920 2023-10-05T19:28:38.630 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.622+0000 7f3f5ee12700 1 -- 172.21.15.167:0/1639253938 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:35 5.17 5:e8000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x55fa994c3240 con 0x7f3f5000d8e0 2023-10-05T19:28:38.630 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.626+0000 7f3f5f613700 1 -- 172.21.15.167:0/1639253938 <== osd.0 v2:172.21.15.118:6800/27690 9 ==== osd_op_reply(35 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f3f58022040 con 0x7f3f5000d8e0 2023-10-05T19:28:38.631 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.626+0000 7f3f5f613700 1 -- 172.21.15.167:0/1639253938 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:36 5.f 5:f0000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x55fa994c3240 con 0x55fa994ba7a0 2023-10-05T19:28:38.631 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.626+0000 7f3f5f613700 1 -- 172.21.15.167:0/1639253938 <== osd.1 v2:172.21.15.167:6800/30035 13 ==== osd_op_reply(36 [pgnls start_epoch 30 out=115b] v23'2 uv2 ondisk = 1) v8 ==== 144+0+115 (crc 0 0 0) 0x7f3f5802f080 con 0x55fa994ba7a0 2023-10-05T19:28:38.631 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.626+0000 7f3f5f613700 1 -- 172.21.15.167:0/1639253938 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:37 5.1f 5:f8000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x55fa994c3240 con 0x7f3f48037070 2023-10-05T19:28:38.632 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.626+0000 7f3f5e611700 1 -- 172.21.15.167:0/1639253938 <== osd.2 v2:172.21.15.167:6816/30037 6 ==== osd_op_reply(37 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f3f50006070 con 0x7f3f48037070 2023-10-05T19:28:38.633 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.626+0000 7f3f612dde40 20 RGWRados::pool_iterate: got zonegroup_info.4f1f719d-9648-417b-b10c-e5d939825419 2023-10-05T19:28:38.633 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.626+0000 7f3f612dde40 20 RGWRados::pool_iterate: got zone_info.9a9df965-6612-45d4-a45f-6e3c889bacc2 2023-10-05T19:28:38.633 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.626+0000 7f3f612dde40 20 RGWRados::pool_iterate: got zone_names.default 2023-10-05T19:28:38.633 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.626+0000 7f3f612dde40 20 RGWRados::pool_iterate: got zonegroups_names.default 2023-10-05T19:28:38.633 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.626+0000 7f3f612dde40 20 rados->read ofs=0 len=0 2023-10-05T19:28:38.633 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.626+0000 7f3f612dde40 1 -- 172.21.15.167:0/1639253938 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:38 5.12 5:49953fa1:::default.realm:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x55fa994bf6d0 con 0x55fa994ba7a0 2023-10-05T19:28:38.634 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.626+0000 7f3f5f613700 1 -- 172.21.15.167:0/1639253938 <== osd.1 v2:172.21.15.167:6800/30035 14 ==== osd_op_reply(38 default.realm [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 ==== 157+0+0 (crc 0 0 0) 0x7f3f5802f080 con 0x55fa994ba7a0 2023-10-05T19:28:38.634 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.626+0000 7f3f612dde40 20 rados_obj.operate() r=-2 bl.length=0 2023-10-05T19:28:38.634 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.626+0000 7f3f612dde40 20 rados->read ofs=0 len=0 2023-10-05T19:28:38.634 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.626+0000 7f3f612dde40 1 -- 172.21.15.167:0/1639253938 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:39 5.f 5:f4c53578:::zonegroups_names.default:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x55fa994bffc0 con 0x55fa994ba7a0 2023-10-05T19:28:38.635 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.626+0000 7f3f5f613700 1 -- 172.21.15.167:0/1639253938 <== osd.1 v2:172.21.15.167:6800/30035 15 ==== osd_op_reply(39 zonegroups_names.default [read 0~46 out=46b] v0'0 uv2 ondisk = 0) v8 ==== 168+0+46 (crc 0 0 0) 0x7f3f5802f080 con 0x55fa994ba7a0 2023-10-05T19:28:38.635 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.626+0000 7f3f612dde40 20 rados_obj.operate() r=0 bl.length=46 2023-10-05T19:28:38.635 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.626+0000 7f3f612dde40 20 rados->read ofs=0 len=0 2023-10-05T19:28:38.635 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.630+0000 7f3f612dde40 1 -- 172.21.15.167:0/1639253938 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:40 5.c 5:30d299a5:::zonegroup_info.4f1f719d-9648-417b-b10c-e5d939825419:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x55fa994c3680 con 0x7f3f5000d8e0 2023-10-05T19:28:38.635 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.630+0000 7f3f5f613700 1 -- 172.21.15.167:0/1639253938 <== osd.0 v2:172.21.15.118:6800/27690 10 ==== osd_op_reply(40 zonegroup_info.4f1f719d-9648-417b-b10c-e5d939825419 [read 0~362 out=362b] v0'0 uv1 ondisk = 0) v8 ==== 195+0+362 (crc 0 0 0) 0x7f3f5802f080 con 0x7f3f5000d8e0 2023-10-05T19:28:38.636 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.630+0000 7f3f612dde40 20 rados_obj.operate() r=0 bl.length=362 2023-10-05T19:28:38.636 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.630+0000 7f3f612dde40 20 rados->read ofs=0 len=0 2023-10-05T19:28:38.636 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.630+0000 7f3f612dde40 1 -- 172.21.15.167:0/1639253938 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:41 5.13 5:cbbf6c60:::converted:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x55fa994c3680 con 0x55fa994c0920 2023-10-05T19:28:38.637 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.630+0000 7f3f5ee12700 1 -- 172.21.15.167:0/1639253938 <== osd.3 v2:172.21.15.167:6808/30036 10 ==== osd_op_reply(41 converted [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 ==== 153+0+0 (crc 0 0 0) 0x7f3f48040040 con 0x55fa994c0920 2023-10-05T19:28:38.637 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.630+0000 7f3f612dde40 20 rados_obj.operate() r=-2 bl.length=0 2023-10-05T19:28:38.637 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.630+0000 7f3f612dde40 20 rados->read ofs=0 len=0 2023-10-05T19:28:38.637 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.630+0000 7f3f612dde40 1 -- 172.21.15.167:0/1639253938 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:42 5.12 5:49953fa1:::default.realm:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x55fa994c3a30 con 0x55fa994ba7a0 2023-10-05T19:28:38.637 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.630+0000 7f3f5f613700 1 -- 172.21.15.167:0/1639253938 <== osd.1 v2:172.21.15.167:6800/30035 16 ==== osd_op_reply(42 default.realm [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 ==== 157+0+0 (crc 0 0 0) 0x7f3f5802f080 con 0x55fa994ba7a0 2023-10-05T19:28:38.637 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.630+0000 7f3f612dde40 20 rados_obj.operate() r=-2 bl.length=0 2023-10-05T19:28:38.637 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.630+0000 7f3f612dde40 20 rados->read ofs=0 len=0 2023-10-05T19:28:38.637 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.630+0000 7f3f612dde40 1 -- 172.21.15.167:0/1639253938 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:43 5.f 5:f4c53578:::zonegroups_names.default:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x55fa994c3e30 con 0x55fa994ba7a0 2023-10-05T19:28:38.638 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.630+0000 7f3f5f613700 1 -- 172.21.15.167:0/1639253938 <== osd.1 v2:172.21.15.167:6800/30035 17 ==== osd_op_reply(43 zonegroups_names.default [read 0~46 out=46b] v0'0 uv2 ondisk = 0) v8 ==== 168+0+46 (crc 0 0 0) 0x7f3f5802f080 con 0x55fa994ba7a0 2023-10-05T19:28:38.638 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.630+0000 7f3f612dde40 20 rados_obj.operate() r=0 bl.length=46 2023-10-05T19:28:38.638 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.630+0000 7f3f612dde40 1 -- 172.21.15.167:0/1639253938 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:44 5.0 5:00000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x55fa994c4e40 con 0x55fa994c0920 2023-10-05T19:28:38.640 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.634+0000 7f3f5ee12700 1 -- 172.21.15.167:0/1639253938 <== osd.3 v2:172.21.15.167:6808/30036 11 ==== osd_op_reply(44 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f3f48040040 con 0x55fa994c0920 2023-10-05T19:28:38.640 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.634+0000 7f3f5ee12700 1 -- 172.21.15.167:0/1639253938 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:45 5.10 5:08000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x55fa994c4e40 con 0x7f3f48037070 2023-10-05T19:28:38.641 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.634+0000 7f3f5e611700 1 -- 172.21.15.167:0/1639253938 <== osd.2 v2:172.21.15.167:6816/30037 7 ==== osd_op_reply(45 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f3f50006070 con 0x7f3f48037070 2023-10-05T19:28:38.641 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.634+0000 7f3f5e611700 1 -- 172.21.15.167:0/1639253938 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:46 5.8 5:10000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x55fa994c4e40 con 0x7f3f5000d8e0 2023-10-05T19:28:38.642 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.638+0000 7f3f5f613700 1 -- 172.21.15.167:0/1639253938 <== osd.0 v2:172.21.15.118:6800/27690 11 ==== osd_op_reply(46 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f3f5802f080 con 0x7f3f5000d8e0 2023-10-05T19:28:38.642 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.638+0000 7f3f5f613700 1 -- 172.21.15.167:0/1639253938 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:47 5.18 5:18000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x55fa994bffc0 con 0x55fa994ba7a0 2023-10-05T19:28:38.643 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.638+0000 7f3f5f613700 1 -- 172.21.15.167:0/1639253938 <== osd.1 v2:172.21.15.167:6800/30035 18 ==== osd_op_reply(47 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f3f5802f080 con 0x55fa994ba7a0 2023-10-05T19:28:38.643 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.638+0000 7f3f5f613700 1 -- 172.21.15.167:0/1639253938 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:48 5.4 5:20000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x55fa994bffc0 con 0x55fa994ba7a0 2023-10-05T19:28:38.644 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.638+0000 7f3f5f613700 1 -- 172.21.15.167:0/1639253938 <== osd.1 v2:172.21.15.167:6800/30035 19 ==== osd_op_reply(48 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f3f5802f080 con 0x55fa994ba7a0 2023-10-05T19:28:38.644 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.638+0000 7f3f5f613700 1 -- 172.21.15.167:0/1639253938 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:49 5.14 5:28000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x55fa994bffc0 con 0x7f3f5000d8e0 2023-10-05T19:28:38.645 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.638+0000 7f3f5f613700 1 -- 172.21.15.167:0/1639253938 <== osd.0 v2:172.21.15.118:6800/27690 12 ==== osd_op_reply(49 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f3f5802f080 con 0x7f3f5000d8e0 2023-10-05T19:28:38.645 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.638+0000 7f3f5f613700 1 -- 172.21.15.167:0/1639253938 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:50 5.c 5:30000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x55fa994bffc0 con 0x7f3f5000d8e0 2023-10-05T19:28:38.646 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.642+0000 7f3f5f613700 1 -- 172.21.15.167:0/1639253938 <== osd.0 v2:172.21.15.118:6800/27690 13 ==== osd_op_reply(50 [pgnls start_epoch 30 out=112b] v23'1 uv1 ondisk = 1) v8 ==== 144+0+112 (crc 0 0 0) 0x7f3f5802f080 con 0x7f3f5000d8e0 2023-10-05T19:28:38.646 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.642+0000 7f3f5f613700 1 -- 172.21.15.167:0/1639253938 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:51 5.1c 5:38000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x55fa994bffc0 con 0x7f3f48037070 2023-10-05T19:28:38.647 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.642+0000 7f3f5e611700 1 -- 172.21.15.167:0/1639253938 <== osd.2 v2:172.21.15.167:6816/30037 8 ==== osd_op_reply(51 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f3f50006070 con 0x7f3f48037070 2023-10-05T19:28:38.647 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.642+0000 7f3f5e611700 1 -- 172.21.15.167:0/1639253938 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:52 5.2 5:40000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x55fa994bffc0 con 0x55fa994c0920 2023-10-05T19:28:38.648 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.642+0000 7f3f5ee12700 1 -- 172.21.15.167:0/1639253938 <== osd.3 v2:172.21.15.167:6808/30036 12 ==== osd_op_reply(52 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f3f48040040 con 0x55fa994c0920 2023-10-05T19:28:38.648 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.642+0000 7f3f5ee12700 1 -- 172.21.15.167:0/1639253938 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:53 5.12 5:48000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x55fa994bffc0 con 0x55fa994ba7a0 2023-10-05T19:28:38.649 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.642+0000 7f3f5f613700 1 -- 172.21.15.167:0/1639253938 <== osd.1 v2:172.21.15.167:6800/30035 20 ==== osd_op_reply(53 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f3f5802f080 con 0x55fa994ba7a0 2023-10-05T19:28:38.649 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.642+0000 7f3f5f613700 1 -- 172.21.15.167:0/1639253938 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:54 5.a 5:50000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x55fa994bffc0 con 0x7f3f5000d8e0 2023-10-05T19:28:38.650 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.646+0000 7f3f5f613700 1 -- 172.21.15.167:0/1639253938 <== osd.0 v2:172.21.15.118:6800/27690 14 ==== osd_op_reply(54 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f3f5802f080 con 0x7f3f5000d8e0 2023-10-05T19:28:38.650 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.646+0000 7f3f5f613700 1 -- 172.21.15.167:0/1639253938 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:55 5.1a 5:58000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x55fa994bffc0 con 0x55fa994ba7a0 2023-10-05T19:28:38.650 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.646+0000 7f3f5f613700 1 -- 172.21.15.167:0/1639253938 <== osd.1 v2:172.21.15.167:6800/30035 21 ==== osd_op_reply(55 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f3f5802f080 con 0x55fa994ba7a0 2023-10-05T19:28:38.651 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.646+0000 7f3f5f613700 1 -- 172.21.15.167:0/1639253938 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:56 5.6 5:60000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x55fa994bffc0 con 0x7f3f48037070 2023-10-05T19:28:38.651 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.646+0000 7f3f5e611700 1 -- 172.21.15.167:0/1639253938 <== osd.2 v2:172.21.15.167:6816/30037 9 ==== osd_op_reply(56 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f3f50006070 con 0x7f3f48037070 2023-10-05T19:28:38.652 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.646+0000 7f3f5e611700 1 -- 172.21.15.167:0/1639253938 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:57 5.16 5:68000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x55fa994bffc0 con 0x55fa994c0920 2023-10-05T19:28:38.652 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.646+0000 7f3f5ee12700 1 -- 172.21.15.167:0/1639253938 <== osd.3 v2:172.21.15.167:6808/30036 13 ==== osd_op_reply(57 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f3f48040040 con 0x55fa994c0920 2023-10-05T19:28:38.652 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.646+0000 7f3f5ee12700 1 -- 172.21.15.167:0/1639253938 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:58 5.e 5:70000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x55fa994bffc0 con 0x55fa994c0920 2023-10-05T19:28:38.653 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.646+0000 7f3f5ee12700 1 -- 172.21.15.167:0/1639253938 <== osd.3 v2:172.21.15.167:6808/30036 14 ==== osd_op_reply(58 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f3f48040040 con 0x55fa994c0920 2023-10-05T19:28:38.653 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.646+0000 7f3f5ee12700 1 -- 172.21.15.167:0/1639253938 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:59 5.1e 5:78000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x55fa994bffc0 con 0x7f3f5000d8e0 2023-10-05T19:28:38.654 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.650+0000 7f3f5f613700 1 -- 172.21.15.167:0/1639253938 <== osd.0 v2:172.21.15.118:6800/27690 15 ==== osd_op_reply(59 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f3f5802f080 con 0x7f3f5000d8e0 2023-10-05T19:28:38.654 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.650+0000 7f3f5f613700 1 -- 172.21.15.167:0/1639253938 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:60 5.1 5:80000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x55fa994bffc0 con 0x55fa994c0920 2023-10-05T19:28:38.655 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.650+0000 7f3f5ee12700 1 -- 172.21.15.167:0/1639253938 <== osd.3 v2:172.21.15.167:6808/30036 15 ==== osd_op_reply(60 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f3f48040040 con 0x55fa994c0920 2023-10-05T19:28:38.655 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.650+0000 7f3f5ee12700 1 -- 172.21.15.167:0/1639253938 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:61 5.11 5:88000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x55fa994bffc0 con 0x55fa994c0920 2023-10-05T19:28:38.655 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.650+0000 7f3f5ee12700 1 -- 172.21.15.167:0/1639253938 <== osd.3 v2:172.21.15.167:6808/30036 16 ==== osd_op_reply(61 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f3f48040040 con 0x55fa994c0920 2023-10-05T19:28:38.655 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.650+0000 7f3f5ee12700 1 -- 172.21.15.167:0/1639253938 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:62 5.9 5:90000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x55fa994bffc0 con 0x55fa994ba7a0 2023-10-05T19:28:38.656 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.650+0000 7f3f5f613700 1 -- 172.21.15.167:0/1639253938 <== osd.1 v2:172.21.15.167:6800/30035 22 ==== osd_op_reply(62 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f3f5802f080 con 0x55fa994ba7a0 2023-10-05T19:28:38.656 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.650+0000 7f3f5f613700 1 -- 172.21.15.167:0/1639253938 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:63 5.19 5:98000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x55fa994bffc0 con 0x55fa994ba7a0 2023-10-05T19:28:38.656 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.650+0000 7f3f5f613700 1 -- 172.21.15.167:0/1639253938 <== osd.1 v2:172.21.15.167:6800/30035 23 ==== osd_op_reply(63 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f3f5802f080 con 0x55fa994ba7a0 2023-10-05T19:28:38.657 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.650+0000 7f3f5f613700 1 -- 172.21.15.167:0/1639253938 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:64 5.5 5:a0000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x55fa994bffc0 con 0x7f3f5000d8e0 2023-10-05T19:28:38.657 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.650+0000 7f3f5f613700 1 -- 172.21.15.167:0/1639253938 <== osd.0 v2:172.21.15.118:6800/27690 16 ==== osd_op_reply(64 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f3f5802f080 con 0x7f3f5000d8e0 2023-10-05T19:28:38.658 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.650+0000 7f3f5f613700 1 -- 172.21.15.167:0/1639253938 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:65 5.15 5:a8000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x55fa994bffc0 con 0x55fa994c0920 2023-10-05T19:28:38.658 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.654+0000 7f3f5ee12700 1 -- 172.21.15.167:0/1639253938 <== osd.3 v2:172.21.15.167:6808/30036 17 ==== osd_op_reply(65 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f3f48040040 con 0x55fa994c0920 2023-10-05T19:28:38.658 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.654+0000 7f3f5ee12700 1 -- 172.21.15.167:0/1639253938 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:66 5.d 5:b0000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x55fa994bffc0 con 0x7f3f48037070 2023-10-05T19:28:38.659 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.654+0000 7f3f5e611700 1 -- 172.21.15.167:0/1639253938 <== osd.2 v2:172.21.15.167:6816/30037 10 ==== osd_op_reply(66 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f3f50006070 con 0x7f3f48037070 2023-10-05T19:28:38.659 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.654+0000 7f3f5e611700 1 -- 172.21.15.167:0/1639253938 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:67 5.1d 5:b8000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x55fa994bffc0 con 0x55fa994ba7a0 2023-10-05T19:28:38.660 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.654+0000 7f3f5f613700 1 -- 172.21.15.167:0/1639253938 <== osd.1 v2:172.21.15.167:6800/30035 24 ==== osd_op_reply(67 [pgnls start_epoch 30 out=107b] v30'2 uv2 ondisk = 1) v8 ==== 144+0+107 (crc 0 0 0) 0x7f3f5802f080 con 0x55fa994ba7a0 2023-10-05T19:28:38.660 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.654+0000 7f3f5f613700 1 -- 172.21.15.167:0/1639253938 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:68 5.3 5:c0000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x55fa994bffc0 con 0x7f3f5000d8e0 2023-10-05T19:28:38.661 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.654+0000 7f3f5f613700 1 -- 172.21.15.167:0/1639253938 <== osd.0 v2:172.21.15.118:6800/27690 17 ==== osd_op_reply(68 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f3f5802f080 con 0x7f3f5000d8e0 2023-10-05T19:28:38.661 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.654+0000 7f3f5f613700 1 -- 172.21.15.167:0/1639253938 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:69 5.13 5:c8000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x55fa994bffc0 con 0x55fa994c0920 2023-10-05T19:28:38.662 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.658+0000 7f3f5ee12700 1 -- 172.21.15.167:0/1639253938 <== osd.3 v2:172.21.15.167:6808/30036 18 ==== osd_op_reply(69 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f3f48040040 con 0x55fa994c0920 2023-10-05T19:28:38.662 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.658+0000 7f3f5ee12700 1 -- 172.21.15.167:0/1639253938 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:70 5.b 5:d0000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x55fa994bffc0 con 0x7f3f5000d8e0 2023-10-05T19:28:38.663 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.658+0000 7f3f5f613700 1 -- 172.21.15.167:0/1639253938 <== osd.0 v2:172.21.15.118:6800/27690 18 ==== osd_op_reply(70 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f3f5802f080 con 0x7f3f5000d8e0 2023-10-05T19:28:38.663 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.658+0000 7f3f5f613700 1 -- 172.21.15.167:0/1639253938 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:71 5.1b 5:d8000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x55fa994bffc0 con 0x7f3f48037070 2023-10-05T19:28:38.664 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.658+0000 7f3f5e611700 1 -- 172.21.15.167:0/1639253938 <== osd.2 v2:172.21.15.167:6816/30037 11 ==== osd_op_reply(71 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f3f50006070 con 0x7f3f48037070 2023-10-05T19:28:38.664 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.658+0000 7f3f5e611700 1 -- 172.21.15.167:0/1639253938 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:72 5.7 5:e0000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x55fa994bffc0 con 0x55fa994c0920 2023-10-05T19:28:38.665 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.658+0000 7f3f5ee12700 1 -- 172.21.15.167:0/1639253938 <== osd.3 v2:172.21.15.167:6808/30036 19 ==== osd_op_reply(72 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f3f48040040 con 0x55fa994c0920 2023-10-05T19:28:38.665 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.658+0000 7f3f5ee12700 1 -- 172.21.15.167:0/1639253938 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:73 5.17 5:e8000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x55fa994bffc0 con 0x7f3f5000d8e0 2023-10-05T19:28:38.666 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.658+0000 7f3f5f613700 1 -- 172.21.15.167:0/1639253938 <== osd.0 v2:172.21.15.118:6800/27690 19 ==== osd_op_reply(73 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f3f5802f080 con 0x7f3f5000d8e0 2023-10-05T19:28:38.666 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.658+0000 7f3f5f613700 1 -- 172.21.15.167:0/1639253938 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:74 5.f 5:f0000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x55fa994bffc0 con 0x55fa994ba7a0 2023-10-05T19:28:38.667 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.662+0000 7f3f5f613700 1 -- 172.21.15.167:0/1639253938 <== osd.1 v2:172.21.15.167:6800/30035 25 ==== osd_op_reply(74 [pgnls start_epoch 30 out=115b] v23'2 uv2 ondisk = 1) v8 ==== 144+0+115 (crc 0 0 0) 0x7f3f5802f080 con 0x55fa994ba7a0 2023-10-05T19:28:38.667 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.662+0000 7f3f5f613700 1 -- 172.21.15.167:0/1639253938 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:75 5.1f 5:f8000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x55fa994bffc0 con 0x7f3f48037070 2023-10-05T19:28:38.668 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.662+0000 7f3f5e611700 1 -- 172.21.15.167:0/1639253938 <== osd.2 v2:172.21.15.167:6816/30037 12 ==== osd_op_reply(75 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f3f50006070 con 0x7f3f48037070 2023-10-05T19:28:38.668 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.662+0000 7f3f612dde40 20 RGWRados::pool_iterate: got zonegroup_info.4f1f719d-9648-417b-b10c-e5d939825419 2023-10-05T19:28:38.668 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.662+0000 7f3f612dde40 20 RGWRados::pool_iterate: got zone_info.9a9df965-6612-45d4-a45f-6e3c889bacc2 2023-10-05T19:28:38.668 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.662+0000 7f3f612dde40 20 RGWRados::pool_iterate: got zone_names.default 2023-10-05T19:28:38.668 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.662+0000 7f3f612dde40 20 RGWRados::pool_iterate: got zonegroups_names.default 2023-10-05T19:28:38.668 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.662+0000 7f3f612dde40 20 rados->read ofs=0 len=0 2023-10-05T19:28:38.669 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.662+0000 7f3f612dde40 1 -- 172.21.15.167:0/1639253938 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:76 5.f 5:f43b6ece:::zone_names.default:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x55fa994c3e30 con 0x55fa994ba7a0 2023-10-05T19:28:38.670 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.662+0000 7f3f5f613700 1 -- 172.21.15.167:0/1639253938 <== osd.1 v2:172.21.15.167:6800/30035 26 ==== osd_op_reply(76 zone_names.default [read 0~46 out=46b] v0'0 uv1 ondisk = 0) v8 ==== 162+0+46 (crc 0 0 0) 0x7f3f5802f080 con 0x55fa994ba7a0 2023-10-05T19:28:38.670 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.662+0000 7f3f612dde40 20 rados_obj.operate() r=0 bl.length=46 2023-10-05T19:28:38.670 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.662+0000 7f3f612dde40 20 rados->read ofs=0 len=0 2023-10-05T19:28:38.670 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.662+0000 7f3f612dde40 1 -- 172.21.15.167:0/1639253938 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:77 5.1d 5:b8374eb4:::zone_info.9a9df965-6612-45d4-a45f-6e3c889bacc2:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x55fa994c41e0 con 0x55fa994ba7a0 2023-10-05T19:28:38.670 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.662+0000 7f3f5f613700 1 -- 172.21.15.167:0/1639253938 <== osd.1 v2:172.21.15.167:6800/30035 27 ==== osd_op_reply(77 zone_info.9a9df965-6612-45d4-a45f-6e3c889bacc2 [read 0~874 out=874b] v0'0 uv2 ondisk = 0) v8 ==== 190+0+874 (crc 0 0 0) 0x7f3f5802f080 con 0x55fa994ba7a0 2023-10-05T19:28:38.670 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.662+0000 7f3f612dde40 20 rados_obj.operate() r=0 bl.length=874 2023-10-05T19:28:38.670 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.662+0000 7f3f612dde40 20 rados->read ofs=0 len=0 2023-10-05T19:28:38.670 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.662+0000 7f3f612dde40 1 -- 172.21.15.167:0/1639253938 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:78 5.f 5:f4c53578:::zonegroups_names.default:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x55fa994bfbc0 con 0x55fa994ba7a0 2023-10-05T19:28:38.671 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.666+0000 7f3f5f613700 1 -- 172.21.15.167:0/1639253938 <== osd.1 v2:172.21.15.167:6800/30035 28 ==== osd_op_reply(78 zonegroups_names.default [read 0~46 out=46b] v0'0 uv2 ondisk = 0) v8 ==== 168+0+46 (crc 0 0 0) 0x7f3f5802f080 con 0x55fa994ba7a0 2023-10-05T19:28:38.671 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.666+0000 7f3f612dde40 20 rados_obj.operate() r=0 bl.length=46 2023-10-05T19:28:38.672 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.666+0000 7f3f612dde40 20 rados->read ofs=0 len=0 2023-10-05T19:28:38.672 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.666+0000 7f3f612dde40 1 -- 172.21.15.167:0/1639253938 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:79 5.c 5:30d299a5:::zonegroup_info.4f1f719d-9648-417b-b10c-e5d939825419:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x55fa994bfbd0 con 0x7f3f5000d8e0 2023-10-05T19:28:38.672 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.666+0000 7f3f5f613700 1 -- 172.21.15.167:0/1639253938 <== osd.0 v2:172.21.15.118:6800/27690 20 ==== osd_op_reply(79 zonegroup_info.4f1f719d-9648-417b-b10c-e5d939825419 [read 0~362 out=362b] v0'0 uv1 ondisk = 0) v8 ==== 195+0+362 (crc 0 0 0) 0x7f3f5802f080 con 0x7f3f5000d8e0 2023-10-05T19:28:38.672 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.666+0000 7f3f612dde40 20 rados_obj.operate() r=0 bl.length=362 2023-10-05T19:28:38.672 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.666+0000 7f3f612dde40 20 rados->read ofs=0 len=0 2023-10-05T19:28:38.672 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.666+0000 7f3f612dde40 1 -- 172.21.15.167:0/1639253938 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:80 5.3 5:c0ad8cbd:::region_map:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x55fa994c4720 con 0x7f3f5000d8e0 2023-10-05T19:28:38.673 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.666+0000 7f3f5f613700 1 -- 172.21.15.167:0/1639253938 <== osd.0 v2:172.21.15.118:6800/27690 21 ==== osd_op_reply(80 region_map [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 ==== 154+0+0 (crc 0 0 0) 0x7f3f5802f080 con 0x7f3f5000d8e0 2023-10-05T19:28:38.673 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.666+0000 7f3f612dde40 20 rados_obj.operate() r=-2 bl.length=0 2023-10-05T19:28:38.673 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.666+0000 7f3f612dde40 20 zone default found 2023-10-05T19:28:38.673 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.666+0000 7f3f612dde40 4 Realm: () 2023-10-05T19:28:38.673 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.666+0000 7f3f612dde40 4 ZoneGroup: default (4f1f719d-9648-417b-b10c-e5d939825419) 2023-10-05T19:28:38.674 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.666+0000 7f3f612dde40 4 Zone: default (9a9df965-6612-45d4-a45f-6e3c889bacc2) 2023-10-05T19:28:38.674 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.666+0000 7f3f612dde40 10 cannot find current period zonegroup using local zonegroup configuration 2023-10-05T19:28:38.674 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.666+0000 7f3f612dde40 20 zonegroup default 2023-10-05T19:28:38.674 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.666+0000 7f3f612dde40 20 rados->read ofs=0 len=0 2023-10-05T19:28:38.674 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.666+0000 7f3f612dde40 1 -- 172.21.15.167:0/1639253938 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:81 5.3 5:c52100b6:::period_config.default:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x55fa994c4720 con 0x7f3f5000d8e0 2023-10-05T19:28:38.674 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.666+0000 7f3f5f613700 1 -- 172.21.15.167:0/1639253938 <== osd.0 v2:172.21.15.118:6800/27690 22 ==== osd_op_reply(81 period_config.default [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 ==== 165+0+0 (crc 0 0 0) 0x7f3f5802f080 con 0x7f3f5000d8e0 2023-10-05T19:28:38.674 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.666+0000 7f3f612dde40 20 rados_obj.operate() r=-2 bl.length=0 2023-10-05T19:28:38.674 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.666+0000 7f3f612dde40 20 rados->read ofs=0 len=0 2023-10-05T19:28:38.674 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.666+0000 7f3f612dde40 1 -- 172.21.15.167:0/1639253938 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:82 6.2 6:4347d321:::bucket.sync-source-hints.:head [call version.read in=11b,read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x55fa994c6050 con 0x55fa994c0920 2023-10-05T19:28:38.675 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.666+0000 7f3f5ee12700 1 -- 172.21.15.167:0/1639253938 <== osd.3 v2:172.21.15.167:6808/30036 20 ==== osd_op_reply(82 bucket.sync-source-hints. [call,read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 ==== 211+0+0 (crc 0 0 0) 0x7f3f48040040 con 0x55fa994c0920 2023-10-05T19:28:38.676 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.670+0000 7f3f612dde40 20 rados_obj.operate() r=-2 bl.length=0 2023-10-05T19:28:38.676 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.670+0000 7f3f612dde40 20 rados->read ofs=0 len=0 2023-10-05T19:28:38.676 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.670+0000 7f3f612dde40 1 -- 172.21.15.167:0/1639253938 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:83 6.b 6:d467b91b:::bucket.sync-target-hints.:head [call version.read in=11b,read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x55fa994c6400 con 0x55fa994c0920 2023-10-05T19:28:38.676 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.670+0000 7f3f5ee12700 1 -- 172.21.15.167:0/1639253938 <== osd.3 v2:172.21.15.167:6808/30036 21 ==== osd_op_reply(83 bucket.sync-target-hints. [call,read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 ==== 211+0+0 (crc 0 0 0) 0x7f3f48040040 con 0x55fa994c0920 2023-10-05T19:28:38.676 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.670+0000 7f3f612dde40 20 rados_obj.operate() r=-2 bl.length=0 2023-10-05T19:28:38.676 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.670+0000 7f3f612dde40 20 started sync module instance, tier type = 2023-10-05T19:28:38.676 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.670+0000 7f3f612dde40 20 started zone id=9a9df965-6612-45d4-a45f-6e3c889bacc2 (name=default) with tier type = 2023-10-05T19:28:38.676 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.670+0000 7f3f612dde40 1 -- 172.21.15.167:0/1639253938 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:84 7.1f 7:f95f44c2:::notify.0:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x55fa994c5710 con 0x7f3f48037070 2023-10-05T19:28:38.677 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.670+0000 7f3f5e611700 1 -- 172.21.15.167:0/1639253938 <== osd.2 v2:172.21.15.167:6816/30037 13 ==== osd_op_reply(84 notify.0 [create] v30'183 uv183 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f3f50006070 con 0x7f3f48037070 2023-10-05T19:28:38.677 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.670+0000 7f3f612dde40 1 -- 172.21.15.167:0/1639253938 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:85 7.1f 7:f95f44c2:::notify.0:head [watch watch cookie 94534802100768] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x55fa994c67b0 con 0x7f3f48037070 2023-10-05T19:28:38.678 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.670+0000 7f3f612dde40 1 -- 172.21.15.167:0/1639253938 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:86 7.0 7:05bf5b68:::notify.1:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x55fa994c7050 con 0x55fa994ba7a0 2023-10-05T19:28:38.680 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.674+0000 7f3f5e611700 1 -- 172.21.15.167:0/1639253938 <== osd.2 v2:172.21.15.167:6816/30037 14 ==== osd_op_reply(85 notify.0 [watch watch cookie 94534802100768] v30'184 uv183 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f3f50006070 con 0x7f3f48037070 2023-10-05T19:28:38.681 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.674+0000 7f3f5f613700 1 -- 172.21.15.167:0/1639253938 <== osd.1 v2:172.21.15.167:6800/30035 29 ==== osd_op_reply(86 notify.1 [create] v30'183 uv183 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f3f5802f080 con 0x55fa994ba7a0 2023-10-05T19:28:38.681 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.674+0000 7f3f612dde40 1 -- 172.21.15.167:0/1639253938 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:87 7.0 7:05bf5b68:::notify.1:head [watch watch cookie 94534802107232] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x55fa994c74a0 con 0x55fa994ba7a0 2023-10-05T19:28:38.681 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.674+0000 7f3f612dde40 1 -- 172.21.15.167:0/1639253938 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:88 7.15 7:a93a5511:::notify.2:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x55fa994c7d40 con 0x55fa994c0920 2023-10-05T19:28:38.684 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.678+0000 7f3f5f613700 1 -- 172.21.15.167:0/1639253938 <== osd.1 v2:172.21.15.167:6800/30035 30 ==== osd_op_reply(87 notify.1 [watch watch cookie 94534802107232] v30'184 uv183 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f3f58050040 con 0x55fa994ba7a0 2023-10-05T19:28:38.685 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.678+0000 7f3f5ee12700 1 -- 172.21.15.167:0/1639253938 <== osd.3 v2:172.21.15.167:6808/30036 22 ==== osd_op_reply(88 notify.2 [create] v30'183 uv183 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f3f48040040 con 0x55fa994c0920 2023-10-05T19:28:38.685 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.678+0000 7f3f612dde40 1 -- 172.21.15.167:0/1639253938 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:89 7.15 7:a93a5511:::notify.2:head [watch watch cookie 94534802108336] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x55fa994c78f0 con 0x55fa994c0920 2023-10-05T19:28:38.685 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.678+0000 7f3f612dde40 1 -- 172.21.15.167:0/1639253938 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:90 7.e 7:7759931f:::notify.3:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x55fa994c87c0 con 0x7f3f48037070 2023-10-05T19:28:38.688 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.682+0000 7f3f5e611700 1 -- 172.21.15.167:0/1639253938 <== osd.2 v2:172.21.15.167:6816/30037 15 ==== osd_op_reply(90 notify.3 [create] v30'183 uv183 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f3f50006070 con 0x7f3f48037070 2023-10-05T19:28:38.688 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.682+0000 7f3f612dde40 1 -- 172.21.15.167:0/1639253938 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:91 7.e 7:7759931f:::notify.3:head [watch watch cookie 94534802115520] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x55fa994c9500 con 0x7f3f48037070 2023-10-05T19:28:38.688 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.682+0000 7f3f612dde40 1 -- 172.21.15.167:0/1639253938 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:92 7.d 7:b4812045:::notify.4:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x55fa994c9de0 con 0x55fa994ba7a0 2023-10-05T19:28:38.688 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.682+0000 7f3f5ee12700 1 -- 172.21.15.167:0/1639253938 <== osd.3 v2:172.21.15.167:6808/30036 23 ==== osd_op_reply(89 notify.2 [watch watch cookie 94534802108336] v30'184 uv183 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f3f48040040 con 0x55fa994c0920 2023-10-05T19:28:38.690 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.686+0000 7f3f5f613700 1 -- 172.21.15.167:0/1639253938 <== osd.1 v2:172.21.15.167:6800/30035 31 ==== osd_op_reply(92 notify.4 [create] v30'183 uv183 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f3f58050040 con 0x55fa994ba7a0 2023-10-05T19:28:38.691 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.686+0000 7f3f612dde40 1 -- 172.21.15.167:0/1639253938 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:93 7.d 7:b4812045:::notify.4:head [watch watch cookie 94534802109440] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x55fa994ca1e0 con 0x55fa994ba7a0 2023-10-05T19:28:38.691 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.686+0000 7f3f612dde40 1 -- 172.21.15.167:0/1639253938 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:94 7.3 7:c609908c:::notify.5:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x55fa994cabf0 con 0x7f3f5000d8e0 2023-10-05T19:28:38.691 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.686+0000 7f3f5e611700 1 -- 172.21.15.167:0/1639253938 <== osd.2 v2:172.21.15.167:6816/30037 16 ==== osd_op_reply(91 notify.3 [watch watch cookie 94534802115520] v30'184 uv183 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f3f50006070 con 0x7f3f48037070 2023-10-05T19:28:38.693 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.686+0000 7f3f5f613700 1 -- 172.21.15.167:0/1639253938 <== osd.1 v2:172.21.15.167:6800/30035 32 ==== osd_op_reply(93 notify.4 [watch watch cookie 94534802109440] v30'184 uv183 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f3f58050040 con 0x55fa994ba7a0 2023-10-05T19:28:38.694 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.686+0000 7f3f5f613700 1 -- 172.21.15.167:0/1639253938 <== osd.0 v2:172.21.15.118:6800/27690 23 ==== osd_op_reply(94 notify.5 [create] v30'183 uv183 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f3f58050040 con 0x7f3f5000d8e0 2023-10-05T19:28:38.694 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.686+0000 7f3f612dde40 1 -- 172.21.15.167:0/1639253938 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:95 7.3 7:c609908c:::notify.5:head [watch watch cookie 94534802116624] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x55fa994ca700 con 0x7f3f5000d8e0 2023-10-05T19:28:38.694 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.690+0000 7f3f612dde40 1 -- 172.21.15.167:0/1639253938 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:96 7.14 7:2b04a3e9:::notify.6:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x55fa994cb110 con 0x7f3f5000d8e0 2023-10-05T19:28:38.697 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.690+0000 7f3f5f613700 1 -- 172.21.15.167:0/1639253938 <== osd.0 v2:172.21.15.118:6800/27690 24 ==== osd_op_reply(96 notify.6 [create] v30'183 uv183 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f3f58050040 con 0x7f3f5000d8e0 2023-10-05T19:28:38.697 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.690+0000 7f3f612dde40 1 -- 172.21.15.167:0/1639253938 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:97 7.14 7:2b04a3e9:::notify.6:head [watch watch cookie 94534802123808] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x55fa994cb560 con 0x7f3f5000d8e0 2023-10-05T19:28:38.697 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.690+0000 7f3f612dde40 1 -- 172.21.15.167:0/1639253938 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:98 7.9 7:93e5b521:::notify.7:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x55fa994cbf70 con 0x7f3f5000d8e0 2023-10-05T19:28:38.697 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.690+0000 7f3f5f613700 1 -- 172.21.15.167:0/1639253938 <== osd.0 v2:172.21.15.118:6800/27690 25 ==== osd_op_reply(95 notify.5 [watch watch cookie 94534802116624] v30'184 uv183 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f3f58050040 con 0x7f3f5000d8e0 2023-10-05T19:28:38.700 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.694+0000 7f3f5f613700 1 -- 172.21.15.167:0/1639253938 <== osd.0 v2:172.21.15.118:6800/27690 26 ==== osd_op_reply(98 notify.7 [create] v30'183 uv183 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f3f58050040 con 0x7f3f5000d8e0 2023-10-05T19:28:38.700 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.694+0000 7f3f5f613700 1 -- 172.21.15.167:0/1639253938 <== osd.0 v2:172.21.15.118:6800/27690 27 ==== osd_op_reply(97 notify.6 [watch watch cookie 94534802123808] v30'184 uv183 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f3f58050040 con 0x7f3f5000d8e0 2023-10-05T19:28:38.700 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.694+0000 7f3f612dde40 1 -- 172.21.15.167:0/1639253938 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:99 7.9 7:93e5b521:::notify.7:head [watch watch cookie 94534802118832] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x55fa994cbf70 con 0x7f3f5000d8e0 2023-10-05T19:28:38.700 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.694+0000 7f3f612dde40 20 add_watcher() i=0 2023-10-05T19:28:38.700 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.694+0000 7f3f612dde40 20 add_watcher() i=1 2023-10-05T19:28:38.700 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.694+0000 7f3f612dde40 20 add_watcher() i=2 2023-10-05T19:28:38.700 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.694+0000 7f3f612dde40 20 add_watcher() i=3 2023-10-05T19:28:38.700 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.694+0000 7f3f612dde40 20 add_watcher() i=4 2023-10-05T19:28:38.700 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.694+0000 7f3f612dde40 20 add_watcher() i=5 2023-10-05T19:28:38.700 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.694+0000 7f3f612dde40 20 add_watcher() i=6 2023-10-05T19:28:38.701 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.694+0000 7f3f5f613700 1 -- 172.21.15.167:0/1639253938 <== osd.0 v2:172.21.15.118:6800/27690 28 ==== osd_op_reply(99 notify.7 [watch watch cookie 94534802118832] v30'184 uv183 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f3f58050040 con 0x7f3f5000d8e0 2023-10-05T19:28:38.701 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.694+0000 7f3f612dde40 20 add_watcher() i=7 2023-10-05T19:28:38.702 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.694+0000 7f3f612dde40 2 all 8 watchers are set, enabling cache 2023-10-05T19:28:38.702 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.694+0000 7f3f612dde40 1 -- 172.21.15.167:0/1639253938 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:100 6.4 6:22d26bf9:::data_loggenerations_metadata:head [call version.check_conds in=74b,call version.read in=11b,read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x55fa994cefa0 con 0x55fa994ba7a0 2023-10-05T19:28:38.702 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.698+0000 7f3f5f613700 1 -- 172.21.15.167:0/1639253938 <== osd.1 v2:172.21.15.167:6800/30035 33 ==== osd_op_reply(100 data_loggenerations_metadata [call,call out=48b,read 0~28 out=28b] v0'0 uv1 ondisk = 0) v8 ==== 256+0+76 (crc 0 0 0) 0x7f3f58050040 con 0x55fa994ba7a0 2023-10-05T19:28:38.702 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.698+0000 7f3f612dde40 1 -- 172.21.15.167:0/1639253938 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:101 6.4 6:22d26bf9:::data_loggenerations_metadata:head [watch watch cookie 94534802127488] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x55fa994cc3c0 con 0x55fa994ba7a0 2023-10-05T19:28:38.705 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.698+0000 7f3f5f613700 1 -- 172.21.15.167:0/1639253938 <== osd.1 v2:172.21.15.167:6800/30035 34 ==== osd_op_reply(101 data_loggenerations_metadata [watch watch cookie 94534802127488] v30'169 uv1 ondisk = 0) v8 ==== 172+0+0 (crc 0 0 0) 0x7f3f58050040 con 0x55fa994ba7a0 2023-10-05T19:28:38.705 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.698+0000 7f3eabfff700 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2023-10-05T19:28:38.705 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.698+0000 7f3f612dde40 20 check_secure_mon_conn(): auth registy supported: methods=[2] modes=[2,1] 2023-10-05T19:28:38.705 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.698+0000 7f3f612dde40 20 check_secure_mon_conn(): mode 1 is insecure 2023-10-05T19:28:38.706 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.698+0000 7f3f612dde40 1 Processor -- start 2023-10-05T19:28:38.706 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.698+0000 7f3f612dde40 1 -- start start 2023-10-05T19:28:38.707 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.702+0000 7f3f612dde40 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55fa994d9420 0x55fa994d97f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-05T19:28:38.707 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.702+0000 7f3f612dde40 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x55fa994d9d30 con 0x55fa994d9420 2023-10-05T19:28:38.707 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.702+0000 7f3f5e611700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55fa994d9420 0x55fa994d97f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-05T19:28:38.707 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.702+0000 7f3f5e611700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55fa994d9420 0x55fa994d97f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.118:3300/0 says I am v2:172.21.15.167:42264/0 (socket says 172.21.15.167:42264) 2023-10-05T19:28:38.707 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.702+0000 7f3f5e611700 1 -- 172.21.15.167:0/1391327158 learned_addr learned my addr 172.21.15.167:0/1391327158 (peer_addr_for_me v2:172.21.15.167:0/0) 2023-10-05T19:28:38.708 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.702+0000 7f3f5e611700 1 -- 172.21.15.167:0/1391327158 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55fa994d9e70 con 0x55fa994d9420 2023-10-05T19:28:38.708 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.702+0000 7f3f5e611700 1 --2- 172.21.15.167:0/1391327158 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55fa994d9420 0x55fa994d97f0 secure :-1 s=READY pgs=504 cs=0 l=1 rev1=1 crypto rx=0x55fa994be3c0 tx=0x7f3f50006000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d90fc211cee141b server_cookie=0 in_seq=0 out_seq=0 2023-10-05T19:28:38.708 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.702+0000 7f3eab7fe700 1 -- 172.21.15.167:0/1391327158 <== mon.0 v2:172.21.15.118:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f3f50023040 con 0x55fa994d9420 2023-10-05T19:28:38.708 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.702+0000 7f3eab7fe700 1 -- 172.21.15.167:0/1391327158 <== mon.0 v2:172.21.15.118:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3f50024040 con 0x55fa994d9420 2023-10-05T19:28:38.708 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.702+0000 7f3eab7fe700 1 -- 172.21.15.167:0/1391327158 <== mon.0 v2:172.21.15.118:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f3f50013040 con 0x55fa994d9420 2023-10-05T19:28:38.709 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.702+0000 7f3f612dde40 1 -- 172.21.15.167:0/1391327158 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55fa994d9420 msgr2=0x55fa994d97f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-05T19:28:38.709 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.702+0000 7f3f612dde40 1 --2- 172.21.15.167:0/1391327158 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55fa994d9420 0x55fa994d97f0 secure :-1 s=READY pgs=504 cs=0 l=1 rev1=1 crypto rx=0x55fa994be3c0 tx=0x7f3f50006000 comp rx=0 tx=0).stop 2023-10-05T19:28:38.709 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.702+0000 7f3f612dde40 1 -- 172.21.15.167:0/1391327158 shutdown_connections 2023-10-05T19:28:38.709 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.702+0000 7f3f612dde40 1 --2- 172.21.15.167:0/1391327158 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55fa994d9420 0x55fa994d97f0 unknown :-1 s=CLOSED pgs=504 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-05T19:28:38.710 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.702+0000 7f3f612dde40 1 -- 172.21.15.167:0/1391327158 >> 172.21.15.167:0/1391327158 conn(0x55fa994d47e0 msgr2=0x55fa994d2930 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-05T19:28:38.710 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.702+0000 7f3f612dde40 1 -- 172.21.15.167:0/1391327158 shutdown_connections 2023-10-05T19:28:38.710 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.702+0000 7f3f612dde40 1 -- 172.21.15.167:0/1391327158 wait complete. 2023-10-05T19:28:38.711 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.706+0000 7f3f612dde40 1 Processor -- start 2023-10-05T19:28:38.711 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.706+0000 7f3f612dde40 1 -- start start 2023-10-05T19:28:38.711 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.706+0000 7f3f612dde40 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55fa994d9420 0x55fa99400af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-05T19:28:38.711 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.706+0000 7f3f612dde40 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x55fa994003f0 con 0x55fa994d9420 2023-10-05T19:28:38.711 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.706+0000 7f3f5e611700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55fa994d9420 0x55fa99400af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-05T19:28:38.711 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.706+0000 7f3f5e611700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55fa994d9420 0x55fa99400af0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.118:3300/0 says I am v2:172.21.15.167:42276/0 (socket says 172.21.15.167:42276) 2023-10-05T19:28:38.711 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.706+0000 7f3f5e611700 1 -- 172.21.15.167:0/2502269580 learned_addr learned my addr 172.21.15.167:0/2502269580 (peer_addr_for_me v2:172.21.15.167:0/0) 2023-10-05T19:28:38.712 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.706+0000 7f3f5e611700 1 -- 172.21.15.167:0/2502269580 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55fa993d7bd0 con 0x55fa994d9420 2023-10-05T19:28:38.712 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.706+0000 7f3f5e611700 1 --2- 172.21.15.167:0/2502269580 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55fa994d9420 0x55fa99400af0 secure :-1 s=READY pgs=505 cs=0 l=1 rev1=1 crypto rx=0x55fa994be080 tx=0x7f3f50030000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-05T19:28:38.712 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.706+0000 7f3ea9ffb700 1 -- 172.21.15.167:0/2502269580 <== mon.0 v2:172.21.15.118:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f3f50023040 con 0x55fa994d9420 2023-10-05T19:28:38.713 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.706+0000 7f3ea9ffb700 1 -- 172.21.15.167:0/2502269580 <== mon.0 v2:172.21.15.118:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3f5003a4f0 con 0x55fa994d9420 2023-10-05T19:28:38.713 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.706+0000 7f3f612dde40 1 -- 172.21.15.167:0/2502269580 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x55fa994010f0 con 0x55fa994d9420 2023-10-05T19:28:38.713 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.706+0000 7f3ea9ffb700 1 -- 172.21.15.167:0/2502269580 <== mon.0 v2:172.21.15.118:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f3f5003a660 con 0x55fa994d9420 2023-10-05T19:28:38.713 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.706+0000 7f3f612dde40 1 -- 172.21.15.167:0/2502269580 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55fa993ff450 con 0x55fa994d9420 2023-10-05T19:28:38.714 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.706+0000 7f3ea9ffb700 1 -- 172.21.15.167:0/2502269580 <== mon.0 v2:172.21.15.118:3300/0 4 ==== mgrmap(e 5) v1 ==== 40329+0+0 (secure 0 0 0) 0x7f3f5003a7c0 con 0x55fa994d9420 2023-10-05T19:28:38.714 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.710+0000 7f3ea9ffb700 1 --2- 172.21.15.167:0/2502269580 >> [v2:172.21.15.167:6824/29889,v1:172.21.15.167:6825/29889] conn(0x7f3e9c030c00 0x7f3e9c0330b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-05T19:28:38.714 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.710+0000 7f3ea9ffb700 1 -- 172.21.15.167:0/2502269580 <== mon.0 v2:172.21.15.118:3300/0 5 ==== osd_map(30..30 src has 1..30) v4 ==== 6057+0+0 (secure 0 0 0) 0x7f3f5004fc60 con 0x55fa994d9420 2023-10-05T19:28:38.715 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.710+0000 7f3f612dde40 5 note: GC not initialized 2023-10-05T19:28:38.715 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.710+0000 7f3f5f613700 1 --2- 172.21.15.167:0/2502269580 >> [v2:172.21.15.167:6824/29889,v1:172.21.15.167:6825/29889] conn(0x7f3e9c030c00 0x7f3e9c0330b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-05T19:28:38.715 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.710+0000 7f3f5f613700 1 --2- 172.21.15.167:0/2502269580 >> [v2:172.21.15.167:6824/29889,v1:172.21.15.167:6825/29889] conn(0x7f3e9c030c00 0x7f3e9c0330b0 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x55fa994cb240 tx=0x7f3f58000f90 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-05T19:28:38.717 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.710+0000 7f3ea2ffd700 20 reqs_thread_entry: start 2023-10-05T19:28:38.717 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.710+0000 7f3ea0ff9700 20 reqs_thread_entry: start 2023-10-05T19:28:38.717 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.710+0000 7f3f612dde40 20 init_complete bucket index max shards: 11 2023-10-05T19:28:38.718 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.710+0000 7f3f612dde40 10 rgw notify: Started notification manager with: 1 workers 2023-10-05T19:28:38.718 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.710+0000 7f3e8effd700 20 rgw notify: INFO: next queues processing will happen at: Thu Oct 5 19:29:08 2023 2023-10-05T19:28:38.718 INFO:teuthology.orchestra.run.smithi167.stderr: 2023-10-05T19:28:38.718 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.710+0000 7f3e8e7fc700 20 reqs_thread_entry: start 2023-10-05T19:28:38.718 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.710+0000 7f3f612dde40 20 get_system_obj_state: rctx=0x7ffdf689ac98 obj=default.rgw.meta:users.uid:foo state=0x55fa993640b0 s->prefetch_data=0 2023-10-05T19:28:38.718 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.710+0000 7f3f612dde40 10 cache get: name=default.rgw.meta+users.uid+foo : miss 2023-10-05T19:28:38.718 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.710+0000 7f3f612dde40 1 -- 172.21.15.167:0/1639253938 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:102 8.18 8:1e6ca5ed:users.uid::foo:head [call version.read in=11b,getxattrs,stat] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x55fa99549a40 con 0x55fa994ba7a0 2023-10-05T19:28:38.719 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.714+0000 7f3f5f613700 1 -- 172.21.15.167:0/1639253938 <== osd.1 v2:172.21.15.167:6800/30035 35 ==== osd_op_reply(102 foo [call out=48b,getxattrs out=75b,stat out=16b] v0'0 uv11 ondisk = 0) v8 ==== 231+0+139 (crc 0 0 0) 0x7f3f58050040 con 0x55fa994ba7a0 2023-10-05T19:28:38.719 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.714+0000 7f3f612dde40 10 cache put: name=default.rgw.meta+users.uid+foo info.flags=0x16 2023-10-05T19:28:38.720 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.714+0000 7f3f612dde40 10 adding default.rgw.meta+users.uid+foo to cache LRU end 2023-10-05T19:28:38.720 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.714+0000 7f3f612dde40 10 updating xattr: name=ceph.objclass.version bl.length()=42 2023-10-05T19:28:38.720 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.714+0000 7f3f612dde40 20 get_system_obj_state: s->obj_tag was set empty 2023-10-05T19:28:38.720 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.714+0000 7f3f612dde40 20 Read xattr: user.rgw.idtag 2023-10-05T19:28:38.720 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.714+0000 7f3f612dde40 10 cache get: name=default.rgw.meta+users.uid+foo : type miss (requested=0x13, cached=0x16) 2023-10-05T19:28:38.720 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.714+0000 7f3f612dde40 20 rados->read ofs=0 len=0 2023-10-05T19:28:38.720 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.714+0000 7f3f612dde40 1 -- 172.21.15.167:0/1639253938 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:103 8.18 8:1e6ca5ed:users.uid::foo:head [call version.check_conds in=122b,call version.read in=11b,read 0~0,getxattrs] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x55fa9954c4d0 con 0x55fa994ba7a0 2023-10-05T19:28:38.720 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.714+0000 7f3f5f613700 1 -- 172.21.15.167:0/1639253938 <== osd.1 v2:172.21.15.167:6800/30035 36 ==== osd_op_reply(103 foo [call,call out=48b,read 0~357 out=357b,getxattrs out=75b] v0'0 uv11 ondisk = 0) v8 ==== 273+0+480 (crc 0 0 0) 0x7f3f58050040 con 0x55fa994ba7a0 2023-10-05T19:28:38.720 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.714+0000 7f3f612dde40 20 rados_obj.operate() r=0 bl.length=357 2023-10-05T19:28:38.720 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.714+0000 7f3f612dde40 10 cache put: name=default.rgw.meta+users.uid+foo info.flags=0x13 2023-10-05T19:28:38.721 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.714+0000 7f3f612dde40 10 moving default.rgw.meta+users.uid+foo to cache LRU end 2023-10-05T19:28:38.721 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.714+0000 7f3f612dde40 10 updating xattr: name=ceph.objclass.version bl.length()=42 2023-10-05T19:28:38.721 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.714+0000 7f3f612dde40 1 -- 172.21.15.167:0/1639253938 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:104 8.9 8:96d98480:users.uid::foo.buckets:head [call user.list_buckets in=34b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x55fa9954c4d0 con 0x7f3f48037070 2023-10-05T19:28:38.722 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.714+0000 7f3f5e611700 1 -- 172.21.15.167:0/1639253938 <== osd.2 v2:172.21.15.167:6816/30037 17 ==== osd_op_reply(104 foo.buckets [call out=183b] v0'0 uv26 ondisk = 0) v8 ==== 155+0+183 (crc 0 0 0) 0x7f3f50006070 con 0x7f3f48037070 2023-10-05T19:28:38.722 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.714+0000 7f3f612dde40 20 disabling bucket name=myfoo 2023-10-05T19:28:38.722 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.714+0000 7f3f612dde40 20 get_system_obj_state: rctx=0x7ffdf689a470 obj=default.rgw.meta:root:myfoo state=0x55fa993640b0 s->prefetch_data=0 2023-10-05T19:28:38.722 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.714+0000 7f3f612dde40 10 cache get: name=default.rgw.meta+root+myfoo : miss 2023-10-05T19:28:38.722 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.714+0000 7f3f612dde40 1 -- 172.21.15.167:0/1639253938 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:105 8.9 8:91b71841:root::myfoo:head [getxattrs,stat] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x55fa9954acd0 con 0x7f3f48037070 2023-10-05T19:28:38.722 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.714+0000 7f3f5e611700 1 -- 172.21.15.167:0/1639253938 <== osd.2 v2:172.21.15.167:6816/30037 18 ==== osd_op_reply(105 myfoo [getxattrs out=75b,stat out=16b] v0'0 uv27 ondisk = 0) v8 ==== 191+0+91 (crc 0 0 0) 0x7f3f50006070 con 0x7f3f48037070 2023-10-05T19:28:38.722 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.714+0000 7f3f612dde40 10 cache put: name=default.rgw.meta+root+myfoo info.flags=0x6 2023-10-05T19:28:38.723 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.714+0000 7f3f612dde40 10 adding default.rgw.meta+root+myfoo to cache LRU end 2023-10-05T19:28:38.723 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.714+0000 7f3f612dde40 10 updating xattr: name=ceph.objclass.version bl.length()=42 2023-10-05T19:28:38.723 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.714+0000 7f3f612dde40 20 get_system_obj_state: s->obj_tag was set empty 2023-10-05T19:28:38.723 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.714+0000 7f3f612dde40 10 cache get: name=default.rgw.meta+root+myfoo : type miss (requested=0x1, cached=0x6) 2023-10-05T19:28:38.723 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.714+0000 7f3f612dde40 20 rados->read ofs=0 len=0 2023-10-05T19:28:38.723 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.714+0000 7f3f612dde40 1 -- 172.21.15.167:0/1639253938 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:106 8.9 8:91b71841:root::myfoo:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x55fa9954acd0 con 0x7f3f48037070 2023-10-05T19:28:38.723 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.714+0000 7f3f5e611700 1 -- 172.21.15.167:0/1639253938 <== osd.2 v2:172.21.15.167:6816/30037 19 ==== osd_op_reply(106 myfoo [read 0~165 out=165b] v0'0 uv27 ondisk = 0) v8 ==== 149+0+165 (crc 0 0 0) 0x7f3f50006070 con 0x7f3f48037070 2023-10-05T19:28:38.723 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.714+0000 7f3f612dde40 20 rados_obj.operate() r=0 bl.length=165 2023-10-05T19:28:38.723 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.714+0000 7f3f612dde40 10 cache put: name=default.rgw.meta+root+myfoo info.flags=0x1 2023-10-05T19:28:38.723 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.714+0000 7f3f612dde40 10 moving default.rgw.meta+root+myfoo to cache LRU end 2023-10-05T19:28:38.723 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.714+0000 7f3f612dde40 20 get_system_obj_state: rctx=0x7ffdf689a470 obj=default.rgw.meta:root:.bucket.meta.myfoo:9a9df965-6612-45d4-a45f-6e3c889bacc2.4223.1 state=0x55fa9933a2b0 s->prefetch_data=0 2023-10-05T19:28:38.724 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.714+0000 7f3f612dde40 10 cache get: name=default.rgw.meta+root+.bucket.meta.myfoo:9a9df965-6612-45d4-a45f-6e3c889bacc2.4223.1 : miss 2023-10-05T19:28:38.724 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.714+0000 7f3f612dde40 1 -- 172.21.15.167:0/1639253938 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:107 8.11 8:8bb4ed75:root::.bucket.meta.myfoo%3a9a9df965-6612-45d4-a45f-6e3c889bacc2.4223.1:head [call version.read in=11b,getxattrs,stat] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x55fa9954d8e0 con 0x55fa994c0920 2023-10-05T19:28:38.725 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.718+0000 7f3f5ee12700 1 -- 172.21.15.167:0/1639253938 <== osd.3 v2:172.21.15.167:6808/30036 24 ==== osd_op_reply(107 .bucket.meta.myfoo:9a9df965-6612-45d4-a45f-6e3c889bacc2.4223.1 [call out=48b,getxattrs out=218b,stat out=16b] v0'0 uv5 ondisk = 0) v8 ==== 290+0+282 (crc 0 0 0) 0x7f3f48040040 con 0x55fa994c0920 2023-10-05T19:28:38.725 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.718+0000 7f3f612dde40 10 cache put: name=default.rgw.meta+root+.bucket.meta.myfoo:9a9df965-6612-45d4-a45f-6e3c889bacc2.4223.1 info.flags=0x16 2023-10-05T19:28:38.725 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.718+0000 7f3f612dde40 10 adding default.rgw.meta+root+.bucket.meta.myfoo:9a9df965-6612-45d4-a45f-6e3c889bacc2.4223.1 to cache LRU end 2023-10-05T19:28:38.725 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.718+0000 7f3f612dde40 10 updating xattr: name=ceph.objclass.version bl.length()=42 2023-10-05T19:28:38.725 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.718+0000 7f3f612dde40 10 updating xattr: name=user.rgw.acl bl.length()=123 2023-10-05T19:28:38.725 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.718+0000 7f3f612dde40 20 get_system_obj_state: s->obj_tag was set empty 2023-10-05T19:28:38.725 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.718+0000 7f3f612dde40 20 Read xattr: user.rgw.acl 2023-10-05T19:28:38.725 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.718+0000 7f3f612dde40 20 Read xattr: user.rgw.idtag 2023-10-05T19:28:38.725 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.718+0000 7f3f612dde40 10 cache get: name=default.rgw.meta+root+.bucket.meta.myfoo:9a9df965-6612-45d4-a45f-6e3c889bacc2.4223.1 : type miss (requested=0x13, cached=0x16) 2023-10-05T19:28:38.725 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.718+0000 7f3f612dde40 20 rados->read ofs=0 len=0 2023-10-05T19:28:38.726 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.718+0000 7f3f612dde40 1 -- 172.21.15.167:0/1639253938 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:108 8.11 8:8bb4ed75:root::.bucket.meta.myfoo%3a9a9df965-6612-45d4-a45f-6e3c889bacc2.4223.1:head [call version.check_conds in=122b,call version.read in=11b,read 0~0,getxattrs] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x55fa99550830 con 0x55fa994c0920 2023-10-05T19:28:38.726 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.718+0000 7f3f5ee12700 1 -- 172.21.15.167:0/1639253938 <== osd.3 v2:172.21.15.167:6808/30036 25 ==== osd_op_reply(108 .bucket.meta.myfoo:9a9df965-6612-45d4-a45f-6e3c889bacc2.4223.1 [call,call out=48b,read 0~352 out=352b,getxattrs out=218b] v0'0 uv5 ondisk = 0) v8 ==== 332+0+618 (crc 0 0 0) 0x7f3f48040040 con 0x55fa994c0920 2023-10-05T19:28:38.726 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.718+0000 7f3f612dde40 20 rados_obj.operate() r=0 bl.length=352 2023-10-05T19:28:38.726 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.718+0000 7f3f612dde40 10 cache put: name=default.rgw.meta+root+.bucket.meta.myfoo:9a9df965-6612-45d4-a45f-6e3c889bacc2.4223.1 info.flags=0x13 2023-10-05T19:28:38.726 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.718+0000 7f3f612dde40 10 moving default.rgw.meta+root+.bucket.meta.myfoo:9a9df965-6612-45d4-a45f-6e3c889bacc2.4223.1 to cache LRU end 2023-10-05T19:28:38.726 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.718+0000 7f3f612dde40 10 updating xattr: name=ceph.objclass.version bl.length()=42 2023-10-05T19:28:38.726 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.718+0000 7f3f612dde40 10 updating xattr: name=user.rgw.acl bl.length()=123 2023-10-05T19:28:38.726 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.718+0000 7f3f612dde40 10 chain_cache_entry: cache_locator=default.rgw.meta+root+.bucket.meta.myfoo:9a9df965-6612-45d4-a45f-6e3c889bacc2.4223.1 2023-10-05T19:28:38.726 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.718+0000 7f3f612dde40 1 -- 172.21.15.167:0/1639253938 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:109 8.11 8:8bb4ed75:root::.bucket.meta.myfoo%3a9a9df965-6612-45d4-a45f-6e3c889bacc2.4223.1:head [delete,create,call version.check_conds in=122b,call version.set in=58b,writefull 0~352 in=352b,setxattr user.rgw.acl (123) in=135b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x55fa99555fd0 con 0x55fa994c0920 2023-10-05T19:28:38.728 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.722+0000 7f3f5ee12700 1 -- 172.21.15.167:0/1639253938 <== osd.3 v2:172.21.15.167:6808/30036 26 ==== osd_op_reply(109 .bucket.meta.myfoo:9a9df965-6612-45d4-a45f-6e3c889bacc2.4223.1 [delete,create,call,call,writefull 0~352,setxattr (123)] v30'6 uv6 ondisk = 0) v8 ==== 416+0+0 (crc 0 0 0) 0x7f3f48040040 con 0x55fa994c0920 2023-10-05T19:28:38.728 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.722+0000 7f3f612dde40 10 cache put: name=default.rgw.meta+root+.bucket.meta.myfoo:9a9df965-6612-45d4-a45f-6e3c889bacc2.4223.1 info.flags=0x17 2023-10-05T19:28:38.728 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.722+0000 7f3f612dde40 10 moving default.rgw.meta+root+.bucket.meta.myfoo:9a9df965-6612-45d4-a45f-6e3c889bacc2.4223.1 to cache LRU end 2023-10-05T19:28:38.728 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.722+0000 7f3f612dde40 10 updating xattr: name=user.rgw.acl bl.length()=123 2023-10-05T19:28:38.728 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.722+0000 7f3f612dde40 10 distributing notification oid=default.rgw.control:notify.7 cni=[op: 0, obj: default.rgw.meta:root:.bucket.meta.myfoo:9a9df965-6612-45d4-a45f-6e3c889bacc2.4223.1, ofs0, ns] 2023-10-05T19:28:38.728 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.722+0000 7f3f612dde40 1 -- 172.21.15.167:0/1639253938 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:110 7.9 7:93e5b521:::notify.7:head [notify cookie 94534802667408 in=737b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x55fa99551f90 con 0x7f3f5000d8e0 2023-10-05T19:28:38.729 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.722+0000 7f3f5f613700 1 -- 172.21.15.167:0/1639253938 <== osd.0 v2:172.21.15.118:6800/27690 29 ==== watch-notify(notify (1) cookie 94534802118832 notify 128849018897 ret 0) v3 ==== 767+0+0 (crc 0 0 0) 0x7f3f5803d040 con 0x7f3f5000d8e0 2023-10-05T19:28:38.729 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.722+0000 7f3f5ce0e700 10 rgw watcher librados: RGWWatcher::handle_notify() notify_id 128849018897 cookie 94534802118832 notifier 4593 bl.length()=725 2023-10-05T19:28:38.729 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.722+0000 7f3f5ce0e700 10 rgw watcher librados: cache put: name=default.rgw.meta+root+.bucket.meta.myfoo:9a9df965-6612-45d4-a45f-6e3c889bacc2.4223.1 info.flags=0x17 2023-10-05T19:28:38.729 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.722+0000 7f3f5ce0e700 10 rgw watcher librados: moving default.rgw.meta+root+.bucket.meta.myfoo:9a9df965-6612-45d4-a45f-6e3c889bacc2.4223.1 to cache LRU end 2023-10-05T19:28:38.729 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.722+0000 7f3f5ce0e700 10 rgw watcher librados: updating xattr: name=user.rgw.acl bl.length()=123 2023-10-05T19:28:38.729 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.722+0000 7f3f5ce0e700 1 -- 172.21.15.167:0/1639253938 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:111 7.9 7:93e5b521:::notify.7:head [notify-ack in=20b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x7f3f44003d90 con 0x7f3f5000d8e0 2023-10-05T19:28:38.729 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.722+0000 7f3f5f613700 1 -- 172.21.15.167:0/1639253938 <== osd.0 v2:172.21.15.118:6800/27690 30 ==== osd_op_reply(110 notify.7 [notify cookie 94534802667408 out=8b] v0'0 uv183 ondisk = 0) v8 ==== 152+0+8 (crc 0 0 0) 0x7f3f58050040 con 0x7f3f5000d8e0 2023-10-05T19:28:38.730 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.722+0000 7f3f5f613700 1 -- 172.21.15.167:0/1639253938 <== osd.0 v2:172.21.15.118:6800/27690 31 ==== osd_op_reply(111 notify.7 [notify-ack] v0'0 uv183 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f3f58050040 con 0x7f3f5000d8e0 2023-10-05T19:28:38.730 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.726+0000 7f3f5f613700 1 -- 172.21.15.167:0/1639253938 <== osd.0 v2:172.21.15.118:6800/27690 32 ==== watch-notify(notify_complete (2) cookie 94534802667408 notify 128849018897 ret 0) v3 ==== 42+0+48 (crc 0 0 0) 0x7f3f58002dd0 con 0x7f3f5000d8e0 2023-10-05T19:28:38.730 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.726+0000 7f3f612dde40 20 handle_bi_update(): bucket=:myfoo[9a9df965-6612-45d4-a45f-6e3c889bacc2.4223.1]): orig_sources= new_sources= 2023-10-05T19:28:38.731 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.726+0000 7f3f612dde40 20 handle_bi_update(): bucket=:myfoo[9a9df965-6612-45d4-a45f-6e3c889bacc2.4223.1]): potential sources added=[] removed=[] 2023-10-05T19:28:38.731 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.726+0000 7f3f612dde40 20 handle_bi_update(): bucket=:myfoo[9a9df965-6612-45d4-a45f-6e3c889bacc2.4223.1]): orig_dests= new_dests= 2023-10-05T19:28:38.731 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.726+0000 7f3f612dde40 20 handle_bi_update(): bucket=:myfoo[9a9df965-6612-45d4-a45f-6e3c889bacc2.4223.1]): potential dests added=[] removed=[] 2023-10-05T19:28:38.731 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.726+0000 7f3f612dde40 1 -- 172.21.15.167:0/1639253938 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:112 8.18 8:1e6ca5ed:users.uid::foo:head [delete,create,call version.set in=58b,writefull 0~357 in=357b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x55fa9954f790 con 0x55fa994ba7a0 2023-10-05T19:28:38.733 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.726+0000 7f3f5f613700 1 -- 172.21.15.167:0/1639253938 <== osd.1 v2:172.21.15.167:6800/30035 37 ==== osd_op_reply(112 foo [delete,create,call,writefull 0~357] v30'12 uv12 ondisk = 0) v8 ==== 273+0+0 (crc 0 0 0) 0x7f3f58050040 con 0x55fa994ba7a0 2023-10-05T19:28:38.734 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.726+0000 7f3f612dde40 10 cache put: name=default.rgw.meta+users.uid+foo info.flags=0x17 2023-10-05T19:28:38.734 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.726+0000 7f3f612dde40 10 moving default.rgw.meta+users.uid+foo to cache LRU end 2023-10-05T19:28:38.734 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.726+0000 7f3f612dde40 10 distributing notification oid=default.rgw.control:notify.4 cni=[op: 0, obj: default.rgw.meta:users.uid:foo, ofs0, ns] 2023-10-05T19:28:38.734 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.726+0000 7f3f612dde40 1 -- 172.21.15.167:0/1639253938 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:113 7.d 7:b4812045:::notify.4:head [notify cookie 94534802667408 in=545b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x55fa9954a260 con 0x55fa994ba7a0 2023-10-05T19:28:38.734 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.730+0000 7f3f5f613700 1 -- 172.21.15.167:0/1639253938 <== osd.1 v2:172.21.15.167:6800/30035 38 ==== watch-notify(notify (1) cookie 94534802109440 notify 128849018897 ret 0) v3 ==== 575+0+0 (crc 0 0 0) 0x7f3f58002dd0 con 0x55fa994ba7a0 2023-10-05T19:28:38.735 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.730+0000 7f3f57fff700 10 rgw watcher librados: RGWWatcher::handle_notify() notify_id 128849018897 cookie 94534802109440 notifier 4593 bl.length()=533 2023-10-05T19:28:38.735 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.730+0000 7f3f57fff700 10 rgw watcher librados: cache put: name=default.rgw.meta+users.uid+foo info.flags=0x17 2023-10-05T19:28:38.735 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.730+0000 7f3f57fff700 10 rgw watcher librados: moving default.rgw.meta+users.uid+foo to cache LRU end 2023-10-05T19:28:38.735 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.730+0000 7f3f5f613700 1 -- 172.21.15.167:0/1639253938 <== osd.1 v2:172.21.15.167:6800/30035 39 ==== osd_op_reply(113 notify.4 [notify cookie 94534802667408 out=8b] v0'0 uv183 ondisk = 0) v8 ==== 152+0+8 (crc 0 0 0) 0x7f3f58050040 con 0x55fa994ba7a0 2023-10-05T19:28:38.735 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.730+0000 7f3f57fff700 1 -- 172.21.15.167:0/1639253938 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:114 7.d 7:b4812045:::notify.4:head [notify-ack in=20b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x7f3eb0003440 con 0x55fa994ba7a0 2023-10-05T19:28:38.736 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.730+0000 7f3f5f613700 1 -- 172.21.15.167:0/1639253938 <== osd.1 v2:172.21.15.167:6800/30035 40 ==== watch-notify(notify_complete (2) cookie 94534802667408 notify 128849018897 ret 0) v3 ==== 42+0+48 (crc 0 0 0) 0x7f3f58002580 con 0x55fa994ba7a0 2023-10-05T19:28:38.736 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.730+0000 7f3f5f613700 1 -- 172.21.15.167:0/1639253938 <== osd.1 v2:172.21.15.167:6800/30035 41 ==== osd_op_reply(114 notify.4 [notify-ack] v0'0 uv183 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f3f58050040 con 0x55fa994ba7a0 2023-10-05T19:28:38.736 INFO:teuthology.orchestra.run.smithi167.stdout:{ 2023-10-05T19:28:38.736 INFO:teuthology.orchestra.run.smithi167.stdout: "user_id": "foo", 2023-10-05T19:28:38.736 INFO:teuthology.orchestra.run.smithi167.stdout: "display_name": "Foo", 2023-10-05T19:28:38.736 INFO:teuthology.orchestra.run.smithi167.stdout: "email": "foo@foo.com", 2023-10-05T19:28:38.736 INFO:teuthology.orchestra.run.smithi167.stdout: "suspended": 1, 2023-10-05T19:28:38.736 INFO:teuthology.orchestra.run.smithi167.stdout: "max_buckets": 4, 2023-10-05T19:28:38.737 INFO:teuthology.orchestra.run.smithi167.stdout: "subusers": [], 2023-10-05T19:28:38.737 INFO:teuthology.orchestra.run.smithi167.stdout: "keys": [ 2023-10-05T19:28:38.737 INFO:teuthology.orchestra.run.smithi167.stdout: { 2023-10-05T19:28:38.737 INFO:teuthology.orchestra.run.smithi167.stdout: "user": "foo", 2023-10-05T19:28:38.737 INFO:teuthology.orchestra.run.smithi167.stdout: "access_key": "9te6NH5mcdcq0Tc5i8i1", 2023-10-05T19:28:38.737 INFO:teuthology.orchestra.run.smithi167.stdout: "secret_key": "Ny4IOauQoL18Gp2zM7lC1vLmoawgqcYP/YGcWfXu" 2023-10-05T19:28:38.737 INFO:teuthology.orchestra.run.smithi167.stdout: } 2023-10-05T19:28:38.737 INFO:teuthology.orchestra.run.smithi167.stdout: ], 2023-10-05T19:28:38.737 INFO:teuthology.orchestra.run.smithi167.stdout: "swift_keys": [], 2023-10-05T19:28:38.737 INFO:teuthology.orchestra.run.smithi167.stdout: "caps": [], 2023-10-05T19:28:38.737 INFO:teuthology.orchestra.run.smithi167.stdout: "op_mask": "read, write, delete", 2023-10-05T19:28:38.737 INFO:teuthology.orchestra.run.smithi167.stdout: "default_placement": "", 2023-10-05T19:28:38.737 INFO:teuthology.orchestra.run.smithi167.stdout: "default_storage_class": "", 2023-10-05T19:28:38.738 INFO:teuthology.orchestra.run.smithi167.stdout: "placement_tags": [], 2023-10-05T19:28:38.738 INFO:teuthology.orchestra.run.smithi167.stdout: "bucket_quota": { 2023-10-05T19:28:38.738 INFO:teuthology.orchestra.run.smithi167.stdout: "enabled": false, 2023-10-05T19:28:38.738 INFO:teuthology.orchestra.run.smithi167.stdout: "check_on_raw": false, 2023-10-05T19:28:38.738 INFO:teuthology.orchestra.run.smithi167.stdout: "max_size": -1, 2023-10-05T19:28:38.738 INFO:teuthology.orchestra.run.smithi167.stdout: "max_size_kb": 0, 2023-10-05T19:28:38.738 INFO:teuthology.orchestra.run.smithi167.stdout: "max_objects": -1 2023-10-05T19:28:38.738 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2023-10-05T19:28:38.738 INFO:teuthology.orchestra.run.smithi167.stdout: "user_quota": { 2023-10-05T19:28:38.738 INFO:teuthology.orchestra.run.smithi167.stdout: "enabled": false, 2023-10-05T19:28:38.738 INFO:teuthology.orchestra.run.smithi167.stdout: "check_on_raw": false, 2023-10-05T19:28:38.738 INFO:teuthology.orchestra.run.smithi167.stdout: "max_size": -1, 2023-10-05T19:28:38.739 INFO:teuthology.orchestra.run.smithi167.stdout: "max_size_kb": 0, 2023-10-05T19:28:38.739 INFO:teuthology.orchestra.run.smithi167.stdout: "max_objects": -1 2023-10-05T19:28:38.739 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2023-10-05T19:28:38.739 INFO:teuthology.orchestra.run.smithi167.stdout: "temp_url_keys": [], 2023-10-05T19:28:38.739 INFO:teuthology.orchestra.run.smithi167.stdout: "type": "rgw", 2023-10-05T19:28:38.739 INFO:teuthology.orchestra.run.smithi167.stdout: "mfa_ids": [] 2023-10-05T19:28:38.739 INFO:teuthology.orchestra.run.smithi167.stdout:} 2023-10-05T19:28:38.739 INFO:teuthology.orchestra.run.smithi167.stdout: 2023-10-05T19:28:38.740 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.730+0000 7f3f612dde40 1 -- 172.21.15.167:0/1639253938 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:115 7.1f 7:f95f44c2:::notify.0:head [watch unwatch cookie 94534802100768] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x55fa994a0700 con 0x7f3f48037070 2023-10-05T19:28:38.740 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.734+0000 7f3f5e611700 1 -- 172.21.15.167:0/1639253938 <== osd.2 v2:172.21.15.167:6816/30037 20 ==== osd_op_reply(115 notify.0 [watch unwatch cookie 94534802100768] v30'185 uv183 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f3f50006070 con 0x7f3f48037070 2023-10-05T19:28:38.740 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.734+0000 7f3f612dde40 20 remove_watcher() i=0 2023-10-05T19:28:38.740 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.734+0000 7f3f612dde40 2 removed watcher, disabling cache 2023-10-05T19:28:38.740 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.734+0000 7f3f612dde40 1 -- 172.21.15.167:0/1639253938 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:116 7.0 7:05bf5b68:::notify.1:head [watch unwatch cookie 94534802107232] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x55fa994a0700 con 0x55fa994ba7a0 2023-10-05T19:28:38.741 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.734+0000 7f3f5f613700 1 -- 172.21.15.167:0/1639253938 <== osd.1 v2:172.21.15.167:6800/30035 42 ==== osd_op_reply(116 notify.1 [watch unwatch cookie 94534802107232] v30'185 uv183 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f3f58050040 con 0x55fa994ba7a0 2023-10-05T19:28:38.741 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.734+0000 7f3f612dde40 20 remove_watcher() i=1 2023-10-05T19:28:38.742 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.734+0000 7f3f612dde40 1 -- 172.21.15.167:0/1639253938 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:117 7.15 7:a93a5511:::notify.2:head [watch unwatch cookie 94534802108336] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x55fa994a0700 con 0x55fa994c0920 2023-10-05T19:28:38.744 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.738+0000 7f3f5ee12700 1 -- 172.21.15.167:0/1639253938 <== osd.3 v2:172.21.15.167:6808/30036 27 ==== osd_op_reply(117 notify.2 [watch unwatch cookie 94534802108336] v30'185 uv183 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f3f48040040 con 0x55fa994c0920 2023-10-05T19:28:38.744 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.738+0000 7f3f612dde40 20 remove_watcher() i=2 2023-10-05T19:28:38.745 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.738+0000 7f3f612dde40 1 -- 172.21.15.167:0/1639253938 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:118 7.e 7:7759931f:::notify.3:head [watch unwatch cookie 94534802115520] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x55fa994a0700 con 0x7f3f48037070 2023-10-05T19:28:38.747 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.742+0000 7f3f5e611700 1 -- 172.21.15.167:0/1639253938 <== osd.2 v2:172.21.15.167:6816/30037 21 ==== osd_op_reply(118 notify.3 [watch unwatch cookie 94534802115520] v30'185 uv183 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f3f50006070 con 0x7f3f48037070 2023-10-05T19:28:38.747 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.742+0000 7f3f612dde40 20 remove_watcher() i=3 2023-10-05T19:28:38.747 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.742+0000 7f3f612dde40 1 -- 172.21.15.167:0/1639253938 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:119 7.d 7:b4812045:::notify.4:head [watch unwatch cookie 94534802109440] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x55fa994a0700 con 0x55fa994ba7a0 2023-10-05T19:28:38.750 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.746+0000 7f3f5f613700 1 -- 172.21.15.167:0/1639253938 <== osd.1 v2:172.21.15.167:6800/30035 43 ==== osd_op_reply(119 notify.4 [watch unwatch cookie 94534802109440] v30'185 uv183 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f3f58050040 con 0x55fa994ba7a0 2023-10-05T19:28:38.750 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.746+0000 7f3f612dde40 20 remove_watcher() i=4 2023-10-05T19:28:38.750 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.746+0000 7f3f612dde40 1 -- 172.21.15.167:0/1639253938 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:120 7.3 7:c609908c:::notify.5:head [watch unwatch cookie 94534802116624] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x55fa994a0700 con 0x7f3f5000d8e0 2023-10-05T19:28:38.753 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.746+0000 7f3f5f613700 1 -- 172.21.15.167:0/1639253938 <== osd.0 v2:172.21.15.118:6800/27690 33 ==== osd_op_reply(120 notify.5 [watch unwatch cookie 94534802116624] v30'185 uv183 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f3f58050040 con 0x7f3f5000d8e0 2023-10-05T19:28:38.753 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.746+0000 7f3f612dde40 20 remove_watcher() i=5 2023-10-05T19:28:38.753 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.746+0000 7f3f612dde40 1 -- 172.21.15.167:0/1639253938 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:121 7.14 7:2b04a3e9:::notify.6:head [watch unwatch cookie 94534802123808] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x55fa994a0700 con 0x7f3f5000d8e0 2023-10-05T19:28:38.757 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.750+0000 7f3f5f613700 1 -- 172.21.15.167:0/1639253938 <== osd.0 v2:172.21.15.118:6800/27690 34 ==== osd_op_reply(121 notify.6 [watch unwatch cookie 94534802123808] v30'185 uv183 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f3f58050040 con 0x7f3f5000d8e0 2023-10-05T19:28:38.757 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.750+0000 7f3f612dde40 20 remove_watcher() i=6 2023-10-05T19:28:38.757 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.750+0000 7f3f612dde40 1 -- 172.21.15.167:0/1639253938 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:122 7.9 7:93e5b521:::notify.7:head [watch unwatch cookie 94534802118832] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x55fa994a0700 con 0x7f3f5000d8e0 2023-10-05T19:28:38.760 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.754+0000 7f3f5f613700 1 -- 172.21.15.167:0/1639253938 <== osd.0 v2:172.21.15.118:6800/27690 35 ==== osd_op_reply(122 notify.7 [watch unwatch cookie 94534802118832] v30'185 uv183 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f3f58050040 con 0x7f3f5000d8e0 2023-10-05T19:28:38.760 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.754+0000 7f3f612dde40 20 remove_watcher() i=7 2023-10-05T19:28:38.761 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.754+0000 7f3eabfff700 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2023-10-05T19:28:38.762 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.754+0000 7f3f612dde40 1 -- 172.21.15.167:0/1639253938 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:123 6.4 6:22d26bf9:::data_loggenerations_metadata:head [watch unwatch cookie 94534802127488] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x55fa994a0700 con 0x55fa994ba7a0 2023-10-05T19:28:38.765 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.758+0000 7f3f5f613700 1 -- 172.21.15.167:0/1639253938 <== osd.1 v2:172.21.15.167:6800/30035 44 ==== osd_op_reply(123 data_loggenerations_metadata [watch unwatch cookie 94534802127488] v30'170 uv1 ondisk = 0) v8 ==== 172+0+0 (crc 0 0 0) 0x7f3f58050040 con 0x55fa994ba7a0 2023-10-05T19:28:38.765 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.758+0000 7f3f612dde40 1 -- 172.21.15.167:0/1639253938 >> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] conn(0x7f3f5000d8e0 msgr2=0x7f3f5000c360 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-05T19:28:38.766 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.758+0000 7f3f612dde40 1 --2- 172.21.15.167:0/1639253938 >> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] conn(0x7f3f5000d8e0 0x7f3f5000c360 crc :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-05T19:28:38.766 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.758+0000 7f3f612dde40 1 -- 172.21.15.167:0/1639253938 >> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] conn(0x55fa994ba7a0 msgr2=0x55fa994bcb80 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-05T19:28:38.766 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.758+0000 7f3f612dde40 1 --2- 172.21.15.167:0/1639253938 >> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] conn(0x55fa994ba7a0 0x55fa994bcb80 crc :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-05T19:28:38.766 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.758+0000 7f3f612dde40 1 -- 172.21.15.167:0/1639253938 >> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] conn(0x7f3f48037070 msgr2=0x7f3f480183b0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-05T19:28:38.766 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.758+0000 7f3f612dde40 1 --2- 172.21.15.167:0/1639253938 >> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] conn(0x7f3f48037070 0x7f3f480183b0 crc :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-05T19:28:38.766 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.758+0000 7f3f612dde40 1 -- 172.21.15.167:0/1639253938 >> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] conn(0x55fa994c0920 msgr2=0x55fa994c2d00 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-05T19:28:38.766 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.758+0000 7f3f612dde40 1 --2- 172.21.15.167:0/1639253938 >> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] conn(0x55fa994c0920 0x55fa994c2d00 crc :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-05T19:28:38.766 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.758+0000 7f3f612dde40 1 -- 172.21.15.167:0/1639253938 >> [v2:172.21.15.167:6824/29889,v1:172.21.15.167:6825/29889] conn(0x7f3f40030c50 msgr2=0x7f3f40033100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-05T19:28:38.766 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.758+0000 7f3f612dde40 1 --2- 172.21.15.167:0/1639253938 >> [v2:172.21.15.167:6824/29889,v1:172.21.15.167:6825/29889] conn(0x7f3f40030c50 0x7f3f40033100 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7f3f50001e90 tx=0x7f3f500013b0 comp rx=0 tx=0).stop 2023-10-05T19:28:38.766 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.758+0000 7f3f5f613700 1 -- 172.21.15.167:0/1639253938 reap_dead start 2023-10-05T19:28:38.767 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.758+0000 7f3f612dde40 1 -- 172.21.15.167:0/1639253938 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55fa9948b740 msgr2=0x55fa994b0c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-05T19:28:38.767 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.758+0000 7f3f612dde40 1 --2- 172.21.15.167:0/1639253938 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55fa9948b740 0x55fa994b0c90 secure :-1 s=READY pgs=503 cs=0 l=1 rev1=1 crypto rx=0x7f3f480058c0 tx=0x7f3f4801d000 comp rx=0 tx=0).stop 2023-10-05T19:28:38.767 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.758+0000 7f3f612dde40 1 -- 172.21.15.167:0/1639253938 shutdown_connections 2023-10-05T19:28:38.767 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.758+0000 7f3f612dde40 1 --2- 172.21.15.167:0/1639253938 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55fa9948b740 0x55fa994b0c90 unknown :-1 s=CLOSED pgs=503 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-05T19:28:38.767 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.758+0000 7f3f612dde40 1 -- 172.21.15.167:0/1639253938 >> 172.21.15.167:0/1639253938 conn(0x55fa99486850 msgr2=0x55fa994a44a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-05T19:28:38.767 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.758+0000 7f3f612dde40 1 -- 172.21.15.167:0/1639253938 shutdown_connections 2023-10-05T19:28:38.767 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.758+0000 7f3f612dde40 1 -- 172.21.15.167:0/1639253938 wait complete. 2023-10-05T19:28:38.768 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.762+0000 7f3f612dde40 1 -- 172.21.15.167:0/2502269580 >> [v2:172.21.15.167:6824/29889,v1:172.21.15.167:6825/29889] conn(0x7f3e9c030c00 msgr2=0x7f3e9c0330b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-05T19:28:38.768 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.762+0000 7f3f612dde40 1 --2- 172.21.15.167:0/2502269580 >> [v2:172.21.15.167:6824/29889,v1:172.21.15.167:6825/29889] conn(0x7f3e9c030c00 0x7f3e9c0330b0 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x55fa994cb240 tx=0x7f3f58000f90 comp rx=0 tx=0).stop 2023-10-05T19:28:38.768 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.762+0000 7f3f612dde40 1 -- 172.21.15.167:0/2502269580 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55fa994d9420 msgr2=0x55fa99400af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-05T19:28:38.768 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.762+0000 7f3f612dde40 1 --2- 172.21.15.167:0/2502269580 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55fa994d9420 0x55fa99400af0 secure :-1 s=READY pgs=505 cs=0 l=1 rev1=1 crypto rx=0x55fa994be080 tx=0x7f3f50030000 comp rx=0 tx=0).stop 2023-10-05T19:28:38.768 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.762+0000 7f3f612dde40 1 -- 172.21.15.167:0/2502269580 shutdown_connections 2023-10-05T19:28:38.769 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.762+0000 7f3f612dde40 1 --2- 172.21.15.167:0/2502269580 >> [v2:172.21.15.167:6824/29889,v1:172.21.15.167:6825/29889] conn(0x7f3e9c030c00 0x7f3e9c0330b0 unknown :-1 s=CLOSED pgs=175 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-05T19:28:38.769 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.762+0000 7f3f612dde40 1 --2- 172.21.15.167:0/2502269580 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55fa994d9420 0x55fa99400af0 unknown :-1 s=CLOSED pgs=505 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-05T19:28:38.769 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.762+0000 7f3f612dde40 1 -- 172.21.15.167:0/2502269580 >> 172.21.15.167:0/2502269580 conn(0x55fa994d47e0 msgr2=0x55fa994d2880 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-05T19:28:38.769 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.762+0000 7f3f612dde40 1 -- 172.21.15.167:0/2502269580 shutdown_connections 2023-10-05T19:28:38.769 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.762+0000 7f3f612dde40 1 -- 172.21.15.167:0/2502269580 wait complete. 2023-10-05T19:28:38.773 DEBUG:tasks.util.rgw: json result: {'user_id': 'foo', 'display_name': 'Foo', 'email': 'foo@foo.com', 'suspended': 1, 'max_buckets': 4, 'subusers': [], 'keys': [{'user': 'foo', 'access_key': '9te6NH5mcdcq0Tc5i8i1', 'secret_key': 'Ny4IOauQoL18Gp2zM7lC1vLmoawgqcYP/YGcWfXu'}], 'swift_keys': [], 'caps': [], 'op_mask': 'read, write, delete', 'default_placement': '', 'default_storage_class': '', 'placement_tags': [], 'bucket_quota': {'enabled': False, 'check_on_raw': False, 'max_size': -1, 'max_size_kb': 0, 'max_objects': -1}, 'user_quota': {'enabled': False, 'check_on_raw': False, 'max_size': -1, 'max_size_kb': 0, 'max_objects': -1}, 'temp_url_keys': [], 'type': 'rgw', 'mfa_ids': []} 2023-10-05T19:28:38.775 INFO:tasks.util.rgw:rgwadmin: client.0 : ['user', 'enable', '--uid', 'foo'] 2023-10-05T19:28:38.775 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'ceph', 'user', 'enable', '--uid', 'foo'] 2023-10-05T19:28:38.775 DEBUG:teuthology.orchestra.run.smithi167:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster ceph user enable --uid foo 2023-10-05T19:28:38.818 INFO:teuthology.orchestra.run.smithi167.stderr:ignoring --setuser ceph since I am not root 2023-10-05T19:28:38.819 INFO:teuthology.orchestra.run.smithi167.stderr:ignoring --setgroup ceph since I am not root 2023-10-05T19:28:38.819 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.814+0000 7f32bf325e40 1 Processor -- start 2023-10-05T19:28:38.819 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.814+0000 7f32bf325e40 1 -- start start 2023-10-05T19:28:38.820 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.814+0000 7f32bf325e40 1 --2- >> v2:172.21.15.118:3300/0 conn(0x561e229531a0 0x561e229e6f40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-05T19:28:38.820 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.814+0000 7f32bf325e40 1 -- --> v1:172.21.15.118:6789/0 -- auth(proto 0 26 bytes epoch 0) v1 -- 0x561e2292abd0 con 0x561e229544b0 2023-10-05T19:28:38.820 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.814+0000 7f32bf325e40 1 -- --> v2:172.21.15.118:3300/0 -- mon_getmap magic: 0 v1 -- 0x561e228e7ee0 con 0x561e229531a0 2023-10-05T19:28:38.821 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.814+0000 7f32b7fff700 1 --2- >> v2:172.21.15.118:3300/0 conn(0x561e229531a0 0x561e229e6f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-05T19:28:38.821 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.814+0000 7f32b7fff700 1 --2- >> v2:172.21.15.118:3300/0 conn(0x561e229531a0 0x561e229e6f40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.118:3300/0 says I am v2:172.21.15.167:42278/0 (socket says 172.21.15.167:42278) 2023-10-05T19:28:38.821 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.814+0000 7f32b7fff700 1 -- 172.21.15.167:0/2102181973 learned_addr learned my addr 172.21.15.167:0/2102181973 (peer_addr_for_me v2:172.21.15.167:0/0) 2023-10-05T19:28:38.821 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.814+0000 7f32b7fff700 1 -- 172.21.15.167:0/2102181973 >> v1:172.21.15.118:6789/0 conn(0x561e229544b0 legacy=0x561e22952a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-05T19:28:38.821 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.814+0000 7f32b7fff700 1 -- 172.21.15.167:0/2102181973 --> v2:172.21.15.118:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x561e22954940 con 0x561e229531a0 2023-10-05T19:28:38.822 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.814+0000 7f32b7fff700 1 --2- 172.21.15.167:0/2102181973 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x561e229531a0 0x561e229e6f40 secure :-1 s=READY pgs=507 cs=0 l=1 rev1=1 crypto rx=0x7f32a800a650 tx=0x7f32a8005b10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f000050ce29ae513 server_cookie=0 in_seq=0 out_seq=0 2023-10-05T19:28:38.822 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.814+0000 7f32b77fe700 1 -- 172.21.15.167:0/2102181973 <== mon.0 v2:172.21.15.118:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f32a8008760 con 0x561e229531a0 2023-10-05T19:28:38.822 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.818+0000 7f32b77fe700 1 -- 172.21.15.167:0/2102181973 <== mon.0 v2:172.21.15.118:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f32a80088c0 con 0x561e229531a0 2023-10-05T19:28:38.823 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.818+0000 7f32bf325e40 1 -- 172.21.15.167:0/2102181973 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x561e229531a0 msgr2=0x561e229e6f40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-05T19:28:38.823 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.818+0000 7f32bf325e40 1 --2- 172.21.15.167:0/2102181973 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x561e229531a0 0x561e229e6f40 secure :-1 s=READY pgs=507 cs=0 l=1 rev1=1 crypto rx=0x7f32a800a650 tx=0x7f32a8005b10 comp rx=0 tx=0).stop 2023-10-05T19:28:38.823 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.818+0000 7f32bf325e40 1 -- 172.21.15.167:0/2102181973 shutdown_connections 2023-10-05T19:28:38.824 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.818+0000 7f32bf325e40 1 --2- 172.21.15.167:0/2102181973 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x561e229531a0 0x561e229e6f40 unknown :-1 s=CLOSED pgs=507 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-05T19:28:38.824 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.818+0000 7f32bf325e40 1 -- 172.21.15.167:0/2102181973 >> 172.21.15.167:0/2102181973 conn(0x561e229d9830 msgr2=0x561e229dbc40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-05T19:28:38.824 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.818+0000 7f32bf325e40 1 -- 172.21.15.167:0/2102181973 shutdown_connections 2023-10-05T19:28:38.824 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.818+0000 7f32bf325e40 1 -- 172.21.15.167:0/2102181973 wait complete. 2023-10-05T19:28:38.825 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.818+0000 7f32bf325e40 1 Processor -- start 2023-10-05T19:28:38.825 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.818+0000 7f32bf325e40 1 -- start start 2023-10-05T19:28:38.825 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.818+0000 7f32bf325e40 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x561e229531a0 0x561e229feff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-05T19:28:38.825 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.818+0000 7f32bf325e40 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x561e228ad3d0 con 0x561e229531a0 2023-10-05T19:28:38.826 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.818+0000 7f32bce5a700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x561e229531a0 0x561e229feff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-05T19:28:38.826 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.818+0000 7f32bce5a700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x561e229531a0 0x561e229feff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.118:3300/0 says I am v2:172.21.15.167:42292/0 (socket says 172.21.15.167:42292) 2023-10-05T19:28:38.826 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.818+0000 7f32bce5a700 1 -- 172.21.15.167:0/928644297 learned_addr learned my addr 172.21.15.167:0/928644297 (peer_addr_for_me v2:172.21.15.167:0/0) 2023-10-05T19:28:38.826 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.818+0000 7f32bce5a700 1 -- 172.21.15.167:0/928644297 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f32a8005db0 con 0x561e229531a0 2023-10-05T19:28:38.826 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.822+0000 7f32bce5a700 1 --2- 172.21.15.167:0/928644297 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x561e229531a0 0x561e229feff0 secure :-1 s=READY pgs=508 cs=0 l=1 rev1=1 crypto rx=0x7f32b0001d30 tx=0x7f32b0001900 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7faea6c847d69aac server_cookie=0 in_seq=0 out_seq=0 2023-10-05T19:28:38.827 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.822+0000 7f32b67fc700 1 -- 172.21.15.167:0/928644297 <== mon.0 v2:172.21.15.118:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f32b0014070 con 0x561e229531a0 2023-10-05T19:28:38.828 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.822+0000 7f32b67fc700 1 -- 172.21.15.167:0/928644297 <== mon.0 v2:172.21.15.118:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f32b0008070 con 0x561e229531a0 2023-10-05T19:28:38.828 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.822+0000 7f32b67fc700 1 -- 172.21.15.167:0/928644297 <== mon.0 v2:172.21.15.118:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f32b000bd30 con 0x561e229531a0 2023-10-05T19:28:38.828 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.822+0000 7f32bf325e40 1 -- 172.21.15.167:0/928644297 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x561e229531a0 msgr2=0x561e229feff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-05T19:28:38.828 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.822+0000 7f32bf325e40 1 --2- 172.21.15.167:0/928644297 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x561e229531a0 0x561e229feff0 secure :-1 s=READY pgs=508 cs=0 l=1 rev1=1 crypto rx=0x7f32b0001d30 tx=0x7f32b0001900 comp rx=0 tx=0).stop 2023-10-05T19:28:38.828 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.822+0000 7f32bf325e40 1 -- 172.21.15.167:0/928644297 shutdown_connections 2023-10-05T19:28:38.828 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.822+0000 7f32bf325e40 1 --2- 172.21.15.167:0/928644297 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x561e229531a0 0x561e229feff0 unknown :-1 s=CLOSED pgs=508 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-05T19:28:38.828 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.822+0000 7f32bf325e40 1 -- 172.21.15.167:0/928644297 >> 172.21.15.167:0/928644297 conn(0x561e229d9830 msgr2=0x561e229fabb0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-05T19:28:38.828 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.822+0000 7f32bf325e40 1 -- 172.21.15.167:0/928644297 shutdown_connections 2023-10-05T19:28:38.828 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.822+0000 7f32bf325e40 1 -- 172.21.15.167:0/928644297 wait complete. 2023-10-05T19:28:38.829 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.822+0000 7f32bf325e40 1 Processor -- start 2023-10-05T19:28:38.830 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.822+0000 7f32bf325e40 1 -- start start 2023-10-05T19:28:38.830 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.822+0000 7f32bf325e40 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x561e229531a0 0x561e22a05d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-05T19:28:38.830 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.822+0000 7f32bf325e40 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x561e228abe30 con 0x561e229531a0 2023-10-05T19:28:38.830 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.822+0000 7f32bce5a700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x561e229531a0 0x561e22a05d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-05T19:28:38.830 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.822+0000 7f32bce5a700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x561e229531a0 0x561e22a05d80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.118:3300/0 says I am v2:172.21.15.167:42294/0 (socket says 172.21.15.167:42294) 2023-10-05T19:28:38.830 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.822+0000 7f32bce5a700 1 -- 172.21.15.167:0/3986483093 learned_addr learned my addr 172.21.15.167:0/3986483093 (peer_addr_for_me v2:172.21.15.167:0/0) 2023-10-05T19:28:38.830 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.822+0000 7f32bce5a700 1 -- 172.21.15.167:0/3986483093 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f32b000a040 con 0x561e229531a0 2023-10-05T19:28:38.830 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.822+0000 7f32bce5a700 1 --2- 172.21.15.167:0/3986483093 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x561e229531a0 0x561e22a05d80 secure :-1 s=READY pgs=509 cs=0 l=1 rev1=1 crypto rx=0x7f32b000b850 tx=0x7f32b0016000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-05T19:28:38.831 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.826+0000 7f32b4ff9700 1 -- 172.21.15.167:0/3986483093 <== mon.0 v2:172.21.15.118:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f32b001a480 con 0x561e229531a0 2023-10-05T19:28:38.831 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.826+0000 7f32b4ff9700 1 -- 172.21.15.167:0/3986483093 <== mon.0 v2:172.21.15.118:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f32b0021070 con 0x561e229531a0 2023-10-05T19:28:38.831 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.826+0000 7f32bf325e40 1 -- 172.21.15.167:0/3986483093 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x561e22a062c0 con 0x561e229531a0 2023-10-05T19:28:38.831 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.826+0000 7f32b4ff9700 1 -- 172.21.15.167:0/3986483093 <== mon.0 v2:172.21.15.118:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f32b001a760 con 0x561e229531a0 2023-10-05T19:28:38.831 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.826+0000 7f32bf325e40 1 -- 172.21.15.167:0/3986483093 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x561e22a06760 con 0x561e229531a0 2023-10-05T19:28:38.832 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.826+0000 7f32b4ff9700 1 -- 172.21.15.167:0/3986483093 <== mon.0 v2:172.21.15.118:3300/0 4 ==== mgrmap(e 5) v1 ==== 40329+0+0 (secure 0 0 0) 0x7f32b001a910 con 0x561e229531a0 2023-10-05T19:28:38.833 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.826+0000 7f32bf325e40 20 rados->read ofs=0 len=0 2023-10-05T19:28:38.833 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.826+0000 7f32b4ff9700 1 --2- 172.21.15.167:0/3986483093 >> [v2:172.21.15.167:6824/29889,v1:172.21.15.167:6825/29889] conn(0x7f32a0030c40 0x7f32a00330f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-05T19:28:38.833 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.826+0000 7f32b4ff9700 1 -- 172.21.15.167:0/3986483093 <== mon.0 v2:172.21.15.118:3300/0 5 ==== osd_map(30..30 src has 1..30) v4 ==== 6057+0+0 (secure 0 0 0) 0x7f32b0044d50 con 0x561e229531a0 2023-10-05T19:28:38.833 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.826+0000 7f32b7fff700 1 --2- 172.21.15.167:0/3986483093 >> [v2:172.21.15.167:6824/29889,v1:172.21.15.167:6825/29889] conn(0x7f32a0030c40 0x7f32a00330f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-05T19:28:38.833 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.826+0000 7f32bf325e40 1 --2- 172.21.15.167:0/3986483093 >> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] conn(0x561e229544b0 0x561e22a11880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-05T19:28:38.833 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.826+0000 7f32bf325e40 1 -- 172.21.15.167:0/3986483093 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:1 5.12 5:49953fa1:::default.realm:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x561e22a11f30 con 0x561e229544b0 2023-10-05T19:28:38.833 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.826+0000 7f32bd65b700 1 --2- 172.21.15.167:0/3986483093 >> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] conn(0x561e229544b0 0x561e22a11880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-05T19:28:38.836 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.826+0000 7f32b7fff700 1 --2- 172.21.15.167:0/3986483093 >> [v2:172.21.15.167:6824/29889,v1:172.21.15.167:6825/29889] conn(0x7f32a0030c40 0x7f32a00330f0 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7f32a8005810 tx=0x7f32a8015000 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-05T19:28:38.836 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.826+0000 7f32bd65b700 1 --2- 172.21.15.167:0/3986483093 >> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] conn(0x561e229544b0 0x561e22a11880 crc :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-05T19:28:38.836 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.826+0000 7f32bd65b700 1 -- 172.21.15.167:0/3986483093 <== osd.1 v2:172.21.15.167:6800/30035 1 ==== osd_op_reply(1 default.realm [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 ==== 157+0+0 (crc 0 0 0) 0x7f32b800a5a0 con 0x561e229544b0 2023-10-05T19:28:38.836 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.826+0000 7f32bf325e40 20 rados_obj.operate() r=-2 bl.length=0 2023-10-05T19:28:38.836 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.826+0000 7f32bf325e40 20 realm 2023-10-05T19:28:38.836 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.826+0000 7f32bf325e40 20 rados->read ofs=0 len=0 2023-10-05T19:28:38.837 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.826+0000 7f32bf325e40 1 -- 172.21.15.167:0/3986483093 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:2 5.12 5:49953fa1:::default.realm:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x561e22a0eea0 con 0x561e229544b0 2023-10-05T19:28:38.837 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.830+0000 7f32bd65b700 1 -- 172.21.15.167:0/3986483093 <== osd.1 v2:172.21.15.167:6800/30035 2 ==== osd_op_reply(2 default.realm [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 ==== 157+0+0 (crc 0 0 0) 0x7f32b800a5a0 con 0x561e229544b0 2023-10-05T19:28:38.837 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.830+0000 7f32bf325e40 20 rados_obj.operate() r=-2 bl.length=0 2023-10-05T19:28:38.837 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.830+0000 7f32bf325e40 4 RGWPeriod::init failed to init realm id : (2) No such file or directory 2023-10-05T19:28:38.837 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.830+0000 7f32bf325e40 20 rados->read ofs=0 len=0 2023-10-05T19:28:38.837 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.830+0000 7f32bf325e40 1 -- 172.21.15.167:0/3986483093 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:3 5.12 5:49953fa1:::default.realm:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x561e22a12a10 con 0x561e229544b0 2023-10-05T19:28:38.837 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.830+0000 7f32bd65b700 1 -- 172.21.15.167:0/3986483093 <== osd.1 v2:172.21.15.167:6800/30035 3 ==== osd_op_reply(3 default.realm [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 ==== 157+0+0 (crc 0 0 0) 0x7f32b800a5a0 con 0x561e229544b0 2023-10-05T19:28:38.837 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.830+0000 7f32bf325e40 20 rados_obj.operate() r=-2 bl.length=0 2023-10-05T19:28:38.837 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.830+0000 7f32bf325e40 20 rados->read ofs=0 len=0 2023-10-05T19:28:38.837 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.830+0000 7f32bf325e40 1 -- 172.21.15.167:0/3986483093 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:4 5.f 5:f43b6ece:::zone_names.default:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x561e22a125f0 con 0x561e229544b0 2023-10-05T19:28:38.837 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.830+0000 7f32bd65b700 1 -- 172.21.15.167:0/3986483093 <== osd.1 v2:172.21.15.167:6800/30035 4 ==== osd_op_reply(4 zone_names.default [read 0~46 out=46b] v0'0 uv1 ondisk = 0) v8 ==== 162+0+46 (crc 0 0 0) 0x7f32b800a5a0 con 0x561e229544b0 2023-10-05T19:28:38.837 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.830+0000 7f32bf325e40 20 rados_obj.operate() r=0 bl.length=46 2023-10-05T19:28:38.838 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.830+0000 7f32bf325e40 20 rados->read ofs=0 len=0 2023-10-05T19:28:38.838 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.830+0000 7f32bf325e40 1 -- 172.21.15.167:0/3986483093 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:5 5.1d 5:b8374eb4:::zone_info.9a9df965-6612-45d4-a45f-6e3c889bacc2:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x561e22a135b0 con 0x561e229544b0 2023-10-05T19:28:38.838 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.830+0000 7f32bd65b700 1 -- 172.21.15.167:0/3986483093 <== osd.1 v2:172.21.15.167:6800/30035 5 ==== osd_op_reply(5 zone_info.9a9df965-6612-45d4-a45f-6e3c889bacc2 [read 0~874 out=874b] v0'0 uv2 ondisk = 0) v8 ==== 190+0+874 (crc 0 0 0) 0x7f32b800a5a0 con 0x561e229544b0 2023-10-05T19:28:38.838 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.830+0000 7f32bf325e40 20 rados_obj.operate() r=0 bl.length=874 2023-10-05T19:28:38.838 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.830+0000 7f32bf325e40 20 searching for the correct realm 2023-10-05T19:28:38.838 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.830+0000 7f32bf325e40 1 --2- 172.21.15.167:0/3986483093 >> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] conn(0x561e22a154f0 0x561e22a17900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-05T19:28:38.838 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.830+0000 7f32bf325e40 1 -- 172.21.15.167:0/3986483093 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:6 5.0 5:00000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x561e22a17f10 con 0x561e22a154f0 2023-10-05T19:28:38.838 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.830+0000 7f32bce5a700 1 --2- 172.21.15.167:0/3986483093 >> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] conn(0x561e22a154f0 0x561e22a17900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-05T19:28:38.839 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.830+0000 7f32bce5a700 1 --2- 172.21.15.167:0/3986483093 >> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] conn(0x561e22a154f0 0x561e22a17900 crc :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-05T19:28:38.840 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.830+0000 7f32bce5a700 1 -- 172.21.15.167:0/3986483093 <== osd.3 v2:172.21.15.167:6808/30036 1 ==== osd_op_reply(6 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f32b0016070 con 0x561e22a154f0 2023-10-05T19:28:38.840 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.834+0000 7f32bce5a700 1 --2- 172.21.15.167:0/3986483093 >> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] conn(0x7f32b0017540 0x7f32b0042780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-05T19:28:38.840 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.834+0000 7f32bce5a700 1 -- 172.21.15.167:0/3986483093 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:7 5.10 5:08000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x561e22a17f10 con 0x7f32b0017540 2023-10-05T19:28:38.840 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.834+0000 7f32b7fff700 1 --2- 172.21.15.167:0/3986483093 >> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] conn(0x7f32b0017540 0x7f32b0042780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-05T19:28:38.840 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.834+0000 7f32b7fff700 1 --2- 172.21.15.167:0/3986483093 >> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] conn(0x7f32b0017540 0x7f32b0042780 crc :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-05T19:28:38.840 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.834+0000 7f32b7fff700 1 -- 172.21.15.167:0/3986483093 <== osd.2 v2:172.21.15.167:6816/30037 1 ==== osd_op_reply(7 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f32a8012820 con 0x7f32b0017540 2023-10-05T19:28:38.840 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.834+0000 7f32b7fff700 1 --2- 172.21.15.167:0/3986483093 >> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] conn(0x7f32a8007030 0x7f32a801ca90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-05T19:28:38.840 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.834+0000 7f32b7fff700 1 -- 172.21.15.167:0/3986483093 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:8 5.8 5:10000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x561e22a17f10 con 0x7f32a8007030 2023-10-05T19:28:38.840 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.834+0000 7f32bd65b700 1 --2- 172.21.15.167:0/3986483093 >> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] conn(0x7f32a8007030 0x7f32a801ca90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-05T19:28:38.841 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.834+0000 7f32bd65b700 1 --2- 172.21.15.167:0/3986483093 >> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] conn(0x7f32a8007030 0x7f32a801ca90 crc :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-05T19:28:38.841 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.834+0000 7f32bd65b700 1 -- 172.21.15.167:0/3986483093 <== osd.0 v2:172.21.15.118:6800/27690 1 ==== osd_op_reply(8 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f32b800a5a0 con 0x7f32a8007030 2023-10-05T19:28:38.841 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.834+0000 7f32bd65b700 1 -- 172.21.15.167:0/3986483093 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:9 5.18 5:18000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x561e22a17f10 con 0x561e229544b0 2023-10-05T19:28:38.842 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.834+0000 7f32bd65b700 1 -- 172.21.15.167:0/3986483093 <== osd.1 v2:172.21.15.167:6800/30035 6 ==== osd_op_reply(9 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f32b800a5a0 con 0x561e229544b0 2023-10-05T19:28:38.842 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.834+0000 7f32bd65b700 1 -- 172.21.15.167:0/3986483093 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:10 5.4 5:20000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x561e22a17f10 con 0x561e229544b0 2023-10-05T19:28:38.842 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.838+0000 7f32bd65b700 1 -- 172.21.15.167:0/3986483093 <== osd.1 v2:172.21.15.167:6800/30035 7 ==== osd_op_reply(10 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f32b800a5a0 con 0x561e229544b0 2023-10-05T19:28:38.842 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.838+0000 7f32bd65b700 1 -- 172.21.15.167:0/3986483093 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:11 5.14 5:28000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x561e22a17f10 con 0x7f32a8007030 2023-10-05T19:28:38.843 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.838+0000 7f32bd65b700 1 -- 172.21.15.167:0/3986483093 <== osd.0 v2:172.21.15.118:6800/27690 2 ==== osd_op_reply(11 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f32b800a5a0 con 0x7f32a8007030 2023-10-05T19:28:38.843 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.838+0000 7f32bd65b700 1 -- 172.21.15.167:0/3986483093 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:12 5.c 5:30000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x561e22a17f10 con 0x7f32a8007030 2023-10-05T19:28:38.844 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.838+0000 7f32bd65b700 1 -- 172.21.15.167:0/3986483093 <== osd.0 v2:172.21.15.118:6800/27690 3 ==== osd_op_reply(12 [pgnls start_epoch 30 out=112b] v23'1 uv1 ondisk = 1) v8 ==== 144+0+112 (crc 0 0 0) 0x7f32b800a5a0 con 0x7f32a8007030 2023-10-05T19:28:38.844 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.838+0000 7f32bd65b700 1 -- 172.21.15.167:0/3986483093 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:13 5.1c 5:38000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x561e22a17f10 con 0x7f32b0017540 2023-10-05T19:28:38.844 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.838+0000 7f32b7fff700 1 -- 172.21.15.167:0/3986483093 <== osd.2 v2:172.21.15.167:6816/30037 2 ==== osd_op_reply(13 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f32a8012820 con 0x7f32b0017540 2023-10-05T19:28:38.844 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.838+0000 7f32b7fff700 1 -- 172.21.15.167:0/3986483093 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:14 5.2 5:40000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x561e22a17f10 con 0x561e22a154f0 2023-10-05T19:28:38.845 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.838+0000 7f32bce5a700 1 -- 172.21.15.167:0/3986483093 <== osd.3 v2:172.21.15.167:6808/30036 2 ==== osd_op_reply(14 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f32b0016070 con 0x561e22a154f0 2023-10-05T19:28:38.845 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.838+0000 7f32bce5a700 1 -- 172.21.15.167:0/3986483093 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:15 5.12 5:48000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x561e22a17f10 con 0x561e229544b0 2023-10-05T19:28:38.845 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.838+0000 7f32bd65b700 1 -- 172.21.15.167:0/3986483093 <== osd.1 v2:172.21.15.167:6800/30035 8 ==== osd_op_reply(15 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f32b800a5a0 con 0x561e229544b0 2023-10-05T19:28:38.846 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.838+0000 7f32bd65b700 1 -- 172.21.15.167:0/3986483093 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:16 5.a 5:50000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x561e22a17f10 con 0x7f32a8007030 2023-10-05T19:28:38.846 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.842+0000 7f32bd65b700 1 -- 172.21.15.167:0/3986483093 <== osd.0 v2:172.21.15.118:6800/27690 4 ==== osd_op_reply(16 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f32b800a5a0 con 0x7f32a8007030 2023-10-05T19:28:38.847 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.842+0000 7f32bd65b700 1 -- 172.21.15.167:0/3986483093 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:17 5.1a 5:58000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x561e22a17f10 con 0x561e229544b0 2023-10-05T19:28:38.847 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.842+0000 7f32bd65b700 1 -- 172.21.15.167:0/3986483093 <== osd.1 v2:172.21.15.167:6800/30035 9 ==== osd_op_reply(17 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f32b800a5a0 con 0x561e229544b0 2023-10-05T19:28:38.847 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.842+0000 7f32bd65b700 1 -- 172.21.15.167:0/3986483093 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:18 5.6 5:60000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x561e22a17f10 con 0x7f32b0017540 2023-10-05T19:28:38.848 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.842+0000 7f32b7fff700 1 -- 172.21.15.167:0/3986483093 <== osd.2 v2:172.21.15.167:6816/30037 3 ==== osd_op_reply(18 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f32a8012820 con 0x7f32b0017540 2023-10-05T19:28:38.848 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.842+0000 7f32b7fff700 1 -- 172.21.15.167:0/3986483093 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:19 5.16 5:68000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x561e22a17f10 con 0x561e22a154f0 2023-10-05T19:28:38.848 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.842+0000 7f32bce5a700 1 -- 172.21.15.167:0/3986483093 <== osd.3 v2:172.21.15.167:6808/30036 3 ==== osd_op_reply(19 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f32b0016070 con 0x561e22a154f0 2023-10-05T19:28:38.849 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.842+0000 7f32bce5a700 1 -- 172.21.15.167:0/3986483093 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:20 5.e 5:70000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x561e22a17f10 con 0x561e22a154f0 2023-10-05T19:28:38.849 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.842+0000 7f32bce5a700 1 -- 172.21.15.167:0/3986483093 <== osd.3 v2:172.21.15.167:6808/30036 4 ==== osd_op_reply(20 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f32b0016070 con 0x561e22a154f0 2023-10-05T19:28:38.849 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.842+0000 7f32bce5a700 1 -- 172.21.15.167:0/3986483093 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:21 5.1e 5:78000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x561e22a17f10 con 0x7f32a8007030 2023-10-05T19:28:38.850 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.846+0000 7f32bd65b700 1 -- 172.21.15.167:0/3986483093 <== osd.0 v2:172.21.15.118:6800/27690 5 ==== osd_op_reply(21 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f32b800a5a0 con 0x7f32a8007030 2023-10-05T19:28:38.850 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.846+0000 7f32bd65b700 1 -- 172.21.15.167:0/3986483093 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:22 5.1 5:80000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x561e22a17f10 con 0x561e22a154f0 2023-10-05T19:28:38.851 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.846+0000 7f32bce5a700 1 -- 172.21.15.167:0/3986483093 <== osd.3 v2:172.21.15.167:6808/30036 5 ==== osd_op_reply(22 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f32b0016070 con 0x561e22a154f0 2023-10-05T19:28:38.851 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.846+0000 7f32bce5a700 1 -- 172.21.15.167:0/3986483093 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:23 5.11 5:88000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x561e22a17f10 con 0x561e22a154f0 2023-10-05T19:28:38.851 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.846+0000 7f32bce5a700 1 -- 172.21.15.167:0/3986483093 <== osd.3 v2:172.21.15.167:6808/30036 6 ==== osd_op_reply(23 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f32b0016070 con 0x561e22a154f0 2023-10-05T19:28:38.851 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.846+0000 7f32bce5a700 1 -- 172.21.15.167:0/3986483093 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:24 5.9 5:90000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x561e22a17f10 con 0x561e229544b0 2023-10-05T19:28:38.852 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.846+0000 7f32bd65b700 1 -- 172.21.15.167:0/3986483093 <== osd.1 v2:172.21.15.167:6800/30035 10 ==== osd_op_reply(24 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f32b800a5a0 con 0x561e229544b0 2023-10-05T19:28:38.852 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.846+0000 7f32bd65b700 1 -- 172.21.15.167:0/3986483093 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:25 5.19 5:98000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x561e22a17f10 con 0x561e229544b0 2023-10-05T19:28:38.852 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.846+0000 7f32bd65b700 1 -- 172.21.15.167:0/3986483093 <== osd.1 v2:172.21.15.167:6800/30035 11 ==== osd_op_reply(25 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f32b800a5a0 con 0x561e229544b0 2023-10-05T19:28:38.853 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.846+0000 7f32bd65b700 1 -- 172.21.15.167:0/3986483093 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:26 5.5 5:a0000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x561e22a17f10 con 0x7f32a8007030 2023-10-05T19:28:38.853 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.846+0000 7f32bd65b700 1 -- 172.21.15.167:0/3986483093 <== osd.0 v2:172.21.15.118:6800/27690 6 ==== osd_op_reply(26 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f32b800a5a0 con 0x7f32a8007030 2023-10-05T19:28:38.854 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.846+0000 7f32bd65b700 1 -- 172.21.15.167:0/3986483093 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:27 5.15 5:a8000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x561e22a17f10 con 0x561e22a154f0 2023-10-05T19:28:38.854 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.850+0000 7f32bce5a700 1 -- 172.21.15.167:0/3986483093 <== osd.3 v2:172.21.15.167:6808/30036 7 ==== osd_op_reply(27 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f32b0016070 con 0x561e22a154f0 2023-10-05T19:28:38.854 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.850+0000 7f32bce5a700 1 -- 172.21.15.167:0/3986483093 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:28 5.d 5:b0000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x561e22a17f10 con 0x7f32b0017540 2023-10-05T19:28:38.855 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.850+0000 7f32b7fff700 1 -- 172.21.15.167:0/3986483093 <== osd.2 v2:172.21.15.167:6816/30037 4 ==== osd_op_reply(28 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f32a8012820 con 0x7f32b0017540 2023-10-05T19:28:38.855 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.850+0000 7f32b7fff700 1 -- 172.21.15.167:0/3986483093 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:29 5.1d 5:b8000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x561e22a17f10 con 0x561e229544b0 2023-10-05T19:28:38.856 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.850+0000 7f32bd65b700 1 -- 172.21.15.167:0/3986483093 <== osd.1 v2:172.21.15.167:6800/30035 12 ==== osd_op_reply(29 [pgnls start_epoch 30 out=107b] v30'2 uv2 ondisk = 1) v8 ==== 144+0+107 (crc 0 0 0) 0x7f32b800a5a0 con 0x561e229544b0 2023-10-05T19:28:38.856 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.850+0000 7f32bd65b700 1 -- 172.21.15.167:0/3986483093 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:30 5.3 5:c0000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x561e22a17f10 con 0x7f32a8007030 2023-10-05T19:28:38.857 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.850+0000 7f32bd65b700 1 -- 172.21.15.167:0/3986483093 <== osd.0 v2:172.21.15.118:6800/27690 7 ==== osd_op_reply(30 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f32b800a5a0 con 0x7f32a8007030 2023-10-05T19:28:38.857 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.850+0000 7f32bd65b700 1 -- 172.21.15.167:0/3986483093 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:31 5.13 5:c8000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x561e22a17f10 con 0x561e22a154f0 2023-10-05T19:28:38.858 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.850+0000 7f32bce5a700 1 -- 172.21.15.167:0/3986483093 <== osd.3 v2:172.21.15.167:6808/30036 8 ==== osd_op_reply(31 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f32b0016070 con 0x561e22a154f0 2023-10-05T19:28:38.858 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.850+0000 7f32bce5a700 1 -- 172.21.15.167:0/3986483093 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:32 5.b 5:d0000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x561e22a17f10 con 0x7f32a8007030 2023-10-05T19:28:38.859 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.854+0000 7f32bd65b700 1 -- 172.21.15.167:0/3986483093 <== osd.0 v2:172.21.15.118:6800/27690 8 ==== osd_op_reply(32 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f32b800a5a0 con 0x7f32a8007030 2023-10-05T19:28:38.859 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.854+0000 7f32bd65b700 1 -- 172.21.15.167:0/3986483093 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:33 5.1b 5:d8000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x561e22a17f10 con 0x7f32b0017540 2023-10-05T19:28:38.860 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.854+0000 7f32b7fff700 1 -- 172.21.15.167:0/3986483093 <== osd.2 v2:172.21.15.167:6816/30037 5 ==== osd_op_reply(33 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f32a8012820 con 0x7f32b0017540 2023-10-05T19:28:38.860 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.854+0000 7f32b7fff700 1 -- 172.21.15.167:0/3986483093 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:34 5.7 5:e0000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x561e22a17f10 con 0x561e22a154f0 2023-10-05T19:28:38.860 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.854+0000 7f32bce5a700 1 -- 172.21.15.167:0/3986483093 <== osd.3 v2:172.21.15.167:6808/30036 9 ==== osd_op_reply(34 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f32b0016070 con 0x561e22a154f0 2023-10-05T19:28:38.860 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.854+0000 7f32bce5a700 1 -- 172.21.15.167:0/3986483093 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:35 5.17 5:e8000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x561e22a17f10 con 0x7f32a8007030 2023-10-05T19:28:38.861 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.854+0000 7f32bd65b700 1 -- 172.21.15.167:0/3986483093 <== osd.0 v2:172.21.15.118:6800/27690 9 ==== osd_op_reply(35 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f32b800a5a0 con 0x7f32a8007030 2023-10-05T19:28:38.861 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.854+0000 7f32bd65b700 1 -- 172.21.15.167:0/3986483093 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:36 5.f 5:f0000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x561e22a17f10 con 0x561e229544b0 2023-10-05T19:28:38.862 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.858+0000 7f32bd65b700 1 -- 172.21.15.167:0/3986483093 <== osd.1 v2:172.21.15.167:6800/30035 13 ==== osd_op_reply(36 [pgnls start_epoch 30 out=115b] v23'2 uv2 ondisk = 1) v8 ==== 144+0+115 (crc 0 0 0) 0x7f32b800a5a0 con 0x561e229544b0 2023-10-05T19:28:38.862 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.858+0000 7f32bd65b700 1 -- 172.21.15.167:0/3986483093 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:37 5.1f 5:f8000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x561e22a17f10 con 0x7f32b0017540 2023-10-05T19:28:38.863 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.858+0000 7f32b7fff700 1 -- 172.21.15.167:0/3986483093 <== osd.2 v2:172.21.15.167:6816/30037 6 ==== osd_op_reply(37 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f32a8012820 con 0x7f32b0017540 2023-10-05T19:28:38.863 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.858+0000 7f32bf325e40 20 RGWRados::pool_iterate: got zonegroup_info.4f1f719d-9648-417b-b10c-e5d939825419 2023-10-05T19:28:38.863 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.858+0000 7f32bf325e40 20 RGWRados::pool_iterate: got zone_info.9a9df965-6612-45d4-a45f-6e3c889bacc2 2023-10-05T19:28:38.864 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.858+0000 7f32bf325e40 20 RGWRados::pool_iterate: got zone_names.default 2023-10-05T19:28:38.864 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.858+0000 7f32bf325e40 20 RGWRados::pool_iterate: got zonegroups_names.default 2023-10-05T19:28:38.864 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.858+0000 7f32bf325e40 20 rados->read ofs=0 len=0 2023-10-05T19:28:38.864 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.858+0000 7f32bf325e40 1 -- 172.21.15.167:0/3986483093 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:38 5.12 5:49953fa1:::default.realm:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x561e22a13fa0 con 0x561e229544b0 2023-10-05T19:28:38.865 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.858+0000 7f32bd65b700 1 -- 172.21.15.167:0/3986483093 <== osd.1 v2:172.21.15.167:6800/30035 14 ==== osd_op_reply(38 default.realm [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 ==== 157+0+0 (crc 0 0 0) 0x7f32b800a5a0 con 0x561e229544b0 2023-10-05T19:28:38.865 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.858+0000 7f32bf325e40 20 rados_obj.operate() r=-2 bl.length=0 2023-10-05T19:28:38.865 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.858+0000 7f32bf325e40 20 rados->read ofs=0 len=0 2023-10-05T19:28:38.865 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.858+0000 7f32bf325e40 1 -- 172.21.15.167:0/3986483093 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:39 5.f 5:f4c53578:::zonegroups_names.default:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x561e22a143a0 con 0x561e229544b0 2023-10-05T19:28:38.865 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.858+0000 7f32bd65b700 1 -- 172.21.15.167:0/3986483093 <== osd.1 v2:172.21.15.167:6800/30035 15 ==== osd_op_reply(39 zonegroups_names.default [read 0~46 out=46b] v0'0 uv2 ondisk = 0) v8 ==== 168+0+46 (crc 0 0 0) 0x7f32b800a5a0 con 0x561e229544b0 2023-10-05T19:28:38.865 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.858+0000 7f32bf325e40 20 rados_obj.operate() r=0 bl.length=46 2023-10-05T19:28:38.865 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.858+0000 7f32bf325e40 20 rados->read ofs=0 len=0 2023-10-05T19:28:38.865 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.858+0000 7f32bf325e40 1 -- 172.21.15.167:0/3986483093 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:40 5.c 5:30d299a5:::zonegroup_info.4f1f719d-9648-417b-b10c-e5d939825419:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x561e22a143a0 con 0x7f32a8007030 2023-10-05T19:28:38.866 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.858+0000 7f32bd65b700 1 -- 172.21.15.167:0/3986483093 <== osd.0 v2:172.21.15.118:6800/27690 10 ==== osd_op_reply(40 zonegroup_info.4f1f719d-9648-417b-b10c-e5d939825419 [read 0~362 out=362b] v0'0 uv1 ondisk = 0) v8 ==== 195+0+362 (crc 0 0 0) 0x7f32b800a5a0 con 0x7f32a8007030 2023-10-05T19:28:38.866 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.858+0000 7f32bf325e40 20 rados_obj.operate() r=0 bl.length=362 2023-10-05T19:28:38.866 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.858+0000 7f32bf325e40 20 rados->read ofs=0 len=0 2023-10-05T19:28:38.866 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.858+0000 7f32bf325e40 1 -- 172.21.15.167:0/3986483093 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:41 5.13 5:cbbf6c60:::converted:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x561e22a143a0 con 0x561e22a154f0 2023-10-05T19:28:38.867 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.862+0000 7f32bce5a700 1 -- 172.21.15.167:0/3986483093 <== osd.3 v2:172.21.15.167:6808/30036 10 ==== osd_op_reply(41 converted [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 ==== 153+0+0 (crc 0 0 0) 0x7f32b0016070 con 0x561e22a154f0 2023-10-05T19:28:38.867 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.862+0000 7f32bf325e40 20 rados_obj.operate() r=-2 bl.length=0 2023-10-05T19:28:38.867 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.862+0000 7f32bf325e40 20 rados->read ofs=0 len=0 2023-10-05T19:28:38.867 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.862+0000 7f32bf325e40 1 -- 172.21.15.167:0/3986483093 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:42 5.12 5:49953fa1:::default.realm:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x561e22a18380 con 0x561e229544b0 2023-10-05T19:28:38.868 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.862+0000 7f32bd65b700 1 -- 172.21.15.167:0/3986483093 <== osd.1 v2:172.21.15.167:6800/30035 16 ==== osd_op_reply(42 default.realm [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 ==== 157+0+0 (crc 0 0 0) 0x7f32b800a5a0 con 0x561e229544b0 2023-10-05T19:28:38.868 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.862+0000 7f32bf325e40 20 rados_obj.operate() r=-2 bl.length=0 2023-10-05T19:28:38.868 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.862+0000 7f32bf325e40 20 rados->read ofs=0 len=0 2023-10-05T19:28:38.868 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.862+0000 7f32bf325e40 1 -- 172.21.15.167:0/3986483093 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:43 5.f 5:f4c53578:::zonegroups_names.default:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x561e22a13bd0 con 0x561e229544b0 2023-10-05T19:28:38.868 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.862+0000 7f32bd65b700 1 -- 172.21.15.167:0/3986483093 <== osd.1 v2:172.21.15.167:6800/30035 17 ==== osd_op_reply(43 zonegroups_names.default [read 0~46 out=46b] v0'0 uv2 ondisk = 0) v8 ==== 168+0+46 (crc 0 0 0) 0x7f32b800a5a0 con 0x561e229544b0 2023-10-05T19:28:38.868 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.862+0000 7f32bf325e40 20 rados_obj.operate() r=0 bl.length=46 2023-10-05T19:28:38.869 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.862+0000 7f32bf325e40 1 -- 172.21.15.167:0/3986483093 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:44 5.0 5:00000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x561e22a13ba0 con 0x561e22a154f0 2023-10-05T19:28:38.871 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.866+0000 7f32bce5a700 1 -- 172.21.15.167:0/3986483093 <== osd.3 v2:172.21.15.167:6808/30036 11 ==== osd_op_reply(44 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f32b0016070 con 0x561e22a154f0 2023-10-05T19:28:38.871 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.866+0000 7f32bce5a700 1 -- 172.21.15.167:0/3986483093 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:45 5.10 5:08000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x561e22a13ba0 con 0x7f32b0017540 2023-10-05T19:28:38.871 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.866+0000 7f32b7fff700 1 -- 172.21.15.167:0/3986483093 <== osd.2 v2:172.21.15.167:6816/30037 7 ==== osd_op_reply(45 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f32a8012820 con 0x7f32b0017540 2023-10-05T19:28:38.872 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.866+0000 7f32b7fff700 1 -- 172.21.15.167:0/3986483093 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:46 5.8 5:10000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x561e22a13ba0 con 0x7f32a8007030 2023-10-05T19:28:38.872 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.866+0000 7f32bd65b700 1 -- 172.21.15.167:0/3986483093 <== osd.0 v2:172.21.15.118:6800/27690 11 ==== osd_op_reply(46 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f32b800a5a0 con 0x7f32a8007030 2023-10-05T19:28:38.872 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.866+0000 7f32bd65b700 1 -- 172.21.15.167:0/3986483093 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:47 5.18 5:18000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x561e22a13fa0 con 0x561e229544b0 2023-10-05T19:28:38.873 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.866+0000 7f32bd65b700 1 -- 172.21.15.167:0/3986483093 <== osd.1 v2:172.21.15.167:6800/30035 18 ==== osd_op_reply(47 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f32b800a5a0 con 0x561e229544b0 2023-10-05T19:28:38.873 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.866+0000 7f32bd65b700 1 -- 172.21.15.167:0/3986483093 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:48 5.4 5:20000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x561e22a13fa0 con 0x561e229544b0 2023-10-05T19:28:38.873 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.866+0000 7f32bd65b700 1 -- 172.21.15.167:0/3986483093 <== osd.1 v2:172.21.15.167:6800/30035 19 ==== osd_op_reply(48 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f32b800a5a0 con 0x561e229544b0 2023-10-05T19:28:38.874 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.866+0000 7f32bd65b700 1 -- 172.21.15.167:0/3986483093 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:49 5.14 5:28000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x561e22a13fa0 con 0x7f32a8007030 2023-10-05T19:28:38.875 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.870+0000 7f32bd65b700 1 -- 172.21.15.167:0/3986483093 <== osd.0 v2:172.21.15.118:6800/27690 12 ==== osd_op_reply(49 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f32b800a5a0 con 0x7f32a8007030 2023-10-05T19:28:38.875 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.870+0000 7f32bd65b700 1 -- 172.21.15.167:0/3986483093 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:50 5.c 5:30000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x561e22a13fa0 con 0x7f32a8007030 2023-10-05T19:28:38.875 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.870+0000 7f32bd65b700 1 -- 172.21.15.167:0/3986483093 <== osd.0 v2:172.21.15.118:6800/27690 13 ==== osd_op_reply(50 [pgnls start_epoch 30 out=112b] v23'1 uv1 ondisk = 1) v8 ==== 144+0+112 (crc 0 0 0) 0x7f32b800a5a0 con 0x7f32a8007030 2023-10-05T19:28:38.876 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.870+0000 7f32bd65b700 1 -- 172.21.15.167:0/3986483093 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:51 5.1c 5:38000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x561e22a13fa0 con 0x7f32b0017540 2023-10-05T19:28:38.876 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.870+0000 7f32b7fff700 1 -- 172.21.15.167:0/3986483093 <== osd.2 v2:172.21.15.167:6816/30037 8 ==== osd_op_reply(51 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f32a8012820 con 0x7f32b0017540 2023-10-05T19:28:38.876 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.870+0000 7f32b7fff700 1 -- 172.21.15.167:0/3986483093 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:52 5.2 5:40000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x561e22a13fa0 con 0x561e22a154f0 2023-10-05T19:28:38.876 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.870+0000 7f32bce5a700 1 -- 172.21.15.167:0/3986483093 <== osd.3 v2:172.21.15.167:6808/30036 12 ==== osd_op_reply(52 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f32b0016070 con 0x561e22a154f0 2023-10-05T19:28:38.877 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.870+0000 7f32bce5a700 1 -- 172.21.15.167:0/3986483093 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:53 5.12 5:48000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x561e22a13fa0 con 0x561e229544b0 2023-10-05T19:28:38.877 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.870+0000 7f32bd65b700 1 -- 172.21.15.167:0/3986483093 <== osd.1 v2:172.21.15.167:6800/30035 20 ==== osd_op_reply(53 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f32b800a5a0 con 0x561e229544b0 2023-10-05T19:28:38.877 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.870+0000 7f32bd65b700 1 -- 172.21.15.167:0/3986483093 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:54 5.a 5:50000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x561e22a13fa0 con 0x7f32a8007030 2023-10-05T19:28:38.878 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.874+0000 7f32bd65b700 1 -- 172.21.15.167:0/3986483093 <== osd.0 v2:172.21.15.118:6800/27690 14 ==== osd_op_reply(54 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f32b800a5a0 con 0x7f32a8007030 2023-10-05T19:28:38.878 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.874+0000 7f32bd65b700 1 -- 172.21.15.167:0/3986483093 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:55 5.1a 5:58000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x561e22a13fa0 con 0x561e229544b0 2023-10-05T19:28:38.879 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.874+0000 7f32bd65b700 1 -- 172.21.15.167:0/3986483093 <== osd.1 v2:172.21.15.167:6800/30035 21 ==== osd_op_reply(55 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f32b800a5a0 con 0x561e229544b0 2023-10-05T19:28:38.879 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.874+0000 7f32bd65b700 1 -- 172.21.15.167:0/3986483093 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:56 5.6 5:60000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x561e22a13fa0 con 0x7f32b0017540 2023-10-05T19:28:38.879 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.874+0000 7f32b7fff700 1 -- 172.21.15.167:0/3986483093 <== osd.2 v2:172.21.15.167:6816/30037 9 ==== osd_op_reply(56 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f32a8012820 con 0x7f32b0017540 2023-10-05T19:28:38.879 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.874+0000 7f32b7fff700 1 -- 172.21.15.167:0/3986483093 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:57 5.16 5:68000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x561e22a13fa0 con 0x561e22a154f0 2023-10-05T19:28:38.880 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.874+0000 7f32bce5a700 1 -- 172.21.15.167:0/3986483093 <== osd.3 v2:172.21.15.167:6808/30036 13 ==== osd_op_reply(57 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f32b0016070 con 0x561e22a154f0 2023-10-05T19:28:38.880 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.874+0000 7f32bce5a700 1 -- 172.21.15.167:0/3986483093 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:58 5.e 5:70000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x561e22a13fa0 con 0x561e22a154f0 2023-10-05T19:28:38.880 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.874+0000 7f32bce5a700 1 -- 172.21.15.167:0/3986483093 <== osd.3 v2:172.21.15.167:6808/30036 14 ==== osd_op_reply(58 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f32b0016070 con 0x561e22a154f0 2023-10-05T19:28:38.881 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.874+0000 7f32bce5a700 1 -- 172.21.15.167:0/3986483093 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:59 5.1e 5:78000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x561e22a13fa0 con 0x7f32a8007030 2023-10-05T19:28:38.882 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.874+0000 7f32bd65b700 1 -- 172.21.15.167:0/3986483093 <== osd.0 v2:172.21.15.118:6800/27690 15 ==== osd_op_reply(59 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f32b803f040 con 0x7f32a8007030 2023-10-05T19:28:38.882 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.874+0000 7f32bd65b700 1 -- 172.21.15.167:0/3986483093 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:60 5.1 5:80000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x561e22a13fa0 con 0x561e22a154f0 2023-10-05T19:28:38.882 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.878+0000 7f32bce5a700 1 -- 172.21.15.167:0/3986483093 <== osd.3 v2:172.21.15.167:6808/30036 15 ==== osd_op_reply(60 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f32b0016070 con 0x561e22a154f0 2023-10-05T19:28:38.882 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.878+0000 7f32bce5a700 1 -- 172.21.15.167:0/3986483093 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:61 5.11 5:88000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x561e22a13fa0 con 0x561e22a154f0 2023-10-05T19:28:38.882 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.878+0000 7f32bce5a700 1 -- 172.21.15.167:0/3986483093 <== osd.3 v2:172.21.15.167:6808/30036 16 ==== osd_op_reply(61 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f32b0016070 con 0x561e22a154f0 2023-10-05T19:28:38.883 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.878+0000 7f32bce5a700 1 -- 172.21.15.167:0/3986483093 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:62 5.9 5:90000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x561e22a13fa0 con 0x561e229544b0 2023-10-05T19:28:38.883 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.878+0000 7f32bd65b700 1 -- 172.21.15.167:0/3986483093 <== osd.1 v2:172.21.15.167:6800/30035 22 ==== osd_op_reply(62 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f32b803f040 con 0x561e229544b0 2023-10-05T19:28:38.883 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.878+0000 7f32bd65b700 1 -- 172.21.15.167:0/3986483093 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:63 5.19 5:98000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x561e22a13fa0 con 0x561e229544b0 2023-10-05T19:28:38.884 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.878+0000 7f32bd65b700 1 -- 172.21.15.167:0/3986483093 <== osd.1 v2:172.21.15.167:6800/30035 23 ==== osd_op_reply(63 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f32b803f040 con 0x561e229544b0 2023-10-05T19:28:38.884 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.878+0000 7f32bd65b700 1 -- 172.21.15.167:0/3986483093 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:64 5.5 5:a0000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x561e22a13fa0 con 0x7f32a8007030 2023-10-05T19:28:38.885 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.878+0000 7f32bd65b700 1 -- 172.21.15.167:0/3986483093 <== osd.0 v2:172.21.15.118:6800/27690 16 ==== osd_op_reply(64 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f32b803f040 con 0x7f32a8007030 2023-10-05T19:28:38.885 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.878+0000 7f32bd65b700 1 -- 172.21.15.167:0/3986483093 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:65 5.15 5:a8000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x561e22a13fa0 con 0x561e22a154f0 2023-10-05T19:28:38.885 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.878+0000 7f32bce5a700 1 -- 172.21.15.167:0/3986483093 <== osd.3 v2:172.21.15.167:6808/30036 17 ==== osd_op_reply(65 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f32b0016070 con 0x561e22a154f0 2023-10-05T19:28:38.885 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.878+0000 7f32bce5a700 1 -- 172.21.15.167:0/3986483093 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:66 5.d 5:b0000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x561e22a13fa0 con 0x7f32b0017540 2023-10-05T19:28:38.886 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.882+0000 7f32b7fff700 1 -- 172.21.15.167:0/3986483093 <== osd.2 v2:172.21.15.167:6816/30037 10 ==== osd_op_reply(66 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f32a8012820 con 0x7f32b0017540 2023-10-05T19:28:38.886 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.882+0000 7f32b7fff700 1 -- 172.21.15.167:0/3986483093 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:67 5.1d 5:b8000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x561e22a13fa0 con 0x561e229544b0 2023-10-05T19:28:38.886 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.882+0000 7f32bd65b700 1 -- 172.21.15.167:0/3986483093 <== osd.1 v2:172.21.15.167:6800/30035 24 ==== osd_op_reply(67 [pgnls start_epoch 30 out=107b] v30'2 uv2 ondisk = 1) v8 ==== 144+0+107 (crc 0 0 0) 0x7f32b803f040 con 0x561e229544b0 2023-10-05T19:28:38.887 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.882+0000 7f32bd65b700 1 -- 172.21.15.167:0/3986483093 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:68 5.3 5:c0000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x561e22a13fa0 con 0x7f32a8007030 2023-10-05T19:28:38.887 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.882+0000 7f32bd65b700 1 -- 172.21.15.167:0/3986483093 <== osd.0 v2:172.21.15.118:6800/27690 17 ==== osd_op_reply(68 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f32b803f040 con 0x7f32a8007030 2023-10-05T19:28:38.887 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.882+0000 7f32bd65b700 1 -- 172.21.15.167:0/3986483093 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:69 5.13 5:c8000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x561e22a13fa0 con 0x561e22a154f0 2023-10-05T19:28:38.888 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.882+0000 7f32bce5a700 1 -- 172.21.15.167:0/3986483093 <== osd.3 v2:172.21.15.167:6808/30036 18 ==== osd_op_reply(69 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f32b0016070 con 0x561e22a154f0 2023-10-05T19:28:38.888 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.882+0000 7f32bce5a700 1 -- 172.21.15.167:0/3986483093 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:70 5.b 5:d0000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x561e22a13fa0 con 0x7f32a8007030 2023-10-05T19:28:38.889 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.882+0000 7f32bd65b700 1 -- 172.21.15.167:0/3986483093 <== osd.0 v2:172.21.15.118:6800/27690 18 ==== osd_op_reply(70 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f32b803f040 con 0x7f32a8007030 2023-10-05T19:28:38.889 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.882+0000 7f32bd65b700 1 -- 172.21.15.167:0/3986483093 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:71 5.1b 5:d8000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x561e22a13fa0 con 0x7f32b0017540 2023-10-05T19:28:38.889 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.882+0000 7f32b7fff700 1 -- 172.21.15.167:0/3986483093 <== osd.2 v2:172.21.15.167:6816/30037 11 ==== osd_op_reply(71 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f32a8012820 con 0x7f32b0017540 2023-10-05T19:28:38.889 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.882+0000 7f32b7fff700 1 -- 172.21.15.167:0/3986483093 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:72 5.7 5:e0000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x561e22a13fa0 con 0x561e22a154f0 2023-10-05T19:28:38.890 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.886+0000 7f32bce5a700 1 -- 172.21.15.167:0/3986483093 <== osd.3 v2:172.21.15.167:6808/30036 19 ==== osd_op_reply(72 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f32b0016070 con 0x561e22a154f0 2023-10-05T19:28:38.890 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.886+0000 7f32bce5a700 1 -- 172.21.15.167:0/3986483093 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:73 5.17 5:e8000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x561e22a13fa0 con 0x7f32a8007030 2023-10-05T19:28:38.891 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.886+0000 7f32bd65b700 1 -- 172.21.15.167:0/3986483093 <== osd.0 v2:172.21.15.118:6800/27690 19 ==== osd_op_reply(73 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f32b803f040 con 0x7f32a8007030 2023-10-05T19:28:38.891 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.886+0000 7f32bd65b700 1 -- 172.21.15.167:0/3986483093 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:74 5.f 5:f0000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x561e22a13fa0 con 0x561e229544b0 2023-10-05T19:28:38.892 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.886+0000 7f32bd65b700 1 -- 172.21.15.167:0/3986483093 <== osd.1 v2:172.21.15.167:6800/30035 25 ==== osd_op_reply(74 [pgnls start_epoch 30 out=115b] v23'2 uv2 ondisk = 1) v8 ==== 144+0+115 (crc 0 0 0) 0x7f32b803f040 con 0x561e229544b0 2023-10-05T19:28:38.892 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.886+0000 7f32bd65b700 1 -- 172.21.15.167:0/3986483093 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:75 5.1f 5:f8000000::::head [pgnls start_epoch 30 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e30) v8 -- 0x561e22a13fa0 con 0x7f32b0017540 2023-10-05T19:28:38.893 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.886+0000 7f32b7fff700 1 -- 172.21.15.167:0/3986483093 <== osd.2 v2:172.21.15.167:6816/30037 12 ==== osd_op_reply(75 [pgnls start_epoch 30 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f32a8012820 con 0x7f32b0017540 2023-10-05T19:28:38.893 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.886+0000 7f32bf325e40 20 RGWRados::pool_iterate: got zonegroup_info.4f1f719d-9648-417b-b10c-e5d939825419 2023-10-05T19:28:38.893 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.886+0000 7f32bf325e40 20 RGWRados::pool_iterate: got zone_info.9a9df965-6612-45d4-a45f-6e3c889bacc2 2023-10-05T19:28:38.893 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.886+0000 7f32bf325e40 20 RGWRados::pool_iterate: got zone_names.default 2023-10-05T19:28:38.893 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.886+0000 7f32bf325e40 20 RGWRados::pool_iterate: got zonegroups_names.default 2023-10-05T19:28:38.893 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.886+0000 7f32bf325e40 20 rados->read ofs=0 len=0 2023-10-05T19:28:38.893 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.886+0000 7f32bf325e40 1 -- 172.21.15.167:0/3986483093 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:76 5.f 5:f43b6ece:::zone_names.default:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x561e22a13ba0 con 0x561e229544b0 2023-10-05T19:28:38.895 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.886+0000 7f32bd65b700 1 -- 172.21.15.167:0/3986483093 <== osd.1 v2:172.21.15.167:6800/30035 26 ==== osd_op_reply(76 zone_names.default [read 0~46 out=46b] v0'0 uv1 ondisk = 0) v8 ==== 162+0+46 (crc 0 0 0) 0x7f32b803f040 con 0x561e229544b0 2023-10-05T19:28:38.895 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.886+0000 7f32bf325e40 20 rados_obj.operate() r=0 bl.length=46 2023-10-05T19:28:38.895 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.886+0000 7f32bf325e40 20 rados->read ofs=0 len=0 2023-10-05T19:28:38.895 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.886+0000 7f32bf325e40 1 -- 172.21.15.167:0/3986483093 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:77 5.1d 5:b8374eb4:::zone_info.9a9df965-6612-45d4-a45f-6e3c889bacc2:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x561e22a188f0 con 0x561e229544b0 2023-10-05T19:28:38.895 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.890+0000 7f32bd65b700 1 -- 172.21.15.167:0/3986483093 <== osd.1 v2:172.21.15.167:6800/30035 27 ==== osd_op_reply(77 zone_info.9a9df965-6612-45d4-a45f-6e3c889bacc2 [read 0~874 out=874b] v0'0 uv2 ondisk = 0) v8 ==== 190+0+874 (crc 0 0 0) 0x7f32b803f040 con 0x561e229544b0 2023-10-05T19:28:38.895 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.890+0000 7f32bf325e40 20 rados_obj.operate() r=0 bl.length=874 2023-10-05T19:28:38.895 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.890+0000 7f32bf325e40 20 rados->read ofs=0 len=0 2023-10-05T19:28:38.895 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.890+0000 7f32bf325e40 1 -- 172.21.15.167:0/3986483093 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:78 5.f 5:f4c53578:::zonegroups_names.default:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x561e22a18de0 con 0x561e229544b0 2023-10-05T19:28:38.896 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.890+0000 7f32bd65b700 1 -- 172.21.15.167:0/3986483093 <== osd.1 v2:172.21.15.167:6800/30035 28 ==== osd_op_reply(78 zonegroups_names.default [read 0~46 out=46b] v0'0 uv2 ondisk = 0) v8 ==== 168+0+46 (crc 0 0 0) 0x7f32b803f040 con 0x561e229544b0 2023-10-05T19:28:38.896 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.890+0000 7f32bf325e40 20 rados_obj.operate() r=0 bl.length=46 2023-10-05T19:28:38.896 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.890+0000 7f32bf325e40 20 rados->read ofs=0 len=0 2023-10-05T19:28:38.896 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.890+0000 7f32bf325e40 1 -- 172.21.15.167:0/3986483093 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:79 5.c 5:30d299a5:::zonegroup_info.4f1f719d-9648-417b-b10c-e5d939825419:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x561e22a18de0 con 0x7f32a8007030 2023-10-05T19:28:38.897 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.890+0000 7f32bd65b700 1 -- 172.21.15.167:0/3986483093 <== osd.0 v2:172.21.15.118:6800/27690 20 ==== osd_op_reply(79 zonegroup_info.4f1f719d-9648-417b-b10c-e5d939825419 [read 0~362 out=362b] v0'0 uv1 ondisk = 0) v8 ==== 195+0+362 (crc 0 0 0) 0x7f32b8052040 con 0x7f32a8007030 2023-10-05T19:28:38.897 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.890+0000 7f32bf325e40 20 rados_obj.operate() r=0 bl.length=362 2023-10-05T19:28:38.897 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.890+0000 7f32bf325e40 20 rados->read ofs=0 len=0 2023-10-05T19:28:38.897 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.890+0000 7f32bf325e40 1 -- 172.21.15.167:0/3986483093 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:80 5.3 5:c0ad8cbd:::region_map:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x561e22a18ee0 con 0x7f32a8007030 2023-10-05T19:28:38.897 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.890+0000 7f32bd65b700 1 -- 172.21.15.167:0/3986483093 <== osd.0 v2:172.21.15.118:6800/27690 21 ==== osd_op_reply(80 region_map [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 ==== 154+0+0 (crc 0 0 0) 0x7f32b8052040 con 0x7f32a8007030 2023-10-05T19:28:38.898 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.890+0000 7f32bf325e40 20 rados_obj.operate() r=-2 bl.length=0 2023-10-05T19:28:38.898 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.890+0000 7f32bf325e40 20 zone default found 2023-10-05T19:28:38.898 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.890+0000 7f32bf325e40 4 Realm: () 2023-10-05T19:28:38.898 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.890+0000 7f32bf325e40 4 ZoneGroup: default (4f1f719d-9648-417b-b10c-e5d939825419) 2023-10-05T19:28:38.898 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.890+0000 7f32bf325e40 4 Zone: default (9a9df965-6612-45d4-a45f-6e3c889bacc2) 2023-10-05T19:28:38.898 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.890+0000 7f32bf325e40 10 cannot find current period zonegroup using local zonegroup configuration 2023-10-05T19:28:38.898 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.890+0000 7f32bf325e40 20 zonegroup default 2023-10-05T19:28:38.898 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.890+0000 7f32bf325e40 20 rados->read ofs=0 len=0 2023-10-05T19:28:38.898 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.890+0000 7f32bf325e40 1 -- 172.21.15.167:0/3986483093 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:81 5.3 5:c52100b6:::period_config.default:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x561e22a18ee0 con 0x7f32a8007030 2023-10-05T19:28:38.900 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.890+0000 7f32bd65b700 1 -- 172.21.15.167:0/3986483093 <== osd.0 v2:172.21.15.118:6800/27690 22 ==== osd_op_reply(81 period_config.default [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 ==== 165+0+0 (crc 0 0 0) 0x7f32b8052040 con 0x7f32a8007030 2023-10-05T19:28:38.900 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.890+0000 7f32bf325e40 20 rados_obj.operate() r=-2 bl.length=0 2023-10-05T19:28:38.900 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.890+0000 7f32bf325e40 20 rados->read ofs=0 len=0 2023-10-05T19:28:38.900 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.890+0000 7f32bf325e40 1 -- 172.21.15.167:0/3986483093 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:82 6.2 6:4347d321:::bucket.sync-source-hints.:head [call version.read in=11b,read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x561e22a1a860 con 0x561e22a154f0 2023-10-05T19:28:38.900 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.894+0000 7f32bce5a700 1 -- 172.21.15.167:0/3986483093 <== osd.3 v2:172.21.15.167:6808/30036 20 ==== osd_op_reply(82 bucket.sync-source-hints. [call,read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 ==== 211+0+0 (crc 0 0 0) 0x7f32b0016070 con 0x561e22a154f0 2023-10-05T19:28:38.900 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.894+0000 7f32bf325e40 20 rados_obj.operate() r=-2 bl.length=0 2023-10-05T19:28:38.900 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.894+0000 7f32bf325e40 20 rados->read ofs=0 len=0 2023-10-05T19:28:38.900 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.894+0000 7f32bf325e40 1 -- 172.21.15.167:0/3986483093 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:83 6.b 6:d467b91b:::bucket.sync-target-hints.:head [call version.read in=11b,read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x561e22a1ac10 con 0x561e22a154f0 2023-10-05T19:28:38.900 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.894+0000 7f32bce5a700 1 -- 172.21.15.167:0/3986483093 <== osd.3 v2:172.21.15.167:6808/30036 21 ==== osd_op_reply(83 bucket.sync-target-hints. [call,read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 ==== 211+0+0 (crc 0 0 0) 0x7f32b0016070 con 0x561e22a154f0 2023-10-05T19:28:38.901 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.894+0000 7f32bf325e40 20 rados_obj.operate() r=-2 bl.length=0 2023-10-05T19:28:38.901 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.894+0000 7f32bf325e40 20 started sync module instance, tier type = 2023-10-05T19:28:38.901 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.894+0000 7f32bf325e40 20 started zone id=9a9df965-6612-45d4-a45f-6e3c889bacc2 (name=default) with tier type = 2023-10-05T19:28:38.901 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.894+0000 7f32bf325e40 1 -- 172.21.15.167:0/3986483093 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:84 7.1f 7:f95f44c2:::notify.0:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x561e22a19f20 con 0x7f32b0017540 2023-10-05T19:28:38.902 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.898+0000 7f32b7fff700 1 -- 172.21.15.167:0/3986483093 <== osd.2 v2:172.21.15.167:6816/30037 13 ==== osd_op_reply(84 notify.0 [create] v30'186 uv186 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f32a8012820 con 0x7f32b0017540 2023-10-05T19:28:38.902 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.898+0000 7f32bf325e40 1 -- 172.21.15.167:0/3986483093 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:85 7.1f 7:f95f44c2:::notify.0:head [watch watch cookie 94687430023728] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x561e22a1afc0 con 0x7f32b0017540 2023-10-05T19:28:38.902 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.898+0000 7f32bf325e40 1 -- 172.21.15.167:0/3986483093 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:86 7.0 7:05bf5b68:::notify.1:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x561e22a1b890 con 0x561e229544b0 2023-10-05T19:28:38.905 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.898+0000 7f32b7fff700 1 -- 172.21.15.167:0/3986483093 <== osd.2 v2:172.21.15.167:6816/30037 14 ==== osd_op_reply(85 notify.0 [watch watch cookie 94687430023728] v30'187 uv186 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f32a8012820 con 0x7f32b0017540 2023-10-05T19:28:38.905 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.898+0000 7f32bd65b700 1 -- 172.21.15.167:0/3986483093 <== osd.1 v2:172.21.15.167:6800/30035 29 ==== osd_op_reply(86 notify.1 [create] v30'186 uv186 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f32b8052040 con 0x561e229544b0 2023-10-05T19:28:38.905 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.898+0000 7f32bf325e40 1 -- 172.21.15.167:0/3986483093 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:87 7.0 7:05bf5b68:::notify.1:head [watch watch cookie 94687430030240] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x561e22a1b7f0 con 0x561e229544b0 2023-10-05T19:28:38.905 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.898+0000 7f32bf325e40 1 -- 172.21.15.167:0/3986483093 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:88 7.15 7:a93a5511:::notify.2:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x561e22a1c670 con 0x561e22a154f0 2023-10-05T19:28:38.908 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.902+0000 7f32bd65b700 1 -- 172.21.15.167:0/3986483093 <== osd.1 v2:172.21.15.167:6800/30035 30 ==== osd_op_reply(87 notify.1 [watch watch cookie 94687430030240] v30'187 uv186 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f32b8052040 con 0x561e229544b0 2023-10-05T19:28:38.909 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.902+0000 7f32bce5a700 1 -- 172.21.15.167:0/3986483093 <== osd.3 v2:172.21.15.167:6808/30036 22 ==== osd_op_reply(88 notify.2 [create] v30'186 uv186 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f32b0016070 con 0x561e22a154f0 2023-10-05T19:28:38.909 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.902+0000 7f32bf325e40 1 -- 172.21.15.167:0/3986483093 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:89 7.15 7:a93a5511:::notify.2:head [watch watch cookie 94687430032528] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x561e22a1b7f0 con 0x561e22a154f0 2023-10-05T19:28:38.909 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.902+0000 7f32bf325e40 1 -- 172.21.15.167:0/3986483093 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:90 7.e 7:7759931f:::notify.3:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x561e22a1d120 con 0x7f32b0017540 2023-10-05T19:28:38.912 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.906+0000 7f32b7fff700 1 -- 172.21.15.167:0/3986483093 <== osd.2 v2:172.21.15.167:6816/30037 15 ==== osd_op_reply(90 notify.3 [create] v30'186 uv186 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f32a8012820 con 0x7f32b0017540 2023-10-05T19:28:38.912 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.906+0000 7f32bce5a700 1 -- 172.21.15.167:0/3986483093 <== osd.3 v2:172.21.15.167:6808/30036 23 ==== osd_op_reply(89 notify.2 [watch watch cookie 94687430032528] v30'187 uv186 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f32b0016070 con 0x561e22a154f0 2023-10-05T19:28:38.912 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.906+0000 7f32bf325e40 1 -- 172.21.15.167:0/3986483093 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:91 7.e 7:7759931f:::notify.3:head [watch watch cookie 94687430038816] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x561e22a1de60 con 0x7f32b0017540 2023-10-05T19:28:38.912 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.906+0000 7f32bf325e40 1 -- 172.21.15.167:0/3986483093 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:92 7.d 7:b4812045:::notify.4:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x561e22a1e380 con 0x561e229544b0 2023-10-05T19:28:38.914 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.910+0000 7f32b7fff700 1 -- 172.21.15.167:0/3986483093 <== osd.2 v2:172.21.15.167:6816/30037 16 ==== osd_op_reply(91 notify.3 [watch watch cookie 94687430038816] v30'187 uv186 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f32a8012820 con 0x7f32b0017540 2023-10-05T19:28:38.915 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.910+0000 7f32bd65b700 1 -- 172.21.15.167:0/3986483093 <== osd.1 v2:172.21.15.167:6800/30035 31 ==== osd_op_reply(92 notify.4 [create] v30'186 uv186 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f32b8052040 con 0x561e229544b0 2023-10-05T19:28:38.915 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.910+0000 7f32bf325e40 1 -- 172.21.15.167:0/3986483093 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:93 7.d 7:b4812045:::notify.4:head [watch watch cookie 94687430039920] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x561e22a1c1f0 con 0x561e229544b0 2023-10-05T19:28:38.915 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.910+0000 7f32bf325e40 1 -- 172.21.15.167:0/3986483093 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:94 7.3 7:c609908c:::notify.5:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x561e22a1ef70 con 0x7f32a8007030 2023-10-05T19:28:38.917 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.910+0000 7f32bd65b700 1 -- 172.21.15.167:0/3986483093 <== osd.1 v2:172.21.15.167:6800/30035 32 ==== osd_op_reply(93 notify.4 [watch watch cookie 94687430039920] v30'187 uv186 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f32b8052040 con 0x561e229544b0 2023-10-05T19:28:38.918 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.910+0000 7f32bd65b700 1 -- 172.21.15.167:0/3986483093 <== osd.0 v2:172.21.15.118:6800/27690 23 ==== osd_op_reply(94 notify.5 [create] v30'186 uv186 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f32b8052040 con 0x7f32a8007030 2023-10-05T19:28:38.918 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.914+0000 7f32bf325e40 1 -- 172.21.15.167:0/3986483093 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:95 7.3 7:c609908c:::notify.5:head [watch watch cookie 94687430042496] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x561e22a1ef30 con 0x7f32a8007030 2023-10-05T19:28:38.918 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.914+0000 7f32bf325e40 1 -- 172.21.15.167:0/3986483093 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:96 7.14 7:2b04a3e9:::notify.6:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x561e22a1f9d0 con 0x7f32a8007030 2023-10-05T19:28:38.920 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.914+0000 7f32bd65b700 1 -- 172.21.15.167:0/3986483093 <== osd.0 v2:172.21.15.118:6800/27690 24 ==== osd_op_reply(95 notify.5 [watch watch cookie 94687430042496] v30'187 uv186 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f32b8052040 con 0x7f32a8007030 2023-10-05T19:28:38.921 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.914+0000 7f32bd65b700 1 -- 172.21.15.167:0/3986483093 <== osd.0 v2:172.21.15.118:6800/27690 25 ==== osd_op_reply(96 notify.6 [create] v30'186 uv186 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f32b8052040 con 0x7f32a8007030 2023-10-05T19:28:38.921 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.914+0000 7f32bf325e40 1 -- 172.21.15.167:0/3986483093 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:97 7.14 7:2b04a3e9:::notify.6:head [watch watch cookie 94687430044784] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x561e22a1f9d0 con 0x7f32a8007030 2023-10-05T19:28:38.921 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.914+0000 7f32bf325e40 1 -- 172.21.15.167:0/3986483093 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:98 7.9 7:93e5b521:::notify.7:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x561e22a203e0 con 0x7f32a8007030 2023-10-05T19:28:38.924 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.918+0000 7f32bd65b700 1 -- 172.21.15.167:0/3986483093 <== osd.0 v2:172.21.15.118:6800/27690 26 ==== osd_op_reply(98 notify.7 [create] v30'186 uv186 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f32b8052040 con 0x7f32a8007030 2023-10-05T19:28:38.924 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.918+0000 7f32bd65b700 1 -- 172.21.15.167:0/3986483093 <== osd.0 v2:172.21.15.118:6800/27690 27 ==== osd_op_reply(97 notify.6 [watch watch cookie 94687430044784] v30'187 uv186 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f32b8052040 con 0x7f32a8007030 2023-10-05T19:28:38.924 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.918+0000 7f32bf325e40 1 -- 172.21.15.167:0/3986483093 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:99 7.9 7:93e5b521:::notify.7:head [watch watch cookie 94687430049520] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x561e22a20340 con 0x7f32a8007030 2023-10-05T19:28:38.924 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.918+0000 7f32bf325e40 20 add_watcher() i=0 2023-10-05T19:28:38.924 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.918+0000 7f32bf325e40 20 add_watcher() i=1 2023-10-05T19:28:38.924 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.918+0000 7f32bf325e40 20 add_watcher() i=2 2023-10-05T19:28:38.924 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.918+0000 7f32bf325e40 20 add_watcher() i=3 2023-10-05T19:28:38.924 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.918+0000 7f32bf325e40 20 add_watcher() i=4 2023-10-05T19:28:38.924 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.918+0000 7f32bf325e40 20 add_watcher() i=5 2023-10-05T19:28:38.925 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.918+0000 7f32bf325e40 20 add_watcher() i=6 2023-10-05T19:28:38.926 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.918+0000 7f32bd65b700 1 -- 172.21.15.167:0/3986483093 <== osd.0 v2:172.21.15.118:6800/27690 28 ==== osd_op_reply(99 notify.7 [watch watch cookie 94687430049520] v30'187 uv186 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f32b8052040 con 0x7f32a8007030 2023-10-05T19:28:38.926 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.918+0000 7f32bf325e40 20 add_watcher() i=7 2023-10-05T19:28:38.926 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.918+0000 7f32bf325e40 2 all 8 watchers are set, enabling cache 2023-10-05T19:28:38.926 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.918+0000 7f32bf325e40 1 -- 172.21.15.167:0/3986483093 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:100 6.4 6:22d26bf9:::data_loggenerations_metadata:head [call version.check_conds in=74b,call version.read in=11b,read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x561e22a23860 con 0x561e229544b0 2023-10-05T19:28:38.926 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.922+0000 7f32bd65b700 1 -- 172.21.15.167:0/3986483093 <== osd.1 v2:172.21.15.167:6800/30035 33 ==== osd_op_reply(100 data_loggenerations_metadata [call,call out=48b,read 0~28 out=28b] v0'0 uv1 ondisk = 0) v8 ==== 256+0+76 (crc 0 0 0) 0x7f32b8052040 con 0x561e229544b0 2023-10-05T19:28:38.927 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.922+0000 7f32bf325e40 1 -- 172.21.15.167:0/3986483093 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:101 6.4 6:22d26bf9:::data_loggenerations_metadata:head [watch watch cookie 94687430050624] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x561e22a20c80 con 0x561e229544b0 2023-10-05T19:28:38.929 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.922+0000 7f32bd65b700 1 -- 172.21.15.167:0/3986483093 <== osd.1 v2:172.21.15.167:6800/30035 34 ==== osd_op_reply(101 data_loggenerations_metadata [watch watch cookie 94687430050624] v30'171 uv1 ondisk = 0) v8 ==== 172+0+0 (crc 0 0 0) 0x7f32b8052040 con 0x561e229544b0 2023-10-05T19:28:38.929 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.922+0000 7f328dfe3700 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2023-10-05T19:28:38.929 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.922+0000 7f32bf325e40 20 check_secure_mon_conn(): auth registy supported: methods=[2] modes=[2,1] 2023-10-05T19:28:38.929 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.922+0000 7f32bf325e40 20 check_secure_mon_conn(): mode 1 is insecure 2023-10-05T19:28:38.930 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.922+0000 7f32bf325e40 1 Processor -- start 2023-10-05T19:28:38.930 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.922+0000 7f32bf325e40 1 -- start start 2023-10-05T19:28:38.930 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.926+0000 7f32bf325e40 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x561e22a2dd70 0x561e22a2e140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-05T19:28:38.930 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.926+0000 7f32bf325e40 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x561e2287eed0 con 0x561e22a2dd70 2023-10-05T19:28:38.931 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.926+0000 7f32b7fff700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x561e22a2dd70 0x561e22a2e140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-05T19:28:38.931 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.926+0000 7f32b7fff700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x561e22a2dd70 0x561e22a2e140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.118:3300/0 says I am v2:172.21.15.167:42298/0 (socket says 172.21.15.167:42298) 2023-10-05T19:28:38.931 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.926+0000 7f32b7fff700 1 -- 172.21.15.167:0/3972045796 learned_addr learned my addr 172.21.15.167:0/3972045796 (peer_addr_for_me v2:172.21.15.167:0/0) 2023-10-05T19:28:38.932 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.926+0000 7f32b7fff700 1 -- 172.21.15.167:0/3972045796 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x561e22a2e680 con 0x561e22a2dd70 2023-10-05T19:28:38.932 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.926+0000 7f32b7fff700 1 --2- 172.21.15.167:0/3972045796 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x561e22a2dd70 0x561e22a2e140 secure :-1 s=READY pgs=510 cs=0 l=1 rev1=1 crypto rx=0x561e22954910 tx=0x7f32a8012000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=43aca507e85bd6ae server_cookie=0 in_seq=0 out_seq=0 2023-10-05T19:28:38.932 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.926+0000 7f328d7e2700 1 -- 172.21.15.167:0/3972045796 <== mon.0 v2:172.21.15.118:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f32a802aa20 con 0x561e22a2dd70 2023-10-05T19:28:38.932 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.926+0000 7f328d7e2700 1 -- 172.21.15.167:0/3972045796 <== mon.0 v2:172.21.15.118:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f32a802ab80 con 0x561e22a2dd70 2023-10-05T19:28:38.932 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.926+0000 7f328d7e2700 1 -- 172.21.15.167:0/3972045796 <== mon.0 v2:172.21.15.118:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f32a802ad30 con 0x561e22a2dd70 2023-10-05T19:28:38.933 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.926+0000 7f32bf325e40 1 -- 172.21.15.167:0/3972045796 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x561e22a2dd70 msgr2=0x561e22a2e140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-05T19:28:38.934 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.926+0000 7f32bf325e40 1 --2- 172.21.15.167:0/3972045796 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x561e22a2dd70 0x561e22a2e140 secure :-1 s=READY pgs=510 cs=0 l=1 rev1=1 crypto rx=0x561e22954910 tx=0x7f32a8012000 comp rx=0 tx=0).stop 2023-10-05T19:28:38.934 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.926+0000 7f32bf325e40 1 -- 172.21.15.167:0/3972045796 shutdown_connections 2023-10-05T19:28:38.934 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.926+0000 7f32bf325e40 1 --2- 172.21.15.167:0/3972045796 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x561e22a2dd70 0x561e22a2e140 unknown :-1 s=CLOSED pgs=510 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-05T19:28:38.934 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.926+0000 7f32bf325e40 1 -- 172.21.15.167:0/3972045796 >> 172.21.15.167:0/3972045796 conn(0x561e22a29130 msgr2=0x561e22a27280 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-05T19:28:38.934 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.926+0000 7f32bf325e40 1 -- 172.21.15.167:0/3972045796 shutdown_connections 2023-10-05T19:28:38.934 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.926+0000 7f32bf325e40 1 -- 172.21.15.167:0/3972045796 wait complete. 2023-10-05T19:28:38.935 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.930+0000 7f32bf325e40 1 Processor -- start 2023-10-05T19:28:38.935 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.930+0000 7f32bf325e40 1 -- start start 2023-10-05T19:28:38.935 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.930+0000 7f32bf325e40 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x561e22a2dd70 0x561e22a33760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-05T19:28:38.935 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.930+0000 7f32bf325e40 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x561e22a34760 con 0x561e22a2dd70 2023-10-05T19:28:38.935 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.930+0000 7f32b7fff700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x561e22a2dd70 0x561e22a33760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-05T19:28:38.935 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.930+0000 7f32b7fff700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x561e22a2dd70 0x561e22a33760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.118:3300/0 says I am v2:172.21.15.167:42308/0 (socket says 172.21.15.167:42308) 2023-10-05T19:28:38.936 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.930+0000 7f32b7fff700 1 -- 172.21.15.167:0/2562358716 learned_addr learned my addr 172.21.15.167:0/2562358716 (peer_addr_for_me v2:172.21.15.167:0/0) 2023-10-05T19:28:38.936 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.930+0000 7f32b7fff700 1 -- 172.21.15.167:0/2562358716 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x561e22954940 con 0x561e22a2dd70 2023-10-05T19:28:38.937 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.930+0000 7f32b7fff700 1 --2- 172.21.15.167:0/2562358716 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x561e22a2dd70 0x561e22a33760 secure :-1 s=READY pgs=511 cs=0 l=1 rev1=1 crypto rx=0x7f32a8012f30 tx=0x7f32a802f000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-05T19:28:38.937 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.930+0000 7f32077fe700 1 -- 172.21.15.167:0/2562358716 <== mon.0 v2:172.21.15.118:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f32a8009bd0 con 0x561e22a2dd70 2023-10-05T19:28:38.937 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.930+0000 7f32077fe700 1 -- 172.21.15.167:0/2562358716 <== mon.0 v2:172.21.15.118:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f32a8009d50 con 0x561e22a2dd70 2023-10-05T19:28:38.937 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.930+0000 7f32bf325e40 1 -- 172.21.15.167:0/2562358716 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x561e22a33d60 con 0x561e22a2dd70 2023-10-05T19:28:38.937 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.930+0000 7f32077fe700 1 -- 172.21.15.167:0/2562358716 <== mon.0 v2:172.21.15.118:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f32a8047480 con 0x561e22a2dd70 2023-10-05T19:28:38.937 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.930+0000 7f32bf325e40 1 -- 172.21.15.167:0/2562358716 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x561e22a34200 con 0x561e22a2dd70 2023-10-05T19:28:38.938 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.934+0000 7f32077fe700 1 -- 172.21.15.167:0/2562358716 <== mon.0 v2:172.21.15.118:3300/0 4 ==== mgrmap(e 5) v1 ==== 40329+0+0 (secure 0 0 0) 0x7f32a8035400 con 0x561e22a2dd70 2023-10-05T19:28:38.938 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.934+0000 7f32077fe700 1 --2- 172.21.15.167:0/2562358716 >> [v2:172.21.15.167:6824/29889,v1:172.21.15.167:6825/29889] conn(0x7f31fc030c50 0x7f31fc033100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-05T19:28:38.938 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.934+0000 7f32077fe700 1 -- 172.21.15.167:0/2562358716 <== mon.0 v2:172.21.15.118:3300/0 5 ==== osd_map(30..30 src has 1..30) v4 ==== 6057+0+0 (secure 0 0 0) 0x7f32a802f070 con 0x561e22a2dd70 2023-10-05T19:28:38.939 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.934+0000 7f32bd65b700 1 --2- 172.21.15.167:0/2562358716 >> [v2:172.21.15.167:6824/29889,v1:172.21.15.167:6825/29889] conn(0x7f31fc030c50 0x7f31fc033100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-05T19:28:38.939 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.934+0000 7f32bf325e40 5 note: GC not initialized 2023-10-05T19:28:38.939 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.934+0000 7f32bd65b700 1 --2- 172.21.15.167:0/2562358716 >> [v2:172.21.15.167:6824/29889,v1:172.21.15.167:6825/29889] conn(0x7f31fc030c50 0x7f31fc033100 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x561e22a1f770 tx=0x7f32b8012e70 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-05T19:28:38.941 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.934+0000 7f3204ff9700 20 reqs_thread_entry: start 2023-10-05T19:28:38.941 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.934+0000 7f31fa7fc700 20 reqs_thread_entry: start 2023-10-05T19:28:38.942 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.934+0000 7f32bf325e40 20 init_complete bucket index max shards: 11 2023-10-05T19:28:38.942 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.938+0000 7f32bf325e40 10 rgw notify: Started notification manager with: 1 workers 2023-10-05T19:28:38.942 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.938+0000 7f31f8ff9700 20 rgw notify: INFO: next queues processing will happen at: Thu Oct 5 19:29:09 2023 2023-10-05T19:28:38.942 INFO:teuthology.orchestra.run.smithi167.stderr: 2023-10-05T19:28:38.943 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.938+0000 7f31dffff700 20 reqs_thread_entry: start 2023-10-05T19:28:38.943 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.938+0000 7f32bf325e40 20 get_system_obj_state: rctx=0x7ffc0b416bf8 obj=default.rgw.meta:users.uid:foo state=0x561e228b70b0 s->prefetch_data=0 2023-10-05T19:28:38.943 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.938+0000 7f32bf325e40 10 cache get: name=default.rgw.meta+users.uid+foo : miss 2023-10-05T19:28:38.943 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.938+0000 7f32bf325e40 1 -- 172.21.15.167:0/3986483093 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:102 8.18 8:1e6ca5ed:users.uid::foo:head [call version.read in=11b,getxattrs,stat] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x561e22a9beb0 con 0x561e229544b0 2023-10-05T19:28:38.943 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.938+0000 7f32bd65b700 1 -- 172.21.15.167:0/3986483093 <== osd.1 v2:172.21.15.167:6800/30035 35 ==== osd_op_reply(102 foo [call out=48b,getxattrs out=75b,stat out=16b] v0'0 uv12 ondisk = 0) v8 ==== 231+0+139 (crc 0 0 0) 0x7f32b8052040 con 0x561e229544b0 2023-10-05T19:28:38.944 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.938+0000 7f32bf325e40 10 cache put: name=default.rgw.meta+users.uid+foo info.flags=0x16 2023-10-05T19:28:38.944 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.938+0000 7f32bf325e40 10 adding default.rgw.meta+users.uid+foo to cache LRU end 2023-10-05T19:28:38.944 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.938+0000 7f32bf325e40 10 updating xattr: name=ceph.objclass.version bl.length()=42 2023-10-05T19:28:38.944 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.938+0000 7f32bf325e40 20 get_system_obj_state: s->obj_tag was set empty 2023-10-05T19:28:38.944 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.938+0000 7f32bf325e40 20 Read xattr: user.rgw.idtag 2023-10-05T19:28:38.944 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.938+0000 7f32bf325e40 10 cache get: name=default.rgw.meta+users.uid+foo : type miss (requested=0x13, cached=0x16) 2023-10-05T19:28:38.944 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.938+0000 7f32bf325e40 20 rados->read ofs=0 len=0 2023-10-05T19:28:38.944 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.938+0000 7f32bf325e40 1 -- 172.21.15.167:0/3986483093 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:103 8.18 8:1e6ca5ed:users.uid::foo:head [call version.check_conds in=122b,call version.read in=11b,read 0~0,getxattrs] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x561e22a9e940 con 0x561e229544b0 2023-10-05T19:28:38.945 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.938+0000 7f32bd65b700 1 -- 172.21.15.167:0/3986483093 <== osd.1 v2:172.21.15.167:6800/30035 36 ==== osd_op_reply(103 foo [call,call out=48b,read 0~357 out=357b,getxattrs out=75b] v0'0 uv12 ondisk = 0) v8 ==== 273+0+480 (crc 0 0 0) 0x7f32b8052040 con 0x561e229544b0 2023-10-05T19:28:38.945 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.938+0000 7f32bf325e40 20 rados_obj.operate() r=0 bl.length=357 2023-10-05T19:28:38.945 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.938+0000 7f32bf325e40 10 cache put: name=default.rgw.meta+users.uid+foo info.flags=0x13 2023-10-05T19:28:38.945 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.938+0000 7f32bf325e40 10 moving default.rgw.meta+users.uid+foo to cache LRU end 2023-10-05T19:28:38.945 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.938+0000 7f32bf325e40 10 updating xattr: name=ceph.objclass.version bl.length()=42 2023-10-05T19:28:38.945 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.938+0000 7f32bf325e40 1 -- 172.21.15.167:0/3986483093 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:104 8.9 8:96d98480:users.uid::foo.buckets:head [call user.list_buckets in=34b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x561e22a9e940 con 0x7f32b0017540 2023-10-05T19:28:38.946 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.938+0000 7f32b7fff700 1 -- 172.21.15.167:0/3986483093 <== osd.2 v2:172.21.15.167:6816/30037 17 ==== osd_op_reply(104 foo.buckets [call out=183b] v0'0 uv26 ondisk = 0) v8 ==== 155+0+183 (crc 0 0 0) 0x7f32a8032040 con 0x7f32b0017540 2023-10-05T19:28:38.946 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.938+0000 7f32bf325e40 20 enabling bucket name=myfoo 2023-10-05T19:28:38.946 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.938+0000 7f32bf325e40 20 get_system_obj_state: rctx=0x7ffc0b4163d0 obj=default.rgw.meta:root:myfoo state=0x561e228b70b0 s->prefetch_data=0 2023-10-05T19:28:38.947 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.938+0000 7f32bf325e40 10 cache get: name=default.rgw.meta+root+myfoo : miss 2023-10-05T19:28:38.947 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.938+0000 7f32bf325e40 1 -- 172.21.15.167:0/3986483093 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:105 8.9 8:91b71841:root::myfoo:head [getxattrs,stat] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x561e22a9d140 con 0x7f32b0017540 2023-10-05T19:28:38.947 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.942+0000 7f32b7fff700 1 -- 172.21.15.167:0/3986483093 <== osd.2 v2:172.21.15.167:6816/30037 18 ==== osd_op_reply(105 myfoo [getxattrs out=75b,stat out=16b] v0'0 uv27 ondisk = 0) v8 ==== 191+0+91 (crc 0 0 0) 0x7f32a8032040 con 0x7f32b0017540 2023-10-05T19:28:38.947 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.942+0000 7f32bf325e40 10 cache put: name=default.rgw.meta+root+myfoo info.flags=0x6 2023-10-05T19:28:38.947 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.942+0000 7f32bf325e40 10 adding default.rgw.meta+root+myfoo to cache LRU end 2023-10-05T19:28:38.947 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.942+0000 7f32bf325e40 10 updating xattr: name=ceph.objclass.version bl.length()=42 2023-10-05T19:28:38.947 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.942+0000 7f32bf325e40 20 get_system_obj_state: s->obj_tag was set empty 2023-10-05T19:28:38.947 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.942+0000 7f32bf325e40 10 cache get: name=default.rgw.meta+root+myfoo : type miss (requested=0x1, cached=0x6) 2023-10-05T19:28:38.947 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.942+0000 7f32bf325e40 20 rados->read ofs=0 len=0 2023-10-05T19:28:38.947 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.942+0000 7f32bf325e40 1 -- 172.21.15.167:0/3986483093 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:106 8.9 8:91b71841:root::myfoo:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x561e22a9d140 con 0x7f32b0017540 2023-10-05T19:28:38.949 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.942+0000 7f32b7fff700 1 -- 172.21.15.167:0/3986483093 <== osd.2 v2:172.21.15.167:6816/30037 19 ==== osd_op_reply(106 myfoo [read 0~165 out=165b] v0'0 uv27 ondisk = 0) v8 ==== 149+0+165 (crc 0 0 0) 0x7f32a8032040 con 0x7f32b0017540 2023-10-05T19:28:38.949 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.942+0000 7f32bf325e40 20 rados_obj.operate() r=0 bl.length=165 2023-10-05T19:28:38.949 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.942+0000 7f32bf325e40 10 cache put: name=default.rgw.meta+root+myfoo info.flags=0x1 2023-10-05T19:28:38.949 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.942+0000 7f32bf325e40 10 moving default.rgw.meta+root+myfoo to cache LRU end 2023-10-05T19:28:38.949 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.942+0000 7f32bf325e40 20 get_system_obj_state: rctx=0x7ffc0b4163d0 obj=default.rgw.meta:root:.bucket.meta.myfoo:9a9df965-6612-45d4-a45f-6e3c889bacc2.4223.1 state=0x561e2288d2b0 s->prefetch_data=0 2023-10-05T19:28:38.949 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.942+0000 7f32bf325e40 10 cache get: name=default.rgw.meta+root+.bucket.meta.myfoo:9a9df965-6612-45d4-a45f-6e3c889bacc2.4223.1 : miss 2023-10-05T19:28:38.949 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.942+0000 7f32bf325e40 1 -- 172.21.15.167:0/3986483093 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:107 8.11 8:8bb4ed75:root::.bucket.meta.myfoo%3a9a9df965-6612-45d4-a45f-6e3c889bacc2.4223.1:head [call version.read in=11b,getxattrs,stat] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x561e22a9fa50 con 0x561e22a154f0 2023-10-05T19:28:38.949 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.942+0000 7f32bce5a700 1 -- 172.21.15.167:0/3986483093 <== osd.3 v2:172.21.15.167:6808/30036 24 ==== osd_op_reply(107 .bucket.meta.myfoo:9a9df965-6612-45d4-a45f-6e3c889bacc2.4223.1 [call out=48b,getxattrs out=218b,stat out=16b] v0'0 uv6 ondisk = 0) v8 ==== 290+0+282 (crc 0 0 0) 0x7f32b0016070 con 0x561e22a154f0 2023-10-05T19:28:38.949 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.942+0000 7f32bf325e40 10 cache put: name=default.rgw.meta+root+.bucket.meta.myfoo:9a9df965-6612-45d4-a45f-6e3c889bacc2.4223.1 info.flags=0x16 2023-10-05T19:28:38.949 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.942+0000 7f32bf325e40 10 adding default.rgw.meta+root+.bucket.meta.myfoo:9a9df965-6612-45d4-a45f-6e3c889bacc2.4223.1 to cache LRU end 2023-10-05T19:28:38.950 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.942+0000 7f32bf325e40 10 updating xattr: name=ceph.objclass.version bl.length()=42 2023-10-05T19:28:38.950 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.942+0000 7f32bf325e40 10 updating xattr: name=user.rgw.acl bl.length()=123 2023-10-05T19:28:38.950 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.942+0000 7f32bf325e40 20 get_system_obj_state: s->obj_tag was set empty 2023-10-05T19:28:38.950 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.942+0000 7f32bf325e40 20 Read xattr: user.rgw.acl 2023-10-05T19:28:38.950 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.942+0000 7f32bf325e40 20 Read xattr: user.rgw.idtag 2023-10-05T19:28:38.950 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.942+0000 7f32bf325e40 10 cache get: name=default.rgw.meta+root+.bucket.meta.myfoo:9a9df965-6612-45d4-a45f-6e3c889bacc2.4223.1 : type miss (requested=0x13, cached=0x16) 2023-10-05T19:28:38.950 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.942+0000 7f32bf325e40 20 rados->read ofs=0 len=0 2023-10-05T19:28:38.950 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.942+0000 7f32bf325e40 1 -- 172.21.15.167:0/3986483093 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:108 8.11 8:8bb4ed75:root::.bucket.meta.myfoo%3a9a9df965-6612-45d4-a45f-6e3c889bacc2.4223.1:head [call version.check_conds in=122b,call version.read in=11b,read 0~0,getxattrs] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x561e22aa2830 con 0x561e22a154f0 2023-10-05T19:28:38.951 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.942+0000 7f32bce5a700 1 -- 172.21.15.167:0/3986483093 <== osd.3 v2:172.21.15.167:6808/30036 25 ==== osd_op_reply(108 .bucket.meta.myfoo:9a9df965-6612-45d4-a45f-6e3c889bacc2.4223.1 [call,call out=48b,read 0~352 out=352b,getxattrs out=218b] v0'0 uv6 ondisk = 0) v8 ==== 332+0+618 (crc 0 0 0) 0x7f32b0016070 con 0x561e22a154f0 2023-10-05T19:28:38.951 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.942+0000 7f32bf325e40 20 rados_obj.operate() r=0 bl.length=352 2023-10-05T19:28:38.951 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.942+0000 7f32bf325e40 10 cache put: name=default.rgw.meta+root+.bucket.meta.myfoo:9a9df965-6612-45d4-a45f-6e3c889bacc2.4223.1 info.flags=0x13 2023-10-05T19:28:38.951 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.942+0000 7f32bf325e40 10 moving default.rgw.meta+root+.bucket.meta.myfoo:9a9df965-6612-45d4-a45f-6e3c889bacc2.4223.1 to cache LRU end 2023-10-05T19:28:38.951 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.942+0000 7f32bf325e40 10 updating xattr: name=ceph.objclass.version bl.length()=42 2023-10-05T19:28:38.951 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.942+0000 7f32bf325e40 10 updating xattr: name=user.rgw.acl bl.length()=123 2023-10-05T19:28:38.951 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.942+0000 7f32bf325e40 10 chain_cache_entry: cache_locator=default.rgw.meta+root+.bucket.meta.myfoo:9a9df965-6612-45d4-a45f-6e3c889bacc2.4223.1 2023-10-05T19:28:38.951 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.942+0000 7f32bf325e40 1 -- 172.21.15.167:0/3986483093 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:109 8.11 8:8bb4ed75:root::.bucket.meta.myfoo%3a9a9df965-6612-45d4-a45f-6e3c889bacc2.4223.1:head [delete,create,call version.check_conds in=122b,call version.set in=58b,writefull 0~352 in=352b,setxattr user.rgw.acl (123) in=135b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x561e22aa7ff0 con 0x561e22a154f0 2023-10-05T19:28:38.952 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.946+0000 7f32bce5a700 1 -- 172.21.15.167:0/3986483093 <== osd.3 v2:172.21.15.167:6808/30036 26 ==== osd_op_reply(109 .bucket.meta.myfoo:9a9df965-6612-45d4-a45f-6e3c889bacc2.4223.1 [delete,create,call,call,writefull 0~352,setxattr (123)] v30'7 uv7 ondisk = 0) v8 ==== 416+0+0 (crc 0 0 0) 0x7f32b0016070 con 0x561e22a154f0 2023-10-05T19:28:38.952 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.946+0000 7f32bf325e40 10 cache put: name=default.rgw.meta+root+.bucket.meta.myfoo:9a9df965-6612-45d4-a45f-6e3c889bacc2.4223.1 info.flags=0x17 2023-10-05T19:28:38.952 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.946+0000 7f32bf325e40 10 moving default.rgw.meta+root+.bucket.meta.myfoo:9a9df965-6612-45d4-a45f-6e3c889bacc2.4223.1 to cache LRU end 2023-10-05T19:28:38.952 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.946+0000 7f32bf325e40 10 updating xattr: name=user.rgw.acl bl.length()=123 2023-10-05T19:28:38.953 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.946+0000 7f32bf325e40 10 distributing notification oid=default.rgw.control:notify.7 cni=[op: 0, obj: default.rgw.meta:root:.bucket.meta.myfoo:9a9df965-6612-45d4-a45f-6e3c889bacc2.4223.1, ofs0, ns] 2023-10-05T19:28:38.953 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.946+0000 7f32bf325e40 1 -- 172.21.15.167:0/3986483093 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:110 7.9 7:93e5b521:::notify.7:head [notify cookie 94687430574032 in=737b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x561e22aa3fb0 con 0x7f32a8007030 2023-10-05T19:28:38.953 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.946+0000 7f32bd65b700 1 -- 172.21.15.167:0/3986483093 <== osd.0 v2:172.21.15.118:6800/27690 29 ==== watch-notify(notify (1) cookie 94687430049520 notify 128849018898 ret 0) v3 ==== 767+0+0 (crc 0 0 0) 0x7f32b804e380 con 0x7f32a8007030 2023-10-05T19:28:38.953 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.946+0000 7f32b67fc700 10 rgw watcher librados: RGWWatcher::handle_notify() notify_id 128849018898 cookie 94687430049520 notifier 4599 bl.length()=725 2023-10-05T19:28:38.954 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.946+0000 7f32bd65b700 1 -- 172.21.15.167:0/3986483093 <== osd.0 v2:172.21.15.118:6800/27690 30 ==== osd_op_reply(110 notify.7 [notify cookie 94687430574032 out=8b] v0'0 uv186 ondisk = 0) v8 ==== 152+0+8 (crc 0 0 0) 0x7f32b8052040 con 0x7f32a8007030 2023-10-05T19:28:38.954 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.946+0000 7f32b67fc700 10 rgw watcher librados: cache put: name=default.rgw.meta+root+.bucket.meta.myfoo:9a9df965-6612-45d4-a45f-6e3c889bacc2.4223.1 info.flags=0x17 2023-10-05T19:28:38.954 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.946+0000 7f32b67fc700 10 rgw watcher librados: moving default.rgw.meta+root+.bucket.meta.myfoo:9a9df965-6612-45d4-a45f-6e3c889bacc2.4223.1 to cache LRU end 2023-10-05T19:28:38.954 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.946+0000 7f32b67fc700 10 rgw watcher librados: updating xattr: name=user.rgw.acl bl.length()=123 2023-10-05T19:28:38.954 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.946+0000 7f32b67fc700 1 -- 172.21.15.167:0/3986483093 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:111 7.9 7:93e5b521:::notify.7:head [notify-ack in=20b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x7f329c003d90 con 0x7f32a8007030 2023-10-05T19:28:38.955 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.946+0000 7f32bd65b700 1 -- 172.21.15.167:0/3986483093 <== osd.0 v2:172.21.15.118:6800/27690 31 ==== watch-notify(notify_complete (2) cookie 94687430574032 notify 128849018898 ret 0) v3 ==== 42+0+48 (crc 0 0 0) 0x7f32b8036950 con 0x7f32a8007030 2023-10-05T19:28:38.955 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.950+0000 7f32bf325e40 20 handle_bi_update(): bucket=:myfoo[9a9df965-6612-45d4-a45f-6e3c889bacc2.4223.1]): orig_sources= new_sources= 2023-10-05T19:28:38.955 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.950+0000 7f32bd65b700 1 -- 172.21.15.167:0/3986483093 <== osd.0 v2:172.21.15.118:6800/27690 32 ==== osd_op_reply(111 notify.7 [notify-ack] v0'0 uv186 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f32b8052040 con 0x7f32a8007030 2023-10-05T19:28:38.955 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.950+0000 7f32bf325e40 20 handle_bi_update(): bucket=:myfoo[9a9df965-6612-45d4-a45f-6e3c889bacc2.4223.1]): potential sources added=[] removed=[] 2023-10-05T19:28:38.955 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.950+0000 7f32bf325e40 20 handle_bi_update(): bucket=:myfoo[9a9df965-6612-45d4-a45f-6e3c889bacc2.4223.1]): orig_dests= new_dests= 2023-10-05T19:28:38.955 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.950+0000 7f32bf325e40 20 handle_bi_update(): bucket=:myfoo[9a9df965-6612-45d4-a45f-6e3c889bacc2.4223.1]): potential dests added=[] removed=[] 2023-10-05T19:28:38.955 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.950+0000 7f32bf325e40 1 -- 172.21.15.167:0/3986483093 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:112 8.18 8:1e6ca5ed:users.uid::foo:head [delete,create,call version.set in=58b,writefull 0~357 in=357b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x561e22a9ffd0 con 0x561e229544b0 2023-10-05T19:28:38.957 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.950+0000 7f32bd65b700 1 -- 172.21.15.167:0/3986483093 <== osd.1 v2:172.21.15.167:6800/30035 37 ==== osd_op_reply(112 foo [delete,create,call,writefull 0~357] v30'13 uv13 ondisk = 0) v8 ==== 273+0+0 (crc 0 0 0) 0x7f32b8052040 con 0x561e229544b0 2023-10-05T19:28:38.957 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.950+0000 7f32bf325e40 10 cache put: name=default.rgw.meta+users.uid+foo info.flags=0x17 2023-10-05T19:28:38.957 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.950+0000 7f32bf325e40 10 moving default.rgw.meta+users.uid+foo to cache LRU end 2023-10-05T19:28:38.957 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.950+0000 7f32bf325e40 10 distributing notification oid=default.rgw.control:notify.4 cni=[op: 0, obj: default.rgw.meta:users.uid:foo, ofs0, ns] 2023-10-05T19:28:38.957 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.950+0000 7f32bf325e40 1 -- 172.21.15.167:0/3986483093 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:113 7.d 7:b4812045:::notify.4:head [notify cookie 94687430574032 in=545b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x561e22a9c6a0 con 0x561e229544b0 2023-10-05T19:28:38.957 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.950+0000 7f32bd65b700 1 -- 172.21.15.167:0/3986483093 <== osd.1 v2:172.21.15.167:6800/30035 38 ==== watch-notify(notify (1) cookie 94687430039920 notify 128849018898 ret 0) v3 ==== 575+0+0 (crc 0 0 0) 0x7f32b8036950 con 0x561e229544b0 2023-10-05T19:28:38.957 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.950+0000 7f32b67fc700 10 rgw watcher librados: RGWWatcher::handle_notify() notify_id 128849018898 cookie 94687430039920 notifier 4599 bl.length()=533 2023-10-05T19:28:38.958 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.950+0000 7f32b67fc700 10 rgw watcher librados: cache put: name=default.rgw.meta+users.uid+foo info.flags=0x17 2023-10-05T19:28:38.958 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.950+0000 7f32b67fc700 10 rgw watcher librados: moving default.rgw.meta+users.uid+foo to cache LRU end 2023-10-05T19:28:38.958 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.950+0000 7f32bd65b700 1 -- 172.21.15.167:0/3986483093 <== osd.1 v2:172.21.15.167:6800/30035 39 ==== osd_op_reply(113 notify.4 [notify cookie 94687430574032 out=8b] v0'0 uv186 ondisk = 0) v8 ==== 152+0+8 (crc 0 0 0) 0x7f32b8052040 con 0x561e229544b0 2023-10-05T19:28:38.958 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.950+0000 7f32b67fc700 1 -- 172.21.15.167:0/3986483093 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:114 7.d 7:b4812045:::notify.4:head [notify-ack in=20b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e30) v8 -- 0x7f329c003d90 con 0x561e229544b0 2023-10-05T19:28:38.958 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.954+0000 7f32bd65b700 1 -- 172.21.15.167:0/3986483093 <== osd.1 v2:172.21.15.167:6800/30035 40 ==== watch-notify(notify_complete (2) cookie 94687430574032 notify 128849018898 ret 0) v3 ==== 42+0+48 (crc 0 0 0) 0x7f32b8036e30 con 0x561e229544b0 2023-10-05T19:28:38.959 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.954+0000 7f32bd65b700 1 -- 172.21.15.167:0/3986483093 <== osd.1 v2:172.21.15.167:6800/30035 41 ==== osd_op_reply(114 notify.4 [notify-ack] v0'0 uv186 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f32b8052040 con 0x561e229544b0 2023-10-05T19:28:38.959 INFO:teuthology.orchestra.run.smithi167.stdout:{ 2023-10-05T19:28:38.959 INFO:teuthology.orchestra.run.smithi167.stdout: "user_id": "foo", 2023-10-05T19:28:38.959 INFO:teuthology.orchestra.run.smithi167.stdout: "display_name": "Foo", 2023-10-05T19:28:38.959 INFO:teuthology.orchestra.run.smithi167.stdout: "email": "foo@foo.com", 2023-10-05T19:28:38.959 INFO:teuthology.orchestra.run.smithi167.stdout: "suspended": 0, 2023-10-05T19:28:38.959 INFO:teuthology.orchestra.run.smithi167.stdout: "max_buckets": 4, 2023-10-05T19:28:38.959 INFO:teuthology.orchestra.run.smithi167.stdout: "subusers": [], 2023-10-05T19:28:38.959 INFO:teuthology.orchestra.run.smithi167.stdout: "keys": [ 2023-10-05T19:28:38.960 INFO:teuthology.orchestra.run.smithi167.stdout: { 2023-10-05T19:28:38.960 INFO:teuthology.orchestra.run.smithi167.stdout: "user": "foo", 2023-10-05T19:28:38.960 INFO:teuthology.orchestra.run.smithi167.stdout: "access_key": "9te6NH5mcdcq0Tc5i8i1", 2023-10-05T19:28:38.960 INFO:teuthology.orchestra.run.smithi167.stdout: "secret_key": "Ny4IOauQoL18Gp2zM7lC1vLmoawgqcYP/YGcWfXu" 2023-10-05T19:28:38.960 INFO:teuthology.orchestra.run.smithi167.stdout: } 2023-10-05T19:28:38.960 INFO:teuthology.orchestra.run.smithi167.stdout: ], 2023-10-05T19:28:38.960 INFO:teuthology.orchestra.run.smithi167.stdout: "swift_keys": [], 2023-10-05T19:28:38.960 INFO:teuthology.orchestra.run.smithi167.stdout: "caps": [], 2023-10-05T19:28:38.960 INFO:teuthology.orchestra.run.smithi167.stdout: "op_mask": "read, write, delete", 2023-10-05T19:28:38.960 INFO:teuthology.orchestra.run.smithi167.stdout: "default_placement": "", 2023-10-05T19:28:38.960 INFO:teuthology.orchestra.run.smithi167.stdout: "default_storage_class": "", 2023-10-05T19:28:38.961 INFO:teuthology.orchestra.run.smithi167.stdout: "placement_tags": [], 2023-10-05T19:28:38.961 INFO:teuthology.orchestra.run.smithi167.stdout: "bucket_quota": { 2023-10-05T19:28:38.961 INFO:teuthology.orchestra.run.smithi167.stdout: "enabled": false, 2023-10-05T19:28:38.961 INFO:teuthology.orchestra.run.smithi167.stdout: "check_on_raw": false, 2023-10-05T19:28:38.961 INFO:teuthology.orchestra.run.smithi167.stdout: "max_size": -1, 2023-10-05T19:28:38.961 INFO:teuthology.orchestra.run.smithi167.stdout: "max_size_kb": 0, 2023-10-05T19:28:38.961 INFO:teuthology.orchestra.run.smithi167.stdout: "max_objects": -1 2023-10-05T19:28:38.961 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2023-10-05T19:28:38.961 INFO:teuthology.orchestra.run.smithi167.stdout: "user_quota": { 2023-10-05T19:28:38.961 INFO:teuthology.orchestra.run.smithi167.stdout: "enabled": false, 2023-10-05T19:28:38.961 INFO:teuthology.orchestra.run.smithi167.stdout: "check_on_raw": false, 2023-10-05T19:28:38.962 INFO:teuthology.orchestra.run.smithi167.stdout: "max_size": -1, 2023-10-05T19:28:38.962 INFO:teuthology.orchestra.run.smithi167.stdout: "max_size_kb": 0, 2023-10-05T19:28:38.962 INFO:teuthology.orchestra.run.smithi167.stdout: "max_objects": -1 2023-10-05T19:28:38.962 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2023-10-05T19:28:38.962 INFO:teuthology.orchestra.run.smithi167.stdout: "temp_url_keys": [], 2023-10-05T19:28:38.962 INFO:teuthology.orchestra.run.smithi167.stdout: "type": "rgw", 2023-10-05T19:28:38.962 INFO:teuthology.orchestra.run.smithi167.stdout: "mfa_ids": [] 2023-10-05T19:28:38.962 INFO:teuthology.orchestra.run.smithi167.stdout:} 2023-10-05T19:28:38.962 INFO:teuthology.orchestra.run.smithi167.stdout: 2023-10-05T19:28:38.963 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.954+0000 7f32bf325e40 1 -- 172.21.15.167:0/3986483093 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:115 7.1f 7:f95f44c2:::notify.0:head [watch unwatch cookie 94687430023728] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x561e229f5890 con 0x7f32b0017540 2023-10-05T19:28:38.963 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.954+0000 7f32b7fff700 1 -- 172.21.15.167:0/3986483093 <== osd.2 v2:172.21.15.167:6816/30037 20 ==== osd_op_reply(115 notify.0 [watch unwatch cookie 94687430023728] v30'188 uv186 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f32a8032040 con 0x7f32b0017540 2023-10-05T19:28:38.963 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.954+0000 7f32bf325e40 20 remove_watcher() i=0 2023-10-05T19:28:38.963 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.954+0000 7f32bf325e40 2 removed watcher, disabling cache 2023-10-05T19:28:38.963 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.954+0000 7f32bf325e40 1 -- 172.21.15.167:0/3986483093 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:116 7.0 7:05bf5b68:::notify.1:head [watch unwatch cookie 94687430030240] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x561e229f5890 con 0x561e229544b0 2023-10-05T19:28:38.964 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.958+0000 7f32bd65b700 1 -- 172.21.15.167:0/3986483093 <== osd.1 v2:172.21.15.167:6800/30035 42 ==== osd_op_reply(116 notify.1 [watch unwatch cookie 94687430030240] v30'188 uv186 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f32b8052040 con 0x561e229544b0 2023-10-05T19:28:38.964 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.958+0000 7f32bf325e40 20 remove_watcher() i=1 2023-10-05T19:28:38.965 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.958+0000 7f32bf325e40 1 -- 172.21.15.167:0/3986483093 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:117 7.15 7:a93a5511:::notify.2:head [watch unwatch cookie 94687430032528] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x561e229f5890 con 0x561e22a154f0 2023-10-05T19:28:38.967 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.962+0000 7f32bce5a700 1 -- 172.21.15.167:0/3986483093 <== osd.3 v2:172.21.15.167:6808/30036 27 ==== osd_op_reply(117 notify.2 [watch unwatch cookie 94687430032528] v30'188 uv186 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f32b0016070 con 0x561e22a154f0 2023-10-05T19:28:38.967 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.962+0000 7f32bf325e40 20 remove_watcher() i=2 2023-10-05T19:28:38.967 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.962+0000 7f32bf325e40 1 -- 172.21.15.167:0/3986483093 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:118 7.e 7:7759931f:::notify.3:head [watch unwatch cookie 94687430038816] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x561e229f5890 con 0x7f32b0017540 2023-10-05T19:28:38.970 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.962+0000 7f32b7fff700 1 -- 172.21.15.167:0/3986483093 <== osd.2 v2:172.21.15.167:6816/30037 21 ==== osd_op_reply(118 notify.3 [watch unwatch cookie 94687430038816] v30'188 uv186 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f32a8032040 con 0x7f32b0017540 2023-10-05T19:28:38.970 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.962+0000 7f32bf325e40 20 remove_watcher() i=3 2023-10-05T19:28:38.970 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.962+0000 7f32bf325e40 1 -- 172.21.15.167:0/3986483093 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:119 7.d 7:b4812045:::notify.4:head [watch unwatch cookie 94687430039920] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x561e229f5890 con 0x561e229544b0 2023-10-05T19:28:38.973 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.966+0000 7f32bd65b700 1 -- 172.21.15.167:0/3986483093 <== osd.1 v2:172.21.15.167:6800/30035 43 ==== osd_op_reply(119 notify.4 [watch unwatch cookie 94687430039920] v30'188 uv186 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f32b8052040 con 0x561e229544b0 2023-10-05T19:28:38.973 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.966+0000 7f32bf325e40 20 remove_watcher() i=4 2023-10-05T19:28:38.973 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.966+0000 7f32bf325e40 1 -- 172.21.15.167:0/3986483093 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:120 7.3 7:c609908c:::notify.5:head [watch unwatch cookie 94687430042496] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x561e229f5890 con 0x7f32a8007030 2023-10-05T19:28:38.976 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.970+0000 7f32bd65b700 1 -- 172.21.15.167:0/3986483093 <== osd.0 v2:172.21.15.118:6800/27690 33 ==== osd_op_reply(120 notify.5 [watch unwatch cookie 94687430042496] v30'188 uv186 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f32b8052040 con 0x7f32a8007030 2023-10-05T19:28:38.976 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.970+0000 7f32bf325e40 20 remove_watcher() i=5 2023-10-05T19:28:38.976 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.970+0000 7f32bf325e40 1 -- 172.21.15.167:0/3986483093 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:121 7.14 7:2b04a3e9:::notify.6:head [watch unwatch cookie 94687430044784] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x561e229f5890 con 0x7f32a8007030 2023-10-05T19:28:38.979 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.974+0000 7f32bd65b700 1 -- 172.21.15.167:0/3986483093 <== osd.0 v2:172.21.15.118:6800/27690 34 ==== osd_op_reply(121 notify.6 [watch unwatch cookie 94687430044784] v30'188 uv186 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f32b8052040 con 0x7f32a8007030 2023-10-05T19:28:38.979 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.974+0000 7f32bf325e40 20 remove_watcher() i=6 2023-10-05T19:28:38.979 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.974+0000 7f32bf325e40 1 -- 172.21.15.167:0/3986483093 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:122 7.9 7:93e5b521:::notify.7:head [watch unwatch cookie 94687430049520] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x561e229f5890 con 0x7f32a8007030 2023-10-05T19:28:38.982 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.978+0000 7f32bd65b700 1 -- 172.21.15.167:0/3986483093 <== osd.0 v2:172.21.15.118:6800/27690 35 ==== osd_op_reply(122 notify.7 [watch unwatch cookie 94687430049520] v30'188 uv186 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f32b8052040 con 0x7f32a8007030 2023-10-05T19:28:38.982 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.978+0000 7f32bf325e40 20 remove_watcher() i=7 2023-10-05T19:28:38.984 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.978+0000 7f328dfe3700 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2023-10-05T19:28:38.984 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.978+0000 7f32bf325e40 1 -- 172.21.15.167:0/3986483093 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:123 6.4 6:22d26bf9:::data_loggenerations_metadata:head [watch unwatch cookie 94687430050624] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x561e229f5890 con 0x561e229544b0 2023-10-05T19:28:38.988 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.982+0000 7f32bd65b700 1 -- 172.21.15.167:0/3986483093 <== osd.1 v2:172.21.15.167:6800/30035 44 ==== osd_op_reply(123 data_loggenerations_metadata [watch unwatch cookie 94687430050624] v30'172 uv1 ondisk = 0) v8 ==== 172+0+0 (crc 0 0 0) 0x7f32b8052040 con 0x561e229544b0 2023-10-05T19:28:38.988 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.982+0000 7f32bf325e40 1 -- 172.21.15.167:0/3986483093 >> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] conn(0x7f32a8007030 msgr2=0x7f32a801ca90 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-05T19:28:38.988 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.982+0000 7f32bf325e40 1 --2- 172.21.15.167:0/3986483093 >> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] conn(0x7f32a8007030 0x7f32a801ca90 crc :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-05T19:28:38.988 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.982+0000 7f32bf325e40 1 -- 172.21.15.167:0/3986483093 >> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] conn(0x561e229544b0 msgr2=0x561e22a11880 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-05T19:28:38.988 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.982+0000 7f32bf325e40 1 --2- 172.21.15.167:0/3986483093 >> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] conn(0x561e229544b0 0x561e22a11880 crc :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-05T19:28:38.988 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.982+0000 7f32bf325e40 1 -- 172.21.15.167:0/3986483093 >> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] conn(0x7f32b0017540 msgr2=0x7f32b0042780 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-05T19:28:38.988 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.982+0000 7f32bf325e40 1 --2- 172.21.15.167:0/3986483093 >> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] conn(0x7f32b0017540 0x7f32b0042780 crc :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-05T19:28:38.988 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.982+0000 7f32bf325e40 1 -- 172.21.15.167:0/3986483093 >> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] conn(0x561e22a154f0 msgr2=0x561e22a17900 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-05T19:28:38.988 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.982+0000 7f32bf325e40 1 --2- 172.21.15.167:0/3986483093 >> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] conn(0x561e22a154f0 0x561e22a17900 crc :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-05T19:28:38.989 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.982+0000 7f32bf325e40 1 -- 172.21.15.167:0/3986483093 >> [v2:172.21.15.167:6824/29889,v1:172.21.15.167:6825/29889] conn(0x7f32a0030c40 msgr2=0x7f32a00330f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-05T19:28:38.989 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.982+0000 7f32bf325e40 1 --2- 172.21.15.167:0/3986483093 >> [v2:172.21.15.167:6824/29889,v1:172.21.15.167:6825/29889] conn(0x7f32a0030c40 0x7f32a00330f0 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7f32a8005810 tx=0x7f32a8015000 comp rx=0 tx=0).stop 2023-10-05T19:28:38.989 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.982+0000 7f32bd65b700 1 -- 172.21.15.167:0/3986483093 reap_dead start 2023-10-05T19:28:38.989 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.982+0000 7f32bf325e40 1 -- 172.21.15.167:0/3986483093 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x561e229531a0 msgr2=0x561e22a05d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-05T19:28:38.989 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.982+0000 7f32bf325e40 1 --2- 172.21.15.167:0/3986483093 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x561e229531a0 0x561e22a05d80 secure :-1 s=READY pgs=509 cs=0 l=1 rev1=1 crypto rx=0x7f32b000b850 tx=0x7f32b0016000 comp rx=0 tx=0).stop 2023-10-05T19:28:38.989 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.982+0000 7f32bf325e40 1 -- 172.21.15.167:0/3986483093 shutdown_connections 2023-10-05T19:28:38.989 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.982+0000 7f32bf325e40 1 --2- 172.21.15.167:0/3986483093 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x561e229531a0 0x561e22a05d80 unknown :-1 s=CLOSED pgs=509 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-05T19:28:38.989 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.982+0000 7f32bf325e40 1 -- 172.21.15.167:0/3986483093 >> 172.21.15.167:0/3986483093 conn(0x561e229d9830 msgr2=0x561e229f9630 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-05T19:28:38.989 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.982+0000 7f32bf325e40 1 -- 172.21.15.167:0/3986483093 shutdown_connections 2023-10-05T19:28:38.989 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.982+0000 7f32bf325e40 1 -- 172.21.15.167:0/3986483093 wait complete. 2023-10-05T19:28:38.990 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.982+0000 7f32bf325e40 1 -- 172.21.15.167:0/2562358716 >> [v2:172.21.15.167:6824/29889,v1:172.21.15.167:6825/29889] conn(0x7f31fc030c50 msgr2=0x7f31fc033100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-05T19:28:38.990 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.982+0000 7f32bf325e40 1 --2- 172.21.15.167:0/2562358716 >> [v2:172.21.15.167:6824/29889,v1:172.21.15.167:6825/29889] conn(0x7f31fc030c50 0x7f31fc033100 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x561e22a1f770 tx=0x7f32b8012e70 comp rx=0 tx=0).stop 2023-10-05T19:28:38.990 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.982+0000 7f32bf325e40 1 -- 172.21.15.167:0/2562358716 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x561e22a2dd70 msgr2=0x561e22a33760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-05T19:28:38.991 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.982+0000 7f32bf325e40 1 --2- 172.21.15.167:0/2562358716 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x561e22a2dd70 0x561e22a33760 secure :-1 s=READY pgs=511 cs=0 l=1 rev1=1 crypto rx=0x7f32a8012f30 tx=0x7f32a802f000 comp rx=0 tx=0).stop 2023-10-05T19:28:38.991 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.982+0000 7f32bf325e40 1 -- 172.21.15.167:0/2562358716 shutdown_connections 2023-10-05T19:28:38.991 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.982+0000 7f32bf325e40 1 --2- 172.21.15.167:0/2562358716 >> [v2:172.21.15.167:6824/29889,v1:172.21.15.167:6825/29889] conn(0x7f31fc030c50 0x7f31fc033100 unknown :-1 s=CLOSED pgs=177 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-05T19:28:38.991 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.982+0000 7f32bf325e40 1 --2- 172.21.15.167:0/2562358716 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x561e22a2dd70 0x561e22a33760 unknown :-1 s=CLOSED pgs=511 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-05T19:28:38.991 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.982+0000 7f32bf325e40 1 -- 172.21.15.167:0/2562358716 >> 172.21.15.167:0/2562358716 conn(0x561e22a29130 msgr2=0x561e22a271d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-05T19:28:38.991 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.982+0000 7f32bf325e40 1 -- 172.21.15.167:0/2562358716 shutdown_connections 2023-10-05T19:28:38.991 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:38.982+0000 7f32bf325e40 1 -- 172.21.15.167:0/2562358716 wait complete. 2023-10-05T19:28:38.995 DEBUG:tasks.util.rgw: json result: {'user_id': 'foo', 'display_name': 'Foo', 'email': 'foo@foo.com', 'suspended': 0, 'max_buckets': 4, 'subusers': [], 'keys': [{'user': 'foo', 'access_key': '9te6NH5mcdcq0Tc5i8i1', 'secret_key': 'Ny4IOauQoL18Gp2zM7lC1vLmoawgqcYP/YGcWfXu'}], 'swift_keys': [], 'caps': [], 'op_mask': 'read, write, delete', 'default_placement': '', 'default_storage_class': '', 'placement_tags': [], 'bucket_quota': {'enabled': False, 'check_on_raw': False, 'max_size': -1, 'max_size_kb': 0, 'max_objects': -1}, 'user_quota': {'enabled': False, 'check_on_raw': False, 'max_size': -1, 'max_size_kb': 0, 'max_objects': -1}, 'temp_url_keys': [], 'type': 'rgw', 'mfa_ids': []} 2023-10-05T19:28:39.009 INFO:tasks.radosgw_admin:RL: put_obj myfoo foo bytes_out=3 bytes_in=0 failed=False 2023-10-05T19:28:39.280 INFO:tasks.radosgw_admin:RL: put_obj myfoo foo bytes_out=30000000 bytes_in=0 failed=False 2023-10-05T19:28:39.293 INFO:tasks.radosgw_admin:RL: delete_obj myfoo foo bytes_out=0 bytes_in=0 failed=False 2023-10-05T19:28:54.294 INFO:tasks.util.rgw:rgwadmin: client.0 : ['gc', 'list', '--include-all'] 2023-10-05T19:28:54.294 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'ceph', 'gc', 'list', '--include-all'] 2023-10-05T19:28:54.294 DEBUG:teuthology.orchestra.run.smithi167:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster ceph gc list --include-all 2023-10-05T19:28:54.334 INFO:teuthology.orchestra.run.smithi167.stderr:ignoring --setuser ceph since I am not root 2023-10-05T19:28:54.334 INFO:teuthology.orchestra.run.smithi167.stderr:ignoring --setgroup ceph since I am not root 2023-10-05T19:28:54.334 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:54.330+0000 7feeef1e2e40 1 Processor -- start 2023-10-05T19:28:54.335 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:54.330+0000 7feeef1e2e40 1 -- start start 2023-10-05T19:28:54.335 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:54.330+0000 7feeef1e2e40 1 --2- >> v2:172.21.15.118:3300/0 conn(0x55c572b972f0 0x55c572b996d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-05T19:28:54.335 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:54.330+0000 7feeef1e2e40 1 -- --> v1:172.21.15.118:6789/0 -- auth(proto 0 26 bytes epoch 0) v1 -- 0x55c572adfbd0 con 0x55c572b073d0 2023-10-05T19:28:54.336 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:54.330+0000 7feeef1e2e40 1 -- --> v2:172.21.15.118:3300/0 -- mon_getmap magic: 0 v1 -- 0x55c572a9cee0 con 0x55c572b972f0 2023-10-05T19:28:54.336 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:54.330+0000 7feeec516700 1 --2- >> v2:172.21.15.118:3300/0 conn(0x55c572b972f0 0x55c572b996d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-05T19:28:54.336 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:54.330+0000 7feeecd17700 1 --1- >> v1:172.21.15.118:6789/0 conn(0x55c572b073d0 0x55c572b95bd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.118:6789/0 says I am v1:172.21.15.167:48658/0 (socket says 172.21.15.167:48658) 2023-10-05T19:28:54.336 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:54.330+0000 7feeecd17700 1 -- 172.21.15.167:0/288186259 learned_addr learned my addr 172.21.15.167:0/288186259 (peer_addr_for_me v1:172.21.15.167:0/0) 2023-10-05T19:28:54.336 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:54.330+0000 7feeec516700 1 -- 172.21.15.167:0/288186259 >> v1:172.21.15.118:6789/0 conn(0x55c572b073d0 legacy=0x55c572b95bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-05T19:28:54.337 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:54.330+0000 7feeec516700 1 -- 172.21.15.167:0/288186259 --> v2:172.21.15.118:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55c572b9ac20 con 0x55c572b972f0 2023-10-05T19:28:54.337 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:54.330+0000 7feeec516700 1 --2- 172.21.15.167:0/288186259 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55c572b972f0 0x55c572b996d0 secure :-1 s=READY pgs=513 cs=0 l=1 rev1=1 crypto rx=0x7feed400a5d0 tx=0x7feed4005a50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f64aea3307839a83 server_cookie=0 in_seq=0 out_seq=0 2023-10-05T19:28:54.338 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:54.330+0000 7feeebd15700 1 -- 172.21.15.167:0/288186259 <== mon.0 v2:172.21.15.118:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7feed40086a0 con 0x55c572b972f0 2023-10-05T19:28:54.338 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:54.330+0000 7feeebd15700 1 -- 172.21.15.167:0/288186259 <== mon.0 v2:172.21.15.118:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7feed4008800 con 0x55c572b972f0 2023-10-05T19:28:54.338 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:54.330+0000 7feeebd15700 1 -- 172.21.15.167:0/288186259 <== mon.0 v2:172.21.15.118:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7feed4008ad0 con 0x55c572b972f0 2023-10-05T19:28:54.338 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:54.334+0000 7feeef1e2e40 1 -- 172.21.15.167:0/288186259 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55c572b972f0 msgr2=0x55c572b996d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-05T19:28:54.338 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:54.334+0000 7feeef1e2e40 1 --2- 172.21.15.167:0/288186259 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55c572b972f0 0x55c572b996d0 secure :-1 s=READY pgs=513 cs=0 l=1 rev1=1 crypto rx=0x7feed400a5d0 tx=0x7feed4005a50 comp rx=0 tx=0).stop 2023-10-05T19:28:54.338 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:54.334+0000 7feeef1e2e40 1 -- 172.21.15.167:0/288186259 shutdown_connections 2023-10-05T19:28:54.338 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:54.334+0000 7feeef1e2e40 1 --2- 172.21.15.167:0/288186259 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55c572b972f0 0x55c572b996d0 unknown :-1 s=CLOSED pgs=513 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-05T19:28:54.338 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:54.334+0000 7feeef1e2e40 1 -- 172.21.15.167:0/288186259 >> 172.21.15.167:0/288186259 conn(0x55c572b8d720 msgr2=0x55c572b8fb30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-05T19:28:54.339 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:54.334+0000 7feeef1e2e40 1 -- 172.21.15.167:0/288186259 shutdown_connections 2023-10-05T19:28:54.339 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:54.334+0000 7feeef1e2e40 1 -- 172.21.15.167:0/288186259 wait complete. 2023-10-05T19:28:54.340 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:54.334+0000 7feeef1e2e40 1 Processor -- start 2023-10-05T19:28:54.340 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:54.334+0000 7feeef1e2e40 1 -- start start 2023-10-05T19:28:54.340 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:54.334+0000 7feeef1e2e40 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55c572b073d0 0x55c572bb1f20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-05T19:28:54.340 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:54.334+0000 7feeef1e2e40 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c572a60e30 con 0x55c572b073d0 2023-10-05T19:28:54.340 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:54.334+0000 7feeecd17700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55c572b073d0 0x55c572bb1f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-05T19:28:54.340 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:54.334+0000 7feeecd17700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55c572b073d0 0x55c572bb1f20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.118:3300/0 says I am v2:172.21.15.167:53124/0 (socket says 172.21.15.167:53124) 2023-10-05T19:28:54.340 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:54.334+0000 7feeecd17700 1 -- 172.21.15.167:0/2756141501 learned_addr learned my addr 172.21.15.167:0/2756141501 (peer_addr_for_me v2:172.21.15.167:0/0) 2023-10-05T19:28:54.340 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:54.334+0000 7feeecd17700 1 -- 172.21.15.167:0/2756141501 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feed4005cf0 con 0x55c572b073d0 2023-10-05T19:28:54.342 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:54.334+0000 7feeecd17700 1 --2- 172.21.15.167:0/2756141501 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55c572b073d0 0x55c572bb1f20 secure :-1 s=READY pgs=514 cs=0 l=1 rev1=1 crypto rx=0x7feedc001bf0 tx=0x7feedc0017a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ed1fab1abfb9ca86 server_cookie=0 in_seq=0 out_seq=0 2023-10-05T19:28:54.342 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:54.334+0000 7feeead13700 1 -- 172.21.15.167:0/2756141501 <== mon.0 v2:172.21.15.118:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7feedc00bbf0 con 0x55c572b073d0 2023-10-05T19:28:54.342 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:54.334+0000 7feeead13700 1 -- 172.21.15.167:0/2756141501 <== mon.0 v2:172.21.15.118:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7feedc00bd50 con 0x55c572b073d0 2023-10-05T19:28:54.342 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:54.334+0000 7feeead13700 1 -- 172.21.15.167:0/2756141501 <== mon.0 v2:172.21.15.118:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7feedc00e470 con 0x55c572b073d0 2023-10-05T19:28:54.342 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:54.334+0000 7feeef1e2e40 1 -- 172.21.15.167:0/2756141501 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55c572b073d0 msgr2=0x55c572bb1f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-05T19:28:54.342 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:54.334+0000 7feeef1e2e40 1 --2- 172.21.15.167:0/2756141501 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55c572b073d0 0x55c572bb1f20 secure :-1 s=READY pgs=514 cs=0 l=1 rev1=1 crypto rx=0x7feedc001bf0 tx=0x7feedc0017a0 comp rx=0 tx=0).stop 2023-10-05T19:28:54.342 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:54.334+0000 7feeef1e2e40 1 -- 172.21.15.167:0/2756141501 shutdown_connections 2023-10-05T19:28:54.343 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:54.334+0000 7feeef1e2e40 1 --2- 172.21.15.167:0/2756141501 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55c572b073d0 0x55c572bb1f20 unknown :-1 s=CLOSED pgs=514 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-05T19:28:54.343 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:54.334+0000 7feeef1e2e40 1 -- 172.21.15.167:0/2756141501 >> 172.21.15.167:0/2756141501 conn(0x55c572b8d720 msgr2=0x55c572badae0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-05T19:28:54.343 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:54.334+0000 7feeef1e2e40 1 -- 172.21.15.167:0/2756141501 shutdown_connections 2023-10-05T19:28:54.343 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:54.334+0000 7feeef1e2e40 1 -- 172.21.15.167:0/2756141501 wait complete. 2023-10-05T19:28:54.343 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:54.338+0000 7feeef1e2e40 1 Processor -- start 2023-10-05T19:28:54.343 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:54.338+0000 7feeef1e2e40 1 -- start start 2023-10-05T19:28:54.343 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:54.338+0000 7feeef1e2e40 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55c572b073d0 0x55c572bbaf30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-05T19:28:54.343 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:54.338+0000 7feeef1e2e40 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c572a33ed0 con 0x55c572b073d0 2023-10-05T19:28:54.345 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:54.338+0000 7feeecd17700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55c572b073d0 0x55c572bbaf30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-05T19:28:54.345 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:54.338+0000 7feeecd17700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55c572b073d0 0x55c572bbaf30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.118:3300/0 says I am v2:172.21.15.167:53136/0 (socket says 172.21.15.167:53136) 2023-10-05T19:28:54.345 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:54.338+0000 7feeecd17700 1 -- 172.21.15.167:0/24894888 learned_addr learned my addr 172.21.15.167:0/24894888 (peer_addr_for_me v2:172.21.15.167:0/0) 2023-10-05T19:28:54.345 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:54.338+0000 7feeecd17700 1 -- 172.21.15.167:0/24894888 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55c572adfbd0 con 0x55c572b073d0 2023-10-05T19:28:54.345 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:54.338+0000 7feeecd17700 1 --2- 172.21.15.167:0/24894888 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55c572b073d0 0x55c572bbaf30 secure :-1 s=READY pgs=515 cs=0 l=1 rev1=1 crypto rx=0x55c572bae600 tx=0x7feedc016000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-05T19:28:54.345 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:54.338+0000 7feee9510700 1 -- 172.21.15.167:0/24894888 <== mon.0 v2:172.21.15.118:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7feedc01f070 con 0x55c572b073d0 2023-10-05T19:28:54.345 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:54.338+0000 7feee9510700 1 -- 172.21.15.167:0/24894888 <== mon.0 v2:172.21.15.118:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7feedc01a2e0 con 0x55c572b073d0 2023-10-05T19:28:54.345 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:54.338+0000 7feeef1e2e40 1 -- 172.21.15.167:0/24894888 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x55c572bbb470 con 0x55c572b073d0 2023-10-05T19:28:54.345 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:54.338+0000 7feee9510700 1 -- 172.21.15.167:0/24894888 <== mon.0 v2:172.21.15.118:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7feedc00bec0 con 0x55c572b073d0 2023-10-05T19:28:54.345 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:54.338+0000 7feeef1e2e40 1 -- 172.21.15.167:0/24894888 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55c572bbb8b0 con 0x55c572b073d0 2023-10-05T19:28:54.346 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:54.338+0000 7feeef1e2e40 20 rados->read ofs=0 len=0 2023-10-05T19:28:54.346 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:54.342+0000 7feee9510700 1 -- 172.21.15.167:0/24894888 <== mon.0 v2:172.21.15.118:3300/0 4 ==== mgrmap(e 5) v1 ==== 40329+0+0 (secure 0 0 0) 0x7feedc01a960 con 0x55c572b073d0 2023-10-05T19:28:54.347 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:54.342+0000 7feee9510700 1 --2- 172.21.15.167:0/24894888 >> [v2:172.21.15.167:6824/29889,v1:172.21.15.167:6825/29889] conn(0x7feecc030c50 0x7feecc033100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-05T19:28:54.347 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:54.342+0000 7feee9510700 1 -- 172.21.15.167:0/24894888 <== mon.0 v2:172.21.15.118:3300/0 5 ==== osd_map(32..32 src has 1..32) v4 ==== 6260+0+0 (secure 0 0 0) 0x7feedc045190 con 0x55c572b073d0 2023-10-05T19:28:54.347 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:54.342+0000 7feeec516700 1 --2- 172.21.15.167:0/24894888 >> [v2:172.21.15.167:6824/29889,v1:172.21.15.167:6825/29889] conn(0x7feecc030c50 0x7feecc033100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-05T19:28:54.347 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:54.342+0000 7feeef1e2e40 1 --2- 172.21.15.167:0/24894888 >> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] conn(0x55c572b972f0 0x55c572bc6970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-05T19:28:54.347 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:54.342+0000 7feeef1e2e40 1 -- 172.21.15.167:0/24894888 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:1 5.12 5:49953fa1:::default.realm:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32) v8 -- 0x55c572bc6fa0 con 0x55c572b972f0 2023-10-05T19:28:54.348 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:54.342+0000 7feeed518700 1 --2- 172.21.15.167:0/24894888 >> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] conn(0x55c572b972f0 0x55c572bc6970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-05T19:28:54.348 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:54.342+0000 7feeec516700 1 --2- 172.21.15.167:0/24894888 >> [v2:172.21.15.167:6824/29889,v1:172.21.15.167:6825/29889] conn(0x7feecc030c50 0x7feecc033100 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7feed4005a90 tx=0x7feed4015000 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-05T19:28:54.348 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:54.342+0000 7feeed518700 1 --2- 172.21.15.167:0/24894888 >> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] conn(0x55c572b972f0 0x55c572bc6970 crc :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-05T19:28:54.348 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:54.342+0000 7feeed518700 1 -- 172.21.15.167:0/24894888 <== osd.1 v2:172.21.15.167:6800/30035 1 ==== osd_op_reply(1 default.realm [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 ==== 157+0+0 (crc 0 0 0) 0x7feee400a5a0 con 0x55c572b972f0 2023-10-05T19:28:54.349 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:54.342+0000 7feeef1e2e40 20 rados_obj.operate() r=-2 bl.length=0 2023-10-05T19:28:54.349 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:54.342+0000 7feeef1e2e40 20 realm 2023-10-05T19:28:54.349 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:54.342+0000 7feeef1e2e40 20 rados->read ofs=0 len=0 2023-10-05T19:28:54.349 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:54.342+0000 7feeef1e2e40 1 -- 172.21.15.167:0/24894888 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:2 5.12 5:49953fa1:::default.realm:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32) v8 -- 0x55c572bc3fd0 con 0x55c572b972f0 2023-10-05T19:28:54.350 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:54.342+0000 7feeed518700 1 -- 172.21.15.167:0/24894888 <== osd.1 v2:172.21.15.167:6800/30035 2 ==== osd_op_reply(2 default.realm [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 ==== 157+0+0 (crc 0 0 0) 0x7feee400a5a0 con 0x55c572b972f0 2023-10-05T19:28:54.351 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:54.342+0000 7feeef1e2e40 20 rados_obj.operate() r=-2 bl.length=0 2023-10-05T19:28:54.351 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:54.342+0000 7feeef1e2e40 4 RGWPeriod::init failed to init realm id : (2) No such file or directory 2023-10-05T19:28:54.351 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:54.342+0000 7feeef1e2e40 20 rados->read ofs=0 len=0 2023-10-05T19:28:54.351 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:54.342+0000 7feeef1e2e40 1 -- 172.21.15.167:0/24894888 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:3 5.12 5:49953fa1:::default.realm:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32) v8 -- 0x55c572bc7a80 con 0x55c572b972f0 2023-10-05T19:28:54.351 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:54.342+0000 7feeed518700 1 -- 172.21.15.167:0/24894888 <== osd.1 v2:172.21.15.167:6800/30035 3 ==== osd_op_reply(3 default.realm [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 ==== 157+0+0 (crc 0 0 0) 0x7feee400a5a0 con 0x55c572b972f0 2023-10-05T19:28:54.351 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:54.342+0000 7feeef1e2e40 20 rados_obj.operate() r=-2 bl.length=0 2023-10-05T19:28:54.351 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:54.342+0000 7feeef1e2e40 20 rados->read ofs=0 len=0 2023-10-05T19:28:54.351 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:54.342+0000 7feeef1e2e40 1 -- 172.21.15.167:0/24894888 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:4 5.f 5:f43b6ece:::zone_names.default:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32) v8 -- 0x55c572bc7660 con 0x55c572b972f0 2023-10-05T19:28:54.351 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:54.346+0000 7feeed518700 1 -- 172.21.15.167:0/24894888 <== osd.1 v2:172.21.15.167:6800/30035 4 ==== osd_op_reply(4 zone_names.default [read 0~46 out=46b] v0'0 uv1 ondisk = 0) v8 ==== 162+0+46 (crc 0 0 0) 0x7feee400a5a0 con 0x55c572b972f0 2023-10-05T19:28:54.352 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:54.346+0000 7feeef1e2e40 20 rados_obj.operate() r=0 bl.length=46 2023-10-05T19:28:54.352 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:54.346+0000 7feeef1e2e40 20 rados->read ofs=0 len=0 2023-10-05T19:28:54.352 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:54.346+0000 7feeef1e2e40 1 -- 172.21.15.167:0/24894888 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:5 5.1d 5:b8374eb4:::zone_info.9a9df965-6612-45d4-a45f-6e3c889bacc2:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32) v8 -- 0x55c572bc84f0 con 0x55c572b972f0 2023-10-05T19:28:54.352 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:54.346+0000 7feeed518700 1 -- 172.21.15.167:0/24894888 <== osd.1 v2:172.21.15.167:6800/30035 5 ==== osd_op_reply(5 zone_info.9a9df965-6612-45d4-a45f-6e3c889bacc2 [read 0~874 out=874b] v0'0 uv2 ondisk = 0) v8 ==== 190+0+874 (crc 0 0 0) 0x7feee400a5a0 con 0x55c572b972f0 2023-10-05T19:28:54.352 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:54.346+0000 7feeef1e2e40 20 rados_obj.operate() r=0 bl.length=874 2023-10-05T19:28:54.352 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:54.346+0000 7feeef1e2e40 20 searching for the correct realm 2023-10-05T19:28:54.352 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:54.346+0000 7feeef1e2e40 1 --2- 172.21.15.167:0/24894888 >> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] conn(0x55c572bca340 0x55c572bcc720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-05T19:28:54.352 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:54.346+0000 7feeef1e2e40 1 -- 172.21.15.167:0/24894888 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:6 5.0 5:00000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x55c572bccc60 con 0x55c572bca340 2023-10-05T19:28:54.352 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:54.346+0000 7feeecd17700 1 --2- 172.21.15.167:0/24894888 >> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] conn(0x55c572bca340 0x55c572bcc720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-05T19:28:54.353 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:54.346+0000 7feeecd17700 1 --2- 172.21.15.167:0/24894888 >> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] conn(0x55c572bca340 0x55c572bcc720 crc :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-05T19:28:54.353 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:54.346+0000 7feeecd17700 1 -- 172.21.15.167:0/24894888 <== osd.3 v2:172.21.15.167:6808/30036 1 ==== osd_op_reply(6 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7feedc014070 con 0x55c572bca340 2023-10-05T19:28:54.354 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:54.346+0000 7feeecd17700 1 --2- 172.21.15.167:0/24894888 >> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] conn(0x7feedc02d020 0x7feedc03d8d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-05T19:28:54.354 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:54.346+0000 7feeecd17700 1 -- 172.21.15.167:0/24894888 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:7 5.10 5:08000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x55c572bccc60 con 0x7feedc02d020 2023-10-05T19:28:54.354 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:54.346+0000 7feeec516700 1 --2- 172.21.15.167:0/24894888 >> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] conn(0x7feedc02d020 0x7feedc03d8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-05T19:28:54.354 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:54.346+0000 7feeec516700 1 --2- 172.21.15.167:0/24894888 >> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] conn(0x7feedc02d020 0x7feedc03d8d0 crc :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-05T19:28:54.354 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:54.346+0000 7feeec516700 1 -- 172.21.15.167:0/24894888 <== osd.2 v2:172.21.15.167:6816/30037 1 ==== osd_op_reply(7 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7feed4012450 con 0x7feedc02d020 2023-10-05T19:28:54.354 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:54.350+0000 7feeec516700 1 --2- 172.21.15.167:0/24894888 >> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] conn(0x7feed4007030 0x7feed401c920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-05T19:28:54.354 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:54.350+0000 7feeec516700 1 -- 172.21.15.167:0/24894888 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:8 5.8 5:10000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x55c572bccc60 con 0x7feed4007030 2023-10-05T19:28:54.355 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:54.350+0000 7feeed518700 1 --2- 172.21.15.167:0/24894888 >> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] conn(0x7feed4007030 0x7feed401c920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-05T19:28:54.355 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:54.350+0000 7feeed518700 1 --2- 172.21.15.167:0/24894888 >> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] conn(0x7feed4007030 0x7feed401c920 crc :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-05T19:28:54.355 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:54.350+0000 7feeed518700 1 -- 172.21.15.167:0/24894888 <== osd.0 v2:172.21.15.118:6800/27690 1 ==== osd_op_reply(8 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7feee400a5a0 con 0x7feed4007030 2023-10-05T19:28:54.355 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:54.350+0000 7feeed518700 1 -- 172.21.15.167:0/24894888 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:9 5.18 5:18000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x55c572bccc60 con 0x55c572b972f0 2023-10-05T19:28:54.357 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:54.350+0000 7feeed518700 1 -- 172.21.15.167:0/24894888 <== osd.1 v2:172.21.15.167:6800/30035 6 ==== osd_op_reply(9 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7feee400a5a0 con 0x55c572b972f0 2023-10-05T19:28:54.357 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:54.350+0000 7feeed518700 1 -- 172.21.15.167:0/24894888 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:10 5.4 5:20000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x55c572bccc60 con 0x55c572b972f0 2023-10-05T19:28:54.358 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:54.350+0000 7feeed518700 1 -- 172.21.15.167:0/24894888 <== osd.1 v2:172.21.15.167:6800/30035 7 ==== osd_op_reply(10 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7feee400a5a0 con 0x55c572b972f0 2023-10-05T19:28:54.358 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:54.350+0000 7feeed518700 1 -- 172.21.15.167:0/24894888 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:11 5.14 5:28000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x55c572bccc60 con 0x7feed4007030 2023-10-05T19:28:54.358 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:54.354+0000 7feeed518700 1 -- 172.21.15.167:0/24894888 <== osd.0 v2:172.21.15.118:6800/27690 2 ==== osd_op_reply(11 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7feee400a5a0 con 0x7feed4007030 2023-10-05T19:28:54.358 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:54.354+0000 7feeed518700 1 -- 172.21.15.167:0/24894888 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:12 5.c 5:30000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x55c572bccc60 con 0x7feed4007030 2023-10-05T19:28:54.359 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:54.354+0000 7feeed518700 1 -- 172.21.15.167:0/24894888 <== osd.0 v2:172.21.15.118:6800/27690 3 ==== osd_op_reply(12 [pgnls start_epoch 32 out=112b] v23'1 uv1 ondisk = 1) v8 ==== 144+0+112 (crc 0 0 0) 0x7feee400a5a0 con 0x7feed4007030 2023-10-05T19:28:54.359 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:54.354+0000 7feeed518700 1 -- 172.21.15.167:0/24894888 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:13 5.1c 5:38000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x55c572bccc60 con 0x7feedc02d020 2023-10-05T19:28:54.360 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:54.354+0000 7feeec516700 1 -- 172.21.15.167:0/24894888 <== osd.2 v2:172.21.15.167:6816/30037 2 ==== osd_op_reply(13 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7feed4012450 con 0x7feedc02d020 2023-10-05T19:28:54.360 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:54.354+0000 7feeec516700 1 -- 172.21.15.167:0/24894888 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:14 5.2 5:40000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x55c572bccc60 con 0x55c572bca340 2023-10-05T19:28:54.361 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:54.354+0000 7feeecd17700 1 -- 172.21.15.167:0/24894888 <== osd.3 v2:172.21.15.167:6808/30036 2 ==== osd_op_reply(14 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7feedc014070 con 0x55c572bca340 2023-10-05T19:28:54.361 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:54.354+0000 7feeecd17700 1 -- 172.21.15.167:0/24894888 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:15 5.12 5:48000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x55c572bccc60 con 0x55c572b972f0 2023-10-05T19:28:54.361 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:54.354+0000 7feeed518700 1 -- 172.21.15.167:0/24894888 <== osd.1 v2:172.21.15.167:6800/30035 8 ==== osd_op_reply(15 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7feee400a5a0 con 0x55c572b972f0 2023-10-05T19:28:54.361 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:54.354+0000 7feeed518700 1 -- 172.21.15.167:0/24894888 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:16 5.a 5:50000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x55c572bccc60 con 0x7feed4007030 2023-10-05T19:28:54.362 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:54.358+0000 7feeed518700 1 -- 172.21.15.167:0/24894888 <== osd.0 v2:172.21.15.118:6800/27690 4 ==== osd_op_reply(16 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7feee400a5a0 con 0x7feed4007030 2023-10-05T19:28:54.362 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:54.358+0000 7feeed518700 1 -- 172.21.15.167:0/24894888 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:17 5.1a 5:58000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x55c572bccc60 con 0x55c572b972f0 2023-10-05T19:28:54.363 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:54.358+0000 7feeed518700 1 -- 172.21.15.167:0/24894888 <== osd.1 v2:172.21.15.167:6800/30035 9 ==== osd_op_reply(17 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7feee400a5a0 con 0x55c572b972f0 2023-10-05T19:28:54.363 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:54.358+0000 7feeed518700 1 -- 172.21.15.167:0/24894888 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:18 5.6 5:60000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x55c572bccc60 con 0x7feedc02d020 2023-10-05T19:28:54.363 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:54.358+0000 7feeec516700 1 -- 172.21.15.167:0/24894888 <== osd.2 v2:172.21.15.167:6816/30037 3 ==== osd_op_reply(18 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7feed4012450 con 0x7feedc02d020 2023-10-05T19:28:54.363 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:54.358+0000 7feeec516700 1 -- 172.21.15.167:0/24894888 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:19 5.16 5:68000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x55c572bccc60 con 0x55c572bca340 2023-10-05T19:28:54.364 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:54.358+0000 7feeecd17700 1 -- 172.21.15.167:0/24894888 <== osd.3 v2:172.21.15.167:6808/30036 3 ==== osd_op_reply(19 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7feedc014070 con 0x55c572bca340 2023-10-05T19:28:54.364 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:54.358+0000 7feeecd17700 1 -- 172.21.15.167:0/24894888 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:20 5.e 5:70000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x55c572bccc60 con 0x55c572bca340 2023-10-05T19:28:54.365 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:54.358+0000 7feeecd17700 1 -- 172.21.15.167:0/24894888 <== osd.3 v2:172.21.15.167:6808/30036 4 ==== osd_op_reply(20 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7feedc014070 con 0x55c572bca340 2023-10-05T19:28:54.365 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:54.358+0000 7feeecd17700 1 -- 172.21.15.167:0/24894888 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:21 5.1e 5:78000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x55c572bccc60 con 0x7feed4007030 2023-10-05T19:28:54.366 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:54.362+0000 7feeed518700 1 -- 172.21.15.167:0/24894888 <== osd.0 v2:172.21.15.118:6800/27690 5 ==== osd_op_reply(21 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7feee400a5a0 con 0x7feed4007030 2023-10-05T19:28:54.366 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:54.362+0000 7feeed518700 1 -- 172.21.15.167:0/24894888 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:22 5.1 5:80000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x55c572bccc60 con 0x55c572bca340 2023-10-05T19:28:54.367 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:54.362+0000 7feeecd17700 1 -- 172.21.15.167:0/24894888 <== osd.3 v2:172.21.15.167:6808/30036 5 ==== osd_op_reply(22 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7feedc014070 con 0x55c572bca340 2023-10-05T19:28:54.367 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:54.362+0000 7feeecd17700 1 -- 172.21.15.167:0/24894888 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:23 5.11 5:88000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x55c572bccc60 con 0x55c572bca340 2023-10-05T19:28:54.368 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:54.362+0000 7feeecd17700 1 -- 172.21.15.167:0/24894888 <== osd.3 v2:172.21.15.167:6808/30036 6 ==== osd_op_reply(23 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7feedc014070 con 0x55c572bca340 2023-10-05T19:28:54.368 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:54.362+0000 7feeecd17700 1 -- 172.21.15.167:0/24894888 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:24 5.9 5:90000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x55c572bccc60 con 0x55c572b972f0 2023-10-05T19:28:54.368 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:54.362+0000 7feeed518700 1 -- 172.21.15.167:0/24894888 <== osd.1 v2:172.21.15.167:6800/30035 10 ==== osd_op_reply(24 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7feee400a5a0 con 0x55c572b972f0 2023-10-05T19:28:54.368 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:54.362+0000 7feeed518700 1 -- 172.21.15.167:0/24894888 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:25 5.19 5:98000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x55c572bccc60 con 0x55c572b972f0 2023-10-05T19:28:54.369 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:54.362+0000 7feeed518700 1 -- 172.21.15.167:0/24894888 <== osd.1 v2:172.21.15.167:6800/30035 11 ==== osd_op_reply(25 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7feee400a5a0 con 0x55c572b972f0 2023-10-05T19:28:54.369 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:54.362+0000 7feeed518700 1 -- 172.21.15.167:0/24894888 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:26 5.5 5:a0000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x55c572bccc60 con 0x7feed4007030 2023-10-05T19:28:54.370 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:54.366+0000 7feeed518700 1 -- 172.21.15.167:0/24894888 <== osd.0 v2:172.21.15.118:6800/27690 6 ==== osd_op_reply(26 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7feee400a5a0 con 0x7feed4007030 2023-10-05T19:28:54.370 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:54.366+0000 7feeed518700 1 -- 172.21.15.167:0/24894888 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:27 5.15 5:a8000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x55c572bccc60 con 0x55c572bca340 2023-10-05T19:28:54.371 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:54.366+0000 7feeecd17700 1 -- 172.21.15.167:0/24894888 <== osd.3 v2:172.21.15.167:6808/30036 7 ==== osd_op_reply(27 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7feedc014070 con 0x55c572bca340 2023-10-05T19:28:54.371 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:54.366+0000 7feeecd17700 1 -- 172.21.15.167:0/24894888 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:28 5.d 5:b0000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x55c572bccc60 con 0x7feedc02d020 2023-10-05T19:28:54.372 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:54.366+0000 7feeec516700 1 -- 172.21.15.167:0/24894888 <== osd.2 v2:172.21.15.167:6816/30037 4 ==== osd_op_reply(28 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7feed4012450 con 0x7feedc02d020 2023-10-05T19:28:54.372 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:54.366+0000 7feeec516700 1 -- 172.21.15.167:0/24894888 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:29 5.1d 5:b8000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x55c572bccc60 con 0x55c572b972f0 2023-10-05T19:28:54.372 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:54.366+0000 7feeed518700 1 -- 172.21.15.167:0/24894888 <== osd.1 v2:172.21.15.167:6800/30035 12 ==== osd_op_reply(29 [pgnls start_epoch 32 out=107b] v30'2 uv2 ondisk = 1) v8 ==== 144+0+107 (crc 0 0 0) 0x7feee400a5a0 con 0x55c572b972f0 2023-10-05T19:28:54.373 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:54.366+0000 7feeed518700 1 -- 172.21.15.167:0/24894888 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:30 5.3 5:c0000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x55c572bccc60 con 0x7feed4007030 2023-10-05T19:28:54.374 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:54.370+0000 7feeed518700 1 -- 172.21.15.167:0/24894888 <== osd.0 v2:172.21.15.118:6800/27690 7 ==== osd_op_reply(30 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7feee400a5a0 con 0x7feed4007030 2023-10-05T19:28:54.374 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:54.370+0000 7feeed518700 1 -- 172.21.15.167:0/24894888 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:31 5.13 5:c8000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x55c572bccc60 con 0x55c572bca340 2023-10-05T19:28:54.374 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:54.370+0000 7feeecd17700 1 -- 172.21.15.167:0/24894888 <== osd.3 v2:172.21.15.167:6808/30036 8 ==== osd_op_reply(31 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7feedc014070 con 0x55c572bca340 2023-10-05T19:28:54.374 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:54.370+0000 7feeecd17700 1 -- 172.21.15.167:0/24894888 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:32 5.b 5:d0000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x55c572bccc60 con 0x7feed4007030 2023-10-05T19:28:54.375 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:54.370+0000 7feeed518700 1 -- 172.21.15.167:0/24894888 <== osd.0 v2:172.21.15.118:6800/27690 8 ==== osd_op_reply(32 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7feee400a5a0 con 0x7feed4007030 2023-10-05T19:28:54.375 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:54.370+0000 7feeed518700 1 -- 172.21.15.167:0/24894888 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:33 5.1b 5:d8000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x55c572bccc60 con 0x7feedc02d020 2023-10-05T19:28:54.376 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:54.370+0000 7feeec516700 1 -- 172.21.15.167:0/24894888 <== osd.2 v2:172.21.15.167:6816/30037 5 ==== osd_op_reply(33 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7feed4012450 con 0x7feedc02d020 2023-10-05T19:28:54.376 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:54.370+0000 7feeec516700 1 -- 172.21.15.167:0/24894888 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:34 5.7 5:e0000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x55c572bccc60 con 0x55c572bca340 2023-10-05T19:28:54.377 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:54.370+0000 7feeecd17700 1 -- 172.21.15.167:0/24894888 <== osd.3 v2:172.21.15.167:6808/30036 9 ==== osd_op_reply(34 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7feedc014070 con 0x55c572bca340 2023-10-05T19:28:54.377 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:54.370+0000 7feeecd17700 1 -- 172.21.15.167:0/24894888 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:35 5.17 5:e8000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x55c572bccc60 con 0x7feed4007030 2023-10-05T19:28:54.378 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:54.374+0000 7feeed518700 1 -- 172.21.15.167:0/24894888 <== osd.0 v2:172.21.15.118:6800/27690 9 ==== osd_op_reply(35 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7feee400a5a0 con 0x7feed4007030 2023-10-05T19:28:54.378 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:54.374+0000 7feeed518700 1 -- 172.21.15.167:0/24894888 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:36 5.f 5:f0000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x55c572bccc60 con 0x55c572b972f0 2023-10-05T19:28:54.379 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:54.374+0000 7feeed518700 1 -- 172.21.15.167:0/24894888 <== osd.1 v2:172.21.15.167:6800/30035 13 ==== osd_op_reply(36 [pgnls start_epoch 32 out=115b] v23'2 uv2 ondisk = 1) v8 ==== 144+0+115 (crc 0 0 0) 0x7feee400a5a0 con 0x55c572b972f0 2023-10-05T19:28:54.379 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:54.374+0000 7feeed518700 1 -- 172.21.15.167:0/24894888 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:37 5.1f 5:f8000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x55c572bccc60 con 0x7feedc02d020 2023-10-05T19:28:54.380 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:54.374+0000 7feeec516700 1 -- 172.21.15.167:0/24894888 <== osd.2 v2:172.21.15.167:6816/30037 6 ==== osd_op_reply(37 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7feed4012450 con 0x7feedc02d020 2023-10-05T19:28:54.380 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:54.374+0000 7feeef1e2e40 20 RGWRados::pool_iterate: got zonegroup_info.4f1f719d-9648-417b-b10c-e5d939825419 2023-10-05T19:28:54.380 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:54.374+0000 7feeef1e2e40 20 RGWRados::pool_iterate: got zone_info.9a9df965-6612-45d4-a45f-6e3c889bacc2 2023-10-05T19:28:54.380 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:54.374+0000 7feeef1e2e40 20 RGWRados::pool_iterate: got zone_names.default 2023-10-05T19:28:54.380 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:54.374+0000 7feeef1e2e40 20 RGWRados::pool_iterate: got zonegroups_names.default 2023-10-05T19:28:54.380 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:54.374+0000 7feeef1e2e40 20 rados->read ofs=0 len=0 2023-10-05T19:28:54.380 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:54.374+0000 7feeef1e2e40 1 -- 172.21.15.167:0/24894888 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:38 5.12 5:49953fa1:::default.realm:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32) v8 -- 0x55c572bc9040 con 0x55c572b972f0 2023-10-05T19:28:54.381 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:54.374+0000 7feeed518700 1 -- 172.21.15.167:0/24894888 <== osd.1 v2:172.21.15.167:6800/30035 14 ==== osd_op_reply(38 default.realm [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 ==== 157+0+0 (crc 0 0 0) 0x7feee400a5a0 con 0x55c572b972f0 2023-10-05T19:28:54.381 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:54.374+0000 7feeef1e2e40 20 rados_obj.operate() r=-2 bl.length=0 2023-10-05T19:28:54.382 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:54.374+0000 7feeef1e2e40 20 rados->read ofs=0 len=0 2023-10-05T19:28:54.382 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:54.374+0000 7feeef1e2e40 1 -- 172.21.15.167:0/24894888 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:39 5.f 5:f4c53578:::zonegroups_names.default:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32) v8 -- 0x55c572bc8c40 con 0x55c572b972f0 2023-10-05T19:28:54.382 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:54.374+0000 7feeed518700 1 -- 172.21.15.167:0/24894888 <== osd.1 v2:172.21.15.167:6800/30035 15 ==== osd_op_reply(39 zonegroups_names.default [read 0~46 out=46b] v0'0 uv2 ondisk = 0) v8 ==== 168+0+46 (crc 0 0 0) 0x7feee400a5a0 con 0x55c572b972f0 2023-10-05T19:28:54.382 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:54.374+0000 7feeef1e2e40 20 rados_obj.operate() r=0 bl.length=46 2023-10-05T19:28:54.382 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:54.374+0000 7feeef1e2e40 20 rados->read ofs=0 len=0 2023-10-05T19:28:54.382 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:54.374+0000 7feeef1e2e40 1 -- 172.21.15.167:0/24894888 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:40 5.c 5:30d299a5:::zonegroup_info.4f1f719d-9648-417b-b10c-e5d939825419:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32) v8 -- 0x55c572bc8c10 con 0x7feed4007030 2023-10-05T19:28:54.383 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:54.378+0000 7feeed518700 1 -- 172.21.15.167:0/24894888 <== osd.0 v2:172.21.15.118:6800/27690 10 ==== osd_op_reply(40 zonegroup_info.4f1f719d-9648-417b-b10c-e5d939825419 [read 0~362 out=362b] v0'0 uv1 ondisk = 0) v8 ==== 195+0+362 (crc 0 0 0) 0x7feee400a5a0 con 0x7feed4007030 2023-10-05T19:28:54.383 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:54.378+0000 7feeef1e2e40 20 rados_obj.operate() r=0 bl.length=362 2023-10-05T19:28:54.383 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:54.378+0000 7feeef1e2e40 20 rados->read ofs=0 len=0 2023-10-05T19:28:54.383 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:54.378+0000 7feeef1e2e40 1 -- 172.21.15.167:0/24894888 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:41 5.13 5:cbbf6c60:::converted:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32) v8 -- 0x55c572bc99d0 con 0x55c572bca340 2023-10-05T19:28:54.383 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:54.378+0000 7feeecd17700 1 -- 172.21.15.167:0/24894888 <== osd.3 v2:172.21.15.167:6808/30036 10 ==== osd_op_reply(41 converted [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 ==== 153+0+0 (crc 0 0 0) 0x7feedc014070 con 0x55c572bca340 2023-10-05T19:28:54.384 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:54.378+0000 7feeef1e2e40 20 rados_obj.operate() r=-2 bl.length=0 2023-10-05T19:28:54.384 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:54.378+0000 7feeef1e2e40 20 rados->read ofs=0 len=0 2023-10-05T19:28:54.384 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:54.378+0000 7feeef1e2e40 1 -- 172.21.15.167:0/24894888 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:42 5.12 5:49953fa1:::default.realm:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32) v8 -- 0x55c572bcd0a0 con 0x55c572b972f0 2023-10-05T19:28:54.384 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:54.378+0000 7feeed518700 1 -- 172.21.15.167:0/24894888 <== osd.1 v2:172.21.15.167:6800/30035 16 ==== osd_op_reply(42 default.realm [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 ==== 157+0+0 (crc 0 0 0) 0x7feee400a5a0 con 0x55c572b972f0 2023-10-05T19:28:54.385 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:54.378+0000 7feeef1e2e40 20 rados_obj.operate() r=-2 bl.length=0 2023-10-05T19:28:54.385 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:54.378+0000 7feeef1e2e40 20 rados->read ofs=0 len=0 2023-10-05T19:28:54.385 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:54.378+0000 7feeef1e2e40 1 -- 172.21.15.167:0/24894888 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:43 5.f 5:f4c53578:::zonegroups_names.default:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32) v8 -- 0x55c572bcd0a0 con 0x55c572b972f0 2023-10-05T19:28:54.385 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:54.378+0000 7feeed518700 1 -- 172.21.15.167:0/24894888 <== osd.1 v2:172.21.15.167:6800/30035 17 ==== osd_op_reply(43 zonegroups_names.default [read 0~46 out=46b] v0'0 uv2 ondisk = 0) v8 ==== 168+0+46 (crc 0 0 0) 0x7feee400a5a0 con 0x55c572b972f0 2023-10-05T19:28:54.385 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:54.378+0000 7feeef1e2e40 20 rados_obj.operate() r=0 bl.length=46 2023-10-05T19:28:54.385 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:54.378+0000 7feeef1e2e40 1 -- 172.21.15.167:0/24894888 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:44 5.0 5:00000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x55c572bce0b0 con 0x55c572bca340 2023-10-05T19:28:54.386 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:54.382+0000 7feeecd17700 1 -- 172.21.15.167:0/24894888 <== osd.3 v2:172.21.15.167:6808/30036 11 ==== osd_op_reply(44 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7feedc014070 con 0x55c572bca340 2023-10-05T19:28:54.386 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:54.382+0000 7feeecd17700 1 -- 172.21.15.167:0/24894888 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:45 5.10 5:08000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x55c572bce0b0 con 0x7feedc02d020 2023-10-05T19:28:54.386 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:54.382+0000 7feeec516700 1 -- 172.21.15.167:0/24894888 <== osd.2 v2:172.21.15.167:6816/30037 7 ==== osd_op_reply(45 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7feed4012450 con 0x7feedc02d020 2023-10-05T19:28:54.387 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:54.382+0000 7feeec516700 1 -- 172.21.15.167:0/24894888 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:46 5.8 5:10000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x55c572bce0b0 con 0x7feed4007030 2023-10-05T19:28:54.387 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:54.382+0000 7feeed518700 1 -- 172.21.15.167:0/24894888 <== osd.0 v2:172.21.15.118:6800/27690 11 ==== osd_op_reply(46 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7feee400a5a0 con 0x7feed4007030 2023-10-05T19:28:54.387 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:54.382+0000 7feeed518700 1 -- 172.21.15.167:0/24894888 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:47 5.18 5:18000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x55c572bc9040 con 0x55c572b972f0 2023-10-05T19:28:54.388 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:54.382+0000 7feeed518700 1 -- 172.21.15.167:0/24894888 <== osd.1 v2:172.21.15.167:6800/30035 18 ==== osd_op_reply(47 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7feee400a5a0 con 0x55c572b972f0 2023-10-05T19:28:54.388 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:54.382+0000 7feeed518700 1 -- 172.21.15.167:0/24894888 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:48 5.4 5:20000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x55c572bc9040 con 0x55c572b972f0 2023-10-05T19:28:54.389 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:54.382+0000 7feeed518700 1 -- 172.21.15.167:0/24894888 <== osd.1 v2:172.21.15.167:6800/30035 19 ==== osd_op_reply(48 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7feee400a5a0 con 0x55c572b972f0 2023-10-05T19:28:54.389 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:54.382+0000 7feeed518700 1 -- 172.21.15.167:0/24894888 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:49 5.14 5:28000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x55c572bc9040 con 0x7feed4007030 2023-10-05T19:28:54.390 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:54.386+0000 7feeed518700 1 -- 172.21.15.167:0/24894888 <== osd.0 v2:172.21.15.118:6800/27690 12 ==== osd_op_reply(49 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7feee400a5a0 con 0x7feed4007030 2023-10-05T19:28:54.390 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:54.386+0000 7feeed518700 1 -- 172.21.15.167:0/24894888 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:50 5.c 5:30000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x55c572bc9040 con 0x7feed4007030 2023-10-05T19:28:54.391 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:54.386+0000 7feeed518700 1 -- 172.21.15.167:0/24894888 <== osd.0 v2:172.21.15.118:6800/27690 13 ==== osd_op_reply(50 [pgnls start_epoch 32 out=112b] v23'1 uv1 ondisk = 1) v8 ==== 144+0+112 (crc 0 0 0) 0x7feee400a5a0 con 0x7feed4007030 2023-10-05T19:28:54.391 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:54.386+0000 7feeed518700 1 -- 172.21.15.167:0/24894888 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:51 5.1c 5:38000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x55c572bc9040 con 0x7feedc02d020 2023-10-05T19:28:54.392 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:54.386+0000 7feeec516700 1 -- 172.21.15.167:0/24894888 <== osd.2 v2:172.21.15.167:6816/30037 8 ==== osd_op_reply(51 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7feed4012450 con 0x7feedc02d020 2023-10-05T19:28:54.392 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:54.386+0000 7feeec516700 1 -- 172.21.15.167:0/24894888 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:52 5.2 5:40000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x55c572bc9040 con 0x55c572bca340 2023-10-05T19:28:54.393 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:54.386+0000 7feeecd17700 1 -- 172.21.15.167:0/24894888 <== osd.3 v2:172.21.15.167:6808/30036 12 ==== osd_op_reply(52 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7feedc014070 con 0x55c572bca340 2023-10-05T19:28:54.393 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:54.386+0000 7feeecd17700 1 -- 172.21.15.167:0/24894888 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:53 5.12 5:48000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x55c572bc9040 con 0x55c572b972f0 2023-10-05T19:28:54.393 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:54.390+0000 7feeed518700 1 -- 172.21.15.167:0/24894888 <== osd.1 v2:172.21.15.167:6800/30035 20 ==== osd_op_reply(53 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7feee400a5a0 con 0x55c572b972f0 2023-10-05T19:28:54.394 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:54.390+0000 7feeed518700 1 -- 172.21.15.167:0/24894888 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:54 5.a 5:50000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x55c572bc9040 con 0x7feed4007030 2023-10-05T19:28:54.395 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:54.390+0000 7feeed518700 1 -- 172.21.15.167:0/24894888 <== osd.0 v2:172.21.15.118:6800/27690 14 ==== osd_op_reply(54 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7feee400a5a0 con 0x7feed4007030 2023-10-05T19:28:54.395 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:54.390+0000 7feeed518700 1 -- 172.21.15.167:0/24894888 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:55 5.1a 5:58000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x55c572bc9040 con 0x55c572b972f0 2023-10-05T19:28:54.395 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:54.390+0000 7feeed518700 1 -- 172.21.15.167:0/24894888 <== osd.1 v2:172.21.15.167:6800/30035 21 ==== osd_op_reply(55 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7feee400a5a0 con 0x55c572b972f0 2023-10-05T19:28:54.395 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:54.390+0000 7feeed518700 1 -- 172.21.15.167:0/24894888 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:56 5.6 5:60000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x55c572bc9040 con 0x7feedc02d020 2023-10-05T19:28:54.396 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:54.390+0000 7feeec516700 1 -- 172.21.15.167:0/24894888 <== osd.2 v2:172.21.15.167:6816/30037 9 ==== osd_op_reply(56 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7feed4012450 con 0x7feedc02d020 2023-10-05T19:28:54.396 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:54.390+0000 7feeec516700 1 -- 172.21.15.167:0/24894888 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:57 5.16 5:68000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x55c572bc9040 con 0x55c572bca340 2023-10-05T19:28:54.397 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:54.390+0000 7feeecd17700 1 -- 172.21.15.167:0/24894888 <== osd.3 v2:172.21.15.167:6808/30036 13 ==== osd_op_reply(57 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7feedc014070 con 0x55c572bca340 2023-10-05T19:28:54.397 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:54.390+0000 7feeecd17700 1 -- 172.21.15.167:0/24894888 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:58 5.e 5:70000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x55c572bc9040 con 0x55c572bca340 2023-10-05T19:28:54.398 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:54.394+0000 7feeecd17700 1 -- 172.21.15.167:0/24894888 <== osd.3 v2:172.21.15.167:6808/30036 14 ==== osd_op_reply(58 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7feedc04d040 con 0x55c572bca340 2023-10-05T19:28:54.398 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:54.394+0000 7feeecd17700 1 -- 172.21.15.167:0/24894888 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:59 5.1e 5:78000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x55c572bc9040 con 0x7feed4007030 2023-10-05T19:28:54.399 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:54.394+0000 7feeed518700 1 -- 172.21.15.167:0/24894888 <== osd.0 v2:172.21.15.118:6800/27690 15 ==== osd_op_reply(59 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7feee403f040 con 0x7feed4007030 2023-10-05T19:28:54.399 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:54.394+0000 7feeed518700 1 -- 172.21.15.167:0/24894888 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:60 5.1 5:80000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x55c572bc9040 con 0x55c572bca340 2023-10-05T19:28:54.399 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:54.394+0000 7feeecd17700 1 -- 172.21.15.167:0/24894888 <== osd.3 v2:172.21.15.167:6808/30036 15 ==== osd_op_reply(60 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7feedc04d040 con 0x55c572bca340 2023-10-05T19:28:54.399 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:54.394+0000 7feeecd17700 1 -- 172.21.15.167:0/24894888 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:61 5.11 5:88000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x55c572bc9040 con 0x55c572bca340 2023-10-05T19:28:54.400 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:54.394+0000 7feeecd17700 1 -- 172.21.15.167:0/24894888 <== osd.3 v2:172.21.15.167:6808/30036 16 ==== osd_op_reply(61 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7feedc04d040 con 0x55c572bca340 2023-10-05T19:28:54.400 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:54.394+0000 7feeecd17700 1 -- 172.21.15.167:0/24894888 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:62 5.9 5:90000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x55c572bc9040 con 0x55c572b972f0 2023-10-05T19:28:54.401 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:54.394+0000 7feeed518700 1 -- 172.21.15.167:0/24894888 <== osd.1 v2:172.21.15.167:6800/30035 22 ==== osd_op_reply(62 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7feee403f040 con 0x55c572b972f0 2023-10-05T19:28:54.401 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:54.394+0000 7feeed518700 1 -- 172.21.15.167:0/24894888 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:63 5.19 5:98000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x55c572bc9040 con 0x55c572b972f0 2023-10-05T19:28:54.401 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:54.398+0000 7feeed518700 1 -- 172.21.15.167:0/24894888 <== osd.1 v2:172.21.15.167:6800/30035 23 ==== osd_op_reply(63 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7feee403f040 con 0x55c572b972f0 2023-10-05T19:28:54.402 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:54.398+0000 7feeed518700 1 -- 172.21.15.167:0/24894888 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:64 5.5 5:a0000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x55c572bc9040 con 0x7feed4007030 2023-10-05T19:28:54.402 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:54.398+0000 7feeed518700 1 -- 172.21.15.167:0/24894888 <== osd.0 v2:172.21.15.118:6800/27690 16 ==== osd_op_reply(64 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7feee403f040 con 0x7feed4007030 2023-10-05T19:28:54.402 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:54.398+0000 7feeed518700 1 -- 172.21.15.167:0/24894888 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:65 5.15 5:a8000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x55c572bc9040 con 0x55c572bca340 2023-10-05T19:28:54.403 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:54.398+0000 7feeecd17700 1 -- 172.21.15.167:0/24894888 <== osd.3 v2:172.21.15.167:6808/30036 17 ==== osd_op_reply(65 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7feedc04d040 con 0x55c572bca340 2023-10-05T19:28:54.403 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:54.398+0000 7feeecd17700 1 -- 172.21.15.167:0/24894888 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:66 5.d 5:b0000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x55c572bc9040 con 0x7feedc02d020 2023-10-05T19:28:54.404 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:54.398+0000 7feeec516700 1 -- 172.21.15.167:0/24894888 <== osd.2 v2:172.21.15.167:6816/30037 10 ==== osd_op_reply(66 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7feed4012450 con 0x7feedc02d020 2023-10-05T19:28:54.404 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:54.398+0000 7feeec516700 1 -- 172.21.15.167:0/24894888 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:67 5.1d 5:b8000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x55c572bc9040 con 0x55c572b972f0 2023-10-05T19:28:54.405 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:54.398+0000 7feeed518700 1 -- 172.21.15.167:0/24894888 <== osd.1 v2:172.21.15.167:6800/30035 24 ==== osd_op_reply(67 [pgnls start_epoch 32 out=107b] v30'2 uv2 ondisk = 1) v8 ==== 144+0+107 (crc 0 0 0) 0x7feee403f040 con 0x55c572b972f0 2023-10-05T19:28:54.405 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:54.398+0000 7feeed518700 1 -- 172.21.15.167:0/24894888 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:68 5.3 5:c0000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x55c572bc9040 con 0x7feed4007030 2023-10-05T19:28:54.406 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:54.402+0000 7feeed518700 1 -- 172.21.15.167:0/24894888 <== osd.0 v2:172.21.15.118:6800/27690 17 ==== osd_op_reply(68 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7feee403f040 con 0x7feed4007030 2023-10-05T19:28:54.406 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:54.402+0000 7feeed518700 1 -- 172.21.15.167:0/24894888 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:69 5.13 5:c8000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x55c572bc9040 con 0x55c572bca340 2023-10-05T19:28:54.406 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:54.402+0000 7feeecd17700 1 -- 172.21.15.167:0/24894888 <== osd.3 v2:172.21.15.167:6808/30036 18 ==== osd_op_reply(69 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7feedc04d040 con 0x55c572bca340 2023-10-05T19:28:54.406 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:54.402+0000 7feeecd17700 1 -- 172.21.15.167:0/24894888 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:70 5.b 5:d0000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x55c572bc9040 con 0x7feed4007030 2023-10-05T19:28:54.408 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:54.402+0000 7feeed518700 1 -- 172.21.15.167:0/24894888 <== osd.0 v2:172.21.15.118:6800/27690 18 ==== osd_op_reply(70 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7feee403f040 con 0x7feed4007030 2023-10-05T19:28:54.408 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:54.402+0000 7feeed518700 1 -- 172.21.15.167:0/24894888 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:71 5.1b 5:d8000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x55c572bc9040 con 0x7feedc02d020 2023-10-05T19:28:54.408 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:54.402+0000 7feeec516700 1 -- 172.21.15.167:0/24894888 <== osd.2 v2:172.21.15.167:6816/30037 11 ==== osd_op_reply(71 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7feed4012450 con 0x7feedc02d020 2023-10-05T19:28:54.408 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:54.402+0000 7feeec516700 1 -- 172.21.15.167:0/24894888 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:72 5.7 5:e0000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x55c572bc9040 con 0x55c572bca340 2023-10-05T19:28:54.409 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:54.402+0000 7feeecd17700 1 -- 172.21.15.167:0/24894888 <== osd.3 v2:172.21.15.167:6808/30036 19 ==== osd_op_reply(72 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7feedc04d040 con 0x55c572bca340 2023-10-05T19:28:54.409 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:54.402+0000 7feeecd17700 1 -- 172.21.15.167:0/24894888 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:73 5.17 5:e8000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x55c572bc9040 con 0x7feed4007030 2023-10-05T19:28:54.410 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:54.406+0000 7feeed518700 1 -- 172.21.15.167:0/24894888 <== osd.0 v2:172.21.15.118:6800/27690 19 ==== osd_op_reply(73 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7feee403f040 con 0x7feed4007030 2023-10-05T19:28:54.410 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:54.406+0000 7feeed518700 1 -- 172.21.15.167:0/24894888 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:74 5.f 5:f0000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x55c572bc9040 con 0x55c572b972f0 2023-10-05T19:28:54.411 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:54.406+0000 7feeed518700 1 -- 172.21.15.167:0/24894888 <== osd.1 v2:172.21.15.167:6800/30035 25 ==== osd_op_reply(74 [pgnls start_epoch 32 out=115b] v23'2 uv2 ondisk = 1) v8 ==== 144+0+115 (crc 0 0 0) 0x7feee403f040 con 0x55c572b972f0 2023-10-05T19:28:54.411 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:54.406+0000 7feeed518700 1 -- 172.21.15.167:0/24894888 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:75 5.1f 5:f8000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x55c572bc9040 con 0x7feedc02d020 2023-10-05T19:28:54.412 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:54.406+0000 7feeec516700 1 -- 172.21.15.167:0/24894888 <== osd.2 v2:172.21.15.167:6816/30037 12 ==== osd_op_reply(75 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7feed4012450 con 0x7feedc02d020 2023-10-05T19:28:54.412 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:54.406+0000 7feeef1e2e40 20 RGWRados::pool_iterate: got zonegroup_info.4f1f719d-9648-417b-b10c-e5d939825419 2023-10-05T19:28:54.412 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:54.406+0000 7feeef1e2e40 20 RGWRados::pool_iterate: got zone_info.9a9df965-6612-45d4-a45f-6e3c889bacc2 2023-10-05T19:28:54.412 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:54.406+0000 7feeef1e2e40 20 RGWRados::pool_iterate: got zone_names.default 2023-10-05T19:28:54.412 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:54.406+0000 7feeef1e2e40 20 RGWRados::pool_iterate: got zonegroups_names.default 2023-10-05T19:28:54.412 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:54.406+0000 7feeef1e2e40 20 rados->read ofs=0 len=0 2023-10-05T19:28:54.413 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:54.406+0000 7feeef1e2e40 1 -- 172.21.15.167:0/24894888 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:76 5.f 5:f43b6ece:::zone_names.default:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32) v8 -- 0x55c572bcd0a0 con 0x55c572b972f0 2023-10-05T19:28:54.414 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:54.406+0000 7feeed518700 1 -- 172.21.15.167:0/24894888 <== osd.1 v2:172.21.15.167:6800/30035 26 ==== osd_op_reply(76 zone_names.default [read 0~46 out=46b] v0'0 uv1 ondisk = 0) v8 ==== 162+0+46 (crc 0 0 0) 0x7feee403f040 con 0x55c572b972f0 2023-10-05T19:28:54.414 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:54.406+0000 7feeef1e2e40 20 rados_obj.operate() r=0 bl.length=46 2023-10-05T19:28:54.414 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:54.406+0000 7feeef1e2e40 20 rados->read ofs=0 len=0 2023-10-05T19:28:54.414 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:54.406+0000 7feeef1e2e40 1 -- 172.21.15.167:0/24894888 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:77 5.1d 5:b8374eb4:::zone_info.9a9df965-6612-45d4-a45f-6e3c889bacc2:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32) v8 -- 0x55c572bcd450 con 0x55c572b972f0 2023-10-05T19:28:54.414 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:54.406+0000 7feeed518700 1 -- 172.21.15.167:0/24894888 <== osd.1 v2:172.21.15.167:6800/30035 27 ==== osd_op_reply(77 zone_info.9a9df965-6612-45d4-a45f-6e3c889bacc2 [read 0~874 out=874b] v0'0 uv2 ondisk = 0) v8 ==== 190+0+874 (crc 0 0 0) 0x7feee403f040 con 0x55c572b972f0 2023-10-05T19:28:54.414 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:54.406+0000 7feeef1e2e40 20 rados_obj.operate() r=0 bl.length=874 2023-10-05T19:28:54.414 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:54.406+0000 7feeef1e2e40 20 rados->read ofs=0 len=0 2023-10-05T19:28:54.415 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:54.406+0000 7feeef1e2e40 1 -- 172.21.15.167:0/24894888 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:78 5.f 5:f4c53578:::zonegroups_names.default:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32) v8 -- 0x55c572bcd940 con 0x55c572b972f0 2023-10-05T19:28:54.415 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:54.410+0000 7feeed518700 1 -- 172.21.15.167:0/24894888 <== osd.1 v2:172.21.15.167:6800/30035 28 ==== osd_op_reply(78 zonegroups_names.default [read 0~46 out=46b] v0'0 uv2 ondisk = 0) v8 ==== 168+0+46 (crc 0 0 0) 0x7feee403f040 con 0x55c572b972f0 2023-10-05T19:28:54.415 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:54.410+0000 7feeef1e2e40 20 rados_obj.operate() r=0 bl.length=46 2023-10-05T19:28:54.415 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:54.410+0000 7feeef1e2e40 20 rados->read ofs=0 len=0 2023-10-05T19:28:54.415 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:54.410+0000 7feeef1e2e40 1 -- 172.21.15.167:0/24894888 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:79 5.c 5:30d299a5:::zonegroup_info.4f1f719d-9648-417b-b10c-e5d939825419:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32) v8 -- 0x55c572bcd940 con 0x7feed4007030 2023-10-05T19:28:54.416 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:54.410+0000 7feeed518700 1 -- 172.21.15.167:0/24894888 <== osd.0 v2:172.21.15.118:6800/27690 20 ==== osd_op_reply(79 zonegroup_info.4f1f719d-9648-417b-b10c-e5d939825419 [read 0~362 out=362b] v0'0 uv1 ondisk = 0) v8 ==== 195+0+362 (crc 0 0 0) 0x7feee4052040 con 0x7feed4007030 2023-10-05T19:28:54.417 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:54.410+0000 7feeef1e2e40 20 rados_obj.operate() r=0 bl.length=362 2023-10-05T19:28:54.417 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:54.410+0000 7feeef1e2e40 20 rados->read ofs=0 len=0 2023-10-05T19:28:54.417 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:54.410+0000 7feeef1e2e40 1 -- 172.21.15.167:0/24894888 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:80 5.3 5:c0ad8cbd:::region_map:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32) v8 -- 0x55c572bcd940 con 0x7feed4007030 2023-10-05T19:28:54.417 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:54.410+0000 7feeed518700 1 -- 172.21.15.167:0/24894888 <== osd.0 v2:172.21.15.118:6800/27690 21 ==== osd_op_reply(80 region_map [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 ==== 154+0+0 (crc 0 0 0) 0x7feee4052040 con 0x7feed4007030 2023-10-05T19:28:54.417 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:54.410+0000 7feeef1e2e40 20 rados_obj.operate() r=-2 bl.length=0 2023-10-05T19:28:54.417 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:54.410+0000 7feeef1e2e40 20 zone default found 2023-10-05T19:28:54.417 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:54.410+0000 7feeef1e2e40 4 Realm: () 2023-10-05T19:28:54.417 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:54.410+0000 7feeef1e2e40 4 ZoneGroup: default (4f1f719d-9648-417b-b10c-e5d939825419) 2023-10-05T19:28:54.417 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:54.410+0000 7feeef1e2e40 4 Zone: default (9a9df965-6612-45d4-a45f-6e3c889bacc2) 2023-10-05T19:28:54.417 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:54.410+0000 7feeef1e2e40 10 cannot find current period zonegroup using local zonegroup configuration 2023-10-05T19:28:54.418 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:54.410+0000 7feeef1e2e40 20 zonegroup default 2023-10-05T19:28:54.418 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:54.410+0000 7feeef1e2e40 20 rados->read ofs=0 len=0 2023-10-05T19:28:54.418 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:54.410+0000 7feeef1e2e40 1 -- 172.21.15.167:0/24894888 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:81 5.3 5:c52100b6:::period_config.default:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32) v8 -- 0x55c572bcd940 con 0x7feed4007030 2023-10-05T19:28:54.418 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:54.410+0000 7feeed518700 1 -- 172.21.15.167:0/24894888 <== osd.0 v2:172.21.15.118:6800/27690 22 ==== osd_op_reply(81 period_config.default [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 ==== 165+0+0 (crc 0 0 0) 0x7feee4052040 con 0x7feed4007030 2023-10-05T19:28:54.418 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:54.410+0000 7feeef1e2e40 20 rados_obj.operate() r=-2 bl.length=0 2023-10-05T19:28:54.418 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:54.410+0000 7feeef1e2e40 20 rados->read ofs=0 len=0 2023-10-05T19:28:54.418 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:54.410+0000 7feeef1e2e40 1 -- 172.21.15.167:0/24894888 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:82 6.2 6:4347d321:::bucket.sync-source-hints.:head [call version.read in=11b,read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32) v8 -- 0x55c572bcf2c0 con 0x55c572bca340 2023-10-05T19:28:54.419 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:54.410+0000 7feeecd17700 1 -- 172.21.15.167:0/24894888 <== osd.3 v2:172.21.15.167:6808/30036 20 ==== osd_op_reply(82 bucket.sync-source-hints. [call,read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 ==== 211+0+0 (crc 0 0 0) 0x7feedc04d040 con 0x55c572bca340 2023-10-05T19:28:54.419 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:54.410+0000 7feeef1e2e40 20 rados_obj.operate() r=-2 bl.length=0 2023-10-05T19:28:54.419 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:54.410+0000 7feeef1e2e40 20 rados->read ofs=0 len=0 2023-10-05T19:28:54.419 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:54.410+0000 7feeef1e2e40 1 -- 172.21.15.167:0/24894888 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:83 6.b 6:d467b91b:::bucket.sync-target-hints.:head [call version.read in=11b,read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32) v8 -- 0x55c572bcf670 con 0x55c572bca340 2023-10-05T19:28:54.419 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:54.414+0000 7feeecd17700 1 -- 172.21.15.167:0/24894888 <== osd.3 v2:172.21.15.167:6808/30036 21 ==== osd_op_reply(83 bucket.sync-target-hints. [call,read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 ==== 211+0+0 (crc 0 0 0) 0x7feedc04d040 con 0x55c572bca340 2023-10-05T19:28:54.419 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:54.414+0000 7feeef1e2e40 20 rados_obj.operate() r=-2 bl.length=0 2023-10-05T19:28:54.420 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:54.414+0000 7feeef1e2e40 20 started sync module instance, tier type = 2023-10-05T19:28:54.420 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:54.414+0000 7feeef1e2e40 20 started zone id=9a9df965-6612-45d4-a45f-6e3c889bacc2 (name=default) with tier type = 2023-10-05T19:28:54.420 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:54.414+0000 7feeef1e2e40 1 -- 172.21.15.167:0/24894888 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:84 7.1f 7:f95f44c2:::notify.0:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e32) v8 -- 0x55c572bce980 con 0x7feedc02d020 2023-10-05T19:28:54.421 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:54.414+0000 7feeec516700 1 -- 172.21.15.167:0/24894888 <== osd.2 v2:172.21.15.167:6816/30037 13 ==== osd_op_reply(84 notify.0 [create] v32'189 uv189 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7feed4012450 con 0x7feedc02d020 2023-10-05T19:28:54.422 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:54.414+0000 7feeef1e2e40 1 -- 172.21.15.167:0/24894888 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:85 7.1f 7:f95f44c2:::notify.0:head [watch watch cookie 94306521902464] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e32) v8 -- 0x55c572bce5b0 con 0x7feedc02d020 2023-10-05T19:28:54.422 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:54.418+0000 7feeef1e2e40 1 -- 172.21.15.167:0/24894888 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:86 7.0 7:05bf5b68:::notify.1:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e32) v8 -- 0x55c572bd0460 con 0x55c572b972f0 2023-10-05T19:28:54.425 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:54.418+0000 7feeed518700 1 -- 172.21.15.167:0/24894888 <== osd.1 v2:172.21.15.167:6800/30035 29 ==== osd_op_reply(86 notify.1 [create] v32'189 uv189 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7feee4052040 con 0x55c572b972f0 2023-10-05T19:28:54.425 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:54.418+0000 7feeec516700 1 -- 172.21.15.167:0/24894888 <== osd.2 v2:172.21.15.167:6816/30037 14 ==== osd_op_reply(85 notify.0 [watch watch cookie 94306521902464] v32'190 uv189 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7feed4012450 con 0x7feedc02d020 2023-10-05T19:28:54.425 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:54.418+0000 7feeef1e2e40 1 -- 172.21.15.167:0/24894888 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:87 7.0 7:05bf5b68:::notify.1:head [watch watch cookie 94306521907344] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e32) v8 -- 0x55c572bd0d50 con 0x55c572b972f0 2023-10-05T19:28:54.425 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:54.418+0000 7feeef1e2e40 1 -- 172.21.15.167:0/24894888 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:88 7.15 7:a93a5511:::notify.2:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e32) v8 -- 0x55c572bd1100 con 0x55c572bca340 2023-10-05T19:28:54.428 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:54.422+0000 7feeed518700 1 -- 172.21.15.167:0/24894888 <== osd.1 v2:172.21.15.167:6800/30035 30 ==== osd_op_reply(87 notify.1 [watch watch cookie 94306521907344] v32'190 uv189 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7feee4052040 con 0x55c572b972f0 2023-10-05T19:28:54.429 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:54.422+0000 7feeecd17700 1 -- 172.21.15.167:0/24894888 <== osd.3 v2:172.21.15.167:6808/30036 22 ==== osd_op_reply(88 notify.2 [create] v32'189 uv189 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7feedc04d040 con 0x55c572bca340 2023-10-05T19:28:54.429 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:54.422+0000 7feeef1e2e40 1 -- 172.21.15.167:0/24894888 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:89 7.15 7:a93a5511:::notify.2:head [watch watch cookie 94306521909344] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e32) v8 -- 0x55c572bcfb40 con 0x55c572bca340 2023-10-05T19:28:54.429 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:54.422+0000 7feeef1e2e40 1 -- 172.21.15.167:0/24894888 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:90 7.e 7:7759931f:::notify.3:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e32) v8 -- 0x55c572bd1e90 con 0x7feedc02d020 2023-10-05T19:28:54.432 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:54.426+0000 7feeec516700 1 -- 172.21.15.167:0/24894888 <== osd.2 v2:172.21.15.167:6816/30037 15 ==== osd_op_reply(90 notify.3 [create] v32'189 uv189 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7feed4012450 con 0x7feedc02d020 2023-10-05T19:28:54.432 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:54.426+0000 7feeef1e2e40 1 -- 172.21.15.167:0/24894888 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:91 7.e 7:7759931f:::notify.3:head [watch watch cookie 94306521913760] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e32) v8 -- 0x55c572bd27b0 con 0x7feedc02d020 2023-10-05T19:28:54.432 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:54.426+0000 7feeef1e2e40 1 -- 172.21.15.167:0/24894888 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:92 7.d 7:b4812045:::notify.4:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e32) v8 -- 0x55c572bd32e0 con 0x55c572b972f0 2023-10-05T19:28:54.432 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:54.426+0000 7feeecd17700 1 -- 172.21.15.167:0/24894888 <== osd.3 v2:172.21.15.167:6808/30036 23 ==== osd_op_reply(89 notify.2 [watch watch cookie 94306521909344] v32'190 uv189 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7feedc04d040 con 0x55c572bca340 2023-10-05T19:28:54.434 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:54.430+0000 7feeec516700 1 -- 172.21.15.167:0/24894888 <== osd.2 v2:172.21.15.167:6816/30037 16 ==== osd_op_reply(91 notify.3 [watch watch cookie 94306521913760] v32'190 uv189 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7feed4012450 con 0x7feedc02d020 2023-10-05T19:28:54.434 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:54.430+0000 7feeed518700 1 -- 172.21.15.167:0/24894888 <== osd.1 v2:172.21.15.167:6800/30035 31 ==== osd_op_reply(92 notify.4 [create] v32'189 uv189 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7feee4052040 con 0x55c572b972f0 2023-10-05T19:28:54.434 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:54.430+0000 7feeef1e2e40 1 -- 172.21.15.167:0/24894888 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:93 7.d 7:b4812045:::notify.4:head [watch watch cookie 94306521912496] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e32) v8 -- 0x55c572bd2240 con 0x55c572b972f0 2023-10-05T19:28:54.435 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:54.430+0000 7feeef1e2e40 1 -- 172.21.15.167:0/24894888 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:94 7.3 7:c609908c:::notify.5:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e32) v8 -- 0x55c572bd3bd0 con 0x7feed4007030 2023-10-05T19:28:54.436 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:54.430+0000 7feeed518700 1 -- 172.21.15.167:0/24894888 <== osd.1 v2:172.21.15.167:6800/30035 32 ==== osd_op_reply(93 notify.4 [watch watch cookie 94306521912496] v32'190 uv189 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7feee4052040 con 0x55c572b972f0 2023-10-05T19:28:54.437 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:54.430+0000 7feeed518700 1 -- 172.21.15.167:0/24894888 <== osd.0 v2:172.21.15.118:6800/27690 23 ==== osd_op_reply(94 notify.5 [create] v32'189 uv189 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7feee4052040 con 0x7feed4007030 2023-10-05T19:28:54.437 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:54.430+0000 7feeef1e2e40 1 -- 172.21.15.167:0/24894888 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:95 7.3 7:c609908c:::notify.5:head [watch watch cookie 94306521918896] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e32) v8 -- 0x55c572bd3b80 con 0x7feed4007030 2023-10-05T19:28:54.437 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:54.430+0000 7feeef1e2e40 1 -- 172.21.15.167:0/24894888 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:96 7.14 7:2b04a3e9:::notify.6:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e32) v8 -- 0x55c572bd4460 con 0x7feed4007030 2023-10-05T19:28:54.440 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:54.434+0000 7feeed518700 1 -- 172.21.15.167:0/24894888 <== osd.0 v2:172.21.15.118:6800/27690 24 ==== osd_op_reply(95 notify.5 [watch watch cookie 94306521918896] v32'190 uv189 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7feee4052040 con 0x7feed4007030 2023-10-05T19:28:54.440 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:54.434+0000 7feeed518700 1 -- 172.21.15.167:0/24894888 <== osd.0 v2:172.21.15.118:6800/27690 25 ==== osd_op_reply(96 notify.6 [create] v32'189 uv189 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7feee4052040 con 0x7feed4007030 2023-10-05T19:28:54.440 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:54.434+0000 7feeef1e2e40 1 -- 172.21.15.167:0/24894888 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:97 7.14 7:2b04a3e9:::notify.6:head [watch watch cookie 94306521921168] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e32) v8 -- 0x55c572bd3fd0 con 0x7feed4007030 2023-10-05T19:28:54.440 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:54.434+0000 7feeef1e2e40 1 -- 172.21.15.167:0/24894888 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:98 7.9 7:93e5b521:::notify.7:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e32) v8 -- 0x55c572bd4ec0 con 0x7feed4007030 2023-10-05T19:28:54.442 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:54.438+0000 7feeed518700 1 -- 172.21.15.167:0/24894888 <== osd.0 v2:172.21.15.118:6800/27690 26 ==== osd_op_reply(97 notify.6 [watch watch cookie 94306521921168] v32'190 uv189 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7feee4052040 con 0x7feed4007030 2023-10-05T19:28:54.443 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:54.438+0000 7feeed518700 1 -- 172.21.15.167:0/24894888 <== osd.0 v2:172.21.15.118:6800/27690 27 ==== osd_op_reply(98 notify.7 [create] v32'189 uv189 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7feee4052040 con 0x7feed4007030 2023-10-05T19:28:54.443 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:54.438+0000 7feeef1e2e40 1 -- 172.21.15.167:0/24894888 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:99 7.9 7:93e5b521:::notify.7:head [watch watch cookie 94306521922832] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e32) v8 -- 0x55c572bd4ec0 con 0x7feed4007030 2023-10-05T19:28:54.443 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:54.438+0000 7feeef1e2e40 20 add_watcher() i=0 2023-10-05T19:28:54.443 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:54.438+0000 7feeef1e2e40 20 add_watcher() i=1 2023-10-05T19:28:54.443 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:54.438+0000 7feeef1e2e40 20 add_watcher() i=2 2023-10-05T19:28:54.443 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:54.438+0000 7feeef1e2e40 20 add_watcher() i=3 2023-10-05T19:28:54.443 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:54.438+0000 7feeef1e2e40 20 add_watcher() i=4 2023-10-05T19:28:54.443 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:54.438+0000 7feeef1e2e40 20 add_watcher() i=5 2023-10-05T19:28:54.444 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:54.438+0000 7feeef1e2e40 20 add_watcher() i=6 2023-10-05T19:28:54.445 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:54.438+0000 7feeed518700 1 -- 172.21.15.167:0/24894888 <== osd.0 v2:172.21.15.118:6800/27690 28 ==== osd_op_reply(99 notify.7 [watch watch cookie 94306521922832] v32'190 uv189 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7feee4052040 con 0x7feed4007030 2023-10-05T19:28:54.445 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:54.438+0000 7feeef1e2e40 20 add_watcher() i=7 2023-10-05T19:28:54.445 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:54.438+0000 7feeef1e2e40 2 all 8 watchers are set, enabling cache 2023-10-05T19:28:54.445 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:54.438+0000 7feeef1e2e40 1 -- 172.21.15.167:0/24894888 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:100 6.4 6:22d26bf9:::data_loggenerations_metadata:head [call version.check_conds in=74b,call version.read in=11b,read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32) v8 -- 0x55c572bd8090 con 0x55c572b972f0 2023-10-05T19:28:54.446 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:54.442+0000 7feeed518700 1 -- 172.21.15.167:0/24894888 <== osd.1 v2:172.21.15.167:6800/30035 33 ==== osd_op_reply(100 data_loggenerations_metadata [call,call out=48b,read 0~28 out=28b] v0'0 uv1 ondisk = 0) v8 ==== 256+0+76 (crc 0 0 0) 0x7feee4052040 con 0x55c572b972f0 2023-10-05T19:28:54.446 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:54.442+0000 7feeef1e2e40 1 -- 172.21.15.167:0/24894888 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:101 6.4 6:22d26bf9:::data_loggenerations_metadata:head [watch watch cookie 94306521926464] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e32) v8 -- 0x55c572bd5480 con 0x55c572b972f0 2023-10-05T19:28:54.449 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:54.442+0000 7feeed518700 1 -- 172.21.15.167:0/24894888 <== osd.1 v2:172.21.15.167:6800/30035 34 ==== osd_op_reply(101 data_loggenerations_metadata [watch watch cookie 94306521926464] v32'174 uv1 ondisk = 0) v8 ==== 172+0+0 (crc 0 0 0) 0x7feee4052040 con 0x55c572b972f0 2023-10-05T19:28:54.449 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:54.442+0000 7feebe7e4700 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2023-10-05T19:28:54.449 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:54.442+0000 7feeef1e2e40 20 check_secure_mon_conn(): auth registy supported: methods=[2] modes=[2,1] 2023-10-05T19:28:54.450 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:54.442+0000 7feeef1e2e40 20 check_secure_mon_conn(): mode 1 is insecure 2023-10-05T19:28:54.450 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:54.442+0000 7feeef1e2e40 1 Processor -- start 2023-10-05T19:28:54.450 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:54.446+0000 7feeef1e2e40 1 -- start start 2023-10-05T19:28:54.450 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:54.446+0000 7feeef1e2e40 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55c572be2510 0x55c572be28e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-05T19:28:54.451 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:54.446+0000 7feeef1e2e40 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c572be2e20 con 0x55c572be2510 2023-10-05T19:28:54.451 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:54.446+0000 7feeec516700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55c572be2510 0x55c572be28e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-05T19:28:54.451 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:54.446+0000 7feeec516700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55c572be2510 0x55c572be28e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.118:3300/0 says I am v2:172.21.15.167:53138/0 (socket says 172.21.15.167:53138) 2023-10-05T19:28:54.451 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:54.446+0000 7feeec516700 1 -- 172.21.15.167:0/4243588109 learned_addr learned my addr 172.21.15.167:0/4243588109 (peer_addr_for_me v2:172.21.15.167:0/0) 2023-10-05T19:28:54.452 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:54.446+0000 7feeec516700 1 -- 172.21.15.167:0/4243588109 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55c572be2f60 con 0x55c572be2510 2023-10-05T19:28:54.452 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:54.446+0000 7feeec516700 1 --2- 172.21.15.167:0/4243588109 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55c572be2510 0x55c572be28e0 secure :-1 s=READY pgs=516 cs=0 l=1 rev1=1 crypto rx=0x55c572b06c60 tx=0x7feed4012000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=674deb2c3bdf30ed server_cookie=0 in_seq=0 out_seq=0 2023-10-05T19:28:54.452 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:54.446+0000 7feebdfe3700 1 -- 172.21.15.167:0/4243588109 <== mon.0 v2:172.21.15.118:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7feed402a5c0 con 0x55c572be2510 2023-10-05T19:28:54.452 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:54.446+0000 7feebdfe3700 1 -- 172.21.15.167:0/4243588109 <== mon.0 v2:172.21.15.118:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7feed402a720 con 0x55c572be2510 2023-10-05T19:28:54.452 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:54.446+0000 7feebdfe3700 1 -- 172.21.15.167:0/4243588109 <== mon.0 v2:172.21.15.118:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7feed402a910 con 0x55c572be2510 2023-10-05T19:28:54.453 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:54.446+0000 7feeef1e2e40 1 -- 172.21.15.167:0/4243588109 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55c572be2510 msgr2=0x55c572be28e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-05T19:28:54.453 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:54.446+0000 7feeef1e2e40 1 --2- 172.21.15.167:0/4243588109 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55c572be2510 0x55c572be28e0 secure :-1 s=READY pgs=516 cs=0 l=1 rev1=1 crypto rx=0x55c572b06c60 tx=0x7feed4012000 comp rx=0 tx=0).stop 2023-10-05T19:28:54.453 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:54.446+0000 7feeef1e2e40 1 -- 172.21.15.167:0/4243588109 shutdown_connections 2023-10-05T19:28:54.454 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:54.446+0000 7feeef1e2e40 1 --2- 172.21.15.167:0/4243588109 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55c572be2510 0x55c572be28e0 unknown :-1 s=CLOSED pgs=516 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-05T19:28:54.454 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:54.446+0000 7feeef1e2e40 1 -- 172.21.15.167:0/4243588109 >> 172.21.15.167:0/4243588109 conn(0x55c572bdd8d0 msgr2=0x55c572bdba80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-05T19:28:54.454 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:54.446+0000 7feeef1e2e40 1 -- 172.21.15.167:0/4243588109 shutdown_connections 2023-10-05T19:28:54.454 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:54.446+0000 7feeef1e2e40 1 -- 172.21.15.167:0/4243588109 wait complete. 2023-10-05T19:28:54.455 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:54.450+0000 7feeef1e2e40 1 Processor -- start 2023-10-05T19:28:54.455 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:54.450+0000 7feeef1e2e40 1 -- start start 2023-10-05T19:28:54.455 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:54.450+0000 7feeef1e2e40 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55c572be2510 0x55c572be7ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-05T19:28:54.455 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:54.450+0000 7feeef1e2e40 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c572be8ff0 con 0x55c572be2510 2023-10-05T19:28:54.455 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:54.450+0000 7feeec516700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55c572be2510 0x55c572be7ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-05T19:28:54.455 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:54.450+0000 7feeec516700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55c572be2510 0x55c572be7ff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.118:3300/0 says I am v2:172.21.15.167:53146/0 (socket says 172.21.15.167:53146) 2023-10-05T19:28:54.456 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:54.450+0000 7feeec516700 1 -- 172.21.15.167:0/733441895 learned_addr learned my addr 172.21.15.167:0/733441895 (peer_addr_for_me v2:172.21.15.167:0/0) 2023-10-05T19:28:54.456 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:54.450+0000 7feeec516700 1 -- 172.21.15.167:0/733441895 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55c572b9ac20 con 0x55c572be2510 2023-10-05T19:28:54.456 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:54.450+0000 7feeec516700 1 --2- 172.21.15.167:0/733441895 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55c572be2510 0x55c572be7ff0 secure :-1 s=READY pgs=517 cs=0 l=1 rev1=1 crypto rx=0x55c572bdc5f0 tx=0x7feed402f000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-05T19:28:54.457 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:54.450+0000 7fee2ffff700 1 -- 172.21.15.167:0/733441895 <== mon.0 v2:172.21.15.118:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7feed402a540 con 0x55c572be2510 2023-10-05T19:28:54.457 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:54.450+0000 7fee2ffff700 1 -- 172.21.15.167:0/733441895 <== mon.0 v2:172.21.15.118:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7feed403ecd0 con 0x55c572be2510 2023-10-05T19:28:54.457 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:54.450+0000 7feeef1e2e40 1 -- 172.21.15.167:0/733441895 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x55c572be85f0 con 0x55c572be2510 2023-10-05T19:28:54.457 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:54.450+0000 7fee2ffff700 1 -- 172.21.15.167:0/733441895 <== mon.0 v2:172.21.15.118:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7feed403ee80 con 0x55c572be2510 2023-10-05T19:28:54.457 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:54.450+0000 7feeef1e2e40 1 -- 172.21.15.167:0/733441895 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55c572be8a30 con 0x55c572be2510 2023-10-05T19:28:54.458 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:54.450+0000 7fee2ffff700 1 -- 172.21.15.167:0/733441895 <== mon.0 v2:172.21.15.118:3300/0 4 ==== mgrmap(e 5) v1 ==== 40329+0+0 (secure 0 0 0) 0x7feed4046070 con 0x55c572be2510 2023-10-05T19:28:54.458 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:54.454+0000 7fee2ffff700 1 --2- 172.21.15.167:0/733441895 >> [v2:172.21.15.167:6824/29889,v1:172.21.15.167:6825/29889] conn(0x7fee34030c50 0x7fee34033100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-05T19:28:54.458 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:54.454+0000 7fee2ffff700 1 -- 172.21.15.167:0/733441895 <== mon.0 v2:172.21.15.118:3300/0 5 ==== osd_map(32..32 src has 1..32) v4 ==== 6260+0+0 (secure 0 0 0) 0x7feed402ae40 con 0x55c572be2510 2023-10-05T19:28:54.458 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:54.454+0000 7feeed518700 1 --2- 172.21.15.167:0/733441895 >> [v2:172.21.15.167:6824/29889,v1:172.21.15.167:6825/29889] conn(0x7fee34030c50 0x7fee34033100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-05T19:28:54.459 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:54.454+0000 7feeef1e2e40 1 --2- 172.21.15.167:0/733441895 >> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] conn(0x55c572bf2be0 0x55c572bf4ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-05T19:28:54.459 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:54.454+0000 7feeef1e2e40 1 -- 172.21.15.167:0/733441895 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:1 6.19 6:9a6dd8c0:gc::gc.0:head [create,call version.check_conds in=74b,call rgw_gc.rgw_gc_queue_init in=45b,call version.set in=34b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e32) v8 -- 0x55c572bf5690 con 0x55c572bf2be0 2023-10-05T19:28:54.459 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:54.454+0000 7feeecd17700 1 --2- 172.21.15.167:0/733441895 >> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] conn(0x55c572bf2be0 0x55c572bf4ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-05T19:28:54.459 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:54.454+0000 7feeed518700 1 --2- 172.21.15.167:0/733441895 >> [v2:172.21.15.167:6824/29889,v1:172.21.15.167:6825/29889] conn(0x7fee34030c50 0x7fee34033100 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x55c572bd8110 tx=0x7feee400aec0 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-05T19:28:54.459 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:54.454+0000 7feeecd17700 1 --2- 172.21.15.167:0/733441895 >> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] conn(0x55c572bf2be0 0x55c572bf4ff0 crc :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-05T19:28:54.462 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:54.458+0000 7feeecd17700 1 -- 172.21.15.167:0/733441895 <== osd.1 v2:172.21.15.167:6800/30035 1 ==== osd_op_reply(1 gc.0 [create,call,call,call] v32'30 uv3 ondisk = -125 ((125) Operation canceled)) v8 ==== 274+0+0 (crc 0 0 0) 0x7feedc04d040 con 0x55c572bf2be0 2023-10-05T19:28:54.462 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:54.458+0000 7feeef1e2e40 1 --2- 172.21.15.167:0/733441895 >> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] conn(0x55c572bee820 0x55c572bf8d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-05T19:28:54.462 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:54.458+0000 7feeef1e2e40 1 -- 172.21.15.167:0/733441895 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:2 6.18 6:1e85421f:gc::gc.1:head [create,call version.check_conds in=74b,call rgw_gc.rgw_gc_queue_init in=45b,call version.set in=34b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e32) v8 -- 0x55c572bf9240 con 0x55c572bee820 2023-10-05T19:28:54.462 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:54.458+0000 7feeec516700 1 --2- 172.21.15.167:0/733441895 >> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] conn(0x55c572bee820 0x55c572bf8d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-05T19:28:54.463 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:54.458+0000 7feeec516700 1 --2- 172.21.15.167:0/733441895 >> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] conn(0x55c572bee820 0x55c572bf8d00 crc :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-05T19:28:54.466 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:54.462+0000 7feeec516700 1 -- 172.21.15.167:0/733441895 <== osd.0 v2:172.21.15.118:6800/27690 1 ==== osd_op_reply(2 gc.1 [create,call,call,call] v32'22 uv5 ondisk = -125 ((125) Operation canceled)) v8 ==== 274+0+0 (crc 0 0 0) 0x7feed4032040 con 0x55c572bee820 2023-10-05T19:28:54.466 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:54.462+0000 7feeef1e2e40 1 --2- 172.21.15.167:0/733441895 >> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] conn(0x55c572bfa290 0x55c572bfc670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-05T19:28:54.466 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:54.462+0000 7feeef1e2e40 1 -- 172.21.15.167:0/733441895 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:3 6.b 6:d242335b:gc::gc.2:head [create,call version.check_conds in=74b,call rgw_gc.rgw_gc_queue_init in=45b,call version.set in=34b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e32) v8 -- 0x55c572bfcd30 con 0x55c572bfa290 2023-10-05T19:28:54.466 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:54.462+0000 7feeed518700 1 --2- 172.21.15.167:0/733441895 >> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] conn(0x55c572bfa290 0x55c572bfc670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-05T19:28:54.466 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:54.462+0000 7feeed518700 1 --2- 172.21.15.167:0/733441895 >> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] conn(0x55c572bfa290 0x55c572bfc670 crc :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-05T19:28:54.469 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:54.462+0000 7feeed518700 1 -- 172.21.15.167:0/733441895 <== osd.3 v2:172.21.15.167:6808/30036 1 ==== osd_op_reply(3 gc.2 [create,call,call,call] v32'18 uv3 ondisk = -125 ((125) Operation canceled)) v8 ==== 274+0+0 (crc 0 0 0) 0x7feee4052040 con 0x55c572bfa290 2023-10-05T19:28:54.469 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:54.462+0000 7feeef1e2e40 1 -- 172.21.15.167:0/733441895 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:4 6.4 6:2166d8e1:gc::gc.3:head [create,call version.check_conds in=74b,call rgw_gc.rgw_gc_queue_init in=45b,call version.set in=34b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e32) v8 -- 0x55c572bfd380 con 0x55c572bf2be0 2023-10-05T19:28:54.472 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:54.466+0000 7feeecd17700 1 -- 172.21.15.167:0/733441895 <== osd.1 v2:172.21.15.167:6800/30035 2 ==== osd_op_reply(4 gc.3 [create,call,call,call] v32'175 uv7 ondisk = -125 ((125) Operation canceled)) v8 ==== 274+0+0 (crc 0 0 0) 0x7feedc04d040 con 0x55c572bf2be0 2023-10-05T19:28:54.472 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:54.466+0000 7feeef1e2e40 1 --2- 172.21.15.167:0/733441895 >> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] conn(0x55c572bfe3a0 0x55c572c00780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-05T19:28:54.472 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:54.466+0000 7feeef1e2e40 1 -- 172.21.15.167:0/733441895 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:5 6.3 6:c60da066:gc::gc.4:head [create,call version.check_conds in=74b,call rgw_gc.rgw_gc_queue_init in=45b,call version.set in=34b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e32) v8 -- 0x55c572c00df0 con 0x55c572bfe3a0 2023-10-05T19:28:54.472 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:54.466+0000 7feeecd17700 1 --2- 172.21.15.167:0/733441895 >> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] conn(0x55c572bfe3a0 0x55c572c00780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-05T19:28:54.473 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:54.466+0000 7feeecd17700 1 --2- 172.21.15.167:0/733441895 >> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] conn(0x55c572bfe3a0 0x55c572c00780 crc :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-05T19:28:54.475 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:54.470+0000 7feeecd17700 1 -- 172.21.15.167:0/733441895 <== osd.2 v2:172.21.15.167:6816/30037 1 ==== osd_op_reply(5 gc.4 [create,call,call,call] v32'44 uv7 ondisk = -125 ((125) Operation canceled)) v8 ==== 274+0+0 (crc 0 0 0) 0x7feedc04d040 con 0x55c572bfe3a0 2023-10-05T19:28:54.476 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:54.470+0000 7feeef1e2e40 1 -- 172.21.15.167:0/733441895 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:6 6.9 6:954be8d2:gc::gc.5:head [create,call version.check_conds in=74b,call rgw_gc.rgw_gc_queue_init in=45b,call version.set in=34b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e32) v8 -- 0x55c572c01440 con 0x55c572bfa290 2023-10-05T19:28:54.478 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:54.474+0000 7feeed518700 1 -- 172.21.15.167:0/733441895 <== osd.3 v2:172.21.15.167:6808/30036 2 ==== osd_op_reply(6 gc.5 [create,call,call,call] v32'32 uv4 ondisk = -125 ((125) Operation canceled)) v8 ==== 274+0+0 (crc 0 0 0) 0x7feee4052040 con 0x55c572bfa290 2023-10-05T19:28:54.478 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:54.474+0000 7feeef1e2e40 1 -- 172.21.15.167:0/733441895 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:7 6.e 6:748722ff:gc::gc.6:head [create,call version.check_conds in=74b,call rgw_gc.rgw_gc_queue_init in=45b,call version.set in=34b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e32) v8 -- 0x55c572c01930 con 0x55c572bf2be0 2023-10-05T19:28:54.481 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:54.474+0000 7feeecd17700 1 -- 172.21.15.167:0/733441895 <== osd.1 v2:172.21.15.167:6800/30035 3 ==== osd_op_reply(7 gc.6 [create,call,call,call] v32'20 uv5 ondisk = -125 ((125) Operation canceled)) v8 ==== 274+0+0 (crc 0 0 0) 0x7feedc04d040 con 0x55c572bf2be0 2023-10-05T19:28:54.481 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:54.474+0000 7feeef1e2e40 1 -- 172.21.15.167:0/733441895 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:8 6.1d 6:beaa4be4:gc::gc.7:head [create,call version.check_conds in=74b,call rgw_gc.rgw_gc_queue_init in=45b,call version.set in=34b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e32) v8 -- 0x55c572c01930 con 0x55c572bee820 2023-10-05T19:28:54.484 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:54.478+0000 7feeec516700 1 -- 172.21.15.167:0/733441895 <== osd.0 v2:172.21.15.118:6800/27690 2 ==== osd_op_reply(8 gc.7 [create,call,call,call] v32'29 uv4 ondisk = -125 ((125) Operation canceled)) v8 ==== 274+0+0 (crc 0 0 0) 0x7feed4032040 con 0x55c572bee820 2023-10-05T19:28:54.484 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:54.478+0000 7feeef1e2e40 1 -- 172.21.15.167:0/733441895 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:9 6.12 6:4f78e9a2:gc::gc.8:head [create,call version.check_conds in=74b,call rgw_gc.rgw_gc_queue_init in=45b,call version.set in=34b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e32) v8 -- 0x55c572c01f50 con 0x55c572bfe3a0 2023-10-05T19:28:54.487 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:54.482+0000 7feeecd17700 1 -- 172.21.15.167:0/733441895 <== osd.2 v2:172.21.15.167:6816/30037 2 ==== osd_op_reply(9 gc.8 [create,call,call,call] v32'18 uv3 ondisk = -125 ((125) Operation canceled)) v8 ==== 274+0+0 (crc 0 0 0) 0x7feedc04d040 con 0x55c572bfe3a0 2023-10-05T19:28:54.487 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:54.482+0000 7feeef1e2e40 1 -- 172.21.15.167:0/733441895 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:10 6.1c 6:3e426232:gc::gc.9:head [create,call version.check_conds in=74b,call rgw_gc.rgw_gc_queue_init in=45b,call version.set in=34b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e32) v8 -- 0x55c572c01f50 con 0x55c572bfa290 2023-10-05T19:28:54.489 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:54.482+0000 7feeed518700 1 -- 172.21.15.167:0/733441895 <== osd.3 v2:172.21.15.167:6808/30036 3 ==== osd_op_reply(10 gc.9 [create,call,call,call] v32'36 uv3 ondisk = -125 ((125) Operation canceled)) v8 ==== 274+0+0 (crc 0 0 0) 0x7feee4052040 con 0x55c572bfa290 2023-10-05T19:28:54.490 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:54.486+0000 7feeef1e2e40 1 -- 172.21.15.167:0/733441895 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:11 6.3 6:c31f41d7:gc::gc.10:head [create,call version.check_conds in=74b,call rgw_gc.rgw_gc_queue_init in=45b,call version.set in=34b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e32) v8 -- 0x55c572c02440 con 0x55c572bfe3a0 2023-10-05T19:28:54.492 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:54.486+0000 7feeecd17700 1 -- 172.21.15.167:0/733441895 <== osd.2 v2:172.21.15.167:6816/30037 3 ==== osd_op_reply(11 gc.10 [create,call,call,call] v32'45 uv11 ondisk = -125 ((125) Operation canceled)) v8 ==== 275+0+0 (crc 0 0 0) 0x7feedc04d040 con 0x55c572bfe3a0 2023-10-05T19:28:54.493 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:54.486+0000 7feeef1e2e40 1 -- 172.21.15.167:0/733441895 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:12 6.13 6:c8333aab:gc::gc.11:head [create,call version.check_conds in=74b,call rgw_gc.rgw_gc_queue_init in=45b,call version.set in=34b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e32) v8 -- 0x55c572c02440 con 0x55c572bee820 2023-10-05T19:28:54.495 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:54.490+0000 7feeec516700 1 -- 172.21.15.167:0/733441895 <== osd.0 v2:172.21.15.118:6800/27690 3 ==== osd_op_reply(12 gc.11 [create,call,call,call] v32'22 uv5 ondisk = -125 ((125) Operation canceled)) v8 ==== 275+0+0 (crc 0 0 0) 0x7feed4032040 con 0x55c572bee820 2023-10-05T19:28:54.496 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:54.490+0000 7feeef1e2e40 1 -- 172.21.15.167:0/733441895 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:13 6.15 6:ab569f81:gc::gc.12:head [create,call version.check_conds in=74b,call rgw_gc.rgw_gc_queue_init in=45b,call version.set in=34b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e32) v8 -- 0x55c572c02a60 con 0x55c572bfe3a0 2023-10-05T19:28:54.498 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:54.494+0000 7feeecd17700 1 -- 172.21.15.167:0/733441895 <== osd.2 v2:172.21.15.167:6816/30037 4 ==== osd_op_reply(13 gc.12 [create,call,call,call] v32'18 uv3 ondisk = -125 ((125) Operation canceled)) v8 ==== 275+0+0 (crc 0 0 0) 0x7feedc04d040 con 0x55c572bfe3a0 2023-10-05T19:28:54.498 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:54.494+0000 7feeef1e2e40 1 -- 172.21.15.167:0/733441895 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:14 6.0 6:0667bb13:gc::gc.13:head [create,call version.check_conds in=74b,call rgw_gc.rgw_gc_queue_init in=45b,call version.set in=34b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e32) v8 -- 0x55c572c02a60 con 0x55c572bee820 2023-10-05T19:28:54.502 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:54.498+0000 7feeec516700 1 -- 172.21.15.167:0/733441895 <== osd.0 v2:172.21.15.118:6800/27690 4 ==== osd_op_reply(14 gc.13 [create,call,call,call] v32'35 uv6 ondisk = -125 ((125) Operation canceled)) v8 ==== 275+0+0 (crc 0 0 0) 0x7feed4032040 con 0x55c572bee820 2023-10-05T19:28:54.502 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:54.498+0000 7feeef1e2e40 1 -- 172.21.15.167:0/733441895 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:15 6.4 6:227f63b5:gc::gc.14:head [create,call version.check_conds in=74b,call rgw_gc.rgw_gc_queue_init in=45b,call version.set in=34b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e32) v8 -- 0x55c572c02f50 con 0x55c572bf2be0 2023-10-05T19:28:54.504 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:54.498+0000 7feeecd17700 1 -- 172.21.15.167:0/733441895 <== osd.1 v2:172.21.15.167:6800/30035 4 ==== osd_op_reply(15 gc.14 [create,call,call,call] v32'176 uv13 ondisk = -125 ((125) Operation canceled)) v8 ==== 275+0+0 (crc 0 0 0) 0x7feedc04d040 con 0x55c572bf2be0 2023-10-05T19:28:54.504 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:54.498+0000 7feeef1e2e40 1 -- 172.21.15.167:0/733441895 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:16 6.7 6:e07a6548:gc::gc.15:head [create,call version.check_conds in=74b,call rgw_gc.rgw_gc_queue_init in=45b,call version.set in=34b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e32) v8 -- 0x55c572c02f50 con 0x55c572bfa290 2023-10-05T19:28:54.507 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:54.502+0000 7feeed518700 1 -- 172.21.15.167:0/733441895 <== osd.3 v2:172.21.15.167:6808/30036 4 ==== osd_op_reply(16 gc.15 [create,call,call,call] v32'38 uv7 ondisk = -125 ((125) Operation canceled)) v8 ==== 275+0+0 (crc 0 0 0) 0x7feee4052040 con 0x55c572bfa290 2023-10-05T19:28:54.507 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:54.502+0000 7feeef1e2e40 1 -- 172.21.15.167:0/733441895 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:17 6.3 6:c623b173:gc::gc.16:head [create,call version.check_conds in=74b,call rgw_gc.rgw_gc_queue_init in=45b,call version.set in=34b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e32) v8 -- 0x55c572c02f50 con 0x55c572bfe3a0 2023-10-05T19:28:54.510 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:54.506+0000 7feeecd17700 1 -- 172.21.15.167:0/733441895 <== osd.2 v2:172.21.15.167:6816/30037 5 ==== osd_op_reply(17 gc.16 [create,call,call,call] v32'46 uv13 ondisk = -125 ((125) Operation canceled)) v8 ==== 275+0+0 (crc 0 0 0) 0x7feedc04d040 con 0x55c572bfe3a0 2023-10-05T19:28:54.510 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:54.506+0000 7feeef1e2e40 1 -- 172.21.15.167:0/733441895 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:18 6.16 6:6849fd06:gc::gc.17:head [create,call version.check_conds in=74b,call rgw_gc.rgw_gc_queue_init in=45b,call version.set in=34b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e32) v8 -- 0x55c572c02f50 con 0x55c572bfa290 2023-10-05T19:28:54.513 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:54.506+0000 7feeed518700 1 -- 172.21.15.167:0/733441895 <== osd.3 v2:172.21.15.167:6808/30036 5 ==== osd_op_reply(18 gc.17 [create,call,call,call] v32'18 uv7 ondisk = -125 ((125) Operation canceled)) v8 ==== 275+0+0 (crc 0 0 0) 0x7feee4052040 con 0x55c572bfa290 2023-10-05T19:28:54.513 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:54.506+0000 7feeef1e2e40 1 -- 172.21.15.167:0/733441895 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:19 6.d 6:b019ea83:gc::gc.18:head [create,call version.check_conds in=74b,call rgw_gc.rgw_gc_queue_init in=45b,call version.set in=34b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e32) v8 -- 0x55c572c02f50 con 0x55c572bf2be0 2023-10-05T19:28:54.515 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:54.510+0000 7feeecd17700 1 -- 172.21.15.167:0/733441895 <== osd.1 v2:172.21.15.167:6800/30035 5 ==== osd_op_reply(19 gc.18 [create,call,call,call] v32'26 uv5 ondisk = -125 ((125) Operation canceled)) v8 ==== 275+0+0 (crc 0 0 0) 0x7feedc04d040 con 0x55c572bf2be0 2023-10-05T19:28:54.516 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:54.510+0000 7feeef1e2e40 1 -- 172.21.15.167:0/733441895 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:20 6.4 6:24c8e1ae:gc::gc.19:head [create,call version.check_conds in=74b,call rgw_gc.rgw_gc_queue_init in=45b,call version.set in=34b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e32) v8 -- 0x55c572c02f50 con 0x55c572bf2be0 2023-10-05T19:28:54.518 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:54.514+0000 7feeecd17700 1 -- 172.21.15.167:0/733441895 <== osd.1 v2:172.21.15.167:6800/30035 6 ==== osd_op_reply(20 gc.19 [create,call,call,call] v32'177 uv173 ondisk = -125 ((125) Operation canceled)) v8 ==== 275+0+0 (crc 0 0 0) 0x7feedc04d040 con 0x55c572bf2be0 2023-10-05T19:28:54.518 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:54.514+0000 7feeef1e2e40 1 -- 172.21.15.167:0/733441895 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:21 6.7 6:e622f76a:gc::gc.20:head [create,call version.check_conds in=74b,call rgw_gc.rgw_gc_queue_init in=45b,call version.set in=34b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e32) v8 -- 0x55c572c02f50 con 0x55c572bfa290 2023-10-05T19:28:54.521 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:54.514+0000 7feeed518700 1 -- 172.21.15.167:0/733441895 <== osd.3 v2:172.21.15.167:6808/30036 6 ==== osd_op_reply(21 gc.20 [create,call,call,call] v32'39 uv9 ondisk = -125 ((125) Operation canceled)) v8 ==== 275+0+0 (crc 0 0 0) 0x7feee4052040 con 0x55c572bfa290 2023-10-05T19:28:54.521 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:54.514+0000 7feeef1e2e40 1 -- 172.21.15.167:0/733441895 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:22 6.a 6:5127e642:gc::gc.21:head [create,call version.check_conds in=74b,call rgw_gc.rgw_gc_queue_init in=45b,call version.set in=34b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e32) v8 -- 0x55c572c02f50 con 0x55c572bfe3a0 2023-10-05T19:28:54.523 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:54.518+0000 7feeecd17700 1 -- 172.21.15.167:0/733441895 <== osd.2 v2:172.21.15.167:6816/30037 6 ==== osd_op_reply(22 gc.21 [create,call,call,call] v32'28 uv7 ondisk = -125 ((125) Operation canceled)) v8 ==== 275+0+0 (crc 0 0 0) 0x7feedc04d040 con 0x55c572bfe3a0 2023-10-05T19:28:54.524 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:54.518+0000 7feeef1e2e40 1 -- 172.21.15.167:0/733441895 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:23 6.11 6:8eb99c53:gc::gc.22:head [create,call version.check_conds in=74b,call rgw_gc.rgw_gc_queue_init in=45b,call version.set in=34b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e32) v8 -- 0x55c572c02f50 con 0x55c572bfa290 2023-10-05T19:28:54.526 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:54.522+0000 7feeed518700 1 -- 172.21.15.167:0/733441895 <== osd.3 v2:172.21.15.167:6808/30036 7 ==== osd_op_reply(23 gc.22 [create,call,call,call] v32'20 uv3 ondisk = -125 ((125) Operation canceled)) v8 ==== 275+0+0 (crc 0 0 0) 0x7feee4052040 con 0x55c572bfa290 2023-10-05T19:28:54.526 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:54.522+0000 7feeef1e2e40 1 -- 172.21.15.167:0/733441895 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:24 6.8 6:17d7e8ed:gc::gc.23:head [create,call version.check_conds in=74b,call rgw_gc.rgw_gc_queue_init in=45b,call version.set in=34b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e32) v8 -- 0x55c572c02f50 con 0x55c572bfa290 2023-10-05T19:28:54.529 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:54.522+0000 7feeed518700 1 -- 172.21.15.167:0/733441895 <== osd.3 v2:172.21.15.167:6808/30036 8 ==== osd_op_reply(24 gc.23 [create,call,call,call] v32'30 uv5 ondisk = -125 ((125) Operation canceled)) v8 ==== 275+0+0 (crc 0 0 0) 0x7feee4052040 con 0x55c572bfa290 2023-10-05T19:28:54.529 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:54.526+0000 7feeef1e2e40 1 -- 172.21.15.167:0/733441895 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:25 6.1b 6:db9d130d:gc::gc.24:head [create,call version.check_conds in=74b,call rgw_gc.rgw_gc_queue_init in=45b,call version.set in=34b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e32) v8 -- 0x55c572c02f50 con 0x55c572bfa290 2023-10-05T19:28:54.532 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:54.526+0000 7feeed518700 1 -- 172.21.15.167:0/733441895 <== osd.3 v2:172.21.15.167:6808/30036 9 ==== osd_op_reply(25 gc.24 [create,call,call,call] v32'14 uv3 ondisk = -125 ((125) Operation canceled)) v8 ==== 275+0+0 (crc 0 0 0) 0x7feee4052040 con 0x55c572bfa290 2023-10-05T19:28:54.532 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:54.526+0000 7feeef1e2e40 1 -- 172.21.15.167:0/733441895 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:26 6.9 6:921ecdb1:gc::gc.25:head [create,call version.check_conds in=74b,call rgw_gc.rgw_gc_queue_init in=45b,call version.set in=34b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e32) v8 -- 0x55c572c02f50 con 0x55c572bfa290 2023-10-05T19:28:54.534 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:54.530+0000 7feeed518700 1 -- 172.21.15.167:0/733441895 <== osd.3 v2:172.21.15.167:6808/30036 10 ==== osd_op_reply(26 gc.25 [create,call,call,call] v32'33 uv10 ondisk = -125 ((125) Operation canceled)) v8 ==== 275+0+0 (crc 0 0 0) 0x7feee4052040 con 0x55c572bfa290 2023-10-05T19:28:54.534 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:54.530+0000 7feeef1e2e40 1 -- 172.21.15.167:0/733441895 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:27 6.1f 6:fde4dd55:gc::gc.26:head [create,call version.check_conds in=74b,call rgw_gc.rgw_gc_queue_init in=45b,call version.set in=34b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e32) v8 -- 0x55c572c02f50 con 0x55c572bee820 2023-10-05T19:28:54.537 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:54.530+0000 7feeec516700 1 -- 172.21.15.167:0/733441895 <== osd.0 v2:172.21.15.118:6800/27690 5 ==== osd_op_reply(27 gc.26 [create,call,call,call] v32'20 uv5 ondisk = -125 ((125) Operation canceled)) v8 ==== 275+0+0 (crc 0 0 0) 0x7feed4032040 con 0x55c572bee820 2023-10-05T19:28:54.537 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:54.530+0000 7feeef1e2e40 1 -- 172.21.15.167:0/733441895 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:28 6.1a 6:5909e671:gc::gc.27:head [create,call version.check_conds in=74b,call rgw_gc.rgw_gc_queue_init in=45b,call version.set in=34b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e32) v8 -- 0x55c572c02f50 con 0x55c572bfa290 2023-10-05T19:28:54.540 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:54.534+0000 7feeed518700 1 -- 172.21.15.167:0/733441895 <== osd.3 v2:172.21.15.167:6808/30036 11 ==== osd_op_reply(28 gc.27 [create,call,call,call] v32'18 uv7 ondisk = -125 ((125) Operation canceled)) v8 ==== 275+0+0 (crc 0 0 0) 0x7feee4052040 con 0x55c572bfa290 2023-10-05T19:28:54.540 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:54.534+0000 7feeef1e2e40 1 -- 172.21.15.167:0/733441895 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:29 6.1d 6:b88e3271:gc::gc.28:head [create,call version.check_conds in=74b,call rgw_gc.rgw_gc_queue_init in=45b,call version.set in=34b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e32) v8 -- 0x55c572c02f50 con 0x55c572bee820 2023-10-05T19:28:54.542 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:54.538+0000 7feeec516700 1 -- 172.21.15.167:0/733441895 <== osd.0 v2:172.21.15.118:6800/27690 6 ==== osd_op_reply(29 gc.28 [create,call,call,call] v32'30 uv8 ondisk = -125 ((125) Operation canceled)) v8 ==== 275+0+0 (crc 0 0 0) 0x7feed4032040 con 0x55c572bee820 2023-10-05T19:28:54.542 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:54.538+0000 7feeef1e2e40 1 -- 172.21.15.167:0/733441895 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:30 6.7 6:e47c9322:gc::gc.29:head [create,call version.check_conds in=74b,call rgw_gc.rgw_gc_queue_init in=45b,call version.set in=34b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e32) v8 -- 0x55c572c02f50 con 0x55c572bfa290 2023-10-05T19:28:54.545 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:54.538+0000 7feeed518700 1 -- 172.21.15.167:0/733441895 <== osd.3 v2:172.21.15.167:6808/30036 12 ==== osd_op_reply(30 gc.29 [create,call,call,call] v32'40 uv11 ondisk = -125 ((125) Operation canceled)) v8 ==== 275+0+0 (crc 0 0 0) 0x7feee4052040 con 0x55c572bfa290 2023-10-05T19:28:54.545 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:54.538+0000 7feeef1e2e40 1 -- 172.21.15.167:0/733441895 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:31 6.0 6:013d6c33:gc::gc.30:head [create,call version.check_conds in=74b,call rgw_gc.rgw_gc_queue_init in=45b,call version.set in=34b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e32) v8 -- 0x55c572c02f50 con 0x55c572bee820 2023-10-05T19:28:54.548 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:54.542+0000 7feeec516700 1 -- 172.21.15.167:0/733441895 <== osd.0 v2:172.21.15.118:6800/27690 7 ==== osd_op_reply(31 gc.30 [create,call,call,call] v32'36 uv12 ondisk = -125 ((125) Operation canceled)) v8 ==== 275+0+0 (crc 0 0 0) 0x7feed4032040 con 0x55c572bee820 2023-10-05T19:28:54.548 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:54.542+0000 7feeef1e2e40 1 -- 172.21.15.167:0/733441895 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:32 6.f 6:f24cb4a8:gc::gc.31:head [create,call version.check_conds in=74b,call rgw_gc.rgw_gc_queue_init in=45b,call version.set in=34b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e32) v8 -- 0x55c572c02f50 con 0x55c572bfe3a0 2023-10-05T19:28:54.550 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:54.546+0000 7feeecd17700 1 -- 172.21.15.167:0/733441895 <== osd.2 v2:172.21.15.167:6816/30037 7 ==== osd_op_reply(32 gc.31 [create,call,call,call] v32'24 uv5 ondisk = -125 ((125) Operation canceled)) v8 ==== 275+0+0 (crc 0 0 0) 0x7feedc04d040 con 0x55c572bfe3a0 2023-10-05T19:28:54.554 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:54.550+0000 7fee2d7fa700 20 reqs_thread_entry: start 2023-10-05T19:28:54.554 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:54.550+0000 7fee26ffd700 20 reqs_thread_entry: start 2023-10-05T19:28:54.555 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:54.550+0000 7feeef1e2e40 20 init_complete bucket index max shards: 11 2023-10-05T19:28:54.555 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:54.550+0000 7feeef1e2e40 10 rgw notify: Started notification manager with: 1 workers 2023-10-05T19:28:54.555 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:54.550+0000 7fee257fa700 20 rgw notify: INFO: next queues processing will happen at: Thu Oct 5 19:29:24 2023 2023-10-05T19:28:54.556 INFO:teuthology.orchestra.run.smithi167.stderr: 2023-10-05T19:28:54.556 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:54.550+0000 7fee24ff9700 20 reqs_thread_entry: start 2023-10-05T19:28:54.556 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:54.550+0000 7feeef1e2e40 1 -- 172.21.15.167:0/733441895 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:33 6.19 6:9a6dd8c0:gc::gc.0:head [call rgw.gc_list in=25b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32) v8 -- 0x55c572c621d0 con 0x55c572bf2be0 2023-10-05T19:28:54.557 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:54.550+0000 7feeecd17700 1 -- 172.21.15.167:0/733441895 <== osd.1 v2:172.21.15.167:6800/30035 7 ==== osd_op_reply(33 gc.0 [call out=15b] v0'0 uv3 ondisk = 0) v8 ==== 148+0+15 (crc 0 0 0) 0x7feedc04d040 con 0x55c572bf2be0 2023-10-05T19:28:54.557 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:54.550+0000 7feeef1e2e40 1 -- 172.21.15.167:0/733441895 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:34 6.19 6:9a6dd8c0:gc::gc.0:head [call version.read in=11b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32) v8 -- 0x55c572c611c0 con 0x55c572bf2be0 2023-10-05T19:28:54.557 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:54.550+0000 7feeecd17700 1 -- 172.21.15.167:0/733441895 <== osd.1 v2:172.21.15.167:6800/30035 8 ==== osd_op_reply(34 gc.0 [call out=24b] v0'0 uv3 ondisk = 0) v8 ==== 148+0+24 (crc 0 0 0) 0x7feedc04d040 con 0x55c572bf2be0 2023-10-05T19:28:54.557 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:54.550+0000 7feeef1e2e40 1 -- 172.21.15.167:0/733441895 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:35 6.19 6:9a6dd8c0:gc::gc.0:head [call rgw_gc.rgw_gc_queue_list_entries in=46b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32) v8 -- 0x55c572c621d0 con 0x55c572bf2be0 2023-10-05T19:28:54.558 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:54.550+0000 7feeecd17700 1 -- 172.21.15.167:0/733441895 <== osd.1 v2:172.21.15.167:6800/30035 9 ==== osd_op_reply(35 gc.0 [call out=15b] v0'0 uv3 ondisk = 0) v8 ==== 148+0+15 (crc 0 0 0) 0x7feedc04d040 con 0x55c572bf2be0 2023-10-05T19:28:54.558 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:54.550+0000 7feeef1e2e40 1 -- 172.21.15.167:0/733441895 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:36 6.18 6:1e85421f:gc::gc.1:head [call rgw.gc_list in=25b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32) v8 -- 0x55c572c621d0 con 0x55c572bee820 2023-10-05T19:28:54.559 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:54.554+0000 7feeec516700 1 -- 172.21.15.167:0/733441895 <== osd.0 v2:172.21.15.118:6800/27690 8 ==== osd_op_reply(36 gc.1 [call out=15b] v0'0 uv5 ondisk = 0) v8 ==== 148+0+15 (crc 0 0 0) 0x7feed4032040 con 0x55c572bee820 2023-10-05T19:28:54.559 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:54.554+0000 7feeef1e2e40 1 -- 172.21.15.167:0/733441895 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:37 6.18 6:1e85421f:gc::gc.1:head [call version.read in=11b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32) v8 -- 0x55c572c611c0 con 0x55c572bee820 2023-10-05T19:28:54.559 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:54.554+0000 7feeec516700 1 -- 172.21.15.167:0/733441895 <== osd.0 v2:172.21.15.118:6800/27690 9 ==== osd_op_reply(37 gc.1 [call out=24b] v0'0 uv5 ondisk = 0) v8 ==== 148+0+24 (crc 0 0 0) 0x7feed4032040 con 0x55c572bee820 2023-10-05T19:28:54.559 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:54.554+0000 7feeef1e2e40 1 -- 172.21.15.167:0/733441895 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:38 6.18 6:1e85421f:gc::gc.1:head [call rgw_gc.rgw_gc_queue_list_entries in=46b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32) v8 -- 0x55c572c621d0 con 0x55c572bee820 2023-10-05T19:28:54.560 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:54.554+0000 7feeec516700 1 -- 172.21.15.167:0/733441895 <== osd.0 v2:172.21.15.118:6800/27690 10 ==== osd_op_reply(38 gc.1 [call out=15b] v0'0 uv5 ondisk = 0) v8 ==== 148+0+15 (crc 0 0 0) 0x7feed4032040 con 0x55c572bee820 2023-10-05T19:28:54.561 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:54.554+0000 7feeef1e2e40 1 -- 172.21.15.167:0/733441895 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:39 6.b 6:d242335b:gc::gc.2:head [call rgw.gc_list in=25b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32) v8 -- 0x55c572c621d0 con 0x55c572bfa290 2023-10-05T19:28:54.561 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:54.554+0000 7feeed518700 1 -- 172.21.15.167:0/733441895 <== osd.3 v2:172.21.15.167:6808/30036 13 ==== osd_op_reply(39 gc.2 [call out=15b] v0'0 uv3 ondisk = 0) v8 ==== 148+0+15 (crc 0 0 0) 0x7feee4052040 con 0x55c572bfa290 2023-10-05T19:28:54.561 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:54.554+0000 7feeef1e2e40 1 -- 172.21.15.167:0/733441895 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:40 6.b 6:d242335b:gc::gc.2:head [call version.read in=11b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32) v8 -- 0x55c572c611c0 con 0x55c572bfa290 2023-10-05T19:28:54.562 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:54.558+0000 7feeed518700 1 -- 172.21.15.167:0/733441895 <== osd.3 v2:172.21.15.167:6808/30036 14 ==== osd_op_reply(40 gc.2 [call out=24b] v0'0 uv3 ondisk = 0) v8 ==== 148+0+24 (crc 0 0 0) 0x7feee4052040 con 0x55c572bfa290 2023-10-05T19:28:54.562 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:54.558+0000 7feeef1e2e40 1 -- 172.21.15.167:0/733441895 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:41 6.b 6:d242335b:gc::gc.2:head [call rgw_gc.rgw_gc_queue_list_entries in=46b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32) v8 -- 0x55c572c621d0 con 0x55c572bfa290 2023-10-05T19:28:54.563 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:54.558+0000 7feeed518700 1 -- 172.21.15.167:0/733441895 <== osd.3 v2:172.21.15.167:6808/30036 15 ==== osd_op_reply(41 gc.2 [call out=15b] v0'0 uv3 ondisk = 0) v8 ==== 148+0+15 (crc 0 0 0) 0x7feee4052040 con 0x55c572bfa290 2023-10-05T19:28:54.563 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:54.558+0000 7feeef1e2e40 1 -- 172.21.15.167:0/733441895 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:42 6.4 6:2166d8e1:gc::gc.3:head [call rgw.gc_list in=25b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32) v8 -- 0x55c572c621d0 con 0x55c572bf2be0 2023-10-05T19:28:54.564 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:54.558+0000 7feeecd17700 1 -- 172.21.15.167:0/733441895 <== osd.1 v2:172.21.15.167:6800/30035 10 ==== osd_op_reply(42 gc.3 [call out=15b] v0'0 uv7 ondisk = 0) v8 ==== 148+0+15 (crc 0 0 0) 0x7feedc04d040 con 0x55c572bf2be0 2023-10-05T19:28:54.564 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:54.558+0000 7feeef1e2e40 1 -- 172.21.15.167:0/733441895 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:43 6.4 6:2166d8e1:gc::gc.3:head [call version.read in=11b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32) v8 -- 0x55c572c611c0 con 0x55c572bf2be0 2023-10-05T19:28:54.564 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:54.558+0000 7feeecd17700 1 -- 172.21.15.167:0/733441895 <== osd.1 v2:172.21.15.167:6800/30035 11 ==== osd_op_reply(43 gc.3 [call out=24b] v0'0 uv7 ondisk = 0) v8 ==== 148+0+24 (crc 0 0 0) 0x7feedc04d040 con 0x55c572bf2be0 2023-10-05T19:28:54.564 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:54.558+0000 7feeef1e2e40 1 -- 172.21.15.167:0/733441895 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:44 6.4 6:2166d8e1:gc::gc.3:head [call rgw_gc.rgw_gc_queue_list_entries in=46b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32) v8 -- 0x55c572c621d0 con 0x55c572bf2be0 2023-10-05T19:28:54.565 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:54.558+0000 7feeecd17700 1 -- 172.21.15.167:0/733441895 <== osd.1 v2:172.21.15.167:6800/30035 12 ==== osd_op_reply(44 gc.3 [call out=15b] v0'0 uv7 ondisk = 0) v8 ==== 148+0+15 (crc 0 0 0) 0x7feedc04d040 con 0x55c572bf2be0 2023-10-05T19:28:54.565 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:54.558+0000 7feeef1e2e40 1 -- 172.21.15.167:0/733441895 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:45 6.3 6:c60da066:gc::gc.4:head [call rgw.gc_list in=25b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32) v8 -- 0x55c572c621d0 con 0x55c572bfe3a0 2023-10-05T19:28:54.566 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:54.562+0000 7feeecd17700 1 -- 172.21.15.167:0/733441895 <== osd.2 v2:172.21.15.167:6816/30037 8 ==== osd_op_reply(45 gc.4 [call out=15b] v0'0 uv7 ondisk = 0) v8 ==== 148+0+15 (crc 0 0 0) 0x7feedc04d040 con 0x55c572bfe3a0 2023-10-05T19:28:54.566 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:54.562+0000 7feeef1e2e40 1 -- 172.21.15.167:0/733441895 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:46 6.3 6:c60da066:gc::gc.4:head [call version.read in=11b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32) v8 -- 0x55c572c611c0 con 0x55c572bfe3a0 2023-10-05T19:28:54.567 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:54.562+0000 7feeecd17700 1 -- 172.21.15.167:0/733441895 <== osd.2 v2:172.21.15.167:6816/30037 9 ==== osd_op_reply(46 gc.4 [call out=24b] v0'0 uv7 ondisk = 0) v8 ==== 148+0+24 (crc 0 0 0) 0x7feedc04d040 con 0x55c572bfe3a0 2023-10-05T19:28:54.567 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:54.562+0000 7feeef1e2e40 1 -- 172.21.15.167:0/733441895 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:47 6.3 6:c60da066:gc::gc.4:head [call rgw_gc.rgw_gc_queue_list_entries in=46b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32) v8 -- 0x55c572c621d0 con 0x55c572bfe3a0 2023-10-05T19:28:54.568 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:54.562+0000 7feeecd17700 1 -- 172.21.15.167:0/733441895 <== osd.2 v2:172.21.15.167:6816/30037 10 ==== osd_op_reply(47 gc.4 [call out=15b] v0'0 uv7 ondisk = 0) v8 ==== 148+0+15 (crc 0 0 0) 0x7feedc04d040 con 0x55c572bfe3a0 2023-10-05T19:28:54.568 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:54.562+0000 7feeef1e2e40 1 -- 172.21.15.167:0/733441895 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:48 6.9 6:954be8d2:gc::gc.5:head [call rgw.gc_list in=25b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32) v8 -- 0x55c572c621d0 con 0x55c572bfa290 2023-10-05T19:28:54.568 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:54.562+0000 7feeed518700 1 -- 172.21.15.167:0/733441895 <== osd.3 v2:172.21.15.167:6808/30036 16 ==== osd_op_reply(48 gc.5 [call out=15b] v0'0 uv4 ondisk = 0) v8 ==== 148+0+15 (crc 0 0 0) 0x7feee4052040 con 0x55c572bfa290 2023-10-05T19:28:54.569 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:54.562+0000 7feeef1e2e40 1 -- 172.21.15.167:0/733441895 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:49 6.9 6:954be8d2:gc::gc.5:head [call version.read in=11b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32) v8 -- 0x55c572c611c0 con 0x55c572bfa290 2023-10-05T19:28:54.569 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:54.562+0000 7feeed518700 1 -- 172.21.15.167:0/733441895 <== osd.3 v2:172.21.15.167:6808/30036 17 ==== osd_op_reply(49 gc.5 [call out=24b] v0'0 uv4 ondisk = 0) v8 ==== 148+0+24 (crc 0 0 0) 0x7feee4052040 con 0x55c572bfa290 2023-10-05T19:28:54.569 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:54.562+0000 7feeef1e2e40 1 -- 172.21.15.167:0/733441895 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:50 6.9 6:954be8d2:gc::gc.5:head [call rgw_gc.rgw_gc_queue_list_entries in=46b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32) v8 -- 0x55c572c621d0 con 0x55c572bfa290 2023-10-05T19:28:54.570 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:54.566+0000 7feeed518700 1 -- 172.21.15.167:0/733441895 <== osd.3 v2:172.21.15.167:6808/30036 18 ==== osd_op_reply(50 gc.5 [call out=15b] v0'0 uv4 ondisk = 0) v8 ==== 148+0+15 (crc 0 0 0) 0x7feee4052040 con 0x55c572bfa290 2023-10-05T19:28:54.570 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:54.566+0000 7feeef1e2e40 1 -- 172.21.15.167:0/733441895 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:51 6.e 6:748722ff:gc::gc.6:head [call rgw.gc_list in=25b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32) v8 -- 0x55c572c621d0 con 0x55c572bf2be0 2023-10-05T19:28:54.570 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:54.566+0000 7feeecd17700 1 -- 172.21.15.167:0/733441895 <== osd.1 v2:172.21.15.167:6800/30035 13 ==== osd_op_reply(51 gc.6 [call out=15b] v0'0 uv5 ondisk = 0) v8 ==== 148+0+15 (crc 0 0 0) 0x7feedc04d040 con 0x55c572bf2be0 2023-10-05T19:28:54.571 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:54.566+0000 7feeef1e2e40 1 -- 172.21.15.167:0/733441895 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:52 6.e 6:748722ff:gc::gc.6:head [call version.read in=11b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32) v8 -- 0x55c572c611c0 con 0x55c572bf2be0 2023-10-05T19:28:54.571 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:54.566+0000 7feeecd17700 1 -- 172.21.15.167:0/733441895 <== osd.1 v2:172.21.15.167:6800/30035 14 ==== osd_op_reply(52 gc.6 [call out=24b] v0'0 uv5 ondisk = 0) v8 ==== 148+0+24 (crc 0 0 0) 0x7feedc04d040 con 0x55c572bf2be0 2023-10-05T19:28:54.571 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:54.566+0000 7feeef1e2e40 1 -- 172.21.15.167:0/733441895 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:53 6.e 6:748722ff:gc::gc.6:head [call rgw_gc.rgw_gc_queue_list_entries in=46b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32) v8 -- 0x55c572c621d0 con 0x55c572bf2be0 2023-10-05T19:28:54.572 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:54.566+0000 7feeecd17700 1 -- 172.21.15.167:0/733441895 <== osd.1 v2:172.21.15.167:6800/30035 15 ==== osd_op_reply(53 gc.6 [call out=15b] v0'0 uv5 ondisk = 0) v8 ==== 148+0+15 (crc 0 0 0) 0x7feedc04d040 con 0x55c572bf2be0 2023-10-05T19:28:54.572 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:54.566+0000 7feeef1e2e40 1 -- 172.21.15.167:0/733441895 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:54 6.1d 6:beaa4be4:gc::gc.7:head [call rgw.gc_list in=25b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32) v8 -- 0x55c572c621d0 con 0x55c572bee820 2023-10-05T19:28:54.573 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:54.566+0000 7feeec516700 1 -- 172.21.15.167:0/733441895 <== osd.0 v2:172.21.15.118:6800/27690 11 ==== osd_op_reply(54 gc.7 [call out=15b] v0'0 uv4 ondisk = 0) v8 ==== 148+0+15 (crc 0 0 0) 0x7feed4032040 con 0x55c572bee820 2023-10-05T19:28:54.573 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:54.566+0000 7feeef1e2e40 1 -- 172.21.15.167:0/733441895 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:55 6.1d 6:beaa4be4:gc::gc.7:head [call version.read in=11b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32) v8 -- 0x55c572c611c0 con 0x55c572bee820 2023-10-05T19:28:54.574 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:54.570+0000 7feeec516700 1 -- 172.21.15.167:0/733441895 <== osd.0 v2:172.21.15.118:6800/27690 12 ==== osd_op_reply(55 gc.7 [call out=24b] v0'0 uv4 ondisk = 0) v8 ==== 148+0+24 (crc 0 0 0) 0x7feed4032040 con 0x55c572bee820 2023-10-05T19:28:54.574 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:54.570+0000 7feeef1e2e40 1 -- 172.21.15.167:0/733441895 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:56 6.1d 6:beaa4be4:gc::gc.7:head [call rgw_gc.rgw_gc_queue_list_entries in=46b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32) v8 -- 0x55c572c621d0 con 0x55c572bee820 2023-10-05T19:28:54.574 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:54.570+0000 7feeec516700 1 -- 172.21.15.167:0/733441895 <== osd.0 v2:172.21.15.118:6800/27690 13 ==== osd_op_reply(56 gc.7 [call out=15b] v0'0 uv4 ondisk = 0) v8 ==== 148+0+15 (crc 0 0 0) 0x7feed4032040 con 0x55c572bee820 2023-10-05T19:28:54.574 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:54.570+0000 7feeef1e2e40 1 -- 172.21.15.167:0/733441895 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:57 6.12 6:4f78e9a2:gc::gc.8:head [call rgw.gc_list in=25b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32) v8 -- 0x55c572c621d0 con 0x55c572bfe3a0 2023-10-05T19:28:54.575 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:54.570+0000 7feeecd17700 1 -- 172.21.15.167:0/733441895 <== osd.2 v2:172.21.15.167:6816/30037 11 ==== osd_op_reply(57 gc.8 [call out=15b] v0'0 uv3 ondisk = 0) v8 ==== 148+0+15 (crc 0 0 0) 0x7feedc04d040 con 0x55c572bfe3a0 2023-10-05T19:28:54.575 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:54.570+0000 7feeef1e2e40 1 -- 172.21.15.167:0/733441895 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:58 6.12 6:4f78e9a2:gc::gc.8:head [call version.read in=11b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32) v8 -- 0x55c572c611c0 con 0x55c572bfe3a0 2023-10-05T19:28:54.576 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:54.570+0000 7feeecd17700 1 -- 172.21.15.167:0/733441895 <== osd.2 v2:172.21.15.167:6816/30037 12 ==== osd_op_reply(58 gc.8 [call out=24b] v0'0 uv3 ondisk = 0) v8 ==== 148+0+24 (crc 0 0 0) 0x7feedc04d040 con 0x55c572bfe3a0 2023-10-05T19:28:54.576 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:54.570+0000 7feeef1e2e40 1 -- 172.21.15.167:0/733441895 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:59 6.12 6:4f78e9a2:gc::gc.8:head [call rgw_gc.rgw_gc_queue_list_entries in=46b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32) v8 -- 0x55c572c621d0 con 0x55c572bfe3a0 2023-10-05T19:28:54.577 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:54.570+0000 7feeecd17700 1 -- 172.21.15.167:0/733441895 <== osd.2 v2:172.21.15.167:6816/30037 13 ==== osd_op_reply(59 gc.8 [call out=15b] v0'0 uv3 ondisk = 0) v8 ==== 148+0+15 (crc 0 0 0) 0x7feedc04d040 con 0x55c572bfe3a0 2023-10-05T19:28:54.577 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:54.570+0000 7feeef1e2e40 1 -- 172.21.15.167:0/733441895 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:60 6.1c 6:3e426232:gc::gc.9:head [call rgw.gc_list in=25b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32) v8 -- 0x55c572c621d0 con 0x55c572bfa290 2023-10-05T19:28:54.578 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:54.570+0000 7feeed518700 1 -- 172.21.15.167:0/733441895 <== osd.3 v2:172.21.15.167:6808/30036 19 ==== osd_op_reply(60 gc.9 [call out=15b] v0'0 uv3 ondisk = 0) v8 ==== 148+0+15 (crc 0 0 0) 0x7feee4052040 con 0x55c572bfa290 2023-10-05T19:28:54.578 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:54.570+0000 7feeef1e2e40 1 -- 172.21.15.167:0/733441895 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:61 6.1c 6:3e426232:gc::gc.9:head [call version.read in=11b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32) v8 -- 0x55c572c611c0 con 0x55c572bfa290 2023-10-05T19:28:54.579 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:54.574+0000 7feeed518700 1 -- 172.21.15.167:0/733441895 <== osd.3 v2:172.21.15.167:6808/30036 20 ==== osd_op_reply(61 gc.9 [call out=24b] v0'0 uv3 ondisk = 0) v8 ==== 148+0+24 (crc 0 0 0) 0x7feee4052040 con 0x55c572bfa290 2023-10-05T19:28:54.579 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:54.574+0000 7feeef1e2e40 1 -- 172.21.15.167:0/733441895 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:62 6.1c 6:3e426232:gc::gc.9:head [call rgw_gc.rgw_gc_queue_list_entries in=46b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32) v8 -- 0x55c572c621d0 con 0x55c572bfa290 2023-10-05T19:28:54.580 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:54.574+0000 7feeed518700 1 -- 172.21.15.167:0/733441895 <== osd.3 v2:172.21.15.167:6808/30036 21 ==== osd_op_reply(62 gc.9 [call out=15b] v0'0 uv3 ondisk = 0) v8 ==== 148+0+15 (crc 0 0 0) 0x7feee4052040 con 0x55c572bfa290 2023-10-05T19:28:54.580 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:54.574+0000 7feeef1e2e40 1 -- 172.21.15.167:0/733441895 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:63 6.3 6:c31f41d7:gc::gc.10:head [call rgw.gc_list in=25b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32) v8 -- 0x55c572c621d0 con 0x55c572bfe3a0 2023-10-05T19:28:54.580 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:54.574+0000 7feeecd17700 1 -- 172.21.15.167:0/733441895 <== osd.2 v2:172.21.15.167:6816/30037 14 ==== osd_op_reply(63 gc.10 [call out=15b] v0'0 uv11 ondisk = 0) v8 ==== 149+0+15 (crc 0 0 0) 0x7feedc04d040 con 0x55c572bfe3a0 2023-10-05T19:28:54.580 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:54.574+0000 7feeef1e2e40 1 -- 172.21.15.167:0/733441895 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:64 6.3 6:c31f41d7:gc::gc.10:head [call version.read in=11b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32) v8 -- 0x55c572c611c0 con 0x55c572bfe3a0 2023-10-05T19:28:54.580 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:54.574+0000 7feeecd17700 1 -- 172.21.15.167:0/733441895 <== osd.2 v2:172.21.15.167:6816/30037 15 ==== osd_op_reply(64 gc.10 [call out=24b] v0'0 uv11 ondisk = 0) v8 ==== 149+0+24 (crc 0 0 0) 0x7feedc04d040 con 0x55c572bfe3a0 2023-10-05T19:28:54.580 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:54.574+0000 7feeef1e2e40 1 -- 172.21.15.167:0/733441895 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:65 6.3 6:c31f41d7:gc::gc.10:head [call rgw_gc.rgw_gc_queue_list_entries in=46b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32) v8 -- 0x55c572c621d0 con 0x55c572bfe3a0 2023-10-05T19:28:54.581 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:54.574+0000 7feeecd17700 1 -- 172.21.15.167:0/733441895 <== osd.2 v2:172.21.15.167:6816/30037 16 ==== osd_op_reply(65 gc.10 [call out=15b] v0'0 uv11 ondisk = 0) v8 ==== 149+0+15 (crc 0 0 0) 0x7feedc04d040 con 0x55c572bfe3a0 2023-10-05T19:28:54.581 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:54.574+0000 7feeef1e2e40 1 -- 172.21.15.167:0/733441895 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:66 6.13 6:c8333aab:gc::gc.11:head [call rgw.gc_list in=25b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32) v8 -- 0x55c572c621d0 con 0x55c572bee820 2023-10-05T19:28:54.582 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:54.578+0000 7feeec516700 1 -- 172.21.15.167:0/733441895 <== osd.0 v2:172.21.15.118:6800/27690 14 ==== osd_op_reply(66 gc.11 [call out=15b] v0'0 uv5 ondisk = 0) v8 ==== 149+0+15 (crc 0 0 0) 0x7feed4032040 con 0x55c572bee820 2023-10-05T19:28:54.582 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:54.578+0000 7feeef1e2e40 1 -- 172.21.15.167:0/733441895 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:67 6.13 6:c8333aab:gc::gc.11:head [call version.read in=11b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32) v8 -- 0x55c572c611c0 con 0x55c572bee820 2023-10-05T19:28:54.583 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:54.578+0000 7feeec516700 1 -- 172.21.15.167:0/733441895 <== osd.0 v2:172.21.15.118:6800/27690 15 ==== osd_op_reply(67 gc.11 [call out=24b] v0'0 uv5 ondisk = 0) v8 ==== 149+0+24 (crc 0 0 0) 0x7feed4032040 con 0x55c572bee820 2023-10-05T19:28:54.583 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:54.578+0000 7feeef1e2e40 1 -- 172.21.15.167:0/733441895 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:68 6.13 6:c8333aab:gc::gc.11:head [call rgw_gc.rgw_gc_queue_list_entries in=46b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32) v8 -- 0x55c572c621d0 con 0x55c572bee820 2023-10-05T19:28:54.584 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:54.578+0000 7feeec516700 1 -- 172.21.15.167:0/733441895 <== osd.0 v2:172.21.15.118:6800/27690 16 ==== osd_op_reply(68 gc.11 [call out=15b] v0'0 uv5 ondisk = 0) v8 ==== 149+0+15 (crc 0 0 0) 0x7feed4032040 con 0x55c572bee820 2023-10-05T19:28:54.584 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:54.578+0000 7feeef1e2e40 1 -- 172.21.15.167:0/733441895 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:69 6.15 6:ab569f81:gc::gc.12:head [call rgw.gc_list in=25b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32) v8 -- 0x55c572c621d0 con 0x55c572bfe3a0 2023-10-05T19:28:54.585 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:54.578+0000 7feeecd17700 1 -- 172.21.15.167:0/733441895 <== osd.2 v2:172.21.15.167:6816/30037 17 ==== osd_op_reply(69 gc.12 [call out=15b] v0'0 uv3 ondisk = 0) v8 ==== 149+0+15 (crc 0 0 0) 0x7feedc04d040 con 0x55c572bfe3a0 2023-10-05T19:28:54.585 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:54.578+0000 7feeef1e2e40 1 -- 172.21.15.167:0/733441895 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:70 6.15 6:ab569f81:gc::gc.12:head [call version.read in=11b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32) v8 -- 0x55c572c611c0 con 0x55c572bfe3a0 2023-10-05T19:28:54.585 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:54.582+0000 7feeecd17700 1 -- 172.21.15.167:0/733441895 <== osd.2 v2:172.21.15.167:6816/30037 18 ==== osd_op_reply(70 gc.12 [call out=24b] v0'0 uv3 ondisk = 0) v8 ==== 149+0+24 (crc 0 0 0) 0x7feedc04d040 con 0x55c572bfe3a0 2023-10-05T19:28:54.585 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:54.582+0000 7feeef1e2e40 1 -- 172.21.15.167:0/733441895 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:71 6.15 6:ab569f81:gc::gc.12:head [call rgw_gc.rgw_gc_queue_list_entries in=46b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32) v8 -- 0x55c572c621d0 con 0x55c572bfe3a0 2023-10-05T19:28:54.586 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:54.582+0000 7feeecd17700 1 -- 172.21.15.167:0/733441895 <== osd.2 v2:172.21.15.167:6816/30037 19 ==== osd_op_reply(71 gc.12 [call out=15b] v0'0 uv3 ondisk = 0) v8 ==== 149+0+15 (crc 0 0 0) 0x7feedc04d040 con 0x55c572bfe3a0 2023-10-05T19:28:54.586 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:54.582+0000 7feeef1e2e40 1 -- 172.21.15.167:0/733441895 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:72 6.0 6:0667bb13:gc::gc.13:head [call rgw.gc_list in=25b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32) v8 -- 0x55c572c621d0 con 0x55c572bee820 2023-10-05T19:28:54.587 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:54.582+0000 7feeec516700 1 -- 172.21.15.167:0/733441895 <== osd.0 v2:172.21.15.118:6800/27690 17 ==== osd_op_reply(72 gc.13 [call out=15b] v0'0 uv6 ondisk = 0) v8 ==== 149+0+15 (crc 0 0 0) 0x7feed4032040 con 0x55c572bee820 2023-10-05T19:28:54.587 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:54.582+0000 7feeef1e2e40 1 -- 172.21.15.167:0/733441895 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:73 6.0 6:0667bb13:gc::gc.13:head [call version.read in=11b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32) v8 -- 0x55c572c611c0 con 0x55c572bee820 2023-10-05T19:28:54.588 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:54.582+0000 7feeec516700 1 -- 172.21.15.167:0/733441895 <== osd.0 v2:172.21.15.118:6800/27690 18 ==== osd_op_reply(73 gc.13 [call out=24b] v0'0 uv6 ondisk = 0) v8 ==== 149+0+24 (crc 0 0 0) 0x7feed4032040 con 0x55c572bee820 2023-10-05T19:28:54.588 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:54.582+0000 7feeef1e2e40 1 -- 172.21.15.167:0/733441895 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:74 6.0 6:0667bb13:gc::gc.13:head [call rgw_gc.rgw_gc_queue_list_entries in=46b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32) v8 -- 0x55c572c621d0 con 0x55c572bee820 2023-10-05T19:28:54.589 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:54.582+0000 7feeec516700 1 -- 172.21.15.167:0/733441895 <== osd.0 v2:172.21.15.118:6800/27690 19 ==== osd_op_reply(74 gc.13 [call out=15b] v0'0 uv6 ondisk = 0) v8 ==== 149+0+15 (crc 0 0 0) 0x7feed4032040 con 0x55c572bee820 2023-10-05T19:28:54.589 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:54.586+0000 7feeef1e2e40 1 -- 172.21.15.167:0/733441895 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:75 6.4 6:227f63b5:gc::gc.14:head [call rgw.gc_list in=25b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32) v8 -- 0x55c572c621d0 con 0x55c572bf2be0 2023-10-05T19:28:54.590 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:54.586+0000 7feeecd17700 1 -- 172.21.15.167:0/733441895 <== osd.1 v2:172.21.15.167:6800/30035 16 ==== osd_op_reply(75 gc.14 [call out=15b] v0'0 uv13 ondisk = 0) v8 ==== 149+0+15 (crc 0 0 0) 0x7feedc04d040 con 0x55c572bf2be0 2023-10-05T19:28:54.590 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:54.586+0000 7feeef1e2e40 1 -- 172.21.15.167:0/733441895 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:76 6.4 6:227f63b5:gc::gc.14:head [call version.read in=11b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32) v8 -- 0x55c572c611c0 con 0x55c572bf2be0 2023-10-05T19:28:54.591 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:54.586+0000 7feeecd17700 1 -- 172.21.15.167:0/733441895 <== osd.1 v2:172.21.15.167:6800/30035 17 ==== osd_op_reply(76 gc.14 [call out=24b] v0'0 uv13 ondisk = 0) v8 ==== 149+0+24 (crc 0 0 0) 0x7feedc04d040 con 0x55c572bf2be0 2023-10-05T19:28:54.591 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:54.586+0000 7feeef1e2e40 1 -- 172.21.15.167:0/733441895 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:77 6.4 6:227f63b5:gc::gc.14:head [call rgw_gc.rgw_gc_queue_list_entries in=46b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32) v8 -- 0x55c572c621d0 con 0x55c572bf2be0 2023-10-05T19:28:54.591 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:54.586+0000 7feeecd17700 1 -- 172.21.15.167:0/733441895 <== osd.1 v2:172.21.15.167:6800/30035 18 ==== osd_op_reply(77 gc.14 [call out=15b] v0'0 uv13 ondisk = 0) v8 ==== 149+0+15 (crc 0 0 0) 0x7feedc04d040 con 0x55c572bf2be0 2023-10-05T19:28:54.592 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:54.586+0000 7feeef1e2e40 1 -- 172.21.15.167:0/733441895 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:78 6.7 6:e07a6548:gc::gc.15:head [call rgw.gc_list in=25b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32) v8 -- 0x55c572c621d0 con 0x55c572bfa290 2023-10-05T19:28:54.592 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:54.586+0000 7feeed518700 1 -- 172.21.15.167:0/733441895 <== osd.3 v2:172.21.15.167:6808/30036 22 ==== osd_op_reply(78 gc.15 [call out=15b] v0'0 uv7 ondisk = 0) v8 ==== 149+0+15 (crc 0 0 0) 0x7feee4052040 con 0x55c572bfa290 2023-10-05T19:28:54.592 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:54.586+0000 7feeef1e2e40 1 -- 172.21.15.167:0/733441895 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:79 6.7 6:e07a6548:gc::gc.15:head [call version.read in=11b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32) v8 -- 0x55c572c611c0 con 0x55c572bfa290 2023-10-05T19:28:54.593 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:54.586+0000 7feeed518700 1 -- 172.21.15.167:0/733441895 <== osd.3 v2:172.21.15.167:6808/30036 23 ==== osd_op_reply(79 gc.15 [call out=24b] v0'0 uv7 ondisk = 0) v8 ==== 149+0+24 (crc 0 0 0) 0x7feee4052040 con 0x55c572bfa290 2023-10-05T19:28:54.593 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:54.586+0000 7feeef1e2e40 1 -- 172.21.15.167:0/733441895 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:80 6.7 6:e07a6548:gc::gc.15:head [call rgw_gc.rgw_gc_queue_list_entries in=46b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32) v8 -- 0x55c572c621d0 con 0x55c572bfa290 2023-10-05T19:28:54.594 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:54.590+0000 7feeed518700 1 -- 172.21.15.167:0/733441895 <== osd.3 v2:172.21.15.167:6808/30036 24 ==== osd_op_reply(80 gc.15 [call out=15b] v0'0 uv7 ondisk = 0) v8 ==== 149+0+15 (crc 0 0 0) 0x7feee4052040 con 0x55c572bfa290 2023-10-05T19:28:54.594 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:54.590+0000 7feeef1e2e40 1 -- 172.21.15.167:0/733441895 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:81 6.3 6:c623b173:gc::gc.16:head [call rgw.gc_list in=25b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32) v8 -- 0x55c572c621d0 con 0x55c572bfe3a0 2023-10-05T19:28:54.595 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:54.590+0000 7feeecd17700 1 -- 172.21.15.167:0/733441895 <== osd.2 v2:172.21.15.167:6816/30037 20 ==== osd_op_reply(81 gc.16 [call out=15b] v0'0 uv13 ondisk = 0) v8 ==== 149+0+15 (crc 0 0 0) 0x7feedc04d040 con 0x55c572bfe3a0 2023-10-05T19:28:54.595 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:54.590+0000 7feeef1e2e40 1 -- 172.21.15.167:0/733441895 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:82 6.3 6:c623b173:gc::gc.16:head [call version.read in=11b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32) v8 -- 0x55c572c611c0 con 0x55c572bfe3a0 2023-10-05T19:28:54.595 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:54.590+0000 7feeecd17700 1 -- 172.21.15.167:0/733441895 <== osd.2 v2:172.21.15.167:6816/30037 21 ==== osd_op_reply(82 gc.16 [call out=24b] v0'0 uv13 ondisk = 0) v8 ==== 149+0+24 (crc 0 0 0) 0x7feedc04d040 con 0x55c572bfe3a0 2023-10-05T19:28:54.596 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:54.590+0000 7feeef1e2e40 1 -- 172.21.15.167:0/733441895 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:83 6.3 6:c623b173:gc::gc.16:head [call rgw_gc.rgw_gc_queue_list_entries in=46b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32) v8 -- 0x55c572c621d0 con 0x55c572bfe3a0 2023-10-05T19:28:54.597 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:54.590+0000 7feeecd17700 1 -- 172.21.15.167:0/733441895 <== osd.2 v2:172.21.15.167:6816/30037 22 ==== osd_op_reply(83 gc.16 [call out=15b] v0'0 uv13 ondisk = 0) v8 ==== 149+0+15 (crc 0 0 0) 0x7feedc04d040 con 0x55c572bfe3a0 2023-10-05T19:28:54.597 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:54.590+0000 7feeef1e2e40 1 -- 172.21.15.167:0/733441895 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:84 6.16 6:6849fd06:gc::gc.17:head [call rgw.gc_list in=25b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32) v8 -- 0x55c572c621d0 con 0x55c572bfa290 2023-10-05T19:28:54.597 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:54.594+0000 7feeed518700 1 -- 172.21.15.167:0/733441895 <== osd.3 v2:172.21.15.167:6808/30036 25 ==== osd_op_reply(84 gc.17 [call out=15b] v0'0 uv7 ondisk = 0) v8 ==== 149+0+15 (crc 0 0 0) 0x7feee4052040 con 0x55c572bfa290 2023-10-05T19:28:54.597 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:54.594+0000 7feeef1e2e40 1 -- 172.21.15.167:0/733441895 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:85 6.16 6:6849fd06:gc::gc.17:head [call version.read in=11b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32) v8 -- 0x55c572c611c0 con 0x55c572bfa290 2023-10-05T19:28:54.598 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:54.594+0000 7feeed518700 1 -- 172.21.15.167:0/733441895 <== osd.3 v2:172.21.15.167:6808/30036 26 ==== osd_op_reply(85 gc.17 [call out=24b] v0'0 uv7 ondisk = 0) v8 ==== 149+0+24 (crc 0 0 0) 0x7feee4052040 con 0x55c572bfa290 2023-10-05T19:28:54.598 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:54.594+0000 7feeef1e2e40 1 -- 172.21.15.167:0/733441895 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:86 6.16 6:6849fd06:gc::gc.17:head [call rgw_gc.rgw_gc_queue_list_entries in=46b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32) v8 -- 0x55c572c621d0 con 0x55c572bfa290 2023-10-05T19:28:54.599 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:54.594+0000 7feeed518700 1 -- 172.21.15.167:0/733441895 <== osd.3 v2:172.21.15.167:6808/30036 27 ==== osd_op_reply(86 gc.17 [call out=15b] v0'0 uv7 ondisk = 0) v8 ==== 149+0+15 (crc 0 0 0) 0x7feee4052040 con 0x55c572bfa290 2023-10-05T19:28:54.599 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:54.594+0000 7feeef1e2e40 1 -- 172.21.15.167:0/733441895 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:87 6.d 6:b019ea83:gc::gc.18:head [call rgw.gc_list in=25b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32) v8 -- 0x55c572c621d0 con 0x55c572bf2be0 2023-10-05T19:28:54.600 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:54.594+0000 7feeecd17700 1 -- 172.21.15.167:0/733441895 <== osd.1 v2:172.21.15.167:6800/30035 19 ==== osd_op_reply(87 gc.18 [call out=15b] v0'0 uv5 ondisk = 0) v8 ==== 149+0+15 (crc 0 0 0) 0x7feedc04d040 con 0x55c572bf2be0 2023-10-05T19:28:54.600 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:54.594+0000 7feeef1e2e40 1 -- 172.21.15.167:0/733441895 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:88 6.d 6:b019ea83:gc::gc.18:head [call version.read in=11b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32) v8 -- 0x55c572c611c0 con 0x55c572bf2be0 2023-10-05T19:28:54.600 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:54.594+0000 7feeecd17700 1 -- 172.21.15.167:0/733441895 <== osd.1 v2:172.21.15.167:6800/30035 20 ==== osd_op_reply(88 gc.18 [call out=24b] v0'0 uv5 ondisk = 0) v8 ==== 149+0+24 (crc 0 0 0) 0x7feedc04d040 con 0x55c572bf2be0 2023-10-05T19:28:54.600 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:54.594+0000 7feeef1e2e40 1 -- 172.21.15.167:0/733441895 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:89 6.d 6:b019ea83:gc::gc.18:head [call rgw_gc.rgw_gc_queue_list_entries in=46b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32) v8 -- 0x55c572c621d0 con 0x55c572bf2be0 2023-10-05T19:28:54.601 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:54.594+0000 7feeecd17700 1 -- 172.21.15.167:0/733441895 <== osd.1 v2:172.21.15.167:6800/30035 21 ==== osd_op_reply(89 gc.18 [call out=15b] v0'0 uv5 ondisk = 0) v8 ==== 149+0+15 (crc 0 0 0) 0x7feedc04d040 con 0x55c572bf2be0 2023-10-05T19:28:54.601 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:54.594+0000 7feeef1e2e40 1 -- 172.21.15.167:0/733441895 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:90 6.4 6:24c8e1ae:gc::gc.19:head [call rgw.gc_list in=25b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32) v8 -- 0x55c572c621d0 con 0x55c572bf2be0 2023-10-05T19:28:54.601 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:54.598+0000 7feeecd17700 1 -- 172.21.15.167:0/733441895 <== osd.1 v2:172.21.15.167:6800/30035 22 ==== osd_op_reply(90 gc.19 [call out=15b] v0'0 uv173 ondisk = 0) v8 ==== 149+0+15 (crc 0 0 0) 0x7feedc04d040 con 0x55c572bf2be0 2023-10-05T19:28:54.602 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:54.598+0000 7feeef1e2e40 1 -- 172.21.15.167:0/733441895 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:91 6.4 6:24c8e1ae:gc::gc.19:head [call version.read in=11b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32) v8 -- 0x55c572c611c0 con 0x55c572bf2be0 2023-10-05T19:28:54.602 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:54.598+0000 7feeecd17700 1 -- 172.21.15.167:0/733441895 <== osd.1 v2:172.21.15.167:6800/30035 23 ==== osd_op_reply(91 gc.19 [call out=24b] v0'0 uv173 ondisk = 0) v8 ==== 149+0+24 (crc 0 0 0) 0x7feedc04d040 con 0x55c572bf2be0 2023-10-05T19:28:54.602 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:54.598+0000 7feeef1e2e40 1 -- 172.21.15.167:0/733441895 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:92 6.4 6:24c8e1ae:gc::gc.19:head [call rgw_gc.rgw_gc_queue_list_entries in=46b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32) v8 -- 0x55c572c621d0 con 0x55c572bf2be0 2023-10-05T19:28:54.603 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:54.598+0000 7feeecd17700 1 -- 172.21.15.167:0/733441895 <== osd.1 v2:172.21.15.167:6800/30035 24 ==== osd_op_reply(92 gc.19 [call out=1701b] v0'0 uv173 ondisk = 0) v8 ==== 149+0+1701 (crc 0 0 0) 0x7feedc04d040 con 0x55c572bf2be0 2023-10-05T19:28:54.603 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:54.598+0000 7feeef1e2e40 1 -- 172.21.15.167:0/733441895 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:93 6.7 6:e622f76a:gc::gc.20:head [call rgw.gc_list in=25b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32) v8 -- 0x55c572c621d0 con 0x55c572bfa290 2023-10-05T19:28:54.604 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:54.598+0000 7feeed518700 1 -- 172.21.15.167:0/733441895 <== osd.3 v2:172.21.15.167:6808/30036 28 ==== osd_op_reply(93 gc.20 [call out=15b] v0'0 uv9 ondisk = 0) v8 ==== 149+0+15 (crc 0 0 0) 0x7feee4052040 con 0x55c572bfa290 2023-10-05T19:28:54.604 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:54.598+0000 7feeef1e2e40 1 -- 172.21.15.167:0/733441895 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:94 6.7 6:e622f76a:gc::gc.20:head [call version.read in=11b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32) v8 -- 0x55c572c611c0 con 0x55c572bfa290 2023-10-05T19:28:54.604 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:54.598+0000 7feeed518700 1 -- 172.21.15.167:0/733441895 <== osd.3 v2:172.21.15.167:6808/30036 29 ==== osd_op_reply(94 gc.20 [call out=24b] v0'0 uv9 ondisk = 0) v8 ==== 149+0+24 (crc 0 0 0) 0x7feee4052040 con 0x55c572bfa290 2023-10-05T19:28:54.604 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:54.598+0000 7feeef1e2e40 1 -- 172.21.15.167:0/733441895 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:95 6.7 6:e622f76a:gc::gc.20:head [call rgw_gc.rgw_gc_queue_list_entries in=46b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32) v8 -- 0x55c572c621d0 con 0x55c572bfa290 2023-10-05T19:28:54.605 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:54.598+0000 7feeed518700 1 -- 172.21.15.167:0/733441895 <== osd.3 v2:172.21.15.167:6808/30036 30 ==== osd_op_reply(95 gc.20 [call out=15b] v0'0 uv9 ondisk = 0) v8 ==== 149+0+15 (crc 0 0 0) 0x7feee4052040 con 0x55c572bfa290 2023-10-05T19:28:54.605 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:54.598+0000 7feeef1e2e40 1 -- 172.21.15.167:0/733441895 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:96 6.a 6:5127e642:gc::gc.21:head [call rgw.gc_list in=25b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32) v8 -- 0x55c572c621d0 con 0x55c572bfe3a0 2023-10-05T19:28:54.606 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:54.602+0000 7feeecd17700 1 -- 172.21.15.167:0/733441895 <== osd.2 v2:172.21.15.167:6816/30037 23 ==== osd_op_reply(96 gc.21 [call out=15b] v0'0 uv7 ondisk = 0) v8 ==== 149+0+15 (crc 0 0 0) 0x7feedc04d040 con 0x55c572bfe3a0 2023-10-05T19:28:54.606 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:54.602+0000 7feeef1e2e40 1 -- 172.21.15.167:0/733441895 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:97 6.a 6:5127e642:gc::gc.21:head [call version.read in=11b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32) v8 -- 0x55c572c611c0 con 0x55c572bfe3a0 2023-10-05T19:28:54.606 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:54.602+0000 7feeecd17700 1 -- 172.21.15.167:0/733441895 <== osd.2 v2:172.21.15.167:6816/30037 24 ==== osd_op_reply(97 gc.21 [call out=24b] v0'0 uv7 ondisk = 0) v8 ==== 149+0+24 (crc 0 0 0) 0x7feedc04d040 con 0x55c572bfe3a0 2023-10-05T19:28:54.606 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:54.602+0000 7feeef1e2e40 1 -- 172.21.15.167:0/733441895 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:98 6.a 6:5127e642:gc::gc.21:head [call rgw_gc.rgw_gc_queue_list_entries in=46b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32) v8 -- 0x55c572c621d0 con 0x55c572bfe3a0 2023-10-05T19:28:54.607 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:54.602+0000 7feeecd17700 1 -- 172.21.15.167:0/733441895 <== osd.2 v2:172.21.15.167:6816/30037 25 ==== osd_op_reply(98 gc.21 [call out=15b] v0'0 uv7 ondisk = 0) v8 ==== 149+0+15 (crc 0 0 0) 0x7feedc04d040 con 0x55c572bfe3a0 2023-10-05T19:28:54.607 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:54.602+0000 7feeef1e2e40 1 -- 172.21.15.167:0/733441895 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:99 6.11 6:8eb99c53:gc::gc.22:head [call rgw.gc_list in=25b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32) v8 -- 0x55c572c621d0 con 0x55c572bfa290 2023-10-05T19:28:54.608 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:54.602+0000 7feeed518700 1 -- 172.21.15.167:0/733441895 <== osd.3 v2:172.21.15.167:6808/30036 31 ==== osd_op_reply(99 gc.22 [call out=15b] v0'0 uv3 ondisk = 0) v8 ==== 149+0+15 (crc 0 0 0) 0x7feee4052040 con 0x55c572bfa290 2023-10-05T19:28:54.608 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:54.602+0000 7feeef1e2e40 1 -- 172.21.15.167:0/733441895 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:100 6.11 6:8eb99c53:gc::gc.22:head [call version.read in=11b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32) v8 -- 0x55c572c611c0 con 0x55c572bfa290 2023-10-05T19:28:54.608 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:54.602+0000 7feeed518700 1 -- 172.21.15.167:0/733441895 <== osd.3 v2:172.21.15.167:6808/30036 32 ==== osd_op_reply(100 gc.22 [call out=24b] v0'0 uv3 ondisk = 0) v8 ==== 149+0+24 (crc 0 0 0) 0x7feee4052040 con 0x55c572bfa290 2023-10-05T19:28:54.608 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:54.602+0000 7feeef1e2e40 1 -- 172.21.15.167:0/733441895 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:101 6.11 6:8eb99c53:gc::gc.22:head [call rgw_gc.rgw_gc_queue_list_entries in=46b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32) v8 -- 0x55c572c621d0 con 0x55c572bfa290 2023-10-05T19:28:54.609 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:54.602+0000 7feeed518700 1 -- 172.21.15.167:0/733441895 <== osd.3 v2:172.21.15.167:6808/30036 33 ==== osd_op_reply(101 gc.22 [call out=15b] v0'0 uv3 ondisk = 0) v8 ==== 149+0+15 (crc 0 0 0) 0x7feee4052040 con 0x55c572bfa290 2023-10-05T19:28:54.609 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:54.602+0000 7feeef1e2e40 1 -- 172.21.15.167:0/733441895 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:102 6.8 6:17d7e8ed:gc::gc.23:head [call rgw.gc_list in=25b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32) v8 -- 0x55c572c621d0 con 0x55c572bfa290 2023-10-05T19:28:54.610 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:54.606+0000 7feeed518700 1 -- 172.21.15.167:0/733441895 <== osd.3 v2:172.21.15.167:6808/30036 34 ==== osd_op_reply(102 gc.23 [call out=15b] v0'0 uv5 ondisk = 0) v8 ==== 149+0+15 (crc 0 0 0) 0x7feee4052040 con 0x55c572bfa290 2023-10-05T19:28:54.610 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:54.606+0000 7feeef1e2e40 1 -- 172.21.15.167:0/733441895 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:103 6.8 6:17d7e8ed:gc::gc.23:head [call version.read in=11b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32) v8 -- 0x55c572c611c0 con 0x55c572bfa290 2023-10-05T19:28:54.610 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:54.606+0000 7feeed518700 1 -- 172.21.15.167:0/733441895 <== osd.3 v2:172.21.15.167:6808/30036 35 ==== osd_op_reply(103 gc.23 [call out=24b] v0'0 uv5 ondisk = 0) v8 ==== 149+0+24 (crc 0 0 0) 0x7feee4052040 con 0x55c572bfa290 2023-10-05T19:28:54.611 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:54.606+0000 7feeef1e2e40 1 -- 172.21.15.167:0/733441895 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:104 6.8 6:17d7e8ed:gc::gc.23:head [call rgw_gc.rgw_gc_queue_list_entries in=46b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32) v8 -- 0x55c572c621d0 con 0x55c572bfa290 2023-10-05T19:28:54.611 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:54.606+0000 7feeed518700 1 -- 172.21.15.167:0/733441895 <== osd.3 v2:172.21.15.167:6808/30036 36 ==== osd_op_reply(104 gc.23 [call out=15b] v0'0 uv5 ondisk = 0) v8 ==== 149+0+15 (crc 0 0 0) 0x7feee4052040 con 0x55c572bfa290 2023-10-05T19:28:54.611 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:54.606+0000 7feeef1e2e40 1 -- 172.21.15.167:0/733441895 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:105 6.1b 6:db9d130d:gc::gc.24:head [call rgw.gc_list in=25b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32) v8 -- 0x55c572c621d0 con 0x55c572bfa290 2023-10-05T19:28:54.611 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:54.606+0000 7feeed518700 1 -- 172.21.15.167:0/733441895 <== osd.3 v2:172.21.15.167:6808/30036 37 ==== osd_op_reply(105 gc.24 [call out=15b] v0'0 uv3 ondisk = 0) v8 ==== 149+0+15 (crc 0 0 0) 0x7feee4052040 con 0x55c572bfa290 2023-10-05T19:28:54.612 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:54.606+0000 7feeef1e2e40 1 -- 172.21.15.167:0/733441895 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:106 6.1b 6:db9d130d:gc::gc.24:head [call version.read in=11b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32) v8 -- 0x55c572c611c0 con 0x55c572bfa290 2023-10-05T19:28:54.612 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:54.606+0000 7feeed518700 1 -- 172.21.15.167:0/733441895 <== osd.3 v2:172.21.15.167:6808/30036 38 ==== osd_op_reply(106 gc.24 [call out=24b] v0'0 uv3 ondisk = 0) v8 ==== 149+0+24 (crc 0 0 0) 0x7feee4052040 con 0x55c572bfa290 2023-10-05T19:28:54.612 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:54.606+0000 7feeef1e2e40 1 -- 172.21.15.167:0/733441895 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:107 6.1b 6:db9d130d:gc::gc.24:head [call rgw_gc.rgw_gc_queue_list_entries in=46b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32) v8 -- 0x55c572c621d0 con 0x55c572bfa290 2023-10-05T19:28:54.612 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:54.606+0000 7feeed518700 1 -- 172.21.15.167:0/733441895 <== osd.3 v2:172.21.15.167:6808/30036 39 ==== osd_op_reply(107 gc.24 [call out=15b] v0'0 uv3 ondisk = 0) v8 ==== 149+0+15 (crc 0 0 0) 0x7feee4052040 con 0x55c572bfa290 2023-10-05T19:28:54.613 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:54.606+0000 7feeef1e2e40 1 -- 172.21.15.167:0/733441895 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:108 6.9 6:921ecdb1:gc::gc.25:head [call rgw.gc_list in=25b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32) v8 -- 0x55c572c621d0 con 0x55c572bfa290 2023-10-05T19:28:54.613 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:54.606+0000 7feeed518700 1 -- 172.21.15.167:0/733441895 <== osd.3 v2:172.21.15.167:6808/30036 40 ==== osd_op_reply(108 gc.25 [call out=15b] v0'0 uv10 ondisk = 0) v8 ==== 149+0+15 (crc 0 0 0) 0x7feee4052040 con 0x55c572bfa290 2023-10-05T19:28:54.613 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:54.606+0000 7feeef1e2e40 1 -- 172.21.15.167:0/733441895 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:109 6.9 6:921ecdb1:gc::gc.25:head [call version.read in=11b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32) v8 -- 0x55c572c611c0 con 0x55c572bfa290 2023-10-05T19:28:54.613 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:54.610+0000 7feeed518700 1 -- 172.21.15.167:0/733441895 <== osd.3 v2:172.21.15.167:6808/30036 41 ==== osd_op_reply(109 gc.25 [call out=24b] v0'0 uv10 ondisk = 0) v8 ==== 149+0+24 (crc 0 0 0) 0x7feee4052040 con 0x55c572bfa290 2023-10-05T19:28:54.614 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:54.610+0000 7feeef1e2e40 1 -- 172.21.15.167:0/733441895 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:110 6.9 6:921ecdb1:gc::gc.25:head [call rgw_gc.rgw_gc_queue_list_entries in=46b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32) v8 -- 0x55c572c621d0 con 0x55c572bfa290 2023-10-05T19:28:54.614 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:54.610+0000 7feeed518700 1 -- 172.21.15.167:0/733441895 <== osd.3 v2:172.21.15.167:6808/30036 42 ==== osd_op_reply(110 gc.25 [call out=15b] v0'0 uv10 ondisk = 0) v8 ==== 149+0+15 (crc 0 0 0) 0x7feee4052040 con 0x55c572bfa290 2023-10-05T19:28:54.614 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:54.610+0000 7feeef1e2e40 1 -- 172.21.15.167:0/733441895 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:111 6.1f 6:fde4dd55:gc::gc.26:head [call rgw.gc_list in=25b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32) v8 -- 0x55c572c621d0 con 0x55c572bee820 2023-10-05T19:28:54.615 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:54.610+0000 7feeec516700 1 -- 172.21.15.167:0/733441895 <== osd.0 v2:172.21.15.118:6800/27690 20 ==== osd_op_reply(111 gc.26 [call out=15b] v0'0 uv5 ondisk = 0) v8 ==== 149+0+15 (crc 0 0 0) 0x7feed4032040 con 0x55c572bee820 2023-10-05T19:28:54.615 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:54.610+0000 7feeef1e2e40 1 -- 172.21.15.167:0/733441895 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:112 6.1f 6:fde4dd55:gc::gc.26:head [call version.read in=11b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32) v8 -- 0x55c572c611c0 con 0x55c572bee820 2023-10-05T19:28:54.616 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:54.610+0000 7feeec516700 1 -- 172.21.15.167:0/733441895 <== osd.0 v2:172.21.15.118:6800/27690 21 ==== osd_op_reply(112 gc.26 [call out=24b] v0'0 uv5 ondisk = 0) v8 ==== 149+0+24 (crc 0 0 0) 0x7feed4032040 con 0x55c572bee820 2023-10-05T19:28:54.616 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:54.610+0000 7feeef1e2e40 1 -- 172.21.15.167:0/733441895 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:113 6.1f 6:fde4dd55:gc::gc.26:head [call rgw_gc.rgw_gc_queue_list_entries in=46b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32) v8 -- 0x55c572c621d0 con 0x55c572bee820 2023-10-05T19:28:54.617 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:54.610+0000 7feeec516700 1 -- 172.21.15.167:0/733441895 <== osd.0 v2:172.21.15.118:6800/27690 22 ==== osd_op_reply(113 gc.26 [call out=15b] v0'0 uv5 ondisk = 0) v8 ==== 149+0+15 (crc 0 0 0) 0x7feed4032040 con 0x55c572bee820 2023-10-05T19:28:54.617 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:54.610+0000 7feeef1e2e40 1 -- 172.21.15.167:0/733441895 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:114 6.1a 6:5909e671:gc::gc.27:head [call rgw.gc_list in=25b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32) v8 -- 0x55c572c621d0 con 0x55c572bfa290 2023-10-05T19:28:54.617 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:54.610+0000 7feeed518700 1 -- 172.21.15.167:0/733441895 <== osd.3 v2:172.21.15.167:6808/30036 43 ==== osd_op_reply(114 gc.27 [call out=15b] v0'0 uv7 ondisk = 0) v8 ==== 149+0+15 (crc 0 0 0) 0x7feee4052040 con 0x55c572bfa290 2023-10-05T19:28:54.617 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:54.610+0000 7feeef1e2e40 1 -- 172.21.15.167:0/733441895 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:115 6.1a 6:5909e671:gc::gc.27:head [call version.read in=11b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32) v8 -- 0x55c572c611c0 con 0x55c572bfa290 2023-10-05T19:28:54.618 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:54.614+0000 7feeed518700 1 -- 172.21.15.167:0/733441895 <== osd.3 v2:172.21.15.167:6808/30036 44 ==== osd_op_reply(115 gc.27 [call out=24b] v0'0 uv7 ondisk = 0) v8 ==== 149+0+24 (crc 0 0 0) 0x7feee4052040 con 0x55c572bfa290 2023-10-05T19:28:54.618 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:54.614+0000 7feeef1e2e40 1 -- 172.21.15.167:0/733441895 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:116 6.1a 6:5909e671:gc::gc.27:head [call rgw_gc.rgw_gc_queue_list_entries in=46b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32) v8 -- 0x55c572c621d0 con 0x55c572bfa290 2023-10-05T19:28:54.618 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:54.614+0000 7feeed518700 1 -- 172.21.15.167:0/733441895 <== osd.3 v2:172.21.15.167:6808/30036 45 ==== osd_op_reply(116 gc.27 [call out=15b] v0'0 uv7 ondisk = 0) v8 ==== 149+0+15 (crc 0 0 0) 0x7feee4052040 con 0x55c572bfa290 2023-10-05T19:28:54.618 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:54.614+0000 7feeef1e2e40 1 -- 172.21.15.167:0/733441895 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:117 6.1d 6:b88e3271:gc::gc.28:head [call rgw.gc_list in=25b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32) v8 -- 0x55c572c621d0 con 0x55c572bee820 2023-10-05T19:28:54.619 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:54.614+0000 7feeec516700 1 -- 172.21.15.167:0/733441895 <== osd.0 v2:172.21.15.118:6800/27690 23 ==== osd_op_reply(117 gc.28 [call out=15b] v0'0 uv8 ondisk = 0) v8 ==== 149+0+15 (crc 0 0 0) 0x7feed4032040 con 0x55c572bee820 2023-10-05T19:28:54.619 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:54.614+0000 7feeef1e2e40 1 -- 172.21.15.167:0/733441895 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:118 6.1d 6:b88e3271:gc::gc.28:head [call version.read in=11b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32) v8 -- 0x55c572c611c0 con 0x55c572bee820 2023-10-05T19:28:54.620 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:54.614+0000 7feeec516700 1 -- 172.21.15.167:0/733441895 <== osd.0 v2:172.21.15.118:6800/27690 24 ==== osd_op_reply(118 gc.28 [call out=24b] v0'0 uv8 ondisk = 0) v8 ==== 149+0+24 (crc 0 0 0) 0x7feed4032040 con 0x55c572bee820 2023-10-05T19:28:54.620 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:54.614+0000 7feeef1e2e40 1 -- 172.21.15.167:0/733441895 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:119 6.1d 6:b88e3271:gc::gc.28:head [call rgw_gc.rgw_gc_queue_list_entries in=46b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32) v8 -- 0x55c572c621d0 con 0x55c572bee820 2023-10-05T19:28:54.621 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:54.614+0000 7feeec516700 1 -- 172.21.15.167:0/733441895 <== osd.0 v2:172.21.15.118:6800/27690 25 ==== osd_op_reply(119 gc.28 [call out=15b] v0'0 uv8 ondisk = 0) v8 ==== 149+0+15 (crc 0 0 0) 0x7feed4032040 con 0x55c572bee820 2023-10-05T19:28:54.621 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:54.614+0000 7feeef1e2e40 1 -- 172.21.15.167:0/733441895 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:120 6.7 6:e47c9322:gc::gc.29:head [call rgw.gc_list in=25b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32) v8 -- 0x55c572c621d0 con 0x55c572bfa290 2023-10-05T19:28:54.622 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:54.618+0000 7feeed518700 1 -- 172.21.15.167:0/733441895 <== osd.3 v2:172.21.15.167:6808/30036 46 ==== osd_op_reply(120 gc.29 [call out=15b] v0'0 uv11 ondisk = 0) v8 ==== 149+0+15 (crc 0 0 0) 0x7feee4052040 con 0x55c572bfa290 2023-10-05T19:28:54.622 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:54.618+0000 7feeef1e2e40 1 -- 172.21.15.167:0/733441895 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:121 6.7 6:e47c9322:gc::gc.29:head [call version.read in=11b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32) v8 -- 0x55c572c611c0 con 0x55c572bfa290 2023-10-05T19:28:54.622 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:54.618+0000 7feeed518700 1 -- 172.21.15.167:0/733441895 <== osd.3 v2:172.21.15.167:6808/30036 47 ==== osd_op_reply(121 gc.29 [call out=24b] v0'0 uv11 ondisk = 0) v8 ==== 149+0+24 (crc 0 0 0) 0x7feee4052040 con 0x55c572bfa290 2023-10-05T19:28:54.622 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:54.618+0000 7feeef1e2e40 1 -- 172.21.15.167:0/733441895 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:122 6.7 6:e47c9322:gc::gc.29:head [call rgw_gc.rgw_gc_queue_list_entries in=46b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32) v8 -- 0x55c572c621d0 con 0x55c572bfa290 2023-10-05T19:28:54.623 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:54.618+0000 7feeed518700 1 -- 172.21.15.167:0/733441895 <== osd.3 v2:172.21.15.167:6808/30036 48 ==== osd_op_reply(122 gc.29 [call out=15b] v0'0 uv11 ondisk = 0) v8 ==== 149+0+15 (crc 0 0 0) 0x7feee4052040 con 0x55c572bfa290 2023-10-05T19:28:54.623 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:54.618+0000 7feeef1e2e40 1 -- 172.21.15.167:0/733441895 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:123 6.0 6:013d6c33:gc::gc.30:head [call rgw.gc_list in=25b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32) v8 -- 0x55c572c621d0 con 0x55c572bee820 2023-10-05T19:28:54.624 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:54.618+0000 7feeec516700 1 -- 172.21.15.167:0/733441895 <== osd.0 v2:172.21.15.118:6800/27690 26 ==== osd_op_reply(123 gc.30 [call out=15b] v0'0 uv12 ondisk = 0) v8 ==== 149+0+15 (crc 0 0 0) 0x7feed4032040 con 0x55c572bee820 2023-10-05T19:28:54.624 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:54.618+0000 7feeef1e2e40 1 -- 172.21.15.167:0/733441895 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:124 6.0 6:013d6c33:gc::gc.30:head [call version.read in=11b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32) v8 -- 0x55c572c611c0 con 0x55c572bee820 2023-10-05T19:28:54.625 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:54.618+0000 7feeec516700 1 -- 172.21.15.167:0/733441895 <== osd.0 v2:172.21.15.118:6800/27690 27 ==== osd_op_reply(124 gc.30 [call out=24b] v0'0 uv12 ondisk = 0) v8 ==== 149+0+24 (crc 0 0 0) 0x7feed4032040 con 0x55c572bee820 2023-10-05T19:28:54.625 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:54.618+0000 7feeef1e2e40 1 -- 172.21.15.167:0/733441895 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:125 6.0 6:013d6c33:gc::gc.30:head [call rgw_gc.rgw_gc_queue_list_entries in=46b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32) v8 -- 0x55c572c621d0 con 0x55c572bee820 2023-10-05T19:28:54.626 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:54.622+0000 7feeec516700 1 -- 172.21.15.167:0/733441895 <== osd.0 v2:172.21.15.118:6800/27690 28 ==== osd_op_reply(125 gc.30 [call out=15b] v0'0 uv12 ondisk = 0) v8 ==== 149+0+15 (crc 0 0 0) 0x7feed4032040 con 0x55c572bee820 2023-10-05T19:28:54.626 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:54.622+0000 7feeef1e2e40 1 -- 172.21.15.167:0/733441895 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:126 6.f 6:f24cb4a8:gc::gc.31:head [call rgw.gc_list in=25b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32) v8 -- 0x55c572c621d0 con 0x55c572bfe3a0 2023-10-05T19:28:54.627 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:54.622+0000 7feeecd17700 1 -- 172.21.15.167:0/733441895 <== osd.2 v2:172.21.15.167:6816/30037 26 ==== osd_op_reply(126 gc.31 [call out=15b] v0'0 uv5 ondisk = 0) v8 ==== 149+0+15 (crc 0 0 0) 0x7feedc04d040 con 0x55c572bfe3a0 2023-10-05T19:28:54.627 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:54.622+0000 7feeef1e2e40 1 -- 172.21.15.167:0/733441895 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:127 6.f 6:f24cb4a8:gc::gc.31:head [call version.read in=11b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32) v8 -- 0x55c572c611c0 con 0x55c572bfe3a0 2023-10-05T19:28:54.627 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:54.622+0000 7feeecd17700 1 -- 172.21.15.167:0/733441895 <== osd.2 v2:172.21.15.167:6816/30037 27 ==== osd_op_reply(127 gc.31 [call out=24b] v0'0 uv5 ondisk = 0) v8 ==== 149+0+24 (crc 0 0 0) 0x7feedc04d040 con 0x55c572bfe3a0 2023-10-05T19:28:54.628 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:54.622+0000 7feeef1e2e40 1 -- 172.21.15.167:0/733441895 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:128 6.f 6:f24cb4a8:gc::gc.31:head [call rgw_gc.rgw_gc_queue_list_entries in=46b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32) v8 -- 0x55c572c621d0 con 0x55c572bfe3a0 2023-10-05T19:28:54.628 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:54.622+0000 7feeecd17700 1 -- 172.21.15.167:0/733441895 <== osd.2 v2:172.21.15.167:6816/30037 28 ==== osd_op_reply(128 gc.31 [call out=15b] v0'0 uv5 ondisk = 0) v8 ==== 149+0+15 (crc 0 0 0) 0x7feedc04d040 con 0x55c572bfe3a0 2023-10-05T19:28:54.629 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:54.622+0000 7feeef1e2e40 1 -- 172.21.15.167:0/24894888 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:102 7.1f 7:f95f44c2:::notify.0:head [watch unwatch cookie 94306521902464] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e32) v8 -- 0x55c572ba8800 con 0x7feedc02d020 2023-10-05T19:28:54.632 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:54.626+0000 7feeec516700 1 -- 172.21.15.167:0/24894888 <== osd.2 v2:172.21.15.167:6816/30037 17 ==== osd_op_reply(102 notify.0 [watch unwatch cookie 94306521902464] v32'191 uv189 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7feed4032040 con 0x7feedc02d020 2023-10-05T19:28:54.632 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:54.626+0000 7feeef1e2e40 20 remove_watcher() i=0 2023-10-05T19:28:54.632 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:54.626+0000 7feeef1e2e40 2 removed watcher, disabling cache 2023-10-05T19:28:54.632 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:54.626+0000 7feeef1e2e40 1 -- 172.21.15.167:0/24894888 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:103 7.0 7:05bf5b68:::notify.1:head [watch unwatch cookie 94306521907344] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e32) v8 -- 0x55c572ba8800 con 0x55c572b972f0 2023-10-05T19:28:54.635 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:54.630+0000 7feeed518700 1 -- 172.21.15.167:0/24894888 <== osd.1 v2:172.21.15.167:6800/30035 35 ==== osd_op_reply(103 notify.1 [watch unwatch cookie 94306521907344] v32'191 uv189 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7feee4052040 con 0x55c572b972f0 2023-10-05T19:28:54.636 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:54.630+0000 7feeef1e2e40 20 remove_watcher() i=1 2023-10-05T19:28:54.636 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:54.630+0000 7feeef1e2e40 1 -- 172.21.15.167:0/24894888 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:104 7.15 7:a93a5511:::notify.2:head [watch unwatch cookie 94306521909344] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e32) v8 -- 0x55c572ba8800 con 0x55c572bca340 2023-10-05T19:28:54.639 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:54.634+0000 7feeecd17700 1 -- 172.21.15.167:0/24894888 <== osd.3 v2:172.21.15.167:6808/30036 24 ==== osd_op_reply(104 notify.2 [watch unwatch cookie 94306521909344] v32'191 uv189 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7feedc04d040 con 0x55c572bca340 2023-10-05T19:28:54.639 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:54.634+0000 7feeef1e2e40 20 remove_watcher() i=2 2023-10-05T19:28:54.639 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:54.634+0000 7feeef1e2e40 1 -- 172.21.15.167:0/24894888 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:105 7.e 7:7759931f:::notify.3:head [watch unwatch cookie 94306521913760] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e32) v8 -- 0x55c572ba8800 con 0x7feedc02d020 2023-10-05T19:28:54.642 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:54.638+0000 7feeec516700 1 -- 172.21.15.167:0/24894888 <== osd.2 v2:172.21.15.167:6816/30037 18 ==== osd_op_reply(105 notify.3 [watch unwatch cookie 94306521913760] v32'191 uv189 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7feed4032040 con 0x7feedc02d020 2023-10-05T19:28:54.642 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:54.638+0000 7feeef1e2e40 20 remove_watcher() i=3 2023-10-05T19:28:54.642 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:54.638+0000 7feeef1e2e40 1 -- 172.21.15.167:0/24894888 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:106 7.d 7:b4812045:::notify.4:head [watch unwatch cookie 94306521912496] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e32) v8 -- 0x55c572ba8800 con 0x55c572b972f0 2023-10-05T19:28:54.645 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:54.638+0000 7feeed518700 1 -- 172.21.15.167:0/24894888 <== osd.1 v2:172.21.15.167:6800/30035 36 ==== osd_op_reply(106 notify.4 [watch unwatch cookie 94306521912496] v32'191 uv189 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7feee4052040 con 0x55c572b972f0 2023-10-05T19:28:54.645 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:54.638+0000 7feeef1e2e40 20 remove_watcher() i=4 2023-10-05T19:28:54.645 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:54.638+0000 7feeef1e2e40 1 -- 172.21.15.167:0/24894888 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:107 7.3 7:c609908c:::notify.5:head [watch unwatch cookie 94306521918896] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e32) v8 -- 0x55c572ba8800 con 0x7feed4007030 2023-10-05T19:28:54.648 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:54.642+0000 7feeed518700 1 -- 172.21.15.167:0/24894888 <== osd.0 v2:172.21.15.118:6800/27690 29 ==== osd_op_reply(107 notify.5 [watch unwatch cookie 94306521918896] v32'191 uv189 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7feee4052040 con 0x7feed4007030 2023-10-05T19:28:54.648 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:54.642+0000 7feeef1e2e40 20 remove_watcher() i=5 2023-10-05T19:28:54.648 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:54.642+0000 7feeef1e2e40 1 -- 172.21.15.167:0/24894888 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:108 7.14 7:2b04a3e9:::notify.6:head [watch unwatch cookie 94306521921168] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e32) v8 -- 0x55c572ba8800 con 0x7feed4007030 2023-10-05T19:28:54.651 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:54.646+0000 7feeed518700 1 -- 172.21.15.167:0/24894888 <== osd.0 v2:172.21.15.118:6800/27690 30 ==== osd_op_reply(108 notify.6 [watch unwatch cookie 94306521921168] v32'191 uv189 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7feee4052040 con 0x7feed4007030 2023-10-05T19:28:54.651 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:54.646+0000 7feeef1e2e40 20 remove_watcher() i=6 2023-10-05T19:28:54.651 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:54.646+0000 7feeef1e2e40 1 -- 172.21.15.167:0/24894888 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:109 7.9 7:93e5b521:::notify.7:head [watch unwatch cookie 94306521922832] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e32) v8 -- 0x55c572ba8800 con 0x7feed4007030 2023-10-05T19:28:54.654 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:54.650+0000 7feeed518700 1 -- 172.21.15.167:0/24894888 <== osd.0 v2:172.21.15.118:6800/27690 31 ==== osd_op_reply(109 notify.7 [watch unwatch cookie 94306521922832] v32'191 uv189 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7feee4052040 con 0x7feed4007030 2023-10-05T19:28:54.654 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:54.650+0000 7feeef1e2e40 20 remove_watcher() i=7 2023-10-05T19:28:54.656 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:54.650+0000 7feebe7e4700 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2023-10-05T19:28:54.656 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:54.650+0000 7feeef1e2e40 1 -- 172.21.15.167:0/24894888 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:110 6.4 6:22d26bf9:::data_loggenerations_metadata:head [watch unwatch cookie 94306521926464] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e32) v8 -- 0x55c572ba8800 con 0x55c572b972f0 2023-10-05T19:28:54.660 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:54.654+0000 7feeed518700 1 -- 172.21.15.167:0/24894888 <== osd.1 v2:172.21.15.167:6800/30035 37 ==== osd_op_reply(110 data_loggenerations_metadata [watch unwatch cookie 94306521926464] v32'178 uv1 ondisk = 0) v8 ==== 172+0+0 (crc 0 0 0) 0x7feee4052040 con 0x55c572b972f0 2023-10-05T19:28:54.660 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:54.654+0000 7feeef1e2e40 1 -- 172.21.15.167:0/24894888 >> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] conn(0x7feed4007030 msgr2=0x7feed401c920 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-05T19:28:54.660 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:54.654+0000 7feeef1e2e40 1 --2- 172.21.15.167:0/24894888 >> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] conn(0x7feed4007030 0x7feed401c920 crc :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-05T19:28:54.660 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:54.654+0000 7feeef1e2e40 1 -- 172.21.15.167:0/24894888 >> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] conn(0x55c572b972f0 msgr2=0x55c572bc6970 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-05T19:28:54.660 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:54.654+0000 7feeef1e2e40 1 --2- 172.21.15.167:0/24894888 >> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] conn(0x55c572b972f0 0x55c572bc6970 crc :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-05T19:28:54.661 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:54.654+0000 7feeef1e2e40 1 -- 172.21.15.167:0/24894888 >> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] conn(0x7feedc02d020 msgr2=0x7feedc03d8d0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-05T19:28:54.661 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:54.654+0000 7feeef1e2e40 1 --2- 172.21.15.167:0/24894888 >> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] conn(0x7feedc02d020 0x7feedc03d8d0 crc :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-05T19:28:54.661 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:54.654+0000 7feeef1e2e40 1 -- 172.21.15.167:0/24894888 >> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] conn(0x55c572bca340 msgr2=0x55c572bcc720 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-05T19:28:54.661 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:54.654+0000 7feeef1e2e40 1 --2- 172.21.15.167:0/24894888 >> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] conn(0x55c572bca340 0x55c572bcc720 crc :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-05T19:28:54.661 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:54.654+0000 7feeef1e2e40 1 -- 172.21.15.167:0/24894888 >> [v2:172.21.15.167:6824/29889,v1:172.21.15.167:6825/29889] conn(0x7feecc030c50 msgr2=0x7feecc033100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-05T19:28:54.661 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:54.654+0000 7feeef1e2e40 1 --2- 172.21.15.167:0/24894888 >> [v2:172.21.15.167:6824/29889,v1:172.21.15.167:6825/29889] conn(0x7feecc030c50 0x7feecc033100 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7feed4005a90 tx=0x7feed4015000 comp rx=0 tx=0).stop 2023-10-05T19:28:54.661 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:54.654+0000 7feeed518700 1 -- 172.21.15.167:0/24894888 reap_dead start 2023-10-05T19:28:54.661 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:54.654+0000 7feeef1e2e40 1 -- 172.21.15.167:0/24894888 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55c572b073d0 msgr2=0x55c572bbaf30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-05T19:28:54.661 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:54.654+0000 7feeef1e2e40 1 --2- 172.21.15.167:0/24894888 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55c572b073d0 0x55c572bbaf30 secure :-1 s=READY pgs=515 cs=0 l=1 rev1=1 crypto rx=0x55c572bae600 tx=0x7feedc016000 comp rx=0 tx=0).stop 2023-10-05T19:28:54.661 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:54.654+0000 7feeef1e2e40 1 -- 172.21.15.167:0/24894888 shutdown_connections 2023-10-05T19:28:54.661 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:54.654+0000 7feeef1e2e40 1 --2- 172.21.15.167:0/24894888 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55c572b073d0 0x55c572bbaf30 secure :-1 s=CLOSED pgs=515 cs=0 l=1 rev1=1 crypto rx=0x55c572bae600 tx=0x7feedc016000 comp rx=0 tx=0).stop 2023-10-05T19:28:54.662 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:54.654+0000 7feeef1e2e40 1 -- 172.21.15.167:0/24894888 >> 172.21.15.167:0/24894888 conn(0x55c572b8d720 msgr2=0x55c572bac560 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-05T19:28:54.662 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:54.654+0000 7feeef1e2e40 1 -- 172.21.15.167:0/24894888 shutdown_connections 2023-10-05T19:28:54.662 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:54.654+0000 7feeef1e2e40 1 -- 172.21.15.167:0/24894888 wait complete. 2023-10-05T19:28:54.663 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:54.654+0000 7feeef1e2e40 1 -- 172.21.15.167:0/733441895 >> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] conn(0x55c572bee820 msgr2=0x55c572bf8d00 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-05T19:28:54.663 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:54.654+0000 7feeef1e2e40 1 --2- 172.21.15.167:0/733441895 >> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] conn(0x55c572bee820 0x55c572bf8d00 crc :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-05T19:28:54.663 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:54.654+0000 7feeef1e2e40 1 -- 172.21.15.167:0/733441895 >> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] conn(0x55c572bf2be0 msgr2=0x55c572bf4ff0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-05T19:28:54.663 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:54.654+0000 7feeef1e2e40 1 --2- 172.21.15.167:0/733441895 >> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] conn(0x55c572bf2be0 0x55c572bf4ff0 crc :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-05T19:28:54.664 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:54.654+0000 7feeef1e2e40 1 -- 172.21.15.167:0/733441895 >> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] conn(0x55c572bfe3a0 msgr2=0x55c572c00780 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-05T19:28:54.664 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:54.654+0000 7feeef1e2e40 1 --2- 172.21.15.167:0/733441895 >> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] conn(0x55c572bfe3a0 0x55c572c00780 crc :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-05T19:28:54.664 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:54.654+0000 7feeef1e2e40 1 -- 172.21.15.167:0/733441895 >> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] conn(0x55c572bfa290 msgr2=0x55c572bfc670 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-05T19:28:54.664 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:54.654+0000 7feeef1e2e40 1 --2- 172.21.15.167:0/733441895 >> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] conn(0x55c572bfa290 0x55c572bfc670 crc :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-05T19:28:54.664 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:54.658+0000 7feeef1e2e40 1 -- 172.21.15.167:0/733441895 >> [v2:172.21.15.167:6824/29889,v1:172.21.15.167:6825/29889] conn(0x7fee34030c50 msgr2=0x7fee34033100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-05T19:28:54.664 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:54.658+0000 7feeef1e2e40 1 --2- 172.21.15.167:0/733441895 >> [v2:172.21.15.167:6824/29889,v1:172.21.15.167:6825/29889] conn(0x7fee34030c50 0x7fee34033100 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x55c572bd8110 tx=0x7feee400aec0 comp rx=0 tx=0).stop 2023-10-05T19:28:54.664 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:54.658+0000 7feeef1e2e40 1 -- 172.21.15.167:0/733441895 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55c572be2510 msgr2=0x55c572be7ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-05T19:28:54.664 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:54.658+0000 7feeecd17700 1 -- 172.21.15.167:0/733441895 reap_dead start 2023-10-05T19:28:54.664 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:54.658+0000 7feeef1e2e40 1 --2- 172.21.15.167:0/733441895 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55c572be2510 0x55c572be7ff0 secure :-1 s=READY pgs=517 cs=0 l=1 rev1=1 crypto rx=0x55c572bdc5f0 tx=0x7feed402f000 comp rx=0 tx=0).stop 2023-10-05T19:28:54.664 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:54.658+0000 7feeef1e2e40 1 -- 172.21.15.167:0/733441895 shutdown_connections 2023-10-05T19:28:54.664 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:54.658+0000 7feeef1e2e40 1 --2- 172.21.15.167:0/733441895 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55c572be2510 0x55c572be7ff0 unknown :-1 s=CLOSED pgs=517 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-05T19:28:54.665 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:54.658+0000 7feeef1e2e40 1 -- 172.21.15.167:0/733441895 >> 172.21.15.167:0/733441895 conn(0x55c572bdd8d0 msgr2=0x55c572bdb990 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-05T19:28:54.665 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:54.658+0000 7feeef1e2e40 1 -- 172.21.15.167:0/733441895 shutdown_connections 2023-10-05T19:28:54.665 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:54.658+0000 7feeef1e2e40 1 -- 172.21.15.167:0/733441895 wait complete. 2023-10-05T19:28:54.665 INFO:teuthology.orchestra.run.smithi167.stdout:[ 2023-10-05T19:28:54.665 INFO:teuthology.orchestra.run.smithi167.stdout: { 2023-10-05T19:28:54.665 INFO:teuthology.orchestra.run.smithi167.stdout: "tag": "9a9df965-6612-45d4-a45f-6e3c889bacc2.4221.6358388886286532197\u0000", 2023-10-05T19:28:54.665 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2023-10-05T19:28:54.286915+0000", 2023-10-05T19:28:54.665 INFO:teuthology.orchestra.run.smithi167.stdout: "objs": [ 2023-10-05T19:28:54.666 INFO:teuthology.orchestra.run.smithi167.stdout: { 2023-10-05T19:28:54.666 INFO:teuthology.orchestra.run.smithi167.stdout: "pool": "default.rgw.buckets.data", 2023-10-05T19:28:54.666 INFO:teuthology.orchestra.run.smithi167.stdout: "oid": "9a9df965-6612-45d4-a45f-6e3c889bacc2.4223.1__shadow_.efCgafU_mcZPUY_vovN7gwdjS1XOZ16_1", 2023-10-05T19:28:54.666 INFO:teuthology.orchestra.run.smithi167.stdout: "key": "", 2023-10-05T19:28:54.666 INFO:teuthology.orchestra.run.smithi167.stdout: "instance": "" 2023-10-05T19:28:54.666 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2023-10-05T19:28:54.666 INFO:teuthology.orchestra.run.smithi167.stdout: { 2023-10-05T19:28:54.666 INFO:teuthology.orchestra.run.smithi167.stdout: "pool": "default.rgw.buckets.data", 2023-10-05T19:28:54.666 INFO:teuthology.orchestra.run.smithi167.stdout: "oid": "9a9df965-6612-45d4-a45f-6e3c889bacc2.4223.1__shadow_.efCgafU_mcZPUY_vovN7gwdjS1XOZ16_2", 2023-10-05T19:28:54.666 INFO:teuthology.orchestra.run.smithi167.stdout: "key": "", 2023-10-05T19:28:54.666 INFO:teuthology.orchestra.run.smithi167.stdout: "instance": "" 2023-10-05T19:28:54.666 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2023-10-05T19:28:54.667 INFO:teuthology.orchestra.run.smithi167.stdout: { 2023-10-05T19:28:54.667 INFO:teuthology.orchestra.run.smithi167.stdout: "pool": "default.rgw.buckets.data", 2023-10-05T19:28:54.667 INFO:teuthology.orchestra.run.smithi167.stdout: "oid": "9a9df965-6612-45d4-a45f-6e3c889bacc2.4223.1__shadow_.efCgafU_mcZPUY_vovN7gwdjS1XOZ16_3", 2023-10-05T19:28:54.667 INFO:teuthology.orchestra.run.smithi167.stdout: "key": "", 2023-10-05T19:28:54.667 INFO:teuthology.orchestra.run.smithi167.stdout: "instance": "" 2023-10-05T19:28:54.667 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2023-10-05T19:28:54.667 INFO:teuthology.orchestra.run.smithi167.stdout: { 2023-10-05T19:28:54.667 INFO:teuthology.orchestra.run.smithi167.stdout: "pool": "default.rgw.buckets.data", 2023-10-05T19:28:54.667 INFO:teuthology.orchestra.run.smithi167.stdout: "oid": "9a9df965-6612-45d4-a45f-6e3c889bacc2.4223.1__shadow_.efCgafU_mcZPUY_vovN7gwdjS1XOZ16_4", 2023-10-05T19:28:54.667 INFO:teuthology.orchestra.run.smithi167.stdout: "key": "", 2023-10-05T19:28:54.667 INFO:teuthology.orchestra.run.smithi167.stdout: "instance": "" 2023-10-05T19:28:54.667 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2023-10-05T19:28:54.668 INFO:teuthology.orchestra.run.smithi167.stdout: { 2023-10-05T19:28:54.668 INFO:teuthology.orchestra.run.smithi167.stdout: "pool": "default.rgw.buckets.data", 2023-10-05T19:28:54.668 INFO:teuthology.orchestra.run.smithi167.stdout: "oid": "9a9df965-6612-45d4-a45f-6e3c889bacc2.4223.1__shadow_.efCgafU_mcZPUY_vovN7gwdjS1XOZ16_5", 2023-10-05T19:28:54.668 INFO:teuthology.orchestra.run.smithi167.stdout: "key": "", 2023-10-05T19:28:54.668 INFO:teuthology.orchestra.run.smithi167.stdout: "instance": "" 2023-10-05T19:28:54.668 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2023-10-05T19:28:54.668 INFO:teuthology.orchestra.run.smithi167.stdout: { 2023-10-05T19:28:54.668 INFO:teuthology.orchestra.run.smithi167.stdout: "pool": "default.rgw.buckets.data", 2023-10-05T19:28:54.668 INFO:teuthology.orchestra.run.smithi167.stdout: "oid": "9a9df965-6612-45d4-a45f-6e3c889bacc2.4223.1__shadow_.efCgafU_mcZPUY_vovN7gwdjS1XOZ16_6", 2023-10-05T19:28:54.668 INFO:teuthology.orchestra.run.smithi167.stdout: "key": "", 2023-10-05T19:28:54.668 INFO:teuthology.orchestra.run.smithi167.stdout: "instance": "" 2023-10-05T19:28:54.668 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2023-10-05T19:28:54.668 INFO:teuthology.orchestra.run.smithi167.stdout: { 2023-10-05T19:28:54.669 INFO:teuthology.orchestra.run.smithi167.stdout: "pool": "default.rgw.buckets.data", 2023-10-05T19:28:54.669 INFO:teuthology.orchestra.run.smithi167.stdout: "oid": "9a9df965-6612-45d4-a45f-6e3c889bacc2.4223.1__shadow_.efCgafU_mcZPUY_vovN7gwdjS1XOZ16_7", 2023-10-05T19:28:54.669 INFO:teuthology.orchestra.run.smithi167.stdout: "key": "", 2023-10-05T19:28:54.669 INFO:teuthology.orchestra.run.smithi167.stdout: "instance": "" 2023-10-05T19:28:54.669 INFO:teuthology.orchestra.run.smithi167.stdout: } 2023-10-05T19:28:54.669 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2023-10-05T19:28:54.669 INFO:teuthology.orchestra.run.smithi167.stdout: } 2023-10-05T19:28:54.669 INFO:teuthology.orchestra.run.smithi167.stdout:] 2023-10-05T19:28:54.670 DEBUG:tasks.util.rgw: json result: [{'tag': '9a9df965-6612-45d4-a45f-6e3c889bacc2.4221.6358388886286532197\x00', 'time': '2023-10-05T19:28:54.286915+0000', 'objs': [{'pool': 'default.rgw.buckets.data', 'oid': '9a9df965-6612-45d4-a45f-6e3c889bacc2.4223.1__shadow_.efCgafU_mcZPUY_vovN7gwdjS1XOZ16_1', 'key': '', 'instance': ''}, {'pool': 'default.rgw.buckets.data', 'oid': '9a9df965-6612-45d4-a45f-6e3c889bacc2.4223.1__shadow_.efCgafU_mcZPUY_vovN7gwdjS1XOZ16_2', 'key': '', 'instance': ''}, {'pool': 'default.rgw.buckets.data', 'oid': '9a9df965-6612-45d4-a45f-6e3c889bacc2.4223.1__shadow_.efCgafU_mcZPUY_vovN7gwdjS1XOZ16_3', 'key': '', 'instance': ''}, {'pool': 'default.rgw.buckets.data', 'oid': '9a9df965-6612-45d4-a45f-6e3c889bacc2.4223.1__shadow_.efCgafU_mcZPUY_vovN7gwdjS1XOZ16_4', 'key': '', 'instance': ''}, {'pool': 'default.rgw.buckets.data', 'oid': '9a9df965-6612-45d4-a45f-6e3c889bacc2.4223.1__shadow_.efCgafU_mcZPUY_vovN7gwdjS1XOZ16_5', 'key': '', 'instance': ''}, {'pool': 'default.rgw.buckets.data', 'oid': '9a9df965-6612-45d4-a45f-6e3c889bacc2.4223.1__shadow_.efCgafU_mcZPUY_vovN7gwdjS1XOZ16_6', 'key': '', 'instance': ''}, {'pool': 'default.rgw.buckets.data', 'oid': '9a9df965-6612-45d4-a45f-6e3c889bacc2.4223.1__shadow_.efCgafU_mcZPUY_vovN7gwdjS1XOZ16_7', 'key': '', 'instance': ''}]}] 2023-10-05T19:28:54.670 INFO:tasks.util.rgw:rgwadmin: client.0 : ['gc', 'process'] 2023-10-05T19:28:54.670 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'ceph', 'gc', 'process'] 2023-10-05T19:28:54.671 DEBUG:teuthology.orchestra.run.smithi167:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster ceph gc process 2023-10-05T19:28:54.754 INFO:teuthology.orchestra.run.smithi167.stderr:ignoring --setuser ceph since I am not root 2023-10-05T19:28:54.754 INFO:teuthology.orchestra.run.smithi167.stderr:ignoring --setgroup ceph since I am not root 2023-10-05T19:28:54.754 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:54.750+0000 7f9f6621fe40 1 Processor -- start 2023-10-05T19:28:54.754 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:54.750+0000 7f9f6621fe40 1 -- start start 2023-10-05T19:28:54.755 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:54.750+0000 7f9f6621fe40 1 --2- >> v2:172.21.15.118:3300/0 conn(0x563697b335d0 0x563697b359b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-05T19:28:54.755 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:54.750+0000 7f9f6621fe40 1 -- --> v1:172.21.15.118:6789/0 -- auth(proto 0 26 bytes epoch 0) v1 -- 0x563697a79c00 con 0x563697b2eff0 2023-10-05T19:28:54.755 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:54.750+0000 7f9f6621fe40 1 -- --> v2:172.21.15.118:3300/0 -- mon_getmap magic: 0 v1 -- 0x563697a36ee0 con 0x563697b335d0 2023-10-05T19:28:54.756 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:54.750+0000 7f9f63553700 1 --2- >> v2:172.21.15.118:3300/0 conn(0x563697b335d0 0x563697b359b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-05T19:28:54.756 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:54.750+0000 7f9f63d54700 1 --1- >> v1:172.21.15.118:6789/0 conn(0x563697b2eff0 0x563697b2f3c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.118:6789/0 says I am v1:172.21.15.167:48672/0 (socket says 172.21.15.167:48672) 2023-10-05T19:28:54.756 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:54.750+0000 7f9f63d54700 1 -- 172.21.15.167:0/3353464685 learned_addr learned my addr 172.21.15.167:0/3353464685 (peer_addr_for_me v1:172.21.15.167:0/0) 2023-10-05T19:28:54.756 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:54.750+0000 7f9f62d52700 1 -- 172.21.15.167:0/3353464685 <== mon.0 v1:172.21.15.118:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4289913166 0 0) 0x563697a79c00 con 0x563697b2eff0 2023-10-05T19:28:54.757 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:54.750+0000 7f9f62d52700 1 -- 172.21.15.167:0/3353464685 --> v1:172.21.15.118:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9f50001e10 con 0x563697b2eff0 2023-10-05T19:28:54.757 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:54.750+0000 7f9f63553700 1 -- 172.21.15.167:0/3353464685 >> v1:172.21.15.118:6789/0 conn(0x563697b2eff0 legacy=0x563697b2f3c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-05T19:28:54.757 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:54.750+0000 7f9f63553700 1 -- 172.21.15.167:0/3353464685 --> v2:172.21.15.118:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x563697b2fc00 con 0x563697b335d0 2023-10-05T19:28:54.757 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:54.750+0000 7f9f63553700 1 --2- 172.21.15.167:0/3353464685 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x563697b335d0 0x563697b359b0 secure :-1 s=READY pgs=519 cs=0 l=1 rev1=1 crypto rx=0x7f9f4c00a5d0 tx=0x7f9f4c005a50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=84bb604a40720715 server_cookie=0 in_seq=0 out_seq=0 2023-10-05T19:28:54.758 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:54.750+0000 7f9f62d52700 1 -- 172.21.15.167:0/3353464685 <== mon.0 v2:172.21.15.118:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f9f4c007610 con 0x563697b335d0 2023-10-05T19:28:54.758 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:54.750+0000 7f9f62d52700 1 -- 172.21.15.167:0/3353464685 <== mon.0 v2:172.21.15.118:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9f4c007770 con 0x563697b335d0 2023-10-05T19:28:54.758 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:54.750+0000 7f9f62d52700 1 -- 172.21.15.167:0/3353464685 <== mon.0 v2:172.21.15.118:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f9f4c007a20 con 0x563697b335d0 2023-10-05T19:28:54.758 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:54.750+0000 7f9f6621fe40 1 -- 172.21.15.167:0/3353464685 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x563697b335d0 msgr2=0x563697b359b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-05T19:28:54.758 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:54.750+0000 7f9f6621fe40 1 --2- 172.21.15.167:0/3353464685 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x563697b335d0 0x563697b359b0 secure :-1 s=READY pgs=519 cs=0 l=1 rev1=1 crypto rx=0x7f9f4c00a5d0 tx=0x7f9f4c005a50 comp rx=0 tx=0).stop 2023-10-05T19:28:54.758 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:54.754+0000 7f9f6621fe40 1 -- 172.21.15.167:0/3353464685 shutdown_connections 2023-10-05T19:28:54.759 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:54.754+0000 7f9f6621fe40 1 --2- 172.21.15.167:0/3353464685 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x563697b335d0 0x563697b359b0 unknown :-1 s=CLOSED pgs=519 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-05T19:28:54.759 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:54.754+0000 7f9f6621fe40 1 -- 172.21.15.167:0/3353464685 >> 172.21.15.167:0/3353464685 conn(0x563697aa2840 msgr2=0x563697b2d4e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-05T19:28:54.759 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:54.754+0000 7f9f6621fe40 1 -- 172.21.15.167:0/3353464685 shutdown_connections 2023-10-05T19:28:54.759 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:54.754+0000 7f9f6621fe40 1 -- 172.21.15.167:0/3353464685 wait complete. 2023-10-05T19:28:54.760 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:54.754+0000 7f9f6621fe40 1 Processor -- start 2023-10-05T19:28:54.760 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:54.754+0000 7f9f6621fe40 1 -- start start 2023-10-05T19:28:54.760 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:54.754+0000 7f9f6621fe40 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x563697b2eff0 0x563697b4df10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-05T19:28:54.760 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:54.754+0000 7f9f6621fe40 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x5636979fae30 con 0x563697b2eff0 2023-10-05T19:28:54.760 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:54.754+0000 7f9f63d54700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x563697b2eff0 0x563697b4df10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-05T19:28:54.760 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:54.754+0000 7f9f63d54700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x563697b2eff0 0x563697b4df10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.118:3300/0 says I am v2:172.21.15.167:53168/0 (socket says 172.21.15.167:53168) 2023-10-05T19:28:54.760 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:54.754+0000 7f9f63d54700 1 -- 172.21.15.167:0/3151286366 learned_addr learned my addr 172.21.15.167:0/3151286366 (peer_addr_for_me v2:172.21.15.167:0/0) 2023-10-05T19:28:54.760 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:54.754+0000 7f9f63d54700 1 -- 172.21.15.167:0/3151286366 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9f4c005cf0 con 0x563697b2eff0 2023-10-05T19:28:54.761 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:54.754+0000 7f9f63d54700 1 --2- 172.21.15.167:0/3151286366 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x563697b2eff0 0x563697b4df10 secure :-1 s=READY pgs=520 cs=0 l=1 rev1=1 crypto rx=0x7f9f5c006e30 tx=0x7f9f5c006a10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=51f6850ad75870da server_cookie=0 in_seq=0 out_seq=0 2023-10-05T19:28:54.761 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:54.754+0000 7f9f61d50700 1 -- 172.21.15.167:0/3151286366 <== mon.0 v2:172.21.15.118:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f9f5c006110 con 0x563697b2eff0 2023-10-05T19:28:54.761 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:54.754+0000 7f9f61d50700 1 -- 172.21.15.167:0/3151286366 <== mon.0 v2:172.21.15.118:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9f5c006290 con 0x563697b2eff0 2023-10-05T19:28:54.761 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:54.754+0000 7f9f61d50700 1 -- 172.21.15.167:0/3151286366 <== mon.0 v2:172.21.15.118:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f9f5c004520 con 0x563697b2eff0 2023-10-05T19:28:54.762 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:54.758+0000 7f9f6621fe40 1 -- 172.21.15.167:0/3151286366 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x563697b2eff0 msgr2=0x563697b4df10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-05T19:28:54.762 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:54.758+0000 7f9f6621fe40 1 --2- 172.21.15.167:0/3151286366 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x563697b2eff0 0x563697b4df10 secure :-1 s=READY pgs=520 cs=0 l=1 rev1=1 crypto rx=0x7f9f5c006e30 tx=0x7f9f5c006a10 comp rx=0 tx=0).stop 2023-10-05T19:28:54.762 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:54.758+0000 7f9f6621fe40 1 -- 172.21.15.167:0/3151286366 shutdown_connections 2023-10-05T19:28:54.762 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:54.758+0000 7f9f6621fe40 1 --2- 172.21.15.167:0/3151286366 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x563697b2eff0 0x563697b4df10 unknown :-1 s=CLOSED pgs=520 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-05T19:28:54.762 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:54.758+0000 7f9f6621fe40 1 -- 172.21.15.167:0/3151286366 >> 172.21.15.167:0/3151286366 conn(0x563697aa2840 msgr2=0x563697b49ad0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-05T19:28:54.763 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:54.758+0000 7f9f6621fe40 1 -- 172.21.15.167:0/3151286366 shutdown_connections 2023-10-05T19:28:54.763 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:54.758+0000 7f9f6621fe40 1 -- 172.21.15.167:0/3151286366 wait complete. 2023-10-05T19:28:54.766 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:54.758+0000 7f9f6621fe40 1 Processor -- start 2023-10-05T19:28:54.766 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:54.758+0000 7f9f6621fe40 1 -- start start 2023-10-05T19:28:54.766 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:54.758+0000 7f9f6621fe40 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x563697b2eff0 0x563697b54cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-05T19:28:54.766 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:54.758+0000 7f9f6621fe40 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x5636979cded0 con 0x563697b2eff0 2023-10-05T19:28:54.766 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:54.758+0000 7f9f63d54700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x563697b2eff0 0x563697b54cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-05T19:28:54.766 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:54.758+0000 7f9f63d54700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x563697b2eff0 0x563697b54cf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.118:3300/0 says I am v2:172.21.15.167:53178/0 (socket says 172.21.15.167:53178) 2023-10-05T19:28:54.766 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:54.758+0000 7f9f63d54700 1 -- 172.21.15.167:0/3004404278 learned_addr learned my addr 172.21.15.167:0/3004404278 (peer_addr_for_me v2:172.21.15.167:0/0) 2023-10-05T19:28:54.766 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:54.758+0000 7f9f63d54700 1 -- 172.21.15.167:0/3004404278 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9f50001e10 con 0x563697b2eff0 2023-10-05T19:28:54.766 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:54.758+0000 7f9f63d54700 1 --2- 172.21.15.167:0/3004404278 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x563697b2eff0 0x563697b54cf0 secure :-1 s=READY pgs=521 cs=0 l=1 rev1=1 crypto rx=0x7f9f5c00f000 tx=0x7f9f5c010000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-05T19:28:54.766 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:54.758+0000 7f9f5bfff700 1 -- 172.21.15.167:0/3004404278 <== mon.0 v2:172.21.15.118:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f9f5c006110 con 0x563697b2eff0 2023-10-05T19:28:54.767 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:54.758+0000 7f9f5bfff700 1 -- 172.21.15.167:0/3004404278 <== mon.0 v2:172.21.15.118:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9f5c0044f0 con 0x563697b2eff0 2023-10-05T19:28:54.767 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:54.758+0000 7f9f6621fe40 1 -- 172.21.15.167:0/3004404278 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x563697b55230 con 0x563697b2eff0 2023-10-05T19:28:54.767 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:54.762+0000 7f9f5bfff700 1 -- 172.21.15.167:0/3004404278 <== mon.0 v2:172.21.15.118:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f9f5c014bd0 con 0x563697b2eff0 2023-10-05T19:28:54.767 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:54.762+0000 7f9f6621fe40 1 -- 172.21.15.167:0/3004404278 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x563697b55670 con 0x563697b2eff0 2023-10-05T19:28:54.768 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:54.762+0000 7f9f5bfff700 1 -- 172.21.15.167:0/3004404278 <== mon.0 v2:172.21.15.118:3300/0 4 ==== mgrmap(e 5) v1 ==== 40329+0+0 (secure 0 0 0) 0x7f9f5c028460 con 0x563697b2eff0 2023-10-05T19:28:54.768 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:54.762+0000 7f9f6621fe40 20 rados->read ofs=0 len=0 2023-10-05T19:28:54.768 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:54.762+0000 7f9f5bfff700 1 --2- 172.21.15.167:0/3004404278 >> [v2:172.21.15.167:6824/29889,v1:172.21.15.167:6825/29889] conn(0x7f9f44030c00 0x7f9f440330b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-05T19:28:54.768 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:54.762+0000 7f9f5bfff700 1 -- 172.21.15.167:0/3004404278 <== mon.0 v2:172.21.15.118:3300/0 5 ==== osd_map(32..32 src has 1..32) v4 ==== 6260+0+0 (secure 0 0 0) 0x7f9f5c0452a0 con 0x563697b2eff0 2023-10-05T19:28:54.768 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:54.762+0000 7f9f63553700 1 --2- 172.21.15.167:0/3004404278 >> [v2:172.21.15.167:6824/29889,v1:172.21.15.167:6825/29889] conn(0x7f9f44030c00 0x7f9f440330b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-05T19:28:54.769 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:54.762+0000 7f9f6621fe40 1 --2- 172.21.15.167:0/3004404278 >> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] conn(0x563697b335d0 0x563697b607b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-05T19:28:54.769 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:54.762+0000 7f9f6621fe40 1 -- 172.21.15.167:0/3004404278 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:1 5.12 5:49953fa1:::default.realm:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32) v8 -- 0x563697b60eb0 con 0x563697b335d0 2023-10-05T19:28:54.769 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:54.762+0000 7f9f64555700 1 --2- 172.21.15.167:0/3004404278 >> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] conn(0x563697b335d0 0x563697b607b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-05T19:28:54.769 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:54.762+0000 7f9f63553700 1 --2- 172.21.15.167:0/3004404278 >> [v2:172.21.15.167:6824/29889,v1:172.21.15.167:6825/29889] conn(0x7f9f44030c00 0x7f9f440330b0 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7f9f4c005cc0 tx=0x7f9f4c014000 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-05T19:28:54.769 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:54.762+0000 7f9f64555700 1 --2- 172.21.15.167:0/3004404278 >> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] conn(0x563697b335d0 0x563697b607b0 crc :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-05T19:28:54.770 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:54.766+0000 7f9f64555700 1 -- 172.21.15.167:0/3004404278 <== osd.1 v2:172.21.15.167:6800/30035 1 ==== osd_op_reply(1 default.realm [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 ==== 157+0+0 (crc 0 0 0) 0x7f9f5400a500 con 0x563697b335d0 2023-10-05T19:28:54.770 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:54.766+0000 7f9f6621fe40 20 rados_obj.operate() r=-2 bl.length=0 2023-10-05T19:28:54.770 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:54.766+0000 7f9f6621fe40 20 realm 2023-10-05T19:28:54.770 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:54.766+0000 7f9f6621fe40 20 rados->read ofs=0 len=0 2023-10-05T19:28:54.770 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:54.766+0000 7f9f6621fe40 1 -- 172.21.15.167:0/3004404278 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:2 5.12 5:49953fa1:::default.realm:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32) v8 -- 0x563697b5ddd0 con 0x563697b335d0 2023-10-05T19:28:54.772 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:54.766+0000 7f9f64555700 1 -- 172.21.15.167:0/3004404278 <== osd.1 v2:172.21.15.167:6800/30035 2 ==== osd_op_reply(2 default.realm [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 ==== 157+0+0 (crc 0 0 0) 0x7f9f5400a500 con 0x563697b335d0 2023-10-05T19:28:54.772 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:54.766+0000 7f9f6621fe40 20 rados_obj.operate() r=-2 bl.length=0 2023-10-05T19:28:54.772 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:54.766+0000 7f9f6621fe40 4 RGWPeriod::init failed to init realm id : (2) No such file or directory 2023-10-05T19:28:54.772 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:54.766+0000 7f9f6621fe40 20 rados->read ofs=0 len=0 2023-10-05T19:28:54.772 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:54.766+0000 7f9f6621fe40 1 -- 172.21.15.167:0/3004404278 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:3 5.12 5:49953fa1:::default.realm:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32) v8 -- 0x563697b61990 con 0x563697b335d0 2023-10-05T19:28:54.772 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:54.766+0000 7f9f64555700 1 -- 172.21.15.167:0/3004404278 <== osd.1 v2:172.21.15.167:6800/30035 3 ==== osd_op_reply(3 default.realm [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 ==== 157+0+0 (crc 0 0 0) 0x7f9f5400a500 con 0x563697b335d0 2023-10-05T19:28:54.772 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:54.766+0000 7f9f6621fe40 20 rados_obj.operate() r=-2 bl.length=0 2023-10-05T19:28:54.772 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:54.766+0000 7f9f6621fe40 20 rados->read ofs=0 len=0 2023-10-05T19:28:54.773 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:54.766+0000 7f9f6621fe40 1 -- 172.21.15.167:0/3004404278 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:4 5.f 5:f43b6ece:::zone_names.default:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32) v8 -- 0x563697b61570 con 0x563697b335d0 2023-10-05T19:28:54.773 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:54.766+0000 7f9f64555700 1 -- 172.21.15.167:0/3004404278 <== osd.1 v2:172.21.15.167:6800/30035 4 ==== osd_op_reply(4 zone_names.default [read 0~46 out=46b] v0'0 uv1 ondisk = 0) v8 ==== 162+0+46 (crc 0 0 0) 0x7f9f5400a500 con 0x563697b335d0 2023-10-05T19:28:54.773 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:54.766+0000 7f9f6621fe40 20 rados_obj.operate() r=0 bl.length=46 2023-10-05T19:28:54.773 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:54.766+0000 7f9f6621fe40 20 rados->read ofs=0 len=0 2023-10-05T19:28:54.773 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:54.766+0000 7f9f6621fe40 1 -- 172.21.15.167:0/3004404278 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:5 5.1d 5:b8374eb4:::zone_info.9a9df965-6612-45d4-a45f-6e3c889bacc2:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32) v8 -- 0x563697b62530 con 0x563697b335d0 2023-10-05T19:28:54.774 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:54.766+0000 7f9f64555700 1 -- 172.21.15.167:0/3004404278 <== osd.1 v2:172.21.15.167:6800/30035 5 ==== osd_op_reply(5 zone_info.9a9df965-6612-45d4-a45f-6e3c889bacc2 [read 0~874 out=874b] v0'0 uv2 ondisk = 0) v8 ==== 190+0+874 (crc 0 0 0) 0x7f9f5400a500 con 0x563697b335d0 2023-10-05T19:28:54.774 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:54.766+0000 7f9f6621fe40 20 rados_obj.operate() r=0 bl.length=874 2023-10-05T19:28:54.774 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:54.766+0000 7f9f6621fe40 20 searching for the correct realm 2023-10-05T19:28:54.774 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:54.766+0000 7f9f6621fe40 1 --2- 172.21.15.167:0/3004404278 >> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] conn(0x563697b64440 0x563697b66850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-05T19:28:54.774 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:54.766+0000 7f9f6621fe40 1 -- 172.21.15.167:0/3004404278 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:6 5.0 5:00000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x563697b66e60 con 0x563697b64440 2023-10-05T19:28:54.774 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:54.766+0000 7f9f63d54700 1 --2- 172.21.15.167:0/3004404278 >> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] conn(0x563697b64440 0x563697b66850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-05T19:28:54.774 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:54.766+0000 7f9f63d54700 1 --2- 172.21.15.167:0/3004404278 >> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] conn(0x563697b64440 0x563697b66850 crc :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-05T19:28:54.775 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:54.770+0000 7f9f63d54700 1 -- 172.21.15.167:0/3004404278 <== osd.3 v2:172.21.15.167:6808/30036 1 ==== osd_op_reply(6 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f9f5c024020 con 0x563697b64440 2023-10-05T19:28:54.775 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:54.770+0000 7f9f63d54700 1 --2- 172.21.15.167:0/3004404278 >> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] conn(0x7f9f5c010c30 0x7f9f5c0439b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-05T19:28:54.775 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:54.770+0000 7f9f63d54700 1 -- 172.21.15.167:0/3004404278 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:7 5.10 5:08000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x563697b66e60 con 0x7f9f5c010c30 2023-10-05T19:28:54.775 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:54.770+0000 7f9f63553700 1 --2- 172.21.15.167:0/3004404278 >> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] conn(0x7f9f5c010c30 0x7f9f5c0439b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-05T19:28:54.776 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:54.770+0000 7f9f63553700 1 --2- 172.21.15.167:0/3004404278 >> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] conn(0x7f9f5c010c30 0x7f9f5c0439b0 crc :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-05T19:28:54.776 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:54.770+0000 7f9f63553700 1 -- 172.21.15.167:0/3004404278 <== osd.2 v2:172.21.15.167:6816/30037 1 ==== osd_op_reply(7 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f9f4c007a50 con 0x7f9f5c010c30 2023-10-05T19:28:54.776 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:54.770+0000 7f9f63553700 1 --2- 172.21.15.167:0/3004404278 >> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] conn(0x7f9f4c007030 0x7f9f4c01d430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-05T19:28:54.777 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:54.770+0000 7f9f63553700 1 -- 172.21.15.167:0/3004404278 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:8 5.8 5:10000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x563697b66e60 con 0x7f9f4c007030 2023-10-05T19:28:54.777 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:54.770+0000 7f9f64555700 1 --2- 172.21.15.167:0/3004404278 >> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] conn(0x7f9f4c007030 0x7f9f4c01d430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-05T19:28:54.777 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:54.770+0000 7f9f64555700 1 --2- 172.21.15.167:0/3004404278 >> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] conn(0x7f9f4c007030 0x7f9f4c01d430 crc :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-05T19:28:54.778 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:54.774+0000 7f9f64555700 1 -- 172.21.15.167:0/3004404278 <== osd.0 v2:172.21.15.118:6800/27690 1 ==== osd_op_reply(8 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f9f54014040 con 0x7f9f4c007030 2023-10-05T19:28:54.778 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:54.774+0000 7f9f64555700 1 -- 172.21.15.167:0/3004404278 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:9 5.18 5:18000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x563697b66e60 con 0x563697b335d0 2023-10-05T19:28:54.779 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:54.774+0000 7f9f64555700 1 -- 172.21.15.167:0/3004404278 <== osd.1 v2:172.21.15.167:6800/30035 6 ==== osd_op_reply(9 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f9f54014040 con 0x563697b335d0 2023-10-05T19:28:54.779 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:54.774+0000 7f9f64555700 1 -- 172.21.15.167:0/3004404278 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:10 5.4 5:20000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x563697b66e60 con 0x563697b335d0 2023-10-05T19:28:54.779 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:54.774+0000 7f9f64555700 1 -- 172.21.15.167:0/3004404278 <== osd.1 v2:172.21.15.167:6800/30035 7 ==== osd_op_reply(10 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f9f54014040 con 0x563697b335d0 2023-10-05T19:28:54.780 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:54.774+0000 7f9f64555700 1 -- 172.21.15.167:0/3004404278 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:11 5.14 5:28000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x563697b66e60 con 0x7f9f4c007030 2023-10-05T19:28:54.780 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:54.774+0000 7f9f64555700 1 -- 172.21.15.167:0/3004404278 <== osd.0 v2:172.21.15.118:6800/27690 2 ==== osd_op_reply(11 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f9f54014040 con 0x7f9f4c007030 2023-10-05T19:28:54.780 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:54.774+0000 7f9f64555700 1 -- 172.21.15.167:0/3004404278 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:12 5.c 5:30000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x563697b66e60 con 0x7f9f4c007030 2023-10-05T19:28:54.781 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:54.774+0000 7f9f64555700 1 -- 172.21.15.167:0/3004404278 <== osd.0 v2:172.21.15.118:6800/27690 3 ==== osd_op_reply(12 [pgnls start_epoch 32 out=112b] v23'1 uv1 ondisk = 1) v8 ==== 144+0+112 (crc 0 0 0) 0x7f9f54014040 con 0x7f9f4c007030 2023-10-05T19:28:54.781 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:54.774+0000 7f9f64555700 1 -- 172.21.15.167:0/3004404278 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:13 5.1c 5:38000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x563697b66e60 con 0x7f9f5c010c30 2023-10-05T19:28:54.782 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:54.778+0000 7f9f63553700 1 -- 172.21.15.167:0/3004404278 <== osd.2 v2:172.21.15.167:6816/30037 2 ==== osd_op_reply(13 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f9f4c029040 con 0x7f9f5c010c30 2023-10-05T19:28:54.782 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:54.778+0000 7f9f63553700 1 -- 172.21.15.167:0/3004404278 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:14 5.2 5:40000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x563697b66e60 con 0x563697b64440 2023-10-05T19:28:54.782 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:54.778+0000 7f9f63d54700 1 -- 172.21.15.167:0/3004404278 <== osd.3 v2:172.21.15.167:6808/30036 2 ==== osd_op_reply(14 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f9f5c024020 con 0x563697b64440 2023-10-05T19:28:54.783 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:54.778+0000 7f9f63d54700 1 -- 172.21.15.167:0/3004404278 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:15 5.12 5:48000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x563697b66e60 con 0x563697b335d0 2023-10-05T19:28:54.783 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:54.778+0000 7f9f64555700 1 -- 172.21.15.167:0/3004404278 <== osd.1 v2:172.21.15.167:6800/30035 8 ==== osd_op_reply(15 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f9f54014040 con 0x563697b335d0 2023-10-05T19:28:54.783 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:54.778+0000 7f9f64555700 1 -- 172.21.15.167:0/3004404278 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:16 5.a 5:50000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x563697b66e60 con 0x7f9f4c007030 2023-10-05T19:28:54.784 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:54.778+0000 7f9f64555700 1 -- 172.21.15.167:0/3004404278 <== osd.0 v2:172.21.15.118:6800/27690 4 ==== osd_op_reply(16 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f9f54022040 con 0x7f9f4c007030 2023-10-05T19:28:54.784 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:54.778+0000 7f9f64555700 1 -- 172.21.15.167:0/3004404278 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:17 5.1a 5:58000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x563697b66e60 con 0x563697b335d0 2023-10-05T19:28:54.785 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:54.778+0000 7f9f64555700 1 -- 172.21.15.167:0/3004404278 <== osd.1 v2:172.21.15.167:6800/30035 9 ==== osd_op_reply(17 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f9f54022040 con 0x563697b335d0 2023-10-05T19:28:54.785 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:54.778+0000 7f9f64555700 1 -- 172.21.15.167:0/3004404278 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:18 5.6 5:60000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x563697b66e60 con 0x7f9f5c010c30 2023-10-05T19:28:54.786 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:54.782+0000 7f9f63553700 1 -- 172.21.15.167:0/3004404278 <== osd.2 v2:172.21.15.167:6816/30037 3 ==== osd_op_reply(18 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f9f4c029040 con 0x7f9f5c010c30 2023-10-05T19:28:54.786 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:54.782+0000 7f9f63553700 1 -- 172.21.15.167:0/3004404278 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:19 5.16 5:68000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x563697b66e60 con 0x563697b64440 2023-10-05T19:28:54.786 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:54.782+0000 7f9f63d54700 1 -- 172.21.15.167:0/3004404278 <== osd.3 v2:172.21.15.167:6808/30036 3 ==== osd_op_reply(19 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f9f5c024020 con 0x563697b64440 2023-10-05T19:28:54.786 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:54.782+0000 7f9f63d54700 1 -- 172.21.15.167:0/3004404278 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:20 5.e 5:70000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x563697b66e60 con 0x563697b64440 2023-10-05T19:28:54.787 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:54.782+0000 7f9f63d54700 1 -- 172.21.15.167:0/3004404278 <== osd.3 v2:172.21.15.167:6808/30036 4 ==== osd_op_reply(20 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f9f5c024020 con 0x563697b64440 2023-10-05T19:28:54.787 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:54.782+0000 7f9f63d54700 1 -- 172.21.15.167:0/3004404278 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:21 5.1e 5:78000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x563697b66e60 con 0x7f9f4c007030 2023-10-05T19:28:54.788 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:54.782+0000 7f9f64555700 1 -- 172.21.15.167:0/3004404278 <== osd.0 v2:172.21.15.118:6800/27690 5 ==== osd_op_reply(21 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f9f54022040 con 0x7f9f4c007030 2023-10-05T19:28:54.788 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:54.782+0000 7f9f64555700 1 -- 172.21.15.167:0/3004404278 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:22 5.1 5:80000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x563697b66e60 con 0x563697b64440 2023-10-05T19:28:54.789 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:54.782+0000 7f9f63d54700 1 -- 172.21.15.167:0/3004404278 <== osd.3 v2:172.21.15.167:6808/30036 5 ==== osd_op_reply(22 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f9f5c024020 con 0x563697b64440 2023-10-05T19:28:54.789 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:54.782+0000 7f9f63d54700 1 -- 172.21.15.167:0/3004404278 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:23 5.11 5:88000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x563697b66e60 con 0x563697b64440 2023-10-05T19:28:54.789 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:54.786+0000 7f9f63d54700 1 -- 172.21.15.167:0/3004404278 <== osd.3 v2:172.21.15.167:6808/30036 6 ==== osd_op_reply(23 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f9f5c024020 con 0x563697b64440 2023-10-05T19:28:54.790 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:54.786+0000 7f9f63d54700 1 -- 172.21.15.167:0/3004404278 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:24 5.9 5:90000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x563697b66e60 con 0x563697b335d0 2023-10-05T19:28:54.790 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:54.786+0000 7f9f64555700 1 -- 172.21.15.167:0/3004404278 <== osd.1 v2:172.21.15.167:6800/30035 10 ==== osd_op_reply(24 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f9f54022040 con 0x563697b335d0 2023-10-05T19:28:54.791 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:54.786+0000 7f9f64555700 1 -- 172.21.15.167:0/3004404278 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:25 5.19 5:98000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x563697b66e60 con 0x563697b335d0 2023-10-05T19:28:54.791 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:54.786+0000 7f9f64555700 1 -- 172.21.15.167:0/3004404278 <== osd.1 v2:172.21.15.167:6800/30035 11 ==== osd_op_reply(25 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f9f54022040 con 0x563697b335d0 2023-10-05T19:28:54.791 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:54.786+0000 7f9f64555700 1 -- 172.21.15.167:0/3004404278 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:26 5.5 5:a0000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x563697b66e60 con 0x7f9f4c007030 2023-10-05T19:28:54.792 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:54.786+0000 7f9f64555700 1 -- 172.21.15.167:0/3004404278 <== osd.0 v2:172.21.15.118:6800/27690 6 ==== osd_op_reply(26 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f9f54022040 con 0x7f9f4c007030 2023-10-05T19:28:54.792 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:54.786+0000 7f9f64555700 1 -- 172.21.15.167:0/3004404278 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:27 5.15 5:a8000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x563697b66e60 con 0x563697b64440 2023-10-05T19:28:54.793 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:54.786+0000 7f9f63d54700 1 -- 172.21.15.167:0/3004404278 <== osd.3 v2:172.21.15.167:6808/30036 7 ==== osd_op_reply(27 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f9f5c024020 con 0x563697b64440 2023-10-05T19:28:54.793 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:54.786+0000 7f9f63d54700 1 -- 172.21.15.167:0/3004404278 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:28 5.d 5:b0000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x563697b66e60 con 0x7f9f5c010c30 2023-10-05T19:28:54.794 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:54.790+0000 7f9f63553700 1 -- 172.21.15.167:0/3004404278 <== osd.2 v2:172.21.15.167:6816/30037 4 ==== osd_op_reply(28 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f9f4c029040 con 0x7f9f5c010c30 2023-10-05T19:28:54.794 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:54.790+0000 7f9f63553700 1 -- 172.21.15.167:0/3004404278 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:29 5.1d 5:b8000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x563697b66e60 con 0x563697b335d0 2023-10-05T19:28:54.795 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:54.790+0000 7f9f64555700 1 -- 172.21.15.167:0/3004404278 <== osd.1 v2:172.21.15.167:6800/30035 12 ==== osd_op_reply(29 [pgnls start_epoch 32 out=107b] v30'2 uv2 ondisk = 1) v8 ==== 144+0+107 (crc 0 0 0) 0x7f9f54022040 con 0x563697b335d0 2023-10-05T19:28:54.795 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:54.790+0000 7f9f64555700 1 -- 172.21.15.167:0/3004404278 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:30 5.3 5:c0000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x563697b66e60 con 0x7f9f4c007030 2023-10-05T19:28:54.795 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:54.790+0000 7f9f64555700 1 -- 172.21.15.167:0/3004404278 <== osd.0 v2:172.21.15.118:6800/27690 7 ==== osd_op_reply(30 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f9f54022040 con 0x7f9f4c007030 2023-10-05T19:28:54.796 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:54.790+0000 7f9f64555700 1 -- 172.21.15.167:0/3004404278 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:31 5.13 5:c8000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x563697b66e60 con 0x563697b64440 2023-10-05T19:28:54.796 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:54.790+0000 7f9f63d54700 1 -- 172.21.15.167:0/3004404278 <== osd.3 v2:172.21.15.167:6808/30036 8 ==== osd_op_reply(31 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f9f5c024020 con 0x563697b64440 2023-10-05T19:28:54.796 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:54.790+0000 7f9f63d54700 1 -- 172.21.15.167:0/3004404278 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:32 5.b 5:d0000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x563697b66e60 con 0x7f9f4c007030 2023-10-05T19:28:54.797 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:54.794+0000 7f9f64555700 1 -- 172.21.15.167:0/3004404278 <== osd.0 v2:172.21.15.118:6800/27690 8 ==== osd_op_reply(32 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f9f54022040 con 0x7f9f4c007030 2023-10-05T19:28:54.798 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:54.794+0000 7f9f64555700 1 -- 172.21.15.167:0/3004404278 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:33 5.1b 5:d8000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x563697b66e60 con 0x7f9f5c010c30 2023-10-05T19:28:54.798 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:54.794+0000 7f9f63553700 1 -- 172.21.15.167:0/3004404278 <== osd.2 v2:172.21.15.167:6816/30037 5 ==== osd_op_reply(33 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f9f4c029040 con 0x7f9f5c010c30 2023-10-05T19:28:54.798 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:54.794+0000 7f9f63553700 1 -- 172.21.15.167:0/3004404278 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:34 5.7 5:e0000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x563697b66e60 con 0x563697b64440 2023-10-05T19:28:54.799 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:54.794+0000 7f9f63d54700 1 -- 172.21.15.167:0/3004404278 <== osd.3 v2:172.21.15.167:6808/30036 9 ==== osd_op_reply(34 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f9f5c024020 con 0x563697b64440 2023-10-05T19:28:54.800 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:54.794+0000 7f9f63d54700 1 -- 172.21.15.167:0/3004404278 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:35 5.17 5:e8000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x563697b66e60 con 0x7f9f4c007030 2023-10-05T19:28:54.801 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:54.794+0000 7f9f64555700 1 -- 172.21.15.167:0/3004404278 <== osd.0 v2:172.21.15.118:6800/27690 9 ==== osd_op_reply(35 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f9f54022040 con 0x7f9f4c007030 2023-10-05T19:28:54.801 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:54.794+0000 7f9f64555700 1 -- 172.21.15.167:0/3004404278 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:36 5.f 5:f0000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x563697b66e60 con 0x563697b335d0 2023-10-05T19:28:54.802 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:54.798+0000 7f9f64555700 1 -- 172.21.15.167:0/3004404278 <== osd.1 v2:172.21.15.167:6800/30035 13 ==== osd_op_reply(36 [pgnls start_epoch 32 out=115b] v23'2 uv2 ondisk = 1) v8 ==== 144+0+115 (crc 0 0 0) 0x7f9f5402f080 con 0x563697b335d0 2023-10-05T19:28:54.802 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:54.798+0000 7f9f64555700 1 -- 172.21.15.167:0/3004404278 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:37 5.1f 5:f8000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x563697b66e60 con 0x7f9f5c010c30 2023-10-05T19:28:54.803 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:54.798+0000 7f9f63553700 1 -- 172.21.15.167:0/3004404278 <== osd.2 v2:172.21.15.167:6816/30037 6 ==== osd_op_reply(37 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f9f4c029040 con 0x7f9f5c010c30 2023-10-05T19:28:54.803 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:54.798+0000 7f9f6621fe40 20 RGWRados::pool_iterate: got zonegroup_info.4f1f719d-9648-417b-b10c-e5d939825419 2023-10-05T19:28:54.803 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:54.798+0000 7f9f6621fe40 20 RGWRados::pool_iterate: got zone_info.9a9df965-6612-45d4-a45f-6e3c889bacc2 2023-10-05T19:28:54.803 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:54.798+0000 7f9f6621fe40 20 RGWRados::pool_iterate: got zone_names.default 2023-10-05T19:28:54.803 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:54.798+0000 7f9f6621fe40 20 RGWRados::pool_iterate: got zonegroups_names.default 2023-10-05T19:28:54.804 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:54.798+0000 7f9f6621fe40 20 rados->read ofs=0 len=0 2023-10-05T19:28:54.804 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:54.798+0000 7f9f6621fe40 1 -- 172.21.15.167:0/3004404278 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:38 5.12 5:49953fa1:::default.realm:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32) v8 -- 0x563697b62ef0 con 0x563697b335d0 2023-10-05T19:28:54.805 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:54.798+0000 7f9f64555700 1 -- 172.21.15.167:0/3004404278 <== osd.1 v2:172.21.15.167:6800/30035 14 ==== osd_op_reply(38 default.realm [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 ==== 157+0+0 (crc 0 0 0) 0x7f9f5402f080 con 0x563697b335d0 2023-10-05T19:28:54.805 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:54.798+0000 7f9f6621fe40 20 rados_obj.operate() r=-2 bl.length=0 2023-10-05T19:28:54.805 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:54.798+0000 7f9f6621fe40 20 rados->read ofs=0 len=0 2023-10-05T19:28:54.805 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:54.798+0000 7f9f6621fe40 1 -- 172.21.15.167:0/3004404278 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:39 5.f 5:f4c53578:::zonegroups_names.default:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32) v8 -- 0x563697b632f0 con 0x563697b335d0 2023-10-05T19:28:54.805 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:54.798+0000 7f9f64555700 1 -- 172.21.15.167:0/3004404278 <== osd.1 v2:172.21.15.167:6800/30035 15 ==== osd_op_reply(39 zonegroups_names.default [read 0~46 out=46b] v0'0 uv2 ondisk = 0) v8 ==== 168+0+46 (crc 0 0 0) 0x7f9f5402f080 con 0x563697b335d0 2023-10-05T19:28:54.805 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:54.798+0000 7f9f6621fe40 20 rados_obj.operate() r=0 bl.length=46 2023-10-05T19:28:54.805 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:54.798+0000 7f9f6621fe40 20 rados->read ofs=0 len=0 2023-10-05T19:28:54.805 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:54.798+0000 7f9f6621fe40 1 -- 172.21.15.167:0/3004404278 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:40 5.c 5:30d299a5:::zonegroup_info.4f1f719d-9648-417b-b10c-e5d939825419:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32) v8 -- 0x563697b672d0 con 0x7f9f4c007030 2023-10-05T19:28:54.806 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:54.798+0000 7f9f64555700 1 -- 172.21.15.167:0/3004404278 <== osd.0 v2:172.21.15.118:6800/27690 10 ==== osd_op_reply(40 zonegroup_info.4f1f719d-9648-417b-b10c-e5d939825419 [read 0~362 out=362b] v0'0 uv1 ondisk = 0) v8 ==== 195+0+362 (crc 0 0 0) 0x7f9f5402f080 con 0x7f9f4c007030 2023-10-05T19:28:54.806 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:54.802+0000 7f9f6621fe40 20 rados_obj.operate() r=0 bl.length=362 2023-10-05T19:28:54.806 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:54.802+0000 7f9f6621fe40 20 rados->read ofs=0 len=0 2023-10-05T19:28:54.806 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:54.802+0000 7f9f6621fe40 1 -- 172.21.15.167:0/3004404278 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:41 5.13 5:cbbf6c60:::converted:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32) v8 -- 0x563697b672d0 con 0x563697b64440 2023-10-05T19:28:54.807 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:54.802+0000 7f9f63d54700 1 -- 172.21.15.167:0/3004404278 <== osd.3 v2:172.21.15.167:6808/30036 10 ==== osd_op_reply(41 converted [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 ==== 153+0+0 (crc 0 0 0) 0x7f9f5c024020 con 0x563697b64440 2023-10-05T19:28:54.807 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:54.802+0000 7f9f6621fe40 20 rados_obj.operate() r=-2 bl.length=0 2023-10-05T19:28:54.807 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:54.802+0000 7f9f6621fe40 20 rados->read ofs=0 len=0 2023-10-05T19:28:54.807 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:54.802+0000 7f9f6621fe40 1 -- 172.21.15.167:0/3004404278 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:42 5.12 5:49953fa1:::default.realm:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32) v8 -- 0x563697b67680 con 0x563697b335d0 2023-10-05T19:28:54.807 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:54.802+0000 7f9f64555700 1 -- 172.21.15.167:0/3004404278 <== osd.1 v2:172.21.15.167:6800/30035 16 ==== osd_op_reply(42 default.realm [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 ==== 157+0+0 (crc 0 0 0) 0x7f9f5402f080 con 0x563697b335d0 2023-10-05T19:28:54.807 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:54.802+0000 7f9f6621fe40 20 rados_obj.operate() r=-2 bl.length=0 2023-10-05T19:28:54.807 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:54.802+0000 7f9f6621fe40 20 rados->read ofs=0 len=0 2023-10-05T19:28:54.808 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:54.802+0000 7f9f6621fe40 1 -- 172.21.15.167:0/3004404278 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:43 5.f 5:f4c53578:::zonegroups_names.default:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32) v8 -- 0x563697b62b20 con 0x563697b335d0 2023-10-05T19:28:54.808 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:54.802+0000 7f9f64555700 1 -- 172.21.15.167:0/3004404278 <== osd.1 v2:172.21.15.167:6800/30035 17 ==== osd_op_reply(43 zonegroups_names.default [read 0~46 out=46b] v0'0 uv2 ondisk = 0) v8 ==== 168+0+46 (crc 0 0 0) 0x7f9f5402f080 con 0x563697b335d0 2023-10-05T19:28:54.808 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:54.802+0000 7f9f6621fe40 20 rados_obj.operate() r=0 bl.length=46 2023-10-05T19:28:54.808 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:54.802+0000 7f9f6621fe40 1 -- 172.21.15.167:0/3004404278 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:44 5.0 5:00000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x563697b62af0 con 0x563697b64440 2023-10-05T19:28:54.809 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:54.802+0000 7f9f63d54700 1 -- 172.21.15.167:0/3004404278 <== osd.3 v2:172.21.15.167:6808/30036 11 ==== osd_op_reply(44 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f9f5c024020 con 0x563697b64440 2023-10-05T19:28:54.809 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:54.802+0000 7f9f63d54700 1 -- 172.21.15.167:0/3004404278 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:45 5.10 5:08000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x563697b62af0 con 0x7f9f5c010c30 2023-10-05T19:28:54.810 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:54.806+0000 7f9f63553700 1 -- 172.21.15.167:0/3004404278 <== osd.2 v2:172.21.15.167:6816/30037 7 ==== osd_op_reply(45 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f9f4c029040 con 0x7f9f5c010c30 2023-10-05T19:28:54.810 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:54.806+0000 7f9f63553700 1 -- 172.21.15.167:0/3004404278 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:46 5.8 5:10000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x563697b62af0 con 0x7f9f4c007030 2023-10-05T19:28:54.811 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:54.806+0000 7f9f64555700 1 -- 172.21.15.167:0/3004404278 <== osd.0 v2:172.21.15.118:6800/27690 11 ==== osd_op_reply(46 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f9f5402f080 con 0x7f9f4c007030 2023-10-05T19:28:54.811 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:54.806+0000 7f9f64555700 1 -- 172.21.15.167:0/3004404278 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:47 5.18 5:18000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x563697b632f0 con 0x563697b335d0 2023-10-05T19:28:54.812 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:54.806+0000 7f9f64555700 1 -- 172.21.15.167:0/3004404278 <== osd.1 v2:172.21.15.167:6800/30035 18 ==== osd_op_reply(47 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f9f5402f080 con 0x563697b335d0 2023-10-05T19:28:54.812 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:54.806+0000 7f9f64555700 1 -- 172.21.15.167:0/3004404278 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:48 5.4 5:20000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x563697b632f0 con 0x563697b335d0 2023-10-05T19:28:54.812 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:54.806+0000 7f9f64555700 1 -- 172.21.15.167:0/3004404278 <== osd.1 v2:172.21.15.167:6800/30035 19 ==== osd_op_reply(48 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f9f5402f080 con 0x563697b335d0 2023-10-05T19:28:54.812 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:54.806+0000 7f9f64555700 1 -- 172.21.15.167:0/3004404278 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:49 5.14 5:28000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x563697b632f0 con 0x7f9f4c007030 2023-10-05T19:28:54.813 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:54.806+0000 7f9f64555700 1 -- 172.21.15.167:0/3004404278 <== osd.0 v2:172.21.15.118:6800/27690 12 ==== osd_op_reply(49 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f9f5402f080 con 0x7f9f4c007030 2023-10-05T19:28:54.813 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:54.806+0000 7f9f64555700 1 -- 172.21.15.167:0/3004404278 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:50 5.c 5:30000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x563697b632f0 con 0x7f9f4c007030 2023-10-05T19:28:54.814 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:54.810+0000 7f9f64555700 1 -- 172.21.15.167:0/3004404278 <== osd.0 v2:172.21.15.118:6800/27690 13 ==== osd_op_reply(50 [pgnls start_epoch 32 out=112b] v23'1 uv1 ondisk = 1) v8 ==== 144+0+112 (crc 0 0 0) 0x7f9f5402f080 con 0x7f9f4c007030 2023-10-05T19:28:54.814 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:54.810+0000 7f9f64555700 1 -- 172.21.15.167:0/3004404278 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:51 5.1c 5:38000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x563697b632f0 con 0x7f9f5c010c30 2023-10-05T19:28:54.814 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:54.810+0000 7f9f63553700 1 -- 172.21.15.167:0/3004404278 <== osd.2 v2:172.21.15.167:6816/30037 8 ==== osd_op_reply(51 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f9f4c029040 con 0x7f9f5c010c30 2023-10-05T19:28:54.814 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:54.810+0000 7f9f63553700 1 -- 172.21.15.167:0/3004404278 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:52 5.2 5:40000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x563697b632f0 con 0x563697b64440 2023-10-05T19:28:54.815 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:54.810+0000 7f9f63d54700 1 -- 172.21.15.167:0/3004404278 <== osd.3 v2:172.21.15.167:6808/30036 12 ==== osd_op_reply(52 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f9f5c024020 con 0x563697b64440 2023-10-05T19:28:54.815 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:54.810+0000 7f9f63d54700 1 -- 172.21.15.167:0/3004404278 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:53 5.12 5:48000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x563697b632f0 con 0x563697b335d0 2023-10-05T19:28:54.815 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:54.810+0000 7f9f64555700 1 -- 172.21.15.167:0/3004404278 <== osd.1 v2:172.21.15.167:6800/30035 20 ==== osd_op_reply(53 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f9f5402f080 con 0x563697b335d0 2023-10-05T19:28:54.815 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:54.810+0000 7f9f64555700 1 -- 172.21.15.167:0/3004404278 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:54 5.a 5:50000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x563697b632f0 con 0x7f9f4c007030 2023-10-05T19:28:54.816 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:54.810+0000 7f9f64555700 1 -- 172.21.15.167:0/3004404278 <== osd.0 v2:172.21.15.118:6800/27690 14 ==== osd_op_reply(54 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f9f5402f080 con 0x7f9f4c007030 2023-10-05T19:28:54.816 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:54.810+0000 7f9f64555700 1 -- 172.21.15.167:0/3004404278 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:55 5.1a 5:58000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x563697b632f0 con 0x563697b335d0 2023-10-05T19:28:54.816 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:54.810+0000 7f9f64555700 1 -- 172.21.15.167:0/3004404278 <== osd.1 v2:172.21.15.167:6800/30035 21 ==== osd_op_reply(55 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f9f5402f080 con 0x563697b335d0 2023-10-05T19:28:54.816 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:54.810+0000 7f9f64555700 1 -- 172.21.15.167:0/3004404278 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:56 5.6 5:60000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x563697b632f0 con 0x7f9f5c010c30 2023-10-05T19:28:54.817 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:54.810+0000 7f9f63553700 1 -- 172.21.15.167:0/3004404278 <== osd.2 v2:172.21.15.167:6816/30037 9 ==== osd_op_reply(56 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f9f4c029040 con 0x7f9f5c010c30 2023-10-05T19:28:54.817 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:54.810+0000 7f9f63553700 1 -- 172.21.15.167:0/3004404278 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:57 5.16 5:68000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x563697b632f0 con 0x563697b64440 2023-10-05T19:28:54.818 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:54.810+0000 7f9f63d54700 1 -- 172.21.15.167:0/3004404278 <== osd.3 v2:172.21.15.167:6808/30036 13 ==== osd_op_reply(57 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f9f5c024020 con 0x563697b64440 2023-10-05T19:28:54.818 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:54.814+0000 7f9f63d54700 1 -- 172.21.15.167:0/3004404278 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:58 5.e 5:70000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x563697b632f0 con 0x563697b64440 2023-10-05T19:28:54.818 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:54.814+0000 7f9f63d54700 1 -- 172.21.15.167:0/3004404278 <== osd.3 v2:172.21.15.167:6808/30036 14 ==== osd_op_reply(58 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f9f5c024020 con 0x563697b64440 2023-10-05T19:28:54.818 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:54.814+0000 7f9f63d54700 1 -- 172.21.15.167:0/3004404278 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:59 5.1e 5:78000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x563697b632f0 con 0x7f9f4c007030 2023-10-05T19:28:54.818 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:54.814+0000 7f9f64555700 1 -- 172.21.15.167:0/3004404278 <== osd.0 v2:172.21.15.118:6800/27690 15 ==== osd_op_reply(59 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f9f5402f080 con 0x7f9f4c007030 2023-10-05T19:28:54.819 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:54.814+0000 7f9f64555700 1 -- 172.21.15.167:0/3004404278 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:60 5.1 5:80000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x563697b632f0 con 0x563697b64440 2023-10-05T19:28:54.819 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:54.814+0000 7f9f63d54700 1 -- 172.21.15.167:0/3004404278 <== osd.3 v2:172.21.15.167:6808/30036 15 ==== osd_op_reply(60 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f9f5c024020 con 0x563697b64440 2023-10-05T19:28:54.819 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:54.814+0000 7f9f63d54700 1 -- 172.21.15.167:0/3004404278 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:61 5.11 5:88000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x563697b632f0 con 0x563697b64440 2023-10-05T19:28:54.820 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:54.814+0000 7f9f63d54700 1 -- 172.21.15.167:0/3004404278 <== osd.3 v2:172.21.15.167:6808/30036 16 ==== osd_op_reply(61 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f9f5c024020 con 0x563697b64440 2023-10-05T19:28:54.820 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:54.814+0000 7f9f63d54700 1 -- 172.21.15.167:0/3004404278 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:62 5.9 5:90000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x563697b632f0 con 0x563697b335d0 2023-10-05T19:28:54.820 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:54.814+0000 7f9f64555700 1 -- 172.21.15.167:0/3004404278 <== osd.1 v2:172.21.15.167:6800/30035 22 ==== osd_op_reply(62 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f9f5402f080 con 0x563697b335d0 2023-10-05T19:28:54.821 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:54.814+0000 7f9f64555700 1 -- 172.21.15.167:0/3004404278 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:63 5.19 5:98000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x563697b632f0 con 0x563697b335d0 2023-10-05T19:28:54.821 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:54.814+0000 7f9f64555700 1 -- 172.21.15.167:0/3004404278 <== osd.1 v2:172.21.15.167:6800/30035 23 ==== osd_op_reply(63 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f9f5402f080 con 0x563697b335d0 2023-10-05T19:28:54.821 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:54.814+0000 7f9f64555700 1 -- 172.21.15.167:0/3004404278 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:64 5.5 5:a0000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x563697b632f0 con 0x7f9f4c007030 2023-10-05T19:28:54.822 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:54.818+0000 7f9f64555700 1 -- 172.21.15.167:0/3004404278 <== osd.0 v2:172.21.15.118:6800/27690 16 ==== osd_op_reply(64 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f9f5402f080 con 0x7f9f4c007030 2023-10-05T19:28:54.822 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:54.818+0000 7f9f64555700 1 -- 172.21.15.167:0/3004404278 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:65 5.15 5:a8000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x563697b632f0 con 0x563697b64440 2023-10-05T19:28:54.822 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:54.818+0000 7f9f63d54700 1 -- 172.21.15.167:0/3004404278 <== osd.3 v2:172.21.15.167:6808/30036 17 ==== osd_op_reply(65 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f9f5c024020 con 0x563697b64440 2023-10-05T19:28:54.823 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:54.818+0000 7f9f63d54700 1 -- 172.21.15.167:0/3004404278 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:66 5.d 5:b0000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x563697b632f0 con 0x7f9f5c010c30 2023-10-05T19:28:54.823 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:54.818+0000 7f9f63553700 1 -- 172.21.15.167:0/3004404278 <== osd.2 v2:172.21.15.167:6816/30037 10 ==== osd_op_reply(66 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f9f4c029040 con 0x7f9f5c010c30 2023-10-05T19:28:54.823 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:54.818+0000 7f9f63553700 1 -- 172.21.15.167:0/3004404278 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:67 5.1d 5:b8000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x563697b632f0 con 0x563697b335d0 2023-10-05T19:28:54.824 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:54.818+0000 7f9f64555700 1 -- 172.21.15.167:0/3004404278 <== osd.1 v2:172.21.15.167:6800/30035 24 ==== osd_op_reply(67 [pgnls start_epoch 32 out=107b] v30'2 uv2 ondisk = 1) v8 ==== 144+0+107 (crc 0 0 0) 0x7f9f5402f080 con 0x563697b335d0 2023-10-05T19:28:54.824 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:54.818+0000 7f9f64555700 1 -- 172.21.15.167:0/3004404278 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:68 5.3 5:c0000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x563697b632f0 con 0x7f9f4c007030 2023-10-05T19:28:54.824 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:54.818+0000 7f9f64555700 1 -- 172.21.15.167:0/3004404278 <== osd.0 v2:172.21.15.118:6800/27690 17 ==== osd_op_reply(68 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f9f5402f080 con 0x7f9f4c007030 2023-10-05T19:28:54.824 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:54.818+0000 7f9f64555700 1 -- 172.21.15.167:0/3004404278 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:69 5.13 5:c8000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x563697b632f0 con 0x563697b64440 2023-10-05T19:28:54.825 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:54.818+0000 7f9f63d54700 1 -- 172.21.15.167:0/3004404278 <== osd.3 v2:172.21.15.167:6808/30036 18 ==== osd_op_reply(69 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f9f5c024020 con 0x563697b64440 2023-10-05T19:28:54.825 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:54.818+0000 7f9f63d54700 1 -- 172.21.15.167:0/3004404278 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:70 5.b 5:d0000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x563697b632f0 con 0x7f9f4c007030 2023-10-05T19:28:54.826 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:54.822+0000 7f9f64555700 1 -- 172.21.15.167:0/3004404278 <== osd.0 v2:172.21.15.118:6800/27690 18 ==== osd_op_reply(70 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f9f5402f080 con 0x7f9f4c007030 2023-10-05T19:28:54.826 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:54.822+0000 7f9f64555700 1 -- 172.21.15.167:0/3004404278 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:71 5.1b 5:d8000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x563697b632f0 con 0x7f9f5c010c30 2023-10-05T19:28:54.827 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:54.822+0000 7f9f63553700 1 -- 172.21.15.167:0/3004404278 <== osd.2 v2:172.21.15.167:6816/30037 11 ==== osd_op_reply(71 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f9f4c029040 con 0x7f9f5c010c30 2023-10-05T19:28:54.827 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:54.822+0000 7f9f63553700 1 -- 172.21.15.167:0/3004404278 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:72 5.7 5:e0000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x563697b632f0 con 0x563697b64440 2023-10-05T19:28:54.827 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:54.822+0000 7f9f63d54700 1 -- 172.21.15.167:0/3004404278 <== osd.3 v2:172.21.15.167:6808/30036 19 ==== osd_op_reply(72 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f9f5c024020 con 0x563697b64440 2023-10-05T19:28:54.827 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:54.822+0000 7f9f63d54700 1 -- 172.21.15.167:0/3004404278 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:73 5.17 5:e8000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x563697b632f0 con 0x7f9f4c007030 2023-10-05T19:28:54.828 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:54.822+0000 7f9f64555700 1 -- 172.21.15.167:0/3004404278 <== osd.0 v2:172.21.15.118:6800/27690 19 ==== osd_op_reply(73 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f9f5402f080 con 0x7f9f4c007030 2023-10-05T19:28:54.828 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:54.822+0000 7f9f64555700 1 -- 172.21.15.167:0/3004404278 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:74 5.f 5:f0000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x563697b632f0 con 0x563697b335d0 2023-10-05T19:28:54.829 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:54.822+0000 7f9f64555700 1 -- 172.21.15.167:0/3004404278 <== osd.1 v2:172.21.15.167:6800/30035 25 ==== osd_op_reply(74 [pgnls start_epoch 32 out=115b] v23'2 uv2 ondisk = 1) v8 ==== 144+0+115 (crc 0 0 0) 0x7f9f5402f080 con 0x563697b335d0 2023-10-05T19:28:54.829 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:54.822+0000 7f9f64555700 1 -- 172.21.15.167:0/3004404278 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:75 5.1f 5:f8000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x563697b632f0 con 0x7f9f5c010c30 2023-10-05T19:28:54.830 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:54.826+0000 7f9f63553700 1 -- 172.21.15.167:0/3004404278 <== osd.2 v2:172.21.15.167:6816/30037 12 ==== osd_op_reply(75 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f9f4c029040 con 0x7f9f5c010c30 2023-10-05T19:28:54.830 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:54.826+0000 7f9f6621fe40 20 RGWRados::pool_iterate: got zonegroup_info.4f1f719d-9648-417b-b10c-e5d939825419 2023-10-05T19:28:54.830 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:54.826+0000 7f9f6621fe40 20 RGWRados::pool_iterate: got zone_info.9a9df965-6612-45d4-a45f-6e3c889bacc2 2023-10-05T19:28:54.830 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:54.826+0000 7f9f6621fe40 20 RGWRados::pool_iterate: got zone_names.default 2023-10-05T19:28:54.830 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:54.826+0000 7f9f6621fe40 20 RGWRados::pool_iterate: got zonegroups_names.default 2023-10-05T19:28:54.830 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:54.826+0000 7f9f6621fe40 20 rados->read ofs=0 len=0 2023-10-05T19:28:54.830 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:54.826+0000 7f9f6621fe40 1 -- 172.21.15.167:0/3004404278 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:76 5.f 5:f43b6ece:::zone_names.default:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32) v8 -- 0x563697b62af0 con 0x563697b335d0 2023-10-05T19:28:54.832 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:54.826+0000 7f9f64555700 1 -- 172.21.15.167:0/3004404278 <== osd.1 v2:172.21.15.167:6800/30035 26 ==== osd_op_reply(76 zone_names.default [read 0~46 out=46b] v0'0 uv1 ondisk = 0) v8 ==== 162+0+46 (crc 0 0 0) 0x7f9f5402f080 con 0x563697b335d0 2023-10-05T19:28:54.832 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:54.826+0000 7f9f6621fe40 20 rados_obj.operate() r=0 bl.length=46 2023-10-05T19:28:54.832 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:54.826+0000 7f9f6621fe40 20 rados->read ofs=0 len=0 2023-10-05T19:28:54.832 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:54.826+0000 7f9f6621fe40 1 -- 172.21.15.167:0/3004404278 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:77 5.1d 5:b8374eb4:::zone_info.9a9df965-6612-45d4-a45f-6e3c889bacc2:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32) v8 -- 0x563697b67bf0 con 0x563697b335d0 2023-10-05T19:28:54.832 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:54.826+0000 7f9f64555700 1 -- 172.21.15.167:0/3004404278 <== osd.1 v2:172.21.15.167:6800/30035 27 ==== osd_op_reply(77 zone_info.9a9df965-6612-45d4-a45f-6e3c889bacc2 [read 0~874 out=874b] v0'0 uv2 ondisk = 0) v8 ==== 190+0+874 (crc 0 0 0) 0x7f9f5402f080 con 0x563697b335d0 2023-10-05T19:28:54.832 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:54.826+0000 7f9f6621fe40 20 rados_obj.operate() r=0 bl.length=874 2023-10-05T19:28:54.832 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:54.826+0000 7f9f6621fe40 20 rados->read ofs=0 len=0 2023-10-05T19:28:54.832 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:54.826+0000 7f9f6621fe40 1 -- 172.21.15.167:0/3004404278 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:78 5.f 5:f4c53578:::zonegroups_names.default:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32) v8 -- 0x563697b680e0 con 0x563697b335d0 2023-10-05T19:28:54.833 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:54.826+0000 7f9f64555700 1 -- 172.21.15.167:0/3004404278 <== osd.1 v2:172.21.15.167:6800/30035 28 ==== osd_op_reply(78 zonegroups_names.default [read 0~46 out=46b] v0'0 uv2 ondisk = 0) v8 ==== 168+0+46 (crc 0 0 0) 0x7f9f5402f080 con 0x563697b335d0 2023-10-05T19:28:54.833 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:54.826+0000 7f9f6621fe40 20 rados_obj.operate() r=0 bl.length=46 2023-10-05T19:28:54.833 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:54.826+0000 7f9f6621fe40 20 rados->read ofs=0 len=0 2023-10-05T19:28:54.833 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:54.826+0000 7f9f6621fe40 1 -- 172.21.15.167:0/3004404278 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:79 5.c 5:30d299a5:::zonegroup_info.4f1f719d-9648-417b-b10c-e5d939825419:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32) v8 -- 0x563697b680e0 con 0x7f9f4c007030 2023-10-05T19:28:54.834 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:54.826+0000 7f9f64555700 1 -- 172.21.15.167:0/3004404278 <== osd.0 v2:172.21.15.118:6800/27690 20 ==== osd_op_reply(79 zonegroup_info.4f1f719d-9648-417b-b10c-e5d939825419 [read 0~362 out=362b] v0'0 uv1 ondisk = 0) v8 ==== 195+0+362 (crc 0 0 0) 0x7f9f5402f080 con 0x7f9f4c007030 2023-10-05T19:28:54.834 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:54.826+0000 7f9f6621fe40 20 rados_obj.operate() r=0 bl.length=362 2023-10-05T19:28:54.834 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:54.826+0000 7f9f6621fe40 20 rados->read ofs=0 len=0 2023-10-05T19:28:54.834 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:54.826+0000 7f9f6621fe40 1 -- 172.21.15.167:0/3004404278 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:80 5.3 5:c0ad8cbd:::region_map:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32) v8 -- 0x563697b681e0 con 0x7f9f4c007030 2023-10-05T19:28:54.834 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:54.830+0000 7f9f64555700 1 -- 172.21.15.167:0/3004404278 <== osd.0 v2:172.21.15.118:6800/27690 21 ==== osd_op_reply(80 region_map [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 ==== 154+0+0 (crc 0 0 0) 0x7f9f54050040 con 0x7f9f4c007030 2023-10-05T19:28:54.834 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:54.830+0000 7f9f6621fe40 20 rados_obj.operate() r=-2 bl.length=0 2023-10-05T19:28:54.834 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:54.830+0000 7f9f6621fe40 20 zone default found 2023-10-05T19:28:54.834 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:54.830+0000 7f9f6621fe40 4 Realm: () 2023-10-05T19:28:54.835 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:54.830+0000 7f9f6621fe40 4 ZoneGroup: default (4f1f719d-9648-417b-b10c-e5d939825419) 2023-10-05T19:28:54.835 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:54.830+0000 7f9f6621fe40 4 Zone: default (9a9df965-6612-45d4-a45f-6e3c889bacc2) 2023-10-05T19:28:54.835 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:54.830+0000 7f9f6621fe40 10 cannot find current period zonegroup using local zonegroup configuration 2023-10-05T19:28:54.835 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:54.830+0000 7f9f6621fe40 20 zonegroup default 2023-10-05T19:28:54.835 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:54.830+0000 7f9f6621fe40 20 rados->read ofs=0 len=0 2023-10-05T19:28:54.835 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:54.830+0000 7f9f6621fe40 1 -- 172.21.15.167:0/3004404278 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:81 5.3 5:c52100b6:::period_config.default:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32) v8 -- 0x563697b681e0 con 0x7f9f4c007030 2023-10-05T19:28:54.836 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:54.830+0000 7f9f64555700 1 -- 172.21.15.167:0/3004404278 <== osd.0 v2:172.21.15.118:6800/27690 22 ==== osd_op_reply(81 period_config.default [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 ==== 165+0+0 (crc 0 0 0) 0x7f9f54050040 con 0x7f9f4c007030 2023-10-05T19:28:54.836 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:54.830+0000 7f9f6621fe40 20 rados_obj.operate() r=-2 bl.length=0 2023-10-05T19:28:54.836 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:54.830+0000 7f9f6621fe40 20 rados->read ofs=0 len=0 2023-10-05T19:28:54.837 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:54.830+0000 7f9f6621fe40 1 -- 172.21.15.167:0/3004404278 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:82 6.2 6:4347d321:::bucket.sync-source-hints.:head [call version.read in=11b,read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32) v8 -- 0x563697b69b60 con 0x563697b64440 2023-10-05T19:28:54.837 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:54.830+0000 7f9f63d54700 1 -- 172.21.15.167:0/3004404278 <== osd.3 v2:172.21.15.167:6808/30036 20 ==== osd_op_reply(82 bucket.sync-source-hints. [call,read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 ==== 211+0+0 (crc 0 0 0) 0x7f9f5c024020 con 0x563697b64440 2023-10-05T19:28:54.837 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:54.830+0000 7f9f6621fe40 20 rados_obj.operate() r=-2 bl.length=0 2023-10-05T19:28:54.837 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:54.830+0000 7f9f6621fe40 20 rados->read ofs=0 len=0 2023-10-05T19:28:54.837 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:54.830+0000 7f9f6621fe40 1 -- 172.21.15.167:0/3004404278 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:83 6.b 6:d467b91b:::bucket.sync-target-hints.:head [call version.read in=11b,read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32) v8 -- 0x563697b69f10 con 0x563697b64440 2023-10-05T19:28:54.837 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:54.830+0000 7f9f63d54700 1 -- 172.21.15.167:0/3004404278 <== osd.3 v2:172.21.15.167:6808/30036 21 ==== osd_op_reply(83 bucket.sync-target-hints. [call,read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 ==== 211+0+0 (crc 0 0 0) 0x7f9f5c024020 con 0x563697b64440 2023-10-05T19:28:54.837 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:54.830+0000 7f9f6621fe40 20 rados_obj.operate() r=-2 bl.length=0 2023-10-05T19:28:54.837 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:54.830+0000 7f9f6621fe40 20 started sync module instance, tier type = 2023-10-05T19:28:54.837 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:54.830+0000 7f9f6621fe40 20 started zone id=9a9df965-6612-45d4-a45f-6e3c889bacc2 (name=default) with tier type = 2023-10-05T19:28:54.837 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:54.830+0000 7f9f6621fe40 1 -- 172.21.15.167:0/3004404278 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:84 7.1f 7:f95f44c2:::notify.0:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e32) v8 -- 0x563697b69220 con 0x7f9f5c010c30 2023-10-05T19:28:54.838 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:54.834+0000 7f9f63553700 1 -- 172.21.15.167:0/3004404278 <== osd.2 v2:172.21.15.167:6816/30037 13 ==== osd_op_reply(84 notify.0 [create] v32'192 uv192 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f9f4c029040 con 0x7f9f5c010c30 2023-10-05T19:28:54.838 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:54.834+0000 7f9f6621fe40 1 -- 172.21.15.167:0/3004404278 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:85 7.1f 7:f95f44c2:::notify.0:head [watch watch cookie 94792473546032] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e32) v8 -- 0x563697b6a2c0 con 0x7f9f5c010c30 2023-10-05T19:28:54.838 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:54.834+0000 7f9f6621fe40 1 -- 172.21.15.167:0/3004404278 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:86 7.0 7:05bf5b68:::notify.1:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e32) v8 -- 0x563697b6ab90 con 0x563697b335d0 2023-10-05T19:28:54.840 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:54.834+0000 7f9f63553700 1 -- 172.21.15.167:0/3004404278 <== osd.2 v2:172.21.15.167:6816/30037 14 ==== osd_op_reply(85 notify.0 [watch watch cookie 94792473546032] v32'193 uv192 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f9f4c029040 con 0x7f9f5c010c30 2023-10-05T19:28:54.840 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:54.834+0000 7f9f64555700 1 -- 172.21.15.167:0/3004404278 <== osd.1 v2:172.21.15.167:6800/30035 29 ==== osd_op_reply(86 notify.1 [create] v32'192 uv192 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f9f54050040 con 0x563697b335d0 2023-10-05T19:28:54.841 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:54.834+0000 7f9f6621fe40 1 -- 172.21.15.167:0/3004404278 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:87 7.0 7:05bf5b68:::notify.1:head [watch watch cookie 94792473552544] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e32) v8 -- 0x563697b6aaf0 con 0x563697b335d0 2023-10-05T19:28:54.841 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:54.834+0000 7f9f6621fe40 1 -- 172.21.15.167:0/3004404278 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:88 7.15 7:a93a5511:::notify.2:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e32) v8 -- 0x563697b6b970 con 0x563697b64440 2023-10-05T19:28:54.844 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:54.838+0000 7f9f63d54700 1 -- 172.21.15.167:0/3004404278 <== osd.3 v2:172.21.15.167:6808/30036 22 ==== osd_op_reply(88 notify.2 [create] v32'192 uv192 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f9f5c024020 con 0x563697b64440 2023-10-05T19:28:54.844 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:54.838+0000 7f9f6621fe40 1 -- 172.21.15.167:0/3004404278 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:89 7.15 7:a93a5511:::notify.2:head [watch watch cookie 94792473556096] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e32) v8 -- 0x563697b6aaf0 con 0x563697b64440 2023-10-05T19:28:54.844 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:54.838+0000 7f9f6621fe40 1 -- 172.21.15.167:0/3004404278 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:90 7.e 7:7759931f:::notify.3:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e32) v8 -- 0x563697b6c9f0 con 0x7f9f5c010c30 2023-10-05T19:28:54.844 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:54.838+0000 7f9f64555700 1 -- 172.21.15.167:0/3004404278 <== osd.1 v2:172.21.15.167:6800/30035 30 ==== osd_op_reply(87 notify.1 [watch watch cookie 94792473552544] v32'193 uv192 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f9f54050040 con 0x563697b335d0 2023-10-05T19:28:54.846 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:54.842+0000 7f9f63553700 1 -- 172.21.15.167:0/3004404278 <== osd.2 v2:172.21.15.167:6816/30037 15 ==== osd_op_reply(90 notify.3 [create] v32'192 uv192 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f9f4c029040 con 0x7f9f5c010c30 2023-10-05T19:28:54.846 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:54.842+0000 7f9f6621fe40 1 -- 172.21.15.167:0/3004404278 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:91 7.e 7:7759931f:::notify.3:head [watch watch cookie 94792473554832] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e32) v8 -- 0x563697b6c620 con 0x7f9f5c010c30 2023-10-05T19:28:54.846 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:54.842+0000 7f9f6621fe40 1 -- 172.21.15.167:0/3004404278 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:92 7.d 7:b4812045:::notify.4:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e32) v8 -- 0x563697b6dac0 con 0x563697b335d0 2023-10-05T19:28:54.847 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:54.842+0000 7f9f63d54700 1 -- 172.21.15.167:0/3004404278 <== osd.3 v2:172.21.15.167:6808/30036 23 ==== osd_op_reply(89 notify.2 [watch watch cookie 94792473556096] v32'193 uv192 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f9f5c024020 con 0x563697b64440 2023-10-05T19:28:54.849 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:54.842+0000 7f9f64555700 1 -- 172.21.15.167:0/3004404278 <== osd.1 v2:172.21.15.167:6800/30035 31 ==== osd_op_reply(92 notify.4 [create] v32'192 uv192 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f9f54050040 con 0x563697b335d0 2023-10-05T19:28:54.849 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:54.842+0000 7f9f63553700 1 -- 172.21.15.167:0/3004404278 <== osd.2 v2:172.21.15.167:6816/30037 16 ==== osd_op_reply(91 notify.3 [watch watch cookie 94792473554832] v32'193 uv192 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f9f4c029040 con 0x7f9f5c010c30 2023-10-05T19:28:54.849 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:54.842+0000 7f9f6621fe40 1 -- 172.21.15.167:0/3004404278 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:93 7.d 7:b4812045:::notify.4:head [watch watch cookie 94792473562608] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e32) v8 -- 0x563697b6d590 con 0x563697b335d0 2023-10-05T19:28:54.849 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:54.842+0000 7f9f6621fe40 1 -- 172.21.15.167:0/3004404278 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:94 7.3 7:c609908c:::notify.5:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e32) v8 -- 0x563697b6e3b0 con 0x7f9f4c007030 2023-10-05T19:28:54.852 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:54.846+0000 7f9f64555700 1 -- 172.21.15.167:0/3004404278 <== osd.0 v2:172.21.15.118:6800/27690 23 ==== osd_op_reply(94 notify.5 [create] v32'192 uv192 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f9f54050040 con 0x7f9f4c007030 2023-10-05T19:28:54.852 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:54.846+0000 7f9f6621fe40 1 -- 172.21.15.167:0/3004404278 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:95 7.3 7:c609908c:::notify.5:head [watch watch cookie 94792473566832] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e32) v8 -- 0x563697b6e7b0 con 0x7f9f4c007030 2023-10-05T19:28:54.852 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:54.846+0000 7f9f6621fe40 1 -- 172.21.15.167:0/3004404278 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:96 7.14 7:2b04a3e9:::notify.6:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e32) v8 -- 0x563697b6f200 con 0x7f9f4c007030 2023-10-05T19:28:54.852 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:54.846+0000 7f9f64555700 1 -- 172.21.15.167:0/3004404278 <== osd.1 v2:172.21.15.167:6800/30035 32 ==== osd_op_reply(93 notify.4 [watch watch cookie 94792473562608] v32'193 uv192 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f9f54050040 con 0x563697b335d0 2023-10-05T19:28:54.854 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:54.850+0000 7f9f64555700 1 -- 172.21.15.167:0/3004404278 <== osd.0 v2:172.21.15.118:6800/27690 24 ==== osd_op_reply(95 notify.5 [watch watch cookie 94792473566832] v32'193 uv192 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f9f54050040 con 0x7f9f4c007030 2023-10-05T19:28:54.855 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:54.850+0000 7f9f64555700 1 -- 172.21.15.167:0/3004404278 <== osd.0 v2:172.21.15.118:6800/27690 25 ==== osd_op_reply(96 notify.6 [create] v32'192 uv192 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f9f54050040 con 0x7f9f4c007030 2023-10-05T19:28:54.855 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:54.850+0000 7f9f6621fe40 1 -- 172.21.15.167:0/3004404278 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:97 7.14 7:2b04a3e9:::notify.6:head [watch watch cookie 94792473558304] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e32) v8 -- 0x563697b6e2c0 con 0x7f9f4c007030 2023-10-05T19:28:54.855 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:54.850+0000 7f9f6621fe40 1 -- 172.21.15.167:0/3004404278 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:98 7.9 7:93e5b521:::notify.7:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e32) v8 -- 0x563697b6f920 con 0x7f9f4c007030 2023-10-05T19:28:54.857 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:54.850+0000 7f9f64555700 1 -- 172.21.15.167:0/3004404278 <== osd.0 v2:172.21.15.118:6800/27690 26 ==== osd_op_reply(97 notify.6 [watch watch cookie 94792473558304] v32'193 uv192 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f9f54050040 con 0x7f9f4c007030 2023-10-05T19:28:54.858 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:54.854+0000 7f9f64555700 1 -- 172.21.15.167:0/3004404278 <== osd.0 v2:172.21.15.118:6800/27690 27 ==== osd_op_reply(98 notify.7 [create] v32'192 uv192 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f9f54050040 con 0x7f9f4c007030 2023-10-05T19:28:54.858 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:54.854+0000 7f9f6621fe40 1 -- 172.21.15.167:0/3004404278 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:99 7.9 7:93e5b521:::notify.7:head [watch watch cookie 94792473570512] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e32) v8 -- 0x563697b6f920 con 0x7f9f4c007030 2023-10-05T19:28:54.858 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:54.854+0000 7f9f6621fe40 20 add_watcher() i=0 2023-10-05T19:28:54.858 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:54.854+0000 7f9f6621fe40 20 add_watcher() i=1 2023-10-05T19:28:54.858 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:54.854+0000 7f9f6621fe40 20 add_watcher() i=2 2023-10-05T19:28:54.858 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:54.854+0000 7f9f6621fe40 20 add_watcher() i=3 2023-10-05T19:28:54.858 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:54.854+0000 7f9f6621fe40 20 add_watcher() i=4 2023-10-05T19:28:54.859 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:54.854+0000 7f9f6621fe40 20 add_watcher() i=5 2023-10-05T19:28:54.859 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:54.854+0000 7f9f6621fe40 20 add_watcher() i=6 2023-10-05T19:28:54.860 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:54.854+0000 7f9f64555700 1 -- 172.21.15.167:0/3004404278 <== osd.0 v2:172.21.15.118:6800/27690 28 ==== osd_op_reply(99 notify.7 [watch watch cookie 94792473570512] v32'193 uv192 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f9f54050040 con 0x7f9f4c007030 2023-10-05T19:28:54.860 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:54.854+0000 7f9f6621fe40 20 add_watcher() i=7 2023-10-05T19:28:54.860 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:54.854+0000 7f9f6621fe40 2 all 8 watchers are set, enabling cache 2023-10-05T19:28:54.860 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:54.854+0000 7f9f6621fe40 1 -- 172.21.15.167:0/3004404278 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:100 6.4 6:22d26bf9:::data_loggenerations_metadata:head [call version.check_conds in=74b,call version.read in=11b,read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32) v8 -- 0x563697b72950 con 0x563697b335d0 2023-10-05T19:28:54.861 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:54.854+0000 7f9f64555700 1 -- 172.21.15.167:0/3004404278 <== osd.1 v2:172.21.15.167:6800/30035 33 ==== osd_op_reply(100 data_loggenerations_metadata [call,call out=48b,read 0~28 out=28b] v0'0 uv1 ondisk = 0) v8 ==== 256+0+76 (crc 0 0 0) 0x7f9f54050040 con 0x563697b335d0 2023-10-05T19:28:54.861 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:54.854+0000 7f9f6621fe40 1 -- 172.21.15.167:0/3004404278 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:101 6.4 6:22d26bf9:::data_loggenerations_metadata:head [watch watch cookie 94792473572400] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e32) v8 -- 0x563697b6fe80 con 0x563697b335d0 2023-10-05T19:28:54.863 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:54.858+0000 7f9f64555700 1 -- 172.21.15.167:0/3004404278 <== osd.1 v2:172.21.15.167:6800/30035 34 ==== osd_op_reply(101 data_loggenerations_metadata [watch watch cookie 94792473572400] v32'179 uv1 ondisk = 0) v8 ==== 172+0+0 (crc 0 0 0) 0x7f9f54050040 con 0x563697b335d0 2023-10-05T19:28:54.864 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:54.858+0000 7f9f357e2700 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2023-10-05T19:28:54.864 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:54.858+0000 7f9f6621fe40 20 check_secure_mon_conn(): auth registy supported: methods=[2] modes=[2,1] 2023-10-05T19:28:54.864 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:54.858+0000 7f9f6621fe40 20 check_secure_mon_conn(): mode 1 is insecure 2023-10-05T19:28:54.864 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:54.858+0000 7f9f6621fe40 1 Processor -- start 2023-10-05T19:28:54.864 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:54.858+0000 7f9f6621fe40 1 -- start start 2023-10-05T19:28:54.865 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:54.858+0000 7f9f6621fe40 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x563697b7d160 0x563697b7d530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-05T19:28:54.865 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:54.858+0000 7f9f6621fe40 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x563697b7da70 con 0x563697b7d160 2023-10-05T19:28:54.865 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:54.858+0000 7f9f63553700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x563697b7d160 0x563697b7d530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-05T19:28:54.865 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:54.858+0000 7f9f63553700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x563697b7d160 0x563697b7d530 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.118:3300/0 says I am v2:172.21.15.167:53192/0 (socket says 172.21.15.167:53192) 2023-10-05T19:28:54.865 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:54.858+0000 7f9f63553700 1 -- 172.21.15.167:0/318356604 learned_addr learned my addr 172.21.15.167:0/318356604 (peer_addr_for_me v2:172.21.15.167:0/0) 2023-10-05T19:28:54.866 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:54.862+0000 7f9f63553700 1 -- 172.21.15.167:0/318356604 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x563697b7dbb0 con 0x563697b7d160 2023-10-05T19:28:54.866 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:54.862+0000 7f9f63553700 1 --2- 172.21.15.167:0/318356604 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x563697b7d160 0x563697b7d530 secure :-1 s=READY pgs=522 cs=0 l=1 rev1=1 crypto rx=0x563697b30070 tx=0x7f9f4c030040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9b46543f5f7343ac server_cookie=0 in_seq=0 out_seq=0 2023-10-05T19:28:54.868 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:54.862+0000 7f9f34fe1700 1 -- 172.21.15.167:0/318356604 <== mon.0 v2:172.21.15.118:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f9f4c025040 con 0x563697b7d160 2023-10-05T19:28:54.869 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:54.862+0000 7f9f34fe1700 1 -- 172.21.15.167:0/318356604 <== mon.0 v2:172.21.15.118:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9f4c034050 con 0x563697b7d160 2023-10-05T19:28:54.869 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:54.862+0000 7f9f34fe1700 1 -- 172.21.15.167:0/318356604 <== mon.0 v2:172.21.15.118:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f9f4c02e4e0 con 0x563697b7d160 2023-10-05T19:28:54.869 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:54.862+0000 7f9f6621fe40 1 -- 172.21.15.167:0/318356604 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x563697b7d160 msgr2=0x563697b7d530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-05T19:28:54.869 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:54.862+0000 7f9f6621fe40 1 --2- 172.21.15.167:0/318356604 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x563697b7d160 0x563697b7d530 secure :-1 s=READY pgs=522 cs=0 l=1 rev1=1 crypto rx=0x563697b30070 tx=0x7f9f4c030040 comp rx=0 tx=0).stop 2023-10-05T19:28:54.869 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:54.862+0000 7f9f6621fe40 1 -- 172.21.15.167:0/318356604 shutdown_connections 2023-10-05T19:28:54.869 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:54.862+0000 7f9f6621fe40 1 --2- 172.21.15.167:0/318356604 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x563697b7d160 0x563697b7d530 unknown :-1 s=CLOSED pgs=522 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-05T19:28:54.869 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:54.862+0000 7f9f6621fe40 1 -- 172.21.15.167:0/318356604 >> 172.21.15.167:0/318356604 conn(0x563697b78520 msgr2=0x563697b76670 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-05T19:28:54.869 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:54.862+0000 7f9f6621fe40 1 -- 172.21.15.167:0/318356604 shutdown_connections 2023-10-05T19:28:54.870 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:54.862+0000 7f9f6621fe40 1 -- 172.21.15.167:0/318356604 wait complete. 2023-10-05T19:28:54.870 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:54.866+0000 7f9f6621fe40 1 Processor -- start 2023-10-05T19:28:54.870 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:54.866+0000 7f9f6621fe40 1 -- start start 2023-10-05T19:28:54.870 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:54.866+0000 7f9f6621fe40 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x563697b7d160 0x563697b87090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-05T19:28:54.870 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:54.866+0000 7f9f6621fe40 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x563697b88090 con 0x563697b7d160 2023-10-05T19:28:54.871 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:54.866+0000 7f9f63553700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x563697b7d160 0x563697b87090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-05T19:28:54.871 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:54.866+0000 7f9f63553700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x563697b7d160 0x563697b87090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.118:3300/0 says I am v2:172.21.15.167:53194/0 (socket says 172.21.15.167:53194) 2023-10-05T19:28:54.871 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:54.866+0000 7f9f63553700 1 -- 172.21.15.167:0/344537782 learned_addr learned my addr 172.21.15.167:0/344537782 (peer_addr_for_me v2:172.21.15.167:0/0) 2023-10-05T19:28:54.871 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:54.866+0000 7f9f63553700 1 -- 172.21.15.167:0/344537782 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x563697b2fc00 con 0x563697b7d160 2023-10-05T19:28:54.872 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:54.866+0000 7f9f63553700 1 --2- 172.21.15.167:0/344537782 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x563697b7d160 0x563697b87090 secure :-1 s=READY pgs=523 cs=0 l=1 rev1=1 crypto rx=0x7f9f4c01f940 tx=0x7f9f4c038000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-05T19:28:54.872 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:54.866+0000 7f9eaeffd700 1 -- 172.21.15.167:0/344537782 <== mon.0 v2:172.21.15.118:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f9f4c025040 con 0x563697b7d160 2023-10-05T19:28:54.872 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:54.866+0000 7f9eaeffd700 1 -- 172.21.15.167:0/344537782 <== mon.0 v2:172.21.15.118:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9f4c039900 con 0x563697b7d160 2023-10-05T19:28:54.872 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:54.866+0000 7f9f6621fe40 1 -- 172.21.15.167:0/344537782 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x563697b87690 con 0x563697b7d160 2023-10-05T19:28:54.872 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:54.866+0000 7f9eaeffd700 1 -- 172.21.15.167:0/344537782 <== mon.0 v2:172.21.15.118:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f9f4c039a90 con 0x563697b7d160 2023-10-05T19:28:54.873 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:54.866+0000 7f9f6621fe40 1 -- 172.21.15.167:0/344537782 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x563697b87b30 con 0x563697b7d160 2023-10-05T19:28:54.873 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:54.866+0000 7f9eaeffd700 1 -- 172.21.15.167:0/344537782 <== mon.0 v2:172.21.15.118:3300/0 4 ==== mgrmap(e 5) v1 ==== 40329+0+0 (secure 0 0 0) 0x7f9f4c039bf0 con 0x563697b7d160 2023-10-05T19:28:54.874 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:54.870+0000 7f9eaeffd700 1 --2- 172.21.15.167:0/344537782 >> [v2:172.21.15.167:6824/29889,v1:172.21.15.167:6825/29889] conn(0x7f9ea4030c00 0x7f9ea40330b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-05T19:28:54.874 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:54.870+0000 7f9eaeffd700 1 -- 172.21.15.167:0/344537782 <== mon.0 v2:172.21.15.118:3300/0 5 ==== osd_map(32..32 src has 1..32) v4 ==== 6260+0+0 (secure 0 0 0) 0x7f9f4c04dd50 con 0x563697b7d160 2023-10-05T19:28:54.874 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:54.870+0000 7f9f64555700 1 --2- 172.21.15.167:0/344537782 >> [v2:172.21.15.167:6824/29889,v1:172.21.15.167:6825/29889] conn(0x7f9ea4030c00 0x7f9ea40330b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-05T19:28:54.874 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:54.870+0000 7f9f6621fe40 1 --2- 172.21.15.167:0/344537782 >> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] conn(0x563697b91ce0 0x563697b940f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-05T19:28:54.874 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:54.870+0000 7f9f6621fe40 1 -- 172.21.15.167:0/344537782 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:1 6.19 6:9a6dd8c0:gc::gc.0:head [create,call version.check_conds in=74b,call rgw_gc.rgw_gc_queue_init in=45b,call version.set in=34b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e32) v8 -- 0x563697b94790 con 0x563697b91ce0 2023-10-05T19:28:54.874 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:54.870+0000 7f9f63d54700 1 --2- 172.21.15.167:0/344537782 >> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] conn(0x563697b91ce0 0x563697b940f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-05T19:28:54.875 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:54.870+0000 7f9f64555700 1 --2- 172.21.15.167:0/344537782 >> [v2:172.21.15.167:6824/29889,v1:172.21.15.167:6825/29889] conn(0x7f9ea4030c00 0x7f9ea40330b0 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x563697b6fde0 tx=0x7f9f54000f90 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-05T19:28:54.875 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:54.870+0000 7f9f63d54700 1 --2- 172.21.15.167:0/344537782 >> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] conn(0x563697b91ce0 0x563697b940f0 crc :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-05T19:28:54.877 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:54.870+0000 7f9f63d54700 1 -- 172.21.15.167:0/344537782 <== osd.1 v2:172.21.15.167:6800/30035 1 ==== osd_op_reply(1 gc.0 [create,call,call,call] v32'31 uv3 ondisk = -125 ((125) Operation canceled)) v8 ==== 274+0+0 (crc 0 0 0) 0x7f9f5c024020 con 0x563697b91ce0 2023-10-05T19:28:54.877 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:54.870+0000 7f9f6621fe40 1 --2- 172.21.15.167:0/344537782 >> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] conn(0x563697b8d8c0 0x563697b97e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-05T19:28:54.877 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:54.870+0000 7f9f6621fe40 1 -- 172.21.15.167:0/344537782 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:2 6.18 6:1e85421f:gc::gc.1:head [create,call version.check_conds in=74b,call rgw_gc.rgw_gc_queue_init in=45b,call version.set in=34b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e32) v8 -- 0x563697b98340 con 0x563697b8d8c0 2023-10-05T19:28:54.878 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:54.874+0000 7f9f63553700 1 --2- 172.21.15.167:0/344537782 >> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] conn(0x563697b8d8c0 0x563697b97e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-05T19:28:54.878 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:54.874+0000 7f9f63553700 1 --2- 172.21.15.167:0/344537782 >> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] conn(0x563697b8d8c0 0x563697b97e00 crc :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-05T19:28:54.880 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:54.874+0000 7f9f63553700 1 -- 172.21.15.167:0/344537782 <== osd.0 v2:172.21.15.118:6800/27690 1 ==== osd_op_reply(2 gc.1 [create,call,call,call] v32'23 uv5 ondisk = -125 ((125) Operation canceled)) v8 ==== 274+0+0 (crc 0 0 0) 0x7f9f4c03f020 con 0x563697b8d8c0 2023-10-05T19:28:54.881 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:54.874+0000 7f9f6621fe40 1 --2- 172.21.15.167:0/344537782 >> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] conn(0x563697b95830 0x563697b9b9b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-05T19:28:54.881 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:54.874+0000 7f9f6621fe40 1 -- 172.21.15.167:0/344537782 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:3 6.b 6:d242335b:gc::gc.2:head [create,call version.check_conds in=74b,call rgw_gc.rgw_gc_queue_init in=45b,call version.set in=34b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e32) v8 -- 0x563697b9bf30 con 0x563697b95830 2023-10-05T19:28:54.881 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:54.874+0000 7f9f64555700 1 --2- 172.21.15.167:0/344537782 >> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] conn(0x563697b95830 0x563697b9b9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-05T19:28:54.881 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:54.874+0000 7f9f64555700 1 --2- 172.21.15.167:0/344537782 >> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] conn(0x563697b95830 0x563697b9b9b0 crc :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-05T19:28:54.883 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:54.878+0000 7f9f64555700 1 -- 172.21.15.167:0/344537782 <== osd.3 v2:172.21.15.167:6808/30036 1 ==== osd_op_reply(3 gc.2 [create,call,call,call] v32'19 uv3 ondisk = -125 ((125) Operation canceled)) v8 ==== 274+0+0 (crc 0 0 0) 0x7f9f54050040 con 0x563697b95830 2023-10-05T19:28:54.883 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:54.878+0000 7f9f6621fe40 1 -- 172.21.15.167:0/344537782 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:4 6.4 6:2166d8e1:gc::gc.3:head [create,call version.check_conds in=74b,call rgw_gc.rgw_gc_queue_init in=45b,call version.set in=34b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e32) v8 -- 0x563697b9c580 con 0x563697b91ce0 2023-10-05T19:28:54.886 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:54.882+0000 7f9f63d54700 1 -- 172.21.15.167:0/344537782 <== osd.1 v2:172.21.15.167:6800/30035 2 ==== osd_op_reply(4 gc.3 [create,call,call,call] v32'180 uv7 ondisk = -125 ((125) Operation canceled)) v8 ==== 274+0+0 (crc 0 0 0) 0x7f9f5c024020 con 0x563697b91ce0 2023-10-05T19:28:54.886 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:54.882+0000 7f9f6621fe40 1 --2- 172.21.15.167:0/344537782 >> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] conn(0x563697b9d5a0 0x563697b9f980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-05T19:28:54.886 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:54.882+0000 7f9f6621fe40 1 -- 172.21.15.167:0/344537782 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:5 6.3 6:c60da066:gc::gc.4:head [create,call version.check_conds in=74b,call rgw_gc.rgw_gc_queue_init in=45b,call version.set in=34b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e32) v8 -- 0x563697ba0030 con 0x563697b9d5a0 2023-10-05T19:28:54.886 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:54.882+0000 7f9f63d54700 1 --2- 172.21.15.167:0/344537782 >> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] conn(0x563697b9d5a0 0x563697b9f980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-05T19:28:54.886 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:54.882+0000 7f9f63d54700 1 --2- 172.21.15.167:0/344537782 >> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] conn(0x563697b9d5a0 0x563697b9f980 crc :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-05T19:28:54.888 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:54.882+0000 7f9f63d54700 1 -- 172.21.15.167:0/344537782 <== osd.2 v2:172.21.15.167:6816/30037 1 ==== osd_op_reply(5 gc.4 [create,call,call,call] v32'47 uv7 ondisk = -125 ((125) Operation canceled)) v8 ==== 274+0+0 (crc 0 0 0) 0x7f9f5c024020 con 0x563697b9d5a0 2023-10-05T19:28:54.888 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:54.882+0000 7f9f6621fe40 1 -- 172.21.15.167:0/344537782 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:6 6.9 6:954be8d2:gc::gc.5:head [create,call version.check_conds in=74b,call rgw_gc.rgw_gc_queue_init in=45b,call version.set in=34b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e32) v8 -- 0x563697ba0680 con 0x563697b95830 2023-10-05T19:28:54.890 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:54.886+0000 7f9f64555700 1 -- 172.21.15.167:0/344537782 <== osd.3 v2:172.21.15.167:6808/30036 2 ==== osd_op_reply(6 gc.5 [create,call,call,call] v32'34 uv4 ondisk = -125 ((125) Operation canceled)) v8 ==== 274+0+0 (crc 0 0 0) 0x7f9f54050040 con 0x563697b95830 2023-10-05T19:28:54.891 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:54.886+0000 7f9f6621fe40 1 -- 172.21.15.167:0/344537782 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:7 6.e 6:748722ff:gc::gc.6:head [create,call version.check_conds in=74b,call rgw_gc.rgw_gc_queue_init in=45b,call version.set in=34b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e32) v8 -- 0x563697ba0ca0 con 0x563697b91ce0 2023-10-05T19:28:54.893 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:54.886+0000 7f9f63d54700 1 -- 172.21.15.167:0/344537782 <== osd.1 v2:172.21.15.167:6800/30035 3 ==== osd_op_reply(7 gc.6 [create,call,call,call] v32'21 uv5 ondisk = -125 ((125) Operation canceled)) v8 ==== 274+0+0 (crc 0 0 0) 0x7f9f5c024020 con 0x563697b91ce0 2023-10-05T19:28:54.893 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:54.886+0000 7f9f6621fe40 1 -- 172.21.15.167:0/344537782 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:8 6.1d 6:beaa4be4:gc::gc.7:head [create,call version.check_conds in=74b,call rgw_gc.rgw_gc_queue_init in=45b,call version.set in=34b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e32) v8 -- 0x563697ba1190 con 0x563697b8d8c0 2023-10-05T19:28:54.896 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:54.890+0000 7f9f63553700 1 -- 172.21.15.167:0/344537782 <== osd.0 v2:172.21.15.118:6800/27690 2 ==== osd_op_reply(8 gc.7 [create,call,call,call] v32'31 uv4 ondisk = -125 ((125) Operation canceled)) v8 ==== 274+0+0 (crc 0 0 0) 0x7f9f4c03f020 con 0x563697b8d8c0 2023-10-05T19:28:54.896 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:54.890+0000 7f9f6621fe40 1 -- 172.21.15.167:0/344537782 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:9 6.12 6:4f78e9a2:gc::gc.8:head [create,call version.check_conds in=74b,call rgw_gc.rgw_gc_queue_init in=45b,call version.set in=34b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e32) v8 -- 0x563697b907d0 con 0x563697b9d5a0 2023-10-05T19:28:54.899 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:54.894+0000 7f9f63d54700 1 -- 172.21.15.167:0/344537782 <== osd.2 v2:172.21.15.167:6816/30037 2 ==== osd_op_reply(9 gc.8 [create,call,call,call] v32'19 uv3 ondisk = -125 ((125) Operation canceled)) v8 ==== 274+0+0 (crc 0 0 0) 0x7f9f5c024020 con 0x563697b9d5a0 2023-10-05T19:28:54.899 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:54.894+0000 7f9f6621fe40 1 -- 172.21.15.167:0/344537782 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:10 6.1c 6:3e426232:gc::gc.9:head [create,call version.check_conds in=74b,call rgw_gc.rgw_gc_queue_init in=45b,call version.set in=34b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e32) v8 -- 0x563697b907d0 con 0x563697b95830 2023-10-05T19:28:54.901 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:54.894+0000 7f9f64555700 1 -- 172.21.15.167:0/344537782 <== osd.3 v2:172.21.15.167:6808/30036 3 ==== osd_op_reply(10 gc.9 [create,call,call,call] v32'37 uv3 ondisk = -125 ((125) Operation canceled)) v8 ==== 274+0+0 (crc 0 0 0) 0x7f9f54050040 con 0x563697b95830 2023-10-05T19:28:54.901 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:54.894+0000 7f9f6621fe40 1 -- 172.21.15.167:0/344537782 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:11 6.3 6:c31f41d7:gc::gc.10:head [create,call version.check_conds in=74b,call rgw_gc.rgw_gc_queue_init in=45b,call version.set in=34b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e32) v8 -- 0x563697b90a40 con 0x563697b9d5a0 2023-10-05T19:28:54.903 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:54.898+0000 7f9f63d54700 1 -- 172.21.15.167:0/344537782 <== osd.2 v2:172.21.15.167:6816/30037 3 ==== osd_op_reply(11 gc.10 [create,call,call,call] v32'48 uv11 ondisk = -125 ((125) Operation canceled)) v8 ==== 275+0+0 (crc 0 0 0) 0x7f9f5c024020 con 0x563697b9d5a0 2023-10-05T19:28:54.903 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:54.898+0000 7f9f6621fe40 1 -- 172.21.15.167:0/344537782 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:12 6.13 6:c8333aab:gc::gc.11:head [create,call version.check_conds in=74b,call rgw_gc.rgw_gc_queue_init in=45b,call version.set in=34b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e32) v8 -- 0x563697b90a40 con 0x563697b8d8c0 2023-10-05T19:28:54.906 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:54.902+0000 7f9f63553700 1 -- 172.21.15.167:0/344537782 <== osd.0 v2:172.21.15.118:6800/27690 3 ==== osd_op_reply(12 gc.11 [create,call,call,call] v32'23 uv5 ondisk = -125 ((125) Operation canceled)) v8 ==== 275+0+0 (crc 0 0 0) 0x7f9f4c03f020 con 0x563697b8d8c0 2023-10-05T19:28:54.906 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:54.902+0000 7f9f6621fe40 1 -- 172.21.15.167:0/344537782 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:13 6.15 6:ab569f81:gc::gc.12:head [create,call version.check_conds in=74b,call rgw_gc.rgw_gc_queue_init in=45b,call version.set in=34b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e32) v8 -- 0x563697ba1f20 con 0x563697b9d5a0 2023-10-05T19:28:54.909 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:54.902+0000 7f9f63d54700 1 -- 172.21.15.167:0/344537782 <== osd.2 v2:172.21.15.167:6816/30037 4 ==== osd_op_reply(13 gc.12 [create,call,call,call] v32'19 uv3 ondisk = -125 ((125) Operation canceled)) v8 ==== 275+0+0 (crc 0 0 0) 0x7f9f5c024020 con 0x563697b9d5a0 2023-10-05T19:28:54.909 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:54.902+0000 7f9f6621fe40 1 -- 172.21.15.167:0/344537782 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:14 6.0 6:0667bb13:gc::gc.13:head [create,call version.check_conds in=74b,call rgw_gc.rgw_gc_queue_init in=45b,call version.set in=34b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e32) v8 -- 0x563697ba1f20 con 0x563697b8d8c0 2023-10-05T19:28:54.912 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:54.906+0000 7f9f63553700 1 -- 172.21.15.167:0/344537782 <== osd.0 v2:172.21.15.118:6800/27690 4 ==== osd_op_reply(14 gc.13 [create,call,call,call] v32'37 uv6 ondisk = -125 ((125) Operation canceled)) v8 ==== 275+0+0 (crc 0 0 0) 0x7f9f4c03f020 con 0x563697b8d8c0 2023-10-05T19:28:54.912 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:54.906+0000 7f9f6621fe40 1 -- 172.21.15.167:0/344537782 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:15 6.4 6:227f63b5:gc::gc.14:head [create,call version.check_conds in=74b,call rgw_gc.rgw_gc_queue_init in=45b,call version.set in=34b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e32) v8 -- 0x563697ba2540 con 0x563697b91ce0 2023-10-05T19:28:54.915 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:54.910+0000 7f9f63d54700 1 -- 172.21.15.167:0/344537782 <== osd.1 v2:172.21.15.167:6800/30035 4 ==== osd_op_reply(15 gc.14 [create,call,call,call] v32'181 uv13 ondisk = -125 ((125) Operation canceled)) v8 ==== 275+0+0 (crc 0 0 0) 0x7f9f5c024020 con 0x563697b91ce0 2023-10-05T19:28:54.915 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:54.910+0000 7f9f6621fe40 1 -- 172.21.15.167:0/344537782 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:16 6.7 6:e07a6548:gc::gc.15:head [create,call version.check_conds in=74b,call rgw_gc.rgw_gc_queue_init in=45b,call version.set in=34b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e32) v8 -- 0x563697ba2540 con 0x563697b95830 2023-10-05T19:28:54.918 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:54.914+0000 7f9f64555700 1 -- 172.21.15.167:0/344537782 <== osd.3 v2:172.21.15.167:6808/30036 4 ==== osd_op_reply(16 gc.15 [create,call,call,call] v32'41 uv7 ondisk = -125 ((125) Operation canceled)) v8 ==== 275+0+0 (crc 0 0 0) 0x7f9f54050040 con 0x563697b95830 2023-10-05T19:28:54.918 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:54.914+0000 7f9f6621fe40 1 -- 172.21.15.167:0/344537782 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:17 6.3 6:c623b173:gc::gc.16:head [create,call version.check_conds in=74b,call rgw_gc.rgw_gc_queue_init in=45b,call version.set in=34b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e32) v8 -- 0x563697ba2540 con 0x563697b9d5a0 2023-10-05T19:28:54.920 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:54.914+0000 7f9f63d54700 1 -- 172.21.15.167:0/344537782 <== osd.2 v2:172.21.15.167:6816/30037 5 ==== osd_op_reply(17 gc.16 [create,call,call,call] v32'49 uv13 ondisk = -125 ((125) Operation canceled)) v8 ==== 275+0+0 (crc 0 0 0) 0x7f9f5c024020 con 0x563697b9d5a0 2023-10-05T19:28:54.920 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:54.914+0000 7f9f6621fe40 1 -- 172.21.15.167:0/344537782 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:18 6.16 6:6849fd06:gc::gc.17:head [create,call version.check_conds in=74b,call rgw_gc.rgw_gc_queue_init in=45b,call version.set in=34b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e32) v8 -- 0x563697ba2540 con 0x563697b95830 2023-10-05T19:28:54.923 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:54.918+0000 7f9f64555700 1 -- 172.21.15.167:0/344537782 <== osd.3 v2:172.21.15.167:6808/30036 5 ==== osd_op_reply(18 gc.17 [create,call,call,call] v32'19 uv7 ondisk = -125 ((125) Operation canceled)) v8 ==== 275+0+0 (crc 0 0 0) 0x7f9f54050040 con 0x563697b95830 2023-10-05T19:28:54.923 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:54.918+0000 7f9f6621fe40 1 -- 172.21.15.167:0/344537782 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:19 6.d 6:b019ea83:gc::gc.18:head [create,call version.check_conds in=74b,call rgw_gc.rgw_gc_queue_init in=45b,call version.set in=34b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e32) v8 -- 0x563697ba2540 con 0x563697b91ce0 2023-10-05T19:28:54.926 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:54.922+0000 7f9f63d54700 1 -- 172.21.15.167:0/344537782 <== osd.1 v2:172.21.15.167:6800/30035 5 ==== osd_op_reply(19 gc.18 [create,call,call,call] v32'27 uv5 ondisk = -125 ((125) Operation canceled)) v8 ==== 275+0+0 (crc 0 0 0) 0x7f9f5c024020 con 0x563697b91ce0 2023-10-05T19:28:54.926 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:54.922+0000 7f9f6621fe40 1 -- 172.21.15.167:0/344537782 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:20 6.4 6:24c8e1ae:gc::gc.19:head [create,call version.check_conds in=74b,call rgw_gc.rgw_gc_queue_init in=45b,call version.set in=34b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e32) v8 -- 0x563697ba2540 con 0x563697b91ce0 2023-10-05T19:28:54.928 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:54.922+0000 7f9f63d54700 1 -- 172.21.15.167:0/344537782 <== osd.1 v2:172.21.15.167:6800/30035 6 ==== osd_op_reply(20 gc.19 [create,call,call,call] v32'182 uv173 ondisk = -125 ((125) Operation canceled)) v8 ==== 275+0+0 (crc 0 0 0) 0x7f9f5c024020 con 0x563697b91ce0 2023-10-05T19:28:54.929 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:54.922+0000 7f9f6621fe40 1 -- 172.21.15.167:0/344537782 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:21 6.7 6:e622f76a:gc::gc.20:head [create,call version.check_conds in=74b,call rgw_gc.rgw_gc_queue_init in=45b,call version.set in=34b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e32) v8 -- 0x563697ba2540 con 0x563697b95830 2023-10-05T19:28:54.932 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:54.926+0000 7f9f64555700 1 -- 172.21.15.167:0/344537782 <== osd.3 v2:172.21.15.167:6808/30036 6 ==== osd_op_reply(21 gc.20 [create,call,call,call] v32'42 uv9 ondisk = -125 ((125) Operation canceled)) v8 ==== 275+0+0 (crc 0 0 0) 0x7f9f54050040 con 0x563697b95830 2023-10-05T19:28:54.932 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:54.926+0000 7f9f6621fe40 1 -- 172.21.15.167:0/344537782 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:22 6.a 6:5127e642:gc::gc.21:head [create,call version.check_conds in=74b,call rgw_gc.rgw_gc_queue_init in=45b,call version.set in=34b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e32) v8 -- 0x563697ba2540 con 0x563697b9d5a0 2023-10-05T19:28:54.934 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:54.930+0000 7f9f63d54700 1 -- 172.21.15.167:0/344537782 <== osd.2 v2:172.21.15.167:6816/30037 6 ==== osd_op_reply(22 gc.21 [create,call,call,call] v32'29 uv7 ondisk = -125 ((125) Operation canceled)) v8 ==== 275+0+0 (crc 0 0 0) 0x7f9f5c024020 con 0x563697b9d5a0 2023-10-05T19:28:54.934 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:54.930+0000 7f9f6621fe40 1 -- 172.21.15.167:0/344537782 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:23 6.11 6:8eb99c53:gc::gc.22:head [create,call version.check_conds in=74b,call rgw_gc.rgw_gc_queue_init in=45b,call version.set in=34b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e32) v8 -- 0x563697ba25d0 con 0x563697b95830 2023-10-05T19:28:54.936 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:54.930+0000 7f9f64555700 1 -- 172.21.15.167:0/344537782 <== osd.3 v2:172.21.15.167:6808/30036 7 ==== osd_op_reply(23 gc.22 [create,call,call,call] v32'21 uv3 ondisk = -125 ((125) Operation canceled)) v8 ==== 275+0+0 (crc 0 0 0) 0x7f9f54050040 con 0x563697b95830 2023-10-05T19:28:54.937 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:54.930+0000 7f9f6621fe40 1 -- 172.21.15.167:0/344537782 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:24 6.8 6:17d7e8ed:gc::gc.23:head [create,call version.check_conds in=74b,call rgw_gc.rgw_gc_queue_init in=45b,call version.set in=34b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e32) v8 -- 0x563697ba2660 con 0x563697b95830 2023-10-05T19:28:54.939 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:54.934+0000 7f9f64555700 1 -- 172.21.15.167:0/344537782 <== osd.3 v2:172.21.15.167:6808/30036 8 ==== osd_op_reply(24 gc.23 [create,call,call,call] v32'31 uv5 ondisk = -125 ((125) Operation canceled)) v8 ==== 275+0+0 (crc 0 0 0) 0x7f9f54050040 con 0x563697b95830 2023-10-05T19:28:54.939 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:54.934+0000 7f9f6621fe40 1 -- 172.21.15.167:0/344537782 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:25 6.1b 6:db9d130d:gc::gc.24:head [create,call version.check_conds in=74b,call rgw_gc.rgw_gc_queue_init in=45b,call version.set in=34b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e32) v8 -- 0x563697ba2660 con 0x563697b95830 2023-10-05T19:28:54.942 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:54.938+0000 7f9f64555700 1 -- 172.21.15.167:0/344537782 <== osd.3 v2:172.21.15.167:6808/30036 9 ==== osd_op_reply(25 gc.24 [create,call,call,call] v32'15 uv3 ondisk = -125 ((125) Operation canceled)) v8 ==== 275+0+0 (crc 0 0 0) 0x7f9f54050040 con 0x563697b95830 2023-10-05T19:28:54.942 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:54.938+0000 7f9f6621fe40 1 -- 172.21.15.167:0/344537782 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:26 6.9 6:921ecdb1:gc::gc.25:head [create,call version.check_conds in=74b,call rgw_gc.rgw_gc_queue_init in=45b,call version.set in=34b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e32) v8 -- 0x563697ba2660 con 0x563697b95830 2023-10-05T19:28:54.944 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:54.938+0000 7f9f64555700 1 -- 172.21.15.167:0/344537782 <== osd.3 v2:172.21.15.167:6808/30036 10 ==== osd_op_reply(26 gc.25 [create,call,call,call] v32'35 uv10 ondisk = -125 ((125) Operation canceled)) v8 ==== 275+0+0 (crc 0 0 0) 0x7f9f54050040 con 0x563697b95830 2023-10-05T19:28:54.944 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:54.938+0000 7f9f6621fe40 1 -- 172.21.15.167:0/344537782 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:27 6.1f 6:fde4dd55:gc::gc.26:head [create,call version.check_conds in=74b,call rgw_gc.rgw_gc_queue_init in=45b,call version.set in=34b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e32) v8 -- 0x563697ba2660 con 0x563697b8d8c0 2023-10-05T19:28:54.946 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:54.942+0000 7f9f63553700 1 -- 172.21.15.167:0/344537782 <== osd.0 v2:172.21.15.118:6800/27690 5 ==== osd_op_reply(27 gc.26 [create,call,call,call] v32'21 uv5 ondisk = -125 ((125) Operation canceled)) v8 ==== 275+0+0 (crc 0 0 0) 0x7f9f4c03f020 con 0x563697b8d8c0 2023-10-05T19:28:54.947 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:54.942+0000 7f9f6621fe40 1 -- 172.21.15.167:0/344537782 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:28 6.1a 6:5909e671:gc::gc.27:head [create,call version.check_conds in=74b,call rgw_gc.rgw_gc_queue_init in=45b,call version.set in=34b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e32) v8 -- 0x563697ba2660 con 0x563697b95830 2023-10-05T19:28:54.949 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:54.942+0000 7f9f64555700 1 -- 172.21.15.167:0/344537782 <== osd.3 v2:172.21.15.167:6808/30036 11 ==== osd_op_reply(28 gc.27 [create,call,call,call] v32'19 uv7 ondisk = -125 ((125) Operation canceled)) v8 ==== 275+0+0 (crc 0 0 0) 0x7f9f54050040 con 0x563697b95830 2023-10-05T19:28:54.949 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:54.942+0000 7f9f6621fe40 1 -- 172.21.15.167:0/344537782 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:29 6.1d 6:b88e3271:gc::gc.28:head [create,call version.check_conds in=74b,call rgw_gc.rgw_gc_queue_init in=45b,call version.set in=34b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e32) v8 -- 0x563697ba2660 con 0x563697b8d8c0 2023-10-05T19:28:54.951 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:54.946+0000 7f9f63553700 1 -- 172.21.15.167:0/344537782 <== osd.0 v2:172.21.15.118:6800/27690 6 ==== osd_op_reply(29 gc.28 [create,call,call,call] v32'32 uv8 ondisk = -125 ((125) Operation canceled)) v8 ==== 275+0+0 (crc 0 0 0) 0x7f9f4c03f020 con 0x563697b8d8c0 2023-10-05T19:28:54.951 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:54.946+0000 7f9f6621fe40 1 -- 172.21.15.167:0/344537782 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:30 6.7 6:e47c9322:gc::gc.29:head [create,call version.check_conds in=74b,call rgw_gc.rgw_gc_queue_init in=45b,call version.set in=34b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e32) v8 -- 0x563697ba2660 con 0x563697b95830 2023-10-05T19:28:54.954 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:54.950+0000 7f9f64555700 1 -- 172.21.15.167:0/344537782 <== osd.3 v2:172.21.15.167:6808/30036 12 ==== osd_op_reply(30 gc.29 [create,call,call,call] v32'43 uv11 ondisk = -125 ((125) Operation canceled)) v8 ==== 275+0+0 (crc 0 0 0) 0x7f9f54050040 con 0x563697b95830 2023-10-05T19:28:54.954 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:54.950+0000 7f9f6621fe40 1 -- 172.21.15.167:0/344537782 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:31 6.0 6:013d6c33:gc::gc.30:head [create,call version.check_conds in=74b,call rgw_gc.rgw_gc_queue_init in=45b,call version.set in=34b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e32) v8 -- 0x563697ba2660 con 0x563697b8d8c0 2023-10-05T19:28:54.957 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:54.950+0000 7f9f63553700 1 -- 172.21.15.167:0/344537782 <== osd.0 v2:172.21.15.118:6800/27690 7 ==== osd_op_reply(31 gc.30 [create,call,call,call] v32'38 uv12 ondisk = -125 ((125) Operation canceled)) v8 ==== 275+0+0 (crc 0 0 0) 0x7f9f4c03f020 con 0x563697b8d8c0 2023-10-05T19:28:54.957 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:54.950+0000 7f9f6621fe40 1 -- 172.21.15.167:0/344537782 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:32 6.f 6:f24cb4a8:gc::gc.31:head [create,call version.check_conds in=74b,call rgw_gc.rgw_gc_queue_init in=45b,call version.set in=34b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e32) v8 -- 0x563697ba2660 con 0x563697b9d5a0 2023-10-05T19:28:54.959 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:54.954+0000 7f9f63d54700 1 -- 172.21.15.167:0/344537782 <== osd.2 v2:172.21.15.167:6816/30037 7 ==== osd_op_reply(32 gc.31 [create,call,call,call] v32'25 uv5 ondisk = -125 ((125) Operation canceled)) v8 ==== 275+0+0 (crc 0 0 0) 0x7f9f5c024020 con 0x563697b9d5a0 2023-10-05T19:28:54.962 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:54.958+0000 7f9ea3fff700 20 reqs_thread_entry: start 2023-10-05T19:28:54.963 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:54.958+0000 7f9ea1ffb700 20 reqs_thread_entry: start 2023-10-05T19:28:54.963 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:54.958+0000 7f9f6621fe40 20 init_complete bucket index max shards: 11 2023-10-05T19:28:54.963 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:54.958+0000 7f9f6621fe40 10 rgw notify: Started notification manager with: 1 workers 2023-10-05T19:28:54.964 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:54.958+0000 7f9e8ffff700 20 rgw notify: INFO: next queues processing will happen at: Thu Oct 5 19:29:25 2023 2023-10-05T19:28:54.964 INFO:teuthology.orchestra.run.smithi167.stderr: 2023-10-05T19:28:54.964 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:54.958+0000 7f9f6621fe40 20 garbage collection: RGWGC::process entered with GC index_shard=1, max_secs=3600, expired_only=1 2023-10-05T19:28:54.964 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:54.958+0000 7f9f6621fe40 1 -- 172.21.15.167:0/344537782 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:33 6.18 6:1e85421f:gc::gc.1:head [call lock.lock in=50b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e32) v8 -- 0x563697c023e0 con 0x563697b8d8c0 2023-10-05T19:28:54.964 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:54.958+0000 7f9e8f7fe700 20 reqs_thread_entry: start 2023-10-05T19:28:54.966 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:54.962+0000 7f9f63553700 1 -- 172.21.15.167:0/344537782 <== osd.0 v2:172.21.15.118:6800/27690 8 ==== osd_op_reply(33 gc.1 [call] v32'24 uv24 ondisk = 0) v8 ==== 148+0+0 (crc 0 0 0) 0x7f9f4c03f020 con 0x563697b8d8c0 2023-10-05T19:28:54.967 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:54.962+0000 7f9f6621fe40 1 -- 172.21.15.167:0/344537782 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:34 6.18 6:1e85421f:gc::gc.1:head [call rgw.gc_list in=25b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32) v8 -- 0x563697c023e0 con 0x563697b8d8c0 2023-10-05T19:28:54.968 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:54.962+0000 7f9f63553700 1 -- 172.21.15.167:0/344537782 <== osd.0 v2:172.21.15.118:6800/27690 9 ==== osd_op_reply(34 gc.1 [call out=15b] v0'0 uv24 ondisk = 0) v8 ==== 148+0+15 (crc 0 0 0) 0x7f9f4c03f020 con 0x563697b8d8c0 2023-10-05T19:28:54.968 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:54.962+0000 7f9f6621fe40 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2023-10-05T19:28:54.968 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:54.962+0000 7f9f6621fe40 1 -- 172.21.15.167:0/344537782 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:35 6.18 6:1e85421f:gc::gc.1:head [call version.read in=11b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32) v8 -- 0x563697c013d0 con 0x563697b8d8c0 2023-10-05T19:28:54.968 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:54.962+0000 7f9f63553700 1 -- 172.21.15.167:0/344537782 <== osd.0 v2:172.21.15.118:6800/27690 10 ==== osd_op_reply(35 gc.1 [call out=24b] v0'0 uv24 ondisk = 0) v8 ==== 148+0+24 (crc 0 0 0) 0x7f9f4c03f020 con 0x563697b8d8c0 2023-10-05T19:28:54.968 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:54.962+0000 7f9f6621fe40 1 -- 172.21.15.167:0/344537782 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:36 6.18 6:1e85421f:gc::gc.1:head [call rgw.gc_list in=25b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32) v8 -- 0x563697c023e0 con 0x563697b8d8c0 2023-10-05T19:28:54.969 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:54.962+0000 7f9f63553700 1 -- 172.21.15.167:0/344537782 <== osd.0 v2:172.21.15.118:6800/27690 11 ==== osd_op_reply(36 gc.1 [call out=15b] v0'0 uv24 ondisk = 0) v8 ==== 148+0+15 (crc 0 0 0) 0x7f9f4c03f020 con 0x563697b8d8c0 2023-10-05T19:28:54.969 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:54.962+0000 7f9f6621fe40 20 garbage collection: RGWGC::process cls_rgw_gc_list returned NO non expired entries, so setting cache entry to TRUE 2023-10-05T19:28:54.969 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:54.962+0000 7f9f6621fe40 1 -- 172.21.15.167:0/344537782 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:37 6.18 6:1e85421f:gc::gc.1:head [call rgw_gc.rgw_gc_queue_list_entries in=46b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32) v8 -- 0x563697c023e0 con 0x563697b8d8c0 2023-10-05T19:28:54.970 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:54.966+0000 7f9f63553700 1 -- 172.21.15.167:0/344537782 <== osd.0 v2:172.21.15.118:6800/27690 12 ==== osd_op_reply(37 gc.1 [call out=15b] v0'0 uv24 ondisk = 0) v8 ==== 148+0+15 (crc 0 0 0) 0x7f9f4c03f020 con 0x563697b8d8c0 2023-10-05T19:28:54.970 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:54.966+0000 7f9f6621fe40 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=0, truncated=0, next_marker='' 2023-10-05T19:28:54.971 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:54.966+0000 7f9f6621fe40 1 -- 172.21.15.167:0/344537782 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:38 6.18 6:1e85421f:gc::gc.1:head [call lock.unlock in=34b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e32) v8 -- 0x563697c023e0 con 0x563697b8d8c0 2023-10-05T19:28:54.973 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:54.966+0000 7f9f63553700 1 -- 172.21.15.167:0/344537782 <== osd.0 v2:172.21.15.118:6800/27690 13 ==== osd_op_reply(38 gc.1 [call] v32'25 uv25 ondisk = 0) v8 ==== 148+0+0 (crc 0 0 0) 0x7f9f4c03f020 con 0x563697b8d8c0 2023-10-05T19:28:54.973 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:54.966+0000 7f9f6621fe40 20 garbage collection: RGWGC::process entered with GC index_shard=2, max_secs=3600, expired_only=1 2023-10-05T19:28:54.973 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:54.966+0000 7f9f6621fe40 1 -- 172.21.15.167:0/344537782 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:39 6.b 6:d242335b:gc::gc.2:head [call lock.lock in=50b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e32) v8 -- 0x563697c023e0 con 0x563697b95830 2023-10-05T19:28:54.976 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:54.970+0000 7f9f64555700 1 -- 172.21.15.167:0/344537782 <== osd.3 v2:172.21.15.167:6808/30036 13 ==== osd_op_reply(39 gc.2 [call] v32'20 uv20 ondisk = 0) v8 ==== 148+0+0 (crc 0 0 0) 0x7f9f54050040 con 0x563697b95830 2023-10-05T19:28:54.976 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:54.970+0000 7f9f6621fe40 1 -- 172.21.15.167:0/344537782 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:40 6.b 6:d242335b:gc::gc.2:head [call rgw.gc_list in=25b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32) v8 -- 0x563697c023e0 con 0x563697b95830 2023-10-05T19:28:54.977 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:54.970+0000 7f9f64555700 1 -- 172.21.15.167:0/344537782 <== osd.3 v2:172.21.15.167:6808/30036 14 ==== osd_op_reply(40 gc.2 [call out=15b] v0'0 uv20 ondisk = 0) v8 ==== 148+0+15 (crc 0 0 0) 0x7f9f54050040 con 0x563697b95830 2023-10-05T19:28:54.977 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:54.970+0000 7f9f6621fe40 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2023-10-05T19:28:54.977 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:54.970+0000 7f9f6621fe40 1 -- 172.21.15.167:0/344537782 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:41 6.b 6:d242335b:gc::gc.2:head [call version.read in=11b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32) v8 -- 0x563697c013d0 con 0x563697b95830 2023-10-05T19:28:54.977 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:54.974+0000 7f9f64555700 1 -- 172.21.15.167:0/344537782 <== osd.3 v2:172.21.15.167:6808/30036 15 ==== osd_op_reply(41 gc.2 [call out=24b] v0'0 uv20 ondisk = 0) v8 ==== 148+0+24 (crc 0 0 0) 0x7f9f54050040 con 0x563697b95830 2023-10-05T19:28:54.978 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:54.974+0000 7f9f6621fe40 1 -- 172.21.15.167:0/344537782 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:42 6.b 6:d242335b:gc::gc.2:head [call rgw.gc_list in=25b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32) v8 -- 0x563697c023e0 con 0x563697b95830 2023-10-05T19:28:54.978 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:54.974+0000 7f9f64555700 1 -- 172.21.15.167:0/344537782 <== osd.3 v2:172.21.15.167:6808/30036 16 ==== osd_op_reply(42 gc.2 [call out=15b] v0'0 uv20 ondisk = 0) v8 ==== 148+0+15 (crc 0 0 0) 0x7f9f54050040 con 0x563697b95830 2023-10-05T19:28:54.978 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:54.974+0000 7f9f6621fe40 20 garbage collection: RGWGC::process cls_rgw_gc_list returned NO non expired entries, so setting cache entry to TRUE 2023-10-05T19:28:54.978 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:54.974+0000 7f9f6621fe40 1 -- 172.21.15.167:0/344537782 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:43 6.b 6:d242335b:gc::gc.2:head [call rgw_gc.rgw_gc_queue_list_entries in=46b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32) v8 -- 0x563697c023e0 con 0x563697b95830 2023-10-05T19:28:54.979 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:54.974+0000 7f9f64555700 1 -- 172.21.15.167:0/344537782 <== osd.3 v2:172.21.15.167:6808/30036 17 ==== osd_op_reply(43 gc.2 [call out=15b] v0'0 uv20 ondisk = 0) v8 ==== 148+0+15 (crc 0 0 0) 0x7f9f54050040 con 0x563697b95830 2023-10-05T19:28:54.979 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:54.974+0000 7f9f6621fe40 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=0, truncated=0, next_marker='' 2023-10-05T19:28:54.979 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:54.974+0000 7f9f6621fe40 1 -- 172.21.15.167:0/344537782 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:44 6.b 6:d242335b:gc::gc.2:head [call lock.unlock in=34b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e32) v8 -- 0x563697c023e0 con 0x563697b95830 2023-10-05T19:28:54.982 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:54.978+0000 7f9f64555700 1 -- 172.21.15.167:0/344537782 <== osd.3 v2:172.21.15.167:6808/30036 18 ==== osd_op_reply(44 gc.2 [call] v32'21 uv21 ondisk = 0) v8 ==== 148+0+0 (crc 0 0 0) 0x7f9f54050040 con 0x563697b95830 2023-10-05T19:28:54.982 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:54.978+0000 7f9f6621fe40 20 garbage collection: RGWGC::process entered with GC index_shard=3, max_secs=3600, expired_only=1 2023-10-05T19:28:54.982 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:54.978+0000 7f9f6621fe40 1 -- 172.21.15.167:0/344537782 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:45 6.4 6:2166d8e1:gc::gc.3:head [call lock.lock in=50b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e32) v8 -- 0x563697c023e0 con 0x563697b91ce0 2023-10-05T19:28:54.985 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:54.978+0000 7f9f63d54700 1 -- 172.21.15.167:0/344537782 <== osd.1 v2:172.21.15.167:6800/30035 7 ==== osd_op_reply(45 gc.3 [call] v32'183 uv183 ondisk = 0) v8 ==== 148+0+0 (crc 0 0 0) 0x7f9f5c024020 con 0x563697b91ce0 2023-10-05T19:28:54.985 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:54.978+0000 7f9f6621fe40 1 -- 172.21.15.167:0/344537782 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:46 6.4 6:2166d8e1:gc::gc.3:head [call rgw.gc_list in=25b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32) v8 -- 0x563697c023e0 con 0x563697b91ce0 2023-10-05T19:28:54.985 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:54.982+0000 7f9f63d54700 1 -- 172.21.15.167:0/344537782 <== osd.1 v2:172.21.15.167:6800/30035 8 ==== osd_op_reply(46 gc.3 [call out=15b] v0'0 uv183 ondisk = 0) v8 ==== 148+0+15 (crc 0 0 0) 0x7f9f5c024020 con 0x563697b91ce0 2023-10-05T19:28:54.986 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:54.982+0000 7f9f6621fe40 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2023-10-05T19:28:54.986 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:54.982+0000 7f9f6621fe40 1 -- 172.21.15.167:0/344537782 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:47 6.4 6:2166d8e1:gc::gc.3:head [call version.read in=11b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32) v8 -- 0x563697c013d0 con 0x563697b91ce0 2023-10-05T19:28:54.986 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:54.982+0000 7f9f63d54700 1 -- 172.21.15.167:0/344537782 <== osd.1 v2:172.21.15.167:6800/30035 9 ==== osd_op_reply(47 gc.3 [call out=24b] v0'0 uv183 ondisk = 0) v8 ==== 148+0+24 (crc 0 0 0) 0x7f9f5c024020 con 0x563697b91ce0 2023-10-05T19:28:54.986 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:54.982+0000 7f9f6621fe40 1 -- 172.21.15.167:0/344537782 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:48 6.4 6:2166d8e1:gc::gc.3:head [call rgw.gc_list in=25b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32) v8 -- 0x563697c023e0 con 0x563697b91ce0 2023-10-05T19:28:54.987 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:54.982+0000 7f9f63d54700 1 -- 172.21.15.167:0/344537782 <== osd.1 v2:172.21.15.167:6800/30035 10 ==== osd_op_reply(48 gc.3 [call out=15b] v0'0 uv183 ondisk = 0) v8 ==== 148+0+15 (crc 0 0 0) 0x7f9f5c024020 con 0x563697b91ce0 2023-10-05T19:28:54.987 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:54.982+0000 7f9f6621fe40 20 garbage collection: RGWGC::process cls_rgw_gc_list returned NO non expired entries, so setting cache entry to TRUE 2023-10-05T19:28:54.987 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:54.982+0000 7f9f6621fe40 1 -- 172.21.15.167:0/344537782 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:49 6.4 6:2166d8e1:gc::gc.3:head [call rgw_gc.rgw_gc_queue_list_entries in=46b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32) v8 -- 0x563697c023e0 con 0x563697b91ce0 2023-10-05T19:28:54.988 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:54.982+0000 7f9f63d54700 1 -- 172.21.15.167:0/344537782 <== osd.1 v2:172.21.15.167:6800/30035 11 ==== osd_op_reply(49 gc.3 [call out=15b] v0'0 uv183 ondisk = 0) v8 ==== 148+0+15 (crc 0 0 0) 0x7f9f5c024020 con 0x563697b91ce0 2023-10-05T19:28:54.988 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:54.982+0000 7f9f6621fe40 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=0, truncated=0, next_marker='' 2023-10-05T19:28:54.988 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:54.982+0000 7f9f6621fe40 1 -- 172.21.15.167:0/344537782 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:50 6.4 6:2166d8e1:gc::gc.3:head [call lock.unlock in=34b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e32) v8 -- 0x563697c023e0 con 0x563697b91ce0 2023-10-05T19:28:54.990 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:54.986+0000 7f9f63d54700 1 -- 172.21.15.167:0/344537782 <== osd.1 v2:172.21.15.167:6800/30035 12 ==== osd_op_reply(50 gc.3 [call] v32'184 uv184 ondisk = 0) v8 ==== 148+0+0 (crc 0 0 0) 0x7f9f5c024020 con 0x563697b91ce0 2023-10-05T19:28:54.990 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:54.986+0000 7f9f6621fe40 20 garbage collection: RGWGC::process entered with GC index_shard=4, max_secs=3600, expired_only=1 2023-10-05T19:28:54.990 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:54.986+0000 7f9f6621fe40 1 -- 172.21.15.167:0/344537782 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:51 6.3 6:c60da066:gc::gc.4:head [call lock.lock in=50b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e32) v8 -- 0x563697c023e0 con 0x563697b9d5a0 2023-10-05T19:28:54.992 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:54.986+0000 7f9f63d54700 1 -- 172.21.15.167:0/344537782 <== osd.2 v2:172.21.15.167:6816/30037 8 ==== osd_op_reply(51 gc.4 [call] v32'50 uv50 ondisk = 0) v8 ==== 148+0+0 (crc 0 0 0) 0x7f9f5c024020 con 0x563697b9d5a0 2023-10-05T19:28:54.992 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:54.986+0000 7f9f6621fe40 1 -- 172.21.15.167:0/344537782 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:52 6.3 6:c60da066:gc::gc.4:head [call rgw.gc_list in=25b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32) v8 -- 0x563697c023e0 con 0x563697b9d5a0 2023-10-05T19:28:54.993 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:54.986+0000 7f9f63d54700 1 -- 172.21.15.167:0/344537782 <== osd.2 v2:172.21.15.167:6816/30037 9 ==== osd_op_reply(52 gc.4 [call out=15b] v0'0 uv50 ondisk = 0) v8 ==== 148+0+15 (crc 0 0 0) 0x7f9f5c024020 con 0x563697b9d5a0 2023-10-05T19:28:54.993 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:54.986+0000 7f9f6621fe40 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2023-10-05T19:28:54.993 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:54.986+0000 7f9f6621fe40 1 -- 172.21.15.167:0/344537782 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:53 6.3 6:c60da066:gc::gc.4:head [call version.read in=11b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32) v8 -- 0x563697c013d0 con 0x563697b9d5a0 2023-10-05T19:28:54.993 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:54.986+0000 7f9f63d54700 1 -- 172.21.15.167:0/344537782 <== osd.2 v2:172.21.15.167:6816/30037 10 ==== osd_op_reply(53 gc.4 [call out=24b] v0'0 uv50 ondisk = 0) v8 ==== 148+0+24 (crc 0 0 0) 0x7f9f5c024020 con 0x563697b9d5a0 2023-10-05T19:28:54.994 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:54.986+0000 7f9f6621fe40 1 -- 172.21.15.167:0/344537782 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:54 6.3 6:c60da066:gc::gc.4:head [call rgw.gc_list in=25b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32) v8 -- 0x563697c023e0 con 0x563697b9d5a0 2023-10-05T19:28:54.994 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:54.990+0000 7f9f63d54700 1 -- 172.21.15.167:0/344537782 <== osd.2 v2:172.21.15.167:6816/30037 11 ==== osd_op_reply(54 gc.4 [call out=15b] v0'0 uv50 ondisk = 0) v8 ==== 148+0+15 (crc 0 0 0) 0x7f9f5c024020 con 0x563697b9d5a0 2023-10-05T19:28:54.994 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:54.990+0000 7f9f6621fe40 20 garbage collection: RGWGC::process cls_rgw_gc_list returned NO non expired entries, so setting cache entry to TRUE 2023-10-05T19:28:54.995 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:54.990+0000 7f9f6621fe40 1 -- 172.21.15.167:0/344537782 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:55 6.3 6:c60da066:gc::gc.4:head [call rgw_gc.rgw_gc_queue_list_entries in=46b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32) v8 -- 0x563697c023e0 con 0x563697b9d5a0 2023-10-05T19:28:54.995 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:54.990+0000 7f9f63d54700 1 -- 172.21.15.167:0/344537782 <== osd.2 v2:172.21.15.167:6816/30037 12 ==== osd_op_reply(55 gc.4 [call out=15b] v0'0 uv50 ondisk = 0) v8 ==== 148+0+15 (crc 0 0 0) 0x7f9f5c024020 con 0x563697b9d5a0 2023-10-05T19:28:54.995 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:54.990+0000 7f9f6621fe40 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=0, truncated=0, next_marker='' 2023-10-05T19:28:54.995 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:54.990+0000 7f9f6621fe40 1 -- 172.21.15.167:0/344537782 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:56 6.3 6:c60da066:gc::gc.4:head [call lock.unlock in=34b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e32) v8 -- 0x563697c023e0 con 0x563697b9d5a0 2023-10-05T19:28:54.997 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:54.990+0000 7f9f63d54700 1 -- 172.21.15.167:0/344537782 <== osd.2 v2:172.21.15.167:6816/30037 13 ==== osd_op_reply(56 gc.4 [call] v32'51 uv51 ondisk = 0) v8 ==== 148+0+0 (crc 0 0 0) 0x7f9f5c024020 con 0x563697b9d5a0 2023-10-05T19:28:54.997 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:54.990+0000 7f9f6621fe40 20 garbage collection: RGWGC::process entered with GC index_shard=5, max_secs=3600, expired_only=1 2023-10-05T19:28:54.997 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:54.990+0000 7f9f6621fe40 1 -- 172.21.15.167:0/344537782 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:57 6.9 6:954be8d2:gc::gc.5:head [call lock.lock in=50b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e32) v8 -- 0x563697c023e0 con 0x563697b95830 2023-10-05T19:28:54.999 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:54.994+0000 7f9f64555700 1 -- 172.21.15.167:0/344537782 <== osd.3 v2:172.21.15.167:6808/30036 19 ==== osd_op_reply(57 gc.5 [call] v32'36 uv36 ondisk = 0) v8 ==== 148+0+0 (crc 0 0 0) 0x7f9f54050040 con 0x563697b95830 2023-10-05T19:28:54.999 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:54.994+0000 7f9f6621fe40 1 -- 172.21.15.167:0/344537782 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:58 6.9 6:954be8d2:gc::gc.5:head [call rgw.gc_list in=25b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32) v8 -- 0x563697c023e0 con 0x563697b95830 2023-10-05T19:28:54.999 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:54.994+0000 7f9f64555700 1 -- 172.21.15.167:0/344537782 <== osd.3 v2:172.21.15.167:6808/30036 20 ==== osd_op_reply(58 gc.5 [call out=15b] v0'0 uv36 ondisk = 0) v8 ==== 148+0+15 (crc 0 0 0) 0x7f9f54050040 con 0x563697b95830 2023-10-05T19:28:54.999 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:54.994+0000 7f9f6621fe40 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2023-10-05T19:28:55.000 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:54.994+0000 7f9f6621fe40 1 -- 172.21.15.167:0/344537782 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:59 6.9 6:954be8d2:gc::gc.5:head [call version.read in=11b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32) v8 -- 0x563697c013d0 con 0x563697b95830 2023-10-05T19:28:55.000 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:54.994+0000 7f9f64555700 1 -- 172.21.15.167:0/344537782 <== osd.3 v2:172.21.15.167:6808/30036 21 ==== osd_op_reply(59 gc.5 [call out=24b] v0'0 uv36 ondisk = 0) v8 ==== 148+0+24 (crc 0 0 0) 0x7f9f54050040 con 0x563697b95830 2023-10-05T19:28:55.000 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:54.994+0000 7f9f6621fe40 1 -- 172.21.15.167:0/344537782 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:60 6.9 6:954be8d2:gc::gc.5:head [call rgw.gc_list in=25b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32) v8 -- 0x563697c023e0 con 0x563697b95830 2023-10-05T19:28:55.001 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:54.994+0000 7f9f64555700 1 -- 172.21.15.167:0/344537782 <== osd.3 v2:172.21.15.167:6808/30036 22 ==== osd_op_reply(60 gc.5 [call out=15b] v0'0 uv36 ondisk = 0) v8 ==== 148+0+15 (crc 0 0 0) 0x7f9f54050040 con 0x563697b95830 2023-10-05T19:28:55.001 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:54.994+0000 7f9f6621fe40 20 garbage collection: RGWGC::process cls_rgw_gc_list returned NO non expired entries, so setting cache entry to TRUE 2023-10-05T19:28:55.001 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:54.994+0000 7f9f6621fe40 1 -- 172.21.15.167:0/344537782 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:61 6.9 6:954be8d2:gc::gc.5:head [call rgw_gc.rgw_gc_queue_list_entries in=46b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32) v8 -- 0x563697c023e0 con 0x563697b95830 2023-10-05T19:28:55.001 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:54.994+0000 7f9f64555700 1 -- 172.21.15.167:0/344537782 <== osd.3 v2:172.21.15.167:6808/30036 23 ==== osd_op_reply(61 gc.5 [call out=15b] v0'0 uv36 ondisk = 0) v8 ==== 148+0+15 (crc 0 0 0) 0x7f9f54050040 con 0x563697b95830 2023-10-05T19:28:55.002 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:54.994+0000 7f9f6621fe40 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=0, truncated=0, next_marker='' 2023-10-05T19:28:55.002 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:54.994+0000 7f9f6621fe40 1 -- 172.21.15.167:0/344537782 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:62 6.9 6:954be8d2:gc::gc.5:head [call lock.unlock in=34b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e32) v8 -- 0x563697c023e0 con 0x563697b95830 2023-10-05T19:28:55.003 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:54.998+0000 7f9f64555700 1 -- 172.21.15.167:0/344537782 <== osd.3 v2:172.21.15.167:6808/30036 24 ==== osd_op_reply(62 gc.5 [call] v32'37 uv37 ondisk = 0) v8 ==== 148+0+0 (crc 0 0 0) 0x7f9f54050040 con 0x563697b95830 2023-10-05T19:28:55.003 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:54.998+0000 7f9f6621fe40 20 garbage collection: RGWGC::process entered with GC index_shard=6, max_secs=3600, expired_only=1 2023-10-05T19:28:55.003 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:54.998+0000 7f9f6621fe40 1 -- 172.21.15.167:0/344537782 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:63 6.e 6:748722ff:gc::gc.6:head [call lock.lock in=50b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e32) v8 -- 0x563697c023e0 con 0x563697b91ce0 2023-10-05T19:28:55.006 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.002+0000 7f9f63d54700 1 -- 172.21.15.167:0/344537782 <== osd.1 v2:172.21.15.167:6800/30035 13 ==== osd_op_reply(63 gc.6 [call] v32'22 uv22 ondisk = 0) v8 ==== 148+0+0 (crc 0 0 0) 0x7f9f5c024020 con 0x563697b91ce0 2023-10-05T19:28:55.006 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.002+0000 7f9f6621fe40 1 -- 172.21.15.167:0/344537782 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:64 6.e 6:748722ff:gc::gc.6:head [call rgw.gc_list in=25b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32) v8 -- 0x563697c023e0 con 0x563697b91ce0 2023-10-05T19:28:55.007 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.002+0000 7f9f63d54700 1 -- 172.21.15.167:0/344537782 <== osd.1 v2:172.21.15.167:6800/30035 14 ==== osd_op_reply(64 gc.6 [call out=15b] v0'0 uv22 ondisk = 0) v8 ==== 148+0+15 (crc 0 0 0) 0x7f9f5c024020 con 0x563697b91ce0 2023-10-05T19:28:55.007 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.002+0000 7f9f6621fe40 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2023-10-05T19:28:55.007 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.002+0000 7f9f6621fe40 1 -- 172.21.15.167:0/344537782 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:65 6.e 6:748722ff:gc::gc.6:head [call version.read in=11b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32) v8 -- 0x563697c013d0 con 0x563697b91ce0 2023-10-05T19:28:55.007 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.002+0000 7f9f63d54700 1 -- 172.21.15.167:0/344537782 <== osd.1 v2:172.21.15.167:6800/30035 15 ==== osd_op_reply(65 gc.6 [call out=24b] v0'0 uv22 ondisk = 0) v8 ==== 148+0+24 (crc 0 0 0) 0x7f9f5c024020 con 0x563697b91ce0 2023-10-05T19:28:55.008 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.002+0000 7f9f6621fe40 1 -- 172.21.15.167:0/344537782 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:66 6.e 6:748722ff:gc::gc.6:head [call rgw.gc_list in=25b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32) v8 -- 0x563697c023e0 con 0x563697b91ce0 2023-10-05T19:28:55.008 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.002+0000 7f9f63d54700 1 -- 172.21.15.167:0/344537782 <== osd.1 v2:172.21.15.167:6800/30035 16 ==== osd_op_reply(66 gc.6 [call out=15b] v0'0 uv22 ondisk = 0) v8 ==== 148+0+15 (crc 0 0 0) 0x7f9f5c024020 con 0x563697b91ce0 2023-10-05T19:28:55.008 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.002+0000 7f9f6621fe40 20 garbage collection: RGWGC::process cls_rgw_gc_list returned NO non expired entries, so setting cache entry to TRUE 2023-10-05T19:28:55.008 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.002+0000 7f9f6621fe40 1 -- 172.21.15.167:0/344537782 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:67 6.e 6:748722ff:gc::gc.6:head [call rgw_gc.rgw_gc_queue_list_entries in=46b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32) v8 -- 0x563697c023e0 con 0x563697b91ce0 2023-10-05T19:28:55.008 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.002+0000 7f9f63d54700 1 -- 172.21.15.167:0/344537782 <== osd.1 v2:172.21.15.167:6800/30035 17 ==== osd_op_reply(67 gc.6 [call out=15b] v0'0 uv22 ondisk = 0) v8 ==== 148+0+15 (crc 0 0 0) 0x7f9f5c024020 con 0x563697b91ce0 2023-10-05T19:28:55.009 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.002+0000 7f9f6621fe40 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=0, truncated=0, next_marker='' 2023-10-05T19:28:55.009 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.002+0000 7f9f6621fe40 1 -- 172.21.15.167:0/344537782 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:68 6.e 6:748722ff:gc::gc.6:head [call lock.unlock in=34b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e32) v8 -- 0x563697c023e0 con 0x563697b91ce0 2023-10-05T19:28:55.011 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.006+0000 7f9f63d54700 1 -- 172.21.15.167:0/344537782 <== osd.1 v2:172.21.15.167:6800/30035 18 ==== osd_op_reply(68 gc.6 [call] v32'23 uv23 ondisk = 0) v8 ==== 148+0+0 (crc 0 0 0) 0x7f9f5c024020 con 0x563697b91ce0 2023-10-05T19:28:55.011 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.006+0000 7f9f6621fe40 20 garbage collection: RGWGC::process entered with GC index_shard=7, max_secs=3600, expired_only=1 2023-10-05T19:28:55.012 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.006+0000 7f9f6621fe40 1 -- 172.21.15.167:0/344537782 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:69 6.1d 6:beaa4be4:gc::gc.7:head [call lock.lock in=50b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e32) v8 -- 0x563697c023e0 con 0x563697b8d8c0 2023-10-05T19:28:55.014 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.010+0000 7f9f63553700 1 -- 172.21.15.167:0/344537782 <== osd.0 v2:172.21.15.118:6800/27690 14 ==== osd_op_reply(69 gc.7 [call] v32'33 uv33 ondisk = 0) v8 ==== 148+0+0 (crc 0 0 0) 0x7f9f4c03f020 con 0x563697b8d8c0 2023-10-05T19:28:55.014 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.010+0000 7f9f6621fe40 1 -- 172.21.15.167:0/344537782 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:70 6.1d 6:beaa4be4:gc::gc.7:head [call rgw.gc_list in=25b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32) v8 -- 0x563697c023e0 con 0x563697b8d8c0 2023-10-05T19:28:55.015 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.010+0000 7f9f63553700 1 -- 172.21.15.167:0/344537782 <== osd.0 v2:172.21.15.118:6800/27690 15 ==== osd_op_reply(70 gc.7 [call out=15b] v0'0 uv33 ondisk = 0) v8 ==== 148+0+15 (crc 0 0 0) 0x7f9f4c03f020 con 0x563697b8d8c0 2023-10-05T19:28:55.015 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.010+0000 7f9f6621fe40 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2023-10-05T19:28:55.015 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.010+0000 7f9f6621fe40 1 -- 172.21.15.167:0/344537782 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:71 6.1d 6:beaa4be4:gc::gc.7:head [call version.read in=11b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32) v8 -- 0x563697c013d0 con 0x563697b8d8c0 2023-10-05T19:28:55.016 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.010+0000 7f9f63553700 1 -- 172.21.15.167:0/344537782 <== osd.0 v2:172.21.15.118:6800/27690 16 ==== osd_op_reply(71 gc.7 [call out=24b] v0'0 uv33 ondisk = 0) v8 ==== 148+0+24 (crc 0 0 0) 0x7f9f4c03f020 con 0x563697b8d8c0 2023-10-05T19:28:55.016 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.010+0000 7f9f6621fe40 1 -- 172.21.15.167:0/344537782 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:72 6.1d 6:beaa4be4:gc::gc.7:head [call rgw.gc_list in=25b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32) v8 -- 0x563697c023e0 con 0x563697b8d8c0 2023-10-05T19:28:55.017 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.010+0000 7f9f63553700 1 -- 172.21.15.167:0/344537782 <== osd.0 v2:172.21.15.118:6800/27690 17 ==== osd_op_reply(72 gc.7 [call out=15b] v0'0 uv33 ondisk = 0) v8 ==== 148+0+15 (crc 0 0 0) 0x7f9f4c03f020 con 0x563697b8d8c0 2023-10-05T19:28:55.017 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.010+0000 7f9f6621fe40 20 garbage collection: RGWGC::process cls_rgw_gc_list returned NO non expired entries, so setting cache entry to TRUE 2023-10-05T19:28:55.017 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.010+0000 7f9f6621fe40 1 -- 172.21.15.167:0/344537782 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:73 6.1d 6:beaa4be4:gc::gc.7:head [call rgw_gc.rgw_gc_queue_list_entries in=46b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32) v8 -- 0x563697c023e0 con 0x563697b8d8c0 2023-10-05T19:28:55.018 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.014+0000 7f9f63553700 1 -- 172.21.15.167:0/344537782 <== osd.0 v2:172.21.15.118:6800/27690 18 ==== osd_op_reply(73 gc.7 [call out=15b] v0'0 uv33 ondisk = 0) v8 ==== 148+0+15 (crc 0 0 0) 0x7f9f4c03f020 con 0x563697b8d8c0 2023-10-05T19:28:55.018 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.014+0000 7f9f6621fe40 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=0, truncated=0, next_marker='' 2023-10-05T19:28:55.018 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.014+0000 7f9f6621fe40 1 -- 172.21.15.167:0/344537782 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:74 6.1d 6:beaa4be4:gc::gc.7:head [call lock.unlock in=34b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e32) v8 -- 0x563697c023e0 con 0x563697b8d8c0 2023-10-05T19:28:55.021 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.014+0000 7f9f63553700 1 -- 172.21.15.167:0/344537782 <== osd.0 v2:172.21.15.118:6800/27690 19 ==== osd_op_reply(74 gc.7 [call] v32'34 uv34 ondisk = 0) v8 ==== 148+0+0 (crc 0 0 0) 0x7f9f4c03f020 con 0x563697b8d8c0 2023-10-05T19:28:55.021 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.014+0000 7f9f6621fe40 20 garbage collection: RGWGC::process entered with GC index_shard=8, max_secs=3600, expired_only=1 2023-10-05T19:28:55.021 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.014+0000 7f9f6621fe40 1 -- 172.21.15.167:0/344537782 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:75 6.12 6:4f78e9a2:gc::gc.8:head [call lock.lock in=50b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e32) v8 -- 0x563697c023e0 con 0x563697b9d5a0 2023-10-05T19:28:55.023 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.018+0000 7f9f63d54700 1 -- 172.21.15.167:0/344537782 <== osd.2 v2:172.21.15.167:6816/30037 14 ==== osd_op_reply(75 gc.8 [call] v32'20 uv20 ondisk = 0) v8 ==== 148+0+0 (crc 0 0 0) 0x7f9f5c024020 con 0x563697b9d5a0 2023-10-05T19:28:55.023 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.018+0000 7f9f6621fe40 1 -- 172.21.15.167:0/344537782 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:76 6.12 6:4f78e9a2:gc::gc.8:head [call rgw.gc_list in=25b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32) v8 -- 0x563697c023e0 con 0x563697b9d5a0 2023-10-05T19:28:55.024 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.018+0000 7f9f63d54700 1 -- 172.21.15.167:0/344537782 <== osd.2 v2:172.21.15.167:6816/30037 15 ==== osd_op_reply(76 gc.8 [call out=15b] v0'0 uv20 ondisk = 0) v8 ==== 148+0+15 (crc 0 0 0) 0x7f9f5c024020 con 0x563697b9d5a0 2023-10-05T19:28:55.024 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.018+0000 7f9f6621fe40 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2023-10-05T19:28:55.024 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.018+0000 7f9f6621fe40 1 -- 172.21.15.167:0/344537782 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:77 6.12 6:4f78e9a2:gc::gc.8:head [call version.read in=11b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32) v8 -- 0x563697c013d0 con 0x563697b9d5a0 2023-10-05T19:28:55.025 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.018+0000 7f9f63d54700 1 -- 172.21.15.167:0/344537782 <== osd.2 v2:172.21.15.167:6816/30037 16 ==== osd_op_reply(77 gc.8 [call out=24b] v0'0 uv20 ondisk = 0) v8 ==== 148+0+24 (crc 0 0 0) 0x7f9f5c024020 con 0x563697b9d5a0 2023-10-05T19:28:55.025 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.018+0000 7f9f6621fe40 1 -- 172.21.15.167:0/344537782 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:78 6.12 6:4f78e9a2:gc::gc.8:head [call rgw.gc_list in=25b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32) v8 -- 0x563697c023e0 con 0x563697b9d5a0 2023-10-05T19:28:55.025 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.018+0000 7f9f63d54700 1 -- 172.21.15.167:0/344537782 <== osd.2 v2:172.21.15.167:6816/30037 17 ==== osd_op_reply(78 gc.8 [call out=15b] v0'0 uv20 ondisk = 0) v8 ==== 148+0+15 (crc 0 0 0) 0x7f9f5c024020 con 0x563697b9d5a0 2023-10-05T19:28:55.026 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.018+0000 7f9f6621fe40 20 garbage collection: RGWGC::process cls_rgw_gc_list returned NO non expired entries, so setting cache entry to TRUE 2023-10-05T19:28:55.026 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.018+0000 7f9f6621fe40 1 -- 172.21.15.167:0/344537782 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:79 6.12 6:4f78e9a2:gc::gc.8:head [call rgw_gc.rgw_gc_queue_list_entries in=46b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32) v8 -- 0x563697c023e0 con 0x563697b9d5a0 2023-10-05T19:28:55.026 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.022+0000 7f9f63d54700 1 -- 172.21.15.167:0/344537782 <== osd.2 v2:172.21.15.167:6816/30037 18 ==== osd_op_reply(79 gc.8 [call out=15b] v0'0 uv20 ondisk = 0) v8 ==== 148+0+15 (crc 0 0 0) 0x7f9f5c024020 con 0x563697b9d5a0 2023-10-05T19:28:55.026 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.022+0000 7f9f6621fe40 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=0, truncated=0, next_marker='' 2023-10-05T19:28:55.026 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.022+0000 7f9f6621fe40 1 -- 172.21.15.167:0/344537782 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:80 6.12 6:4f78e9a2:gc::gc.8:head [call lock.unlock in=34b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e32) v8 -- 0x563697c023e0 con 0x563697b9d5a0 2023-10-05T19:28:55.029 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.022+0000 7f9f63d54700 1 -- 172.21.15.167:0/344537782 <== osd.2 v2:172.21.15.167:6816/30037 19 ==== osd_op_reply(80 gc.8 [call] v32'21 uv21 ondisk = 0) v8 ==== 148+0+0 (crc 0 0 0) 0x7f9f5c024020 con 0x563697b9d5a0 2023-10-05T19:28:55.029 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.022+0000 7f9f6621fe40 20 garbage collection: RGWGC::process entered with GC index_shard=9, max_secs=3600, expired_only=1 2023-10-05T19:28:55.029 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.022+0000 7f9f6621fe40 1 -- 172.21.15.167:0/344537782 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:81 6.1c 6:3e426232:gc::gc.9:head [call lock.lock in=50b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e32) v8 -- 0x563697c023e0 con 0x563697b95830 2023-10-05T19:28:55.032 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.026+0000 7f9f64555700 1 -- 172.21.15.167:0/344537782 <== osd.3 v2:172.21.15.167:6808/30036 25 ==== osd_op_reply(81 gc.9 [call] v32'38 uv38 ondisk = 0) v8 ==== 148+0+0 (crc 0 0 0) 0x7f9f54050040 con 0x563697b95830 2023-10-05T19:28:55.032 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.026+0000 7f9f6621fe40 1 -- 172.21.15.167:0/344537782 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:82 6.1c 6:3e426232:gc::gc.9:head [call rgw.gc_list in=25b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32) v8 -- 0x563697c023e0 con 0x563697b95830 2023-10-05T19:28:55.032 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.026+0000 7f9f64555700 1 -- 172.21.15.167:0/344537782 <== osd.3 v2:172.21.15.167:6808/30036 26 ==== osd_op_reply(82 gc.9 [call out=15b] v0'0 uv38 ondisk = 0) v8 ==== 148+0+15 (crc 0 0 0) 0x7f9f54050040 con 0x563697b95830 2023-10-05T19:28:55.033 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.026+0000 7f9f6621fe40 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2023-10-05T19:28:55.033 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.026+0000 7f9f6621fe40 1 -- 172.21.15.167:0/344537782 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:83 6.1c 6:3e426232:gc::gc.9:head [call version.read in=11b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32) v8 -- 0x563697c013d0 con 0x563697b95830 2023-10-05T19:28:55.033 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.026+0000 7f9f64555700 1 -- 172.21.15.167:0/344537782 <== osd.3 v2:172.21.15.167:6808/30036 27 ==== osd_op_reply(83 gc.9 [call out=24b] v0'0 uv38 ondisk = 0) v8 ==== 148+0+24 (crc 0 0 0) 0x7f9f54050040 con 0x563697b95830 2023-10-05T19:28:55.033 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.026+0000 7f9f6621fe40 1 -- 172.21.15.167:0/344537782 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:84 6.1c 6:3e426232:gc::gc.9:head [call rgw.gc_list in=25b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32) v8 -- 0x563697c023e0 con 0x563697b95830 2023-10-05T19:28:55.034 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.030+0000 7f9f64555700 1 -- 172.21.15.167:0/344537782 <== osd.3 v2:172.21.15.167:6808/30036 28 ==== osd_op_reply(84 gc.9 [call out=15b] v0'0 uv38 ondisk = 0) v8 ==== 148+0+15 (crc 0 0 0) 0x7f9f54050040 con 0x563697b95830 2023-10-05T19:28:55.034 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.030+0000 7f9f6621fe40 20 garbage collection: RGWGC::process cls_rgw_gc_list returned NO non expired entries, so setting cache entry to TRUE 2023-10-05T19:28:55.034 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.030+0000 7f9f6621fe40 1 -- 172.21.15.167:0/344537782 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:85 6.1c 6:3e426232:gc::gc.9:head [call rgw_gc.rgw_gc_queue_list_entries in=46b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32) v8 -- 0x563697c023e0 con 0x563697b95830 2023-10-05T19:28:55.035 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.030+0000 7f9f64555700 1 -- 172.21.15.167:0/344537782 <== osd.3 v2:172.21.15.167:6808/30036 29 ==== osd_op_reply(85 gc.9 [call out=15b] v0'0 uv38 ondisk = 0) v8 ==== 148+0+15 (crc 0 0 0) 0x7f9f54050040 con 0x563697b95830 2023-10-05T19:28:55.035 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.030+0000 7f9f6621fe40 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=0, truncated=0, next_marker='' 2023-10-05T19:28:55.035 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.030+0000 7f9f6621fe40 1 -- 172.21.15.167:0/344537782 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:86 6.1c 6:3e426232:gc::gc.9:head [call lock.unlock in=34b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e32) v8 -- 0x563697c023e0 con 0x563697b95830 2023-10-05T19:28:55.037 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.030+0000 7f9f64555700 1 -- 172.21.15.167:0/344537782 <== osd.3 v2:172.21.15.167:6808/30036 30 ==== osd_op_reply(86 gc.9 [call] v32'39 uv39 ondisk = 0) v8 ==== 148+0+0 (crc 0 0 0) 0x7f9f54050040 con 0x563697b95830 2023-10-05T19:28:55.037 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.030+0000 7f9f6621fe40 20 garbage collection: RGWGC::process entered with GC index_shard=10, max_secs=3600, expired_only=1 2023-10-05T19:28:55.037 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.030+0000 7f9f6621fe40 1 -- 172.21.15.167:0/344537782 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:87 6.3 6:c31f41d7:gc::gc.10:head [call lock.lock in=50b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e32) v8 -- 0x563697c023e0 con 0x563697b9d5a0 2023-10-05T19:28:55.039 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.034+0000 7f9f63d54700 1 -- 172.21.15.167:0/344537782 <== osd.2 v2:172.21.15.167:6816/30037 20 ==== osd_op_reply(87 gc.10 [call] v32'52 uv52 ondisk = 0) v8 ==== 149+0+0 (crc 0 0 0) 0x7f9f5c024020 con 0x563697b9d5a0 2023-10-05T19:28:55.040 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.034+0000 7f9f6621fe40 1 -- 172.21.15.167:0/344537782 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:88 6.3 6:c31f41d7:gc::gc.10:head [call rgw.gc_list in=25b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32) v8 -- 0x563697c023e0 con 0x563697b9d5a0 2023-10-05T19:28:55.040 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.034+0000 7f9f63d54700 1 -- 172.21.15.167:0/344537782 <== osd.2 v2:172.21.15.167:6816/30037 21 ==== osd_op_reply(88 gc.10 [call out=15b] v0'0 uv52 ondisk = 0) v8 ==== 149+0+15 (crc 0 0 0) 0x7f9f5c024020 con 0x563697b9d5a0 2023-10-05T19:28:55.040 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.034+0000 7f9f6621fe40 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2023-10-05T19:28:55.040 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.034+0000 7f9f6621fe40 1 -- 172.21.15.167:0/344537782 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:89 6.3 6:c31f41d7:gc::gc.10:head [call version.read in=11b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32) v8 -- 0x563697c013d0 con 0x563697b9d5a0 2023-10-05T19:28:55.041 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.034+0000 7f9f63d54700 1 -- 172.21.15.167:0/344537782 <== osd.2 v2:172.21.15.167:6816/30037 22 ==== osd_op_reply(89 gc.10 [call out=24b] v0'0 uv52 ondisk = 0) v8 ==== 149+0+24 (crc 0 0 0) 0x7f9f5c024020 con 0x563697b9d5a0 2023-10-05T19:28:55.041 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.034+0000 7f9f6621fe40 1 -- 172.21.15.167:0/344537782 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:90 6.3 6:c31f41d7:gc::gc.10:head [call rgw.gc_list in=25b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32) v8 -- 0x563697c023e0 con 0x563697b9d5a0 2023-10-05T19:28:55.042 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.038+0000 7f9f63d54700 1 -- 172.21.15.167:0/344537782 <== osd.2 v2:172.21.15.167:6816/30037 23 ==== osd_op_reply(90 gc.10 [call out=15b] v0'0 uv52 ondisk = 0) v8 ==== 149+0+15 (crc 0 0 0) 0x7f9f5c024020 con 0x563697b9d5a0 2023-10-05T19:28:55.042 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.038+0000 7f9f6621fe40 20 garbage collection: RGWGC::process cls_rgw_gc_list returned NO non expired entries, so setting cache entry to TRUE 2023-10-05T19:28:55.042 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.038+0000 7f9f6621fe40 1 -- 172.21.15.167:0/344537782 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:91 6.3 6:c31f41d7:gc::gc.10:head [call rgw_gc.rgw_gc_queue_list_entries in=46b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32) v8 -- 0x563697c023e0 con 0x563697b9d5a0 2023-10-05T19:28:55.042 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.038+0000 7f9f63d54700 1 -- 172.21.15.167:0/344537782 <== osd.2 v2:172.21.15.167:6816/30037 24 ==== osd_op_reply(91 gc.10 [call out=15b] v0'0 uv52 ondisk = 0) v8 ==== 149+0+15 (crc 0 0 0) 0x7f9f5c024020 con 0x563697b9d5a0 2023-10-05T19:28:55.043 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.038+0000 7f9f6621fe40 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=0, truncated=0, next_marker='' 2023-10-05T19:28:55.043 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.038+0000 7f9f6621fe40 1 -- 172.21.15.167:0/344537782 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:92 6.3 6:c31f41d7:gc::gc.10:head [call lock.unlock in=34b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e32) v8 -- 0x563697c023e0 con 0x563697b9d5a0 2023-10-05T19:28:55.045 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.038+0000 7f9f63d54700 1 -- 172.21.15.167:0/344537782 <== osd.2 v2:172.21.15.167:6816/30037 25 ==== osd_op_reply(92 gc.10 [call] v32'53 uv53 ondisk = 0) v8 ==== 149+0+0 (crc 0 0 0) 0x7f9f5c024020 con 0x563697b9d5a0 2023-10-05T19:28:55.045 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.038+0000 7f9f6621fe40 20 garbage collection: RGWGC::process entered with GC index_shard=11, max_secs=3600, expired_only=1 2023-10-05T19:28:55.045 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.038+0000 7f9f6621fe40 1 -- 172.21.15.167:0/344537782 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:93 6.13 6:c8333aab:gc::gc.11:head [call lock.lock in=50b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e32) v8 -- 0x563697c023e0 con 0x563697b8d8c0 2023-10-05T19:28:55.048 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.042+0000 7f9f63553700 1 -- 172.21.15.167:0/344537782 <== osd.0 v2:172.21.15.118:6800/27690 20 ==== osd_op_reply(93 gc.11 [call] v32'24 uv24 ondisk = 0) v8 ==== 149+0+0 (crc 0 0 0) 0x7f9f4c03f020 con 0x563697b8d8c0 2023-10-05T19:28:55.048 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.042+0000 7f9f6621fe40 1 -- 172.21.15.167:0/344537782 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:94 6.13 6:c8333aab:gc::gc.11:head [call rgw.gc_list in=25b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32) v8 -- 0x563697c023e0 con 0x563697b8d8c0 2023-10-05T19:28:55.048 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.042+0000 7f9f63553700 1 -- 172.21.15.167:0/344537782 <== osd.0 v2:172.21.15.118:6800/27690 21 ==== osd_op_reply(94 gc.11 [call out=15b] v0'0 uv24 ondisk = 0) v8 ==== 149+0+15 (crc 0 0 0) 0x7f9f4c03f020 con 0x563697b8d8c0 2023-10-05T19:28:55.048 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.042+0000 7f9f6621fe40 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2023-10-05T19:28:55.049 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.042+0000 7f9f6621fe40 1 -- 172.21.15.167:0/344537782 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:95 6.13 6:c8333aab:gc::gc.11:head [call version.read in=11b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32) v8 -- 0x563697c013d0 con 0x563697b8d8c0 2023-10-05T19:28:55.049 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.042+0000 7f9f63553700 1 -- 172.21.15.167:0/344537782 <== osd.0 v2:172.21.15.118:6800/27690 22 ==== osd_op_reply(95 gc.11 [call out=24b] v0'0 uv24 ondisk = 0) v8 ==== 149+0+24 (crc 0 0 0) 0x7f9f4c03f020 con 0x563697b8d8c0 2023-10-05T19:28:55.049 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.042+0000 7f9f6621fe40 1 -- 172.21.15.167:0/344537782 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:96 6.13 6:c8333aab:gc::gc.11:head [call rgw.gc_list in=25b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32) v8 -- 0x563697c023e0 con 0x563697b8d8c0 2023-10-05T19:28:55.050 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.046+0000 7f9f63553700 1 -- 172.21.15.167:0/344537782 <== osd.0 v2:172.21.15.118:6800/27690 23 ==== osd_op_reply(96 gc.11 [call out=15b] v0'0 uv24 ondisk = 0) v8 ==== 149+0+15 (crc 0 0 0) 0x7f9f4c03f020 con 0x563697b8d8c0 2023-10-05T19:28:55.050 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.046+0000 7f9f6621fe40 20 garbage collection: RGWGC::process cls_rgw_gc_list returned NO non expired entries, so setting cache entry to TRUE 2023-10-05T19:28:55.050 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.046+0000 7f9f6621fe40 1 -- 172.21.15.167:0/344537782 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:97 6.13 6:c8333aab:gc::gc.11:head [call rgw_gc.rgw_gc_queue_list_entries in=46b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32) v8 -- 0x563697c023e0 con 0x563697b8d8c0 2023-10-05T19:28:55.051 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.046+0000 7f9f63553700 1 -- 172.21.15.167:0/344537782 <== osd.0 v2:172.21.15.118:6800/27690 24 ==== osd_op_reply(97 gc.11 [call out=15b] v0'0 uv24 ondisk = 0) v8 ==== 149+0+15 (crc 0 0 0) 0x7f9f4c03f020 con 0x563697b8d8c0 2023-10-05T19:28:55.051 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.046+0000 7f9f6621fe40 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=0, truncated=0, next_marker='' 2023-10-05T19:28:55.051 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.046+0000 7f9f6621fe40 1 -- 172.21.15.167:0/344537782 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:98 6.13 6:c8333aab:gc::gc.11:head [call lock.unlock in=34b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e32) v8 -- 0x563697c023e0 con 0x563697b8d8c0 2023-10-05T19:28:55.054 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.050+0000 7f9f63553700 1 -- 172.21.15.167:0/344537782 <== osd.0 v2:172.21.15.118:6800/27690 25 ==== osd_op_reply(98 gc.11 [call] v32'25 uv25 ondisk = 0) v8 ==== 149+0+0 (crc 0 0 0) 0x7f9f4c03f020 con 0x563697b8d8c0 2023-10-05T19:28:55.054 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.050+0000 7f9f6621fe40 20 garbage collection: RGWGC::process entered with GC index_shard=12, max_secs=3600, expired_only=1 2023-10-05T19:28:55.054 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.050+0000 7f9f6621fe40 1 -- 172.21.15.167:0/344537782 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:99 6.15 6:ab569f81:gc::gc.12:head [call lock.lock in=50b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e32) v8 -- 0x563697c023e0 con 0x563697b9d5a0 2023-10-05T19:28:55.056 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.050+0000 7f9f63d54700 1 -- 172.21.15.167:0/344537782 <== osd.2 v2:172.21.15.167:6816/30037 26 ==== osd_op_reply(99 gc.12 [call] v32'20 uv20 ondisk = 0) v8 ==== 149+0+0 (crc 0 0 0) 0x7f9f5c024020 con 0x563697b9d5a0 2023-10-05T19:28:55.057 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.050+0000 7f9f6621fe40 1 -- 172.21.15.167:0/344537782 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:100 6.15 6:ab569f81:gc::gc.12:head [call rgw.gc_list in=25b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32) v8 -- 0x563697c023e0 con 0x563697b9d5a0 2023-10-05T19:28:55.057 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.050+0000 7f9f63d54700 1 -- 172.21.15.167:0/344537782 <== osd.2 v2:172.21.15.167:6816/30037 27 ==== osd_op_reply(100 gc.12 [call out=15b] v0'0 uv20 ondisk = 0) v8 ==== 149+0+15 (crc 0 0 0) 0x7f9f5c024020 con 0x563697b9d5a0 2023-10-05T19:28:55.057 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.050+0000 7f9f6621fe40 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2023-10-05T19:28:55.057 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.050+0000 7f9f6621fe40 1 -- 172.21.15.167:0/344537782 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:101 6.15 6:ab569f81:gc::gc.12:head [call version.read in=11b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32) v8 -- 0x563697c013d0 con 0x563697b9d5a0 2023-10-05T19:28:55.058 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.054+0000 7f9f63d54700 1 -- 172.21.15.167:0/344537782 <== osd.2 v2:172.21.15.167:6816/30037 28 ==== osd_op_reply(101 gc.12 [call out=24b] v0'0 uv20 ondisk = 0) v8 ==== 149+0+24 (crc 0 0 0) 0x7f9f5c024020 con 0x563697b9d5a0 2023-10-05T19:28:55.058 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.054+0000 7f9f6621fe40 1 -- 172.21.15.167:0/344537782 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:102 6.15 6:ab569f81:gc::gc.12:head [call rgw.gc_list in=25b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32) v8 -- 0x563697c023e0 con 0x563697b9d5a0 2023-10-05T19:28:55.059 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.054+0000 7f9f63d54700 1 -- 172.21.15.167:0/344537782 <== osd.2 v2:172.21.15.167:6816/30037 29 ==== osd_op_reply(102 gc.12 [call out=15b] v0'0 uv20 ondisk = 0) v8 ==== 149+0+15 (crc 0 0 0) 0x7f9f5c024020 con 0x563697b9d5a0 2023-10-05T19:28:55.059 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.054+0000 7f9f6621fe40 20 garbage collection: RGWGC::process cls_rgw_gc_list returned NO non expired entries, so setting cache entry to TRUE 2023-10-05T19:28:55.059 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.054+0000 7f9f6621fe40 1 -- 172.21.15.167:0/344537782 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:103 6.15 6:ab569f81:gc::gc.12:head [call rgw_gc.rgw_gc_queue_list_entries in=46b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32) v8 -- 0x563697c023e0 con 0x563697b9d5a0 2023-10-05T19:28:55.059 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.054+0000 7f9f63d54700 1 -- 172.21.15.167:0/344537782 <== osd.2 v2:172.21.15.167:6816/30037 30 ==== osd_op_reply(103 gc.12 [call out=15b] v0'0 uv20 ondisk = 0) v8 ==== 149+0+15 (crc 0 0 0) 0x7f9f5c024020 con 0x563697b9d5a0 2023-10-05T19:28:55.059 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.054+0000 7f9f6621fe40 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=0, truncated=0, next_marker='' 2023-10-05T19:28:55.060 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.054+0000 7f9f6621fe40 1 -- 172.21.15.167:0/344537782 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:104 6.15 6:ab569f81:gc::gc.12:head [call lock.unlock in=34b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e32) v8 -- 0x563697c023e0 con 0x563697b9d5a0 2023-10-05T19:28:55.062 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.058+0000 7f9f63d54700 1 -- 172.21.15.167:0/344537782 <== osd.2 v2:172.21.15.167:6816/30037 31 ==== osd_op_reply(104 gc.12 [call] v32'21 uv21 ondisk = 0) v8 ==== 149+0+0 (crc 0 0 0) 0x7f9f5c024020 con 0x563697b9d5a0 2023-10-05T19:28:55.062 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.058+0000 7f9f6621fe40 20 garbage collection: RGWGC::process entered with GC index_shard=13, max_secs=3600, expired_only=1 2023-10-05T19:28:55.062 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.058+0000 7f9f6621fe40 1 -- 172.21.15.167:0/344537782 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:105 6.0 6:0667bb13:gc::gc.13:head [call lock.lock in=50b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e32) v8 -- 0x563697c023e0 con 0x563697b8d8c0 2023-10-05T19:28:55.065 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.058+0000 7f9f63553700 1 -- 172.21.15.167:0/344537782 <== osd.0 v2:172.21.15.118:6800/27690 26 ==== osd_op_reply(105 gc.13 [call] v32'39 uv39 ondisk = 0) v8 ==== 149+0+0 (crc 0 0 0) 0x7f9f4c03f020 con 0x563697b8d8c0 2023-10-05T19:28:55.065 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.058+0000 7f9f6621fe40 1 -- 172.21.15.167:0/344537782 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:106 6.0 6:0667bb13:gc::gc.13:head [call rgw.gc_list in=25b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32) v8 -- 0x563697c023e0 con 0x563697b8d8c0 2023-10-05T19:28:55.065 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.062+0000 7f9f63553700 1 -- 172.21.15.167:0/344537782 <== osd.0 v2:172.21.15.118:6800/27690 27 ==== osd_op_reply(106 gc.13 [call out=15b] v0'0 uv39 ondisk = 0) v8 ==== 149+0+15 (crc 0 0 0) 0x7f9f4c03f020 con 0x563697b8d8c0 2023-10-05T19:28:55.066 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.062+0000 7f9f6621fe40 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2023-10-05T19:28:55.066 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.062+0000 7f9f6621fe40 1 -- 172.21.15.167:0/344537782 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:107 6.0 6:0667bb13:gc::gc.13:head [call version.read in=11b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32) v8 -- 0x563697c013d0 con 0x563697b8d8c0 2023-10-05T19:28:55.066 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.062+0000 7f9f63553700 1 -- 172.21.15.167:0/344537782 <== osd.0 v2:172.21.15.118:6800/27690 28 ==== osd_op_reply(107 gc.13 [call out=24b] v0'0 uv39 ondisk = 0) v8 ==== 149+0+24 (crc 0 0 0) 0x7f9f4c03f020 con 0x563697b8d8c0 2023-10-05T19:28:55.066 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.062+0000 7f9f6621fe40 1 -- 172.21.15.167:0/344537782 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:108 6.0 6:0667bb13:gc::gc.13:head [call rgw.gc_list in=25b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32) v8 -- 0x563697c023e0 con 0x563697b8d8c0 2023-10-05T19:28:55.067 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.062+0000 7f9f63553700 1 -- 172.21.15.167:0/344537782 <== osd.0 v2:172.21.15.118:6800/27690 29 ==== osd_op_reply(108 gc.13 [call out=15b] v0'0 uv39 ondisk = 0) v8 ==== 149+0+15 (crc 0 0 0) 0x7f9f4c03f020 con 0x563697b8d8c0 2023-10-05T19:28:55.067 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.062+0000 7f9f6621fe40 20 garbage collection: RGWGC::process cls_rgw_gc_list returned NO non expired entries, so setting cache entry to TRUE 2023-10-05T19:28:55.067 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.062+0000 7f9f6621fe40 1 -- 172.21.15.167:0/344537782 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:109 6.0 6:0667bb13:gc::gc.13:head [call rgw_gc.rgw_gc_queue_list_entries in=46b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32) v8 -- 0x563697c023e0 con 0x563697b8d8c0 2023-10-05T19:28:55.068 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.062+0000 7f9f63553700 1 -- 172.21.15.167:0/344537782 <== osd.0 v2:172.21.15.118:6800/27690 30 ==== osd_op_reply(109 gc.13 [call out=15b] v0'0 uv39 ondisk = 0) v8 ==== 149+0+15 (crc 0 0 0) 0x7f9f4c03f020 con 0x563697b8d8c0 2023-10-05T19:28:55.068 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.062+0000 7f9f6621fe40 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=0, truncated=0, next_marker='' 2023-10-05T19:28:55.068 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.062+0000 7f9f6621fe40 1 -- 172.21.15.167:0/344537782 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:110 6.0 6:0667bb13:gc::gc.13:head [call lock.unlock in=34b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e32) v8 -- 0x563697c023e0 con 0x563697b8d8c0 2023-10-05T19:28:55.071 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.066+0000 7f9f63553700 1 -- 172.21.15.167:0/344537782 <== osd.0 v2:172.21.15.118:6800/27690 31 ==== osd_op_reply(110 gc.13 [call] v32'40 uv40 ondisk = 0) v8 ==== 149+0+0 (crc 0 0 0) 0x7f9f4c03f020 con 0x563697b8d8c0 2023-10-05T19:28:55.071 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.066+0000 7f9f6621fe40 20 garbage collection: RGWGC::process entered with GC index_shard=14, max_secs=3600, expired_only=1 2023-10-05T19:28:55.071 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.066+0000 7f9f6621fe40 1 -- 172.21.15.167:0/344537782 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:111 6.4 6:227f63b5:gc::gc.14:head [call lock.lock in=50b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e32) v8 -- 0x563697c023e0 con 0x563697b91ce0 2023-10-05T19:28:55.074 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.070+0000 7f9f63d54700 1 -- 172.21.15.167:0/344537782 <== osd.1 v2:172.21.15.167:6800/30035 19 ==== osd_op_reply(111 gc.14 [call] v32'185 uv185 ondisk = 0) v8 ==== 149+0+0 (crc 0 0 0) 0x7f9f5c024020 con 0x563697b91ce0 2023-10-05T19:28:55.074 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.070+0000 7f9f6621fe40 1 -- 172.21.15.167:0/344537782 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:112 6.4 6:227f63b5:gc::gc.14:head [call rgw.gc_list in=25b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32) v8 -- 0x563697c023e0 con 0x563697b91ce0 2023-10-05T19:28:55.075 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.070+0000 7f9f63d54700 1 -- 172.21.15.167:0/344537782 <== osd.1 v2:172.21.15.167:6800/30035 20 ==== osd_op_reply(112 gc.14 [call out=15b] v0'0 uv185 ondisk = 0) v8 ==== 149+0+15 (crc 0 0 0) 0x7f9f5c024020 con 0x563697b91ce0 2023-10-05T19:28:55.075 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.070+0000 7f9f6621fe40 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2023-10-05T19:28:55.075 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.070+0000 7f9f6621fe40 1 -- 172.21.15.167:0/344537782 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:113 6.4 6:227f63b5:gc::gc.14:head [call version.read in=11b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32) v8 -- 0x563697c013d0 con 0x563697b91ce0 2023-10-05T19:28:55.075 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.070+0000 7f9f63d54700 1 -- 172.21.15.167:0/344537782 <== osd.1 v2:172.21.15.167:6800/30035 21 ==== osd_op_reply(113 gc.14 [call out=24b] v0'0 uv185 ondisk = 0) v8 ==== 149+0+24 (crc 0 0 0) 0x7f9f5c024020 con 0x563697b91ce0 2023-10-05T19:28:55.076 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.070+0000 7f9f6621fe40 1 -- 172.21.15.167:0/344537782 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:114 6.4 6:227f63b5:gc::gc.14:head [call rgw.gc_list in=25b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32) v8 -- 0x563697c023e0 con 0x563697b91ce0 2023-10-05T19:28:55.076 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.070+0000 7f9f63d54700 1 -- 172.21.15.167:0/344537782 <== osd.1 v2:172.21.15.167:6800/30035 22 ==== osd_op_reply(114 gc.14 [call out=15b] v0'0 uv185 ondisk = 0) v8 ==== 149+0+15 (crc 0 0 0) 0x7f9f5c024020 con 0x563697b91ce0 2023-10-05T19:28:55.076 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.070+0000 7f9f6621fe40 20 garbage collection: RGWGC::process cls_rgw_gc_list returned NO non expired entries, so setting cache entry to TRUE 2023-10-05T19:28:55.076 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.070+0000 7f9f6621fe40 1 -- 172.21.15.167:0/344537782 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:115 6.4 6:227f63b5:gc::gc.14:head [call rgw_gc.rgw_gc_queue_list_entries in=46b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32) v8 -- 0x563697c023e0 con 0x563697b91ce0 2023-10-05T19:28:55.077 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.070+0000 7f9f63d54700 1 -- 172.21.15.167:0/344537782 <== osd.1 v2:172.21.15.167:6800/30035 23 ==== osd_op_reply(115 gc.14 [call out=15b] v0'0 uv185 ondisk = 0) v8 ==== 149+0+15 (crc 0 0 0) 0x7f9f5c024020 con 0x563697b91ce0 2023-10-05T19:28:55.077 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.070+0000 7f9f6621fe40 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=0, truncated=0, next_marker='' 2023-10-05T19:28:55.077 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.070+0000 7f9f6621fe40 1 -- 172.21.15.167:0/344537782 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:116 6.4 6:227f63b5:gc::gc.14:head [call lock.unlock in=34b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e32) v8 -- 0x563697c023e0 con 0x563697b91ce0 2023-10-05T19:28:55.080 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.074+0000 7f9f63d54700 1 -- 172.21.15.167:0/344537782 <== osd.1 v2:172.21.15.167:6800/30035 24 ==== osd_op_reply(116 gc.14 [call] v32'186 uv186 ondisk = 0) v8 ==== 149+0+0 (crc 0 0 0) 0x7f9f5c024020 con 0x563697b91ce0 2023-10-05T19:28:55.080 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.074+0000 7f9f6621fe40 20 garbage collection: RGWGC::process entered with GC index_shard=15, max_secs=3600, expired_only=1 2023-10-05T19:28:55.080 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.074+0000 7f9f6621fe40 1 -- 172.21.15.167:0/344537782 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:117 6.7 6:e07a6548:gc::gc.15:head [call lock.lock in=50b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e32) v8 -- 0x563697c023e0 con 0x563697b95830 2023-10-05T19:28:55.083 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.078+0000 7f9f64555700 1 -- 172.21.15.167:0/344537782 <== osd.3 v2:172.21.15.167:6808/30036 31 ==== osd_op_reply(117 gc.15 [call] v32'44 uv44 ondisk = 0) v8 ==== 149+0+0 (crc 0 0 0) 0x7f9f54050040 con 0x563697b95830 2023-10-05T19:28:55.083 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.078+0000 7f9f6621fe40 1 -- 172.21.15.167:0/344537782 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:118 6.7 6:e07a6548:gc::gc.15:head [call rgw.gc_list in=25b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32) v8 -- 0x563697c023e0 con 0x563697b95830 2023-10-05T19:28:55.083 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.078+0000 7f9f64555700 1 -- 172.21.15.167:0/344537782 <== osd.3 v2:172.21.15.167:6808/30036 32 ==== osd_op_reply(118 gc.15 [call out=15b] v0'0 uv44 ondisk = 0) v8 ==== 149+0+15 (crc 0 0 0) 0x7f9f54050040 con 0x563697b95830 2023-10-05T19:28:55.084 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.078+0000 7f9f6621fe40 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2023-10-05T19:28:55.084 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.078+0000 7f9f6621fe40 1 -- 172.21.15.167:0/344537782 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:119 6.7 6:e07a6548:gc::gc.15:head [call version.read in=11b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32) v8 -- 0x563697c013d0 con 0x563697b95830 2023-10-05T19:28:55.084 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.078+0000 7f9f64555700 1 -- 172.21.15.167:0/344537782 <== osd.3 v2:172.21.15.167:6808/30036 33 ==== osd_op_reply(119 gc.15 [call out=24b] v0'0 uv44 ondisk = 0) v8 ==== 149+0+24 (crc 0 0 0) 0x7f9f54050040 con 0x563697b95830 2023-10-05T19:28:55.084 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.078+0000 7f9f6621fe40 1 -- 172.21.15.167:0/344537782 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:120 6.7 6:e07a6548:gc::gc.15:head [call rgw.gc_list in=25b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32) v8 -- 0x563697c023e0 con 0x563697b95830 2023-10-05T19:28:55.085 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.078+0000 7f9f64555700 1 -- 172.21.15.167:0/344537782 <== osd.3 v2:172.21.15.167:6808/30036 34 ==== osd_op_reply(120 gc.15 [call out=15b] v0'0 uv44 ondisk = 0) v8 ==== 149+0+15 (crc 0 0 0) 0x7f9f54050040 con 0x563697b95830 2023-10-05T19:28:55.085 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.078+0000 7f9f6621fe40 20 garbage collection: RGWGC::process cls_rgw_gc_list returned NO non expired entries, so setting cache entry to TRUE 2023-10-05T19:28:55.085 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.078+0000 7f9f6621fe40 1 -- 172.21.15.167:0/344537782 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:121 6.7 6:e07a6548:gc::gc.15:head [call rgw_gc.rgw_gc_queue_list_entries in=46b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32) v8 -- 0x563697c023e0 con 0x563697b95830 2023-10-05T19:28:55.085 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.082+0000 7f9f64555700 1 -- 172.21.15.167:0/344537782 <== osd.3 v2:172.21.15.167:6808/30036 35 ==== osd_op_reply(121 gc.15 [call out=15b] v0'0 uv44 ondisk = 0) v8 ==== 149+0+15 (crc 0 0 0) 0x7f9f54050040 con 0x563697b95830 2023-10-05T19:28:55.086 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.082+0000 7f9f6621fe40 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=0, truncated=0, next_marker='' 2023-10-05T19:28:55.086 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.082+0000 7f9f6621fe40 1 -- 172.21.15.167:0/344537782 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:122 6.7 6:e07a6548:gc::gc.15:head [call lock.unlock in=34b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e32) v8 -- 0x563697c023e0 con 0x563697b95830 2023-10-05T19:28:55.088 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.082+0000 7f9f64555700 1 -- 172.21.15.167:0/344537782 <== osd.3 v2:172.21.15.167:6808/30036 36 ==== osd_op_reply(122 gc.15 [call] v32'45 uv45 ondisk = 0) v8 ==== 149+0+0 (crc 0 0 0) 0x7f9f54050040 con 0x563697b95830 2023-10-05T19:28:55.088 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.082+0000 7f9f6621fe40 20 garbage collection: RGWGC::process entered with GC index_shard=16, max_secs=3600, expired_only=1 2023-10-05T19:28:55.088 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.082+0000 7f9f6621fe40 1 -- 172.21.15.167:0/344537782 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:123 6.3 6:c623b173:gc::gc.16:head [call lock.lock in=50b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e32) v8 -- 0x563697c023e0 con 0x563697b9d5a0 2023-10-05T19:28:55.091 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.086+0000 7f9f63d54700 1 -- 172.21.15.167:0/344537782 <== osd.2 v2:172.21.15.167:6816/30037 32 ==== osd_op_reply(123 gc.16 [call] v32'54 uv54 ondisk = 0) v8 ==== 149+0+0 (crc 0 0 0) 0x7f9f5c024020 con 0x563697b9d5a0 2023-10-05T19:28:55.091 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.086+0000 7f9f6621fe40 1 -- 172.21.15.167:0/344537782 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:124 6.3 6:c623b173:gc::gc.16:head [call rgw.gc_list in=25b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32) v8 -- 0x563697c023e0 con 0x563697b9d5a0 2023-10-05T19:28:55.092 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.086+0000 7f9f63d54700 1 -- 172.21.15.167:0/344537782 <== osd.2 v2:172.21.15.167:6816/30037 33 ==== osd_op_reply(124 gc.16 [call out=15b] v0'0 uv54 ondisk = 0) v8 ==== 149+0+15 (crc 0 0 0) 0x7f9f5c024020 con 0x563697b9d5a0 2023-10-05T19:28:55.092 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.086+0000 7f9f6621fe40 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2023-10-05T19:28:55.092 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.086+0000 7f9f6621fe40 1 -- 172.21.15.167:0/344537782 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:125 6.3 6:c623b173:gc::gc.16:head [call version.read in=11b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32) v8 -- 0x563697c013d0 con 0x563697b9d5a0 2023-10-05T19:28:55.092 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.086+0000 7f9f63d54700 1 -- 172.21.15.167:0/344537782 <== osd.2 v2:172.21.15.167:6816/30037 34 ==== osd_op_reply(125 gc.16 [call out=24b] v0'0 uv54 ondisk = 0) v8 ==== 149+0+24 (crc 0 0 0) 0x7f9f5c024020 con 0x563697b9d5a0 2023-10-05T19:28:55.092 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.086+0000 7f9f6621fe40 1 -- 172.21.15.167:0/344537782 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:126 6.3 6:c623b173:gc::gc.16:head [call rgw.gc_list in=25b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32) v8 -- 0x563697c023e0 con 0x563697b9d5a0 2023-10-05T19:28:55.093 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.086+0000 7f9f63d54700 1 -- 172.21.15.167:0/344537782 <== osd.2 v2:172.21.15.167:6816/30037 35 ==== osd_op_reply(126 gc.16 [call out=15b] v0'0 uv54 ondisk = 0) v8 ==== 149+0+15 (crc 0 0 0) 0x7f9f5c024020 con 0x563697b9d5a0 2023-10-05T19:28:55.093 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.086+0000 7f9f6621fe40 20 garbage collection: RGWGC::process cls_rgw_gc_list returned NO non expired entries, so setting cache entry to TRUE 2023-10-05T19:28:55.093 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.086+0000 7f9f6621fe40 1 -- 172.21.15.167:0/344537782 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:127 6.3 6:c623b173:gc::gc.16:head [call rgw_gc.rgw_gc_queue_list_entries in=46b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32) v8 -- 0x563697c023e0 con 0x563697b9d5a0 2023-10-05T19:28:55.093 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.086+0000 7f9f63d54700 1 -- 172.21.15.167:0/344537782 <== osd.2 v2:172.21.15.167:6816/30037 36 ==== osd_op_reply(127 gc.16 [call out=15b] v0'0 uv54 ondisk = 0) v8 ==== 149+0+15 (crc 0 0 0) 0x7f9f5c024020 con 0x563697b9d5a0 2023-10-05T19:28:55.093 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.086+0000 7f9f6621fe40 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=0, truncated=0, next_marker='' 2023-10-05T19:28:55.093 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.086+0000 7f9f6621fe40 1 -- 172.21.15.167:0/344537782 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:128 6.3 6:c623b173:gc::gc.16:head [call lock.unlock in=34b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e32) v8 -- 0x563697c023e0 con 0x563697b9d5a0 2023-10-05T19:28:55.095 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.090+0000 7f9f63d54700 1 -- 172.21.15.167:0/344537782 <== osd.2 v2:172.21.15.167:6816/30037 37 ==== osd_op_reply(128 gc.16 [call] v32'55 uv55 ondisk = 0) v8 ==== 149+0+0 (crc 0 0 0) 0x7f9f5c024020 con 0x563697b9d5a0 2023-10-05T19:28:55.095 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.090+0000 7f9f6621fe40 20 garbage collection: RGWGC::process entered with GC index_shard=17, max_secs=3600, expired_only=1 2023-10-05T19:28:55.095 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.090+0000 7f9f6621fe40 1 -- 172.21.15.167:0/344537782 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:129 6.16 6:6849fd06:gc::gc.17:head [call lock.lock in=50b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e32) v8 -- 0x563697c023e0 con 0x563697b95830 2023-10-05T19:28:55.097 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.090+0000 7f9f64555700 1 -- 172.21.15.167:0/344537782 <== osd.3 v2:172.21.15.167:6808/30036 37 ==== osd_op_reply(129 gc.17 [call] v32'20 uv20 ondisk = 0) v8 ==== 149+0+0 (crc 0 0 0) 0x7f9f54050040 con 0x563697b95830 2023-10-05T19:28:55.097 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.090+0000 7f9f6621fe40 1 -- 172.21.15.167:0/344537782 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:130 6.16 6:6849fd06:gc::gc.17:head [call rgw.gc_list in=25b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32) v8 -- 0x563697c023e0 con 0x563697b95830 2023-10-05T19:28:55.098 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.094+0000 7f9f64555700 1 -- 172.21.15.167:0/344537782 <== osd.3 v2:172.21.15.167:6808/30036 38 ==== osd_op_reply(130 gc.17 [call out=15b] v0'0 uv20 ondisk = 0) v8 ==== 149+0+15 (crc 0 0 0) 0x7f9f54050040 con 0x563697b95830 2023-10-05T19:28:55.098 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.094+0000 7f9f6621fe40 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2023-10-05T19:28:55.098 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.094+0000 7f9f6621fe40 1 -- 172.21.15.167:0/344537782 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:131 6.16 6:6849fd06:gc::gc.17:head [call version.read in=11b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32) v8 -- 0x563697c013d0 con 0x563697b95830 2023-10-05T19:28:55.098 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.094+0000 7f9f64555700 1 -- 172.21.15.167:0/344537782 <== osd.3 v2:172.21.15.167:6808/30036 39 ==== osd_op_reply(131 gc.17 [call out=24b] v0'0 uv20 ondisk = 0) v8 ==== 149+0+24 (crc 0 0 0) 0x7f9f54050040 con 0x563697b95830 2023-10-05T19:28:55.099 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.094+0000 7f9f6621fe40 1 -- 172.21.15.167:0/344537782 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:132 6.16 6:6849fd06:gc::gc.17:head [call rgw.gc_list in=25b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32) v8 -- 0x563697c023e0 con 0x563697b95830 2023-10-05T19:28:55.099 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.094+0000 7f9f64555700 1 -- 172.21.15.167:0/344537782 <== osd.3 v2:172.21.15.167:6808/30036 40 ==== osd_op_reply(132 gc.17 [call out=15b] v0'0 uv20 ondisk = 0) v8 ==== 149+0+15 (crc 0 0 0) 0x7f9f54050040 con 0x563697b95830 2023-10-05T19:28:55.099 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.094+0000 7f9f6621fe40 20 garbage collection: RGWGC::process cls_rgw_gc_list returned NO non expired entries, so setting cache entry to TRUE 2023-10-05T19:28:55.099 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.094+0000 7f9f6621fe40 1 -- 172.21.15.167:0/344537782 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:133 6.16 6:6849fd06:gc::gc.17:head [call rgw_gc.rgw_gc_queue_list_entries in=46b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32) v8 -- 0x563697c023e0 con 0x563697b95830 2023-10-05T19:28:55.100 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.094+0000 7f9f64555700 1 -- 172.21.15.167:0/344537782 <== osd.3 v2:172.21.15.167:6808/30036 41 ==== osd_op_reply(133 gc.17 [call out=15b] v0'0 uv20 ondisk = 0) v8 ==== 149+0+15 (crc 0 0 0) 0x7f9f54050040 con 0x563697b95830 2023-10-05T19:28:55.100 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.094+0000 7f9f6621fe40 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=0, truncated=0, next_marker='' 2023-10-05T19:28:55.100 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.094+0000 7f9f6621fe40 1 -- 172.21.15.167:0/344537782 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:134 6.16 6:6849fd06:gc::gc.17:head [call lock.unlock in=34b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e32) v8 -- 0x563697c023e0 con 0x563697b95830 2023-10-05T19:28:55.102 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.098+0000 7f9f64555700 1 -- 172.21.15.167:0/344537782 <== osd.3 v2:172.21.15.167:6808/30036 42 ==== osd_op_reply(134 gc.17 [call] v32'21 uv21 ondisk = 0) v8 ==== 149+0+0 (crc 0 0 0) 0x7f9f54050040 con 0x563697b95830 2023-10-05T19:28:55.102 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.098+0000 7f9f6621fe40 20 garbage collection: RGWGC::process entered with GC index_shard=18, max_secs=3600, expired_only=1 2023-10-05T19:28:55.102 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.098+0000 7f9f6621fe40 1 -- 172.21.15.167:0/344537782 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:135 6.d 6:b019ea83:gc::gc.18:head [call lock.lock in=50b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e32) v8 -- 0x563697c023e0 con 0x563697b91ce0 2023-10-05T19:28:55.105 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.098+0000 7f9f63d54700 1 -- 172.21.15.167:0/344537782 <== osd.1 v2:172.21.15.167:6800/30035 25 ==== osd_op_reply(135 gc.18 [call] v32'28 uv28 ondisk = 0) v8 ==== 149+0+0 (crc 0 0 0) 0x7f9f5c024020 con 0x563697b91ce0 2023-10-05T19:28:55.105 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.098+0000 7f9f6621fe40 1 -- 172.21.15.167:0/344537782 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:136 6.d 6:b019ea83:gc::gc.18:head [call rgw.gc_list in=25b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32) v8 -- 0x563697c023e0 con 0x563697b91ce0 2023-10-05T19:28:55.106 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.098+0000 7f9f63d54700 1 -- 172.21.15.167:0/344537782 <== osd.1 v2:172.21.15.167:6800/30035 26 ==== osd_op_reply(136 gc.18 [call out=15b] v0'0 uv28 ondisk = 0) v8 ==== 149+0+15 (crc 0 0 0) 0x7f9f5c024020 con 0x563697b91ce0 2023-10-05T19:28:55.106 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.098+0000 7f9f6621fe40 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2023-10-05T19:28:55.106 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.098+0000 7f9f6621fe40 1 -- 172.21.15.167:0/344537782 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:137 6.d 6:b019ea83:gc::gc.18:head [call version.read in=11b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32) v8 -- 0x563697c013d0 con 0x563697b91ce0 2023-10-05T19:28:55.106 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.102+0000 7f9f63d54700 1 -- 172.21.15.167:0/344537782 <== osd.1 v2:172.21.15.167:6800/30035 27 ==== osd_op_reply(137 gc.18 [call out=24b] v0'0 uv28 ondisk = 0) v8 ==== 149+0+24 (crc 0 0 0) 0x7f9f5c024020 con 0x563697b91ce0 2023-10-05T19:28:55.106 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.102+0000 7f9f6621fe40 1 -- 172.21.15.167:0/344537782 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:138 6.d 6:b019ea83:gc::gc.18:head [call rgw.gc_list in=25b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32) v8 -- 0x563697c023e0 con 0x563697b91ce0 2023-10-05T19:28:55.107 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.102+0000 7f9f63d54700 1 -- 172.21.15.167:0/344537782 <== osd.1 v2:172.21.15.167:6800/30035 28 ==== osd_op_reply(138 gc.18 [call out=15b] v0'0 uv28 ondisk = 0) v8 ==== 149+0+15 (crc 0 0 0) 0x7f9f5c024020 con 0x563697b91ce0 2023-10-05T19:28:55.107 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.102+0000 7f9f6621fe40 20 garbage collection: RGWGC::process cls_rgw_gc_list returned NO non expired entries, so setting cache entry to TRUE 2023-10-05T19:28:55.107 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.102+0000 7f9f6621fe40 1 -- 172.21.15.167:0/344537782 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:139 6.d 6:b019ea83:gc::gc.18:head [call rgw_gc.rgw_gc_queue_list_entries in=46b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32) v8 -- 0x563697c023e0 con 0x563697b91ce0 2023-10-05T19:28:55.107 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.102+0000 7f9f63d54700 1 -- 172.21.15.167:0/344537782 <== osd.1 v2:172.21.15.167:6800/30035 29 ==== osd_op_reply(139 gc.18 [call out=15b] v0'0 uv28 ondisk = 0) v8 ==== 149+0+15 (crc 0 0 0) 0x7f9f5c024020 con 0x563697b91ce0 2023-10-05T19:28:55.107 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.102+0000 7f9f6621fe40 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=0, truncated=0, next_marker='' 2023-10-05T19:28:55.108 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.102+0000 7f9f6621fe40 1 -- 172.21.15.167:0/344537782 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:140 6.d 6:b019ea83:gc::gc.18:head [call lock.unlock in=34b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e32) v8 -- 0x563697c023e0 con 0x563697b91ce0 2023-10-05T19:28:55.109 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.102+0000 7f9f63d54700 1 -- 172.21.15.167:0/344537782 <== osd.1 v2:172.21.15.167:6800/30035 30 ==== osd_op_reply(140 gc.18 [call] v32'29 uv29 ondisk = 0) v8 ==== 149+0+0 (crc 0 0 0) 0x7f9f5c024020 con 0x563697b91ce0 2023-10-05T19:28:55.110 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.106+0000 7f9f6621fe40 20 garbage collection: RGWGC::process entered with GC index_shard=19, max_secs=3600, expired_only=1 2023-10-05T19:28:55.110 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.106+0000 7f9f6621fe40 1 -- 172.21.15.167:0/344537782 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:141 6.4 6:24c8e1ae:gc::gc.19:head [call lock.lock in=50b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e32) v8 -- 0x563697c023e0 con 0x563697b91ce0 2023-10-05T19:28:55.112 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.106+0000 7f9f63d54700 1 -- 172.21.15.167:0/344537782 <== osd.1 v2:172.21.15.167:6800/30035 31 ==== osd_op_reply(141 gc.19 [call] v32'187 uv187 ondisk = 0) v8 ==== 149+0+0 (crc 0 0 0) 0x7f9f5c024020 con 0x563697b91ce0 2023-10-05T19:28:55.112 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.106+0000 7f9f6621fe40 1 -- 172.21.15.167:0/344537782 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:142 6.4 6:24c8e1ae:gc::gc.19:head [call rgw.gc_list in=25b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32) v8 -- 0x563697c023e0 con 0x563697b91ce0 2023-10-05T19:28:55.113 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.106+0000 7f9f63d54700 1 -- 172.21.15.167:0/344537782 <== osd.1 v2:172.21.15.167:6800/30035 32 ==== osd_op_reply(142 gc.19 [call out=15b] v0'0 uv187 ondisk = 0) v8 ==== 149+0+15 (crc 0 0 0) 0x7f9f5c024020 con 0x563697b91ce0 2023-10-05T19:28:55.113 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.106+0000 7f9f6621fe40 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2023-10-05T19:28:55.113 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.106+0000 7f9f6621fe40 1 -- 172.21.15.167:0/344537782 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:143 6.4 6:24c8e1ae:gc::gc.19:head [call version.read in=11b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32) v8 -- 0x563697c013d0 con 0x563697b91ce0 2023-10-05T19:28:55.114 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.106+0000 7f9f63d54700 1 -- 172.21.15.167:0/344537782 <== osd.1 v2:172.21.15.167:6800/30035 33 ==== osd_op_reply(143 gc.19 [call out=24b] v0'0 uv187 ondisk = 0) v8 ==== 149+0+24 (crc 0 0 0) 0x7f9f5c024020 con 0x563697b91ce0 2023-10-05T19:28:55.114 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.106+0000 7f9f6621fe40 1 -- 172.21.15.167:0/344537782 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:144 6.4 6:24c8e1ae:gc::gc.19:head [call rgw.gc_list in=25b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32) v8 -- 0x563697c023e0 con 0x563697b91ce0 2023-10-05T19:28:55.114 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.110+0000 7f9f63d54700 1 -- 172.21.15.167:0/344537782 <== osd.1 v2:172.21.15.167:6800/30035 34 ==== osd_op_reply(144 gc.19 [call out=15b] v0'0 uv187 ondisk = 0) v8 ==== 149+0+15 (crc 0 0 0) 0x7f9f5c024020 con 0x563697b91ce0 2023-10-05T19:28:55.114 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.110+0000 7f9f6621fe40 20 garbage collection: RGWGC::process cls_rgw_gc_list returned NO non expired entries, so setting cache entry to TRUE 2023-10-05T19:28:55.114 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.110+0000 7f9f6621fe40 1 -- 172.21.15.167:0/344537782 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:145 6.4 6:24c8e1ae:gc::gc.19:head [call rgw_gc.rgw_gc_queue_list_entries in=46b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32) v8 -- 0x563697c023e0 con 0x563697b91ce0 2023-10-05T19:28:55.116 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.110+0000 7f9f63d54700 1 -- 172.21.15.167:0/344537782 <== osd.1 v2:172.21.15.167:6800/30035 35 ==== osd_op_reply(145 gc.19 [call out=1701b] v0'0 uv187 ondisk = 0) v8 ==== 149+0+1701 (crc 0 0 0) 0x7f9f5c024020 con 0x563697b91ce0 2023-10-05T19:28:55.116 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.110+0000 7f9f6621fe40 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=1, truncated=0, next_marker='' 2023-10-05T19:28:55.116 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.110+0000 7f9f6621fe40 20 garbage collection: RGWGC::process iterating over entry tag='9a9df965-6612-45d4-a45f-6e3c889bacc2.4221.6358388886286532197', time=2023-10-05T19:28:54.286915+0000, chain.objs.size()=7 2023-10-05T19:28:55.116 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.110+0000 7f9f6621fe40 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:9a9df965-6612-45d4-a45f-6e3c889bacc2.4223.1__shadow_.efCgafU_mcZPUY_vovN7gwdjS1XOZ16_1 2023-10-05T19:28:55.116 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.110+0000 7f9f6621fe40 1 -- 172.21.15.167:0/344537782 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:146 3.1s0 3:8276627b:::9a9df965-6612-45d4-a45f-6e3c889bacc2.4223.1__shadow_.efCgafU_mcZPUY_vovN7gwdjS1XOZ16_1:head [call refcount.put in=84b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e32) v8 -- 0x563697c023e0 con 0x563697b8d8c0 2023-10-05T19:28:55.116 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.110+0000 7f9f6621fe40 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:9a9df965-6612-45d4-a45f-6e3c889bacc2.4223.1__shadow_.efCgafU_mcZPUY_vovN7gwdjS1XOZ16_2 2023-10-05T19:28:55.116 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.110+0000 7f9f6621fe40 1 -- 172.21.15.167:0/344537782 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:147 3.1es0 3:7a7f21e5:::9a9df965-6612-45d4-a45f-6e3c889bacc2.4223.1__shadow_.efCgafU_mcZPUY_vovN7gwdjS1XOZ16_2:head [call refcount.put in=84b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e32) v8 -- 0x563697c04ca0 con 0x563697b8d8c0 2023-10-05T19:28:55.116 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.110+0000 7f9f6621fe40 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:9a9df965-6612-45d4-a45f-6e3c889bacc2.4223.1__shadow_.efCgafU_mcZPUY_vovN7gwdjS1XOZ16_3 2023-10-05T19:28:55.116 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.110+0000 7f9f6621fe40 1 -- 172.21.15.167:0/344537782 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:148 3.19s0 3:99908955:::9a9df965-6612-45d4-a45f-6e3c889bacc2.4223.1__shadow_.efCgafU_mcZPUY_vovN7gwdjS1XOZ16_3:head [call refcount.put in=84b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e32) v8 -- 0x563697c023e0 con 0x563697b91ce0 2023-10-05T19:28:55.117 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.110+0000 7f9f6621fe40 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:9a9df965-6612-45d4-a45f-6e3c889bacc2.4223.1__shadow_.efCgafU_mcZPUY_vovN7gwdjS1XOZ16_4 2023-10-05T19:28:55.117 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.110+0000 7f9f6621fe40 1 -- 172.21.15.167:0/344537782 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:149 3.29s0 3:96d1fdc4:::9a9df965-6612-45d4-a45f-6e3c889bacc2.4223.1__shadow_.efCgafU_mcZPUY_vovN7gwdjS1XOZ16_4:head [call refcount.put in=84b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e32) v8 -- 0x563697c09a70 con 0x563697b8d8c0 2023-10-05T19:28:55.117 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.110+0000 7f9f6621fe40 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:9a9df965-6612-45d4-a45f-6e3c889bacc2.4223.1__shadow_.efCgafU_mcZPUY_vovN7gwdjS1XOZ16_5 2023-10-05T19:28:55.117 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.110+0000 7f9f6621fe40 1 -- 172.21.15.167:0/344537782 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:150 3.39s0 3:9d912608:::9a9df965-6612-45d4-a45f-6e3c889bacc2.4223.1__shadow_.efCgafU_mcZPUY_vovN7gwdjS1XOZ16_5:head [call refcount.put in=84b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e32) v8 -- 0x563697c0c5c0 con 0x563697b95830 2023-10-05T19:28:55.117 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.110+0000 7f9f6621fe40 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:9a9df965-6612-45d4-a45f-6e3c889bacc2.4223.1__shadow_.efCgafU_mcZPUY_vovN7gwdjS1XOZ16_6 2023-10-05T19:28:55.117 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.110+0000 7f9f6621fe40 1 -- 172.21.15.167:0/344537782 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:151 3.18s0 3:1bac2154:::9a9df965-6612-45d4-a45f-6e3c889bacc2.4223.1__shadow_.efCgafU_mcZPUY_vovN7gwdjS1XOZ16_6:head [call refcount.put in=84b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e32) v8 -- 0x563697c09a70 con 0x563697b95830 2023-10-05T19:28:55.117 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.110+0000 7f9f6621fe40 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:9a9df965-6612-45d4-a45f-6e3c889bacc2.4223.1__shadow_.efCgafU_mcZPUY_vovN7gwdjS1XOZ16_7 2023-10-05T19:28:55.117 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.110+0000 7f9f6621fe40 1 -- 172.21.15.167:0/344537782 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:152 3.33s0 3:ce912f48:::9a9df965-6612-45d4-a45f-6e3c889bacc2.4223.1__shadow_.efCgafU_mcZPUY_vovN7gwdjS1XOZ16_7:head [call refcount.put in=84b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e32) v8 -- 0x563697c116c0 con 0x563697b8d8c0 2023-10-05T19:28:55.122 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.118+0000 7f9f64555700 1 -- 172.21.15.167:0/344537782 <== osd.3 v2:172.21.15.167:6808/30036 43 ==== osd_op_reply(150 9a9df965-6612-45d4-a45f-6e3c889bacc2.4223.1__shadow_.efCgafU_mcZPUY_vovN7gwdjS1XOZ16_5 [call] v32'2 uv2 ondisk = 0) v8 ==== 230+0+0 (crc 0 0 0) 0x7f9f54050040 con 0x563697b95830 2023-10-05T19:28:55.122 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.118+0000 7f9f63d54700 1 -- 172.21.15.167:0/344537782 <== osd.1 v2:172.21.15.167:6800/30035 36 ==== osd_op_reply(148 9a9df965-6612-45d4-a45f-6e3c889bacc2.4223.1__shadow_.efCgafU_mcZPUY_vovN7gwdjS1XOZ16_3 [call] v32'2 uv2 ondisk = 0) v8 ==== 230+0+0 (crc 0 0 0) 0x7f9f5c024020 con 0x563697b91ce0 2023-10-05T19:28:55.122 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.118+0000 7f9f63553700 1 -- 172.21.15.167:0/344537782 <== osd.0 v2:172.21.15.118:6800/27690 32 ==== osd_op_reply(147 9a9df965-6612-45d4-a45f-6e3c889bacc2.4223.1__shadow_.efCgafU_mcZPUY_vovN7gwdjS1XOZ16_2 [call] v32'2 uv2 ondisk = 0) v8 ==== 230+0+0 (crc 0 0 0) 0x7f9f4c03f020 con 0x563697b8d8c0 2023-10-05T19:28:55.123 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.118+0000 7f9f63553700 1 -- 172.21.15.167:0/344537782 <== osd.0 v2:172.21.15.118:6800/27690 33 ==== osd_op_reply(152 9a9df965-6612-45d4-a45f-6e3c889bacc2.4223.1__shadow_.efCgafU_mcZPUY_vovN7gwdjS1XOZ16_7 [call] v32'2 uv2 ondisk = 0) v8 ==== 230+0+0 (crc 0 0 0) 0x7f9f4c03f020 con 0x563697b8d8c0 2023-10-05T19:28:55.123 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.118+0000 7f9f64555700 1 -- 172.21.15.167:0/344537782 <== osd.3 v2:172.21.15.167:6808/30036 44 ==== osd_op_reply(151 9a9df965-6612-45d4-a45f-6e3c889bacc2.4223.1__shadow_.efCgafU_mcZPUY_vovN7gwdjS1XOZ16_6 [call] v32'2 uv2 ondisk = 0) v8 ==== 230+0+0 (crc 0 0 0) 0x7f9f54050040 con 0x563697b95830 2023-10-05T19:28:55.125 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.118+0000 7f9f63553700 1 -- 172.21.15.167:0/344537782 <== osd.0 v2:172.21.15.118:6800/27690 34 ==== osd_op_reply(149 9a9df965-6612-45d4-a45f-6e3c889bacc2.4223.1__shadow_.efCgafU_mcZPUY_vovN7gwdjS1XOZ16_4 [call] v32'2 uv2 ondisk = 0) v8 ==== 230+0+0 (crc 0 0 0) 0x7f9f4c03f020 con 0x563697b8d8c0 2023-10-05T19:28:55.125 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.118+0000 7f9f63553700 1 -- 172.21.15.167:0/344537782 <== osd.0 v2:172.21.15.118:6800/27690 35 ==== osd_op_reply(146 9a9df965-6612-45d4-a45f-6e3c889bacc2.4223.1__shadow_.efCgafU_mcZPUY_vovN7gwdjS1XOZ16_1 [call] v32'2 uv2 ondisk = 0) v8 ==== 230+0+0 (crc 0 0 0) 0x7f9f4c03f020 con 0x563697b8d8c0 2023-10-05T19:28:55.125 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.118+0000 7f9f6621fe40 5 garbage collection: RGWGC::process removing entries, marker: 2023-10-05T19:28:55.126 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.118+0000 7f9f6621fe40 1 -- 172.21.15.167:0/344537782 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:153 6.4 6:24c8e1ae:gc::gc.19:head [call rgw_gc.rgw_gc_queue_remove_entries in=47b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e32) v8 -- 0x563697bffed0 con 0x563697b91ce0 2023-10-05T19:28:55.128 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.122+0000 7f9f63d54700 1 -- 172.21.15.167:0/344537782 <== osd.1 v2:172.21.15.167:6800/30035 37 ==== osd_op_reply(153 gc.19 [call] v32'188 uv188 ondisk = 0) v8 ==== 149+0+0 (crc 0 0 0) 0x7f9f5c024020 con 0x563697b91ce0 2023-10-05T19:28:55.128 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.122+0000 7f9f6621fe40 1 -- 172.21.15.167:0/344537782 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:154 6.4 6:24c8e1ae:gc::gc.19:head [call lock.unlock in=34b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e32) v8 -- 0x563697bffed0 con 0x563697b91ce0 2023-10-05T19:28:55.129 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.126+0000 7f9f63d54700 1 -- 172.21.15.167:0/344537782 <== osd.1 v2:172.21.15.167:6800/30035 38 ==== osd_op_reply(154 gc.19 [call] v32'189 uv189 ondisk = 0) v8 ==== 149+0+0 (crc 0 0 0) 0x7f9f5c024020 con 0x563697b91ce0 2023-10-05T19:28:55.130 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.126+0000 7f9f6621fe40 20 garbage collection: RGWGC::process entered with GC index_shard=20, max_secs=3600, expired_only=1 2023-10-05T19:28:55.130 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.126+0000 7f9f6621fe40 1 -- 172.21.15.167:0/344537782 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:155 6.7 6:e622f76a:gc::gc.20:head [call lock.lock in=50b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e32) v8 -- 0x563697bffed0 con 0x563697b95830 2023-10-05T19:28:55.131 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.126+0000 7f9f64555700 1 -- 172.21.15.167:0/344537782 <== osd.3 v2:172.21.15.167:6808/30036 45 ==== osd_op_reply(155 gc.20 [call] v32'46 uv46 ondisk = 0) v8 ==== 149+0+0 (crc 0 0 0) 0x7f9f54050040 con 0x563697b95830 2023-10-05T19:28:55.131 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.126+0000 7f9f6621fe40 1 -- 172.21.15.167:0/344537782 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:156 6.7 6:e622f76a:gc::gc.20:head [call rgw.gc_list in=25b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32) v8 -- 0x563697bffed0 con 0x563697b95830 2023-10-05T19:28:55.132 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.126+0000 7f9f64555700 1 -- 172.21.15.167:0/344537782 <== osd.3 v2:172.21.15.167:6808/30036 46 ==== osd_op_reply(156 gc.20 [call out=15b] v0'0 uv46 ondisk = 0) v8 ==== 149+0+15 (crc 0 0 0) 0x7f9f54050040 con 0x563697b95830 2023-10-05T19:28:55.132 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.126+0000 7f9f6621fe40 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2023-10-05T19:28:55.132 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.126+0000 7f9f6621fe40 1 -- 172.21.15.167:0/344537782 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:157 6.7 6:e622f76a:gc::gc.20:head [call version.read in=11b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32) v8 -- 0x563697bffed0 con 0x563697b95830 2023-10-05T19:28:55.133 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.126+0000 7f9f64555700 1 -- 172.21.15.167:0/344537782 <== osd.3 v2:172.21.15.167:6808/30036 47 ==== osd_op_reply(157 gc.20 [call out=24b] v0'0 uv46 ondisk = 0) v8 ==== 149+0+24 (crc 0 0 0) 0x7f9f54050040 con 0x563697b95830 2023-10-05T19:28:55.133 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.126+0000 7f9f6621fe40 1 -- 172.21.15.167:0/344537782 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:158 6.7 6:e622f76a:gc::gc.20:head [call rgw.gc_list in=25b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32) v8 -- 0x563697bffed0 con 0x563697b95830 2023-10-05T19:28:55.133 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.126+0000 7f9f64555700 1 -- 172.21.15.167:0/344537782 <== osd.3 v2:172.21.15.167:6808/30036 48 ==== osd_op_reply(158 gc.20 [call out=15b] v0'0 uv46 ondisk = 0) v8 ==== 149+0+15 (crc 0 0 0) 0x7f9f54050040 con 0x563697b95830 2023-10-05T19:28:55.133 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.126+0000 7f9f6621fe40 20 garbage collection: RGWGC::process cls_rgw_gc_list returned NO non expired entries, so setting cache entry to TRUE 2023-10-05T19:28:55.133 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.126+0000 7f9f6621fe40 1 -- 172.21.15.167:0/344537782 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:159 6.7 6:e622f76a:gc::gc.20:head [call rgw_gc.rgw_gc_queue_list_entries in=46b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32) v8 -- 0x563697bffed0 con 0x563697b95830 2023-10-05T19:28:55.133 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.126+0000 7f9f64555700 1 -- 172.21.15.167:0/344537782 <== osd.3 v2:172.21.15.167:6808/30036 49 ==== osd_op_reply(159 gc.20 [call out=15b] v0'0 uv46 ondisk = 0) v8 ==== 149+0+15 (crc 0 0 0) 0x7f9f54050040 con 0x563697b95830 2023-10-05T19:28:55.133 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.126+0000 7f9f6621fe40 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=0, truncated=0, next_marker='' 2023-10-05T19:28:55.134 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.126+0000 7f9f6621fe40 1 -- 172.21.15.167:0/344537782 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:160 6.7 6:e622f76a:gc::gc.20:head [call lock.unlock in=34b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e32) v8 -- 0x563697bffed0 con 0x563697b95830 2023-10-05T19:28:55.134 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.130+0000 7f9f64555700 1 -- 172.21.15.167:0/344537782 <== osd.3 v2:172.21.15.167:6808/30036 50 ==== osd_op_reply(160 gc.20 [call] v32'47 uv47 ondisk = 0) v8 ==== 149+0+0 (crc 0 0 0) 0x7f9f54050040 con 0x563697b95830 2023-10-05T19:28:55.134 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.130+0000 7f9f6621fe40 20 garbage collection: RGWGC::process entered with GC index_shard=21, max_secs=3600, expired_only=1 2023-10-05T19:28:55.135 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.130+0000 7f9f6621fe40 1 -- 172.21.15.167:0/344537782 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:161 6.a 6:5127e642:gc::gc.21:head [call lock.lock in=50b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e32) v8 -- 0x563697bffed0 con 0x563697b9d5a0 2023-10-05T19:28:55.136 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.130+0000 7f9f63d54700 1 -- 172.21.15.167:0/344537782 <== osd.2 v2:172.21.15.167:6816/30037 38 ==== osd_op_reply(161 gc.21 [call] v32'30 uv30 ondisk = 0) v8 ==== 149+0+0 (crc 0 0 0) 0x7f9f5c024020 con 0x563697b9d5a0 2023-10-05T19:28:55.136 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.130+0000 7f9f6621fe40 1 -- 172.21.15.167:0/344537782 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:162 6.a 6:5127e642:gc::gc.21:head [call rgw.gc_list in=25b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32) v8 -- 0x563697bffed0 con 0x563697b9d5a0 2023-10-05T19:28:55.137 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.130+0000 7f9f63d54700 1 -- 172.21.15.167:0/344537782 <== osd.2 v2:172.21.15.167:6816/30037 39 ==== osd_op_reply(162 gc.21 [call out=15b] v0'0 uv30 ondisk = 0) v8 ==== 149+0+15 (crc 0 0 0) 0x7f9f5c024020 con 0x563697b9d5a0 2023-10-05T19:28:55.137 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.130+0000 7f9f6621fe40 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2023-10-05T19:28:55.137 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.130+0000 7f9f6621fe40 1 -- 172.21.15.167:0/344537782 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:163 6.a 6:5127e642:gc::gc.21:head [call version.read in=11b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32) v8 -- 0x563697bffed0 con 0x563697b9d5a0 2023-10-05T19:28:55.138 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.130+0000 7f9f63d54700 1 -- 172.21.15.167:0/344537782 <== osd.2 v2:172.21.15.167:6816/30037 40 ==== osd_op_reply(163 gc.21 [call out=24b] v0'0 uv30 ondisk = 0) v8 ==== 149+0+24 (crc 0 0 0) 0x7f9f5c024020 con 0x563697b9d5a0 2023-10-05T19:28:55.138 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.130+0000 7f9f6621fe40 1 -- 172.21.15.167:0/344537782 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:164 6.a 6:5127e642:gc::gc.21:head [call rgw.gc_list in=25b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32) v8 -- 0x563697bffed0 con 0x563697b9d5a0 2023-10-05T19:28:55.138 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.130+0000 7f9f63d54700 1 -- 172.21.15.167:0/344537782 <== osd.2 v2:172.21.15.167:6816/30037 41 ==== osd_op_reply(164 gc.21 [call out=15b] v0'0 uv30 ondisk = 0) v8 ==== 149+0+15 (crc 0 0 0) 0x7f9f5c024020 con 0x563697b9d5a0 2023-10-05T19:28:55.138 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.130+0000 7f9f6621fe40 20 garbage collection: RGWGC::process cls_rgw_gc_list returned NO non expired entries, so setting cache entry to TRUE 2023-10-05T19:28:55.138 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.130+0000 7f9f6621fe40 1 -- 172.21.15.167:0/344537782 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:165 6.a 6:5127e642:gc::gc.21:head [call rgw_gc.rgw_gc_queue_list_entries in=46b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32) v8 -- 0x563697bffed0 con 0x563697b9d5a0 2023-10-05T19:28:55.138 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.130+0000 7f9f63d54700 1 -- 172.21.15.167:0/344537782 <== osd.2 v2:172.21.15.167:6816/30037 42 ==== osd_op_reply(165 gc.21 [call out=15b] v0'0 uv30 ondisk = 0) v8 ==== 149+0+15 (crc 0 0 0) 0x7f9f5c024020 con 0x563697b9d5a0 2023-10-05T19:28:55.138 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.130+0000 7f9f6621fe40 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=0, truncated=0, next_marker='' 2023-10-05T19:28:55.138 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.130+0000 7f9f6621fe40 1 -- 172.21.15.167:0/344537782 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:166 6.a 6:5127e642:gc::gc.21:head [call lock.unlock in=34b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e32) v8 -- 0x563697bffed0 con 0x563697b9d5a0 2023-10-05T19:28:55.139 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.134+0000 7f9f63d54700 1 -- 172.21.15.167:0/344537782 <== osd.2 v2:172.21.15.167:6816/30037 43 ==== osd_op_reply(166 gc.21 [call] v32'31 uv31 ondisk = 0) v8 ==== 149+0+0 (crc 0 0 0) 0x7f9f5c024020 con 0x563697b9d5a0 2023-10-05T19:28:55.139 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.134+0000 7f9f6621fe40 20 garbage collection: RGWGC::process entered with GC index_shard=22, max_secs=3600, expired_only=1 2023-10-05T19:28:55.139 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.134+0000 7f9f6621fe40 1 -- 172.21.15.167:0/344537782 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:167 6.11 6:8eb99c53:gc::gc.22:head [call lock.lock in=50b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e32) v8 -- 0x563697bffed0 con 0x563697b95830 2023-10-05T19:28:55.140 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.134+0000 7f9f64555700 1 -- 172.21.15.167:0/344537782 <== osd.3 v2:172.21.15.167:6808/30036 51 ==== osd_op_reply(167 gc.22 [call] v32'22 uv22 ondisk = 0) v8 ==== 149+0+0 (crc 0 0 0) 0x7f9f54050040 con 0x563697b95830 2023-10-05T19:28:55.140 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.134+0000 7f9f6621fe40 1 -- 172.21.15.167:0/344537782 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:168 6.11 6:8eb99c53:gc::gc.22:head [call rgw.gc_list in=25b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32) v8 -- 0x563697bffed0 con 0x563697b95830 2023-10-05T19:28:55.142 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.134+0000 7f9f64555700 1 -- 172.21.15.167:0/344537782 <== osd.3 v2:172.21.15.167:6808/30036 52 ==== osd_op_reply(168 gc.22 [call out=15b] v0'0 uv22 ondisk = 0) v8 ==== 149+0+15 (crc 0 0 0) 0x7f9f54050040 con 0x563697b95830 2023-10-05T19:28:55.142 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.134+0000 7f9f6621fe40 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2023-10-05T19:28:55.142 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.134+0000 7f9f6621fe40 1 -- 172.21.15.167:0/344537782 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:169 6.11 6:8eb99c53:gc::gc.22:head [call version.read in=11b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32) v8 -- 0x563697bffed0 con 0x563697b95830 2023-10-05T19:28:55.142 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.134+0000 7f9f64555700 1 -- 172.21.15.167:0/344537782 <== osd.3 v2:172.21.15.167:6808/30036 53 ==== osd_op_reply(169 gc.22 [call out=24b] v0'0 uv22 ondisk = 0) v8 ==== 149+0+24 (crc 0 0 0) 0x7f9f54050040 con 0x563697b95830 2023-10-05T19:28:55.142 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.134+0000 7f9f6621fe40 1 -- 172.21.15.167:0/344537782 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:170 6.11 6:8eb99c53:gc::gc.22:head [call rgw.gc_list in=25b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32) v8 -- 0x563697bffed0 con 0x563697b95830 2023-10-05T19:28:55.142 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.134+0000 7f9f64555700 1 -- 172.21.15.167:0/344537782 <== osd.3 v2:172.21.15.167:6808/30036 54 ==== osd_op_reply(170 gc.22 [call out=15b] v0'0 uv22 ondisk = 0) v8 ==== 149+0+15 (crc 0 0 0) 0x7f9f54050040 con 0x563697b95830 2023-10-05T19:28:55.142 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.138+0000 7f9f6621fe40 20 garbage collection: RGWGC::process cls_rgw_gc_list returned NO non expired entries, so setting cache entry to TRUE 2023-10-05T19:28:55.143 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.138+0000 7f9f6621fe40 1 -- 172.21.15.167:0/344537782 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:171 6.11 6:8eb99c53:gc::gc.22:head [call rgw_gc.rgw_gc_queue_list_entries in=46b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32) v8 -- 0x563697bffed0 con 0x563697b95830 2023-10-05T19:28:55.143 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.138+0000 7f9f64555700 1 -- 172.21.15.167:0/344537782 <== osd.3 v2:172.21.15.167:6808/30036 55 ==== osd_op_reply(171 gc.22 [call out=15b] v0'0 uv22 ondisk = 0) v8 ==== 149+0+15 (crc 0 0 0) 0x7f9f54050040 con 0x563697b95830 2023-10-05T19:28:55.143 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.138+0000 7f9f6621fe40 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=0, truncated=0, next_marker='' 2023-10-05T19:28:55.143 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.138+0000 7f9f6621fe40 1 -- 172.21.15.167:0/344537782 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:172 6.11 6:8eb99c53:gc::gc.22:head [call lock.unlock in=34b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e32) v8 -- 0x563697bffed0 con 0x563697b95830 2023-10-05T19:28:55.143 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.138+0000 7f9f64555700 1 -- 172.21.15.167:0/344537782 <== osd.3 v2:172.21.15.167:6808/30036 56 ==== osd_op_reply(172 gc.22 [call] v32'23 uv23 ondisk = 0) v8 ==== 149+0+0 (crc 0 0 0) 0x7f9f54050040 con 0x563697b95830 2023-10-05T19:28:55.144 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.138+0000 7f9f6621fe40 20 garbage collection: RGWGC::process entered with GC index_shard=23, max_secs=3600, expired_only=1 2023-10-05T19:28:55.144 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.138+0000 7f9f6621fe40 1 -- 172.21.15.167:0/344537782 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:173 6.8 6:17d7e8ed:gc::gc.23:head [call lock.lock in=50b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e32) v8 -- 0x563697bffed0 con 0x563697b95830 2023-10-05T19:28:55.145 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.138+0000 7f9f64555700 1 -- 172.21.15.167:0/344537782 <== osd.3 v2:172.21.15.167:6808/30036 57 ==== osd_op_reply(173 gc.23 [call] v32'32 uv32 ondisk = 0) v8 ==== 149+0+0 (crc 0 0 0) 0x7f9f54050040 con 0x563697b95830 2023-10-05T19:28:55.145 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.138+0000 7f9f6621fe40 1 -- 172.21.15.167:0/344537782 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:174 6.8 6:17d7e8ed:gc::gc.23:head [call rgw.gc_list in=25b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32) v8 -- 0x563697bffed0 con 0x563697b95830 2023-10-05T19:28:55.146 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.138+0000 7f9f64555700 1 -- 172.21.15.167:0/344537782 <== osd.3 v2:172.21.15.167:6808/30036 58 ==== osd_op_reply(174 gc.23 [call out=15b] v0'0 uv32 ondisk = 0) v8 ==== 149+0+15 (crc 0 0 0) 0x7f9f54050040 con 0x563697b95830 2023-10-05T19:28:55.146 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.138+0000 7f9f6621fe40 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2023-10-05T19:28:55.146 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.138+0000 7f9f6621fe40 1 -- 172.21.15.167:0/344537782 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:175 6.8 6:17d7e8ed:gc::gc.23:head [call version.read in=11b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32) v8 -- 0x563697bffed0 con 0x563697b95830 2023-10-05T19:28:55.146 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.142+0000 7f9f64555700 1 -- 172.21.15.167:0/344537782 <== osd.3 v2:172.21.15.167:6808/30036 59 ==== osd_op_reply(175 gc.23 [call out=24b] v0'0 uv32 ondisk = 0) v8 ==== 149+0+24 (crc 0 0 0) 0x7f9f54050040 con 0x563697b95830 2023-10-05T19:28:55.146 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.142+0000 7f9f6621fe40 1 -- 172.21.15.167:0/344537782 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:176 6.8 6:17d7e8ed:gc::gc.23:head [call rgw.gc_list in=25b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32) v8 -- 0x563697bffed0 con 0x563697b95830 2023-10-05T19:28:55.147 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.142+0000 7f9f64555700 1 -- 172.21.15.167:0/344537782 <== osd.3 v2:172.21.15.167:6808/30036 60 ==== osd_op_reply(176 gc.23 [call out=15b] v0'0 uv32 ondisk = 0) v8 ==== 149+0+15 (crc 0 0 0) 0x7f9f54050040 con 0x563697b95830 2023-10-05T19:28:55.147 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.142+0000 7f9f6621fe40 20 garbage collection: RGWGC::process cls_rgw_gc_list returned NO non expired entries, so setting cache entry to TRUE 2023-10-05T19:28:55.147 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.142+0000 7f9f6621fe40 1 -- 172.21.15.167:0/344537782 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:177 6.8 6:17d7e8ed:gc::gc.23:head [call rgw_gc.rgw_gc_queue_list_entries in=46b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32) v8 -- 0x563697bffed0 con 0x563697b95830 2023-10-05T19:28:55.147 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.142+0000 7f9f64555700 1 -- 172.21.15.167:0/344537782 <== osd.3 v2:172.21.15.167:6808/30036 61 ==== osd_op_reply(177 gc.23 [call out=15b] v0'0 uv32 ondisk = 0) v8 ==== 149+0+15 (crc 0 0 0) 0x7f9f54050040 con 0x563697b95830 2023-10-05T19:28:55.147 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.142+0000 7f9f6621fe40 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=0, truncated=0, next_marker='' 2023-10-05T19:28:55.147 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.142+0000 7f9f6621fe40 1 -- 172.21.15.167:0/344537782 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:178 6.8 6:17d7e8ed:gc::gc.23:head [call lock.unlock in=34b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e32) v8 -- 0x563697bffed0 con 0x563697b95830 2023-10-05T19:28:55.148 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.142+0000 7f9f64555700 1 -- 172.21.15.167:0/344537782 <== osd.3 v2:172.21.15.167:6808/30036 62 ==== osd_op_reply(178 gc.23 [call] v32'33 uv33 ondisk = 0) v8 ==== 149+0+0 (crc 0 0 0) 0x7f9f54050040 con 0x563697b95830 2023-10-05T19:28:55.148 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.142+0000 7f9f6621fe40 20 garbage collection: RGWGC::process entered with GC index_shard=24, max_secs=3600, expired_only=1 2023-10-05T19:28:55.149 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.142+0000 7f9f6621fe40 1 -- 172.21.15.167:0/344537782 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:179 6.1b 6:db9d130d:gc::gc.24:head [call lock.lock in=50b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e32) v8 -- 0x563697bffed0 con 0x563697b95830 2023-10-05T19:28:55.150 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.146+0000 7f9f64555700 1 -- 172.21.15.167:0/344537782 <== osd.3 v2:172.21.15.167:6808/30036 63 ==== osd_op_reply(179 gc.24 [call] v32'16 uv16 ondisk = 0) v8 ==== 149+0+0 (crc 0 0 0) 0x7f9f54050040 con 0x563697b95830 2023-10-05T19:28:55.150 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.146+0000 7f9f6621fe40 1 -- 172.21.15.167:0/344537782 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:180 6.1b 6:db9d130d:gc::gc.24:head [call rgw.gc_list in=25b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32) v8 -- 0x563697bffed0 con 0x563697b95830 2023-10-05T19:28:55.151 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.146+0000 7f9f64555700 1 -- 172.21.15.167:0/344537782 <== osd.3 v2:172.21.15.167:6808/30036 64 ==== osd_op_reply(180 gc.24 [call out=15b] v0'0 uv16 ondisk = 0) v8 ==== 149+0+15 (crc 0 0 0) 0x7f9f54050040 con 0x563697b95830 2023-10-05T19:28:55.151 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.146+0000 7f9f6621fe40 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2023-10-05T19:28:55.151 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.146+0000 7f9f6621fe40 1 -- 172.21.15.167:0/344537782 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:181 6.1b 6:db9d130d:gc::gc.24:head [call version.read in=11b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32) v8 -- 0x563697bffed0 con 0x563697b95830 2023-10-05T19:28:55.152 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.146+0000 7f9f64555700 1 -- 172.21.15.167:0/344537782 <== osd.3 v2:172.21.15.167:6808/30036 65 ==== osd_op_reply(181 gc.24 [call out=24b] v0'0 uv16 ondisk = 0) v8 ==== 149+0+24 (crc 0 0 0) 0x7f9f54050040 con 0x563697b95830 2023-10-05T19:28:55.152 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.146+0000 7f9f6621fe40 1 -- 172.21.15.167:0/344537782 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:182 6.1b 6:db9d130d:gc::gc.24:head [call rgw.gc_list in=25b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32) v8 -- 0x563697bffed0 con 0x563697b95830 2023-10-05T19:28:55.152 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.146+0000 7f9f64555700 1 -- 172.21.15.167:0/344537782 <== osd.3 v2:172.21.15.167:6808/30036 66 ==== osd_op_reply(182 gc.24 [call out=15b] v0'0 uv16 ondisk = 0) v8 ==== 149+0+15 (crc 0 0 0) 0x7f9f54050040 con 0x563697b95830 2023-10-05T19:28:55.152 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.146+0000 7f9f6621fe40 20 garbage collection: RGWGC::process cls_rgw_gc_list returned NO non expired entries, so setting cache entry to TRUE 2023-10-05T19:28:55.152 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.146+0000 7f9f6621fe40 1 -- 172.21.15.167:0/344537782 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:183 6.1b 6:db9d130d:gc::gc.24:head [call rgw_gc.rgw_gc_queue_list_entries in=46b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32) v8 -- 0x563697bffed0 con 0x563697b95830 2023-10-05T19:28:55.153 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.146+0000 7f9f64555700 1 -- 172.21.15.167:0/344537782 <== osd.3 v2:172.21.15.167:6808/30036 67 ==== osd_op_reply(183 gc.24 [call out=15b] v0'0 uv16 ondisk = 0) v8 ==== 149+0+15 (crc 0 0 0) 0x7f9f54050040 con 0x563697b95830 2023-10-05T19:28:55.153 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.146+0000 7f9f6621fe40 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=0, truncated=0, next_marker='' 2023-10-05T19:28:55.153 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.146+0000 7f9f6621fe40 1 -- 172.21.15.167:0/344537782 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:184 6.1b 6:db9d130d:gc::gc.24:head [call lock.unlock in=34b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e32) v8 -- 0x563697bffed0 con 0x563697b95830 2023-10-05T19:28:55.154 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.150+0000 7f9f64555700 1 -- 172.21.15.167:0/344537782 <== osd.3 v2:172.21.15.167:6808/30036 68 ==== osd_op_reply(184 gc.24 [call] v32'17 uv17 ondisk = 0) v8 ==== 149+0+0 (crc 0 0 0) 0x7f9f54050040 con 0x563697b95830 2023-10-05T19:28:55.155 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.150+0000 7f9f6621fe40 20 garbage collection: RGWGC::process entered with GC index_shard=25, max_secs=3600, expired_only=1 2023-10-05T19:28:55.155 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.150+0000 7f9f6621fe40 1 -- 172.21.15.167:0/344537782 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:185 6.9 6:921ecdb1:gc::gc.25:head [call lock.lock in=50b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e32) v8 -- 0x563697bffed0 con 0x563697b95830 2023-10-05T19:28:55.156 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.150+0000 7f9f64555700 1 -- 172.21.15.167:0/344537782 <== osd.3 v2:172.21.15.167:6808/30036 69 ==== osd_op_reply(185 gc.25 [call] v32'38 uv38 ondisk = 0) v8 ==== 149+0+0 (crc 0 0 0) 0x7f9f54050040 con 0x563697b95830 2023-10-05T19:28:55.156 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.150+0000 7f9f6621fe40 1 -- 172.21.15.167:0/344537782 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:186 6.9 6:921ecdb1:gc::gc.25:head [call rgw.gc_list in=25b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32) v8 -- 0x563697bffed0 con 0x563697b95830 2023-10-05T19:28:55.157 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.150+0000 7f9f64555700 1 -- 172.21.15.167:0/344537782 <== osd.3 v2:172.21.15.167:6808/30036 70 ==== osd_op_reply(186 gc.25 [call out=15b] v0'0 uv38 ondisk = 0) v8 ==== 149+0+15 (crc 0 0 0) 0x7f9f54050040 con 0x563697b95830 2023-10-05T19:28:55.157 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.150+0000 7f9f6621fe40 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2023-10-05T19:28:55.157 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.150+0000 7f9f6621fe40 1 -- 172.21.15.167:0/344537782 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:187 6.9 6:921ecdb1:gc::gc.25:head [call version.read in=11b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32) v8 -- 0x563697bffed0 con 0x563697b95830 2023-10-05T19:28:55.157 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.150+0000 7f9f64555700 1 -- 172.21.15.167:0/344537782 <== osd.3 v2:172.21.15.167:6808/30036 71 ==== osd_op_reply(187 gc.25 [call out=24b] v0'0 uv38 ondisk = 0) v8 ==== 149+0+24 (crc 0 0 0) 0x7f9f54050040 con 0x563697b95830 2023-10-05T19:28:55.157 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.150+0000 7f9f6621fe40 1 -- 172.21.15.167:0/344537782 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:188 6.9 6:921ecdb1:gc::gc.25:head [call rgw.gc_list in=25b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32) v8 -- 0x563697bffed0 con 0x563697b95830 2023-10-05T19:28:55.158 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.154+0000 7f9f64555700 1 -- 172.21.15.167:0/344537782 <== osd.3 v2:172.21.15.167:6808/30036 72 ==== osd_op_reply(188 gc.25 [call out=15b] v0'0 uv38 ondisk = 0) v8 ==== 149+0+15 (crc 0 0 0) 0x7f9f54050040 con 0x563697b95830 2023-10-05T19:28:55.158 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.154+0000 7f9f6621fe40 20 garbage collection: RGWGC::process cls_rgw_gc_list returned NO non expired entries, so setting cache entry to TRUE 2023-10-05T19:28:55.159 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.154+0000 7f9f6621fe40 1 -- 172.21.15.167:0/344537782 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:189 6.9 6:921ecdb1:gc::gc.25:head [call rgw_gc.rgw_gc_queue_list_entries in=46b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32) v8 -- 0x563697bffed0 con 0x563697b95830 2023-10-05T19:28:55.159 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.154+0000 7f9f64555700 1 -- 172.21.15.167:0/344537782 <== osd.3 v2:172.21.15.167:6808/30036 73 ==== osd_op_reply(189 gc.25 [call out=15b] v0'0 uv38 ondisk = 0) v8 ==== 149+0+15 (crc 0 0 0) 0x7f9f54050040 con 0x563697b95830 2023-10-05T19:28:55.159 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.154+0000 7f9f6621fe40 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=0, truncated=0, next_marker='' 2023-10-05T19:28:55.159 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.154+0000 7f9f6621fe40 1 -- 172.21.15.167:0/344537782 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:190 6.9 6:921ecdb1:gc::gc.25:head [call lock.unlock in=34b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e32) v8 -- 0x563697bffed0 con 0x563697b95830 2023-10-05T19:28:55.160 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.154+0000 7f9f64555700 1 -- 172.21.15.167:0/344537782 <== osd.3 v2:172.21.15.167:6808/30036 74 ==== osd_op_reply(190 gc.25 [call] v32'39 uv39 ondisk = 0) v8 ==== 149+0+0 (crc 0 0 0) 0x7f9f54050040 con 0x563697b95830 2023-10-05T19:28:55.160 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.154+0000 7f9f6621fe40 20 garbage collection: RGWGC::process entered with GC index_shard=26, max_secs=3600, expired_only=1 2023-10-05T19:28:55.160 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.154+0000 7f9f6621fe40 1 -- 172.21.15.167:0/344537782 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:191 6.1f 6:fde4dd55:gc::gc.26:head [call lock.lock in=50b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e32) v8 -- 0x563697bffed0 con 0x563697b8d8c0 2023-10-05T19:28:55.162 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.158+0000 7f9f63553700 1 -- 172.21.15.167:0/344537782 <== osd.0 v2:172.21.15.118:6800/27690 36 ==== osd_op_reply(191 gc.26 [call] v32'22 uv22 ondisk = 0) v8 ==== 149+0+0 (crc 0 0 0) 0x7f9f4c03f020 con 0x563697b8d8c0 2023-10-05T19:28:55.162 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.158+0000 7f9f6621fe40 1 -- 172.21.15.167:0/344537782 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:192 6.1f 6:fde4dd55:gc::gc.26:head [call rgw.gc_list in=25b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32) v8 -- 0x563697bffed0 con 0x563697b8d8c0 2023-10-05T19:28:55.163 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.158+0000 7f9f63553700 1 -- 172.21.15.167:0/344537782 <== osd.0 v2:172.21.15.118:6800/27690 37 ==== osd_op_reply(192 gc.26 [call out=15b] v0'0 uv22 ondisk = 0) v8 ==== 149+0+15 (crc 0 0 0) 0x7f9f4c03f020 con 0x563697b8d8c0 2023-10-05T19:28:55.163 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.158+0000 7f9f6621fe40 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2023-10-05T19:28:55.163 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.158+0000 7f9f6621fe40 1 -- 172.21.15.167:0/344537782 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:193 6.1f 6:fde4dd55:gc::gc.26:head [call version.read in=11b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32) v8 -- 0x563697bffed0 con 0x563697b8d8c0 2023-10-05T19:28:55.164 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.158+0000 7f9f63553700 1 -- 172.21.15.167:0/344537782 <== osd.0 v2:172.21.15.118:6800/27690 38 ==== osd_op_reply(193 gc.26 [call out=24b] v0'0 uv22 ondisk = 0) v8 ==== 149+0+24 (crc 0 0 0) 0x7f9f4c03f020 con 0x563697b8d8c0 2023-10-05T19:28:55.164 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.158+0000 7f9f6621fe40 1 -- 172.21.15.167:0/344537782 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:194 6.1f 6:fde4dd55:gc::gc.26:head [call rgw.gc_list in=25b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32) v8 -- 0x563697bffed0 con 0x563697b8d8c0 2023-10-05T19:28:55.165 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.158+0000 7f9f63553700 1 -- 172.21.15.167:0/344537782 <== osd.0 v2:172.21.15.118:6800/27690 39 ==== osd_op_reply(194 gc.26 [call out=15b] v0'0 uv22 ondisk = 0) v8 ==== 149+0+15 (crc 0 0 0) 0x7f9f4c03f020 con 0x563697b8d8c0 2023-10-05T19:28:55.165 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.158+0000 7f9f6621fe40 20 garbage collection: RGWGC::process cls_rgw_gc_list returned NO non expired entries, so setting cache entry to TRUE 2023-10-05T19:28:55.165 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.158+0000 7f9f6621fe40 1 -- 172.21.15.167:0/344537782 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:195 6.1f 6:fde4dd55:gc::gc.26:head [call rgw_gc.rgw_gc_queue_list_entries in=46b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32) v8 -- 0x563697bffed0 con 0x563697b8d8c0 2023-10-05T19:28:55.166 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.162+0000 7f9f63553700 1 -- 172.21.15.167:0/344537782 <== osd.0 v2:172.21.15.118:6800/27690 40 ==== osd_op_reply(195 gc.26 [call out=15b] v0'0 uv22 ondisk = 0) v8 ==== 149+0+15 (crc 0 0 0) 0x7f9f4c03f020 con 0x563697b8d8c0 2023-10-05T19:28:55.166 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.162+0000 7f9f6621fe40 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=0, truncated=0, next_marker='' 2023-10-05T19:28:55.166 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.162+0000 7f9f6621fe40 1 -- 172.21.15.167:0/344537782 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:196 6.1f 6:fde4dd55:gc::gc.26:head [call lock.unlock in=34b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e32) v8 -- 0x563697bffed0 con 0x563697b8d8c0 2023-10-05T19:28:55.168 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.162+0000 7f9f63553700 1 -- 172.21.15.167:0/344537782 <== osd.0 v2:172.21.15.118:6800/27690 41 ==== osd_op_reply(196 gc.26 [call] v32'23 uv23 ondisk = 0) v8 ==== 149+0+0 (crc 0 0 0) 0x7f9f4c03f020 con 0x563697b8d8c0 2023-10-05T19:28:55.168 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.162+0000 7f9f6621fe40 20 garbage collection: RGWGC::process entered with GC index_shard=27, max_secs=3600, expired_only=1 2023-10-05T19:28:55.168 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.162+0000 7f9f6621fe40 1 -- 172.21.15.167:0/344537782 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:197 6.1a 6:5909e671:gc::gc.27:head [call lock.lock in=50b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e32) v8 -- 0x563697bffed0 con 0x563697b95830 2023-10-05T19:28:55.171 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.166+0000 7f9f64555700 1 -- 172.21.15.167:0/344537782 <== osd.3 v2:172.21.15.167:6808/30036 75 ==== osd_op_reply(197 gc.27 [call] v32'20 uv20 ondisk = 0) v8 ==== 149+0+0 (crc 0 0 0) 0x7f9f54050040 con 0x563697b95830 2023-10-05T19:28:55.171 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.166+0000 7f9f6621fe40 1 -- 172.21.15.167:0/344537782 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:198 6.1a 6:5909e671:gc::gc.27:head [call rgw.gc_list in=25b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32) v8 -- 0x563697bffed0 con 0x563697b95830 2023-10-05T19:28:55.171 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.166+0000 7f9f64555700 1 -- 172.21.15.167:0/344537782 <== osd.3 v2:172.21.15.167:6808/30036 76 ==== osd_op_reply(198 gc.27 [call out=15b] v0'0 uv20 ondisk = 0) v8 ==== 149+0+15 (crc 0 0 0) 0x7f9f54050040 con 0x563697b95830 2023-10-05T19:28:55.171 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.166+0000 7f9f6621fe40 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2023-10-05T19:28:55.171 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.166+0000 7f9f6621fe40 1 -- 172.21.15.167:0/344537782 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:199 6.1a 6:5909e671:gc::gc.27:head [call version.read in=11b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32) v8 -- 0x563697bffed0 con 0x563697b95830 2023-10-05T19:28:55.172 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.166+0000 7f9f64555700 1 -- 172.21.15.167:0/344537782 <== osd.3 v2:172.21.15.167:6808/30036 77 ==== osd_op_reply(199 gc.27 [call out=24b] v0'0 uv20 ondisk = 0) v8 ==== 149+0+24 (crc 0 0 0) 0x7f9f54050040 con 0x563697b95830 2023-10-05T19:28:55.172 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.166+0000 7f9f6621fe40 1 -- 172.21.15.167:0/344537782 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:200 6.1a 6:5909e671:gc::gc.27:head [call rgw.gc_list in=25b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32) v8 -- 0x563697bffed0 con 0x563697b95830 2023-10-05T19:28:55.172 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.166+0000 7f9f64555700 1 -- 172.21.15.167:0/344537782 <== osd.3 v2:172.21.15.167:6808/30036 78 ==== osd_op_reply(200 gc.27 [call out=15b] v0'0 uv20 ondisk = 0) v8 ==== 149+0+15 (crc 0 0 0) 0x7f9f54050040 con 0x563697b95830 2023-10-05T19:28:55.172 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.166+0000 7f9f6621fe40 20 garbage collection: RGWGC::process cls_rgw_gc_list returned NO non expired entries, so setting cache entry to TRUE 2023-10-05T19:28:55.173 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.166+0000 7f9f6621fe40 1 -- 172.21.15.167:0/344537782 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:201 6.1a 6:5909e671:gc::gc.27:head [call rgw_gc.rgw_gc_queue_list_entries in=46b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32) v8 -- 0x563697bffed0 con 0x563697b95830 2023-10-05T19:28:55.173 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.166+0000 7f9f64555700 1 -- 172.21.15.167:0/344537782 <== osd.3 v2:172.21.15.167:6808/30036 79 ==== osd_op_reply(201 gc.27 [call out=15b] v0'0 uv20 ondisk = 0) v8 ==== 149+0+15 (crc 0 0 0) 0x7f9f54050040 con 0x563697b95830 2023-10-05T19:28:55.173 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.166+0000 7f9f6621fe40 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=0, truncated=0, next_marker='' 2023-10-05T19:28:55.173 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.166+0000 7f9f6621fe40 1 -- 172.21.15.167:0/344537782 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:202 6.1a 6:5909e671:gc::gc.27:head [call lock.unlock in=34b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e32) v8 -- 0x563697bffed0 con 0x563697b95830 2023-10-05T19:28:55.175 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.170+0000 7f9f64555700 1 -- 172.21.15.167:0/344537782 <== osd.3 v2:172.21.15.167:6808/30036 80 ==== osd_op_reply(202 gc.27 [call] v32'21 uv21 ondisk = 0) v8 ==== 149+0+0 (crc 0 0 0) 0x7f9f54050040 con 0x563697b95830 2023-10-05T19:28:55.177 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.170+0000 7f9f6621fe40 20 garbage collection: RGWGC::process entered with GC index_shard=28, max_secs=3600, expired_only=1 2023-10-05T19:28:55.177 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.170+0000 7f9f6621fe40 1 -- 172.21.15.167:0/344537782 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:203 6.1d 6:b88e3271:gc::gc.28:head [call lock.lock in=50b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e32) v8 -- 0x563697bffed0 con 0x563697b8d8c0 2023-10-05T19:28:55.178 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.174+0000 7f9f63553700 1 -- 172.21.15.167:0/344537782 <== osd.0 v2:172.21.15.118:6800/27690 42 ==== osd_op_reply(203 gc.28 [call] v32'35 uv35 ondisk = 0) v8 ==== 149+0+0 (crc 0 0 0) 0x7f9f4c03f020 con 0x563697b8d8c0 2023-10-05T19:28:55.178 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.174+0000 7f9f6621fe40 1 -- 172.21.15.167:0/344537782 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:204 6.1d 6:b88e3271:gc::gc.28:head [call rgw.gc_list in=25b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32) v8 -- 0x563697bffed0 con 0x563697b8d8c0 2023-10-05T19:28:55.179 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.174+0000 7f9f63553700 1 -- 172.21.15.167:0/344537782 <== osd.0 v2:172.21.15.118:6800/27690 43 ==== osd_op_reply(204 gc.28 [call out=15b] v0'0 uv35 ondisk = 0) v8 ==== 149+0+15 (crc 0 0 0) 0x7f9f4c03f020 con 0x563697b8d8c0 2023-10-05T19:28:55.179 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.174+0000 7f9f6621fe40 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2023-10-05T19:28:55.179 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.174+0000 7f9f6621fe40 1 -- 172.21.15.167:0/344537782 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:205 6.1d 6:b88e3271:gc::gc.28:head [call version.read in=11b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32) v8 -- 0x563697bffed0 con 0x563697b8d8c0 2023-10-05T19:28:55.179 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.174+0000 7f9f63553700 1 -- 172.21.15.167:0/344537782 <== osd.0 v2:172.21.15.118:6800/27690 44 ==== osd_op_reply(205 gc.28 [call out=24b] v0'0 uv35 ondisk = 0) v8 ==== 149+0+24 (crc 0 0 0) 0x7f9f4c03f020 con 0x563697b8d8c0 2023-10-05T19:28:55.179 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.174+0000 7f9f6621fe40 1 -- 172.21.15.167:0/344537782 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:206 6.1d 6:b88e3271:gc::gc.28:head [call rgw.gc_list in=25b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32) v8 -- 0x563697bffed0 con 0x563697b8d8c0 2023-10-05T19:28:55.180 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.174+0000 7f9f63553700 1 -- 172.21.15.167:0/344537782 <== osd.0 v2:172.21.15.118:6800/27690 45 ==== osd_op_reply(206 gc.28 [call out=15b] v0'0 uv35 ondisk = 0) v8 ==== 149+0+15 (crc 0 0 0) 0x7f9f4c03f020 con 0x563697b8d8c0 2023-10-05T19:28:55.180 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.174+0000 7f9f6621fe40 20 garbage collection: RGWGC::process cls_rgw_gc_list returned NO non expired entries, so setting cache entry to TRUE 2023-10-05T19:28:55.181 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.174+0000 7f9f6621fe40 1 -- 172.21.15.167:0/344537782 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:207 6.1d 6:b88e3271:gc::gc.28:head [call rgw_gc.rgw_gc_queue_list_entries in=46b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32) v8 -- 0x563697bffed0 con 0x563697b8d8c0 2023-10-05T19:28:55.181 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.174+0000 7f9f63553700 1 -- 172.21.15.167:0/344537782 <== osd.0 v2:172.21.15.118:6800/27690 46 ==== osd_op_reply(207 gc.28 [call out=15b] v0'0 uv35 ondisk = 0) v8 ==== 149+0+15 (crc 0 0 0) 0x7f9f4c03f020 con 0x563697b8d8c0 2023-10-05T19:28:55.182 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.178+0000 7f9f6621fe40 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=0, truncated=0, next_marker='' 2023-10-05T19:28:55.182 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.178+0000 7f9f6621fe40 1 -- 172.21.15.167:0/344537782 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:208 6.1d 6:b88e3271:gc::gc.28:head [call lock.unlock in=34b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e32) v8 -- 0x563697bffed0 con 0x563697b8d8c0 2023-10-05T19:28:55.184 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.178+0000 7f9f63553700 1 -- 172.21.15.167:0/344537782 <== osd.0 v2:172.21.15.118:6800/27690 47 ==== osd_op_reply(208 gc.28 [call] v32'36 uv36 ondisk = 0) v8 ==== 149+0+0 (crc 0 0 0) 0x7f9f4c03f020 con 0x563697b8d8c0 2023-10-05T19:28:55.185 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.178+0000 7f9f6621fe40 20 garbage collection: RGWGC::process entered with GC index_shard=29, max_secs=3600, expired_only=1 2023-10-05T19:28:55.185 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.178+0000 7f9f6621fe40 1 -- 172.21.15.167:0/344537782 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:209 6.7 6:e47c9322:gc::gc.29:head [call lock.lock in=50b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e32) v8 -- 0x563697bffed0 con 0x563697b95830 2023-10-05T19:28:55.187 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.182+0000 7f9f64555700 1 -- 172.21.15.167:0/344537782 <== osd.3 v2:172.21.15.167:6808/30036 81 ==== osd_op_reply(209 gc.29 [call] v32'48 uv48 ondisk = 0) v8 ==== 149+0+0 (crc 0 0 0) 0x7f9f54050040 con 0x563697b95830 2023-10-05T19:28:55.187 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.182+0000 7f9f6621fe40 1 -- 172.21.15.167:0/344537782 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:210 6.7 6:e47c9322:gc::gc.29:head [call rgw.gc_list in=25b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32) v8 -- 0x563697bffed0 con 0x563697b95830 2023-10-05T19:28:55.188 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.182+0000 7f9f64555700 1 -- 172.21.15.167:0/344537782 <== osd.3 v2:172.21.15.167:6808/30036 82 ==== osd_op_reply(210 gc.29 [call out=15b] v0'0 uv48 ondisk = 0) v8 ==== 149+0+15 (crc 0 0 0) 0x7f9f54050040 con 0x563697b95830 2023-10-05T19:28:55.188 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.182+0000 7f9f6621fe40 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2023-10-05T19:28:55.188 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.182+0000 7f9f6621fe40 1 -- 172.21.15.167:0/344537782 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:211 6.7 6:e47c9322:gc::gc.29:head [call version.read in=11b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32) v8 -- 0x563697bffed0 con 0x563697b95830 2023-10-05T19:28:55.189 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.182+0000 7f9f64555700 1 -- 172.21.15.167:0/344537782 <== osd.3 v2:172.21.15.167:6808/30036 83 ==== osd_op_reply(211 gc.29 [call out=24b] v0'0 uv48 ondisk = 0) v8 ==== 149+0+24 (crc 0 0 0) 0x7f9f54050040 con 0x563697b95830 2023-10-05T19:28:55.189 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.182+0000 7f9f6621fe40 1 -- 172.21.15.167:0/344537782 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:212 6.7 6:e47c9322:gc::gc.29:head [call rgw.gc_list in=25b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32) v8 -- 0x563697bffed0 con 0x563697b95830 2023-10-05T19:28:55.189 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.182+0000 7f9f64555700 1 -- 172.21.15.167:0/344537782 <== osd.3 v2:172.21.15.167:6808/30036 84 ==== osd_op_reply(212 gc.29 [call out=15b] v0'0 uv48 ondisk = 0) v8 ==== 149+0+15 (crc 0 0 0) 0x7f9f54050040 con 0x563697b95830 2023-10-05T19:28:55.189 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.182+0000 7f9f6621fe40 20 garbage collection: RGWGC::process cls_rgw_gc_list returned NO non expired entries, so setting cache entry to TRUE 2023-10-05T19:28:55.189 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.182+0000 7f9f6621fe40 1 -- 172.21.15.167:0/344537782 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:213 6.7 6:e47c9322:gc::gc.29:head [call rgw_gc.rgw_gc_queue_list_entries in=46b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32) v8 -- 0x563697bffed0 con 0x563697b95830 2023-10-05T19:28:55.190 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.186+0000 7f9f64555700 1 -- 172.21.15.167:0/344537782 <== osd.3 v2:172.21.15.167:6808/30036 85 ==== osd_op_reply(213 gc.29 [call out=15b] v0'0 uv48 ondisk = 0) v8 ==== 149+0+15 (crc 0 0 0) 0x7f9f54050040 con 0x563697b95830 2023-10-05T19:28:55.190 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.186+0000 7f9f6621fe40 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=0, truncated=0, next_marker='' 2023-10-05T19:28:55.190 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.186+0000 7f9f6621fe40 1 -- 172.21.15.167:0/344537782 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:214 6.7 6:e47c9322:gc::gc.29:head [call lock.unlock in=34b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e32) v8 -- 0x563697bffed0 con 0x563697b95830 2023-10-05T19:28:55.192 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.186+0000 7f9f64555700 1 -- 172.21.15.167:0/344537782 <== osd.3 v2:172.21.15.167:6808/30036 86 ==== osd_op_reply(214 gc.29 [call] v32'49 uv49 ondisk = 0) v8 ==== 149+0+0 (crc 0 0 0) 0x7f9f54050040 con 0x563697b95830 2023-10-05T19:28:55.192 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.186+0000 7f9f6621fe40 20 garbage collection: RGWGC::process entered with GC index_shard=30, max_secs=3600, expired_only=1 2023-10-05T19:28:55.193 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.186+0000 7f9f6621fe40 1 -- 172.21.15.167:0/344537782 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:215 6.0 6:013d6c33:gc::gc.30:head [call lock.lock in=50b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e32) v8 -- 0x563697bffed0 con 0x563697b8d8c0 2023-10-05T19:28:55.195 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.190+0000 7f9f63553700 1 -- 172.21.15.167:0/344537782 <== osd.0 v2:172.21.15.118:6800/27690 48 ==== osd_op_reply(215 gc.30 [call] v32'41 uv41 ondisk = 0) v8 ==== 149+0+0 (crc 0 0 0) 0x7f9f4c03f020 con 0x563697b8d8c0 2023-10-05T19:28:55.195 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.190+0000 7f9f6621fe40 1 -- 172.21.15.167:0/344537782 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:216 6.0 6:013d6c33:gc::gc.30:head [call rgw.gc_list in=25b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32) v8 -- 0x563697bffed0 con 0x563697b8d8c0 2023-10-05T19:28:55.196 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.190+0000 7f9f63553700 1 -- 172.21.15.167:0/344537782 <== osd.0 v2:172.21.15.118:6800/27690 49 ==== osd_op_reply(216 gc.30 [call out=15b] v0'0 uv41 ondisk = 0) v8 ==== 149+0+15 (crc 0 0 0) 0x7f9f4c03f020 con 0x563697b8d8c0 2023-10-05T19:28:55.196 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.190+0000 7f9f6621fe40 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2023-10-05T19:28:55.196 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.190+0000 7f9f6621fe40 1 -- 172.21.15.167:0/344537782 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:217 6.0 6:013d6c33:gc::gc.30:head [call version.read in=11b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32) v8 -- 0x563697bffed0 con 0x563697b8d8c0 2023-10-05T19:28:55.197 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.190+0000 7f9f63553700 1 -- 172.21.15.167:0/344537782 <== osd.0 v2:172.21.15.118:6800/27690 50 ==== osd_op_reply(217 gc.30 [call out=24b] v0'0 uv41 ondisk = 0) v8 ==== 149+0+24 (crc 0 0 0) 0x7f9f4c03f020 con 0x563697b8d8c0 2023-10-05T19:28:55.197 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.190+0000 7f9f6621fe40 1 -- 172.21.15.167:0/344537782 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:218 6.0 6:013d6c33:gc::gc.30:head [call rgw.gc_list in=25b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32) v8 -- 0x563697bffed0 con 0x563697b8d8c0 2023-10-05T19:28:55.198 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.194+0000 7f9f63553700 1 -- 172.21.15.167:0/344537782 <== osd.0 v2:172.21.15.118:6800/27690 51 ==== osd_op_reply(218 gc.30 [call out=15b] v0'0 uv41 ondisk = 0) v8 ==== 149+0+15 (crc 0 0 0) 0x7f9f4c03f020 con 0x563697b8d8c0 2023-10-05T19:28:55.198 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.194+0000 7f9f6621fe40 20 garbage collection: RGWGC::process cls_rgw_gc_list returned NO non expired entries, so setting cache entry to TRUE 2023-10-05T19:28:55.198 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.194+0000 7f9f6621fe40 1 -- 172.21.15.167:0/344537782 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:219 6.0 6:013d6c33:gc::gc.30:head [call rgw_gc.rgw_gc_queue_list_entries in=46b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32) v8 -- 0x563697bffed0 con 0x563697b8d8c0 2023-10-05T19:28:55.199 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.194+0000 7f9f63553700 1 -- 172.21.15.167:0/344537782 <== osd.0 v2:172.21.15.118:6800/27690 52 ==== osd_op_reply(219 gc.30 [call out=15b] v0'0 uv41 ondisk = 0) v8 ==== 149+0+15 (crc 0 0 0) 0x7f9f4c03f020 con 0x563697b8d8c0 2023-10-05T19:28:55.199 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.194+0000 7f9f6621fe40 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=0, truncated=0, next_marker='' 2023-10-05T19:28:55.199 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.194+0000 7f9f6621fe40 1 -- 172.21.15.167:0/344537782 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:220 6.0 6:013d6c33:gc::gc.30:head [call lock.unlock in=34b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e32) v8 -- 0x563697bffed0 con 0x563697b8d8c0 2023-10-05T19:28:55.202 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.198+0000 7f9f63553700 1 -- 172.21.15.167:0/344537782 <== osd.0 v2:172.21.15.118:6800/27690 53 ==== osd_op_reply(220 gc.30 [call] v32'42 uv42 ondisk = 0) v8 ==== 149+0+0 (crc 0 0 0) 0x7f9f4c03f020 con 0x563697b8d8c0 2023-10-05T19:28:55.202 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.198+0000 7f9f6621fe40 20 garbage collection: RGWGC::process entered with GC index_shard=31, max_secs=3600, expired_only=1 2023-10-05T19:28:55.202 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.198+0000 7f9f6621fe40 1 -- 172.21.15.167:0/344537782 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:221 6.f 6:f24cb4a8:gc::gc.31:head [call lock.lock in=50b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e32) v8 -- 0x563697bffed0 con 0x563697b9d5a0 2023-10-05T19:28:55.205 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.198+0000 7f9f63d54700 1 -- 172.21.15.167:0/344537782 <== osd.2 v2:172.21.15.167:6816/30037 44 ==== osd_op_reply(221 gc.31 [call] v32'26 uv26 ondisk = 0) v8 ==== 149+0+0 (crc 0 0 0) 0x7f9f5c024020 con 0x563697b9d5a0 2023-10-05T19:28:55.205 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.198+0000 7f9f6621fe40 1 -- 172.21.15.167:0/344537782 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:222 6.f 6:f24cb4a8:gc::gc.31:head [call rgw.gc_list in=25b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32) v8 -- 0x563697c0bf50 con 0x563697b9d5a0 2023-10-05T19:28:55.206 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.202+0000 7f9f63d54700 1 -- 172.21.15.167:0/344537782 <== osd.2 v2:172.21.15.167:6816/30037 45 ==== osd_op_reply(222 gc.31 [call out=15b] v0'0 uv26 ondisk = 0) v8 ==== 149+0+15 (crc 0 0 0) 0x7f9f5c024020 con 0x563697b9d5a0 2023-10-05T19:28:55.206 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.202+0000 7f9f6621fe40 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2023-10-05T19:28:55.206 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.202+0000 7f9f6621fe40 1 -- 172.21.15.167:0/344537782 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:223 6.f 6:f24cb4a8:gc::gc.31:head [call version.read in=11b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32) v8 -- 0x563697c0bf50 con 0x563697b9d5a0 2023-10-05T19:28:55.207 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.202+0000 7f9f63d54700 1 -- 172.21.15.167:0/344537782 <== osd.2 v2:172.21.15.167:6816/30037 46 ==== osd_op_reply(223 gc.31 [call out=24b] v0'0 uv26 ondisk = 0) v8 ==== 149+0+24 (crc 0 0 0) 0x7f9f5c024020 con 0x563697b9d5a0 2023-10-05T19:28:55.207 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.202+0000 7f9f6621fe40 1 -- 172.21.15.167:0/344537782 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:224 6.f 6:f24cb4a8:gc::gc.31:head [call rgw.gc_list in=25b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32) v8 -- 0x563697c0bf50 con 0x563697b9d5a0 2023-10-05T19:28:55.207 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.202+0000 7f9f63d54700 1 -- 172.21.15.167:0/344537782 <== osd.2 v2:172.21.15.167:6816/30037 47 ==== osd_op_reply(224 gc.31 [call out=15b] v0'0 uv26 ondisk = 0) v8 ==== 149+0+15 (crc 0 0 0) 0x7f9f5c024020 con 0x563697b9d5a0 2023-10-05T19:28:55.207 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.202+0000 7f9f6621fe40 20 garbage collection: RGWGC::process cls_rgw_gc_list returned NO non expired entries, so setting cache entry to TRUE 2023-10-05T19:28:55.208 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.202+0000 7f9f6621fe40 1 -- 172.21.15.167:0/344537782 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:225 6.f 6:f24cb4a8:gc::gc.31:head [call rgw_gc.rgw_gc_queue_list_entries in=46b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32) v8 -- 0x563697c0bf50 con 0x563697b9d5a0 2023-10-05T19:28:55.208 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.202+0000 7f9f63d54700 1 -- 172.21.15.167:0/344537782 <== osd.2 v2:172.21.15.167:6816/30037 48 ==== osd_op_reply(225 gc.31 [call out=15b] v0'0 uv26 ondisk = 0) v8 ==== 149+0+15 (crc 0 0 0) 0x7f9f5c024020 con 0x563697b9d5a0 2023-10-05T19:28:55.208 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.202+0000 7f9f6621fe40 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=0, truncated=0, next_marker='' 2023-10-05T19:28:55.208 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.202+0000 7f9f6621fe40 1 -- 172.21.15.167:0/344537782 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:226 6.f 6:f24cb4a8:gc::gc.31:head [call lock.unlock in=34b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e32) v8 -- 0x563697c0bf50 con 0x563697b9d5a0 2023-10-05T19:28:55.211 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.206+0000 7f9f63d54700 1 -- 172.21.15.167:0/344537782 <== osd.2 v2:172.21.15.167:6816/30037 49 ==== osd_op_reply(226 gc.31 [call] v32'27 uv27 ondisk = 0) v8 ==== 149+0+0 (crc 0 0 0) 0x7f9f5c024020 con 0x563697b9d5a0 2023-10-05T19:28:55.211 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.206+0000 7f9f6621fe40 20 garbage collection: RGWGC::process entered with GC index_shard=0, max_secs=3600, expired_only=1 2023-10-05T19:28:55.211 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.206+0000 7f9f6621fe40 1 -- 172.21.15.167:0/344537782 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:227 6.19 6:9a6dd8c0:gc::gc.0:head [call lock.lock in=50b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e32) v8 -- 0x563697c0bf50 con 0x563697b91ce0 2023-10-05T19:28:55.214 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.210+0000 7f9f63d54700 1 -- 172.21.15.167:0/344537782 <== osd.1 v2:172.21.15.167:6800/30035 39 ==== osd_op_reply(227 gc.0 [call] v32'32 uv32 ondisk = 0) v8 ==== 148+0+0 (crc 0 0 0) 0x7f9f5c024020 con 0x563697b91ce0 2023-10-05T19:28:55.214 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.210+0000 7f9f6621fe40 1 -- 172.21.15.167:0/344537782 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:228 6.19 6:9a6dd8c0:gc::gc.0:head [call rgw.gc_list in=25b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32) v8 -- 0x563697c0bf50 con 0x563697b91ce0 2023-10-05T19:28:55.215 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.210+0000 7f9f63d54700 1 -- 172.21.15.167:0/344537782 <== osd.1 v2:172.21.15.167:6800/30035 40 ==== osd_op_reply(228 gc.0 [call out=15b] v0'0 uv32 ondisk = 0) v8 ==== 148+0+15 (crc 0 0 0) 0x7f9f5c024020 con 0x563697b91ce0 2023-10-05T19:28:55.215 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.210+0000 7f9f6621fe40 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2023-10-05T19:28:55.215 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.210+0000 7f9f6621fe40 1 -- 172.21.15.167:0/344537782 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:229 6.19 6:9a6dd8c0:gc::gc.0:head [call version.read in=11b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32) v8 -- 0x563697c0bf50 con 0x563697b91ce0 2023-10-05T19:28:55.216 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.210+0000 7f9f63d54700 1 -- 172.21.15.167:0/344537782 <== osd.1 v2:172.21.15.167:6800/30035 41 ==== osd_op_reply(229 gc.0 [call out=24b] v0'0 uv32 ondisk = 0) v8 ==== 148+0+24 (crc 0 0 0) 0x7f9f5c024020 con 0x563697b91ce0 2023-10-05T19:28:55.216 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.210+0000 7f9f6621fe40 1 -- 172.21.15.167:0/344537782 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:230 6.19 6:9a6dd8c0:gc::gc.0:head [call rgw.gc_list in=25b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32) v8 -- 0x563697c0bf50 con 0x563697b91ce0 2023-10-05T19:28:55.216 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.210+0000 7f9f63d54700 1 -- 172.21.15.167:0/344537782 <== osd.1 v2:172.21.15.167:6800/30035 42 ==== osd_op_reply(230 gc.0 [call out=15b] v0'0 uv32 ondisk = 0) v8 ==== 148+0+15 (crc 0 0 0) 0x7f9f5c024020 con 0x563697b91ce0 2023-10-05T19:28:55.217 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.210+0000 7f9f6621fe40 20 garbage collection: RGWGC::process cls_rgw_gc_list returned NO non expired entries, so setting cache entry to TRUE 2023-10-05T19:28:55.217 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.210+0000 7f9f6621fe40 1 -- 172.21.15.167:0/344537782 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:231 6.19 6:9a6dd8c0:gc::gc.0:head [call rgw_gc.rgw_gc_queue_list_entries in=46b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32) v8 -- 0x563697c0bf50 con 0x563697b91ce0 2023-10-05T19:28:55.217 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.210+0000 7f9f63d54700 1 -- 172.21.15.167:0/344537782 <== osd.1 v2:172.21.15.167:6800/30035 43 ==== osd_op_reply(231 gc.0 [call out=15b] v0'0 uv32 ondisk = 0) v8 ==== 148+0+15 (crc 0 0 0) 0x7f9f5c024020 con 0x563697b91ce0 2023-10-05T19:28:55.217 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.214+0000 7f9f6621fe40 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=0, truncated=0, next_marker='' 2023-10-05T19:28:55.218 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.214+0000 7f9f6621fe40 1 -- 172.21.15.167:0/344537782 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:232 6.19 6:9a6dd8c0:gc::gc.0:head [call lock.unlock in=34b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e32) v8 -- 0x563697c0bf50 con 0x563697b91ce0 2023-10-05T19:28:55.220 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.214+0000 7f9f63d54700 1 -- 172.21.15.167:0/344537782 <== osd.1 v2:172.21.15.167:6800/30035 44 ==== osd_op_reply(232 gc.0 [call] v32'33 uv33 ondisk = 0) v8 ==== 148+0+0 (crc 0 0 0) 0x7f9f5c024020 con 0x563697b91ce0 2023-10-05T19:28:55.221 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.214+0000 7f9f6621fe40 1 -- 172.21.15.167:0/3004404278 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:102 7.1f 7:f95f44c2:::notify.0:head [watch unwatch cookie 94792473546032] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e32) v8 -- 0x563697bff670 con 0x7f9f5c010c30 2023-10-05T19:28:55.224 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.218+0000 7f9f63553700 1 -- 172.21.15.167:0/3004404278 <== osd.2 v2:172.21.15.167:6816/30037 17 ==== osd_op_reply(102 notify.0 [watch unwatch cookie 94792473546032] v32'194 uv192 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f9f4c03f020 con 0x7f9f5c010c30 2023-10-05T19:28:55.224 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.218+0000 7f9f6621fe40 20 remove_watcher() i=0 2023-10-05T19:28:55.224 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.218+0000 7f9f6621fe40 2 removed watcher, disabling cache 2023-10-05T19:28:55.224 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.218+0000 7f9f6621fe40 1 -- 172.21.15.167:0/3004404278 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:103 7.0 7:05bf5b68:::notify.1:head [watch unwatch cookie 94792473552544] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e32) v8 -- 0x563697b68d30 con 0x563697b335d0 2023-10-05T19:28:55.227 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.222+0000 7f9f64555700 1 -- 172.21.15.167:0/3004404278 <== osd.1 v2:172.21.15.167:6800/30035 35 ==== osd_op_reply(103 notify.1 [watch unwatch cookie 94792473552544] v32'194 uv192 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f9f54050040 con 0x563697b335d0 2023-10-05T19:28:55.227 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.222+0000 7f9f6621fe40 20 remove_watcher() i=1 2023-10-05T19:28:55.227 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.222+0000 7f9f6621fe40 1 -- 172.21.15.167:0/3004404278 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:104 7.15 7:a93a5511:::notify.2:head [watch unwatch cookie 94792473556096] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e32) v8 -- 0x563697b68d30 con 0x563697b64440 2023-10-05T19:28:55.230 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.226+0000 7f9f63d54700 1 -- 172.21.15.167:0/3004404278 <== osd.3 v2:172.21.15.167:6808/30036 24 ==== osd_op_reply(104 notify.2 [watch unwatch cookie 94792473556096] v32'194 uv192 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f9f5c024020 con 0x563697b64440 2023-10-05T19:28:55.230 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.226+0000 7f9f6621fe40 20 remove_watcher() i=2 2023-10-05T19:28:55.230 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.226+0000 7f9f6621fe40 1 -- 172.21.15.167:0/3004404278 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:105 7.e 7:7759931f:::notify.3:head [watch unwatch cookie 94792473554832] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e32) v8 -- 0x563697b68d30 con 0x7f9f5c010c30 2023-10-05T19:28:55.232 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.226+0000 7f9f63553700 1 -- 172.21.15.167:0/3004404278 <== osd.2 v2:172.21.15.167:6816/30037 18 ==== osd_op_reply(105 notify.3 [watch unwatch cookie 94792473554832] v32'194 uv192 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f9f4c03f020 con 0x7f9f5c010c30 2023-10-05T19:28:55.232 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.226+0000 7f9f6621fe40 20 remove_watcher() i=3 2023-10-05T19:28:55.233 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.226+0000 7f9f6621fe40 1 -- 172.21.15.167:0/3004404278 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:106 7.d 7:b4812045:::notify.4:head [watch unwatch cookie 94792473562608] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e32) v8 -- 0x563697b6af90 con 0x563697b335d0 2023-10-05T19:28:55.235 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.230+0000 7f9f64555700 1 -- 172.21.15.167:0/3004404278 <== osd.1 v2:172.21.15.167:6800/30035 36 ==== osd_op_reply(106 notify.4 [watch unwatch cookie 94792473562608] v32'194 uv192 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f9f54050040 con 0x563697b335d0 2023-10-05T19:28:55.235 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.230+0000 7f9f6621fe40 20 remove_watcher() i=4 2023-10-05T19:28:55.235 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.230+0000 7f9f6621fe40 1 -- 172.21.15.167:0/3004404278 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:107 7.3 7:c609908c:::notify.5:head [watch unwatch cookie 94792473566832] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e32) v8 -- 0x563697b68d30 con 0x7f9f4c007030 2023-10-05T19:28:55.238 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.234+0000 7f9f64555700 1 -- 172.21.15.167:0/3004404278 <== osd.0 v2:172.21.15.118:6800/27690 29 ==== osd_op_reply(107 notify.5 [watch unwatch cookie 94792473566832] v32'194 uv192 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f9f54050040 con 0x7f9f4c007030 2023-10-05T19:28:55.238 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.234+0000 7f9f6621fe40 20 remove_watcher() i=5 2023-10-05T19:28:55.238 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.234+0000 7f9f6621fe40 1 -- 172.21.15.167:0/3004404278 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:108 7.14 7:2b04a3e9:::notify.6:head [watch unwatch cookie 94792473558304] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e32) v8 -- 0x563697b6cdf0 con 0x7f9f4c007030 2023-10-05T19:28:55.241 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.238+0000 7f9f64555700 1 -- 172.21.15.167:0/3004404278 <== osd.0 v2:172.21.15.118:6800/27690 30 ==== osd_op_reply(108 notify.6 [watch unwatch cookie 94792473558304] v32'194 uv192 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f9f54050040 con 0x7f9f4c007030 2023-10-05T19:28:55.242 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.238+0000 7f9f6621fe40 20 remove_watcher() i=6 2023-10-05T19:28:55.242 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.238+0000 7f9f6621fe40 1 -- 172.21.15.167:0/3004404278 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:109 7.9 7:93e5b521:::notify.7:head [watch unwatch cookie 94792473570512] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e32) v8 -- 0x563697b6de70 con 0x7f9f4c007030 2023-10-05T19:28:55.245 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.238+0000 7f9f64555700 1 -- 172.21.15.167:0/3004404278 <== osd.0 v2:172.21.15.118:6800/27690 31 ==== osd_op_reply(109 notify.7 [watch unwatch cookie 94792473570512] v32'194 uv192 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f9f54050040 con 0x7f9f4c007030 2023-10-05T19:28:55.245 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.238+0000 7f9f6621fe40 20 remove_watcher() i=7 2023-10-05T19:28:55.247 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.242+0000 7f9f357e2700 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2023-10-05T19:28:55.247 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.242+0000 7f9f6621fe40 1 -- 172.21.15.167:0/3004404278 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:110 6.4 6:22d26bf9:::data_loggenerations_metadata:head [watch unwatch cookie 94792473572400] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e32) v8 -- 0x563697bc1cd0 con 0x563697b335d0 2023-10-05T19:28:55.250 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.246+0000 7f9f64555700 1 -- 172.21.15.167:0/3004404278 <== osd.1 v2:172.21.15.167:6800/30035 37 ==== osd_op_reply(110 data_loggenerations_metadata [watch unwatch cookie 94792473572400] v32'190 uv1 ondisk = 0) v8 ==== 172+0+0 (crc 0 0 0) 0x7f9f54050040 con 0x563697b335d0 2023-10-05T19:28:55.250 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.246+0000 7f9f6621fe40 1 -- 172.21.15.167:0/3004404278 >> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] conn(0x7f9f4c007030 msgr2=0x7f9f4c01d430 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-05T19:28:55.250 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.246+0000 7f9f6621fe40 1 --2- 172.21.15.167:0/3004404278 >> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] conn(0x7f9f4c007030 0x7f9f4c01d430 crc :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-05T19:28:55.250 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.246+0000 7f9f6621fe40 1 -- 172.21.15.167:0/3004404278 >> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] conn(0x563697b335d0 msgr2=0x563697b607b0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-05T19:28:55.250 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.246+0000 7f9f6621fe40 1 --2- 172.21.15.167:0/3004404278 >> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] conn(0x563697b335d0 0x563697b607b0 crc :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-05T19:28:55.251 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.246+0000 7f9f6621fe40 1 -- 172.21.15.167:0/3004404278 >> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] conn(0x7f9f5c010c30 msgr2=0x7f9f5c0439b0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-05T19:28:55.251 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.246+0000 7f9f6621fe40 1 --2- 172.21.15.167:0/3004404278 >> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] conn(0x7f9f5c010c30 0x7f9f5c0439b0 crc :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-05T19:28:55.251 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.246+0000 7f9f6621fe40 1 -- 172.21.15.167:0/3004404278 >> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] conn(0x563697b64440 msgr2=0x563697b66850 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-05T19:28:55.251 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.246+0000 7f9f6621fe40 1 --2- 172.21.15.167:0/3004404278 >> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] conn(0x563697b64440 0x563697b66850 crc :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-05T19:28:55.253 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.246+0000 7f9f6621fe40 1 -- 172.21.15.167:0/3004404278 >> [v2:172.21.15.167:6824/29889,v1:172.21.15.167:6825/29889] conn(0x7f9f44030c00 msgr2=0x7f9f440330b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-05T19:28:55.253 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.246+0000 7f9f6621fe40 1 --2- 172.21.15.167:0/3004404278 >> [v2:172.21.15.167:6824/29889,v1:172.21.15.167:6825/29889] conn(0x7f9f44030c00 0x7f9f440330b0 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7f9f4c005cc0 tx=0x7f9f4c014000 comp rx=0 tx=0).stop 2023-10-05T19:28:55.253 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.246+0000 7f9f6621fe40 1 -- 172.21.15.167:0/3004404278 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x563697b2eff0 msgr2=0x563697b54cf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-05T19:28:55.253 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.246+0000 7f9f6621fe40 1 --2- 172.21.15.167:0/3004404278 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x563697b2eff0 0x563697b54cf0 secure :-1 s=READY pgs=521 cs=0 l=1 rev1=1 crypto rx=0x7f9f5c00f000 tx=0x7f9f5c010000 comp rx=0 tx=0).stop 2023-10-05T19:28:55.253 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.246+0000 7f9f64555700 1 -- 172.21.15.167:0/3004404278 reap_dead start 2023-10-05T19:28:55.253 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.246+0000 7f9f6621fe40 1 -- 172.21.15.167:0/3004404278 shutdown_connections 2023-10-05T19:28:55.253 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.246+0000 7f9f6621fe40 1 --2- 172.21.15.167:0/3004404278 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x563697b2eff0 0x563697b54cf0 unknown :-1 s=CLOSED pgs=521 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-05T19:28:55.253 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.246+0000 7f9f6621fe40 1 -- 172.21.15.167:0/3004404278 >> 172.21.15.167:0/3004404278 conn(0x563697aa2840 msgr2=0x563697b48550 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-05T19:28:55.253 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.246+0000 7f9f6621fe40 1 -- 172.21.15.167:0/3004404278 shutdown_connections 2023-10-05T19:28:55.253 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.246+0000 7f9f6621fe40 1 -- 172.21.15.167:0/3004404278 wait complete. 2023-10-05T19:28:55.254 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.246+0000 7f9f6621fe40 1 -- 172.21.15.167:0/344537782 >> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] conn(0x563697b8d8c0 msgr2=0x563697b97e00 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-05T19:28:55.254 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.246+0000 7f9f6621fe40 1 --2- 172.21.15.167:0/344537782 >> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] conn(0x563697b8d8c0 0x563697b97e00 crc :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-05T19:28:55.254 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.246+0000 7f9f6621fe40 1 -- 172.21.15.167:0/344537782 >> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] conn(0x563697b91ce0 msgr2=0x563697b940f0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-05T19:28:55.254 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.246+0000 7f9f6621fe40 1 --2- 172.21.15.167:0/344537782 >> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] conn(0x563697b91ce0 0x563697b940f0 crc :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-05T19:28:55.254 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.246+0000 7f9f6621fe40 1 -- 172.21.15.167:0/344537782 >> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] conn(0x563697b9d5a0 msgr2=0x563697b9f980 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-05T19:28:55.254 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.246+0000 7f9f6621fe40 1 --2- 172.21.15.167:0/344537782 >> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] conn(0x563697b9d5a0 0x563697b9f980 crc :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-05T19:28:55.254 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.246+0000 7f9f6621fe40 1 -- 172.21.15.167:0/344537782 >> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] conn(0x563697b95830 msgr2=0x563697b9b9b0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-05T19:28:55.254 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.246+0000 7f9f6621fe40 1 --2- 172.21.15.167:0/344537782 >> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] conn(0x563697b95830 0x563697b9b9b0 crc :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-05T19:28:55.254 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.246+0000 7f9f6621fe40 1 -- 172.21.15.167:0/344537782 >> [v2:172.21.15.167:6824/29889,v1:172.21.15.167:6825/29889] conn(0x7f9ea4030c00 msgr2=0x7f9ea40330b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-05T19:28:55.254 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.246+0000 7f9f6621fe40 1 --2- 172.21.15.167:0/344537782 >> [v2:172.21.15.167:6824/29889,v1:172.21.15.167:6825/29889] conn(0x7f9ea4030c00 0x7f9ea40330b0 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x563697b6fde0 tx=0x7f9f54000f90 comp rx=0 tx=0).stop 2023-10-05T19:28:55.254 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.246+0000 7f9f63d54700 1 -- 172.21.15.167:0/344537782 reap_dead start 2023-10-05T19:28:55.255 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.246+0000 7f9f6621fe40 1 -- 172.21.15.167:0/344537782 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x563697b7d160 msgr2=0x563697b87090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-05T19:28:55.255 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.246+0000 7f9f6621fe40 1 --2- 172.21.15.167:0/344537782 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x563697b7d160 0x563697b87090 secure :-1 s=READY pgs=523 cs=0 l=1 rev1=1 crypto rx=0x7f9f4c01f940 tx=0x7f9f4c038000 comp rx=0 tx=0).stop 2023-10-05T19:28:55.255 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.246+0000 7f9f6621fe40 1 -- 172.21.15.167:0/344537782 shutdown_connections 2023-10-05T19:28:55.255 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.246+0000 7f9f6621fe40 1 --2- 172.21.15.167:0/344537782 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x563697b7d160 0x563697b87090 unknown :-1 s=CLOSED pgs=523 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-05T19:28:55.255 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.246+0000 7f9f6621fe40 1 -- 172.21.15.167:0/344537782 >> 172.21.15.167:0/344537782 conn(0x563697b78520 msgr2=0x563697b765c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-05T19:28:55.255 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.246+0000 7f9f6621fe40 1 -- 172.21.15.167:0/344537782 shutdown_connections 2023-10-05T19:28:55.255 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.246+0000 7f9f6621fe40 1 -- 172.21.15.167:0/344537782 wait complete. 2023-10-05T19:28:55.259 INFO:tasks.util.rgw:rgwadmin: client.0 : ['gc', 'list', '--include-all'] 2023-10-05T19:28:55.259 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'ceph', 'gc', 'list', '--include-all'] 2023-10-05T19:28:55.259 DEBUG:teuthology.orchestra.run.smithi167:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster ceph gc list --include-all 2023-10-05T19:28:55.295 INFO:teuthology.orchestra.run.smithi167.stderr:ignoring --setuser ceph since I am not root 2023-10-05T19:28:55.295 INFO:teuthology.orchestra.run.smithi167.stderr:ignoring --setgroup ceph since I am not root 2023-10-05T19:28:55.296 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.290+0000 7fc9e0230e40 1 Processor -- start 2023-10-05T19:28:55.296 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.290+0000 7fc9e0230e40 1 -- start start 2023-10-05T19:28:55.296 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.290+0000 7fc9e0230e40 1 --2- >> v2:172.21.15.118:3300/0 conn(0x5606235e66f0 0x5606235e8ad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-05T19:28:55.297 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.290+0000 7fc9e0230e40 1 -- --> v1:172.21.15.118:6789/0 -- auth(proto 0 26 bytes epoch 0) v1 -- 0x56062352cbd0 con 0x5606235543d0 2023-10-05T19:28:55.297 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.290+0000 7fc9e0230e40 1 -- --> v2:172.21.15.118:3300/0 -- mon_getmap magic: 0 v1 -- 0x5606234e9ee0 con 0x5606235e66f0 2023-10-05T19:28:55.297 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.290+0000 7fc9dd564700 1 --2- >> v2:172.21.15.118:3300/0 conn(0x5606235e66f0 0x5606235e8ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-05T19:28:55.298 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.290+0000 7fc9ddd65700 1 --1- >> v1:172.21.15.118:6789/0 conn(0x5606235543d0 0x5606235e4fd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.118:6789/0 says I am v1:172.21.15.167:48676/0 (socket says 172.21.15.167:48676) 2023-10-05T19:28:55.298 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.290+0000 7fc9ddd65700 1 -- 172.21.15.167:0/519419647 learned_addr learned my addr 172.21.15.167:0/519419647 (peer_addr_for_me v1:172.21.15.167:0/0) 2023-10-05T19:28:55.298 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.294+0000 7fc9dd564700 1 -- 172.21.15.167:0/519419647 >> v1:172.21.15.118:6789/0 conn(0x5606235543d0 legacy=0x5606235e4fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-05T19:28:55.298 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.294+0000 7fc9dd564700 1 -- 172.21.15.167:0/519419647 --> v2:172.21.15.118:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5606235547a0 con 0x5606235e66f0 2023-10-05T19:28:55.299 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.294+0000 7fc9dd564700 1 --2- 172.21.15.167:0/519419647 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5606235e66f0 0x5606235e8ad0 secure :-1 s=READY pgs=525 cs=0 l=1 rev1=1 crypto rx=0x7fc9d400a5d0 tx=0x7fc9d4005a50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=30d8b9ba85b0d4d0 server_cookie=0 in_seq=0 out_seq=0 2023-10-05T19:28:55.299 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.294+0000 7fc9dcd63700 1 -- 172.21.15.167:0/519419647 <== mon.0 v2:172.21.15.118:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fc9d40086a0 con 0x5606235e66f0 2023-10-05T19:28:55.299 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.294+0000 7fc9dcd63700 1 -- 172.21.15.167:0/519419647 <== mon.0 v2:172.21.15.118:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc9d4008800 con 0x5606235e66f0 2023-10-05T19:28:55.299 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.294+0000 7fc9dcd63700 1 -- 172.21.15.167:0/519419647 <== mon.0 v2:172.21.15.118:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fc9d4008ad0 con 0x5606235e66f0 2023-10-05T19:28:55.300 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.294+0000 7fc9e0230e40 1 -- 172.21.15.167:0/519419647 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5606235e66f0 msgr2=0x5606235e8ad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-05T19:28:55.300 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.294+0000 7fc9e0230e40 1 --2- 172.21.15.167:0/519419647 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5606235e66f0 0x5606235e8ad0 secure :-1 s=READY pgs=525 cs=0 l=1 rev1=1 crypto rx=0x7fc9d400a5d0 tx=0x7fc9d4005a50 comp rx=0 tx=0).stop 2023-10-05T19:28:55.300 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.294+0000 7fc9e0230e40 1 -- 172.21.15.167:0/519419647 shutdown_connections 2023-10-05T19:28:55.300 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.294+0000 7fc9e0230e40 1 --2- 172.21.15.167:0/519419647 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5606235e66f0 0x5606235e8ad0 unknown :-1 s=CLOSED pgs=525 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-05T19:28:55.300 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.294+0000 7fc9e0230e40 1 -- 172.21.15.167:0/519419647 >> 172.21.15.167:0/519419647 conn(0x5606235da720 msgr2=0x5606235dcb30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-05T19:28:55.300 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.294+0000 7fc9e0230e40 1 -- 172.21.15.167:0/519419647 shutdown_connections 2023-10-05T19:28:55.300 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.294+0000 7fc9e0230e40 1 -- 172.21.15.167:0/519419647 wait complete. 2023-10-05T19:28:55.301 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.294+0000 7fc9e0230e40 1 Processor -- start 2023-10-05T19:28:55.301 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.294+0000 7fc9e0230e40 1 -- start start 2023-10-05T19:28:55.302 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.298+0000 7fc9e0230e40 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5606235543d0 0x560623601140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-05T19:28:55.302 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.298+0000 7fc9e0230e40 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x5606234ade30 con 0x5606235543d0 2023-10-05T19:28:55.302 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.298+0000 7fc9ddd65700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5606235543d0 0x560623601140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-05T19:28:55.302 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.298+0000 7fc9ddd65700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5606235543d0 0x560623601140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.118:3300/0 says I am v2:172.21.15.167:53218/0 (socket says 172.21.15.167:53218) 2023-10-05T19:28:55.302 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.298+0000 7fc9ddd65700 1 -- 172.21.15.167:0/2560301927 learned_addr learned my addr 172.21.15.167:0/2560301927 (peer_addr_for_me v2:172.21.15.167:0/0) 2023-10-05T19:28:55.303 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.298+0000 7fc9ddd65700 1 -- 172.21.15.167:0/2560301927 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc9d4005cf0 con 0x5606235543d0 2023-10-05T19:28:55.303 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.298+0000 7fc9ddd65700 1 --2- 172.21.15.167:0/2560301927 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5606235543d0 0x560623601140 secure :-1 s=READY pgs=526 cs=0 l=1 rev1=1 crypto rx=0x7fc9d8006fa0 tx=0x7fc9d8006b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=118807efad622d5d server_cookie=0 in_seq=0 out_seq=0 2023-10-05T19:28:55.303 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.298+0000 7fc9cf7fe700 1 -- 172.21.15.167:0/2560301927 <== mon.0 v2:172.21.15.118:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fc9d8006110 con 0x5606235543d0 2023-10-05T19:28:55.303 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.298+0000 7fc9cf7fe700 1 -- 172.21.15.167:0/2560301927 <== mon.0 v2:172.21.15.118:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc9d8006270 con 0x5606235543d0 2023-10-05T19:28:55.303 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.298+0000 7fc9cf7fe700 1 -- 172.21.15.167:0/2560301927 <== mon.0 v2:172.21.15.118:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fc9d8004520 con 0x5606235543d0 2023-10-05T19:28:55.304 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.298+0000 7fc9e0230e40 1 -- 172.21.15.167:0/2560301927 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5606235543d0 msgr2=0x560623601140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-05T19:28:55.304 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.298+0000 7fc9e0230e40 1 --2- 172.21.15.167:0/2560301927 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5606235543d0 0x560623601140 secure :-1 s=READY pgs=526 cs=0 l=1 rev1=1 crypto rx=0x7fc9d8006fa0 tx=0x7fc9d8006b50 comp rx=0 tx=0).stop 2023-10-05T19:28:55.304 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.298+0000 7fc9e0230e40 1 -- 172.21.15.167:0/2560301927 shutdown_connections 2023-10-05T19:28:55.304 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.298+0000 7fc9e0230e40 1 --2- 172.21.15.167:0/2560301927 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5606235543d0 0x560623601140 unknown :-1 s=CLOSED pgs=526 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-05T19:28:55.305 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.298+0000 7fc9e0230e40 1 -- 172.21.15.167:0/2560301927 >> 172.21.15.167:0/2560301927 conn(0x5606235da720 msgr2=0x5606235fcd00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-05T19:28:55.305 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.298+0000 7fc9e0230e40 1 -- 172.21.15.167:0/2560301927 shutdown_connections 2023-10-05T19:28:55.305 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.298+0000 7fc9e0230e40 1 -- 172.21.15.167:0/2560301927 wait complete. 2023-10-05T19:28:55.305 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.298+0000 7fc9e0230e40 1 Processor -- start 2023-10-05T19:28:55.306 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.298+0000 7fc9e0230e40 1 -- start start 2023-10-05T19:28:55.306 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.302+0000 7fc9e0230e40 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5606235543d0 0x560623607f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-05T19:28:55.306 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.302+0000 7fc9e0230e40 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x560623480ed0 con 0x5606235543d0 2023-10-05T19:28:55.306 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.302+0000 7fc9ddd65700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5606235543d0 0x560623607f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-05T19:28:55.306 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.302+0000 7fc9ddd65700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5606235543d0 0x560623607f10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.118:3300/0 says I am v2:172.21.15.167:53232/0 (socket says 172.21.15.167:53232) 2023-10-05T19:28:55.306 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.302+0000 7fc9ddd65700 1 -- 172.21.15.167:0/2053399812 learned_addr learned my addr 172.21.15.167:0/2053399812 (peer_addr_for_me v2:172.21.15.167:0/0) 2023-10-05T19:28:55.307 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.302+0000 7fc9ddd65700 1 -- 172.21.15.167:0/2053399812 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x56062352cbd0 con 0x5606235543d0 2023-10-05T19:28:55.307 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.302+0000 7fc9ddd65700 1 --2- 172.21.15.167:0/2053399812 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5606235543d0 0x560623607f10 secure :-1 s=READY pgs=527 cs=0 l=1 rev1=1 crypto rx=0x7fc9d8000f80 tx=0x7fc9d8014000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-05T19:28:55.308 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.302+0000 7fc9cdffb700 1 -- 172.21.15.167:0/2053399812 <== mon.0 v2:172.21.15.118:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fc9d8013420 con 0x5606235543d0 2023-10-05T19:28:55.308 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.302+0000 7fc9cdffb700 1 -- 172.21.15.167:0/2053399812 <== mon.0 v2:172.21.15.118:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc9d8013580 con 0x5606235543d0 2023-10-05T19:28:55.308 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.302+0000 7fc9e0230e40 1 -- 172.21.15.167:0/2053399812 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x560623608450 con 0x5606235543d0 2023-10-05T19:28:55.308 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.302+0000 7fc9cdffb700 1 -- 172.21.15.167:0/2053399812 <== mon.0 v2:172.21.15.118:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fc9d8013850 con 0x5606235543d0 2023-10-05T19:28:55.308 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.302+0000 7fc9e0230e40 1 -- 172.21.15.167:0/2053399812 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x560623608890 con 0x5606235543d0 2023-10-05T19:28:55.309 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.302+0000 7fc9e0230e40 20 rados->read ofs=0 len=0 2023-10-05T19:28:55.309 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.306+0000 7fc9cdffb700 1 -- 172.21.15.167:0/2053399812 <== mon.0 v2:172.21.15.118:3300/0 4 ==== mgrmap(e 5) v1 ==== 40329+0+0 (secure 0 0 0) 0x7fc9d801a070 con 0x5606235543d0 2023-10-05T19:28:55.310 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.306+0000 7fc9cdffb700 1 --2- 172.21.15.167:0/2053399812 >> [v2:172.21.15.167:6824/29889,v1:172.21.15.167:6825/29889] conn(0x7fc9c0030c00 0x7fc9c00330b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-05T19:28:55.310 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.306+0000 7fc9cdffb700 1 -- 172.21.15.167:0/2053399812 <== mon.0 v2:172.21.15.118:3300/0 5 ==== osd_map(32..32 src has 1..32) v4 ==== 6260+0+0 (secure 0 0 0) 0x7fc9d8044ee0 con 0x5606235543d0 2023-10-05T19:28:55.310 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.306+0000 7fc9dd564700 1 --2- 172.21.15.167:0/2053399812 >> [v2:172.21.15.167:6824/29889,v1:172.21.15.167:6825/29889] conn(0x7fc9c0030c00 0x7fc9c00330b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-05T19:28:55.311 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.306+0000 7fc9e0230e40 1 --2- 172.21.15.167:0/2053399812 >> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] conn(0x5606235e66f0 0x5606236139d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-05T19:28:55.311 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.306+0000 7fc9e0230e40 1 -- 172.21.15.167:0/2053399812 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:1 5.12 5:49953fa1:::default.realm:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32) v8 -- 0x5606236140d0 con 0x5606235e66f0 2023-10-05T19:28:55.311 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.306+0000 7fc9dd564700 1 --2- 172.21.15.167:0/2053399812 >> [v2:172.21.15.167:6824/29889,v1:172.21.15.167:6825/29889] conn(0x7fc9c0030c00 0x7fc9c00330b0 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7fc9d4005a90 tx=0x7fc9d4015000 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-05T19:28:55.311 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.306+0000 7fc9de566700 1 --2- 172.21.15.167:0/2053399812 >> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] conn(0x5606235e66f0 0x5606236139d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-05T19:28:55.312 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.306+0000 7fc9de566700 1 --2- 172.21.15.167:0/2053399812 >> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] conn(0x5606235e66f0 0x5606236139d0 crc :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-05T19:28:55.313 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.306+0000 7fc9de566700 1 -- 172.21.15.167:0/2053399812 <== osd.1 v2:172.21.15.167:6800/30035 1 ==== osd_op_reply(1 default.realm [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 ==== 157+0+0 (crc 0 0 0) 0x7fc9d000a500 con 0x5606235e66f0 2023-10-05T19:28:55.313 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.306+0000 7fc9e0230e40 20 rados_obj.operate() r=-2 bl.length=0 2023-10-05T19:28:55.313 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.306+0000 7fc9e0230e40 20 realm 2023-10-05T19:28:55.313 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.306+0000 7fc9e0230e40 20 rados->read ofs=0 len=0 2023-10-05T19:28:55.313 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.306+0000 7fc9e0230e40 1 -- 172.21.15.167:0/2053399812 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:2 5.12 5:49953fa1:::default.realm:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32) v8 -- 0x560623611030 con 0x5606235e66f0 2023-10-05T19:28:55.314 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.306+0000 7fc9de566700 1 -- 172.21.15.167:0/2053399812 <== osd.1 v2:172.21.15.167:6800/30035 2 ==== osd_op_reply(2 default.realm [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 ==== 157+0+0 (crc 0 0 0) 0x7fc9d000a500 con 0x5606235e66f0 2023-10-05T19:28:55.314 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.306+0000 7fc9e0230e40 20 rados_obj.operate() r=-2 bl.length=0 2023-10-05T19:28:55.314 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.306+0000 7fc9e0230e40 4 RGWPeriod::init failed to init realm id : (2) No such file or directory 2023-10-05T19:28:55.314 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.306+0000 7fc9e0230e40 20 rados->read ofs=0 len=0 2023-10-05T19:28:55.314 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.306+0000 7fc9e0230e40 1 -- 172.21.15.167:0/2053399812 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:3 5.12 5:49953fa1:::default.realm:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32) v8 -- 0x560623614bb0 con 0x5606235e66f0 2023-10-05T19:28:55.314 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.306+0000 7fc9de566700 1 -- 172.21.15.167:0/2053399812 <== osd.1 v2:172.21.15.167:6800/30035 3 ==== osd_op_reply(3 default.realm [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 ==== 157+0+0 (crc 0 0 0) 0x7fc9d000a500 con 0x5606235e66f0 2023-10-05T19:28:55.314 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.306+0000 7fc9e0230e40 20 rados_obj.operate() r=-2 bl.length=0 2023-10-05T19:28:55.314 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.306+0000 7fc9e0230e40 20 rados->read ofs=0 len=0 2023-10-05T19:28:55.314 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.306+0000 7fc9e0230e40 1 -- 172.21.15.167:0/2053399812 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:4 5.f 5:f43b6ece:::zone_names.default:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32) v8 -- 0x560623614790 con 0x5606235e66f0 2023-10-05T19:28:55.315 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.310+0000 7fc9de566700 1 -- 172.21.15.167:0/2053399812 <== osd.1 v2:172.21.15.167:6800/30035 4 ==== osd_op_reply(4 zone_names.default [read 0~46 out=46b] v0'0 uv1 ondisk = 0) v8 ==== 162+0+46 (crc 0 0 0) 0x7fc9d000a500 con 0x5606235e66f0 2023-10-05T19:28:55.316 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.310+0000 7fc9e0230e40 20 rados_obj.operate() r=0 bl.length=46 2023-10-05T19:28:55.316 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.310+0000 7fc9e0230e40 20 rados->read ofs=0 len=0 2023-10-05T19:28:55.316 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.310+0000 7fc9e0230e40 1 -- 172.21.15.167:0/2053399812 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:5 5.1d 5:b8374eb4:::zone_info.9a9df965-6612-45d4-a45f-6e3c889bacc2:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32) v8 -- 0x560623615620 con 0x5606235e66f0 2023-10-05T19:28:55.316 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.310+0000 7fc9de566700 1 -- 172.21.15.167:0/2053399812 <== osd.1 v2:172.21.15.167:6800/30035 5 ==== osd_op_reply(5 zone_info.9a9df965-6612-45d4-a45f-6e3c889bacc2 [read 0~874 out=874b] v0'0 uv2 ondisk = 0) v8 ==== 190+0+874 (crc 0 0 0) 0x7fc9d000a500 con 0x5606235e66f0 2023-10-05T19:28:55.316 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.310+0000 7fc9e0230e40 20 rados_obj.operate() r=0 bl.length=874 2023-10-05T19:28:55.316 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.310+0000 7fc9e0230e40 20 searching for the correct realm 2023-10-05T19:28:55.316 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.310+0000 7fc9e0230e40 1 --2- 172.21.15.167:0/2053399812 >> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] conn(0x560623617470 0x560623619850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-05T19:28:55.316 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.310+0000 7fc9e0230e40 1 -- 172.21.15.167:0/2053399812 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:6 5.0 5:00000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x560623619d90 con 0x560623617470 2023-10-05T19:28:55.316 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.310+0000 7fc9ddd65700 1 --2- 172.21.15.167:0/2053399812 >> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] conn(0x560623617470 0x560623619850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-05T19:28:55.316 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.310+0000 7fc9ddd65700 1 --2- 172.21.15.167:0/2053399812 >> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] conn(0x560623617470 0x560623619850 crc :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-05T19:28:55.317 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.310+0000 7fc9ddd65700 1 -- 172.21.15.167:0/2053399812 <== osd.3 v2:172.21.15.167:6808/30036 1 ==== osd_op_reply(6 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fc9d8025020 con 0x560623617470 2023-10-05T19:28:55.317 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.310+0000 7fc9ddd65700 1 --2- 172.21.15.167:0/2053399812 >> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] conn(0x7fc9d801ec10 0x7fc9d801fa80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-05T19:28:55.317 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.310+0000 7fc9ddd65700 1 -- 172.21.15.167:0/2053399812 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:7 5.10 5:08000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x560623619d90 con 0x7fc9d801ec10 2023-10-05T19:28:55.317 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.310+0000 7fc9dd564700 1 --2- 172.21.15.167:0/2053399812 >> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] conn(0x7fc9d801ec10 0x7fc9d801fa80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-05T19:28:55.318 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.314+0000 7fc9dd564700 1 --2- 172.21.15.167:0/2053399812 >> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] conn(0x7fc9d801ec10 0x7fc9d801fa80 crc :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-05T19:28:55.319 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.314+0000 7fc9dd564700 1 -- 172.21.15.167:0/2053399812 <== osd.2 v2:172.21.15.167:6816/30037 1 ==== osd_op_reply(7 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fc9d4012450 con 0x7fc9d801ec10 2023-10-05T19:28:55.319 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.314+0000 7fc9dd564700 1 --2- 172.21.15.167:0/2053399812 >> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] conn(0x7fc9d4007030 0x7fc9d401c920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-05T19:28:55.319 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.314+0000 7fc9dd564700 1 -- 172.21.15.167:0/2053399812 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:8 5.8 5:10000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x560623619d90 con 0x7fc9d4007030 2023-10-05T19:28:55.319 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.314+0000 7fc9de566700 1 --2- 172.21.15.167:0/2053399812 >> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] conn(0x7fc9d4007030 0x7fc9d401c920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-05T19:28:55.320 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.314+0000 7fc9de566700 1 --2- 172.21.15.167:0/2053399812 >> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] conn(0x7fc9d4007030 0x7fc9d401c920 crc :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-05T19:28:55.321 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.314+0000 7fc9de566700 1 -- 172.21.15.167:0/2053399812 <== osd.0 v2:172.21.15.118:6800/27690 1 ==== osd_op_reply(8 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fc9d0014040 con 0x7fc9d4007030 2023-10-05T19:28:55.321 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.314+0000 7fc9de566700 1 -- 172.21.15.167:0/2053399812 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:9 5.18 5:18000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x560623619d90 con 0x5606235e66f0 2023-10-05T19:28:55.321 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.318+0000 7fc9de566700 1 -- 172.21.15.167:0/2053399812 <== osd.1 v2:172.21.15.167:6800/30035 6 ==== osd_op_reply(9 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fc9d0014040 con 0x5606235e66f0 2023-10-05T19:28:55.321 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.318+0000 7fc9de566700 1 -- 172.21.15.167:0/2053399812 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:10 5.4 5:20000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x560623619d90 con 0x5606235e66f0 2023-10-05T19:28:55.322 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.318+0000 7fc9de566700 1 -- 172.21.15.167:0/2053399812 <== osd.1 v2:172.21.15.167:6800/30035 7 ==== osd_op_reply(10 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fc9d0014040 con 0x5606235e66f0 2023-10-05T19:28:55.322 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.318+0000 7fc9de566700 1 -- 172.21.15.167:0/2053399812 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:11 5.14 5:28000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x560623619d90 con 0x7fc9d4007030 2023-10-05T19:28:55.323 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.318+0000 7fc9de566700 1 -- 172.21.15.167:0/2053399812 <== osd.0 v2:172.21.15.118:6800/27690 2 ==== osd_op_reply(11 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fc9d0014040 con 0x7fc9d4007030 2023-10-05T19:28:55.323 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.318+0000 7fc9de566700 1 -- 172.21.15.167:0/2053399812 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:12 5.c 5:30000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x560623619d90 con 0x7fc9d4007030 2023-10-05T19:28:55.324 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.318+0000 7fc9de566700 1 -- 172.21.15.167:0/2053399812 <== osd.0 v2:172.21.15.118:6800/27690 3 ==== osd_op_reply(12 [pgnls start_epoch 32 out=112b] v23'1 uv1 ondisk = 1) v8 ==== 144+0+112 (crc 0 0 0) 0x7fc9d0014040 con 0x7fc9d4007030 2023-10-05T19:28:55.324 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.318+0000 7fc9de566700 1 -- 172.21.15.167:0/2053399812 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:13 5.1c 5:38000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x560623619d90 con 0x7fc9d801ec10 2023-10-05T19:28:55.324 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.318+0000 7fc9dd564700 1 -- 172.21.15.167:0/2053399812 <== osd.2 v2:172.21.15.167:6816/30037 2 ==== osd_op_reply(13 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fc9d4012450 con 0x7fc9d801ec10 2023-10-05T19:28:55.325 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.318+0000 7fc9dd564700 1 -- 172.21.15.167:0/2053399812 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:14 5.2 5:40000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x560623619d90 con 0x560623617470 2023-10-05T19:28:55.325 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.322+0000 7fc9ddd65700 1 -- 172.21.15.167:0/2053399812 <== osd.3 v2:172.21.15.167:6808/30036 2 ==== osd_op_reply(14 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fc9d8025020 con 0x560623617470 2023-10-05T19:28:55.325 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.322+0000 7fc9ddd65700 1 -- 172.21.15.167:0/2053399812 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:15 5.12 5:48000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x560623619d90 con 0x5606235e66f0 2023-10-05T19:28:55.326 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.322+0000 7fc9de566700 1 -- 172.21.15.167:0/2053399812 <== osd.1 v2:172.21.15.167:6800/30035 8 ==== osd_op_reply(15 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fc9d0014040 con 0x5606235e66f0 2023-10-05T19:28:55.326 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.322+0000 7fc9de566700 1 -- 172.21.15.167:0/2053399812 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:16 5.a 5:50000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x560623619d90 con 0x7fc9d4007030 2023-10-05T19:28:55.327 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.322+0000 7fc9de566700 1 -- 172.21.15.167:0/2053399812 <== osd.0 v2:172.21.15.118:6800/27690 4 ==== osd_op_reply(16 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fc9d0022040 con 0x7fc9d4007030 2023-10-05T19:28:55.327 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.322+0000 7fc9de566700 1 -- 172.21.15.167:0/2053399812 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:17 5.1a 5:58000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x560623619d90 con 0x5606235e66f0 2023-10-05T19:28:55.328 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.322+0000 7fc9de566700 1 -- 172.21.15.167:0/2053399812 <== osd.1 v2:172.21.15.167:6800/30035 9 ==== osd_op_reply(17 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fc9d0022040 con 0x5606235e66f0 2023-10-05T19:28:55.328 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.322+0000 7fc9de566700 1 -- 172.21.15.167:0/2053399812 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:18 5.6 5:60000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x560623619d90 con 0x7fc9d801ec10 2023-10-05T19:28:55.328 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.322+0000 7fc9dd564700 1 -- 172.21.15.167:0/2053399812 <== osd.2 v2:172.21.15.167:6816/30037 3 ==== osd_op_reply(18 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fc9d4012450 con 0x7fc9d801ec10 2023-10-05T19:28:55.329 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.322+0000 7fc9dd564700 1 -- 172.21.15.167:0/2053399812 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:19 5.16 5:68000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x560623619d90 con 0x560623617470 2023-10-05T19:28:55.329 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.322+0000 7fc9ddd65700 1 -- 172.21.15.167:0/2053399812 <== osd.3 v2:172.21.15.167:6808/30036 3 ==== osd_op_reply(19 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fc9d8025020 con 0x560623617470 2023-10-05T19:28:55.329 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.322+0000 7fc9ddd65700 1 -- 172.21.15.167:0/2053399812 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:20 5.e 5:70000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x560623619d90 con 0x560623617470 2023-10-05T19:28:55.330 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.326+0000 7fc9ddd65700 1 -- 172.21.15.167:0/2053399812 <== osd.3 v2:172.21.15.167:6808/30036 4 ==== osd_op_reply(20 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fc9d8025020 con 0x560623617470 2023-10-05T19:28:55.330 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.326+0000 7fc9ddd65700 1 -- 172.21.15.167:0/2053399812 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:21 5.1e 5:78000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x560623619d90 con 0x7fc9d4007030 2023-10-05T19:28:55.331 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.326+0000 7fc9de566700 1 -- 172.21.15.167:0/2053399812 <== osd.0 v2:172.21.15.118:6800/27690 5 ==== osd_op_reply(21 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fc9d0022040 con 0x7fc9d4007030 2023-10-05T19:28:55.331 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.326+0000 7fc9de566700 1 -- 172.21.15.167:0/2053399812 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:22 5.1 5:80000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x560623619d90 con 0x560623617470 2023-10-05T19:28:55.332 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.326+0000 7fc9ddd65700 1 -- 172.21.15.167:0/2053399812 <== osd.3 v2:172.21.15.167:6808/30036 5 ==== osd_op_reply(22 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fc9d8025020 con 0x560623617470 2023-10-05T19:28:55.332 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.326+0000 7fc9ddd65700 1 -- 172.21.15.167:0/2053399812 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:23 5.11 5:88000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x560623619d90 con 0x560623617470 2023-10-05T19:28:55.332 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.326+0000 7fc9ddd65700 1 -- 172.21.15.167:0/2053399812 <== osd.3 v2:172.21.15.167:6808/30036 6 ==== osd_op_reply(23 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fc9d8025020 con 0x560623617470 2023-10-05T19:28:55.332 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.326+0000 7fc9ddd65700 1 -- 172.21.15.167:0/2053399812 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:24 5.9 5:90000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x560623619d90 con 0x5606235e66f0 2023-10-05T19:28:55.333 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.326+0000 7fc9de566700 1 -- 172.21.15.167:0/2053399812 <== osd.1 v2:172.21.15.167:6800/30035 10 ==== osd_op_reply(24 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fc9d0022040 con 0x5606235e66f0 2023-10-05T19:28:55.333 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.326+0000 7fc9de566700 1 -- 172.21.15.167:0/2053399812 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:25 5.19 5:98000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x560623619d90 con 0x5606235e66f0 2023-10-05T19:28:55.334 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.330+0000 7fc9de566700 1 -- 172.21.15.167:0/2053399812 <== osd.1 v2:172.21.15.167:6800/30035 11 ==== osd_op_reply(25 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fc9d0022040 con 0x5606235e66f0 2023-10-05T19:28:55.334 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.330+0000 7fc9de566700 1 -- 172.21.15.167:0/2053399812 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:26 5.5 5:a0000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x560623619d90 con 0x7fc9d4007030 2023-10-05T19:28:55.335 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.330+0000 7fc9de566700 1 -- 172.21.15.167:0/2053399812 <== osd.0 v2:172.21.15.118:6800/27690 6 ==== osd_op_reply(26 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fc9d0022040 con 0x7fc9d4007030 2023-10-05T19:28:55.335 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.330+0000 7fc9de566700 1 -- 172.21.15.167:0/2053399812 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:27 5.15 5:a8000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x560623619d90 con 0x560623617470 2023-10-05T19:28:55.335 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.330+0000 7fc9ddd65700 1 -- 172.21.15.167:0/2053399812 <== osd.3 v2:172.21.15.167:6808/30036 7 ==== osd_op_reply(27 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fc9d8025020 con 0x560623617470 2023-10-05T19:28:55.335 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.330+0000 7fc9ddd65700 1 -- 172.21.15.167:0/2053399812 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:28 5.d 5:b0000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x560623619d90 con 0x7fc9d801ec10 2023-10-05T19:28:55.336 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.330+0000 7fc9dd564700 1 -- 172.21.15.167:0/2053399812 <== osd.2 v2:172.21.15.167:6816/30037 4 ==== osd_op_reply(28 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fc9d4012450 con 0x7fc9d801ec10 2023-10-05T19:28:55.336 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.330+0000 7fc9dd564700 1 -- 172.21.15.167:0/2053399812 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:29 5.1d 5:b8000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x560623619d90 con 0x5606235e66f0 2023-10-05T19:28:55.337 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.330+0000 7fc9de566700 1 -- 172.21.15.167:0/2053399812 <== osd.1 v2:172.21.15.167:6800/30035 12 ==== osd_op_reply(29 [pgnls start_epoch 32 out=107b] v30'2 uv2 ondisk = 1) v8 ==== 144+0+107 (crc 0 0 0) 0x7fc9d0022040 con 0x5606235e66f0 2023-10-05T19:28:55.337 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.330+0000 7fc9de566700 1 -- 172.21.15.167:0/2053399812 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:30 5.3 5:c0000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x560623619d90 con 0x7fc9d4007030 2023-10-05T19:28:55.338 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.334+0000 7fc9de566700 1 -- 172.21.15.167:0/2053399812 <== osd.0 v2:172.21.15.118:6800/27690 7 ==== osd_op_reply(30 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fc9d0022040 con 0x7fc9d4007030 2023-10-05T19:28:55.338 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.334+0000 7fc9de566700 1 -- 172.21.15.167:0/2053399812 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:31 5.13 5:c8000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x560623619d90 con 0x560623617470 2023-10-05T19:28:55.339 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.334+0000 7fc9ddd65700 1 -- 172.21.15.167:0/2053399812 <== osd.3 v2:172.21.15.167:6808/30036 8 ==== osd_op_reply(31 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fc9d8025020 con 0x560623617470 2023-10-05T19:28:55.339 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.334+0000 7fc9ddd65700 1 -- 172.21.15.167:0/2053399812 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:32 5.b 5:d0000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x560623619d90 con 0x7fc9d4007030 2023-10-05T19:28:55.340 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.334+0000 7fc9de566700 1 -- 172.21.15.167:0/2053399812 <== osd.0 v2:172.21.15.118:6800/27690 8 ==== osd_op_reply(32 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fc9d0022040 con 0x7fc9d4007030 2023-10-05T19:28:55.340 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.334+0000 7fc9de566700 1 -- 172.21.15.167:0/2053399812 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:33 5.1b 5:d8000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x560623619d90 con 0x7fc9d801ec10 2023-10-05T19:28:55.341 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.334+0000 7fc9dd564700 1 -- 172.21.15.167:0/2053399812 <== osd.2 v2:172.21.15.167:6816/30037 5 ==== osd_op_reply(33 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fc9d4012450 con 0x7fc9d801ec10 2023-10-05T19:28:55.341 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.334+0000 7fc9dd564700 1 -- 172.21.15.167:0/2053399812 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:34 5.7 5:e0000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x560623619d90 con 0x560623617470 2023-10-05T19:28:55.341 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.338+0000 7fc9ddd65700 1 -- 172.21.15.167:0/2053399812 <== osd.3 v2:172.21.15.167:6808/30036 9 ==== osd_op_reply(34 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fc9d8025020 con 0x560623617470 2023-10-05T19:28:55.341 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.338+0000 7fc9ddd65700 1 -- 172.21.15.167:0/2053399812 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:35 5.17 5:e8000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x560623619d90 con 0x7fc9d4007030 2023-10-05T19:28:55.342 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.338+0000 7fc9de566700 1 -- 172.21.15.167:0/2053399812 <== osd.0 v2:172.21.15.118:6800/27690 9 ==== osd_op_reply(35 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fc9d0022040 con 0x7fc9d4007030 2023-10-05T19:28:55.343 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.338+0000 7fc9de566700 1 -- 172.21.15.167:0/2053399812 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:36 5.f 5:f0000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x560623619d90 con 0x5606235e66f0 2023-10-05T19:28:55.343 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.338+0000 7fc9de566700 1 -- 172.21.15.167:0/2053399812 <== osd.1 v2:172.21.15.167:6800/30035 13 ==== osd_op_reply(36 [pgnls start_epoch 32 out=115b] v23'2 uv2 ondisk = 1) v8 ==== 144+0+115 (crc 0 0 0) 0x7fc9d002f080 con 0x5606235e66f0 2023-10-05T19:28:55.343 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.338+0000 7fc9de566700 1 -- 172.21.15.167:0/2053399812 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:37 5.1f 5:f8000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x560623619d90 con 0x7fc9d801ec10 2023-10-05T19:28:55.344 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.338+0000 7fc9dd564700 1 -- 172.21.15.167:0/2053399812 <== osd.2 v2:172.21.15.167:6816/30037 6 ==== osd_op_reply(37 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fc9d4012450 con 0x7fc9d801ec10 2023-10-05T19:28:55.344 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.338+0000 7fc9e0230e40 20 RGWRados::pool_iterate: got zonegroup_info.4f1f719d-9648-417b-b10c-e5d939825419 2023-10-05T19:28:55.344 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.338+0000 7fc9e0230e40 20 RGWRados::pool_iterate: got zone_info.9a9df965-6612-45d4-a45f-6e3c889bacc2 2023-10-05T19:28:55.345 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.338+0000 7fc9e0230e40 20 RGWRados::pool_iterate: got zone_names.default 2023-10-05T19:28:55.345 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.338+0000 7fc9e0230e40 20 RGWRados::pool_iterate: got zonegroups_names.default 2023-10-05T19:28:55.345 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.338+0000 7fc9e0230e40 20 rados->read ofs=0 len=0 2023-10-05T19:28:55.345 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.338+0000 7fc9e0230e40 1 -- 172.21.15.167:0/2053399812 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:38 5.12 5:49953fa1:::default.realm:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32) v8 -- 0x560623616220 con 0x5606235e66f0 2023-10-05T19:28:55.345 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.338+0000 7fc9de566700 1 -- 172.21.15.167:0/2053399812 <== osd.1 v2:172.21.15.167:6800/30035 14 ==== osd_op_reply(38 default.realm [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 ==== 157+0+0 (crc 0 0 0) 0x7fc9d002f080 con 0x5606235e66f0 2023-10-05T19:28:55.346 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.338+0000 7fc9e0230e40 20 rados_obj.operate() r=-2 bl.length=0 2023-10-05T19:28:55.346 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.338+0000 7fc9e0230e40 20 rados->read ofs=0 len=0 2023-10-05T19:28:55.346 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.338+0000 7fc9e0230e40 1 -- 172.21.15.167:0/2053399812 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:39 5.f 5:f4c53578:::zonegroups_names.default:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32) v8 -- 0x560623616ba0 con 0x5606235e66f0 2023-10-05T19:28:55.346 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.342+0000 7fc9de566700 1 -- 172.21.15.167:0/2053399812 <== osd.1 v2:172.21.15.167:6800/30035 15 ==== osd_op_reply(39 zonegroups_names.default [read 0~46 out=46b] v0'0 uv2 ondisk = 0) v8 ==== 168+0+46 (crc 0 0 0) 0x7fc9d002f080 con 0x5606235e66f0 2023-10-05T19:28:55.346 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.342+0000 7fc9e0230e40 20 rados_obj.operate() r=0 bl.length=46 2023-10-05T19:28:55.346 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.342+0000 7fc9e0230e40 20 rados->read ofs=0 len=0 2023-10-05T19:28:55.346 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.342+0000 7fc9e0230e40 1 -- 172.21.15.167:0/2053399812 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:40 5.c 5:30d299a5:::zonegroup_info.4f1f719d-9648-417b-b10c-e5d939825419:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32) v8 -- 0x560623615e10 con 0x7fc9d4007030 2023-10-05T19:28:55.347 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.342+0000 7fc9de566700 1 -- 172.21.15.167:0/2053399812 <== osd.0 v2:172.21.15.118:6800/27690 10 ==== osd_op_reply(40 zonegroup_info.4f1f719d-9648-417b-b10c-e5d939825419 [read 0~362 out=362b] v0'0 uv1 ondisk = 0) v8 ==== 195+0+362 (crc 0 0 0) 0x7fc9d002f080 con 0x7fc9d4007030 2023-10-05T19:28:55.347 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.342+0000 7fc9e0230e40 20 rados_obj.operate() r=0 bl.length=362 2023-10-05T19:28:55.347 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.342+0000 7fc9e0230e40 20 rados->read ofs=0 len=0 2023-10-05T19:28:55.347 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.342+0000 7fc9e0230e40 1 -- 172.21.15.167:0/2053399812 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:41 5.13 5:cbbf6c60:::converted:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32) v8 -- 0x56062361a680 con 0x560623617470 2023-10-05T19:28:55.348 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.342+0000 7fc9ddd65700 1 -- 172.21.15.167:0/2053399812 <== osd.3 v2:172.21.15.167:6808/30036 10 ==== osd_op_reply(41 converted [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 ==== 153+0+0 (crc 0 0 0) 0x7fc9d8025020 con 0x560623617470 2023-10-05T19:28:55.348 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.342+0000 7fc9e0230e40 20 rados_obj.operate() r=-2 bl.length=0 2023-10-05T19:28:55.348 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.342+0000 7fc9e0230e40 20 rados->read ofs=0 len=0 2023-10-05T19:28:55.348 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.342+0000 7fc9e0230e40 1 -- 172.21.15.167:0/2053399812 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:42 5.12 5:49953fa1:::default.realm:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32) v8 -- 0x56062361aa30 con 0x5606235e66f0 2023-10-05T19:28:55.348 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.342+0000 7fc9de566700 1 -- 172.21.15.167:0/2053399812 <== osd.1 v2:172.21.15.167:6800/30035 16 ==== osd_op_reply(42 default.realm [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 ==== 157+0+0 (crc 0 0 0) 0x7fc9d002f080 con 0x5606235e66f0 2023-10-05T19:28:55.349 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.342+0000 7fc9e0230e40 20 rados_obj.operate() r=-2 bl.length=0 2023-10-05T19:28:55.349 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.342+0000 7fc9e0230e40 20 rados->read ofs=0 len=0 2023-10-05T19:28:55.349 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.342+0000 7fc9e0230e40 1 -- 172.21.15.167:0/2053399812 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:43 5.f 5:f4c53578:::zonegroups_names.default:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32) v8 -- 0x56062361aa30 con 0x5606235e66f0 2023-10-05T19:28:55.349 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.342+0000 7fc9de566700 1 -- 172.21.15.167:0/2053399812 <== osd.1 v2:172.21.15.167:6800/30035 17 ==== osd_op_reply(43 zonegroups_names.default [read 0~46 out=46b] v0'0 uv2 ondisk = 0) v8 ==== 168+0+46 (crc 0 0 0) 0x7fc9d002f080 con 0x5606235e66f0 2023-10-05T19:28:55.349 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.342+0000 7fc9e0230e40 20 rados_obj.operate() r=0 bl.length=46 2023-10-05T19:28:55.349 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.342+0000 7fc9e0230e40 1 -- 172.21.15.167:0/2053399812 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:44 5.0 5:00000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x56062361ba40 con 0x560623617470 2023-10-05T19:28:55.350 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.346+0000 7fc9ddd65700 1 -- 172.21.15.167:0/2053399812 <== osd.3 v2:172.21.15.167:6808/30036 11 ==== osd_op_reply(44 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fc9d8025020 con 0x560623617470 2023-10-05T19:28:55.350 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.346+0000 7fc9ddd65700 1 -- 172.21.15.167:0/2053399812 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:45 5.10 5:08000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x56062361ba40 con 0x7fc9d801ec10 2023-10-05T19:28:55.351 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.346+0000 7fc9dd564700 1 -- 172.21.15.167:0/2053399812 <== osd.2 v2:172.21.15.167:6816/30037 7 ==== osd_op_reply(45 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fc9d4012450 con 0x7fc9d801ec10 2023-10-05T19:28:55.351 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.346+0000 7fc9dd564700 1 -- 172.21.15.167:0/2053399812 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:46 5.8 5:10000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x56062361ba40 con 0x7fc9d4007030 2023-10-05T19:28:55.352 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.346+0000 7fc9de566700 1 -- 172.21.15.167:0/2053399812 <== osd.0 v2:172.21.15.118:6800/27690 11 ==== osd_op_reply(46 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fc9d002f080 con 0x7fc9d4007030 2023-10-05T19:28:55.352 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.346+0000 7fc9de566700 1 -- 172.21.15.167:0/2053399812 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:47 5.18 5:18000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x560623615e10 con 0x5606235e66f0 2023-10-05T19:28:55.353 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.346+0000 7fc9de566700 1 -- 172.21.15.167:0/2053399812 <== osd.1 v2:172.21.15.167:6800/30035 18 ==== osd_op_reply(47 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fc9d002f080 con 0x5606235e66f0 2023-10-05T19:28:55.353 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.346+0000 7fc9de566700 1 -- 172.21.15.167:0/2053399812 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:48 5.4 5:20000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x560623615e10 con 0x5606235e66f0 2023-10-05T19:28:55.353 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.346+0000 7fc9de566700 1 -- 172.21.15.167:0/2053399812 <== osd.1 v2:172.21.15.167:6800/30035 19 ==== osd_op_reply(48 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fc9d002f080 con 0x5606235e66f0 2023-10-05T19:28:55.353 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.350+0000 7fc9de566700 1 -- 172.21.15.167:0/2053399812 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:49 5.14 5:28000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x560623615e10 con 0x7fc9d4007030 2023-10-05T19:28:55.354 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.350+0000 7fc9de566700 1 -- 172.21.15.167:0/2053399812 <== osd.0 v2:172.21.15.118:6800/27690 12 ==== osd_op_reply(49 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fc9d002f080 con 0x7fc9d4007030 2023-10-05T19:28:55.354 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.350+0000 7fc9de566700 1 -- 172.21.15.167:0/2053399812 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:50 5.c 5:30000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x560623615e10 con 0x7fc9d4007030 2023-10-05T19:28:55.355 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.350+0000 7fc9de566700 1 -- 172.21.15.167:0/2053399812 <== osd.0 v2:172.21.15.118:6800/27690 13 ==== osd_op_reply(50 [pgnls start_epoch 32 out=112b] v23'1 uv1 ondisk = 1) v8 ==== 144+0+112 (crc 0 0 0) 0x7fc9d002f080 con 0x7fc9d4007030 2023-10-05T19:28:55.355 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.350+0000 7fc9de566700 1 -- 172.21.15.167:0/2053399812 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:51 5.1c 5:38000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x560623615e10 con 0x7fc9d801ec10 2023-10-05T19:28:55.356 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.350+0000 7fc9dd564700 1 -- 172.21.15.167:0/2053399812 <== osd.2 v2:172.21.15.167:6816/30037 8 ==== osd_op_reply(51 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fc9d4012450 con 0x7fc9d801ec10 2023-10-05T19:28:55.356 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.350+0000 7fc9dd564700 1 -- 172.21.15.167:0/2053399812 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:52 5.2 5:40000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x560623615e10 con 0x560623617470 2023-10-05T19:28:55.357 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.350+0000 7fc9ddd65700 1 -- 172.21.15.167:0/2053399812 <== osd.3 v2:172.21.15.167:6808/30036 12 ==== osd_op_reply(52 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fc9d8025020 con 0x560623617470 2023-10-05T19:28:55.357 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.350+0000 7fc9ddd65700 1 -- 172.21.15.167:0/2053399812 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:53 5.12 5:48000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x560623615e10 con 0x5606235e66f0 2023-10-05T19:28:55.358 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.354+0000 7fc9de566700 1 -- 172.21.15.167:0/2053399812 <== osd.1 v2:172.21.15.167:6800/30035 20 ==== osd_op_reply(53 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fc9d002f080 con 0x5606235e66f0 2023-10-05T19:28:55.358 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.354+0000 7fc9de566700 1 -- 172.21.15.167:0/2053399812 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:54 5.a 5:50000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x560623615e10 con 0x7fc9d4007030 2023-10-05T19:28:55.359 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.354+0000 7fc9de566700 1 -- 172.21.15.167:0/2053399812 <== osd.0 v2:172.21.15.118:6800/27690 14 ==== osd_op_reply(54 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fc9d002f080 con 0x7fc9d4007030 2023-10-05T19:28:55.359 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.354+0000 7fc9de566700 1 -- 172.21.15.167:0/2053399812 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:55 5.1a 5:58000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x560623615e10 con 0x5606235e66f0 2023-10-05T19:28:55.359 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.354+0000 7fc9de566700 1 -- 172.21.15.167:0/2053399812 <== osd.1 v2:172.21.15.167:6800/30035 21 ==== osd_op_reply(55 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fc9d002f080 con 0x5606235e66f0 2023-10-05T19:28:55.360 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.354+0000 7fc9de566700 1 -- 172.21.15.167:0/2053399812 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:56 5.6 5:60000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x560623615e10 con 0x7fc9d801ec10 2023-10-05T19:28:55.360 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.354+0000 7fc9dd564700 1 -- 172.21.15.167:0/2053399812 <== osd.2 v2:172.21.15.167:6816/30037 9 ==== osd_op_reply(56 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fc9d4012450 con 0x7fc9d801ec10 2023-10-05T19:28:55.360 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.354+0000 7fc9dd564700 1 -- 172.21.15.167:0/2053399812 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:57 5.16 5:68000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x560623615e10 con 0x560623617470 2023-10-05T19:28:55.361 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.354+0000 7fc9ddd65700 1 -- 172.21.15.167:0/2053399812 <== osd.3 v2:172.21.15.167:6808/30036 13 ==== osd_op_reply(57 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fc9d8025020 con 0x560623617470 2023-10-05T19:28:55.361 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.354+0000 7fc9ddd65700 1 -- 172.21.15.167:0/2053399812 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:58 5.e 5:70000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x560623615e10 con 0x560623617470 2023-10-05T19:28:55.362 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.358+0000 7fc9ddd65700 1 -- 172.21.15.167:0/2053399812 <== osd.3 v2:172.21.15.167:6808/30036 14 ==== osd_op_reply(58 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fc9d8025020 con 0x560623617470 2023-10-05T19:28:55.362 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.358+0000 7fc9ddd65700 1 -- 172.21.15.167:0/2053399812 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:59 5.1e 5:78000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x560623615e10 con 0x7fc9d4007030 2023-10-05T19:28:55.363 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.358+0000 7fc9de566700 1 -- 172.21.15.167:0/2053399812 <== osd.0 v2:172.21.15.118:6800/27690 15 ==== osd_op_reply(59 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fc9d002f080 con 0x7fc9d4007030 2023-10-05T19:28:55.363 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.358+0000 7fc9de566700 1 -- 172.21.15.167:0/2053399812 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:60 5.1 5:80000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x560623615e10 con 0x560623617470 2023-10-05T19:28:55.364 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.358+0000 7fc9ddd65700 1 -- 172.21.15.167:0/2053399812 <== osd.3 v2:172.21.15.167:6808/30036 15 ==== osd_op_reply(60 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fc9d8025020 con 0x560623617470 2023-10-05T19:28:55.364 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.358+0000 7fc9ddd65700 1 -- 172.21.15.167:0/2053399812 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:61 5.11 5:88000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x560623615e10 con 0x560623617470 2023-10-05T19:28:55.365 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.358+0000 7fc9ddd65700 1 -- 172.21.15.167:0/2053399812 <== osd.3 v2:172.21.15.167:6808/30036 16 ==== osd_op_reply(61 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fc9d8025020 con 0x560623617470 2023-10-05T19:28:55.365 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.358+0000 7fc9ddd65700 1 -- 172.21.15.167:0/2053399812 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:62 5.9 5:90000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x560623615e10 con 0x5606235e66f0 2023-10-05T19:28:55.365 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.362+0000 7fc9de566700 1 -- 172.21.15.167:0/2053399812 <== osd.1 v2:172.21.15.167:6800/30035 22 ==== osd_op_reply(62 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fc9d002f080 con 0x5606235e66f0 2023-10-05T19:28:55.365 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.362+0000 7fc9de566700 1 -- 172.21.15.167:0/2053399812 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:63 5.19 5:98000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x560623615e10 con 0x5606235e66f0 2023-10-05T19:28:55.366 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.362+0000 7fc9de566700 1 -- 172.21.15.167:0/2053399812 <== osd.1 v2:172.21.15.167:6800/30035 23 ==== osd_op_reply(63 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fc9d002f080 con 0x5606235e66f0 2023-10-05T19:28:55.366 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.362+0000 7fc9de566700 1 -- 172.21.15.167:0/2053399812 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:64 5.5 5:a0000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x560623615e10 con 0x7fc9d4007030 2023-10-05T19:28:55.367 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.362+0000 7fc9de566700 1 -- 172.21.15.167:0/2053399812 <== osd.0 v2:172.21.15.118:6800/27690 16 ==== osd_op_reply(64 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fc9d002f080 con 0x7fc9d4007030 2023-10-05T19:28:55.367 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.362+0000 7fc9de566700 1 -- 172.21.15.167:0/2053399812 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:65 5.15 5:a8000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x560623615e10 con 0x560623617470 2023-10-05T19:28:55.368 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.362+0000 7fc9ddd65700 1 -- 172.21.15.167:0/2053399812 <== osd.3 v2:172.21.15.167:6808/30036 17 ==== osd_op_reply(65 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fc9d8025020 con 0x560623617470 2023-10-05T19:28:55.368 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.362+0000 7fc9ddd65700 1 -- 172.21.15.167:0/2053399812 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:66 5.d 5:b0000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x560623615e10 con 0x7fc9d801ec10 2023-10-05T19:28:55.369 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.362+0000 7fc9dd564700 1 -- 172.21.15.167:0/2053399812 <== osd.2 v2:172.21.15.167:6816/30037 10 ==== osd_op_reply(66 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fc9d4012450 con 0x7fc9d801ec10 2023-10-05T19:28:55.369 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.362+0000 7fc9dd564700 1 -- 172.21.15.167:0/2053399812 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:67 5.1d 5:b8000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x560623615e10 con 0x5606235e66f0 2023-10-05T19:28:55.370 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.366+0000 7fc9de566700 1 -- 172.21.15.167:0/2053399812 <== osd.1 v2:172.21.15.167:6800/30035 24 ==== osd_op_reply(67 [pgnls start_epoch 32 out=107b] v30'2 uv2 ondisk = 1) v8 ==== 144+0+107 (crc 0 0 0) 0x7fc9d002f080 con 0x5606235e66f0 2023-10-05T19:28:55.370 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.366+0000 7fc9de566700 1 -- 172.21.15.167:0/2053399812 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:68 5.3 5:c0000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x560623615e10 con 0x7fc9d4007030 2023-10-05T19:28:55.371 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.366+0000 7fc9de566700 1 -- 172.21.15.167:0/2053399812 <== osd.0 v2:172.21.15.118:6800/27690 17 ==== osd_op_reply(68 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fc9d002f080 con 0x7fc9d4007030 2023-10-05T19:28:55.371 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.366+0000 7fc9de566700 1 -- 172.21.15.167:0/2053399812 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:69 5.13 5:c8000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x560623615e10 con 0x560623617470 2023-10-05T19:28:55.372 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.366+0000 7fc9ddd65700 1 -- 172.21.15.167:0/2053399812 <== osd.3 v2:172.21.15.167:6808/30036 18 ==== osd_op_reply(69 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fc9d8025020 con 0x560623617470 2023-10-05T19:28:55.372 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.366+0000 7fc9ddd65700 1 -- 172.21.15.167:0/2053399812 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:70 5.b 5:d0000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x560623615e10 con 0x7fc9d4007030 2023-10-05T19:28:55.373 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.366+0000 7fc9de566700 1 -- 172.21.15.167:0/2053399812 <== osd.0 v2:172.21.15.118:6800/27690 18 ==== osd_op_reply(70 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fc9d002f080 con 0x7fc9d4007030 2023-10-05T19:28:55.373 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.366+0000 7fc9de566700 1 -- 172.21.15.167:0/2053399812 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:71 5.1b 5:d8000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x560623615e10 con 0x7fc9d801ec10 2023-10-05T19:28:55.374 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.370+0000 7fc9dd564700 1 -- 172.21.15.167:0/2053399812 <== osd.2 v2:172.21.15.167:6816/30037 11 ==== osd_op_reply(71 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fc9d4012450 con 0x7fc9d801ec10 2023-10-05T19:28:55.374 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.370+0000 7fc9dd564700 1 -- 172.21.15.167:0/2053399812 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:72 5.7 5:e0000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x560623615e10 con 0x560623617470 2023-10-05T19:28:55.374 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.370+0000 7fc9ddd65700 1 -- 172.21.15.167:0/2053399812 <== osd.3 v2:172.21.15.167:6808/30036 19 ==== osd_op_reply(72 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fc9d8025020 con 0x560623617470 2023-10-05T19:28:55.375 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.370+0000 7fc9ddd65700 1 -- 172.21.15.167:0/2053399812 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:73 5.17 5:e8000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x560623615e10 con 0x7fc9d4007030 2023-10-05T19:28:55.376 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.370+0000 7fc9de566700 1 -- 172.21.15.167:0/2053399812 <== osd.0 v2:172.21.15.118:6800/27690 19 ==== osd_op_reply(73 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fc9d002f080 con 0x7fc9d4007030 2023-10-05T19:28:55.376 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.370+0000 7fc9de566700 1 -- 172.21.15.167:0/2053399812 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:74 5.f 5:f0000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x560623615e10 con 0x5606235e66f0 2023-10-05T19:28:55.377 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.370+0000 7fc9de566700 1 -- 172.21.15.167:0/2053399812 <== osd.1 v2:172.21.15.167:6800/30035 25 ==== osd_op_reply(74 [pgnls start_epoch 32 out=115b] v23'2 uv2 ondisk = 1) v8 ==== 144+0+115 (crc 0 0 0) 0x7fc9d002f080 con 0x5606235e66f0 2023-10-05T19:28:55.377 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.370+0000 7fc9de566700 1 -- 172.21.15.167:0/2053399812 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:75 5.1f 5:f8000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x560623615e10 con 0x7fc9d801ec10 2023-10-05T19:28:55.378 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.374+0000 7fc9dd564700 1 -- 172.21.15.167:0/2053399812 <== osd.2 v2:172.21.15.167:6816/30037 12 ==== osd_op_reply(75 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fc9d4012450 con 0x7fc9d801ec10 2023-10-05T19:28:55.378 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.374+0000 7fc9e0230e40 20 RGWRados::pool_iterate: got zonegroup_info.4f1f719d-9648-417b-b10c-e5d939825419 2023-10-05T19:28:55.378 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.374+0000 7fc9e0230e40 20 RGWRados::pool_iterate: got zone_info.9a9df965-6612-45d4-a45f-6e3c889bacc2 2023-10-05T19:28:55.378 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.374+0000 7fc9e0230e40 20 RGWRados::pool_iterate: got zone_names.default 2023-10-05T19:28:55.378 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.374+0000 7fc9e0230e40 20 RGWRados::pool_iterate: got zonegroups_names.default 2023-10-05T19:28:55.378 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.374+0000 7fc9e0230e40 20 rados->read ofs=0 len=0 2023-10-05T19:28:55.378 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.374+0000 7fc9e0230e40 1 -- 172.21.15.167:0/2053399812 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:76 5.f 5:f43b6ece:::zone_names.default:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32) v8 -- 0x56062361aa30 con 0x5606235e66f0 2023-10-05T19:28:55.379 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.374+0000 7fc9de566700 1 -- 172.21.15.167:0/2053399812 <== osd.1 v2:172.21.15.167:6800/30035 26 ==== osd_op_reply(76 zone_names.default [read 0~46 out=46b] v0'0 uv1 ondisk = 0) v8 ==== 162+0+46 (crc 0 0 0) 0x7fc9d002f080 con 0x5606235e66f0 2023-10-05T19:28:55.379 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.374+0000 7fc9e0230e40 20 rados_obj.operate() r=0 bl.length=46 2023-10-05T19:28:55.381 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.374+0000 7fc9e0230e40 20 rados->read ofs=0 len=0 2023-10-05T19:28:55.381 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.374+0000 7fc9e0230e40 1 -- 172.21.15.167:0/2053399812 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:77 5.1d 5:b8374eb4:::zone_info.9a9df965-6612-45d4-a45f-6e3c889bacc2:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32) v8 -- 0x56062361ade0 con 0x5606235e66f0 2023-10-05T19:28:55.381 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.374+0000 7fc9de566700 1 -- 172.21.15.167:0/2053399812 <== osd.1 v2:172.21.15.167:6800/30035 27 ==== osd_op_reply(77 zone_info.9a9df965-6612-45d4-a45f-6e3c889bacc2 [read 0~874 out=874b] v0'0 uv2 ondisk = 0) v8 ==== 190+0+874 (crc 0 0 0) 0x7fc9d002f080 con 0x5606235e66f0 2023-10-05T19:28:55.381 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.374+0000 7fc9e0230e40 20 rados_obj.operate() r=0 bl.length=874 2023-10-05T19:28:55.381 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.374+0000 7fc9e0230e40 20 rados->read ofs=0 len=0 2023-10-05T19:28:55.381 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.374+0000 7fc9e0230e40 1 -- 172.21.15.167:0/2053399812 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:78 5.f 5:f4c53578:::zonegroups_names.default:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32) v8 -- 0x56062361a220 con 0x5606235e66f0 2023-10-05T19:28:55.383 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.374+0000 7fc9de566700 1 -- 172.21.15.167:0/2053399812 <== osd.1 v2:172.21.15.167:6800/30035 28 ==== osd_op_reply(78 zonegroups_names.default [read 0~46 out=46b] v0'0 uv2 ondisk = 0) v8 ==== 168+0+46 (crc 0 0 0) 0x7fc9d002f080 con 0x5606235e66f0 2023-10-05T19:28:55.383 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.374+0000 7fc9e0230e40 20 rados_obj.operate() r=0 bl.length=46 2023-10-05T19:28:55.383 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.374+0000 7fc9e0230e40 20 rados->read ofs=0 len=0 2023-10-05T19:28:55.384 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.374+0000 7fc9e0230e40 1 -- 172.21.15.167:0/2053399812 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:79 5.c 5:30d299a5:::zonegroup_info.4f1f719d-9648-417b-b10c-e5d939825419:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32) v8 -- 0x56062361b2d0 con 0x7fc9d4007030 2023-10-05T19:28:55.384 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.374+0000 7fc9de566700 1 -- 172.21.15.167:0/2053399812 <== osd.0 v2:172.21.15.118:6800/27690 20 ==== osd_op_reply(79 zonegroup_info.4f1f719d-9648-417b-b10c-e5d939825419 [read 0~362 out=362b] v0'0 uv1 ondisk = 0) v8 ==== 195+0+362 (crc 0 0 0) 0x7fc9d002f080 con 0x7fc9d4007030 2023-10-05T19:28:55.384 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.374+0000 7fc9e0230e40 20 rados_obj.operate() r=0 bl.length=362 2023-10-05T19:28:55.384 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.374+0000 7fc9e0230e40 20 rados->read ofs=0 len=0 2023-10-05T19:28:55.384 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.374+0000 7fc9e0230e40 1 -- 172.21.15.167:0/2053399812 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:80 5.3 5:c0ad8cbd:::region_map:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32) v8 -- 0x56062361b2d0 con 0x7fc9d4007030 2023-10-05T19:28:55.384 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.378+0000 7fc9de566700 1 -- 172.21.15.167:0/2053399812 <== osd.0 v2:172.21.15.118:6800/27690 21 ==== osd_op_reply(80 region_map [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 ==== 154+0+0 (crc 0 0 0) 0x7fc9d0050040 con 0x7fc9d4007030 2023-10-05T19:28:55.384 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.378+0000 7fc9e0230e40 20 rados_obj.operate() r=-2 bl.length=0 2023-10-05T19:28:55.384 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.378+0000 7fc9e0230e40 20 zone default found 2023-10-05T19:28:55.384 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.378+0000 7fc9e0230e40 4 Realm: () 2023-10-05T19:28:55.384 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.378+0000 7fc9e0230e40 4 ZoneGroup: default (4f1f719d-9648-417b-b10c-e5d939825419) 2023-10-05T19:28:55.385 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.378+0000 7fc9e0230e40 4 Zone: default (9a9df965-6612-45d4-a45f-6e3c889bacc2) 2023-10-05T19:28:55.385 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.378+0000 7fc9e0230e40 10 cannot find current period zonegroup using local zonegroup configuration 2023-10-05T19:28:55.385 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.378+0000 7fc9e0230e40 20 zonegroup default 2023-10-05T19:28:55.385 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.378+0000 7fc9e0230e40 20 rados->read ofs=0 len=0 2023-10-05T19:28:55.385 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.378+0000 7fc9e0230e40 1 -- 172.21.15.167:0/2053399812 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:81 5.3 5:c52100b6:::period_config.default:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32) v8 -- 0x56062361b2d0 con 0x7fc9d4007030 2023-10-05T19:28:55.385 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.378+0000 7fc9de566700 1 -- 172.21.15.167:0/2053399812 <== osd.0 v2:172.21.15.118:6800/27690 22 ==== osd_op_reply(81 period_config.default [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 ==== 165+0+0 (crc 0 0 0) 0x7fc9d0050040 con 0x7fc9d4007030 2023-10-05T19:28:55.385 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.378+0000 7fc9e0230e40 20 rados_obj.operate() r=-2 bl.length=0 2023-10-05T19:28:55.385 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.378+0000 7fc9e0230e40 20 rados->read ofs=0 len=0 2023-10-05T19:28:55.385 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.378+0000 7fc9e0230e40 1 -- 172.21.15.167:0/2053399812 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:82 6.2 6:4347d321:::bucket.sync-source-hints.:head [call version.read in=11b,read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32) v8 -- 0x56062361cc50 con 0x560623617470 2023-10-05T19:28:55.386 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.378+0000 7fc9ddd65700 1 -- 172.21.15.167:0/2053399812 <== osd.3 v2:172.21.15.167:6808/30036 20 ==== osd_op_reply(82 bucket.sync-source-hints. [call,read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 ==== 211+0+0 (crc 0 0 0) 0x7fc9d8025020 con 0x560623617470 2023-10-05T19:28:55.386 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.378+0000 7fc9e0230e40 20 rados_obj.operate() r=-2 bl.length=0 2023-10-05T19:28:55.386 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.378+0000 7fc9e0230e40 20 rados->read ofs=0 len=0 2023-10-05T19:28:55.387 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.378+0000 7fc9e0230e40 1 -- 172.21.15.167:0/2053399812 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:83 6.b 6:d467b91b:::bucket.sync-target-hints.:head [call version.read in=11b,read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32) v8 -- 0x56062361d000 con 0x560623617470 2023-10-05T19:28:55.387 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.378+0000 7fc9ddd65700 1 -- 172.21.15.167:0/2053399812 <== osd.3 v2:172.21.15.167:6808/30036 21 ==== osd_op_reply(83 bucket.sync-target-hints. [call,read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 ==== 211+0+0 (crc 0 0 0) 0x7fc9d8025020 con 0x560623617470 2023-10-05T19:28:55.387 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.378+0000 7fc9e0230e40 20 rados_obj.operate() r=-2 bl.length=0 2023-10-05T19:28:55.387 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.378+0000 7fc9e0230e40 20 started sync module instance, tier type = 2023-10-05T19:28:55.387 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.378+0000 7fc9e0230e40 20 started zone id=9a9df965-6612-45d4-a45f-6e3c889bacc2 (name=default) with tier type = 2023-10-05T19:28:55.387 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.378+0000 7fc9e0230e40 1 -- 172.21.15.167:0/2053399812 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:84 7.1f 7:f95f44c2:::notify.0:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e32) v8 -- 0x56062361c310 con 0x7fc9d801ec10 2023-10-05T19:28:55.388 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.382+0000 7fc9dd564700 1 -- 172.21.15.167:0/2053399812 <== osd.2 v2:172.21.15.167:6816/30037 13 ==== osd_op_reply(84 notify.0 [create] v32'195 uv195 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7fc9d4012450 con 0x7fc9d801ec10 2023-10-05T19:28:55.388 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.382+0000 7fc9e0230e40 1 -- 172.21.15.167:0/2053399812 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:85 7.1f 7:f95f44c2:::notify.0:head [watch watch cookie 94584363400736] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e32) v8 -- 0x56062361d3b0 con 0x7fc9d801ec10 2023-10-05T19:28:55.388 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.382+0000 7fc9e0230e40 1 -- 172.21.15.167:0/2053399812 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:86 7.0 7:05bf5b68:::notify.1:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e32) v8 -- 0x56062361dcb0 con 0x5606235e66f0 2023-10-05T19:28:55.391 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.386+0000 7fc9dd564700 1 -- 172.21.15.167:0/2053399812 <== osd.2 v2:172.21.15.167:6816/30037 14 ==== osd_op_reply(85 notify.0 [watch watch cookie 94584363400736] v32'196 uv195 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7fc9d4012450 con 0x7fc9d801ec10 2023-10-05T19:28:55.391 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.386+0000 7fc9de566700 1 -- 172.21.15.167:0/2053399812 <== osd.1 v2:172.21.15.167:6800/30035 29 ==== osd_op_reply(86 notify.1 [create] v32'195 uv195 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7fc9d0050040 con 0x5606235e66f0 2023-10-05T19:28:55.391 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.386+0000 7fc9e0230e40 1 -- 172.21.15.167:0/2053399812 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:87 7.0 7:05bf5b68:::notify.1:head [watch watch cookie 94584363407296] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e32) v8 -- 0x56062361dc10 con 0x5606235e66f0 2023-10-05T19:28:55.391 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.386+0000 7fc9e0230e40 1 -- 172.21.15.167:0/2053399812 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:88 7.15 7:a93a5511:::notify.2:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e32) v8 -- 0x56062361ea90 con 0x560623617470 2023-10-05T19:28:55.394 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.390+0000 7fc9de566700 1 -- 172.21.15.167:0/2053399812 <== osd.1 v2:172.21.15.167:6800/30035 30 ==== osd_op_reply(87 notify.1 [watch watch cookie 94584363407296] v32'196 uv195 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7fc9d0050040 con 0x5606235e66f0 2023-10-05T19:28:55.395 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.390+0000 7fc9ddd65700 1 -- 172.21.15.167:0/2053399812 <== osd.3 v2:172.21.15.167:6808/30036 22 ==== osd_op_reply(88 notify.2 [create] v32'195 uv195 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7fc9d8025020 con 0x560623617470 2023-10-05T19:28:55.395 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.390+0000 7fc9e0230e40 1 -- 172.21.15.167:0/2053399812 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:89 7.15 7:a93a5511:::notify.2:head [watch watch cookie 94584363409584] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e32) v8 -- 0x56062361dc10 con 0x560623617470 2023-10-05T19:28:55.395 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.390+0000 7fc9e0230e40 1 -- 172.21.15.167:0/2053399812 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:90 7.e 7:7759931f:::notify.3:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e32) v8 -- 0x56062361f5a0 con 0x7fc9d801ec10 2023-10-05T19:28:55.397 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.390+0000 7fc9dd564700 1 -- 172.21.15.167:0/2053399812 <== osd.2 v2:172.21.15.167:6816/30037 15 ==== osd_op_reply(90 notify.3 [create] v32'195 uv195 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7fc9d4012450 con 0x7fc9d801ec10 2023-10-05T19:28:55.398 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.390+0000 7fc9e0230e40 1 -- 172.21.15.167:0/2053399812 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:91 7.e 7:7759931f:::notify.3:head [watch watch cookie 94584363415968] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e32) v8 -- 0x5606236202e0 con 0x7fc9d801ec10 2023-10-05T19:28:55.398 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.394+0000 7fc9e0230e40 1 -- 172.21.15.167:0/2053399812 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:92 7.d 7:b4812045:::notify.4:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e32) v8 -- 0x560623620bc0 con 0x5606235e66f0 2023-10-05T19:28:55.398 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.394+0000 7fc9ddd65700 1 -- 172.21.15.167:0/2053399812 <== osd.3 v2:172.21.15.167:6808/30036 23 ==== osd_op_reply(89 notify.2 [watch watch cookie 94584363409584] v32'196 uv195 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7fc9d8025020 con 0x560623617470 2023-10-05T19:28:55.400 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.394+0000 7fc9dd564700 1 -- 172.21.15.167:0/2053399812 <== osd.2 v2:172.21.15.167:6816/30037 16 ==== osd_op_reply(91 notify.3 [watch watch cookie 94584363415968] v32'196 uv195 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7fc9d4012450 con 0x7fc9d801ec10 2023-10-05T19:28:55.400 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.394+0000 7fc9de566700 1 -- 172.21.15.167:0/2053399812 <== osd.1 v2:172.21.15.167:6800/30035 31 ==== osd_op_reply(92 notify.4 [create] v32'195 uv195 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7fc9d0050040 con 0x5606235e66f0 2023-10-05T19:28:55.401 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.394+0000 7fc9e0230e40 1 -- 172.21.15.167:0/2053399812 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:93 7.d 7:b4812045:::notify.4:head [watch watch cookie 94584363417072] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e32) v8 -- 0x560623620690 con 0x5606235e66f0 2023-10-05T19:28:55.401 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.394+0000 7fc9e0230e40 1 -- 172.21.15.167:0/2053399812 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:94 7.3 7:c609908c:::notify.5:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e32) v8 -- 0x5606236214b0 con 0x7fc9d4007030 2023-10-05T19:28:55.403 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.398+0000 7fc9de566700 1 -- 172.21.15.167:0/2053399812 <== osd.1 v2:172.21.15.167:6800/30035 32 ==== osd_op_reply(93 notify.4 [watch watch cookie 94584363417072] v32'196 uv195 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7fc9d0050040 con 0x5606235e66f0 2023-10-05T19:28:55.403 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.398+0000 7fc9de566700 1 -- 172.21.15.167:0/2053399812 <== osd.0 v2:172.21.15.118:6800/27690 23 ==== osd_op_reply(94 notify.5 [create] v32'195 uv195 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7fc9d0050040 con 0x7fc9d4007030 2023-10-05T19:28:55.403 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.398+0000 7fc9e0230e40 1 -- 172.21.15.167:0/2053399812 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:95 7.3 7:c609908c:::notify.5:head [watch watch cookie 94584363410688] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e32) v8 -- 0x560623621460 con 0x7fc9d4007030 2023-10-05T19:28:55.404 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.398+0000 7fc9e0230e40 1 -- 172.21.15.167:0/2053399812 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:96 7.14 7:2b04a3e9:::notify.6:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e32) v8 -- 0x560623621e70 con 0x7fc9d4007030 2023-10-05T19:28:55.406 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.402+0000 7fc9de566700 1 -- 172.21.15.167:0/2053399812 <== osd.0 v2:172.21.15.118:6800/27690 24 ==== osd_op_reply(96 notify.6 [create] v32'195 uv195 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7fc9d0050040 con 0x7fc9d4007030 2023-10-05T19:28:55.406 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.402+0000 7fc9e0230e40 1 -- 172.21.15.167:0/2053399812 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:97 7.14 7:2b04a3e9:::notify.6:head [watch watch cookie 94584363424128] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e32) v8 -- 0x5606236222c0 con 0x7fc9d4007030 2023-10-05T19:28:55.406 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.402+0000 7fc9e0230e40 1 -- 172.21.15.167:0/2053399812 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:98 7.9 7:93e5b521:::notify.7:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e32) v8 -- 0x560623622cd0 con 0x7fc9d4007030 2023-10-05T19:28:55.406 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.402+0000 7fc9de566700 1 -- 172.21.15.167:0/2053399812 <== osd.0 v2:172.21.15.118:6800/27690 25 ==== osd_op_reply(95 notify.5 [watch watch cookie 94584363410688] v32'196 uv195 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7fc9d0050040 con 0x7fc9d4007030 2023-10-05T19:28:55.409 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.402+0000 7fc9de566700 1 -- 172.21.15.167:0/2053399812 <== osd.0 v2:172.21.15.118:6800/27690 26 ==== osd_op_reply(98 notify.7 [create] v32'195 uv195 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7fc9d0050040 con 0x7fc9d4007030 2023-10-05T19:28:55.409 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.402+0000 7fc9de566700 1 -- 172.21.15.167:0/2053399812 <== osd.0 v2:172.21.15.118:6800/27690 27 ==== osd_op_reply(97 notify.6 [watch watch cookie 94584363424128] v32'196 uv195 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7fc9d0050040 con 0x7fc9d4007030 2023-10-05T19:28:55.409 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.402+0000 7fc9e0230e40 1 -- 172.21.15.167:0/2053399812 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:99 7.9 7:93e5b521:::notify.7:head [watch watch cookie 94584363421552] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e32) v8 -- 0x5606236227e0 con 0x7fc9d4007030 2023-10-05T19:28:55.409 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.402+0000 7fc9e0230e40 20 add_watcher() i=0 2023-10-05T19:28:55.409 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.402+0000 7fc9e0230e40 20 add_watcher() i=1 2023-10-05T19:28:55.409 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.402+0000 7fc9e0230e40 20 add_watcher() i=2 2023-10-05T19:28:55.409 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.402+0000 7fc9e0230e40 20 add_watcher() i=3 2023-10-05T19:28:55.410 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.402+0000 7fc9e0230e40 20 add_watcher() i=4 2023-10-05T19:28:55.410 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.402+0000 7fc9e0230e40 20 add_watcher() i=5 2023-10-05T19:28:55.410 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.402+0000 7fc9e0230e40 20 add_watcher() i=6 2023-10-05T19:28:55.411 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.406+0000 7fc9de566700 1 -- 172.21.15.167:0/2053399812 <== osd.0 v2:172.21.15.118:6800/27690 28 ==== osd_op_reply(99 notify.7 [watch watch cookie 94584363421552] v32'196 uv195 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7fc9d0050040 con 0x7fc9d4007030 2023-10-05T19:28:55.412 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.406+0000 7fc9e0230e40 20 add_watcher() i=7 2023-10-05T19:28:55.412 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.406+0000 7fc9e0230e40 2 all 8 watchers are set, enabling cache 2023-10-05T19:28:55.412 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.406+0000 7fc9e0230e40 1 -- 172.21.15.167:0/2053399812 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:100 6.4 6:22d26bf9:::data_loggenerations_metadata:head [call version.check_conds in=74b,call version.read in=11b,read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32) v8 -- 0x5606236258c0 con 0x5606235e66f0 2023-10-05T19:28:55.412 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.406+0000 7fc9de566700 1 -- 172.21.15.167:0/2053399812 <== osd.1 v2:172.21.15.167:6800/30035 33 ==== osd_op_reply(100 data_loggenerations_metadata [call,call out=48b,read 0~28 out=28b] v0'0 uv1 ondisk = 0) v8 ==== 256+0+76 (crc 0 0 0) 0x7fc9d0050040 con 0x5606235e66f0 2023-10-05T19:28:55.413 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.406+0000 7fc9e0230e40 1 -- 172.21.15.167:0/2053399812 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:101 6.4 6:22d26bf9:::data_loggenerations_metadata:head [watch watch cookie 94584363425232] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e32) v8 -- 0x560623622cd0 con 0x5606235e66f0 2023-10-05T19:28:55.415 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.410+0000 7fc9de566700 1 -- 172.21.15.167:0/2053399812 <== osd.1 v2:172.21.15.167:6800/30035 34 ==== osd_op_reply(101 data_loggenerations_metadata [watch watch cookie 94584363425232] v32'191 uv1 ondisk = 0) v8 ==== 172+0+0 (crc 0 0 0) 0x7fc9d0050040 con 0x5606235e66f0 2023-10-05T19:28:55.415 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.410+0000 7fc9aefe5700 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2023-10-05T19:28:55.415 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.410+0000 7fc9e0230e40 20 check_secure_mon_conn(): auth registy supported: methods=[2] modes=[2,1] 2023-10-05T19:28:55.416 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.410+0000 7fc9e0230e40 20 check_secure_mon_conn(): mode 1 is insecure 2023-10-05T19:28:55.416 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.410+0000 7fc9e0230e40 1 Processor -- start 2023-10-05T19:28:55.416 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.410+0000 7fc9e0230e40 1 -- start start 2023-10-05T19:28:55.417 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.410+0000 7fc9e0230e40 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x56062362fd70 0x560623630140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-05T19:28:55.417 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.410+0000 7fc9e0230e40 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x560623630680 con 0x56062362fd70 2023-10-05T19:28:55.417 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.410+0000 7fc9dd564700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x56062362fd70 0x560623630140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-05T19:28:55.417 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.410+0000 7fc9dd564700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x56062362fd70 0x560623630140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.118:3300/0 says I am v2:172.21.15.167:53240/0 (socket says 172.21.15.167:53240) 2023-10-05T19:28:55.417 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.410+0000 7fc9dd564700 1 -- 172.21.15.167:0/541688592 learned_addr learned my addr 172.21.15.167:0/541688592 (peer_addr_for_me v2:172.21.15.167:0/0) 2023-10-05T19:28:55.417 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.410+0000 7fc9dd564700 1 -- 172.21.15.167:0/541688592 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5606236307c0 con 0x56062362fd70 2023-10-05T19:28:55.418 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.410+0000 7fc9dd564700 1 --2- 172.21.15.167:0/541688592 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x56062362fd70 0x560623630140 secure :-1 s=READY pgs=528 cs=0 l=1 rev1=1 crypto rx=0x560623553c60 tx=0x7fc9d4012000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=de91cbbc7169cec7 server_cookie=0 in_seq=0 out_seq=0 2023-10-05T19:28:55.418 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.414+0000 7fc9ae7e4700 1 -- 172.21.15.167:0/541688592 <== mon.0 v2:172.21.15.118:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fc9d402a5c0 con 0x56062362fd70 2023-10-05T19:28:55.418 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.414+0000 7fc9ae7e4700 1 -- 172.21.15.167:0/541688592 <== mon.0 v2:172.21.15.118:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc9d402a720 con 0x56062362fd70 2023-10-05T19:28:55.418 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.414+0000 7fc9ae7e4700 1 -- 172.21.15.167:0/541688592 <== mon.0 v2:172.21.15.118:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fc9d402a910 con 0x56062362fd70 2023-10-05T19:28:55.419 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.414+0000 7fc9e0230e40 1 -- 172.21.15.167:0/541688592 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x56062362fd70 msgr2=0x560623630140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-05T19:28:55.419 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.414+0000 7fc9e0230e40 1 --2- 172.21.15.167:0/541688592 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x56062362fd70 0x560623630140 secure :-1 s=READY pgs=528 cs=0 l=1 rev1=1 crypto rx=0x560623553c60 tx=0x7fc9d4012000 comp rx=0 tx=0).stop 2023-10-05T19:28:55.420 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.414+0000 7fc9e0230e40 1 -- 172.21.15.167:0/541688592 shutdown_connections 2023-10-05T19:28:55.420 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.414+0000 7fc9e0230e40 1 --2- 172.21.15.167:0/541688592 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x56062362fd70 0x560623630140 unknown :-1 s=CLOSED pgs=528 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-05T19:28:55.420 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.414+0000 7fc9e0230e40 1 -- 172.21.15.167:0/541688592 >> 172.21.15.167:0/541688592 conn(0x56062362b130 msgr2=0x5606236292e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-05T19:28:55.420 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.414+0000 7fc9e0230e40 1 -- 172.21.15.167:0/541688592 shutdown_connections 2023-10-05T19:28:55.420 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.414+0000 7fc9e0230e40 1 -- 172.21.15.167:0/541688592 wait complete. 2023-10-05T19:28:55.421 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.414+0000 7fc9e0230e40 1 Processor -- start 2023-10-05T19:28:55.421 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.414+0000 7fc9e0230e40 1 -- start start 2023-10-05T19:28:55.421 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.414+0000 7fc9e0230e40 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x56062362fd70 0x5606236358a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-05T19:28:55.422 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.414+0000 7fc9e0230e40 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x5606236368a0 con 0x56062362fd70 2023-10-05T19:28:55.422 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.414+0000 7fc9dd564700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x56062362fd70 0x5606236358a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-05T19:28:55.422 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.414+0000 7fc9dd564700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x56062362fd70 0x5606236358a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.118:3300/0 says I am v2:172.21.15.167:53250/0 (socket says 172.21.15.167:53250) 2023-10-05T19:28:55.422 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.414+0000 7fc9dd564700 1 -- 172.21.15.167:0/1093196717 learned_addr learned my addr 172.21.15.167:0/1093196717 (peer_addr_for_me v2:172.21.15.167:0/0) 2023-10-05T19:28:55.422 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.414+0000 7fc9dd564700 1 -- 172.21.15.167:0/1093196717 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5606235547a0 con 0x56062362fd70 2023-10-05T19:28:55.422 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.418+0000 7fc9dd564700 1 --2- 172.21.15.167:0/1093196717 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x56062362fd70 0x5606236358a0 secure :-1 s=READY pgs=529 cs=0 l=1 rev1=1 crypto rx=0x560623629e50 tx=0x7fc9d402f000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-05T19:28:55.423 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.418+0000 7fc9acfe1700 1 -- 172.21.15.167:0/1093196717 <== mon.0 v2:172.21.15.118:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fc9d403e950 con 0x56062362fd70 2023-10-05T19:28:55.423 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.418+0000 7fc9acfe1700 1 -- 172.21.15.167:0/1093196717 <== mon.0 v2:172.21.15.118:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc9d403eab0 con 0x56062362fd70 2023-10-05T19:28:55.423 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.418+0000 7fc9e0230e40 1 -- 172.21.15.167:0/1093196717 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x560623635ea0 con 0x56062362fd70 2023-10-05T19:28:55.423 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.418+0000 7fc9acfe1700 1 -- 172.21.15.167:0/1093196717 <== mon.0 v2:172.21.15.118:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fc9d403ec80 con 0x56062362fd70 2023-10-05T19:28:55.423 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.418+0000 7fc9e0230e40 1 -- 172.21.15.167:0/1093196717 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x5606236362e0 con 0x56062362fd70 2023-10-05T19:28:55.424 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.418+0000 7fc9acfe1700 1 -- 172.21.15.167:0/1093196717 <== mon.0 v2:172.21.15.118:3300/0 4 ==== mgrmap(e 5) v1 ==== 40329+0+0 (secure 0 0 0) 0x7fc9d4046070 con 0x56062362fd70 2023-10-05T19:28:55.425 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.418+0000 7fc9acfe1700 1 --2- 172.21.15.167:0/1093196717 >> [v2:172.21.15.167:6824/29889,v1:172.21.15.167:6825/29889] conn(0x7fc924030c50 0x7fc924033100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-05T19:28:55.425 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.418+0000 7fc9acfe1700 1 -- 172.21.15.167:0/1093196717 <== mon.0 v2:172.21.15.118:3300/0 5 ==== osd_map(32..32 src has 1..32) v4 ==== 6260+0+0 (secure 0 0 0) 0x7fc9d402f070 con 0x56062362fd70 2023-10-05T19:28:55.425 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.418+0000 7fc9de566700 1 --2- 172.21.15.167:0/1093196717 >> [v2:172.21.15.167:6824/29889,v1:172.21.15.167:6825/29889] conn(0x7fc924030c50 0x7fc924033100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-05T19:28:55.425 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.418+0000 7fc9e0230e40 1 --2- 172.21.15.167:0/1093196717 >> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] conn(0x5606236404f0 0x560623642900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-05T19:28:55.425 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.418+0000 7fc9e0230e40 1 -- 172.21.15.167:0/1093196717 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:1 6.19 6:9a6dd8c0:gc::gc.0:head [create,call version.check_conds in=74b,call rgw_gc.rgw_gc_queue_init in=45b,call version.set in=34b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e32) v8 -- 0x560623642f50 con 0x5606236404f0 2023-10-05T19:28:55.425 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.418+0000 7fc9ddd65700 1 --2- 172.21.15.167:0/1093196717 >> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] conn(0x5606236404f0 0x560623642900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-05T19:28:55.425 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.418+0000 7fc9de566700 1 --2- 172.21.15.167:0/1093196717 >> [v2:172.21.15.167:6824/29889,v1:172.21.15.167:6825/29889] conn(0x7fc924030c50 0x7fc924033100 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x5606236258b0 tx=0x7fc9d0000f90 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-05T19:28:55.426 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.418+0000 7fc9ddd65700 1 --2- 172.21.15.167:0/1093196717 >> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] conn(0x5606236404f0 0x560623642900 crc :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-05T19:28:55.428 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.422+0000 7fc9ddd65700 1 -- 172.21.15.167:0/1093196717 <== osd.1 v2:172.21.15.167:6800/30035 1 ==== osd_op_reply(1 gc.0 [create,call,call,call] v32'34 uv33 ondisk = -125 ((125) Operation canceled)) v8 ==== 274+0+0 (crc 0 0 0) 0x7fc9d8025020 con 0x5606236404f0 2023-10-05T19:28:55.428 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.422+0000 7fc9e0230e40 1 --2- 172.21.15.167:0/1093196717 >> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] conn(0x56062363c0d0 0x5606236465c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-05T19:28:55.428 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.422+0000 7fc9e0230e40 1 -- 172.21.15.167:0/1093196717 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:2 6.18 6:1e85421f:gc::gc.1:head [create,call version.check_conds in=74b,call rgw_gc.rgw_gc_queue_init in=45b,call version.set in=34b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e32) v8 -- 0x560623646b00 con 0x56062363c0d0 2023-10-05T19:28:55.428 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.422+0000 7fc9dd564700 1 --2- 172.21.15.167:0/1093196717 >> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] conn(0x56062363c0d0 0x5606236465c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-05T19:28:55.429 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.422+0000 7fc9dd564700 1 --2- 172.21.15.167:0/1093196717 >> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] conn(0x56062363c0d0 0x5606236465c0 crc :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-05T19:28:55.431 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.426+0000 7fc9dd564700 1 -- 172.21.15.167:0/1093196717 <== osd.0 v2:172.21.15.118:6800/27690 1 ==== osd_op_reply(2 gc.1 [create,call,call,call] v32'26 uv25 ondisk = -125 ((125) Operation canceled)) v8 ==== 274+0+0 (crc 0 0 0) 0x7fc9d4032040 con 0x56062363c0d0 2023-10-05T19:28:55.431 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.426+0000 7fc9e0230e40 1 --2- 172.21.15.167:0/1093196717 >> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] conn(0x5606236477d0 0x560623649bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-05T19:28:55.431 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.426+0000 7fc9e0230e40 1 -- 172.21.15.167:0/1093196717 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:3 6.b 6:d242335b:gc::gc.2:head [create,call version.check_conds in=74b,call rgw_gc.rgw_gc_queue_init in=45b,call version.set in=34b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e32) v8 -- 0x56062364a2b0 con 0x5606236477d0 2023-10-05T19:28:55.432 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.426+0000 7fc9de566700 1 --2- 172.21.15.167:0/1093196717 >> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] conn(0x5606236477d0 0x560623649bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-05T19:28:55.432 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.426+0000 7fc9de566700 1 --2- 172.21.15.167:0/1093196717 >> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] conn(0x5606236477d0 0x560623649bb0 crc :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-05T19:28:55.434 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.430+0000 7fc9de566700 1 -- 172.21.15.167:0/1093196717 <== osd.3 v2:172.21.15.167:6808/30036 1 ==== osd_op_reply(3 gc.2 [create,call,call,call] v32'22 uv21 ondisk = -125 ((125) Operation canceled)) v8 ==== 274+0+0 (crc 0 0 0) 0x7fc9d0050040 con 0x5606236477d0 2023-10-05T19:28:55.435 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.430+0000 7fc9e0230e40 1 -- 172.21.15.167:0/1093196717 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:4 6.4 6:2166d8e1:gc::gc.3:head [create,call version.check_conds in=74b,call rgw_gc.rgw_gc_queue_init in=45b,call version.set in=34b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e32) v8 -- 0x56062364a8d0 con 0x5606236404f0 2023-10-05T19:28:55.437 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.434+0000 7fc9ddd65700 1 -- 172.21.15.167:0/1093196717 <== osd.1 v2:172.21.15.167:6800/30035 2 ==== osd_op_reply(4 gc.3 [create,call,call,call] v32'192 uv184 ondisk = -125 ((125) Operation canceled)) v8 ==== 274+0+0 (crc 0 0 0) 0x7fc9d8025020 con 0x5606236404f0 2023-10-05T19:28:55.438 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.434+0000 7fc9e0230e40 1 --2- 172.21.15.167:0/1093196717 >> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] conn(0x56062364b8f0 0x56062364dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-05T19:28:55.438 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.434+0000 7fc9e0230e40 1 -- 172.21.15.167:0/1093196717 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:5 6.3 6:c60da066:gc::gc.4:head [create,call version.check_conds in=74b,call rgw_gc.rgw_gc_queue_init in=45b,call version.set in=34b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e32) v8 -- 0x56062364e380 con 0x56062364b8f0 2023-10-05T19:28:55.438 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.434+0000 7fc9ddd65700 1 --2- 172.21.15.167:0/1093196717 >> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] conn(0x56062364b8f0 0x56062364dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-05T19:28:55.438 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.434+0000 7fc9ddd65700 1 --2- 172.21.15.167:0/1093196717 >> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] conn(0x56062364b8f0 0x56062364dcd0 crc :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-05T19:28:55.441 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.434+0000 7fc9ddd65700 1 -- 172.21.15.167:0/1093196717 <== osd.2 v2:172.21.15.167:6816/30037 1 ==== osd_op_reply(5 gc.4 [create,call,call,call] v32'56 uv51 ondisk = -125 ((125) Operation canceled)) v8 ==== 274+0+0 (crc 0 0 0) 0x7fc9d8025020 con 0x56062364b8f0 2023-10-05T19:28:55.441 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.434+0000 7fc9e0230e40 1 -- 172.21.15.167:0/1093196717 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:6 6.9 6:954be8d2:gc::gc.5:head [create,call version.check_conds in=74b,call rgw_gc.rgw_gc_queue_init in=45b,call version.set in=34b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e32) v8 -- 0x56062364e9d0 con 0x5606236477d0 2023-10-05T19:28:55.444 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.438+0000 7fc9de566700 1 -- 172.21.15.167:0/1093196717 <== osd.3 v2:172.21.15.167:6808/30036 2 ==== osd_op_reply(6 gc.5 [create,call,call,call] v32'40 uv37 ondisk = -125 ((125) Operation canceled)) v8 ==== 274+0+0 (crc 0 0 0) 0x7fc9d0050040 con 0x5606236477d0 2023-10-05T19:28:55.444 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.438+0000 7fc9e0230e40 1 -- 172.21.15.167:0/1093196717 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:7 6.e 6:748722ff:gc::gc.6:head [create,call version.check_conds in=74b,call rgw_gc.rgw_gc_queue_init in=45b,call version.set in=34b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e32) v8 -- 0x56062364eff0 con 0x5606236404f0 2023-10-05T19:28:55.446 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.442+0000 7fc9ddd65700 1 -- 172.21.15.167:0/1093196717 <== osd.1 v2:172.21.15.167:6800/30035 3 ==== osd_op_reply(7 gc.6 [create,call,call,call] v32'24 uv23 ondisk = -125 ((125) Operation canceled)) v8 ==== 274+0+0 (crc 0 0 0) 0x7fc9d8025020 con 0x5606236404f0 2023-10-05T19:28:55.446 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.442+0000 7fc9e0230e40 1 -- 172.21.15.167:0/1093196717 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:8 6.1d 6:beaa4be4:gc::gc.7:head [create,call version.check_conds in=74b,call rgw_gc.rgw_gc_queue_init in=45b,call version.set in=34b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e32) v8 -- 0x56062364f4e0 con 0x56062363c0d0 2023-10-05T19:28:55.449 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.442+0000 7fc9dd564700 1 -- 172.21.15.167:0/1093196717 <== osd.0 v2:172.21.15.118:6800/27690 2 ==== osd_op_reply(8 gc.7 [create,call,call,call] v32'37 uv34 ondisk = -125 ((125) Operation canceled)) v8 ==== 274+0+0 (crc 0 0 0) 0x7fc9d4032040 con 0x56062363c0d0 2023-10-05T19:28:55.449 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.442+0000 7fc9e0230e40 1 -- 172.21.15.167:0/1093196717 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:9 6.12 6:4f78e9a2:gc::gc.8:head [create,call version.check_conds in=74b,call rgw_gc.rgw_gc_queue_init in=45b,call version.set in=34b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e32) v8 -- 0x56062363f010 con 0x56062364b8f0 2023-10-05T19:28:55.452 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.446+0000 7fc9ddd65700 1 -- 172.21.15.167:0/1093196717 <== osd.2 v2:172.21.15.167:6816/30037 2 ==== osd_op_reply(9 gc.8 [create,call,call,call] v32'22 uv21 ondisk = -125 ((125) Operation canceled)) v8 ==== 274+0+0 (crc 0 0 0) 0x7fc9d8025020 con 0x56062364b8f0 2023-10-05T19:28:55.452 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.446+0000 7fc9e0230e40 1 -- 172.21.15.167:0/1093196717 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:10 6.1c 6:3e426232:gc::gc.9:head [create,call version.check_conds in=74b,call rgw_gc.rgw_gc_queue_init in=45b,call version.set in=34b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e32) v8 -- 0x56062363f010 con 0x5606236477d0 2023-10-05T19:28:55.454 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.450+0000 7fc9de566700 1 -- 172.21.15.167:0/1093196717 <== osd.3 v2:172.21.15.167:6808/30036 3 ==== osd_op_reply(10 gc.9 [create,call,call,call] v32'40 uv39 ondisk = -125 ((125) Operation canceled)) v8 ==== 274+0+0 (crc 0 0 0) 0x7fc9d0050040 con 0x5606236477d0 2023-10-05T19:28:55.455 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.450+0000 7fc9e0230e40 1 -- 172.21.15.167:0/1093196717 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:11 6.3 6:c31f41d7:gc::gc.10:head [create,call version.check_conds in=74b,call rgw_gc.rgw_gc_queue_init in=45b,call version.set in=34b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e32) v8 -- 0x560623650000 con 0x56062364b8f0 2023-10-05T19:28:55.457 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.450+0000 7fc9ddd65700 1 -- 172.21.15.167:0/1093196717 <== osd.2 v2:172.21.15.167:6816/30037 3 ==== osd_op_reply(11 gc.10 [create,call,call,call] v32'57 uv53 ondisk = -125 ((125) Operation canceled)) v8 ==== 275+0+0 (crc 0 0 0) 0x7fc9d8025020 con 0x56062364b8f0 2023-10-05T19:28:55.457 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.450+0000 7fc9e0230e40 1 -- 172.21.15.167:0/1093196717 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:12 6.13 6:c8333aab:gc::gc.11:head [create,call version.check_conds in=74b,call rgw_gc.rgw_gc_queue_init in=45b,call version.set in=34b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e32) v8 -- 0x560623650000 con 0x56062363c0d0 2023-10-05T19:28:55.460 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.454+0000 7fc9dd564700 1 -- 172.21.15.167:0/1093196717 <== osd.0 v2:172.21.15.118:6800/27690 3 ==== osd_op_reply(12 gc.11 [create,call,call,call] v32'26 uv25 ondisk = -125 ((125) Operation canceled)) v8 ==== 275+0+0 (crc 0 0 0) 0x7fc9d4032040 con 0x56062363c0d0 2023-10-05T19:28:55.460 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.454+0000 7fc9e0230e40 1 -- 172.21.15.167:0/1093196717 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:13 6.15 6:ab569f81:gc::gc.12:head [create,call version.check_conds in=74b,call rgw_gc.rgw_gc_queue_init in=45b,call version.set in=34b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e32) v8 -- 0x5606236504f0 con 0x56062364b8f0 2023-10-05T19:28:55.463 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.458+0000 7fc9ddd65700 1 -- 172.21.15.167:0/1093196717 <== osd.2 v2:172.21.15.167:6816/30037 4 ==== osd_op_reply(13 gc.12 [create,call,call,call] v32'22 uv21 ondisk = -125 ((125) Operation canceled)) v8 ==== 275+0+0 (crc 0 0 0) 0x7fc9d8025020 con 0x56062364b8f0 2023-10-05T19:28:55.463 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.458+0000 7fc9e0230e40 1 -- 172.21.15.167:0/1093196717 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:14 6.0 6:0667bb13:gc::gc.13:head [create,call version.check_conds in=74b,call rgw_gc.rgw_gc_queue_init in=45b,call version.set in=34b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e32) v8 -- 0x5606236504f0 con 0x56062363c0d0 2023-10-05T19:28:55.465 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.462+0000 7fc9dd564700 1 -- 172.21.15.167:0/1093196717 <== osd.0 v2:172.21.15.118:6800/27690 4 ==== osd_op_reply(14 gc.13 [create,call,call,call] v32'43 uv40 ondisk = -125 ((125) Operation canceled)) v8 ==== 275+0+0 (crc 0 0 0) 0x7fc9d4032040 con 0x56062363c0d0 2023-10-05T19:28:55.466 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.462+0000 7fc9e0230e40 1 -- 172.21.15.167:0/1093196717 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:15 6.4 6:227f63b5:gc::gc.14:head [create,call version.check_conds in=74b,call rgw_gc.rgw_gc_queue_init in=45b,call version.set in=34b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e32) v8 -- 0x560623650b10 con 0x5606236404f0 2023-10-05T19:28:55.468 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.462+0000 7fc9ddd65700 1 -- 172.21.15.167:0/1093196717 <== osd.1 v2:172.21.15.167:6800/30035 4 ==== osd_op_reply(15 gc.14 [create,call,call,call] v32'193 uv186 ondisk = -125 ((125) Operation canceled)) v8 ==== 275+0+0 (crc 0 0 0) 0x7fc9d8025020 con 0x5606236404f0 2023-10-05T19:28:55.468 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.462+0000 7fc9e0230e40 1 -- 172.21.15.167:0/1093196717 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:16 6.7 6:e07a6548:gc::gc.15:head [create,call version.check_conds in=74b,call rgw_gc.rgw_gc_queue_init in=45b,call version.set in=34b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e32) v8 -- 0x560623650b10 con 0x5606236477d0 2023-10-05T19:28:55.471 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.466+0000 7fc9de566700 1 -- 172.21.15.167:0/1093196717 <== osd.3 v2:172.21.15.167:6808/30036 4 ==== osd_op_reply(16 gc.15 [create,call,call,call] v32'50 uv45 ondisk = -125 ((125) Operation canceled)) v8 ==== 275+0+0 (crc 0 0 0) 0x7fc9d0050040 con 0x5606236477d0 2023-10-05T19:28:55.471 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.466+0000 7fc9e0230e40 1 -- 172.21.15.167:0/1093196717 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:17 6.3 6:c623b173:gc::gc.16:head [create,call version.check_conds in=74b,call rgw_gc.rgw_gc_queue_init in=45b,call version.set in=34b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e32) v8 -- 0x560623650b10 con 0x56062364b8f0 2023-10-05T19:28:55.474 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.470+0000 7fc9ddd65700 1 -- 172.21.15.167:0/1093196717 <== osd.2 v2:172.21.15.167:6816/30037 5 ==== osd_op_reply(17 gc.16 [create,call,call,call] v32'58 uv55 ondisk = -125 ((125) Operation canceled)) v8 ==== 275+0+0 (crc 0 0 0) 0x7fc9d8025020 con 0x56062364b8f0 2023-10-05T19:28:55.474 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.470+0000 7fc9e0230e40 1 -- 172.21.15.167:0/1093196717 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:18 6.16 6:6849fd06:gc::gc.17:head [create,call version.check_conds in=74b,call rgw_gc.rgw_gc_queue_init in=45b,call version.set in=34b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e32) v8 -- 0x560623650b10 con 0x5606236477d0 2023-10-05T19:28:55.477 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.470+0000 7fc9de566700 1 -- 172.21.15.167:0/1093196717 <== osd.3 v2:172.21.15.167:6808/30036 5 ==== osd_op_reply(18 gc.17 [create,call,call,call] v32'22 uv21 ondisk = -125 ((125) Operation canceled)) v8 ==== 275+0+0 (crc 0 0 0) 0x7fc9d0050040 con 0x5606236477d0 2023-10-05T19:28:55.477 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.470+0000 7fc9e0230e40 1 -- 172.21.15.167:0/1093196717 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:19 6.d 6:b019ea83:gc::gc.18:head [create,call version.check_conds in=74b,call rgw_gc.rgw_gc_queue_init in=45b,call version.set in=34b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e32) v8 -- 0x560623650b10 con 0x5606236404f0 2023-10-05T19:28:55.480 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.474+0000 7fc9ddd65700 1 -- 172.21.15.167:0/1093196717 <== osd.1 v2:172.21.15.167:6800/30035 5 ==== osd_op_reply(19 gc.18 [create,call,call,call] v32'30 uv29 ondisk = -125 ((125) Operation canceled)) v8 ==== 275+0+0 (crc 0 0 0) 0x7fc9d8025020 con 0x5606236404f0 2023-10-05T19:28:55.480 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.474+0000 7fc9e0230e40 1 -- 172.21.15.167:0/1093196717 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:20 6.4 6:24c8e1ae:gc::gc.19:head [create,call version.check_conds in=74b,call rgw_gc.rgw_gc_queue_init in=45b,call version.set in=34b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e32) v8 -- 0x560623650b10 con 0x5606236404f0 2023-10-05T19:28:55.482 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.478+0000 7fc9ddd65700 1 -- 172.21.15.167:0/1093196717 <== osd.1 v2:172.21.15.167:6800/30035 6 ==== osd_op_reply(20 gc.19 [create,call,call,call] v32'194 uv189 ondisk = -125 ((125) Operation canceled)) v8 ==== 275+0+0 (crc 0 0 0) 0x7fc9d8025020 con 0x5606236404f0 2023-10-05T19:28:55.482 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.478+0000 7fc9e0230e40 1 -- 172.21.15.167:0/1093196717 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:21 6.7 6:e622f76a:gc::gc.20:head [create,call version.check_conds in=74b,call rgw_gc.rgw_gc_queue_init in=45b,call version.set in=34b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e32) v8 -- 0x560623650b10 con 0x5606236477d0 2023-10-05T19:28:55.485 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.478+0000 7fc9de566700 1 -- 172.21.15.167:0/1093196717 <== osd.3 v2:172.21.15.167:6808/30036 6 ==== osd_op_reply(21 gc.20 [create,call,call,call] v32'51 uv47 ondisk = -125 ((125) Operation canceled)) v8 ==== 275+0+0 (crc 0 0 0) 0x7fc9d0050040 con 0x5606236477d0 2023-10-05T19:28:55.485 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.478+0000 7fc9e0230e40 1 -- 172.21.15.167:0/1093196717 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:22 6.a 6:5127e642:gc::gc.21:head [create,call version.check_conds in=74b,call rgw_gc.rgw_gc_queue_init in=45b,call version.set in=34b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e32) v8 -- 0x560623650b10 con 0x56062364b8f0 2023-10-05T19:28:55.487 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.482+0000 7fc9ddd65700 1 -- 172.21.15.167:0/1093196717 <== osd.2 v2:172.21.15.167:6816/30037 6 ==== osd_op_reply(22 gc.21 [create,call,call,call] v32'32 uv31 ondisk = -125 ((125) Operation canceled)) v8 ==== 275+0+0 (crc 0 0 0) 0x7fc9d8025020 con 0x56062364b8f0 2023-10-05T19:28:55.487 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.482+0000 7fc9e0230e40 1 -- 172.21.15.167:0/1093196717 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:23 6.11 6:8eb99c53:gc::gc.22:head [create,call version.check_conds in=74b,call rgw_gc.rgw_gc_queue_init in=45b,call version.set in=34b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e32) v8 -- 0x560623650b10 con 0x5606236477d0 2023-10-05T19:28:55.489 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.486+0000 7fc9de566700 1 -- 172.21.15.167:0/1093196717 <== osd.3 v2:172.21.15.167:6808/30036 7 ==== osd_op_reply(23 gc.22 [create,call,call,call] v32'24 uv23 ondisk = -125 ((125) Operation canceled)) v8 ==== 275+0+0 (crc 0 0 0) 0x7fc9d0050040 con 0x5606236477d0 2023-10-05T19:28:55.490 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.486+0000 7fc9e0230e40 1 -- 172.21.15.167:0/1093196717 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:24 6.8 6:17d7e8ed:gc::gc.23:head [create,call version.check_conds in=74b,call rgw_gc.rgw_gc_queue_init in=45b,call version.set in=34b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e32) v8 -- 0x560623650b10 con 0x5606236477d0 2023-10-05T19:28:55.492 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.486+0000 7fc9de566700 1 -- 172.21.15.167:0/1093196717 <== osd.3 v2:172.21.15.167:6808/30036 8 ==== osd_op_reply(24 gc.23 [create,call,call,call] v32'34 uv33 ondisk = -125 ((125) Operation canceled)) v8 ==== 275+0+0 (crc 0 0 0) 0x7fc9d0050040 con 0x5606236477d0 2023-10-05T19:28:55.492 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.486+0000 7fc9e0230e40 1 -- 172.21.15.167:0/1093196717 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:25 6.1b 6:db9d130d:gc::gc.24:head [create,call version.check_conds in=74b,call rgw_gc.rgw_gc_queue_init in=45b,call version.set in=34b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e32) v8 -- 0x560623650b10 con 0x5606236477d0 2023-10-05T19:28:55.494 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.490+0000 7fc9de566700 1 -- 172.21.15.167:0/1093196717 <== osd.3 v2:172.21.15.167:6808/30036 9 ==== osd_op_reply(25 gc.24 [create,call,call,call] v32'18 uv17 ondisk = -125 ((125) Operation canceled)) v8 ==== 275+0+0 (crc 0 0 0) 0x7fc9d0050040 con 0x5606236477d0 2023-10-05T19:28:55.494 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.490+0000 7fc9e0230e40 1 -- 172.21.15.167:0/1093196717 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:26 6.9 6:921ecdb1:gc::gc.25:head [create,call version.check_conds in=74b,call rgw_gc.rgw_gc_queue_init in=45b,call version.set in=34b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e32) v8 -- 0x560623650b10 con 0x5606236477d0 2023-10-05T19:28:55.496 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.490+0000 7fc9de566700 1 -- 172.21.15.167:0/1093196717 <== osd.3 v2:172.21.15.167:6808/30036 10 ==== osd_op_reply(26 gc.25 [create,call,call,call] v32'41 uv39 ondisk = -125 ((125) Operation canceled)) v8 ==== 275+0+0 (crc 0 0 0) 0x7fc9d0050040 con 0x5606236477d0 2023-10-05T19:28:55.496 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.490+0000 7fc9e0230e40 1 -- 172.21.15.167:0/1093196717 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:27 6.1f 6:fde4dd55:gc::gc.26:head [create,call version.check_conds in=74b,call rgw_gc.rgw_gc_queue_init in=45b,call version.set in=34b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e32) v8 -- 0x560623650b10 con 0x56062363c0d0 2023-10-05T19:28:55.499 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.494+0000 7fc9dd564700 1 -- 172.21.15.167:0/1093196717 <== osd.0 v2:172.21.15.118:6800/27690 5 ==== osd_op_reply(27 gc.26 [create,call,call,call] v32'24 uv23 ondisk = -125 ((125) Operation canceled)) v8 ==== 275+0+0 (crc 0 0 0) 0x7fc9d4032040 con 0x56062363c0d0 2023-10-05T19:28:55.499 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.494+0000 7fc9e0230e40 1 -- 172.21.15.167:0/1093196717 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:28 6.1a 6:5909e671:gc::gc.27:head [create,call version.check_conds in=74b,call rgw_gc.rgw_gc_queue_init in=45b,call version.set in=34b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e32) v8 -- 0x560623650d90 con 0x5606236477d0 2023-10-05T19:28:55.501 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.494+0000 7fc9de566700 1 -- 172.21.15.167:0/1093196717 <== osd.3 v2:172.21.15.167:6808/30036 11 ==== osd_op_reply(28 gc.27 [create,call,call,call] v32'22 uv21 ondisk = -125 ((125) Operation canceled)) v8 ==== 275+0+0 (crc 0 0 0) 0x7fc9d0050040 con 0x5606236477d0 2023-10-05T19:28:55.501 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.494+0000 7fc9e0230e40 1 -- 172.21.15.167:0/1093196717 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:29 6.1d 6:b88e3271:gc::gc.28:head [create,call version.check_conds in=74b,call rgw_gc.rgw_gc_queue_init in=45b,call version.set in=34b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e32) v8 -- 0x560623650d90 con 0x56062363c0d0 2023-10-05T19:28:55.504 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.498+0000 7fc9dd564700 1 -- 172.21.15.167:0/1093196717 <== osd.0 v2:172.21.15.118:6800/27690 6 ==== osd_op_reply(29 gc.28 [create,call,call,call] v32'38 uv36 ondisk = -125 ((125) Operation canceled)) v8 ==== 275+0+0 (crc 0 0 0) 0x7fc9d4032040 con 0x56062363c0d0 2023-10-05T19:28:55.504 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.498+0000 7fc9e0230e40 1 -- 172.21.15.167:0/1093196717 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:30 6.7 6:e47c9322:gc::gc.29:head [create,call version.check_conds in=74b,call rgw_gc.rgw_gc_queue_init in=45b,call version.set in=34b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e32) v8 -- 0x560623650d90 con 0x5606236477d0 2023-10-05T19:28:55.507 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.502+0000 7fc9de566700 1 -- 172.21.15.167:0/1093196717 <== osd.3 v2:172.21.15.167:6808/30036 12 ==== osd_op_reply(30 gc.29 [create,call,call,call] v32'52 uv49 ondisk = -125 ((125) Operation canceled)) v8 ==== 275+0+0 (crc 0 0 0) 0x7fc9d0050040 con 0x5606236477d0 2023-10-05T19:28:55.507 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.502+0000 7fc9e0230e40 1 -- 172.21.15.167:0/1093196717 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:31 6.0 6:013d6c33:gc::gc.30:head [create,call version.check_conds in=74b,call rgw_gc.rgw_gc_queue_init in=45b,call version.set in=34b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e32) v8 -- 0x560623650d90 con 0x56062363c0d0 2023-10-05T19:28:55.510 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.506+0000 7fc9dd564700 1 -- 172.21.15.167:0/1093196717 <== osd.0 v2:172.21.15.118:6800/27690 7 ==== osd_op_reply(31 gc.30 [create,call,call,call] v32'44 uv42 ondisk = -125 ((125) Operation canceled)) v8 ==== 275+0+0 (crc 0 0 0) 0x7fc9d4032040 con 0x56062363c0d0 2023-10-05T19:28:55.510 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.506+0000 7fc9e0230e40 1 -- 172.21.15.167:0/1093196717 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:32 6.f 6:f24cb4a8:gc::gc.31:head [create,call version.check_conds in=74b,call rgw_gc.rgw_gc_queue_init in=45b,call version.set in=34b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e32) v8 -- 0x560623650d90 con 0x56062364b8f0 2023-10-05T19:28:55.512 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.506+0000 7fc9ddd65700 1 -- 172.21.15.167:0/1093196717 <== osd.2 v2:172.21.15.167:6816/30037 7 ==== osd_op_reply(32 gc.31 [create,call,call,call] v32'28 uv27 ondisk = -125 ((125) Operation canceled)) v8 ==== 275+0+0 (crc 0 0 0) 0x7fc9d8025020 con 0x56062364b8f0 2023-10-05T19:28:55.515 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.510+0000 7fc91dffb700 20 reqs_thread_entry: start 2023-10-05T19:28:55.516 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.510+0000 7fc9e0230e40 20 init_complete bucket index max shards: 11 2023-10-05T19:28:55.516 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.510+0000 7fc9177fe700 20 reqs_thread_entry: start 2023-10-05T19:28:55.516 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.510+0000 7fc9e0230e40 10 rgw notify: Started notification manager with: 1 workers 2023-10-05T19:28:55.516 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.510+0000 7fc915ffb700 20 rgw notify: INFO: next queues processing will happen at: Thu Oct 5 19:29:25 2023 2023-10-05T19:28:55.517 INFO:teuthology.orchestra.run.smithi167.stderr: 2023-10-05T19:28:55.517 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.510+0000 7fc9157fa700 20 reqs_thread_entry: start 2023-10-05T19:28:55.517 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.510+0000 7fc9e0230e40 1 -- 172.21.15.167:0/1093196717 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:33 6.19 6:9a6dd8c0:gc::gc.0:head [call rgw.gc_list in=25b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32) v8 -- 0x5606236b0100 con 0x5606236404f0 2023-10-05T19:28:55.517 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.510+0000 7fc9ddd65700 1 -- 172.21.15.167:0/1093196717 <== osd.1 v2:172.21.15.167:6800/30035 7 ==== osd_op_reply(33 gc.0 [call out=15b] v0'0 uv33 ondisk = 0) v8 ==== 148+0+15 (crc 0 0 0) 0x7fc9d8025020 con 0x5606236404f0 2023-10-05T19:28:55.518 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.510+0000 7fc9e0230e40 1 -- 172.21.15.167:0/1093196717 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:34 6.19 6:9a6dd8c0:gc::gc.0:head [call version.read in=11b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32) v8 -- 0x5606236af0f0 con 0x5606236404f0 2023-10-05T19:28:55.518 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.514+0000 7fc9ddd65700 1 -- 172.21.15.167:0/1093196717 <== osd.1 v2:172.21.15.167:6800/30035 8 ==== osd_op_reply(34 gc.0 [call out=24b] v0'0 uv33 ondisk = 0) v8 ==== 148+0+24 (crc 0 0 0) 0x7fc9d8025020 con 0x5606236404f0 2023-10-05T19:28:55.518 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.514+0000 7fc9e0230e40 1 -- 172.21.15.167:0/1093196717 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:35 6.19 6:9a6dd8c0:gc::gc.0:head [call rgw_gc.rgw_gc_queue_list_entries in=46b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32) v8 -- 0x5606236b0100 con 0x5606236404f0 2023-10-05T19:28:55.519 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.514+0000 7fc9ddd65700 1 -- 172.21.15.167:0/1093196717 <== osd.1 v2:172.21.15.167:6800/30035 9 ==== osd_op_reply(35 gc.0 [call out=15b] v0'0 uv33 ondisk = 0) v8 ==== 148+0+15 (crc 0 0 0) 0x7fc9d8025020 con 0x5606236404f0 2023-10-05T19:28:55.519 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.514+0000 7fc9e0230e40 1 -- 172.21.15.167:0/1093196717 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:36 6.18 6:1e85421f:gc::gc.1:head [call rgw.gc_list in=25b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32) v8 -- 0x5606236b0100 con 0x56062363c0d0 2023-10-05T19:28:55.520 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.514+0000 7fc9dd564700 1 -- 172.21.15.167:0/1093196717 <== osd.0 v2:172.21.15.118:6800/27690 8 ==== osd_op_reply(36 gc.1 [call out=15b] v0'0 uv25 ondisk = 0) v8 ==== 148+0+15 (crc 0 0 0) 0x7fc9d4032040 con 0x56062363c0d0 2023-10-05T19:28:55.520 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.514+0000 7fc9e0230e40 1 -- 172.21.15.167:0/1093196717 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:37 6.18 6:1e85421f:gc::gc.1:head [call version.read in=11b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32) v8 -- 0x5606236af0f0 con 0x56062363c0d0 2023-10-05T19:28:55.520 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.514+0000 7fc9dd564700 1 -- 172.21.15.167:0/1093196717 <== osd.0 v2:172.21.15.118:6800/27690 9 ==== osd_op_reply(37 gc.1 [call out=24b] v0'0 uv25 ondisk = 0) v8 ==== 148+0+24 (crc 0 0 0) 0x7fc9d4032040 con 0x56062363c0d0 2023-10-05T19:28:55.521 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.514+0000 7fc9e0230e40 1 -- 172.21.15.167:0/1093196717 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:38 6.18 6:1e85421f:gc::gc.1:head [call rgw_gc.rgw_gc_queue_list_entries in=46b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32) v8 -- 0x5606236b0100 con 0x56062363c0d0 2023-10-05T19:28:55.521 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.518+0000 7fc9dd564700 1 -- 172.21.15.167:0/1093196717 <== osd.0 v2:172.21.15.118:6800/27690 10 ==== osd_op_reply(38 gc.1 [call out=15b] v0'0 uv25 ondisk = 0) v8 ==== 148+0+15 (crc 0 0 0) 0x7fc9d4032040 con 0x56062363c0d0 2023-10-05T19:28:55.522 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.518+0000 7fc9e0230e40 1 -- 172.21.15.167:0/1093196717 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:39 6.b 6:d242335b:gc::gc.2:head [call rgw.gc_list in=25b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32) v8 -- 0x5606236b0100 con 0x5606236477d0 2023-10-05T19:28:55.522 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.518+0000 7fc9de566700 1 -- 172.21.15.167:0/1093196717 <== osd.3 v2:172.21.15.167:6808/30036 13 ==== osd_op_reply(39 gc.2 [call out=15b] v0'0 uv21 ondisk = 0) v8 ==== 148+0+15 (crc 0 0 0) 0x7fc9d0050040 con 0x5606236477d0 2023-10-05T19:28:55.522 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.518+0000 7fc9e0230e40 1 -- 172.21.15.167:0/1093196717 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:40 6.b 6:d242335b:gc::gc.2:head [call version.read in=11b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32) v8 -- 0x5606236af0f0 con 0x5606236477d0 2023-10-05T19:28:55.523 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.518+0000 7fc9de566700 1 -- 172.21.15.167:0/1093196717 <== osd.3 v2:172.21.15.167:6808/30036 14 ==== osd_op_reply(40 gc.2 [call out=24b] v0'0 uv21 ondisk = 0) v8 ==== 148+0+24 (crc 0 0 0) 0x7fc9d0050040 con 0x5606236477d0 2023-10-05T19:28:55.523 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.518+0000 7fc9e0230e40 1 -- 172.21.15.167:0/1093196717 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:41 6.b 6:d242335b:gc::gc.2:head [call rgw_gc.rgw_gc_queue_list_entries in=46b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32) v8 -- 0x5606236b0100 con 0x5606236477d0 2023-10-05T19:28:55.524 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.518+0000 7fc9de566700 1 -- 172.21.15.167:0/1093196717 <== osd.3 v2:172.21.15.167:6808/30036 15 ==== osd_op_reply(41 gc.2 [call out=15b] v0'0 uv21 ondisk = 0) v8 ==== 148+0+15 (crc 0 0 0) 0x7fc9d0050040 con 0x5606236477d0 2023-10-05T19:28:55.524 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.518+0000 7fc9e0230e40 1 -- 172.21.15.167:0/1093196717 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:42 6.4 6:2166d8e1:gc::gc.3:head [call rgw.gc_list in=25b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32) v8 -- 0x5606236b0100 con 0x5606236404f0 2023-10-05T19:28:55.525 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.518+0000 7fc9ddd65700 1 -- 172.21.15.167:0/1093196717 <== osd.1 v2:172.21.15.167:6800/30035 10 ==== osd_op_reply(42 gc.3 [call out=15b] v0'0 uv184 ondisk = 0) v8 ==== 148+0+15 (crc 0 0 0) 0x7fc9d8025020 con 0x5606236404f0 2023-10-05T19:28:55.525 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.518+0000 7fc9e0230e40 1 -- 172.21.15.167:0/1093196717 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:43 6.4 6:2166d8e1:gc::gc.3:head [call version.read in=11b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32) v8 -- 0x5606236af0f0 con 0x5606236404f0 2023-10-05T19:28:55.525 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.522+0000 7fc9ddd65700 1 -- 172.21.15.167:0/1093196717 <== osd.1 v2:172.21.15.167:6800/30035 11 ==== osd_op_reply(43 gc.3 [call out=24b] v0'0 uv184 ondisk = 0) v8 ==== 148+0+24 (crc 0 0 0) 0x7fc9d8025020 con 0x5606236404f0 2023-10-05T19:28:55.526 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.522+0000 7fc9e0230e40 1 -- 172.21.15.167:0/1093196717 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:44 6.4 6:2166d8e1:gc::gc.3:head [call rgw_gc.rgw_gc_queue_list_entries in=46b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32) v8 -- 0x5606236b0100 con 0x5606236404f0 2023-10-05T19:28:55.526 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.522+0000 7fc9ddd65700 1 -- 172.21.15.167:0/1093196717 <== osd.1 v2:172.21.15.167:6800/30035 12 ==== osd_op_reply(44 gc.3 [call out=15b] v0'0 uv184 ondisk = 0) v8 ==== 148+0+15 (crc 0 0 0) 0x7fc9d8025020 con 0x5606236404f0 2023-10-05T19:28:55.526 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.522+0000 7fc9e0230e40 1 -- 172.21.15.167:0/1093196717 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:45 6.3 6:c60da066:gc::gc.4:head [call rgw.gc_list in=25b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32) v8 -- 0x5606236b0100 con 0x56062364b8f0 2023-10-05T19:28:55.527 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.522+0000 7fc9ddd65700 1 -- 172.21.15.167:0/1093196717 <== osd.2 v2:172.21.15.167:6816/30037 8 ==== osd_op_reply(45 gc.4 [call out=15b] v0'0 uv51 ondisk = 0) v8 ==== 148+0+15 (crc 0 0 0) 0x7fc9d8025020 con 0x56062364b8f0 2023-10-05T19:28:55.527 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.522+0000 7fc9e0230e40 1 -- 172.21.15.167:0/1093196717 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:46 6.3 6:c60da066:gc::gc.4:head [call version.read in=11b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32) v8 -- 0x5606236af0f0 con 0x56062364b8f0 2023-10-05T19:28:55.528 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.522+0000 7fc9ddd65700 1 -- 172.21.15.167:0/1093196717 <== osd.2 v2:172.21.15.167:6816/30037 9 ==== osd_op_reply(46 gc.4 [call out=24b] v0'0 uv51 ondisk = 0) v8 ==== 148+0+24 (crc 0 0 0) 0x7fc9d8025020 con 0x56062364b8f0 2023-10-05T19:28:55.528 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.522+0000 7fc9e0230e40 1 -- 172.21.15.167:0/1093196717 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:47 6.3 6:c60da066:gc::gc.4:head [call rgw_gc.rgw_gc_queue_list_entries in=46b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32) v8 -- 0x5606236b0100 con 0x56062364b8f0 2023-10-05T19:28:55.529 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.522+0000 7fc9ddd65700 1 -- 172.21.15.167:0/1093196717 <== osd.2 v2:172.21.15.167:6816/30037 10 ==== osd_op_reply(47 gc.4 [call out=15b] v0'0 uv51 ondisk = 0) v8 ==== 148+0+15 (crc 0 0 0) 0x7fc9d8025020 con 0x56062364b8f0 2023-10-05T19:28:55.529 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.522+0000 7fc9e0230e40 1 -- 172.21.15.167:0/1093196717 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:48 6.9 6:954be8d2:gc::gc.5:head [call rgw.gc_list in=25b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32) v8 -- 0x5606236b0100 con 0x5606236477d0 2023-10-05T19:28:55.529 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.526+0000 7fc9de566700 1 -- 172.21.15.167:0/1093196717 <== osd.3 v2:172.21.15.167:6808/30036 16 ==== osd_op_reply(48 gc.5 [call out=15b] v0'0 uv37 ondisk = 0) v8 ==== 148+0+15 (crc 0 0 0) 0x7fc9d0050040 con 0x5606236477d0 2023-10-05T19:28:55.529 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.526+0000 7fc9e0230e40 1 -- 172.21.15.167:0/1093196717 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:49 6.9 6:954be8d2:gc::gc.5:head [call version.read in=11b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32) v8 -- 0x5606236af0f0 con 0x5606236477d0 2023-10-05T19:28:55.530 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.526+0000 7fc9de566700 1 -- 172.21.15.167:0/1093196717 <== osd.3 v2:172.21.15.167:6808/30036 17 ==== osd_op_reply(49 gc.5 [call out=24b] v0'0 uv37 ondisk = 0) v8 ==== 148+0+24 (crc 0 0 0) 0x7fc9d0050040 con 0x5606236477d0 2023-10-05T19:28:55.530 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.526+0000 7fc9e0230e40 1 -- 172.21.15.167:0/1093196717 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:50 6.9 6:954be8d2:gc::gc.5:head [call rgw_gc.rgw_gc_queue_list_entries in=46b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32) v8 -- 0x5606236b0100 con 0x5606236477d0 2023-10-05T19:28:55.531 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.526+0000 7fc9de566700 1 -- 172.21.15.167:0/1093196717 <== osd.3 v2:172.21.15.167:6808/30036 18 ==== osd_op_reply(50 gc.5 [call out=15b] v0'0 uv37 ondisk = 0) v8 ==== 148+0+15 (crc 0 0 0) 0x7fc9d0050040 con 0x5606236477d0 2023-10-05T19:28:55.531 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.526+0000 7fc9e0230e40 1 -- 172.21.15.167:0/1093196717 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:51 6.e 6:748722ff:gc::gc.6:head [call rgw.gc_list in=25b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32) v8 -- 0x5606236b0100 con 0x5606236404f0 2023-10-05T19:28:55.531 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.526+0000 7fc9ddd65700 1 -- 172.21.15.167:0/1093196717 <== osd.1 v2:172.21.15.167:6800/30035 13 ==== osd_op_reply(51 gc.6 [call out=15b] v0'0 uv23 ondisk = 0) v8 ==== 148+0+15 (crc 0 0 0) 0x7fc9d8025020 con 0x5606236404f0 2023-10-05T19:28:55.531 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.526+0000 7fc9e0230e40 1 -- 172.21.15.167:0/1093196717 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:52 6.e 6:748722ff:gc::gc.6:head [call version.read in=11b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32) v8 -- 0x5606236af0f0 con 0x5606236404f0 2023-10-05T19:28:55.532 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.526+0000 7fc9ddd65700 1 -- 172.21.15.167:0/1093196717 <== osd.1 v2:172.21.15.167:6800/30035 14 ==== osd_op_reply(52 gc.6 [call out=24b] v0'0 uv23 ondisk = 0) v8 ==== 148+0+24 (crc 0 0 0) 0x7fc9d8025020 con 0x5606236404f0 2023-10-05T19:28:55.532 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.526+0000 7fc9e0230e40 1 -- 172.21.15.167:0/1093196717 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:53 6.e 6:748722ff:gc::gc.6:head [call rgw_gc.rgw_gc_queue_list_entries in=46b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32) v8 -- 0x5606236b0100 con 0x5606236404f0 2023-10-05T19:28:55.533 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.526+0000 7fc9ddd65700 1 -- 172.21.15.167:0/1093196717 <== osd.1 v2:172.21.15.167:6800/30035 15 ==== osd_op_reply(53 gc.6 [call out=15b] v0'0 uv23 ondisk = 0) v8 ==== 148+0+15 (crc 0 0 0) 0x7fc9d8025020 con 0x5606236404f0 2023-10-05T19:28:55.533 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.526+0000 7fc9e0230e40 1 -- 172.21.15.167:0/1093196717 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:54 6.1d 6:beaa4be4:gc::gc.7:head [call rgw.gc_list in=25b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32) v8 -- 0x5606236b0100 con 0x56062363c0d0 2023-10-05T19:28:55.534 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.530+0000 7fc9dd564700 1 -- 172.21.15.167:0/1093196717 <== osd.0 v2:172.21.15.118:6800/27690 11 ==== osd_op_reply(54 gc.7 [call out=15b] v0'0 uv34 ondisk = 0) v8 ==== 148+0+15 (crc 0 0 0) 0x7fc9d4032040 con 0x56062363c0d0 2023-10-05T19:28:55.534 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.530+0000 7fc9e0230e40 1 -- 172.21.15.167:0/1093196717 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:55 6.1d 6:beaa4be4:gc::gc.7:head [call version.read in=11b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32) v8 -- 0x5606236af0f0 con 0x56062363c0d0 2023-10-05T19:28:55.535 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.530+0000 7fc9dd564700 1 -- 172.21.15.167:0/1093196717 <== osd.0 v2:172.21.15.118:6800/27690 12 ==== osd_op_reply(55 gc.7 [call out=24b] v0'0 uv34 ondisk = 0) v8 ==== 148+0+24 (crc 0 0 0) 0x7fc9d4032040 con 0x56062363c0d0 2023-10-05T19:28:55.535 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.530+0000 7fc9e0230e40 1 -- 172.21.15.167:0/1093196717 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:56 6.1d 6:beaa4be4:gc::gc.7:head [call rgw_gc.rgw_gc_queue_list_entries in=46b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32) v8 -- 0x5606236b0100 con 0x56062363c0d0 2023-10-05T19:28:55.536 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.530+0000 7fc9dd564700 1 -- 172.21.15.167:0/1093196717 <== osd.0 v2:172.21.15.118:6800/27690 13 ==== osd_op_reply(56 gc.7 [call out=15b] v0'0 uv34 ondisk = 0) v8 ==== 148+0+15 (crc 0 0 0) 0x7fc9d4032040 con 0x56062363c0d0 2023-10-05T19:28:55.536 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.530+0000 7fc9e0230e40 1 -- 172.21.15.167:0/1093196717 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:57 6.12 6:4f78e9a2:gc::gc.8:head [call rgw.gc_list in=25b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32) v8 -- 0x5606236b0100 con 0x56062364b8f0 2023-10-05T19:28:55.536 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.530+0000 7fc9ddd65700 1 -- 172.21.15.167:0/1093196717 <== osd.2 v2:172.21.15.167:6816/30037 11 ==== osd_op_reply(57 gc.8 [call out=15b] v0'0 uv21 ondisk = 0) v8 ==== 148+0+15 (crc 0 0 0) 0x7fc9d8025020 con 0x56062364b8f0 2023-10-05T19:28:55.536 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.530+0000 7fc9e0230e40 1 -- 172.21.15.167:0/1093196717 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:58 6.12 6:4f78e9a2:gc::gc.8:head [call version.read in=11b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32) v8 -- 0x5606236af0f0 con 0x56062364b8f0 2023-10-05T19:28:55.537 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.530+0000 7fc9ddd65700 1 -- 172.21.15.167:0/1093196717 <== osd.2 v2:172.21.15.167:6816/30037 12 ==== osd_op_reply(58 gc.8 [call out=24b] v0'0 uv21 ondisk = 0) v8 ==== 148+0+24 (crc 0 0 0) 0x7fc9d8025020 con 0x56062364b8f0 2023-10-05T19:28:55.537 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.530+0000 7fc9e0230e40 1 -- 172.21.15.167:0/1093196717 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:59 6.12 6:4f78e9a2:gc::gc.8:head [call rgw_gc.rgw_gc_queue_list_entries in=46b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32) v8 -- 0x5606236b0100 con 0x56062364b8f0 2023-10-05T19:28:55.537 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.534+0000 7fc9ddd65700 1 -- 172.21.15.167:0/1093196717 <== osd.2 v2:172.21.15.167:6816/30037 13 ==== osd_op_reply(59 gc.8 [call out=15b] v0'0 uv21 ondisk = 0) v8 ==== 148+0+15 (crc 0 0 0) 0x7fc9d8025020 con 0x56062364b8f0 2023-10-05T19:28:55.538 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.534+0000 7fc9e0230e40 1 -- 172.21.15.167:0/1093196717 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:60 6.1c 6:3e426232:gc::gc.9:head [call rgw.gc_list in=25b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32) v8 -- 0x5606236b0100 con 0x5606236477d0 2023-10-05T19:28:55.538 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.534+0000 7fc9de566700 1 -- 172.21.15.167:0/1093196717 <== osd.3 v2:172.21.15.167:6808/30036 19 ==== osd_op_reply(60 gc.9 [call out=15b] v0'0 uv39 ondisk = 0) v8 ==== 148+0+15 (crc 0 0 0) 0x7fc9d0050040 con 0x5606236477d0 2023-10-05T19:28:55.538 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.534+0000 7fc9e0230e40 1 -- 172.21.15.167:0/1093196717 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:61 6.1c 6:3e426232:gc::gc.9:head [call version.read in=11b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32) v8 -- 0x5606236af0f0 con 0x5606236477d0 2023-10-05T19:28:55.539 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.534+0000 7fc9de566700 1 -- 172.21.15.167:0/1093196717 <== osd.3 v2:172.21.15.167:6808/30036 20 ==== osd_op_reply(61 gc.9 [call out=24b] v0'0 uv39 ondisk = 0) v8 ==== 148+0+24 (crc 0 0 0) 0x7fc9d0050040 con 0x5606236477d0 2023-10-05T19:28:55.539 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.534+0000 7fc9e0230e40 1 -- 172.21.15.167:0/1093196717 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:62 6.1c 6:3e426232:gc::gc.9:head [call rgw_gc.rgw_gc_queue_list_entries in=46b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32) v8 -- 0x5606236b0100 con 0x5606236477d0 2023-10-05T19:28:55.539 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.534+0000 7fc9de566700 1 -- 172.21.15.167:0/1093196717 <== osd.3 v2:172.21.15.167:6808/30036 21 ==== osd_op_reply(62 gc.9 [call out=15b] v0'0 uv39 ondisk = 0) v8 ==== 148+0+15 (crc 0 0 0) 0x7fc9d0050040 con 0x5606236477d0 2023-10-05T19:28:55.539 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.534+0000 7fc9e0230e40 1 -- 172.21.15.167:0/1093196717 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:63 6.3 6:c31f41d7:gc::gc.10:head [call rgw.gc_list in=25b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32) v8 -- 0x5606236b0100 con 0x56062364b8f0 2023-10-05T19:28:55.540 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.534+0000 7fc9ddd65700 1 -- 172.21.15.167:0/1093196717 <== osd.2 v2:172.21.15.167:6816/30037 14 ==== osd_op_reply(63 gc.10 [call out=15b] v0'0 uv53 ondisk = 0) v8 ==== 149+0+15 (crc 0 0 0) 0x7fc9d8025020 con 0x56062364b8f0 2023-10-05T19:28:55.540 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.534+0000 7fc9e0230e40 1 -- 172.21.15.167:0/1093196717 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:64 6.3 6:c31f41d7:gc::gc.10:head [call version.read in=11b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32) v8 -- 0x5606236af0f0 con 0x56062364b8f0 2023-10-05T19:28:55.541 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.534+0000 7fc9ddd65700 1 -- 172.21.15.167:0/1093196717 <== osd.2 v2:172.21.15.167:6816/30037 15 ==== osd_op_reply(64 gc.10 [call out=24b] v0'0 uv53 ondisk = 0) v8 ==== 149+0+24 (crc 0 0 0) 0x7fc9d8025020 con 0x56062364b8f0 2023-10-05T19:28:55.541 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.534+0000 7fc9e0230e40 1 -- 172.21.15.167:0/1093196717 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:65 6.3 6:c31f41d7:gc::gc.10:head [call rgw_gc.rgw_gc_queue_list_entries in=46b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32) v8 -- 0x5606236b0100 con 0x56062364b8f0 2023-10-05T19:28:55.542 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.538+0000 7fc9ddd65700 1 -- 172.21.15.167:0/1093196717 <== osd.2 v2:172.21.15.167:6816/30037 16 ==== osd_op_reply(65 gc.10 [call out=15b] v0'0 uv53 ondisk = 0) v8 ==== 149+0+15 (crc 0 0 0) 0x7fc9d8025020 con 0x56062364b8f0 2023-10-05T19:28:55.542 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.538+0000 7fc9e0230e40 1 -- 172.21.15.167:0/1093196717 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:66 6.13 6:c8333aab:gc::gc.11:head [call rgw.gc_list in=25b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32) v8 -- 0x5606236b0100 con 0x56062363c0d0 2023-10-05T19:28:55.543 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.538+0000 7fc9dd564700 1 -- 172.21.15.167:0/1093196717 <== osd.0 v2:172.21.15.118:6800/27690 14 ==== osd_op_reply(66 gc.11 [call out=15b] v0'0 uv25 ondisk = 0) v8 ==== 149+0+15 (crc 0 0 0) 0x7fc9d4032040 con 0x56062363c0d0 2023-10-05T19:28:55.543 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.538+0000 7fc9e0230e40 1 -- 172.21.15.167:0/1093196717 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:67 6.13 6:c8333aab:gc::gc.11:head [call version.read in=11b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32) v8 -- 0x5606236af0f0 con 0x56062363c0d0 2023-10-05T19:28:55.544 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.538+0000 7fc9dd564700 1 -- 172.21.15.167:0/1093196717 <== osd.0 v2:172.21.15.118:6800/27690 15 ==== osd_op_reply(67 gc.11 [call out=24b] v0'0 uv25 ondisk = 0) v8 ==== 149+0+24 (crc 0 0 0) 0x7fc9d4032040 con 0x56062363c0d0 2023-10-05T19:28:55.544 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.538+0000 7fc9e0230e40 1 -- 172.21.15.167:0/1093196717 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:68 6.13 6:c8333aab:gc::gc.11:head [call rgw_gc.rgw_gc_queue_list_entries in=46b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32) v8 -- 0x5606236b0100 con 0x56062363c0d0 2023-10-05T19:28:55.545 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.538+0000 7fc9dd564700 1 -- 172.21.15.167:0/1093196717 <== osd.0 v2:172.21.15.118:6800/27690 16 ==== osd_op_reply(68 gc.11 [call out=15b] v0'0 uv25 ondisk = 0) v8 ==== 149+0+15 (crc 0 0 0) 0x7fc9d4032040 con 0x56062363c0d0 2023-10-05T19:28:55.545 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.538+0000 7fc9e0230e40 1 -- 172.21.15.167:0/1093196717 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:69 6.15 6:ab569f81:gc::gc.12:head [call rgw.gc_list in=25b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32) v8 -- 0x5606236b0100 con 0x56062364b8f0 2023-10-05T19:28:55.546 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.542+0000 7fc9ddd65700 1 -- 172.21.15.167:0/1093196717 <== osd.2 v2:172.21.15.167:6816/30037 17 ==== osd_op_reply(69 gc.12 [call out=15b] v0'0 uv21 ondisk = 0) v8 ==== 149+0+15 (crc 0 0 0) 0x7fc9d8025020 con 0x56062364b8f0 2023-10-05T19:28:55.546 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.542+0000 7fc9e0230e40 1 -- 172.21.15.167:0/1093196717 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:70 6.15 6:ab569f81:gc::gc.12:head [call version.read in=11b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32) v8 -- 0x5606236af0f0 con 0x56062364b8f0 2023-10-05T19:28:55.547 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.542+0000 7fc9ddd65700 1 -- 172.21.15.167:0/1093196717 <== osd.2 v2:172.21.15.167:6816/30037 18 ==== osd_op_reply(70 gc.12 [call out=24b] v0'0 uv21 ondisk = 0) v8 ==== 149+0+24 (crc 0 0 0) 0x7fc9d8025020 con 0x56062364b8f0 2023-10-05T19:28:55.547 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.542+0000 7fc9e0230e40 1 -- 172.21.15.167:0/1093196717 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:71 6.15 6:ab569f81:gc::gc.12:head [call rgw_gc.rgw_gc_queue_list_entries in=46b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32) v8 -- 0x5606236b0100 con 0x56062364b8f0 2023-10-05T19:28:55.547 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.542+0000 7fc9ddd65700 1 -- 172.21.15.167:0/1093196717 <== osd.2 v2:172.21.15.167:6816/30037 19 ==== osd_op_reply(71 gc.12 [call out=15b] v0'0 uv21 ondisk = 0) v8 ==== 149+0+15 (crc 0 0 0) 0x7fc9d8025020 con 0x56062364b8f0 2023-10-05T19:28:55.547 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.542+0000 7fc9e0230e40 1 -- 172.21.15.167:0/1093196717 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:72 6.0 6:0667bb13:gc::gc.13:head [call rgw.gc_list in=25b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32) v8 -- 0x5606236b0100 con 0x56062363c0d0 2023-10-05T19:28:55.548 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.542+0000 7fc9dd564700 1 -- 172.21.15.167:0/1093196717 <== osd.0 v2:172.21.15.118:6800/27690 17 ==== osd_op_reply(72 gc.13 [call out=15b] v0'0 uv40 ondisk = 0) v8 ==== 149+0+15 (crc 0 0 0) 0x7fc9d4032040 con 0x56062363c0d0 2023-10-05T19:28:55.548 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.542+0000 7fc9e0230e40 1 -- 172.21.15.167:0/1093196717 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:73 6.0 6:0667bb13:gc::gc.13:head [call version.read in=11b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32) v8 -- 0x5606236af0f0 con 0x56062363c0d0 2023-10-05T19:28:55.549 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.542+0000 7fc9dd564700 1 -- 172.21.15.167:0/1093196717 <== osd.0 v2:172.21.15.118:6800/27690 18 ==== osd_op_reply(73 gc.13 [call out=24b] v0'0 uv40 ondisk = 0) v8 ==== 149+0+24 (crc 0 0 0) 0x7fc9d4032040 con 0x56062363c0d0 2023-10-05T19:28:55.549 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.542+0000 7fc9e0230e40 1 -- 172.21.15.167:0/1093196717 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:74 6.0 6:0667bb13:gc::gc.13:head [call rgw_gc.rgw_gc_queue_list_entries in=46b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32) v8 -- 0x5606236b0100 con 0x56062363c0d0 2023-10-05T19:28:55.550 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.546+0000 7fc9dd564700 1 -- 172.21.15.167:0/1093196717 <== osd.0 v2:172.21.15.118:6800/27690 19 ==== osd_op_reply(74 gc.13 [call out=15b] v0'0 uv40 ondisk = 0) v8 ==== 149+0+15 (crc 0 0 0) 0x7fc9d4032040 con 0x56062363c0d0 2023-10-05T19:28:55.550 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.546+0000 7fc9e0230e40 1 -- 172.21.15.167:0/1093196717 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:75 6.4 6:227f63b5:gc::gc.14:head [call rgw.gc_list in=25b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32) v8 -- 0x5606236b0100 con 0x5606236404f0 2023-10-05T19:28:55.551 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.546+0000 7fc9ddd65700 1 -- 172.21.15.167:0/1093196717 <== osd.1 v2:172.21.15.167:6800/30035 16 ==== osd_op_reply(75 gc.14 [call out=15b] v0'0 uv186 ondisk = 0) v8 ==== 149+0+15 (crc 0 0 0) 0x7fc9d8025020 con 0x5606236404f0 2023-10-05T19:28:55.551 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.546+0000 7fc9e0230e40 1 -- 172.21.15.167:0/1093196717 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:76 6.4 6:227f63b5:gc::gc.14:head [call version.read in=11b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32) v8 -- 0x5606236af0f0 con 0x5606236404f0 2023-10-05T19:28:55.551 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.546+0000 7fc9ddd65700 1 -- 172.21.15.167:0/1093196717 <== osd.1 v2:172.21.15.167:6800/30035 17 ==== osd_op_reply(76 gc.14 [call out=24b] v0'0 uv186 ondisk = 0) v8 ==== 149+0+24 (crc 0 0 0) 0x7fc9d8025020 con 0x5606236404f0 2023-10-05T19:28:55.552 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.546+0000 7fc9e0230e40 1 -- 172.21.15.167:0/1093196717 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:77 6.4 6:227f63b5:gc::gc.14:head [call rgw_gc.rgw_gc_queue_list_entries in=46b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32) v8 -- 0x5606236b0100 con 0x5606236404f0 2023-10-05T19:28:55.552 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.546+0000 7fc9ddd65700 1 -- 172.21.15.167:0/1093196717 <== osd.1 v2:172.21.15.167:6800/30035 18 ==== osd_op_reply(77 gc.14 [call out=15b] v0'0 uv186 ondisk = 0) v8 ==== 149+0+15 (crc 0 0 0) 0x7fc9d8025020 con 0x5606236404f0 2023-10-05T19:28:55.552 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.546+0000 7fc9e0230e40 1 -- 172.21.15.167:0/1093196717 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:78 6.7 6:e07a6548:gc::gc.15:head [call rgw.gc_list in=25b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32) v8 -- 0x5606236b0100 con 0x5606236477d0 2023-10-05T19:28:55.553 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.546+0000 7fc9de566700 1 -- 172.21.15.167:0/1093196717 <== osd.3 v2:172.21.15.167:6808/30036 22 ==== osd_op_reply(78 gc.15 [call out=15b] v0'0 uv45 ondisk = 0) v8 ==== 149+0+15 (crc 0 0 0) 0x7fc9d0050040 con 0x5606236477d0 2023-10-05T19:28:55.553 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.546+0000 7fc9e0230e40 1 -- 172.21.15.167:0/1093196717 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:79 6.7 6:e07a6548:gc::gc.15:head [call version.read in=11b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32) v8 -- 0x5606236af0f0 con 0x5606236477d0 2023-10-05T19:28:55.553 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.550+0000 7fc9de566700 1 -- 172.21.15.167:0/1093196717 <== osd.3 v2:172.21.15.167:6808/30036 23 ==== osd_op_reply(79 gc.15 [call out=24b] v0'0 uv45 ondisk = 0) v8 ==== 149+0+24 (crc 0 0 0) 0x7fc9d0050040 con 0x5606236477d0 2023-10-05T19:28:55.553 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.550+0000 7fc9e0230e40 1 -- 172.21.15.167:0/1093196717 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:80 6.7 6:e07a6548:gc::gc.15:head [call rgw_gc.rgw_gc_queue_list_entries in=46b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32) v8 -- 0x5606236b0100 con 0x5606236477d0 2023-10-05T19:28:55.554 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.550+0000 7fc9de566700 1 -- 172.21.15.167:0/1093196717 <== osd.3 v2:172.21.15.167:6808/30036 24 ==== osd_op_reply(80 gc.15 [call out=15b] v0'0 uv45 ondisk = 0) v8 ==== 149+0+15 (crc 0 0 0) 0x7fc9d0050040 con 0x5606236477d0 2023-10-05T19:28:55.554 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.550+0000 7fc9e0230e40 1 -- 172.21.15.167:0/1093196717 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:81 6.3 6:c623b173:gc::gc.16:head [call rgw.gc_list in=25b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32) v8 -- 0x5606236b0100 con 0x56062364b8f0 2023-10-05T19:28:55.555 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.550+0000 7fc9ddd65700 1 -- 172.21.15.167:0/1093196717 <== osd.2 v2:172.21.15.167:6816/30037 20 ==== osd_op_reply(81 gc.16 [call out=15b] v0'0 uv55 ondisk = 0) v8 ==== 149+0+15 (crc 0 0 0) 0x7fc9d8025020 con 0x56062364b8f0 2023-10-05T19:28:55.555 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.550+0000 7fc9e0230e40 1 -- 172.21.15.167:0/1093196717 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:82 6.3 6:c623b173:gc::gc.16:head [call version.read in=11b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32) v8 -- 0x5606236af0f0 con 0x56062364b8f0 2023-10-05T19:28:55.556 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.550+0000 7fc9ddd65700 1 -- 172.21.15.167:0/1093196717 <== osd.2 v2:172.21.15.167:6816/30037 21 ==== osd_op_reply(82 gc.16 [call out=24b] v0'0 uv55 ondisk = 0) v8 ==== 149+0+24 (crc 0 0 0) 0x7fc9d8025020 con 0x56062364b8f0 2023-10-05T19:28:55.556 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.550+0000 7fc9e0230e40 1 -- 172.21.15.167:0/1093196717 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:83 6.3 6:c623b173:gc::gc.16:head [call rgw_gc.rgw_gc_queue_list_entries in=46b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32) v8 -- 0x5606236b0100 con 0x56062364b8f0 2023-10-05T19:28:55.557 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.550+0000 7fc9ddd65700 1 -- 172.21.15.167:0/1093196717 <== osd.2 v2:172.21.15.167:6816/30037 22 ==== osd_op_reply(83 gc.16 [call out=15b] v0'0 uv55 ondisk = 0) v8 ==== 149+0+15 (crc 0 0 0) 0x7fc9d8025020 con 0x56062364b8f0 2023-10-05T19:28:55.557 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.550+0000 7fc9e0230e40 1 -- 172.21.15.167:0/1093196717 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:84 6.16 6:6849fd06:gc::gc.17:head [call rgw.gc_list in=25b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32) v8 -- 0x5606236b0100 con 0x5606236477d0 2023-10-05T19:28:55.557 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.554+0000 7fc9de566700 1 -- 172.21.15.167:0/1093196717 <== osd.3 v2:172.21.15.167:6808/30036 25 ==== osd_op_reply(84 gc.17 [call out=15b] v0'0 uv21 ondisk = 0) v8 ==== 149+0+15 (crc 0 0 0) 0x7fc9d0050040 con 0x5606236477d0 2023-10-05T19:28:55.557 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.554+0000 7fc9e0230e40 1 -- 172.21.15.167:0/1093196717 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:85 6.16 6:6849fd06:gc::gc.17:head [call version.read in=11b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32) v8 -- 0x5606236af0f0 con 0x5606236477d0 2023-10-05T19:28:55.558 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.554+0000 7fc9de566700 1 -- 172.21.15.167:0/1093196717 <== osd.3 v2:172.21.15.167:6808/30036 26 ==== osd_op_reply(85 gc.17 [call out=24b] v0'0 uv21 ondisk = 0) v8 ==== 149+0+24 (crc 0 0 0) 0x7fc9d0050040 con 0x5606236477d0 2023-10-05T19:28:55.558 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.554+0000 7fc9e0230e40 1 -- 172.21.15.167:0/1093196717 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:86 6.16 6:6849fd06:gc::gc.17:head [call rgw_gc.rgw_gc_queue_list_entries in=46b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32) v8 -- 0x5606236b0100 con 0x5606236477d0 2023-10-05T19:28:55.559 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.554+0000 7fc9de566700 1 -- 172.21.15.167:0/1093196717 <== osd.3 v2:172.21.15.167:6808/30036 27 ==== osd_op_reply(86 gc.17 [call out=15b] v0'0 uv21 ondisk = 0) v8 ==== 149+0+15 (crc 0 0 0) 0x7fc9d0050040 con 0x5606236477d0 2023-10-05T19:28:55.559 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.554+0000 7fc9e0230e40 1 -- 172.21.15.167:0/1093196717 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:87 6.d 6:b019ea83:gc::gc.18:head [call rgw.gc_list in=25b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32) v8 -- 0x5606236b0100 con 0x5606236404f0 2023-10-05T19:28:55.560 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.554+0000 7fc9ddd65700 1 -- 172.21.15.167:0/1093196717 <== osd.1 v2:172.21.15.167:6800/30035 19 ==== osd_op_reply(87 gc.18 [call out=15b] v0'0 uv29 ondisk = 0) v8 ==== 149+0+15 (crc 0 0 0) 0x7fc9d8025020 con 0x5606236404f0 2023-10-05T19:28:55.560 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.554+0000 7fc9e0230e40 1 -- 172.21.15.167:0/1093196717 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:88 6.d 6:b019ea83:gc::gc.18:head [call version.read in=11b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32) v8 -- 0x5606236af0f0 con 0x5606236404f0 2023-10-05T19:28:55.560 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.554+0000 7fc9ddd65700 1 -- 172.21.15.167:0/1093196717 <== osd.1 v2:172.21.15.167:6800/30035 20 ==== osd_op_reply(88 gc.18 [call out=24b] v0'0 uv29 ondisk = 0) v8 ==== 149+0+24 (crc 0 0 0) 0x7fc9d8025020 con 0x5606236404f0 2023-10-05T19:28:55.561 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.554+0000 7fc9e0230e40 1 -- 172.21.15.167:0/1093196717 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:89 6.d 6:b019ea83:gc::gc.18:head [call rgw_gc.rgw_gc_queue_list_entries in=46b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32) v8 -- 0x5606236b0100 con 0x5606236404f0 2023-10-05T19:28:55.561 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.554+0000 7fc9ddd65700 1 -- 172.21.15.167:0/1093196717 <== osd.1 v2:172.21.15.167:6800/30035 21 ==== osd_op_reply(89 gc.18 [call out=15b] v0'0 uv29 ondisk = 0) v8 ==== 149+0+15 (crc 0 0 0) 0x7fc9d8025020 con 0x5606236404f0 2023-10-05T19:28:55.561 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.558+0000 7fc9e0230e40 1 -- 172.21.15.167:0/1093196717 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:90 6.4 6:24c8e1ae:gc::gc.19:head [call rgw.gc_list in=25b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32) v8 -- 0x5606236b0100 con 0x5606236404f0 2023-10-05T19:28:55.562 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.558+0000 7fc9ddd65700 1 -- 172.21.15.167:0/1093196717 <== osd.1 v2:172.21.15.167:6800/30035 22 ==== osd_op_reply(90 gc.19 [call out=15b] v0'0 uv189 ondisk = 0) v8 ==== 149+0+15 (crc 0 0 0) 0x7fc9d8025020 con 0x5606236404f0 2023-10-05T19:28:55.562 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.558+0000 7fc9e0230e40 1 -- 172.21.15.167:0/1093196717 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:91 6.4 6:24c8e1ae:gc::gc.19:head [call version.read in=11b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32) v8 -- 0x5606236af0f0 con 0x5606236404f0 2023-10-05T19:28:55.563 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.558+0000 7fc9ddd65700 1 -- 172.21.15.167:0/1093196717 <== osd.1 v2:172.21.15.167:6800/30035 23 ==== osd_op_reply(91 gc.19 [call out=24b] v0'0 uv189 ondisk = 0) v8 ==== 149+0+24 (crc 0 0 0) 0x7fc9d8025020 con 0x5606236404f0 2023-10-05T19:28:55.563 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.558+0000 7fc9e0230e40 1 -- 172.21.15.167:0/1093196717 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:92 6.4 6:24c8e1ae:gc::gc.19:head [call rgw_gc.rgw_gc_queue_list_entries in=46b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32) v8 -- 0x5606236b0100 con 0x5606236404f0 2023-10-05T19:28:55.563 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.558+0000 7fc9ddd65700 1 -- 172.21.15.167:0/1093196717 <== osd.1 v2:172.21.15.167:6800/30035 24 ==== osd_op_reply(92 gc.19 [call out=15b] v0'0 uv189 ondisk = 0) v8 ==== 149+0+15 (crc 0 0 0) 0x7fc9d8025020 con 0x5606236404f0 2023-10-05T19:28:55.564 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.558+0000 7fc9e0230e40 1 -- 172.21.15.167:0/1093196717 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:93 6.7 6:e622f76a:gc::gc.20:head [call rgw.gc_list in=25b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32) v8 -- 0x5606236b0100 con 0x5606236477d0 2023-10-05T19:28:55.564 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.558+0000 7fc9de566700 1 -- 172.21.15.167:0/1093196717 <== osd.3 v2:172.21.15.167:6808/30036 28 ==== osd_op_reply(93 gc.20 [call out=15b] v0'0 uv47 ondisk = 0) v8 ==== 149+0+15 (crc 0 0 0) 0x7fc9d0050040 con 0x5606236477d0 2023-10-05T19:28:55.564 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.558+0000 7fc9e0230e40 1 -- 172.21.15.167:0/1093196717 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:94 6.7 6:e622f76a:gc::gc.20:head [call version.read in=11b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32) v8 -- 0x5606236af0f0 con 0x5606236477d0 2023-10-05T19:28:55.565 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.558+0000 7fc9de566700 1 -- 172.21.15.167:0/1093196717 <== osd.3 v2:172.21.15.167:6808/30036 29 ==== osd_op_reply(94 gc.20 [call out=24b] v0'0 uv47 ondisk = 0) v8 ==== 149+0+24 (crc 0 0 0) 0x7fc9d0050040 con 0x5606236477d0 2023-10-05T19:28:55.565 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.558+0000 7fc9e0230e40 1 -- 172.21.15.167:0/1093196717 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:95 6.7 6:e622f76a:gc::gc.20:head [call rgw_gc.rgw_gc_queue_list_entries in=46b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32) v8 -- 0x5606236b0100 con 0x5606236477d0 2023-10-05T19:28:55.565 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.562+0000 7fc9de566700 1 -- 172.21.15.167:0/1093196717 <== osd.3 v2:172.21.15.167:6808/30036 30 ==== osd_op_reply(95 gc.20 [call out=15b] v0'0 uv47 ondisk = 0) v8 ==== 149+0+15 (crc 0 0 0) 0x7fc9d0050040 con 0x5606236477d0 2023-10-05T19:28:55.566 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.562+0000 7fc9e0230e40 1 -- 172.21.15.167:0/1093196717 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:96 6.a 6:5127e642:gc::gc.21:head [call rgw.gc_list in=25b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32) v8 -- 0x5606236b0100 con 0x56062364b8f0 2023-10-05T19:28:55.566 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.562+0000 7fc9ddd65700 1 -- 172.21.15.167:0/1093196717 <== osd.2 v2:172.21.15.167:6816/30037 23 ==== osd_op_reply(96 gc.21 [call out=15b] v0'0 uv31 ondisk = 0) v8 ==== 149+0+15 (crc 0 0 0) 0x7fc9d8025020 con 0x56062364b8f0 2023-10-05T19:28:55.566 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.562+0000 7fc9e0230e40 1 -- 172.21.15.167:0/1093196717 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:97 6.a 6:5127e642:gc::gc.21:head [call version.read in=11b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32) v8 -- 0x5606236af0f0 con 0x56062364b8f0 2023-10-05T19:28:55.567 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.562+0000 7fc9ddd65700 1 -- 172.21.15.167:0/1093196717 <== osd.2 v2:172.21.15.167:6816/30037 24 ==== osd_op_reply(97 gc.21 [call out=24b] v0'0 uv31 ondisk = 0) v8 ==== 149+0+24 (crc 0 0 0) 0x7fc9d8025020 con 0x56062364b8f0 2023-10-05T19:28:55.567 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.562+0000 7fc9e0230e40 1 -- 172.21.15.167:0/1093196717 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:98 6.a 6:5127e642:gc::gc.21:head [call rgw_gc.rgw_gc_queue_list_entries in=46b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32) v8 -- 0x5606236b0100 con 0x56062364b8f0 2023-10-05T19:28:55.568 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.562+0000 7fc9ddd65700 1 -- 172.21.15.167:0/1093196717 <== osd.2 v2:172.21.15.167:6816/30037 25 ==== osd_op_reply(98 gc.21 [call out=15b] v0'0 uv31 ondisk = 0) v8 ==== 149+0+15 (crc 0 0 0) 0x7fc9d8025020 con 0x56062364b8f0 2023-10-05T19:28:55.568 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.562+0000 7fc9e0230e40 1 -- 172.21.15.167:0/1093196717 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:99 6.11 6:8eb99c53:gc::gc.22:head [call rgw.gc_list in=25b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32) v8 -- 0x5606236b0100 con 0x5606236477d0 2023-10-05T19:28:55.568 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.562+0000 7fc9de566700 1 -- 172.21.15.167:0/1093196717 <== osd.3 v2:172.21.15.167:6808/30036 31 ==== osd_op_reply(99 gc.22 [call out=15b] v0'0 uv23 ondisk = 0) v8 ==== 149+0+15 (crc 0 0 0) 0x7fc9d0050040 con 0x5606236477d0 2023-10-05T19:28:55.568 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.562+0000 7fc9e0230e40 1 -- 172.21.15.167:0/1093196717 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:100 6.11 6:8eb99c53:gc::gc.22:head [call version.read in=11b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32) v8 -- 0x5606236af0f0 con 0x5606236477d0 2023-10-05T19:28:55.569 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.562+0000 7fc9de566700 1 -- 172.21.15.167:0/1093196717 <== osd.3 v2:172.21.15.167:6808/30036 32 ==== osd_op_reply(100 gc.22 [call out=24b] v0'0 uv23 ondisk = 0) v8 ==== 149+0+24 (crc 0 0 0) 0x7fc9d0050040 con 0x5606236477d0 2023-10-05T19:28:55.569 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.562+0000 7fc9e0230e40 1 -- 172.21.15.167:0/1093196717 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:101 6.11 6:8eb99c53:gc::gc.22:head [call rgw_gc.rgw_gc_queue_list_entries in=46b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32) v8 -- 0x5606236b0100 con 0x5606236477d0 2023-10-05T19:28:55.569 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.566+0000 7fc9de566700 1 -- 172.21.15.167:0/1093196717 <== osd.3 v2:172.21.15.167:6808/30036 33 ==== osd_op_reply(101 gc.22 [call out=15b] v0'0 uv23 ondisk = 0) v8 ==== 149+0+15 (crc 0 0 0) 0x7fc9d0050040 con 0x5606236477d0 2023-10-05T19:28:55.570 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.566+0000 7fc9e0230e40 1 -- 172.21.15.167:0/1093196717 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:102 6.8 6:17d7e8ed:gc::gc.23:head [call rgw.gc_list in=25b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32) v8 -- 0x5606236b0100 con 0x5606236477d0 2023-10-05T19:28:55.570 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.566+0000 7fc9de566700 1 -- 172.21.15.167:0/1093196717 <== osd.3 v2:172.21.15.167:6808/30036 34 ==== osd_op_reply(102 gc.23 [call out=15b] v0'0 uv33 ondisk = 0) v8 ==== 149+0+15 (crc 0 0 0) 0x7fc9d0050040 con 0x5606236477d0 2023-10-05T19:28:55.570 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.566+0000 7fc9e0230e40 1 -- 172.21.15.167:0/1093196717 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:103 6.8 6:17d7e8ed:gc::gc.23:head [call version.read in=11b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32) v8 -- 0x5606236af0f0 con 0x5606236477d0 2023-10-05T19:28:55.571 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.566+0000 7fc9de566700 1 -- 172.21.15.167:0/1093196717 <== osd.3 v2:172.21.15.167:6808/30036 35 ==== osd_op_reply(103 gc.23 [call out=24b] v0'0 uv33 ondisk = 0) v8 ==== 149+0+24 (crc 0 0 0) 0x7fc9d0050040 con 0x5606236477d0 2023-10-05T19:28:55.571 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.566+0000 7fc9e0230e40 1 -- 172.21.15.167:0/1093196717 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:104 6.8 6:17d7e8ed:gc::gc.23:head [call rgw_gc.rgw_gc_queue_list_entries in=46b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32) v8 -- 0x5606236b0100 con 0x5606236477d0 2023-10-05T19:28:55.571 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.566+0000 7fc9de566700 1 -- 172.21.15.167:0/1093196717 <== osd.3 v2:172.21.15.167:6808/30036 36 ==== osd_op_reply(104 gc.23 [call out=15b] v0'0 uv33 ondisk = 0) v8 ==== 149+0+15 (crc 0 0 0) 0x7fc9d0050040 con 0x5606236477d0 2023-10-05T19:28:55.571 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.566+0000 7fc9e0230e40 1 -- 172.21.15.167:0/1093196717 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:105 6.1b 6:db9d130d:gc::gc.24:head [call rgw.gc_list in=25b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32) v8 -- 0x5606236b0100 con 0x5606236477d0 2023-10-05T19:28:55.572 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.566+0000 7fc9de566700 1 -- 172.21.15.167:0/1093196717 <== osd.3 v2:172.21.15.167:6808/30036 37 ==== osd_op_reply(105 gc.24 [call out=15b] v0'0 uv17 ondisk = 0) v8 ==== 149+0+15 (crc 0 0 0) 0x7fc9d0050040 con 0x5606236477d0 2023-10-05T19:28:55.572 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.566+0000 7fc9e0230e40 1 -- 172.21.15.167:0/1093196717 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:106 6.1b 6:db9d130d:gc::gc.24:head [call version.read in=11b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32) v8 -- 0x5606236af0f0 con 0x5606236477d0 2023-10-05T19:28:55.573 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.566+0000 7fc9de566700 1 -- 172.21.15.167:0/1093196717 <== osd.3 v2:172.21.15.167:6808/30036 38 ==== osd_op_reply(106 gc.24 [call out=24b] v0'0 uv17 ondisk = 0) v8 ==== 149+0+24 (crc 0 0 0) 0x7fc9d0050040 con 0x5606236477d0 2023-10-05T19:28:55.573 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.566+0000 7fc9e0230e40 1 -- 172.21.15.167:0/1093196717 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:107 6.1b 6:db9d130d:gc::gc.24:head [call rgw_gc.rgw_gc_queue_list_entries in=46b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32) v8 -- 0x5606236b0100 con 0x5606236477d0 2023-10-05T19:28:55.573 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.566+0000 7fc9de566700 1 -- 172.21.15.167:0/1093196717 <== osd.3 v2:172.21.15.167:6808/30036 39 ==== osd_op_reply(107 gc.24 [call out=15b] v0'0 uv17 ondisk = 0) v8 ==== 149+0+15 (crc 0 0 0) 0x7fc9d0050040 con 0x5606236477d0 2023-10-05T19:28:55.573 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.566+0000 7fc9e0230e40 1 -- 172.21.15.167:0/1093196717 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:108 6.9 6:921ecdb1:gc::gc.25:head [call rgw.gc_list in=25b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32) v8 -- 0x5606236b0100 con 0x5606236477d0 2023-10-05T19:28:55.574 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.566+0000 7fc9de566700 1 -- 172.21.15.167:0/1093196717 <== osd.3 v2:172.21.15.167:6808/30036 40 ==== osd_op_reply(108 gc.25 [call out=15b] v0'0 uv39 ondisk = 0) v8 ==== 149+0+15 (crc 0 0 0) 0x7fc9d0050040 con 0x5606236477d0 2023-10-05T19:28:55.574 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.566+0000 7fc9e0230e40 1 -- 172.21.15.167:0/1093196717 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:109 6.9 6:921ecdb1:gc::gc.25:head [call version.read in=11b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32) v8 -- 0x5606236af0f0 con 0x5606236477d0 2023-10-05T19:28:55.574 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.570+0000 7fc9de566700 1 -- 172.21.15.167:0/1093196717 <== osd.3 v2:172.21.15.167:6808/30036 41 ==== osd_op_reply(109 gc.25 [call out=24b] v0'0 uv39 ondisk = 0) v8 ==== 149+0+24 (crc 0 0 0) 0x7fc9d0050040 con 0x5606236477d0 2023-10-05T19:28:55.574 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.570+0000 7fc9e0230e40 1 -- 172.21.15.167:0/1093196717 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:110 6.9 6:921ecdb1:gc::gc.25:head [call rgw_gc.rgw_gc_queue_list_entries in=46b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32) v8 -- 0x5606236b0100 con 0x5606236477d0 2023-10-05T19:28:55.575 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.570+0000 7fc9de566700 1 -- 172.21.15.167:0/1093196717 <== osd.3 v2:172.21.15.167:6808/30036 42 ==== osd_op_reply(110 gc.25 [call out=15b] v0'0 uv39 ondisk = 0) v8 ==== 149+0+15 (crc 0 0 0) 0x7fc9d0050040 con 0x5606236477d0 2023-10-05T19:28:55.575 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.570+0000 7fc9e0230e40 1 -- 172.21.15.167:0/1093196717 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:111 6.1f 6:fde4dd55:gc::gc.26:head [call rgw.gc_list in=25b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32) v8 -- 0x5606236b0100 con 0x56062363c0d0 2023-10-05T19:28:55.575 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.570+0000 7fc9dd564700 1 -- 172.21.15.167:0/1093196717 <== osd.0 v2:172.21.15.118:6800/27690 20 ==== osd_op_reply(111 gc.26 [call out=15b] v0'0 uv23 ondisk = 0) v8 ==== 149+0+15 (crc 0 0 0) 0x7fc9d4032040 con 0x56062363c0d0 2023-10-05T19:28:55.575 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.570+0000 7fc9e0230e40 1 -- 172.21.15.167:0/1093196717 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:112 6.1f 6:fde4dd55:gc::gc.26:head [call version.read in=11b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32) v8 -- 0x5606236af0f0 con 0x56062363c0d0 2023-10-05T19:28:55.576 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.570+0000 7fc9dd564700 1 -- 172.21.15.167:0/1093196717 <== osd.0 v2:172.21.15.118:6800/27690 21 ==== osd_op_reply(112 gc.26 [call out=24b] v0'0 uv23 ondisk = 0) v8 ==== 149+0+24 (crc 0 0 0) 0x7fc9d4032040 con 0x56062363c0d0 2023-10-05T19:28:55.576 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.570+0000 7fc9e0230e40 1 -- 172.21.15.167:0/1093196717 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:113 6.1f 6:fde4dd55:gc::gc.26:head [call rgw_gc.rgw_gc_queue_list_entries in=46b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32) v8 -- 0x5606236b0100 con 0x56062363c0d0 2023-10-05T19:28:55.577 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.570+0000 7fc9dd564700 1 -- 172.21.15.167:0/1093196717 <== osd.0 v2:172.21.15.118:6800/27690 22 ==== osd_op_reply(113 gc.26 [call out=15b] v0'0 uv23 ondisk = 0) v8 ==== 149+0+15 (crc 0 0 0) 0x7fc9d4032040 con 0x56062363c0d0 2023-10-05T19:28:55.577 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.570+0000 7fc9e0230e40 1 -- 172.21.15.167:0/1093196717 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:114 6.1a 6:5909e671:gc::gc.27:head [call rgw.gc_list in=25b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32) v8 -- 0x5606236b0100 con 0x5606236477d0 2023-10-05T19:28:55.577 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.574+0000 7fc9de566700 1 -- 172.21.15.167:0/1093196717 <== osd.3 v2:172.21.15.167:6808/30036 43 ==== osd_op_reply(114 gc.27 [call out=15b] v0'0 uv21 ondisk = 0) v8 ==== 149+0+15 (crc 0 0 0) 0x7fc9d0050040 con 0x5606236477d0 2023-10-05T19:28:55.578 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.574+0000 7fc9e0230e40 1 -- 172.21.15.167:0/1093196717 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:115 6.1a 6:5909e671:gc::gc.27:head [call version.read in=11b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32) v8 -- 0x5606236af0f0 con 0x5606236477d0 2023-10-05T19:28:55.578 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.574+0000 7fc9de566700 1 -- 172.21.15.167:0/1093196717 <== osd.3 v2:172.21.15.167:6808/30036 44 ==== osd_op_reply(115 gc.27 [call out=24b] v0'0 uv21 ondisk = 0) v8 ==== 149+0+24 (crc 0 0 0) 0x7fc9d0050040 con 0x5606236477d0 2023-10-05T19:28:55.578 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.574+0000 7fc9e0230e40 1 -- 172.21.15.167:0/1093196717 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:116 6.1a 6:5909e671:gc::gc.27:head [call rgw_gc.rgw_gc_queue_list_entries in=46b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32) v8 -- 0x5606236b0100 con 0x5606236477d0 2023-10-05T19:28:55.579 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.574+0000 7fc9de566700 1 -- 172.21.15.167:0/1093196717 <== osd.3 v2:172.21.15.167:6808/30036 45 ==== osd_op_reply(116 gc.27 [call out=15b] v0'0 uv21 ondisk = 0) v8 ==== 149+0+15 (crc 0 0 0) 0x7fc9d0050040 con 0x5606236477d0 2023-10-05T19:28:55.579 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.574+0000 7fc9e0230e40 1 -- 172.21.15.167:0/1093196717 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:117 6.1d 6:b88e3271:gc::gc.28:head [call rgw.gc_list in=25b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32) v8 -- 0x5606236b0100 con 0x56062363c0d0 2023-10-05T19:28:55.579 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.574+0000 7fc9dd564700 1 -- 172.21.15.167:0/1093196717 <== osd.0 v2:172.21.15.118:6800/27690 23 ==== osd_op_reply(117 gc.28 [call out=15b] v0'0 uv36 ondisk = 0) v8 ==== 149+0+15 (crc 0 0 0) 0x7fc9d4032040 con 0x56062363c0d0 2023-10-05T19:28:55.580 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.574+0000 7fc9e0230e40 1 -- 172.21.15.167:0/1093196717 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:118 6.1d 6:b88e3271:gc::gc.28:head [call version.read in=11b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32) v8 -- 0x5606236af0f0 con 0x56062363c0d0 2023-10-05T19:28:55.581 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.574+0000 7fc9dd564700 1 -- 172.21.15.167:0/1093196717 <== osd.0 v2:172.21.15.118:6800/27690 24 ==== osd_op_reply(118 gc.28 [call out=24b] v0'0 uv36 ondisk = 0) v8 ==== 149+0+24 (crc 0 0 0) 0x7fc9d4032040 con 0x56062363c0d0 2023-10-05T19:28:55.581 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.574+0000 7fc9e0230e40 1 -- 172.21.15.167:0/1093196717 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:119 6.1d 6:b88e3271:gc::gc.28:head [call rgw_gc.rgw_gc_queue_list_entries in=46b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32) v8 -- 0x5606236b0100 con 0x56062363c0d0 2023-10-05T19:28:55.582 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.578+0000 7fc9dd564700 1 -- 172.21.15.167:0/1093196717 <== osd.0 v2:172.21.15.118:6800/27690 25 ==== osd_op_reply(119 gc.28 [call out=15b] v0'0 uv36 ondisk = 0) v8 ==== 149+0+15 (crc 0 0 0) 0x7fc9d4032040 con 0x56062363c0d0 2023-10-05T19:28:55.582 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.578+0000 7fc9e0230e40 1 -- 172.21.15.167:0/1093196717 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:120 6.7 6:e47c9322:gc::gc.29:head [call rgw.gc_list in=25b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32) v8 -- 0x5606236b0100 con 0x5606236477d0 2023-10-05T19:28:55.582 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.578+0000 7fc9de566700 1 -- 172.21.15.167:0/1093196717 <== osd.3 v2:172.21.15.167:6808/30036 46 ==== osd_op_reply(120 gc.29 [call out=15b] v0'0 uv49 ondisk = 0) v8 ==== 149+0+15 (crc 0 0 0) 0x7fc9d0050040 con 0x5606236477d0 2023-10-05T19:28:55.582 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.578+0000 7fc9e0230e40 1 -- 172.21.15.167:0/1093196717 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:121 6.7 6:e47c9322:gc::gc.29:head [call version.read in=11b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32) v8 -- 0x5606236af0f0 con 0x5606236477d0 2023-10-05T19:28:55.583 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.578+0000 7fc9de566700 1 -- 172.21.15.167:0/1093196717 <== osd.3 v2:172.21.15.167:6808/30036 47 ==== osd_op_reply(121 gc.29 [call out=24b] v0'0 uv49 ondisk = 0) v8 ==== 149+0+24 (crc 0 0 0) 0x7fc9d0050040 con 0x5606236477d0 2023-10-05T19:28:55.583 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.578+0000 7fc9e0230e40 1 -- 172.21.15.167:0/1093196717 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:122 6.7 6:e47c9322:gc::gc.29:head [call rgw_gc.rgw_gc_queue_list_entries in=46b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32) v8 -- 0x5606236b0100 con 0x5606236477d0 2023-10-05T19:28:55.586 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.578+0000 7fc9de566700 1 -- 172.21.15.167:0/1093196717 <== osd.3 v2:172.21.15.167:6808/30036 48 ==== osd_op_reply(122 gc.29 [call out=15b] v0'0 uv49 ondisk = 0) v8 ==== 149+0+15 (crc 0 0 0) 0x7fc9d0050040 con 0x5606236477d0 2023-10-05T19:28:55.586 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.578+0000 7fc9e0230e40 1 -- 172.21.15.167:0/1093196717 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:123 6.0 6:013d6c33:gc::gc.30:head [call rgw.gc_list in=25b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32) v8 -- 0x5606236b0100 con 0x56062363c0d0 2023-10-05T19:28:55.586 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.578+0000 7fc9dd564700 1 -- 172.21.15.167:0/1093196717 <== osd.0 v2:172.21.15.118:6800/27690 26 ==== osd_op_reply(123 gc.30 [call out=15b] v0'0 uv42 ondisk = 0) v8 ==== 149+0+15 (crc 0 0 0) 0x7fc9d4032040 con 0x56062363c0d0 2023-10-05T19:28:55.586 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.578+0000 7fc9e0230e40 1 -- 172.21.15.167:0/1093196717 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:124 6.0 6:013d6c33:gc::gc.30:head [call version.read in=11b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32) v8 -- 0x5606236af0f0 con 0x56062363c0d0 2023-10-05T19:28:55.586 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.578+0000 7fc9dd564700 1 -- 172.21.15.167:0/1093196717 <== osd.0 v2:172.21.15.118:6800/27690 27 ==== osd_op_reply(124 gc.30 [call out=24b] v0'0 uv42 ondisk = 0) v8 ==== 149+0+24 (crc 0 0 0) 0x7fc9d4032040 con 0x56062363c0d0 2023-10-05T19:28:55.586 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.582+0000 7fc9e0230e40 1 -- 172.21.15.167:0/1093196717 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:125 6.0 6:013d6c33:gc::gc.30:head [call rgw_gc.rgw_gc_queue_list_entries in=46b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32) v8 -- 0x5606236b0100 con 0x56062363c0d0 2023-10-05T19:28:55.587 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.582+0000 7fc9dd564700 1 -- 172.21.15.167:0/1093196717 <== osd.0 v2:172.21.15.118:6800/27690 28 ==== osd_op_reply(125 gc.30 [call out=15b] v0'0 uv42 ondisk = 0) v8 ==== 149+0+15 (crc 0 0 0) 0x7fc9d4032040 con 0x56062363c0d0 2023-10-05T19:28:55.587 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.582+0000 7fc9e0230e40 1 -- 172.21.15.167:0/1093196717 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:126 6.f 6:f24cb4a8:gc::gc.31:head [call rgw.gc_list in=25b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32) v8 -- 0x5606236b0100 con 0x56062364b8f0 2023-10-05T19:28:55.587 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.582+0000 7fc9ddd65700 1 -- 172.21.15.167:0/1093196717 <== osd.2 v2:172.21.15.167:6816/30037 26 ==== osd_op_reply(126 gc.31 [call out=15b] v0'0 uv27 ondisk = 0) v8 ==== 149+0+15 (crc 0 0 0) 0x7fc9d8025020 con 0x56062364b8f0 2023-10-05T19:28:55.587 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.582+0000 7fc9e0230e40 1 -- 172.21.15.167:0/1093196717 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:127 6.f 6:f24cb4a8:gc::gc.31:head [call version.read in=11b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32) v8 -- 0x5606236af0f0 con 0x56062364b8f0 2023-10-05T19:28:55.588 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.582+0000 7fc9ddd65700 1 -- 172.21.15.167:0/1093196717 <== osd.2 v2:172.21.15.167:6816/30037 27 ==== osd_op_reply(127 gc.31 [call out=24b] v0'0 uv27 ondisk = 0) v8 ==== 149+0+24 (crc 0 0 0) 0x7fc9d8025020 con 0x56062364b8f0 2023-10-05T19:28:55.588 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.582+0000 7fc9e0230e40 1 -- 172.21.15.167:0/1093196717 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:128 6.f 6:f24cb4a8:gc::gc.31:head [call rgw_gc.rgw_gc_queue_list_entries in=46b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32) v8 -- 0x5606236b0100 con 0x56062364b8f0 2023-10-05T19:28:55.588 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.582+0000 7fc9ddd65700 1 -- 172.21.15.167:0/1093196717 <== osd.2 v2:172.21.15.167:6816/30037 28 ==== osd_op_reply(128 gc.31 [call out=15b] v0'0 uv27 ondisk = 0) v8 ==== 149+0+15 (crc 0 0 0) 0x7fc9d8025020 con 0x56062364b8f0 2023-10-05T19:28:55.588 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.582+0000 7fc9e0230e40 1 -- 172.21.15.167:0/2053399812 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:102 7.1f 7:f95f44c2:::notify.0:head [watch unwatch cookie 94584363400736] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e32) v8 -- 0x5606235f7b20 con 0x7fc9d801ec10 2023-10-05T19:28:55.592 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.586+0000 7fc9dd564700 1 -- 172.21.15.167:0/2053399812 <== osd.2 v2:172.21.15.167:6816/30037 17 ==== osd_op_reply(102 notify.0 [watch unwatch cookie 94584363400736] v32'197 uv195 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7fc9d4032040 con 0x7fc9d801ec10 2023-10-05T19:28:55.592 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.586+0000 7fc9e0230e40 20 remove_watcher() i=0 2023-10-05T19:28:55.592 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.586+0000 7fc9e0230e40 2 removed watcher, disabling cache 2023-10-05T19:28:55.592 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.586+0000 7fc9e0230e40 1 -- 172.21.15.167:0/2053399812 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:103 7.0 7:05bf5b68:::notify.1:head [watch unwatch cookie 94584363407296] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e32) v8 -- 0x5606235f7b20 con 0x5606235e66f0 2023-10-05T19:28:55.595 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.590+0000 7fc9de566700 1 -- 172.21.15.167:0/2053399812 <== osd.1 v2:172.21.15.167:6800/30035 35 ==== osd_op_reply(103 notify.1 [watch unwatch cookie 94584363407296] v32'197 uv195 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7fc9d0050040 con 0x5606235e66f0 2023-10-05T19:28:55.595 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.590+0000 7fc9e0230e40 20 remove_watcher() i=1 2023-10-05T19:28:55.595 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.590+0000 7fc9e0230e40 1 -- 172.21.15.167:0/2053399812 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:104 7.15 7:a93a5511:::notify.2:head [watch unwatch cookie 94584363409584] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e32) v8 -- 0x5606235f7b20 con 0x560623617470 2023-10-05T19:28:55.598 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.594+0000 7fc9ddd65700 1 -- 172.21.15.167:0/2053399812 <== osd.3 v2:172.21.15.167:6808/30036 24 ==== osd_op_reply(104 notify.2 [watch unwatch cookie 94584363409584] v32'197 uv195 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7fc9d8025020 con 0x560623617470 2023-10-05T19:28:55.598 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.594+0000 7fc9e0230e40 20 remove_watcher() i=2 2023-10-05T19:28:55.598 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.594+0000 7fc9e0230e40 1 -- 172.21.15.167:0/2053399812 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:105 7.e 7:7759931f:::notify.3:head [watch unwatch cookie 94584363415968] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e32) v8 -- 0x5606235f7b20 con 0x7fc9d801ec10 2023-10-05T19:28:55.600 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.594+0000 7fc9dd564700 1 -- 172.21.15.167:0/2053399812 <== osd.2 v2:172.21.15.167:6816/30037 18 ==== osd_op_reply(105 notify.3 [watch unwatch cookie 94584363415968] v32'197 uv195 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7fc9d4032040 con 0x7fc9d801ec10 2023-10-05T19:28:55.601 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.594+0000 7fc9e0230e40 20 remove_watcher() i=3 2023-10-05T19:28:55.601 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.594+0000 7fc9e0230e40 1 -- 172.21.15.167:0/2053399812 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:106 7.d 7:b4812045:::notify.4:head [watch unwatch cookie 94584363417072] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e32) v8 -- 0x5606235f7b20 con 0x5606235e66f0 2023-10-05T19:28:55.604 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.598+0000 7fc9de566700 1 -- 172.21.15.167:0/2053399812 <== osd.1 v2:172.21.15.167:6800/30035 36 ==== osd_op_reply(106 notify.4 [watch unwatch cookie 94584363417072] v32'197 uv195 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7fc9d0050040 con 0x5606235e66f0 2023-10-05T19:28:55.604 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.598+0000 7fc9e0230e40 20 remove_watcher() i=4 2023-10-05T19:28:55.604 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.598+0000 7fc9e0230e40 1 -- 172.21.15.167:0/2053399812 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:107 7.3 7:c609908c:::notify.5:head [watch unwatch cookie 94584363410688] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e32) v8 -- 0x5606235f7b20 con 0x7fc9d4007030 2023-10-05T19:28:55.607 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.602+0000 7fc9de566700 1 -- 172.21.15.167:0/2053399812 <== osd.0 v2:172.21.15.118:6800/27690 29 ==== osd_op_reply(107 notify.5 [watch unwatch cookie 94584363410688] v32'197 uv195 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7fc9d0050040 con 0x7fc9d4007030 2023-10-05T19:28:55.607 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.602+0000 7fc9e0230e40 20 remove_watcher() i=5 2023-10-05T19:28:55.607 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.602+0000 7fc9e0230e40 1 -- 172.21.15.167:0/2053399812 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:108 7.14 7:2b04a3e9:::notify.6:head [watch unwatch cookie 94584363424128] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e32) v8 -- 0x5606235f7b20 con 0x7fc9d4007030 2023-10-05T19:28:55.610 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.606+0000 7fc9de566700 1 -- 172.21.15.167:0/2053399812 <== osd.0 v2:172.21.15.118:6800/27690 30 ==== osd_op_reply(108 notify.6 [watch unwatch cookie 94584363424128] v32'197 uv195 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7fc9d0050040 con 0x7fc9d4007030 2023-10-05T19:28:55.610 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.606+0000 7fc9e0230e40 20 remove_watcher() i=6 2023-10-05T19:28:55.610 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.606+0000 7fc9e0230e40 1 -- 172.21.15.167:0/2053399812 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:109 7.9 7:93e5b521:::notify.7:head [watch unwatch cookie 94584363421552] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e32) v8 -- 0x5606235f7b20 con 0x7fc9d4007030 2023-10-05T19:28:55.613 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.606+0000 7fc9de566700 1 -- 172.21.15.167:0/2053399812 <== osd.0 v2:172.21.15.118:6800/27690 31 ==== osd_op_reply(109 notify.7 [watch unwatch cookie 94584363421552] v32'197 uv195 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7fc9d0050040 con 0x7fc9d4007030 2023-10-05T19:28:55.613 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.606+0000 7fc9e0230e40 20 remove_watcher() i=7 2023-10-05T19:28:55.615 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.610+0000 7fc9aefe5700 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2023-10-05T19:28:55.615 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.610+0000 7fc9e0230e40 1 -- 172.21.15.167:0/2053399812 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:110 6.4 6:22d26bf9:::data_loggenerations_metadata:head [watch unwatch cookie 94584363425232] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e32) v8 -- 0x5606235f7b20 con 0x5606235e66f0 2023-10-05T19:28:55.618 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.614+0000 7fc9de566700 1 -- 172.21.15.167:0/2053399812 <== osd.1 v2:172.21.15.167:6800/30035 37 ==== osd_op_reply(110 data_loggenerations_metadata [watch unwatch cookie 94584363425232] v32'195 uv1 ondisk = 0) v8 ==== 172+0+0 (crc 0 0 0) 0x7fc9d0050040 con 0x5606235e66f0 2023-10-05T19:28:55.619 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.614+0000 7fc9e0230e40 1 -- 172.21.15.167:0/2053399812 >> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] conn(0x7fc9d4007030 msgr2=0x7fc9d401c920 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-05T19:28:55.619 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.614+0000 7fc9e0230e40 1 --2- 172.21.15.167:0/2053399812 >> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] conn(0x7fc9d4007030 0x7fc9d401c920 crc :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-05T19:28:55.619 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.614+0000 7fc9e0230e40 1 -- 172.21.15.167:0/2053399812 >> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] conn(0x5606235e66f0 msgr2=0x5606236139d0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-05T19:28:55.619 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.614+0000 7fc9e0230e40 1 --2- 172.21.15.167:0/2053399812 >> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] conn(0x5606235e66f0 0x5606236139d0 crc :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-05T19:28:55.619 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.614+0000 7fc9e0230e40 1 -- 172.21.15.167:0/2053399812 >> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] conn(0x7fc9d801ec10 msgr2=0x7fc9d801fa80 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-05T19:28:55.619 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.614+0000 7fc9e0230e40 1 --2- 172.21.15.167:0/2053399812 >> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] conn(0x7fc9d801ec10 0x7fc9d801fa80 crc :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-05T19:28:55.619 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.614+0000 7fc9e0230e40 1 -- 172.21.15.167:0/2053399812 >> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] conn(0x560623617470 msgr2=0x560623619850 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-05T19:28:55.619 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.614+0000 7fc9e0230e40 1 --2- 172.21.15.167:0/2053399812 >> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] conn(0x560623617470 0x560623619850 crc :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-05T19:28:55.619 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.614+0000 7fc9e0230e40 1 -- 172.21.15.167:0/2053399812 >> [v2:172.21.15.167:6824/29889,v1:172.21.15.167:6825/29889] conn(0x7fc9c0030c00 msgr2=0x7fc9c00330b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-05T19:28:55.619 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.614+0000 7fc9e0230e40 1 --2- 172.21.15.167:0/2053399812 >> [v2:172.21.15.167:6824/29889,v1:172.21.15.167:6825/29889] conn(0x7fc9c0030c00 0x7fc9c00330b0 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7fc9d4005a90 tx=0x7fc9d4015000 comp rx=0 tx=0).stop 2023-10-05T19:28:55.619 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.614+0000 7fc9e0230e40 1 -- 172.21.15.167:0/2053399812 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5606235543d0 msgr2=0x560623607f10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-05T19:28:55.620 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.614+0000 7fc9de566700 1 -- 172.21.15.167:0/2053399812 reap_dead start 2023-10-05T19:28:55.620 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.614+0000 7fc9e0230e40 1 --2- 172.21.15.167:0/2053399812 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5606235543d0 0x560623607f10 secure :-1 s=READY pgs=527 cs=0 l=1 rev1=1 crypto rx=0x7fc9d8000f80 tx=0x7fc9d8014000 comp rx=0 tx=0).stop 2023-10-05T19:28:55.620 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.614+0000 7fc9e0230e40 1 -- 172.21.15.167:0/2053399812 shutdown_connections 2023-10-05T19:28:55.620 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.614+0000 7fc9e0230e40 1 --2- 172.21.15.167:0/2053399812 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5606235543d0 0x560623607f10 unknown :-1 s=CLOSED pgs=527 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-05T19:28:55.620 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.614+0000 7fc9e0230e40 1 -- 172.21.15.167:0/2053399812 >> 172.21.15.167:0/2053399812 conn(0x5606235da720 msgr2=0x5606235fb780 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-05T19:28:55.620 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.614+0000 7fc9e0230e40 1 -- 172.21.15.167:0/2053399812 shutdown_connections 2023-10-05T19:28:55.620 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.614+0000 7fc9e0230e40 1 -- 172.21.15.167:0/2053399812 wait complete. 2023-10-05T19:28:55.621 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.614+0000 7fc9e0230e40 1 -- 172.21.15.167:0/1093196717 >> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] conn(0x56062363c0d0 msgr2=0x5606236465c0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-05T19:28:55.621 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.614+0000 7fc9e0230e40 1 --2- 172.21.15.167:0/1093196717 >> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] conn(0x56062363c0d0 0x5606236465c0 crc :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-05T19:28:55.621 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.614+0000 7fc9e0230e40 1 -- 172.21.15.167:0/1093196717 >> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] conn(0x5606236404f0 msgr2=0x560623642900 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-05T19:28:55.621 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.614+0000 7fc9e0230e40 1 --2- 172.21.15.167:0/1093196717 >> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] conn(0x5606236404f0 0x560623642900 crc :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-05T19:28:55.621 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.614+0000 7fc9e0230e40 1 -- 172.21.15.167:0/1093196717 >> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] conn(0x56062364b8f0 msgr2=0x56062364dcd0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-05T19:28:55.622 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.614+0000 7fc9e0230e40 1 --2- 172.21.15.167:0/1093196717 >> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] conn(0x56062364b8f0 0x56062364dcd0 crc :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-05T19:28:55.622 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.614+0000 7fc9e0230e40 1 -- 172.21.15.167:0/1093196717 >> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] conn(0x5606236477d0 msgr2=0x560623649bb0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-05T19:28:55.622 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.614+0000 7fc9e0230e40 1 --2- 172.21.15.167:0/1093196717 >> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] conn(0x5606236477d0 0x560623649bb0 crc :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-05T19:28:55.622 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.614+0000 7fc9e0230e40 1 -- 172.21.15.167:0/1093196717 >> [v2:172.21.15.167:6824/29889,v1:172.21.15.167:6825/29889] conn(0x7fc924030c50 msgr2=0x7fc924033100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-05T19:28:55.622 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.614+0000 7fc9e0230e40 1 --2- 172.21.15.167:0/1093196717 >> [v2:172.21.15.167:6824/29889,v1:172.21.15.167:6825/29889] conn(0x7fc924030c50 0x7fc924033100 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x5606236258b0 tx=0x7fc9d0000f90 comp rx=0 tx=0).stop 2023-10-05T19:28:55.622 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.614+0000 7fc9e0230e40 1 -- 172.21.15.167:0/1093196717 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x56062362fd70 msgr2=0x5606236358a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-05T19:28:55.622 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.614+0000 7fc9e0230e40 1 --2- 172.21.15.167:0/1093196717 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x56062362fd70 0x5606236358a0 secure :-1 s=READY pgs=529 cs=0 l=1 rev1=1 crypto rx=0x560623629e50 tx=0x7fc9d402f000 comp rx=0 tx=0).stop 2023-10-05T19:28:55.622 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.614+0000 7fc9ddd65700 1 -- 172.21.15.167:0/1093196717 reap_dead start 2023-10-05T19:28:55.622 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.614+0000 7fc9e0230e40 1 -- 172.21.15.167:0/1093196717 shutdown_connections 2023-10-05T19:28:55.622 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.614+0000 7fc9e0230e40 1 -- 172.21.15.167:0/1093196717 >> 172.21.15.167:0/1093196717 conn(0x56062362b130 msgr2=0x560623629230 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-05T19:28:55.622 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.614+0000 7fc9e0230e40 1 -- 172.21.15.167:0/1093196717 shutdown_connections 2023-10-05T19:28:55.623 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.614+0000 7fc9e0230e40 1 -- 172.21.15.167:0/1093196717 wait complete. 2023-10-05T19:28:55.623 INFO:teuthology.orchestra.run.smithi167.stdout:[] 2023-10-05T19:28:55.626 DEBUG:tasks.util.rgw: json result: [] 2023-10-05T19:28:55.626 INFO:tasks.util.rgw:rgwadmin: client.0 : ['user', 'rm', '--uid', 'foo'] 2023-10-05T19:28:55.626 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'ceph', 'user', 'rm', '--uid', 'foo'] 2023-10-05T19:28:55.627 DEBUG:teuthology.orchestra.run.smithi167:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster ceph user rm --uid foo 2023-10-05T19:28:55.667 INFO:teuthology.orchestra.run.smithi167.stderr:ignoring --setuser ceph since I am not root 2023-10-05T19:28:55.668 INFO:teuthology.orchestra.run.smithi167.stderr:ignoring --setgroup ceph since I am not root 2023-10-05T19:28:55.668 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.662+0000 7f8f4c788e40 1 Processor -- start 2023-10-05T19:28:55.668 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.662+0000 7f8f4c788e40 1 -- start start 2023-10-05T19:28:55.669 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.662+0000 7f8f4c788e40 1 --2- >> v2:172.21.15.118:3300/0 conn(0x5645238b0c20 0x5645238af200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-05T19:28:55.669 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.662+0000 7f8f4c788e40 1 -- --> v1:172.21.15.118:6789/0 -- auth(proto 0 26 bytes epoch 0) v1 -- 0x564523887bd0 con 0x5645238af740 2023-10-05T19:28:55.669 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.662+0000 7f8f4c788e40 1 -- --> v2:172.21.15.118:3300/0 -- mon_getmap magic: 0 v1 -- 0x564523844ee0 con 0x5645238b0c20 2023-10-05T19:28:55.669 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.662+0000 7f8f4a2bd700 1 --2- >> v2:172.21.15.118:3300/0 conn(0x5645238b0c20 0x5645238af200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-05T19:28:55.670 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.662+0000 7f8f49abc700 1 --1- >> v1:172.21.15.118:6789/0 conn(0x5645238af740 0x564523942f50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.118:6789/0 says I am v1:172.21.15.167:48678/0 (socket says 172.21.15.167:48678) 2023-10-05T19:28:55.670 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.662+0000 7f8f49abc700 1 -- 172.21.15.167:0/1892423424 learned_addr learned my addr 172.21.15.167:0/1892423424 (peer_addr_for_me v1:172.21.15.167:0/0) 2023-10-05T19:28:55.670 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.666+0000 7f8f492bb700 1 -- 172.21.15.167:0/1892423424 <== mon.0 v1:172.21.15.118:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1592111011 0 0) 0x564523887bd0 con 0x5645238af740 2023-10-05T19:28:55.670 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.666+0000 7f8f4a2bd700 1 -- 172.21.15.167:0/1892423424 >> v1:172.21.15.118:6789/0 conn(0x5645238af740 legacy=0x564523942f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-05T19:28:55.671 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.666+0000 7f8f4a2bd700 1 -- 172.21.15.167:0/1892423424 --> v2:172.21.15.118:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5645238b10b0 con 0x5645238b0c20 2023-10-05T19:28:55.671 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.666+0000 7f8f4a2bd700 1 --2- 172.21.15.167:0/1892423424 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5645238b0c20 0x5645238af200 secure :-1 s=READY pgs=531 cs=0 l=1 rev1=1 crypto rx=0x7f8f4400a520 tx=0x7f8f44005d40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b0eb40df0635a020 server_cookie=0 in_seq=0 out_seq=0 2023-10-05T19:28:55.672 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.666+0000 7f8f492bb700 1 -- 172.21.15.167:0/1892423424 <== mon.0 v2:172.21.15.118:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f8f4400f590 con 0x5645238b0c20 2023-10-05T19:28:55.672 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.666+0000 7f8f492bb700 1 -- 172.21.15.167:0/1892423424 <== mon.0 v2:172.21.15.118:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8f4400f6f0 con 0x5645238b0c20 2023-10-05T19:28:55.672 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.666+0000 7f8f492bb700 1 -- 172.21.15.167:0/1892423424 <== mon.0 v2:172.21.15.118:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f8f4400f880 con 0x5645238b0c20 2023-10-05T19:28:55.672 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.666+0000 7f8f4c788e40 1 -- 172.21.15.167:0/1892423424 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5645238b0c20 msgr2=0x5645238af200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-05T19:28:55.672 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.666+0000 7f8f4c788e40 1 --2- 172.21.15.167:0/1892423424 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5645238b0c20 0x5645238af200 secure :-1 s=READY pgs=531 cs=0 l=1 rev1=1 crypto rx=0x7f8f4400a520 tx=0x7f8f44005d40 comp rx=0 tx=0).stop 2023-10-05T19:28:55.672 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.666+0000 7f8f4c788e40 1 -- 172.21.15.167:0/1892423424 shutdown_connections 2023-10-05T19:28:55.672 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.666+0000 7f8f4c788e40 1 --2- 172.21.15.167:0/1892423424 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5645238b0c20 0x5645238af200 unknown :-1 s=CLOSED pgs=531 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-05T19:28:55.672 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.666+0000 7f8f4c788e40 1 -- 172.21.15.167:0/1892423424 >> 172.21.15.167:0/1892423424 conn(0x564523936850 msgr2=0x564523938c60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-05T19:28:55.673 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.666+0000 7f8f4c788e40 1 -- 172.21.15.167:0/1892423424 shutdown_connections 2023-10-05T19:28:55.673 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.666+0000 7f8f4c788e40 1 -- 172.21.15.167:0/1892423424 wait complete. 2023-10-05T19:28:55.673 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.666+0000 7f8f4c788e40 1 Processor -- start 2023-10-05T19:28:55.674 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.666+0000 7f8f4c788e40 1 -- start start 2023-10-05T19:28:55.674 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.666+0000 7f8f4c788e40 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5645238af740 0x56452395bec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-05T19:28:55.674 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.666+0000 7f8f4c788e40 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x564523808e30 con 0x5645238af740 2023-10-05T19:28:55.674 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.670+0000 7f8f4a2bd700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5645238af740 0x56452395bec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-05T19:28:55.674 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.670+0000 7f8f4a2bd700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5645238af740 0x56452395bec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.118:3300/0 says I am v2:172.21.15.167:53268/0 (socket says 172.21.15.167:53268) 2023-10-05T19:28:55.674 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.670+0000 7f8f4a2bd700 1 -- 172.21.15.167:0/1094652593 learned_addr learned my addr 172.21.15.167:0/1094652593 (peer_addr_for_me v2:172.21.15.167:0/0) 2023-10-05T19:28:55.675 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.670+0000 7f8f4a2bd700 1 -- 172.21.15.167:0/1094652593 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8f4400a0e0 con 0x5645238af740 2023-10-05T19:28:55.675 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.670+0000 7f8f4a2bd700 1 --2- 172.21.15.167:0/1094652593 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5645238af740 0x56452395bec0 secure :-1 s=READY pgs=532 cs=0 l=1 rev1=1 crypto rx=0x7f8f4400a470 tx=0x7f8f44014000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4d1aeef392110293 server_cookie=0 in_seq=0 out_seq=0 2023-10-05T19:28:55.675 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.670+0000 7f8f43fff700 1 -- 172.21.15.167:0/1094652593 <== mon.0 v2:172.21.15.118:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f8f4400f860 con 0x5645238af740 2023-10-05T19:28:55.675 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.670+0000 7f8f43fff700 1 -- 172.21.15.167:0/1094652593 <== mon.0 v2:172.21.15.118:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8f4400f9e0 con 0x5645238af740 2023-10-05T19:28:55.675 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.670+0000 7f8f43fff700 1 -- 172.21.15.167:0/1094652593 <== mon.0 v2:172.21.15.118:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f8f4400fb90 con 0x5645238af740 2023-10-05T19:28:55.676 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.670+0000 7f8f4c788e40 1 -- 172.21.15.167:0/1094652593 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5645238af740 msgr2=0x56452395bec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-05T19:28:55.676 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.670+0000 7f8f4c788e40 1 --2- 172.21.15.167:0/1094652593 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5645238af740 0x56452395bec0 secure :-1 s=READY pgs=532 cs=0 l=1 rev1=1 crypto rx=0x7f8f4400a470 tx=0x7f8f44014000 comp rx=0 tx=0).stop 2023-10-05T19:28:55.677 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.670+0000 7f8f4c788e40 1 -- 172.21.15.167:0/1094652593 shutdown_connections 2023-10-05T19:28:55.677 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.670+0000 7f8f4c788e40 1 --2- 172.21.15.167:0/1094652593 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5645238af740 0x56452395bec0 unknown :-1 s=CLOSED pgs=532 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-05T19:28:55.677 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.670+0000 7f8f4c788e40 1 -- 172.21.15.167:0/1094652593 >> 172.21.15.167:0/1094652593 conn(0x564523936850 msgr2=0x564523957a80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-05T19:28:55.677 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.670+0000 7f8f4c788e40 1 -- 172.21.15.167:0/1094652593 shutdown_connections 2023-10-05T19:28:55.677 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.670+0000 7f8f4c788e40 1 -- 172.21.15.167:0/1094652593 wait complete. 2023-10-05T19:28:55.677 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.670+0000 7f8f4c788e40 1 Processor -- start 2023-10-05T19:28:55.677 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.670+0000 7f8f4c788e40 1 -- start start 2023-10-05T19:28:55.677 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.670+0000 7f8f4c788e40 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5645238af740 0x564523962b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-05T19:28:55.677 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.670+0000 7f8f4c788e40 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x5645237dbed0 con 0x5645238af740 2023-10-05T19:28:55.677 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.670+0000 7f8f4a2bd700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5645238af740 0x564523962b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-05T19:28:55.677 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.670+0000 7f8f4a2bd700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5645238af740 0x564523962b20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.118:3300/0 says I am v2:172.21.15.167:53274/0 (socket says 172.21.15.167:53274) 2023-10-05T19:28:55.677 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.670+0000 7f8f4a2bd700 1 -- 172.21.15.167:0/1437397465 learned_addr learned my addr 172.21.15.167:0/1437397465 (peer_addr_for_me v2:172.21.15.167:0/0) 2023-10-05T19:28:55.678 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.670+0000 7f8f4a2bd700 1 -- 172.21.15.167:0/1437397465 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5645238b10b0 con 0x5645238af740 2023-10-05T19:28:55.678 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.674+0000 7f8f4a2bd700 1 --2- 172.21.15.167:0/1437397465 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5645238af740 0x564523962b20 secure :-1 s=READY pgs=533 cs=0 l=1 rev1=1 crypto rx=0x7f8f4400a470 tx=0x7f8f4401d000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-05T19:28:55.679 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.674+0000 7f8f427fc700 1 -- 172.21.15.167:0/1437397465 <== mon.0 v2:172.21.15.118:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f8f4401ce00 con 0x5645238af740 2023-10-05T19:28:55.679 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.674+0000 7f8f427fc700 1 -- 172.21.15.167:0/1437397465 <== mon.0 v2:172.21.15.118:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8f44010630 con 0x5645238af740 2023-10-05T19:28:55.679 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.674+0000 7f8f4c788e40 1 -- 172.21.15.167:0/1437397465 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x564523963060 con 0x5645238af740 2023-10-05T19:28:55.679 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.674+0000 7f8f427fc700 1 -- 172.21.15.167:0/1437397465 <== mon.0 v2:172.21.15.118:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f8f44010820 con 0x5645238af740 2023-10-05T19:28:55.679 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.674+0000 7f8f4c788e40 1 -- 172.21.15.167:0/1437397465 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x5645239634a0 con 0x5645238af740 2023-10-05T19:28:55.679 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.674+0000 7f8f4c788e40 20 rados->read ofs=0 len=0 2023-10-05T19:28:55.680 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.674+0000 7f8f427fc700 1 -- 172.21.15.167:0/1437397465 <== mon.0 v2:172.21.15.118:3300/0 4 ==== mgrmap(e 5) v1 ==== 40329+0+0 (secure 0 0 0) 0x7f8f44010a90 con 0x5645238af740 2023-10-05T19:28:55.680 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.674+0000 7f8f427fc700 1 --2- 172.21.15.167:0/1437397465 >> [v2:172.21.15.167:6824/29889,v1:172.21.15.167:6825/29889] conn(0x7f8f2c030c00 0x7f8f2c0330b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-05T19:28:55.680 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.674+0000 7f8f427fc700 1 -- 172.21.15.167:0/1437397465 <== mon.0 v2:172.21.15.118:3300/0 5 ==== osd_map(32..32 src has 1..32) v4 ==== 6260+0+0 (secure 0 0 0) 0x7f8f4402b070 con 0x5645238af740 2023-10-05T19:28:55.681 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.674+0000 7f8f49abc700 1 --2- 172.21.15.167:0/1437397465 >> [v2:172.21.15.167:6824/29889,v1:172.21.15.167:6825/29889] conn(0x7f8f2c030c00 0x7f8f2c0330b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-05T19:28:55.681 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.674+0000 7f8f4c788e40 1 --2- 172.21.15.167:0/1437397465 >> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] conn(0x5645238b0c20 0x56452396e5b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-05T19:28:55.681 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.674+0000 7f8f4c788e40 1 -- 172.21.15.167:0/1437397465 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:1 5.12 5:49953fa1:::default.realm:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32) v8 -- 0x56452396ec60 con 0x5645238b0c20 2023-10-05T19:28:55.681 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.674+0000 7f8f4aabe700 1 --2- 172.21.15.167:0/1437397465 >> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] conn(0x5645238b0c20 0x56452396e5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-05T19:28:55.682 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.678+0000 7f8f49abc700 1 --2- 172.21.15.167:0/1437397465 >> [v2:172.21.15.167:6824/29889,v1:172.21.15.167:6825/29889] conn(0x7f8f2c030c00 0x7f8f2c0330b0 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7f8f340011e0 tx=0x7f8f340013a0 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-05T19:28:55.682 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.678+0000 7f8f4aabe700 1 --2- 172.21.15.167:0/1437397465 >> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] conn(0x5645238b0c20 0x56452396e5b0 crc :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-05T19:28:55.683 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.678+0000 7f8f4aabe700 1 -- 172.21.15.167:0/1437397465 <== osd.1 v2:172.21.15.167:6800/30035 1 ==== osd_op_reply(1 default.realm [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 ==== 157+0+0 (crc 0 0 0) 0x7f8f3c00a500 con 0x5645238b0c20 2023-10-05T19:28:55.683 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.678+0000 7f8f4c788e40 20 rados_obj.operate() r=-2 bl.length=0 2023-10-05T19:28:55.683 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.678+0000 7f8f4c788e40 20 realm 2023-10-05T19:28:55.683 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.678+0000 7f8f4c788e40 20 rados->read ofs=0 len=0 2023-10-05T19:28:55.683 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.678+0000 7f8f4c788e40 1 -- 172.21.15.167:0/1437397465 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:2 5.12 5:49953fa1:::default.realm:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32) v8 -- 0x56452396bc10 con 0x5645238b0c20 2023-10-05T19:28:55.683 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.678+0000 7f8f4aabe700 1 -- 172.21.15.167:0/1437397465 <== osd.1 v2:172.21.15.167:6800/30035 2 ==== osd_op_reply(2 default.realm [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 ==== 157+0+0 (crc 0 0 0) 0x7f8f3c00a500 con 0x5645238b0c20 2023-10-05T19:28:55.683 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.678+0000 7f8f4c788e40 20 rados_obj.operate() r=-2 bl.length=0 2023-10-05T19:28:55.684 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.678+0000 7f8f4c788e40 4 RGWPeriod::init failed to init realm id : (2) No such file or directory 2023-10-05T19:28:55.684 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.678+0000 7f8f4c788e40 20 rados->read ofs=0 len=0 2023-10-05T19:28:55.684 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.678+0000 7f8f4c788e40 1 -- 172.21.15.167:0/1437397465 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:3 5.12 5:49953fa1:::default.realm:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32) v8 -- 0x56452396f740 con 0x5645238b0c20 2023-10-05T19:28:55.685 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.678+0000 7f8f4aabe700 1 -- 172.21.15.167:0/1437397465 <== osd.1 v2:172.21.15.167:6800/30035 3 ==== osd_op_reply(3 default.realm [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 ==== 157+0+0 (crc 0 0 0) 0x7f8f3c00a500 con 0x5645238b0c20 2023-10-05T19:28:55.685 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.678+0000 7f8f4c788e40 20 rados_obj.operate() r=-2 bl.length=0 2023-10-05T19:28:55.685 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.678+0000 7f8f4c788e40 20 rados->read ofs=0 len=0 2023-10-05T19:28:55.686 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.678+0000 7f8f4c788e40 1 -- 172.21.15.167:0/1437397465 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:4 5.f 5:f43b6ece:::zone_names.default:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32) v8 -- 0x56452396f350 con 0x5645238b0c20 2023-10-05T19:28:55.686 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.678+0000 7f8f4aabe700 1 -- 172.21.15.167:0/1437397465 <== osd.1 v2:172.21.15.167:6800/30035 4 ==== osd_op_reply(4 zone_names.default [read 0~46 out=46b] v0'0 uv1 ondisk = 0) v8 ==== 162+0+46 (crc 0 0 0) 0x7f8f3c00a500 con 0x5645238b0c20 2023-10-05T19:28:55.686 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.678+0000 7f8f4c788e40 20 rados_obj.operate() r=0 bl.length=46 2023-10-05T19:28:55.686 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.678+0000 7f8f4c788e40 20 rados->read ofs=0 len=0 2023-10-05T19:28:55.686 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.678+0000 7f8f4c788e40 1 -- 172.21.15.167:0/1437397465 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:5 5.1d 5:b8374eb4:::zone_info.9a9df965-6612-45d4-a45f-6e3c889bacc2:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32) v8 -- 0x5645239705d0 con 0x5645238b0c20 2023-10-05T19:28:55.686 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.678+0000 7f8f4aabe700 1 -- 172.21.15.167:0/1437397465 <== osd.1 v2:172.21.15.167:6800/30035 5 ==== osd_op_reply(5 zone_info.9a9df965-6612-45d4-a45f-6e3c889bacc2 [read 0~874 out=874b] v0'0 uv2 ondisk = 0) v8 ==== 190+0+874 (crc 0 0 0) 0x7f8f3c00a500 con 0x5645238b0c20 2023-10-05T19:28:55.686 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.678+0000 7f8f4c788e40 20 rados_obj.operate() r=0 bl.length=874 2023-10-05T19:28:55.686 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.678+0000 7f8f4c788e40 20 searching for the correct realm 2023-10-05T19:28:55.686 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.678+0000 7f8f4c788e40 1 --2- 172.21.15.167:0/1437397465 >> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] conn(0x564523972390 0x564523974770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-05T19:28:55.686 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.678+0000 7f8f4c788e40 1 -- 172.21.15.167:0/1437397465 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:6 5.0 5:00000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x564523974cb0 con 0x564523972390 2023-10-05T19:28:55.686 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.678+0000 7f8f4a2bd700 1 --2- 172.21.15.167:0/1437397465 >> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] conn(0x564523972390 0x564523974770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-05T19:28:55.687 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.682+0000 7f8f4a2bd700 1 --2- 172.21.15.167:0/1437397465 >> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] conn(0x564523972390 0x564523974770 crc :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-05T19:28:55.687 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.682+0000 7f8f4a2bd700 1 -- 172.21.15.167:0/1437397465 <== osd.3 v2:172.21.15.167:6808/30036 1 ==== osd_op_reply(6 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f8f4403ad70 con 0x564523972390 2023-10-05T19:28:55.687 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.682+0000 7f8f4a2bd700 1 --2- 172.21.15.167:0/1437397465 >> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] conn(0x7f8f4401a3c0 0x7f8f44039540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-05T19:28:55.687 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.682+0000 7f8f4a2bd700 1 -- 172.21.15.167:0/1437397465 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:7 5.10 5:08000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x564523974cb0 con 0x7f8f4401a3c0 2023-10-05T19:28:55.688 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.682+0000 7f8f49abc700 1 --2- 172.21.15.167:0/1437397465 >> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] conn(0x7f8f4401a3c0 0x7f8f44039540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-05T19:28:55.688 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.682+0000 7f8f49abc700 1 --2- 172.21.15.167:0/1437397465 >> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] conn(0x7f8f4401a3c0 0x7f8f44039540 crc :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-05T19:28:55.689 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.682+0000 7f8f49abc700 1 -- 172.21.15.167:0/1437397465 <== osd.2 v2:172.21.15.167:6816/30037 1 ==== osd_op_reply(7 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f8f34006070 con 0x7f8f4401a3c0 2023-10-05T19:28:55.689 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.682+0000 7f8f49abc700 1 --2- 172.21.15.167:0/1437397465 >> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] conn(0x7f8f3400d8e0 0x7f8f3400c360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-05T19:28:55.689 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.682+0000 7f8f49abc700 1 -- 172.21.15.167:0/1437397465 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:8 5.8 5:10000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x564523974cb0 con 0x7f8f3400d8e0 2023-10-05T19:28:55.689 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.682+0000 7f8f4aabe700 1 --2- 172.21.15.167:0/1437397465 >> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] conn(0x7f8f3400d8e0 0x7f8f3400c360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-05T19:28:55.690 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.686+0000 7f8f4aabe700 1 --2- 172.21.15.167:0/1437397465 >> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] conn(0x7f8f3400d8e0 0x7f8f3400c360 crc :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-05T19:28:55.691 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.686+0000 7f8f4aabe700 1 -- 172.21.15.167:0/1437397465 <== osd.0 v2:172.21.15.118:6800/27690 1 ==== osd_op_reply(8 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f8f3c014040 con 0x7f8f3400d8e0 2023-10-05T19:28:55.691 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.686+0000 7f8f4aabe700 1 -- 172.21.15.167:0/1437397465 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:9 5.18 5:18000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x564523974cb0 con 0x5645238b0c20 2023-10-05T19:28:55.692 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.686+0000 7f8f4aabe700 1 -- 172.21.15.167:0/1437397465 <== osd.1 v2:172.21.15.167:6800/30035 6 ==== osd_op_reply(9 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f8f3c014040 con 0x5645238b0c20 2023-10-05T19:28:55.692 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.686+0000 7f8f4aabe700 1 -- 172.21.15.167:0/1437397465 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:10 5.4 5:20000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x564523974cb0 con 0x5645238b0c20 2023-10-05T19:28:55.692 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.686+0000 7f8f4aabe700 1 -- 172.21.15.167:0/1437397465 <== osd.1 v2:172.21.15.167:6800/30035 7 ==== osd_op_reply(10 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f8f3c014040 con 0x5645238b0c20 2023-10-05T19:28:55.692 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.686+0000 7f8f4aabe700 1 -- 172.21.15.167:0/1437397465 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:11 5.14 5:28000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x564523974cb0 con 0x7f8f3400d8e0 2023-10-05T19:28:55.693 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.686+0000 7f8f4aabe700 1 -- 172.21.15.167:0/1437397465 <== osd.0 v2:172.21.15.118:6800/27690 2 ==== osd_op_reply(11 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f8f3c014040 con 0x7f8f3400d8e0 2023-10-05T19:28:55.693 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.686+0000 7f8f4aabe700 1 -- 172.21.15.167:0/1437397465 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:12 5.c 5:30000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x564523974cb0 con 0x7f8f3400d8e0 2023-10-05T19:28:55.694 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.690+0000 7f8f4aabe700 1 -- 172.21.15.167:0/1437397465 <== osd.0 v2:172.21.15.118:6800/27690 3 ==== osd_op_reply(12 [pgnls start_epoch 32 out=112b] v23'1 uv1 ondisk = 1) v8 ==== 144+0+112 (crc 0 0 0) 0x7f8f3c014040 con 0x7f8f3400d8e0 2023-10-05T19:28:55.694 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.690+0000 7f8f4aabe700 1 -- 172.21.15.167:0/1437397465 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:13 5.1c 5:38000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x564523974cb0 con 0x7f8f4401a3c0 2023-10-05T19:28:55.695 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.690+0000 7f8f49abc700 1 -- 172.21.15.167:0/1437397465 <== osd.2 v2:172.21.15.167:6816/30037 2 ==== osd_op_reply(13 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f8f34006070 con 0x7f8f4401a3c0 2023-10-05T19:28:55.695 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.690+0000 7f8f49abc700 1 -- 172.21.15.167:0/1437397465 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:14 5.2 5:40000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x564523974cb0 con 0x564523972390 2023-10-05T19:28:55.696 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.690+0000 7f8f4a2bd700 1 -- 172.21.15.167:0/1437397465 <== osd.3 v2:172.21.15.167:6808/30036 2 ==== osd_op_reply(14 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f8f4403ad70 con 0x564523972390 2023-10-05T19:28:55.696 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.690+0000 7f8f4a2bd700 1 -- 172.21.15.167:0/1437397465 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:15 5.12 5:48000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x564523974cb0 con 0x5645238b0c20 2023-10-05T19:28:55.696 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.690+0000 7f8f4aabe700 1 -- 172.21.15.167:0/1437397465 <== osd.1 v2:172.21.15.167:6800/30035 8 ==== osd_op_reply(15 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f8f3c014040 con 0x5645238b0c20 2023-10-05T19:28:55.697 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.690+0000 7f8f4aabe700 1 -- 172.21.15.167:0/1437397465 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:16 5.a 5:50000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x564523974cb0 con 0x7f8f3400d8e0 2023-10-05T19:28:55.697 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.694+0000 7f8f4aabe700 1 -- 172.21.15.167:0/1437397465 <== osd.0 v2:172.21.15.118:6800/27690 4 ==== osd_op_reply(16 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f8f3c022040 con 0x7f8f3400d8e0 2023-10-05T19:28:55.697 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.694+0000 7f8f4aabe700 1 -- 172.21.15.167:0/1437397465 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:17 5.1a 5:58000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x564523974cb0 con 0x5645238b0c20 2023-10-05T19:28:55.698 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.694+0000 7f8f4aabe700 1 -- 172.21.15.167:0/1437397465 <== osd.1 v2:172.21.15.167:6800/30035 9 ==== osd_op_reply(17 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f8f3c022040 con 0x5645238b0c20 2023-10-05T19:28:55.698 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.694+0000 7f8f4aabe700 1 -- 172.21.15.167:0/1437397465 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:18 5.6 5:60000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x564523974cb0 con 0x7f8f4401a3c0 2023-10-05T19:28:55.699 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.694+0000 7f8f49abc700 1 -- 172.21.15.167:0/1437397465 <== osd.2 v2:172.21.15.167:6816/30037 3 ==== osd_op_reply(18 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f8f34006070 con 0x7f8f4401a3c0 2023-10-05T19:28:55.699 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.694+0000 7f8f49abc700 1 -- 172.21.15.167:0/1437397465 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:19 5.16 5:68000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x564523974cb0 con 0x564523972390 2023-10-05T19:28:55.700 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.694+0000 7f8f4a2bd700 1 -- 172.21.15.167:0/1437397465 <== osd.3 v2:172.21.15.167:6808/30036 3 ==== osd_op_reply(19 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f8f4403ad70 con 0x564523972390 2023-10-05T19:28:55.700 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.694+0000 7f8f4a2bd700 1 -- 172.21.15.167:0/1437397465 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:20 5.e 5:70000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x564523974cb0 con 0x564523972390 2023-10-05T19:28:55.701 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.694+0000 7f8f4a2bd700 1 -- 172.21.15.167:0/1437397465 <== osd.3 v2:172.21.15.167:6808/30036 4 ==== osd_op_reply(20 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f8f4403ad70 con 0x564523972390 2023-10-05T19:28:55.701 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.694+0000 7f8f4a2bd700 1 -- 172.21.15.167:0/1437397465 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:21 5.1e 5:78000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x564523974cb0 con 0x7f8f3400d8e0 2023-10-05T19:28:55.702 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.698+0000 7f8f4aabe700 1 -- 172.21.15.167:0/1437397465 <== osd.0 v2:172.21.15.118:6800/27690 5 ==== osd_op_reply(21 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f8f3c022040 con 0x7f8f3400d8e0 2023-10-05T19:28:55.702 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.698+0000 7f8f4aabe700 1 -- 172.21.15.167:0/1437397465 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:22 5.1 5:80000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x564523974cb0 con 0x564523972390 2023-10-05T19:28:55.703 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.698+0000 7f8f4a2bd700 1 -- 172.21.15.167:0/1437397465 <== osd.3 v2:172.21.15.167:6808/30036 5 ==== osd_op_reply(22 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f8f4403ad70 con 0x564523972390 2023-10-05T19:28:55.703 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.698+0000 7f8f4a2bd700 1 -- 172.21.15.167:0/1437397465 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:23 5.11 5:88000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x564523974cb0 con 0x564523972390 2023-10-05T19:28:55.703 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.698+0000 7f8f4a2bd700 1 -- 172.21.15.167:0/1437397465 <== osd.3 v2:172.21.15.167:6808/30036 6 ==== osd_op_reply(23 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f8f4403ad70 con 0x564523972390 2023-10-05T19:28:55.704 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.698+0000 7f8f4a2bd700 1 -- 172.21.15.167:0/1437397465 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:24 5.9 5:90000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x564523974cb0 con 0x5645238b0c20 2023-10-05T19:28:55.704 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.698+0000 7f8f4aabe700 1 -- 172.21.15.167:0/1437397465 <== osd.1 v2:172.21.15.167:6800/30035 10 ==== osd_op_reply(24 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f8f3c022040 con 0x5645238b0c20 2023-10-05T19:28:55.705 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.698+0000 7f8f4aabe700 1 -- 172.21.15.167:0/1437397465 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:25 5.19 5:98000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x564523974cb0 con 0x5645238b0c20 2023-10-05T19:28:55.705 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.698+0000 7f8f4aabe700 1 -- 172.21.15.167:0/1437397465 <== osd.1 v2:172.21.15.167:6800/30035 11 ==== osd_op_reply(25 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f8f3c022040 con 0x5645238b0c20 2023-10-05T19:28:55.705 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.698+0000 7f8f4aabe700 1 -- 172.21.15.167:0/1437397465 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:26 5.5 5:a0000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x564523974cb0 con 0x7f8f3400d8e0 2023-10-05T19:28:55.706 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.702+0000 7f8f4aabe700 1 -- 172.21.15.167:0/1437397465 <== osd.0 v2:172.21.15.118:6800/27690 6 ==== osd_op_reply(26 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f8f3c022040 con 0x7f8f3400d8e0 2023-10-05T19:28:55.706 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.702+0000 7f8f4aabe700 1 -- 172.21.15.167:0/1437397465 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:27 5.15 5:a8000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x564523974cb0 con 0x564523972390 2023-10-05T19:28:55.707 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.702+0000 7f8f4a2bd700 1 -- 172.21.15.167:0/1437397465 <== osd.3 v2:172.21.15.167:6808/30036 7 ==== osd_op_reply(27 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f8f4403ad70 con 0x564523972390 2023-10-05T19:28:55.708 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.702+0000 7f8f4a2bd700 1 -- 172.21.15.167:0/1437397465 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:28 5.d 5:b0000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x564523974cb0 con 0x7f8f4401a3c0 2023-10-05T19:28:55.708 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.702+0000 7f8f49abc700 1 -- 172.21.15.167:0/1437397465 <== osd.2 v2:172.21.15.167:6816/30037 4 ==== osd_op_reply(28 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f8f34006070 con 0x7f8f4401a3c0 2023-10-05T19:28:55.709 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.702+0000 7f8f49abc700 1 -- 172.21.15.167:0/1437397465 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:29 5.1d 5:b8000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x564523974cb0 con 0x5645238b0c20 2023-10-05T19:28:55.709 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.702+0000 7f8f4aabe700 1 -- 172.21.15.167:0/1437397465 <== osd.1 v2:172.21.15.167:6800/30035 12 ==== osd_op_reply(29 [pgnls start_epoch 32 out=107b] v30'2 uv2 ondisk = 1) v8 ==== 144+0+107 (crc 0 0 0) 0x7f8f3c022040 con 0x5645238b0c20 2023-10-05T19:28:55.709 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.706+0000 7f8f4aabe700 1 -- 172.21.15.167:0/1437397465 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:30 5.3 5:c0000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x564523974cb0 con 0x7f8f3400d8e0 2023-10-05T19:28:55.710 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.706+0000 7f8f4aabe700 1 -- 172.21.15.167:0/1437397465 <== osd.0 v2:172.21.15.118:6800/27690 7 ==== osd_op_reply(30 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f8f3c022040 con 0x7f8f3400d8e0 2023-10-05T19:28:55.710 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.706+0000 7f8f4aabe700 1 -- 172.21.15.167:0/1437397465 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:31 5.13 5:c8000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x564523974cb0 con 0x564523972390 2023-10-05T19:28:55.712 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.706+0000 7f8f4a2bd700 1 -- 172.21.15.167:0/1437397465 <== osd.3 v2:172.21.15.167:6808/30036 8 ==== osd_op_reply(31 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f8f4403ad70 con 0x564523972390 2023-10-05T19:28:55.712 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.706+0000 7f8f4a2bd700 1 -- 172.21.15.167:0/1437397465 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:32 5.b 5:d0000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x564523974cb0 con 0x7f8f3400d8e0 2023-10-05T19:28:55.712 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.706+0000 7f8f4aabe700 1 -- 172.21.15.167:0/1437397465 <== osd.0 v2:172.21.15.118:6800/27690 8 ==== osd_op_reply(32 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f8f3c022040 con 0x7f8f3400d8e0 2023-10-05T19:28:55.713 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.706+0000 7f8f4aabe700 1 -- 172.21.15.167:0/1437397465 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:33 5.1b 5:d8000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x564523974cb0 con 0x7f8f4401a3c0 2023-10-05T19:28:55.713 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.706+0000 7f8f49abc700 1 -- 172.21.15.167:0/1437397465 <== osd.2 v2:172.21.15.167:6816/30037 5 ==== osd_op_reply(33 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f8f34006070 con 0x7f8f4401a3c0 2023-10-05T19:28:55.713 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.706+0000 7f8f49abc700 1 -- 172.21.15.167:0/1437397465 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:34 5.7 5:e0000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x564523974cb0 con 0x564523972390 2023-10-05T19:28:55.714 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.710+0000 7f8f4a2bd700 1 -- 172.21.15.167:0/1437397465 <== osd.3 v2:172.21.15.167:6808/30036 9 ==== osd_op_reply(34 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f8f4403ad70 con 0x564523972390 2023-10-05T19:28:55.714 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.710+0000 7f8f4a2bd700 1 -- 172.21.15.167:0/1437397465 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:35 5.17 5:e8000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x564523974cb0 con 0x7f8f3400d8e0 2023-10-05T19:28:55.715 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.710+0000 7f8f4aabe700 1 -- 172.21.15.167:0/1437397465 <== osd.0 v2:172.21.15.118:6800/27690 9 ==== osd_op_reply(35 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f8f3c022040 con 0x7f8f3400d8e0 2023-10-05T19:28:55.715 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.710+0000 7f8f4aabe700 1 -- 172.21.15.167:0/1437397465 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:36 5.f 5:f0000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x564523974cb0 con 0x5645238b0c20 2023-10-05T19:28:55.716 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.710+0000 7f8f4aabe700 1 -- 172.21.15.167:0/1437397465 <== osd.1 v2:172.21.15.167:6800/30035 13 ==== osd_op_reply(36 [pgnls start_epoch 32 out=115b] v23'2 uv2 ondisk = 1) v8 ==== 144+0+115 (crc 0 0 0) 0x7f8f3c02f080 con 0x5645238b0c20 2023-10-05T19:28:55.716 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.710+0000 7f8f4aabe700 1 -- 172.21.15.167:0/1437397465 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:37 5.1f 5:f8000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x564523974cb0 con 0x7f8f4401a3c0 2023-10-05T19:28:55.717 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.710+0000 7f8f49abc700 1 -- 172.21.15.167:0/1437397465 <== osd.2 v2:172.21.15.167:6816/30037 6 ==== osd_op_reply(37 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f8f34006070 con 0x7f8f4401a3c0 2023-10-05T19:28:55.717 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.710+0000 7f8f4c788e40 20 RGWRados::pool_iterate: got zonegroup_info.4f1f719d-9648-417b-b10c-e5d939825419 2023-10-05T19:28:55.717 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.710+0000 7f8f4c788e40 20 RGWRados::pool_iterate: got zone_info.9a9df965-6612-45d4-a45f-6e3c889bacc2 2023-10-05T19:28:55.718 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.710+0000 7f8f4c788e40 20 RGWRados::pool_iterate: got zone_names.default 2023-10-05T19:28:55.718 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.710+0000 7f8f4c788e40 20 RGWRados::pool_iterate: got zonegroups_names.default 2023-10-05T19:28:55.718 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.710+0000 7f8f4c788e40 20 rados->read ofs=0 len=0 2023-10-05T19:28:55.718 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.710+0000 7f8f4c788e40 1 -- 172.21.15.167:0/1437397465 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:38 5.12 5:49953fa1:::default.realm:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32) v8 -- 0x564523971140 con 0x5645238b0c20 2023-10-05T19:28:55.719 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.714+0000 7f8f4aabe700 1 -- 172.21.15.167:0/1437397465 <== osd.1 v2:172.21.15.167:6800/30035 14 ==== osd_op_reply(38 default.realm [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 ==== 157+0+0 (crc 0 0 0) 0x7f8f3c02f080 con 0x5645238b0c20 2023-10-05T19:28:55.719 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.714+0000 7f8f4c788e40 20 rados_obj.operate() r=-2 bl.length=0 2023-10-05T19:28:55.720 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.714+0000 7f8f4c788e40 20 rados->read ofs=0 len=0 2023-10-05T19:28:55.720 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.714+0000 7f8f4c788e40 1 -- 172.21.15.167:0/1437397465 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:39 5.f 5:f4c53578:::zonegroups_names.default:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32) v8 -- 0x564523971a30 con 0x5645238b0c20 2023-10-05T19:28:55.720 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.714+0000 7f8f4aabe700 1 -- 172.21.15.167:0/1437397465 <== osd.1 v2:172.21.15.167:6800/30035 15 ==== osd_op_reply(39 zonegroups_names.default [read 0~46 out=46b] v0'0 uv2 ondisk = 0) v8 ==== 168+0+46 (crc 0 0 0) 0x7f8f3c02f080 con 0x5645238b0c20 2023-10-05T19:28:55.720 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.714+0000 7f8f4c788e40 20 rados_obj.operate() r=0 bl.length=46 2023-10-05T19:28:55.720 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.714+0000 7f8f4c788e40 20 rados->read ofs=0 len=0 2023-10-05T19:28:55.721 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.714+0000 7f8f4c788e40 1 -- 172.21.15.167:0/1437397465 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:40 5.c 5:30d299a5:::zonegroup_info.4f1f719d-9648-417b-b10c-e5d939825419:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32) v8 -- 0x5645239750f0 con 0x7f8f3400d8e0 2023-10-05T19:28:55.721 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.714+0000 7f8f4aabe700 1 -- 172.21.15.167:0/1437397465 <== osd.0 v2:172.21.15.118:6800/27690 10 ==== osd_op_reply(40 zonegroup_info.4f1f719d-9648-417b-b10c-e5d939825419 [read 0~362 out=362b] v0'0 uv1 ondisk = 0) v8 ==== 195+0+362 (crc 0 0 0) 0x7f8f3c02f080 con 0x7f8f3400d8e0 2023-10-05T19:28:55.721 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.714+0000 7f8f4c788e40 20 rados_obj.operate() r=0 bl.length=362 2023-10-05T19:28:55.721 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.714+0000 7f8f4c788e40 20 rados->read ofs=0 len=0 2023-10-05T19:28:55.721 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.714+0000 7f8f4c788e40 1 -- 172.21.15.167:0/1437397465 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:41 5.13 5:cbbf6c60:::converted:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32) v8 -- 0x5645239750f0 con 0x564523972390 2023-10-05T19:28:55.722 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.714+0000 7f8f4a2bd700 1 -- 172.21.15.167:0/1437397465 <== osd.3 v2:172.21.15.167:6808/30036 10 ==== osd_op_reply(41 converted [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 ==== 153+0+0 (crc 0 0 0) 0x7f8f4403ad70 con 0x564523972390 2023-10-05T19:28:55.722 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.714+0000 7f8f4c788e40 20 rados_obj.operate() r=-2 bl.length=0 2023-10-05T19:28:55.723 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.714+0000 7f8f4c788e40 20 rados->read ofs=0 len=0 2023-10-05T19:28:55.723 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.714+0000 7f8f4c788e40 1 -- 172.21.15.167:0/1437397465 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:42 5.12 5:49953fa1:::default.realm:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32) v8 -- 0x5645239754a0 con 0x5645238b0c20 2023-10-05T19:28:55.723 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.714+0000 7f8f4aabe700 1 -- 172.21.15.167:0/1437397465 <== osd.1 v2:172.21.15.167:6800/30035 16 ==== osd_op_reply(42 default.realm [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 ==== 157+0+0 (crc 0 0 0) 0x7f8f3c02f080 con 0x5645238b0c20 2023-10-05T19:28:55.723 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.714+0000 7f8f4c788e40 20 rados_obj.operate() r=-2 bl.length=0 2023-10-05T19:28:55.723 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.714+0000 7f8f4c788e40 20 rados->read ofs=0 len=0 2023-10-05T19:28:55.723 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.714+0000 7f8f4c788e40 1 -- 172.21.15.167:0/1437397465 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:43 5.f 5:f4c53578:::zonegroups_names.default:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32) v8 -- 0x5645239758d0 con 0x5645238b0c20 2023-10-05T19:28:55.724 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.718+0000 7f8f4aabe700 1 -- 172.21.15.167:0/1437397465 <== osd.1 v2:172.21.15.167:6800/30035 17 ==== osd_op_reply(43 zonegroups_names.default [read 0~46 out=46b] v0'0 uv2 ondisk = 0) v8 ==== 168+0+46 (crc 0 0 0) 0x7f8f3c02f080 con 0x5645238b0c20 2023-10-05T19:28:55.724 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.718+0000 7f8f4c788e40 20 rados_obj.operate() r=0 bl.length=46 2023-10-05T19:28:55.724 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.718+0000 7f8f4c788e40 1 -- 172.21.15.167:0/1437397465 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:44 5.0 5:00000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x5645239768e0 con 0x564523972390 2023-10-05T19:28:55.725 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.718+0000 7f8f4a2bd700 1 -- 172.21.15.167:0/1437397465 <== osd.3 v2:172.21.15.167:6808/30036 11 ==== osd_op_reply(44 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f8f4403ad70 con 0x564523972390 2023-10-05T19:28:55.725 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.718+0000 7f8f4a2bd700 1 -- 172.21.15.167:0/1437397465 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:45 5.10 5:08000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x5645239768e0 con 0x7f8f4401a3c0 2023-10-05T19:28:55.725 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.718+0000 7f8f49abc700 1 -- 172.21.15.167:0/1437397465 <== osd.2 v2:172.21.15.167:6816/30037 7 ==== osd_op_reply(45 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f8f34006070 con 0x7f8f4401a3c0 2023-10-05T19:28:55.725 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.718+0000 7f8f49abc700 1 -- 172.21.15.167:0/1437397465 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:46 5.8 5:10000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x5645239768e0 con 0x7f8f3400d8e0 2023-10-05T19:28:55.726 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.718+0000 7f8f4aabe700 1 -- 172.21.15.167:0/1437397465 <== osd.0 v2:172.21.15.118:6800/27690 11 ==== osd_op_reply(46 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f8f3c02f080 con 0x7f8f3400d8e0 2023-10-05T19:28:55.726 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.718+0000 7f8f4aabe700 1 -- 172.21.15.167:0/1437397465 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:47 5.18 5:18000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x564523971a30 con 0x5645238b0c20 2023-10-05T19:28:55.726 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.722+0000 7f8f4aabe700 1 -- 172.21.15.167:0/1437397465 <== osd.1 v2:172.21.15.167:6800/30035 18 ==== osd_op_reply(47 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f8f3c02f080 con 0x5645238b0c20 2023-10-05T19:28:55.726 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.722+0000 7f8f4aabe700 1 -- 172.21.15.167:0/1437397465 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:48 5.4 5:20000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x564523971a30 con 0x5645238b0c20 2023-10-05T19:28:55.727 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.722+0000 7f8f4aabe700 1 -- 172.21.15.167:0/1437397465 <== osd.1 v2:172.21.15.167:6800/30035 19 ==== osd_op_reply(48 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f8f3c02f080 con 0x5645238b0c20 2023-10-05T19:28:55.727 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.722+0000 7f8f4aabe700 1 -- 172.21.15.167:0/1437397465 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:49 5.14 5:28000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x564523971a30 con 0x7f8f3400d8e0 2023-10-05T19:28:55.727 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.722+0000 7f8f4aabe700 1 -- 172.21.15.167:0/1437397465 <== osd.0 v2:172.21.15.118:6800/27690 12 ==== osd_op_reply(49 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f8f3c02f080 con 0x7f8f3400d8e0 2023-10-05T19:28:55.727 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.722+0000 7f8f4aabe700 1 -- 172.21.15.167:0/1437397465 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:50 5.c 5:30000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x564523971a30 con 0x7f8f3400d8e0 2023-10-05T19:28:55.728 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.722+0000 7f8f4aabe700 1 -- 172.21.15.167:0/1437397465 <== osd.0 v2:172.21.15.118:6800/27690 13 ==== osd_op_reply(50 [pgnls start_epoch 32 out=112b] v23'1 uv1 ondisk = 1) v8 ==== 144+0+112 (crc 0 0 0) 0x7f8f3c02f080 con 0x7f8f3400d8e0 2023-10-05T19:28:55.728 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.722+0000 7f8f4aabe700 1 -- 172.21.15.167:0/1437397465 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:51 5.1c 5:38000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x564523971a30 con 0x7f8f4401a3c0 2023-10-05T19:28:55.729 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.722+0000 7f8f49abc700 1 -- 172.21.15.167:0/1437397465 <== osd.2 v2:172.21.15.167:6816/30037 8 ==== osd_op_reply(51 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f8f34006070 con 0x7f8f4401a3c0 2023-10-05T19:28:55.729 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.722+0000 7f8f49abc700 1 -- 172.21.15.167:0/1437397465 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:52 5.2 5:40000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x564523971a30 con 0x564523972390 2023-10-05T19:28:55.730 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.726+0000 7f8f4a2bd700 1 -- 172.21.15.167:0/1437397465 <== osd.3 v2:172.21.15.167:6808/30036 12 ==== osd_op_reply(52 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f8f4403ad70 con 0x564523972390 2023-10-05T19:28:55.730 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.726+0000 7f8f4a2bd700 1 -- 172.21.15.167:0/1437397465 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:53 5.12 5:48000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x564523971a30 con 0x5645238b0c20 2023-10-05T19:28:55.731 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.726+0000 7f8f4aabe700 1 -- 172.21.15.167:0/1437397465 <== osd.1 v2:172.21.15.167:6800/30035 20 ==== osd_op_reply(53 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f8f3c02f080 con 0x5645238b0c20 2023-10-05T19:28:55.731 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.726+0000 7f8f4aabe700 1 -- 172.21.15.167:0/1437397465 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:54 5.a 5:50000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x564523971a30 con 0x7f8f3400d8e0 2023-10-05T19:28:55.732 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.726+0000 7f8f4aabe700 1 -- 172.21.15.167:0/1437397465 <== osd.0 v2:172.21.15.118:6800/27690 14 ==== osd_op_reply(54 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f8f3c02f080 con 0x7f8f3400d8e0 2023-10-05T19:28:55.732 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.726+0000 7f8f4aabe700 1 -- 172.21.15.167:0/1437397465 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:55 5.1a 5:58000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x564523971a30 con 0x5645238b0c20 2023-10-05T19:28:55.732 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.726+0000 7f8f4aabe700 1 -- 172.21.15.167:0/1437397465 <== osd.1 v2:172.21.15.167:6800/30035 21 ==== osd_op_reply(55 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f8f3c02f080 con 0x5645238b0c20 2023-10-05T19:28:55.733 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.726+0000 7f8f4aabe700 1 -- 172.21.15.167:0/1437397465 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:56 5.6 5:60000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x564523971a30 con 0x7f8f4401a3c0 2023-10-05T19:28:55.734 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.730+0000 7f8f49abc700 1 -- 172.21.15.167:0/1437397465 <== osd.2 v2:172.21.15.167:6816/30037 9 ==== osd_op_reply(56 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f8f34006070 con 0x7f8f4401a3c0 2023-10-05T19:28:55.734 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.730+0000 7f8f49abc700 1 -- 172.21.15.167:0/1437397465 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:57 5.16 5:68000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x564523971a30 con 0x564523972390 2023-10-05T19:28:55.734 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.730+0000 7f8f4a2bd700 1 -- 172.21.15.167:0/1437397465 <== osd.3 v2:172.21.15.167:6808/30036 13 ==== osd_op_reply(57 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f8f4403ad70 con 0x564523972390 2023-10-05T19:28:55.735 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.730+0000 7f8f4a2bd700 1 -- 172.21.15.167:0/1437397465 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:58 5.e 5:70000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x564523971a30 con 0x564523972390 2023-10-05T19:28:55.735 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.730+0000 7f8f4a2bd700 1 -- 172.21.15.167:0/1437397465 <== osd.3 v2:172.21.15.167:6808/30036 14 ==== osd_op_reply(58 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f8f4403ad70 con 0x564523972390 2023-10-05T19:28:55.735 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.730+0000 7f8f4a2bd700 1 -- 172.21.15.167:0/1437397465 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:59 5.1e 5:78000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x564523971a30 con 0x7f8f3400d8e0 2023-10-05T19:28:55.736 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.730+0000 7f8f4aabe700 1 -- 172.21.15.167:0/1437397465 <== osd.0 v2:172.21.15.118:6800/27690 15 ==== osd_op_reply(59 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f8f3c02f080 con 0x7f8f3400d8e0 2023-10-05T19:28:55.736 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.730+0000 7f8f4aabe700 1 -- 172.21.15.167:0/1437397465 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:60 5.1 5:80000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x564523971a30 con 0x564523972390 2023-10-05T19:28:55.737 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.730+0000 7f8f4a2bd700 1 -- 172.21.15.167:0/1437397465 <== osd.3 v2:172.21.15.167:6808/30036 15 ==== osd_op_reply(60 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f8f4403ad70 con 0x564523972390 2023-10-05T19:28:55.737 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.730+0000 7f8f4a2bd700 1 -- 172.21.15.167:0/1437397465 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:61 5.11 5:88000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x564523971a30 con 0x564523972390 2023-10-05T19:28:55.738 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.734+0000 7f8f4a2bd700 1 -- 172.21.15.167:0/1437397465 <== osd.3 v2:172.21.15.167:6808/30036 16 ==== osd_op_reply(61 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f8f4403ad70 con 0x564523972390 2023-10-05T19:28:55.738 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.734+0000 7f8f4a2bd700 1 -- 172.21.15.167:0/1437397465 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:62 5.9 5:90000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x564523971a30 con 0x5645238b0c20 2023-10-05T19:28:55.739 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.734+0000 7f8f4aabe700 1 -- 172.21.15.167:0/1437397465 <== osd.1 v2:172.21.15.167:6800/30035 22 ==== osd_op_reply(62 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f8f3c02f080 con 0x5645238b0c20 2023-10-05T19:28:55.739 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.734+0000 7f8f4aabe700 1 -- 172.21.15.167:0/1437397465 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:63 5.19 5:98000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x564523971a30 con 0x5645238b0c20 2023-10-05T19:28:55.740 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.734+0000 7f8f4aabe700 1 -- 172.21.15.167:0/1437397465 <== osd.1 v2:172.21.15.167:6800/30035 23 ==== osd_op_reply(63 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f8f3c02f080 con 0x5645238b0c20 2023-10-05T19:28:55.740 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.734+0000 7f8f4aabe700 1 -- 172.21.15.167:0/1437397465 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:64 5.5 5:a0000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x564523971a30 con 0x7f8f3400d8e0 2023-10-05T19:28:55.741 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.734+0000 7f8f4aabe700 1 -- 172.21.15.167:0/1437397465 <== osd.0 v2:172.21.15.118:6800/27690 16 ==== osd_op_reply(64 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f8f3c02f080 con 0x7f8f3400d8e0 2023-10-05T19:28:55.741 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.734+0000 7f8f4aabe700 1 -- 172.21.15.167:0/1437397465 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:65 5.15 5:a8000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x564523971a30 con 0x564523972390 2023-10-05T19:28:55.742 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.738+0000 7f8f4a2bd700 1 -- 172.21.15.167:0/1437397465 <== osd.3 v2:172.21.15.167:6808/30036 17 ==== osd_op_reply(65 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f8f4403ad70 con 0x564523972390 2023-10-05T19:28:55.742 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.738+0000 7f8f4a2bd700 1 -- 172.21.15.167:0/1437397465 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:66 5.d 5:b0000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x564523971a30 con 0x7f8f4401a3c0 2023-10-05T19:28:55.743 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.738+0000 7f8f49abc700 1 -- 172.21.15.167:0/1437397465 <== osd.2 v2:172.21.15.167:6816/30037 10 ==== osd_op_reply(66 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f8f34006070 con 0x7f8f4401a3c0 2023-10-05T19:28:55.743 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.738+0000 7f8f49abc700 1 -- 172.21.15.167:0/1437397465 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:67 5.1d 5:b8000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x564523971a30 con 0x5645238b0c20 2023-10-05T19:28:55.743 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.738+0000 7f8f4aabe700 1 -- 172.21.15.167:0/1437397465 <== osd.1 v2:172.21.15.167:6800/30035 24 ==== osd_op_reply(67 [pgnls start_epoch 32 out=107b] v30'2 uv2 ondisk = 1) v8 ==== 144+0+107 (crc 0 0 0) 0x7f8f3c02f080 con 0x5645238b0c20 2023-10-05T19:28:55.744 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.738+0000 7f8f4aabe700 1 -- 172.21.15.167:0/1437397465 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:68 5.3 5:c0000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x564523971a30 con 0x7f8f3400d8e0 2023-10-05T19:28:55.744 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.738+0000 7f8f4aabe700 1 -- 172.21.15.167:0/1437397465 <== osd.0 v2:172.21.15.118:6800/27690 17 ==== osd_op_reply(68 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f8f3c02f080 con 0x7f8f3400d8e0 2023-10-05T19:28:55.744 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.738+0000 7f8f4aabe700 1 -- 172.21.15.167:0/1437397465 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:69 5.13 5:c8000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x564523971a30 con 0x564523972390 2023-10-05T19:28:55.745 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.742+0000 7f8f4a2bd700 1 -- 172.21.15.167:0/1437397465 <== osd.3 v2:172.21.15.167:6808/30036 18 ==== osd_op_reply(69 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f8f4403ad70 con 0x564523972390 2023-10-05T19:28:55.746 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.742+0000 7f8f4a2bd700 1 -- 172.21.15.167:0/1437397465 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:70 5.b 5:d0000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x564523971a30 con 0x7f8f3400d8e0 2023-10-05T19:28:55.746 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.742+0000 7f8f4aabe700 1 -- 172.21.15.167:0/1437397465 <== osd.0 v2:172.21.15.118:6800/27690 18 ==== osd_op_reply(70 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f8f3c02f080 con 0x7f8f3400d8e0 2023-10-05T19:28:55.746 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.742+0000 7f8f4aabe700 1 -- 172.21.15.167:0/1437397465 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:71 5.1b 5:d8000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x564523971a30 con 0x7f8f4401a3c0 2023-10-05T19:28:55.747 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.742+0000 7f8f49abc700 1 -- 172.21.15.167:0/1437397465 <== osd.2 v2:172.21.15.167:6816/30037 11 ==== osd_op_reply(71 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f8f34006070 con 0x7f8f4401a3c0 2023-10-05T19:28:55.747 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.742+0000 7f8f49abc700 1 -- 172.21.15.167:0/1437397465 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:72 5.7 5:e0000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x564523971a30 con 0x564523972390 2023-10-05T19:28:55.748 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.742+0000 7f8f4a2bd700 1 -- 172.21.15.167:0/1437397465 <== osd.3 v2:172.21.15.167:6808/30036 19 ==== osd_op_reply(72 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f8f4403ad70 con 0x564523972390 2023-10-05T19:28:55.748 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.742+0000 7f8f4a2bd700 1 -- 172.21.15.167:0/1437397465 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:73 5.17 5:e8000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x564523971a30 con 0x7f8f3400d8e0 2023-10-05T19:28:55.749 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.742+0000 7f8f4aabe700 1 -- 172.21.15.167:0/1437397465 <== osd.0 v2:172.21.15.118:6800/27690 19 ==== osd_op_reply(73 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f8f3c02f080 con 0x7f8f3400d8e0 2023-10-05T19:28:55.749 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.742+0000 7f8f4aabe700 1 -- 172.21.15.167:0/1437397465 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:74 5.f 5:f0000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x564523971a30 con 0x5645238b0c20 2023-10-05T19:28:55.750 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.746+0000 7f8f4aabe700 1 -- 172.21.15.167:0/1437397465 <== osd.1 v2:172.21.15.167:6800/30035 25 ==== osd_op_reply(74 [pgnls start_epoch 32 out=115b] v23'2 uv2 ondisk = 1) v8 ==== 144+0+115 (crc 0 0 0) 0x7f8f3c02f080 con 0x5645238b0c20 2023-10-05T19:28:55.750 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.746+0000 7f8f4aabe700 1 -- 172.21.15.167:0/1437397465 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:75 5.1f 5:f8000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x564523971a30 con 0x7f8f4401a3c0 2023-10-05T19:28:55.751 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.746+0000 7f8f49abc700 1 -- 172.21.15.167:0/1437397465 <== osd.2 v2:172.21.15.167:6816/30037 12 ==== osd_op_reply(75 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f8f34006070 con 0x7f8f4401a3c0 2023-10-05T19:28:55.751 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.746+0000 7f8f4c788e40 20 RGWRados::pool_iterate: got zonegroup_info.4f1f719d-9648-417b-b10c-e5d939825419 2023-10-05T19:28:55.751 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.746+0000 7f8f4c788e40 20 RGWRados::pool_iterate: got zone_info.9a9df965-6612-45d4-a45f-6e3c889bacc2 2023-10-05T19:28:55.751 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.746+0000 7f8f4c788e40 20 RGWRados::pool_iterate: got zone_names.default 2023-10-05T19:28:55.751 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.746+0000 7f8f4c788e40 20 RGWRados::pool_iterate: got zonegroups_names.default 2023-10-05T19:28:55.752 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.746+0000 7f8f4c788e40 20 rados->read ofs=0 len=0 2023-10-05T19:28:55.752 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.746+0000 7f8f4c788e40 1 -- 172.21.15.167:0/1437397465 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:76 5.f 5:f43b6ece:::zone_names.default:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32) v8 -- 0x5645239758d0 con 0x5645238b0c20 2023-10-05T19:28:55.753 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.746+0000 7f8f4aabe700 1 -- 172.21.15.167:0/1437397465 <== osd.1 v2:172.21.15.167:6800/30035 26 ==== osd_op_reply(76 zone_names.default [read 0~46 out=46b] v0'0 uv1 ondisk = 0) v8 ==== 162+0+46 (crc 0 0 0) 0x7f8f3c02f080 con 0x5645238b0c20 2023-10-05T19:28:55.753 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.746+0000 7f8f4c788e40 20 rados_obj.operate() r=0 bl.length=46 2023-10-05T19:28:55.753 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.746+0000 7f8f4c788e40 20 rados->read ofs=0 len=0 2023-10-05T19:28:55.753 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.746+0000 7f8f4c788e40 1 -- 172.21.15.167:0/1437397465 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:77 5.1d 5:b8374eb4:::zone_info.9a9df965-6612-45d4-a45f-6e3c889bacc2:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32) v8 -- 0x564523975c80 con 0x5645238b0c20 2023-10-05T19:28:55.753 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.746+0000 7f8f4aabe700 1 -- 172.21.15.167:0/1437397465 <== osd.1 v2:172.21.15.167:6800/30035 27 ==== osd_op_reply(77 zone_info.9a9df965-6612-45d4-a45f-6e3c889bacc2 [read 0~874 out=874b] v0'0 uv2 ondisk = 0) v8 ==== 190+0+874 (crc 0 0 0) 0x7f8f3c02f080 con 0x5645238b0c20 2023-10-05T19:28:55.753 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.746+0000 7f8f4c788e40 20 rados_obj.operate() r=0 bl.length=874 2023-10-05T19:28:55.753 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.746+0000 7f8f4c788e40 20 rados->read ofs=0 len=0 2023-10-05T19:28:55.753 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.746+0000 7f8f4c788e40 1 -- 172.21.15.167:0/1437397465 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:78 5.f 5:f4c53578:::zonegroups_names.default:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32) v8 -- 0x564523975cb0 con 0x5645238b0c20 2023-10-05T19:28:55.754 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.746+0000 7f8f4aabe700 1 -- 172.21.15.167:0/1437397465 <== osd.1 v2:172.21.15.167:6800/30035 28 ==== osd_op_reply(78 zonegroups_names.default [read 0~46 out=46b] v0'0 uv2 ondisk = 0) v8 ==== 168+0+46 (crc 0 0 0) 0x7f8f3c02f080 con 0x5645238b0c20 2023-10-05T19:28:55.754 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.746+0000 7f8f4c788e40 20 rados_obj.operate() r=0 bl.length=46 2023-10-05T19:28:55.754 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.746+0000 7f8f4c788e40 20 rados->read ofs=0 len=0 2023-10-05T19:28:55.755 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.750+0000 7f8f4c788e40 1 -- 172.21.15.167:0/1437397465 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:79 5.c 5:30d299a5:::zonegroup_info.4f1f719d-9648-417b-b10c-e5d939825419:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32) v8 -- 0x564523975cc0 con 0x7f8f3400d8e0 2023-10-05T19:28:55.755 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.750+0000 7f8f4aabe700 1 -- 172.21.15.167:0/1437397465 <== osd.0 v2:172.21.15.118:6800/27690 20 ==== osd_op_reply(79 zonegroup_info.4f1f719d-9648-417b-b10c-e5d939825419 [read 0~362 out=362b] v0'0 uv1 ondisk = 0) v8 ==== 195+0+362 (crc 0 0 0) 0x7f8f3c02f080 con 0x7f8f3400d8e0 2023-10-05T19:28:55.755 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.750+0000 7f8f4c788e40 20 rados_obj.operate() r=0 bl.length=362 2023-10-05T19:28:55.755 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.750+0000 7f8f4c788e40 20 rados->read ofs=0 len=0 2023-10-05T19:28:55.755 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.750+0000 7f8f4c788e40 1 -- 172.21.15.167:0/1437397465 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:80 5.3 5:c0ad8cbd:::region_map:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32) v8 -- 0x564523976170 con 0x7f8f3400d8e0 2023-10-05T19:28:55.755 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.750+0000 7f8f4aabe700 1 -- 172.21.15.167:0/1437397465 <== osd.0 v2:172.21.15.118:6800/27690 21 ==== osd_op_reply(80 region_map [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 ==== 154+0+0 (crc 0 0 0) 0x7f8f3c050040 con 0x7f8f3400d8e0 2023-10-05T19:28:55.755 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.750+0000 7f8f4c788e40 20 rados_obj.operate() r=-2 bl.length=0 2023-10-05T19:28:55.756 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.750+0000 7f8f4c788e40 20 zone default found 2023-10-05T19:28:55.756 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.750+0000 7f8f4c788e40 4 Realm: () 2023-10-05T19:28:55.756 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.750+0000 7f8f4c788e40 4 ZoneGroup: default (4f1f719d-9648-417b-b10c-e5d939825419) 2023-10-05T19:28:55.756 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.750+0000 7f8f4c788e40 4 Zone: default (9a9df965-6612-45d4-a45f-6e3c889bacc2) 2023-10-05T19:28:55.756 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.750+0000 7f8f4c788e40 10 cannot find current period zonegroup using local zonegroup configuration 2023-10-05T19:28:55.756 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.750+0000 7f8f4c788e40 20 zonegroup default 2023-10-05T19:28:55.756 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.750+0000 7f8f4c788e40 20 rados->read ofs=0 len=0 2023-10-05T19:28:55.756 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.750+0000 7f8f4c788e40 1 -- 172.21.15.167:0/1437397465 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:81 5.3 5:c52100b6:::period_config.default:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32) v8 -- 0x564523976170 con 0x7f8f3400d8e0 2023-10-05T19:28:55.757 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.750+0000 7f8f4aabe700 1 -- 172.21.15.167:0/1437397465 <== osd.0 v2:172.21.15.118:6800/27690 22 ==== osd_op_reply(81 period_config.default [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 ==== 165+0+0 (crc 0 0 0) 0x7f8f3c050040 con 0x7f8f3400d8e0 2023-10-05T19:28:55.757 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.750+0000 7f8f4c788e40 20 rados_obj.operate() r=-2 bl.length=0 2023-10-05T19:28:55.757 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.750+0000 7f8f4c788e40 20 rados->read ofs=0 len=0 2023-10-05T19:28:55.758 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.750+0000 7f8f4c788e40 1 -- 172.21.15.167:0/1437397465 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:82 6.2 6:4347d321:::bucket.sync-source-hints.:head [call version.read in=11b,read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32) v8 -- 0x564523977860 con 0x564523972390 2023-10-05T19:28:55.758 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.750+0000 7f8f4a2bd700 1 -- 172.21.15.167:0/1437397465 <== osd.3 v2:172.21.15.167:6808/30036 20 ==== osd_op_reply(82 bucket.sync-source-hints. [call,read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 ==== 211+0+0 (crc 0 0 0) 0x7f8f4403ad70 con 0x564523972390 2023-10-05T19:28:55.758 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.750+0000 7f8f4c788e40 20 rados_obj.operate() r=-2 bl.length=0 2023-10-05T19:28:55.758 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.750+0000 7f8f4c788e40 20 rados->read ofs=0 len=0 2023-10-05T19:28:55.758 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.750+0000 7f8f4c788e40 1 -- 172.21.15.167:0/1437397465 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:83 6.b 6:d467b91b:::bucket.sync-target-hints.:head [call version.read in=11b,read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32) v8 -- 0x564523977c10 con 0x564523972390 2023-10-05T19:28:55.758 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.750+0000 7f8f4a2bd700 1 -- 172.21.15.167:0/1437397465 <== osd.3 v2:172.21.15.167:6808/30036 21 ==== osd_op_reply(83 bucket.sync-target-hints. [call,read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 ==== 211+0+0 (crc 0 0 0) 0x7f8f4405f040 con 0x564523972390 2023-10-05T19:28:55.758 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.750+0000 7f8f4c788e40 20 rados_obj.operate() r=-2 bl.length=0 2023-10-05T19:28:55.758 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.750+0000 7f8f4c788e40 20 started sync module instance, tier type = 2023-10-05T19:28:55.758 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.750+0000 7f8f4c788e40 20 started zone id=9a9df965-6612-45d4-a45f-6e3c889bacc2 (name=default) with tier type = 2023-10-05T19:28:55.758 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.750+0000 7f8f4c788e40 1 -- 172.21.15.167:0/1437397465 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:84 7.1f 7:f95f44c2:::notify.0:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e32) v8 -- 0x564523976f20 con 0x7f8f4401a3c0 2023-10-05T19:28:55.760 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.754+0000 7f8f49abc700 1 -- 172.21.15.167:0/1437397465 <== osd.2 v2:172.21.15.167:6816/30037 13 ==== osd_op_reply(84 notify.0 [create] v32'198 uv198 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f8f34006070 con 0x7f8f4401a3c0 2023-10-05T19:28:55.760 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.754+0000 7f8f4c788e40 1 -- 172.21.15.167:0/1437397465 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:85 7.1f 7:f95f44c2:::notify.0:head [watch watch cookie 94854949860128] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e32) v8 -- 0x564523971660 con 0x7f8f4401a3c0 2023-10-05T19:28:55.761 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.754+0000 7f8f4c788e40 1 -- 172.21.15.167:0/1437397465 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:86 7.0 7:05bf5b68:::notify.1:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e32) v8 -- 0x564523978a30 con 0x5645238b0c20 2023-10-05T19:28:55.763 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.758+0000 7f8f4aabe700 1 -- 172.21.15.167:0/1437397465 <== osd.1 v2:172.21.15.167:6800/30035 29 ==== osd_op_reply(86 notify.1 [create] v32'198 uv198 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f8f3c050040 con 0x5645238b0c20 2023-10-05T19:28:55.763 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.758+0000 7f8f4c788e40 1 -- 172.21.15.167:0/1437397465 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:87 7.0 7:05bf5b68:::notify.1:head [watch watch cookie 94854949865056] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e32) v8 -- 0x564523979320 con 0x5645238b0c20 2023-10-05T19:28:55.764 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.758+0000 7f8f4c788e40 1 -- 172.21.15.167:0/1437397465 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:88 7.15 7:a93a5511:::notify.2:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e32) v8 -- 0x564523979bc0 con 0x564523972390 2023-10-05T19:28:55.764 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.758+0000 7f8f49abc700 1 -- 172.21.15.167:0/1437397465 <== osd.2 v2:172.21.15.167:6816/30037 14 ==== osd_op_reply(85 notify.0 [watch watch cookie 94854949860128] v32'199 uv198 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f8f34006070 con 0x7f8f4401a3c0 2023-10-05T19:28:55.766 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.762+0000 7f8f4a2bd700 1 -- 172.21.15.167:0/1437397465 <== osd.3 v2:172.21.15.167:6808/30036 22 ==== osd_op_reply(88 notify.2 [create] v32'198 uv198 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f8f4405f040 con 0x564523972390 2023-10-05T19:28:55.766 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.762+0000 7f8f4c788e40 1 -- 172.21.15.167:0/1437397465 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:89 7.15 7:a93a5511:::notify.2:head [watch watch cookie 94854949868320] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e32) v8 -- 0x564523979770 con 0x564523972390 2023-10-05T19:28:55.766 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.762+0000 7f8f4c788e40 1 -- 172.21.15.167:0/1437397465 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:90 7.e 7:7759931f:::notify.3:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e32) v8 -- 0x56452397a950 con 0x7f8f4401a3c0 2023-10-05T19:28:55.767 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.762+0000 7f8f4aabe700 1 -- 172.21.15.167:0/1437397465 <== osd.1 v2:172.21.15.167:6800/30035 30 ==== osd_op_reply(87 notify.1 [watch watch cookie 94854949865056] v32'199 uv198 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f8f3c050040 con 0x5645238b0c20 2023-10-05T19:28:55.769 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.762+0000 7f8f4a2bd700 1 -- 172.21.15.167:0/1437397465 <== osd.3 v2:172.21.15.167:6808/30036 23 ==== osd_op_reply(89 notify.2 [watch watch cookie 94854949868320] v32'199 uv198 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f8f4405f040 con 0x564523972390 2023-10-05T19:28:55.769 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.762+0000 7f8f49abc700 1 -- 172.21.15.167:0/1437397465 <== osd.2 v2:172.21.15.167:6816/30037 15 ==== osd_op_reply(90 notify.3 [create] v32'198 uv198 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f8f34006070 con 0x7f8f4401a3c0 2023-10-05T19:28:55.769 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.762+0000 7f8f4c788e40 1 -- 172.21.15.167:0/1437397465 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:91 7.e 7:7759931f:::notify.3:head [watch watch cookie 94854949867056] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e32) v8 -- 0x56452397a460 con 0x7f8f4401a3c0 2023-10-05T19:28:55.769 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.762+0000 7f8f4c788e40 1 -- 172.21.15.167:0/1437397465 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:92 7.d 7:b4812045:::notify.4:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e32) v8 -- 0x56452397b360 con 0x5645238b0c20 2023-10-05T19:28:55.771 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.766+0000 7f8f49abc700 1 -- 172.21.15.167:0/1437397465 <== osd.2 v2:172.21.15.167:6816/30037 16 ==== osd_op_reply(91 notify.3 [watch watch cookie 94854949867056] v32'199 uv198 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f8f34006070 con 0x7f8f4401a3c0 2023-10-05T19:28:55.772 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.766+0000 7f8f4aabe700 1 -- 172.21.15.167:0/1437397465 <== osd.1 v2:172.21.15.167:6800/30035 31 ==== osd_op_reply(92 notify.4 [create] v32'198 uv198 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f8f3c050040 con 0x5645238b0c20 2023-10-05T19:28:55.772 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.766+0000 7f8f4c788e40 1 -- 172.21.15.167:0/1437397465 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:93 7.d 7:b4812045:::notify.4:head [watch watch cookie 94854949875312] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e32) v8 -- 0x56452397b2c0 con 0x5645238b0c20 2023-10-05T19:28:55.772 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.766+0000 7f8f4c788e40 1 -- 172.21.15.167:0/1437397465 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:94 7.3 7:c609908c:::notify.5:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e32) v8 -- 0x56452397c140 con 0x7f8f3400d8e0 2023-10-05T19:28:55.774 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.770+0000 7f8f4aabe700 1 -- 172.21.15.167:0/1437397465 <== osd.0 v2:172.21.15.118:6800/27690 23 ==== osd_op_reply(94 notify.5 [create] v32'198 uv198 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f8f3c050040 con 0x7f8f3400d8e0 2023-10-05T19:28:55.774 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.770+0000 7f8f4c788e40 1 -- 172.21.15.167:0/1437397465 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:95 7.3 7:c609908c:::notify.5:head [watch watch cookie 94854949878864] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e32) v8 -- 0x56452397c590 con 0x7f8f3400d8e0 2023-10-05T19:28:55.775 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.770+0000 7f8f4c788e40 1 -- 172.21.15.167:0/1437397465 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:96 7.14 7:2b04a3e9:::notify.6:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e32) v8 -- 0x56452397cfe0 con 0x7f8f3400d8e0 2023-10-05T19:28:55.775 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.770+0000 7f8f4aabe700 1 -- 172.21.15.167:0/1437397465 <== osd.1 v2:172.21.15.167:6800/30035 32 ==== osd_op_reply(93 notify.4 [watch watch cookie 94854949875312] v32'199 uv198 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f8f3c050040 con 0x5645238b0c20 2023-10-05T19:28:55.776 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.770+0000 7f8f4aabe700 1 -- 172.21.15.167:0/1437397465 <== osd.0 v2:172.21.15.118:6800/27690 24 ==== osd_op_reply(95 notify.5 [watch watch cookie 94854949878864] v32'199 uv198 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f8f3c050040 con 0x7f8f3400d8e0 2023-10-05T19:28:55.777 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.770+0000 7f8f4aabe700 1 -- 172.21.15.167:0/1437397465 <== osd.0 v2:172.21.15.118:6800/27690 25 ==== osd_op_reply(96 notify.6 [create] v32'198 uv198 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f8f3c050040 con 0x7f8f3400d8e0 2023-10-05T19:28:55.777 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.770+0000 7f8f4c788e40 1 -- 172.21.15.167:0/1437397465 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:97 7.14 7:2b04a3e9:::notify.6:head [watch watch cookie 94854949877504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e32) v8 -- 0x56452397c590 con 0x7f8f3400d8e0 2023-10-05T19:28:55.777 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.770+0000 7f8f4c788e40 1 -- 172.21.15.167:0/1437397465 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:98 7.9 7:93e5b521:::notify.7:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e32) v8 -- 0x56452397d410 con 0x7f8f3400d8e0 2023-10-05T19:28:55.780 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.774+0000 7f8f4aabe700 1 -- 172.21.15.167:0/1437397465 <== osd.0 v2:172.21.15.118:6800/27690 26 ==== osd_op_reply(98 notify.7 [create] v32'198 uv198 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f8f3c050040 con 0x7f8f3400d8e0 2023-10-05T19:28:55.780 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.774+0000 7f8f4c788e40 1 -- 172.21.15.167:0/1437397465 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:99 7.9 7:93e5b521:::notify.7:head [watch watch cookie 94854949883616] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e32) v8 -- 0x56452397d820 con 0x7f8f3400d8e0 2023-10-05T19:28:55.780 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.774+0000 7f8f4c788e40 20 add_watcher() i=0 2023-10-05T19:28:55.780 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.774+0000 7f8f4c788e40 20 add_watcher() i=1 2023-10-05T19:28:55.781 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.774+0000 7f8f4c788e40 20 add_watcher() i=2 2023-10-05T19:28:55.781 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.774+0000 7f8f4c788e40 20 add_watcher() i=3 2023-10-05T19:28:55.781 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.774+0000 7f8f4c788e40 20 add_watcher() i=4 2023-10-05T19:28:55.781 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.774+0000 7f8f4c788e40 20 add_watcher() i=5 2023-10-05T19:28:55.781 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.774+0000 7f8f4aabe700 1 -- 172.21.15.167:0/1437397465 <== osd.0 v2:172.21.15.118:6800/27690 27 ==== osd_op_reply(97 notify.6 [watch watch cookie 94854949877504] v32'199 uv198 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f8f3c050040 con 0x7f8f3400d8e0 2023-10-05T19:28:55.781 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.774+0000 7f8f4c788e40 20 add_watcher() i=6 2023-10-05T19:28:55.782 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.778+0000 7f8f4aabe700 1 -- 172.21.15.167:0/1437397465 <== osd.0 v2:172.21.15.118:6800/27690 28 ==== osd_op_reply(99 notify.7 [watch watch cookie 94854949883616] v32'199 uv198 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f8f3c050040 con 0x7f8f3400d8e0 2023-10-05T19:28:55.782 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.778+0000 7f8f4c788e40 20 add_watcher() i=7 2023-10-05T19:28:55.782 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.778+0000 7f8f4c788e40 2 all 8 watchers are set, enabling cache 2023-10-05T19:28:55.783 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.778+0000 7f8f4c788e40 1 -- 172.21.15.167:0/1437397465 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:100 6.4 6:22d26bf9:::data_loggenerations_metadata:head [call version.check_conds in=74b,call version.read in=11b,read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32) v8 -- 0x56452397c0a0 con 0x5645238b0c20 2023-10-05T19:28:55.783 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.778+0000 7f8f4aabe700 1 -- 172.21.15.167:0/1437397465 <== osd.1 v2:172.21.15.167:6800/30035 33 ==== osd_op_reply(100 data_loggenerations_metadata [call,call out=48b,read 0~28 out=28b] v0'0 uv1 ondisk = 0) v8 ==== 256+0+76 (crc 0 0 0) 0x7f8f3c050040 con 0x5645238b0c20 2023-10-05T19:28:55.783 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.778+0000 7f8f4c788e40 1 -- 172.21.15.167:0/1437397465 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:101 6.4 6:22d26bf9:::data_loggenerations_metadata:head [watch watch cookie 94854949884720] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e32) v8 -- 0x56452397dcc0 con 0x5645238b0c20 2023-10-05T19:28:55.785 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.778+0000 7f8f4aabe700 1 -- 172.21.15.167:0/1437397465 <== osd.1 v2:172.21.15.167:6800/30035 34 ==== osd_op_reply(101 data_loggenerations_metadata [watch watch cookie 94854949884720] v32'196 uv1 ondisk = 0) v8 ==== 172+0+0 (crc 0 0 0) 0x7f8f3c050040 con 0x5645238b0c20 2023-10-05T19:28:55.785 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.778+0000 7f8f1bfdf700 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2023-10-05T19:28:55.785 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.778+0000 7f8f4c788e40 20 check_secure_mon_conn(): auth registy supported: methods=[2] modes=[2,1] 2023-10-05T19:28:55.786 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.778+0000 7f8f4c788e40 20 check_secure_mon_conn(): mode 1 is insecure 2023-10-05T19:28:55.786 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.782+0000 7f8f4c788e40 1 Processor -- start 2023-10-05T19:28:55.789 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.782+0000 7f8f4c788e40 1 -- start start 2023-10-05T19:28:55.789 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.782+0000 7f8f4c788e40 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x56452398ae30 0x56452398b200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-05T19:28:55.789 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.782+0000 7f8f4c788e40 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x56452398b740 con 0x56452398ae30 2023-10-05T19:28:55.789 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.782+0000 7f8f49abc700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x56452398ae30 0x56452398b200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-05T19:28:55.789 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.782+0000 7f8f49abc700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x56452398ae30 0x56452398b200 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.118:3300/0 says I am v2:172.21.15.167:53282/0 (socket says 172.21.15.167:53282) 2023-10-05T19:28:55.789 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.782+0000 7f8f49abc700 1 -- 172.21.15.167:0/4049343241 learned_addr learned my addr 172.21.15.167:0/4049343241 (peer_addr_for_me v2:172.21.15.167:0/0) 2023-10-05T19:28:55.792 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.782+0000 7f8f49abc700 1 -- 172.21.15.167:0/4049343241 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x56452398b880 con 0x56452398ae30 2023-10-05T19:28:55.792 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.782+0000 7f8f49abc700 1 --2- 172.21.15.167:0/4049343241 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x56452398ae30 0x56452398b200 secure :-1 s=READY pgs=534 cs=0 l=1 rev1=1 crypto rx=0x56452396fe30 tx=0x7f8f34006000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=396025b9c919d3eb server_cookie=0 in_seq=0 out_seq=0 2023-10-05T19:28:55.792 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.782+0000 7f8f1b7de700 1 -- 172.21.15.167:0/4049343241 <== mon.0 v2:172.21.15.118:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f8f34023040 con 0x56452398ae30 2023-10-05T19:28:55.792 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.782+0000 7f8f1b7de700 1 -- 172.21.15.167:0/4049343241 <== mon.0 v2:172.21.15.118:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8f34024040 con 0x56452398ae30 2023-10-05T19:28:55.792 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.782+0000 7f8f1b7de700 1 -- 172.21.15.167:0/4049343241 <== mon.0 v2:172.21.15.118:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f8f34013040 con 0x56452398ae30 2023-10-05T19:28:55.792 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.782+0000 7f8f4c788e40 1 -- 172.21.15.167:0/4049343241 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x56452398ae30 msgr2=0x56452398b200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-05T19:28:55.792 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.782+0000 7f8f4c788e40 1 --2- 172.21.15.167:0/4049343241 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x56452398ae30 0x56452398b200 secure :-1 s=READY pgs=534 cs=0 l=1 rev1=1 crypto rx=0x56452396fe30 tx=0x7f8f34006000 comp rx=0 tx=0).stop 2023-10-05T19:28:55.792 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.782+0000 7f8f4c788e40 1 -- 172.21.15.167:0/4049343241 shutdown_connections 2023-10-05T19:28:55.793 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.782+0000 7f8f4c788e40 1 --2- 172.21.15.167:0/4049343241 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x56452398ae30 0x56452398b200 unknown :-1 s=CLOSED pgs=534 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-05T19:28:55.793 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.782+0000 7f8f4c788e40 1 -- 172.21.15.167:0/4049343241 >> 172.21.15.167:0/4049343241 conn(0x5645239861f0 msgr2=0x564523984340 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-05T19:28:55.793 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.782+0000 7f8f4c788e40 1 -- 172.21.15.167:0/4049343241 shutdown_connections 2023-10-05T19:28:55.793 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.782+0000 7f8f4c788e40 1 -- 172.21.15.167:0/4049343241 wait complete. 2023-10-05T19:28:55.793 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.786+0000 7f8f4c788e40 1 Processor -- start 2023-10-05T19:28:55.793 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.786+0000 7f8f4c788e40 1 -- start start 2023-10-05T19:28:55.793 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.786+0000 7f8f4c788e40 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x56452398ae30 0x564523990a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-05T19:28:55.793 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.786+0000 7f8f4c788e40 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x564523991a10 con 0x56452398ae30 2023-10-05T19:28:55.793 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.786+0000 7f8f49abc700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x56452398ae30 0x564523990a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-05T19:28:55.793 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.786+0000 7f8f49abc700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x56452398ae30 0x564523990a10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.118:3300/0 says I am v2:172.21.15.167:53292/0 (socket says 172.21.15.167:53292) 2023-10-05T19:28:55.793 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.786+0000 7f8f49abc700 1 -- 172.21.15.167:0/2730877213 learned_addr learned my addr 172.21.15.167:0/2730877213 (peer_addr_for_me v2:172.21.15.167:0/0) 2023-10-05T19:28:55.793 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.786+0000 7f8f49abc700 1 -- 172.21.15.167:0/2730877213 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8f3400ed60 con 0x56452398ae30 2023-10-05T19:28:55.794 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.786+0000 7f8f49abc700 1 --2- 172.21.15.167:0/2730877213 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x56452398ae30 0x564523990a10 secure :-1 s=READY pgs=535 cs=0 l=1 rev1=1 crypto rx=0x564523984eb0 tx=0x7f8f34030000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-05T19:28:55.794 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.786+0000 7f8f19fdb700 1 -- 172.21.15.167:0/2730877213 <== mon.0 v2:172.21.15.118:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f8f34023040 con 0x56452398ae30 2023-10-05T19:28:55.794 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.786+0000 7f8f19fdb700 1 -- 172.21.15.167:0/2730877213 <== mon.0 v2:172.21.15.118:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8f3403b580 con 0x56452398ae30 2023-10-05T19:28:55.794 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.786+0000 7f8f4c788e40 1 -- 172.21.15.167:0/2730877213 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x564523991010 con 0x56452398ae30 2023-10-05T19:28:55.794 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.786+0000 7f8f19fdb700 1 -- 172.21.15.167:0/2730877213 <== mon.0 v2:172.21.15.118:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f8f3403b6f0 con 0x56452398ae30 2023-10-05T19:28:55.794 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.786+0000 7f8f4c788e40 1 -- 172.21.15.167:0/2730877213 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x564523991450 con 0x56452398ae30 2023-10-05T19:28:55.795 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.786+0000 7f8f19fdb700 1 -- 172.21.15.167:0/2730877213 <== mon.0 v2:172.21.15.118:3300/0 4 ==== mgrmap(e 5) v1 ==== 40329+0+0 (secure 0 0 0) 0x7f8f3403b850 con 0x56452398ae30 2023-10-05T19:28:55.795 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.790+0000 7f8f19fdb700 1 --2- 172.21.15.167:0/2730877213 >> [v2:172.21.15.167:6824/29889,v1:172.21.15.167:6825/29889] conn(0x7f8e90030c50 0x7f8e90033100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-05T19:28:55.795 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.790+0000 7f8f19fdb700 1 -- 172.21.15.167:0/2730877213 <== mon.0 v2:172.21.15.118:3300/0 5 ==== osd_map(32..32 src has 1..32) v4 ==== 6260+0+0 (secure 0 0 0) 0x7f8f3404ed30 con 0x56452398ae30 2023-10-05T19:28:55.795 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.790+0000 7f8f4aabe700 1 --2- 172.21.15.167:0/2730877213 >> [v2:172.21.15.167:6824/29889,v1:172.21.15.167:6825/29889] conn(0x7f8e90030c50 0x7f8e90033100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-05T19:28:55.795 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.790+0000 7f8f4c788e40 1 --2- 172.21.15.167:0/2730877213 >> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] conn(0x56452399b600 0x56452399da10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-05T19:28:55.796 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.790+0000 7f8f4c788e40 1 -- 172.21.15.167:0/2730877213 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:1 6.19 6:9a6dd8c0:gc::gc.0:head [create,call version.check_conds in=74b,call rgw_gc.rgw_gc_queue_init in=45b,call version.set in=34b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e32) v8 -- 0x56452399e060 con 0x56452399b600 2023-10-05T19:28:55.796 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.790+0000 7f8f4a2bd700 1 --2- 172.21.15.167:0/2730877213 >> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] conn(0x56452399b600 0x56452399da10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-05T19:28:55.796 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.790+0000 7f8f4aabe700 1 --2- 172.21.15.167:0/2730877213 >> [v2:172.21.15.167:6824/29889,v1:172.21.15.167:6825/29889] conn(0x7f8e90030c50 0x7f8e90033100 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x56452397bbc0 tx=0x7f8f3c000f90 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-05T19:28:55.796 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.790+0000 7f8f4a2bd700 1 --2- 172.21.15.167:0/2730877213 >> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] conn(0x56452399b600 0x56452399da10 crc :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-05T19:28:55.797 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.790+0000 7f8f4a2bd700 1 -- 172.21.15.167:0/2730877213 <== osd.1 v2:172.21.15.167:6800/30035 1 ==== osd_op_reply(1 gc.0 [create,call,call,call] v32'35 uv33 ondisk = -125 ((125) Operation canceled)) v8 ==== 274+0+0 (crc 0 0 0) 0x7f8f4405f040 con 0x56452399b600 2023-10-05T19:28:55.797 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.794+0000 7f8f4c788e40 1 --2- 172.21.15.167:0/2730877213 >> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] conn(0x56452399ed30 0x5645239a1110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-05T19:28:55.798 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.794+0000 7f8f4c788e40 1 -- 172.21.15.167:0/2730877213 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:2 6.18 6:1e85421f:gc::gc.1:head [create,call version.check_conds in=74b,call rgw_gc.rgw_gc_queue_init in=45b,call version.set in=34b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e32) v8 -- 0x5645239a1840 con 0x56452399ed30 2023-10-05T19:28:55.798 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.794+0000 7f8f49abc700 1 --2- 172.21.15.167:0/2730877213 >> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] conn(0x56452399ed30 0x5645239a1110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-05T19:28:55.798 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.794+0000 7f8f49abc700 1 --2- 172.21.15.167:0/2730877213 >> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] conn(0x56452399ed30 0x5645239a1110 crc :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-05T19:28:55.801 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.794+0000 7f8f49abc700 1 -- 172.21.15.167:0/2730877213 <== osd.0 v2:172.21.15.118:6800/27690 1 ==== osd_op_reply(2 gc.1 [create,call,call,call] v32'27 uv25 ondisk = -125 ((125) Operation canceled)) v8 ==== 274+0+0 (crc 0 0 0) 0x7f8f34006070 con 0x56452399ed30 2023-10-05T19:28:55.801 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.794+0000 7f8f4c788e40 1 --2- 172.21.15.167:0/2730877213 >> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] conn(0x564523997240 0x5645239a4eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-05T19:28:55.802 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.794+0000 7f8f4c788e40 1 -- 172.21.15.167:0/2730877213 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:3 6.b 6:d242335b:gc::gc.2:head [create,call version.check_conds in=74b,call rgw_gc.rgw_gc_queue_init in=45b,call version.set in=34b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e32) v8 -- 0x5645239a5430 con 0x564523997240 2023-10-05T19:28:55.802 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.798+0000 7f8f4aabe700 1 --2- 172.21.15.167:0/2730877213 >> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] conn(0x564523997240 0x5645239a4eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-05T19:28:55.802 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.798+0000 7f8f4aabe700 1 --2- 172.21.15.167:0/2730877213 >> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] conn(0x564523997240 0x5645239a4eb0 crc :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-05T19:28:55.804 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.798+0000 7f8f4aabe700 1 -- 172.21.15.167:0/2730877213 <== osd.3 v2:172.21.15.167:6808/30036 1 ==== osd_op_reply(3 gc.2 [create,call,call,call] v32'23 uv21 ondisk = -125 ((125) Operation canceled)) v8 ==== 274+0+0 (crc 0 0 0) 0x7f8f3c050040 con 0x564523997240 2023-10-05T19:28:55.804 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.798+0000 7f8f4c788e40 1 -- 172.21.15.167:0/2730877213 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:4 6.4 6:2166d8e1:gc::gc.3:head [create,call version.check_conds in=74b,call rgw_gc.rgw_gc_queue_init in=45b,call version.set in=34b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e32) v8 -- 0x5645239a5a80 con 0x56452399b600 2023-10-05T19:28:55.807 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.802+0000 7f8f4a2bd700 1 -- 172.21.15.167:0/2730877213 <== osd.1 v2:172.21.15.167:6800/30035 2 ==== osd_op_reply(4 gc.3 [create,call,call,call] v32'197 uv184 ondisk = -125 ((125) Operation canceled)) v8 ==== 274+0+0 (crc 0 0 0) 0x7f8f4405f040 con 0x56452399b600 2023-10-05T19:28:55.808 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.802+0000 7f8f4c788e40 1 --2- 172.21.15.167:0/2730877213 >> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] conn(0x5645239a6720 0x5645239a8b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-05T19:28:55.808 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.802+0000 7f8f4c788e40 1 -- 172.21.15.167:0/2730877213 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:5 6.3 6:c60da066:gc::gc.4:head [create,call version.check_conds in=74b,call rgw_gc.rgw_gc_queue_init in=45b,call version.set in=34b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e32) v8 -- 0x5645239a91b0 con 0x5645239a6720 2023-10-05T19:28:55.808 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.802+0000 7f8f4a2bd700 1 --2- 172.21.15.167:0/2730877213 >> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] conn(0x5645239a6720 0x5645239a8b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-05T19:28:55.808 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.802+0000 7f8f4a2bd700 1 --2- 172.21.15.167:0/2730877213 >> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] conn(0x5645239a6720 0x5645239a8b00 crc :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-05T19:28:55.811 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.806+0000 7f8f4a2bd700 1 -- 172.21.15.167:0/2730877213 <== osd.2 v2:172.21.15.167:6816/30037 1 ==== osd_op_reply(5 gc.4 [create,call,call,call] v32'59 uv51 ondisk = -125 ((125) Operation canceled)) v8 ==== 274+0+0 (crc 0 0 0) 0x7f8f4405f040 con 0x5645239a6720 2023-10-05T19:28:55.811 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.806+0000 7f8f4c788e40 1 -- 172.21.15.167:0/2730877213 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:6 6.9 6:954be8d2:gc::gc.5:head [create,call version.check_conds in=74b,call rgw_gc.rgw_gc_queue_init in=45b,call version.set in=34b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e32) v8 -- 0x5645239a97d0 con 0x564523997240 2023-10-05T19:28:55.813 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.810+0000 7f8f4aabe700 1 -- 172.21.15.167:0/2730877213 <== osd.3 v2:172.21.15.167:6808/30036 2 ==== osd_op_reply(6 gc.5 [create,call,call,call] v32'42 uv37 ondisk = -125 ((125) Operation canceled)) v8 ==== 274+0+0 (crc 0 0 0) 0x7f8f3c050040 con 0x564523997240 2023-10-05T19:28:55.813 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.810+0000 7f8f4c788e40 1 -- 172.21.15.167:0/2730877213 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:7 6.e 6:748722ff:gc::gc.6:head [create,call version.check_conds in=74b,call rgw_gc.rgw_gc_queue_init in=45b,call version.set in=34b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e32) v8 -- 0x5645239a9df0 con 0x56452399b600 2023-10-05T19:28:55.816 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.810+0000 7f8f4a2bd700 1 -- 172.21.15.167:0/2730877213 <== osd.1 v2:172.21.15.167:6800/30035 3 ==== osd_op_reply(7 gc.6 [create,call,call,call] v32'25 uv23 ondisk = -125 ((125) Operation canceled)) v8 ==== 274+0+0 (crc 0 0 0) 0x7f8f4405f040 con 0x56452399b600 2023-10-05T19:28:55.816 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.810+0000 7f8f4c788e40 1 -- 172.21.15.167:0/2730877213 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:8 6.1d 6:beaa4be4:gc::gc.7:head [create,call version.check_conds in=74b,call rgw_gc.rgw_gc_queue_init in=45b,call version.set in=34b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e32) v8 -- 0x5645239a9df0 con 0x56452399ed30 2023-10-05T19:28:55.819 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.814+0000 7f8f49abc700 1 -- 172.21.15.167:0/2730877213 <== osd.0 v2:172.21.15.118:6800/27690 2 ==== osd_op_reply(8 gc.7 [create,call,call,call] v32'39 uv34 ondisk = -125 ((125) Operation canceled)) v8 ==== 274+0+0 (crc 0 0 0) 0x7f8f34006070 con 0x56452399ed30 2023-10-05T19:28:55.819 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.814+0000 7f8f4c788e40 1 -- 172.21.15.167:0/2730877213 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:9 6.12 6:4f78e9a2:gc::gc.8:head [create,call version.check_conds in=74b,call rgw_gc.rgw_gc_queue_init in=45b,call version.set in=34b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e32) v8 -- 0x5645239aa410 con 0x5645239a6720 2023-10-05T19:28:55.822 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.818+0000 7f8f4a2bd700 1 -- 172.21.15.167:0/2730877213 <== osd.2 v2:172.21.15.167:6816/30037 2 ==== osd_op_reply(9 gc.8 [create,call,call,call] v32'23 uv21 ondisk = -125 ((125) Operation canceled)) v8 ==== 274+0+0 (crc 0 0 0) 0x7f8f4405f040 con 0x5645239a6720 2023-10-05T19:28:55.822 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.818+0000 7f8f4c788e40 1 -- 172.21.15.167:0/2730877213 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:10 6.1c 6:3e426232:gc::gc.9:head [create,call version.check_conds in=74b,call rgw_gc.rgw_gc_queue_init in=45b,call version.set in=34b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e32) v8 -- 0x5645239aa410 con 0x564523997240 2023-10-05T19:28:55.825 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.818+0000 7f8f4aabe700 1 -- 172.21.15.167:0/2730877213 <== osd.3 v2:172.21.15.167:6808/30036 3 ==== osd_op_reply(10 gc.9 [create,call,call,call] v32'41 uv39 ondisk = -125 ((125) Operation canceled)) v8 ==== 274+0+0 (crc 0 0 0) 0x7f8f3c050040 con 0x564523997240 2023-10-05T19:28:55.825 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.818+0000 7f8f4c788e40 1 -- 172.21.15.167:0/2730877213 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:11 6.3 6:c31f41d7:gc::gc.10:head [create,call version.check_conds in=74b,call rgw_gc.rgw_gc_queue_init in=45b,call version.set in=34b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e32) v8 -- 0x5645239aa900 con 0x5645239a6720 2023-10-05T19:28:55.827 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.822+0000 7f8f4a2bd700 1 -- 172.21.15.167:0/2730877213 <== osd.2 v2:172.21.15.167:6816/30037 3 ==== osd_op_reply(11 gc.10 [create,call,call,call] v32'60 uv53 ondisk = -125 ((125) Operation canceled)) v8 ==== 275+0+0 (crc 0 0 0) 0x7f8f4405f040 con 0x5645239a6720 2023-10-05T19:28:55.828 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.822+0000 7f8f4c788e40 1 -- 172.21.15.167:0/2730877213 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:12 6.13 6:c8333aab:gc::gc.11:head [create,call version.check_conds in=74b,call rgw_gc.rgw_gc_queue_init in=45b,call version.set in=34b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e32) v8 -- 0x5645239aa9d0 con 0x56452399ed30 2023-10-05T19:28:55.831 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.826+0000 7f8f49abc700 1 -- 172.21.15.167:0/2730877213 <== osd.0 v2:172.21.15.118:6800/27690 3 ==== osd_op_reply(12 gc.11 [create,call,call,call] v32'27 uv25 ondisk = -125 ((125) Operation canceled)) v8 ==== 275+0+0 (crc 0 0 0) 0x7f8f34006070 con 0x56452399ed30 2023-10-05T19:28:55.831 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.826+0000 7f8f4c788e40 1 -- 172.21.15.167:0/2730877213 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:13 6.15 6:ab569f81:gc::gc.12:head [create,call version.check_conds in=74b,call rgw_gc.rgw_gc_queue_init in=45b,call version.set in=34b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e32) v8 -- 0x5645239ab070 con 0x5645239a6720 2023-10-05T19:28:55.833 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.830+0000 7f8f4a2bd700 1 -- 172.21.15.167:0/2730877213 <== osd.2 v2:172.21.15.167:6816/30037 4 ==== osd_op_reply(13 gc.12 [create,call,call,call] v32'23 uv21 ondisk = -125 ((125) Operation canceled)) v8 ==== 275+0+0 (crc 0 0 0) 0x7f8f4405f040 con 0x5645239a6720 2023-10-05T19:28:55.833 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.830+0000 7f8f4c788e40 1 -- 172.21.15.167:0/2730877213 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:14 6.0 6:0667bb13:gc::gc.13:head [create,call version.check_conds in=74b,call rgw_gc.rgw_gc_queue_init in=45b,call version.set in=34b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e32) v8 -- 0x5645239ab070 con 0x56452399ed30 2023-10-05T19:28:55.836 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.830+0000 7f8f49abc700 1 -- 172.21.15.167:0/2730877213 <== osd.0 v2:172.21.15.118:6800/27690 4 ==== osd_op_reply(14 gc.13 [create,call,call,call] v32'45 uv40 ondisk = -125 ((125) Operation canceled)) v8 ==== 275+0+0 (crc 0 0 0) 0x7f8f34006070 con 0x56452399ed30 2023-10-05T19:28:55.836 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.830+0000 7f8f4c788e40 1 -- 172.21.15.167:0/2730877213 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:15 6.4 6:227f63b5:gc::gc.14:head [create,call version.check_conds in=74b,call rgw_gc.rgw_gc_queue_init in=45b,call version.set in=34b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e32) v8 -- 0x5645239ab0a0 con 0x56452399b600 2023-10-05T19:28:55.839 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.834+0000 7f8f4a2bd700 1 -- 172.21.15.167:0/2730877213 <== osd.1 v2:172.21.15.167:6800/30035 4 ==== osd_op_reply(15 gc.14 [create,call,call,call] v32'198 uv186 ondisk = -125 ((125) Operation canceled)) v8 ==== 275+0+0 (crc 0 0 0) 0x7f8f4405f040 con 0x56452399b600 2023-10-05T19:28:55.839 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.834+0000 7f8f4c788e40 1 -- 172.21.15.167:0/2730877213 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:16 6.7 6:e07a6548:gc::gc.15:head [create,call version.check_conds in=74b,call rgw_gc.rgw_gc_queue_init in=45b,call version.set in=34b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e32) v8 -- 0x5645239ab130 con 0x564523997240 2023-10-05T19:28:55.842 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.838+0000 7f8f4aabe700 1 -- 172.21.15.167:0/2730877213 <== osd.3 v2:172.21.15.167:6808/30036 4 ==== osd_op_reply(16 gc.15 [create,call,call,call] v32'53 uv45 ondisk = -125 ((125) Operation canceled)) v8 ==== 275+0+0 (crc 0 0 0) 0x7f8f3c050040 con 0x564523997240 2023-10-05T19:28:55.842 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.838+0000 7f8f4c788e40 1 -- 172.21.15.167:0/2730877213 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:17 6.3 6:c623b173:gc::gc.16:head [create,call version.check_conds in=74b,call rgw_gc.rgw_gc_queue_init in=45b,call version.set in=34b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e32) v8 -- 0x5645239ab1c0 con 0x5645239a6720 2023-10-05T19:28:55.845 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.838+0000 7f8f4a2bd700 1 -- 172.21.15.167:0/2730877213 <== osd.2 v2:172.21.15.167:6816/30037 5 ==== osd_op_reply(17 gc.16 [create,call,call,call] v32'61 uv55 ondisk = -125 ((125) Operation canceled)) v8 ==== 275+0+0 (crc 0 0 0) 0x7f8f4405f040 con 0x5645239a6720 2023-10-05T19:28:55.845 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.842+0000 7f8f4c788e40 1 -- 172.21.15.167:0/2730877213 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:18 6.16 6:6849fd06:gc::gc.17:head [create,call version.check_conds in=74b,call rgw_gc.rgw_gc_queue_init in=45b,call version.set in=34b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e32) v8 -- 0x5645239ab1c0 con 0x564523997240 2023-10-05T19:28:55.848 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.842+0000 7f8f4aabe700 1 -- 172.21.15.167:0/2730877213 <== osd.3 v2:172.21.15.167:6808/30036 5 ==== osd_op_reply(18 gc.17 [create,call,call,call] v32'23 uv21 ondisk = -125 ((125) Operation canceled)) v8 ==== 275+0+0 (crc 0 0 0) 0x7f8f3c050040 con 0x564523997240 2023-10-05T19:28:55.848 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.842+0000 7f8f4c788e40 1 -- 172.21.15.167:0/2730877213 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:19 6.d 6:b019ea83:gc::gc.18:head [create,call version.check_conds in=74b,call rgw_gc.rgw_gc_queue_init in=45b,call version.set in=34b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e32) v8 -- 0x5645239ab250 con 0x56452399b600 2023-10-05T19:28:55.851 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.846+0000 7f8f4a2bd700 1 -- 172.21.15.167:0/2730877213 <== osd.1 v2:172.21.15.167:6800/30035 5 ==== osd_op_reply(19 gc.18 [create,call,call,call] v32'31 uv29 ondisk = -125 ((125) Operation canceled)) v8 ==== 275+0+0 (crc 0 0 0) 0x7f8f4405f040 con 0x56452399b600 2023-10-05T19:28:55.851 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.846+0000 7f8f4c788e40 1 -- 172.21.15.167:0/2730877213 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:20 6.4 6:24c8e1ae:gc::gc.19:head [create,call version.check_conds in=74b,call rgw_gc.rgw_gc_queue_init in=45b,call version.set in=34b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e32) v8 -- 0x5645239ab250 con 0x56452399b600 2023-10-05T19:28:55.853 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.846+0000 7f8f4a2bd700 1 -- 172.21.15.167:0/2730877213 <== osd.1 v2:172.21.15.167:6800/30035 6 ==== osd_op_reply(20 gc.19 [create,call,call,call] v32'199 uv189 ondisk = -125 ((125) Operation canceled)) v8 ==== 275+0+0 (crc 0 0 0) 0x7f8f4405f040 con 0x56452399b600 2023-10-05T19:28:55.853 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.850+0000 7f8f4c788e40 1 -- 172.21.15.167:0/2730877213 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:21 6.7 6:e622f76a:gc::gc.20:head [create,call version.check_conds in=74b,call rgw_gc.rgw_gc_queue_init in=45b,call version.set in=34b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e32) v8 -- 0x5645239ab250 con 0x564523997240 2023-10-05T19:28:55.856 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.850+0000 7f8f4aabe700 1 -- 172.21.15.167:0/2730877213 <== osd.3 v2:172.21.15.167:6808/30036 6 ==== osd_op_reply(21 gc.20 [create,call,call,call] v32'54 uv47 ondisk = -125 ((125) Operation canceled)) v8 ==== 275+0+0 (crc 0 0 0) 0x7f8f3c050040 con 0x564523997240 2023-10-05T19:28:55.856 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.850+0000 7f8f4c788e40 1 -- 172.21.15.167:0/2730877213 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:22 6.a 6:5127e642:gc::gc.21:head [create,call version.check_conds in=74b,call rgw_gc.rgw_gc_queue_init in=45b,call version.set in=34b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e32) v8 -- 0x5645239ab250 con 0x5645239a6720 2023-10-05T19:28:55.859 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.854+0000 7f8f4a2bd700 1 -- 172.21.15.167:0/2730877213 <== osd.2 v2:172.21.15.167:6816/30037 6 ==== osd_op_reply(22 gc.21 [create,call,call,call] v32'33 uv31 ondisk = -125 ((125) Operation canceled)) v8 ==== 275+0+0 (crc 0 0 0) 0x7f8f4405f040 con 0x5645239a6720 2023-10-05T19:28:55.859 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.854+0000 7f8f4c788e40 1 -- 172.21.15.167:0/2730877213 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:23 6.11 6:8eb99c53:gc::gc.22:head [create,call version.check_conds in=74b,call rgw_gc.rgw_gc_queue_init in=45b,call version.set in=34b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e32) v8 -- 0x5645239ab250 con 0x564523997240 2023-10-05T19:28:55.862 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.858+0000 7f8f4aabe700 1 -- 172.21.15.167:0/2730877213 <== osd.3 v2:172.21.15.167:6808/30036 7 ==== osd_op_reply(23 gc.22 [create,call,call,call] v32'25 uv23 ondisk = -125 ((125) Operation canceled)) v8 ==== 275+0+0 (crc 0 0 0) 0x7f8f3c050040 con 0x564523997240 2023-10-05T19:28:55.862 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.858+0000 7f8f4c788e40 1 -- 172.21.15.167:0/2730877213 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:24 6.8 6:17d7e8ed:gc::gc.23:head [create,call version.check_conds in=74b,call rgw_gc.rgw_gc_queue_init in=45b,call version.set in=34b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e32) v8 -- 0x5645239ab250 con 0x564523997240 2023-10-05T19:28:55.864 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.858+0000 7f8f4aabe700 1 -- 172.21.15.167:0/2730877213 <== osd.3 v2:172.21.15.167:6808/30036 8 ==== osd_op_reply(24 gc.23 [create,call,call,call] v32'35 uv33 ondisk = -125 ((125) Operation canceled)) v8 ==== 275+0+0 (crc 0 0 0) 0x7f8f3c050040 con 0x564523997240 2023-10-05T19:28:55.864 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.858+0000 7f8f4c788e40 1 -- 172.21.15.167:0/2730877213 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:25 6.1b 6:db9d130d:gc::gc.24:head [create,call version.check_conds in=74b,call rgw_gc.rgw_gc_queue_init in=45b,call version.set in=34b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e32) v8 -- 0x5645239ab250 con 0x564523997240 2023-10-05T19:28:55.867 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.862+0000 7f8f4aabe700 1 -- 172.21.15.167:0/2730877213 <== osd.3 v2:172.21.15.167:6808/30036 9 ==== osd_op_reply(25 gc.24 [create,call,call,call] v32'19 uv17 ondisk = -125 ((125) Operation canceled)) v8 ==== 275+0+0 (crc 0 0 0) 0x7f8f3c050040 con 0x564523997240 2023-10-05T19:28:55.867 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.862+0000 7f8f4c788e40 1 -- 172.21.15.167:0/2730877213 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:26 6.9 6:921ecdb1:gc::gc.25:head [create,call version.check_conds in=74b,call rgw_gc.rgw_gc_queue_init in=45b,call version.set in=34b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e32) v8 -- 0x5645239ab250 con 0x564523997240 2023-10-05T19:28:55.869 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.866+0000 7f8f4aabe700 1 -- 172.21.15.167:0/2730877213 <== osd.3 v2:172.21.15.167:6808/30036 10 ==== osd_op_reply(26 gc.25 [create,call,call,call] v32'43 uv39 ondisk = -125 ((125) Operation canceled)) v8 ==== 275+0+0 (crc 0 0 0) 0x7f8f3c050040 con 0x564523997240 2023-10-05T19:28:55.870 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.866+0000 7f8f4c788e40 1 -- 172.21.15.167:0/2730877213 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:27 6.1f 6:fde4dd55:gc::gc.26:head [create,call version.check_conds in=74b,call rgw_gc.rgw_gc_queue_init in=45b,call version.set in=34b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e32) v8 -- 0x5645239ab250 con 0x56452399ed30 2023-10-05T19:28:55.872 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.866+0000 7f8f49abc700 1 -- 172.21.15.167:0/2730877213 <== osd.0 v2:172.21.15.118:6800/27690 5 ==== osd_op_reply(27 gc.26 [create,call,call,call] v32'25 uv23 ondisk = -125 ((125) Operation canceled)) v8 ==== 275+0+0 (crc 0 0 0) 0x7f8f34006070 con 0x56452399ed30 2023-10-05T19:28:55.873 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.866+0000 7f8f4c788e40 1 -- 172.21.15.167:0/2730877213 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:28 6.1a 6:5909e671:gc::gc.27:head [create,call version.check_conds in=74b,call rgw_gc.rgw_gc_queue_init in=45b,call version.set in=34b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e32) v8 -- 0x5645239ab250 con 0x564523997240 2023-10-05T19:28:55.875 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.870+0000 7f8f4aabe700 1 -- 172.21.15.167:0/2730877213 <== osd.3 v2:172.21.15.167:6808/30036 11 ==== osd_op_reply(28 gc.27 [create,call,call,call] v32'23 uv21 ondisk = -125 ((125) Operation canceled)) v8 ==== 275+0+0 (crc 0 0 0) 0x7f8f3c050040 con 0x564523997240 2023-10-05T19:28:55.876 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.870+0000 7f8f4c788e40 1 -- 172.21.15.167:0/2730877213 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:29 6.1d 6:b88e3271:gc::gc.28:head [create,call version.check_conds in=74b,call rgw_gc.rgw_gc_queue_init in=45b,call version.set in=34b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e32) v8 -- 0x5645239ab250 con 0x56452399ed30 2023-10-05T19:28:55.878 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.874+0000 7f8f49abc700 1 -- 172.21.15.167:0/2730877213 <== osd.0 v2:172.21.15.118:6800/27690 6 ==== osd_op_reply(29 gc.28 [create,call,call,call] v32'40 uv36 ondisk = -125 ((125) Operation canceled)) v8 ==== 275+0+0 (crc 0 0 0) 0x7f8f34006070 con 0x56452399ed30 2023-10-05T19:28:55.878 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.874+0000 7f8f4c788e40 1 -- 172.21.15.167:0/2730877213 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:30 6.7 6:e47c9322:gc::gc.29:head [create,call version.check_conds in=74b,call rgw_gc.rgw_gc_queue_init in=45b,call version.set in=34b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e32) v8 -- 0x5645239ab250 con 0x564523997240 2023-10-05T19:28:55.881 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.874+0000 7f8f4aabe700 1 -- 172.21.15.167:0/2730877213 <== osd.3 v2:172.21.15.167:6808/30036 12 ==== osd_op_reply(30 gc.29 [create,call,call,call] v32'55 uv49 ondisk = -125 ((125) Operation canceled)) v8 ==== 275+0+0 (crc 0 0 0) 0x7f8f3c050040 con 0x564523997240 2023-10-05T19:28:55.881 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.874+0000 7f8f4c788e40 1 -- 172.21.15.167:0/2730877213 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:31 6.0 6:013d6c33:gc::gc.30:head [create,call version.check_conds in=74b,call rgw_gc.rgw_gc_queue_init in=45b,call version.set in=34b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e32) v8 -- 0x5645239ab250 con 0x56452399ed30 2023-10-05T19:28:55.883 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.878+0000 7f8f49abc700 1 -- 172.21.15.167:0/2730877213 <== osd.0 v2:172.21.15.118:6800/27690 7 ==== osd_op_reply(31 gc.30 [create,call,call,call] v32'46 uv42 ondisk = -125 ((125) Operation canceled)) v8 ==== 275+0+0 (crc 0 0 0) 0x7f8f34006070 con 0x56452399ed30 2023-10-05T19:28:55.883 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.878+0000 7f8f4c788e40 1 -- 172.21.15.167:0/2730877213 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:32 6.f 6:f24cb4a8:gc::gc.31:head [create,call version.check_conds in=74b,call rgw_gc.rgw_gc_queue_init in=45b,call version.set in=34b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e32) v8 -- 0x5645239ab250 con 0x5645239a6720 2023-10-05T19:28:55.886 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.882+0000 7f8f4a2bd700 1 -- 172.21.15.167:0/2730877213 <== osd.2 v2:172.21.15.167:6816/30037 7 ==== osd_op_reply(32 gc.31 [create,call,call,call] v32'29 uv27 ondisk = -125 ((125) Operation canceled)) v8 ==== 275+0+0 (crc 0 0 0) 0x7f8f4405f040 con 0x5645239a6720 2023-10-05T19:28:55.889 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.882+0000 7f8e8affd700 20 reqs_thread_entry: start 2023-10-05T19:28:55.890 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.886+0000 7f8e88ff9700 20 reqs_thread_entry: start 2023-10-05T19:28:55.890 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.886+0000 7f8f4c788e40 20 init_complete bucket index max shards: 11 2023-10-05T19:28:55.891 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.886+0000 7f8f4c788e40 10 rgw notify: Started notification manager with: 1 workers 2023-10-05T19:28:55.891 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.886+0000 7f8e7affd700 20 rgw notify: INFO: next queues processing will happen at: Thu Oct 5 19:29:26 2023 2023-10-05T19:28:55.891 INFO:teuthology.orchestra.run.smithi167.stderr: 2023-10-05T19:28:55.891 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.886+0000 7f8f4c788e40 20 get_system_obj_state: rctx=0x7ffe1934e698 obj=default.rgw.meta:users.uid:foo state=0x5645238140b0 s->prefetch_data=0 2023-10-05T19:28:55.891 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.886+0000 7f8f4c788e40 10 cache get: name=default.rgw.meta+users.uid+foo : miss 2023-10-05T19:28:55.891 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.886+0000 7f8f4c788e40 1 -- 172.21.15.167:0/1437397465 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:102 8.18 8:1e6ca5ed:users.uid::foo:head [call version.read in=11b,getxattrs,stat] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32) v8 -- 0x564523a09d30 con 0x5645238b0c20 2023-10-05T19:28:55.891 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.886+0000 7f8e7a7fc700 20 reqs_thread_entry: start 2023-10-05T19:28:55.892 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.886+0000 7f8f4aabe700 1 -- 172.21.15.167:0/1437397465 <== osd.1 v2:172.21.15.167:6800/30035 35 ==== osd_op_reply(102 foo [call out=48b,getxattrs out=75b,stat out=16b] v0'0 uv13 ondisk = 0) v8 ==== 231+0+139 (crc 0 0 0) 0x7f8f3c050040 con 0x5645238b0c20 2023-10-05T19:28:55.893 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.886+0000 7f8f4c788e40 10 cache put: name=default.rgw.meta+users.uid+foo info.flags=0x16 2023-10-05T19:28:55.893 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.886+0000 7f8f4c788e40 10 adding default.rgw.meta+users.uid+foo to cache LRU end 2023-10-05T19:28:55.893 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.886+0000 7f8f4c788e40 10 updating xattr: name=ceph.objclass.version bl.length()=42 2023-10-05T19:28:55.893 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.886+0000 7f8f4c788e40 20 get_system_obj_state: s->obj_tag was set empty 2023-10-05T19:28:55.893 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.886+0000 7f8f4c788e40 20 Read xattr: user.rgw.idtag 2023-10-05T19:28:55.893 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.886+0000 7f8f4c788e40 10 cache get: name=default.rgw.meta+users.uid+foo : type miss (requested=0x13, cached=0x16) 2023-10-05T19:28:55.893 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.886+0000 7f8f4c788e40 20 rados->read ofs=0 len=0 2023-10-05T19:28:55.893 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.886+0000 7f8f4c788e40 1 -- 172.21.15.167:0/1437397465 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:103 8.18 8:1e6ca5ed:users.uid::foo:head [call version.check_conds in=122b,call version.read in=11b,read 0~0,getxattrs] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32) v8 -- 0x564523a0c7c0 con 0x5645238b0c20 2023-10-05T19:28:55.893 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.886+0000 7f8f4aabe700 1 -- 172.21.15.167:0/1437397465 <== osd.1 v2:172.21.15.167:6800/30035 36 ==== osd_op_reply(103 foo [call,call out=48b,read 0~357 out=357b,getxattrs out=75b] v0'0 uv13 ondisk = 0) v8 ==== 273+0+480 (crc 0 0 0) 0x7f8f3c050040 con 0x5645238b0c20 2023-10-05T19:28:55.893 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.886+0000 7f8f4c788e40 20 rados_obj.operate() r=0 bl.length=357 2023-10-05T19:28:55.894 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.886+0000 7f8f4c788e40 10 cache put: name=default.rgw.meta+users.uid+foo info.flags=0x13 2023-10-05T19:28:55.894 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.886+0000 7f8f4c788e40 10 moving default.rgw.meta+users.uid+foo to cache LRU end 2023-10-05T19:28:55.894 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.886+0000 7f8f4c788e40 10 updating xattr: name=ceph.objclass.version bl.length()=42 2023-10-05T19:28:55.894 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.886+0000 7f8f4c788e40 1 -- 172.21.15.167:0/1437397465 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:104 8.9 8:96d98480:users.uid::foo.buckets:head [call user.list_buckets in=34b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32) v8 -- 0x564523a0c7c0 con 0x7f8f4401a3c0 2023-10-05T19:28:55.894 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.886+0000 7f8f49abc700 1 -- 172.21.15.167:0/1437397465 <== osd.2 v2:172.21.15.167:6816/30037 17 ==== osd_op_reply(104 foo.buckets [call out=183b] v0'0 uv26 ondisk = 0) v8 ==== 155+0+183 (crc 0 0 0) 0x7f8f34006070 con 0x7f8f4401a3c0 2023-10-05T19:28:55.895 INFO:teuthology.orchestra.run.smithi167.stderr:could not remove user: unable to remove user, must specify purge data to remove user with buckets 2023-10-05T19:28:55.895 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.890+0000 7f8f4c788e40 1 -- 172.21.15.167:0/1437397465 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:105 7.1f 7:f95f44c2:::notify.0:head [watch unwatch cookie 94854949860128] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e32) v8 -- 0x5645239c82e0 con 0x7f8f4401a3c0 2023-10-05T19:28:55.896 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.890+0000 7f8f49abc700 1 -- 172.21.15.167:0/1437397465 <== osd.2 v2:172.21.15.167:6816/30037 18 ==== osd_op_reply(105 notify.0 [watch unwatch cookie 94854949860128] v32'200 uv198 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f8f34006070 con 0x7f8f4401a3c0 2023-10-05T19:28:55.896 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.890+0000 7f8f4c788e40 20 remove_watcher() i=0 2023-10-05T19:28:55.896 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.890+0000 7f8f4c788e40 2 removed watcher, disabling cache 2023-10-05T19:28:55.896 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.890+0000 7f8f4c788e40 1 -- 172.21.15.167:0/1437397465 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:106 7.0 7:05bf5b68:::notify.1:head [watch unwatch cookie 94854949865056] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e32) v8 -- 0x564523952760 con 0x5645238b0c20 2023-10-05T19:28:55.898 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.894+0000 7f8f4aabe700 1 -- 172.21.15.167:0/1437397465 <== osd.1 v2:172.21.15.167:6800/30035 37 ==== osd_op_reply(106 notify.1 [watch unwatch cookie 94854949865056] v32'200 uv198 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f8f3c050040 con 0x5645238b0c20 2023-10-05T19:28:55.898 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.894+0000 7f8f4c788e40 20 remove_watcher() i=1 2023-10-05T19:28:55.899 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.894+0000 7f8f4c788e40 1 -- 172.21.15.167:0/1437397465 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:107 7.15 7:a93a5511:::notify.2:head [watch unwatch cookie 94854949868320] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e32) v8 -- 0x564523952760 con 0x564523972390 2023-10-05T19:28:55.902 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.898+0000 7f8f4a2bd700 1 -- 172.21.15.167:0/1437397465 <== osd.3 v2:172.21.15.167:6808/30036 24 ==== osd_op_reply(107 notify.2 [watch unwatch cookie 94854949868320] v32'200 uv198 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f8f4405f040 con 0x564523972390 2023-10-05T19:28:55.902 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.898+0000 7f8f4c788e40 20 remove_watcher() i=2 2023-10-05T19:28:55.902 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.898+0000 7f8f4c788e40 1 -- 172.21.15.167:0/1437397465 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:108 7.e 7:7759931f:::notify.3:head [watch unwatch cookie 94854949867056] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e32) v8 -- 0x564523952760 con 0x7f8f4401a3c0 2023-10-05T19:28:55.904 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.898+0000 7f8f49abc700 1 -- 172.21.15.167:0/1437397465 <== osd.2 v2:172.21.15.167:6816/30037 19 ==== osd_op_reply(108 notify.3 [watch unwatch cookie 94854949867056] v32'200 uv198 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f8f34006070 con 0x7f8f4401a3c0 2023-10-05T19:28:55.904 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.898+0000 7f8f4c788e40 20 remove_watcher() i=3 2023-10-05T19:28:55.904 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.898+0000 7f8f4c788e40 1 -- 172.21.15.167:0/1437397465 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:109 7.d 7:b4812045:::notify.4:head [watch unwatch cookie 94854949875312] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e32) v8 -- 0x564523952760 con 0x5645238b0c20 2023-10-05T19:28:55.907 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.902+0000 7f8f4aabe700 1 -- 172.21.15.167:0/1437397465 <== osd.1 v2:172.21.15.167:6800/30035 38 ==== osd_op_reply(109 notify.4 [watch unwatch cookie 94854949875312] v32'200 uv198 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f8f3c050040 con 0x5645238b0c20 2023-10-05T19:28:55.907 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.902+0000 7f8f4c788e40 20 remove_watcher() i=4 2023-10-05T19:28:55.907 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.902+0000 7f8f4c788e40 1 -- 172.21.15.167:0/1437397465 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:110 7.3 7:c609908c:::notify.5:head [watch unwatch cookie 94854949878864] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e32) v8 -- 0x564523952760 con 0x7f8f3400d8e0 2023-10-05T19:28:55.910 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.906+0000 7f8f4aabe700 1 -- 172.21.15.167:0/1437397465 <== osd.0 v2:172.21.15.118:6800/27690 29 ==== osd_op_reply(110 notify.5 [watch unwatch cookie 94854949878864] v32'200 uv198 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f8f3c050040 con 0x7f8f3400d8e0 2023-10-05T19:28:55.910 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.906+0000 7f8f4c788e40 20 remove_watcher() i=5 2023-10-05T19:28:55.910 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.906+0000 7f8f4c788e40 1 -- 172.21.15.167:0/1437397465 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:111 7.14 7:2b04a3e9:::notify.6:head [watch unwatch cookie 94854949877504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e32) v8 -- 0x564523952760 con 0x7f8f3400d8e0 2023-10-05T19:28:55.913 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.906+0000 7f8f4aabe700 1 -- 172.21.15.167:0/1437397465 <== osd.0 v2:172.21.15.118:6800/27690 30 ==== osd_op_reply(111 notify.6 [watch unwatch cookie 94854949877504] v32'200 uv198 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f8f3c050040 con 0x7f8f3400d8e0 2023-10-05T19:28:55.913 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.906+0000 7f8f4c788e40 20 remove_watcher() i=6 2023-10-05T19:28:55.913 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.906+0000 7f8f4c788e40 1 -- 172.21.15.167:0/1437397465 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:112 7.9 7:93e5b521:::notify.7:head [watch unwatch cookie 94854949883616] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e32) v8 -- 0x564523952760 con 0x7f8f3400d8e0 2023-10-05T19:28:55.916 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.910+0000 7f8f4aabe700 1 -- 172.21.15.167:0/1437397465 <== osd.0 v2:172.21.15.118:6800/27690 31 ==== osd_op_reply(112 notify.7 [watch unwatch cookie 94854949883616] v32'200 uv198 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f8f3c050040 con 0x7f8f3400d8e0 2023-10-05T19:28:55.916 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.910+0000 7f8f4c788e40 20 remove_watcher() i=7 2023-10-05T19:28:55.917 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.914+0000 7f8f1bfdf700 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2023-10-05T19:28:55.917 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.914+0000 7f8f4c788e40 1 -- 172.21.15.167:0/1437397465 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:113 6.4 6:22d26bf9:::data_loggenerations_metadata:head [watch unwatch cookie 94854949884720] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e32) v8 -- 0x564523952760 con 0x5645238b0c20 2023-10-05T19:28:55.921 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.914+0000 7f8f4aabe700 1 -- 172.21.15.167:0/1437397465 <== osd.1 v2:172.21.15.167:6800/30035 39 ==== osd_op_reply(113 data_loggenerations_metadata [watch unwatch cookie 94854949884720] v32'200 uv1 ondisk = 0) v8 ==== 172+0+0 (crc 0 0 0) 0x7f8f3c050040 con 0x5645238b0c20 2023-10-05T19:28:55.921 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.914+0000 7f8f4c788e40 1 -- 172.21.15.167:0/1437397465 >> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] conn(0x7f8f3400d8e0 msgr2=0x7f8f3400c360 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-05T19:28:55.921 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.914+0000 7f8f4c788e40 1 --2- 172.21.15.167:0/1437397465 >> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] conn(0x7f8f3400d8e0 0x7f8f3400c360 crc :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-05T19:28:55.921 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.914+0000 7f8f4c788e40 1 -- 172.21.15.167:0/1437397465 >> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] conn(0x5645238b0c20 msgr2=0x56452396e5b0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-05T19:28:55.921 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.914+0000 7f8f4c788e40 1 --2- 172.21.15.167:0/1437397465 >> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] conn(0x5645238b0c20 0x56452396e5b0 crc :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-05T19:28:55.921 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.914+0000 7f8f4c788e40 1 -- 172.21.15.167:0/1437397465 >> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] conn(0x7f8f4401a3c0 msgr2=0x7f8f44039540 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-05T19:28:55.922 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.914+0000 7f8f4c788e40 1 --2- 172.21.15.167:0/1437397465 >> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] conn(0x7f8f4401a3c0 0x7f8f44039540 crc :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-05T19:28:55.922 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.914+0000 7f8f4c788e40 1 -- 172.21.15.167:0/1437397465 >> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] conn(0x564523972390 msgr2=0x564523974770 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-05T19:28:55.922 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.914+0000 7f8f4c788e40 1 --2- 172.21.15.167:0/1437397465 >> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] conn(0x564523972390 0x564523974770 crc :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-05T19:28:55.922 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.914+0000 7f8f4c788e40 1 -- 172.21.15.167:0/1437397465 >> [v2:172.21.15.167:6824/29889,v1:172.21.15.167:6825/29889] conn(0x7f8f2c030c00 msgr2=0x7f8f2c0330b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-05T19:28:55.922 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.914+0000 7f8f4c788e40 1 --2- 172.21.15.167:0/1437397465 >> [v2:172.21.15.167:6824/29889,v1:172.21.15.167:6825/29889] conn(0x7f8f2c030c00 0x7f8f2c0330b0 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7f8f340011e0 tx=0x7f8f340013a0 comp rx=0 tx=0).stop 2023-10-05T19:28:55.922 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.914+0000 7f8f4c788e40 1 -- 172.21.15.167:0/1437397465 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5645238af740 msgr2=0x564523962b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-05T19:28:55.922 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.914+0000 7f8f4aabe700 1 -- 172.21.15.167:0/1437397465 reap_dead start 2023-10-05T19:28:55.922 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.914+0000 7f8f4c788e40 1 --2- 172.21.15.167:0/1437397465 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5645238af740 0x564523962b20 secure :-1 s=READY pgs=533 cs=0 l=1 rev1=1 crypto rx=0x7f8f4400a470 tx=0x7f8f4401d000 comp rx=0 tx=0).stop 2023-10-05T19:28:55.923 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.914+0000 7f8f4c788e40 1 -- 172.21.15.167:0/1437397465 shutdown_connections 2023-10-05T19:28:55.923 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.914+0000 7f8f4c788e40 1 --2- 172.21.15.167:0/1437397465 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5645238af740 0x564523962b20 unknown :-1 s=CLOSED pgs=533 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-05T19:28:55.923 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.914+0000 7f8f4c788e40 1 -- 172.21.15.167:0/1437397465 >> 172.21.15.167:0/1437397465 conn(0x564523936850 msgr2=0x564523956500 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-05T19:28:55.923 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.914+0000 7f8f4c788e40 1 -- 172.21.15.167:0/1437397465 shutdown_connections 2023-10-05T19:28:55.923 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.914+0000 7f8f4c788e40 1 -- 172.21.15.167:0/1437397465 wait complete. 2023-10-05T19:28:55.924 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.918+0000 7f8f4c788e40 1 -- 172.21.15.167:0/2730877213 >> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] conn(0x56452399ed30 msgr2=0x5645239a1110 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-05T19:28:55.925 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.918+0000 7f8f4c788e40 1 --2- 172.21.15.167:0/2730877213 >> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] conn(0x56452399ed30 0x5645239a1110 crc :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-05T19:28:55.925 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.918+0000 7f8f4c788e40 1 -- 172.21.15.167:0/2730877213 >> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] conn(0x56452399b600 msgr2=0x56452399da10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-05T19:28:55.925 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.918+0000 7f8f4c788e40 1 --2- 172.21.15.167:0/2730877213 >> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] conn(0x56452399b600 0x56452399da10 crc :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-05T19:28:55.925 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.918+0000 7f8f4c788e40 1 -- 172.21.15.167:0/2730877213 >> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] conn(0x5645239a6720 msgr2=0x5645239a8b00 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-05T19:28:55.925 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.918+0000 7f8f4c788e40 1 --2- 172.21.15.167:0/2730877213 >> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] conn(0x5645239a6720 0x5645239a8b00 crc :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-05T19:28:55.925 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.918+0000 7f8f4c788e40 1 -- 172.21.15.167:0/2730877213 >> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] conn(0x564523997240 msgr2=0x5645239a4eb0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-05T19:28:55.925 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.918+0000 7f8f4c788e40 1 --2- 172.21.15.167:0/2730877213 >> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] conn(0x564523997240 0x5645239a4eb0 crc :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-05T19:28:55.925 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.918+0000 7f8f4c788e40 1 -- 172.21.15.167:0/2730877213 >> [v2:172.21.15.167:6824/29889,v1:172.21.15.167:6825/29889] conn(0x7f8e90030c50 msgr2=0x7f8e90033100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-05T19:28:55.925 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.918+0000 7f8f4c788e40 1 --2- 172.21.15.167:0/2730877213 >> [v2:172.21.15.167:6824/29889,v1:172.21.15.167:6825/29889] conn(0x7f8e90030c50 0x7f8e90033100 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x56452397bbc0 tx=0x7f8f3c000f90 comp rx=0 tx=0).stop 2023-10-05T19:28:55.926 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.918+0000 7f8f4c788e40 1 -- 172.21.15.167:0/2730877213 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x56452398ae30 msgr2=0x564523990a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-05T19:28:55.926 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.918+0000 7f8f4c788e40 1 --2- 172.21.15.167:0/2730877213 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x56452398ae30 0x564523990a10 secure :-1 s=READY pgs=535 cs=0 l=1 rev1=1 crypto rx=0x564523984eb0 tx=0x7f8f34030000 comp rx=0 tx=0).stop 2023-10-05T19:28:55.926 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.918+0000 7f8f4c788e40 1 -- 172.21.15.167:0/2730877213 shutdown_connections 2023-10-05T19:28:55.926 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.918+0000 7f8f4c788e40 1 --2- 172.21.15.167:0/2730877213 >> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] conn(0x564523997240 0x5645239a4eb0 unknown :-1 s=CLOSED pgs=96 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-05T19:28:55.926 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.918+0000 7f8f4c788e40 1 --2- 172.21.15.167:0/2730877213 >> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] conn(0x56452399ed30 0x5645239a1110 unknown :-1 s=CLOSED pgs=91 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-05T19:28:55.926 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.918+0000 7f8f4c788e40 1 --2- 172.21.15.167:0/2730877213 >> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] conn(0x56452399b600 0x56452399da10 unknown :-1 s=CLOSED pgs=93 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-05T19:28:55.926 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.918+0000 7f8f4c788e40 1 --2- 172.21.15.167:0/2730877213 >> [v2:172.21.15.167:6824/29889,v1:172.21.15.167:6825/29889] conn(0x7f8e90030c50 0x7f8e90033100 unknown :-1 s=CLOSED pgs=185 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-05T19:28:55.926 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.918+0000 7f8f4c788e40 1 --2- 172.21.15.167:0/2730877213 >> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] conn(0x5645239a6720 0x5645239a8b00 unknown :-1 s=CLOSED pgs=90 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-05T19:28:55.926 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.918+0000 7f8f4c788e40 1 --2- 172.21.15.167:0/2730877213 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x56452398ae30 0x564523990a10 unknown :-1 s=CLOSED pgs=535 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-05T19:28:55.926 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.918+0000 7f8f4c788e40 1 -- 172.21.15.167:0/2730877213 >> 172.21.15.167:0/2730877213 conn(0x5645239861f0 msgr2=0x564523984290 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-05T19:28:55.926 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.918+0000 7f8f4a2bd700 1 -- 172.21.15.167:0/2730877213 reap_dead start 2023-10-05T19:28:55.927 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.918+0000 7f8f4c788e40 1 -- 172.21.15.167:0/2730877213 shutdown_connections 2023-10-05T19:28:55.927 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:55.918+0000 7f8f4c788e40 1 -- 172.21.15.167:0/2730877213 wait complete. 2023-10-05T19:28:55.927 DEBUG:teuthology.orchestra.run:got remote process result: 17 2023-10-05T19:28:55.945 INFO:tasks.radosgw_admin:RL: delete_bucket myfoo foo bytes_out=0 bytes_in=217 failed=True 2023-10-05T19:28:55.954 INFO:tasks.radosgw_admin:RL: delete_obj myfoo foo bytes_out=0 bytes_in=0 failed=False 2023-10-05T19:28:56.035 INFO:tasks.radosgw_admin:RL: delete_bucket myfoo foo bytes_out=0 bytes_in=0 failed=False 2023-10-05T19:28:56.055 INFO:tasks.radosgw_admin:RL: create_bucket myfoo foo bytes_out=0 bytes_in=0 failed=False 2023-10-05T19:28:56.103 INFO:tasks.radosgw_admin:RL: put_obj myfoo foo bytes_out=5 bytes_in=0 failed=False 2023-10-05T19:28:56.115 INFO:tasks.radosgw_admin:RL: put_acls myfoo foo bytes_out=0 bytes_in=0 failed=False 2023-10-05T19:28:56.115 INFO:tasks.util.rgw:rgwadmin: client.0 : ['policy', '--bucket', 'myfoo', '--object', 'bb3aec0fdcdbc2974890f805c585d432'] 2023-10-05T19:28:56.115 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'xml', '-n', 'client.0', '--cluster', 'ceph', 'policy', '--bucket', 'myfoo', '--object', 'bb3aec0fdcdbc2974890f805c585d432'] 2023-10-05T19:28:56.115 DEBUG:teuthology.orchestra.run.smithi167:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format xml -n client.0 --cluster ceph policy --bucket myfoo --object bb3aec0fdcdbc2974890f805c585d432 2023-10-05T19:28:56.154 INFO:teuthology.orchestra.run.smithi167.stderr:ignoring --setuser ceph since I am not root 2023-10-05T19:28:56.154 INFO:teuthology.orchestra.run.smithi167.stderr:ignoring --setgroup ceph since I am not root 2023-10-05T19:28:56.155 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.150+0000 7f3daeb5fe40 1 Processor -- start 2023-10-05T19:28:56.155 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.150+0000 7f3daeb5fe40 1 -- start start 2023-10-05T19:28:56.155 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.150+0000 7f3daeb5fe40 1 --2- >> v2:172.21.15.118:3300/0 conn(0x563030d64420 0x563030d62fd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-05T19:28:56.155 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.150+0000 7f3daeb5fe40 1 -- --> v1:172.21.15.118:6789/0 -- auth(proto 0 26 bytes epoch 0) v1 -- 0x563030cb0bd0 con 0x563030d647f0 2023-10-05T19:28:56.155 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.150+0000 7f3daeb5fe40 1 -- --> v2:172.21.15.118:3300/0 -- mon_getmap magic: 0 v1 -- 0x563030c6dee0 con 0x563030d64420 2023-10-05T19:28:56.157 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.150+0000 7f3dac694700 1 --2- >> v2:172.21.15.118:3300/0 conn(0x563030d64420 0x563030d62fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-05T19:28:56.157 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.150+0000 7f3dac694700 1 --2- >> v2:172.21.15.118:3300/0 conn(0x563030d64420 0x563030d62fd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.118:3300/0 says I am v2:172.21.15.167:53302/0 (socket says 172.21.15.167:53302) 2023-10-05T19:28:56.157 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.150+0000 7f3dac694700 1 -- 172.21.15.167:0/1562280099 learned_addr learned my addr 172.21.15.167:0/1562280099 (peer_addr_for_me v2:172.21.15.167:0/0) 2023-10-05T19:28:56.157 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.150+0000 7f3dabe93700 1 --1- 172.21.15.167:0/1562280099 >> v1:172.21.15.118:6789/0 conn(0x563030d647f0 0x563030d63510 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.118:6789/0 says I am v1:172.21.15.167:48690/0 (socket says 172.21.15.167:48690) 2023-10-05T19:28:56.157 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.150+0000 7f3dac694700 1 -- 172.21.15.167:0/1562280099 >> v1:172.21.15.118:6789/0 conn(0x563030d647f0 legacy=0x563030d63510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-05T19:28:56.157 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.150+0000 7f3dac694700 1 -- 172.21.15.167:0/1562280099 --> v2:172.21.15.118:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x563030d63ce0 con 0x563030d64420 2023-10-05T19:28:56.157 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.150+0000 7f3dac694700 1 --2- 172.21.15.167:0/1562280099 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x563030d64420 0x563030d62fd0 secure :-1 s=READY pgs=537 cs=0 l=1 rev1=1 crypto rx=0x7f3d9400a650 tx=0x7f3d94005b10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5a91e422d5ecc89 server_cookie=0 in_seq=0 out_seq=0 2023-10-05T19:28:56.157 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.150+0000 7f3dab692700 1 -- 172.21.15.167:0/1562280099 <== mon.0 v2:172.21.15.118:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f3d94008760 con 0x563030d64420 2023-10-05T19:28:56.157 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.150+0000 7f3dab692700 1 -- 172.21.15.167:0/1562280099 <== mon.0 v2:172.21.15.118:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3d940088c0 con 0x563030d64420 2023-10-05T19:28:56.158 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.150+0000 7f3dab692700 1 -- 172.21.15.167:0/1562280099 <== mon.0 v2:172.21.15.118:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f3d94008bb0 con 0x563030d64420 2023-10-05T19:28:56.159 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.150+0000 7f3daeb5fe40 1 -- 172.21.15.167:0/1562280099 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x563030d64420 msgr2=0x563030d62fd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-05T19:28:56.159 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.150+0000 7f3daeb5fe40 1 --2- 172.21.15.167:0/1562280099 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x563030d64420 0x563030d62fd0 secure :-1 s=READY pgs=537 cs=0 l=1 rev1=1 crypto rx=0x7f3d9400a650 tx=0x7f3d94005b10 comp rx=0 tx=0).stop 2023-10-05T19:28:56.159 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.150+0000 7f3daeb5fe40 1 -- 172.21.15.167:0/1562280099 shutdown_connections 2023-10-05T19:28:56.159 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.150+0000 7f3daeb5fe40 1 --2- 172.21.15.167:0/1562280099 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x563030d64420 0x563030d62fd0 unknown :-1 s=CLOSED pgs=537 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-05T19:28:56.160 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.150+0000 7f3daeb5fe40 1 -- 172.21.15.167:0/1562280099 >> 172.21.15.167:0/1562280099 conn(0x563030d5e730 msgr2=0x563030d60b40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-05T19:28:56.160 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.154+0000 7f3daeb5fe40 1 -- 172.21.15.167:0/1562280099 shutdown_connections 2023-10-05T19:28:56.160 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.154+0000 7f3daeb5fe40 1 -- 172.21.15.167:0/1562280099 wait complete. 2023-10-05T19:28:56.160 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.154+0000 7f3daeb5fe40 1 Processor -- start 2023-10-05T19:28:56.160 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.154+0000 7f3daeb5fe40 1 -- start start 2023-10-05T19:28:56.160 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.154+0000 7f3daeb5fe40 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x563030d64420 0x563030d851d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-05T19:28:56.160 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.154+0000 7f3daeb5fe40 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x563030c31e30 con 0x563030d64420 2023-10-05T19:28:56.160 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.154+0000 7f3dac694700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x563030d64420 0x563030d851d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-05T19:28:56.160 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.154+0000 7f3dac694700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x563030d64420 0x563030d851d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.118:3300/0 says I am v2:172.21.15.167:53314/0 (socket says 172.21.15.167:53314) 2023-10-05T19:28:56.161 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.154+0000 7f3dac694700 1 -- 172.21.15.167:0/637732725 learned_addr learned my addr 172.21.15.167:0/637732725 (peer_addr_for_me v2:172.21.15.167:0/0) 2023-10-05T19:28:56.162 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.154+0000 7f3dac694700 1 -- 172.21.15.167:0/637732725 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3d94005db0 con 0x563030d64420 2023-10-05T19:28:56.162 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.154+0000 7f3dac694700 1 --2- 172.21.15.167:0/637732725 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x563030d64420 0x563030d851d0 secure :-1 s=READY pgs=538 cs=0 l=1 rev1=1 crypto rx=0x7f3d94005b50 tx=0x7f3d94015000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a9bb9efc9299e5c8 server_cookie=0 in_seq=0 out_seq=0 2023-10-05T19:28:56.163 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.154+0000 7f3daa690700 1 -- 172.21.15.167:0/637732725 <== mon.0 v2:172.21.15.118:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f3d940055f0 con 0x563030d64420 2023-10-05T19:28:56.163 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.154+0000 7f3daa690700 1 -- 172.21.15.167:0/637732725 <== mon.0 v2:172.21.15.118:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3d94008a30 con 0x563030d64420 2023-10-05T19:28:56.163 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.154+0000 7f3daa690700 1 -- 172.21.15.167:0/637732725 <== mon.0 v2:172.21.15.118:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f3d94008d00 con 0x563030d64420 2023-10-05T19:28:56.163 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.154+0000 7f3daeb5fe40 1 -- 172.21.15.167:0/637732725 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x563030d64420 msgr2=0x563030d851d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-05T19:28:56.163 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.154+0000 7f3daeb5fe40 1 --2- 172.21.15.167:0/637732725 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x563030d64420 0x563030d851d0 secure :-1 s=READY pgs=538 cs=0 l=1 rev1=1 crypto rx=0x7f3d94005b50 tx=0x7f3d94015000 comp rx=0 tx=0).stop 2023-10-05T19:28:56.163 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.154+0000 7f3daeb5fe40 1 -- 172.21.15.167:0/637732725 shutdown_connections 2023-10-05T19:28:56.163 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.154+0000 7f3daeb5fe40 1 --2- 172.21.15.167:0/637732725 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x563030d64420 0x563030d851d0 unknown :-1 s=CLOSED pgs=538 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-05T19:28:56.163 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.154+0000 7f3daeb5fe40 1 -- 172.21.15.167:0/637732725 >> 172.21.15.167:0/637732725 conn(0x563030d5e730 msgr2=0x563030d80d90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-05T19:28:56.164 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.154+0000 7f3daeb5fe40 1 -- 172.21.15.167:0/637732725 shutdown_connections 2023-10-05T19:28:56.164 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.154+0000 7f3daeb5fe40 1 -- 172.21.15.167:0/637732725 wait complete. 2023-10-05T19:28:56.164 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.158+0000 7f3daeb5fe40 1 Processor -- start 2023-10-05T19:28:56.164 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.158+0000 7f3daeb5fe40 1 -- start start 2023-10-05T19:28:56.164 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.158+0000 7f3daeb5fe40 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x563030d64420 0x563030d8be30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-05T19:28:56.164 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.158+0000 7f3daeb5fe40 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x563030c04ed0 con 0x563030d64420 2023-10-05T19:28:56.164 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.158+0000 7f3dac694700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x563030d64420 0x563030d8be30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-05T19:28:56.164 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.158+0000 7f3dac694700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x563030d64420 0x563030d8be30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.118:3300/0 says I am v2:172.21.15.167:53318/0 (socket says 172.21.15.167:53318) 2023-10-05T19:28:56.164 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.158+0000 7f3dac694700 1 -- 172.21.15.167:0/1367369157 learned_addr learned my addr 172.21.15.167:0/1367369157 (peer_addr_for_me v2:172.21.15.167:0/0) 2023-10-05T19:28:56.166 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.158+0000 7f3dac694700 1 -- 172.21.15.167:0/1367369157 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x563030d63ce0 con 0x563030d64420 2023-10-05T19:28:56.166 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.158+0000 7f3dac694700 1 --2- 172.21.15.167:0/1367369157 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x563030d64420 0x563030d8be30 secure :-1 s=READY pgs=539 cs=0 l=1 rev1=1 crypto rx=0x7f3d9400a650 tx=0x7f3d9401c000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-05T19:28:56.166 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.158+0000 7f3da8e8d700 1 -- 172.21.15.167:0/1367369157 <== mon.0 v2:172.21.15.118:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f3d940055b0 con 0x563030d64420 2023-10-05T19:28:56.166 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.158+0000 7f3da8e8d700 1 -- 172.21.15.167:0/1367369157 <== mon.0 v2:172.21.15.118:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3d940269b0 con 0x563030d64420 2023-10-05T19:28:56.166 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.158+0000 7f3daeb5fe40 1 -- 172.21.15.167:0/1367369157 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x563030d8c370 con 0x563030d64420 2023-10-05T19:28:56.166 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.158+0000 7f3da8e8d700 1 -- 172.21.15.167:0/1367369157 <== mon.0 v2:172.21.15.118:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f3d94026ca0 con 0x563030d64420 2023-10-05T19:28:56.166 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.158+0000 7f3daeb5fe40 1 -- 172.21.15.167:0/1367369157 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x563030d8c810 con 0x563030d64420 2023-10-05T19:28:56.166 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.162+0000 7f3da8e8d700 1 -- 172.21.15.167:0/1367369157 <== mon.0 v2:172.21.15.118:3300/0 4 ==== mgrmap(e 5) v1 ==== 40329+0+0 (secure 0 0 0) 0x7f3d9402e030 con 0x563030d64420 2023-10-05T19:28:56.167 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.162+0000 7f3da8e8d700 1 --2- 172.21.15.167:0/1367369157 >> [v2:172.21.15.167:6824/29889,v1:172.21.15.167:6825/29889] conn(0x7f3d8c030c50 0x7f3d8c033100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-05T19:28:56.167 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.162+0000 7f3da8e8d700 1 -- 172.21.15.167:0/1367369157 <== mon.0 v2:172.21.15.118:3300/0 5 ==== osd_map(32..32 src has 1..32) v4 ==== 6260+0+0 (secure 0 0 0) 0x7f3d9403bce0 con 0x563030d64420 2023-10-05T19:28:56.168 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.162+0000 7f3dabe93700 1 --2- 172.21.15.167:0/1367369157 >> [v2:172.21.15.167:6824/29889,v1:172.21.15.167:6825/29889] conn(0x7f3d8c030c50 0x7f3d8c033100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-05T19:28:56.168 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.162+0000 7f3daeb5fe40 20 rados->read ofs=0 len=0 2023-10-05T19:28:56.168 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.162+0000 7f3daeb5fe40 1 --2- 172.21.15.167:0/1367369157 >> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] conn(0x563030d647f0 0x563030d978d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-05T19:28:56.168 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.162+0000 7f3daeb5fe40 1 -- 172.21.15.167:0/1367369157 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:1 5.12 5:49953fa1:::default.realm:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32) v8 -- 0x563030d97f80 con 0x563030d647f0 2023-10-05T19:28:56.169 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.162+0000 7f3dace95700 1 --2- 172.21.15.167:0/1367369157 >> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] conn(0x563030d647f0 0x563030d978d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-05T19:28:56.169 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.162+0000 7f3dabe93700 1 --2- 172.21.15.167:0/1367369157 >> [v2:172.21.15.167:6824/29889,v1:172.21.15.167:6825/29889] conn(0x7f3d8c030c50 0x7f3d8c033100 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7f3d9c0016b0 tx=0x7f3d9c0067b0 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-05T19:28:56.169 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.162+0000 7f3dace95700 1 --2- 172.21.15.167:0/1367369157 >> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] conn(0x563030d647f0 0x563030d978d0 crc :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-05T19:28:56.169 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.162+0000 7f3dace95700 1 -- 172.21.15.167:0/1367369157 <== osd.1 v2:172.21.15.167:6800/30035 1 ==== osd_op_reply(1 default.realm [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 ==== 157+0+0 (crc 0 0 0) 0x7f3da400a500 con 0x563030d647f0 2023-10-05T19:28:56.169 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.162+0000 7f3daeb5fe40 20 rados_obj.operate() r=-2 bl.length=0 2023-10-05T19:28:56.169 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.162+0000 7f3daeb5fe40 20 realm 2023-10-05T19:28:56.169 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.162+0000 7f3daeb5fe40 20 rados->read ofs=0 len=0 2023-10-05T19:28:56.169 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.162+0000 7f3daeb5fe40 1 -- 172.21.15.167:0/1367369157 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:2 5.12 5:49953fa1:::default.realm:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32) v8 -- 0x563030d94f30 con 0x563030d647f0 2023-10-05T19:28:56.171 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.162+0000 7f3dace95700 1 -- 172.21.15.167:0/1367369157 <== osd.1 v2:172.21.15.167:6800/30035 2 ==== osd_op_reply(2 default.realm [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 ==== 157+0+0 (crc 0 0 0) 0x7f3da400a500 con 0x563030d647f0 2023-10-05T19:28:56.171 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.162+0000 7f3daeb5fe40 20 rados_obj.operate() r=-2 bl.length=0 2023-10-05T19:28:56.171 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.162+0000 7f3daeb5fe40 4 RGWPeriod::init failed to init realm id : (2) No such file or directory 2023-10-05T19:28:56.171 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.162+0000 7f3daeb5fe40 20 rados->read ofs=0 len=0 2023-10-05T19:28:56.171 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.162+0000 7f3daeb5fe40 1 -- 172.21.15.167:0/1367369157 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:3 5.12 5:49953fa1:::default.realm:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32) v8 -- 0x563030d98a60 con 0x563030d647f0 2023-10-05T19:28:56.171 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.162+0000 7f3dace95700 1 -- 172.21.15.167:0/1367369157 <== osd.1 v2:172.21.15.167:6800/30035 3 ==== osd_op_reply(3 default.realm [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 ==== 157+0+0 (crc 0 0 0) 0x7f3da400a500 con 0x563030d647f0 2023-10-05T19:28:56.172 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.162+0000 7f3daeb5fe40 20 rados_obj.operate() r=-2 bl.length=0 2023-10-05T19:28:56.172 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.162+0000 7f3daeb5fe40 20 rados->read ofs=0 len=0 2023-10-05T19:28:56.172 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.162+0000 7f3daeb5fe40 1 -- 172.21.15.167:0/1367369157 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:4 5.f 5:f43b6ece:::zone_names.default:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32) v8 -- 0x563030d98670 con 0x563030d647f0 2023-10-05T19:28:56.172 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.166+0000 7f3dace95700 1 -- 172.21.15.167:0/1367369157 <== osd.1 v2:172.21.15.167:6800/30035 4 ==== osd_op_reply(4 zone_names.default [read 0~46 out=46b] v0'0 uv1 ondisk = 0) v8 ==== 162+0+46 (crc 0 0 0) 0x7f3da400a500 con 0x563030d647f0 2023-10-05T19:28:56.172 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.166+0000 7f3daeb5fe40 20 rados_obj.operate() r=0 bl.length=46 2023-10-05T19:28:56.172 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.166+0000 7f3daeb5fe40 20 rados->read ofs=0 len=0 2023-10-05T19:28:56.172 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.166+0000 7f3daeb5fe40 1 -- 172.21.15.167:0/1367369157 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:5 5.1d 5:b8374eb4:::zone_info.9a9df965-6612-45d4-a45f-6e3c889bacc2:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32) v8 -- 0x563030d998f0 con 0x563030d647f0 2023-10-05T19:28:56.172 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.166+0000 7f3dace95700 1 -- 172.21.15.167:0/1367369157 <== osd.1 v2:172.21.15.167:6800/30035 5 ==== osd_op_reply(5 zone_info.9a9df965-6612-45d4-a45f-6e3c889bacc2 [read 0~874 out=874b] v0'0 uv2 ondisk = 0) v8 ==== 190+0+874 (crc 0 0 0) 0x7f3da400a500 con 0x563030d647f0 2023-10-05T19:28:56.172 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.166+0000 7f3daeb5fe40 20 rados_obj.operate() r=0 bl.length=874 2023-10-05T19:28:56.173 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.166+0000 7f3daeb5fe40 20 searching for the correct realm 2023-10-05T19:28:56.173 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.166+0000 7f3daeb5fe40 1 --2- 172.21.15.167:0/1367369157 >> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] conn(0x563030d9b6b0 0x563030d9da90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-05T19:28:56.173 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.166+0000 7f3daeb5fe40 1 -- 172.21.15.167:0/1367369157 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:6 5.0 5:00000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x563030d9dfd0 con 0x563030d9b6b0 2023-10-05T19:28:56.173 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.166+0000 7f3dac694700 1 --2- 172.21.15.167:0/1367369157 >> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] conn(0x563030d9b6b0 0x563030d9da90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-05T19:28:56.174 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.166+0000 7f3dac694700 1 --2- 172.21.15.167:0/1367369157 >> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] conn(0x563030d9b6b0 0x563030d9da90 crc :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-05T19:28:56.174 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.166+0000 7f3dac694700 1 -- 172.21.15.167:0/1367369157 <== osd.3 v2:172.21.15.167:6808/30036 1 ==== osd_op_reply(6 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f3d94012450 con 0x563030d9b6b0 2023-10-05T19:28:56.174 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.166+0000 7f3dac694700 1 --2- 172.21.15.167:0/1367369157 >> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] conn(0x7f3d94007030 0x7f3d9401e3b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-05T19:28:56.174 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.166+0000 7f3dac694700 1 -- 172.21.15.167:0/1367369157 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:7 5.10 5:08000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x563030d9dfd0 con 0x7f3d94007030 2023-10-05T19:28:56.174 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.166+0000 7f3dabe93700 1 --2- 172.21.15.167:0/1367369157 >> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] conn(0x7f3d94007030 0x7f3d9401e3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-05T19:28:56.174 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.170+0000 7f3dabe93700 1 --2- 172.21.15.167:0/1367369157 >> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] conn(0x7f3d94007030 0x7f3d9401e3b0 crc :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-05T19:28:56.175 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.170+0000 7f3dabe93700 1 -- 172.21.15.167:0/1367369157 <== osd.2 v2:172.21.15.167:6816/30037 1 ==== osd_op_reply(7 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f3d9c003450 con 0x7f3d94007030 2023-10-05T19:28:56.175 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.170+0000 7f3dabe93700 1 --2- 172.21.15.167:0/1367369157 >> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] conn(0x7f3d9c011040 0x7f3d9c015960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-05T19:28:56.175 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.170+0000 7f3dabe93700 1 -- 172.21.15.167:0/1367369157 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:8 5.8 5:10000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x563030d9dfd0 con 0x7f3d9c011040 2023-10-05T19:28:56.175 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.170+0000 7f3dace95700 1 --2- 172.21.15.167:0/1367369157 >> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] conn(0x7f3d9c011040 0x7f3d9c015960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-05T19:28:56.176 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.170+0000 7f3dace95700 1 --2- 172.21.15.167:0/1367369157 >> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] conn(0x7f3d9c011040 0x7f3d9c015960 crc :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-05T19:28:56.176 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.170+0000 7f3dace95700 1 -- 172.21.15.167:0/1367369157 <== osd.0 v2:172.21.15.118:6800/27690 1 ==== osd_op_reply(8 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f3da4014040 con 0x7f3d9c011040 2023-10-05T19:28:56.176 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.170+0000 7f3dace95700 1 -- 172.21.15.167:0/1367369157 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:9 5.18 5:18000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x563030d9dfd0 con 0x563030d647f0 2023-10-05T19:28:56.177 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.170+0000 7f3dace95700 1 -- 172.21.15.167:0/1367369157 <== osd.1 v2:172.21.15.167:6800/30035 6 ==== osd_op_reply(9 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f3da4014040 con 0x563030d647f0 2023-10-05T19:28:56.177 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.170+0000 7f3dace95700 1 -- 172.21.15.167:0/1367369157 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:10 5.4 5:20000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x563030d9dfd0 con 0x563030d647f0 2023-10-05T19:28:56.178 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.174+0000 7f3dace95700 1 -- 172.21.15.167:0/1367369157 <== osd.1 v2:172.21.15.167:6800/30035 7 ==== osd_op_reply(10 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f3da4014040 con 0x563030d647f0 2023-10-05T19:28:56.178 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.174+0000 7f3dace95700 1 -- 172.21.15.167:0/1367369157 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:11 5.14 5:28000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x563030d9dfd0 con 0x7f3d9c011040 2023-10-05T19:28:56.179 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.174+0000 7f3dace95700 1 -- 172.21.15.167:0/1367369157 <== osd.0 v2:172.21.15.118:6800/27690 2 ==== osd_op_reply(11 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f3da4014040 con 0x7f3d9c011040 2023-10-05T19:28:56.179 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.174+0000 7f3dace95700 1 -- 172.21.15.167:0/1367369157 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:12 5.c 5:30000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x563030d9dfd0 con 0x7f3d9c011040 2023-10-05T19:28:56.180 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.174+0000 7f3dace95700 1 -- 172.21.15.167:0/1367369157 <== osd.0 v2:172.21.15.118:6800/27690 3 ==== osd_op_reply(12 [pgnls start_epoch 32 out=112b] v23'1 uv1 ondisk = 1) v8 ==== 144+0+112 (crc 0 0 0) 0x7f3da4014040 con 0x7f3d9c011040 2023-10-05T19:28:56.180 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.174+0000 7f3dace95700 1 -- 172.21.15.167:0/1367369157 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:13 5.1c 5:38000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x563030d9dfd0 con 0x7f3d94007030 2023-10-05T19:28:56.181 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.174+0000 7f3dabe93700 1 -- 172.21.15.167:0/1367369157 <== osd.2 v2:172.21.15.167:6816/30037 2 ==== osd_op_reply(13 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f3d9c003450 con 0x7f3d94007030 2023-10-05T19:28:56.181 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.174+0000 7f3dabe93700 1 -- 172.21.15.167:0/1367369157 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:14 5.2 5:40000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x563030d9dfd0 con 0x563030d9b6b0 2023-10-05T19:28:56.181 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.174+0000 7f3dac694700 1 -- 172.21.15.167:0/1367369157 <== osd.3 v2:172.21.15.167:6808/30036 2 ==== osd_op_reply(14 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f3d94012450 con 0x563030d9b6b0 2023-10-05T19:28:56.182 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.174+0000 7f3dac694700 1 -- 172.21.15.167:0/1367369157 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:15 5.12 5:48000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x563030d9dfd0 con 0x563030d647f0 2023-10-05T19:28:56.182 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.178+0000 7f3dace95700 1 -- 172.21.15.167:0/1367369157 <== osd.1 v2:172.21.15.167:6800/30035 8 ==== osd_op_reply(15 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f3da4014040 con 0x563030d647f0 2023-10-05T19:28:56.182 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.178+0000 7f3dace95700 1 -- 172.21.15.167:0/1367369157 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:16 5.a 5:50000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x563030d9dfd0 con 0x7f3d9c011040 2023-10-05T19:28:56.183 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.178+0000 7f3dace95700 1 -- 172.21.15.167:0/1367369157 <== osd.0 v2:172.21.15.118:6800/27690 4 ==== osd_op_reply(16 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f3da4022040 con 0x7f3d9c011040 2023-10-05T19:28:56.183 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.178+0000 7f3dace95700 1 -- 172.21.15.167:0/1367369157 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:17 5.1a 5:58000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x563030d9dfd0 con 0x563030d647f0 2023-10-05T19:28:56.184 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.178+0000 7f3dace95700 1 -- 172.21.15.167:0/1367369157 <== osd.1 v2:172.21.15.167:6800/30035 9 ==== osd_op_reply(17 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f3da4022040 con 0x563030d647f0 2023-10-05T19:28:56.184 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.178+0000 7f3dace95700 1 -- 172.21.15.167:0/1367369157 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:18 5.6 5:60000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x563030d9dfd0 con 0x7f3d94007030 2023-10-05T19:28:56.184 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.178+0000 7f3dabe93700 1 -- 172.21.15.167:0/1367369157 <== osd.2 v2:172.21.15.167:6816/30037 3 ==== osd_op_reply(18 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f3d9c003450 con 0x7f3d94007030 2023-10-05T19:28:56.184 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.178+0000 7f3dabe93700 1 -- 172.21.15.167:0/1367369157 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:19 5.16 5:68000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x563030d9dfd0 con 0x563030d9b6b0 2023-10-05T19:28:56.185 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.178+0000 7f3dac694700 1 -- 172.21.15.167:0/1367369157 <== osd.3 v2:172.21.15.167:6808/30036 3 ==== osd_op_reply(19 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f3d94012450 con 0x563030d9b6b0 2023-10-05T19:28:56.185 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.178+0000 7f3dac694700 1 -- 172.21.15.167:0/1367369157 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:20 5.e 5:70000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x563030d9dfd0 con 0x563030d9b6b0 2023-10-05T19:28:56.186 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.182+0000 7f3dac694700 1 -- 172.21.15.167:0/1367369157 <== osd.3 v2:172.21.15.167:6808/30036 4 ==== osd_op_reply(20 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f3d94012450 con 0x563030d9b6b0 2023-10-05T19:28:56.186 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.182+0000 7f3dac694700 1 -- 172.21.15.167:0/1367369157 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:21 5.1e 5:78000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x563030d9dfd0 con 0x7f3d9c011040 2023-10-05T19:28:56.187 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.182+0000 7f3dace95700 1 -- 172.21.15.167:0/1367369157 <== osd.0 v2:172.21.15.118:6800/27690 5 ==== osd_op_reply(21 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f3da4022040 con 0x7f3d9c011040 2023-10-05T19:28:56.187 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.182+0000 7f3dace95700 1 -- 172.21.15.167:0/1367369157 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:22 5.1 5:80000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x563030d9dfd0 con 0x563030d9b6b0 2023-10-05T19:28:56.188 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.182+0000 7f3dac694700 1 -- 172.21.15.167:0/1367369157 <== osd.3 v2:172.21.15.167:6808/30036 5 ==== osd_op_reply(22 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f3d94012450 con 0x563030d9b6b0 2023-10-05T19:28:56.188 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.182+0000 7f3dac694700 1 -- 172.21.15.167:0/1367369157 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:23 5.11 5:88000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x563030d9dfd0 con 0x563030d9b6b0 2023-10-05T19:28:56.188 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.182+0000 7f3dac694700 1 -- 172.21.15.167:0/1367369157 <== osd.3 v2:172.21.15.167:6808/30036 6 ==== osd_op_reply(23 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f3d94012450 con 0x563030d9b6b0 2023-10-05T19:28:56.188 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.182+0000 7f3dac694700 1 -- 172.21.15.167:0/1367369157 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:24 5.9 5:90000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x563030d9dfd0 con 0x563030d647f0 2023-10-05T19:28:56.189 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.182+0000 7f3dace95700 1 -- 172.21.15.167:0/1367369157 <== osd.1 v2:172.21.15.167:6800/30035 10 ==== osd_op_reply(24 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f3da4022040 con 0x563030d647f0 2023-10-05T19:28:56.189 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.182+0000 7f3dace95700 1 -- 172.21.15.167:0/1367369157 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:25 5.19 5:98000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x563030d9dfd0 con 0x563030d647f0 2023-10-05T19:28:56.190 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.186+0000 7f3dace95700 1 -- 172.21.15.167:0/1367369157 <== osd.1 v2:172.21.15.167:6800/30035 11 ==== osd_op_reply(25 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f3da4022040 con 0x563030d647f0 2023-10-05T19:28:56.190 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.186+0000 7f3dace95700 1 -- 172.21.15.167:0/1367369157 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:26 5.5 5:a0000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x563030d9dfd0 con 0x7f3d9c011040 2023-10-05T19:28:56.191 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.186+0000 7f3dace95700 1 -- 172.21.15.167:0/1367369157 <== osd.0 v2:172.21.15.118:6800/27690 6 ==== osd_op_reply(26 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f3da4022040 con 0x7f3d9c011040 2023-10-05T19:28:56.191 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.186+0000 7f3dace95700 1 -- 172.21.15.167:0/1367369157 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:27 5.15 5:a8000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x563030d9dfd0 con 0x563030d9b6b0 2023-10-05T19:28:56.192 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.186+0000 7f3dac694700 1 -- 172.21.15.167:0/1367369157 <== osd.3 v2:172.21.15.167:6808/30036 7 ==== osd_op_reply(27 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f3d94012450 con 0x563030d9b6b0 2023-10-05T19:28:56.192 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.186+0000 7f3dac694700 1 -- 172.21.15.167:0/1367369157 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:28 5.d 5:b0000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x563030d9dfd0 con 0x7f3d94007030 2023-10-05T19:28:56.192 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.186+0000 7f3dabe93700 1 -- 172.21.15.167:0/1367369157 <== osd.2 v2:172.21.15.167:6816/30037 4 ==== osd_op_reply(28 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f3d9c003450 con 0x7f3d94007030 2023-10-05T19:28:56.192 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.186+0000 7f3dabe93700 1 -- 172.21.15.167:0/1367369157 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:29 5.1d 5:b8000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x563030d9dfd0 con 0x563030d647f0 2023-10-05T19:28:56.193 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.186+0000 7f3dace95700 1 -- 172.21.15.167:0/1367369157 <== osd.1 v2:172.21.15.167:6800/30035 12 ==== osd_op_reply(29 [pgnls start_epoch 32 out=107b] v30'2 uv2 ondisk = 1) v8 ==== 144+0+107 (crc 0 0 0) 0x7f3da4022040 con 0x563030d647f0 2023-10-05T19:28:56.193 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.186+0000 7f3dace95700 1 -- 172.21.15.167:0/1367369157 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:30 5.3 5:c0000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x563030d9dfd0 con 0x7f3d9c011040 2023-10-05T19:28:56.194 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.190+0000 7f3dace95700 1 -- 172.21.15.167:0/1367369157 <== osd.0 v2:172.21.15.118:6800/27690 7 ==== osd_op_reply(30 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f3da4022040 con 0x7f3d9c011040 2023-10-05T19:28:56.194 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.190+0000 7f3dace95700 1 -- 172.21.15.167:0/1367369157 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:31 5.13 5:c8000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x563030d9dfd0 con 0x563030d9b6b0 2023-10-05T19:28:56.195 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.190+0000 7f3dac694700 1 -- 172.21.15.167:0/1367369157 <== osd.3 v2:172.21.15.167:6808/30036 8 ==== osd_op_reply(31 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f3d94012450 con 0x563030d9b6b0 2023-10-05T19:28:56.195 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.190+0000 7f3dac694700 1 -- 172.21.15.167:0/1367369157 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:32 5.b 5:d0000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x563030d9dfd0 con 0x7f3d9c011040 2023-10-05T19:28:56.196 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.190+0000 7f3dace95700 1 -- 172.21.15.167:0/1367369157 <== osd.0 v2:172.21.15.118:6800/27690 8 ==== osd_op_reply(32 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f3da4022040 con 0x7f3d9c011040 2023-10-05T19:28:56.196 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.190+0000 7f3dace95700 1 -- 172.21.15.167:0/1367369157 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:33 5.1b 5:d8000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x563030d9dfd0 con 0x7f3d94007030 2023-10-05T19:28:56.197 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.190+0000 7f3dabe93700 1 -- 172.21.15.167:0/1367369157 <== osd.2 v2:172.21.15.167:6816/30037 5 ==== osd_op_reply(33 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f3d9c003450 con 0x7f3d94007030 2023-10-05T19:28:56.197 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.190+0000 7f3dabe93700 1 -- 172.21.15.167:0/1367369157 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:34 5.7 5:e0000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x563030d9dfd0 con 0x563030d9b6b0 2023-10-05T19:28:56.198 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.194+0000 7f3dac694700 1 -- 172.21.15.167:0/1367369157 <== osd.3 v2:172.21.15.167:6808/30036 9 ==== osd_op_reply(34 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f3d94012450 con 0x563030d9b6b0 2023-10-05T19:28:56.198 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.194+0000 7f3dac694700 1 -- 172.21.15.167:0/1367369157 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:35 5.17 5:e8000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x563030d9dfd0 con 0x7f3d9c011040 2023-10-05T19:28:56.203 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.194+0000 7f3dace95700 1 -- 172.21.15.167:0/1367369157 <== osd.0 v2:172.21.15.118:6800/27690 9 ==== osd_op_reply(35 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f3da4022040 con 0x7f3d9c011040 2023-10-05T19:28:56.203 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.194+0000 7f3dace95700 1 -- 172.21.15.167:0/1367369157 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:36 5.f 5:f0000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x563030d9dfd0 con 0x563030d647f0 2023-10-05T19:28:56.203 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.194+0000 7f3dace95700 1 -- 172.21.15.167:0/1367369157 <== osd.1 v2:172.21.15.167:6800/30035 13 ==== osd_op_reply(36 [pgnls start_epoch 32 out=115b] v23'2 uv2 ondisk = 1) v8 ==== 144+0+115 (crc 0 0 0) 0x7f3da402f080 con 0x563030d647f0 2023-10-05T19:28:56.203 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.194+0000 7f3dace95700 1 -- 172.21.15.167:0/1367369157 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:37 5.1f 5:f8000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x563030d9dfd0 con 0x7f3d94007030 2023-10-05T19:28:56.204 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.194+0000 7f3dabe93700 1 -- 172.21.15.167:0/1367369157 <== osd.2 v2:172.21.15.167:6816/30037 6 ==== osd_op_reply(37 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f3d9c003450 con 0x7f3d94007030 2023-10-05T19:28:56.204 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.194+0000 7f3daeb5fe40 20 RGWRados::pool_iterate: got zonegroup_info.4f1f719d-9648-417b-b10c-e5d939825419 2023-10-05T19:28:56.204 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.194+0000 7f3daeb5fe40 20 RGWRados::pool_iterate: got zone_info.9a9df965-6612-45d4-a45f-6e3c889bacc2 2023-10-05T19:28:56.204 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.194+0000 7f3daeb5fe40 20 RGWRados::pool_iterate: got zone_names.default 2023-10-05T19:28:56.204 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.194+0000 7f3daeb5fe40 20 RGWRados::pool_iterate: got zonegroups_names.default 2023-10-05T19:28:56.204 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.194+0000 7f3daeb5fe40 20 rados->read ofs=0 len=0 2023-10-05T19:28:56.204 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.194+0000 7f3daeb5fe40 1 -- 172.21.15.167:0/1367369157 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:38 5.12 5:49953fa1:::default.realm:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32) v8 -- 0x563030d9a460 con 0x563030d647f0 2023-10-05T19:28:56.204 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.194+0000 7f3dace95700 1 -- 172.21.15.167:0/1367369157 <== osd.1 v2:172.21.15.167:6800/30035 14 ==== osd_op_reply(38 default.realm [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 ==== 157+0+0 (crc 0 0 0) 0x7f3da402f080 con 0x563030d647f0 2023-10-05T19:28:56.204 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.194+0000 7f3daeb5fe40 20 rados_obj.operate() r=-2 bl.length=0 2023-10-05T19:28:56.204 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.194+0000 7f3daeb5fe40 20 rados->read ofs=0 len=0 2023-10-05T19:28:56.204 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.194+0000 7f3daeb5fe40 1 -- 172.21.15.167:0/1367369157 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:39 5.f 5:f4c53578:::zonegroups_names.default:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32) v8 -- 0x563030d9ad50 con 0x563030d647f0 2023-10-05T19:28:56.205 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.198+0000 7f3dace95700 1 -- 172.21.15.167:0/1367369157 <== osd.1 v2:172.21.15.167:6800/30035 15 ==== osd_op_reply(39 zonegroups_names.default [read 0~46 out=46b] v0'0 uv2 ondisk = 0) v8 ==== 168+0+46 (crc 0 0 0) 0x7f3da402f080 con 0x563030d647f0 2023-10-05T19:28:56.205 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.198+0000 7f3daeb5fe40 20 rados_obj.operate() r=0 bl.length=46 2023-10-05T19:28:56.205 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.198+0000 7f3daeb5fe40 20 rados->read ofs=0 len=0 2023-10-05T19:28:56.205 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.198+0000 7f3daeb5fe40 1 -- 172.21.15.167:0/1367369157 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:40 5.c 5:30d299a5:::zonegroup_info.4f1f719d-9648-417b-b10c-e5d939825419:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32) v8 -- 0x563030d9e410 con 0x7f3d9c011040 2023-10-05T19:28:56.205 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.198+0000 7f3dace95700 1 -- 172.21.15.167:0/1367369157 <== osd.0 v2:172.21.15.118:6800/27690 10 ==== osd_op_reply(40 zonegroup_info.4f1f719d-9648-417b-b10c-e5d939825419 [read 0~362 out=362b] v0'0 uv1 ondisk = 0) v8 ==== 195+0+362 (crc 0 0 0) 0x7f3da402f080 con 0x7f3d9c011040 2023-10-05T19:28:56.205 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.198+0000 7f3daeb5fe40 20 rados_obj.operate() r=0 bl.length=362 2023-10-05T19:28:56.205 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.198+0000 7f3daeb5fe40 20 rados->read ofs=0 len=0 2023-10-05T19:28:56.205 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.198+0000 7f3daeb5fe40 1 -- 172.21.15.167:0/1367369157 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:41 5.13 5:cbbf6c60:::converted:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32) v8 -- 0x563030d9e410 con 0x563030d9b6b0 2023-10-05T19:28:56.207 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.198+0000 7f3dac694700 1 -- 172.21.15.167:0/1367369157 <== osd.3 v2:172.21.15.167:6808/30036 10 ==== osd_op_reply(41 converted [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 ==== 153+0+0 (crc 0 0 0) 0x7f3d94012450 con 0x563030d9b6b0 2023-10-05T19:28:56.207 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.198+0000 7f3daeb5fe40 20 rados_obj.operate() r=-2 bl.length=0 2023-10-05T19:28:56.207 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.198+0000 7f3daeb5fe40 20 rados->read ofs=0 len=0 2023-10-05T19:28:56.207 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.198+0000 7f3daeb5fe40 1 -- 172.21.15.167:0/1367369157 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:42 5.12 5:49953fa1:::default.realm:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32) v8 -- 0x563030d9e7c0 con 0x563030d647f0 2023-10-05T19:28:56.207 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.198+0000 7f3dace95700 1 -- 172.21.15.167:0/1367369157 <== osd.1 v2:172.21.15.167:6800/30035 16 ==== osd_op_reply(42 default.realm [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 ==== 157+0+0 (crc 0 0 0) 0x7f3da402f080 con 0x563030d647f0 2023-10-05T19:28:56.207 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.198+0000 7f3daeb5fe40 20 rados_obj.operate() r=-2 bl.length=0 2023-10-05T19:28:56.207 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.198+0000 7f3daeb5fe40 20 rados->read ofs=0 len=0 2023-10-05T19:28:56.207 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.198+0000 7f3daeb5fe40 1 -- 172.21.15.167:0/1367369157 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:43 5.f 5:f4c53578:::zonegroups_names.default:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32) v8 -- 0x563030d9ebc0 con 0x563030d647f0 2023-10-05T19:28:56.207 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.198+0000 7f3dace95700 1 -- 172.21.15.167:0/1367369157 <== osd.1 v2:172.21.15.167:6800/30035 17 ==== osd_op_reply(43 zonegroups_names.default [read 0~46 out=46b] v0'0 uv2 ondisk = 0) v8 ==== 168+0+46 (crc 0 0 0) 0x7f3da402f080 con 0x563030d647f0 2023-10-05T19:28:56.208 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.198+0000 7f3daeb5fe40 20 rados_obj.operate() r=0 bl.length=46 2023-10-05T19:28:56.208 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.198+0000 7f3daeb5fe40 1 -- 172.21.15.167:0/1367369157 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:44 5.0 5:00000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x563030d9fbd0 con 0x563030d9b6b0 2023-10-05T19:28:56.208 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.202+0000 7f3dac694700 1 -- 172.21.15.167:0/1367369157 <== osd.3 v2:172.21.15.167:6808/30036 11 ==== osd_op_reply(44 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f3d94012450 con 0x563030d9b6b0 2023-10-05T19:28:56.208 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.202+0000 7f3dac694700 1 -- 172.21.15.167:0/1367369157 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:45 5.10 5:08000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x563030d9fbd0 con 0x7f3d94007030 2023-10-05T19:28:56.209 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.202+0000 7f3dabe93700 1 -- 172.21.15.167:0/1367369157 <== osd.2 v2:172.21.15.167:6816/30037 7 ==== osd_op_reply(45 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f3d9c003450 con 0x7f3d94007030 2023-10-05T19:28:56.209 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.202+0000 7f3dabe93700 1 -- 172.21.15.167:0/1367369157 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:46 5.8 5:10000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x563030d9fbd0 con 0x7f3d9c011040 2023-10-05T19:28:56.209 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.202+0000 7f3dace95700 1 -- 172.21.15.167:0/1367369157 <== osd.0 v2:172.21.15.118:6800/27690 11 ==== osd_op_reply(46 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f3da402f080 con 0x7f3d9c011040 2023-10-05T19:28:56.209 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.202+0000 7f3dace95700 1 -- 172.21.15.167:0/1367369157 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:47 5.18 5:18000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x563030d9ad50 con 0x563030d647f0 2023-10-05T19:28:56.210 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.202+0000 7f3dace95700 1 -- 172.21.15.167:0/1367369157 <== osd.1 v2:172.21.15.167:6800/30035 18 ==== osd_op_reply(47 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f3da402f080 con 0x563030d647f0 2023-10-05T19:28:56.210 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.202+0000 7f3dace95700 1 -- 172.21.15.167:0/1367369157 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:48 5.4 5:20000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x563030d9ad50 con 0x563030d647f0 2023-10-05T19:28:56.210 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.206+0000 7f3dace95700 1 -- 172.21.15.167:0/1367369157 <== osd.1 v2:172.21.15.167:6800/30035 19 ==== osd_op_reply(48 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f3da402f080 con 0x563030d647f0 2023-10-05T19:28:56.210 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.206+0000 7f3dace95700 1 -- 172.21.15.167:0/1367369157 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:49 5.14 5:28000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x563030d9ad50 con 0x7f3d9c011040 2023-10-05T19:28:56.211 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.206+0000 7f3dace95700 1 -- 172.21.15.167:0/1367369157 <== osd.0 v2:172.21.15.118:6800/27690 12 ==== osd_op_reply(49 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f3da402f080 con 0x7f3d9c011040 2023-10-05T19:28:56.211 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.206+0000 7f3dace95700 1 -- 172.21.15.167:0/1367369157 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:50 5.c 5:30000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x563030d9ad50 con 0x7f3d9c011040 2023-10-05T19:28:56.212 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.206+0000 7f3dace95700 1 -- 172.21.15.167:0/1367369157 <== osd.0 v2:172.21.15.118:6800/27690 13 ==== osd_op_reply(50 [pgnls start_epoch 32 out=112b] v23'1 uv1 ondisk = 1) v8 ==== 144+0+112 (crc 0 0 0) 0x7f3da402f080 con 0x7f3d9c011040 2023-10-05T19:28:56.212 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.206+0000 7f3dace95700 1 -- 172.21.15.167:0/1367369157 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:51 5.1c 5:38000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x563030d9ad50 con 0x7f3d94007030 2023-10-05T19:28:56.213 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.206+0000 7f3dabe93700 1 -- 172.21.15.167:0/1367369157 <== osd.2 v2:172.21.15.167:6816/30037 8 ==== osd_op_reply(51 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f3d9c003450 con 0x7f3d94007030 2023-10-05T19:28:56.213 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.206+0000 7f3dabe93700 1 -- 172.21.15.167:0/1367369157 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:52 5.2 5:40000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x563030d9ad50 con 0x563030d9b6b0 2023-10-05T19:28:56.214 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.210+0000 7f3dac694700 1 -- 172.21.15.167:0/1367369157 <== osd.3 v2:172.21.15.167:6808/30036 12 ==== osd_op_reply(52 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f3d94012450 con 0x563030d9b6b0 2023-10-05T19:28:56.214 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.210+0000 7f3dac694700 1 -- 172.21.15.167:0/1367369157 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:53 5.12 5:48000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x563030d9ad50 con 0x563030d647f0 2023-10-05T19:28:56.215 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.210+0000 7f3dace95700 1 -- 172.21.15.167:0/1367369157 <== osd.1 v2:172.21.15.167:6800/30035 20 ==== osd_op_reply(53 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f3da402f080 con 0x563030d647f0 2023-10-05T19:28:56.215 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.210+0000 7f3dace95700 1 -- 172.21.15.167:0/1367369157 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:54 5.a 5:50000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x563030d9ad50 con 0x7f3d9c011040 2023-10-05T19:28:56.216 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.210+0000 7f3dace95700 1 -- 172.21.15.167:0/1367369157 <== osd.0 v2:172.21.15.118:6800/27690 14 ==== osd_op_reply(54 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f3da402f080 con 0x7f3d9c011040 2023-10-05T19:28:56.216 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.210+0000 7f3dace95700 1 -- 172.21.15.167:0/1367369157 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:55 5.1a 5:58000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x563030d9ad50 con 0x563030d647f0 2023-10-05T19:28:56.217 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.210+0000 7f3dace95700 1 -- 172.21.15.167:0/1367369157 <== osd.1 v2:172.21.15.167:6800/30035 21 ==== osd_op_reply(55 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f3da402f080 con 0x563030d647f0 2023-10-05T19:28:56.217 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.210+0000 7f3dace95700 1 -- 172.21.15.167:0/1367369157 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:56 5.6 5:60000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x563030d9ad50 con 0x7f3d94007030 2023-10-05T19:28:56.218 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.214+0000 7f3dabe93700 1 -- 172.21.15.167:0/1367369157 <== osd.2 v2:172.21.15.167:6816/30037 9 ==== osd_op_reply(56 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f3d9c003450 con 0x7f3d94007030 2023-10-05T19:28:56.218 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.214+0000 7f3dabe93700 1 -- 172.21.15.167:0/1367369157 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:57 5.16 5:68000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x563030d9ad50 con 0x563030d9b6b0 2023-10-05T19:28:56.218 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.214+0000 7f3dac694700 1 -- 172.21.15.167:0/1367369157 <== osd.3 v2:172.21.15.167:6808/30036 13 ==== osd_op_reply(57 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f3d94012450 con 0x563030d9b6b0 2023-10-05T19:28:56.219 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.214+0000 7f3dac694700 1 -- 172.21.15.167:0/1367369157 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:58 5.e 5:70000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x563030d9ad50 con 0x563030d9b6b0 2023-10-05T19:28:56.219 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.214+0000 7f3dac694700 1 -- 172.21.15.167:0/1367369157 <== osd.3 v2:172.21.15.167:6808/30036 14 ==== osd_op_reply(58 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f3d94012450 con 0x563030d9b6b0 2023-10-05T19:28:56.219 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.214+0000 7f3dac694700 1 -- 172.21.15.167:0/1367369157 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:59 5.1e 5:78000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x563030d9ad50 con 0x7f3d9c011040 2023-10-05T19:28:56.220 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.214+0000 7f3dace95700 1 -- 172.21.15.167:0/1367369157 <== osd.0 v2:172.21.15.118:6800/27690 15 ==== osd_op_reply(59 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f3da402f080 con 0x7f3d9c011040 2023-10-05T19:28:56.220 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.214+0000 7f3dace95700 1 -- 172.21.15.167:0/1367369157 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:60 5.1 5:80000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x563030d9ad50 con 0x563030d9b6b0 2023-10-05T19:28:56.221 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.214+0000 7f3dac694700 1 -- 172.21.15.167:0/1367369157 <== osd.3 v2:172.21.15.167:6808/30036 15 ==== osd_op_reply(60 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f3d94012450 con 0x563030d9b6b0 2023-10-05T19:28:56.221 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.214+0000 7f3dac694700 1 -- 172.21.15.167:0/1367369157 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:61 5.11 5:88000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x563030d9ad50 con 0x563030d9b6b0 2023-10-05T19:28:56.222 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.218+0000 7f3dac694700 1 -- 172.21.15.167:0/1367369157 <== osd.3 v2:172.21.15.167:6808/30036 16 ==== osd_op_reply(61 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f3d94012450 con 0x563030d9b6b0 2023-10-05T19:28:56.222 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.218+0000 7f3dac694700 1 -- 172.21.15.167:0/1367369157 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:62 5.9 5:90000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x563030d9ad50 con 0x563030d647f0 2023-10-05T19:28:56.223 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.218+0000 7f3dace95700 1 -- 172.21.15.167:0/1367369157 <== osd.1 v2:172.21.15.167:6800/30035 22 ==== osd_op_reply(62 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f3da402f080 con 0x563030d647f0 2023-10-05T19:28:56.223 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.218+0000 7f3dace95700 1 -- 172.21.15.167:0/1367369157 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:63 5.19 5:98000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x563030d9ad50 con 0x563030d647f0 2023-10-05T19:28:56.224 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.218+0000 7f3dace95700 1 -- 172.21.15.167:0/1367369157 <== osd.1 v2:172.21.15.167:6800/30035 23 ==== osd_op_reply(63 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f3da402f080 con 0x563030d647f0 2023-10-05T19:28:56.224 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.218+0000 7f3dace95700 1 -- 172.21.15.167:0/1367369157 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:64 5.5 5:a0000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x563030d9ad50 con 0x7f3d9c011040 2023-10-05T19:28:56.225 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.218+0000 7f3dace95700 1 -- 172.21.15.167:0/1367369157 <== osd.0 v2:172.21.15.118:6800/27690 16 ==== osd_op_reply(64 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f3da402f080 con 0x7f3d9c011040 2023-10-05T19:28:56.225 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.218+0000 7f3dace95700 1 -- 172.21.15.167:0/1367369157 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:65 5.15 5:a8000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x563030d9ad50 con 0x563030d9b6b0 2023-10-05T19:28:56.226 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.222+0000 7f3dac694700 1 -- 172.21.15.167:0/1367369157 <== osd.3 v2:172.21.15.167:6808/30036 17 ==== osd_op_reply(65 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f3d94012450 con 0x563030d9b6b0 2023-10-05T19:28:56.226 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.222+0000 7f3dac694700 1 -- 172.21.15.167:0/1367369157 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:66 5.d 5:b0000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x563030d9ad50 con 0x7f3d94007030 2023-10-05T19:28:56.227 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.222+0000 7f3dabe93700 1 -- 172.21.15.167:0/1367369157 <== osd.2 v2:172.21.15.167:6816/30037 10 ==== osd_op_reply(66 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f3d9c003450 con 0x7f3d94007030 2023-10-05T19:28:56.227 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.222+0000 7f3dabe93700 1 -- 172.21.15.167:0/1367369157 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:67 5.1d 5:b8000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x563030d9ad50 con 0x563030d647f0 2023-10-05T19:28:56.227 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.222+0000 7f3dace95700 1 -- 172.21.15.167:0/1367369157 <== osd.1 v2:172.21.15.167:6800/30035 24 ==== osd_op_reply(67 [pgnls start_epoch 32 out=107b] v30'2 uv2 ondisk = 1) v8 ==== 144+0+107 (crc 0 0 0) 0x7f3da402f080 con 0x563030d647f0 2023-10-05T19:28:56.228 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.222+0000 7f3dace95700 1 -- 172.21.15.167:0/1367369157 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:68 5.3 5:c0000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x563030d9ad50 con 0x7f3d9c011040 2023-10-05T19:28:56.228 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.222+0000 7f3dace95700 1 -- 172.21.15.167:0/1367369157 <== osd.0 v2:172.21.15.118:6800/27690 17 ==== osd_op_reply(68 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f3da402f080 con 0x7f3d9c011040 2023-10-05T19:28:56.229 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.222+0000 7f3dace95700 1 -- 172.21.15.167:0/1367369157 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:69 5.13 5:c8000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x563030d9ad50 con 0x563030d9b6b0 2023-10-05T19:28:56.229 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.222+0000 7f3dac694700 1 -- 172.21.15.167:0/1367369157 <== osd.3 v2:172.21.15.167:6808/30036 18 ==== osd_op_reply(69 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f3d94012450 con 0x563030d9b6b0 2023-10-05T19:28:56.229 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.226+0000 7f3dac694700 1 -- 172.21.15.167:0/1367369157 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:70 5.b 5:d0000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x563030d9ad50 con 0x7f3d9c011040 2023-10-05T19:28:56.230 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.226+0000 7f3dace95700 1 -- 172.21.15.167:0/1367369157 <== osd.0 v2:172.21.15.118:6800/27690 18 ==== osd_op_reply(70 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f3da402f080 con 0x7f3d9c011040 2023-10-05T19:28:56.230 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.226+0000 7f3dace95700 1 -- 172.21.15.167:0/1367369157 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:71 5.1b 5:d8000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x563030d9ad50 con 0x7f3d94007030 2023-10-05T19:28:56.231 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.226+0000 7f3dabe93700 1 -- 172.21.15.167:0/1367369157 <== osd.2 v2:172.21.15.167:6816/30037 11 ==== osd_op_reply(71 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f3d9c003450 con 0x7f3d94007030 2023-10-05T19:28:56.231 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.226+0000 7f3dabe93700 1 -- 172.21.15.167:0/1367369157 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:72 5.7 5:e0000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x563030d9ad50 con 0x563030d9b6b0 2023-10-05T19:28:56.232 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.226+0000 7f3dac694700 1 -- 172.21.15.167:0/1367369157 <== osd.3 v2:172.21.15.167:6808/30036 19 ==== osd_op_reply(72 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f3d94012450 con 0x563030d9b6b0 2023-10-05T19:28:56.232 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.226+0000 7f3dac694700 1 -- 172.21.15.167:0/1367369157 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:73 5.17 5:e8000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x563030d9ad50 con 0x7f3d9c011040 2023-10-05T19:28:56.233 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.226+0000 7f3dace95700 1 -- 172.21.15.167:0/1367369157 <== osd.0 v2:172.21.15.118:6800/27690 19 ==== osd_op_reply(73 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f3da402f080 con 0x7f3d9c011040 2023-10-05T19:28:56.233 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.226+0000 7f3dace95700 1 -- 172.21.15.167:0/1367369157 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:74 5.f 5:f0000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x563030d9ad50 con 0x563030d647f0 2023-10-05T19:28:56.234 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.230+0000 7f3dace95700 1 -- 172.21.15.167:0/1367369157 <== osd.1 v2:172.21.15.167:6800/30035 25 ==== osd_op_reply(74 [pgnls start_epoch 32 out=115b] v23'2 uv2 ondisk = 1) v8 ==== 144+0+115 (crc 0 0 0) 0x7f3da402f080 con 0x563030d647f0 2023-10-05T19:28:56.234 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.230+0000 7f3dace95700 1 -- 172.21.15.167:0/1367369157 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:75 5.1f 5:f8000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x563030d9ad50 con 0x7f3d94007030 2023-10-05T19:28:56.235 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.230+0000 7f3dabe93700 1 -- 172.21.15.167:0/1367369157 <== osd.2 v2:172.21.15.167:6816/30037 12 ==== osd_op_reply(75 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f3d9c003450 con 0x7f3d94007030 2023-10-05T19:28:56.235 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.230+0000 7f3daeb5fe40 20 RGWRados::pool_iterate: got zonegroup_info.4f1f719d-9648-417b-b10c-e5d939825419 2023-10-05T19:28:56.235 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.230+0000 7f3daeb5fe40 20 RGWRados::pool_iterate: got zone_info.9a9df965-6612-45d4-a45f-6e3c889bacc2 2023-10-05T19:28:56.235 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.230+0000 7f3daeb5fe40 20 RGWRados::pool_iterate: got zone_names.default 2023-10-05T19:28:56.235 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.230+0000 7f3daeb5fe40 20 RGWRados::pool_iterate: got zonegroups_names.default 2023-10-05T19:28:56.235 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.230+0000 7f3daeb5fe40 20 rados->read ofs=0 len=0 2023-10-05T19:28:56.235 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.230+0000 7f3daeb5fe40 1 -- 172.21.15.167:0/1367369157 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:76 5.f 5:f43b6ece:::zone_names.default:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32) v8 -- 0x563030d9ebc0 con 0x563030d647f0 2023-10-05T19:28:56.237 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.230+0000 7f3dace95700 1 -- 172.21.15.167:0/1367369157 <== osd.1 v2:172.21.15.167:6800/30035 26 ==== osd_op_reply(76 zone_names.default [read 0~46 out=46b] v0'0 uv1 ondisk = 0) v8 ==== 162+0+46 (crc 0 0 0) 0x7f3da402f080 con 0x563030d647f0 2023-10-05T19:28:56.237 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.230+0000 7f3daeb5fe40 20 rados_obj.operate() r=0 bl.length=46 2023-10-05T19:28:56.237 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.230+0000 7f3daeb5fe40 20 rados->read ofs=0 len=0 2023-10-05T19:28:56.237 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.230+0000 7f3daeb5fe40 1 -- 172.21.15.167:0/1367369157 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:77 5.1d 5:b8374eb4:::zone_info.9a9df965-6612-45d4-a45f-6e3c889bacc2:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32) v8 -- 0x563030d9ef70 con 0x563030d647f0 2023-10-05T19:28:56.237 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.230+0000 7f3dace95700 1 -- 172.21.15.167:0/1367369157 <== osd.1 v2:172.21.15.167:6800/30035 27 ==== osd_op_reply(77 zone_info.9a9df965-6612-45d4-a45f-6e3c889bacc2 [read 0~874 out=874b] v0'0 uv2 ondisk = 0) v8 ==== 190+0+874 (crc 0 0 0) 0x7f3da402f080 con 0x563030d647f0 2023-10-05T19:28:56.237 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.230+0000 7f3daeb5fe40 20 rados_obj.operate() r=0 bl.length=874 2023-10-05T19:28:56.237 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.230+0000 7f3daeb5fe40 20 rados->read ofs=0 len=0 2023-10-05T19:28:56.238 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.230+0000 7f3daeb5fe40 1 -- 172.21.15.167:0/1367369157 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:78 5.f 5:f4c53578:::zonegroups_names.default:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32) v8 -- 0x563030d9a8c0 con 0x563030d647f0 2023-10-05T19:28:56.238 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.230+0000 7f3dace95700 1 -- 172.21.15.167:0/1367369157 <== osd.1 v2:172.21.15.167:6800/30035 28 ==== osd_op_reply(78 zonegroups_names.default [read 0~46 out=46b] v0'0 uv2 ondisk = 0) v8 ==== 168+0+46 (crc 0 0 0) 0x7f3da402f080 con 0x563030d647f0 2023-10-05T19:28:56.238 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.230+0000 7f3daeb5fe40 20 rados_obj.operate() r=0 bl.length=46 2023-10-05T19:28:56.238 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.230+0000 7f3daeb5fe40 20 rados->read ofs=0 len=0 2023-10-05T19:28:56.238 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.230+0000 7f3daeb5fe40 1 -- 172.21.15.167:0/1367369157 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:79 5.c 5:30d299a5:::zonegroup_info.4f1f719d-9648-417b-b10c-e5d939825419:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32) v8 -- 0x563030d9a8a0 con 0x7f3d9c011040 2023-10-05T19:28:56.239 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.234+0000 7f3dace95700 1 -- 172.21.15.167:0/1367369157 <== osd.0 v2:172.21.15.118:6800/27690 20 ==== osd_op_reply(79 zonegroup_info.4f1f719d-9648-417b-b10c-e5d939825419 [read 0~362 out=362b] v0'0 uv1 ondisk = 0) v8 ==== 195+0+362 (crc 0 0 0) 0x7f3da402f080 con 0x7f3d9c011040 2023-10-05T19:28:56.239 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.234+0000 7f3daeb5fe40 20 rados_obj.operate() r=0 bl.length=362 2023-10-05T19:28:56.239 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.234+0000 7f3daeb5fe40 20 rados->read ofs=0 len=0 2023-10-05T19:28:56.239 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.234+0000 7f3daeb5fe40 1 -- 172.21.15.167:0/1367369157 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:80 5.3 5:c0ad8cbd:::region_map:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32) v8 -- 0x563030d9f4b0 con 0x7f3d9c011040 2023-10-05T19:28:56.239 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.234+0000 7f3dace95700 1 -- 172.21.15.167:0/1367369157 <== osd.0 v2:172.21.15.118:6800/27690 21 ==== osd_op_reply(80 region_map [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 ==== 154+0+0 (crc 0 0 0) 0x7f3da402f080 con 0x7f3d9c011040 2023-10-05T19:28:56.240 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.234+0000 7f3daeb5fe40 20 rados_obj.operate() r=-2 bl.length=0 2023-10-05T19:28:56.240 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.234+0000 7f3daeb5fe40 20 zone default found 2023-10-05T19:28:56.240 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.234+0000 7f3daeb5fe40 4 Realm: () 2023-10-05T19:28:56.240 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.234+0000 7f3daeb5fe40 4 ZoneGroup: default (4f1f719d-9648-417b-b10c-e5d939825419) 2023-10-05T19:28:56.240 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.234+0000 7f3daeb5fe40 4 Zone: default (9a9df965-6612-45d4-a45f-6e3c889bacc2) 2023-10-05T19:28:56.240 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.234+0000 7f3daeb5fe40 10 cannot find current period zonegroup using local zonegroup configuration 2023-10-05T19:28:56.240 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.234+0000 7f3daeb5fe40 20 zonegroup default 2023-10-05T19:28:56.240 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.234+0000 7f3daeb5fe40 20 rados->read ofs=0 len=0 2023-10-05T19:28:56.240 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.234+0000 7f3daeb5fe40 1 -- 172.21.15.167:0/1367369157 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:81 5.3 5:c52100b6:::period_config.default:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32) v8 -- 0x563030d9f4b0 con 0x7f3d9c011040 2023-10-05T19:28:56.242 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.234+0000 7f3dace95700 1 -- 172.21.15.167:0/1367369157 <== osd.0 v2:172.21.15.118:6800/27690 22 ==== osd_op_reply(81 period_config.default [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 ==== 165+0+0 (crc 0 0 0) 0x7f3da402f080 con 0x7f3d9c011040 2023-10-05T19:28:56.242 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.234+0000 7f3daeb5fe40 20 rados_obj.operate() r=-2 bl.length=0 2023-10-05T19:28:56.242 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.234+0000 7f3daeb5fe40 20 rados->read ofs=0 len=0 2023-10-05T19:28:56.242 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.234+0000 7f3daeb5fe40 1 -- 172.21.15.167:0/1367369157 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:82 6.2 6:4347d321:::bucket.sync-source-hints.:head [call version.read in=11b,read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32) v8 -- 0x563030da0de0 con 0x563030d9b6b0 2023-10-05T19:28:56.242 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.234+0000 7f3dac694700 1 -- 172.21.15.167:0/1367369157 <== osd.3 v2:172.21.15.167:6808/30036 20 ==== osd_op_reply(82 bucket.sync-source-hints. [call,read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 ==== 211+0+0 (crc 0 0 0) 0x7f3d94012450 con 0x563030d9b6b0 2023-10-05T19:28:56.242 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.234+0000 7f3daeb5fe40 20 rados_obj.operate() r=-2 bl.length=0 2023-10-05T19:28:56.242 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.234+0000 7f3daeb5fe40 20 rados->read ofs=0 len=0 2023-10-05T19:28:56.242 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.234+0000 7f3daeb5fe40 1 -- 172.21.15.167:0/1367369157 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:83 6.b 6:d467b91b:::bucket.sync-target-hints.:head [call version.read in=11b,read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32) v8 -- 0x563030da1190 con 0x563030d9b6b0 2023-10-05T19:28:56.243 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.238+0000 7f3dac694700 1 -- 172.21.15.167:0/1367369157 <== osd.3 v2:172.21.15.167:6808/30036 21 ==== osd_op_reply(83 bucket.sync-target-hints. [call,read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 ==== 211+0+0 (crc 0 0 0) 0x7f3d94012450 con 0x563030d9b6b0 2023-10-05T19:28:56.243 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.238+0000 7f3daeb5fe40 20 rados_obj.operate() r=-2 bl.length=0 2023-10-05T19:28:56.243 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.238+0000 7f3daeb5fe40 20 started sync module instance, tier type = 2023-10-05T19:28:56.243 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.238+0000 7f3daeb5fe40 20 started zone id=9a9df965-6612-45d4-a45f-6e3c889bacc2 (name=default) with tier type = 2023-10-05T19:28:56.243 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.238+0000 7f3daeb5fe40 1 -- 172.21.15.167:0/1367369157 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:84 7.1f 7:f95f44c2:::notify.0:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e32) v8 -- 0x563030da04a0 con 0x7f3d94007030 2023-10-05T19:28:56.245 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.238+0000 7f3dabe93700 1 -- 172.21.15.167:0/1367369157 <== osd.2 v2:172.21.15.167:6816/30037 13 ==== osd_op_reply(84 notify.0 [create] v32'201 uv201 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f3d9c02f040 con 0x7f3d94007030 2023-10-05T19:28:56.245 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.238+0000 7f3daeb5fe40 1 -- 172.21.15.167:0/1367369157 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:85 7.1f 7:f95f44c2:::notify.0:head [watch watch cookie 94764978012080] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e32) v8 -- 0x563030da1540 con 0x7f3d94007030 2023-10-05T19:28:56.245 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.238+0000 7f3daeb5fe40 1 -- 172.21.15.167:0/1367369157 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:86 7.0 7:05bf5b68:::notify.1:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e32) v8 -- 0x563030da1de0 con 0x563030d647f0 2023-10-05T19:28:56.248 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.242+0000 7f3dabe93700 1 -- 172.21.15.167:0/1367369157 <== osd.2 v2:172.21.15.167:6816/30037 14 ==== osd_op_reply(85 notify.0 [watch watch cookie 94764978012080] v32'202 uv201 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f3d9c02f040 con 0x7f3d94007030 2023-10-05T19:28:56.248 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.242+0000 7f3dace95700 1 -- 172.21.15.167:0/1367369157 <== osd.1 v2:172.21.15.167:6800/30035 29 ==== osd_op_reply(86 notify.1 [create] v32'201 uv201 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f3da402f080 con 0x563030d647f0 2023-10-05T19:28:56.248 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.242+0000 7f3daeb5fe40 1 -- 172.21.15.167:0/1367369157 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:87 7.0 7:05bf5b68:::notify.1:head [watch watch cookie 94764978018544] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e32) v8 -- 0x563030da2230 con 0x563030d647f0 2023-10-05T19:28:56.248 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.242+0000 7f3daeb5fe40 1 -- 172.21.15.167:0/1367369157 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:88 7.15 7:a93a5511:::notify.2:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e32) v8 -- 0x563030da2ad0 con 0x563030d9b6b0 2023-10-05T19:28:56.251 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.246+0000 7f3dac694700 1 -- 172.21.15.167:0/1367369157 <== osd.3 v2:172.21.15.167:6808/30036 22 ==== osd_op_reply(88 notify.2 [create] v32'201 uv201 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f3d94012450 con 0x563030d9b6b0 2023-10-05T19:28:56.251 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.246+0000 7f3daeb5fe40 1 -- 172.21.15.167:0/1367369157 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:89 7.15 7:a93a5511:::notify.2:head [watch watch cookie 94764978020912] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e32) v8 -- 0x563030da26c0 con 0x563030d9b6b0 2023-10-05T19:28:56.252 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.246+0000 7f3daeb5fe40 1 -- 172.21.15.167:0/1367369157 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:90 7.e 7:7759931f:::notify.3:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e32) v8 -- 0x563030da3b20 con 0x7f3d94007030 2023-10-05T19:28:56.252 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.246+0000 7f3dace95700 1 -- 172.21.15.167:0/1367369157 <== osd.1 v2:172.21.15.167:6800/30035 30 ==== osd_op_reply(87 notify.1 [watch watch cookie 94764978018544] v32'202 uv201 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f3da4050040 con 0x563030d647f0 2023-10-05T19:28:56.254 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.250+0000 7f3dabe93700 1 -- 172.21.15.167:0/1367369157 <== osd.2 v2:172.21.15.167:6816/30037 15 ==== osd_op_reply(90 notify.3 [create] v32'201 uv201 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f3d9c02f040 con 0x7f3d94007030 2023-10-05T19:28:56.254 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.250+0000 7f3daeb5fe40 1 -- 172.21.15.167:0/1367369157 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:91 7.e 7:7759931f:::notify.3:head [watch watch cookie 94764978019648] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e32) v8 -- 0x563030da3710 con 0x7f3d94007030 2023-10-05T19:28:56.254 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.250+0000 7f3daeb5fe40 1 -- 172.21.15.167:0/1367369157 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:92 7.d 7:b4812045:::notify.4:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e32) v8 -- 0x563030da4bc0 con 0x563030d647f0 2023-10-05T19:28:56.254 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.250+0000 7f3dac694700 1 -- 172.21.15.167:0/1367369157 <== osd.3 v2:172.21.15.167:6808/30036 23 ==== osd_op_reply(89 notify.2 [watch watch cookie 94764978020912] v32'202 uv201 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f3d94012450 con 0x563030d9b6b0 2023-10-05T19:28:56.257 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.250+0000 7f3dace95700 1 -- 172.21.15.167:0/1367369157 <== osd.1 v2:172.21.15.167:6800/30035 31 ==== osd_op_reply(92 notify.4 [create] v32'201 uv201 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f3da4050040 con 0x563030d647f0 2023-10-05T19:28:56.257 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.250+0000 7f3daeb5fe40 1 -- 172.21.15.167:0/1367369157 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:93 7.d 7:b4812045:::notify.4:head [watch watch cookie 94764978024064] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e32) v8 -- 0x563030da4fc0 con 0x563030d647f0 2023-10-05T19:28:56.257 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.250+0000 7f3daeb5fe40 1 -- 172.21.15.167:0/1367369157 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:94 7.3 7:c609908c:::notify.5:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e32) v8 -- 0x563030da59d0 con 0x7f3d9c011040 2023-10-05T19:28:56.257 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.250+0000 7f3dabe93700 1 -- 172.21.15.167:0/1367369157 <== osd.2 v2:172.21.15.167:6816/30037 16 ==== osd_op_reply(91 notify.3 [watch watch cookie 94764978019648] v32'202 uv201 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f3d9c02f040 con 0x7f3d94007030 2023-10-05T19:28:56.259 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.254+0000 7f3dace95700 1 -- 172.21.15.167:0/1367369157 <== osd.0 v2:172.21.15.118:6800/27690 23 ==== osd_op_reply(94 notify.5 [create] v32'201 uv201 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f3da4050040 con 0x7f3d9c011040 2023-10-05T19:28:56.259 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.254+0000 7f3daeb5fe40 1 -- 172.21.15.167:0/1367369157 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:95 7.3 7:c609908c:::notify.5:head [watch watch cookie 94764978028320] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e32) v8 -- 0x563030da59d0 con 0x7f3d9c011040 2023-10-05T19:28:56.259 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.254+0000 7f3daeb5fe40 1 -- 172.21.15.167:0/1367369157 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:96 7.14 7:2b04a3e9:::notify.6:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e32) v8 -- 0x563030da63e0 con 0x7f3d9c011040 2023-10-05T19:28:56.260 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.254+0000 7f3dace95700 1 -- 172.21.15.167:0/1367369157 <== osd.1 v2:172.21.15.167:6800/30035 32 ==== osd_op_reply(93 notify.4 [watch watch cookie 94764978024064] v32'202 uv201 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f3da4050040 con 0x563030d647f0 2023-10-05T19:28:56.261 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.258+0000 7f3dace95700 1 -- 172.21.15.167:0/1367369157 <== osd.0 v2:172.21.15.118:6800/27690 24 ==== osd_op_reply(95 notify.5 [watch watch cookie 94764978028320] v32'202 uv201 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f3da4050040 con 0x7f3d9c011040 2023-10-05T19:28:56.262 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.258+0000 7f3dace95700 1 -- 172.21.15.167:0/1367369157 <== osd.0 v2:172.21.15.118:6800/27690 25 ==== osd_op_reply(96 notify.6 [create] v32'201 uv201 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f3da4050040 con 0x7f3d9c011040 2023-10-05T19:28:56.262 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.258+0000 7f3daeb5fe40 1 -- 172.21.15.167:0/1367369157 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:97 7.14 7:2b04a3e9:::notify.6:head [watch watch cookie 94764978033888] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e32) v8 -- 0x563030da5930 con 0x7f3d9c011040 2023-10-05T19:28:56.262 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.258+0000 7f3daeb5fe40 1 -- 172.21.15.167:0/1367369157 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:98 7.9 7:93e5b521:::notify.7:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e32) v8 -- 0x563030da68d0 con 0x7f3d9c011040 2023-10-05T19:28:56.264 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.258+0000 7f3dace95700 1 -- 172.21.15.167:0/1367369157 <== osd.0 v2:172.21.15.118:6800/27690 26 ==== osd_op_reply(97 notify.6 [watch watch cookie 94764978033888] v32'202 uv201 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f3da4050040 con 0x7f3d9c011040 2023-10-05T19:28:56.265 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.258+0000 7f3dace95700 1 -- 172.21.15.167:0/1367369157 <== osd.0 v2:172.21.15.118:6800/27690 27 ==== osd_op_reply(98 notify.7 [create] v32'201 uv201 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f3da4050040 con 0x7f3d9c011040 2023-10-05T19:28:56.265 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.262+0000 7f3daeb5fe40 1 -- 172.21.15.167:0/1367369157 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:99 7.9 7:93e5b521:::notify.7:head [watch watch cookie 94764978036464] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e32) v8 -- 0x563030da6830 con 0x7f3d9c011040 2023-10-05T19:28:56.266 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.262+0000 7f3daeb5fe40 20 add_watcher() i=0 2023-10-05T19:28:56.266 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.262+0000 7f3daeb5fe40 20 add_watcher() i=1 2023-10-05T19:28:56.266 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.262+0000 7f3daeb5fe40 20 add_watcher() i=2 2023-10-05T19:28:56.266 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.262+0000 7f3daeb5fe40 20 add_watcher() i=3 2023-10-05T19:28:56.266 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.262+0000 7f3daeb5fe40 20 add_watcher() i=4 2023-10-05T19:28:56.266 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.262+0000 7f3daeb5fe40 20 add_watcher() i=5 2023-10-05T19:28:56.266 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.262+0000 7f3daeb5fe40 20 add_watcher() i=6 2023-10-05T19:28:56.267 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.262+0000 7f3dace95700 1 -- 172.21.15.167:0/1367369157 <== osd.0 v2:172.21.15.118:6800/27690 28 ==== osd_op_reply(99 notify.7 [watch watch cookie 94764978036464] v32'202 uv201 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f3da4050040 con 0x7f3d9c011040 2023-10-05T19:28:56.268 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.262+0000 7f3daeb5fe40 20 add_watcher() i=7 2023-10-05T19:28:56.268 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.262+0000 7f3daeb5fe40 2 all 8 watchers are set, enabling cache 2023-10-05T19:28:56.268 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.262+0000 7f3daeb5fe40 1 -- 172.21.15.167:0/1367369157 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:100 6.4 6:22d26bf9:::data_loggenerations_metadata:head [call version.check_conds in=74b,call version.read in=11b,read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32) v8 -- 0x563030da9a00 con 0x563030d647f0 2023-10-05T19:28:56.268 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.262+0000 7f3dace95700 1 -- 172.21.15.167:0/1367369157 <== osd.1 v2:172.21.15.167:6800/30035 33 ==== osd_op_reply(100 data_loggenerations_metadata [call,call out=48b,read 0~28 out=28b] v0'0 uv1 ondisk = 0) v8 ==== 256+0+76 (crc 0 0 0) 0x7f3da4050040 con 0x563030d647f0 2023-10-05T19:28:56.268 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.262+0000 7f3daeb5fe40 1 -- 172.21.15.167:0/1367369157 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:101 6.4 6:22d26bf9:::data_loggenerations_metadata:head [watch watch cookie 94764978037936] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e32) v8 -- 0x563030da6df0 con 0x563030d647f0 2023-10-05T19:28:56.270 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.266+0000 7f3dace95700 1 -- 172.21.15.167:0/1367369157 <== osd.1 v2:172.21.15.167:6800/30035 34 ==== osd_op_reply(101 data_loggenerations_metadata [watch watch cookie 94764978037936] v32'201 uv1 ondisk = 0) v8 ==== 172+0+0 (crc 0 0 0) 0x7f3da4050040 con 0x563030d647f0 2023-10-05T19:28:56.270 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.266+0000 7f3d7dfe3700 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2023-10-05T19:28:56.271 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.266+0000 7f3daeb5fe40 20 check_secure_mon_conn(): auth registy supported: methods=[2] modes=[2,1] 2023-10-05T19:28:56.271 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.266+0000 7f3daeb5fe40 20 check_secure_mon_conn(): mode 1 is insecure 2023-10-05T19:28:56.271 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.266+0000 7f3daeb5fe40 1 Processor -- start 2023-10-05T19:28:56.272 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.266+0000 7f3daeb5fe40 1 -- start start 2023-10-05T19:28:56.272 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.266+0000 7f3daeb5fe40 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x563030db4270 0x563030db4640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-05T19:28:56.272 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.266+0000 7f3daeb5fe40 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x563030db4b80 con 0x563030db4270 2023-10-05T19:28:56.272 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.266+0000 7f3dabe93700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x563030db4270 0x563030db4640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-05T19:28:56.272 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.266+0000 7f3dabe93700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x563030db4270 0x563030db4640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.118:3300/0 says I am v2:172.21.15.167:53330/0 (socket says 172.21.15.167:53330) 2023-10-05T19:28:56.272 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.266+0000 7f3dabe93700 1 -- 172.21.15.167:0/263877284 learned_addr learned my addr 172.21.15.167:0/263877284 (peer_addr_for_me v2:172.21.15.167:0/0) 2023-10-05T19:28:56.273 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.266+0000 7f3dabe93700 1 -- 172.21.15.167:0/263877284 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x563030db4cc0 con 0x563030db4270 2023-10-05T19:28:56.273 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.266+0000 7f3dabe93700 1 --2- 172.21.15.167:0/263877284 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x563030db4270 0x563030db4640 secure :-1 s=READY pgs=540 cs=0 l=1 rev1=1 crypto rx=0x7f3d9c001370 tx=0x7f3d9c006000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=784ee83327a33d0a server_cookie=0 in_seq=0 out_seq=0 2023-10-05T19:28:56.273 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.266+0000 7f3d7d7e2700 1 -- 172.21.15.167:0/263877284 <== mon.0 v2:172.21.15.118:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f3d9c02c040 con 0x563030db4270 2023-10-05T19:28:56.273 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.266+0000 7f3d7d7e2700 1 -- 172.21.15.167:0/263877284 <== mon.0 v2:172.21.15.118:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3d9c02d040 con 0x563030db4270 2023-10-05T19:28:56.273 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.266+0000 7f3d7d7e2700 1 -- 172.21.15.167:0/263877284 <== mon.0 v2:172.21.15.118:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f3d9c0289c0 con 0x563030db4270 2023-10-05T19:28:56.274 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.270+0000 7f3daeb5fe40 1 -- 172.21.15.167:0/263877284 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x563030db4270 msgr2=0x563030db4640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-05T19:28:56.275 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.270+0000 7f3daeb5fe40 1 --2- 172.21.15.167:0/263877284 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x563030db4270 0x563030db4640 secure :-1 s=READY pgs=540 cs=0 l=1 rev1=1 crypto rx=0x7f3d9c001370 tx=0x7f3d9c006000 comp rx=0 tx=0).stop 2023-10-05T19:28:56.275 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.270+0000 7f3daeb5fe40 1 -- 172.21.15.167:0/263877284 shutdown_connections 2023-10-05T19:28:56.275 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.270+0000 7f3daeb5fe40 1 --2- 172.21.15.167:0/263877284 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x563030db4270 0x563030db4640 unknown :-1 s=CLOSED pgs=540 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-05T19:28:56.275 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.270+0000 7f3daeb5fe40 1 -- 172.21.15.167:0/263877284 >> 172.21.15.167:0/263877284 conn(0x563030daf630 msgr2=0x563030dad780 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-05T19:28:56.275 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.270+0000 7f3daeb5fe40 1 -- 172.21.15.167:0/263877284 shutdown_connections 2023-10-05T19:28:56.275 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.270+0000 7f3daeb5fe40 1 -- 172.21.15.167:0/263877284 wait complete. 2023-10-05T19:28:56.276 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.270+0000 7f3daeb5fe40 1 Processor -- start 2023-10-05T19:28:56.276 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.270+0000 7f3daeb5fe40 1 -- start start 2023-10-05T19:28:56.276 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.270+0000 7f3daeb5fe40 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x563030db4270 0x563030db9ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-05T19:28:56.276 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.270+0000 7f3daeb5fe40 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x563030dbaea0 con 0x563030db4270 2023-10-05T19:28:56.277 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.270+0000 7f3dabe93700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x563030db4270 0x563030db9ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-05T19:28:56.277 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.270+0000 7f3dabe93700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x563030db4270 0x563030db9ea0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.118:3300/0 says I am v2:172.21.15.167:53336/0 (socket says 172.21.15.167:53336) 2023-10-05T19:28:56.277 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.270+0000 7f3dabe93700 1 -- 172.21.15.167:0/1327780082 learned_addr learned my addr 172.21.15.167:0/1327780082 (peer_addr_for_me v2:172.21.15.167:0/0) 2023-10-05T19:28:56.277 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.270+0000 7f3dabe93700 1 -- 172.21.15.167:0/1327780082 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x563030cb0bd0 con 0x563030db4270 2023-10-05T19:28:56.278 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.270+0000 7f3dabe93700 1 --2- 172.21.15.167:0/1327780082 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x563030db4270 0x563030db9ea0 secure :-1 s=READY pgs=541 cs=0 l=1 rev1=1 crypto rx=0x7f3d9c001370 tx=0x7f3d9c03c000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-05T19:28:56.278 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.270+0000 7f3cf77fe700 1 -- 172.21.15.167:0/1327780082 <== mon.0 v2:172.21.15.118:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f3d9c02c070 con 0x563030db4270 2023-10-05T19:28:56.278 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.270+0000 7f3cf77fe700 1 -- 172.21.15.167:0/1327780082 <== mon.0 v2:172.21.15.118:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3d9c01c040 con 0x563030db4270 2023-10-05T19:28:56.278 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.270+0000 7f3cf77fe700 1 -- 172.21.15.167:0/1327780082 <== mon.0 v2:172.21.15.118:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f3d9c028ee0 con 0x563030db4270 2023-10-05T19:28:56.278 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.270+0000 7f3daeb5fe40 1 -- 172.21.15.167:0/1327780082 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x563030dba4a0 con 0x563030db4270 2023-10-05T19:28:56.278 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.270+0000 7f3daeb5fe40 1 -- 172.21.15.167:0/1327780082 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x563030dba8e0 con 0x563030db4270 2023-10-05T19:28:56.279 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.274+0000 7f3cf77fe700 1 -- 172.21.15.167:0/1327780082 <== mon.0 v2:172.21.15.118:3300/0 4 ==== mgrmap(e 5) v1 ==== 40329+0+0 (secure 0 0 0) 0x7f3d9c045040 con 0x563030db4270 2023-10-05T19:28:56.279 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.274+0000 7f3cf77fe700 1 --2- 172.21.15.167:0/1327780082 >> [v2:172.21.15.167:6824/29889,v1:172.21.15.167:6825/29889] conn(0x7f3cec030c50 0x7f3cec033100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-05T19:28:56.279 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.274+0000 7f3cf77fe700 1 -- 172.21.15.167:0/1327780082 <== mon.0 v2:172.21.15.118:3300/0 5 ==== osd_map(32..32 src has 1..32) v4 ==== 6260+0+0 (secure 0 0 0) 0x7f3d9c051cf0 con 0x563030db4270 2023-10-05T19:28:56.279 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.274+0000 7f3daeb5fe40 5 note: GC not initialized 2023-10-05T19:28:56.279 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.274+0000 7f3dace95700 1 --2- 172.21.15.167:0/1327780082 >> [v2:172.21.15.167:6824/29889,v1:172.21.15.167:6825/29889] conn(0x7f3cec030c50 0x7f3cec033100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-05T19:28:56.280 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.274+0000 7f3dace95700 1 --2- 172.21.15.167:0/1327780082 >> [v2:172.21.15.167:6824/29889,v1:172.21.15.167:6825/29889] conn(0x7f3cec030c50 0x7f3cec033100 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x563030da9ab0 tx=0x7f3da4000f90 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-05T19:28:56.282 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.278+0000 7f3cf4ff9700 20 reqs_thread_entry: start 2023-10-05T19:28:56.282 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.278+0000 7f3cea7fc700 20 reqs_thread_entry: start 2023-10-05T19:28:56.282 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.278+0000 7f3daeb5fe40 20 init_complete bucket index max shards: 11 2023-10-05T19:28:56.283 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.278+0000 7f3daeb5fe40 10 rgw notify: Started notification manager with: 1 workers 2023-10-05T19:28:56.283 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.278+0000 7f3ce8ff9700 20 rgw notify: INFO: next queues processing will happen at: Thu Oct 5 19:29:26 2023 2023-10-05T19:28:56.283 INFO:teuthology.orchestra.run.smithi167.stderr: 2023-10-05T19:28:56.283 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.278+0000 7f3ccffff700 20 reqs_thread_entry: start 2023-10-05T19:28:56.283 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.278+0000 7f3daeb5fe40 20 get_system_obj_state: rctx=0x7fff4de62880 obj=default.rgw.meta:root:myfoo state=0x563030c3d0b0 s->prefetch_data=0 2023-10-05T19:28:56.284 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.278+0000 7f3daeb5fe40 10 cache get: name=default.rgw.meta+root+myfoo : miss 2023-10-05T19:28:56.284 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.278+0000 7f3daeb5fe40 1 -- 172.21.15.167:0/1367369157 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:102 8.9 8:91b71841:root::myfoo:head [call version.read in=11b,getxattrs,stat] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32) v8 -- 0x563030e22f50 con 0x7f3d94007030 2023-10-05T19:28:56.284 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.278+0000 7f3dabe93700 1 -- 172.21.15.167:0/1367369157 <== osd.2 v2:172.21.15.167:6816/30037 17 ==== osd_op_reply(102 myfoo [call out=48b,getxattrs out=75b,stat out=16b] v0'0 uv34 ondisk = 0) v8 ==== 233+0+139 (crc 0 0 0) 0x7f3d9c02f040 con 0x7f3d94007030 2023-10-05T19:28:56.284 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.278+0000 7f3daeb5fe40 10 cache put: name=default.rgw.meta+root+myfoo info.flags=0x16 2023-10-05T19:28:56.284 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.278+0000 7f3daeb5fe40 10 adding default.rgw.meta+root+myfoo to cache LRU end 2023-10-05T19:28:56.285 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.278+0000 7f3daeb5fe40 10 updating xattr: name=ceph.objclass.version bl.length()=42 2023-10-05T19:28:56.285 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.278+0000 7f3daeb5fe40 20 get_system_obj_state: s->obj_tag was set empty 2023-10-05T19:28:56.285 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.278+0000 7f3daeb5fe40 10 cache get: name=default.rgw.meta+root+myfoo : type miss (requested=0x11, cached=0x16) 2023-10-05T19:28:56.285 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.278+0000 7f3daeb5fe40 20 rados->read ofs=0 len=0 2023-10-05T19:28:56.285 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.278+0000 7f3daeb5fe40 1 -- 172.21.15.167:0/1367369157 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:103 8.9 8:91b71841:root::myfoo:head [call version.check_conds in=122b,call version.read in=11b,read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32) v8 -- 0x563030e259e0 con 0x7f3d94007030 2023-10-05T19:28:56.287 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.278+0000 7f3dabe93700 1 -- 172.21.15.167:0/1367369157 <== osd.2 v2:172.21.15.167:6816/30037 18 ==== osd_op_reply(103 myfoo [call,call out=48b,read 0~165 out=165b] v0'0 uv34 ondisk = 0) v8 ==== 233+0+213 (crc 0 0 0) 0x7f3d9c02f040 con 0x7f3d94007030 2023-10-05T19:28:56.287 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.278+0000 7f3daeb5fe40 20 rados_obj.operate() r=0 bl.length=165 2023-10-05T19:28:56.287 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.278+0000 7f3daeb5fe40 10 cache put: name=default.rgw.meta+root+myfoo info.flags=0x11 2023-10-05T19:28:56.287 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.278+0000 7f3daeb5fe40 10 moving default.rgw.meta+root+myfoo to cache LRU end 2023-10-05T19:28:56.287 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.278+0000 7f3daeb5fe40 20 get_system_obj_state: rctx=0x7fff4de62880 obj=default.rgw.meta:root:.bucket.meta.myfoo:9a9df965-6612-45d4-a45f-6e3c889bacc2.4223.5 state=0x563030c132b0 s->prefetch_data=0 2023-10-05T19:28:56.287 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.278+0000 7f3daeb5fe40 10 cache get: name=default.rgw.meta+root+.bucket.meta.myfoo:9a9df965-6612-45d4-a45f-6e3c889bacc2.4223.5 : miss 2023-10-05T19:28:56.287 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.278+0000 7f3daeb5fe40 1 -- 172.21.15.167:0/1367369157 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:104 8.9 8:96cb4b0f:root::.bucket.meta.myfoo%3a9a9df965-6612-45d4-a45f-6e3c889bacc2.4223.5:head [call version.read in=11b,getxattrs,stat] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32) v8 -- 0x563030e24df0 con 0x7f3d94007030 2023-10-05T19:28:56.287 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.282+0000 7f3dabe93700 1 -- 172.21.15.167:0/1367369157 <== osd.2 v2:172.21.15.167:6816/30037 19 ==== osd_op_reply(104 .bucket.meta.myfoo:9a9df965-6612-45d4-a45f-6e3c889bacc2.4223.5 [call out=48b,getxattrs out=218b,stat out=16b] v0'0 uv33 ondisk = 0) v8 ==== 290+0+282 (crc 0 0 0) 0x7f3d9c02f040 con 0x7f3d94007030 2023-10-05T19:28:56.287 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.282+0000 7f3daeb5fe40 10 cache put: name=default.rgw.meta+root+.bucket.meta.myfoo:9a9df965-6612-45d4-a45f-6e3c889bacc2.4223.5 info.flags=0x16 2023-10-05T19:28:56.287 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.282+0000 7f3daeb5fe40 10 adding default.rgw.meta+root+.bucket.meta.myfoo:9a9df965-6612-45d4-a45f-6e3c889bacc2.4223.5 to cache LRU end 2023-10-05T19:28:56.288 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.282+0000 7f3daeb5fe40 10 updating xattr: name=ceph.objclass.version bl.length()=42 2023-10-05T19:28:56.288 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.282+0000 7f3daeb5fe40 10 updating xattr: name=user.rgw.acl bl.length()=123 2023-10-05T19:28:56.288 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.282+0000 7f3daeb5fe40 20 get_system_obj_state: s->obj_tag was set empty 2023-10-05T19:28:56.288 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.282+0000 7f3daeb5fe40 20 Read xattr: user.rgw.acl 2023-10-05T19:28:56.288 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.282+0000 7f3daeb5fe40 20 Read xattr: user.rgw.idtag 2023-10-05T19:28:56.288 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.282+0000 7f3daeb5fe40 10 cache get: name=default.rgw.meta+root+.bucket.meta.myfoo:9a9df965-6612-45d4-a45f-6e3c889bacc2.4223.5 : type miss (requested=0x13, cached=0x16) 2023-10-05T19:28:56.288 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.282+0000 7f3daeb5fe40 20 rados->read ofs=0 len=0 2023-10-05T19:28:56.288 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.282+0000 7f3daeb5fe40 1 -- 172.21.15.167:0/1367369157 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:105 8.9 8:96cb4b0f:root::.bucket.meta.myfoo%3a9a9df965-6612-45d4-a45f-6e3c889bacc2.4223.5:head [call version.check_conds in=122b,call version.read in=11b,read 0~0,getxattrs] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32) v8 -- 0x563030e25980 con 0x7f3d94007030 2023-10-05T19:28:56.288 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.282+0000 7f3dabe93700 1 -- 172.21.15.167:0/1367369157 <== osd.2 v2:172.21.15.167:6816/30037 20 ==== osd_op_reply(105 .bucket.meta.myfoo:9a9df965-6612-45d4-a45f-6e3c889bacc2.4223.5 [call,call out=48b,read 0~352 out=352b,getxattrs out=218b] v0'0 uv33 ondisk = 0) v8 ==== 332+0+618 (crc 0 0 0) 0x7f3d9c02f040 con 0x7f3d94007030 2023-10-05T19:28:56.288 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.282+0000 7f3daeb5fe40 20 rados_obj.operate() r=0 bl.length=352 2023-10-05T19:28:56.288 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.282+0000 7f3daeb5fe40 10 cache put: name=default.rgw.meta+root+.bucket.meta.myfoo:9a9df965-6612-45d4-a45f-6e3c889bacc2.4223.5 info.flags=0x13 2023-10-05T19:28:56.289 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.282+0000 7f3daeb5fe40 10 moving default.rgw.meta+root+.bucket.meta.myfoo:9a9df965-6612-45d4-a45f-6e3c889bacc2.4223.5 to cache LRU end 2023-10-05T19:28:56.289 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.282+0000 7f3daeb5fe40 10 updating xattr: name=ceph.objclass.version bl.length()=42 2023-10-05T19:28:56.289 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.282+0000 7f3daeb5fe40 10 updating xattr: name=user.rgw.acl bl.length()=123 2023-10-05T19:28:56.289 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.282+0000 7f3daeb5fe40 10 chain_cache_entry: cache_locator=default.rgw.meta+root+.bucket.meta.myfoo:9a9df965-6612-45d4-a45f-6e3c889bacc2.4223.5 2023-10-05T19:28:56.289 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.282+0000 7f3daeb5fe40 20 get_obj_state: rctx=0x7fff4de62ac0 obj=myfoo:bb3aec0fdcdbc2974890f805c585d432 state=0x563030e283d8 s->prefetch_data=0 2023-10-05T19:28:56.289 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.282+0000 7f3daeb5fe40 1 -- 172.21.15.167:0/1367369157 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:106 3.1cs0 3:38675d0b:::9a9df965-6612-45d4-a45f-6e3c889bacc2.4223.5_bb3aec0fdcdbc2974890f805c585d432:head [getxattrs,stat] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32) v8 -- 0x563030e28d40 con 0x563030d647f0 2023-10-05T19:28:56.290 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.282+0000 7f3dace95700 1 -- 172.21.15.167:0/1367369157 <== osd.1 v2:172.21.15.167:6800/30035 35 ==== osd_op_reply(106 9a9df965-6612-45d4-a45f-6e3c889bacc2.4223.5_bb3aec0fdcdbc2974890f805c585d432 [getxattrs out=848b,stat out=16b] v0'0 uv2 ondisk = 0) v8 ==== 262+0+864 (crc 0 0 0) 0x7f3da4050040 con 0x563030d647f0 2023-10-05T19:28:56.290 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.282+0000 7f3daeb5fe40 10 manifest: total_size = 5 2023-10-05T19:28:56.290 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.282+0000 7f3daeb5fe40 20 get_obj_state: setting s->obj_tag to _qh1-J0T-HPdTyrcxHf-9FVkWh8X8Nh0 2023-10-05T19:28:56.290 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.282+0000 7f3daeb5fe40 1 -- 172.21.15.167:0/1367369157 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:107 7.1f 7:f95f44c2:::notify.0:head [watch unwatch cookie 94764978012080] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e32) v8 -- 0x563030d7ba70 con 0x7f3d94007030 2023-10-05T19:28:56.290 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.286+0000 7f3dabe93700 1 -- 172.21.15.167:0/1367369157 <== osd.2 v2:172.21.15.167:6816/30037 21 ==== osd_op_reply(107 notify.0 [watch unwatch cookie 94764978012080] v32'203 uv201 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f3d9c02f040 con 0x7f3d94007030 2023-10-05T19:28:56.291 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.286+0000 7f3daeb5fe40 20 remove_watcher() i=0 2023-10-05T19:28:56.291 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.286+0000 7f3daeb5fe40 2 removed watcher, disabling cache 2023-10-05T19:28:56.291 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.286+0000 7f3daeb5fe40 1 -- 172.21.15.167:0/1367369157 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:108 7.0 7:05bf5b68:::notify.1:head [watch unwatch cookie 94764978018544] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e32) v8 -- 0x563030d7ba70 con 0x563030d647f0 2023-10-05T19:28:56.293 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.286+0000 7f3dace95700 1 -- 172.21.15.167:0/1367369157 <== osd.1 v2:172.21.15.167:6800/30035 36 ==== osd_op_reply(108 notify.1 [watch unwatch cookie 94764978018544] v32'203 uv201 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f3da4050040 con 0x563030d647f0 2023-10-05T19:28:56.293 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.286+0000 7f3daeb5fe40 20 remove_watcher() i=1 2023-10-05T19:28:56.293 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.286+0000 7f3daeb5fe40 1 -- 172.21.15.167:0/1367369157 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:109 7.15 7:a93a5511:::notify.2:head [watch unwatch cookie 94764978020912] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e32) v8 -- 0x563030d7ba70 con 0x563030d9b6b0 2023-10-05T19:28:56.296 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.290+0000 7f3dac694700 1 -- 172.21.15.167:0/1367369157 <== osd.3 v2:172.21.15.167:6808/30036 24 ==== osd_op_reply(109 notify.2 [watch unwatch cookie 94764978020912] v32'203 uv201 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f3d94012450 con 0x563030d9b6b0 2023-10-05T19:28:56.296 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.290+0000 7f3daeb5fe40 20 remove_watcher() i=2 2023-10-05T19:28:56.296 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.290+0000 7f3daeb5fe40 1 -- 172.21.15.167:0/1367369157 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:110 7.e 7:7759931f:::notify.3:head [watch unwatch cookie 94764978019648] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e32) v8 -- 0x563030da2230 con 0x7f3d94007030 2023-10-05T19:28:56.299 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.294+0000 7f3dabe93700 1 -- 172.21.15.167:0/1367369157 <== osd.2 v2:172.21.15.167:6816/30037 22 ==== osd_op_reply(110 notify.3 [watch unwatch cookie 94764978019648] v32'203 uv201 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f3d9c02f040 con 0x7f3d94007030 2023-10-05T19:28:56.299 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.294+0000 7f3daeb5fe40 20 remove_watcher() i=3 2023-10-05T19:28:56.299 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.294+0000 7f3daeb5fe40 1 -- 172.21.15.167:0/1367369157 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:111 7.d 7:b4812045:::notify.4:head [watch unwatch cookie 94764978024064] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e32) v8 -- 0x563030da2230 con 0x563030d647f0 2023-10-05T19:28:56.301 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.298+0000 7f3dace95700 1 -- 172.21.15.167:0/1367369157 <== osd.1 v2:172.21.15.167:6800/30035 37 ==== osd_op_reply(111 notify.4 [watch unwatch cookie 94764978024064] v32'203 uv201 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f3da4050040 con 0x563030d647f0 2023-10-05T19:28:56.302 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.298+0000 7f3daeb5fe40 20 remove_watcher() i=4 2023-10-05T19:28:56.302 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.298+0000 7f3daeb5fe40 1 -- 172.21.15.167:0/1367369157 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:112 7.3 7:c609908c:::notify.5:head [watch unwatch cookie 94764978028320] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e32) v8 -- 0x563030da2e80 con 0x7f3d9c011040 2023-10-05T19:28:56.304 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.298+0000 7f3dace95700 1 -- 172.21.15.167:0/1367369157 <== osd.0 v2:172.21.15.118:6800/27690 29 ==== osd_op_reply(112 notify.5 [watch unwatch cookie 94764978028320] v32'203 uv201 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f3da4050040 con 0x7f3d9c011040 2023-10-05T19:28:56.304 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.298+0000 7f3daeb5fe40 20 remove_watcher() i=5 2023-10-05T19:28:56.304 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.298+0000 7f3daeb5fe40 1 -- 172.21.15.167:0/1367369157 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:113 7.14 7:2b04a3e9:::notify.6:head [watch unwatch cookie 94764978033888] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e32) v8 -- 0x563030da2e80 con 0x7f3d9c011040 2023-10-05T19:28:56.307 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.302+0000 7f3dace95700 1 -- 172.21.15.167:0/1367369157 <== osd.0 v2:172.21.15.118:6800/27690 30 ==== osd_op_reply(113 notify.6 [watch unwatch cookie 94764978033888] v32'203 uv201 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f3da4050040 con 0x7f3d9c011040 2023-10-05T19:28:56.307 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.302+0000 7f3daeb5fe40 20 remove_watcher() i=6 2023-10-05T19:28:56.307 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.302+0000 7f3daeb5fe40 1 -- 172.21.15.167:0/1367369157 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:114 7.9 7:93e5b521:::notify.7:head [watch unwatch cookie 94764978036464] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e32) v8 -- 0x563030da2e80 con 0x7f3d9c011040 2023-10-05T19:28:56.310 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.306+0000 7f3dace95700 1 -- 172.21.15.167:0/1367369157 <== osd.0 v2:172.21.15.118:6800/27690 31 ==== osd_op_reply(114 notify.7 [watch unwatch cookie 94764978036464] v32'203 uv201 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f3da4050040 con 0x7f3d9c011040 2023-10-05T19:28:56.310 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.306+0000 7f3daeb5fe40 20 remove_watcher() i=7 2023-10-05T19:28:56.312 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.306+0000 7f3d7dfe3700 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2023-10-05T19:28:56.312 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.306+0000 7f3daeb5fe40 1 -- 172.21.15.167:0/1367369157 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:115 6.4 6:22d26bf9:::data_loggenerations_metadata:head [watch unwatch cookie 94764978037936] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e32) v8 -- 0x563030da5ef0 con 0x563030d647f0 2023-10-05T19:28:56.316 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.310+0000 7f3dace95700 1 -- 172.21.15.167:0/1367369157 <== osd.1 v2:172.21.15.167:6800/30035 38 ==== osd_op_reply(115 data_loggenerations_metadata [watch unwatch cookie 94764978037936] v32'202 uv1 ondisk = 0) v8 ==== 172+0+0 (crc 0 0 0) 0x7f3da4050040 con 0x563030d647f0 2023-10-05T19:28:56.316 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.310+0000 7f3daeb5fe40 1 -- 172.21.15.167:0/1367369157 >> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] conn(0x7f3d9c011040 msgr2=0x7f3d9c015960 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-05T19:28:56.317 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.310+0000 7f3daeb5fe40 1 --2- 172.21.15.167:0/1367369157 >> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] conn(0x7f3d9c011040 0x7f3d9c015960 crc :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-05T19:28:56.317 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.310+0000 7f3daeb5fe40 1 -- 172.21.15.167:0/1367369157 >> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] conn(0x563030d647f0 msgr2=0x563030d978d0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-05T19:28:56.317 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.310+0000 7f3daeb5fe40 1 --2- 172.21.15.167:0/1367369157 >> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] conn(0x563030d647f0 0x563030d978d0 crc :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-05T19:28:56.317 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.310+0000 7f3daeb5fe40 1 -- 172.21.15.167:0/1367369157 >> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] conn(0x7f3d94007030 msgr2=0x7f3d9401e3b0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-05T19:28:56.317 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.310+0000 7f3daeb5fe40 1 --2- 172.21.15.167:0/1367369157 >> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] conn(0x7f3d94007030 0x7f3d9401e3b0 crc :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-05T19:28:56.317 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.310+0000 7f3daeb5fe40 1 -- 172.21.15.167:0/1367369157 >> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] conn(0x563030d9b6b0 msgr2=0x563030d9da90 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-05T19:28:56.317 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.310+0000 7f3daeb5fe40 1 --2- 172.21.15.167:0/1367369157 >> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] conn(0x563030d9b6b0 0x563030d9da90 crc :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-05T19:28:56.317 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.310+0000 7f3daeb5fe40 1 -- 172.21.15.167:0/1367369157 >> [v2:172.21.15.167:6824/29889,v1:172.21.15.167:6825/29889] conn(0x7f3d8c030c50 msgr2=0x7f3d8c033100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-05T19:28:56.317 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.310+0000 7f3daeb5fe40 1 --2- 172.21.15.167:0/1367369157 >> [v2:172.21.15.167:6824/29889,v1:172.21.15.167:6825/29889] conn(0x7f3d8c030c50 0x7f3d8c033100 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7f3d9c0016b0 tx=0x7f3d9c0067b0 comp rx=0 tx=0).stop 2023-10-05T19:28:56.317 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.310+0000 7f3daeb5fe40 1 -- 172.21.15.167:0/1367369157 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x563030d64420 msgr2=0x563030d8be30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-05T19:28:56.317 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.310+0000 7f3dace95700 1 -- 172.21.15.167:0/1367369157 reap_dead start 2023-10-05T19:28:56.318 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.310+0000 7f3daeb5fe40 1 --2- 172.21.15.167:0/1367369157 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x563030d64420 0x563030d8be30 secure :-1 s=READY pgs=539 cs=0 l=1 rev1=1 crypto rx=0x7f3d9400a650 tx=0x7f3d9401c000 comp rx=0 tx=0).stop 2023-10-05T19:28:56.318 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.310+0000 7f3daeb5fe40 1 -- 172.21.15.167:0/1367369157 shutdown_connections 2023-10-05T19:28:56.318 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.310+0000 7f3daeb5fe40 1 --2- 172.21.15.167:0/1367369157 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x563030d64420 0x563030d8be30 unknown :-1 s=CLOSED pgs=539 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-05T19:28:56.318 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.310+0000 7f3daeb5fe40 1 -- 172.21.15.167:0/1367369157 >> 172.21.15.167:0/1367369157 conn(0x563030d5e730 msgr2=0x563030d7f810 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-05T19:28:56.318 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.310+0000 7f3daeb5fe40 1 -- 172.21.15.167:0/1367369157 shutdown_connections 2023-10-05T19:28:56.318 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.310+0000 7f3daeb5fe40 1 -- 172.21.15.167:0/1367369157 wait complete. 2023-10-05T19:28:56.319 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.310+0000 7f3daeb5fe40 1 -- 172.21.15.167:0/1327780082 >> [v2:172.21.15.167:6824/29889,v1:172.21.15.167:6825/29889] conn(0x7f3cec030c50 msgr2=0x7f3cec033100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-05T19:28:56.319 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.310+0000 7f3daeb5fe40 1 --2- 172.21.15.167:0/1327780082 >> [v2:172.21.15.167:6824/29889,v1:172.21.15.167:6825/29889] conn(0x7f3cec030c50 0x7f3cec033100 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x563030da9ab0 tx=0x7f3da4000f90 comp rx=0 tx=0).stop 2023-10-05T19:28:56.319 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.310+0000 7f3daeb5fe40 1 -- 172.21.15.167:0/1327780082 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x563030db4270 msgr2=0x563030db9ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-05T19:28:56.319 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.310+0000 7f3daeb5fe40 1 --2- 172.21.15.167:0/1327780082 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x563030db4270 0x563030db9ea0 secure :-1 s=READY pgs=541 cs=0 l=1 rev1=1 crypto rx=0x7f3d9c001370 tx=0x7f3d9c03c000 comp rx=0 tx=0).stop 2023-10-05T19:28:56.319 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.310+0000 7f3daeb5fe40 1 -- 172.21.15.167:0/1327780082 shutdown_connections 2023-10-05T19:28:56.319 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.310+0000 7f3daeb5fe40 1 --2- 172.21.15.167:0/1327780082 >> [v2:172.21.15.167:6824/29889,v1:172.21.15.167:6825/29889] conn(0x7f3cec030c50 0x7f3cec033100 unknown :-1 s=CLOSED pgs=187 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-05T19:28:56.319 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.310+0000 7f3daeb5fe40 1 --2- 172.21.15.167:0/1327780082 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x563030db4270 0x563030db9ea0 unknown :-1 s=CLOSED pgs=541 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-05T19:28:56.320 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.310+0000 7f3daeb5fe40 1 -- 172.21.15.167:0/1327780082 >> 172.21.15.167:0/1327780082 conn(0x563030daf630 msgr2=0x563030dad6d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-05T19:28:56.320 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.310+0000 7f3daeb5fe40 1 -- 172.21.15.167:0/1327780082 shutdown_connections 2023-10-05T19:28:56.320 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.310+0000 7f3daeb5fe40 1 -- 172.21.15.167:0/1327780082 wait complete. 2023-10-05T19:28:56.323 INFO:teuthology.orchestra.run.smithi167.stdout:fooFoofooFooFULL_CONTROL 2023-10-05T19:28:56.323 DEBUG:tasks.util.rgw: raw result: fooFoofooFooFULL_CONTROL 2023-10-05T19:28:56.327 INFO:tasks.radosgw_admin:RL: get_acls myfoo foo bytes_out=0 bytes_in=417 failed=False 2023-10-05T19:28:56.337 INFO:tasks.radosgw_admin:RL: put_acls myfoo foo bytes_out=0 bytes_in=0 failed=False 2023-10-05T19:28:56.338 INFO:tasks.util.rgw:rgwadmin: client.0 : ['policy', '--bucket', 'myfoo', '--object', 'bb3aec0fdcdbc2974890f805c585d432'] 2023-10-05T19:28:56.338 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'xml', '-n', 'client.0', '--cluster', 'ceph', 'policy', '--bucket', 'myfoo', '--object', 'bb3aec0fdcdbc2974890f805c585d432'] 2023-10-05T19:28:56.338 DEBUG:teuthology.orchestra.run.smithi167:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format xml -n client.0 --cluster ceph policy --bucket myfoo --object bb3aec0fdcdbc2974890f805c585d432 2023-10-05T19:28:56.376 INFO:teuthology.orchestra.run.smithi167.stderr:ignoring --setuser ceph since I am not root 2023-10-05T19:28:56.376 INFO:teuthology.orchestra.run.smithi167.stderr:ignoring --setgroup ceph since I am not root 2023-10-05T19:28:56.377 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.370+0000 7fced3a8ee40 1 Processor -- start 2023-10-05T19:28:56.377 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.370+0000 7fced3a8ee40 1 -- start start 2023-10-05T19:28:56.377 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.370+0000 7fced3a8ee40 1 --2- >> v2:172.21.15.118:3300/0 conn(0x5584b0af1110 0x5584b0af3520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-05T19:28:56.377 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.370+0000 7fced3a8ee40 1 -- --> v1:172.21.15.118:6789/0 -- auth(proto 0 26 bytes epoch 0) v1 -- 0x5584b0a3bbd0 con 0x5584b0a633e0 2023-10-05T19:28:56.377 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.370+0000 7fced3a8ee40 1 -- --> v2:172.21.15.118:3300/0 -- mon_getmap magic: 0 v1 -- 0x5584b09f8ee0 con 0x5584b0af1110 2023-10-05T19:28:56.378 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.374+0000 7fced0dc2700 1 --2- >> v2:172.21.15.118:3300/0 conn(0x5584b0af1110 0x5584b0af3520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-05T19:28:56.378 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.374+0000 7fced0dc2700 1 --2- >> v2:172.21.15.118:3300/0 conn(0x5584b0af1110 0x5584b0af3520 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.118:3300/0 says I am v2:172.21.15.167:53346/0 (socket says 172.21.15.167:53346) 2023-10-05T19:28:56.378 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.374+0000 7fced0dc2700 1 -- 172.21.15.167:0/3882064982 learned_addr learned my addr 172.21.15.167:0/3882064982 (peer_addr_for_me v2:172.21.15.167:0/0) 2023-10-05T19:28:56.378 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.374+0000 7fced15c3700 1 --1- 172.21.15.167:0/3882064982 >> v1:172.21.15.118:6789/0 conn(0x5584b0a633e0 0x5584b0aef9f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.118:6789/0 says I am v1:172.21.15.167:48698/0 (socket says 172.21.15.167:48698) 2023-10-05T19:28:56.378 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.374+0000 7fced0dc2700 1 -- 172.21.15.167:0/3882064982 >> v1:172.21.15.118:6789/0 conn(0x5584b0a633e0 legacy=0x5584b0aef9f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-05T19:28:56.378 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.374+0000 7fced0dc2700 1 -- 172.21.15.167:0/3882064982 --> v2:172.21.15.118:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5584b0af3a60 con 0x5584b0af1110 2023-10-05T19:28:56.379 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.374+0000 7fced0dc2700 1 --2- 172.21.15.167:0/3882064982 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5584b0af1110 0x5584b0af3520 secure :-1 s=READY pgs=543 cs=0 l=1 rev1=1 crypto rx=0x7fcebc00a5c0 tx=0x7fcebc005e00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6fd4818711b740fa server_cookie=0 in_seq=0 out_seq=0 2023-10-05T19:28:56.379 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.374+0000 7fcecbfff700 1 -- 172.21.15.167:0/3882064982 <== mon.0 v2:172.21.15.118:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fcebc00e510 con 0x5584b0af1110 2023-10-05T19:28:56.380 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.374+0000 7fcecbfff700 1 -- 172.21.15.167:0/3882064982 <== mon.0 v2:172.21.15.118:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcebc00e670 con 0x5584b0af1110 2023-10-05T19:28:56.380 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.374+0000 7fcecbfff700 1 -- 172.21.15.167:0/3882064982 <== mon.0 v2:172.21.15.118:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fcebc00e920 con 0x5584b0af1110 2023-10-05T19:28:56.380 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.374+0000 7fced3a8ee40 1 -- 172.21.15.167:0/3882064982 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5584b0af1110 msgr2=0x5584b0af3520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-05T19:28:56.380 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.374+0000 7fced3a8ee40 1 --2- 172.21.15.167:0/3882064982 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5584b0af1110 0x5584b0af3520 secure :-1 s=READY pgs=543 cs=0 l=1 rev1=1 crypto rx=0x7fcebc00a5c0 tx=0x7fcebc005e00 comp rx=0 tx=0).stop 2023-10-05T19:28:56.380 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.374+0000 7fced3a8ee40 1 -- 172.21.15.167:0/3882064982 shutdown_connections 2023-10-05T19:28:56.380 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.374+0000 7fced3a8ee40 1 --2- 172.21.15.167:0/3882064982 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5584b0af1110 0x5584b0af3520 unknown :-1 s=CLOSED pgs=543 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-05T19:28:56.380 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.374+0000 7fced3a8ee40 1 -- 172.21.15.167:0/3882064982 >> 172.21.15.167:0/3882064982 conn(0x5584b0ae9750 msgr2=0x5584b0aebb60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-05T19:28:56.380 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.374+0000 7fced3a8ee40 1 -- 172.21.15.167:0/3882064982 shutdown_connections 2023-10-05T19:28:56.380 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.374+0000 7fced3a8ee40 1 -- 172.21.15.167:0/3882064982 wait complete. 2023-10-05T19:28:56.381 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.374+0000 7fced3a8ee40 1 Processor -- start 2023-10-05T19:28:56.381 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.374+0000 7fced3a8ee40 1 -- start start 2023-10-05T19:28:56.381 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.374+0000 7fced3a8ee40 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5584b0a633e0 0x5584b0b14570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-05T19:28:56.381 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.374+0000 7fced3a8ee40 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x5584b09bce30 con 0x5584b0a633e0 2023-10-05T19:28:56.382 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.374+0000 7fced15c3700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5584b0a633e0 0x5584b0b14570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-05T19:28:56.382 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.374+0000 7fced15c3700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5584b0a633e0 0x5584b0b14570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.118:3300/0 says I am v2:172.21.15.167:53356/0 (socket says 172.21.15.167:53356) 2023-10-05T19:28:56.382 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.374+0000 7fced15c3700 1 -- 172.21.15.167:0/369551947 learned_addr learned my addr 172.21.15.167:0/369551947 (peer_addr_for_me v2:172.21.15.167:0/0) 2023-10-05T19:28:56.382 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.378+0000 7fced15c3700 1 -- 172.21.15.167:0/369551947 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcebc00a180 con 0x5584b0a633e0 2023-10-05T19:28:56.383 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.378+0000 7fced15c3700 1 --2- 172.21.15.167:0/369551947 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5584b0a633e0 0x5584b0b14570 secure :-1 s=READY pgs=544 cs=0 l=1 rev1=1 crypto rx=0x7fcec4001cf0 tx=0x7fcec40018a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b6b75aa0a4631c81 server_cookie=0 in_seq=0 out_seq=0 2023-10-05T19:28:56.383 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.378+0000 7fcecaffd700 1 -- 172.21.15.167:0/369551947 <== mon.0 v2:172.21.15.118:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fcec4014070 con 0x5584b0a633e0 2023-10-05T19:28:56.384 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.378+0000 7fcecaffd700 1 -- 172.21.15.167:0/369551947 <== mon.0 v2:172.21.15.118:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcec400be10 con 0x5584b0a633e0 2023-10-05T19:28:56.384 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.378+0000 7fcecaffd700 1 -- 172.21.15.167:0/369551947 <== mon.0 v2:172.21.15.118:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fcec400f590 con 0x5584b0a633e0 2023-10-05T19:28:56.384 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.378+0000 7fced3a8ee40 1 -- 172.21.15.167:0/369551947 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5584b0a633e0 msgr2=0x5584b0b14570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-05T19:28:56.384 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.378+0000 7fced3a8ee40 1 --2- 172.21.15.167:0/369551947 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5584b0a633e0 0x5584b0b14570 secure :-1 s=READY pgs=544 cs=0 l=1 rev1=1 crypto rx=0x7fcec4001cf0 tx=0x7fcec40018a0 comp rx=0 tx=0).stop 2023-10-05T19:28:56.384 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.378+0000 7fced3a8ee40 1 -- 172.21.15.167:0/369551947 shutdown_connections 2023-10-05T19:28:56.384 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.378+0000 7fced3a8ee40 1 --2- 172.21.15.167:0/369551947 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5584b0a633e0 0x5584b0b14570 unknown :-1 s=CLOSED pgs=544 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-05T19:28:56.384 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.378+0000 7fced3a8ee40 1 -- 172.21.15.167:0/369551947 >> 172.21.15.167:0/369551947 conn(0x5584b0ae9750 msgr2=0x5584b0b10130 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-05T19:28:56.384 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.378+0000 7fced3a8ee40 1 -- 172.21.15.167:0/369551947 shutdown_connections 2023-10-05T19:28:56.384 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.378+0000 7fced3a8ee40 1 -- 172.21.15.167:0/369551947 wait complete. 2023-10-05T19:28:56.384 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.378+0000 7fced3a8ee40 1 Processor -- start 2023-10-05T19:28:56.384 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.378+0000 7fced3a8ee40 1 -- start start 2023-10-05T19:28:56.384 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.378+0000 7fced3a8ee40 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5584b0a633e0 0x5584b0b1b340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-05T19:28:56.385 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.378+0000 7fced3a8ee40 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x5584b098fed0 con 0x5584b0a633e0 2023-10-05T19:28:56.386 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.378+0000 7fced15c3700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5584b0a633e0 0x5584b0b1b340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-05T19:28:56.386 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.378+0000 7fced15c3700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5584b0a633e0 0x5584b0b1b340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.118:3300/0 says I am v2:172.21.15.167:53358/0 (socket says 172.21.15.167:53358) 2023-10-05T19:28:56.386 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.378+0000 7fced15c3700 1 -- 172.21.15.167:0/3622536379 learned_addr learned my addr 172.21.15.167:0/3622536379 (peer_addr_for_me v2:172.21.15.167:0/0) 2023-10-05T19:28:56.386 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.378+0000 7fced15c3700 1 -- 172.21.15.167:0/3622536379 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5584b0a3bbd0 con 0x5584b0a633e0 2023-10-05T19:28:56.386 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.378+0000 7fced15c3700 1 --2- 172.21.15.167:0/3622536379 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5584b0a633e0 0x5584b0b1b340 secure :-1 s=READY pgs=545 cs=0 l=1 rev1=1 crypto rx=0x7fcec40018e0 tx=0x7fcec4016000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-05T19:28:56.386 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.378+0000 7fcec97fa700 1 -- 172.21.15.167:0/3622536379 <== mon.0 v2:172.21.15.118:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fcec401c670 con 0x5584b0a633e0 2023-10-05T19:28:56.386 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.382+0000 7fcec97fa700 1 -- 172.21.15.167:0/3622536379 <== mon.0 v2:172.21.15.118:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcec401c7f0 con 0x5584b0a633e0 2023-10-05T19:28:56.386 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.382+0000 7fced3a8ee40 1 -- 172.21.15.167:0/3622536379 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x5584b0b1b880 con 0x5584b0a633e0 2023-10-05T19:28:56.387 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.382+0000 7fcec97fa700 1 -- 172.21.15.167:0/3622536379 <== mon.0 v2:172.21.15.118:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fcec401cae0 con 0x5584b0a633e0 2023-10-05T19:28:56.387 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.382+0000 7fced3a8ee40 1 -- 172.21.15.167:0/3622536379 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x5584b0b1bcc0 con 0x5584b0a633e0 2023-10-05T19:28:56.387 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.382+0000 7fced3a8ee40 20 rados->read ofs=0 len=0 2023-10-05T19:28:56.387 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.382+0000 7fcec97fa700 1 -- 172.21.15.167:0/3622536379 <== mon.0 v2:172.21.15.118:3300/0 4 ==== mgrmap(e 5) v1 ==== 40329+0+0 (secure 0 0 0) 0x7fcec4023020 con 0x5584b0a633e0 2023-10-05T19:28:56.388 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.382+0000 7fcec97fa700 1 --2- 172.21.15.167:0/3622536379 >> [v2:172.21.15.167:6824/29889,v1:172.21.15.167:6825/29889] conn(0x7fceb4030c50 0x7fceb4033100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-05T19:28:56.388 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.382+0000 7fcec97fa700 1 -- 172.21.15.167:0/3622536379 <== mon.0 v2:172.21.15.118:3300/0 5 ==== osd_map(32..32 src has 1..32) v4 ==== 6260+0+0 (secure 0 0 0) 0x7fcec4008030 con 0x5584b0a633e0 2023-10-05T19:28:56.388 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.382+0000 7fced3a8ee40 1 --2- 172.21.15.167:0/3622536379 >> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] conn(0x5584b0af1110 0x5584b0b26e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-05T19:28:56.389 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.382+0000 7fced0dc2700 1 --2- 172.21.15.167:0/3622536379 >> [v2:172.21.15.167:6824/29889,v1:172.21.15.167:6825/29889] conn(0x7fceb4030c50 0x7fceb4033100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-05T19:28:56.389 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.382+0000 7fced3a8ee40 1 -- 172.21.15.167:0/3622536379 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:1 5.12 5:49953fa1:::default.realm:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32) v8 -- 0x5584b0b274e0 con 0x5584b0af1110 2023-10-05T19:28:56.389 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.382+0000 7fced1dc4700 1 --2- 172.21.15.167:0/3622536379 >> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] conn(0x5584b0af1110 0x5584b0b26e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-05T19:28:56.389 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.382+0000 7fced0dc2700 1 --2- 172.21.15.167:0/3622536379 >> [v2:172.21.15.167:6824/29889,v1:172.21.15.167:6825/29889] conn(0x7fceb4030c50 0x7fceb4033100 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7fcebc000f80 tx=0x7fcebc013000 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-05T19:28:56.389 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.382+0000 7fced1dc4700 1 --2- 172.21.15.167:0/3622536379 >> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] conn(0x5584b0af1110 0x5584b0b26e30 crc :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-05T19:28:56.390 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.386+0000 7fced1dc4700 1 -- 172.21.15.167:0/3622536379 <== osd.1 v2:172.21.15.167:6800/30035 1 ==== osd_op_reply(1 default.realm [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 ==== 157+0+0 (crc 0 0 0) 0x7fcecc00a500 con 0x5584b0af1110 2023-10-05T19:28:56.390 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.386+0000 7fced3a8ee40 20 rados_obj.operate() r=-2 bl.length=0 2023-10-05T19:28:56.391 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.386+0000 7fced3a8ee40 20 realm 2023-10-05T19:28:56.391 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.386+0000 7fced3a8ee40 20 rados->read ofs=0 len=0 2023-10-05T19:28:56.391 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.386+0000 7fced3a8ee40 1 -- 172.21.15.167:0/3622536379 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:2 5.12 5:49953fa1:::default.realm:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32) v8 -- 0x5584b0b24490 con 0x5584b0af1110 2023-10-05T19:28:56.391 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.386+0000 7fced1dc4700 1 -- 172.21.15.167:0/3622536379 <== osd.1 v2:172.21.15.167:6800/30035 2 ==== osd_op_reply(2 default.realm [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 ==== 157+0+0 (crc 0 0 0) 0x7fcecc00a500 con 0x5584b0af1110 2023-10-05T19:28:56.391 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.386+0000 7fced3a8ee40 20 rados_obj.operate() r=-2 bl.length=0 2023-10-05T19:28:56.391 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.386+0000 7fced3a8ee40 4 RGWPeriod::init failed to init realm id : (2) No such file or directory 2023-10-05T19:28:56.391 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.386+0000 7fced3a8ee40 20 rados->read ofs=0 len=0 2023-10-05T19:28:56.391 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.386+0000 7fced3a8ee40 1 -- 172.21.15.167:0/3622536379 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:3 5.12 5:49953fa1:::default.realm:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32) v8 -- 0x5584b0b27fc0 con 0x5584b0af1110 2023-10-05T19:28:56.392 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.386+0000 7fced1dc4700 1 -- 172.21.15.167:0/3622536379 <== osd.1 v2:172.21.15.167:6800/30035 3 ==== osd_op_reply(3 default.realm [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 ==== 157+0+0 (crc 0 0 0) 0x7fcecc00a500 con 0x5584b0af1110 2023-10-05T19:28:56.393 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.386+0000 7fced3a8ee40 20 rados_obj.operate() r=-2 bl.length=0 2023-10-05T19:28:56.393 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.386+0000 7fced3a8ee40 20 rados->read ofs=0 len=0 2023-10-05T19:28:56.393 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.386+0000 7fced3a8ee40 1 -- 172.21.15.167:0/3622536379 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:4 5.f 5:f43b6ece:::zone_names.default:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32) v8 -- 0x5584b0b27bd0 con 0x5584b0af1110 2023-10-05T19:28:56.393 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.386+0000 7fced1dc4700 1 -- 172.21.15.167:0/3622536379 <== osd.1 v2:172.21.15.167:6800/30035 4 ==== osd_op_reply(4 zone_names.default [read 0~46 out=46b] v0'0 uv1 ondisk = 0) v8 ==== 162+0+46 (crc 0 0 0) 0x7fcecc00a500 con 0x5584b0af1110 2023-10-05T19:28:56.393 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.386+0000 7fced3a8ee40 20 rados_obj.operate() r=0 bl.length=46 2023-10-05T19:28:56.393 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.386+0000 7fced3a8ee40 20 rados->read ofs=0 len=0 2023-10-05T19:28:56.393 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.386+0000 7fced3a8ee40 1 -- 172.21.15.167:0/3622536379 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:5 5.1d 5:b8374eb4:::zone_info.9a9df965-6612-45d4-a45f-6e3c889bacc2:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32) v8 -- 0x5584b0b28e50 con 0x5584b0af1110 2023-10-05T19:28:56.393 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.386+0000 7fced1dc4700 1 -- 172.21.15.167:0/3622536379 <== osd.1 v2:172.21.15.167:6800/30035 5 ==== osd_op_reply(5 zone_info.9a9df965-6612-45d4-a45f-6e3c889bacc2 [read 0~874 out=874b] v0'0 uv2 ondisk = 0) v8 ==== 190+0+874 (crc 0 0 0) 0x7fcecc00a500 con 0x5584b0af1110 2023-10-05T19:28:56.393 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.386+0000 7fced3a8ee40 20 rados_obj.operate() r=0 bl.length=874 2023-10-05T19:28:56.393 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.386+0000 7fced3a8ee40 20 searching for the correct realm 2023-10-05T19:28:56.393 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.386+0000 7fced3a8ee40 1 --2- 172.21.15.167:0/3622536379 >> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] conn(0x5584b0b2ac10 0x5584b0b2cff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-05T19:28:56.394 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.386+0000 7fced3a8ee40 1 -- 172.21.15.167:0/3622536379 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:6 5.0 5:00000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x5584b0b2d530 con 0x5584b0b2ac10 2023-10-05T19:28:56.394 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.386+0000 7fced15c3700 1 --2- 172.21.15.167:0/3622536379 >> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] conn(0x5584b0b2ac10 0x5584b0b2cff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-05T19:28:56.394 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.386+0000 7fced15c3700 1 --2- 172.21.15.167:0/3622536379 >> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] conn(0x5584b0b2ac10 0x5584b0b2cff0 crc :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-05T19:28:56.395 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.390+0000 7fced15c3700 1 -- 172.21.15.167:0/3622536379 <== osd.3 v2:172.21.15.167:6808/30036 1 ==== osd_op_reply(6 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fcec4016070 con 0x5584b0b2ac10 2023-10-05T19:28:56.395 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.390+0000 7fced15c3700 1 --2- 172.21.15.167:0/3622536379 >> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] conn(0x7fcec401a070 0x7fcec4043760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-05T19:28:56.395 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.390+0000 7fced15c3700 1 -- 172.21.15.167:0/3622536379 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:7 5.10 5:08000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x5584b0b2d530 con 0x7fcec401a070 2023-10-05T19:28:56.395 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.390+0000 7fced0dc2700 1 --2- 172.21.15.167:0/3622536379 >> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] conn(0x7fcec401a070 0x7fcec4043760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-05T19:28:56.396 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.390+0000 7fced0dc2700 1 --2- 172.21.15.167:0/3622536379 >> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] conn(0x7fcec401a070 0x7fcec4043760 crc :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-05T19:28:56.396 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.390+0000 7fced0dc2700 1 -- 172.21.15.167:0/3622536379 <== osd.2 v2:172.21.15.167:6816/30037 1 ==== osd_op_reply(7 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fcebc017050 con 0x7fcec401a070 2023-10-05T19:28:56.396 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.390+0000 7fced0dc2700 1 --2- 172.21.15.167:0/3622536379 >> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] conn(0x7fcebc010070 0x7fcebc01c430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-05T19:28:56.396 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.390+0000 7fced0dc2700 1 -- 172.21.15.167:0/3622536379 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:8 5.8 5:10000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x5584b0b2d530 con 0x7fcebc010070 2023-10-05T19:28:56.396 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.390+0000 7fced1dc4700 1 --2- 172.21.15.167:0/3622536379 >> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] conn(0x7fcebc010070 0x7fcebc01c430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-05T19:28:56.397 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.390+0000 7fced1dc4700 1 --2- 172.21.15.167:0/3622536379 >> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] conn(0x7fcebc010070 0x7fcebc01c430 crc :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-05T19:28:56.397 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.394+0000 7fced1dc4700 1 -- 172.21.15.167:0/3622536379 <== osd.0 v2:172.21.15.118:6800/27690 1 ==== osd_op_reply(8 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fcecc014040 con 0x7fcebc010070 2023-10-05T19:28:56.397 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.394+0000 7fced1dc4700 1 -- 172.21.15.167:0/3622536379 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:9 5.18 5:18000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x5584b0b2d530 con 0x5584b0af1110 2023-10-05T19:28:56.398 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.394+0000 7fced1dc4700 1 -- 172.21.15.167:0/3622536379 <== osd.1 v2:172.21.15.167:6800/30035 6 ==== osd_op_reply(9 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fcecc014040 con 0x5584b0af1110 2023-10-05T19:28:56.398 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.394+0000 7fced1dc4700 1 -- 172.21.15.167:0/3622536379 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:10 5.4 5:20000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x5584b0b2d530 con 0x5584b0af1110 2023-10-05T19:28:56.399 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.394+0000 7fced1dc4700 1 -- 172.21.15.167:0/3622536379 <== osd.1 v2:172.21.15.167:6800/30035 7 ==== osd_op_reply(10 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fcecc014040 con 0x5584b0af1110 2023-10-05T19:28:56.399 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.394+0000 7fced1dc4700 1 -- 172.21.15.167:0/3622536379 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:11 5.14 5:28000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x5584b0b2d530 con 0x7fcebc010070 2023-10-05T19:28:56.400 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.394+0000 7fced1dc4700 1 -- 172.21.15.167:0/3622536379 <== osd.0 v2:172.21.15.118:6800/27690 2 ==== osd_op_reply(11 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fcecc014040 con 0x7fcebc010070 2023-10-05T19:28:56.400 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.394+0000 7fced1dc4700 1 -- 172.21.15.167:0/3622536379 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:12 5.c 5:30000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x5584b0b2d530 con 0x7fcebc010070 2023-10-05T19:28:56.400 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.394+0000 7fced1dc4700 1 -- 172.21.15.167:0/3622536379 <== osd.0 v2:172.21.15.118:6800/27690 3 ==== osd_op_reply(12 [pgnls start_epoch 32 out=112b] v23'1 uv1 ondisk = 1) v8 ==== 144+0+112 (crc 0 0 0) 0x7fcecc014040 con 0x7fcebc010070 2023-10-05T19:28:56.400 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.394+0000 7fced1dc4700 1 -- 172.21.15.167:0/3622536379 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:13 5.1c 5:38000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x5584b0b2d530 con 0x7fcec401a070 2023-10-05T19:28:56.401 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.394+0000 7fced0dc2700 1 -- 172.21.15.167:0/3622536379 <== osd.2 v2:172.21.15.167:6816/30037 2 ==== osd_op_reply(13 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fcebc017050 con 0x7fcec401a070 2023-10-05T19:28:56.401 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.394+0000 7fced0dc2700 1 -- 172.21.15.167:0/3622536379 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:14 5.2 5:40000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x5584b0b2d530 con 0x5584b0b2ac10 2023-10-05T19:28:56.402 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.398+0000 7fced15c3700 1 -- 172.21.15.167:0/3622536379 <== osd.3 v2:172.21.15.167:6808/30036 2 ==== osd_op_reply(14 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fcec4016070 con 0x5584b0b2ac10 2023-10-05T19:28:56.402 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.398+0000 7fced15c3700 1 -- 172.21.15.167:0/3622536379 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:15 5.12 5:48000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x5584b0b2d530 con 0x5584b0af1110 2023-10-05T19:28:56.402 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.398+0000 7fced1dc4700 1 -- 172.21.15.167:0/3622536379 <== osd.1 v2:172.21.15.167:6800/30035 8 ==== osd_op_reply(15 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fcecc014040 con 0x5584b0af1110 2023-10-05T19:28:56.403 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.398+0000 7fced1dc4700 1 -- 172.21.15.167:0/3622536379 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:16 5.a 5:50000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x5584b0b2d530 con 0x7fcebc010070 2023-10-05T19:28:56.407 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.398+0000 7fced1dc4700 1 -- 172.21.15.167:0/3622536379 <== osd.0 v2:172.21.15.118:6800/27690 4 ==== osd_op_reply(16 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fcecc022040 con 0x7fcebc010070 2023-10-05T19:28:56.407 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.398+0000 7fced1dc4700 1 -- 172.21.15.167:0/3622536379 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:17 5.1a 5:58000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x5584b0b2d530 con 0x5584b0af1110 2023-10-05T19:28:56.407 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.398+0000 7fced1dc4700 1 -- 172.21.15.167:0/3622536379 <== osd.1 v2:172.21.15.167:6800/30035 9 ==== osd_op_reply(17 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fcecc022040 con 0x5584b0af1110 2023-10-05T19:28:56.407 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.398+0000 7fced1dc4700 1 -- 172.21.15.167:0/3622536379 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:18 5.6 5:60000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x5584b0b2d530 con 0x7fcec401a070 2023-10-05T19:28:56.408 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.398+0000 7fced0dc2700 1 -- 172.21.15.167:0/3622536379 <== osd.2 v2:172.21.15.167:6816/30037 3 ==== osd_op_reply(18 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fcebc017050 con 0x7fcec401a070 2023-10-05T19:28:56.408 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.398+0000 7fced0dc2700 1 -- 172.21.15.167:0/3622536379 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:19 5.16 5:68000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x5584b0b2d530 con 0x5584b0b2ac10 2023-10-05T19:28:56.408 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.402+0000 7fced15c3700 1 -- 172.21.15.167:0/3622536379 <== osd.3 v2:172.21.15.167:6808/30036 3 ==== osd_op_reply(19 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fcec4016070 con 0x5584b0b2ac10 2023-10-05T19:28:56.408 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.402+0000 7fced15c3700 1 -- 172.21.15.167:0/3622536379 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:20 5.e 5:70000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x5584b0b2d530 con 0x5584b0b2ac10 2023-10-05T19:28:56.408 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.402+0000 7fced15c3700 1 -- 172.21.15.167:0/3622536379 <== osd.3 v2:172.21.15.167:6808/30036 4 ==== osd_op_reply(20 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fcec4016070 con 0x5584b0b2ac10 2023-10-05T19:28:56.408 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.402+0000 7fced15c3700 1 -- 172.21.15.167:0/3622536379 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:21 5.1e 5:78000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x5584b0b2d530 con 0x7fcebc010070 2023-10-05T19:28:56.408 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.402+0000 7fced1dc4700 1 -- 172.21.15.167:0/3622536379 <== osd.0 v2:172.21.15.118:6800/27690 5 ==== osd_op_reply(21 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fcecc022040 con 0x7fcebc010070 2023-10-05T19:28:56.408 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.402+0000 7fced1dc4700 1 -- 172.21.15.167:0/3622536379 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:22 5.1 5:80000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x5584b0b2d530 con 0x5584b0b2ac10 2023-10-05T19:28:56.409 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.402+0000 7fced15c3700 1 -- 172.21.15.167:0/3622536379 <== osd.3 v2:172.21.15.167:6808/30036 5 ==== osd_op_reply(22 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fcec4039040 con 0x5584b0b2ac10 2023-10-05T19:28:56.409 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.402+0000 7fced15c3700 1 -- 172.21.15.167:0/3622536379 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:23 5.11 5:88000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x5584b0b2d530 con 0x5584b0b2ac10 2023-10-05T19:28:56.409 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.402+0000 7fced15c3700 1 -- 172.21.15.167:0/3622536379 <== osd.3 v2:172.21.15.167:6808/30036 6 ==== osd_op_reply(23 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fcec4039040 con 0x5584b0b2ac10 2023-10-05T19:28:56.409 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.402+0000 7fced15c3700 1 -- 172.21.15.167:0/3622536379 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:24 5.9 5:90000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x5584b0b2d530 con 0x5584b0af1110 2023-10-05T19:28:56.409 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.406+0000 7fced1dc4700 1 -- 172.21.15.167:0/3622536379 <== osd.1 v2:172.21.15.167:6800/30035 10 ==== osd_op_reply(24 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fcecc022040 con 0x5584b0af1110 2023-10-05T19:28:56.410 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.406+0000 7fced1dc4700 1 -- 172.21.15.167:0/3622536379 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:25 5.19 5:98000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x5584b0b2d530 con 0x5584b0af1110 2023-10-05T19:28:56.410 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.406+0000 7fced1dc4700 1 -- 172.21.15.167:0/3622536379 <== osd.1 v2:172.21.15.167:6800/30035 11 ==== osd_op_reply(25 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fcecc022040 con 0x5584b0af1110 2023-10-05T19:28:56.410 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.406+0000 7fced1dc4700 1 -- 172.21.15.167:0/3622536379 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:26 5.5 5:a0000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x5584b0b2d530 con 0x7fcebc010070 2023-10-05T19:28:56.411 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.406+0000 7fced1dc4700 1 -- 172.21.15.167:0/3622536379 <== osd.0 v2:172.21.15.118:6800/27690 6 ==== osd_op_reply(26 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fcecc022040 con 0x7fcebc010070 2023-10-05T19:28:56.411 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.406+0000 7fced1dc4700 1 -- 172.21.15.167:0/3622536379 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:27 5.15 5:a8000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x5584b0b2d530 con 0x5584b0b2ac10 2023-10-05T19:28:56.412 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.406+0000 7fced15c3700 1 -- 172.21.15.167:0/3622536379 <== osd.3 v2:172.21.15.167:6808/30036 7 ==== osd_op_reply(27 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fcec4039040 con 0x5584b0b2ac10 2023-10-05T19:28:56.412 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.406+0000 7fced15c3700 1 -- 172.21.15.167:0/3622536379 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:28 5.d 5:b0000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x5584b0b2d530 con 0x7fcec401a070 2023-10-05T19:28:56.413 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.406+0000 7fced0dc2700 1 -- 172.21.15.167:0/3622536379 <== osd.2 v2:172.21.15.167:6816/30037 4 ==== osd_op_reply(28 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fcebc017050 con 0x7fcec401a070 2023-10-05T19:28:56.413 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.406+0000 7fced0dc2700 1 -- 172.21.15.167:0/3622536379 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:29 5.1d 5:b8000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x5584b0b2d530 con 0x5584b0af1110 2023-10-05T19:28:56.413 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.410+0000 7fced1dc4700 1 -- 172.21.15.167:0/3622536379 <== osd.1 v2:172.21.15.167:6800/30035 12 ==== osd_op_reply(29 [pgnls start_epoch 32 out=107b] v30'2 uv2 ondisk = 1) v8 ==== 144+0+107 (crc 0 0 0) 0x7fcecc022040 con 0x5584b0af1110 2023-10-05T19:28:56.413 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.410+0000 7fced1dc4700 1 -- 172.21.15.167:0/3622536379 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:30 5.3 5:c0000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x5584b0b2d530 con 0x7fcebc010070 2023-10-05T19:28:56.414 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.410+0000 7fced1dc4700 1 -- 172.21.15.167:0/3622536379 <== osd.0 v2:172.21.15.118:6800/27690 7 ==== osd_op_reply(30 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fcecc022040 con 0x7fcebc010070 2023-10-05T19:28:56.415 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.410+0000 7fced1dc4700 1 -- 172.21.15.167:0/3622536379 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:31 5.13 5:c8000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x5584b0b2d530 con 0x5584b0b2ac10 2023-10-05T19:28:56.415 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.410+0000 7fced15c3700 1 -- 172.21.15.167:0/3622536379 <== osd.3 v2:172.21.15.167:6808/30036 8 ==== osd_op_reply(31 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fcec4039040 con 0x5584b0b2ac10 2023-10-05T19:28:56.415 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.410+0000 7fced15c3700 1 -- 172.21.15.167:0/3622536379 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:32 5.b 5:d0000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x5584b0b2d530 con 0x7fcebc010070 2023-10-05T19:28:56.416 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.410+0000 7fced1dc4700 1 -- 172.21.15.167:0/3622536379 <== osd.0 v2:172.21.15.118:6800/27690 8 ==== osd_op_reply(32 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fcecc022040 con 0x7fcebc010070 2023-10-05T19:28:56.416 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.410+0000 7fced1dc4700 1 -- 172.21.15.167:0/3622536379 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:33 5.1b 5:d8000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x5584b0b2d530 con 0x7fcec401a070 2023-10-05T19:28:56.417 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.410+0000 7fced0dc2700 1 -- 172.21.15.167:0/3622536379 <== osd.2 v2:172.21.15.167:6816/30037 5 ==== osd_op_reply(33 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fcebc017050 con 0x7fcec401a070 2023-10-05T19:28:56.417 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.410+0000 7fced0dc2700 1 -- 172.21.15.167:0/3622536379 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:34 5.7 5:e0000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x5584b0b2d530 con 0x5584b0b2ac10 2023-10-05T19:28:56.418 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.414+0000 7fced15c3700 1 -- 172.21.15.167:0/3622536379 <== osd.3 v2:172.21.15.167:6808/30036 9 ==== osd_op_reply(34 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fcec4039040 con 0x5584b0b2ac10 2023-10-05T19:28:56.418 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.414+0000 7fced15c3700 1 -- 172.21.15.167:0/3622536379 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:35 5.17 5:e8000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x5584b0b2d530 con 0x7fcebc010070 2023-10-05T19:28:56.419 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.414+0000 7fced1dc4700 1 -- 172.21.15.167:0/3622536379 <== osd.0 v2:172.21.15.118:6800/27690 9 ==== osd_op_reply(35 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fcecc022040 con 0x7fcebc010070 2023-10-05T19:28:56.419 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.414+0000 7fced1dc4700 1 -- 172.21.15.167:0/3622536379 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:36 5.f 5:f0000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x5584b0b2d530 con 0x5584b0af1110 2023-10-05T19:28:56.420 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.414+0000 7fced1dc4700 1 -- 172.21.15.167:0/3622536379 <== osd.1 v2:172.21.15.167:6800/30035 13 ==== osd_op_reply(36 [pgnls start_epoch 32 out=115b] v23'2 uv2 ondisk = 1) v8 ==== 144+0+115 (crc 0 0 0) 0x7fcecc02f080 con 0x5584b0af1110 2023-10-05T19:28:56.420 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.414+0000 7fced1dc4700 1 -- 172.21.15.167:0/3622536379 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:37 5.1f 5:f8000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x5584b0b2d530 con 0x7fcec401a070 2023-10-05T19:28:56.421 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.414+0000 7fced0dc2700 1 -- 172.21.15.167:0/3622536379 <== osd.2 v2:172.21.15.167:6816/30037 6 ==== osd_op_reply(37 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fcebc017050 con 0x7fcec401a070 2023-10-05T19:28:56.421 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.414+0000 7fced3a8ee40 20 RGWRados::pool_iterate: got zonegroup_info.4f1f719d-9648-417b-b10c-e5d939825419 2023-10-05T19:28:56.421 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.414+0000 7fced3a8ee40 20 RGWRados::pool_iterate: got zone_info.9a9df965-6612-45d4-a45f-6e3c889bacc2 2023-10-05T19:28:56.421 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.414+0000 7fced3a8ee40 20 RGWRados::pool_iterate: got zone_names.default 2023-10-05T19:28:56.421 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.414+0000 7fced3a8ee40 20 RGWRados::pool_iterate: got zonegroups_names.default 2023-10-05T19:28:56.421 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.414+0000 7fced3a8ee40 20 rados->read ofs=0 len=0 2023-10-05T19:28:56.421 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.414+0000 7fced3a8ee40 1 -- 172.21.15.167:0/3622536379 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:38 5.12 5:49953fa1:::default.realm:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32) v8 -- 0x5584b0b299c0 con 0x5584b0af1110 2023-10-05T19:28:56.422 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.414+0000 7fced1dc4700 1 -- 172.21.15.167:0/3622536379 <== osd.1 v2:172.21.15.167:6800/30035 14 ==== osd_op_reply(38 default.realm [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 ==== 157+0+0 (crc 0 0 0) 0x7fcecc02f080 con 0x5584b0af1110 2023-10-05T19:28:56.422 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.414+0000 7fced3a8ee40 20 rados_obj.operate() r=-2 bl.length=0 2023-10-05T19:28:56.422 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.414+0000 7fced3a8ee40 20 rados->read ofs=0 len=0 2023-10-05T19:28:56.422 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.414+0000 7fced3a8ee40 1 -- 172.21.15.167:0/3622536379 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:39 5.f 5:f4c53578:::zonegroups_names.default:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32) v8 -- 0x5584b0b2a2b0 con 0x5584b0af1110 2023-10-05T19:28:56.422 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.418+0000 7fced1dc4700 1 -- 172.21.15.167:0/3622536379 <== osd.1 v2:172.21.15.167:6800/30035 15 ==== osd_op_reply(39 zonegroups_names.default [read 0~46 out=46b] v0'0 uv2 ondisk = 0) v8 ==== 168+0+46 (crc 0 0 0) 0x7fcecc02f080 con 0x5584b0af1110 2023-10-05T19:28:56.423 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.418+0000 7fced3a8ee40 20 rados_obj.operate() r=0 bl.length=46 2023-10-05T19:28:56.423 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.418+0000 7fced3a8ee40 20 rados->read ofs=0 len=0 2023-10-05T19:28:56.423 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.418+0000 7fced3a8ee40 1 -- 172.21.15.167:0/3622536379 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:40 5.c 5:30d299a5:::zonegroup_info.4f1f719d-9648-417b-b10c-e5d939825419:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32) v8 -- 0x5584b0b2d970 con 0x7fcebc010070 2023-10-05T19:28:56.424 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.418+0000 7fced1dc4700 1 -- 172.21.15.167:0/3622536379 <== osd.0 v2:172.21.15.118:6800/27690 10 ==== osd_op_reply(40 zonegroup_info.4f1f719d-9648-417b-b10c-e5d939825419 [read 0~362 out=362b] v0'0 uv1 ondisk = 0) v8 ==== 195+0+362 (crc 0 0 0) 0x7fcecc02f080 con 0x7fcebc010070 2023-10-05T19:28:56.424 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.418+0000 7fced3a8ee40 20 rados_obj.operate() r=0 bl.length=362 2023-10-05T19:28:56.424 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.418+0000 7fced3a8ee40 20 rados->read ofs=0 len=0 2023-10-05T19:28:56.424 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.418+0000 7fced3a8ee40 1 -- 172.21.15.167:0/3622536379 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:41 5.13 5:cbbf6c60:::converted:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32) v8 -- 0x5584b0b2d970 con 0x5584b0b2ac10 2023-10-05T19:28:56.424 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.418+0000 7fced15c3700 1 -- 172.21.15.167:0/3622536379 <== osd.3 v2:172.21.15.167:6808/30036 10 ==== osd_op_reply(41 converted [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 ==== 153+0+0 (crc 0 0 0) 0x7fcec4039040 con 0x5584b0b2ac10 2023-10-05T19:28:56.424 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.418+0000 7fced3a8ee40 20 rados_obj.operate() r=-2 bl.length=0 2023-10-05T19:28:56.424 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.418+0000 7fced3a8ee40 20 rados->read ofs=0 len=0 2023-10-05T19:28:56.424 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.418+0000 7fced3a8ee40 1 -- 172.21.15.167:0/3622536379 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:42 5.12 5:49953fa1:::default.realm:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32) v8 -- 0x5584b0b2dd20 con 0x5584b0af1110 2023-10-05T19:28:56.425 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.418+0000 7fced1dc4700 1 -- 172.21.15.167:0/3622536379 <== osd.1 v2:172.21.15.167:6800/30035 16 ==== osd_op_reply(42 default.realm [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 ==== 157+0+0 (crc 0 0 0) 0x7fcecc02f080 con 0x5584b0af1110 2023-10-05T19:28:56.425 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.418+0000 7fced3a8ee40 20 rados_obj.operate() r=-2 bl.length=0 2023-10-05T19:28:56.425 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.418+0000 7fced3a8ee40 20 rados->read ofs=0 len=0 2023-10-05T19:28:56.425 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.418+0000 7fced3a8ee40 1 -- 172.21.15.167:0/3622536379 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:43 5.f 5:f4c53578:::zonegroups_names.default:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32) v8 -- 0x5584b0b2e120 con 0x5584b0af1110 2023-10-05T19:28:56.426 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.418+0000 7fced1dc4700 1 -- 172.21.15.167:0/3622536379 <== osd.1 v2:172.21.15.167:6800/30035 17 ==== osd_op_reply(43 zonegroups_names.default [read 0~46 out=46b] v0'0 uv2 ondisk = 0) v8 ==== 168+0+46 (crc 0 0 0) 0x7fcecc02f080 con 0x5584b0af1110 2023-10-05T19:28:56.426 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.418+0000 7fced3a8ee40 20 rados_obj.operate() r=0 bl.length=46 2023-10-05T19:28:56.426 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.418+0000 7fced3a8ee40 1 -- 172.21.15.167:0/3622536379 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:44 5.0 5:00000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x5584b0b2f130 con 0x5584b0b2ac10 2023-10-05T19:28:56.426 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.422+0000 7fced15c3700 1 -- 172.21.15.167:0/3622536379 <== osd.3 v2:172.21.15.167:6808/30036 11 ==== osd_op_reply(44 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fcec4039040 con 0x5584b0b2ac10 2023-10-05T19:28:56.427 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.422+0000 7fced15c3700 1 -- 172.21.15.167:0/3622536379 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:45 5.10 5:08000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x5584b0b2f130 con 0x7fcec401a070 2023-10-05T19:28:56.428 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.422+0000 7fced0dc2700 1 -- 172.21.15.167:0/3622536379 <== osd.2 v2:172.21.15.167:6816/30037 7 ==== osd_op_reply(45 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fcebc017050 con 0x7fcec401a070 2023-10-05T19:28:56.428 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.422+0000 7fced0dc2700 1 -- 172.21.15.167:0/3622536379 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:46 5.8 5:10000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x5584b0b2f130 con 0x7fcebc010070 2023-10-05T19:28:56.429 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.422+0000 7fced1dc4700 1 -- 172.21.15.167:0/3622536379 <== osd.0 v2:172.21.15.118:6800/27690 11 ==== osd_op_reply(46 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fcecc02f080 con 0x7fcebc010070 2023-10-05T19:28:56.429 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.422+0000 7fced1dc4700 1 -- 172.21.15.167:0/3622536379 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:47 5.18 5:18000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x5584b0b2a2b0 con 0x5584b0af1110 2023-10-05T19:28:56.429 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.426+0000 7fced1dc4700 1 -- 172.21.15.167:0/3622536379 <== osd.1 v2:172.21.15.167:6800/30035 18 ==== osd_op_reply(47 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fcecc02f080 con 0x5584b0af1110 2023-10-05T19:28:56.430 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.426+0000 7fced1dc4700 1 -- 172.21.15.167:0/3622536379 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:48 5.4 5:20000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x5584b0b2a2b0 con 0x5584b0af1110 2023-10-05T19:28:56.430 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.426+0000 7fced1dc4700 1 -- 172.21.15.167:0/3622536379 <== osd.1 v2:172.21.15.167:6800/30035 19 ==== osd_op_reply(48 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fcecc02f080 con 0x5584b0af1110 2023-10-05T19:28:56.430 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.426+0000 7fced1dc4700 1 -- 172.21.15.167:0/3622536379 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:49 5.14 5:28000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x5584b0b2a2b0 con 0x7fcebc010070 2023-10-05T19:28:56.431 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.426+0000 7fced1dc4700 1 -- 172.21.15.167:0/3622536379 <== osd.0 v2:172.21.15.118:6800/27690 12 ==== osd_op_reply(49 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fcecc02f080 con 0x7fcebc010070 2023-10-05T19:28:56.431 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.426+0000 7fced1dc4700 1 -- 172.21.15.167:0/3622536379 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:50 5.c 5:30000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x5584b0b2a2b0 con 0x7fcebc010070 2023-10-05T19:28:56.432 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.426+0000 7fced1dc4700 1 -- 172.21.15.167:0/3622536379 <== osd.0 v2:172.21.15.118:6800/27690 13 ==== osd_op_reply(50 [pgnls start_epoch 32 out=112b] v23'1 uv1 ondisk = 1) v8 ==== 144+0+112 (crc 0 0 0) 0x7fcecc02f080 con 0x7fcebc010070 2023-10-05T19:28:56.432 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.426+0000 7fced1dc4700 1 -- 172.21.15.167:0/3622536379 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:51 5.1c 5:38000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x5584b0b2a2b0 con 0x7fcec401a070 2023-10-05T19:28:56.433 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.426+0000 7fced0dc2700 1 -- 172.21.15.167:0/3622536379 <== osd.2 v2:172.21.15.167:6816/30037 8 ==== osd_op_reply(51 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fcebc017050 con 0x7fcec401a070 2023-10-05T19:28:56.433 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.426+0000 7fced0dc2700 1 -- 172.21.15.167:0/3622536379 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:52 5.2 5:40000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x5584b0b2a2b0 con 0x5584b0b2ac10 2023-10-05T19:28:56.434 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.430+0000 7fced15c3700 1 -- 172.21.15.167:0/3622536379 <== osd.3 v2:172.21.15.167:6808/30036 12 ==== osd_op_reply(52 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fcec4039040 con 0x5584b0b2ac10 2023-10-05T19:28:56.434 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.430+0000 7fced15c3700 1 -- 172.21.15.167:0/3622536379 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:53 5.12 5:48000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x5584b0b2a2b0 con 0x5584b0af1110 2023-10-05T19:28:56.435 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.430+0000 7fced1dc4700 1 -- 172.21.15.167:0/3622536379 <== osd.1 v2:172.21.15.167:6800/30035 20 ==== osd_op_reply(53 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fcecc02f080 con 0x5584b0af1110 2023-10-05T19:28:56.435 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.430+0000 7fced1dc4700 1 -- 172.21.15.167:0/3622536379 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:54 5.a 5:50000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x5584b0b2a2b0 con 0x7fcebc010070 2023-10-05T19:28:56.436 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.430+0000 7fced1dc4700 1 -- 172.21.15.167:0/3622536379 <== osd.0 v2:172.21.15.118:6800/27690 14 ==== osd_op_reply(54 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fcecc02f080 con 0x7fcebc010070 2023-10-05T19:28:56.436 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.430+0000 7fced1dc4700 1 -- 172.21.15.167:0/3622536379 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:55 5.1a 5:58000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x5584b0b2a2b0 con 0x5584b0af1110 2023-10-05T19:28:56.436 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.430+0000 7fced1dc4700 1 -- 172.21.15.167:0/3622536379 <== osd.1 v2:172.21.15.167:6800/30035 21 ==== osd_op_reply(55 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fcecc02f080 con 0x5584b0af1110 2023-10-05T19:28:56.437 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.430+0000 7fced1dc4700 1 -- 172.21.15.167:0/3622536379 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:56 5.6 5:60000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x5584b0b2a2b0 con 0x7fcec401a070 2023-10-05T19:28:56.437 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.434+0000 7fced0dc2700 1 -- 172.21.15.167:0/3622536379 <== osd.2 v2:172.21.15.167:6816/30037 9 ==== osd_op_reply(56 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fcebc017050 con 0x7fcec401a070 2023-10-05T19:28:56.438 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.434+0000 7fced0dc2700 1 -- 172.21.15.167:0/3622536379 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:57 5.16 5:68000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x5584b0b2a2b0 con 0x5584b0b2ac10 2023-10-05T19:28:56.438 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.434+0000 7fced15c3700 1 -- 172.21.15.167:0/3622536379 <== osd.3 v2:172.21.15.167:6808/30036 13 ==== osd_op_reply(57 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fcec4039040 con 0x5584b0b2ac10 2023-10-05T19:28:56.438 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.434+0000 7fced15c3700 1 -- 172.21.15.167:0/3622536379 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:58 5.e 5:70000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x5584b0b2a2b0 con 0x5584b0b2ac10 2023-10-05T19:28:56.439 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.434+0000 7fced15c3700 1 -- 172.21.15.167:0/3622536379 <== osd.3 v2:172.21.15.167:6808/30036 14 ==== osd_op_reply(58 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fcec4039040 con 0x5584b0b2ac10 2023-10-05T19:28:56.439 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.434+0000 7fced15c3700 1 -- 172.21.15.167:0/3622536379 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:59 5.1e 5:78000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x5584b0b2a2b0 con 0x7fcebc010070 2023-10-05T19:28:56.440 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.434+0000 7fced1dc4700 1 -- 172.21.15.167:0/3622536379 <== osd.0 v2:172.21.15.118:6800/27690 15 ==== osd_op_reply(59 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fcecc02f080 con 0x7fcebc010070 2023-10-05T19:28:56.440 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.434+0000 7fced1dc4700 1 -- 172.21.15.167:0/3622536379 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:60 5.1 5:80000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x5584b0b2a2b0 con 0x5584b0b2ac10 2023-10-05T19:28:56.441 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.434+0000 7fced15c3700 1 -- 172.21.15.167:0/3622536379 <== osd.3 v2:172.21.15.167:6808/30036 15 ==== osd_op_reply(60 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fcec4039040 con 0x5584b0b2ac10 2023-10-05T19:28:56.441 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.434+0000 7fced15c3700 1 -- 172.21.15.167:0/3622536379 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:61 5.11 5:88000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x5584b0b2a2b0 con 0x5584b0b2ac10 2023-10-05T19:28:56.442 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.438+0000 7fced15c3700 1 -- 172.21.15.167:0/3622536379 <== osd.3 v2:172.21.15.167:6808/30036 16 ==== osd_op_reply(61 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fcec4039040 con 0x5584b0b2ac10 2023-10-05T19:28:56.442 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.438+0000 7fced15c3700 1 -- 172.21.15.167:0/3622536379 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:62 5.9 5:90000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x5584b0b2a2b0 con 0x5584b0af1110 2023-10-05T19:28:56.442 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.438+0000 7fced1dc4700 1 -- 172.21.15.167:0/3622536379 <== osd.1 v2:172.21.15.167:6800/30035 22 ==== osd_op_reply(62 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fcecc02f080 con 0x5584b0af1110 2023-10-05T19:28:56.443 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.438+0000 7fced1dc4700 1 -- 172.21.15.167:0/3622536379 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:63 5.19 5:98000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x5584b0b2a2b0 con 0x5584b0af1110 2023-10-05T19:28:56.443 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.438+0000 7fced1dc4700 1 -- 172.21.15.167:0/3622536379 <== osd.1 v2:172.21.15.167:6800/30035 23 ==== osd_op_reply(63 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fcecc02f080 con 0x5584b0af1110 2023-10-05T19:28:56.443 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.438+0000 7fced1dc4700 1 -- 172.21.15.167:0/3622536379 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:64 5.5 5:a0000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x5584b0b2a2b0 con 0x7fcebc010070 2023-10-05T19:28:56.444 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.438+0000 7fced1dc4700 1 -- 172.21.15.167:0/3622536379 <== osd.0 v2:172.21.15.118:6800/27690 16 ==== osd_op_reply(64 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fcecc02f080 con 0x7fcebc010070 2023-10-05T19:28:56.444 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.438+0000 7fced1dc4700 1 -- 172.21.15.167:0/3622536379 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:65 5.15 5:a8000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x5584b0b2a2b0 con 0x5584b0b2ac10 2023-10-05T19:28:56.445 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.438+0000 7fced15c3700 1 -- 172.21.15.167:0/3622536379 <== osd.3 v2:172.21.15.167:6808/30036 17 ==== osd_op_reply(65 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fcec4039040 con 0x5584b0b2ac10 2023-10-05T19:28:56.445 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.438+0000 7fced15c3700 1 -- 172.21.15.167:0/3622536379 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:66 5.d 5:b0000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x5584b0b2a2b0 con 0x7fcec401a070 2023-10-05T19:28:56.446 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.442+0000 7fced0dc2700 1 -- 172.21.15.167:0/3622536379 <== osd.2 v2:172.21.15.167:6816/30037 10 ==== osd_op_reply(66 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fcebc017050 con 0x7fcec401a070 2023-10-05T19:28:56.446 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.442+0000 7fced0dc2700 1 -- 172.21.15.167:0/3622536379 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:67 5.1d 5:b8000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x5584b0b2a2b0 con 0x5584b0af1110 2023-10-05T19:28:56.447 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.442+0000 7fced1dc4700 1 -- 172.21.15.167:0/3622536379 <== osd.1 v2:172.21.15.167:6800/30035 24 ==== osd_op_reply(67 [pgnls start_epoch 32 out=107b] v30'2 uv2 ondisk = 1) v8 ==== 144+0+107 (crc 0 0 0) 0x7fcecc02f080 con 0x5584b0af1110 2023-10-05T19:28:56.447 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.442+0000 7fced1dc4700 1 -- 172.21.15.167:0/3622536379 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:68 5.3 5:c0000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x5584b0b2a2b0 con 0x7fcebc010070 2023-10-05T19:28:56.448 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.442+0000 7fced1dc4700 1 -- 172.21.15.167:0/3622536379 <== osd.0 v2:172.21.15.118:6800/27690 17 ==== osd_op_reply(68 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fcecc02f080 con 0x7fcebc010070 2023-10-05T19:28:56.448 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.442+0000 7fced1dc4700 1 -- 172.21.15.167:0/3622536379 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:69 5.13 5:c8000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x5584b0b2a2b0 con 0x5584b0b2ac10 2023-10-05T19:28:56.449 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.442+0000 7fced15c3700 1 -- 172.21.15.167:0/3622536379 <== osd.3 v2:172.21.15.167:6808/30036 18 ==== osd_op_reply(69 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fcec4039040 con 0x5584b0b2ac10 2023-10-05T19:28:56.449 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.442+0000 7fced15c3700 1 -- 172.21.15.167:0/3622536379 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:70 5.b 5:d0000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x5584b0b2a2b0 con 0x7fcebc010070 2023-10-05T19:28:56.450 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.446+0000 7fced1dc4700 1 -- 172.21.15.167:0/3622536379 <== osd.0 v2:172.21.15.118:6800/27690 18 ==== osd_op_reply(70 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fcecc02f080 con 0x7fcebc010070 2023-10-05T19:28:56.450 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.446+0000 7fced1dc4700 1 -- 172.21.15.167:0/3622536379 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:71 5.1b 5:d8000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x5584b0b2a2b0 con 0x7fcec401a070 2023-10-05T19:28:56.451 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.446+0000 7fced0dc2700 1 -- 172.21.15.167:0/3622536379 <== osd.2 v2:172.21.15.167:6816/30037 11 ==== osd_op_reply(71 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fcebc017050 con 0x7fcec401a070 2023-10-05T19:28:56.451 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.446+0000 7fced0dc2700 1 -- 172.21.15.167:0/3622536379 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:72 5.7 5:e0000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x5584b0b2a2b0 con 0x5584b0b2ac10 2023-10-05T19:28:56.452 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.446+0000 7fced15c3700 1 -- 172.21.15.167:0/3622536379 <== osd.3 v2:172.21.15.167:6808/30036 19 ==== osd_op_reply(72 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fcec4039040 con 0x5584b0b2ac10 2023-10-05T19:28:56.452 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.446+0000 7fced15c3700 1 -- 172.21.15.167:0/3622536379 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:73 5.17 5:e8000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x5584b0b2a2b0 con 0x7fcebc010070 2023-10-05T19:28:56.453 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.446+0000 7fced1dc4700 1 -- 172.21.15.167:0/3622536379 <== osd.0 v2:172.21.15.118:6800/27690 19 ==== osd_op_reply(73 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fcecc02f080 con 0x7fcebc010070 2023-10-05T19:28:56.453 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.446+0000 7fced1dc4700 1 -- 172.21.15.167:0/3622536379 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:74 5.f 5:f0000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x5584b0b2a2b0 con 0x5584b0af1110 2023-10-05T19:28:56.454 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.450+0000 7fced1dc4700 1 -- 172.21.15.167:0/3622536379 <== osd.1 v2:172.21.15.167:6800/30035 25 ==== osd_op_reply(74 [pgnls start_epoch 32 out=115b] v23'2 uv2 ondisk = 1) v8 ==== 144+0+115 (crc 0 0 0) 0x7fcecc02f080 con 0x5584b0af1110 2023-10-05T19:28:56.454 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.450+0000 7fced1dc4700 1 -- 172.21.15.167:0/3622536379 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:75 5.1f 5:f8000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x5584b0b2a2b0 con 0x7fcec401a070 2023-10-05T19:28:56.455 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.450+0000 7fced0dc2700 1 -- 172.21.15.167:0/3622536379 <== osd.2 v2:172.21.15.167:6816/30037 12 ==== osd_op_reply(75 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fcebc017050 con 0x7fcec401a070 2023-10-05T19:28:56.455 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.450+0000 7fced3a8ee40 20 RGWRados::pool_iterate: got zonegroup_info.4f1f719d-9648-417b-b10c-e5d939825419 2023-10-05T19:28:56.455 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.450+0000 7fced3a8ee40 20 RGWRados::pool_iterate: got zone_info.9a9df965-6612-45d4-a45f-6e3c889bacc2 2023-10-05T19:28:56.456 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.450+0000 7fced3a8ee40 20 RGWRados::pool_iterate: got zone_names.default 2023-10-05T19:28:56.456 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.450+0000 7fced3a8ee40 20 RGWRados::pool_iterate: got zonegroups_names.default 2023-10-05T19:28:56.456 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.450+0000 7fced3a8ee40 20 rados->read ofs=0 len=0 2023-10-05T19:28:56.456 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.450+0000 7fced3a8ee40 1 -- 172.21.15.167:0/3622536379 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:76 5.f 5:f43b6ece:::zone_names.default:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32) v8 -- 0x5584b0b2e120 con 0x5584b0af1110 2023-10-05T19:28:56.457 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.450+0000 7fced1dc4700 1 -- 172.21.15.167:0/3622536379 <== osd.1 v2:172.21.15.167:6800/30035 26 ==== osd_op_reply(76 zone_names.default [read 0~46 out=46b] v0'0 uv1 ondisk = 0) v8 ==== 162+0+46 (crc 0 0 0) 0x7fcecc02f080 con 0x5584b0af1110 2023-10-05T19:28:56.457 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.450+0000 7fced3a8ee40 20 rados_obj.operate() r=0 bl.length=46 2023-10-05T19:28:56.457 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.450+0000 7fced3a8ee40 20 rados->read ofs=0 len=0 2023-10-05T19:28:56.457 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.450+0000 7fced3a8ee40 1 -- 172.21.15.167:0/3622536379 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:77 5.1d 5:b8374eb4:::zone_info.9a9df965-6612-45d4-a45f-6e3c889bacc2:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32) v8 -- 0x5584b0b2e4d0 con 0x5584b0af1110 2023-10-05T19:28:56.457 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.450+0000 7fced1dc4700 1 -- 172.21.15.167:0/3622536379 <== osd.1 v2:172.21.15.167:6800/30035 27 ==== osd_op_reply(77 zone_info.9a9df965-6612-45d4-a45f-6e3c889bacc2 [read 0~874 out=874b] v0'0 uv2 ondisk = 0) v8 ==== 190+0+874 (crc 0 0 0) 0x7fcecc02f080 con 0x5584b0af1110 2023-10-05T19:28:56.457 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.450+0000 7fced3a8ee40 20 rados_obj.operate() r=0 bl.length=874 2023-10-05T19:28:56.457 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.450+0000 7fced3a8ee40 20 rados->read ofs=0 len=0 2023-10-05T19:28:56.457 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.450+0000 7fced3a8ee40 1 -- 172.21.15.167:0/3622536379 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:78 5.f 5:f4c53578:::zonegroups_names.default:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32) v8 -- 0x5584b0b2e4d0 con 0x5584b0af1110 2023-10-05T19:28:56.458 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.454+0000 7fced1dc4700 1 -- 172.21.15.167:0/3622536379 <== osd.1 v2:172.21.15.167:6800/30035 28 ==== osd_op_reply(78 zonegroups_names.default [read 0~46 out=46b] v0'0 uv2 ondisk = 0) v8 ==== 168+0+46 (crc 0 0 0) 0x7fcecc02f080 con 0x5584b0af1110 2023-10-05T19:28:56.459 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.454+0000 7fced3a8ee40 20 rados_obj.operate() r=0 bl.length=46 2023-10-05T19:28:56.459 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.454+0000 7fced3a8ee40 20 rados->read ofs=0 len=0 2023-10-05T19:28:56.459 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.454+0000 7fced3a8ee40 1 -- 172.21.15.167:0/3622536379 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:79 5.c 5:30d299a5:::zonegroup_info.4f1f719d-9648-417b-b10c-e5d939825419:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32) v8 -- 0x5584b0b2e900 con 0x7fcebc010070 2023-10-05T19:28:56.459 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.454+0000 7fced1dc4700 1 -- 172.21.15.167:0/3622536379 <== osd.0 v2:172.21.15.118:6800/27690 20 ==== osd_op_reply(79 zonegroup_info.4f1f719d-9648-417b-b10c-e5d939825419 [read 0~362 out=362b] v0'0 uv1 ondisk = 0) v8 ==== 195+0+362 (crc 0 0 0) 0x7fcecc02f080 con 0x7fcebc010070 2023-10-05T19:28:56.459 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.454+0000 7fced3a8ee40 20 rados_obj.operate() r=0 bl.length=362 2023-10-05T19:28:56.459 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.454+0000 7fced3a8ee40 20 rados->read ofs=0 len=0 2023-10-05T19:28:56.459 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.454+0000 7fced3a8ee40 1 -- 172.21.15.167:0/3622536379 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:80 5.3 5:c0ad8cbd:::region_map:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32) v8 -- 0x5584b0b2e900 con 0x7fcebc010070 2023-10-05T19:28:56.460 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.454+0000 7fced1dc4700 1 -- 172.21.15.167:0/3622536379 <== osd.0 v2:172.21.15.118:6800/27690 21 ==== osd_op_reply(80 region_map [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 ==== 154+0+0 (crc 0 0 0) 0x7fcecc02f080 con 0x7fcebc010070 2023-10-05T19:28:56.460 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.454+0000 7fced3a8ee40 20 rados_obj.operate() r=-2 bl.length=0 2023-10-05T19:28:56.460 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.454+0000 7fced3a8ee40 20 zone default found 2023-10-05T19:28:56.460 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.454+0000 7fced3a8ee40 4 Realm: () 2023-10-05T19:28:56.460 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.454+0000 7fced3a8ee40 4 ZoneGroup: default (4f1f719d-9648-417b-b10c-e5d939825419) 2023-10-05T19:28:56.460 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.454+0000 7fced3a8ee40 4 Zone: default (9a9df965-6612-45d4-a45f-6e3c889bacc2) 2023-10-05T19:28:56.460 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.454+0000 7fced3a8ee40 10 cannot find current period zonegroup using local zonegroup configuration 2023-10-05T19:28:56.460 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.454+0000 7fced3a8ee40 20 zonegroup default 2023-10-05T19:28:56.460 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.454+0000 7fced3a8ee40 20 rados->read ofs=0 len=0 2023-10-05T19:28:56.461 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.454+0000 7fced3a8ee40 1 -- 172.21.15.167:0/3622536379 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:81 5.3 5:c52100b6:::period_config.default:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32) v8 -- 0x5584b0b2e7c0 con 0x7fcebc010070 2023-10-05T19:28:56.461 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.454+0000 7fced1dc4700 1 -- 172.21.15.167:0/3622536379 <== osd.0 v2:172.21.15.118:6800/27690 22 ==== osd_op_reply(81 period_config.default [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 ==== 165+0+0 (crc 0 0 0) 0x7fcecc050040 con 0x7fcebc010070 2023-10-05T19:28:56.461 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.454+0000 7fced3a8ee40 20 rados_obj.operate() r=-2 bl.length=0 2023-10-05T19:28:56.461 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.454+0000 7fced3a8ee40 20 rados->read ofs=0 len=0 2023-10-05T19:28:56.461 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.454+0000 7fced3a8ee40 1 -- 172.21.15.167:0/3622536379 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:82 6.2 6:4347d321:::bucket.sync-source-hints.:head [call version.read in=11b,read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32) v8 -- 0x5584b0b30140 con 0x5584b0b2ac10 2023-10-05T19:28:56.462 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.458+0000 7fced15c3700 1 -- 172.21.15.167:0/3622536379 <== osd.3 v2:172.21.15.167:6808/30036 20 ==== osd_op_reply(82 bucket.sync-source-hints. [call,read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 ==== 211+0+0 (crc 0 0 0) 0x7fcec4039040 con 0x5584b0b2ac10 2023-10-05T19:28:56.462 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.458+0000 7fced3a8ee40 20 rados_obj.operate() r=-2 bl.length=0 2023-10-05T19:28:56.463 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.458+0000 7fced3a8ee40 20 rados->read ofs=0 len=0 2023-10-05T19:28:56.463 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.458+0000 7fced3a8ee40 1 -- 172.21.15.167:0/3622536379 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:83 6.b 6:d467b91b:::bucket.sync-target-hints.:head [call version.read in=11b,read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32) v8 -- 0x5584b0b304f0 con 0x5584b0b2ac10 2023-10-05T19:28:56.463 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.458+0000 7fced15c3700 1 -- 172.21.15.167:0/3622536379 <== osd.3 v2:172.21.15.167:6808/30036 21 ==== osd_op_reply(83 bucket.sync-target-hints. [call,read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 ==== 211+0+0 (crc 0 0 0) 0x7fcec4039040 con 0x5584b0b2ac10 2023-10-05T19:28:56.463 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.458+0000 7fced3a8ee40 20 rados_obj.operate() r=-2 bl.length=0 2023-10-05T19:28:56.463 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.458+0000 7fced3a8ee40 20 started sync module instance, tier type = 2023-10-05T19:28:56.463 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.458+0000 7fced3a8ee40 20 started zone id=9a9df965-6612-45d4-a45f-6e3c889bacc2 (name=default) with tier type = 2023-10-05T19:28:56.463 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.458+0000 7fced3a8ee40 1 -- 172.21.15.167:0/3622536379 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:84 7.1f 7:f95f44c2:::notify.0:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e32) v8 -- 0x5584b0b2f800 con 0x7fcec401a070 2023-10-05T19:28:56.466 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.462+0000 7fced0dc2700 1 -- 172.21.15.167:0/3622536379 <== osd.2 v2:172.21.15.167:6816/30037 13 ==== osd_op_reply(84 notify.0 [create] v32'204 uv204 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7fcebc017050 con 0x7fcec401a070 2023-10-05T19:28:56.466 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.462+0000 7fced3a8ee40 1 -- 172.21.15.167:0/3622536379 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:85 7.1f 7:f95f44c2:::notify.0:head [watch watch cookie 94028388563968] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e32) v8 -- 0x5584b0b29ee0 con 0x7fcec401a070 2023-10-05T19:28:56.466 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.462+0000 7fced3a8ee40 1 -- 172.21.15.167:0/3622536379 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:86 7.0 7:05bf5b68:::notify.1:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e32) v8 -- 0x5584b0b31280 con 0x5584b0af1110 2023-10-05T19:28:56.469 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.462+0000 7fced1dc4700 1 -- 172.21.15.167:0/3622536379 <== osd.1 v2:172.21.15.167:6800/30035 29 ==== osd_op_reply(86 notify.1 [create] v32'204 uv204 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7fcecc050040 con 0x5584b0af1110 2023-10-05T19:28:56.469 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.462+0000 7fced3a8ee40 1 -- 172.21.15.167:0/3622536379 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:87 7.0 7:05bf5b68:::notify.1:head [watch watch cookie 94028388568752] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e32) v8 -- 0x5584b0b31b70 con 0x5584b0af1110 2023-10-05T19:28:56.469 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.462+0000 7fced3a8ee40 1 -- 172.21.15.167:0/3622536379 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:88 7.15 7:a93a5511:::notify.2:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e32) v8 -- 0x5584b0b32410 con 0x5584b0b2ac10 2023-10-05T19:28:56.469 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.466+0000 7fced0dc2700 1 -- 172.21.15.167:0/3622536379 <== osd.2 v2:172.21.15.167:6816/30037 14 ==== osd_op_reply(85 notify.0 [watch watch cookie 94028388563968] v32'205 uv204 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7fcebc017050 con 0x7fcec401a070 2023-10-05T19:28:56.472 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.466+0000 7fced15c3700 1 -- 172.21.15.167:0/3622536379 <== osd.3 v2:172.21.15.167:6808/30036 22 ==== osd_op_reply(88 notify.2 [create] v32'204 uv204 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7fcec4039040 con 0x5584b0b2ac10 2023-10-05T19:28:56.472 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.466+0000 7fced1dc4700 1 -- 172.21.15.167:0/3622536379 <== osd.1 v2:172.21.15.167:6800/30035 30 ==== osd_op_reply(87 notify.1 [watch watch cookie 94028388568752] v32'205 uv204 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7fcecc050040 con 0x5584b0af1110 2023-10-05T19:28:56.473 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.466+0000 7fced3a8ee40 1 -- 172.21.15.167:0/3622536379 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:89 7.15 7:a93a5511:::notify.2:head [watch watch cookie 94028388572016] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e32) v8 -- 0x5584b0b309c0 con 0x5584b0b2ac10 2023-10-05T19:28:56.473 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.466+0000 7fced3a8ee40 1 -- 172.21.15.167:0/3622536379 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:90 7.e 7:7759931f:::notify.3:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e32) v8 -- 0x5584b0b32cb0 con 0x7fcec401a070 2023-10-05T19:28:56.475 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.470+0000 7fced0dc2700 1 -- 172.21.15.167:0/3622536379 <== osd.2 v2:172.21.15.167:6816/30037 15 ==== osd_op_reply(90 notify.3 [create] v32'204 uv204 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7fcebc017050 con 0x7fcec401a070 2023-10-05T19:28:56.475 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.470+0000 7fced3a8ee40 1 -- 172.21.15.167:0/3622536379 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:91 7.e 7:7759931f:::notify.3:head [watch watch cookie 94028388575168] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e32) v8 -- 0x5584b0b33550 con 0x7fcec401a070 2023-10-05T19:28:56.475 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.470+0000 7fced3a8ee40 1 -- 172.21.15.167:0/3622536379 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:92 7.d 7:b4812045:::notify.4:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e32) v8 -- 0x5584b0b340e0 con 0x5584b0af1110 2023-10-05T19:28:56.476 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.470+0000 7fced15c3700 1 -- 172.21.15.167:0/3622536379 <== osd.3 v2:172.21.15.167:6808/30036 23 ==== osd_op_reply(89 notify.2 [watch watch cookie 94028388572016] v32'205 uv204 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7fcec4039040 con 0x5584b0b2ac10 2023-10-05T19:28:56.477 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.474+0000 7fced0dc2700 1 -- 172.21.15.167:0/3622536379 <== osd.2 v2:172.21.15.167:6816/30037 16 ==== osd_op_reply(91 notify.3 [watch watch cookie 94028388575168] v32'205 uv204 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7fcebc017050 con 0x7fcec401a070 2023-10-05T19:28:56.478 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.474+0000 7fced1dc4700 1 -- 172.21.15.167:0/3622536379 <== osd.1 v2:172.21.15.167:6800/30035 31 ==== osd_op_reply(92 notify.4 [create] v32'204 uv204 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7fcecc050040 con 0x5584b0af1110 2023-10-05T19:28:56.478 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.474+0000 7fced3a8ee40 1 -- 172.21.15.167:0/3622536379 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:93 7.d 7:b4812045:::notify.4:head [watch watch cookie 94028388577376] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e32) v8 -- 0x5584b0b31600 con 0x5584b0af1110 2023-10-05T19:28:56.478 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.474+0000 7fced3a8ee40 1 -- 172.21.15.167:0/3622536379 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:94 7.3 7:c609908c:::notify.5:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e32) v8 -- 0x5584b0b349d0 con 0x7fcebc010070 2023-10-05T19:28:56.480 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.474+0000 7fced1dc4700 1 -- 172.21.15.167:0/3622536379 <== osd.1 v2:172.21.15.167:6800/30035 32 ==== osd_op_reply(93 notify.4 [watch watch cookie 94028388577376] v32'205 uv204 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7fcecc050040 con 0x5584b0af1110 2023-10-05T19:28:56.481 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.474+0000 7fced1dc4700 1 -- 172.21.15.167:0/3622536379 <== osd.0 v2:172.21.15.118:6800/27690 23 ==== osd_op_reply(94 notify.5 [create] v32'204 uv204 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7fcecc050040 con 0x7fcebc010070 2023-10-05T19:28:56.481 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.474+0000 7fced3a8ee40 1 -- 172.21.15.167:0/3622536379 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:95 7.3 7:c609908c:::notify.5:head [watch watch cookie 94028388580272] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e32) v8 -- 0x5584b0b34980 con 0x7fcebc010070 2023-10-05T19:28:56.481 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.474+0000 7fced3a8ee40 1 -- 172.21.15.167:0/3622536379 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:96 7.14 7:2b04a3e9:::notify.6:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e32) v8 -- 0x5584b0b35390 con 0x7fcebc010070 2023-10-05T19:28:56.483 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.478+0000 7fced1dc4700 1 -- 172.21.15.167:0/3622536379 <== osd.0 v2:172.21.15.118:6800/27690 24 ==== osd_op_reply(95 notify.5 [watch watch cookie 94028388580272] v32'205 uv204 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7fcecc050040 con 0x7fcebc010070 2023-10-05T19:28:56.484 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.478+0000 7fced1dc4700 1 -- 172.21.15.167:0/3622536379 <== osd.0 v2:172.21.15.118:6800/27690 25 ==== osd_op_reply(96 notify.6 [create] v32'204 uv204 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7fcecc050040 con 0x7fcebc010070 2023-10-05T19:28:56.484 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.478+0000 7fced3a8ee40 1 -- 172.21.15.167:0/3622536379 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:97 7.14 7:2b04a3e9:::notify.6:head [watch watch cookie 94028388582816] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e32) v8 -- 0x5584b0b35390 con 0x7fcebc010070 2023-10-05T19:28:56.484 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.478+0000 7fced3a8ee40 1 -- 172.21.15.167:0/3622536379 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:98 7.9 7:93e5b521:::notify.7:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e32) v8 -- 0x5584b0b35da0 con 0x7fcebc010070 2023-10-05T19:28:56.486 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.482+0000 7fced1dc4700 1 -- 172.21.15.167:0/3622536379 <== osd.0 v2:172.21.15.118:6800/27690 26 ==== osd_op_reply(97 notify.6 [watch watch cookie 94028388582816] v32'205 uv204 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7fcecc050040 con 0x7fcebc010070 2023-10-05T19:28:56.487 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.482+0000 7fced1dc4700 1 -- 172.21.15.167:0/3622536379 <== osd.0 v2:172.21.15.118:6800/27690 27 ==== osd_op_reply(98 notify.7 [create] v32'204 uv204 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7fcecc050040 con 0x7fcebc010070 2023-10-05T19:28:56.487 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.482+0000 7fced3a8ee40 1 -- 172.21.15.167:0/3622536379 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:99 7.9 7:93e5b521:::notify.7:head [watch watch cookie 94028388585680] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e32) v8 -- 0x5584b0b35da0 con 0x7fcebc010070 2023-10-05T19:28:56.487 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.482+0000 7fced3a8ee40 20 add_watcher() i=0 2023-10-05T19:28:56.487 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.482+0000 7fced3a8ee40 20 add_watcher() i=1 2023-10-05T19:28:56.488 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.482+0000 7fced3a8ee40 20 add_watcher() i=2 2023-10-05T19:28:56.488 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.482+0000 7fced3a8ee40 20 add_watcher() i=3 2023-10-05T19:28:56.488 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.482+0000 7fced3a8ee40 20 add_watcher() i=4 2023-10-05T19:28:56.488 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.482+0000 7fced3a8ee40 20 add_watcher() i=5 2023-10-05T19:28:56.488 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.482+0000 7fced3a8ee40 20 add_watcher() i=6 2023-10-05T19:28:56.489 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.486+0000 7fced1dc4700 1 -- 172.21.15.167:0/3622536379 <== osd.0 v2:172.21.15.118:6800/27690 28 ==== osd_op_reply(99 notify.7 [watch watch cookie 94028388585680] v32'205 uv204 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7fcecc050040 con 0x7fcebc010070 2023-10-05T19:28:56.490 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.486+0000 7fced3a8ee40 20 add_watcher() i=7 2023-10-05T19:28:56.490 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.486+0000 7fced3a8ee40 2 all 8 watchers are set, enabling cache 2023-10-05T19:28:56.490 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.486+0000 7fced3a8ee40 1 -- 172.21.15.167:0/3622536379 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:100 6.4 6:22d26bf9:::data_loggenerations_metadata:head [call version.check_conds in=74b,call version.read in=11b,read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32) v8 -- 0x5584b0b38f40 con 0x5584b0af1110 2023-10-05T19:28:56.490 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.486+0000 7fced1dc4700 1 -- 172.21.15.167:0/3622536379 <== osd.1 v2:172.21.15.167:6800/30035 33 ==== osd_op_reply(100 data_loggenerations_metadata [call,call out=48b,read 0~28 out=28b] v0'0 uv1 ondisk = 0) v8 ==== 256+0+76 (crc 0 0 0) 0x7fcecc050040 con 0x5584b0af1110 2023-10-05T19:28:56.490 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.486+0000 7fced3a8ee40 1 -- 172.21.15.167:0/3622536379 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:101 6.4 6:22d26bf9:::data_loggenerations_metadata:head [watch watch cookie 94028388588064] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e32) v8 -- 0x5584b0b364b0 con 0x5584b0af1110 2023-10-05T19:28:56.492 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.486+0000 7fced1dc4700 1 -- 172.21.15.167:0/3622536379 <== osd.1 v2:172.21.15.167:6800/30035 34 ==== osd_op_reply(101 data_loggenerations_metadata [watch watch cookie 94028388588064] v32'203 uv1 ondisk = 0) v8 ==== 172+0+0 (crc 0 0 0) 0x7fcecc050040 con 0x5584b0af1110 2023-10-05T19:28:56.492 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.486+0000 7fcea27e4700 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2023-10-05T19:28:56.493 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.486+0000 7fced3a8ee40 20 check_secure_mon_conn(): auth registy supported: methods=[2] modes=[2,1] 2023-10-05T19:28:56.493 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.486+0000 7fced3a8ee40 20 check_secure_mon_conn(): mode 1 is insecure 2023-10-05T19:28:56.493 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.486+0000 7fced3a8ee40 1 Processor -- start 2023-10-05T19:28:56.493 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.486+0000 7fced3a8ee40 1 -- start start 2023-10-05T19:28:56.494 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.490+0000 7fced3a8ee40 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5584b0b43710 0x5584b0b43ae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-05T19:28:56.494 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.490+0000 7fced3a8ee40 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x5584b0b44020 con 0x5584b0b43710 2023-10-05T19:28:56.494 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.490+0000 7fced0dc2700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5584b0b43710 0x5584b0b43ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-05T19:28:56.494 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.490+0000 7fced0dc2700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5584b0b43710 0x5584b0b43ae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.118:3300/0 says I am v2:172.21.15.167:53360/0 (socket says 172.21.15.167:53360) 2023-10-05T19:28:56.494 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.490+0000 7fced0dc2700 1 -- 172.21.15.167:0/3854953730 learned_addr learned my addr 172.21.15.167:0/3854953730 (peer_addr_for_me v2:172.21.15.167:0/0) 2023-10-05T19:28:56.495 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.490+0000 7fced0dc2700 1 -- 172.21.15.167:0/3854953730 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5584b0b44160 con 0x5584b0b43710 2023-10-05T19:28:56.495 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.490+0000 7fced0dc2700 1 --2- 172.21.15.167:0/3854953730 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5584b0b43710 0x5584b0b43ae0 secure :-1 s=READY pgs=546 cs=0 l=1 rev1=1 crypto rx=0x5584b0b288a0 tx=0x7fcebc010000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a50bde04c80cefb server_cookie=0 in_seq=0 out_seq=0 2023-10-05T19:28:56.496 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.490+0000 7fcea1fe3700 1 -- 172.21.15.167:0/3854953730 <== mon.0 v2:172.21.15.118:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fcebc01c9f0 con 0x5584b0b43710 2023-10-05T19:28:56.496 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.490+0000 7fcea1fe3700 1 -- 172.21.15.167:0/3854953730 <== mon.0 v2:172.21.15.118:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcebc01cb50 con 0x5584b0b43710 2023-10-05T19:28:56.496 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.490+0000 7fcea1fe3700 1 -- 172.21.15.167:0/3854953730 <== mon.0 v2:172.21.15.118:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fcebc01cd00 con 0x5584b0b43710 2023-10-05T19:28:56.497 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.490+0000 7fced3a8ee40 1 -- 172.21.15.167:0/3854953730 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5584b0b43710 msgr2=0x5584b0b43ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-05T19:28:56.497 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.490+0000 7fced3a8ee40 1 --2- 172.21.15.167:0/3854953730 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5584b0b43710 0x5584b0b43ae0 secure :-1 s=READY pgs=546 cs=0 l=1 rev1=1 crypto rx=0x5584b0b288a0 tx=0x7fcebc010000 comp rx=0 tx=0).stop 2023-10-05T19:28:56.497 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.490+0000 7fced3a8ee40 1 -- 172.21.15.167:0/3854953730 shutdown_connections 2023-10-05T19:28:56.497 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.490+0000 7fced3a8ee40 1 --2- 172.21.15.167:0/3854953730 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5584b0b43710 0x5584b0b43ae0 unknown :-1 s=CLOSED pgs=546 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-05T19:28:56.497 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.490+0000 7fced3a8ee40 1 -- 172.21.15.167:0/3854953730 >> 172.21.15.167:0/3854953730 conn(0x5584b0b3ead0 msgr2=0x5584b0b3cc20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-05T19:28:56.497 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.490+0000 7fced3a8ee40 1 -- 172.21.15.167:0/3854953730 shutdown_connections 2023-10-05T19:28:56.497 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.490+0000 7fced3a8ee40 1 -- 172.21.15.167:0/3854953730 wait complete. 2023-10-05T19:28:56.498 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.494+0000 7fced3a8ee40 1 Processor -- start 2023-10-05T19:28:56.498 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.494+0000 7fced3a8ee40 1 -- start start 2023-10-05T19:28:56.498 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.494+0000 7fced3a8ee40 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5584b0b43710 0x5584b0b49210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-05T19:28:56.498 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.494+0000 7fced3a8ee40 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x5584b0b4a210 con 0x5584b0b43710 2023-10-05T19:28:56.499 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.494+0000 7fced0dc2700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5584b0b43710 0x5584b0b49210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-05T19:28:56.499 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.494+0000 7fced0dc2700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5584b0b43710 0x5584b0b49210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.118:3300/0 says I am v2:172.21.15.167:53376/0 (socket says 172.21.15.167:53376) 2023-10-05T19:28:56.499 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.494+0000 7fced0dc2700 1 -- 172.21.15.167:0/2392103459 learned_addr learned my addr 172.21.15.167:0/2392103459 (peer_addr_for_me v2:172.21.15.167:0/0) 2023-10-05T19:28:56.500 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.494+0000 7fced0dc2700 1 -- 172.21.15.167:0/2392103459 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5584b0af3a60 con 0x5584b0b43710 2023-10-05T19:28:56.500 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.494+0000 7fced0dc2700 1 --2- 172.21.15.167:0/2392103459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5584b0b43710 0x5584b0b49210 secure :-1 s=READY pgs=547 cs=0 l=1 rev1=1 crypto rx=0x7fcebc010fd0 tx=0x7fcebc02f000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-05T19:28:56.500 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.494+0000 7fce13fff700 1 -- 172.21.15.167:0/2392103459 <== mon.0 v2:172.21.15.118:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fcebc047460 con 0x5584b0b43710 2023-10-05T19:28:56.500 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.494+0000 7fce13fff700 1 -- 172.21.15.167:0/2392103459 <== mon.0 v2:172.21.15.118:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcebc047600 con 0x5584b0b43710 2023-10-05T19:28:56.500 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.494+0000 7fced3a8ee40 1 -- 172.21.15.167:0/2392103459 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x5584b0b49810 con 0x5584b0b43710 2023-10-05T19:28:56.500 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.494+0000 7fce13fff700 1 -- 172.21.15.167:0/2392103459 <== mon.0 v2:172.21.15.118:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fcebc0477b0 con 0x5584b0b43710 2023-10-05T19:28:56.500 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.494+0000 7fced3a8ee40 1 -- 172.21.15.167:0/2392103459 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x5584b0b49c50 con 0x5584b0b43710 2023-10-05T19:28:56.501 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.494+0000 7fce13fff700 1 -- 172.21.15.167:0/2392103459 <== mon.0 v2:172.21.15.118:3300/0 4 ==== mgrmap(e 5) v1 ==== 40329+0+0 (secure 0 0 0) 0x7fcebc035400 con 0x5584b0b43710 2023-10-05T19:28:56.502 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.498+0000 7fce13fff700 1 --2- 172.21.15.167:0/2392103459 >> [v2:172.21.15.167:6824/29889,v1:172.21.15.167:6825/29889] conn(0x7fce18030c50 0x7fce18033100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-05T19:28:56.502 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.498+0000 7fce13fff700 1 -- 172.21.15.167:0/2392103459 <== mon.0 v2:172.21.15.118:3300/0 5 ==== osd_map(32..32 src has 1..32) v4 ==== 6260+0+0 (secure 0 0 0) 0x7fcebc02f070 con 0x5584b0b43710 2023-10-05T19:28:56.502 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.498+0000 7fced3a8ee40 5 note: GC not initialized 2023-10-05T19:28:56.502 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.498+0000 7fced1dc4700 1 --2- 172.21.15.167:0/2392103459 >> [v2:172.21.15.167:6824/29889,v1:172.21.15.167:6825/29889] conn(0x7fce18030c50 0x7fce18033100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-05T19:28:56.502 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.498+0000 7fced1dc4700 1 --2- 172.21.15.167:0/2392103459 >> [v2:172.21.15.167:6824/29889,v1:172.21.15.167:6825/29889] conn(0x7fce18030c50 0x7fce18033100 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x5584b0b36410 tx=0x7fcecc000f90 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-05T19:28:56.504 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.498+0000 7fce117fa700 20 reqs_thread_entry: start 2023-10-05T19:28:56.504 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.498+0000 7fced3a8ee40 20 init_complete bucket index max shards: 11 2023-10-05T19:28:56.504 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.498+0000 7fce0affd700 20 reqs_thread_entry: start 2023-10-05T19:28:56.505 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.498+0000 7fced3a8ee40 10 rgw notify: Started notification manager with: 1 workers 2023-10-05T19:28:56.505 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.498+0000 7fce097fa700 20 rgw notify: INFO: next queues processing will happen at: Thu Oct 5 19:29:26 2023 2023-10-05T19:28:56.505 INFO:teuthology.orchestra.run.smithi167.stderr: 2023-10-05T19:28:56.506 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.498+0000 7fce08ff9700 20 reqs_thread_entry: start 2023-10-05T19:28:56.506 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.498+0000 7fced3a8ee40 20 get_system_obj_state: rctx=0x7ffcccacb440 obj=default.rgw.meta:root:myfoo state=0x5584b09c80b0 s->prefetch_data=0 2023-10-05T19:28:56.506 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.498+0000 7fced3a8ee40 10 cache get: name=default.rgw.meta+root+myfoo : miss 2023-10-05T19:28:56.506 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.498+0000 7fced3a8ee40 1 -- 172.21.15.167:0/3622536379 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:102 8.9 8:91b71841:root::myfoo:head [call version.read in=11b,getxattrs,stat] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32) v8 -- 0x5584b0bb2220 con 0x7fcec401a070 2023-10-05T19:28:56.507 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.502+0000 7fced0dc2700 1 -- 172.21.15.167:0/3622536379 <== osd.2 v2:172.21.15.167:6816/30037 17 ==== osd_op_reply(102 myfoo [call out=48b,getxattrs out=75b,stat out=16b] v0'0 uv34 ondisk = 0) v8 ==== 233+0+139 (crc 0 0 0) 0x7fcebc032040 con 0x7fcec401a070 2023-10-05T19:28:56.507 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.502+0000 7fced3a8ee40 10 cache put: name=default.rgw.meta+root+myfoo info.flags=0x16 2023-10-05T19:28:56.507 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.502+0000 7fced3a8ee40 10 adding default.rgw.meta+root+myfoo to cache LRU end 2023-10-05T19:28:56.507 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.502+0000 7fced3a8ee40 10 updating xattr: name=ceph.objclass.version bl.length()=42 2023-10-05T19:28:56.507 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.502+0000 7fced3a8ee40 20 get_system_obj_state: s->obj_tag was set empty 2023-10-05T19:28:56.507 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.502+0000 7fced3a8ee40 10 cache get: name=default.rgw.meta+root+myfoo : type miss (requested=0x11, cached=0x16) 2023-10-05T19:28:56.507 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.502+0000 7fced3a8ee40 20 rados->read ofs=0 len=0 2023-10-05T19:28:56.507 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.502+0000 7fced3a8ee40 1 -- 172.21.15.167:0/3622536379 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:103 8.9 8:91b71841:root::myfoo:head [call version.check_conds in=122b,call version.read in=11b,read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32) v8 -- 0x5584b0bb4cb0 con 0x7fcec401a070 2023-10-05T19:28:56.508 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.502+0000 7fced0dc2700 1 -- 172.21.15.167:0/3622536379 <== osd.2 v2:172.21.15.167:6816/30037 18 ==== osd_op_reply(103 myfoo [call,call out=48b,read 0~165 out=165b] v0'0 uv34 ondisk = 0) v8 ==== 233+0+213 (crc 0 0 0) 0x7fcebc032040 con 0x7fcec401a070 2023-10-05T19:28:56.508 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.502+0000 7fced3a8ee40 20 rados_obj.operate() r=0 bl.length=165 2023-10-05T19:28:56.508 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.502+0000 7fced3a8ee40 10 cache put: name=default.rgw.meta+root+myfoo info.flags=0x11 2023-10-05T19:28:56.508 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.502+0000 7fced3a8ee40 10 moving default.rgw.meta+root+myfoo to cache LRU end 2023-10-05T19:28:56.508 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.502+0000 7fced3a8ee40 20 get_system_obj_state: rctx=0x7ffcccacb440 obj=default.rgw.meta:root:.bucket.meta.myfoo:9a9df965-6612-45d4-a45f-6e3c889bacc2.4223.5 state=0x5584b099e2b0 s->prefetch_data=0 2023-10-05T19:28:56.508 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.502+0000 7fced3a8ee40 10 cache get: name=default.rgw.meta+root+.bucket.meta.myfoo:9a9df965-6612-45d4-a45f-6e3c889bacc2.4223.5 : miss 2023-10-05T19:28:56.508 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.502+0000 7fced3a8ee40 1 -- 172.21.15.167:0/3622536379 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:104 8.9 8:96cb4b0f:root::.bucket.meta.myfoo%3a9a9df965-6612-45d4-a45f-6e3c889bacc2.4223.5:head [call version.read in=11b,getxattrs,stat] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32) v8 -- 0x5584b0bb40d0 con 0x7fcec401a070 2023-10-05T19:28:56.509 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.502+0000 7fced0dc2700 1 -- 172.21.15.167:0/3622536379 <== osd.2 v2:172.21.15.167:6816/30037 19 ==== osd_op_reply(104 .bucket.meta.myfoo:9a9df965-6612-45d4-a45f-6e3c889bacc2.4223.5 [call out=48b,getxattrs out=218b,stat out=16b] v0'0 uv33 ondisk = 0) v8 ==== 290+0+282 (crc 0 0 0) 0x7fcebc032040 con 0x7fcec401a070 2023-10-05T19:28:56.510 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.502+0000 7fced3a8ee40 10 cache put: name=default.rgw.meta+root+.bucket.meta.myfoo:9a9df965-6612-45d4-a45f-6e3c889bacc2.4223.5 info.flags=0x16 2023-10-05T19:28:56.510 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.502+0000 7fced3a8ee40 10 adding default.rgw.meta+root+.bucket.meta.myfoo:9a9df965-6612-45d4-a45f-6e3c889bacc2.4223.5 to cache LRU end 2023-10-05T19:28:56.510 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.502+0000 7fced3a8ee40 10 updating xattr: name=ceph.objclass.version bl.length()=42 2023-10-05T19:28:56.510 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.502+0000 7fced3a8ee40 10 updating xattr: name=user.rgw.acl bl.length()=123 2023-10-05T19:28:56.510 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.502+0000 7fced3a8ee40 20 get_system_obj_state: s->obj_tag was set empty 2023-10-05T19:28:56.510 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.502+0000 7fced3a8ee40 20 Read xattr: user.rgw.acl 2023-10-05T19:28:56.510 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.502+0000 7fced3a8ee40 20 Read xattr: user.rgw.idtag 2023-10-05T19:28:56.510 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.502+0000 7fced3a8ee40 10 cache get: name=default.rgw.meta+root+.bucket.meta.myfoo:9a9df965-6612-45d4-a45f-6e3c889bacc2.4223.5 : type miss (requested=0x13, cached=0x16) 2023-10-05T19:28:56.510 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.502+0000 7fced3a8ee40 20 rados->read ofs=0 len=0 2023-10-05T19:28:56.510 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.502+0000 7fced3a8ee40 1 -- 172.21.15.167:0/3622536379 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:105 8.9 8:96cb4b0f:root::.bucket.meta.myfoo%3a9a9df965-6612-45d4-a45f-6e3c889bacc2.4223.5:head [call version.check_conds in=122b,call version.read in=11b,read 0~0,getxattrs] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32) v8 -- 0x5584b0bb4c90 con 0x7fcec401a070 2023-10-05T19:28:56.511 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.502+0000 7fced0dc2700 1 -- 172.21.15.167:0/3622536379 <== osd.2 v2:172.21.15.167:6816/30037 20 ==== osd_op_reply(105 .bucket.meta.myfoo:9a9df965-6612-45d4-a45f-6e3c889bacc2.4223.5 [call,call out=48b,read 0~352 out=352b,getxattrs out=218b] v0'0 uv33 ondisk = 0) v8 ==== 332+0+618 (crc 0 0 0) 0x7fcebc032040 con 0x7fcec401a070 2023-10-05T19:28:56.511 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.502+0000 7fced3a8ee40 20 rados_obj.operate() r=0 bl.length=352 2023-10-05T19:28:56.511 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.502+0000 7fced3a8ee40 10 cache put: name=default.rgw.meta+root+.bucket.meta.myfoo:9a9df965-6612-45d4-a45f-6e3c889bacc2.4223.5 info.flags=0x13 2023-10-05T19:28:56.511 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.502+0000 7fced3a8ee40 10 moving default.rgw.meta+root+.bucket.meta.myfoo:9a9df965-6612-45d4-a45f-6e3c889bacc2.4223.5 to cache LRU end 2023-10-05T19:28:56.511 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.502+0000 7fced3a8ee40 10 updating xattr: name=ceph.objclass.version bl.length()=42 2023-10-05T19:28:56.511 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.502+0000 7fced3a8ee40 10 updating xattr: name=user.rgw.acl bl.length()=123 2023-10-05T19:28:56.511 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.502+0000 7fced3a8ee40 10 chain_cache_entry: cache_locator=default.rgw.meta+root+.bucket.meta.myfoo:9a9df965-6612-45d4-a45f-6e3c889bacc2.4223.5 2023-10-05T19:28:56.511 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.502+0000 7fced3a8ee40 20 get_obj_state: rctx=0x7ffcccacb680 obj=myfoo:bb3aec0fdcdbc2974890f805c585d432 state=0x5584b0bb76a8 s->prefetch_data=0 2023-10-05T19:28:56.511 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.502+0000 7fced3a8ee40 1 -- 172.21.15.167:0/3622536379 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:106 3.1cs0 3:38675d0b:::9a9df965-6612-45d4-a45f-6e3c889bacc2.4223.5_bb3aec0fdcdbc2974890f805c585d432:head [getxattrs,stat] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32) v8 -- 0x5584b0bb8010 con 0x5584b0af1110 2023-10-05T19:28:56.511 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.506+0000 7fced1dc4700 1 -- 172.21.15.167:0/3622536379 <== osd.1 v2:172.21.15.167:6800/30035 35 ==== osd_op_reply(106 9a9df965-6612-45d4-a45f-6e3c889bacc2.4223.5_bb3aec0fdcdbc2974890f805c585d432 [getxattrs out=910b,stat out=16b] v0'0 uv3 ondisk = 0) v8 ==== 262+0+926 (crc 0 0 0) 0x7fcecc050040 con 0x5584b0af1110 2023-10-05T19:28:56.511 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.506+0000 7fced3a8ee40 10 manifest: total_size = 5 2023-10-05T19:28:56.511 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.506+0000 7fced3a8ee40 20 get_obj_state: setting s->obj_tag to _RiM11gV4_HUAssJBJPt8rz1mj38MntN 2023-10-05T19:28:56.512 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.506+0000 7fced3a8ee40 1 -- 172.21.15.167:0/3622536379 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:107 7.1f 7:f95f44c2:::notify.0:head [watch unwatch cookie 94028388563968] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e32) v8 -- 0x7fcecc049000 con 0x7fcec401a070 2023-10-05T19:28:56.513 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.506+0000 7fced0dc2700 1 -- 172.21.15.167:0/3622536379 <== osd.2 v2:172.21.15.167:6816/30037 21 ==== osd_op_reply(107 notify.0 [watch unwatch cookie 94028388563968] v32'206 uv204 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7fcebc032040 con 0x7fcec401a070 2023-10-05T19:28:56.513 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.506+0000 7fced3a8ee40 20 remove_watcher() i=0 2023-10-05T19:28:56.513 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.506+0000 7fced3a8ee40 2 removed watcher, disabling cache 2023-10-05T19:28:56.513 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.506+0000 7fced3a8ee40 1 -- 172.21.15.167:0/3622536379 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:108 7.0 7:05bf5b68:::notify.1:head [watch unwatch cookie 94028388568752] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e32) v8 -- 0x5584b0b0ae10 con 0x5584b0af1110 2023-10-05T19:28:56.516 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.510+0000 7fced1dc4700 1 -- 172.21.15.167:0/3622536379 <== osd.1 v2:172.21.15.167:6800/30035 36 ==== osd_op_reply(108 notify.1 [watch unwatch cookie 94028388568752] v32'206 uv204 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7fcecc050040 con 0x5584b0af1110 2023-10-05T19:28:56.516 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.510+0000 7fced3a8ee40 20 remove_watcher() i=1 2023-10-05T19:28:56.516 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.510+0000 7fced3a8ee40 1 -- 172.21.15.167:0/3622536379 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:109 7.15 7:a93a5511:::notify.2:head [watch unwatch cookie 94028388572016] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e32) v8 -- 0x5584b0b0ae10 con 0x5584b0b2ac10 2023-10-05T19:28:56.519 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.514+0000 7fced15c3700 1 -- 172.21.15.167:0/3622536379 <== osd.3 v2:172.21.15.167:6808/30036 24 ==== osd_op_reply(109 notify.2 [watch unwatch cookie 94028388572016] v32'206 uv204 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7fcec4039040 con 0x5584b0b2ac10 2023-10-05T19:28:56.519 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.514+0000 7fced3a8ee40 20 remove_watcher() i=2 2023-10-05T19:28:56.519 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.514+0000 7fced3a8ee40 1 -- 172.21.15.167:0/3622536379 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:110 7.e 7:7759931f:::notify.3:head [watch unwatch cookie 94028388575168] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e32) v8 -- 0x5584b0b30eb0 con 0x7fcec401a070 2023-10-05T19:28:56.521 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.514+0000 7fced0dc2700 1 -- 172.21.15.167:0/3622536379 <== osd.2 v2:172.21.15.167:6816/30037 22 ==== osd_op_reply(110 notify.3 [watch unwatch cookie 94028388575168] v32'206 uv204 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7fcebc032040 con 0x7fcec401a070 2023-10-05T19:28:56.521 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.514+0000 7fced3a8ee40 20 remove_watcher() i=3 2023-10-05T19:28:56.521 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.514+0000 7fced3a8ee40 1 -- 172.21.15.167:0/3622536379 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:111 7.d 7:b4812045:::notify.4:head [watch unwatch cookie 94028388577376] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e32) v8 -- 0x5584b0b31b70 con 0x5584b0af1110 2023-10-05T19:28:56.524 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.518+0000 7fced1dc4700 1 -- 172.21.15.167:0/3622536379 <== osd.1 v2:172.21.15.167:6800/30035 37 ==== osd_op_reply(111 notify.4 [watch unwatch cookie 94028388577376] v32'206 uv204 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7fcecc050040 con 0x5584b0af1110 2023-10-05T19:28:56.524 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.518+0000 7fced3a8ee40 20 remove_watcher() i=4 2023-10-05T19:28:56.524 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.518+0000 7fced3a8ee40 1 -- 172.21.15.167:0/3622536379 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:112 7.3 7:c609908c:::notify.5:head [watch unwatch cookie 94028388580272] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e32) v8 -- 0x5584b0b30eb0 con 0x7fcebc010070 2023-10-05T19:28:56.527 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.522+0000 7fced1dc4700 1 -- 172.21.15.167:0/3622536379 <== osd.0 v2:172.21.15.118:6800/27690 29 ==== osd_op_reply(112 notify.5 [watch unwatch cookie 94028388580272] v32'206 uv204 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7fcecc050040 con 0x7fcebc010070 2023-10-05T19:28:56.527 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.522+0000 7fced3a8ee40 20 remove_watcher() i=5 2023-10-05T19:28:56.528 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.522+0000 7fced3a8ee40 1 -- 172.21.15.167:0/3622536379 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:113 7.14 7:2b04a3e9:::notify.6:head [watch unwatch cookie 94028388582816] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e32) v8 -- 0x5584b0b33060 con 0x7fcebc010070 2023-10-05T19:28:56.531 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.526+0000 7fced1dc4700 1 -- 172.21.15.167:0/3622536379 <== osd.0 v2:172.21.15.118:6800/27690 30 ==== osd_op_reply(113 notify.6 [watch unwatch cookie 94028388582816] v32'206 uv204 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7fcecc050040 con 0x7fcebc010070 2023-10-05T19:28:56.531 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.526+0000 7fced3a8ee40 20 remove_watcher() i=6 2023-10-05T19:28:56.531 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.526+0000 7fced3a8ee40 1 -- 172.21.15.167:0/3622536379 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:114 7.9 7:93e5b521:::notify.7:head [watch unwatch cookie 94028388585680] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e32) v8 -- 0x5584b0b33bb0 con 0x7fcebc010070 2023-10-05T19:28:56.534 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.530+0000 7fced1dc4700 1 -- 172.21.15.167:0/3622536379 <== osd.0 v2:172.21.15.118:6800/27690 31 ==== osd_op_reply(114 notify.7 [watch unwatch cookie 94028388585680] v32'206 uv204 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7fcecc050040 con 0x7fcebc010070 2023-10-05T19:28:56.534 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.530+0000 7fced3a8ee40 20 remove_watcher() i=7 2023-10-05T19:28:56.536 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.530+0000 7fcea27e4700 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2023-10-05T19:28:56.536 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.530+0000 7fced3a8ee40 1 -- 172.21.15.167:0/3622536379 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:115 6.4 6:22d26bf9:::data_loggenerations_metadata:head [watch unwatch cookie 94028388588064] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e32) v8 -- 0x5584b0b33060 con 0x5584b0af1110 2023-10-05T19:28:56.539 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.534+0000 7fced1dc4700 1 -- 172.21.15.167:0/3622536379 <== osd.1 v2:172.21.15.167:6800/30035 38 ==== osd_op_reply(115 data_loggenerations_metadata [watch unwatch cookie 94028388588064] v32'204 uv1 ondisk = 0) v8 ==== 172+0+0 (crc 0 0 0) 0x7fcecc050040 con 0x5584b0af1110 2023-10-05T19:28:56.539 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.534+0000 7fced3a8ee40 1 -- 172.21.15.167:0/3622536379 >> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] conn(0x7fcebc010070 msgr2=0x7fcebc01c430 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-05T19:28:56.539 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.534+0000 7fced3a8ee40 1 --2- 172.21.15.167:0/3622536379 >> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] conn(0x7fcebc010070 0x7fcebc01c430 crc :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-05T19:28:56.540 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.534+0000 7fced3a8ee40 1 -- 172.21.15.167:0/3622536379 >> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] conn(0x5584b0af1110 msgr2=0x5584b0b26e30 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-05T19:28:56.540 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.534+0000 7fced3a8ee40 1 --2- 172.21.15.167:0/3622536379 >> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] conn(0x5584b0af1110 0x5584b0b26e30 crc :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-05T19:28:56.540 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.534+0000 7fced3a8ee40 1 -- 172.21.15.167:0/3622536379 >> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] conn(0x7fcec401a070 msgr2=0x7fcec4043760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-05T19:28:56.540 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.534+0000 7fced3a8ee40 1 --2- 172.21.15.167:0/3622536379 >> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] conn(0x7fcec401a070 0x7fcec4043760 crc :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-05T19:28:56.540 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.534+0000 7fced3a8ee40 1 -- 172.21.15.167:0/3622536379 >> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] conn(0x5584b0b2ac10 msgr2=0x5584b0b2cff0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-05T19:28:56.540 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.534+0000 7fced3a8ee40 1 --2- 172.21.15.167:0/3622536379 >> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] conn(0x5584b0b2ac10 0x5584b0b2cff0 crc :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-05T19:28:56.540 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.534+0000 7fced3a8ee40 1 -- 172.21.15.167:0/3622536379 >> [v2:172.21.15.167:6824/29889,v1:172.21.15.167:6825/29889] conn(0x7fceb4030c50 msgr2=0x7fceb4033100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-05T19:28:56.540 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.534+0000 7fced3a8ee40 1 --2- 172.21.15.167:0/3622536379 >> [v2:172.21.15.167:6824/29889,v1:172.21.15.167:6825/29889] conn(0x7fceb4030c50 0x7fceb4033100 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7fcebc000f80 tx=0x7fcebc013000 comp rx=0 tx=0).stop 2023-10-05T19:28:56.540 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.534+0000 7fced3a8ee40 1 -- 172.21.15.167:0/3622536379 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5584b0a633e0 msgr2=0x5584b0b1b340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-05T19:28:56.540 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.534+0000 7fced1dc4700 1 -- 172.21.15.167:0/3622536379 reap_dead start 2023-10-05T19:28:56.541 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.534+0000 7fced3a8ee40 1 --2- 172.21.15.167:0/3622536379 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5584b0a633e0 0x5584b0b1b340 secure :-1 s=READY pgs=545 cs=0 l=1 rev1=1 crypto rx=0x7fcec40018e0 tx=0x7fcec4016000 comp rx=0 tx=0).stop 2023-10-05T19:28:56.541 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.534+0000 7fced3a8ee40 1 -- 172.21.15.167:0/3622536379 shutdown_connections 2023-10-05T19:28:56.541 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.534+0000 7fced3a8ee40 1 --2- 172.21.15.167:0/3622536379 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5584b0a633e0 0x5584b0b1b340 unknown :-1 s=CLOSED pgs=545 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-05T19:28:56.541 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.534+0000 7fced3a8ee40 1 -- 172.21.15.167:0/3622536379 >> 172.21.15.167:0/3622536379 conn(0x5584b0ae9750 msgr2=0x5584b0b0ebb0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-05T19:28:56.541 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.534+0000 7fced3a8ee40 1 -- 172.21.15.167:0/3622536379 shutdown_connections 2023-10-05T19:28:56.541 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.534+0000 7fced3a8ee40 1 -- 172.21.15.167:0/3622536379 wait complete. 2023-10-05T19:28:56.542 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.534+0000 7fced3a8ee40 1 -- 172.21.15.167:0/2392103459 >> [v2:172.21.15.167:6824/29889,v1:172.21.15.167:6825/29889] conn(0x7fce18030c50 msgr2=0x7fce18033100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-05T19:28:56.542 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.534+0000 7fced3a8ee40 1 --2- 172.21.15.167:0/2392103459 >> [v2:172.21.15.167:6824/29889,v1:172.21.15.167:6825/29889] conn(0x7fce18030c50 0x7fce18033100 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x5584b0b36410 tx=0x7fcecc000f90 comp rx=0 tx=0).stop 2023-10-05T19:28:56.542 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.534+0000 7fced3a8ee40 1 -- 172.21.15.167:0/2392103459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5584b0b43710 msgr2=0x5584b0b49210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-05T19:28:56.542 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.534+0000 7fced3a8ee40 1 --2- 172.21.15.167:0/2392103459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5584b0b43710 0x5584b0b49210 secure :-1 s=READY pgs=547 cs=0 l=1 rev1=1 crypto rx=0x7fcebc010fd0 tx=0x7fcebc02f000 comp rx=0 tx=0).stop 2023-10-05T19:28:56.542 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.534+0000 7fced3a8ee40 1 -- 172.21.15.167:0/2392103459 shutdown_connections 2023-10-05T19:28:56.542 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.534+0000 7fced3a8ee40 1 --2- 172.21.15.167:0/2392103459 >> [v2:172.21.15.167:6824/29889,v1:172.21.15.167:6825/29889] conn(0x7fce18030c50 0x7fce18033100 unknown :-1 s=CLOSED pgs=189 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-05T19:28:56.542 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.534+0000 7fced3a8ee40 1 --2- 172.21.15.167:0/2392103459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5584b0b43710 0x5584b0b49210 unknown :-1 s=CLOSED pgs=547 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-05T19:28:56.543 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.534+0000 7fced3a8ee40 1 -- 172.21.15.167:0/2392103459 >> 172.21.15.167:0/2392103459 conn(0x5584b0b3ead0 msgr2=0x5584b0b3cb70 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-05T19:28:56.543 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.534+0000 7fced3a8ee40 1 -- 172.21.15.167:0/2392103459 shutdown_connections 2023-10-05T19:28:56.543 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.534+0000 7fced3a8ee40 1 -- 172.21.15.167:0/2392103459 wait complete. 2023-10-05T19:28:56.547 INFO:teuthology.orchestra.run.smithi167.stdout:fooFoohttp://acs.amazonaws.com/groups/global/AllUsersREADfooFooFULL_CONTROL 2023-10-05T19:28:56.547 DEBUG:tasks.util.rgw: raw result: fooFoohttp://acs.amazonaws.com/groups/global/AllUsersREADfooFooFULL_CONTROL 2023-10-05T19:28:56.550 INFO:tasks.radosgw_admin:RL: get_acls myfoo foo bytes_out=0 bytes_in=609 failed=False 2023-10-05T19:28:56.556 INFO:tasks.radosgw_admin:RL: create_bucket myfoo foo bytes_out=0 bytes_in=0 failed=False 2023-10-05T19:28:56.663 INFO:tasks.radosgw_admin:RL: put_obj myfoo foo bytes_out=5 bytes_in=0 failed=False 2023-10-05T19:28:56.663 INFO:tasks.radosgw_admin:RL: put_obj myfoo foo bytes_out=4 bytes_in=0 failed=False 2023-10-05T19:28:56.663 INFO:tasks.radosgw_admin:RL: put_obj myfoo foo bytes_out=3 bytes_in=0 failed=False 2023-10-05T19:28:56.663 INFO:tasks.radosgw_admin:RL: put_obj myfoo foo bytes_out=6 bytes_in=0 failed=False 2023-10-05T19:28:56.663 INFO:tasks.util.rgw:rgwadmin: client.0 : ['bucket', 'rm', '--bucket', 'myfoo', '--purge-objects'] 2023-10-05T19:28:56.664 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'ceph', 'bucket', 'rm', '--bucket', 'myfoo', '--purge-objects'] 2023-10-05T19:28:56.664 DEBUG:teuthology.orchestra.run.smithi167:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster ceph bucket rm --bucket myfoo --purge-objects 2023-10-05T19:28:56.704 INFO:teuthology.orchestra.run.smithi167.stderr:ignoring --setuser ceph since I am not root 2023-10-05T19:28:56.704 INFO:teuthology.orchestra.run.smithi167.stderr:ignoring --setgroup ceph since I am not root 2023-10-05T19:28:56.705 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.698+0000 7f5475af1e40 1 Processor -- start 2023-10-05T19:28:56.705 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.698+0000 7f5475af1e40 1 -- start start 2023-10-05T19:28:56.705 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.698+0000 7f5475af1e40 1 --2- >> v2:172.21.15.118:3300/0 conn(0x565382cabe40 0x565382cac210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-05T19:28:56.705 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.698+0000 7f5475af1e40 1 -- --> v1:172.21.15.118:6789/0 -- auth(proto 0 26 bytes epoch 0) v1 -- 0x565382bf8bd0 con 0x565382cac750 2023-10-05T19:28:56.705 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.698+0000 7f5475af1e40 1 -- --> v2:172.21.15.118:3300/0 -- mon_getmap magic: 0 v1 -- 0x565382bb5ee0 con 0x565382cabe40 2023-10-05T19:28:56.706 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.702+0000 7f5473626700 1 --2- >> v2:172.21.15.118:3300/0 conn(0x565382cabe40 0x565382cac210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-05T19:28:56.706 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.702+0000 7f5472e25700 1 --1- >> v1:172.21.15.118:6789/0 conn(0x565382cac750 0x565382cad1f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.118:6789/0 says I am v1:172.21.15.167:48702/0 (socket says 172.21.15.167:48702) 2023-10-05T19:28:56.706 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.702+0000 7f5472e25700 1 -- 172.21.15.167:0/898431901 learned_addr learned my addr 172.21.15.167:0/898431901 (peer_addr_for_me v1:172.21.15.167:0/0) 2023-10-05T19:28:56.706 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.702+0000 7f5473626700 1 -- 172.21.15.167:0/898431901 >> v1:172.21.15.118:6789/0 conn(0x565382cac750 legacy=0x565382cad1f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-05T19:28:56.707 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.702+0000 7f5473626700 1 -- 172.21.15.167:0/898431901 --> v2:172.21.15.118:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x565382cad900 con 0x565382cabe40 2023-10-05T19:28:56.707 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.702+0000 7f5473626700 1 --2- 172.21.15.167:0/898431901 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x565382cabe40 0x565382cac210 secure :-1 s=READY pgs=549 cs=0 l=1 rev1=1 crypto rx=0x7f546c00a520 tx=0x7f546c005d40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a3b76687ac7daeca server_cookie=0 in_seq=0 out_seq=0 2023-10-05T19:28:56.707 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.702+0000 7f5472624700 1 -- 172.21.15.167:0/898431901 <== mon.0 v2:172.21.15.118:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f546c00f590 con 0x565382cabe40 2023-10-05T19:28:56.707 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.702+0000 7f5472624700 1 -- 172.21.15.167:0/898431901 <== mon.0 v2:172.21.15.118:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f546c00f6f0 con 0x565382cabe40 2023-10-05T19:28:56.707 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.702+0000 7f5472624700 1 -- 172.21.15.167:0/898431901 <== mon.0 v2:172.21.15.118:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f546c00f880 con 0x565382cabe40 2023-10-05T19:28:56.708 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.702+0000 7f5475af1e40 1 -- 172.21.15.167:0/898431901 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x565382cabe40 msgr2=0x565382cac210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-05T19:28:56.708 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.702+0000 7f5475af1e40 1 --2- 172.21.15.167:0/898431901 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x565382cabe40 0x565382cac210 secure :-1 s=READY pgs=549 cs=0 l=1 rev1=1 crypto rx=0x7f546c00a520 tx=0x7f546c005d40 comp rx=0 tx=0).stop 2023-10-05T19:28:56.708 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.702+0000 7f5475af1e40 1 -- 172.21.15.167:0/898431901 shutdown_connections 2023-10-05T19:28:56.709 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.702+0000 7f5475af1e40 1 --2- 172.21.15.167:0/898431901 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x565382cabe40 0x565382cac210 unknown :-1 s=CLOSED pgs=549 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-05T19:28:56.709 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.702+0000 7f5475af1e40 1 -- 172.21.15.167:0/898431901 >> 172.21.15.167:0/898431901 conn(0x565382ca7860 msgr2=0x565382ca9c70 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-05T19:28:56.709 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.702+0000 7f5475af1e40 1 -- 172.21.15.167:0/898431901 shutdown_connections 2023-10-05T19:28:56.709 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.702+0000 7f5475af1e40 1 -- 172.21.15.167:0/898431901 wait complete. 2023-10-05T19:28:56.709 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.702+0000 7f5475af1e40 1 Processor -- start 2023-10-05T19:28:56.710 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.702+0000 7f5475af1e40 1 -- start start 2023-10-05T19:28:56.710 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.702+0000 7f5475af1e40 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x565382cabe40 0x565382ccae70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-05T19:28:56.710 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.702+0000 7f5475af1e40 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x565382b79e30 con 0x565382cabe40 2023-10-05T19:28:56.710 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.706+0000 7f5473626700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x565382cabe40 0x565382ccae70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-05T19:28:56.710 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.706+0000 7f5473626700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x565382cabe40 0x565382ccae70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.118:3300/0 says I am v2:172.21.15.167:53386/0 (socket says 172.21.15.167:53386) 2023-10-05T19:28:56.710 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.706+0000 7f5473626700 1 -- 172.21.15.167:0/531232883 learned_addr learned my addr 172.21.15.167:0/531232883 (peer_addr_for_me v2:172.21.15.167:0/0) 2023-10-05T19:28:56.710 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.706+0000 7f5473626700 1 -- 172.21.15.167:0/531232883 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f546c00a0e0 con 0x565382cabe40 2023-10-05T19:28:56.712 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.706+0000 7f5473626700 1 --2- 172.21.15.167:0/531232883 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x565382cabe40 0x565382ccae70 secure :-1 s=READY pgs=550 cs=0 l=1 rev1=1 crypto rx=0x7f546c005b20 tx=0x7f546c014000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=21b8cad6cf370b64 server_cookie=0 in_seq=0 out_seq=0 2023-10-05T19:28:56.712 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.706+0000 7f5471622700 1 -- 172.21.15.167:0/531232883 <== mon.0 v2:172.21.15.118:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f546c00f860 con 0x565382cabe40 2023-10-05T19:28:56.712 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.706+0000 7f5471622700 1 -- 172.21.15.167:0/531232883 <== mon.0 v2:172.21.15.118:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f546c00f9c0 con 0x565382cabe40 2023-10-05T19:28:56.712 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.706+0000 7f5471622700 1 -- 172.21.15.167:0/531232883 <== mon.0 v2:172.21.15.118:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f546c00fb30 con 0x565382cabe40 2023-10-05T19:28:56.712 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.706+0000 7f5475af1e40 1 -- 172.21.15.167:0/531232883 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x565382cabe40 msgr2=0x565382ccae70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-05T19:28:56.712 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.706+0000 7f5475af1e40 1 --2- 172.21.15.167:0/531232883 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x565382cabe40 0x565382ccae70 secure :-1 s=READY pgs=550 cs=0 l=1 rev1=1 crypto rx=0x7f546c005b20 tx=0x7f546c014000 comp rx=0 tx=0).stop 2023-10-05T19:28:56.712 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.706+0000 7f5475af1e40 1 -- 172.21.15.167:0/531232883 shutdown_connections 2023-10-05T19:28:56.713 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.706+0000 7f5475af1e40 1 --2- 172.21.15.167:0/531232883 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x565382cabe40 0x565382ccae70 unknown :-1 s=CLOSED pgs=550 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-05T19:28:56.713 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.706+0000 7f5475af1e40 1 -- 172.21.15.167:0/531232883 >> 172.21.15.167:0/531232883 conn(0x565382ca7860 msgr2=0x565382cc6a30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-05T19:28:56.713 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.706+0000 7f5475af1e40 1 -- 172.21.15.167:0/531232883 shutdown_connections 2023-10-05T19:28:56.713 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.706+0000 7f5475af1e40 1 -- 172.21.15.167:0/531232883 wait complete. 2023-10-05T19:28:56.713 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.706+0000 7f5475af1e40 1 Processor -- start 2023-10-05T19:28:56.713 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.706+0000 7f5475af1e40 1 -- start start 2023-10-05T19:28:56.713 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.706+0000 7f5475af1e40 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x565382cabe40 0x565382cd1b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-05T19:28:56.713 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.706+0000 7f5475af1e40 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x565382b4ced0 con 0x565382cabe40 2023-10-05T19:28:56.714 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.706+0000 7f5473626700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x565382cabe40 0x565382cd1b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-05T19:28:56.714 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.706+0000 7f5473626700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x565382cabe40 0x565382cd1b70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.118:3300/0 says I am v2:172.21.15.167:53402/0 (socket says 172.21.15.167:53402) 2023-10-05T19:28:56.714 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.706+0000 7f5473626700 1 -- 172.21.15.167:0/1974318067 learned_addr learned my addr 172.21.15.167:0/1974318067 (peer_addr_for_me v2:172.21.15.167:0/0) 2023-10-05T19:28:56.715 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.706+0000 7f5473626700 1 -- 172.21.15.167:0/1974318067 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x565382cad900 con 0x565382cabe40 2023-10-05T19:28:56.715 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.710+0000 7f5473626700 1 --2- 172.21.15.167:0/1974318067 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x565382cabe40 0x565382cd1b70 secure :-1 s=READY pgs=551 cs=0 l=1 rev1=1 crypto rx=0x565382cc7550 tx=0x7f546c01d000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-05T19:28:56.715 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.710+0000 7f546b7fe700 1 -- 172.21.15.167:0/1974318067 <== mon.0 v2:172.21.15.118:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f546c02a070 con 0x565382cabe40 2023-10-05T19:28:56.715 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.710+0000 7f546b7fe700 1 -- 172.21.15.167:0/1974318067 <== mon.0 v2:172.21.15.118:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f546c010420 con 0x565382cabe40 2023-10-05T19:28:56.715 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.710+0000 7f5475af1e40 1 -- 172.21.15.167:0/1974318067 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x565382cd20b0 con 0x565382cabe40 2023-10-05T19:28:56.715 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.710+0000 7f546b7fe700 1 -- 172.21.15.167:0/1974318067 <== mon.0 v2:172.21.15.118:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f546c0105d0 con 0x565382cabe40 2023-10-05T19:28:56.715 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.710+0000 7f5475af1e40 1 -- 172.21.15.167:0/1974318067 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x565382cd2490 con 0x565382cabe40 2023-10-05T19:28:56.716 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.710+0000 7f5475af1e40 20 rados->read ofs=0 len=0 2023-10-05T19:28:56.716 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.710+0000 7f546b7fe700 1 -- 172.21.15.167:0/1974318067 <== mon.0 v2:172.21.15.118:3300/0 4 ==== mgrmap(e 5) v1 ==== 40329+0+0 (secure 0 0 0) 0x7f546c010800 con 0x565382cabe40 2023-10-05T19:28:56.716 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.710+0000 7f546b7fe700 1 --2- 172.21.15.167:0/1974318067 >> [v2:172.21.15.167:6824/29889,v1:172.21.15.167:6825/29889] conn(0x7f5454030ca0 0x7f5454033150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-05T19:28:56.716 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.710+0000 7f546b7fe700 1 -- 172.21.15.167:0/1974318067 <== mon.0 v2:172.21.15.118:3300/0 5 ==== osd_map(32..32 src has 1..32) v4 ==== 6260+0+0 (secure 0 0 0) 0x7f546c0465d0 con 0x565382cabe40 2023-10-05T19:28:56.716 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.710+0000 7f5472e25700 1 --2- 172.21.15.167:0/1974318067 >> [v2:172.21.15.167:6824/29889,v1:172.21.15.167:6825/29889] conn(0x7f5454030ca0 0x7f5454033150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-05T19:28:56.717 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.710+0000 7f5475af1e40 1 --2- 172.21.15.167:0/1974318067 >> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] conn(0x565382cac750 0x565382cdd5e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-05T19:28:56.717 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.710+0000 7f5475af1e40 1 -- 172.21.15.167:0/1974318067 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:1 5.12 5:49953fa1:::default.realm:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32) v8 -- 0x565382cddce0 con 0x565382cac750 2023-10-05T19:28:56.717 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.710+0000 7f5473e27700 1 --2- 172.21.15.167:0/1974318067 >> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] conn(0x565382cac750 0x565382cdd5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-05T19:28:56.717 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.710+0000 7f5472e25700 1 --2- 172.21.15.167:0/1974318067 >> [v2:172.21.15.167:6824/29889,v1:172.21.15.167:6825/29889] conn(0x7f5454030ca0 0x7f5454033150 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7f545c001e90 tx=0x7f545c0013b0 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-05T19:28:56.718 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.714+0000 7f5473e27700 1 --2- 172.21.15.167:0/1974318067 >> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] conn(0x565382cac750 0x565382cdd5e0 crc :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-05T19:28:56.718 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.714+0000 7f5473e27700 1 -- 172.21.15.167:0/1974318067 <== osd.1 v2:172.21.15.167:6800/30035 1 ==== osd_op_reply(1 default.realm [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 ==== 157+0+0 (crc 0 0 0) 0x7f546400a500 con 0x565382cac750 2023-10-05T19:28:56.719 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.714+0000 7f5475af1e40 20 rados_obj.operate() r=-2 bl.length=0 2023-10-05T19:28:56.719 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.714+0000 7f5475af1e40 20 realm 2023-10-05T19:28:56.719 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.714+0000 7f5475af1e40 20 rados->read ofs=0 len=0 2023-10-05T19:28:56.719 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.714+0000 7f5475af1e40 1 -- 172.21.15.167:0/1974318067 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:2 5.12 5:49953fa1:::default.realm:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32) v8 -- 0x565382cdac40 con 0x565382cac750 2023-10-05T19:28:56.720 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.714+0000 7f5473e27700 1 -- 172.21.15.167:0/1974318067 <== osd.1 v2:172.21.15.167:6800/30035 2 ==== osd_op_reply(2 default.realm [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 ==== 157+0+0 (crc 0 0 0) 0x7f546400a500 con 0x565382cac750 2023-10-05T19:28:56.720 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.714+0000 7f5475af1e40 20 rados_obj.operate() r=-2 bl.length=0 2023-10-05T19:28:56.720 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.714+0000 7f5475af1e40 4 RGWPeriod::init failed to init realm id : (2) No such file or directory 2023-10-05T19:28:56.720 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.714+0000 7f5475af1e40 20 rados->read ofs=0 len=0 2023-10-05T19:28:56.720 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.714+0000 7f5475af1e40 1 -- 172.21.15.167:0/1974318067 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:3 5.12 5:49953fa1:::default.realm:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32) v8 -- 0x565382cde7c0 con 0x565382cac750 2023-10-05T19:28:56.720 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.714+0000 7f5473e27700 1 -- 172.21.15.167:0/1974318067 <== osd.1 v2:172.21.15.167:6800/30035 3 ==== osd_op_reply(3 default.realm [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 ==== 157+0+0 (crc 0 0 0) 0x7f546400a500 con 0x565382cac750 2023-10-05T19:28:56.720 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.714+0000 7f5475af1e40 20 rados_obj.operate() r=-2 bl.length=0 2023-10-05T19:28:56.721 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.714+0000 7f5475af1e40 20 rados->read ofs=0 len=0 2023-10-05T19:28:56.721 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.714+0000 7f5475af1e40 1 -- 172.21.15.167:0/1974318067 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:4 5.f 5:f43b6ece:::zone_names.default:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32) v8 -- 0x565382cde3d0 con 0x565382cac750 2023-10-05T19:28:56.722 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.714+0000 7f5473e27700 1 -- 172.21.15.167:0/1974318067 <== osd.1 v2:172.21.15.167:6800/30035 4 ==== osd_op_reply(4 zone_names.default [read 0~46 out=46b] v0'0 uv1 ondisk = 0) v8 ==== 162+0+46 (crc 0 0 0) 0x7f546400a500 con 0x565382cac750 2023-10-05T19:28:56.722 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.714+0000 7f5475af1e40 20 rados_obj.operate() r=0 bl.length=46 2023-10-05T19:28:56.722 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.714+0000 7f5475af1e40 20 rados->read ofs=0 len=0 2023-10-05T19:28:56.722 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.714+0000 7f5475af1e40 1 -- 172.21.15.167:0/1974318067 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:5 5.1d 5:b8374eb4:::zone_info.9a9df965-6612-45d4-a45f-6e3c889bacc2:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32) v8 -- 0x565382cdf650 con 0x565382cac750 2023-10-05T19:28:56.722 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.714+0000 7f5473e27700 1 -- 172.21.15.167:0/1974318067 <== osd.1 v2:172.21.15.167:6800/30035 5 ==== osd_op_reply(5 zone_info.9a9df965-6612-45d4-a45f-6e3c889bacc2 [read 0~874 out=874b] v0'0 uv2 ondisk = 0) v8 ==== 190+0+874 (crc 0 0 0) 0x7f546400a500 con 0x565382cac750 2023-10-05T19:28:56.722 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.714+0000 7f5475af1e40 20 rados_obj.operate() r=0 bl.length=874 2023-10-05T19:28:56.722 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.714+0000 7f5475af1e40 20 searching for the correct realm 2023-10-05T19:28:56.723 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.718+0000 7f5475af1e40 1 --2- 172.21.15.167:0/1974318067 >> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] conn(0x565382ce1410 0x565382ce37f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-05T19:28:56.723 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.718+0000 7f5475af1e40 1 -- 172.21.15.167:0/1974318067 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:6 5.0 5:00000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x565382ce3d30 con 0x565382ce1410 2023-10-05T19:28:56.723 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.718+0000 7f5473626700 1 --2- 172.21.15.167:0/1974318067 >> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] conn(0x565382ce1410 0x565382ce37f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-05T19:28:56.723 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.718+0000 7f5473626700 1 --2- 172.21.15.167:0/1974318067 >> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] conn(0x565382ce1410 0x565382ce37f0 crc :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-05T19:28:56.724 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.718+0000 7f5473626700 1 -- 172.21.15.167:0/1974318067 <== osd.3 v2:172.21.15.167:6808/30036 1 ==== osd_op_reply(6 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f546c034070 con 0x565382ce1410 2023-10-05T19:28:56.724 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.718+0000 7f5473626700 1 --2- 172.21.15.167:0/1974318067 >> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] conn(0x7f546c01b740 0x7f546c01a5f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-05T19:28:56.724 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.718+0000 7f5473626700 1 -- 172.21.15.167:0/1974318067 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:7 5.10 5:08000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x565382ce3d30 con 0x7f546c01b740 2023-10-05T19:28:56.724 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.718+0000 7f5472e25700 1 --2- 172.21.15.167:0/1974318067 >> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] conn(0x7f546c01b740 0x7f546c01a5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-05T19:28:56.724 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.718+0000 7f5472e25700 1 --2- 172.21.15.167:0/1974318067 >> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] conn(0x7f546c01b740 0x7f546c01a5f0 crc :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-05T19:28:56.725 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.718+0000 7f5472e25700 1 -- 172.21.15.167:0/1974318067 <== osd.2 v2:172.21.15.167:6816/30037 1 ==== osd_op_reply(7 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f545c006070 con 0x7f546c01b740 2023-10-05T19:28:56.725 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.718+0000 7f5472e25700 1 --2- 172.21.15.167:0/1974318067 >> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] conn(0x7f545c00d8e0 0x7f545c00c360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-05T19:28:56.725 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.718+0000 7f5472e25700 1 -- 172.21.15.167:0/1974318067 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:8 5.8 5:10000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x565382ce3d30 con 0x7f545c00d8e0 2023-10-05T19:28:56.726 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.722+0000 7f5473e27700 1 --2- 172.21.15.167:0/1974318067 >> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] conn(0x7f545c00d8e0 0x7f545c00c360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-05T19:28:56.726 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.722+0000 7f5473e27700 1 --2- 172.21.15.167:0/1974318067 >> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] conn(0x7f545c00d8e0 0x7f545c00c360 crc :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-05T19:28:56.727 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.722+0000 7f5473e27700 1 -- 172.21.15.167:0/1974318067 <== osd.0 v2:172.21.15.118:6800/27690 1 ==== osd_op_reply(8 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f5464014040 con 0x7f545c00d8e0 2023-10-05T19:28:56.727 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.722+0000 7f5473e27700 1 -- 172.21.15.167:0/1974318067 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:9 5.18 5:18000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x565382ce3d30 con 0x565382cac750 2023-10-05T19:28:56.728 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.722+0000 7f5473e27700 1 -- 172.21.15.167:0/1974318067 <== osd.1 v2:172.21.15.167:6800/30035 6 ==== osd_op_reply(9 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f5464014040 con 0x565382cac750 2023-10-05T19:28:56.728 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.722+0000 7f5473e27700 1 -- 172.21.15.167:0/1974318067 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:10 5.4 5:20000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x565382ce3d30 con 0x565382cac750 2023-10-05T19:28:56.729 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.722+0000 7f5473e27700 1 -- 172.21.15.167:0/1974318067 <== osd.1 v2:172.21.15.167:6800/30035 7 ==== osd_op_reply(10 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f5464014040 con 0x565382cac750 2023-10-05T19:28:56.729 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.722+0000 7f5473e27700 1 -- 172.21.15.167:0/1974318067 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:11 5.14 5:28000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x565382ce3d30 con 0x7f545c00d8e0 2023-10-05T19:28:56.729 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.722+0000 7f5473e27700 1 -- 172.21.15.167:0/1974318067 <== osd.0 v2:172.21.15.118:6800/27690 2 ==== osd_op_reply(11 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f5464014040 con 0x7f545c00d8e0 2023-10-05T19:28:56.729 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.726+0000 7f5473e27700 1 -- 172.21.15.167:0/1974318067 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:12 5.c 5:30000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x565382ce3d30 con 0x7f545c00d8e0 2023-10-05T19:28:56.730 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.726+0000 7f5473e27700 1 -- 172.21.15.167:0/1974318067 <== osd.0 v2:172.21.15.118:6800/27690 3 ==== osd_op_reply(12 [pgnls start_epoch 32 out=112b] v23'1 uv1 ondisk = 1) v8 ==== 144+0+112 (crc 0 0 0) 0x7f5464014040 con 0x7f545c00d8e0 2023-10-05T19:28:56.730 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.726+0000 7f5473e27700 1 -- 172.21.15.167:0/1974318067 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:13 5.1c 5:38000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x565382ce3d30 con 0x7f546c01b740 2023-10-05T19:28:56.731 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.726+0000 7f5472e25700 1 -- 172.21.15.167:0/1974318067 <== osd.2 v2:172.21.15.167:6816/30037 2 ==== osd_op_reply(13 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f545c006070 con 0x7f546c01b740 2023-10-05T19:28:56.731 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.726+0000 7f5472e25700 1 -- 172.21.15.167:0/1974318067 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:14 5.2 5:40000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x565382ce3d30 con 0x565382ce1410 2023-10-05T19:28:56.731 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.726+0000 7f5473626700 1 -- 172.21.15.167:0/1974318067 <== osd.3 v2:172.21.15.167:6808/30036 2 ==== osd_op_reply(14 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f546c034070 con 0x565382ce1410 2023-10-05T19:28:56.732 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.726+0000 7f5473626700 1 -- 172.21.15.167:0/1974318067 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:15 5.12 5:48000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x565382ce3d30 con 0x565382cac750 2023-10-05T19:28:56.732 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.726+0000 7f5473e27700 1 -- 172.21.15.167:0/1974318067 <== osd.1 v2:172.21.15.167:6800/30035 8 ==== osd_op_reply(15 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f5464014040 con 0x565382cac750 2023-10-05T19:28:56.732 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.726+0000 7f5473e27700 1 -- 172.21.15.167:0/1974318067 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:16 5.a 5:50000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x565382ce3d30 con 0x7f545c00d8e0 2023-10-05T19:28:56.733 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.730+0000 7f5473e27700 1 -- 172.21.15.167:0/1974318067 <== osd.0 v2:172.21.15.118:6800/27690 4 ==== osd_op_reply(16 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f5464022040 con 0x7f545c00d8e0 2023-10-05T19:28:56.733 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.730+0000 7f5473e27700 1 -- 172.21.15.167:0/1974318067 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:17 5.1a 5:58000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x565382ce3d30 con 0x565382cac750 2023-10-05T19:28:56.734 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.730+0000 7f5473e27700 1 -- 172.21.15.167:0/1974318067 <== osd.1 v2:172.21.15.167:6800/30035 9 ==== osd_op_reply(17 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f5464022040 con 0x565382cac750 2023-10-05T19:28:56.734 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.730+0000 7f5473e27700 1 -- 172.21.15.167:0/1974318067 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:18 5.6 5:60000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x565382ce3d30 con 0x7f546c01b740 2023-10-05T19:28:56.734 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.730+0000 7f5472e25700 1 -- 172.21.15.167:0/1974318067 <== osd.2 v2:172.21.15.167:6816/30037 3 ==== osd_op_reply(18 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f545c006070 con 0x7f546c01b740 2023-10-05T19:28:56.734 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.730+0000 7f5472e25700 1 -- 172.21.15.167:0/1974318067 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:19 5.16 5:68000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x565382ce3d30 con 0x565382ce1410 2023-10-05T19:28:56.735 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.730+0000 7f5473626700 1 -- 172.21.15.167:0/1974318067 <== osd.3 v2:172.21.15.167:6808/30036 3 ==== osd_op_reply(19 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f546c034070 con 0x565382ce1410 2023-10-05T19:28:56.735 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.730+0000 7f5473626700 1 -- 172.21.15.167:0/1974318067 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:20 5.e 5:70000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x565382ce3d30 con 0x565382ce1410 2023-10-05T19:28:56.736 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.730+0000 7f5473626700 1 -- 172.21.15.167:0/1974318067 <== osd.3 v2:172.21.15.167:6808/30036 4 ==== osd_op_reply(20 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f546c034070 con 0x565382ce1410 2023-10-05T19:28:56.736 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.730+0000 7f5473626700 1 -- 172.21.15.167:0/1974318067 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:21 5.1e 5:78000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x565382ce3d30 con 0x7f545c00d8e0 2023-10-05T19:28:56.737 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.730+0000 7f5473e27700 1 -- 172.21.15.167:0/1974318067 <== osd.0 v2:172.21.15.118:6800/27690 5 ==== osd_op_reply(21 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f5464022040 con 0x7f545c00d8e0 2023-10-05T19:28:56.737 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.730+0000 7f5473e27700 1 -- 172.21.15.167:0/1974318067 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:22 5.1 5:80000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x565382ce3d30 con 0x565382ce1410 2023-10-05T19:28:56.738 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.734+0000 7f5473626700 1 -- 172.21.15.167:0/1974318067 <== osd.3 v2:172.21.15.167:6808/30036 5 ==== osd_op_reply(22 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f546c034070 con 0x565382ce1410 2023-10-05T19:28:56.738 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.734+0000 7f5473626700 1 -- 172.21.15.167:0/1974318067 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:23 5.11 5:88000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x565382ce3d30 con 0x565382ce1410 2023-10-05T19:28:56.738 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.734+0000 7f5473626700 1 -- 172.21.15.167:0/1974318067 <== osd.3 v2:172.21.15.167:6808/30036 6 ==== osd_op_reply(23 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f546c034070 con 0x565382ce1410 2023-10-05T19:28:56.739 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.734+0000 7f5473626700 1 -- 172.21.15.167:0/1974318067 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:24 5.9 5:90000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x565382ce3d30 con 0x565382cac750 2023-10-05T19:28:56.739 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.734+0000 7f5473e27700 1 -- 172.21.15.167:0/1974318067 <== osd.1 v2:172.21.15.167:6800/30035 10 ==== osd_op_reply(24 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f5464022040 con 0x565382cac750 2023-10-05T19:28:56.739 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.734+0000 7f5473e27700 1 -- 172.21.15.167:0/1974318067 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:25 5.19 5:98000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x565382ce3d30 con 0x565382cac750 2023-10-05T19:28:56.740 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.734+0000 7f5473e27700 1 -- 172.21.15.167:0/1974318067 <== osd.1 v2:172.21.15.167:6800/30035 11 ==== osd_op_reply(25 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f5464022040 con 0x565382cac750 2023-10-05T19:28:56.740 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.734+0000 7f5473e27700 1 -- 172.21.15.167:0/1974318067 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:26 5.5 5:a0000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x565382ce3d30 con 0x7f545c00d8e0 2023-10-05T19:28:56.741 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.734+0000 7f5473e27700 1 -- 172.21.15.167:0/1974318067 <== osd.0 v2:172.21.15.118:6800/27690 6 ==== osd_op_reply(26 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f5464022040 con 0x7f545c00d8e0 2023-10-05T19:28:56.741 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.734+0000 7f5473e27700 1 -- 172.21.15.167:0/1974318067 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:27 5.15 5:a8000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x565382ce3d30 con 0x565382ce1410 2023-10-05T19:28:56.742 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.738+0000 7f5473626700 1 -- 172.21.15.167:0/1974318067 <== osd.3 v2:172.21.15.167:6808/30036 7 ==== osd_op_reply(27 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f546c034070 con 0x565382ce1410 2023-10-05T19:28:56.742 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.738+0000 7f5473626700 1 -- 172.21.15.167:0/1974318067 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:28 5.d 5:b0000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x565382ce3d30 con 0x7f546c01b740 2023-10-05T19:28:56.742 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.738+0000 7f5472e25700 1 -- 172.21.15.167:0/1974318067 <== osd.2 v2:172.21.15.167:6816/30037 4 ==== osd_op_reply(28 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f545c006070 con 0x7f546c01b740 2023-10-05T19:28:56.743 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.738+0000 7f5472e25700 1 -- 172.21.15.167:0/1974318067 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:29 5.1d 5:b8000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x565382ce3d30 con 0x565382cac750 2023-10-05T19:28:56.743 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.738+0000 7f5473e27700 1 -- 172.21.15.167:0/1974318067 <== osd.1 v2:172.21.15.167:6800/30035 12 ==== osd_op_reply(29 [pgnls start_epoch 32 out=107b] v30'2 uv2 ondisk = 1) v8 ==== 144+0+107 (crc 0 0 0) 0x7f5464022040 con 0x565382cac750 2023-10-05T19:28:56.743 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.738+0000 7f5473e27700 1 -- 172.21.15.167:0/1974318067 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:30 5.3 5:c0000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x565382ce3d30 con 0x7f545c00d8e0 2023-10-05T19:28:56.744 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.738+0000 7f5473e27700 1 -- 172.21.15.167:0/1974318067 <== osd.0 v2:172.21.15.118:6800/27690 7 ==== osd_op_reply(30 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f5464022040 con 0x7f545c00d8e0 2023-10-05T19:28:56.745 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.738+0000 7f5473e27700 1 -- 172.21.15.167:0/1974318067 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:31 5.13 5:c8000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x565382ce3d30 con 0x565382ce1410 2023-10-05T19:28:56.745 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.742+0000 7f5473626700 1 -- 172.21.15.167:0/1974318067 <== osd.3 v2:172.21.15.167:6808/30036 8 ==== osd_op_reply(31 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f546c034070 con 0x565382ce1410 2023-10-05T19:28:56.745 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.742+0000 7f5473626700 1 -- 172.21.15.167:0/1974318067 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:32 5.b 5:d0000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x565382ce3d30 con 0x7f545c00d8e0 2023-10-05T19:28:56.746 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.742+0000 7f5473e27700 1 -- 172.21.15.167:0/1974318067 <== osd.0 v2:172.21.15.118:6800/27690 8 ==== osd_op_reply(32 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f5464022040 con 0x7f545c00d8e0 2023-10-05T19:28:56.746 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.742+0000 7f5473e27700 1 -- 172.21.15.167:0/1974318067 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:33 5.1b 5:d8000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x565382ce3d30 con 0x7f546c01b740 2023-10-05T19:28:56.747 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.742+0000 7f5472e25700 1 -- 172.21.15.167:0/1974318067 <== osd.2 v2:172.21.15.167:6816/30037 5 ==== osd_op_reply(33 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f545c006070 con 0x7f546c01b740 2023-10-05T19:28:56.747 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.742+0000 7f5472e25700 1 -- 172.21.15.167:0/1974318067 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:34 5.7 5:e0000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x565382ce3d30 con 0x565382ce1410 2023-10-05T19:28:56.748 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.742+0000 7f5473626700 1 -- 172.21.15.167:0/1974318067 <== osd.3 v2:172.21.15.167:6808/30036 9 ==== osd_op_reply(34 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f546c034070 con 0x565382ce1410 2023-10-05T19:28:56.748 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.742+0000 7f5473626700 1 -- 172.21.15.167:0/1974318067 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:35 5.17 5:e8000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x565382ce3d30 con 0x7f545c00d8e0 2023-10-05T19:28:56.749 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.742+0000 7f5473e27700 1 -- 172.21.15.167:0/1974318067 <== osd.0 v2:172.21.15.118:6800/27690 9 ==== osd_op_reply(35 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f5464022040 con 0x7f545c00d8e0 2023-10-05T19:28:56.749 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.742+0000 7f5473e27700 1 -- 172.21.15.167:0/1974318067 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:36 5.f 5:f0000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x565382ce3d30 con 0x565382cac750 2023-10-05T19:28:56.750 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.746+0000 7f5473e27700 1 -- 172.21.15.167:0/1974318067 <== osd.1 v2:172.21.15.167:6800/30035 13 ==== osd_op_reply(36 [pgnls start_epoch 32 out=115b] v23'2 uv2 ondisk = 1) v8 ==== 144+0+115 (crc 0 0 0) 0x7f546402f080 con 0x565382cac750 2023-10-05T19:28:56.750 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.746+0000 7f5473e27700 1 -- 172.21.15.167:0/1974318067 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:37 5.1f 5:f8000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x565382ce3d30 con 0x7f546c01b740 2023-10-05T19:28:56.751 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.746+0000 7f5472e25700 1 -- 172.21.15.167:0/1974318067 <== osd.2 v2:172.21.15.167:6816/30037 6 ==== osd_op_reply(37 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f545c006070 con 0x7f546c01b740 2023-10-05T19:28:56.751 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.746+0000 7f5475af1e40 20 RGWRados::pool_iterate: got zonegroup_info.4f1f719d-9648-417b-b10c-e5d939825419 2023-10-05T19:28:56.751 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.746+0000 7f5475af1e40 20 RGWRados::pool_iterate: got zone_info.9a9df965-6612-45d4-a45f-6e3c889bacc2 2023-10-05T19:28:56.751 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.746+0000 7f5475af1e40 20 RGWRados::pool_iterate: got zone_names.default 2023-10-05T19:28:56.751 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.746+0000 7f5475af1e40 20 RGWRados::pool_iterate: got zonegroups_names.default 2023-10-05T19:28:56.751 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.746+0000 7f5475af1e40 20 rados->read ofs=0 len=0 2023-10-05T19:28:56.751 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.746+0000 7f5475af1e40 1 -- 172.21.15.167:0/1974318067 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:38 5.12 5:49953fa1:::default.realm:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32) v8 -- 0x565382ce01c0 con 0x565382cac750 2023-10-05T19:28:56.752 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.746+0000 7f5473e27700 1 -- 172.21.15.167:0/1974318067 <== osd.1 v2:172.21.15.167:6800/30035 14 ==== osd_op_reply(38 default.realm [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 ==== 157+0+0 (crc 0 0 0) 0x7f546402f080 con 0x565382cac750 2023-10-05T19:28:56.752 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.746+0000 7f5475af1e40 20 rados_obj.operate() r=-2 bl.length=0 2023-10-05T19:28:56.752 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.746+0000 7f5475af1e40 20 rados->read ofs=0 len=0 2023-10-05T19:28:56.752 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.746+0000 7f5475af1e40 1 -- 172.21.15.167:0/1974318067 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:39 5.f 5:f4c53578:::zonegroups_names.default:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32) v8 -- 0x565382ce0ab0 con 0x565382cac750 2023-10-05T19:28:56.753 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.746+0000 7f5473e27700 1 -- 172.21.15.167:0/1974318067 <== osd.1 v2:172.21.15.167:6800/30035 15 ==== osd_op_reply(39 zonegroups_names.default [read 0~46 out=46b] v0'0 uv2 ondisk = 0) v8 ==== 168+0+46 (crc 0 0 0) 0x7f546402f080 con 0x565382cac750 2023-10-05T19:28:56.753 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.746+0000 7f5475af1e40 20 rados_obj.operate() r=0 bl.length=46 2023-10-05T19:28:56.753 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.746+0000 7f5475af1e40 20 rados->read ofs=0 len=0 2023-10-05T19:28:56.753 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.746+0000 7f5475af1e40 1 -- 172.21.15.167:0/1974318067 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:40 5.c 5:30d299a5:::zonegroup_info.4f1f719d-9648-417b-b10c-e5d939825419:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32) v8 -- 0x565382ce4170 con 0x7f545c00d8e0 2023-10-05T19:28:56.754 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.746+0000 7f5473e27700 1 -- 172.21.15.167:0/1974318067 <== osd.0 v2:172.21.15.118:6800/27690 10 ==== osd_op_reply(40 zonegroup_info.4f1f719d-9648-417b-b10c-e5d939825419 [read 0~362 out=362b] v0'0 uv1 ondisk = 0) v8 ==== 195+0+362 (crc 0 0 0) 0x7f546402f080 con 0x7f545c00d8e0 2023-10-05T19:28:56.754 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.746+0000 7f5475af1e40 20 rados_obj.operate() r=0 bl.length=362 2023-10-05T19:28:56.754 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.750+0000 7f5475af1e40 20 rados->read ofs=0 len=0 2023-10-05T19:28:56.754 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.750+0000 7f5475af1e40 1 -- 172.21.15.167:0/1974318067 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:41 5.13 5:cbbf6c60:::converted:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32) v8 -- 0x565382ce4170 con 0x565382ce1410 2023-10-05T19:28:56.755 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.750+0000 7f5473626700 1 -- 172.21.15.167:0/1974318067 <== osd.3 v2:172.21.15.167:6808/30036 10 ==== osd_op_reply(41 converted [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 ==== 153+0+0 (crc 0 0 0) 0x7f546c034070 con 0x565382ce1410 2023-10-05T19:28:56.755 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.750+0000 7f5475af1e40 20 rados_obj.operate() r=-2 bl.length=0 2023-10-05T19:28:56.755 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.750+0000 7f5475af1e40 20 rados->read ofs=0 len=0 2023-10-05T19:28:56.755 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.750+0000 7f5475af1e40 1 -- 172.21.15.167:0/1974318067 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:42 5.12 5:49953fa1:::default.realm:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32) v8 -- 0x565382ce4520 con 0x565382cac750 2023-10-05T19:28:56.755 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.750+0000 7f5473e27700 1 -- 172.21.15.167:0/1974318067 <== osd.1 v2:172.21.15.167:6800/30035 16 ==== osd_op_reply(42 default.realm [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 ==== 157+0+0 (crc 0 0 0) 0x7f546402f080 con 0x565382cac750 2023-10-05T19:28:56.755 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.750+0000 7f5475af1e40 20 rados_obj.operate() r=-2 bl.length=0 2023-10-05T19:28:56.755 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.750+0000 7f5475af1e40 20 rados->read ofs=0 len=0 2023-10-05T19:28:56.755 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.750+0000 7f5475af1e40 1 -- 172.21.15.167:0/1974318067 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:43 5.f 5:f4c53578:::zonegroups_names.default:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32) v8 -- 0x565382ce4920 con 0x565382cac750 2023-10-05T19:28:56.756 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.750+0000 7f5473e27700 1 -- 172.21.15.167:0/1974318067 <== osd.1 v2:172.21.15.167:6800/30035 17 ==== osd_op_reply(43 zonegroups_names.default [read 0~46 out=46b] v0'0 uv2 ondisk = 0) v8 ==== 168+0+46 (crc 0 0 0) 0x7f546402f080 con 0x565382cac750 2023-10-05T19:28:56.756 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.750+0000 7f5475af1e40 20 rados_obj.operate() r=0 bl.length=46 2023-10-05T19:28:56.756 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.750+0000 7f5475af1e40 1 -- 172.21.15.167:0/1974318067 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:44 5.0 5:00000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x565382ce5930 con 0x565382ce1410 2023-10-05T19:28:56.757 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.754+0000 7f5473626700 1 -- 172.21.15.167:0/1974318067 <== osd.3 v2:172.21.15.167:6808/30036 11 ==== osd_op_reply(44 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f546c034070 con 0x565382ce1410 2023-10-05T19:28:56.757 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.754+0000 7f5473626700 1 -- 172.21.15.167:0/1974318067 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:45 5.10 5:08000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x565382ce5930 con 0x7f546c01b740 2023-10-05T19:28:56.758 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.754+0000 7f5472e25700 1 -- 172.21.15.167:0/1974318067 <== osd.2 v2:172.21.15.167:6816/30037 7 ==== osd_op_reply(45 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f545c006070 con 0x7f546c01b740 2023-10-05T19:28:56.758 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.754+0000 7f5472e25700 1 -- 172.21.15.167:0/1974318067 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:46 5.8 5:10000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x565382ce5930 con 0x7f545c00d8e0 2023-10-05T19:28:56.759 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.754+0000 7f5473e27700 1 -- 172.21.15.167:0/1974318067 <== osd.0 v2:172.21.15.118:6800/27690 11 ==== osd_op_reply(46 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f546402f080 con 0x7f545c00d8e0 2023-10-05T19:28:56.759 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.754+0000 7f5473e27700 1 -- 172.21.15.167:0/1974318067 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:47 5.18 5:18000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x565382ce0ab0 con 0x565382cac750 2023-10-05T19:28:56.760 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.754+0000 7f5473e27700 1 -- 172.21.15.167:0/1974318067 <== osd.1 v2:172.21.15.167:6800/30035 18 ==== osd_op_reply(47 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f546402f080 con 0x565382cac750 2023-10-05T19:28:56.760 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.754+0000 7f5473e27700 1 -- 172.21.15.167:0/1974318067 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:48 5.4 5:20000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x565382ce0ab0 con 0x565382cac750 2023-10-05T19:28:56.761 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.754+0000 7f5473e27700 1 -- 172.21.15.167:0/1974318067 <== osd.1 v2:172.21.15.167:6800/30035 19 ==== osd_op_reply(48 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f546402f080 con 0x565382cac750 2023-10-05T19:28:56.761 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.754+0000 7f5473e27700 1 -- 172.21.15.167:0/1974318067 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:49 5.14 5:28000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x565382ce0ab0 con 0x7f545c00d8e0 2023-10-05T19:28:56.762 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.758+0000 7f5473e27700 1 -- 172.21.15.167:0/1974318067 <== osd.0 v2:172.21.15.118:6800/27690 12 ==== osd_op_reply(49 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f546402f080 con 0x7f545c00d8e0 2023-10-05T19:28:56.762 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.758+0000 7f5473e27700 1 -- 172.21.15.167:0/1974318067 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:50 5.c 5:30000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x565382ce0ab0 con 0x7f545c00d8e0 2023-10-05T19:28:56.763 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.758+0000 7f5473e27700 1 -- 172.21.15.167:0/1974318067 <== osd.0 v2:172.21.15.118:6800/27690 13 ==== osd_op_reply(50 [pgnls start_epoch 32 out=112b] v23'1 uv1 ondisk = 1) v8 ==== 144+0+112 (crc 0 0 0) 0x7f546402f080 con 0x7f545c00d8e0 2023-10-05T19:28:56.763 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.758+0000 7f5473e27700 1 -- 172.21.15.167:0/1974318067 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:51 5.1c 5:38000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x565382ce0ab0 con 0x7f546c01b740 2023-10-05T19:28:56.764 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.758+0000 7f5472e25700 1 -- 172.21.15.167:0/1974318067 <== osd.2 v2:172.21.15.167:6816/30037 8 ==== osd_op_reply(51 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f545c006070 con 0x7f546c01b740 2023-10-05T19:28:56.764 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.758+0000 7f5472e25700 1 -- 172.21.15.167:0/1974318067 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:52 5.2 5:40000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x565382ce0ab0 con 0x565382ce1410 2023-10-05T19:28:56.764 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.758+0000 7f5473626700 1 -- 172.21.15.167:0/1974318067 <== osd.3 v2:172.21.15.167:6808/30036 12 ==== osd_op_reply(52 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f546c034070 con 0x565382ce1410 2023-10-05T19:28:56.764 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.758+0000 7f5473626700 1 -- 172.21.15.167:0/1974318067 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:53 5.12 5:48000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x565382ce0ab0 con 0x565382cac750 2023-10-05T19:28:56.766 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.762+0000 7f5473e27700 1 -- 172.21.15.167:0/1974318067 <== osd.1 v2:172.21.15.167:6800/30035 20 ==== osd_op_reply(53 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f546402f080 con 0x565382cac750 2023-10-05T19:28:56.766 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.762+0000 7f5473e27700 1 -- 172.21.15.167:0/1974318067 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:54 5.a 5:50000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x565382ce0ab0 con 0x7f545c00d8e0 2023-10-05T19:28:56.767 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.762+0000 7f5473e27700 1 -- 172.21.15.167:0/1974318067 <== osd.0 v2:172.21.15.118:6800/27690 14 ==== osd_op_reply(54 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f546402f080 con 0x7f545c00d8e0 2023-10-05T19:28:56.767 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.762+0000 7f5473e27700 1 -- 172.21.15.167:0/1974318067 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:55 5.1a 5:58000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x565382ce0ab0 con 0x565382cac750 2023-10-05T19:28:56.767 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.762+0000 7f5473e27700 1 -- 172.21.15.167:0/1974318067 <== osd.1 v2:172.21.15.167:6800/30035 21 ==== osd_op_reply(55 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f546402f080 con 0x565382cac750 2023-10-05T19:28:56.767 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.762+0000 7f5473e27700 1 -- 172.21.15.167:0/1974318067 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:56 5.6 5:60000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x565382ce0ab0 con 0x7f546c01b740 2023-10-05T19:28:56.768 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.762+0000 7f5472e25700 1 -- 172.21.15.167:0/1974318067 <== osd.2 v2:172.21.15.167:6816/30037 9 ==== osd_op_reply(56 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f545c006070 con 0x7f546c01b740 2023-10-05T19:28:56.768 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.762+0000 7f5472e25700 1 -- 172.21.15.167:0/1974318067 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:57 5.16 5:68000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x565382ce0ab0 con 0x565382ce1410 2023-10-05T19:28:56.769 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.762+0000 7f5473626700 1 -- 172.21.15.167:0/1974318067 <== osd.3 v2:172.21.15.167:6808/30036 13 ==== osd_op_reply(57 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f546c034070 con 0x565382ce1410 2023-10-05T19:28:56.769 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.762+0000 7f5473626700 1 -- 172.21.15.167:0/1974318067 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:58 5.e 5:70000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x565382ce0ab0 con 0x565382ce1410 2023-10-05T19:28:56.770 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.766+0000 7f5473626700 1 -- 172.21.15.167:0/1974318067 <== osd.3 v2:172.21.15.167:6808/30036 14 ==== osd_op_reply(58 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f546c034070 con 0x565382ce1410 2023-10-05T19:28:56.770 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.766+0000 7f5473626700 1 -- 172.21.15.167:0/1974318067 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:59 5.1e 5:78000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x565382ce0ab0 con 0x7f545c00d8e0 2023-10-05T19:28:56.771 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.766+0000 7f5473e27700 1 -- 172.21.15.167:0/1974318067 <== osd.0 v2:172.21.15.118:6800/27690 15 ==== osd_op_reply(59 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f546402f080 con 0x7f545c00d8e0 2023-10-05T19:28:56.771 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.766+0000 7f5473e27700 1 -- 172.21.15.167:0/1974318067 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:60 5.1 5:80000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x565382ce0ab0 con 0x565382ce1410 2023-10-05T19:28:56.771 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.766+0000 7f5473626700 1 -- 172.21.15.167:0/1974318067 <== osd.3 v2:172.21.15.167:6808/30036 15 ==== osd_op_reply(60 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f546c034070 con 0x565382ce1410 2023-10-05T19:28:56.772 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.766+0000 7f5473626700 1 -- 172.21.15.167:0/1974318067 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:61 5.11 5:88000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x565382ce0ab0 con 0x565382ce1410 2023-10-05T19:28:56.772 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.766+0000 7f5473626700 1 -- 172.21.15.167:0/1974318067 <== osd.3 v2:172.21.15.167:6808/30036 16 ==== osd_op_reply(61 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f546c034070 con 0x565382ce1410 2023-10-05T19:28:56.772 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.766+0000 7f5473626700 1 -- 172.21.15.167:0/1974318067 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:62 5.9 5:90000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x565382ce0ab0 con 0x565382cac750 2023-10-05T19:28:56.773 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.766+0000 7f5473e27700 1 -- 172.21.15.167:0/1974318067 <== osd.1 v2:172.21.15.167:6800/30035 22 ==== osd_op_reply(62 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f546402f080 con 0x565382cac750 2023-10-05T19:28:56.773 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.770+0000 7f5473e27700 1 -- 172.21.15.167:0/1974318067 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:63 5.19 5:98000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x565382ce0ab0 con 0x565382cac750 2023-10-05T19:28:56.774 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.770+0000 7f5473e27700 1 -- 172.21.15.167:0/1974318067 <== osd.1 v2:172.21.15.167:6800/30035 23 ==== osd_op_reply(63 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f546402f080 con 0x565382cac750 2023-10-05T19:28:56.774 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.770+0000 7f5473e27700 1 -- 172.21.15.167:0/1974318067 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:64 5.5 5:a0000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x565382ce0ab0 con 0x7f545c00d8e0 2023-10-05T19:28:56.775 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.770+0000 7f5473e27700 1 -- 172.21.15.167:0/1974318067 <== osd.0 v2:172.21.15.118:6800/27690 16 ==== osd_op_reply(64 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f546402f080 con 0x7f545c00d8e0 2023-10-05T19:28:56.775 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.770+0000 7f5473e27700 1 -- 172.21.15.167:0/1974318067 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:65 5.15 5:a8000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x565382ce0ab0 con 0x565382ce1410 2023-10-05T19:28:56.776 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.770+0000 7f5473626700 1 -- 172.21.15.167:0/1974318067 <== osd.3 v2:172.21.15.167:6808/30036 17 ==== osd_op_reply(65 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f546c034070 con 0x565382ce1410 2023-10-05T19:28:56.776 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.770+0000 7f5473626700 1 -- 172.21.15.167:0/1974318067 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:66 5.d 5:b0000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x565382ce0ab0 con 0x7f546c01b740 2023-10-05T19:28:56.777 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.770+0000 7f5472e25700 1 -- 172.21.15.167:0/1974318067 <== osd.2 v2:172.21.15.167:6816/30037 10 ==== osd_op_reply(66 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f545c006070 con 0x7f546c01b740 2023-10-05T19:28:56.777 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.770+0000 7f5472e25700 1 -- 172.21.15.167:0/1974318067 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:67 5.1d 5:b8000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x565382ce0ab0 con 0x565382cac750 2023-10-05T19:28:56.778 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.774+0000 7f5473e27700 1 -- 172.21.15.167:0/1974318067 <== osd.1 v2:172.21.15.167:6800/30035 24 ==== osd_op_reply(67 [pgnls start_epoch 32 out=107b] v30'2 uv2 ondisk = 1) v8 ==== 144+0+107 (crc 0 0 0) 0x7f546402f080 con 0x565382cac750 2023-10-05T19:28:56.778 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.774+0000 7f5473e27700 1 -- 172.21.15.167:0/1974318067 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:68 5.3 5:c0000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x565382ce0ab0 con 0x7f545c00d8e0 2023-10-05T19:28:56.779 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.774+0000 7f5473e27700 1 -- 172.21.15.167:0/1974318067 <== osd.0 v2:172.21.15.118:6800/27690 17 ==== osd_op_reply(68 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f546402f080 con 0x7f545c00d8e0 2023-10-05T19:28:56.779 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.774+0000 7f5473e27700 1 -- 172.21.15.167:0/1974318067 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:69 5.13 5:c8000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x565382ce0ab0 con 0x565382ce1410 2023-10-05T19:28:56.780 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.774+0000 7f5473626700 1 -- 172.21.15.167:0/1974318067 <== osd.3 v2:172.21.15.167:6808/30036 18 ==== osd_op_reply(69 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f546c034070 con 0x565382ce1410 2023-10-05T19:28:56.780 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.774+0000 7f5473626700 1 -- 172.21.15.167:0/1974318067 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:70 5.b 5:d0000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x565382ce0ab0 con 0x7f545c00d8e0 2023-10-05T19:28:56.781 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.774+0000 7f5473e27700 1 -- 172.21.15.167:0/1974318067 <== osd.0 v2:172.21.15.118:6800/27690 18 ==== osd_op_reply(70 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f546402f080 con 0x7f545c00d8e0 2023-10-05T19:28:56.781 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.774+0000 7f5473e27700 1 -- 172.21.15.167:0/1974318067 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:71 5.1b 5:d8000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x565382ce0ab0 con 0x7f546c01b740 2023-10-05T19:28:56.781 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.778+0000 7f5472e25700 1 -- 172.21.15.167:0/1974318067 <== osd.2 v2:172.21.15.167:6816/30037 11 ==== osd_op_reply(71 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f545c006070 con 0x7f546c01b740 2023-10-05T19:28:56.782 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.778+0000 7f5472e25700 1 -- 172.21.15.167:0/1974318067 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:72 5.7 5:e0000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x565382ce0ab0 con 0x565382ce1410 2023-10-05T19:28:56.782 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.778+0000 7f5473626700 1 -- 172.21.15.167:0/1974318067 <== osd.3 v2:172.21.15.167:6808/30036 19 ==== osd_op_reply(72 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f546c034070 con 0x565382ce1410 2023-10-05T19:28:56.782 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.778+0000 7f5473626700 1 -- 172.21.15.167:0/1974318067 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:73 5.17 5:e8000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x565382ce0ab0 con 0x7f545c00d8e0 2023-10-05T19:28:56.783 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.778+0000 7f5473e27700 1 -- 172.21.15.167:0/1974318067 <== osd.0 v2:172.21.15.118:6800/27690 19 ==== osd_op_reply(73 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f546402f080 con 0x7f545c00d8e0 2023-10-05T19:28:56.783 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.778+0000 7f5473e27700 1 -- 172.21.15.167:0/1974318067 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:74 5.f 5:f0000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x565382ce0ab0 con 0x565382cac750 2023-10-05T19:28:56.784 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.778+0000 7f5473e27700 1 -- 172.21.15.167:0/1974318067 <== osd.1 v2:172.21.15.167:6800/30035 25 ==== osd_op_reply(74 [pgnls start_epoch 32 out=115b] v23'2 uv2 ondisk = 1) v8 ==== 144+0+115 (crc 0 0 0) 0x7f546402f080 con 0x565382cac750 2023-10-05T19:28:56.784 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.778+0000 7f5473e27700 1 -- 172.21.15.167:0/1974318067 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:75 5.1f 5:f8000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x565382ce0ab0 con 0x7f546c01b740 2023-10-05T19:28:56.785 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.778+0000 7f5472e25700 1 -- 172.21.15.167:0/1974318067 <== osd.2 v2:172.21.15.167:6816/30037 12 ==== osd_op_reply(75 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f545c006070 con 0x7f546c01b740 2023-10-05T19:28:56.785 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.778+0000 7f5475af1e40 20 RGWRados::pool_iterate: got zonegroup_info.4f1f719d-9648-417b-b10c-e5d939825419 2023-10-05T19:28:56.785 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.778+0000 7f5475af1e40 20 RGWRados::pool_iterate: got zone_info.9a9df965-6612-45d4-a45f-6e3c889bacc2 2023-10-05T19:28:56.786 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.778+0000 7f5475af1e40 20 RGWRados::pool_iterate: got zone_names.default 2023-10-05T19:28:56.786 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.778+0000 7f5475af1e40 20 RGWRados::pool_iterate: got zonegroups_names.default 2023-10-05T19:28:56.786 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.778+0000 7f5475af1e40 20 rados->read ofs=0 len=0 2023-10-05T19:28:56.786 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.778+0000 7f5475af1e40 1 -- 172.21.15.167:0/1974318067 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:76 5.f 5:f43b6ece:::zone_names.default:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32) v8 -- 0x565382ce4920 con 0x565382cac750 2023-10-05T19:28:56.786 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.782+0000 7f5473e27700 1 -- 172.21.15.167:0/1974318067 <== osd.1 v2:172.21.15.167:6800/30035 26 ==== osd_op_reply(76 zone_names.default [read 0~46 out=46b] v0'0 uv1 ondisk = 0) v8 ==== 162+0+46 (crc 0 0 0) 0x7f546402f080 con 0x565382cac750 2023-10-05T19:28:56.787 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.782+0000 7f5475af1e40 20 rados_obj.operate() r=0 bl.length=46 2023-10-05T19:28:56.787 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.782+0000 7f5475af1e40 20 rados->read ofs=0 len=0 2023-10-05T19:28:56.787 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.782+0000 7f5475af1e40 1 -- 172.21.15.167:0/1974318067 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:77 5.1d 5:b8374eb4:::zone_info.9a9df965-6612-45d4-a45f-6e3c889bacc2:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32) v8 -- 0x565382ce4cd0 con 0x565382cac750 2023-10-05T19:28:56.788 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.782+0000 7f5473e27700 1 -- 172.21.15.167:0/1974318067 <== osd.1 v2:172.21.15.167:6800/30035 27 ==== osd_op_reply(77 zone_info.9a9df965-6612-45d4-a45f-6e3c889bacc2 [read 0~874 out=874b] v0'0 uv2 ondisk = 0) v8 ==== 190+0+874 (crc 0 0 0) 0x7f546402f080 con 0x565382cac750 2023-10-05T19:28:56.788 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.782+0000 7f5475af1e40 20 rados_obj.operate() r=0 bl.length=874 2023-10-05T19:28:56.788 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.782+0000 7f5475af1e40 20 rados->read ofs=0 len=0 2023-10-05T19:28:56.788 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.782+0000 7f5475af1e40 1 -- 172.21.15.167:0/1974318067 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:78 5.f 5:f4c53578:::zonegroups_names.default:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32) v8 -- 0x565382ce0650 con 0x565382cac750 2023-10-05T19:28:56.788 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.782+0000 7f5473e27700 1 -- 172.21.15.167:0/1974318067 <== osd.1 v2:172.21.15.167:6800/30035 28 ==== osd_op_reply(78 zonegroups_names.default [read 0~46 out=46b] v0'0 uv2 ondisk = 0) v8 ==== 168+0+46 (crc 0 0 0) 0x7f546402f080 con 0x565382cac750 2023-10-05T19:28:56.788 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.782+0000 7f5475af1e40 20 rados_obj.operate() r=0 bl.length=46 2023-10-05T19:28:56.788 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.782+0000 7f5475af1e40 20 rados->read ofs=0 len=0 2023-10-05T19:28:56.788 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.782+0000 7f5475af1e40 1 -- 172.21.15.167:0/1974318067 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:79 5.c 5:30d299a5:::zonegroup_info.4f1f719d-9648-417b-b10c-e5d939825419:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32) v8 -- 0x565382ce0670 con 0x7f545c00d8e0 2023-10-05T19:28:56.789 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.782+0000 7f5473e27700 1 -- 172.21.15.167:0/1974318067 <== osd.0 v2:172.21.15.118:6800/27690 20 ==== osd_op_reply(79 zonegroup_info.4f1f719d-9648-417b-b10c-e5d939825419 [read 0~362 out=362b] v0'0 uv1 ondisk = 0) v8 ==== 195+0+362 (crc 0 0 0) 0x7f546402f080 con 0x7f545c00d8e0 2023-10-05T19:28:56.789 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.782+0000 7f5475af1e40 20 rados_obj.operate() r=0 bl.length=362 2023-10-05T19:28:56.789 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.782+0000 7f5475af1e40 20 rados->read ofs=0 len=0 2023-10-05T19:28:56.789 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.782+0000 7f5475af1e40 1 -- 172.21.15.167:0/1974318067 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:80 5.3 5:c0ad8cbd:::region_map:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32) v8 -- 0x565382ce5740 con 0x7f545c00d8e0 2023-10-05T19:28:56.790 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.786+0000 7f5473e27700 1 -- 172.21.15.167:0/1974318067 <== osd.0 v2:172.21.15.118:6800/27690 21 ==== osd_op_reply(80 region_map [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 ==== 154+0+0 (crc 0 0 0) 0x7f546402f080 con 0x7f545c00d8e0 2023-10-05T19:28:56.790 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.786+0000 7f5475af1e40 20 rados_obj.operate() r=-2 bl.length=0 2023-10-05T19:28:56.790 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.786+0000 7f5475af1e40 20 zone default found 2023-10-05T19:28:56.790 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.786+0000 7f5475af1e40 4 Realm: () 2023-10-05T19:28:56.791 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.786+0000 7f5475af1e40 4 ZoneGroup: default (4f1f719d-9648-417b-b10c-e5d939825419) 2023-10-05T19:28:56.791 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.786+0000 7f5475af1e40 4 Zone: default (9a9df965-6612-45d4-a45f-6e3c889bacc2) 2023-10-05T19:28:56.791 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.786+0000 7f5475af1e40 10 cannot find current period zonegroup using local zonegroup configuration 2023-10-05T19:28:56.791 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.786+0000 7f5475af1e40 20 zonegroup default 2023-10-05T19:28:56.791 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.786+0000 7f5475af1e40 20 rados->read ofs=0 len=0 2023-10-05T19:28:56.791 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.786+0000 7f5475af1e40 1 -- 172.21.15.167:0/1974318067 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:81 5.3 5:c52100b6:::period_config.default:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32) v8 -- 0x565382ce5740 con 0x7f545c00d8e0 2023-10-05T19:28:56.791 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.786+0000 7f5473e27700 1 -- 172.21.15.167:0/1974318067 <== osd.0 v2:172.21.15.118:6800/27690 22 ==== osd_op_reply(81 period_config.default [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 ==== 165+0+0 (crc 0 0 0) 0x7f546402f080 con 0x7f545c00d8e0 2023-10-05T19:28:56.791 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.786+0000 7f5475af1e40 20 rados_obj.operate() r=-2 bl.length=0 2023-10-05T19:28:56.791 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.786+0000 7f5475af1e40 20 rados->read ofs=0 len=0 2023-10-05T19:28:56.791 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.786+0000 7f5475af1e40 1 -- 172.21.15.167:0/1974318067 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:82 6.2 6:4347d321:::bucket.sync-source-hints.:head [call version.read in=11b,read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32) v8 -- 0x565382ce6d60 con 0x565382ce1410 2023-10-05T19:28:56.792 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.786+0000 7f5473626700 1 -- 172.21.15.167:0/1974318067 <== osd.3 v2:172.21.15.167:6808/30036 20 ==== osd_op_reply(82 bucket.sync-source-hints. [call,read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 ==== 211+0+0 (crc 0 0 0) 0x7f546c034070 con 0x565382ce1410 2023-10-05T19:28:56.793 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.786+0000 7f5475af1e40 20 rados_obj.operate() r=-2 bl.length=0 2023-10-05T19:28:56.793 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.786+0000 7f5475af1e40 20 rados->read ofs=0 len=0 2023-10-05T19:28:56.793 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.786+0000 7f5475af1e40 1 -- 172.21.15.167:0/1974318067 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:83 6.b 6:d467b91b:::bucket.sync-target-hints.:head [call version.read in=11b,read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32) v8 -- 0x565382ce7110 con 0x565382ce1410 2023-10-05T19:28:56.793 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.786+0000 7f5473626700 1 -- 172.21.15.167:0/1974318067 <== osd.3 v2:172.21.15.167:6808/30036 21 ==== osd_op_reply(83 bucket.sync-target-hints. [call,read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 ==== 211+0+0 (crc 0 0 0) 0x7f546c034070 con 0x565382ce1410 2023-10-05T19:28:56.793 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.786+0000 7f5475af1e40 20 rados_obj.operate() r=-2 bl.length=0 2023-10-05T19:28:56.793 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.786+0000 7f5475af1e40 20 started sync module instance, tier type = 2023-10-05T19:28:56.793 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.786+0000 7f5475af1e40 20 started zone id=9a9df965-6612-45d4-a45f-6e3c889bacc2 (name=default) with tier type = 2023-10-05T19:28:56.793 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.786+0000 7f5475af1e40 1 -- 172.21.15.167:0/1974318067 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:84 7.1f 7:f95f44c2:::notify.0:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e32) v8 -- 0x565382ce6420 con 0x7f546c01b740 2023-10-05T19:28:56.795 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.790+0000 7f5472e25700 1 -- 172.21.15.167:0/1974318067 <== osd.2 v2:172.21.15.167:6816/30037 13 ==== osd_op_reply(84 notify.0 [create] v32'207 uv207 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f545c006070 con 0x7f546c01b740 2023-10-05T19:28:56.795 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.790+0000 7f5475af1e40 1 -- 172.21.15.167:0/1974318067 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:85 7.1f 7:f95f44c2:::notify.0:head [watch watch cookie 94916676839376] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e32) v8 -- 0x565382ce6050 con 0x7f546c01b740 2023-10-05T19:28:56.795 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.790+0000 7f5475af1e40 1 -- 172.21.15.167:0/1974318067 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:86 7.0 7:05bf5b68:::notify.1:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e32) v8 -- 0x565382ce7ea0 con 0x565382cac750 2023-10-05T19:28:56.798 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.794+0000 7f5473e27700 1 -- 172.21.15.167:0/1974318067 <== osd.1 v2:172.21.15.167:6800/30035 29 ==== osd_op_reply(86 notify.1 [create] v32'207 uv207 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f546402f080 con 0x565382cac750 2023-10-05T19:28:56.798 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.794+0000 7f5472e25700 1 -- 172.21.15.167:0/1974318067 <== osd.2 v2:172.21.15.167:6816/30037 14 ==== osd_op_reply(85 notify.0 [watch watch cookie 94916676839376] v32'208 uv207 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f545c006070 con 0x7f546c01b740 2023-10-05T19:28:56.798 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.794+0000 7f5475af1e40 1 -- 172.21.15.167:0/1974318067 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:87 7.0 7:05bf5b68:::notify.1:head [watch watch cookie 94916676842976] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e32) v8 -- 0x565382ce82a0 con 0x565382cac750 2023-10-05T19:28:56.798 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.794+0000 7f5475af1e40 1 -- 172.21.15.167:0/1974318067 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:88 7.15 7:a93a5511:::notify.2:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e32) v8 -- 0x565382ce8b40 con 0x565382ce1410 2023-10-05T19:28:56.801 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.794+0000 7f5473e27700 1 -- 172.21.15.167:0/1974318067 <== osd.1 v2:172.21.15.167:6800/30035 30 ==== osd_op_reply(87 notify.1 [watch watch cookie 94916676842976] v32'208 uv207 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f5464050040 con 0x565382cac750 2023-10-05T19:28:56.802 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.798+0000 7f5473626700 1 -- 172.21.15.167:0/1974318067 <== osd.3 v2:172.21.15.167:6808/30036 22 ==== osd_op_reply(88 notify.2 [create] v32'207 uv207 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f546c034070 con 0x565382ce1410 2023-10-05T19:28:56.802 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.798+0000 7f5475af1e40 1 -- 172.21.15.167:0/1974318067 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:89 7.15 7:a93a5511:::notify.2:head [watch watch cookie 94916676844080] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e32) v8 -- 0x565382ce8790 con 0x565382ce1410 2023-10-05T19:28:56.802 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.798+0000 7f5475af1e40 1 -- 172.21.15.167:0/1974318067 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:90 7.e 7:7759931f:::notify.3:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e32) v8 -- 0x565382ce9680 con 0x7f546c01b740 2023-10-05T19:28:56.805 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.798+0000 7f5473626700 1 -- 172.21.15.167:0/1974318067 <== osd.3 v2:172.21.15.167:6808/30036 23 ==== osd_op_reply(89 notify.2 [watch watch cookie 94916676844080] v32'208 uv207 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f546c034070 con 0x565382ce1410 2023-10-05T19:28:56.805 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.798+0000 7f5472e25700 1 -- 172.21.15.167:0/1974318067 <== osd.2 v2:172.21.15.167:6816/30037 15 ==== osd_op_reply(90 notify.3 [create] v32'207 uv207 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f545c006070 con 0x7f546c01b740 2023-10-05T19:28:56.805 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.798+0000 7f5475af1e40 1 -- 172.21.15.167:0/1974318067 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:91 7.e 7:7759931f:::notify.3:head [watch watch cookie 94916676846240] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e32) v8 -- 0x565382ce92b0 con 0x7f546c01b740 2023-10-05T19:28:56.805 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.798+0000 7f5475af1e40 1 -- 172.21.15.167:0/1974318067 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:92 7.d 7:b4812045:::notify.4:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e32) v8 -- 0x565382cea720 con 0x565382cac750 2023-10-05T19:28:56.807 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.802+0000 7f5473e27700 1 -- 172.21.15.167:0/1974318067 <== osd.1 v2:172.21.15.167:6800/30035 31 ==== osd_op_reply(92 notify.4 [create] v32'207 uv207 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f5464050040 con 0x565382cac750 2023-10-05T19:28:56.807 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.802+0000 7f5475af1e40 1 -- 172.21.15.167:0/1974318067 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:93 7.d 7:b4812045:::notify.4:head [watch watch cookie 94916676854256] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e32) v8 -- 0x565382ceb010 con 0x565382cac750 2023-10-05T19:28:56.807 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.802+0000 7f5475af1e40 1 -- 172.21.15.167:0/1974318067 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:94 7.3 7:c609908c:::notify.5:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e32) v8 -- 0x565382cebac0 con 0x7f545c00d8e0 2023-10-05T19:28:56.808 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.802+0000 7f5472e25700 1 -- 172.21.15.167:0/1974318067 <== osd.2 v2:172.21.15.167:6816/30037 16 ==== osd_op_reply(91 notify.3 [watch watch cookie 94916676846240] v32'208 uv207 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f545c006070 con 0x7f546c01b740 2023-10-05T19:28:56.809 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.806+0000 7f5473e27700 1 -- 172.21.15.167:0/1974318067 <== osd.1 v2:172.21.15.167:6800/30035 32 ==== osd_op_reply(93 notify.4 [watch watch cookie 94916676854256] v32'208 uv207 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f5464050040 con 0x565382cac750 2023-10-05T19:28:56.810 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.806+0000 7f5473e27700 1 -- 172.21.15.167:0/1974318067 <== osd.0 v2:172.21.15.118:6800/27690 23 ==== osd_op_reply(94 notify.5 [create] v32'207 uv207 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f5464050040 con 0x7f545c00d8e0 2023-10-05T19:28:56.810 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.806+0000 7f5475af1e40 1 -- 172.21.15.167:0/1974318067 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:95 7.3 7:c609908c:::notify.5:head [watch watch cookie 94916676852352] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e32) v8 -- 0x565382ceaab0 con 0x7f545c00d8e0 2023-10-05T19:28:56.810 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.806+0000 7f5475af1e40 1 -- 172.21.15.167:0/1974318067 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:96 7.14 7:2b04a3e9:::notify.6:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e32) v8 -- 0x565382cec3b0 con 0x7f545c00d8e0 2023-10-05T19:28:56.812 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.806+0000 7f5473e27700 1 -- 172.21.15.167:0/1974318067 <== osd.0 v2:172.21.15.118:6800/27690 24 ==== osd_op_reply(96 notify.6 [create] v32'207 uv207 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f5464050040 con 0x7f545c00d8e0 2023-10-05T19:28:56.813 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.806+0000 7f5475af1e40 1 -- 172.21.15.167:0/1974318067 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:97 7.14 7:2b04a3e9:::notify.6:head [watch watch cookie 94916676861168] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e32) v8 -- 0x565382ceaab0 con 0x7f545c00d8e0 2023-10-05T19:28:56.813 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.806+0000 7f5475af1e40 1 -- 172.21.15.167:0/1974318067 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:98 7.9 7:93e5b521:::notify.7:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e32) v8 -- 0x565382cecde0 con 0x7f545c00d8e0 2023-10-05T19:28:56.813 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.806+0000 7f5473e27700 1 -- 172.21.15.167:0/1974318067 <== osd.0 v2:172.21.15.118:6800/27690 25 ==== osd_op_reply(95 notify.5 [watch watch cookie 94916676852352] v32'208 uv207 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f5464050040 con 0x7f545c00d8e0 2023-10-05T19:28:56.815 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.810+0000 7f5473e27700 1 -- 172.21.15.167:0/1974318067 <== osd.0 v2:172.21.15.118:6800/27690 26 ==== osd_op_reply(97 notify.6 [watch watch cookie 94916676861168] v32'208 uv207 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f5464050040 con 0x7f545c00d8e0 2023-10-05T19:28:56.815 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.810+0000 7f5473e27700 1 -- 172.21.15.167:0/1974318067 <== osd.0 v2:172.21.15.118:6800/27690 27 ==== osd_op_reply(98 notify.7 [create] v32'207 uv207 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f5464050040 con 0x7f545c00d8e0 2023-10-05T19:28:56.815 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.810+0000 7f5475af1e40 1 -- 172.21.15.167:0/1974318067 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:99 7.9 7:93e5b521:::notify.7:head [watch watch cookie 94916676862272] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e32) v8 -- 0x565382cec8b0 con 0x7f545c00d8e0 2023-10-05T19:28:56.815 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.810+0000 7f5475af1e40 20 add_watcher() i=0 2023-10-05T19:28:56.815 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.810+0000 7f5475af1e40 20 add_watcher() i=1 2023-10-05T19:28:56.815 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.810+0000 7f5475af1e40 20 add_watcher() i=2 2023-10-05T19:28:56.815 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.810+0000 7f5475af1e40 20 add_watcher() i=3 2023-10-05T19:28:56.816 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.810+0000 7f5475af1e40 20 add_watcher() i=4 2023-10-05T19:28:56.816 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.810+0000 7f5475af1e40 20 add_watcher() i=5 2023-10-05T19:28:56.816 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.810+0000 7f5475af1e40 20 add_watcher() i=6 2023-10-05T19:28:56.816 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.810+0000 7f5473e27700 1 -- 172.21.15.167:0/1974318067 <== osd.0 v2:172.21.15.118:6800/27690 28 ==== osd_op_reply(99 notify.7 [watch watch cookie 94916676862272] v32'208 uv207 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f5464050040 con 0x7f545c00d8e0 2023-10-05T19:28:56.817 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.810+0000 7f5475af1e40 20 add_watcher() i=7 2023-10-05T19:28:56.817 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.810+0000 7f5475af1e40 2 all 8 watchers are set, enabling cache 2023-10-05T19:28:56.817 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.810+0000 7f5475af1e40 1 -- 172.21.15.167:0/1974318067 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:100 6.4 6:22d26bf9:::data_loggenerations_metadata:head [call version.check_conds in=74b,call version.read in=11b,read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32) v8 -- 0x565382cec8b0 con 0x565382cac750 2023-10-05T19:28:56.817 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.810+0000 7f5473e27700 1 -- 172.21.15.167:0/1974318067 <== osd.1 v2:172.21.15.167:6800/30035 33 ==== osd_op_reply(100 data_loggenerations_metadata [call,call out=48b,read 0~28 out=28b] v0'0 uv1 ondisk = 0) v8 ==== 256+0+76 (crc 0 0 0) 0x7f5464050040 con 0x565382cac750 2023-10-05T19:28:56.817 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.810+0000 7f5475af1e40 1 -- 172.21.15.167:0/1974318067 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:101 6.4 6:22d26bf9:::data_loggenerations_metadata:head [watch watch cookie 94916676859280] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e32) v8 -- 0x565382ced2c0 con 0x565382cac750 2023-10-05T19:28:56.819 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.814+0000 7f5473e27700 1 -- 172.21.15.167:0/1974318067 <== osd.1 v2:172.21.15.167:6800/30035 34 ==== osd_op_reply(101 data_loggenerations_metadata [watch watch cookie 94916676859280] v32'205 uv1 ondisk = 0) v8 ==== 172+0+0 (crc 0 0 0) 0x7f5464050040 con 0x565382cac750 2023-10-05T19:28:56.819 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.814+0000 7f5444fe1700 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2023-10-05T19:28:56.819 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.814+0000 7f5475af1e40 20 check_secure_mon_conn(): auth registy supported: methods=[2] modes=[2,1] 2023-10-05T19:28:56.819 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.814+0000 7f5475af1e40 20 check_secure_mon_conn(): mode 1 is insecure 2023-10-05T19:28:56.820 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.814+0000 7f5475af1e40 1 Processor -- start 2023-10-05T19:28:56.820 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.814+0000 7f5475af1e40 1 -- start start 2023-10-05T19:28:56.820 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.814+0000 7f5475af1e40 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x565382cfa260 0x565382cfa630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-05T19:28:56.820 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.814+0000 7f5475af1e40 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x565382cfab70 con 0x565382cfa260 2023-10-05T19:28:56.821 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.814+0000 7f5472e25700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x565382cfa260 0x565382cfa630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-05T19:28:56.821 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.814+0000 7f5472e25700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x565382cfa260 0x565382cfa630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.118:3300/0 says I am v2:172.21.15.167:53418/0 (socket says 172.21.15.167:53418) 2023-10-05T19:28:56.821 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.814+0000 7f5472e25700 1 -- 172.21.15.167:0/3955177280 learned_addr learned my addr 172.21.15.167:0/3955177280 (peer_addr_for_me v2:172.21.15.167:0/0) 2023-10-05T19:28:56.821 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.814+0000 7f5472e25700 1 -- 172.21.15.167:0/3955177280 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x565382cfacb0 con 0x565382cfa260 2023-10-05T19:28:56.822 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.818+0000 7f5472e25700 1 --2- 172.21.15.167:0/3955177280 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x565382cfa260 0x565382cfa630 secure :-1 s=READY pgs=552 cs=0 l=1 rev1=1 crypto rx=0x565382cdeb70 tx=0x7f545c006000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=14e1aea4859b001f server_cookie=0 in_seq=0 out_seq=0 2023-10-05T19:28:56.822 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.818+0000 7f53bffff700 1 -- 172.21.15.167:0/3955177280 <== mon.0 v2:172.21.15.118:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f545c023040 con 0x565382cfa260 2023-10-05T19:28:56.822 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.818+0000 7f53bffff700 1 -- 172.21.15.167:0/3955177280 <== mon.0 v2:172.21.15.118:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f545c024040 con 0x565382cfa260 2023-10-05T19:28:56.822 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.818+0000 7f53bffff700 1 -- 172.21.15.167:0/3955177280 <== mon.0 v2:172.21.15.118:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f545c013040 con 0x565382cfa260 2023-10-05T19:28:56.823 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.818+0000 7f5475af1e40 1 -- 172.21.15.167:0/3955177280 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x565382cfa260 msgr2=0x565382cfa630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-05T19:28:56.823 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.818+0000 7f5475af1e40 1 --2- 172.21.15.167:0/3955177280 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x565382cfa260 0x565382cfa630 secure :-1 s=READY pgs=552 cs=0 l=1 rev1=1 crypto rx=0x565382cdeb70 tx=0x7f545c006000 comp rx=0 tx=0).stop 2023-10-05T19:28:56.823 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.818+0000 7f5475af1e40 1 -- 172.21.15.167:0/3955177280 shutdown_connections 2023-10-05T19:28:56.823 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.818+0000 7f5475af1e40 1 --2- 172.21.15.167:0/3955177280 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x565382cfa260 0x565382cfa630 unknown :-1 s=CLOSED pgs=552 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-05T19:28:56.823 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.818+0000 7f5475af1e40 1 -- 172.21.15.167:0/3955177280 >> 172.21.15.167:0/3955177280 conn(0x565382cf5620 msgr2=0x565382cf3770 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-05T19:28:56.823 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.818+0000 7f5475af1e40 1 -- 172.21.15.167:0/3955177280 shutdown_connections 2023-10-05T19:28:56.823 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.818+0000 7f5475af1e40 1 -- 172.21.15.167:0/3955177280 wait complete. 2023-10-05T19:28:56.824 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.818+0000 7f5475af1e40 1 Processor -- start 2023-10-05T19:28:56.824 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.818+0000 7f5475af1e40 1 -- start start 2023-10-05T19:28:56.824 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.818+0000 7f5475af1e40 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x565382cfff90 0x565382d00360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-05T19:28:56.825 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.818+0000 7f5475af1e40 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x565382d00f90 con 0x565382cfff90 2023-10-05T19:28:56.825 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.818+0000 7f5472e25700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x565382cfff90 0x565382d00360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-05T19:28:56.825 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.818+0000 7f5472e25700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x565382cfff90 0x565382d00360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.118:3300/0 says I am v2:172.21.15.167:53434/0 (socket says 172.21.15.167:53434) 2023-10-05T19:28:56.825 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.818+0000 7f5472e25700 1 -- 172.21.15.167:0/4231517449 learned_addr learned my addr 172.21.15.167:0/4231517449 (peer_addr_for_me v2:172.21.15.167:0/0) 2023-10-05T19:28:56.825 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.818+0000 7f5472e25700 1 -- 172.21.15.167:0/4231517449 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x565382bf8bd0 con 0x565382cfff90 2023-10-05T19:28:56.826 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.818+0000 7f5472e25700 1 --2- 172.21.15.167:0/4231517449 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x565382cfff90 0x565382d00360 secure :-1 s=READY pgs=553 cs=0 l=1 rev1=1 crypto rx=0x565382cdb070 tx=0x7f545c030000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-05T19:28:56.826 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.818+0000 7f53be7fc700 1 -- 172.21.15.167:0/4231517449 <== mon.0 v2:172.21.15.118:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f545c023040 con 0x565382cfff90 2023-10-05T19:28:56.826 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.822+0000 7f53be7fc700 1 -- 172.21.15.167:0/4231517449 <== mon.0 v2:172.21.15.118:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f545c03a420 con 0x565382cfff90 2023-10-05T19:28:56.826 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.822+0000 7f5475af1e40 1 -- 172.21.15.167:0/4231517449 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x565382d00960 con 0x565382cfff90 2023-10-05T19:28:56.826 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.822+0000 7f53be7fc700 1 -- 172.21.15.167:0/4231517449 <== mon.0 v2:172.21.15.118:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f545c03a590 con 0x565382cfff90 2023-10-05T19:28:56.826 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.822+0000 7f5475af1e40 1 -- 172.21.15.167:0/4231517449 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x565382d034b0 con 0x565382cfff90 2023-10-05T19:28:56.827 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.822+0000 7f53be7fc700 1 -- 172.21.15.167:0/4231517449 <== mon.0 v2:172.21.15.118:3300/0 4 ==== mgrmap(e 5) v1 ==== 40329+0+0 (secure 0 0 0) 0x7f545c03a6f0 con 0x565382cfff90 2023-10-05T19:28:56.827 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.822+0000 7f53be7fc700 1 --2- 172.21.15.167:0/4231517449 >> [v2:172.21.15.167:6824/29889,v1:172.21.15.167:6825/29889] conn(0x7f53b4030c50 0x7f53b4033100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-05T19:28:56.827 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.822+0000 7f53be7fc700 1 -- 172.21.15.167:0/4231517449 <== mon.0 v2:172.21.15.118:3300/0 5 ==== osd_map(32..32 src has 1..32) v4 ==== 6260+0+0 (secure 0 0 0) 0x7f545c04fd30 con 0x565382cfff90 2023-10-05T19:28:56.827 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.822+0000 7f5473e27700 1 --2- 172.21.15.167:0/4231517449 >> [v2:172.21.15.167:6824/29889,v1:172.21.15.167:6825/29889] conn(0x7f53b4030c50 0x7f53b4033100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-05T19:28:56.827 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.822+0000 7f5475af1e40 1 --2- 172.21.15.167:0/4231517449 >> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] conn(0x565382d0afe0 0x565382d0d3f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-05T19:28:56.828 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.822+0000 7f5475af1e40 1 -- 172.21.15.167:0/4231517449 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:1 6.19 6:9a6dd8c0:gc::gc.0:head [create,call version.check_conds in=74b,call rgw_gc.rgw_gc_queue_init in=45b,call version.set in=34b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e32) v8 -- 0x565382d0da40 con 0x565382d0afe0 2023-10-05T19:28:56.828 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.822+0000 7f5473626700 1 --2- 172.21.15.167:0/4231517449 >> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] conn(0x565382d0afe0 0x565382d0d3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-05T19:28:56.828 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.822+0000 7f5473e27700 1 --2- 172.21.15.167:0/4231517449 >> [v2:172.21.15.167:6824/29889,v1:172.21.15.167:6825/29889] conn(0x7f53b4030c50 0x7f53b4033100 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x565382cec920 tx=0x7f5464000f90 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-05T19:28:56.828 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.822+0000 7f5473626700 1 --2- 172.21.15.167:0/4231517449 >> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] conn(0x565382d0afe0 0x565382d0d3f0 crc :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-05T19:28:56.831 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.826+0000 7f5473626700 1 -- 172.21.15.167:0/4231517449 <== osd.1 v2:172.21.15.167:6800/30035 1 ==== osd_op_reply(1 gc.0 [create,call,call,call] v32'36 uv33 ondisk = -125 ((125) Operation canceled)) v8 ==== 274+0+0 (crc 0 0 0) 0x7f546c034070 con 0x565382d0afe0 2023-10-05T19:28:56.831 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.826+0000 7f5475af1e40 1 --2- 172.21.15.167:0/4231517449 >> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] conn(0x565382d0ea90 0x565382d10e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-05T19:28:56.831 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.826+0000 7f5475af1e40 1 -- 172.21.15.167:0/4231517449 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:2 6.18 6:1e85421f:gc::gc.1:head [create,call version.check_conds in=74b,call rgw_gc.rgw_gc_queue_init in=45b,call version.set in=34b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e32) v8 -- 0x565382d115a0 con 0x565382d0ea90 2023-10-05T19:28:56.831 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.826+0000 7f5472e25700 1 --2- 172.21.15.167:0/4231517449 >> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] conn(0x565382d0ea90 0x565382d10e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-05T19:28:56.832 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.826+0000 7f5472e25700 1 --2- 172.21.15.167:0/4231517449 >> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] conn(0x565382d0ea90 0x565382d10e70 crc :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-05T19:28:56.834 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.830+0000 7f5472e25700 1 -- 172.21.15.167:0/4231517449 <== osd.0 v2:172.21.15.118:6800/27690 1 ==== osd_op_reply(2 gc.1 [create,call,call,call] v32'28 uv25 ondisk = -125 ((125) Operation canceled)) v8 ==== 274+0+0 (crc 0 0 0) 0x7f545c006070 con 0x565382d0ea90 2023-10-05T19:28:56.834 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.830+0000 7f5475af1e40 1 --2- 172.21.15.167:0/4231517449 >> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] conn(0x565382d12760 0x565382d14b40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-05T19:28:56.834 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.830+0000 7f5475af1e40 1 -- 172.21.15.167:0/4231517449 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:3 6.b 6:d242335b:gc::gc.2:head [create,call version.check_conds in=74b,call rgw_gc.rgw_gc_queue_init in=45b,call version.set in=34b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e32) v8 -- 0x565382d15160 con 0x565382d12760 2023-10-05T19:28:56.834 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.830+0000 7f5473e27700 1 --2- 172.21.15.167:0/4231517449 >> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] conn(0x565382d12760 0x565382d14b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-05T19:28:56.835 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.830+0000 7f5473e27700 1 --2- 172.21.15.167:0/4231517449 >> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] conn(0x565382d12760 0x565382d14b40 crc :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-05T19:28:56.837 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.834+0000 7f5473e27700 1 -- 172.21.15.167:0/4231517449 <== osd.3 v2:172.21.15.167:6808/30036 1 ==== osd_op_reply(3 gc.2 [create,call,call,call] v32'24 uv21 ondisk = -125 ((125) Operation canceled)) v8 ==== 274+0+0 (crc 0 0 0) 0x7f5464050040 con 0x565382d12760 2023-10-05T19:28:56.837 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.834+0000 7f5475af1e40 1 -- 172.21.15.167:0/4231517449 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:4 6.4 6:2166d8e1:gc::gc.3:head [create,call version.check_conds in=74b,call rgw_gc.rgw_gc_queue_init in=45b,call version.set in=34b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e32) v8 -- 0x565382d157b0 con 0x565382d0afe0 2023-10-05T19:28:56.840 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.834+0000 7f5473626700 1 -- 172.21.15.167:0/4231517449 <== osd.1 v2:172.21.15.167:6800/30035 2 ==== osd_op_reply(4 gc.3 [create,call,call,call] v32'206 uv184 ondisk = -125 ((125) Operation canceled)) v8 ==== 274+0+0 (crc 0 0 0) 0x7f546c034070 con 0x565382d0afe0 2023-10-05T19:28:56.840 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.834+0000 7f5475af1e40 1 --2- 172.21.15.167:0/4231517449 >> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] conn(0x565382d16450 0x565382d18830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-05T19:28:56.840 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.834+0000 7f5475af1e40 1 -- 172.21.15.167:0/4231517449 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:5 6.3 6:c60da066:gc::gc.4:head [create,call version.check_conds in=74b,call rgw_gc.rgw_gc_queue_init in=45b,call version.set in=34b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e32) v8 -- 0x565382d18ee0 con 0x565382d16450 2023-10-05T19:28:56.841 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.834+0000 7f5473626700 1 --2- 172.21.15.167:0/4231517449 >> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] conn(0x565382d16450 0x565382d18830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-05T19:28:56.841 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.834+0000 7f5473626700 1 --2- 172.21.15.167:0/4231517449 >> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] conn(0x565382d16450 0x565382d18830 crc :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-05T19:28:56.844 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.838+0000 7f5473626700 1 -- 172.21.15.167:0/4231517449 <== osd.2 v2:172.21.15.167:6816/30037 1 ==== osd_op_reply(5 gc.4 [create,call,call,call] v32'62 uv51 ondisk = -125 ((125) Operation canceled)) v8 ==== 274+0+0 (crc 0 0 0) 0x7f546c034070 con 0x565382d16450 2023-10-05T19:28:56.844 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.838+0000 7f5475af1e40 1 -- 172.21.15.167:0/4231517449 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:6 6.9 6:954be8d2:gc::gc.5:head [create,call version.check_conds in=74b,call rgw_gc.rgw_gc_queue_init in=45b,call version.set in=34b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e32) v8 -- 0x565382d19500 con 0x565382d12760 2023-10-05T19:28:56.846 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.842+0000 7f5473e27700 1 -- 172.21.15.167:0/4231517449 <== osd.3 v2:172.21.15.167:6808/30036 2 ==== osd_op_reply(6 gc.5 [create,call,call,call] v32'44 uv37 ondisk = -125 ((125) Operation canceled)) v8 ==== 274+0+0 (crc 0 0 0) 0x7f5464050040 con 0x565382d12760 2023-10-05T19:28:56.846 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.842+0000 7f5475af1e40 1 -- 172.21.15.167:0/4231517449 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:7 6.e 6:748722ff:gc::gc.6:head [create,call version.check_conds in=74b,call rgw_gc.rgw_gc_queue_init in=45b,call version.set in=34b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e32) v8 -- 0x565382d19b20 con 0x565382d0afe0 2023-10-05T19:28:56.849 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.842+0000 7f5473626700 1 -- 172.21.15.167:0/4231517449 <== osd.1 v2:172.21.15.167:6800/30035 3 ==== osd_op_reply(7 gc.6 [create,call,call,call] v32'26 uv23 ondisk = -125 ((125) Operation canceled)) v8 ==== 274+0+0 (crc 0 0 0) 0x7f546c034070 con 0x565382d0afe0 2023-10-05T19:28:56.849 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.842+0000 7f5475af1e40 1 -- 172.21.15.167:0/4231517449 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:8 6.1d 6:beaa4be4:gc::gc.7:head [create,call version.check_conds in=74b,call rgw_gc.rgw_gc_queue_init in=45b,call version.set in=34b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e32) v8 -- 0x565382d1a010 con 0x565382d0ea90 2023-10-05T19:28:56.851 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.846+0000 7f5472e25700 1 -- 172.21.15.167:0/4231517449 <== osd.0 v2:172.21.15.118:6800/27690 2 ==== osd_op_reply(8 gc.7 [create,call,call,call] v32'41 uv34 ondisk = -125 ((125) Operation canceled)) v8 ==== 274+0+0 (crc 0 0 0) 0x7f545c006070 con 0x565382d0ea90 2023-10-05T19:28:56.852 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.846+0000 7f5475af1e40 1 -- 172.21.15.167:0/4231517449 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:9 6.12 6:4f78e9a2:gc::gc.8:head [create,call version.check_conds in=74b,call rgw_gc.rgw_gc_queue_init in=45b,call version.set in=34b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e32) v8 -- 0x565382d067d0 con 0x565382d16450 2023-10-05T19:28:56.854 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.850+0000 7f5473626700 1 -- 172.21.15.167:0/4231517449 <== osd.2 v2:172.21.15.167:6816/30037 2 ==== osd_op_reply(9 gc.8 [create,call,call,call] v32'24 uv21 ondisk = -125 ((125) Operation canceled)) v8 ==== 274+0+0 (crc 0 0 0) 0x7f546c034070 con 0x565382d16450 2023-10-05T19:28:56.854 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.850+0000 7f5475af1e40 1 -- 172.21.15.167:0/4231517449 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:10 6.1c 6:3e426232:gc::gc.9:head [create,call version.check_conds in=74b,call rgw_gc.rgw_gc_queue_init in=45b,call version.set in=34b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e32) v8 -- 0x565382d067d0 con 0x565382d12760 2023-10-05T19:28:56.857 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.850+0000 7f5473e27700 1 -- 172.21.15.167:0/4231517449 <== osd.3 v2:172.21.15.167:6808/30036 3 ==== osd_op_reply(10 gc.9 [create,call,call,call] v32'45 uv39 ondisk = -125 ((125) Operation canceled)) v8 ==== 274+0+0 (crc 0 0 0) 0x7f5464050040 con 0x565382d12760 2023-10-05T19:28:56.857 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.850+0000 7f5475af1e40 1 -- 172.21.15.167:0/4231517449 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:11 6.3 6:c31f41d7:gc::gc.10:head [create,call version.check_conds in=74b,call rgw_gc.rgw_gc_queue_init in=45b,call version.set in=34b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e32) v8 -- 0x565382d1ab30 con 0x565382d16450 2023-10-05T19:28:56.859 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.854+0000 7f5473626700 1 -- 172.21.15.167:0/4231517449 <== osd.2 v2:172.21.15.167:6816/30037 3 ==== osd_op_reply(11 gc.10 [create,call,call,call] v32'63 uv53 ondisk = -125 ((125) Operation canceled)) v8 ==== 275+0+0 (crc 0 0 0) 0x7f546c034070 con 0x565382d16450 2023-10-05T19:28:56.860 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.854+0000 7f5475af1e40 1 -- 172.21.15.167:0/4231517449 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:12 6.13 6:c8333aab:gc::gc.11:head [create,call version.check_conds in=74b,call rgw_gc.rgw_gc_queue_init in=45b,call version.set in=34b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e32) v8 -- 0x565382d1ab30 con 0x565382d0ea90 2023-10-05T19:28:56.862 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.858+0000 7f5472e25700 1 -- 172.21.15.167:0/4231517449 <== osd.0 v2:172.21.15.118:6800/27690 3 ==== osd_op_reply(12 gc.11 [create,call,call,call] v32'28 uv25 ondisk = -125 ((125) Operation canceled)) v8 ==== 275+0+0 (crc 0 0 0) 0x7f545c006070 con 0x565382d0ea90 2023-10-05T19:28:56.862 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.858+0000 7f5475af1e40 1 -- 172.21.15.167:0/4231517449 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:13 6.15 6:ab569f81:gc::gc.12:head [create,call version.check_conds in=74b,call rgw_gc.rgw_gc_queue_init in=45b,call version.set in=34b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e32) v8 -- 0x565382d1b020 con 0x565382d16450 2023-10-05T19:28:56.865 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.858+0000 7f5473626700 1 -- 172.21.15.167:0/4231517449 <== osd.2 v2:172.21.15.167:6816/30037 4 ==== osd_op_reply(13 gc.12 [create,call,call,call] v32'24 uv21 ondisk = -125 ((125) Operation canceled)) v8 ==== 275+0+0 (crc 0 0 0) 0x7f546c034070 con 0x565382d16450 2023-10-05T19:28:56.865 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.858+0000 7f5475af1e40 1 -- 172.21.15.167:0/4231517449 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:14 6.0 6:0667bb13:gc::gc.13:head [create,call version.check_conds in=74b,call rgw_gc.rgw_gc_queue_init in=45b,call version.set in=34b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e32) v8 -- 0x565382d1b020 con 0x565382d0ea90 2023-10-05T19:28:56.867 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.862+0000 7f5472e25700 1 -- 172.21.15.167:0/4231517449 <== osd.0 v2:172.21.15.118:6800/27690 4 ==== osd_op_reply(14 gc.13 [create,call,call,call] v32'47 uv40 ondisk = -125 ((125) Operation canceled)) v8 ==== 275+0+0 (crc 0 0 0) 0x7f545c006070 con 0x565382d0ea90 2023-10-05T19:28:56.868 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.862+0000 7f5475af1e40 1 -- 172.21.15.167:0/4231517449 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:15 6.4 6:227f63b5:gc::gc.14:head [create,call version.check_conds in=74b,call rgw_gc.rgw_gc_queue_init in=45b,call version.set in=34b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e32) v8 -- 0x565382d1b290 con 0x565382d0afe0 2023-10-05T19:28:56.871 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.866+0000 7f5473626700 1 -- 172.21.15.167:0/4231517449 <== osd.1 v2:172.21.15.167:6800/30035 4 ==== osd_op_reply(15 gc.14 [create,call,call,call] v32'207 uv186 ondisk = -125 ((125) Operation canceled)) v8 ==== 275+0+0 (crc 0 0 0) 0x7f546c034070 con 0x565382d0afe0 2023-10-05T19:28:56.871 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.866+0000 7f5475af1e40 1 -- 172.21.15.167:0/4231517449 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:16 6.7 6:e07a6548:gc::gc.15:head [create,call version.check_conds in=74b,call rgw_gc.rgw_gc_queue_init in=45b,call version.set in=34b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e32) v8 -- 0x565382d1b290 con 0x565382d12760 2023-10-05T19:28:56.873 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.870+0000 7f5473e27700 1 -- 172.21.15.167:0/4231517449 <== osd.3 v2:172.21.15.167:6808/30036 4 ==== osd_op_reply(16 gc.15 [create,call,call,call] v32'56 uv45 ondisk = -125 ((125) Operation canceled)) v8 ==== 275+0+0 (crc 0 0 0) 0x7f5464050040 con 0x565382d12760 2023-10-05T19:28:56.874 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.870+0000 7f5475af1e40 1 -- 172.21.15.167:0/4231517449 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:17 6.3 6:c623b173:gc::gc.16:head [create,call version.check_conds in=74b,call rgw_gc.rgw_gc_queue_init in=45b,call version.set in=34b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e32) v8 -- 0x565382d1b320 con 0x565382d16450 2023-10-05T19:28:56.876 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.870+0000 7f5473626700 1 -- 172.21.15.167:0/4231517449 <== osd.2 v2:172.21.15.167:6816/30037 5 ==== osd_op_reply(17 gc.16 [create,call,call,call] v32'64 uv55 ondisk = -125 ((125) Operation canceled)) v8 ==== 275+0+0 (crc 0 0 0) 0x7f546c034070 con 0x565382d16450 2023-10-05T19:28:56.876 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.870+0000 7f5475af1e40 1 -- 172.21.15.167:0/4231517449 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:18 6.16 6:6849fd06:gc::gc.17:head [create,call version.check_conds in=74b,call rgw_gc.rgw_gc_queue_init in=45b,call version.set in=34b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e32) v8 -- 0x565382d1b320 con 0x565382d12760 2023-10-05T19:28:56.879 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.874+0000 7f5473e27700 1 -- 172.21.15.167:0/4231517449 <== osd.3 v2:172.21.15.167:6808/30036 5 ==== osd_op_reply(18 gc.17 [create,call,call,call] v32'24 uv21 ondisk = -125 ((125) Operation canceled)) v8 ==== 275+0+0 (crc 0 0 0) 0x7f5464050040 con 0x565382d12760 2023-10-05T19:28:56.879 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.874+0000 7f5475af1e40 1 -- 172.21.15.167:0/4231517449 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:19 6.d 6:b019ea83:gc::gc.18:head [create,call version.check_conds in=74b,call rgw_gc.rgw_gc_queue_init in=45b,call version.set in=34b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e32) v8 -- 0x565382d1b350 con 0x565382d0afe0 2023-10-05T19:28:56.882 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.878+0000 7f5473626700 1 -- 172.21.15.167:0/4231517449 <== osd.1 v2:172.21.15.167:6800/30035 5 ==== osd_op_reply(19 gc.18 [create,call,call,call] v32'32 uv29 ondisk = -125 ((125) Operation canceled)) v8 ==== 275+0+0 (crc 0 0 0) 0x7f546c034070 con 0x565382d0afe0 2023-10-05T19:28:56.882 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.878+0000 7f5475af1e40 1 -- 172.21.15.167:0/4231517449 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:20 6.4 6:24c8e1ae:gc::gc.19:head [create,call version.check_conds in=74b,call rgw_gc.rgw_gc_queue_init in=45b,call version.set in=34b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e32) v8 -- 0x565382d1b3e0 con 0x565382d0afe0 2023-10-05T19:28:56.885 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.878+0000 7f5473626700 1 -- 172.21.15.167:0/4231517449 <== osd.1 v2:172.21.15.167:6800/30035 6 ==== osd_op_reply(20 gc.19 [create,call,call,call] v32'208 uv189 ondisk = -125 ((125) Operation canceled)) v8 ==== 275+0+0 (crc 0 0 0) 0x7f546c034070 con 0x565382d0afe0 2023-10-05T19:28:56.885 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.878+0000 7f5475af1e40 1 -- 172.21.15.167:0/4231517449 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:21 6.7 6:e622f76a:gc::gc.20:head [create,call version.check_conds in=74b,call rgw_gc.rgw_gc_queue_init in=45b,call version.set in=34b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e32) v8 -- 0x565382d1b3e0 con 0x565382d12760 2023-10-05T19:28:56.887 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.882+0000 7f5473e27700 1 -- 172.21.15.167:0/4231517449 <== osd.3 v2:172.21.15.167:6808/30036 6 ==== osd_op_reply(21 gc.20 [create,call,call,call] v32'57 uv47 ondisk = -125 ((125) Operation canceled)) v8 ==== 275+0+0 (crc 0 0 0) 0x7f5464050040 con 0x565382d12760 2023-10-05T19:28:56.888 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.882+0000 7f5475af1e40 1 -- 172.21.15.167:0/4231517449 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:22 6.a 6:5127e642:gc::gc.21:head [create,call version.check_conds in=74b,call rgw_gc.rgw_gc_queue_init in=45b,call version.set in=34b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e32) v8 -- 0x565382d1b430 con 0x565382d16450 2023-10-05T19:28:56.890 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.886+0000 7f5473626700 1 -- 172.21.15.167:0/4231517449 <== osd.2 v2:172.21.15.167:6816/30037 6 ==== osd_op_reply(22 gc.21 [create,call,call,call] v32'34 uv31 ondisk = -125 ((125) Operation canceled)) v8 ==== 275+0+0 (crc 0 0 0) 0x7f546c034070 con 0x565382d16450 2023-10-05T19:28:56.890 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.886+0000 7f5475af1e40 1 -- 172.21.15.167:0/4231517449 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:23 6.11 6:8eb99c53:gc::gc.22:head [create,call version.check_conds in=74b,call rgw_gc.rgw_gc_queue_init in=45b,call version.set in=34b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e32) v8 -- 0x565382d1b430 con 0x565382d12760 2023-10-05T19:28:56.893 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.886+0000 7f5473e27700 1 -- 172.21.15.167:0/4231517449 <== osd.3 v2:172.21.15.167:6808/30036 7 ==== osd_op_reply(23 gc.22 [create,call,call,call] v32'26 uv23 ondisk = -125 ((125) Operation canceled)) v8 ==== 275+0+0 (crc 0 0 0) 0x7f5464050040 con 0x565382d12760 2023-10-05T19:28:56.893 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.886+0000 7f5475af1e40 1 -- 172.21.15.167:0/4231517449 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:24 6.8 6:17d7e8ed:gc::gc.23:head [create,call version.check_conds in=74b,call rgw_gc.rgw_gc_queue_init in=45b,call version.set in=34b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e32) v8 -- 0x565382d1b4c0 con 0x565382d12760 2023-10-05T19:28:56.896 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.890+0000 7f5473e27700 1 -- 172.21.15.167:0/4231517449 <== osd.3 v2:172.21.15.167:6808/30036 8 ==== osd_op_reply(24 gc.23 [create,call,call,call] v32'37 uv33 ondisk = -125 ((125) Operation canceled)) v8 ==== 275+0+0 (crc 0 0 0) 0x7f5464050040 con 0x565382d12760 2023-10-05T19:28:56.896 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.890+0000 7f5475af1e40 1 -- 172.21.15.167:0/4231517449 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:25 6.1b 6:db9d130d:gc::gc.24:head [create,call version.check_conds in=74b,call rgw_gc.rgw_gc_queue_init in=45b,call version.set in=34b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e32) v8 -- 0x565382d1b4c0 con 0x565382d12760 2023-10-05T19:28:56.898 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.894+0000 7f5473e27700 1 -- 172.21.15.167:0/4231517449 <== osd.3 v2:172.21.15.167:6808/30036 9 ==== osd_op_reply(25 gc.24 [create,call,call,call] v32'20 uv17 ondisk = -125 ((125) Operation canceled)) v8 ==== 275+0+0 (crc 0 0 0) 0x7f5464050040 con 0x565382d12760 2023-10-05T19:28:56.899 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.894+0000 7f5475af1e40 1 -- 172.21.15.167:0/4231517449 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:26 6.9 6:921ecdb1:gc::gc.25:head [create,call version.check_conds in=74b,call rgw_gc.rgw_gc_queue_init in=45b,call version.set in=34b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e32) v8 -- 0x565382d1b4c0 con 0x565382d12760 2023-10-05T19:28:56.901 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.894+0000 7f5473e27700 1 -- 172.21.15.167:0/4231517449 <== osd.3 v2:172.21.15.167:6808/30036 10 ==== osd_op_reply(26 gc.25 [create,call,call,call] v32'45 uv39 ondisk = -125 ((125) Operation canceled)) v8 ==== 275+0+0 (crc 0 0 0) 0x7f5464050040 con 0x565382d12760 2023-10-05T19:28:56.901 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.894+0000 7f5475af1e40 1 -- 172.21.15.167:0/4231517449 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:27 6.1f 6:fde4dd55:gc::gc.26:head [create,call version.check_conds in=74b,call rgw_gc.rgw_gc_queue_init in=45b,call version.set in=34b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e32) v8 -- 0x565382d1b4c0 con 0x565382d0ea90 2023-10-05T19:28:56.904 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.898+0000 7f5472e25700 1 -- 172.21.15.167:0/4231517449 <== osd.0 v2:172.21.15.118:6800/27690 5 ==== osd_op_reply(27 gc.26 [create,call,call,call] v32'37 uv23 ondisk = -125 ((125) Operation canceled)) v8 ==== 275+0+0 (crc 0 0 0) 0x7f545c006070 con 0x565382d0ea90 2023-10-05T19:28:56.904 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.898+0000 7f5475af1e40 1 -- 172.21.15.167:0/4231517449 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:28 6.1a 6:5909e671:gc::gc.27:head [create,call version.check_conds in=74b,call rgw_gc.rgw_gc_queue_init in=45b,call version.set in=34b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e32) v8 -- 0x565382d1b4c0 con 0x565382d12760 2023-10-05T19:28:56.906 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.902+0000 7f5473e27700 1 -- 172.21.15.167:0/4231517449 <== osd.3 v2:172.21.15.167:6808/30036 11 ==== osd_op_reply(28 gc.27 [create,call,call,call] v32'24 uv21 ondisk = -125 ((125) Operation canceled)) v8 ==== 275+0+0 (crc 0 0 0) 0x7f5464050040 con 0x565382d12760 2023-10-05T19:28:56.906 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.902+0000 7f5475af1e40 1 -- 172.21.15.167:0/4231517449 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:29 6.1d 6:b88e3271:gc::gc.28:head [create,call version.check_conds in=74b,call rgw_gc.rgw_gc_queue_init in=45b,call version.set in=34b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e32) v8 -- 0x565382d1b4c0 con 0x565382d0ea90 2023-10-05T19:28:56.908 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.902+0000 7f5472e25700 1 -- 172.21.15.167:0/4231517449 <== osd.0 v2:172.21.15.118:6800/27690 6 ==== osd_op_reply(29 gc.28 [create,call,call,call] v32'42 uv36 ondisk = -125 ((125) Operation canceled)) v8 ==== 275+0+0 (crc 0 0 0) 0x7f545c006070 con 0x565382d0ea90 2023-10-05T19:28:56.909 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.902+0000 7f5475af1e40 1 -- 172.21.15.167:0/4231517449 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:30 6.7 6:e47c9322:gc::gc.29:head [create,call version.check_conds in=74b,call rgw_gc.rgw_gc_queue_init in=45b,call version.set in=34b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e32) v8 -- 0x565382d1b4c0 con 0x565382d12760 2023-10-05T19:28:56.911 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.906+0000 7f5473e27700 1 -- 172.21.15.167:0/4231517449 <== osd.3 v2:172.21.15.167:6808/30036 12 ==== osd_op_reply(30 gc.29 [create,call,call,call] v32'58 uv49 ondisk = -125 ((125) Operation canceled)) v8 ==== 275+0+0 (crc 0 0 0) 0x7f5464050040 con 0x565382d12760 2023-10-05T19:28:56.911 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.906+0000 7f5475af1e40 1 -- 172.21.15.167:0/4231517449 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:31 6.0 6:013d6c33:gc::gc.30:head [create,call version.check_conds in=74b,call rgw_gc.rgw_gc_queue_init in=45b,call version.set in=34b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e32) v8 -- 0x565382d1b4c0 con 0x565382d0ea90 2023-10-05T19:28:56.914 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.910+0000 7f5472e25700 1 -- 172.21.15.167:0/4231517449 <== osd.0 v2:172.21.15.118:6800/27690 7 ==== osd_op_reply(31 gc.30 [create,call,call,call] v32'48 uv42 ondisk = -125 ((125) Operation canceled)) v8 ==== 275+0+0 (crc 0 0 0) 0x7f545c006070 con 0x565382d0ea90 2023-10-05T19:28:56.914 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.910+0000 7f5475af1e40 1 -- 172.21.15.167:0/4231517449 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:32 6.f 6:f24cb4a8:gc::gc.31:head [create,call version.check_conds in=74b,call rgw_gc.rgw_gc_queue_init in=45b,call version.set in=34b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e32) v8 -- 0x565382d1b4c0 con 0x565382d16450 2023-10-05T19:28:56.917 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.910+0000 7f5473626700 1 -- 172.21.15.167:0/4231517449 <== osd.2 v2:172.21.15.167:6816/30037 7 ==== osd_op_reply(32 gc.31 [create,call,call,call] v32'30 uv27 ondisk = -125 ((125) Operation canceled)) v8 ==== 275+0+0 (crc 0 0 0) 0x7f546c034070 con 0x565382d16450 2023-10-05T19:28:56.920 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.914+0000 7f53b37fe700 20 reqs_thread_entry: start 2023-10-05T19:28:56.920 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.914+0000 7f5475af1e40 20 init_complete bucket index max shards: 11 2023-10-05T19:28:56.921 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.914+0000 7f53b17fa700 20 reqs_thread_entry: start 2023-10-05T19:28:56.921 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.914+0000 7f5475af1e40 10 rgw notify: Started notification manager with: 1 workers 2023-10-05T19:28:56.921 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.914+0000 7f539f7fe700 20 rgw notify: INFO: next queues processing will happen at: Thu Oct 5 19:29:27 2023 2023-10-05T19:28:56.921 INFO:teuthology.orchestra.run.smithi167.stderr: 2023-10-05T19:28:56.922 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.914+0000 7f5475af1e40 20 get_system_obj_state: rctx=0x7fff8b74c570 obj=default.rgw.meta:root:myfoo state=0x565382b850b0 s->prefetch_data=0 2023-10-05T19:28:56.922 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.914+0000 7f5475af1e40 10 cache get: name=default.rgw.meta+root+myfoo : miss 2023-10-05T19:28:56.922 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.914+0000 7f539effd700 20 reqs_thread_entry: start 2023-10-05T19:28:56.922 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.914+0000 7f5475af1e40 1 -- 172.21.15.167:0/1974318067 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:102 8.9 8:91b71841:root::myfoo:head [call version.read in=11b,getxattrs,stat] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32) v8 -- 0x565382d7a7f0 con 0x7f546c01b740 2023-10-05T19:28:56.923 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.918+0000 7f5472e25700 1 -- 172.21.15.167:0/1974318067 <== osd.2 v2:172.21.15.167:6816/30037 17 ==== osd_op_reply(102 myfoo [call out=48b,getxattrs out=75b,stat out=16b] v0'0 uv34 ondisk = 0) v8 ==== 233+0+139 (crc 0 0 0) 0x7f545c006070 con 0x7f546c01b740 2023-10-05T19:28:56.923 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.918+0000 7f5475af1e40 10 cache put: name=default.rgw.meta+root+myfoo info.flags=0x16 2023-10-05T19:28:56.923 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.918+0000 7f5475af1e40 10 adding default.rgw.meta+root+myfoo to cache LRU end 2023-10-05T19:28:56.923 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.918+0000 7f5475af1e40 10 updating xattr: name=ceph.objclass.version bl.length()=42 2023-10-05T19:28:56.923 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.918+0000 7f5475af1e40 20 get_system_obj_state: s->obj_tag was set empty 2023-10-05T19:28:56.923 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.918+0000 7f5475af1e40 10 cache get: name=default.rgw.meta+root+myfoo : type miss (requested=0x11, cached=0x16) 2023-10-05T19:28:56.923 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.918+0000 7f5475af1e40 20 rados->read ofs=0 len=0 2023-10-05T19:28:56.923 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.918+0000 7f5475af1e40 1 -- 172.21.15.167:0/1974318067 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:103 8.9 8:91b71841:root::myfoo:head [call version.check_conds in=122b,call version.read in=11b,read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32) v8 -- 0x565382d7a720 con 0x7f546c01b740 2023-10-05T19:28:56.927 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.918+0000 7f5472e25700 1 -- 172.21.15.167:0/1974318067 <== osd.2 v2:172.21.15.167:6816/30037 18 ==== osd_op_reply(103 myfoo [call,call out=48b,read 0~165 out=165b] v0'0 uv34 ondisk = 0) v8 ==== 233+0+213 (crc 0 0 0) 0x7f545c006070 con 0x7f546c01b740 2023-10-05T19:28:56.927 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.918+0000 7f5475af1e40 20 rados_obj.operate() r=0 bl.length=165 2023-10-05T19:28:56.927 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.918+0000 7f5475af1e40 10 cache put: name=default.rgw.meta+root+myfoo info.flags=0x11 2023-10-05T19:28:56.927 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.918+0000 7f5475af1e40 10 moving default.rgw.meta+root+myfoo to cache LRU end 2023-10-05T19:28:56.927 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.918+0000 7f5475af1e40 20 get_system_obj_state: rctx=0x7fff8b74c570 obj=default.rgw.meta:root:.bucket.meta.myfoo:9a9df965-6612-45d4-a45f-6e3c889bacc2.4223.5 state=0x565382b5b2b0 s->prefetch_data=0 2023-10-05T19:28:56.927 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.918+0000 7f5475af1e40 10 cache get: name=default.rgw.meta+root+.bucket.meta.myfoo:9a9df965-6612-45d4-a45f-6e3c889bacc2.4223.5 : miss 2023-10-05T19:28:56.927 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.918+0000 7f5475af1e40 1 -- 172.21.15.167:0/1974318067 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:104 8.9 8:96cb4b0f:root::.bucket.meta.myfoo%3a9a9df965-6612-45d4-a45f-6e3c889bacc2.4223.5:head [call version.read in=11b,getxattrs,stat] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32) v8 -- 0x565382d7c270 con 0x7f546c01b740 2023-10-05T19:28:56.927 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.918+0000 7f5472e25700 1 -- 172.21.15.167:0/1974318067 <== osd.2 v2:172.21.15.167:6816/30037 19 ==== osd_op_reply(104 .bucket.meta.myfoo:9a9df965-6612-45d4-a45f-6e3c889bacc2.4223.5 [call out=48b,getxattrs out=218b,stat out=16b] v0'0 uv33 ondisk = 0) v8 ==== 290+0+282 (crc 0 0 0) 0x7f545c006070 con 0x7f546c01b740 2023-10-05T19:28:56.927 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.918+0000 7f5475af1e40 10 cache put: name=default.rgw.meta+root+.bucket.meta.myfoo:9a9df965-6612-45d4-a45f-6e3c889bacc2.4223.5 info.flags=0x16 2023-10-05T19:28:56.928 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.918+0000 7f5475af1e40 10 adding default.rgw.meta+root+.bucket.meta.myfoo:9a9df965-6612-45d4-a45f-6e3c889bacc2.4223.5 to cache LRU end 2023-10-05T19:28:56.928 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.918+0000 7f5475af1e40 10 updating xattr: name=ceph.objclass.version bl.length()=42 2023-10-05T19:28:56.928 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.918+0000 7f5475af1e40 10 updating xattr: name=user.rgw.acl bl.length()=123 2023-10-05T19:28:56.928 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.918+0000 7f5475af1e40 20 get_system_obj_state: s->obj_tag was set empty 2023-10-05T19:28:56.928 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.918+0000 7f5475af1e40 20 Read xattr: user.rgw.acl 2023-10-05T19:28:56.928 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.918+0000 7f5475af1e40 20 Read xattr: user.rgw.idtag 2023-10-05T19:28:56.928 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.918+0000 7f5475af1e40 10 cache get: name=default.rgw.meta+root+.bucket.meta.myfoo:9a9df965-6612-45d4-a45f-6e3c889bacc2.4223.5 : type miss (requested=0x13, cached=0x16) 2023-10-05T19:28:56.928 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.918+0000 7f5475af1e40 20 rados->read ofs=0 len=0 2023-10-05T19:28:56.928 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.918+0000 7f5475af1e40 1 -- 172.21.15.167:0/1974318067 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:105 8.9 8:96cb4b0f:root::.bucket.meta.myfoo%3a9a9df965-6612-45d4-a45f-6e3c889bacc2.4223.5:head [call version.check_conds in=122b,call version.read in=11b,read 0~0,getxattrs] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32) v8 -- 0x565382d7ce00 con 0x7f546c01b740 2023-10-05T19:28:56.928 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.918+0000 7f5472e25700 1 -- 172.21.15.167:0/1974318067 <== osd.2 v2:172.21.15.167:6816/30037 20 ==== osd_op_reply(105 .bucket.meta.myfoo:9a9df965-6612-45d4-a45f-6e3c889bacc2.4223.5 [call,call out=48b,read 0~352 out=352b,getxattrs out=218b] v0'0 uv33 ondisk = 0) v8 ==== 332+0+618 (crc 0 0 0) 0x7f545c006070 con 0x7f546c01b740 2023-10-05T19:28:56.928 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.918+0000 7f5475af1e40 20 rados_obj.operate() r=0 bl.length=352 2023-10-05T19:28:56.928 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.918+0000 7f5475af1e40 10 cache put: name=default.rgw.meta+root+.bucket.meta.myfoo:9a9df965-6612-45d4-a45f-6e3c889bacc2.4223.5 info.flags=0x13 2023-10-05T19:28:56.929 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.918+0000 7f5475af1e40 10 moving default.rgw.meta+root+.bucket.meta.myfoo:9a9df965-6612-45d4-a45f-6e3c889bacc2.4223.5 to cache LRU end 2023-10-05T19:28:56.929 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.918+0000 7f5475af1e40 10 updating xattr: name=ceph.objclass.version bl.length()=42 2023-10-05T19:28:56.929 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.918+0000 7f5475af1e40 10 updating xattr: name=user.rgw.acl bl.length()=123 2023-10-05T19:28:56.929 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.918+0000 7f5475af1e40 10 chain_cache_entry: cache_locator=default.rgw.meta+root+.bucket.meta.myfoo:9a9df965-6612-45d4-a45f-6e3c889bacc2.4223.5 2023-10-05T19:28:56.929 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.918+0000 7f5475af1e40 10 int RGWRados::cls_bucket_list_unordered(const DoutPrefixProvider*, RGWBucketInfo&, int, const rgw_obj_index_key&, const string&, uint32_t, bool, std::vector&, bool*, rgw_obj_index_key*, optional_yield, RGWBucketListNameFilter) :myfoo[9a9df965-6612-45d4-a45f-6e3c889bacc2.4223.5]) start_after="", prefix="", shard_id=-1", num_entries=1100, list_versions=1, force_check_filter is unset 2023-10-05T19:28:56.929 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.918+0000 7f5475af1e40 1 -- 172.21.15.167:0/1974318067 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:106 4.2d 4:b5ae0298:::.dir.9a9df965-6612-45d4-a45f-6e3c889bacc2.4223.5.0:head [call rgw.bucket_list in=47b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32) v8 -- 0x565382d7cec0 con 0x7f545c00d8e0 2023-10-05T19:28:56.929 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.922+0000 7f5473e27700 1 -- 172.21.15.167:0/1974318067 <== osd.0 v2:172.21.15.118:6800/27690 29 ==== osd_op_reply(106 .dir.9a9df965-6612-45d4-a45f-6e3c889bacc2.4223.5.0 [call out=85b] v0'0 uv5 ondisk = 0) v8 ==== 194+0+85 (crc 0 0 0) 0x7f5464050040 con 0x7f545c00d8e0 2023-10-05T19:28:56.929 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.922+0000 7f5475af1e40 1 -- 172.21.15.167:0/1974318067 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:107 4.34 4:2ef4a173:::.dir.9a9df965-6612-45d4-a45f-6e3c889bacc2.4223.5.1:head [call rgw.bucket_list in=47b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32) v8 -- 0x565382dfa1f0 con 0x7f546c01b740 2023-10-05T19:28:56.930 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.922+0000 7f5472e25700 1 -- 172.21.15.167:0/1974318067 <== osd.2 v2:172.21.15.167:6816/30037 21 ==== osd_op_reply(107 .dir.9a9df965-6612-45d4-a45f-6e3c889bacc2.4223.5.1 [call out=85b] v0'0 uv1 ondisk = 0) v8 ==== 194+0+85 (crc 0 0 0) 0x7f545c006070 con 0x7f546c01b740 2023-10-05T19:28:56.930 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.922+0000 7f5475af1e40 1 -- 172.21.15.167:0/1974318067 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:108 4.2 4:42981e29:::.dir.9a9df965-6612-45d4-a45f-6e3c889bacc2.4223.5.2:head [call rgw.bucket_list in=47b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32) v8 -- 0x565382dfa1f0 con 0x565382cac750 2023-10-05T19:28:56.930 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.922+0000 7f5473e27700 1 -- 172.21.15.167:0/1974318067 <== osd.1 v2:172.21.15.167:6800/30035 35 ==== osd_op_reply(108 .dir.9a9df965-6612-45d4-a45f-6e3c889bacc2.4223.5.2 [call out=425b] v0'0 uv7 ondisk = 0) v8 ==== 194+0+425 (crc 0 0 0) 0x7f5464050040 con 0x565382cac750 2023-10-05T19:28:56.930 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.922+0000 7f5475af1e40 10 cls_bucket_list_unordered: got 9c8454ddf7aa50116496bac348d7550d 2023-10-05T19:28:56.930 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.922+0000 7f5475af1e40 1 -- 172.21.15.167:0/1974318067 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:109 4.3d 4:bdb71a6b:::.dir.9a9df965-6612-45d4-a45f-6e3c889bacc2.4223.5.3:head [call rgw.bucket_list in=47b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32) v8 -- 0x565382dfa1f0 con 0x565382ce1410 2023-10-05T19:28:56.931 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.926+0000 7f5473626700 1 -- 172.21.15.167:0/1974318067 <== osd.3 v2:172.21.15.167:6808/30036 24 ==== osd_op_reply(109 .dir.9a9df965-6612-45d4-a45f-6e3c889bacc2.4223.5.3 [call out=85b] v0'0 uv5 ondisk = 0) v8 ==== 194+0+85 (crc 0 0 0) 0x7f546c034070 con 0x565382ce1410 2023-10-05T19:28:56.931 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.926+0000 7f5475af1e40 1 -- 172.21.15.167:0/1974318067 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:110 4.33 4:cc3cd978:::.dir.9a9df965-6612-45d4-a45f-6e3c889bacc2.4223.5.4:head [call rgw.bucket_list in=47b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32) v8 -- 0x565382dfa1f0 con 0x7f545c00d8e0 2023-10-05T19:28:56.931 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.926+0000 7f5473e27700 1 -- 172.21.15.167:0/1974318067 <== osd.0 v2:172.21.15.118:6800/27690 30 ==== osd_op_reply(110 .dir.9a9df965-6612-45d4-a45f-6e3c889bacc2.4223.5.4 [call out=85b] v0'0 uv3 ondisk = 0) v8 ==== 194+0+85 (crc 0 0 0) 0x7f5464050040 con 0x7f545c00d8e0 2023-10-05T19:28:56.931 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.926+0000 7f5475af1e40 1 -- 172.21.15.167:0/1974318067 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:111 4.16 4:6b936802:::.dir.9a9df965-6612-45d4-a45f-6e3c889bacc2.4223.5.5:head [call rgw.bucket_list in=47b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32) v8 -- 0x565382dfa6c0 con 0x565382ce1410 2023-10-05T19:28:56.932 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.926+0000 7f5473626700 1 -- 172.21.15.167:0/1974318067 <== osd.3 v2:172.21.15.167:6808/30036 25 ==== osd_op_reply(111 .dir.9a9df965-6612-45d4-a45f-6e3c889bacc2.4223.5.5 [call out=85b] v0'0 uv3 ondisk = 0) v8 ==== 194+0+85 (crc 0 0 0) 0x7f546c034070 con 0x565382ce1410 2023-10-05T19:28:56.933 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.926+0000 7f5475af1e40 1 -- 172.21.15.167:0/1974318067 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:112 4.8 4:1342e9a7:::.dir.9a9df965-6612-45d4-a45f-6e3c889bacc2.4223.5.6:head [call rgw.bucket_list in=47b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32) v8 -- 0x565382dfa6c0 con 0x565382ce1410 2023-10-05T19:28:56.934 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.930+0000 7f5473626700 1 -- 172.21.15.167:0/1974318067 <== osd.3 v2:172.21.15.167:6808/30036 26 ==== osd_op_reply(112 .dir.9a9df965-6612-45d4-a45f-6e3c889bacc2.4223.5.6 [call out=426b] v0'0 uv3 ondisk = 0) v8 ==== 194+0+426 (crc 0 0 0) 0x7f546c034070 con 0x565382ce1410 2023-10-05T19:28:56.934 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.930+0000 7f5475af1e40 10 cls_bucket_list_unordered: got b1b9a972ccd8c962a473909b97007eb4 2023-10-05T19:28:56.934 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.930+0000 7f5475af1e40 1 -- 172.21.15.167:0/1974318067 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:113 4.d 4:b00cdfe0:::.dir.9a9df965-6612-45d4-a45f-6e3c889bacc2.4223.5.7:head [call rgw.bucket_list in=47b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32) v8 -- 0x565382dfa6c0 con 0x565382ce1410 2023-10-05T19:28:56.934 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.930+0000 7f5473626700 1 -- 172.21.15.167:0/1974318067 <== osd.3 v2:172.21.15.167:6808/30036 27 ==== osd_op_reply(113 .dir.9a9df965-6612-45d4-a45f-6e3c889bacc2.4223.5.7 [call out=85b] v0'0 uv1 ondisk = 0) v8 ==== 194+0+85 (crc 0 0 0) 0x7f546c034070 con 0x565382ce1410 2023-10-05T19:28:56.934 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.930+0000 7f5475af1e40 1 -- 172.21.15.167:0/1974318067 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:114 4.21 4:84ddff17:::.dir.9a9df965-6612-45d4-a45f-6e3c889bacc2.4223.5.8:head [call rgw.bucket_list in=47b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32) v8 -- 0x565382dfa6c0 con 0x7f545c00d8e0 2023-10-05T19:28:56.936 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.930+0000 7f5473e27700 1 -- 172.21.15.167:0/1974318067 <== osd.0 v2:172.21.15.118:6800/27690 31 ==== osd_op_reply(114 .dir.9a9df965-6612-45d4-a45f-6e3c889bacc2.4223.5.8 [call out=426b] v0'0 uv5 ondisk = 0) v8 ==== 194+0+426 (crc 0 0 0) 0x7f5464050040 con 0x7f545c00d8e0 2023-10-05T19:28:56.936 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.930+0000 7f5475af1e40 10 cls_bucket_list_unordered: got c785e1ed2950e3e36b1e2ca01f299a54 2023-10-05T19:28:56.936 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.930+0000 7f5475af1e40 1 -- 172.21.15.167:0/1974318067 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:115 4.4 4:20b95fb7:::.dir.9a9df965-6612-45d4-a45f-6e3c889bacc2.4223.5.9:head [call rgw.bucket_list in=47b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32) v8 -- 0x565382dfa6c0 con 0x7f545c00d8e0 2023-10-05T19:28:56.937 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.930+0000 7f5473e27700 1 -- 172.21.15.167:0/1974318067 <== osd.0 v2:172.21.15.118:6800/27690 32 ==== osd_op_reply(115 .dir.9a9df965-6612-45d4-a45f-6e3c889bacc2.4223.5.9 [call out=85b] v0'0 uv3 ondisk = 0) v8 ==== 194+0+85 (crc 0 0 0) 0x7f5464050040 con 0x7f545c00d8e0 2023-10-05T19:28:56.937 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.930+0000 7f5475af1e40 1 -- 172.21.15.167:0/1974318067 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:116 4.25 4:a63b86a5:::.dir.9a9df965-6612-45d4-a45f-6e3c889bacc2.4223.5.10:head [call rgw.bucket_list in=47b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32) v8 -- 0x565382dfa6c0 con 0x565382ce1410 2023-10-05T19:28:56.938 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.934+0000 7f5473626700 1 -- 172.21.15.167:0/1974318067 <== osd.3 v2:172.21.15.167:6808/30036 28 ==== osd_op_reply(116 .dir.9a9df965-6612-45d4-a45f-6e3c889bacc2.4223.5.10 [call out=698b] v0'0 uv9 ondisk = 0) v8 ==== 195+0+698 (crc 0 0 0) 0x7f546c034070 con 0x565382ce1410 2023-10-05T19:28:56.938 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.934+0000 7f5475af1e40 10 cls_bucket_list_unordered: got 24d27c169c2c881eb09a065116f2aa5c 2023-10-05T19:28:56.938 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.934+0000 7f5475af1e40 10 cls_bucket_list_unordered: got bb3aec0fdcdbc2974890f805c585d432 2023-10-05T19:28:56.939 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.934+0000 7f5475af1e40 20 get_obj_state: rctx=0x7fff8b74c1a0 obj=myfoo:9c8454ddf7aa50116496bac348d7550d state=0x565382d7f6e8 s->prefetch_data=0 2023-10-05T19:28:56.939 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.934+0000 7f5475af1e40 1 -- 172.21.15.167:0/1974318067 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:117 3.36s0 3:6fce0d3c:::9a9df965-6612-45d4-a45f-6e3c889bacc2.4223.5_9c8454ddf7aa50116496bac348d7550d:head [getxattrs,stat] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32) v8 -- 0x565382d7e9d0 con 0x7f546c01b740 2023-10-05T19:28:56.939 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.934+0000 7f5472e25700 1 -- 172.21.15.167:0/1974318067 <== osd.2 v2:172.21.15.167:6816/30037 22 ==== osd_op_reply(117 9a9df965-6612-45d4-a45f-6e3c889bacc2.4223.5_9c8454ddf7aa50116496bac348d7550d [getxattrs out=877b,stat out=16b] v0'0 uv1 ondisk = 0) v8 ==== 262+0+893 (crc 0 0 0) 0x7f545c006070 con 0x7f546c01b740 2023-10-05T19:28:56.939 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.934+0000 7f5475af1e40 10 manifest: total_size = 6 2023-10-05T19:28:56.940 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.934+0000 7f5475af1e40 20 get_obj_state: setting s->obj_tag to 9a9df965-6612-45d4-a45f-6e3c889bacc2.4221.1711147289124211764 2023-10-05T19:28:56.940 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.934+0000 7f5475af1e40 20 get_obj_state: rctx=0x7fff8b74c1a0 obj=myfoo:9c8454ddf7aa50116496bac348d7550d state=0x565382d7f6e8 s->prefetch_data=0 2023-10-05T19:28:56.940 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.934+0000 7f5475af1e40 20 prepare_atomic_modification: state is not atomic. state=0x565382d7f6e8 2023-10-05T19:28:56.940 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.934+0000 7f5475af1e40 20 bucket index object: default.rgw.buckets.index:.dir.9a9df965-6612-45d4-a45f-6e3c889bacc2.4223.5.2 2023-10-05T19:28:56.940 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.934+0000 7f5475af1e40 1 -- 172.21.15.167:0/1974318067 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:118 4.2 4:42981e29:::.dir.9a9df965-6612-45d4-a45f-6e3c889bacc2.4223.5.2:head [stat,call rgw.guard_bucket_resharding in=36b,call rgw.bucket_prepare_op in=210b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e32) v8 -- 0x565382d84090 con 0x565382cac750 2023-10-05T19:28:56.942 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.938+0000 7f5473e27700 1 -- 172.21.15.167:0/1974318067 <== osd.1 v2:172.21.15.167:6800/30035 36 ==== osd_op_reply(118 .dir.9a9df965-6612-45d4-a45f-6e3c889bacc2.4223.5.2 [stat,call,call] v32'8 uv8 ondisk = 0) v8 ==== 278+0+0 (crc 0 0 0) 0x7f5464050040 con 0x565382cac750 2023-10-05T19:28:56.942 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.938+0000 7f5475af1e40 1 -- 172.21.15.167:0/1974318067 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:119 3.36s0 3:6fce0d3c:::9a9df965-6612-45d4-a45f-6e3c889bacc2.4223.5_9c8454ddf7aa50116496bac348d7550d:head [call rgw.obj_remove in=40b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e32) v8 -- 0x565382d82070 con 0x7f546c01b740 2023-10-05T19:28:56.945 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.938+0000 7f5472e25700 1 -- 172.21.15.167:0/1974318067 <== osd.2 v2:172.21.15.167:6816/30037 23 ==== osd_op_reply(119 9a9df965-6612-45d4-a45f-6e3c889bacc2.4223.5_9c8454ddf7aa50116496bac348d7550d [call] v32'2 uv2 ondisk = 0) v8 ==== 220+0+0 (crc 0 0 0) 0x7f545c006070 con 0x7f546c01b740 2023-10-05T19:28:56.945 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.942+0000 7f5475af1e40 1 -- 172.21.15.167:0/1974318067 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:120 4.2 4:42981e29:::.dir.9a9df965-6612-45d4-a45f-6e3c889bacc2.4223.5.2:head [stat,call rgw.guard_bucket_resharding in=36b,call rgw.bucket_complete_op in=287b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e32) v8 -- 0x565382d846f0 con 0x565382cac750 2023-10-05T19:28:56.946 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.942+0000 7f5475af1e40 10 cache get: name=default.rgw.log++bucket.sync-source-hints.myfoo : miss 2023-10-05T19:28:56.946 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.942+0000 7f5475af1e40 20 rados->read ofs=0 len=0 2023-10-05T19:28:56.946 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.942+0000 7f5475af1e40 1 -- 172.21.15.167:0/1974318067 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:121 6.18 6:1df25bda:::bucket.sync-source-hints.myfoo:head [call version.read in=11b,read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32) v8 -- 0x565382d87350 con 0x7f545c00d8e0 2023-10-05T19:28:56.947 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.942+0000 7f5473e27700 1 -- 172.21.15.167:0/1974318067 <== osd.0 v2:172.21.15.118:6800/27690 33 ==== osd_op_reply(121 bucket.sync-source-hints.myfoo [call,read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 ==== 216+0+0 (crc 0 0 0) 0x7f5464050040 con 0x7f545c00d8e0 2023-10-05T19:28:56.947 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.942+0000 7f5475af1e40 20 rados_obj.operate() r=-2 bl.length=0 2023-10-05T19:28:56.947 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.942+0000 7f5475af1e40 10 cache put: name=default.rgw.log++bucket.sync-source-hints.myfoo info.flags=0x0 2023-10-05T19:28:56.947 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.942+0000 7f5475af1e40 10 adding default.rgw.log++bucket.sync-source-hints.myfoo to cache LRU end 2023-10-05T19:28:56.947 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.942+0000 7f5475af1e40 10 cache get: name=default.rgw.log++bucket.sync-target-hints.myfoo : miss 2023-10-05T19:28:56.947 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.942+0000 7f5475af1e40 20 rados->read ofs=0 len=0 2023-10-05T19:28:56.947 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.942+0000 7f5475af1e40 1 -- 172.21.15.167:0/1974318067 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:122 6.4 6:26e4c4bc:::bucket.sync-target-hints.myfoo:head [call version.read in=11b,read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32) v8 -- 0x565382d87350 con 0x565382cac750 2023-10-05T19:28:56.948 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.942+0000 7f5473e27700 1 -- 172.21.15.167:0/1974318067 <== osd.1 v2:172.21.15.167:6800/30035 37 ==== osd_op_reply(122 bucket.sync-target-hints.myfoo [call,read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 ==== 216+0+0 (crc 0 0 0) 0x7f5464050040 con 0x565382cac750 2023-10-05T19:28:56.949 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.942+0000 7f5475af1e40 20 rados_obj.operate() r=-2 bl.length=0 2023-10-05T19:28:56.949 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.942+0000 7f5475af1e40 10 cache put: name=default.rgw.log++bucket.sync-target-hints.myfoo info.flags=0x0 2023-10-05T19:28:56.949 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.942+0000 7f5475af1e40 10 adding default.rgw.log++bucket.sync-target-hints.myfoo to cache LRU end 2023-10-05T19:28:56.949 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.942+0000 7f5475af1e40 10 chain_cache_entry: cache_locator= 2023-10-05T19:28:56.949 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.942+0000 7f5475af1e40 20 chain_cache_entry: couldn't find cache locator 2023-10-05T19:28:56.949 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.942+0000 7f5475af1e40 20 couldn't put bucket_sync_policy cache entry, might have raced with data changes 2023-10-05T19:28:56.949 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.942+0000 7f5475af1e40 20 get_obj_state: rctx=0x7fff8b74c1a0 obj=myfoo:b1b9a972ccd8c962a473909b97007eb4 state=0x565382d7f6e8 s->prefetch_data=0 2023-10-05T19:28:56.949 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.942+0000 7f5475af1e40 1 -- 172.21.15.167:0/1974318067 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:123 3.30s0 3:0f812532:::9a9df965-6612-45d4-a45f-6e3c889bacc2.4223.5_b1b9a972ccd8c962a473909b97007eb4:head [getxattrs,stat] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32) v8 -- 0x565382d850a0 con 0x565382ce1410 2023-10-05T19:28:56.949 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.942+0000 7f5473626700 1 -- 172.21.15.167:0/1974318067 <== osd.3 v2:172.21.15.167:6808/30036 29 ==== osd_op_reply(123 9a9df965-6612-45d4-a45f-6e3c889bacc2.4223.5_b1b9a972ccd8c962a473909b97007eb4 [getxattrs out=879b,stat out=16b] v0'0 uv1 ondisk = 0) v8 ==== 262+0+895 (crc 0 0 0) 0x7f546c034070 con 0x565382ce1410 2023-10-05T19:28:56.949 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.942+0000 7f5475af1e40 10 manifest: total_size = 3 2023-10-05T19:28:56.949 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.942+0000 7f5475af1e40 20 get_obj_state: setting s->obj_tag to 9a9df965-6612-45d4-a45f-6e3c889bacc2.4221.16505509926696333407 2023-10-05T19:28:56.950 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.942+0000 7f5475af1e40 20 get_obj_state: rctx=0x7fff8b74c1a0 obj=myfoo:b1b9a972ccd8c962a473909b97007eb4 state=0x565382d7f6e8 s->prefetch_data=0 2023-10-05T19:28:56.950 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.942+0000 7f5475af1e40 20 prepare_atomic_modification: state is not atomic. state=0x565382d7f6e8 2023-10-05T19:28:56.950 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.942+0000 7f5475af1e40 20 bucket index object: default.rgw.buckets.index:.dir.9a9df965-6612-45d4-a45f-6e3c889bacc2.4223.5.6 2023-10-05T19:28:56.950 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.942+0000 7f5475af1e40 1 -- 172.21.15.167:0/1974318067 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:124 4.8 4:1342e9a7:::.dir.9a9df965-6612-45d4-a45f-6e3c889bacc2.4223.5.6:head [stat,call rgw.guard_bucket_resharding in=36b,call rgw.bucket_prepare_op in=210b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e32) v8 -- 0x565382d850a0 con 0x565382ce1410 2023-10-05T19:28:56.950 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.942+0000 7f5473e27700 1 -- 172.21.15.167:0/1974318067 <== osd.1 v2:172.21.15.167:6800/30035 38 ==== osd_op_reply(120 .dir.9a9df965-6612-45d4-a45f-6e3c889bacc2.4223.5.2 [stat,call,call] v32'9 uv9 ondisk = 0) v8 ==== 278+0+0 (crc 0 0 0) 0x7f5464050040 con 0x565382cac750 2023-10-05T19:28:56.950 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.942+0000 7f5471622700 20 handle_completion(): completion ok for obj=9c8454ddf7aa50116496bac348d7550d 2023-10-05T19:28:56.951 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.946+0000 7f5473626700 1 -- 172.21.15.167:0/1974318067 <== osd.3 v2:172.21.15.167:6808/30036 30 ==== osd_op_reply(124 .dir.9a9df965-6612-45d4-a45f-6e3c889bacc2.4223.5.6 [stat,call,call] v32'4 uv4 ondisk = 0) v8 ==== 278+0+0 (crc 0 0 0) 0x7f546c034070 con 0x565382ce1410 2023-10-05T19:28:56.951 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.946+0000 7f5475af1e40 1 -- 172.21.15.167:0/1974318067 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:125 3.30s0 3:0f812532:::9a9df965-6612-45d4-a45f-6e3c889bacc2.4223.5_b1b9a972ccd8c962a473909b97007eb4:head [call rgw.obj_remove in=40b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e32) v8 -- 0x565382d85700 con 0x565382ce1410 2023-10-05T19:28:56.955 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.950+0000 7f5473626700 1 -- 172.21.15.167:0/1974318067 <== osd.3 v2:172.21.15.167:6808/30036 31 ==== osd_op_reply(125 9a9df965-6612-45d4-a45f-6e3c889bacc2.4223.5_b1b9a972ccd8c962a473909b97007eb4 [call] v32'2 uv2 ondisk = 0) v8 ==== 220+0+0 (crc 0 0 0) 0x7f546c034070 con 0x565382ce1410 2023-10-05T19:28:56.955 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.950+0000 7f5475af1e40 1 -- 172.21.15.167:0/1974318067 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:126 4.8 4:1342e9a7:::.dir.9a9df965-6612-45d4-a45f-6e3c889bacc2.4223.5.6:head [stat,call rgw.guard_bucket_resharding in=36b,call rgw.bucket_complete_op in=287b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e32) v8 -- 0x565382d83080 con 0x565382ce1410 2023-10-05T19:28:56.955 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.950+0000 7f5475af1e40 10 cache get: name=default.rgw.log++bucket.sync-source-hints.myfoo : hit (negative entry) 2023-10-05T19:28:56.955 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.950+0000 7f5475af1e40 10 cache get: name=default.rgw.log++bucket.sync-target-hints.myfoo : hit (negative entry) 2023-10-05T19:28:56.956 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.950+0000 7f5475af1e40 10 chain_cache_entry: cache_locator= 2023-10-05T19:28:56.956 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.950+0000 7f5475af1e40 20 chain_cache_entry: couldn't find cache locator 2023-10-05T19:28:56.956 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.950+0000 7f5475af1e40 20 couldn't put bucket_sync_policy cache entry, might have raced with data changes 2023-10-05T19:28:56.956 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.950+0000 7f5475af1e40 20 get_obj_state: rctx=0x7fff8b74c1a0 obj=myfoo:c785e1ed2950e3e36b1e2ca01f299a54 state=0x565382d7f6e8 s->prefetch_data=0 2023-10-05T19:28:56.956 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.950+0000 7f5475af1e40 1 -- 172.21.15.167:0/1974318067 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:127 3.2fs0 3:f4880c09:::9a9df965-6612-45d4-a45f-6e3c889bacc2.4223.5_c785e1ed2950e3e36b1e2ca01f299a54:head [getxattrs,stat] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32) v8 -- 0x565382d83080 con 0x565382ce1410 2023-10-05T19:28:56.956 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.950+0000 7f5473626700 1 -- 172.21.15.167:0/1974318067 <== osd.3 v2:172.21.15.167:6808/30036 32 ==== osd_op_reply(127 9a9df965-6612-45d4-a45f-6e3c889bacc2.4223.5_c785e1ed2950e3e36b1e2ca01f299a54 [getxattrs out=879b,stat out=16b] v0'0 uv1 ondisk = 0) v8 ==== 262+0+895 (crc 0 0 0) 0x7f546c034070 con 0x565382ce1410 2023-10-05T19:28:56.956 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.950+0000 7f5475af1e40 10 manifest: total_size = 4 2023-10-05T19:28:56.956 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.950+0000 7f5475af1e40 20 get_obj_state: setting s->obj_tag to 9a9df965-6612-45d4-a45f-6e3c889bacc2.4221.15426725478028318743 2023-10-05T19:28:56.957 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.950+0000 7f5475af1e40 20 get_obj_state: rctx=0x7fff8b74c1a0 obj=myfoo:c785e1ed2950e3e36b1e2ca01f299a54 state=0x565382d7f6e8 s->prefetch_data=0 2023-10-05T19:28:56.957 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.950+0000 7f5475af1e40 20 prepare_atomic_modification: state is not atomic. state=0x565382d7f6e8 2023-10-05T19:28:56.957 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.950+0000 7f5475af1e40 20 bucket index object: default.rgw.buckets.index:.dir.9a9df965-6612-45d4-a45f-6e3c889bacc2.4223.5.8 2023-10-05T19:28:56.957 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.950+0000 7f5475af1e40 1 -- 172.21.15.167:0/1974318067 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:128 4.21 4:84ddff17:::.dir.9a9df965-6612-45d4-a45f-6e3c889bacc2.4223.5.8:head [stat,call rgw.guard_bucket_resharding in=36b,call rgw.bucket_prepare_op in=210b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e32) v8 -- 0x565382d83080 con 0x7f545c00d8e0 2023-10-05T19:28:56.957 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.950+0000 7f5473626700 1 -- 172.21.15.167:0/1974318067 <== osd.3 v2:172.21.15.167:6808/30036 33 ==== osd_op_reply(126 .dir.9a9df965-6612-45d4-a45f-6e3c889bacc2.4223.5.6 [stat,call,call] v32'5 uv5 ondisk = 0) v8 ==== 278+0+0 (crc 0 0 0) 0x7f546c034070 con 0x565382ce1410 2023-10-05T19:28:56.958 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.950+0000 7f5470e21700 20 handle_completion(): completion ok for obj=b1b9a972ccd8c962a473909b97007eb4 2023-10-05T19:28:56.958 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.954+0000 7f5473e27700 1 -- 172.21.15.167:0/1974318067 <== osd.0 v2:172.21.15.118:6800/27690 34 ==== osd_op_reply(128 .dir.9a9df965-6612-45d4-a45f-6e3c889bacc2.4223.5.8 [stat,call,call] v32'6 uv6 ondisk = 0) v8 ==== 278+0+0 (crc 0 0 0) 0x7f5464050040 con 0x7f545c00d8e0 2023-10-05T19:28:56.958 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.954+0000 7f5475af1e40 1 -- 172.21.15.167:0/1974318067 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:129 3.2fs0 3:f4880c09:::9a9df965-6612-45d4-a45f-6e3c889bacc2.4223.5_c785e1ed2950e3e36b1e2ca01f299a54:head [call rgw.obj_remove in=40b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e32) v8 -- 0x565382d83a10 con 0x565382ce1410 2023-10-05T19:28:56.961 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.954+0000 7f5473626700 1 -- 172.21.15.167:0/1974318067 <== osd.3 v2:172.21.15.167:6808/30036 34 ==== osd_op_reply(129 9a9df965-6612-45d4-a45f-6e3c889bacc2.4223.5_c785e1ed2950e3e36b1e2ca01f299a54 [call] v32'2 uv2 ondisk = 0) v8 ==== 220+0+0 (crc 0 0 0) 0x7f546c034070 con 0x565382ce1410 2023-10-05T19:28:56.961 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.954+0000 7f5475af1e40 1 -- 172.21.15.167:0/1974318067 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:130 4.21 4:84ddff17:::.dir.9a9df965-6612-45d4-a45f-6e3c889bacc2.4223.5.8:head [stat,call rgw.guard_bucket_resharding in=36b,call rgw.bucket_complete_op in=287b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e32) v8 -- 0x565382d83a10 con 0x7f545c00d8e0 2023-10-05T19:28:56.961 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.954+0000 7f5475af1e40 10 cache get: name=default.rgw.log++bucket.sync-source-hints.myfoo : hit (negative entry) 2023-10-05T19:28:56.961 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.954+0000 7f5475af1e40 10 cache get: name=default.rgw.log++bucket.sync-target-hints.myfoo : hit (negative entry) 2023-10-05T19:28:56.962 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.954+0000 7f5475af1e40 10 chain_cache_entry: cache_locator= 2023-10-05T19:28:56.962 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.954+0000 7f5475af1e40 20 chain_cache_entry: couldn't find cache locator 2023-10-05T19:28:56.962 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.954+0000 7f5475af1e40 20 couldn't put bucket_sync_policy cache entry, might have raced with data changes 2023-10-05T19:28:56.962 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.954+0000 7f5475af1e40 20 get_obj_state: rctx=0x7fff8b74c1a0 obj=myfoo:24d27c169c2c881eb09a065116f2aa5c state=0x565382d7f6e8 s->prefetch_data=0 2023-10-05T19:28:56.962 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.954+0000 7f5475af1e40 1 -- 172.21.15.167:0/1974318067 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:131 3.2cs0 3:344ef3ef:::9a9df965-6612-45d4-a45f-6e3c889bacc2.4223.5_24d27c169c2c881eb09a065116f2aa5c:head [getxattrs,stat] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32) v8 -- 0x565382d87750 con 0x565382ce1410 2023-10-05T19:28:56.962 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.958+0000 7f5473626700 1 -- 172.21.15.167:0/1974318067 <== osd.3 v2:172.21.15.167:6808/30036 35 ==== osd_op_reply(131 9a9df965-6612-45d4-a45f-6e3c889bacc2.4223.5_24d27c169c2c881eb09a065116f2aa5c [getxattrs out=879b,stat out=16b] v0'0 uv1 ondisk = 0) v8 ==== 262+0+895 (crc 0 0 0) 0x7f546c034070 con 0x565382ce1410 2023-10-05T19:28:56.962 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.958+0000 7f5475af1e40 10 manifest: total_size = 5 2023-10-05T19:28:56.963 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.958+0000 7f5475af1e40 20 get_obj_state: setting s->obj_tag to 9a9df965-6612-45d4-a45f-6e3c889bacc2.4221.12203978142987706629 2023-10-05T19:28:56.963 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.958+0000 7f5475af1e40 20 get_obj_state: rctx=0x7fff8b74c1a0 obj=myfoo:24d27c169c2c881eb09a065116f2aa5c state=0x565382d7f6e8 s->prefetch_data=0 2023-10-05T19:28:56.963 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.958+0000 7f5475af1e40 20 prepare_atomic_modification: state is not atomic. state=0x565382d7f6e8 2023-10-05T19:28:56.963 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.958+0000 7f5475af1e40 20 bucket index object: default.rgw.buckets.index:.dir.9a9df965-6612-45d4-a45f-6e3c889bacc2.4223.5.10 2023-10-05T19:28:56.963 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.958+0000 7f5475af1e40 1 -- 172.21.15.167:0/1974318067 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:132 4.25 4:a63b86a5:::.dir.9a9df965-6612-45d4-a45f-6e3c889bacc2.4223.5.10:head [stat,call rgw.guard_bucket_resharding in=36b,call rgw.bucket_prepare_op in=210b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e32) v8 -- 0x565382d8ba00 con 0x565382ce1410 2023-10-05T19:28:56.964 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.958+0000 7f5473e27700 1 -- 172.21.15.167:0/1974318067 <== osd.0 v2:172.21.15.118:6800/27690 35 ==== osd_op_reply(130 .dir.9a9df965-6612-45d4-a45f-6e3c889bacc2.4223.5.8 [stat,call,call] v32'7 uv7 ondisk = 0) v8 ==== 278+0+0 (crc 0 0 0) 0x7f5464050040 con 0x7f545c00d8e0 2023-10-05T19:28:56.964 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.958+0000 7f5471622700 20 handle_completion(): completion ok for obj=c785e1ed2950e3e36b1e2ca01f299a54 2023-10-05T19:28:56.964 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.958+0000 7f5473626700 1 -- 172.21.15.167:0/1974318067 <== osd.3 v2:172.21.15.167:6808/30036 36 ==== osd_op_reply(132 .dir.9a9df965-6612-45d4-a45f-6e3c889bacc2.4223.5.10 [stat,call,call] v32'10 uv10 ondisk = 0) v8 ==== 279+0+0 (crc 0 0 0) 0x7f546c034070 con 0x565382ce1410 2023-10-05T19:28:56.964 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.958+0000 7f5475af1e40 1 -- 172.21.15.167:0/1974318067 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:133 3.2cs0 3:344ef3ef:::9a9df965-6612-45d4-a45f-6e3c889bacc2.4223.5_24d27c169c2c881eb09a065116f2aa5c:head [call rgw.obj_remove in=40b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e32) v8 -- 0x565382d83810 con 0x565382ce1410 2023-10-05T19:28:56.967 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.962+0000 7f5473626700 1 -- 172.21.15.167:0/1974318067 <== osd.3 v2:172.21.15.167:6808/30036 37 ==== osd_op_reply(133 9a9df965-6612-45d4-a45f-6e3c889bacc2.4223.5_24d27c169c2c881eb09a065116f2aa5c [call] v32'2 uv2 ondisk = 0) v8 ==== 220+0+0 (crc 0 0 0) 0x7f546c034070 con 0x565382ce1410 2023-10-05T19:28:56.967 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.962+0000 7f5475af1e40 1 -- 172.21.15.167:0/1974318067 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:134 4.25 4:a63b86a5:::.dir.9a9df965-6612-45d4-a45f-6e3c889bacc2.4223.5.10:head [stat,call rgw.guard_bucket_resharding in=36b,call rgw.bucket_complete_op in=287b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e32) v8 -- 0x565382d8ba00 con 0x565382ce1410 2023-10-05T19:28:56.968 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.962+0000 7f5475af1e40 10 cache get: name=default.rgw.log++bucket.sync-source-hints.myfoo : hit (negative entry) 2023-10-05T19:28:56.968 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.962+0000 7f5475af1e40 10 cache get: name=default.rgw.log++bucket.sync-target-hints.myfoo : hit (negative entry) 2023-10-05T19:28:56.968 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.962+0000 7f5475af1e40 10 chain_cache_entry: cache_locator= 2023-10-05T19:28:56.968 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.962+0000 7f5475af1e40 20 chain_cache_entry: couldn't find cache locator 2023-10-05T19:28:56.968 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.962+0000 7f5475af1e40 20 couldn't put bucket_sync_policy cache entry, might have raced with data changes 2023-10-05T19:28:56.968 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.962+0000 7f5475af1e40 20 get_obj_state: rctx=0x7fff8b74c1a0 obj=myfoo:bb3aec0fdcdbc2974890f805c585d432 state=0x565382d7f6e8 s->prefetch_data=0 2023-10-05T19:28:56.968 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.962+0000 7f5475af1e40 1 -- 172.21.15.167:0/1974318067 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:135 3.1cs0 3:38675d0b:::9a9df965-6612-45d4-a45f-6e3c889bacc2.4223.5_bb3aec0fdcdbc2974890f805c585d432:head [getxattrs,stat] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32) v8 -- 0x565382d87e10 con 0x565382cac750 2023-10-05T19:28:56.969 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.962+0000 7f5473e27700 1 -- 172.21.15.167:0/1974318067 <== osd.1 v2:172.21.15.167:6800/30035 39 ==== osd_op_reply(135 9a9df965-6612-45d4-a45f-6e3c889bacc2.4223.5_bb3aec0fdcdbc2974890f805c585d432 [getxattrs out=910b,stat out=16b] v0'0 uv3 ondisk = 0) v8 ==== 262+0+926 (crc 0 0 0) 0x7f5464050040 con 0x565382cac750 2023-10-05T19:28:56.969 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.962+0000 7f5475af1e40 10 manifest: total_size = 5 2023-10-05T19:28:56.969 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.962+0000 7f5475af1e40 20 get_obj_state: setting s->obj_tag to _RiM11gV4_HUAssJBJPt8rz1mj38MntN 2023-10-05T19:28:56.969 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.962+0000 7f5475af1e40 20 get_obj_state: rctx=0x7fff8b74c1a0 obj=myfoo:bb3aec0fdcdbc2974890f805c585d432 state=0x565382d7f6e8 s->prefetch_data=0 2023-10-05T19:28:56.969 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.962+0000 7f5475af1e40 20 prepare_atomic_modification: state is not atomic. state=0x565382d7f6e8 2023-10-05T19:28:56.969 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.962+0000 7f5475af1e40 20 bucket index object: default.rgw.buckets.index:.dir.9a9df965-6612-45d4-a45f-6e3c889bacc2.4223.5.10 2023-10-05T19:28:56.969 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.962+0000 7f5475af1e40 1 -- 172.21.15.167:0/1974318067 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:136 4.25 4:a63b86a5:::.dir.9a9df965-6612-45d4-a45f-6e3c889bacc2.4223.5.10:head [stat,call rgw.guard_bucket_resharding in=36b,call rgw.bucket_prepare_op in=210b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e32) v8 -- 0x565382d8bb30 con 0x565382ce1410 2023-10-05T19:28:56.970 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.966+0000 7f5473626700 1 -- 172.21.15.167:0/1974318067 <== osd.3 v2:172.21.15.167:6808/30036 38 ==== osd_op_reply(134 .dir.9a9df965-6612-45d4-a45f-6e3c889bacc2.4223.5.10 [stat,call,call] v32'11 uv11 ondisk = 0) v8 ==== 279+0+0 (crc 0 0 0) 0x7f546c034070 con 0x565382ce1410 2023-10-05T19:28:56.970 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.966+0000 7f5470e21700 20 handle_completion(): completion ok for obj=24d27c169c2c881eb09a065116f2aa5c 2023-10-05T19:28:56.972 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.966+0000 7f5473626700 1 -- 172.21.15.167:0/1974318067 <== osd.3 v2:172.21.15.167:6808/30036 39 ==== osd_op_reply(136 .dir.9a9df965-6612-45d4-a45f-6e3c889bacc2.4223.5.10 [stat,call,call] v32'12 uv12 ondisk = 0) v8 ==== 279+0+0 (crc 0 0 0) 0x7f546c034070 con 0x565382ce1410 2023-10-05T19:28:56.972 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.966+0000 7f5475af1e40 1 -- 172.21.15.167:0/1974318067 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:137 3.1cs0 3:38675d0b:::9a9df965-6612-45d4-a45f-6e3c889bacc2.4223.5_bb3aec0fdcdbc2974890f805c585d432:head [call rgw.obj_remove in=40b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e32) v8 -- 0x565382d8bb30 con 0x565382cac750 2023-10-05T19:28:56.975 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.970+0000 7f5473e27700 1 -- 172.21.15.167:0/1974318067 <== osd.1 v2:172.21.15.167:6800/30035 40 ==== osd_op_reply(137 9a9df965-6612-45d4-a45f-6e3c889bacc2.4223.5_bb3aec0fdcdbc2974890f805c585d432 [call] v32'4 uv4 ondisk = 0) v8 ==== 220+0+0 (crc 0 0 0) 0x7f5464050040 con 0x565382cac750 2023-10-05T19:28:56.975 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.970+0000 7f5475af1e40 1 -- 172.21.15.167:0/1974318067 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:138 4.25 4:a63b86a5:::.dir.9a9df965-6612-45d4-a45f-6e3c889bacc2.4223.5.10:head [stat,call rgw.guard_bucket_resharding in=36b,call rgw.bucket_complete_op in=287b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e32) v8 -- 0x565382d83640 con 0x565382ce1410 2023-10-05T19:28:56.976 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.970+0000 7f5475af1e40 10 cache get: name=default.rgw.log++bucket.sync-source-hints.myfoo : hit (negative entry) 2023-10-05T19:28:56.976 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.970+0000 7f5475af1e40 10 cache get: name=default.rgw.log++bucket.sync-target-hints.myfoo : hit (negative entry) 2023-10-05T19:28:56.976 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.970+0000 7f5475af1e40 10 chain_cache_entry: cache_locator= 2023-10-05T19:28:56.976 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.970+0000 7f5475af1e40 20 chain_cache_entry: couldn't find cache locator 2023-10-05T19:28:56.976 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.970+0000 7f5475af1e40 20 couldn't put bucket_sync_policy cache entry, might have raced with data changes 2023-10-05T19:28:56.976 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.970+0000 7f5475af1e40 20 int RGWRados::Bucket::List::list_objects_ordered(const DoutPrefixProvider*, int64_t, std::vector*, std::map, bool>*, bool*, optional_yield): starting attempt 1 2023-10-05T19:28:56.976 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.970+0000 7f5475af1e40 10 int RGWRados::cls_bucket_list_ordered(const DoutPrefixProvider*, RGWBucketInfo&, int, const rgw_obj_index_key&, const string&, const string&, uint32_t, bool, uint16_t, RGWRados::ent_map_t&, bool*, bool*, rgw_obj_index_key*, optional_yield, RGWBucketListNameFilter): :myfoo[9a9df965-6612-45d4-a45f-6e3c889bacc2.4223.5]) start_after="_multipart_", prefix="_multipart_, delimiter="", shard_id=-1", num_entries=1001, list_versions=0, expansion_factor=1, force_check_filter is unset 2023-10-05T19:28:56.976 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.970+0000 7f5475af1e40 10 int RGWRados::cls_bucket_list_ordered(const DoutPrefixProvider*, RGWBucketInfo&, int, const rgw_obj_index_key&, const string&, const string&, uint32_t, bool, uint16_t, RGWRados::ent_map_t&, bool*, bool*, rgw_obj_index_key*, optional_yield, RGWBucketListNameFilter): request from each of 11 shard(s) for 112 entries to get 1001 total entries 2023-10-05T19:28:56.976 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.970+0000 7f5475af1e40 1 -- 172.21.15.167:0/1974318067 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:139 4.2d 4:b5ae0298:::.dir.9a9df965-6612-45d4-a45f-6e3c889bacc2.4223.5.0:head [call rgw.bucket_list in=69b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32) v8 -- 0x7f5464073000 con 0x7f545c00d8e0 2023-10-05T19:28:56.976 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.970+0000 7f5475af1e40 1 -- 172.21.15.167:0/1974318067 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:140 4.34 4:2ef4a173:::.dir.9a9df965-6612-45d4-a45f-6e3c889bacc2.4223.5.1:head [call rgw.bucket_list in=69b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32) v8 -- 0x565382d82a20 con 0x7f546c01b740 2023-10-05T19:28:56.977 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.970+0000 7f5475af1e40 1 -- 172.21.15.167:0/1974318067 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:141 4.2 4:42981e29:::.dir.9a9df965-6612-45d4-a45f-6e3c889bacc2.4223.5.2:head [call rgw.bucket_list in=69b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32) v8 -- 0x565382d8ec50 con 0x565382cac750 2023-10-05T19:28:56.979 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.970+0000 7f5475af1e40 1 -- 172.21.15.167:0/1974318067 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:142 4.3d 4:bdb71a6b:::.dir.9a9df965-6612-45d4-a45f-6e3c889bacc2.4223.5.3:head [call rgw.bucket_list in=69b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32) v8 -- 0x565382d8ec50 con 0x565382ce1410 2023-10-05T19:28:56.979 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.970+0000 7f5475af1e40 1 -- 172.21.15.167:0/1974318067 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:143 4.33 4:cc3cd978:::.dir.9a9df965-6612-45d4-a45f-6e3c889bacc2.4223.5.4:head [call rgw.bucket_list in=69b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32) v8 -- 0x565382d93d30 con 0x7f545c00d8e0 2023-10-05T19:28:56.979 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.970+0000 7f5475af1e40 1 -- 172.21.15.167:0/1974318067 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:144 4.16 4:6b936802:::.dir.9a9df965-6612-45d4-a45f-6e3c889bacc2.4223.5.5:head [call rgw.bucket_list in=69b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32) v8 -- 0x565382d93db0 con 0x565382ce1410 2023-10-05T19:28:56.979 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.970+0000 7f5473e27700 1 -- 172.21.15.167:0/1974318067 <== osd.0 v2:172.21.15.118:6800/27690 36 ==== osd_op_reply(139 .dir.9a9df965-6612-45d4-a45f-6e3c889bacc2.4223.5.0 [call out=85b] v0'0 uv5 ondisk = 0) v8 ==== 194+0+85 (crc 0 0 0) 0x7f5464050040 con 0x7f545c00d8e0 2023-10-05T19:28:56.980 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.970+0000 7f5475af1e40 1 -- 172.21.15.167:0/1974318067 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:145 4.8 4:1342e9a7:::.dir.9a9df965-6612-45d4-a45f-6e3c889bacc2.4223.5.6:head [call rgw.bucket_list in=69b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32) v8 -- 0x565382d991b0 con 0x565382ce1410 2023-10-05T19:28:56.980 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.970+0000 7f5475af1e40 1 -- 172.21.15.167:0/1974318067 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:146 4.d 4:b00cdfe0:::.dir.9a9df965-6612-45d4-a45f-6e3c889bacc2.4223.5.7:head [call rgw.bucket_list in=69b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32) v8 -- 0x565382d836a0 con 0x565382ce1410 2023-10-05T19:28:56.980 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.970+0000 7f5472e25700 1 -- 172.21.15.167:0/1974318067 <== osd.2 v2:172.21.15.167:6816/30037 24 ==== osd_op_reply(140 .dir.9a9df965-6612-45d4-a45f-6e3c889bacc2.4223.5.1 [call out=85b] v0'0 uv1 ondisk = 0) v8 ==== 194+0+85 (crc 0 0 0) 0x7f545c006070 con 0x7f546c01b740 2023-10-05T19:28:56.980 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.970+0000 7f5475af1e40 1 -- 172.21.15.167:0/1974318067 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:147 4.21 4:84ddff17:::.dir.9a9df965-6612-45d4-a45f-6e3c889bacc2.4223.5.8:head [call rgw.bucket_list in=69b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32) v8 -- 0x565382d9c9c0 con 0x7f545c00d8e0 2023-10-05T19:28:56.980 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.970+0000 7f5475af1e40 1 -- 172.21.15.167:0/1974318067 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:148 4.4 4:20b95fb7:::.dir.9a9df965-6612-45d4-a45f-6e3c889bacc2.4223.5.9:head [call rgw.bucket_list in=69b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32) v8 -- 0x565382d9d300 con 0x7f545c00d8e0 2023-10-05T19:28:56.980 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.970+0000 7f5473e27700 1 -- 172.21.15.167:0/1974318067 <== osd.1 v2:172.21.15.167:6800/30035 41 ==== osd_op_reply(141 .dir.9a9df965-6612-45d4-a45f-6e3c889bacc2.4223.5.2 [call out=124b] v0'0 uv9 ondisk = 0) v8 ==== 194+0+124 (crc 0 0 0) 0x7f5464050040 con 0x565382cac750 2023-10-05T19:28:56.980 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.970+0000 7f5475af1e40 1 -- 172.21.15.167:0/1974318067 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:149 4.25 4:a63b86a5:::.dir.9a9df965-6612-45d4-a45f-6e3c889bacc2.4223.5.10:head [call rgw.bucket_list in=69b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32) v8 -- 0x565382d9ff20 con 0x565382ce1410 2023-10-05T19:28:56.980 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.970+0000 7f5473626700 1 -- 172.21.15.167:0/1974318067 <== osd.3 v2:172.21.15.167:6808/30036 40 ==== osd_op_reply(142 .dir.9a9df965-6612-45d4-a45f-6e3c889bacc2.4223.5.3 [call out=85b] v0'0 uv5 ondisk = 0) v8 ==== 194+0+85 (crc 0 0 0) 0x7f546c034070 con 0x565382ce1410 2023-10-05T19:28:56.980 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.970+0000 7f5473e27700 1 -- 172.21.15.167:0/1974318067 <== osd.0 v2:172.21.15.118:6800/27690 37 ==== osd_op_reply(143 .dir.9a9df965-6612-45d4-a45f-6e3c889bacc2.4223.5.4 [call out=85b] v0'0 uv3 ondisk = 0) v8 ==== 194+0+85 (crc 0 0 0) 0x7f5464050040 con 0x7f545c00d8e0 2023-10-05T19:28:56.980 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.970+0000 7f5473e27700 1 -- 172.21.15.167:0/1974318067 <== osd.0 v2:172.21.15.118:6800/27690 38 ==== osd_op_reply(147 .dir.9a9df965-6612-45d4-a45f-6e3c889bacc2.4223.5.8 [call out=124b] v0'0 uv7 ondisk = 0) v8 ==== 194+0+124 (crc 0 0 0) 0x7f5464050040 con 0x7f545c00d8e0 2023-10-05T19:28:56.980 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.970+0000 7f5473e27700 1 -- 172.21.15.167:0/1974318067 <== osd.0 v2:172.21.15.118:6800/27690 39 ==== osd_op_reply(148 .dir.9a9df965-6612-45d4-a45f-6e3c889bacc2.4223.5.9 [call out=85b] v0'0 uv3 ondisk = 0) v8 ==== 194+0+85 (crc 0 0 0) 0x7f5464050040 con 0x7f545c00d8e0 2023-10-05T19:28:56.981 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.970+0000 7f5473626700 1 -- 172.21.15.167:0/1974318067 <== osd.3 v2:172.21.15.167:6808/30036 41 ==== osd_op_reply(145 .dir.9a9df965-6612-45d4-a45f-6e3c889bacc2.4223.5.6 [call out=124b] v0'0 uv5 ondisk = 0) v8 ==== 194+0+124 (crc 0 0 0) 0x7f546c034070 con 0x565382ce1410 2023-10-05T19:28:56.981 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.970+0000 7f5473626700 1 -- 172.21.15.167:0/1974318067 <== osd.3 v2:172.21.15.167:6808/30036 42 ==== osd_op_reply(144 .dir.9a9df965-6612-45d4-a45f-6e3c889bacc2.4223.5.5 [call out=85b] v0'0 uv3 ondisk = 0) v8 ==== 194+0+85 (crc 0 0 0) 0x7f546c034070 con 0x565382ce1410 2023-10-05T19:28:56.981 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.970+0000 7f5473626700 1 -- 172.21.15.167:0/1974318067 <== osd.3 v2:172.21.15.167:6808/30036 43 ==== osd_op_reply(146 .dir.9a9df965-6612-45d4-a45f-6e3c889bacc2.4223.5.7 [call out=85b] v0'0 uv1 ondisk = 0) v8 ==== 194+0+85 (crc 0 0 0) 0x7f546c034070 con 0x565382ce1410 2023-10-05T19:28:56.981 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.974+0000 7f5473626700 1 -- 172.21.15.167:0/1974318067 <== osd.3 v2:172.21.15.167:6808/30036 44 ==== osd_op_reply(138 .dir.9a9df965-6612-45d4-a45f-6e3c889bacc2.4223.5.10 [stat,call,call] v32'13 uv13 ondisk = 0) v8 ==== 279+0+0 (crc 0 0 0) 0x7f546c034070 con 0x565382ce1410 2023-10-05T19:28:56.981 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.974+0000 7f5471622700 20 handle_completion(): completion ok for obj=bb3aec0fdcdbc2974890f805c585d432 2023-10-05T19:28:56.981 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.974+0000 7f5473626700 1 -- 172.21.15.167:0/1974318067 <== osd.3 v2:172.21.15.167:6808/30036 45 ==== osd_op_reply(149 .dir.9a9df965-6612-45d4-a45f-6e3c889bacc2.4223.5.10 [call out=124b] v0'0 uv13 ondisk = 0) v8 ==== 195+0+124 (crc 0 0 0) 0x7f546c034070 con 0x565382ce1410 2023-10-05T19:28:56.981 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.974+0000 7f5475af1e40 20 int RGWRados::cls_bucket_list_ordered(const DoutPrefixProvider*, RGWBucketInfo&, int, const rgw_obj_index_key&, const string&, const string&, uint32_t, bool, uint16_t, RGWRados::ent_map_t&, bool*, bool*, rgw_obj_index_key*, optional_yield, RGWBucketListNameFilter): returning, count=0, is_truncated=0 2023-10-05T19:28:56.981 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.974+0000 7f5475af1e40 20 int RGWRados::cls_bucket_list_ordered(const DoutPrefixProvider*, RGWBucketInfo&, int, const rgw_obj_index_key&, const string&, const string&, uint32_t, bool, uint16_t, RGWRados::ent_map_t&, bool*, bool*, rgw_obj_index_key*, optional_yield, RGWBucketListNameFilter): returning, last_entry NOT SET 2023-10-05T19:28:56.981 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.974+0000 7f5475af1e40 10 int RGWRados::Bucket::List::list_objects_ordered(const DoutPrefixProvider*, int64_t, std::vector*, std::map, bool>*, bool*, optional_yield): end of outer loop, truncated=0, count=0, attempt=1 2023-10-05T19:28:56.981 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.974+0000 7f5475af1e40 10 int RGWRados::Bucket::List::list_objects_ordered(const DoutPrefixProvider*, int64_t, std::vector*, std::map, bool>*, bool*, optional_yield): exiting attempt loop because we reached end (0) or we're returning half the requested entries (0 of 1000) 2023-10-05T19:28:56.982 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.974+0000 7f5475af1e40 20 abort_multiparts INFO: aborting and cleaning up multipart upload(s); bucket=":myfoo[9a9df965-6612-45d4-a45f-6e3c889bacc2.4223.5])"; uploads.size()=0; is_truncated=0 2023-10-05T19:28:56.982 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.974+0000 7f5475af1e40 10 lifecycle: RGWRados::convert_old_bucket_info(): bucket=:myfoo[9a9df965-6612-45d4-a45f-6e3c889bacc2.4223.5]) 2023-10-05T19:28:56.982 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.974+0000 7f5475af1e40 20 lifecycle: get_system_obj_state: rctx=0x7fff8b74bc38 obj=default.rgw.meta:root:myfoo state=0x565382b5b2b0 s->prefetch_data=0 2023-10-05T19:28:56.982 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.974+0000 7f5475af1e40 10 lifecycle: cache get: name=default.rgw.meta+root+myfoo : type miss (requested=0x16, cached=0x13) 2023-10-05T19:28:56.982 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.974+0000 7f5475af1e40 1 -- 172.21.15.167:0/1974318067 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:150 8.9 8:91b71841:root::myfoo:head [call version.read in=11b,getxattrs,stat] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32) v8 -- 0x565382d93db0 con 0x7f546c01b740 2023-10-05T19:28:56.982 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.974+0000 7f5472e25700 1 -- 172.21.15.167:0/1974318067 <== osd.2 v2:172.21.15.167:6816/30037 25 ==== osd_op_reply(150 myfoo [call out=48b,getxattrs out=75b,stat out=16b] v0'0 uv34 ondisk = 0) v8 ==== 233+0+139 (crc 0 0 0) 0x7f545c006070 con 0x7f546c01b740 2023-10-05T19:28:56.982 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.974+0000 7f5475af1e40 10 lifecycle: cache put: name=default.rgw.meta+root+myfoo info.flags=0x16 2023-10-05T19:28:56.982 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.974+0000 7f5475af1e40 10 lifecycle: moving default.rgw.meta+root+myfoo to cache LRU end 2023-10-05T19:28:56.982 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.974+0000 7f5475af1e40 10 lifecycle: updating xattr: name=ceph.objclass.version bl.length()=42 2023-10-05T19:28:56.982 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.974+0000 7f5475af1e40 20 lifecycle: get_system_obj_state: s->obj_tag was set empty 2023-10-05T19:28:56.982 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.974+0000 7f5475af1e40 20 lifecycle: Read xattr: user.rgw.idtag 2023-10-05T19:28:56.983 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.974+0000 7f5475af1e40 10 lifecycle: cache get: name=default.rgw.meta+root+myfoo : hit (requested=0x13, cached=0x17) 2023-10-05T19:28:56.983 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.974+0000 7f5475af1e40 1 -- 172.21.15.167:0/1974318067 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:151 8.9 8:96cb4b0f:root::.bucket.meta.myfoo%3a9a9df965-6612-45d4-a45f-6e3c889bacc2.4223.5:head [delete,create,call version.check_conds in=122b,call version.set in=58b,writefull 0~352 in=352b,setxattr user.rgw.acl (123) in=135b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e32) v8 -- 0x565382d9ff20 con 0x7f546c01b740 2023-10-05T19:28:56.984 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.978+0000 7f5472e25700 1 -- 172.21.15.167:0/1974318067 <== osd.2 v2:172.21.15.167:6816/30037 26 ==== osd_op_reply(151 .bucket.meta.myfoo:9a9df965-6612-45d4-a45f-6e3c889bacc2.4223.5 [delete,create,call,call,writefull 0~352,setxattr (123)] v32'37 uv37 ondisk = 0) v8 ==== 416+0+0 (crc 0 0 0) 0x7f545c006070 con 0x7f546c01b740 2023-10-05T19:28:56.984 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.978+0000 7f5475af1e40 10 lifecycle: cache put: name=default.rgw.meta+root+.bucket.meta.myfoo:9a9df965-6612-45d4-a45f-6e3c889bacc2.4223.5 info.flags=0x17 2023-10-05T19:28:56.984 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.978+0000 7f5475af1e40 10 lifecycle: moving default.rgw.meta+root+.bucket.meta.myfoo:9a9df965-6612-45d4-a45f-6e3c889bacc2.4223.5 to cache LRU end 2023-10-05T19:28:56.984 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.978+0000 7f5475af1e40 10 lifecycle: updating xattr: name=user.rgw.acl bl.length()=123 2023-10-05T19:28:56.984 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.978+0000 7f5475af1e40 10 lifecycle: distributing notification oid=default.rgw.control:notify.7 cni=[op: 0, obj: default.rgw.meta:root:.bucket.meta.myfoo:9a9df965-6612-45d4-a45f-6e3c889bacc2.4223.5, ofs0, ns] 2023-10-05T19:28:56.984 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.978+0000 7f5475af1e40 1 -- 172.21.15.167:0/1974318067 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:152 7.9 7:93e5b521:::notify.7:head [notify cookie 94916677482048 in=737b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32) v8 -- 0x565382d9ff20 con 0x7f545c00d8e0 2023-10-05T19:28:56.985 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.978+0000 7f5473e27700 1 -- 172.21.15.167:0/1974318067 <== osd.0 v2:172.21.15.118:6800/27690 40 ==== watch-notify(notify (1) cookie 94916676862272 notify 137438953497 ret 0) v3 ==== 767+0+0 (crc 0 0 0) 0x7f5464074040 con 0x7f545c00d8e0 2023-10-05T19:28:56.985 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.978+0000 7f5471622700 10 rgw watcher librados: RGWWatcher::handle_notify() notify_id 137438953497 cookie 94916676862272 notifier 4641 bl.length()=725 2023-10-05T19:28:56.985 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.978+0000 7f5473e27700 1 -- 172.21.15.167:0/1974318067 <== osd.0 v2:172.21.15.118:6800/27690 41 ==== osd_op_reply(152 notify.7 [notify cookie 94916677482048 out=8b] v0'0 uv207 ondisk = 0) v8 ==== 152+0+8 (crc 0 0 0) 0x7f5464050040 con 0x7f545c00d8e0 2023-10-05T19:28:56.985 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.978+0000 7f5471622700 10 rgw watcher librados: cache put: name=default.rgw.meta+root+.bucket.meta.myfoo:9a9df965-6612-45d4-a45f-6e3c889bacc2.4223.5 info.flags=0x17 2023-10-05T19:28:56.985 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.978+0000 7f5471622700 10 rgw watcher librados: moving default.rgw.meta+root+.bucket.meta.myfoo:9a9df965-6612-45d4-a45f-6e3c889bacc2.4223.5 to cache LRU end 2023-10-05T19:28:56.985 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.978+0000 7f5471622700 10 rgw watcher librados: updating xattr: name=user.rgw.acl bl.length()=123 2023-10-05T19:28:56.985 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.978+0000 7f5471622700 1 -- 172.21.15.167:0/1974318067 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:153 7.9 7:93e5b521:::notify.7:head [notify-ack in=20b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32) v8 -- 0x7f5458003d90 con 0x7f545c00d8e0 2023-10-05T19:28:56.985 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.978+0000 7f5473e27700 1 -- 172.21.15.167:0/1974318067 <== osd.0 v2:172.21.15.118:6800/27690 42 ==== osd_op_reply(153 notify.7 [notify-ack] v0'0 uv207 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f5464050040 con 0x7f545c00d8e0 2023-10-05T19:28:56.985 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.978+0000 7f5473e27700 1 -- 172.21.15.167:0/1974318067 <== osd.0 v2:172.21.15.118:6800/27690 43 ==== watch-notify(notify_complete (2) cookie 94916677482048 notify 137438953497 ret 0) v3 ==== 42+0+48 (crc 0 0 0) 0x7f546403d040 con 0x7f545c00d8e0 2023-10-05T19:28:56.985 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.978+0000 7f5475af1e40 20 lifecycle: handle_bi_update(): bucket=:myfoo[9a9df965-6612-45d4-a45f-6e3c889bacc2.4223.5]): orig_sources= new_sources= 2023-10-05T19:28:56.985 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.978+0000 7f5475af1e40 20 lifecycle: handle_bi_update(): bucket=:myfoo[9a9df965-6612-45d4-a45f-6e3c889bacc2.4223.5]): potential sources added=[] removed=[] 2023-10-05T19:28:56.986 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.978+0000 7f5475af1e40 20 lifecycle: handle_bi_update(): bucket=:myfoo[9a9df965-6612-45d4-a45f-6e3c889bacc2.4223.5]): orig_dests= new_dests= 2023-10-05T19:28:56.986 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.978+0000 7f5475af1e40 20 lifecycle: handle_bi_update(): bucket=:myfoo[9a9df965-6612-45d4-a45f-6e3c889bacc2.4223.5]): potential dests added=[] removed=[] 2023-10-05T19:28:56.986 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.978+0000 7f5475af1e40 1 -- 172.21.15.167:0/4231517449 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:33 6.17 6:ee62d5d2:lc::lc.1:head [call lock.lock in=65b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e32) v8 -- 0x565382d7c8f0 con 0x565382d0afe0 2023-10-05T19:28:56.986 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.982+0000 7f5473626700 1 -- 172.21.15.167:0/4231517449 <== osd.1 v2:172.21.15.167:6800/30035 7 ==== osd_op_reply(33 lc.1 [call] v32'10 uv10 ondisk = 0) v8 ==== 148+0+0 (crc 0 0 0) 0x7f546c034070 con 0x565382d0afe0 2023-10-05T19:28:56.986 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.982+0000 7f5475af1e40 1 -- 172.21.15.167:0/4231517449 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:34 6.17 6:ee62d5d2:lc::lc.1:head [call rgw.lc_rm_entry in=92b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32) v8 -- 0x565382d7c8f0 con 0x565382d0afe0 2023-10-05T19:28:56.988 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.982+0000 7f5473626700 1 -- 172.21.15.167:0/4231517449 <== osd.1 v2:172.21.15.167:6800/30035 8 ==== osd_op_reply(34 lc.1 [call] v32'11 uv11 ondisk = 0) v8 ==== 148+0+0 (crc 0 0 0) 0x7f546c034070 con 0x565382d0afe0 2023-10-05T19:28:56.988 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.982+0000 7f5475af1e40 1 -- 172.21.15.167:0/4231517449 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:35 6.17 6:ee62d5d2:lc::lc.1:head [call lock.unlock in=49b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e32) v8 -- 0x565382d7c8f0 con 0x565382d0afe0 2023-10-05T19:28:56.992 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.986+0000 7f5473626700 1 -- 172.21.15.167:0/4231517449 <== osd.1 v2:172.21.15.167:6800/30035 9 ==== osd_op_reply(35 lc.1 [call] v32'12 uv12 ondisk = 0) v8 ==== 148+0+0 (crc 0 0 0) 0x7f546c034070 con 0x565382d0afe0 2023-10-05T19:28:56.992 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.986+0000 7f5475af1e40 1 -- 172.21.15.167:0/1974318067 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:154 4.2d 4:b5ae0298:::.dir.9a9df965-6612-45d4-a45f-6e3c889bacc2.4223.5.0:head [call rgw.bucket_list in=47b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32) v8 -- 0x565382d7c8f0 con 0x7f545c00d8e0 2023-10-05T19:28:56.992 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.986+0000 7f5475af1e40 1 -- 172.21.15.167:0/1974318067 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:155 4.34 4:2ef4a173:::.dir.9a9df965-6612-45d4-a45f-6e3c889bacc2.4223.5.1:head [call rgw.bucket_list in=47b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32) v8 -- 0x565382d936e0 con 0x7f546c01b740 2023-10-05T19:28:56.992 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.986+0000 7f5475af1e40 1 -- 172.21.15.167:0/1974318067 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:156 4.2 4:42981e29:::.dir.9a9df965-6612-45d4-a45f-6e3c889bacc2.4223.5.2:head [call rgw.bucket_list in=47b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32) v8 -- 0x565382d946a0 con 0x565382cac750 2023-10-05T19:28:56.993 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.986+0000 7f5475af1e40 1 -- 172.21.15.167:0/1974318067 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:157 4.3d 4:bdb71a6b:::.dir.9a9df965-6612-45d4-a45f-6e3c889bacc2.4223.5.3:head [call rgw.bucket_list in=47b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32) v8 -- 0x565382d7c8f0 con 0x565382ce1410 2023-10-05T19:28:56.993 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.986+0000 7f5475af1e40 1 -- 172.21.15.167:0/1974318067 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:158 4.33 4:cc3cd978:::.dir.9a9df965-6612-45d4-a45f-6e3c889bacc2.4223.5.4:head [call rgw.bucket_list in=47b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32) v8 -- 0x565382d936e0 con 0x7f545c00d8e0 2023-10-05T19:28:56.993 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.986+0000 7f5475af1e40 1 -- 172.21.15.167:0/1974318067 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:159 4.16 4:6b936802:::.dir.9a9df965-6612-45d4-a45f-6e3c889bacc2.4223.5.5:head [call rgw.bucket_list in=47b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32) v8 -- 0x565382d946a0 con 0x565382ce1410 2023-10-05T19:28:56.993 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.986+0000 7f5475af1e40 1 -- 172.21.15.167:0/1974318067 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:160 4.8 4:1342e9a7:::.dir.9a9df965-6612-45d4-a45f-6e3c889bacc2.4223.5.6:head [call rgw.bucket_list in=47b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32) v8 -- 0x565382d936e0 con 0x565382ce1410 2023-10-05T19:28:56.993 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.986+0000 7f5475af1e40 1 -- 172.21.15.167:0/1974318067 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:161 4.d 4:b00cdfe0:::.dir.9a9df965-6612-45d4-a45f-6e3c889bacc2.4223.5.7:head [call rgw.bucket_list in=47b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32) v8 -- 0x565382d7c8f0 con 0x565382ce1410 2023-10-05T19:28:56.993 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.986+0000 7f5475af1e40 1 -- 172.21.15.167:0/1974318067 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:162 4.21 4:84ddff17:::.dir.9a9df965-6612-45d4-a45f-6e3c889bacc2.4223.5.8:head [call rgw.bucket_list in=47b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32) v8 -- 0x565382d946a0 con 0x7f545c00d8e0 2023-10-05T19:28:56.993 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.986+0000 7f5475af1e40 1 -- 172.21.15.167:0/1974318067 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:163 4.4 4:20b95fb7:::.dir.9a9df965-6612-45d4-a45f-6e3c889bacc2.4223.5.9:head [call rgw.bucket_list in=47b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32) v8 -- 0x565382d7c8f0 con 0x7f545c00d8e0 2023-10-05T19:28:56.993 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.986+0000 7f5475af1e40 1 -- 172.21.15.167:0/1974318067 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:164 4.25 4:a63b86a5:::.dir.9a9df965-6612-45d4-a45f-6e3c889bacc2.4223.5.10:head [call rgw.bucket_list in=47b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32) v8 -- 0x565382d936e0 con 0x565382ce1410 2023-10-05T19:28:56.993 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.986+0000 7f5473e27700 1 -- 172.21.15.167:0/1974318067 <== osd.1 v2:172.21.15.167:6800/30035 42 ==== osd_op_reply(156 .dir.9a9df965-6612-45d4-a45f-6e3c889bacc2.4223.5.2 [call out=124b] v0'0 uv9 ondisk = 0) v8 ==== 194+0+124 (crc 0 0 0) 0x7f5464050040 con 0x565382cac750 2023-10-05T19:28:56.993 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.986+0000 7f5472e25700 1 -- 172.21.15.167:0/1974318067 <== osd.2 v2:172.21.15.167:6816/30037 27 ==== osd_op_reply(155 .dir.9a9df965-6612-45d4-a45f-6e3c889bacc2.4223.5.1 [call out=85b] v0'0 uv1 ondisk = 0) v8 ==== 194+0+85 (crc 0 0 0) 0x7f545c006070 con 0x7f546c01b740 2023-10-05T19:28:56.994 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.986+0000 7f5473e27700 1 -- 172.21.15.167:0/1974318067 <== osd.0 v2:172.21.15.118:6800/27690 44 ==== osd_op_reply(154 .dir.9a9df965-6612-45d4-a45f-6e3c889bacc2.4223.5.0 [call out=85b] v0'0 uv5 ondisk = 0) v8 ==== 194+0+85 (crc 0 0 0) 0x7f5464050040 con 0x7f545c00d8e0 2023-10-05T19:28:56.994 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.986+0000 7f5473e27700 1 -- 172.21.15.167:0/1974318067 <== osd.0 v2:172.21.15.118:6800/27690 45 ==== osd_op_reply(158 .dir.9a9df965-6612-45d4-a45f-6e3c889bacc2.4223.5.4 [call out=85b] v0'0 uv3 ondisk = 0) v8 ==== 194+0+85 (crc 0 0 0) 0x7f5464050040 con 0x7f545c00d8e0 2023-10-05T19:28:56.994 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.986+0000 7f5473e27700 1 -- 172.21.15.167:0/1974318067 <== osd.0 v2:172.21.15.118:6800/27690 46 ==== osd_op_reply(163 .dir.9a9df965-6612-45d4-a45f-6e3c889bacc2.4223.5.9 [call out=85b] v0'0 uv3 ondisk = 0) v8 ==== 194+0+85 (crc 0 0 0) 0x7f5464050040 con 0x7f545c00d8e0 2023-10-05T19:28:56.994 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.986+0000 7f5473e27700 1 -- 172.21.15.167:0/1974318067 <== osd.0 v2:172.21.15.118:6800/27690 47 ==== osd_op_reply(162 .dir.9a9df965-6612-45d4-a45f-6e3c889bacc2.4223.5.8 [call out=124b] v0'0 uv7 ondisk = 0) v8 ==== 194+0+124 (crc 0 0 0) 0x7f5464050040 con 0x7f545c00d8e0 2023-10-05T19:28:56.994 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.986+0000 7f5473626700 1 -- 172.21.15.167:0/1974318067 <== osd.3 v2:172.21.15.167:6808/30036 46 ==== osd_op_reply(157 .dir.9a9df965-6612-45d4-a45f-6e3c889bacc2.4223.5.3 [call out=85b] v0'0 uv5 ondisk = 0) v8 ==== 194+0+85 (crc 0 0 0) 0x7f546c034070 con 0x565382ce1410 2023-10-05T19:28:56.994 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.986+0000 7f5473626700 1 -- 172.21.15.167:0/1974318067 <== osd.3 v2:172.21.15.167:6808/30036 47 ==== osd_op_reply(159 .dir.9a9df965-6612-45d4-a45f-6e3c889bacc2.4223.5.5 [call out=85b] v0'0 uv3 ondisk = 0) v8 ==== 194+0+85 (crc 0 0 0) 0x7f546c034070 con 0x565382ce1410 2023-10-05T19:28:56.994 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.986+0000 7f5473626700 1 -- 172.21.15.167:0/1974318067 <== osd.3 v2:172.21.15.167:6808/30036 48 ==== osd_op_reply(160 .dir.9a9df965-6612-45d4-a45f-6e3c889bacc2.4223.5.6 [call out=124b] v0'0 uv5 ondisk = 0) v8 ==== 194+0+124 (crc 0 0 0) 0x7f546c034070 con 0x565382ce1410 2023-10-05T19:28:56.994 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.986+0000 7f5473626700 1 -- 172.21.15.167:0/1974318067 <== osd.3 v2:172.21.15.167:6808/30036 49 ==== osd_op_reply(161 .dir.9a9df965-6612-45d4-a45f-6e3c889bacc2.4223.5.7 [call out=85b] v0'0 uv1 ondisk = 0) v8 ==== 194+0+85 (crc 0 0 0) 0x7f546c034070 con 0x565382ce1410 2023-10-05T19:28:56.994 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.986+0000 7f5473626700 1 -- 172.21.15.167:0/1974318067 <== osd.3 v2:172.21.15.167:6808/30036 50 ==== osd_op_reply(164 .dir.9a9df965-6612-45d4-a45f-6e3c889bacc2.4223.5.10 [call out=124b] v0'0 uv13 ondisk = 0) v8 ==== 195+0+124 (crc 0 0 0) 0x7f546c034070 con 0x565382ce1410 2023-10-05T19:28:56.994 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.986+0000 7f5475af1e40 1 -- 172.21.15.167:0/1974318067 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:165 8.9 8:96d98480:users.uid::foo.buckets:head [call user.set_buckets_info in=207b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e32) v8 -- 0x565382d936e0 con 0x7f546c01b740 2023-10-05T19:28:56.995 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.990+0000 7f5472e25700 1 -- 172.21.15.167:0/1974318067 <== osd.2 v2:172.21.15.167:6816/30037 28 ==== osd_op_reply(165 foo.buckets [call] v32'38 uv38 ondisk = 0) v8 ==== 155+0+0 (crc 0 0 0) 0x7f545c006070 con 0x7f546c01b740 2023-10-05T19:28:56.996 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.990+0000 7f5475af1e40 20 get_system_obj_state: rctx=0x7fff8b74b6b8 obj=default.rgw.meta:root:myfoo state=0x565382b5b2b0 s->prefetch_data=0 2023-10-05T19:28:56.996 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.990+0000 7f5475af1e40 10 cache get: name=default.rgw.meta+root+myfoo : hit (requested=0x16, cached=0x17) 2023-10-05T19:28:56.996 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.990+0000 7f5475af1e40 20 get_system_obj_state: s->obj_tag was set empty 2023-10-05T19:28:56.996 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.990+0000 7f5475af1e40 10 cache get: name=default.rgw.meta+root+myfoo : hit (requested=0x11, cached=0x17) 2023-10-05T19:28:56.996 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.990+0000 7f5475af1e40 10 removing default.rgw.meta+root+myfoo from cache 2023-10-05T19:28:56.996 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.990+0000 7f5475af1e40 10 distributing notification oid=default.rgw.control:notify.6 cni=[op: 1, obj: default.rgw.meta:root:myfoo, ofs0, ns] 2023-10-05T19:28:56.996 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.990+0000 7f5475af1e40 1 -- 172.21.15.167:0/1974318067 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:166 7.14 7:2b04a3e9:::notify.6:head [notify cookie 94916677467472 in=161b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32) v8 -- 0x565382d936e0 con 0x7f545c00d8e0 2023-10-05T19:28:56.996 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.990+0000 7f5473e27700 1 -- 172.21.15.167:0/1974318067 <== osd.0 v2:172.21.15.118:6800/27690 48 ==== watch-notify(notify (1) cookie 94916676861168 notify 137438953498 ret 0) v3 ==== 191+0+0 (crc 0 0 0) 0x7f546403d040 con 0x7f545c00d8e0 2023-10-05T19:28:56.996 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.990+0000 7f5471622700 10 rgw watcher librados: RGWWatcher::handle_notify() notify_id 137438953498 cookie 94916676861168 notifier 4641 bl.length()=149 2023-10-05T19:28:56.996 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.990+0000 7f5473e27700 1 -- 172.21.15.167:0/1974318067 <== osd.0 v2:172.21.15.118:6800/27690 49 ==== osd_op_reply(166 notify.6 [notify cookie 94916677467472 out=8b] v0'0 uv207 ondisk = 0) v8 ==== 152+0+8 (crc 0 0 0) 0x7f5464050040 con 0x7f545c00d8e0 2023-10-05T19:28:56.997 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.990+0000 7f5471622700 1 -- 172.21.15.167:0/1974318067 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:167 7.14 7:2b04a3e9:::notify.6:head [notify-ack in=20b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32) v8 -- 0x7f5458003d90 con 0x7f545c00d8e0 2023-10-05T19:28:56.997 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.990+0000 7f5473e27700 1 -- 172.21.15.167:0/1974318067 <== osd.0 v2:172.21.15.118:6800/27690 50 ==== osd_op_reply(167 notify.6 [notify-ack] v0'0 uv207 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f5464050040 con 0x7f545c00d8e0 2023-10-05T19:28:56.997 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.990+0000 7f5473e27700 1 -- 172.21.15.167:0/1974318067 <== osd.0 v2:172.21.15.118:6800/27690 51 ==== watch-notify(notify_complete (2) cookie 94916677467472 notify 137438953498 ret 0) v3 ==== 42+0+48 (crc 0 0 0) 0x7f54640731e0 con 0x7f545c00d8e0 2023-10-05T19:28:56.997 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.990+0000 7f5475af1e40 1 -- 172.21.15.167:0/1974318067 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:168 8.9 8:91b71841:root::myfoo:head [call version.check_conds in=122b,call version.set in=58b,delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e32) v8 -- 0x565382d936e0 con 0x7f546c01b740 2023-10-05T19:28:56.999 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.994+0000 7f5472e25700 1 -- 172.21.15.167:0/1974318067 <== osd.2 v2:172.21.15.167:6816/30037 29 ==== osd_op_reply(168 myfoo [call,call,delete] v32'39 uv39 ondisk = 0) v8 ==== 233+0+0 (crc 0 0 0) 0x7f545c006070 con 0x7f546c01b740 2023-10-05T19:28:56.999 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.994+0000 7f5475af1e40 10 removing default.rgw.meta+root+.bucket.meta.myfoo:9a9df965-6612-45d4-a45f-6e3c889bacc2.4223.5 from cache 2023-10-05T19:28:56.999 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.994+0000 7f5475af1e40 10 distributing notification oid=default.rgw.control:notify.7 cni=[op: 1, obj: default.rgw.meta:root:.bucket.meta.myfoo:9a9df965-6612-45d4-a45f-6e3c889bacc2.4223.5, ofs0, ns] 2023-10-05T19:28:56.999 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.994+0000 7f5475af1e40 1 -- 172.21.15.167:0/1974318067 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:169 7.9 7:93e5b521:::notify.7:head [notify cookie 94916677467472 in=218b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32) v8 -- 0x565382d936e0 con 0x7f545c00d8e0 2023-10-05T19:28:57.000 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.994+0000 7f5473e27700 1 -- 172.21.15.167:0/1974318067 <== osd.0 v2:172.21.15.118:6800/27690 52 ==== watch-notify(notify (1) cookie 94916676862272 notify 137438953499 ret 0) v3 ==== 248+0+0 (crc 0 0 0) 0x7f54640731e0 con 0x7f545c00d8e0 2023-10-05T19:28:57.000 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.994+0000 7f5471622700 10 rgw watcher librados: RGWWatcher::handle_notify() notify_id 137438953499 cookie 94916676862272 notifier 4641 bl.length()=206 2023-10-05T19:28:57.000 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.994+0000 7f5473e27700 1 -- 172.21.15.167:0/1974318067 <== osd.0 v2:172.21.15.118:6800/27690 53 ==== osd_op_reply(169 notify.7 [notify cookie 94916677467472 out=8b] v0'0 uv207 ondisk = 0) v8 ==== 152+0+8 (crc 0 0 0) 0x7f5464050040 con 0x7f545c00d8e0 2023-10-05T19:28:57.000 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.994+0000 7f5471622700 1 -- 172.21.15.167:0/1974318067 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:170 7.9 7:93e5b521:::notify.7:head [notify-ack in=20b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32) v8 -- 0x7f5458003d90 con 0x7f545c00d8e0 2023-10-05T19:28:57.001 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.994+0000 7f5473e27700 1 -- 172.21.15.167:0/1974318067 <== osd.0 v2:172.21.15.118:6800/27690 54 ==== osd_op_reply(170 notify.7 [notify-ack] v0'0 uv207 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f5464050040 con 0x7f545c00d8e0 2023-10-05T19:28:57.001 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.994+0000 7f5473e27700 1 -- 172.21.15.167:0/1974318067 <== osd.0 v2:172.21.15.118:6800/27690 55 ==== watch-notify(notify_complete (2) cookie 94916677467472 notify 137438953499 ret 0) v3 ==== 42+0+48 (crc 0 0 0) 0x7f54640338f0 con 0x7f545c00d8e0 2023-10-05T19:28:57.001 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.994+0000 7f5475af1e40 1 -- 172.21.15.167:0/1974318067 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:171 8.9 8:96cb4b0f:root::.bucket.meta.myfoo%3a9a9df965-6612-45d4-a45f-6e3c889bacc2.4223.5:head [call version.check_conds in=122b,call version.inc in=38b,delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e32) v8 -- 0x565382d936e0 con 0x7f546c01b740 2023-10-05T19:28:57.004 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.998+0000 7f5472e25700 1 -- 172.21.15.167:0/1974318067 <== osd.2 v2:172.21.15.167:6816/30037 30 ==== osd_op_reply(171 .bucket.meta.myfoo:9a9df965-6612-45d4-a45f-6e3c889bacc2.4223.5 [call,call,delete] v32'40 uv40 ondisk = 0) v8 ==== 290+0+0 (crc 0 0 0) 0x7f545c006070 con 0x7f546c01b740 2023-10-05T19:28:57.004 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.998+0000 7f5475af1e40 1 -- 172.21.15.167:0/1974318067 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:172 4.2d 4:b5ae0298:::.dir.9a9df965-6612-45d4-a45f-6e3c889bacc2.4223.5.0:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e32) v8 -- 0x565382d936e0 con 0x7f545c00d8e0 2023-10-05T19:28:57.004 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.998+0000 7f5475af1e40 1 -- 172.21.15.167:0/1974318067 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:173 4.34 4:2ef4a173:::.dir.9a9df965-6612-45d4-a45f-6e3c889bacc2.4223.5.1:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e32) v8 -- 0x565382d7c8f0 con 0x7f546c01b740 2023-10-05T19:28:57.004 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.998+0000 7f5475af1e40 1 -- 172.21.15.167:0/1974318067 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:174 4.2 4:42981e29:::.dir.9a9df965-6612-45d4-a45f-6e3c889bacc2.4223.5.2:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e32) v8 -- 0x565382d941b0 con 0x565382cac750 2023-10-05T19:28:57.005 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.998+0000 7f5475af1e40 1 -- 172.21.15.167:0/1974318067 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:175 4.3d 4:bdb71a6b:::.dir.9a9df965-6612-45d4-a45f-6e3c889bacc2.4223.5.3:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e32) v8 -- 0x565382d936e0 con 0x565382ce1410 2023-10-05T19:28:57.005 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.998+0000 7f5475af1e40 1 -- 172.21.15.167:0/1974318067 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:176 4.33 4:cc3cd978:::.dir.9a9df965-6612-45d4-a45f-6e3c889bacc2.4223.5.4:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e32) v8 -- 0x565382d941b0 con 0x7f545c00d8e0 2023-10-05T19:28:57.005 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.998+0000 7f5475af1e40 1 -- 172.21.15.167:0/1974318067 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:177 4.16 4:6b936802:::.dir.9a9df965-6612-45d4-a45f-6e3c889bacc2.4223.5.5:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e32) v8 -- 0x565382d7c8f0 con 0x565382ce1410 2023-10-05T19:28:57.005 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.998+0000 7f5475af1e40 1 -- 172.21.15.167:0/1974318067 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:178 4.8 4:1342e9a7:::.dir.9a9df965-6612-45d4-a45f-6e3c889bacc2.4223.5.6:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e32) v8 -- 0x565382d941b0 con 0x565382ce1410 2023-10-05T19:28:57.005 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.998+0000 7f5475af1e40 1 -- 172.21.15.167:0/1974318067 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:179 4.d 4:b00cdfe0:::.dir.9a9df965-6612-45d4-a45f-6e3c889bacc2.4223.5.7:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e32) v8 -- 0x565382d936e0 con 0x565382ce1410 2023-10-05T19:28:57.005 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.998+0000 7f5475af1e40 1 -- 172.21.15.167:0/1974318067 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:180 4.21 4:84ddff17:::.dir.9a9df965-6612-45d4-a45f-6e3c889bacc2.4223.5.8:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e32) v8 -- 0x565382d7c8f0 con 0x7f545c00d8e0 2023-10-05T19:28:57.005 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.998+0000 7f5475af1e40 1 -- 172.21.15.167:0/1974318067 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:181 4.4 4:20b95fb7:::.dir.9a9df965-6612-45d4-a45f-6e3c889bacc2.4223.5.9:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e32) v8 -- 0x565382d941b0 con 0x7f545c00d8e0 2023-10-05T19:28:57.006 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:56.998+0000 7f5475af1e40 1 -- 172.21.15.167:0/1974318067 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:182 4.25 4:a63b86a5:::.dir.9a9df965-6612-45d4-a45f-6e3c889bacc2.4223.5.10:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e32) v8 -- 0x565382d936e0 con 0x565382ce1410 2023-10-05T19:28:57.007 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.002+0000 7f5473e27700 1 -- 172.21.15.167:0/1974318067 <== osd.0 v2:172.21.15.118:6800/27690 56 ==== osd_op_reply(176 .dir.9a9df965-6612-45d4-a45f-6e3c889bacc2.4223.5.4 [delete] v32'4 uv4 ondisk = 0) v8 ==== 194+0+0 (crc 0 0 0) 0x7f5464050040 con 0x7f545c00d8e0 2023-10-05T19:28:57.008 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.002+0000 7f5473e27700 1 -- 172.21.15.167:0/1974318067 <== osd.0 v2:172.21.15.118:6800/27690 57 ==== osd_op_reply(181 .dir.9a9df965-6612-45d4-a45f-6e3c889bacc2.4223.5.9 [delete] v32'4 uv4 ondisk = 0) v8 ==== 194+0+0 (crc 0 0 0) 0x7f5464050040 con 0x7f545c00d8e0 2023-10-05T19:28:57.008 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.002+0000 7f5473626700 1 -- 172.21.15.167:0/1974318067 <== osd.3 v2:172.21.15.167:6808/30036 51 ==== osd_op_reply(177 .dir.9a9df965-6612-45d4-a45f-6e3c889bacc2.4223.5.5 [delete] v32'4 uv4 ondisk = 0) v8 ==== 194+0+0 (crc 0 0 0) 0x7f546c034070 con 0x565382ce1410 2023-10-05T19:28:57.009 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.002+0000 7f5473e27700 1 -- 172.21.15.167:0/1974318067 <== osd.0 v2:172.21.15.118:6800/27690 58 ==== osd_op_reply(172 .dir.9a9df965-6612-45d4-a45f-6e3c889bacc2.4223.5.0 [delete] v32'6 uv6 ondisk = 0) v8 ==== 194+0+0 (crc 0 0 0) 0x7f5464050040 con 0x7f545c00d8e0 2023-10-05T19:28:57.009 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.002+0000 7f5472e25700 1 -- 172.21.15.167:0/1974318067 <== osd.2 v2:172.21.15.167:6816/30037 31 ==== osd_op_reply(173 .dir.9a9df965-6612-45d4-a45f-6e3c889bacc2.4223.5.1 [delete] v32'2 uv2 ondisk = 0) v8 ==== 194+0+0 (crc 0 0 0) 0x7f545c006070 con 0x7f546c01b740 2023-10-05T19:28:57.009 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.002+0000 7f5473e27700 1 -- 172.21.15.167:0/1974318067 <== osd.1 v2:172.21.15.167:6800/30035 43 ==== osd_op_reply(174 .dir.9a9df965-6612-45d4-a45f-6e3c889bacc2.4223.5.2 [delete] v32'10 uv10 ondisk = 0) v8 ==== 194+0+0 (crc 0 0 0) 0x7f5464050040 con 0x565382cac750 2023-10-05T19:28:57.009 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.002+0000 7f5473e27700 1 -- 172.21.15.167:0/1974318067 <== osd.0 v2:172.21.15.118:6800/27690 59 ==== osd_op_reply(180 .dir.9a9df965-6612-45d4-a45f-6e3c889bacc2.4223.5.8 [delete] v32'8 uv8 ondisk = 0) v8 ==== 194+0+0 (crc 0 0 0) 0x7f5464050040 con 0x7f545c00d8e0 2023-10-05T19:28:57.009 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.002+0000 7f5473626700 1 -- 172.21.15.167:0/1974318067 <== osd.3 v2:172.21.15.167:6808/30036 52 ==== osd_op_reply(175 .dir.9a9df965-6612-45d4-a45f-6e3c889bacc2.4223.5.3 [delete] v32'6 uv6 ondisk = 0) v8 ==== 194+0+0 (crc 0 0 0) 0x7f546c034070 con 0x565382ce1410 2023-10-05T19:28:57.009 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.002+0000 7f5473626700 1 -- 172.21.15.167:0/1974318067 <== osd.3 v2:172.21.15.167:6808/30036 53 ==== osd_op_reply(182 .dir.9a9df965-6612-45d4-a45f-6e3c889bacc2.4223.5.10 [delete] v32'14 uv14 ondisk = 0) v8 ==== 195+0+0 (crc 0 0 0) 0x7f546c034070 con 0x565382ce1410 2023-10-05T19:28:57.010 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.006+0000 7f5473626700 1 -- 172.21.15.167:0/1974318067 <== osd.3 v2:172.21.15.167:6808/30036 54 ==== osd_op_reply(178 .dir.9a9df965-6612-45d4-a45f-6e3c889bacc2.4223.5.6 [delete] v32'6 uv6 ondisk = 0) v8 ==== 194+0+0 (crc 0 0 0) 0x7f546c034070 con 0x565382ce1410 2023-10-05T19:28:57.010 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.006+0000 7f5473626700 1 -- 172.21.15.167:0/1974318067 <== osd.3 v2:172.21.15.167:6808/30036 55 ==== osd_op_reply(179 .dir.9a9df965-6612-45d4-a45f-6e3c889bacc2.4223.5.7 [delete] v32'2 uv2 ondisk = 0) v8 ==== 194+0+0 (crc 0 0 0) 0x7f546c034070 con 0x565382ce1410 2023-10-05T19:28:57.010 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.006+0000 7f5475af1e40 1 -- 172.21.15.167:0/1974318067 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:183 6.10 6:0ff42df5:::pubsub..bucket.myfoo%2f9a9df965-6612-45d4-a45f-6e3c889bacc2.4223.5:head [getxattrs,stat] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32) v8 -- 0x565382d936e0 con 0x7f545c00d8e0 2023-10-05T19:28:57.011 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.006+0000 7f5473e27700 1 -- 172.21.15.167:0/1974318067 <== osd.0 v2:172.21.15.118:6800/27690 60 ==== osd_op_reply(183 pubsub..bucket.myfoo/9a9df965-6612-45d4-a45f-6e3c889bacc2.4223.5 [getxattrs,stat] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 ==== 250+0+0 (crc 0 0 0) 0x7f5464050040 con 0x7f545c00d8e0 2023-10-05T19:28:57.011 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.006+0000 7f5475af1e40 10 removing default.rgw.log++pubsub..bucket.myfoo/9a9df965-6612-45d4-a45f-6e3c889bacc2.4223.5 from cache 2023-10-05T19:28:57.011 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.006+0000 7f5475af1e40 10 distributing notification oid=default.rgw.control:notify.6 cni=[op: 1, obj: default.rgw.log:pubsub..bucket.myfoo/9a9df965-6612-45d4-a45f-6e3c889bacc2.4223.5, ofs0, ns] 2023-10-05T19:28:57.011 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.006+0000 7f5475af1e40 1 -- 172.21.15.167:0/1974318067 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:184 7.14 7:2b04a3e9:::notify.6:head [notify cookie 94916677467472 in=215b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32) v8 -- 0x565382d936e0 con 0x7f545c00d8e0 2023-10-05T19:28:57.012 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.006+0000 7f5473e27700 1 -- 172.21.15.167:0/1974318067 <== osd.0 v2:172.21.15.118:6800/27690 61 ==== watch-notify(notify (1) cookie 94916676861168 notify 137438953500 ret 0) v3 ==== 245+0+0 (crc 0 0 0) 0x7f54640338f0 con 0x7f545c00d8e0 2023-10-05T19:28:57.012 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.006+0000 7f5470e21700 10 rgw watcher librados: RGWWatcher::handle_notify() notify_id 137438953500 cookie 94916676861168 notifier 4641 bl.length()=203 2023-10-05T19:28:57.012 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.006+0000 7f5473e27700 1 -- 172.21.15.167:0/1974318067 <== osd.0 v2:172.21.15.118:6800/27690 62 ==== osd_op_reply(184 notify.6 [notify cookie 94916677467472 out=8b] v0'0 uv207 ondisk = 0) v8 ==== 152+0+8 (crc 0 0 0) 0x7f5464050040 con 0x7f545c00d8e0 2023-10-05T19:28:57.012 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.006+0000 7f5470e21700 1 -- 172.21.15.167:0/1974318067 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:185 7.14 7:2b04a3e9:::notify.6:head [notify-ack in=20b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32) v8 -- 0x7f53c4003250 con 0x7f545c00d8e0 2023-10-05T19:28:57.012 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.006+0000 7f5473e27700 1 -- 172.21.15.167:0/1974318067 <== osd.0 v2:172.21.15.118:6800/27690 63 ==== osd_op_reply(185 notify.6 [notify-ack] v0'0 uv207 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f5464050040 con 0x7f545c00d8e0 2023-10-05T19:28:57.012 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.006+0000 7f5473e27700 1 -- 172.21.15.167:0/1974318067 <== osd.0 v2:172.21.15.118:6800/27690 64 ==== watch-notify(notify_complete (2) cookie 94916677467472 notify 137438953500 ret 0) v3 ==== 42+0+48 (crc 0 0 0) 0x7f5464033c50 con 0x7f545c00d8e0 2023-10-05T19:28:57.012 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.006+0000 7f5475af1e40 1 -- 172.21.15.167:0/1974318067 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:186 6.10 6:0ff42df5:::pubsub..bucket.myfoo%2f9a9df965-6612-45d4-a45f-6e3c889bacc2.4223.5:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e32) v8 -- 0x565382d936e0 con 0x7f545c00d8e0 2023-10-05T19:28:57.015 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.010+0000 7f5473e27700 1 -- 172.21.15.167:0/1974318067 <== osd.0 v2:172.21.15.118:6800/27690 65 ==== osd_op_reply(186 pubsub..bucket.myfoo/9a9df965-6612-45d4-a45f-6e3c889bacc2.4223.5 [delete] v32'6 uv5 ondisk = -2 ((2) No such file or directory)) v8 ==== 208+0+0 (crc 0 0 0) 0x7f5464050040 con 0x7f545c00d8e0 2023-10-05T19:28:57.015 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.010+0000 7f5475af1e40 1 -- 172.21.15.167:0/1974318067 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:187 8.9 8:96d98480:users.uid::foo.buckets:head [call user.remove_bucket in=58b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e32) v8 -- 0x565382d936e0 con 0x7f546c01b740 2023-10-05T19:28:57.017 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.010+0000 7f5472e25700 1 -- 172.21.15.167:0/1974318067 <== osd.2 v2:172.21.15.167:6816/30037 32 ==== osd_op_reply(187 foo.buckets [call] v32'41 uv41 ondisk = 0) v8 ==== 155+0+0 (crc 0 0 0) 0x7f545c006070 con 0x7f546c01b740 2023-10-05T19:28:57.017 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.014+0000 7f5475af1e40 1 -- 172.21.15.167:0/1974318067 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:188 7.1f 7:f95f44c2:::notify.0:head [watch unwatch cookie 94916676839376] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e32) v8 -- 0x565382d936e0 con 0x7f546c01b740 2023-10-05T19:28:57.019 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.014+0000 7f5472e25700 1 -- 172.21.15.167:0/1974318067 <== osd.2 v2:172.21.15.167:6816/30037 33 ==== osd_op_reply(188 notify.0 [watch unwatch cookie 94916676839376] v32'209 uv207 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f545c006070 con 0x7f546c01b740 2023-10-05T19:28:57.019 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.014+0000 7f5475af1e40 20 remove_watcher() i=0 2023-10-05T19:28:57.019 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.014+0000 7f5475af1e40 2 removed watcher, disabling cache 2023-10-05T19:28:57.020 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.014+0000 7f5475af1e40 1 -- 172.21.15.167:0/1974318067 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:189 7.0 7:05bf5b68:::notify.1:head [watch unwatch cookie 94916676842976] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e32) v8 -- 0x565382d936e0 con 0x565382cac750 2023-10-05T19:28:57.022 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.018+0000 7f5473e27700 1 -- 172.21.15.167:0/1974318067 <== osd.1 v2:172.21.15.167:6800/30035 44 ==== osd_op_reply(189 notify.1 [watch unwatch cookie 94916676842976] v32'209 uv207 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f5464050040 con 0x565382cac750 2023-10-05T19:28:57.022 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.018+0000 7f5475af1e40 20 remove_watcher() i=1 2023-10-05T19:28:57.022 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.018+0000 7f5475af1e40 1 -- 172.21.15.167:0/1974318067 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:190 7.15 7:a93a5511:::notify.2:head [watch unwatch cookie 94916676844080] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e32) v8 -- 0x565382d936e0 con 0x565382ce1410 2023-10-05T19:28:57.024 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.018+0000 7f5473626700 1 -- 172.21.15.167:0/1974318067 <== osd.3 v2:172.21.15.167:6808/30036 56 ==== osd_op_reply(190 notify.2 [watch unwatch cookie 94916676844080] v32'209 uv207 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f546c034070 con 0x565382ce1410 2023-10-05T19:28:57.024 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.018+0000 7f5475af1e40 20 remove_watcher() i=2 2023-10-05T19:28:57.024 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.018+0000 7f5475af1e40 1 -- 172.21.15.167:0/1974318067 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:191 7.e 7:7759931f:::notify.3:head [watch unwatch cookie 94916676846240] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e32) v8 -- 0x565382d936e0 con 0x7f546c01b740 2023-10-05T19:28:57.026 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.022+0000 7f5472e25700 1 -- 172.21.15.167:0/1974318067 <== osd.2 v2:172.21.15.167:6816/30037 34 ==== osd_op_reply(191 notify.3 [watch unwatch cookie 94916676846240] v32'209 uv207 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f545c006070 con 0x7f546c01b740 2023-10-05T19:28:57.026 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.022+0000 7f5475af1e40 20 remove_watcher() i=3 2023-10-05T19:28:57.026 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.022+0000 7f5475af1e40 1 -- 172.21.15.167:0/1974318067 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:192 7.d 7:b4812045:::notify.4:head [watch unwatch cookie 94916676854256] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e32) v8 -- 0x565382d936e0 con 0x565382cac750 2023-10-05T19:28:57.029 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.022+0000 7f5473e27700 1 -- 172.21.15.167:0/1974318067 <== osd.1 v2:172.21.15.167:6800/30035 45 ==== osd_op_reply(192 notify.4 [watch unwatch cookie 94916676854256] v32'209 uv207 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f5464050040 con 0x565382cac750 2023-10-05T19:28:57.029 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.022+0000 7f5475af1e40 20 remove_watcher() i=4 2023-10-05T19:28:57.029 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.022+0000 7f5475af1e40 1 -- 172.21.15.167:0/1974318067 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:193 7.3 7:c609908c:::notify.5:head [watch unwatch cookie 94916676852352] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e32) v8 -- 0x565382d936e0 con 0x7f545c00d8e0 2023-10-05T19:28:57.031 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.026+0000 7f5473e27700 1 -- 172.21.15.167:0/1974318067 <== osd.0 v2:172.21.15.118:6800/27690 66 ==== osd_op_reply(193 notify.5 [watch unwatch cookie 94916676852352] v32'209 uv207 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f5464050040 con 0x7f545c00d8e0 2023-10-05T19:28:57.032 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.026+0000 7f5475af1e40 20 remove_watcher() i=5 2023-10-05T19:28:57.032 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.026+0000 7f5475af1e40 1 -- 172.21.15.167:0/1974318067 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:194 7.14 7:2b04a3e9:::notify.6:head [watch unwatch cookie 94916676861168] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e32) v8 -- 0x565382d936e0 con 0x7f545c00d8e0 2023-10-05T19:28:57.034 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.030+0000 7f5473e27700 1 -- 172.21.15.167:0/1974318067 <== osd.0 v2:172.21.15.118:6800/27690 67 ==== osd_op_reply(194 notify.6 [watch unwatch cookie 94916676861168] v32'209 uv207 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f5464050040 con 0x7f545c00d8e0 2023-10-05T19:28:57.034 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.030+0000 7f5475af1e40 20 remove_watcher() i=6 2023-10-05T19:28:57.034 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.030+0000 7f5475af1e40 1 -- 172.21.15.167:0/1974318067 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:195 7.9 7:93e5b521:::notify.7:head [watch unwatch cookie 94916676862272] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e32) v8 -- 0x565382d936e0 con 0x7f545c00d8e0 2023-10-05T19:28:57.037 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.030+0000 7f5473e27700 1 -- 172.21.15.167:0/1974318067 <== osd.0 v2:172.21.15.118:6800/27690 68 ==== osd_op_reply(195 notify.7 [watch unwatch cookie 94916676862272] v32'209 uv207 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f5464050040 con 0x7f545c00d8e0 2023-10-05T19:28:57.037 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.030+0000 7f5475af1e40 20 remove_watcher() i=7 2023-10-05T19:28:57.038 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.034+0000 7f5444fe1700 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2023-10-05T19:28:57.038 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.034+0000 7f5475af1e40 1 -- 172.21.15.167:0/1974318067 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:196 6.4 6:22d26bf9:::data_loggenerations_metadata:head [watch unwatch cookie 94916676859280] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e32) v8 -- 0x565382d936e0 con 0x565382cac750 2023-10-05T19:28:57.043 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.034+0000 7f5473e27700 1 -- 172.21.15.167:0/1974318067 <== osd.1 v2:172.21.15.167:6800/30035 46 ==== osd_op_reply(196 data_loggenerations_metadata [watch unwatch cookie 94916676859280] v32'209 uv1 ondisk = 0) v8 ==== 172+0+0 (crc 0 0 0) 0x7f5464050040 con 0x565382cac750 2023-10-05T19:28:57.043 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.034+0000 7f5475af1e40 1 -- 172.21.15.167:0/1974318067 >> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] conn(0x7f545c00d8e0 msgr2=0x7f545c00c360 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-05T19:28:57.043 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.034+0000 7f5475af1e40 1 --2- 172.21.15.167:0/1974318067 >> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] conn(0x7f545c00d8e0 0x7f545c00c360 crc :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-05T19:28:57.044 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.034+0000 7f5475af1e40 1 -- 172.21.15.167:0/1974318067 >> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] conn(0x565382cac750 msgr2=0x565382cdd5e0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-05T19:28:57.044 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.034+0000 7f5475af1e40 1 --2- 172.21.15.167:0/1974318067 >> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] conn(0x565382cac750 0x565382cdd5e0 crc :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-05T19:28:57.044 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.034+0000 7f5475af1e40 1 -- 172.21.15.167:0/1974318067 >> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] conn(0x7f546c01b740 msgr2=0x7f546c01a5f0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-05T19:28:57.044 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.034+0000 7f5475af1e40 1 --2- 172.21.15.167:0/1974318067 >> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] conn(0x7f546c01b740 0x7f546c01a5f0 crc :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-05T19:28:57.044 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.034+0000 7f5475af1e40 1 -- 172.21.15.167:0/1974318067 >> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] conn(0x565382ce1410 msgr2=0x565382ce37f0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-05T19:28:57.044 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.034+0000 7f5475af1e40 1 --2- 172.21.15.167:0/1974318067 >> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] conn(0x565382ce1410 0x565382ce37f0 crc :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-05T19:28:57.044 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.038+0000 7f5475af1e40 1 -- 172.21.15.167:0/1974318067 >> [v2:172.21.15.167:6824/29889,v1:172.21.15.167:6825/29889] conn(0x7f5454030ca0 msgr2=0x7f5454033150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-05T19:28:57.044 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.038+0000 7f5475af1e40 1 --2- 172.21.15.167:0/1974318067 >> [v2:172.21.15.167:6824/29889,v1:172.21.15.167:6825/29889] conn(0x7f5454030ca0 0x7f5454033150 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7f545c001e90 tx=0x7f545c0013b0 comp rx=0 tx=0).stop 2023-10-05T19:28:57.044 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.038+0000 7f5475af1e40 1 -- 172.21.15.167:0/1974318067 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x565382cabe40 msgr2=0x565382cd1b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-05T19:28:57.044 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.038+0000 7f5475af1e40 1 --2- 172.21.15.167:0/1974318067 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x565382cabe40 0x565382cd1b70 secure :-1 s=READY pgs=551 cs=0 l=1 rev1=1 crypto rx=0x565382cc7550 tx=0x7f546c01d000 comp rx=0 tx=0).stop 2023-10-05T19:28:57.044 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.038+0000 7f5473e27700 1 -- 172.21.15.167:0/1974318067 reap_dead start 2023-10-05T19:28:57.044 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.038+0000 7f5475af1e40 1 -- 172.21.15.167:0/1974318067 shutdown_connections 2023-10-05T19:28:57.045 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.038+0000 7f5475af1e40 1 --2- 172.21.15.167:0/1974318067 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x565382cabe40 0x565382cd1b70 unknown :-1 s=CLOSED pgs=551 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-05T19:28:57.045 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.038+0000 7f5475af1e40 1 -- 172.21.15.167:0/1974318067 >> 172.21.15.167:0/1974318067 conn(0x565382ca7860 msgr2=0x565382cc54b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-05T19:28:57.045 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.038+0000 7f5475af1e40 1 -- 172.21.15.167:0/1974318067 shutdown_connections 2023-10-05T19:28:57.045 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.038+0000 7f5475af1e40 1 -- 172.21.15.167:0/1974318067 wait complete. 2023-10-05T19:28:57.045 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.038+0000 7f5475af1e40 1 -- 172.21.15.167:0/4231517449 >> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] conn(0x565382d0ea90 msgr2=0x565382d10e70 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-05T19:28:57.045 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.038+0000 7f5475af1e40 1 --2- 172.21.15.167:0/4231517449 >> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] conn(0x565382d0ea90 0x565382d10e70 crc :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-05T19:28:57.045 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.038+0000 7f5475af1e40 1 -- 172.21.15.167:0/4231517449 >> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] conn(0x565382d0afe0 msgr2=0x565382d0d3f0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-05T19:28:57.045 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.038+0000 7f5475af1e40 1 --2- 172.21.15.167:0/4231517449 >> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] conn(0x565382d0afe0 0x565382d0d3f0 crc :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-05T19:28:57.045 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.038+0000 7f5475af1e40 1 -- 172.21.15.167:0/4231517449 >> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] conn(0x565382d16450 msgr2=0x565382d18830 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-05T19:28:57.045 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.038+0000 7f5475af1e40 1 --2- 172.21.15.167:0/4231517449 >> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] conn(0x565382d16450 0x565382d18830 crc :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-05T19:28:57.045 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.038+0000 7f5475af1e40 1 -- 172.21.15.167:0/4231517449 >> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] conn(0x565382d12760 msgr2=0x565382d14b40 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-05T19:28:57.046 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.038+0000 7f5475af1e40 1 --2- 172.21.15.167:0/4231517449 >> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] conn(0x565382d12760 0x565382d14b40 crc :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-05T19:28:57.046 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.038+0000 7f5475af1e40 1 -- 172.21.15.167:0/4231517449 >> [v2:172.21.15.167:6824/29889,v1:172.21.15.167:6825/29889] conn(0x7f53b4030c50 msgr2=0x7f53b4033100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-05T19:28:57.046 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.038+0000 7f5475af1e40 1 --2- 172.21.15.167:0/4231517449 >> [v2:172.21.15.167:6824/29889,v1:172.21.15.167:6825/29889] conn(0x7f53b4030c50 0x7f53b4033100 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x565382cec920 tx=0x7f5464000f90 comp rx=0 tx=0).stop 2023-10-05T19:28:57.046 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.038+0000 7f5475af1e40 1 -- 172.21.15.167:0/4231517449 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x565382cfff90 msgr2=0x565382d00360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-05T19:28:57.046 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.038+0000 7f5475af1e40 1 --2- 172.21.15.167:0/4231517449 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x565382cfff90 0x565382d00360 secure :-1 s=READY pgs=553 cs=0 l=1 rev1=1 crypto rx=0x565382cdb070 tx=0x7f545c030000 comp rx=0 tx=0).stop 2023-10-05T19:28:57.046 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.038+0000 7f5473626700 1 -- 172.21.15.167:0/4231517449 reap_dead start 2023-10-05T19:28:57.046 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.038+0000 7f5473626700 1 -- 172.21.15.167:0/4231517449 reap_dead start 2023-10-05T19:28:57.046 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.038+0000 7f5475af1e40 1 -- 172.21.15.167:0/4231517449 shutdown_connections 2023-10-05T19:28:57.046 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.038+0000 7f5475af1e40 1 -- 172.21.15.167:0/4231517449 >> 172.21.15.167:0/4231517449 conn(0x565382cf5620 msgr2=0x565382cf36c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-05T19:28:57.046 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.038+0000 7f5475af1e40 1 -- 172.21.15.167:0/4231517449 shutdown_connections 2023-10-05T19:28:57.046 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.038+0000 7f5475af1e40 1 -- 172.21.15.167:0/4231517449 wait complete. 2023-10-05T19:28:57.049 INFO:tasks.util.rgw:rgwadmin: client.0 : ['caps', 'add', '--uid', 'foo', '--caps', 'user=read'] 2023-10-05T19:28:57.049 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'ceph', 'caps', 'add', '--uid', 'foo', '--caps', 'user=read'] 2023-10-05T19:28:57.049 DEBUG:teuthology.orchestra.run.smithi167:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster ceph caps add --uid foo --caps user=read 2023-10-05T19:28:57.133 INFO:teuthology.orchestra.run.smithi167.stderr:ignoring --setuser ceph since I am not root 2023-10-05T19:28:57.134 INFO:teuthology.orchestra.run.smithi167.stderr:ignoring --setgroup ceph since I am not root 2023-10-05T19:28:57.134 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.130+0000 7f583fe1ae40 1 Processor -- start 2023-10-05T19:28:57.134 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.130+0000 7f583fe1ae40 1 -- start start 2023-10-05T19:28:57.135 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.130+0000 7f583fe1ae40 1 --2- >> v2:172.21.15.118:3300/0 conn(0x563bbf74b020 0x563bbf74b3f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-05T19:28:57.135 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.130+0000 7f583fe1ae40 1 -- --> v1:172.21.15.118:6789/0 -- auth(proto 0 26 bytes epoch 0) v1 -- 0x563bbf695bd0 con 0x563bbf74b930 2023-10-05T19:28:57.135 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.130+0000 7f583fe1ae40 1 -- --> v2:172.21.15.118:3300/0 -- mon_getmap magic: 0 v1 -- 0x563bbf652ee0 con 0x563bbf74b020 2023-10-05T19:28:57.135 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.130+0000 7f583d94f700 1 --2- >> v2:172.21.15.118:3300/0 conn(0x563bbf74b020 0x563bbf74b3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-05T19:28:57.136 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.130+0000 7f583d14e700 1 --1- >> v1:172.21.15.118:6789/0 conn(0x563bbf74b930 0x563bbf74c3d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.118:6789/0 says I am v1:172.21.15.167:48712/0 (socket says 172.21.15.167:48712) 2023-10-05T19:28:57.136 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.130+0000 7f583d14e700 1 -- 172.21.15.167:0/1428577185 learned_addr learned my addr 172.21.15.167:0/1428577185 (peer_addr_for_me v1:172.21.15.167:0/0) 2023-10-05T19:28:57.137 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.130+0000 7f583d94f700 1 -- 172.21.15.167:0/1428577185 >> v1:172.21.15.118:6789/0 conn(0x563bbf74b930 legacy=0x563bbf74c3d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-05T19:28:57.137 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.130+0000 7f583c94d700 1 -- 172.21.15.167:0/1428577185 <== mon.0 v1:172.21.15.118:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 887863768 0 0) 0x563bbf695bd0 con 0x563bbf74b930 2023-10-05T19:28:57.137 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.130+0000 7f583d94f700 1 -- 172.21.15.167:0/1428577185 --> v2:172.21.15.118:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x563bbf74cae0 con 0x563bbf74b020 2023-10-05T19:28:57.137 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.130+0000 7f583d94f700 1 --2- 172.21.15.167:0/1428577185 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x563bbf74b020 0x563bbf74b3f0 secure :-1 s=READY pgs=555 cs=0 l=1 rev1=1 crypto rx=0x7f583000a5d0 tx=0x7f5830005a50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=720c747757348e46 server_cookie=0 in_seq=0 out_seq=0 2023-10-05T19:28:57.137 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.130+0000 7f583c94d700 1 -- 172.21.15.167:0/1428577185 <== mon.0 v2:172.21.15.118:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f58300086a0 con 0x563bbf74b020 2023-10-05T19:28:57.137 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.130+0000 7f583c94d700 1 -- 172.21.15.167:0/1428577185 <== mon.0 v2:172.21.15.118:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5830008800 con 0x563bbf74b020 2023-10-05T19:28:57.137 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.130+0000 7f583c94d700 1 -- 172.21.15.167:0/1428577185 <== mon.0 v2:172.21.15.118:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f5830008ad0 con 0x563bbf74b020 2023-10-05T19:28:57.138 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.130+0000 7f583fe1ae40 1 -- 172.21.15.167:0/1428577185 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x563bbf74b020 msgr2=0x563bbf74b3f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-05T19:28:57.139 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.130+0000 7f583fe1ae40 1 --2- 172.21.15.167:0/1428577185 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x563bbf74b020 0x563bbf74b3f0 secure :-1 s=READY pgs=555 cs=0 l=1 rev1=1 crypto rx=0x7f583000a5d0 tx=0x7f5830005a50 comp rx=0 tx=0).stop 2023-10-05T19:28:57.139 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.130+0000 7f583fe1ae40 1 -- 172.21.15.167:0/1428577185 shutdown_connections 2023-10-05T19:28:57.139 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.130+0000 7f583fe1ae40 1 --2- 172.21.15.167:0/1428577185 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x563bbf74b020 0x563bbf74b3f0 unknown :-1 s=CLOSED pgs=555 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-05T19:28:57.139 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.130+0000 7f583fe1ae40 1 -- 172.21.15.167:0/1428577185 >> 172.21.15.167:0/1428577185 conn(0x563bbf6be870 msgr2=0x563bbf6bec70 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-05T19:28:57.139 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.134+0000 7f583fe1ae40 1 -- 172.21.15.167:0/1428577185 shutdown_connections 2023-10-05T19:28:57.139 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.134+0000 7f583fe1ae40 1 -- 172.21.15.167:0/1428577185 wait complete. 2023-10-05T19:28:57.139 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.134+0000 7f583fe1ae40 1 Processor -- start 2023-10-05T19:28:57.139 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.134+0000 7f583fe1ae40 1 -- start start 2023-10-05T19:28:57.139 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.134+0000 7f583fe1ae40 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x563bbf74b020 0x563bbf769f30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-05T19:28:57.139 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.134+0000 7f583fe1ae40 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x563bbf616e30 con 0x563bbf74b020 2023-10-05T19:28:57.140 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.134+0000 7f583d94f700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x563bbf74b020 0x563bbf769f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-05T19:28:57.140 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.134+0000 7f583d94f700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x563bbf74b020 0x563bbf769f30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.118:3300/0 says I am v2:172.21.15.167:53456/0 (socket says 172.21.15.167:53456) 2023-10-05T19:28:57.140 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.134+0000 7f583d94f700 1 -- 172.21.15.167:0/3006115720 learned_addr learned my addr 172.21.15.167:0/3006115720 (peer_addr_for_me v2:172.21.15.167:0/0) 2023-10-05T19:28:57.141 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.134+0000 7f583d94f700 1 -- 172.21.15.167:0/3006115720 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5830005cf0 con 0x563bbf74b020 2023-10-05T19:28:57.141 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.134+0000 7f583d94f700 1 --2- 172.21.15.167:0/3006115720 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x563bbf74b020 0x563bbf769f30 secure :-1 s=READY pgs=556 cs=0 l=1 rev1=1 crypto rx=0x7f5830005cc0 tx=0x7f5830015000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a3c4625ed51d7ffd server_cookie=0 in_seq=0 out_seq=0 2023-10-05T19:28:57.141 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.134+0000 7f58377fe700 1 -- 172.21.15.167:0/3006115720 <== mon.0 v2:172.21.15.118:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f5830008970 con 0x563bbf74b020 2023-10-05T19:28:57.141 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.134+0000 7f58377fe700 1 -- 172.21.15.167:0/3006115720 <== mon.0 v2:172.21.15.118:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5830008ad0 con 0x563bbf74b020 2023-10-05T19:28:57.141 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.134+0000 7f58377fe700 1 -- 172.21.15.167:0/3006115720 <== mon.0 v2:172.21.15.118:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f5830008dc0 con 0x563bbf74b020 2023-10-05T19:28:57.142 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.138+0000 7f583fe1ae40 1 -- 172.21.15.167:0/3006115720 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x563bbf74b020 msgr2=0x563bbf769f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-05T19:28:57.142 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.138+0000 7f583fe1ae40 1 --2- 172.21.15.167:0/3006115720 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x563bbf74b020 0x563bbf769f30 secure :-1 s=READY pgs=556 cs=0 l=1 rev1=1 crypto rx=0x7f5830005cc0 tx=0x7f5830015000 comp rx=0 tx=0).stop 2023-10-05T19:28:57.142 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.138+0000 7f583fe1ae40 1 -- 172.21.15.167:0/3006115720 shutdown_connections 2023-10-05T19:28:57.142 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.138+0000 7f583fe1ae40 1 --2- 172.21.15.167:0/3006115720 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x563bbf74b020 0x563bbf769f30 unknown :-1 s=CLOSED pgs=556 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-05T19:28:57.142 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.138+0000 7f583fe1ae40 1 -- 172.21.15.167:0/3006115720 >> 172.21.15.167:0/3006115720 conn(0x563bbf6be870 msgr2=0x563bbf765af0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-05T19:28:57.142 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.138+0000 7f583fe1ae40 1 -- 172.21.15.167:0/3006115720 shutdown_connections 2023-10-05T19:28:57.142 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.138+0000 7f583fe1ae40 1 -- 172.21.15.167:0/3006115720 wait complete. 2023-10-05T19:28:57.143 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.138+0000 7f583fe1ae40 1 Processor -- start 2023-10-05T19:28:57.143 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.138+0000 7f583fe1ae40 1 -- start start 2023-10-05T19:28:57.143 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.138+0000 7f583fe1ae40 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x563bbf74b020 0x563bbf770b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-05T19:28:57.143 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.138+0000 7f583fe1ae40 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x563bbf5e9ed0 con 0x563bbf74b020 2023-10-05T19:28:57.144 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.138+0000 7f583d94f700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x563bbf74b020 0x563bbf770b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-05T19:28:57.144 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.138+0000 7f583d94f700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x563bbf74b020 0x563bbf770b90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.118:3300/0 says I am v2:172.21.15.167:53458/0 (socket says 172.21.15.167:53458) 2023-10-05T19:28:57.144 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.138+0000 7f583d94f700 1 -- 172.21.15.167:0/4078798006 learned_addr learned my addr 172.21.15.167:0/4078798006 (peer_addr_for_me v2:172.21.15.167:0/0) 2023-10-05T19:28:57.144 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.138+0000 7f583d94f700 1 -- 172.21.15.167:0/4078798006 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x563bbf74cae0 con 0x563bbf74b020 2023-10-05T19:28:57.145 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.138+0000 7f583d94f700 1 --2- 172.21.15.167:0/4078798006 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x563bbf74b020 0x563bbf770b90 secure :-1 s=READY pgs=557 cs=0 l=1 rev1=1 crypto rx=0x7f583001e040 tx=0x7f583001c000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-05T19:28:57.145 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.142+0000 7f5835ffb700 1 -- 172.21.15.167:0/4078798006 <== mon.0 v2:172.21.15.118:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f58300086a0 con 0x563bbf74b020 2023-10-05T19:28:57.146 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.142+0000 7f5835ffb700 1 -- 172.21.15.167:0/4078798006 <== mon.0 v2:172.21.15.118:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f583002a070 con 0x563bbf74b020 2023-10-05T19:28:57.146 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.142+0000 7f583fe1ae40 1 -- 172.21.15.167:0/4078798006 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x563bbf7710d0 con 0x563bbf74b020 2023-10-05T19:28:57.146 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.142+0000 7f5835ffb700 1 -- 172.21.15.167:0/4078798006 <== mon.0 v2:172.21.15.118:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f5830012cd0 con 0x563bbf74b020 2023-10-05T19:28:57.146 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.142+0000 7f583fe1ae40 1 -- 172.21.15.167:0/4078798006 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x563bbf771570 con 0x563bbf74b020 2023-10-05T19:28:57.147 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.142+0000 7f5835ffb700 1 -- 172.21.15.167:0/4078798006 <== mon.0 v2:172.21.15.118:3300/0 4 ==== mgrmap(e 5) v1 ==== 40329+0+0 (secure 0 0 0) 0x7f5830019070 con 0x563bbf74b020 2023-10-05T19:28:57.148 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.142+0000 7f5835ffb700 1 --2- 172.21.15.167:0/4078798006 >> [v2:172.21.15.167:6824/29889,v1:172.21.15.167:6825/29889] conn(0x7f5824030c50 0x7f5824033100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-05T19:28:57.148 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.142+0000 7f583fe1ae40 20 rados->read ofs=0 len=0 2023-10-05T19:28:57.148 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.142+0000 7f5835ffb700 1 -- 172.21.15.167:0/4078798006 <== mon.0 v2:172.21.15.118:3300/0 5 ==== osd_map(32..32 src has 1..32) v4 ==== 6260+0+0 (secure 0 0 0) 0x7f583002c070 con 0x563bbf74b020 2023-10-05T19:28:57.148 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.142+0000 7f583d14e700 1 --2- 172.21.15.167:0/4078798006 >> [v2:172.21.15.167:6824/29889,v1:172.21.15.167:6825/29889] conn(0x7f5824030c50 0x7f5824033100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-05T19:28:57.149 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.142+0000 7f583fe1ae40 1 --2- 172.21.15.167:0/4078798006 >> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] conn(0x563bbf74b930 0x563bbf77c630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-05T19:28:57.149 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.142+0000 7f583fe1ae40 1 -- 172.21.15.167:0/4078798006 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:1 5.12 5:49953fa1:::default.realm:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32) v8 -- 0x563bbf77cca0 con 0x563bbf74b930 2023-10-05T19:28:57.149 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.142+0000 7f583e150700 1 --2- 172.21.15.167:0/4078798006 >> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] conn(0x563bbf74b930 0x563bbf77c630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-05T19:28:57.149 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.142+0000 7f583d14e700 1 --2- 172.21.15.167:0/4078798006 >> [v2:172.21.15.167:6824/29889,v1:172.21.15.167:6825/29889] conn(0x7f5824030c50 0x7f5824033100 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7f58280011e0 tx=0x7f58280013a0 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-05T19:28:57.149 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.146+0000 7f583e150700 1 --2- 172.21.15.167:0/4078798006 >> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] conn(0x563bbf74b930 0x563bbf77c630 crc :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-05T19:28:57.150 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.146+0000 7f583e150700 1 -- 172.21.15.167:0/4078798006 <== osd.1 v2:172.21.15.167:6800/30035 1 ==== osd_op_reply(1 default.realm [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 ==== 157+0+0 (crc 0 0 0) 0x7f583800a5a0 con 0x563bbf74b930 2023-10-05T19:28:57.151 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.146+0000 7f583fe1ae40 20 rados_obj.operate() r=-2 bl.length=0 2023-10-05T19:28:57.151 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.146+0000 7f583fe1ae40 20 realm 2023-10-05T19:28:57.151 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.146+0000 7f583fe1ae40 20 rados->read ofs=0 len=0 2023-10-05T19:28:57.151 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.146+0000 7f583fe1ae40 1 -- 172.21.15.167:0/4078798006 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:2 5.12 5:49953fa1:::default.realm:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32) v8 -- 0x563bbf779c90 con 0x563bbf74b930 2023-10-05T19:28:57.153 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.146+0000 7f583e150700 1 -- 172.21.15.167:0/4078798006 <== osd.1 v2:172.21.15.167:6800/30035 2 ==== osd_op_reply(2 default.realm [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 ==== 157+0+0 (crc 0 0 0) 0x7f583800a5a0 con 0x563bbf74b930 2023-10-05T19:28:57.153 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.146+0000 7f583fe1ae40 20 rados_obj.operate() r=-2 bl.length=0 2023-10-05T19:28:57.153 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.146+0000 7f583fe1ae40 4 RGWPeriod::init failed to init realm id : (2) No such file or directory 2023-10-05T19:28:57.153 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.146+0000 7f583fe1ae40 20 rados->read ofs=0 len=0 2023-10-05T19:28:57.153 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.146+0000 7f583fe1ae40 1 -- 172.21.15.167:0/4078798006 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:3 5.12 5:49953fa1:::default.realm:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32) v8 -- 0x563bbf77d780 con 0x563bbf74b930 2023-10-05T19:28:57.153 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.146+0000 7f583e150700 1 -- 172.21.15.167:0/4078798006 <== osd.1 v2:172.21.15.167:6800/30035 3 ==== osd_op_reply(3 default.realm [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 ==== 157+0+0 (crc 0 0 0) 0x7f583800a5a0 con 0x563bbf74b930 2023-10-05T19:28:57.153 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.146+0000 7f583fe1ae40 20 rados_obj.operate() r=-2 bl.length=0 2023-10-05T19:28:57.153 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.146+0000 7f583fe1ae40 20 rados->read ofs=0 len=0 2023-10-05T19:28:57.153 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.146+0000 7f583fe1ae40 1 -- 172.21.15.167:0/4078798006 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:4 5.f 5:f43b6ece:::zone_names.default:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32) v8 -- 0x563bbf77d390 con 0x563bbf74b930 2023-10-05T19:28:57.153 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.146+0000 7f583e150700 1 -- 172.21.15.167:0/4078798006 <== osd.1 v2:172.21.15.167:6800/30035 4 ==== osd_op_reply(4 zone_names.default [read 0~46 out=46b] v0'0 uv1 ondisk = 0) v8 ==== 162+0+46 (crc 0 0 0) 0x7f583800a5a0 con 0x563bbf74b930 2023-10-05T19:28:57.154 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.146+0000 7f583fe1ae40 20 rados_obj.operate() r=0 bl.length=46 2023-10-05T19:28:57.154 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.146+0000 7f583fe1ae40 20 rados->read ofs=0 len=0 2023-10-05T19:28:57.154 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.146+0000 7f583fe1ae40 1 -- 172.21.15.167:0/4078798006 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:5 5.1d 5:b8374eb4:::zone_info.9a9df965-6612-45d4-a45f-6e3c889bacc2:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32) v8 -- 0x563bbf77e610 con 0x563bbf74b930 2023-10-05T19:28:57.155 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.146+0000 7f583e150700 1 -- 172.21.15.167:0/4078798006 <== osd.1 v2:172.21.15.167:6800/30035 5 ==== osd_op_reply(5 zone_info.9a9df965-6612-45d4-a45f-6e3c889bacc2 [read 0~874 out=874b] v0'0 uv2 ondisk = 0) v8 ==== 190+0+874 (crc 0 0 0) 0x7f583800a5a0 con 0x563bbf74b930 2023-10-05T19:28:57.155 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.146+0000 7f583fe1ae40 20 rados_obj.operate() r=0 bl.length=874 2023-10-05T19:28:57.155 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.146+0000 7f583fe1ae40 20 searching for the correct realm 2023-10-05T19:28:57.155 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.146+0000 7f583fe1ae40 1 --2- 172.21.15.167:0/4078798006 >> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] conn(0x563bbf7803d0 0x563bbf7827b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-05T19:28:57.155 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.150+0000 7f583fe1ae40 1 -- 172.21.15.167:0/4078798006 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:6 5.0 5:00000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x563bbf782cf0 con 0x563bbf7803d0 2023-10-05T19:28:57.155 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.150+0000 7f583d94f700 1 --2- 172.21.15.167:0/4078798006 >> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] conn(0x563bbf7803d0 0x563bbf7827b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-05T19:28:57.155 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.150+0000 7f583d94f700 1 --2- 172.21.15.167:0/4078798006 >> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] conn(0x563bbf7803d0 0x563bbf7827b0 crc :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-05T19:28:57.156 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.150+0000 7f583d94f700 1 -- 172.21.15.167:0/4078798006 <== osd.3 v2:172.21.15.167:6808/30036 1 ==== osd_op_reply(6 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f5830022020 con 0x563bbf7803d0 2023-10-05T19:28:57.156 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.150+0000 7f583d94f700 1 --2- 172.21.15.167:0/4078798006 >> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] conn(0x7f5830007030 0x7f58300245b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-05T19:28:57.156 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.150+0000 7f583d94f700 1 -- 172.21.15.167:0/4078798006 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:7 5.10 5:08000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x563bbf782cf0 con 0x7f5830007030 2023-10-05T19:28:57.156 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.150+0000 7f583d14e700 1 --2- 172.21.15.167:0/4078798006 >> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] conn(0x7f5830007030 0x7f58300245b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-05T19:28:57.157 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.150+0000 7f583d14e700 1 --2- 172.21.15.167:0/4078798006 >> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] conn(0x7f5830007030 0x7f58300245b0 crc :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-05T19:28:57.157 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.150+0000 7f583d14e700 1 -- 172.21.15.167:0/4078798006 <== osd.2 v2:172.21.15.167:6816/30037 1 ==== osd_op_reply(7 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f5828006070 con 0x7f5830007030 2023-10-05T19:28:57.157 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.150+0000 7f583d14e700 1 --2- 172.21.15.167:0/4078798006 >> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] conn(0x7f582800d8e0 0x7f582800c360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-05T19:28:57.157 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.150+0000 7f583d14e700 1 -- 172.21.15.167:0/4078798006 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:8 5.8 5:10000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x563bbf782cf0 con 0x7f582800d8e0 2023-10-05T19:28:57.157 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.150+0000 7f583e150700 1 --2- 172.21.15.167:0/4078798006 >> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] conn(0x7f582800d8e0 0x7f582800c360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-05T19:28:57.158 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.154+0000 7f583e150700 1 --2- 172.21.15.167:0/4078798006 >> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] conn(0x7f582800d8e0 0x7f582800c360 crc :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-05T19:28:57.159 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.154+0000 7f583e150700 1 -- 172.21.15.167:0/4078798006 <== osd.0 v2:172.21.15.118:6800/27690 1 ==== osd_op_reply(8 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f583800a5a0 con 0x7f582800d8e0 2023-10-05T19:28:57.159 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.154+0000 7f583e150700 1 -- 172.21.15.167:0/4078798006 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:9 5.18 5:18000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x563bbf782cf0 con 0x563bbf74b930 2023-10-05T19:28:57.159 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.154+0000 7f583e150700 1 -- 172.21.15.167:0/4078798006 <== osd.1 v2:172.21.15.167:6800/30035 6 ==== osd_op_reply(9 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f583800a5a0 con 0x563bbf74b930 2023-10-05T19:28:57.160 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.154+0000 7f583e150700 1 -- 172.21.15.167:0/4078798006 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:10 5.4 5:20000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x563bbf782cf0 con 0x563bbf74b930 2023-10-05T19:28:57.160 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.154+0000 7f583e150700 1 -- 172.21.15.167:0/4078798006 <== osd.1 v2:172.21.15.167:6800/30035 7 ==== osd_op_reply(10 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f583800a5a0 con 0x563bbf74b930 2023-10-05T19:28:57.160 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.154+0000 7f583e150700 1 -- 172.21.15.167:0/4078798006 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:11 5.14 5:28000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x563bbf782cf0 con 0x7f582800d8e0 2023-10-05T19:28:57.161 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.154+0000 7f583e150700 1 -- 172.21.15.167:0/4078798006 <== osd.0 v2:172.21.15.118:6800/27690 2 ==== osd_op_reply(11 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f583800a5a0 con 0x7f582800d8e0 2023-10-05T19:28:57.161 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.154+0000 7f583e150700 1 -- 172.21.15.167:0/4078798006 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:12 5.c 5:30000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x563bbf782cf0 con 0x7f582800d8e0 2023-10-05T19:28:57.162 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.158+0000 7f583e150700 1 -- 172.21.15.167:0/4078798006 <== osd.0 v2:172.21.15.118:6800/27690 3 ==== osd_op_reply(12 [pgnls start_epoch 32 out=112b] v23'1 uv1 ondisk = 1) v8 ==== 144+0+112 (crc 0 0 0) 0x7f583800a5a0 con 0x7f582800d8e0 2023-10-05T19:28:57.162 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.158+0000 7f583e150700 1 -- 172.21.15.167:0/4078798006 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:13 5.1c 5:38000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x563bbf782cf0 con 0x7f5830007030 2023-10-05T19:28:57.163 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.158+0000 7f583d14e700 1 -- 172.21.15.167:0/4078798006 <== osd.2 v2:172.21.15.167:6816/30037 2 ==== osd_op_reply(13 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f5828006070 con 0x7f5830007030 2023-10-05T19:28:57.163 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.158+0000 7f583d14e700 1 -- 172.21.15.167:0/4078798006 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:14 5.2 5:40000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x563bbf782cf0 con 0x563bbf7803d0 2023-10-05T19:28:57.163 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.158+0000 7f583d94f700 1 -- 172.21.15.167:0/4078798006 <== osd.3 v2:172.21.15.167:6808/30036 2 ==== osd_op_reply(14 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f5830022020 con 0x563bbf7803d0 2023-10-05T19:28:57.163 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.158+0000 7f583d94f700 1 -- 172.21.15.167:0/4078798006 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:15 5.12 5:48000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x563bbf782cf0 con 0x563bbf74b930 2023-10-05T19:28:57.164 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.158+0000 7f583e150700 1 -- 172.21.15.167:0/4078798006 <== osd.1 v2:172.21.15.167:6800/30035 8 ==== osd_op_reply(15 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f583800a5a0 con 0x563bbf74b930 2023-10-05T19:28:57.164 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.158+0000 7f583e150700 1 -- 172.21.15.167:0/4078798006 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:16 5.a 5:50000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x563bbf782cf0 con 0x7f582800d8e0 2023-10-05T19:28:57.165 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.158+0000 7f583e150700 1 -- 172.21.15.167:0/4078798006 <== osd.0 v2:172.21.15.118:6800/27690 4 ==== osd_op_reply(16 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f583800a5a0 con 0x7f582800d8e0 2023-10-05T19:28:57.165 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.158+0000 7f583e150700 1 -- 172.21.15.167:0/4078798006 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:17 5.1a 5:58000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x563bbf782cf0 con 0x563bbf74b930 2023-10-05T19:28:57.165 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.162+0000 7f583e150700 1 -- 172.21.15.167:0/4078798006 <== osd.1 v2:172.21.15.167:6800/30035 9 ==== osd_op_reply(17 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f583800a5a0 con 0x563bbf74b930 2023-10-05T19:28:57.166 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.162+0000 7f583e150700 1 -- 172.21.15.167:0/4078798006 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:18 5.6 5:60000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x563bbf782cf0 con 0x7f5830007030 2023-10-05T19:28:57.166 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.162+0000 7f583d14e700 1 -- 172.21.15.167:0/4078798006 <== osd.2 v2:172.21.15.167:6816/30037 3 ==== osd_op_reply(18 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f5828006070 con 0x7f5830007030 2023-10-05T19:28:57.166 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.162+0000 7f583d14e700 1 -- 172.21.15.167:0/4078798006 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:19 5.16 5:68000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x563bbf782cf0 con 0x563bbf7803d0 2023-10-05T19:28:57.167 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.162+0000 7f583d94f700 1 -- 172.21.15.167:0/4078798006 <== osd.3 v2:172.21.15.167:6808/30036 3 ==== osd_op_reply(19 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f5830022020 con 0x563bbf7803d0 2023-10-05T19:28:57.167 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.162+0000 7f583d94f700 1 -- 172.21.15.167:0/4078798006 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:20 5.e 5:70000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x563bbf782cf0 con 0x563bbf7803d0 2023-10-05T19:28:57.167 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.162+0000 7f583d94f700 1 -- 172.21.15.167:0/4078798006 <== osd.3 v2:172.21.15.167:6808/30036 4 ==== osd_op_reply(20 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f5830022020 con 0x563bbf7803d0 2023-10-05T19:28:57.167 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.162+0000 7f583d94f700 1 -- 172.21.15.167:0/4078798006 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:21 5.1e 5:78000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x563bbf782cf0 con 0x7f582800d8e0 2023-10-05T19:28:57.168 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.162+0000 7f583e150700 1 -- 172.21.15.167:0/4078798006 <== osd.0 v2:172.21.15.118:6800/27690 5 ==== osd_op_reply(21 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f583800a5a0 con 0x7f582800d8e0 2023-10-05T19:28:57.168 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.162+0000 7f583e150700 1 -- 172.21.15.167:0/4078798006 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:22 5.1 5:80000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x563bbf782cf0 con 0x563bbf7803d0 2023-10-05T19:28:57.169 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.162+0000 7f583d94f700 1 -- 172.21.15.167:0/4078798006 <== osd.3 v2:172.21.15.167:6808/30036 5 ==== osd_op_reply(22 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f5830022020 con 0x563bbf7803d0 2023-10-05T19:28:57.169 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.162+0000 7f583d94f700 1 -- 172.21.15.167:0/4078798006 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:23 5.11 5:88000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x563bbf782cf0 con 0x563bbf7803d0 2023-10-05T19:28:57.170 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.166+0000 7f583d94f700 1 -- 172.21.15.167:0/4078798006 <== osd.3 v2:172.21.15.167:6808/30036 6 ==== osd_op_reply(23 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f5830022020 con 0x563bbf7803d0 2023-10-05T19:28:57.170 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.166+0000 7f583d94f700 1 -- 172.21.15.167:0/4078798006 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:24 5.9 5:90000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x563bbf782cf0 con 0x563bbf74b930 2023-10-05T19:28:57.170 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.166+0000 7f583e150700 1 -- 172.21.15.167:0/4078798006 <== osd.1 v2:172.21.15.167:6800/30035 10 ==== osd_op_reply(24 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f583800a5a0 con 0x563bbf74b930 2023-10-05T19:28:57.171 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.166+0000 7f583e150700 1 -- 172.21.15.167:0/4078798006 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:25 5.19 5:98000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x563bbf782cf0 con 0x563bbf74b930 2023-10-05T19:28:57.171 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.166+0000 7f583e150700 1 -- 172.21.15.167:0/4078798006 <== osd.1 v2:172.21.15.167:6800/30035 11 ==== osd_op_reply(25 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f583800a5a0 con 0x563bbf74b930 2023-10-05T19:28:57.171 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.166+0000 7f583e150700 1 -- 172.21.15.167:0/4078798006 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:26 5.5 5:a0000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x563bbf782cf0 con 0x7f582800d8e0 2023-10-05T19:28:57.172 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.166+0000 7f583e150700 1 -- 172.21.15.167:0/4078798006 <== osd.0 v2:172.21.15.118:6800/27690 6 ==== osd_op_reply(26 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f583800a5a0 con 0x7f582800d8e0 2023-10-05T19:28:57.172 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.166+0000 7f583e150700 1 -- 172.21.15.167:0/4078798006 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:27 5.15 5:a8000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x563bbf782cf0 con 0x563bbf7803d0 2023-10-05T19:28:57.173 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.166+0000 7f583d94f700 1 -- 172.21.15.167:0/4078798006 <== osd.3 v2:172.21.15.167:6808/30036 7 ==== osd_op_reply(27 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f5830022020 con 0x563bbf7803d0 2023-10-05T19:28:57.173 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.166+0000 7f583d94f700 1 -- 172.21.15.167:0/4078798006 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:28 5.d 5:b0000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x563bbf782cf0 con 0x7f5830007030 2023-10-05T19:28:57.174 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.170+0000 7f583d14e700 1 -- 172.21.15.167:0/4078798006 <== osd.2 v2:172.21.15.167:6816/30037 4 ==== osd_op_reply(28 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f5828006070 con 0x7f5830007030 2023-10-05T19:28:57.174 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.170+0000 7f583d14e700 1 -- 172.21.15.167:0/4078798006 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:29 5.1d 5:b8000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x563bbf782cf0 con 0x563bbf74b930 2023-10-05T19:28:57.174 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.170+0000 7f583e150700 1 -- 172.21.15.167:0/4078798006 <== osd.1 v2:172.21.15.167:6800/30035 12 ==== osd_op_reply(29 [pgnls start_epoch 32 out=107b] v30'2 uv2 ondisk = 1) v8 ==== 144+0+107 (crc 0 0 0) 0x7f583800a5a0 con 0x563bbf74b930 2023-10-05T19:28:57.174 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.170+0000 7f583e150700 1 -- 172.21.15.167:0/4078798006 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:30 5.3 5:c0000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x563bbf782cf0 con 0x7f582800d8e0 2023-10-05T19:28:57.175 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.170+0000 7f583e150700 1 -- 172.21.15.167:0/4078798006 <== osd.0 v2:172.21.15.118:6800/27690 7 ==== osd_op_reply(30 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f583800a5a0 con 0x7f582800d8e0 2023-10-05T19:28:57.176 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.170+0000 7f583e150700 1 -- 172.21.15.167:0/4078798006 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:31 5.13 5:c8000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x563bbf782cf0 con 0x563bbf7803d0 2023-10-05T19:28:57.176 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.170+0000 7f583d94f700 1 -- 172.21.15.167:0/4078798006 <== osd.3 v2:172.21.15.167:6808/30036 8 ==== osd_op_reply(31 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f5830022020 con 0x563bbf7803d0 2023-10-05T19:28:57.176 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.170+0000 7f583d94f700 1 -- 172.21.15.167:0/4078798006 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:32 5.b 5:d0000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x563bbf782cf0 con 0x7f582800d8e0 2023-10-05T19:28:57.177 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.170+0000 7f583e150700 1 -- 172.21.15.167:0/4078798006 <== osd.0 v2:172.21.15.118:6800/27690 8 ==== osd_op_reply(32 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f583800a5a0 con 0x7f582800d8e0 2023-10-05T19:28:57.177 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.174+0000 7f583e150700 1 -- 172.21.15.167:0/4078798006 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:33 5.1b 5:d8000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x563bbf782cf0 con 0x7f5830007030 2023-10-05T19:28:57.178 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.174+0000 7f583d14e700 1 -- 172.21.15.167:0/4078798006 <== osd.2 v2:172.21.15.167:6816/30037 5 ==== osd_op_reply(33 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f5828006070 con 0x7f5830007030 2023-10-05T19:28:57.178 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.174+0000 7f583d14e700 1 -- 172.21.15.167:0/4078798006 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:34 5.7 5:e0000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x563bbf782cf0 con 0x563bbf7803d0 2023-10-05T19:28:57.179 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.174+0000 7f583d94f700 1 -- 172.21.15.167:0/4078798006 <== osd.3 v2:172.21.15.167:6808/30036 9 ==== osd_op_reply(34 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f5830022020 con 0x563bbf7803d0 2023-10-05T19:28:57.179 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.174+0000 7f583d94f700 1 -- 172.21.15.167:0/4078798006 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:35 5.17 5:e8000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x563bbf782cf0 con 0x7f582800d8e0 2023-10-05T19:28:57.180 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.174+0000 7f583e150700 1 -- 172.21.15.167:0/4078798006 <== osd.0 v2:172.21.15.118:6800/27690 9 ==== osd_op_reply(35 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f583800a5a0 con 0x7f582800d8e0 2023-10-05T19:28:57.180 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.174+0000 7f583e150700 1 -- 172.21.15.167:0/4078798006 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:36 5.f 5:f0000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x563bbf782cf0 con 0x563bbf74b930 2023-10-05T19:28:57.181 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.174+0000 7f583e150700 1 -- 172.21.15.167:0/4078798006 <== osd.1 v2:172.21.15.167:6800/30035 13 ==== osd_op_reply(36 [pgnls start_epoch 32 out=115b] v23'2 uv2 ondisk = 1) v8 ==== 144+0+115 (crc 0 0 0) 0x7f583800a5a0 con 0x563bbf74b930 2023-10-05T19:28:57.181 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.174+0000 7f583e150700 1 -- 172.21.15.167:0/4078798006 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:37 5.1f 5:f8000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x563bbf782cf0 con 0x7f5830007030 2023-10-05T19:28:57.182 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.178+0000 7f583d14e700 1 -- 172.21.15.167:0/4078798006 <== osd.2 v2:172.21.15.167:6816/30037 6 ==== osd_op_reply(37 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f5828006070 con 0x7f5830007030 2023-10-05T19:28:57.182 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.178+0000 7f583fe1ae40 20 RGWRados::pool_iterate: got zonegroup_info.4f1f719d-9648-417b-b10c-e5d939825419 2023-10-05T19:28:57.182 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.178+0000 7f583fe1ae40 20 RGWRados::pool_iterate: got zone_info.9a9df965-6612-45d4-a45f-6e3c889bacc2 2023-10-05T19:28:57.182 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.178+0000 7f583fe1ae40 20 RGWRados::pool_iterate: got zone_names.default 2023-10-05T19:28:57.182 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.178+0000 7f583fe1ae40 20 RGWRados::pool_iterate: got zonegroups_names.default 2023-10-05T19:28:57.182 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.178+0000 7f583fe1ae40 20 rados->read ofs=0 len=0 2023-10-05T19:28:57.182 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.178+0000 7f583fe1ae40 1 -- 172.21.15.167:0/4078798006 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:38 5.12 5:49953fa1:::default.realm:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32) v8 -- 0x563bbf77f180 con 0x563bbf74b930 2023-10-05T19:28:57.183 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.178+0000 7f583e150700 1 -- 172.21.15.167:0/4078798006 <== osd.1 v2:172.21.15.167:6800/30035 14 ==== osd_op_reply(38 default.realm [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 ==== 157+0+0 (crc 0 0 0) 0x7f583800a5a0 con 0x563bbf74b930 2023-10-05T19:28:57.184 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.178+0000 7f583fe1ae40 20 rados_obj.operate() r=-2 bl.length=0 2023-10-05T19:28:57.184 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.178+0000 7f583fe1ae40 20 rados->read ofs=0 len=0 2023-10-05T19:28:57.184 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.178+0000 7f583fe1ae40 1 -- 172.21.15.167:0/4078798006 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:39 5.f 5:f4c53578:::zonegroups_names.default:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32) v8 -- 0x563bbf77fa70 con 0x563bbf74b930 2023-10-05T19:28:57.184 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.178+0000 7f583e150700 1 -- 172.21.15.167:0/4078798006 <== osd.1 v2:172.21.15.167:6800/30035 15 ==== osd_op_reply(39 zonegroups_names.default [read 0~46 out=46b] v0'0 uv2 ondisk = 0) v8 ==== 168+0+46 (crc 0 0 0) 0x7f583800a5a0 con 0x563bbf74b930 2023-10-05T19:28:57.184 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.178+0000 7f583fe1ae40 20 rados_obj.operate() r=0 bl.length=46 2023-10-05T19:28:57.184 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.178+0000 7f583fe1ae40 20 rados->read ofs=0 len=0 2023-10-05T19:28:57.184 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.178+0000 7f583fe1ae40 1 -- 172.21.15.167:0/4078798006 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:40 5.c 5:30d299a5:::zonegroup_info.4f1f719d-9648-417b-b10c-e5d939825419:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32) v8 -- 0x563bbf77fa70 con 0x7f582800d8e0 2023-10-05T19:28:57.185 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.178+0000 7f583e150700 1 -- 172.21.15.167:0/4078798006 <== osd.0 v2:172.21.15.118:6800/27690 10 ==== osd_op_reply(40 zonegroup_info.4f1f719d-9648-417b-b10c-e5d939825419 [read 0~362 out=362b] v0'0 uv1 ondisk = 0) v8 ==== 195+0+362 (crc 0 0 0) 0x7f583800a5a0 con 0x7f582800d8e0 2023-10-05T19:28:57.185 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.178+0000 7f583fe1ae40 20 rados_obj.operate() r=0 bl.length=362 2023-10-05T19:28:57.185 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.178+0000 7f583fe1ae40 20 rados->read ofs=0 len=0 2023-10-05T19:28:57.185 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.178+0000 7f583fe1ae40 1 -- 172.21.15.167:0/4078798006 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:41 5.13 5:cbbf6c60:::converted:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32) v8 -- 0x563bbf77fa70 con 0x563bbf7803d0 2023-10-05T19:28:57.186 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.178+0000 7f583d94f700 1 -- 172.21.15.167:0/4078798006 <== osd.3 v2:172.21.15.167:6808/30036 10 ==== osd_op_reply(41 converted [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 ==== 153+0+0 (crc 0 0 0) 0x7f5830050040 con 0x563bbf7803d0 2023-10-05T19:28:57.186 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.178+0000 7f583fe1ae40 20 rados_obj.operate() r=-2 bl.length=0 2023-10-05T19:28:57.186 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.178+0000 7f583fe1ae40 20 rados->read ofs=0 len=0 2023-10-05T19:28:57.186 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.178+0000 7f583fe1ae40 1 -- 172.21.15.167:0/4078798006 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:42 5.12 5:49953fa1:::default.realm:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32) v8 -- 0x563bbf783160 con 0x563bbf74b930 2023-10-05T19:28:57.187 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.182+0000 7f583e150700 1 -- 172.21.15.167:0/4078798006 <== osd.1 v2:172.21.15.167:6800/30035 16 ==== osd_op_reply(42 default.realm [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 ==== 157+0+0 (crc 0 0 0) 0x7f583800a5a0 con 0x563bbf74b930 2023-10-05T19:28:57.187 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.182+0000 7f583fe1ae40 20 rados_obj.operate() r=-2 bl.length=0 2023-10-05T19:28:57.187 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.182+0000 7f583fe1ae40 20 rados->read ofs=0 len=0 2023-10-05T19:28:57.187 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.182+0000 7f583fe1ae40 1 -- 172.21.15.167:0/4078798006 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:43 5.f 5:f4c53578:::zonegroups_names.default:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32) v8 -- 0x563bbf783560 con 0x563bbf74b930 2023-10-05T19:28:57.187 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.182+0000 7f583e150700 1 -- 172.21.15.167:0/4078798006 <== osd.1 v2:172.21.15.167:6800/30035 17 ==== osd_op_reply(43 zonegroups_names.default [read 0~46 out=46b] v0'0 uv2 ondisk = 0) v8 ==== 168+0+46 (crc 0 0 0) 0x7f583800a5a0 con 0x563bbf74b930 2023-10-05T19:28:57.187 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.182+0000 7f583fe1ae40 20 rados_obj.operate() r=0 bl.length=46 2023-10-05T19:28:57.187 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.182+0000 7f583fe1ae40 1 -- 172.21.15.167:0/4078798006 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:44 5.0 5:00000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x563bbf784570 con 0x563bbf7803d0 2023-10-05T19:28:57.188 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.182+0000 7f583d94f700 1 -- 172.21.15.167:0/4078798006 <== osd.3 v2:172.21.15.167:6808/30036 11 ==== osd_op_reply(44 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f5830050040 con 0x563bbf7803d0 2023-10-05T19:28:57.188 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.182+0000 7f583d94f700 1 -- 172.21.15.167:0/4078798006 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:45 5.10 5:08000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x563bbf784570 con 0x7f5830007030 2023-10-05T19:28:57.189 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.182+0000 7f583d14e700 1 -- 172.21.15.167:0/4078798006 <== osd.2 v2:172.21.15.167:6816/30037 7 ==== osd_op_reply(45 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f5828006070 con 0x7f5830007030 2023-10-05T19:28:57.189 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.182+0000 7f583d14e700 1 -- 172.21.15.167:0/4078798006 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:46 5.8 5:10000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x563bbf784570 con 0x7f582800d8e0 2023-10-05T19:28:57.190 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.186+0000 7f583e150700 1 -- 172.21.15.167:0/4078798006 <== osd.0 v2:172.21.15.118:6800/27690 11 ==== osd_op_reply(46 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f583800a5a0 con 0x7f582800d8e0 2023-10-05T19:28:57.190 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.186+0000 7f583e150700 1 -- 172.21.15.167:0/4078798006 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:47 5.18 5:18000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x563bbf77f180 con 0x563bbf74b930 2023-10-05T19:28:57.190 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.186+0000 7f583e150700 1 -- 172.21.15.167:0/4078798006 <== osd.1 v2:172.21.15.167:6800/30035 18 ==== osd_op_reply(47 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f583800a5a0 con 0x563bbf74b930 2023-10-05T19:28:57.191 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.186+0000 7f583e150700 1 -- 172.21.15.167:0/4078798006 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:48 5.4 5:20000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x563bbf77f180 con 0x563bbf74b930 2023-10-05T19:28:57.191 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.186+0000 7f583e150700 1 -- 172.21.15.167:0/4078798006 <== osd.1 v2:172.21.15.167:6800/30035 19 ==== osd_op_reply(48 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f583800a5a0 con 0x563bbf74b930 2023-10-05T19:28:57.191 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.186+0000 7f583e150700 1 -- 172.21.15.167:0/4078798006 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:49 5.14 5:28000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x563bbf77f180 con 0x7f582800d8e0 2023-10-05T19:28:57.192 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.186+0000 7f583e150700 1 -- 172.21.15.167:0/4078798006 <== osd.0 v2:172.21.15.118:6800/27690 12 ==== osd_op_reply(49 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f583800a5a0 con 0x7f582800d8e0 2023-10-05T19:28:57.192 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.186+0000 7f583e150700 1 -- 172.21.15.167:0/4078798006 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:50 5.c 5:30000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x563bbf77f180 con 0x7f582800d8e0 2023-10-05T19:28:57.193 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.186+0000 7f583e150700 1 -- 172.21.15.167:0/4078798006 <== osd.0 v2:172.21.15.118:6800/27690 13 ==== osd_op_reply(50 [pgnls start_epoch 32 out=112b] v23'1 uv1 ondisk = 1) v8 ==== 144+0+112 (crc 0 0 0) 0x7f583800a5a0 con 0x7f582800d8e0 2023-10-05T19:28:57.193 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.186+0000 7f583e150700 1 -- 172.21.15.167:0/4078798006 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:51 5.1c 5:38000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x563bbf77f180 con 0x7f5830007030 2023-10-05T19:28:57.194 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.190+0000 7f583d14e700 1 -- 172.21.15.167:0/4078798006 <== osd.2 v2:172.21.15.167:6816/30037 8 ==== osd_op_reply(51 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f5828006070 con 0x7f5830007030 2023-10-05T19:28:57.194 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.190+0000 7f583d14e700 1 -- 172.21.15.167:0/4078798006 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:52 5.2 5:40000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x563bbf77f180 con 0x563bbf7803d0 2023-10-05T19:28:57.195 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.190+0000 7f583d94f700 1 -- 172.21.15.167:0/4078798006 <== osd.3 v2:172.21.15.167:6808/30036 12 ==== osd_op_reply(52 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f5830050040 con 0x563bbf7803d0 2023-10-05T19:28:57.195 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.190+0000 7f583d94f700 1 -- 172.21.15.167:0/4078798006 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:53 5.12 5:48000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x563bbf77f180 con 0x563bbf74b930 2023-10-05T19:28:57.195 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.190+0000 7f583e150700 1 -- 172.21.15.167:0/4078798006 <== osd.1 v2:172.21.15.167:6800/30035 20 ==== osd_op_reply(53 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f583800a5a0 con 0x563bbf74b930 2023-10-05T19:28:57.195 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.190+0000 7f583e150700 1 -- 172.21.15.167:0/4078798006 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:54 5.a 5:50000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x563bbf77f180 con 0x7f582800d8e0 2023-10-05T19:28:57.196 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.190+0000 7f583e150700 1 -- 172.21.15.167:0/4078798006 <== osd.0 v2:172.21.15.118:6800/27690 14 ==== osd_op_reply(54 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f583800a5a0 con 0x7f582800d8e0 2023-10-05T19:28:57.197 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.190+0000 7f583e150700 1 -- 172.21.15.167:0/4078798006 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:55 5.1a 5:58000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x563bbf77f180 con 0x563bbf74b930 2023-10-05T19:28:57.197 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.194+0000 7f583e150700 1 -- 172.21.15.167:0/4078798006 <== osd.1 v2:172.21.15.167:6800/30035 21 ==== osd_op_reply(55 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f583800a5a0 con 0x563bbf74b930 2023-10-05T19:28:57.197 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.194+0000 7f583e150700 1 -- 172.21.15.167:0/4078798006 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:56 5.6 5:60000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x563bbf77f180 con 0x7f5830007030 2023-10-05T19:28:57.198 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.194+0000 7f583d14e700 1 -- 172.21.15.167:0/4078798006 <== osd.2 v2:172.21.15.167:6816/30037 9 ==== osd_op_reply(56 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f5828006070 con 0x7f5830007030 2023-10-05T19:28:57.199 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.194+0000 7f583d14e700 1 -- 172.21.15.167:0/4078798006 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:57 5.16 5:68000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x563bbf77f180 con 0x563bbf7803d0 2023-10-05T19:28:57.199 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.194+0000 7f583d94f700 1 -- 172.21.15.167:0/4078798006 <== osd.3 v2:172.21.15.167:6808/30036 13 ==== osd_op_reply(57 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f5830050040 con 0x563bbf7803d0 2023-10-05T19:28:57.199 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.194+0000 7f583d94f700 1 -- 172.21.15.167:0/4078798006 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:58 5.e 5:70000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x563bbf77f180 con 0x563bbf7803d0 2023-10-05T19:28:57.200 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.194+0000 7f583d94f700 1 -- 172.21.15.167:0/4078798006 <== osd.3 v2:172.21.15.167:6808/30036 14 ==== osd_op_reply(58 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f5830050040 con 0x563bbf7803d0 2023-10-05T19:28:57.200 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.194+0000 7f583d94f700 1 -- 172.21.15.167:0/4078798006 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:59 5.1e 5:78000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x563bbf77f180 con 0x7f582800d8e0 2023-10-05T19:28:57.201 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.194+0000 7f583e150700 1 -- 172.21.15.167:0/4078798006 <== osd.0 v2:172.21.15.118:6800/27690 15 ==== osd_op_reply(59 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f583803f040 con 0x7f582800d8e0 2023-10-05T19:28:57.201 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.194+0000 7f583e150700 1 -- 172.21.15.167:0/4078798006 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:60 5.1 5:80000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x563bbf77f180 con 0x563bbf7803d0 2023-10-05T19:28:57.201 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.198+0000 7f583d94f700 1 -- 172.21.15.167:0/4078798006 <== osd.3 v2:172.21.15.167:6808/30036 15 ==== osd_op_reply(60 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f5830050040 con 0x563bbf7803d0 2023-10-05T19:28:57.201 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.198+0000 7f583d94f700 1 -- 172.21.15.167:0/4078798006 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:61 5.11 5:88000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x563bbf77f180 con 0x563bbf7803d0 2023-10-05T19:28:57.202 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.198+0000 7f583d94f700 1 -- 172.21.15.167:0/4078798006 <== osd.3 v2:172.21.15.167:6808/30036 16 ==== osd_op_reply(61 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f5830050040 con 0x563bbf7803d0 2023-10-05T19:28:57.202 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.198+0000 7f583d94f700 1 -- 172.21.15.167:0/4078798006 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:62 5.9 5:90000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x563bbf77f180 con 0x563bbf74b930 2023-10-05T19:28:57.203 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.198+0000 7f583e150700 1 -- 172.21.15.167:0/4078798006 <== osd.1 v2:172.21.15.167:6800/30035 22 ==== osd_op_reply(62 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f583803f040 con 0x563bbf74b930 2023-10-05T19:28:57.203 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.198+0000 7f583e150700 1 -- 172.21.15.167:0/4078798006 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:63 5.19 5:98000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x563bbf77f180 con 0x563bbf74b930 2023-10-05T19:28:57.204 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.198+0000 7f583e150700 1 -- 172.21.15.167:0/4078798006 <== osd.1 v2:172.21.15.167:6800/30035 23 ==== osd_op_reply(63 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f583803f040 con 0x563bbf74b930 2023-10-05T19:28:57.204 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.198+0000 7f583e150700 1 -- 172.21.15.167:0/4078798006 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:64 5.5 5:a0000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x563bbf77f180 con 0x7f582800d8e0 2023-10-05T19:28:57.205 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.198+0000 7f583e150700 1 -- 172.21.15.167:0/4078798006 <== osd.0 v2:172.21.15.118:6800/27690 16 ==== osd_op_reply(64 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f583803f040 con 0x7f582800d8e0 2023-10-05T19:28:57.205 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.198+0000 7f583e150700 1 -- 172.21.15.167:0/4078798006 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:65 5.15 5:a8000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x563bbf77f180 con 0x563bbf7803d0 2023-10-05T19:28:57.205 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.202+0000 7f583d94f700 1 -- 172.21.15.167:0/4078798006 <== osd.3 v2:172.21.15.167:6808/30036 17 ==== osd_op_reply(65 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f5830050040 con 0x563bbf7803d0 2023-10-05T19:28:57.206 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.202+0000 7f583d94f700 1 -- 172.21.15.167:0/4078798006 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:66 5.d 5:b0000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x563bbf77f180 con 0x7f5830007030 2023-10-05T19:28:57.206 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.202+0000 7f583d14e700 1 -- 172.21.15.167:0/4078798006 <== osd.2 v2:172.21.15.167:6816/30037 10 ==== osd_op_reply(66 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f5828006070 con 0x7f5830007030 2023-10-05T19:28:57.207 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.202+0000 7f583d14e700 1 -- 172.21.15.167:0/4078798006 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:67 5.1d 5:b8000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x563bbf77f180 con 0x563bbf74b930 2023-10-05T19:28:57.208 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.202+0000 7f583e150700 1 -- 172.21.15.167:0/4078798006 <== osd.1 v2:172.21.15.167:6800/30035 24 ==== osd_op_reply(67 [pgnls start_epoch 32 out=107b] v30'2 uv2 ondisk = 1) v8 ==== 144+0+107 (crc 0 0 0) 0x7f583803f040 con 0x563bbf74b930 2023-10-05T19:28:57.208 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.202+0000 7f583e150700 1 -- 172.21.15.167:0/4078798006 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:68 5.3 5:c0000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x563bbf77f180 con 0x7f582800d8e0 2023-10-05T19:28:57.208 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.202+0000 7f583e150700 1 -- 172.21.15.167:0/4078798006 <== osd.0 v2:172.21.15.118:6800/27690 17 ==== osd_op_reply(68 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f583803f040 con 0x7f582800d8e0 2023-10-05T19:28:57.208 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.202+0000 7f583e150700 1 -- 172.21.15.167:0/4078798006 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:69 5.13 5:c8000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x563bbf77f180 con 0x563bbf7803d0 2023-10-05T19:28:57.209 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.206+0000 7f583d94f700 1 -- 172.21.15.167:0/4078798006 <== osd.3 v2:172.21.15.167:6808/30036 18 ==== osd_op_reply(69 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f5830050040 con 0x563bbf7803d0 2023-10-05T19:28:57.209 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.206+0000 7f583d94f700 1 -- 172.21.15.167:0/4078798006 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:70 5.b 5:d0000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x563bbf77f180 con 0x7f582800d8e0 2023-10-05T19:28:57.210 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.206+0000 7f583e150700 1 -- 172.21.15.167:0/4078798006 <== osd.0 v2:172.21.15.118:6800/27690 18 ==== osd_op_reply(70 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f583803f040 con 0x7f582800d8e0 2023-10-05T19:28:57.210 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.206+0000 7f583e150700 1 -- 172.21.15.167:0/4078798006 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:71 5.1b 5:d8000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x563bbf77f180 con 0x7f5830007030 2023-10-05T19:28:57.211 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.206+0000 7f583d14e700 1 -- 172.21.15.167:0/4078798006 <== osd.2 v2:172.21.15.167:6816/30037 11 ==== osd_op_reply(71 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f5828006070 con 0x7f5830007030 2023-10-05T19:28:57.211 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.206+0000 7f583d14e700 1 -- 172.21.15.167:0/4078798006 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:72 5.7 5:e0000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x563bbf77f180 con 0x563bbf7803d0 2023-10-05T19:28:57.212 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.206+0000 7f583d94f700 1 -- 172.21.15.167:0/4078798006 <== osd.3 v2:172.21.15.167:6808/30036 19 ==== osd_op_reply(72 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f5830050040 con 0x563bbf7803d0 2023-10-05T19:28:57.212 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.206+0000 7f583d94f700 1 -- 172.21.15.167:0/4078798006 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:73 5.17 5:e8000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x563bbf77f180 con 0x7f582800d8e0 2023-10-05T19:28:57.213 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.206+0000 7f583e150700 1 -- 172.21.15.167:0/4078798006 <== osd.0 v2:172.21.15.118:6800/27690 19 ==== osd_op_reply(73 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f583803f040 con 0x7f582800d8e0 2023-10-05T19:28:57.213 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.206+0000 7f583e150700 1 -- 172.21.15.167:0/4078798006 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:74 5.f 5:f0000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x563bbf77f180 con 0x563bbf74b930 2023-10-05T19:28:57.214 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.210+0000 7f583e150700 1 -- 172.21.15.167:0/4078798006 <== osd.1 v2:172.21.15.167:6800/30035 25 ==== osd_op_reply(74 [pgnls start_epoch 32 out=115b] v23'2 uv2 ondisk = 1) v8 ==== 144+0+115 (crc 0 0 0) 0x7f583803f040 con 0x563bbf74b930 2023-10-05T19:28:57.214 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.210+0000 7f583e150700 1 -- 172.21.15.167:0/4078798006 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:75 5.1f 5:f8000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x563bbf77f180 con 0x7f5830007030 2023-10-05T19:28:57.215 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.210+0000 7f583d14e700 1 -- 172.21.15.167:0/4078798006 <== osd.2 v2:172.21.15.167:6816/30037 12 ==== osd_op_reply(75 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f5828006070 con 0x7f5830007030 2023-10-05T19:28:57.215 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.210+0000 7f583fe1ae40 20 RGWRados::pool_iterate: got zonegroup_info.4f1f719d-9648-417b-b10c-e5d939825419 2023-10-05T19:28:57.215 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.210+0000 7f583fe1ae40 20 RGWRados::pool_iterate: got zone_info.9a9df965-6612-45d4-a45f-6e3c889bacc2 2023-10-05T19:28:57.216 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.210+0000 7f583fe1ae40 20 RGWRados::pool_iterate: got zone_names.default 2023-10-05T19:28:57.216 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.210+0000 7f583fe1ae40 20 RGWRados::pool_iterate: got zonegroups_names.default 2023-10-05T19:28:57.216 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.210+0000 7f583fe1ae40 20 rados->read ofs=0 len=0 2023-10-05T19:28:57.216 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.210+0000 7f583fe1ae40 1 -- 172.21.15.167:0/4078798006 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:76 5.f 5:f43b6ece:::zone_names.default:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32) v8 -- 0x563bbf783560 con 0x563bbf74b930 2023-10-05T19:28:57.217 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.210+0000 7f583e150700 1 -- 172.21.15.167:0/4078798006 <== osd.1 v2:172.21.15.167:6800/30035 26 ==== osd_op_reply(76 zone_names.default [read 0~46 out=46b] v0'0 uv1 ondisk = 0) v8 ==== 162+0+46 (crc 0 0 0) 0x7f583803f040 con 0x563bbf74b930 2023-10-05T19:28:57.217 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.210+0000 7f583fe1ae40 20 rados_obj.operate() r=0 bl.length=46 2023-10-05T19:28:57.217 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.210+0000 7f583fe1ae40 20 rados->read ofs=0 len=0 2023-10-05T19:28:57.217 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.210+0000 7f583fe1ae40 1 -- 172.21.15.167:0/4078798006 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:77 5.1d 5:b8374eb4:::zone_info.9a9df965-6612-45d4-a45f-6e3c889bacc2:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32) v8 -- 0x563bbf783910 con 0x563bbf74b930 2023-10-05T19:28:57.217 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.210+0000 7f583e150700 1 -- 172.21.15.167:0/4078798006 <== osd.1 v2:172.21.15.167:6800/30035 27 ==== osd_op_reply(77 zone_info.9a9df965-6612-45d4-a45f-6e3c889bacc2 [read 0~874 out=874b] v0'0 uv2 ondisk = 0) v8 ==== 190+0+874 (crc 0 0 0) 0x7f583803f040 con 0x563bbf74b930 2023-10-05T19:28:57.217 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.210+0000 7f583fe1ae40 20 rados_obj.operate() r=0 bl.length=874 2023-10-05T19:28:57.217 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.210+0000 7f583fe1ae40 20 rados->read ofs=0 len=0 2023-10-05T19:28:57.217 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.210+0000 7f583fe1ae40 1 -- 172.21.15.167:0/4078798006 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:78 5.f 5:f4c53578:::zonegroups_names.default:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32) v8 -- 0x563bbf77f5e0 con 0x563bbf74b930 2023-10-05T19:28:57.218 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.214+0000 7f583e150700 1 -- 172.21.15.167:0/4078798006 <== osd.1 v2:172.21.15.167:6800/30035 28 ==== osd_op_reply(78 zonegroups_names.default [read 0~46 out=46b] v0'0 uv2 ondisk = 0) v8 ==== 168+0+46 (crc 0 0 0) 0x7f583803f040 con 0x563bbf74b930 2023-10-05T19:28:57.219 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.214+0000 7f583fe1ae40 20 rados_obj.operate() r=0 bl.length=46 2023-10-05T19:28:57.219 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.214+0000 7f583fe1ae40 20 rados->read ofs=0 len=0 2023-10-05T19:28:57.219 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.214+0000 7f583fe1ae40 1 -- 172.21.15.167:0/4078798006 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:79 5.c 5:30d299a5:::zonegroup_info.4f1f719d-9648-417b-b10c-e5d939825419:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32) v8 -- 0x563bbf77f5c0 con 0x7f582800d8e0 2023-10-05T19:28:57.219 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.214+0000 7f583e150700 1 -- 172.21.15.167:0/4078798006 <== osd.0 v2:172.21.15.118:6800/27690 20 ==== osd_op_reply(79 zonegroup_info.4f1f719d-9648-417b-b10c-e5d939825419 [read 0~362 out=362b] v0'0 uv1 ondisk = 0) v8 ==== 195+0+362 (crc 0 0 0) 0x7f583803f040 con 0x7f582800d8e0 2023-10-05T19:28:57.219 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.214+0000 7f583fe1ae40 20 rados_obj.operate() r=0 bl.length=362 2023-10-05T19:28:57.219 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.214+0000 7f583fe1ae40 20 rados->read ofs=0 len=0 2023-10-05T19:28:57.219 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.214+0000 7f583fe1ae40 1 -- 172.21.15.167:0/4078798006 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:80 5.3 5:c0ad8cbd:::region_map:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32) v8 -- 0x563bbf783e50 con 0x7f582800d8e0 2023-10-05T19:28:57.220 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.214+0000 7f583e150700 1 -- 172.21.15.167:0/4078798006 <== osd.0 v2:172.21.15.118:6800/27690 21 ==== osd_op_reply(80 region_map [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 ==== 154+0+0 (crc 0 0 0) 0x7f583803f040 con 0x7f582800d8e0 2023-10-05T19:28:57.220 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.214+0000 7f583fe1ae40 20 rados_obj.operate() r=-2 bl.length=0 2023-10-05T19:28:57.220 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.214+0000 7f583fe1ae40 20 zone default found 2023-10-05T19:28:57.220 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.214+0000 7f583fe1ae40 4 Realm: () 2023-10-05T19:28:57.221 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.214+0000 7f583fe1ae40 4 ZoneGroup: default (4f1f719d-9648-417b-b10c-e5d939825419) 2023-10-05T19:28:57.221 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.214+0000 7f583fe1ae40 4 Zone: default (9a9df965-6612-45d4-a45f-6e3c889bacc2) 2023-10-05T19:28:57.221 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.214+0000 7f583fe1ae40 10 cannot find current period zonegroup using local zonegroup configuration 2023-10-05T19:28:57.221 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.214+0000 7f583fe1ae40 20 zonegroup default 2023-10-05T19:28:57.221 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.214+0000 7f583fe1ae40 20 rados->read ofs=0 len=0 2023-10-05T19:28:57.221 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.214+0000 7f583fe1ae40 1 -- 172.21.15.167:0/4078798006 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:81 5.3 5:c52100b6:::period_config.default:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32) v8 -- 0x563bbf783e50 con 0x7f582800d8e0 2023-10-05T19:28:57.221 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.214+0000 7f583e150700 1 -- 172.21.15.167:0/4078798006 <== osd.0 v2:172.21.15.118:6800/27690 22 ==== osd_op_reply(81 period_config.default [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 ==== 165+0+0 (crc 0 0 0) 0x7f583803f040 con 0x7f582800d8e0 2023-10-05T19:28:57.221 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.214+0000 7f583fe1ae40 20 rados_obj.operate() r=-2 bl.length=0 2023-10-05T19:28:57.221 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.214+0000 7f583fe1ae40 20 rados->read ofs=0 len=0 2023-10-05T19:28:57.221 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.214+0000 7f583fe1ae40 1 -- 172.21.15.167:0/4078798006 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:82 6.2 6:4347d321:::bucket.sync-source-hints.:head [call version.read in=11b,read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32) v8 -- 0x563bbf785780 con 0x563bbf7803d0 2023-10-05T19:28:57.222 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.214+0000 7f583d94f700 1 -- 172.21.15.167:0/4078798006 <== osd.3 v2:172.21.15.167:6808/30036 20 ==== osd_op_reply(82 bucket.sync-source-hints. [call,read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 ==== 211+0+0 (crc 0 0 0) 0x7f5830050040 con 0x563bbf7803d0 2023-10-05T19:28:57.222 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.214+0000 7f583fe1ae40 20 rados_obj.operate() r=-2 bl.length=0 2023-10-05T19:28:57.222 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.214+0000 7f583fe1ae40 20 rados->read ofs=0 len=0 2023-10-05T19:28:57.223 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.214+0000 7f583fe1ae40 1 -- 172.21.15.167:0/4078798006 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:83 6.b 6:d467b91b:::bucket.sync-target-hints.:head [call version.read in=11b,read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32) v8 -- 0x563bbf785b30 con 0x563bbf7803d0 2023-10-05T19:28:57.223 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.218+0000 7f583d94f700 1 -- 172.21.15.167:0/4078798006 <== osd.3 v2:172.21.15.167:6808/30036 21 ==== osd_op_reply(83 bucket.sync-target-hints. [call,read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 ==== 211+0+0 (crc 0 0 0) 0x7f5830050040 con 0x563bbf7803d0 2023-10-05T19:28:57.223 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.218+0000 7f583fe1ae40 20 rados_obj.operate() r=-2 bl.length=0 2023-10-05T19:28:57.223 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.218+0000 7f583fe1ae40 20 started sync module instance, tier type = 2023-10-05T19:28:57.223 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.218+0000 7f583fe1ae40 20 started zone id=9a9df965-6612-45d4-a45f-6e3c889bacc2 (name=default) with tier type = 2023-10-05T19:28:57.223 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.218+0000 7f583fe1ae40 1 -- 172.21.15.167:0/4078798006 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:84 7.1f 7:f95f44c2:::notify.0:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e32) v8 -- 0x563bbf784e40 con 0x7f5830007030 2023-10-05T19:28:57.225 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.218+0000 7f583d14e700 1 -- 172.21.15.167:0/4078798006 <== osd.2 v2:172.21.15.167:6816/30037 13 ==== osd_op_reply(84 notify.0 [create] v32'210 uv210 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f5828006070 con 0x7f5830007030 2023-10-05T19:28:57.225 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.218+0000 7f583fe1ae40 1 -- 172.21.15.167:0/4078798006 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:85 7.1f 7:f95f44c2:::notify.0:head [watch watch cookie 94814615390544] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e32) v8 -- 0x563bbf785ee0 con 0x7f5830007030 2023-10-05T19:28:57.225 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.218+0000 7f583fe1ae40 1 -- 172.21.15.167:0/4078798006 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:86 7.0 7:05bf5b68:::notify.1:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e32) v8 -- 0x563bbf786780 con 0x563bbf74b930 2023-10-05T19:28:57.228 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.222+0000 7f583d14e700 1 -- 172.21.15.167:0/4078798006 <== osd.2 v2:172.21.15.167:6816/30037 14 ==== osd_op_reply(85 notify.0 [watch watch cookie 94814615390544] v32'211 uv210 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f5828006070 con 0x7f5830007030 2023-10-05T19:28:57.228 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.222+0000 7f583e150700 1 -- 172.21.15.167:0/4078798006 <== osd.1 v2:172.21.15.167:6800/30035 29 ==== osd_op_reply(86 notify.1 [create] v32'210 uv210 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f583803f040 con 0x563bbf74b930 2023-10-05T19:28:57.228 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.222+0000 7f583fe1ae40 1 -- 172.21.15.167:0/4078798006 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:87 7.0 7:05bf5b68:::notify.1:head [watch watch cookie 94814615397008] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e32) v8 -- 0x563bbf786bd0 con 0x563bbf74b930 2023-10-05T19:28:57.228 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.222+0000 7f583fe1ae40 1 -- 172.21.15.167:0/4078798006 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:88 7.15 7:a93a5511:::notify.2:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e32) v8 -- 0x563bbf787470 con 0x563bbf7803d0 2023-10-05T19:28:57.231 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.226+0000 7f583e150700 1 -- 172.21.15.167:0/4078798006 <== osd.1 v2:172.21.15.167:6800/30035 30 ==== osd_op_reply(87 notify.1 [watch watch cookie 94814615397008] v32'211 uv210 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f583803f040 con 0x563bbf74b930 2023-10-05T19:28:57.232 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.226+0000 7f583d94f700 1 -- 172.21.15.167:0/4078798006 <== osd.3 v2:172.21.15.167:6808/30036 22 ==== osd_op_reply(88 notify.2 [create] v32'210 uv210 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f5830050040 con 0x563bbf7803d0 2023-10-05T19:28:57.232 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.226+0000 7f583fe1ae40 1 -- 172.21.15.167:0/4078798006 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:89 7.15 7:a93a5511:::notify.2:head [watch watch cookie 94814615398112] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e32) v8 -- 0x563bbf787020 con 0x563bbf7803d0 2023-10-05T19:28:57.232 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.226+0000 7f583fe1ae40 1 -- 172.21.15.167:0/4078798006 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:90 7.e 7:7759931f:::notify.3:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e32) v8 -- 0x563bbf787eb0 con 0x7f5830007030 2023-10-05T19:28:57.235 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.230+0000 7f583d14e700 1 -- 172.21.15.167:0/4078798006 <== osd.2 v2:172.21.15.167:6816/30037 15 ==== osd_op_reply(90 notify.3 [create] v32'210 uv210 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f5828006070 con 0x7f5830007030 2023-10-05T19:28:57.235 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.230+0000 7f583fe1ae40 1 -- 172.21.15.167:0/4078798006 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:91 7.e 7:7759931f:::notify.3:head [watch watch cookie 94814615405232] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e32) v8 -- 0x563bbf788bf0 con 0x7f5830007030 2023-10-05T19:28:57.235 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.230+0000 7f583fe1ae40 1 -- 172.21.15.167:0/4078798006 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:92 7.d 7:b4812045:::notify.4:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e32) v8 -- 0x563bbf789600 con 0x563bbf74b930 2023-10-05T19:28:57.235 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.230+0000 7f583d94f700 1 -- 172.21.15.167:0/4078798006 <== osd.3 v2:172.21.15.167:6808/30036 23 ==== osd_op_reply(89 notify.2 [watch watch cookie 94814615398112] v32'211 uv210 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f5830050040 con 0x563bbf7803d0 2023-10-05T19:28:57.237 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.230+0000 7f583d14e700 1 -- 172.21.15.167:0/4078798006 <== osd.2 v2:172.21.15.167:6816/30037 16 ==== osd_op_reply(91 notify.3 [watch watch cookie 94814615405232] v32'211 uv210 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f5828006070 con 0x7f5830007030 2023-10-05T19:28:57.237 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.230+0000 7f583e150700 1 -- 172.21.15.167:0/4078798006 <== osd.1 v2:172.21.15.167:6800/30035 31 ==== osd_op_reply(92 notify.4 [create] v32'210 uv210 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f583803f040 con 0x563bbf74b930 2023-10-05T19:28:57.237 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.230+0000 7f583fe1ae40 1 -- 172.21.15.167:0/4078798006 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:93 7.d 7:b4812045:::notify.4:head [watch watch cookie 94814615399216] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e32) v8 -- 0x563bbf789600 con 0x563bbf74b930 2023-10-05T19:28:57.237 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.230+0000 7f583fe1ae40 1 -- 172.21.15.167:0/4078798006 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:94 7.3 7:c609908c:::notify.5:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e32) v8 -- 0x563bbf789ea0 con 0x7f582800d8e0 2023-10-05T19:28:57.240 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.234+0000 7f583e150700 1 -- 172.21.15.167:0/4078798006 <== osd.1 v2:172.21.15.167:6800/30035 32 ==== osd_op_reply(93 notify.4 [watch watch cookie 94814615399216] v32'211 uv210 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f583803f040 con 0x563bbf74b930 2023-10-05T19:28:57.240 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.234+0000 7f583e150700 1 -- 172.21.15.167:0/4078798006 <== osd.0 v2:172.21.15.118:6800/27690 23 ==== osd_op_reply(94 notify.5 [create] v32'210 uv210 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f583803f040 con 0x7f582800d8e0 2023-10-05T19:28:57.240 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.234+0000 7f583fe1ae40 1 -- 172.21.15.167:0/4078798006 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:95 7.3 7:c609908c:::notify.5:head [watch watch cookie 94814615409184] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e32) v8 -- 0x563bbf789b60 con 0x7f582800d8e0 2023-10-05T19:28:57.240 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.234+0000 7f583fe1ae40 1 -- 172.21.15.167:0/4078798006 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:96 7.14 7:2b04a3e9:::notify.6:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e32) v8 -- 0x563bbf78a660 con 0x7f582800d8e0 2023-10-05T19:28:57.243 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.238+0000 7f583e150700 1 -- 172.21.15.167:0/4078798006 <== osd.0 v2:172.21.15.118:6800/27690 24 ==== osd_op_reply(95 notify.5 [watch watch cookie 94814615409184] v32'211 uv210 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f583803f040 con 0x7f582800d8e0 2023-10-05T19:28:57.243 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.238+0000 7f583e150700 1 -- 172.21.15.167:0/4078798006 <== osd.0 v2:172.21.15.118:6800/27690 25 ==== osd_op_reply(96 notify.6 [create] v32'210 uv210 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f583803f040 con 0x7f582800d8e0 2023-10-05T19:28:57.243 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.238+0000 7f583fe1ae40 1 -- 172.21.15.167:0/4078798006 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:97 7.14 7:2b04a3e9:::notify.6:head [watch watch cookie 94814615410288] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e32) v8 -- 0x563bbf789b00 con 0x7f582800d8e0 2023-10-05T19:28:57.243 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.238+0000 7f583fe1ae40 1 -- 172.21.15.167:0/4078798006 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:98 7.9 7:93e5b521:::notify.7:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e32) v8 -- 0x563bbf78afb0 con 0x7f582800d8e0 2023-10-05T19:28:57.245 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.238+0000 7f583e150700 1 -- 172.21.15.167:0/4078798006 <== osd.0 v2:172.21.15.118:6800/27690 26 ==== osd_op_reply(97 notify.6 [watch watch cookie 94814615410288] v32'211 uv210 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f583803f040 con 0x7f582800d8e0 2023-10-05T19:28:57.246 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.238+0000 7f583e150700 1 -- 172.21.15.167:0/4078798006 <== osd.0 v2:172.21.15.118:6800/27690 27 ==== osd_op_reply(98 notify.7 [create] v32'210 uv210 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f583803f040 con 0x7f582800d8e0 2023-10-05T19:28:57.246 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.238+0000 7f583fe1ae40 1 -- 172.21.15.167:0/4078798006 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:99 7.9 7:93e5b521:::notify.7:head [watch watch cookie 94814615413664] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e32) v8 -- 0x563bbf78afb0 con 0x7f582800d8e0 2023-10-05T19:28:57.246 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.238+0000 7f583fe1ae40 20 add_watcher() i=0 2023-10-05T19:28:57.246 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.242+0000 7f583fe1ae40 20 add_watcher() i=1 2023-10-05T19:28:57.246 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.242+0000 7f583fe1ae40 20 add_watcher() i=2 2023-10-05T19:28:57.246 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.242+0000 7f583fe1ae40 20 add_watcher() i=3 2023-10-05T19:28:57.246 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.242+0000 7f583fe1ae40 20 add_watcher() i=4 2023-10-05T19:28:57.246 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.242+0000 7f583fe1ae40 20 add_watcher() i=5 2023-10-05T19:28:57.246 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.242+0000 7f583fe1ae40 20 add_watcher() i=6 2023-10-05T19:28:57.247 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.242+0000 7f583e150700 1 -- 172.21.15.167:0/4078798006 <== osd.0 v2:172.21.15.118:6800/27690 28 ==== osd_op_reply(99 notify.7 [watch watch cookie 94814615413664] v32'211 uv210 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f583803f040 con 0x7f582800d8e0 2023-10-05T19:28:57.248 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.242+0000 7f583fe1ae40 20 add_watcher() i=7 2023-10-05T19:28:57.248 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.242+0000 7f583fe1ae40 2 all 8 watchers are set, enabling cache 2023-10-05T19:28:57.248 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.242+0000 7f583fe1ae40 1 -- 172.21.15.167:0/4078798006 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:100 6.4 6:22d26bf9:::data_loggenerations_metadata:head [call version.check_conds in=74b,call version.read in=11b,read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32) v8 -- 0x563bbf78e260 con 0x563bbf74b930 2023-10-05T19:28:57.248 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.242+0000 7f583e150700 1 -- 172.21.15.167:0/4078798006 <== osd.1 v2:172.21.15.167:6800/30035 33 ==== osd_op_reply(100 data_loggenerations_metadata [call,call out=48b,read 0~28 out=28b] v0'0 uv1 ondisk = 0) v8 ==== 256+0+76 (crc 0 0 0) 0x7f583803f040 con 0x563bbf74b930 2023-10-05T19:28:57.248 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.242+0000 7f583fe1ae40 1 -- 172.21.15.167:0/4078798006 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:101 6.4 6:22d26bf9:::data_loggenerations_metadata:head [watch watch cookie 94814615415856] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e32) v8 -- 0x563bbf78b570 con 0x563bbf74b930 2023-10-05T19:28:57.251 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.246+0000 7f583e150700 1 -- 172.21.15.167:0/4078798006 <== osd.1 v2:172.21.15.167:6800/30035 34 ==== osd_op_reply(101 data_loggenerations_metadata [watch watch cookie 94814615415856] v32'210 uv1 ondisk = 0) v8 ==== 172+0+0 (crc 0 0 0) 0x7f583803f040 con 0x563bbf74b930 2023-10-05T19:28:57.251 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.246+0000 7f580e7ec700 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2023-10-05T19:28:57.251 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.246+0000 7f583fe1ae40 20 check_secure_mon_conn(): auth registy supported: methods=[2] modes=[2,1] 2023-10-05T19:28:57.251 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.246+0000 7f583fe1ae40 20 check_secure_mon_conn(): mode 1 is insecure 2023-10-05T19:28:57.252 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.246+0000 7f583fe1ae40 1 Processor -- start 2023-10-05T19:28:57.252 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.246+0000 7f583fe1ae40 1 -- start start 2023-10-05T19:28:57.252 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.246+0000 7f583fe1ae40 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x563bbf798a40 0x563bbf798e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-05T19:28:57.252 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.246+0000 7f583fe1ae40 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x563bbf799350 con 0x563bbf798a40 2023-10-05T19:28:57.252 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.246+0000 7f583d14e700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x563bbf798a40 0x563bbf798e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-05T19:28:57.253 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.246+0000 7f583d14e700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x563bbf798a40 0x563bbf798e10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.118:3300/0 says I am v2:172.21.15.167:53468/0 (socket says 172.21.15.167:53468) 2023-10-05T19:28:57.253 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.246+0000 7f583d14e700 1 -- 172.21.15.167:0/864155833 learned_addr learned my addr 172.21.15.167:0/864155833 (peer_addr_for_me v2:172.21.15.167:0/0) 2023-10-05T19:28:57.253 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.246+0000 7f583d14e700 1 -- 172.21.15.167:0/864155833 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x563bbf799490 con 0x563bbf798a40 2023-10-05T19:28:57.253 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.246+0000 7f583d14e700 1 --2- 172.21.15.167:0/864155833 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x563bbf798a40 0x563bbf798e10 secure :-1 s=READY pgs=558 cs=0 l=1 rev1=1 crypto rx=0x563bbf77db30 tx=0x7f5828006000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5d26fffde2465f92 server_cookie=0 in_seq=0 out_seq=0 2023-10-05T19:28:57.254 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.250+0000 7f580dfeb700 1 -- 172.21.15.167:0/864155833 <== mon.0 v2:172.21.15.118:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f5828023040 con 0x563bbf798a40 2023-10-05T19:28:57.254 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.250+0000 7f580dfeb700 1 -- 172.21.15.167:0/864155833 <== mon.0 v2:172.21.15.118:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5828024040 con 0x563bbf798a40 2023-10-05T19:28:57.254 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.250+0000 7f580dfeb700 1 -- 172.21.15.167:0/864155833 <== mon.0 v2:172.21.15.118:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f5828013040 con 0x563bbf798a40 2023-10-05T19:28:57.255 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.250+0000 7f583fe1ae40 1 -- 172.21.15.167:0/864155833 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x563bbf798a40 msgr2=0x563bbf798e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-05T19:28:57.255 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.250+0000 7f583fe1ae40 1 --2- 172.21.15.167:0/864155833 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x563bbf798a40 0x563bbf798e10 secure :-1 s=READY pgs=558 cs=0 l=1 rev1=1 crypto rx=0x563bbf77db30 tx=0x7f5828006000 comp rx=0 tx=0).stop 2023-10-05T19:28:57.255 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.250+0000 7f583fe1ae40 1 -- 172.21.15.167:0/864155833 shutdown_connections 2023-10-05T19:28:57.255 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.250+0000 7f583fe1ae40 1 --2- 172.21.15.167:0/864155833 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x563bbf798a40 0x563bbf798e10 unknown :-1 s=CLOSED pgs=558 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-05T19:28:57.255 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.250+0000 7f583fe1ae40 1 -- 172.21.15.167:0/864155833 >> 172.21.15.167:0/864155833 conn(0x563bbf793e00 msgr2=0x563bbf791f50 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-05T19:28:57.255 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.250+0000 7f583fe1ae40 1 -- 172.21.15.167:0/864155833 shutdown_connections 2023-10-05T19:28:57.256 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.250+0000 7f583fe1ae40 1 -- 172.21.15.167:0/864155833 wait complete. 2023-10-05T19:28:57.256 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.250+0000 7f583fe1ae40 1 Processor -- start 2023-10-05T19:28:57.257 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.250+0000 7f583fe1ae40 1 -- start start 2023-10-05T19:28:57.257 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.250+0000 7f583fe1ae40 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x563bbf798a40 0x563bbf79e5d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-05T19:28:57.257 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.250+0000 7f583fe1ae40 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x563bbf79f5d0 con 0x563bbf798a40 2023-10-05T19:28:57.257 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.250+0000 7f583d14e700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x563bbf798a40 0x563bbf79e5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-05T19:28:57.257 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.250+0000 7f583d14e700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x563bbf798a40 0x563bbf79e5d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.118:3300/0 says I am v2:172.21.15.167:53476/0 (socket says 172.21.15.167:53476) 2023-10-05T19:28:57.257 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.250+0000 7f583d14e700 1 -- 172.21.15.167:0/2334422951 learned_addr learned my addr 172.21.15.167:0/2334422951 (peer_addr_for_me v2:172.21.15.167:0/0) 2023-10-05T19:28:57.258 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.250+0000 7f583d14e700 1 -- 172.21.15.167:0/2334422951 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f582800ed20 con 0x563bbf798a40 2023-10-05T19:28:57.258 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.254+0000 7f583d14e700 1 --2- 172.21.15.167:0/2334422951 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x563bbf798a40 0x563bbf79e5d0 secure :-1 s=READY pgs=559 cs=0 l=1 rev1=1 crypto rx=0x563bbf77a0c0 tx=0x7f5828030000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-05T19:28:57.258 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.254+0000 7f577ffff700 1 -- 172.21.15.167:0/2334422951 <== mon.0 v2:172.21.15.118:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f5828023040 con 0x563bbf798a40 2023-10-05T19:28:57.258 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.254+0000 7f577ffff700 1 -- 172.21.15.167:0/2334422951 <== mon.0 v2:172.21.15.118:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f582803b580 con 0x563bbf798a40 2023-10-05T19:28:57.258 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.254+0000 7f583fe1ae40 1 -- 172.21.15.167:0/2334422951 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x563bbf79ebd0 con 0x563bbf798a40 2023-10-05T19:28:57.258 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.254+0000 7f577ffff700 1 -- 172.21.15.167:0/2334422951 <== mon.0 v2:172.21.15.118:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f582803b750 con 0x563bbf798a40 2023-10-05T19:28:57.259 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.254+0000 7f583fe1ae40 1 -- 172.21.15.167:0/2334422951 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x563bbf79f010 con 0x563bbf798a40 2023-10-05T19:28:57.259 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.254+0000 7f577ffff700 1 -- 172.21.15.167:0/2334422951 <== mon.0 v2:172.21.15.118:3300/0 4 ==== mgrmap(e 5) v1 ==== 40329+0+0 (secure 0 0 0) 0x7f582803b8b0 con 0x563bbf798a40 2023-10-05T19:28:57.259 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.254+0000 7f577ffff700 1 --2- 172.21.15.167:0/2334422951 >> [v2:172.21.15.167:6824/29889,v1:172.21.15.167:6825/29889] conn(0x7f5784030c50 0x7f5784033100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-05T19:28:57.260 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.254+0000 7f577ffff700 1 -- 172.21.15.167:0/2334422951 <== mon.0 v2:172.21.15.118:3300/0 5 ==== osd_map(32..32 src has 1..32) v4 ==== 6260+0+0 (secure 0 0 0) 0x7f5828044ce0 con 0x563bbf798a40 2023-10-05T19:28:57.260 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.254+0000 7f583fe1ae40 5 note: GC not initialized 2023-10-05T19:28:57.260 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.254+0000 7f583e150700 1 --2- 172.21.15.167:0/2334422951 >> [v2:172.21.15.167:6824/29889,v1:172.21.15.167:6825/29889] conn(0x7f5784030c50 0x7f5784033100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-05T19:28:57.261 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.254+0000 7f583e150700 1 --2- 172.21.15.167:0/2334422951 >> [v2:172.21.15.167:6824/29889,v1:172.21.15.167:6825/29889] conn(0x7f5784030c50 0x7f5784033100 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x563bbf78e2e0 tx=0x7f583800aec0 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-05T19:28:57.262 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.258+0000 7f577d7fa700 20 reqs_thread_entry: start 2023-10-05T19:28:57.263 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.258+0000 7f5776ffd700 20 reqs_thread_entry: start 2023-10-05T19:28:57.263 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.258+0000 7f583fe1ae40 20 init_complete bucket index max shards: 11 2023-10-05T19:28:57.263 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.258+0000 7f583fe1ae40 10 rgw notify: Started notification manager with: 1 workers 2023-10-05T19:28:57.263 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.258+0000 7f57757fa700 20 rgw notify: INFO: next queues processing will happen at: Thu Oct 5 19:29:27 2023 2023-10-05T19:28:57.264 INFO:teuthology.orchestra.run.smithi167.stderr: 2023-10-05T19:28:57.264 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.258+0000 7f5774ff9700 20 reqs_thread_entry: start 2023-10-05T19:28:57.264 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.258+0000 7f583fe1ae40 20 get_system_obj_state: rctx=0x7ffc18958788 obj=default.rgw.meta:users.uid:foo state=0x563bbf6220b0 s->prefetch_data=0 2023-10-05T19:28:57.264 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.258+0000 7f583fe1ae40 10 cache get: name=default.rgw.meta+users.uid+foo : miss 2023-10-05T19:28:57.264 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.258+0000 7f583fe1ae40 1 -- 172.21.15.167:0/4078798006 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:102 8.18 8:1e6ca5ed:users.uid::foo:head [call version.read in=11b,getxattrs,stat] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32) v8 -- 0x563bbf806e40 con 0x563bbf74b930 2023-10-05T19:28:57.265 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.258+0000 7f583e150700 1 -- 172.21.15.167:0/4078798006 <== osd.1 v2:172.21.15.167:6800/30035 35 ==== osd_op_reply(102 foo [call out=48b,getxattrs out=75b,stat out=16b] v0'0 uv13 ondisk = 0) v8 ==== 231+0+139 (crc 0 0 0) 0x7f583803f040 con 0x563bbf74b930 2023-10-05T19:28:57.265 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.258+0000 7f583fe1ae40 10 cache put: name=default.rgw.meta+users.uid+foo info.flags=0x16 2023-10-05T19:28:57.265 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.258+0000 7f583fe1ae40 10 adding default.rgw.meta+users.uid+foo to cache LRU end 2023-10-05T19:28:57.265 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.258+0000 7f583fe1ae40 10 updating xattr: name=ceph.objclass.version bl.length()=42 2023-10-05T19:28:57.265 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.258+0000 7f583fe1ae40 20 get_system_obj_state: s->obj_tag was set empty 2023-10-05T19:28:57.265 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.258+0000 7f583fe1ae40 20 Read xattr: user.rgw.idtag 2023-10-05T19:28:57.265 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.258+0000 7f583fe1ae40 10 cache get: name=default.rgw.meta+users.uid+foo : type miss (requested=0x13, cached=0x16) 2023-10-05T19:28:57.266 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.258+0000 7f583fe1ae40 20 rados->read ofs=0 len=0 2023-10-05T19:28:57.266 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.258+0000 7f583fe1ae40 1 -- 172.21.15.167:0/4078798006 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:103 8.18 8:1e6ca5ed:users.uid::foo:head [call version.check_conds in=122b,call version.read in=11b,read 0~0,getxattrs] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32) v8 -- 0x563bbf8098d0 con 0x563bbf74b930 2023-10-05T19:28:57.266 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.258+0000 7f583e150700 1 -- 172.21.15.167:0/4078798006 <== osd.1 v2:172.21.15.167:6800/30035 36 ==== osd_op_reply(103 foo [call,call out=48b,read 0~357 out=357b,getxattrs out=75b] v0'0 uv13 ondisk = 0) v8 ==== 273+0+480 (crc 0 0 0) 0x7f583803f040 con 0x563bbf74b930 2023-10-05T19:28:57.266 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.258+0000 7f583fe1ae40 20 rados_obj.operate() r=0 bl.length=357 2023-10-05T19:28:57.266 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.258+0000 7f583fe1ae40 10 cache put: name=default.rgw.meta+users.uid+foo info.flags=0x13 2023-10-05T19:28:57.266 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.258+0000 7f583fe1ae40 10 moving default.rgw.meta+users.uid+foo to cache LRU end 2023-10-05T19:28:57.266 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.258+0000 7f583fe1ae40 10 updating xattr: name=ceph.objclass.version bl.length()=42 2023-10-05T19:28:57.266 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.258+0000 7f583fe1ae40 1 -- 172.21.15.167:0/4078798006 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:104 8.18 8:1e6ca5ed:users.uid::foo:head [delete,create,call version.set in=58b,writefull 0~369 in=369b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e32) v8 -- 0x563bbf80ab50 con 0x563bbf74b930 2023-10-05T19:28:57.267 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.262+0000 7f583e150700 1 -- 172.21.15.167:0/4078798006 <== osd.1 v2:172.21.15.167:6800/30035 37 ==== osd_op_reply(104 foo [delete,create,call,writefull 0~369] v32'14 uv14 ondisk = 0) v8 ==== 273+0+0 (crc 0 0 0) 0x7f583803f040 con 0x563bbf74b930 2023-10-05T19:28:57.268 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.262+0000 7f583fe1ae40 10 cache put: name=default.rgw.meta+users.uid+foo info.flags=0x17 2023-10-05T19:28:57.268 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.262+0000 7f583fe1ae40 10 moving default.rgw.meta+users.uid+foo to cache LRU end 2023-10-05T19:28:57.268 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.262+0000 7f583fe1ae40 10 distributing notification oid=default.rgw.control:notify.4 cni=[op: 0, obj: default.rgw.meta:users.uid:foo, ofs0, ns] 2023-10-05T19:28:57.268 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.262+0000 7f583fe1ae40 1 -- 172.21.15.167:0/4078798006 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:105 7.d 7:b4812045:::notify.4:head [notify cookie 94814615938656 in=557b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32) v8 -- 0x563bbf8098d0 con 0x563bbf74b930 2023-10-05T19:28:57.269 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.262+0000 7f583e150700 1 -- 172.21.15.167:0/4078798006 <== osd.1 v2:172.21.15.167:6800/30035 38 ==== watch-notify(notify (1) cookie 94814615399216 notify 137438953491 ret 0) v3 ==== 587+0+0 (crc 0 0 0) 0x7f583804e040 con 0x563bbf74b930 2023-10-05T19:28:57.269 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.262+0000 7f58377fe700 10 rgw watcher librados: RGWWatcher::handle_notify() notify_id 137438953491 cookie 94814615399216 notifier 4647 bl.length()=545 2023-10-05T19:28:57.269 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.262+0000 7f58377fe700 10 rgw watcher librados: cache put: name=default.rgw.meta+users.uid+foo info.flags=0x17 2023-10-05T19:28:57.269 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.262+0000 7f58377fe700 10 rgw watcher librados: moving default.rgw.meta+users.uid+foo to cache LRU end 2023-10-05T19:28:57.269 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.262+0000 7f583e150700 1 -- 172.21.15.167:0/4078798006 <== osd.1 v2:172.21.15.167:6800/30035 39 ==== osd_op_reply(105 notify.4 [notify cookie 94814615938656 out=8b] v0'0 uv210 ondisk = 0) v8 ==== 152+0+8 (crc 0 0 0) 0x7f583803f040 con 0x563bbf74b930 2023-10-05T19:28:57.269 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.262+0000 7f58377fe700 1 -- 172.21.15.167:0/4078798006 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:106 7.d 7:b4812045:::notify.4:head [notify-ack in=20b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32) v8 -- 0x7f5818003d90 con 0x563bbf74b930 2023-10-05T19:28:57.269 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.262+0000 7f583e150700 1 -- 172.21.15.167:0/4078798006 <== osd.1 v2:172.21.15.167:6800/30035 40 ==== osd_op_reply(106 notify.4 [notify-ack] v0'0 uv210 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f583803f040 con 0x563bbf74b930 2023-10-05T19:28:57.269 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.262+0000 7f583e150700 1 -- 172.21.15.167:0/4078798006 <== osd.1 v2:172.21.15.167:6800/30035 41 ==== watch-notify(notify_complete (2) cookie 94814615938656 notify 137438953491 ret 0) v3 ==== 42+0+48 (crc 0 0 0) 0x7f5838036990 con 0x563bbf74b930 2023-10-05T19:28:57.270 INFO:teuthology.orchestra.run.smithi167.stdout:{ 2023-10-05T19:28:57.270 INFO:teuthology.orchestra.run.smithi167.stdout: "user_id": "foo", 2023-10-05T19:28:57.270 INFO:teuthology.orchestra.run.smithi167.stdout: "display_name": "Foo", 2023-10-05T19:28:57.270 INFO:teuthology.orchestra.run.smithi167.stdout: "email": "foo@foo.com", 2023-10-05T19:28:57.270 INFO:teuthology.orchestra.run.smithi167.stdout: "suspended": 0, 2023-10-05T19:28:57.270 INFO:teuthology.orchestra.run.smithi167.stdout: "max_buckets": 4, 2023-10-05T19:28:57.270 INFO:teuthology.orchestra.run.smithi167.stdout: "subusers": [], 2023-10-05T19:28:57.270 INFO:teuthology.orchestra.run.smithi167.stdout: "keys": [ 2023-10-05T19:28:57.270 INFO:teuthology.orchestra.run.smithi167.stdout: { 2023-10-05T19:28:57.271 INFO:teuthology.orchestra.run.smithi167.stdout: "user": "foo", 2023-10-05T19:28:57.271 INFO:teuthology.orchestra.run.smithi167.stdout: "access_key": "9te6NH5mcdcq0Tc5i8i1", 2023-10-05T19:28:57.271 INFO:teuthology.orchestra.run.smithi167.stdout: "secret_key": "Ny4IOauQoL18Gp2zM7lC1vLmoawgqcYP/YGcWfXu" 2023-10-05T19:28:57.271 INFO:teuthology.orchestra.run.smithi167.stdout: } 2023-10-05T19:28:57.271 INFO:teuthology.orchestra.run.smithi167.stdout: ], 2023-10-05T19:28:57.271 INFO:teuthology.orchestra.run.smithi167.stdout: "swift_keys": [], 2023-10-05T19:28:57.271 INFO:teuthology.orchestra.run.smithi167.stdout: "caps": [ 2023-10-05T19:28:57.271 INFO:teuthology.orchestra.run.smithi167.stdout: { 2023-10-05T19:28:57.271 INFO:teuthology.orchestra.run.smithi167.stdout: "type": "user", 2023-10-05T19:28:57.271 INFO:teuthology.orchestra.run.smithi167.stdout: "perm": "read" 2023-10-05T19:28:57.271 INFO:teuthology.orchestra.run.smithi167.stdout: } 2023-10-05T19:28:57.271 INFO:teuthology.orchestra.run.smithi167.stdout: ], 2023-10-05T19:28:57.272 INFO:teuthology.orchestra.run.smithi167.stdout: "op_mask": "read, write, delete", 2023-10-05T19:28:57.272 INFO:teuthology.orchestra.run.smithi167.stdout: "default_placement": "", 2023-10-05T19:28:57.272 INFO:teuthology.orchestra.run.smithi167.stdout: "default_storage_class": "", 2023-10-05T19:28:57.272 INFO:teuthology.orchestra.run.smithi167.stdout: "placement_tags": [], 2023-10-05T19:28:57.272 INFO:teuthology.orchestra.run.smithi167.stdout: "bucket_quota": { 2023-10-05T19:28:57.272 INFO:teuthology.orchestra.run.smithi167.stdout: "enabled": false, 2023-10-05T19:28:57.272 INFO:teuthology.orchestra.run.smithi167.stdout: "check_on_raw": false, 2023-10-05T19:28:57.272 INFO:teuthology.orchestra.run.smithi167.stdout: "max_size": -1, 2023-10-05T19:28:57.272 INFO:teuthology.orchestra.run.smithi167.stdout: "max_size_kb": 0, 2023-10-05T19:28:57.272 INFO:teuthology.orchestra.run.smithi167.stdout: "max_objects": -1 2023-10-05T19:28:57.272 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2023-10-05T19:28:57.272 INFO:teuthology.orchestra.run.smithi167.stdout: "user_quota": { 2023-10-05T19:28:57.273 INFO:teuthology.orchestra.run.smithi167.stdout: "enabled": false, 2023-10-05T19:28:57.273 INFO:teuthology.orchestra.run.smithi167.stdout: "check_on_raw": false, 2023-10-05T19:28:57.273 INFO:teuthology.orchestra.run.smithi167.stdout: "max_size": -1, 2023-10-05T19:28:57.273 INFO:teuthology.orchestra.run.smithi167.stdout: "max_size_kb": 0, 2023-10-05T19:28:57.273 INFO:teuthology.orchestra.run.smithi167.stdout: "max_objects": -1 2023-10-05T19:28:57.273 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2023-10-05T19:28:57.273 INFO:teuthology.orchestra.run.smithi167.stdout: "temp_url_keys": [], 2023-10-05T19:28:57.273 INFO:teuthology.orchestra.run.smithi167.stdout: "type": "rgw", 2023-10-05T19:28:57.273 INFO:teuthology.orchestra.run.smithi167.stdout: "mfa_ids": [] 2023-10-05T19:28:57.273 INFO:teuthology.orchestra.run.smithi167.stdout:} 2023-10-05T19:28:57.273 INFO:teuthology.orchestra.run.smithi167.stdout: 2023-10-05T19:28:57.274 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.266+0000 7f583fe1ae40 1 -- 172.21.15.167:0/4078798006 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:107 7.1f 7:f95f44c2:::notify.0:head [watch unwatch cookie 94814615390544] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e32) v8 -- 0x563bbf80ab50 con 0x7f5830007030 2023-10-05T19:28:57.274 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.266+0000 7f583d14e700 1 -- 172.21.15.167:0/4078798006 <== osd.2 v2:172.21.15.167:6816/30037 17 ==== osd_op_reply(107 notify.0 [watch unwatch cookie 94814615390544] v32'212 uv210 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f5828006070 con 0x7f5830007030 2023-10-05T19:28:57.274 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.266+0000 7f583fe1ae40 20 remove_watcher() i=0 2023-10-05T19:28:57.274 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.266+0000 7f583fe1ae40 2 removed watcher, disabling cache 2023-10-05T19:28:57.274 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.266+0000 7f583fe1ae40 1 -- 172.21.15.167:0/4078798006 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:108 7.0 7:05bf5b68:::notify.1:head [watch unwatch cookie 94814615397008] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e32) v8 -- 0x563bbf760810 con 0x563bbf74b930 2023-10-05T19:28:57.275 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.270+0000 7f583e150700 1 -- 172.21.15.167:0/4078798006 <== osd.1 v2:172.21.15.167:6800/30035 42 ==== osd_op_reply(108 notify.1 [watch unwatch cookie 94814615397008] v32'212 uv210 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f583803f040 con 0x563bbf74b930 2023-10-05T19:28:57.275 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.270+0000 7f583fe1ae40 20 remove_watcher() i=1 2023-10-05T19:28:57.275 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.270+0000 7f583fe1ae40 1 -- 172.21.15.167:0/4078798006 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:109 7.15 7:a93a5511:::notify.2:head [watch unwatch cookie 94814615398112] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e32) v8 -- 0x563bbf760810 con 0x563bbf7803d0 2023-10-05T19:28:57.278 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.274+0000 7f583d94f700 1 -- 172.21.15.167:0/4078798006 <== osd.3 v2:172.21.15.167:6808/30036 24 ==== osd_op_reply(109 notify.2 [watch unwatch cookie 94814615398112] v32'212 uv210 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f5830050040 con 0x563bbf7803d0 2023-10-05T19:28:57.278 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.274+0000 7f583fe1ae40 20 remove_watcher() i=2 2023-10-05T19:28:57.278 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.274+0000 7f583fe1ae40 1 -- 172.21.15.167:0/4078798006 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:110 7.e 7:7759931f:::notify.3:head [watch unwatch cookie 94814615405232] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e32) v8 -- 0x563bbf760810 con 0x7f5830007030 2023-10-05T19:28:57.280 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.274+0000 7f583d14e700 1 -- 172.21.15.167:0/4078798006 <== osd.2 v2:172.21.15.167:6816/30037 18 ==== osd_op_reply(110 notify.3 [watch unwatch cookie 94814615405232] v32'212 uv210 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f5828006070 con 0x7f5830007030 2023-10-05T19:28:57.281 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.274+0000 7f583fe1ae40 20 remove_watcher() i=3 2023-10-05T19:28:57.281 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.274+0000 7f583fe1ae40 1 -- 172.21.15.167:0/4078798006 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:111 7.d 7:b4812045:::notify.4:head [watch unwatch cookie 94814615399216] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e32) v8 -- 0x563bbf760810 con 0x563bbf74b930 2023-10-05T19:28:57.283 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.278+0000 7f583e150700 1 -- 172.21.15.167:0/4078798006 <== osd.1 v2:172.21.15.167:6800/30035 43 ==== osd_op_reply(111 notify.4 [watch unwatch cookie 94814615399216] v32'212 uv210 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f583803f040 con 0x563bbf74b930 2023-10-05T19:28:57.283 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.278+0000 7f583fe1ae40 20 remove_watcher() i=4 2023-10-05T19:28:57.283 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.278+0000 7f583fe1ae40 1 -- 172.21.15.167:0/4078798006 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:112 7.3 7:c609908c:::notify.5:head [watch unwatch cookie 94814615409184] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e32) v8 -- 0x563bbf760810 con 0x7f582800d8e0 2023-10-05T19:28:57.286 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.282+0000 7f583e150700 1 -- 172.21.15.167:0/4078798006 <== osd.0 v2:172.21.15.118:6800/27690 29 ==== osd_op_reply(112 notify.5 [watch unwatch cookie 94814615409184] v32'212 uv210 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f583803f040 con 0x7f582800d8e0 2023-10-05T19:28:57.286 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.282+0000 7f583fe1ae40 20 remove_watcher() i=5 2023-10-05T19:28:57.286 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.282+0000 7f583fe1ae40 1 -- 172.21.15.167:0/4078798006 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:113 7.14 7:2b04a3e9:::notify.6:head [watch unwatch cookie 94814615410288] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e32) v8 -- 0x563bbf760810 con 0x7f582800d8e0 2023-10-05T19:28:57.289 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.282+0000 7f583e150700 1 -- 172.21.15.167:0/4078798006 <== osd.0 v2:172.21.15.118:6800/27690 30 ==== osd_op_reply(113 notify.6 [watch unwatch cookie 94814615410288] v32'212 uv210 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f583803f040 con 0x7f582800d8e0 2023-10-05T19:28:57.289 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.282+0000 7f583fe1ae40 20 remove_watcher() i=6 2023-10-05T19:28:57.289 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.282+0000 7f583fe1ae40 1 -- 172.21.15.167:0/4078798006 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:114 7.9 7:93e5b521:::notify.7:head [watch unwatch cookie 94814615413664] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e32) v8 -- 0x563bbf789710 con 0x7f582800d8e0 2023-10-05T19:28:57.292 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.286+0000 7f583e150700 1 -- 172.21.15.167:0/4078798006 <== osd.0 v2:172.21.15.118:6800/27690 31 ==== osd_op_reply(114 notify.7 [watch unwatch cookie 94814615413664] v32'212 uv210 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f583803f040 con 0x7f582800d8e0 2023-10-05T19:28:57.292 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.286+0000 7f583fe1ae40 20 remove_watcher() i=7 2023-10-05T19:28:57.294 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.290+0000 7f580e7ec700 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2023-10-05T19:28:57.294 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.290+0000 7f583fe1ae40 1 -- 172.21.15.167:0/4078798006 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:115 6.4 6:22d26bf9:::data_loggenerations_metadata:head [watch unwatch cookie 94814615415856] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e32) v8 -- 0x563bbf760810 con 0x563bbf74b930 2023-10-05T19:28:57.298 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.290+0000 7f583e150700 1 -- 172.21.15.167:0/4078798006 <== osd.1 v2:172.21.15.167:6800/30035 44 ==== osd_op_reply(115 data_loggenerations_metadata [watch unwatch cookie 94814615415856] v32'211 uv1 ondisk = 0) v8 ==== 172+0+0 (crc 0 0 0) 0x7f583803f040 con 0x563bbf74b930 2023-10-05T19:28:57.298 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.290+0000 7f583fe1ae40 1 -- 172.21.15.167:0/4078798006 >> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] conn(0x7f582800d8e0 msgr2=0x7f582800c360 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-05T19:28:57.298 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.290+0000 7f583fe1ae40 1 --2- 172.21.15.167:0/4078798006 >> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] conn(0x7f582800d8e0 0x7f582800c360 crc :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-05T19:28:57.298 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.290+0000 7f583fe1ae40 1 -- 172.21.15.167:0/4078798006 >> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] conn(0x563bbf74b930 msgr2=0x563bbf77c630 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-05T19:28:57.298 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.290+0000 7f583fe1ae40 1 --2- 172.21.15.167:0/4078798006 >> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] conn(0x563bbf74b930 0x563bbf77c630 crc :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-05T19:28:57.298 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.290+0000 7f583fe1ae40 1 -- 172.21.15.167:0/4078798006 >> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] conn(0x7f5830007030 msgr2=0x7f58300245b0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-05T19:28:57.298 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.290+0000 7f583fe1ae40 1 --2- 172.21.15.167:0/4078798006 >> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] conn(0x7f5830007030 0x7f58300245b0 crc :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-05T19:28:57.298 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.290+0000 7f583fe1ae40 1 -- 172.21.15.167:0/4078798006 >> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] conn(0x563bbf7803d0 msgr2=0x563bbf7827b0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-05T19:28:57.299 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.290+0000 7f583fe1ae40 1 --2- 172.21.15.167:0/4078798006 >> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] conn(0x563bbf7803d0 0x563bbf7827b0 crc :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-05T19:28:57.299 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.294+0000 7f583fe1ae40 1 -- 172.21.15.167:0/4078798006 >> [v2:172.21.15.167:6824/29889,v1:172.21.15.167:6825/29889] conn(0x7f5824030c50 msgr2=0x7f5824033100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-05T19:28:57.299 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.294+0000 7f583fe1ae40 1 --2- 172.21.15.167:0/4078798006 >> [v2:172.21.15.167:6824/29889,v1:172.21.15.167:6825/29889] conn(0x7f5824030c50 0x7f5824033100 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7f58280011e0 tx=0x7f58280013a0 comp rx=0 tx=0).stop 2023-10-05T19:28:57.299 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.294+0000 7f583e150700 1 -- 172.21.15.167:0/4078798006 reap_dead start 2023-10-05T19:28:57.299 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.294+0000 7f583fe1ae40 1 -- 172.21.15.167:0/4078798006 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x563bbf74b020 msgr2=0x563bbf770b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-05T19:28:57.299 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.294+0000 7f583fe1ae40 1 --2- 172.21.15.167:0/4078798006 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x563bbf74b020 0x563bbf770b90 secure :-1 s=READY pgs=557 cs=0 l=1 rev1=1 crypto rx=0x7f583001e040 tx=0x7f583001c000 comp rx=0 tx=0).stop 2023-10-05T19:28:57.299 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.294+0000 7f583fe1ae40 1 -- 172.21.15.167:0/4078798006 shutdown_connections 2023-10-05T19:28:57.299 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.294+0000 7f583fe1ae40 1 --2- 172.21.15.167:0/4078798006 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x563bbf74b020 0x563bbf770b90 unknown :-1 s=CLOSED pgs=557 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-05T19:28:57.299 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.294+0000 7f583fe1ae40 1 -- 172.21.15.167:0/4078798006 >> 172.21.15.167:0/4078798006 conn(0x563bbf6be870 msgr2=0x563bbf764570 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-05T19:28:57.299 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.294+0000 7f583fe1ae40 1 -- 172.21.15.167:0/4078798006 shutdown_connections 2023-10-05T19:28:57.299 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.294+0000 7f583fe1ae40 1 -- 172.21.15.167:0/4078798006 wait complete. 2023-10-05T19:28:57.300 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.294+0000 7f583fe1ae40 1 -- 172.21.15.167:0/2334422951 >> [v2:172.21.15.167:6824/29889,v1:172.21.15.167:6825/29889] conn(0x7f5784030c50 msgr2=0x7f5784033100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-05T19:28:57.300 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.294+0000 7f583fe1ae40 1 --2- 172.21.15.167:0/2334422951 >> [v2:172.21.15.167:6824/29889,v1:172.21.15.167:6825/29889] conn(0x7f5784030c50 0x7f5784033100 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x563bbf78e2e0 tx=0x7f583800aec0 comp rx=0 tx=0).stop 2023-10-05T19:28:57.301 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.294+0000 7f583fe1ae40 1 -- 172.21.15.167:0/2334422951 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x563bbf798a40 msgr2=0x563bbf79e5d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-05T19:28:57.301 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.294+0000 7f583fe1ae40 1 --2- 172.21.15.167:0/2334422951 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x563bbf798a40 0x563bbf79e5d0 secure :-1 s=READY pgs=559 cs=0 l=1 rev1=1 crypto rx=0x563bbf77a0c0 tx=0x7f5828030000 comp rx=0 tx=0).stop 2023-10-05T19:28:57.301 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.294+0000 7f583fe1ae40 1 -- 172.21.15.167:0/2334422951 shutdown_connections 2023-10-05T19:28:57.301 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.294+0000 7f583fe1ae40 1 --2- 172.21.15.167:0/2334422951 >> [v2:172.21.15.167:6824/29889,v1:172.21.15.167:6825/29889] conn(0x7f5784030c50 0x7f5784033100 unknown :-1 s=CLOSED pgs=193 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-05T19:28:57.301 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.294+0000 7f583fe1ae40 1 --2- 172.21.15.167:0/2334422951 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x563bbf798a40 0x563bbf79e5d0 unknown :-1 s=CLOSED pgs=559 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-05T19:28:57.301 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.294+0000 7f583fe1ae40 1 -- 172.21.15.167:0/2334422951 >> 172.21.15.167:0/2334422951 conn(0x563bbf793e00 msgr2=0x563bbf791ea0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-05T19:28:57.301 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.294+0000 7f583fe1ae40 1 -- 172.21.15.167:0/2334422951 shutdown_connections 2023-10-05T19:28:57.301 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.294+0000 7f583fe1ae40 1 -- 172.21.15.167:0/2334422951 wait complete. 2023-10-05T19:28:57.304 DEBUG:tasks.util.rgw: json result: {'user_id': 'foo', 'display_name': 'Foo', 'email': 'foo@foo.com', 'suspended': 0, 'max_buckets': 4, 'subusers': [], 'keys': [{'user': 'foo', 'access_key': '9te6NH5mcdcq0Tc5i8i1', 'secret_key': 'Ny4IOauQoL18Gp2zM7lC1vLmoawgqcYP/YGcWfXu'}], 'swift_keys': [], 'caps': [{'type': 'user', 'perm': 'read'}], 'op_mask': 'read, write, delete', 'default_placement': '', 'default_storage_class': '', 'placement_tags': [], 'bucket_quota': {'enabled': False, 'check_on_raw': False, 'max_size': -1, 'max_size_kb': 0, 'max_objects': -1}, 'user_quota': {'enabled': False, 'check_on_raw': False, 'max_size': -1, 'max_size_kb': 0, 'max_objects': -1}, 'temp_url_keys': [], 'type': 'rgw', 'mfa_ids': []} 2023-10-05T19:28:57.304 INFO:tasks.util.rgw:rgwadmin: client.0 : ['caps', 'rm', '--uid', 'foo', '--caps', 'user=read'] 2023-10-05T19:28:57.304 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'ceph', 'caps', 'rm', '--uid', 'foo', '--caps', 'user=read'] 2023-10-05T19:28:57.304 DEBUG:teuthology.orchestra.run.smithi167:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster ceph caps rm --uid foo --caps user=read 2023-10-05T19:28:57.345 INFO:teuthology.orchestra.run.smithi167.stderr:ignoring --setuser ceph since I am not root 2023-10-05T19:28:57.346 INFO:teuthology.orchestra.run.smithi167.stderr:ignoring --setgroup ceph since I am not root 2023-10-05T19:28:57.346 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.342+0000 7fcaa91fee40 1 Processor -- start 2023-10-05T19:28:57.346 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.342+0000 7fcaa91fee40 1 -- start start 2023-10-05T19:28:57.347 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.342+0000 7fcaa91fee40 1 --2- >> v2:172.21.15.118:3300/0 conn(0x563014413420 0x563014415800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-05T19:28:57.347 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.342+0000 7fcaa91fee40 1 -- --> v1:172.21.15.118:6789/0 -- auth(proto 0 26 bytes epoch 0) v1 -- 0x56301435bbd0 con 0x56301440ee40 2023-10-05T19:28:57.347 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.342+0000 7fcaa91fee40 1 -- --> v2:172.21.15.118:3300/0 -- mon_getmap magic: 0 v1 -- 0x563014318ee0 con 0x563014413420 2023-10-05T19:28:57.348 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.342+0000 7fcaa6532700 1 --2- >> v2:172.21.15.118:3300/0 conn(0x563014413420 0x563014415800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-05T19:28:57.348 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.342+0000 7fcaa6d33700 1 --1- >> v1:172.21.15.118:6789/0 conn(0x56301440ee40 0x56301440f210 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.118:6789/0 says I am v1:172.21.15.167:48722/0 (socket says 172.21.15.167:48722) 2023-10-05T19:28:57.348 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.342+0000 7fcaa6d33700 1 -- 172.21.15.167:0/4018895440 learned_addr learned my addr 172.21.15.167:0/4018895440 (peer_addr_for_me v1:172.21.15.167:0/0) 2023-10-05T19:28:57.349 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.342+0000 7fcaa5d31700 1 -- 172.21.15.167:0/4018895440 <== mon.0 v1:172.21.15.118:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3366889785 0 0) 0x56301435bbd0 con 0x56301440ee40 2023-10-05T19:28:57.349 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.342+0000 7fcaa5d31700 1 -- 172.21.15.167:0/4018895440 --> v1:172.21.15.118:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fca94001e10 con 0x56301440ee40 2023-10-05T19:28:57.350 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.342+0000 7fcaa6532700 1 -- 172.21.15.167:0/4018895440 >> v1:172.21.15.118:6789/0 conn(0x56301440ee40 legacy=0x56301440f210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-05T19:28:57.350 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.342+0000 7fcaa6532700 1 -- 172.21.15.167:0/4018895440 --> v2:172.21.15.118:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x56301440fa50 con 0x563014413420 2023-10-05T19:28:57.350 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.342+0000 7fcaa6532700 1 --2- 172.21.15.167:0/4018895440 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x563014413420 0x563014415800 secure :-1 s=READY pgs=561 cs=0 l=1 rev1=1 crypto rx=0x7fca9000a5d0 tx=0x7fca90005a50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d20f717c9f8ceb72 server_cookie=0 in_seq=0 out_seq=0 2023-10-05T19:28:57.350 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.342+0000 7fcaa5d31700 1 -- 172.21.15.167:0/4018895440 <== mon.0 v2:172.21.15.118:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fca90007610 con 0x563014413420 2023-10-05T19:28:57.350 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.342+0000 7fcaa5d31700 1 -- 172.21.15.167:0/4018895440 <== mon.0 v2:172.21.15.118:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fca90007770 con 0x563014413420 2023-10-05T19:28:57.350 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.342+0000 7fcaa5d31700 1 -- 172.21.15.167:0/4018895440 <== mon.0 v2:172.21.15.118:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fca90007a20 con 0x563014413420 2023-10-05T19:28:57.350 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.342+0000 7fcaa91fee40 1 -- 172.21.15.167:0/4018895440 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x563014413420 msgr2=0x563014415800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-05T19:28:57.350 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.342+0000 7fcaa91fee40 1 --2- 172.21.15.167:0/4018895440 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x563014413420 0x563014415800 secure :-1 s=READY pgs=561 cs=0 l=1 rev1=1 crypto rx=0x7fca9000a5d0 tx=0x7fca90005a50 comp rx=0 tx=0).stop 2023-10-05T19:28:57.350 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.346+0000 7fcaa91fee40 1 -- 172.21.15.167:0/4018895440 shutdown_connections 2023-10-05T19:28:57.350 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.346+0000 7fcaa91fee40 1 --2- 172.21.15.167:0/4018895440 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x563014413420 0x563014415800 unknown :-1 s=CLOSED pgs=561 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-05T19:28:57.350 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.346+0000 7fcaa91fee40 1 -- 172.21.15.167:0/4018895440 >> 172.21.15.167:0/4018895440 conn(0x56301440a860 msgr2=0x56301440cc70 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-05T19:28:57.351 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.346+0000 7fcaa91fee40 1 -- 172.21.15.167:0/4018895440 shutdown_connections 2023-10-05T19:28:57.351 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.346+0000 7fcaa91fee40 1 -- 172.21.15.167:0/4018895440 wait complete. 2023-10-05T19:28:57.351 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.346+0000 7fcaa91fee40 1 Processor -- start 2023-10-05T19:28:57.352 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.346+0000 7fcaa91fee40 1 -- start start 2023-10-05T19:28:57.352 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.346+0000 7fcaa91fee40 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x56301440ee40 0x563014432810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-05T19:28:57.352 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.346+0000 7fcaa91fee40 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x5630142dce30 con 0x56301440ee40 2023-10-05T19:28:57.352 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.346+0000 7fcaa6d33700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x56301440ee40 0x563014432810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-05T19:28:57.352 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.346+0000 7fcaa6d33700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x56301440ee40 0x563014432810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.118:3300/0 says I am v2:172.21.15.167:53492/0 (socket says 172.21.15.167:53492) 2023-10-05T19:28:57.352 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.346+0000 7fcaa6d33700 1 -- 172.21.15.167:0/3312566257 learned_addr learned my addr 172.21.15.167:0/3312566257 (peer_addr_for_me v2:172.21.15.167:0/0) 2023-10-05T19:28:57.352 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.346+0000 7fcaa6d33700 1 -- 172.21.15.167:0/3312566257 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fca90005cf0 con 0x56301440ee40 2023-10-05T19:28:57.353 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.346+0000 7fcaa6d33700 1 --2- 172.21.15.167:0/3312566257 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x56301440ee40 0x563014432810 secure :-1 s=READY pgs=562 cs=0 l=1 rev1=1 crypto rx=0x7fcaa0001c30 tx=0x7fcaa0001810 comp rx=0 tx=0).ready entity=mon.0 client_cookie=77e7e0b0510307c2 server_cookie=0 in_seq=0 out_seq=0 2023-10-05T19:28:57.353 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.346+0000 7fcaa4d2f700 1 -- 172.21.15.167:0/3312566257 <== mon.0 v2:172.21.15.118:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fcaa000bbf0 con 0x56301440ee40 2023-10-05T19:28:57.353 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.346+0000 7fcaa4d2f700 1 -- 172.21.15.167:0/3312566257 <== mon.0 v2:172.21.15.118:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcaa000bd50 con 0x56301440ee40 2023-10-05T19:28:57.353 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.346+0000 7fcaa4d2f700 1 -- 172.21.15.167:0/3312566257 <== mon.0 v2:172.21.15.118:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fcaa000e470 con 0x56301440ee40 2023-10-05T19:28:57.354 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.350+0000 7fcaa91fee40 1 -- 172.21.15.167:0/3312566257 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x56301440ee40 msgr2=0x563014432810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-05T19:28:57.354 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.350+0000 7fcaa91fee40 1 --2- 172.21.15.167:0/3312566257 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x56301440ee40 0x563014432810 secure :-1 s=READY pgs=562 cs=0 l=1 rev1=1 crypto rx=0x7fcaa0001c30 tx=0x7fcaa0001810 comp rx=0 tx=0).stop 2023-10-05T19:28:57.354 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.350+0000 7fcaa91fee40 1 -- 172.21.15.167:0/3312566257 shutdown_connections 2023-10-05T19:28:57.354 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.350+0000 7fcaa91fee40 1 --2- 172.21.15.167:0/3312566257 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x56301440ee40 0x563014432810 unknown :-1 s=CLOSED pgs=562 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-05T19:28:57.354 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.350+0000 7fcaa91fee40 1 -- 172.21.15.167:0/3312566257 >> 172.21.15.167:0/3312566257 conn(0x56301440a860 msgr2=0x563014383210 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-05T19:28:57.354 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.350+0000 7fcaa91fee40 1 -- 172.21.15.167:0/3312566257 shutdown_connections 2023-10-05T19:28:57.355 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.350+0000 7fcaa91fee40 1 -- 172.21.15.167:0/3312566257 wait complete. 2023-10-05T19:28:57.355 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.350+0000 7fcaa91fee40 1 Processor -- start 2023-10-05T19:28:57.355 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.350+0000 7fcaa91fee40 1 -- start start 2023-10-05T19:28:57.355 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.350+0000 7fcaa91fee40 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x56301440ee40 0x563014436da0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-05T19:28:57.355 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.350+0000 7fcaa91fee40 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x5630142afed0 con 0x56301440ee40 2023-10-05T19:28:57.356 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.350+0000 7fcaa6d33700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x56301440ee40 0x563014436da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-05T19:28:57.356 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.350+0000 7fcaa6d33700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x56301440ee40 0x563014436da0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.118:3300/0 says I am v2:172.21.15.167:53508/0 (socket says 172.21.15.167:53508) 2023-10-05T19:28:57.356 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.350+0000 7fcaa6d33700 1 -- 172.21.15.167:0/2066703832 learned_addr learned my addr 172.21.15.167:0/2066703832 (peer_addr_for_me v2:172.21.15.167:0/0) 2023-10-05T19:28:57.357 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.350+0000 7fcaa6d33700 1 -- 172.21.15.167:0/2066703832 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fca94001e10 con 0x56301440ee40 2023-10-05T19:28:57.357 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.350+0000 7fcaa6d33700 1 --2- 172.21.15.167:0/2066703832 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x56301440ee40 0x563014436da0 secure :-1 s=READY pgs=563 cs=0 l=1 rev1=1 crypto rx=0x7fcaa000b960 tx=0x7fcaa0014000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-05T19:28:57.358 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.350+0000 7fca9effd700 1 -- 172.21.15.167:0/2066703832 <== mon.0 v2:172.21.15.118:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fcaa001f070 con 0x56301440ee40 2023-10-05T19:28:57.358 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.354+0000 7fcaa91fee40 1 -- 172.21.15.167:0/2066703832 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x5630144372e0 con 0x56301440ee40 2023-10-05T19:28:57.358 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.354+0000 7fca9effd700 1 -- 172.21.15.167:0/2066703832 <== mon.0 v2:172.21.15.118:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcaa001a2e0 con 0x56301440ee40 2023-10-05T19:28:57.358 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.354+0000 7fca9effd700 1 -- 172.21.15.167:0/2066703832 <== mon.0 v2:172.21.15.118:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fcaa000bec0 con 0x56301440ee40 2023-10-05T19:28:57.358 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.354+0000 7fcaa91fee40 1 -- 172.21.15.167:0/2066703832 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x563014437720 con 0x56301440ee40 2023-10-05T19:28:57.358 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.354+0000 7fcaa91fee40 20 rados->read ofs=0 len=0 2023-10-05T19:28:57.359 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.354+0000 7fca9effd700 1 -- 172.21.15.167:0/2066703832 <== mon.0 v2:172.21.15.118:3300/0 4 ==== mgrmap(e 5) v1 ==== 40329+0+0 (secure 0 0 0) 0x7fcaa001aa10 con 0x56301440ee40 2023-10-05T19:28:57.360 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.354+0000 7fca9effd700 1 --2- 172.21.15.167:0/2066703832 >> [v2:172.21.15.167:6824/29889,v1:172.21.15.167:6825/29889] conn(0x7fca88030bf0 0x7fca880330a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-05T19:28:57.360 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.354+0000 7fca9effd700 1 -- 172.21.15.167:0/2066703832 <== mon.0 v2:172.21.15.118:3300/0 5 ==== osd_map(32..32 src has 1..32) v4 ==== 6260+0+0 (secure 0 0 0) 0x7fcaa0045170 con 0x56301440ee40 2023-10-05T19:28:57.360 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.354+0000 7fcaa6532700 1 --2- 172.21.15.167:0/2066703832 >> [v2:172.21.15.167:6824/29889,v1:172.21.15.167:6825/29889] conn(0x7fca88030bf0 0x7fca880330a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-05T19:28:57.360 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.354+0000 7fcaa91fee40 1 --2- 172.21.15.167:0/2066703832 >> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] conn(0x563014413420 0x563014442830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-05T19:28:57.360 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.354+0000 7fcaa91fee40 1 -- 172.21.15.167:0/2066703832 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:1 5.12 5:49953fa1:::default.realm:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32) v8 -- 0x563014442ee0 con 0x563014413420 2023-10-05T19:28:57.360 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.354+0000 7fcaa7534700 1 --2- 172.21.15.167:0/2066703832 >> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] conn(0x563014413420 0x563014442830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-05T19:28:57.361 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.354+0000 7fcaa6532700 1 --2- 172.21.15.167:0/2066703832 >> [v2:172.21.15.167:6824/29889,v1:172.21.15.167:6825/29889] conn(0x7fca88030bf0 0x7fca880330a0 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7fca900030c0 tx=0x7fca90014000 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-05T19:28:57.361 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.354+0000 7fcaa7534700 1 --2- 172.21.15.167:0/2066703832 >> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] conn(0x563014413420 0x563014442830 crc :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-05T19:28:57.362 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.358+0000 7fcaa7534700 1 -- 172.21.15.167:0/2066703832 <== osd.1 v2:172.21.15.167:6800/30035 1 ==== osd_op_reply(1 default.realm [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 ==== 157+0+0 (crc 0 0 0) 0x7fca9800a5a0 con 0x563014413420 2023-10-05T19:28:57.362 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.358+0000 7fcaa91fee40 20 rados_obj.operate() r=-2 bl.length=0 2023-10-05T19:28:57.362 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.358+0000 7fcaa91fee40 20 realm 2023-10-05T19:28:57.362 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.358+0000 7fcaa91fee40 20 rados->read ofs=0 len=0 2023-10-05T19:28:57.362 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.358+0000 7fcaa91fee40 1 -- 172.21.15.167:0/2066703832 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:2 5.12 5:49953fa1:::default.realm:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32) v8 -- 0x56301443fe50 con 0x563014413420 2023-10-05T19:28:57.363 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.358+0000 7fcaa7534700 1 -- 172.21.15.167:0/2066703832 <== osd.1 v2:172.21.15.167:6800/30035 2 ==== osd_op_reply(2 default.realm [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 ==== 157+0+0 (crc 0 0 0) 0x7fca9800a5a0 con 0x563014413420 2023-10-05T19:28:57.363 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.358+0000 7fcaa91fee40 20 rados_obj.operate() r=-2 bl.length=0 2023-10-05T19:28:57.363 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.358+0000 7fcaa91fee40 4 RGWPeriod::init failed to init realm id : (2) No such file or directory 2023-10-05T19:28:57.363 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.358+0000 7fcaa91fee40 20 rados->read ofs=0 len=0 2023-10-05T19:28:57.363 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.358+0000 7fcaa91fee40 1 -- 172.21.15.167:0/2066703832 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:3 5.12 5:49953fa1:::default.realm:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32) v8 -- 0x5630144439c0 con 0x563014413420 2023-10-05T19:28:57.364 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.358+0000 7fcaa7534700 1 -- 172.21.15.167:0/2066703832 <== osd.1 v2:172.21.15.167:6800/30035 3 ==== osd_op_reply(3 default.realm [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 ==== 157+0+0 (crc 0 0 0) 0x7fca9800a5a0 con 0x563014413420 2023-10-05T19:28:57.364 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.358+0000 7fcaa91fee40 20 rados_obj.operate() r=-2 bl.length=0 2023-10-05T19:28:57.364 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.358+0000 7fcaa91fee40 20 rados->read ofs=0 len=0 2023-10-05T19:28:57.364 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.358+0000 7fcaa91fee40 1 -- 172.21.15.167:0/2066703832 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:4 5.f 5:f43b6ece:::zone_names.default:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32) v8 -- 0x5630144435a0 con 0x563014413420 2023-10-05T19:28:57.365 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.358+0000 7fcaa7534700 1 -- 172.21.15.167:0/2066703832 <== osd.1 v2:172.21.15.167:6800/30035 4 ==== osd_op_reply(4 zone_names.default [read 0~46 out=46b] v0'0 uv1 ondisk = 0) v8 ==== 162+0+46 (crc 0 0 0) 0x7fca9800a5a0 con 0x563014413420 2023-10-05T19:28:57.365 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.358+0000 7fcaa91fee40 20 rados_obj.operate() r=0 bl.length=46 2023-10-05T19:28:57.365 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.358+0000 7fcaa91fee40 20 rados->read ofs=0 len=0 2023-10-05T19:28:57.365 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.358+0000 7fcaa91fee40 1 -- 172.21.15.167:0/2066703832 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:5 5.1d 5:b8374eb4:::zone_info.9a9df965-6612-45d4-a45f-6e3c889bacc2:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32) v8 -- 0x563014444560 con 0x563014413420 2023-10-05T19:28:57.365 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.358+0000 7fcaa7534700 1 -- 172.21.15.167:0/2066703832 <== osd.1 v2:172.21.15.167:6800/30035 5 ==== osd_op_reply(5 zone_info.9a9df965-6612-45d4-a45f-6e3c889bacc2 [read 0~874 out=874b] v0'0 uv2 ondisk = 0) v8 ==== 190+0+874 (crc 0 0 0) 0x7fca9800a5a0 con 0x563014413420 2023-10-05T19:28:57.365 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.358+0000 7fcaa91fee40 20 rados_obj.operate() r=0 bl.length=874 2023-10-05T19:28:57.365 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.358+0000 7fcaa91fee40 20 searching for the correct realm 2023-10-05T19:28:57.365 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.358+0000 7fcaa91fee40 1 --2- 172.21.15.167:0/2066703832 >> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] conn(0x5630144464a0 0x5630144488b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-05T19:28:57.366 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.358+0000 7fcaa91fee40 1 -- 172.21.15.167:0/2066703832 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:6 5.0 5:00000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x563014448ec0 con 0x5630144464a0 2023-10-05T19:28:57.366 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.358+0000 7fcaa6d33700 1 --2- 172.21.15.167:0/2066703832 >> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] conn(0x5630144464a0 0x5630144488b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-05T19:28:57.366 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.362+0000 7fcaa6d33700 1 --2- 172.21.15.167:0/2066703832 >> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] conn(0x5630144464a0 0x5630144488b0 crc :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-05T19:28:57.366 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.362+0000 7fcaa6d33700 1 -- 172.21.15.167:0/2066703832 <== osd.3 v2:172.21.15.167:6808/30036 1 ==== osd_op_reply(6 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fcaa0014070 con 0x5630144464a0 2023-10-05T19:28:57.367 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.362+0000 7fcaa6d33700 1 --2- 172.21.15.167:0/2066703832 >> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] conn(0x7fcaa0022360 0x7fcaa002d3f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-05T19:28:57.367 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.362+0000 7fcaa6d33700 1 -- 172.21.15.167:0/2066703832 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:7 5.10 5:08000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x563014448ec0 con 0x7fcaa0022360 2023-10-05T19:28:57.367 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.362+0000 7fcaa6532700 1 --2- 172.21.15.167:0/2066703832 >> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] conn(0x7fcaa0022360 0x7fcaa002d3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-05T19:28:57.367 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.362+0000 7fcaa6532700 1 --2- 172.21.15.167:0/2066703832 >> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] conn(0x7fcaa0022360 0x7fcaa002d3f0 crc :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-05T19:28:57.368 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.362+0000 7fcaa6532700 1 -- 172.21.15.167:0/2066703832 <== osd.2 v2:172.21.15.167:6816/30037 1 ==== osd_op_reply(7 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fca900079f0 con 0x7fcaa0022360 2023-10-05T19:28:57.368 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.362+0000 7fcaa6532700 1 --2- 172.21.15.167:0/2066703832 >> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] conn(0x7fca90007030 0x7fca9001d430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-05T19:28:57.368 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.362+0000 7fcaa6532700 1 -- 172.21.15.167:0/2066703832 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:8 5.8 5:10000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x563014448ec0 con 0x7fca90007030 2023-10-05T19:28:57.368 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.362+0000 7fcaa7534700 1 --2- 172.21.15.167:0/2066703832 >> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] conn(0x7fca90007030 0x7fca9001d430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-05T19:28:57.369 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.362+0000 7fcaa7534700 1 --2- 172.21.15.167:0/2066703832 >> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] conn(0x7fca90007030 0x7fca9001d430 crc :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-05T19:28:57.369 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.366+0000 7fcaa7534700 1 -- 172.21.15.167:0/2066703832 <== osd.0 v2:172.21.15.118:6800/27690 1 ==== osd_op_reply(8 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fca9800a5a0 con 0x7fca90007030 2023-10-05T19:28:57.370 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.366+0000 7fcaa7534700 1 -- 172.21.15.167:0/2066703832 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:9 5.18 5:18000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x563014448ec0 con 0x563014413420 2023-10-05T19:28:57.370 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.366+0000 7fcaa7534700 1 -- 172.21.15.167:0/2066703832 <== osd.1 v2:172.21.15.167:6800/30035 6 ==== osd_op_reply(9 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fca9800a5a0 con 0x563014413420 2023-10-05T19:28:57.370 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.366+0000 7fcaa7534700 1 -- 172.21.15.167:0/2066703832 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:10 5.4 5:20000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x563014448ec0 con 0x563014413420 2023-10-05T19:28:57.371 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.366+0000 7fcaa7534700 1 -- 172.21.15.167:0/2066703832 <== osd.1 v2:172.21.15.167:6800/30035 7 ==== osd_op_reply(10 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fca9800a5a0 con 0x563014413420 2023-10-05T19:28:57.371 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.366+0000 7fcaa7534700 1 -- 172.21.15.167:0/2066703832 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:11 5.14 5:28000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x563014448ec0 con 0x7fca90007030 2023-10-05T19:28:57.372 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.366+0000 7fcaa7534700 1 -- 172.21.15.167:0/2066703832 <== osd.0 v2:172.21.15.118:6800/27690 2 ==== osd_op_reply(11 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fca9800a5a0 con 0x7fca90007030 2023-10-05T19:28:57.372 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.366+0000 7fcaa7534700 1 -- 172.21.15.167:0/2066703832 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:12 5.c 5:30000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x563014448ec0 con 0x7fca90007030 2023-10-05T19:28:57.372 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.366+0000 7fcaa7534700 1 -- 172.21.15.167:0/2066703832 <== osd.0 v2:172.21.15.118:6800/27690 3 ==== osd_op_reply(12 [pgnls start_epoch 32 out=112b] v23'1 uv1 ondisk = 1) v8 ==== 144+0+112 (crc 0 0 0) 0x7fca9800a5a0 con 0x7fca90007030 2023-10-05T19:28:57.373 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.366+0000 7fcaa7534700 1 -- 172.21.15.167:0/2066703832 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:13 5.1c 5:38000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x563014448ec0 con 0x7fcaa0022360 2023-10-05T19:28:57.373 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.370+0000 7fcaa6532700 1 -- 172.21.15.167:0/2066703832 <== osd.2 v2:172.21.15.167:6816/30037 2 ==== osd_op_reply(13 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fca90029040 con 0x7fcaa0022360 2023-10-05T19:28:57.373 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.370+0000 7fcaa6532700 1 -- 172.21.15.167:0/2066703832 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:14 5.2 5:40000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x563014448ec0 con 0x5630144464a0 2023-10-05T19:28:57.374 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.370+0000 7fcaa6d33700 1 -- 172.21.15.167:0/2066703832 <== osd.3 v2:172.21.15.167:6808/30036 2 ==== osd_op_reply(14 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fcaa0014070 con 0x5630144464a0 2023-10-05T19:28:57.374 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.370+0000 7fcaa6d33700 1 -- 172.21.15.167:0/2066703832 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:15 5.12 5:48000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x563014448ec0 con 0x563014413420 2023-10-05T19:28:57.375 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.370+0000 7fcaa7534700 1 -- 172.21.15.167:0/2066703832 <== osd.1 v2:172.21.15.167:6800/30035 8 ==== osd_op_reply(15 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fca9800a5a0 con 0x563014413420 2023-10-05T19:28:57.375 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.370+0000 7fcaa7534700 1 -- 172.21.15.167:0/2066703832 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:16 5.a 5:50000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x563014448ec0 con 0x7fca90007030 2023-10-05T19:28:57.376 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.370+0000 7fcaa7534700 1 -- 172.21.15.167:0/2066703832 <== osd.0 v2:172.21.15.118:6800/27690 4 ==== osd_op_reply(16 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fca9800a5a0 con 0x7fca90007030 2023-10-05T19:28:57.376 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.370+0000 7fcaa7534700 1 -- 172.21.15.167:0/2066703832 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:17 5.1a 5:58000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x563014448ec0 con 0x563014413420 2023-10-05T19:28:57.376 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.370+0000 7fcaa7534700 1 -- 172.21.15.167:0/2066703832 <== osd.1 v2:172.21.15.167:6800/30035 9 ==== osd_op_reply(17 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fca9800a5a0 con 0x563014413420 2023-10-05T19:28:57.377 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.370+0000 7fcaa7534700 1 -- 172.21.15.167:0/2066703832 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:18 5.6 5:60000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x563014448ec0 con 0x7fcaa0022360 2023-10-05T19:28:57.377 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.374+0000 7fcaa6532700 1 -- 172.21.15.167:0/2066703832 <== osd.2 v2:172.21.15.167:6816/30037 3 ==== osd_op_reply(18 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fca90029040 con 0x7fcaa0022360 2023-10-05T19:28:57.377 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.374+0000 7fcaa6532700 1 -- 172.21.15.167:0/2066703832 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:19 5.16 5:68000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x563014448ec0 con 0x5630144464a0 2023-10-05T19:28:57.378 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.374+0000 7fcaa6d33700 1 -- 172.21.15.167:0/2066703832 <== osd.3 v2:172.21.15.167:6808/30036 3 ==== osd_op_reply(19 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fcaa0014070 con 0x5630144464a0 2023-10-05T19:28:57.378 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.374+0000 7fcaa6d33700 1 -- 172.21.15.167:0/2066703832 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:20 5.e 5:70000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x563014448ec0 con 0x5630144464a0 2023-10-05T19:28:57.379 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.374+0000 7fcaa6d33700 1 -- 172.21.15.167:0/2066703832 <== osd.3 v2:172.21.15.167:6808/30036 4 ==== osd_op_reply(20 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fcaa0014070 con 0x5630144464a0 2023-10-05T19:28:57.379 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.374+0000 7fcaa6d33700 1 -- 172.21.15.167:0/2066703832 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:21 5.1e 5:78000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x563014448ec0 con 0x7fca90007030 2023-10-05T19:28:57.380 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.374+0000 7fcaa7534700 1 -- 172.21.15.167:0/2066703832 <== osd.0 v2:172.21.15.118:6800/27690 5 ==== osd_op_reply(21 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fca9800a5a0 con 0x7fca90007030 2023-10-05T19:28:57.380 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.374+0000 7fcaa7534700 1 -- 172.21.15.167:0/2066703832 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:22 5.1 5:80000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x563014448ec0 con 0x5630144464a0 2023-10-05T19:28:57.380 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.374+0000 7fcaa6d33700 1 -- 172.21.15.167:0/2066703832 <== osd.3 v2:172.21.15.167:6808/30036 5 ==== osd_op_reply(22 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fcaa0014070 con 0x5630144464a0 2023-10-05T19:28:57.381 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.374+0000 7fcaa6d33700 1 -- 172.21.15.167:0/2066703832 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:23 5.11 5:88000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x563014448ec0 con 0x5630144464a0 2023-10-05T19:28:57.381 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.374+0000 7fcaa6d33700 1 -- 172.21.15.167:0/2066703832 <== osd.3 v2:172.21.15.167:6808/30036 6 ==== osd_op_reply(23 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fcaa0014070 con 0x5630144464a0 2023-10-05T19:28:57.381 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.374+0000 7fcaa6d33700 1 -- 172.21.15.167:0/2066703832 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:24 5.9 5:90000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x563014448ec0 con 0x563014413420 2023-10-05T19:28:57.382 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.378+0000 7fcaa7534700 1 -- 172.21.15.167:0/2066703832 <== osd.1 v2:172.21.15.167:6800/30035 10 ==== osd_op_reply(24 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fca9800a5a0 con 0x563014413420 2023-10-05T19:28:57.382 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.378+0000 7fcaa7534700 1 -- 172.21.15.167:0/2066703832 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:25 5.19 5:98000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x563014448ec0 con 0x563014413420 2023-10-05T19:28:57.383 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.378+0000 7fcaa7534700 1 -- 172.21.15.167:0/2066703832 <== osd.1 v2:172.21.15.167:6800/30035 11 ==== osd_op_reply(25 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fca9800a5a0 con 0x563014413420 2023-10-05T19:28:57.383 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.378+0000 7fcaa7534700 1 -- 172.21.15.167:0/2066703832 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:26 5.5 5:a0000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x563014448ec0 con 0x7fca90007030 2023-10-05T19:28:57.384 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.378+0000 7fcaa7534700 1 -- 172.21.15.167:0/2066703832 <== osd.0 v2:172.21.15.118:6800/27690 6 ==== osd_op_reply(26 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fca9800a5a0 con 0x7fca90007030 2023-10-05T19:28:57.384 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.378+0000 7fcaa7534700 1 -- 172.21.15.167:0/2066703832 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:27 5.15 5:a8000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x563014448ec0 con 0x5630144464a0 2023-10-05T19:28:57.384 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.378+0000 7fcaa6d33700 1 -- 172.21.15.167:0/2066703832 <== osd.3 v2:172.21.15.167:6808/30036 7 ==== osd_op_reply(27 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fcaa0014070 con 0x5630144464a0 2023-10-05T19:28:57.385 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.378+0000 7fcaa6d33700 1 -- 172.21.15.167:0/2066703832 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:28 5.d 5:b0000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x563014448ec0 con 0x7fcaa0022360 2023-10-05T19:28:57.385 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.382+0000 7fcaa6532700 1 -- 172.21.15.167:0/2066703832 <== osd.2 v2:172.21.15.167:6816/30037 4 ==== osd_op_reply(28 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fca90029040 con 0x7fcaa0022360 2023-10-05T19:28:57.385 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.382+0000 7fcaa6532700 1 -- 172.21.15.167:0/2066703832 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:29 5.1d 5:b8000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x563014448ec0 con 0x563014413420 2023-10-05T19:28:57.386 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.382+0000 7fcaa7534700 1 -- 172.21.15.167:0/2066703832 <== osd.1 v2:172.21.15.167:6800/30035 12 ==== osd_op_reply(29 [pgnls start_epoch 32 out=107b] v30'2 uv2 ondisk = 1) v8 ==== 144+0+107 (crc 0 0 0) 0x7fca9800a5a0 con 0x563014413420 2023-10-05T19:28:57.386 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.382+0000 7fcaa7534700 1 -- 172.21.15.167:0/2066703832 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:30 5.3 5:c0000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x563014448ec0 con 0x7fca90007030 2023-10-05T19:28:57.387 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.382+0000 7fcaa7534700 1 -- 172.21.15.167:0/2066703832 <== osd.0 v2:172.21.15.118:6800/27690 7 ==== osd_op_reply(30 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fca9800a5a0 con 0x7fca90007030 2023-10-05T19:28:57.387 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.382+0000 7fcaa7534700 1 -- 172.21.15.167:0/2066703832 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:31 5.13 5:c8000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x563014448ec0 con 0x5630144464a0 2023-10-05T19:28:57.388 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.382+0000 7fcaa6d33700 1 -- 172.21.15.167:0/2066703832 <== osd.3 v2:172.21.15.167:6808/30036 8 ==== osd_op_reply(31 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fcaa0014070 con 0x5630144464a0 2023-10-05T19:28:57.388 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.382+0000 7fcaa6d33700 1 -- 172.21.15.167:0/2066703832 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:32 5.b 5:d0000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x563014448ec0 con 0x7fca90007030 2023-10-05T19:28:57.389 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.382+0000 7fcaa7534700 1 -- 172.21.15.167:0/2066703832 <== osd.0 v2:172.21.15.118:6800/27690 8 ==== osd_op_reply(32 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fca9800a5a0 con 0x7fca90007030 2023-10-05T19:28:57.389 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.382+0000 7fcaa7534700 1 -- 172.21.15.167:0/2066703832 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:33 5.1b 5:d8000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x563014448ec0 con 0x7fcaa0022360 2023-10-05T19:28:57.390 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.386+0000 7fcaa6532700 1 -- 172.21.15.167:0/2066703832 <== osd.2 v2:172.21.15.167:6816/30037 5 ==== osd_op_reply(33 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fca90029040 con 0x7fcaa0022360 2023-10-05T19:28:57.390 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.386+0000 7fcaa6532700 1 -- 172.21.15.167:0/2066703832 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:34 5.7 5:e0000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x563014448ec0 con 0x5630144464a0 2023-10-05T19:28:57.390 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.386+0000 7fcaa6d33700 1 -- 172.21.15.167:0/2066703832 <== osd.3 v2:172.21.15.167:6808/30036 9 ==== osd_op_reply(34 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fcaa0014070 con 0x5630144464a0 2023-10-05T19:28:57.391 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.386+0000 7fcaa6d33700 1 -- 172.21.15.167:0/2066703832 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:35 5.17 5:e8000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x563014448ec0 con 0x7fca90007030 2023-10-05T19:28:57.391 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.386+0000 7fcaa7534700 1 -- 172.21.15.167:0/2066703832 <== osd.0 v2:172.21.15.118:6800/27690 9 ==== osd_op_reply(35 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fca9800a5a0 con 0x7fca90007030 2023-10-05T19:28:57.392 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.386+0000 7fcaa7534700 1 -- 172.21.15.167:0/2066703832 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:36 5.f 5:f0000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x563014448ec0 con 0x563014413420 2023-10-05T19:28:57.392 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.386+0000 7fcaa7534700 1 -- 172.21.15.167:0/2066703832 <== osd.1 v2:172.21.15.167:6800/30035 13 ==== osd_op_reply(36 [pgnls start_epoch 32 out=115b] v23'2 uv2 ondisk = 1) v8 ==== 144+0+115 (crc 0 0 0) 0x7fca9800a5a0 con 0x563014413420 2023-10-05T19:28:57.393 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.386+0000 7fcaa7534700 1 -- 172.21.15.167:0/2066703832 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:37 5.1f 5:f8000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x563014448ec0 con 0x7fcaa0022360 2023-10-05T19:28:57.394 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.390+0000 7fcaa6532700 1 -- 172.21.15.167:0/2066703832 <== osd.2 v2:172.21.15.167:6816/30037 6 ==== osd_op_reply(37 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fca90029040 con 0x7fcaa0022360 2023-10-05T19:28:57.394 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.390+0000 7fcaa91fee40 20 RGWRados::pool_iterate: got zonegroup_info.4f1f719d-9648-417b-b10c-e5d939825419 2023-10-05T19:28:57.394 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.390+0000 7fcaa91fee40 20 RGWRados::pool_iterate: got zone_info.9a9df965-6612-45d4-a45f-6e3c889bacc2 2023-10-05T19:28:57.394 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.390+0000 7fcaa91fee40 20 RGWRados::pool_iterate: got zone_names.default 2023-10-05T19:28:57.394 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.390+0000 7fcaa91fee40 20 RGWRados::pool_iterate: got zonegroups_names.default 2023-10-05T19:28:57.394 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.390+0000 7fcaa91fee40 20 rados->read ofs=0 len=0 2023-10-05T19:28:57.394 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.390+0000 7fcaa91fee40 1 -- 172.21.15.167:0/2066703832 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:38 5.12 5:49953fa1:::default.realm:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32) v8 -- 0x563014444f50 con 0x563014413420 2023-10-05T19:28:57.395 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.390+0000 7fcaa7534700 1 -- 172.21.15.167:0/2066703832 <== osd.1 v2:172.21.15.167:6800/30035 14 ==== osd_op_reply(38 default.realm [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 ==== 157+0+0 (crc 0 0 0) 0x7fca9800a5a0 con 0x563014413420 2023-10-05T19:28:57.395 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.390+0000 7fcaa91fee40 20 rados_obj.operate() r=-2 bl.length=0 2023-10-05T19:28:57.395 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.390+0000 7fcaa91fee40 20 rados->read ofs=0 len=0 2023-10-05T19:28:57.395 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.390+0000 7fcaa91fee40 1 -- 172.21.15.167:0/2066703832 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:39 5.f 5:f4c53578:::zonegroups_names.default:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32) v8 -- 0x563014445350 con 0x563014413420 2023-10-05T19:28:57.396 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.390+0000 7fcaa7534700 1 -- 172.21.15.167:0/2066703832 <== osd.1 v2:172.21.15.167:6800/30035 15 ==== osd_op_reply(39 zonegroups_names.default [read 0~46 out=46b] v0'0 uv2 ondisk = 0) v8 ==== 168+0+46 (crc 0 0 0) 0x7fca9800a5a0 con 0x563014413420 2023-10-05T19:28:57.396 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.390+0000 7fcaa91fee40 20 rados_obj.operate() r=0 bl.length=46 2023-10-05T19:28:57.396 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.390+0000 7fcaa91fee40 20 rados->read ofs=0 len=0 2023-10-05T19:28:57.396 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.390+0000 7fcaa91fee40 1 -- 172.21.15.167:0/2066703832 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:40 5.c 5:30d299a5:::zonegroup_info.4f1f719d-9648-417b-b10c-e5d939825419:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32) v8 -- 0x563014445350 con 0x7fca90007030 2023-10-05T19:28:57.396 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.390+0000 7fcaa7534700 1 -- 172.21.15.167:0/2066703832 <== osd.0 v2:172.21.15.118:6800/27690 10 ==== osd_op_reply(40 zonegroup_info.4f1f719d-9648-417b-b10c-e5d939825419 [read 0~362 out=362b] v0'0 uv1 ondisk = 0) v8 ==== 195+0+362 (crc 0 0 0) 0x7fca9800a5a0 con 0x7fca90007030 2023-10-05T19:28:57.397 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.390+0000 7fcaa91fee40 20 rados_obj.operate() r=0 bl.length=362 2023-10-05T19:28:57.397 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.390+0000 7fcaa91fee40 20 rados->read ofs=0 len=0 2023-10-05T19:28:57.397 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.390+0000 7fcaa91fee40 1 -- 172.21.15.167:0/2066703832 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:41 5.13 5:cbbf6c60:::converted:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32) v8 -- 0x563014445350 con 0x5630144464a0 2023-10-05T19:28:57.397 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.390+0000 7fcaa6d33700 1 -- 172.21.15.167:0/2066703832 <== osd.3 v2:172.21.15.167:6808/30036 10 ==== osd_op_reply(41 converted [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 ==== 153+0+0 (crc 0 0 0) 0x7fcaa0014070 con 0x5630144464a0 2023-10-05T19:28:57.398 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.390+0000 7fcaa91fee40 20 rados_obj.operate() r=-2 bl.length=0 2023-10-05T19:28:57.398 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.390+0000 7fcaa91fee40 20 rados->read ofs=0 len=0 2023-10-05T19:28:57.398 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.390+0000 7fcaa91fee40 1 -- 172.21.15.167:0/2066703832 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:42 5.12 5:49953fa1:::default.realm:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32) v8 -- 0x563014449330 con 0x563014413420 2023-10-05T19:28:57.398 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.394+0000 7fcaa7534700 1 -- 172.21.15.167:0/2066703832 <== osd.1 v2:172.21.15.167:6800/30035 16 ==== osd_op_reply(42 default.realm [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 ==== 157+0+0 (crc 0 0 0) 0x7fca9800a5a0 con 0x563014413420 2023-10-05T19:28:57.399 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.394+0000 7fcaa91fee40 20 rados_obj.operate() r=-2 bl.length=0 2023-10-05T19:28:57.399 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.394+0000 7fcaa91fee40 20 rados->read ofs=0 len=0 2023-10-05T19:28:57.399 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.394+0000 7fcaa91fee40 1 -- 172.21.15.167:0/2066703832 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:43 5.f 5:f4c53578:::zonegroups_names.default:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32) v8 -- 0x563014449710 con 0x563014413420 2023-10-05T19:28:57.399 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.394+0000 7fcaa7534700 1 -- 172.21.15.167:0/2066703832 <== osd.1 v2:172.21.15.167:6800/30035 17 ==== osd_op_reply(43 zonegroups_names.default [read 0~46 out=46b] v0'0 uv2 ondisk = 0) v8 ==== 168+0+46 (crc 0 0 0) 0x7fca9800a5a0 con 0x563014413420 2023-10-05T19:28:57.399 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.394+0000 7fcaa91fee40 20 rados_obj.operate() r=0 bl.length=46 2023-10-05T19:28:57.399 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.394+0000 7fcaa91fee40 1 -- 172.21.15.167:0/2066703832 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:44 5.0 5:00000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x56301444a720 con 0x5630144464a0 2023-10-05T19:28:57.400 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.394+0000 7fcaa6d33700 1 -- 172.21.15.167:0/2066703832 <== osd.3 v2:172.21.15.167:6808/30036 11 ==== osd_op_reply(44 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fcaa0014070 con 0x5630144464a0 2023-10-05T19:28:57.401 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.394+0000 7fcaa6d33700 1 -- 172.21.15.167:0/2066703832 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:45 5.10 5:08000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x56301444a720 con 0x7fcaa0022360 2023-10-05T19:28:57.401 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.394+0000 7fcaa6532700 1 -- 172.21.15.167:0/2066703832 <== osd.2 v2:172.21.15.167:6816/30037 7 ==== osd_op_reply(45 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fca90029040 con 0x7fcaa0022360 2023-10-05T19:28:57.401 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.394+0000 7fcaa6532700 1 -- 172.21.15.167:0/2066703832 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:46 5.8 5:10000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x56301444a720 con 0x7fca90007030 2023-10-05T19:28:57.402 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.398+0000 7fcaa7534700 1 -- 172.21.15.167:0/2066703832 <== osd.0 v2:172.21.15.118:6800/27690 11 ==== osd_op_reply(46 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fca9800a5a0 con 0x7fca90007030 2023-10-05T19:28:57.402 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.398+0000 7fcaa7534700 1 -- 172.21.15.167:0/2066703832 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:47 5.18 5:18000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x563014444f50 con 0x563014413420 2023-10-05T19:28:57.403 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.398+0000 7fcaa7534700 1 -- 172.21.15.167:0/2066703832 <== osd.1 v2:172.21.15.167:6800/30035 18 ==== osd_op_reply(47 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fca9800a5a0 con 0x563014413420 2023-10-05T19:28:57.403 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.398+0000 7fcaa7534700 1 -- 172.21.15.167:0/2066703832 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:48 5.4 5:20000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x563014444f50 con 0x563014413420 2023-10-05T19:28:57.404 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.398+0000 7fcaa7534700 1 -- 172.21.15.167:0/2066703832 <== osd.1 v2:172.21.15.167:6800/30035 19 ==== osd_op_reply(48 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fca9800a5a0 con 0x563014413420 2023-10-05T19:28:57.404 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.398+0000 7fcaa7534700 1 -- 172.21.15.167:0/2066703832 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:49 5.14 5:28000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x563014444f50 con 0x7fca90007030 2023-10-05T19:28:57.405 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.398+0000 7fcaa7534700 1 -- 172.21.15.167:0/2066703832 <== osd.0 v2:172.21.15.118:6800/27690 12 ==== osd_op_reply(49 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fca9800a5a0 con 0x7fca90007030 2023-10-05T19:28:57.405 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.398+0000 7fcaa7534700 1 -- 172.21.15.167:0/2066703832 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:50 5.c 5:30000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x563014444f50 con 0x7fca90007030 2023-10-05T19:28:57.406 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.402+0000 7fcaa7534700 1 -- 172.21.15.167:0/2066703832 <== osd.0 v2:172.21.15.118:6800/27690 13 ==== osd_op_reply(50 [pgnls start_epoch 32 out=112b] v23'1 uv1 ondisk = 1) v8 ==== 144+0+112 (crc 0 0 0) 0x7fca9800a5a0 con 0x7fca90007030 2023-10-05T19:28:57.406 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.402+0000 7fcaa7534700 1 -- 172.21.15.167:0/2066703832 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:51 5.1c 5:38000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x563014444f50 con 0x7fcaa0022360 2023-10-05T19:28:57.407 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.402+0000 7fcaa6532700 1 -- 172.21.15.167:0/2066703832 <== osd.2 v2:172.21.15.167:6816/30037 8 ==== osd_op_reply(51 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fca90029040 con 0x7fcaa0022360 2023-10-05T19:28:57.407 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.402+0000 7fcaa6532700 1 -- 172.21.15.167:0/2066703832 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:52 5.2 5:40000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x563014444f50 con 0x5630144464a0 2023-10-05T19:28:57.407 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.402+0000 7fcaa6d33700 1 -- 172.21.15.167:0/2066703832 <== osd.3 v2:172.21.15.167:6808/30036 12 ==== osd_op_reply(52 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fcaa0014070 con 0x5630144464a0 2023-10-05T19:28:57.407 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.402+0000 7fcaa6d33700 1 -- 172.21.15.167:0/2066703832 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:53 5.12 5:48000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x563014444f50 con 0x563014413420 2023-10-05T19:28:57.408 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.402+0000 7fcaa7534700 1 -- 172.21.15.167:0/2066703832 <== osd.1 v2:172.21.15.167:6800/30035 20 ==== osd_op_reply(53 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fca9800a5a0 con 0x563014413420 2023-10-05T19:28:57.409 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.402+0000 7fcaa7534700 1 -- 172.21.15.167:0/2066703832 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:54 5.a 5:50000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x563014444f50 con 0x7fca90007030 2023-10-05T19:28:57.409 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.406+0000 7fcaa7534700 1 -- 172.21.15.167:0/2066703832 <== osd.0 v2:172.21.15.118:6800/27690 14 ==== osd_op_reply(54 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fca9800a5a0 con 0x7fca90007030 2023-10-05T19:28:57.409 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.406+0000 7fcaa7534700 1 -- 172.21.15.167:0/2066703832 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:55 5.1a 5:58000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x563014444f50 con 0x563014413420 2023-10-05T19:28:57.410 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.406+0000 7fcaa7534700 1 -- 172.21.15.167:0/2066703832 <== osd.1 v2:172.21.15.167:6800/30035 21 ==== osd_op_reply(55 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fca9800a5a0 con 0x563014413420 2023-10-05T19:28:57.410 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.406+0000 7fcaa7534700 1 -- 172.21.15.167:0/2066703832 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:56 5.6 5:60000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x563014444f50 con 0x7fcaa0022360 2023-10-05T19:28:57.411 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.406+0000 7fcaa6532700 1 -- 172.21.15.167:0/2066703832 <== osd.2 v2:172.21.15.167:6816/30037 9 ==== osd_op_reply(56 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fca90029040 con 0x7fcaa0022360 2023-10-05T19:28:57.411 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.406+0000 7fcaa6532700 1 -- 172.21.15.167:0/2066703832 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:57 5.16 5:68000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x563014444f50 con 0x5630144464a0 2023-10-05T19:28:57.412 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.406+0000 7fcaa6d33700 1 -- 172.21.15.167:0/2066703832 <== osd.3 v2:172.21.15.167:6808/30036 13 ==== osd_op_reply(57 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fcaa0014070 con 0x5630144464a0 2023-10-05T19:28:57.412 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.406+0000 7fcaa6d33700 1 -- 172.21.15.167:0/2066703832 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:58 5.e 5:70000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x563014444f50 con 0x5630144464a0 2023-10-05T19:28:57.413 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.406+0000 7fcaa6d33700 1 -- 172.21.15.167:0/2066703832 <== osd.3 v2:172.21.15.167:6808/30036 14 ==== osd_op_reply(58 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fcaa0014070 con 0x5630144464a0 2023-10-05T19:28:57.413 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.406+0000 7fcaa6d33700 1 -- 172.21.15.167:0/2066703832 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:59 5.1e 5:78000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x563014444f50 con 0x7fca90007030 2023-10-05T19:28:57.413 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.410+0000 7fcaa7534700 1 -- 172.21.15.167:0/2066703832 <== osd.0 v2:172.21.15.118:6800/27690 15 ==== osd_op_reply(59 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fca9803f040 con 0x7fca90007030 2023-10-05T19:28:57.414 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.410+0000 7fcaa7534700 1 -- 172.21.15.167:0/2066703832 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:60 5.1 5:80000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x563014444f50 con 0x5630144464a0 2023-10-05T19:28:57.414 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.410+0000 7fcaa6d33700 1 -- 172.21.15.167:0/2066703832 <== osd.3 v2:172.21.15.167:6808/30036 15 ==== osd_op_reply(60 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fcaa0014070 con 0x5630144464a0 2023-10-05T19:28:57.415 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.410+0000 7fcaa6d33700 1 -- 172.21.15.167:0/2066703832 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:61 5.11 5:88000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x563014444f50 con 0x5630144464a0 2023-10-05T19:28:57.415 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.410+0000 7fcaa6d33700 1 -- 172.21.15.167:0/2066703832 <== osd.3 v2:172.21.15.167:6808/30036 16 ==== osd_op_reply(61 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fcaa0014070 con 0x5630144464a0 2023-10-05T19:28:57.415 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.410+0000 7fcaa6d33700 1 -- 172.21.15.167:0/2066703832 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:62 5.9 5:90000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x563014444f50 con 0x563014413420 2023-10-05T19:28:57.416 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.410+0000 7fcaa7534700 1 -- 172.21.15.167:0/2066703832 <== osd.1 v2:172.21.15.167:6800/30035 22 ==== osd_op_reply(62 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fca9803f040 con 0x563014413420 2023-10-05T19:28:57.416 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.410+0000 7fcaa7534700 1 -- 172.21.15.167:0/2066703832 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:63 5.19 5:98000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x563014444f50 con 0x563014413420 2023-10-05T19:28:57.417 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.410+0000 7fcaa7534700 1 -- 172.21.15.167:0/2066703832 <== osd.1 v2:172.21.15.167:6800/30035 23 ==== osd_op_reply(63 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fca9803f040 con 0x563014413420 2023-10-05T19:28:57.417 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.410+0000 7fcaa7534700 1 -- 172.21.15.167:0/2066703832 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:64 5.5 5:a0000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x563014444f50 con 0x7fca90007030 2023-10-05T19:28:57.418 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.414+0000 7fcaa7534700 1 -- 172.21.15.167:0/2066703832 <== osd.0 v2:172.21.15.118:6800/27690 16 ==== osd_op_reply(64 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fca9803f040 con 0x7fca90007030 2023-10-05T19:28:57.418 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.414+0000 7fcaa7534700 1 -- 172.21.15.167:0/2066703832 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:65 5.15 5:a8000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x563014444f50 con 0x5630144464a0 2023-10-05T19:28:57.419 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.414+0000 7fcaa6d33700 1 -- 172.21.15.167:0/2066703832 <== osd.3 v2:172.21.15.167:6808/30036 17 ==== osd_op_reply(65 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fcaa0014070 con 0x5630144464a0 2023-10-05T19:28:57.419 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.414+0000 7fcaa6d33700 1 -- 172.21.15.167:0/2066703832 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:66 5.d 5:b0000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x563014444f50 con 0x7fcaa0022360 2023-10-05T19:28:57.420 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.414+0000 7fcaa6532700 1 -- 172.21.15.167:0/2066703832 <== osd.2 v2:172.21.15.167:6816/30037 10 ==== osd_op_reply(66 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fca90029040 con 0x7fcaa0022360 2023-10-05T19:28:57.420 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.414+0000 7fcaa6532700 1 -- 172.21.15.167:0/2066703832 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:67 5.1d 5:b8000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x563014444f50 con 0x563014413420 2023-10-05T19:28:57.421 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.414+0000 7fcaa7534700 1 -- 172.21.15.167:0/2066703832 <== osd.1 v2:172.21.15.167:6800/30035 24 ==== osd_op_reply(67 [pgnls start_epoch 32 out=107b] v30'2 uv2 ondisk = 1) v8 ==== 144+0+107 (crc 0 0 0) 0x7fca9803f040 con 0x563014413420 2023-10-05T19:28:57.421 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.414+0000 7fcaa7534700 1 -- 172.21.15.167:0/2066703832 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:68 5.3 5:c0000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x563014444f50 con 0x7fca90007030 2023-10-05T19:28:57.421 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.418+0000 7fcaa7534700 1 -- 172.21.15.167:0/2066703832 <== osd.0 v2:172.21.15.118:6800/27690 17 ==== osd_op_reply(68 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fca9803f040 con 0x7fca90007030 2023-10-05T19:28:57.422 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.418+0000 7fcaa7534700 1 -- 172.21.15.167:0/2066703832 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:69 5.13 5:c8000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x563014444f50 con 0x5630144464a0 2023-10-05T19:28:57.422 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.418+0000 7fcaa6d33700 1 -- 172.21.15.167:0/2066703832 <== osd.3 v2:172.21.15.167:6808/30036 18 ==== osd_op_reply(69 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fcaa0014070 con 0x5630144464a0 2023-10-05T19:28:57.422 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.418+0000 7fcaa6d33700 1 -- 172.21.15.167:0/2066703832 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:70 5.b 5:d0000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x563014444f50 con 0x7fca90007030 2023-10-05T19:28:57.423 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.418+0000 7fcaa7534700 1 -- 172.21.15.167:0/2066703832 <== osd.0 v2:172.21.15.118:6800/27690 18 ==== osd_op_reply(70 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fca9803f040 con 0x7fca90007030 2023-10-05T19:28:57.424 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.418+0000 7fcaa7534700 1 -- 172.21.15.167:0/2066703832 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:71 5.1b 5:d8000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x563014444f50 con 0x7fcaa0022360 2023-10-05T19:28:57.424 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.418+0000 7fcaa6532700 1 -- 172.21.15.167:0/2066703832 <== osd.2 v2:172.21.15.167:6816/30037 11 ==== osd_op_reply(71 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fca90029040 con 0x7fcaa0022360 2023-10-05T19:28:57.424 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.418+0000 7fcaa6532700 1 -- 172.21.15.167:0/2066703832 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:72 5.7 5:e0000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x563014444f50 con 0x5630144464a0 2023-10-05T19:28:57.425 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.418+0000 7fcaa6d33700 1 -- 172.21.15.167:0/2066703832 <== osd.3 v2:172.21.15.167:6808/30036 19 ==== osd_op_reply(72 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fcaa0014070 con 0x5630144464a0 2023-10-05T19:28:57.425 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.418+0000 7fcaa6d33700 1 -- 172.21.15.167:0/2066703832 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:73 5.17 5:e8000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x563014444f50 con 0x7fca90007030 2023-10-05T19:28:57.426 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.422+0000 7fcaa7534700 1 -- 172.21.15.167:0/2066703832 <== osd.0 v2:172.21.15.118:6800/27690 19 ==== osd_op_reply(73 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fca9803f040 con 0x7fca90007030 2023-10-05T19:28:57.426 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.422+0000 7fcaa7534700 1 -- 172.21.15.167:0/2066703832 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:74 5.f 5:f0000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x563014444f50 con 0x563014413420 2023-10-05T19:28:57.427 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.422+0000 7fcaa7534700 1 -- 172.21.15.167:0/2066703832 <== osd.1 v2:172.21.15.167:6800/30035 25 ==== osd_op_reply(74 [pgnls start_epoch 32 out=115b] v23'2 uv2 ondisk = 1) v8 ==== 144+0+115 (crc 0 0 0) 0x7fca9803f040 con 0x563014413420 2023-10-05T19:28:57.427 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.422+0000 7fcaa7534700 1 -- 172.21.15.167:0/2066703832 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:75 5.1f 5:f8000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x563014444f50 con 0x7fcaa0022360 2023-10-05T19:28:57.428 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.422+0000 7fcaa6532700 1 -- 172.21.15.167:0/2066703832 <== osd.2 v2:172.21.15.167:6816/30037 12 ==== osd_op_reply(75 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fca90029040 con 0x7fcaa0022360 2023-10-05T19:28:57.428 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.422+0000 7fcaa91fee40 20 RGWRados::pool_iterate: got zonegroup_info.4f1f719d-9648-417b-b10c-e5d939825419 2023-10-05T19:28:57.428 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.422+0000 7fcaa91fee40 20 RGWRados::pool_iterate: got zone_info.9a9df965-6612-45d4-a45f-6e3c889bacc2 2023-10-05T19:28:57.428 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.422+0000 7fcaa91fee40 20 RGWRados::pool_iterate: got zone_names.default 2023-10-05T19:28:57.429 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.422+0000 7fcaa91fee40 20 RGWRados::pool_iterate: got zonegroups_names.default 2023-10-05T19:28:57.429 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.422+0000 7fcaa91fee40 20 rados->read ofs=0 len=0 2023-10-05T19:28:57.429 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.422+0000 7fcaa91fee40 1 -- 172.21.15.167:0/2066703832 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:76 5.f 5:f43b6ece:::zone_names.default:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32) v8 -- 0x563014449710 con 0x563014413420 2023-10-05T19:28:57.429 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.422+0000 7fcaa7534700 1 -- 172.21.15.167:0/2066703832 <== osd.1 v2:172.21.15.167:6800/30035 26 ==== osd_op_reply(76 zone_names.default [read 0~46 out=46b] v0'0 uv1 ondisk = 0) v8 ==== 162+0+46 (crc 0 0 0) 0x7fca9803f040 con 0x563014413420 2023-10-05T19:28:57.429 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.426+0000 7fcaa91fee40 20 rados_obj.operate() r=0 bl.length=46 2023-10-05T19:28:57.430 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.426+0000 7fcaa91fee40 20 rados->read ofs=0 len=0 2023-10-05T19:28:57.430 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.426+0000 7fcaa91fee40 1 -- 172.21.15.167:0/2066703832 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:77 5.1d 5:b8374eb4:::zone_info.9a9df965-6612-45d4-a45f-6e3c889bacc2:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32) v8 -- 0x563014449ac0 con 0x563014413420 2023-10-05T19:28:57.430 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.426+0000 7fcaa7534700 1 -- 172.21.15.167:0/2066703832 <== osd.1 v2:172.21.15.167:6800/30035 27 ==== osd_op_reply(77 zone_info.9a9df965-6612-45d4-a45f-6e3c889bacc2 [read 0~874 out=874b] v0'0 uv2 ondisk = 0) v8 ==== 190+0+874 (crc 0 0 0) 0x7fca9803f040 con 0x563014413420 2023-10-05T19:28:57.430 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.426+0000 7fcaa91fee40 20 rados_obj.operate() r=0 bl.length=874 2023-10-05T19:28:57.430 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.426+0000 7fcaa91fee40 20 rados->read ofs=0 len=0 2023-10-05T19:28:57.431 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.426+0000 7fcaa91fee40 1 -- 172.21.15.167:0/2066703832 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:78 5.f 5:f4c53578:::zonegroups_names.default:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32) v8 -- 0x563014449ac0 con 0x563014413420 2023-10-05T19:28:57.431 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.426+0000 7fcaa7534700 1 -- 172.21.15.167:0/2066703832 <== osd.1 v2:172.21.15.167:6800/30035 28 ==== osd_op_reply(78 zonegroups_names.default [read 0~46 out=46b] v0'0 uv2 ondisk = 0) v8 ==== 168+0+46 (crc 0 0 0) 0x7fca9803f040 con 0x563014413420 2023-10-05T19:28:57.431 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.426+0000 7fcaa91fee40 20 rados_obj.operate() r=0 bl.length=46 2023-10-05T19:28:57.431 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.426+0000 7fcaa91fee40 20 rados->read ofs=0 len=0 2023-10-05T19:28:57.431 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.426+0000 7fcaa91fee40 1 -- 172.21.15.167:0/2066703832 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:79 5.c 5:30d299a5:::zonegroup_info.4f1f719d-9648-417b-b10c-e5d939825419:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32) v8 -- 0x563014449ac0 con 0x7fca90007030 2023-10-05T19:28:57.432 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.426+0000 7fcaa7534700 1 -- 172.21.15.167:0/2066703832 <== osd.0 v2:172.21.15.118:6800/27690 20 ==== osd_op_reply(79 zonegroup_info.4f1f719d-9648-417b-b10c-e5d939825419 [read 0~362 out=362b] v0'0 uv1 ondisk = 0) v8 ==== 195+0+362 (crc 0 0 0) 0x7fca9803f040 con 0x7fca90007030 2023-10-05T19:28:57.432 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.426+0000 7fcaa91fee40 20 rados_obj.operate() r=0 bl.length=362 2023-10-05T19:28:57.432 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.426+0000 7fcaa91fee40 20 rados->read ofs=0 len=0 2023-10-05T19:28:57.432 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.426+0000 7fcaa91fee40 1 -- 172.21.15.167:0/2066703832 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:80 5.3 5:c0ad8cbd:::region_map:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32) v8 -- 0x563014449f50 con 0x7fca90007030 2023-10-05T19:28:57.433 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.426+0000 7fcaa7534700 1 -- 172.21.15.167:0/2066703832 <== osd.0 v2:172.21.15.118:6800/27690 21 ==== osd_op_reply(80 region_map [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 ==== 154+0+0 (crc 0 0 0) 0x7fca9803f040 con 0x7fca90007030 2023-10-05T19:28:57.433 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.426+0000 7fcaa91fee40 20 rados_obj.operate() r=-2 bl.length=0 2023-10-05T19:28:57.433 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.426+0000 7fcaa91fee40 20 zone default found 2023-10-05T19:28:57.433 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.426+0000 7fcaa91fee40 4 Realm: () 2023-10-05T19:28:57.433 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.426+0000 7fcaa91fee40 4 ZoneGroup: default (4f1f719d-9648-417b-b10c-e5d939825419) 2023-10-05T19:28:57.433 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.426+0000 7fcaa91fee40 4 Zone: default (9a9df965-6612-45d4-a45f-6e3c889bacc2) 2023-10-05T19:28:57.433 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.426+0000 7fcaa91fee40 10 cannot find current period zonegroup using local zonegroup configuration 2023-10-05T19:28:57.434 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.426+0000 7fcaa91fee40 20 zonegroup default 2023-10-05T19:28:57.434 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.426+0000 7fcaa91fee40 20 rados->read ofs=0 len=0 2023-10-05T19:28:57.434 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.426+0000 7fcaa91fee40 1 -- 172.21.15.167:0/2066703832 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:81 5.3 5:c52100b6:::period_config.default:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32) v8 -- 0x563014449f50 con 0x7fca90007030 2023-10-05T19:28:57.435 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.430+0000 7fcaa7534700 1 -- 172.21.15.167:0/2066703832 <== osd.0 v2:172.21.15.118:6800/27690 22 ==== osd_op_reply(81 period_config.default [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 ==== 165+0+0 (crc 0 0 0) 0x7fca9803f040 con 0x7fca90007030 2023-10-05T19:28:57.435 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.430+0000 7fcaa91fee40 20 rados_obj.operate() r=-2 bl.length=0 2023-10-05T19:28:57.435 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.430+0000 7fcaa91fee40 20 rados->read ofs=0 len=0 2023-10-05T19:28:57.435 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.430+0000 7fcaa91fee40 1 -- 172.21.15.167:0/2066703832 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:82 6.2 6:4347d321:::bucket.sync-source-hints.:head [call version.read in=11b,read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32) v8 -- 0x56301444b660 con 0x5630144464a0 2023-10-05T19:28:57.435 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.430+0000 7fcaa6d33700 1 -- 172.21.15.167:0/2066703832 <== osd.3 v2:172.21.15.167:6808/30036 20 ==== osd_op_reply(82 bucket.sync-source-hints. [call,read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 ==== 211+0+0 (crc 0 0 0) 0x7fcaa0014070 con 0x5630144464a0 2023-10-05T19:28:57.435 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.430+0000 7fcaa91fee40 20 rados_obj.operate() r=-2 bl.length=0 2023-10-05T19:28:57.435 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.430+0000 7fcaa91fee40 20 rados->read ofs=0 len=0 2023-10-05T19:28:57.435 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.430+0000 7fcaa91fee40 1 -- 172.21.15.167:0/2066703832 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:83 6.b 6:d467b91b:::bucket.sync-target-hints.:head [call version.read in=11b,read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32) v8 -- 0x56301444ba10 con 0x5630144464a0 2023-10-05T19:28:57.436 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.430+0000 7fcaa6d33700 1 -- 172.21.15.167:0/2066703832 <== osd.3 v2:172.21.15.167:6808/30036 21 ==== osd_op_reply(83 bucket.sync-target-hints. [call,read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 ==== 211+0+0 (crc 0 0 0) 0x7fcaa0014070 con 0x5630144464a0 2023-10-05T19:28:57.436 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.430+0000 7fcaa91fee40 20 rados_obj.operate() r=-2 bl.length=0 2023-10-05T19:28:57.436 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.430+0000 7fcaa91fee40 20 started sync module instance, tier type = 2023-10-05T19:28:57.436 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.430+0000 7fcaa91fee40 20 started zone id=9a9df965-6612-45d4-a45f-6e3c889bacc2 (name=default) with tier type = 2023-10-05T19:28:57.436 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.430+0000 7fcaa91fee40 1 -- 172.21.15.167:0/2066703832 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:84 7.1f 7:f95f44c2:::notify.0:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e32) v8 -- 0x56301444ad20 con 0x7fcaa0022360 2023-10-05T19:28:57.438 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.434+0000 7fcaa6532700 1 -- 172.21.15.167:0/2066703832 <== osd.2 v2:172.21.15.167:6816/30037 13 ==== osd_op_reply(84 notify.0 [create] v32'213 uv213 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7fca90029040 con 0x7fcaa0022360 2023-10-05T19:28:57.439 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.434+0000 7fcaa91fee40 1 -- 172.21.15.167:0/2066703832 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:85 7.1f 7:f95f44c2:::notify.0:head [watch watch cookie 94764498465056] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e32) v8 -- 0x563014444b50 con 0x7fcaa0022360 2023-10-05T19:28:57.439 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.434+0000 7fcaa91fee40 1 -- 172.21.15.167:0/2066703832 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:86 7.0 7:05bf5b68:::notify.1:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e32) v8 -- 0x56301444c7d0 con 0x563014413420 2023-10-05T19:28:57.441 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.434+0000 7fcaa6532700 1 -- 172.21.15.167:0/2066703832 <== osd.2 v2:172.21.15.167:6816/30037 14 ==== osd_op_reply(85 notify.0 [watch watch cookie 94764498465056] v32'214 uv213 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7fca90029040 con 0x7fcaa0022360 2023-10-05T19:28:57.442 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.438+0000 7fcaa7534700 1 -- 172.21.15.167:0/2066703832 <== osd.1 v2:172.21.15.167:6800/30035 29 ==== osd_op_reply(86 notify.1 [create] v32'213 uv213 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7fca98052040 con 0x563014413420 2023-10-05T19:28:57.442 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.438+0000 7fcaa91fee40 1 -- 172.21.15.167:0/2066703832 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:87 7.0 7:05bf5b68:::notify.1:head [watch watch cookie 94764498469600] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e32) v8 -- 0x56301444c730 con 0x563014413420 2023-10-05T19:28:57.442 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.438+0000 7fcaa91fee40 1 -- 172.21.15.167:0/2066703832 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:88 7.15 7:a93a5511:::notify.2:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e32) v8 -- 0x56301444d5b0 con 0x5630144464a0 2023-10-05T19:28:57.445 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.438+0000 7fcaa6d33700 1 -- 172.21.15.167:0/2066703832 <== osd.3 v2:172.21.15.167:6808/30036 22 ==== osd_op_reply(88 notify.2 [create] v32'213 uv213 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7fcaa0014070 con 0x5630144464a0 2023-10-05T19:28:57.445 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.438+0000 7fcaa91fee40 1 -- 172.21.15.167:0/2066703832 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:89 7.15 7:a93a5511:::notify.2:head [watch watch cookie 94764498473152] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e32) v8 -- 0x56301444c730 con 0x5630144464a0 2023-10-05T19:28:57.445 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.438+0000 7fcaa91fee40 1 -- 172.21.15.167:0/2066703832 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:90 7.e 7:7759931f:::notify.3:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e32) v8 -- 0x56301444e550 con 0x7fcaa0022360 2023-10-05T19:28:57.446 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.442+0000 7fcaa7534700 1 -- 172.21.15.167:0/2066703832 <== osd.1 v2:172.21.15.167:6800/30035 30 ==== osd_op_reply(87 notify.1 [watch watch cookie 94764498469600] v32'214 uv213 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7fca98052040 con 0x563014413420 2023-10-05T19:28:57.447 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.442+0000 7fcaa6532700 1 -- 172.21.15.167:0/2066703832 <== osd.2 v2:172.21.15.167:6816/30037 15 ==== osd_op_reply(90 notify.3 [create] v32'213 uv213 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7fca90029040 con 0x7fcaa0022360 2023-10-05T19:28:57.447 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.442+0000 7fcaa91fee40 1 -- 172.21.15.167:0/2066703832 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:91 7.e 7:7759931f:::notify.3:head [watch watch cookie 94764498471888] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e32) v8 -- 0x56301444e140 con 0x7fcaa0022360 2023-10-05T19:28:57.448 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.442+0000 7fcaa91fee40 1 -- 172.21.15.167:0/2066703832 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:92 7.d 7:b4812045:::notify.4:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e32) v8 -- 0x56301444f5f0 con 0x563014413420 2023-10-05T19:28:57.448 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.442+0000 7fcaa6d33700 1 -- 172.21.15.167:0/2066703832 <== osd.3 v2:172.21.15.167:6808/30036 23 ==== osd_op_reply(89 notify.2 [watch watch cookie 94764498473152] v32'214 uv213 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7fcaa0014070 con 0x5630144464a0 2023-10-05T19:28:57.450 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.446+0000 7fcaa6532700 1 -- 172.21.15.167:0/2066703832 <== osd.2 v2:172.21.15.167:6816/30037 16 ==== osd_op_reply(91 notify.3 [watch watch cookie 94764498471888] v32'214 uv213 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7fca90029040 con 0x7fcaa0022360 2023-10-05T19:28:57.450 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.446+0000 7fcaa7534700 1 -- 172.21.15.167:0/2066703832 <== osd.1 v2:172.21.15.167:6800/30035 31 ==== osd_op_reply(92 notify.4 [create] v32'213 uv213 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7fca98052040 con 0x563014413420 2023-10-05T19:28:57.450 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.446+0000 7fcaa91fee40 1 -- 172.21.15.167:0/2066703832 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:93 7.d 7:b4812045:::notify.4:head [watch watch cookie 94764498479440] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e32) v8 -- 0x56301444f0c0 con 0x563014413420 2023-10-05T19:28:57.450 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.446+0000 7fcaa91fee40 1 -- 172.21.15.167:0/2066703832 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:94 7.3 7:c609908c:::notify.5:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e32) v8 -- 0x56301444fee0 con 0x7fca90007030 2023-10-05T19:28:57.453 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.446+0000 7fcaa7534700 1 -- 172.21.15.167:0/2066703832 <== osd.0 v2:172.21.15.118:6800/27690 23 ==== osd_op_reply(94 notify.5 [create] v32'213 uv213 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7fca98052040 con 0x7fca90007030 2023-10-05T19:28:57.453 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.446+0000 7fcaa91fee40 1 -- 172.21.15.167:0/2066703832 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:95 7.3 7:c609908c:::notify.5:head [watch watch cookie 94764498483616] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e32) v8 -- 0x5630144502e0 con 0x7fca90007030 2023-10-05T19:28:57.453 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.446+0000 7fcaa91fee40 1 -- 172.21.15.167:0/2066703832 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:96 7.14 7:2b04a3e9:::notify.6:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e32) v8 -- 0x563014450cf0 con 0x7fca90007030 2023-10-05T19:28:57.453 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.446+0000 7fcaa7534700 1 -- 172.21.15.167:0/2066703832 <== osd.1 v2:172.21.15.167:6800/30035 32 ==== osd_op_reply(93 notify.4 [watch watch cookie 94764498479440] v32'214 uv213 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7fca98052040 con 0x563014413420 2023-10-05T19:28:57.455 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.450+0000 7fcaa7534700 1 -- 172.21.15.167:0/2066703832 <== osd.0 v2:172.21.15.118:6800/27690 24 ==== osd_op_reply(96 notify.6 [create] v32'213 uv213 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7fca98052040 con 0x7fca90007030 2023-10-05T19:28:57.455 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.450+0000 7fcaa91fee40 1 -- 172.21.15.167:0/2066703832 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:97 7.14 7:2b04a3e9:::notify.6:head [watch watch cookie 94764498475360] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e32) v8 -- 0x563014450cf0 con 0x7fca90007030 2023-10-05T19:28:57.455 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.450+0000 7fcaa91fee40 1 -- 172.21.15.167:0/2066703832 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:98 7.9 7:93e5b521:::notify.7:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e32) v8 -- 0x563014451750 con 0x7fca90007030 2023-10-05T19:28:57.456 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.450+0000 7fcaa7534700 1 -- 172.21.15.167:0/2066703832 <== osd.0 v2:172.21.15.118:6800/27690 25 ==== osd_op_reply(95 notify.5 [watch watch cookie 94764498483616] v32'214 uv213 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7fca98052040 con 0x7fca90007030 2023-10-05T19:28:57.458 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.454+0000 7fcaa7534700 1 -- 172.21.15.167:0/2066703832 <== osd.0 v2:172.21.15.118:6800/27690 26 ==== osd_op_reply(98 notify.7 [create] v32'213 uv213 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7fca98052040 con 0x7fca90007030 2023-10-05T19:28:57.458 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.454+0000 7fcaa91fee40 1 -- 172.21.15.167:0/2066703832 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:99 7.9 7:93e5b521:::notify.7:head [watch watch cookie 94764498484720] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e32) v8 -- 0x563014451750 con 0x7fca90007030 2023-10-05T19:28:57.458 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.454+0000 7fcaa91fee40 20 add_watcher() i=0 2023-10-05T19:28:57.458 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.454+0000 7fcaa91fee40 20 add_watcher() i=1 2023-10-05T19:28:57.458 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.454+0000 7fcaa91fee40 20 add_watcher() i=2 2023-10-05T19:28:57.458 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.454+0000 7fcaa91fee40 20 add_watcher() i=3 2023-10-05T19:28:57.458 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.454+0000 7fcaa91fee40 20 add_watcher() i=4 2023-10-05T19:28:57.458 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.454+0000 7fcaa91fee40 20 add_watcher() i=5 2023-10-05T19:28:57.458 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.454+0000 7fcaa7534700 1 -- 172.21.15.167:0/2066703832 <== osd.0 v2:172.21.15.118:6800/27690 27 ==== osd_op_reply(97 notify.6 [watch watch cookie 94764498475360] v32'214 uv213 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7fca98052040 con 0x7fca90007030 2023-10-05T19:28:57.458 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.454+0000 7fcaa91fee40 20 add_watcher() i=6 2023-10-05T19:28:57.460 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.454+0000 7fcaa7534700 1 -- 172.21.15.167:0/2066703832 <== osd.0 v2:172.21.15.118:6800/27690 28 ==== osd_op_reply(99 notify.7 [watch watch cookie 94764498484720] v32'214 uv213 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7fca98052040 con 0x7fca90007030 2023-10-05T19:28:57.460 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.454+0000 7fcaa91fee40 20 add_watcher() i=7 2023-10-05T19:28:57.460 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.454+0000 7fcaa91fee40 2 all 8 watchers are set, enabling cache 2023-10-05T19:28:57.460 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.454+0000 7fcaa91fee40 1 -- 172.21.15.167:0/2066703832 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:100 6.4 6:22d26bf9:::data_loggenerations_metadata:head [call version.check_conds in=74b,call version.read in=11b,read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32) v8 -- 0x563014454330 con 0x563014413420 2023-10-05T19:28:57.460 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.454+0000 7fcaa7534700 1 -- 172.21.15.167:0/2066703832 <== osd.1 v2:172.21.15.167:6800/30035 33 ==== osd_op_reply(100 data_loggenerations_metadata [call,call out=48b,read 0~28 out=28b] v0'0 uv1 ondisk = 0) v8 ==== 256+0+76 (crc 0 0 0) 0x7fca98052040 con 0x563014413420 2023-10-05T19:28:57.461 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.454+0000 7fcaa91fee40 1 -- 172.21.15.167:0/2066703832 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:101 6.4 6:22d26bf9:::data_loggenerations_metadata:head [watch watch cookie 94764498487296] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e32) v8 -- 0x563014451750 con 0x563014413420 2023-10-05T19:28:57.463 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.458+0000 7fcaa7534700 1 -- 172.21.15.167:0/2066703832 <== osd.1 v2:172.21.15.167:6800/30035 34 ==== osd_op_reply(101 data_loggenerations_metadata [watch watch cookie 94764498487296] v32'212 uv1 ondisk = 0) v8 ==== 172+0+0 (crc 0 0 0) 0x7fca98052040 con 0x563014413420 2023-10-05T19:28:57.463 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.458+0000 7fc9f3fff700 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2023-10-05T19:28:57.463 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.458+0000 7fcaa91fee40 20 check_secure_mon_conn(): auth registy supported: methods=[2] modes=[2,1] 2023-10-05T19:28:57.463 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.458+0000 7fcaa91fee40 20 check_secure_mon_conn(): mode 1 is insecure 2023-10-05T19:28:57.464 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.458+0000 7fcaa91fee40 1 Processor -- start 2023-10-05T19:28:57.464 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.458+0000 7fcaa91fee40 1 -- start start 2023-10-05T19:28:57.464 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.458+0000 7fcaa91fee40 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x56301445ebb0 0x56301445ef80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-05T19:28:57.464 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.458+0000 7fcaa91fee40 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x56301445f4c0 con 0x56301445ebb0 2023-10-05T19:28:57.465 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.458+0000 7fcaa6532700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x56301445ebb0 0x56301445ef80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-05T19:28:57.465 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.458+0000 7fcaa6532700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x56301445ebb0 0x56301445ef80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.118:3300/0 says I am v2:172.21.15.167:53514/0 (socket says 172.21.15.167:53514) 2023-10-05T19:28:57.465 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.458+0000 7fcaa6532700 1 -- 172.21.15.167:0/2800354327 learned_addr learned my addr 172.21.15.167:0/2800354327 (peer_addr_for_me v2:172.21.15.167:0/0) 2023-10-05T19:28:57.465 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.458+0000 7fcaa6532700 1 -- 172.21.15.167:0/2800354327 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x56301445f600 con 0x56301445ebb0 2023-10-05T19:28:57.465 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.462+0000 7fcaa6532700 1 --2- 172.21.15.167:0/2800354327 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x56301445ebb0 0x56301445ef80 secure :-1 s=READY pgs=564 cs=0 l=1 rev1=1 crypto rx=0x56301440fec0 tx=0x7fca90030040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3c094c421cb9228 server_cookie=0 in_seq=0 out_seq=0 2023-10-05T19:28:57.466 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.462+0000 7fc9f37fe700 1 -- 172.21.15.167:0/2800354327 <== mon.0 v2:172.21.15.118:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fca90025040 con 0x56301445ebb0 2023-10-05T19:28:57.466 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.462+0000 7fc9f37fe700 1 -- 172.21.15.167:0/2800354327 <== mon.0 v2:172.21.15.118:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fca90034050 con 0x56301445ebb0 2023-10-05T19:28:57.466 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.462+0000 7fc9f37fe700 1 -- 172.21.15.167:0/2800354327 <== mon.0 v2:172.21.15.118:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fca9002e4e0 con 0x56301445ebb0 2023-10-05T19:28:57.467 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.462+0000 7fcaa91fee40 1 -- 172.21.15.167:0/2800354327 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x56301445ebb0 msgr2=0x56301445ef80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-05T19:28:57.467 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.462+0000 7fcaa91fee40 1 --2- 172.21.15.167:0/2800354327 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x56301445ebb0 0x56301445ef80 secure :-1 s=READY pgs=564 cs=0 l=1 rev1=1 crypto rx=0x56301440fec0 tx=0x7fca90030040 comp rx=0 tx=0).stop 2023-10-05T19:28:57.467 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.462+0000 7fcaa91fee40 1 -- 172.21.15.167:0/2800354327 shutdown_connections 2023-10-05T19:28:57.468 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.462+0000 7fcaa91fee40 1 --2- 172.21.15.167:0/2800354327 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x56301445ebb0 0x56301445ef80 unknown :-1 s=CLOSED pgs=564 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-05T19:28:57.468 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.462+0000 7fcaa91fee40 1 -- 172.21.15.167:0/2800354327 >> 172.21.15.167:0/2800354327 conn(0x563014459f70 msgr2=0x5630144580c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-05T19:28:57.468 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.462+0000 7fcaa91fee40 1 -- 172.21.15.167:0/2800354327 shutdown_connections 2023-10-05T19:28:57.468 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.462+0000 7fcaa91fee40 1 -- 172.21.15.167:0/2800354327 wait complete. 2023-10-05T19:28:57.469 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.462+0000 7fcaa91fee40 1 Processor -- start 2023-10-05T19:28:57.469 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.462+0000 7fcaa91fee40 1 -- start start 2023-10-05T19:28:57.469 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.462+0000 7fcaa91fee40 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x56301445ebb0 0x563014464710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-05T19:28:57.469 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.462+0000 7fcaa91fee40 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x563014465710 con 0x56301445ebb0 2023-10-05T19:28:57.469 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.462+0000 7fcaa6532700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x56301445ebb0 0x563014464710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-05T19:28:57.469 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.462+0000 7fcaa6532700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x56301445ebb0 0x563014464710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.118:3300/0 says I am v2:172.21.15.167:53518/0 (socket says 172.21.15.167:53518) 2023-10-05T19:28:57.469 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.462+0000 7fcaa6532700 1 -- 172.21.15.167:0/4236346686 learned_addr learned my addr 172.21.15.167:0/4236346686 (peer_addr_for_me v2:172.21.15.167:0/0) 2023-10-05T19:28:57.470 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.466+0000 7fcaa6532700 1 -- 172.21.15.167:0/4236346686 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x56301440fa50 con 0x56301445ebb0 2023-10-05T19:28:57.470 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.466+0000 7fcaa6532700 1 --2- 172.21.15.167:0/4236346686 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x56301445ebb0 0x563014464710 secure :-1 s=READY pgs=565 cs=0 l=1 rev1=1 crypto rx=0x563014458c30 tx=0x7fca90038000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-05T19:28:57.471 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.466+0000 7fc9f1ffb700 1 -- 172.21.15.167:0/4236346686 <== mon.0 v2:172.21.15.118:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fca90025040 con 0x56301445ebb0 2023-10-05T19:28:57.471 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.466+0000 7fc9f1ffb700 1 -- 172.21.15.167:0/4236346686 <== mon.0 v2:172.21.15.118:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fca90039850 con 0x56301445ebb0 2023-10-05T19:28:57.471 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.466+0000 7fcaa91fee40 1 -- 172.21.15.167:0/4236346686 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x563014464d10 con 0x56301445ebb0 2023-10-05T19:28:57.471 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.466+0000 7fc9f1ffb700 1 -- 172.21.15.167:0/4236346686 <== mon.0 v2:172.21.15.118:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fca90039a00 con 0x56301445ebb0 2023-10-05T19:28:57.471 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.466+0000 7fcaa91fee40 1 -- 172.21.15.167:0/4236346686 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x5630144651b0 con 0x56301445ebb0 2023-10-05T19:28:57.472 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.466+0000 7fc9f1ffb700 1 -- 172.21.15.167:0/4236346686 <== mon.0 v2:172.21.15.118:3300/0 4 ==== mgrmap(e 5) v1 ==== 40329+0+0 (secure 0 0 0) 0x7fca90039b60 con 0x56301445ebb0 2023-10-05T19:28:57.473 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.466+0000 7fc9f1ffb700 1 --2- 172.21.15.167:0/4236346686 >> [v2:172.21.15.167:6824/29889,v1:172.21.15.167:6825/29889] conn(0x7fc9e4030bb0 0x7fc9e4033060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-05T19:28:57.473 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.466+0000 7fc9f1ffb700 1 -- 172.21.15.167:0/4236346686 <== mon.0 v2:172.21.15.118:3300/0 5 ==== osd_map(32..32 src has 1..32) v4 ==== 6260+0+0 (secure 0 0 0) 0x7fca90044070 con 0x56301445ebb0 2023-10-05T19:28:57.473 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.466+0000 7fcaa7534700 1 --2- 172.21.15.167:0/4236346686 >> [v2:172.21.15.167:6824/29889,v1:172.21.15.167:6825/29889] conn(0x7fc9e4030bb0 0x7fc9e4033060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-05T19:28:57.473 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.466+0000 7fcaa91fee40 5 note: GC not initialized 2023-10-05T19:28:57.473 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.470+0000 7fcaa7534700 1 --2- 172.21.15.167:0/4236346686 >> [v2:172.21.15.167:6824/29889,v1:172.21.15.167:6825/29889] conn(0x7fc9e4030bb0 0x7fc9e4033060 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x563014454360 tx=0x7fca9800a280 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-05T19:28:57.475 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.470+0000 7fc9eaffd700 20 reqs_thread_entry: start 2023-10-05T19:28:57.475 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.470+0000 7fc9e8ff9700 20 reqs_thread_entry: start 2023-10-05T19:28:57.475 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.470+0000 7fcaa91fee40 20 init_complete bucket index max shards: 11 2023-10-05T19:28:57.476 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.470+0000 7fcaa91fee40 10 rgw notify: Started notification manager with: 1 workers 2023-10-05T19:28:57.476 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.470+0000 7fc9deffd700 20 rgw notify: INFO: next queues processing will happen at: Thu Oct 5 19:29:27 2023 2023-10-05T19:28:57.476 INFO:teuthology.orchestra.run.smithi167.stderr: 2023-10-05T19:28:57.476 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.470+0000 7fc9de7fc700 20 reqs_thread_entry: start 2023-10-05T19:28:57.476 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.470+0000 7fcaa91fee40 20 get_system_obj_state: rctx=0x7ffd8ff6c338 obj=default.rgw.meta:users.uid:foo state=0x5630142e80b0 s->prefetch_data=0 2023-10-05T19:28:57.477 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.470+0000 7fcaa91fee40 10 cache get: name=default.rgw.meta+users.uid+foo : miss 2023-10-05T19:28:57.477 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.470+0000 7fcaa91fee40 1 -- 172.21.15.167:0/2066703832 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:102 8.18 8:1e6ca5ed:users.uid::foo:head [call version.read in=11b,getxattrs,stat] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32) v8 -- 0x5630144ccfe0 con 0x563014413420 2023-10-05T19:28:57.477 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.470+0000 7fcaa7534700 1 -- 172.21.15.167:0/2066703832 <== osd.1 v2:172.21.15.167:6800/30035 35 ==== osd_op_reply(102 foo [call out=48b,getxattrs out=75b,stat out=16b] v0'0 uv14 ondisk = 0) v8 ==== 231+0+139 (crc 0 0 0) 0x7fca98052040 con 0x563014413420 2023-10-05T19:28:57.477 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.470+0000 7fcaa91fee40 10 cache put: name=default.rgw.meta+users.uid+foo info.flags=0x16 2023-10-05T19:28:57.478 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.470+0000 7fcaa91fee40 10 adding default.rgw.meta+users.uid+foo to cache LRU end 2023-10-05T19:28:57.478 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.470+0000 7fcaa91fee40 10 updating xattr: name=ceph.objclass.version bl.length()=42 2023-10-05T19:28:57.478 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.470+0000 7fcaa91fee40 20 get_system_obj_state: s->obj_tag was set empty 2023-10-05T19:28:57.478 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.470+0000 7fcaa91fee40 20 Read xattr: user.rgw.idtag 2023-10-05T19:28:57.478 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.470+0000 7fcaa91fee40 10 cache get: name=default.rgw.meta+users.uid+foo : type miss (requested=0x13, cached=0x16) 2023-10-05T19:28:57.478 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.470+0000 7fcaa91fee40 20 rados->read ofs=0 len=0 2023-10-05T19:28:57.478 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.470+0000 7fcaa91fee40 1 -- 172.21.15.167:0/2066703832 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:103 8.18 8:1e6ca5ed:users.uid::foo:head [call version.check_conds in=122b,call version.read in=11b,read 0~0,getxattrs] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32) v8 -- 0x5630144cfa70 con 0x563014413420 2023-10-05T19:28:57.479 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.474+0000 7fcaa7534700 1 -- 172.21.15.167:0/2066703832 <== osd.1 v2:172.21.15.167:6800/30035 36 ==== osd_op_reply(103 foo [call,call out=48b,read 0~369 out=369b,getxattrs out=75b] v0'0 uv14 ondisk = 0) v8 ==== 273+0+492 (crc 0 0 0) 0x7fca98052040 con 0x563014413420 2023-10-05T19:28:57.479 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.474+0000 7fcaa91fee40 20 rados_obj.operate() r=0 bl.length=369 2023-10-05T19:28:57.479 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.474+0000 7fcaa91fee40 10 cache put: name=default.rgw.meta+users.uid+foo info.flags=0x13 2023-10-05T19:28:57.479 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.474+0000 7fcaa91fee40 10 moving default.rgw.meta+users.uid+foo to cache LRU end 2023-10-05T19:28:57.479 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.474+0000 7fcaa91fee40 10 updating xattr: name=ceph.objclass.version bl.length()=42 2023-10-05T19:28:57.479 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.474+0000 7fcaa91fee40 1 -- 172.21.15.167:0/2066703832 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:104 8.18 8:1e6ca5ed:users.uid::foo:head [delete,create,call version.set in=58b,writefull 0~357 in=357b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e32) v8 -- 0x5630144d0cf0 con 0x563014413420 2023-10-05T19:28:57.481 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.478+0000 7fcaa7534700 1 -- 172.21.15.167:0/2066703832 <== osd.1 v2:172.21.15.167:6800/30035 37 ==== osd_op_reply(104 foo [delete,create,call,writefull 0~357] v32'15 uv15 ondisk = 0) v8 ==== 273+0+0 (crc 0 0 0) 0x7fca98052040 con 0x563014413420 2023-10-05T19:28:57.482 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.478+0000 7fcaa91fee40 10 cache put: name=default.rgw.meta+users.uid+foo info.flags=0x17 2023-10-05T19:28:57.482 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.478+0000 7fcaa91fee40 10 moving default.rgw.meta+users.uid+foo to cache LRU end 2023-10-05T19:28:57.482 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.478+0000 7fcaa91fee40 10 distributing notification oid=default.rgw.control:notify.4 cni=[op: 0, obj: default.rgw.meta:users.uid:foo, ofs0, ns] 2023-10-05T19:28:57.482 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.478+0000 7fcaa91fee40 1 -- 172.21.15.167:0/2066703832 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:105 7.d 7:b4812045:::notify.4:head [notify cookie 94764499011904 in=545b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32) v8 -- 0x5630144cfa70 con 0x563014413420 2023-10-05T19:28:57.482 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.478+0000 7fcaa7534700 1 -- 172.21.15.167:0/2066703832 <== osd.1 v2:172.21.15.167:6800/30035 38 ==== watch-notify(notify (1) cookie 94764498479440 notify 137438953492 ret 0) v3 ==== 575+0+0 (crc 0 0 0) 0x7fca9804e380 con 0x563014413420 2023-10-05T19:28:57.483 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.478+0000 7fcaa4d2f700 10 rgw watcher librados: RGWWatcher::handle_notify() notify_id 137438953492 cookie 94764498479440 notifier 4653 bl.length()=533 2023-10-05T19:28:57.483 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.478+0000 7fcaa4d2f700 10 rgw watcher librados: cache put: name=default.rgw.meta+users.uid+foo info.flags=0x17 2023-10-05T19:28:57.483 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.478+0000 7fcaa4d2f700 10 rgw watcher librados: moving default.rgw.meta+users.uid+foo to cache LRU end 2023-10-05T19:28:57.483 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.478+0000 7fcaa7534700 1 -- 172.21.15.167:0/2066703832 <== osd.1 v2:172.21.15.167:6800/30035 39 ==== osd_op_reply(105 notify.4 [notify cookie 94764499011904 out=8b] v0'0 uv213 ondisk = 0) v8 ==== 152+0+8 (crc 0 0 0) 0x7fca98052040 con 0x563014413420 2023-10-05T19:28:57.483 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.478+0000 7fcaa4d2f700 1 -- 172.21.15.167:0/2066703832 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:106 7.d 7:b4812045:::notify.4:head [notify-ack in=20b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32) v8 -- 0x7fca8c003d90 con 0x563014413420 2023-10-05T19:28:57.483 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.478+0000 7fcaa7534700 1 -- 172.21.15.167:0/2066703832 <== osd.1 v2:172.21.15.167:6800/30035 40 ==== watch-notify(notify_complete (2) cookie 94764499011904 notify 137438953492 ret 0) v3 ==== 42+0+48 (crc 0 0 0) 0x7fca98036c80 con 0x563014413420 2023-10-05T19:28:57.484 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.478+0000 7fcaa7534700 1 -- 172.21.15.167:0/2066703832 <== osd.1 v2:172.21.15.167:6800/30035 41 ==== osd_op_reply(106 notify.4 [notify-ack] v0'0 uv213 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7fca98052040 con 0x563014413420 2023-10-05T19:28:57.484 INFO:teuthology.orchestra.run.smithi167.stdout:{ 2023-10-05T19:28:57.484 INFO:teuthology.orchestra.run.smithi167.stdout: "user_id": "foo", 2023-10-05T19:28:57.484 INFO:teuthology.orchestra.run.smithi167.stdout: "display_name": "Foo", 2023-10-05T19:28:57.484 INFO:teuthology.orchestra.run.smithi167.stdout: "email": "foo@foo.com", 2023-10-05T19:28:57.484 INFO:teuthology.orchestra.run.smithi167.stdout: "suspended": 0, 2023-10-05T19:28:57.484 INFO:teuthology.orchestra.run.smithi167.stdout: "max_buckets": 4, 2023-10-05T19:28:57.484 INFO:teuthology.orchestra.run.smithi167.stdout: "subusers": [], 2023-10-05T19:28:57.484 INFO:teuthology.orchestra.run.smithi167.stdout: "keys": [ 2023-10-05T19:28:57.485 INFO:teuthology.orchestra.run.smithi167.stdout: { 2023-10-05T19:28:57.485 INFO:teuthology.orchestra.run.smithi167.stdout: "user": "foo", 2023-10-05T19:28:57.485 INFO:teuthology.orchestra.run.smithi167.stdout: "access_key": "9te6NH5mcdcq0Tc5i8i1", 2023-10-05T19:28:57.485 INFO:teuthology.orchestra.run.smithi167.stdout: "secret_key": "Ny4IOauQoL18Gp2zM7lC1vLmoawgqcYP/YGcWfXu" 2023-10-05T19:28:57.485 INFO:teuthology.orchestra.run.smithi167.stdout: } 2023-10-05T19:28:57.485 INFO:teuthology.orchestra.run.smithi167.stdout: ], 2023-10-05T19:28:57.485 INFO:teuthology.orchestra.run.smithi167.stdout: "swift_keys": [], 2023-10-05T19:28:57.485 INFO:teuthology.orchestra.run.smithi167.stdout: "caps": [], 2023-10-05T19:28:57.485 INFO:teuthology.orchestra.run.smithi167.stdout: "op_mask": "read, write, delete", 2023-10-05T19:28:57.485 INFO:teuthology.orchestra.run.smithi167.stdout: "default_placement": "", 2023-10-05T19:28:57.485 INFO:teuthology.orchestra.run.smithi167.stdout: "default_storage_class": "", 2023-10-05T19:28:57.485 INFO:teuthology.orchestra.run.smithi167.stdout: "placement_tags": [], 2023-10-05T19:28:57.485 INFO:teuthology.orchestra.run.smithi167.stdout: "bucket_quota": { 2023-10-05T19:28:57.486 INFO:teuthology.orchestra.run.smithi167.stdout: "enabled": false, 2023-10-05T19:28:57.486 INFO:teuthology.orchestra.run.smithi167.stdout: "check_on_raw": false, 2023-10-05T19:28:57.486 INFO:teuthology.orchestra.run.smithi167.stdout: "max_size": -1, 2023-10-05T19:28:57.486 INFO:teuthology.orchestra.run.smithi167.stdout: "max_size_kb": 0, 2023-10-05T19:28:57.486 INFO:teuthology.orchestra.run.smithi167.stdout: "max_objects": -1 2023-10-05T19:28:57.486 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2023-10-05T19:28:57.486 INFO:teuthology.orchestra.run.smithi167.stdout: "user_quota": { 2023-10-05T19:28:57.486 INFO:teuthology.orchestra.run.smithi167.stdout: "enabled": false, 2023-10-05T19:28:57.486 INFO:teuthology.orchestra.run.smithi167.stdout: "check_on_raw": false, 2023-10-05T19:28:57.486 INFO:teuthology.orchestra.run.smithi167.stdout: "max_size": -1, 2023-10-05T19:28:57.486 INFO:teuthology.orchestra.run.smithi167.stdout: "max_size_kb": 0, 2023-10-05T19:28:57.487 INFO:teuthology.orchestra.run.smithi167.stdout: "max_objects": -1 2023-10-05T19:28:57.487 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2023-10-05T19:28:57.487 INFO:teuthology.orchestra.run.smithi167.stdout: "temp_url_keys": [], 2023-10-05T19:28:57.487 INFO:teuthology.orchestra.run.smithi167.stdout: "type": "rgw", 2023-10-05T19:28:57.487 INFO:teuthology.orchestra.run.smithi167.stdout: "mfa_ids": [] 2023-10-05T19:28:57.487 INFO:teuthology.orchestra.run.smithi167.stdout:} 2023-10-05T19:28:57.487 INFO:teuthology.orchestra.run.smithi167.stdout: 2023-10-05T19:28:57.488 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.478+0000 7fcaa91fee40 1 -- 172.21.15.167:0/2066703832 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:107 7.1f 7:f95f44c2:::notify.0:head [watch unwatch cookie 94764498465056] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e32) v8 -- 0x5630144d0940 con 0x7fcaa0022360 2023-10-05T19:28:57.488 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.482+0000 7fcaa6532700 1 -- 172.21.15.167:0/2066703832 <== osd.2 v2:172.21.15.167:6816/30037 17 ==== osd_op_reply(107 notify.0 [watch unwatch cookie 94764498465056] v32'215 uv213 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7fca9003f020 con 0x7fcaa0022360 2023-10-05T19:28:57.488 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.482+0000 7fcaa91fee40 20 remove_watcher() i=0 2023-10-05T19:28:57.488 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.482+0000 7fcaa91fee40 2 removed watcher, disabling cache 2023-10-05T19:28:57.488 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.482+0000 7fcaa91fee40 1 -- 172.21.15.167:0/2066703832 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:108 7.0 7:05bf5b68:::notify.1:head [watch unwatch cookie 94764498469600] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e32) v8 -- 0x56301444b120 con 0x563014413420 2023-10-05T19:28:57.489 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.482+0000 7fcaa7534700 1 -- 172.21.15.167:0/2066703832 <== osd.1 v2:172.21.15.167:6800/30035 42 ==== osd_op_reply(108 notify.1 [watch unwatch cookie 94764498469600] v32'215 uv213 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7fca98052040 con 0x563014413420 2023-10-05T19:28:57.489 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.486+0000 7fcaa91fee40 20 remove_watcher() i=1 2023-10-05T19:28:57.489 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.486+0000 7fcaa91fee40 1 -- 172.21.15.167:0/2066703832 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:109 7.15 7:a93a5511:::notify.2:head [watch unwatch cookie 94764498473152] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e32) v8 -- 0x56301444b120 con 0x5630144464a0 2023-10-05T19:28:57.492 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.486+0000 7fcaa6d33700 1 -- 172.21.15.167:0/2066703832 <== osd.3 v2:172.21.15.167:6808/30036 24 ==== osd_op_reply(109 notify.2 [watch unwatch cookie 94764498473152] v32'215 uv213 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7fcaa0014070 con 0x5630144464a0 2023-10-05T19:28:57.492 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.486+0000 7fcaa91fee40 20 remove_watcher() i=2 2023-10-05T19:28:57.492 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.486+0000 7fcaa91fee40 1 -- 172.21.15.167:0/2066703832 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:110 7.e 7:7759931f:::notify.3:head [watch unwatch cookie 94764498471888] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e32) v8 -- 0x56301444d0c0 con 0x7fcaa0022360 2023-10-05T19:28:57.494 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.490+0000 7fcaa6532700 1 -- 172.21.15.167:0/2066703832 <== osd.2 v2:172.21.15.167:6816/30037 18 ==== osd_op_reply(110 notify.3 [watch unwatch cookie 94764498471888] v32'215 uv213 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7fca9003f020 con 0x7fcaa0022360 2023-10-05T19:28:57.494 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.490+0000 7fcaa91fee40 20 remove_watcher() i=3 2023-10-05T19:28:57.495 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.490+0000 7fcaa91fee40 1 -- 172.21.15.167:0/2066703832 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:111 7.d 7:b4812045:::notify.4:head [watch unwatch cookie 94764498479440] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e32) v8 -- 0x56301444cbd0 con 0x563014413420 2023-10-05T19:28:57.497 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.490+0000 7fcaa7534700 1 -- 172.21.15.167:0/2066703832 <== osd.1 v2:172.21.15.167:6800/30035 43 ==== osd_op_reply(111 notify.4 [watch unwatch cookie 94764498479440] v32'215 uv213 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7fca98052040 con 0x563014413420 2023-10-05T19:28:57.497 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.490+0000 7fcaa91fee40 20 remove_watcher() i=4 2023-10-05T19:28:57.497 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.490+0000 7fcaa91fee40 1 -- 172.21.15.167:0/2066703832 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:112 7.3 7:c609908c:::notify.5:head [watch unwatch cookie 94764498483616] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e32) v8 -- 0x56301444d0c0 con 0x7fca90007030 2023-10-05T19:28:57.500 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.494+0000 7fcaa7534700 1 -- 172.21.15.167:0/2066703832 <== osd.0 v2:172.21.15.118:6800/27690 29 ==== osd_op_reply(112 notify.5 [watch unwatch cookie 94764498483616] v32'215 uv213 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7fca98052040 con 0x7fca90007030 2023-10-05T19:28:57.500 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.494+0000 7fcaa91fee40 20 remove_watcher() i=5 2023-10-05T19:28:57.500 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.494+0000 7fcaa91fee40 1 -- 172.21.15.167:0/2066703832 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:113 7.14 7:2b04a3e9:::notify.6:head [watch unwatch cookie 94764498475360] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e32) v8 -- 0x56301444e950 con 0x7fca90007030 2023-10-05T19:28:57.503 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.498+0000 7fcaa7534700 1 -- 172.21.15.167:0/2066703832 <== osd.0 v2:172.21.15.118:6800/27690 30 ==== osd_op_reply(113 notify.6 [watch unwatch cookie 94764498475360] v32'215 uv213 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7fca98052040 con 0x7fca90007030 2023-10-05T19:28:57.503 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.498+0000 7fcaa91fee40 20 remove_watcher() i=6 2023-10-05T19:28:57.503 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.498+0000 7fcaa91fee40 1 -- 172.21.15.167:0/2066703832 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:114 7.9 7:93e5b521:::notify.7:head [watch unwatch cookie 94764498484720] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e32) v8 -- 0x56301444f9a0 con 0x7fca90007030 2023-10-05T19:28:57.506 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.502+0000 7fcaa7534700 1 -- 172.21.15.167:0/2066703832 <== osd.0 v2:172.21.15.118:6800/27690 31 ==== osd_op_reply(114 notify.7 [watch unwatch cookie 94764498484720] v32'215 uv213 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7fca98052040 con 0x7fca90007030 2023-10-05T19:28:57.506 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.502+0000 7fcaa91fee40 20 remove_watcher() i=7 2023-10-05T19:28:57.508 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.502+0000 7fc9f3fff700 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2023-10-05T19:28:57.508 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.502+0000 7fcaa91fee40 1 -- 172.21.15.167:0/2066703832 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:115 6.4 6:22d26bf9:::data_loggenerations_metadata:head [watch unwatch cookie 94764498487296] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e32) v8 -- 0x56301444fe90 con 0x563014413420 2023-10-05T19:28:57.512 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.506+0000 7fcaa7534700 1 -- 172.21.15.167:0/2066703832 <== osd.1 v2:172.21.15.167:6800/30035 44 ==== osd_op_reply(115 data_loggenerations_metadata [watch unwatch cookie 94764498487296] v32'213 uv1 ondisk = 0) v8 ==== 172+0+0 (crc 0 0 0) 0x7fca98052040 con 0x563014413420 2023-10-05T19:28:57.512 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.506+0000 7fcaa91fee40 1 -- 172.21.15.167:0/2066703832 >> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] conn(0x7fca90007030 msgr2=0x7fca9001d430 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-05T19:28:57.512 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.506+0000 7fcaa91fee40 1 --2- 172.21.15.167:0/2066703832 >> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] conn(0x7fca90007030 0x7fca9001d430 crc :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-05T19:28:57.512 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.506+0000 7fcaa91fee40 1 -- 172.21.15.167:0/2066703832 >> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] conn(0x563014413420 msgr2=0x563014442830 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-05T19:28:57.512 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.506+0000 7fcaa91fee40 1 --2- 172.21.15.167:0/2066703832 >> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] conn(0x563014413420 0x563014442830 crc :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-05T19:28:57.512 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.506+0000 7fcaa91fee40 1 -- 172.21.15.167:0/2066703832 >> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] conn(0x7fcaa0022360 msgr2=0x7fcaa002d3f0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-05T19:28:57.512 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.506+0000 7fcaa91fee40 1 --2- 172.21.15.167:0/2066703832 >> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] conn(0x7fcaa0022360 0x7fcaa002d3f0 crc :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-05T19:28:57.512 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.506+0000 7fcaa91fee40 1 -- 172.21.15.167:0/2066703832 >> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] conn(0x5630144464a0 msgr2=0x5630144488b0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-05T19:28:57.513 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.506+0000 7fcaa91fee40 1 --2- 172.21.15.167:0/2066703832 >> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] conn(0x5630144464a0 0x5630144488b0 crc :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-05T19:28:57.513 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.506+0000 7fcaa91fee40 1 -- 172.21.15.167:0/2066703832 >> [v2:172.21.15.167:6824/29889,v1:172.21.15.167:6825/29889] conn(0x7fca88030bf0 msgr2=0x7fca880330a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-05T19:28:57.513 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.506+0000 7fcaa91fee40 1 --2- 172.21.15.167:0/2066703832 >> [v2:172.21.15.167:6824/29889,v1:172.21.15.167:6825/29889] conn(0x7fca88030bf0 0x7fca880330a0 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7fca900030c0 tx=0x7fca90014000 comp rx=0 tx=0).stop 2023-10-05T19:28:57.513 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.506+0000 7fcaa7534700 1 -- 172.21.15.167:0/2066703832 reap_dead start 2023-10-05T19:28:57.513 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.506+0000 7fcaa91fee40 1 -- 172.21.15.167:0/2066703832 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x56301440ee40 msgr2=0x563014436da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-05T19:28:57.513 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.506+0000 7fcaa91fee40 1 --2- 172.21.15.167:0/2066703832 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x56301440ee40 0x563014436da0 secure :-1 s=READY pgs=563 cs=0 l=1 rev1=1 crypto rx=0x7fcaa000b960 tx=0x7fcaa0014000 comp rx=0 tx=0).stop 2023-10-05T19:28:57.513 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.506+0000 7fcaa91fee40 1 -- 172.21.15.167:0/2066703832 shutdown_connections 2023-10-05T19:28:57.513 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.506+0000 7fcaa91fee40 1 --2- 172.21.15.167:0/2066703832 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x56301440ee40 0x563014436da0 unknown :-1 s=CLOSED pgs=563 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-05T19:28:57.513 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.506+0000 7fcaa91fee40 1 -- 172.21.15.167:0/2066703832 >> 172.21.15.167:0/2066703832 conn(0x56301440a860 msgr2=0x5630144283b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-05T19:28:57.513 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.506+0000 7fcaa91fee40 1 -- 172.21.15.167:0/2066703832 shutdown_connections 2023-10-05T19:28:57.513 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.506+0000 7fcaa91fee40 1 -- 172.21.15.167:0/2066703832 wait complete. 2023-10-05T19:28:57.515 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.506+0000 7fcaa91fee40 1 -- 172.21.15.167:0/4236346686 >> [v2:172.21.15.167:6824/29889,v1:172.21.15.167:6825/29889] conn(0x7fc9e4030bb0 msgr2=0x7fc9e4033060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-05T19:28:57.515 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.506+0000 7fcaa91fee40 1 --2- 172.21.15.167:0/4236346686 >> [v2:172.21.15.167:6824/29889,v1:172.21.15.167:6825/29889] conn(0x7fc9e4030bb0 0x7fc9e4033060 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x563014454360 tx=0x7fca9800a280 comp rx=0 tx=0).stop 2023-10-05T19:28:57.515 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.506+0000 7fcaa91fee40 1 -- 172.21.15.167:0/4236346686 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x56301445ebb0 msgr2=0x563014464710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-05T19:28:57.515 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.506+0000 7fcaa91fee40 1 --2- 172.21.15.167:0/4236346686 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x56301445ebb0 0x563014464710 secure :-1 s=READY pgs=565 cs=0 l=1 rev1=1 crypto rx=0x563014458c30 tx=0x7fca90038000 comp rx=0 tx=0).stop 2023-10-05T19:28:57.515 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.506+0000 7fcaa91fee40 1 -- 172.21.15.167:0/4236346686 shutdown_connections 2023-10-05T19:28:57.515 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.506+0000 7fcaa91fee40 1 --2- 172.21.15.167:0/4236346686 >> [v2:172.21.15.167:6824/29889,v1:172.21.15.167:6825/29889] conn(0x7fc9e4030bb0 0x7fc9e4033060 unknown :-1 s=CLOSED pgs=195 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-05T19:28:57.515 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.506+0000 7fcaa91fee40 1 --2- 172.21.15.167:0/4236346686 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x56301445ebb0 0x563014464710 unknown :-1 s=CLOSED pgs=565 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-05T19:28:57.515 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.506+0000 7fcaa91fee40 1 -- 172.21.15.167:0/4236346686 >> 172.21.15.167:0/4236346686 conn(0x563014459f70 msgr2=0x563014458010 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-05T19:28:57.515 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.506+0000 7fcaa91fee40 1 -- 172.21.15.167:0/4236346686 shutdown_connections 2023-10-05T19:28:57.515 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.506+0000 7fcaa91fee40 1 -- 172.21.15.167:0/4236346686 wait complete. 2023-10-05T19:28:57.520 DEBUG:tasks.util.rgw: json result: {'user_id': 'foo', 'display_name': 'Foo', 'email': 'foo@foo.com', 'suspended': 0, 'max_buckets': 4, 'subusers': [], 'keys': [{'user': 'foo', 'access_key': '9te6NH5mcdcq0Tc5i8i1', 'secret_key': 'Ny4IOauQoL18Gp2zM7lC1vLmoawgqcYP/YGcWfXu'}], 'swift_keys': [], 'caps': [], 'op_mask': 'read, write, delete', 'default_placement': '', 'default_storage_class': '', 'placement_tags': [], 'bucket_quota': {'enabled': False, 'check_on_raw': False, 'max_size': -1, 'max_size_kb': 0, 'max_objects': -1}, 'user_quota': {'enabled': False, 'check_on_raw': False, 'max_size': -1, 'max_size_kb': 0, 'max_objects': -1}, 'temp_url_keys': [], 'type': 'rgw', 'mfa_ids': []} 2023-10-05T19:28:57.541 INFO:tasks.radosgw_admin:RL: create_bucket myfoo foo bytes_out=0 bytes_in=0 failed=False 2023-10-05T19:28:57.541 INFO:tasks.util.rgw:rgwadmin: client.0 : ['user', 'rm', '--uid', 'foo'] 2023-10-05T19:28:57.541 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'ceph', 'user', 'rm', '--uid', 'foo'] 2023-10-05T19:28:57.541 DEBUG:teuthology.orchestra.run.smithi167:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster ceph user rm --uid foo 2023-10-05T19:28:57.579 INFO:teuthology.orchestra.run.smithi167.stderr:ignoring --setuser ceph since I am not root 2023-10-05T19:28:57.579 INFO:teuthology.orchestra.run.smithi167.stderr:ignoring --setgroup ceph since I am not root 2023-10-05T19:28:57.580 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.574+0000 7f94071c4e40 1 Processor -- start 2023-10-05T19:28:57.580 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.574+0000 7f94071c4e40 1 -- start start 2023-10-05T19:28:57.580 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.574+0000 7f94071c4e40 1 --2- >> v2:172.21.15.118:3300/0 conn(0x55c469eb0310 0x55c469eb26f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-05T19:28:57.580 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.574+0000 7f94071c4e40 1 -- --> v1:172.21.15.118:6789/0 -- auth(proto 0 26 bytes epoch 0) v1 -- 0x55c469df8bd0 con 0x55c469e1fb40 2023-10-05T19:28:57.581 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.574+0000 7f94071c4e40 1 -- --> v2:172.21.15.118:3300/0 -- mon_getmap magic: 0 v1 -- 0x55c469db5ee0 con 0x55c469eb0310 2023-10-05T19:28:57.581 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.574+0000 7f9404cf9700 1 --1- >> v1:172.21.15.118:6789/0 conn(0x55c469e1fb40 0x55c469eaebf0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.118:6789/0 says I am v1:172.21.15.167:48738/0 (socket says 172.21.15.167:48738) 2023-10-05T19:28:57.581 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.574+0000 7f94044f8700 1 --2- >> v2:172.21.15.118:3300/0 conn(0x55c469eb0310 0x55c469eb26f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-05T19:28:57.581 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.574+0000 7f9404cf9700 1 -- 172.21.15.167:0/1988069002 learned_addr learned my addr 172.21.15.167:0/1988069002 (peer_addr_for_me v1:172.21.15.167:0/0) 2023-10-05T19:28:57.582 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.574+0000 7f94044f8700 1 -- 172.21.15.167:0/1988069002 >> v1:172.21.15.118:6789/0 conn(0x55c469e1fb40 legacy=0x55c469eaebf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-05T19:28:57.582 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.578+0000 7f94044f8700 1 -- 172.21.15.167:0/1988069002 --> v2:172.21.15.118:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55c469eb3c40 con 0x55c469eb0310 2023-10-05T19:28:57.582 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.578+0000 7f94044f8700 1 --2- 172.21.15.167:0/1988069002 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55c469eb0310 0x55c469eb26f0 secure :-1 s=READY pgs=567 cs=0 l=1 rev1=1 crypto rx=0x7f93ec00a5d0 tx=0x7f93ec005a50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a15372dc55e87a5 server_cookie=0 in_seq=0 out_seq=0 2023-10-05T19:28:57.582 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.578+0000 7f9403cf7700 1 -- 172.21.15.167:0/1988069002 <== mon.0 v2:172.21.15.118:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f93ec0086a0 con 0x55c469eb0310 2023-10-05T19:28:57.582 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.578+0000 7f9403cf7700 1 -- 172.21.15.167:0/1988069002 <== mon.0 v2:172.21.15.118:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f93ec008800 con 0x55c469eb0310 2023-10-05T19:28:57.583 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.578+0000 7f9403cf7700 1 -- 172.21.15.167:0/1988069002 <== mon.0 v2:172.21.15.118:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f93ec008ad0 con 0x55c469eb0310 2023-10-05T19:28:57.583 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.578+0000 7f94071c4e40 1 -- 172.21.15.167:0/1988069002 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55c469eb0310 msgr2=0x55c469eb26f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-05T19:28:57.583 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.578+0000 7f94071c4e40 1 --2- 172.21.15.167:0/1988069002 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55c469eb0310 0x55c469eb26f0 secure :-1 s=READY pgs=567 cs=0 l=1 rev1=1 crypto rx=0x7f93ec00a5d0 tx=0x7f93ec005a50 comp rx=0 tx=0).stop 2023-10-05T19:28:57.584 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.578+0000 7f94071c4e40 1 -- 172.21.15.167:0/1988069002 shutdown_connections 2023-10-05T19:28:57.584 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.578+0000 7f94071c4e40 1 --2- 172.21.15.167:0/1988069002 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55c469eb0310 0x55c469eb26f0 unknown :-1 s=CLOSED pgs=567 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-05T19:28:57.584 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.578+0000 7f94071c4e40 1 -- 172.21.15.167:0/1988069002 >> 172.21.15.167:0/1988069002 conn(0x55c469ea6740 msgr2=0x55c469ea8b50 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-05T19:28:57.584 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.578+0000 7f94071c4e40 1 -- 172.21.15.167:0/1988069002 shutdown_connections 2023-10-05T19:28:57.584 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.578+0000 7f94071c4e40 1 -- 172.21.15.167:0/1988069002 wait complete. 2023-10-05T19:28:57.585 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.578+0000 7f94071c4e40 1 Processor -- start 2023-10-05T19:28:57.585 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.578+0000 7f94071c4e40 1 -- start start 2023-10-05T19:28:57.585 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.578+0000 7f94071c4e40 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55c469e1fb40 0x55c469eac170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-05T19:28:57.585 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.578+0000 7f94071c4e40 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c469d79e30 con 0x55c469e1fb40 2023-10-05T19:28:57.585 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.578+0000 7f9404cf9700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55c469e1fb40 0x55c469eac170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-05T19:28:57.585 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.578+0000 7f9404cf9700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55c469e1fb40 0x55c469eac170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.118:3300/0 says I am v2:172.21.15.167:53538/0 (socket says 172.21.15.167:53538) 2023-10-05T19:28:57.586 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.578+0000 7f9404cf9700 1 -- 172.21.15.167:0/1964461895 learned_addr learned my addr 172.21.15.167:0/1964461895 (peer_addr_for_me v2:172.21.15.167:0/0) 2023-10-05T19:28:57.586 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.578+0000 7f9404cf9700 1 -- 172.21.15.167:0/1964461895 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f93ec005cf0 con 0x55c469e1fb40 2023-10-05T19:28:57.586 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.582+0000 7f9404cf9700 1 --2- 172.21.15.167:0/1964461895 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55c469e1fb40 0x55c469eac170 secure :-1 s=READY pgs=568 cs=0 l=1 rev1=1 crypto rx=0x7f93fc001c30 tx=0x7f93fc0017e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=43b55ba6e053fdad server_cookie=0 in_seq=0 out_seq=0 2023-10-05T19:28:57.586 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.582+0000 7f9402cf5700 1 -- 172.21.15.167:0/1964461895 <== mon.0 v2:172.21.15.118:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f93fc00bbf0 con 0x55c469e1fb40 2023-10-05T19:28:57.587 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.582+0000 7f9402cf5700 1 -- 172.21.15.167:0/1964461895 <== mon.0 v2:172.21.15.118:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f93fc00bd50 con 0x55c469e1fb40 2023-10-05T19:28:57.587 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.582+0000 7f9402cf5700 1 -- 172.21.15.167:0/1964461895 <== mon.0 v2:172.21.15.118:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f93fc00e470 con 0x55c469e1fb40 2023-10-05T19:28:57.587 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.582+0000 7f94071c4e40 1 -- 172.21.15.167:0/1964461895 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55c469e1fb40 msgr2=0x55c469eac170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-05T19:28:57.588 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.582+0000 7f94071c4e40 1 --2- 172.21.15.167:0/1964461895 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55c469e1fb40 0x55c469eac170 secure :-1 s=READY pgs=568 cs=0 l=1 rev1=1 crypto rx=0x7f93fc001c30 tx=0x7f93fc0017e0 comp rx=0 tx=0).stop 2023-10-05T19:28:57.588 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.582+0000 7f94071c4e40 1 -- 172.21.15.167:0/1964461895 shutdown_connections 2023-10-05T19:28:57.588 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.582+0000 7f94071c4e40 1 --2- 172.21.15.167:0/1964461895 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55c469e1fb40 0x55c469eac170 unknown :-1 s=CLOSED pgs=568 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-05T19:28:57.588 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.582+0000 7f94071c4e40 1 -- 172.21.15.167:0/1964461895 >> 172.21.15.167:0/1964461895 conn(0x55c469ea6740 msgr2=0x55c469ec6ad0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-05T19:28:57.588 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.582+0000 7f94071c4e40 1 -- 172.21.15.167:0/1964461895 shutdown_connections 2023-10-05T19:28:57.588 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.582+0000 7f94071c4e40 1 -- 172.21.15.167:0/1964461895 wait complete. 2023-10-05T19:28:57.589 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.582+0000 7f94071c4e40 1 Processor -- start 2023-10-05T19:28:57.589 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.582+0000 7f94071c4e40 1 -- start start 2023-10-05T19:28:57.589 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.582+0000 7f94071c4e40 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55c469e1fb40 0x55c469ed3ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-05T19:28:57.589 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.582+0000 7f94071c4e40 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c469d4ced0 con 0x55c469e1fb40 2023-10-05T19:28:57.589 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.582+0000 7f9404cf9700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55c469e1fb40 0x55c469ed3ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-05T19:28:57.590 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.582+0000 7f9404cf9700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55c469e1fb40 0x55c469ed3ee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.118:3300/0 says I am v2:172.21.15.167:53550/0 (socket says 172.21.15.167:53550) 2023-10-05T19:28:57.590 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.582+0000 7f9404cf9700 1 -- 172.21.15.167:0/1717304001 learned_addr learned my addr 172.21.15.167:0/1717304001 (peer_addr_for_me v2:172.21.15.167:0/0) 2023-10-05T19:28:57.591 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.586+0000 7f9404cf9700 1 -- 172.21.15.167:0/1717304001 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55c469df8bd0 con 0x55c469e1fb40 2023-10-05T19:28:57.591 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.586+0000 7f9404cf9700 1 --2- 172.21.15.167:0/1717304001 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55c469e1fb40 0x55c469ed3ee0 secure :-1 s=READY pgs=569 cs=0 l=1 rev1=1 crypto rx=0x7f93fc001820 tx=0x7f93fc016000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-05T19:28:57.591 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.586+0000 7f94014f2700 1 -- 172.21.15.167:0/1717304001 <== mon.0 v2:172.21.15.118:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f93fc01f070 con 0x55c469e1fb40 2023-10-05T19:28:57.591 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.586+0000 7f94014f2700 1 -- 172.21.15.167:0/1717304001 <== mon.0 v2:172.21.15.118:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f93fc0197b0 con 0x55c469e1fb40 2023-10-05T19:28:57.591 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.586+0000 7f94071c4e40 1 -- 172.21.15.167:0/1717304001 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x55c469ed4420 con 0x55c469e1fb40 2023-10-05T19:28:57.591 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.586+0000 7f94014f2700 1 -- 172.21.15.167:0/1717304001 <== mon.0 v2:172.21.15.118:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f93fc019960 con 0x55c469e1fb40 2023-10-05T19:28:57.591 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.586+0000 7f94071c4e40 1 -- 172.21.15.167:0/1717304001 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55c469ed4860 con 0x55c469e1fb40 2023-10-05T19:28:57.593 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.586+0000 7f94014f2700 1 -- 172.21.15.167:0/1717304001 <== mon.0 v2:172.21.15.118:3300/0 4 ==== mgrmap(e 5) v1 ==== 40329+0+0 (secure 0 0 0) 0x7f93fc022020 con 0x55c469e1fb40 2023-10-05T19:28:57.593 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.586+0000 7f94071c4e40 20 rados->read ofs=0 len=0 2023-10-05T19:28:57.593 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.586+0000 7f94014f2700 1 --2- 172.21.15.167:0/1717304001 >> [v2:172.21.15.167:6824/29889,v1:172.21.15.167:6825/29889] conn(0x7f93e4030c00 0x7f93e40330b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-05T19:28:57.593 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.586+0000 7f94014f2700 1 -- 172.21.15.167:0/1717304001 <== mon.0 v2:172.21.15.118:3300/0 5 ==== osd_map(32..32 src has 1..32) v4 ==== 6260+0+0 (secure 0 0 0) 0x7f93fc044f50 con 0x55c469e1fb40 2023-10-05T19:28:57.594 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.590+0000 7f94071c4e40 1 --2- 172.21.15.167:0/1717304001 >> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] conn(0x55c469eb0310 0x55c469edf9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-05T19:28:57.594 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.590+0000 7f94044f8700 1 --2- 172.21.15.167:0/1717304001 >> [v2:172.21.15.167:6824/29889,v1:172.21.15.167:6825/29889] conn(0x7f93e4030c00 0x7f93e40330b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-05T19:28:57.594 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.590+0000 7f94071c4e40 1 -- 172.21.15.167:0/1717304001 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:1 5.12 5:49953fa1:::default.realm:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32) v8 -- 0x55c469ee00a0 con 0x55c469eb0310 2023-10-05T19:28:57.594 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.590+0000 7f94054fa700 1 --2- 172.21.15.167:0/1717304001 >> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] conn(0x55c469eb0310 0x55c469edf9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-05T19:28:57.595 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.590+0000 7f94044f8700 1 --2- 172.21.15.167:0/1717304001 >> [v2:172.21.15.167:6824/29889,v1:172.21.15.167:6825/29889] conn(0x7f93e4030c00 0x7f93e40330b0 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7f93ec005a90 tx=0x7f93ec015000 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-05T19:28:57.595 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.590+0000 7f94054fa700 1 --2- 172.21.15.167:0/1717304001 >> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] conn(0x55c469eb0310 0x55c469edf9a0 crc :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-05T19:28:57.595 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.590+0000 7f94054fa700 1 -- 172.21.15.167:0/1717304001 <== osd.1 v2:172.21.15.167:6800/30035 1 ==== osd_op_reply(1 default.realm [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 ==== 157+0+0 (crc 0 0 0) 0x7f93f400a5a0 con 0x55c469eb0310 2023-10-05T19:28:57.595 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.590+0000 7f94071c4e40 20 rados_obj.operate() r=-2 bl.length=0 2023-10-05T19:28:57.596 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.590+0000 7f94071c4e40 20 realm 2023-10-05T19:28:57.596 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.590+0000 7f94071c4e40 20 rados->read ofs=0 len=0 2023-10-05T19:28:57.596 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.590+0000 7f94071c4e40 1 -- 172.21.15.167:0/1717304001 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:2 5.12 5:49953fa1:::default.realm:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32) v8 -- 0x55c469edd000 con 0x55c469eb0310 2023-10-05T19:28:57.597 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.590+0000 7f94054fa700 1 -- 172.21.15.167:0/1717304001 <== osd.1 v2:172.21.15.167:6800/30035 2 ==== osd_op_reply(2 default.realm [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 ==== 157+0+0 (crc 0 0 0) 0x7f93f400a5a0 con 0x55c469eb0310 2023-10-05T19:28:57.597 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.590+0000 7f94071c4e40 20 rados_obj.operate() r=-2 bl.length=0 2023-10-05T19:28:57.597 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.590+0000 7f94071c4e40 4 RGWPeriod::init failed to init realm id : (2) No such file or directory 2023-10-05T19:28:57.597 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.590+0000 7f94071c4e40 20 rados->read ofs=0 len=0 2023-10-05T19:28:57.597 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.590+0000 7f94071c4e40 1 -- 172.21.15.167:0/1717304001 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:3 5.12 5:49953fa1:::default.realm:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32) v8 -- 0x55c469ee0b80 con 0x55c469eb0310 2023-10-05T19:28:57.597 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.590+0000 7f94054fa700 1 -- 172.21.15.167:0/1717304001 <== osd.1 v2:172.21.15.167:6800/30035 3 ==== osd_op_reply(3 default.realm [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 ==== 157+0+0 (crc 0 0 0) 0x7f93f400a5a0 con 0x55c469eb0310 2023-10-05T19:28:57.597 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.590+0000 7f94071c4e40 20 rados_obj.operate() r=-2 bl.length=0 2023-10-05T19:28:57.597 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.590+0000 7f94071c4e40 20 rados->read ofs=0 len=0 2023-10-05T19:28:57.597 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.590+0000 7f94071c4e40 1 -- 172.21.15.167:0/1717304001 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:4 5.f 5:f43b6ece:::zone_names.default:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32) v8 -- 0x55c469ee0790 con 0x55c469eb0310 2023-10-05T19:28:57.599 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.590+0000 7f94054fa700 1 -- 172.21.15.167:0/1717304001 <== osd.1 v2:172.21.15.167:6800/30035 4 ==== osd_op_reply(4 zone_names.default [read 0~46 out=46b] v0'0 uv1 ondisk = 0) v8 ==== 162+0+46 (crc 0 0 0) 0x7f93f400a5a0 con 0x55c469eb0310 2023-10-05T19:28:57.599 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.590+0000 7f94071c4e40 20 rados_obj.operate() r=0 bl.length=46 2023-10-05T19:28:57.599 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.594+0000 7f94071c4e40 20 rados->read ofs=0 len=0 2023-10-05T19:28:57.599 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.594+0000 7f94071c4e40 1 -- 172.21.15.167:0/1717304001 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:5 5.1d 5:b8374eb4:::zone_info.9a9df965-6612-45d4-a45f-6e3c889bacc2:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32) v8 -- 0x55c469ee1a10 con 0x55c469eb0310 2023-10-05T19:28:57.599 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.594+0000 7f94054fa700 1 -- 172.21.15.167:0/1717304001 <== osd.1 v2:172.21.15.167:6800/30035 5 ==== osd_op_reply(5 zone_info.9a9df965-6612-45d4-a45f-6e3c889bacc2 [read 0~874 out=874b] v0'0 uv2 ondisk = 0) v8 ==== 190+0+874 (crc 0 0 0) 0x7f93f400a5a0 con 0x55c469eb0310 2023-10-05T19:28:57.599 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.594+0000 7f94071c4e40 20 rados_obj.operate() r=0 bl.length=874 2023-10-05T19:28:57.599 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.594+0000 7f94071c4e40 20 searching for the correct realm 2023-10-05T19:28:57.599 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.594+0000 7f94071c4e40 1 --2- 172.21.15.167:0/1717304001 >> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] conn(0x55c469ee37d0 0x55c469ee5bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-05T19:28:57.599 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.594+0000 7f94071c4e40 1 -- 172.21.15.167:0/1717304001 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:6 5.0 5:00000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x55c469ee60f0 con 0x55c469ee37d0 2023-10-05T19:28:57.599 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.594+0000 7f9404cf9700 1 --2- 172.21.15.167:0/1717304001 >> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] conn(0x55c469ee37d0 0x55c469ee5bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-05T19:28:57.600 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.594+0000 7f9404cf9700 1 --2- 172.21.15.167:0/1717304001 >> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] conn(0x55c469ee37d0 0x55c469ee5bb0 crc :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-05T19:28:57.600 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.594+0000 7f9404cf9700 1 -- 172.21.15.167:0/1717304001 <== osd.3 v2:172.21.15.167:6808/30036 1 ==== osd_op_reply(6 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f93fc0173e0 con 0x55c469ee37d0 2023-10-05T19:28:57.600 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.594+0000 7f9404cf9700 1 --2- 172.21.15.167:0/1717304001 >> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] conn(0x7f93fc016b60 0x7f93fc0243d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-05T19:28:57.600 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.594+0000 7f9404cf9700 1 -- 172.21.15.167:0/1717304001 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:7 5.10 5:08000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x55c469ee60f0 con 0x7f93fc016b60 2023-10-05T19:28:57.601 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.594+0000 7f94044f8700 1 --2- 172.21.15.167:0/1717304001 >> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] conn(0x7f93fc016b60 0x7f93fc0243d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-05T19:28:57.601 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.594+0000 7f94044f8700 1 --2- 172.21.15.167:0/1717304001 >> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] conn(0x7f93fc016b60 0x7f93fc0243d0 crc :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-05T19:28:57.601 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.598+0000 7f94044f8700 1 -- 172.21.15.167:0/1717304001 <== osd.2 v2:172.21.15.167:6816/30037 1 ==== osd_op_reply(7 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f93ec008a80 con 0x7f93fc016b60 2023-10-05T19:28:57.602 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.598+0000 7f94044f8700 1 --2- 172.21.15.167:0/1717304001 >> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] conn(0x7f93ec007030 0x7f93ec01c410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-05T19:28:57.602 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.598+0000 7f94044f8700 1 -- 172.21.15.167:0/1717304001 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:8 5.8 5:10000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x55c469ee60f0 con 0x7f93ec007030 2023-10-05T19:28:57.602 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.598+0000 7f94054fa700 1 --2- 172.21.15.167:0/1717304001 >> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] conn(0x7f93ec007030 0x7f93ec01c410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-05T19:28:57.602 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.598+0000 7f94054fa700 1 --2- 172.21.15.167:0/1717304001 >> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] conn(0x7f93ec007030 0x7f93ec01c410 crc :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-05T19:28:57.603 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.598+0000 7f94054fa700 1 -- 172.21.15.167:0/1717304001 <== osd.0 v2:172.21.15.118:6800/27690 1 ==== osd_op_reply(8 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f93f400a5a0 con 0x7f93ec007030 2023-10-05T19:28:57.603 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.598+0000 7f94054fa700 1 -- 172.21.15.167:0/1717304001 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:9 5.18 5:18000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x55c469ee60f0 con 0x55c469eb0310 2023-10-05T19:28:57.604 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.598+0000 7f94054fa700 1 -- 172.21.15.167:0/1717304001 <== osd.1 v2:172.21.15.167:6800/30035 6 ==== osd_op_reply(9 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f93f400a5a0 con 0x55c469eb0310 2023-10-05T19:28:57.604 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.598+0000 7f94054fa700 1 -- 172.21.15.167:0/1717304001 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:10 5.4 5:20000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x55c469ee60f0 con 0x55c469eb0310 2023-10-05T19:28:57.604 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.598+0000 7f94054fa700 1 -- 172.21.15.167:0/1717304001 <== osd.1 v2:172.21.15.167:6800/30035 7 ==== osd_op_reply(10 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f93f400a5a0 con 0x55c469eb0310 2023-10-05T19:28:57.605 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.598+0000 7f94054fa700 1 -- 172.21.15.167:0/1717304001 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:11 5.14 5:28000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x55c469ee60f0 con 0x7f93ec007030 2023-10-05T19:28:57.605 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.602+0000 7f94054fa700 1 -- 172.21.15.167:0/1717304001 <== osd.0 v2:172.21.15.118:6800/27690 2 ==== osd_op_reply(11 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f93f400a5a0 con 0x7f93ec007030 2023-10-05T19:28:57.606 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.602+0000 7f94054fa700 1 -- 172.21.15.167:0/1717304001 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:12 5.c 5:30000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x55c469ee60f0 con 0x7f93ec007030 2023-10-05T19:28:57.606 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.602+0000 7f94054fa700 1 -- 172.21.15.167:0/1717304001 <== osd.0 v2:172.21.15.118:6800/27690 3 ==== osd_op_reply(12 [pgnls start_epoch 32 out=112b] v23'1 uv1 ondisk = 1) v8 ==== 144+0+112 (crc 0 0 0) 0x7f93f400a5a0 con 0x7f93ec007030 2023-10-05T19:28:57.607 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.602+0000 7f94054fa700 1 -- 172.21.15.167:0/1717304001 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:13 5.1c 5:38000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x55c469ee60f0 con 0x7f93fc016b60 2023-10-05T19:28:57.607 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.602+0000 7f94044f8700 1 -- 172.21.15.167:0/1717304001 <== osd.2 v2:172.21.15.167:6816/30037 2 ==== osd_op_reply(13 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f93ec008a80 con 0x7f93fc016b60 2023-10-05T19:28:57.607 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.602+0000 7f94044f8700 1 -- 172.21.15.167:0/1717304001 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:14 5.2 5:40000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x55c469ee60f0 con 0x55c469ee37d0 2023-10-05T19:28:57.608 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.602+0000 7f9404cf9700 1 -- 172.21.15.167:0/1717304001 <== osd.3 v2:172.21.15.167:6808/30036 2 ==== osd_op_reply(14 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f93fc0173e0 con 0x55c469ee37d0 2023-10-05T19:28:57.608 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.602+0000 7f9404cf9700 1 -- 172.21.15.167:0/1717304001 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:15 5.12 5:48000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x55c469ee60f0 con 0x55c469eb0310 2023-10-05T19:28:57.608 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.602+0000 7f94054fa700 1 -- 172.21.15.167:0/1717304001 <== osd.1 v2:172.21.15.167:6800/30035 8 ==== osd_op_reply(15 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f93f400a5a0 con 0x55c469eb0310 2023-10-05T19:28:57.608 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.602+0000 7f94054fa700 1 -- 172.21.15.167:0/1717304001 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:16 5.a 5:50000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x55c469ee60f0 con 0x7f93ec007030 2023-10-05T19:28:57.609 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.606+0000 7f94054fa700 1 -- 172.21.15.167:0/1717304001 <== osd.0 v2:172.21.15.118:6800/27690 4 ==== osd_op_reply(16 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f93f400a5a0 con 0x7f93ec007030 2023-10-05T19:28:57.609 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.606+0000 7f94054fa700 1 -- 172.21.15.167:0/1717304001 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:17 5.1a 5:58000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x55c469ee60f0 con 0x55c469eb0310 2023-10-05T19:28:57.610 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.606+0000 7f94054fa700 1 -- 172.21.15.167:0/1717304001 <== osd.1 v2:172.21.15.167:6800/30035 9 ==== osd_op_reply(17 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f93f400a5a0 con 0x55c469eb0310 2023-10-05T19:28:57.610 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.606+0000 7f94054fa700 1 -- 172.21.15.167:0/1717304001 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:18 5.6 5:60000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x55c469ee60f0 con 0x7f93fc016b60 2023-10-05T19:28:57.611 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.606+0000 7f94044f8700 1 -- 172.21.15.167:0/1717304001 <== osd.2 v2:172.21.15.167:6816/30037 3 ==== osd_op_reply(18 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f93ec008a80 con 0x7f93fc016b60 2023-10-05T19:28:57.611 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.606+0000 7f94044f8700 1 -- 172.21.15.167:0/1717304001 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:19 5.16 5:68000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x55c469ee60f0 con 0x55c469ee37d0 2023-10-05T19:28:57.611 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.606+0000 7f9404cf9700 1 -- 172.21.15.167:0/1717304001 <== osd.3 v2:172.21.15.167:6808/30036 3 ==== osd_op_reply(19 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f93fc0173e0 con 0x55c469ee37d0 2023-10-05T19:28:57.612 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.606+0000 7f9404cf9700 1 -- 172.21.15.167:0/1717304001 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:20 5.e 5:70000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x55c469ee60f0 con 0x55c469ee37d0 2023-10-05T19:28:57.612 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.606+0000 7f9404cf9700 1 -- 172.21.15.167:0/1717304001 <== osd.3 v2:172.21.15.167:6808/30036 4 ==== osd_op_reply(20 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f93fc0173e0 con 0x55c469ee37d0 2023-10-05T19:28:57.612 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.606+0000 7f9404cf9700 1 -- 172.21.15.167:0/1717304001 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:21 5.1e 5:78000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x55c469ee60f0 con 0x7f93ec007030 2023-10-05T19:28:57.613 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.610+0000 7f94054fa700 1 -- 172.21.15.167:0/1717304001 <== osd.0 v2:172.21.15.118:6800/27690 5 ==== osd_op_reply(21 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f93f400a5a0 con 0x7f93ec007030 2023-10-05T19:28:57.613 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.610+0000 7f94054fa700 1 -- 172.21.15.167:0/1717304001 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:22 5.1 5:80000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x55c469ee60f0 con 0x55c469ee37d0 2023-10-05T19:28:57.614 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.610+0000 7f9404cf9700 1 -- 172.21.15.167:0/1717304001 <== osd.3 v2:172.21.15.167:6808/30036 5 ==== osd_op_reply(22 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f93fc0173e0 con 0x55c469ee37d0 2023-10-05T19:28:57.614 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.610+0000 7f9404cf9700 1 -- 172.21.15.167:0/1717304001 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:23 5.11 5:88000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x55c469ee60f0 con 0x55c469ee37d0 2023-10-05T19:28:57.614 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.610+0000 7f9404cf9700 1 -- 172.21.15.167:0/1717304001 <== osd.3 v2:172.21.15.167:6808/30036 6 ==== osd_op_reply(23 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f93fc0173e0 con 0x55c469ee37d0 2023-10-05T19:28:57.615 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.610+0000 7f9404cf9700 1 -- 172.21.15.167:0/1717304001 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:24 5.9 5:90000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x55c469ee60f0 con 0x55c469eb0310 2023-10-05T19:28:57.615 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.610+0000 7f94054fa700 1 -- 172.21.15.167:0/1717304001 <== osd.1 v2:172.21.15.167:6800/30035 10 ==== osd_op_reply(24 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f93f400a5a0 con 0x55c469eb0310 2023-10-05T19:28:57.615 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.610+0000 7f94054fa700 1 -- 172.21.15.167:0/1717304001 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:25 5.19 5:98000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x55c469ee60f0 con 0x55c469eb0310 2023-10-05T19:28:57.616 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.610+0000 7f94054fa700 1 -- 172.21.15.167:0/1717304001 <== osd.1 v2:172.21.15.167:6800/30035 11 ==== osd_op_reply(25 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f93f400a5a0 con 0x55c469eb0310 2023-10-05T19:28:57.616 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.610+0000 7f94054fa700 1 -- 172.21.15.167:0/1717304001 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:26 5.5 5:a0000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x55c469ee60f0 con 0x7f93ec007030 2023-10-05T19:28:57.617 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.610+0000 7f94054fa700 1 -- 172.21.15.167:0/1717304001 <== osd.0 v2:172.21.15.118:6800/27690 6 ==== osd_op_reply(26 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f93f400a5a0 con 0x7f93ec007030 2023-10-05T19:28:57.617 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.610+0000 7f94054fa700 1 -- 172.21.15.167:0/1717304001 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:27 5.15 5:a8000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x55c469ee60f0 con 0x55c469ee37d0 2023-10-05T19:28:57.618 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.614+0000 7f9404cf9700 1 -- 172.21.15.167:0/1717304001 <== osd.3 v2:172.21.15.167:6808/30036 7 ==== osd_op_reply(27 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f93fc0173e0 con 0x55c469ee37d0 2023-10-05T19:28:57.618 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.614+0000 7f9404cf9700 1 -- 172.21.15.167:0/1717304001 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:28 5.d 5:b0000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x55c469ee60f0 con 0x7f93fc016b60 2023-10-05T19:28:57.618 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.614+0000 7f94044f8700 1 -- 172.21.15.167:0/1717304001 <== osd.2 v2:172.21.15.167:6816/30037 4 ==== osd_op_reply(28 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f93ec008a80 con 0x7f93fc016b60 2023-10-05T19:28:57.619 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.614+0000 7f94044f8700 1 -- 172.21.15.167:0/1717304001 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:29 5.1d 5:b8000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x55c469ee60f0 con 0x55c469eb0310 2023-10-05T19:28:57.619 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.614+0000 7f94054fa700 1 -- 172.21.15.167:0/1717304001 <== osd.1 v2:172.21.15.167:6800/30035 12 ==== osd_op_reply(29 [pgnls start_epoch 32 out=107b] v30'2 uv2 ondisk = 1) v8 ==== 144+0+107 (crc 0 0 0) 0x7f93f400a5a0 con 0x55c469eb0310 2023-10-05T19:28:57.619 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.614+0000 7f94054fa700 1 -- 172.21.15.167:0/1717304001 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:30 5.3 5:c0000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x55c469ee60f0 con 0x7f93ec007030 2023-10-05T19:28:57.620 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.614+0000 7f94054fa700 1 -- 172.21.15.167:0/1717304001 <== osd.0 v2:172.21.15.118:6800/27690 7 ==== osd_op_reply(30 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f93f400a5a0 con 0x7f93ec007030 2023-10-05T19:28:57.620 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.614+0000 7f94054fa700 1 -- 172.21.15.167:0/1717304001 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:31 5.13 5:c8000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x55c469ee60f0 con 0x55c469ee37d0 2023-10-05T19:28:57.621 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.618+0000 7f9404cf9700 1 -- 172.21.15.167:0/1717304001 <== osd.3 v2:172.21.15.167:6808/30036 8 ==== osd_op_reply(31 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f93fc0173e0 con 0x55c469ee37d0 2023-10-05T19:28:57.621 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.618+0000 7f9404cf9700 1 -- 172.21.15.167:0/1717304001 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:32 5.b 5:d0000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x55c469ee60f0 con 0x7f93ec007030 2023-10-05T19:28:57.622 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.618+0000 7f94054fa700 1 -- 172.21.15.167:0/1717304001 <== osd.0 v2:172.21.15.118:6800/27690 8 ==== osd_op_reply(32 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f93f400a5a0 con 0x7f93ec007030 2023-10-05T19:28:57.622 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.618+0000 7f94054fa700 1 -- 172.21.15.167:0/1717304001 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:33 5.1b 5:d8000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x55c469ee60f0 con 0x7f93fc016b60 2023-10-05T19:28:57.623 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.618+0000 7f94044f8700 1 -- 172.21.15.167:0/1717304001 <== osd.2 v2:172.21.15.167:6816/30037 5 ==== osd_op_reply(33 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f93ec008a80 con 0x7f93fc016b60 2023-10-05T19:28:57.623 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.618+0000 7f94044f8700 1 -- 172.21.15.167:0/1717304001 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:34 5.7 5:e0000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x55c469ee60f0 con 0x55c469ee37d0 2023-10-05T19:28:57.624 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.618+0000 7f9404cf9700 1 -- 172.21.15.167:0/1717304001 <== osd.3 v2:172.21.15.167:6808/30036 9 ==== osd_op_reply(34 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f93fc0173e0 con 0x55c469ee37d0 2023-10-05T19:28:57.624 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.618+0000 7f9404cf9700 1 -- 172.21.15.167:0/1717304001 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:35 5.17 5:e8000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x55c469ee60f0 con 0x7f93ec007030 2023-10-05T19:28:57.625 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.618+0000 7f94054fa700 1 -- 172.21.15.167:0/1717304001 <== osd.0 v2:172.21.15.118:6800/27690 9 ==== osd_op_reply(35 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f93f400a5a0 con 0x7f93ec007030 2023-10-05T19:28:57.625 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.618+0000 7f94054fa700 1 -- 172.21.15.167:0/1717304001 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:36 5.f 5:f0000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x55c469ee60f0 con 0x55c469eb0310 2023-10-05T19:28:57.626 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.622+0000 7f94054fa700 1 -- 172.21.15.167:0/1717304001 <== osd.1 v2:172.21.15.167:6800/30035 13 ==== osd_op_reply(36 [pgnls start_epoch 32 out=115b] v23'2 uv2 ondisk = 1) v8 ==== 144+0+115 (crc 0 0 0) 0x7f93f400a5a0 con 0x55c469eb0310 2023-10-05T19:28:57.626 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.622+0000 7f94054fa700 1 -- 172.21.15.167:0/1717304001 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:37 5.1f 5:f8000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x55c469ee60f0 con 0x7f93fc016b60 2023-10-05T19:28:57.627 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.622+0000 7f94044f8700 1 -- 172.21.15.167:0/1717304001 <== osd.2 v2:172.21.15.167:6816/30037 6 ==== osd_op_reply(37 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f93ec008a80 con 0x7f93fc016b60 2023-10-05T19:28:57.627 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.622+0000 7f94071c4e40 20 RGWRados::pool_iterate: got zonegroup_info.4f1f719d-9648-417b-b10c-e5d939825419 2023-10-05T19:28:57.627 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.622+0000 7f94071c4e40 20 RGWRados::pool_iterate: got zone_info.9a9df965-6612-45d4-a45f-6e3c889bacc2 2023-10-05T19:28:57.627 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.622+0000 7f94071c4e40 20 RGWRados::pool_iterate: got zone_names.default 2023-10-05T19:28:57.627 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.622+0000 7f94071c4e40 20 RGWRados::pool_iterate: got zonegroups_names.default 2023-10-05T19:28:57.627 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.622+0000 7f94071c4e40 20 rados->read ofs=0 len=0 2023-10-05T19:28:57.627 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.622+0000 7f94071c4e40 1 -- 172.21.15.167:0/1717304001 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:38 5.12 5:49953fa1:::default.realm:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32) v8 -- 0x55c469ee2550 con 0x55c469eb0310 2023-10-05T19:28:57.628 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.622+0000 7f94054fa700 1 -- 172.21.15.167:0/1717304001 <== osd.1 v2:172.21.15.167:6800/30035 14 ==== osd_op_reply(38 default.realm [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 ==== 157+0+0 (crc 0 0 0) 0x7f93f400a5a0 con 0x55c469eb0310 2023-10-05T19:28:57.628 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.622+0000 7f94071c4e40 20 rados_obj.operate() r=-2 bl.length=0 2023-10-05T19:28:57.628 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.622+0000 7f94071c4e40 20 rados->read ofs=0 len=0 2023-10-05T19:28:57.629 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.622+0000 7f94071c4e40 1 -- 172.21.15.167:0/1717304001 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:39 5.f 5:f4c53578:::zonegroups_names.default:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32) v8 -- 0x55c469ee2e40 con 0x55c469eb0310 2023-10-05T19:28:57.629 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.622+0000 7f94054fa700 1 -- 172.21.15.167:0/1717304001 <== osd.1 v2:172.21.15.167:6800/30035 15 ==== osd_op_reply(39 zonegroups_names.default [read 0~46 out=46b] v0'0 uv2 ondisk = 0) v8 ==== 168+0+46 (crc 0 0 0) 0x7f93f400a5a0 con 0x55c469eb0310 2023-10-05T19:28:57.629 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.622+0000 7f94071c4e40 20 rados_obj.operate() r=0 bl.length=46 2023-10-05T19:28:57.629 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.622+0000 7f94071c4e40 20 rados->read ofs=0 len=0 2023-10-05T19:28:57.629 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.622+0000 7f94071c4e40 1 -- 172.21.15.167:0/1717304001 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:40 5.c 5:30d299a5:::zonegroup_info.4f1f719d-9648-417b-b10c-e5d939825419:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32) v8 -- 0x55c469ee2e40 con 0x7f93ec007030 2023-10-05T19:28:57.629 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.622+0000 7f94054fa700 1 -- 172.21.15.167:0/1717304001 <== osd.0 v2:172.21.15.118:6800/27690 10 ==== osd_op_reply(40 zonegroup_info.4f1f719d-9648-417b-b10c-e5d939825419 [read 0~362 out=362b] v0'0 uv1 ondisk = 0) v8 ==== 195+0+362 (crc 0 0 0) 0x7f93f400a5a0 con 0x7f93ec007030 2023-10-05T19:28:57.630 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.622+0000 7f94071c4e40 20 rados_obj.operate() r=0 bl.length=362 2023-10-05T19:28:57.630 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.626+0000 7f94071c4e40 20 rados->read ofs=0 len=0 2023-10-05T19:28:57.630 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.626+0000 7f94071c4e40 1 -- 172.21.15.167:0/1717304001 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:41 5.13 5:cbbf6c60:::converted:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32) v8 -- 0x55c469ee2e40 con 0x55c469ee37d0 2023-10-05T19:28:57.630 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.626+0000 7f9404cf9700 1 -- 172.21.15.167:0/1717304001 <== osd.3 v2:172.21.15.167:6808/30036 10 ==== osd_op_reply(41 converted [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 ==== 153+0+0 (crc 0 0 0) 0x7f93fc0173e0 con 0x55c469ee37d0 2023-10-05T19:28:57.630 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.626+0000 7f94071c4e40 20 rados_obj.operate() r=-2 bl.length=0 2023-10-05T19:28:57.631 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.626+0000 7f94071c4e40 20 rados->read ofs=0 len=0 2023-10-05T19:28:57.631 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.626+0000 7f94071c4e40 1 -- 172.21.15.167:0/1717304001 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:42 5.12 5:49953fa1:::default.realm:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32) v8 -- 0x55c469ee6560 con 0x55c469eb0310 2023-10-05T19:28:57.634 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.626+0000 7f94054fa700 1 -- 172.21.15.167:0/1717304001 <== osd.1 v2:172.21.15.167:6800/30035 16 ==== osd_op_reply(42 default.realm [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 ==== 157+0+0 (crc 0 0 0) 0x7f93f400a5a0 con 0x55c469eb0310 2023-10-05T19:28:57.634 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.626+0000 7f94071c4e40 20 rados_obj.operate() r=-2 bl.length=0 2023-10-05T19:28:57.634 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.626+0000 7f94071c4e40 20 rados->read ofs=0 len=0 2023-10-05T19:28:57.634 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.626+0000 7f94071c4e40 1 -- 172.21.15.167:0/1717304001 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:43 5.f 5:f4c53578:::zonegroups_names.default:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32) v8 -- 0x55c469ee2a10 con 0x55c469eb0310 2023-10-05T19:28:57.634 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.626+0000 7f94054fa700 1 -- 172.21.15.167:0/1717304001 <== osd.1 v2:172.21.15.167:6800/30035 17 ==== osd_op_reply(43 zonegroups_names.default [read 0~46 out=46b] v0'0 uv2 ondisk = 0) v8 ==== 168+0+46 (crc 0 0 0) 0x7f93f400a5a0 con 0x55c469eb0310 2023-10-05T19:28:57.634 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.626+0000 7f94071c4e40 20 rados_obj.operate() r=0 bl.length=46 2023-10-05T19:28:57.634 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.626+0000 7f94071c4e40 1 -- 172.21.15.167:0/1717304001 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:44 5.0 5:00000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x55c469ee29e0 con 0x55c469ee37d0 2023-10-05T19:28:57.634 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.626+0000 7f9404cf9700 1 -- 172.21.15.167:0/1717304001 <== osd.3 v2:172.21.15.167:6808/30036 11 ==== osd_op_reply(44 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f93fc0173e0 con 0x55c469ee37d0 2023-10-05T19:28:57.635 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.626+0000 7f9404cf9700 1 -- 172.21.15.167:0/1717304001 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:45 5.10 5:08000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x55c469ee29e0 con 0x7f93fc016b60 2023-10-05T19:28:57.635 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.630+0000 7f94044f8700 1 -- 172.21.15.167:0/1717304001 <== osd.2 v2:172.21.15.167:6816/30037 7 ==== osd_op_reply(45 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f93ec008a80 con 0x7f93fc016b60 2023-10-05T19:28:57.635 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.630+0000 7f94044f8700 1 -- 172.21.15.167:0/1717304001 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:46 5.8 5:10000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x55c469ee29e0 con 0x7f93ec007030 2023-10-05T19:28:57.635 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.630+0000 7f94054fa700 1 -- 172.21.15.167:0/1717304001 <== osd.0 v2:172.21.15.118:6800/27690 11 ==== osd_op_reply(46 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f93f400a5a0 con 0x7f93ec007030 2023-10-05T19:28:57.635 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.630+0000 7f94054fa700 1 -- 172.21.15.167:0/1717304001 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:47 5.18 5:18000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x55c469ee2550 con 0x55c469eb0310 2023-10-05T19:28:57.636 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.630+0000 7f94054fa700 1 -- 172.21.15.167:0/1717304001 <== osd.1 v2:172.21.15.167:6800/30035 18 ==== osd_op_reply(47 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f93f400a5a0 con 0x55c469eb0310 2023-10-05T19:28:57.636 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.630+0000 7f94054fa700 1 -- 172.21.15.167:0/1717304001 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:48 5.4 5:20000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x55c469ee2550 con 0x55c469eb0310 2023-10-05T19:28:57.636 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.630+0000 7f94054fa700 1 -- 172.21.15.167:0/1717304001 <== osd.1 v2:172.21.15.167:6800/30035 19 ==== osd_op_reply(48 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f93f400a5a0 con 0x55c469eb0310 2023-10-05T19:28:57.636 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.630+0000 7f94054fa700 1 -- 172.21.15.167:0/1717304001 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:49 5.14 5:28000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x55c469ee2550 con 0x7f93ec007030 2023-10-05T19:28:57.637 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.634+0000 7f94054fa700 1 -- 172.21.15.167:0/1717304001 <== osd.0 v2:172.21.15.118:6800/27690 12 ==== osd_op_reply(49 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f93f400a5a0 con 0x7f93ec007030 2023-10-05T19:28:57.637 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.634+0000 7f94054fa700 1 -- 172.21.15.167:0/1717304001 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:50 5.c 5:30000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x55c469ee2550 con 0x7f93ec007030 2023-10-05T19:28:57.638 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.634+0000 7f94054fa700 1 -- 172.21.15.167:0/1717304001 <== osd.0 v2:172.21.15.118:6800/27690 13 ==== osd_op_reply(50 [pgnls start_epoch 32 out=112b] v23'1 uv1 ondisk = 1) v8 ==== 144+0+112 (crc 0 0 0) 0x7f93f400a5a0 con 0x7f93ec007030 2023-10-05T19:28:57.638 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.634+0000 7f94054fa700 1 -- 172.21.15.167:0/1717304001 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:51 5.1c 5:38000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x55c469ee2550 con 0x7f93fc016b60 2023-10-05T19:28:57.639 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.634+0000 7f94044f8700 1 -- 172.21.15.167:0/1717304001 <== osd.2 v2:172.21.15.167:6816/30037 8 ==== osd_op_reply(51 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f93ec008a80 con 0x7f93fc016b60 2023-10-05T19:28:57.639 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.634+0000 7f94044f8700 1 -- 172.21.15.167:0/1717304001 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:52 5.2 5:40000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x55c469ee2550 con 0x55c469ee37d0 2023-10-05T19:28:57.640 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.634+0000 7f9404cf9700 1 -- 172.21.15.167:0/1717304001 <== osd.3 v2:172.21.15.167:6808/30036 12 ==== osd_op_reply(52 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f93fc0173e0 con 0x55c469ee37d0 2023-10-05T19:28:57.640 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.634+0000 7f9404cf9700 1 -- 172.21.15.167:0/1717304001 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:53 5.12 5:48000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x55c469ee2550 con 0x55c469eb0310 2023-10-05T19:28:57.641 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.634+0000 7f94054fa700 1 -- 172.21.15.167:0/1717304001 <== osd.1 v2:172.21.15.167:6800/30035 20 ==== osd_op_reply(53 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f93f400a5a0 con 0x55c469eb0310 2023-10-05T19:28:57.641 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.634+0000 7f94054fa700 1 -- 172.21.15.167:0/1717304001 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:54 5.a 5:50000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x55c469ee2550 con 0x7f93ec007030 2023-10-05T19:28:57.642 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.638+0000 7f94054fa700 1 -- 172.21.15.167:0/1717304001 <== osd.0 v2:172.21.15.118:6800/27690 14 ==== osd_op_reply(54 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f93f400a5a0 con 0x7f93ec007030 2023-10-05T19:28:57.642 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.638+0000 7f94054fa700 1 -- 172.21.15.167:0/1717304001 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:55 5.1a 5:58000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x55c469ee2550 con 0x55c469eb0310 2023-10-05T19:28:57.642 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.638+0000 7f94054fa700 1 -- 172.21.15.167:0/1717304001 <== osd.1 v2:172.21.15.167:6800/30035 21 ==== osd_op_reply(55 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f93f400a5a0 con 0x55c469eb0310 2023-10-05T19:28:57.642 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.638+0000 7f94054fa700 1 -- 172.21.15.167:0/1717304001 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:56 5.6 5:60000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x55c469ee2550 con 0x7f93fc016b60 2023-10-05T19:28:57.643 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.638+0000 7f94044f8700 1 -- 172.21.15.167:0/1717304001 <== osd.2 v2:172.21.15.167:6816/30037 9 ==== osd_op_reply(56 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f93ec008a80 con 0x7f93fc016b60 2023-10-05T19:28:57.643 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.638+0000 7f94044f8700 1 -- 172.21.15.167:0/1717304001 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:57 5.16 5:68000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x55c469ee2550 con 0x55c469ee37d0 2023-10-05T19:28:57.644 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.638+0000 7f9404cf9700 1 -- 172.21.15.167:0/1717304001 <== osd.3 v2:172.21.15.167:6808/30036 13 ==== osd_op_reply(57 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f93fc0173e0 con 0x55c469ee37d0 2023-10-05T19:28:57.644 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.638+0000 7f9404cf9700 1 -- 172.21.15.167:0/1717304001 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:58 5.e 5:70000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x55c469ee2550 con 0x55c469ee37d0 2023-10-05T19:28:57.645 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.638+0000 7f9404cf9700 1 -- 172.21.15.167:0/1717304001 <== osd.3 v2:172.21.15.167:6808/30036 14 ==== osd_op_reply(58 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f93fc0173e0 con 0x55c469ee37d0 2023-10-05T19:28:57.645 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.638+0000 7f9404cf9700 1 -- 172.21.15.167:0/1717304001 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:59 5.1e 5:78000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x55c469ee2550 con 0x7f93ec007030 2023-10-05T19:28:57.646 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.642+0000 7f94054fa700 1 -- 172.21.15.167:0/1717304001 <== osd.0 v2:172.21.15.118:6800/27690 15 ==== osd_op_reply(59 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f93f403f040 con 0x7f93ec007030 2023-10-05T19:28:57.646 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.642+0000 7f94054fa700 1 -- 172.21.15.167:0/1717304001 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:60 5.1 5:80000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x55c469ee2550 con 0x55c469ee37d0 2023-10-05T19:28:57.647 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.642+0000 7f9404cf9700 1 -- 172.21.15.167:0/1717304001 <== osd.3 v2:172.21.15.167:6808/30036 15 ==== osd_op_reply(60 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f93fc0173e0 con 0x55c469ee37d0 2023-10-05T19:28:57.647 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.642+0000 7f9404cf9700 1 -- 172.21.15.167:0/1717304001 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:61 5.11 5:88000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x55c469ee2550 con 0x55c469ee37d0 2023-10-05T19:28:57.648 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.642+0000 7f9404cf9700 1 -- 172.21.15.167:0/1717304001 <== osd.3 v2:172.21.15.167:6808/30036 16 ==== osd_op_reply(61 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f93fc0173e0 con 0x55c469ee37d0 2023-10-05T19:28:57.648 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.642+0000 7f9404cf9700 1 -- 172.21.15.167:0/1717304001 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:62 5.9 5:90000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x55c469ee2550 con 0x55c469eb0310 2023-10-05T19:28:57.649 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.642+0000 7f94054fa700 1 -- 172.21.15.167:0/1717304001 <== osd.1 v2:172.21.15.167:6800/30035 22 ==== osd_op_reply(62 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f93f403f040 con 0x55c469eb0310 2023-10-05T19:28:57.649 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.642+0000 7f94054fa700 1 -- 172.21.15.167:0/1717304001 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:63 5.19 5:98000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x55c469ee2550 con 0x55c469eb0310 2023-10-05T19:28:57.650 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.646+0000 7f94054fa700 1 -- 172.21.15.167:0/1717304001 <== osd.1 v2:172.21.15.167:6800/30035 23 ==== osd_op_reply(63 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f93f403f040 con 0x55c469eb0310 2023-10-05T19:28:57.650 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.646+0000 7f94054fa700 1 -- 172.21.15.167:0/1717304001 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:64 5.5 5:a0000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x55c469ee2550 con 0x7f93ec007030 2023-10-05T19:28:57.650 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.646+0000 7f94054fa700 1 -- 172.21.15.167:0/1717304001 <== osd.0 v2:172.21.15.118:6800/27690 16 ==== osd_op_reply(64 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f93f403f040 con 0x7f93ec007030 2023-10-05T19:28:57.651 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.646+0000 7f94054fa700 1 -- 172.21.15.167:0/1717304001 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:65 5.15 5:a8000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x55c469ee2550 con 0x55c469ee37d0 2023-10-05T19:28:57.651 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.646+0000 7f9404cf9700 1 -- 172.21.15.167:0/1717304001 <== osd.3 v2:172.21.15.167:6808/30036 17 ==== osd_op_reply(65 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f93fc0173e0 con 0x55c469ee37d0 2023-10-05T19:28:57.651 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.646+0000 7f9404cf9700 1 -- 172.21.15.167:0/1717304001 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:66 5.d 5:b0000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x55c469ee2550 con 0x7f93fc016b60 2023-10-05T19:28:57.652 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.646+0000 7f94044f8700 1 -- 172.21.15.167:0/1717304001 <== osd.2 v2:172.21.15.167:6816/30037 10 ==== osd_op_reply(66 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f93ec008a80 con 0x7f93fc016b60 2023-10-05T19:28:57.652 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.646+0000 7f94044f8700 1 -- 172.21.15.167:0/1717304001 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:67 5.1d 5:b8000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x55c469ee2550 con 0x55c469eb0310 2023-10-05T19:28:57.653 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.646+0000 7f94054fa700 1 -- 172.21.15.167:0/1717304001 <== osd.1 v2:172.21.15.167:6800/30035 24 ==== osd_op_reply(67 [pgnls start_epoch 32 out=107b] v30'2 uv2 ondisk = 1) v8 ==== 144+0+107 (crc 0 0 0) 0x7f93f403f040 con 0x55c469eb0310 2023-10-05T19:28:57.653 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.650+0000 7f94054fa700 1 -- 172.21.15.167:0/1717304001 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:68 5.3 5:c0000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x55c469ee2550 con 0x7f93ec007030 2023-10-05T19:28:57.654 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.650+0000 7f94054fa700 1 -- 172.21.15.167:0/1717304001 <== osd.0 v2:172.21.15.118:6800/27690 17 ==== osd_op_reply(68 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f93f403f040 con 0x7f93ec007030 2023-10-05T19:28:57.654 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.650+0000 7f94054fa700 1 -- 172.21.15.167:0/1717304001 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:69 5.13 5:c8000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x55c469ee2550 con 0x55c469ee37d0 2023-10-05T19:28:57.655 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.650+0000 7f9404cf9700 1 -- 172.21.15.167:0/1717304001 <== osd.3 v2:172.21.15.167:6808/30036 18 ==== osd_op_reply(69 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f93fc0173e0 con 0x55c469ee37d0 2023-10-05T19:28:57.655 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.650+0000 7f9404cf9700 1 -- 172.21.15.167:0/1717304001 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:70 5.b 5:d0000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x55c469ee2550 con 0x7f93ec007030 2023-10-05T19:28:57.656 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.650+0000 7f94054fa700 1 -- 172.21.15.167:0/1717304001 <== osd.0 v2:172.21.15.118:6800/27690 18 ==== osd_op_reply(70 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f93f403f040 con 0x7f93ec007030 2023-10-05T19:28:57.656 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.650+0000 7f94054fa700 1 -- 172.21.15.167:0/1717304001 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:71 5.1b 5:d8000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x55c469ee2550 con 0x7f93fc016b60 2023-10-05T19:28:57.657 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.650+0000 7f94044f8700 1 -- 172.21.15.167:0/1717304001 <== osd.2 v2:172.21.15.167:6816/30037 11 ==== osd_op_reply(71 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f93ec008a80 con 0x7f93fc016b60 2023-10-05T19:28:57.657 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.650+0000 7f94044f8700 1 -- 172.21.15.167:0/1717304001 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:72 5.7 5:e0000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x55c469ee2550 con 0x55c469ee37d0 2023-10-05T19:28:57.658 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.654+0000 7f9404cf9700 1 -- 172.21.15.167:0/1717304001 <== osd.3 v2:172.21.15.167:6808/30036 19 ==== osd_op_reply(72 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f93fc050040 con 0x55c469ee37d0 2023-10-05T19:28:57.658 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.654+0000 7f9404cf9700 1 -- 172.21.15.167:0/1717304001 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:73 5.17 5:e8000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x55c469ee2550 con 0x7f93ec007030 2023-10-05T19:28:57.659 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.654+0000 7f94054fa700 1 -- 172.21.15.167:0/1717304001 <== osd.0 v2:172.21.15.118:6800/27690 19 ==== osd_op_reply(73 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f93f403f040 con 0x7f93ec007030 2023-10-05T19:28:57.659 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.654+0000 7f94054fa700 1 -- 172.21.15.167:0/1717304001 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:74 5.f 5:f0000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x55c469ee2550 con 0x55c469eb0310 2023-10-05T19:28:57.660 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.654+0000 7f94054fa700 1 -- 172.21.15.167:0/1717304001 <== osd.1 v2:172.21.15.167:6800/30035 25 ==== osd_op_reply(74 [pgnls start_epoch 32 out=115b] v23'2 uv2 ondisk = 1) v8 ==== 144+0+115 (crc 0 0 0) 0x7f93f403f040 con 0x55c469eb0310 2023-10-05T19:28:57.660 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.654+0000 7f94054fa700 1 -- 172.21.15.167:0/1717304001 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:75 5.1f 5:f8000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x55c469ee2550 con 0x7f93fc016b60 2023-10-05T19:28:57.661 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.654+0000 7f94044f8700 1 -- 172.21.15.167:0/1717304001 <== osd.2 v2:172.21.15.167:6816/30037 12 ==== osd_op_reply(75 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f93ec008a80 con 0x7f93fc016b60 2023-10-05T19:28:57.661 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.654+0000 7f94071c4e40 20 RGWRados::pool_iterate: got zonegroup_info.4f1f719d-9648-417b-b10c-e5d939825419 2023-10-05T19:28:57.661 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.654+0000 7f94071c4e40 20 RGWRados::pool_iterate: got zone_info.9a9df965-6612-45d4-a45f-6e3c889bacc2 2023-10-05T19:28:57.661 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.654+0000 7f94071c4e40 20 RGWRados::pool_iterate: got zone_names.default 2023-10-05T19:28:57.661 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.654+0000 7f94071c4e40 20 RGWRados::pool_iterate: got zonegroups_names.default 2023-10-05T19:28:57.661 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.654+0000 7f94071c4e40 20 rados->read ofs=0 len=0 2023-10-05T19:28:57.661 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.654+0000 7f94071c4e40 1 -- 172.21.15.167:0/1717304001 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:76 5.f 5:f43b6ece:::zone_names.default:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32) v8 -- 0x55c469ee29e0 con 0x55c469eb0310 2023-10-05T19:28:57.662 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.658+0000 7f94054fa700 1 -- 172.21.15.167:0/1717304001 <== osd.1 v2:172.21.15.167:6800/30035 26 ==== osd_op_reply(76 zone_names.default [read 0~46 out=46b] v0'0 uv1 ondisk = 0) v8 ==== 162+0+46 (crc 0 0 0) 0x7f93f403f040 con 0x55c469eb0310 2023-10-05T19:28:57.662 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.658+0000 7f94071c4e40 20 rados_obj.operate() r=0 bl.length=46 2023-10-05T19:28:57.662 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.658+0000 7f94071c4e40 20 rados->read ofs=0 len=0 2023-10-05T19:28:57.662 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.658+0000 7f94071c4e40 1 -- 172.21.15.167:0/1717304001 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:77 5.1d 5:b8374eb4:::zone_info.9a9df965-6612-45d4-a45f-6e3c889bacc2:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32) v8 -- 0x55c469ee6a50 con 0x55c469eb0310 2023-10-05T19:28:57.663 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.658+0000 7f94054fa700 1 -- 172.21.15.167:0/1717304001 <== osd.1 v2:172.21.15.167:6800/30035 27 ==== osd_op_reply(77 zone_info.9a9df965-6612-45d4-a45f-6e3c889bacc2 [read 0~874 out=874b] v0'0 uv2 ondisk = 0) v8 ==== 190+0+874 (crc 0 0 0) 0x7f93f403f040 con 0x55c469eb0310 2023-10-05T19:28:57.663 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.658+0000 7f94071c4e40 20 rados_obj.operate() r=0 bl.length=874 2023-10-05T19:28:57.663 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.658+0000 7f94071c4e40 20 rados->read ofs=0 len=0 2023-10-05T19:28:57.663 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.658+0000 7f94071c4e40 1 -- 172.21.15.167:0/1717304001 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:78 5.f 5:f4c53578:::zonegroups_names.default:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32) v8 -- 0x55c469ee6c60 con 0x55c469eb0310 2023-10-05T19:28:57.663 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.658+0000 7f94054fa700 1 -- 172.21.15.167:0/1717304001 <== osd.1 v2:172.21.15.167:6800/30035 28 ==== osd_op_reply(78 zonegroups_names.default [read 0~46 out=46b] v0'0 uv2 ondisk = 0) v8 ==== 168+0+46 (crc 0 0 0) 0x7f93f403f040 con 0x55c469eb0310 2023-10-05T19:28:57.663 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.658+0000 7f94071c4e40 20 rados_obj.operate() r=0 bl.length=46 2023-10-05T19:28:57.664 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.658+0000 7f94071c4e40 20 rados->read ofs=0 len=0 2023-10-05T19:28:57.664 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.658+0000 7f94071c4e40 1 -- 172.21.15.167:0/1717304001 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:79 5.c 5:30d299a5:::zonegroup_info.4f1f719d-9648-417b-b10c-e5d939825419:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32) v8 -- 0x55c469ee6c80 con 0x7f93ec007030 2023-10-05T19:28:57.664 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.658+0000 7f94054fa700 1 -- 172.21.15.167:0/1717304001 <== osd.0 v2:172.21.15.118:6800/27690 20 ==== osd_op_reply(79 zonegroup_info.4f1f719d-9648-417b-b10c-e5d939825419 [read 0~362 out=362b] v0'0 uv1 ondisk = 0) v8 ==== 195+0+362 (crc 0 0 0) 0x7f93f403f040 con 0x7f93ec007030 2023-10-05T19:28:57.664 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.658+0000 7f94071c4e40 20 rados_obj.operate() r=0 bl.length=362 2023-10-05T19:28:57.664 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.658+0000 7f94071c4e40 20 rados->read ofs=0 len=0 2023-10-05T19:28:57.665 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.658+0000 7f94071c4e40 1 -- 172.21.15.167:0/1717304001 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:80 5.3 5:c0ad8cbd:::region_map:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32) v8 -- 0x55c469ee7610 con 0x7f93ec007030 2023-10-05T19:28:57.665 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.658+0000 7f94054fa700 1 -- 172.21.15.167:0/1717304001 <== osd.0 v2:172.21.15.118:6800/27690 21 ==== osd_op_reply(80 region_map [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 ==== 154+0+0 (crc 0 0 0) 0x7f93f403f040 con 0x7f93ec007030 2023-10-05T19:28:57.665 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.662+0000 7f94071c4e40 20 rados_obj.operate() r=-2 bl.length=0 2023-10-05T19:28:57.666 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.662+0000 7f94071c4e40 20 zone default found 2023-10-05T19:28:57.666 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.662+0000 7f94071c4e40 4 Realm: () 2023-10-05T19:28:57.666 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.662+0000 7f94071c4e40 4 ZoneGroup: default (4f1f719d-9648-417b-b10c-e5d939825419) 2023-10-05T19:28:57.666 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.662+0000 7f94071c4e40 4 Zone: default (9a9df965-6612-45d4-a45f-6e3c889bacc2) 2023-10-05T19:28:57.666 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.662+0000 7f94071c4e40 10 cannot find current period zonegroup using local zonegroup configuration 2023-10-05T19:28:57.666 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.662+0000 7f94071c4e40 20 zonegroup default 2023-10-05T19:28:57.666 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.662+0000 7f94071c4e40 20 rados->read ofs=0 len=0 2023-10-05T19:28:57.666 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.662+0000 7f94071c4e40 1 -- 172.21.15.167:0/1717304001 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:81 5.3 5:c52100b6:::period_config.default:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32) v8 -- 0x55c469ee7610 con 0x7f93ec007030 2023-10-05T19:28:57.667 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.662+0000 7f94054fa700 1 -- 172.21.15.167:0/1717304001 <== osd.0 v2:172.21.15.118:6800/27690 22 ==== osd_op_reply(81 period_config.default [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 ==== 165+0+0 (crc 0 0 0) 0x7f93f403f040 con 0x7f93ec007030 2023-10-05T19:28:57.667 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.662+0000 7f94071c4e40 20 rados_obj.operate() r=-2 bl.length=0 2023-10-05T19:28:57.667 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.662+0000 7f94071c4e40 20 rados->read ofs=0 len=0 2023-10-05T19:28:57.667 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.662+0000 7f94071c4e40 1 -- 172.21.15.167:0/1717304001 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:82 6.2 6:4347d321:::bucket.sync-source-hints.:head [call version.read in=11b,read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32) v8 -- 0x55c469ee8eb0 con 0x55c469ee37d0 2023-10-05T19:28:57.667 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.662+0000 7f9404cf9700 1 -- 172.21.15.167:0/1717304001 <== osd.3 v2:172.21.15.167:6808/30036 20 ==== osd_op_reply(82 bucket.sync-source-hints. [call,read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 ==== 211+0+0 (crc 0 0 0) 0x7f93fc050040 con 0x55c469ee37d0 2023-10-05T19:28:57.668 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.662+0000 7f94071c4e40 20 rados_obj.operate() r=-2 bl.length=0 2023-10-05T19:28:57.668 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.662+0000 7f94071c4e40 20 rados->read ofs=0 len=0 2023-10-05T19:28:57.668 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.662+0000 7f94071c4e40 1 -- 172.21.15.167:0/1717304001 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:83 6.b 6:d467b91b:::bucket.sync-target-hints.:head [call version.read in=11b,read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32) v8 -- 0x55c469ee9260 con 0x55c469ee37d0 2023-10-05T19:28:57.668 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.662+0000 7f9404cf9700 1 -- 172.21.15.167:0/1717304001 <== osd.3 v2:172.21.15.167:6808/30036 21 ==== osd_op_reply(83 bucket.sync-target-hints. [call,read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 ==== 211+0+0 (crc 0 0 0) 0x7f93fc050040 con 0x55c469ee37d0 2023-10-05T19:28:57.668 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.662+0000 7f94071c4e40 20 rados_obj.operate() r=-2 bl.length=0 2023-10-05T19:28:57.668 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.662+0000 7f94071c4e40 20 started sync module instance, tier type = 2023-10-05T19:28:57.669 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.662+0000 7f94071c4e40 20 started zone id=9a9df965-6612-45d4-a45f-6e3c889bacc2 (name=default) with tier type = 2023-10-05T19:28:57.669 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.662+0000 7f94071c4e40 1 -- 172.21.15.167:0/1717304001 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:84 7.1f 7:f95f44c2:::notify.0:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e32) v8 -- 0x55c469ee8570 con 0x7f93fc016b60 2023-10-05T19:28:57.671 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.666+0000 7f94044f8700 1 -- 172.21.15.167:0/1717304001 <== osd.2 v2:172.21.15.167:6816/30037 13 ==== osd_op_reply(84 notify.0 [create] v32'216 uv216 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f93ec008a80 con 0x7f93fc016b60 2023-10-05T19:28:57.671 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.666+0000 7f94071c4e40 1 -- 172.21.15.167:0/1717304001 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:85 7.1f 7:f95f44c2:::notify.0:head [watch watch cookie 94302079189120] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e32) v8 -- 0x55c469ee9610 con 0x7f93fc016b60 2023-10-05T19:28:57.671 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.666+0000 7f94071c4e40 1 -- 172.21.15.167:0/1717304001 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:86 7.0 7:05bf5b68:::notify.1:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e32) v8 -- 0x55c469ee9eb0 con 0x55c469eb0310 2023-10-05T19:28:57.674 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.670+0000 7f94054fa700 1 -- 172.21.15.167:0/1717304001 <== osd.1 v2:172.21.15.167:6800/30035 29 ==== osd_op_reply(86 notify.1 [create] v32'216 uv216 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f93f4052040 con 0x55c469eb0310 2023-10-05T19:28:57.674 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.670+0000 7f94071c4e40 1 -- 172.21.15.167:0/1717304001 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:87 7.0 7:05bf5b68:::notify.1:head [watch watch cookie 94302079195584] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e32) v8 -- 0x55c469eea380 con 0x55c469eb0310 2023-10-05T19:28:57.675 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.670+0000 7f94071c4e40 1 -- 172.21.15.167:0/1717304001 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:88 7.15 7:a93a5511:::notify.2:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e32) v8 -- 0x55c469eeadd0 con 0x55c469ee37d0 2023-10-05T19:28:57.675 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.670+0000 7f94044f8700 1 -- 172.21.15.167:0/1717304001 <== osd.2 v2:172.21.15.167:6816/30037 14 ==== osd_op_reply(85 notify.0 [watch watch cookie 94302079189120] v32'217 uv216 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f93ec008a80 con 0x7f93fc016b60 2023-10-05T19:28:57.677 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.674+0000 7f94054fa700 1 -- 172.21.15.167:0/1717304001 <== osd.1 v2:172.21.15.167:6800/30035 30 ==== osd_op_reply(87 notify.1 [watch watch cookie 94302079195584] v32'217 uv216 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f93f4052040 con 0x55c469eb0310 2023-10-05T19:28:57.678 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.674+0000 7f9404cf9700 1 -- 172.21.15.167:0/1717304001 <== osd.3 v2:172.21.15.167:6808/30036 22 ==== osd_op_reply(88 notify.2 [create] v32'216 uv216 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f93fc050040 con 0x55c469ee37d0 2023-10-05T19:28:57.678 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.674+0000 7f94071c4e40 1 -- 172.21.15.167:0/1717304001 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:89 7.15 7:a93a5511:::notify.2:head [watch watch cookie 94302079191328] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e32) v8 -- 0x55c469ee9e10 con 0x55c469ee37d0 2023-10-05T19:28:57.679 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.674+0000 7f94071c4e40 1 -- 172.21.15.167:0/1717304001 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:90 7.e 7:7759931f:::notify.3:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e32) v8 -- 0x55c469eeb6c0 con 0x7f93fc016b60 2023-10-05T19:28:57.681 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.674+0000 7f94044f8700 1 -- 172.21.15.167:0/1717304001 <== osd.2 v2:172.21.15.167:6816/30037 15 ==== osd_op_reply(90 notify.3 [create] v32'216 uv216 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f93ec008a80 con 0x7f93fc016b60 2023-10-05T19:28:57.681 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.674+0000 7f94071c4e40 1 -- 172.21.15.167:0/1717304001 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:91 7.e 7:7759931f:::notify.3:head [watch watch cookie 94302079203952] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e32) v8 -- 0x55c469eec3b0 con 0x7f93fc016b60 2023-10-05T19:28:57.681 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.674+0000 7f94071c4e40 1 -- 172.21.15.167:0/1717304001 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:92 7.d 7:b4812045:::notify.4:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e32) v8 -- 0x55c469eecc50 con 0x55c469eb0310 2023-10-05T19:28:57.681 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.674+0000 7f9404cf9700 1 -- 172.21.15.167:0/1717304001 <== osd.3 v2:172.21.15.167:6808/30036 23 ==== osd_op_reply(89 notify.2 [watch watch cookie 94302079191328] v32'217 uv216 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f93fc050040 con 0x55c469ee37d0 2023-10-05T19:28:57.684 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.678+0000 7f94044f8700 1 -- 172.21.15.167:0/1717304001 <== osd.2 v2:172.21.15.167:6816/30037 16 ==== osd_op_reply(91 notify.3 [watch watch cookie 94302079203952] v32'217 uv216 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f93ec008a80 con 0x7f93fc016b60 2023-10-05T19:28:57.684 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.678+0000 7f94054fa700 1 -- 172.21.15.167:0/1717304001 <== osd.1 v2:172.21.15.167:6800/30035 31 ==== osd_op_reply(92 notify.4 [create] v32'216 uv216 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f93f4052040 con 0x55c469eb0310 2023-10-05T19:28:57.684 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.678+0000 7f94071c4e40 1 -- 172.21.15.167:0/1717304001 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:93 7.d 7:b4812045:::notify.4:head [watch watch cookie 94302079199392] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e32) v8 -- 0x55c469eecc50 con 0x55c469eb0310 2023-10-05T19:28:57.684 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.678+0000 7f94071c4e40 1 -- 172.21.15.167:0/1717304001 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:94 7.3 7:c609908c:::notify.5:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e32) v8 -- 0x55c469eed4f0 con 0x7f93ec007030 2023-10-05T19:28:57.687 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.682+0000 7f94054fa700 1 -- 172.21.15.167:0/1717304001 <== osd.0 v2:172.21.15.118:6800/27690 23 ==== osd_op_reply(94 notify.5 [create] v32'216 uv216 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f93f4052040 con 0x7f93ec007030 2023-10-05T19:28:57.687 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.682+0000 7f94071c4e40 1 -- 172.21.15.167:0/1717304001 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:95 7.3 7:c609908c:::notify.5:head [watch watch cookie 94302079208800] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e32) v8 -- 0x55c469eed770 con 0x7f93ec007030 2023-10-05T19:28:57.687 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.682+0000 7f94071c4e40 1 -- 172.21.15.167:0/1717304001 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:96 7.14 7:2b04a3e9:::notify.6:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e32) v8 -- 0x55c469eee300 con 0x7f93ec007030 2023-10-05T19:28:57.687 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.682+0000 7f94054fa700 1 -- 172.21.15.167:0/1717304001 <== osd.1 v2:172.21.15.167:6800/30035 32 ==== osd_op_reply(93 notify.4 [watch watch cookie 94302079199392] v32'217 uv216 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f93f4052040 con 0x55c469eb0310 2023-10-05T19:28:57.690 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.686+0000 7f94054fa700 1 -- 172.21.15.167:0/1717304001 <== osd.0 v2:172.21.15.118:6800/27690 24 ==== osd_op_reply(95 notify.5 [watch watch cookie 94302079208800] v32'217 uv216 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f93f4052040 con 0x7f93ec007030 2023-10-05T19:28:57.690 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.686+0000 7f94054fa700 1 -- 172.21.15.167:0/1717304001 <== osd.0 v2:172.21.15.118:6800/27690 25 ==== osd_op_reply(96 notify.6 [create] v32'216 uv216 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f93f4052040 con 0x7f93ec007030 2023-10-05T19:28:57.690 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.686+0000 7f94071c4e40 1 -- 172.21.15.167:0/1717304001 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:97 7.14 7:2b04a3e9:::notify.6:head [watch watch cookie 94302079207264] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e32) v8 -- 0x55c469eeddd0 con 0x7f93ec007030 2023-10-05T19:28:57.690 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.686+0000 7f94071c4e40 1 -- 172.21.15.167:0/1717304001 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:98 7.9 7:93e5b521:::notify.7:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e32) v8 -- 0x55c469eeebf0 con 0x7f93ec007030 2023-10-05T19:28:57.692 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.686+0000 7f94054fa700 1 -- 172.21.15.167:0/1717304001 <== osd.0 v2:172.21.15.118:6800/27690 26 ==== osd_op_reply(97 notify.6 [watch watch cookie 94302079207264] v32'217 uv216 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f93f4052040 con 0x7f93ec007030 2023-10-05T19:28:57.693 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.686+0000 7f94054fa700 1 -- 172.21.15.167:0/1717304001 <== osd.0 v2:172.21.15.118:6800/27690 27 ==== osd_op_reply(98 notify.7 [create] v32'216 uv216 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f93f4052040 con 0x7f93ec007030 2023-10-05T19:28:57.693 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.686+0000 7f94071c4e40 1 -- 172.21.15.167:0/1717304001 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:99 7.9 7:93e5b521:::notify.7:head [watch watch cookie 94302079209904] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e32) v8 -- 0x55c469eeddd0 con 0x7f93ec007030 2023-10-05T19:28:57.693 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.686+0000 7f94071c4e40 20 add_watcher() i=0 2023-10-05T19:28:57.693 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.686+0000 7f94071c4e40 20 add_watcher() i=1 2023-10-05T19:28:57.693 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.686+0000 7f94071c4e40 20 add_watcher() i=2 2023-10-05T19:28:57.693 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.686+0000 7f94071c4e40 20 add_watcher() i=3 2023-10-05T19:28:57.693 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.686+0000 7f94071c4e40 20 add_watcher() i=4 2023-10-05T19:28:57.693 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.686+0000 7f94071c4e40 20 add_watcher() i=5 2023-10-05T19:28:57.693 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.686+0000 7f94071c4e40 20 add_watcher() i=6 2023-10-05T19:28:57.694 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.690+0000 7f94054fa700 1 -- 172.21.15.167:0/1717304001 <== osd.0 v2:172.21.15.118:6800/27690 28 ==== osd_op_reply(99 notify.7 [watch watch cookie 94302079209904] v32'217 uv216 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f93f4052040 con 0x7f93ec007030 2023-10-05T19:28:57.695 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.690+0000 7f94071c4e40 20 add_watcher() i=7 2023-10-05T19:28:57.695 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.690+0000 7f94071c4e40 2 all 8 watchers are set, enabling cache 2023-10-05T19:28:57.695 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.690+0000 7f94071c4e40 1 -- 172.21.15.167:0/1717304001 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:100 6.4 6:22d26bf9:::data_loggenerations_metadata:head [call version.check_conds in=74b,call version.read in=11b,read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32) v8 -- 0x55c469ef1930 con 0x55c469eb0310 2023-10-05T19:28:57.695 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.690+0000 7f94054fa700 1 -- 172.21.15.167:0/1717304001 <== osd.1 v2:172.21.15.167:6800/30035 33 ==== osd_op_reply(100 data_loggenerations_metadata [call,call out=48b,read 0~28 out=28b] v0'0 uv1 ondisk = 0) v8 ==== 256+0+76 (crc 0 0 0) 0x7f93f4052040 con 0x55c469eb0310 2023-10-05T19:28:57.695 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.690+0000 7f94071c4e40 1 -- 172.21.15.167:0/1717304001 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:101 6.4 6:22d26bf9:::data_loggenerations_metadata:head [watch watch cookie 94302079214608] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e32) v8 -- 0x55c469eeeea0 con 0x55c469eb0310 2023-10-05T19:28:57.698 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.694+0000 7f94054fa700 1 -- 172.21.15.167:0/1717304001 <== osd.1 v2:172.21.15.167:6800/30035 34 ==== osd_op_reply(101 data_loggenerations_metadata [watch watch cookie 94302079214608] v32'214 uv1 ondisk = 0) v8 ==== 172+0+0 (crc 0 0 0) 0x7f93f4052040 con 0x55c469eb0310 2023-10-05T19:28:57.698 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.694+0000 7f93d67e4700 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2023-10-05T19:28:57.698 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.694+0000 7f94071c4e40 20 check_secure_mon_conn(): auth registy supported: methods=[2] modes=[2,1] 2023-10-05T19:28:57.698 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.694+0000 7f94071c4e40 20 check_secure_mon_conn(): mode 1 is insecure 2023-10-05T19:28:57.699 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.694+0000 7f94071c4e40 1 Processor -- start 2023-10-05T19:28:57.699 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.694+0000 7f94071c4e40 1 -- start start 2023-10-05T19:28:57.699 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.694+0000 7f94071c4e40 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55c469efc100 0x55c469efc4d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-05T19:28:57.699 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.694+0000 7f94071c4e40 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c469efca10 con 0x55c469efc100 2023-10-05T19:28:57.700 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.694+0000 7f94044f8700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55c469efc100 0x55c469efc4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-05T19:28:57.700 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.694+0000 7f94044f8700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55c469efc100 0x55c469efc4d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.118:3300/0 says I am v2:172.21.15.167:53564/0 (socket says 172.21.15.167:53564) 2023-10-05T19:28:57.700 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.694+0000 7f94044f8700 1 -- 172.21.15.167:0/1488608634 learned_addr learned my addr 172.21.15.167:0/1488608634 (peer_addr_for_me v2:172.21.15.167:0/0) 2023-10-05T19:28:57.700 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.694+0000 7f94044f8700 1 -- 172.21.15.167:0/1488608634 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55c469efcb50 con 0x55c469efc100 2023-10-05T19:28:57.701 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.694+0000 7f94044f8700 1 --2- 172.21.15.167:0/1488608634 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55c469efc100 0x55c469efc4d0 secure :-1 s=READY pgs=570 cs=0 l=1 rev1=1 crypto rx=0x55c469e1f3d0 tx=0x7f93ec012000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=46d9763a425d2a09 server_cookie=0 in_seq=0 out_seq=0 2023-10-05T19:28:57.701 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.694+0000 7f93d5fe3700 1 -- 172.21.15.167:0/1488608634 <== mon.0 v2:172.21.15.118:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f93ec0127d0 con 0x55c469efc100 2023-10-05T19:28:57.701 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.694+0000 7f93d5fe3700 1 -- 172.21.15.167:0/1488608634 <== mon.0 v2:172.21.15.118:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f93ec01cdf0 con 0x55c469efc100 2023-10-05T19:28:57.701 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.694+0000 7f93d5fe3700 1 -- 172.21.15.167:0/1488608634 <== mon.0 v2:172.21.15.118:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f93ec02a8a0 con 0x55c469efc100 2023-10-05T19:28:57.703 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.698+0000 7f94071c4e40 1 -- 172.21.15.167:0/1488608634 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55c469efc100 msgr2=0x55c469efc4d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-05T19:28:57.703 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.698+0000 7f94071c4e40 1 --2- 172.21.15.167:0/1488608634 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55c469efc100 0x55c469efc4d0 secure :-1 s=READY pgs=570 cs=0 l=1 rev1=1 crypto rx=0x55c469e1f3d0 tx=0x7f93ec012000 comp rx=0 tx=0).stop 2023-10-05T19:28:57.703 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.698+0000 7f94071c4e40 1 -- 172.21.15.167:0/1488608634 shutdown_connections 2023-10-05T19:28:57.703 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.698+0000 7f94071c4e40 1 --2- 172.21.15.167:0/1488608634 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55c469efc100 0x55c469efc4d0 unknown :-1 s=CLOSED pgs=570 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-05T19:28:57.703 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.698+0000 7f94071c4e40 1 -- 172.21.15.167:0/1488608634 >> 172.21.15.167:0/1488608634 conn(0x55c469ef74c0 msgr2=0x55c469ef5610 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-05T19:28:57.703 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.698+0000 7f94071c4e40 1 -- 172.21.15.167:0/1488608634 shutdown_connections 2023-10-05T19:28:57.703 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.698+0000 7f94071c4e40 1 -- 172.21.15.167:0/1488608634 wait complete. 2023-10-05T19:28:57.704 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.698+0000 7f94071c4e40 1 Processor -- start 2023-10-05T19:28:57.704 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.698+0000 7f94071c4e40 1 -- start start 2023-10-05T19:28:57.704 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.698+0000 7f94071c4e40 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55c469efc100 0x55c469f01c30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-05T19:28:57.704 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.698+0000 7f94071c4e40 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c469f02c30 con 0x55c469efc100 2023-10-05T19:28:57.705 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.698+0000 7f94044f8700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55c469efc100 0x55c469f01c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-05T19:28:57.705 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.698+0000 7f94044f8700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55c469efc100 0x55c469f01c30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.118:3300/0 says I am v2:172.21.15.167:53568/0 (socket says 172.21.15.167:53568) 2023-10-05T19:28:57.705 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.698+0000 7f94044f8700 1 -- 172.21.15.167:0/438558575 learned_addr learned my addr 172.21.15.167:0/438558575 (peer_addr_for_me v2:172.21.15.167:0/0) 2023-10-05T19:28:57.705 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.698+0000 7f94044f8700 1 -- 172.21.15.167:0/438558575 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55c469eb3c40 con 0x55c469efc100 2023-10-05T19:28:57.705 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.698+0000 7f94044f8700 1 --2- 172.21.15.167:0/438558575 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55c469efc100 0x55c469f01c30 secure :-1 s=READY pgs=571 cs=0 l=1 rev1=1 crypto rx=0x55c469ef6180 tx=0x7f93ec02f000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-05T19:28:57.706 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.702+0000 7f9347fff700 1 -- 172.21.15.167:0/438558575 <== mon.0 v2:172.21.15.118:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f93ec03eb20 con 0x55c469efc100 2023-10-05T19:28:57.706 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.702+0000 7f9347fff700 1 -- 172.21.15.167:0/438558575 <== mon.0 v2:172.21.15.118:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f93ec03eca0 con 0x55c469efc100 2023-10-05T19:28:57.706 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.702+0000 7f94071c4e40 1 -- 172.21.15.167:0/438558575 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x55c469f02230 con 0x55c469efc100 2023-10-05T19:28:57.706 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.702+0000 7f9347fff700 1 -- 172.21.15.167:0/438558575 <== mon.0 v2:172.21.15.118:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f93ec03ee50 con 0x55c469efc100 2023-10-05T19:28:57.706 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.702+0000 7f94071c4e40 1 -- 172.21.15.167:0/438558575 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55c469f02610 con 0x55c469efc100 2023-10-05T19:28:57.707 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.702+0000 7f9347fff700 1 -- 172.21.15.167:0/438558575 <== mon.0 v2:172.21.15.118:3300/0 4 ==== mgrmap(e 5) v1 ==== 40329+0+0 (secure 0 0 0) 0x7f93ec035400 con 0x55c469efc100 2023-10-05T19:28:57.707 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.702+0000 7f9347fff700 1 --2- 172.21.15.167:0/438558575 >> [v2:172.21.15.167:6824/29889,v1:172.21.15.167:6825/29889] conn(0x7f934c030ca0 0x7f934c033150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-05T19:28:57.708 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.702+0000 7f9347fff700 1 -- 172.21.15.167:0/438558575 <== mon.0 v2:172.21.15.118:3300/0 5 ==== osd_map(32..32 src has 1..32) v4 ==== 6260+0+0 (secure 0 0 0) 0x7f93ec02f070 con 0x55c469efc100 2023-10-05T19:28:57.708 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.702+0000 7f94054fa700 1 --2- 172.21.15.167:0/438558575 >> [v2:172.21.15.167:6824/29889,v1:172.21.15.167:6825/29889] conn(0x7f934c030ca0 0x7f934c033150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-05T19:28:57.708 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.702+0000 7f94071c4e40 1 --2- 172.21.15.167:0/438558575 >> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] conn(0x55c469f0c880 0x55c469f0ec90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-05T19:28:57.708 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.702+0000 7f94071c4e40 1 -- 172.21.15.167:0/438558575 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:1 6.19 6:9a6dd8c0:gc::gc.0:head [create,call version.check_conds in=74b,call rgw_gc.rgw_gc_queue_init in=45b,call version.set in=34b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e32) v8 -- 0x55c469f0f2e0 con 0x55c469f0c880 2023-10-05T19:28:57.708 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.702+0000 7f9404cf9700 1 --2- 172.21.15.167:0/438558575 >> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] conn(0x55c469f0c880 0x55c469f0ec90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-05T19:28:57.709 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.702+0000 7f94054fa700 1 --2- 172.21.15.167:0/438558575 >> [v2:172.21.15.167:6824/29889,v1:172.21.15.167:6825/29889] conn(0x7f934c030ca0 0x7f934c033150 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x55c469eeee00 tx=0x7f93f400a280 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-05T19:28:57.709 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.702+0000 7f9404cf9700 1 --2- 172.21.15.167:0/438558575 >> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] conn(0x55c469f0c880 0x55c469f0ec90 crc :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-05T19:28:57.712 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.706+0000 7f9404cf9700 1 -- 172.21.15.167:0/438558575 <== osd.1 v2:172.21.15.167:6800/30035 1 ==== osd_op_reply(1 gc.0 [create,call,call,call] v32'37 uv33 ondisk = -125 ((125) Operation canceled)) v8 ==== 274+0+0 (crc 0 0 0) 0x7f93fc050040 con 0x55c469f0c880 2023-10-05T19:28:57.712 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.706+0000 7f94071c4e40 1 --2- 172.21.15.167:0/438558575 >> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] conn(0x55c469f08460 0x55c469f12950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-05T19:28:57.712 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.706+0000 7f94071c4e40 1 -- 172.21.15.167:0/438558575 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:2 6.18 6:1e85421f:gc::gc.1:head [create,call version.check_conds in=74b,call rgw_gc.rgw_gc_queue_init in=45b,call version.set in=34b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e32) v8 -- 0x55c469f12e90 con 0x55c469f08460 2023-10-05T19:28:57.712 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.706+0000 7f94044f8700 1 --2- 172.21.15.167:0/438558575 >> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] conn(0x55c469f08460 0x55c469f12950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-05T19:28:57.713 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.706+0000 7f94044f8700 1 --2- 172.21.15.167:0/438558575 >> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] conn(0x55c469f08460 0x55c469f12950 crc :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-05T19:28:57.716 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.710+0000 7f94044f8700 1 -- 172.21.15.167:0/438558575 <== osd.0 v2:172.21.15.118:6800/27690 1 ==== osd_op_reply(2 gc.1 [create,call,call,call] v32'29 uv25 ondisk = -125 ((125) Operation canceled)) v8 ==== 274+0+0 (crc 0 0 0) 0x7f93ec032040 con 0x55c469f08460 2023-10-05T19:28:57.716 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.710+0000 7f94071c4e40 1 --2- 172.21.15.167:0/438558575 >> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] conn(0x55c469f13ee0 0x55c469f162c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-05T19:28:57.716 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.710+0000 7f94071c4e40 1 -- 172.21.15.167:0/438558575 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:3 6.b 6:d242335b:gc::gc.2:head [create,call version.check_conds in=74b,call rgw_gc.rgw_gc_queue_init in=45b,call version.set in=34b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e32) v8 -- 0x55c469f169c0 con 0x55c469f13ee0 2023-10-05T19:28:57.716 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.710+0000 7f94054fa700 1 --2- 172.21.15.167:0/438558575 >> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] conn(0x55c469f13ee0 0x55c469f162c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-05T19:28:57.717 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.710+0000 7f94054fa700 1 --2- 172.21.15.167:0/438558575 >> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] conn(0x55c469f13ee0 0x55c469f162c0 crc :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-05T19:28:57.719 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.714+0000 7f94054fa700 1 -- 172.21.15.167:0/438558575 <== osd.3 v2:172.21.15.167:6808/30036 1 ==== osd_op_reply(3 gc.2 [create,call,call,call] v32'25 uv21 ondisk = -125 ((125) Operation canceled)) v8 ==== 274+0+0 (crc 0 0 0) 0x7f93f4052040 con 0x55c469f13ee0 2023-10-05T19:28:57.719 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.714+0000 7f94071c4e40 1 -- 172.21.15.167:0/438558575 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:4 6.4 6:2166d8e1:gc::gc.3:head [create,call version.check_conds in=74b,call rgw_gc.rgw_gc_queue_init in=45b,call version.set in=34b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e32) v8 -- 0x55c469f17010 con 0x55c469f0c880 2023-10-05T19:28:57.722 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.718+0000 7f9404cf9700 1 -- 172.21.15.167:0/438558575 <== osd.1 v2:172.21.15.167:6800/30035 2 ==== osd_op_reply(4 gc.3 [create,call,call,call] v32'215 uv184 ondisk = -125 ((125) Operation canceled)) v8 ==== 274+0+0 (crc 0 0 0) 0x7f93fc050040 con 0x55c469f0c880 2023-10-05T19:28:57.722 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.718+0000 7f94071c4e40 1 --2- 172.21.15.167:0/438558575 >> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] conn(0x55c469f18030 0x55c469f1a410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-05T19:28:57.722 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.718+0000 7f94071c4e40 1 -- 172.21.15.167:0/438558575 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:5 6.3 6:c60da066:gc::gc.4:head [create,call version.check_conds in=74b,call rgw_gc.rgw_gc_queue_init in=45b,call version.set in=34b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e32) v8 -- 0x55c469f1aac0 con 0x55c469f18030 2023-10-05T19:28:57.722 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.718+0000 7f9404cf9700 1 --2- 172.21.15.167:0/438558575 >> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] conn(0x55c469f18030 0x55c469f1a410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-05T19:28:57.723 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.718+0000 7f9404cf9700 1 --2- 172.21.15.167:0/438558575 >> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] conn(0x55c469f18030 0x55c469f1a410 crc :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-05T19:28:57.725 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.722+0000 7f9404cf9700 1 -- 172.21.15.167:0/438558575 <== osd.2 v2:172.21.15.167:6816/30037 1 ==== osd_op_reply(5 gc.4 [create,call,call,call] v32'65 uv51 ondisk = -125 ((125) Operation canceled)) v8 ==== 274+0+0 (crc 0 0 0) 0x7f93fc050040 con 0x55c469f18030 2023-10-05T19:28:57.725 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.722+0000 7f94071c4e40 1 -- 172.21.15.167:0/438558575 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:6 6.9 6:954be8d2:gc::gc.5:head [create,call version.check_conds in=74b,call rgw_gc.rgw_gc_queue_init in=45b,call version.set in=34b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e32) v8 -- 0x55c469f1b110 con 0x55c469f13ee0 2023-10-05T19:28:57.728 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.722+0000 7f94054fa700 1 -- 172.21.15.167:0/438558575 <== osd.3 v2:172.21.15.167:6808/30036 2 ==== osd_op_reply(6 gc.5 [create,call,call,call] v32'46 uv37 ondisk = -125 ((125) Operation canceled)) v8 ==== 274+0+0 (crc 0 0 0) 0x7f93f4052040 con 0x55c469f13ee0 2023-10-05T19:28:57.728 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.722+0000 7f94071c4e40 1 -- 172.21.15.167:0/438558575 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:7 6.e 6:748722ff:gc::gc.6:head [create,call version.check_conds in=74b,call rgw_gc.rgw_gc_queue_init in=45b,call version.set in=34b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e32) v8 -- 0x55c469f1b730 con 0x55c469f0c880 2023-10-05T19:28:57.731 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.726+0000 7f9404cf9700 1 -- 172.21.15.167:0/438558575 <== osd.1 v2:172.21.15.167:6800/30035 3 ==== osd_op_reply(7 gc.6 [create,call,call,call] v32'27 uv23 ondisk = -125 ((125) Operation canceled)) v8 ==== 274+0+0 (crc 0 0 0) 0x7f93fc050040 con 0x55c469f0c880 2023-10-05T19:28:57.731 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.726+0000 7f94071c4e40 1 -- 172.21.15.167:0/438558575 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:8 6.1d 6:beaa4be4:gc::gc.7:head [create,call version.check_conds in=74b,call rgw_gc.rgw_gc_queue_init in=45b,call version.set in=34b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e32) v8 -- 0x55c469f1b730 con 0x55c469f08460 2023-10-05T19:28:57.733 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.730+0000 7f94044f8700 1 -- 172.21.15.167:0/438558575 <== osd.0 v2:172.21.15.118:6800/27690 2 ==== osd_op_reply(8 gc.7 [create,call,call,call] v32'43 uv34 ondisk = -125 ((125) Operation canceled)) v8 ==== 274+0+0 (crc 0 0 0) 0x7f93ec032040 con 0x55c469f08460 2023-10-05T19:28:57.733 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.730+0000 7f94071c4e40 1 -- 172.21.15.167:0/438558575 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:9 6.12 6:4f78e9a2:gc::gc.8:head [create,call version.check_conds in=74b,call rgw_gc.rgw_gc_queue_init in=45b,call version.set in=34b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e32) v8 -- 0x55c469f1bd50 con 0x55c469f18030 2023-10-05T19:28:57.736 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.730+0000 7f9404cf9700 1 -- 172.21.15.167:0/438558575 <== osd.2 v2:172.21.15.167:6816/30037 2 ==== osd_op_reply(9 gc.8 [create,call,call,call] v32'25 uv21 ondisk = -125 ((125) Operation canceled)) v8 ==== 274+0+0 (crc 0 0 0) 0x7f93fc050040 con 0x55c469f18030 2023-10-05T19:28:57.736 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.730+0000 7f94071c4e40 1 -- 172.21.15.167:0/438558575 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:10 6.1c 6:3e426232:gc::gc.9:head [create,call version.check_conds in=74b,call rgw_gc.rgw_gc_queue_init in=45b,call version.set in=34b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e32) v8 -- 0x55c469f1bd50 con 0x55c469f13ee0 2023-10-05T19:28:57.739 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.734+0000 7f94054fa700 1 -- 172.21.15.167:0/438558575 <== osd.3 v2:172.21.15.167:6808/30036 3 ==== osd_op_reply(10 gc.9 [create,call,call,call] v32'46 uv39 ondisk = -125 ((125) Operation canceled)) v8 ==== 274+0+0 (crc 0 0 0) 0x7f93f4052040 con 0x55c469f13ee0 2023-10-05T19:28:57.739 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.734+0000 7f94071c4e40 1 -- 172.21.15.167:0/438558575 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:11 6.3 6:c31f41d7:gc::gc.10:head [create,call version.check_conds in=74b,call rgw_gc.rgw_gc_queue_init in=45b,call version.set in=34b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e32) v8 -- 0x55c469f1c240 con 0x55c469f18030 2023-10-05T19:28:57.742 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.738+0000 7f9404cf9700 1 -- 172.21.15.167:0/438558575 <== osd.2 v2:172.21.15.167:6816/30037 3 ==== osd_op_reply(11 gc.10 [create,call,call,call] v32'66 uv53 ondisk = -125 ((125) Operation canceled)) v8 ==== 275+0+0 (crc 0 0 0) 0x7f93fc050040 con 0x55c469f18030 2023-10-05T19:28:57.742 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.738+0000 7f94071c4e40 1 -- 172.21.15.167:0/438558575 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:12 6.13 6:c8333aab:gc::gc.11:head [create,call version.check_conds in=74b,call rgw_gc.rgw_gc_queue_init in=45b,call version.set in=34b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e32) v8 -- 0x55c469f1c240 con 0x55c469f08460 2023-10-05T19:28:57.745 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.738+0000 7f94044f8700 1 -- 172.21.15.167:0/438558575 <== osd.0 v2:172.21.15.118:6800/27690 3 ==== osd_op_reply(12 gc.11 [create,call,call,call] v32'29 uv25 ondisk = -125 ((125) Operation canceled)) v8 ==== 275+0+0 (crc 0 0 0) 0x7f93ec032040 con 0x55c469f08460 2023-10-05T19:28:57.745 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.738+0000 7f94071c4e40 1 -- 172.21.15.167:0/438558575 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:13 6.15 6:ab569f81:gc::gc.12:head [create,call version.check_conds in=74b,call rgw_gc.rgw_gc_queue_init in=45b,call version.set in=34b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e32) v8 -- 0x55c469f1c860 con 0x55c469f18030 2023-10-05T19:28:57.748 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.742+0000 7f9404cf9700 1 -- 172.21.15.167:0/438558575 <== osd.2 v2:172.21.15.167:6816/30037 4 ==== osd_op_reply(13 gc.12 [create,call,call,call] v32'25 uv21 ondisk = -125 ((125) Operation canceled)) v8 ==== 275+0+0 (crc 0 0 0) 0x7f93fc050040 con 0x55c469f18030 2023-10-05T19:28:57.748 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.742+0000 7f94071c4e40 1 -- 172.21.15.167:0/438558575 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:14 6.0 6:0667bb13:gc::gc.13:head [create,call version.check_conds in=74b,call rgw_gc.rgw_gc_queue_init in=45b,call version.set in=34b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e32) v8 -- 0x55c469f1c860 con 0x55c469f08460 2023-10-05T19:28:57.751 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.746+0000 7f94044f8700 1 -- 172.21.15.167:0/438558575 <== osd.0 v2:172.21.15.118:6800/27690 4 ==== osd_op_reply(14 gc.13 [create,call,call,call] v32'49 uv40 ondisk = -125 ((125) Operation canceled)) v8 ==== 275+0+0 (crc 0 0 0) 0x7f93ec032040 con 0x55c469f08460 2023-10-05T19:28:57.751 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.746+0000 7f94071c4e40 1 -- 172.21.15.167:0/438558575 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:15 6.4 6:227f63b5:gc::gc.14:head [create,call version.check_conds in=74b,call rgw_gc.rgw_gc_queue_init in=45b,call version.set in=34b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e32) v8 -- 0x55c469f1cd50 con 0x55c469f0c880 2023-10-05T19:28:57.754 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.750+0000 7f9404cf9700 1 -- 172.21.15.167:0/438558575 <== osd.1 v2:172.21.15.167:6800/30035 4 ==== osd_op_reply(15 gc.14 [create,call,call,call] v32'216 uv186 ondisk = -125 ((125) Operation canceled)) v8 ==== 275+0+0 (crc 0 0 0) 0x7f93fc050040 con 0x55c469f0c880 2023-10-05T19:28:57.754 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.750+0000 7f94071c4e40 1 -- 172.21.15.167:0/438558575 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:16 6.7 6:e07a6548:gc::gc.15:head [create,call version.check_conds in=74b,call rgw_gc.rgw_gc_queue_init in=45b,call version.set in=34b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e32) v8 -- 0x55c469f1cd50 con 0x55c469f13ee0 2023-10-05T19:28:57.757 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.750+0000 7f94054fa700 1 -- 172.21.15.167:0/438558575 <== osd.3 v2:172.21.15.167:6808/30036 4 ==== osd_op_reply(16 gc.15 [create,call,call,call] v32'60 uv45 ondisk = -125 ((125) Operation canceled)) v8 ==== 275+0+0 (crc 0 0 0) 0x7f93f4052040 con 0x55c469f13ee0 2023-10-05T19:28:57.757 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.750+0000 7f94071c4e40 1 -- 172.21.15.167:0/438558575 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:17 6.3 6:c623b173:gc::gc.16:head [create,call version.check_conds in=74b,call rgw_gc.rgw_gc_queue_init in=45b,call version.set in=34b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e32) v8 -- 0x55c469f1cd50 con 0x55c469f18030 2023-10-05T19:28:57.760 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.754+0000 7f9404cf9700 1 -- 172.21.15.167:0/438558575 <== osd.2 v2:172.21.15.167:6816/30037 5 ==== osd_op_reply(17 gc.16 [create,call,call,call] v32'67 uv55 ondisk = -125 ((125) Operation canceled)) v8 ==== 275+0+0 (crc 0 0 0) 0x7f93fc050040 con 0x55c469f18030 2023-10-05T19:28:57.760 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.754+0000 7f94071c4e40 1 -- 172.21.15.167:0/438558575 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:18 6.16 6:6849fd06:gc::gc.17:head [create,call version.check_conds in=74b,call rgw_gc.rgw_gc_queue_init in=45b,call version.set in=34b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e32) v8 -- 0x55c469f1c860 con 0x55c469f13ee0 2023-10-05T19:28:57.763 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.758+0000 7f94054fa700 1 -- 172.21.15.167:0/438558575 <== osd.3 v2:172.21.15.167:6808/30036 5 ==== osd_op_reply(18 gc.17 [create,call,call,call] v32'25 uv21 ondisk = -125 ((125) Operation canceled)) v8 ==== 275+0+0 (crc 0 0 0) 0x7f93f4052040 con 0x55c469f13ee0 2023-10-05T19:28:57.763 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.758+0000 7f94071c4e40 1 -- 172.21.15.167:0/438558575 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:19 6.d 6:b019ea83:gc::gc.18:head [create,call version.check_conds in=74b,call rgw_gc.rgw_gc_queue_init in=45b,call version.set in=34b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e32) v8 -- 0x55c469f1c860 con 0x55c469f0c880 2023-10-05T19:28:57.766 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.762+0000 7f9404cf9700 1 -- 172.21.15.167:0/438558575 <== osd.1 v2:172.21.15.167:6800/30035 5 ==== osd_op_reply(19 gc.18 [create,call,call,call] v32'33 uv29 ondisk = -125 ((125) Operation canceled)) v8 ==== 275+0+0 (crc 0 0 0) 0x7f93fc050040 con 0x55c469f0c880 2023-10-05T19:28:57.766 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.762+0000 7f94071c4e40 1 -- 172.21.15.167:0/438558575 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:20 6.4 6:24c8e1ae:gc::gc.19:head [create,call version.check_conds in=74b,call rgw_gc.rgw_gc_queue_init in=45b,call version.set in=34b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e32) v8 -- 0x55c469f1c860 con 0x55c469f0c880 2023-10-05T19:28:57.768 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.762+0000 7f9404cf9700 1 -- 172.21.15.167:0/438558575 <== osd.1 v2:172.21.15.167:6800/30035 6 ==== osd_op_reply(20 gc.19 [create,call,call,call] v32'217 uv189 ondisk = -125 ((125) Operation canceled)) v8 ==== 275+0+0 (crc 0 0 0) 0x7f93fc050040 con 0x55c469f0c880 2023-10-05T19:28:57.769 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.762+0000 7f94071c4e40 1 -- 172.21.15.167:0/438558575 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:21 6.7 6:e622f76a:gc::gc.20:head [create,call version.check_conds in=74b,call rgw_gc.rgw_gc_queue_init in=45b,call version.set in=34b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e32) v8 -- 0x55c469f1c860 con 0x55c469f13ee0 2023-10-05T19:28:57.771 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.766+0000 7f94054fa700 1 -- 172.21.15.167:0/438558575 <== osd.3 v2:172.21.15.167:6808/30036 6 ==== osd_op_reply(21 gc.20 [create,call,call,call] v32'61 uv47 ondisk = -125 ((125) Operation canceled)) v8 ==== 275+0+0 (crc 0 0 0) 0x7f93f4052040 con 0x55c469f13ee0 2023-10-05T19:28:57.771 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.766+0000 7f94071c4e40 1 -- 172.21.15.167:0/438558575 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:22 6.a 6:5127e642:gc::gc.21:head [create,call version.check_conds in=74b,call rgw_gc.rgw_gc_queue_init in=45b,call version.set in=34b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e32) v8 -- 0x55c469f1c860 con 0x55c469f18030 2023-10-05T19:28:57.774 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.770+0000 7f9404cf9700 1 -- 172.21.15.167:0/438558575 <== osd.2 v2:172.21.15.167:6816/30037 6 ==== osd_op_reply(22 gc.21 [create,call,call,call] v32'35 uv31 ondisk = -125 ((125) Operation canceled)) v8 ==== 275+0+0 (crc 0 0 0) 0x7f93fc050040 con 0x55c469f18030 2023-10-05T19:28:57.774 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.770+0000 7f94071c4e40 1 -- 172.21.15.167:0/438558575 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:23 6.11 6:8eb99c53:gc::gc.22:head [create,call version.check_conds in=74b,call rgw_gc.rgw_gc_queue_init in=45b,call version.set in=34b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e32) v8 -- 0x55c469f1c860 con 0x55c469f13ee0 2023-10-05T19:28:57.776 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.770+0000 7f94054fa700 1 -- 172.21.15.167:0/438558575 <== osd.3 v2:172.21.15.167:6808/30036 7 ==== osd_op_reply(23 gc.22 [create,call,call,call] v32'27 uv23 ondisk = -125 ((125) Operation canceled)) v8 ==== 275+0+0 (crc 0 0 0) 0x7f93f4052040 con 0x55c469f13ee0 2023-10-05T19:28:57.776 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.770+0000 7f94071c4e40 1 -- 172.21.15.167:0/438558575 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:24 6.8 6:17d7e8ed:gc::gc.23:head [create,call version.check_conds in=74b,call rgw_gc.rgw_gc_queue_init in=45b,call version.set in=34b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e32) v8 -- 0x55c469f1c860 con 0x55c469f13ee0 2023-10-05T19:28:57.778 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.774+0000 7f94054fa700 1 -- 172.21.15.167:0/438558575 <== osd.3 v2:172.21.15.167:6808/30036 8 ==== osd_op_reply(24 gc.23 [create,call,call,call] v32'38 uv33 ondisk = -125 ((125) Operation canceled)) v8 ==== 275+0+0 (crc 0 0 0) 0x7f93f4052040 con 0x55c469f13ee0 2023-10-05T19:28:57.778 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.774+0000 7f94071c4e40 1 -- 172.21.15.167:0/438558575 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:25 6.1b 6:db9d130d:gc::gc.24:head [create,call version.check_conds in=74b,call rgw_gc.rgw_gc_queue_init in=45b,call version.set in=34b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e32) v8 -- 0x55c469f1c860 con 0x55c469f13ee0 2023-10-05T19:28:57.780 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.774+0000 7f94054fa700 1 -- 172.21.15.167:0/438558575 <== osd.3 v2:172.21.15.167:6808/30036 9 ==== osd_op_reply(25 gc.24 [create,call,call,call] v32'21 uv17 ondisk = -125 ((125) Operation canceled)) v8 ==== 275+0+0 (crc 0 0 0) 0x7f93f4052040 con 0x55c469f13ee0 2023-10-05T19:28:57.780 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.774+0000 7f94071c4e40 1 -- 172.21.15.167:0/438558575 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:26 6.9 6:921ecdb1:gc::gc.25:head [create,call version.check_conds in=74b,call rgw_gc.rgw_gc_queue_init in=45b,call version.set in=34b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e32) v8 -- 0x55c469f1c860 con 0x55c469f13ee0 2023-10-05T19:28:57.783 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.778+0000 7f94054fa700 1 -- 172.21.15.167:0/438558575 <== osd.3 v2:172.21.15.167:6808/30036 10 ==== osd_op_reply(26 gc.25 [create,call,call,call] v32'47 uv39 ondisk = -125 ((125) Operation canceled)) v8 ==== 275+0+0 (crc 0 0 0) 0x7f93f4052040 con 0x55c469f13ee0 2023-10-05T19:28:57.783 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.778+0000 7f94071c4e40 1 -- 172.21.15.167:0/438558575 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:27 6.1f 6:fde4dd55:gc::gc.26:head [create,call version.check_conds in=74b,call rgw_gc.rgw_gc_queue_init in=45b,call version.set in=34b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e32) v8 -- 0x55c469f1c860 con 0x55c469f08460 2023-10-05T19:28:57.785 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.782+0000 7f94044f8700 1 -- 172.21.15.167:0/438558575 <== osd.0 v2:172.21.15.118:6800/27690 5 ==== osd_op_reply(27 gc.26 [create,call,call,call] v32'38 uv23 ondisk = -125 ((125) Operation canceled)) v8 ==== 275+0+0 (crc 0 0 0) 0x7f93ec032040 con 0x55c469f08460 2023-10-05T19:28:57.786 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.782+0000 7f94071c4e40 1 -- 172.21.15.167:0/438558575 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:28 6.1a 6:5909e671:gc::gc.27:head [create,call version.check_conds in=74b,call rgw_gc.rgw_gc_queue_init in=45b,call version.set in=34b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e32) v8 -- 0x55c469f1c8f0 con 0x55c469f13ee0 2023-10-05T19:28:57.788 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.782+0000 7f94054fa700 1 -- 172.21.15.167:0/438558575 <== osd.3 v2:172.21.15.167:6808/30036 11 ==== osd_op_reply(28 gc.27 [create,call,call,call] v32'25 uv21 ondisk = -125 ((125) Operation canceled)) v8 ==== 275+0+0 (crc 0 0 0) 0x7f93f4052040 con 0x55c469f13ee0 2023-10-05T19:28:57.788 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.782+0000 7f94071c4e40 1 -- 172.21.15.167:0/438558575 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:29 6.1d 6:b88e3271:gc::gc.28:head [create,call version.check_conds in=74b,call rgw_gc.rgw_gc_queue_init in=45b,call version.set in=34b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e32) v8 -- 0x55c469f1c8f0 con 0x55c469f08460 2023-10-05T19:28:57.790 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.786+0000 7f94044f8700 1 -- 172.21.15.167:0/438558575 <== osd.0 v2:172.21.15.118:6800/27690 6 ==== osd_op_reply(29 gc.28 [create,call,call,call] v32'44 uv36 ondisk = -125 ((125) Operation canceled)) v8 ==== 275+0+0 (crc 0 0 0) 0x7f93ec032040 con 0x55c469f08460 2023-10-05T19:28:57.791 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.786+0000 7f94071c4e40 1 -- 172.21.15.167:0/438558575 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:30 6.7 6:e47c9322:gc::gc.29:head [create,call version.check_conds in=74b,call rgw_gc.rgw_gc_queue_init in=45b,call version.set in=34b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e32) v8 -- 0x55c469f1c860 con 0x55c469f13ee0 2023-10-05T19:28:57.793 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.786+0000 7f94054fa700 1 -- 172.21.15.167:0/438558575 <== osd.3 v2:172.21.15.167:6808/30036 12 ==== osd_op_reply(30 gc.29 [create,call,call,call] v32'62 uv49 ondisk = -125 ((125) Operation canceled)) v8 ==== 275+0+0 (crc 0 0 0) 0x7f93f4052040 con 0x55c469f13ee0 2023-10-05T19:28:57.793 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.786+0000 7f94071c4e40 1 -- 172.21.15.167:0/438558575 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:31 6.0 6:013d6c33:gc::gc.30:head [create,call version.check_conds in=74b,call rgw_gc.rgw_gc_queue_init in=45b,call version.set in=34b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e32) v8 -- 0x55c469f1c860 con 0x55c469f08460 2023-10-05T19:28:57.796 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.790+0000 7f94044f8700 1 -- 172.21.15.167:0/438558575 <== osd.0 v2:172.21.15.118:6800/27690 7 ==== osd_op_reply(31 gc.30 [create,call,call,call] v32'50 uv42 ondisk = -125 ((125) Operation canceled)) v8 ==== 275+0+0 (crc 0 0 0) 0x7f93ec032040 con 0x55c469f08460 2023-10-05T19:28:57.796 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.790+0000 7f94071c4e40 1 -- 172.21.15.167:0/438558575 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:32 6.f 6:f24cb4a8:gc::gc.31:head [create,call version.check_conds in=74b,call rgw_gc.rgw_gc_queue_init in=45b,call version.set in=34b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e32) v8 -- 0x55c469f1c8f0 con 0x55c469f18030 2023-10-05T19:28:57.799 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.794+0000 7f9404cf9700 1 -- 172.21.15.167:0/438558575 <== osd.2 v2:172.21.15.167:6816/30037 7 ==== osd_op_reply(32 gc.31 [create,call,call,call] v32'31 uv27 ondisk = -125 ((125) Operation canceled)) v8 ==== 275+0+0 (crc 0 0 0) 0x7f93fc050040 con 0x55c469f18030 2023-10-05T19:28:57.801 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.798+0000 7f93457fa700 20 reqs_thread_entry: start 2023-10-05T19:28:57.802 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.798+0000 7f933effd700 20 reqs_thread_entry: start 2023-10-05T19:28:57.802 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.798+0000 7f94071c4e40 20 init_complete bucket index max shards: 11 2023-10-05T19:28:57.803 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.798+0000 7f94071c4e40 10 rgw notify: Started notification manager with: 1 workers 2023-10-05T19:28:57.803 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.798+0000 7f933d7fa700 20 rgw notify: INFO: next queues processing will happen at: Thu Oct 5 19:29:28 2023 2023-10-05T19:28:57.803 INFO:teuthology.orchestra.run.smithi167.stderr: 2023-10-05T19:28:57.803 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.798+0000 7f933cff9700 20 reqs_thread_entry: start 2023-10-05T19:28:57.803 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.798+0000 7f94071c4e40 20 get_system_obj_state: rctx=0x7ffd8e315808 obj=default.rgw.meta:users.uid:foo state=0x55c469d850b0 s->prefetch_data=0 2023-10-05T19:28:57.803 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.798+0000 7f94071c4e40 10 cache get: name=default.rgw.meta+users.uid+foo : miss 2023-10-05T19:28:57.803 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.798+0000 7f94071c4e40 1 -- 172.21.15.167:0/1717304001 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:102 8.18 8:1e6ca5ed:users.uid::foo:head [call version.read in=11b,getxattrs,stat] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32) v8 -- 0x55c469f7b3b0 con 0x55c469eb0310 2023-10-05T19:28:57.804 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.798+0000 7f94054fa700 1 -- 172.21.15.167:0/1717304001 <== osd.1 v2:172.21.15.167:6800/30035 35 ==== osd_op_reply(102 foo [call out=48b,getxattrs out=75b,stat out=16b] v0'0 uv15 ondisk = 0) v8 ==== 231+0+139 (crc 0 0 0) 0x7f93f4052040 con 0x55c469eb0310 2023-10-05T19:28:57.805 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.798+0000 7f94071c4e40 10 cache put: name=default.rgw.meta+users.uid+foo info.flags=0x16 2023-10-05T19:28:57.805 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.798+0000 7f94071c4e40 10 adding default.rgw.meta+users.uid+foo to cache LRU end 2023-10-05T19:28:57.805 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.798+0000 7f94071c4e40 10 updating xattr: name=ceph.objclass.version bl.length()=42 2023-10-05T19:28:57.805 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.798+0000 7f94071c4e40 20 get_system_obj_state: s->obj_tag was set empty 2023-10-05T19:28:57.805 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.798+0000 7f94071c4e40 20 Read xattr: user.rgw.idtag 2023-10-05T19:28:57.805 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.798+0000 7f94071c4e40 10 cache get: name=default.rgw.meta+users.uid+foo : type miss (requested=0x13, cached=0x16) 2023-10-05T19:28:57.805 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.798+0000 7f94071c4e40 20 rados->read ofs=0 len=0 2023-10-05T19:28:57.805 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.798+0000 7f94071c4e40 1 -- 172.21.15.167:0/1717304001 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:103 8.18 8:1e6ca5ed:users.uid::foo:head [call version.check_conds in=122b,call version.read in=11b,read 0~0,getxattrs] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32) v8 -- 0x55c469f7de40 con 0x55c469eb0310 2023-10-05T19:28:57.805 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.798+0000 7f94054fa700 1 -- 172.21.15.167:0/1717304001 <== osd.1 v2:172.21.15.167:6800/30035 36 ==== osd_op_reply(103 foo [call,call out=48b,read 0~357 out=357b,getxattrs out=75b] v0'0 uv15 ondisk = 0) v8 ==== 273+0+480 (crc 0 0 0) 0x7f93f4052040 con 0x55c469eb0310 2023-10-05T19:28:57.805 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.798+0000 7f94071c4e40 20 rados_obj.operate() r=0 bl.length=357 2023-10-05T19:28:57.806 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.798+0000 7f94071c4e40 10 cache put: name=default.rgw.meta+users.uid+foo info.flags=0x13 2023-10-05T19:28:57.806 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.798+0000 7f94071c4e40 10 moving default.rgw.meta+users.uid+foo to cache LRU end 2023-10-05T19:28:57.806 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.798+0000 7f94071c4e40 10 updating xattr: name=ceph.objclass.version bl.length()=42 2023-10-05T19:28:57.806 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.798+0000 7f94071c4e40 1 -- 172.21.15.167:0/1717304001 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:104 8.9 8:96d98480:users.uid::foo.buckets:head [call user.list_buckets in=34b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32) v8 -- 0x55c469f7de40 con 0x7f93fc016b60 2023-10-05T19:28:57.806 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.802+0000 7f94044f8700 1 -- 172.21.15.167:0/1717304001 <== osd.2 v2:172.21.15.167:6816/30037 17 ==== osd_op_reply(104 foo.buckets [call out=183b] v0'0 uv43 ondisk = 0) v8 ==== 155+0+183 (crc 0 0 0) 0x7f93ec05b0c0 con 0x7f93fc016b60 2023-10-05T19:28:57.806 INFO:teuthology.orchestra.run.smithi167.stderr:could not remove user: unable to remove user, must specify purge data to remove user with buckets 2023-10-05T19:28:57.807 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.802+0000 7f94071c4e40 1 -- 172.21.15.167:0/1717304001 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:105 7.1f 7:f95f44c2:::notify.0:head [watch unwatch cookie 94302079189120] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e32) v8 -- 0x55c469f39960 con 0x7f93fc016b60 2023-10-05T19:28:57.809 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.806+0000 7f94044f8700 1 -- 172.21.15.167:0/1717304001 <== osd.2 v2:172.21.15.167:6816/30037 18 ==== osd_op_reply(105 notify.0 [watch unwatch cookie 94302079189120] v32'218 uv216 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f93ec05b0c0 con 0x7f93fc016b60 2023-10-05T19:28:57.809 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.806+0000 7f94071c4e40 20 remove_watcher() i=0 2023-10-05T19:28:57.809 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.806+0000 7f94071c4e40 2 removed watcher, disabling cache 2023-10-05T19:28:57.810 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.806+0000 7f94071c4e40 1 -- 172.21.15.167:0/1717304001 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:106 7.0 7:05bf5b68:::notify.1:head [watch unwatch cookie 94302079195584] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e32) v8 -- 0x55c469ec17b0 con 0x55c469eb0310 2023-10-05T19:28:57.812 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.806+0000 7f94054fa700 1 -- 172.21.15.167:0/1717304001 <== osd.1 v2:172.21.15.167:6800/30035 37 ==== osd_op_reply(106 notify.1 [watch unwatch cookie 94302079195584] v32'218 uv216 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f93f4052040 con 0x55c469eb0310 2023-10-05T19:28:57.812 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.806+0000 7f94071c4e40 20 remove_watcher() i=1 2023-10-05T19:28:57.812 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.806+0000 7f94071c4e40 1 -- 172.21.15.167:0/1717304001 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:107 7.15 7:a93a5511:::notify.2:head [watch unwatch cookie 94302079191328] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e32) v8 -- 0x55c469ec17b0 con 0x55c469ee37d0 2023-10-05T19:28:57.815 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.810+0000 7f9404cf9700 1 -- 172.21.15.167:0/1717304001 <== osd.3 v2:172.21.15.167:6808/30036 24 ==== osd_op_reply(107 notify.2 [watch unwatch cookie 94302079191328] v32'218 uv216 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f93fc050040 con 0x55c469ee37d0 2023-10-05T19:28:57.815 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.810+0000 7f94071c4e40 20 remove_watcher() i=2 2023-10-05T19:28:57.815 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.810+0000 7f94071c4e40 1 -- 172.21.15.167:0/1717304001 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:108 7.e 7:7759931f:::notify.3:head [watch unwatch cookie 94302079203952] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e32) v8 -- 0x55c469ec17b0 con 0x7f93fc016b60 2023-10-05T19:28:57.817 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.814+0000 7f94044f8700 1 -- 172.21.15.167:0/1717304001 <== osd.2 v2:172.21.15.167:6816/30037 19 ==== osd_op_reply(108 notify.3 [watch unwatch cookie 94302079203952] v32'218 uv216 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f93ec05b0c0 con 0x7f93fc016b60 2023-10-05T19:28:57.818 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.814+0000 7f94071c4e40 20 remove_watcher() i=3 2023-10-05T19:28:57.818 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.814+0000 7f94071c4e40 1 -- 172.21.15.167:0/1717304001 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:109 7.d 7:b4812045:::notify.4:head [watch unwatch cookie 94302079199392] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e32) v8 -- 0x55c469ec17b0 con 0x55c469eb0310 2023-10-05T19:28:57.821 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.814+0000 7f94054fa700 1 -- 172.21.15.167:0/1717304001 <== osd.1 v2:172.21.15.167:6800/30035 38 ==== osd_op_reply(109 notify.4 [watch unwatch cookie 94302079199392] v32'218 uv216 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f93f4052040 con 0x55c469eb0310 2023-10-05T19:28:57.821 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.814+0000 7f94071c4e40 20 remove_watcher() i=4 2023-10-05T19:28:57.821 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.814+0000 7f94071c4e40 1 -- 172.21.15.167:0/1717304001 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:110 7.3 7:c609908c:::notify.5:head [watch unwatch cookie 94302079208800] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e32) v8 -- 0x55c469ec17b0 con 0x7f93ec007030 2023-10-05T19:28:57.824 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.818+0000 7f94054fa700 1 -- 172.21.15.167:0/1717304001 <== osd.0 v2:172.21.15.118:6800/27690 29 ==== osd_op_reply(110 notify.5 [watch unwatch cookie 94302079208800] v32'218 uv216 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f93f4052040 con 0x7f93ec007030 2023-10-05T19:28:57.824 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.818+0000 7f94071c4e40 20 remove_watcher() i=5 2023-10-05T19:28:57.825 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.818+0000 7f94071c4e40 1 -- 172.21.15.167:0/1717304001 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:111 7.14 7:2b04a3e9:::notify.6:head [watch unwatch cookie 94302079207264] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e32) v8 -- 0x55c469ec17b0 con 0x7f93ec007030 2023-10-05T19:28:57.827 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.822+0000 7f94054fa700 1 -- 172.21.15.167:0/1717304001 <== osd.0 v2:172.21.15.118:6800/27690 30 ==== osd_op_reply(111 notify.6 [watch unwatch cookie 94302079207264] v32'218 uv216 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f93f4052040 con 0x7f93ec007030 2023-10-05T19:28:57.828 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.822+0000 7f94071c4e40 20 remove_watcher() i=6 2023-10-05T19:28:57.828 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.822+0000 7f94071c4e40 1 -- 172.21.15.167:0/1717304001 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:112 7.9 7:93e5b521:::notify.7:head [watch unwatch cookie 94302079209904] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e32) v8 -- 0x55c469ec17b0 con 0x7f93ec007030 2023-10-05T19:28:57.831 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.826+0000 7f94054fa700 1 -- 172.21.15.167:0/1717304001 <== osd.0 v2:172.21.15.118:6800/27690 31 ==== osd_op_reply(112 notify.7 [watch unwatch cookie 94302079209904] v32'218 uv216 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f93f4052040 con 0x7f93ec007030 2023-10-05T19:28:57.831 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.826+0000 7f94071c4e40 20 remove_watcher() i=7 2023-10-05T19:28:57.833 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.826+0000 7f93d67e4700 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2023-10-05T19:28:57.833 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.826+0000 7f94071c4e40 1 -- 172.21.15.167:0/1717304001 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:113 6.4 6:22d26bf9:::data_loggenerations_metadata:head [watch unwatch cookie 94302079214608] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e32) v8 -- 0x55c469ec17b0 con 0x55c469eb0310 2023-10-05T19:28:57.839 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.830+0000 7f94054fa700 1 -- 172.21.15.167:0/1717304001 <== osd.1 v2:172.21.15.167:6800/30035 39 ==== osd_op_reply(113 data_loggenerations_metadata [watch unwatch cookie 94302079214608] v32'218 uv1 ondisk = 0) v8 ==== 172+0+0 (crc 0 0 0) 0x7f93f4052040 con 0x55c469eb0310 2023-10-05T19:28:57.839 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.830+0000 7f94071c4e40 1 -- 172.21.15.167:0/1717304001 >> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] conn(0x7f93ec007030 msgr2=0x7f93ec01c410 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-05T19:28:57.839 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.830+0000 7f94071c4e40 1 --2- 172.21.15.167:0/1717304001 >> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] conn(0x7f93ec007030 0x7f93ec01c410 crc :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-05T19:28:57.839 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.830+0000 7f94071c4e40 1 -- 172.21.15.167:0/1717304001 >> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] conn(0x55c469eb0310 msgr2=0x55c469edf9a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-05T19:28:57.839 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.830+0000 7f94071c4e40 1 --2- 172.21.15.167:0/1717304001 >> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] conn(0x55c469eb0310 0x55c469edf9a0 crc :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-05T19:28:57.839 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.830+0000 7f94071c4e40 1 -- 172.21.15.167:0/1717304001 >> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] conn(0x7f93fc016b60 msgr2=0x7f93fc0243d0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-05T19:28:57.839 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.830+0000 7f94071c4e40 1 --2- 172.21.15.167:0/1717304001 >> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] conn(0x7f93fc016b60 0x7f93fc0243d0 crc :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-05T19:28:57.839 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.830+0000 7f94071c4e40 1 -- 172.21.15.167:0/1717304001 >> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] conn(0x55c469ee37d0 msgr2=0x55c469ee5bb0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-05T19:28:57.839 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.830+0000 7f94071c4e40 1 --2- 172.21.15.167:0/1717304001 >> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] conn(0x55c469ee37d0 0x55c469ee5bb0 crc :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-05T19:28:57.840 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.830+0000 7f94071c4e40 1 -- 172.21.15.167:0/1717304001 >> [v2:172.21.15.167:6824/29889,v1:172.21.15.167:6825/29889] conn(0x7f93e4030c00 msgr2=0x7f93e40330b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-05T19:28:57.840 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.830+0000 7f94071c4e40 1 --2- 172.21.15.167:0/1717304001 >> [v2:172.21.15.167:6824/29889,v1:172.21.15.167:6825/29889] conn(0x7f93e4030c00 0x7f93e40330b0 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7f93ec005a90 tx=0x7f93ec015000 comp rx=0 tx=0).stop 2023-10-05T19:28:57.840 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.830+0000 7f94071c4e40 1 -- 172.21.15.167:0/1717304001 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55c469e1fb40 msgr2=0x55c469ed3ee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-05T19:28:57.840 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.830+0000 7f94054fa700 1 -- 172.21.15.167:0/1717304001 reap_dead start 2023-10-05T19:28:57.840 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.830+0000 7f94071c4e40 1 --2- 172.21.15.167:0/1717304001 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55c469e1fb40 0x55c469ed3ee0 secure :-1 s=READY pgs=569 cs=0 l=1 rev1=1 crypto rx=0x7f93fc001820 tx=0x7f93fc016000 comp rx=0 tx=0).stop 2023-10-05T19:28:57.840 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.830+0000 7f94071c4e40 1 -- 172.21.15.167:0/1717304001 shutdown_connections 2023-10-05T19:28:57.840 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.830+0000 7f94071c4e40 1 --2- 172.21.15.167:0/1717304001 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55c469e1fb40 0x55c469ed3ee0 unknown :-1 s=CLOSED pgs=569 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-05T19:28:57.840 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.830+0000 7f94071c4e40 1 -- 172.21.15.167:0/1717304001 >> 172.21.15.167:0/1717304001 conn(0x55c469ea6740 msgr2=0x55c469ec5550 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-05T19:28:57.840 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.830+0000 7f94071c4e40 1 -- 172.21.15.167:0/1717304001 shutdown_connections 2023-10-05T19:28:57.840 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.830+0000 7f94071c4e40 1 -- 172.21.15.167:0/1717304001 wait complete. 2023-10-05T19:28:57.840 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.834+0000 7f94071c4e40 1 -- 172.21.15.167:0/438558575 >> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] conn(0x55c469f08460 msgr2=0x55c469f12950 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-05T19:28:57.840 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.834+0000 7f94071c4e40 1 --2- 172.21.15.167:0/438558575 >> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] conn(0x55c469f08460 0x55c469f12950 crc :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-05T19:28:57.841 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.834+0000 7f94071c4e40 1 -- 172.21.15.167:0/438558575 >> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] conn(0x55c469f0c880 msgr2=0x55c469f0ec90 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-05T19:28:57.841 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.834+0000 7f94071c4e40 1 --2- 172.21.15.167:0/438558575 >> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] conn(0x55c469f0c880 0x55c469f0ec90 crc :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-05T19:28:57.841 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.834+0000 7f94071c4e40 1 -- 172.21.15.167:0/438558575 >> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] conn(0x55c469f18030 msgr2=0x55c469f1a410 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-05T19:28:57.841 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.834+0000 7f94071c4e40 1 --2- 172.21.15.167:0/438558575 >> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] conn(0x55c469f18030 0x55c469f1a410 crc :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-05T19:28:57.841 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.834+0000 7f94071c4e40 1 -- 172.21.15.167:0/438558575 >> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] conn(0x55c469f13ee0 msgr2=0x55c469f162c0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-05T19:28:57.841 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.834+0000 7f94071c4e40 1 --2- 172.21.15.167:0/438558575 >> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] conn(0x55c469f13ee0 0x55c469f162c0 crc :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-05T19:28:57.841 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.834+0000 7f94071c4e40 1 -- 172.21.15.167:0/438558575 >> [v2:172.21.15.167:6824/29889,v1:172.21.15.167:6825/29889] conn(0x7f934c030ca0 msgr2=0x7f934c033150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-05T19:28:57.841 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.834+0000 7f94071c4e40 1 --2- 172.21.15.167:0/438558575 >> [v2:172.21.15.167:6824/29889,v1:172.21.15.167:6825/29889] conn(0x7f934c030ca0 0x7f934c033150 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x55c469eeee00 tx=0x7f93f400a280 comp rx=0 tx=0).stop 2023-10-05T19:28:57.841 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.834+0000 7f94071c4e40 1 -- 172.21.15.167:0/438558575 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55c469efc100 msgr2=0x55c469f01c30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-05T19:28:57.841 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.834+0000 7f94071c4e40 1 --2- 172.21.15.167:0/438558575 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55c469efc100 0x55c469f01c30 secure :-1 s=READY pgs=571 cs=0 l=1 rev1=1 crypto rx=0x55c469ef6180 tx=0x7f93ec02f000 comp rx=0 tx=0).stop 2023-10-05T19:28:57.841 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.834+0000 7f9404cf9700 1 -- 172.21.15.167:0/438558575 reap_dead start 2023-10-05T19:28:57.841 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.834+0000 7f9404cf9700 1 -- 172.21.15.167:0/438558575 reap_dead start 2023-10-05T19:28:57.841 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.834+0000 7f94071c4e40 1 -- 172.21.15.167:0/438558575 shutdown_connections 2023-10-05T19:28:57.842 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.834+0000 7f94071c4e40 1 -- 172.21.15.167:0/438558575 >> 172.21.15.167:0/438558575 conn(0x55c469ef74c0 msgr2=0x55c469ef5560 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-05T19:28:57.842 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.834+0000 7f94071c4e40 1 -- 172.21.15.167:0/438558575 shutdown_connections 2023-10-05T19:28:57.842 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:28:57.834+0000 7f94071c4e40 1 -- 172.21.15.167:0/438558575 wait complete. 2023-10-05T19:28:57.844 DEBUG:teuthology.orchestra.run:got remote process result: 17 2023-10-05T19:28:57.851 INFO:tasks.radosgw_admin:RL: create_bucket myfoo foo bytes_out=0 bytes_in=0 failed=False 2023-10-05T19:28:57.860 INFO:tasks.radosgw_admin:RL: put_obj myfoo foo bytes_out=6 bytes_in=0 failed=False 2023-10-05T19:29:32.862 INFO:tasks.util.rgw:rgwadmin: client.0 : ['usage', 'show', '--categories', 'delete_obj'] 2023-10-05T19:29:32.862 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'ceph', 'usage', 'show', '--categories', 'delete_obj'] 2023-10-05T19:29:32.863 DEBUG:teuthology.orchestra.run.smithi167:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster ceph usage show --categories delete_obj 2023-10-05T19:29:32.901 INFO:teuthology.orchestra.run.smithi167.stderr:ignoring --setuser ceph since I am not root 2023-10-05T19:29:32.901 INFO:teuthology.orchestra.run.smithi167.stderr:ignoring --setgroup ceph since I am not root 2023-10-05T19:29:32.902 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:32.898+0000 7fd8309f5e40 1 Processor -- start 2023-10-05T19:29:32.902 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:32.898+0000 7fd8309f5e40 1 -- start start 2023-10-05T19:29:32.902 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:32.898+0000 7fd8309f5e40 1 --2- >> v2:172.21.15.118:3300/0 conn(0x5576a688a410 0x5576a6888fc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-05T19:29:32.902 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:32.898+0000 7fd8309f5e40 1 -- --> v1:172.21.15.118:6789/0 -- auth(proto 0 26 bytes epoch 0) v1 -- 0x5576a67d6bd0 con 0x5576a688a7e0 2023-10-05T19:29:32.902 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:32.898+0000 7fd8309f5e40 1 -- --> v2:172.21.15.118:3300/0 -- mon_getmap magic: 0 v1 -- 0x5576a6793ee0 con 0x5576a688a410 2023-10-05T19:29:32.903 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:32.898+0000 7fd82e52a700 1 --2- >> v2:172.21.15.118:3300/0 conn(0x5576a688a410 0x5576a6888fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-05T19:29:32.903 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:32.898+0000 7fd82dd29700 1 --1- >> v1:172.21.15.118:6789/0 conn(0x5576a688a7e0 0x5576a6889500 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.118:6789/0 says I am v1:172.21.15.167:52576/0 (socket says 172.21.15.167:52576) 2023-10-05T19:29:32.903 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:32.898+0000 7fd82e52a700 1 --2- >> v2:172.21.15.118:3300/0 conn(0x5576a688a410 0x5576a6888fc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.118:3300/0 says I am v2:172.21.15.167:46110/0 (socket says 172.21.15.167:46110) 2023-10-05T19:29:32.904 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:32.898+0000 7fd82dd29700 1 -- 172.21.15.167:0/2958760642 learned_addr learned my addr 172.21.15.167:0/2958760642 (peer_addr_for_me v1:172.21.15.167:0/0) 2023-10-05T19:29:32.904 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:32.898+0000 7fd82e52a700 1 -- 172.21.15.167:0/2958760642 >> v1:172.21.15.118:6789/0 conn(0x5576a688a7e0 legacy=0x5576a6889500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-05T19:29:32.904 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:32.898+0000 7fd82e52a700 1 -- 172.21.15.167:0/2958760642 --> v2:172.21.15.118:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5576a6889cd0 con 0x5576a688a410 2023-10-05T19:29:32.905 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:32.898+0000 7fd82e52a700 1 --2- 172.21.15.167:0/2958760642 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5576a688a410 0x5576a6888fc0 secure :-1 s=READY pgs=573 cs=0 l=1 rev1=1 crypto rx=0x7fd82800a520 tx=0x7fd828005d40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=93bbd4ca5231851e server_cookie=0 in_seq=0 out_seq=0 2023-10-05T19:29:32.905 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:32.898+0000 7fd82d528700 1 -- 172.21.15.167:0/2958760642 <== mon.0 v2:172.21.15.118:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fd82800f590 con 0x5576a688a410 2023-10-05T19:29:32.905 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:32.898+0000 7fd82d528700 1 -- 172.21.15.167:0/2958760642 <== mon.0 v2:172.21.15.118:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd82800f6f0 con 0x5576a688a410 2023-10-05T19:29:32.905 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:32.898+0000 7fd82d528700 1 -- 172.21.15.167:0/2958760642 <== mon.0 v2:172.21.15.118:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fd82800f880 con 0x5576a688a410 2023-10-05T19:29:32.905 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:32.902+0000 7fd8309f5e40 1 -- 172.21.15.167:0/2958760642 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5576a688a410 msgr2=0x5576a6888fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-05T19:29:32.906 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:32.902+0000 7fd8309f5e40 1 --2- 172.21.15.167:0/2958760642 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5576a688a410 0x5576a6888fc0 secure :-1 s=READY pgs=573 cs=0 l=1 rev1=1 crypto rx=0x7fd82800a520 tx=0x7fd828005d40 comp rx=0 tx=0).stop 2023-10-05T19:29:32.906 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:32.902+0000 7fd8309f5e40 1 -- 172.21.15.167:0/2958760642 shutdown_connections 2023-10-05T19:29:32.906 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:32.902+0000 7fd8309f5e40 1 --2- 172.21.15.167:0/2958760642 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5576a688a410 0x5576a6888fc0 unknown :-1 s=CLOSED pgs=573 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-05T19:29:32.906 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:32.902+0000 7fd8309f5e40 1 -- 172.21.15.167:0/2958760642 >> 172.21.15.167:0/2958760642 conn(0x5576a6884720 msgr2=0x5576a6886b30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-05T19:29:32.906 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:32.902+0000 7fd8309f5e40 1 -- 172.21.15.167:0/2958760642 shutdown_connections 2023-10-05T19:29:32.906 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:32.902+0000 7fd8309f5e40 1 -- 172.21.15.167:0/2958760642 wait complete. 2023-10-05T19:29:32.907 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:32.902+0000 7fd8309f5e40 1 Processor -- start 2023-10-05T19:29:32.907 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:32.902+0000 7fd8309f5e40 1 -- start start 2023-10-05T19:29:32.908 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:32.902+0000 7fd8309f5e40 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5576a688a410 0x5576a68ab140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-05T19:29:32.908 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:32.902+0000 7fd8309f5e40 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x5576a6757e30 con 0x5576a688a410 2023-10-05T19:29:32.908 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:32.902+0000 7fd82e52a700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5576a688a410 0x5576a68ab140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-05T19:29:32.908 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:32.902+0000 7fd82e52a700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5576a688a410 0x5576a68ab140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.118:3300/0 says I am v2:172.21.15.167:46122/0 (socket says 172.21.15.167:46122) 2023-10-05T19:29:32.908 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:32.902+0000 7fd82e52a700 1 -- 172.21.15.167:0/1996252263 learned_addr learned my addr 172.21.15.167:0/1996252263 (peer_addr_for_me v2:172.21.15.167:0/0) 2023-10-05T19:29:32.908 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:32.902+0000 7fd82e52a700 1 -- 172.21.15.167:0/1996252263 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd82800a0e0 con 0x5576a688a410 2023-10-05T19:29:32.908 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:32.902+0000 7fd82e52a700 1 --2- 172.21.15.167:0/1996252263 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5576a688a410 0x5576a68ab140 secure :-1 s=READY pgs=574 cs=0 l=1 rev1=1 crypto rx=0x7fd82800a470 tx=0x7fd828014000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6368eeef6e9b0c4c server_cookie=0 in_seq=0 out_seq=0 2023-10-05T19:29:32.908 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:32.902+0000 7fd827fff700 1 -- 172.21.15.167:0/1996252263 <== mon.0 v2:172.21.15.118:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fd82800f860 con 0x5576a688a410 2023-10-05T19:29:32.908 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:32.902+0000 7fd827fff700 1 -- 172.21.15.167:0/1996252263 <== mon.0 v2:172.21.15.118:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd82800f9e0 con 0x5576a688a410 2023-10-05T19:29:32.909 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:32.902+0000 7fd827fff700 1 -- 172.21.15.167:0/1996252263 <== mon.0 v2:172.21.15.118:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fd82800fb90 con 0x5576a688a410 2023-10-05T19:29:32.911 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:32.902+0000 7fd8309f5e40 1 -- 172.21.15.167:0/1996252263 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5576a688a410 msgr2=0x5576a68ab140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-05T19:29:32.911 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:32.902+0000 7fd8309f5e40 1 --2- 172.21.15.167:0/1996252263 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5576a688a410 0x5576a68ab140 secure :-1 s=READY pgs=574 cs=0 l=1 rev1=1 crypto rx=0x7fd82800a470 tx=0x7fd828014000 comp rx=0 tx=0).stop 2023-10-05T19:29:32.911 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:32.902+0000 7fd8309f5e40 1 -- 172.21.15.167:0/1996252263 shutdown_connections 2023-10-05T19:29:32.911 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:32.902+0000 7fd8309f5e40 1 --2- 172.21.15.167:0/1996252263 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5576a688a410 0x5576a68ab140 unknown :-1 s=CLOSED pgs=574 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-05T19:29:32.911 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:32.902+0000 7fd8309f5e40 1 -- 172.21.15.167:0/1996252263 >> 172.21.15.167:0/1996252263 conn(0x5576a6884720 msgr2=0x5576a68a6d00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-05T19:29:32.911 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:32.902+0000 7fd8309f5e40 1 -- 172.21.15.167:0/1996252263 shutdown_connections 2023-10-05T19:29:32.911 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:32.902+0000 7fd8309f5e40 1 -- 172.21.15.167:0/1996252263 wait complete. 2023-10-05T19:29:32.911 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:32.902+0000 7fd8309f5e40 1 Processor -- start 2023-10-05T19:29:32.911 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:32.906+0000 7fd8309f5e40 1 -- start start 2023-10-05T19:29:32.911 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:32.906+0000 7fd8309f5e40 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5576a688a410 0x5576a68b1da0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-05T19:29:32.912 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:32.906+0000 7fd8309f5e40 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x5576a672aed0 con 0x5576a688a410 2023-10-05T19:29:32.912 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:32.906+0000 7fd82e52a700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5576a688a410 0x5576a68b1da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-05T19:29:32.912 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:32.906+0000 7fd82e52a700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5576a688a410 0x5576a68b1da0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.118:3300/0 says I am v2:172.21.15.167:46136/0 (socket says 172.21.15.167:46136) 2023-10-05T19:29:32.912 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:32.906+0000 7fd82e52a700 1 -- 172.21.15.167:0/3169388355 learned_addr learned my addr 172.21.15.167:0/3169388355 (peer_addr_for_me v2:172.21.15.167:0/0) 2023-10-05T19:29:32.912 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:32.906+0000 7fd82e52a700 1 -- 172.21.15.167:0/3169388355 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5576a6889cd0 con 0x5576a688a410 2023-10-05T19:29:32.912 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:32.906+0000 7fd82e52a700 1 --2- 172.21.15.167:0/3169388355 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5576a688a410 0x5576a68b1da0 secure :-1 s=READY pgs=575 cs=0 l=1 rev1=1 crypto rx=0x7fd82800a470 tx=0x7fd82801d000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-05T19:29:32.912 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:32.906+0000 7fd8267fc700 1 -- 172.21.15.167:0/3169388355 <== mon.0 v2:172.21.15.118:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fd82801ce00 con 0x5576a688a410 2023-10-05T19:29:32.912 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:32.906+0000 7fd8267fc700 1 -- 172.21.15.167:0/3169388355 <== mon.0 v2:172.21.15.118:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd828010630 con 0x5576a688a410 2023-10-05T19:29:32.912 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:32.906+0000 7fd8309f5e40 1 -- 172.21.15.167:0/3169388355 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x5576a68b22e0 con 0x5576a688a410 2023-10-05T19:29:32.912 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:32.906+0000 7fd8267fc700 1 -- 172.21.15.167:0/3169388355 <== mon.0 v2:172.21.15.118:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fd828010820 con 0x5576a688a410 2023-10-05T19:29:32.912 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:32.906+0000 7fd8309f5e40 1 -- 172.21.15.167:0/3169388355 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x5576a68b2720 con 0x5576a688a410 2023-10-05T19:29:32.914 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:32.906+0000 7fd8267fc700 1 -- 172.21.15.167:0/3169388355 <== mon.0 v2:172.21.15.118:3300/0 4 ==== mgrmap(e 5) v1 ==== 40329+0+0 (secure 0 0 0) 0x7fd828010a90 con 0x5576a688a410 2023-10-05T19:29:32.914 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:32.906+0000 7fd8309f5e40 20 rados->read ofs=0 len=0 2023-10-05T19:29:32.914 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:32.906+0000 7fd8267fc700 1 --2- 172.21.15.167:0/3169388355 >> [v2:172.21.15.167:6824/29889,v1:172.21.15.167:6825/29889] conn(0x7fd810030c00 0x7fd8100330b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-05T19:29:32.914 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:32.906+0000 7fd8267fc700 1 -- 172.21.15.167:0/3169388355 <== mon.0 v2:172.21.15.118:3300/0 5 ==== osd_map(32..32 src has 1..32) v4 ==== 6260+0+0 (secure 0 0 0) 0x7fd82802b070 con 0x5576a688a410 2023-10-05T19:29:32.914 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:32.906+0000 7fd82dd29700 1 --2- 172.21.15.167:0/3169388355 >> [v2:172.21.15.167:6824/29889,v1:172.21.15.167:6825/29889] conn(0x7fd810030c00 0x7fd8100330b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-05T19:29:32.915 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:32.910+0000 7fd8309f5e40 1 --2- 172.21.15.167:0/3169388355 >> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] conn(0x5576a688a7e0 0x5576a68bd800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-05T19:29:32.915 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:32.910+0000 7fd8309f5e40 1 -- 172.21.15.167:0/3169388355 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:1 5.12 5:49953fa1:::default.realm:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32) v8 -- 0x5576a68bdeb0 con 0x5576a688a7e0 2023-10-05T19:29:32.915 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:32.910+0000 7fd82ed2b700 1 --2- 172.21.15.167:0/3169388355 >> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] conn(0x5576a688a7e0 0x5576a68bd800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-05T19:29:32.915 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:32.910+0000 7fd82dd29700 1 --2- 172.21.15.167:0/3169388355 >> [v2:172.21.15.167:6824/29889,v1:172.21.15.167:6825/29889] conn(0x7fd810030c00 0x7fd8100330b0 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7fd818004170 tx=0x7fd818001380 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-05T19:29:32.915 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:32.910+0000 7fd82ed2b700 1 --2- 172.21.15.167:0/3169388355 >> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] conn(0x5576a688a7e0 0x5576a68bd800 crc :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-05T19:29:32.915 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:32.910+0000 7fd82ed2b700 1 -- 172.21.15.167:0/3169388355 <== osd.1 v2:172.21.15.167:6800/30035 1 ==== osd_op_reply(1 default.realm [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 ==== 157+0+0 (crc 0 0 0) 0x7fd82000a500 con 0x5576a688a7e0 2023-10-05T19:29:32.915 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:32.910+0000 7fd8309f5e40 20 rados_obj.operate() r=-2 bl.length=0 2023-10-05T19:29:32.915 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:32.910+0000 7fd8309f5e40 20 realm 2023-10-05T19:29:32.915 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:32.910+0000 7fd8309f5e40 20 rados->read ofs=0 len=0 2023-10-05T19:29:32.915 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:32.910+0000 7fd8309f5e40 1 -- 172.21.15.167:0/3169388355 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:2 5.12 5:49953fa1:::default.realm:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32) v8 -- 0x5576a68bae60 con 0x5576a688a7e0 2023-10-05T19:29:32.917 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:32.910+0000 7fd82ed2b700 1 -- 172.21.15.167:0/3169388355 <== osd.1 v2:172.21.15.167:6800/30035 2 ==== osd_op_reply(2 default.realm [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 ==== 157+0+0 (crc 0 0 0) 0x7fd82000a500 con 0x5576a688a7e0 2023-10-05T19:29:32.917 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:32.910+0000 7fd8309f5e40 20 rados_obj.operate() r=-2 bl.length=0 2023-10-05T19:29:32.917 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:32.910+0000 7fd8309f5e40 4 RGWPeriod::init failed to init realm id : (2) No such file or directory 2023-10-05T19:29:32.917 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:32.910+0000 7fd8309f5e40 20 rados->read ofs=0 len=0 2023-10-05T19:29:32.917 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:32.910+0000 7fd8309f5e40 1 -- 172.21.15.167:0/3169388355 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:3 5.12 5:49953fa1:::default.realm:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32) v8 -- 0x5576a68be990 con 0x5576a688a7e0 2023-10-05T19:29:32.917 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:32.910+0000 7fd82ed2b700 1 -- 172.21.15.167:0/3169388355 <== osd.1 v2:172.21.15.167:6800/30035 3 ==== osd_op_reply(3 default.realm [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 ==== 157+0+0 (crc 0 0 0) 0x7fd82000a500 con 0x5576a688a7e0 2023-10-05T19:29:32.918 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:32.910+0000 7fd8309f5e40 20 rados_obj.operate() r=-2 bl.length=0 2023-10-05T19:29:32.918 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:32.910+0000 7fd8309f5e40 20 rados->read ofs=0 len=0 2023-10-05T19:29:32.918 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:32.910+0000 7fd8309f5e40 1 -- 172.21.15.167:0/3169388355 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:4 5.f 5:f43b6ece:::zone_names.default:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32) v8 -- 0x5576a68be570 con 0x5576a688a7e0 2023-10-05T19:29:32.918 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:32.910+0000 7fd82ed2b700 1 -- 172.21.15.167:0/3169388355 <== osd.1 v2:172.21.15.167:6800/30035 4 ==== osd_op_reply(4 zone_names.default [read 0~46 out=46b] v0'0 uv1 ondisk = 0) v8 ==== 162+0+46 (crc 0 0 0) 0x7fd82000a500 con 0x5576a688a7e0 2023-10-05T19:29:32.918 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:32.910+0000 7fd8309f5e40 20 rados_obj.operate() r=0 bl.length=46 2023-10-05T19:29:32.918 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:32.910+0000 7fd8309f5e40 20 rados->read ofs=0 len=0 2023-10-05T19:29:32.918 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:32.910+0000 7fd8309f5e40 1 -- 172.21.15.167:0/3169388355 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:5 5.1d 5:b8374eb4:::zone_info.9a9df965-6612-45d4-a45f-6e3c889bacc2:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32) v8 -- 0x5576a68bf400 con 0x5576a688a7e0 2023-10-05T19:29:32.918 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:32.910+0000 7fd82ed2b700 1 -- 172.21.15.167:0/3169388355 <== osd.1 v2:172.21.15.167:6800/30035 5 ==== osd_op_reply(5 zone_info.9a9df965-6612-45d4-a45f-6e3c889bacc2 [read 0~874 out=874b] v0'0 uv2 ondisk = 0) v8 ==== 190+0+874 (crc 0 0 0) 0x7fd82000a500 con 0x5576a688a7e0 2023-10-05T19:29:32.918 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:32.910+0000 7fd8309f5e40 20 rados_obj.operate() r=0 bl.length=874 2023-10-05T19:29:32.918 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:32.914+0000 7fd8309f5e40 20 searching for the correct realm 2023-10-05T19:29:32.918 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:32.914+0000 7fd8309f5e40 1 --2- 172.21.15.167:0/3169388355 >> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] conn(0x5576a68c1250 0x5576a68c3630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-05T19:29:32.919 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:32.914+0000 7fd8309f5e40 1 -- 172.21.15.167:0/3169388355 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:6 5.0 5:00000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x5576a68c3b70 con 0x5576a68c1250 2023-10-05T19:29:32.919 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:32.914+0000 7fd82e52a700 1 --2- 172.21.15.167:0/3169388355 >> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] conn(0x5576a68c1250 0x5576a68c3630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-05T19:29:32.919 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:32.914+0000 7fd82e52a700 1 --2- 172.21.15.167:0/3169388355 >> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] conn(0x5576a68c1250 0x5576a68c3630 crc :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-05T19:29:32.919 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:32.914+0000 7fd82e52a700 1 -- 172.21.15.167:0/3169388355 <== osd.3 v2:172.21.15.167:6808/30036 1 ==== osd_op_reply(6 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fd82803ad70 con 0x5576a68c1250 2023-10-05T19:29:32.920 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:32.914+0000 7fd82e52a700 1 --2- 172.21.15.167:0/3169388355 >> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] conn(0x7fd82801a3c0 0x7fd828039540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-05T19:29:32.920 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:32.914+0000 7fd82e52a700 1 -- 172.21.15.167:0/3169388355 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:7 5.10 5:08000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x5576a68c3b70 con 0x7fd82801a3c0 2023-10-05T19:29:32.920 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:32.914+0000 7fd82dd29700 1 --2- 172.21.15.167:0/3169388355 >> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] conn(0x7fd82801a3c0 0x7fd828039540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-05T19:29:32.920 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:32.914+0000 7fd82dd29700 1 --2- 172.21.15.167:0/3169388355 >> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] conn(0x7fd82801a3c0 0x7fd828039540 crc :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-05T19:29:32.920 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:32.914+0000 7fd82dd29700 1 -- 172.21.15.167:0/3169388355 <== osd.2 v2:172.21.15.167:6816/30037 1 ==== osd_op_reply(7 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fd818006070 con 0x7fd82801a3c0 2023-10-05T19:29:32.921 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:32.914+0000 7fd82dd29700 1 --2- 172.21.15.167:0/3169388355 >> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] conn(0x7fd81800d3d0 0x7fd81800d820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-05T19:29:32.921 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:32.914+0000 7fd82dd29700 1 -- 172.21.15.167:0/3169388355 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:8 5.8 5:10000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x5576a68c3b70 con 0x7fd81800d3d0 2023-10-05T19:29:32.921 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:32.918+0000 7fd82ed2b700 1 --2- 172.21.15.167:0/3169388355 >> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] conn(0x7fd81800d3d0 0x7fd81800d820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-05T19:29:32.921 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:32.918+0000 7fd82ed2b700 1 --2- 172.21.15.167:0/3169388355 >> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] conn(0x7fd81800d3d0 0x7fd81800d820 crc :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-05T19:29:32.922 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:32.918+0000 7fd82ed2b700 1 -- 172.21.15.167:0/3169388355 <== osd.0 v2:172.21.15.118:6800/27690 1 ==== osd_op_reply(8 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fd820014040 con 0x7fd81800d3d0 2023-10-05T19:29:32.923 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:32.918+0000 7fd82ed2b700 1 -- 172.21.15.167:0/3169388355 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:9 5.18 5:18000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x5576a68c3b70 con 0x5576a688a7e0 2023-10-05T19:29:32.923 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:32.918+0000 7fd82ed2b700 1 -- 172.21.15.167:0/3169388355 <== osd.1 v2:172.21.15.167:6800/30035 6 ==== osd_op_reply(9 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fd820014040 con 0x5576a688a7e0 2023-10-05T19:29:32.923 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:32.918+0000 7fd82ed2b700 1 -- 172.21.15.167:0/3169388355 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:10 5.4 5:20000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x5576a68c3b70 con 0x5576a688a7e0 2023-10-05T19:29:32.924 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:32.918+0000 7fd82ed2b700 1 -- 172.21.15.167:0/3169388355 <== osd.1 v2:172.21.15.167:6800/30035 7 ==== osd_op_reply(10 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fd820014040 con 0x5576a688a7e0 2023-10-05T19:29:32.924 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:32.918+0000 7fd82ed2b700 1 -- 172.21.15.167:0/3169388355 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:11 5.14 5:28000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x5576a68c3b70 con 0x7fd81800d3d0 2023-10-05T19:29:32.925 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:32.922+0000 7fd82ed2b700 1 -- 172.21.15.167:0/3169388355 <== osd.0 v2:172.21.15.118:6800/27690 2 ==== osd_op_reply(11 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fd820014040 con 0x7fd81800d3d0 2023-10-05T19:29:32.925 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:32.922+0000 7fd82ed2b700 1 -- 172.21.15.167:0/3169388355 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:12 5.c 5:30000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x5576a68c3b70 con 0x7fd81800d3d0 2023-10-05T19:29:32.926 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:32.922+0000 7fd82ed2b700 1 -- 172.21.15.167:0/3169388355 <== osd.0 v2:172.21.15.118:6800/27690 3 ==== osd_op_reply(12 [pgnls start_epoch 32 out=112b] v23'1 uv1 ondisk = 1) v8 ==== 144+0+112 (crc 0 0 0) 0x7fd820014040 con 0x7fd81800d3d0 2023-10-05T19:29:32.926 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:32.922+0000 7fd82ed2b700 1 -- 172.21.15.167:0/3169388355 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:13 5.1c 5:38000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x5576a68c3b70 con 0x7fd82801a3c0 2023-10-05T19:29:32.927 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:32.922+0000 7fd82dd29700 1 -- 172.21.15.167:0/3169388355 <== osd.2 v2:172.21.15.167:6816/30037 2 ==== osd_op_reply(13 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fd818006070 con 0x7fd82801a3c0 2023-10-05T19:29:32.927 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:32.922+0000 7fd82dd29700 1 -- 172.21.15.167:0/3169388355 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:14 5.2 5:40000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x5576a68c3b70 con 0x5576a68c1250 2023-10-05T19:29:32.927 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:32.922+0000 7fd82e52a700 1 -- 172.21.15.167:0/3169388355 <== osd.3 v2:172.21.15.167:6808/30036 2 ==== osd_op_reply(14 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fd82803ad70 con 0x5576a68c1250 2023-10-05T19:29:32.927 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:32.922+0000 7fd82e52a700 1 -- 172.21.15.167:0/3169388355 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:15 5.12 5:48000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x5576a68c3b70 con 0x5576a688a7e0 2023-10-05T19:29:32.928 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:32.922+0000 7fd82ed2b700 1 -- 172.21.15.167:0/3169388355 <== osd.1 v2:172.21.15.167:6800/30035 8 ==== osd_op_reply(15 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fd820014040 con 0x5576a688a7e0 2023-10-05T19:29:32.928 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:32.922+0000 7fd82ed2b700 1 -- 172.21.15.167:0/3169388355 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:16 5.a 5:50000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x5576a68c3b70 con 0x7fd81800d3d0 2023-10-05T19:29:32.929 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:32.926+0000 7fd82ed2b700 1 -- 172.21.15.167:0/3169388355 <== osd.0 v2:172.21.15.118:6800/27690 4 ==== osd_op_reply(16 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fd820022040 con 0x7fd81800d3d0 2023-10-05T19:29:32.929 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:32.926+0000 7fd82ed2b700 1 -- 172.21.15.167:0/3169388355 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:17 5.1a 5:58000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x5576a68c3b70 con 0x5576a688a7e0 2023-10-05T19:29:32.930 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:32.926+0000 7fd82ed2b700 1 -- 172.21.15.167:0/3169388355 <== osd.1 v2:172.21.15.167:6800/30035 9 ==== osd_op_reply(17 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fd820022040 con 0x5576a688a7e0 2023-10-05T19:29:32.930 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:32.926+0000 7fd82ed2b700 1 -- 172.21.15.167:0/3169388355 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:18 5.6 5:60000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x5576a68c3b70 con 0x7fd82801a3c0 2023-10-05T19:29:32.930 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:32.926+0000 7fd82dd29700 1 -- 172.21.15.167:0/3169388355 <== osd.2 v2:172.21.15.167:6816/30037 3 ==== osd_op_reply(18 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fd818006070 con 0x7fd82801a3c0 2023-10-05T19:29:32.931 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:32.926+0000 7fd82dd29700 1 -- 172.21.15.167:0/3169388355 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:19 5.16 5:68000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x5576a68c3b70 con 0x5576a68c1250 2023-10-05T19:29:32.931 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:32.926+0000 7fd82e52a700 1 -- 172.21.15.167:0/3169388355 <== osd.3 v2:172.21.15.167:6808/30036 3 ==== osd_op_reply(19 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fd82803ad70 con 0x5576a68c1250 2023-10-05T19:29:32.931 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:32.926+0000 7fd82e52a700 1 -- 172.21.15.167:0/3169388355 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:20 5.e 5:70000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x5576a68c3b70 con 0x5576a68c1250 2023-10-05T19:29:32.931 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:32.926+0000 7fd82e52a700 1 -- 172.21.15.167:0/3169388355 <== osd.3 v2:172.21.15.167:6808/30036 4 ==== osd_op_reply(20 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fd82803ad70 con 0x5576a68c1250 2023-10-05T19:29:32.932 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:32.926+0000 7fd82e52a700 1 -- 172.21.15.167:0/3169388355 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:21 5.1e 5:78000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x5576a68c3b70 con 0x7fd81800d3d0 2023-10-05T19:29:32.933 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:32.930+0000 7fd82ed2b700 1 -- 172.21.15.167:0/3169388355 <== osd.0 v2:172.21.15.118:6800/27690 5 ==== osd_op_reply(21 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fd820022040 con 0x7fd81800d3d0 2023-10-05T19:29:32.933 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:32.930+0000 7fd82ed2b700 1 -- 172.21.15.167:0/3169388355 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:22 5.1 5:80000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x5576a68c3b70 con 0x5576a68c1250 2023-10-05T19:29:32.933 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:32.930+0000 7fd82e52a700 1 -- 172.21.15.167:0/3169388355 <== osd.3 v2:172.21.15.167:6808/30036 5 ==== osd_op_reply(22 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fd82803ad70 con 0x5576a68c1250 2023-10-05T19:29:32.933 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:32.930+0000 7fd82e52a700 1 -- 172.21.15.167:0/3169388355 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:23 5.11 5:88000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x5576a68c3b70 con 0x5576a68c1250 2023-10-05T19:29:32.934 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:32.930+0000 7fd82e52a700 1 -- 172.21.15.167:0/3169388355 <== osd.3 v2:172.21.15.167:6808/30036 6 ==== osd_op_reply(23 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fd82803ad70 con 0x5576a68c1250 2023-10-05T19:29:32.934 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:32.930+0000 7fd82e52a700 1 -- 172.21.15.167:0/3169388355 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:24 5.9 5:90000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x5576a68c3b70 con 0x5576a688a7e0 2023-10-05T19:29:32.935 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:32.930+0000 7fd82ed2b700 1 -- 172.21.15.167:0/3169388355 <== osd.1 v2:172.21.15.167:6800/30035 10 ==== osd_op_reply(24 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fd820022040 con 0x5576a688a7e0 2023-10-05T19:29:32.935 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:32.930+0000 7fd82ed2b700 1 -- 172.21.15.167:0/3169388355 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:25 5.19 5:98000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x5576a68c3b70 con 0x5576a688a7e0 2023-10-05T19:29:32.935 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:32.930+0000 7fd82ed2b700 1 -- 172.21.15.167:0/3169388355 <== osd.1 v2:172.21.15.167:6800/30035 11 ==== osd_op_reply(25 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fd820022040 con 0x5576a688a7e0 2023-10-05T19:29:32.936 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:32.930+0000 7fd82ed2b700 1 -- 172.21.15.167:0/3169388355 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:26 5.5 5:a0000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x5576a68c3b70 con 0x7fd81800d3d0 2023-10-05T19:29:32.936 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:32.934+0000 7fd82ed2b700 1 -- 172.21.15.167:0/3169388355 <== osd.0 v2:172.21.15.118:6800/27690 6 ==== osd_op_reply(26 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fd820022040 con 0x7fd81800d3d0 2023-10-05T19:29:32.936 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:32.934+0000 7fd82ed2b700 1 -- 172.21.15.167:0/3169388355 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:27 5.15 5:a8000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x5576a68c3b70 con 0x5576a68c1250 2023-10-05T19:29:32.937 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:32.934+0000 7fd82e52a700 1 -- 172.21.15.167:0/3169388355 <== osd.3 v2:172.21.15.167:6808/30036 7 ==== osd_op_reply(27 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fd82803ad70 con 0x5576a68c1250 2023-10-05T19:29:32.937 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:32.934+0000 7fd82e52a700 1 -- 172.21.15.167:0/3169388355 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:28 5.d 5:b0000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x5576a68c3b70 con 0x7fd82801a3c0 2023-10-05T19:29:32.938 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:32.934+0000 7fd82dd29700 1 -- 172.21.15.167:0/3169388355 <== osd.2 v2:172.21.15.167:6816/30037 4 ==== osd_op_reply(28 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fd818006070 con 0x7fd82801a3c0 2023-10-05T19:29:32.938 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:32.934+0000 7fd82dd29700 1 -- 172.21.15.167:0/3169388355 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:29 5.1d 5:b8000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x5576a68c3b70 con 0x5576a688a7e0 2023-10-05T19:29:32.939 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:32.934+0000 7fd82ed2b700 1 -- 172.21.15.167:0/3169388355 <== osd.1 v2:172.21.15.167:6800/30035 12 ==== osd_op_reply(29 [pgnls start_epoch 32 out=107b] v30'2 uv2 ondisk = 1) v8 ==== 144+0+107 (crc 0 0 0) 0x7fd820022040 con 0x5576a688a7e0 2023-10-05T19:29:32.939 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:32.934+0000 7fd82ed2b700 1 -- 172.21.15.167:0/3169388355 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:30 5.3 5:c0000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x5576a68c3b70 con 0x7fd81800d3d0 2023-10-05T19:29:32.940 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:32.934+0000 7fd82ed2b700 1 -- 172.21.15.167:0/3169388355 <== osd.0 v2:172.21.15.118:6800/27690 7 ==== osd_op_reply(30 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fd820022040 con 0x7fd81800d3d0 2023-10-05T19:29:32.940 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:32.934+0000 7fd82ed2b700 1 -- 172.21.15.167:0/3169388355 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:31 5.13 5:c8000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x5576a68c3b70 con 0x5576a68c1250 2023-10-05T19:29:32.940 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:32.938+0000 7fd82e52a700 1 -- 172.21.15.167:0/3169388355 <== osd.3 v2:172.21.15.167:6808/30036 8 ==== osd_op_reply(31 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fd82803ad70 con 0x5576a68c1250 2023-10-05T19:29:32.941 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:32.938+0000 7fd82e52a700 1 -- 172.21.15.167:0/3169388355 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:32 5.b 5:d0000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x5576a68c3b70 con 0x7fd81800d3d0 2023-10-05T19:29:32.942 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:32.938+0000 7fd82ed2b700 1 -- 172.21.15.167:0/3169388355 <== osd.0 v2:172.21.15.118:6800/27690 8 ==== osd_op_reply(32 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fd820022040 con 0x7fd81800d3d0 2023-10-05T19:29:32.942 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:32.938+0000 7fd82ed2b700 1 -- 172.21.15.167:0/3169388355 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:33 5.1b 5:d8000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x5576a68c3b70 con 0x7fd82801a3c0 2023-10-05T19:29:32.942 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:32.938+0000 7fd82dd29700 1 -- 172.21.15.167:0/3169388355 <== osd.2 v2:172.21.15.167:6816/30037 5 ==== osd_op_reply(33 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fd818006070 con 0x7fd82801a3c0 2023-10-05T19:29:32.942 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:32.938+0000 7fd82dd29700 1 -- 172.21.15.167:0/3169388355 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:34 5.7 5:e0000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x5576a68c3b70 con 0x5576a68c1250 2023-10-05T19:29:32.943 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:32.938+0000 7fd82e52a700 1 -- 172.21.15.167:0/3169388355 <== osd.3 v2:172.21.15.167:6808/30036 9 ==== osd_op_reply(34 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fd82803ad70 con 0x5576a68c1250 2023-10-05T19:29:32.943 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:32.938+0000 7fd82e52a700 1 -- 172.21.15.167:0/3169388355 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:35 5.17 5:e8000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x5576a68c3b70 con 0x7fd81800d3d0 2023-10-05T19:29:32.944 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:32.938+0000 7fd82ed2b700 1 -- 172.21.15.167:0/3169388355 <== osd.0 v2:172.21.15.118:6800/27690 9 ==== osd_op_reply(35 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fd820022040 con 0x7fd81800d3d0 2023-10-05T19:29:32.944 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:32.938+0000 7fd82ed2b700 1 -- 172.21.15.167:0/3169388355 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:36 5.f 5:f0000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x5576a68c3b70 con 0x5576a688a7e0 2023-10-05T19:29:32.945 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:32.942+0000 7fd82ed2b700 1 -- 172.21.15.167:0/3169388355 <== osd.1 v2:172.21.15.167:6800/30035 13 ==== osd_op_reply(36 [pgnls start_epoch 32 out=115b] v23'2 uv2 ondisk = 1) v8 ==== 144+0+115 (crc 0 0 0) 0x7fd82002f080 con 0x5576a688a7e0 2023-10-05T19:29:32.945 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:32.942+0000 7fd82ed2b700 1 -- 172.21.15.167:0/3169388355 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:37 5.1f 5:f8000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x5576a68c3b70 con 0x7fd82801a3c0 2023-10-05T19:29:32.946 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:32.942+0000 7fd82dd29700 1 -- 172.21.15.167:0/3169388355 <== osd.2 v2:172.21.15.167:6816/30037 6 ==== osd_op_reply(37 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fd818006070 con 0x7fd82801a3c0 2023-10-05T19:29:32.946 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:32.942+0000 7fd8309f5e40 20 RGWRados::pool_iterate: got zonegroup_info.4f1f719d-9648-417b-b10c-e5d939825419 2023-10-05T19:29:32.946 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:32.942+0000 7fd8309f5e40 20 RGWRados::pool_iterate: got zone_info.9a9df965-6612-45d4-a45f-6e3c889bacc2 2023-10-05T19:29:32.947 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:32.942+0000 7fd8309f5e40 20 RGWRados::pool_iterate: got zone_names.default 2023-10-05T19:29:32.947 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:32.942+0000 7fd8309f5e40 20 RGWRados::pool_iterate: got zonegroups_names.default 2023-10-05T19:29:32.947 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:32.942+0000 7fd8309f5e40 20 rados->read ofs=0 len=0 2023-10-05T19:29:32.947 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:32.942+0000 7fd8309f5e40 1 -- 172.21.15.167:0/3169388355 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:38 5.12 5:49953fa1:::default.realm:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32) v8 -- 0x5576a68c0000 con 0x5576a688a7e0 2023-10-05T19:29:32.948 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:32.942+0000 7fd82ed2b700 1 -- 172.21.15.167:0/3169388355 <== osd.1 v2:172.21.15.167:6800/30035 14 ==== osd_op_reply(38 default.realm [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 ==== 157+0+0 (crc 0 0 0) 0x7fd82002f080 con 0x5576a688a7e0 2023-10-05T19:29:32.948 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:32.942+0000 7fd8309f5e40 20 rados_obj.operate() r=-2 bl.length=0 2023-10-05T19:29:32.948 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:32.942+0000 7fd8309f5e40 20 rados->read ofs=0 len=0 2023-10-05T19:29:32.948 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:32.942+0000 7fd8309f5e40 1 -- 172.21.15.167:0/3169388355 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:39 5.f 5:f4c53578:::zonegroups_names.default:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32) v8 -- 0x5576a68c0980 con 0x5576a688a7e0 2023-10-05T19:29:32.948 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:32.942+0000 7fd82ed2b700 1 -- 172.21.15.167:0/3169388355 <== osd.1 v2:172.21.15.167:6800/30035 15 ==== osd_op_reply(39 zonegroups_names.default [read 0~46 out=46b] v0'0 uv2 ondisk = 0) v8 ==== 168+0+46 (crc 0 0 0) 0x7fd82002f080 con 0x5576a688a7e0 2023-10-05T19:29:32.948 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:32.942+0000 7fd8309f5e40 20 rados_obj.operate() r=0 bl.length=46 2023-10-05T19:29:32.948 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:32.942+0000 7fd8309f5e40 20 rados->read ofs=0 len=0 2023-10-05T19:29:32.948 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:32.942+0000 7fd8309f5e40 1 -- 172.21.15.167:0/3169388355 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:40 5.c 5:30d299a5:::zonegroup_info.4f1f719d-9648-417b-b10c-e5d939825419:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32) v8 -- 0x5576a68bfbf0 con 0x7fd81800d3d0 2023-10-05T19:29:32.949 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:32.946+0000 7fd82ed2b700 1 -- 172.21.15.167:0/3169388355 <== osd.0 v2:172.21.15.118:6800/27690 10 ==== osd_op_reply(40 zonegroup_info.4f1f719d-9648-417b-b10c-e5d939825419 [read 0~362 out=362b] v0'0 uv1 ondisk = 0) v8 ==== 195+0+362 (crc 0 0 0) 0x7fd82002f080 con 0x7fd81800d3d0 2023-10-05T19:29:32.949 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:32.946+0000 7fd8309f5e40 20 rados_obj.operate() r=0 bl.length=362 2023-10-05T19:29:32.949 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:32.946+0000 7fd8309f5e40 20 rados->read ofs=0 len=0 2023-10-05T19:29:32.949 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:32.946+0000 7fd8309f5e40 1 -- 172.21.15.167:0/3169388355 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:41 5.13 5:cbbf6c60:::converted:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32) v8 -- 0x5576a68c4460 con 0x5576a68c1250 2023-10-05T19:29:32.950 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:32.946+0000 7fd82e52a700 1 -- 172.21.15.167:0/3169388355 <== osd.3 v2:172.21.15.167:6808/30036 10 ==== osd_op_reply(41 converted [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 ==== 153+0+0 (crc 0 0 0) 0x7fd82803ad70 con 0x5576a68c1250 2023-10-05T19:29:32.950 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:32.946+0000 7fd8309f5e40 20 rados_obj.operate() r=-2 bl.length=0 2023-10-05T19:29:32.950 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:32.946+0000 7fd8309f5e40 20 rados->read ofs=0 len=0 2023-10-05T19:29:32.950 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:32.946+0000 7fd8309f5e40 1 -- 172.21.15.167:0/3169388355 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:42 5.12 5:49953fa1:::default.realm:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32) v8 -- 0x5576a68c4810 con 0x5576a688a7e0 2023-10-05T19:29:32.951 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:32.946+0000 7fd82ed2b700 1 -- 172.21.15.167:0/3169388355 <== osd.1 v2:172.21.15.167:6800/30035 16 ==== osd_op_reply(42 default.realm [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 ==== 157+0+0 (crc 0 0 0) 0x7fd82002f080 con 0x5576a688a7e0 2023-10-05T19:29:32.951 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:32.946+0000 7fd8309f5e40 20 rados_obj.operate() r=-2 bl.length=0 2023-10-05T19:29:32.951 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:32.946+0000 7fd8309f5e40 20 rados->read ofs=0 len=0 2023-10-05T19:29:32.951 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:32.946+0000 7fd8309f5e40 1 -- 172.21.15.167:0/3169388355 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:43 5.f 5:f4c53578:::zonegroups_names.default:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32) v8 -- 0x5576a68c4810 con 0x5576a688a7e0 2023-10-05T19:29:32.952 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:32.946+0000 7fd82ed2b700 1 -- 172.21.15.167:0/3169388355 <== osd.1 v2:172.21.15.167:6800/30035 17 ==== osd_op_reply(43 zonegroups_names.default [read 0~46 out=46b] v0'0 uv2 ondisk = 0) v8 ==== 168+0+46 (crc 0 0 0) 0x7fd82002f080 con 0x5576a688a7e0 2023-10-05T19:29:32.952 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:32.946+0000 7fd8309f5e40 20 rados_obj.operate() r=0 bl.length=46 2023-10-05T19:29:32.952 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:32.946+0000 7fd8309f5e40 1 -- 172.21.15.167:0/3169388355 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:44 5.0 5:00000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x5576a68c5820 con 0x5576a68c1250 2023-10-05T19:29:32.953 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:32.950+0000 7fd82e52a700 1 -- 172.21.15.167:0/3169388355 <== osd.3 v2:172.21.15.167:6808/30036 11 ==== osd_op_reply(44 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fd82803ad70 con 0x5576a68c1250 2023-10-05T19:29:32.953 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:32.950+0000 7fd82e52a700 1 -- 172.21.15.167:0/3169388355 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:45 5.10 5:08000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x5576a68c5820 con 0x7fd82801a3c0 2023-10-05T19:29:32.954 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:32.950+0000 7fd82dd29700 1 -- 172.21.15.167:0/3169388355 <== osd.2 v2:172.21.15.167:6816/30037 7 ==== osd_op_reply(45 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fd818006070 con 0x7fd82801a3c0 2023-10-05T19:29:32.954 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:32.950+0000 7fd82dd29700 1 -- 172.21.15.167:0/3169388355 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:46 5.8 5:10000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x5576a68c5820 con 0x7fd81800d3d0 2023-10-05T19:29:32.955 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:32.950+0000 7fd82ed2b700 1 -- 172.21.15.167:0/3169388355 <== osd.0 v2:172.21.15.118:6800/27690 11 ==== osd_op_reply(46 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fd82002f080 con 0x7fd81800d3d0 2023-10-05T19:29:32.955 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:32.950+0000 7fd82ed2b700 1 -- 172.21.15.167:0/3169388355 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:47 5.18 5:18000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x5576a68bfbf0 con 0x5576a688a7e0 2023-10-05T19:29:32.956 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:32.950+0000 7fd82ed2b700 1 -- 172.21.15.167:0/3169388355 <== osd.1 v2:172.21.15.167:6800/30035 18 ==== osd_op_reply(47 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fd82002f080 con 0x5576a688a7e0 2023-10-05T19:29:32.956 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:32.950+0000 7fd82ed2b700 1 -- 172.21.15.167:0/3169388355 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:48 5.4 5:20000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x5576a68bfbf0 con 0x5576a688a7e0 2023-10-05T19:29:32.956 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:32.954+0000 7fd82ed2b700 1 -- 172.21.15.167:0/3169388355 <== osd.1 v2:172.21.15.167:6800/30035 19 ==== osd_op_reply(48 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fd82002f080 con 0x5576a688a7e0 2023-10-05T19:29:32.956 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:32.954+0000 7fd82ed2b700 1 -- 172.21.15.167:0/3169388355 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:49 5.14 5:28000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x5576a68bfbf0 con 0x7fd81800d3d0 2023-10-05T19:29:32.957 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:32.954+0000 7fd82ed2b700 1 -- 172.21.15.167:0/3169388355 <== osd.0 v2:172.21.15.118:6800/27690 12 ==== osd_op_reply(49 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fd82002f080 con 0x7fd81800d3d0 2023-10-05T19:29:32.958 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:32.954+0000 7fd82ed2b700 1 -- 172.21.15.167:0/3169388355 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:50 5.c 5:30000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x5576a68bfbf0 con 0x7fd81800d3d0 2023-10-05T19:29:32.959 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:32.954+0000 7fd82ed2b700 1 -- 172.21.15.167:0/3169388355 <== osd.0 v2:172.21.15.118:6800/27690 13 ==== osd_op_reply(50 [pgnls start_epoch 32 out=112b] v23'1 uv1 ondisk = 1) v8 ==== 144+0+112 (crc 0 0 0) 0x7fd82002f080 con 0x7fd81800d3d0 2023-10-05T19:29:32.959 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:32.954+0000 7fd82ed2b700 1 -- 172.21.15.167:0/3169388355 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:51 5.1c 5:38000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x5576a68bfbf0 con 0x7fd82801a3c0 2023-10-05T19:29:32.960 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:32.954+0000 7fd82dd29700 1 -- 172.21.15.167:0/3169388355 <== osd.2 v2:172.21.15.167:6816/30037 8 ==== osd_op_reply(51 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fd818006070 con 0x7fd82801a3c0 2023-10-05T19:29:32.960 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:32.954+0000 7fd82dd29700 1 -- 172.21.15.167:0/3169388355 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:52 5.2 5:40000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x5576a68bfbf0 con 0x5576a68c1250 2023-10-05T19:29:32.960 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:32.958+0000 7fd82e52a700 1 -- 172.21.15.167:0/3169388355 <== osd.3 v2:172.21.15.167:6808/30036 12 ==== osd_op_reply(52 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fd82803ad70 con 0x5576a68c1250 2023-10-05T19:29:32.961 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:32.958+0000 7fd82e52a700 1 -- 172.21.15.167:0/3169388355 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:53 5.12 5:48000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x5576a68bfbf0 con 0x5576a688a7e0 2023-10-05T19:29:32.961 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:32.958+0000 7fd82ed2b700 1 -- 172.21.15.167:0/3169388355 <== osd.1 v2:172.21.15.167:6800/30035 20 ==== osd_op_reply(53 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fd82002f080 con 0x5576a688a7e0 2023-10-05T19:29:32.961 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:32.958+0000 7fd82ed2b700 1 -- 172.21.15.167:0/3169388355 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:54 5.a 5:50000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x5576a68bfbf0 con 0x7fd81800d3d0 2023-10-05T19:29:32.962 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:32.958+0000 7fd82ed2b700 1 -- 172.21.15.167:0/3169388355 <== osd.0 v2:172.21.15.118:6800/27690 14 ==== osd_op_reply(54 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fd82002f080 con 0x7fd81800d3d0 2023-10-05T19:29:32.962 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:32.958+0000 7fd82ed2b700 1 -- 172.21.15.167:0/3169388355 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:55 5.1a 5:58000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x5576a68bfbf0 con 0x5576a688a7e0 2023-10-05T19:29:32.963 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:32.958+0000 7fd82ed2b700 1 -- 172.21.15.167:0/3169388355 <== osd.1 v2:172.21.15.167:6800/30035 21 ==== osd_op_reply(55 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fd82002f080 con 0x5576a688a7e0 2023-10-05T19:29:32.963 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:32.958+0000 7fd82ed2b700 1 -- 172.21.15.167:0/3169388355 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:56 5.6 5:60000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x5576a68bfbf0 con 0x7fd82801a3c0 2023-10-05T19:29:32.964 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:32.958+0000 7fd82dd29700 1 -- 172.21.15.167:0/3169388355 <== osd.2 v2:172.21.15.167:6816/30037 9 ==== osd_op_reply(56 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fd818006070 con 0x7fd82801a3c0 2023-10-05T19:29:32.964 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:32.958+0000 7fd82dd29700 1 -- 172.21.15.167:0/3169388355 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:57 5.16 5:68000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x5576a68bfbf0 con 0x5576a68c1250 2023-10-05T19:29:32.965 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:32.962+0000 7fd82e52a700 1 -- 172.21.15.167:0/3169388355 <== osd.3 v2:172.21.15.167:6808/30036 13 ==== osd_op_reply(57 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fd82803ad70 con 0x5576a68c1250 2023-10-05T19:29:32.965 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:32.962+0000 7fd82e52a700 1 -- 172.21.15.167:0/3169388355 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:58 5.e 5:70000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x5576a68bfbf0 con 0x5576a68c1250 2023-10-05T19:29:32.966 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:32.962+0000 7fd82e52a700 1 -- 172.21.15.167:0/3169388355 <== osd.3 v2:172.21.15.167:6808/30036 14 ==== osd_op_reply(58 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fd82803ad70 con 0x5576a68c1250 2023-10-05T19:29:32.966 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:32.962+0000 7fd82e52a700 1 -- 172.21.15.167:0/3169388355 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:59 5.1e 5:78000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x5576a68bfbf0 con 0x7fd81800d3d0 2023-10-05T19:29:32.967 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:32.962+0000 7fd82ed2b700 1 -- 172.21.15.167:0/3169388355 <== osd.0 v2:172.21.15.118:6800/27690 15 ==== osd_op_reply(59 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fd82002f080 con 0x7fd81800d3d0 2023-10-05T19:29:32.967 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:32.962+0000 7fd82ed2b700 1 -- 172.21.15.167:0/3169388355 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:60 5.1 5:80000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x5576a68bfbf0 con 0x5576a68c1250 2023-10-05T19:29:32.968 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:32.962+0000 7fd82e52a700 1 -- 172.21.15.167:0/3169388355 <== osd.3 v2:172.21.15.167:6808/30036 15 ==== osd_op_reply(60 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fd82803ad70 con 0x5576a68c1250 2023-10-05T19:29:32.968 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:32.962+0000 7fd82e52a700 1 -- 172.21.15.167:0/3169388355 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:61 5.11 5:88000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x5576a68bfbf0 con 0x5576a68c1250 2023-10-05T19:29:32.969 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:32.966+0000 7fd82e52a700 1 -- 172.21.15.167:0/3169388355 <== osd.3 v2:172.21.15.167:6808/30036 16 ==== osd_op_reply(61 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fd82803ad70 con 0x5576a68c1250 2023-10-05T19:29:32.969 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:32.966+0000 7fd82e52a700 1 -- 172.21.15.167:0/3169388355 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:62 5.9 5:90000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x5576a68bfbf0 con 0x5576a688a7e0 2023-10-05T19:29:32.969 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:32.966+0000 7fd82ed2b700 1 -- 172.21.15.167:0/3169388355 <== osd.1 v2:172.21.15.167:6800/30035 22 ==== osd_op_reply(62 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fd82002f080 con 0x5576a688a7e0 2023-10-05T19:29:32.970 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:32.966+0000 7fd82ed2b700 1 -- 172.21.15.167:0/3169388355 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:63 5.19 5:98000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x5576a68bfbf0 con 0x5576a688a7e0 2023-10-05T19:29:32.970 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:32.966+0000 7fd82ed2b700 1 -- 172.21.15.167:0/3169388355 <== osd.1 v2:172.21.15.167:6800/30035 23 ==== osd_op_reply(63 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fd82002f080 con 0x5576a688a7e0 2023-10-05T19:29:32.970 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:32.966+0000 7fd82ed2b700 1 -- 172.21.15.167:0/3169388355 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:64 5.5 5:a0000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x5576a68bfbf0 con 0x7fd81800d3d0 2023-10-05T19:29:32.971 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:32.966+0000 7fd82ed2b700 1 -- 172.21.15.167:0/3169388355 <== osd.0 v2:172.21.15.118:6800/27690 16 ==== osd_op_reply(64 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fd82002f080 con 0x7fd81800d3d0 2023-10-05T19:29:32.972 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:32.966+0000 7fd82ed2b700 1 -- 172.21.15.167:0/3169388355 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:65 5.15 5:a8000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x5576a68bfbf0 con 0x5576a68c1250 2023-10-05T19:29:32.972 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:32.966+0000 7fd82e52a700 1 -- 172.21.15.167:0/3169388355 <== osd.3 v2:172.21.15.167:6808/30036 17 ==== osd_op_reply(65 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fd82803ad70 con 0x5576a68c1250 2023-10-05T19:29:32.973 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:32.970+0000 7fd82e52a700 1 -- 172.21.15.167:0/3169388355 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:66 5.d 5:b0000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x5576a68bfbf0 con 0x7fd82801a3c0 2023-10-05T19:29:32.974 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:32.970+0000 7fd82dd29700 1 -- 172.21.15.167:0/3169388355 <== osd.2 v2:172.21.15.167:6816/30037 10 ==== osd_op_reply(66 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fd818006070 con 0x7fd82801a3c0 2023-10-05T19:29:32.974 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:32.970+0000 7fd82dd29700 1 -- 172.21.15.167:0/3169388355 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:67 5.1d 5:b8000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x5576a68bfbf0 con 0x5576a688a7e0 2023-10-05T19:29:32.974 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:32.970+0000 7fd82ed2b700 1 -- 172.21.15.167:0/3169388355 <== osd.1 v2:172.21.15.167:6800/30035 24 ==== osd_op_reply(67 [pgnls start_epoch 32 out=107b] v30'2 uv2 ondisk = 1) v8 ==== 144+0+107 (crc 0 0 0) 0x7fd82002f080 con 0x5576a688a7e0 2023-10-05T19:29:32.974 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:32.970+0000 7fd82ed2b700 1 -- 172.21.15.167:0/3169388355 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:68 5.3 5:c0000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x5576a68bfbf0 con 0x7fd81800d3d0 2023-10-05T19:29:32.975 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:32.970+0000 7fd82ed2b700 1 -- 172.21.15.167:0/3169388355 <== osd.0 v2:172.21.15.118:6800/27690 17 ==== osd_op_reply(68 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fd82002f080 con 0x7fd81800d3d0 2023-10-05T19:29:32.975 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:32.970+0000 7fd82ed2b700 1 -- 172.21.15.167:0/3169388355 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:69 5.13 5:c8000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x5576a68bfbf0 con 0x5576a68c1250 2023-10-05T19:29:32.976 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:32.970+0000 7fd82e52a700 1 -- 172.21.15.167:0/3169388355 <== osd.3 v2:172.21.15.167:6808/30036 18 ==== osd_op_reply(69 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fd82803ad70 con 0x5576a68c1250 2023-10-05T19:29:32.976 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:32.970+0000 7fd82e52a700 1 -- 172.21.15.167:0/3169388355 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:70 5.b 5:d0000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x5576a68bfbf0 con 0x7fd81800d3d0 2023-10-05T19:29:32.977 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:32.974+0000 7fd82ed2b700 1 -- 172.21.15.167:0/3169388355 <== osd.0 v2:172.21.15.118:6800/27690 18 ==== osd_op_reply(70 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fd82002f080 con 0x7fd81800d3d0 2023-10-05T19:29:32.977 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:32.974+0000 7fd82ed2b700 1 -- 172.21.15.167:0/3169388355 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:71 5.1b 5:d8000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x5576a68bfbf0 con 0x7fd82801a3c0 2023-10-05T19:29:32.978 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:32.974+0000 7fd82dd29700 1 -- 172.21.15.167:0/3169388355 <== osd.2 v2:172.21.15.167:6816/30037 11 ==== osd_op_reply(71 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fd818006070 con 0x7fd82801a3c0 2023-10-05T19:29:32.978 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:32.974+0000 7fd82dd29700 1 -- 172.21.15.167:0/3169388355 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:72 5.7 5:e0000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x5576a68bfbf0 con 0x5576a68c1250 2023-10-05T19:29:32.979 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:32.974+0000 7fd82e52a700 1 -- 172.21.15.167:0/3169388355 <== osd.3 v2:172.21.15.167:6808/30036 19 ==== osd_op_reply(72 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fd82803ad70 con 0x5576a68c1250 2023-10-05T19:29:32.979 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:32.974+0000 7fd82e52a700 1 -- 172.21.15.167:0/3169388355 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:73 5.17 5:e8000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x5576a68bfbf0 con 0x7fd81800d3d0 2023-10-05T19:29:32.980 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:32.974+0000 7fd82ed2b700 1 -- 172.21.15.167:0/3169388355 <== osd.0 v2:172.21.15.118:6800/27690 19 ==== osd_op_reply(73 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fd82002f080 con 0x7fd81800d3d0 2023-10-05T19:29:32.980 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:32.974+0000 7fd82ed2b700 1 -- 172.21.15.167:0/3169388355 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:74 5.f 5:f0000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x5576a68bfbf0 con 0x5576a688a7e0 2023-10-05T19:29:32.981 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:32.978+0000 7fd82ed2b700 1 -- 172.21.15.167:0/3169388355 <== osd.1 v2:172.21.15.167:6800/30035 25 ==== osd_op_reply(74 [pgnls start_epoch 32 out=115b] v23'2 uv2 ondisk = 1) v8 ==== 144+0+115 (crc 0 0 0) 0x7fd82002f080 con 0x5576a688a7e0 2023-10-05T19:29:32.981 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:32.978+0000 7fd82ed2b700 1 -- 172.21.15.167:0/3169388355 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:75 5.1f 5:f8000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x5576a68bfbf0 con 0x7fd82801a3c0 2023-10-05T19:29:32.982 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:32.978+0000 7fd82dd29700 1 -- 172.21.15.167:0/3169388355 <== osd.2 v2:172.21.15.167:6816/30037 12 ==== osd_op_reply(75 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fd818006070 con 0x7fd82801a3c0 2023-10-05T19:29:32.982 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:32.978+0000 7fd8309f5e40 20 RGWRados::pool_iterate: got zonegroup_info.4f1f719d-9648-417b-b10c-e5d939825419 2023-10-05T19:29:32.982 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:32.978+0000 7fd8309f5e40 20 RGWRados::pool_iterate: got zone_info.9a9df965-6612-45d4-a45f-6e3c889bacc2 2023-10-05T19:29:32.982 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:32.978+0000 7fd8309f5e40 20 RGWRados::pool_iterate: got zone_names.default 2023-10-05T19:29:32.982 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:32.978+0000 7fd8309f5e40 20 RGWRados::pool_iterate: got zonegroups_names.default 2023-10-05T19:29:32.982 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:32.978+0000 7fd8309f5e40 20 rados->read ofs=0 len=0 2023-10-05T19:29:32.983 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:32.978+0000 7fd8309f5e40 1 -- 172.21.15.167:0/3169388355 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:76 5.f 5:f43b6ece:::zone_names.default:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32) v8 -- 0x5576a68c4810 con 0x5576a688a7e0 2023-10-05T19:29:32.983 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:32.978+0000 7fd82ed2b700 1 -- 172.21.15.167:0/3169388355 <== osd.1 v2:172.21.15.167:6800/30035 26 ==== osd_op_reply(76 zone_names.default [read 0~46 out=46b] v0'0 uv1 ondisk = 0) v8 ==== 162+0+46 (crc 0 0 0) 0x7fd82002f080 con 0x5576a688a7e0 2023-10-05T19:29:32.983 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:32.978+0000 7fd8309f5e40 20 rados_obj.operate() r=0 bl.length=46 2023-10-05T19:29:32.986 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:32.978+0000 7fd8309f5e40 20 rados->read ofs=0 len=0 2023-10-05T19:29:32.986 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:32.978+0000 7fd8309f5e40 1 -- 172.21.15.167:0/3169388355 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:77 5.1d 5:b8374eb4:::zone_info.9a9df965-6612-45d4-a45f-6e3c889bacc2:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32) v8 -- 0x5576a68c4bc0 con 0x5576a688a7e0 2023-10-05T19:29:32.988 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:32.978+0000 7fd82ed2b700 1 -- 172.21.15.167:0/3169388355 <== osd.1 v2:172.21.15.167:6800/30035 27 ==== osd_op_reply(77 zone_info.9a9df965-6612-45d4-a45f-6e3c889bacc2 [read 0~874 out=874b] v0'0 uv2 ondisk = 0) v8 ==== 190+0+874 (crc 0 0 0) 0x7fd82002f080 con 0x5576a688a7e0 2023-10-05T19:29:32.988 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:32.978+0000 7fd8309f5e40 20 rados_obj.operate() r=0 bl.length=874 2023-10-05T19:29:32.988 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:32.978+0000 7fd8309f5e40 20 rados->read ofs=0 len=0 2023-10-05T19:29:32.989 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:32.978+0000 7fd8309f5e40 1 -- 172.21.15.167:0/3169388355 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:78 5.f 5:f4c53578:::zonegroups_names.default:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32) v8 -- 0x5576a68c50b0 con 0x5576a688a7e0 2023-10-05T19:29:32.989 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:32.982+0000 7fd82ed2b700 1 -- 172.21.15.167:0/3169388355 <== osd.1 v2:172.21.15.167:6800/30035 28 ==== osd_op_reply(78 zonegroups_names.default [read 0~46 out=46b] v0'0 uv2 ondisk = 0) v8 ==== 168+0+46 (crc 0 0 0) 0x7fd82002f080 con 0x5576a688a7e0 2023-10-05T19:29:32.989 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:32.982+0000 7fd8309f5e40 20 rados_obj.operate() r=0 bl.length=46 2023-10-05T19:29:32.989 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:32.982+0000 7fd8309f5e40 20 rados->read ofs=0 len=0 2023-10-05T19:29:32.989 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:32.982+0000 7fd8309f5e40 1 -- 172.21.15.167:0/3169388355 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:79 5.c 5:30d299a5:::zonegroup_info.4f1f719d-9648-417b-b10c-e5d939825419:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32) v8 -- 0x5576a68c50b0 con 0x7fd81800d3d0 2023-10-05T19:29:32.989 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:32.982+0000 7fd82ed2b700 1 -- 172.21.15.167:0/3169388355 <== osd.0 v2:172.21.15.118:6800/27690 20 ==== osd_op_reply(79 zonegroup_info.4f1f719d-9648-417b-b10c-e5d939825419 [read 0~362 out=362b] v0'0 uv1 ondisk = 0) v8 ==== 195+0+362 (crc 0 0 0) 0x7fd82002f080 con 0x7fd81800d3d0 2023-10-05T19:29:32.989 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:32.982+0000 7fd8309f5e40 20 rados_obj.operate() r=0 bl.length=362 2023-10-05T19:29:32.989 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:32.982+0000 7fd8309f5e40 20 rados->read ofs=0 len=0 2023-10-05T19:29:32.989 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:32.982+0000 7fd8309f5e40 1 -- 172.21.15.167:0/3169388355 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:80 5.3 5:c0ad8cbd:::region_map:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32) v8 -- 0x5576a68c50e0 con 0x7fd81800d3d0 2023-10-05T19:29:32.989 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:32.982+0000 7fd82ed2b700 1 -- 172.21.15.167:0/3169388355 <== osd.0 v2:172.21.15.118:6800/27690 21 ==== osd_op_reply(80 region_map [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 ==== 154+0+0 (crc 0 0 0) 0x7fd820050040 con 0x7fd81800d3d0 2023-10-05T19:29:32.990 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:32.982+0000 7fd8309f5e40 20 rados_obj.operate() r=-2 bl.length=0 2023-10-05T19:29:32.990 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:32.982+0000 7fd8309f5e40 20 zone default found 2023-10-05T19:29:32.990 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:32.982+0000 7fd8309f5e40 4 Realm: () 2023-10-05T19:29:32.990 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:32.982+0000 7fd8309f5e40 4 ZoneGroup: default (4f1f719d-9648-417b-b10c-e5d939825419) 2023-10-05T19:29:32.990 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:32.982+0000 7fd8309f5e40 4 Zone: default (9a9df965-6612-45d4-a45f-6e3c889bacc2) 2023-10-05T19:29:32.990 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:32.982+0000 7fd8309f5e40 10 cannot find current period zonegroup using local zonegroup configuration 2023-10-05T19:29:32.990 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:32.982+0000 7fd8309f5e40 20 zonegroup default 2023-10-05T19:29:32.990 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:32.982+0000 7fd8309f5e40 20 rados->read ofs=0 len=0 2023-10-05T19:29:32.990 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:32.982+0000 7fd8309f5e40 1 -- 172.21.15.167:0/3169388355 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:81 5.3 5:c52100b6:::period_config.default:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32) v8 -- 0x5576a68c50e0 con 0x7fd81800d3d0 2023-10-05T19:29:32.991 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:32.982+0000 7fd82ed2b700 1 -- 172.21.15.167:0/3169388355 <== osd.0 v2:172.21.15.118:6800/27690 22 ==== osd_op_reply(81 period_config.default [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 ==== 165+0+0 (crc 0 0 0) 0x7fd820050040 con 0x7fd81800d3d0 2023-10-05T19:29:32.991 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:32.982+0000 7fd8309f5e40 20 rados_obj.operate() r=-2 bl.length=0 2023-10-05T19:29:32.991 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:32.982+0000 7fd8309f5e40 20 rados->read ofs=0 len=0 2023-10-05T19:29:32.991 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:32.982+0000 7fd8309f5e40 1 -- 172.21.15.167:0/3169388355 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:82 6.2 6:4347d321:::bucket.sync-source-hints.:head [call version.read in=11b,read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32) v8 -- 0x5576a68c6a60 con 0x5576a68c1250 2023-10-05T19:29:32.991 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:32.982+0000 7fd82e52a700 1 -- 172.21.15.167:0/3169388355 <== osd.3 v2:172.21.15.167:6808/30036 20 ==== osd_op_reply(82 bucket.sync-source-hints. [call,read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 ==== 211+0+0 (crc 0 0 0) 0x7fd82803ad70 con 0x5576a68c1250 2023-10-05T19:29:32.991 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:32.982+0000 7fd8309f5e40 20 rados_obj.operate() r=-2 bl.length=0 2023-10-05T19:29:32.991 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:32.982+0000 7fd8309f5e40 20 rados->read ofs=0 len=0 2023-10-05T19:29:32.991 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:32.982+0000 7fd8309f5e40 1 -- 172.21.15.167:0/3169388355 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:83 6.b 6:d467b91b:::bucket.sync-target-hints.:head [call version.read in=11b,read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32) v8 -- 0x5576a68c6e10 con 0x5576a68c1250 2023-10-05T19:29:32.992 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:32.986+0000 7fd82e52a700 1 -- 172.21.15.167:0/3169388355 <== osd.3 v2:172.21.15.167:6808/30036 21 ==== osd_op_reply(83 bucket.sync-target-hints. [call,read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 ==== 211+0+0 (crc 0 0 0) 0x7fd82805f040 con 0x5576a68c1250 2023-10-05T19:29:32.992 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:32.986+0000 7fd8309f5e40 20 rados_obj.operate() r=-2 bl.length=0 2023-10-05T19:29:32.992 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:32.986+0000 7fd8309f5e40 20 started sync module instance, tier type = 2023-10-05T19:29:32.993 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:32.986+0000 7fd8309f5e40 20 started zone id=9a9df965-6612-45d4-a45f-6e3c889bacc2 (name=default) with tier type = 2023-10-05T19:29:32.993 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:32.986+0000 7fd8309f5e40 1 -- 172.21.15.167:0/3169388355 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:84 7.1f 7:f95f44c2:::notify.0:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e32) v8 -- 0x5576a68c6120 con 0x7fd82801a3c0 2023-10-05T19:29:32.993 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:32.986+0000 7fd82dd29700 1 -- 172.21.15.167:0/3169388355 <== osd.2 v2:172.21.15.167:6816/30037 13 ==== osd_op_reply(84 notify.0 [create] v32'219 uv219 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7fd818006070 con 0x7fd82801a3c0 2023-10-05T19:29:32.993 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:32.986+0000 7fd8309f5e40 1 -- 172.21.15.167:0/3169388355 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:85 7.1f 7:f95f44c2:::notify.0:head [watch watch cookie 93968088720672] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e32) v8 -- 0x5576a68c5d50 con 0x7fd82801a3c0 2023-10-05T19:29:32.993 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:32.986+0000 7fd8309f5e40 1 -- 172.21.15.167:0/3169388355 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:86 7.0 7:05bf5b68:::notify.1:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e32) v8 -- 0x5576a68c7c30 con 0x5576a688a7e0 2023-10-05T19:29:32.996 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:32.990+0000 7fd82ed2b700 1 -- 172.21.15.167:0/3169388355 <== osd.1 v2:172.21.15.167:6800/30035 29 ==== osd_op_reply(86 notify.1 [create] v32'219 uv219 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7fd820050040 con 0x5576a688a7e0 2023-10-05T19:29:32.996 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:32.990+0000 7fd8309f5e40 1 -- 172.21.15.167:0/3169388355 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:87 7.0 7:05bf5b68:::notify.1:head [watch watch cookie 93968088725600] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e32) v8 -- 0x5576a68c8520 con 0x5576a688a7e0 2023-10-05T19:29:32.996 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:32.990+0000 7fd82dd29700 1 -- 172.21.15.167:0/3169388355 <== osd.2 v2:172.21.15.167:6816/30037 14 ==== osd_op_reply(85 notify.0 [watch watch cookie 93968088720672] v32'220 uv219 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7fd818006070 con 0x7fd82801a3c0 2023-10-05T19:29:32.996 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:32.990+0000 7fd8309f5e40 1 -- 172.21.15.167:0/3169388355 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:88 7.15 7:a93a5511:::notify.2:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e32) v8 -- 0x5576a68c88d0 con 0x5576a68c1250 2023-10-05T19:29:32.999 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:32.994+0000 7fd82e52a700 1 -- 172.21.15.167:0/3169388355 <== osd.3 v2:172.21.15.167:6808/30036 22 ==== osd_op_reply(88 notify.2 [create] v32'219 uv219 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7fd82805f040 con 0x5576a68c1250 2023-10-05T19:29:32.999 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:32.994+0000 7fd8309f5e40 1 -- 172.21.15.167:0/3169388355 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:89 7.15 7:a93a5511:::notify.2:head [watch watch cookie 93968088723904] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e32) v8 -- 0x5576a68c9170 con 0x5576a68c1250 2023-10-05T19:29:32.999 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:32.994+0000 7fd8309f5e40 1 -- 172.21.15.167:0/3169388355 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:90 7.e 7:7759931f:::notify.3:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e32) v8 -- 0x5576a68c9a50 con 0x7fd82801a3c0 2023-10-05T19:29:32.999 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:32.994+0000 7fd82ed2b700 1 -- 172.21.15.167:0/3169388355 <== osd.1 v2:172.21.15.167:6800/30035 30 ==== osd_op_reply(87 notify.1 [watch watch cookie 93968088725600] v32'220 uv219 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7fd820050040 con 0x5576a688a7e0 2023-10-05T19:29:33.001 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:32.998+0000 7fd82dd29700 1 -- 172.21.15.167:0/3169388355 <== osd.2 v2:172.21.15.167:6816/30037 15 ==== osd_op_reply(90 notify.3 [create] v32'219 uv219 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7fd818006070 con 0x7fd82801a3c0 2023-10-05T19:29:33.002 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:32.998+0000 7fd8309f5e40 1 -- 172.21.15.167:0/3169388355 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:91 7.e 7:7759931f:::notify.3:head [watch watch cookie 93968088727600] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e32) v8 -- 0x5576a68c9680 con 0x7fd82801a3c0 2023-10-05T19:29:33.002 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:32.998+0000 7fd8309f5e40 1 -- 172.21.15.167:0/3169388355 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:92 7.d 7:b4812045:::notify.4:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e32) v8 -- 0x5576a68cab20 con 0x5576a688a7e0 2023-10-05T19:29:33.002 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:32.998+0000 7fd82e52a700 1 -- 172.21.15.167:0/3169388355 <== osd.3 v2:172.21.15.167:6808/30036 23 ==== osd_op_reply(89 notify.2 [watch watch cookie 93968088723904] v32'220 uv219 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7fd82805f040 con 0x5576a68c1250 2023-10-05T19:29:33.003 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:32.998+0000 7fd82dd29700 1 -- 172.21.15.167:0/3169388355 <== osd.2 v2:172.21.15.167:6816/30037 16 ==== osd_op_reply(91 notify.3 [watch watch cookie 93968088727600] v32'220 uv219 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7fd818006070 con 0x7fd82801a3c0 2023-10-05T19:29:33.004 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:32.998+0000 7fd82ed2b700 1 -- 172.21.15.167:0/3169388355 <== osd.1 v2:172.21.15.167:6800/30035 31 ==== osd_op_reply(92 notify.4 [create] v32'219 uv219 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7fd820050040 con 0x5576a688a7e0 2023-10-05T19:29:33.004 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:32.998+0000 7fd8309f5e40 1 -- 172.21.15.167:0/3169388355 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:93 7.d 7:b4812045:::notify.4:head [watch watch cookie 93968088735312] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e32) v8 -- 0x5576a68ca5f0 con 0x5576a688a7e0 2023-10-05T19:29:33.004 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:32.998+0000 7fd8309f5e40 1 -- 172.21.15.167:0/3169388355 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:94 7.3 7:c609908c:::notify.5:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e32) v8 -- 0x5576a68cb410 con 0x7fd81800d3d0 2023-10-05T19:29:33.007 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.002+0000 7fd82ed2b700 1 -- 172.21.15.167:0/3169388355 <== osd.0 v2:172.21.15.118:6800/27690 23 ==== osd_op_reply(94 notify.5 [create] v32'219 uv219 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7fd820050040 con 0x7fd81800d3d0 2023-10-05T19:29:33.008 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.002+0000 7fd8309f5e40 1 -- 172.21.15.167:0/3169388355 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:95 7.3 7:c609908c:::notify.5:head [watch watch cookie 93968088739536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e32) v8 -- 0x5576a68cb810 con 0x7fd81800d3d0 2023-10-05T19:29:33.008 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.002+0000 7fd8309f5e40 1 -- 172.21.15.167:0/3169388355 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:96 7.14 7:2b04a3e9:::notify.6:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e32) v8 -- 0x5576a68cc260 con 0x7fd81800d3d0 2023-10-05T19:29:33.008 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.002+0000 7fd82ed2b700 1 -- 172.21.15.167:0/3169388355 <== osd.1 v2:172.21.15.167:6800/30035 32 ==== osd_op_reply(93 notify.4 [watch watch cookie 93968088735312] v32'220 uv219 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7fd820050040 con 0x5576a688a7e0 2023-10-05T19:29:33.010 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.006+0000 7fd82ed2b700 1 -- 172.21.15.167:0/3169388355 <== osd.0 v2:172.21.15.118:6800/27690 24 ==== osd_op_reply(96 notify.6 [create] v32'219 uv219 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7fd820050040 con 0x7fd81800d3d0 2023-10-05T19:29:33.010 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.006+0000 7fd8309f5e40 1 -- 172.21.15.167:0/3169388355 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:97 7.14 7:2b04a3e9:::notify.6:head [watch watch cookie 93968088730752] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e32) v8 -- 0x5576a68cc220 con 0x7fd81800d3d0 2023-10-05T19:29:33.010 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.006+0000 7fd8309f5e40 1 -- 172.21.15.167:0/3169388355 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:98 7.9 7:93e5b521:::notify.7:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e32) v8 -- 0x5576a68ccc30 con 0x7fd81800d3d0 2023-10-05T19:29:33.011 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.006+0000 7fd82ed2b700 1 -- 172.21.15.167:0/3169388355 <== osd.0 v2:172.21.15.118:6800/27690 25 ==== osd_op_reply(95 notify.5 [watch watch cookie 93968088739536] v32'220 uv219 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7fd820050040 con 0x7fd81800d3d0 2023-10-05T19:29:33.013 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.010+0000 7fd82ed2b700 1 -- 172.21.15.167:0/3169388355 <== osd.0 v2:172.21.15.118:6800/27690 26 ==== osd_op_reply(97 notify.6 [watch watch cookie 93968088730752] v32'220 uv219 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7fd820050040 con 0x7fd81800d3d0 2023-10-05T19:29:33.013 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.010+0000 7fd82ed2b700 1 -- 172.21.15.167:0/3169388355 <== osd.0 v2:172.21.15.118:6800/27690 27 ==== osd_op_reply(98 notify.7 [create] v32'219 uv219 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7fd820050040 con 0x7fd81800d3d0 2023-10-05T19:29:33.013 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.010+0000 7fd8309f5e40 1 -- 172.21.15.167:0/3169388355 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:99 7.9 7:93e5b521:::notify.7:head [watch watch cookie 93968088743216] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e32) v8 -- 0x5576a68cc180 con 0x7fd81800d3d0 2023-10-05T19:29:33.014 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.010+0000 7fd8309f5e40 20 add_watcher() i=0 2023-10-05T19:29:33.014 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.010+0000 7fd8309f5e40 20 add_watcher() i=1 2023-10-05T19:29:33.014 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.010+0000 7fd8309f5e40 20 add_watcher() i=2 2023-10-05T19:29:33.014 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.010+0000 7fd8309f5e40 20 add_watcher() i=3 2023-10-05T19:29:33.014 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.010+0000 7fd8309f5e40 20 add_watcher() i=4 2023-10-05T19:29:33.014 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.010+0000 7fd8309f5e40 20 add_watcher() i=5 2023-10-05T19:29:33.014 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.010+0000 7fd8309f5e40 20 add_watcher() i=6 2023-10-05T19:29:33.015 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.010+0000 7fd82ed2b700 1 -- 172.21.15.167:0/3169388355 <== osd.0 v2:172.21.15.118:6800/27690 28 ==== osd_op_reply(99 notify.7 [watch watch cookie 93968088743216] v32'220 uv219 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7fd820050040 con 0x7fd81800d3d0 2023-10-05T19:29:33.016 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.010+0000 7fd8309f5e40 20 add_watcher() i=7 2023-10-05T19:29:33.016 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.010+0000 7fd8309f5e40 2 all 8 watchers are set, enabling cache 2023-10-05T19:29:33.016 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.010+0000 7fd8309f5e40 1 -- 172.21.15.167:0/3169388355 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:100 6.4 6:22d26bf9:::data_loggenerations_metadata:head [call version.check_conds in=74b,call version.read in=11b,read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32) v8 -- 0x5576a68cb440 con 0x5576a688a7e0 2023-10-05T19:29:33.016 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.010+0000 7fd82ed2b700 1 -- 172.21.15.167:0/3169388355 <== osd.1 v2:172.21.15.167:6800/30035 33 ==== osd_op_reply(100 data_loggenerations_metadata [call,call out=48b,read 0~28 out=28b] v0'0 uv1 ondisk = 0) v8 ==== 256+0+76 (crc 0 0 0) 0x7fd820050040 con 0x5576a688a7e0 2023-10-05T19:29:33.017 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.010+0000 7fd8309f5e40 1 -- 172.21.15.167:0/3169388355 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:101 6.4 6:22d26bf9:::data_loggenerations_metadata:head [watch watch cookie 93968088745792] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e32) v8 -- 0x5576a68cd080 con 0x5576a688a7e0 2023-10-05T19:29:33.019 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.014+0000 7fd82ed2b700 1 -- 172.21.15.167:0/3169388355 <== osd.1 v2:172.21.15.167:6800/30035 34 ==== osd_op_reply(101 data_loggenerations_metadata [watch watch cookie 93968088745792] v32'219 uv1 ondisk = 0) v8 ==== 172+0+0 (crc 0 0 0) 0x7fd820050040 con 0x5576a688a7e0 2023-10-05T19:29:33.019 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.014+0000 7fd7fffdf700 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2023-10-05T19:29:33.019 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.014+0000 7fd8309f5e40 20 check_secure_mon_conn(): auth registy supported: methods=[2] modes=[2,1] 2023-10-05T19:29:33.019 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.014+0000 7fd8309f5e40 20 check_secure_mon_conn(): mode 1 is insecure 2023-10-05T19:29:33.020 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.014+0000 7fd8309f5e40 1 Processor -- start 2023-10-05T19:29:33.020 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.014+0000 7fd8309f5e40 1 -- start start 2023-10-05T19:29:33.020 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.014+0000 7fd8309f5e40 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5576a68d9cb0 0x5576a68da080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-05T19:29:33.020 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.014+0000 7fd8309f5e40 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x5576a68da5c0 con 0x5576a68d9cb0 2023-10-05T19:29:33.021 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.014+0000 7fd82dd29700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5576a68d9cb0 0x5576a68da080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-05T19:29:33.021 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.014+0000 7fd82dd29700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5576a68d9cb0 0x5576a68da080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.118:3300/0 says I am v2:172.21.15.167:46152/0 (socket says 172.21.15.167:46152) 2023-10-05T19:29:33.022 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.014+0000 7fd82dd29700 1 -- 172.21.15.167:0/1829063577 learned_addr learned my addr 172.21.15.167:0/1829063577 (peer_addr_for_me v2:172.21.15.167:0/0) 2023-10-05T19:29:33.022 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.018+0000 7fd82dd29700 1 -- 172.21.15.167:0/1829063577 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5576a68da700 con 0x5576a68d9cb0 2023-10-05T19:29:33.022 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.018+0000 7fd82dd29700 1 --2- 172.21.15.167:0/1829063577 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5576a68d9cb0 0x5576a68da080 secure :-1 s=READY pgs=576 cs=0 l=1 rev1=1 crypto rx=0x5576a68bed40 tx=0x7fd818006000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=555e166452a944b7 server_cookie=0 in_seq=0 out_seq=0 2023-10-05T19:29:33.023 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.018+0000 7fd7ff7de700 1 -- 172.21.15.167:0/1829063577 <== mon.0 v2:172.21.15.118:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fd818023040 con 0x5576a68d9cb0 2023-10-05T19:29:33.023 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.018+0000 7fd7ff7de700 1 -- 172.21.15.167:0/1829063577 <== mon.0 v2:172.21.15.118:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd818024040 con 0x5576a68d9cb0 2023-10-05T19:29:33.023 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.018+0000 7fd7ff7de700 1 -- 172.21.15.167:0/1829063577 <== mon.0 v2:172.21.15.118:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fd818013040 con 0x5576a68d9cb0 2023-10-05T19:29:33.023 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.018+0000 7fd8309f5e40 1 -- 172.21.15.167:0/1829063577 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5576a68d9cb0 msgr2=0x5576a68da080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-05T19:29:33.024 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.018+0000 7fd8309f5e40 1 --2- 172.21.15.167:0/1829063577 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5576a68d9cb0 0x5576a68da080 secure :-1 s=READY pgs=576 cs=0 l=1 rev1=1 crypto rx=0x5576a68bed40 tx=0x7fd818006000 comp rx=0 tx=0).stop 2023-10-05T19:29:33.024 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.018+0000 7fd8309f5e40 1 -- 172.21.15.167:0/1829063577 shutdown_connections 2023-10-05T19:29:33.024 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.018+0000 7fd8309f5e40 1 --2- 172.21.15.167:0/1829063577 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5576a68d9cb0 0x5576a68da080 unknown :-1 s=CLOSED pgs=576 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-05T19:29:33.024 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.018+0000 7fd8309f5e40 1 -- 172.21.15.167:0/1829063577 >> 172.21.15.167:0/1829063577 conn(0x5576a68d5070 msgr2=0x5576a68d3220 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-05T19:29:33.024 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.018+0000 7fd8309f5e40 1 -- 172.21.15.167:0/1829063577 shutdown_connections 2023-10-05T19:29:33.024 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.018+0000 7fd8309f5e40 1 -- 172.21.15.167:0/1829063577 wait complete. 2023-10-05T19:29:33.025 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.018+0000 7fd8309f5e40 1 Processor -- start 2023-10-05T19:29:33.026 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.018+0000 7fd8309f5e40 1 -- start start 2023-10-05T19:29:33.026 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.018+0000 7fd8309f5e40 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5576a68d9cb0 0x5576a68df7b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-05T19:29:33.026 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.018+0000 7fd8309f5e40 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x5576a68e07b0 con 0x5576a68d9cb0 2023-10-05T19:29:33.026 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.022+0000 7fd82dd29700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5576a68d9cb0 0x5576a68df7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-05T19:29:33.026 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.022+0000 7fd82dd29700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5576a68d9cb0 0x5576a68df7b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.118:3300/0 says I am v2:172.21.15.167:46154/0 (socket says 172.21.15.167:46154) 2023-10-05T19:29:33.026 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.022+0000 7fd82dd29700 1 -- 172.21.15.167:0/3942372936 learned_addr learned my addr 172.21.15.167:0/3942372936 (peer_addr_for_me v2:172.21.15.167:0/0) 2023-10-05T19:29:33.026 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.022+0000 7fd82dd29700 1 -- 172.21.15.167:0/3942372936 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5576a67d6bd0 con 0x5576a68d9cb0 2023-10-05T19:29:33.027 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.022+0000 7fd82dd29700 1 --2- 172.21.15.167:0/3942372936 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5576a68d9cb0 0x5576a68df7b0 secure :-1 s=READY pgs=577 cs=0 l=1 rev1=1 crypto rx=0x5576a68d3d90 tx=0x7fd81802f000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-05T19:29:33.027 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.022+0000 7fd7fdfdb700 1 -- 172.21.15.167:0/3942372936 <== mon.0 v2:172.21.15.118:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fd818023040 con 0x5576a68d9cb0 2023-10-05T19:29:33.027 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.022+0000 7fd7fdfdb700 1 -- 172.21.15.167:0/3942372936 <== mon.0 v2:172.21.15.118:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd81802c410 con 0x5576a68d9cb0 2023-10-05T19:29:33.028 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.022+0000 7fd8309f5e40 1 -- 172.21.15.167:0/3942372936 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x5576a68dfdb0 con 0x5576a68d9cb0 2023-10-05T19:29:33.028 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.022+0000 7fd7fdfdb700 1 -- 172.21.15.167:0/3942372936 <== mon.0 v2:172.21.15.118:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fd81802c5a0 con 0x5576a68d9cb0 2023-10-05T19:29:33.028 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.022+0000 7fd8309f5e40 1 -- 172.21.15.167:0/3942372936 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x5576a68e0190 con 0x5576a68d9cb0 2023-10-05T19:29:33.029 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.022+0000 7fd7fdfdb700 1 -- 172.21.15.167:0/3942372936 <== mon.0 v2:172.21.15.118:3300/0 4 ==== mgrmap(e 5) v1 ==== 40329+0+0 (secure 0 0 0) 0x7fd81802c700 con 0x5576a68d9cb0 2023-10-05T19:29:33.029 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.026+0000 7fd7fdfdb700 1 --2- 172.21.15.167:0/3942372936 >> [v2:172.21.15.167:6824/29889,v1:172.21.15.167:6825/29889] conn(0x7fd774030c50 0x7fd774033100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-05T19:29:33.029 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.026+0000 7fd7fdfdb700 1 -- 172.21.15.167:0/3942372936 <== mon.0 v2:172.21.15.118:3300/0 5 ==== osd_map(32..32 src has 1..32) v4 ==== 6260+0+0 (secure 0 0 0) 0x7fd81803bd10 con 0x5576a68d9cb0 2023-10-05T19:29:33.029 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.026+0000 7fd82ed2b700 1 --2- 172.21.15.167:0/3942372936 >> [v2:172.21.15.167:6824/29889,v1:172.21.15.167:6825/29889] conn(0x7fd774030c50 0x7fd774033100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-05T19:29:33.029 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.026+0000 7fd8309f5e40 5 note: GC not initialized 2023-10-05T19:29:33.030 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.026+0000 7fd82ed2b700 1 --2- 172.21.15.167:0/3942372936 >> [v2:172.21.15.167:6824/29889,v1:172.21.15.167:6825/29889] conn(0x7fd774030c50 0x7fd774033100 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x5576a68cb470 tx=0x7fd820000f90 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-05T19:29:33.032 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.026+0000 7fd76effd700 20 reqs_thread_entry: start 2023-10-05T19:29:33.032 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.030+0000 7fd76cff9700 20 reqs_thread_entry: start 2023-10-05T19:29:33.033 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.030+0000 7fd8309f5e40 20 init_complete bucket index max shards: 11 2023-10-05T19:29:33.034 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.030+0000 7fd8309f5e40 10 rgw notify: Started notification manager with: 1 workers 2023-10-05T19:29:33.034 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.030+0000 7fd75effd700 20 rgw notify: INFO: next queues processing will happen at: Thu Oct 5 19:30:03 2023 2023-10-05T19:29:33.034 INFO:teuthology.orchestra.run.smithi167.stderr: 2023-10-05T19:29:33.034 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.030+0000 7fd75e7fc700 20 reqs_thread_entry: start 2023-10-05T19:29:33.034 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.030+0000 7fd8309f5e40 1 -- 172.21.15.167:0/3169388355 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:102 6.e 6:74cd889a:usage::usage.0:head [call rgw.user_usage_log_read in=60b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32) v8 -- 0x5576a69488d0 con 0x5576a688a7e0 2023-10-05T19:29:33.035 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.030+0000 7fd82ed2b700 1 -- 172.21.15.167:0/3169388355 <== osd.1 v2:172.21.15.167:6800/30035 35 ==== osd_op_reply(102 usage.0 [call] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 ==== 151+0+0 (crc 0 0 0) 0x7fd820050040 con 0x5576a688a7e0 2023-10-05T19:29:33.035 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.030+0000 7fd8309f5e40 1 -- 172.21.15.167:0/3169388355 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:103 6.3 6:c0e78a64:usage::usage.1:head [call rgw.user_usage_log_read in=60b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32) v8 -- 0x5576a69488d0 con 0x7fd82801a3c0 2023-10-05T19:29:33.035 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.030+0000 7fd82dd29700 1 -- 172.21.15.167:0/3169388355 <== osd.2 v2:172.21.15.167:6816/30037 17 ==== osd_op_reply(103 usage.1 [call] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 ==== 151+0+0 (crc 0 0 0) 0x7fd81804a070 con 0x7fd82801a3c0 2023-10-05T19:29:33.035 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.030+0000 7fd8309f5e40 1 -- 172.21.15.167:0/3169388355 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:104 6.d 6:b564efa4:usage::usage.2:head [call rgw.user_usage_log_read in=60b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32) v8 -- 0x5576a6948c80 con 0x5576a688a7e0 2023-10-05T19:29:33.036 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.030+0000 7fd82ed2b700 1 -- 172.21.15.167:0/3169388355 <== osd.1 v2:172.21.15.167:6800/30035 36 ==== osd_op_reply(104 usage.2 [call] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 ==== 151+0+0 (crc 0 0 0) 0x7fd820050040 con 0x5576a688a7e0 2023-10-05T19:29:33.036 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.030+0000 7fd8309f5e40 1 -- 172.21.15.167:0/3169388355 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:105 6.2 6:42d64401:usage::usage.3:head [call rgw.user_usage_log_read in=60b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32) v8 -- 0x5576a6948c80 con 0x5576a68c1250 2023-10-05T19:29:33.037 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.030+0000 7fd82e52a700 1 -- 172.21.15.167:0/3169388355 <== osd.3 v2:172.21.15.167:6808/30036 24 ==== osd_op_reply(105 usage.3 [call] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 ==== 151+0+0 (crc 0 0 0) 0x7fd82805f040 con 0x5576a68c1250 2023-10-05T19:29:33.037 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.030+0000 7fd8309f5e40 1 -- 172.21.15.167:0/3169388355 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:106 6.7 6:e09b6f01:usage::usage.4:head [call rgw.user_usage_log_read in=60b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32) v8 -- 0x5576a6949030 con 0x5576a68c1250 2023-10-05T19:29:33.037 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.034+0000 7fd82e52a700 1 -- 172.21.15.167:0/3169388355 <== osd.3 v2:172.21.15.167:6808/30036 25 ==== osd_op_reply(106 usage.4 [call] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 ==== 151+0+0 (crc 0 0 0) 0x7fd82805f040 con 0x5576a68c1250 2023-10-05T19:29:33.037 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.034+0000 7fd8309f5e40 1 -- 172.21.15.167:0/3169388355 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:107 6.6 6:672810f1:usage::usage.5:head [call rgw.user_usage_log_read in=60b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32) v8 -- 0x5576a6949030 con 0x5576a68c1250 2023-10-05T19:29:33.038 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.034+0000 7fd82e52a700 1 -- 172.21.15.167:0/3169388355 <== osd.3 v2:172.21.15.167:6808/30036 26 ==== osd_op_reply(107 usage.5 [call] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 ==== 151+0+0 (crc 0 0 0) 0x7fd82805f040 con 0x5576a68c1250 2023-10-05T19:29:33.038 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.034+0000 7fd8309f5e40 1 -- 172.21.15.167:0/3169388355 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:108 6.4 6:25a86931:usage::usage.6:head [call rgw.user_usage_log_read in=60b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32) v8 -- 0x5576a6949030 con 0x5576a688a7e0 2023-10-05T19:29:33.038 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.034+0000 7fd82ed2b700 1 -- 172.21.15.167:0/3169388355 <== osd.1 v2:172.21.15.167:6800/30035 37 ==== osd_op_reply(108 usage.6 [call] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 ==== 151+0+0 (crc 0 0 0) 0x7fd820050040 con 0x5576a688a7e0 2023-10-05T19:29:33.038 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.034+0000 7fd8309f5e40 1 -- 172.21.15.167:0/3169388355 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:109 6.17 6:ebbe7322:usage::usage.7:head [call rgw.user_usage_log_read in=60b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32) v8 -- 0x5576a6949030 con 0x5576a688a7e0 2023-10-05T19:29:33.039 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.034+0000 7fd82ed2b700 1 -- 172.21.15.167:0/3169388355 <== osd.1 v2:172.21.15.167:6800/30035 38 ==== osd_op_reply(109 usage.7 [call] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 ==== 151+0+0 (crc 0 0 0) 0x7fd820050040 con 0x5576a688a7e0 2023-10-05T19:29:33.039 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.034+0000 7fd8309f5e40 1 -- 172.21.15.167:0/3169388355 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:110 6.6 6:607e1922:usage::usage.8:head [call rgw.user_usage_log_read in=60b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32) v8 -- 0x5576a6949030 con 0x5576a68c1250 2023-10-05T19:29:33.039 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.034+0000 7fd82e52a700 1 -- 172.21.15.167:0/3169388355 <== osd.3 v2:172.21.15.167:6808/30036 27 ==== osd_op_reply(110 usage.8 [call] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 ==== 151+0+0 (crc 0 0 0) 0x7fd82805f040 con 0x5576a68c1250 2023-10-05T19:29:33.040 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.034+0000 7fd8309f5e40 1 -- 172.21.15.167:0/3169388355 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:111 6.b 6:d7809252:usage::usage.9:head [call rgw.user_usage_log_read in=60b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32) v8 -- 0x5576a6949030 con 0x5576a68c1250 2023-10-05T19:29:33.040 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.034+0000 7fd82e52a700 1 -- 172.21.15.167:0/3169388355 <== osd.3 v2:172.21.15.167:6808/30036 28 ==== osd_op_reply(111 usage.9 [call] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 ==== 151+0+0 (crc 0 0 0) 0x7fd82805f040 con 0x5576a68c1250 2023-10-05T19:29:33.040 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.034+0000 7fd8309f5e40 1 -- 172.21.15.167:0/3169388355 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:112 6.13 6:cea599e4:usage::usage.10:head [call rgw.user_usage_log_read in=60b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32) v8 -- 0x5576a6949030 con 0x7fd81800d3d0 2023-10-05T19:29:33.041 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.038+0000 7fd82ed2b700 1 -- 172.21.15.167:0/3169388355 <== osd.0 v2:172.21.15.118:6800/27690 29 ==== osd_op_reply(112 usage.10 [call out=1268b] v0'0 uv30 ondisk = 0) v8 ==== 152+0+1268 (crc 0 0 0) 0x7fd820050040 con 0x7fd81800d3d0 2023-10-05T19:29:33.041 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.038+0000 7fd8309f5e40 1 -- 172.21.15.167:0/3169388355 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:113 6.4 6:21f845ae:usage::usage.11:head [call rgw.user_usage_log_read in=60b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32) v8 -- 0x5576a694a530 con 0x5576a688a7e0 2023-10-05T19:29:33.042 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.038+0000 7fd82ed2b700 1 -- 172.21.15.167:0/3169388355 <== osd.1 v2:172.21.15.167:6800/30035 39 ==== osd_op_reply(113 usage.11 [call] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 ==== 152+0+0 (crc 0 0 0) 0x7fd820050040 con 0x5576a688a7e0 2023-10-05T19:29:33.042 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.038+0000 7fd8309f5e40 1 -- 172.21.15.167:0/3169388355 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:114 6.f 6:f0386354:usage::usage.12:head [call rgw.user_usage_log_read in=60b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32) v8 -- 0x5576a694a530 con 0x7fd82801a3c0 2023-10-05T19:29:33.043 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.038+0000 7fd82dd29700 1 -- 172.21.15.167:0/3169388355 <== osd.2 v2:172.21.15.167:6816/30037 18 ==== osd_op_reply(114 usage.12 [call] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 ==== 152+0+0 (crc 0 0 0) 0x7fd81804a070 con 0x7fd82801a3c0 2023-10-05T19:29:33.043 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.038+0000 7fd8309f5e40 1 -- 172.21.15.167:0/3169388355 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:115 6.c 6:3735eec4:usage::usage.13:head [call rgw.user_usage_log_read in=60b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32) v8 -- 0x5576a694a530 con 0x5576a68c1250 2023-10-05T19:29:33.043 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.038+0000 7fd82e52a700 1 -- 172.21.15.167:0/3169388355 <== osd.3 v2:172.21.15.167:6808/30036 29 ==== osd_op_reply(115 usage.13 [call] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 ==== 152+0+0 (crc 0 0 0) 0x7fd82805f040 con 0x5576a68c1250 2023-10-05T19:29:33.043 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.038+0000 7fd8309f5e40 1 -- 172.21.15.167:0/3169388355 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:116 6.18 6:1f520d0a:usage::usage.14:head [call rgw.user_usage_log_read in=60b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32) v8 -- 0x5576a694a530 con 0x7fd81800d3d0 2023-10-05T19:29:33.044 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.038+0000 7fd82ed2b700 1 -- 172.21.15.167:0/3169388355 <== osd.0 v2:172.21.15.118:6800/27690 30 ==== osd_op_reply(116 usage.14 [call] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 ==== 152+0+0 (crc 0 0 0) 0x7fd820050040 con 0x7fd81800d3d0 2023-10-05T19:29:33.044 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.038+0000 7fd8309f5e40 1 -- 172.21.15.167:0/3169388355 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:117 6.0 6:029cee12:usage::usage.15:head [call rgw.user_usage_log_read in=60b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32) v8 -- 0x5576a694a530 con 0x7fd81800d3d0 2023-10-05T19:29:33.045 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.042+0000 7fd82ed2b700 1 -- 172.21.15.167:0/3169388355 <== osd.0 v2:172.21.15.118:6800/27690 31 ==== osd_op_reply(117 usage.15 [call] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 ==== 152+0+0 (crc 0 0 0) 0x7fd820050040 con 0x7fd81800d3d0 2023-10-05T19:29:33.045 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.042+0000 7fd8309f5e40 1 -- 172.21.15.167:0/3169388355 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:118 6.1e 6:7f3e030e:usage::usage.16:head [call rgw.user_usage_log_read in=60b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32) v8 -- 0x5576a694a530 con 0x5576a68c1250 2023-10-05T19:29:33.045 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.042+0000 7fd82e52a700 1 -- 172.21.15.167:0/3169388355 <== osd.3 v2:172.21.15.167:6808/30036 30 ==== osd_op_reply(118 usage.16 [call] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 ==== 152+0+0 (crc 0 0 0) 0x7fd82805f040 con 0x5576a68c1250 2023-10-05T19:29:33.045 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.042+0000 7fd8309f5e40 1 -- 172.21.15.167:0/3169388355 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:119 6.10 6:0fbdcb32:usage::usage.17:head [call rgw.user_usage_log_read in=60b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32) v8 -- 0x5576a694a530 con 0x7fd81800d3d0 2023-10-05T19:29:33.046 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.042+0000 7fd82ed2b700 1 -- 172.21.15.167:0/3169388355 <== osd.0 v2:172.21.15.118:6800/27690 32 ==== osd_op_reply(119 usage.17 [call out=163b] v0'0 uv5 ondisk = 0) v8 ==== 152+0+163 (crc 0 0 0) 0x7fd820050040 con 0x7fd81800d3d0 2023-10-05T19:29:33.047 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.042+0000 7fd8309f5e40 1 -- 172.21.15.167:0/3169388355 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:120 6.19 6:993b03e9:usage::usage.18:head [call rgw.user_usage_log_read in=60b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32) v8 -- 0x5576a694a530 con 0x5576a688a7e0 2023-10-05T19:29:33.047 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.042+0000 7fd82ed2b700 1 -- 172.21.15.167:0/3169388355 <== osd.1 v2:172.21.15.167:6800/30035 40 ==== osd_op_reply(120 usage.18 [call] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 ==== 152+0+0 (crc 0 0 0) 0x7fd820050040 con 0x5576a688a7e0 2023-10-05T19:29:33.047 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.042+0000 7fd8309f5e40 1 -- 172.21.15.167:0/3169388355 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:121 6.1a 6:58fcd270:usage::usage.19:head [call rgw.user_usage_log_read in=60b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32) v8 -- 0x5576a694a530 con 0x5576a68c1250 2023-10-05T19:29:33.048 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.042+0000 7fd82e52a700 1 -- 172.21.15.167:0/3169388355 <== osd.3 v2:172.21.15.167:6808/30036 31 ==== osd_op_reply(121 usage.19 [call] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 ==== 152+0+0 (crc 0 0 0) 0x7fd82805f040 con 0x5576a68c1250 2023-10-05T19:29:33.048 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.042+0000 7fd8309f5e40 1 -- 172.21.15.167:0/3169388355 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:122 6.1f 6:fcd63d49:usage::usage.20:head [call rgw.user_usage_log_read in=60b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32) v8 -- 0x5576a694a530 con 0x7fd81800d3d0 2023-10-05T19:29:33.048 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.042+0000 7fd82ed2b700 1 -- 172.21.15.167:0/3169388355 <== osd.0 v2:172.21.15.118:6800/27690 33 ==== osd_op_reply(122 usage.20 [call] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 ==== 152+0+0 (crc 0 0 0) 0x7fd820050040 con 0x7fd81800d3d0 2023-10-05T19:29:33.048 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.042+0000 7fd8309f5e40 1 -- 172.21.15.167:0/3169388355 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:123 6.5 6:a2c1a412:usage::usage.21:head [call rgw.user_usage_log_read in=60b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32) v8 -- 0x5576a694a530 con 0x7fd82801a3c0 2023-10-05T19:29:33.049 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.046+0000 7fd82dd29700 1 -- 172.21.15.167:0/3169388355 <== osd.2 v2:172.21.15.167:6816/30037 19 ==== osd_op_reply(123 usage.21 [call] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 ==== 152+0+0 (crc 0 0 0) 0x7fd81804a070 con 0x7fd82801a3c0 2023-10-05T19:29:33.049 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.046+0000 7fd8309f5e40 1 -- 172.21.15.167:0/3169388355 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:124 6.13 6:cdb71d68:usage::usage.22:head [call rgw.user_usage_log_read in=60b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32) v8 -- 0x5576a694a530 con 0x7fd81800d3d0 2023-10-05T19:29:33.050 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.046+0000 7fd82ed2b700 1 -- 172.21.15.167:0/3169388355 <== osd.0 v2:172.21.15.118:6800/27690 34 ==== osd_op_reply(124 usage.22 [call] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 ==== 152+0+0 (crc 0 0 0) 0x7fd820050040 con 0x7fd81800d3d0 2023-10-05T19:29:33.050 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.046+0000 7fd8309f5e40 1 -- 172.21.15.167:0/3169388355 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:125 6.17 6:ef3bca03:usage::usage.23:head [call rgw.user_usage_log_read in=60b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32) v8 -- 0x5576a694a530 con 0x5576a688a7e0 2023-10-05T19:29:33.050 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.046+0000 7fd82ed2b700 1 -- 172.21.15.167:0/3169388355 <== osd.1 v2:172.21.15.167:6800/30035 41 ==== osd_op_reply(125 usage.23 [call] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 ==== 152+0+0 (crc 0 0 0) 0x7fd820050040 con 0x5576a688a7e0 2023-10-05T19:29:33.051 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.046+0000 7fd8309f5e40 1 -- 172.21.15.167:0/3169388355 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:126 6.e 6:7412960a:usage::usage.24:head [call rgw.user_usage_log_read in=60b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32) v8 -- 0x5576a694a530 con 0x5576a688a7e0 2023-10-05T19:29:33.051 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.046+0000 7fd82ed2b700 1 -- 172.21.15.167:0/3169388355 <== osd.1 v2:172.21.15.167:6800/30035 42 ==== osd_op_reply(126 usage.24 [call] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 ==== 152+0+0 (crc 0 0 0) 0x7fd820050040 con 0x5576a688a7e0 2023-10-05T19:29:33.052 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.046+0000 7fd8309f5e40 1 -- 172.21.15.167:0/3169388355 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:127 6.7 6:e62d1024:usage::usage.25:head [call rgw.user_usage_log_read in=60b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32) v8 -- 0x5576a694a530 con 0x5576a68c1250 2023-10-05T19:29:33.052 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.046+0000 7fd82e52a700 1 -- 172.21.15.167:0/3169388355 <== osd.3 v2:172.21.15.167:6808/30036 32 ==== osd_op_reply(127 usage.25 [call] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 ==== 152+0+0 (crc 0 0 0) 0x7fd82805f040 con 0x5576a68c1250 2023-10-05T19:29:33.052 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.046+0000 7fd8309f5e40 1 -- 172.21.15.167:0/3169388355 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:128 6.d 6:b172cd59:usage::usage.26:head [call rgw.user_usage_log_read in=60b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32) v8 -- 0x5576a694a530 con 0x5576a688a7e0 2023-10-05T19:29:33.053 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.046+0000 7fd82ed2b700 1 -- 172.21.15.167:0/3169388355 <== osd.1 v2:172.21.15.167:6800/30035 43 ==== osd_op_reply(128 usage.26 [call] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 ==== 152+0+0 (crc 0 0 0) 0x7fd820050040 con 0x5576a688a7e0 2023-10-05T19:29:33.053 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.046+0000 7fd8309f5e40 1 -- 172.21.15.167:0/3169388355 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:129 6.1 6:83912023:usage::usage.27:head [call rgw.user_usage_log_read in=60b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32) v8 -- 0x5576a694a530 con 0x5576a688a7e0 2023-10-05T19:29:33.053 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.050+0000 7fd82ed2b700 1 -- 172.21.15.167:0/3169388355 <== osd.1 v2:172.21.15.167:6800/30035 44 ==== osd_op_reply(129 usage.27 [call] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 ==== 152+0+0 (crc 0 0 0) 0x7fd820050040 con 0x5576a688a7e0 2023-10-05T19:29:33.053 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.050+0000 7fd8309f5e40 1 -- 172.21.15.167:0/3169388355 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:130 6.1e 6:7c73bdb3:usage::usage.28:head [call rgw.user_usage_log_read in=60b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32) v8 -- 0x5576a694a530 con 0x5576a68c1250 2023-10-05T19:29:33.054 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.050+0000 7fd82e52a700 1 -- 172.21.15.167:0/3169388355 <== osd.3 v2:172.21.15.167:6808/30036 33 ==== osd_op_reply(130 usage.28 [call] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 ==== 152+0+0 (crc 0 0 0) 0x7fd82805f040 con 0x5576a68c1250 2023-10-05T19:29:33.054 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.050+0000 7fd8309f5e40 1 -- 172.21.15.167:0/3169388355 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:131 6.16 6:6e59fa35:usage::usage.29:head [call rgw.user_usage_log_read in=60b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32) v8 -- 0x5576a694a530 con 0x5576a68c1250 2023-10-05T19:29:33.054 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.050+0000 7fd82e52a700 1 -- 172.21.15.167:0/3169388355 <== osd.3 v2:172.21.15.167:6808/30036 34 ==== osd_op_reply(131 usage.29 [call] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 ==== 152+0+0 (crc 0 0 0) 0x7fd82805f040 con 0x5576a68c1250 2023-10-05T19:29:33.054 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.050+0000 7fd8309f5e40 1 -- 172.21.15.167:0/3169388355 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:132 6.1c 6:38fe6885:usage::usage.30:head [call rgw.user_usage_log_read in=60b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32) v8 -- 0x5576a694a530 con 0x5576a68c1250 2023-10-05T19:29:33.055 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.050+0000 7fd82e52a700 1 -- 172.21.15.167:0/3169388355 <== osd.3 v2:172.21.15.167:6808/30036 35 ==== osd_op_reply(132 usage.30 [call] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 ==== 152+0+0 (crc 0 0 0) 0x7fd82805f040 con 0x5576a68c1250 2023-10-05T19:29:33.055 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.050+0000 7fd8309f5e40 1 -- 172.21.15.167:0/3169388355 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:133 6.1d 6:bb28dff0:usage::usage.31:head [call rgw.user_usage_log_read in=60b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32) v8 -- 0x5576a694a530 con 0x7fd81800d3d0 2023-10-05T19:29:33.055 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.050+0000 7fd82ed2b700 1 -- 172.21.15.167:0/3169388355 <== osd.0 v2:172.21.15.118:6800/27690 35 ==== osd_op_reply(133 usage.31 [call] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 ==== 152+0+0 (crc 0 0 0) 0x7fd820050040 con 0x7fd81800d3d0 2023-10-05T19:29:33.056 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.050+0000 7fd8309f5e40 1 -- 172.21.15.167:0/3169388355 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:134 7.1f 7:f95f44c2:::notify.0:head [watch unwatch cookie 93968088720672] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e32) v8 -- 0x5576a69063f0 con 0x7fd82801a3c0 2023-10-05T19:29:33.059 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.054+0000 7fd82dd29700 1 -- 172.21.15.167:0/3169388355 <== osd.2 v2:172.21.15.167:6816/30037 20 ==== osd_op_reply(134 notify.0 [watch unwatch cookie 93968088720672] v32'221 uv219 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7fd818053040 con 0x7fd82801a3c0 2023-10-05T19:29:33.059 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.054+0000 7fd8309f5e40 20 remove_watcher() i=0 2023-10-05T19:29:33.059 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.054+0000 7fd8309f5e40 2 removed watcher, disabling cache 2023-10-05T19:29:33.059 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.054+0000 7fd8309f5e40 1 -- 172.21.15.167:0/3169388355 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:135 7.0 7:05bf5b68:::notify.1:head [watch unwatch cookie 93968088725600] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e32) v8 -- 0x5576a68c6520 con 0x5576a688a7e0 2023-10-05T19:29:33.062 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.058+0000 7fd82ed2b700 1 -- 172.21.15.167:0/3169388355 <== osd.1 v2:172.21.15.167:6800/30035 45 ==== osd_op_reply(135 notify.1 [watch unwatch cookie 93968088725600] v32'221 uv219 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7fd820050040 con 0x5576a688a7e0 2023-10-05T19:29:33.062 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.058+0000 7fd8309f5e40 20 remove_watcher() i=1 2023-10-05T19:29:33.062 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.058+0000 7fd8309f5e40 1 -- 172.21.15.167:0/3169388355 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:136 7.15 7:a93a5511:::notify.2:head [watch unwatch cookie 93968088723904] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e32) v8 -- 0x5576a68c6520 con 0x5576a68c1250 2023-10-05T19:29:33.065 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.062+0000 7fd82e52a700 1 -- 172.21.15.167:0/3169388355 <== osd.3 v2:172.21.15.167:6808/30036 36 ==== osd_op_reply(136 notify.2 [watch unwatch cookie 93968088723904] v32'221 uv219 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7fd82805f040 con 0x5576a68c1250 2023-10-05T19:29:33.065 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.062+0000 7fd8309f5e40 20 remove_watcher() i=2 2023-10-05T19:29:33.065 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.062+0000 7fd8309f5e40 1 -- 172.21.15.167:0/3169388355 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:137 7.e 7:7759931f:::notify.3:head [watch unwatch cookie 93968088727600] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e32) v8 -- 0x5576a68c71c0 con 0x7fd82801a3c0 2023-10-05T19:29:33.067 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.062+0000 7fd82dd29700 1 -- 172.21.15.167:0/3169388355 <== osd.2 v2:172.21.15.167:6816/30037 21 ==== osd_op_reply(137 notify.3 [watch unwatch cookie 93968088727600] v32'221 uv219 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7fd818053040 con 0x7fd82801a3c0 2023-10-05T19:29:33.067 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.062+0000 7fd8309f5e40 20 remove_watcher() i=3 2023-10-05T19:29:33.068 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.062+0000 7fd8309f5e40 1 -- 172.21.15.167:0/3169388355 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:138 7.d 7:b4812045:::notify.4:head [watch unwatch cookie 93968088735312] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e32) v8 -- 0x5576a68c6520 con 0x5576a688a7e0 2023-10-05T19:29:33.070 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.066+0000 7fd82ed2b700 1 -- 172.21.15.167:0/3169388355 <== osd.1 v2:172.21.15.167:6800/30035 46 ==== osd_op_reply(138 notify.4 [watch unwatch cookie 93968088735312] v32'221 uv219 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7fd820050040 con 0x5576a688a7e0 2023-10-05T19:29:33.071 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.066+0000 7fd8309f5e40 20 remove_watcher() i=4 2023-10-05T19:29:33.071 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.066+0000 7fd8309f5e40 1 -- 172.21.15.167:0/3169388355 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:139 7.3 7:c609908c:::notify.5:head [watch unwatch cookie 93968088739536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e32) v8 -- 0x5576a68c9e50 con 0x7fd81800d3d0 2023-10-05T19:29:33.073 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.070+0000 7fd82ed2b700 1 -- 172.21.15.167:0/3169388355 <== osd.0 v2:172.21.15.118:6800/27690 36 ==== osd_op_reply(139 notify.5 [watch unwatch cookie 93968088739536] v32'221 uv219 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7fd820050040 con 0x7fd81800d3d0 2023-10-05T19:29:33.074 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.070+0000 7fd8309f5e40 20 remove_watcher() i=5 2023-10-05T19:29:33.074 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.070+0000 7fd8309f5e40 1 -- 172.21.15.167:0/3169388355 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:140 7.14 7:2b04a3e9:::notify.6:head [watch unwatch cookie 93968088730752] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e32) v8 -- 0x5576a68c6520 con 0x7fd81800d3d0 2023-10-05T19:29:33.077 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.074+0000 7fd82ed2b700 1 -- 172.21.15.167:0/3169388355 <== osd.0 v2:172.21.15.118:6800/27690 37 ==== osd_op_reply(140 notify.6 [watch unwatch cookie 93968088730752] v32'221 uv219 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7fd820050040 con 0x7fd81800d3d0 2023-10-05T19:29:33.077 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.074+0000 7fd8309f5e40 20 remove_watcher() i=6 2023-10-05T19:29:33.077 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.074+0000 7fd8309f5e40 1 -- 172.21.15.167:0/3169388355 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:141 7.9 7:93e5b521:::notify.7:head [watch unwatch cookie 93968088743216] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e32) v8 -- 0x5576a68caed0 con 0x7fd81800d3d0 2023-10-05T19:29:33.080 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.074+0000 7fd82ed2b700 1 -- 172.21.15.167:0/3169388355 <== osd.0 v2:172.21.15.118:6800/27690 38 ==== osd_op_reply(141 notify.7 [watch unwatch cookie 93968088743216] v32'221 uv219 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7fd820050040 con 0x7fd81800d3d0 2023-10-05T19:29:33.080 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.074+0000 7fd8309f5e40 20 remove_watcher() i=7 2023-10-05T19:29:33.082 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.078+0000 7fd7fffdf700 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2023-10-05T19:29:33.082 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.078+0000 7fd8309f5e40 1 -- 172.21.15.167:0/3169388355 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:142 6.4 6:22d26bf9:::data_loggenerations_metadata:head [watch unwatch cookie 93968088745792] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e32) v8 -- 0x5576a68c6520 con 0x5576a688a7e0 2023-10-05T19:29:33.085 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.078+0000 7fd82ed2b700 1 -- 172.21.15.167:0/3169388355 <== osd.1 v2:172.21.15.167:6800/30035 47 ==== osd_op_reply(142 data_loggenerations_metadata [watch unwatch cookie 93968088745792] v32'220 uv1 ondisk = 0) v8 ==== 172+0+0 (crc 0 0 0) 0x7fd820050040 con 0x5576a688a7e0 2023-10-05T19:29:33.085 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.078+0000 7fd8309f5e40 1 -- 172.21.15.167:0/3169388355 >> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] conn(0x7fd81800d3d0 msgr2=0x7fd81800d820 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-05T19:29:33.085 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.078+0000 7fd8309f5e40 1 --2- 172.21.15.167:0/3169388355 >> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] conn(0x7fd81800d3d0 0x7fd81800d820 crc :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-05T19:29:33.085 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.078+0000 7fd8309f5e40 1 -- 172.21.15.167:0/3169388355 >> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] conn(0x5576a688a7e0 msgr2=0x5576a68bd800 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-05T19:29:33.086 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.078+0000 7fd8309f5e40 1 --2- 172.21.15.167:0/3169388355 >> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] conn(0x5576a688a7e0 0x5576a68bd800 crc :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-05T19:29:33.086 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.078+0000 7fd8309f5e40 1 -- 172.21.15.167:0/3169388355 >> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] conn(0x7fd82801a3c0 msgr2=0x7fd828039540 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-05T19:29:33.086 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.078+0000 7fd8309f5e40 1 --2- 172.21.15.167:0/3169388355 >> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] conn(0x7fd82801a3c0 0x7fd828039540 crc :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-05T19:29:33.086 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.078+0000 7fd8309f5e40 1 -- 172.21.15.167:0/3169388355 >> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] conn(0x5576a68c1250 msgr2=0x5576a68c3630 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-05T19:29:33.086 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.078+0000 7fd8309f5e40 1 --2- 172.21.15.167:0/3169388355 >> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] conn(0x5576a68c1250 0x5576a68c3630 crc :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-05T19:29:33.086 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.082+0000 7fd8309f5e40 1 -- 172.21.15.167:0/3169388355 >> [v2:172.21.15.167:6824/29889,v1:172.21.15.167:6825/29889] conn(0x7fd810030c00 msgr2=0x7fd8100330b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-05T19:29:33.086 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.082+0000 7fd8309f5e40 1 --2- 172.21.15.167:0/3169388355 >> [v2:172.21.15.167:6824/29889,v1:172.21.15.167:6825/29889] conn(0x7fd810030c00 0x7fd8100330b0 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7fd818004170 tx=0x7fd818001380 comp rx=0 tx=0).stop 2023-10-05T19:29:33.086 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.082+0000 7fd8309f5e40 1 -- 172.21.15.167:0/3169388355 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5576a688a410 msgr2=0x5576a68b1da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-05T19:29:33.086 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.082+0000 7fd8309f5e40 1 --2- 172.21.15.167:0/3169388355 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5576a688a410 0x5576a68b1da0 secure :-1 s=READY pgs=575 cs=0 l=1 rev1=1 crypto rx=0x7fd82800a470 tx=0x7fd82801d000 comp rx=0 tx=0).stop 2023-10-05T19:29:33.087 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.082+0000 7fd82ed2b700 1 -- 172.21.15.167:0/3169388355 reap_dead start 2023-10-05T19:29:33.087 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.082+0000 7fd8309f5e40 1 -- 172.21.15.167:0/3169388355 shutdown_connections 2023-10-05T19:29:33.087 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.082+0000 7fd8309f5e40 1 --2- 172.21.15.167:0/3169388355 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5576a688a410 0x5576a68b1da0 unknown :-1 s=CLOSED pgs=575 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-05T19:29:33.087 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.082+0000 7fd8309f5e40 1 -- 172.21.15.167:0/3169388355 >> 172.21.15.167:0/3169388355 conn(0x5576a6884720 msgr2=0x5576a68a5780 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-05T19:29:33.087 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.082+0000 7fd8309f5e40 1 -- 172.21.15.167:0/3169388355 shutdown_connections 2023-10-05T19:29:33.087 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.082+0000 7fd8309f5e40 1 -- 172.21.15.167:0/3169388355 wait complete. 2023-10-05T19:29:33.088 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.082+0000 7fd8309f5e40 1 -- 172.21.15.167:0/3942372936 >> [v2:172.21.15.167:6824/29889,v1:172.21.15.167:6825/29889] conn(0x7fd774030c50 msgr2=0x7fd774033100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-05T19:29:33.088 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.082+0000 7fd8309f5e40 1 --2- 172.21.15.167:0/3942372936 >> [v2:172.21.15.167:6824/29889,v1:172.21.15.167:6825/29889] conn(0x7fd774030c50 0x7fd774033100 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x5576a68cb470 tx=0x7fd820000f90 comp rx=0 tx=0).stop 2023-10-05T19:29:33.088 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.082+0000 7fd8309f5e40 1 -- 172.21.15.167:0/3942372936 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5576a68d9cb0 msgr2=0x5576a68df7b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-05T19:29:33.089 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.082+0000 7fd8309f5e40 1 --2- 172.21.15.167:0/3942372936 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5576a68d9cb0 0x5576a68df7b0 secure :-1 s=READY pgs=577 cs=0 l=1 rev1=1 crypto rx=0x5576a68d3d90 tx=0x7fd81802f000 comp rx=0 tx=0).stop 2023-10-05T19:29:33.089 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.082+0000 7fd8309f5e40 1 -- 172.21.15.167:0/3942372936 shutdown_connections 2023-10-05T19:29:33.089 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.082+0000 7fd8309f5e40 1 --2- 172.21.15.167:0/3942372936 >> [v2:172.21.15.167:6824/29889,v1:172.21.15.167:6825/29889] conn(0x7fd774030c50 0x7fd774033100 unknown :-1 s=CLOSED pgs=199 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-05T19:29:33.089 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.082+0000 7fd8309f5e40 1 --2- 172.21.15.167:0/3942372936 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5576a68d9cb0 0x5576a68df7b0 unknown :-1 s=CLOSED pgs=577 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-05T19:29:33.089 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.082+0000 7fd8309f5e40 1 -- 172.21.15.167:0/3942372936 >> 172.21.15.167:0/3942372936 conn(0x5576a68d5070 msgr2=0x5576a68d3170 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-05T19:29:33.089 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.082+0000 7fd8309f5e40 1 -- 172.21.15.167:0/3942372936 shutdown_connections 2023-10-05T19:29:33.089 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.082+0000 7fd8309f5e40 1 -- 172.21.15.167:0/3942372936 wait complete. 2023-10-05T19:29:33.090 INFO:teuthology.orchestra.run.smithi167.stdout:{ 2023-10-05T19:29:33.090 INFO:teuthology.orchestra.run.smithi167.stdout: "entries": [ 2023-10-05T19:29:33.090 INFO:teuthology.orchestra.run.smithi167.stdout: { 2023-10-05T19:29:33.090 INFO:teuthology.orchestra.run.smithi167.stdout: "user": "anonymous", 2023-10-05T19:29:33.090 INFO:teuthology.orchestra.run.smithi167.stdout: "buckets": [ 2023-10-05T19:29:33.090 INFO:teuthology.orchestra.run.smithi167.stdout: { 2023-10-05T19:29:33.090 INFO:teuthology.orchestra.run.smithi167.stdout: "bucket": "", 2023-10-05T19:29:33.090 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2023-10-05T19:00:00.000000Z", 2023-10-05T19:29:33.091 INFO:teuthology.orchestra.run.smithi167.stdout: "epoch": 1696532400, 2023-10-05T19:29:33.091 INFO:teuthology.orchestra.run.smithi167.stdout: "owner": "anonymous", 2023-10-05T19:29:33.091 INFO:teuthology.orchestra.run.smithi167.stdout: "categories": [] 2023-10-05T19:29:33.091 INFO:teuthology.orchestra.run.smithi167.stdout: } 2023-10-05T19:29:33.091 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2023-10-05T19:29:33.091 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2023-10-05T19:29:33.091 INFO:teuthology.orchestra.run.smithi167.stdout: { 2023-10-05T19:29:33.091 INFO:teuthology.orchestra.run.smithi167.stdout: "user": "foo", 2023-10-05T19:29:33.091 INFO:teuthology.orchestra.run.smithi167.stdout: "buckets": [ 2023-10-05T19:29:33.091 INFO:teuthology.orchestra.run.smithi167.stdout: { 2023-10-05T19:29:33.092 INFO:teuthology.orchestra.run.smithi167.stdout: "bucket": "", 2023-10-05T19:29:33.092 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2023-10-05T19:00:00.000000Z", 2023-10-05T19:29:33.092 INFO:teuthology.orchestra.run.smithi167.stdout: "epoch": 1696532400, 2023-10-05T19:29:33.092 INFO:teuthology.orchestra.run.smithi167.stdout: "owner": "foo", 2023-10-05T19:29:33.092 INFO:teuthology.orchestra.run.smithi167.stdout: "categories": [] 2023-10-05T19:29:33.092 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2023-10-05T19:29:33.092 INFO:teuthology.orchestra.run.smithi167.stdout: { 2023-10-05T19:29:33.092 INFO:teuthology.orchestra.run.smithi167.stdout: "bucket": "myfoo", 2023-10-05T19:29:33.092 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2023-10-05T19:00:00.000000Z", 2023-10-05T19:29:33.092 INFO:teuthology.orchestra.run.smithi167.stdout: "epoch": 1696532400, 2023-10-05T19:29:33.093 INFO:teuthology.orchestra.run.smithi167.stdout: "owner": "foo", 2023-10-05T19:29:33.093 INFO:teuthology.orchestra.run.smithi167.stdout: "categories": [ 2023-10-05T19:29:33.093 INFO:teuthology.orchestra.run.smithi167.stdout: { 2023-10-05T19:29:33.093 INFO:teuthology.orchestra.run.smithi167.stdout: "category": "delete_obj", 2023-10-05T19:29:33.093 INFO:teuthology.orchestra.run.smithi167.stdout: "bytes_sent": 0, 2023-10-05T19:29:33.093 INFO:teuthology.orchestra.run.smithi167.stdout: "bytes_received": 0, 2023-10-05T19:29:33.093 INFO:teuthology.orchestra.run.smithi167.stdout: "ops": 4, 2023-10-05T19:29:33.093 INFO:teuthology.orchestra.run.smithi167.stdout: "successful_ops": 4 2023-10-05T19:29:33.093 INFO:teuthology.orchestra.run.smithi167.stdout: } 2023-10-05T19:29:33.093 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2023-10-05T19:29:33.094 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2023-10-05T19:29:33.094 INFO:teuthology.orchestra.run.smithi167.stdout: { 2023-10-05T19:29:33.094 INFO:teuthology.orchestra.run.smithi167.stdout: "bucket": "myfoo2", 2023-10-05T19:29:33.094 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2023-10-05T19:00:00.000000Z", 2023-10-05T19:29:33.094 INFO:teuthology.orchestra.run.smithi167.stdout: "epoch": 1696532400, 2023-10-05T19:29:33.094 INFO:teuthology.orchestra.run.smithi167.stdout: "owner": "foo", 2023-10-05T19:29:33.094 INFO:teuthology.orchestra.run.smithi167.stdout: "categories": [] 2023-10-05T19:29:33.094 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2023-10-05T19:29:33.094 INFO:teuthology.orchestra.run.smithi167.stdout: { 2023-10-05T19:29:33.094 INFO:teuthology.orchestra.run.smithi167.stdout: "bucket": "myfoo3", 2023-10-05T19:29:33.095 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2023-10-05T19:00:00.000000Z", 2023-10-05T19:29:33.095 INFO:teuthology.orchestra.run.smithi167.stdout: "epoch": 1696532400, 2023-10-05T19:29:33.095 INFO:teuthology.orchestra.run.smithi167.stdout: "owner": "foo", 2023-10-05T19:29:33.095 INFO:teuthology.orchestra.run.smithi167.stdout: "categories": [] 2023-10-05T19:29:33.095 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2023-10-05T19:29:33.095 INFO:teuthology.orchestra.run.smithi167.stdout: { 2023-10-05T19:29:33.095 INFO:teuthology.orchestra.run.smithi167.stdout: "bucket": "myfoo4", 2023-10-05T19:29:33.095 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2023-10-05T19:00:00.000000Z", 2023-10-05T19:29:33.095 INFO:teuthology.orchestra.run.smithi167.stdout: "epoch": 1696532400, 2023-10-05T19:29:33.096 INFO:teuthology.orchestra.run.smithi167.stdout: "owner": "foo", 2023-10-05T19:29:33.096 INFO:teuthology.orchestra.run.smithi167.stdout: "categories": [] 2023-10-05T19:29:33.096 INFO:teuthology.orchestra.run.smithi167.stdout: } 2023-10-05T19:29:33.096 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2023-10-05T19:29:33.096 INFO:teuthology.orchestra.run.smithi167.stdout: } 2023-10-05T19:29:33.096 INFO:teuthology.orchestra.run.smithi167.stdout: ], 2023-10-05T19:29:33.096 INFO:teuthology.orchestra.run.smithi167.stdout: "summary": [ 2023-10-05T19:29:33.096 INFO:teuthology.orchestra.run.smithi167.stdout: { 2023-10-05T19:29:33.096 INFO:teuthology.orchestra.run.smithi167.stdout: "user": "anonymous", 2023-10-05T19:29:33.096 INFO:teuthology.orchestra.run.smithi167.stdout: "categories": [], 2023-10-05T19:29:33.097 INFO:teuthology.orchestra.run.smithi167.stdout: "total": { 2023-10-05T19:29:33.097 INFO:teuthology.orchestra.run.smithi167.stdout: "bytes_sent": 0, 2023-10-05T19:29:33.097 INFO:teuthology.orchestra.run.smithi167.stdout: "bytes_received": 0, 2023-10-05T19:29:33.097 INFO:teuthology.orchestra.run.smithi167.stdout: "ops": 0, 2023-10-05T19:29:33.097 INFO:teuthology.orchestra.run.smithi167.stdout: "successful_ops": 0 2023-10-05T19:29:33.097 INFO:teuthology.orchestra.run.smithi167.stdout: } 2023-10-05T19:29:33.097 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2023-10-05T19:29:33.097 INFO:teuthology.orchestra.run.smithi167.stdout: { 2023-10-05T19:29:33.097 INFO:teuthology.orchestra.run.smithi167.stdout: "user": "foo", 2023-10-05T19:29:33.097 INFO:teuthology.orchestra.run.smithi167.stdout: "categories": [ 2023-10-05T19:29:33.097 INFO:teuthology.orchestra.run.smithi167.stdout: { 2023-10-05T19:29:33.098 INFO:teuthology.orchestra.run.smithi167.stdout: "category": "delete_obj", 2023-10-05T19:29:33.098 INFO:teuthology.orchestra.run.smithi167.stdout: "bytes_sent": 0, 2023-10-05T19:29:33.098 INFO:teuthology.orchestra.run.smithi167.stdout: "bytes_received": 0, 2023-10-05T19:29:33.098 INFO:teuthology.orchestra.run.smithi167.stdout: "ops": 4, 2023-10-05T19:29:33.098 INFO:teuthology.orchestra.run.smithi167.stdout: "successful_ops": 4 2023-10-05T19:29:33.098 INFO:teuthology.orchestra.run.smithi167.stdout: } 2023-10-05T19:29:33.098 INFO:teuthology.orchestra.run.smithi167.stdout: ], 2023-10-05T19:29:33.098 INFO:teuthology.orchestra.run.smithi167.stdout: "total": { 2023-10-05T19:29:33.098 INFO:teuthology.orchestra.run.smithi167.stdout: "bytes_sent": 0, 2023-10-05T19:29:33.098 INFO:teuthology.orchestra.run.smithi167.stdout: "bytes_received": 0, 2023-10-05T19:29:33.099 INFO:teuthology.orchestra.run.smithi167.stdout: "ops": 4, 2023-10-05T19:29:33.099 INFO:teuthology.orchestra.run.smithi167.stdout: "successful_ops": 4 2023-10-05T19:29:33.099 INFO:teuthology.orchestra.run.smithi167.stdout: } 2023-10-05T19:29:33.099 INFO:teuthology.orchestra.run.smithi167.stdout: } 2023-10-05T19:29:33.099 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2023-10-05T19:29:33.099 INFO:teuthology.orchestra.run.smithi167.stdout:} 2023-10-05T19:29:33.100 DEBUG:tasks.util.rgw: json result: {'entries': [{'user': 'anonymous', 'buckets': [{'bucket': '', 'time': '2023-10-05T19:00:00.000000Z', 'epoch': 1696532400, 'owner': 'anonymous', 'categories': []}]}, {'user': 'foo', 'buckets': [{'bucket': '', 'time': '2023-10-05T19:00:00.000000Z', 'epoch': 1696532400, 'owner': 'foo', 'categories': []}, {'bucket': 'myfoo', 'time': '2023-10-05T19:00:00.000000Z', 'epoch': 1696532400, 'owner': 'foo', 'categories': [{'category': 'delete_obj', 'bytes_sent': 0, 'bytes_received': 0, 'ops': 4, 'successful_ops': 4}]}, {'bucket': 'myfoo2', 'time': '2023-10-05T19:00:00.000000Z', 'epoch': 1696532400, 'owner': 'foo', 'categories': []}, {'bucket': 'myfoo3', 'time': '2023-10-05T19:00:00.000000Z', 'epoch': 1696532400, 'owner': 'foo', 'categories': []}, {'bucket': 'myfoo4', 'time': '2023-10-05T19:00:00.000000Z', 'epoch': 1696532400, 'owner': 'foo', 'categories': []}]}], 'summary': [{'user': 'anonymous', 'categories': [], 'total': {'bytes_sent': 0, 'bytes_received': 0, 'ops': 0, 'successful_ops': 0}}, {'user': 'foo', 'categories': [{'category': 'delete_obj', 'bytes_sent': 0, 'bytes_received': 0, 'ops': 4, 'successful_ops': 4}], 'total': {'bytes_sent': 0, 'bytes_received': 0, 'ops': 4, 'successful_ops': 4}}]} 2023-10-05T19:29:33.100 INFO:tasks.util.rgw:rgwadmin: client.0 : ['usage', 'show'] 2023-10-05T19:29:33.101 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'ceph', 'usage', 'show'] 2023-10-05T19:29:33.101 DEBUG:teuthology.orchestra.run.smithi167:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster ceph usage show 2023-10-05T19:29:33.138 INFO:teuthology.orchestra.run.smithi167.stderr:ignoring --setuser ceph since I am not root 2023-10-05T19:29:33.138 INFO:teuthology.orchestra.run.smithi167.stderr:ignoring --setgroup ceph since I am not root 2023-10-05T19:29:33.139 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.134+0000 7f25033e0e40 1 Processor -- start 2023-10-05T19:29:33.139 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.134+0000 7f25033e0e40 1 -- start start 2023-10-05T19:29:33.140 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.134+0000 7f25033e0e40 1 --2- >> v2:172.21.15.118:3300/0 conn(0x564c7867d3e0 0x564c7867f7c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-05T19:29:33.140 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.134+0000 7f25033e0e40 1 -- --> v1:172.21.15.118:6789/0 -- auth(proto 0 26 bytes epoch 0) v1 -- 0x564c785c5c00 con 0x564c78678e00 2023-10-05T19:29:33.140 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.134+0000 7f25033e0e40 1 -- --> v2:172.21.15.118:3300/0 -- mon_getmap magic: 0 v1 -- 0x564c78582ee0 con 0x564c7867d3e0 2023-10-05T19:29:33.141 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.134+0000 7f2500714700 1 --2- >> v2:172.21.15.118:3300/0 conn(0x564c7867d3e0 0x564c7867f7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-05T19:29:33.141 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.134+0000 7f2500714700 1 --2- >> v2:172.21.15.118:3300/0 conn(0x564c7867d3e0 0x564c7867f7c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.118:3300/0 says I am v2:172.21.15.167:46156/0 (socket says 172.21.15.167:46156) 2023-10-05T19:29:33.141 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.134+0000 7f2500f15700 1 --1- >> v1:172.21.15.118:6789/0 conn(0x564c78678e00 0x564c786791d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.118:6789/0 says I am v1:172.21.15.167:52592/0 (socket says 172.21.15.167:52592) 2023-10-05T19:29:33.141 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.134+0000 7f2500714700 1 -- 172.21.15.167:0/1480648825 learned_addr learned my addr 172.21.15.167:0/1480648825 (peer_addr_for_me v2:172.21.15.167:0/0) 2023-10-05T19:29:33.141 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.138+0000 7f2500714700 1 -- 172.21.15.167:0/1480648825 >> v1:172.21.15.118:6789/0 conn(0x564c78678e00 legacy=0x564c786791d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-05T19:29:33.141 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.138+0000 7f2500714700 1 -- 172.21.15.167:0/1480648825 --> v2:172.21.15.118:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x564c78679a10 con 0x564c7867d3e0 2023-10-05T19:29:33.142 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.138+0000 7f2500714700 1 --2- 172.21.15.167:0/1480648825 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x564c7867d3e0 0x564c7867f7c0 secure :-1 s=READY pgs=579 cs=0 l=1 rev1=1 crypto rx=0x7f24e800a650 tx=0x7f24e8005b10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9a3b2b6f5c57067a server_cookie=0 in_seq=0 out_seq=0 2023-10-05T19:29:33.143 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.138+0000 7f24fff13700 1 -- 172.21.15.167:0/1480648825 <== mon.0 v2:172.21.15.118:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f24e8008760 con 0x564c7867d3e0 2023-10-05T19:29:33.143 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.138+0000 7f24fff13700 1 -- 172.21.15.167:0/1480648825 <== mon.0 v2:172.21.15.118:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f24e80088c0 con 0x564c7867d3e0 2023-10-05T19:29:33.143 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.138+0000 7f24fff13700 1 -- 172.21.15.167:0/1480648825 <== mon.0 v2:172.21.15.118:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f24e8008bb0 con 0x564c7867d3e0 2023-10-05T19:29:33.143 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.138+0000 7f25033e0e40 1 -- 172.21.15.167:0/1480648825 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x564c7867d3e0 msgr2=0x564c7867f7c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-05T19:29:33.143 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.138+0000 7f25033e0e40 1 --2- 172.21.15.167:0/1480648825 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x564c7867d3e0 0x564c7867f7c0 secure :-1 s=READY pgs=579 cs=0 l=1 rev1=1 crypto rx=0x7f24e800a650 tx=0x7f24e8005b10 comp rx=0 tx=0).stop 2023-10-05T19:29:33.143 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.138+0000 7f25033e0e40 1 -- 172.21.15.167:0/1480648825 shutdown_connections 2023-10-05T19:29:33.143 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.138+0000 7f25033e0e40 1 --2- 172.21.15.167:0/1480648825 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x564c7867d3e0 0x564c7867f7c0 unknown :-1 s=CLOSED pgs=579 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-05T19:29:33.143 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.138+0000 7f25033e0e40 1 -- 172.21.15.167:0/1480648825 >> 172.21.15.167:0/1480648825 conn(0x564c78674820 msgr2=0x564c78676c30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-05T19:29:33.143 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.138+0000 7f25033e0e40 1 -- 172.21.15.167:0/1480648825 shutdown_connections 2023-10-05T19:29:33.144 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.138+0000 7f25033e0e40 1 -- 172.21.15.167:0/1480648825 wait complete. 2023-10-05T19:29:33.145 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.138+0000 7f25033e0e40 1 Processor -- start 2023-10-05T19:29:33.145 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.138+0000 7f25033e0e40 1 -- start start 2023-10-05T19:29:33.145 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.138+0000 7f25033e0e40 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x564c78678e00 0x564c78697f30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-05T19:29:33.145 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.138+0000 7f25033e0e40 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x564c78546e30 con 0x564c78678e00 2023-10-05T19:29:33.145 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.138+0000 7f2500f15700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x564c78678e00 0x564c78697f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-05T19:29:33.145 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.138+0000 7f2500f15700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x564c78678e00 0x564c78697f30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.118:3300/0 says I am v2:172.21.15.167:46164/0 (socket says 172.21.15.167:46164) 2023-10-05T19:29:33.146 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.138+0000 7f2500f15700 1 -- 172.21.15.167:0/2539582203 learned_addr learned my addr 172.21.15.167:0/2539582203 (peer_addr_for_me v2:172.21.15.167:0/0) 2023-10-05T19:29:33.146 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.138+0000 7f2500f15700 1 -- 172.21.15.167:0/2539582203 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f24e8005db0 con 0x564c78678e00 2023-10-05T19:29:33.146 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.142+0000 7f2500f15700 1 --2- 172.21.15.167:0/2539582203 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x564c78678e00 0x564c78697f30 secure :-1 s=READY pgs=580 cs=0 l=1 rev1=1 crypto rx=0x7f24f80014a0 tx=0x7f24f8006b90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4ec54c46bb117746 server_cookie=0 in_seq=0 out_seq=0 2023-10-05T19:29:33.146 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.142+0000 7f24fef11700 1 -- 172.21.15.167:0/2539582203 <== mon.0 v2:172.21.15.118:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f24f8006110 con 0x564c78678e00 2023-10-05T19:29:33.146 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.142+0000 7f24fef11700 1 -- 172.21.15.167:0/2539582203 <== mon.0 v2:172.21.15.118:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f24f8006270 con 0x564c78678e00 2023-10-05T19:29:33.146 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.142+0000 7f24fef11700 1 -- 172.21.15.167:0/2539582203 <== mon.0 v2:172.21.15.118:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f24f80135a0 con 0x564c78678e00 2023-10-05T19:29:33.148 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.142+0000 7f25033e0e40 1 -- 172.21.15.167:0/2539582203 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x564c78678e00 msgr2=0x564c78697f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-05T19:29:33.148 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.142+0000 7f25033e0e40 1 --2- 172.21.15.167:0/2539582203 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x564c78678e00 0x564c78697f30 secure :-1 s=READY pgs=580 cs=0 l=1 rev1=1 crypto rx=0x7f24f80014a0 tx=0x7f24f8006b90 comp rx=0 tx=0).stop 2023-10-05T19:29:33.148 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.142+0000 7f25033e0e40 1 -- 172.21.15.167:0/2539582203 shutdown_connections 2023-10-05T19:29:33.148 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.142+0000 7f25033e0e40 1 --2- 172.21.15.167:0/2539582203 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x564c78678e00 0x564c78697f30 unknown :-1 s=CLOSED pgs=580 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-05T19:29:33.148 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.142+0000 7f25033e0e40 1 -- 172.21.15.167:0/2539582203 >> 172.21.15.167:0/2539582203 conn(0x564c78674820 msgr2=0x564c78693af0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-05T19:29:33.148 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.142+0000 7f25033e0e40 1 -- 172.21.15.167:0/2539582203 shutdown_connections 2023-10-05T19:29:33.148 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.142+0000 7f25033e0e40 1 -- 172.21.15.167:0/2539582203 wait complete. 2023-10-05T19:29:33.148 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.142+0000 7f25033e0e40 1 Processor -- start 2023-10-05T19:29:33.148 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.142+0000 7f25033e0e40 1 -- start start 2023-10-05T19:29:33.149 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.142+0000 7f25033e0e40 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x564c7867d3e0 0x564c786a1020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-05T19:29:33.149 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.142+0000 7f25033e0e40 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x564c78519ed0 con 0x564c7867d3e0 2023-10-05T19:29:33.149 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.142+0000 7f2500f15700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x564c7867d3e0 0x564c786a1020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-05T19:29:33.149 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.142+0000 7f2500f15700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x564c7867d3e0 0x564c786a1020 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.118:3300/0 says I am v2:172.21.15.167:46170/0 (socket says 172.21.15.167:46170) 2023-10-05T19:29:33.149 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.142+0000 7f2500f15700 1 -- 172.21.15.167:0/3463850574 learned_addr learned my addr 172.21.15.167:0/3463850574 (peer_addr_for_me v2:172.21.15.167:0/0) 2023-10-05T19:29:33.149 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.142+0000 7f2500f15700 1 -- 172.21.15.167:0/3463850574 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x564c785c5c00 con 0x564c7867d3e0 2023-10-05T19:29:33.149 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.142+0000 7f2500f15700 1 --2- 172.21.15.167:0/3463850574 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x564c7867d3e0 0x564c786a1020 secure :-1 s=READY pgs=581 cs=0 l=1 rev1=1 crypto rx=0x7f24f800f000 tx=0x7f24f8014000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-05T19:29:33.151 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.142+0000 7f24fd70e700 1 -- 172.21.15.167:0/3463850574 <== mon.0 v2:172.21.15.118:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f24f80046f0 con 0x564c7867d3e0 2023-10-05T19:29:33.151 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.142+0000 7f24fd70e700 1 -- 172.21.15.167:0/3463850574 <== mon.0 v2:172.21.15.118:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f24f8004850 con 0x564c7867d3e0 2023-10-05T19:29:33.151 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.142+0000 7f25033e0e40 1 -- 172.21.15.167:0/3463850574 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x564c786a1560 con 0x564c7867d3e0 2023-10-05T19:29:33.151 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.142+0000 7f24fd70e700 1 -- 172.21.15.167:0/3463850574 <== mon.0 v2:172.21.15.118:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f24f8004b40 con 0x564c7867d3e0 2023-10-05T19:29:33.151 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.142+0000 7f25033e0e40 1 -- 172.21.15.167:0/3463850574 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x564c786a19a0 con 0x564c7867d3e0 2023-10-05T19:29:33.151 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.146+0000 7f24fd70e700 1 -- 172.21.15.167:0/3463850574 <== mon.0 v2:172.21.15.118:3300/0 4 ==== mgrmap(e 5) v1 ==== 40329+0+0 (secure 0 0 0) 0x7f24f8004ca0 con 0x564c7867d3e0 2023-10-05T19:29:33.151 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.146+0000 7f24fd70e700 1 --2- 172.21.15.167:0/3463850574 >> [v2:172.21.15.167:6824/29889,v1:172.21.15.167:6825/29889] conn(0x7f24e0030c00 0x7f24e00330b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-05T19:29:33.151 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.146+0000 7f24fd70e700 1 -- 172.21.15.167:0/3463850574 <== mon.0 v2:172.21.15.118:3300/0 5 ==== osd_map(32..32 src has 1..32) v4 ==== 6260+0+0 (secure 0 0 0) 0x7f24f80452e0 con 0x564c7867d3e0 2023-10-05T19:29:33.151 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.146+0000 7f2500714700 1 --2- 172.21.15.167:0/3463850574 >> [v2:172.21.15.167:6824/29889,v1:172.21.15.167:6825/29889] conn(0x7f24e0030c00 0x7f24e00330b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-05T19:29:33.152 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.146+0000 7f25033e0e40 20 rados->read ofs=0 len=0 2023-10-05T19:29:33.152 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.146+0000 7f25033e0e40 1 --2- 172.21.15.167:0/3463850574 >> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] conn(0x564c786aaa50 0x564c786ace60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-05T19:29:33.152 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.146+0000 7f25033e0e40 1 -- 172.21.15.167:0/3463850574 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:1 5.12 5:49953fa1:::default.realm:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32) v8 -- 0x564c786ad590 con 0x564c786aaa50 2023-10-05T19:29:33.153 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.146+0000 7f2501716700 1 --2- 172.21.15.167:0/3463850574 >> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] conn(0x564c786aaa50 0x564c786ace60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-05T19:29:33.153 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.146+0000 7f2500714700 1 --2- 172.21.15.167:0/3463850574 >> [v2:172.21.15.167:6824/29889,v1:172.21.15.167:6825/29889] conn(0x7f24e0030c00 0x7f24e00330b0 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7f24e80030a0 tx=0x7f24e8015000 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-05T19:29:33.153 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.146+0000 7f2501716700 1 --2- 172.21.15.167:0/3463850574 >> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] conn(0x564c786aaa50 0x564c786ace60 crc :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-05T19:29:33.153 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.150+0000 7f2501716700 1 -- 172.21.15.167:0/3463850574 <== osd.1 v2:172.21.15.167:6800/30035 1 ==== osd_op_reply(1 default.realm [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 ==== 157+0+0 (crc 0 0 0) 0x7f24f000a500 con 0x564c786aaa50 2023-10-05T19:29:33.154 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.150+0000 7f25033e0e40 20 rados_obj.operate() r=-2 bl.length=0 2023-10-05T19:29:33.154 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.150+0000 7f25033e0e40 20 realm 2023-10-05T19:29:33.154 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.150+0000 7f25033e0e40 20 rados->read ofs=0 len=0 2023-10-05T19:29:33.154 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.150+0000 7f25033e0e40 1 -- 172.21.15.167:0/3463850574 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:2 5.12 5:49953fa1:::default.realm:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32) v8 -- 0x564c786aa030 con 0x564c786aaa50 2023-10-05T19:29:33.154 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.150+0000 7f2501716700 1 -- 172.21.15.167:0/3463850574 <== osd.1 v2:172.21.15.167:6800/30035 2 ==== osd_op_reply(2 default.realm [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 ==== 157+0+0 (crc 0 0 0) 0x7f24f000a500 con 0x564c786aaa50 2023-10-05T19:29:33.154 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.150+0000 7f25033e0e40 20 rados_obj.operate() r=-2 bl.length=0 2023-10-05T19:29:33.154 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.150+0000 7f25033e0e40 4 RGWPeriod::init failed to init realm id : (2) No such file or directory 2023-10-05T19:29:33.154 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.150+0000 7f25033e0e40 20 rados->read ofs=0 len=0 2023-10-05T19:29:33.154 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.150+0000 7f25033e0e40 1 -- 172.21.15.167:0/3463850574 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:3 5.12 5:49953fa1:::default.realm:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32) v8 -- 0x564c786ae070 con 0x564c786aaa50 2023-10-05T19:29:33.156 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.150+0000 7f2501716700 1 -- 172.21.15.167:0/3463850574 <== osd.1 v2:172.21.15.167:6800/30035 3 ==== osd_op_reply(3 default.realm [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 ==== 157+0+0 (crc 0 0 0) 0x7f24f000a500 con 0x564c786aaa50 2023-10-05T19:29:33.156 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.150+0000 7f25033e0e40 20 rados_obj.operate() r=-2 bl.length=0 2023-10-05T19:29:33.156 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.150+0000 7f25033e0e40 20 rados->read ofs=0 len=0 2023-10-05T19:29:33.157 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.150+0000 7f25033e0e40 1 -- 172.21.15.167:0/3463850574 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:4 5.f 5:f43b6ece:::zone_names.default:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32) v8 -- 0x564c786adc20 con 0x564c786aaa50 2023-10-05T19:29:33.157 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.150+0000 7f2501716700 1 -- 172.21.15.167:0/3463850574 <== osd.1 v2:172.21.15.167:6800/30035 4 ==== osd_op_reply(4 zone_names.default [read 0~46 out=46b] v0'0 uv1 ondisk = 0) v8 ==== 162+0+46 (crc 0 0 0) 0x7f24f000a500 con 0x564c786aaa50 2023-10-05T19:29:33.157 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.150+0000 7f25033e0e40 20 rados_obj.operate() r=0 bl.length=46 2023-10-05T19:29:33.157 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.150+0000 7f25033e0e40 20 rados->read ofs=0 len=0 2023-10-05T19:29:33.157 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.150+0000 7f25033e0e40 1 -- 172.21.15.167:0/3463850574 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:5 5.1d 5:b8374eb4:::zone_info.9a9df965-6612-45d4-a45f-6e3c889bacc2:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32) v8 -- 0x564c786aeaa0 con 0x564c786aaa50 2023-10-05T19:29:33.157 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.150+0000 7f2501716700 1 -- 172.21.15.167:0/3463850574 <== osd.1 v2:172.21.15.167:6800/30035 5 ==== osd_op_reply(5 zone_info.9a9df965-6612-45d4-a45f-6e3c889bacc2 [read 0~874 out=874b] v0'0 uv2 ondisk = 0) v8 ==== 190+0+874 (crc 0 0 0) 0x7f24f000a500 con 0x564c786aaa50 2023-10-05T19:29:33.157 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.150+0000 7f25033e0e40 20 rados_obj.operate() r=0 bl.length=874 2023-10-05T19:29:33.157 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.150+0000 7f25033e0e40 20 searching for the correct realm 2023-10-05T19:29:33.157 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.150+0000 7f25033e0e40 1 --2- 172.21.15.167:0/3463850574 >> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] conn(0x564c786b08f0 0x564c786b2cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-05T19:29:33.158 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.150+0000 7f25033e0e40 1 -- 172.21.15.167:0/3463850574 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:6 5.0 5:00000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x564c786b3210 con 0x564c786b08f0 2023-10-05T19:29:33.158 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.150+0000 7f2500f15700 1 --2- 172.21.15.167:0/3463850574 >> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] conn(0x564c786b08f0 0x564c786b2cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-05T19:29:33.158 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.150+0000 7f2500f15700 1 --2- 172.21.15.167:0/3463850574 >> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] conn(0x564c786b08f0 0x564c786b2cd0 crc :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-05T19:29:33.159 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.154+0000 7f2500f15700 1 -- 172.21.15.167:0/3463850574 <== osd.3 v2:172.21.15.167:6808/30036 1 ==== osd_op_reply(6 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f24f8024020 con 0x564c786b08f0 2023-10-05T19:29:33.159 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.154+0000 7f2500f15700 1 --2- 172.21.15.167:0/3463850574 >> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] conn(0x564c78678e00 0x7f24f80439f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-05T19:29:33.159 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.154+0000 7f2500f15700 1 -- 172.21.15.167:0/3463850574 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:7 5.10 5:08000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x564c786b3210 con 0x564c78678e00 2023-10-05T19:29:33.159 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.154+0000 7f2500714700 1 --2- 172.21.15.167:0/3463850574 >> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] conn(0x564c78678e00 0x7f24f80439f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-05T19:29:33.159 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.154+0000 7f2500714700 1 --2- 172.21.15.167:0/3463850574 >> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] conn(0x564c78678e00 0x7f24f80439f0 crc :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-05T19:29:33.160 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.154+0000 7f2500714700 1 -- 172.21.15.167:0/3463850574 <== osd.2 v2:172.21.15.167:6816/30037 1 ==== osd_op_reply(7 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f24e8008ad0 con 0x564c78678e00 2023-10-05T19:29:33.160 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.154+0000 7f2500714700 1 --2- 172.21.15.167:0/3463850574 >> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] conn(0x7f24e8007030 0x7f24e801c920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-05T19:29:33.160 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.154+0000 7f2500714700 1 -- 172.21.15.167:0/3463850574 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:8 5.8 5:10000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x564c786b3210 con 0x7f24e8007030 2023-10-05T19:29:33.160 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.154+0000 7f2501716700 1 --2- 172.21.15.167:0/3463850574 >> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] conn(0x7f24e8007030 0x7f24e801c920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-05T19:29:33.161 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.154+0000 7f2501716700 1 --2- 172.21.15.167:0/3463850574 >> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] conn(0x7f24e8007030 0x7f24e801c920 crc :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-05T19:29:33.161 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.158+0000 7f2501716700 1 -- 172.21.15.167:0/3463850574 <== osd.0 v2:172.21.15.118:6800/27690 1 ==== osd_op_reply(8 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f24f0014040 con 0x7f24e8007030 2023-10-05T19:29:33.161 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.158+0000 7f2501716700 1 -- 172.21.15.167:0/3463850574 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:9 5.18 5:18000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x564c786b3210 con 0x564c786aaa50 2023-10-05T19:29:33.161 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.158+0000 7f2501716700 1 -- 172.21.15.167:0/3463850574 <== osd.1 v2:172.21.15.167:6800/30035 6 ==== osd_op_reply(9 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f24f0014040 con 0x564c786aaa50 2023-10-05T19:29:33.161 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.158+0000 7f2501716700 1 -- 172.21.15.167:0/3463850574 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:10 5.4 5:20000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x564c786b3210 con 0x564c786aaa50 2023-10-05T19:29:33.162 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.158+0000 7f2501716700 1 -- 172.21.15.167:0/3463850574 <== osd.1 v2:172.21.15.167:6800/30035 7 ==== osd_op_reply(10 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f24f0014040 con 0x564c786aaa50 2023-10-05T19:29:33.162 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.158+0000 7f2501716700 1 -- 172.21.15.167:0/3463850574 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:11 5.14 5:28000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x564c786b3210 con 0x7f24e8007030 2023-10-05T19:29:33.162 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.158+0000 7f2501716700 1 -- 172.21.15.167:0/3463850574 <== osd.0 v2:172.21.15.118:6800/27690 2 ==== osd_op_reply(11 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f24f0014040 con 0x7f24e8007030 2023-10-05T19:29:33.163 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.158+0000 7f2501716700 1 -- 172.21.15.167:0/3463850574 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:12 5.c 5:30000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x564c786b3210 con 0x7f24e8007030 2023-10-05T19:29:33.163 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.158+0000 7f2501716700 1 -- 172.21.15.167:0/3463850574 <== osd.0 v2:172.21.15.118:6800/27690 3 ==== osd_op_reply(12 [pgnls start_epoch 32 out=112b] v23'1 uv1 ondisk = 1) v8 ==== 144+0+112 (crc 0 0 0) 0x7f24f0014040 con 0x7f24e8007030 2023-10-05T19:29:33.163 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.158+0000 7f2501716700 1 -- 172.21.15.167:0/3463850574 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:13 5.1c 5:38000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x564c786b3210 con 0x564c78678e00 2023-10-05T19:29:33.164 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.158+0000 7f2500714700 1 -- 172.21.15.167:0/3463850574 <== osd.2 v2:172.21.15.167:6816/30037 2 ==== osd_op_reply(13 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f24e8008ad0 con 0x564c78678e00 2023-10-05T19:29:33.164 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.158+0000 7f2500714700 1 -- 172.21.15.167:0/3463850574 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:14 5.2 5:40000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x564c786b3210 con 0x564c786b08f0 2023-10-05T19:29:33.165 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.162+0000 7f2500f15700 1 -- 172.21.15.167:0/3463850574 <== osd.3 v2:172.21.15.167:6808/30036 2 ==== osd_op_reply(14 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f24f8024020 con 0x564c786b08f0 2023-10-05T19:29:33.165 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.162+0000 7f2500f15700 1 -- 172.21.15.167:0/3463850574 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:15 5.12 5:48000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x564c786b3210 con 0x564c786aaa50 2023-10-05T19:29:33.165 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.162+0000 7f2501716700 1 -- 172.21.15.167:0/3463850574 <== osd.1 v2:172.21.15.167:6800/30035 8 ==== osd_op_reply(15 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f24f0014040 con 0x564c786aaa50 2023-10-05T19:29:33.165 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.162+0000 7f2501716700 1 -- 172.21.15.167:0/3463850574 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:16 5.a 5:50000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x564c786b3210 con 0x7f24e8007030 2023-10-05T19:29:33.166 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.162+0000 7f2501716700 1 -- 172.21.15.167:0/3463850574 <== osd.0 v2:172.21.15.118:6800/27690 4 ==== osd_op_reply(16 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f24f0022040 con 0x7f24e8007030 2023-10-05T19:29:33.167 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.162+0000 7f2501716700 1 -- 172.21.15.167:0/3463850574 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:17 5.1a 5:58000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x564c786b3210 con 0x564c786aaa50 2023-10-05T19:29:33.167 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.162+0000 7f2501716700 1 -- 172.21.15.167:0/3463850574 <== osd.1 v2:172.21.15.167:6800/30035 9 ==== osd_op_reply(17 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f24f0022040 con 0x564c786aaa50 2023-10-05T19:29:33.167 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.162+0000 7f2501716700 1 -- 172.21.15.167:0/3463850574 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:18 5.6 5:60000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x564c786b3210 con 0x564c78678e00 2023-10-05T19:29:33.168 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.162+0000 7f2500714700 1 -- 172.21.15.167:0/3463850574 <== osd.2 v2:172.21.15.167:6816/30037 3 ==== osd_op_reply(18 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f24e8008ad0 con 0x564c78678e00 2023-10-05T19:29:33.168 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.162+0000 7f2500714700 1 -- 172.21.15.167:0/3463850574 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:19 5.16 5:68000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x564c786b3210 con 0x564c786b08f0 2023-10-05T19:29:33.168 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.166+0000 7f2500f15700 1 -- 172.21.15.167:0/3463850574 <== osd.3 v2:172.21.15.167:6808/30036 3 ==== osd_op_reply(19 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f24f8024020 con 0x564c786b08f0 2023-10-05T19:29:33.169 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.166+0000 7f2500f15700 1 -- 172.21.15.167:0/3463850574 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:20 5.e 5:70000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x564c786b3210 con 0x564c786b08f0 2023-10-05T19:29:33.169 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.166+0000 7f2500f15700 1 -- 172.21.15.167:0/3463850574 <== osd.3 v2:172.21.15.167:6808/30036 4 ==== osd_op_reply(20 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f24f8024020 con 0x564c786b08f0 2023-10-05T19:29:33.169 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.166+0000 7f2500f15700 1 -- 172.21.15.167:0/3463850574 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:21 5.1e 5:78000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x564c786b3210 con 0x7f24e8007030 2023-10-05T19:29:33.170 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.166+0000 7f2501716700 1 -- 172.21.15.167:0/3463850574 <== osd.0 v2:172.21.15.118:6800/27690 5 ==== osd_op_reply(21 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f24f0022040 con 0x7f24e8007030 2023-10-05T19:29:33.170 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.166+0000 7f2501716700 1 -- 172.21.15.167:0/3463850574 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:22 5.1 5:80000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x564c786b3210 con 0x564c786b08f0 2023-10-05T19:29:33.171 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.166+0000 7f2500f15700 1 -- 172.21.15.167:0/3463850574 <== osd.3 v2:172.21.15.167:6808/30036 5 ==== osd_op_reply(22 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f24f8024020 con 0x564c786b08f0 2023-10-05T19:29:33.171 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.166+0000 7f2500f15700 1 -- 172.21.15.167:0/3463850574 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:23 5.11 5:88000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x564c786b3210 con 0x564c786b08f0 2023-10-05T19:29:33.171 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.166+0000 7f2500f15700 1 -- 172.21.15.167:0/3463850574 <== osd.3 v2:172.21.15.167:6808/30036 6 ==== osd_op_reply(23 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f24f8024020 con 0x564c786b08f0 2023-10-05T19:29:33.171 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.166+0000 7f2500f15700 1 -- 172.21.15.167:0/3463850574 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:24 5.9 5:90000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x564c786b3210 con 0x564c786aaa50 2023-10-05T19:29:33.172 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.166+0000 7f2501716700 1 -- 172.21.15.167:0/3463850574 <== osd.1 v2:172.21.15.167:6800/30035 10 ==== osd_op_reply(24 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f24f0022040 con 0x564c786aaa50 2023-10-05T19:29:33.172 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.166+0000 7f2501716700 1 -- 172.21.15.167:0/3463850574 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:25 5.19 5:98000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x564c786b3210 con 0x564c786aaa50 2023-10-05T19:29:33.173 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.170+0000 7f2501716700 1 -- 172.21.15.167:0/3463850574 <== osd.1 v2:172.21.15.167:6800/30035 11 ==== osd_op_reply(25 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f24f0022040 con 0x564c786aaa50 2023-10-05T19:29:33.173 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.170+0000 7f2501716700 1 -- 172.21.15.167:0/3463850574 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:26 5.5 5:a0000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x564c786b3210 con 0x7f24e8007030 2023-10-05T19:29:33.174 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.170+0000 7f2501716700 1 -- 172.21.15.167:0/3463850574 <== osd.0 v2:172.21.15.118:6800/27690 6 ==== osd_op_reply(26 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f24f0022040 con 0x7f24e8007030 2023-10-05T19:29:33.174 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.170+0000 7f2501716700 1 -- 172.21.15.167:0/3463850574 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:27 5.15 5:a8000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x564c786b3210 con 0x564c786b08f0 2023-10-05T19:29:33.174 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.170+0000 7f2500f15700 1 -- 172.21.15.167:0/3463850574 <== osd.3 v2:172.21.15.167:6808/30036 7 ==== osd_op_reply(27 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f24f8024020 con 0x564c786b08f0 2023-10-05T19:29:33.175 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.170+0000 7f2500f15700 1 -- 172.21.15.167:0/3463850574 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:28 5.d 5:b0000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x564c786b3210 con 0x564c78678e00 2023-10-05T19:29:33.175 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.170+0000 7f2500714700 1 -- 172.21.15.167:0/3463850574 <== osd.2 v2:172.21.15.167:6816/30037 4 ==== osd_op_reply(28 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f24e8008ad0 con 0x564c78678e00 2023-10-05T19:29:33.175 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.170+0000 7f2500714700 1 -- 172.21.15.167:0/3463850574 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:29 5.1d 5:b8000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x564c786b3210 con 0x564c786aaa50 2023-10-05T19:29:33.176 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.170+0000 7f2501716700 1 -- 172.21.15.167:0/3463850574 <== osd.1 v2:172.21.15.167:6800/30035 12 ==== osd_op_reply(29 [pgnls start_epoch 32 out=107b] v30'2 uv2 ondisk = 1) v8 ==== 144+0+107 (crc 0 0 0) 0x7f24f0022040 con 0x564c786aaa50 2023-10-05T19:29:33.176 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.170+0000 7f2501716700 1 -- 172.21.15.167:0/3463850574 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:30 5.3 5:c0000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x564c786b3210 con 0x7f24e8007030 2023-10-05T19:29:33.177 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.174+0000 7f2501716700 1 -- 172.21.15.167:0/3463850574 <== osd.0 v2:172.21.15.118:6800/27690 7 ==== osd_op_reply(30 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f24f0022040 con 0x7f24e8007030 2023-10-05T19:29:33.177 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.174+0000 7f2501716700 1 -- 172.21.15.167:0/3463850574 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:31 5.13 5:c8000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x564c786b3210 con 0x564c786b08f0 2023-10-05T19:29:33.178 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.174+0000 7f2500f15700 1 -- 172.21.15.167:0/3463850574 <== osd.3 v2:172.21.15.167:6808/30036 8 ==== osd_op_reply(31 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f24f8024020 con 0x564c786b08f0 2023-10-05T19:29:33.178 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.174+0000 7f2500f15700 1 -- 172.21.15.167:0/3463850574 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:32 5.b 5:d0000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x564c786b3210 con 0x7f24e8007030 2023-10-05T19:29:33.179 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.174+0000 7f2501716700 1 -- 172.21.15.167:0/3463850574 <== osd.0 v2:172.21.15.118:6800/27690 8 ==== osd_op_reply(32 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f24f0022040 con 0x7f24e8007030 2023-10-05T19:29:33.179 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.174+0000 7f2501716700 1 -- 172.21.15.167:0/3463850574 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:33 5.1b 5:d8000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x564c786b3210 con 0x564c78678e00 2023-10-05T19:29:33.180 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.174+0000 7f2500714700 1 -- 172.21.15.167:0/3463850574 <== osd.2 v2:172.21.15.167:6816/30037 5 ==== osd_op_reply(33 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f24e8008ad0 con 0x564c78678e00 2023-10-05T19:29:33.180 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.174+0000 7f2500714700 1 -- 172.21.15.167:0/3463850574 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:34 5.7 5:e0000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x564c786b3210 con 0x564c786b08f0 2023-10-05T19:29:33.180 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.174+0000 7f2500f15700 1 -- 172.21.15.167:0/3463850574 <== osd.3 v2:172.21.15.167:6808/30036 9 ==== osd_op_reply(34 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f24f8024020 con 0x564c786b08f0 2023-10-05T19:29:33.180 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.178+0000 7f2500f15700 1 -- 172.21.15.167:0/3463850574 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:35 5.17 5:e8000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x564c786b3210 con 0x7f24e8007030 2023-10-05T19:29:33.181 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.178+0000 7f2501716700 1 -- 172.21.15.167:0/3463850574 <== osd.0 v2:172.21.15.118:6800/27690 9 ==== osd_op_reply(35 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f24f0022040 con 0x7f24e8007030 2023-10-05T19:29:33.182 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.178+0000 7f2501716700 1 -- 172.21.15.167:0/3463850574 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:36 5.f 5:f0000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x564c786b3210 con 0x564c786aaa50 2023-10-05T19:29:33.182 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.178+0000 7f2501716700 1 -- 172.21.15.167:0/3463850574 <== osd.1 v2:172.21.15.167:6800/30035 13 ==== osd_op_reply(36 [pgnls start_epoch 32 out=115b] v23'2 uv2 ondisk = 1) v8 ==== 144+0+115 (crc 0 0 0) 0x7f24f002f080 con 0x564c786aaa50 2023-10-05T19:29:33.182 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.178+0000 7f2501716700 1 -- 172.21.15.167:0/3463850574 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:37 5.1f 5:f8000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x564c786b3210 con 0x564c78678e00 2023-10-05T19:29:33.183 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.178+0000 7f2500714700 1 -- 172.21.15.167:0/3463850574 <== osd.2 v2:172.21.15.167:6816/30037 6 ==== osd_op_reply(37 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f24e8008ad0 con 0x564c78678e00 2023-10-05T19:29:33.183 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.178+0000 7f25033e0e40 20 RGWRados::pool_iterate: got zonegroup_info.4f1f719d-9648-417b-b10c-e5d939825419 2023-10-05T19:29:33.183 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.178+0000 7f25033e0e40 20 RGWRados::pool_iterate: got zone_info.9a9df965-6612-45d4-a45f-6e3c889bacc2 2023-10-05T19:29:33.183 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.178+0000 7f25033e0e40 20 RGWRados::pool_iterate: got zone_names.default 2023-10-05T19:29:33.184 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.178+0000 7f25033e0e40 20 RGWRados::pool_iterate: got zonegroups_names.default 2023-10-05T19:29:33.184 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.178+0000 7f25033e0e40 20 rados->read ofs=0 len=0 2023-10-05T19:29:33.184 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.178+0000 7f25033e0e40 1 -- 172.21.15.167:0/3463850574 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:38 5.12 5:49953fa1:::default.realm:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32) v8 -- 0x564c786af670 con 0x564c786aaa50 2023-10-05T19:29:33.185 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.178+0000 7f2501716700 1 -- 172.21.15.167:0/3463850574 <== osd.1 v2:172.21.15.167:6800/30035 14 ==== osd_op_reply(38 default.realm [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 ==== 157+0+0 (crc 0 0 0) 0x7f24f002f080 con 0x564c786aaa50 2023-10-05T19:29:33.185 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.178+0000 7f25033e0e40 20 rados_obj.operate() r=-2 bl.length=0 2023-10-05T19:29:33.185 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.178+0000 7f25033e0e40 20 rados->read ofs=0 len=0 2023-10-05T19:29:33.185 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.178+0000 7f25033e0e40 1 -- 172.21.15.167:0/3463850574 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:39 5.f 5:f4c53578:::zonegroups_names.default:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32) v8 -- 0x564c786afff0 con 0x564c786aaa50 2023-10-05T19:29:33.185 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.178+0000 7f2501716700 1 -- 172.21.15.167:0/3463850574 <== osd.1 v2:172.21.15.167:6800/30035 15 ==== osd_op_reply(39 zonegroups_names.default [read 0~46 out=46b] v0'0 uv2 ondisk = 0) v8 ==== 168+0+46 (crc 0 0 0) 0x7f24f002f080 con 0x564c786aaa50 2023-10-05T19:29:33.185 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.182+0000 7f25033e0e40 20 rados_obj.operate() r=0 bl.length=46 2023-10-05T19:29:33.186 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.182+0000 7f25033e0e40 20 rados->read ofs=0 len=0 2023-10-05T19:29:33.186 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.182+0000 7f25033e0e40 1 -- 172.21.15.167:0/3463850574 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:40 5.c 5:30d299a5:::zonegroup_info.4f1f719d-9648-417b-b10c-e5d939825419:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32) v8 -- 0x564c786af230 con 0x7f24e8007030 2023-10-05T19:29:33.187 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.182+0000 7f2501716700 1 -- 172.21.15.167:0/3463850574 <== osd.0 v2:172.21.15.118:6800/27690 10 ==== osd_op_reply(40 zonegroup_info.4f1f719d-9648-417b-b10c-e5d939825419 [read 0~362 out=362b] v0'0 uv1 ondisk = 0) v8 ==== 195+0+362 (crc 0 0 0) 0x7f24f002f080 con 0x7f24e8007030 2023-10-05T19:29:33.187 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.182+0000 7f25033e0e40 20 rados_obj.operate() r=0 bl.length=362 2023-10-05T19:29:33.187 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.182+0000 7f25033e0e40 20 rados->read ofs=0 len=0 2023-10-05T19:29:33.187 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.182+0000 7f25033e0e40 1 -- 172.21.15.167:0/3463850574 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:41 5.13 5:cbbf6c60:::converted:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32) v8 -- 0x564c786b35c0 con 0x564c786b08f0 2023-10-05T19:29:33.188 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.182+0000 7f2500f15700 1 -- 172.21.15.167:0/3463850574 <== osd.3 v2:172.21.15.167:6808/30036 10 ==== osd_op_reply(41 converted [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 ==== 153+0+0 (crc 0 0 0) 0x7f24f8024020 con 0x564c786b08f0 2023-10-05T19:29:33.188 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.182+0000 7f25033e0e40 20 rados_obj.operate() r=-2 bl.length=0 2023-10-05T19:29:33.188 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.182+0000 7f25033e0e40 20 rados->read ofs=0 len=0 2023-10-05T19:29:33.188 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.182+0000 7f25033e0e40 1 -- 172.21.15.167:0/3463850574 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:42 5.12 5:49953fa1:::default.realm:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32) v8 -- 0x564c786b3970 con 0x564c786aaa50 2023-10-05T19:29:33.188 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.182+0000 7f2501716700 1 -- 172.21.15.167:0/3463850574 <== osd.1 v2:172.21.15.167:6800/30035 16 ==== osd_op_reply(42 default.realm [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 ==== 157+0+0 (crc 0 0 0) 0x7f24f002f080 con 0x564c786aaa50 2023-10-05T19:29:33.188 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.182+0000 7f25033e0e40 20 rados_obj.operate() r=-2 bl.length=0 2023-10-05T19:29:33.188 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.182+0000 7f25033e0e40 20 rados->read ofs=0 len=0 2023-10-05T19:29:33.188 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.182+0000 7f25033e0e40 1 -- 172.21.15.167:0/3463850574 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:43 5.f 5:f4c53578:::zonegroups_names.default:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32) v8 -- 0x564c786b3970 con 0x564c786aaa50 2023-10-05T19:29:33.189 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.182+0000 7f2501716700 1 -- 172.21.15.167:0/3463850574 <== osd.1 v2:172.21.15.167:6800/30035 17 ==== osd_op_reply(43 zonegroups_names.default [read 0~46 out=46b] v0'0 uv2 ondisk = 0) v8 ==== 168+0+46 (crc 0 0 0) 0x7f24f002f080 con 0x564c786aaa50 2023-10-05T19:29:33.189 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.182+0000 7f25033e0e40 20 rados_obj.operate() r=0 bl.length=46 2023-10-05T19:29:33.189 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.182+0000 7f25033e0e40 1 -- 172.21.15.167:0/3463850574 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:44 5.0 5:00000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x564c786b3970 con 0x564c786b08f0 2023-10-05T19:29:33.190 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.186+0000 7f2500f15700 1 -- 172.21.15.167:0/3463850574 <== osd.3 v2:172.21.15.167:6808/30036 11 ==== osd_op_reply(44 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f24f8024020 con 0x564c786b08f0 2023-10-05T19:29:33.190 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.186+0000 7f2500f15700 1 -- 172.21.15.167:0/3463850574 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:45 5.10 5:08000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x564c786b3970 con 0x564c78678e00 2023-10-05T19:29:33.190 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.186+0000 7f2500714700 1 -- 172.21.15.167:0/3463850574 <== osd.2 v2:172.21.15.167:6816/30037 7 ==== osd_op_reply(45 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f24e8008ad0 con 0x564c78678e00 2023-10-05T19:29:33.190 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.186+0000 7f2500714700 1 -- 172.21.15.167:0/3463850574 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:46 5.8 5:10000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x564c786b3970 con 0x7f24e8007030 2023-10-05T19:29:33.191 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.186+0000 7f2501716700 1 -- 172.21.15.167:0/3463850574 <== osd.0 v2:172.21.15.118:6800/27690 11 ==== osd_op_reply(46 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f24f002f080 con 0x7f24e8007030 2023-10-05T19:29:33.191 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.186+0000 7f2501716700 1 -- 172.21.15.167:0/3463850574 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:47 5.18 5:18000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x564c786afff0 con 0x564c786aaa50 2023-10-05T19:29:33.192 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.186+0000 7f2501716700 1 -- 172.21.15.167:0/3463850574 <== osd.1 v2:172.21.15.167:6800/30035 18 ==== osd_op_reply(47 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f24f002f080 con 0x564c786aaa50 2023-10-05T19:29:33.192 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.186+0000 7f2501716700 1 -- 172.21.15.167:0/3463850574 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:48 5.4 5:20000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x564c786afff0 con 0x564c786aaa50 2023-10-05T19:29:33.193 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.190+0000 7f2501716700 1 -- 172.21.15.167:0/3463850574 <== osd.1 v2:172.21.15.167:6800/30035 19 ==== osd_op_reply(48 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f24f002f080 con 0x564c786aaa50 2023-10-05T19:29:33.193 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.190+0000 7f2501716700 1 -- 172.21.15.167:0/3463850574 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:49 5.14 5:28000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x564c786afff0 con 0x7f24e8007030 2023-10-05T19:29:33.194 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.190+0000 7f2501716700 1 -- 172.21.15.167:0/3463850574 <== osd.0 v2:172.21.15.118:6800/27690 12 ==== osd_op_reply(49 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f24f002f080 con 0x7f24e8007030 2023-10-05T19:29:33.194 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.190+0000 7f2501716700 1 -- 172.21.15.167:0/3463850574 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:50 5.c 5:30000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x564c786afff0 con 0x7f24e8007030 2023-10-05T19:29:33.195 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.190+0000 7f2501716700 1 -- 172.21.15.167:0/3463850574 <== osd.0 v2:172.21.15.118:6800/27690 13 ==== osd_op_reply(50 [pgnls start_epoch 32 out=112b] v23'1 uv1 ondisk = 1) v8 ==== 144+0+112 (crc 0 0 0) 0x7f24f002f080 con 0x7f24e8007030 2023-10-05T19:29:33.195 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.190+0000 7f2501716700 1 -- 172.21.15.167:0/3463850574 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:51 5.1c 5:38000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x564c786afff0 con 0x564c78678e00 2023-10-05T19:29:33.196 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.190+0000 7f2500714700 1 -- 172.21.15.167:0/3463850574 <== osd.2 v2:172.21.15.167:6816/30037 8 ==== osd_op_reply(51 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f24e8008ad0 con 0x564c78678e00 2023-10-05T19:29:33.196 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.190+0000 7f2500714700 1 -- 172.21.15.167:0/3463850574 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:52 5.2 5:40000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x564c786afff0 con 0x564c786b08f0 2023-10-05T19:29:33.197 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.194+0000 7f2500f15700 1 -- 172.21.15.167:0/3463850574 <== osd.3 v2:172.21.15.167:6808/30036 12 ==== osd_op_reply(52 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f24f8024020 con 0x564c786b08f0 2023-10-05T19:29:33.197 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.194+0000 7f2500f15700 1 -- 172.21.15.167:0/3463850574 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:53 5.12 5:48000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x564c786afff0 con 0x564c786aaa50 2023-10-05T19:29:33.197 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.194+0000 7f2501716700 1 -- 172.21.15.167:0/3463850574 <== osd.1 v2:172.21.15.167:6800/30035 20 ==== osd_op_reply(53 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f24f002f080 con 0x564c786aaa50 2023-10-05T19:29:33.198 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.194+0000 7f2501716700 1 -- 172.21.15.167:0/3463850574 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:54 5.a 5:50000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x564c786afff0 con 0x7f24e8007030 2023-10-05T19:29:33.198 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.194+0000 7f2501716700 1 -- 172.21.15.167:0/3463850574 <== osd.0 v2:172.21.15.118:6800/27690 14 ==== osd_op_reply(54 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f24f002f080 con 0x7f24e8007030 2023-10-05T19:29:33.199 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.194+0000 7f2501716700 1 -- 172.21.15.167:0/3463850574 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:55 5.1a 5:58000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x564c786afff0 con 0x564c786aaa50 2023-10-05T19:29:33.199 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.194+0000 7f2501716700 1 -- 172.21.15.167:0/3463850574 <== osd.1 v2:172.21.15.167:6800/30035 21 ==== osd_op_reply(55 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f24f002f080 con 0x564c786aaa50 2023-10-05T19:29:33.199 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.194+0000 7f2501716700 1 -- 172.21.15.167:0/3463850574 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:56 5.6 5:60000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x564c786afff0 con 0x564c78678e00 2023-10-05T19:29:33.200 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.194+0000 7f2500714700 1 -- 172.21.15.167:0/3463850574 <== osd.2 v2:172.21.15.167:6816/30037 9 ==== osd_op_reply(56 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f24e8008ad0 con 0x564c78678e00 2023-10-05T19:29:33.200 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.194+0000 7f2500714700 1 -- 172.21.15.167:0/3463850574 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:57 5.16 5:68000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x564c786afff0 con 0x564c786b08f0 2023-10-05T19:29:33.201 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.198+0000 7f2500f15700 1 -- 172.21.15.167:0/3463850574 <== osd.3 v2:172.21.15.167:6808/30036 13 ==== osd_op_reply(57 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f24f8024020 con 0x564c786b08f0 2023-10-05T19:29:33.201 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.198+0000 7f2500f15700 1 -- 172.21.15.167:0/3463850574 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:58 5.e 5:70000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x564c786afff0 con 0x564c786b08f0 2023-10-05T19:29:33.202 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.198+0000 7f2500f15700 1 -- 172.21.15.167:0/3463850574 <== osd.3 v2:172.21.15.167:6808/30036 14 ==== osd_op_reply(58 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f24f8024020 con 0x564c786b08f0 2023-10-05T19:29:33.202 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.198+0000 7f2500f15700 1 -- 172.21.15.167:0/3463850574 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:59 5.1e 5:78000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x564c786afff0 con 0x7f24e8007030 2023-10-05T19:29:33.203 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.198+0000 7f2501716700 1 -- 172.21.15.167:0/3463850574 <== osd.0 v2:172.21.15.118:6800/27690 15 ==== osd_op_reply(59 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f24f002f080 con 0x7f24e8007030 2023-10-05T19:29:33.203 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.198+0000 7f2501716700 1 -- 172.21.15.167:0/3463850574 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:60 5.1 5:80000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x564c786afff0 con 0x564c786b08f0 2023-10-05T19:29:33.203 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.198+0000 7f2500f15700 1 -- 172.21.15.167:0/3463850574 <== osd.3 v2:172.21.15.167:6808/30036 15 ==== osd_op_reply(60 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f24f8024020 con 0x564c786b08f0 2023-10-05T19:29:33.203 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.198+0000 7f2500f15700 1 -- 172.21.15.167:0/3463850574 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:61 5.11 5:88000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x564c786afff0 con 0x564c786b08f0 2023-10-05T19:29:33.204 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.198+0000 7f2500f15700 1 -- 172.21.15.167:0/3463850574 <== osd.3 v2:172.21.15.167:6808/30036 16 ==== osd_op_reply(61 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f24f8024020 con 0x564c786b08f0 2023-10-05T19:29:33.204 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.198+0000 7f2500f15700 1 -- 172.21.15.167:0/3463850574 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:62 5.9 5:90000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x564c786afff0 con 0x564c786aaa50 2023-10-05T19:29:33.205 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.202+0000 7f2501716700 1 -- 172.21.15.167:0/3463850574 <== osd.1 v2:172.21.15.167:6800/30035 22 ==== osd_op_reply(62 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f24f002f080 con 0x564c786aaa50 2023-10-05T19:29:33.205 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.202+0000 7f2501716700 1 -- 172.21.15.167:0/3463850574 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:63 5.19 5:98000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x564c786afff0 con 0x564c786aaa50 2023-10-05T19:29:33.206 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.202+0000 7f2501716700 1 -- 172.21.15.167:0/3463850574 <== osd.1 v2:172.21.15.167:6800/30035 23 ==== osd_op_reply(63 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f24f002f080 con 0x564c786aaa50 2023-10-05T19:29:33.206 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.202+0000 7f2501716700 1 -- 172.21.15.167:0/3463850574 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:64 5.5 5:a0000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x564c786afff0 con 0x7f24e8007030 2023-10-05T19:29:33.207 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.202+0000 7f2501716700 1 -- 172.21.15.167:0/3463850574 <== osd.0 v2:172.21.15.118:6800/27690 16 ==== osd_op_reply(64 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f24f002f080 con 0x7f24e8007030 2023-10-05T19:29:33.207 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.202+0000 7f2501716700 1 -- 172.21.15.167:0/3463850574 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:65 5.15 5:a8000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x564c786afff0 con 0x564c786b08f0 2023-10-05T19:29:33.208 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.202+0000 7f2500f15700 1 -- 172.21.15.167:0/3463850574 <== osd.3 v2:172.21.15.167:6808/30036 17 ==== osd_op_reply(65 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f24f8024020 con 0x564c786b08f0 2023-10-05T19:29:33.208 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.202+0000 7f2500f15700 1 -- 172.21.15.167:0/3463850574 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:66 5.d 5:b0000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x564c786afff0 con 0x564c78678e00 2023-10-05T19:29:33.209 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.206+0000 7f2500714700 1 -- 172.21.15.167:0/3463850574 <== osd.2 v2:172.21.15.167:6816/30037 10 ==== osd_op_reply(66 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f24e8008ad0 con 0x564c78678e00 2023-10-05T19:29:33.209 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.206+0000 7f2500714700 1 -- 172.21.15.167:0/3463850574 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:67 5.1d 5:b8000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x564c786afff0 con 0x564c786aaa50 2023-10-05T19:29:33.210 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.206+0000 7f2501716700 1 -- 172.21.15.167:0/3463850574 <== osd.1 v2:172.21.15.167:6800/30035 24 ==== osd_op_reply(67 [pgnls start_epoch 32 out=107b] v30'2 uv2 ondisk = 1) v8 ==== 144+0+107 (crc 0 0 0) 0x7f24f002f080 con 0x564c786aaa50 2023-10-05T19:29:33.210 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.206+0000 7f2501716700 1 -- 172.21.15.167:0/3463850574 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:68 5.3 5:c0000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x564c786afff0 con 0x7f24e8007030 2023-10-05T19:29:33.211 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.206+0000 7f2501716700 1 -- 172.21.15.167:0/3463850574 <== osd.0 v2:172.21.15.118:6800/27690 17 ==== osd_op_reply(68 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f24f002f080 con 0x7f24e8007030 2023-10-05T19:29:33.211 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.206+0000 7f2501716700 1 -- 172.21.15.167:0/3463850574 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:69 5.13 5:c8000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x564c786afff0 con 0x564c786b08f0 2023-10-05T19:29:33.213 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.206+0000 7f2500f15700 1 -- 172.21.15.167:0/3463850574 <== osd.3 v2:172.21.15.167:6808/30036 18 ==== osd_op_reply(69 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f24f8024020 con 0x564c786b08f0 2023-10-05T19:29:33.214 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.206+0000 7f2500f15700 1 -- 172.21.15.167:0/3463850574 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:70 5.b 5:d0000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x564c786afff0 con 0x7f24e8007030 2023-10-05T19:29:33.214 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.210+0000 7f2501716700 1 -- 172.21.15.167:0/3463850574 <== osd.0 v2:172.21.15.118:6800/27690 18 ==== osd_op_reply(70 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f24f002f080 con 0x7f24e8007030 2023-10-05T19:29:33.214 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.210+0000 7f2501716700 1 -- 172.21.15.167:0/3463850574 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:71 5.1b 5:d8000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x564c786afff0 con 0x564c78678e00 2023-10-05T19:29:33.215 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.210+0000 7f2500714700 1 -- 172.21.15.167:0/3463850574 <== osd.2 v2:172.21.15.167:6816/30037 11 ==== osd_op_reply(71 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f24e8008ad0 con 0x564c78678e00 2023-10-05T19:29:33.215 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.210+0000 7f2500714700 1 -- 172.21.15.167:0/3463850574 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:72 5.7 5:e0000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x564c786afff0 con 0x564c786b08f0 2023-10-05T19:29:33.215 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.210+0000 7f2500f15700 1 -- 172.21.15.167:0/3463850574 <== osd.3 v2:172.21.15.167:6808/30036 19 ==== osd_op_reply(72 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f24f8024020 con 0x564c786b08f0 2023-10-05T19:29:33.215 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.210+0000 7f2500f15700 1 -- 172.21.15.167:0/3463850574 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:73 5.17 5:e8000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x564c786afff0 con 0x7f24e8007030 2023-10-05T19:29:33.215 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.210+0000 7f2501716700 1 -- 172.21.15.167:0/3463850574 <== osd.0 v2:172.21.15.118:6800/27690 19 ==== osd_op_reply(73 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f24f002f080 con 0x7f24e8007030 2023-10-05T19:29:33.215 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.210+0000 7f2501716700 1 -- 172.21.15.167:0/3463850574 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:74 5.f 5:f0000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x564c786afff0 con 0x564c786aaa50 2023-10-05T19:29:33.216 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.210+0000 7f2501716700 1 -- 172.21.15.167:0/3463850574 <== osd.1 v2:172.21.15.167:6800/30035 25 ==== osd_op_reply(74 [pgnls start_epoch 32 out=115b] v23'2 uv2 ondisk = 1) v8 ==== 144+0+115 (crc 0 0 0) 0x7f24f002f080 con 0x564c786aaa50 2023-10-05T19:29:33.216 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.210+0000 7f2501716700 1 -- 172.21.15.167:0/3463850574 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:75 5.1f 5:f8000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x564c786afff0 con 0x564c78678e00 2023-10-05T19:29:33.217 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.214+0000 7f2500714700 1 -- 172.21.15.167:0/3463850574 <== osd.2 v2:172.21.15.167:6816/30037 12 ==== osd_op_reply(75 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f24e8008ad0 con 0x564c78678e00 2023-10-05T19:29:33.218 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.214+0000 7f25033e0e40 20 RGWRados::pool_iterate: got zonegroup_info.4f1f719d-9648-417b-b10c-e5d939825419 2023-10-05T19:29:33.218 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.214+0000 7f25033e0e40 20 RGWRados::pool_iterate: got zone_info.9a9df965-6612-45d4-a45f-6e3c889bacc2 2023-10-05T19:29:33.218 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.214+0000 7f25033e0e40 20 RGWRados::pool_iterate: got zone_names.default 2023-10-05T19:29:33.218 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.214+0000 7f25033e0e40 20 RGWRados::pool_iterate: got zonegroups_names.default 2023-10-05T19:29:33.218 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.214+0000 7f25033e0e40 20 rados->read ofs=0 len=0 2023-10-05T19:29:33.218 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.214+0000 7f25033e0e40 1 -- 172.21.15.167:0/3463850574 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:76 5.f 5:f43b6ece:::zone_names.default:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32) v8 -- 0x564c786b3970 con 0x564c786aaa50 2023-10-05T19:29:33.219 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.214+0000 7f2501716700 1 -- 172.21.15.167:0/3463850574 <== osd.1 v2:172.21.15.167:6800/30035 26 ==== osd_op_reply(76 zone_names.default [read 0~46 out=46b] v0'0 uv1 ondisk = 0) v8 ==== 162+0+46 (crc 0 0 0) 0x7f24f002f080 con 0x564c786aaa50 2023-10-05T19:29:33.219 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.214+0000 7f25033e0e40 20 rados_obj.operate() r=0 bl.length=46 2023-10-05T19:29:33.219 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.214+0000 7f25033e0e40 20 rados->read ofs=0 len=0 2023-10-05T19:29:33.220 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.214+0000 7f25033e0e40 1 -- 172.21.15.167:0/3463850574 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:77 5.1d 5:b8374eb4:::zone_info.9a9df965-6612-45d4-a45f-6e3c889bacc2:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32) v8 -- 0x564c786b3d70 con 0x564c786aaa50 2023-10-05T19:29:33.220 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.214+0000 7f2501716700 1 -- 172.21.15.167:0/3463850574 <== osd.1 v2:172.21.15.167:6800/30035 27 ==== osd_op_reply(77 zone_info.9a9df965-6612-45d4-a45f-6e3c889bacc2 [read 0~874 out=874b] v0'0 uv2 ondisk = 0) v8 ==== 190+0+874 (crc 0 0 0) 0x7f24f002f080 con 0x564c786aaa50 2023-10-05T19:29:33.220 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.214+0000 7f25033e0e40 20 rados_obj.operate() r=0 bl.length=874 2023-10-05T19:29:33.220 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.214+0000 7f25033e0e40 20 rados->read ofs=0 len=0 2023-10-05T19:29:33.220 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.214+0000 7f25033e0e40 1 -- 172.21.15.167:0/3463850574 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:78 5.f 5:f4c53578:::zonegroups_names.default:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32) v8 -- 0x564c786b4260 con 0x564c786aaa50 2023-10-05T19:29:33.221 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.214+0000 7f2501716700 1 -- 172.21.15.167:0/3463850574 <== osd.1 v2:172.21.15.167:6800/30035 28 ==== osd_op_reply(78 zonegroups_names.default [read 0~46 out=46b] v0'0 uv2 ondisk = 0) v8 ==== 168+0+46 (crc 0 0 0) 0x7f24f002f080 con 0x564c786aaa50 2023-10-05T19:29:33.221 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.214+0000 7f25033e0e40 20 rados_obj.operate() r=0 bl.length=46 2023-10-05T19:29:33.221 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.214+0000 7f25033e0e40 20 rados->read ofs=0 len=0 2023-10-05T19:29:33.221 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.214+0000 7f25033e0e40 1 -- 172.21.15.167:0/3463850574 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:79 5.c 5:30d299a5:::zonegroup_info.4f1f719d-9648-417b-b10c-e5d939825419:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32) v8 -- 0x564c786b4260 con 0x7f24e8007030 2023-10-05T19:29:33.222 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.218+0000 7f2501716700 1 -- 172.21.15.167:0/3463850574 <== osd.0 v2:172.21.15.118:6800/27690 20 ==== osd_op_reply(79 zonegroup_info.4f1f719d-9648-417b-b10c-e5d939825419 [read 0~362 out=362b] v0'0 uv1 ondisk = 0) v8 ==== 195+0+362 (crc 0 0 0) 0x7f24f002f080 con 0x7f24e8007030 2023-10-05T19:29:33.222 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.218+0000 7f25033e0e40 20 rados_obj.operate() r=0 bl.length=362 2023-10-05T19:29:33.222 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.218+0000 7f25033e0e40 20 rados->read ofs=0 len=0 2023-10-05T19:29:33.222 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.218+0000 7f25033e0e40 1 -- 172.21.15.167:0/3463850574 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:80 5.3 5:c0ad8cbd:::region_map:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32) v8 -- 0x564c786b4360 con 0x7f24e8007030 2023-10-05T19:29:33.223 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.218+0000 7f2501716700 1 -- 172.21.15.167:0/3463850574 <== osd.0 v2:172.21.15.118:6800/27690 21 ==== osd_op_reply(80 region_map [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 ==== 154+0+0 (crc 0 0 0) 0x7f24f0050040 con 0x7f24e8007030 2023-10-05T19:29:33.223 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.218+0000 7f25033e0e40 20 rados_obj.operate() r=-2 bl.length=0 2023-10-05T19:29:33.223 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.218+0000 7f25033e0e40 20 zone default found 2023-10-05T19:29:33.223 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.218+0000 7f25033e0e40 4 Realm: () 2023-10-05T19:29:33.224 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.218+0000 7f25033e0e40 4 ZoneGroup: default (4f1f719d-9648-417b-b10c-e5d939825419) 2023-10-05T19:29:33.224 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.218+0000 7f25033e0e40 4 Zone: default (9a9df965-6612-45d4-a45f-6e3c889bacc2) 2023-10-05T19:29:33.224 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.218+0000 7f25033e0e40 10 cannot find current period zonegroup using local zonegroup configuration 2023-10-05T19:29:33.224 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.218+0000 7f25033e0e40 20 zonegroup default 2023-10-05T19:29:33.224 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.218+0000 7f25033e0e40 20 rados->read ofs=0 len=0 2023-10-05T19:29:33.224 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.218+0000 7f25033e0e40 1 -- 172.21.15.167:0/3463850574 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:81 5.3 5:c52100b6:::period_config.default:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32) v8 -- 0x564c786b4360 con 0x7f24e8007030 2023-10-05T19:29:33.224 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.218+0000 7f2501716700 1 -- 172.21.15.167:0/3463850574 <== osd.0 v2:172.21.15.118:6800/27690 22 ==== osd_op_reply(81 period_config.default [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 ==== 165+0+0 (crc 0 0 0) 0x7f24f0050040 con 0x7f24e8007030 2023-10-05T19:29:33.224 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.218+0000 7f25033e0e40 20 rados_obj.operate() r=-2 bl.length=0 2023-10-05T19:29:33.224 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.218+0000 7f25033e0e40 20 rados->read ofs=0 len=0 2023-10-05T19:29:33.225 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.218+0000 7f25033e0e40 1 -- 172.21.15.167:0/3463850574 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:82 6.2 6:4347d321:::bucket.sync-source-hints.:head [call version.read in=11b,read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32) v8 -- 0x564c786b5ce0 con 0x564c786b08f0 2023-10-05T19:29:33.226 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.218+0000 7f2500f15700 1 -- 172.21.15.167:0/3463850574 <== osd.3 v2:172.21.15.167:6808/30036 20 ==== osd_op_reply(82 bucket.sync-source-hints. [call,read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 ==== 211+0+0 (crc 0 0 0) 0x7f24f8024020 con 0x564c786b08f0 2023-10-05T19:29:33.226 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.218+0000 7f25033e0e40 20 rados_obj.operate() r=-2 bl.length=0 2023-10-05T19:29:33.226 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.218+0000 7f25033e0e40 20 rados->read ofs=0 len=0 2023-10-05T19:29:33.226 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.218+0000 7f25033e0e40 1 -- 172.21.15.167:0/3463850574 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:83 6.b 6:d467b91b:::bucket.sync-target-hints.:head [call version.read in=11b,read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32) v8 -- 0x564c786b6090 con 0x564c786b08f0 2023-10-05T19:29:33.226 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.222+0000 7f2500f15700 1 -- 172.21.15.167:0/3463850574 <== osd.3 v2:172.21.15.167:6808/30036 21 ==== osd_op_reply(83 bucket.sync-target-hints. [call,read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 ==== 211+0+0 (crc 0 0 0) 0x7f24f8024020 con 0x564c786b08f0 2023-10-05T19:29:33.226 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.222+0000 7f25033e0e40 20 rados_obj.operate() r=-2 bl.length=0 2023-10-05T19:29:33.226 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.222+0000 7f25033e0e40 20 started sync module instance, tier type = 2023-10-05T19:29:33.226 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.222+0000 7f25033e0e40 20 started zone id=9a9df965-6612-45d4-a45f-6e3c889bacc2 (name=default) with tier type = 2023-10-05T19:29:33.227 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.222+0000 7f25033e0e40 1 -- 172.21.15.167:0/3463850574 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:84 7.1f 7:f95f44c2:::notify.0:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e32) v8 -- 0x564c786b53a0 con 0x564c78678e00 2023-10-05T19:29:33.228 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.222+0000 7f2500714700 1 -- 172.21.15.167:0/3463850574 <== osd.2 v2:172.21.15.167:6816/30037 13 ==== osd_op_reply(84 notify.0 [create] v32'222 uv222 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f24e8008ad0 con 0x564c78678e00 2023-10-05T19:29:33.228 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.222+0000 7f25033e0e40 1 -- 172.21.15.167:0/3463850574 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:85 7.1f 7:f95f44c2:::notify.0:head [watch watch cookie 94886437801648] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e32) v8 -- 0x564c786b6440 con 0x564c78678e00 2023-10-05T19:29:33.228 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.222+0000 7f25033e0e40 1 -- 172.21.15.167:0/3463850574 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:86 7.0 7:05bf5b68:::notify.1:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e32) v8 -- 0x564c786b6db0 con 0x564c786aaa50 2023-10-05T19:29:33.231 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.226+0000 7f2501716700 1 -- 172.21.15.167:0/3463850574 <== osd.1 v2:172.21.15.167:6800/30035 29 ==== osd_op_reply(86 notify.1 [create] v32'222 uv222 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f24f0050040 con 0x564c786aaa50 2023-10-05T19:29:33.231 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.226+0000 7f25033e0e40 1 -- 172.21.15.167:0/3463850574 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:87 7.0 7:05bf5b68:::notify.1:head [watch watch cookie 94886437808256] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e32) v8 -- 0x564c786b76a0 con 0x564c786aaa50 2023-10-05T19:29:33.231 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.226+0000 7f2500714700 1 -- 172.21.15.167:0/3463850574 <== osd.2 v2:172.21.15.167:6816/30037 14 ==== osd_op_reply(85 notify.0 [watch watch cookie 94886437801648] v32'223 uv222 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f24e8008ad0 con 0x564c78678e00 2023-10-05T19:29:33.231 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.226+0000 7f25033e0e40 1 -- 172.21.15.167:0/3463850574 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:88 7.15 7:a93a5511:::notify.2:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e32) v8 -- 0x564c786b7f40 con 0x564c786b08f0 2023-10-05T19:29:33.234 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.230+0000 7f2501716700 1 -- 172.21.15.167:0/3463850574 <== osd.1 v2:172.21.15.167:6800/30035 30 ==== osd_op_reply(87 notify.1 [watch watch cookie 94886437808256] v32'223 uv222 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f24f0050040 con 0x564c786aaa50 2023-10-05T19:29:33.234 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.230+0000 7f2500f15700 1 -- 172.21.15.167:0/3463850574 <== osd.3 v2:172.21.15.167:6808/30036 22 ==== osd_op_reply(88 notify.2 [create] v32'222 uv222 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f24f8024020 con 0x564c786b08f0 2023-10-05T19:29:33.235 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.230+0000 7f25033e0e40 1 -- 172.21.15.167:0/3463850574 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:89 7.15 7:a93a5511:::notify.2:head [watch watch cookie 94886437810608] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e32) v8 -- 0x564c786b58c0 con 0x564c786b08f0 2023-10-05T19:29:33.235 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.230+0000 7f25033e0e40 1 -- 172.21.15.167:0/3463850574 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:90 7.e 7:7759931f:::notify.3:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e32) v8 -- 0x564c786b87e0 con 0x564c78678e00 2023-10-05T19:29:33.237 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.234+0000 7f2500714700 1 -- 172.21.15.167:0/3463850574 <== osd.2 v2:172.21.15.167:6816/30037 15 ==== osd_op_reply(90 notify.3 [create] v32'222 uv222 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f24e8008ad0 con 0x564c78678e00 2023-10-05T19:29:33.237 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.234+0000 7f2500f15700 1 -- 172.21.15.167:0/3463850574 <== osd.3 v2:172.21.15.167:6808/30036 23 ==== osd_op_reply(89 notify.2 [watch watch cookie 94886437810608] v32'223 uv222 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f24f8024020 con 0x564c786b08f0 2023-10-05T19:29:33.237 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.234+0000 7f25033e0e40 1 -- 172.21.15.167:0/3463850574 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:91 7.e 7:7759931f:::notify.3:head [watch watch cookie 94886437815024] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e32) v8 -- 0x564c786b9080 con 0x564c78678e00 2023-10-05T19:29:33.238 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.234+0000 7f25033e0e40 1 -- 172.21.15.167:0/3463850574 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:92 7.d 7:b4812045:::notify.4:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e32) v8 -- 0x564c786b9a00 con 0x564c786aaa50 2023-10-05T19:29:33.240 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.234+0000 7f2500714700 1 -- 172.21.15.167:0/3463850574 <== osd.2 v2:172.21.15.167:6816/30037 16 ==== osd_op_reply(91 notify.3 [watch watch cookie 94886437815024] v32'223 uv222 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f24e8008ad0 con 0x564c78678e00 2023-10-05T19:29:33.240 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.234+0000 7f2501716700 1 -- 172.21.15.167:0/3463850574 <== osd.1 v2:172.21.15.167:6800/30035 31 ==== osd_op_reply(92 notify.4 [create] v32'222 uv222 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f24f0050040 con 0x564c786aaa50 2023-10-05T19:29:33.240 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.234+0000 7f25033e0e40 1 -- 172.21.15.167:0/3463850574 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:93 7.d 7:b4812045:::notify.4:head [watch watch cookie 94886437817232] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e32) v8 -- 0x564c786b9a00 con 0x564c786aaa50 2023-10-05T19:29:33.241 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.238+0000 7f25033e0e40 1 -- 172.21.15.167:0/3463850574 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:94 7.3 7:c609908c:::notify.5:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e32) v8 -- 0x564c786ba410 con 0x7f24e8007030 2023-10-05T19:29:33.242 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.238+0000 7f2501716700 1 -- 172.21.15.167:0/3463850574 <== osd.1 v2:172.21.15.167:6800/30035 32 ==== osd_op_reply(93 notify.4 [watch watch cookie 94886437817232] v32'223 uv222 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f24f0050040 con 0x564c786aaa50 2023-10-05T19:29:33.243 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.238+0000 7f2501716700 1 -- 172.21.15.167:0/3463850574 <== osd.0 v2:172.21.15.118:6800/27690 23 ==== osd_op_reply(94 notify.5 [create] v32'222 uv222 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f24f0050040 con 0x7f24e8007030 2023-10-05T19:29:33.243 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.238+0000 7f25033e0e40 1 -- 172.21.15.167:0/3463850574 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:95 7.3 7:c609908c:::notify.5:head [watch watch cookie 94886437820224] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e32) v8 -- 0x564c786ba410 con 0x7f24e8007030 2023-10-05T19:29:33.243 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.238+0000 7f25033e0e40 1 -- 172.21.15.167:0/3463850574 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:96 7.14 7:2b04a3e9:::notify.6:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e32) v8 -- 0x564c786bae20 con 0x7f24e8007030 2023-10-05T19:29:33.245 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.242+0000 7f2501716700 1 -- 172.21.15.167:0/3463850574 <== osd.0 v2:172.21.15.118:6800/27690 24 ==== osd_op_reply(95 notify.5 [watch watch cookie 94886437820224] v32'223 uv222 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f24f0050040 con 0x7f24e8007030 2023-10-05T19:29:33.246 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.242+0000 7f2501716700 1 -- 172.21.15.167:0/3463850574 <== osd.0 v2:172.21.15.118:6800/27690 25 ==== osd_op_reply(96 notify.6 [create] v32'222 uv222 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f24f0050040 con 0x7f24e8007030 2023-10-05T19:29:33.246 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.242+0000 7f25033e0e40 1 -- 172.21.15.167:0/3463850574 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:97 7.14 7:2b04a3e9:::notify.6:head [watch watch cookie 94886437822512] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e32) v8 -- 0x564c786bae20 con 0x7f24e8007030 2023-10-05T19:29:33.246 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.242+0000 7f25033e0e40 1 -- 172.21.15.167:0/3463850574 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:98 7.9 7:93e5b521:::notify.7:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e32) v8 -- 0x564c786bb6c0 con 0x7f24e8007030 2023-10-05T19:29:33.249 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.246+0000 7f2501716700 1 -- 172.21.15.167:0/3463850574 <== osd.0 v2:172.21.15.118:6800/27690 26 ==== osd_op_reply(98 notify.7 [create] v32'222 uv222 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f24f0050040 con 0x7f24e8007030 2023-10-05T19:29:33.249 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.246+0000 7f25033e0e40 1 -- 172.21.15.167:0/3463850574 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:99 7.9 7:93e5b521:::notify.7:head [watch watch cookie 94886437826640] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e32) v8 -- 0x564c786bba50 con 0x7f24e8007030 2023-10-05T19:29:33.249 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.246+0000 7f25033e0e40 20 add_watcher() i=0 2023-10-05T19:29:33.249 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.246+0000 7f25033e0e40 20 add_watcher() i=1 2023-10-05T19:29:33.249 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.246+0000 7f25033e0e40 20 add_watcher() i=2 2023-10-05T19:29:33.249 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.246+0000 7f25033e0e40 20 add_watcher() i=3 2023-10-05T19:29:33.250 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.246+0000 7f25033e0e40 20 add_watcher() i=4 2023-10-05T19:29:33.250 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.246+0000 7f25033e0e40 20 add_watcher() i=5 2023-10-05T19:29:33.250 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.246+0000 7f2501716700 1 -- 172.21.15.167:0/3463850574 <== osd.0 v2:172.21.15.118:6800/27690 27 ==== osd_op_reply(97 notify.6 [watch watch cookie 94886437822512] v32'223 uv222 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f24f0050040 con 0x7f24e8007030 2023-10-05T19:29:33.250 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.246+0000 7f25033e0e40 20 add_watcher() i=6 2023-10-05T19:29:33.251 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.246+0000 7f2501716700 1 -- 172.21.15.167:0/3463850574 <== osd.0 v2:172.21.15.118:6800/27690 28 ==== osd_op_reply(99 notify.7 [watch watch cookie 94886437826640] v32'223 uv222 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f24f0050040 con 0x7f24e8007030 2023-10-05T19:29:33.251 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.246+0000 7f25033e0e40 20 add_watcher() i=7 2023-10-05T19:29:33.251 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.246+0000 7f25033e0e40 2 all 8 watchers are set, enabling cache 2023-10-05T19:29:33.251 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.246+0000 7f25033e0e40 1 -- 172.21.15.167:0/3463850574 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:100 6.4 6:22d26bf9:::data_loggenerations_metadata:head [call version.check_conds in=74b,call version.read in=11b,read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32) v8 -- 0x564c786ba930 con 0x564c786aaa50 2023-10-05T19:29:33.252 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.246+0000 7f2501716700 1 -- 172.21.15.167:0/3463850574 <== osd.1 v2:172.21.15.167:6800/30035 33 ==== osd_op_reply(100 data_loggenerations_metadata [call,call out=48b,read 0~28 out=28b] v0'0 uv1 ondisk = 0) v8 ==== 256+0+76 (crc 0 0 0) 0x7f24f0050040 con 0x564c786aaa50 2023-10-05T19:29:33.252 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.246+0000 7f25033e0e40 1 -- 172.21.15.167:0/3463850574 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:101 6.4 6:22d26bf9:::data_loggenerations_metadata:head [watch watch cookie 94886437827872] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e32) v8 -- 0x564c786bbf70 con 0x564c786aaa50 2023-10-05T19:29:33.254 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.250+0000 7f2501716700 1 -- 172.21.15.167:0/3463850574 <== osd.1 v2:172.21.15.167:6800/30035 34 ==== osd_op_reply(101 data_loggenerations_metadata [watch watch cookie 94886437827872] v32'221 uv1 ondisk = 0) v8 ==== 172+0+0 (crc 0 0 0) 0x7f24f0050040 con 0x564c786aaa50 2023-10-05T19:29:33.254 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.250+0000 7f24d27e4700 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2023-10-05T19:29:33.254 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.250+0000 7f25033e0e40 20 check_secure_mon_conn(): auth registy supported: methods=[2] modes=[2,1] 2023-10-05T19:29:33.255 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.250+0000 7f25033e0e40 20 check_secure_mon_conn(): mode 1 is insecure 2023-10-05T19:29:33.256 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.250+0000 7f25033e0e40 1 Processor -- start 2023-10-05T19:29:33.256 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.250+0000 7f25033e0e40 1 -- start start 2023-10-05T19:29:33.256 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.250+0000 7f25033e0e40 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x564c786c8da0 0x564c786c9170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-05T19:29:33.256 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.250+0000 7f25033e0e40 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x564c786c96b0 con 0x564c786c8da0 2023-10-05T19:29:33.256 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.250+0000 7f2500714700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x564c786c8da0 0x564c786c9170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-05T19:29:33.256 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.250+0000 7f2500714700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x564c786c8da0 0x564c786c9170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.118:3300/0 says I am v2:172.21.15.167:46174/0 (socket says 172.21.15.167:46174) 2023-10-05T19:29:33.256 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.250+0000 7f2500714700 1 -- 172.21.15.167:0/1117195991 learned_addr learned my addr 172.21.15.167:0/1117195991 (peer_addr_for_me v2:172.21.15.167:0/0) 2023-10-05T19:29:33.257 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.254+0000 7f2500714700 1 -- 172.21.15.167:0/1117195991 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x564c786c97f0 con 0x564c786c8da0 2023-10-05T19:29:33.257 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.254+0000 7f2500714700 1 --2- 172.21.15.167:0/1117195991 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x564c786c8da0 0x564c786c9170 secure :-1 s=READY pgs=582 cs=0 l=1 rev1=1 crypto rx=0x564c78679e80 tx=0x7f24e8012000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5bb113ba1e498810 server_cookie=0 in_seq=0 out_seq=0 2023-10-05T19:29:33.257 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.254+0000 7f24d1fe3700 1 -- 172.21.15.167:0/1117195991 <== mon.0 v2:172.21.15.118:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f24e8012c30 con 0x564c786c8da0 2023-10-05T19:29:33.257 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.254+0000 7f24d1fe3700 1 -- 172.21.15.167:0/1117195991 <== mon.0 v2:172.21.15.118:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f24e8012e40 con 0x564c786c8da0 2023-10-05T19:29:33.258 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.254+0000 7f24d1fe3700 1 -- 172.21.15.167:0/1117195991 <== mon.0 v2:172.21.15.118:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f24e802a8a0 con 0x564c786c8da0 2023-10-05T19:29:33.259 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.254+0000 7f25033e0e40 1 -- 172.21.15.167:0/1117195991 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x564c786c8da0 msgr2=0x564c786c9170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-05T19:29:33.259 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.254+0000 7f25033e0e40 1 --2- 172.21.15.167:0/1117195991 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x564c786c8da0 0x564c786c9170 secure :-1 s=READY pgs=582 cs=0 l=1 rev1=1 crypto rx=0x564c78679e80 tx=0x7f24e8012000 comp rx=0 tx=0).stop 2023-10-05T19:29:33.259 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.254+0000 7f25033e0e40 1 -- 172.21.15.167:0/1117195991 shutdown_connections 2023-10-05T19:29:33.259 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.254+0000 7f25033e0e40 1 --2- 172.21.15.167:0/1117195991 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x564c786c8da0 0x564c786c9170 unknown :-1 s=CLOSED pgs=582 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-05T19:29:33.259 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.254+0000 7f25033e0e40 1 -- 172.21.15.167:0/1117195991 >> 172.21.15.167:0/1117195991 conn(0x564c786c4160 msgr2=0x564c786c2310 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-05T19:29:33.259 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.254+0000 7f25033e0e40 1 -- 172.21.15.167:0/1117195991 shutdown_connections 2023-10-05T19:29:33.259 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.254+0000 7f25033e0e40 1 -- 172.21.15.167:0/1117195991 wait complete. 2023-10-05T19:29:33.260 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.254+0000 7f25033e0e40 1 Processor -- start 2023-10-05T19:29:33.260 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.254+0000 7f25033e0e40 1 -- start start 2023-10-05T19:29:33.261 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.254+0000 7f25033e0e40 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x564c786c8da0 0x564c786ce810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-05T19:29:33.261 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.254+0000 7f25033e0e40 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x564c786cf810 con 0x564c786c8da0 2023-10-05T19:29:33.261 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.254+0000 7f2500714700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x564c786c8da0 0x564c786ce810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-05T19:29:33.261 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.254+0000 7f2500714700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x564c786c8da0 0x564c786ce810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.118:3300/0 says I am v2:172.21.15.167:46176/0 (socket says 172.21.15.167:46176) 2023-10-05T19:29:33.261 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.254+0000 7f2500714700 1 -- 172.21.15.167:0/2482662757 learned_addr learned my addr 172.21.15.167:0/2482662757 (peer_addr_for_me v2:172.21.15.167:0/0) 2023-10-05T19:29:33.262 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.258+0000 7f2500714700 1 -- 172.21.15.167:0/2482662757 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x564c78679a10 con 0x564c786c8da0 2023-10-05T19:29:33.262 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.258+0000 7f2500714700 1 --2- 172.21.15.167:0/2482662757 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x564c786c8da0 0x564c786ce810 secure :-1 s=READY pgs=583 cs=0 l=1 rev1=1 crypto rx=0x7f24e8012fd0 tx=0x7f24e802f000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-05T19:29:33.262 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.258+0000 7f2443fff700 1 -- 172.21.15.167:0/2482662757 <== mon.0 v2:172.21.15.118:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f24e8009890 con 0x564c786c8da0 2023-10-05T19:29:33.262 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.258+0000 7f2443fff700 1 -- 172.21.15.167:0/2482662757 <== mon.0 v2:172.21.15.118:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f24e80099f0 con 0x564c786c8da0 2023-10-05T19:29:33.262 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.258+0000 7f25033e0e40 1 -- 172.21.15.167:0/2482662757 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x564c786cee10 con 0x564c786c8da0 2023-10-05T19:29:33.263 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.258+0000 7f2443fff700 1 -- 172.21.15.167:0/2482662757 <== mon.0 v2:172.21.15.118:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f24e8009ba0 con 0x564c786c8da0 2023-10-05T19:29:33.263 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.258+0000 7f25033e0e40 1 -- 172.21.15.167:0/2482662757 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x564c786cf250 con 0x564c786c8da0 2023-10-05T19:29:33.263 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.258+0000 7f2443fff700 1 -- 172.21.15.167:0/2482662757 <== mon.0 v2:172.21.15.118:3300/0 4 ==== mgrmap(e 5) v1 ==== 40329+0+0 (secure 0 0 0) 0x7f24e8009d00 con 0x564c786c8da0 2023-10-05T19:29:33.264 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.258+0000 7f2443fff700 1 --2- 172.21.15.167:0/2482662757 >> [v2:172.21.15.167:6824/29889,v1:172.21.15.167:6825/29889] conn(0x7f2448030c00 0x7f24480330b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-05T19:29:33.264 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.258+0000 7f2443fff700 1 -- 172.21.15.167:0/2482662757 <== mon.0 v2:172.21.15.118:3300/0 5 ==== osd_map(32..32 src has 1..32) v4 ==== 6260+0+0 (secure 0 0 0) 0x7f24e802f070 con 0x564c786c8da0 2023-10-05T19:29:33.264 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.258+0000 7f2501716700 1 --2- 172.21.15.167:0/2482662757 >> [v2:172.21.15.167:6824/29889,v1:172.21.15.167:6825/29889] conn(0x7f2448030c00 0x7f24480330b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-05T19:29:33.264 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.258+0000 7f25033e0e40 5 note: GC not initialized 2023-10-05T19:29:33.265 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.262+0000 7f2501716700 1 --2- 172.21.15.167:0/2482662757 >> [v2:172.21.15.167:6824/29889,v1:172.21.15.167:6825/29889] conn(0x7f2448030c00 0x7f24480330b0 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x564c786b9050 tx=0x7f24f0000f90 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-05T19:29:33.266 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.262+0000 7f24417fa700 20 reqs_thread_entry: start 2023-10-05T19:29:33.266 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.262+0000 7f25033e0e40 20 init_complete bucket index max shards: 11 2023-10-05T19:29:33.266 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.262+0000 7f243affd700 20 reqs_thread_entry: start 2023-10-05T19:29:33.267 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.262+0000 7f25033e0e40 10 rgw notify: Started notification manager with: 1 workers 2023-10-05T19:29:33.267 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.262+0000 7f24397fa700 20 rgw notify: INFO: next queues processing will happen at: Thu Oct 5 19:30:03 2023 2023-10-05T19:29:33.267 INFO:teuthology.orchestra.run.smithi167.stderr: 2023-10-05T19:29:33.267 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.262+0000 7f2438ff9700 20 reqs_thread_entry: start 2023-10-05T19:29:33.268 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.262+0000 7f25033e0e40 1 -- 172.21.15.167:0/3463850574 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:102 6.e 6:74cd889a:usage::usage.0:head [call rgw.user_usage_log_read in=60b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32) v8 -- 0x564c787378d0 con 0x564c786aaa50 2023-10-05T19:29:33.268 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.262+0000 7f2501716700 1 -- 172.21.15.167:0/3463850574 <== osd.1 v2:172.21.15.167:6800/30035 35 ==== osd_op_reply(102 usage.0 [call] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 ==== 151+0+0 (crc 0 0 0) 0x7f24f0050040 con 0x564c786aaa50 2023-10-05T19:29:33.268 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.262+0000 7f25033e0e40 1 -- 172.21.15.167:0/3463850574 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:103 6.3 6:c0e78a64:usage::usage.1:head [call rgw.user_usage_log_read in=60b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32) v8 -- 0x564c787378d0 con 0x564c78678e00 2023-10-05T19:29:33.269 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.266+0000 7f2500714700 1 -- 172.21.15.167:0/3463850574 <== osd.2 v2:172.21.15.167:6816/30037 17 ==== osd_op_reply(103 usage.1 [call] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 ==== 151+0+0 (crc 0 0 0) 0x7f24e8032040 con 0x564c78678e00 2023-10-05T19:29:33.269 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.266+0000 7f25033e0e40 1 -- 172.21.15.167:0/3463850574 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:104 6.d 6:b564efa4:usage::usage.2:head [call rgw.user_usage_log_read in=60b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32) v8 -- 0x564c78737c80 con 0x564c786aaa50 2023-10-05T19:29:33.269 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.266+0000 7f2501716700 1 -- 172.21.15.167:0/3463850574 <== osd.1 v2:172.21.15.167:6800/30035 36 ==== osd_op_reply(104 usage.2 [call] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 ==== 151+0+0 (crc 0 0 0) 0x7f24f0050040 con 0x564c786aaa50 2023-10-05T19:29:33.269 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.266+0000 7f25033e0e40 1 -- 172.21.15.167:0/3463850574 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:105 6.2 6:42d64401:usage::usage.3:head [call rgw.user_usage_log_read in=60b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32) v8 -- 0x564c78737c80 con 0x564c786b08f0 2023-10-05T19:29:33.270 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.266+0000 7f2500f15700 1 -- 172.21.15.167:0/3463850574 <== osd.3 v2:172.21.15.167:6808/30036 24 ==== osd_op_reply(105 usage.3 [call] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 ==== 151+0+0 (crc 0 0 0) 0x7f24f8024020 con 0x564c786b08f0 2023-10-05T19:29:33.270 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.266+0000 7f25033e0e40 1 -- 172.21.15.167:0/3463850574 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:106 6.7 6:e09b6f01:usage::usage.4:head [call rgw.user_usage_log_read in=60b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32) v8 -- 0x564c78738030 con 0x564c786b08f0 2023-10-05T19:29:33.271 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.266+0000 7f2500f15700 1 -- 172.21.15.167:0/3463850574 <== osd.3 v2:172.21.15.167:6808/30036 25 ==== osd_op_reply(106 usage.4 [call] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 ==== 151+0+0 (crc 0 0 0) 0x7f24f8024020 con 0x564c786b08f0 2023-10-05T19:29:33.271 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.266+0000 7f25033e0e40 1 -- 172.21.15.167:0/3463850574 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:107 6.6 6:672810f1:usage::usage.5:head [call rgw.user_usage_log_read in=60b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32) v8 -- 0x564c787383e0 con 0x564c786b08f0 2023-10-05T19:29:33.271 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.266+0000 7f2500f15700 1 -- 172.21.15.167:0/3463850574 <== osd.3 v2:172.21.15.167:6808/30036 26 ==== osd_op_reply(107 usage.5 [call] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 ==== 151+0+0 (crc 0 0 0) 0x7f24f8024020 con 0x564c786b08f0 2023-10-05T19:29:33.272 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.266+0000 7f25033e0e40 1 -- 172.21.15.167:0/3463850574 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:108 6.4 6:25a86931:usage::usage.6:head [call rgw.user_usage_log_read in=60b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32) v8 -- 0x564c787383e0 con 0x564c786aaa50 2023-10-05T19:29:33.272 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.266+0000 7f2501716700 1 -- 172.21.15.167:0/3463850574 <== osd.1 v2:172.21.15.167:6800/30035 37 ==== osd_op_reply(108 usage.6 [call] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 ==== 151+0+0 (crc 0 0 0) 0x7f24f0050040 con 0x564c786aaa50 2023-10-05T19:29:33.272 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.266+0000 7f25033e0e40 1 -- 172.21.15.167:0/3463850574 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:109 6.17 6:ebbe7322:usage::usage.7:head [call rgw.user_usage_log_read in=60b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32) v8 -- 0x564c787383e0 con 0x564c786aaa50 2023-10-05T19:29:33.272 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.266+0000 7f2501716700 1 -- 172.21.15.167:0/3463850574 <== osd.1 v2:172.21.15.167:6800/30035 38 ==== osd_op_reply(109 usage.7 [call] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 ==== 151+0+0 (crc 0 0 0) 0x7f24f0050040 con 0x564c786aaa50 2023-10-05T19:29:33.272 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.266+0000 7f25033e0e40 1 -- 172.21.15.167:0/3463850574 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:110 6.6 6:607e1922:usage::usage.8:head [call rgw.user_usage_log_read in=60b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32) v8 -- 0x564c787383e0 con 0x564c786b08f0 2023-10-05T19:29:33.273 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.270+0000 7f2500f15700 1 -- 172.21.15.167:0/3463850574 <== osd.3 v2:172.21.15.167:6808/30036 27 ==== osd_op_reply(110 usage.8 [call] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 ==== 151+0+0 (crc 0 0 0) 0x7f24f8024020 con 0x564c786b08f0 2023-10-05T19:29:33.273 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.270+0000 7f25033e0e40 1 -- 172.21.15.167:0/3463850574 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:111 6.b 6:d7809252:usage::usage.9:head [call rgw.user_usage_log_read in=60b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32) v8 -- 0x564c787383e0 con 0x564c786b08f0 2023-10-05T19:29:33.273 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.270+0000 7f2500f15700 1 -- 172.21.15.167:0/3463850574 <== osd.3 v2:172.21.15.167:6808/30036 28 ==== osd_op_reply(111 usage.9 [call] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 ==== 151+0+0 (crc 0 0 0) 0x7f24f8024020 con 0x564c786b08f0 2023-10-05T19:29:33.274 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.270+0000 7f25033e0e40 1 -- 172.21.15.167:0/3463850574 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:112 6.13 6:cea599e4:usage::usage.10:head [call rgw.user_usage_log_read in=60b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32) v8 -- 0x564c787383e0 con 0x7f24e8007030 2023-10-05T19:29:33.275 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.270+0000 7f2501716700 1 -- 172.21.15.167:0/3463850574 <== osd.0 v2:172.21.15.118:6800/27690 29 ==== osd_op_reply(112 usage.10 [call out=1268b] v0'0 uv30 ondisk = 0) v8 ==== 152+0+1268 (crc 0 0 0) 0x7f24f0050040 con 0x7f24e8007030 2023-10-05T19:29:33.275 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.270+0000 7f25033e0e40 1 -- 172.21.15.167:0/3463850574 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:113 6.4 6:21f845ae:usage::usage.11:head [call rgw.user_usage_log_read in=60b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32) v8 -- 0x564c787398e0 con 0x564c786aaa50 2023-10-05T19:29:33.275 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.270+0000 7f2501716700 1 -- 172.21.15.167:0/3463850574 <== osd.1 v2:172.21.15.167:6800/30035 39 ==== osd_op_reply(113 usage.11 [call] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 ==== 152+0+0 (crc 0 0 0) 0x7f24f0050040 con 0x564c786aaa50 2023-10-05T19:29:33.276 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.270+0000 7f25033e0e40 1 -- 172.21.15.167:0/3463850574 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:114 6.f 6:f0386354:usage::usage.12:head [call rgw.user_usage_log_read in=60b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32) v8 -- 0x564c787398e0 con 0x564c78678e00 2023-10-05T19:29:33.276 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.270+0000 7f2500714700 1 -- 172.21.15.167:0/3463850574 <== osd.2 v2:172.21.15.167:6816/30037 18 ==== osd_op_reply(114 usage.12 [call] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 ==== 152+0+0 (crc 0 0 0) 0x7f24e8032040 con 0x564c78678e00 2023-10-05T19:29:33.276 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.270+0000 7f25033e0e40 1 -- 172.21.15.167:0/3463850574 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:115 6.c 6:3735eec4:usage::usage.13:head [call rgw.user_usage_log_read in=60b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32) v8 -- 0x564c787398e0 con 0x564c786b08f0 2023-10-05T19:29:33.277 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.274+0000 7f2500f15700 1 -- 172.21.15.167:0/3463850574 <== osd.3 v2:172.21.15.167:6808/30036 29 ==== osd_op_reply(115 usage.13 [call] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 ==== 152+0+0 (crc 0 0 0) 0x7f24f8024020 con 0x564c786b08f0 2023-10-05T19:29:33.277 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.274+0000 7f25033e0e40 1 -- 172.21.15.167:0/3463850574 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:116 6.18 6:1f520d0a:usage::usage.14:head [call rgw.user_usage_log_read in=60b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32) v8 -- 0x564c787398e0 con 0x7f24e8007030 2023-10-05T19:29:33.278 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.274+0000 7f2501716700 1 -- 172.21.15.167:0/3463850574 <== osd.0 v2:172.21.15.118:6800/27690 30 ==== osd_op_reply(116 usage.14 [call] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 ==== 152+0+0 (crc 0 0 0) 0x7f24f0050040 con 0x7f24e8007030 2023-10-05T19:29:33.278 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.274+0000 7f25033e0e40 1 -- 172.21.15.167:0/3463850574 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:117 6.0 6:029cee12:usage::usage.15:head [call rgw.user_usage_log_read in=60b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32) v8 -- 0x564c787398e0 con 0x7f24e8007030 2023-10-05T19:29:33.278 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.274+0000 7f2501716700 1 -- 172.21.15.167:0/3463850574 <== osd.0 v2:172.21.15.118:6800/27690 31 ==== osd_op_reply(117 usage.15 [call] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 ==== 152+0+0 (crc 0 0 0) 0x7f24f0050040 con 0x7f24e8007030 2023-10-05T19:29:33.279 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.274+0000 7f25033e0e40 1 -- 172.21.15.167:0/3463850574 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:118 6.1e 6:7f3e030e:usage::usage.16:head [call rgw.user_usage_log_read in=60b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32) v8 -- 0x564c787398e0 con 0x564c786b08f0 2023-10-05T19:29:33.279 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.274+0000 7f2500f15700 1 -- 172.21.15.167:0/3463850574 <== osd.3 v2:172.21.15.167:6808/30036 30 ==== osd_op_reply(118 usage.16 [call] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 ==== 152+0+0 (crc 0 0 0) 0x7f24f8024020 con 0x564c786b08f0 2023-10-05T19:29:33.279 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.274+0000 7f25033e0e40 1 -- 172.21.15.167:0/3463850574 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:119 6.10 6:0fbdcb32:usage::usage.17:head [call rgw.user_usage_log_read in=60b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32) v8 -- 0x564c787398e0 con 0x7f24e8007030 2023-10-05T19:29:33.280 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.274+0000 7f2501716700 1 -- 172.21.15.167:0/3463850574 <== osd.0 v2:172.21.15.118:6800/27690 32 ==== osd_op_reply(119 usage.17 [call out=163b] v0'0 uv5 ondisk = 0) v8 ==== 152+0+163 (crc 0 0 0) 0x7f24f0050040 con 0x7f24e8007030 2023-10-05T19:29:33.280 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.274+0000 7f25033e0e40 1 -- 172.21.15.167:0/3463850574 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:120 6.19 6:993b03e9:usage::usage.18:head [call rgw.user_usage_log_read in=60b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32) v8 -- 0x564c787398e0 con 0x564c786aaa50 2023-10-05T19:29:33.281 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.278+0000 7f2501716700 1 -- 172.21.15.167:0/3463850574 <== osd.1 v2:172.21.15.167:6800/30035 40 ==== osd_op_reply(120 usage.18 [call] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 ==== 152+0+0 (crc 0 0 0) 0x7f24f0050040 con 0x564c786aaa50 2023-10-05T19:29:33.281 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.278+0000 7f25033e0e40 1 -- 172.21.15.167:0/3463850574 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:121 6.1a 6:58fcd270:usage::usage.19:head [call rgw.user_usage_log_read in=60b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32) v8 -- 0x564c787398e0 con 0x564c786b08f0 2023-10-05T19:29:33.281 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.278+0000 7f2500f15700 1 -- 172.21.15.167:0/3463850574 <== osd.3 v2:172.21.15.167:6808/30036 31 ==== osd_op_reply(121 usage.19 [call] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 ==== 152+0+0 (crc 0 0 0) 0x7f24f8024020 con 0x564c786b08f0 2023-10-05T19:29:33.282 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.278+0000 7f25033e0e40 1 -- 172.21.15.167:0/3463850574 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:122 6.1f 6:fcd63d49:usage::usage.20:head [call rgw.user_usage_log_read in=60b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32) v8 -- 0x564c787398e0 con 0x7f24e8007030 2023-10-05T19:29:33.282 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.278+0000 7f2501716700 1 -- 172.21.15.167:0/3463850574 <== osd.0 v2:172.21.15.118:6800/27690 33 ==== osd_op_reply(122 usage.20 [call] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 ==== 152+0+0 (crc 0 0 0) 0x7f24f0050040 con 0x7f24e8007030 2023-10-05T19:29:33.282 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.278+0000 7f25033e0e40 1 -- 172.21.15.167:0/3463850574 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:123 6.5 6:a2c1a412:usage::usage.21:head [call rgw.user_usage_log_read in=60b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32) v8 -- 0x564c787398e0 con 0x564c78678e00 2023-10-05T19:29:33.283 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.278+0000 7f2500714700 1 -- 172.21.15.167:0/3463850574 <== osd.2 v2:172.21.15.167:6816/30037 19 ==== osd_op_reply(123 usage.21 [call] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 ==== 152+0+0 (crc 0 0 0) 0x7f24e8032040 con 0x564c78678e00 2023-10-05T19:29:33.283 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.278+0000 7f25033e0e40 1 -- 172.21.15.167:0/3463850574 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:124 6.13 6:cdb71d68:usage::usage.22:head [call rgw.user_usage_log_read in=60b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32) v8 -- 0x564c787398e0 con 0x7f24e8007030 2023-10-05T19:29:33.284 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.278+0000 7f2501716700 1 -- 172.21.15.167:0/3463850574 <== osd.0 v2:172.21.15.118:6800/27690 34 ==== osd_op_reply(124 usage.22 [call] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 ==== 152+0+0 (crc 0 0 0) 0x7f24f0050040 con 0x7f24e8007030 2023-10-05T19:29:33.284 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.278+0000 7f25033e0e40 1 -- 172.21.15.167:0/3463850574 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:125 6.17 6:ef3bca03:usage::usage.23:head [call rgw.user_usage_log_read in=60b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32) v8 -- 0x564c787398e0 con 0x564c786aaa50 2023-10-05T19:29:33.284 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.282+0000 7f2501716700 1 -- 172.21.15.167:0/3463850574 <== osd.1 v2:172.21.15.167:6800/30035 41 ==== osd_op_reply(125 usage.23 [call] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 ==== 152+0+0 (crc 0 0 0) 0x7f24f0050040 con 0x564c786aaa50 2023-10-05T19:29:33.285 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.282+0000 7f25033e0e40 1 -- 172.21.15.167:0/3463850574 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:126 6.e 6:7412960a:usage::usage.24:head [call rgw.user_usage_log_read in=60b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32) v8 -- 0x564c787398e0 con 0x564c786aaa50 2023-10-05T19:29:33.285 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.282+0000 7f2501716700 1 -- 172.21.15.167:0/3463850574 <== osd.1 v2:172.21.15.167:6800/30035 42 ==== osd_op_reply(126 usage.24 [call] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 ==== 152+0+0 (crc 0 0 0) 0x7f24f0050040 con 0x564c786aaa50 2023-10-05T19:29:33.285 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.282+0000 7f25033e0e40 1 -- 172.21.15.167:0/3463850574 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:127 6.7 6:e62d1024:usage::usage.25:head [call rgw.user_usage_log_read in=60b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32) v8 -- 0x564c787398e0 con 0x564c786b08f0 2023-10-05T19:29:33.286 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.282+0000 7f2500f15700 1 -- 172.21.15.167:0/3463850574 <== osd.3 v2:172.21.15.167:6808/30036 32 ==== osd_op_reply(127 usage.25 [call] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 ==== 152+0+0 (crc 0 0 0) 0x7f24f8024020 con 0x564c786b08f0 2023-10-05T19:29:33.286 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.282+0000 7f25033e0e40 1 -- 172.21.15.167:0/3463850574 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:128 6.d 6:b172cd59:usage::usage.26:head [call rgw.user_usage_log_read in=60b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32) v8 -- 0x564c787398e0 con 0x564c786aaa50 2023-10-05T19:29:33.287 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.282+0000 7f2501716700 1 -- 172.21.15.167:0/3463850574 <== osd.1 v2:172.21.15.167:6800/30035 43 ==== osd_op_reply(128 usage.26 [call] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 ==== 152+0+0 (crc 0 0 0) 0x7f24f0050040 con 0x564c786aaa50 2023-10-05T19:29:33.287 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.282+0000 7f25033e0e40 1 -- 172.21.15.167:0/3463850574 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:129 6.1 6:83912023:usage::usage.27:head [call rgw.user_usage_log_read in=60b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32) v8 -- 0x564c787398e0 con 0x564c786aaa50 2023-10-05T19:29:33.287 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.282+0000 7f2501716700 1 -- 172.21.15.167:0/3463850574 <== osd.1 v2:172.21.15.167:6800/30035 44 ==== osd_op_reply(129 usage.27 [call] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 ==== 152+0+0 (crc 0 0 0) 0x7f24f0050040 con 0x564c786aaa50 2023-10-05T19:29:33.287 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.282+0000 7f25033e0e40 1 -- 172.21.15.167:0/3463850574 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:130 6.1e 6:7c73bdb3:usage::usage.28:head [call rgw.user_usage_log_read in=60b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32) v8 -- 0x564c787398e0 con 0x564c786b08f0 2023-10-05T19:29:33.288 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.282+0000 7f2500f15700 1 -- 172.21.15.167:0/3463850574 <== osd.3 v2:172.21.15.167:6808/30036 33 ==== osd_op_reply(130 usage.28 [call] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 ==== 152+0+0 (crc 0 0 0) 0x7f24f8024020 con 0x564c786b08f0 2023-10-05T19:29:33.288 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.282+0000 7f25033e0e40 1 -- 172.21.15.167:0/3463850574 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:131 6.16 6:6e59fa35:usage::usage.29:head [call rgw.user_usage_log_read in=60b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32) v8 -- 0x564c787398e0 con 0x564c786b08f0 2023-10-05T19:29:33.289 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.282+0000 7f2500f15700 1 -- 172.21.15.167:0/3463850574 <== osd.3 v2:172.21.15.167:6808/30036 34 ==== osd_op_reply(131 usage.29 [call] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 ==== 152+0+0 (crc 0 0 0) 0x7f24f8024020 con 0x564c786b08f0 2023-10-05T19:29:33.289 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.286+0000 7f25033e0e40 1 -- 172.21.15.167:0/3463850574 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:132 6.1c 6:38fe6885:usage::usage.30:head [call rgw.user_usage_log_read in=60b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32) v8 -- 0x564c787398e0 con 0x564c786b08f0 2023-10-05T19:29:33.289 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.286+0000 7f2500f15700 1 -- 172.21.15.167:0/3463850574 <== osd.3 v2:172.21.15.167:6808/30036 35 ==== osd_op_reply(132 usage.30 [call] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 ==== 152+0+0 (crc 0 0 0) 0x7f24f8024020 con 0x564c786b08f0 2023-10-05T19:29:33.289 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.286+0000 7f25033e0e40 1 -- 172.21.15.167:0/3463850574 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:133 6.1d 6:bb28dff0:usage::usage.31:head [call rgw.user_usage_log_read in=60b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32) v8 -- 0x564c787398e0 con 0x7f24e8007030 2023-10-05T19:29:33.290 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.286+0000 7f2501716700 1 -- 172.21.15.167:0/3463850574 <== osd.0 v2:172.21.15.118:6800/27690 35 ==== osd_op_reply(133 usage.31 [call] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 ==== 152+0+0 (crc 0 0 0) 0x7f24f0050040 con 0x7f24e8007030 2023-10-05T19:29:33.290 INFO:teuthology.orchestra.run.smithi167.stdout:{ 2023-10-05T19:29:33.290 INFO:teuthology.orchestra.run.smithi167.stdout: "entries": [ 2023-10-05T19:29:33.291 INFO:teuthology.orchestra.run.smithi167.stdout: { 2023-10-05T19:29:33.291 INFO:teuthology.orchestra.run.smithi167.stdout: "user": "anonymous", 2023-10-05T19:29:33.291 INFO:teuthology.orchestra.run.smithi167.stdout: "buckets": [ 2023-10-05T19:29:33.291 INFO:teuthology.orchestra.run.smithi167.stdout: { 2023-10-05T19:29:33.291 INFO:teuthology.orchestra.run.smithi167.stdout: "bucket": "", 2023-10-05T19:29:33.291 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2023-10-05T19:00:00.000000Z", 2023-10-05T19:29:33.291 INFO:teuthology.orchestra.run.smithi167.stdout: "epoch": 1696532400, 2023-10-05T19:29:33.291 INFO:teuthology.orchestra.run.smithi167.stdout: "owner": "anonymous", 2023-10-05T19:29:33.291 INFO:teuthology.orchestra.run.smithi167.stdout: "categories": [ 2023-10-05T19:29:33.291 INFO:teuthology.orchestra.run.smithi167.stdout: { 2023-10-05T19:29:33.292 INFO:teuthology.orchestra.run.smithi167.stdout: "category": "list_buckets", 2023-10-05T19:29:33.292 INFO:teuthology.orchestra.run.smithi167.stdout: "bytes_sent": 231, 2023-10-05T19:29:33.292 INFO:teuthology.orchestra.run.smithi167.stdout: "bytes_received": 0, 2023-10-05T19:29:33.292 INFO:teuthology.orchestra.run.smithi167.stdout: "ops": 1, 2023-10-05T19:29:33.292 INFO:teuthology.orchestra.run.smithi167.stdout: "successful_ops": 1 2023-10-05T19:29:33.292 INFO:teuthology.orchestra.run.smithi167.stdout: } 2023-10-05T19:29:33.292 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2023-10-05T19:29:33.292 INFO:teuthology.orchestra.run.smithi167.stdout: } 2023-10-05T19:29:33.292 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2023-10-05T19:29:33.292 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2023-10-05T19:29:33.292 INFO:teuthology.orchestra.run.smithi167.stdout: { 2023-10-05T19:29:33.293 INFO:teuthology.orchestra.run.smithi167.stdout: "user": "foo", 2023-10-05T19:29:33.293 INFO:teuthology.orchestra.run.smithi167.stdout: "buckets": [ 2023-10-05T19:29:33.293 INFO:teuthology.orchestra.run.smithi167.stdout: { 2023-10-05T19:29:33.293 INFO:teuthology.orchestra.run.smithi167.stdout: "bucket": "", 2023-10-05T19:29:33.293 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2023-10-05T19:00:00.000000Z", 2023-10-05T19:29:33.293 INFO:teuthology.orchestra.run.smithi167.stdout: "epoch": 1696532400, 2023-10-05T19:29:33.293 INFO:teuthology.orchestra.run.smithi167.stdout: "owner": "foo", 2023-10-05T19:29:33.293 INFO:teuthology.orchestra.run.smithi167.stdout: "categories": [ 2023-10-05T19:29:33.293 INFO:teuthology.orchestra.run.smithi167.stdout: { 2023-10-05T19:29:33.293 INFO:teuthology.orchestra.run.smithi167.stdout: "category": "list_buckets", 2023-10-05T19:29:33.294 INFO:teuthology.orchestra.run.smithi167.stdout: "bytes_sent": 322, 2023-10-05T19:29:33.294 INFO:teuthology.orchestra.run.smithi167.stdout: "bytes_received": 0, 2023-10-05T19:29:33.294 INFO:teuthology.orchestra.run.smithi167.stdout: "ops": 1, 2023-10-05T19:29:33.294 INFO:teuthology.orchestra.run.smithi167.stdout: "successful_ops": 1 2023-10-05T19:29:33.294 INFO:teuthology.orchestra.run.smithi167.stdout: } 2023-10-05T19:29:33.294 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2023-10-05T19:29:33.294 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2023-10-05T19:29:33.294 INFO:teuthology.orchestra.run.smithi167.stdout: { 2023-10-05T19:29:33.294 INFO:teuthology.orchestra.run.smithi167.stdout: "bucket": "myfoo", 2023-10-05T19:29:33.295 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2023-10-05T19:00:00.000000Z", 2023-10-05T19:29:33.295 INFO:teuthology.orchestra.run.smithi167.stdout: "epoch": 1696532400, 2023-10-05T19:29:33.295 INFO:teuthology.orchestra.run.smithi167.stdout: "owner": "foo", 2023-10-05T19:29:33.295 INFO:teuthology.orchestra.run.smithi167.stdout: "categories": [ 2023-10-05T19:29:33.295 INFO:teuthology.orchestra.run.smithi167.stdout: { 2023-10-05T19:29:33.295 INFO:teuthology.orchestra.run.smithi167.stdout: "category": "create_bucket", 2023-10-05T19:29:33.295 INFO:teuthology.orchestra.run.smithi167.stdout: "bytes_sent": 0, 2023-10-05T19:29:33.295 INFO:teuthology.orchestra.run.smithi167.stdout: "bytes_received": 0, 2023-10-05T19:29:33.295 INFO:teuthology.orchestra.run.smithi167.stdout: "ops": 5, 2023-10-05T19:29:33.295 INFO:teuthology.orchestra.run.smithi167.stdout: "successful_ops": 5 2023-10-05T19:29:33.296 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2023-10-05T19:29:33.296 INFO:teuthology.orchestra.run.smithi167.stdout: { 2023-10-05T19:29:33.296 INFO:teuthology.orchestra.run.smithi167.stdout: "category": "delete_bucket", 2023-10-05T19:29:33.296 INFO:teuthology.orchestra.run.smithi167.stdout: "bytes_sent": 217, 2023-10-05T19:29:33.296 INFO:teuthology.orchestra.run.smithi167.stdout: "bytes_received": 0, 2023-10-05T19:29:33.296 INFO:teuthology.orchestra.run.smithi167.stdout: "ops": 2, 2023-10-05T19:29:33.296 INFO:teuthology.orchestra.run.smithi167.stdout: "successful_ops": 1 2023-10-05T19:29:33.296 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2023-10-05T19:29:33.296 INFO:teuthology.orchestra.run.smithi167.stdout: { 2023-10-05T19:29:33.296 INFO:teuthology.orchestra.run.smithi167.stdout: "category": "delete_obj", 2023-10-05T19:29:33.297 INFO:teuthology.orchestra.run.smithi167.stdout: "bytes_sent": 0, 2023-10-05T19:29:33.297 INFO:teuthology.orchestra.run.smithi167.stdout: "bytes_received": 0, 2023-10-05T19:29:33.297 INFO:teuthology.orchestra.run.smithi167.stdout: "ops": 4, 2023-10-05T19:29:33.297 INFO:teuthology.orchestra.run.smithi167.stdout: "successful_ops": 4 2023-10-05T19:29:33.297 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2023-10-05T19:29:33.297 INFO:teuthology.orchestra.run.smithi167.stdout: { 2023-10-05T19:29:33.297 INFO:teuthology.orchestra.run.smithi167.stdout: "category": "get_acls", 2023-10-05T19:29:33.297 INFO:teuthology.orchestra.run.smithi167.stdout: "bytes_sent": 1026, 2023-10-05T19:29:33.297 INFO:teuthology.orchestra.run.smithi167.stdout: "bytes_received": 0, 2023-10-05T19:29:33.298 INFO:teuthology.orchestra.run.smithi167.stdout: "ops": 2, 2023-10-05T19:29:33.298 INFO:teuthology.orchestra.run.smithi167.stdout: "successful_ops": 2 2023-10-05T19:29:33.298 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2023-10-05T19:29:33.298 INFO:teuthology.orchestra.run.smithi167.stdout: { 2023-10-05T19:29:33.298 INFO:teuthology.orchestra.run.smithi167.stdout: "category": "get_obj", 2023-10-05T19:29:33.298 INFO:teuthology.orchestra.run.smithi167.stdout: "bytes_sent": 4, 2023-10-05T19:29:33.298 INFO:teuthology.orchestra.run.smithi167.stdout: "bytes_received": 0, 2023-10-05T19:29:33.298 INFO:teuthology.orchestra.run.smithi167.stdout: "ops": 1, 2023-10-05T19:29:33.298 INFO:teuthology.orchestra.run.smithi167.stdout: "successful_ops": 1 2023-10-05T19:29:33.298 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2023-10-05T19:29:33.299 INFO:teuthology.orchestra.run.smithi167.stdout: { 2023-10-05T19:29:33.299 INFO:teuthology.orchestra.run.smithi167.stdout: "category": "list_bucket", 2023-10-05T19:29:33.299 INFO:teuthology.orchestra.run.smithi167.stdout: "bytes_sent": 515, 2023-10-05T19:29:33.299 INFO:teuthology.orchestra.run.smithi167.stdout: "bytes_received": 0, 2023-10-05T19:29:33.299 INFO:teuthology.orchestra.run.smithi167.stdout: "ops": 1, 2023-10-05T19:29:33.299 INFO:teuthology.orchestra.run.smithi167.stdout: "successful_ops": 1 2023-10-05T19:29:33.299 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2023-10-05T19:29:33.299 INFO:teuthology.orchestra.run.smithi167.stdout: { 2023-10-05T19:29:33.299 INFO:teuthology.orchestra.run.smithi167.stdout: "category": "put_acls", 2023-10-05T19:29:33.299 INFO:teuthology.orchestra.run.smithi167.stdout: "bytes_sent": 0, 2023-10-05T19:29:33.300 INFO:teuthology.orchestra.run.smithi167.stdout: "bytes_received": 0, 2023-10-05T19:29:33.300 INFO:teuthology.orchestra.run.smithi167.stdout: "ops": 2, 2023-10-05T19:29:33.300 INFO:teuthology.orchestra.run.smithi167.stdout: "successful_ops": 2 2023-10-05T19:29:33.300 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2023-10-05T19:29:33.300 INFO:teuthology.orchestra.run.smithi167.stdout: { 2023-10-05T19:29:33.300 INFO:teuthology.orchestra.run.smithi167.stdout: "category": "put_obj", 2023-10-05T19:29:33.300 INFO:teuthology.orchestra.run.smithi167.stdout: "bytes_sent": 0, 2023-10-05T19:29:33.300 INFO:teuthology.orchestra.run.smithi167.stdout: "bytes_received": 30000042, 2023-10-05T19:29:33.300 INFO:teuthology.orchestra.run.smithi167.stdout: "ops": 11, 2023-10-05T19:29:33.300 INFO:teuthology.orchestra.run.smithi167.stdout: "successful_ops": 11 2023-10-05T19:29:33.301 INFO:teuthology.orchestra.run.smithi167.stdout: } 2023-10-05T19:29:33.301 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2023-10-05T19:29:33.301 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2023-10-05T19:29:33.301 INFO:teuthology.orchestra.run.smithi167.stdout: { 2023-10-05T19:29:33.301 INFO:teuthology.orchestra.run.smithi167.stdout: "bucket": "myfoo2", 2023-10-05T19:29:33.301 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2023-10-05T19:00:00.000000Z", 2023-10-05T19:29:33.301 INFO:teuthology.orchestra.run.smithi167.stdout: "epoch": 1696532400, 2023-10-05T19:29:33.301 INFO:teuthology.orchestra.run.smithi167.stdout: "owner": "foo", 2023-10-05T19:29:33.301 INFO:teuthology.orchestra.run.smithi167.stdout: "categories": [ 2023-10-05T19:29:33.302 INFO:teuthology.orchestra.run.smithi167.stdout: { 2023-10-05T19:29:33.302 INFO:teuthology.orchestra.run.smithi167.stdout: "category": "create_bucket", 2023-10-05T19:29:33.302 INFO:teuthology.orchestra.run.smithi167.stdout: "bytes_sent": 0, 2023-10-05T19:29:33.302 INFO:teuthology.orchestra.run.smithi167.stdout: "bytes_received": 0, 2023-10-05T19:29:33.302 INFO:teuthology.orchestra.run.smithi167.stdout: "ops": 1, 2023-10-05T19:29:33.302 INFO:teuthology.orchestra.run.smithi167.stdout: "successful_ops": 1 2023-10-05T19:29:33.302 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2023-10-05T19:29:33.302 INFO:teuthology.orchestra.run.smithi167.stdout: { 2023-10-05T19:29:33.302 INFO:teuthology.orchestra.run.smithi167.stdout: "category": "delete_bucket", 2023-10-05T19:29:33.303 INFO:teuthology.orchestra.run.smithi167.stdout: "bytes_sent": 0, 2023-10-05T19:29:33.303 INFO:teuthology.orchestra.run.smithi167.stdout: "bytes_received": 0, 2023-10-05T19:29:33.303 INFO:teuthology.orchestra.run.smithi167.stdout: "ops": 1, 2023-10-05T19:29:33.303 INFO:teuthology.orchestra.run.smithi167.stdout: "successful_ops": 1 2023-10-05T19:29:33.303 INFO:teuthology.orchestra.run.smithi167.stdout: } 2023-10-05T19:29:33.303 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2023-10-05T19:29:33.303 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2023-10-05T19:29:33.303 INFO:teuthology.orchestra.run.smithi167.stdout: { 2023-10-05T19:29:33.303 INFO:teuthology.orchestra.run.smithi167.stdout: "bucket": "myfoo3", 2023-10-05T19:29:33.304 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2023-10-05T19:00:00.000000Z", 2023-10-05T19:29:33.304 INFO:teuthology.orchestra.run.smithi167.stdout: "epoch": 1696532400, 2023-10-05T19:29:33.304 INFO:teuthology.orchestra.run.smithi167.stdout: "owner": "foo", 2023-10-05T19:29:33.304 INFO:teuthology.orchestra.run.smithi167.stdout: "categories": [ 2023-10-05T19:29:33.304 INFO:teuthology.orchestra.run.smithi167.stdout: { 2023-10-05T19:29:33.304 INFO:teuthology.orchestra.run.smithi167.stdout: "category": "create_bucket", 2023-10-05T19:29:33.304 INFO:teuthology.orchestra.run.smithi167.stdout: "bytes_sent": 0, 2023-10-05T19:29:33.304 INFO:teuthology.orchestra.run.smithi167.stdout: "bytes_received": 0, 2023-10-05T19:29:33.304 INFO:teuthology.orchestra.run.smithi167.stdout: "ops": 1, 2023-10-05T19:29:33.304 INFO:teuthology.orchestra.run.smithi167.stdout: "successful_ops": 1 2023-10-05T19:29:33.305 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2023-10-05T19:29:33.305 INFO:teuthology.orchestra.run.smithi167.stdout: { 2023-10-05T19:29:33.305 INFO:teuthology.orchestra.run.smithi167.stdout: "category": "delete_bucket", 2023-10-05T19:29:33.305 INFO:teuthology.orchestra.run.smithi167.stdout: "bytes_sent": 0, 2023-10-05T19:29:33.305 INFO:teuthology.orchestra.run.smithi167.stdout: "bytes_received": 0, 2023-10-05T19:29:33.305 INFO:teuthology.orchestra.run.smithi167.stdout: "ops": 1, 2023-10-05T19:29:33.305 INFO:teuthology.orchestra.run.smithi167.stdout: "successful_ops": 1 2023-10-05T19:29:33.305 INFO:teuthology.orchestra.run.smithi167.stdout: } 2023-10-05T19:29:33.305 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2023-10-05T19:29:33.306 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2023-10-05T19:29:33.306 INFO:teuthology.orchestra.run.smithi167.stdout: { 2023-10-05T19:29:33.306 INFO:teuthology.orchestra.run.smithi167.stdout: "bucket": "myfoo4", 2023-10-05T19:29:33.306 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2023-10-05T19:00:00.000000Z", 2023-10-05T19:29:33.306 INFO:teuthology.orchestra.run.smithi167.stdout: "epoch": 1696532400, 2023-10-05T19:29:33.306 INFO:teuthology.orchestra.run.smithi167.stdout: "owner": "foo", 2023-10-05T19:29:33.306 INFO:teuthology.orchestra.run.smithi167.stdout: "categories": [ 2023-10-05T19:29:33.306 INFO:teuthology.orchestra.run.smithi167.stdout: { 2023-10-05T19:29:33.306 INFO:teuthology.orchestra.run.smithi167.stdout: "category": "create_bucket", 2023-10-05T19:29:33.307 INFO:teuthology.orchestra.run.smithi167.stdout: "bytes_sent": 0, 2023-10-05T19:29:33.307 INFO:teuthology.orchestra.run.smithi167.stdout: "bytes_received": 0, 2023-10-05T19:29:33.307 INFO:teuthology.orchestra.run.smithi167.stdout: "ops": 1, 2023-10-05T19:29:33.307 INFO:teuthology.orchestra.run.smithi167.stdout: "successful_ops": 1 2023-10-05T19:29:33.307 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2023-10-05T19:29:33.307 INFO:teuthology.orchestra.run.smithi167.stdout: { 2023-10-05T19:29:33.307 INFO:teuthology.orchestra.run.smithi167.stdout: "category": "delete_bucket", 2023-10-05T19:29:33.307 INFO:teuthology.orchestra.run.smithi167.stdout: "bytes_sent": 0, 2023-10-05T19:29:33.307 INFO:teuthology.orchestra.run.smithi167.stdout: "bytes_received": 0, 2023-10-05T19:29:33.307 INFO:teuthology.orchestra.run.smithi167.stdout: "ops": 1, 2023-10-05T19:29:33.307 INFO:teuthology.orchestra.run.smithi167.stdout: "successful_ops": 1 2023-10-05T19:29:33.308 INFO:teuthology.orchestra.run.smithi167.stdout: } 2023-10-05T19:29:33.308 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2023-10-05T19:29:33.308 INFO:teuthology.orchestra.run.smithi167.stdout: } 2023-10-05T19:29:33.308 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2023-10-05T19:29:33.308 INFO:teuthology.orchestra.run.smithi167.stdout: } 2023-10-05T19:29:33.308 INFO:teuthology.orchestra.run.smithi167.stdout: ], 2023-10-05T19:29:33.308 INFO:teuthology.orchestra.run.smithi167.stdout: "summary": [ 2023-10-05T19:29:33.308 INFO:teuthology.orchestra.run.smithi167.stdout: { 2023-10-05T19:29:33.308 INFO:teuthology.orchestra.run.smithi167.stdout: "user": "anonymous", 2023-10-05T19:29:33.308 INFO:teuthology.orchestra.run.smithi167.stdout: "categories": [ 2023-10-05T19:29:33.309 INFO:teuthology.orchestra.run.smithi167.stdout: { 2023-10-05T19:29:33.309 INFO:teuthology.orchestra.run.smithi167.stdout: "category": "list_buckets", 2023-10-05T19:29:33.309 INFO:teuthology.orchestra.run.smithi167.stdout: "bytes_sent": 231, 2023-10-05T19:29:33.309 INFO:teuthology.orchestra.run.smithi167.stdout: "bytes_received": 0, 2023-10-05T19:29:33.309 INFO:teuthology.orchestra.run.smithi167.stdout: "ops": 1, 2023-10-05T19:29:33.309 INFO:teuthology.orchestra.run.smithi167.stdout: "successful_ops": 1 2023-10-05T19:29:33.309 INFO:teuthology.orchestra.run.smithi167.stdout: } 2023-10-05T19:29:33.309 INFO:teuthology.orchestra.run.smithi167.stdout: ], 2023-10-05T19:29:33.309 INFO:teuthology.orchestra.run.smithi167.stdout: "total": { 2023-10-05T19:29:33.309 INFO:teuthology.orchestra.run.smithi167.stdout: "bytes_sent": 231, 2023-10-05T19:29:33.309 INFO:teuthology.orchestra.run.smithi167.stdout: "bytes_received": 0, 2023-10-05T19:29:33.310 INFO:teuthology.orchestra.run.smithi167.stdout: "ops": 1, 2023-10-05T19:29:33.310 INFO:teuthology.orchestra.run.smithi167.stdout: "successful_ops": 1 2023-10-05T19:29:33.310 INFO:teuthology.orchestra.run.smithi167.stdout: } 2023-10-05T19:29:33.310 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2023-10-05T19:29:33.310 INFO:teuthology.orchestra.run.smithi167.stdout: { 2023-10-05T19:29:33.310 INFO:teuthology.orchestra.run.smithi167.stdout: "user": "foo", 2023-10-05T19:29:33.310 INFO:teuthology.orchestra.run.smithi167.stdout: "categories": [ 2023-10-05T19:29:33.310 INFO:teuthology.orchestra.run.smithi167.stdout: { 2023-10-05T19:29:33.310 INFO:teuthology.orchestra.run.smithi167.stdout: "category": "create_bucket", 2023-10-05T19:29:33.310 INFO:teuthology.orchestra.run.smithi167.stdout: "bytes_sent": 0, 2023-10-05T19:29:33.311 INFO:teuthology.orchestra.run.smithi167.stdout: "bytes_received": 0, 2023-10-05T19:29:33.311 INFO:teuthology.orchestra.run.smithi167.stdout: "ops": 8, 2023-10-05T19:29:33.311 INFO:teuthology.orchestra.run.smithi167.stdout: "successful_ops": 8 2023-10-05T19:29:33.311 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2023-10-05T19:29:33.311 INFO:teuthology.orchestra.run.smithi167.stdout: { 2023-10-05T19:29:33.311 INFO:teuthology.orchestra.run.smithi167.stdout: "category": "delete_bucket", 2023-10-05T19:29:33.311 INFO:teuthology.orchestra.run.smithi167.stdout: "bytes_sent": 217, 2023-10-05T19:29:33.311 INFO:teuthology.orchestra.run.smithi167.stdout: "bytes_received": 0, 2023-10-05T19:29:33.311 INFO:teuthology.orchestra.run.smithi167.stdout: "ops": 5, 2023-10-05T19:29:33.311 INFO:teuthology.orchestra.run.smithi167.stdout: "successful_ops": 4 2023-10-05T19:29:33.311 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2023-10-05T19:29:33.312 INFO:teuthology.orchestra.run.smithi167.stdout: { 2023-10-05T19:29:33.312 INFO:teuthology.orchestra.run.smithi167.stdout: "category": "delete_obj", 2023-10-05T19:29:33.312 INFO:teuthology.orchestra.run.smithi167.stdout: "bytes_sent": 0, 2023-10-05T19:29:33.312 INFO:teuthology.orchestra.run.smithi167.stdout: "bytes_received": 0, 2023-10-05T19:29:33.312 INFO:teuthology.orchestra.run.smithi167.stdout: "ops": 4, 2023-10-05T19:29:33.312 INFO:teuthology.orchestra.run.smithi167.stdout: "successful_ops": 4 2023-10-05T19:29:33.312 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2023-10-05T19:29:33.312 INFO:teuthology.orchestra.run.smithi167.stdout: { 2023-10-05T19:29:33.312 INFO:teuthology.orchestra.run.smithi167.stdout: "category": "get_acls", 2023-10-05T19:29:33.312 INFO:teuthology.orchestra.run.smithi167.stdout: "bytes_sent": 1026, 2023-10-05T19:29:33.313 INFO:teuthology.orchestra.run.smithi167.stdout: "bytes_received": 0, 2023-10-05T19:29:33.313 INFO:teuthology.orchestra.run.smithi167.stdout: "ops": 2, 2023-10-05T19:29:33.313 INFO:teuthology.orchestra.run.smithi167.stdout: "successful_ops": 2 2023-10-05T19:29:33.313 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2023-10-05T19:29:33.313 INFO:teuthology.orchestra.run.smithi167.stdout: { 2023-10-05T19:29:33.313 INFO:teuthology.orchestra.run.smithi167.stdout: "category": "get_obj", 2023-10-05T19:29:33.313 INFO:teuthology.orchestra.run.smithi167.stdout: "bytes_sent": 4, 2023-10-05T19:29:33.313 INFO:teuthology.orchestra.run.smithi167.stdout: "bytes_received": 0, 2023-10-05T19:29:33.313 INFO:teuthology.orchestra.run.smithi167.stdout: "ops": 1, 2023-10-05T19:29:33.313 INFO:teuthology.orchestra.run.smithi167.stdout: "successful_ops": 1 2023-10-05T19:29:33.314 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2023-10-05T19:29:33.316 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.286+0000 7f25033e0e40 1 -- 172.21.15.167:0/3463850574 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:134 7.1f 7:f95f44c2:::notify.0:head [watch unwatch cookie 94886437801648] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e32) v8 -- 0x564c786f53f0 con 0x564c78678e00 2023-10-05T19:29:33.317 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.290+0000 7f2500714700 1 -- 172.21.15.167:0/3463850574 <== osd.2 v2:172.21.15.167:6816/30037 20 ==== osd_op_reply(134 notify.0 [watch unwatch cookie 94886437801648] v32'224 uv222 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f24e8032040 con 0x564c78678e00 2023-10-05T19:29:33.317 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.290+0000 7f25033e0e40 20 remove_watcher() i=0 2023-10-05T19:29:33.317 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.290+0000 7f25033e0e40 2 removed watcher, disabling cache 2023-10-05T19:29:33.317 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.290+0000 7f25033e0e40 1 -- 172.21.15.167:0/3463850574 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:135 7.0 7:05bf5b68:::notify.1:head [watch unwatch cookie 94886437808256] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e32) v8 -- 0x564c786b4eb0 con 0x564c786aaa50 2023-10-05T19:29:33.317 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.294+0000 7f2501716700 1 -- 172.21.15.167:0/3463850574 <== osd.1 v2:172.21.15.167:6800/30035 45 ==== osd_op_reply(135 notify.1 [watch unwatch cookie 94886437808256] v32'224 uv222 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f24f0050040 con 0x564c786aaa50 2023-10-05T19:29:33.317 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.294+0000 7f25033e0e40 20 remove_watcher() i=1 2023-10-05T19:29:33.317 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.294+0000 7f25033e0e40 1 -- 172.21.15.167:0/3463850574 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:136 7.15 7:a93a5511:::notify.2:head [watch unwatch cookie 94886437810608] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e32) v8 -- 0x564c786b4eb0 con 0x564c786b08f0 2023-10-05T19:29:33.317 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.294+0000 7f2500f15700 1 -- 172.21.15.167:0/3463850574 <== osd.3 v2:172.21.15.167:6808/30036 36 ==== osd_op_reply(136 notify.2 [watch unwatch cookie 94886437810608] v32'224 uv222 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f24f8024020 con 0x564c786b08f0 2023-10-05T19:29:33.317 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.294+0000 7f25033e0e40 20 remove_watcher() i=2 2023-10-05T19:29:33.317 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.294+0000 7f25033e0e40 1 -- 172.21.15.167:0/3463850574 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:137 7.e 7:7759931f:::notify.3:head [watch unwatch cookie 94886437815024] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e32) v8 -- 0x564c786b4eb0 con 0x564c78678e00 2023-10-05T19:29:33.318 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.298+0000 7f2500714700 1 -- 172.21.15.167:0/3463850574 <== osd.2 v2:172.21.15.167:6816/30037 21 ==== osd_op_reply(137 notify.3 [watch unwatch cookie 94886437815024] v32'224 uv222 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f24e8032040 con 0x564c78678e00 2023-10-05T19:29:33.318 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.298+0000 7f25033e0e40 20 remove_watcher() i=3 2023-10-05T19:29:33.318 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.298+0000 7f25033e0e40 1 -- 172.21.15.167:0/3463850574 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:138 7.d 7:b4812045:::notify.4:head [watch unwatch cookie 94886437817232] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e32) v8 -- 0x564c786b82f0 con 0x564c786aaa50 2023-10-05T19:29:33.318 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.302+0000 7f2501716700 1 -- 172.21.15.167:0/3463850574 <== osd.1 v2:172.21.15.167:6800/30035 46 ==== osd_op_reply(138 notify.4 [watch unwatch cookie 94886437817232] v32'224 uv222 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f24f0050040 con 0x564c786aaa50 2023-10-05T19:29:33.318 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.302+0000 7f25033e0e40 20 remove_watcher() i=4 2023-10-05T19:29:33.318 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.302+0000 7f25033e0e40 1 -- 172.21.15.167:0/3463850574 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:139 7.3 7:c609908c:::notify.5:head [watch unwatch cookie 94886437820224] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e32) v8 -- 0x564c786b4eb0 con 0x7f24e8007030 2023-10-05T19:29:33.318 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.306+0000 7f2501716700 1 -- 172.21.15.167:0/3463850574 <== osd.0 v2:172.21.15.118:6800/27690 36 ==== osd_op_reply(139 notify.5 [watch unwatch cookie 94886437820224] v32'224 uv222 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f24f0050040 con 0x7f24e8007030 2023-10-05T19:29:33.318 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.306+0000 7f25033e0e40 20 remove_watcher() i=5 2023-10-05T19:29:33.318 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.306+0000 7f25033e0e40 1 -- 172.21.15.167:0/3463850574 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:140 7.14 7:2b04a3e9:::notify.6:head [watch unwatch cookie 94886437822512] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e32) v8 -- 0x564c786b8b90 con 0x7f24e8007030 2023-10-05T19:29:33.319 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.310+0000 7f2501716700 1 -- 172.21.15.167:0/3463850574 <== osd.0 v2:172.21.15.118:6800/27690 37 ==== osd_op_reply(140 notify.6 [watch unwatch cookie 94886437822512] v32'224 uv222 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f24f0050040 con 0x7f24e8007030 2023-10-05T19:29:33.319 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.310+0000 7f25033e0e40 20 remove_watcher() i=6 2023-10-05T19:29:33.319 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.310+0000 7f25033e0e40 1 -- 172.21.15.167:0/3463850574 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:141 7.9 7:93e5b521:::notify.7:head [watch unwatch cookie 94886437826640] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e32) v8 -- 0x564c786b9740 con 0x7f24e8007030 2023-10-05T19:29:33.319 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.310+0000 7f2501716700 1 -- 172.21.15.167:0/3463850574 <== osd.0 v2:172.21.15.118:6800/27690 38 ==== osd_op_reply(141 notify.7 [watch unwatch cookie 94886437826640] v32'224 uv222 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f24f0050040 con 0x7f24e8007030 2023-10-05T19:29:33.319 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.310+0000 7f25033e0e40 20 remove_watcher() i=7 2023-10-05T19:29:33.319 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.314+0000 7f24d27e4700 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2023-10-05T19:29:33.320 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.314+0000 7f25033e0e40 1 -- 172.21.15.167:0/3463850574 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:142 6.4 6:22d26bf9:::data_loggenerations_metadata:head [watch unwatch cookie 94886437827872] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e32) v8 -- 0x564c786b4eb0 con 0x564c786aaa50 2023-10-05T19:29:33.322 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.318+0000 7f2501716700 1 -- 172.21.15.167:0/3463850574 <== osd.1 v2:172.21.15.167:6800/30035 47 ==== osd_op_reply(142 data_loggenerations_metadata [watch unwatch cookie 94886437827872] v32'222 uv1 ondisk = 0) v8 ==== 172+0+0 (crc 0 0 0) 0x7f24f0050040 con 0x564c786aaa50 2023-10-05T19:29:33.322 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.318+0000 7f25033e0e40 1 -- 172.21.15.167:0/3463850574 >> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] conn(0x7f24e8007030 msgr2=0x7f24e801c920 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-05T19:29:33.322 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.318+0000 7f25033e0e40 1 --2- 172.21.15.167:0/3463850574 >> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] conn(0x7f24e8007030 0x7f24e801c920 crc :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-05T19:29:33.323 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.318+0000 7f25033e0e40 1 -- 172.21.15.167:0/3463850574 >> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] conn(0x564c786aaa50 msgr2=0x564c786ace60 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-05T19:29:33.323 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.318+0000 7f25033e0e40 1 --2- 172.21.15.167:0/3463850574 >> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] conn(0x564c786aaa50 0x564c786ace60 crc :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-05T19:29:33.323 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.318+0000 7f25033e0e40 1 -- 172.21.15.167:0/3463850574 >> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] conn(0x564c78678e00 msgr2=0x7f24f80439f0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-05T19:29:33.323 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.318+0000 7f25033e0e40 1 --2- 172.21.15.167:0/3463850574 >> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] conn(0x564c78678e00 0x7f24f80439f0 crc :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-05T19:29:33.323 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.318+0000 7f25033e0e40 1 -- 172.21.15.167:0/3463850574 >> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] conn(0x564c786b08f0 msgr2=0x564c786b2cd0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-05T19:29:33.323 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.318+0000 7f25033e0e40 1 --2- 172.21.15.167:0/3463850574 >> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] conn(0x564c786b08f0 0x564c786b2cd0 crc :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-05T19:29:33.323 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.318+0000 7f25033e0e40 1 -- 172.21.15.167:0/3463850574 >> [v2:172.21.15.167:6824/29889,v1:172.21.15.167:6825/29889] conn(0x7f24e0030c00 msgr2=0x7f24e00330b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-05T19:29:33.323 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.318+0000 7f25033e0e40 1 --2- 172.21.15.167:0/3463850574 >> [v2:172.21.15.167:6824/29889,v1:172.21.15.167:6825/29889] conn(0x7f24e0030c00 0x7f24e00330b0 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7f24e80030a0 tx=0x7f24e8015000 comp rx=0 tx=0).stop 2023-10-05T19:29:33.323 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.318+0000 7f2501716700 1 -- 172.21.15.167:0/3463850574 reap_dead start 2023-10-05T19:29:33.323 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.318+0000 7f25033e0e40 1 -- 172.21.15.167:0/3463850574 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x564c7867d3e0 msgr2=0x564c786a1020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-05T19:29:33.324 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.318+0000 7f25033e0e40 1 --2- 172.21.15.167:0/3463850574 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x564c7867d3e0 0x564c786a1020 secure :-1 s=READY pgs=581 cs=0 l=1 rev1=1 crypto rx=0x7f24f800f000 tx=0x7f24f8014000 comp rx=0 tx=0).stop 2023-10-05T19:29:33.324 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.318+0000 7f25033e0e40 1 -- 172.21.15.167:0/3463850574 shutdown_connections 2023-10-05T19:29:33.324 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.318+0000 7f25033e0e40 1 --2- 172.21.15.167:0/3463850574 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x564c7867d3e0 0x564c786a1020 unknown :-1 s=CLOSED pgs=581 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-05T19:29:33.324 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.318+0000 7f25033e0e40 1 -- 172.21.15.167:0/3463850574 >> 172.21.15.167:0/3463850574 conn(0x564c78674820 msgr2=0x564c78692570 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-05T19:29:33.324 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.318+0000 7f25033e0e40 1 -- 172.21.15.167:0/3463850574 shutdown_connections 2023-10-05T19:29:33.324 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.318+0000 7f25033e0e40 1 -- 172.21.15.167:0/3463850574 wait complete. 2023-10-05T19:29:33.325 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.318+0000 7f25033e0e40 1 -- 172.21.15.167:0/2482662757 >> [v2:172.21.15.167:6824/29889,v1:172.21.15.167:6825/29889] conn(0x7f2448030c00 msgr2=0x7f24480330b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-05T19:29:33.325 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.318+0000 7f25033e0e40 1 --2- 172.21.15.167:0/2482662757 >> [v2:172.21.15.167:6824/29889,v1:172.21.15.167:6825/29889] conn(0x7f2448030c00 0x7f24480330b0 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x564c786b9050 tx=0x7f24f0000f90 comp rx=0 tx=0).stop 2023-10-05T19:29:33.325 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.318+0000 7f25033e0e40 1 -- 172.21.15.167:0/2482662757 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x564c786c8da0 msgr2=0x564c786ce810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-05T19:29:33.325 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.318+0000 7f25033e0e40 1 --2- 172.21.15.167:0/2482662757 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x564c786c8da0 0x564c786ce810 secure :-1 s=READY pgs=583 cs=0 l=1 rev1=1 crypto rx=0x7f24e8012fd0 tx=0x7f24e802f000 comp rx=0 tx=0).stop 2023-10-05T19:29:33.326 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.318+0000 7f25033e0e40 1 -- 172.21.15.167:0/2482662757 shutdown_connections 2023-10-05T19:29:33.326 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.318+0000 7f25033e0e40 1 --2- 172.21.15.167:0/2482662757 >> [v2:172.21.15.167:6824/29889,v1:172.21.15.167:6825/29889] conn(0x7f2448030c00 0x7f24480330b0 unknown :-1 s=CLOSED pgs=201 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-05T19:29:33.326 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.318+0000 7f25033e0e40 1 --2- 172.21.15.167:0/2482662757 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x564c786c8da0 0x564c786ce810 unknown :-1 s=CLOSED pgs=583 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-05T19:29:33.326 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.318+0000 7f25033e0e40 1 -- 172.21.15.167:0/2482662757 >> 172.21.15.167:0/2482662757 conn(0x564c786c4160 msgr2=0x564c786c2260 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-05T19:29:33.326 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.318+0000 7f25033e0e40 1 -- 172.21.15.167:0/2482662757 shutdown_connections 2023-10-05T19:29:33.326 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.318+0000 7f25033e0e40 1 -- 172.21.15.167:0/2482662757 wait complete. 2023-10-05T19:29:33.326 INFO:teuthology.orchestra.run.smithi167.stdout: { 2023-10-05T19:29:33.327 INFO:teuthology.orchestra.run.smithi167.stdout: "category": "list_bucket", 2023-10-05T19:29:33.327 INFO:teuthology.orchestra.run.smithi167.stdout: "bytes_sent": 515, 2023-10-05T19:29:33.327 INFO:teuthology.orchestra.run.smithi167.stdout: "bytes_received": 0, 2023-10-05T19:29:33.327 INFO:teuthology.orchestra.run.smithi167.stdout: "ops": 1, 2023-10-05T19:29:33.327 INFO:teuthology.orchestra.run.smithi167.stdout: "successful_ops": 1 2023-10-05T19:29:33.327 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2023-10-05T19:29:33.327 INFO:teuthology.orchestra.run.smithi167.stdout: { 2023-10-05T19:29:33.327 INFO:teuthology.orchestra.run.smithi167.stdout: "category": "list_buckets", 2023-10-05T19:29:33.327 INFO:teuthology.orchestra.run.smithi167.stdout: "bytes_sent": 322, 2023-10-05T19:29:33.327 INFO:teuthology.orchestra.run.smithi167.stdout: "bytes_received": 0, 2023-10-05T19:29:33.328 INFO:teuthology.orchestra.run.smithi167.stdout: "ops": 1, 2023-10-05T19:29:33.328 INFO:teuthology.orchestra.run.smithi167.stdout: "successful_ops": 1 2023-10-05T19:29:33.328 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2023-10-05T19:29:33.328 INFO:teuthology.orchestra.run.smithi167.stdout: { 2023-10-05T19:29:33.328 INFO:teuthology.orchestra.run.smithi167.stdout: "category": "put_acls", 2023-10-05T19:29:33.328 INFO:teuthology.orchestra.run.smithi167.stdout: "bytes_sent": 0, 2023-10-05T19:29:33.328 INFO:teuthology.orchestra.run.smithi167.stdout: "bytes_received": 0, 2023-10-05T19:29:33.328 INFO:teuthology.orchestra.run.smithi167.stdout: "ops": 2, 2023-10-05T19:29:33.328 INFO:teuthology.orchestra.run.smithi167.stdout: "successful_ops": 2 2023-10-05T19:29:33.328 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2023-10-05T19:29:33.329 INFO:teuthology.orchestra.run.smithi167.stdout: { 2023-10-05T19:29:33.329 INFO:teuthology.orchestra.run.smithi167.stdout: "category": "put_obj", 2023-10-05T19:29:33.329 INFO:teuthology.orchestra.run.smithi167.stdout: "bytes_sent": 0, 2023-10-05T19:29:33.329 INFO:teuthology.orchestra.run.smithi167.stdout: "bytes_received": 30000042, 2023-10-05T19:29:33.329 INFO:teuthology.orchestra.run.smithi167.stdout: "ops": 11, 2023-10-05T19:29:33.329 INFO:teuthology.orchestra.run.smithi167.stdout: "successful_ops": 11 2023-10-05T19:29:33.329 INFO:teuthology.orchestra.run.smithi167.stdout: } 2023-10-05T19:29:33.329 INFO:teuthology.orchestra.run.smithi167.stdout: ], 2023-10-05T19:29:33.329 INFO:teuthology.orchestra.run.smithi167.stdout: "total": { 2023-10-05T19:29:33.329 INFO:teuthology.orchestra.run.smithi167.stdout: "bytes_sent": 2084, 2023-10-05T19:29:33.329 INFO:teuthology.orchestra.run.smithi167.stdout: "bytes_received": 30000042, 2023-10-05T19:29:33.330 INFO:teuthology.orchestra.run.smithi167.stdout: "ops": 35, 2023-10-05T19:29:33.330 INFO:teuthology.orchestra.run.smithi167.stdout: "successful_ops": 34 2023-10-05T19:29:33.330 INFO:teuthology.orchestra.run.smithi167.stdout: } 2023-10-05T19:29:33.330 INFO:teuthology.orchestra.run.smithi167.stdout: } 2023-10-05T19:29:33.330 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2023-10-05T19:29:33.330 INFO:teuthology.orchestra.run.smithi167.stdout:} 2023-10-05T19:29:33.331 DEBUG:tasks.util.rgw: json result: {'entries': [{'user': 'anonymous', 'buckets': [{'bucket': '', 'time': '2023-10-05T19:00:00.000000Z', 'epoch': 1696532400, 'owner': 'anonymous', 'categories': [{'category': 'list_buckets', 'bytes_sent': 231, 'bytes_received': 0, 'ops': 1, 'successful_ops': 1}]}]}, {'user': 'foo', 'buckets': [{'bucket': '', 'time': '2023-10-05T19:00:00.000000Z', 'epoch': 1696532400, 'owner': 'foo', 'categories': [{'category': 'list_buckets', 'bytes_sent': 322, 'bytes_received': 0, 'ops': 1, 'successful_ops': 1}]}, {'bucket': 'myfoo', 'time': '2023-10-05T19:00:00.000000Z', 'epoch': 1696532400, 'owner': 'foo', 'categories': [{'category': 'create_bucket', 'bytes_sent': 0, 'bytes_received': 0, 'ops': 5, 'successful_ops': 5}, {'category': 'delete_bucket', 'bytes_sent': 217, 'bytes_received': 0, 'ops': 2, 'successful_ops': 1}, {'category': 'delete_obj', 'bytes_sent': 0, 'bytes_received': 0, 'ops': 4, 'successful_ops': 4}, {'category': 'get_acls', 'bytes_sent': 1026, 'bytes_received': 0, 'ops': 2, 'successful_ops': 2}, {'category': 'get_obj', 'bytes_sent': 4, 'bytes_received': 0, 'ops': 1, 'successful_ops': 1}, {'category': 'list_bucket', 'bytes_sent': 515, 'bytes_received': 0, 'ops': 1, 'successful_ops': 1}, {'category': 'put_acls', 'bytes_sent': 0, 'bytes_received': 0, 'ops': 2, 'successful_ops': 2}, {'category': 'put_obj', 'bytes_sent': 0, 'bytes_received': 30000042, 'ops': 11, 'successful_ops': 11}]}, {'bucket': 'myfoo2', 'time': '2023-10-05T19:00:00.000000Z', 'epoch': 1696532400, 'owner': 'foo', 'categories': [{'category': 'create_bucket', 'bytes_sent': 0, 'bytes_received': 0, 'ops': 1, 'successful_ops': 1}, {'category': 'delete_bucket', 'bytes_sent': 0, 'bytes_received': 0, 'ops': 1, 'successful_ops': 1}]}, {'bucket': 'myfoo3', 'time': '2023-10-05T19:00:00.000000Z', 'epoch': 1696532400, 'owner': 'foo', 'categories': [{'category': 'create_bucket', 'bytes_sent': 0, 'bytes_received': 0, 'ops': 1, 'successful_ops': 1}, {'category': 'delete_bucket', 'bytes_sent': 0, 'bytes_received': 0, 'ops': 1, 'successful_ops': 1}]}, {'bucket': 'myfoo4', 'time': '2023-10-05T19:00:00.000000Z', 'epoch': 1696532400, 'owner': 'foo', 'categories': [{'category': 'create_bucket', 'bytes_sent': 0, 'bytes_received': 0, 'ops': 1, 'successful_ops': 1}, {'category': 'delete_bucket', 'bytes_sent': 0, 'bytes_received': 0, 'ops': 1, 'successful_ops': 1}]}]}], 'summary': [{'user': 'anonymous', 'categories': [{'category': 'list_buckets', 'bytes_sent': 231, 'bytes_received': 0, 'ops': 1, 'successful_ops': 1}], 'total': {'bytes_sent': 231, 'bytes_received': 0, 'ops': 1, 'successful_ops': 1}}, {'user': 'foo', 'categories': [{'category': 'create_bucket', 'bytes_sent': 0, 'bytes_received': 0, 'ops': 8, 'successful_ops': 8}, {'category': 'delete_bucket', 'bytes_sent': 217, 'bytes_received': 0, 'ops': 5, 'successful_ops': 4}, {'category': 'delete_obj', 'bytes_sent': 0, 'bytes_received': 0, 'ops': 4, 'successful_ops': 4}, {'category': 'get_acls', 'bytes_sent': 1026, 'bytes_received': 0, 'ops': 2, 'successful_ops': 2}, {'category': 'get_obj', 'bytes_sent': 4, 'bytes_received': 0, 'ops': 1, 'successful_ops': 1}, {'category': 'list_bucket', 'bytes_sent': 515, 'bytes_received': 0, 'ops': 1, 'successful_ops': 1}, {'category': 'list_buckets', 'bytes_sent': 322, 'bytes_received': 0, 'ops': 1, 'successful_ops': 1}, {'category': 'put_acls', 'bytes_sent': 0, 'bytes_received': 0, 'ops': 2, 'successful_ops': 2}, {'category': 'put_obj', 'bytes_sent': 0, 'bytes_received': 30000042, 'ops': 11, 'successful_ops': 11}], 'total': {'bytes_sent': 2084, 'bytes_received': 30000042, 'ops': 35, 'successful_ops': 34}}]} 2023-10-05T19:29:33.331 INFO:tasks.util.rgw:rgwadmin: client.0 : ['usage', 'show', '--uid', 'foo'] 2023-10-05T19:29:33.332 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'ceph', 'usage', 'show', '--uid', 'foo'] 2023-10-05T19:29:33.332 DEBUG:teuthology.orchestra.run.smithi167:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster ceph usage show --uid foo 2023-10-05T19:29:33.415 INFO:teuthology.orchestra.run.smithi167.stderr:ignoring --setuser ceph since I am not root 2023-10-05T19:29:33.415 INFO:teuthology.orchestra.run.smithi167.stderr:ignoring --setgroup ceph since I am not root 2023-10-05T19:29:33.415 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.410+0000 7f02161abe40 1 Processor -- start 2023-10-05T19:29:33.416 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.410+0000 7f02161abe40 1 -- start start 2023-10-05T19:29:33.416 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.410+0000 7f02161abe40 1 --2- >> v2:172.21.15.118:3300/0 conn(0x55aab2af2600 0x55aab2af49e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-05T19:29:33.416 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.410+0000 7f02161abe40 1 -- --> v1:172.21.15.118:6789/0 -- auth(proto 0 26 bytes epoch 0) v1 -- 0x55aab2a38bd0 con 0x55aab2aee020 2023-10-05T19:29:33.416 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.410+0000 7f02161abe40 1 -- --> v2:172.21.15.118:3300/0 -- mon_getmap magic: 0 v1 -- 0x55aab29f5ee0 con 0x55aab2af2600 2023-10-05T19:29:33.417 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.410+0000 7f02134df700 1 --2- >> v2:172.21.15.118:3300/0 conn(0x55aab2af2600 0x55aab2af49e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-05T19:29:33.417 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.410+0000 7f0213ce0700 1 --1- >> v1:172.21.15.118:6789/0 conn(0x55aab2aee020 0x55aab2aee3f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.118:6789/0 says I am v1:172.21.15.167:52602/0 (socket says 172.21.15.167:52602) 2023-10-05T19:29:33.417 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.410+0000 7f0213ce0700 1 -- 172.21.15.167:0/2468106586 learned_addr learned my addr 172.21.15.167:0/2468106586 (peer_addr_for_me v1:172.21.15.167:0/0) 2023-10-05T19:29:33.417 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.414+0000 7f0212cde700 1 -- 172.21.15.167:0/2468106586 <== mon.0 v1:172.21.15.118:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 8427775 0 0) 0x55aab2a38bd0 con 0x55aab2aee020 2023-10-05T19:29:33.417 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.414+0000 7f0212cde700 1 -- 172.21.15.167:0/2468106586 --> v1:172.21.15.118:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f01f4001e10 con 0x55aab2aee020 2023-10-05T19:29:33.417 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.414+0000 7f02134df700 1 -- 172.21.15.167:0/2468106586 >> v1:172.21.15.118:6789/0 conn(0x55aab2aee020 legacy=0x55aab2aee3f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-05T19:29:33.418 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.414+0000 7f02134df700 1 -- 172.21.15.167:0/2468106586 --> v2:172.21.15.118:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55aab2aeec30 con 0x55aab2af2600 2023-10-05T19:29:33.419 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.414+0000 7f02134df700 1 --2- 172.21.15.167:0/2468106586 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55aab2af2600 0x55aab2af49e0 secure :-1 s=READY pgs=585 cs=0 l=1 rev1=1 crypto rx=0x7f01fc00a5d0 tx=0x7f01fc005a50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=717770c66b3de983 server_cookie=0 in_seq=0 out_seq=0 2023-10-05T19:29:33.419 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.414+0000 7f0212cde700 1 -- 172.21.15.167:0/2468106586 <== mon.0 v2:172.21.15.118:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f01fc007610 con 0x55aab2af2600 2023-10-05T19:29:33.419 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.414+0000 7f0212cde700 1 -- 172.21.15.167:0/2468106586 <== mon.0 v2:172.21.15.118:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f01fc007770 con 0x55aab2af2600 2023-10-05T19:29:33.419 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.414+0000 7f0212cde700 1 -- 172.21.15.167:0/2468106586 <== mon.0 v2:172.21.15.118:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f01fc007a20 con 0x55aab2af2600 2023-10-05T19:29:33.419 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.414+0000 7f02161abe40 1 -- 172.21.15.167:0/2468106586 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55aab2af2600 msgr2=0x55aab2af49e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-05T19:29:33.419 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.414+0000 7f02161abe40 1 --2- 172.21.15.167:0/2468106586 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55aab2af2600 0x55aab2af49e0 secure :-1 s=READY pgs=585 cs=0 l=1 rev1=1 crypto rx=0x7f01fc00a5d0 tx=0x7f01fc005a50 comp rx=0 tx=0).stop 2023-10-05T19:29:33.419 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.414+0000 7f02161abe40 1 -- 172.21.15.167:0/2468106586 shutdown_connections 2023-10-05T19:29:33.419 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.414+0000 7f02161abe40 1 --2- 172.21.15.167:0/2468106586 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55aab2af2600 0x55aab2af49e0 unknown :-1 s=CLOSED pgs=585 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-05T19:29:33.420 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.414+0000 7f02161abe40 1 -- 172.21.15.167:0/2468106586 >> 172.21.15.167:0/2468106586 conn(0x55aab2a61870 msgr2=0x55aab2a61c70 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-05T19:29:33.420 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.414+0000 7f02161abe40 1 -- 172.21.15.167:0/2468106586 shutdown_connections 2023-10-05T19:29:33.420 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.414+0000 7f02161abe40 1 -- 172.21.15.167:0/2468106586 wait complete. 2023-10-05T19:29:33.421 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.414+0000 7f02161abe40 1 Processor -- start 2023-10-05T19:29:33.421 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.414+0000 7f02161abe40 1 -- start start 2023-10-05T19:29:33.421 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.414+0000 7f02161abe40 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55aab2aee020 0x55aab2b0cf60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-05T19:29:33.421 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.414+0000 7f02161abe40 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x55aab29b9e30 con 0x55aab2aee020 2023-10-05T19:29:33.421 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.414+0000 7f0213ce0700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55aab2aee020 0x55aab2b0cf60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-05T19:29:33.422 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.414+0000 7f0213ce0700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55aab2aee020 0x55aab2b0cf60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.118:3300/0 says I am v2:172.21.15.167:46192/0 (socket says 172.21.15.167:46192) 2023-10-05T19:29:33.422 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.414+0000 7f0213ce0700 1 -- 172.21.15.167:0/2225927647 learned_addr learned my addr 172.21.15.167:0/2225927647 (peer_addr_for_me v2:172.21.15.167:0/0) 2023-10-05T19:29:33.422 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.418+0000 7f0213ce0700 1 -- 172.21.15.167:0/2225927647 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f01fc005cf0 con 0x55aab2aee020 2023-10-05T19:29:33.422 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.418+0000 7f0213ce0700 1 --2- 172.21.15.167:0/2225927647 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55aab2aee020 0x55aab2b0cf60 secure :-1 s=READY pgs=586 cs=0 l=1 rev1=1 crypto rx=0x7f0204001c30 tx=0x7f0204001810 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d9a45ea105f32d46 server_cookie=0 in_seq=0 out_seq=0 2023-10-05T19:29:33.422 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.418+0000 7f0211cdc700 1 -- 172.21.15.167:0/2225927647 <== mon.0 v2:172.21.15.118:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f020400bbf0 con 0x55aab2aee020 2023-10-05T19:29:33.422 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.418+0000 7f0211cdc700 1 -- 172.21.15.167:0/2225927647 <== mon.0 v2:172.21.15.118:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f020400bd50 con 0x55aab2aee020 2023-10-05T19:29:33.422 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.418+0000 7f0211cdc700 1 -- 172.21.15.167:0/2225927647 <== mon.0 v2:172.21.15.118:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f020400e470 con 0x55aab2aee020 2023-10-05T19:29:33.424 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.418+0000 7f02161abe40 1 -- 172.21.15.167:0/2225927647 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55aab2aee020 msgr2=0x55aab2b0cf60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-05T19:29:33.425 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.418+0000 7f02161abe40 1 --2- 172.21.15.167:0/2225927647 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55aab2aee020 0x55aab2b0cf60 secure :-1 s=READY pgs=586 cs=0 l=1 rev1=1 crypto rx=0x7f0204001c30 tx=0x7f0204001810 comp rx=0 tx=0).stop 2023-10-05T19:29:33.425 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.418+0000 7f02161abe40 1 -- 172.21.15.167:0/2225927647 shutdown_connections 2023-10-05T19:29:33.425 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.418+0000 7f02161abe40 1 --2- 172.21.15.167:0/2225927647 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55aab2aee020 0x55aab2b0cf60 unknown :-1 s=CLOSED pgs=586 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-05T19:29:33.425 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.418+0000 7f02161abe40 1 -- 172.21.15.167:0/2225927647 >> 172.21.15.167:0/2225927647 conn(0x55aab2a61870 msgr2=0x55aab2b08b20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-05T19:29:33.425 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.418+0000 7f02161abe40 1 -- 172.21.15.167:0/2225927647 shutdown_connections 2023-10-05T19:29:33.425 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.418+0000 7f02161abe40 1 -- 172.21.15.167:0/2225927647 wait complete. 2023-10-05T19:29:33.425 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.418+0000 7f02161abe40 1 Processor -- start 2023-10-05T19:29:33.425 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.418+0000 7f02161abe40 1 -- start start 2023-10-05T19:29:33.426 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.418+0000 7f02161abe40 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55aab2aee020 0x55aab2b13cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-05T19:29:33.426 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.418+0000 7f02161abe40 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x55aab298ced0 con 0x55aab2aee020 2023-10-05T19:29:33.426 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.418+0000 7f0213ce0700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55aab2aee020 0x55aab2b13cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-05T19:29:33.426 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.418+0000 7f0213ce0700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55aab2aee020 0x55aab2b13cf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.118:3300/0 says I am v2:172.21.15.167:46196/0 (socket says 172.21.15.167:46196) 2023-10-05T19:29:33.426 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.418+0000 7f0213ce0700 1 -- 172.21.15.167:0/2873792612 learned_addr learned my addr 172.21.15.167:0/2873792612 (peer_addr_for_me v2:172.21.15.167:0/0) 2023-10-05T19:29:33.426 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.418+0000 7f0213ce0700 1 -- 172.21.15.167:0/2873792612 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f01f4001e10 con 0x55aab2aee020 2023-10-05T19:29:33.426 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.418+0000 7f0213ce0700 1 --2- 172.21.15.167:0/2873792612 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55aab2aee020 0x55aab2b13cf0 secure :-1 s=READY pgs=587 cs=0 l=1 rev1=1 crypto rx=0x7f020400b960 tx=0x7f0204014000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-05T19:29:33.426 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.418+0000 7f020bfff700 1 -- 172.21.15.167:0/2873792612 <== mon.0 v2:172.21.15.118:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f020401f070 con 0x55aab2aee020 2023-10-05T19:29:33.426 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.418+0000 7f020bfff700 1 -- 172.21.15.167:0/2873792612 <== mon.0 v2:172.21.15.118:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f020401a2e0 con 0x55aab2aee020 2023-10-05T19:29:33.427 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.418+0000 7f020bfff700 1 -- 172.21.15.167:0/2873792612 <== mon.0 v2:172.21.15.118:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f020400e470 con 0x55aab2aee020 2023-10-05T19:29:33.427 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.418+0000 7f02161abe40 1 -- 172.21.15.167:0/2873792612 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x55aab2b14230 con 0x55aab2aee020 2023-10-05T19:29:33.427 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.422+0000 7f02161abe40 1 -- 172.21.15.167:0/2873792612 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55aab2b14670 con 0x55aab2aee020 2023-10-05T19:29:33.428 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.422+0000 7f020bfff700 1 -- 172.21.15.167:0/2873792612 <== mon.0 v2:172.21.15.118:3300/0 4 ==== mgrmap(e 5) v1 ==== 40329+0+0 (secure 0 0 0) 0x7f020401aa10 con 0x55aab2aee020 2023-10-05T19:29:33.428 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.422+0000 7f020bfff700 1 --2- 172.21.15.167:0/2873792612 >> [v2:172.21.15.167:6824/29889,v1:172.21.15.167:6825/29889] conn(0x7f01f8030bf0 0x7f01f80330a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-05T19:29:33.428 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.422+0000 7f020bfff700 1 -- 172.21.15.167:0/2873792612 <== mon.0 v2:172.21.15.118:3300/0 5 ==== osd_map(32..32 src has 1..32) v4 ==== 6260+0+0 (secure 0 0 0) 0x7f0204045420 con 0x55aab2aee020 2023-10-05T19:29:33.428 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.422+0000 7f02134df700 1 --2- 172.21.15.167:0/2873792612 >> [v2:172.21.15.167:6824/29889,v1:172.21.15.167:6825/29889] conn(0x7f01f8030bf0 0x7f01f80330a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-05T19:29:33.428 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.422+0000 7f02161abe40 20 rados->read ofs=0 len=0 2023-10-05T19:29:33.429 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.422+0000 7f02161abe40 1 --2- 172.21.15.167:0/2873792612 >> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] conn(0x55aab2af2600 0x55aab2b1f6e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-05T19:29:33.429 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.422+0000 7f02161abe40 1 -- 172.21.15.167:0/2873792612 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:1 5.12 5:49953fa1:::default.realm:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32) v8 -- 0x55aab2b1fe10 con 0x55aab2af2600 2023-10-05T19:29:33.429 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.422+0000 7f02144e1700 1 --2- 172.21.15.167:0/2873792612 >> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] conn(0x55aab2af2600 0x55aab2b1f6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-05T19:29:33.429 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.422+0000 7f02134df700 1 --2- 172.21.15.167:0/2873792612 >> [v2:172.21.15.167:6824/29889,v1:172.21.15.167:6825/29889] conn(0x7f01f8030bf0 0x7f01f80330a0 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7f01fc0030c0 tx=0x7f01fc014000 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-05T19:29:33.429 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.422+0000 7f02144e1700 1 --2- 172.21.15.167:0/2873792612 >> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] conn(0x55aab2af2600 0x55aab2b1f6e0 crc :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-05T19:29:33.431 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.426+0000 7f02144e1700 1 -- 172.21.15.167:0/2873792612 <== osd.1 v2:172.21.15.167:6800/30035 1 ==== osd_op_reply(1 default.realm [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 ==== 157+0+0 (crc 0 0 0) 0x7f020c00a5a0 con 0x55aab2af2600 2023-10-05T19:29:33.431 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.426+0000 7f02161abe40 20 rados_obj.operate() r=-2 bl.length=0 2023-10-05T19:29:33.432 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.426+0000 7f02161abe40 20 realm 2023-10-05T19:29:33.432 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.426+0000 7f02161abe40 20 rados->read ofs=0 len=0 2023-10-05T19:29:33.432 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.426+0000 7f02161abe40 1 -- 172.21.15.167:0/2873792612 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:2 5.12 5:49953fa1:::default.realm:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32) v8 -- 0x55aab2b1ccd0 con 0x55aab2af2600 2023-10-05T19:29:33.432 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.426+0000 7f02144e1700 1 -- 172.21.15.167:0/2873792612 <== osd.1 v2:172.21.15.167:6800/30035 2 ==== osd_op_reply(2 default.realm [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 ==== 157+0+0 (crc 0 0 0) 0x7f020c00a5a0 con 0x55aab2af2600 2023-10-05T19:29:33.432 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.426+0000 7f02161abe40 20 rados_obj.operate() r=-2 bl.length=0 2023-10-05T19:29:33.432 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.426+0000 7f02161abe40 4 RGWPeriod::init failed to init realm id : (2) No such file or directory 2023-10-05T19:29:33.432 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.426+0000 7f02161abe40 20 rados->read ofs=0 len=0 2023-10-05T19:29:33.432 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.426+0000 7f02161abe40 1 -- 172.21.15.167:0/2873792612 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:3 5.12 5:49953fa1:::default.realm:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32) v8 -- 0x55aab2b20940 con 0x55aab2af2600 2023-10-05T19:29:33.432 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.426+0000 7f02144e1700 1 -- 172.21.15.167:0/2873792612 <== osd.1 v2:172.21.15.167:6800/30035 3 ==== osd_op_reply(3 default.realm [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 ==== 157+0+0 (crc 0 0 0) 0x7f020c00a5a0 con 0x55aab2af2600 2023-10-05T19:29:33.433 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.426+0000 7f02161abe40 20 rados_obj.operate() r=-2 bl.length=0 2023-10-05T19:29:33.433 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.426+0000 7f02161abe40 20 rados->read ofs=0 len=0 2023-10-05T19:29:33.433 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.426+0000 7f02161abe40 1 -- 172.21.15.167:0/2873792612 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:4 5.f 5:f43b6ece:::zone_names.default:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32) v8 -- 0x55aab2b204f0 con 0x55aab2af2600 2023-10-05T19:29:33.433 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.426+0000 7f02144e1700 1 -- 172.21.15.167:0/2873792612 <== osd.1 v2:172.21.15.167:6800/30035 4 ==== osd_op_reply(4 zone_names.default [read 0~46 out=46b] v0'0 uv1 ondisk = 0) v8 ==== 162+0+46 (crc 0 0 0) 0x7f020c00a5a0 con 0x55aab2af2600 2023-10-05T19:29:33.433 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.426+0000 7f02161abe40 20 rados_obj.operate() r=0 bl.length=46 2023-10-05T19:29:33.433 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.426+0000 7f02161abe40 20 rados->read ofs=0 len=0 2023-10-05T19:29:33.433 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.426+0000 7f02161abe40 1 -- 172.21.15.167:0/2873792612 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:5 5.1d 5:b8374eb4:::zone_info.9a9df965-6612-45d4-a45f-6e3c889bacc2:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32) v8 -- 0x55aab2b21370 con 0x55aab2af2600 2023-10-05T19:29:33.433 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.426+0000 7f02144e1700 1 -- 172.21.15.167:0/2873792612 <== osd.1 v2:172.21.15.167:6800/30035 5 ==== osd_op_reply(5 zone_info.9a9df965-6612-45d4-a45f-6e3c889bacc2 [read 0~874 out=874b] v0'0 uv2 ondisk = 0) v8 ==== 190+0+874 (crc 0 0 0) 0x7f020c00a5a0 con 0x55aab2af2600 2023-10-05T19:29:33.433 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.426+0000 7f02161abe40 20 rados_obj.operate() r=0 bl.length=874 2023-10-05T19:29:33.433 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.426+0000 7f02161abe40 20 searching for the correct realm 2023-10-05T19:29:33.434 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.426+0000 7f02161abe40 1 --2- 172.21.15.167:0/2873792612 >> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] conn(0x55aab2b231c0 0x55aab2b255a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-05T19:29:33.434 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.426+0000 7f02161abe40 1 -- 172.21.15.167:0/2873792612 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:6 5.0 5:00000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x55aab2b25ae0 con 0x55aab2b231c0 2023-10-05T19:29:33.434 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.426+0000 7f0213ce0700 1 --2- 172.21.15.167:0/2873792612 >> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] conn(0x55aab2b231c0 0x55aab2b255a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-05T19:29:33.434 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.426+0000 7f0213ce0700 1 --2- 172.21.15.167:0/2873792612 >> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] conn(0x55aab2b231c0 0x55aab2b255a0 crc :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-05T19:29:33.435 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.430+0000 7f0213ce0700 1 -- 172.21.15.167:0/2873792612 <== osd.3 v2:172.21.15.167:6808/30036 1 ==== osd_op_reply(6 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f0204014070 con 0x55aab2b231c0 2023-10-05T19:29:33.435 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.430+0000 7f0213ce0700 1 --2- 172.21.15.167:0/2873792612 >> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] conn(0x7f020402d020 0x7f020402c5d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-05T19:29:33.435 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.430+0000 7f0213ce0700 1 -- 172.21.15.167:0/2873792612 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:7 5.10 5:08000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x55aab2b25ae0 con 0x7f020402d020 2023-10-05T19:29:33.435 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.430+0000 7f02134df700 1 --2- 172.21.15.167:0/2873792612 >> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] conn(0x7f020402d020 0x7f020402c5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-05T19:29:33.436 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.430+0000 7f02134df700 1 --2- 172.21.15.167:0/2873792612 >> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] conn(0x7f020402d020 0x7f020402c5d0 crc :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-05T19:29:33.436 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.430+0000 7f02134df700 1 -- 172.21.15.167:0/2873792612 <== osd.2 v2:172.21.15.167:6816/30037 1 ==== osd_op_reply(7 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f01fc0079f0 con 0x7f020402d020 2023-10-05T19:29:33.436 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.430+0000 7f02134df700 1 --2- 172.21.15.167:0/2873792612 >> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] conn(0x7f01fc007030 0x7f01fc01d430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-05T19:29:33.436 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.430+0000 7f02134df700 1 -- 172.21.15.167:0/2873792612 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:8 5.8 5:10000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x55aab2b25ae0 con 0x7f01fc007030 2023-10-05T19:29:33.436 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.430+0000 7f02144e1700 1 --2- 172.21.15.167:0/2873792612 >> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] conn(0x7f01fc007030 0x7f01fc01d430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-05T19:29:33.436 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.430+0000 7f02144e1700 1 --2- 172.21.15.167:0/2873792612 >> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] conn(0x7f01fc007030 0x7f01fc01d430 crc :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-05T19:29:33.437 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.430+0000 7f02144e1700 1 -- 172.21.15.167:0/2873792612 <== osd.0 v2:172.21.15.118:6800/27690 1 ==== osd_op_reply(8 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f020c00a5a0 con 0x7f01fc007030 2023-10-05T19:29:33.437 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.430+0000 7f02144e1700 1 -- 172.21.15.167:0/2873792612 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:9 5.18 5:18000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x55aab2b25ae0 con 0x55aab2af2600 2023-10-05T19:29:33.437 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.434+0000 7f02144e1700 1 -- 172.21.15.167:0/2873792612 <== osd.1 v2:172.21.15.167:6800/30035 6 ==== osd_op_reply(9 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f020c00a5a0 con 0x55aab2af2600 2023-10-05T19:29:33.437 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.434+0000 7f02144e1700 1 -- 172.21.15.167:0/2873792612 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:10 5.4 5:20000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x55aab2b25ae0 con 0x55aab2af2600 2023-10-05T19:29:33.437 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.434+0000 7f02144e1700 1 -- 172.21.15.167:0/2873792612 <== osd.1 v2:172.21.15.167:6800/30035 7 ==== osd_op_reply(10 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f020c00a5a0 con 0x55aab2af2600 2023-10-05T19:29:33.438 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.434+0000 7f02144e1700 1 -- 172.21.15.167:0/2873792612 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:11 5.14 5:28000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x55aab2b25ae0 con 0x7f01fc007030 2023-10-05T19:29:33.438 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.434+0000 7f02144e1700 1 -- 172.21.15.167:0/2873792612 <== osd.0 v2:172.21.15.118:6800/27690 2 ==== osd_op_reply(11 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f020c00a5a0 con 0x7f01fc007030 2023-10-05T19:29:33.438 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.434+0000 7f02144e1700 1 -- 172.21.15.167:0/2873792612 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:12 5.c 5:30000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x55aab2b25ae0 con 0x7f01fc007030 2023-10-05T19:29:33.439 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.434+0000 7f02144e1700 1 -- 172.21.15.167:0/2873792612 <== osd.0 v2:172.21.15.118:6800/27690 3 ==== osd_op_reply(12 [pgnls start_epoch 32 out=112b] v23'1 uv1 ondisk = 1) v8 ==== 144+0+112 (crc 0 0 0) 0x7f020c00a5a0 con 0x7f01fc007030 2023-10-05T19:29:33.439 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.434+0000 7f02144e1700 1 -- 172.21.15.167:0/2873792612 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:13 5.1c 5:38000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x55aab2b25ae0 con 0x7f020402d020 2023-10-05T19:29:33.440 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.434+0000 7f02134df700 1 -- 172.21.15.167:0/2873792612 <== osd.2 v2:172.21.15.167:6816/30037 2 ==== osd_op_reply(13 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f01fc029040 con 0x7f020402d020 2023-10-05T19:29:33.440 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.434+0000 7f02134df700 1 -- 172.21.15.167:0/2873792612 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:14 5.2 5:40000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x55aab2b25ae0 con 0x55aab2b231c0 2023-10-05T19:29:33.440 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.438+0000 7f0213ce0700 1 -- 172.21.15.167:0/2873792612 <== osd.3 v2:172.21.15.167:6808/30036 2 ==== osd_op_reply(14 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f0204014070 con 0x55aab2b231c0 2023-10-05T19:29:33.441 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.438+0000 7f0213ce0700 1 -- 172.21.15.167:0/2873792612 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:15 5.12 5:48000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x55aab2b25ae0 con 0x55aab2af2600 2023-10-05T19:29:33.441 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.438+0000 7f02144e1700 1 -- 172.21.15.167:0/2873792612 <== osd.1 v2:172.21.15.167:6800/30035 8 ==== osd_op_reply(15 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f020c00a5a0 con 0x55aab2af2600 2023-10-05T19:29:33.441 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.438+0000 7f02144e1700 1 -- 172.21.15.167:0/2873792612 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:16 5.a 5:50000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x55aab2b25ae0 con 0x7f01fc007030 2023-10-05T19:29:33.442 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.438+0000 7f02144e1700 1 -- 172.21.15.167:0/2873792612 <== osd.0 v2:172.21.15.118:6800/27690 4 ==== osd_op_reply(16 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f020c00a5a0 con 0x7f01fc007030 2023-10-05T19:29:33.442 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.438+0000 7f02144e1700 1 -- 172.21.15.167:0/2873792612 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:17 5.1a 5:58000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x55aab2b25ae0 con 0x55aab2af2600 2023-10-05T19:29:33.443 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.438+0000 7f02144e1700 1 -- 172.21.15.167:0/2873792612 <== osd.1 v2:172.21.15.167:6800/30035 9 ==== osd_op_reply(17 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f020c00a5a0 con 0x55aab2af2600 2023-10-05T19:29:33.443 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.438+0000 7f02144e1700 1 -- 172.21.15.167:0/2873792612 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:18 5.6 5:60000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x55aab2b25ae0 con 0x7f020402d020 2023-10-05T19:29:33.443 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.438+0000 7f02134df700 1 -- 172.21.15.167:0/2873792612 <== osd.2 v2:172.21.15.167:6816/30037 3 ==== osd_op_reply(18 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f01fc029040 con 0x7f020402d020 2023-10-05T19:29:33.443 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.438+0000 7f02134df700 1 -- 172.21.15.167:0/2873792612 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:19 5.16 5:68000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x55aab2b25ae0 con 0x55aab2b231c0 2023-10-05T19:29:33.444 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.438+0000 7f0213ce0700 1 -- 172.21.15.167:0/2873792612 <== osd.3 v2:172.21.15.167:6808/30036 3 ==== osd_op_reply(19 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f0204014070 con 0x55aab2b231c0 2023-10-05T19:29:33.444 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.438+0000 7f0213ce0700 1 -- 172.21.15.167:0/2873792612 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:20 5.e 5:70000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x55aab2b25ae0 con 0x55aab2b231c0 2023-10-05T19:29:33.445 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.442+0000 7f0213ce0700 1 -- 172.21.15.167:0/2873792612 <== osd.3 v2:172.21.15.167:6808/30036 4 ==== osd_op_reply(20 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f0204014070 con 0x55aab2b231c0 2023-10-05T19:29:33.445 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.442+0000 7f0213ce0700 1 -- 172.21.15.167:0/2873792612 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:21 5.1e 5:78000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x55aab2b25ae0 con 0x7f01fc007030 2023-10-05T19:29:33.446 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.442+0000 7f02144e1700 1 -- 172.21.15.167:0/2873792612 <== osd.0 v2:172.21.15.118:6800/27690 5 ==== osd_op_reply(21 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f020c00a5a0 con 0x7f01fc007030 2023-10-05T19:29:33.446 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.442+0000 7f02144e1700 1 -- 172.21.15.167:0/2873792612 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:22 5.1 5:80000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x55aab2b25ae0 con 0x55aab2b231c0 2023-10-05T19:29:33.446 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.442+0000 7f0213ce0700 1 -- 172.21.15.167:0/2873792612 <== osd.3 v2:172.21.15.167:6808/30036 5 ==== osd_op_reply(22 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f0204014070 con 0x55aab2b231c0 2023-10-05T19:29:33.447 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.442+0000 7f0213ce0700 1 -- 172.21.15.167:0/2873792612 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:23 5.11 5:88000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x55aab2b25ae0 con 0x55aab2b231c0 2023-10-05T19:29:33.447 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.442+0000 7f0213ce0700 1 -- 172.21.15.167:0/2873792612 <== osd.3 v2:172.21.15.167:6808/30036 6 ==== osd_op_reply(23 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f0204014070 con 0x55aab2b231c0 2023-10-05T19:29:33.447 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.442+0000 7f0213ce0700 1 -- 172.21.15.167:0/2873792612 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:24 5.9 5:90000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x55aab2b25ae0 con 0x55aab2af2600 2023-10-05T19:29:33.448 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.442+0000 7f02144e1700 1 -- 172.21.15.167:0/2873792612 <== osd.1 v2:172.21.15.167:6800/30035 10 ==== osd_op_reply(24 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f020c00a5a0 con 0x55aab2af2600 2023-10-05T19:29:33.448 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.442+0000 7f02144e1700 1 -- 172.21.15.167:0/2873792612 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:25 5.19 5:98000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x55aab2b25ae0 con 0x55aab2af2600 2023-10-05T19:29:33.449 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.446+0000 7f02144e1700 1 -- 172.21.15.167:0/2873792612 <== osd.1 v2:172.21.15.167:6800/30035 11 ==== osd_op_reply(25 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f020c00a5a0 con 0x55aab2af2600 2023-10-05T19:29:33.449 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.446+0000 7f02144e1700 1 -- 172.21.15.167:0/2873792612 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:26 5.5 5:a0000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x55aab2b25ae0 con 0x7f01fc007030 2023-10-05T19:29:33.450 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.446+0000 7f02144e1700 1 -- 172.21.15.167:0/2873792612 <== osd.0 v2:172.21.15.118:6800/27690 6 ==== osd_op_reply(26 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f020c00a5a0 con 0x7f01fc007030 2023-10-05T19:29:33.450 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.446+0000 7f02144e1700 1 -- 172.21.15.167:0/2873792612 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:27 5.15 5:a8000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x55aab2b25ae0 con 0x55aab2b231c0 2023-10-05T19:29:33.450 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.446+0000 7f0213ce0700 1 -- 172.21.15.167:0/2873792612 <== osd.3 v2:172.21.15.167:6808/30036 7 ==== osd_op_reply(27 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f0204014070 con 0x55aab2b231c0 2023-10-05T19:29:33.450 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.446+0000 7f0213ce0700 1 -- 172.21.15.167:0/2873792612 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:28 5.d 5:b0000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x55aab2b25ae0 con 0x7f020402d020 2023-10-05T19:29:33.451 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.446+0000 7f02134df700 1 -- 172.21.15.167:0/2873792612 <== osd.2 v2:172.21.15.167:6816/30037 4 ==== osd_op_reply(28 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f01fc029040 con 0x7f020402d020 2023-10-05T19:29:33.451 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.446+0000 7f02134df700 1 -- 172.21.15.167:0/2873792612 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:29 5.1d 5:b8000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x55aab2b25ae0 con 0x55aab2af2600 2023-10-05T19:29:33.452 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.446+0000 7f02144e1700 1 -- 172.21.15.167:0/2873792612 <== osd.1 v2:172.21.15.167:6800/30035 12 ==== osd_op_reply(29 [pgnls start_epoch 32 out=107b] v30'2 uv2 ondisk = 1) v8 ==== 144+0+107 (crc 0 0 0) 0x7f020c00a5a0 con 0x55aab2af2600 2023-10-05T19:29:33.452 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.446+0000 7f02144e1700 1 -- 172.21.15.167:0/2873792612 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:30 5.3 5:c0000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x55aab2b25ae0 con 0x7f01fc007030 2023-10-05T19:29:33.453 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.450+0000 7f02144e1700 1 -- 172.21.15.167:0/2873792612 <== osd.0 v2:172.21.15.118:6800/27690 7 ==== osd_op_reply(30 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f020c00a5a0 con 0x7f01fc007030 2023-10-05T19:29:33.453 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.450+0000 7f02144e1700 1 -- 172.21.15.167:0/2873792612 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:31 5.13 5:c8000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x55aab2b25ae0 con 0x55aab2b231c0 2023-10-05T19:29:33.454 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.450+0000 7f0213ce0700 1 -- 172.21.15.167:0/2873792612 <== osd.3 v2:172.21.15.167:6808/30036 8 ==== osd_op_reply(31 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f0204014070 con 0x55aab2b231c0 2023-10-05T19:29:33.454 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.450+0000 7f0213ce0700 1 -- 172.21.15.167:0/2873792612 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:32 5.b 5:d0000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x55aab2b25ae0 con 0x7f01fc007030 2023-10-05T19:29:33.455 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.450+0000 7f02144e1700 1 -- 172.21.15.167:0/2873792612 <== osd.0 v2:172.21.15.118:6800/27690 8 ==== osd_op_reply(32 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f020c00a5a0 con 0x7f01fc007030 2023-10-05T19:29:33.455 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.450+0000 7f02144e1700 1 -- 172.21.15.167:0/2873792612 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:33 5.1b 5:d8000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x55aab2b25ae0 con 0x7f020402d020 2023-10-05T19:29:33.456 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.450+0000 7f02134df700 1 -- 172.21.15.167:0/2873792612 <== osd.2 v2:172.21.15.167:6816/30037 5 ==== osd_op_reply(33 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f01fc029040 con 0x7f020402d020 2023-10-05T19:29:33.456 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.450+0000 7f02134df700 1 -- 172.21.15.167:0/2873792612 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:34 5.7 5:e0000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x55aab2b25ae0 con 0x55aab2b231c0 2023-10-05T19:29:33.457 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.454+0000 7f0213ce0700 1 -- 172.21.15.167:0/2873792612 <== osd.3 v2:172.21.15.167:6808/30036 9 ==== osd_op_reply(34 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f0204014070 con 0x55aab2b231c0 2023-10-05T19:29:33.457 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.454+0000 7f0213ce0700 1 -- 172.21.15.167:0/2873792612 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:35 5.17 5:e8000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x55aab2b25ae0 con 0x7f01fc007030 2023-10-05T19:29:33.458 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.454+0000 7f02144e1700 1 -- 172.21.15.167:0/2873792612 <== osd.0 v2:172.21.15.118:6800/27690 9 ==== osd_op_reply(35 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f020c00a5a0 con 0x7f01fc007030 2023-10-05T19:29:33.458 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.454+0000 7f02144e1700 1 -- 172.21.15.167:0/2873792612 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:36 5.f 5:f0000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x55aab2b25ae0 con 0x55aab2af2600 2023-10-05T19:29:33.459 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.454+0000 7f02144e1700 1 -- 172.21.15.167:0/2873792612 <== osd.1 v2:172.21.15.167:6800/30035 13 ==== osd_op_reply(36 [pgnls start_epoch 32 out=115b] v23'2 uv2 ondisk = 1) v8 ==== 144+0+115 (crc 0 0 0) 0x7f020c00a5a0 con 0x55aab2af2600 2023-10-05T19:29:33.459 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.454+0000 7f02144e1700 1 -- 172.21.15.167:0/2873792612 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:37 5.1f 5:f8000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x55aab2b25ae0 con 0x7f020402d020 2023-10-05T19:29:33.460 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.454+0000 7f02134df700 1 -- 172.21.15.167:0/2873792612 <== osd.2 v2:172.21.15.167:6816/30037 6 ==== osd_op_reply(37 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f01fc029040 con 0x7f020402d020 2023-10-05T19:29:33.460 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.454+0000 7f02161abe40 20 RGWRados::pool_iterate: got zonegroup_info.4f1f719d-9648-417b-b10c-e5d939825419 2023-10-05T19:29:33.460 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.454+0000 7f02161abe40 20 RGWRados::pool_iterate: got zone_info.9a9df965-6612-45d4-a45f-6e3c889bacc2 2023-10-05T19:29:33.460 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.454+0000 7f02161abe40 20 RGWRados::pool_iterate: got zone_names.default 2023-10-05T19:29:33.460 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.454+0000 7f02161abe40 20 RGWRados::pool_iterate: got zonegroups_names.default 2023-10-05T19:29:33.460 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.454+0000 7f02161abe40 20 rados->read ofs=0 len=0 2023-10-05T19:29:33.460 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.454+0000 7f02161abe40 1 -- 172.21.15.167:0/2873792612 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:38 5.12 5:49953fa1:::default.realm:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32) v8 -- 0x55aab2b21f50 con 0x55aab2af2600 2023-10-05T19:29:33.461 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.454+0000 7f02144e1700 1 -- 172.21.15.167:0/2873792612 <== osd.1 v2:172.21.15.167:6800/30035 14 ==== osd_op_reply(38 default.realm [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 ==== 157+0+0 (crc 0 0 0) 0x7f020c00a5a0 con 0x55aab2af2600 2023-10-05T19:29:33.461 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.454+0000 7f02161abe40 20 rados_obj.operate() r=-2 bl.length=0 2023-10-05T19:29:33.461 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.454+0000 7f02161abe40 20 rados->read ofs=0 len=0 2023-10-05T19:29:33.461 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.454+0000 7f02161abe40 1 -- 172.21.15.167:0/2873792612 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:39 5.f 5:f4c53578:::zonegroups_names.default:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32) v8 -- 0x55aab2b228d0 con 0x55aab2af2600 2023-10-05T19:29:33.462 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.458+0000 7f02144e1700 1 -- 172.21.15.167:0/2873792612 <== osd.1 v2:172.21.15.167:6800/30035 15 ==== osd_op_reply(39 zonegroups_names.default [read 0~46 out=46b] v0'0 uv2 ondisk = 0) v8 ==== 168+0+46 (crc 0 0 0) 0x7f020c00a5a0 con 0x55aab2af2600 2023-10-05T19:29:33.462 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.458+0000 7f02161abe40 20 rados_obj.operate() r=0 bl.length=46 2023-10-05T19:29:33.462 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.458+0000 7f02161abe40 20 rados->read ofs=0 len=0 2023-10-05T19:29:33.462 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.458+0000 7f02161abe40 1 -- 172.21.15.167:0/2873792612 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:40 5.c 5:30d299a5:::zonegroup_info.4f1f719d-9648-417b-b10c-e5d939825419:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32) v8 -- 0x55aab2b228e0 con 0x7f01fc007030 2023-10-05T19:29:33.463 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.458+0000 7f02144e1700 1 -- 172.21.15.167:0/2873792612 <== osd.0 v2:172.21.15.118:6800/27690 10 ==== osd_op_reply(40 zonegroup_info.4f1f719d-9648-417b-b10c-e5d939825419 [read 0~362 out=362b] v0'0 uv1 ondisk = 0) v8 ==== 195+0+362 (crc 0 0 0) 0x7f020c00a5a0 con 0x7f01fc007030 2023-10-05T19:29:33.463 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.458+0000 7f02161abe40 20 rados_obj.operate() r=0 bl.length=362 2023-10-05T19:29:33.463 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.458+0000 7f02161abe40 20 rados->read ofs=0 len=0 2023-10-05T19:29:33.463 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.458+0000 7f02161abe40 1 -- 172.21.15.167:0/2873792612 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:41 5.13 5:cbbf6c60:::converted:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32) v8 -- 0x55aab2b228e0 con 0x55aab2b231c0 2023-10-05T19:29:33.464 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.458+0000 7f0213ce0700 1 -- 172.21.15.167:0/2873792612 <== osd.3 v2:172.21.15.167:6808/30036 10 ==== osd_op_reply(41 converted [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 ==== 153+0+0 (crc 0 0 0) 0x7f0204014070 con 0x55aab2b231c0 2023-10-05T19:29:33.464 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.458+0000 7f02161abe40 20 rados_obj.operate() r=-2 bl.length=0 2023-10-05T19:29:33.464 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.458+0000 7f02161abe40 20 rados->read ofs=0 len=0 2023-10-05T19:29:33.464 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.458+0000 7f02161abe40 1 -- 172.21.15.167:0/2873792612 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:42 5.12 5:49953fa1:::default.realm:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32) v8 -- 0x55aab2b25e90 con 0x55aab2af2600 2023-10-05T19:29:33.464 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.458+0000 7f02144e1700 1 -- 172.21.15.167:0/2873792612 <== osd.1 v2:172.21.15.167:6800/30035 16 ==== osd_op_reply(42 default.realm [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 ==== 157+0+0 (crc 0 0 0) 0x7f020c00a5a0 con 0x55aab2af2600 2023-10-05T19:29:33.464 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.458+0000 7f02161abe40 20 rados_obj.operate() r=-2 bl.length=0 2023-10-05T19:29:33.465 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.458+0000 7f02161abe40 20 rados->read ofs=0 len=0 2023-10-05T19:29:33.465 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.458+0000 7f02161abe40 1 -- 172.21.15.167:0/2873792612 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:43 5.f 5:f4c53578:::zonegroups_names.default:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32) v8 -- 0x55aab2b25e90 con 0x55aab2af2600 2023-10-05T19:29:33.465 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.462+0000 7f02144e1700 1 -- 172.21.15.167:0/2873792612 <== osd.1 v2:172.21.15.167:6800/30035 17 ==== osd_op_reply(43 zonegroups_names.default [read 0~46 out=46b] v0'0 uv2 ondisk = 0) v8 ==== 168+0+46 (crc 0 0 0) 0x7f020c00a5a0 con 0x55aab2af2600 2023-10-05T19:29:33.465 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.462+0000 7f02161abe40 20 rados_obj.operate() r=0 bl.length=46 2023-10-05T19:29:33.465 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.462+0000 7f02161abe40 1 -- 172.21.15.167:0/2873792612 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:44 5.0 5:00000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x55aab2b26ea0 con 0x55aab2b231c0 2023-10-05T19:29:33.466 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.462+0000 7f0213ce0700 1 -- 172.21.15.167:0/2873792612 <== osd.3 v2:172.21.15.167:6808/30036 11 ==== osd_op_reply(44 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f0204014070 con 0x55aab2b231c0 2023-10-05T19:29:33.466 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.462+0000 7f0213ce0700 1 -- 172.21.15.167:0/2873792612 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:45 5.10 5:08000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x55aab2b26ea0 con 0x7f020402d020 2023-10-05T19:29:33.467 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.462+0000 7f02134df700 1 -- 172.21.15.167:0/2873792612 <== osd.2 v2:172.21.15.167:6816/30037 7 ==== osd_op_reply(45 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f01fc029040 con 0x7f020402d020 2023-10-05T19:29:33.467 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.462+0000 7f02134df700 1 -- 172.21.15.167:0/2873792612 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:46 5.8 5:10000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x55aab2b26ea0 con 0x7f01fc007030 2023-10-05T19:29:33.468 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.462+0000 7f02144e1700 1 -- 172.21.15.167:0/2873792612 <== osd.0 v2:172.21.15.118:6800/27690 11 ==== osd_op_reply(46 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f020c00a5a0 con 0x7f01fc007030 2023-10-05T19:29:33.468 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.462+0000 7f02144e1700 1 -- 172.21.15.167:0/2873792612 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:47 5.18 5:18000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x55aab2b21f50 con 0x55aab2af2600 2023-10-05T19:29:33.469 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.466+0000 7f02144e1700 1 -- 172.21.15.167:0/2873792612 <== osd.1 v2:172.21.15.167:6800/30035 18 ==== osd_op_reply(47 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f020c00a5a0 con 0x55aab2af2600 2023-10-05T19:29:33.469 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.466+0000 7f02144e1700 1 -- 172.21.15.167:0/2873792612 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:48 5.4 5:20000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x55aab2b21f50 con 0x55aab2af2600 2023-10-05T19:29:33.469 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.466+0000 7f02144e1700 1 -- 172.21.15.167:0/2873792612 <== osd.1 v2:172.21.15.167:6800/30035 19 ==== osd_op_reply(48 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f020c00a5a0 con 0x55aab2af2600 2023-10-05T19:29:33.470 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.466+0000 7f02144e1700 1 -- 172.21.15.167:0/2873792612 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:49 5.14 5:28000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x55aab2b21f50 con 0x7f01fc007030 2023-10-05T19:29:33.474 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.466+0000 7f02144e1700 1 -- 172.21.15.167:0/2873792612 <== osd.0 v2:172.21.15.118:6800/27690 12 ==== osd_op_reply(49 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f020c00a5a0 con 0x7f01fc007030 2023-10-05T19:29:33.474 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.466+0000 7f02144e1700 1 -- 172.21.15.167:0/2873792612 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:50 5.c 5:30000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x55aab2b21f50 con 0x7f01fc007030 2023-10-05T19:29:33.474 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.466+0000 7f02144e1700 1 -- 172.21.15.167:0/2873792612 <== osd.0 v2:172.21.15.118:6800/27690 13 ==== osd_op_reply(50 [pgnls start_epoch 32 out=112b] v23'1 uv1 ondisk = 1) v8 ==== 144+0+112 (crc 0 0 0) 0x7f020c00a5a0 con 0x7f01fc007030 2023-10-05T19:29:33.474 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.466+0000 7f02144e1700 1 -- 172.21.15.167:0/2873792612 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:51 5.1c 5:38000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x55aab2b21f50 con 0x7f020402d020 2023-10-05T19:29:33.474 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.470+0000 7f02134df700 1 -- 172.21.15.167:0/2873792612 <== osd.2 v2:172.21.15.167:6816/30037 8 ==== osd_op_reply(51 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f01fc029040 con 0x7f020402d020 2023-10-05T19:29:33.474 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.470+0000 7f02134df700 1 -- 172.21.15.167:0/2873792612 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:52 5.2 5:40000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x55aab2b21f50 con 0x55aab2b231c0 2023-10-05T19:29:33.475 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.470+0000 7f0213ce0700 1 -- 172.21.15.167:0/2873792612 <== osd.3 v2:172.21.15.167:6808/30036 12 ==== osd_op_reply(52 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f0204014070 con 0x55aab2b231c0 2023-10-05T19:29:33.475 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.470+0000 7f0213ce0700 1 -- 172.21.15.167:0/2873792612 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:53 5.12 5:48000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x55aab2b21f50 con 0x55aab2af2600 2023-10-05T19:29:33.475 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.470+0000 7f02144e1700 1 -- 172.21.15.167:0/2873792612 <== osd.1 v2:172.21.15.167:6800/30035 20 ==== osd_op_reply(53 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f020c00a5a0 con 0x55aab2af2600 2023-10-05T19:29:33.475 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.470+0000 7f02144e1700 1 -- 172.21.15.167:0/2873792612 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:54 5.a 5:50000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x55aab2b21f50 con 0x7f01fc007030 2023-10-05T19:29:33.475 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.470+0000 7f02144e1700 1 -- 172.21.15.167:0/2873792612 <== osd.0 v2:172.21.15.118:6800/27690 14 ==== osd_op_reply(54 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f020c00a5a0 con 0x7f01fc007030 2023-10-05T19:29:33.476 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.470+0000 7f02144e1700 1 -- 172.21.15.167:0/2873792612 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:55 5.1a 5:58000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x55aab2b21f50 con 0x55aab2af2600 2023-10-05T19:29:33.476 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.470+0000 7f02144e1700 1 -- 172.21.15.167:0/2873792612 <== osd.1 v2:172.21.15.167:6800/30035 21 ==== osd_op_reply(55 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f020c00a5a0 con 0x55aab2af2600 2023-10-05T19:29:33.476 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.470+0000 7f02144e1700 1 -- 172.21.15.167:0/2873792612 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:56 5.6 5:60000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x55aab2b21f50 con 0x7f020402d020 2023-10-05T19:29:33.477 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.474+0000 7f02134df700 1 -- 172.21.15.167:0/2873792612 <== osd.2 v2:172.21.15.167:6816/30037 9 ==== osd_op_reply(56 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f01fc029040 con 0x7f020402d020 2023-10-05T19:29:33.477 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.474+0000 7f02134df700 1 -- 172.21.15.167:0/2873792612 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:57 5.16 5:68000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x55aab2b21f50 con 0x55aab2b231c0 2023-10-05T19:29:33.478 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.474+0000 7f0213ce0700 1 -- 172.21.15.167:0/2873792612 <== osd.3 v2:172.21.15.167:6808/30036 13 ==== osd_op_reply(57 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f0204014070 con 0x55aab2b231c0 2023-10-05T19:29:33.478 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.474+0000 7f0213ce0700 1 -- 172.21.15.167:0/2873792612 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:58 5.e 5:70000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x55aab2b21f50 con 0x55aab2b231c0 2023-10-05T19:29:33.479 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.474+0000 7f0213ce0700 1 -- 172.21.15.167:0/2873792612 <== osd.3 v2:172.21.15.167:6808/30036 14 ==== osd_op_reply(58 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f020404d040 con 0x55aab2b231c0 2023-10-05T19:29:33.479 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.474+0000 7f0213ce0700 1 -- 172.21.15.167:0/2873792612 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:59 5.1e 5:78000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x55aab2b21f50 con 0x7f01fc007030 2023-10-05T19:29:33.480 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.474+0000 7f02144e1700 1 -- 172.21.15.167:0/2873792612 <== osd.0 v2:172.21.15.118:6800/27690 15 ==== osd_op_reply(59 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f020c03f040 con 0x7f01fc007030 2023-10-05T19:29:33.480 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.474+0000 7f02144e1700 1 -- 172.21.15.167:0/2873792612 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:60 5.1 5:80000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x55aab2b21f50 con 0x55aab2b231c0 2023-10-05T19:29:33.480 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.478+0000 7f0213ce0700 1 -- 172.21.15.167:0/2873792612 <== osd.3 v2:172.21.15.167:6808/30036 15 ==== osd_op_reply(60 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f020404d040 con 0x55aab2b231c0 2023-10-05T19:29:33.481 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.478+0000 7f0213ce0700 1 -- 172.21.15.167:0/2873792612 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:61 5.11 5:88000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x55aab2b21f50 con 0x55aab2b231c0 2023-10-05T19:29:33.481 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.478+0000 7f0213ce0700 1 -- 172.21.15.167:0/2873792612 <== osd.3 v2:172.21.15.167:6808/30036 16 ==== osd_op_reply(61 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f020404d040 con 0x55aab2b231c0 2023-10-05T19:29:33.482 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.478+0000 7f0213ce0700 1 -- 172.21.15.167:0/2873792612 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:62 5.9 5:90000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x55aab2b21f50 con 0x55aab2af2600 2023-10-05T19:29:33.482 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.478+0000 7f02144e1700 1 -- 172.21.15.167:0/2873792612 <== osd.1 v2:172.21.15.167:6800/30035 22 ==== osd_op_reply(62 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f020c03f040 con 0x55aab2af2600 2023-10-05T19:29:33.482 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.478+0000 7f02144e1700 1 -- 172.21.15.167:0/2873792612 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:63 5.19 5:98000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x55aab2b21f50 con 0x55aab2af2600 2023-10-05T19:29:33.483 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.478+0000 7f02144e1700 1 -- 172.21.15.167:0/2873792612 <== osd.1 v2:172.21.15.167:6800/30035 23 ==== osd_op_reply(63 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f020c03f040 con 0x55aab2af2600 2023-10-05T19:29:33.483 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.478+0000 7f02144e1700 1 -- 172.21.15.167:0/2873792612 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:64 5.5 5:a0000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x55aab2b21f50 con 0x7f01fc007030 2023-10-05T19:29:33.484 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.478+0000 7f02144e1700 1 -- 172.21.15.167:0/2873792612 <== osd.0 v2:172.21.15.118:6800/27690 16 ==== osd_op_reply(64 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f020c03f040 con 0x7f01fc007030 2023-10-05T19:29:33.484 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.478+0000 7f02144e1700 1 -- 172.21.15.167:0/2873792612 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:65 5.15 5:a8000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x55aab2b21f50 con 0x55aab2b231c0 2023-10-05T19:29:33.485 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.482+0000 7f0213ce0700 1 -- 172.21.15.167:0/2873792612 <== osd.3 v2:172.21.15.167:6808/30036 17 ==== osd_op_reply(65 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f020404d040 con 0x55aab2b231c0 2023-10-05T19:29:33.485 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.482+0000 7f0213ce0700 1 -- 172.21.15.167:0/2873792612 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:66 5.d 5:b0000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x55aab2b21f50 con 0x7f020402d020 2023-10-05T19:29:33.485 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.482+0000 7f02134df700 1 -- 172.21.15.167:0/2873792612 <== osd.2 v2:172.21.15.167:6816/30037 10 ==== osd_op_reply(66 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f01fc029040 con 0x7f020402d020 2023-10-05T19:29:33.486 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.482+0000 7f02134df700 1 -- 172.21.15.167:0/2873792612 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:67 5.1d 5:b8000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x55aab2b21f50 con 0x55aab2af2600 2023-10-05T19:29:33.486 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.482+0000 7f02144e1700 1 -- 172.21.15.167:0/2873792612 <== osd.1 v2:172.21.15.167:6800/30035 24 ==== osd_op_reply(67 [pgnls start_epoch 32 out=107b] v30'2 uv2 ondisk = 1) v8 ==== 144+0+107 (crc 0 0 0) 0x7f020c03f040 con 0x55aab2af2600 2023-10-05T19:29:33.486 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.482+0000 7f02144e1700 1 -- 172.21.15.167:0/2873792612 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:68 5.3 5:c0000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x55aab2b21f50 con 0x7f01fc007030 2023-10-05T19:29:33.487 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.482+0000 7f02144e1700 1 -- 172.21.15.167:0/2873792612 <== osd.0 v2:172.21.15.118:6800/27690 17 ==== osd_op_reply(68 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f020c03f040 con 0x7f01fc007030 2023-10-05T19:29:33.488 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.482+0000 7f02144e1700 1 -- 172.21.15.167:0/2873792612 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:69 5.13 5:c8000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x55aab2b21f50 con 0x55aab2b231c0 2023-10-05T19:29:33.488 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.482+0000 7f0213ce0700 1 -- 172.21.15.167:0/2873792612 <== osd.3 v2:172.21.15.167:6808/30036 18 ==== osd_op_reply(69 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f020404d040 con 0x55aab2b231c0 2023-10-05T19:29:33.488 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.482+0000 7f0213ce0700 1 -- 172.21.15.167:0/2873792612 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:70 5.b 5:d0000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x55aab2b21f50 con 0x7f01fc007030 2023-10-05T19:29:33.489 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.486+0000 7f02144e1700 1 -- 172.21.15.167:0/2873792612 <== osd.0 v2:172.21.15.118:6800/27690 18 ==== osd_op_reply(70 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f020c03f040 con 0x7f01fc007030 2023-10-05T19:29:33.489 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.486+0000 7f02144e1700 1 -- 172.21.15.167:0/2873792612 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:71 5.1b 5:d8000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x55aab2b21f50 con 0x7f020402d020 2023-10-05T19:29:33.490 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.486+0000 7f02134df700 1 -- 172.21.15.167:0/2873792612 <== osd.2 v2:172.21.15.167:6816/30037 11 ==== osd_op_reply(71 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f01fc029040 con 0x7f020402d020 2023-10-05T19:29:33.490 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.486+0000 7f02134df700 1 -- 172.21.15.167:0/2873792612 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:72 5.7 5:e0000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x55aab2b21f50 con 0x55aab2b231c0 2023-10-05T19:29:33.491 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.486+0000 7f0213ce0700 1 -- 172.21.15.167:0/2873792612 <== osd.3 v2:172.21.15.167:6808/30036 19 ==== osd_op_reply(72 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f020404d040 con 0x55aab2b231c0 2023-10-05T19:29:33.491 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.486+0000 7f0213ce0700 1 -- 172.21.15.167:0/2873792612 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:73 5.17 5:e8000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x55aab2b21f50 con 0x7f01fc007030 2023-10-05T19:29:33.492 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.486+0000 7f02144e1700 1 -- 172.21.15.167:0/2873792612 <== osd.0 v2:172.21.15.118:6800/27690 19 ==== osd_op_reply(73 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f020c03f040 con 0x7f01fc007030 2023-10-05T19:29:33.492 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.486+0000 7f02144e1700 1 -- 172.21.15.167:0/2873792612 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:74 5.f 5:f0000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x55aab2b21f50 con 0x55aab2af2600 2023-10-05T19:29:33.493 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.490+0000 7f02144e1700 1 -- 172.21.15.167:0/2873792612 <== osd.1 v2:172.21.15.167:6800/30035 25 ==== osd_op_reply(74 [pgnls start_epoch 32 out=115b] v23'2 uv2 ondisk = 1) v8 ==== 144+0+115 (crc 0 0 0) 0x7f020c03f040 con 0x55aab2af2600 2023-10-05T19:29:33.493 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.490+0000 7f02144e1700 1 -- 172.21.15.167:0/2873792612 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:75 5.1f 5:f8000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x55aab2b21f50 con 0x7f020402d020 2023-10-05T19:29:33.494 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.490+0000 7f02134df700 1 -- 172.21.15.167:0/2873792612 <== osd.2 v2:172.21.15.167:6816/30037 12 ==== osd_op_reply(75 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f01fc029040 con 0x7f020402d020 2023-10-05T19:29:33.494 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.490+0000 7f02161abe40 20 RGWRados::pool_iterate: got zonegroup_info.4f1f719d-9648-417b-b10c-e5d939825419 2023-10-05T19:29:33.494 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.490+0000 7f02161abe40 20 RGWRados::pool_iterate: got zone_info.9a9df965-6612-45d4-a45f-6e3c889bacc2 2023-10-05T19:29:33.494 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.490+0000 7f02161abe40 20 RGWRados::pool_iterate: got zone_names.default 2023-10-05T19:29:33.494 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.490+0000 7f02161abe40 20 RGWRados::pool_iterate: got zonegroups_names.default 2023-10-05T19:29:33.494 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.490+0000 7f02161abe40 20 rados->read ofs=0 len=0 2023-10-05T19:29:33.495 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.490+0000 7f02161abe40 1 -- 172.21.15.167:0/2873792612 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:76 5.f 5:f43b6ece:::zone_names.default:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32) v8 -- 0x55aab2b25e90 con 0x55aab2af2600 2023-10-05T19:29:33.495 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.490+0000 7f02144e1700 1 -- 172.21.15.167:0/2873792612 <== osd.1 v2:172.21.15.167:6800/30035 26 ==== osd_op_reply(76 zone_names.default [read 0~46 out=46b] v0'0 uv1 ondisk = 0) v8 ==== 162+0+46 (crc 0 0 0) 0x7f020c03f040 con 0x55aab2af2600 2023-10-05T19:29:33.495 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.490+0000 7f02161abe40 20 rados_obj.operate() r=0 bl.length=46 2023-10-05T19:29:33.495 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.490+0000 7f02161abe40 20 rados->read ofs=0 len=0 2023-10-05T19:29:33.495 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.490+0000 7f02161abe40 1 -- 172.21.15.167:0/2873792612 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:77 5.1d 5:b8374eb4:::zone_info.9a9df965-6612-45d4-a45f-6e3c889bacc2:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32) v8 -- 0x55aab2b26240 con 0x55aab2af2600 2023-10-05T19:29:33.496 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.490+0000 7f02144e1700 1 -- 172.21.15.167:0/2873792612 <== osd.1 v2:172.21.15.167:6800/30035 27 ==== osd_op_reply(77 zone_info.9a9df965-6612-45d4-a45f-6e3c889bacc2 [read 0~874 out=874b] v0'0 uv2 ondisk = 0) v8 ==== 190+0+874 (crc 0 0 0) 0x7f020c03f040 con 0x55aab2af2600 2023-10-05T19:29:33.496 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.490+0000 7f02161abe40 20 rados_obj.operate() r=0 bl.length=874 2023-10-05T19:29:33.496 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.490+0000 7f02161abe40 20 rados->read ofs=0 len=0 2023-10-05T19:29:33.496 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.490+0000 7f02161abe40 1 -- 172.21.15.167:0/2873792612 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:78 5.f 5:f4c53578:::zonegroups_names.default:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32) v8 -- 0x55aab2b26680 con 0x55aab2af2600 2023-10-05T19:29:33.497 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.494+0000 7f02144e1700 1 -- 172.21.15.167:0/2873792612 <== osd.1 v2:172.21.15.167:6800/30035 28 ==== osd_op_reply(78 zonegroups_names.default [read 0~46 out=46b] v0'0 uv2 ondisk = 0) v8 ==== 168+0+46 (crc 0 0 0) 0x7f020c03f040 con 0x55aab2af2600 2023-10-05T19:29:33.497 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.494+0000 7f02161abe40 20 rados_obj.operate() r=0 bl.length=46 2023-10-05T19:29:33.497 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.494+0000 7f02161abe40 20 rados->read ofs=0 len=0 2023-10-05T19:29:33.497 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.494+0000 7f02161abe40 1 -- 172.21.15.167:0/2873792612 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:79 5.c 5:30d299a5:::zonegroup_info.4f1f719d-9648-417b-b10c-e5d939825419:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32) v8 -- 0x55aab2b26680 con 0x7f01fc007030 2023-10-05T19:29:33.498 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.494+0000 7f02144e1700 1 -- 172.21.15.167:0/2873792612 <== osd.0 v2:172.21.15.118:6800/27690 20 ==== osd_op_reply(79 zonegroup_info.4f1f719d-9648-417b-b10c-e5d939825419 [read 0~362 out=362b] v0'0 uv1 ondisk = 0) v8 ==== 195+0+362 (crc 0 0 0) 0x7f020c052040 con 0x7f01fc007030 2023-10-05T19:29:33.498 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.494+0000 7f02161abe40 20 rados_obj.operate() r=0 bl.length=362 2023-10-05T19:29:33.498 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.494+0000 7f02161abe40 20 rados->read ofs=0 len=0 2023-10-05T19:29:33.498 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.494+0000 7f02161abe40 1 -- 172.21.15.167:0/2873792612 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:80 5.3 5:c0ad8cbd:::region_map:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32) v8 -- 0x55aab2b26ab0 con 0x7f01fc007030 2023-10-05T19:29:33.499 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.494+0000 7f02144e1700 1 -- 172.21.15.167:0/2873792612 <== osd.0 v2:172.21.15.118:6800/27690 21 ==== osd_op_reply(80 region_map [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 ==== 154+0+0 (crc 0 0 0) 0x7f020c052040 con 0x7f01fc007030 2023-10-05T19:29:33.499 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.494+0000 7f02161abe40 20 rados_obj.operate() r=-2 bl.length=0 2023-10-05T19:29:33.499 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.494+0000 7f02161abe40 20 zone default found 2023-10-05T19:29:33.499 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.494+0000 7f02161abe40 4 Realm: () 2023-10-05T19:29:33.499 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.494+0000 7f02161abe40 4 ZoneGroup: default (4f1f719d-9648-417b-b10c-e5d939825419) 2023-10-05T19:29:33.500 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.494+0000 7f02161abe40 4 Zone: default (9a9df965-6612-45d4-a45f-6e3c889bacc2) 2023-10-05T19:29:33.500 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.494+0000 7f02161abe40 10 cannot find current period zonegroup using local zonegroup configuration 2023-10-05T19:29:33.500 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.494+0000 7f02161abe40 20 zonegroup default 2023-10-05T19:29:33.500 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.494+0000 7f02161abe40 20 rados->read ofs=0 len=0 2023-10-05T19:29:33.500 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.494+0000 7f02161abe40 1 -- 172.21.15.167:0/2873792612 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:81 5.3 5:c52100b6:::period_config.default:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32) v8 -- 0x55aab2b26900 con 0x7f01fc007030 2023-10-05T19:29:33.501 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.494+0000 7f02144e1700 1 -- 172.21.15.167:0/2873792612 <== osd.0 v2:172.21.15.118:6800/27690 22 ==== osd_op_reply(81 period_config.default [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 ==== 165+0+0 (crc 0 0 0) 0x7f020c052040 con 0x7f01fc007030 2023-10-05T19:29:33.501 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.494+0000 7f02161abe40 20 rados_obj.operate() r=-2 bl.length=0 2023-10-05T19:29:33.501 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.494+0000 7f02161abe40 20 rados->read ofs=0 len=0 2023-10-05T19:29:33.501 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.494+0000 7f02161abe40 1 -- 172.21.15.167:0/2873792612 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:82 6.2 6:4347d321:::bucket.sync-source-hints.:head [call version.read in=11b,read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32) v8 -- 0x55aab2b28280 con 0x55aab2b231c0 2023-10-05T19:29:33.501 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.498+0000 7f0213ce0700 1 -- 172.21.15.167:0/2873792612 <== osd.3 v2:172.21.15.167:6808/30036 20 ==== osd_op_reply(82 bucket.sync-source-hints. [call,read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 ==== 211+0+0 (crc 0 0 0) 0x7f020404d040 con 0x55aab2b231c0 2023-10-05T19:29:33.502 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.498+0000 7f02161abe40 20 rados_obj.operate() r=-2 bl.length=0 2023-10-05T19:29:33.502 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.498+0000 7f02161abe40 20 rados->read ofs=0 len=0 2023-10-05T19:29:33.502 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.498+0000 7f02161abe40 1 -- 172.21.15.167:0/2873792612 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:83 6.b 6:d467b91b:::bucket.sync-target-hints.:head [call version.read in=11b,read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32) v8 -- 0x55aab2b28630 con 0x55aab2b231c0 2023-10-05T19:29:33.502 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.498+0000 7f0213ce0700 1 -- 172.21.15.167:0/2873792612 <== osd.3 v2:172.21.15.167:6808/30036 21 ==== osd_op_reply(83 bucket.sync-target-hints. [call,read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 ==== 211+0+0 (crc 0 0 0) 0x7f020404d040 con 0x55aab2b231c0 2023-10-05T19:29:33.502 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.498+0000 7f02161abe40 20 rados_obj.operate() r=-2 bl.length=0 2023-10-05T19:29:33.502 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.498+0000 7f02161abe40 20 started sync module instance, tier type = 2023-10-05T19:29:33.503 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.498+0000 7f02161abe40 20 started zone id=9a9df965-6612-45d4-a45f-6e3c889bacc2 (name=default) with tier type = 2023-10-05T19:29:33.503 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.498+0000 7f02161abe40 1 -- 172.21.15.167:0/2873792612 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:84 7.1f 7:f95f44c2:::notify.0:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e32) v8 -- 0x55aab2b27940 con 0x7f020402d020 2023-10-05T19:29:33.505 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.502+0000 7f02134df700 1 -- 172.21.15.167:0/2873792612 <== osd.2 v2:172.21.15.167:6816/30037 13 ==== osd_op_reply(84 notify.0 [create] v32'225 uv225 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f01fc029040 con 0x7f020402d020 2023-10-05T19:29:33.505 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.502+0000 7f02161abe40 1 -- 172.21.15.167:0/2873792612 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:85 7.1f 7:f95f44c2:::notify.0:head [watch watch cookie 94191630843200] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e32) v8 -- 0x55aab2b22470 con 0x7f020402d020 2023-10-05T19:29:33.505 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.502+0000 7f02161abe40 1 -- 172.21.15.167:0/2873792612 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:86 7.0 7:05bf5b68:::notify.1:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e32) v8 -- 0x55aab2b293c0 con 0x55aab2af2600 2023-10-05T19:29:33.508 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.502+0000 7f02134df700 1 -- 172.21.15.167:0/2873792612 <== osd.2 v2:172.21.15.167:6816/30037 14 ==== osd_op_reply(85 notify.0 [watch watch cookie 94191630843200] v32'226 uv225 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f01fc029040 con 0x7f020402d020 2023-10-05T19:29:33.508 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.502+0000 7f02144e1700 1 -- 172.21.15.167:0/2873792612 <== osd.1 v2:172.21.15.167:6800/30035 29 ==== osd_op_reply(86 notify.1 [create] v32'225 uv225 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f020c052040 con 0x55aab2af2600 2023-10-05T19:29:33.508 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.502+0000 7f02161abe40 1 -- 172.21.15.167:0/2873792612 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:87 7.0 7:05bf5b68:::notify.1:head [watch watch cookie 94191630846720] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e32) v8 -- 0x55aab2b294c0 con 0x55aab2af2600 2023-10-05T19:29:33.508 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.502+0000 7f02161abe40 1 -- 172.21.15.167:0/2873792612 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:88 7.15 7:a93a5511:::notify.2:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e32) v8 -- 0x55aab2b29eb0 con 0x55aab2b231c0 2023-10-05T19:29:33.511 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.506+0000 7f0213ce0700 1 -- 172.21.15.167:0/2873792612 <== osd.3 v2:172.21.15.167:6808/30036 22 ==== osd_op_reply(88 notify.2 [create] v32'225 uv225 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f020404d040 con 0x55aab2b231c0 2023-10-05T19:29:33.511 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.506+0000 7f02161abe40 1 -- 172.21.15.167:0/2873792612 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:89 7.15 7:a93a5511:::notify.2:head [watch watch cookie 94191630850432] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e32) v8 -- 0x55aab2b2a7a0 con 0x55aab2b231c0 2023-10-05T19:29:33.512 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.506+0000 7f02161abe40 1 -- 172.21.15.167:0/2873792612 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:90 7.e 7:7759931f:::notify.3:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e32) v8 -- 0x55aab2b2b1b0 con 0x7f020402d020 2023-10-05T19:29:33.512 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.506+0000 7f02144e1700 1 -- 172.21.15.167:0/2873792612 <== osd.1 v2:172.21.15.167:6800/30035 30 ==== osd_op_reply(87 notify.1 [watch watch cookie 94191630846720] v32'226 uv225 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f020c052040 con 0x55aab2af2600 2023-10-05T19:29:33.513 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.510+0000 7f02134df700 1 -- 172.21.15.167:0/2873792612 <== osd.2 v2:172.21.15.167:6816/30037 15 ==== osd_op_reply(90 notify.3 [create] v32'225 uv225 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f01fc029040 con 0x7f020402d020 2023-10-05T19:29:33.514 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.510+0000 7f02161abe40 1 -- 172.21.15.167:0/2873792612 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:91 7.e 7:7759931f:::notify.3:head [watch watch cookie 94191630847824] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e32) v8 -- 0x55aab2b2ade0 con 0x7f020402d020 2023-10-05T19:29:33.514 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.510+0000 7f02161abe40 1 -- 172.21.15.167:0/2873792612 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:92 7.d 7:b4812045:::notify.4:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e32) v8 -- 0x55aab2b2c1d0 con 0x55aab2af2600 2023-10-05T19:29:33.514 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.510+0000 7f0213ce0700 1 -- 172.21.15.167:0/2873792612 <== osd.3 v2:172.21.15.167:6808/30036 23 ==== osd_op_reply(89 notify.2 [watch watch cookie 94191630850432] v32'226 uv225 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f020404d040 con 0x55aab2b231c0 2023-10-05T19:29:33.516 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.510+0000 7f02134df700 1 -- 172.21.15.167:0/2873792612 <== osd.2 v2:172.21.15.167:6816/30037 16 ==== osd_op_reply(91 notify.3 [watch watch cookie 94191630847824] v32'226 uv225 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f01fc029040 con 0x7f020402d020 2023-10-05T19:29:33.516 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.510+0000 7f02144e1700 1 -- 172.21.15.167:0/2873792612 <== osd.1 v2:172.21.15.167:6800/30035 31 ==== osd_op_reply(92 notify.4 [create] v32'225 uv225 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f020c052040 con 0x55aab2af2600 2023-10-05T19:29:33.516 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.510+0000 7f02161abe40 1 -- 172.21.15.167:0/2873792612 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:93 7.d 7:b4812045:::notify.4:head [watch watch cookie 94191630857568] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e32) v8 -- 0x55aab2b2bca0 con 0x55aab2af2600 2023-10-05T19:29:33.516 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.510+0000 7f02161abe40 1 -- 172.21.15.167:0/2873792612 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:94 7.3 7:c609908c:::notify.5:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e32) v8 -- 0x55aab2b2cac0 con 0x7f01fc007030 2023-10-05T19:29:33.518 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.514+0000 7f02144e1700 1 -- 172.21.15.167:0/2873792612 <== osd.1 v2:172.21.15.167:6800/30035 32 ==== osd_op_reply(93 notify.4 [watch watch cookie 94191630857568] v32'226 uv225 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f020c052040 con 0x55aab2af2600 2023-10-05T19:29:33.519 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.514+0000 7f02144e1700 1 -- 172.21.15.167:0/2873792612 <== osd.0 v2:172.21.15.118:6800/27690 23 ==== osd_op_reply(94 notify.5 [create] v32'225 uv225 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f020c052040 con 0x7f01fc007030 2023-10-05T19:29:33.519 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.514+0000 7f02161abe40 1 -- 172.21.15.167:0/2873792612 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:95 7.3 7:c609908c:::notify.5:head [watch watch cookie 94191630852784] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e32) v8 -- 0x55aab2b2ca70 con 0x7f01fc007030 2023-10-05T19:29:33.519 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.514+0000 7f02161abe40 1 -- 172.21.15.167:0/2873792612 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:96 7.14 7:2b04a3e9:::notify.6:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e32) v8 -- 0x55aab2b2d310 con 0x7f01fc007030 2023-10-05T19:29:33.521 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.518+0000 7f02144e1700 1 -- 172.21.15.167:0/2873792612 <== osd.0 v2:172.21.15.118:6800/27690 24 ==== osd_op_reply(95 notify.5 [watch watch cookie 94191630852784] v32'226 uv225 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f020c052040 con 0x7f01fc007030 2023-10-05T19:29:33.522 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.518+0000 7f02144e1700 1 -- 172.21.15.167:0/2873792612 <== osd.0 v2:172.21.15.118:6800/27690 25 ==== osd_op_reply(96 notify.6 [create] v32'225 uv225 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f020c052040 con 0x7f01fc007030 2023-10-05T19:29:33.522 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.518+0000 7f02161abe40 1 -- 172.21.15.167:0/2873792612 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:97 7.14 7:2b04a3e9:::notify.6:head [watch watch cookie 94191630862256] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e32) v8 -- 0x55aab2b2d0f0 con 0x7f01fc007030 2023-10-05T19:29:33.522 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.518+0000 7f02161abe40 1 -- 172.21.15.167:0/2873792612 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:98 7.9 7:93e5b521:::notify.7:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e32) v8 -- 0x55aab2b2dc70 con 0x7f01fc007030 2023-10-05T19:29:33.524 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.518+0000 7f02144e1700 1 -- 172.21.15.167:0/2873792612 <== osd.0 v2:172.21.15.118:6800/27690 26 ==== osd_op_reply(98 notify.7 [create] v32'225 uv225 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f020c052040 con 0x7f01fc007030 2023-10-05T19:29:33.524 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.518+0000 7f02161abe40 1 -- 172.21.15.167:0/2873792612 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:99 7.9 7:93e5b521:::notify.7:head [watch watch cookie 94191630866304] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e32) v8 -- 0x55aab2b2e0c0 con 0x7f01fc007030 2023-10-05T19:29:33.524 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.518+0000 7f02161abe40 20 add_watcher() i=0 2023-10-05T19:29:33.524 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.518+0000 7f02161abe40 20 add_watcher() i=1 2023-10-05T19:29:33.524 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.518+0000 7f02161abe40 20 add_watcher() i=2 2023-10-05T19:29:33.524 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.518+0000 7f02161abe40 20 add_watcher() i=3 2023-10-05T19:29:33.525 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.518+0000 7f02161abe40 20 add_watcher() i=4 2023-10-05T19:29:33.525 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.518+0000 7f02161abe40 20 add_watcher() i=5 2023-10-05T19:29:33.525 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.522+0000 7f02144e1700 1 -- 172.21.15.167:0/2873792612 <== osd.0 v2:172.21.15.118:6800/27690 27 ==== osd_op_reply(97 notify.6 [watch watch cookie 94191630862256] v32'226 uv225 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f020c052040 con 0x7f01fc007030 2023-10-05T19:29:33.525 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.522+0000 7f02161abe40 20 add_watcher() i=6 2023-10-05T19:29:33.526 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.522+0000 7f02144e1700 1 -- 172.21.15.167:0/2873792612 <== osd.0 v2:172.21.15.118:6800/27690 28 ==== osd_op_reply(99 notify.7 [watch watch cookie 94191630866304] v32'226 uv225 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f020c052040 con 0x7f01fc007030 2023-10-05T19:29:33.526 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.522+0000 7f02161abe40 20 add_watcher() i=7 2023-10-05T19:29:33.526 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.522+0000 7f02161abe40 2 all 8 watchers are set, enabling cache 2023-10-05T19:29:33.526 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.522+0000 7f02161abe40 1 -- 172.21.15.167:0/2873792612 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:100 6.4 6:22d26bf9:::data_loggenerations_metadata:head [call version.check_conds in=74b,call version.read in=11b,read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32) v8 -- 0x55aab2b2cc00 con 0x55aab2af2600 2023-10-05T19:29:33.527 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.522+0000 7f02144e1700 1 -- 172.21.15.167:0/2873792612 <== osd.1 v2:172.21.15.167:6800/30035 33 ==== osd_op_reply(100 data_loggenerations_metadata [call,call out=48b,read 0~28 out=28b] v0'0 uv1 ondisk = 0) v8 ==== 256+0+76 (crc 0 0 0) 0x7f020c052040 con 0x55aab2af2600 2023-10-05T19:29:33.527 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.522+0000 7f02161abe40 1 -- 172.21.15.167:0/2873792612 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:101 6.4 6:22d26bf9:::data_loggenerations_metadata:head [watch watch cookie 94191630867408] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e32) v8 -- 0x55aab2b2e530 con 0x55aab2af2600 2023-10-05T19:29:33.529 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.526+0000 7f02144e1700 1 -- 172.21.15.167:0/2873792612 <== osd.1 v2:172.21.15.167:6800/30035 34 ==== osd_op_reply(101 data_loggenerations_metadata [watch watch cookie 94191630867408] v32'223 uv1 ondisk = 0) v8 ==== 172+0+0 (crc 0 0 0) 0x7f020c052040 con 0x55aab2af2600 2023-10-05T19:29:33.529 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.526+0000 7f015ffff700 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2023-10-05T19:29:33.529 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.526+0000 7f02161abe40 20 check_secure_mon_conn(): auth registy supported: methods=[2] modes=[2,1] 2023-10-05T19:29:33.529 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.526+0000 7f02161abe40 20 check_secure_mon_conn(): mode 1 is insecure 2023-10-05T19:29:33.530 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.526+0000 7f02161abe40 1 Processor -- start 2023-10-05T19:29:33.530 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.526+0000 7f02161abe40 1 -- start start 2023-10-05T19:29:33.530 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.526+0000 7f02161abe40 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55aab2b3b340 0x55aab2b3b710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-05T19:29:33.531 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.526+0000 7f02161abe40 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x55aab2b3bc50 con 0x55aab2b3b340 2023-10-05T19:29:33.531 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.526+0000 7f02134df700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55aab2b3b340 0x55aab2b3b710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-05T19:29:33.531 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.526+0000 7f02134df700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55aab2b3b340 0x55aab2b3b710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.118:3300/0 says I am v2:172.21.15.167:46206/0 (socket says 172.21.15.167:46206) 2023-10-05T19:29:33.531 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.526+0000 7f02134df700 1 -- 172.21.15.167:0/131793748 learned_addr learned my addr 172.21.15.167:0/131793748 (peer_addr_for_me v2:172.21.15.167:0/0) 2023-10-05T19:29:33.531 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.526+0000 7f02134df700 1 -- 172.21.15.167:0/131793748 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55aab2b3bd90 con 0x55aab2b3b340 2023-10-05T19:29:33.531 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.526+0000 7f02134df700 1 --2- 172.21.15.167:0/131793748 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55aab2b3b340 0x55aab2b3b710 secure :-1 s=READY pgs=588 cs=0 l=1 rev1=1 crypto rx=0x55aab2aef0a0 tx=0x7f01fc030040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fe4d40b90a77791c server_cookie=0 in_seq=0 out_seq=0 2023-10-05T19:29:33.532 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.526+0000 7f015f7fe700 1 -- 172.21.15.167:0/131793748 <== mon.0 v2:172.21.15.118:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f01fc025040 con 0x55aab2b3b340 2023-10-05T19:29:33.532 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.526+0000 7f015f7fe700 1 -- 172.21.15.167:0/131793748 <== mon.0 v2:172.21.15.118:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f01fc034050 con 0x55aab2b3b340 2023-10-05T19:29:33.532 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.526+0000 7f015f7fe700 1 -- 172.21.15.167:0/131793748 <== mon.0 v2:172.21.15.118:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f01fc02e4e0 con 0x55aab2b3b340 2023-10-05T19:29:33.533 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.530+0000 7f02161abe40 1 -- 172.21.15.167:0/131793748 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55aab2b3b340 msgr2=0x55aab2b3b710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-05T19:29:33.533 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.530+0000 7f02161abe40 1 --2- 172.21.15.167:0/131793748 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55aab2b3b340 0x55aab2b3b710 secure :-1 s=READY pgs=588 cs=0 l=1 rev1=1 crypto rx=0x55aab2aef0a0 tx=0x7f01fc030040 comp rx=0 tx=0).stop 2023-10-05T19:29:33.533 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.530+0000 7f02161abe40 1 -- 172.21.15.167:0/131793748 shutdown_connections 2023-10-05T19:29:33.533 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.530+0000 7f02161abe40 1 --2- 172.21.15.167:0/131793748 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55aab2b3b340 0x55aab2b3b710 unknown :-1 s=CLOSED pgs=588 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-05T19:29:33.534 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.530+0000 7f02161abe40 1 -- 172.21.15.167:0/131793748 >> 172.21.15.167:0/131793748 conn(0x55aab2b36700 msgr2=0x55aab2b348b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-05T19:29:33.534 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.530+0000 7f02161abe40 1 -- 172.21.15.167:0/131793748 shutdown_connections 2023-10-05T19:29:33.534 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.530+0000 7f02161abe40 1 -- 172.21.15.167:0/131793748 wait complete. 2023-10-05T19:29:33.535 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.530+0000 7f02161abe40 1 Processor -- start 2023-10-05T19:29:33.535 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.530+0000 7f02161abe40 1 -- start start 2023-10-05T19:29:33.535 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.530+0000 7f02161abe40 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55aab2b3b340 0x55aab2b40db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-05T19:29:33.535 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.530+0000 7f02161abe40 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x55aab2b41db0 con 0x55aab2b3b340 2023-10-05T19:29:33.535 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.530+0000 7f02134df700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55aab2b3b340 0x55aab2b40db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-05T19:29:33.535 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.530+0000 7f02134df700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55aab2b3b340 0x55aab2b40db0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.118:3300/0 says I am v2:172.21.15.167:46210/0 (socket says 172.21.15.167:46210) 2023-10-05T19:29:33.535 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.530+0000 7f02134df700 1 -- 172.21.15.167:0/282595458 learned_addr learned my addr 172.21.15.167:0/282595458 (peer_addr_for_me v2:172.21.15.167:0/0) 2023-10-05T19:29:33.535 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.530+0000 7f02134df700 1 -- 172.21.15.167:0/282595458 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55aab2aeec30 con 0x55aab2b3b340 2023-10-05T19:29:33.536 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.530+0000 7f02134df700 1 --2- 172.21.15.167:0/282595458 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55aab2b3b340 0x55aab2b40db0 secure :-1 s=READY pgs=589 cs=0 l=1 rev1=1 crypto rx=0x7f01fc01f940 tx=0x7f01fc038000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-05T19:29:33.536 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.530+0000 7f015dffb700 1 -- 172.21.15.167:0/282595458 <== mon.0 v2:172.21.15.118:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f01fc025040 con 0x55aab2b3b340 2023-10-05T19:29:33.536 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.530+0000 7f015dffb700 1 -- 172.21.15.167:0/282595458 <== mon.0 v2:172.21.15.118:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f01fc039920 con 0x55aab2b3b340 2023-10-05T19:29:33.537 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.530+0000 7f02161abe40 1 -- 172.21.15.167:0/282595458 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x55aab2b413b0 con 0x55aab2b3b340 2023-10-05T19:29:33.537 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.530+0000 7f015dffb700 1 -- 172.21.15.167:0/282595458 <== mon.0 v2:172.21.15.118:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f01fc039af0 con 0x55aab2b3b340 2023-10-05T19:29:33.537 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.530+0000 7f02161abe40 1 -- 172.21.15.167:0/282595458 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55aab2b41850 con 0x55aab2b3b340 2023-10-05T19:29:33.538 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.534+0000 7f015dffb700 1 -- 172.21.15.167:0/282595458 <== mon.0 v2:172.21.15.118:3300/0 4 ==== mgrmap(e 5) v1 ==== 40329+0+0 (secure 0 0 0) 0x7f01fc039c50 con 0x55aab2b3b340 2023-10-05T19:29:33.538 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.534+0000 7f015dffb700 1 --2- 172.21.15.167:0/282595458 >> [v2:172.21.15.167:6824/29889,v1:172.21.15.167:6825/29889] conn(0x7f0150030c50 0x7f0150033100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-05T19:29:33.538 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.534+0000 7f015dffb700 1 -- 172.21.15.167:0/282595458 <== mon.0 v2:172.21.15.118:3300/0 5 ==== osd_map(32..32 src has 1..32) v4 ==== 6260+0+0 (secure 0 0 0) 0x7f01fc04dd50 con 0x55aab2b3b340 2023-10-05T19:29:33.539 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.534+0000 7f02144e1700 1 --2- 172.21.15.167:0/282595458 >> [v2:172.21.15.167:6824/29889,v1:172.21.15.167:6825/29889] conn(0x7f0150030c50 0x7f0150033100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-05T19:29:33.539 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.534+0000 7f02161abe40 5 note: GC not initialized 2023-10-05T19:29:33.539 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.534+0000 7f02144e1700 1 --2- 172.21.15.167:0/282595458 >> [v2:172.21.15.167:6824/29889,v1:172.21.15.167:6825/29889] conn(0x7f0150030c50 0x7f0150033100 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x55aab2b2e060 tx=0x7f020c00aec0 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-05T19:29:33.541 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.538+0000 7f0156ffd700 20 reqs_thread_entry: start 2023-10-05T19:29:33.541 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.538+0000 7f02161abe40 20 init_complete bucket index max shards: 11 2023-10-05T19:29:33.541 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.538+0000 7f0154ff9700 20 reqs_thread_entry: start 2023-10-05T19:29:33.542 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.538+0000 7f02161abe40 10 rgw notify: Started notification manager with: 1 workers 2023-10-05T19:29:33.542 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.538+0000 7f0142ffd700 20 rgw notify: INFO: next queues processing will happen at: Thu Oct 5 19:30:04 2023 2023-10-05T19:29:33.542 INFO:teuthology.orchestra.run.smithi167.stderr: 2023-10-05T19:29:33.542 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.538+0000 7f01427fc700 20 reqs_thread_entry: start 2023-10-05T19:29:33.542 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.538+0000 7f02161abe40 20 get_system_obj_state: rctx=0x7fff9928b818 obj=default.rgw.meta:users.uid:foo state=0x55aab29c50b0 s->prefetch_data=0 2023-10-05T19:29:33.542 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.538+0000 7f02161abe40 1 -- 172.21.15.167:0/2873792612 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:102 8.18 8:1e6ca5ed:users.uid::foo:head [call version.read in=11b,getxattrs,stat] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32) v8 -- 0x55aab2ba95f0 con 0x55aab2af2600 2023-10-05T19:29:33.543 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.538+0000 7f02144e1700 1 -- 172.21.15.167:0/2873792612 <== osd.1 v2:172.21.15.167:6800/30035 35 ==== osd_op_reply(102 foo [call out=48b,getxattrs out=75b,stat out=16b] v0'0 uv15 ondisk = 0) v8 ==== 231+0+139 (crc 0 0 0) 0x7f020c052040 con 0x55aab2af2600 2023-10-05T19:29:33.543 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.538+0000 7f02161abe40 20 get_system_obj_state: s->obj_tag was set empty 2023-10-05T19:29:33.543 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.538+0000 7f02161abe40 20 Read xattr: user.rgw.idtag 2023-10-05T19:29:33.543 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.538+0000 7f02161abe40 20 rados->read ofs=0 len=0 2023-10-05T19:29:33.543 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.538+0000 7f02161abe40 1 -- 172.21.15.167:0/2873792612 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:103 8.18 8:1e6ca5ed:users.uid::foo:head [call version.check_conds in=122b,call version.read in=11b,read 0~0,getxattrs] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32) v8 -- 0x55aab2ba95b0 con 0x55aab2af2600 2023-10-05T19:29:33.544 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.538+0000 7f02144e1700 1 -- 172.21.15.167:0/2873792612 <== osd.1 v2:172.21.15.167:6800/30035 36 ==== osd_op_reply(103 foo [call,call out=48b,read 0~357 out=357b,getxattrs out=75b] v0'0 uv15 ondisk = 0) v8 ==== 273+0+480 (crc 0 0 0) 0x7f020c052040 con 0x55aab2af2600 2023-10-05T19:29:33.544 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.538+0000 7f02161abe40 20 rados_obj.operate() r=0 bl.length=357 2023-10-05T19:29:33.544 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.538+0000 7f02161abe40 1 -- 172.21.15.167:0/2873792612 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:104 6.13 6:cea599e4:usage::usage.10:head [call rgw.user_usage_log_read in=63b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32) v8 -- 0x55aab2ba95b0 con 0x7f01fc007030 2023-10-05T19:29:33.545 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.542+0000 7f02144e1700 1 -- 172.21.15.167:0/2873792612 <== osd.0 v2:172.21.15.118:6800/27690 29 ==== osd_op_reply(104 usage.10 [call out=1268b] v0'0 uv30 ondisk = 0) v8 ==== 152+0+1268 (crc 0 0 0) 0x7f020c052040 con 0x7f01fc007030 2023-10-05T19:29:33.545 INFO:teuthology.orchestra.run.smithi167.stdout:{ 2023-10-05T19:29:33.545 INFO:teuthology.orchestra.run.smithi167.stdout: "entries": [ 2023-10-05T19:29:33.545 INFO:teuthology.orchestra.run.smithi167.stdout: { 2023-10-05T19:29:33.545 INFO:teuthology.orchestra.run.smithi167.stdout: "user": "foo", 2023-10-05T19:29:33.545 INFO:teuthology.orchestra.run.smithi167.stdout: "buckets": [ 2023-10-05T19:29:33.545 INFO:teuthology.orchestra.run.smithi167.stdout: { 2023-10-05T19:29:33.545 INFO:teuthology.orchestra.run.smithi167.stdout: "bucket": "", 2023-10-05T19:29:33.546 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2023-10-05T19:00:00.000000Z", 2023-10-05T19:29:33.546 INFO:teuthology.orchestra.run.smithi167.stdout: "epoch": 1696532400, 2023-10-05T19:29:33.546 INFO:teuthology.orchestra.run.smithi167.stdout: "owner": "foo", 2023-10-05T19:29:33.546 INFO:teuthology.orchestra.run.smithi167.stdout: "categories": [ 2023-10-05T19:29:33.546 INFO:teuthology.orchestra.run.smithi167.stdout: { 2023-10-05T19:29:33.546 INFO:teuthology.orchestra.run.smithi167.stdout: "category": "list_buckets", 2023-10-05T19:29:33.546 INFO:teuthology.orchestra.run.smithi167.stdout: "bytes_sent": 322, 2023-10-05T19:29:33.546 INFO:teuthology.orchestra.run.smithi167.stdout: "bytes_received": 0, 2023-10-05T19:29:33.546 INFO:teuthology.orchestra.run.smithi167.stdout: "ops": 1, 2023-10-05T19:29:33.546 INFO:teuthology.orchestra.run.smithi167.stdout: "successful_ops": 1 2023-10-05T19:29:33.546 INFO:teuthology.orchestra.run.smithi167.stdout: } 2023-10-05T19:29:33.546 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2023-10-05T19:29:33.547 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2023-10-05T19:29:33.547 INFO:teuthology.orchestra.run.smithi167.stdout: { 2023-10-05T19:29:33.547 INFO:teuthology.orchestra.run.smithi167.stdout: "bucket": "myfoo", 2023-10-05T19:29:33.547 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2023-10-05T19:00:00.000000Z", 2023-10-05T19:29:33.547 INFO:teuthology.orchestra.run.smithi167.stdout: "epoch": 1696532400, 2023-10-05T19:29:33.547 INFO:teuthology.orchestra.run.smithi167.stdout: "owner": "foo", 2023-10-05T19:29:33.547 INFO:teuthology.orchestra.run.smithi167.stdout: "categories": [ 2023-10-05T19:29:33.547 INFO:teuthology.orchestra.run.smithi167.stdout: { 2023-10-05T19:29:33.547 INFO:teuthology.orchestra.run.smithi167.stdout: "category": "create_bucket", 2023-10-05T19:29:33.547 INFO:teuthology.orchestra.run.smithi167.stdout: "bytes_sent": 0, 2023-10-05T19:29:33.547 INFO:teuthology.orchestra.run.smithi167.stdout: "bytes_received": 0, 2023-10-05T19:29:33.547 INFO:teuthology.orchestra.run.smithi167.stdout: "ops": 5, 2023-10-05T19:29:33.547 INFO:teuthology.orchestra.run.smithi167.stdout: "successful_ops": 5 2023-10-05T19:29:33.548 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2023-10-05T19:29:33.548 INFO:teuthology.orchestra.run.smithi167.stdout: { 2023-10-05T19:29:33.548 INFO:teuthology.orchestra.run.smithi167.stdout: "category": "delete_bucket", 2023-10-05T19:29:33.548 INFO:teuthology.orchestra.run.smithi167.stdout: "bytes_sent": 217, 2023-10-05T19:29:33.548 INFO:teuthology.orchestra.run.smithi167.stdout: "bytes_received": 0, 2023-10-05T19:29:33.548 INFO:teuthology.orchestra.run.smithi167.stdout: "ops": 2, 2023-10-05T19:29:33.548 INFO:teuthology.orchestra.run.smithi167.stdout: "successful_ops": 1 2023-10-05T19:29:33.548 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2023-10-05T19:29:33.548 INFO:teuthology.orchestra.run.smithi167.stdout: { 2023-10-05T19:29:33.548 INFO:teuthology.orchestra.run.smithi167.stdout: "category": "delete_obj", 2023-10-05T19:29:33.548 INFO:teuthology.orchestra.run.smithi167.stdout: "bytes_sent": 0, 2023-10-05T19:29:33.548 INFO:teuthology.orchestra.run.smithi167.stdout: "bytes_received": 0, 2023-10-05T19:29:33.549 INFO:teuthology.orchestra.run.smithi167.stdout: "ops": 4, 2023-10-05T19:29:33.549 INFO:teuthology.orchestra.run.smithi167.stdout: "successful_ops": 4 2023-10-05T19:29:33.549 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2023-10-05T19:29:33.549 INFO:teuthology.orchestra.run.smithi167.stdout: { 2023-10-05T19:29:33.549 INFO:teuthology.orchestra.run.smithi167.stdout: "category": "get_acls", 2023-10-05T19:29:33.549 INFO:teuthology.orchestra.run.smithi167.stdout: "bytes_sent": 1026, 2023-10-05T19:29:33.549 INFO:teuthology.orchestra.run.smithi167.stdout: "bytes_received": 0, 2023-10-05T19:29:33.549 INFO:teuthology.orchestra.run.smithi167.stdout: "ops": 2, 2023-10-05T19:29:33.549 INFO:teuthology.orchestra.run.smithi167.stdout: "successful_ops": 2 2023-10-05T19:29:33.549 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2023-10-05T19:29:33.549 INFO:teuthology.orchestra.run.smithi167.stdout: { 2023-10-05T19:29:33.549 INFO:teuthology.orchestra.run.smithi167.stdout: "category": "get_obj", 2023-10-05T19:29:33.549 INFO:teuthology.orchestra.run.smithi167.stdout: "bytes_sent": 4, 2023-10-05T19:29:33.550 INFO:teuthology.orchestra.run.smithi167.stdout: "bytes_received": 0, 2023-10-05T19:29:33.550 INFO:teuthology.orchestra.run.smithi167.stdout: "ops": 1, 2023-10-05T19:29:33.550 INFO:teuthology.orchestra.run.smithi167.stdout: "successful_ops": 1 2023-10-05T19:29:33.550 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2023-10-05T19:29:33.550 INFO:teuthology.orchestra.run.smithi167.stdout: { 2023-10-05T19:29:33.550 INFO:teuthology.orchestra.run.smithi167.stdout: "category": "list_bucket", 2023-10-05T19:29:33.550 INFO:teuthology.orchestra.run.smithi167.stdout: "bytes_sent": 515, 2023-10-05T19:29:33.550 INFO:teuthology.orchestra.run.smithi167.stdout: "bytes_received": 0, 2023-10-05T19:29:33.550 INFO:teuthology.orchestra.run.smithi167.stdout: "ops": 1, 2023-10-05T19:29:33.550 INFO:teuthology.orchestra.run.smithi167.stdout: "successful_ops": 1 2023-10-05T19:29:33.550 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2023-10-05T19:29:33.550 INFO:teuthology.orchestra.run.smithi167.stdout: { 2023-10-05T19:29:33.551 INFO:teuthology.orchestra.run.smithi167.stdout: "category": "put_acls", 2023-10-05T19:29:33.551 INFO:teuthology.orchestra.run.smithi167.stdout: "bytes_sent": 0, 2023-10-05T19:29:33.551 INFO:teuthology.orchestra.run.smithi167.stdout: "bytes_received": 0, 2023-10-05T19:29:33.551 INFO:teuthology.orchestra.run.smithi167.stdout: "ops": 2, 2023-10-05T19:29:33.551 INFO:teuthology.orchestra.run.smithi167.stdout: "successful_ops": 2 2023-10-05T19:29:33.551 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2023-10-05T19:29:33.551 INFO:teuthology.orchestra.run.smithi167.stdout: { 2023-10-05T19:29:33.551 INFO:teuthology.orchestra.run.smithi167.stdout: "category": "put_obj", 2023-10-05T19:29:33.551 INFO:teuthology.orchestra.run.smithi167.stdout: "bytes_sent": 0, 2023-10-05T19:29:33.551 INFO:teuthology.orchestra.run.smithi167.stdout: "bytes_received": 30000042, 2023-10-05T19:29:33.551 INFO:teuthology.orchestra.run.smithi167.stdout: "ops": 11, 2023-10-05T19:29:33.551 INFO:teuthology.orchestra.run.smithi167.stdout: "successful_ops": 11 2023-10-05T19:29:33.552 INFO:teuthology.orchestra.run.smithi167.stdout: } 2023-10-05T19:29:33.552 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2023-10-05T19:29:33.552 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2023-10-05T19:29:33.552 INFO:teuthology.orchestra.run.smithi167.stdout: { 2023-10-05T19:29:33.552 INFO:teuthology.orchestra.run.smithi167.stdout: "bucket": "myfoo2", 2023-10-05T19:29:33.552 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2023-10-05T19:00:00.000000Z", 2023-10-05T19:29:33.552 INFO:teuthology.orchestra.run.smithi167.stdout: "epoch": 1696532400, 2023-10-05T19:29:33.552 INFO:teuthology.orchestra.run.smithi167.stdout: "owner": "foo", 2023-10-05T19:29:33.552 INFO:teuthology.orchestra.run.smithi167.stdout: "categories": [ 2023-10-05T19:29:33.552 INFO:teuthology.orchestra.run.smithi167.stdout: { 2023-10-05T19:29:33.552 INFO:teuthology.orchestra.run.smithi167.stdout: "category": "create_bucket", 2023-10-05T19:29:33.552 INFO:teuthology.orchestra.run.smithi167.stdout: "bytes_sent": 0, 2023-10-05T19:29:33.552 INFO:teuthology.orchestra.run.smithi167.stdout: "bytes_received": 0, 2023-10-05T19:29:33.553 INFO:teuthology.orchestra.run.smithi167.stdout: "ops": 1, 2023-10-05T19:29:33.553 INFO:teuthology.orchestra.run.smithi167.stdout: "successful_ops": 1 2023-10-05T19:29:33.553 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2023-10-05T19:29:33.553 INFO:teuthology.orchestra.run.smithi167.stdout: { 2023-10-05T19:29:33.553 INFO:teuthology.orchestra.run.smithi167.stdout: "category": "delete_bucket", 2023-10-05T19:29:33.553 INFO:teuthology.orchestra.run.smithi167.stdout: "bytes_sent": 0, 2023-10-05T19:29:33.553 INFO:teuthology.orchestra.run.smithi167.stdout: "bytes_received": 0, 2023-10-05T19:29:33.553 INFO:teuthology.orchestra.run.smithi167.stdout: "ops": 1, 2023-10-05T19:29:33.553 INFO:teuthology.orchestra.run.smithi167.stdout: "successful_ops": 1 2023-10-05T19:29:33.553 INFO:teuthology.orchestra.run.smithi167.stdout: } 2023-10-05T19:29:33.553 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2023-10-05T19:29:33.553 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2023-10-05T19:29:33.554 INFO:teuthology.orchestra.run.smithi167.stdout: { 2023-10-05T19:29:33.555 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.542+0000 7f02161abe40 1 -- 172.21.15.167:0/2873792612 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:105 7.1f 7:f95f44c2:::notify.0:head [watch unwatch cookie 94191630843200] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e32) v8 -- 0x55aab2b03880 con 0x7f020402d020 2023-10-05T19:29:33.555 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.542+0000 7f02134df700 1 -- 172.21.15.167:0/2873792612 <== osd.2 v2:172.21.15.167:6816/30037 17 ==== osd_op_reply(105 notify.0 [watch unwatch cookie 94191630843200] v32'227 uv225 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f01fc03f020 con 0x7f020402d020 2023-10-05T19:29:33.555 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.546+0000 7f02161abe40 20 remove_watcher() i=0 2023-10-05T19:29:33.555 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.546+0000 7f02161abe40 2 removed watcher, disabling cache 2023-10-05T19:29:33.555 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.546+0000 7f02161abe40 1 -- 172.21.15.167:0/2873792612 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:106 7.0 7:05bf5b68:::notify.1:head [watch unwatch cookie 94191630846720] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e32) v8 -- 0x55aab2b27d40 con 0x55aab2af2600 2023-10-05T19:29:33.555 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.546+0000 7f02144e1700 1 -- 172.21.15.167:0/2873792612 <== osd.1 v2:172.21.15.167:6800/30035 37 ==== osd_op_reply(106 notify.1 [watch unwatch cookie 94191630846720] v32'227 uv225 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f020c052040 con 0x55aab2af2600 2023-10-05T19:29:33.555 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.546+0000 7f02161abe40 20 remove_watcher() i=1 2023-10-05T19:29:33.555 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.546+0000 7f02161abe40 1 -- 172.21.15.167:0/2873792612 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:107 7.15 7:a93a5511:::notify.2:head [watch unwatch cookie 94191630850432] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e32) v8 -- 0x55aab2b28b00 con 0x55aab2b231c0 2023-10-05T19:29:33.556 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.550+0000 7f0213ce0700 1 -- 172.21.15.167:0/2873792612 <== osd.3 v2:172.21.15.167:6808/30036 24 ==== osd_op_reply(107 notify.2 [watch unwatch cookie 94191630850432] v32'227 uv225 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f020404d040 con 0x55aab2b231c0 2023-10-05T19:29:33.556 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.550+0000 7f02161abe40 20 remove_watcher() i=2 2023-10-05T19:29:33.556 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.550+0000 7f02161abe40 1 -- 172.21.15.167:0/2873792612 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:108 7.e 7:7759931f:::notify.3:head [watch unwatch cookie 94191630847824] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e32) v8 -- 0x55aab2b29980 con 0x7f020402d020 2023-10-05T19:29:33.557 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.554+0000 7f02134df700 1 -- 172.21.15.167:0/2873792612 <== osd.2 v2:172.21.15.167:6816/30037 18 ==== osd_op_reply(108 notify.3 [watch unwatch cookie 94191630847824] v32'227 uv225 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f01fc03f020 con 0x7f020402d020 2023-10-05T19:29:33.557 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.554+0000 7f02161abe40 20 remove_watcher() i=3 2023-10-05T19:29:33.558 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.554+0000 7f02161abe40 1 -- 172.21.15.167:0/2873792612 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:109 7.d 7:b4812045:::notify.4:head [watch unwatch cookie 94191630857568] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e32) v8 -- 0x55aab2b29980 con 0x55aab2af2600 2023-10-05T19:29:33.560 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.554+0000 7f02144e1700 1 -- 172.21.15.167:0/2873792612 <== osd.1 v2:172.21.15.167:6800/30035 38 ==== osd_op_reply(109 notify.4 [watch unwatch cookie 94191630857568] v32'227 uv225 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f020c052040 con 0x55aab2af2600 2023-10-05T19:29:33.560 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.554+0000 7f02161abe40 20 remove_watcher() i=4 2023-10-05T19:29:33.560 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.554+0000 7f02161abe40 1 -- 172.21.15.167:0/2873792612 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:110 7.3 7:c609908c:::notify.5:head [watch unwatch cookie 94191630852784] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e32) v8 -- 0x55aab2b29980 con 0x7f01fc007030 2023-10-05T19:29:33.563 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.558+0000 7f02144e1700 1 -- 172.21.15.167:0/2873792612 <== osd.0 v2:172.21.15.118:6800/27690 30 ==== osd_op_reply(110 notify.5 [watch unwatch cookie 94191630852784] v32'227 uv225 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f020c052040 con 0x7f01fc007030 2023-10-05T19:29:33.563 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.558+0000 7f02161abe40 20 remove_watcher() i=5 2023-10-05T19:29:33.563 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.558+0000 7f02161abe40 1 -- 172.21.15.167:0/2873792612 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:111 7.14 7:2b04a3e9:::notify.6:head [watch unwatch cookie 94191630862256] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e32) v8 -- 0x55aab2b2a2b0 con 0x7f01fc007030 2023-10-05T19:29:33.566 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.562+0000 7f02144e1700 1 -- 172.21.15.167:0/2873792612 <== osd.0 v2:172.21.15.118:6800/27690 31 ==== osd_op_reply(111 notify.6 [watch unwatch cookie 94191630862256] v32'227 uv225 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f020c052040 con 0x7f01fc007030 2023-10-05T19:29:33.566 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.562+0000 7f02161abe40 20 remove_watcher() i=6 2023-10-05T19:29:33.566 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.562+0000 7f02161abe40 1 -- 172.21.15.167:0/2873792612 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:112 7.9 7:93e5b521:::notify.7:head [watch unwatch cookie 94191630866304] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e32) v8 -- 0x55aab2b29980 con 0x7f01fc007030 2023-10-05T19:29:33.569 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.566+0000 7f02144e1700 1 -- 172.21.15.167:0/2873792612 <== osd.0 v2:172.21.15.118:6800/27690 32 ==== osd_op_reply(112 notify.7 [watch unwatch cookie 94191630866304] v32'227 uv225 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f020c052040 con 0x7f01fc007030 2023-10-05T19:29:33.569 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.566+0000 7f02161abe40 20 remove_watcher() i=7 2023-10-05T19:29:33.570 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.566+0000 7f015ffff700 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2023-10-05T19:29:33.570 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.566+0000 7f02161abe40 1 -- 172.21.15.167:0/2873792612 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:113 6.4 6:22d26bf9:::data_loggenerations_metadata:head [watch unwatch cookie 94191630867408] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e32) v8 -- 0x55aab2b2c7b0 con 0x55aab2af2600 2023-10-05T19:29:33.574 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.570+0000 7f02144e1700 1 -- 172.21.15.167:0/2873792612 <== osd.1 v2:172.21.15.167:6800/30035 39 ==== osd_op_reply(113 data_loggenerations_metadata [watch unwatch cookie 94191630867408] v32'224 uv1 ondisk = 0) v8 ==== 172+0+0 (crc 0 0 0) 0x7f020c052040 con 0x55aab2af2600 2023-10-05T19:29:33.574 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.570+0000 7f02161abe40 1 -- 172.21.15.167:0/2873792612 >> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] conn(0x7f01fc007030 msgr2=0x7f01fc01d430 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-05T19:29:33.574 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.570+0000 7f02161abe40 1 --2- 172.21.15.167:0/2873792612 >> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] conn(0x7f01fc007030 0x7f01fc01d430 crc :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-05T19:29:33.574 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.570+0000 7f02161abe40 1 -- 172.21.15.167:0/2873792612 >> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] conn(0x55aab2af2600 msgr2=0x55aab2b1f6e0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-05T19:29:33.574 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.570+0000 7f02161abe40 1 --2- 172.21.15.167:0/2873792612 >> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] conn(0x55aab2af2600 0x55aab2b1f6e0 crc :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-05T19:29:33.574 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.570+0000 7f02161abe40 1 -- 172.21.15.167:0/2873792612 >> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] conn(0x7f020402d020 msgr2=0x7f020402c5d0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-05T19:29:33.574 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.570+0000 7f02161abe40 1 --2- 172.21.15.167:0/2873792612 >> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] conn(0x7f020402d020 0x7f020402c5d0 crc :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-05T19:29:33.575 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.570+0000 7f02161abe40 1 -- 172.21.15.167:0/2873792612 >> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] conn(0x55aab2b231c0 msgr2=0x55aab2b255a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-05T19:29:33.575 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.570+0000 7f02161abe40 1 --2- 172.21.15.167:0/2873792612 >> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] conn(0x55aab2b231c0 0x55aab2b255a0 crc :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-05T19:29:33.575 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.570+0000 7f02161abe40 1 -- 172.21.15.167:0/2873792612 >> [v2:172.21.15.167:6824/29889,v1:172.21.15.167:6825/29889] conn(0x7f01f8030bf0 msgr2=0x7f01f80330a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-05T19:29:33.575 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.570+0000 7f02161abe40 1 --2- 172.21.15.167:0/2873792612 >> [v2:172.21.15.167:6824/29889,v1:172.21.15.167:6825/29889] conn(0x7f01f8030bf0 0x7f01f80330a0 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7f01fc0030c0 tx=0x7f01fc014000 comp rx=0 tx=0).stop 2023-10-05T19:29:33.575 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.570+0000 7f02144e1700 1 -- 172.21.15.167:0/2873792612 reap_dead start 2023-10-05T19:29:33.575 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.570+0000 7f02161abe40 1 -- 172.21.15.167:0/2873792612 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55aab2aee020 msgr2=0x55aab2b13cf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-05T19:29:33.575 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.570+0000 7f02161abe40 1 --2- 172.21.15.167:0/2873792612 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55aab2aee020 0x55aab2b13cf0 secure :-1 s=READY pgs=587 cs=0 l=1 rev1=1 crypto rx=0x7f020400b960 tx=0x7f0204014000 comp rx=0 tx=0).stop 2023-10-05T19:29:33.575 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.570+0000 7f02161abe40 1 -- 172.21.15.167:0/2873792612 shutdown_connections 2023-10-05T19:29:33.575 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.570+0000 7f02161abe40 1 --2- 172.21.15.167:0/2873792612 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55aab2aee020 0x55aab2b13cf0 unknown :-1 s=CLOSED pgs=587 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-05T19:29:33.575 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.570+0000 7f02161abe40 1 -- 172.21.15.167:0/2873792612 >> 172.21.15.167:0/2873792612 conn(0x55aab2a61870 msgr2=0x55aab2b075a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-05T19:29:33.576 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.570+0000 7f02161abe40 1 -- 172.21.15.167:0/2873792612 shutdown_connections 2023-10-05T19:29:33.576 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.570+0000 7f02161abe40 1 -- 172.21.15.167:0/2873792612 wait complete. 2023-10-05T19:29:33.576 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.570+0000 7f02161abe40 1 -- 172.21.15.167:0/282595458 >> [v2:172.21.15.167:6824/29889,v1:172.21.15.167:6825/29889] conn(0x7f0150030c50 msgr2=0x7f0150033100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-05T19:29:33.577 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.570+0000 7f02161abe40 1 --2- 172.21.15.167:0/282595458 >> [v2:172.21.15.167:6824/29889,v1:172.21.15.167:6825/29889] conn(0x7f0150030c50 0x7f0150033100 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x55aab2b2e060 tx=0x7f020c00aec0 comp rx=0 tx=0).stop 2023-10-05T19:29:33.577 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.570+0000 7f02161abe40 1 -- 172.21.15.167:0/282595458 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55aab2b3b340 msgr2=0x55aab2b40db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-05T19:29:33.577 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.570+0000 7f02161abe40 1 --2- 172.21.15.167:0/282595458 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55aab2b3b340 0x55aab2b40db0 secure :-1 s=READY pgs=589 cs=0 l=1 rev1=1 crypto rx=0x7f01fc01f940 tx=0x7f01fc038000 comp rx=0 tx=0).stop 2023-10-05T19:29:33.577 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.570+0000 7f02161abe40 1 -- 172.21.15.167:0/282595458 shutdown_connections 2023-10-05T19:29:33.577 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.570+0000 7f02161abe40 1 --2- 172.21.15.167:0/282595458 >> [v2:172.21.15.167:6824/29889,v1:172.21.15.167:6825/29889] conn(0x7f0150030c50 0x7f0150033100 unknown :-1 s=CLOSED pgs=203 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-05T19:29:33.577 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.570+0000 7f02161abe40 1 --2- 172.21.15.167:0/282595458 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55aab2b3b340 0x55aab2b40db0 unknown :-1 s=CLOSED pgs=589 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-05T19:29:33.577 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.570+0000 7f02161abe40 1 -- 172.21.15.167:0/282595458 >> 172.21.15.167:0/282595458 conn(0x55aab2b36700 msgr2=0x55aab2b34800 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-05T19:29:33.577 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.570+0000 7f02161abe40 1 -- 172.21.15.167:0/282595458 shutdown_connections 2023-10-05T19:29:33.577 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.570+0000 7f02161abe40 1 -- 172.21.15.167:0/282595458 wait complete. 2023-10-05T19:29:33.578 INFO:teuthology.orchestra.run.smithi167.stdout: "bucket": "myfoo3", 2023-10-05T19:29:33.579 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2023-10-05T19:00:00.000000Z", 2023-10-05T19:29:33.579 INFO:teuthology.orchestra.run.smithi167.stdout: "epoch": 1696532400, 2023-10-05T19:29:33.579 INFO:teuthology.orchestra.run.smithi167.stdout: "owner": "foo", 2023-10-05T19:29:33.579 INFO:teuthology.orchestra.run.smithi167.stdout: "categories": [ 2023-10-05T19:29:33.579 INFO:teuthology.orchestra.run.smithi167.stdout: { 2023-10-05T19:29:33.579 INFO:teuthology.orchestra.run.smithi167.stdout: "category": "create_bucket", 2023-10-05T19:29:33.579 INFO:teuthology.orchestra.run.smithi167.stdout: "bytes_sent": 0, 2023-10-05T19:29:33.579 INFO:teuthology.orchestra.run.smithi167.stdout: "bytes_received": 0, 2023-10-05T19:29:33.579 INFO:teuthology.orchestra.run.smithi167.stdout: "ops": 1, 2023-10-05T19:29:33.579 INFO:teuthology.orchestra.run.smithi167.stdout: "successful_ops": 1 2023-10-05T19:29:33.579 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2023-10-05T19:29:33.579 INFO:teuthology.orchestra.run.smithi167.stdout: { 2023-10-05T19:29:33.580 INFO:teuthology.orchestra.run.smithi167.stdout: "category": "delete_bucket", 2023-10-05T19:29:33.580 INFO:teuthology.orchestra.run.smithi167.stdout: "bytes_sent": 0, 2023-10-05T19:29:33.580 INFO:teuthology.orchestra.run.smithi167.stdout: "bytes_received": 0, 2023-10-05T19:29:33.580 INFO:teuthology.orchestra.run.smithi167.stdout: "ops": 1, 2023-10-05T19:29:33.580 INFO:teuthology.orchestra.run.smithi167.stdout: "successful_ops": 1 2023-10-05T19:29:33.580 INFO:teuthology.orchestra.run.smithi167.stdout: } 2023-10-05T19:29:33.580 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2023-10-05T19:29:33.580 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2023-10-05T19:29:33.580 INFO:teuthology.orchestra.run.smithi167.stdout: { 2023-10-05T19:29:33.580 INFO:teuthology.orchestra.run.smithi167.stdout: "bucket": "myfoo4", 2023-10-05T19:29:33.580 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2023-10-05T19:00:00.000000Z", 2023-10-05T19:29:33.580 INFO:teuthology.orchestra.run.smithi167.stdout: "epoch": 1696532400, 2023-10-05T19:29:33.581 INFO:teuthology.orchestra.run.smithi167.stdout: "owner": "foo", 2023-10-05T19:29:33.581 INFO:teuthology.orchestra.run.smithi167.stdout: "categories": [ 2023-10-05T19:29:33.581 INFO:teuthology.orchestra.run.smithi167.stdout: { 2023-10-05T19:29:33.581 INFO:teuthology.orchestra.run.smithi167.stdout: "category": "create_bucket", 2023-10-05T19:29:33.581 INFO:teuthology.orchestra.run.smithi167.stdout: "bytes_sent": 0, 2023-10-05T19:29:33.581 INFO:teuthology.orchestra.run.smithi167.stdout: "bytes_received": 0, 2023-10-05T19:29:33.581 INFO:teuthology.orchestra.run.smithi167.stdout: "ops": 1, 2023-10-05T19:29:33.581 INFO:teuthology.orchestra.run.smithi167.stdout: "successful_ops": 1 2023-10-05T19:29:33.581 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2023-10-05T19:29:33.581 INFO:teuthology.orchestra.run.smithi167.stdout: { 2023-10-05T19:29:33.581 INFO:teuthology.orchestra.run.smithi167.stdout: "category": "delete_bucket", 2023-10-05T19:29:33.581 INFO:teuthology.orchestra.run.smithi167.stdout: "bytes_sent": 0, 2023-10-05T19:29:33.582 INFO:teuthology.orchestra.run.smithi167.stdout: "bytes_received": 0, 2023-10-05T19:29:33.582 INFO:teuthology.orchestra.run.smithi167.stdout: "ops": 1, 2023-10-05T19:29:33.582 INFO:teuthology.orchestra.run.smithi167.stdout: "successful_ops": 1 2023-10-05T19:29:33.582 INFO:teuthology.orchestra.run.smithi167.stdout: } 2023-10-05T19:29:33.582 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2023-10-05T19:29:33.582 INFO:teuthology.orchestra.run.smithi167.stdout: } 2023-10-05T19:29:33.582 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2023-10-05T19:29:33.582 INFO:teuthology.orchestra.run.smithi167.stdout: } 2023-10-05T19:29:33.582 INFO:teuthology.orchestra.run.smithi167.stdout: ], 2023-10-05T19:29:33.582 INFO:teuthology.orchestra.run.smithi167.stdout: "summary": [ 2023-10-05T19:29:33.582 INFO:teuthology.orchestra.run.smithi167.stdout: { 2023-10-05T19:29:33.582 INFO:teuthology.orchestra.run.smithi167.stdout: "user": "foo", 2023-10-05T19:29:33.582 INFO:teuthology.orchestra.run.smithi167.stdout: "categories": [ 2023-10-05T19:29:33.583 INFO:teuthology.orchestra.run.smithi167.stdout: { 2023-10-05T19:29:33.583 INFO:teuthology.orchestra.run.smithi167.stdout: "category": "create_bucket", 2023-10-05T19:29:33.583 INFO:teuthology.orchestra.run.smithi167.stdout: "bytes_sent": 0, 2023-10-05T19:29:33.583 INFO:teuthology.orchestra.run.smithi167.stdout: "bytes_received": 0, 2023-10-05T19:29:33.583 INFO:teuthology.orchestra.run.smithi167.stdout: "ops": 8, 2023-10-05T19:29:33.583 INFO:teuthology.orchestra.run.smithi167.stdout: "successful_ops": 8 2023-10-05T19:29:33.583 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2023-10-05T19:29:33.583 INFO:teuthology.orchestra.run.smithi167.stdout: { 2023-10-05T19:29:33.583 INFO:teuthology.orchestra.run.smithi167.stdout: "category": "delete_bucket", 2023-10-05T19:29:33.583 INFO:teuthology.orchestra.run.smithi167.stdout: "bytes_sent": 217, 2023-10-05T19:29:33.583 INFO:teuthology.orchestra.run.smithi167.stdout: "bytes_received": 0, 2023-10-05T19:29:33.583 INFO:teuthology.orchestra.run.smithi167.stdout: "ops": 5, 2023-10-05T19:29:33.584 INFO:teuthology.orchestra.run.smithi167.stdout: "successful_ops": 4 2023-10-05T19:29:33.584 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2023-10-05T19:29:33.584 INFO:teuthology.orchestra.run.smithi167.stdout: { 2023-10-05T19:29:33.584 INFO:teuthology.orchestra.run.smithi167.stdout: "category": "delete_obj", 2023-10-05T19:29:33.584 INFO:teuthology.orchestra.run.smithi167.stdout: "bytes_sent": 0, 2023-10-05T19:29:33.584 INFO:teuthology.orchestra.run.smithi167.stdout: "bytes_received": 0, 2023-10-05T19:29:33.584 INFO:teuthology.orchestra.run.smithi167.stdout: "ops": 4, 2023-10-05T19:29:33.584 INFO:teuthology.orchestra.run.smithi167.stdout: "successful_ops": 4 2023-10-05T19:29:33.584 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2023-10-05T19:29:33.584 INFO:teuthology.orchestra.run.smithi167.stdout: { 2023-10-05T19:29:33.584 INFO:teuthology.orchestra.run.smithi167.stdout: "category": "get_acls", 2023-10-05T19:29:33.584 INFO:teuthology.orchestra.run.smithi167.stdout: "bytes_sent": 1026, 2023-10-05T19:29:33.585 INFO:teuthology.orchestra.run.smithi167.stdout: "bytes_received": 0, 2023-10-05T19:29:33.585 INFO:teuthology.orchestra.run.smithi167.stdout: "ops": 2, 2023-10-05T19:29:33.585 INFO:teuthology.orchestra.run.smithi167.stdout: "successful_ops": 2 2023-10-05T19:29:33.585 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2023-10-05T19:29:33.585 INFO:teuthology.orchestra.run.smithi167.stdout: { 2023-10-05T19:29:33.585 INFO:teuthology.orchestra.run.smithi167.stdout: "category": "get_obj", 2023-10-05T19:29:33.585 INFO:teuthology.orchestra.run.smithi167.stdout: "bytes_sent": 4, 2023-10-05T19:29:33.585 INFO:teuthology.orchestra.run.smithi167.stdout: "bytes_received": 0, 2023-10-05T19:29:33.585 INFO:teuthology.orchestra.run.smithi167.stdout: "ops": 1, 2023-10-05T19:29:33.585 INFO:teuthology.orchestra.run.smithi167.stdout: "successful_ops": 1 2023-10-05T19:29:33.585 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2023-10-05T19:29:33.585 INFO:teuthology.orchestra.run.smithi167.stdout: { 2023-10-05T19:29:33.585 INFO:teuthology.orchestra.run.smithi167.stdout: "category": "list_bucket", 2023-10-05T19:29:33.586 INFO:teuthology.orchestra.run.smithi167.stdout: "bytes_sent": 515, 2023-10-05T19:29:33.586 INFO:teuthology.orchestra.run.smithi167.stdout: "bytes_received": 0, 2023-10-05T19:29:33.586 INFO:teuthology.orchestra.run.smithi167.stdout: "ops": 1, 2023-10-05T19:29:33.586 INFO:teuthology.orchestra.run.smithi167.stdout: "successful_ops": 1 2023-10-05T19:29:33.586 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2023-10-05T19:29:33.586 INFO:teuthology.orchestra.run.smithi167.stdout: { 2023-10-05T19:29:33.586 INFO:teuthology.orchestra.run.smithi167.stdout: "category": "list_buckets", 2023-10-05T19:29:33.586 INFO:teuthology.orchestra.run.smithi167.stdout: "bytes_sent": 322, 2023-10-05T19:29:33.586 INFO:teuthology.orchestra.run.smithi167.stdout: "bytes_received": 0, 2023-10-05T19:29:33.586 INFO:teuthology.orchestra.run.smithi167.stdout: "ops": 1, 2023-10-05T19:29:33.586 INFO:teuthology.orchestra.run.smithi167.stdout: "successful_ops": 1 2023-10-05T19:29:33.586 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2023-10-05T19:29:33.587 INFO:teuthology.orchestra.run.smithi167.stdout: { 2023-10-05T19:29:33.587 INFO:teuthology.orchestra.run.smithi167.stdout: "category": "put_acls", 2023-10-05T19:29:33.587 INFO:teuthology.orchestra.run.smithi167.stdout: "bytes_sent": 0, 2023-10-05T19:29:33.587 INFO:teuthology.orchestra.run.smithi167.stdout: "bytes_received": 0, 2023-10-05T19:29:33.587 INFO:teuthology.orchestra.run.smithi167.stdout: "ops": 2, 2023-10-05T19:29:33.587 INFO:teuthology.orchestra.run.smithi167.stdout: "successful_ops": 2 2023-10-05T19:29:33.587 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2023-10-05T19:29:33.587 INFO:teuthology.orchestra.run.smithi167.stdout: { 2023-10-05T19:29:33.587 INFO:teuthology.orchestra.run.smithi167.stdout: "category": "put_obj", 2023-10-05T19:29:33.587 INFO:teuthology.orchestra.run.smithi167.stdout: "bytes_sent": 0, 2023-10-05T19:29:33.587 INFO:teuthology.orchestra.run.smithi167.stdout: "bytes_received": 30000042, 2023-10-05T19:29:33.587 INFO:teuthology.orchestra.run.smithi167.stdout: "ops": 11, 2023-10-05T19:29:33.588 INFO:teuthology.orchestra.run.smithi167.stdout: "successful_ops": 11 2023-10-05T19:29:33.588 INFO:teuthology.orchestra.run.smithi167.stdout: } 2023-10-05T19:29:33.588 INFO:teuthology.orchestra.run.smithi167.stdout: ], 2023-10-05T19:29:33.588 INFO:teuthology.orchestra.run.smithi167.stdout: "total": { 2023-10-05T19:29:33.588 INFO:teuthology.orchestra.run.smithi167.stdout: "bytes_sent": 2084, 2023-10-05T19:29:33.588 INFO:teuthology.orchestra.run.smithi167.stdout: "bytes_received": 30000042, 2023-10-05T19:29:33.588 INFO:teuthology.orchestra.run.smithi167.stdout: "ops": 35, 2023-10-05T19:29:33.588 INFO:teuthology.orchestra.run.smithi167.stdout: "successful_ops": 34 2023-10-05T19:29:33.588 INFO:teuthology.orchestra.run.smithi167.stdout: } 2023-10-05T19:29:33.588 INFO:teuthology.orchestra.run.smithi167.stdout: } 2023-10-05T19:29:33.588 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2023-10-05T19:29:33.588 INFO:teuthology.orchestra.run.smithi167.stdout:} 2023-10-05T19:29:33.589 DEBUG:tasks.util.rgw: json result: {'entries': [{'user': 'foo', 'buckets': [{'bucket': '', 'time': '2023-10-05T19:00:00.000000Z', 'epoch': 1696532400, 'owner': 'foo', 'categories': [{'category': 'list_buckets', 'bytes_sent': 322, 'bytes_received': 0, 'ops': 1, 'successful_ops': 1}]}, {'bucket': 'myfoo', 'time': '2023-10-05T19:00:00.000000Z', 'epoch': 1696532400, 'owner': 'foo', 'categories': [{'category': 'create_bucket', 'bytes_sent': 0, 'bytes_received': 0, 'ops': 5, 'successful_ops': 5}, {'category': 'delete_bucket', 'bytes_sent': 217, 'bytes_received': 0, 'ops': 2, 'successful_ops': 1}, {'category': 'delete_obj', 'bytes_sent': 0, 'bytes_received': 0, 'ops': 4, 'successful_ops': 4}, {'category': 'get_acls', 'bytes_sent': 1026, 'bytes_received': 0, 'ops': 2, 'successful_ops': 2}, {'category': 'get_obj', 'bytes_sent': 4, 'bytes_received': 0, 'ops': 1, 'successful_ops': 1}, {'category': 'list_bucket', 'bytes_sent': 515, 'bytes_received': 0, 'ops': 1, 'successful_ops': 1}, {'category': 'put_acls', 'bytes_sent': 0, 'bytes_received': 0, 'ops': 2, 'successful_ops': 2}, {'category': 'put_obj', 'bytes_sent': 0, 'bytes_received': 30000042, 'ops': 11, 'successful_ops': 11}]}, {'bucket': 'myfoo2', 'time': '2023-10-05T19:00:00.000000Z', 'epoch': 1696532400, 'owner': 'foo', 'categories': [{'category': 'create_bucket', 'bytes_sent': 0, 'bytes_received': 0, 'ops': 1, 'successful_ops': 1}, {'category': 'delete_bucket', 'bytes_sent': 0, 'bytes_received': 0, 'ops': 1, 'successful_ops': 1}]}, {'bucket': 'myfoo3', 'time': '2023-10-05T19:00:00.000000Z', 'epoch': 1696532400, 'owner': 'foo', 'categories': [{'category': 'create_bucket', 'bytes_sent': 0, 'bytes_received': 0, 'ops': 1, 'successful_ops': 1}, {'category': 'delete_bucket', 'bytes_sent': 0, 'bytes_received': 0, 'ops': 1, 'successful_ops': 1}]}, {'bucket': 'myfoo4', 'time': '2023-10-05T19:00:00.000000Z', 'epoch': 1696532400, 'owner': 'foo', 'categories': [{'category': 'create_bucket', 'bytes_sent': 0, 'bytes_received': 0, 'ops': 1, 'successful_ops': 1}, {'category': 'delete_bucket', 'bytes_sent': 0, 'bytes_received': 0, 'ops': 1, 'successful_ops': 1}]}]}], 'summary': [{'user': 'foo', 'categories': [{'category': 'create_bucket', 'bytes_sent': 0, 'bytes_received': 0, 'ops': 8, 'successful_ops': 8}, {'category': 'delete_bucket', 'bytes_sent': 217, 'bytes_received': 0, 'ops': 5, 'successful_ops': 4}, {'category': 'delete_obj', 'bytes_sent': 0, 'bytes_received': 0, 'ops': 4, 'successful_ops': 4}, {'category': 'get_acls', 'bytes_sent': 1026, 'bytes_received': 0, 'ops': 2, 'successful_ops': 2}, {'category': 'get_obj', 'bytes_sent': 4, 'bytes_received': 0, 'ops': 1, 'successful_ops': 1}, {'category': 'list_bucket', 'bytes_sent': 515, 'bytes_received': 0, 'ops': 1, 'successful_ops': 1}, {'category': 'list_buckets', 'bytes_sent': 322, 'bytes_received': 0, 'ops': 1, 'successful_ops': 1}, {'category': 'put_acls', 'bytes_sent': 0, 'bytes_received': 0, 'ops': 2, 'successful_ops': 2}, {'category': 'put_obj', 'bytes_sent': 0, 'bytes_received': 30000042, 'ops': 11, 'successful_ops': 11}], 'total': {'bytes_sent': 2084, 'bytes_received': 30000042, 'ops': 35, 'successful_ops': 34}}]} 2023-10-05T19:29:33.590 INFO:tasks.util.rgw:rgwadmin: client.0 : ['usage', 'show', '--uid', 'foo', '--categories', 'create_bucket'] 2023-10-05T19:29:33.590 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'ceph', 'usage', 'show', '--uid', 'foo', '--categories', 'create_bucket'] 2023-10-05T19:29:33.590 DEBUG:teuthology.orchestra.run.smithi167:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster ceph usage show --uid foo --categories create_bucket 2023-10-05T19:29:33.630 INFO:teuthology.orchestra.run.smithi167.stderr:ignoring --setuser ceph since I am not root 2023-10-05T19:29:33.630 INFO:teuthology.orchestra.run.smithi167.stderr:ignoring --setgroup ceph since I am not root 2023-10-05T19:29:33.631 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.626+0000 7f4079625e40 1 Processor -- start 2023-10-05T19:29:33.631 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.626+0000 7f4079625e40 1 -- start start 2023-10-05T19:29:33.631 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.626+0000 7f4079625e40 1 --2- >> v2:172.21.15.118:3300/0 conn(0x564f0b979030 0x564f0b979400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-05T19:29:33.631 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.626+0000 7f4079625e40 1 -- --> v1:172.21.15.118:6789/0 -- auth(proto 0 26 bytes epoch 0) v1 -- 0x564f0b8c3bd0 con 0x564f0b979940 2023-10-05T19:29:33.631 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.626+0000 7f4079625e40 1 -- --> v2:172.21.15.118:3300/0 -- mon_getmap magic: 0 v1 -- 0x564f0b880ee0 con 0x564f0b979030 2023-10-05T19:29:33.632 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.626+0000 7f407715a700 1 --2- >> v2:172.21.15.118:3300/0 conn(0x564f0b979030 0x564f0b979400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-05T19:29:33.632 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.626+0000 7f4076959700 1 --1- >> v1:172.21.15.118:6789/0 conn(0x564f0b979940 0x564f0b97a3e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.118:6789/0 says I am v1:172.21.15.167:52610/0 (socket says 172.21.15.167:52610) 2023-10-05T19:29:33.632 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.626+0000 7f4076959700 1 -- 172.21.15.167:0/24694540 learned_addr learned my addr 172.21.15.167:0/24694540 (peer_addr_for_me v1:172.21.15.167:0/0) 2023-10-05T19:29:33.632 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.626+0000 7f407715a700 1 -- 172.21.15.167:0/24694540 >> v1:172.21.15.118:6789/0 conn(0x564f0b979940 legacy=0x564f0b97a3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-05T19:29:33.632 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.626+0000 7f407715a700 1 -- 172.21.15.167:0/24694540 --> v2:172.21.15.118:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x564f0b97aaf0 con 0x564f0b979030 2023-10-05T19:29:33.633 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.626+0000 7f407715a700 1 --2- 172.21.15.167:0/24694540 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x564f0b979030 0x564f0b979400 secure :-1 s=READY pgs=591 cs=0 l=1 rev1=1 crypto rx=0x7f406000a5d0 tx=0x7f4060005a50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=335c0406eea97701 server_cookie=0 in_seq=0 out_seq=0 2023-10-05T19:29:33.633 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.630+0000 7f4076158700 1 -- 172.21.15.167:0/24694540 <== mon.0 v2:172.21.15.118:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f40600086a0 con 0x564f0b979030 2023-10-05T19:29:33.633 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.630+0000 7f4076158700 1 -- 172.21.15.167:0/24694540 <== mon.0 v2:172.21.15.118:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4060008800 con 0x564f0b979030 2023-10-05T19:29:33.633 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.630+0000 7f4076158700 1 -- 172.21.15.167:0/24694540 <== mon.0 v2:172.21.15.118:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f4060008ad0 con 0x564f0b979030 2023-10-05T19:29:33.634 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.630+0000 7f4079625e40 1 -- 172.21.15.167:0/24694540 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x564f0b979030 msgr2=0x564f0b979400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-05T19:29:33.634 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.630+0000 7f4079625e40 1 --2- 172.21.15.167:0/24694540 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x564f0b979030 0x564f0b979400 secure :-1 s=READY pgs=591 cs=0 l=1 rev1=1 crypto rx=0x7f406000a5d0 tx=0x7f4060005a50 comp rx=0 tx=0).stop 2023-10-05T19:29:33.634 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.630+0000 7f4079625e40 1 -- 172.21.15.167:0/24694540 shutdown_connections 2023-10-05T19:29:33.634 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.630+0000 7f4079625e40 1 --2- 172.21.15.167:0/24694540 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x564f0b979030 0x564f0b979400 unknown :-1 s=CLOSED pgs=591 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-05T19:29:33.634 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.630+0000 7f4079625e40 1 -- 172.21.15.167:0/24694540 >> 172.21.15.167:0/24694540 conn(0x564f0b8ec880 msgr2=0x564f0b977520 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-05T19:29:33.635 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.630+0000 7f4079625e40 1 -- 172.21.15.167:0/24694540 shutdown_connections 2023-10-05T19:29:33.635 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.630+0000 7f4079625e40 1 -- 172.21.15.167:0/24694540 wait complete. 2023-10-05T19:29:33.636 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.630+0000 7f4079625e40 1 Processor -- start 2023-10-05T19:29:33.636 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.630+0000 7f4079625e40 1 -- start start 2023-10-05T19:29:33.637 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.634+0000 7f4079625e40 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x564f0b979030 0x564f0b998100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-05T19:29:33.637 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.634+0000 7f4079625e40 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x564f0b844e30 con 0x564f0b979030 2023-10-05T19:29:33.637 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.634+0000 7f407715a700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x564f0b979030 0x564f0b998100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-05T19:29:33.637 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.634+0000 7f407715a700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x564f0b979030 0x564f0b998100 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.118:3300/0 says I am v2:172.21.15.167:46216/0 (socket says 172.21.15.167:46216) 2023-10-05T19:29:33.637 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.634+0000 7f407715a700 1 -- 172.21.15.167:0/686177304 learned_addr learned my addr 172.21.15.167:0/686177304 (peer_addr_for_me v2:172.21.15.167:0/0) 2023-10-05T19:29:33.638 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.634+0000 7f407715a700 1 -- 172.21.15.167:0/686177304 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4060005cf0 con 0x564f0b979030 2023-10-05T19:29:33.638 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.634+0000 7f407715a700 1 --2- 172.21.15.167:0/686177304 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x564f0b979030 0x564f0b998100 secure :-1 s=READY pgs=592 cs=0 l=1 rev1=1 crypto rx=0x7f406000a520 tx=0x7f4060015000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bec0d73551e166d2 server_cookie=0 in_seq=0 out_seq=0 2023-10-05T19:29:33.638 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.634+0000 7f4075156700 1 -- 172.21.15.167:0/686177304 <== mon.0 v2:172.21.15.118:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f4060008970 con 0x564f0b979030 2023-10-05T19:29:33.639 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.634+0000 7f4075156700 1 -- 172.21.15.167:0/686177304 <== mon.0 v2:172.21.15.118:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4060008ad0 con 0x564f0b979030 2023-10-05T19:29:33.639 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.634+0000 7f4075156700 1 -- 172.21.15.167:0/686177304 <== mon.0 v2:172.21.15.118:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f4060008dc0 con 0x564f0b979030 2023-10-05T19:29:33.639 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.634+0000 7f4079625e40 1 -- 172.21.15.167:0/686177304 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x564f0b979030 msgr2=0x564f0b998100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-05T19:29:33.639 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.634+0000 7f4079625e40 1 --2- 172.21.15.167:0/686177304 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x564f0b979030 0x564f0b998100 secure :-1 s=READY pgs=592 cs=0 l=1 rev1=1 crypto rx=0x7f406000a520 tx=0x7f4060015000 comp rx=0 tx=0).stop 2023-10-05T19:29:33.639 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.634+0000 7f4079625e40 1 -- 172.21.15.167:0/686177304 shutdown_connections 2023-10-05T19:29:33.640 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.634+0000 7f4079625e40 1 --2- 172.21.15.167:0/686177304 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x564f0b979030 0x564f0b998100 unknown :-1 s=CLOSED pgs=592 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-05T19:29:33.640 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.634+0000 7f4079625e40 1 -- 172.21.15.167:0/686177304 >> 172.21.15.167:0/686177304 conn(0x564f0b8ec880 msgr2=0x564f0b993cc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-05T19:29:33.640 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.634+0000 7f4079625e40 1 -- 172.21.15.167:0/686177304 shutdown_connections 2023-10-05T19:29:33.640 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.634+0000 7f4079625e40 1 -- 172.21.15.167:0/686177304 wait complete. 2023-10-05T19:29:33.641 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.634+0000 7f4079625e40 1 Processor -- start 2023-10-05T19:29:33.641 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.634+0000 7f4079625e40 1 -- start start 2023-10-05T19:29:33.641 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.634+0000 7f4079625e40 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x564f0b979940 0x564f0b99eed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-05T19:29:33.641 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.634+0000 7f4079625e40 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x564f0b817ed0 con 0x564f0b979940 2023-10-05T19:29:33.641 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.638+0000 7f407715a700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x564f0b979940 0x564f0b99eed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-05T19:29:33.641 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.638+0000 7f407715a700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x564f0b979940 0x564f0b99eed0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.118:3300/0 says I am v2:172.21.15.167:46230/0 (socket says 172.21.15.167:46230) 2023-10-05T19:29:33.641 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.638+0000 7f407715a700 1 -- 172.21.15.167:0/153311785 learned_addr learned my addr 172.21.15.167:0/153311785 (peer_addr_for_me v2:172.21.15.167:0/0) 2023-10-05T19:29:33.642 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.638+0000 7f407715a700 1 -- 172.21.15.167:0/153311785 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x564f0b97aaf0 con 0x564f0b979940 2023-10-05T19:29:33.642 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.638+0000 7f407715a700 1 --2- 172.21.15.167:0/153311785 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x564f0b979940 0x564f0b99eed0 secure :-1 s=READY pgs=593 cs=0 l=1 rev1=1 crypto rx=0x7f406000c010 tx=0x7f4060003050 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-05T19:29:33.643 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.638+0000 7f406f7fe700 1 -- 172.21.15.167:0/153311785 <== mon.0 v2:172.21.15.118:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f406001c410 con 0x564f0b979940 2023-10-05T19:29:33.643 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.638+0000 7f406f7fe700 1 -- 172.21.15.167:0/153311785 <== mon.0 v2:172.21.15.118:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f406001c570 con 0x564f0b979940 2023-10-05T19:29:33.643 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.638+0000 7f4079625e40 1 -- 172.21.15.167:0/153311785 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x564f0b99f410 con 0x564f0b979940 2023-10-05T19:29:33.643 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.638+0000 7f406f7fe700 1 -- 172.21.15.167:0/153311785 <== mon.0 v2:172.21.15.118:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f406001c820 con 0x564f0b979940 2023-10-05T19:29:33.643 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.638+0000 7f4079625e40 1 -- 172.21.15.167:0/153311785 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x564f0b99f850 con 0x564f0b979940 2023-10-05T19:29:33.644 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.638+0000 7f406f7fe700 1 -- 172.21.15.167:0/153311785 <== mon.0 v2:172.21.15.118:3300/0 4 ==== mgrmap(e 5) v1 ==== 40329+0+0 (secure 0 0 0) 0x7f4060019070 con 0x564f0b979940 2023-10-05T19:29:33.644 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.638+0000 7f4079625e40 20 rados->read ofs=0 len=0 2023-10-05T19:29:33.644 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.638+0000 7f406f7fe700 1 --2- 172.21.15.167:0/153311785 >> [v2:172.21.15.167:6824/29889,v1:172.21.15.167:6825/29889] conn(0x7f4058030c50 0x7f4058033100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-05T19:29:33.644 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.642+0000 7f406f7fe700 1 -- 172.21.15.167:0/153311785 <== mon.0 v2:172.21.15.118:3300/0 5 ==== osd_map(32..32 src has 1..32) v4 ==== 6260+0+0 (secure 0 0 0) 0x7f4060046270 con 0x564f0b979940 2023-10-05T19:29:33.645 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.642+0000 7f4076959700 1 --2- 172.21.15.167:0/153311785 >> [v2:172.21.15.167:6824/29889,v1:172.21.15.167:6825/29889] conn(0x7f4058030c50 0x7f4058033100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-05T19:29:33.645 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.642+0000 7f4079625e40 1 --2- 172.21.15.167:0/153311785 >> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] conn(0x564f0b9a8950 0x564f0b9aad30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-05T19:29:33.645 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.642+0000 7f4079625e40 1 -- 172.21.15.167:0/153311785 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:1 5.12 5:49953fa1:::default.realm:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32) v8 -- 0x564f0b9ab3e0 con 0x564f0b9a8950 2023-10-05T19:29:33.645 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.642+0000 7f407795b700 1 --2- 172.21.15.167:0/153311785 >> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] conn(0x564f0b9a8950 0x564f0b9aad30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-05T19:29:33.646 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.642+0000 7f4076959700 1 --2- 172.21.15.167:0/153311785 >> [v2:172.21.15.167:6824/29889,v1:172.21.15.167:6825/29889] conn(0x7f4058030c50 0x7f4058033100 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7f40700044f0 tx=0x7f40700067c0 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-05T19:29:33.646 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.642+0000 7f407795b700 1 --2- 172.21.15.167:0/153311785 >> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] conn(0x564f0b9a8950 0x564f0b9aad30 crc :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-05T19:29:33.647 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.642+0000 7f407795b700 1 -- 172.21.15.167:0/153311785 <== osd.1 v2:172.21.15.167:6800/30035 1 ==== osd_op_reply(1 default.realm [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 ==== 157+0+0 (crc 0 0 0) 0x7f406800a500 con 0x564f0b9a8950 2023-10-05T19:29:33.647 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.642+0000 7f4079625e40 20 rados_obj.operate() r=-2 bl.length=0 2023-10-05T19:29:33.647 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.642+0000 7f4079625e40 20 realm 2023-10-05T19:29:33.647 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.642+0000 7f4079625e40 20 rados->read ofs=0 len=0 2023-10-05T19:29:33.647 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.642+0000 7f4079625e40 1 -- 172.21.15.167:0/153311785 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:2 5.12 5:49953fa1:::default.realm:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32) v8 -- 0x564f0b9a7fc0 con 0x564f0b9a8950 2023-10-05T19:29:33.647 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.642+0000 7f407795b700 1 -- 172.21.15.167:0/153311785 <== osd.1 v2:172.21.15.167:6800/30035 2 ==== osd_op_reply(2 default.realm [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 ==== 157+0+0 (crc 0 0 0) 0x7f406800a500 con 0x564f0b9a8950 2023-10-05T19:29:33.647 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.642+0000 7f4079625e40 20 rados_obj.operate() r=-2 bl.length=0 2023-10-05T19:29:33.648 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.642+0000 7f4079625e40 4 RGWPeriod::init failed to init realm id : (2) No such file or directory 2023-10-05T19:29:33.648 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.642+0000 7f4079625e40 20 rados->read ofs=0 len=0 2023-10-05T19:29:33.648 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.642+0000 7f4079625e40 1 -- 172.21.15.167:0/153311785 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:3 5.12 5:49953fa1:::default.realm:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32) v8 -- 0x564f0b9abec0 con 0x564f0b9a8950 2023-10-05T19:29:33.649 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.642+0000 7f407795b700 1 -- 172.21.15.167:0/153311785 <== osd.1 v2:172.21.15.167:6800/30035 3 ==== osd_op_reply(3 default.realm [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 ==== 157+0+0 (crc 0 0 0) 0x7f406800a500 con 0x564f0b9a8950 2023-10-05T19:29:33.649 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.642+0000 7f4079625e40 20 rados_obj.operate() r=-2 bl.length=0 2023-10-05T19:29:33.649 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.642+0000 7f4079625e40 20 rados->read ofs=0 len=0 2023-10-05T19:29:33.649 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.642+0000 7f4079625e40 1 -- 172.21.15.167:0/153311785 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:4 5.f 5:f43b6ece:::zone_names.default:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32) v8 -- 0x564f0b9abaa0 con 0x564f0b9a8950 2023-10-05T19:29:33.650 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.642+0000 7f407795b700 1 -- 172.21.15.167:0/153311785 <== osd.1 v2:172.21.15.167:6800/30035 4 ==== osd_op_reply(4 zone_names.default [read 0~46 out=46b] v0'0 uv1 ondisk = 0) v8 ==== 162+0+46 (crc 0 0 0) 0x7f406800a500 con 0x564f0b9a8950 2023-10-05T19:29:33.650 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.642+0000 7f4079625e40 20 rados_obj.operate() r=0 bl.length=46 2023-10-05T19:29:33.650 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.642+0000 7f4079625e40 20 rados->read ofs=0 len=0 2023-10-05T19:29:33.650 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.646+0000 7f4079625e40 1 -- 172.21.15.167:0/153311785 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:5 5.1d 5:b8374eb4:::zone_info.9a9df965-6612-45d4-a45f-6e3c889bacc2:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32) v8 -- 0x564f0b9ac930 con 0x564f0b9a8950 2023-10-05T19:29:33.650 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.646+0000 7f407795b700 1 -- 172.21.15.167:0/153311785 <== osd.1 v2:172.21.15.167:6800/30035 5 ==== osd_op_reply(5 zone_info.9a9df965-6612-45d4-a45f-6e3c889bacc2 [read 0~874 out=874b] v0'0 uv2 ondisk = 0) v8 ==== 190+0+874 (crc 0 0 0) 0x7f406800a500 con 0x564f0b9a8950 2023-10-05T19:29:33.650 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.646+0000 7f4079625e40 20 rados_obj.operate() r=0 bl.length=874 2023-10-05T19:29:33.650 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.646+0000 7f4079625e40 20 searching for the correct realm 2023-10-05T19:29:33.650 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.646+0000 7f4079625e40 1 --2- 172.21.15.167:0/153311785 >> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] conn(0x564f0b9ae780 0x564f0b9b0b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-05T19:29:33.650 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.646+0000 7f4079625e40 1 -- 172.21.15.167:0/153311785 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:6 5.0 5:00000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x564f0b9b10a0 con 0x564f0b9ae780 2023-10-05T19:29:33.650 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.646+0000 7f407715a700 1 --2- 172.21.15.167:0/153311785 >> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] conn(0x564f0b9ae780 0x564f0b9b0b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-05T19:29:33.651 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.646+0000 7f407715a700 1 --2- 172.21.15.167:0/153311785 >> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] conn(0x564f0b9ae780 0x564f0b9b0b60 crc :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-05T19:29:33.651 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.646+0000 7f407715a700 1 -- 172.21.15.167:0/153311785 <== osd.3 v2:172.21.15.167:6808/30036 1 ==== osd_op_reply(6 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f4060022020 con 0x564f0b9ae780 2023-10-05T19:29:33.652 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.646+0000 7f407715a700 1 --2- 172.21.15.167:0/153311785 >> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] conn(0x7f4060038070 0x7f40600113b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-05T19:29:33.652 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.646+0000 7f407715a700 1 -- 172.21.15.167:0/153311785 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:7 5.10 5:08000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x564f0b9b10a0 con 0x7f4060038070 2023-10-05T19:29:33.652 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.646+0000 7f4076959700 1 --2- 172.21.15.167:0/153311785 >> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] conn(0x7f4060038070 0x7f40600113b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-05T19:29:33.652 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.646+0000 7f4076959700 1 --2- 172.21.15.167:0/153311785 >> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] conn(0x7f4060038070 0x7f40600113b0 crc :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-05T19:29:33.653 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.650+0000 7f4076959700 1 -- 172.21.15.167:0/153311785 <== osd.2 v2:172.21.15.167:6816/30037 1 ==== osd_op_reply(7 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f4070004520 con 0x7f4060038070 2023-10-05T19:29:33.653 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.650+0000 7f4076959700 1 --2- 172.21.15.167:0/153311785 >> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] conn(0x7f4070011040 0x7f4070014950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-05T19:29:33.653 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.650+0000 7f4076959700 1 -- 172.21.15.167:0/153311785 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:8 5.8 5:10000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x564f0b9b10a0 con 0x7f4070011040 2023-10-05T19:29:33.653 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.650+0000 7f407795b700 1 --2- 172.21.15.167:0/153311785 >> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] conn(0x7f4070011040 0x7f4070014950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-05T19:29:33.654 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.650+0000 7f407795b700 1 --2- 172.21.15.167:0/153311785 >> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] conn(0x7f4070011040 0x7f4070014950 crc :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-05T19:29:33.655 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.650+0000 7f407795b700 1 -- 172.21.15.167:0/153311785 <== osd.0 v2:172.21.15.118:6800/27690 1 ==== osd_op_reply(8 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f4068014040 con 0x7f4070011040 2023-10-05T19:29:33.655 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.650+0000 7f407795b700 1 -- 172.21.15.167:0/153311785 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:9 5.18 5:18000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x564f0b9b10a0 con 0x564f0b9a8950 2023-10-05T19:29:33.656 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.650+0000 7f407795b700 1 -- 172.21.15.167:0/153311785 <== osd.1 v2:172.21.15.167:6800/30035 6 ==== osd_op_reply(9 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f4068014040 con 0x564f0b9a8950 2023-10-05T19:29:33.656 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.650+0000 7f407795b700 1 -- 172.21.15.167:0/153311785 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:10 5.4 5:20000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x564f0b9b10a0 con 0x564f0b9a8950 2023-10-05T19:29:33.656 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.650+0000 7f407795b700 1 -- 172.21.15.167:0/153311785 <== osd.1 v2:172.21.15.167:6800/30035 7 ==== osd_op_reply(10 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f4068014040 con 0x564f0b9a8950 2023-10-05T19:29:33.656 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.650+0000 7f407795b700 1 -- 172.21.15.167:0/153311785 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:11 5.14 5:28000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x564f0b9b10a0 con 0x7f4070011040 2023-10-05T19:29:33.657 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.654+0000 7f407795b700 1 -- 172.21.15.167:0/153311785 <== osd.0 v2:172.21.15.118:6800/27690 2 ==== osd_op_reply(11 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f4068014040 con 0x7f4070011040 2023-10-05T19:29:33.657 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.654+0000 7f407795b700 1 -- 172.21.15.167:0/153311785 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:12 5.c 5:30000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x564f0b9b10a0 con 0x7f4070011040 2023-10-05T19:29:33.658 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.654+0000 7f407795b700 1 -- 172.21.15.167:0/153311785 <== osd.0 v2:172.21.15.118:6800/27690 3 ==== osd_op_reply(12 [pgnls start_epoch 32 out=112b] v23'1 uv1 ondisk = 1) v8 ==== 144+0+112 (crc 0 0 0) 0x7f4068014040 con 0x7f4070011040 2023-10-05T19:29:33.658 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.654+0000 7f407795b700 1 -- 172.21.15.167:0/153311785 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:13 5.1c 5:38000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x564f0b9b10a0 con 0x7f4060038070 2023-10-05T19:29:33.659 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.654+0000 7f4076959700 1 -- 172.21.15.167:0/153311785 <== osd.2 v2:172.21.15.167:6816/30037 2 ==== osd_op_reply(13 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f4070004520 con 0x7f4060038070 2023-10-05T19:29:33.659 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.654+0000 7f4076959700 1 -- 172.21.15.167:0/153311785 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:14 5.2 5:40000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x564f0b9b10a0 con 0x564f0b9ae780 2023-10-05T19:29:33.660 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.654+0000 7f407715a700 1 -- 172.21.15.167:0/153311785 <== osd.3 v2:172.21.15.167:6808/30036 2 ==== osd_op_reply(14 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f4060022020 con 0x564f0b9ae780 2023-10-05T19:29:33.660 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.654+0000 7f407715a700 1 -- 172.21.15.167:0/153311785 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:15 5.12 5:48000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x564f0b9b10a0 con 0x564f0b9a8950 2023-10-05T19:29:33.660 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.658+0000 7f407795b700 1 -- 172.21.15.167:0/153311785 <== osd.1 v2:172.21.15.167:6800/30035 8 ==== osd_op_reply(15 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f4068014040 con 0x564f0b9a8950 2023-10-05T19:29:33.660 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.658+0000 7f407795b700 1 -- 172.21.15.167:0/153311785 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:16 5.a 5:50000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x564f0b9b10a0 con 0x7f4070011040 2023-10-05T19:29:33.661 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.658+0000 7f407795b700 1 -- 172.21.15.167:0/153311785 <== osd.0 v2:172.21.15.118:6800/27690 4 ==== osd_op_reply(16 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f4068022040 con 0x7f4070011040 2023-10-05T19:29:33.661 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.658+0000 7f407795b700 1 -- 172.21.15.167:0/153311785 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:17 5.1a 5:58000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x564f0b9b10a0 con 0x564f0b9a8950 2023-10-05T19:29:33.662 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.658+0000 7f407795b700 1 -- 172.21.15.167:0/153311785 <== osd.1 v2:172.21.15.167:6800/30035 9 ==== osd_op_reply(17 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f4068022040 con 0x564f0b9a8950 2023-10-05T19:29:33.662 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.658+0000 7f407795b700 1 -- 172.21.15.167:0/153311785 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:18 5.6 5:60000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x564f0b9b10a0 con 0x7f4060038070 2023-10-05T19:29:33.663 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.658+0000 7f4076959700 1 -- 172.21.15.167:0/153311785 <== osd.2 v2:172.21.15.167:6816/30037 3 ==== osd_op_reply(18 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f4070004520 con 0x7f4060038070 2023-10-05T19:29:33.663 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.658+0000 7f4076959700 1 -- 172.21.15.167:0/153311785 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:19 5.16 5:68000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x564f0b9b10a0 con 0x564f0b9ae780 2023-10-05T19:29:33.663 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.658+0000 7f407715a700 1 -- 172.21.15.167:0/153311785 <== osd.3 v2:172.21.15.167:6808/30036 3 ==== osd_op_reply(19 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f4060022020 con 0x564f0b9ae780 2023-10-05T19:29:33.663 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.658+0000 7f407715a700 1 -- 172.21.15.167:0/153311785 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:20 5.e 5:70000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x564f0b9b10a0 con 0x564f0b9ae780 2023-10-05T19:29:33.664 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.658+0000 7f407715a700 1 -- 172.21.15.167:0/153311785 <== osd.3 v2:172.21.15.167:6808/30036 4 ==== osd_op_reply(20 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f4060022020 con 0x564f0b9ae780 2023-10-05T19:29:33.664 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.658+0000 7f407715a700 1 -- 172.21.15.167:0/153311785 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:21 5.1e 5:78000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x564f0b9b10a0 con 0x7f4070011040 2023-10-05T19:29:33.665 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.662+0000 7f407795b700 1 -- 172.21.15.167:0/153311785 <== osd.0 v2:172.21.15.118:6800/27690 5 ==== osd_op_reply(21 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f4068022040 con 0x7f4070011040 2023-10-05T19:29:33.665 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.662+0000 7f407795b700 1 -- 172.21.15.167:0/153311785 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:22 5.1 5:80000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x564f0b9b10a0 con 0x564f0b9ae780 2023-10-05T19:29:33.666 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.662+0000 7f407715a700 1 -- 172.21.15.167:0/153311785 <== osd.3 v2:172.21.15.167:6808/30036 5 ==== osd_op_reply(22 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f4060022020 con 0x564f0b9ae780 2023-10-05T19:29:33.666 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.662+0000 7f407715a700 1 -- 172.21.15.167:0/153311785 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:23 5.11 5:88000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x564f0b9b10a0 con 0x564f0b9ae780 2023-10-05T19:29:33.667 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.662+0000 7f407715a700 1 -- 172.21.15.167:0/153311785 <== osd.3 v2:172.21.15.167:6808/30036 6 ==== osd_op_reply(23 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f4060022020 con 0x564f0b9ae780 2023-10-05T19:29:33.667 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.662+0000 7f407715a700 1 -- 172.21.15.167:0/153311785 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:24 5.9 5:90000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x564f0b9b10a0 con 0x564f0b9a8950 2023-10-05T19:29:33.667 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.662+0000 7f407795b700 1 -- 172.21.15.167:0/153311785 <== osd.1 v2:172.21.15.167:6800/30035 10 ==== osd_op_reply(24 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f4068022040 con 0x564f0b9a8950 2023-10-05T19:29:33.668 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.662+0000 7f407795b700 1 -- 172.21.15.167:0/153311785 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:25 5.19 5:98000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x564f0b9b10a0 con 0x564f0b9a8950 2023-10-05T19:29:33.668 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.662+0000 7f407795b700 1 -- 172.21.15.167:0/153311785 <== osd.1 v2:172.21.15.167:6800/30035 11 ==== osd_op_reply(25 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f4068022040 con 0x564f0b9a8950 2023-10-05T19:29:33.668 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.662+0000 7f407795b700 1 -- 172.21.15.167:0/153311785 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:26 5.5 5:a0000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x564f0b9b10a0 con 0x7f4070011040 2023-10-05T19:29:33.669 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.666+0000 7f407795b700 1 -- 172.21.15.167:0/153311785 <== osd.0 v2:172.21.15.118:6800/27690 6 ==== osd_op_reply(26 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f4068022040 con 0x7f4070011040 2023-10-05T19:29:33.669 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.666+0000 7f407795b700 1 -- 172.21.15.167:0/153311785 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:27 5.15 5:a8000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x564f0b9b10a0 con 0x564f0b9ae780 2023-10-05T19:29:33.670 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.666+0000 7f407715a700 1 -- 172.21.15.167:0/153311785 <== osd.3 v2:172.21.15.167:6808/30036 7 ==== osd_op_reply(27 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f4060022020 con 0x564f0b9ae780 2023-10-05T19:29:33.670 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.666+0000 7f407715a700 1 -- 172.21.15.167:0/153311785 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:28 5.d 5:b0000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x564f0b9b10a0 con 0x7f4060038070 2023-10-05T19:29:33.670 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.666+0000 7f4076959700 1 -- 172.21.15.167:0/153311785 <== osd.2 v2:172.21.15.167:6816/30037 4 ==== osd_op_reply(28 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f4070004520 con 0x7f4060038070 2023-10-05T19:29:33.670 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.666+0000 7f4076959700 1 -- 172.21.15.167:0/153311785 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:29 5.1d 5:b8000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x564f0b9b10a0 con 0x564f0b9a8950 2023-10-05T19:29:33.671 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.666+0000 7f407795b700 1 -- 172.21.15.167:0/153311785 <== osd.1 v2:172.21.15.167:6800/30035 12 ==== osd_op_reply(29 [pgnls start_epoch 32 out=107b] v30'2 uv2 ondisk = 1) v8 ==== 144+0+107 (crc 0 0 0) 0x7f4068022040 con 0x564f0b9a8950 2023-10-05T19:29:33.671 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.666+0000 7f407795b700 1 -- 172.21.15.167:0/153311785 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:30 5.3 5:c0000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x564f0b9b10a0 con 0x7f4070011040 2023-10-05T19:29:33.672 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.666+0000 7f407795b700 1 -- 172.21.15.167:0/153311785 <== osd.0 v2:172.21.15.118:6800/27690 7 ==== osd_op_reply(30 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f4068022040 con 0x7f4070011040 2023-10-05T19:29:33.673 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.666+0000 7f407795b700 1 -- 172.21.15.167:0/153311785 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:31 5.13 5:c8000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x564f0b9b10a0 con 0x564f0b9ae780 2023-10-05T19:29:33.673 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.670+0000 7f407715a700 1 -- 172.21.15.167:0/153311785 <== osd.3 v2:172.21.15.167:6808/30036 8 ==== osd_op_reply(31 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f4060022020 con 0x564f0b9ae780 2023-10-05T19:29:33.673 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.670+0000 7f407715a700 1 -- 172.21.15.167:0/153311785 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:32 5.b 5:d0000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x564f0b9b10a0 con 0x7f4070011040 2023-10-05T19:29:33.674 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.670+0000 7f407795b700 1 -- 172.21.15.167:0/153311785 <== osd.0 v2:172.21.15.118:6800/27690 8 ==== osd_op_reply(32 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f4068022040 con 0x7f4070011040 2023-10-05T19:29:33.674 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.670+0000 7f407795b700 1 -- 172.21.15.167:0/153311785 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:33 5.1b 5:d8000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x564f0b9b10a0 con 0x7f4060038070 2023-10-05T19:29:33.678 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.670+0000 7f4076959700 1 -- 172.21.15.167:0/153311785 <== osd.2 v2:172.21.15.167:6816/30037 5 ==== osd_op_reply(33 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f4070004520 con 0x7f4060038070 2023-10-05T19:29:33.678 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.670+0000 7f4076959700 1 -- 172.21.15.167:0/153311785 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:34 5.7 5:e0000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x564f0b9b10a0 con 0x564f0b9ae780 2023-10-05T19:29:33.678 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.670+0000 7f407715a700 1 -- 172.21.15.167:0/153311785 <== osd.3 v2:172.21.15.167:6808/30036 9 ==== osd_op_reply(34 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f4060022020 con 0x564f0b9ae780 2023-10-05T19:29:33.678 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.670+0000 7f407715a700 1 -- 172.21.15.167:0/153311785 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:35 5.17 5:e8000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x564f0b9b10a0 con 0x7f4070011040 2023-10-05T19:29:33.678 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.674+0000 7f407795b700 1 -- 172.21.15.167:0/153311785 <== osd.0 v2:172.21.15.118:6800/27690 9 ==== osd_op_reply(35 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f4068022040 con 0x7f4070011040 2023-10-05T19:29:33.678 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.674+0000 7f407795b700 1 -- 172.21.15.167:0/153311785 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:36 5.f 5:f0000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x564f0b9b10a0 con 0x564f0b9a8950 2023-10-05T19:29:33.678 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.674+0000 7f407795b700 1 -- 172.21.15.167:0/153311785 <== osd.1 v2:172.21.15.167:6800/30035 13 ==== osd_op_reply(36 [pgnls start_epoch 32 out=115b] v23'2 uv2 ondisk = 1) v8 ==== 144+0+115 (crc 0 0 0) 0x7f406802f080 con 0x564f0b9a8950 2023-10-05T19:29:33.678 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.674+0000 7f407795b700 1 -- 172.21.15.167:0/153311785 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:37 5.1f 5:f8000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x564f0b9b10a0 con 0x7f4060038070 2023-10-05T19:29:33.679 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.674+0000 7f4076959700 1 -- 172.21.15.167:0/153311785 <== osd.2 v2:172.21.15.167:6816/30037 6 ==== osd_op_reply(37 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f4070004520 con 0x7f4060038070 2023-10-05T19:29:33.679 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.674+0000 7f4079625e40 20 RGWRados::pool_iterate: got zonegroup_info.4f1f719d-9648-417b-b10c-e5d939825419 2023-10-05T19:29:33.679 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.674+0000 7f4079625e40 20 RGWRados::pool_iterate: got zone_info.9a9df965-6612-45d4-a45f-6e3c889bacc2 2023-10-05T19:29:33.679 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.674+0000 7f4079625e40 20 RGWRados::pool_iterate: got zone_names.default 2023-10-05T19:29:33.679 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.674+0000 7f4079625e40 20 RGWRados::pool_iterate: got zonegroups_names.default 2023-10-05T19:29:33.680 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.674+0000 7f4079625e40 20 rados->read ofs=0 len=0 2023-10-05T19:29:33.680 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.674+0000 7f4079625e40 1 -- 172.21.15.167:0/153311785 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:38 5.12 5:49953fa1:::default.realm:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32) v8 -- 0x564f0b9ad530 con 0x564f0b9a8950 2023-10-05T19:29:33.680 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.674+0000 7f407795b700 1 -- 172.21.15.167:0/153311785 <== osd.1 v2:172.21.15.167:6800/30035 14 ==== osd_op_reply(38 default.realm [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 ==== 157+0+0 (crc 0 0 0) 0x7f406802f080 con 0x564f0b9a8950 2023-10-05T19:29:33.680 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.674+0000 7f4079625e40 20 rados_obj.operate() r=-2 bl.length=0 2023-10-05T19:29:33.680 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.674+0000 7f4079625e40 20 rados->read ofs=0 len=0 2023-10-05T19:29:33.680 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.674+0000 7f4079625e40 1 -- 172.21.15.167:0/153311785 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:39 5.f 5:f4c53578:::zonegroups_names.default:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32) v8 -- 0x564f0b9adeb0 con 0x564f0b9a8950 2023-10-05T19:29:33.681 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.674+0000 7f407795b700 1 -- 172.21.15.167:0/153311785 <== osd.1 v2:172.21.15.167:6800/30035 15 ==== osd_op_reply(39 zonegroups_names.default [read 0~46 out=46b] v0'0 uv2 ondisk = 0) v8 ==== 168+0+46 (crc 0 0 0) 0x7f406802f080 con 0x564f0b9a8950 2023-10-05T19:29:33.681 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.674+0000 7f4079625e40 20 rados_obj.operate() r=0 bl.length=46 2023-10-05T19:29:33.681 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.674+0000 7f4079625e40 20 rados->read ofs=0 len=0 2023-10-05T19:29:33.681 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.674+0000 7f4079625e40 1 -- 172.21.15.167:0/153311785 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:40 5.c 5:30d299a5:::zonegroup_info.4f1f719d-9648-417b-b10c-e5d939825419:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32) v8 -- 0x564f0b9ad120 con 0x7f4070011040 2023-10-05T19:29:33.681 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.678+0000 7f407795b700 1 -- 172.21.15.167:0/153311785 <== osd.0 v2:172.21.15.118:6800/27690 10 ==== osd_op_reply(40 zonegroup_info.4f1f719d-9648-417b-b10c-e5d939825419 [read 0~362 out=362b] v0'0 uv1 ondisk = 0) v8 ==== 195+0+362 (crc 0 0 0) 0x7f406802f080 con 0x7f4070011040 2023-10-05T19:29:33.681 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.678+0000 7f4079625e40 20 rados_obj.operate() r=0 bl.length=362 2023-10-05T19:29:33.681 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.678+0000 7f4079625e40 20 rados->read ofs=0 len=0 2023-10-05T19:29:33.681 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.678+0000 7f4079625e40 1 -- 172.21.15.167:0/153311785 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:41 5.13 5:cbbf6c60:::converted:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32) v8 -- 0x564f0b9b1990 con 0x564f0b9ae780 2023-10-05T19:29:33.682 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.678+0000 7f407715a700 1 -- 172.21.15.167:0/153311785 <== osd.3 v2:172.21.15.167:6808/30036 10 ==== osd_op_reply(41 converted [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 ==== 153+0+0 (crc 0 0 0) 0x7f4060022020 con 0x564f0b9ae780 2023-10-05T19:29:33.682 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.678+0000 7f4079625e40 20 rados_obj.operate() r=-2 bl.length=0 2023-10-05T19:29:33.682 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.678+0000 7f4079625e40 20 rados->read ofs=0 len=0 2023-10-05T19:29:33.683 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.678+0000 7f4079625e40 1 -- 172.21.15.167:0/153311785 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:42 5.12 5:49953fa1:::default.realm:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32) v8 -- 0x564f0b9b1d40 con 0x564f0b9a8950 2023-10-05T19:29:33.683 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.678+0000 7f407795b700 1 -- 172.21.15.167:0/153311785 <== osd.1 v2:172.21.15.167:6800/30035 16 ==== osd_op_reply(42 default.realm [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 ==== 157+0+0 (crc 0 0 0) 0x7f406802f080 con 0x564f0b9a8950 2023-10-05T19:29:33.683 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.678+0000 7f4079625e40 20 rados_obj.operate() r=-2 bl.length=0 2023-10-05T19:29:33.683 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.678+0000 7f4079625e40 20 rados->read ofs=0 len=0 2023-10-05T19:29:33.683 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.678+0000 7f4079625e40 1 -- 172.21.15.167:0/153311785 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:43 5.f 5:f4c53578:::zonegroups_names.default:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32) v8 -- 0x564f0b9b1d40 con 0x564f0b9a8950 2023-10-05T19:29:33.684 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.678+0000 7f407795b700 1 -- 172.21.15.167:0/153311785 <== osd.1 v2:172.21.15.167:6800/30035 17 ==== osd_op_reply(43 zonegroups_names.default [read 0~46 out=46b] v0'0 uv2 ondisk = 0) v8 ==== 168+0+46 (crc 0 0 0) 0x7f406802f080 con 0x564f0b9a8950 2023-10-05T19:29:33.684 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.678+0000 7f4079625e40 20 rados_obj.operate() r=0 bl.length=46 2023-10-05T19:29:33.684 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.678+0000 7f4079625e40 1 -- 172.21.15.167:0/153311785 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:44 5.0 5:00000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x564f0b9b2d50 con 0x564f0b9ae780 2023-10-05T19:29:33.684 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.678+0000 7f407715a700 1 -- 172.21.15.167:0/153311785 <== osd.3 v2:172.21.15.167:6808/30036 11 ==== osd_op_reply(44 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f4060022020 con 0x564f0b9ae780 2023-10-05T19:29:33.684 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.678+0000 7f407715a700 1 -- 172.21.15.167:0/153311785 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:45 5.10 5:08000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x564f0b9b2d50 con 0x7f4060038070 2023-10-05T19:29:33.685 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.682+0000 7f4076959700 1 -- 172.21.15.167:0/153311785 <== osd.2 v2:172.21.15.167:6816/30037 7 ==== osd_op_reply(45 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f4070004520 con 0x7f4060038070 2023-10-05T19:29:33.685 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.682+0000 7f4076959700 1 -- 172.21.15.167:0/153311785 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:46 5.8 5:10000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x564f0b9b2d50 con 0x7f4070011040 2023-10-05T19:29:33.686 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.682+0000 7f407795b700 1 -- 172.21.15.167:0/153311785 <== osd.0 v2:172.21.15.118:6800/27690 11 ==== osd_op_reply(46 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f406802f080 con 0x7f4070011040 2023-10-05T19:29:33.686 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.682+0000 7f407795b700 1 -- 172.21.15.167:0/153311785 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:47 5.18 5:18000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x564f0b9ad120 con 0x564f0b9a8950 2023-10-05T19:29:33.687 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.682+0000 7f407795b700 1 -- 172.21.15.167:0/153311785 <== osd.1 v2:172.21.15.167:6800/30035 18 ==== osd_op_reply(47 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f406802f080 con 0x564f0b9a8950 2023-10-05T19:29:33.687 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.682+0000 7f407795b700 1 -- 172.21.15.167:0/153311785 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:48 5.4 5:20000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x564f0b9ad120 con 0x564f0b9a8950 2023-10-05T19:29:33.687 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.682+0000 7f407795b700 1 -- 172.21.15.167:0/153311785 <== osd.1 v2:172.21.15.167:6800/30035 19 ==== osd_op_reply(48 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f406802f080 con 0x564f0b9a8950 2023-10-05T19:29:33.688 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.682+0000 7f407795b700 1 -- 172.21.15.167:0/153311785 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:49 5.14 5:28000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x564f0b9ad120 con 0x7f4070011040 2023-10-05T19:29:33.688 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.686+0000 7f407795b700 1 -- 172.21.15.167:0/153311785 <== osd.0 v2:172.21.15.118:6800/27690 12 ==== osd_op_reply(49 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f406802f080 con 0x7f4070011040 2023-10-05T19:29:33.689 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.686+0000 7f407795b700 1 -- 172.21.15.167:0/153311785 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:50 5.c 5:30000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x564f0b9ad120 con 0x7f4070011040 2023-10-05T19:29:33.689 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.686+0000 7f407795b700 1 -- 172.21.15.167:0/153311785 <== osd.0 v2:172.21.15.118:6800/27690 13 ==== osd_op_reply(50 [pgnls start_epoch 32 out=112b] v23'1 uv1 ondisk = 1) v8 ==== 144+0+112 (crc 0 0 0) 0x7f406802f080 con 0x7f4070011040 2023-10-05T19:29:33.690 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.686+0000 7f407795b700 1 -- 172.21.15.167:0/153311785 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:51 5.1c 5:38000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x564f0b9ad120 con 0x7f4060038070 2023-10-05T19:29:33.690 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.686+0000 7f4076959700 1 -- 172.21.15.167:0/153311785 <== osd.2 v2:172.21.15.167:6816/30037 8 ==== osd_op_reply(51 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f4070004520 con 0x7f4060038070 2023-10-05T19:29:33.690 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.686+0000 7f4076959700 1 -- 172.21.15.167:0/153311785 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:52 5.2 5:40000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x564f0b9ad120 con 0x564f0b9ae780 2023-10-05T19:29:33.691 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.686+0000 7f407715a700 1 -- 172.21.15.167:0/153311785 <== osd.3 v2:172.21.15.167:6808/30036 12 ==== osd_op_reply(52 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f4060022020 con 0x564f0b9ae780 2023-10-05T19:29:33.691 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.686+0000 7f407715a700 1 -- 172.21.15.167:0/153311785 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:53 5.12 5:48000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x564f0b9ad120 con 0x564f0b9a8950 2023-10-05T19:29:33.692 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.686+0000 7f407795b700 1 -- 172.21.15.167:0/153311785 <== osd.1 v2:172.21.15.167:6800/30035 20 ==== osd_op_reply(53 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f406802f080 con 0x564f0b9a8950 2023-10-05T19:29:33.692 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.686+0000 7f407795b700 1 -- 172.21.15.167:0/153311785 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:54 5.a 5:50000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x564f0b9ad120 con 0x7f4070011040 2023-10-05T19:29:33.693 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.690+0000 7f407795b700 1 -- 172.21.15.167:0/153311785 <== osd.0 v2:172.21.15.118:6800/27690 14 ==== osd_op_reply(54 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f406802f080 con 0x7f4070011040 2023-10-05T19:29:33.693 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.690+0000 7f407795b700 1 -- 172.21.15.167:0/153311785 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:55 5.1a 5:58000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x564f0b9ad120 con 0x564f0b9a8950 2023-10-05T19:29:33.694 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.690+0000 7f407795b700 1 -- 172.21.15.167:0/153311785 <== osd.1 v2:172.21.15.167:6800/30035 21 ==== osd_op_reply(55 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f406802f080 con 0x564f0b9a8950 2023-10-05T19:29:33.694 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.690+0000 7f407795b700 1 -- 172.21.15.167:0/153311785 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:56 5.6 5:60000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x564f0b9ad120 con 0x7f4060038070 2023-10-05T19:29:33.695 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.690+0000 7f4076959700 1 -- 172.21.15.167:0/153311785 <== osd.2 v2:172.21.15.167:6816/30037 9 ==== osd_op_reply(56 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f4070004520 con 0x7f4060038070 2023-10-05T19:29:33.695 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.690+0000 7f4076959700 1 -- 172.21.15.167:0/153311785 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:57 5.16 5:68000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x564f0b9ad120 con 0x564f0b9ae780 2023-10-05T19:29:33.695 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.690+0000 7f407715a700 1 -- 172.21.15.167:0/153311785 <== osd.3 v2:172.21.15.167:6808/30036 13 ==== osd_op_reply(57 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f4060022020 con 0x564f0b9ae780 2023-10-05T19:29:33.696 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.690+0000 7f407715a700 1 -- 172.21.15.167:0/153311785 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:58 5.e 5:70000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x564f0b9ad120 con 0x564f0b9ae780 2023-10-05T19:29:33.696 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.694+0000 7f407715a700 1 -- 172.21.15.167:0/153311785 <== osd.3 v2:172.21.15.167:6808/30036 14 ==== osd_op_reply(58 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f4060022020 con 0x564f0b9ae780 2023-10-05T19:29:33.697 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.694+0000 7f407715a700 1 -- 172.21.15.167:0/153311785 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:59 5.1e 5:78000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x564f0b9ad120 con 0x7f4070011040 2023-10-05T19:29:33.697 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.694+0000 7f407795b700 1 -- 172.21.15.167:0/153311785 <== osd.0 v2:172.21.15.118:6800/27690 15 ==== osd_op_reply(59 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f406802f080 con 0x7f4070011040 2023-10-05T19:29:33.697 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.694+0000 7f407795b700 1 -- 172.21.15.167:0/153311785 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:60 5.1 5:80000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x564f0b9ad120 con 0x564f0b9ae780 2023-10-05T19:29:33.698 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.694+0000 7f407715a700 1 -- 172.21.15.167:0/153311785 <== osd.3 v2:172.21.15.167:6808/30036 15 ==== osd_op_reply(60 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f4060022020 con 0x564f0b9ae780 2023-10-05T19:29:33.698 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.694+0000 7f407715a700 1 -- 172.21.15.167:0/153311785 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:61 5.11 5:88000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x564f0b9ad120 con 0x564f0b9ae780 2023-10-05T19:29:33.699 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.694+0000 7f407715a700 1 -- 172.21.15.167:0/153311785 <== osd.3 v2:172.21.15.167:6808/30036 16 ==== osd_op_reply(61 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f4060022020 con 0x564f0b9ae780 2023-10-05T19:29:33.699 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.694+0000 7f407715a700 1 -- 172.21.15.167:0/153311785 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:62 5.9 5:90000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x564f0b9ad120 con 0x564f0b9a8950 2023-10-05T19:29:33.700 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.694+0000 7f407795b700 1 -- 172.21.15.167:0/153311785 <== osd.1 v2:172.21.15.167:6800/30035 22 ==== osd_op_reply(62 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f406802f080 con 0x564f0b9a8950 2023-10-05T19:29:33.700 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.694+0000 7f407795b700 1 -- 172.21.15.167:0/153311785 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:63 5.19 5:98000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x564f0b9ad120 con 0x564f0b9a8950 2023-10-05T19:29:33.701 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.698+0000 7f407795b700 1 -- 172.21.15.167:0/153311785 <== osd.1 v2:172.21.15.167:6800/30035 23 ==== osd_op_reply(63 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f406802f080 con 0x564f0b9a8950 2023-10-05T19:29:33.701 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.698+0000 7f407795b700 1 -- 172.21.15.167:0/153311785 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:64 5.5 5:a0000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x564f0b9ad120 con 0x7f4070011040 2023-10-05T19:29:33.701 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.698+0000 7f407795b700 1 -- 172.21.15.167:0/153311785 <== osd.0 v2:172.21.15.118:6800/27690 16 ==== osd_op_reply(64 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f406802f080 con 0x7f4070011040 2023-10-05T19:29:33.702 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.698+0000 7f407795b700 1 -- 172.21.15.167:0/153311785 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:65 5.15 5:a8000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x564f0b9ad120 con 0x564f0b9ae780 2023-10-05T19:29:33.702 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.698+0000 7f407715a700 1 -- 172.21.15.167:0/153311785 <== osd.3 v2:172.21.15.167:6808/30036 17 ==== osd_op_reply(65 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f4060022020 con 0x564f0b9ae780 2023-10-05T19:29:33.702 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.698+0000 7f407715a700 1 -- 172.21.15.167:0/153311785 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:66 5.d 5:b0000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x564f0b9ad120 con 0x7f4060038070 2023-10-05T19:29:33.703 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.698+0000 7f4076959700 1 -- 172.21.15.167:0/153311785 <== osd.2 v2:172.21.15.167:6816/30037 10 ==== osd_op_reply(66 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f4070004520 con 0x7f4060038070 2023-10-05T19:29:33.703 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.698+0000 7f4076959700 1 -- 172.21.15.167:0/153311785 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:67 5.1d 5:b8000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x564f0b9ad120 con 0x564f0b9a8950 2023-10-05T19:29:33.704 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.698+0000 7f407795b700 1 -- 172.21.15.167:0/153311785 <== osd.1 v2:172.21.15.167:6800/30035 24 ==== osd_op_reply(67 [pgnls start_epoch 32 out=107b] v30'2 uv2 ondisk = 1) v8 ==== 144+0+107 (crc 0 0 0) 0x7f406802f080 con 0x564f0b9a8950 2023-10-05T19:29:33.704 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.698+0000 7f407795b700 1 -- 172.21.15.167:0/153311785 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:68 5.3 5:c0000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x564f0b9ad120 con 0x7f4070011040 2023-10-05T19:29:33.705 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.702+0000 7f407795b700 1 -- 172.21.15.167:0/153311785 <== osd.0 v2:172.21.15.118:6800/27690 17 ==== osd_op_reply(68 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f406802f080 con 0x7f4070011040 2023-10-05T19:29:33.705 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.702+0000 7f407795b700 1 -- 172.21.15.167:0/153311785 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:69 5.13 5:c8000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x564f0b9ad120 con 0x564f0b9ae780 2023-10-05T19:29:33.706 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.702+0000 7f407715a700 1 -- 172.21.15.167:0/153311785 <== osd.3 v2:172.21.15.167:6808/30036 18 ==== osd_op_reply(69 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f4060022020 con 0x564f0b9ae780 2023-10-05T19:29:33.706 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.702+0000 7f407715a700 1 -- 172.21.15.167:0/153311785 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:70 5.b 5:d0000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x564f0b9ad120 con 0x7f4070011040 2023-10-05T19:29:33.707 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.702+0000 7f407795b700 1 -- 172.21.15.167:0/153311785 <== osd.0 v2:172.21.15.118:6800/27690 18 ==== osd_op_reply(70 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f406802f080 con 0x7f4070011040 2023-10-05T19:29:33.707 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.702+0000 7f407795b700 1 -- 172.21.15.167:0/153311785 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:71 5.1b 5:d8000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x564f0b9ad120 con 0x7f4060038070 2023-10-05T19:29:33.708 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.702+0000 7f4076959700 1 -- 172.21.15.167:0/153311785 <== osd.2 v2:172.21.15.167:6816/30037 11 ==== osd_op_reply(71 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f4070004520 con 0x7f4060038070 2023-10-05T19:29:33.708 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.702+0000 7f4076959700 1 -- 172.21.15.167:0/153311785 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:72 5.7 5:e0000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x564f0b9ad120 con 0x564f0b9ae780 2023-10-05T19:29:33.708 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.706+0000 7f407715a700 1 -- 172.21.15.167:0/153311785 <== osd.3 v2:172.21.15.167:6808/30036 19 ==== osd_op_reply(72 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f4060022020 con 0x564f0b9ae780 2023-10-05T19:29:33.709 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.706+0000 7f407715a700 1 -- 172.21.15.167:0/153311785 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:73 5.17 5:e8000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x564f0b9ad120 con 0x7f4070011040 2023-10-05T19:29:33.710 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.706+0000 7f407795b700 1 -- 172.21.15.167:0/153311785 <== osd.0 v2:172.21.15.118:6800/27690 19 ==== osd_op_reply(73 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f406802f080 con 0x7f4070011040 2023-10-05T19:29:33.710 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.706+0000 7f407795b700 1 -- 172.21.15.167:0/153311785 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:74 5.f 5:f0000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x564f0b9ad120 con 0x564f0b9a8950 2023-10-05T19:29:33.711 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.706+0000 7f407795b700 1 -- 172.21.15.167:0/153311785 <== osd.1 v2:172.21.15.167:6800/30035 25 ==== osd_op_reply(74 [pgnls start_epoch 32 out=115b] v23'2 uv2 ondisk = 1) v8 ==== 144+0+115 (crc 0 0 0) 0x7f406802f080 con 0x564f0b9a8950 2023-10-05T19:29:33.711 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.706+0000 7f407795b700 1 -- 172.21.15.167:0/153311785 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:75 5.1f 5:f8000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x564f0b9ad120 con 0x7f4060038070 2023-10-05T19:29:33.712 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.706+0000 7f4076959700 1 -- 172.21.15.167:0/153311785 <== osd.2 v2:172.21.15.167:6816/30037 12 ==== osd_op_reply(75 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f4070004520 con 0x7f4060038070 2023-10-05T19:29:33.712 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.706+0000 7f4079625e40 20 RGWRados::pool_iterate: got zonegroup_info.4f1f719d-9648-417b-b10c-e5d939825419 2023-10-05T19:29:33.712 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.706+0000 7f4079625e40 20 RGWRados::pool_iterate: got zone_info.9a9df965-6612-45d4-a45f-6e3c889bacc2 2023-10-05T19:29:33.712 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.706+0000 7f4079625e40 20 RGWRados::pool_iterate: got zone_names.default 2023-10-05T19:29:33.712 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.706+0000 7f4079625e40 20 RGWRados::pool_iterate: got zonegroups_names.default 2023-10-05T19:29:33.712 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.706+0000 7f4079625e40 20 rados->read ofs=0 len=0 2023-10-05T19:29:33.712 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.706+0000 7f4079625e40 1 -- 172.21.15.167:0/153311785 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:76 5.f 5:f43b6ece:::zone_names.default:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32) v8 -- 0x564f0b9b1d40 con 0x564f0b9a8950 2023-10-05T19:29:33.713 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.710+0000 7f407795b700 1 -- 172.21.15.167:0/153311785 <== osd.1 v2:172.21.15.167:6800/30035 26 ==== osd_op_reply(76 zone_names.default [read 0~46 out=46b] v0'0 uv1 ondisk = 0) v8 ==== 162+0+46 (crc 0 0 0) 0x7f406802f080 con 0x564f0b9a8950 2023-10-05T19:29:33.713 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.710+0000 7f4079625e40 20 rados_obj.operate() r=0 bl.length=46 2023-10-05T19:29:33.713 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.710+0000 7f4079625e40 20 rados->read ofs=0 len=0 2023-10-05T19:29:33.713 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.710+0000 7f4079625e40 1 -- 172.21.15.167:0/153311785 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:77 5.1d 5:b8374eb4:::zone_info.9a9df965-6612-45d4-a45f-6e3c889bacc2:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32) v8 -- 0x564f0b9b20f0 con 0x564f0b9a8950 2023-10-05T19:29:33.714 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.710+0000 7f407795b700 1 -- 172.21.15.167:0/153311785 <== osd.1 v2:172.21.15.167:6800/30035 27 ==== osd_op_reply(77 zone_info.9a9df965-6612-45d4-a45f-6e3c889bacc2 [read 0~874 out=874b] v0'0 uv2 ondisk = 0) v8 ==== 190+0+874 (crc 0 0 0) 0x7f406802f080 con 0x564f0b9a8950 2023-10-05T19:29:33.714 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.710+0000 7f4079625e40 20 rados_obj.operate() r=0 bl.length=874 2023-10-05T19:29:33.714 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.710+0000 7f4079625e40 20 rados->read ofs=0 len=0 2023-10-05T19:29:33.714 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.710+0000 7f4079625e40 1 -- 172.21.15.167:0/153311785 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:78 5.f 5:f4c53578:::zonegroups_names.default:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32) v8 -- 0x564f0b9b25e0 con 0x564f0b9a8950 2023-10-05T19:29:33.714 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.710+0000 7f407795b700 1 -- 172.21.15.167:0/153311785 <== osd.1 v2:172.21.15.167:6800/30035 28 ==== osd_op_reply(78 zonegroups_names.default [read 0~46 out=46b] v0'0 uv2 ondisk = 0) v8 ==== 168+0+46 (crc 0 0 0) 0x7f406802f080 con 0x564f0b9a8950 2023-10-05T19:29:33.714 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.710+0000 7f4079625e40 20 rados_obj.operate() r=0 bl.length=46 2023-10-05T19:29:33.714 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.710+0000 7f4079625e40 20 rados->read ofs=0 len=0 2023-10-05T19:29:33.714 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.710+0000 7f4079625e40 1 -- 172.21.15.167:0/153311785 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:79 5.c 5:30d299a5:::zonegroup_info.4f1f719d-9648-417b-b10c-e5d939825419:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32) v8 -- 0x564f0b9b25e0 con 0x7f4070011040 2023-10-05T19:29:33.715 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.710+0000 7f407795b700 1 -- 172.21.15.167:0/153311785 <== osd.0 v2:172.21.15.118:6800/27690 20 ==== osd_op_reply(79 zonegroup_info.4f1f719d-9648-417b-b10c-e5d939825419 [read 0~362 out=362b] v0'0 uv1 ondisk = 0) v8 ==== 195+0+362 (crc 0 0 0) 0x7f406802f080 con 0x7f4070011040 2023-10-05T19:29:33.715 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.710+0000 7f4079625e40 20 rados_obj.operate() r=0 bl.length=362 2023-10-05T19:29:33.715 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.710+0000 7f4079625e40 20 rados->read ofs=0 len=0 2023-10-05T19:29:33.715 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.710+0000 7f4079625e40 1 -- 172.21.15.167:0/153311785 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:80 5.3 5:c0ad8cbd:::region_map:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32) v8 -- 0x564f0b9b26e0 con 0x7f4070011040 2023-10-05T19:29:33.716 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.710+0000 7f407795b700 1 -- 172.21.15.167:0/153311785 <== osd.0 v2:172.21.15.118:6800/27690 21 ==== osd_op_reply(80 region_map [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 ==== 154+0+0 (crc 0 0 0) 0x7f4068050040 con 0x7f4070011040 2023-10-05T19:29:33.716 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.710+0000 7f4079625e40 20 rados_obj.operate() r=-2 bl.length=0 2023-10-05T19:29:33.716 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.710+0000 7f4079625e40 20 zone default found 2023-10-05T19:29:33.716 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.710+0000 7f4079625e40 4 Realm: () 2023-10-05T19:29:33.716 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.710+0000 7f4079625e40 4 ZoneGroup: default (4f1f719d-9648-417b-b10c-e5d939825419) 2023-10-05T19:29:33.716 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.710+0000 7f4079625e40 4 Zone: default (9a9df965-6612-45d4-a45f-6e3c889bacc2) 2023-10-05T19:29:33.716 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.710+0000 7f4079625e40 10 cannot find current period zonegroup using local zonegroup configuration 2023-10-05T19:29:33.717 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.710+0000 7f4079625e40 20 zonegroup default 2023-10-05T19:29:33.717 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.710+0000 7f4079625e40 20 rados->read ofs=0 len=0 2023-10-05T19:29:33.717 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.710+0000 7f4079625e40 1 -- 172.21.15.167:0/153311785 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:81 5.3 5:c52100b6:::period_config.default:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32) v8 -- 0x564f0b9b26e0 con 0x7f4070011040 2023-10-05T19:29:33.718 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.714+0000 7f407795b700 1 -- 172.21.15.167:0/153311785 <== osd.0 v2:172.21.15.118:6800/27690 22 ==== osd_op_reply(81 period_config.default [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 ==== 165+0+0 (crc 0 0 0) 0x7f4068050040 con 0x7f4070011040 2023-10-05T19:29:33.718 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.714+0000 7f4079625e40 20 rados_obj.operate() r=-2 bl.length=0 2023-10-05T19:29:33.718 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.714+0000 7f4079625e40 20 rados->read ofs=0 len=0 2023-10-05T19:29:33.718 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.714+0000 7f4079625e40 1 -- 172.21.15.167:0/153311785 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:82 6.2 6:4347d321:::bucket.sync-source-hints.:head [call version.read in=11b,read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32) v8 -- 0x564f0b9b4060 con 0x564f0b9ae780 2023-10-05T19:29:33.718 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.714+0000 7f407715a700 1 -- 172.21.15.167:0/153311785 <== osd.3 v2:172.21.15.167:6808/30036 20 ==== osd_op_reply(82 bucket.sync-source-hints. [call,read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 ==== 211+0+0 (crc 0 0 0) 0x7f4060022020 con 0x564f0b9ae780 2023-10-05T19:29:33.718 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.714+0000 7f4079625e40 20 rados_obj.operate() r=-2 bl.length=0 2023-10-05T19:29:33.718 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.714+0000 7f4079625e40 20 rados->read ofs=0 len=0 2023-10-05T19:29:33.718 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.714+0000 7f4079625e40 1 -- 172.21.15.167:0/153311785 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:83 6.b 6:d467b91b:::bucket.sync-target-hints.:head [call version.read in=11b,read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32) v8 -- 0x564f0b9b4410 con 0x564f0b9ae780 2023-10-05T19:29:33.719 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.714+0000 7f407715a700 1 -- 172.21.15.167:0/153311785 <== osd.3 v2:172.21.15.167:6808/30036 21 ==== osd_op_reply(83 bucket.sync-target-hints. [call,read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 ==== 211+0+0 (crc 0 0 0) 0x7f4060022020 con 0x564f0b9ae780 2023-10-05T19:29:33.719 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.714+0000 7f4079625e40 20 rados_obj.operate() r=-2 bl.length=0 2023-10-05T19:29:33.719 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.714+0000 7f4079625e40 20 started sync module instance, tier type = 2023-10-05T19:29:33.719 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.714+0000 7f4079625e40 20 started zone id=9a9df965-6612-45d4-a45f-6e3c889bacc2 (name=default) with tier type = 2023-10-05T19:29:33.719 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.714+0000 7f4079625e40 1 -- 172.21.15.167:0/153311785 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:84 7.1f 7:f95f44c2:::notify.0:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e32) v8 -- 0x564f0b9b3720 con 0x7f4060038070 2023-10-05T19:29:33.722 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.718+0000 7f4076959700 1 -- 172.21.15.167:0/153311785 <== osd.2 v2:172.21.15.167:6816/30037 13 ==== osd_op_reply(84 notify.0 [create] v32'228 uv228 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f407002f040 con 0x7f4060038070 2023-10-05T19:29:33.722 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.718+0000 7f4079625e40 1 -- 172.21.15.167:0/153311785 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:85 7.1f 7:f95f44c2:::notify.0:head [watch watch cookie 94897497125424] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e32) v8 -- 0x564f0b9b47c0 con 0x7f4060038070 2023-10-05T19:29:33.722 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.718+0000 7f4079625e40 1 -- 172.21.15.167:0/153311785 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:86 7.0 7:05bf5b68:::notify.1:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e32) v8 -- 0x564f0b9b51e0 con 0x564f0b9a8950 2023-10-05T19:29:33.725 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.722+0000 7f4076959700 1 -- 172.21.15.167:0/153311785 <== osd.2 v2:172.21.15.167:6816/30037 14 ==== osd_op_reply(85 notify.0 [watch watch cookie 94897497125424] v32'229 uv228 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f407002f040 con 0x7f4060038070 2023-10-05T19:29:33.725 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.722+0000 7f407795b700 1 -- 172.21.15.167:0/153311785 <== osd.1 v2:172.21.15.167:6800/30035 29 ==== osd_op_reply(86 notify.1 [create] v32'228 uv228 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f4068050040 con 0x564f0b9a8950 2023-10-05T19:29:33.725 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.722+0000 7f4079625e40 1 -- 172.21.15.167:0/153311785 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:87 7.0 7:05bf5b68:::notify.1:head [watch watch cookie 94897497127760] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e32) v8 -- 0x564f0b9b55e0 con 0x564f0b9a8950 2023-10-05T19:29:33.725 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.722+0000 7f4079625e40 1 -- 172.21.15.167:0/153311785 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:88 7.15 7:a93a5511:::notify.2:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e32) v8 -- 0x564f0b9b5e80 con 0x564f0b9ae780 2023-10-05T19:29:33.728 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.722+0000 7f407795b700 1 -- 172.21.15.167:0/153311785 <== osd.1 v2:172.21.15.167:6800/30035 30 ==== osd_op_reply(87 notify.1 [watch watch cookie 94897497127760] v32'229 uv228 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f4068050040 con 0x564f0b9a8950 2023-10-05T19:29:33.728 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.722+0000 7f407715a700 1 -- 172.21.15.167:0/153311785 <== osd.3 v2:172.21.15.167:6808/30036 22 ==== osd_op_reply(88 notify.2 [create] v32'228 uv228 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f4060022020 con 0x564f0b9ae780 2023-10-05T19:29:33.728 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.726+0000 7f4079625e40 1 -- 172.21.15.167:0/153311785 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:89 7.15 7:a93a5511:::notify.2:head [watch watch cookie 94897497134560] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e32) v8 -- 0x564f0b9b4dd0 con 0x564f0b9ae780 2023-10-05T19:29:33.728 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.726+0000 7f4079625e40 1 -- 172.21.15.167:0/153311785 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:90 7.e 7:7759931f:::notify.3:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e32) v8 -- 0x564f0b9b6c10 con 0x7f4060038070 2023-10-05T19:29:33.732 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.726+0000 7f407715a700 1 -- 172.21.15.167:0/153311785 <== osd.3 v2:172.21.15.167:6808/30036 23 ==== osd_op_reply(89 notify.2 [watch watch cookie 94897497134560] v32'229 uv228 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f4060022020 con 0x564f0b9ae780 2023-10-05T19:29:33.732 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.726+0000 7f4076959700 1 -- 172.21.15.167:0/153311785 <== osd.2 v2:172.21.15.167:6816/30037 15 ==== osd_op_reply(90 notify.3 [create] v32'228 uv228 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f407002f040 con 0x7f4060038070 2023-10-05T19:29:33.732 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.726+0000 7f4079625e40 1 -- 172.21.15.167:0/153311785 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:91 7.e 7:7759931f:::notify.3:head [watch watch cookie 94897497137712] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e32) v8 -- 0x564f0b9b6fc0 con 0x7f4060038070 2023-10-05T19:29:33.732 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.726+0000 7f4079625e40 1 -- 172.21.15.167:0/153311785 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:92 7.d 7:b4812045:::notify.4:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e32) v8 -- 0x564f0b9b7ae0 con 0x564f0b9a8950 2023-10-05T19:29:33.734 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.730+0000 7f407795b700 1 -- 172.21.15.167:0/153311785 <== osd.1 v2:172.21.15.167:6800/30035 31 ==== osd_op_reply(92 notify.4 [create] v32'228 uv228 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f4068050040 con 0x564f0b9a8950 2023-10-05T19:29:33.735 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.730+0000 7f4079625e40 1 -- 172.21.15.167:0/153311785 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:93 7.d 7:b4812045:::notify.4:head [watch watch cookie 94897497143328] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e32) v8 -- 0x564f0b9b83d0 con 0x564f0b9a8950 2023-10-05T19:29:33.735 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.730+0000 7f4076959700 1 -- 172.21.15.167:0/153311785 <== osd.2 v2:172.21.15.167:6816/30037 16 ==== osd_op_reply(91 notify.3 [watch watch cookie 94897497137712] v32'229 uv228 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f407002f040 con 0x7f4060038070 2023-10-05T19:29:33.735 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.730+0000 7f4079625e40 1 -- 172.21.15.167:0/153311785 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:94 7.3 7:c609908c:::notify.5:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e32) v8 -- 0x564f0b9b8de0 con 0x7f4070011040 2023-10-05T19:29:33.737 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.734+0000 7f407795b700 1 -- 172.21.15.167:0/153311785 <== osd.0 v2:172.21.15.118:6800/27690 23 ==== osd_op_reply(94 notify.5 [create] v32'228 uv228 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f4068050040 con 0x7f4070011040 2023-10-05T19:29:33.737 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.734+0000 7f4079625e40 1 -- 172.21.15.167:0/153311785 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:95 7.3 7:c609908c:::notify.5:head [watch watch cookie 94897497138816] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e32) v8 -- 0x564f0b9b8de0 con 0x7f4070011040 2023-10-05T19:29:33.737 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.734+0000 7f4079625e40 1 -- 172.21.15.167:0/153311785 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:96 7.14 7:2b04a3e9:::notify.6:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e32) v8 -- 0x564f0b9b9840 con 0x7f4070011040 2023-10-05T19:29:33.737 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.734+0000 7f407795b700 1 -- 172.21.15.167:0/153311785 <== osd.1 v2:172.21.15.167:6800/30035 32 ==== osd_op_reply(93 notify.4 [watch watch cookie 94897497143328] v32'229 uv228 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f4068050040 con 0x564f0b9a8950 2023-10-05T19:29:33.739 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.734+0000 7f407795b700 1 -- 172.21.15.167:0/153311785 <== osd.0 v2:172.21.15.118:6800/27690 24 ==== osd_op_reply(95 notify.5 [watch watch cookie 94897497138816] v32'229 uv228 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f4068050040 con 0x7f4070011040 2023-10-05T19:29:33.740 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.734+0000 7f407795b700 1 -- 172.21.15.167:0/153311785 <== osd.0 v2:172.21.15.118:6800/27690 25 ==== osd_op_reply(96 notify.6 [create] v32'228 uv228 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f4068050040 con 0x7f4070011040 2023-10-05T19:29:33.740 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.734+0000 7f4079625e40 1 -- 172.21.15.167:0/153311785 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:97 7.14 7:2b04a3e9:::notify.6:head [watch watch cookie 94897497147632] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e32) v8 -- 0x564f0b9b8d40 con 0x7f4070011040 2023-10-05T19:29:33.740 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.734+0000 7f4079625e40 1 -- 172.21.15.167:0/153311785 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:98 7.9 7:93e5b521:::notify.7:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e32) v8 -- 0x564f0b9b9d30 con 0x7f4070011040 2023-10-05T19:29:33.742 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.738+0000 7f407795b700 1 -- 172.21.15.167:0/153311785 <== osd.0 v2:172.21.15.118:6800/27690 26 ==== osd_op_reply(97 notify.6 [watch watch cookie 94897497147632] v32'229 uv228 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f4068050040 con 0x7f4070011040 2023-10-05T19:29:33.743 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.738+0000 7f407795b700 1 -- 172.21.15.167:0/153311785 <== osd.0 v2:172.21.15.118:6800/27690 27 ==== osd_op_reply(98 notify.7 [create] v32'228 uv228 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f4068050040 con 0x7f4070011040 2023-10-05T19:29:33.743 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.738+0000 7f4079625e40 1 -- 172.21.15.167:0/153311785 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:99 7.9 7:93e5b521:::notify.7:head [watch watch cookie 94897497150288] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e32) v8 -- 0x564f0b9b9c90 con 0x7f4070011040 2023-10-05T19:29:33.743 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.738+0000 7f4079625e40 20 add_watcher() i=0 2023-10-05T19:29:33.743 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.738+0000 7f4079625e40 20 add_watcher() i=1 2023-10-05T19:29:33.743 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.738+0000 7f4079625e40 20 add_watcher() i=2 2023-10-05T19:29:33.743 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.738+0000 7f4079625e40 20 add_watcher() i=3 2023-10-05T19:29:33.743 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.738+0000 7f4079625e40 20 add_watcher() i=4 2023-10-05T19:29:33.744 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.738+0000 7f4079625e40 20 add_watcher() i=5 2023-10-05T19:29:33.744 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.738+0000 7f4079625e40 20 add_watcher() i=6 2023-10-05T19:29:33.745 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.742+0000 7f407795b700 1 -- 172.21.15.167:0/153311785 <== osd.0 v2:172.21.15.118:6800/27690 28 ==== osd_op_reply(99 notify.7 [watch watch cookie 94897497150288] v32'229 uv228 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f4068050040 con 0x7f4070011040 2023-10-05T19:29:33.745 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.742+0000 7f4079625e40 20 add_watcher() i=7 2023-10-05T19:29:33.745 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.742+0000 7f4079625e40 2 all 8 watchers are set, enabling cache 2023-10-05T19:29:33.745 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.742+0000 7f4079625e40 1 -- 172.21.15.167:0/153311785 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:100 6.4 6:22d26bf9:::data_loggenerations_metadata:head [call version.check_conds in=74b,call version.read in=11b,read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32) v8 -- 0x564f0b9bce30 con 0x564f0b9a8950 2023-10-05T19:29:33.746 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.742+0000 7f407795b700 1 -- 172.21.15.167:0/153311785 <== osd.1 v2:172.21.15.167:6800/30035 33 ==== osd_op_reply(100 data_loggenerations_metadata [call,call out=48b,read 0~28 out=28b] v0'0 uv1 ondisk = 0) v8 ==== 256+0+76 (crc 0 0 0) 0x7f4068050040 con 0x564f0b9a8950 2023-10-05T19:29:33.746 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.742+0000 7f4079625e40 1 -- 172.21.15.167:0/153311785 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:101 6.4 6:22d26bf9:::data_loggenerations_metadata:head [watch watch cookie 94897497151760] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e32) v8 -- 0x564f0b9ba360 con 0x564f0b9a8950 2023-10-05T19:29:33.749 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.746+0000 7f407795b700 1 -- 172.21.15.167:0/153311785 <== osd.1 v2:172.21.15.167:6800/30035 34 ==== osd_op_reply(101 data_loggenerations_metadata [watch watch cookie 94897497151760] v32'225 uv1 ondisk = 0) v8 ==== 172+0+0 (crc 0 0 0) 0x7f4068050040 con 0x564f0b9a8950 2023-10-05T19:29:33.749 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.746+0000 7f4048fe1700 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2023-10-05T19:29:33.749 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.746+0000 7f4079625e40 20 check_secure_mon_conn(): auth registy supported: methods=[2] modes=[2,1] 2023-10-05T19:29:33.749 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.746+0000 7f4079625e40 20 check_secure_mon_conn(): mode 1 is insecure 2023-10-05T19:29:33.749 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.746+0000 7f4079625e40 1 Processor -- start 2023-10-05T19:29:33.750 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.746+0000 7f4079625e40 1 -- start start 2023-10-05T19:29:33.750 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.746+0000 7f4079625e40 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x564f0b9c7200 0x564f0b9c75d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-05T19:29:33.750 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.746+0000 7f4079625e40 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x564f0b9c7b10 con 0x564f0b9c7200 2023-10-05T19:29:33.751 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.746+0000 7f4076959700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x564f0b9c7200 0x564f0b9c75d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-05T19:29:33.751 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.746+0000 7f4076959700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x564f0b9c7200 0x564f0b9c75d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.118:3300/0 says I am v2:172.21.15.167:46244/0 (socket says 172.21.15.167:46244) 2023-10-05T19:29:33.751 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.746+0000 7f4076959700 1 -- 172.21.15.167:0/408043416 learned_addr learned my addr 172.21.15.167:0/408043416 (peer_addr_for_me v2:172.21.15.167:0/0) 2023-10-05T19:29:33.751 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.746+0000 7f4076959700 1 -- 172.21.15.167:0/408043416 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x564f0b9c7c50 con 0x564f0b9c7200 2023-10-05T19:29:33.751 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.746+0000 7f4076959700 1 --2- 172.21.15.167:0/408043416 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x564f0b9c7200 0x564f0b9c75d0 secure :-1 s=READY pgs=594 cs=0 l=1 rev1=1 crypto rx=0x564f0b9ac270 tx=0x7f4070006000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d11dae9ea2f2fc4e server_cookie=0 in_seq=0 out_seq=0 2023-10-05T19:29:33.752 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.746+0000 7f3fc3fff700 1 -- 172.21.15.167:0/408043416 <== mon.0 v2:172.21.15.118:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f407002c040 con 0x564f0b9c7200 2023-10-05T19:29:33.752 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.746+0000 7f3fc3fff700 1 -- 172.21.15.167:0/408043416 <== mon.0 v2:172.21.15.118:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f407002d040 con 0x564f0b9c7200 2023-10-05T19:29:33.752 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.746+0000 7f3fc3fff700 1 -- 172.21.15.167:0/408043416 <== mon.0 v2:172.21.15.118:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f40700289c0 con 0x564f0b9c7200 2023-10-05T19:29:33.753 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.750+0000 7f4079625e40 1 -- 172.21.15.167:0/408043416 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x564f0b9c7200 msgr2=0x564f0b9c75d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-05T19:29:33.753 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.750+0000 7f4079625e40 1 --2- 172.21.15.167:0/408043416 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x564f0b9c7200 0x564f0b9c75d0 secure :-1 s=READY pgs=594 cs=0 l=1 rev1=1 crypto rx=0x564f0b9ac270 tx=0x7f4070006000 comp rx=0 tx=0).stop 2023-10-05T19:29:33.753 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.750+0000 7f4079625e40 1 -- 172.21.15.167:0/408043416 shutdown_connections 2023-10-05T19:29:33.754 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.750+0000 7f4079625e40 1 --2- 172.21.15.167:0/408043416 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x564f0b9c7200 0x564f0b9c75d0 unknown :-1 s=CLOSED pgs=594 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-05T19:29:33.754 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.750+0000 7f4079625e40 1 -- 172.21.15.167:0/408043416 >> 172.21.15.167:0/408043416 conn(0x564f0b9c25c0 msgr2=0x564f0b9c0770 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-05T19:29:33.754 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.750+0000 7f4079625e40 1 -- 172.21.15.167:0/408043416 shutdown_connections 2023-10-05T19:29:33.754 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.750+0000 7f4079625e40 1 -- 172.21.15.167:0/408043416 wait complete. 2023-10-05T19:29:33.755 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.750+0000 7f4079625e40 1 Processor -- start 2023-10-05T19:29:33.755 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.750+0000 7f4079625e40 1 -- start start 2023-10-05T19:29:33.755 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.750+0000 7f4079625e40 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x564f0b9c7200 0x564f0b9ccd60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-05T19:29:33.755 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.750+0000 7f4079625e40 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x564f0b9cdd60 con 0x564f0b9c7200 2023-10-05T19:29:33.755 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.750+0000 7f4076959700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x564f0b9c7200 0x564f0b9ccd60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-05T19:29:33.755 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.750+0000 7f4076959700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x564f0b9c7200 0x564f0b9ccd60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.118:3300/0 says I am v2:172.21.15.167:46248/0 (socket says 172.21.15.167:46248) 2023-10-05T19:29:33.755 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.750+0000 7f4076959700 1 -- 172.21.15.167:0/558751109 learned_addr learned my addr 172.21.15.167:0/558751109 (peer_addr_for_me v2:172.21.15.167:0/0) 2023-10-05T19:29:33.756 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.750+0000 7f4076959700 1 -- 172.21.15.167:0/558751109 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x564f0b8c3bd0 con 0x564f0b9c7200 2023-10-05T19:29:33.756 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.750+0000 7f4076959700 1 --2- 172.21.15.167:0/558751109 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x564f0b9c7200 0x564f0b9ccd60 secure :-1 s=READY pgs=595 cs=0 l=1 rev1=1 crypto rx=0x7f4070006040 tx=0x7f407003c000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-05T19:29:33.756 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.750+0000 7f3fc27fc700 1 -- 172.21.15.167:0/558751109 <== mon.0 v2:172.21.15.118:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f407002c030 con 0x564f0b9c7200 2023-10-05T19:29:33.756 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.750+0000 7f3fc27fc700 1 -- 172.21.15.167:0/558751109 <== mon.0 v2:172.21.15.118:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f407001c040 con 0x564f0b9c7200 2023-10-05T19:29:33.757 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.750+0000 7f4079625e40 1 -- 172.21.15.167:0/558751109 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x564f0b9cd360 con 0x564f0b9c7200 2023-10-05T19:29:33.757 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.750+0000 7f3fc27fc700 1 -- 172.21.15.167:0/558751109 <== mon.0 v2:172.21.15.118:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f4070028ee0 con 0x564f0b9c7200 2023-10-05T19:29:33.757 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.750+0000 7f4079625e40 1 -- 172.21.15.167:0/558751109 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x564f0b9cd7a0 con 0x564f0b9c7200 2023-10-05T19:29:33.757 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.754+0000 7f3fc27fc700 1 -- 172.21.15.167:0/558751109 <== mon.0 v2:172.21.15.118:3300/0 4 ==== mgrmap(e 5) v1 ==== 40329+0+0 (secure 0 0 0) 0x7f4070045070 con 0x564f0b9c7200 2023-10-05T19:29:33.758 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.754+0000 7f3fc27fc700 1 --2- 172.21.15.167:0/558751109 >> [v2:172.21.15.167:6824/29889,v1:172.21.15.167:6825/29889] conn(0x7f3fb8030b70 0x7f3fb8033020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-05T19:29:33.758 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.754+0000 7f3fc27fc700 1 -- 172.21.15.167:0/558751109 <== mon.0 v2:172.21.15.118:3300/0 5 ==== osd_map(32..32 src has 1..32) v4 ==== 6260+0+0 (secure 0 0 0) 0x7f4070049d00 con 0x564f0b9c7200 2023-10-05T19:29:33.758 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.754+0000 7f407795b700 1 --2- 172.21.15.167:0/558751109 >> [v2:172.21.15.167:6824/29889,v1:172.21.15.167:6825/29889] conn(0x7f3fb8030b70 0x7f3fb8033020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-05T19:29:33.758 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.754+0000 7f4079625e40 5 note: GC not initialized 2023-10-05T19:29:33.759 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.754+0000 7f407795b700 1 --2- 172.21.15.167:0/558751109 >> [v2:172.21.15.167:6824/29889,v1:172.21.15.167:6825/29889] conn(0x7f3fb8030b70 0x7f3fb8033020 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x564f0b9ba2c0 tx=0x7f4068000f90 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-05T19:29:33.760 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.758+0000 7f3fb77fe700 20 reqs_thread_entry: start 2023-10-05T19:29:33.761 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.758+0000 7f4079625e40 20 init_complete bucket index max shards: 11 2023-10-05T19:29:33.761 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.758+0000 7f3fb57fa700 20 reqs_thread_entry: start 2023-10-05T19:29:33.762 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.758+0000 7f4079625e40 10 rgw notify: Started notification manager with: 1 workers 2023-10-05T19:29:33.762 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.758+0000 7f3fa37fe700 20 rgw notify: INFO: next queues processing will happen at: Thu Oct 5 19:30:03 2023 2023-10-05T19:29:33.762 INFO:teuthology.orchestra.run.smithi167.stderr: 2023-10-05T19:29:33.762 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.758+0000 7f4079625e40 20 get_system_obj_state: rctx=0x7ffd49957318 obj=default.rgw.meta:users.uid:foo state=0x564f0b8500b0 s->prefetch_data=0 2023-10-05T19:29:33.762 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.758+0000 7f4079625e40 1 -- 172.21.15.167:0/153311785 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:102 8.18 8:1e6ca5ed:users.uid::foo:head [call version.read in=11b,getxattrs,stat] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32) v8 -- 0x564f0ba35550 con 0x564f0b9a8950 2023-10-05T19:29:33.762 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.758+0000 7f3fa2ffd700 20 reqs_thread_entry: start 2023-10-05T19:29:33.763 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.758+0000 7f407795b700 1 -- 172.21.15.167:0/153311785 <== osd.1 v2:172.21.15.167:6800/30035 35 ==== osd_op_reply(102 foo [call out=48b,getxattrs out=75b,stat out=16b] v0'0 uv15 ondisk = 0) v8 ==== 231+0+139 (crc 0 0 0) 0x7f4068050040 con 0x564f0b9a8950 2023-10-05T19:29:33.763 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.758+0000 7f4079625e40 20 get_system_obj_state: s->obj_tag was set empty 2023-10-05T19:29:33.763 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.758+0000 7f4079625e40 20 Read xattr: user.rgw.idtag 2023-10-05T19:29:33.763 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.758+0000 7f4079625e40 20 rados->read ofs=0 len=0 2023-10-05T19:29:33.763 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.758+0000 7f4079625e40 1 -- 172.21.15.167:0/153311785 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:103 8.18 8:1e6ca5ed:users.uid::foo:head [call version.check_conds in=122b,call version.read in=11b,read 0~0,getxattrs] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32) v8 -- 0x564f0ba35550 con 0x564f0b9a8950 2023-10-05T19:29:33.764 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.758+0000 7f407795b700 1 -- 172.21.15.167:0/153311785 <== osd.1 v2:172.21.15.167:6800/30035 36 ==== osd_op_reply(103 foo [call,call out=48b,read 0~357 out=357b,getxattrs out=75b] v0'0 uv15 ondisk = 0) v8 ==== 273+0+480 (crc 0 0 0) 0x7f4068050040 con 0x564f0b9a8950 2023-10-05T19:29:33.764 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.758+0000 7f4079625e40 20 rados_obj.operate() r=0 bl.length=357 2023-10-05T19:29:33.764 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.758+0000 7f4079625e40 1 -- 172.21.15.167:0/153311785 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:104 6.13 6:cea599e4:usage::usage.10:head [call rgw.user_usage_log_read in=63b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32) v8 -- 0x564f0ba35550 con 0x7f4070011040 2023-10-05T19:29:33.765 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.762+0000 7f407795b700 1 -- 172.21.15.167:0/153311785 <== osd.0 v2:172.21.15.118:6800/27690 29 ==== osd_op_reply(104 usage.10 [call out=1268b] v0'0 uv30 ondisk = 0) v8 ==== 152+0+1268 (crc 0 0 0) 0x7f4068050040 con 0x7f4070011040 2023-10-05T19:29:33.765 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.762+0000 7f4079625e40 1 -- 172.21.15.167:0/153311785 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:105 7.1f 7:f95f44c2:::notify.0:head [watch unwatch cookie 94897497125424] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e32) v8 -- 0x564f0b9f3940 con 0x7f4060038070 2023-10-05T19:29:33.768 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.762+0000 7f4076959700 1 -- 172.21.15.167:0/153311785 <== osd.2 v2:172.21.15.167:6816/30037 17 ==== osd_op_reply(105 notify.0 [watch unwatch cookie 94897497125424] v32'230 uv228 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f407002f040 con 0x7f4060038070 2023-10-05T19:29:33.768 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.762+0000 7f4079625e40 20 remove_watcher() i=0 2023-10-05T19:29:33.768 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.762+0000 7f4079625e40 2 removed watcher, disabling cache 2023-10-05T19:29:33.768 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.762+0000 7f4079625e40 1 -- 172.21.15.167:0/153311785 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:106 7.0 7:05bf5b68:::notify.1:head [watch unwatch cookie 94897497127760] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e32) v8 -- 0x564f0b9b3230 con 0x564f0b9a8950 2023-10-05T19:29:33.771 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.766+0000 7f407795b700 1 -- 172.21.15.167:0/153311785 <== osd.1 v2:172.21.15.167:6800/30035 37 ==== osd_op_reply(106 notify.1 [watch unwatch cookie 94897497127760] v32'230 uv228 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f4068050040 con 0x564f0b9a8950 2023-10-05T19:29:33.771 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.766+0000 7f4079625e40 20 remove_watcher() i=1 2023-10-05T19:29:33.771 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.766+0000 7f4079625e40 1 -- 172.21.15.167:0/153311785 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:107 7.15 7:a93a5511:::notify.2:head [watch unwatch cookie 94897497134560] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e32) v8 -- 0x564f0b9b3230 con 0x564f0b9ae780 2023-10-05T19:29:33.774 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.770+0000 7f407715a700 1 -- 172.21.15.167:0/153311785 <== osd.3 v2:172.21.15.167:6808/30036 24 ==== osd_op_reply(107 notify.2 [watch unwatch cookie 94897497134560] v32'230 uv228 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f4060022020 con 0x564f0b9ae780 2023-10-05T19:29:33.774 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.770+0000 7f4079625e40 20 remove_watcher() i=2 2023-10-05T19:29:33.774 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.770+0000 7f4079625e40 1 -- 172.21.15.167:0/153311785 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:108 7.e 7:7759931f:::notify.3:head [watch unwatch cookie 94897497137712] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e32) v8 -- 0x564f0b9b55e0 con 0x7f4060038070 2023-10-05T19:29:33.776 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.774+0000 7f4076959700 1 -- 172.21.15.167:0/153311785 <== osd.2 v2:172.21.15.167:6816/30037 18 ==== osd_op_reply(108 notify.3 [watch unwatch cookie 94897497137712] v32'230 uv228 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f407002f040 con 0x7f4060038070 2023-10-05T19:29:33.777 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.774+0000 7f4079625e40 20 remove_watcher() i=3 2023-10-05T19:29:33.777 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.774+0000 7f4079625e40 1 -- 172.21.15.167:0/153311785 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:109 7.d 7:b4812045:::notify.4:head [watch unwatch cookie 94897497143328] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e32) v8 -- 0x564f0b9b6230 con 0x564f0b9a8950 2023-10-05T19:29:33.779 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.774+0000 7f407795b700 1 -- 172.21.15.167:0/153311785 <== osd.1 v2:172.21.15.167:6800/30035 38 ==== osd_op_reply(109 notify.4 [watch unwatch cookie 94897497143328] v32'230 uv228 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f4068050040 con 0x564f0b9a8950 2023-10-05T19:29:33.779 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.774+0000 7f4079625e40 20 remove_watcher() i=4 2023-10-05T19:29:33.779 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.774+0000 7f4079625e40 1 -- 172.21.15.167:0/153311785 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:110 7.3 7:c609908c:::notify.5:head [watch unwatch cookie 94897497138816] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e32) v8 -- 0x564f0b9b7820 con 0x7f4070011040 2023-10-05T19:29:33.782 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.778+0000 7f407795b700 1 -- 172.21.15.167:0/153311785 <== osd.0 v2:172.21.15.118:6800/27690 30 ==== osd_op_reply(110 notify.5 [watch unwatch cookie 94897497138816] v32'230 uv228 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f4068050040 con 0x7f4070011040 2023-10-05T19:29:33.782 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.778+0000 7f4079625e40 20 remove_watcher() i=5 2023-10-05T19:29:33.783 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.778+0000 7f4079625e40 1 -- 172.21.15.167:0/153311785 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:111 7.14 7:2b04a3e9:::notify.6:head [watch unwatch cookie 94897497147632] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e32) v8 -- 0x564f0b9b3b50 con 0x7f4070011040 2023-10-05T19:29:33.785 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.782+0000 7f407795b700 1 -- 172.21.15.167:0/153311785 <== osd.0 v2:172.21.15.118:6800/27690 31 ==== osd_op_reply(111 notify.6 [watch unwatch cookie 94897497147632] v32'230 uv228 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f4068050040 con 0x7f4070011040 2023-10-05T19:29:33.785 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.782+0000 7f4079625e40 20 remove_watcher() i=6 2023-10-05T19:29:33.786 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.782+0000 7f4079625e40 1 -- 172.21.15.167:0/153311785 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:112 7.9 7:93e5b521:::notify.7:head [watch unwatch cookie 94897497150288] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e32) v8 -- 0x564f0b9b3b50 con 0x7f4070011040 2023-10-05T19:29:33.788 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.782+0000 7f407795b700 1 -- 172.21.15.167:0/153311785 <== osd.0 v2:172.21.15.118:6800/27690 32 ==== osd_op_reply(112 notify.7 [watch unwatch cookie 94897497150288] v32'230 uv228 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f4068050040 con 0x7f4070011040 2023-10-05T19:29:33.788 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.782+0000 7f4079625e40 20 remove_watcher() i=7 2023-10-05T19:29:33.790 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.786+0000 7f4048fe1700 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2023-10-05T19:29:33.790 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.786+0000 7f4079625e40 1 -- 172.21.15.167:0/153311785 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:113 6.4 6:22d26bf9:::data_loggenerations_metadata:head [watch unwatch cookie 94897497151760] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e32) v8 -- 0x564f0b9b3b50 con 0x564f0b9a8950 2023-10-05T19:29:33.795 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.790+0000 7f407795b700 1 -- 172.21.15.167:0/153311785 <== osd.1 v2:172.21.15.167:6800/30035 39 ==== osd_op_reply(113 data_loggenerations_metadata [watch unwatch cookie 94897497151760] v32'226 uv1 ondisk = 0) v8 ==== 172+0+0 (crc 0 0 0) 0x7f4068050040 con 0x564f0b9a8950 2023-10-05T19:29:33.795 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.790+0000 7f4079625e40 1 -- 172.21.15.167:0/153311785 >> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] conn(0x7f4070011040 msgr2=0x7f4070014950 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-05T19:29:33.795 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.790+0000 7f4079625e40 1 --2- 172.21.15.167:0/153311785 >> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] conn(0x7f4070011040 0x7f4070014950 crc :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-05T19:29:33.795 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.790+0000 7f4079625e40 1 -- 172.21.15.167:0/153311785 >> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] conn(0x564f0b9a8950 msgr2=0x564f0b9aad30 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-05T19:29:33.795 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.790+0000 7f4079625e40 1 --2- 172.21.15.167:0/153311785 >> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] conn(0x564f0b9a8950 0x564f0b9aad30 crc :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-05T19:29:33.795 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.790+0000 7f4079625e40 1 -- 172.21.15.167:0/153311785 >> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] conn(0x7f4060038070 msgr2=0x7f40600113b0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-05T19:29:33.795 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.790+0000 7f4079625e40 1 --2- 172.21.15.167:0/153311785 >> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] conn(0x7f4060038070 0x7f40600113b0 crc :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-05T19:29:33.795 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.790+0000 7f4079625e40 1 -- 172.21.15.167:0/153311785 >> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] conn(0x564f0b9ae780 msgr2=0x564f0b9b0b60 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-05T19:29:33.795 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.790+0000 7f4079625e40 1 --2- 172.21.15.167:0/153311785 >> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] conn(0x564f0b9ae780 0x564f0b9b0b60 crc :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-05T19:29:33.796 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.790+0000 7f4079625e40 1 -- 172.21.15.167:0/153311785 >> [v2:172.21.15.167:6824/29889,v1:172.21.15.167:6825/29889] conn(0x7f4058030c50 msgr2=0x7f4058033100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-05T19:29:33.796 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.790+0000 7f4079625e40 1 --2- 172.21.15.167:0/153311785 >> [v2:172.21.15.167:6824/29889,v1:172.21.15.167:6825/29889] conn(0x7f4058030c50 0x7f4058033100 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7f40700044f0 tx=0x7f40700067c0 comp rx=0 tx=0).stop 2023-10-05T19:29:33.796 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.790+0000 7f407795b700 1 -- 172.21.15.167:0/153311785 reap_dead start 2023-10-05T19:29:33.796 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.790+0000 7f4079625e40 1 -- 172.21.15.167:0/153311785 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x564f0b979940 msgr2=0x564f0b99eed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-05T19:29:33.796 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.790+0000 7f4079625e40 1 --2- 172.21.15.167:0/153311785 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x564f0b979940 0x564f0b99eed0 secure :-1 s=READY pgs=593 cs=0 l=1 rev1=1 crypto rx=0x7f406000c010 tx=0x7f4060003050 comp rx=0 tx=0).stop 2023-10-05T19:29:33.796 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.790+0000 7f4079625e40 1 -- 172.21.15.167:0/153311785 shutdown_connections 2023-10-05T19:29:33.796 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.790+0000 7f4079625e40 1 --2- 172.21.15.167:0/153311785 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x564f0b979940 0x564f0b99eed0 unknown :-1 s=CLOSED pgs=593 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-05T19:29:33.796 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.790+0000 7f4079625e40 1 -- 172.21.15.167:0/153311785 >> 172.21.15.167:0/153311785 conn(0x564f0b8ec880 msgr2=0x564f0b992740 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-05T19:29:33.796 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.790+0000 7f4079625e40 1 -- 172.21.15.167:0/153311785 shutdown_connections 2023-10-05T19:29:33.796 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.790+0000 7f4079625e40 1 -- 172.21.15.167:0/153311785 wait complete. 2023-10-05T19:29:33.796 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.790+0000 7f4079625e40 1 -- 172.21.15.167:0/558751109 >> [v2:172.21.15.167:6824/29889,v1:172.21.15.167:6825/29889] conn(0x7f3fb8030b70 msgr2=0x7f3fb8033020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-05T19:29:33.797 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.790+0000 7f4079625e40 1 --2- 172.21.15.167:0/558751109 >> [v2:172.21.15.167:6824/29889,v1:172.21.15.167:6825/29889] conn(0x7f3fb8030b70 0x7f3fb8033020 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x564f0b9ba2c0 tx=0x7f4068000f90 comp rx=0 tx=0).stop 2023-10-05T19:29:33.797 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.790+0000 7f4079625e40 1 -- 172.21.15.167:0/558751109 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x564f0b9c7200 msgr2=0x564f0b9ccd60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-05T19:29:33.797 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.790+0000 7f4079625e40 1 --2- 172.21.15.167:0/558751109 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x564f0b9c7200 0x564f0b9ccd60 secure :-1 s=READY pgs=595 cs=0 l=1 rev1=1 crypto rx=0x7f4070006040 tx=0x7f407003c000 comp rx=0 tx=0).stop 2023-10-05T19:29:33.797 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.790+0000 7f4079625e40 1 -- 172.21.15.167:0/558751109 shutdown_connections 2023-10-05T19:29:33.797 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.790+0000 7f4079625e40 1 --2- 172.21.15.167:0/558751109 >> [v2:172.21.15.167:6824/29889,v1:172.21.15.167:6825/29889] conn(0x7f3fb8030b70 0x7f3fb8033020 unknown :-1 s=CLOSED pgs=205 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-05T19:29:33.797 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.790+0000 7f4079625e40 1 --2- 172.21.15.167:0/558751109 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x564f0b9c7200 0x564f0b9ccd60 unknown :-1 s=CLOSED pgs=595 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-05T19:29:33.797 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.790+0000 7f4079625e40 1 -- 172.21.15.167:0/558751109 >> 172.21.15.167:0/558751109 conn(0x564f0b9c25c0 msgr2=0x564f0b9c06c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-05T19:29:33.797 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.790+0000 7f4079625e40 1 -- 172.21.15.167:0/558751109 shutdown_connections 2023-10-05T19:29:33.797 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.790+0000 7f4079625e40 1 -- 172.21.15.167:0/558751109 wait complete. 2023-10-05T19:29:33.798 INFO:teuthology.orchestra.run.smithi167.stdout:{ 2023-10-05T19:29:33.798 INFO:teuthology.orchestra.run.smithi167.stdout: "entries": [ 2023-10-05T19:29:33.798 INFO:teuthology.orchestra.run.smithi167.stdout: { 2023-10-05T19:29:33.798 INFO:teuthology.orchestra.run.smithi167.stdout: "user": "foo", 2023-10-05T19:29:33.798 INFO:teuthology.orchestra.run.smithi167.stdout: "buckets": [ 2023-10-05T19:29:33.798 INFO:teuthology.orchestra.run.smithi167.stdout: { 2023-10-05T19:29:33.798 INFO:teuthology.orchestra.run.smithi167.stdout: "bucket": "", 2023-10-05T19:29:33.798 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2023-10-05T19:00:00.000000Z", 2023-10-05T19:29:33.798 INFO:teuthology.orchestra.run.smithi167.stdout: "epoch": 1696532400, 2023-10-05T19:29:33.798 INFO:teuthology.orchestra.run.smithi167.stdout: "owner": "foo", 2023-10-05T19:29:33.798 INFO:teuthology.orchestra.run.smithi167.stdout: "categories": [] 2023-10-05T19:29:33.799 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2023-10-05T19:29:33.799 INFO:teuthology.orchestra.run.smithi167.stdout: { 2023-10-05T19:29:33.799 INFO:teuthology.orchestra.run.smithi167.stdout: "bucket": "myfoo", 2023-10-05T19:29:33.799 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2023-10-05T19:00:00.000000Z", 2023-10-05T19:29:33.799 INFO:teuthology.orchestra.run.smithi167.stdout: "epoch": 1696532400, 2023-10-05T19:29:33.799 INFO:teuthology.orchestra.run.smithi167.stdout: "owner": "foo", 2023-10-05T19:29:33.799 INFO:teuthology.orchestra.run.smithi167.stdout: "categories": [ 2023-10-05T19:29:33.799 INFO:teuthology.orchestra.run.smithi167.stdout: { 2023-10-05T19:29:33.799 INFO:teuthology.orchestra.run.smithi167.stdout: "category": "create_bucket", 2023-10-05T19:29:33.799 INFO:teuthology.orchestra.run.smithi167.stdout: "bytes_sent": 0, 2023-10-05T19:29:33.799 INFO:teuthology.orchestra.run.smithi167.stdout: "bytes_received": 0, 2023-10-05T19:29:33.799 INFO:teuthology.orchestra.run.smithi167.stdout: "ops": 5, 2023-10-05T19:29:33.800 INFO:teuthology.orchestra.run.smithi167.stdout: "successful_ops": 5 2023-10-05T19:29:33.800 INFO:teuthology.orchestra.run.smithi167.stdout: } 2023-10-05T19:29:33.800 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2023-10-05T19:29:33.800 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2023-10-05T19:29:33.800 INFO:teuthology.orchestra.run.smithi167.stdout: { 2023-10-05T19:29:33.800 INFO:teuthology.orchestra.run.smithi167.stdout: "bucket": "myfoo2", 2023-10-05T19:29:33.800 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2023-10-05T19:00:00.000000Z", 2023-10-05T19:29:33.800 INFO:teuthology.orchestra.run.smithi167.stdout: "epoch": 1696532400, 2023-10-05T19:29:33.800 INFO:teuthology.orchestra.run.smithi167.stdout: "owner": "foo", 2023-10-05T19:29:33.800 INFO:teuthology.orchestra.run.smithi167.stdout: "categories": [ 2023-10-05T19:29:33.800 INFO:teuthology.orchestra.run.smithi167.stdout: { 2023-10-05T19:29:33.801 INFO:teuthology.orchestra.run.smithi167.stdout: "category": "create_bucket", 2023-10-05T19:29:33.801 INFO:teuthology.orchestra.run.smithi167.stdout: "bytes_sent": 0, 2023-10-05T19:29:33.801 INFO:teuthology.orchestra.run.smithi167.stdout: "bytes_received": 0, 2023-10-05T19:29:33.801 INFO:teuthology.orchestra.run.smithi167.stdout: "ops": 1, 2023-10-05T19:29:33.801 INFO:teuthology.orchestra.run.smithi167.stdout: "successful_ops": 1 2023-10-05T19:29:33.801 INFO:teuthology.orchestra.run.smithi167.stdout: } 2023-10-05T19:29:33.801 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2023-10-05T19:29:33.801 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2023-10-05T19:29:33.801 INFO:teuthology.orchestra.run.smithi167.stdout: { 2023-10-05T19:29:33.801 INFO:teuthology.orchestra.run.smithi167.stdout: "bucket": "myfoo3", 2023-10-05T19:29:33.801 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2023-10-05T19:00:00.000000Z", 2023-10-05T19:29:33.801 INFO:teuthology.orchestra.run.smithi167.stdout: "epoch": 1696532400, 2023-10-05T19:29:33.802 INFO:teuthology.orchestra.run.smithi167.stdout: "owner": "foo", 2023-10-05T19:29:33.802 INFO:teuthology.orchestra.run.smithi167.stdout: "categories": [ 2023-10-05T19:29:33.802 INFO:teuthology.orchestra.run.smithi167.stdout: { 2023-10-05T19:29:33.802 INFO:teuthology.orchestra.run.smithi167.stdout: "category": "create_bucket", 2023-10-05T19:29:33.802 INFO:teuthology.orchestra.run.smithi167.stdout: "bytes_sent": 0, 2023-10-05T19:29:33.802 INFO:teuthology.orchestra.run.smithi167.stdout: "bytes_received": 0, 2023-10-05T19:29:33.802 INFO:teuthology.orchestra.run.smithi167.stdout: "ops": 1, 2023-10-05T19:29:33.802 INFO:teuthology.orchestra.run.smithi167.stdout: "successful_ops": 1 2023-10-05T19:29:33.802 INFO:teuthology.orchestra.run.smithi167.stdout: } 2023-10-05T19:29:33.802 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2023-10-05T19:29:33.802 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2023-10-05T19:29:33.802 INFO:teuthology.orchestra.run.smithi167.stdout: { 2023-10-05T19:29:33.803 INFO:teuthology.orchestra.run.smithi167.stdout: "bucket": "myfoo4", 2023-10-05T19:29:33.803 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2023-10-05T19:00:00.000000Z", 2023-10-05T19:29:33.803 INFO:teuthology.orchestra.run.smithi167.stdout: "epoch": 1696532400, 2023-10-05T19:29:33.803 INFO:teuthology.orchestra.run.smithi167.stdout: "owner": "foo", 2023-10-05T19:29:33.803 INFO:teuthology.orchestra.run.smithi167.stdout: "categories": [ 2023-10-05T19:29:33.803 INFO:teuthology.orchestra.run.smithi167.stdout: { 2023-10-05T19:29:33.803 INFO:teuthology.orchestra.run.smithi167.stdout: "category": "create_bucket", 2023-10-05T19:29:33.803 INFO:teuthology.orchestra.run.smithi167.stdout: "bytes_sent": 0, 2023-10-05T19:29:33.803 INFO:teuthology.orchestra.run.smithi167.stdout: "bytes_received": 0, 2023-10-05T19:29:33.803 INFO:teuthology.orchestra.run.smithi167.stdout: "ops": 1, 2023-10-05T19:29:33.803 INFO:teuthology.orchestra.run.smithi167.stdout: "successful_ops": 1 2023-10-05T19:29:33.803 INFO:teuthology.orchestra.run.smithi167.stdout: } 2023-10-05T19:29:33.803 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2023-10-05T19:29:33.804 INFO:teuthology.orchestra.run.smithi167.stdout: } 2023-10-05T19:29:33.804 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2023-10-05T19:29:33.804 INFO:teuthology.orchestra.run.smithi167.stdout: } 2023-10-05T19:29:33.804 INFO:teuthology.orchestra.run.smithi167.stdout: ], 2023-10-05T19:29:33.804 INFO:teuthology.orchestra.run.smithi167.stdout: "summary": [ 2023-10-05T19:29:33.804 INFO:teuthology.orchestra.run.smithi167.stdout: { 2023-10-05T19:29:33.804 INFO:teuthology.orchestra.run.smithi167.stdout: "user": "foo", 2023-10-05T19:29:33.804 INFO:teuthology.orchestra.run.smithi167.stdout: "categories": [ 2023-10-05T19:29:33.804 INFO:teuthology.orchestra.run.smithi167.stdout: { 2023-10-05T19:29:33.804 INFO:teuthology.orchestra.run.smithi167.stdout: "category": "create_bucket", 2023-10-05T19:29:33.804 INFO:teuthology.orchestra.run.smithi167.stdout: "bytes_sent": 0, 2023-10-05T19:29:33.804 INFO:teuthology.orchestra.run.smithi167.stdout: "bytes_received": 0, 2023-10-05T19:29:33.805 INFO:teuthology.orchestra.run.smithi167.stdout: "ops": 8, 2023-10-05T19:29:33.805 INFO:teuthology.orchestra.run.smithi167.stdout: "successful_ops": 8 2023-10-05T19:29:33.805 INFO:teuthology.orchestra.run.smithi167.stdout: } 2023-10-05T19:29:33.805 INFO:teuthology.orchestra.run.smithi167.stdout: ], 2023-10-05T19:29:33.805 INFO:teuthology.orchestra.run.smithi167.stdout: "total": { 2023-10-05T19:29:33.805 INFO:teuthology.orchestra.run.smithi167.stdout: "bytes_sent": 0, 2023-10-05T19:29:33.805 INFO:teuthology.orchestra.run.smithi167.stdout: "bytes_received": 0, 2023-10-05T19:29:33.805 INFO:teuthology.orchestra.run.smithi167.stdout: "ops": 8, 2023-10-05T19:29:33.805 INFO:teuthology.orchestra.run.smithi167.stdout: "successful_ops": 8 2023-10-05T19:29:33.805 INFO:teuthology.orchestra.run.smithi167.stdout: } 2023-10-05T19:29:33.805 INFO:teuthology.orchestra.run.smithi167.stdout: } 2023-10-05T19:29:33.805 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2023-10-05T19:29:33.806 INFO:teuthology.orchestra.run.smithi167.stdout:} 2023-10-05T19:29:33.806 DEBUG:tasks.util.rgw: json result: {'entries': [{'user': 'foo', 'buckets': [{'bucket': '', 'time': '2023-10-05T19:00:00.000000Z', 'epoch': 1696532400, 'owner': 'foo', 'categories': []}, {'bucket': 'myfoo', 'time': '2023-10-05T19:00:00.000000Z', 'epoch': 1696532400, 'owner': 'foo', 'categories': [{'category': 'create_bucket', 'bytes_sent': 0, 'bytes_received': 0, 'ops': 5, 'successful_ops': 5}]}, {'bucket': 'myfoo2', 'time': '2023-10-05T19:00:00.000000Z', 'epoch': 1696532400, 'owner': 'foo', 'categories': [{'category': 'create_bucket', 'bytes_sent': 0, 'bytes_received': 0, 'ops': 1, 'successful_ops': 1}]}, {'bucket': 'myfoo3', 'time': '2023-10-05T19:00:00.000000Z', 'epoch': 1696532400, 'owner': 'foo', 'categories': [{'category': 'create_bucket', 'bytes_sent': 0, 'bytes_received': 0, 'ops': 1, 'successful_ops': 1}]}, {'bucket': 'myfoo4', 'time': '2023-10-05T19:00:00.000000Z', 'epoch': 1696532400, 'owner': 'foo', 'categories': [{'category': 'create_bucket', 'bytes_sent': 0, 'bytes_received': 0, 'ops': 1, 'successful_ops': 1}]}]}], 'summary': [{'user': 'foo', 'categories': [{'category': 'create_bucket', 'bytes_sent': 0, 'bytes_received': 0, 'ops': 8, 'successful_ops': 8}], 'total': {'bytes_sent': 0, 'bytes_received': 0, 'ops': 8, 'successful_ops': 8}}]} 2023-10-05T19:29:33.807 INFO:tasks.util.rgw:rgwadmin: client.0 : ['usage', 'show', '--uid', 'foo', '--categories', 'put_obj'] 2023-10-05T19:29:33.807 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'ceph', 'usage', 'show', '--uid', 'foo', '--categories', 'put_obj'] 2023-10-05T19:29:33.807 DEBUG:teuthology.orchestra.run.smithi167:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster ceph usage show --uid foo --categories put_obj 2023-10-05T19:29:33.889 INFO:teuthology.orchestra.run.smithi167.stderr:ignoring --setuser ceph since I am not root 2023-10-05T19:29:33.889 INFO:teuthology.orchestra.run.smithi167.stderr:ignoring --setgroup ceph since I am not root 2023-10-05T19:29:33.890 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.886+0000 7f5726a27e40 1 Processor -- start 2023-10-05T19:29:33.890 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.886+0000 7f5726a27e40 1 -- start start 2023-10-05T19:29:33.890 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.886+0000 7f5726a27e40 1 --2- >> v2:172.21.15.118:3300/0 conn(0x55d8e571a420 0x55d8e5718fd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-05T19:29:33.890 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.886+0000 7f5726a27e40 1 -- --> v1:172.21.15.118:6789/0 -- auth(proto 0 26 bytes epoch 0) v1 -- 0x55d8e5666bd0 con 0x55d8e571a7f0 2023-10-05T19:29:33.890 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.886+0000 7f5726a27e40 1 -- --> v2:172.21.15.118:3300/0 -- mon_getmap magic: 0 v1 -- 0x55d8e5623ee0 con 0x55d8e571a420 2023-10-05T19:29:33.891 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.886+0000 7f572455c700 1 --2- >> v2:172.21.15.118:3300/0 conn(0x55d8e571a420 0x55d8e5718fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-05T19:29:33.892 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.886+0000 7f5723d5b700 1 --1- >> v1:172.21.15.118:6789/0 conn(0x55d8e571a7f0 0x55d8e5719510 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.118:6789/0 says I am v1:172.21.15.167:52620/0 (socket says 172.21.15.167:52620) 2023-10-05T19:29:33.892 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.886+0000 7f5723d5b700 1 -- 172.21.15.167:0/2371695189 learned_addr learned my addr 172.21.15.167:0/2371695189 (peer_addr_for_me v1:172.21.15.167:0/0) 2023-10-05T19:29:33.892 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.886+0000 7f572455c700 1 -- 172.21.15.167:0/2371695189 >> v1:172.21.15.118:6789/0 conn(0x55d8e571a7f0 legacy=0x55d8e5719510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-05T19:29:33.892 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.886+0000 7f572355a700 1 -- 172.21.15.167:0/2371695189 <== mon.0 v1:172.21.15.118:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 510956459 0 0) 0x55d8e5666bd0 con 0x55d8e571a7f0 2023-10-05T19:29:33.892 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.886+0000 7f572455c700 1 -- 172.21.15.167:0/2371695189 --> v2:172.21.15.118:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55d8e5719ce0 con 0x55d8e571a420 2023-10-05T19:29:33.893 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.886+0000 7f572455c700 1 --2- 172.21.15.167:0/2371695189 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55d8e571a420 0x55d8e5718fd0 secure :-1 s=READY pgs=597 cs=0 l=1 rev1=1 crypto rx=0x7f571c00a520 tx=0x7f571c005d40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e56e1b09a9adee1 server_cookie=0 in_seq=0 out_seq=0 2023-10-05T19:29:33.893 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.886+0000 7f572355a700 1 -- 172.21.15.167:0/2371695189 <== mon.0 v2:172.21.15.118:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f571c00f590 con 0x55d8e571a420 2023-10-05T19:29:33.893 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.890+0000 7f572355a700 1 -- 172.21.15.167:0/2371695189 <== mon.0 v2:172.21.15.118:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f571c00f6f0 con 0x55d8e571a420 2023-10-05T19:29:33.893 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.890+0000 7f572355a700 1 -- 172.21.15.167:0/2371695189 <== mon.0 v2:172.21.15.118:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f571c00f880 con 0x55d8e571a420 2023-10-05T19:29:33.894 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.890+0000 7f5726a27e40 1 -- 172.21.15.167:0/2371695189 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55d8e571a420 msgr2=0x55d8e5718fd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-05T19:29:33.894 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.890+0000 7f5726a27e40 1 --2- 172.21.15.167:0/2371695189 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55d8e571a420 0x55d8e5718fd0 secure :-1 s=READY pgs=597 cs=0 l=1 rev1=1 crypto rx=0x7f571c00a520 tx=0x7f571c005d40 comp rx=0 tx=0).stop 2023-10-05T19:29:33.894 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.890+0000 7f5726a27e40 1 -- 172.21.15.167:0/2371695189 shutdown_connections 2023-10-05T19:29:33.894 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.890+0000 7f5726a27e40 1 --2- 172.21.15.167:0/2371695189 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55d8e571a420 0x55d8e5718fd0 unknown :-1 s=CLOSED pgs=597 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-05T19:29:33.894 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.890+0000 7f5726a27e40 1 -- 172.21.15.167:0/2371695189 >> 172.21.15.167:0/2371695189 conn(0x55d8e5714730 msgr2=0x55d8e5716b40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-05T19:29:33.894 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.890+0000 7f5726a27e40 1 -- 172.21.15.167:0/2371695189 shutdown_connections 2023-10-05T19:29:33.894 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.890+0000 7f5726a27e40 1 -- 172.21.15.167:0/2371695189 wait complete. 2023-10-05T19:29:33.895 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.890+0000 7f5726a27e40 1 Processor -- start 2023-10-05T19:29:33.895 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.890+0000 7f5726a27e40 1 -- start start 2023-10-05T19:29:33.895 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.890+0000 7f5726a27e40 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55d8e571a420 0x55d8e573af50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-05T19:29:33.895 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.890+0000 7f5726a27e40 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d8e55e7e30 con 0x55d8e571a420 2023-10-05T19:29:33.896 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.890+0000 7f572455c700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55d8e571a420 0x55d8e573af50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-05T19:29:33.896 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.890+0000 7f572455c700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55d8e571a420 0x55d8e573af50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.118:3300/0 says I am v2:172.21.15.167:46262/0 (socket says 172.21.15.167:46262) 2023-10-05T19:29:33.896 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.890+0000 7f572455c700 1 -- 172.21.15.167:0/1724674810 learned_addr learned my addr 172.21.15.167:0/1724674810 (peer_addr_for_me v2:172.21.15.167:0/0) 2023-10-05T19:29:33.896 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.890+0000 7f572455c700 1 -- 172.21.15.167:0/1724674810 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f571c00a0e0 con 0x55d8e571a420 2023-10-05T19:29:33.896 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.890+0000 7f572455c700 1 --2- 172.21.15.167:0/1724674810 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55d8e571a420 0x55d8e573af50 secure :-1 s=READY pgs=598 cs=0 l=1 rev1=1 crypto rx=0x7f571c00a470 tx=0x7f571c014000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=995ad236cdd551bf server_cookie=0 in_seq=0 out_seq=0 2023-10-05T19:29:33.897 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.890+0000 7f5722558700 1 -- 172.21.15.167:0/1724674810 <== mon.0 v2:172.21.15.118:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f571c00f860 con 0x55d8e571a420 2023-10-05T19:29:33.897 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.890+0000 7f5722558700 1 -- 172.21.15.167:0/1724674810 <== mon.0 v2:172.21.15.118:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f571c00f9e0 con 0x55d8e571a420 2023-10-05T19:29:33.897 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.890+0000 7f5722558700 1 -- 172.21.15.167:0/1724674810 <== mon.0 v2:172.21.15.118:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f571c00fb90 con 0x55d8e571a420 2023-10-05T19:29:33.897 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.894+0000 7f5726a27e40 1 -- 172.21.15.167:0/1724674810 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55d8e571a420 msgr2=0x55d8e573af50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-05T19:29:33.898 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.894+0000 7f5726a27e40 1 --2- 172.21.15.167:0/1724674810 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55d8e571a420 0x55d8e573af50 secure :-1 s=READY pgs=598 cs=0 l=1 rev1=1 crypto rx=0x7f571c00a470 tx=0x7f571c014000 comp rx=0 tx=0).stop 2023-10-05T19:29:33.898 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.894+0000 7f5726a27e40 1 -- 172.21.15.167:0/1724674810 shutdown_connections 2023-10-05T19:29:33.898 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.894+0000 7f5726a27e40 1 --2- 172.21.15.167:0/1724674810 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55d8e571a420 0x55d8e573af50 unknown :-1 s=CLOSED pgs=598 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-05T19:29:33.898 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.894+0000 7f5726a27e40 1 -- 172.21.15.167:0/1724674810 >> 172.21.15.167:0/1724674810 conn(0x55d8e5714730 msgr2=0x55d8e5736b10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-05T19:29:33.898 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.894+0000 7f5726a27e40 1 -- 172.21.15.167:0/1724674810 shutdown_connections 2023-10-05T19:29:33.898 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.894+0000 7f5726a27e40 1 -- 172.21.15.167:0/1724674810 wait complete. 2023-10-05T19:29:33.899 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.894+0000 7f5726a27e40 1 Processor -- start 2023-10-05T19:29:33.899 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.894+0000 7f5726a27e40 1 -- start start 2023-10-05T19:29:33.899 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.894+0000 7f5726a27e40 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55d8e571a420 0x55d8e5741bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-05T19:29:33.899 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.894+0000 7f5726a27e40 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d8e55baed0 con 0x55d8e571a420 2023-10-05T19:29:33.900 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.894+0000 7f572455c700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55d8e571a420 0x55d8e5741bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-05T19:29:33.900 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.894+0000 7f572455c700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55d8e571a420 0x55d8e5741bb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.118:3300/0 says I am v2:172.21.15.167:46272/0 (socket says 172.21.15.167:46272) 2023-10-05T19:29:33.900 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.894+0000 7f572455c700 1 -- 172.21.15.167:0/3631952810 learned_addr learned my addr 172.21.15.167:0/3631952810 (peer_addr_for_me v2:172.21.15.167:0/0) 2023-10-05T19:29:33.900 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.894+0000 7f572455c700 1 -- 172.21.15.167:0/3631952810 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55d8e5719ce0 con 0x55d8e571a420 2023-10-05T19:29:33.900 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.898+0000 7f572455c700 1 --2- 172.21.15.167:0/3631952810 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55d8e571a420 0x55d8e5741bb0 secure :-1 s=READY pgs=599 cs=0 l=1 rev1=1 crypto rx=0x7f571c00a470 tx=0x7f571c01d000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-05T19:29:33.901 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.898+0000 7f5720d55700 1 -- 172.21.15.167:0/3631952810 <== mon.0 v2:172.21.15.118:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f571c01ce00 con 0x55d8e571a420 2023-10-05T19:29:33.901 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.898+0000 7f5720d55700 1 -- 172.21.15.167:0/3631952810 <== mon.0 v2:172.21.15.118:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f571c010630 con 0x55d8e571a420 2023-10-05T19:29:33.901 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.898+0000 7f5720d55700 1 -- 172.21.15.167:0/3631952810 <== mon.0 v2:172.21.15.118:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f571c010820 con 0x55d8e571a420 2023-10-05T19:29:33.901 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.898+0000 7f5726a27e40 1 -- 172.21.15.167:0/3631952810 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x55d8e57420f0 con 0x55d8e571a420 2023-10-05T19:29:33.902 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.898+0000 7f5726a27e40 1 -- 172.21.15.167:0/3631952810 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55d8e5742530 con 0x55d8e571a420 2023-10-05T19:29:33.902 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.898+0000 7f5726a27e40 20 rados->read ofs=0 len=0 2023-10-05T19:29:33.903 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.898+0000 7f5720d55700 1 -- 172.21.15.167:0/3631952810 <== mon.0 v2:172.21.15.118:3300/0 4 ==== mgrmap(e 5) v1 ==== 40329+0+0 (secure 0 0 0) 0x7f571c010980 con 0x55d8e571a420 2023-10-05T19:29:33.904 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.898+0000 7f5720d55700 1 --2- 172.21.15.167:0/3631952810 >> [v2:172.21.15.167:6824/29889,v1:172.21.15.167:6825/29889] conn(0x7f5704030c50 0x7f5704033100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-05T19:29:33.904 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.898+0000 7f5720d55700 1 -- 172.21.15.167:0/3631952810 <== mon.0 v2:172.21.15.118:3300/0 5 ==== osd_map(32..32 src has 1..32) v4 ==== 6260+0+0 (secure 0 0 0) 0x7f571c02b070 con 0x55d8e571a420 2023-10-05T19:29:33.904 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.898+0000 7f5723d5b700 1 --2- 172.21.15.167:0/3631952810 >> [v2:172.21.15.167:6824/29889,v1:172.21.15.167:6825/29889] conn(0x7f5704030c50 0x7f5704033100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-05T19:29:33.904 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.898+0000 7f5726a27e40 1 --2- 172.21.15.167:0/3631952810 >> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] conn(0x55d8e571a7f0 0x55d8e574d610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-05T19:29:33.905 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.898+0000 7f5726a27e40 1 -- 172.21.15.167:0/3631952810 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:1 5.12 5:49953fa1:::default.realm:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32) v8 -- 0x55d8e574dcc0 con 0x55d8e571a7f0 2023-10-05T19:29:33.905 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.902+0000 7f5724d5d700 1 --2- 172.21.15.167:0/3631952810 >> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] conn(0x55d8e571a7f0 0x55d8e574d610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-05T19:29:33.905 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.902+0000 7f5723d5b700 1 --2- 172.21.15.167:0/3631952810 >> [v2:172.21.15.167:6824/29889,v1:172.21.15.167:6825/29889] conn(0x7f5704030c50 0x7f5704033100 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7f570c0011e0 tx=0x7f570c0013a0 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-05T19:29:33.905 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.902+0000 7f5724d5d700 1 --2- 172.21.15.167:0/3631952810 >> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] conn(0x55d8e571a7f0 0x55d8e574d610 crc :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-05T19:29:33.906 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.902+0000 7f5724d5d700 1 -- 172.21.15.167:0/3631952810 <== osd.1 v2:172.21.15.167:6800/30035 1 ==== osd_op_reply(1 default.realm [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 ==== 157+0+0 (crc 0 0 0) 0x7f571400a500 con 0x55d8e571a7f0 2023-10-05T19:29:33.906 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.902+0000 7f5726a27e40 20 rados_obj.operate() r=-2 bl.length=0 2023-10-05T19:29:33.906 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.902+0000 7f5726a27e40 20 realm 2023-10-05T19:29:33.906 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.902+0000 7f5726a27e40 20 rados->read ofs=0 len=0 2023-10-05T19:29:33.907 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.902+0000 7f5726a27e40 1 -- 172.21.15.167:0/3631952810 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:2 5.12 5:49953fa1:::default.realm:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32) v8 -- 0x55d8e574ac70 con 0x55d8e571a7f0 2023-10-05T19:29:33.907 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.902+0000 7f5724d5d700 1 -- 172.21.15.167:0/3631952810 <== osd.1 v2:172.21.15.167:6800/30035 2 ==== osd_op_reply(2 default.realm [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 ==== 157+0+0 (crc 0 0 0) 0x7f571400a500 con 0x55d8e571a7f0 2023-10-05T19:29:33.907 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.902+0000 7f5726a27e40 20 rados_obj.operate() r=-2 bl.length=0 2023-10-05T19:29:33.907 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.902+0000 7f5726a27e40 4 RGWPeriod::init failed to init realm id : (2) No such file or directory 2023-10-05T19:29:33.907 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.902+0000 7f5726a27e40 20 rados->read ofs=0 len=0 2023-10-05T19:29:33.907 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.902+0000 7f5726a27e40 1 -- 172.21.15.167:0/3631952810 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:3 5.12 5:49953fa1:::default.realm:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32) v8 -- 0x55d8e574e7a0 con 0x55d8e571a7f0 2023-10-05T19:29:33.909 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.902+0000 7f5724d5d700 1 -- 172.21.15.167:0/3631952810 <== osd.1 v2:172.21.15.167:6800/30035 3 ==== osd_op_reply(3 default.realm [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 ==== 157+0+0 (crc 0 0 0) 0x7f571400a500 con 0x55d8e571a7f0 2023-10-05T19:29:33.909 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.902+0000 7f5726a27e40 20 rados_obj.operate() r=-2 bl.length=0 2023-10-05T19:29:33.909 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.902+0000 7f5726a27e40 20 rados->read ofs=0 len=0 2023-10-05T19:29:33.909 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.902+0000 7f5726a27e40 1 -- 172.21.15.167:0/3631952810 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:4 5.f 5:f43b6ece:::zone_names.default:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32) v8 -- 0x55d8e574e380 con 0x55d8e571a7f0 2023-10-05T19:29:33.909 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.902+0000 7f5724d5d700 1 -- 172.21.15.167:0/3631952810 <== osd.1 v2:172.21.15.167:6800/30035 4 ==== osd_op_reply(4 zone_names.default [read 0~46 out=46b] v0'0 uv1 ondisk = 0) v8 ==== 162+0+46 (crc 0 0 0) 0x7f571400a500 con 0x55d8e571a7f0 2023-10-05T19:29:33.909 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.902+0000 7f5726a27e40 20 rados_obj.operate() r=0 bl.length=46 2023-10-05T19:29:33.909 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.902+0000 7f5726a27e40 20 rados->read ofs=0 len=0 2023-10-05T19:29:33.909 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.902+0000 7f5726a27e40 1 -- 172.21.15.167:0/3631952810 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:5 5.1d 5:b8374eb4:::zone_info.9a9df965-6612-45d4-a45f-6e3c889bacc2:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32) v8 -- 0x55d8e574f210 con 0x55d8e571a7f0 2023-10-05T19:29:33.909 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.902+0000 7f5724d5d700 1 -- 172.21.15.167:0/3631952810 <== osd.1 v2:172.21.15.167:6800/30035 5 ==== osd_op_reply(5 zone_info.9a9df965-6612-45d4-a45f-6e3c889bacc2 [read 0~874 out=874b] v0'0 uv2 ondisk = 0) v8 ==== 190+0+874 (crc 0 0 0) 0x7f571400a500 con 0x55d8e571a7f0 2023-10-05T19:29:33.909 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.902+0000 7f5726a27e40 20 rados_obj.operate() r=0 bl.length=874 2023-10-05T19:29:33.910 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.902+0000 7f5726a27e40 20 searching for the correct realm 2023-10-05T19:29:33.910 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.902+0000 7f5726a27e40 1 --2- 172.21.15.167:0/3631952810 >> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] conn(0x55d8e5751060 0x55d8e5753440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-05T19:29:33.910 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.902+0000 7f5726a27e40 1 -- 172.21.15.167:0/3631952810 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:6 5.0 5:00000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x55d8e5753980 con 0x55d8e5751060 2023-10-05T19:29:33.910 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.902+0000 7f572455c700 1 --2- 172.21.15.167:0/3631952810 >> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] conn(0x55d8e5751060 0x55d8e5753440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-05T19:29:33.910 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.906+0000 7f572455c700 1 --2- 172.21.15.167:0/3631952810 >> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] conn(0x55d8e5751060 0x55d8e5753440 crc :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-05T19:29:33.911 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.906+0000 7f572455c700 1 -- 172.21.15.167:0/3631952810 <== osd.3 v2:172.21.15.167:6808/30036 1 ==== osd_op_reply(6 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f571c024920 con 0x55d8e5751060 2023-10-05T19:29:33.911 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.906+0000 7f572455c700 1 --2- 172.21.15.167:0/3631952810 >> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] conn(0x7f571c037c00 0x7f571c004620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-05T19:29:33.911 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.906+0000 7f572455c700 1 -- 172.21.15.167:0/3631952810 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:7 5.10 5:08000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x55d8e5753980 con 0x7f571c037c00 2023-10-05T19:29:33.911 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.906+0000 7f5723d5b700 1 --2- 172.21.15.167:0/3631952810 >> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] conn(0x7f571c037c00 0x7f571c004620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-05T19:29:33.911 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.906+0000 7f5723d5b700 1 --2- 172.21.15.167:0/3631952810 >> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] conn(0x7f571c037c00 0x7f571c004620 crc :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-05T19:29:33.912 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.906+0000 7f5723d5b700 1 -- 172.21.15.167:0/3631952810 <== osd.2 v2:172.21.15.167:6816/30037 1 ==== osd_op_reply(7 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f570c006070 con 0x7f571c037c00 2023-10-05T19:29:33.912 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.906+0000 7f5723d5b700 1 --2- 172.21.15.167:0/3631952810 >> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] conn(0x7f570c00d8e0 0x7f570c00c360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-05T19:29:33.912 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.906+0000 7f5723d5b700 1 -- 172.21.15.167:0/3631952810 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:8 5.8 5:10000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x55d8e5753980 con 0x7f570c00d8e0 2023-10-05T19:29:33.912 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.906+0000 7f5724d5d700 1 --2- 172.21.15.167:0/3631952810 >> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] conn(0x7f570c00d8e0 0x7f570c00c360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-05T19:29:33.912 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.910+0000 7f5724d5d700 1 --2- 172.21.15.167:0/3631952810 >> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] conn(0x7f570c00d8e0 0x7f570c00c360 crc :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-05T19:29:33.913 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.910+0000 7f5724d5d700 1 -- 172.21.15.167:0/3631952810 <== osd.0 v2:172.21.15.118:6800/27690 1 ==== osd_op_reply(8 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f5714014040 con 0x7f570c00d8e0 2023-10-05T19:29:33.913 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.910+0000 7f5724d5d700 1 -- 172.21.15.167:0/3631952810 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:9 5.18 5:18000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x55d8e5753980 con 0x55d8e571a7f0 2023-10-05T19:29:33.915 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.910+0000 7f5724d5d700 1 -- 172.21.15.167:0/3631952810 <== osd.1 v2:172.21.15.167:6800/30035 6 ==== osd_op_reply(9 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f5714014040 con 0x55d8e571a7f0 2023-10-05T19:29:33.915 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.910+0000 7f5724d5d700 1 -- 172.21.15.167:0/3631952810 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:10 5.4 5:20000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x55d8e5753980 con 0x55d8e571a7f0 2023-10-05T19:29:33.915 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.910+0000 7f5724d5d700 1 -- 172.21.15.167:0/3631952810 <== osd.1 v2:172.21.15.167:6800/30035 7 ==== osd_op_reply(10 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f5714014040 con 0x55d8e571a7f0 2023-10-05T19:29:33.915 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.910+0000 7f5724d5d700 1 -- 172.21.15.167:0/3631952810 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:11 5.14 5:28000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x55d8e5753980 con 0x7f570c00d8e0 2023-10-05T19:29:33.916 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.910+0000 7f5724d5d700 1 -- 172.21.15.167:0/3631952810 <== osd.0 v2:172.21.15.118:6800/27690 2 ==== osd_op_reply(11 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f5714014040 con 0x7f570c00d8e0 2023-10-05T19:29:33.916 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.910+0000 7f5724d5d700 1 -- 172.21.15.167:0/3631952810 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:12 5.c 5:30000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x55d8e5753980 con 0x7f570c00d8e0 2023-10-05T19:29:33.916 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.910+0000 7f5724d5d700 1 -- 172.21.15.167:0/3631952810 <== osd.0 v2:172.21.15.118:6800/27690 3 ==== osd_op_reply(12 [pgnls start_epoch 32 out=112b] v23'1 uv1 ondisk = 1) v8 ==== 144+0+112 (crc 0 0 0) 0x7f5714014040 con 0x7f570c00d8e0 2023-10-05T19:29:33.916 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.910+0000 7f5724d5d700 1 -- 172.21.15.167:0/3631952810 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:13 5.1c 5:38000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x55d8e5753980 con 0x7f571c037c00 2023-10-05T19:29:33.917 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.914+0000 7f5723d5b700 1 -- 172.21.15.167:0/3631952810 <== osd.2 v2:172.21.15.167:6816/30037 2 ==== osd_op_reply(13 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f570c006070 con 0x7f571c037c00 2023-10-05T19:29:33.917 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.914+0000 7f5723d5b700 1 -- 172.21.15.167:0/3631952810 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:14 5.2 5:40000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x55d8e5753980 con 0x55d8e5751060 2023-10-05T19:29:33.917 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.914+0000 7f572455c700 1 -- 172.21.15.167:0/3631952810 <== osd.3 v2:172.21.15.167:6808/30036 2 ==== osd_op_reply(14 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f571c024920 con 0x55d8e5751060 2023-10-05T19:29:33.917 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.914+0000 7f572455c700 1 -- 172.21.15.167:0/3631952810 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:15 5.12 5:48000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x55d8e5753980 con 0x55d8e571a7f0 2023-10-05T19:29:33.918 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.914+0000 7f5724d5d700 1 -- 172.21.15.167:0/3631952810 <== osd.1 v2:172.21.15.167:6800/30035 8 ==== osd_op_reply(15 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f5714014040 con 0x55d8e571a7f0 2023-10-05T19:29:33.918 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.914+0000 7f5724d5d700 1 -- 172.21.15.167:0/3631952810 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:16 5.a 5:50000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x55d8e5753980 con 0x7f570c00d8e0 2023-10-05T19:29:33.919 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.914+0000 7f5724d5d700 1 -- 172.21.15.167:0/3631952810 <== osd.0 v2:172.21.15.118:6800/27690 4 ==== osd_op_reply(16 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f5714022040 con 0x7f570c00d8e0 2023-10-05T19:29:33.919 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.914+0000 7f5724d5d700 1 -- 172.21.15.167:0/3631952810 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:17 5.1a 5:58000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x55d8e5753980 con 0x55d8e571a7f0 2023-10-05T19:29:33.919 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.914+0000 7f5724d5d700 1 -- 172.21.15.167:0/3631952810 <== osd.1 v2:172.21.15.167:6800/30035 9 ==== osd_op_reply(17 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f5714022040 con 0x55d8e571a7f0 2023-10-05T19:29:33.920 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.914+0000 7f5724d5d700 1 -- 172.21.15.167:0/3631952810 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:18 5.6 5:60000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x55d8e5753980 con 0x7f571c037c00 2023-10-05T19:29:33.920 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.914+0000 7f5723d5b700 1 -- 172.21.15.167:0/3631952810 <== osd.2 v2:172.21.15.167:6816/30037 3 ==== osd_op_reply(18 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f570c006070 con 0x7f571c037c00 2023-10-05T19:29:33.920 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.914+0000 7f5723d5b700 1 -- 172.21.15.167:0/3631952810 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:19 5.16 5:68000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x55d8e5753980 con 0x55d8e5751060 2023-10-05T19:29:33.921 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.918+0000 7f572455c700 1 -- 172.21.15.167:0/3631952810 <== osd.3 v2:172.21.15.167:6808/30036 3 ==== osd_op_reply(19 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f571c024920 con 0x55d8e5751060 2023-10-05T19:29:33.921 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.918+0000 7f572455c700 1 -- 172.21.15.167:0/3631952810 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:20 5.e 5:70000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x55d8e5753980 con 0x55d8e5751060 2023-10-05T19:29:33.922 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.918+0000 7f572455c700 1 -- 172.21.15.167:0/3631952810 <== osd.3 v2:172.21.15.167:6808/30036 4 ==== osd_op_reply(20 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f571c024920 con 0x55d8e5751060 2023-10-05T19:29:33.922 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.918+0000 7f572455c700 1 -- 172.21.15.167:0/3631952810 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:21 5.1e 5:78000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x55d8e5753980 con 0x7f570c00d8e0 2023-10-05T19:29:33.923 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.918+0000 7f5724d5d700 1 -- 172.21.15.167:0/3631952810 <== osd.0 v2:172.21.15.118:6800/27690 5 ==== osd_op_reply(21 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f5714022040 con 0x7f570c00d8e0 2023-10-05T19:29:33.923 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.918+0000 7f5724d5d700 1 -- 172.21.15.167:0/3631952810 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:22 5.1 5:80000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x55d8e5753980 con 0x55d8e5751060 2023-10-05T19:29:33.923 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.918+0000 7f572455c700 1 -- 172.21.15.167:0/3631952810 <== osd.3 v2:172.21.15.167:6808/30036 5 ==== osd_op_reply(22 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f571c024920 con 0x55d8e5751060 2023-10-05T19:29:33.924 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.918+0000 7f572455c700 1 -- 172.21.15.167:0/3631952810 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:23 5.11 5:88000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x55d8e5753980 con 0x55d8e5751060 2023-10-05T19:29:33.924 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.918+0000 7f572455c700 1 -- 172.21.15.167:0/3631952810 <== osd.3 v2:172.21.15.167:6808/30036 6 ==== osd_op_reply(23 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f571c024920 con 0x55d8e5751060 2023-10-05T19:29:33.924 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.918+0000 7f572455c700 1 -- 172.21.15.167:0/3631952810 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:24 5.9 5:90000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x55d8e5753980 con 0x55d8e571a7f0 2023-10-05T19:29:33.925 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.922+0000 7f5724d5d700 1 -- 172.21.15.167:0/3631952810 <== osd.1 v2:172.21.15.167:6800/30035 10 ==== osd_op_reply(24 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f5714022040 con 0x55d8e571a7f0 2023-10-05T19:29:33.925 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.922+0000 7f5724d5d700 1 -- 172.21.15.167:0/3631952810 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:25 5.19 5:98000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x55d8e5753980 con 0x55d8e571a7f0 2023-10-05T19:29:33.926 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.922+0000 7f5724d5d700 1 -- 172.21.15.167:0/3631952810 <== osd.1 v2:172.21.15.167:6800/30035 11 ==== osd_op_reply(25 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f5714022040 con 0x55d8e571a7f0 2023-10-05T19:29:33.926 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.922+0000 7f5724d5d700 1 -- 172.21.15.167:0/3631952810 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:26 5.5 5:a0000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x55d8e5753980 con 0x7f570c00d8e0 2023-10-05T19:29:33.927 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.922+0000 7f5724d5d700 1 -- 172.21.15.167:0/3631952810 <== osd.0 v2:172.21.15.118:6800/27690 6 ==== osd_op_reply(26 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f5714022040 con 0x7f570c00d8e0 2023-10-05T19:29:33.927 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.922+0000 7f5724d5d700 1 -- 172.21.15.167:0/3631952810 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:27 5.15 5:a8000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x55d8e5753980 con 0x55d8e5751060 2023-10-05T19:29:33.927 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.922+0000 7f572455c700 1 -- 172.21.15.167:0/3631952810 <== osd.3 v2:172.21.15.167:6808/30036 7 ==== osd_op_reply(27 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f571c024920 con 0x55d8e5751060 2023-10-05T19:29:33.927 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.922+0000 7f572455c700 1 -- 172.21.15.167:0/3631952810 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:28 5.d 5:b0000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x55d8e5753980 con 0x7f571c037c00 2023-10-05T19:29:33.928 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.922+0000 7f5723d5b700 1 -- 172.21.15.167:0/3631952810 <== osd.2 v2:172.21.15.167:6816/30037 4 ==== osd_op_reply(28 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f570c006070 con 0x7f571c037c00 2023-10-05T19:29:33.928 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.922+0000 7f5723d5b700 1 -- 172.21.15.167:0/3631952810 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:29 5.1d 5:b8000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x55d8e5753980 con 0x55d8e571a7f0 2023-10-05T19:29:33.929 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.926+0000 7f5724d5d700 1 -- 172.21.15.167:0/3631952810 <== osd.1 v2:172.21.15.167:6800/30035 12 ==== osd_op_reply(29 [pgnls start_epoch 32 out=107b] v30'2 uv2 ondisk = 1) v8 ==== 144+0+107 (crc 0 0 0) 0x7f5714022040 con 0x55d8e571a7f0 2023-10-05T19:29:33.929 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.926+0000 7f5724d5d700 1 -- 172.21.15.167:0/3631952810 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:30 5.3 5:c0000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x55d8e5753980 con 0x7f570c00d8e0 2023-10-05T19:29:33.930 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.926+0000 7f5724d5d700 1 -- 172.21.15.167:0/3631952810 <== osd.0 v2:172.21.15.118:6800/27690 7 ==== osd_op_reply(30 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f5714022040 con 0x7f570c00d8e0 2023-10-05T19:29:33.930 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.926+0000 7f5724d5d700 1 -- 172.21.15.167:0/3631952810 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:31 5.13 5:c8000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x55d8e5753980 con 0x55d8e5751060 2023-10-05T19:29:33.931 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.926+0000 7f572455c700 1 -- 172.21.15.167:0/3631952810 <== osd.3 v2:172.21.15.167:6808/30036 8 ==== osd_op_reply(31 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f571c024920 con 0x55d8e5751060 2023-10-05T19:29:33.931 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.926+0000 7f572455c700 1 -- 172.21.15.167:0/3631952810 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:32 5.b 5:d0000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x55d8e5753980 con 0x7f570c00d8e0 2023-10-05T19:29:33.932 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.926+0000 7f5724d5d700 1 -- 172.21.15.167:0/3631952810 <== osd.0 v2:172.21.15.118:6800/27690 8 ==== osd_op_reply(32 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f5714022040 con 0x7f570c00d8e0 2023-10-05T19:29:33.932 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.926+0000 7f5724d5d700 1 -- 172.21.15.167:0/3631952810 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:33 5.1b 5:d8000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x55d8e5753980 con 0x7f571c037c00 2023-10-05T19:29:33.932 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.930+0000 7f5723d5b700 1 -- 172.21.15.167:0/3631952810 <== osd.2 v2:172.21.15.167:6816/30037 5 ==== osd_op_reply(33 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f570c006070 con 0x7f571c037c00 2023-10-05T19:29:33.933 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.930+0000 7f5723d5b700 1 -- 172.21.15.167:0/3631952810 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:34 5.7 5:e0000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x55d8e5753980 con 0x55d8e5751060 2023-10-05T19:29:33.933 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.930+0000 7f572455c700 1 -- 172.21.15.167:0/3631952810 <== osd.3 v2:172.21.15.167:6808/30036 9 ==== osd_op_reply(34 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f571c024920 con 0x55d8e5751060 2023-10-05T19:29:33.933 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.930+0000 7f572455c700 1 -- 172.21.15.167:0/3631952810 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:35 5.17 5:e8000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x55d8e5753980 con 0x7f570c00d8e0 2023-10-05T19:29:33.934 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.930+0000 7f5724d5d700 1 -- 172.21.15.167:0/3631952810 <== osd.0 v2:172.21.15.118:6800/27690 9 ==== osd_op_reply(35 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f5714022040 con 0x7f570c00d8e0 2023-10-05T19:29:33.934 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.930+0000 7f5724d5d700 1 -- 172.21.15.167:0/3631952810 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:36 5.f 5:f0000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x55d8e5753980 con 0x55d8e571a7f0 2023-10-05T19:29:33.935 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.930+0000 7f5724d5d700 1 -- 172.21.15.167:0/3631952810 <== osd.1 v2:172.21.15.167:6800/30035 13 ==== osd_op_reply(36 [pgnls start_epoch 32 out=115b] v23'2 uv2 ondisk = 1) v8 ==== 144+0+115 (crc 0 0 0) 0x7f571402f080 con 0x55d8e571a7f0 2023-10-05T19:29:33.935 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.930+0000 7f5724d5d700 1 -- 172.21.15.167:0/3631952810 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:37 5.1f 5:f8000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x55d8e5753980 con 0x7f571c037c00 2023-10-05T19:29:33.936 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.930+0000 7f5723d5b700 1 -- 172.21.15.167:0/3631952810 <== osd.2 v2:172.21.15.167:6816/30037 6 ==== osd_op_reply(37 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f570c006070 con 0x7f571c037c00 2023-10-05T19:29:33.936 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.930+0000 7f5726a27e40 20 RGWRados::pool_iterate: got zonegroup_info.4f1f719d-9648-417b-b10c-e5d939825419 2023-10-05T19:29:33.936 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.930+0000 7f5726a27e40 20 RGWRados::pool_iterate: got zone_info.9a9df965-6612-45d4-a45f-6e3c889bacc2 2023-10-05T19:29:33.936 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.930+0000 7f5726a27e40 20 RGWRados::pool_iterate: got zone_names.default 2023-10-05T19:29:33.936 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.930+0000 7f5726a27e40 20 RGWRados::pool_iterate: got zonegroups_names.default 2023-10-05T19:29:33.937 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.930+0000 7f5726a27e40 20 rados->read ofs=0 len=0 2023-10-05T19:29:33.937 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.930+0000 7f5726a27e40 1 -- 172.21.15.167:0/3631952810 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:38 5.12 5:49953fa1:::default.realm:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32) v8 -- 0x55d8e574fe10 con 0x55d8e571a7f0 2023-10-05T19:29:33.937 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.934+0000 7f5724d5d700 1 -- 172.21.15.167:0/3631952810 <== osd.1 v2:172.21.15.167:6800/30035 14 ==== osd_op_reply(38 default.realm [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 ==== 157+0+0 (crc 0 0 0) 0x7f571402f080 con 0x55d8e571a7f0 2023-10-05T19:29:33.938 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.934+0000 7f5726a27e40 20 rados_obj.operate() r=-2 bl.length=0 2023-10-05T19:29:33.938 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.934+0000 7f5726a27e40 20 rados->read ofs=0 len=0 2023-10-05T19:29:33.938 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.934+0000 7f5726a27e40 1 -- 172.21.15.167:0/3631952810 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:39 5.f 5:f4c53578:::zonegroups_names.default:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32) v8 -- 0x55d8e5750790 con 0x55d8e571a7f0 2023-10-05T19:29:33.938 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.934+0000 7f5724d5d700 1 -- 172.21.15.167:0/3631952810 <== osd.1 v2:172.21.15.167:6800/30035 15 ==== osd_op_reply(39 zonegroups_names.default [read 0~46 out=46b] v0'0 uv2 ondisk = 0) v8 ==== 168+0+46 (crc 0 0 0) 0x7f571402f080 con 0x55d8e571a7f0 2023-10-05T19:29:33.938 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.934+0000 7f5726a27e40 20 rados_obj.operate() r=0 bl.length=46 2023-10-05T19:29:33.938 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.934+0000 7f5726a27e40 20 rados->read ofs=0 len=0 2023-10-05T19:29:33.938 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.934+0000 7f5726a27e40 1 -- 172.21.15.167:0/3631952810 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:40 5.c 5:30d299a5:::zonegroup_info.4f1f719d-9648-417b-b10c-e5d939825419:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32) v8 -- 0x55d8e574fa00 con 0x7f570c00d8e0 2023-10-05T19:29:33.939 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.934+0000 7f5724d5d700 1 -- 172.21.15.167:0/3631952810 <== osd.0 v2:172.21.15.118:6800/27690 10 ==== osd_op_reply(40 zonegroup_info.4f1f719d-9648-417b-b10c-e5d939825419 [read 0~362 out=362b] v0'0 uv1 ondisk = 0) v8 ==== 195+0+362 (crc 0 0 0) 0x7f571402f080 con 0x7f570c00d8e0 2023-10-05T19:29:33.939 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.934+0000 7f5726a27e40 20 rados_obj.operate() r=0 bl.length=362 2023-10-05T19:29:33.939 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.934+0000 7f5726a27e40 20 rados->read ofs=0 len=0 2023-10-05T19:29:33.939 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.934+0000 7f5726a27e40 1 -- 172.21.15.167:0/3631952810 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:41 5.13 5:cbbf6c60:::converted:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32) v8 -- 0x55d8e5754220 con 0x55d8e5751060 2023-10-05T19:29:33.940 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.934+0000 7f572455c700 1 -- 172.21.15.167:0/3631952810 <== osd.3 v2:172.21.15.167:6808/30036 10 ==== osd_op_reply(41 converted [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 ==== 153+0+0 (crc 0 0 0) 0x7f571c024920 con 0x55d8e5751060 2023-10-05T19:29:33.940 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.934+0000 7f5726a27e40 20 rados_obj.operate() r=-2 bl.length=0 2023-10-05T19:29:33.940 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.934+0000 7f5726a27e40 20 rados->read ofs=0 len=0 2023-10-05T19:29:33.940 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.934+0000 7f5726a27e40 1 -- 172.21.15.167:0/3631952810 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:42 5.12 5:49953fa1:::default.realm:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32) v8 -- 0x55d8e57545d0 con 0x55d8e571a7f0 2023-10-05T19:29:33.940 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.934+0000 7f5724d5d700 1 -- 172.21.15.167:0/3631952810 <== osd.1 v2:172.21.15.167:6800/30035 16 ==== osd_op_reply(42 default.realm [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 ==== 157+0+0 (crc 0 0 0) 0x7f571402f080 con 0x55d8e571a7f0 2023-10-05T19:29:33.940 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.934+0000 7f5726a27e40 20 rados_obj.operate() r=-2 bl.length=0 2023-10-05T19:29:33.940 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.934+0000 7f5726a27e40 20 rados->read ofs=0 len=0 2023-10-05T19:29:33.941 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.934+0000 7f5726a27e40 1 -- 172.21.15.167:0/3631952810 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:43 5.f 5:f4c53578:::zonegroups_names.default:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32) v8 -- 0x55d8e57545d0 con 0x55d8e571a7f0 2023-10-05T19:29:33.941 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.938+0000 7f5724d5d700 1 -- 172.21.15.167:0/3631952810 <== osd.1 v2:172.21.15.167:6800/30035 17 ==== osd_op_reply(43 zonegroups_names.default [read 0~46 out=46b] v0'0 uv2 ondisk = 0) v8 ==== 168+0+46 (crc 0 0 0) 0x7f571402f080 con 0x55d8e571a7f0 2023-10-05T19:29:33.941 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.938+0000 7f5726a27e40 20 rados_obj.operate() r=0 bl.length=46 2023-10-05T19:29:33.941 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.938+0000 7f5726a27e40 1 -- 172.21.15.167:0/3631952810 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:44 5.0 5:00000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x55d8e57555e0 con 0x55d8e5751060 2023-10-05T19:29:33.942 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.938+0000 7f572455c700 1 -- 172.21.15.167:0/3631952810 <== osd.3 v2:172.21.15.167:6808/30036 11 ==== osd_op_reply(44 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f571c024920 con 0x55d8e5751060 2023-10-05T19:29:33.942 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.938+0000 7f572455c700 1 -- 172.21.15.167:0/3631952810 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:45 5.10 5:08000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x55d8e57555e0 con 0x7f571c037c00 2023-10-05T19:29:33.943 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.938+0000 7f5723d5b700 1 -- 172.21.15.167:0/3631952810 <== osd.2 v2:172.21.15.167:6816/30037 7 ==== osd_op_reply(45 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f570c006070 con 0x7f571c037c00 2023-10-05T19:29:33.943 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.938+0000 7f5723d5b700 1 -- 172.21.15.167:0/3631952810 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:46 5.8 5:10000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x55d8e57555e0 con 0x7f570c00d8e0 2023-10-05T19:29:33.944 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.938+0000 7f5724d5d700 1 -- 172.21.15.167:0/3631952810 <== osd.0 v2:172.21.15.118:6800/27690 11 ==== osd_op_reply(46 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f571402f080 con 0x7f570c00d8e0 2023-10-05T19:29:33.944 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.938+0000 7f5724d5d700 1 -- 172.21.15.167:0/3631952810 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:47 5.18 5:18000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x55d8e574fa00 con 0x55d8e571a7f0 2023-10-05T19:29:33.945 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.942+0000 7f5724d5d700 1 -- 172.21.15.167:0/3631952810 <== osd.1 v2:172.21.15.167:6800/30035 18 ==== osd_op_reply(47 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f571402f080 con 0x55d8e571a7f0 2023-10-05T19:29:33.945 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.942+0000 7f5724d5d700 1 -- 172.21.15.167:0/3631952810 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:48 5.4 5:20000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x55d8e574fa00 con 0x55d8e571a7f0 2023-10-05T19:29:33.945 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.942+0000 7f5724d5d700 1 -- 172.21.15.167:0/3631952810 <== osd.1 v2:172.21.15.167:6800/30035 19 ==== osd_op_reply(48 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f571402f080 con 0x55d8e571a7f0 2023-10-05T19:29:33.946 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.942+0000 7f5724d5d700 1 -- 172.21.15.167:0/3631952810 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:49 5.14 5:28000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x55d8e574fa00 con 0x7f570c00d8e0 2023-10-05T19:29:33.946 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.942+0000 7f5724d5d700 1 -- 172.21.15.167:0/3631952810 <== osd.0 v2:172.21.15.118:6800/27690 12 ==== osd_op_reply(49 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f571402f080 con 0x7f570c00d8e0 2023-10-05T19:29:33.947 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.942+0000 7f5724d5d700 1 -- 172.21.15.167:0/3631952810 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:50 5.c 5:30000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x55d8e574fa00 con 0x7f570c00d8e0 2023-10-05T19:29:33.947 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.942+0000 7f5724d5d700 1 -- 172.21.15.167:0/3631952810 <== osd.0 v2:172.21.15.118:6800/27690 13 ==== osd_op_reply(50 [pgnls start_epoch 32 out=112b] v23'1 uv1 ondisk = 1) v8 ==== 144+0+112 (crc 0 0 0) 0x7f571402f080 con 0x7f570c00d8e0 2023-10-05T19:29:33.947 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.942+0000 7f5724d5d700 1 -- 172.21.15.167:0/3631952810 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:51 5.1c 5:38000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x55d8e574fa00 con 0x7f571c037c00 2023-10-05T19:29:33.948 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.946+0000 7f5723d5b700 1 -- 172.21.15.167:0/3631952810 <== osd.2 v2:172.21.15.167:6816/30037 8 ==== osd_op_reply(51 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f570c006070 con 0x7f571c037c00 2023-10-05T19:29:33.949 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.946+0000 7f5723d5b700 1 -- 172.21.15.167:0/3631952810 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:52 5.2 5:40000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x55d8e574fa00 con 0x55d8e5751060 2023-10-05T19:29:33.949 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.946+0000 7f572455c700 1 -- 172.21.15.167:0/3631952810 <== osd.3 v2:172.21.15.167:6808/30036 12 ==== osd_op_reply(52 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f571c024920 con 0x55d8e5751060 2023-10-05T19:29:33.949 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.946+0000 7f572455c700 1 -- 172.21.15.167:0/3631952810 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:53 5.12 5:48000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x55d8e574fa00 con 0x55d8e571a7f0 2023-10-05T19:29:33.950 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.946+0000 7f5724d5d700 1 -- 172.21.15.167:0/3631952810 <== osd.1 v2:172.21.15.167:6800/30035 20 ==== osd_op_reply(53 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f571402f080 con 0x55d8e571a7f0 2023-10-05T19:29:33.950 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.946+0000 7f5724d5d700 1 -- 172.21.15.167:0/3631952810 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:54 5.a 5:50000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x55d8e574fa00 con 0x7f570c00d8e0 2023-10-05T19:29:33.951 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.946+0000 7f5724d5d700 1 -- 172.21.15.167:0/3631952810 <== osd.0 v2:172.21.15.118:6800/27690 14 ==== osd_op_reply(54 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f571402f080 con 0x7f570c00d8e0 2023-10-05T19:29:33.951 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.946+0000 7f5724d5d700 1 -- 172.21.15.167:0/3631952810 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:55 5.1a 5:58000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x55d8e574fa00 con 0x55d8e571a7f0 2023-10-05T19:29:33.952 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.946+0000 7f5724d5d700 1 -- 172.21.15.167:0/3631952810 <== osd.1 v2:172.21.15.167:6800/30035 21 ==== osd_op_reply(55 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f571402f080 con 0x55d8e571a7f0 2023-10-05T19:29:33.952 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.946+0000 7f5724d5d700 1 -- 172.21.15.167:0/3631952810 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:56 5.6 5:60000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x55d8e574fa00 con 0x7f571c037c00 2023-10-05T19:29:33.953 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.950+0000 7f5723d5b700 1 -- 172.21.15.167:0/3631952810 <== osd.2 v2:172.21.15.167:6816/30037 9 ==== osd_op_reply(56 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f570c006070 con 0x7f571c037c00 2023-10-05T19:29:33.953 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.950+0000 7f5723d5b700 1 -- 172.21.15.167:0/3631952810 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:57 5.16 5:68000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x55d8e574fa00 con 0x55d8e5751060 2023-10-05T19:29:33.954 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.950+0000 7f572455c700 1 -- 172.21.15.167:0/3631952810 <== osd.3 v2:172.21.15.167:6808/30036 13 ==== osd_op_reply(57 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f571c024920 con 0x55d8e5751060 2023-10-05T19:29:33.954 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.950+0000 7f572455c700 1 -- 172.21.15.167:0/3631952810 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:58 5.e 5:70000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x55d8e574fa00 con 0x55d8e5751060 2023-10-05T19:29:33.954 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.950+0000 7f572455c700 1 -- 172.21.15.167:0/3631952810 <== osd.3 v2:172.21.15.167:6808/30036 14 ==== osd_op_reply(58 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f571c024920 con 0x55d8e5751060 2023-10-05T19:29:33.955 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.950+0000 7f572455c700 1 -- 172.21.15.167:0/3631952810 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:59 5.1e 5:78000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x55d8e574fa00 con 0x7f570c00d8e0 2023-10-05T19:29:33.956 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.950+0000 7f5724d5d700 1 -- 172.21.15.167:0/3631952810 <== osd.0 v2:172.21.15.118:6800/27690 15 ==== osd_op_reply(59 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f571402f080 con 0x7f570c00d8e0 2023-10-05T19:29:33.956 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.950+0000 7f5724d5d700 1 -- 172.21.15.167:0/3631952810 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:60 5.1 5:80000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x55d8e574fa00 con 0x55d8e5751060 2023-10-05T19:29:33.956 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.954+0000 7f572455c700 1 -- 172.21.15.167:0/3631952810 <== osd.3 v2:172.21.15.167:6808/30036 15 ==== osd_op_reply(60 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f571c024920 con 0x55d8e5751060 2023-10-05T19:29:33.956 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.954+0000 7f572455c700 1 -- 172.21.15.167:0/3631952810 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:61 5.11 5:88000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x55d8e574fa00 con 0x55d8e5751060 2023-10-05T19:29:33.957 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.954+0000 7f572455c700 1 -- 172.21.15.167:0/3631952810 <== osd.3 v2:172.21.15.167:6808/30036 16 ==== osd_op_reply(61 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f571c024920 con 0x55d8e5751060 2023-10-05T19:29:33.957 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.954+0000 7f572455c700 1 -- 172.21.15.167:0/3631952810 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:62 5.9 5:90000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x55d8e574fa00 con 0x55d8e571a7f0 2023-10-05T19:29:33.958 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.954+0000 7f5724d5d700 1 -- 172.21.15.167:0/3631952810 <== osd.1 v2:172.21.15.167:6800/30035 22 ==== osd_op_reply(62 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f571402f080 con 0x55d8e571a7f0 2023-10-05T19:29:33.958 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.954+0000 7f5724d5d700 1 -- 172.21.15.167:0/3631952810 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:63 5.19 5:98000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x55d8e574fa00 con 0x55d8e571a7f0 2023-10-05T19:29:33.959 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.954+0000 7f5724d5d700 1 -- 172.21.15.167:0/3631952810 <== osd.1 v2:172.21.15.167:6800/30035 23 ==== osd_op_reply(63 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f571402f080 con 0x55d8e571a7f0 2023-10-05T19:29:33.959 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.954+0000 7f5724d5d700 1 -- 172.21.15.167:0/3631952810 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:64 5.5 5:a0000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x55d8e574fa00 con 0x7f570c00d8e0 2023-10-05T19:29:33.960 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.954+0000 7f5724d5d700 1 -- 172.21.15.167:0/3631952810 <== osd.0 v2:172.21.15.118:6800/27690 16 ==== osd_op_reply(64 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f571402f080 con 0x7f570c00d8e0 2023-10-05T19:29:33.960 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.954+0000 7f5724d5d700 1 -- 172.21.15.167:0/3631952810 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:65 5.15 5:a8000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x55d8e574fa00 con 0x55d8e5751060 2023-10-05T19:29:33.961 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.958+0000 7f572455c700 1 -- 172.21.15.167:0/3631952810 <== osd.3 v2:172.21.15.167:6808/30036 17 ==== osd_op_reply(65 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f571c024920 con 0x55d8e5751060 2023-10-05T19:29:33.961 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.958+0000 7f572455c700 1 -- 172.21.15.167:0/3631952810 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:66 5.d 5:b0000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x55d8e574fa00 con 0x7f571c037c00 2023-10-05T19:29:33.962 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.958+0000 7f5723d5b700 1 -- 172.21.15.167:0/3631952810 <== osd.2 v2:172.21.15.167:6816/30037 10 ==== osd_op_reply(66 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f570c006070 con 0x7f571c037c00 2023-10-05T19:29:33.962 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.958+0000 7f5723d5b700 1 -- 172.21.15.167:0/3631952810 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:67 5.1d 5:b8000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x55d8e574fa00 con 0x55d8e571a7f0 2023-10-05T19:29:33.963 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.958+0000 7f5724d5d700 1 -- 172.21.15.167:0/3631952810 <== osd.1 v2:172.21.15.167:6800/30035 24 ==== osd_op_reply(67 [pgnls start_epoch 32 out=107b] v30'2 uv2 ondisk = 1) v8 ==== 144+0+107 (crc 0 0 0) 0x7f571402f080 con 0x55d8e571a7f0 2023-10-05T19:29:33.963 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.958+0000 7f5724d5d700 1 -- 172.21.15.167:0/3631952810 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:68 5.3 5:c0000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x55d8e574fa00 con 0x7f570c00d8e0 2023-10-05T19:29:33.964 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.958+0000 7f5724d5d700 1 -- 172.21.15.167:0/3631952810 <== osd.0 v2:172.21.15.118:6800/27690 17 ==== osd_op_reply(68 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f571402f080 con 0x7f570c00d8e0 2023-10-05T19:29:33.964 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.958+0000 7f5724d5d700 1 -- 172.21.15.167:0/3631952810 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:69 5.13 5:c8000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x55d8e574fa00 con 0x55d8e5751060 2023-10-05T19:29:33.965 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.962+0000 7f572455c700 1 -- 172.21.15.167:0/3631952810 <== osd.3 v2:172.21.15.167:6808/30036 18 ==== osd_op_reply(69 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f571c024920 con 0x55d8e5751060 2023-10-05T19:29:33.965 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.962+0000 7f572455c700 1 -- 172.21.15.167:0/3631952810 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:70 5.b 5:d0000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x55d8e574fa00 con 0x7f570c00d8e0 2023-10-05T19:29:33.966 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.962+0000 7f5724d5d700 1 -- 172.21.15.167:0/3631952810 <== osd.0 v2:172.21.15.118:6800/27690 18 ==== osd_op_reply(70 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f571402f080 con 0x7f570c00d8e0 2023-10-05T19:29:33.966 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.962+0000 7f5724d5d700 1 -- 172.21.15.167:0/3631952810 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:71 5.1b 5:d8000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x55d8e574fa00 con 0x7f571c037c00 2023-10-05T19:29:33.967 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.962+0000 7f5723d5b700 1 -- 172.21.15.167:0/3631952810 <== osd.2 v2:172.21.15.167:6816/30037 11 ==== osd_op_reply(71 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f570c006070 con 0x7f571c037c00 2023-10-05T19:29:33.967 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.962+0000 7f5723d5b700 1 -- 172.21.15.167:0/3631952810 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:72 5.7 5:e0000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x55d8e574fa00 con 0x55d8e5751060 2023-10-05T19:29:33.968 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.962+0000 7f572455c700 1 -- 172.21.15.167:0/3631952810 <== osd.3 v2:172.21.15.167:6808/30036 19 ==== osd_op_reply(72 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f571c024920 con 0x55d8e5751060 2023-10-05T19:29:33.968 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.962+0000 7f572455c700 1 -- 172.21.15.167:0/3631952810 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:73 5.17 5:e8000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x55d8e574fa00 con 0x7f570c00d8e0 2023-10-05T19:29:33.969 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.966+0000 7f5724d5d700 1 -- 172.21.15.167:0/3631952810 <== osd.0 v2:172.21.15.118:6800/27690 19 ==== osd_op_reply(73 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f571402f080 con 0x7f570c00d8e0 2023-10-05T19:29:33.969 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.966+0000 7f5724d5d700 1 -- 172.21.15.167:0/3631952810 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:74 5.f 5:f0000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x55d8e574fa00 con 0x55d8e571a7f0 2023-10-05T19:29:33.970 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.966+0000 7f5724d5d700 1 -- 172.21.15.167:0/3631952810 <== osd.1 v2:172.21.15.167:6800/30035 25 ==== osd_op_reply(74 [pgnls start_epoch 32 out=115b] v23'2 uv2 ondisk = 1) v8 ==== 144+0+115 (crc 0 0 0) 0x7f571402f080 con 0x55d8e571a7f0 2023-10-05T19:29:33.970 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.966+0000 7f5724d5d700 1 -- 172.21.15.167:0/3631952810 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:75 5.1f 5:f8000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x55d8e574fa00 con 0x7f571c037c00 2023-10-05T19:29:33.971 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.966+0000 7f5723d5b700 1 -- 172.21.15.167:0/3631952810 <== osd.2 v2:172.21.15.167:6816/30037 12 ==== osd_op_reply(75 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f570c006070 con 0x7f571c037c00 2023-10-05T19:29:33.971 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.966+0000 7f5726a27e40 20 RGWRados::pool_iterate: got zonegroup_info.4f1f719d-9648-417b-b10c-e5d939825419 2023-10-05T19:29:33.971 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.966+0000 7f5726a27e40 20 RGWRados::pool_iterate: got zone_info.9a9df965-6612-45d4-a45f-6e3c889bacc2 2023-10-05T19:29:33.971 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.966+0000 7f5726a27e40 20 RGWRados::pool_iterate: got zone_names.default 2023-10-05T19:29:33.971 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.966+0000 7f5726a27e40 20 RGWRados::pool_iterate: got zonegroups_names.default 2023-10-05T19:29:33.971 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.966+0000 7f5726a27e40 20 rados->read ofs=0 len=0 2023-10-05T19:29:33.971 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.966+0000 7f5726a27e40 1 -- 172.21.15.167:0/3631952810 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:76 5.f 5:f43b6ece:::zone_names.default:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32) v8 -- 0x55d8e57545d0 con 0x55d8e571a7f0 2023-10-05T19:29:33.973 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.966+0000 7f5724d5d700 1 -- 172.21.15.167:0/3631952810 <== osd.1 v2:172.21.15.167:6800/30035 26 ==== osd_op_reply(76 zone_names.default [read 0~46 out=46b] v0'0 uv1 ondisk = 0) v8 ==== 162+0+46 (crc 0 0 0) 0x7f571402f080 con 0x55d8e571a7f0 2023-10-05T19:29:33.973 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.966+0000 7f5726a27e40 20 rados_obj.operate() r=0 bl.length=46 2023-10-05T19:29:33.973 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.966+0000 7f5726a27e40 20 rados->read ofs=0 len=0 2023-10-05T19:29:33.973 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.966+0000 7f5726a27e40 1 -- 172.21.15.167:0/3631952810 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:77 5.1d 5:b8374eb4:::zone_info.9a9df965-6612-45d4-a45f-6e3c889bacc2:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32) v8 -- 0x55d8e5754980 con 0x55d8e571a7f0 2023-10-05T19:29:33.973 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.966+0000 7f5724d5d700 1 -- 172.21.15.167:0/3631952810 <== osd.1 v2:172.21.15.167:6800/30035 27 ==== osd_op_reply(77 zone_info.9a9df965-6612-45d4-a45f-6e3c889bacc2 [read 0~874 out=874b] v0'0 uv2 ondisk = 0) v8 ==== 190+0+874 (crc 0 0 0) 0x7f571402f080 con 0x55d8e571a7f0 2023-10-05T19:29:33.973 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.970+0000 7f5726a27e40 20 rados_obj.operate() r=0 bl.length=874 2023-10-05T19:29:33.973 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.970+0000 7f5726a27e40 20 rados->read ofs=0 len=0 2023-10-05T19:29:33.973 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.970+0000 7f5726a27e40 1 -- 172.21.15.167:0/3631952810 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:78 5.f 5:f4c53578:::zonegroups_names.default:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32) v8 -- 0x55d8e5754e70 con 0x55d8e571a7f0 2023-10-05T19:29:33.974 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.970+0000 7f5724d5d700 1 -- 172.21.15.167:0/3631952810 <== osd.1 v2:172.21.15.167:6800/30035 28 ==== osd_op_reply(78 zonegroups_names.default [read 0~46 out=46b] v0'0 uv2 ondisk = 0) v8 ==== 168+0+46 (crc 0 0 0) 0x7f571402f080 con 0x55d8e571a7f0 2023-10-05T19:29:33.974 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.970+0000 7f5726a27e40 20 rados_obj.operate() r=0 bl.length=46 2023-10-05T19:29:33.974 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.970+0000 7f5726a27e40 20 rados->read ofs=0 len=0 2023-10-05T19:29:33.975 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.970+0000 7f5726a27e40 1 -- 172.21.15.167:0/3631952810 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:79 5.c 5:30d299a5:::zonegroup_info.4f1f719d-9648-417b-b10c-e5d939825419:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32) v8 -- 0x55d8e5754e70 con 0x7f570c00d8e0 2023-10-05T19:29:33.975 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.970+0000 7f5724d5d700 1 -- 172.21.15.167:0/3631952810 <== osd.0 v2:172.21.15.118:6800/27690 20 ==== osd_op_reply(79 zonegroup_info.4f1f719d-9648-417b-b10c-e5d939825419 [read 0~362 out=362b] v0'0 uv1 ondisk = 0) v8 ==== 195+0+362 (crc 0 0 0) 0x7f571402f080 con 0x7f570c00d8e0 2023-10-05T19:29:33.975 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.970+0000 7f5726a27e40 20 rados_obj.operate() r=0 bl.length=362 2023-10-05T19:29:33.975 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.970+0000 7f5726a27e40 20 rados->read ofs=0 len=0 2023-10-05T19:29:33.975 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.970+0000 7f5726a27e40 1 -- 172.21.15.167:0/3631952810 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:80 5.3 5:c0ad8cbd:::region_map:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32) v8 -- 0x55d8e5754f70 con 0x7f570c00d8e0 2023-10-05T19:29:33.976 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.970+0000 7f5724d5d700 1 -- 172.21.15.167:0/3631952810 <== osd.0 v2:172.21.15.118:6800/27690 21 ==== osd_op_reply(80 region_map [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 ==== 154+0+0 (crc 0 0 0) 0x7f5714050040 con 0x7f570c00d8e0 2023-10-05T19:29:33.976 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.970+0000 7f5726a27e40 20 rados_obj.operate() r=-2 bl.length=0 2023-10-05T19:29:33.976 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.970+0000 7f5726a27e40 20 zone default found 2023-10-05T19:29:33.976 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.970+0000 7f5726a27e40 4 Realm: () 2023-10-05T19:29:33.976 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.970+0000 7f5726a27e40 4 ZoneGroup: default (4f1f719d-9648-417b-b10c-e5d939825419) 2023-10-05T19:29:33.976 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.970+0000 7f5726a27e40 4 Zone: default (9a9df965-6612-45d4-a45f-6e3c889bacc2) 2023-10-05T19:29:33.976 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.970+0000 7f5726a27e40 10 cannot find current period zonegroup using local zonegroup configuration 2023-10-05T19:29:33.976 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.970+0000 7f5726a27e40 20 zonegroup default 2023-10-05T19:29:33.976 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.970+0000 7f5726a27e40 20 rados->read ofs=0 len=0 2023-10-05T19:29:33.976 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.970+0000 7f5726a27e40 1 -- 172.21.15.167:0/3631952810 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:81 5.3 5:c52100b6:::period_config.default:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32) v8 -- 0x55d8e5754f70 con 0x7f570c00d8e0 2023-10-05T19:29:33.977 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.970+0000 7f5724d5d700 1 -- 172.21.15.167:0/3631952810 <== osd.0 v2:172.21.15.118:6800/27690 22 ==== osd_op_reply(81 period_config.default [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 ==== 165+0+0 (crc 0 0 0) 0x7f5714050040 con 0x7f570c00d8e0 2023-10-05T19:29:33.977 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.970+0000 7f5726a27e40 20 rados_obj.operate() r=-2 bl.length=0 2023-10-05T19:29:33.977 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.970+0000 7f5726a27e40 20 rados->read ofs=0 len=0 2023-10-05T19:29:33.978 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.970+0000 7f5726a27e40 1 -- 172.21.15.167:0/3631952810 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:82 6.2 6:4347d321:::bucket.sync-source-hints.:head [call version.read in=11b,read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32) v8 -- 0x55d8e57568f0 con 0x55d8e5751060 2023-10-05T19:29:33.978 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.974+0000 7f572455c700 1 -- 172.21.15.167:0/3631952810 <== osd.3 v2:172.21.15.167:6808/30036 20 ==== osd_op_reply(82 bucket.sync-source-hints. [call,read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 ==== 211+0+0 (crc 0 0 0) 0x7f571c024920 con 0x55d8e5751060 2023-10-05T19:29:33.978 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.974+0000 7f5726a27e40 20 rados_obj.operate() r=-2 bl.length=0 2023-10-05T19:29:33.978 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.974+0000 7f5726a27e40 20 rados->read ofs=0 len=0 2023-10-05T19:29:33.978 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.974+0000 7f5726a27e40 1 -- 172.21.15.167:0/3631952810 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:83 6.b 6:d467b91b:::bucket.sync-target-hints.:head [call version.read in=11b,read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32) v8 -- 0x55d8e5756ca0 con 0x55d8e5751060 2023-10-05T19:29:33.979 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.974+0000 7f572455c700 1 -- 172.21.15.167:0/3631952810 <== osd.3 v2:172.21.15.167:6808/30036 21 ==== osd_op_reply(83 bucket.sync-target-hints. [call,read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 ==== 211+0+0 (crc 0 0 0) 0x7f571c05f040 con 0x55d8e5751060 2023-10-05T19:29:33.979 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.974+0000 7f5726a27e40 20 rados_obj.operate() r=-2 bl.length=0 2023-10-05T19:29:33.979 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.974+0000 7f5726a27e40 20 started sync module instance, tier type = 2023-10-05T19:29:33.979 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.974+0000 7f5726a27e40 20 started zone id=9a9df965-6612-45d4-a45f-6e3c889bacc2 (name=default) with tier type = 2023-10-05T19:29:33.979 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.974+0000 7f5726a27e40 1 -- 172.21.15.167:0/3631952810 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:84 7.1f 7:f95f44c2:::notify.0:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e32) v8 -- 0x55d8e5755fb0 con 0x7f571c037c00 2023-10-05T19:29:33.981 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.978+0000 7f5723d5b700 1 -- 172.21.15.167:0/3631952810 <== osd.2 v2:172.21.15.167:6816/30037 13 ==== osd_op_reply(84 notify.0 [create] v32'231 uv231 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f570c006070 con 0x7f571c037c00 2023-10-05T19:29:33.981 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.978+0000 7f5726a27e40 1 -- 172.21.15.167:0/3631952810 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:85 7.1f 7:f95f44c2:::notify.0:head [watch watch cookie 94390050972592] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e32) v8 -- 0x55d8e5755be0 con 0x7f571c037c00 2023-10-05T19:29:33.981 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.978+0000 7f5726a27e40 1 -- 172.21.15.167:0/3631952810 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:86 7.0 7:05bf5b68:::notify.1:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e32) v8 -- 0x55d8e5757a70 con 0x55d8e571a7f0 2023-10-05T19:29:33.983 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.978+0000 7f5723d5b700 1 -- 172.21.15.167:0/3631952810 <== osd.2 v2:172.21.15.167:6816/30037 14 ==== osd_op_reply(85 notify.0 [watch watch cookie 94390050972592] v32'232 uv231 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f570c006070 con 0x7f571c037c00 2023-10-05T19:29:33.984 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.978+0000 7f5724d5d700 1 -- 172.21.15.167:0/3631952810 <== osd.1 v2:172.21.15.167:6800/30035 29 ==== osd_op_reply(86 notify.1 [create] v32'231 uv231 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f5714050040 con 0x55d8e571a7f0 2023-10-05T19:29:33.984 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.982+0000 7f5726a27e40 1 -- 172.21.15.167:0/3631952810 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:87 7.0 7:05bf5b68:::notify.1:head [watch watch cookie 94390050975824] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e32) v8 -- 0x55d8e5757e70 con 0x55d8e571a7f0 2023-10-05T19:29:33.984 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.982+0000 7f5726a27e40 1 -- 172.21.15.167:0/3631952810 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:88 7.15 7:a93a5511:::notify.2:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e32) v8 -- 0x55d8e5758710 con 0x55d8e5751060 2023-10-05T19:29:33.987 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.982+0000 7f572455c700 1 -- 172.21.15.167:0/3631952810 <== osd.3 v2:172.21.15.167:6808/30036 22 ==== osd_op_reply(88 notify.2 [create] v32'231 uv231 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f571c05f040 con 0x55d8e5751060 2023-10-05T19:29:33.988 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.982+0000 7f5726a27e40 1 -- 172.21.15.167:0/3631952810 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:89 7.15 7:a93a5511:::notify.2:head [watch watch cookie 94390050979440] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e32) v8 -- 0x55d8e5758fb0 con 0x55d8e5751060 2023-10-05T19:29:33.988 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.982+0000 7f5726a27e40 1 -- 172.21.15.167:0/3631952810 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:90 7.e 7:7759931f:::notify.3:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e32) v8 -- 0x55d8e5759850 con 0x7f571c037c00 2023-10-05T19:29:33.989 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.986+0000 7f5724d5d700 1 -- 172.21.15.167:0/3631952810 <== osd.1 v2:172.21.15.167:6800/30035 30 ==== osd_op_reply(87 notify.1 [watch watch cookie 94390050975824] v32'232 uv231 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f5714050040 con 0x55d8e571a7f0 2023-10-05T19:29:33.990 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.986+0000 7f572455c700 1 -- 172.21.15.167:0/3631952810 <== osd.3 v2:172.21.15.167:6808/30036 23 ==== osd_op_reply(89 notify.2 [watch watch cookie 94390050979440] v32'232 uv231 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f571c05f040 con 0x55d8e5751060 2023-10-05T19:29:33.991 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.986+0000 7f5723d5b700 1 -- 172.21.15.167:0/3631952810 <== osd.2 v2:172.21.15.167:6816/30037 15 ==== osd_op_reply(90 notify.3 [create] v32'231 uv231 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f570c006070 con 0x7f571c037c00 2023-10-05T19:29:33.991 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.986+0000 7f5726a27e40 1 -- 172.21.15.167:0/3631952810 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:91 7.e 7:7759931f:::notify.3:head [watch watch cookie 94390050984800] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e32) v8 -- 0x55d8e5758be0 con 0x7f571c037c00 2023-10-05T19:29:33.991 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.986+0000 7f5726a27e40 1 -- 172.21.15.167:0/3631952810 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:92 7.d 7:b4812045:::notify.4:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e32) v8 -- 0x55d8e575a2f0 con 0x55d8e571a7f0 2023-10-05T19:29:33.993 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.990+0000 7f5723d5b700 1 -- 172.21.15.167:0/3631952810 <== osd.2 v2:172.21.15.167:6816/30037 16 ==== osd_op_reply(91 notify.3 [watch watch cookie 94390050984800] v32'232 uv231 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f570c006070 con 0x7f571c037c00 2023-10-05T19:29:33.994 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.990+0000 7f5724d5d700 1 -- 172.21.15.167:0/3631952810 <== osd.1 v2:172.21.15.167:6800/30035 31 ==== osd_op_reply(92 notify.4 [create] v32'231 uv231 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f5714050040 con 0x55d8e571a7f0 2023-10-05T19:29:33.994 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.990+0000 7f5726a27e40 1 -- 172.21.15.167:0/3631952810 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:93 7.d 7:b4812045:::notify.4:head [watch watch cookie 94390050987520] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e32) v8 -- 0x55d8e575a740 con 0x55d8e571a7f0 2023-10-05T19:29:33.994 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.990+0000 7f5726a27e40 1 -- 172.21.15.167:0/3631952810 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:94 7.3 7:c609908c:::notify.5:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e32) v8 -- 0x55d8e575afe0 con 0x7f570c00d8e0 2023-10-05T19:29:33.996 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.990+0000 7f5724d5d700 1 -- 172.21.15.167:0/3631952810 <== osd.1 v2:172.21.15.167:6800/30035 32 ==== osd_op_reply(93 notify.4 [watch watch cookie 94390050987520] v32'232 uv231 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f5714050040 con 0x55d8e571a7f0 2023-10-05T19:29:33.996 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.990+0000 7f5724d5d700 1 -- 172.21.15.167:0/3631952810 <== osd.0 v2:172.21.15.118:6800/27690 23 ==== osd_op_reply(94 notify.5 [create] v32'231 uv231 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f5714050040 con 0x7f570c00d8e0 2023-10-05T19:29:33.997 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.994+0000 7f5726a27e40 1 -- 172.21.15.167:0/3631952810 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:95 7.3 7:c609908c:::notify.5:head [watch watch cookie 94390050989184] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e32) v8 -- 0x55d8e575adc0 con 0x7f570c00d8e0 2023-10-05T19:29:33.997 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.994+0000 7f5726a27e40 1 -- 172.21.15.167:0/3631952810 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:96 7.14 7:2b04a3e9:::notify.6:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e32) v8 -- 0x55d8e575ba40 con 0x7f570c00d8e0 2023-10-05T19:29:33.999 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.994+0000 7f5724d5d700 1 -- 172.21.15.167:0/3631952810 <== osd.0 v2:172.21.15.118:6800/27690 24 ==== osd_op_reply(96 notify.6 [create] v32'231 uv231 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f5714050040 con 0x7f570c00d8e0 2023-10-05T19:29:33.999 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.994+0000 7f5726a27e40 1 -- 172.21.15.167:0/3631952810 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:97 7.14 7:2b04a3e9:::notify.6:head [watch watch cookie 94390050993488] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e32) v8 -- 0x55d8e575be90 con 0x7f570c00d8e0 2023-10-05T19:29:33.999 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.994+0000 7f5726a27e40 1 -- 172.21.15.167:0/3631952810 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:98 7.9 7:93e5b521:::notify.7:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e32) v8 -- 0x55d8e575c8a0 con 0x7f570c00d8e0 2023-10-05T19:29:34.000 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.994+0000 7f5724d5d700 1 -- 172.21.15.167:0/3631952810 <== osd.0 v2:172.21.15.118:6800/27690 25 ==== osd_op_reply(95 notify.5 [watch watch cookie 94390050989184] v32'232 uv231 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f5714050040 con 0x7f570c00d8e0 2023-10-05T19:29:34.002 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.998+0000 7f5724d5d700 1 -- 172.21.15.167:0/3631952810 <== osd.0 v2:172.21.15.118:6800/27690 26 ==== osd_op_reply(98 notify.7 [create] v32'231 uv231 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f5714050040 con 0x7f570c00d8e0 2023-10-05T19:29:34.002 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.998+0000 7f5726a27e40 1 -- 172.21.15.167:0/3631952810 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:99 7.9 7:93e5b521:::notify.7:head [watch watch cookie 94390050990288] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e32) v8 -- 0x55d8e575c8a0 con 0x7f570c00d8e0 2023-10-05T19:29:34.002 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.998+0000 7f5726a27e40 20 add_watcher() i=0 2023-10-05T19:29:34.002 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.998+0000 7f5726a27e40 20 add_watcher() i=1 2023-10-05T19:29:34.002 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.998+0000 7f5726a27e40 20 add_watcher() i=2 2023-10-05T19:29:34.002 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.998+0000 7f5726a27e40 20 add_watcher() i=3 2023-10-05T19:29:34.002 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.998+0000 7f5726a27e40 20 add_watcher() i=4 2023-10-05T19:29:34.002 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.998+0000 7f5726a27e40 20 add_watcher() i=5 2023-10-05T19:29:34.003 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.998+0000 7f5724d5d700 1 -- 172.21.15.167:0/3631952810 <== osd.0 v2:172.21.15.118:6800/27690 27 ==== osd_op_reply(97 notify.6 [watch watch cookie 94390050993488] v32'232 uv231 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f5714050040 con 0x7f570c00d8e0 2023-10-05T19:29:34.003 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.998+0000 7f5726a27e40 20 add_watcher() i=6 2023-10-05T19:29:34.004 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.998+0000 7f5724d5d700 1 -- 172.21.15.167:0/3631952810 <== osd.0 v2:172.21.15.118:6800/27690 28 ==== osd_op_reply(99 notify.7 [watch watch cookie 94390050990288] v32'232 uv231 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f5714050040 con 0x7f570c00d8e0 2023-10-05T19:29:34.004 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.998+0000 7f5726a27e40 20 add_watcher() i=7 2023-10-05T19:29:34.004 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.998+0000 7f5726a27e40 2 all 8 watchers are set, enabling cache 2023-10-05T19:29:34.004 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:33.998+0000 7f5726a27e40 1 -- 172.21.15.167:0/3631952810 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:100 6.4 6:22d26bf9:::data_loggenerations_metadata:head [call version.check_conds in=74b,call version.read in=11b,read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32) v8 -- 0x55d8e575f8d0 con 0x55d8e571a7f0 2023-10-05T19:29:34.005 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.002+0000 7f5724d5d700 1 -- 172.21.15.167:0/3631952810 <== osd.1 v2:172.21.15.167:6800/30035 33 ==== osd_op_reply(100 data_loggenerations_metadata [call,call out=48b,read 0~28 out=28b] v0'0 uv1 ondisk = 0) v8 ==== 256+0+76 (crc 0 0 0) 0x7f5714050040 con 0x55d8e571a7f0 2023-10-05T19:29:34.005 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.002+0000 7f5726a27e40 1 -- 172.21.15.167:0/3631952810 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:101 6.4 6:22d26bf9:::data_loggenerations_metadata:head [watch watch cookie 94390050997168] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e32) v8 -- 0x55d8e575ccf0 con 0x55d8e571a7f0 2023-10-05T19:29:34.007 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.002+0000 7f5724d5d700 1 -- 172.21.15.167:0/3631952810 <== osd.1 v2:172.21.15.167:6800/30035 34 ==== osd_op_reply(101 data_loggenerations_metadata [watch watch cookie 94390050997168] v32'227 uv1 ondisk = 0) v8 ==== 172+0+0 (crc 0 0 0) 0x7f5714050040 con 0x55d8e571a7f0 2023-10-05T19:29:34.007 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.002+0000 7f56f5fe3700 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2023-10-05T19:29:34.007 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.002+0000 7f5726a27e40 20 check_secure_mon_conn(): auth registy supported: methods=[2] modes=[2,1] 2023-10-05T19:29:34.008 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.002+0000 7f5726a27e40 20 check_secure_mon_conn(): mode 1 is insecure 2023-10-05T19:29:34.008 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.002+0000 7f5726a27e40 1 Processor -- start 2023-10-05T19:29:34.008 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.002+0000 7f5726a27e40 1 -- start start 2023-10-05T19:29:34.008 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.002+0000 7f5726a27e40 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55d8e5769ac0 0x55d8e5769e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-05T19:29:34.008 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.002+0000 7f5726a27e40 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d8e576a3d0 con 0x55d8e5769ac0 2023-10-05T19:29:34.009 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.006+0000 7f5723d5b700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55d8e5769ac0 0x55d8e5769e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-05T19:29:34.009 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.006+0000 7f5723d5b700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55d8e5769ac0 0x55d8e5769e90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.118:3300/0 says I am v2:172.21.15.167:46288/0 (socket says 172.21.15.167:46288) 2023-10-05T19:29:34.009 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.006+0000 7f5723d5b700 1 -- 172.21.15.167:0/2378383556 learned_addr learned my addr 172.21.15.167:0/2378383556 (peer_addr_for_me v2:172.21.15.167:0/0) 2023-10-05T19:29:34.009 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.006+0000 7f5723d5b700 1 -- 172.21.15.167:0/2378383556 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55d8e576a510 con 0x55d8e5769ac0 2023-10-05T19:29:34.010 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.006+0000 7f5723d5b700 1 --2- 172.21.15.167:0/2378383556 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55d8e5769ac0 0x55d8e5769e90 secure :-1 s=READY pgs=600 cs=0 l=1 rev1=1 crypto rx=0x55d8e574edb0 tx=0x7f570c006000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bd4489afc847ac35 server_cookie=0 in_seq=0 out_seq=0 2023-10-05T19:29:34.010 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.006+0000 7f56f57e2700 1 -- 172.21.15.167:0/2378383556 <== mon.0 v2:172.21.15.118:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f570c023040 con 0x55d8e5769ac0 2023-10-05T19:29:34.010 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.006+0000 7f56f57e2700 1 -- 172.21.15.167:0/2378383556 <== mon.0 v2:172.21.15.118:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f570c024040 con 0x55d8e5769ac0 2023-10-05T19:29:34.010 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.006+0000 7f56f57e2700 1 -- 172.21.15.167:0/2378383556 <== mon.0 v2:172.21.15.118:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f570c013040 con 0x55d8e5769ac0 2023-10-05T19:29:34.011 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.006+0000 7f5726a27e40 1 -- 172.21.15.167:0/2378383556 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55d8e5769ac0 msgr2=0x55d8e5769e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-05T19:29:34.011 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.006+0000 7f5726a27e40 1 --2- 172.21.15.167:0/2378383556 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55d8e5769ac0 0x55d8e5769e90 secure :-1 s=READY pgs=600 cs=0 l=1 rev1=1 crypto rx=0x55d8e574edb0 tx=0x7f570c006000 comp rx=0 tx=0).stop 2023-10-05T19:29:34.012 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.006+0000 7f5726a27e40 1 -- 172.21.15.167:0/2378383556 shutdown_connections 2023-10-05T19:29:34.012 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.006+0000 7f5726a27e40 1 --2- 172.21.15.167:0/2378383556 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55d8e5769ac0 0x55d8e5769e90 unknown :-1 s=CLOSED pgs=600 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-05T19:29:34.012 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.006+0000 7f5726a27e40 1 -- 172.21.15.167:0/2378383556 >> 172.21.15.167:0/2378383556 conn(0x55d8e5764e80 msgr2=0x55d8e5763030 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-05T19:29:34.012 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.006+0000 7f5726a27e40 1 -- 172.21.15.167:0/2378383556 shutdown_connections 2023-10-05T19:29:34.012 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.006+0000 7f5726a27e40 1 -- 172.21.15.167:0/2378383556 wait complete. 2023-10-05T19:29:34.013 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.006+0000 7f5726a27e40 1 Processor -- start 2023-10-05T19:29:34.013 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.006+0000 7f5726a27e40 1 -- start start 2023-10-05T19:29:34.013 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.010+0000 7f5726a27e40 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55d8e5769ac0 0x55d8e576f580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-05T19:29:34.013 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.010+0000 7f5726a27e40 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d8e5770580 con 0x55d8e5769ac0 2023-10-05T19:29:34.013 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.010+0000 7f5723d5b700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55d8e5769ac0 0x55d8e576f580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-05T19:29:34.013 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.010+0000 7f5723d5b700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55d8e5769ac0 0x55d8e576f580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.118:3300/0 says I am v2:172.21.15.167:46300/0 (socket says 172.21.15.167:46300) 2023-10-05T19:29:34.013 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.010+0000 7f5723d5b700 1 -- 172.21.15.167:0/1194590054 learned_addr learned my addr 172.21.15.167:0/1194590054 (peer_addr_for_me v2:172.21.15.167:0/0) 2023-10-05T19:29:34.014 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.010+0000 7f5723d5b700 1 -- 172.21.15.167:0/1194590054 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f570c00ece0 con 0x55d8e5769ac0 2023-10-05T19:29:34.014 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.010+0000 7f5723d5b700 1 --2- 172.21.15.167:0/1194590054 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55d8e5769ac0 0x55d8e576f580 secure :-1 s=READY pgs=601 cs=0 l=1 rev1=1 crypto rx=0x55d8e574eb50 tx=0x7f570c030000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-05T19:29:34.015 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.010+0000 7f566f7fe700 1 -- 172.21.15.167:0/1194590054 <== mon.0 v2:172.21.15.118:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f570c023040 con 0x55d8e5769ac0 2023-10-05T19:29:34.015 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.010+0000 7f566f7fe700 1 -- 172.21.15.167:0/1194590054 <== mon.0 v2:172.21.15.118:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f570c01e570 con 0x55d8e5769ac0 2023-10-05T19:29:34.015 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.010+0000 7f5726a27e40 1 -- 172.21.15.167:0/1194590054 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x55d8e576fb80 con 0x55d8e5769ac0 2023-10-05T19:29:34.015 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.010+0000 7f566f7fe700 1 -- 172.21.15.167:0/1194590054 <== mon.0 v2:172.21.15.118:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f570c01e720 con 0x55d8e5769ac0 2023-10-05T19:29:34.015 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.010+0000 7f5726a27e40 1 -- 172.21.15.167:0/1194590054 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55d8e576ffc0 con 0x55d8e5769ac0 2023-10-05T19:29:34.015 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.010+0000 7f566f7fe700 1 -- 172.21.15.167:0/1194590054 <== mon.0 v2:172.21.15.118:3300/0 4 ==== mgrmap(e 5) v1 ==== 40329+0+0 (secure 0 0 0) 0x7f570c01e880 con 0x55d8e5769ac0 2023-10-05T19:29:34.016 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.010+0000 7f566f7fe700 1 --2- 172.21.15.167:0/1194590054 >> [v2:172.21.15.167:6824/29889,v1:172.21.15.167:6825/29889] conn(0x7f5664030c50 0x7f5664033100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-05T19:29:34.016 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.010+0000 7f566f7fe700 1 -- 172.21.15.167:0/1194590054 <== mon.0 v2:172.21.15.118:3300/0 5 ==== osd_map(32..32 src has 1..32) v4 ==== 6260+0+0 (secure 0 0 0) 0x7f570c04fd30 con 0x55d8e5769ac0 2023-10-05T19:29:34.016 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.010+0000 7f5724d5d700 1 --2- 172.21.15.167:0/1194590054 >> [v2:172.21.15.167:6824/29889,v1:172.21.15.167:6825/29889] conn(0x7f5664030c50 0x7f5664033100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-05T19:29:34.016 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.014+0000 7f5726a27e40 5 note: GC not initialized 2023-10-05T19:29:34.017 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.014+0000 7f5724d5d700 1 --2- 172.21.15.167:0/1194590054 >> [v2:172.21.15.167:6824/29889,v1:172.21.15.167:6825/29889] conn(0x7f5664030c50 0x7f5664033100 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x55d8e575bbe0 tx=0x7f5714000f90 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-05T19:29:34.019 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.014+0000 7f566cff9700 20 reqs_thread_entry: start 2023-10-05T19:29:34.019 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.014+0000 7f5726a27e40 20 init_complete bucket index max shards: 11 2023-10-05T19:29:34.019 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.014+0000 7f56627fc700 20 reqs_thread_entry: start 2023-10-05T19:29:34.020 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.014+0000 7f5726a27e40 10 rgw notify: Started notification manager with: 1 workers 2023-10-05T19:29:34.020 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.014+0000 7f5660ff9700 20 rgw notify: INFO: next queues processing will happen at: Thu Oct 5 19:30:04 2023 2023-10-05T19:29:34.020 INFO:teuthology.orchestra.run.smithi167.stderr: 2023-10-05T19:29:34.020 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.014+0000 7f564ffff700 20 reqs_thread_entry: start 2023-10-05T19:29:34.020 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.014+0000 7f5726a27e40 20 get_system_obj_state: rctx=0x7ffdf52bdfe8 obj=default.rgw.meta:users.uid:foo state=0x55d8e55f30b0 s->prefetch_data=0 2023-10-05T19:29:34.020 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.014+0000 7f5726a27e40 1 -- 172.21.15.167:0/3631952810 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:102 8.18 8:1e6ca5ed:users.uid::foo:head [call version.read in=11b,getxattrs,stat] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32) v8 -- 0x55d8e57d7cb0 con 0x55d8e571a7f0 2023-10-05T19:29:34.021 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.018+0000 7f5724d5d700 1 -- 172.21.15.167:0/3631952810 <== osd.1 v2:172.21.15.167:6800/30035 35 ==== osd_op_reply(102 foo [call out=48b,getxattrs out=75b,stat out=16b] v0'0 uv15 ondisk = 0) v8 ==== 231+0+139 (crc 0 0 0) 0x7f5714050040 con 0x55d8e571a7f0 2023-10-05T19:29:34.021 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.018+0000 7f5726a27e40 20 get_system_obj_state: s->obj_tag was set empty 2023-10-05T19:29:34.021 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.018+0000 7f5726a27e40 20 Read xattr: user.rgw.idtag 2023-10-05T19:29:34.022 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.018+0000 7f5726a27e40 20 rados->read ofs=0 len=0 2023-10-05T19:29:34.022 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.018+0000 7f5726a27e40 1 -- 172.21.15.167:0/3631952810 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:103 8.18 8:1e6ca5ed:users.uid::foo:head [call version.check_conds in=122b,call version.read in=11b,read 0~0,getxattrs] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32) v8 -- 0x55d8e57d7ca0 con 0x55d8e571a7f0 2023-10-05T19:29:34.022 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.018+0000 7f5724d5d700 1 -- 172.21.15.167:0/3631952810 <== osd.1 v2:172.21.15.167:6800/30035 36 ==== osd_op_reply(103 foo [call,call out=48b,read 0~357 out=357b,getxattrs out=75b] v0'0 uv15 ondisk = 0) v8 ==== 273+0+480 (crc 0 0 0) 0x7f5714050040 con 0x55d8e571a7f0 2023-10-05T19:29:34.022 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.018+0000 7f5726a27e40 20 rados_obj.operate() r=0 bl.length=357 2023-10-05T19:29:34.023 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.018+0000 7f5726a27e40 1 -- 172.21.15.167:0/3631952810 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:104 6.13 6:cea599e4:usage::usage.10:head [call rgw.user_usage_log_read in=63b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32) v8 -- 0x55d8e57d9730 con 0x7f570c00d8e0 2023-10-05T19:29:34.023 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.018+0000 7f5724d5d700 1 -- 172.21.15.167:0/3631952810 <== osd.0 v2:172.21.15.118:6800/27690 29 ==== osd_op_reply(104 usage.10 [call out=1268b] v0'0 uv30 ondisk = 0) v8 ==== 152+0+1268 (crc 0 0 0) 0x7f5714050040 con 0x7f570c00d8e0 2023-10-05T19:29:34.024 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.018+0000 7f5726a27e40 1 -- 172.21.15.167:0/3631952810 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:105 7.1f 7:f95f44c2:::notify.0:head [watch unwatch cookie 94390050972592] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e32) v8 -- 0x55d8e57960a0 con 0x7f571c037c00 2023-10-05T19:29:34.027 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.022+0000 7f5723d5b700 1 -- 172.21.15.167:0/3631952810 <== osd.2 v2:172.21.15.167:6816/30037 17 ==== osd_op_reply(105 notify.0 [watch unwatch cookie 94390050972592] v32'233 uv231 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f570c006070 con 0x7f571c037c00 2023-10-05T19:29:34.027 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.022+0000 7f5726a27e40 20 remove_watcher() i=0 2023-10-05T19:29:34.027 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.022+0000 7f5726a27e40 2 removed watcher, disabling cache 2023-10-05T19:29:34.027 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.022+0000 7f5726a27e40 1 -- 172.21.15.167:0/3631952810 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:106 7.0 7:05bf5b68:::notify.1:head [watch unwatch cookie 94390050975824] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e32) v8 -- 0x55d8e57563b0 con 0x55d8e571a7f0 2023-10-05T19:29:34.030 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.026+0000 7f5724d5d700 1 -- 172.21.15.167:0/3631952810 <== osd.1 v2:172.21.15.167:6800/30035 37 ==== osd_op_reply(106 notify.1 [watch unwatch cookie 94390050975824] v32'233 uv231 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f5714050040 con 0x55d8e571a7f0 2023-10-05T19:29:34.030 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.026+0000 7f5726a27e40 20 remove_watcher() i=1 2023-10-05T19:29:34.030 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.026+0000 7f5726a27e40 1 -- 172.21.15.167:0/3631952810 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:107 7.15 7:a93a5511:::notify.2:head [watch unwatch cookie 94390050979440] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e32) v8 -- 0x55d8e57563b0 con 0x55d8e5751060 2023-10-05T19:29:34.033 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.030+0000 7f572455c700 1 -- 172.21.15.167:0/3631952810 <== osd.3 v2:172.21.15.167:6808/30036 24 ==== osd_op_reply(107 notify.2 [watch unwatch cookie 94390050979440] v32'233 uv231 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f571c05f040 con 0x55d8e5751060 2023-10-05T19:29:34.033 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.030+0000 7f5726a27e40 20 remove_watcher() i=2 2023-10-05T19:29:34.033 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.030+0000 7f5726a27e40 1 -- 172.21.15.167:0/3631952810 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:108 7.e 7:7759931f:::notify.3:head [watch unwatch cookie 94390050984800] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e32) v8 -- 0x55d8e5757e70 con 0x7f571c037c00 2023-10-05T19:29:34.035 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.030+0000 7f5723d5b700 1 -- 172.21.15.167:0/3631952810 <== osd.2 v2:172.21.15.167:6816/30037 18 ==== osd_op_reply(108 notify.3 [watch unwatch cookie 94390050984800] v32'233 uv231 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f570c006070 con 0x7f571c037c00 2023-10-05T19:29:34.035 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.030+0000 7f5726a27e40 20 remove_watcher() i=3 2023-10-05T19:29:34.035 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.030+0000 7f5726a27e40 1 -- 172.21.15.167:0/3631952810 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:109 7.d 7:b4812045:::notify.4:head [watch unwatch cookie 94390050987520] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e32) v8 -- 0x55d8e5757e70 con 0x55d8e571a7f0 2023-10-05T19:29:34.038 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.034+0000 7f5724d5d700 1 -- 172.21.15.167:0/3631952810 <== osd.1 v2:172.21.15.167:6800/30035 38 ==== osd_op_reply(109 notify.4 [watch unwatch cookie 94390050987520] v32'233 uv231 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f5714050040 con 0x55d8e571a7f0 2023-10-05T19:29:34.038 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.034+0000 7f5726a27e40 20 remove_watcher() i=4 2023-10-05T19:29:34.038 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.034+0000 7f5726a27e40 1 -- 172.21.15.167:0/3631952810 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:110 7.3 7:c609908c:::notify.5:head [watch unwatch cookie 94390050989184] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e32) v8 -- 0x55d8e5759e00 con 0x7f570c00d8e0 2023-10-05T19:29:34.041 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.038+0000 7f5724d5d700 1 -- 172.21.15.167:0/3631952810 <== osd.0 v2:172.21.15.118:6800/27690 30 ==== osd_op_reply(110 notify.5 [watch unwatch cookie 94390050989184] v32'233 uv231 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f5714050040 con 0x7f570c00d8e0 2023-10-05T19:29:34.041 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.038+0000 7f5726a27e40 20 remove_watcher() i=5 2023-10-05T19:29:34.041 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.038+0000 7f5726a27e40 1 -- 172.21.15.167:0/3631952810 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:111 7.14 7:2b04a3e9:::notify.6:head [watch unwatch cookie 94390050993488] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e32) v8 -- 0x55d8e5757e70 con 0x7f570c00d8e0 2023-10-05T19:29:34.044 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.038+0000 7f5724d5d700 1 -- 172.21.15.167:0/3631952810 <== osd.0 v2:172.21.15.118:6800/27690 31 ==== osd_op_reply(111 notify.6 [watch unwatch cookie 94390050993488] v32'233 uv231 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f5714050040 con 0x7f570c00d8e0 2023-10-05T19:29:34.044 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.038+0000 7f5726a27e40 20 remove_watcher() i=6 2023-10-05T19:29:34.044 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.038+0000 7f5726a27e40 1 -- 172.21.15.167:0/3631952810 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:112 7.9 7:93e5b521:::notify.7:head [watch unwatch cookie 94390050990288] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e32) v8 -- 0x55d8e5757e70 con 0x7f570c00d8e0 2023-10-05T19:29:34.047 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.042+0000 7f5724d5d700 1 -- 172.21.15.167:0/3631952810 <== osd.0 v2:172.21.15.118:6800/27690 32 ==== osd_op_reply(112 notify.7 [watch unwatch cookie 94390050990288] v32'233 uv231 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f5714050040 con 0x7f570c00d8e0 2023-10-05T19:29:34.047 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.042+0000 7f5726a27e40 20 remove_watcher() i=7 2023-10-05T19:29:34.049 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.046+0000 7f56f5fe3700 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2023-10-05T19:29:34.049 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.046+0000 7f5726a27e40 1 -- 172.21.15.167:0/3631952810 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:113 6.4 6:22d26bf9:::data_loggenerations_metadata:head [watch unwatch cookie 94390050997168] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e32) v8 -- 0x55d8e5757e70 con 0x55d8e571a7f0 2023-10-05T19:29:34.053 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.046+0000 7f5724d5d700 1 -- 172.21.15.167:0/3631952810 <== osd.1 v2:172.21.15.167:6800/30035 39 ==== osd_op_reply(113 data_loggenerations_metadata [watch unwatch cookie 94390050997168] v32'228 uv1 ondisk = 0) v8 ==== 172+0+0 (crc 0 0 0) 0x7f5714050040 con 0x55d8e571a7f0 2023-10-05T19:29:34.053 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.046+0000 7f5726a27e40 1 -- 172.21.15.167:0/3631952810 >> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] conn(0x7f570c00d8e0 msgr2=0x7f570c00c360 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-05T19:29:34.053 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.046+0000 7f5726a27e40 1 --2- 172.21.15.167:0/3631952810 >> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] conn(0x7f570c00d8e0 0x7f570c00c360 crc :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-05T19:29:34.053 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.046+0000 7f5726a27e40 1 -- 172.21.15.167:0/3631952810 >> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] conn(0x55d8e571a7f0 msgr2=0x55d8e574d610 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-05T19:29:34.053 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.046+0000 7f5726a27e40 1 --2- 172.21.15.167:0/3631952810 >> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] conn(0x55d8e571a7f0 0x55d8e574d610 crc :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-05T19:29:34.053 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.046+0000 7f5726a27e40 1 -- 172.21.15.167:0/3631952810 >> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] conn(0x7f571c037c00 msgr2=0x7f571c004620 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-05T19:29:34.053 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.046+0000 7f5726a27e40 1 --2- 172.21.15.167:0/3631952810 >> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] conn(0x7f571c037c00 0x7f571c004620 crc :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-05T19:29:34.053 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.046+0000 7f5726a27e40 1 -- 172.21.15.167:0/3631952810 >> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] conn(0x55d8e5751060 msgr2=0x55d8e5753440 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-05T19:29:34.054 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.046+0000 7f5726a27e40 1 --2- 172.21.15.167:0/3631952810 >> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] conn(0x55d8e5751060 0x55d8e5753440 crc :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-05T19:29:34.054 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.050+0000 7f5726a27e40 1 -- 172.21.15.167:0/3631952810 >> [v2:172.21.15.167:6824/29889,v1:172.21.15.167:6825/29889] conn(0x7f5704030c50 msgr2=0x7f5704033100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-05T19:29:34.054 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.050+0000 7f5726a27e40 1 --2- 172.21.15.167:0/3631952810 >> [v2:172.21.15.167:6824/29889,v1:172.21.15.167:6825/29889] conn(0x7f5704030c50 0x7f5704033100 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7f570c0011e0 tx=0x7f570c0013a0 comp rx=0 tx=0).stop 2023-10-05T19:29:34.054 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.050+0000 7f5726a27e40 1 -- 172.21.15.167:0/3631952810 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55d8e571a420 msgr2=0x55d8e5741bb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-05T19:29:34.054 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.050+0000 7f5724d5d700 1 -- 172.21.15.167:0/3631952810 reap_dead start 2023-10-05T19:29:34.054 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.050+0000 7f5726a27e40 1 --2- 172.21.15.167:0/3631952810 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55d8e571a420 0x55d8e5741bb0 secure :-1 s=READY pgs=599 cs=0 l=1 rev1=1 crypto rx=0x7f571c00a470 tx=0x7f571c01d000 comp rx=0 tx=0).stop 2023-10-05T19:29:34.054 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.050+0000 7f5726a27e40 1 -- 172.21.15.167:0/3631952810 shutdown_connections 2023-10-05T19:29:34.054 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.050+0000 7f5726a27e40 1 --2- 172.21.15.167:0/3631952810 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55d8e571a420 0x55d8e5741bb0 unknown :-1 s=CLOSED pgs=599 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-05T19:29:34.054 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.050+0000 7f5726a27e40 1 -- 172.21.15.167:0/3631952810 >> 172.21.15.167:0/3631952810 conn(0x55d8e5714730 msgr2=0x55d8e5735590 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-05T19:29:34.054 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.050+0000 7f5726a27e40 1 -- 172.21.15.167:0/3631952810 shutdown_connections 2023-10-05T19:29:34.054 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.050+0000 7f5726a27e40 1 -- 172.21.15.167:0/3631952810 wait complete. 2023-10-05T19:29:34.055 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.050+0000 7f5726a27e40 1 -- 172.21.15.167:0/1194590054 >> [v2:172.21.15.167:6824/29889,v1:172.21.15.167:6825/29889] conn(0x7f5664030c50 msgr2=0x7f5664033100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-05T19:29:34.056 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.050+0000 7f5726a27e40 1 --2- 172.21.15.167:0/1194590054 >> [v2:172.21.15.167:6824/29889,v1:172.21.15.167:6825/29889] conn(0x7f5664030c50 0x7f5664033100 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x55d8e575bbe0 tx=0x7f5714000f90 comp rx=0 tx=0).stop 2023-10-05T19:29:34.056 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.050+0000 7f5726a27e40 1 -- 172.21.15.167:0/1194590054 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55d8e5769ac0 msgr2=0x55d8e576f580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-05T19:29:34.056 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.050+0000 7f5726a27e40 1 --2- 172.21.15.167:0/1194590054 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55d8e5769ac0 0x55d8e576f580 secure :-1 s=READY pgs=601 cs=0 l=1 rev1=1 crypto rx=0x55d8e574eb50 tx=0x7f570c030000 comp rx=0 tx=0).stop 2023-10-05T19:29:34.056 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.050+0000 7f5726a27e40 1 -- 172.21.15.167:0/1194590054 shutdown_connections 2023-10-05T19:29:34.056 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.050+0000 7f5726a27e40 1 --2- 172.21.15.167:0/1194590054 >> [v2:172.21.15.167:6824/29889,v1:172.21.15.167:6825/29889] conn(0x7f5664030c50 0x7f5664033100 unknown :-1 s=CLOSED pgs=207 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-05T19:29:34.056 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.050+0000 7f5726a27e40 1 --2- 172.21.15.167:0/1194590054 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55d8e5769ac0 0x55d8e576f580 unknown :-1 s=CLOSED pgs=601 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-05T19:29:34.056 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.050+0000 7f5726a27e40 1 -- 172.21.15.167:0/1194590054 >> 172.21.15.167:0/1194590054 conn(0x55d8e5764e80 msgr2=0x55d8e5762f80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-05T19:29:34.056 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.050+0000 7f5726a27e40 1 -- 172.21.15.167:0/1194590054 shutdown_connections 2023-10-05T19:29:34.056 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.050+0000 7f5726a27e40 1 -- 172.21.15.167:0/1194590054 wait complete. 2023-10-05T19:29:34.057 INFO:teuthology.orchestra.run.smithi167.stdout:{ 2023-10-05T19:29:34.057 INFO:teuthology.orchestra.run.smithi167.stdout: "entries": [ 2023-10-05T19:29:34.057 INFO:teuthology.orchestra.run.smithi167.stdout: { 2023-10-05T19:29:34.057 INFO:teuthology.orchestra.run.smithi167.stdout: "user": "foo", 2023-10-05T19:29:34.057 INFO:teuthology.orchestra.run.smithi167.stdout: "buckets": [ 2023-10-05T19:29:34.057 INFO:teuthology.orchestra.run.smithi167.stdout: { 2023-10-05T19:29:34.057 INFO:teuthology.orchestra.run.smithi167.stdout: "bucket": "", 2023-10-05T19:29:34.057 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2023-10-05T19:00:00.000000Z", 2023-10-05T19:29:34.058 INFO:teuthology.orchestra.run.smithi167.stdout: "epoch": 1696532400, 2023-10-05T19:29:34.058 INFO:teuthology.orchestra.run.smithi167.stdout: "owner": "foo", 2023-10-05T19:29:34.058 INFO:teuthology.orchestra.run.smithi167.stdout: "categories": [] 2023-10-05T19:29:34.058 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2023-10-05T19:29:34.058 INFO:teuthology.orchestra.run.smithi167.stdout: { 2023-10-05T19:29:34.058 INFO:teuthology.orchestra.run.smithi167.stdout: "bucket": "myfoo", 2023-10-05T19:29:34.058 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2023-10-05T19:00:00.000000Z", 2023-10-05T19:29:34.058 INFO:teuthology.orchestra.run.smithi167.stdout: "epoch": 1696532400, 2023-10-05T19:29:34.058 INFO:teuthology.orchestra.run.smithi167.stdout: "owner": "foo", 2023-10-05T19:29:34.058 INFO:teuthology.orchestra.run.smithi167.stdout: "categories": [ 2023-10-05T19:29:34.058 INFO:teuthology.orchestra.run.smithi167.stdout: { 2023-10-05T19:29:34.058 INFO:teuthology.orchestra.run.smithi167.stdout: "category": "put_obj", 2023-10-05T19:29:34.059 INFO:teuthology.orchestra.run.smithi167.stdout: "bytes_sent": 0, 2023-10-05T19:29:34.059 INFO:teuthology.orchestra.run.smithi167.stdout: "bytes_received": 30000042, 2023-10-05T19:29:34.059 INFO:teuthology.orchestra.run.smithi167.stdout: "ops": 11, 2023-10-05T19:29:34.059 INFO:teuthology.orchestra.run.smithi167.stdout: "successful_ops": 11 2023-10-05T19:29:34.059 INFO:teuthology.orchestra.run.smithi167.stdout: } 2023-10-05T19:29:34.059 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2023-10-05T19:29:34.059 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2023-10-05T19:29:34.059 INFO:teuthology.orchestra.run.smithi167.stdout: { 2023-10-05T19:29:34.059 INFO:teuthology.orchestra.run.smithi167.stdout: "bucket": "myfoo2", 2023-10-05T19:29:34.059 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2023-10-05T19:00:00.000000Z", 2023-10-05T19:29:34.059 INFO:teuthology.orchestra.run.smithi167.stdout: "epoch": 1696532400, 2023-10-05T19:29:34.059 INFO:teuthology.orchestra.run.smithi167.stdout: "owner": "foo", 2023-10-05T19:29:34.060 INFO:teuthology.orchestra.run.smithi167.stdout: "categories": [] 2023-10-05T19:29:34.060 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2023-10-05T19:29:34.060 INFO:teuthology.orchestra.run.smithi167.stdout: { 2023-10-05T19:29:34.060 INFO:teuthology.orchestra.run.smithi167.stdout: "bucket": "myfoo3", 2023-10-05T19:29:34.060 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2023-10-05T19:00:00.000000Z", 2023-10-05T19:29:34.060 INFO:teuthology.orchestra.run.smithi167.stdout: "epoch": 1696532400, 2023-10-05T19:29:34.060 INFO:teuthology.orchestra.run.smithi167.stdout: "owner": "foo", 2023-10-05T19:29:34.060 INFO:teuthology.orchestra.run.smithi167.stdout: "categories": [] 2023-10-05T19:29:34.060 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2023-10-05T19:29:34.060 INFO:teuthology.orchestra.run.smithi167.stdout: { 2023-10-05T19:29:34.060 INFO:teuthology.orchestra.run.smithi167.stdout: "bucket": "myfoo4", 2023-10-05T19:29:34.060 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2023-10-05T19:00:00.000000Z", 2023-10-05T19:29:34.060 INFO:teuthology.orchestra.run.smithi167.stdout: "epoch": 1696532400, 2023-10-05T19:29:34.061 INFO:teuthology.orchestra.run.smithi167.stdout: "owner": "foo", 2023-10-05T19:29:34.061 INFO:teuthology.orchestra.run.smithi167.stdout: "categories": [] 2023-10-05T19:29:34.061 INFO:teuthology.orchestra.run.smithi167.stdout: } 2023-10-05T19:29:34.061 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2023-10-05T19:29:34.061 INFO:teuthology.orchestra.run.smithi167.stdout: } 2023-10-05T19:29:34.061 INFO:teuthology.orchestra.run.smithi167.stdout: ], 2023-10-05T19:29:34.061 INFO:teuthology.orchestra.run.smithi167.stdout: "summary": [ 2023-10-05T19:29:34.061 INFO:teuthology.orchestra.run.smithi167.stdout: { 2023-10-05T19:29:34.061 INFO:teuthology.orchestra.run.smithi167.stdout: "user": "foo", 2023-10-05T19:29:34.061 INFO:teuthology.orchestra.run.smithi167.stdout: "categories": [ 2023-10-05T19:29:34.061 INFO:teuthology.orchestra.run.smithi167.stdout: { 2023-10-05T19:29:34.061 INFO:teuthology.orchestra.run.smithi167.stdout: "category": "put_obj", 2023-10-05T19:29:34.062 INFO:teuthology.orchestra.run.smithi167.stdout: "bytes_sent": 0, 2023-10-05T19:29:34.062 INFO:teuthology.orchestra.run.smithi167.stdout: "bytes_received": 30000042, 2023-10-05T19:29:34.062 INFO:teuthology.orchestra.run.smithi167.stdout: "ops": 11, 2023-10-05T19:29:34.062 INFO:teuthology.orchestra.run.smithi167.stdout: "successful_ops": 11 2023-10-05T19:29:34.062 INFO:teuthology.orchestra.run.smithi167.stdout: } 2023-10-05T19:29:34.062 INFO:teuthology.orchestra.run.smithi167.stdout: ], 2023-10-05T19:29:34.062 INFO:teuthology.orchestra.run.smithi167.stdout: "total": { 2023-10-05T19:29:34.062 INFO:teuthology.orchestra.run.smithi167.stdout: "bytes_sent": 0, 2023-10-05T19:29:34.062 INFO:teuthology.orchestra.run.smithi167.stdout: "bytes_received": 30000042, 2023-10-05T19:29:34.062 INFO:teuthology.orchestra.run.smithi167.stdout: "ops": 11, 2023-10-05T19:29:34.062 INFO:teuthology.orchestra.run.smithi167.stdout: "successful_ops": 11 2023-10-05T19:29:34.062 INFO:teuthology.orchestra.run.smithi167.stdout: } 2023-10-05T19:29:34.063 INFO:teuthology.orchestra.run.smithi167.stdout: } 2023-10-05T19:29:34.063 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2023-10-05T19:29:34.063 INFO:teuthology.orchestra.run.smithi167.stdout:} 2023-10-05T19:29:34.064 DEBUG:tasks.util.rgw: json result: {'entries': [{'user': 'foo', 'buckets': [{'bucket': '', 'time': '2023-10-05T19:00:00.000000Z', 'epoch': 1696532400, 'owner': 'foo', 'categories': []}, {'bucket': 'myfoo', 'time': '2023-10-05T19:00:00.000000Z', 'epoch': 1696532400, 'owner': 'foo', 'categories': [{'category': 'put_obj', 'bytes_sent': 0, 'bytes_received': 30000042, 'ops': 11, 'successful_ops': 11}]}, {'bucket': 'myfoo2', 'time': '2023-10-05T19:00:00.000000Z', 'epoch': 1696532400, 'owner': 'foo', 'categories': []}, {'bucket': 'myfoo3', 'time': '2023-10-05T19:00:00.000000Z', 'epoch': 1696532400, 'owner': 'foo', 'categories': []}, {'bucket': 'myfoo4', 'time': '2023-10-05T19:00:00.000000Z', 'epoch': 1696532400, 'owner': 'foo', 'categories': []}]}], 'summary': [{'user': 'foo', 'categories': [{'category': 'put_obj', 'bytes_sent': 0, 'bytes_received': 30000042, 'ops': 11, 'successful_ops': 11}], 'total': {'bytes_sent': 0, 'bytes_received': 30000042, 'ops': 11, 'successful_ops': 11}}]} 2023-10-05T19:29:34.064 INFO:tasks.util.rgw:rgwadmin: client.0 : ['usage', 'show', '--uid', 'foo', '--categories', 'delete_obj'] 2023-10-05T19:29:34.064 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'ceph', 'usage', 'show', '--uid', 'foo', '--categories', 'delete_obj'] 2023-10-05T19:29:34.064 DEBUG:teuthology.orchestra.run.smithi167:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster ceph usage show --uid foo --categories delete_obj 2023-10-05T19:29:34.105 INFO:teuthology.orchestra.run.smithi167.stderr:ignoring --setuser ceph since I am not root 2023-10-05T19:29:34.105 INFO:teuthology.orchestra.run.smithi167.stderr:ignoring --setgroup ceph since I am not root 2023-10-05T19:29:34.105 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.102+0000 7f1ce7539e40 1 Processor -- start 2023-10-05T19:29:34.106 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.102+0000 7f1ce7539e40 1 -- start start 2023-10-05T19:29:34.106 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.102+0000 7f1ce7539e40 1 --2- >> v2:172.21.15.118:3300/0 conn(0x5641f585e600 0x5641f58609e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-05T19:29:34.106 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.102+0000 7f1ce7539e40 1 -- --> v1:172.21.15.118:6789/0 -- auth(proto 0 26 bytes epoch 0) v1 -- 0x5641f57a4bd0 con 0x5641f585a020 2023-10-05T19:29:34.106 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.102+0000 7f1ce7539e40 1 -- --> v2:172.21.15.118:3300/0 -- mon_getmap magic: 0 v1 -- 0x5641f5761ee0 con 0x5641f585e600 2023-10-05T19:29:34.107 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.102+0000 7f1ce486d700 1 --2- >> v2:172.21.15.118:3300/0 conn(0x5641f585e600 0x5641f58609e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-05T19:29:34.107 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.102+0000 7f1ce506e700 1 --1- >> v1:172.21.15.118:6789/0 conn(0x5641f585a020 0x5641f585a3f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.118:6789/0 says I am v1:172.21.15.167:52634/0 (socket says 172.21.15.167:52634) 2023-10-05T19:29:34.107 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.102+0000 7f1ce506e700 1 -- 172.21.15.167:0/218835299 learned_addr learned my addr 172.21.15.167:0/218835299 (peer_addr_for_me v1:172.21.15.167:0/0) 2023-10-05T19:29:34.107 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.102+0000 7f1cdffff700 1 -- 172.21.15.167:0/218835299 <== mon.0 v1:172.21.15.118:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2258570252 0 0) 0x5641f57a4bd0 con 0x5641f585a020 2023-10-05T19:29:34.107 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.102+0000 7f1cdffff700 1 -- 172.21.15.167:0/218835299 --> v1:172.21.15.118:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1cd4001e10 con 0x5641f585a020 2023-10-05T19:29:34.107 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.102+0000 7f1ce486d700 1 -- 172.21.15.167:0/218835299 >> v1:172.21.15.118:6789/0 conn(0x5641f585a020 legacy=0x5641f585a3f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-05T19:29:34.108 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.102+0000 7f1ce486d700 1 -- 172.21.15.167:0/218835299 --> v2:172.21.15.118:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5641f585ac30 con 0x5641f585e600 2023-10-05T19:29:34.109 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.102+0000 7f1ce486d700 1 --2- 172.21.15.167:0/218835299 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5641f585e600 0x5641f58609e0 secure :-1 s=READY pgs=603 cs=0 l=1 rev1=1 crypto rx=0x7f1cd000a520 tx=0x7f1cd0005d40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5c2541cc33720fbb server_cookie=0 in_seq=0 out_seq=0 2023-10-05T19:29:34.109 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.102+0000 7f1cdffff700 1 -- 172.21.15.167:0/218835299 <== mon.0 v2:172.21.15.118:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f1cd000f590 con 0x5641f585e600 2023-10-05T19:29:34.109 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.102+0000 7f1cdffff700 1 -- 172.21.15.167:0/218835299 <== mon.0 v2:172.21.15.118:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1cd000f6f0 con 0x5641f585e600 2023-10-05T19:29:34.109 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.102+0000 7f1cdffff700 1 -- 172.21.15.167:0/218835299 <== mon.0 v2:172.21.15.118:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f1cd000f860 con 0x5641f585e600 2023-10-05T19:29:34.109 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.102+0000 7f1ce7539e40 1 -- 172.21.15.167:0/218835299 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5641f585e600 msgr2=0x5641f58609e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-05T19:29:34.109 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.102+0000 7f1ce7539e40 1 --2- 172.21.15.167:0/218835299 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5641f585e600 0x5641f58609e0 secure :-1 s=READY pgs=603 cs=0 l=1 rev1=1 crypto rx=0x7f1cd000a520 tx=0x7f1cd0005d40 comp rx=0 tx=0).stop 2023-10-05T19:29:34.109 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.102+0000 7f1ce7539e40 1 -- 172.21.15.167:0/218835299 shutdown_connections 2023-10-05T19:29:34.109 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.102+0000 7f1ce7539e40 1 --2- 172.21.15.167:0/218835299 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5641f585e600 0x5641f58609e0 unknown :-1 s=CLOSED pgs=603 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-05T19:29:34.110 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.102+0000 7f1ce7539e40 1 -- 172.21.15.167:0/218835299 >> 172.21.15.167:0/218835299 conn(0x5641f57cd870 msgr2=0x5641f57cdc70 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-05T19:29:34.110 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.102+0000 7f1ce7539e40 1 -- 172.21.15.167:0/218835299 shutdown_connections 2023-10-05T19:29:34.110 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.106+0000 7f1ce7539e40 1 -- 172.21.15.167:0/218835299 wait complete. 2023-10-05T19:29:34.111 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.106+0000 7f1ce7539e40 1 Processor -- start 2023-10-05T19:29:34.111 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.106+0000 7f1ce7539e40 1 -- start start 2023-10-05T19:29:34.111 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.106+0000 7f1ce7539e40 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5641f585a020 0x5641f5879000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-05T19:29:34.111 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.106+0000 7f1ce7539e40 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x5641f5725e30 con 0x5641f585a020 2023-10-05T19:29:34.111 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.106+0000 7f1ce506e700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5641f585a020 0x5641f5879000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-05T19:29:34.111 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.106+0000 7f1ce506e700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5641f585a020 0x5641f5879000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.118:3300/0 says I am v2:172.21.15.167:46318/0 (socket says 172.21.15.167:46318) 2023-10-05T19:29:34.111 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.106+0000 7f1ce506e700 1 -- 172.21.15.167:0/1513949079 learned_addr learned my addr 172.21.15.167:0/1513949079 (peer_addr_for_me v2:172.21.15.167:0/0) 2023-10-05T19:29:34.111 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.106+0000 7f1ce506e700 1 -- 172.21.15.167:0/1513949079 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1cd000a0e0 con 0x5641f585a020 2023-10-05T19:29:34.113 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.106+0000 7f1ce506e700 1 --2- 172.21.15.167:0/1513949079 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5641f585a020 0x5641f5879000 secure :-1 s=READY pgs=604 cs=0 l=1 rev1=1 crypto rx=0x7f1cd8001c30 tx=0x7f1cd8001810 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e2ded9ee9df3c54a server_cookie=0 in_seq=0 out_seq=0 2023-10-05T19:29:34.113 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.106+0000 7f1cdeffd700 1 -- 172.21.15.167:0/1513949079 <== mon.0 v2:172.21.15.118:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f1cd800bbf0 con 0x5641f585a020 2023-10-05T19:29:34.113 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.106+0000 7f1cdeffd700 1 -- 172.21.15.167:0/1513949079 <== mon.0 v2:172.21.15.118:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1cd800bd50 con 0x5641f585a020 2023-10-05T19:29:34.113 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.106+0000 7f1cdeffd700 1 -- 172.21.15.167:0/1513949079 <== mon.0 v2:172.21.15.118:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f1cd800e470 con 0x5641f585a020 2023-10-05T19:29:34.113 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.106+0000 7f1ce7539e40 1 -- 172.21.15.167:0/1513949079 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5641f585a020 msgr2=0x5641f5879000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-05T19:29:34.114 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.106+0000 7f1ce7539e40 1 --2- 172.21.15.167:0/1513949079 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5641f585a020 0x5641f5879000 secure :-1 s=READY pgs=604 cs=0 l=1 rev1=1 crypto rx=0x7f1cd8001c30 tx=0x7f1cd8001810 comp rx=0 tx=0).stop 2023-10-05T19:29:34.114 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.106+0000 7f1ce7539e40 1 -- 172.21.15.167:0/1513949079 shutdown_connections 2023-10-05T19:29:34.114 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.106+0000 7f1ce7539e40 1 --2- 172.21.15.167:0/1513949079 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5641f585a020 0x5641f5879000 unknown :-1 s=CLOSED pgs=604 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-05T19:29:34.114 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.106+0000 7f1ce7539e40 1 -- 172.21.15.167:0/1513949079 >> 172.21.15.167:0/1513949079 conn(0x5641f57cd870 msgr2=0x5641f5874bc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-05T19:29:34.114 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.106+0000 7f1ce7539e40 1 -- 172.21.15.167:0/1513949079 shutdown_connections 2023-10-05T19:29:34.114 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.106+0000 7f1ce7539e40 1 -- 172.21.15.167:0/1513949079 wait complete. 2023-10-05T19:29:34.114 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.110+0000 7f1ce7539e40 1 Processor -- start 2023-10-05T19:29:34.114 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.110+0000 7f1ce7539e40 1 -- start start 2023-10-05T19:29:34.114 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.110+0000 7f1ce7539e40 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5641f585a020 0x5641f587fd90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-05T19:29:34.114 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.110+0000 7f1ce7539e40 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x5641f56f8ed0 con 0x5641f585a020 2023-10-05T19:29:34.115 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.110+0000 7f1ce506e700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5641f585a020 0x5641f587fd90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-05T19:29:34.115 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.110+0000 7f1ce506e700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5641f585a020 0x5641f587fd90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.118:3300/0 says I am v2:172.21.15.167:46326/0 (socket says 172.21.15.167:46326) 2023-10-05T19:29:34.115 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.110+0000 7f1ce506e700 1 -- 172.21.15.167:0/2391133210 learned_addr learned my addr 172.21.15.167:0/2391133210 (peer_addr_for_me v2:172.21.15.167:0/0) 2023-10-05T19:29:34.115 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.110+0000 7f1ce506e700 1 -- 172.21.15.167:0/2391133210 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1cd4001e10 con 0x5641f585a020 2023-10-05T19:29:34.116 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.110+0000 7f1ce506e700 1 --2- 172.21.15.167:0/2391133210 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5641f585a020 0x5641f587fd90 secure :-1 s=READY pgs=605 cs=0 l=1 rev1=1 crypto rx=0x7f1cd800b960 tx=0x7f1cd8014000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-05T19:29:34.116 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.110+0000 7f1cdd7fa700 1 -- 172.21.15.167:0/2391133210 <== mon.0 v2:172.21.15.118:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f1cd801f070 con 0x5641f585a020 2023-10-05T19:29:34.116 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.110+0000 7f1ce7539e40 1 -- 172.21.15.167:0/2391133210 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x5641f58802d0 con 0x5641f585a020 2023-10-05T19:29:34.116 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.110+0000 7f1cdd7fa700 1 -- 172.21.15.167:0/2391133210 <== mon.0 v2:172.21.15.118:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1cd801a2e0 con 0x5641f585a020 2023-10-05T19:29:34.116 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.110+0000 7f1cdd7fa700 1 -- 172.21.15.167:0/2391133210 <== mon.0 v2:172.21.15.118:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f1cd800e470 con 0x5641f585a020 2023-10-05T19:29:34.116 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.110+0000 7f1ce7539e40 1 -- 172.21.15.167:0/2391133210 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x5641f5880710 con 0x5641f585a020 2023-10-05T19:29:34.117 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.110+0000 7f1ce7539e40 20 rados->read ofs=0 len=0 2023-10-05T19:29:34.117 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.114+0000 7f1cdd7fa700 1 -- 172.21.15.167:0/2391133210 <== mon.0 v2:172.21.15.118:3300/0 4 ==== mgrmap(e 5) v1 ==== 40329+0+0 (secure 0 0 0) 0x7f1cd801aa10 con 0x5641f585a020 2023-10-05T19:29:34.117 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.114+0000 7f1cdd7fa700 1 --2- 172.21.15.167:0/2391133210 >> [v2:172.21.15.167:6824/29889,v1:172.21.15.167:6825/29889] conn(0x7f1cc8030c40 0x7f1cc80330f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-05T19:29:34.117 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.114+0000 7f1cdd7fa700 1 -- 172.21.15.167:0/2391133210 <== mon.0 v2:172.21.15.118:3300/0 5 ==== osd_map(32..32 src has 1..32) v4 ==== 6260+0+0 (secure 0 0 0) 0x7f1cd8045340 con 0x5641f585a020 2023-10-05T19:29:34.118 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.114+0000 7f1ce486d700 1 --2- 172.21.15.167:0/2391133210 >> [v2:172.21.15.167:6824/29889,v1:172.21.15.167:6825/29889] conn(0x7f1cc8030c40 0x7f1cc80330f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-05T19:29:34.118 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.114+0000 7f1ce7539e40 1 --2- 172.21.15.167:0/2391133210 >> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] conn(0x5641f585e600 0x5641f588b7f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-05T19:29:34.118 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.114+0000 7f1ce7539e40 1 -- 172.21.15.167:0/2391133210 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:1 5.12 5:49953fa1:::default.realm:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32) v8 -- 0x5641f588be60 con 0x5641f585e600 2023-10-05T19:29:34.118 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.114+0000 7f1ce586f700 1 --2- 172.21.15.167:0/2391133210 >> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] conn(0x5641f585e600 0x5641f588b7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-05T19:29:34.119 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.114+0000 7f1ce486d700 1 --2- 172.21.15.167:0/2391133210 >> [v2:172.21.15.167:6824/29889,v1:172.21.15.167:6825/29889] conn(0x7f1cc8030c40 0x7f1cc80330f0 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7f1cd0000f80 tx=0x7f1cd0014000 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-05T19:29:34.119 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.114+0000 7f1ce586f700 1 --2- 172.21.15.167:0/2391133210 >> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] conn(0x5641f585e600 0x5641f588b7f0 crc :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-05T19:29:34.120 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.114+0000 7f1ce586f700 1 -- 172.21.15.167:0/2391133210 <== osd.1 v2:172.21.15.167:6800/30035 1 ==== osd_op_reply(1 default.realm [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 ==== 157+0+0 (crc 0 0 0) 0x7f1ce000a500 con 0x5641f585e600 2023-10-05T19:29:34.120 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.114+0000 7f1ce7539e40 20 rados_obj.operate() r=-2 bl.length=0 2023-10-05T19:29:34.120 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.114+0000 7f1ce7539e40 20 realm 2023-10-05T19:29:34.120 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.114+0000 7f1ce7539e40 20 rados->read ofs=0 len=0 2023-10-05T19:29:34.120 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.114+0000 7f1ce7539e40 1 -- 172.21.15.167:0/2391133210 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:2 5.12 5:49953fa1:::default.realm:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32) v8 -- 0x5641f5888e50 con 0x5641f585e600 2023-10-05T19:29:34.121 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.114+0000 7f1ce586f700 1 -- 172.21.15.167:0/2391133210 <== osd.1 v2:172.21.15.167:6800/30035 2 ==== osd_op_reply(2 default.realm [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 ==== 157+0+0 (crc 0 0 0) 0x7f1ce000a500 con 0x5641f585e600 2023-10-05T19:29:34.121 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.118+0000 7f1ce7539e40 20 rados_obj.operate() r=-2 bl.length=0 2023-10-05T19:29:34.121 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.118+0000 7f1ce7539e40 4 RGWPeriod::init failed to init realm id : (2) No such file or directory 2023-10-05T19:29:34.121 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.118+0000 7f1ce7539e40 20 rados->read ofs=0 len=0 2023-10-05T19:29:34.121 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.118+0000 7f1ce7539e40 1 -- 172.21.15.167:0/2391133210 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:3 5.12 5:49953fa1:::default.realm:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32) v8 -- 0x5641f588c940 con 0x5641f585e600 2023-10-05T19:29:34.121 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.118+0000 7f1ce586f700 1 -- 172.21.15.167:0/2391133210 <== osd.1 v2:172.21.15.167:6800/30035 3 ==== osd_op_reply(3 default.realm [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 ==== 157+0+0 (crc 0 0 0) 0x7f1ce000a500 con 0x5641f585e600 2023-10-05T19:29:34.122 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.118+0000 7f1ce7539e40 20 rados_obj.operate() r=-2 bl.length=0 2023-10-05T19:29:34.122 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.118+0000 7f1ce7539e40 20 rados->read ofs=0 len=0 2023-10-05T19:29:34.122 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.118+0000 7f1ce7539e40 1 -- 172.21.15.167:0/2391133210 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:4 5.f 5:f43b6ece:::zone_names.default:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32) v8 -- 0x5641f588c520 con 0x5641f585e600 2023-10-05T19:29:34.123 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.118+0000 7f1ce586f700 1 -- 172.21.15.167:0/2391133210 <== osd.1 v2:172.21.15.167:6800/30035 4 ==== osd_op_reply(4 zone_names.default [read 0~46 out=46b] v0'0 uv1 ondisk = 0) v8 ==== 162+0+46 (crc 0 0 0) 0x7f1ce000a500 con 0x5641f585e600 2023-10-05T19:29:34.123 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.118+0000 7f1ce7539e40 20 rados_obj.operate() r=0 bl.length=46 2023-10-05T19:29:34.123 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.118+0000 7f1ce7539e40 20 rados->read ofs=0 len=0 2023-10-05T19:29:34.123 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.118+0000 7f1ce7539e40 1 -- 172.21.15.167:0/2391133210 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:5 5.1d 5:b8374eb4:::zone_info.9a9df965-6612-45d4-a45f-6e3c889bacc2:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32) v8 -- 0x5641f588d3b0 con 0x5641f585e600 2023-10-05T19:29:34.123 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.118+0000 7f1ce586f700 1 -- 172.21.15.167:0/2391133210 <== osd.1 v2:172.21.15.167:6800/30035 5 ==== osd_op_reply(5 zone_info.9a9df965-6612-45d4-a45f-6e3c889bacc2 [read 0~874 out=874b] v0'0 uv2 ondisk = 0) v8 ==== 190+0+874 (crc 0 0 0) 0x7f1ce000a500 con 0x5641f585e600 2023-10-05T19:29:34.123 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.118+0000 7f1ce7539e40 20 rados_obj.operate() r=0 bl.length=874 2023-10-05T19:29:34.124 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.118+0000 7f1ce7539e40 20 searching for the correct realm 2023-10-05T19:29:34.124 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.118+0000 7f1ce7539e40 1 --2- 172.21.15.167:0/2391133210 >> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] conn(0x5641f588f200 0x5641f58915e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-05T19:29:34.124 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.118+0000 7f1ce7539e40 1 -- 172.21.15.167:0/2391133210 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:6 5.0 5:00000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x5641f5891b20 con 0x5641f588f200 2023-10-05T19:29:34.124 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.118+0000 7f1ce506e700 1 --2- 172.21.15.167:0/2391133210 >> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] conn(0x5641f588f200 0x5641f58915e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-05T19:29:34.124 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.118+0000 7f1ce506e700 1 --2- 172.21.15.167:0/2391133210 >> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] conn(0x5641f588f200 0x5641f58915e0 crc :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-05T19:29:34.125 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.118+0000 7f1ce506e700 1 -- 172.21.15.167:0/2391133210 <== osd.3 v2:172.21.15.167:6808/30036 1 ==== osd_op_reply(6 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f1cd8014070 con 0x5641f588f200 2023-10-05T19:29:34.125 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.118+0000 7f1ce506e700 1 --2- 172.21.15.167:0/2391133210 >> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] conn(0x7f1cd802d020 0x7f1cd8024830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-05T19:29:34.125 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.118+0000 7f1ce506e700 1 -- 172.21.15.167:0/2391133210 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:7 5.10 5:08000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x5641f5891b20 con 0x7f1cd802d020 2023-10-05T19:29:34.125 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.122+0000 7f1ce486d700 1 --2- 172.21.15.167:0/2391133210 >> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] conn(0x7f1cd802d020 0x7f1cd8024830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-05T19:29:34.125 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.122+0000 7f1ce486d700 1 --2- 172.21.15.167:0/2391133210 >> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] conn(0x7f1cd802d020 0x7f1cd8024830 crc :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-05T19:29:34.126 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.122+0000 7f1ce486d700 1 -- 172.21.15.167:0/2391133210 <== osd.2 v2:172.21.15.167:6816/30037 1 ==== osd_op_reply(7 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f1cd000fb90 con 0x7f1cd802d020 2023-10-05T19:29:34.126 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.122+0000 7f1ce486d700 1 --2- 172.21.15.167:0/2391133210 >> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] conn(0x7f1cd0029420 0x7f1cd0029890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-05T19:29:34.126 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.122+0000 7f1ce486d700 1 -- 172.21.15.167:0/2391133210 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:8 5.8 5:10000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x5641f5891b20 con 0x7f1cd0029420 2023-10-05T19:29:34.127 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.122+0000 7f1ce586f700 1 --2- 172.21.15.167:0/2391133210 >> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] conn(0x7f1cd0029420 0x7f1cd0029890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-05T19:29:34.127 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.122+0000 7f1ce586f700 1 --2- 172.21.15.167:0/2391133210 >> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] conn(0x7f1cd0029420 0x7f1cd0029890 crc :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-05T19:29:34.128 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.122+0000 7f1ce586f700 1 -- 172.21.15.167:0/2391133210 <== osd.0 v2:172.21.15.118:6800/27690 1 ==== osd_op_reply(8 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f1ce0014040 con 0x7f1cd0029420 2023-10-05T19:29:34.128 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.122+0000 7f1ce586f700 1 -- 172.21.15.167:0/2391133210 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:9 5.18 5:18000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x5641f5891b20 con 0x5641f585e600 2023-10-05T19:29:34.128 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.126+0000 7f1ce586f700 1 -- 172.21.15.167:0/2391133210 <== osd.1 v2:172.21.15.167:6800/30035 6 ==== osd_op_reply(9 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f1ce0014040 con 0x5641f585e600 2023-10-05T19:29:34.129 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.126+0000 7f1ce586f700 1 -- 172.21.15.167:0/2391133210 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:10 5.4 5:20000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x5641f5891b20 con 0x5641f585e600 2023-10-05T19:29:34.129 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.126+0000 7f1ce586f700 1 -- 172.21.15.167:0/2391133210 <== osd.1 v2:172.21.15.167:6800/30035 7 ==== osd_op_reply(10 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f1ce0014040 con 0x5641f585e600 2023-10-05T19:29:34.129 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.126+0000 7f1ce586f700 1 -- 172.21.15.167:0/2391133210 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:11 5.14 5:28000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x5641f5891b20 con 0x7f1cd0029420 2023-10-05T19:29:34.130 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.126+0000 7f1ce586f700 1 -- 172.21.15.167:0/2391133210 <== osd.0 v2:172.21.15.118:6800/27690 2 ==== osd_op_reply(11 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f1ce0014040 con 0x7f1cd0029420 2023-10-05T19:29:34.130 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.126+0000 7f1ce586f700 1 -- 172.21.15.167:0/2391133210 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:12 5.c 5:30000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x5641f5891b20 con 0x7f1cd0029420 2023-10-05T19:29:34.131 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.126+0000 7f1ce586f700 1 -- 172.21.15.167:0/2391133210 <== osd.0 v2:172.21.15.118:6800/27690 3 ==== osd_op_reply(12 [pgnls start_epoch 32 out=112b] v23'1 uv1 ondisk = 1) v8 ==== 144+0+112 (crc 0 0 0) 0x7f1ce0014040 con 0x7f1cd0029420 2023-10-05T19:29:34.131 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.126+0000 7f1ce586f700 1 -- 172.21.15.167:0/2391133210 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:13 5.1c 5:38000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x5641f5891b20 con 0x7f1cd802d020 2023-10-05T19:29:34.132 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.126+0000 7f1ce486d700 1 -- 172.21.15.167:0/2391133210 <== osd.2 v2:172.21.15.167:6816/30037 2 ==== osd_op_reply(13 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f1cd000fb90 con 0x7f1cd802d020 2023-10-05T19:29:34.132 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.126+0000 7f1ce486d700 1 -- 172.21.15.167:0/2391133210 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:14 5.2 5:40000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x5641f5891b20 con 0x5641f588f200 2023-10-05T19:29:34.133 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.130+0000 7f1ce506e700 1 -- 172.21.15.167:0/2391133210 <== osd.3 v2:172.21.15.167:6808/30036 2 ==== osd_op_reply(14 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f1cd8014070 con 0x5641f588f200 2023-10-05T19:29:34.133 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.130+0000 7f1ce506e700 1 -- 172.21.15.167:0/2391133210 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:15 5.12 5:48000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x5641f5891b20 con 0x5641f585e600 2023-10-05T19:29:34.134 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.130+0000 7f1ce586f700 1 -- 172.21.15.167:0/2391133210 <== osd.1 v2:172.21.15.167:6800/30035 8 ==== osd_op_reply(15 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f1ce0014040 con 0x5641f585e600 2023-10-05T19:29:34.134 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.130+0000 7f1ce586f700 1 -- 172.21.15.167:0/2391133210 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:16 5.a 5:50000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x5641f5891b20 con 0x7f1cd0029420 2023-10-05T19:29:34.134 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.130+0000 7f1ce586f700 1 -- 172.21.15.167:0/2391133210 <== osd.0 v2:172.21.15.118:6800/27690 4 ==== osd_op_reply(16 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f1ce0022040 con 0x7f1cd0029420 2023-10-05T19:29:34.135 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.130+0000 7f1ce586f700 1 -- 172.21.15.167:0/2391133210 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:17 5.1a 5:58000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x5641f5891b20 con 0x5641f585e600 2023-10-05T19:29:34.135 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.130+0000 7f1ce586f700 1 -- 172.21.15.167:0/2391133210 <== osd.1 v2:172.21.15.167:6800/30035 9 ==== osd_op_reply(17 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f1ce0022040 con 0x5641f585e600 2023-10-05T19:29:34.135 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.130+0000 7f1ce586f700 1 -- 172.21.15.167:0/2391133210 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:18 5.6 5:60000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x5641f5891b20 con 0x7f1cd802d020 2023-10-05T19:29:34.136 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.130+0000 7f1ce486d700 1 -- 172.21.15.167:0/2391133210 <== osd.2 v2:172.21.15.167:6816/30037 3 ==== osd_op_reply(18 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f1cd000fb90 con 0x7f1cd802d020 2023-10-05T19:29:34.136 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.130+0000 7f1ce486d700 1 -- 172.21.15.167:0/2391133210 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:19 5.16 5:68000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x5641f5891b20 con 0x5641f588f200 2023-10-05T19:29:34.137 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.134+0000 7f1ce506e700 1 -- 172.21.15.167:0/2391133210 <== osd.3 v2:172.21.15.167:6808/30036 3 ==== osd_op_reply(19 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f1cd8014070 con 0x5641f588f200 2023-10-05T19:29:34.137 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.134+0000 7f1ce506e700 1 -- 172.21.15.167:0/2391133210 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:20 5.e 5:70000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x5641f5891b20 con 0x5641f588f200 2023-10-05T19:29:34.138 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.134+0000 7f1ce506e700 1 -- 172.21.15.167:0/2391133210 <== osd.3 v2:172.21.15.167:6808/30036 4 ==== osd_op_reply(20 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f1cd8014070 con 0x5641f588f200 2023-10-05T19:29:34.138 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.134+0000 7f1ce506e700 1 -- 172.21.15.167:0/2391133210 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:21 5.1e 5:78000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x5641f5891b20 con 0x7f1cd0029420 2023-10-05T19:29:34.138 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.134+0000 7f1ce586f700 1 -- 172.21.15.167:0/2391133210 <== osd.0 v2:172.21.15.118:6800/27690 5 ==== osd_op_reply(21 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f1ce0022040 con 0x7f1cd0029420 2023-10-05T19:29:34.139 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.134+0000 7f1ce586f700 1 -- 172.21.15.167:0/2391133210 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:22 5.1 5:80000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x5641f5891b20 con 0x5641f588f200 2023-10-05T19:29:34.139 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.134+0000 7f1ce506e700 1 -- 172.21.15.167:0/2391133210 <== osd.3 v2:172.21.15.167:6808/30036 5 ==== osd_op_reply(22 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f1cd8014070 con 0x5641f588f200 2023-10-05T19:29:34.139 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.134+0000 7f1ce506e700 1 -- 172.21.15.167:0/2391133210 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:23 5.11 5:88000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x5641f5891b20 con 0x5641f588f200 2023-10-05T19:29:34.140 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.134+0000 7f1ce506e700 1 -- 172.21.15.167:0/2391133210 <== osd.3 v2:172.21.15.167:6808/30036 6 ==== osd_op_reply(23 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f1cd8014070 con 0x5641f588f200 2023-10-05T19:29:34.140 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.134+0000 7f1ce506e700 1 -- 172.21.15.167:0/2391133210 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:24 5.9 5:90000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x5641f5891b20 con 0x5641f585e600 2023-10-05T19:29:34.141 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.138+0000 7f1ce586f700 1 -- 172.21.15.167:0/2391133210 <== osd.1 v2:172.21.15.167:6800/30035 10 ==== osd_op_reply(24 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f1ce0022040 con 0x5641f585e600 2023-10-05T19:29:34.141 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.138+0000 7f1ce586f700 1 -- 172.21.15.167:0/2391133210 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:25 5.19 5:98000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x5641f5891b20 con 0x5641f585e600 2023-10-05T19:29:34.142 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.138+0000 7f1ce586f700 1 -- 172.21.15.167:0/2391133210 <== osd.1 v2:172.21.15.167:6800/30035 11 ==== osd_op_reply(25 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f1ce0022040 con 0x5641f585e600 2023-10-05T19:29:34.142 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.138+0000 7f1ce586f700 1 -- 172.21.15.167:0/2391133210 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:26 5.5 5:a0000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x5641f5891b20 con 0x7f1cd0029420 2023-10-05T19:29:34.143 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.138+0000 7f1ce586f700 1 -- 172.21.15.167:0/2391133210 <== osd.0 v2:172.21.15.118:6800/27690 6 ==== osd_op_reply(26 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f1ce0022040 con 0x7f1cd0029420 2023-10-05T19:29:34.143 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.138+0000 7f1ce586f700 1 -- 172.21.15.167:0/2391133210 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:27 5.15 5:a8000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x5641f5891b20 con 0x5641f588f200 2023-10-05T19:29:34.144 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.138+0000 7f1ce506e700 1 -- 172.21.15.167:0/2391133210 <== osd.3 v2:172.21.15.167:6808/30036 7 ==== osd_op_reply(27 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f1cd8014070 con 0x5641f588f200 2023-10-05T19:29:34.144 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.138+0000 7f1ce506e700 1 -- 172.21.15.167:0/2391133210 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:28 5.d 5:b0000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x5641f5891b20 con 0x7f1cd802d020 2023-10-05T19:29:34.145 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.142+0000 7f1ce486d700 1 -- 172.21.15.167:0/2391133210 <== osd.2 v2:172.21.15.167:6816/30037 4 ==== osd_op_reply(28 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f1cd000fb90 con 0x7f1cd802d020 2023-10-05T19:29:34.145 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.142+0000 7f1ce486d700 1 -- 172.21.15.167:0/2391133210 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:29 5.1d 5:b8000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x5641f5891b20 con 0x5641f585e600 2023-10-05T19:29:34.146 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.142+0000 7f1ce586f700 1 -- 172.21.15.167:0/2391133210 <== osd.1 v2:172.21.15.167:6800/30035 12 ==== osd_op_reply(29 [pgnls start_epoch 32 out=107b] v30'2 uv2 ondisk = 1) v8 ==== 144+0+107 (crc 0 0 0) 0x7f1ce0022040 con 0x5641f585e600 2023-10-05T19:29:34.146 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.142+0000 7f1ce586f700 1 -- 172.21.15.167:0/2391133210 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:30 5.3 5:c0000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x5641f5891b20 con 0x7f1cd0029420 2023-10-05T19:29:34.146 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.142+0000 7f1ce586f700 1 -- 172.21.15.167:0/2391133210 <== osd.0 v2:172.21.15.118:6800/27690 7 ==== osd_op_reply(30 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f1ce0022040 con 0x7f1cd0029420 2023-10-05T19:29:34.147 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.142+0000 7f1ce586f700 1 -- 172.21.15.167:0/2391133210 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:31 5.13 5:c8000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x5641f5891b20 con 0x5641f588f200 2023-10-05T19:29:34.148 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.142+0000 7f1ce506e700 1 -- 172.21.15.167:0/2391133210 <== osd.3 v2:172.21.15.167:6808/30036 8 ==== osd_op_reply(31 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f1cd8014070 con 0x5641f588f200 2023-10-05T19:29:34.148 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.142+0000 7f1ce506e700 1 -- 172.21.15.167:0/2391133210 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:32 5.b 5:d0000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x5641f5891b20 con 0x7f1cd0029420 2023-10-05T19:29:34.148 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.146+0000 7f1ce586f700 1 -- 172.21.15.167:0/2391133210 <== osd.0 v2:172.21.15.118:6800/27690 8 ==== osd_op_reply(32 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f1ce0022040 con 0x7f1cd0029420 2023-10-05T19:29:34.148 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.146+0000 7f1ce586f700 1 -- 172.21.15.167:0/2391133210 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:33 5.1b 5:d8000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x5641f5891b20 con 0x7f1cd802d020 2023-10-05T19:29:34.149 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.146+0000 7f1ce486d700 1 -- 172.21.15.167:0/2391133210 <== osd.2 v2:172.21.15.167:6816/30037 5 ==== osd_op_reply(33 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f1cd000fb90 con 0x7f1cd802d020 2023-10-05T19:29:34.150 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.146+0000 7f1ce486d700 1 -- 172.21.15.167:0/2391133210 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:34 5.7 5:e0000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x5641f5891b20 con 0x5641f588f200 2023-10-05T19:29:34.150 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.146+0000 7f1ce506e700 1 -- 172.21.15.167:0/2391133210 <== osd.3 v2:172.21.15.167:6808/30036 9 ==== osd_op_reply(34 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f1cd8014070 con 0x5641f588f200 2023-10-05T19:29:34.150 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.146+0000 7f1ce506e700 1 -- 172.21.15.167:0/2391133210 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:35 5.17 5:e8000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x5641f5891b20 con 0x7f1cd0029420 2023-10-05T19:29:34.152 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.146+0000 7f1ce586f700 1 -- 172.21.15.167:0/2391133210 <== osd.0 v2:172.21.15.118:6800/27690 9 ==== osd_op_reply(35 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f1ce0022040 con 0x7f1cd0029420 2023-10-05T19:29:34.152 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.146+0000 7f1ce586f700 1 -- 172.21.15.167:0/2391133210 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:36 5.f 5:f0000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x5641f5891b20 con 0x5641f585e600 2023-10-05T19:29:34.153 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.150+0000 7f1ce586f700 1 -- 172.21.15.167:0/2391133210 <== osd.1 v2:172.21.15.167:6800/30035 13 ==== osd_op_reply(36 [pgnls start_epoch 32 out=115b] v23'2 uv2 ondisk = 1) v8 ==== 144+0+115 (crc 0 0 0) 0x7f1ce002f080 con 0x5641f585e600 2023-10-05T19:29:34.153 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.150+0000 7f1ce586f700 1 -- 172.21.15.167:0/2391133210 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:37 5.1f 5:f8000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x5641f5891b20 con 0x7f1cd802d020 2023-10-05T19:29:34.154 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.150+0000 7f1ce486d700 1 -- 172.21.15.167:0/2391133210 <== osd.2 v2:172.21.15.167:6816/30037 6 ==== osd_op_reply(37 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f1cd000fb90 con 0x7f1cd802d020 2023-10-05T19:29:34.154 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.150+0000 7f1ce7539e40 20 RGWRados::pool_iterate: got zonegroup_info.4f1f719d-9648-417b-b10c-e5d939825419 2023-10-05T19:29:34.154 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.150+0000 7f1ce7539e40 20 RGWRados::pool_iterate: got zone_info.9a9df965-6612-45d4-a45f-6e3c889bacc2 2023-10-05T19:29:34.154 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.150+0000 7f1ce7539e40 20 RGWRados::pool_iterate: got zone_names.default 2023-10-05T19:29:34.154 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.150+0000 7f1ce7539e40 20 RGWRados::pool_iterate: got zonegroups_names.default 2023-10-05T19:29:34.154 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.150+0000 7f1ce7539e40 20 rados->read ofs=0 len=0 2023-10-05T19:29:34.154 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.150+0000 7f1ce7539e40 1 -- 172.21.15.167:0/2391133210 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:38 5.12 5:49953fa1:::default.realm:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32) v8 -- 0x5641f588dfb0 con 0x5641f585e600 2023-10-05T19:29:34.155 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.150+0000 7f1ce586f700 1 -- 172.21.15.167:0/2391133210 <== osd.1 v2:172.21.15.167:6800/30035 14 ==== osd_op_reply(38 default.realm [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 ==== 157+0+0 (crc 0 0 0) 0x7f1ce002f080 con 0x5641f585e600 2023-10-05T19:29:34.155 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.150+0000 7f1ce7539e40 20 rados_obj.operate() r=-2 bl.length=0 2023-10-05T19:29:34.155 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.150+0000 7f1ce7539e40 20 rados->read ofs=0 len=0 2023-10-05T19:29:34.156 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.150+0000 7f1ce7539e40 1 -- 172.21.15.167:0/2391133210 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:39 5.f 5:f4c53578:::zonegroups_names.default:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32) v8 -- 0x5641f588e930 con 0x5641f585e600 2023-10-05T19:29:34.156 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.150+0000 7f1ce586f700 1 -- 172.21.15.167:0/2391133210 <== osd.1 v2:172.21.15.167:6800/30035 15 ==== osd_op_reply(39 zonegroups_names.default [read 0~46 out=46b] v0'0 uv2 ondisk = 0) v8 ==== 168+0+46 (crc 0 0 0) 0x7f1ce002f080 con 0x5641f585e600 2023-10-05T19:29:34.156 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.150+0000 7f1ce7539e40 20 rados_obj.operate() r=0 bl.length=46 2023-10-05T19:29:34.156 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.150+0000 7f1ce7539e40 20 rados->read ofs=0 len=0 2023-10-05T19:29:34.156 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.150+0000 7f1ce7539e40 1 -- 172.21.15.167:0/2391133210 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:40 5.c 5:30d299a5:::zonegroup_info.4f1f719d-9648-417b-b10c-e5d939825419:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32) v8 -- 0x5641f588dba0 con 0x7f1cd0029420 2023-10-05T19:29:34.157 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.150+0000 7f1ce586f700 1 -- 172.21.15.167:0/2391133210 <== osd.0 v2:172.21.15.118:6800/27690 10 ==== osd_op_reply(40 zonegroup_info.4f1f719d-9648-417b-b10c-e5d939825419 [read 0~362 out=362b] v0'0 uv1 ondisk = 0) v8 ==== 195+0+362 (crc 0 0 0) 0x7f1ce002f080 con 0x7f1cd0029420 2023-10-05T19:29:34.157 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.150+0000 7f1ce7539e40 20 rados_obj.operate() r=0 bl.length=362 2023-10-05T19:29:34.157 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.150+0000 7f1ce7539e40 20 rados->read ofs=0 len=0 2023-10-05T19:29:34.157 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.150+0000 7f1ce7539e40 1 -- 172.21.15.167:0/2391133210 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:41 5.13 5:cbbf6c60:::converted:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32) v8 -- 0x5641f58923c0 con 0x5641f588f200 2023-10-05T19:29:34.157 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.154+0000 7f1ce506e700 1 -- 172.21.15.167:0/2391133210 <== osd.3 v2:172.21.15.167:6808/30036 10 ==== osd_op_reply(41 converted [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 ==== 153+0+0 (crc 0 0 0) 0x7f1cd8014070 con 0x5641f588f200 2023-10-05T19:29:34.157 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.154+0000 7f1ce7539e40 20 rados_obj.operate() r=-2 bl.length=0 2023-10-05T19:29:34.157 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.154+0000 7f1ce7539e40 20 rados->read ofs=0 len=0 2023-10-05T19:29:34.157 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.154+0000 7f1ce7539e40 1 -- 172.21.15.167:0/2391133210 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:42 5.12 5:49953fa1:::default.realm:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32) v8 -- 0x5641f5892770 con 0x5641f585e600 2023-10-05T19:29:34.158 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.154+0000 7f1ce586f700 1 -- 172.21.15.167:0/2391133210 <== osd.1 v2:172.21.15.167:6800/30035 16 ==== osd_op_reply(42 default.realm [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 ==== 157+0+0 (crc 0 0 0) 0x7f1ce002f080 con 0x5641f585e600 2023-10-05T19:29:34.158 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.154+0000 7f1ce7539e40 20 rados_obj.operate() r=-2 bl.length=0 2023-10-05T19:29:34.158 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.154+0000 7f1ce7539e40 20 rados->read ofs=0 len=0 2023-10-05T19:29:34.159 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.154+0000 7f1ce7539e40 1 -- 172.21.15.167:0/2391133210 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:43 5.f 5:f4c53578:::zonegroups_names.default:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32) v8 -- 0x5641f5892770 con 0x5641f585e600 2023-10-05T19:29:34.159 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.154+0000 7f1ce586f700 1 -- 172.21.15.167:0/2391133210 <== osd.1 v2:172.21.15.167:6800/30035 17 ==== osd_op_reply(43 zonegroups_names.default [read 0~46 out=46b] v0'0 uv2 ondisk = 0) v8 ==== 168+0+46 (crc 0 0 0) 0x7f1ce002f080 con 0x5641f585e600 2023-10-05T19:29:34.159 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.154+0000 7f1ce7539e40 20 rados_obj.operate() r=0 bl.length=46 2023-10-05T19:29:34.159 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.154+0000 7f1ce7539e40 1 -- 172.21.15.167:0/2391133210 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:44 5.0 5:00000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x5641f5893780 con 0x5641f588f200 2023-10-05T19:29:34.160 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.154+0000 7f1ce506e700 1 -- 172.21.15.167:0/2391133210 <== osd.3 v2:172.21.15.167:6808/30036 11 ==== osd_op_reply(44 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f1cd8014070 con 0x5641f588f200 2023-10-05T19:29:34.160 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.154+0000 7f1ce506e700 1 -- 172.21.15.167:0/2391133210 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:45 5.10 5:08000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x5641f5893780 con 0x7f1cd802d020 2023-10-05T19:29:34.161 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.158+0000 7f1ce486d700 1 -- 172.21.15.167:0/2391133210 <== osd.2 v2:172.21.15.167:6816/30037 7 ==== osd_op_reply(45 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f1cd000fb90 con 0x7f1cd802d020 2023-10-05T19:29:34.161 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.158+0000 7f1ce486d700 1 -- 172.21.15.167:0/2391133210 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:46 5.8 5:10000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x5641f5893780 con 0x7f1cd0029420 2023-10-05T19:29:34.162 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.158+0000 7f1ce586f700 1 -- 172.21.15.167:0/2391133210 <== osd.0 v2:172.21.15.118:6800/27690 11 ==== osd_op_reply(46 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f1ce002f080 con 0x7f1cd0029420 2023-10-05T19:29:34.162 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.158+0000 7f1ce586f700 1 -- 172.21.15.167:0/2391133210 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:47 5.18 5:18000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x5641f588dba0 con 0x5641f585e600 2023-10-05T19:29:34.163 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.158+0000 7f1ce586f700 1 -- 172.21.15.167:0/2391133210 <== osd.1 v2:172.21.15.167:6800/30035 18 ==== osd_op_reply(47 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f1ce002f080 con 0x5641f585e600 2023-10-05T19:29:34.163 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.158+0000 7f1ce586f700 1 -- 172.21.15.167:0/2391133210 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:48 5.4 5:20000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x5641f588dba0 con 0x5641f585e600 2023-10-05T19:29:34.163 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.158+0000 7f1ce586f700 1 -- 172.21.15.167:0/2391133210 <== osd.1 v2:172.21.15.167:6800/30035 19 ==== osd_op_reply(48 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f1ce002f080 con 0x5641f585e600 2023-10-05T19:29:34.163 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.158+0000 7f1ce586f700 1 -- 172.21.15.167:0/2391133210 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:49 5.14 5:28000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x5641f588dba0 con 0x7f1cd0029420 2023-10-05T19:29:34.164 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.162+0000 7f1ce586f700 1 -- 172.21.15.167:0/2391133210 <== osd.0 v2:172.21.15.118:6800/27690 12 ==== osd_op_reply(49 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f1ce002f080 con 0x7f1cd0029420 2023-10-05T19:29:34.164 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.162+0000 7f1ce586f700 1 -- 172.21.15.167:0/2391133210 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:50 5.c 5:30000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x5641f588dba0 con 0x7f1cd0029420 2023-10-05T19:29:34.165 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.162+0000 7f1ce586f700 1 -- 172.21.15.167:0/2391133210 <== osd.0 v2:172.21.15.118:6800/27690 13 ==== osd_op_reply(50 [pgnls start_epoch 32 out=112b] v23'1 uv1 ondisk = 1) v8 ==== 144+0+112 (crc 0 0 0) 0x7f1ce002f080 con 0x7f1cd0029420 2023-10-05T19:29:34.165 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.162+0000 7f1ce586f700 1 -- 172.21.15.167:0/2391133210 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:51 5.1c 5:38000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x5641f588dba0 con 0x7f1cd802d020 2023-10-05T19:29:34.166 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.162+0000 7f1ce486d700 1 -- 172.21.15.167:0/2391133210 <== osd.2 v2:172.21.15.167:6816/30037 8 ==== osd_op_reply(51 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f1cd000fb90 con 0x7f1cd802d020 2023-10-05T19:29:34.166 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.162+0000 7f1ce486d700 1 -- 172.21.15.167:0/2391133210 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:52 5.2 5:40000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x5641f588dba0 con 0x5641f588f200 2023-10-05T19:29:34.167 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.162+0000 7f1ce506e700 1 -- 172.21.15.167:0/2391133210 <== osd.3 v2:172.21.15.167:6808/30036 12 ==== osd_op_reply(52 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f1cd8014070 con 0x5641f588f200 2023-10-05T19:29:34.167 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.162+0000 7f1ce506e700 1 -- 172.21.15.167:0/2391133210 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:53 5.12 5:48000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x5641f588dba0 con 0x5641f585e600 2023-10-05T19:29:34.168 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.162+0000 7f1ce586f700 1 -- 172.21.15.167:0/2391133210 <== osd.1 v2:172.21.15.167:6800/30035 20 ==== osd_op_reply(53 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f1ce002f080 con 0x5641f585e600 2023-10-05T19:29:34.168 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.162+0000 7f1ce586f700 1 -- 172.21.15.167:0/2391133210 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:54 5.a 5:50000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x5641f588dba0 con 0x7f1cd0029420 2023-10-05T19:29:34.169 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.166+0000 7f1ce586f700 1 -- 172.21.15.167:0/2391133210 <== osd.0 v2:172.21.15.118:6800/27690 14 ==== osd_op_reply(54 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f1ce002f080 con 0x7f1cd0029420 2023-10-05T19:29:34.169 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.166+0000 7f1ce586f700 1 -- 172.21.15.167:0/2391133210 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:55 5.1a 5:58000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x5641f588dba0 con 0x5641f585e600 2023-10-05T19:29:34.169 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.166+0000 7f1ce586f700 1 -- 172.21.15.167:0/2391133210 <== osd.1 v2:172.21.15.167:6800/30035 21 ==== osd_op_reply(55 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f1ce002f080 con 0x5641f585e600 2023-10-05T19:29:34.170 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.166+0000 7f1ce586f700 1 -- 172.21.15.167:0/2391133210 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:56 5.6 5:60000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x5641f588dba0 con 0x7f1cd802d020 2023-10-05T19:29:34.170 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.166+0000 7f1ce486d700 1 -- 172.21.15.167:0/2391133210 <== osd.2 v2:172.21.15.167:6816/30037 9 ==== osd_op_reply(56 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f1cd000fb90 con 0x7f1cd802d020 2023-10-05T19:29:34.171 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.166+0000 7f1ce486d700 1 -- 172.21.15.167:0/2391133210 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:57 5.16 5:68000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x5641f588dba0 con 0x5641f588f200 2023-10-05T19:29:34.171 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.166+0000 7f1ce506e700 1 -- 172.21.15.167:0/2391133210 <== osd.3 v2:172.21.15.167:6808/30036 13 ==== osd_op_reply(57 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f1cd8014070 con 0x5641f588f200 2023-10-05T19:29:34.171 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.166+0000 7f1ce506e700 1 -- 172.21.15.167:0/2391133210 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:58 5.e 5:70000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x5641f588dba0 con 0x5641f588f200 2023-10-05T19:29:34.172 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.166+0000 7f1ce506e700 1 -- 172.21.15.167:0/2391133210 <== osd.3 v2:172.21.15.167:6808/30036 14 ==== osd_op_reply(58 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f1cd8014070 con 0x5641f588f200 2023-10-05T19:29:34.172 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.166+0000 7f1ce506e700 1 -- 172.21.15.167:0/2391133210 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:59 5.1e 5:78000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x5641f588dba0 con 0x7f1cd0029420 2023-10-05T19:29:34.173 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.170+0000 7f1ce586f700 1 -- 172.21.15.167:0/2391133210 <== osd.0 v2:172.21.15.118:6800/27690 15 ==== osd_op_reply(59 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f1ce002f080 con 0x7f1cd0029420 2023-10-05T19:29:34.173 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.170+0000 7f1ce586f700 1 -- 172.21.15.167:0/2391133210 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:60 5.1 5:80000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x5641f588dba0 con 0x5641f588f200 2023-10-05T19:29:34.174 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.170+0000 7f1ce506e700 1 -- 172.21.15.167:0/2391133210 <== osd.3 v2:172.21.15.167:6808/30036 15 ==== osd_op_reply(60 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f1cd8014070 con 0x5641f588f200 2023-10-05T19:29:34.174 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.170+0000 7f1ce506e700 1 -- 172.21.15.167:0/2391133210 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:61 5.11 5:88000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x5641f588dba0 con 0x5641f588f200 2023-10-05T19:29:34.175 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.170+0000 7f1ce506e700 1 -- 172.21.15.167:0/2391133210 <== osd.3 v2:172.21.15.167:6808/30036 16 ==== osd_op_reply(61 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f1cd8014070 con 0x5641f588f200 2023-10-05T19:29:34.175 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.170+0000 7f1ce506e700 1 -- 172.21.15.167:0/2391133210 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:62 5.9 5:90000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x5641f588dba0 con 0x5641f585e600 2023-10-05T19:29:34.175 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.170+0000 7f1ce586f700 1 -- 172.21.15.167:0/2391133210 <== osd.1 v2:172.21.15.167:6800/30035 22 ==== osd_op_reply(62 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f1ce002f080 con 0x5641f585e600 2023-10-05T19:29:34.176 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.170+0000 7f1ce586f700 1 -- 172.21.15.167:0/2391133210 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:63 5.19 5:98000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x5641f588dba0 con 0x5641f585e600 2023-10-05T19:29:34.176 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.174+0000 7f1ce586f700 1 -- 172.21.15.167:0/2391133210 <== osd.1 v2:172.21.15.167:6800/30035 23 ==== osd_op_reply(63 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f1ce002f080 con 0x5641f585e600 2023-10-05T19:29:34.176 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.174+0000 7f1ce586f700 1 -- 172.21.15.167:0/2391133210 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:64 5.5 5:a0000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x5641f588dba0 con 0x7f1cd0029420 2023-10-05T19:29:34.177 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.174+0000 7f1ce586f700 1 -- 172.21.15.167:0/2391133210 <== osd.0 v2:172.21.15.118:6800/27690 16 ==== osd_op_reply(64 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f1ce002f080 con 0x7f1cd0029420 2023-10-05T19:29:34.177 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.174+0000 7f1ce586f700 1 -- 172.21.15.167:0/2391133210 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:65 5.15 5:a8000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x5641f588dba0 con 0x5641f588f200 2023-10-05T19:29:34.178 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.174+0000 7f1ce506e700 1 -- 172.21.15.167:0/2391133210 <== osd.3 v2:172.21.15.167:6808/30036 17 ==== osd_op_reply(65 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f1cd8014070 con 0x5641f588f200 2023-10-05T19:29:34.178 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.174+0000 7f1ce506e700 1 -- 172.21.15.167:0/2391133210 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:66 5.d 5:b0000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x5641f588dba0 con 0x7f1cd802d020 2023-10-05T19:29:34.179 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.174+0000 7f1ce486d700 1 -- 172.21.15.167:0/2391133210 <== osd.2 v2:172.21.15.167:6816/30037 10 ==== osd_op_reply(66 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f1cd000fb90 con 0x7f1cd802d020 2023-10-05T19:29:34.179 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.174+0000 7f1ce486d700 1 -- 172.21.15.167:0/2391133210 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:67 5.1d 5:b8000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x5641f588dba0 con 0x5641f585e600 2023-10-05T19:29:34.180 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.174+0000 7f1ce586f700 1 -- 172.21.15.167:0/2391133210 <== osd.1 v2:172.21.15.167:6800/30035 24 ==== osd_op_reply(67 [pgnls start_epoch 32 out=107b] v30'2 uv2 ondisk = 1) v8 ==== 144+0+107 (crc 0 0 0) 0x7f1ce002f080 con 0x5641f585e600 2023-10-05T19:29:34.180 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.174+0000 7f1ce586f700 1 -- 172.21.15.167:0/2391133210 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:68 5.3 5:c0000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x5641f588dba0 con 0x7f1cd0029420 2023-10-05T19:29:34.181 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.178+0000 7f1ce586f700 1 -- 172.21.15.167:0/2391133210 <== osd.0 v2:172.21.15.118:6800/27690 17 ==== osd_op_reply(68 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f1ce002f080 con 0x7f1cd0029420 2023-10-05T19:29:34.181 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.178+0000 7f1ce586f700 1 -- 172.21.15.167:0/2391133210 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:69 5.13 5:c8000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x5641f588dba0 con 0x5641f588f200 2023-10-05T19:29:34.182 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.178+0000 7f1ce506e700 1 -- 172.21.15.167:0/2391133210 <== osd.3 v2:172.21.15.167:6808/30036 18 ==== osd_op_reply(69 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f1cd8014070 con 0x5641f588f200 2023-10-05T19:29:34.182 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.178+0000 7f1ce506e700 1 -- 172.21.15.167:0/2391133210 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:70 5.b 5:d0000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x5641f588dba0 con 0x7f1cd0029420 2023-10-05T19:29:34.183 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.178+0000 7f1ce586f700 1 -- 172.21.15.167:0/2391133210 <== osd.0 v2:172.21.15.118:6800/27690 18 ==== osd_op_reply(70 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f1ce002f080 con 0x7f1cd0029420 2023-10-05T19:29:34.183 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.178+0000 7f1ce586f700 1 -- 172.21.15.167:0/2391133210 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:71 5.1b 5:d8000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x5641f588dba0 con 0x7f1cd802d020 2023-10-05T19:29:34.184 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.178+0000 7f1ce486d700 1 -- 172.21.15.167:0/2391133210 <== osd.2 v2:172.21.15.167:6816/30037 11 ==== osd_op_reply(71 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f1cd000fb90 con 0x7f1cd802d020 2023-10-05T19:29:34.184 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.178+0000 7f1ce486d700 1 -- 172.21.15.167:0/2391133210 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:72 5.7 5:e0000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x5641f588dba0 con 0x5641f588f200 2023-10-05T19:29:34.184 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.182+0000 7f1ce506e700 1 -- 172.21.15.167:0/2391133210 <== osd.3 v2:172.21.15.167:6808/30036 19 ==== osd_op_reply(72 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f1cd8014070 con 0x5641f588f200 2023-10-05T19:29:34.184 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.182+0000 7f1ce506e700 1 -- 172.21.15.167:0/2391133210 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:73 5.17 5:e8000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x5641f588dba0 con 0x7f1cd0029420 2023-10-05T19:29:34.185 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.182+0000 7f1ce586f700 1 -- 172.21.15.167:0/2391133210 <== osd.0 v2:172.21.15.118:6800/27690 19 ==== osd_op_reply(73 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f1ce002f080 con 0x7f1cd0029420 2023-10-05T19:29:34.186 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.182+0000 7f1ce586f700 1 -- 172.21.15.167:0/2391133210 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:74 5.f 5:f0000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x5641f588dba0 con 0x5641f585e600 2023-10-05T19:29:34.187 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.182+0000 7f1ce586f700 1 -- 172.21.15.167:0/2391133210 <== osd.1 v2:172.21.15.167:6800/30035 25 ==== osd_op_reply(74 [pgnls start_epoch 32 out=115b] v23'2 uv2 ondisk = 1) v8 ==== 144+0+115 (crc 0 0 0) 0x7f1ce002f080 con 0x5641f585e600 2023-10-05T19:29:34.187 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.182+0000 7f1ce586f700 1 -- 172.21.15.167:0/2391133210 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:75 5.1f 5:f8000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x5641f588dba0 con 0x7f1cd802d020 2023-10-05T19:29:34.188 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.182+0000 7f1ce486d700 1 -- 172.21.15.167:0/2391133210 <== osd.2 v2:172.21.15.167:6816/30037 12 ==== osd_op_reply(75 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f1cd000fb90 con 0x7f1cd802d020 2023-10-05T19:29:34.188 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.182+0000 7f1ce7539e40 20 RGWRados::pool_iterate: got zonegroup_info.4f1f719d-9648-417b-b10c-e5d939825419 2023-10-05T19:29:34.188 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.182+0000 7f1ce7539e40 20 RGWRados::pool_iterate: got zone_info.9a9df965-6612-45d4-a45f-6e3c889bacc2 2023-10-05T19:29:34.188 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.182+0000 7f1ce7539e40 20 RGWRados::pool_iterate: got zone_names.default 2023-10-05T19:29:34.188 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.182+0000 7f1ce7539e40 20 RGWRados::pool_iterate: got zonegroups_names.default 2023-10-05T19:29:34.188 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.182+0000 7f1ce7539e40 20 rados->read ofs=0 len=0 2023-10-05T19:29:34.188 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.182+0000 7f1ce7539e40 1 -- 172.21.15.167:0/2391133210 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:76 5.f 5:f43b6ece:::zone_names.default:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32) v8 -- 0x5641f5892770 con 0x5641f585e600 2023-10-05T19:29:34.189 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.186+0000 7f1ce586f700 1 -- 172.21.15.167:0/2391133210 <== osd.1 v2:172.21.15.167:6800/30035 26 ==== osd_op_reply(76 zone_names.default [read 0~46 out=46b] v0'0 uv1 ondisk = 0) v8 ==== 162+0+46 (crc 0 0 0) 0x7f1ce002f080 con 0x5641f585e600 2023-10-05T19:29:34.189 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.186+0000 7f1ce7539e40 20 rados_obj.operate() r=0 bl.length=46 2023-10-05T19:29:34.189 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.186+0000 7f1ce7539e40 20 rados->read ofs=0 len=0 2023-10-05T19:29:34.189 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.186+0000 7f1ce7539e40 1 -- 172.21.15.167:0/2391133210 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:77 5.1d 5:b8374eb4:::zone_info.9a9df965-6612-45d4-a45f-6e3c889bacc2:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32) v8 -- 0x5641f5892b20 con 0x5641f585e600 2023-10-05T19:29:34.190 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.186+0000 7f1ce586f700 1 -- 172.21.15.167:0/2391133210 <== osd.1 v2:172.21.15.167:6800/30035 27 ==== osd_op_reply(77 zone_info.9a9df965-6612-45d4-a45f-6e3c889bacc2 [read 0~874 out=874b] v0'0 uv2 ondisk = 0) v8 ==== 190+0+874 (crc 0 0 0) 0x7f1ce002f080 con 0x5641f585e600 2023-10-05T19:29:34.190 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.186+0000 7f1ce7539e40 20 rados_obj.operate() r=0 bl.length=874 2023-10-05T19:29:34.190 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.186+0000 7f1ce7539e40 20 rados->read ofs=0 len=0 2023-10-05T19:29:34.190 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.186+0000 7f1ce7539e40 1 -- 172.21.15.167:0/2391133210 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:78 5.f 5:f4c53578:::zonegroups_names.default:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32) v8 -- 0x5641f5891ff0 con 0x5641f585e600 2023-10-05T19:29:34.191 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.186+0000 7f1ce586f700 1 -- 172.21.15.167:0/2391133210 <== osd.1 v2:172.21.15.167:6800/30035 28 ==== osd_op_reply(78 zonegroups_names.default [read 0~46 out=46b] v0'0 uv2 ondisk = 0) v8 ==== 168+0+46 (crc 0 0 0) 0x7f1ce002f080 con 0x5641f585e600 2023-10-05T19:29:34.191 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.186+0000 7f1ce7539e40 20 rados_obj.operate() r=0 bl.length=46 2023-10-05T19:29:34.191 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.186+0000 7f1ce7539e40 20 rados->read ofs=0 len=0 2023-10-05T19:29:34.191 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.186+0000 7f1ce7539e40 1 -- 172.21.15.167:0/2391133210 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:79 5.c 5:30d299a5:::zonegroup_info.4f1f719d-9648-417b-b10c-e5d939825419:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32) v8 -- 0x5641f5892000 con 0x7f1cd0029420 2023-10-05T19:29:34.192 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.186+0000 7f1ce586f700 1 -- 172.21.15.167:0/2391133210 <== osd.0 v2:172.21.15.118:6800/27690 20 ==== osd_op_reply(79 zonegroup_info.4f1f719d-9648-417b-b10c-e5d939825419 [read 0~362 out=362b] v0'0 uv1 ondisk = 0) v8 ==== 195+0+362 (crc 0 0 0) 0x7f1ce002f080 con 0x7f1cd0029420 2023-10-05T19:29:34.192 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.186+0000 7f1ce7539e40 20 rados_obj.operate() r=0 bl.length=362 2023-10-05T19:29:34.192 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.186+0000 7f1ce7539e40 20 rados->read ofs=0 len=0 2023-10-05T19:29:34.192 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.186+0000 7f1ce7539e40 1 -- 172.21.15.167:0/2391133210 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:80 5.3 5:c0ad8cbd:::region_map:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32) v8 -- 0x5641f5893610 con 0x7f1cd0029420 2023-10-05T19:29:34.192 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.186+0000 7f1ce586f700 1 -- 172.21.15.167:0/2391133210 <== osd.0 v2:172.21.15.118:6800/27690 21 ==== osd_op_reply(80 region_map [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 ==== 154+0+0 (crc 0 0 0) 0x7f1ce0050040 con 0x7f1cd0029420 2023-10-05T19:29:34.193 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.186+0000 7f1ce7539e40 20 rados_obj.operate() r=-2 bl.length=0 2023-10-05T19:29:34.193 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.186+0000 7f1ce7539e40 20 zone default found 2023-10-05T19:29:34.193 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.186+0000 7f1ce7539e40 4 Realm: () 2023-10-05T19:29:34.193 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.186+0000 7f1ce7539e40 4 ZoneGroup: default (4f1f719d-9648-417b-b10c-e5d939825419) 2023-10-05T19:29:34.193 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.186+0000 7f1ce7539e40 4 Zone: default (9a9df965-6612-45d4-a45f-6e3c889bacc2) 2023-10-05T19:29:34.193 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.186+0000 7f1ce7539e40 10 cannot find current period zonegroup using local zonegroup configuration 2023-10-05T19:29:34.193 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.186+0000 7f1ce7539e40 20 zonegroup default 2023-10-05T19:29:34.193 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.186+0000 7f1ce7539e40 20 rados->read ofs=0 len=0 2023-10-05T19:29:34.193 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.186+0000 7f1ce7539e40 1 -- 172.21.15.167:0/2391133210 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:81 5.3 5:c52100b6:::period_config.default:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32) v8 -- 0x5641f5893610 con 0x7f1cd0029420 2023-10-05T19:29:34.194 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.190+0000 7f1ce586f700 1 -- 172.21.15.167:0/2391133210 <== osd.0 v2:172.21.15.118:6800/27690 22 ==== osd_op_reply(81 period_config.default [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 ==== 165+0+0 (crc 0 0 0) 0x7f1ce0050040 con 0x7f1cd0029420 2023-10-05T19:29:34.194 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.190+0000 7f1ce7539e40 20 rados_obj.operate() r=-2 bl.length=0 2023-10-05T19:29:34.194 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.190+0000 7f1ce7539e40 20 rados->read ofs=0 len=0 2023-10-05T19:29:34.195 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.190+0000 7f1ce7539e40 1 -- 172.21.15.167:0/2391133210 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:82 6.2 6:4347d321:::bucket.sync-source-hints.:head [call version.read in=11b,read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32) v8 -- 0x5641f5894be0 con 0x5641f588f200 2023-10-05T19:29:34.195 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.190+0000 7f1ce506e700 1 -- 172.21.15.167:0/2391133210 <== osd.3 v2:172.21.15.167:6808/30036 20 ==== osd_op_reply(82 bucket.sync-source-hints. [call,read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 ==== 211+0+0 (crc 0 0 0) 0x7f1cd8014070 con 0x5641f588f200 2023-10-05T19:29:34.195 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.190+0000 7f1ce7539e40 20 rados_obj.operate() r=-2 bl.length=0 2023-10-05T19:29:34.195 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.190+0000 7f1ce7539e40 20 rados->read ofs=0 len=0 2023-10-05T19:29:34.195 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.190+0000 7f1ce7539e40 1 -- 172.21.15.167:0/2391133210 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:83 6.b 6:d467b91b:::bucket.sync-target-hints.:head [call version.read in=11b,read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32) v8 -- 0x5641f5894f90 con 0x5641f588f200 2023-10-05T19:29:34.195 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.190+0000 7f1ce506e700 1 -- 172.21.15.167:0/2391133210 <== osd.3 v2:172.21.15.167:6808/30036 21 ==== osd_op_reply(83 bucket.sync-target-hints. [call,read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 ==== 211+0+0 (crc 0 0 0) 0x7f1cd8014070 con 0x5641f588f200 2023-10-05T19:29:34.196 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.190+0000 7f1ce7539e40 20 rados_obj.operate() r=-2 bl.length=0 2023-10-05T19:29:34.196 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.190+0000 7f1ce7539e40 20 started sync module instance, tier type = 2023-10-05T19:29:34.196 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.190+0000 7f1ce7539e40 20 started zone id=9a9df965-6612-45d4-a45f-6e3c889bacc2 (name=default) with tier type = 2023-10-05T19:29:34.196 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.190+0000 7f1ce7539e40 1 -- 172.21.15.167:0/2391133210 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:84 7.1f 7:f95f44c2:::notify.0:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e32) v8 -- 0x5641f58942a0 con 0x7f1cd802d020 2023-10-05T19:29:34.198 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.194+0000 7f1ce486d700 1 -- 172.21.15.167:0/2391133210 <== osd.2 v2:172.21.15.167:6816/30037 13 ==== osd_op_reply(84 notify.0 [create] v32'234 uv234 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f1cd000fb90 con 0x7f1cd802d020 2023-10-05T19:29:34.198 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.194+0000 7f1ce7539e40 1 -- 172.21.15.167:0/2391133210 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:85 7.1f 7:f95f44c2:::notify.0:head [watch watch cookie 94841292277408] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e32) v8 -- 0x5641f5893ed0 con 0x7f1cd802d020 2023-10-05T19:29:34.198 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.194+0000 7f1ce7539e40 1 -- 172.21.15.167:0/2391133210 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:86 7.0 7:05bf5b68:::notify.1:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e32) v8 -- 0x5641f5895e50 con 0x5641f585e600 2023-10-05T19:29:34.201 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.198+0000 7f1ce486d700 1 -- 172.21.15.167:0/2391133210 <== osd.2 v2:172.21.15.167:6816/30037 14 ==== osd_op_reply(85 notify.0 [watch watch cookie 94841292277408] v32'235 uv234 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f1cd000fb90 con 0x7f1cd802d020 2023-10-05T19:29:34.201 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.198+0000 7f1ce586f700 1 -- 172.21.15.167:0/2391133210 <== osd.1 v2:172.21.15.167:6800/30035 29 ==== osd_op_reply(86 notify.1 [create] v32'234 uv234 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f1ce0050040 con 0x5641f585e600 2023-10-05T19:29:34.201 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.198+0000 7f1ce7539e40 1 -- 172.21.15.167:0/2391133210 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:87 7.0 7:05bf5b68:::notify.1:head [watch watch cookie 94841292282144] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e32) v8 -- 0x5641f5896250 con 0x5641f585e600 2023-10-05T19:29:34.201 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.198+0000 7f1ce7539e40 1 -- 172.21.15.167:0/2391133210 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:88 7.15 7:a93a5511:::notify.2:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e32) v8 -- 0x5641f5896af0 con 0x5641f588f200 2023-10-05T19:29:34.204 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.198+0000 7f1ce506e700 1 -- 172.21.15.167:0/2391133210 <== osd.3 v2:172.21.15.167:6808/30036 22 ==== osd_op_reply(88 notify.2 [create] v32'234 uv234 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f1cd8014070 con 0x5641f588f200 2023-10-05T19:29:34.205 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.198+0000 7f1ce7539e40 1 -- 172.21.15.167:0/2391133210 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:89 7.15 7:a93a5511:::notify.2:head [watch watch cookie 94841292284784] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e32) v8 -- 0x5641f5897390 con 0x5641f588f200 2023-10-05T19:29:34.205 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.198+0000 7f1ce7539e40 1 -- 172.21.15.167:0/2391133210 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:90 7.e 7:7759931f:::notify.3:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e32) v8 -- 0x5641f5897c30 con 0x7f1cd802d020 2023-10-05T19:29:34.205 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.202+0000 7f1ce586f700 1 -- 172.21.15.167:0/2391133210 <== osd.1 v2:172.21.15.167:6800/30035 30 ==== osd_op_reply(87 notify.1 [watch watch cookie 94841292282144] v32'235 uv234 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f1ce0050040 con 0x5641f585e600 2023-10-05T19:29:34.207 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.202+0000 7f1ce506e700 1 -- 172.21.15.167:0/2391133210 <== osd.3 v2:172.21.15.167:6808/30036 23 ==== osd_op_reply(89 notify.2 [watch watch cookie 94841292284784] v32'235 uv234 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f1cd8014070 con 0x5641f588f200 2023-10-05T19:29:34.207 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.202+0000 7f1ce486d700 1 -- 172.21.15.167:0/2391133210 <== osd.2 v2:172.21.15.167:6816/30037 15 ==== osd_op_reply(90 notify.3 [create] v32'234 uv234 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f1cd000fb90 con 0x7f1cd802d020 2023-10-05T19:29:34.207 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.202+0000 7f1ce7539e40 1 -- 172.21.15.167:0/2391133210 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:91 7.e 7:7759931f:::notify.3:head [watch watch cookie 94841292289856] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e32) v8 -- 0x5641f5896fc0 con 0x7f1cd802d020 2023-10-05T19:29:34.208 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.202+0000 7f1ce7539e40 1 -- 172.21.15.167:0/2391133210 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:92 7.d 7:b4812045:::notify.4:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e32) v8 -- 0x5641f5898730 con 0x5641f585e600 2023-10-05T19:29:34.210 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.206+0000 7f1ce486d700 1 -- 172.21.15.167:0/2391133210 <== osd.2 v2:172.21.15.167:6816/30037 16 ==== osd_op_reply(91 notify.3 [watch watch cookie 94841292289856] v32'235 uv234 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f1cd000fb90 con 0x7f1cd802d020 2023-10-05T19:29:34.210 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.206+0000 7f1ce586f700 1 -- 172.21.15.167:0/2391133210 <== osd.1 v2:172.21.15.167:6800/30035 31 ==== osd_op_reply(92 notify.4 [create] v32'234 uv234 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f1ce0050040 con 0x5641f585e600 2023-10-05T19:29:34.210 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.206+0000 7f1ce7539e40 1 -- 172.21.15.167:0/2391133210 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:93 7.d 7:b4812045:::notify.4:head [watch watch cookie 94841292280640] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e32) v8 -- 0x5641f5898b30 con 0x5641f585e600 2023-10-05T19:29:34.210 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.206+0000 7f1ce7539e40 1 -- 172.21.15.167:0/2391133210 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:94 7.3 7:c609908c:::notify.5:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e32) v8 -- 0x5641f5899540 con 0x7f1cd0029420 2023-10-05T19:29:34.212 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.206+0000 7f1ce586f700 1 -- 172.21.15.167:0/2391133210 <== osd.0 v2:172.21.15.118:6800/27690 23 ==== osd_op_reply(94 notify.5 [create] v32'234 uv234 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f1ce0050040 con 0x7f1cd0029420 2023-10-05T19:29:34.212 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.206+0000 7f1ce7539e40 1 -- 172.21.15.167:0/2391133210 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:95 7.3 7:c609908c:::notify.5:head [watch watch cookie 94841292296272] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e32) v8 -- 0x5641f5899990 con 0x7f1cd0029420 2023-10-05T19:29:34.213 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.210+0000 7f1ce7539e40 1 -- 172.21.15.167:0/2391133210 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:96 7.14 7:2b04a3e9:::notify.6:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e32) v8 -- 0x5641f589a410 con 0x7f1cd0029420 2023-10-05T19:29:34.213 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.210+0000 7f1ce586f700 1 -- 172.21.15.167:0/2391133210 <== osd.1 v2:172.21.15.167:6800/30035 32 ==== osd_op_reply(93 notify.4 [watch watch cookie 94841292280640] v32'235 uv234 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f1ce0050040 con 0x5641f585e600 2023-10-05T19:29:34.215 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.210+0000 7f1ce586f700 1 -- 172.21.15.167:0/2391133210 <== osd.0 v2:172.21.15.118:6800/27690 24 ==== osd_op_reply(95 notify.5 [watch watch cookie 94841292296272] v32'235 uv234 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f1ce0050040 con 0x7f1cd0029420 2023-10-05T19:29:34.215 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.210+0000 7f1ce586f700 1 -- 172.21.15.167:0/2391133210 <== osd.0 v2:172.21.15.118:6800/27690 25 ==== osd_op_reply(96 notify.6 [create] v32'234 uv234 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f1ce0050040 con 0x7f1cd0029420 2023-10-05T19:29:34.216 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.210+0000 7f1ce7539e40 1 -- 172.21.15.167:0/2391133210 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:97 7.14 7:2b04a3e9:::notify.6:head [watch watch cookie 94841292292608] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e32) v8 -- 0x5641f5899990 con 0x7f1cd0029420 2023-10-05T19:29:34.216 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.210+0000 7f1ce7539e40 1 -- 172.21.15.167:0/2391133210 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:98 7.9 7:93e5b521:::notify.7:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e32) v8 -- 0x5641f589a870 con 0x7f1cd0029420 2023-10-05T19:29:34.218 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.214+0000 7f1ce586f700 1 -- 172.21.15.167:0/2391133210 <== osd.0 v2:172.21.15.118:6800/27690 26 ==== osd_op_reply(98 notify.7 [create] v32'234 uv234 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f1ce0050040 con 0x7f1cd0029420 2023-10-05T19:29:34.218 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.214+0000 7f1ce7539e40 1 -- 172.21.15.167:0/2391133210 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:99 7.9 7:93e5b521:::notify.7:head [watch watch cookie 94841292301120] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e32) v8 -- 0x5641f589a790 con 0x7f1cd0029420 2023-10-05T19:29:34.218 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.214+0000 7f1ce7539e40 20 add_watcher() i=0 2023-10-05T19:29:34.218 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.214+0000 7f1ce7539e40 20 add_watcher() i=1 2023-10-05T19:29:34.218 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.214+0000 7f1ce7539e40 20 add_watcher() i=2 2023-10-05T19:29:34.218 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.214+0000 7f1ce7539e40 20 add_watcher() i=3 2023-10-05T19:29:34.218 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.214+0000 7f1ce7539e40 20 add_watcher() i=4 2023-10-05T19:29:34.218 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.214+0000 7f1ce7539e40 20 add_watcher() i=5 2023-10-05T19:29:34.219 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.214+0000 7f1ce586f700 1 -- 172.21.15.167:0/2391133210 <== osd.0 v2:172.21.15.118:6800/27690 27 ==== osd_op_reply(97 notify.6 [watch watch cookie 94841292292608] v32'235 uv234 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f1ce0050040 con 0x7f1cd0029420 2023-10-05T19:29:34.219 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.214+0000 7f1ce7539e40 20 add_watcher() i=6 2023-10-05T19:29:34.220 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.214+0000 7f1ce586f700 1 -- 172.21.15.167:0/2391133210 <== osd.0 v2:172.21.15.118:6800/27690 28 ==== osd_op_reply(99 notify.7 [watch watch cookie 94841292301120] v32'235 uv234 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f1ce0050040 con 0x7f1cd0029420 2023-10-05T19:29:34.220 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.214+0000 7f1ce7539e40 20 add_watcher() i=7 2023-10-05T19:29:34.220 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.214+0000 7f1ce7539e40 2 all 8 watchers are set, enabling cache 2023-10-05T19:29:34.220 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.214+0000 7f1ce7539e40 1 -- 172.21.15.167:0/2391133210 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:100 6.4 6:22d26bf9:::data_loggenerations_metadata:head [call version.check_conds in=74b,call version.read in=11b,read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32) v8 -- 0x5641f589dca0 con 0x5641f585e600 2023-10-05T19:29:34.221 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.218+0000 7f1ce586f700 1 -- 172.21.15.167:0/2391133210 <== osd.1 v2:172.21.15.167:6800/30035 33 ==== osd_op_reply(100 data_loggenerations_metadata [call,call out=48b,read 0~28 out=28b] v0'0 uv1 ondisk = 0) v8 ==== 256+0+76 (crc 0 0 0) 0x7f1ce0050040 con 0x5641f585e600 2023-10-05T19:29:34.221 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.218+0000 7f1ce7539e40 1 -- 172.21.15.167:0/2391133210 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:101 6.4 6:22d26bf9:::data_loggenerations_metadata:head [watch watch cookie 94841292297376] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e32) v8 -- 0x5641f589b110 con 0x5641f585e600 2023-10-05T19:29:34.223 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.218+0000 7f1ce586f700 1 -- 172.21.15.167:0/2391133210 <== osd.1 v2:172.21.15.167:6800/30035 34 ==== osd_op_reply(101 data_loggenerations_metadata [watch watch cookie 94841292297376] v32'229 uv1 ondisk = 0) v8 ==== 172+0+0 (crc 0 0 0) 0x7f1ce0050040 con 0x5641f585e600 2023-10-05T19:29:34.223 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.218+0000 7f1cb67e4700 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2023-10-05T19:29:34.223 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.218+0000 7f1ce7539e40 20 check_secure_mon_conn(): auth registy supported: methods=[2] modes=[2,1] 2023-10-05T19:29:34.223 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.218+0000 7f1ce7539e40 20 check_secure_mon_conn(): mode 1 is insecure 2023-10-05T19:29:34.224 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.218+0000 7f1ce7539e40 1 Processor -- start 2023-10-05T19:29:34.224 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.218+0000 7f1ce7539e40 1 -- start start 2023-10-05T19:29:34.224 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.218+0000 7f1ce7539e40 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5641f58a7f30 0x5641f58a8300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-05T19:29:34.224 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.218+0000 7f1ce7539e40 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x5641f58a8840 con 0x5641f58a7f30 2023-10-05T19:29:34.225 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.222+0000 7f1ce486d700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5641f58a7f30 0x5641f58a8300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-05T19:29:34.225 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.222+0000 7f1ce486d700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5641f58a7f30 0x5641f58a8300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.118:3300/0 says I am v2:172.21.15.167:46338/0 (socket says 172.21.15.167:46338) 2023-10-05T19:29:34.225 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.222+0000 7f1ce486d700 1 -- 172.21.15.167:0/2237857072 learned_addr learned my addr 172.21.15.167:0/2237857072 (peer_addr_for_me v2:172.21.15.167:0/0) 2023-10-05T19:29:34.225 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.222+0000 7f1ce486d700 1 -- 172.21.15.167:0/2237857072 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5641f58a8980 con 0x5641f58a7f30 2023-10-05T19:29:34.226 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.222+0000 7f1ce486d700 1 --2- 172.21.15.167:0/2237857072 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5641f58a7f30 0x5641f58a8300 secure :-1 s=READY pgs=606 cs=0 l=1 rev1=1 crypto rx=0x7f1cd0003000 tx=0x7f1cd0011000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6c2957f46bf7415e server_cookie=0 in_seq=0 out_seq=0 2023-10-05T19:29:34.226 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.222+0000 7f1cb5fe3700 1 -- 172.21.15.167:0/2237857072 <== mon.0 v2:172.21.15.118:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f1cd0030040 con 0x5641f58a7f30 2023-10-05T19:29:34.226 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.222+0000 7f1cb5fe3700 1 -- 172.21.15.167:0/2237857072 <== mon.0 v2:172.21.15.118:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1cd0029e80 con 0x5641f58a7f30 2023-10-05T19:29:34.226 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.222+0000 7f1cb5fe3700 1 -- 172.21.15.167:0/2237857072 <== mon.0 v2:172.21.15.118:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f1cd001c7c0 con 0x5641f58a7f30 2023-10-05T19:29:34.227 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.222+0000 7f1ce7539e40 1 -- 172.21.15.167:0/2237857072 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5641f58a7f30 msgr2=0x5641f58a8300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-05T19:29:34.227 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.222+0000 7f1ce7539e40 1 --2- 172.21.15.167:0/2237857072 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5641f58a7f30 0x5641f58a8300 secure :-1 s=READY pgs=606 cs=0 l=1 rev1=1 crypto rx=0x7f1cd0003000 tx=0x7f1cd0011000 comp rx=0 tx=0).stop 2023-10-05T19:29:34.227 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.222+0000 7f1ce7539e40 1 -- 172.21.15.167:0/2237857072 shutdown_connections 2023-10-05T19:29:34.228 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.222+0000 7f1ce7539e40 1 --2- 172.21.15.167:0/2237857072 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5641f58a7f30 0x5641f58a8300 unknown :-1 s=CLOSED pgs=606 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-05T19:29:34.228 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.222+0000 7f1ce7539e40 1 -- 172.21.15.167:0/2237857072 >> 172.21.15.167:0/2237857072 conn(0x5641f58a32f0 msgr2=0x5641f58a14a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-05T19:29:34.228 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.222+0000 7f1ce7539e40 1 -- 172.21.15.167:0/2237857072 shutdown_connections 2023-10-05T19:29:34.228 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.222+0000 7f1ce7539e40 1 -- 172.21.15.167:0/2237857072 wait complete. 2023-10-05T19:29:34.229 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.222+0000 7f1ce7539e40 1 Processor -- start 2023-10-05T19:29:34.229 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.222+0000 7f1ce7539e40 1 -- start start 2023-10-05T19:29:34.229 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.222+0000 7f1ce7539e40 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5641f58a7f30 0x5641f58adb30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-05T19:29:34.229 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.222+0000 7f1ce7539e40 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x5641f58aeb30 con 0x5641f58a7f30 2023-10-05T19:29:34.229 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.226+0000 7f1ce486d700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5641f58a7f30 0x5641f58adb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-05T19:29:34.229 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.226+0000 7f1ce486d700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5641f58a7f30 0x5641f58adb30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.118:3300/0 says I am v2:172.21.15.167:46354/0 (socket says 172.21.15.167:46354) 2023-10-05T19:29:34.229 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.226+0000 7f1ce486d700 1 -- 172.21.15.167:0/1343801672 learned_addr learned my addr 172.21.15.167:0/1343801672 (peer_addr_for_me v2:172.21.15.167:0/0) 2023-10-05T19:29:34.230 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.226+0000 7f1ce486d700 1 -- 172.21.15.167:0/1343801672 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5641f585ac30 con 0x5641f58a7f30 2023-10-05T19:29:34.230 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.226+0000 7f1ce486d700 1 --2- 172.21.15.167:0/1343801672 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5641f58a7f30 0x5641f58adb30 secure :-1 s=READY pgs=607 cs=0 l=1 rev1=1 crypto rx=0x7f1cd00059e0 tx=0x7f1cd0042000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-05T19:29:34.230 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.226+0000 7f1c27fff700 1 -- 172.21.15.167:0/1343801672 <== mon.0 v2:172.21.15.118:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f1cd0030050 con 0x5641f58a7f30 2023-10-05T19:29:34.230 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.226+0000 7f1c27fff700 1 -- 172.21.15.167:0/1343801672 <== mon.0 v2:172.21.15.118:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1cd004e070 con 0x5641f58a7f30 2023-10-05T19:29:34.231 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.226+0000 7f1ce7539e40 1 -- 172.21.15.167:0/1343801672 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x5641f58ae130 con 0x5641f58a7f30 2023-10-05T19:29:34.231 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.226+0000 7f1c27fff700 1 -- 172.21.15.167:0/1343801672 <== mon.0 v2:172.21.15.118:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f1cd0042ae0 con 0x5641f58a7f30 2023-10-05T19:29:34.231 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.226+0000 7f1ce7539e40 1 -- 172.21.15.167:0/1343801672 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x5641f58ae570 con 0x5641f58a7f30 2023-10-05T19:29:34.232 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.226+0000 7f1c27fff700 1 -- 172.21.15.167:0/1343801672 <== mon.0 v2:172.21.15.118:3300/0 4 ==== mgrmap(e 5) v1 ==== 40329+0+0 (secure 0 0 0) 0x7f1cd0050070 con 0x5641f58a7f30 2023-10-05T19:29:34.232 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.226+0000 7f1c27fff700 1 --2- 172.21.15.167:0/1343801672 >> [v2:172.21.15.167:6824/29889,v1:172.21.15.167:6825/29889] conn(0x7f1c2c030c50 0x7f1c2c033100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-05T19:29:34.232 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.226+0000 7f1c27fff700 1 -- 172.21.15.167:0/1343801672 <== mon.0 v2:172.21.15.118:3300/0 5 ==== osd_map(32..32 src has 1..32) v4 ==== 6260+0+0 (secure 0 0 0) 0x7f1cd003b030 con 0x5641f58a7f30 2023-10-05T19:29:34.233 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.230+0000 7f1ce586f700 1 --2- 172.21.15.167:0/1343801672 >> [v2:172.21.15.167:6824/29889,v1:172.21.15.167:6825/29889] conn(0x7f1c2c030c50 0x7f1c2c033100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-05T19:29:34.233 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.230+0000 7f1ce7539e40 5 note: GC not initialized 2023-10-05T19:29:34.233 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.230+0000 7f1ce586f700 1 --2- 172.21.15.167:0/1343801672 >> [v2:172.21.15.167:6824/29889,v1:172.21.15.167:6825/29889] conn(0x7f1c2c030c50 0x7f1c2c033100 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x5641f589a9e0 tx=0x7f1ce0000f90 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-05T19:29:34.235 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.230+0000 7f1c257fa700 20 reqs_thread_entry: start 2023-10-05T19:29:34.235 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.230+0000 7f1ce7539e40 20 init_complete bucket index max shards: 11 2023-10-05T19:29:34.235 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.230+0000 7f1c1effd700 20 reqs_thread_entry: start 2023-10-05T19:29:34.236 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.230+0000 7f1ce7539e40 10 rgw notify: Started notification manager with: 1 workers 2023-10-05T19:29:34.236 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.230+0000 7f1c1d7fa700 20 rgw notify: INFO: next queues processing will happen at: Thu Oct 5 19:30:04 2023 2023-10-05T19:29:34.236 INFO:teuthology.orchestra.run.smithi167.stderr: 2023-10-05T19:29:34.236 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.230+0000 7f1c1cff9700 20 reqs_thread_entry: start 2023-10-05T19:29:34.236 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.230+0000 7f1ce7539e40 20 get_system_obj_state: rctx=0x7fff390cc858 obj=default.rgw.meta:users.uid:foo state=0x5641f57310b0 s->prefetch_data=0 2023-10-05T19:29:34.236 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.230+0000 7f1ce7539e40 1 -- 172.21.15.167:0/2391133210 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:102 8.18 8:1e6ca5ed:users.uid::foo:head [call version.read in=11b,getxattrs,stat] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32) v8 -- 0x5641f5916260 con 0x5641f585e600 2023-10-05T19:29:34.237 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.234+0000 7f1ce586f700 1 -- 172.21.15.167:0/2391133210 <== osd.1 v2:172.21.15.167:6800/30035 35 ==== osd_op_reply(102 foo [call out=48b,getxattrs out=75b,stat out=16b] v0'0 uv15 ondisk = 0) v8 ==== 231+0+139 (crc 0 0 0) 0x7f1ce0050040 con 0x5641f585e600 2023-10-05T19:29:34.237 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.234+0000 7f1ce7539e40 20 get_system_obj_state: s->obj_tag was set empty 2023-10-05T19:29:34.237 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.234+0000 7f1ce7539e40 20 Read xattr: user.rgw.idtag 2023-10-05T19:29:34.237 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.234+0000 7f1ce7539e40 20 rados->read ofs=0 len=0 2023-10-05T19:29:34.238 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.234+0000 7f1ce7539e40 1 -- 172.21.15.167:0/2391133210 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:103 8.18 8:1e6ca5ed:users.uid::foo:head [call version.check_conds in=122b,call version.read in=11b,read 0~0,getxattrs] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32) v8 -- 0x5641f5918cf0 con 0x5641f585e600 2023-10-05T19:29:34.238 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.234+0000 7f1ce586f700 1 -- 172.21.15.167:0/2391133210 <== osd.1 v2:172.21.15.167:6800/30035 36 ==== osd_op_reply(103 foo [call,call out=48b,read 0~357 out=357b,getxattrs out=75b] v0'0 uv15 ondisk = 0) v8 ==== 273+0+480 (crc 0 0 0) 0x7f1ce0050040 con 0x5641f585e600 2023-10-05T19:29:34.238 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.234+0000 7f1ce7539e40 20 rados_obj.operate() r=0 bl.length=357 2023-10-05T19:29:34.238 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.234+0000 7f1ce7539e40 1 -- 172.21.15.167:0/2391133210 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:104 6.13 6:cea599e4:usage::usage.10:head [call rgw.user_usage_log_read in=63b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32) v8 -- 0x5641f5917ce0 con 0x7f1cd0029420 2023-10-05T19:29:34.239 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.234+0000 7f1ce586f700 1 -- 172.21.15.167:0/2391133210 <== osd.0 v2:172.21.15.118:6800/27690 29 ==== osd_op_reply(104 usage.10 [call out=1268b] v0'0 uv30 ondisk = 0) v8 ==== 152+0+1268 (crc 0 0 0) 0x7f1ce0050040 con 0x7f1cd0029420 2023-10-05T19:29:34.239 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.234+0000 7f1ce7539e40 1 -- 172.21.15.167:0/2391133210 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:105 7.1f 7:f95f44c2:::notify.0:head [watch unwatch cookie 94841292277408] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e32) v8 -- 0x5641f58d4650 con 0x7f1cd802d020 2023-10-05T19:29:34.243 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.238+0000 7f1ce486d700 1 -- 172.21.15.167:0/2391133210 <== osd.2 v2:172.21.15.167:6816/30037 17 ==== osd_op_reply(105 notify.0 [watch unwatch cookie 94841292277408] v32'236 uv234 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f1cd0048020 con 0x7f1cd802d020 2023-10-05T19:29:34.243 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.238+0000 7f1ce7539e40 20 remove_watcher() i=0 2023-10-05T19:29:34.243 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.238+0000 7f1ce7539e40 2 removed watcher, disabling cache 2023-10-05T19:29:34.243 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.238+0000 7f1ce7539e40 1 -- 172.21.15.167:0/2391133210 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:106 7.0 7:05bf5b68:::notify.1:head [watch unwatch cookie 94841292282144] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e32) v8 -- 0x5641f58946a0 con 0x5641f585e600 2023-10-05T19:29:34.245 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.242+0000 7f1ce586f700 1 -- 172.21.15.167:0/2391133210 <== osd.1 v2:172.21.15.167:6800/30035 37 ==== osd_op_reply(106 notify.1 [watch unwatch cookie 94841292282144] v32'236 uv234 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f1ce0050040 con 0x5641f585e600 2023-10-05T19:29:34.245 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.242+0000 7f1ce7539e40 20 remove_watcher() i=1 2023-10-05T19:29:34.246 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.242+0000 7f1ce7539e40 1 -- 172.21.15.167:0/2391133210 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:107 7.15 7:a93a5511:::notify.2:head [watch unwatch cookie 94841292284784] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e32) v8 -- 0x5641f5895920 con 0x5641f588f200 2023-10-05T19:29:34.248 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.242+0000 7f1ce506e700 1 -- 172.21.15.167:0/2391133210 <== osd.3 v2:172.21.15.167:6808/30036 24 ==== osd_op_reply(107 notify.2 [watch unwatch cookie 94841292284784] v32'236 uv234 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f1cd8014070 con 0x5641f588f200 2023-10-05T19:29:34.248 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.242+0000 7f1ce7539e40 20 remove_watcher() i=2 2023-10-05T19:29:34.248 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.242+0000 7f1ce7539e40 1 -- 172.21.15.167:0/2391133210 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:108 7.e 7:7759931f:::notify.3:head [watch unwatch cookie 94841292289856] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e32) v8 -- 0x5641f5896370 con 0x7f1cd802d020 2023-10-05T19:29:34.251 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.246+0000 7f1ce486d700 1 -- 172.21.15.167:0/2391133210 <== osd.2 v2:172.21.15.167:6816/30037 18 ==== osd_op_reply(108 notify.3 [watch unwatch cookie 94841292289856] v32'236 uv234 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f1cd0048020 con 0x7f1cd802d020 2023-10-05T19:29:34.251 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.246+0000 7f1ce7539e40 20 remove_watcher() i=3 2023-10-05T19:29:34.251 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.246+0000 7f1ce7539e40 1 -- 172.21.15.167:0/2391133210 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:109 7.d 7:b4812045:::notify.4:head [watch unwatch cookie 94841292280640] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e32) v8 -- 0x5641f5896370 con 0x5641f585e600 2023-10-05T19:29:34.254 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.250+0000 7f1ce586f700 1 -- 172.21.15.167:0/2391133210 <== osd.1 v2:172.21.15.167:6800/30035 38 ==== osd_op_reply(109 notify.4 [watch unwatch cookie 94841292280640] v32'236 uv234 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f1ce0050040 con 0x5641f585e600 2023-10-05T19:29:34.254 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.250+0000 7f1ce7539e40 20 remove_watcher() i=4 2023-10-05T19:29:34.254 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.250+0000 7f1ce7539e40 1 -- 172.21.15.167:0/2391133210 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:110 7.3 7:c609908c:::notify.5:head [watch unwatch cookie 94841292296272] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e32) v8 -- 0x5641f5895340 con 0x7f1cd0029420 2023-10-05T19:29:34.257 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.254+0000 7f1ce586f700 1 -- 172.21.15.167:0/2391133210 <== osd.0 v2:172.21.15.118:6800/27690 30 ==== osd_op_reply(110 notify.5 [watch unwatch cookie 94841292296272] v32'236 uv234 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f1ce0050040 con 0x7f1cd0029420 2023-10-05T19:29:34.257 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.254+0000 7f1ce7539e40 20 remove_watcher() i=5 2023-10-05T19:29:34.257 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.254+0000 7f1ce7539e40 1 -- 172.21.15.167:0/2391133210 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:111 7.14 7:2b04a3e9:::notify.6:head [watch unwatch cookie 94841292292608] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e32) v8 -- 0x5641f5896370 con 0x7f1cd0029420 2023-10-05T19:29:34.260 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.254+0000 7f1ce586f700 1 -- 172.21.15.167:0/2391133210 <== osd.0 v2:172.21.15.118:6800/27690 31 ==== osd_op_reply(111 notify.6 [watch unwatch cookie 94841292292608] v32'236 uv234 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f1ce0050040 con 0x7f1cd0029420 2023-10-05T19:29:34.260 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.254+0000 7f1ce7539e40 20 remove_watcher() i=6 2023-10-05T19:29:34.260 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.254+0000 7f1ce7539e40 1 -- 172.21.15.167:0/2391133210 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:112 7.9 7:93e5b521:::notify.7:head [watch unwatch cookie 94841292301120] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e32) v8 -- 0x5641f5899050 con 0x7f1cd0029420 2023-10-05T19:29:34.263 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.258+0000 7f1ce586f700 1 -- 172.21.15.167:0/2391133210 <== osd.0 v2:172.21.15.118:6800/27690 32 ==== osd_op_reply(112 notify.7 [watch unwatch cookie 94841292301120] v32'236 uv234 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f1ce0050040 con 0x7f1cd0029420 2023-10-05T19:29:34.263 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.258+0000 7f1ce7539e40 20 remove_watcher() i=7 2023-10-05T19:29:34.264 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.262+0000 7f1cb67e4700 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2023-10-05T19:29:34.264 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.262+0000 7f1ce7539e40 1 -- 172.21.15.167:0/2391133210 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:113 6.4 6:22d26bf9:::data_loggenerations_metadata:head [watch unwatch cookie 94841292297376] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e32) v8 -- 0x5641f5895340 con 0x5641f585e600 2023-10-05T19:29:34.268 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.262+0000 7f1ce586f700 1 -- 172.21.15.167:0/2391133210 <== osd.1 v2:172.21.15.167:6800/30035 39 ==== osd_op_reply(113 data_loggenerations_metadata [watch unwatch cookie 94841292297376] v32'230 uv1 ondisk = 0) v8 ==== 172+0+0 (crc 0 0 0) 0x7f1ce0050040 con 0x5641f585e600 2023-10-05T19:29:34.268 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.262+0000 7f1ce7539e40 1 -- 172.21.15.167:0/2391133210 >> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] conn(0x7f1cd0029420 msgr2=0x7f1cd0029890 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-05T19:29:34.268 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.262+0000 7f1ce7539e40 1 --2- 172.21.15.167:0/2391133210 >> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] conn(0x7f1cd0029420 0x7f1cd0029890 crc :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-05T19:29:34.268 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.262+0000 7f1ce7539e40 1 -- 172.21.15.167:0/2391133210 >> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] conn(0x5641f585e600 msgr2=0x5641f588b7f0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-05T19:29:34.268 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.262+0000 7f1ce7539e40 1 --2- 172.21.15.167:0/2391133210 >> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] conn(0x5641f585e600 0x5641f588b7f0 crc :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-05T19:29:34.268 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.262+0000 7f1ce7539e40 1 -- 172.21.15.167:0/2391133210 >> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] conn(0x7f1cd802d020 msgr2=0x7f1cd8024830 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-05T19:29:34.268 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.262+0000 7f1ce7539e40 1 --2- 172.21.15.167:0/2391133210 >> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] conn(0x7f1cd802d020 0x7f1cd8024830 crc :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-05T19:29:34.268 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.262+0000 7f1ce7539e40 1 -- 172.21.15.167:0/2391133210 >> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] conn(0x5641f588f200 msgr2=0x5641f58915e0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-05T19:29:34.269 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.262+0000 7f1ce7539e40 1 --2- 172.21.15.167:0/2391133210 >> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] conn(0x5641f588f200 0x5641f58915e0 crc :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-05T19:29:34.269 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.262+0000 7f1ce7539e40 1 -- 172.21.15.167:0/2391133210 >> [v2:172.21.15.167:6824/29889,v1:172.21.15.167:6825/29889] conn(0x7f1cc8030c40 msgr2=0x7f1cc80330f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-05T19:29:34.269 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.262+0000 7f1ce7539e40 1 --2- 172.21.15.167:0/2391133210 >> [v2:172.21.15.167:6824/29889,v1:172.21.15.167:6825/29889] conn(0x7f1cc8030c40 0x7f1cc80330f0 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7f1cd0000f80 tx=0x7f1cd0014000 comp rx=0 tx=0).stop 2023-10-05T19:29:34.269 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.262+0000 7f1ce586f700 1 -- 172.21.15.167:0/2391133210 reap_dead start 2023-10-05T19:29:34.269 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.262+0000 7f1ce7539e40 1 -- 172.21.15.167:0/2391133210 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5641f585a020 msgr2=0x5641f587fd90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-05T19:29:34.269 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.262+0000 7f1ce7539e40 1 --2- 172.21.15.167:0/2391133210 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5641f585a020 0x5641f587fd90 secure :-1 s=READY pgs=605 cs=0 l=1 rev1=1 crypto rx=0x7f1cd800b960 tx=0x7f1cd8014000 comp rx=0 tx=0).stop 2023-10-05T19:29:34.269 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.262+0000 7f1ce7539e40 1 -- 172.21.15.167:0/2391133210 shutdown_connections 2023-10-05T19:29:34.269 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.262+0000 7f1ce7539e40 1 --2- 172.21.15.167:0/2391133210 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5641f585a020 0x5641f587fd90 unknown :-1 s=CLOSED pgs=605 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-05T19:29:34.269 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.262+0000 7f1ce7539e40 1 -- 172.21.15.167:0/2391133210 >> 172.21.15.167:0/2391133210 conn(0x5641f57cd870 msgr2=0x5641f5873640 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-05T19:29:34.269 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.262+0000 7f1ce7539e40 1 -- 172.21.15.167:0/2391133210 shutdown_connections 2023-10-05T19:29:34.269 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.262+0000 7f1ce7539e40 1 -- 172.21.15.167:0/2391133210 wait complete. 2023-10-05T19:29:34.270 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.266+0000 7f1ce7539e40 1 -- 172.21.15.167:0/1343801672 >> [v2:172.21.15.167:6824/29889,v1:172.21.15.167:6825/29889] conn(0x7f1c2c030c50 msgr2=0x7f1c2c033100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-05T19:29:34.270 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.266+0000 7f1ce7539e40 1 --2- 172.21.15.167:0/1343801672 >> [v2:172.21.15.167:6824/29889,v1:172.21.15.167:6825/29889] conn(0x7f1c2c030c50 0x7f1c2c033100 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x5641f589a9e0 tx=0x7f1ce0000f90 comp rx=0 tx=0).stop 2023-10-05T19:29:34.271 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.266+0000 7f1ce7539e40 1 -- 172.21.15.167:0/1343801672 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5641f58a7f30 msgr2=0x5641f58adb30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-05T19:29:34.271 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.266+0000 7f1ce7539e40 1 --2- 172.21.15.167:0/1343801672 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5641f58a7f30 0x5641f58adb30 secure :-1 s=READY pgs=607 cs=0 l=1 rev1=1 crypto rx=0x7f1cd00059e0 tx=0x7f1cd0042000 comp rx=0 tx=0).stop 2023-10-05T19:29:34.271 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.266+0000 7f1ce7539e40 1 -- 172.21.15.167:0/1343801672 shutdown_connections 2023-10-05T19:29:34.271 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.266+0000 7f1ce7539e40 1 --2- 172.21.15.167:0/1343801672 >> [v2:172.21.15.167:6824/29889,v1:172.21.15.167:6825/29889] conn(0x7f1c2c030c50 0x7f1c2c033100 unknown :-1 s=CLOSED pgs=209 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-05T19:29:34.271 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.266+0000 7f1ce7539e40 1 --2- 172.21.15.167:0/1343801672 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5641f58a7f30 0x5641f58adb30 unknown :-1 s=CLOSED pgs=607 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-05T19:29:34.271 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.266+0000 7f1ce7539e40 1 -- 172.21.15.167:0/1343801672 >> 172.21.15.167:0/1343801672 conn(0x5641f58a32f0 msgr2=0x5641f58a13f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-05T19:29:34.271 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.266+0000 7f1ce7539e40 1 -- 172.21.15.167:0/1343801672 shutdown_connections 2023-10-05T19:29:34.271 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.266+0000 7f1ce7539e40 1 -- 172.21.15.167:0/1343801672 wait complete. 2023-10-05T19:29:34.272 INFO:teuthology.orchestra.run.smithi167.stdout:{ 2023-10-05T19:29:34.272 INFO:teuthology.orchestra.run.smithi167.stdout: "entries": [ 2023-10-05T19:29:34.272 INFO:teuthology.orchestra.run.smithi167.stdout: { 2023-10-05T19:29:34.272 INFO:teuthology.orchestra.run.smithi167.stdout: "user": "foo", 2023-10-05T19:29:34.272 INFO:teuthology.orchestra.run.smithi167.stdout: "buckets": [ 2023-10-05T19:29:34.272 INFO:teuthology.orchestra.run.smithi167.stdout: { 2023-10-05T19:29:34.272 INFO:teuthology.orchestra.run.smithi167.stdout: "bucket": "", 2023-10-05T19:29:34.272 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2023-10-05T19:00:00.000000Z", 2023-10-05T19:29:34.272 INFO:teuthology.orchestra.run.smithi167.stdout: "epoch": 1696532400, 2023-10-05T19:29:34.273 INFO:teuthology.orchestra.run.smithi167.stdout: "owner": "foo", 2023-10-05T19:29:34.273 INFO:teuthology.orchestra.run.smithi167.stdout: "categories": [] 2023-10-05T19:29:34.273 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2023-10-05T19:29:34.273 INFO:teuthology.orchestra.run.smithi167.stdout: { 2023-10-05T19:29:34.273 INFO:teuthology.orchestra.run.smithi167.stdout: "bucket": "myfoo", 2023-10-05T19:29:34.273 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2023-10-05T19:00:00.000000Z", 2023-10-05T19:29:34.273 INFO:teuthology.orchestra.run.smithi167.stdout: "epoch": 1696532400, 2023-10-05T19:29:34.273 INFO:teuthology.orchestra.run.smithi167.stdout: "owner": "foo", 2023-10-05T19:29:34.273 INFO:teuthology.orchestra.run.smithi167.stdout: "categories": [ 2023-10-05T19:29:34.273 INFO:teuthology.orchestra.run.smithi167.stdout: { 2023-10-05T19:29:34.273 INFO:teuthology.orchestra.run.smithi167.stdout: "category": "delete_obj", 2023-10-05T19:29:34.274 INFO:teuthology.orchestra.run.smithi167.stdout: "bytes_sent": 0, 2023-10-05T19:29:34.274 INFO:teuthology.orchestra.run.smithi167.stdout: "bytes_received": 0, 2023-10-05T19:29:34.274 INFO:teuthology.orchestra.run.smithi167.stdout: "ops": 4, 2023-10-05T19:29:34.274 INFO:teuthology.orchestra.run.smithi167.stdout: "successful_ops": 4 2023-10-05T19:29:34.274 INFO:teuthology.orchestra.run.smithi167.stdout: } 2023-10-05T19:29:34.274 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2023-10-05T19:29:34.274 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2023-10-05T19:29:34.274 INFO:teuthology.orchestra.run.smithi167.stdout: { 2023-10-05T19:29:34.274 INFO:teuthology.orchestra.run.smithi167.stdout: "bucket": "myfoo2", 2023-10-05T19:29:34.274 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2023-10-05T19:00:00.000000Z", 2023-10-05T19:29:34.275 INFO:teuthology.orchestra.run.smithi167.stdout: "epoch": 1696532400, 2023-10-05T19:29:34.275 INFO:teuthology.orchestra.run.smithi167.stdout: "owner": "foo", 2023-10-05T19:29:34.275 INFO:teuthology.orchestra.run.smithi167.stdout: "categories": [] 2023-10-05T19:29:34.275 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2023-10-05T19:29:34.275 INFO:teuthology.orchestra.run.smithi167.stdout: { 2023-10-05T19:29:34.275 INFO:teuthology.orchestra.run.smithi167.stdout: "bucket": "myfoo3", 2023-10-05T19:29:34.275 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2023-10-05T19:00:00.000000Z", 2023-10-05T19:29:34.275 INFO:teuthology.orchestra.run.smithi167.stdout: "epoch": 1696532400, 2023-10-05T19:29:34.275 INFO:teuthology.orchestra.run.smithi167.stdout: "owner": "foo", 2023-10-05T19:29:34.275 INFO:teuthology.orchestra.run.smithi167.stdout: "categories": [] 2023-10-05T19:29:34.275 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2023-10-05T19:29:34.276 INFO:teuthology.orchestra.run.smithi167.stdout: { 2023-10-05T19:29:34.276 INFO:teuthology.orchestra.run.smithi167.stdout: "bucket": "myfoo4", 2023-10-05T19:29:34.276 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2023-10-05T19:00:00.000000Z", 2023-10-05T19:29:34.276 INFO:teuthology.orchestra.run.smithi167.stdout: "epoch": 1696532400, 2023-10-05T19:29:34.276 INFO:teuthology.orchestra.run.smithi167.stdout: "owner": "foo", 2023-10-05T19:29:34.276 INFO:teuthology.orchestra.run.smithi167.stdout: "categories": [] 2023-10-05T19:29:34.276 INFO:teuthology.orchestra.run.smithi167.stdout: } 2023-10-05T19:29:34.276 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2023-10-05T19:29:34.276 INFO:teuthology.orchestra.run.smithi167.stdout: } 2023-10-05T19:29:34.276 INFO:teuthology.orchestra.run.smithi167.stdout: ], 2023-10-05T19:29:34.276 INFO:teuthology.orchestra.run.smithi167.stdout: "summary": [ 2023-10-05T19:29:34.276 INFO:teuthology.orchestra.run.smithi167.stdout: { 2023-10-05T19:29:34.277 INFO:teuthology.orchestra.run.smithi167.stdout: "user": "foo", 2023-10-05T19:29:34.277 INFO:teuthology.orchestra.run.smithi167.stdout: "categories": [ 2023-10-05T19:29:34.277 INFO:teuthology.orchestra.run.smithi167.stdout: { 2023-10-05T19:29:34.277 INFO:teuthology.orchestra.run.smithi167.stdout: "category": "delete_obj", 2023-10-05T19:29:34.277 INFO:teuthology.orchestra.run.smithi167.stdout: "bytes_sent": 0, 2023-10-05T19:29:34.277 INFO:teuthology.orchestra.run.smithi167.stdout: "bytes_received": 0, 2023-10-05T19:29:34.277 INFO:teuthology.orchestra.run.smithi167.stdout: "ops": 4, 2023-10-05T19:29:34.277 INFO:teuthology.orchestra.run.smithi167.stdout: "successful_ops": 4 2023-10-05T19:29:34.277 INFO:teuthology.orchestra.run.smithi167.stdout: } 2023-10-05T19:29:34.277 INFO:teuthology.orchestra.run.smithi167.stdout: ], 2023-10-05T19:29:34.277 INFO:teuthology.orchestra.run.smithi167.stdout: "total": { 2023-10-05T19:29:34.277 INFO:teuthology.orchestra.run.smithi167.stdout: "bytes_sent": 0, 2023-10-05T19:29:34.277 INFO:teuthology.orchestra.run.smithi167.stdout: "bytes_received": 0, 2023-10-05T19:29:34.278 INFO:teuthology.orchestra.run.smithi167.stdout: "ops": 4, 2023-10-05T19:29:34.278 INFO:teuthology.orchestra.run.smithi167.stdout: "successful_ops": 4 2023-10-05T19:29:34.278 INFO:teuthology.orchestra.run.smithi167.stdout: } 2023-10-05T19:29:34.278 INFO:teuthology.orchestra.run.smithi167.stdout: } 2023-10-05T19:29:34.278 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2023-10-05T19:29:34.278 INFO:teuthology.orchestra.run.smithi167.stdout:} 2023-10-05T19:29:34.279 DEBUG:tasks.util.rgw: json result: {'entries': [{'user': 'foo', 'buckets': [{'bucket': '', 'time': '2023-10-05T19:00:00.000000Z', 'epoch': 1696532400, 'owner': 'foo', 'categories': []}, {'bucket': 'myfoo', 'time': '2023-10-05T19:00:00.000000Z', 'epoch': 1696532400, 'owner': 'foo', 'categories': [{'category': 'delete_obj', 'bytes_sent': 0, 'bytes_received': 0, 'ops': 4, 'successful_ops': 4}]}, {'bucket': 'myfoo2', 'time': '2023-10-05T19:00:00.000000Z', 'epoch': 1696532400, 'owner': 'foo', 'categories': []}, {'bucket': 'myfoo3', 'time': '2023-10-05T19:00:00.000000Z', 'epoch': 1696532400, 'owner': 'foo', 'categories': []}, {'bucket': 'myfoo4', 'time': '2023-10-05T19:00:00.000000Z', 'epoch': 1696532400, 'owner': 'foo', 'categories': []}]}], 'summary': [{'user': 'foo', 'categories': [{'category': 'delete_obj', 'bytes_sent': 0, 'bytes_received': 0, 'ops': 4, 'successful_ops': 4}], 'total': {'bytes_sent': 0, 'bytes_received': 0, 'ops': 4, 'successful_ops': 4}}]} 2023-10-05T19:29:34.279 INFO:tasks.util.rgw:rgwadmin: client.0 : ['usage', 'show', '--uid', 'foo', '--categories', 'delete_bucket'] 2023-10-05T19:29:34.279 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'ceph', 'usage', 'show', '--uid', 'foo', '--categories', 'delete_bucket'] 2023-10-05T19:29:34.279 DEBUG:teuthology.orchestra.run.smithi167:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster ceph usage show --uid foo --categories delete_bucket 2023-10-05T19:29:34.320 INFO:teuthology.orchestra.run.smithi167.stderr:ignoring --setuser ceph since I am not root 2023-10-05T19:29:34.320 INFO:teuthology.orchestra.run.smithi167.stderr:ignoring --setgroup ceph since I am not root 2023-10-05T19:29:34.320 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.318+0000 7f8e616f9e40 1 Processor -- start 2023-10-05T19:29:34.320 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.318+0000 7f8e616f9e40 1 -- start start 2023-10-05T19:29:34.321 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.318+0000 7f8e616f9e40 1 --2- >> v2:172.21.15.118:3300/0 conn(0x56003d1ccd10 0x56003d1cd0e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-05T19:29:34.321 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.318+0000 7f8e616f9e40 1 -- --> v1:172.21.15.118:6789/0 -- auth(proto 0 26 bytes epoch 0) v1 -- 0x56003d11abd0 con 0x56003d1cd620 2023-10-05T19:29:34.321 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.318+0000 7f8e616f9e40 1 -- --> v2:172.21.15.118:3300/0 -- mon_getmap magic: 0 v1 -- 0x56003d0d7ee0 con 0x56003d1ccd10 2023-10-05T19:29:34.322 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.318+0000 7f8e5f22e700 1 --2- >> v2:172.21.15.118:3300/0 conn(0x56003d1ccd10 0x56003d1cd0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-05T19:29:34.322 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.318+0000 7f8e5ea2d700 1 --1- >> v1:172.21.15.118:6789/0 conn(0x56003d1cd620 0x56003d1ce0c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.118:6789/0 says I am v1:172.21.15.167:33298/0 (socket says 172.21.15.167:33298) 2023-10-05T19:29:34.322 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.318+0000 7f8e5ea2d700 1 -- 172.21.15.167:0/2247620119 learned_addr learned my addr 172.21.15.167:0/2247620119 (peer_addr_for_me v1:172.21.15.167:0/0) 2023-10-05T19:29:34.323 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.318+0000 7f8e5e22c700 1 -- 172.21.15.167:0/2247620119 <== mon.0 v1:172.21.15.118:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1604275072 0 0) 0x56003d11abd0 con 0x56003d1cd620 2023-10-05T19:29:34.323 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.318+0000 7f8e5e22c700 1 -- 172.21.15.167:0/2247620119 --> v1:172.21.15.118:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8e40001e10 con 0x56003d1cd620 2023-10-05T19:29:34.323 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.318+0000 7f8e5f22e700 1 -- 172.21.15.167:0/2247620119 >> v1:172.21.15.118:6789/0 conn(0x56003d1cd620 legacy=0x56003d1ce0c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-05T19:29:34.323 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.318+0000 7f8e5f22e700 1 -- 172.21.15.167:0/2247620119 --> v2:172.21.15.118:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x56003d1ce7d0 con 0x56003d1ccd10 2023-10-05T19:29:34.323 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.318+0000 7f8e5f22e700 1 --2- 172.21.15.167:0/2247620119 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x56003d1ccd10 0x56003d1cd0e0 secure :-1 s=READY pgs=609 cs=0 l=1 rev1=1 crypto rx=0x7f8e5800a5d0 tx=0x7f8e58005a50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d2936f99d0bd7901 server_cookie=0 in_seq=0 out_seq=0 2023-10-05T19:29:34.323 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.318+0000 7f8e5e22c700 1 -- 172.21.15.167:0/2247620119 <== mon.0 v2:172.21.15.118:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f8e58007610 con 0x56003d1ccd10 2023-10-05T19:29:34.324 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.318+0000 7f8e5e22c700 1 -- 172.21.15.167:0/2247620119 <== mon.0 v2:172.21.15.118:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8e58007770 con 0x56003d1ccd10 2023-10-05T19:29:34.324 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.318+0000 7f8e5e22c700 1 -- 172.21.15.167:0/2247620119 <== mon.0 v2:172.21.15.118:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f8e58007a40 con 0x56003d1ccd10 2023-10-05T19:29:34.325 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.318+0000 7f8e616f9e40 1 -- 172.21.15.167:0/2247620119 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x56003d1ccd10 msgr2=0x56003d1cd0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-05T19:29:34.325 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.318+0000 7f8e616f9e40 1 --2- 172.21.15.167:0/2247620119 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x56003d1ccd10 0x56003d1cd0e0 secure :-1 s=READY pgs=609 cs=0 l=1 rev1=1 crypto rx=0x7f8e5800a5d0 tx=0x7f8e58005a50 comp rx=0 tx=0).stop 2023-10-05T19:29:34.325 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.318+0000 7f8e616f9e40 1 -- 172.21.15.167:0/2247620119 shutdown_connections 2023-10-05T19:29:34.325 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.318+0000 7f8e616f9e40 1 --2- 172.21.15.167:0/2247620119 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x56003d1ccd10 0x56003d1cd0e0 unknown :-1 s=CLOSED pgs=609 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-05T19:29:34.325 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.318+0000 7f8e616f9e40 1 -- 172.21.15.167:0/2247620119 >> 172.21.15.167:0/2247620119 conn(0x56003d1c8730 msgr2=0x56003d1cab40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-05T19:29:34.325 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.318+0000 7f8e616f9e40 1 -- 172.21.15.167:0/2247620119 shutdown_connections 2023-10-05T19:29:34.325 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.318+0000 7f8e616f9e40 1 -- 172.21.15.167:0/2247620119 wait complete. 2023-10-05T19:29:34.326 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.322+0000 7f8e616f9e40 1 Processor -- start 2023-10-05T19:29:34.326 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.322+0000 7f8e616f9e40 1 -- start start 2023-10-05T19:29:34.326 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.322+0000 7f8e616f9e40 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x56003d1ccd10 0x56003d1d2ab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-05T19:29:34.326 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.322+0000 7f8e616f9e40 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x56003d09be30 con 0x56003d1ccd10 2023-10-05T19:29:34.327 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.322+0000 7f8e5f22e700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x56003d1ccd10 0x56003d1d2ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-05T19:29:34.327 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.322+0000 7f8e5f22e700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x56003d1ccd10 0x56003d1d2ab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.118:3300/0 says I am v2:172.21.15.167:56670/0 (socket says 172.21.15.167:56670) 2023-10-05T19:29:34.327 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.322+0000 7f8e5f22e700 1 -- 172.21.15.167:0/1407465063 learned_addr learned my addr 172.21.15.167:0/1407465063 (peer_addr_for_me v2:172.21.15.167:0/0) 2023-10-05T19:29:34.327 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.322+0000 7f8e5f22e700 1 -- 172.21.15.167:0/1407465063 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8e58005cf0 con 0x56003d1ccd10 2023-10-05T19:29:34.327 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.322+0000 7f8e5f22e700 1 --2- 172.21.15.167:0/1407465063 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x56003d1ccd10 0x56003d1d2ab0 secure :-1 s=READY pgs=610 cs=0 l=1 rev1=1 crypto rx=0x7f8e58000c00 tx=0x7f8e58014000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=af6431f2788848b2 server_cookie=0 in_seq=0 out_seq=0 2023-10-05T19:29:34.327 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.322+0000 7f8e5d22a700 1 -- 172.21.15.167:0/1407465063 <== mon.0 v2:172.21.15.118:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f8e580078e0 con 0x56003d1ccd10 2023-10-05T19:29:34.327 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.322+0000 7f8e5d22a700 1 -- 172.21.15.167:0/1407465063 <== mon.0 v2:172.21.15.118:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8e58007a40 con 0x56003d1ccd10 2023-10-05T19:29:34.328 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.322+0000 7f8e5d22a700 1 -- 172.21.15.167:0/1407465063 <== mon.0 v2:172.21.15.118:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f8e58007bd0 con 0x56003d1ccd10 2023-10-05T19:29:34.329 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.322+0000 7f8e616f9e40 1 -- 172.21.15.167:0/1407465063 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x56003d1ccd10 msgr2=0x56003d1d2ab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-05T19:29:34.329 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.322+0000 7f8e616f9e40 1 --2- 172.21.15.167:0/1407465063 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x56003d1ccd10 0x56003d1d2ab0 secure :-1 s=READY pgs=610 cs=0 l=1 rev1=1 crypto rx=0x7f8e58000c00 tx=0x7f8e58014000 comp rx=0 tx=0).stop 2023-10-05T19:29:34.329 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.322+0000 7f8e616f9e40 1 -- 172.21.15.167:0/1407465063 shutdown_connections 2023-10-05T19:29:34.329 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.322+0000 7f8e616f9e40 1 --2- 172.21.15.167:0/1407465063 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x56003d1ccd10 0x56003d1d2ab0 unknown :-1 s=CLOSED pgs=610 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-05T19:29:34.329 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.322+0000 7f8e616f9e40 1 -- 172.21.15.167:0/1407465063 >> 172.21.15.167:0/1407465063 conn(0x56003d1c8730 msgr2=0x56003d1e8970 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-05T19:29:34.330 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.322+0000 7f8e616f9e40 1 -- 172.21.15.167:0/1407465063 shutdown_connections 2023-10-05T19:29:34.330 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.322+0000 7f8e616f9e40 1 -- 172.21.15.167:0/1407465063 wait complete. 2023-10-05T19:29:34.330 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.322+0000 7f8e616f9e40 1 Processor -- start 2023-10-05T19:29:34.330 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.322+0000 7f8e616f9e40 1 -- start start 2023-10-05T19:29:34.330 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.322+0000 7f8e616f9e40 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x56003d1ccd10 0x56003d1f5c60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-05T19:29:34.330 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.322+0000 7f8e616f9e40 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x56003d06eed0 con 0x56003d1ccd10 2023-10-05T19:29:34.330 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.326+0000 7f8e5f22e700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x56003d1ccd10 0x56003d1f5c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-05T19:29:34.330 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.326+0000 7f8e5f22e700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x56003d1ccd10 0x56003d1f5c60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.118:3300/0 says I am v2:172.21.15.167:56676/0 (socket says 172.21.15.167:56676) 2023-10-05T19:29:34.330 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.326+0000 7f8e5f22e700 1 -- 172.21.15.167:0/2399019461 learned_addr learned my addr 172.21.15.167:0/2399019461 (peer_addr_for_me v2:172.21.15.167:0/0) 2023-10-05T19:29:34.331 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.326+0000 7f8e5f22e700 1 -- 172.21.15.167:0/2399019461 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x56003d1ce7d0 con 0x56003d1ccd10 2023-10-05T19:29:34.331 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.326+0000 7f8e5f22e700 1 --2- 172.21.15.167:0/2399019461 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x56003d1ccd10 0x56003d1f5c60 secure :-1 s=READY pgs=611 cs=0 l=1 rev1=1 crypto rx=0x7f8e5801f040 tx=0x7f8e5801c000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-05T19:29:34.331 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.326+0000 7f8e577fe700 1 -- 172.21.15.167:0/2399019461 <== mon.0 v2:172.21.15.118:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f8e58007bb0 con 0x56003d1ccd10 2023-10-05T19:29:34.332 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.326+0000 7f8e577fe700 1 -- 172.21.15.167:0/2399019461 <== mon.0 v2:172.21.15.118:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8e58029070 con 0x56003d1ccd10 2023-10-05T19:29:34.332 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.326+0000 7f8e616f9e40 1 -- 172.21.15.167:0/2399019461 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x56003d1f61a0 con 0x56003d1ccd10 2023-10-05T19:29:34.332 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.326+0000 7f8e577fe700 1 -- 172.21.15.167:0/2399019461 <== mon.0 v2:172.21.15.118:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f8e58012610 con 0x56003d1ccd10 2023-10-05T19:29:34.332 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.326+0000 7f8e616f9e40 1 -- 172.21.15.167:0/2399019461 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x56003d1f65e0 con 0x56003d1ccd10 2023-10-05T19:29:34.333 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.326+0000 7f8e616f9e40 20 rados->read ofs=0 len=0 2023-10-05T19:29:34.333 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.326+0000 7f8e577fe700 1 -- 172.21.15.167:0/2399019461 <== mon.0 v2:172.21.15.118:3300/0 4 ==== mgrmap(e 5) v1 ==== 40329+0+0 (secure 0 0 0) 0x7f8e58012770 con 0x56003d1ccd10 2023-10-05T19:29:34.333 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.326+0000 7f8e577fe700 1 --2- 172.21.15.167:0/2399019461 >> [v2:172.21.15.167:6824/29889,v1:172.21.15.167:6825/29889] conn(0x7f8e44030ca0 0x7f8e44033150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-05T19:29:34.333 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.326+0000 7f8e577fe700 1 -- 172.21.15.167:0/2399019461 <== mon.0 v2:172.21.15.118:3300/0 5 ==== osd_map(32..32 src has 1..32) v4 ==== 6260+0+0 (secure 0 0 0) 0x7f8e5802b070 con 0x56003d1ccd10 2023-10-05T19:29:34.333 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.330+0000 7f8e5ea2d700 1 --2- 172.21.15.167:0/2399019461 >> [v2:172.21.15.167:6824/29889,v1:172.21.15.167:6825/29889] conn(0x7f8e44030ca0 0x7f8e44033150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-05T19:29:34.333 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.330+0000 7f8e616f9e40 1 --2- 172.21.15.167:0/2399019461 >> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] conn(0x56003d1cd620 0x56003d2016f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-05T19:29:34.333 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.330+0000 7f8e616f9e40 1 -- 172.21.15.167:0/2399019461 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:1 5.12 5:49953fa1:::default.realm:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32) v8 -- 0x56003d201d60 con 0x56003d1cd620 2023-10-05T19:29:34.334 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.330+0000 7f8e5fa2f700 1 --2- 172.21.15.167:0/2399019461 >> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] conn(0x56003d1cd620 0x56003d2016f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-05T19:29:34.335 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.330+0000 7f8e5fa2f700 1 --2- 172.21.15.167:0/2399019461 >> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] conn(0x56003d1cd620 0x56003d2016f0 crc :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-05T19:29:34.335 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.330+0000 7f8e5ea2d700 1 --2- 172.21.15.167:0/2399019461 >> [v2:172.21.15.167:6824/29889,v1:172.21.15.167:6825/29889] conn(0x7f8e44030ca0 0x7f8e44033150 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7f8e480044f0 tx=0x7f8e48006690 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-05T19:29:34.335 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.330+0000 7f8e5fa2f700 1 -- 172.21.15.167:0/2399019461 <== osd.1 v2:172.21.15.167:6800/30035 1 ==== osd_op_reply(1 default.realm [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 ==== 157+0+0 (crc 0 0 0) 0x7f8e5000a500 con 0x56003d1cd620 2023-10-05T19:29:34.335 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.330+0000 7f8e616f9e40 20 rados_obj.operate() r=-2 bl.length=0 2023-10-05T19:29:34.335 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.330+0000 7f8e616f9e40 20 realm 2023-10-05T19:29:34.335 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.330+0000 7f8e616f9e40 20 rados->read ofs=0 len=0 2023-10-05T19:29:34.335 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.330+0000 7f8e616f9e40 1 -- 172.21.15.167:0/2399019461 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:2 5.12 5:49953fa1:::default.realm:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32) v8 -- 0x56003d1fed50 con 0x56003d1cd620 2023-10-05T19:29:34.335 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.330+0000 7f8e5fa2f700 1 -- 172.21.15.167:0/2399019461 <== osd.1 v2:172.21.15.167:6800/30035 2 ==== osd_op_reply(2 default.realm [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 ==== 157+0+0 (crc 0 0 0) 0x7f8e5000a500 con 0x56003d1cd620 2023-10-05T19:29:34.336 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.330+0000 7f8e616f9e40 20 rados_obj.operate() r=-2 bl.length=0 2023-10-05T19:29:34.336 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.330+0000 7f8e616f9e40 4 RGWPeriod::init failed to init realm id : (2) No such file or directory 2023-10-05T19:29:34.336 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.330+0000 7f8e616f9e40 20 rados->read ofs=0 len=0 2023-10-05T19:29:34.336 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.330+0000 7f8e616f9e40 1 -- 172.21.15.167:0/2399019461 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:3 5.12 5:49953fa1:::default.realm:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32) v8 -- 0x56003d202840 con 0x56003d1cd620 2023-10-05T19:29:34.337 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.330+0000 7f8e5fa2f700 1 -- 172.21.15.167:0/2399019461 <== osd.1 v2:172.21.15.167:6800/30035 3 ==== osd_op_reply(3 default.realm [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 ==== 157+0+0 (crc 0 0 0) 0x7f8e5000a500 con 0x56003d1cd620 2023-10-05T19:29:34.337 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.330+0000 7f8e616f9e40 20 rados_obj.operate() r=-2 bl.length=0 2023-10-05T19:29:34.337 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.330+0000 7f8e616f9e40 20 rados->read ofs=0 len=0 2023-10-05T19:29:34.337 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.330+0000 7f8e616f9e40 1 -- 172.21.15.167:0/2399019461 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:4 5.f 5:f43b6ece:::zone_names.default:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32) v8 -- 0x56003d202420 con 0x56003d1cd620 2023-10-05T19:29:34.338 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.330+0000 7f8e5fa2f700 1 -- 172.21.15.167:0/2399019461 <== osd.1 v2:172.21.15.167:6800/30035 4 ==== osd_op_reply(4 zone_names.default [read 0~46 out=46b] v0'0 uv1 ondisk = 0) v8 ==== 162+0+46 (crc 0 0 0) 0x7f8e5000a500 con 0x56003d1cd620 2023-10-05T19:29:34.338 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.330+0000 7f8e616f9e40 20 rados_obj.operate() r=0 bl.length=46 2023-10-05T19:29:34.338 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.330+0000 7f8e616f9e40 20 rados->read ofs=0 len=0 2023-10-05T19:29:34.338 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.330+0000 7f8e616f9e40 1 -- 172.21.15.167:0/2399019461 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:5 5.1d 5:b8374eb4:::zone_info.9a9df965-6612-45d4-a45f-6e3c889bacc2:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32) v8 -- 0x56003d2032b0 con 0x56003d1cd620 2023-10-05T19:29:34.338 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.334+0000 7f8e5fa2f700 1 -- 172.21.15.167:0/2399019461 <== osd.1 v2:172.21.15.167:6800/30035 5 ==== osd_op_reply(5 zone_info.9a9df965-6612-45d4-a45f-6e3c889bacc2 [read 0~874 out=874b] v0'0 uv2 ondisk = 0) v8 ==== 190+0+874 (crc 0 0 0) 0x7f8e5000a500 con 0x56003d1cd620 2023-10-05T19:29:34.338 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.334+0000 7f8e616f9e40 20 rados_obj.operate() r=0 bl.length=874 2023-10-05T19:29:34.338 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.334+0000 7f8e616f9e40 20 searching for the correct realm 2023-10-05T19:29:34.338 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.334+0000 7f8e616f9e40 1 --2- 172.21.15.167:0/2399019461 >> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] conn(0x56003d205100 0x56003d2074e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-05T19:29:34.338 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.334+0000 7f8e616f9e40 1 -- 172.21.15.167:0/2399019461 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:6 5.0 5:00000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x56003d207a20 con 0x56003d205100 2023-10-05T19:29:34.338 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.334+0000 7f8e5f22e700 1 --2- 172.21.15.167:0/2399019461 >> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] conn(0x56003d205100 0x56003d2074e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-05T19:29:34.339 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.334+0000 7f8e5f22e700 1 --2- 172.21.15.167:0/2399019461 >> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] conn(0x56003d205100 0x56003d2074e0 crc :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-05T19:29:34.339 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.334+0000 7f8e5f22e700 1 -- 172.21.15.167:0/2399019461 <== osd.3 v2:172.21.15.167:6808/30036 1 ==== osd_op_reply(6 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f8e58041040 con 0x56003d205100 2023-10-05T19:29:34.339 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.334+0000 7f8e5f22e700 1 --2- 172.21.15.167:0/2399019461 >> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] conn(0x7f8e58007030 0x7f8e58031490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-05T19:29:34.340 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.334+0000 7f8e5f22e700 1 -- 172.21.15.167:0/2399019461 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:7 5.10 5:08000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x56003d207a20 con 0x7f8e58007030 2023-10-05T19:29:34.340 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.334+0000 7f8e5ea2d700 1 --2- 172.21.15.167:0/2399019461 >> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] conn(0x7f8e58007030 0x7f8e58031490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-05T19:29:34.340 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.334+0000 7f8e5ea2d700 1 --2- 172.21.15.167:0/2399019461 >> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] conn(0x7f8e58007030 0x7f8e58031490 crc :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-05T19:29:34.341 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.338+0000 7f8e5ea2d700 1 -- 172.21.15.167:0/2399019461 <== osd.2 v2:172.21.15.167:6816/30037 1 ==== osd_op_reply(7 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f8e48004520 con 0x7f8e58007030 2023-10-05T19:29:34.341 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.338+0000 7f8e5ea2d700 1 --2- 172.21.15.167:0/2399019461 >> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] conn(0x7f8e48011040 0x7f8e48014950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-05T19:29:34.341 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.338+0000 7f8e5ea2d700 1 -- 172.21.15.167:0/2399019461 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:8 5.8 5:10000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x56003d207a20 con 0x7f8e48011040 2023-10-05T19:29:34.341 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.338+0000 7f8e5fa2f700 1 --2- 172.21.15.167:0/2399019461 >> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] conn(0x7f8e48011040 0x7f8e48014950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-05T19:29:34.342 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.338+0000 7f8e5fa2f700 1 --2- 172.21.15.167:0/2399019461 >> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] conn(0x7f8e48011040 0x7f8e48014950 crc :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-05T19:29:34.342 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.338+0000 7f8e5fa2f700 1 -- 172.21.15.167:0/2399019461 <== osd.0 v2:172.21.15.118:6800/27690 1 ==== osd_op_reply(8 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f8e50014040 con 0x7f8e48011040 2023-10-05T19:29:34.342 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.338+0000 7f8e5fa2f700 1 -- 172.21.15.167:0/2399019461 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:9 5.18 5:18000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x56003d207a20 con 0x56003d1cd620 2023-10-05T19:29:34.343 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.338+0000 7f8e5fa2f700 1 -- 172.21.15.167:0/2399019461 <== osd.1 v2:172.21.15.167:6800/30035 6 ==== osd_op_reply(9 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f8e50014040 con 0x56003d1cd620 2023-10-05T19:29:34.343 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.338+0000 7f8e5fa2f700 1 -- 172.21.15.167:0/2399019461 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:10 5.4 5:20000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x56003d207a20 con 0x56003d1cd620 2023-10-05T19:29:34.344 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.338+0000 7f8e5fa2f700 1 -- 172.21.15.167:0/2399019461 <== osd.1 v2:172.21.15.167:6800/30035 7 ==== osd_op_reply(10 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f8e50014040 con 0x56003d1cd620 2023-10-05T19:29:34.344 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.338+0000 7f8e5fa2f700 1 -- 172.21.15.167:0/2399019461 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:11 5.14 5:28000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x56003d207a20 con 0x7f8e48011040 2023-10-05T19:29:34.345 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.342+0000 7f8e5fa2f700 1 -- 172.21.15.167:0/2399019461 <== osd.0 v2:172.21.15.118:6800/27690 2 ==== osd_op_reply(11 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f8e50014040 con 0x7f8e48011040 2023-10-05T19:29:34.345 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.342+0000 7f8e5fa2f700 1 -- 172.21.15.167:0/2399019461 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:12 5.c 5:30000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x56003d207a20 con 0x7f8e48011040 2023-10-05T19:29:34.346 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.342+0000 7f8e5fa2f700 1 -- 172.21.15.167:0/2399019461 <== osd.0 v2:172.21.15.118:6800/27690 3 ==== osd_op_reply(12 [pgnls start_epoch 32 out=112b] v23'1 uv1 ondisk = 1) v8 ==== 144+0+112 (crc 0 0 0) 0x7f8e50014040 con 0x7f8e48011040 2023-10-05T19:29:34.346 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.342+0000 7f8e5fa2f700 1 -- 172.21.15.167:0/2399019461 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:13 5.1c 5:38000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x56003d207a20 con 0x7f8e58007030 2023-10-05T19:29:34.347 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.342+0000 7f8e5ea2d700 1 -- 172.21.15.167:0/2399019461 <== osd.2 v2:172.21.15.167:6816/30037 2 ==== osd_op_reply(13 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f8e48004520 con 0x7f8e58007030 2023-10-05T19:29:34.347 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.342+0000 7f8e5ea2d700 1 -- 172.21.15.167:0/2399019461 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:14 5.2 5:40000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x56003d207a20 con 0x56003d205100 2023-10-05T19:29:34.347 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.342+0000 7f8e5f22e700 1 -- 172.21.15.167:0/2399019461 <== osd.3 v2:172.21.15.167:6808/30036 2 ==== osd_op_reply(14 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f8e58041040 con 0x56003d205100 2023-10-05T19:29:34.347 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.342+0000 7f8e5f22e700 1 -- 172.21.15.167:0/2399019461 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:15 5.12 5:48000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x56003d207a20 con 0x56003d1cd620 2023-10-05T19:29:34.348 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.342+0000 7f8e5fa2f700 1 -- 172.21.15.167:0/2399019461 <== osd.1 v2:172.21.15.167:6800/30035 8 ==== osd_op_reply(15 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f8e50014040 con 0x56003d1cd620 2023-10-05T19:29:34.348 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.342+0000 7f8e5fa2f700 1 -- 172.21.15.167:0/2399019461 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:16 5.a 5:50000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x56003d207a20 con 0x7f8e48011040 2023-10-05T19:29:34.349 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.346+0000 7f8e5fa2f700 1 -- 172.21.15.167:0/2399019461 <== osd.0 v2:172.21.15.118:6800/27690 4 ==== osd_op_reply(16 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f8e50022040 con 0x7f8e48011040 2023-10-05T19:29:34.349 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.346+0000 7f8e5fa2f700 1 -- 172.21.15.167:0/2399019461 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:17 5.1a 5:58000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x56003d207a20 con 0x56003d1cd620 2023-10-05T19:29:34.350 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.346+0000 7f8e5fa2f700 1 -- 172.21.15.167:0/2399019461 <== osd.1 v2:172.21.15.167:6800/30035 9 ==== osd_op_reply(17 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f8e50022040 con 0x56003d1cd620 2023-10-05T19:29:34.350 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.346+0000 7f8e5fa2f700 1 -- 172.21.15.167:0/2399019461 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:18 5.6 5:60000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x56003d207a20 con 0x7f8e58007030 2023-10-05T19:29:34.351 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.346+0000 7f8e5ea2d700 1 -- 172.21.15.167:0/2399019461 <== osd.2 v2:172.21.15.167:6816/30037 3 ==== osd_op_reply(18 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f8e48004520 con 0x7f8e58007030 2023-10-05T19:29:34.351 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.346+0000 7f8e5ea2d700 1 -- 172.21.15.167:0/2399019461 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:19 5.16 5:68000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x56003d207a20 con 0x56003d205100 2023-10-05T19:29:34.351 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.346+0000 7f8e5f22e700 1 -- 172.21.15.167:0/2399019461 <== osd.3 v2:172.21.15.167:6808/30036 3 ==== osd_op_reply(19 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f8e58041040 con 0x56003d205100 2023-10-05T19:29:34.351 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.346+0000 7f8e5f22e700 1 -- 172.21.15.167:0/2399019461 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:20 5.e 5:70000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x56003d207a20 con 0x56003d205100 2023-10-05T19:29:34.352 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.346+0000 7f8e5f22e700 1 -- 172.21.15.167:0/2399019461 <== osd.3 v2:172.21.15.167:6808/30036 4 ==== osd_op_reply(20 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f8e58041040 con 0x56003d205100 2023-10-05T19:29:34.352 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.346+0000 7f8e5f22e700 1 -- 172.21.15.167:0/2399019461 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:21 5.1e 5:78000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x56003d207a20 con 0x7f8e48011040 2023-10-05T19:29:34.353 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.350+0000 7f8e5fa2f700 1 -- 172.21.15.167:0/2399019461 <== osd.0 v2:172.21.15.118:6800/27690 5 ==== osd_op_reply(21 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f8e50022040 con 0x7f8e48011040 2023-10-05T19:29:34.353 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.350+0000 7f8e5fa2f700 1 -- 172.21.15.167:0/2399019461 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:22 5.1 5:80000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x56003d207a20 con 0x56003d205100 2023-10-05T19:29:34.354 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.350+0000 7f8e5f22e700 1 -- 172.21.15.167:0/2399019461 <== osd.3 v2:172.21.15.167:6808/30036 5 ==== osd_op_reply(22 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f8e58041040 con 0x56003d205100 2023-10-05T19:29:34.354 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.350+0000 7f8e5f22e700 1 -- 172.21.15.167:0/2399019461 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:23 5.11 5:88000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x56003d207a20 con 0x56003d205100 2023-10-05T19:29:34.355 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.350+0000 7f8e5f22e700 1 -- 172.21.15.167:0/2399019461 <== osd.3 v2:172.21.15.167:6808/30036 6 ==== osd_op_reply(23 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f8e58041040 con 0x56003d205100 2023-10-05T19:29:34.355 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.350+0000 7f8e5f22e700 1 -- 172.21.15.167:0/2399019461 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:24 5.9 5:90000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x56003d207a20 con 0x56003d1cd620 2023-10-05T19:29:34.355 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.350+0000 7f8e5fa2f700 1 -- 172.21.15.167:0/2399019461 <== osd.1 v2:172.21.15.167:6800/30035 10 ==== osd_op_reply(24 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f8e50022040 con 0x56003d1cd620 2023-10-05T19:29:34.355 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.350+0000 7f8e5fa2f700 1 -- 172.21.15.167:0/2399019461 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:25 5.19 5:98000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x56003d207a20 con 0x56003d1cd620 2023-10-05T19:29:34.356 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.350+0000 7f8e5fa2f700 1 -- 172.21.15.167:0/2399019461 <== osd.1 v2:172.21.15.167:6800/30035 11 ==== osd_op_reply(25 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f8e50022040 con 0x56003d1cd620 2023-10-05T19:29:34.356 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.350+0000 7f8e5fa2f700 1 -- 172.21.15.167:0/2399019461 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:26 5.5 5:a0000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x56003d207a20 con 0x7f8e48011040 2023-10-05T19:29:34.357 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.354+0000 7f8e5fa2f700 1 -- 172.21.15.167:0/2399019461 <== osd.0 v2:172.21.15.118:6800/27690 6 ==== osd_op_reply(26 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f8e50022040 con 0x7f8e48011040 2023-10-05T19:29:34.357 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.354+0000 7f8e5fa2f700 1 -- 172.21.15.167:0/2399019461 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:27 5.15 5:a8000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x56003d207a20 con 0x56003d205100 2023-10-05T19:29:34.358 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.354+0000 7f8e5f22e700 1 -- 172.21.15.167:0/2399019461 <== osd.3 v2:172.21.15.167:6808/30036 7 ==== osd_op_reply(27 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f8e58041040 con 0x56003d205100 2023-10-05T19:29:34.358 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.354+0000 7f8e5f22e700 1 -- 172.21.15.167:0/2399019461 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:28 5.d 5:b0000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x56003d207a20 con 0x7f8e58007030 2023-10-05T19:29:34.359 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.354+0000 7f8e5ea2d700 1 -- 172.21.15.167:0/2399019461 <== osd.2 v2:172.21.15.167:6816/30037 4 ==== osd_op_reply(28 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f8e48004520 con 0x7f8e58007030 2023-10-05T19:29:34.359 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.354+0000 7f8e5ea2d700 1 -- 172.21.15.167:0/2399019461 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:29 5.1d 5:b8000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x56003d207a20 con 0x56003d1cd620 2023-10-05T19:29:34.359 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.354+0000 7f8e5fa2f700 1 -- 172.21.15.167:0/2399019461 <== osd.1 v2:172.21.15.167:6800/30035 12 ==== osd_op_reply(29 [pgnls start_epoch 32 out=107b] v30'2 uv2 ondisk = 1) v8 ==== 144+0+107 (crc 0 0 0) 0x7f8e50022040 con 0x56003d1cd620 2023-10-05T19:29:34.360 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.354+0000 7f8e5fa2f700 1 -- 172.21.15.167:0/2399019461 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:30 5.3 5:c0000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x56003d207a20 con 0x7f8e48011040 2023-10-05T19:29:34.360 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.358+0000 7f8e5fa2f700 1 -- 172.21.15.167:0/2399019461 <== osd.0 v2:172.21.15.118:6800/27690 7 ==== osd_op_reply(30 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f8e50022040 con 0x7f8e48011040 2023-10-05T19:29:34.360 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.358+0000 7f8e5fa2f700 1 -- 172.21.15.167:0/2399019461 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:31 5.13 5:c8000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x56003d207a20 con 0x56003d205100 2023-10-05T19:29:34.361 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.358+0000 7f8e5f22e700 1 -- 172.21.15.167:0/2399019461 <== osd.3 v2:172.21.15.167:6808/30036 8 ==== osd_op_reply(31 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f8e58041040 con 0x56003d205100 2023-10-05T19:29:34.361 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.358+0000 7f8e5f22e700 1 -- 172.21.15.167:0/2399019461 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:32 5.b 5:d0000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x56003d207a20 con 0x7f8e48011040 2023-10-05T19:29:34.362 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.358+0000 7f8e5fa2f700 1 -- 172.21.15.167:0/2399019461 <== osd.0 v2:172.21.15.118:6800/27690 8 ==== osd_op_reply(32 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f8e50022040 con 0x7f8e48011040 2023-10-05T19:29:34.362 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.358+0000 7f8e5fa2f700 1 -- 172.21.15.167:0/2399019461 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:33 5.1b 5:d8000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x56003d207a20 con 0x7f8e58007030 2023-10-05T19:29:34.363 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.358+0000 7f8e5ea2d700 1 -- 172.21.15.167:0/2399019461 <== osd.2 v2:172.21.15.167:6816/30037 5 ==== osd_op_reply(33 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f8e48004520 con 0x7f8e58007030 2023-10-05T19:29:34.363 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.358+0000 7f8e5ea2d700 1 -- 172.21.15.167:0/2399019461 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:34 5.7 5:e0000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x56003d207a20 con 0x56003d205100 2023-10-05T19:29:34.364 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.358+0000 7f8e5f22e700 1 -- 172.21.15.167:0/2399019461 <== osd.3 v2:172.21.15.167:6808/30036 9 ==== osd_op_reply(34 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f8e58041040 con 0x56003d205100 2023-10-05T19:29:34.364 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.358+0000 7f8e5f22e700 1 -- 172.21.15.167:0/2399019461 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:35 5.17 5:e8000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x56003d207a20 con 0x7f8e48011040 2023-10-05T19:29:34.365 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.362+0000 7f8e5fa2f700 1 -- 172.21.15.167:0/2399019461 <== osd.0 v2:172.21.15.118:6800/27690 9 ==== osd_op_reply(35 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f8e50022040 con 0x7f8e48011040 2023-10-05T19:29:34.365 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.362+0000 7f8e5fa2f700 1 -- 172.21.15.167:0/2399019461 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:36 5.f 5:f0000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x56003d207a20 con 0x56003d1cd620 2023-10-05T19:29:34.366 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.362+0000 7f8e5fa2f700 1 -- 172.21.15.167:0/2399019461 <== osd.1 v2:172.21.15.167:6800/30035 13 ==== osd_op_reply(36 [pgnls start_epoch 32 out=115b] v23'2 uv2 ondisk = 1) v8 ==== 144+0+115 (crc 0 0 0) 0x7f8e5002f080 con 0x56003d1cd620 2023-10-05T19:29:34.366 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.362+0000 7f8e5fa2f700 1 -- 172.21.15.167:0/2399019461 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:37 5.1f 5:f8000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x56003d207a20 con 0x7f8e58007030 2023-10-05T19:29:34.367 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.362+0000 7f8e5ea2d700 1 -- 172.21.15.167:0/2399019461 <== osd.2 v2:172.21.15.167:6816/30037 6 ==== osd_op_reply(37 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f8e48004520 con 0x7f8e58007030 2023-10-05T19:29:34.367 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.362+0000 7f8e616f9e40 20 RGWRados::pool_iterate: got zonegroup_info.4f1f719d-9648-417b-b10c-e5d939825419 2023-10-05T19:29:34.367 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.362+0000 7f8e616f9e40 20 RGWRados::pool_iterate: got zone_info.9a9df965-6612-45d4-a45f-6e3c889bacc2 2023-10-05T19:29:34.367 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.362+0000 7f8e616f9e40 20 RGWRados::pool_iterate: got zone_names.default 2023-10-05T19:29:34.367 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.362+0000 7f8e616f9e40 20 RGWRados::pool_iterate: got zonegroups_names.default 2023-10-05T19:29:34.367 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.362+0000 7f8e616f9e40 20 rados->read ofs=0 len=0 2023-10-05T19:29:34.367 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.362+0000 7f8e616f9e40 1 -- 172.21.15.167:0/2399019461 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:38 5.12 5:49953fa1:::default.realm:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32) v8 -- 0x56003d203eb0 con 0x56003d1cd620 2023-10-05T19:29:34.368 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.362+0000 7f8e5fa2f700 1 -- 172.21.15.167:0/2399019461 <== osd.1 v2:172.21.15.167:6800/30035 14 ==== osd_op_reply(38 default.realm [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 ==== 157+0+0 (crc 0 0 0) 0x7f8e5002f080 con 0x56003d1cd620 2023-10-05T19:29:34.368 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.362+0000 7f8e616f9e40 20 rados_obj.operate() r=-2 bl.length=0 2023-10-05T19:29:34.368 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.362+0000 7f8e616f9e40 20 rados->read ofs=0 len=0 2023-10-05T19:29:34.369 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.362+0000 7f8e616f9e40 1 -- 172.21.15.167:0/2399019461 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:39 5.f 5:f4c53578:::zonegroups_names.default:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32) v8 -- 0x56003d204830 con 0x56003d1cd620 2023-10-05T19:29:34.369 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.362+0000 7f8e5fa2f700 1 -- 172.21.15.167:0/2399019461 <== osd.1 v2:172.21.15.167:6800/30035 15 ==== osd_op_reply(39 zonegroups_names.default [read 0~46 out=46b] v0'0 uv2 ondisk = 0) v8 ==== 168+0+46 (crc 0 0 0) 0x7f8e5002f080 con 0x56003d1cd620 2023-10-05T19:29:34.369 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.362+0000 7f8e616f9e40 20 rados_obj.operate() r=0 bl.length=46 2023-10-05T19:29:34.369 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.362+0000 7f8e616f9e40 20 rados->read ofs=0 len=0 2023-10-05T19:29:34.369 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.362+0000 7f8e616f9e40 1 -- 172.21.15.167:0/2399019461 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:40 5.c 5:30d299a5:::zonegroup_info.4f1f719d-9648-417b-b10c-e5d939825419:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32) v8 -- 0x56003d203aa0 con 0x7f8e48011040 2023-10-05T19:29:34.370 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.366+0000 7f8e5fa2f700 1 -- 172.21.15.167:0/2399019461 <== osd.0 v2:172.21.15.118:6800/27690 10 ==== osd_op_reply(40 zonegroup_info.4f1f719d-9648-417b-b10c-e5d939825419 [read 0~362 out=362b] v0'0 uv1 ondisk = 0) v8 ==== 195+0+362 (crc 0 0 0) 0x7f8e5002f080 con 0x7f8e48011040 2023-10-05T19:29:34.370 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.366+0000 7f8e616f9e40 20 rados_obj.operate() r=0 bl.length=362 2023-10-05T19:29:34.370 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.366+0000 7f8e616f9e40 20 rados->read ofs=0 len=0 2023-10-05T19:29:34.370 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.366+0000 7f8e616f9e40 1 -- 172.21.15.167:0/2399019461 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:41 5.13 5:cbbf6c60:::converted:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32) v8 -- 0x56003d208310 con 0x56003d205100 2023-10-05T19:29:34.370 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.366+0000 7f8e5f22e700 1 -- 172.21.15.167:0/2399019461 <== osd.3 v2:172.21.15.167:6808/30036 10 ==== osd_op_reply(41 converted [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 ==== 153+0+0 (crc 0 0 0) 0x7f8e58041040 con 0x56003d205100 2023-10-05T19:29:34.370 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.366+0000 7f8e616f9e40 20 rados_obj.operate() r=-2 bl.length=0 2023-10-05T19:29:34.370 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.366+0000 7f8e616f9e40 20 rados->read ofs=0 len=0 2023-10-05T19:29:34.370 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.366+0000 7f8e616f9e40 1 -- 172.21.15.167:0/2399019461 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:42 5.12 5:49953fa1:::default.realm:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32) v8 -- 0x56003d2086c0 con 0x56003d1cd620 2023-10-05T19:29:34.371 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.366+0000 7f8e5fa2f700 1 -- 172.21.15.167:0/2399019461 <== osd.1 v2:172.21.15.167:6800/30035 16 ==== osd_op_reply(42 default.realm [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 ==== 157+0+0 (crc 0 0 0) 0x7f8e5002f080 con 0x56003d1cd620 2023-10-05T19:29:34.371 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.366+0000 7f8e616f9e40 20 rados_obj.operate() r=-2 bl.length=0 2023-10-05T19:29:34.371 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.366+0000 7f8e616f9e40 20 rados->read ofs=0 len=0 2023-10-05T19:29:34.371 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.366+0000 7f8e616f9e40 1 -- 172.21.15.167:0/2399019461 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:43 5.f 5:f4c53578:::zonegroups_names.default:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32) v8 -- 0x56003d2086c0 con 0x56003d1cd620 2023-10-05T19:29:34.371 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.366+0000 7f8e5fa2f700 1 -- 172.21.15.167:0/2399019461 <== osd.1 v2:172.21.15.167:6800/30035 17 ==== osd_op_reply(43 zonegroups_names.default [read 0~46 out=46b] v0'0 uv2 ondisk = 0) v8 ==== 168+0+46 (crc 0 0 0) 0x7f8e5002f080 con 0x56003d1cd620 2023-10-05T19:29:34.372 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.366+0000 7f8e616f9e40 20 rados_obj.operate() r=0 bl.length=46 2023-10-05T19:29:34.372 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.366+0000 7f8e616f9e40 1 -- 172.21.15.167:0/2399019461 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:44 5.0 5:00000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x56003d2096d0 con 0x56003d205100 2023-10-05T19:29:34.372 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.366+0000 7f8e5f22e700 1 -- 172.21.15.167:0/2399019461 <== osd.3 v2:172.21.15.167:6808/30036 11 ==== osd_op_reply(44 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f8e58041040 con 0x56003d205100 2023-10-05T19:29:34.372 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.366+0000 7f8e5f22e700 1 -- 172.21.15.167:0/2399019461 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:45 5.10 5:08000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x56003d2096d0 con 0x7f8e58007030 2023-10-05T19:29:34.373 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.370+0000 7f8e5ea2d700 1 -- 172.21.15.167:0/2399019461 <== osd.2 v2:172.21.15.167:6816/30037 7 ==== osd_op_reply(45 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f8e48004520 con 0x7f8e58007030 2023-10-05T19:29:34.373 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.370+0000 7f8e5ea2d700 1 -- 172.21.15.167:0/2399019461 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:46 5.8 5:10000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x56003d2096d0 con 0x7f8e48011040 2023-10-05T19:29:34.374 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.370+0000 7f8e5fa2f700 1 -- 172.21.15.167:0/2399019461 <== osd.0 v2:172.21.15.118:6800/27690 11 ==== osd_op_reply(46 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f8e5002f080 con 0x7f8e48011040 2023-10-05T19:29:34.374 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.370+0000 7f8e5fa2f700 1 -- 172.21.15.167:0/2399019461 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:47 5.18 5:18000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x56003d203aa0 con 0x56003d1cd620 2023-10-05T19:29:34.375 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.370+0000 7f8e5fa2f700 1 -- 172.21.15.167:0/2399019461 <== osd.1 v2:172.21.15.167:6800/30035 18 ==== osd_op_reply(47 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f8e5002f080 con 0x56003d1cd620 2023-10-05T19:29:34.375 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.370+0000 7f8e5fa2f700 1 -- 172.21.15.167:0/2399019461 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:48 5.4 5:20000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x56003d203aa0 con 0x56003d1cd620 2023-10-05T19:29:34.375 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.370+0000 7f8e5fa2f700 1 -- 172.21.15.167:0/2399019461 <== osd.1 v2:172.21.15.167:6800/30035 19 ==== osd_op_reply(48 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f8e5002f080 con 0x56003d1cd620 2023-10-05T19:29:34.376 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.370+0000 7f8e5fa2f700 1 -- 172.21.15.167:0/2399019461 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:49 5.14 5:28000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x56003d203aa0 con 0x7f8e48011040 2023-10-05T19:29:34.377 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.374+0000 7f8e5fa2f700 1 -- 172.21.15.167:0/2399019461 <== osd.0 v2:172.21.15.118:6800/27690 12 ==== osd_op_reply(49 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f8e5002f080 con 0x7f8e48011040 2023-10-05T19:29:34.377 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.374+0000 7f8e5fa2f700 1 -- 172.21.15.167:0/2399019461 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:50 5.c 5:30000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x56003d203aa0 con 0x7f8e48011040 2023-10-05T19:29:34.377 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.374+0000 7f8e5fa2f700 1 -- 172.21.15.167:0/2399019461 <== osd.0 v2:172.21.15.118:6800/27690 13 ==== osd_op_reply(50 [pgnls start_epoch 32 out=112b] v23'1 uv1 ondisk = 1) v8 ==== 144+0+112 (crc 0 0 0) 0x7f8e5002f080 con 0x7f8e48011040 2023-10-05T19:29:34.377 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.374+0000 7f8e5fa2f700 1 -- 172.21.15.167:0/2399019461 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:51 5.1c 5:38000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x56003d203aa0 con 0x7f8e58007030 2023-10-05T19:29:34.378 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.374+0000 7f8e5ea2d700 1 -- 172.21.15.167:0/2399019461 <== osd.2 v2:172.21.15.167:6816/30037 8 ==== osd_op_reply(51 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f8e48004520 con 0x7f8e58007030 2023-10-05T19:29:34.378 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.374+0000 7f8e5ea2d700 1 -- 172.21.15.167:0/2399019461 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:52 5.2 5:40000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x56003d203aa0 con 0x56003d205100 2023-10-05T19:29:34.379 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.374+0000 7f8e5f22e700 1 -- 172.21.15.167:0/2399019461 <== osd.3 v2:172.21.15.167:6808/30036 12 ==== osd_op_reply(52 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f8e58041040 con 0x56003d205100 2023-10-05T19:29:34.379 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.374+0000 7f8e5f22e700 1 -- 172.21.15.167:0/2399019461 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:53 5.12 5:48000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x56003d203aa0 con 0x56003d1cd620 2023-10-05T19:29:34.380 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.374+0000 7f8e5fa2f700 1 -- 172.21.15.167:0/2399019461 <== osd.1 v2:172.21.15.167:6800/30035 20 ==== osd_op_reply(53 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f8e5002f080 con 0x56003d1cd620 2023-10-05T19:29:34.380 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.374+0000 7f8e5fa2f700 1 -- 172.21.15.167:0/2399019461 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:54 5.a 5:50000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x56003d203aa0 con 0x7f8e48011040 2023-10-05T19:29:34.381 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.378+0000 7f8e5fa2f700 1 -- 172.21.15.167:0/2399019461 <== osd.0 v2:172.21.15.118:6800/27690 14 ==== osd_op_reply(54 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f8e5002f080 con 0x7f8e48011040 2023-10-05T19:29:34.381 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.378+0000 7f8e5fa2f700 1 -- 172.21.15.167:0/2399019461 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:55 5.1a 5:58000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x56003d203aa0 con 0x56003d1cd620 2023-10-05T19:29:34.381 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.378+0000 7f8e5fa2f700 1 -- 172.21.15.167:0/2399019461 <== osd.1 v2:172.21.15.167:6800/30035 21 ==== osd_op_reply(55 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f8e5002f080 con 0x56003d1cd620 2023-10-05T19:29:34.382 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.378+0000 7f8e5fa2f700 1 -- 172.21.15.167:0/2399019461 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:56 5.6 5:60000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x56003d203aa0 con 0x7f8e58007030 2023-10-05T19:29:34.382 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.378+0000 7f8e5ea2d700 1 -- 172.21.15.167:0/2399019461 <== osd.2 v2:172.21.15.167:6816/30037 9 ==== osd_op_reply(56 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f8e48004520 con 0x7f8e58007030 2023-10-05T19:29:34.383 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.378+0000 7f8e5ea2d700 1 -- 172.21.15.167:0/2399019461 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:57 5.16 5:68000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x56003d203aa0 con 0x56003d205100 2023-10-05T19:29:34.383 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.378+0000 7f8e5f22e700 1 -- 172.21.15.167:0/2399019461 <== osd.3 v2:172.21.15.167:6808/30036 13 ==== osd_op_reply(57 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f8e58041040 con 0x56003d205100 2023-10-05T19:29:34.383 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.378+0000 7f8e5f22e700 1 -- 172.21.15.167:0/2399019461 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:58 5.e 5:70000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x56003d203aa0 con 0x56003d205100 2023-10-05T19:29:34.384 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.378+0000 7f8e5f22e700 1 -- 172.21.15.167:0/2399019461 <== osd.3 v2:172.21.15.167:6808/30036 14 ==== osd_op_reply(58 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f8e58041040 con 0x56003d205100 2023-10-05T19:29:34.384 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.378+0000 7f8e5f22e700 1 -- 172.21.15.167:0/2399019461 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:59 5.1e 5:78000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x56003d203aa0 con 0x7f8e48011040 2023-10-05T19:29:34.385 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.382+0000 7f8e5fa2f700 1 -- 172.21.15.167:0/2399019461 <== osd.0 v2:172.21.15.118:6800/27690 15 ==== osd_op_reply(59 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f8e5002f080 con 0x7f8e48011040 2023-10-05T19:29:34.385 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.382+0000 7f8e5fa2f700 1 -- 172.21.15.167:0/2399019461 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:60 5.1 5:80000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x56003d203aa0 con 0x56003d205100 2023-10-05T19:29:34.386 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.382+0000 7f8e5f22e700 1 -- 172.21.15.167:0/2399019461 <== osd.3 v2:172.21.15.167:6808/30036 15 ==== osd_op_reply(60 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f8e58041040 con 0x56003d205100 2023-10-05T19:29:34.386 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.382+0000 7f8e5f22e700 1 -- 172.21.15.167:0/2399019461 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:61 5.11 5:88000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x56003d203aa0 con 0x56003d205100 2023-10-05T19:29:34.386 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.382+0000 7f8e5f22e700 1 -- 172.21.15.167:0/2399019461 <== osd.3 v2:172.21.15.167:6808/30036 16 ==== osd_op_reply(61 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f8e58041040 con 0x56003d205100 2023-10-05T19:29:34.387 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.382+0000 7f8e5f22e700 1 -- 172.21.15.167:0/2399019461 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:62 5.9 5:90000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x56003d203aa0 con 0x56003d1cd620 2023-10-05T19:29:34.387 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.382+0000 7f8e5fa2f700 1 -- 172.21.15.167:0/2399019461 <== osd.1 v2:172.21.15.167:6800/30035 22 ==== osd_op_reply(62 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f8e5002f080 con 0x56003d1cd620 2023-10-05T19:29:34.387 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.382+0000 7f8e5fa2f700 1 -- 172.21.15.167:0/2399019461 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:63 5.19 5:98000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x56003d203aa0 con 0x56003d1cd620 2023-10-05T19:29:34.388 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.382+0000 7f8e5fa2f700 1 -- 172.21.15.167:0/2399019461 <== osd.1 v2:172.21.15.167:6800/30035 23 ==== osd_op_reply(63 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f8e5002f080 con 0x56003d1cd620 2023-10-05T19:29:34.388 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.382+0000 7f8e5fa2f700 1 -- 172.21.15.167:0/2399019461 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:64 5.5 5:a0000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x56003d203aa0 con 0x7f8e48011040 2023-10-05T19:29:34.389 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.386+0000 7f8e5fa2f700 1 -- 172.21.15.167:0/2399019461 <== osd.0 v2:172.21.15.118:6800/27690 16 ==== osd_op_reply(64 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f8e5002f080 con 0x7f8e48011040 2023-10-05T19:29:34.389 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.386+0000 7f8e5fa2f700 1 -- 172.21.15.167:0/2399019461 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:65 5.15 5:a8000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x56003d203aa0 con 0x56003d205100 2023-10-05T19:29:34.390 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.386+0000 7f8e5f22e700 1 -- 172.21.15.167:0/2399019461 <== osd.3 v2:172.21.15.167:6808/30036 17 ==== osd_op_reply(65 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f8e58041040 con 0x56003d205100 2023-10-05T19:29:34.390 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.386+0000 7f8e5f22e700 1 -- 172.21.15.167:0/2399019461 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:66 5.d 5:b0000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x56003d203aa0 con 0x7f8e58007030 2023-10-05T19:29:34.391 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.386+0000 7f8e5ea2d700 1 -- 172.21.15.167:0/2399019461 <== osd.2 v2:172.21.15.167:6816/30037 10 ==== osd_op_reply(66 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f8e48004520 con 0x7f8e58007030 2023-10-05T19:29:34.391 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.386+0000 7f8e5ea2d700 1 -- 172.21.15.167:0/2399019461 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:67 5.1d 5:b8000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x56003d203aa0 con 0x56003d1cd620 2023-10-05T19:29:34.392 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.386+0000 7f8e5fa2f700 1 -- 172.21.15.167:0/2399019461 <== osd.1 v2:172.21.15.167:6800/30035 24 ==== osd_op_reply(67 [pgnls start_epoch 32 out=107b] v30'2 uv2 ondisk = 1) v8 ==== 144+0+107 (crc 0 0 0) 0x7f8e5002f080 con 0x56003d1cd620 2023-10-05T19:29:34.392 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.386+0000 7f8e5fa2f700 1 -- 172.21.15.167:0/2399019461 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:68 5.3 5:c0000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x56003d203aa0 con 0x7f8e48011040 2023-10-05T19:29:34.392 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.390+0000 7f8e5fa2f700 1 -- 172.21.15.167:0/2399019461 <== osd.0 v2:172.21.15.118:6800/27690 17 ==== osd_op_reply(68 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f8e5002f080 con 0x7f8e48011040 2023-10-05T19:29:34.393 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.390+0000 7f8e5fa2f700 1 -- 172.21.15.167:0/2399019461 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:69 5.13 5:c8000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x56003d203aa0 con 0x56003d205100 2023-10-05T19:29:34.393 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.390+0000 7f8e5f22e700 1 -- 172.21.15.167:0/2399019461 <== osd.3 v2:172.21.15.167:6808/30036 18 ==== osd_op_reply(69 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f8e58041040 con 0x56003d205100 2023-10-05T19:29:34.393 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.390+0000 7f8e5f22e700 1 -- 172.21.15.167:0/2399019461 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:70 5.b 5:d0000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x56003d203aa0 con 0x7f8e48011040 2023-10-05T19:29:34.394 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.390+0000 7f8e5fa2f700 1 -- 172.21.15.167:0/2399019461 <== osd.0 v2:172.21.15.118:6800/27690 18 ==== osd_op_reply(70 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f8e5002f080 con 0x7f8e48011040 2023-10-05T19:29:34.394 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.390+0000 7f8e5fa2f700 1 -- 172.21.15.167:0/2399019461 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:71 5.1b 5:d8000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x56003d203aa0 con 0x7f8e58007030 2023-10-05T19:29:34.395 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.390+0000 7f8e5ea2d700 1 -- 172.21.15.167:0/2399019461 <== osd.2 v2:172.21.15.167:6816/30037 11 ==== osd_op_reply(71 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f8e48004520 con 0x7f8e58007030 2023-10-05T19:29:34.395 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.390+0000 7f8e5ea2d700 1 -- 172.21.15.167:0/2399019461 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:72 5.7 5:e0000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x56003d203aa0 con 0x56003d205100 2023-10-05T19:29:34.396 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.390+0000 7f8e5f22e700 1 -- 172.21.15.167:0/2399019461 <== osd.3 v2:172.21.15.167:6808/30036 19 ==== osd_op_reply(72 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f8e58041040 con 0x56003d205100 2023-10-05T19:29:34.396 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.390+0000 7f8e5f22e700 1 -- 172.21.15.167:0/2399019461 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:73 5.17 5:e8000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x56003d203aa0 con 0x7f8e48011040 2023-10-05T19:29:34.397 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.394+0000 7f8e5fa2f700 1 -- 172.21.15.167:0/2399019461 <== osd.0 v2:172.21.15.118:6800/27690 19 ==== osd_op_reply(73 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f8e5002f080 con 0x7f8e48011040 2023-10-05T19:29:34.397 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.394+0000 7f8e5fa2f700 1 -- 172.21.15.167:0/2399019461 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:74 5.f 5:f0000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x56003d203aa0 con 0x56003d1cd620 2023-10-05T19:29:34.398 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.394+0000 7f8e5fa2f700 1 -- 172.21.15.167:0/2399019461 <== osd.1 v2:172.21.15.167:6800/30035 25 ==== osd_op_reply(74 [pgnls start_epoch 32 out=115b] v23'2 uv2 ondisk = 1) v8 ==== 144+0+115 (crc 0 0 0) 0x7f8e5002f080 con 0x56003d1cd620 2023-10-05T19:29:34.398 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.394+0000 7f8e5fa2f700 1 -- 172.21.15.167:0/2399019461 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:75 5.1f 5:f8000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x56003d203aa0 con 0x7f8e58007030 2023-10-05T19:29:34.399 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.394+0000 7f8e5ea2d700 1 -- 172.21.15.167:0/2399019461 <== osd.2 v2:172.21.15.167:6816/30037 12 ==== osd_op_reply(75 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f8e48004520 con 0x7f8e58007030 2023-10-05T19:29:34.399 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.394+0000 7f8e616f9e40 20 RGWRados::pool_iterate: got zonegroup_info.4f1f719d-9648-417b-b10c-e5d939825419 2023-10-05T19:29:34.399 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.394+0000 7f8e616f9e40 20 RGWRados::pool_iterate: got zone_info.9a9df965-6612-45d4-a45f-6e3c889bacc2 2023-10-05T19:29:34.399 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.394+0000 7f8e616f9e40 20 RGWRados::pool_iterate: got zone_names.default 2023-10-05T19:29:34.399 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.394+0000 7f8e616f9e40 20 RGWRados::pool_iterate: got zonegroups_names.default 2023-10-05T19:29:34.399 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.394+0000 7f8e616f9e40 20 rados->read ofs=0 len=0 2023-10-05T19:29:34.400 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.394+0000 7f8e616f9e40 1 -- 172.21.15.167:0/2399019461 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:76 5.f 5:f43b6ece:::zone_names.default:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32) v8 -- 0x56003d2086c0 con 0x56003d1cd620 2023-10-05T19:29:34.400 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.394+0000 7f8e5fa2f700 1 -- 172.21.15.167:0/2399019461 <== osd.1 v2:172.21.15.167:6800/30035 26 ==== osd_op_reply(76 zone_names.default [read 0~46 out=46b] v0'0 uv1 ondisk = 0) v8 ==== 162+0+46 (crc 0 0 0) 0x7f8e5002f080 con 0x56003d1cd620 2023-10-05T19:29:34.401 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.394+0000 7f8e616f9e40 20 rados_obj.operate() r=0 bl.length=46 2023-10-05T19:29:34.401 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.394+0000 7f8e616f9e40 20 rados->read ofs=0 len=0 2023-10-05T19:29:34.401 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.394+0000 7f8e616f9e40 1 -- 172.21.15.167:0/2399019461 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:77 5.1d 5:b8374eb4:::zone_info.9a9df965-6612-45d4-a45f-6e3c889bacc2:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32) v8 -- 0x56003d208a70 con 0x56003d1cd620 2023-10-05T19:29:34.401 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.398+0000 7f8e5fa2f700 1 -- 172.21.15.167:0/2399019461 <== osd.1 v2:172.21.15.167:6800/30035 27 ==== osd_op_reply(77 zone_info.9a9df965-6612-45d4-a45f-6e3c889bacc2 [read 0~874 out=874b] v0'0 uv2 ondisk = 0) v8 ==== 190+0+874 (crc 0 0 0) 0x7f8e5002f080 con 0x56003d1cd620 2023-10-05T19:29:34.401 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.398+0000 7f8e616f9e40 20 rados_obj.operate() r=0 bl.length=874 2023-10-05T19:29:34.401 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.398+0000 7f8e616f9e40 20 rados->read ofs=0 len=0 2023-10-05T19:29:34.401 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.398+0000 7f8e616f9e40 1 -- 172.21.15.167:0/2399019461 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:78 5.f 5:f4c53578:::zonegroups_names.default:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32) v8 -- 0x56003d208f60 con 0x56003d1cd620 2023-10-05T19:29:34.402 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.398+0000 7f8e5fa2f700 1 -- 172.21.15.167:0/2399019461 <== osd.1 v2:172.21.15.167:6800/30035 28 ==== osd_op_reply(78 zonegroups_names.default [read 0~46 out=46b] v0'0 uv2 ondisk = 0) v8 ==== 168+0+46 (crc 0 0 0) 0x7f8e5002f080 con 0x56003d1cd620 2023-10-05T19:29:34.402 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.398+0000 7f8e616f9e40 20 rados_obj.operate() r=0 bl.length=46 2023-10-05T19:29:34.402 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.398+0000 7f8e616f9e40 20 rados->read ofs=0 len=0 2023-10-05T19:29:34.402 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.398+0000 7f8e616f9e40 1 -- 172.21.15.167:0/2399019461 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:79 5.c 5:30d299a5:::zonegroup_info.4f1f719d-9648-417b-b10c-e5d939825419:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32) v8 -- 0x56003d208f60 con 0x7f8e48011040 2023-10-05T19:29:34.402 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.398+0000 7f8e5fa2f700 1 -- 172.21.15.167:0/2399019461 <== osd.0 v2:172.21.15.118:6800/27690 20 ==== osd_op_reply(79 zonegroup_info.4f1f719d-9648-417b-b10c-e5d939825419 [read 0~362 out=362b] v0'0 uv1 ondisk = 0) v8 ==== 195+0+362 (crc 0 0 0) 0x7f8e5002f080 con 0x7f8e48011040 2023-10-05T19:29:34.402 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.398+0000 7f8e616f9e40 20 rados_obj.operate() r=0 bl.length=362 2023-10-05T19:29:34.402 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.398+0000 7f8e616f9e40 20 rados->read ofs=0 len=0 2023-10-05T19:29:34.402 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.398+0000 7f8e616f9e40 1 -- 172.21.15.167:0/2399019461 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:80 5.3 5:c0ad8cbd:::region_map:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32) v8 -- 0x56003d209060 con 0x7f8e48011040 2023-10-05T19:29:34.403 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.398+0000 7f8e5fa2f700 1 -- 172.21.15.167:0/2399019461 <== osd.0 v2:172.21.15.118:6800/27690 21 ==== osd_op_reply(80 region_map [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 ==== 154+0+0 (crc 0 0 0) 0x7f8e50050040 con 0x7f8e48011040 2023-10-05T19:29:34.403 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.398+0000 7f8e616f9e40 20 rados_obj.operate() r=-2 bl.length=0 2023-10-05T19:29:34.403 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.398+0000 7f8e616f9e40 20 zone default found 2023-10-05T19:29:34.403 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.398+0000 7f8e616f9e40 4 Realm: () 2023-10-05T19:29:34.403 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.398+0000 7f8e616f9e40 4 ZoneGroup: default (4f1f719d-9648-417b-b10c-e5d939825419) 2023-10-05T19:29:34.403 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.398+0000 7f8e616f9e40 4 Zone: default (9a9df965-6612-45d4-a45f-6e3c889bacc2) 2023-10-05T19:29:34.404 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.398+0000 7f8e616f9e40 10 cannot find current period zonegroup using local zonegroup configuration 2023-10-05T19:29:34.404 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.398+0000 7f8e616f9e40 20 zonegroup default 2023-10-05T19:29:34.404 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.398+0000 7f8e616f9e40 20 rados->read ofs=0 len=0 2023-10-05T19:29:34.404 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.398+0000 7f8e616f9e40 1 -- 172.21.15.167:0/2399019461 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:81 5.3 5:c52100b6:::period_config.default:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32) v8 -- 0x56003d209060 con 0x7f8e48011040 2023-10-05T19:29:34.405 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.398+0000 7f8e5fa2f700 1 -- 172.21.15.167:0/2399019461 <== osd.0 v2:172.21.15.118:6800/27690 22 ==== osd_op_reply(81 period_config.default [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 ==== 165+0+0 (crc 0 0 0) 0x7f8e50050040 con 0x7f8e48011040 2023-10-05T19:29:34.405 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.398+0000 7f8e616f9e40 20 rados_obj.operate() r=-2 bl.length=0 2023-10-05T19:29:34.405 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.398+0000 7f8e616f9e40 20 rados->read ofs=0 len=0 2023-10-05T19:29:34.405 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.398+0000 7f8e616f9e40 1 -- 172.21.15.167:0/2399019461 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:82 6.2 6:4347d321:::bucket.sync-source-hints.:head [call version.read in=11b,read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32) v8 -- 0x56003d20a9e0 con 0x56003d205100 2023-10-05T19:29:34.405 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.398+0000 7f8e5f22e700 1 -- 172.21.15.167:0/2399019461 <== osd.3 v2:172.21.15.167:6808/30036 20 ==== osd_op_reply(82 bucket.sync-source-hints. [call,read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 ==== 211+0+0 (crc 0 0 0) 0x7f8e58041040 con 0x56003d205100 2023-10-05T19:29:34.405 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.402+0000 7f8e616f9e40 20 rados_obj.operate() r=-2 bl.length=0 2023-10-05T19:29:34.405 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.402+0000 7f8e616f9e40 20 rados->read ofs=0 len=0 2023-10-05T19:29:34.405 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.402+0000 7f8e616f9e40 1 -- 172.21.15.167:0/2399019461 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:83 6.b 6:d467b91b:::bucket.sync-target-hints.:head [call version.read in=11b,read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32) v8 -- 0x56003d20ad90 con 0x56003d205100 2023-10-05T19:29:34.406 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.402+0000 7f8e5f22e700 1 -- 172.21.15.167:0/2399019461 <== osd.3 v2:172.21.15.167:6808/30036 21 ==== osd_op_reply(83 bucket.sync-target-hints. [call,read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 ==== 211+0+0 (crc 0 0 0) 0x7f8e58041040 con 0x56003d205100 2023-10-05T19:29:34.406 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.402+0000 7f8e616f9e40 20 rados_obj.operate() r=-2 bl.length=0 2023-10-05T19:29:34.406 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.402+0000 7f8e616f9e40 20 started sync module instance, tier type = 2023-10-05T19:29:34.406 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.402+0000 7f8e616f9e40 20 started zone id=9a9df965-6612-45d4-a45f-6e3c889bacc2 (name=default) with tier type = 2023-10-05T19:29:34.406 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.402+0000 7f8e616f9e40 1 -- 172.21.15.167:0/2399019461 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:84 7.1f 7:f95f44c2:::notify.0:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e32) v8 -- 0x56003d20a0a0 con 0x7f8e58007030 2023-10-05T19:29:34.409 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.406+0000 7f8e5ea2d700 1 -- 172.21.15.167:0/2399019461 <== osd.2 v2:172.21.15.167:6816/30037 13 ==== osd_op_reply(84 notify.0 [create] v32'237 uv237 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f8e4802f040 con 0x7f8e58007030 2023-10-05T19:29:34.409 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.406+0000 7f8e616f9e40 1 -- 172.21.15.167:0/2399019461 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:85 7.1f 7:f95f44c2:::notify.0:head [watch watch cookie 94559025538128] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e32) v8 -- 0x56003d209cd0 con 0x7f8e58007030 2023-10-05T19:29:34.409 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.406+0000 7f8e616f9e40 1 -- 172.21.15.167:0/2399019461 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:86 7.0 7:05bf5b68:::notify.1:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e32) v8 -- 0x56003d20bb20 con 0x56003d1cd620 2023-10-05T19:29:34.412 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.406+0000 7f8e5ea2d700 1 -- 172.21.15.167:0/2399019461 <== osd.2 v2:172.21.15.167:6816/30037 14 ==== osd_op_reply(85 notify.0 [watch watch cookie 94559025538128] v32'238 uv237 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f8e4802f040 con 0x7f8e58007030 2023-10-05T19:29:34.412 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.406+0000 7f8e5fa2f700 1 -- 172.21.15.167:0/2399019461 <== osd.1 v2:172.21.15.167:6800/30035 29 ==== osd_op_reply(86 notify.1 [create] v32'237 uv237 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f8e50050040 con 0x56003d1cd620 2023-10-05T19:29:34.412 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.406+0000 7f8e616f9e40 1 -- 172.21.15.167:0/2399019461 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:87 7.0 7:05bf5b68:::notify.1:head [watch watch cookie 94559025541728] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e32) v8 -- 0x56003d20bf20 con 0x56003d1cd620 2023-10-05T19:29:34.412 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.406+0000 7f8e616f9e40 1 -- 172.21.15.167:0/2399019461 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:88 7.15 7:a93a5511:::notify.2:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e32) v8 -- 0x56003d20c7c0 con 0x56003d205100 2023-10-05T19:29:34.415 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.410+0000 7f8e5fa2f700 1 -- 172.21.15.167:0/2399019461 <== osd.1 v2:172.21.15.167:6800/30035 30 ==== osd_op_reply(87 notify.1 [watch watch cookie 94559025541728] v32'238 uv237 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f8e50050040 con 0x56003d1cd620 2023-10-05T19:29:34.415 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.410+0000 7f8e5f22e700 1 -- 172.21.15.167:0/2399019461 <== osd.3 v2:172.21.15.167:6808/30036 22 ==== osd_op_reply(88 notify.2 [create] v32'237 uv237 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f8e58041040 con 0x56003d205100 2023-10-05T19:29:34.415 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.410+0000 7f8e616f9e40 1 -- 172.21.15.167:0/2399019461 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:89 7.15 7:a93a5511:::notify.2:head [watch watch cookie 94559025544992] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e32) v8 -- 0x56003d20b7d0 con 0x56003d205100 2023-10-05T19:29:34.415 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.410+0000 7f8e616f9e40 1 -- 172.21.15.167:0/2399019461 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:90 7.e 7:7759931f:::notify.3:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e32) v8 -- 0x56003d20d550 con 0x7f8e58007030 2023-10-05T19:29:34.418 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.414+0000 7f8e5f22e700 1 -- 172.21.15.167:0/2399019461 <== osd.3 v2:172.21.15.167:6808/30036 23 ==== osd_op_reply(89 notify.2 [watch watch cookie 94559025544992] v32'238 uv237 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f8e58041040 con 0x56003d205100 2023-10-05T19:29:34.418 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.414+0000 7f8e5ea2d700 1 -- 172.21.15.167:0/2399019461 <== osd.2 v2:172.21.15.167:6816/30037 15 ==== osd_op_reply(90 notify.3 [create] v32'237 uv237 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f8e4802f040 con 0x7f8e58007030 2023-10-05T19:29:34.418 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.414+0000 7f8e616f9e40 1 -- 172.21.15.167:0/2399019461 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:91 7.e 7:7759931f:::notify.3:head [watch watch cookie 94559025548144] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e32) v8 -- 0x56003d20d900 con 0x7f8e58007030 2023-10-05T19:29:34.418 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.414+0000 7f8e616f9e40 1 -- 172.21.15.167:0/2399019461 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:92 7.d 7:b4812045:::notify.4:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e32) v8 -- 0x56003d20e490 con 0x56003d1cd620 2023-10-05T19:29:34.420 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.414+0000 7f8e5ea2d700 1 -- 172.21.15.167:0/2399019461 <== osd.2 v2:172.21.15.167:6816/30037 16 ==== osd_op_reply(91 notify.3 [watch watch cookie 94559025548144] v32'238 uv237 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f8e4802f040 con 0x7f8e58007030 2023-10-05T19:29:34.421 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.418+0000 7f8e5fa2f700 1 -- 172.21.15.167:0/2399019461 <== osd.1 v2:172.21.15.167:6800/30035 31 ==== osd_op_reply(92 notify.4 [create] v32'237 uv237 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f8e50050040 con 0x56003d1cd620 2023-10-05T19:29:34.421 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.418+0000 7f8e616f9e40 1 -- 172.21.15.167:0/2399019461 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:93 7.d 7:b4812045:::notify.4:head [watch watch cookie 94559025549248] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e32) v8 -- 0x56003d20e890 con 0x56003d1cd620 2023-10-05T19:29:34.421 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.418+0000 7f8e616f9e40 1 -- 172.21.15.167:0/2399019461 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:94 7.3 7:c609908c:::notify.5:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e32) v8 -- 0x56003d20f2a0 con 0x7f8e48011040 2023-10-05T19:29:34.423 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.418+0000 7f8e5fa2f700 1 -- 172.21.15.167:0/2399019461 <== osd.0 v2:172.21.15.118:6800/27690 23 ==== osd_op_reply(94 notify.5 [create] v32'237 uv237 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f8e50050040 con 0x7f8e48011040 2023-10-05T19:29:34.423 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.418+0000 7f8e616f9e40 1 -- 172.21.15.167:0/2399019461 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:95 7.3 7:c609908c:::notify.5:head [watch watch cookie 94559025556912] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e32) v8 -- 0x56003d20f6f0 con 0x7f8e48011040 2023-10-05T19:29:34.424 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.418+0000 7f8e616f9e40 1 -- 172.21.15.167:0/2399019461 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:96 7.14 7:2b04a3e9:::notify.6:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e32) v8 -- 0x56003d210100 con 0x7f8e48011040 2023-10-05T19:29:34.424 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.418+0000 7f8e5fa2f700 1 -- 172.21.15.167:0/2399019461 <== osd.1 v2:172.21.15.167:6800/30035 32 ==== osd_op_reply(93 notify.4 [watch watch cookie 94559025549248] v32'238 uv237 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f8e50050040 con 0x56003d1cd620 2023-10-05T19:29:34.426 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.422+0000 7f8e5fa2f700 1 -- 172.21.15.167:0/2399019461 <== osd.0 v2:172.21.15.118:6800/27690 24 ==== osd_op_reply(95 notify.5 [watch watch cookie 94559025556912] v32'238 uv237 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f8e50050040 con 0x7f8e48011040 2023-10-05T19:29:34.427 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.422+0000 7f8e5fa2f700 1 -- 172.21.15.167:0/2399019461 <== osd.0 v2:172.21.15.118:6800/27690 25 ==== osd_op_reply(96 notify.6 [create] v32'237 uv237 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f8e50050040 con 0x7f8e48011040 2023-10-05T19:29:34.427 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.422+0000 7f8e616f9e40 1 -- 172.21.15.167:0/2399019461 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:97 7.14 7:2b04a3e9:::notify.6:head [watch watch cookie 94559025560592] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e32) v8 -- 0x56003d20f6f0 con 0x7f8e48011040 2023-10-05T19:29:34.427 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.422+0000 7f8e616f9e40 1 -- 172.21.15.167:0/2399019461 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:98 7.9 7:93e5b521:::notify.7:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e32) v8 -- 0x56003d210060 con 0x7f8e48011040 2023-10-05T19:29:34.428 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.426+0000 7f8e5fa2f700 1 -- 172.21.15.167:0/2399019461 <== osd.0 v2:172.21.15.118:6800/27690 26 ==== osd_op_reply(97 notify.6 [watch watch cookie 94559025560592] v32'238 uv237 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f8e50050040 con 0x7f8e48011040 2023-10-05T19:29:34.429 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.426+0000 7f8e5fa2f700 1 -- 172.21.15.167:0/2399019461 <== osd.0 v2:172.21.15.118:6800/27690 27 ==== osd_op_reply(98 notify.7 [create] v32'237 uv237 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f8e50050040 con 0x7f8e48011040 2023-10-05T19:29:34.429 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.426+0000 7f8e616f9e40 1 -- 172.21.15.167:0/2399019461 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:99 7.9 7:93e5b521:::notify.7:head [watch watch cookie 94559025558576] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e32) v8 -- 0x56003d210550 con 0x7f8e48011040 2023-10-05T19:29:34.429 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.426+0000 7f8e616f9e40 20 add_watcher() i=0 2023-10-05T19:29:34.429 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.426+0000 7f8e616f9e40 20 add_watcher() i=1 2023-10-05T19:29:34.430 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.426+0000 7f8e616f9e40 20 add_watcher() i=2 2023-10-05T19:29:34.430 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.426+0000 7f8e616f9e40 20 add_watcher() i=3 2023-10-05T19:29:34.430 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.426+0000 7f8e616f9e40 20 add_watcher() i=4 2023-10-05T19:29:34.430 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.426+0000 7f8e616f9e40 20 add_watcher() i=5 2023-10-05T19:29:34.430 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.426+0000 7f8e616f9e40 20 add_watcher() i=6 2023-10-05T19:29:34.431 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.426+0000 7f8e5fa2f700 1 -- 172.21.15.167:0/2399019461 <== osd.0 v2:172.21.15.118:6800/27690 28 ==== osd_op_reply(99 notify.7 [watch watch cookie 94559025558576] v32'238 uv237 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f8e50050040 con 0x7f8e48011040 2023-10-05T19:29:34.432 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.426+0000 7f8e616f9e40 20 add_watcher() i=7 2023-10-05T19:29:34.432 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.426+0000 7f8e616f9e40 2 all 8 watchers are set, enabling cache 2023-10-05T19:29:34.432 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.426+0000 7f8e616f9e40 1 -- 172.21.15.167:0/2399019461 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:100 6.4 6:22d26bf9:::data_loggenerations_metadata:head [call version.check_conds in=74b,call version.read in=11b,read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32) v8 -- 0x56003d2136f0 con 0x56003d1cd620 2023-10-05T19:29:34.432 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.430+0000 7f8e5fa2f700 1 -- 172.21.15.167:0/2399019461 <== osd.1 v2:172.21.15.167:6800/30035 33 ==== osd_op_reply(100 data_loggenerations_metadata [call,call out=48b,read 0~28 out=28b] v0'0 uv1 ondisk = 0) v8 ==== 256+0+76 (crc 0 0 0) 0x7f8e50050040 con 0x56003d1cd620 2023-10-05T19:29:34.432 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.430+0000 7f8e616f9e40 1 -- 172.21.15.167:0/2399019461 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:101 6.4 6:22d26bf9:::data_loggenerations_metadata:head [watch watch cookie 94559025562064] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e32) v8 -- 0x56003d210c80 con 0x56003d1cd620 2023-10-05T19:29:34.435 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.430+0000 7f8e5fa2f700 1 -- 172.21.15.167:0/2399019461 <== osd.1 v2:172.21.15.167:6800/30035 34 ==== osd_op_reply(101 data_loggenerations_metadata [watch watch cookie 94559025562064] v32'231 uv1 ondisk = 0) v8 ==== 172+0+0 (crc 0 0 0) 0x7f8e50050040 con 0x56003d1cd620 2023-10-05T19:29:34.435 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.430+0000 7f8e2ffef700 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2023-10-05T19:29:34.435 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.430+0000 7f8e616f9e40 20 check_secure_mon_conn(): auth registy supported: methods=[2] modes=[2,1] 2023-10-05T19:29:34.435 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.430+0000 7f8e616f9e40 20 check_secure_mon_conn(): mode 1 is insecure 2023-10-05T19:29:34.436 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.430+0000 7f8e616f9e40 1 Processor -- start 2023-10-05T19:29:34.436 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.430+0000 7f8e616f9e40 1 -- start start 2023-10-05T19:29:34.436 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.430+0000 7f8e616f9e40 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x56003d21dbb0 0x56003d21df80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-05T19:29:34.436 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.430+0000 7f8e616f9e40 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x56003d21e4c0 con 0x56003d21dbb0 2023-10-05T19:29:34.437 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.434+0000 7f8e5ea2d700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x56003d21dbb0 0x56003d21df80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-05T19:29:34.437 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.434+0000 7f8e5ea2d700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x56003d21dbb0 0x56003d21df80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.118:3300/0 says I am v2:172.21.15.167:56684/0 (socket says 172.21.15.167:56684) 2023-10-05T19:29:34.437 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.434+0000 7f8e5ea2d700 1 -- 172.21.15.167:0/959882501 learned_addr learned my addr 172.21.15.167:0/959882501 (peer_addr_for_me v2:172.21.15.167:0/0) 2023-10-05T19:29:34.437 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.434+0000 7f8e5ea2d700 1 -- 172.21.15.167:0/959882501 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x56003d21e600 con 0x56003d21dbb0 2023-10-05T19:29:34.438 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.434+0000 7f8e5ea2d700 1 --2- 172.21.15.167:0/959882501 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x56003d21dbb0 0x56003d21df80 secure :-1 s=READY pgs=612 cs=0 l=1 rev1=1 crypto rx=0x56003d202e50 tx=0x7f8e48006000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b2b296800aa5c57d server_cookie=0 in_seq=0 out_seq=0 2023-10-05T19:29:34.438 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.434+0000 7f8e2f7ee700 1 -- 172.21.15.167:0/959882501 <== mon.0 v2:172.21.15.118:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f8e4802c040 con 0x56003d21dbb0 2023-10-05T19:29:34.438 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.434+0000 7f8e2f7ee700 1 -- 172.21.15.167:0/959882501 <== mon.0 v2:172.21.15.118:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8e4802d040 con 0x56003d21dbb0 2023-10-05T19:29:34.438 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.434+0000 7f8e2f7ee700 1 -- 172.21.15.167:0/959882501 <== mon.0 v2:172.21.15.118:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f8e48028ba0 con 0x56003d21dbb0 2023-10-05T19:29:34.439 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.434+0000 7f8e616f9e40 1 -- 172.21.15.167:0/959882501 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x56003d21dbb0 msgr2=0x56003d21df80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-05T19:29:34.439 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.434+0000 7f8e616f9e40 1 --2- 172.21.15.167:0/959882501 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x56003d21dbb0 0x56003d21df80 secure :-1 s=READY pgs=612 cs=0 l=1 rev1=1 crypto rx=0x56003d202e50 tx=0x7f8e48006000 comp rx=0 tx=0).stop 2023-10-05T19:29:34.440 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.434+0000 7f8e616f9e40 1 -- 172.21.15.167:0/959882501 shutdown_connections 2023-10-05T19:29:34.440 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.434+0000 7f8e616f9e40 1 --2- 172.21.15.167:0/959882501 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x56003d21dbb0 0x56003d21df80 unknown :-1 s=CLOSED pgs=612 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-05T19:29:34.440 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.434+0000 7f8e616f9e40 1 -- 172.21.15.167:0/959882501 >> 172.21.15.167:0/959882501 conn(0x56003d218f70 msgr2=0x56003d217120 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-05T19:29:34.440 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.434+0000 7f8e616f9e40 1 -- 172.21.15.167:0/959882501 shutdown_connections 2023-10-05T19:29:34.440 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.434+0000 7f8e616f9e40 1 -- 172.21.15.167:0/959882501 wait complete. 2023-10-05T19:29:34.441 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.434+0000 7f8e616f9e40 1 Processor -- start 2023-10-05T19:29:34.441 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.434+0000 7f8e616f9e40 1 -- start start 2023-10-05T19:29:34.441 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.438+0000 7f8e616f9e40 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x56003d21dbb0 0x56003d2236c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-05T19:29:34.441 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.438+0000 7f8e616f9e40 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x56003d2246c0 con 0x56003d21dbb0 2023-10-05T19:29:34.441 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.438+0000 7f8e5ea2d700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x56003d21dbb0 0x56003d2236c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-05T19:29:34.441 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.438+0000 7f8e5ea2d700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x56003d21dbb0 0x56003d2236c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.118:3300/0 says I am v2:172.21.15.167:56696/0 (socket says 172.21.15.167:56696) 2023-10-05T19:29:34.441 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.438+0000 7f8e5ea2d700 1 -- 172.21.15.167:0/1892848108 learned_addr learned my addr 172.21.15.167:0/1892848108 (peer_addr_for_me v2:172.21.15.167:0/0) 2023-10-05T19:29:34.442 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.438+0000 7f8e5ea2d700 1 -- 172.21.15.167:0/1892848108 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8e40001e10 con 0x56003d21dbb0 2023-10-05T19:29:34.442 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.438+0000 7f8e5ea2d700 1 --2- 172.21.15.167:0/1892848108 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x56003d21dbb0 0x56003d2236c0 secure :-1 s=READY pgs=613 cs=0 l=1 rev1=1 crypto rx=0x56003d217c90 tx=0x7f8e48035030 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-05T19:29:34.442 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.438+0000 7f8e2dfeb700 1 -- 172.21.15.167:0/1892848108 <== mon.0 v2:172.21.15.118:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f8e4802c030 con 0x56003d21dbb0 2023-10-05T19:29:34.442 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.438+0000 7f8e616f9e40 1 -- 172.21.15.167:0/1892848108 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x56003d223cc0 con 0x56003d21dbb0 2023-10-05T19:29:34.443 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.438+0000 7f8e2dfeb700 1 -- 172.21.15.167:0/1892848108 <== mon.0 v2:172.21.15.118:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8e4801c040 con 0x56003d21dbb0 2023-10-05T19:29:34.443 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.438+0000 7f8e2dfeb700 1 -- 172.21.15.167:0/1892848108 <== mon.0 v2:172.21.15.118:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f8e48004190 con 0x56003d21dbb0 2023-10-05T19:29:34.443 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.438+0000 7f8e616f9e40 1 -- 172.21.15.167:0/1892848108 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x56003d224100 con 0x56003d21dbb0 2023-10-05T19:29:34.444 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.438+0000 7f8e2dfeb700 1 -- 172.21.15.167:0/1892848108 <== mon.0 v2:172.21.15.118:3300/0 4 ==== mgrmap(e 5) v1 ==== 40329+0+0 (secure 0 0 0) 0x7f8e48045070 con 0x56003d21dbb0 2023-10-05T19:29:34.444 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.438+0000 7f8e2dfeb700 1 --2- 172.21.15.167:0/1892848108 >> [v2:172.21.15.167:6824/29889,v1:172.21.15.167:6825/29889] conn(0x7f8da4030b70 0x7f8da4033020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-05T19:29:34.444 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.438+0000 7f8e2dfeb700 1 -- 172.21.15.167:0/1892848108 <== mon.0 v2:172.21.15.118:3300/0 5 ==== osd_map(32..32 src has 1..32) v4 ==== 6260+0+0 (secure 0 0 0) 0x7f8e48052d30 con 0x56003d21dbb0 2023-10-05T19:29:34.445 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.442+0000 7f8e5fa2f700 1 --2- 172.21.15.167:0/1892848108 >> [v2:172.21.15.167:6824/29889,v1:172.21.15.167:6825/29889] conn(0x7f8da4030b70 0x7f8da4033020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-05T19:29:34.445 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.442+0000 7f8e616f9e40 5 note: GC not initialized 2023-10-05T19:29:34.445 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.442+0000 7f8e5fa2f700 1 --2- 172.21.15.167:0/1892848108 >> [v2:172.21.15.167:6824/29889,v1:172.21.15.167:6825/29889] conn(0x7f8da4030b70 0x7f8da4033020 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x56003d210be0 tx=0x7f8e50000f90 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-05T19:29:34.447 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.442+0000 7f8d9effd700 20 reqs_thread_entry: start 2023-10-05T19:29:34.447 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.442+0000 7f8d9cff9700 20 reqs_thread_entry: start 2023-10-05T19:29:34.447 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.442+0000 7f8e616f9e40 20 init_complete bucket index max shards: 11 2023-10-05T19:29:34.448 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.442+0000 7f8e616f9e40 10 rgw notify: Started notification manager with: 1 workers 2023-10-05T19:29:34.448 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.442+0000 7f8d8effd700 20 rgw notify: INFO: next queues processing will happen at: Thu Oct 5 19:30:04 2023 2023-10-05T19:29:34.448 INFO:teuthology.orchestra.run.smithi167.stderr: 2023-10-05T19:29:34.448 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.442+0000 7f8d8e7fc700 20 reqs_thread_entry: start 2023-10-05T19:29:34.449 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.442+0000 7f8e616f9e40 20 get_system_obj_state: rctx=0x7ffe801a6338 obj=default.rgw.meta:users.uid:foo state=0x56003d0a70b0 s->prefetch_data=0 2023-10-05T19:29:34.449 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.442+0000 7f8e616f9e40 1 -- 172.21.15.167:0/2399019461 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:102 8.18 8:1e6ca5ed:users.uid::foo:head [call version.read in=11b,getxattrs,stat] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32) v8 -- 0x56003d28beb0 con 0x56003d1cd620 2023-10-05T19:29:34.449 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.446+0000 7f8e5fa2f700 1 -- 172.21.15.167:0/2399019461 <== osd.1 v2:172.21.15.167:6800/30035 35 ==== osd_op_reply(102 foo [call out=48b,getxattrs out=75b,stat out=16b] v0'0 uv15 ondisk = 0) v8 ==== 231+0+139 (crc 0 0 0) 0x7f8e50050040 con 0x56003d1cd620 2023-10-05T19:29:34.449 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.446+0000 7f8e616f9e40 20 get_system_obj_state: s->obj_tag was set empty 2023-10-05T19:29:34.450 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.446+0000 7f8e616f9e40 20 Read xattr: user.rgw.idtag 2023-10-05T19:29:34.450 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.446+0000 7f8e616f9e40 20 rados->read ofs=0 len=0 2023-10-05T19:29:34.450 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.446+0000 7f8e616f9e40 1 -- 172.21.15.167:0/2399019461 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:103 8.18 8:1e6ca5ed:users.uid::foo:head [call version.check_conds in=122b,call version.read in=11b,read 0~0,getxattrs] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32) v8 -- 0x56003d28e940 con 0x56003d1cd620 2023-10-05T19:29:34.450 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.446+0000 7f8e5fa2f700 1 -- 172.21.15.167:0/2399019461 <== osd.1 v2:172.21.15.167:6800/30035 36 ==== osd_op_reply(103 foo [call,call out=48b,read 0~357 out=357b,getxattrs out=75b] v0'0 uv15 ondisk = 0) v8 ==== 273+0+480 (crc 0 0 0) 0x7f8e50050040 con 0x56003d1cd620 2023-10-05T19:29:34.451 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.446+0000 7f8e616f9e40 20 rados_obj.operate() r=0 bl.length=357 2023-10-05T19:29:34.451 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.446+0000 7f8e616f9e40 1 -- 172.21.15.167:0/2399019461 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:104 6.13 6:cea599e4:usage::usage.10:head [call rgw.user_usage_log_read in=63b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32) v8 -- 0x56003d28d930 con 0x7f8e48011040 2023-10-05T19:29:34.451 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.446+0000 7f8e5fa2f700 1 -- 172.21.15.167:0/2399019461 <== osd.0 v2:172.21.15.118:6800/27690 29 ==== osd_op_reply(104 usage.10 [call out=1268b] v0'0 uv30 ondisk = 0) v8 ==== 152+0+1268 (crc 0 0 0) 0x7f8e50050040 con 0x7f8e48011040 2023-10-05T19:29:34.452 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.446+0000 7f8e616f9e40 1 -- 172.21.15.167:0/2399019461 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:105 7.1f 7:f95f44c2:::notify.0:head [watch unwatch cookie 94559025538128] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e32) v8 -- 0x56003d24a2a0 con 0x7f8e58007030 2023-10-05T19:29:34.455 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.450+0000 7f8e5ea2d700 1 -- 172.21.15.167:0/2399019461 <== osd.2 v2:172.21.15.167:6816/30037 17 ==== osd_op_reply(105 notify.0 [watch unwatch cookie 94559025538128] v32'239 uv237 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f8e4802f040 con 0x7f8e58007030 2023-10-05T19:29:34.455 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.450+0000 7f8e616f9e40 20 remove_watcher() i=0 2023-10-05T19:29:34.455 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.450+0000 7f8e616f9e40 2 removed watcher, disabling cache 2023-10-05T19:29:34.456 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.450+0000 7f8e616f9e40 1 -- 172.21.15.167:0/2399019461 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:106 7.0 7:05bf5b68:::notify.1:head [watch unwatch cookie 94559025541728] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e32) v8 -- 0x56003d20a450 con 0x56003d1cd620 2023-10-05T19:29:34.458 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.454+0000 7f8e5fa2f700 1 -- 172.21.15.167:0/2399019461 <== osd.1 v2:172.21.15.167:6800/30035 37 ==== osd_op_reply(106 notify.1 [watch unwatch cookie 94559025541728] v32'239 uv237 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f8e50050040 con 0x56003d1cd620 2023-10-05T19:29:34.458 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.454+0000 7f8e616f9e40 20 remove_watcher() i=1 2023-10-05T19:29:34.458 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.454+0000 7f8e616f9e40 1 -- 172.21.15.167:0/2399019461 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:107 7.15 7:a93a5511:::notify.2:head [watch unwatch cookie 94559025544992] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e32) v8 -- 0x56003d20a450 con 0x56003d205100 2023-10-05T19:29:34.461 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.458+0000 7f8e5f22e700 1 -- 172.21.15.167:0/2399019461 <== osd.3 v2:172.21.15.167:6808/30036 24 ==== osd_op_reply(107 notify.2 [watch unwatch cookie 94559025544992] v32'239 uv237 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f8e58041040 con 0x56003d205100 2023-10-05T19:29:34.461 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.458+0000 7f8e616f9e40 20 remove_watcher() i=2 2023-10-05T19:29:34.461 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.458+0000 7f8e616f9e40 1 -- 172.21.15.167:0/2399019461 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:108 7.e 7:7759931f:::notify.3:head [watch unwatch cookie 94559025548144] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e32) v8 -- 0x56003d20bf20 con 0x7f8e58007030 2023-10-05T19:29:34.464 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.458+0000 7f8e5ea2d700 1 -- 172.21.15.167:0/2399019461 <== osd.2 v2:172.21.15.167:6816/30037 18 ==== osd_op_reply(108 notify.3 [watch unwatch cookie 94559025548144] v32'239 uv237 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f8e4802f040 con 0x7f8e58007030 2023-10-05T19:29:34.464 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.458+0000 7f8e616f9e40 20 remove_watcher() i=3 2023-10-05T19:29:34.464 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.458+0000 7f8e616f9e40 1 -- 172.21.15.167:0/2399019461 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:109 7.d 7:b4812045:::notify.4:head [watch unwatch cookie 94559025549248] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e32) v8 -- 0x56003d20a450 con 0x56003d1cd620 2023-10-05T19:29:34.467 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.462+0000 7f8e5fa2f700 1 -- 172.21.15.167:0/2399019461 <== osd.1 v2:172.21.15.167:6800/30035 38 ==== osd_op_reply(109 notify.4 [watch unwatch cookie 94559025549248] v32'239 uv237 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f8e50050040 con 0x56003d1cd620 2023-10-05T19:29:34.467 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.462+0000 7f8e616f9e40 20 remove_watcher() i=4 2023-10-05T19:29:34.467 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.462+0000 7f8e616f9e40 1 -- 172.21.15.167:0/2399019461 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:110 7.3 7:c609908c:::notify.5:head [watch unwatch cookie 94559025556912] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e32) v8 -- 0x56003d20d060 con 0x7f8e48011040 2023-10-05T19:29:34.470 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.466+0000 7f8e5fa2f700 1 -- 172.21.15.167:0/2399019461 <== osd.0 v2:172.21.15.118:6800/27690 30 ==== osd_op_reply(110 notify.5 [watch unwatch cookie 94559025556912] v32'239 uv237 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f8e50050040 con 0x7f8e48011040 2023-10-05T19:29:34.470 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.466+0000 7f8e616f9e40 20 remove_watcher() i=5 2023-10-05T19:29:34.470 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.466+0000 7f8e616f9e40 1 -- 172.21.15.167:0/2399019461 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:111 7.14 7:2b04a3e9:::notify.6:head [watch unwatch cookie 94559025560592] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e32) v8 -- 0x56003d20a450 con 0x7f8e48011040 2023-10-05T19:29:34.473 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.470+0000 7f8e5fa2f700 1 -- 172.21.15.167:0/2399019461 <== osd.0 v2:172.21.15.118:6800/27690 31 ==== osd_op_reply(111 notify.6 [watch unwatch cookie 94559025560592] v32'239 uv237 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f8e50050040 con 0x7f8e48011040 2023-10-05T19:29:34.473 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.470+0000 7f8e616f9e40 20 remove_watcher() i=6 2023-10-05T19:29:34.473 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.470+0000 7f8e616f9e40 1 -- 172.21.15.167:0/2399019461 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:112 7.9 7:93e5b521:::notify.7:head [watch unwatch cookie 94559025558576] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e32) v8 -- 0x56003d20a450 con 0x7f8e48011040 2023-10-05T19:29:34.476 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.470+0000 7f8e5fa2f700 1 -- 172.21.15.167:0/2399019461 <== osd.0 v2:172.21.15.118:6800/27690 32 ==== osd_op_reply(112 notify.7 [watch unwatch cookie 94559025558576] v32'239 uv237 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f8e50050040 con 0x7f8e48011040 2023-10-05T19:29:34.476 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.470+0000 7f8e616f9e40 20 remove_watcher() i=7 2023-10-05T19:29:34.477 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.474+0000 7f8e2ffef700 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2023-10-05T19:29:34.478 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.474+0000 7f8e616f9e40 1 -- 172.21.15.167:0/2399019461 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:113 6.4 6:22d26bf9:::data_loggenerations_metadata:head [watch unwatch cookie 94559025562064] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e32) v8 -- 0x56003d20a450 con 0x56003d1cd620 2023-10-05T19:29:34.481 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.474+0000 7f8e5fa2f700 1 -- 172.21.15.167:0/2399019461 <== osd.1 v2:172.21.15.167:6800/30035 39 ==== osd_op_reply(113 data_loggenerations_metadata [watch unwatch cookie 94559025562064] v32'232 uv1 ondisk = 0) v8 ==== 172+0+0 (crc 0 0 0) 0x7f8e50050040 con 0x56003d1cd620 2023-10-05T19:29:34.481 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.478+0000 7f8e616f9e40 1 -- 172.21.15.167:0/2399019461 >> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] conn(0x7f8e48011040 msgr2=0x7f8e48014950 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-05T19:29:34.481 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.478+0000 7f8e616f9e40 1 --2- 172.21.15.167:0/2399019461 >> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] conn(0x7f8e48011040 0x7f8e48014950 crc :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-05T19:29:34.481 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.478+0000 7f8e616f9e40 1 -- 172.21.15.167:0/2399019461 >> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] conn(0x56003d1cd620 msgr2=0x56003d2016f0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-05T19:29:34.481 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.478+0000 7f8e616f9e40 1 --2- 172.21.15.167:0/2399019461 >> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] conn(0x56003d1cd620 0x56003d2016f0 crc :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-05T19:29:34.481 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.478+0000 7f8e616f9e40 1 -- 172.21.15.167:0/2399019461 >> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] conn(0x7f8e58007030 msgr2=0x7f8e58031490 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-05T19:29:34.481 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.478+0000 7f8e616f9e40 1 --2- 172.21.15.167:0/2399019461 >> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] conn(0x7f8e58007030 0x7f8e58031490 crc :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-05T19:29:34.481 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.478+0000 7f8e616f9e40 1 -- 172.21.15.167:0/2399019461 >> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] conn(0x56003d205100 msgr2=0x56003d2074e0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-05T19:29:34.482 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.478+0000 7f8e616f9e40 1 --2- 172.21.15.167:0/2399019461 >> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] conn(0x56003d205100 0x56003d2074e0 crc :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-05T19:29:34.482 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.478+0000 7f8e616f9e40 1 -- 172.21.15.167:0/2399019461 >> [v2:172.21.15.167:6824/29889,v1:172.21.15.167:6825/29889] conn(0x7f8e44030ca0 msgr2=0x7f8e44033150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-05T19:29:34.482 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.478+0000 7f8e616f9e40 1 --2- 172.21.15.167:0/2399019461 >> [v2:172.21.15.167:6824/29889,v1:172.21.15.167:6825/29889] conn(0x7f8e44030ca0 0x7f8e44033150 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7f8e480044f0 tx=0x7f8e48006690 comp rx=0 tx=0).stop 2023-10-05T19:29:34.482 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.478+0000 7f8e616f9e40 1 -- 172.21.15.167:0/2399019461 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x56003d1ccd10 msgr2=0x56003d1f5c60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-05T19:29:34.482 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.478+0000 7f8e5fa2f700 1 -- 172.21.15.167:0/2399019461 reap_dead start 2023-10-05T19:29:34.482 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.478+0000 7f8e616f9e40 1 --2- 172.21.15.167:0/2399019461 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x56003d1ccd10 0x56003d1f5c60 secure :-1 s=READY pgs=611 cs=0 l=1 rev1=1 crypto rx=0x7f8e5801f040 tx=0x7f8e5801c000 comp rx=0 tx=0).stop 2023-10-05T19:29:34.482 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.478+0000 7f8e616f9e40 1 -- 172.21.15.167:0/2399019461 shutdown_connections 2023-10-05T19:29:34.482 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.478+0000 7f8e616f9e40 1 --2- 172.21.15.167:0/2399019461 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x56003d1ccd10 0x56003d1f5c60 unknown :-1 s=CLOSED pgs=611 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-05T19:29:34.482 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.478+0000 7f8e616f9e40 1 -- 172.21.15.167:0/2399019461 >> 172.21.15.167:0/2399019461 conn(0x56003d1c8730 msgr2=0x56003d1e73f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-05T19:29:34.483 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.478+0000 7f8e616f9e40 1 -- 172.21.15.167:0/2399019461 shutdown_connections 2023-10-05T19:29:34.483 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.478+0000 7f8e616f9e40 1 -- 172.21.15.167:0/2399019461 wait complete. 2023-10-05T19:29:34.483 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.478+0000 7f8e616f9e40 1 -- 172.21.15.167:0/1892848108 >> [v2:172.21.15.167:6824/29889,v1:172.21.15.167:6825/29889] conn(0x7f8da4030b70 msgr2=0x7f8da4033020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-05T19:29:34.484 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.478+0000 7f8e616f9e40 1 --2- 172.21.15.167:0/1892848108 >> [v2:172.21.15.167:6824/29889,v1:172.21.15.167:6825/29889] conn(0x7f8da4030b70 0x7f8da4033020 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x56003d210be0 tx=0x7f8e50000f90 comp rx=0 tx=0).stop 2023-10-05T19:29:34.484 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.478+0000 7f8e616f9e40 1 -- 172.21.15.167:0/1892848108 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x56003d21dbb0 msgr2=0x56003d2236c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-05T19:29:34.484 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.478+0000 7f8e616f9e40 1 --2- 172.21.15.167:0/1892848108 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x56003d21dbb0 0x56003d2236c0 secure :-1 s=READY pgs=613 cs=0 l=1 rev1=1 crypto rx=0x56003d217c90 tx=0x7f8e48035030 comp rx=0 tx=0).stop 2023-10-05T19:29:34.484 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.478+0000 7f8e616f9e40 1 -- 172.21.15.167:0/1892848108 shutdown_connections 2023-10-05T19:29:34.484 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.478+0000 7f8e616f9e40 1 --2- 172.21.15.167:0/1892848108 >> [v2:172.21.15.167:6824/29889,v1:172.21.15.167:6825/29889] conn(0x7f8da4030b70 0x7f8da4033020 unknown :-1 s=CLOSED pgs=211 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-05T19:29:34.484 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.478+0000 7f8e616f9e40 1 --2- 172.21.15.167:0/1892848108 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x56003d21dbb0 0x56003d2236c0 unknown :-1 s=CLOSED pgs=613 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-05T19:29:34.484 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.478+0000 7f8e616f9e40 1 -- 172.21.15.167:0/1892848108 >> 172.21.15.167:0/1892848108 conn(0x56003d218f70 msgr2=0x56003d217070 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-05T19:29:34.484 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.478+0000 7f8e616f9e40 1 -- 172.21.15.167:0/1892848108 shutdown_connections 2023-10-05T19:29:34.484 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.478+0000 7f8e616f9e40 1 -- 172.21.15.167:0/1892848108 wait complete. 2023-10-05T19:29:34.485 INFO:teuthology.orchestra.run.smithi167.stdout:{ 2023-10-05T19:29:34.485 INFO:teuthology.orchestra.run.smithi167.stdout: "entries": [ 2023-10-05T19:29:34.485 INFO:teuthology.orchestra.run.smithi167.stdout: { 2023-10-05T19:29:34.485 INFO:teuthology.orchestra.run.smithi167.stdout: "user": "foo", 2023-10-05T19:29:34.485 INFO:teuthology.orchestra.run.smithi167.stdout: "buckets": [ 2023-10-05T19:29:34.485 INFO:teuthology.orchestra.run.smithi167.stdout: { 2023-10-05T19:29:34.485 INFO:teuthology.orchestra.run.smithi167.stdout: "bucket": "", 2023-10-05T19:29:34.485 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2023-10-05T19:00:00.000000Z", 2023-10-05T19:29:34.486 INFO:teuthology.orchestra.run.smithi167.stdout: "epoch": 1696532400, 2023-10-05T19:29:34.486 INFO:teuthology.orchestra.run.smithi167.stdout: "owner": "foo", 2023-10-05T19:29:34.486 INFO:teuthology.orchestra.run.smithi167.stdout: "categories": [] 2023-10-05T19:29:34.486 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2023-10-05T19:29:34.486 INFO:teuthology.orchestra.run.smithi167.stdout: { 2023-10-05T19:29:34.486 INFO:teuthology.orchestra.run.smithi167.stdout: "bucket": "myfoo", 2023-10-05T19:29:34.486 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2023-10-05T19:00:00.000000Z", 2023-10-05T19:29:34.486 INFO:teuthology.orchestra.run.smithi167.stdout: "epoch": 1696532400, 2023-10-05T19:29:34.486 INFO:teuthology.orchestra.run.smithi167.stdout: "owner": "foo", 2023-10-05T19:29:34.486 INFO:teuthology.orchestra.run.smithi167.stdout: "categories": [ 2023-10-05T19:29:34.486 INFO:teuthology.orchestra.run.smithi167.stdout: { 2023-10-05T19:29:34.486 INFO:teuthology.orchestra.run.smithi167.stdout: "category": "delete_bucket", 2023-10-05T19:29:34.487 INFO:teuthology.orchestra.run.smithi167.stdout: "bytes_sent": 217, 2023-10-05T19:29:34.487 INFO:teuthology.orchestra.run.smithi167.stdout: "bytes_received": 0, 2023-10-05T19:29:34.487 INFO:teuthology.orchestra.run.smithi167.stdout: "ops": 2, 2023-10-05T19:29:34.487 INFO:teuthology.orchestra.run.smithi167.stdout: "successful_ops": 1 2023-10-05T19:29:34.487 INFO:teuthology.orchestra.run.smithi167.stdout: } 2023-10-05T19:29:34.487 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2023-10-05T19:29:34.487 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2023-10-05T19:29:34.487 INFO:teuthology.orchestra.run.smithi167.stdout: { 2023-10-05T19:29:34.487 INFO:teuthology.orchestra.run.smithi167.stdout: "bucket": "myfoo2", 2023-10-05T19:29:34.487 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2023-10-05T19:00:00.000000Z", 2023-10-05T19:29:34.487 INFO:teuthology.orchestra.run.smithi167.stdout: "epoch": 1696532400, 2023-10-05T19:29:34.487 INFO:teuthology.orchestra.run.smithi167.stdout: "owner": "foo", 2023-10-05T19:29:34.488 INFO:teuthology.orchestra.run.smithi167.stdout: "categories": [ 2023-10-05T19:29:34.488 INFO:teuthology.orchestra.run.smithi167.stdout: { 2023-10-05T19:29:34.488 INFO:teuthology.orchestra.run.smithi167.stdout: "category": "delete_bucket", 2023-10-05T19:29:34.488 INFO:teuthology.orchestra.run.smithi167.stdout: "bytes_sent": 0, 2023-10-05T19:29:34.488 INFO:teuthology.orchestra.run.smithi167.stdout: "bytes_received": 0, 2023-10-05T19:29:34.488 INFO:teuthology.orchestra.run.smithi167.stdout: "ops": 1, 2023-10-05T19:29:34.488 INFO:teuthology.orchestra.run.smithi167.stdout: "successful_ops": 1 2023-10-05T19:29:34.488 INFO:teuthology.orchestra.run.smithi167.stdout: } 2023-10-05T19:29:34.488 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2023-10-05T19:29:34.488 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2023-10-05T19:29:34.488 INFO:teuthology.orchestra.run.smithi167.stdout: { 2023-10-05T19:29:34.488 INFO:teuthology.orchestra.run.smithi167.stdout: "bucket": "myfoo3", 2023-10-05T19:29:34.489 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2023-10-05T19:00:00.000000Z", 2023-10-05T19:29:34.489 INFO:teuthology.orchestra.run.smithi167.stdout: "epoch": 1696532400, 2023-10-05T19:29:34.489 INFO:teuthology.orchestra.run.smithi167.stdout: "owner": "foo", 2023-10-05T19:29:34.489 INFO:teuthology.orchestra.run.smithi167.stdout: "categories": [ 2023-10-05T19:29:34.489 INFO:teuthology.orchestra.run.smithi167.stdout: { 2023-10-05T19:29:34.489 INFO:teuthology.orchestra.run.smithi167.stdout: "category": "delete_bucket", 2023-10-05T19:29:34.489 INFO:teuthology.orchestra.run.smithi167.stdout: "bytes_sent": 0, 2023-10-05T19:29:34.489 INFO:teuthology.orchestra.run.smithi167.stdout: "bytes_received": 0, 2023-10-05T19:29:34.489 INFO:teuthology.orchestra.run.smithi167.stdout: "ops": 1, 2023-10-05T19:29:34.489 INFO:teuthology.orchestra.run.smithi167.stdout: "successful_ops": 1 2023-10-05T19:29:34.489 INFO:teuthology.orchestra.run.smithi167.stdout: } 2023-10-05T19:29:34.489 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2023-10-05T19:29:34.490 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2023-10-05T19:29:34.490 INFO:teuthology.orchestra.run.smithi167.stdout: { 2023-10-05T19:29:34.490 INFO:teuthology.orchestra.run.smithi167.stdout: "bucket": "myfoo4", 2023-10-05T19:29:34.490 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2023-10-05T19:00:00.000000Z", 2023-10-05T19:29:34.490 INFO:teuthology.orchestra.run.smithi167.stdout: "epoch": 1696532400, 2023-10-05T19:29:34.490 INFO:teuthology.orchestra.run.smithi167.stdout: "owner": "foo", 2023-10-05T19:29:34.490 INFO:teuthology.orchestra.run.smithi167.stdout: "categories": [ 2023-10-05T19:29:34.490 INFO:teuthology.orchestra.run.smithi167.stdout: { 2023-10-05T19:29:34.490 INFO:teuthology.orchestra.run.smithi167.stdout: "category": "delete_bucket", 2023-10-05T19:29:34.490 INFO:teuthology.orchestra.run.smithi167.stdout: "bytes_sent": 0, 2023-10-05T19:29:34.490 INFO:teuthology.orchestra.run.smithi167.stdout: "bytes_received": 0, 2023-10-05T19:29:34.490 INFO:teuthology.orchestra.run.smithi167.stdout: "ops": 1, 2023-10-05T19:29:34.491 INFO:teuthology.orchestra.run.smithi167.stdout: "successful_ops": 1 2023-10-05T19:29:34.491 INFO:teuthology.orchestra.run.smithi167.stdout: } 2023-10-05T19:29:34.491 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2023-10-05T19:29:34.491 INFO:teuthology.orchestra.run.smithi167.stdout: } 2023-10-05T19:29:34.491 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2023-10-05T19:29:34.491 INFO:teuthology.orchestra.run.smithi167.stdout: } 2023-10-05T19:29:34.491 INFO:teuthology.orchestra.run.smithi167.stdout: ], 2023-10-05T19:29:34.491 INFO:teuthology.orchestra.run.smithi167.stdout: "summary": [ 2023-10-05T19:29:34.491 INFO:teuthology.orchestra.run.smithi167.stdout: { 2023-10-05T19:29:34.491 INFO:teuthology.orchestra.run.smithi167.stdout: "user": "foo", 2023-10-05T19:29:34.491 INFO:teuthology.orchestra.run.smithi167.stdout: "categories": [ 2023-10-05T19:29:34.491 INFO:teuthology.orchestra.run.smithi167.stdout: { 2023-10-05T19:29:34.492 INFO:teuthology.orchestra.run.smithi167.stdout: "category": "delete_bucket", 2023-10-05T19:29:34.492 INFO:teuthology.orchestra.run.smithi167.stdout: "bytes_sent": 217, 2023-10-05T19:29:34.492 INFO:teuthology.orchestra.run.smithi167.stdout: "bytes_received": 0, 2023-10-05T19:29:34.492 INFO:teuthology.orchestra.run.smithi167.stdout: "ops": 5, 2023-10-05T19:29:34.492 INFO:teuthology.orchestra.run.smithi167.stdout: "successful_ops": 4 2023-10-05T19:29:34.492 INFO:teuthology.orchestra.run.smithi167.stdout: } 2023-10-05T19:29:34.492 INFO:teuthology.orchestra.run.smithi167.stdout: ], 2023-10-05T19:29:34.492 INFO:teuthology.orchestra.run.smithi167.stdout: "total": { 2023-10-05T19:29:34.492 INFO:teuthology.orchestra.run.smithi167.stdout: "bytes_sent": 217, 2023-10-05T19:29:34.492 INFO:teuthology.orchestra.run.smithi167.stdout: "bytes_received": 0, 2023-10-05T19:29:34.492 INFO:teuthology.orchestra.run.smithi167.stdout: "ops": 5, 2023-10-05T19:29:34.492 INFO:teuthology.orchestra.run.smithi167.stdout: "successful_ops": 4 2023-10-05T19:29:34.492 INFO:teuthology.orchestra.run.smithi167.stdout: } 2023-10-05T19:29:34.493 INFO:teuthology.orchestra.run.smithi167.stdout: } 2023-10-05T19:29:34.493 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2023-10-05T19:29:34.493 INFO:teuthology.orchestra.run.smithi167.stdout:} 2023-10-05T19:29:34.494 DEBUG:tasks.util.rgw: json result: {'entries': [{'user': 'foo', 'buckets': [{'bucket': '', 'time': '2023-10-05T19:00:00.000000Z', 'epoch': 1696532400, 'owner': 'foo', 'categories': []}, {'bucket': 'myfoo', 'time': '2023-10-05T19:00:00.000000Z', 'epoch': 1696532400, 'owner': 'foo', 'categories': [{'category': 'delete_bucket', 'bytes_sent': 217, 'bytes_received': 0, 'ops': 2, 'successful_ops': 1}]}, {'bucket': 'myfoo2', 'time': '2023-10-05T19:00:00.000000Z', 'epoch': 1696532400, 'owner': 'foo', 'categories': [{'category': 'delete_bucket', 'bytes_sent': 0, 'bytes_received': 0, 'ops': 1, 'successful_ops': 1}]}, {'bucket': 'myfoo3', 'time': '2023-10-05T19:00:00.000000Z', 'epoch': 1696532400, 'owner': 'foo', 'categories': [{'category': 'delete_bucket', 'bytes_sent': 0, 'bytes_received': 0, 'ops': 1, 'successful_ops': 1}]}, {'bucket': 'myfoo4', 'time': '2023-10-05T19:00:00.000000Z', 'epoch': 1696532400, 'owner': 'foo', 'categories': [{'category': 'delete_bucket', 'bytes_sent': 0, 'bytes_received': 0, 'ops': 1, 'successful_ops': 1}]}]}], 'summary': [{'user': 'foo', 'categories': [{'category': 'delete_bucket', 'bytes_sent': 217, 'bytes_received': 0, 'ops': 5, 'successful_ops': 4}], 'total': {'bytes_sent': 217, 'bytes_received': 0, 'ops': 5, 'successful_ops': 4}}]} 2023-10-05T19:29:34.494 INFO:tasks.util.rgw:rgwadmin: client.0 : ['user', 'create', '--uid', 'bar', '--display-name', 'Bar', '--access-key', 'NX5QOQKC6BH2IDN8HC7A', '--secret', 'LnEsqNNqZIpkzauboDcLXLcYaWwLQ3Kop0zAnKIn', '--max-buckets', '4'] 2023-10-05T19:29:34.494 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'ceph', 'user', 'create', '--uid', 'bar', '--display-name', 'Bar', '--access-key', 'NX5QOQKC6BH2IDN8HC7A', '--secret', 'LnEsqNNqZIpkzauboDcLXLcYaWwLQ3Kop0zAnKIn', '--max-buckets', '4'] 2023-10-05T19:29:34.494 DEBUG:teuthology.orchestra.run.smithi167:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster ceph user create --uid bar --display-name Bar --access-key NX5QOQKC6BH2IDN8HC7A --secret LnEsqNNqZIpkzauboDcLXLcYaWwLQ3Kop0zAnKIn --max-buckets 4 2023-10-05T19:29:34.582 INFO:teuthology.orchestra.run.smithi167.stderr:ignoring --setuser ceph since I am not root 2023-10-05T19:29:34.583 INFO:teuthology.orchestra.run.smithi167.stderr:ignoring --setgroup ceph since I am not root 2023-10-05T19:29:34.584 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.578+0000 7fa104ce2e40 1 Processor -- start 2023-10-05T19:29:34.584 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.578+0000 7fa104ce2e40 1 -- start start 2023-10-05T19:29:34.584 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.578+0000 7fa104ce2e40 1 --2- >> v2:172.21.15.118:3300/0 conn(0x558f84505d40 0x558f84506110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-05T19:29:34.584 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.578+0000 7fa104ce2e40 1 -- --> v1:172.21.15.118:6789/0 -- auth(proto 0 26 bytes epoch 0) v1 -- 0x558f84453bd0 con 0x558f84506650 2023-10-05T19:29:34.584 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.578+0000 7fa104ce2e40 1 -- --> v2:172.21.15.118:3300/0 -- mon_getmap magic: 0 v1 -- 0x558f84410ee0 con 0x558f84505d40 2023-10-05T19:29:34.585 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.578+0000 7fa102016700 1 --1- >> v1:172.21.15.118:6789/0 conn(0x558f84506650 0x558f845070f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.118:6789/0 says I am v1:172.21.15.167:33310/0 (socket says 172.21.15.167:33310) 2023-10-05T19:29:34.585 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.578+0000 7fa102016700 1 -- 172.21.15.167:0/975570866 learned_addr learned my addr 172.21.15.167:0/975570866 (peer_addr_for_me v1:172.21.15.167:0/0) 2023-10-05T19:29:34.585 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.578+0000 7fa102817700 1 --2- >> v2:172.21.15.118:3300/0 conn(0x558f84505d40 0x558f84506110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-05T19:29:34.585 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.582+0000 7fa101815700 1 -- 172.21.15.167:0/975570866 <== mon.0 v1:172.21.15.118:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3902305569 0 0) 0x558f84453bd0 con 0x558f84506650 2023-10-05T19:29:34.585 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.582+0000 7fa101815700 1 -- 172.21.15.167:0/975570866 --> v1:172.21.15.118:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa0f0001e10 con 0x558f84506650 2023-10-05T19:29:34.585 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.582+0000 7fa102817700 1 -- 172.21.15.167:0/975570866 >> v1:172.21.15.118:6789/0 conn(0x558f84506650 legacy=0x558f845070f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-05T19:29:34.585 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.582+0000 7fa102817700 1 -- 172.21.15.167:0/975570866 --> v2:172.21.15.118:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x558f84507800 con 0x558f84505d40 2023-10-05T19:29:34.586 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.582+0000 7fa102817700 1 --2- 172.21.15.167:0/975570866 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x558f84505d40 0x558f84506110 secure :-1 s=READY pgs=615 cs=0 l=1 rev1=1 crypto rx=0x7fa0ec00a5d0 tx=0x7fa0ec005a50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=856c3a29730e60e6 server_cookie=0 in_seq=0 out_seq=0 2023-10-05T19:29:34.586 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.582+0000 7fa101815700 1 -- 172.21.15.167:0/975570866 <== mon.0 v2:172.21.15.118:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fa0ec007610 con 0x558f84505d40 2023-10-05T19:29:34.586 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.582+0000 7fa101815700 1 -- 172.21.15.167:0/975570866 <== mon.0 v2:172.21.15.118:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa0ec007770 con 0x558f84505d40 2023-10-05T19:29:34.586 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.582+0000 7fa101815700 1 -- 172.21.15.167:0/975570866 <== mon.0 v2:172.21.15.118:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fa0ec007a40 con 0x558f84505d40 2023-10-05T19:29:34.587 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.582+0000 7fa104ce2e40 1 -- 172.21.15.167:0/975570866 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x558f84505d40 msgr2=0x558f84506110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-05T19:29:34.587 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.582+0000 7fa104ce2e40 1 --2- 172.21.15.167:0/975570866 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x558f84505d40 0x558f84506110 secure :-1 s=READY pgs=615 cs=0 l=1 rev1=1 crypto rx=0x7fa0ec00a5d0 tx=0x7fa0ec005a50 comp rx=0 tx=0).stop 2023-10-05T19:29:34.587 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.582+0000 7fa104ce2e40 1 -- 172.21.15.167:0/975570866 shutdown_connections 2023-10-05T19:29:34.587 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.582+0000 7fa104ce2e40 1 --2- 172.21.15.167:0/975570866 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x558f84505d40 0x558f84506110 unknown :-1 s=CLOSED pgs=615 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-05T19:29:34.587 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.582+0000 7fa104ce2e40 1 -- 172.21.15.167:0/975570866 >> 172.21.15.167:0/975570866 conn(0x558f84501760 msgr2=0x558f84503b70 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-05T19:29:34.588 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.582+0000 7fa104ce2e40 1 -- 172.21.15.167:0/975570866 shutdown_connections 2023-10-05T19:29:34.588 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.582+0000 7fa104ce2e40 1 -- 172.21.15.167:0/975570866 wait complete. 2023-10-05T19:29:34.588 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.582+0000 7fa104ce2e40 1 Processor -- start 2023-10-05T19:29:34.588 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.582+0000 7fa104ce2e40 1 -- start start 2023-10-05T19:29:34.589 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.582+0000 7fa104ce2e40 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x558f84505d40 0x558f8452a9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-05T19:29:34.589 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.582+0000 7fa104ce2e40 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x558f843d4e30 con 0x558f84505d40 2023-10-05T19:29:34.589 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.582+0000 7fa102817700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x558f84505d40 0x558f8452a9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-05T19:29:34.589 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.582+0000 7fa102817700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x558f84505d40 0x558f8452a9d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.118:3300/0 says I am v2:172.21.15.167:56716/0 (socket says 172.21.15.167:56716) 2023-10-05T19:29:34.589 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.582+0000 7fa102817700 1 -- 172.21.15.167:0/4234761229 learned_addr learned my addr 172.21.15.167:0/4234761229 (peer_addr_for_me v2:172.21.15.167:0/0) 2023-10-05T19:29:34.590 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.586+0000 7fa102817700 1 -- 172.21.15.167:0/4234761229 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa0ec005cf0 con 0x558f84505d40 2023-10-05T19:29:34.590 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.586+0000 7fa102817700 1 --2- 172.21.15.167:0/4234761229 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x558f84505d40 0x558f8452a9d0 secure :-1 s=READY pgs=616 cs=0 l=1 rev1=1 crypto rx=0x7fa0ec000c00 tx=0x7fa0ec014000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7241c2c42dae5619 server_cookie=0 in_seq=0 out_seq=0 2023-10-05T19:29:34.590 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.586+0000 7fa100813700 1 -- 172.21.15.167:0/4234761229 <== mon.0 v2:172.21.15.118:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fa0ec007a00 con 0x558f84505d40 2023-10-05T19:29:34.590 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.586+0000 7fa100813700 1 -- 172.21.15.167:0/4234761229 <== mon.0 v2:172.21.15.118:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa0ec007b60 con 0x558f84505d40 2023-10-05T19:29:34.590 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.586+0000 7fa100813700 1 -- 172.21.15.167:0/4234761229 <== mon.0 v2:172.21.15.118:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fa0ec007d10 con 0x558f84505d40 2023-10-05T19:29:34.591 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.586+0000 7fa104ce2e40 1 -- 172.21.15.167:0/4234761229 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x558f84505d40 msgr2=0x558f8452a9d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-05T19:29:34.591 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.586+0000 7fa104ce2e40 1 --2- 172.21.15.167:0/4234761229 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x558f84505d40 0x558f8452a9d0 secure :-1 s=READY pgs=616 cs=0 l=1 rev1=1 crypto rx=0x7fa0ec000c00 tx=0x7fa0ec014000 comp rx=0 tx=0).stop 2023-10-05T19:29:34.592 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.586+0000 7fa104ce2e40 1 -- 172.21.15.167:0/4234761229 shutdown_connections 2023-10-05T19:29:34.592 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.586+0000 7fa104ce2e40 1 --2- 172.21.15.167:0/4234761229 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x558f84505d40 0x558f8452a9d0 unknown :-1 s=CLOSED pgs=616 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-05T19:29:34.592 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.586+0000 7fa104ce2e40 1 -- 172.21.15.167:0/4234761229 >> 172.21.15.167:0/4234761229 conn(0x558f84501760 msgr2=0x558f8447ae40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-05T19:29:34.592 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.586+0000 7fa104ce2e40 1 -- 172.21.15.167:0/4234761229 shutdown_connections 2023-10-05T19:29:34.592 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.586+0000 7fa104ce2e40 1 -- 172.21.15.167:0/4234761229 wait complete. 2023-10-05T19:29:34.592 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.586+0000 7fa104ce2e40 1 Processor -- start 2023-10-05T19:29:34.592 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.586+0000 7fa104ce2e40 1 -- start start 2023-10-05T19:29:34.592 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.586+0000 7fa104ce2e40 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x558f84506650 0x558f8452ee70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-05T19:29:34.592 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.586+0000 7fa104ce2e40 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x558f843a7ed0 con 0x558f84506650 2023-10-05T19:29:34.592 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.586+0000 7fa102817700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x558f84506650 0x558f8452ee70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-05T19:29:34.592 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.586+0000 7fa102817700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x558f84506650 0x558f8452ee70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.118:3300/0 says I am v2:172.21.15.167:56722/0 (socket says 172.21.15.167:56722) 2023-10-05T19:29:34.592 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.586+0000 7fa102817700 1 -- 172.21.15.167:0/4173470315 learned_addr learned my addr 172.21.15.167:0/4173470315 (peer_addr_for_me v2:172.21.15.167:0/0) 2023-10-05T19:29:34.593 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.586+0000 7fa102817700 1 -- 172.21.15.167:0/4173470315 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x558f84507800 con 0x558f84506650 2023-10-05T19:29:34.593 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.590+0000 7fa102817700 1 --2- 172.21.15.167:0/4173470315 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x558f84506650 0x558f8452ee70 secure :-1 s=READY pgs=617 cs=0 l=1 rev1=1 crypto rx=0x7fa0ec00afd0 tx=0x7fa0ec01c000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-05T19:29:34.593 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.590+0000 7fa0faffd700 1 -- 172.21.15.167:0/4173470315 <== mon.0 v2:172.21.15.118:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fa0ec0079b0 con 0x558f84506650 2023-10-05T19:29:34.594 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.590+0000 7fa104ce2e40 1 -- 172.21.15.167:0/4173470315 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x558f8452f3b0 con 0x558f84506650 2023-10-05T19:29:34.594 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.590+0000 7fa0faffd700 1 -- 172.21.15.167:0/4173470315 <== mon.0 v2:172.21.15.118:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa0ec029070 con 0x558f84506650 2023-10-05T19:29:34.594 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.590+0000 7fa0faffd700 1 -- 172.21.15.167:0/4173470315 <== mon.0 v2:172.21.15.118:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fa0ec02f400 con 0x558f84506650 2023-10-05T19:29:34.594 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.590+0000 7fa104ce2e40 1 -- 172.21.15.167:0/4173470315 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x558f8452f790 con 0x558f84506650 2023-10-05T19:29:34.594 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.590+0000 7fa104ce2e40 20 rados->read ofs=0 len=0 2023-10-05T19:29:34.595 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.590+0000 7fa0faffd700 1 -- 172.21.15.167:0/4173470315 <== mon.0 v2:172.21.15.118:3300/0 4 ==== mgrmap(e 5) v1 ==== 40329+0+0 (secure 0 0 0) 0x7fa0ec012400 con 0x558f84506650 2023-10-05T19:29:34.595 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.590+0000 7fa0faffd700 1 --2- 172.21.15.167:0/4173470315 >> [v2:172.21.15.167:6824/29889,v1:172.21.15.167:6825/29889] conn(0x7fa0e4030ca0 0x7fa0e4033150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-05T19:29:34.596 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.590+0000 7fa0faffd700 1 -- 172.21.15.167:0/4173470315 <== mon.0 v2:172.21.15.118:3300/0 5 ==== osd_map(32..32 src has 1..32) v4 ==== 6260+0+0 (secure 0 0 0) 0x7fa0ec046380 con 0x558f84506650 2023-10-05T19:29:34.596 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.590+0000 7fa102016700 1 --2- 172.21.15.167:0/4173470315 >> [v2:172.21.15.167:6824/29889,v1:172.21.15.167:6825/29889] conn(0x7fa0e4030ca0 0x7fa0e4033150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-05T19:29:34.596 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.590+0000 7fa104ce2e40 1 --2- 172.21.15.167:0/4173470315 >> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] conn(0x558f84538920 0x558f8453ad00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-05T19:29:34.596 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.590+0000 7fa104ce2e40 1 -- 172.21.15.167:0/4173470315 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:1 5.12 5:49953fa1:::default.realm:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32) v8 -- 0x558f8453b370 con 0x558f84538920 2023-10-05T19:29:34.596 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.590+0000 7fa102016700 1 --2- 172.21.15.167:0/4173470315 >> [v2:172.21.15.167:6824/29889,v1:172.21.15.167:6825/29889] conn(0x7fa0e4030ca0 0x7fa0e4033150 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7fa0fc0044f0 tx=0x7fa0fc006690 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-05T19:29:34.597 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.590+0000 7fa103018700 1 --2- 172.21.15.167:0/4173470315 >> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] conn(0x558f84538920 0x558f8453ad00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-05T19:29:34.597 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.594+0000 7fa103018700 1 --2- 172.21.15.167:0/4173470315 >> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] conn(0x558f84538920 0x558f8453ad00 crc :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-05T19:29:34.598 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.594+0000 7fa103018700 1 -- 172.21.15.167:0/4173470315 <== osd.1 v2:172.21.15.167:6800/30035 1 ==== osd_op_reply(1 default.realm [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 ==== 157+0+0 (crc 0 0 0) 0x7fa0f400a500 con 0x558f84538920 2023-10-05T19:29:34.598 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.594+0000 7fa104ce2e40 20 rados_obj.operate() r=-2 bl.length=0 2023-10-05T19:29:34.598 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.594+0000 7fa104ce2e40 20 realm 2023-10-05T19:29:34.598 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.594+0000 7fa104ce2e40 20 rados->read ofs=0 len=0 2023-10-05T19:29:34.598 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.594+0000 7fa104ce2e40 1 -- 172.21.15.167:0/4173470315 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:2 5.12 5:49953fa1:::default.realm:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32) v8 -- 0x558f84537f90 con 0x558f84538920 2023-10-05T19:29:34.599 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.594+0000 7fa103018700 1 -- 172.21.15.167:0/4173470315 <== osd.1 v2:172.21.15.167:6800/30035 2 ==== osd_op_reply(2 default.realm [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 ==== 157+0+0 (crc 0 0 0) 0x7fa0f400a500 con 0x558f84538920 2023-10-05T19:29:34.599 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.594+0000 7fa104ce2e40 20 rados_obj.operate() r=-2 bl.length=0 2023-10-05T19:29:34.599 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.594+0000 7fa104ce2e40 4 RGWPeriod::init failed to init realm id : (2) No such file or directory 2023-10-05T19:29:34.599 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.594+0000 7fa104ce2e40 20 rados->read ofs=0 len=0 2023-10-05T19:29:34.600 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.594+0000 7fa104ce2e40 1 -- 172.21.15.167:0/4173470315 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:3 5.12 5:49953fa1:::default.realm:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32) v8 -- 0x558f8453be50 con 0x558f84538920 2023-10-05T19:29:34.600 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.594+0000 7fa103018700 1 -- 172.21.15.167:0/4173470315 <== osd.1 v2:172.21.15.167:6800/30035 3 ==== osd_op_reply(3 default.realm [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 ==== 157+0+0 (crc 0 0 0) 0x7fa0f400a500 con 0x558f84538920 2023-10-05T19:29:34.600 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.594+0000 7fa104ce2e40 20 rados_obj.operate() r=-2 bl.length=0 2023-10-05T19:29:34.600 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.594+0000 7fa104ce2e40 20 rados->read ofs=0 len=0 2023-10-05T19:29:34.600 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.594+0000 7fa104ce2e40 1 -- 172.21.15.167:0/4173470315 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:4 5.f 5:f43b6ece:::zone_names.default:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32) v8 -- 0x558f8453ba60 con 0x558f84538920 2023-10-05T19:29:34.600 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.594+0000 7fa103018700 1 -- 172.21.15.167:0/4173470315 <== osd.1 v2:172.21.15.167:6800/30035 4 ==== osd_op_reply(4 zone_names.default [read 0~46 out=46b] v0'0 uv1 ondisk = 0) v8 ==== 162+0+46 (crc 0 0 0) 0x7fa0f400a500 con 0x558f84538920 2023-10-05T19:29:34.600 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.594+0000 7fa104ce2e40 20 rados_obj.operate() r=0 bl.length=46 2023-10-05T19:29:34.600 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.594+0000 7fa104ce2e40 20 rados->read ofs=0 len=0 2023-10-05T19:29:34.600 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.594+0000 7fa104ce2e40 1 -- 172.21.15.167:0/4173470315 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:5 5.1d 5:b8374eb4:::zone_info.9a9df965-6612-45d4-a45f-6e3c889bacc2:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32) v8 -- 0x558f8453cce0 con 0x558f84538920 2023-10-05T19:29:34.601 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.594+0000 7fa103018700 1 -- 172.21.15.167:0/4173470315 <== osd.1 v2:172.21.15.167:6800/30035 5 ==== osd_op_reply(5 zone_info.9a9df965-6612-45d4-a45f-6e3c889bacc2 [read 0~874 out=874b] v0'0 uv2 ondisk = 0) v8 ==== 190+0+874 (crc 0 0 0) 0x7fa0f400a500 con 0x558f84538920 2023-10-05T19:29:34.601 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.594+0000 7fa104ce2e40 20 rados_obj.operate() r=0 bl.length=874 2023-10-05T19:29:34.601 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.594+0000 7fa104ce2e40 20 searching for the correct realm 2023-10-05T19:29:34.601 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.594+0000 7fa104ce2e40 1 --2- 172.21.15.167:0/4173470315 >> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] conn(0x558f8453eaa0 0x558f84540e80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-05T19:29:34.601 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.594+0000 7fa104ce2e40 1 -- 172.21.15.167:0/4173470315 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:6 5.0 5:00000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x558f845413c0 con 0x558f8453eaa0 2023-10-05T19:29:34.602 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.598+0000 7fa102817700 1 --2- 172.21.15.167:0/4173470315 >> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] conn(0x558f8453eaa0 0x558f84540e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-05T19:29:34.602 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.598+0000 7fa102817700 1 --2- 172.21.15.167:0/4173470315 >> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] conn(0x558f8453eaa0 0x558f84540e80 crc :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-05T19:29:34.602 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.598+0000 7fa102817700 1 -- 172.21.15.167:0/4173470315 <== osd.3 v2:172.21.15.167:6808/30036 1 ==== osd_op_reply(6 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fa0ec035070 con 0x558f8453eaa0 2023-10-05T19:29:34.602 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.598+0000 7fa102817700 1 --2- 172.21.15.167:0/4173470315 >> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] conn(0x558f84505d40 0x7fa0ec01ca90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-05T19:29:34.603 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.598+0000 7fa102817700 1 -- 172.21.15.167:0/4173470315 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:7 5.10 5:08000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x558f845413c0 con 0x558f84505d40 2023-10-05T19:29:34.603 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.598+0000 7fa102016700 1 --2- 172.21.15.167:0/4173470315 >> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] conn(0x558f84505d40 0x7fa0ec01ca90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-05T19:29:34.603 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.598+0000 7fa102016700 1 --2- 172.21.15.167:0/4173470315 >> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] conn(0x558f84505d40 0x7fa0ec01ca90 crc :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-05T19:29:34.604 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.598+0000 7fa102016700 1 -- 172.21.15.167:0/4173470315 <== osd.2 v2:172.21.15.167:6816/30037 1 ==== osd_op_reply(7 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fa0fc004520 con 0x558f84505d40 2023-10-05T19:29:34.604 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.598+0000 7fa102016700 1 --2- 172.21.15.167:0/4173470315 >> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] conn(0x7fa0fc011040 0x7fa0fc014950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-05T19:29:34.604 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.598+0000 7fa102016700 1 -- 172.21.15.167:0/4173470315 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:8 5.8 5:10000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x558f845413c0 con 0x7fa0fc011040 2023-10-05T19:29:34.604 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.598+0000 7fa103018700 1 --2- 172.21.15.167:0/4173470315 >> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] conn(0x7fa0fc011040 0x7fa0fc014950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-05T19:29:34.605 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.602+0000 7fa103018700 1 --2- 172.21.15.167:0/4173470315 >> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] conn(0x7fa0fc011040 0x7fa0fc014950 crc :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-05T19:29:34.605 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.602+0000 7fa103018700 1 -- 172.21.15.167:0/4173470315 <== osd.0 v2:172.21.15.118:6800/27690 1 ==== osd_op_reply(8 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fa0f4014040 con 0x7fa0fc011040 2023-10-05T19:29:34.606 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.602+0000 7fa103018700 1 -- 172.21.15.167:0/4173470315 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:9 5.18 5:18000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x558f845413c0 con 0x558f84538920 2023-10-05T19:29:34.606 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.602+0000 7fa103018700 1 -- 172.21.15.167:0/4173470315 <== osd.1 v2:172.21.15.167:6800/30035 6 ==== osd_op_reply(9 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fa0f4014040 con 0x558f84538920 2023-10-05T19:29:34.606 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.602+0000 7fa103018700 1 -- 172.21.15.167:0/4173470315 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:10 5.4 5:20000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x558f845413c0 con 0x558f84538920 2023-10-05T19:29:34.607 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.602+0000 7fa103018700 1 -- 172.21.15.167:0/4173470315 <== osd.1 v2:172.21.15.167:6800/30035 7 ==== osd_op_reply(10 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fa0f4014040 con 0x558f84538920 2023-10-05T19:29:34.607 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.602+0000 7fa103018700 1 -- 172.21.15.167:0/4173470315 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:11 5.14 5:28000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x558f845413c0 con 0x7fa0fc011040 2023-10-05T19:29:34.608 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.602+0000 7fa103018700 1 -- 172.21.15.167:0/4173470315 <== osd.0 v2:172.21.15.118:6800/27690 2 ==== osd_op_reply(11 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fa0f4014040 con 0x7fa0fc011040 2023-10-05T19:29:34.608 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.602+0000 7fa103018700 1 -- 172.21.15.167:0/4173470315 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:12 5.c 5:30000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x558f845413c0 con 0x7fa0fc011040 2023-10-05T19:29:34.608 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.606+0000 7fa103018700 1 -- 172.21.15.167:0/4173470315 <== osd.0 v2:172.21.15.118:6800/27690 3 ==== osd_op_reply(12 [pgnls start_epoch 32 out=112b] v23'1 uv1 ondisk = 1) v8 ==== 144+0+112 (crc 0 0 0) 0x7fa0f4014040 con 0x7fa0fc011040 2023-10-05T19:29:34.608 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.606+0000 7fa103018700 1 -- 172.21.15.167:0/4173470315 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:13 5.1c 5:38000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x558f845413c0 con 0x558f84505d40 2023-10-05T19:29:34.609 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.606+0000 7fa102016700 1 -- 172.21.15.167:0/4173470315 <== osd.2 v2:172.21.15.167:6816/30037 2 ==== osd_op_reply(13 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fa0fc004520 con 0x558f84505d40 2023-10-05T19:29:34.609 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.606+0000 7fa102016700 1 -- 172.21.15.167:0/4173470315 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:14 5.2 5:40000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x558f845413c0 con 0x558f8453eaa0 2023-10-05T19:29:34.610 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.606+0000 7fa102817700 1 -- 172.21.15.167:0/4173470315 <== osd.3 v2:172.21.15.167:6808/30036 2 ==== osd_op_reply(14 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fa0ec035070 con 0x558f8453eaa0 2023-10-05T19:29:34.610 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.606+0000 7fa102817700 1 -- 172.21.15.167:0/4173470315 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:15 5.12 5:48000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x558f845413c0 con 0x558f84538920 2023-10-05T19:29:34.611 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.606+0000 7fa103018700 1 -- 172.21.15.167:0/4173470315 <== osd.1 v2:172.21.15.167:6800/30035 8 ==== osd_op_reply(15 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fa0f4014040 con 0x558f84538920 2023-10-05T19:29:34.611 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.606+0000 7fa103018700 1 -- 172.21.15.167:0/4173470315 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:16 5.a 5:50000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x558f845413c0 con 0x7fa0fc011040 2023-10-05T19:29:34.611 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.606+0000 7fa103018700 1 -- 172.21.15.167:0/4173470315 <== osd.0 v2:172.21.15.118:6800/27690 4 ==== osd_op_reply(16 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fa0f4014040 con 0x7fa0fc011040 2023-10-05T19:29:34.612 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.606+0000 7fa103018700 1 -- 172.21.15.167:0/4173470315 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:17 5.1a 5:58000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x558f845413c0 con 0x558f84538920 2023-10-05T19:29:34.612 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.606+0000 7fa103018700 1 -- 172.21.15.167:0/4173470315 <== osd.1 v2:172.21.15.167:6800/30035 9 ==== osd_op_reply(17 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fa0f4014040 con 0x558f84538920 2023-10-05T19:29:34.612 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.606+0000 7fa103018700 1 -- 172.21.15.167:0/4173470315 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:18 5.6 5:60000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x558f845413c0 con 0x558f84505d40 2023-10-05T19:29:34.613 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.610+0000 7fa102016700 1 -- 172.21.15.167:0/4173470315 <== osd.2 v2:172.21.15.167:6816/30037 3 ==== osd_op_reply(18 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fa0fc004520 con 0x558f84505d40 2023-10-05T19:29:34.613 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.610+0000 7fa102016700 1 -- 172.21.15.167:0/4173470315 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:19 5.16 5:68000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x558f845413c0 con 0x558f8453eaa0 2023-10-05T19:29:34.614 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.610+0000 7fa102817700 1 -- 172.21.15.167:0/4173470315 <== osd.3 v2:172.21.15.167:6808/30036 3 ==== osd_op_reply(19 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fa0ec035070 con 0x558f8453eaa0 2023-10-05T19:29:34.614 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.610+0000 7fa102817700 1 -- 172.21.15.167:0/4173470315 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:20 5.e 5:70000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x558f845413c0 con 0x558f8453eaa0 2023-10-05T19:29:34.614 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.610+0000 7fa102817700 1 -- 172.21.15.167:0/4173470315 <== osd.3 v2:172.21.15.167:6808/30036 4 ==== osd_op_reply(20 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fa0ec035070 con 0x558f8453eaa0 2023-10-05T19:29:34.614 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.610+0000 7fa102817700 1 -- 172.21.15.167:0/4173470315 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:21 5.1e 5:78000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x558f845413c0 con 0x7fa0fc011040 2023-10-05T19:29:34.615 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.610+0000 7fa103018700 1 -- 172.21.15.167:0/4173470315 <== osd.0 v2:172.21.15.118:6800/27690 5 ==== osd_op_reply(21 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fa0f4022040 con 0x7fa0fc011040 2023-10-05T19:29:34.615 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.610+0000 7fa103018700 1 -- 172.21.15.167:0/4173470315 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:22 5.1 5:80000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x558f845413c0 con 0x558f8453eaa0 2023-10-05T19:29:34.616 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.610+0000 7fa102817700 1 -- 172.21.15.167:0/4173470315 <== osd.3 v2:172.21.15.167:6808/30036 5 ==== osd_op_reply(22 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fa0ec035070 con 0x558f8453eaa0 2023-10-05T19:29:34.616 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.610+0000 7fa102817700 1 -- 172.21.15.167:0/4173470315 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:23 5.11 5:88000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x558f845413c0 con 0x558f8453eaa0 2023-10-05T19:29:34.617 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.614+0000 7fa102817700 1 -- 172.21.15.167:0/4173470315 <== osd.3 v2:172.21.15.167:6808/30036 6 ==== osd_op_reply(23 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fa0ec035070 con 0x558f8453eaa0 2023-10-05T19:29:34.617 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.614+0000 7fa102817700 1 -- 172.21.15.167:0/4173470315 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:24 5.9 5:90000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x558f845413c0 con 0x558f84538920 2023-10-05T19:29:34.617 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.614+0000 7fa103018700 1 -- 172.21.15.167:0/4173470315 <== osd.1 v2:172.21.15.167:6800/30035 10 ==== osd_op_reply(24 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fa0f4022040 con 0x558f84538920 2023-10-05T19:29:34.617 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.614+0000 7fa103018700 1 -- 172.21.15.167:0/4173470315 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:25 5.19 5:98000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x558f845413c0 con 0x558f84538920 2023-10-05T19:29:34.618 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.614+0000 7fa103018700 1 -- 172.21.15.167:0/4173470315 <== osd.1 v2:172.21.15.167:6800/30035 11 ==== osd_op_reply(25 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fa0f4022040 con 0x558f84538920 2023-10-05T19:29:34.618 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.614+0000 7fa103018700 1 -- 172.21.15.167:0/4173470315 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:26 5.5 5:a0000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x558f845413c0 con 0x7fa0fc011040 2023-10-05T19:29:34.619 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.614+0000 7fa103018700 1 -- 172.21.15.167:0/4173470315 <== osd.0 v2:172.21.15.118:6800/27690 6 ==== osd_op_reply(26 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fa0f4022040 con 0x7fa0fc011040 2023-10-05T19:29:34.619 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.614+0000 7fa103018700 1 -- 172.21.15.167:0/4173470315 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:27 5.15 5:a8000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x558f845413c0 con 0x558f8453eaa0 2023-10-05T19:29:34.620 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.614+0000 7fa102817700 1 -- 172.21.15.167:0/4173470315 <== osd.3 v2:172.21.15.167:6808/30036 7 ==== osd_op_reply(27 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fa0ec035070 con 0x558f8453eaa0 2023-10-05T19:29:34.620 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.614+0000 7fa102817700 1 -- 172.21.15.167:0/4173470315 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:28 5.d 5:b0000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x558f845413c0 con 0x558f84505d40 2023-10-05T19:29:34.621 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.618+0000 7fa102016700 1 -- 172.21.15.167:0/4173470315 <== osd.2 v2:172.21.15.167:6816/30037 4 ==== osd_op_reply(28 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fa0fc004520 con 0x558f84505d40 2023-10-05T19:29:34.621 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.618+0000 7fa102016700 1 -- 172.21.15.167:0/4173470315 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:29 5.1d 5:b8000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x558f845413c0 con 0x558f84538920 2023-10-05T19:29:34.622 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.618+0000 7fa103018700 1 -- 172.21.15.167:0/4173470315 <== osd.1 v2:172.21.15.167:6800/30035 12 ==== osd_op_reply(29 [pgnls start_epoch 32 out=107b] v30'2 uv2 ondisk = 1) v8 ==== 144+0+107 (crc 0 0 0) 0x7fa0f4022040 con 0x558f84538920 2023-10-05T19:29:34.622 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.618+0000 7fa103018700 1 -- 172.21.15.167:0/4173470315 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:30 5.3 5:c0000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x558f845413c0 con 0x7fa0fc011040 2023-10-05T19:29:34.623 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.618+0000 7fa103018700 1 -- 172.21.15.167:0/4173470315 <== osd.0 v2:172.21.15.118:6800/27690 7 ==== osd_op_reply(30 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fa0f4022040 con 0x7fa0fc011040 2023-10-05T19:29:34.623 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.618+0000 7fa103018700 1 -- 172.21.15.167:0/4173470315 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:31 5.13 5:c8000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x558f845413c0 con 0x558f8453eaa0 2023-10-05T19:29:34.623 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.618+0000 7fa102817700 1 -- 172.21.15.167:0/4173470315 <== osd.3 v2:172.21.15.167:6808/30036 8 ==== osd_op_reply(31 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fa0ec035070 con 0x558f8453eaa0 2023-10-05T19:29:34.624 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.618+0000 7fa102817700 1 -- 172.21.15.167:0/4173470315 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:32 5.b 5:d0000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x558f845413c0 con 0x7fa0fc011040 2023-10-05T19:29:34.624 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.622+0000 7fa103018700 1 -- 172.21.15.167:0/4173470315 <== osd.0 v2:172.21.15.118:6800/27690 8 ==== osd_op_reply(32 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fa0f4022040 con 0x7fa0fc011040 2023-10-05T19:29:34.625 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.622+0000 7fa103018700 1 -- 172.21.15.167:0/4173470315 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:33 5.1b 5:d8000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x558f845413c0 con 0x558f84505d40 2023-10-05T19:29:34.625 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.622+0000 7fa102016700 1 -- 172.21.15.167:0/4173470315 <== osd.2 v2:172.21.15.167:6816/30037 5 ==== osd_op_reply(33 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fa0fc004520 con 0x558f84505d40 2023-10-05T19:29:34.625 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.622+0000 7fa102016700 1 -- 172.21.15.167:0/4173470315 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:34 5.7 5:e0000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x558f845413c0 con 0x558f8453eaa0 2023-10-05T19:29:34.626 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.622+0000 7fa102817700 1 -- 172.21.15.167:0/4173470315 <== osd.3 v2:172.21.15.167:6808/30036 9 ==== osd_op_reply(34 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fa0ec035070 con 0x558f8453eaa0 2023-10-05T19:29:34.626 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.622+0000 7fa102817700 1 -- 172.21.15.167:0/4173470315 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:35 5.17 5:e8000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x558f845413c0 con 0x7fa0fc011040 2023-10-05T19:29:34.627 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.622+0000 7fa103018700 1 -- 172.21.15.167:0/4173470315 <== osd.0 v2:172.21.15.118:6800/27690 9 ==== osd_op_reply(35 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fa0f4022040 con 0x7fa0fc011040 2023-10-05T19:29:34.627 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.622+0000 7fa103018700 1 -- 172.21.15.167:0/4173470315 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:36 5.f 5:f0000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x558f845413c0 con 0x558f84538920 2023-10-05T19:29:34.628 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.622+0000 7fa103018700 1 -- 172.21.15.167:0/4173470315 <== osd.1 v2:172.21.15.167:6800/30035 13 ==== osd_op_reply(36 [pgnls start_epoch 32 out=115b] v23'2 uv2 ondisk = 1) v8 ==== 144+0+115 (crc 0 0 0) 0x7fa0f4022040 con 0x558f84538920 2023-10-05T19:29:34.628 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.622+0000 7fa103018700 1 -- 172.21.15.167:0/4173470315 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:37 5.1f 5:f8000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x558f845413c0 con 0x558f84505d40 2023-10-05T19:29:34.629 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.626+0000 7fa102016700 1 -- 172.21.15.167:0/4173470315 <== osd.2 v2:172.21.15.167:6816/30037 6 ==== osd_op_reply(37 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fa0fc004520 con 0x558f84505d40 2023-10-05T19:29:34.629 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.626+0000 7fa104ce2e40 20 RGWRados::pool_iterate: got zonegroup_info.4f1f719d-9648-417b-b10c-e5d939825419 2023-10-05T19:29:34.629 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.626+0000 7fa104ce2e40 20 RGWRados::pool_iterate: got zone_info.9a9df965-6612-45d4-a45f-6e3c889bacc2 2023-10-05T19:29:34.629 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.626+0000 7fa104ce2e40 20 RGWRados::pool_iterate: got zone_names.default 2023-10-05T19:29:34.629 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.626+0000 7fa104ce2e40 20 RGWRados::pool_iterate: got zonegroups_names.default 2023-10-05T19:29:34.629 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.626+0000 7fa104ce2e40 20 rados->read ofs=0 len=0 2023-10-05T19:29:34.629 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.626+0000 7fa104ce2e40 1 -- 172.21.15.167:0/4173470315 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:38 5.12 5:49953fa1:::default.realm:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32) v8 -- 0x558f8453d850 con 0x558f84538920 2023-10-05T19:29:34.630 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.626+0000 7fa103018700 1 -- 172.21.15.167:0/4173470315 <== osd.1 v2:172.21.15.167:6800/30035 14 ==== osd_op_reply(38 default.realm [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 ==== 157+0+0 (crc 0 0 0) 0x7fa0f4022040 con 0x558f84538920 2023-10-05T19:29:34.630 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.626+0000 7fa104ce2e40 20 rados_obj.operate() r=-2 bl.length=0 2023-10-05T19:29:34.631 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.626+0000 7fa104ce2e40 20 rados->read ofs=0 len=0 2023-10-05T19:29:34.631 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.626+0000 7fa104ce2e40 1 -- 172.21.15.167:0/4173470315 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:39 5.f 5:f4c53578:::zonegroups_names.default:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32) v8 -- 0x558f8453e140 con 0x558f84538920 2023-10-05T19:29:34.631 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.626+0000 7fa103018700 1 -- 172.21.15.167:0/4173470315 <== osd.1 v2:172.21.15.167:6800/30035 15 ==== osd_op_reply(39 zonegroups_names.default [read 0~46 out=46b] v0'0 uv2 ondisk = 0) v8 ==== 168+0+46 (crc 0 0 0) 0x7fa0f4022040 con 0x558f84538920 2023-10-05T19:29:34.631 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.626+0000 7fa104ce2e40 20 rados_obj.operate() r=0 bl.length=46 2023-10-05T19:29:34.631 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.626+0000 7fa104ce2e40 20 rados->read ofs=0 len=0 2023-10-05T19:29:34.631 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.626+0000 7fa104ce2e40 1 -- 172.21.15.167:0/4173470315 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:40 5.c 5:30d299a5:::zonegroup_info.4f1f719d-9648-417b-b10c-e5d939825419:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32) v8 -- 0x558f84541800 con 0x7fa0fc011040 2023-10-05T19:29:34.631 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.626+0000 7fa103018700 1 -- 172.21.15.167:0/4173470315 <== osd.0 v2:172.21.15.118:6800/27690 10 ==== osd_op_reply(40 zonegroup_info.4f1f719d-9648-417b-b10c-e5d939825419 [read 0~362 out=362b] v0'0 uv1 ondisk = 0) v8 ==== 195+0+362 (crc 0 0 0) 0x7fa0f4022040 con 0x7fa0fc011040 2023-10-05T19:29:34.632 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.626+0000 7fa104ce2e40 20 rados_obj.operate() r=0 bl.length=362 2023-10-05T19:29:34.632 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.626+0000 7fa104ce2e40 20 rados->read ofs=0 len=0 2023-10-05T19:29:34.632 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.626+0000 7fa104ce2e40 1 -- 172.21.15.167:0/4173470315 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:41 5.13 5:cbbf6c60:::converted:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32) v8 -- 0x558f84541800 con 0x558f8453eaa0 2023-10-05T19:29:34.633 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.626+0000 7fa102817700 1 -- 172.21.15.167:0/4173470315 <== osd.3 v2:172.21.15.167:6808/30036 10 ==== osd_op_reply(41 converted [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 ==== 153+0+0 (crc 0 0 0) 0x7fa0ec035070 con 0x558f8453eaa0 2023-10-05T19:29:34.633 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.626+0000 7fa104ce2e40 20 rados_obj.operate() r=-2 bl.length=0 2023-10-05T19:29:34.633 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.626+0000 7fa104ce2e40 20 rados->read ofs=0 len=0 2023-10-05T19:29:34.633 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.626+0000 7fa104ce2e40 1 -- 172.21.15.167:0/4173470315 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:42 5.12 5:49953fa1:::default.realm:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32) v8 -- 0x558f84541bb0 con 0x558f84538920 2023-10-05T19:29:34.633 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.630+0000 7fa103018700 1 -- 172.21.15.167:0/4173470315 <== osd.1 v2:172.21.15.167:6800/30035 16 ==== osd_op_reply(42 default.realm [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 ==== 157+0+0 (crc 0 0 0) 0x7fa0f4022040 con 0x558f84538920 2023-10-05T19:29:34.633 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.630+0000 7fa104ce2e40 20 rados_obj.operate() r=-2 bl.length=0 2023-10-05T19:29:34.633 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.630+0000 7fa104ce2e40 20 rados->read ofs=0 len=0 2023-10-05T19:29:34.633 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.630+0000 7fa104ce2e40 1 -- 172.21.15.167:0/4173470315 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:43 5.f 5:f4c53578:::zonegroups_names.default:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32) v8 -- 0x558f84541fb0 con 0x558f84538920 2023-10-05T19:29:34.634 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.630+0000 7fa103018700 1 -- 172.21.15.167:0/4173470315 <== osd.1 v2:172.21.15.167:6800/30035 17 ==== osd_op_reply(43 zonegroups_names.default [read 0~46 out=46b] v0'0 uv2 ondisk = 0) v8 ==== 168+0+46 (crc 0 0 0) 0x7fa0f4022040 con 0x558f84538920 2023-10-05T19:29:34.634 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.630+0000 7fa104ce2e40 20 rados_obj.operate() r=0 bl.length=46 2023-10-05T19:29:34.634 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.630+0000 7fa104ce2e40 1 -- 172.21.15.167:0/4173470315 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:44 5.0 5:00000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x558f84542fc0 con 0x558f8453eaa0 2023-10-05T19:29:34.635 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.630+0000 7fa102817700 1 -- 172.21.15.167:0/4173470315 <== osd.3 v2:172.21.15.167:6808/30036 11 ==== osd_op_reply(44 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fa0ec035070 con 0x558f8453eaa0 2023-10-05T19:29:34.635 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.630+0000 7fa102817700 1 -- 172.21.15.167:0/4173470315 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:45 5.10 5:08000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x558f84542fc0 con 0x558f84505d40 2023-10-05T19:29:34.635 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.630+0000 7fa102016700 1 -- 172.21.15.167:0/4173470315 <== osd.2 v2:172.21.15.167:6816/30037 7 ==== osd_op_reply(45 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fa0fc004520 con 0x558f84505d40 2023-10-05T19:29:34.636 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.630+0000 7fa102016700 1 -- 172.21.15.167:0/4173470315 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:46 5.8 5:10000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x558f84542fc0 con 0x7fa0fc011040 2023-10-05T19:29:34.636 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.634+0000 7fa103018700 1 -- 172.21.15.167:0/4173470315 <== osd.0 v2:172.21.15.118:6800/27690 11 ==== osd_op_reply(46 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fa0f4022040 con 0x7fa0fc011040 2023-10-05T19:29:34.637 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.634+0000 7fa103018700 1 -- 172.21.15.167:0/4173470315 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:47 5.18 5:18000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x558f8453e140 con 0x558f84538920 2023-10-05T19:29:34.637 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.634+0000 7fa103018700 1 -- 172.21.15.167:0/4173470315 <== osd.1 v2:172.21.15.167:6800/30035 18 ==== osd_op_reply(47 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fa0f4022040 con 0x558f84538920 2023-10-05T19:29:34.637 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.634+0000 7fa103018700 1 -- 172.21.15.167:0/4173470315 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:48 5.4 5:20000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x558f8453e140 con 0x558f84538920 2023-10-05T19:29:34.638 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.634+0000 7fa103018700 1 -- 172.21.15.167:0/4173470315 <== osd.1 v2:172.21.15.167:6800/30035 19 ==== osd_op_reply(48 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fa0f4022040 con 0x558f84538920 2023-10-05T19:29:34.638 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.634+0000 7fa103018700 1 -- 172.21.15.167:0/4173470315 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:49 5.14 5:28000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x558f8453e140 con 0x7fa0fc011040 2023-10-05T19:29:34.639 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.634+0000 7fa103018700 1 -- 172.21.15.167:0/4173470315 <== osd.0 v2:172.21.15.118:6800/27690 12 ==== osd_op_reply(49 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fa0f4022040 con 0x7fa0fc011040 2023-10-05T19:29:34.639 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.634+0000 7fa103018700 1 -- 172.21.15.167:0/4173470315 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:50 5.c 5:30000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x558f8453e140 con 0x7fa0fc011040 2023-10-05T19:29:34.640 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.634+0000 7fa103018700 1 -- 172.21.15.167:0/4173470315 <== osd.0 v2:172.21.15.118:6800/27690 13 ==== osd_op_reply(50 [pgnls start_epoch 32 out=112b] v23'1 uv1 ondisk = 1) v8 ==== 144+0+112 (crc 0 0 0) 0x7fa0f4022040 con 0x7fa0fc011040 2023-10-05T19:29:34.640 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.634+0000 7fa103018700 1 -- 172.21.15.167:0/4173470315 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:51 5.1c 5:38000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x558f8453e140 con 0x558f84505d40 2023-10-05T19:29:34.641 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.638+0000 7fa102016700 1 -- 172.21.15.167:0/4173470315 <== osd.2 v2:172.21.15.167:6816/30037 8 ==== osd_op_reply(51 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fa0fc004520 con 0x558f84505d40 2023-10-05T19:29:34.641 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.638+0000 7fa102016700 1 -- 172.21.15.167:0/4173470315 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:52 5.2 5:40000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x558f8453e140 con 0x558f8453eaa0 2023-10-05T19:29:34.642 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.638+0000 7fa102817700 1 -- 172.21.15.167:0/4173470315 <== osd.3 v2:172.21.15.167:6808/30036 12 ==== osd_op_reply(52 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fa0ec035070 con 0x558f8453eaa0 2023-10-05T19:29:34.642 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.638+0000 7fa102817700 1 -- 172.21.15.167:0/4173470315 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:53 5.12 5:48000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x558f8453e140 con 0x558f84538920 2023-10-05T19:29:34.643 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.638+0000 7fa103018700 1 -- 172.21.15.167:0/4173470315 <== osd.1 v2:172.21.15.167:6800/30035 20 ==== osd_op_reply(53 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fa0f4022040 con 0x558f84538920 2023-10-05T19:29:34.643 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.638+0000 7fa103018700 1 -- 172.21.15.167:0/4173470315 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:54 5.a 5:50000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x558f8453e140 con 0x7fa0fc011040 2023-10-05T19:29:34.644 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.638+0000 7fa103018700 1 -- 172.21.15.167:0/4173470315 <== osd.0 v2:172.21.15.118:6800/27690 14 ==== osd_op_reply(54 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fa0f4022040 con 0x7fa0fc011040 2023-10-05T19:29:34.644 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.638+0000 7fa103018700 1 -- 172.21.15.167:0/4173470315 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:55 5.1a 5:58000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x558f8453e140 con 0x558f84538920 2023-10-05T19:29:34.645 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.642+0000 7fa103018700 1 -- 172.21.15.167:0/4173470315 <== osd.1 v2:172.21.15.167:6800/30035 21 ==== osd_op_reply(55 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fa0f4022040 con 0x558f84538920 2023-10-05T19:29:34.645 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.642+0000 7fa103018700 1 -- 172.21.15.167:0/4173470315 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:56 5.6 5:60000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x558f8453e140 con 0x558f84505d40 2023-10-05T19:29:34.646 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.642+0000 7fa102016700 1 -- 172.21.15.167:0/4173470315 <== osd.2 v2:172.21.15.167:6816/30037 9 ==== osd_op_reply(56 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fa0fc004520 con 0x558f84505d40 2023-10-05T19:29:34.646 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.642+0000 7fa102016700 1 -- 172.21.15.167:0/4173470315 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:57 5.16 5:68000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x558f8453e140 con 0x558f8453eaa0 2023-10-05T19:29:34.647 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.642+0000 7fa102817700 1 -- 172.21.15.167:0/4173470315 <== osd.3 v2:172.21.15.167:6808/30036 13 ==== osd_op_reply(57 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fa0ec035070 con 0x558f8453eaa0 2023-10-05T19:29:34.647 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.642+0000 7fa102817700 1 -- 172.21.15.167:0/4173470315 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:58 5.e 5:70000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x558f8453e140 con 0x558f8453eaa0 2023-10-05T19:29:34.648 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.642+0000 7fa102817700 1 -- 172.21.15.167:0/4173470315 <== osd.3 v2:172.21.15.167:6808/30036 14 ==== osd_op_reply(58 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fa0ec035070 con 0x558f8453eaa0 2023-10-05T19:29:34.648 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.642+0000 7fa102817700 1 -- 172.21.15.167:0/4173470315 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:59 5.1e 5:78000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x558f8453e140 con 0x7fa0fc011040 2023-10-05T19:29:34.649 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.646+0000 7fa103018700 1 -- 172.21.15.167:0/4173470315 <== osd.0 v2:172.21.15.118:6800/27690 15 ==== osd_op_reply(59 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fa0f4022040 con 0x7fa0fc011040 2023-10-05T19:29:34.649 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.646+0000 7fa103018700 1 -- 172.21.15.167:0/4173470315 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:60 5.1 5:80000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x558f8453e140 con 0x558f8453eaa0 2023-10-05T19:29:34.650 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.646+0000 7fa102817700 1 -- 172.21.15.167:0/4173470315 <== osd.3 v2:172.21.15.167:6808/30036 15 ==== osd_op_reply(60 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fa0ec035070 con 0x558f8453eaa0 2023-10-05T19:29:34.650 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.646+0000 7fa102817700 1 -- 172.21.15.167:0/4173470315 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:61 5.11 5:88000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x558f8453e140 con 0x558f8453eaa0 2023-10-05T19:29:34.650 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.646+0000 7fa102817700 1 -- 172.21.15.167:0/4173470315 <== osd.3 v2:172.21.15.167:6808/30036 16 ==== osd_op_reply(61 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fa0ec035070 con 0x558f8453eaa0 2023-10-05T19:29:34.650 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.646+0000 7fa102817700 1 -- 172.21.15.167:0/4173470315 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:62 5.9 5:90000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x558f8453e140 con 0x558f84538920 2023-10-05T19:29:34.651 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.646+0000 7fa103018700 1 -- 172.21.15.167:0/4173470315 <== osd.1 v2:172.21.15.167:6800/30035 22 ==== osd_op_reply(62 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fa0f4022040 con 0x558f84538920 2023-10-05T19:29:34.651 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.646+0000 7fa103018700 1 -- 172.21.15.167:0/4173470315 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:63 5.19 5:98000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x558f8453e140 con 0x558f84538920 2023-10-05T19:29:34.652 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.646+0000 7fa103018700 1 -- 172.21.15.167:0/4173470315 <== osd.1 v2:172.21.15.167:6800/30035 23 ==== osd_op_reply(63 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fa0f4022040 con 0x558f84538920 2023-10-05T19:29:34.652 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.646+0000 7fa103018700 1 -- 172.21.15.167:0/4173470315 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:64 5.5 5:a0000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x558f8453e140 con 0x7fa0fc011040 2023-10-05T19:29:34.653 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.650+0000 7fa103018700 1 -- 172.21.15.167:0/4173470315 <== osd.0 v2:172.21.15.118:6800/27690 16 ==== osd_op_reply(64 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fa0f4022040 con 0x7fa0fc011040 2023-10-05T19:29:34.653 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.650+0000 7fa103018700 1 -- 172.21.15.167:0/4173470315 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:65 5.15 5:a8000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x558f8453e140 con 0x558f8453eaa0 2023-10-05T19:29:34.654 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.650+0000 7fa102817700 1 -- 172.21.15.167:0/4173470315 <== osd.3 v2:172.21.15.167:6808/30036 17 ==== osd_op_reply(65 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fa0ec035070 con 0x558f8453eaa0 2023-10-05T19:29:34.654 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.650+0000 7fa102817700 1 -- 172.21.15.167:0/4173470315 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:66 5.d 5:b0000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x558f8453e140 con 0x558f84505d40 2023-10-05T19:29:34.655 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.650+0000 7fa102016700 1 -- 172.21.15.167:0/4173470315 <== osd.2 v2:172.21.15.167:6816/30037 10 ==== osd_op_reply(66 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fa0fc004520 con 0x558f84505d40 2023-10-05T19:29:34.655 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.650+0000 7fa102016700 1 -- 172.21.15.167:0/4173470315 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:67 5.1d 5:b8000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x558f8453e140 con 0x558f84538920 2023-10-05T19:29:34.656 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.650+0000 7fa103018700 1 -- 172.21.15.167:0/4173470315 <== osd.1 v2:172.21.15.167:6800/30035 24 ==== osd_op_reply(67 [pgnls start_epoch 32 out=107b] v30'2 uv2 ondisk = 1) v8 ==== 144+0+107 (crc 0 0 0) 0x7fa0f4022040 con 0x558f84538920 2023-10-05T19:29:34.656 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.650+0000 7fa103018700 1 -- 172.21.15.167:0/4173470315 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:68 5.3 5:c0000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x558f8453e140 con 0x7fa0fc011040 2023-10-05T19:29:34.657 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.654+0000 7fa103018700 1 -- 172.21.15.167:0/4173470315 <== osd.0 v2:172.21.15.118:6800/27690 17 ==== osd_op_reply(68 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fa0f4022040 con 0x7fa0fc011040 2023-10-05T19:29:34.657 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.654+0000 7fa103018700 1 -- 172.21.15.167:0/4173470315 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:69 5.13 5:c8000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x558f8453e140 con 0x558f8453eaa0 2023-10-05T19:29:34.658 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.654+0000 7fa102817700 1 -- 172.21.15.167:0/4173470315 <== osd.3 v2:172.21.15.167:6808/30036 18 ==== osd_op_reply(69 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fa0ec035070 con 0x558f8453eaa0 2023-10-05T19:29:34.658 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.654+0000 7fa102817700 1 -- 172.21.15.167:0/4173470315 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:70 5.b 5:d0000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x558f8453e140 con 0x7fa0fc011040 2023-10-05T19:29:34.659 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.654+0000 7fa103018700 1 -- 172.21.15.167:0/4173470315 <== osd.0 v2:172.21.15.118:6800/27690 18 ==== osd_op_reply(70 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fa0f4022040 con 0x7fa0fc011040 2023-10-05T19:29:34.659 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.654+0000 7fa103018700 1 -- 172.21.15.167:0/4173470315 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:71 5.1b 5:d8000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x558f8453e140 con 0x558f84505d40 2023-10-05T19:29:34.660 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.654+0000 7fa102016700 1 -- 172.21.15.167:0/4173470315 <== osd.2 v2:172.21.15.167:6816/30037 11 ==== osd_op_reply(71 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fa0fc004520 con 0x558f84505d40 2023-10-05T19:29:34.660 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.654+0000 7fa102016700 1 -- 172.21.15.167:0/4173470315 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:72 5.7 5:e0000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x558f8453e140 con 0x558f8453eaa0 2023-10-05T19:29:34.661 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.658+0000 7fa102817700 1 -- 172.21.15.167:0/4173470315 <== osd.3 v2:172.21.15.167:6808/30036 19 ==== osd_op_reply(72 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fa0ec035070 con 0x558f8453eaa0 2023-10-05T19:29:34.661 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.658+0000 7fa102817700 1 -- 172.21.15.167:0/4173470315 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:73 5.17 5:e8000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x558f8453e140 con 0x7fa0fc011040 2023-10-05T19:29:34.662 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.658+0000 7fa103018700 1 -- 172.21.15.167:0/4173470315 <== osd.0 v2:172.21.15.118:6800/27690 19 ==== osd_op_reply(73 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fa0f4022040 con 0x7fa0fc011040 2023-10-05T19:29:34.662 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.658+0000 7fa103018700 1 -- 172.21.15.167:0/4173470315 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:74 5.f 5:f0000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x558f8453e140 con 0x558f84538920 2023-10-05T19:29:34.663 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.658+0000 7fa103018700 1 -- 172.21.15.167:0/4173470315 <== osd.1 v2:172.21.15.167:6800/30035 25 ==== osd_op_reply(74 [pgnls start_epoch 32 out=115b] v23'2 uv2 ondisk = 1) v8 ==== 144+0+115 (crc 0 0 0) 0x7fa0f4022040 con 0x558f84538920 2023-10-05T19:29:34.663 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.658+0000 7fa103018700 1 -- 172.21.15.167:0/4173470315 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:75 5.1f 5:f8000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x558f8453e140 con 0x558f84505d40 2023-10-05T19:29:34.664 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.658+0000 7fa102016700 1 -- 172.21.15.167:0/4173470315 <== osd.2 v2:172.21.15.167:6816/30037 12 ==== osd_op_reply(75 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fa0fc004520 con 0x558f84505d40 2023-10-05T19:29:34.664 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.658+0000 7fa104ce2e40 20 RGWRados::pool_iterate: got zonegroup_info.4f1f719d-9648-417b-b10c-e5d939825419 2023-10-05T19:29:34.664 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.658+0000 7fa104ce2e40 20 RGWRados::pool_iterate: got zone_info.9a9df965-6612-45d4-a45f-6e3c889bacc2 2023-10-05T19:29:34.664 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.658+0000 7fa104ce2e40 20 RGWRados::pool_iterate: got zone_names.default 2023-10-05T19:29:34.664 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.658+0000 7fa104ce2e40 20 RGWRados::pool_iterate: got zonegroups_names.default 2023-10-05T19:29:34.664 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.658+0000 7fa104ce2e40 20 rados->read ofs=0 len=0 2023-10-05T19:29:34.664 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.658+0000 7fa104ce2e40 1 -- 172.21.15.167:0/4173470315 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:76 5.f 5:f43b6ece:::zone_names.default:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32) v8 -- 0x558f84541fb0 con 0x558f84538920 2023-10-05T19:29:34.665 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.662+0000 7fa103018700 1 -- 172.21.15.167:0/4173470315 <== osd.1 v2:172.21.15.167:6800/30035 26 ==== osd_op_reply(76 zone_names.default [read 0~46 out=46b] v0'0 uv1 ondisk = 0) v8 ==== 162+0+46 (crc 0 0 0) 0x7fa0f4022040 con 0x558f84538920 2023-10-05T19:29:34.665 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.662+0000 7fa104ce2e40 20 rados_obj.operate() r=0 bl.length=46 2023-10-05T19:29:34.665 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.662+0000 7fa104ce2e40 20 rados->read ofs=0 len=0 2023-10-05T19:29:34.665 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.662+0000 7fa104ce2e40 1 -- 172.21.15.167:0/4173470315 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:77 5.1d 5:b8374eb4:::zone_info.9a9df965-6612-45d4-a45f-6e3c889bacc2:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32) v8 -- 0x558f84542360 con 0x558f84538920 2023-10-05T19:29:34.666 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.662+0000 7fa103018700 1 -- 172.21.15.167:0/4173470315 <== osd.1 v2:172.21.15.167:6800/30035 27 ==== osd_op_reply(77 zone_info.9a9df965-6612-45d4-a45f-6e3c889bacc2 [read 0~874 out=874b] v0'0 uv2 ondisk = 0) v8 ==== 190+0+874 (crc 0 0 0) 0x7fa0f4022040 con 0x558f84538920 2023-10-05T19:29:34.666 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.662+0000 7fa104ce2e40 20 rados_obj.operate() r=0 bl.length=874 2023-10-05T19:29:34.666 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.662+0000 7fa104ce2e40 20 rados->read ofs=0 len=0 2023-10-05T19:29:34.666 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.662+0000 7fa104ce2e40 1 -- 172.21.15.167:0/4173470315 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:78 5.f 5:f4c53578:::zonegroups_names.default:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32) v8 -- 0x558f8453dce0 con 0x558f84538920 2023-10-05T19:29:34.667 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.662+0000 7fa103018700 1 -- 172.21.15.167:0/4173470315 <== osd.1 v2:172.21.15.167:6800/30035 28 ==== osd_op_reply(78 zonegroups_names.default [read 0~46 out=46b] v0'0 uv2 ondisk = 0) v8 ==== 168+0+46 (crc 0 0 0) 0x7fa0f4022040 con 0x558f84538920 2023-10-05T19:29:34.667 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.662+0000 7fa104ce2e40 20 rados_obj.operate() r=0 bl.length=46 2023-10-05T19:29:34.667 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.662+0000 7fa104ce2e40 20 rados->read ofs=0 len=0 2023-10-05T19:29:34.667 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.662+0000 7fa104ce2e40 1 -- 172.21.15.167:0/4173470315 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:79 5.c 5:30d299a5:::zonegroup_info.4f1f719d-9648-417b-b10c-e5d939825419:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32) v8 -- 0x558f8453dd00 con 0x7fa0fc011040 2023-10-05T19:29:34.667 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.662+0000 7fa103018700 1 -- 172.21.15.167:0/4173470315 <== osd.0 v2:172.21.15.118:6800/27690 20 ==== osd_op_reply(79 zonegroup_info.4f1f719d-9648-417b-b10c-e5d939825419 [read 0~362 out=362b] v0'0 uv1 ondisk = 0) v8 ==== 195+0+362 (crc 0 0 0) 0x7fa0f4022040 con 0x7fa0fc011040 2023-10-05T19:29:34.668 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.662+0000 7fa104ce2e40 20 rados_obj.operate() r=0 bl.length=362 2023-10-05T19:29:34.668 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.662+0000 7fa104ce2e40 20 rados->read ofs=0 len=0 2023-10-05T19:29:34.668 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.662+0000 7fa104ce2e40 1 -- 172.21.15.167:0/4173470315 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:80 5.3 5:c0ad8cbd:::region_map:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32) v8 -- 0x558f84542dd0 con 0x7fa0fc011040 2023-10-05T19:29:34.668 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.662+0000 7fa103018700 1 -- 172.21.15.167:0/4173470315 <== osd.0 v2:172.21.15.118:6800/27690 21 ==== osd_op_reply(80 region_map [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 ==== 154+0+0 (crc 0 0 0) 0x7fa0f4022040 con 0x7fa0fc011040 2023-10-05T19:29:34.668 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.662+0000 7fa104ce2e40 20 rados_obj.operate() r=-2 bl.length=0 2023-10-05T19:29:34.669 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.662+0000 7fa104ce2e40 20 zone default found 2023-10-05T19:29:34.669 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.662+0000 7fa104ce2e40 4 Realm: () 2023-10-05T19:29:34.669 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.662+0000 7fa104ce2e40 4 ZoneGroup: default (4f1f719d-9648-417b-b10c-e5d939825419) 2023-10-05T19:29:34.669 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.662+0000 7fa104ce2e40 4 Zone: default (9a9df965-6612-45d4-a45f-6e3c889bacc2) 2023-10-05T19:29:34.669 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.662+0000 7fa104ce2e40 10 cannot find current period zonegroup using local zonegroup configuration 2023-10-05T19:29:34.669 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.662+0000 7fa104ce2e40 20 zonegroup default 2023-10-05T19:29:34.669 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.662+0000 7fa104ce2e40 20 rados->read ofs=0 len=0 2023-10-05T19:29:34.669 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.662+0000 7fa104ce2e40 1 -- 172.21.15.167:0/4173470315 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:81 5.3 5:c52100b6:::period_config.default:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32) v8 -- 0x558f84542dd0 con 0x7fa0fc011040 2023-10-05T19:29:34.671 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.666+0000 7fa103018700 1 -- 172.21.15.167:0/4173470315 <== osd.0 v2:172.21.15.118:6800/27690 22 ==== osd_op_reply(81 period_config.default [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 ==== 165+0+0 (crc 0 0 0) 0x7fa0f4045040 con 0x7fa0fc011040 2023-10-05T19:29:34.671 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.666+0000 7fa104ce2e40 20 rados_obj.operate() r=-2 bl.length=0 2023-10-05T19:29:34.671 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.666+0000 7fa104ce2e40 20 rados->read ofs=0 len=0 2023-10-05T19:29:34.671 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.666+0000 7fa104ce2e40 1 -- 172.21.15.167:0/4173470315 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:82 6.2 6:4347d321:::bucket.sync-source-hints.:head [call version.read in=11b,read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32) v8 -- 0x558f845443f0 con 0x558f8453eaa0 2023-10-05T19:29:34.671 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.666+0000 7fa102817700 1 -- 172.21.15.167:0/4173470315 <== osd.3 v2:172.21.15.167:6808/30036 20 ==== osd_op_reply(82 bucket.sync-source-hints. [call,read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 ==== 211+0+0 (crc 0 0 0) 0x7fa0ec035070 con 0x558f8453eaa0 2023-10-05T19:29:34.671 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.666+0000 7fa104ce2e40 20 rados_obj.operate() r=-2 bl.length=0 2023-10-05T19:29:34.671 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.666+0000 7fa104ce2e40 20 rados->read ofs=0 len=0 2023-10-05T19:29:34.671 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.666+0000 7fa104ce2e40 1 -- 172.21.15.167:0/4173470315 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:83 6.b 6:d467b91b:::bucket.sync-target-hints.:head [call version.read in=11b,read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32) v8 -- 0x558f845447a0 con 0x558f8453eaa0 2023-10-05T19:29:34.671 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.666+0000 7fa102817700 1 -- 172.21.15.167:0/4173470315 <== osd.3 v2:172.21.15.167:6808/30036 21 ==== osd_op_reply(83 bucket.sync-target-hints. [call,read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 ==== 211+0+0 (crc 0 0 0) 0x7fa0ec035070 con 0x558f8453eaa0 2023-10-05T19:29:34.672 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.666+0000 7fa104ce2e40 20 rados_obj.operate() r=-2 bl.length=0 2023-10-05T19:29:34.672 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.666+0000 7fa104ce2e40 20 started sync module instance, tier type = 2023-10-05T19:29:34.672 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.666+0000 7fa104ce2e40 20 started zone id=9a9df965-6612-45d4-a45f-6e3c889bacc2 (name=default) with tier type = 2023-10-05T19:29:34.672 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.666+0000 7fa104ce2e40 1 -- 172.21.15.167:0/4173470315 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:84 7.1f 7:f95f44c2:::notify.0:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e32) v8 -- 0x558f84543ab0 con 0x558f84505d40 2023-10-05T19:29:34.674 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.670+0000 7fa102016700 1 -- 172.21.15.167:0/4173470315 <== osd.2 v2:172.21.15.167:6816/30037 13 ==== osd_op_reply(84 notify.0 [create] v32'240 uv240 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7fa0fc02f040 con 0x558f84505d40 2023-10-05T19:29:34.674 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.670+0000 7fa104ce2e40 1 -- 172.21.15.167:0/4173470315 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:85 7.1f 7:f95f44c2:::notify.0:head [watch watch cookie 94074888797792] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e32) v8 -- 0x558f845436e0 con 0x558f84505d40 2023-10-05T19:29:34.674 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.670+0000 7fa104ce2e40 1 -- 172.21.15.167:0/4173470315 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:86 7.0 7:05bf5b68:::notify.1:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e32) v8 -- 0x558f84545560 con 0x558f84538920 2023-10-05T19:29:34.677 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.674+0000 7fa102016700 1 -- 172.21.15.167:0/4173470315 <== osd.2 v2:172.21.15.167:6816/30037 14 ==== osd_op_reply(85 notify.0 [watch watch cookie 94074888797792] v32'241 uv240 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7fa0fc02f040 con 0x558f84505d40 2023-10-05T19:29:34.677 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.674+0000 7fa103018700 1 -- 172.21.15.167:0/4173470315 <== osd.1 v2:172.21.15.167:6800/30035 29 ==== osd_op_reply(86 notify.1 [create] v32'240 uv240 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7fa0f4045040 con 0x558f84538920 2023-10-05T19:29:34.677 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.674+0000 7fa104ce2e40 1 -- 172.21.15.167:0/4173470315 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:87 7.0 7:05bf5b68:::notify.1:head [watch watch cookie 94074888802416] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e32) v8 -- 0x558f84544c70 con 0x558f84538920 2023-10-05T19:29:34.677 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.674+0000 7fa104ce2e40 1 -- 172.21.15.167:0/4173470315 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:88 7.15 7:a93a5511:::notify.2:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e32) v8 -- 0x558f84546340 con 0x558f8453eaa0 2023-10-05T19:29:34.679 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.674+0000 7fa103018700 1 -- 172.21.15.167:0/4173470315 <== osd.1 v2:172.21.15.167:6800/30035 30 ==== osd_op_reply(87 notify.1 [watch watch cookie 94074888802416] v32'241 uv240 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7fa0f4045040 con 0x558f84538920 2023-10-05T19:29:34.680 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.674+0000 7fa102817700 1 -- 172.21.15.167:0/4173470315 <== osd.3 v2:172.21.15.167:6808/30036 22 ==== osd_op_reply(88 notify.2 [create] v32'240 uv240 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7fa0ec035070 con 0x558f8453eaa0 2023-10-05T19:29:34.680 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.674+0000 7fa104ce2e40 1 -- 172.21.15.167:0/4173470315 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:89 7.15 7:a93a5511:::notify.2:head [watch watch cookie 94074888804704] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e32) v8 -- 0x558f84544c70 con 0x558f8453eaa0 2023-10-05T19:29:34.681 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.678+0000 7fa104ce2e40 1 -- 172.21.15.167:0/4173470315 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:90 7.e 7:7759931f:::notify.3:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e32) v8 -- 0x558f84546d50 con 0x558f84505d40 2023-10-05T19:29:34.683 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.678+0000 7fa102016700 1 -- 172.21.15.167:0/4173470315 <== osd.2 v2:172.21.15.167:6816/30037 15 ==== osd_op_reply(90 notify.3 [create] v32'240 uv240 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7fa0fc02f040 con 0x558f84505d40 2023-10-05T19:29:34.683 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.678+0000 7fa104ce2e40 1 -- 172.21.15.167:0/4173470315 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:91 7.e 7:7759931f:::notify.3:head [watch watch cookie 94074888810832] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e32) v8 -- 0x558f84547a90 con 0x558f84505d40 2023-10-05T19:29:34.683 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.678+0000 7fa104ce2e40 1 -- 172.21.15.167:0/4173470315 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:92 7.d 7:b4812045:::notify.4:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e32) v8 -- 0x558f84548330 con 0x558f84538920 2023-10-05T19:29:34.684 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.678+0000 7fa102817700 1 -- 172.21.15.167:0/4173470315 <== osd.3 v2:172.21.15.167:6808/30036 23 ==== osd_op_reply(89 notify.2 [watch watch cookie 94074888804704] v32'241 uv240 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7fa0ec035070 con 0x558f8453eaa0 2023-10-05T19:29:34.686 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.682+0000 7fa102016700 1 -- 172.21.15.167:0/4173470315 <== osd.2 v2:172.21.15.167:6816/30037 16 ==== osd_op_reply(91 notify.3 [watch watch cookie 94074888810832] v32'241 uv240 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7fa0fc02f040 con 0x558f84505d40 2023-10-05T19:29:34.686 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.682+0000 7fa103018700 1 -- 172.21.15.167:0/4173470315 <== osd.1 v2:172.21.15.167:6800/30035 31 ==== osd_op_reply(92 notify.4 [create] v32'240 uv240 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7fa0f4045040 con 0x558f84538920 2023-10-05T19:29:34.686 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.682+0000 7fa104ce2e40 1 -- 172.21.15.167:0/4173470315 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:93 7.d 7:b4812045:::notify.4:head [watch watch cookie 94074888805808] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e32) v8 -- 0x558f84548330 con 0x558f84538920 2023-10-05T19:29:34.686 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.682+0000 7fa104ce2e40 1 -- 172.21.15.167:0/4173470315 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:94 7.3 7:c609908c:::notify.5:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e32) v8 -- 0x558f84548c20 con 0x7fa0fc011040 2023-10-05T19:29:34.689 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.686+0000 7fa103018700 1 -- 172.21.15.167:0/4173470315 <== osd.1 v2:172.21.15.167:6800/30035 32 ==== osd_op_reply(93 notify.4 [watch watch cookie 94074888805808] v32'241 uv240 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7fa0f4045040 con 0x558f84538920 2023-10-05T19:29:34.689 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.686+0000 7fa103018700 1 -- 172.21.15.167:0/4173470315 <== osd.0 v2:172.21.15.118:6800/27690 23 ==== osd_op_reply(94 notify.5 [create] v32'240 uv240 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7fa0f4045040 con 0x7fa0fc011040 2023-10-05T19:29:34.689 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.686+0000 7fa104ce2e40 1 -- 172.21.15.167:0/4173470315 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:95 7.3 7:c609908c:::notify.5:head [watch watch cookie 94074888814704] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e32) v8 -- 0x558f845489b0 con 0x7fa0fc011040 2023-10-05T19:29:34.689 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.686+0000 7fa104ce2e40 1 -- 172.21.15.167:0/4173470315 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:96 7.14 7:2b04a3e9:::notify.6:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e32) v8 -- 0x558f845494a0 con 0x7fa0fc011040 2023-10-05T19:29:34.691 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.686+0000 7fa103018700 1 -- 172.21.15.167:0/4173470315 <== osd.0 v2:172.21.15.118:6800/27690 24 ==== osd_op_reply(95 notify.5 [watch watch cookie 94074888814704] v32'241 uv240 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7fa0f4045040 con 0x7fa0fc011040 2023-10-05T19:29:34.692 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.686+0000 7fa103018700 1 -- 172.21.15.167:0/4173470315 <== osd.0 v2:172.21.15.118:6800/27690 25 ==== osd_op_reply(96 notify.6 [create] v32'240 uv240 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7fa0f4045040 con 0x7fa0fc011040 2023-10-05T19:29:34.692 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.686+0000 7fa104ce2e40 1 -- 172.21.15.167:0/4173470315 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:97 7.14 7:2b04a3e9:::notify.6:head [watch watch cookie 94074888815808] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e32) v8 -- 0x558f84548950 con 0x7fa0fc011040 2023-10-05T19:29:34.692 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.686+0000 7fa104ce2e40 1 -- 172.21.15.167:0/4173470315 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:98 7.9 7:93e5b521:::notify.7:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e32) v8 -- 0x558f84549eb0 con 0x7fa0fc011040 2023-10-05T19:29:34.695 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.690+0000 7fa103018700 1 -- 172.21.15.167:0/4173470315 <== osd.0 v2:172.21.15.118:6800/27690 26 ==== osd_op_reply(97 notify.6 [watch watch cookie 94074888815808] v32'241 uv240 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7fa0f4045040 con 0x7fa0fc011040 2023-10-05T19:29:34.695 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.690+0000 7fa103018700 1 -- 172.21.15.167:0/4173470315 <== osd.0 v2:172.21.15.118:6800/27690 27 ==== osd_op_reply(98 notify.7 [create] v32'240 uv240 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7fa0f4045040 con 0x7fa0fc011040 2023-10-05T19:29:34.695 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.690+0000 7fa104ce2e40 1 -- 172.21.15.167:0/4173470315 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:99 7.9 7:93e5b521:::notify.7:head [watch watch cookie 94074888819168] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e32) v8 -- 0x558f84549e70 con 0x7fa0fc011040 2023-10-05T19:29:34.695 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.690+0000 7fa104ce2e40 20 add_watcher() i=0 2023-10-05T19:29:34.696 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.690+0000 7fa104ce2e40 20 add_watcher() i=1 2023-10-05T19:29:34.696 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.690+0000 7fa104ce2e40 20 add_watcher() i=2 2023-10-05T19:29:34.696 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.690+0000 7fa104ce2e40 20 add_watcher() i=3 2023-10-05T19:29:34.696 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.690+0000 7fa104ce2e40 20 add_watcher() i=4 2023-10-05T19:29:34.696 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.690+0000 7fa104ce2e40 20 add_watcher() i=5 2023-10-05T19:29:34.696 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.690+0000 7fa104ce2e40 20 add_watcher() i=6 2023-10-05T19:29:34.698 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.694+0000 7fa103018700 1 -- 172.21.15.167:0/4173470315 <== osd.0 v2:172.21.15.118:6800/27690 28 ==== osd_op_reply(99 notify.7 [watch watch cookie 94074888819168] v32'241 uv240 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7fa0f4045040 con 0x7fa0fc011040 2023-10-05T19:29:34.698 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.694+0000 7fa104ce2e40 20 add_watcher() i=7 2023-10-05T19:29:34.698 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.694+0000 7fa104ce2e40 2 all 8 watchers are set, enabling cache 2023-10-05T19:29:34.698 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.694+0000 7fa104ce2e40 1 -- 172.21.15.167:0/4173470315 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:100 6.4 6:22d26bf9:::data_loggenerations_metadata:head [call version.check_conds in=74b,call version.read in=11b,read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32) v8 -- 0x558f8454d280 con 0x558f84538920 2023-10-05T19:29:34.701 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.694+0000 7fa103018700 1 -- 172.21.15.167:0/4173470315 <== osd.1 v2:172.21.15.167:6800/30035 33 ==== osd_op_reply(100 data_loggenerations_metadata [call,call out=48b,read 0~28 out=28b] v0'0 uv1 ondisk = 0) v8 ==== 256+0+76 (crc 0 0 0) 0x7fa0f4045040 con 0x558f84538920 2023-10-05T19:29:34.701 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.694+0000 7fa104ce2e40 1 -- 172.21.15.167:0/4173470315 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:101 6.4 6:22d26bf9:::data_loggenerations_metadata:head [watch watch cookie 94074888821584] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e32) v8 -- 0x558f8454a740 con 0x558f84538920 2023-10-05T19:29:34.702 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.698+0000 7fa103018700 1 -- 172.21.15.167:0/4173470315 <== osd.1 v2:172.21.15.167:6800/30035 34 ==== osd_op_reply(101 data_loggenerations_metadata [watch watch cookie 94074888821584] v32'233 uv1 ondisk = 0) v8 ==== 172+0+0 (crc 0 0 0) 0x7fa0f4045040 con 0x558f84538920 2023-10-05T19:29:34.702 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.698+0000 7fa04ffff700 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2023-10-05T19:29:34.702 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.698+0000 7fa104ce2e40 20 check_secure_mon_conn(): auth registy supported: methods=[2] modes=[2,1] 2023-10-05T19:29:34.702 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.698+0000 7fa104ce2e40 20 check_secure_mon_conn(): mode 1 is insecure 2023-10-05T19:29:34.702 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.698+0000 7fa104ce2e40 1 Processor -- start 2023-10-05T19:29:34.702 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.698+0000 7fa104ce2e40 1 -- start start 2023-10-05T19:29:34.703 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.698+0000 7fa104ce2e40 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x558f84557900 0x558f84557cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-05T19:29:34.703 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.698+0000 7fa104ce2e40 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x558f84558210 con 0x558f84557900 2023-10-05T19:29:34.704 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.698+0000 7fa102016700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x558f84557900 0x558f84557cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-05T19:29:34.704 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.698+0000 7fa102016700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x558f84557900 0x558f84557cd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.118:3300/0 says I am v2:172.21.15.167:56728/0 (socket says 172.21.15.167:56728) 2023-10-05T19:29:34.704 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.698+0000 7fa102016700 1 -- 172.21.15.167:0/31268495 learned_addr learned my addr 172.21.15.167:0/31268495 (peer_addr_for_me v2:172.21.15.167:0/0) 2023-10-05T19:29:34.704 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.698+0000 7fa102016700 1 -- 172.21.15.167:0/31268495 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x558f84558350 con 0x558f84557900 2023-10-05T19:29:34.704 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.698+0000 7fa102016700 1 --2- 172.21.15.167:0/31268495 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x558f84557900 0x558f84557cd0 secure :-1 s=READY pgs=618 cs=0 l=1 rev1=1 crypto rx=0x558f8453c730 tx=0x7fa0fc006000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=49d379820e4ff7e7 server_cookie=0 in_seq=0 out_seq=0 2023-10-05T19:29:34.704 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.698+0000 7fa04f7fe700 1 -- 172.21.15.167:0/31268495 <== mon.0 v2:172.21.15.118:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fa0fc02c040 con 0x558f84557900 2023-10-05T19:29:34.705 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.698+0000 7fa04f7fe700 1 -- 172.21.15.167:0/31268495 <== mon.0 v2:172.21.15.118:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa0fc02d040 con 0x558f84557900 2023-10-05T19:29:34.705 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.698+0000 7fa04f7fe700 1 -- 172.21.15.167:0/31268495 <== mon.0 v2:172.21.15.118:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fa0fc028a20 con 0x558f84557900 2023-10-05T19:29:34.706 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.702+0000 7fa104ce2e40 1 -- 172.21.15.167:0/31268495 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x558f84557900 msgr2=0x558f84557cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-05T19:29:34.706 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.702+0000 7fa104ce2e40 1 --2- 172.21.15.167:0/31268495 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x558f84557900 0x558f84557cd0 secure :-1 s=READY pgs=618 cs=0 l=1 rev1=1 crypto rx=0x558f8453c730 tx=0x7fa0fc006000 comp rx=0 tx=0).stop 2023-10-05T19:29:34.706 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.702+0000 7fa104ce2e40 1 -- 172.21.15.167:0/31268495 shutdown_connections 2023-10-05T19:29:34.706 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.702+0000 7fa104ce2e40 1 --2- 172.21.15.167:0/31268495 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x558f84557900 0x558f84557cd0 unknown :-1 s=CLOSED pgs=618 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-05T19:29:34.706 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.702+0000 7fa104ce2e40 1 -- 172.21.15.167:0/31268495 >> 172.21.15.167:0/31268495 conn(0x558f84552cc0 msgr2=0x558f84550e10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-05T19:29:34.706 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.702+0000 7fa104ce2e40 1 -- 172.21.15.167:0/31268495 shutdown_connections 2023-10-05T19:29:34.706 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.702+0000 7fa104ce2e40 1 -- 172.21.15.167:0/31268495 wait complete. 2023-10-05T19:29:34.706 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.702+0000 7fa104ce2e40 1 Processor -- start 2023-10-05T19:29:34.706 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.702+0000 7fa104ce2e40 1 -- start start 2023-10-05T19:29:34.707 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.702+0000 7fa104ce2e40 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x558f84557900 0x558f8455d400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-05T19:29:34.707 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.702+0000 7fa104ce2e40 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x558f8455e400 con 0x558f84557900 2023-10-05T19:29:34.707 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.702+0000 7fa102016700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x558f84557900 0x558f8455d400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-05T19:29:34.707 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.702+0000 7fa102016700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x558f84557900 0x558f8455d400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.118:3300/0 says I am v2:172.21.15.167:56742/0 (socket says 172.21.15.167:56742) 2023-10-05T19:29:34.708 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.702+0000 7fa102016700 1 -- 172.21.15.167:0/989994640 learned_addr learned my addr 172.21.15.167:0/989994640 (peer_addr_for_me v2:172.21.15.167:0/0) 2023-10-05T19:29:34.708 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.702+0000 7fa102016700 1 -- 172.21.15.167:0/989994640 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa0f0001e10 con 0x558f84557900 2023-10-05T19:29:34.708 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.702+0000 7fa102016700 1 --2- 172.21.15.167:0/989994640 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x558f84557900 0x558f8455d400 secure :-1 s=READY pgs=619 cs=0 l=1 rev1=1 crypto rx=0x7fa0fc006040 tx=0x7fa0fc036000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-05T19:29:34.709 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.702+0000 7fa04dffb700 1 -- 172.21.15.167:0/989994640 <== mon.0 v2:172.21.15.118:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fa0fc02c030 con 0x558f84557900 2023-10-05T19:29:34.709 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.702+0000 7fa04dffb700 1 -- 172.21.15.167:0/989994640 <== mon.0 v2:172.21.15.118:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa0fc01c040 con 0x558f84557900 2023-10-05T19:29:34.709 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.702+0000 7fa104ce2e40 1 -- 172.21.15.167:0/989994640 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x558f8455da00 con 0x558f84557900 2023-10-05T19:29:34.709 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.702+0000 7fa04dffb700 1 -- 172.21.15.167:0/989994640 <== mon.0 v2:172.21.15.118:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fa0fc0041d0 con 0x558f84557900 2023-10-05T19:29:34.709 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.702+0000 7fa104ce2e40 1 -- 172.21.15.167:0/989994640 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x558f8455dea0 con 0x558f84557900 2023-10-05T19:29:34.709 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.706+0000 7fa04dffb700 1 -- 172.21.15.167:0/989994640 <== mon.0 v2:172.21.15.118:3300/0 4 ==== mgrmap(e 5) v1 ==== 40329+0+0 (secure 0 0 0) 0x7fa0fc003450 con 0x558f84557900 2023-10-05T19:29:34.710 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.706+0000 7fa04dffb700 1 --2- 172.21.15.167:0/989994640 >> [v2:172.21.15.167:6824/29889,v1:172.21.15.167:6825/29889] conn(0x7fa040030c00 0x7fa0400330b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-05T19:29:34.710 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.706+0000 7fa04dffb700 1 -- 172.21.15.167:0/989994640 <== mon.0 v2:172.21.15.118:3300/0 5 ==== osd_map(32..32 src has 1..32) v4 ==== 6260+0+0 (secure 0 0 0) 0x7fa0fc049d50 con 0x558f84557900 2023-10-05T19:29:34.710 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.706+0000 7fa104ce2e40 5 note: GC not initialized 2023-10-05T19:29:34.710 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.706+0000 7fa103018700 1 --2- 172.21.15.167:0/989994640 >> [v2:172.21.15.167:6824/29889,v1:172.21.15.167:6825/29889] conn(0x7fa040030c00 0x7fa0400330b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-05T19:29:34.711 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.706+0000 7fa103018700 1 --2- 172.21.15.167:0/989994640 >> [v2:172.21.15.167:6824/29889,v1:172.21.15.167:6825/29889] conn(0x7fa040030c00 0x7fa0400330b0 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x558f8454a0b0 tx=0x7fa0f4000f90 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-05T19:29:34.713 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.710+0000 7fa046ffd700 20 reqs_thread_entry: start 2023-10-05T19:29:34.714 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.710+0000 7fa044ff9700 20 reqs_thread_entry: start 2023-10-05T19:29:34.714 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.710+0000 7fa104ce2e40 20 init_complete bucket index max shards: 11 2023-10-05T19:29:34.714 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.710+0000 7fa104ce2e40 10 rgw notify: Started notification manager with: 1 workers 2023-10-05T19:29:34.715 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.710+0000 7fa032ffd700 20 rgw notify: INFO: next queues processing will happen at: Thu Oct 5 19:30:05 2023 2023-10-05T19:29:34.715 INFO:teuthology.orchestra.run.smithi167.stderr: 2023-10-05T19:29:34.715 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.710+0000 7fa0327fc700 20 reqs_thread_entry: start 2023-10-05T19:29:34.715 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.710+0000 7fa104ce2e40 20 get_system_obj_state: rctx=0x7ffd1969d358 obj=default.rgw.meta:users.uid:bar state=0x558f843e00b0 s->prefetch_data=0 2023-10-05T19:29:34.715 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.710+0000 7fa104ce2e40 10 cache get: name=default.rgw.meta+users.uid+bar : miss 2023-10-05T19:29:34.715 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.710+0000 7fa104ce2e40 1 -- 172.21.15.167:0/4173470315 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:102 8.e 8:718f6a54:users.uid::bar:head [call version.read in=11b,getxattrs,stat] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32) v8 -- 0x558f845c5c30 con 0x7fa0fc011040 2023-10-05T19:29:34.716 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.710+0000 7fa103018700 1 -- 172.21.15.167:0/4173470315 <== osd.0 v2:172.21.15.118:6800/27690 29 ==== osd_op_reply(102 bar [call,getxattrs,stat] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 ==== 231+0+0 (crc 0 0 0) 0x7fa0f4045040 con 0x7fa0fc011040 2023-10-05T19:29:34.716 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.710+0000 7fa104ce2e40 10 cache put: name=default.rgw.meta+users.uid+bar info.flags=0x0 2023-10-05T19:29:34.716 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.710+0000 7fa104ce2e40 10 adding default.rgw.meta+users.uid+bar to cache LRU end 2023-10-05T19:29:34.717 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.710+0000 7fa104ce2e40 20 get_system_obj_state: rctx=0x7ffd1969d048 obj=default.rgw.meta:users.keys:NX5QOQKC6BH2IDN8HC7A state=0x558f843e00b0 s->prefetch_data=0 2023-10-05T19:29:34.717 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.710+0000 7fa104ce2e40 10 cache get: name=default.rgw.meta+users.keys+NX5QOQKC6BH2IDN8HC7A : miss 2023-10-05T19:29:34.717 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.710+0000 7fa104ce2e40 1 -- 172.21.15.167:0/4173470315 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:103 8.a 8:50c8eddb:users.keys::NX5QOQKC6BH2IDN8HC7A:head [getxattrs,stat] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32) v8 -- 0x558f845c5c30 con 0x558f84538920 2023-10-05T19:29:34.717 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.710+0000 7fa103018700 1 -- 172.21.15.167:0/4173470315 <== osd.1 v2:172.21.15.167:6800/30035 35 ==== osd_op_reply(103 NX5QOQKC6BH2IDN8HC7A [getxattrs,stat] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 ==== 206+0+0 (crc 0 0 0) 0x7fa0f4045040 con 0x558f84538920 2023-10-05T19:29:34.717 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.710+0000 7fa104ce2e40 10 cache put: name=default.rgw.meta+users.keys+NX5QOQKC6BH2IDN8HC7A info.flags=0x0 2023-10-05T19:29:34.717 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.710+0000 7fa104ce2e40 10 adding default.rgw.meta+users.keys+NX5QOQKC6BH2IDN8HC7A to cache LRU end 2023-10-05T19:29:34.717 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.710+0000 7fa104ce2e40 20 get_system_obj_state: rctx=0x7ffd1969c8e8 obj=default.rgw.meta:users.keys:NX5QOQKC6BH2IDN8HC7A state=0x558f843e00b0 s->prefetch_data=0 2023-10-05T19:29:34.717 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.710+0000 7fa104ce2e40 10 cache get: name=default.rgw.meta+users.keys+NX5QOQKC6BH2IDN8HC7A : hit (negative entry) 2023-10-05T19:29:34.717 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.710+0000 7fa104ce2e40 20 get_system_obj_state: rctx=0x7ffd1969cf88 obj=default.rgw.meta:users.keys:NX5QOQKC6BH2IDN8HC7A state=0x558f843e00b0 s->prefetch_data=0 2023-10-05T19:29:34.717 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.710+0000 7fa104ce2e40 10 cache get: name=default.rgw.meta+users.keys+NX5QOQKC6BH2IDN8HC7A : hit (negative entry) 2023-10-05T19:29:34.717 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.710+0000 7fa104ce2e40 1 -- 172.21.15.167:0/4173470315 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:104 8.e 8:718f6a54:users.uid::bar:head [delete,create,call version.set in=58b,writefull 0~346 in=346b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e32) v8 -- 0x558f845c81d0 con 0x7fa0fc011040 2023-10-05T19:29:34.720 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.714+0000 7fa103018700 1 -- 172.21.15.167:0/4173470315 <== osd.0 v2:172.21.15.118:6800/27690 30 ==== osd_op_reply(104 bar [delete,create,call,writefull 0~346] v32'3 uv3 ondisk = 0) v8 ==== 273+0+0 (crc 0 0 0) 0x7fa0f4045040 con 0x7fa0fc011040 2023-10-05T19:29:34.720 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.714+0000 7fa104ce2e40 10 cache put: name=default.rgw.meta+users.uid+bar info.flags=0x17 2023-10-05T19:29:34.720 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.714+0000 7fa104ce2e40 10 moving default.rgw.meta+users.uid+bar to cache LRU end 2023-10-05T19:29:34.720 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.714+0000 7fa104ce2e40 10 distributing notification oid=default.rgw.control:notify.7 cni=[op: 0, obj: default.rgw.meta:users.uid:bar, ofs0, ns] 2023-10-05T19:29:34.720 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.714+0000 7fa104ce2e40 1 -- 172.21.15.167:0/4173470315 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:105 7.9 7:93e5b521:::notify.7:head [notify cookie 94074889335232 in=534b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32) v8 -- 0x558f845c7b60 con 0x7fa0fc011040 2023-10-05T19:29:34.721 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.718+0000 7fa103018700 1 -- 172.21.15.167:0/4173470315 <== osd.0 v2:172.21.15.118:6800/27690 31 ==== watch-notify(notify (1) cookie 94074888819168 notify 137438953503 ret 0) v3 ==== 564+0+0 (crc 0 0 0) 0x7fa0f4031040 con 0x7fa0fc011040 2023-10-05T19:29:34.721 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.718+0000 7fa100813700 10 rgw watcher librados: RGWWatcher::handle_notify() notify_id 137438953503 cookie 94074888819168 notifier 4707 bl.length()=522 2023-10-05T19:29:34.721 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.718+0000 7fa103018700 1 -- 172.21.15.167:0/4173470315 <== osd.0 v2:172.21.15.118:6800/27690 32 ==== osd_op_reply(105 notify.7 [notify cookie 94074889335232 out=8b] v0'0 uv240 ondisk = 0) v8 ==== 152+0+8 (crc 0 0 0) 0x7fa0f4045040 con 0x7fa0fc011040 2023-10-05T19:29:34.721 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.718+0000 7fa100813700 10 rgw watcher librados: cache put: name=default.rgw.meta+users.uid+bar info.flags=0x17 2023-10-05T19:29:34.721 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.718+0000 7fa100813700 10 rgw watcher librados: moving default.rgw.meta+users.uid+bar to cache LRU end 2023-10-05T19:29:34.721 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.718+0000 7fa100813700 1 -- 172.21.15.167:0/4173470315 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:106 7.9 7:93e5b521:::notify.7:head [notify-ack in=20b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32) v8 -- 0x7fa0e8003d90 con 0x7fa0fc011040 2023-10-05T19:29:34.722 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.718+0000 7fa103018700 1 -- 172.21.15.167:0/4173470315 <== osd.0 v2:172.21.15.118:6800/27690 33 ==== osd_op_reply(106 notify.7 [notify-ack] v0'0 uv240 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7fa0f4045040 con 0x7fa0fc011040 2023-10-05T19:29:34.722 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.718+0000 7fa103018700 1 -- 172.21.15.167:0/4173470315 <== osd.0 v2:172.21.15.118:6800/27690 34 ==== watch-notify(notify_complete (2) cookie 94074889335232 notify 137438953503 ret 0) v3 ==== 42+0+48 (crc 0 0 0) 0x7fa0f404b210 con 0x7fa0fc011040 2023-10-05T19:29:34.722 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.718+0000 7fa104ce2e40 1 -- 172.21.15.167:0/4173470315 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:107 8.a 8:50c8eddb:users.keys::NX5QOQKC6BH2IDN8HC7A:head [delete,create,writefull 0~7 in=7b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e32) v8 -- 0x558f845c61b0 con 0x558f84538920 2023-10-05T19:29:34.725 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.722+0000 7fa103018700 1 -- 172.21.15.167:0/4173470315 <== osd.1 v2:172.21.15.167:6800/30035 36 ==== osd_op_reply(107 NX5QOQKC6BH2IDN8HC7A [delete,create,writefull 0~7] v32'1 uv1 ondisk = 0) v8 ==== 248+0+0 (crc 0 0 0) 0x7fa0f4045040 con 0x558f84538920 2023-10-05T19:29:34.725 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.722+0000 7fa104ce2e40 10 cache put: name=default.rgw.meta+users.keys+NX5QOQKC6BH2IDN8HC7A info.flags=0x7 2023-10-05T19:29:34.725 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.722+0000 7fa104ce2e40 10 moving default.rgw.meta+users.keys+NX5QOQKC6BH2IDN8HC7A to cache LRU end 2023-10-05T19:29:34.725 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.722+0000 7fa104ce2e40 10 distributing notification oid=default.rgw.control:notify.5 cni=[op: 0, obj: default.rgw.meta:users.keys:NX5QOQKC6BH2IDN8HC7A, ofs0, ns] 2023-10-05T19:29:34.725 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.722+0000 7fa104ce2e40 1 -- 172.21.15.167:0/4173470315 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:108 7.3 7:c609908c:::notify.5:head [notify cookie 94074889338816 in=189b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32) v8 -- 0x558f845c61b0 con 0x7fa0fc011040 2023-10-05T19:29:34.726 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.722+0000 7fa103018700 1 -- 172.21.15.167:0/4173470315 <== osd.0 v2:172.21.15.118:6800/27690 35 ==== watch-notify(notify (1) cookie 94074888814704 notify 137438953504 ret 0) v3 ==== 219+0+0 (crc 0 0 0) 0x7fa0f404b210 con 0x7fa0fc011040 2023-10-05T19:29:34.726 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.722+0000 7fa100813700 10 rgw watcher librados: RGWWatcher::handle_notify() notify_id 137438953504 cookie 94074888814704 notifier 4707 bl.length()=177 2023-10-05T19:29:34.726 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.722+0000 7fa103018700 1 -- 172.21.15.167:0/4173470315 <== osd.0 v2:172.21.15.118:6800/27690 36 ==== osd_op_reply(108 notify.5 [notify cookie 94074889338816 out=8b] v0'0 uv240 ondisk = 0) v8 ==== 152+0+8 (crc 0 0 0) 0x7fa0f4045040 con 0x7fa0fc011040 2023-10-05T19:29:34.726 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.722+0000 7fa100813700 10 rgw watcher librados: cache put: name=default.rgw.meta+users.keys+NX5QOQKC6BH2IDN8HC7A info.flags=0x7 2023-10-05T19:29:34.727 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.722+0000 7fa100813700 10 rgw watcher librados: moving default.rgw.meta+users.keys+NX5QOQKC6BH2IDN8HC7A to cache LRU end 2023-10-05T19:29:34.727 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.722+0000 7fa100813700 1 -- 172.21.15.167:0/4173470315 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:109 7.3 7:c609908c:::notify.5:head [notify-ack in=20b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32) v8 -- 0x7fa0e8003d90 con 0x7fa0fc011040 2023-10-05T19:29:34.727 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.722+0000 7fa103018700 1 -- 172.21.15.167:0/4173470315 <== osd.0 v2:172.21.15.118:6800/27690 37 ==== osd_op_reply(109 notify.5 [notify-ack] v0'0 uv240 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7fa0f4045040 con 0x7fa0fc011040 2023-10-05T19:29:34.727 INFO:teuthology.orchestra.run.smithi167.stdout:{ 2023-10-05T19:29:34.727 INFO:teuthology.orchestra.run.smithi167.stdout: "user_id": "bar", 2023-10-05T19:29:34.727 INFO:teuthology.orchestra.run.smithi167.stdout: "display_name": "Bar", 2023-10-05T19:29:34.728 INFO:teuthology.orchestra.run.smithi167.stdout: "email": "", 2023-10-05T19:29:34.728 INFO:teuthology.orchestra.run.smithi167.stdout: "suspended": 0, 2023-10-05T19:29:34.728 INFO:teuthology.orchestra.run.smithi167.stdout: "max_buckets": 4, 2023-10-05T19:29:34.728 INFO:teuthology.orchestra.run.smithi167.stdout: "subusers": [], 2023-10-05T19:29:34.728 INFO:teuthology.orchestra.run.smithi167.stdout: "keys": [ 2023-10-05T19:29:34.728 INFO:teuthology.orchestra.run.smithi167.stdout: { 2023-10-05T19:29:34.728 INFO:teuthology.orchestra.run.smithi167.stdout: "user": "bar", 2023-10-05T19:29:34.728 INFO:teuthology.orchestra.run.smithi167.stdout: "access_key": "NX5QOQKC6BH2IDN8HC7A", 2023-10-05T19:29:34.728 INFO:teuthology.orchestra.run.smithi167.stdout: "secret_key": "LnEsqNNqZIpkzauboDcLXLcYaWwLQ3Kop0zAnKIn" 2023-10-05T19:29:34.728 INFO:teuthology.orchestra.run.smithi167.stdout: } 2023-10-05T19:29:34.728 INFO:teuthology.orchestra.run.smithi167.stdout: ], 2023-10-05T19:29:34.728 INFO:teuthology.orchestra.run.smithi167.stdout: "swift_keys": [], 2023-10-05T19:29:34.729 INFO:teuthology.orchestra.run.smithi167.stdout: "caps": [], 2023-10-05T19:29:34.729 INFO:teuthology.orchestra.run.smithi167.stdout: "op_mask": "read, write, delete", 2023-10-05T19:29:34.729 INFO:teuthology.orchestra.run.smithi167.stdout: "default_placement": "", 2023-10-05T19:29:34.729 INFO:teuthology.orchestra.run.smithi167.stdout: "default_storage_class": "", 2023-10-05T19:29:34.729 INFO:teuthology.orchestra.run.smithi167.stdout: "placement_tags": [], 2023-10-05T19:29:34.729 INFO:teuthology.orchestra.run.smithi167.stdout: "bucket_quota": { 2023-10-05T19:29:34.729 INFO:teuthology.orchestra.run.smithi167.stdout: "enabled": false, 2023-10-05T19:29:34.729 INFO:teuthology.orchestra.run.smithi167.stdout: "check_on_raw": false, 2023-10-05T19:29:34.729 INFO:teuthology.orchestra.run.smithi167.stdout: "max_size": -1, 2023-10-05T19:29:34.729 INFO:teuthology.orchestra.run.smithi167.stdout: "max_size_kb": 0, 2023-10-05T19:29:34.729 INFO:teuthology.orchestra.run.smithi167.stdout: "max_objects": -1 2023-10-05T19:29:34.729 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2023-10-05T19:29:34.730 INFO:teuthology.orchestra.run.smithi167.stdout: "user_quota": { 2023-10-05T19:29:34.730 INFO:teuthology.orchestra.run.smithi167.stdout: "enabled": false, 2023-10-05T19:29:34.730 INFO:teuthology.orchestra.run.smithi167.stdout: "check_on_raw": false, 2023-10-05T19:29:34.730 INFO:teuthology.orchestra.run.smithi167.stdout: "max_size": -1, 2023-10-05T19:29:34.730 INFO:teuthology.orchestra.run.smithi167.stdout: "max_size_kb": 0, 2023-10-05T19:29:34.730 INFO:teuthology.orchestra.run.smithi167.stdout: "max_objects": -1 2023-10-05T19:29:34.730 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2023-10-05T19:29:34.730 INFO:teuthology.orchestra.run.smithi167.stdout: "temp_url_keys": [], 2023-10-05T19:29:34.730 INFO:teuthology.orchestra.run.smithi167.stdout: "type": "rgw", 2023-10-05T19:29:34.730 INFO:teuthology.orchestra.run.smithi167.stdout: "mfa_ids": [] 2023-10-05T19:29:34.730 INFO:teuthology.orchestra.run.smithi167.stdout:} 2023-10-05T19:29:34.731 INFO:teuthology.orchestra.run.smithi167.stdout: 2023-10-05T19:29:34.731 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.722+0000 7fa103018700 1 -- 172.21.15.167:0/4173470315 <== osd.0 v2:172.21.15.118:6800/27690 38 ==== watch-notify(notify_complete (2) cookie 94074889338816 notify 137438953504 ret 0) v3 ==== 42+0+48 (crc 0 0 0) 0x7fa0f404b5e0 con 0x7fa0fc011040 2023-10-05T19:29:34.731 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.722+0000 7fa104ce2e40 1 -- 172.21.15.167:0/4173470315 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:110 7.1f 7:f95f44c2:::notify.0:head [watch unwatch cookie 94074888797792] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e32) v8 -- 0x558f8451b5d0 con 0x558f84505d40 2023-10-05T19:29:34.731 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.726+0000 7fa102016700 1 -- 172.21.15.167:0/4173470315 <== osd.2 v2:172.21.15.167:6816/30037 17 ==== osd_op_reply(110 notify.0 [watch unwatch cookie 94074888797792] v32'242 uv240 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7fa0fc02f040 con 0x558f84505d40 2023-10-05T19:29:34.732 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.726+0000 7fa104ce2e40 20 remove_watcher() i=0 2023-10-05T19:29:34.732 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.726+0000 7fa104ce2e40 2 removed watcher, disabling cache 2023-10-05T19:29:34.732 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.726+0000 7fa104ce2e40 1 -- 172.21.15.167:0/4173470315 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:111 7.0 7:05bf5b68:::notify.1:head [watch unwatch cookie 94074888802416] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e32) v8 -- 0x558f845c8170 con 0x558f84538920 2023-10-05T19:29:34.733 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.730+0000 7fa103018700 1 -- 172.21.15.167:0/4173470315 <== osd.1 v2:172.21.15.167:6800/30035 37 ==== osd_op_reply(111 notify.1 [watch unwatch cookie 94074888802416] v32'242 uv240 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7fa0f4045040 con 0x558f84538920 2023-10-05T19:29:34.733 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.730+0000 7fa104ce2e40 20 remove_watcher() i=1 2023-10-05T19:29:34.734 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.730+0000 7fa104ce2e40 1 -- 172.21.15.167:0/4173470315 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:112 7.15 7:a93a5511:::notify.2:head [watch unwatch cookie 94074888804704] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e32) v8 -- 0x558f845c8170 con 0x558f8453eaa0 2023-10-05T19:29:34.736 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.730+0000 7fa102817700 1 -- 172.21.15.167:0/4173470315 <== osd.3 v2:172.21.15.167:6808/30036 24 ==== osd_op_reply(112 notify.2 [watch unwatch cookie 94074888804704] v32'242 uv240 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7fa0ec035070 con 0x558f8453eaa0 2023-10-05T19:29:34.736 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.730+0000 7fa104ce2e40 20 remove_watcher() i=2 2023-10-05T19:29:34.736 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.730+0000 7fa104ce2e40 1 -- 172.21.15.167:0/4173470315 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:113 7.e 7:7759931f:::notify.3:head [watch unwatch cookie 94074888810832] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e32) v8 -- 0x558f84545070 con 0x558f84505d40 2023-10-05T19:29:34.739 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.734+0000 7fa102016700 1 -- 172.21.15.167:0/4173470315 <== osd.2 v2:172.21.15.167:6816/30037 18 ==== osd_op_reply(113 notify.3 [watch unwatch cookie 94074888810832] v32'242 uv240 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7fa0fc02f040 con 0x558f84505d40 2023-10-05T19:29:34.739 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.734+0000 7fa104ce2e40 20 remove_watcher() i=3 2023-10-05T19:29:34.739 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.734+0000 7fa104ce2e40 1 -- 172.21.15.167:0/4173470315 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:114 7.d 7:b4812045:::notify.4:head [watch unwatch cookie 94074888805808] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e32) v8 -- 0x558f84545960 con 0x558f84538920 2023-10-05T19:29:34.742 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.738+0000 7fa103018700 1 -- 172.21.15.167:0/4173470315 <== osd.1 v2:172.21.15.167:6800/30035 38 ==== osd_op_reply(114 notify.4 [watch unwatch cookie 94074888805808] v32'242 uv240 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7fa0f405f040 con 0x558f84538920 2023-10-05T19:29:34.742 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.738+0000 7fa104ce2e40 20 remove_watcher() i=4 2023-10-05T19:29:34.742 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.738+0000 7fa104ce2e40 1 -- 172.21.15.167:0/4173470315 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:115 7.3 7:c609908c:::notify.5:head [watch unwatch cookie 94074888814704] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e32) v8 -- 0x558f84545e50 con 0x7fa0fc011040 2023-10-05T19:29:34.745 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.742+0000 7fa103018700 1 -- 172.21.15.167:0/4173470315 <== osd.0 v2:172.21.15.118:6800/27690 39 ==== osd_op_reply(115 notify.5 [watch unwatch cookie 94074888814704] v32'242 uv240 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7fa0f405f040 con 0x7fa0fc011040 2023-10-05T19:29:34.745 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.742+0000 7fa104ce2e40 20 remove_watcher() i=5 2023-10-05T19:29:34.745 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.742+0000 7fa104ce2e40 1 -- 172.21.15.167:0/4173470315 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:116 7.14 7:2b04a3e9:::notify.6:head [watch unwatch cookie 94074888815808] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e32) v8 -- 0x558f84545e50 con 0x7fa0fc011040 2023-10-05T19:29:34.748 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.742+0000 7fa103018700 1 -- 172.21.15.167:0/4173470315 <== osd.0 v2:172.21.15.118:6800/27690 40 ==== osd_op_reply(116 notify.6 [watch unwatch cookie 94074888815808] v32'242 uv240 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7fa0f405f040 con 0x7fa0fc011040 2023-10-05T19:29:34.748 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.742+0000 7fa104ce2e40 20 remove_watcher() i=6 2023-10-05T19:29:34.748 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.742+0000 7fa104ce2e40 1 -- 172.21.15.167:0/4173470315 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:117 7.9 7:93e5b521:::notify.7:head [watch unwatch cookie 94074888819168] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e32) v8 -- 0x558f84545e50 con 0x7fa0fc011040 2023-10-05T19:29:34.751 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.746+0000 7fa103018700 1 -- 172.21.15.167:0/4173470315 <== osd.0 v2:172.21.15.118:6800/27690 41 ==== osd_op_reply(117 notify.7 [watch unwatch cookie 94074888819168] v32'242 uv240 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7fa0f405f040 con 0x7fa0fc011040 2023-10-05T19:29:34.751 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.746+0000 7fa104ce2e40 20 remove_watcher() i=7 2023-10-05T19:29:34.753 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.750+0000 7fa04ffff700 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2023-10-05T19:29:34.753 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.750+0000 7fa104ce2e40 1 -- 172.21.15.167:0/4173470315 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:118 6.4 6:22d26bf9:::data_loggenerations_metadata:head [watch unwatch cookie 94074888821584] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e32) v8 -- 0x558f84548560 con 0x558f84538920 2023-10-05T19:29:34.756 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.750+0000 7fa103018700 1 -- 172.21.15.167:0/4173470315 <== osd.1 v2:172.21.15.167:6800/30035 39 ==== osd_op_reply(118 data_loggenerations_metadata [watch unwatch cookie 94074888821584] v32'234 uv1 ondisk = 0) v8 ==== 172+0+0 (crc 0 0 0) 0x7fa0f405f040 con 0x558f84538920 2023-10-05T19:29:34.756 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.750+0000 7fa104ce2e40 1 -- 172.21.15.167:0/4173470315 >> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] conn(0x7fa0fc011040 msgr2=0x7fa0fc014950 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-05T19:29:34.756 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.750+0000 7fa104ce2e40 1 --2- 172.21.15.167:0/4173470315 >> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] conn(0x7fa0fc011040 0x7fa0fc014950 crc :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-05T19:29:34.757 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.750+0000 7fa104ce2e40 1 -- 172.21.15.167:0/4173470315 >> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] conn(0x558f84538920 msgr2=0x558f8453ad00 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-05T19:29:34.757 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.750+0000 7fa104ce2e40 1 --2- 172.21.15.167:0/4173470315 >> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] conn(0x558f84538920 0x558f8453ad00 crc :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-05T19:29:34.757 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.750+0000 7fa104ce2e40 1 -- 172.21.15.167:0/4173470315 >> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] conn(0x558f84505d40 msgr2=0x7fa0ec01ca90 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-05T19:29:34.757 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.750+0000 7fa104ce2e40 1 --2- 172.21.15.167:0/4173470315 >> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] conn(0x558f84505d40 0x7fa0ec01ca90 crc :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-05T19:29:34.757 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.750+0000 7fa104ce2e40 1 -- 172.21.15.167:0/4173470315 >> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] conn(0x558f8453eaa0 msgr2=0x558f84540e80 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-05T19:29:34.757 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.750+0000 7fa104ce2e40 1 --2- 172.21.15.167:0/4173470315 >> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] conn(0x558f8453eaa0 0x558f84540e80 crc :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-05T19:29:34.757 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.750+0000 7fa104ce2e40 1 -- 172.21.15.167:0/4173470315 >> [v2:172.21.15.167:6824/29889,v1:172.21.15.167:6825/29889] conn(0x7fa0e4030ca0 msgr2=0x7fa0e4033150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-05T19:29:34.757 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.754+0000 7fa104ce2e40 1 --2- 172.21.15.167:0/4173470315 >> [v2:172.21.15.167:6824/29889,v1:172.21.15.167:6825/29889] conn(0x7fa0e4030ca0 0x7fa0e4033150 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7fa0fc0044f0 tx=0x7fa0fc006690 comp rx=0 tx=0).stop 2023-10-05T19:29:34.757 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.754+0000 7fa103018700 1 -- 172.21.15.167:0/4173470315 reap_dead start 2023-10-05T19:29:34.758 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.754+0000 7fa104ce2e40 1 -- 172.21.15.167:0/4173470315 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x558f84506650 msgr2=0x558f8452ee70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-05T19:29:34.758 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.754+0000 7fa104ce2e40 1 --2- 172.21.15.167:0/4173470315 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x558f84506650 0x558f8452ee70 secure :-1 s=READY pgs=617 cs=0 l=1 rev1=1 crypto rx=0x7fa0ec00afd0 tx=0x7fa0ec01c000 comp rx=0 tx=0).stop 2023-10-05T19:29:34.759 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.754+0000 7fa104ce2e40 1 -- 172.21.15.167:0/4173470315 shutdown_connections 2023-10-05T19:29:34.759 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.754+0000 7fa104ce2e40 1 --2- 172.21.15.167:0/4173470315 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x558f84506650 0x558f8452ee70 unknown :-1 s=CLOSED pgs=617 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-05T19:29:34.759 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.754+0000 7fa104ce2e40 1 -- 172.21.15.167:0/4173470315 >> 172.21.15.167:0/4173470315 conn(0x558f84501760 msgr2=0x558f8451f370 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-05T19:29:34.759 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.754+0000 7fa104ce2e40 1 -- 172.21.15.167:0/4173470315 shutdown_connections 2023-10-05T19:29:34.759 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.754+0000 7fa104ce2e40 1 -- 172.21.15.167:0/4173470315 wait complete. 2023-10-05T19:29:34.759 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.754+0000 7fa104ce2e40 1 -- 172.21.15.167:0/989994640 >> [v2:172.21.15.167:6824/29889,v1:172.21.15.167:6825/29889] conn(0x7fa040030c00 msgr2=0x7fa0400330b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-05T19:29:34.759 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.754+0000 7fa104ce2e40 1 --2- 172.21.15.167:0/989994640 >> [v2:172.21.15.167:6824/29889,v1:172.21.15.167:6825/29889] conn(0x7fa040030c00 0x7fa0400330b0 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x558f8454a0b0 tx=0x7fa0f4000f90 comp rx=0 tx=0).stop 2023-10-05T19:29:34.759 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.754+0000 7fa104ce2e40 1 -- 172.21.15.167:0/989994640 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x558f84557900 msgr2=0x558f8455d400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-05T19:29:34.759 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.754+0000 7fa104ce2e40 1 --2- 172.21.15.167:0/989994640 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x558f84557900 0x558f8455d400 secure :-1 s=READY pgs=619 cs=0 l=1 rev1=1 crypto rx=0x7fa0fc006040 tx=0x7fa0fc036000 comp rx=0 tx=0).stop 2023-10-05T19:29:34.759 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.754+0000 7fa104ce2e40 1 -- 172.21.15.167:0/989994640 shutdown_connections 2023-10-05T19:29:34.759 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.754+0000 7fa104ce2e40 1 --2- 172.21.15.167:0/989994640 >> [v2:172.21.15.167:6824/29889,v1:172.21.15.167:6825/29889] conn(0x7fa040030c00 0x7fa0400330b0 unknown :-1 s=CLOSED pgs=213 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-05T19:29:34.760 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.754+0000 7fa104ce2e40 1 --2- 172.21.15.167:0/989994640 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x558f84557900 0x558f8455d400 unknown :-1 s=CLOSED pgs=619 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-05T19:29:34.760 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.754+0000 7fa104ce2e40 1 -- 172.21.15.167:0/989994640 >> 172.21.15.167:0/989994640 conn(0x558f84552cc0 msgr2=0x558f84550d60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-05T19:29:34.760 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.754+0000 7fa104ce2e40 1 -- 172.21.15.167:0/989994640 shutdown_connections 2023-10-05T19:29:34.760 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.754+0000 7fa104ce2e40 1 -- 172.21.15.167:0/989994640 wait complete. 2023-10-05T19:29:34.763 DEBUG:tasks.util.rgw: json result: {'user_id': 'bar', 'display_name': 'Bar', 'email': '', 'suspended': 0, 'max_buckets': 4, 'subusers': [], 'keys': [{'user': 'bar', 'access_key': 'NX5QOQKC6BH2IDN8HC7A', 'secret_key': 'LnEsqNNqZIpkzauboDcLXLcYaWwLQ3Kop0zAnKIn'}], 'swift_keys': [], 'caps': [], 'op_mask': 'read, write, delete', 'default_placement': '', 'default_storage_class': '', 'placement_tags': [], 'bucket_quota': {'enabled': False, 'check_on_raw': False, 'max_size': -1, 'max_size_kb': 0, 'max_objects': -1}, 'user_quota': {'enabled': False, 'check_on_raw': False, 'max_size': -1, 'max_size_kb': 0, 'max_objects': -1}, 'temp_url_keys': [], 'type': 'rgw', 'mfa_ids': []} 2023-10-05T19:29:34.786 INFO:tasks.radosgw_admin:RL: create_bucket myfoo6 bar bytes_out=0 bytes_in=0 failed=False 2023-10-05T19:29:34.797 INFO:tasks.radosgw_admin:RL: put_obj myfoo6 bar bytes_out=8 bytes_in=0 failed=False 2023-10-05T19:29:34.797 INFO:tasks.util.rgw:rgwadmin: client.0 : ['user', 'rename', '--uid', 'bar', '--new-uid', 'bud'] 2023-10-05T19:29:34.797 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'ceph', 'user', 'rename', '--uid', 'bar', '--new-uid', 'bud'] 2023-10-05T19:29:34.797 DEBUG:teuthology.orchestra.run.smithi167:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster ceph user rename --uid bar --new-uid bud 2023-10-05T19:29:34.837 INFO:teuthology.orchestra.run.smithi167.stderr:ignoring --setuser ceph since I am not root 2023-10-05T19:29:34.837 INFO:teuthology.orchestra.run.smithi167.stderr:ignoring --setgroup ceph since I am not root 2023-10-05T19:29:34.838 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.834+0000 7fae89306e40 1 Processor -- start 2023-10-05T19:29:34.838 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.834+0000 7fae89306e40 1 -- start start 2023-10-05T19:29:34.838 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.834+0000 7fae89306e40 1 --2- >> v2:172.21.15.118:3300/0 conn(0x56000ae95d10 0x56000ae960e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-05T19:29:34.838 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.834+0000 7fae89306e40 1 -- --> v1:172.21.15.118:6789/0 -- auth(proto 0 26 bytes epoch 0) v1 -- 0x56000ade3bd0 con 0x56000ae96620 2023-10-05T19:29:34.839 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.834+0000 7fae89306e40 1 -- --> v2:172.21.15.118:3300/0 -- mon_getmap magic: 0 v1 -- 0x56000ada0ee0 con 0x56000ae95d10 2023-10-05T19:29:34.839 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.834+0000 7fae86e3b700 1 --2- >> v2:172.21.15.118:3300/0 conn(0x56000ae95d10 0x56000ae960e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-05T19:29:34.839 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.834+0000 7fae86e3b700 1 --2- >> v2:172.21.15.118:3300/0 conn(0x56000ae95d10 0x56000ae960e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.118:3300/0 says I am v2:172.21.15.167:56744/0 (socket says 172.21.15.167:56744) 2023-10-05T19:29:34.839 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.834+0000 7fae86e3b700 1 -- 172.21.15.167:0/1779181816 learned_addr learned my addr 172.21.15.167:0/1779181816 (peer_addr_for_me v2:172.21.15.167:0/0) 2023-10-05T19:29:34.840 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.834+0000 7fae86e3b700 1 -- 172.21.15.167:0/1779181816 >> v1:172.21.15.118:6789/0 conn(0x56000ae96620 legacy=0x56000ae970c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-05T19:29:34.840 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.834+0000 7fae86e3b700 1 -- 172.21.15.167:0/1779181816 --> v2:172.21.15.118:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x56000ae977d0 con 0x56000ae95d10 2023-10-05T19:29:34.841 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.834+0000 7fae86e3b700 1 --2- 172.21.15.167:0/1779181816 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x56000ae95d10 0x56000ae960e0 secure :-1 s=READY pgs=621 cs=0 l=1 rev1=1 crypto rx=0x7fae8000a5c0 tx=0x7fae80005e00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2661068ebe677b5b server_cookie=0 in_seq=0 out_seq=0 2023-10-05T19:29:34.841 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.834+0000 7fae85e39700 1 -- 172.21.15.167:0/1779181816 <== mon.0 v2:172.21.15.118:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fae8000e510 con 0x56000ae95d10 2023-10-05T19:29:34.841 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.834+0000 7fae85e39700 1 -- 172.21.15.167:0/1779181816 <== mon.0 v2:172.21.15.118:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fae8000e670 con 0x56000ae95d10 2023-10-05T19:29:34.841 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.834+0000 7fae85e39700 1 -- 172.21.15.167:0/1779181816 <== mon.0 v2:172.21.15.118:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fae8000e920 con 0x56000ae95d10 2023-10-05T19:29:34.841 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.838+0000 7fae89306e40 1 -- 172.21.15.167:0/1779181816 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x56000ae95d10 msgr2=0x56000ae960e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-05T19:29:34.841 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.838+0000 7fae89306e40 1 --2- 172.21.15.167:0/1779181816 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x56000ae95d10 0x56000ae960e0 secure :-1 s=READY pgs=621 cs=0 l=1 rev1=1 crypto rx=0x7fae8000a5c0 tx=0x7fae80005e00 comp rx=0 tx=0).stop 2023-10-05T19:29:34.842 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.838+0000 7fae89306e40 1 -- 172.21.15.167:0/1779181816 shutdown_connections 2023-10-05T19:29:34.842 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.838+0000 7fae89306e40 1 --2- 172.21.15.167:0/1779181816 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x56000ae95d10 0x56000ae960e0 unknown :-1 s=CLOSED pgs=621 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-05T19:29:34.842 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.838+0000 7fae89306e40 1 -- 172.21.15.167:0/1779181816 >> 172.21.15.167:0/1779181816 conn(0x56000ae91730 msgr2=0x56000ae93b40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-05T19:29:34.842 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.838+0000 7fae89306e40 1 -- 172.21.15.167:0/1779181816 shutdown_connections 2023-10-05T19:29:34.842 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.838+0000 7fae89306e40 1 -- 172.21.15.167:0/1779181816 wait complete. 2023-10-05T19:29:34.844 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.838+0000 7fae89306e40 1 Processor -- start 2023-10-05T19:29:34.844 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.838+0000 7fae89306e40 1 -- start start 2023-10-05T19:29:34.844 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.838+0000 7fae89306e40 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x56000ae95d10 0x56000aeb4e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-05T19:29:34.844 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.838+0000 7fae89306e40 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x56000ad64e30 con 0x56000ae95d10 2023-10-05T19:29:34.844 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.838+0000 7fae86e3b700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x56000ae95d10 0x56000aeb4e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-05T19:29:34.844 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.838+0000 7fae86e3b700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x56000ae95d10 0x56000aeb4e30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.118:3300/0 says I am v2:172.21.15.167:56756/0 (socket says 172.21.15.167:56756) 2023-10-05T19:29:34.845 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.838+0000 7fae86e3b700 1 -- 172.21.15.167:0/3267989995 learned_addr learned my addr 172.21.15.167:0/3267989995 (peer_addr_for_me v2:172.21.15.167:0/0) 2023-10-05T19:29:34.845 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.838+0000 7fae86e3b700 1 -- 172.21.15.167:0/3267989995 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fae8000a180 con 0x56000ae95d10 2023-10-05T19:29:34.845 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.838+0000 7fae86e3b700 1 --2- 172.21.15.167:0/3267989995 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x56000ae95d10 0x56000aeb4e30 secure :-1 s=READY pgs=622 cs=0 l=1 rev1=1 crypto rx=0x7fae8000ad10 tx=0x7fae80013000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7d279aad719b4746 server_cookie=0 in_seq=0 out_seq=0 2023-10-05T19:29:34.845 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.838+0000 7fae84e37700 1 -- 172.21.15.167:0/3267989995 <== mon.0 v2:172.21.15.118:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fae800032f0 con 0x56000ae95d10 2023-10-05T19:29:34.845 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.838+0000 7fae84e37700 1 -- 172.21.15.167:0/3267989995 <== mon.0 v2:172.21.15.118:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fae8000e7e0 con 0x56000ae95d10 2023-10-05T19:29:34.845 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.838+0000 7fae84e37700 1 -- 172.21.15.167:0/3267989995 <== mon.0 v2:172.21.15.118:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fae8000eab0 con 0x56000ae95d10 2023-10-05T19:29:34.845 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.838+0000 7fae89306e40 1 -- 172.21.15.167:0/3267989995 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x56000ae95d10 msgr2=0x56000aeb4e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-05T19:29:34.845 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.838+0000 7fae89306e40 1 --2- 172.21.15.167:0/3267989995 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x56000ae95d10 0x56000aeb4e30 secure :-1 s=READY pgs=622 cs=0 l=1 rev1=1 crypto rx=0x7fae8000ad10 tx=0x7fae80013000 comp rx=0 tx=0).stop 2023-10-05T19:29:34.845 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.838+0000 7fae89306e40 1 -- 172.21.15.167:0/3267989995 shutdown_connections 2023-10-05T19:29:34.845 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.838+0000 7fae89306e40 1 --2- 172.21.15.167:0/3267989995 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x56000ae95d10 0x56000aeb4e30 unknown :-1 s=CLOSED pgs=622 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-05T19:29:34.845 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.838+0000 7fae89306e40 1 -- 172.21.15.167:0/3267989995 >> 172.21.15.167:0/3267989995 conn(0x56000ae91730 msgr2=0x56000aeb09f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-05T19:29:34.846 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.838+0000 7fae89306e40 1 -- 172.21.15.167:0/3267989995 shutdown_connections 2023-10-05T19:29:34.846 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.838+0000 7fae89306e40 1 -- 172.21.15.167:0/3267989995 wait complete. 2023-10-05T19:29:34.847 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.842+0000 7fae89306e40 1 Processor -- start 2023-10-05T19:29:34.847 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.842+0000 7fae89306e40 1 -- start start 2023-10-05T19:29:34.847 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.842+0000 7fae89306e40 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x56000ae96620 0x56000aeb9ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-05T19:29:34.847 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.842+0000 7fae89306e40 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x56000ad37ed0 con 0x56000ae96620 2023-10-05T19:29:34.847 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.842+0000 7fae86e3b700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x56000ae96620 0x56000aeb9ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-05T19:29:34.848 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.842+0000 7fae86e3b700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x56000ae96620 0x56000aeb9ad0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.118:3300/0 says I am v2:172.21.15.167:56766/0 (socket says 172.21.15.167:56766) 2023-10-05T19:29:34.848 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.842+0000 7fae86e3b700 1 -- 172.21.15.167:0/3844986762 learned_addr learned my addr 172.21.15.167:0/3844986762 (peer_addr_for_me v2:172.21.15.167:0/0) 2023-10-05T19:29:34.848 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.842+0000 7fae86e3b700 1 -- 172.21.15.167:0/3844986762 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x56000ae977d0 con 0x56000ae96620 2023-10-05T19:29:34.848 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.842+0000 7fae86e3b700 1 --2- 172.21.15.167:0/3844986762 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x56000ae96620 0x56000aeb9ad0 secure :-1 s=READY pgs=623 cs=0 l=1 rev1=1 crypto rx=0x7fae8001e040 tx=0x7fae8001c000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-05T19:29:34.848 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.842+0000 7fae7effd700 1 -- 172.21.15.167:0/3844986762 <== mon.0 v2:172.21.15.118:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fae8001c9f0 con 0x56000ae96620 2023-10-05T19:29:34.848 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.842+0000 7fae7effd700 1 -- 172.21.15.167:0/3844986762 <== mon.0 v2:172.21.15.118:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fae8001cb70 con 0x56000ae96620 2023-10-05T19:29:34.848 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.842+0000 7fae89306e40 1 -- 172.21.15.167:0/3844986762 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x56000aeba010 con 0x56000ae96620 2023-10-05T19:29:34.848 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.842+0000 7fae7effd700 1 -- 172.21.15.167:0/3844986762 <== mon.0 v2:172.21.15.118:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fae8001ce60 con 0x56000ae96620 2023-10-05T19:29:34.848 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.842+0000 7fae89306e40 1 -- 172.21.15.167:0/3844986762 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x56000aeb83d0 con 0x56000ae96620 2023-10-05T19:29:34.849 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.842+0000 7fae89306e40 20 rados->read ofs=0 len=0 2023-10-05T19:29:34.849 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.842+0000 7fae7effd700 1 -- 172.21.15.167:0/3844986762 <== mon.0 v2:172.21.15.118:3300/0 4 ==== mgrmap(e 5) v1 ==== 40329+0+0 (secure 0 0 0) 0x7fae80010440 con 0x56000ae96620 2023-10-05T19:29:34.849 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.846+0000 7fae7effd700 1 --2- 172.21.15.167:0/3844986762 >> [v2:172.21.15.167:6824/29889,v1:172.21.15.167:6825/29889] conn(0x7fae68030ca0 0x7fae68033150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-05T19:29:34.850 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.846+0000 7fae7effd700 1 -- 172.21.15.167:0/3844986762 <== mon.0 v2:172.21.15.118:3300/0 5 ==== osd_map(32..32 src has 1..32) v4 ==== 6260+0+0 (secure 0 0 0) 0x7fae80046380 con 0x56000ae96620 2023-10-05T19:29:34.850 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.846+0000 7fae8663a700 1 --2- 172.21.15.167:0/3844986762 >> [v2:172.21.15.167:6824/29889,v1:172.21.15.167:6825/29889] conn(0x7fae68030ca0 0x7fae68033150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-05T19:29:34.850 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.846+0000 7fae89306e40 1 --2- 172.21.15.167:0/3844986762 >> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] conn(0x56000aec89a0 0x56000aecad80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-05T19:29:34.850 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.846+0000 7fae89306e40 1 -- 172.21.15.167:0/3844986762 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:1 5.12 5:49953fa1:::default.realm:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32) v8 -- 0x56000aecb430 con 0x56000aec89a0 2023-10-05T19:29:34.850 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.846+0000 7fae8763c700 1 --2- 172.21.15.167:0/3844986762 >> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] conn(0x56000aec89a0 0x56000aecad80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-05T19:29:34.851 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.846+0000 7fae8663a700 1 --2- 172.21.15.167:0/3844986762 >> [v2:172.21.15.167:6824/29889,v1:172.21.15.167:6825/29889] conn(0x7fae68030ca0 0x7fae68033150 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7fae7000b2b0 tx=0x7fae70001490 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-05T19:29:34.851 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.846+0000 7fae8763c700 1 --2- 172.21.15.167:0/3844986762 >> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] conn(0x56000aec89a0 0x56000aecad80 crc :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-05T19:29:34.851 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.846+0000 7fae8763c700 1 -- 172.21.15.167:0/3844986762 <== osd.1 v2:172.21.15.167:6800/30035 1 ==== osd_op_reply(1 default.realm [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 ==== 157+0+0 (crc 0 0 0) 0x7fae7800a500 con 0x56000aec89a0 2023-10-05T19:29:34.851 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.846+0000 7fae89306e40 20 rados_obj.operate() r=-2 bl.length=0 2023-10-05T19:29:34.851 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.846+0000 7fae89306e40 20 realm 2023-10-05T19:29:34.852 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.846+0000 7fae89306e40 20 rados->read ofs=0 len=0 2023-10-05T19:29:34.852 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.846+0000 7fae89306e40 1 -- 172.21.15.167:0/3844986762 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:2 5.12 5:49953fa1:::default.realm:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32) v8 -- 0x56000aec8010 con 0x56000aec89a0 2023-10-05T19:29:34.853 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.846+0000 7fae8763c700 1 -- 172.21.15.167:0/3844986762 <== osd.1 v2:172.21.15.167:6800/30035 2 ==== osd_op_reply(2 default.realm [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 ==== 157+0+0 (crc 0 0 0) 0x7fae7800a500 con 0x56000aec89a0 2023-10-05T19:29:34.853 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.846+0000 7fae89306e40 20 rados_obj.operate() r=-2 bl.length=0 2023-10-05T19:29:34.853 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.846+0000 7fae89306e40 4 RGWPeriod::init failed to init realm id : (2) No such file or directory 2023-10-05T19:29:34.853 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.846+0000 7fae89306e40 20 rados->read ofs=0 len=0 2023-10-05T19:29:34.853 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.846+0000 7fae89306e40 1 -- 172.21.15.167:0/3844986762 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:3 5.12 5:49953fa1:::default.realm:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32) v8 -- 0x56000aecbf10 con 0x56000aec89a0 2023-10-05T19:29:34.854 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.846+0000 7fae8763c700 1 -- 172.21.15.167:0/3844986762 <== osd.1 v2:172.21.15.167:6800/30035 3 ==== osd_op_reply(3 default.realm [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 ==== 157+0+0 (crc 0 0 0) 0x7fae7800a500 con 0x56000aec89a0 2023-10-05T19:29:34.854 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.846+0000 7fae89306e40 20 rados_obj.operate() r=-2 bl.length=0 2023-10-05T19:29:34.854 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.846+0000 7fae89306e40 20 rados->read ofs=0 len=0 2023-10-05T19:29:34.854 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.846+0000 7fae89306e40 1 -- 172.21.15.167:0/3844986762 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:4 5.f 5:f43b6ece:::zone_names.default:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32) v8 -- 0x56000aecbb20 con 0x56000aec89a0 2023-10-05T19:29:34.854 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.850+0000 7fae8763c700 1 -- 172.21.15.167:0/3844986762 <== osd.1 v2:172.21.15.167:6800/30035 4 ==== osd_op_reply(4 zone_names.default [read 0~46 out=46b] v0'0 uv1 ondisk = 0) v8 ==== 162+0+46 (crc 0 0 0) 0x7fae7800a500 con 0x56000aec89a0 2023-10-05T19:29:34.854 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.850+0000 7fae89306e40 20 rados_obj.operate() r=0 bl.length=46 2023-10-05T19:29:34.854 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.850+0000 7fae89306e40 20 rados->read ofs=0 len=0 2023-10-05T19:29:34.854 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.850+0000 7fae89306e40 1 -- 172.21.15.167:0/3844986762 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:5 5.1d 5:b8374eb4:::zone_info.9a9df965-6612-45d4-a45f-6e3c889bacc2:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32) v8 -- 0x56000aeccda0 con 0x56000aec89a0 2023-10-05T19:29:34.855 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.850+0000 7fae8763c700 1 -- 172.21.15.167:0/3844986762 <== osd.1 v2:172.21.15.167:6800/30035 5 ==== osd_op_reply(5 zone_info.9a9df965-6612-45d4-a45f-6e3c889bacc2 [read 0~874 out=874b] v0'0 uv2 ondisk = 0) v8 ==== 190+0+874 (crc 0 0 0) 0x7fae7800a500 con 0x56000aec89a0 2023-10-05T19:29:34.855 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.850+0000 7fae89306e40 20 rados_obj.operate() r=0 bl.length=874 2023-10-05T19:29:34.855 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.850+0000 7fae89306e40 20 searching for the correct realm 2023-10-05T19:29:34.855 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.850+0000 7fae89306e40 1 --2- 172.21.15.167:0/3844986762 >> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] conn(0x56000aeceb60 0x56000aed0f40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-05T19:29:34.856 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.850+0000 7fae89306e40 1 -- 172.21.15.167:0/3844986762 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:6 5.0 5:00000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x56000aed1480 con 0x56000aeceb60 2023-10-05T19:29:34.856 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.850+0000 7fae86e3b700 1 --2- 172.21.15.167:0/3844986762 >> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] conn(0x56000aeceb60 0x56000aed0f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-05T19:29:34.856 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.850+0000 7fae86e3b700 1 --2- 172.21.15.167:0/3844986762 >> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] conn(0x56000aeceb60 0x56000aed0f40 crc :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-05T19:29:34.856 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.850+0000 7fae86e3b700 1 -- 172.21.15.167:0/3844986762 <== osd.3 v2:172.21.15.167:6808/30036 1 ==== osd_op_reply(6 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fae8002b070 con 0x56000aeceb60 2023-10-05T19:29:34.857 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.850+0000 7fae86e3b700 1 --2- 172.21.15.167:0/3844986762 >> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] conn(0x56000ae95d10 0x7fae80025400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-05T19:29:34.857 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.850+0000 7fae86e3b700 1 -- 172.21.15.167:0/3844986762 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:7 5.10 5:08000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x56000aed1480 con 0x56000ae95d10 2023-10-05T19:29:34.857 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.850+0000 7fae8663a700 1 --2- 172.21.15.167:0/3844986762 >> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] conn(0x56000ae95d10 0x7fae80025400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-05T19:29:34.857 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.854+0000 7fae8663a700 1 --2- 172.21.15.167:0/3844986762 >> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] conn(0x56000ae95d10 0x7fae80025400 crc :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-05T19:29:34.857 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.854+0000 7fae8663a700 1 -- 172.21.15.167:0/3844986762 <== osd.2 v2:172.21.15.167:6816/30037 1 ==== osd_op_reply(7 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fae7000bc70 con 0x56000ae95d10 2023-10-05T19:29:34.858 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.854+0000 7fae8663a700 1 --2- 172.21.15.167:0/3844986762 >> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] conn(0x7fae7000f8d0 0x7fae7000e360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-05T19:29:34.858 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.854+0000 7fae8663a700 1 -- 172.21.15.167:0/3844986762 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:8 5.8 5:10000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x56000aed1480 con 0x7fae7000f8d0 2023-10-05T19:29:34.858 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.854+0000 7fae8763c700 1 --2- 172.21.15.167:0/3844986762 >> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] conn(0x7fae7000f8d0 0x7fae7000e360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-05T19:29:34.858 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.854+0000 7fae8763c700 1 --2- 172.21.15.167:0/3844986762 >> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] conn(0x7fae7000f8d0 0x7fae7000e360 crc :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-05T19:29:34.859 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.854+0000 7fae8763c700 1 -- 172.21.15.167:0/3844986762 <== osd.0 v2:172.21.15.118:6800/27690 1 ==== osd_op_reply(8 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fae78014040 con 0x7fae7000f8d0 2023-10-05T19:29:34.860 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.854+0000 7fae8763c700 1 -- 172.21.15.167:0/3844986762 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:9 5.18 5:18000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x56000aed1480 con 0x56000aec89a0 2023-10-05T19:29:34.860 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.854+0000 7fae8763c700 1 -- 172.21.15.167:0/3844986762 <== osd.1 v2:172.21.15.167:6800/30035 6 ==== osd_op_reply(9 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fae78014040 con 0x56000aec89a0 2023-10-05T19:29:34.860 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.854+0000 7fae8763c700 1 -- 172.21.15.167:0/3844986762 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:10 5.4 5:20000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x56000aed1480 con 0x56000aec89a0 2023-10-05T19:29:34.861 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.858+0000 7fae8763c700 1 -- 172.21.15.167:0/3844986762 <== osd.1 v2:172.21.15.167:6800/30035 7 ==== osd_op_reply(10 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fae78014040 con 0x56000aec89a0 2023-10-05T19:29:34.861 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.858+0000 7fae8763c700 1 -- 172.21.15.167:0/3844986762 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:11 5.14 5:28000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x56000aed1480 con 0x7fae7000f8d0 2023-10-05T19:29:34.862 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.858+0000 7fae8763c700 1 -- 172.21.15.167:0/3844986762 <== osd.0 v2:172.21.15.118:6800/27690 2 ==== osd_op_reply(11 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fae78014040 con 0x7fae7000f8d0 2023-10-05T19:29:34.862 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.858+0000 7fae8763c700 1 -- 172.21.15.167:0/3844986762 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:12 5.c 5:30000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x56000aed1480 con 0x7fae7000f8d0 2023-10-05T19:29:34.863 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.858+0000 7fae8763c700 1 -- 172.21.15.167:0/3844986762 <== osd.0 v2:172.21.15.118:6800/27690 3 ==== osd_op_reply(12 [pgnls start_epoch 32 out=112b] v23'1 uv1 ondisk = 1) v8 ==== 144+0+112 (crc 0 0 0) 0x7fae78014040 con 0x7fae7000f8d0 2023-10-05T19:29:34.863 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.858+0000 7fae8763c700 1 -- 172.21.15.167:0/3844986762 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:13 5.1c 5:38000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x56000aed1480 con 0x56000ae95d10 2023-10-05T19:29:34.864 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.858+0000 7fae8663a700 1 -- 172.21.15.167:0/3844986762 <== osd.2 v2:172.21.15.167:6816/30037 2 ==== osd_op_reply(13 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fae7000bc70 con 0x56000ae95d10 2023-10-05T19:29:34.864 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.858+0000 7fae8663a700 1 -- 172.21.15.167:0/3844986762 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:14 5.2 5:40000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x56000aed1480 con 0x56000aeceb60 2023-10-05T19:29:34.864 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.858+0000 7fae86e3b700 1 -- 172.21.15.167:0/3844986762 <== osd.3 v2:172.21.15.167:6808/30036 2 ==== osd_op_reply(14 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fae8002b070 con 0x56000aeceb60 2023-10-05T19:29:34.864 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.858+0000 7fae86e3b700 1 -- 172.21.15.167:0/3844986762 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:15 5.12 5:48000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x56000aed1480 con 0x56000aec89a0 2023-10-05T19:29:34.865 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.862+0000 7fae8763c700 1 -- 172.21.15.167:0/3844986762 <== osd.1 v2:172.21.15.167:6800/30035 8 ==== osd_op_reply(15 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fae78014040 con 0x56000aec89a0 2023-10-05T19:29:34.865 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.862+0000 7fae8763c700 1 -- 172.21.15.167:0/3844986762 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:16 5.a 5:50000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x56000aed1480 con 0x7fae7000f8d0 2023-10-05T19:29:34.866 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.862+0000 7fae8763c700 1 -- 172.21.15.167:0/3844986762 <== osd.0 v2:172.21.15.118:6800/27690 4 ==== osd_op_reply(16 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fae78022040 con 0x7fae7000f8d0 2023-10-05T19:29:34.866 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.862+0000 7fae8763c700 1 -- 172.21.15.167:0/3844986762 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:17 5.1a 5:58000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x56000aed1480 con 0x56000aec89a0 2023-10-05T19:29:34.867 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.862+0000 7fae8763c700 1 -- 172.21.15.167:0/3844986762 <== osd.1 v2:172.21.15.167:6800/30035 9 ==== osd_op_reply(17 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fae78022040 con 0x56000aec89a0 2023-10-05T19:29:34.867 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.862+0000 7fae8763c700 1 -- 172.21.15.167:0/3844986762 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:18 5.6 5:60000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x56000aed1480 con 0x56000ae95d10 2023-10-05T19:29:34.868 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.862+0000 7fae8663a700 1 -- 172.21.15.167:0/3844986762 <== osd.2 v2:172.21.15.167:6816/30037 3 ==== osd_op_reply(18 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fae7000bc70 con 0x56000ae95d10 2023-10-05T19:29:34.868 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.862+0000 7fae8663a700 1 -- 172.21.15.167:0/3844986762 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:19 5.16 5:68000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x56000aed1480 con 0x56000aeceb60 2023-10-05T19:29:34.869 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.866+0000 7fae86e3b700 1 -- 172.21.15.167:0/3844986762 <== osd.3 v2:172.21.15.167:6808/30036 3 ==== osd_op_reply(19 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fae8002b070 con 0x56000aeceb60 2023-10-05T19:29:34.869 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.866+0000 7fae86e3b700 1 -- 172.21.15.167:0/3844986762 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:20 5.e 5:70000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x56000aed1480 con 0x56000aeceb60 2023-10-05T19:29:34.870 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.866+0000 7fae86e3b700 1 -- 172.21.15.167:0/3844986762 <== osd.3 v2:172.21.15.167:6808/30036 4 ==== osd_op_reply(20 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fae8002b070 con 0x56000aeceb60 2023-10-05T19:29:34.870 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.866+0000 7fae86e3b700 1 -- 172.21.15.167:0/3844986762 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:21 5.1e 5:78000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x56000aed1480 con 0x7fae7000f8d0 2023-10-05T19:29:34.871 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.866+0000 7fae8763c700 1 -- 172.21.15.167:0/3844986762 <== osd.0 v2:172.21.15.118:6800/27690 5 ==== osd_op_reply(21 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fae78022040 con 0x7fae7000f8d0 2023-10-05T19:29:34.871 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.866+0000 7fae8763c700 1 -- 172.21.15.167:0/3844986762 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:22 5.1 5:80000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x56000aed1480 con 0x56000aeceb60 2023-10-05T19:29:34.871 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.866+0000 7fae86e3b700 1 -- 172.21.15.167:0/3844986762 <== osd.3 v2:172.21.15.167:6808/30036 5 ==== osd_op_reply(22 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fae8002b070 con 0x56000aeceb60 2023-10-05T19:29:34.872 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.866+0000 7fae86e3b700 1 -- 172.21.15.167:0/3844986762 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:23 5.11 5:88000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x56000aed1480 con 0x56000aeceb60 2023-10-05T19:29:34.872 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.866+0000 7fae86e3b700 1 -- 172.21.15.167:0/3844986762 <== osd.3 v2:172.21.15.167:6808/30036 6 ==== osd_op_reply(23 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fae8002b070 con 0x56000aeceb60 2023-10-05T19:29:34.872 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.866+0000 7fae86e3b700 1 -- 172.21.15.167:0/3844986762 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:24 5.9 5:90000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x56000aed1480 con 0x56000aec89a0 2023-10-05T19:29:34.873 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.870+0000 7fae8763c700 1 -- 172.21.15.167:0/3844986762 <== osd.1 v2:172.21.15.167:6800/30035 10 ==== osd_op_reply(24 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fae78022040 con 0x56000aec89a0 2023-10-05T19:29:34.873 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.870+0000 7fae8763c700 1 -- 172.21.15.167:0/3844986762 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:25 5.19 5:98000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x56000aed1480 con 0x56000aec89a0 2023-10-05T19:29:34.873 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.870+0000 7fae8763c700 1 -- 172.21.15.167:0/3844986762 <== osd.1 v2:172.21.15.167:6800/30035 11 ==== osd_op_reply(25 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fae78022040 con 0x56000aec89a0 2023-10-05T19:29:34.873 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.870+0000 7fae8763c700 1 -- 172.21.15.167:0/3844986762 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:26 5.5 5:a0000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x56000aed1480 con 0x7fae7000f8d0 2023-10-05T19:29:34.875 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.870+0000 7fae8763c700 1 -- 172.21.15.167:0/3844986762 <== osd.0 v2:172.21.15.118:6800/27690 6 ==== osd_op_reply(26 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fae78022040 con 0x7fae7000f8d0 2023-10-05T19:29:34.875 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.870+0000 7fae8763c700 1 -- 172.21.15.167:0/3844986762 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:27 5.15 5:a8000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x56000aed1480 con 0x56000aeceb60 2023-10-05T19:29:34.875 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.870+0000 7fae86e3b700 1 -- 172.21.15.167:0/3844986762 <== osd.3 v2:172.21.15.167:6808/30036 7 ==== osd_op_reply(27 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fae8002b070 con 0x56000aeceb60 2023-10-05T19:29:34.875 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.870+0000 7fae86e3b700 1 -- 172.21.15.167:0/3844986762 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:28 5.d 5:b0000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x56000aed1480 con 0x56000ae95d10 2023-10-05T19:29:34.876 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.870+0000 7fae8663a700 1 -- 172.21.15.167:0/3844986762 <== osd.2 v2:172.21.15.167:6816/30037 4 ==== osd_op_reply(28 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fae7000bc70 con 0x56000ae95d10 2023-10-05T19:29:34.876 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.870+0000 7fae8663a700 1 -- 172.21.15.167:0/3844986762 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:29 5.1d 5:b8000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x56000aed1480 con 0x56000aec89a0 2023-10-05T19:29:34.876 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.874+0000 7fae8763c700 1 -- 172.21.15.167:0/3844986762 <== osd.1 v2:172.21.15.167:6800/30035 12 ==== osd_op_reply(29 [pgnls start_epoch 32 out=107b] v30'2 uv2 ondisk = 1) v8 ==== 144+0+107 (crc 0 0 0) 0x7fae78022040 con 0x56000aec89a0 2023-10-05T19:29:34.877 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.874+0000 7fae8763c700 1 -- 172.21.15.167:0/3844986762 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:30 5.3 5:c0000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x56000aed1480 con 0x7fae7000f8d0 2023-10-05T19:29:34.877 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.874+0000 7fae8763c700 1 -- 172.21.15.167:0/3844986762 <== osd.0 v2:172.21.15.118:6800/27690 7 ==== osd_op_reply(30 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fae78022040 con 0x7fae7000f8d0 2023-10-05T19:29:34.877 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.874+0000 7fae8763c700 1 -- 172.21.15.167:0/3844986762 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:31 5.13 5:c8000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x56000aed1480 con 0x56000aeceb60 2023-10-05T19:29:34.878 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.874+0000 7fae86e3b700 1 -- 172.21.15.167:0/3844986762 <== osd.3 v2:172.21.15.167:6808/30036 8 ==== osd_op_reply(31 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fae8002b070 con 0x56000aeceb60 2023-10-05T19:29:34.878 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.874+0000 7fae86e3b700 1 -- 172.21.15.167:0/3844986762 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:32 5.b 5:d0000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x56000aed1480 con 0x7fae7000f8d0 2023-10-05T19:29:34.879 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.874+0000 7fae8763c700 1 -- 172.21.15.167:0/3844986762 <== osd.0 v2:172.21.15.118:6800/27690 8 ==== osd_op_reply(32 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fae78022040 con 0x7fae7000f8d0 2023-10-05T19:29:34.879 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.874+0000 7fae8763c700 1 -- 172.21.15.167:0/3844986762 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:33 5.1b 5:d8000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x56000aed1480 con 0x56000ae95d10 2023-10-05T19:29:34.879 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.874+0000 7fae8663a700 1 -- 172.21.15.167:0/3844986762 <== osd.2 v2:172.21.15.167:6816/30037 5 ==== osd_op_reply(33 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fae7000bc70 con 0x56000ae95d10 2023-10-05T19:29:34.880 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.874+0000 7fae8663a700 1 -- 172.21.15.167:0/3844986762 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:34 5.7 5:e0000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x56000aed1480 con 0x56000aeceb60 2023-10-05T19:29:34.880 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.874+0000 7fae86e3b700 1 -- 172.21.15.167:0/3844986762 <== osd.3 v2:172.21.15.167:6808/30036 9 ==== osd_op_reply(34 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fae8002b070 con 0x56000aeceb60 2023-10-05T19:29:34.880 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.874+0000 7fae86e3b700 1 -- 172.21.15.167:0/3844986762 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:35 5.17 5:e8000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x56000aed1480 con 0x7fae7000f8d0 2023-10-05T19:29:34.881 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.878+0000 7fae8763c700 1 -- 172.21.15.167:0/3844986762 <== osd.0 v2:172.21.15.118:6800/27690 9 ==== osd_op_reply(35 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fae78022040 con 0x7fae7000f8d0 2023-10-05T19:29:34.881 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.878+0000 7fae8763c700 1 -- 172.21.15.167:0/3844986762 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:36 5.f 5:f0000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x56000aed1480 con 0x56000aec89a0 2023-10-05T19:29:34.882 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.878+0000 7fae8763c700 1 -- 172.21.15.167:0/3844986762 <== osd.1 v2:172.21.15.167:6800/30035 13 ==== osd_op_reply(36 [pgnls start_epoch 32 out=115b] v23'2 uv2 ondisk = 1) v8 ==== 144+0+115 (crc 0 0 0) 0x7fae7802f080 con 0x56000aec89a0 2023-10-05T19:29:34.882 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.878+0000 7fae8763c700 1 -- 172.21.15.167:0/3844986762 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:37 5.1f 5:f8000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x56000aed1480 con 0x56000ae95d10 2023-10-05T19:29:34.883 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.878+0000 7fae8663a700 1 -- 172.21.15.167:0/3844986762 <== osd.2 v2:172.21.15.167:6816/30037 6 ==== osd_op_reply(37 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fae7000bc70 con 0x56000ae95d10 2023-10-05T19:29:34.883 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.878+0000 7fae89306e40 20 RGWRados::pool_iterate: got zonegroup_info.4f1f719d-9648-417b-b10c-e5d939825419 2023-10-05T19:29:34.883 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.878+0000 7fae89306e40 20 RGWRados::pool_iterate: got zone_info.9a9df965-6612-45d4-a45f-6e3c889bacc2 2023-10-05T19:29:34.883 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.878+0000 7fae89306e40 20 RGWRados::pool_iterate: got zone_names.default 2023-10-05T19:29:34.883 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.878+0000 7fae89306e40 20 RGWRados::pool_iterate: got zonegroups_names.default 2023-10-05T19:29:34.883 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.878+0000 7fae89306e40 20 rados->read ofs=0 len=0 2023-10-05T19:29:34.883 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.878+0000 7fae89306e40 1 -- 172.21.15.167:0/3844986762 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:38 5.12 5:49953fa1:::default.realm:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32) v8 -- 0x56000aecd890 con 0x56000aec89a0 2023-10-05T19:29:34.884 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.878+0000 7fae8763c700 1 -- 172.21.15.167:0/3844986762 <== osd.1 v2:172.21.15.167:6800/30035 14 ==== osd_op_reply(38 default.realm [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 ==== 157+0+0 (crc 0 0 0) 0x7fae7802f080 con 0x56000aec89a0 2023-10-05T19:29:34.884 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.878+0000 7fae89306e40 20 rados_obj.operate() r=-2 bl.length=0 2023-10-05T19:29:34.884 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.878+0000 7fae89306e40 20 rados->read ofs=0 len=0 2023-10-05T19:29:34.884 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.878+0000 7fae89306e40 1 -- 172.21.15.167:0/3844986762 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:39 5.f 5:f4c53578:::zonegroups_names.default:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32) v8 -- 0x56000aece180 con 0x56000aec89a0 2023-10-05T19:29:34.885 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.878+0000 7fae8763c700 1 -- 172.21.15.167:0/3844986762 <== osd.1 v2:172.21.15.167:6800/30035 15 ==== osd_op_reply(39 zonegroups_names.default [read 0~46 out=46b] v0'0 uv2 ondisk = 0) v8 ==== 168+0+46 (crc 0 0 0) 0x7fae7802f080 con 0x56000aec89a0 2023-10-05T19:29:34.885 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.878+0000 7fae89306e40 20 rados_obj.operate() r=0 bl.length=46 2023-10-05T19:29:34.885 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.878+0000 7fae89306e40 20 rados->read ofs=0 len=0 2023-10-05T19:29:34.885 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.878+0000 7fae89306e40 1 -- 172.21.15.167:0/3844986762 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:40 5.c 5:30d299a5:::zonegroup_info.4f1f719d-9648-417b-b10c-e5d939825419:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32) v8 -- 0x56000aed18c0 con 0x7fae7000f8d0 2023-10-05T19:29:34.886 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.882+0000 7fae8763c700 1 -- 172.21.15.167:0/3844986762 <== osd.0 v2:172.21.15.118:6800/27690 10 ==== osd_op_reply(40 zonegroup_info.4f1f719d-9648-417b-b10c-e5d939825419 [read 0~362 out=362b] v0'0 uv1 ondisk = 0) v8 ==== 195+0+362 (crc 0 0 0) 0x7fae7802f080 con 0x7fae7000f8d0 2023-10-05T19:29:34.886 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.882+0000 7fae89306e40 20 rados_obj.operate() r=0 bl.length=362 2023-10-05T19:29:34.886 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.882+0000 7fae89306e40 20 rados->read ofs=0 len=0 2023-10-05T19:29:34.886 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.882+0000 7fae89306e40 1 -- 172.21.15.167:0/3844986762 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:41 5.13 5:cbbf6c60:::converted:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32) v8 -- 0x56000aed18c0 con 0x56000aeceb60 2023-10-05T19:29:34.886 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.882+0000 7fae86e3b700 1 -- 172.21.15.167:0/3844986762 <== osd.3 v2:172.21.15.167:6808/30036 10 ==== osd_op_reply(41 converted [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 ==== 153+0+0 (crc 0 0 0) 0x7fae8002b070 con 0x56000aeceb60 2023-10-05T19:29:34.886 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.882+0000 7fae89306e40 20 rados_obj.operate() r=-2 bl.length=0 2023-10-05T19:29:34.886 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.882+0000 7fae89306e40 20 rados->read ofs=0 len=0 2023-10-05T19:29:34.886 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.882+0000 7fae89306e40 1 -- 172.21.15.167:0/3844986762 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:42 5.12 5:49953fa1:::default.realm:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32) v8 -- 0x56000aed1c70 con 0x56000aec89a0 2023-10-05T19:29:34.887 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.882+0000 7fae8763c700 1 -- 172.21.15.167:0/3844986762 <== osd.1 v2:172.21.15.167:6800/30035 16 ==== osd_op_reply(42 default.realm [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 ==== 157+0+0 (crc 0 0 0) 0x7fae7802f080 con 0x56000aec89a0 2023-10-05T19:29:34.887 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.882+0000 7fae89306e40 20 rados_obj.operate() r=-2 bl.length=0 2023-10-05T19:29:34.887 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.882+0000 7fae89306e40 20 rados->read ofs=0 len=0 2023-10-05T19:29:34.887 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.882+0000 7fae89306e40 1 -- 172.21.15.167:0/3844986762 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:43 5.f 5:f4c53578:::zonegroups_names.default:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32) v8 -- 0x56000aed2070 con 0x56000aec89a0 2023-10-05T19:29:34.888 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.882+0000 7fae8763c700 1 -- 172.21.15.167:0/3844986762 <== osd.1 v2:172.21.15.167:6800/30035 17 ==== osd_op_reply(43 zonegroups_names.default [read 0~46 out=46b] v0'0 uv2 ondisk = 0) v8 ==== 168+0+46 (crc 0 0 0) 0x7fae7802f080 con 0x56000aec89a0 2023-10-05T19:29:34.888 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.882+0000 7fae89306e40 20 rados_obj.operate() r=0 bl.length=46 2023-10-05T19:29:34.888 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.882+0000 7fae89306e40 1 -- 172.21.15.167:0/3844986762 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:44 5.0 5:00000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x56000aed3080 con 0x56000aeceb60 2023-10-05T19:29:34.888 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.882+0000 7fae86e3b700 1 -- 172.21.15.167:0/3844986762 <== osd.3 v2:172.21.15.167:6808/30036 11 ==== osd_op_reply(44 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fae8002b070 con 0x56000aeceb60 2023-10-05T19:29:34.888 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.882+0000 7fae86e3b700 1 -- 172.21.15.167:0/3844986762 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:45 5.10 5:08000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x56000aed3080 con 0x56000ae95d10 2023-10-05T19:29:34.889 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.886+0000 7fae8663a700 1 -- 172.21.15.167:0/3844986762 <== osd.2 v2:172.21.15.167:6816/30037 7 ==== osd_op_reply(45 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fae7000bc70 con 0x56000ae95d10 2023-10-05T19:29:34.889 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.886+0000 7fae8663a700 1 -- 172.21.15.167:0/3844986762 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:46 5.8 5:10000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x56000aed3080 con 0x7fae7000f8d0 2023-10-05T19:29:34.889 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.886+0000 7fae8763c700 1 -- 172.21.15.167:0/3844986762 <== osd.0 v2:172.21.15.118:6800/27690 11 ==== osd_op_reply(46 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fae7802f080 con 0x7fae7000f8d0 2023-10-05T19:29:34.889 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.886+0000 7fae8763c700 1 -- 172.21.15.167:0/3844986762 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:47 5.18 5:18000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x56000aece180 con 0x56000aec89a0 2023-10-05T19:29:34.890 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.886+0000 7fae8763c700 1 -- 172.21.15.167:0/3844986762 <== osd.1 v2:172.21.15.167:6800/30035 18 ==== osd_op_reply(47 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fae7802f080 con 0x56000aec89a0 2023-10-05T19:29:34.890 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.886+0000 7fae8763c700 1 -- 172.21.15.167:0/3844986762 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:48 5.4 5:20000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x56000aece180 con 0x56000aec89a0 2023-10-05T19:29:34.891 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.886+0000 7fae8763c700 1 -- 172.21.15.167:0/3844986762 <== osd.1 v2:172.21.15.167:6800/30035 19 ==== osd_op_reply(48 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fae7802f080 con 0x56000aec89a0 2023-10-05T19:29:34.891 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.886+0000 7fae8763c700 1 -- 172.21.15.167:0/3844986762 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:49 5.14 5:28000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x56000aece180 con 0x7fae7000f8d0 2023-10-05T19:29:34.891 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.886+0000 7fae8763c700 1 -- 172.21.15.167:0/3844986762 <== osd.0 v2:172.21.15.118:6800/27690 12 ==== osd_op_reply(49 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fae7802f080 con 0x7fae7000f8d0 2023-10-05T19:29:34.892 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.886+0000 7fae8763c700 1 -- 172.21.15.167:0/3844986762 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:50 5.c 5:30000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x56000aece180 con 0x7fae7000f8d0 2023-10-05T19:29:34.892 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.890+0000 7fae8763c700 1 -- 172.21.15.167:0/3844986762 <== osd.0 v2:172.21.15.118:6800/27690 13 ==== osd_op_reply(50 [pgnls start_epoch 32 out=112b] v23'1 uv1 ondisk = 1) v8 ==== 144+0+112 (crc 0 0 0) 0x7fae7802f080 con 0x7fae7000f8d0 2023-10-05T19:29:34.893 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.890+0000 7fae8763c700 1 -- 172.21.15.167:0/3844986762 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:51 5.1c 5:38000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x56000aece180 con 0x56000ae95d10 2023-10-05T19:29:34.893 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.890+0000 7fae8663a700 1 -- 172.21.15.167:0/3844986762 <== osd.2 v2:172.21.15.167:6816/30037 8 ==== osd_op_reply(51 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fae7000bc70 con 0x56000ae95d10 2023-10-05T19:29:34.893 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.890+0000 7fae8663a700 1 -- 172.21.15.167:0/3844986762 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:52 5.2 5:40000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x56000aece180 con 0x56000aeceb60 2023-10-05T19:29:34.894 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.890+0000 7fae86e3b700 1 -- 172.21.15.167:0/3844986762 <== osd.3 v2:172.21.15.167:6808/30036 12 ==== osd_op_reply(52 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fae8002b070 con 0x56000aeceb60 2023-10-05T19:29:34.894 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.890+0000 7fae86e3b700 1 -- 172.21.15.167:0/3844986762 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:53 5.12 5:48000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x56000aece180 con 0x56000aec89a0 2023-10-05T19:29:34.895 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.890+0000 7fae8763c700 1 -- 172.21.15.167:0/3844986762 <== osd.1 v2:172.21.15.167:6800/30035 20 ==== osd_op_reply(53 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fae7802f080 con 0x56000aec89a0 2023-10-05T19:29:34.895 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.890+0000 7fae8763c700 1 -- 172.21.15.167:0/3844986762 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:54 5.a 5:50000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x56000aece180 con 0x7fae7000f8d0 2023-10-05T19:29:34.896 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.890+0000 7fae8763c700 1 -- 172.21.15.167:0/3844986762 <== osd.0 v2:172.21.15.118:6800/27690 14 ==== osd_op_reply(54 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fae7802f080 con 0x7fae7000f8d0 2023-10-05T19:29:34.896 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.890+0000 7fae8763c700 1 -- 172.21.15.167:0/3844986762 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:55 5.1a 5:58000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x56000aece180 con 0x56000aec89a0 2023-10-05T19:29:34.896 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.894+0000 7fae8763c700 1 -- 172.21.15.167:0/3844986762 <== osd.1 v2:172.21.15.167:6800/30035 21 ==== osd_op_reply(55 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fae7802f080 con 0x56000aec89a0 2023-10-05T19:29:34.897 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.894+0000 7fae8763c700 1 -- 172.21.15.167:0/3844986762 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:56 5.6 5:60000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x56000aece180 con 0x56000ae95d10 2023-10-05T19:29:34.897 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.894+0000 7fae8663a700 1 -- 172.21.15.167:0/3844986762 <== osd.2 v2:172.21.15.167:6816/30037 9 ==== osd_op_reply(56 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fae7000bc70 con 0x56000ae95d10 2023-10-05T19:29:34.897 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.894+0000 7fae8663a700 1 -- 172.21.15.167:0/3844986762 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:57 5.16 5:68000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x56000aece180 con 0x56000aeceb60 2023-10-05T19:29:34.898 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.894+0000 7fae86e3b700 1 -- 172.21.15.167:0/3844986762 <== osd.3 v2:172.21.15.167:6808/30036 13 ==== osd_op_reply(57 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fae8002b070 con 0x56000aeceb60 2023-10-05T19:29:34.898 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.894+0000 7fae86e3b700 1 -- 172.21.15.167:0/3844986762 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:58 5.e 5:70000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x56000aece180 con 0x56000aeceb60 2023-10-05T19:29:34.898 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.894+0000 7fae86e3b700 1 -- 172.21.15.167:0/3844986762 <== osd.3 v2:172.21.15.167:6808/30036 14 ==== osd_op_reply(58 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fae8002b070 con 0x56000aeceb60 2023-10-05T19:29:34.899 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.894+0000 7fae86e3b700 1 -- 172.21.15.167:0/3844986762 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:59 5.1e 5:78000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x56000aece180 con 0x7fae7000f8d0 2023-10-05T19:29:34.900 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.894+0000 7fae8763c700 1 -- 172.21.15.167:0/3844986762 <== osd.0 v2:172.21.15.118:6800/27690 15 ==== osd_op_reply(59 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fae7802f080 con 0x7fae7000f8d0 2023-10-05T19:29:34.900 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.894+0000 7fae8763c700 1 -- 172.21.15.167:0/3844986762 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:60 5.1 5:80000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x56000aece180 con 0x56000aeceb60 2023-10-05T19:29:34.900 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.898+0000 7fae86e3b700 1 -- 172.21.15.167:0/3844986762 <== osd.3 v2:172.21.15.167:6808/30036 15 ==== osd_op_reply(60 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fae8002b070 con 0x56000aeceb60 2023-10-05T19:29:34.901 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.898+0000 7fae86e3b700 1 -- 172.21.15.167:0/3844986762 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:61 5.11 5:88000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x56000aece180 con 0x56000aeceb60 2023-10-05T19:29:34.901 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.898+0000 7fae86e3b700 1 -- 172.21.15.167:0/3844986762 <== osd.3 v2:172.21.15.167:6808/30036 16 ==== osd_op_reply(61 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fae8002b070 con 0x56000aeceb60 2023-10-05T19:29:34.901 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.898+0000 7fae86e3b700 1 -- 172.21.15.167:0/3844986762 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:62 5.9 5:90000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x56000aece180 con 0x56000aec89a0 2023-10-05T19:29:34.902 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.898+0000 7fae8763c700 1 -- 172.21.15.167:0/3844986762 <== osd.1 v2:172.21.15.167:6800/30035 22 ==== osd_op_reply(62 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fae7802f080 con 0x56000aec89a0 2023-10-05T19:29:34.902 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.898+0000 7fae8763c700 1 -- 172.21.15.167:0/3844986762 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:63 5.19 5:98000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x56000aece180 con 0x56000aec89a0 2023-10-05T19:29:34.903 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.898+0000 7fae8763c700 1 -- 172.21.15.167:0/3844986762 <== osd.1 v2:172.21.15.167:6800/30035 23 ==== osd_op_reply(63 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fae7802f080 con 0x56000aec89a0 2023-10-05T19:29:34.903 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.898+0000 7fae8763c700 1 -- 172.21.15.167:0/3844986762 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:64 5.5 5:a0000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x56000aece180 con 0x7fae7000f8d0 2023-10-05T19:29:34.904 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.898+0000 7fae8763c700 1 -- 172.21.15.167:0/3844986762 <== osd.0 v2:172.21.15.118:6800/27690 16 ==== osd_op_reply(64 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fae7802f080 con 0x7fae7000f8d0 2023-10-05T19:29:34.904 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.898+0000 7fae8763c700 1 -- 172.21.15.167:0/3844986762 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:65 5.15 5:a8000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x56000aece180 con 0x56000aeceb60 2023-10-05T19:29:34.904 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.902+0000 7fae86e3b700 1 -- 172.21.15.167:0/3844986762 <== osd.3 v2:172.21.15.167:6808/30036 17 ==== osd_op_reply(65 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fae8002b070 con 0x56000aeceb60 2023-10-05T19:29:34.904 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.902+0000 7fae86e3b700 1 -- 172.21.15.167:0/3844986762 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:66 5.d 5:b0000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x56000aece180 con 0x56000ae95d10 2023-10-05T19:29:34.905 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.902+0000 7fae8663a700 1 -- 172.21.15.167:0/3844986762 <== osd.2 v2:172.21.15.167:6816/30037 10 ==== osd_op_reply(66 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fae7000bc70 con 0x56000ae95d10 2023-10-05T19:29:34.905 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.902+0000 7fae8663a700 1 -- 172.21.15.167:0/3844986762 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:67 5.1d 5:b8000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x56000aece180 con 0x56000aec89a0 2023-10-05T19:29:34.906 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.902+0000 7fae8763c700 1 -- 172.21.15.167:0/3844986762 <== osd.1 v2:172.21.15.167:6800/30035 24 ==== osd_op_reply(67 [pgnls start_epoch 32 out=107b] v30'2 uv2 ondisk = 1) v8 ==== 144+0+107 (crc 0 0 0) 0x7fae7802f080 con 0x56000aec89a0 2023-10-05T19:29:34.906 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.902+0000 7fae8763c700 1 -- 172.21.15.167:0/3844986762 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:68 5.3 5:c0000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x56000aece180 con 0x7fae7000f8d0 2023-10-05T19:29:34.907 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.902+0000 7fae8763c700 1 -- 172.21.15.167:0/3844986762 <== osd.0 v2:172.21.15.118:6800/27690 17 ==== osd_op_reply(68 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fae7802f080 con 0x7fae7000f8d0 2023-10-05T19:29:34.907 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.902+0000 7fae8763c700 1 -- 172.21.15.167:0/3844986762 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:69 5.13 5:c8000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x56000aece180 con 0x56000aeceb60 2023-10-05T19:29:34.908 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.902+0000 7fae86e3b700 1 -- 172.21.15.167:0/3844986762 <== osd.3 v2:172.21.15.167:6808/30036 18 ==== osd_op_reply(69 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fae8002b070 con 0x56000aeceb60 2023-10-05T19:29:34.908 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.902+0000 7fae86e3b700 1 -- 172.21.15.167:0/3844986762 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:70 5.b 5:d0000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x56000aece180 con 0x7fae7000f8d0 2023-10-05T19:29:34.909 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.906+0000 7fae8763c700 1 -- 172.21.15.167:0/3844986762 <== osd.0 v2:172.21.15.118:6800/27690 18 ==== osd_op_reply(70 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fae7802f080 con 0x7fae7000f8d0 2023-10-05T19:29:34.909 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.906+0000 7fae8763c700 1 -- 172.21.15.167:0/3844986762 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:71 5.1b 5:d8000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x56000aece180 con 0x56000ae95d10 2023-10-05T19:29:34.910 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.906+0000 7fae8663a700 1 -- 172.21.15.167:0/3844986762 <== osd.2 v2:172.21.15.167:6816/30037 11 ==== osd_op_reply(71 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fae7000bc70 con 0x56000ae95d10 2023-10-05T19:29:34.910 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.906+0000 7fae8663a700 1 -- 172.21.15.167:0/3844986762 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:72 5.7 5:e0000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x56000aece180 con 0x56000aeceb60 2023-10-05T19:29:34.911 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.906+0000 7fae86e3b700 1 -- 172.21.15.167:0/3844986762 <== osd.3 v2:172.21.15.167:6808/30036 19 ==== osd_op_reply(72 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fae8002b070 con 0x56000aeceb60 2023-10-05T19:29:34.911 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.906+0000 7fae86e3b700 1 -- 172.21.15.167:0/3844986762 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:73 5.17 5:e8000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x56000aece180 con 0x7fae7000f8d0 2023-10-05T19:29:34.911 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.906+0000 7fae8763c700 1 -- 172.21.15.167:0/3844986762 <== osd.0 v2:172.21.15.118:6800/27690 19 ==== osd_op_reply(73 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fae7802f080 con 0x7fae7000f8d0 2023-10-05T19:29:34.912 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.906+0000 7fae8763c700 1 -- 172.21.15.167:0/3844986762 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:74 5.f 5:f0000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x56000aece180 con 0x56000aec89a0 2023-10-05T19:29:34.912 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.910+0000 7fae8763c700 1 -- 172.21.15.167:0/3844986762 <== osd.1 v2:172.21.15.167:6800/30035 25 ==== osd_op_reply(74 [pgnls start_epoch 32 out=115b] v23'2 uv2 ondisk = 1) v8 ==== 144+0+115 (crc 0 0 0) 0x7fae7802f080 con 0x56000aec89a0 2023-10-05T19:29:34.913 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.910+0000 7fae8763c700 1 -- 172.21.15.167:0/3844986762 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:75 5.1f 5:f8000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x56000aece180 con 0x56000ae95d10 2023-10-05T19:29:34.913 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.910+0000 7fae8663a700 1 -- 172.21.15.167:0/3844986762 <== osd.2 v2:172.21.15.167:6816/30037 12 ==== osd_op_reply(75 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fae7000bc70 con 0x56000ae95d10 2023-10-05T19:29:34.913 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.910+0000 7fae89306e40 20 RGWRados::pool_iterate: got zonegroup_info.4f1f719d-9648-417b-b10c-e5d939825419 2023-10-05T19:29:34.913 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.910+0000 7fae89306e40 20 RGWRados::pool_iterate: got zone_info.9a9df965-6612-45d4-a45f-6e3c889bacc2 2023-10-05T19:29:34.914 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.910+0000 7fae89306e40 20 RGWRados::pool_iterate: got zone_names.default 2023-10-05T19:29:34.914 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.910+0000 7fae89306e40 20 RGWRados::pool_iterate: got zonegroups_names.default 2023-10-05T19:29:34.914 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.910+0000 7fae89306e40 20 rados->read ofs=0 len=0 2023-10-05T19:29:34.914 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.910+0000 7fae89306e40 1 -- 172.21.15.167:0/3844986762 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:76 5.f 5:f43b6ece:::zone_names.default:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32) v8 -- 0x56000aed2070 con 0x56000aec89a0 2023-10-05T19:29:34.915 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.910+0000 7fae8763c700 1 -- 172.21.15.167:0/3844986762 <== osd.1 v2:172.21.15.167:6800/30035 26 ==== osd_op_reply(76 zone_names.default [read 0~46 out=46b] v0'0 uv1 ondisk = 0) v8 ==== 162+0+46 (crc 0 0 0) 0x7fae7802f080 con 0x56000aec89a0 2023-10-05T19:29:34.915 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.910+0000 7fae89306e40 20 rados_obj.operate() r=0 bl.length=46 2023-10-05T19:29:34.915 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.910+0000 7fae89306e40 20 rados->read ofs=0 len=0 2023-10-05T19:29:34.915 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.910+0000 7fae89306e40 1 -- 172.21.15.167:0/3844986762 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:77 5.1d 5:b8374eb4:::zone_info.9a9df965-6612-45d4-a45f-6e3c889bacc2:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32) v8 -- 0x56000aed2420 con 0x56000aec89a0 2023-10-05T19:29:34.915 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.910+0000 7fae8763c700 1 -- 172.21.15.167:0/3844986762 <== osd.1 v2:172.21.15.167:6800/30035 27 ==== osd_op_reply(77 zone_info.9a9df965-6612-45d4-a45f-6e3c889bacc2 [read 0~874 out=874b] v0'0 uv2 ondisk = 0) v8 ==== 190+0+874 (crc 0 0 0) 0x7fae7802f080 con 0x56000aec89a0 2023-10-05T19:29:34.915 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.910+0000 7fae89306e40 20 rados_obj.operate() r=0 bl.length=874 2023-10-05T19:29:34.915 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.910+0000 7fae89306e40 20 rados->read ofs=0 len=0 2023-10-05T19:29:34.916 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.910+0000 7fae89306e40 1 -- 172.21.15.167:0/3844986762 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:78 5.f 5:f4c53578:::zonegroups_names.default:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32) v8 -- 0x56000aecdd50 con 0x56000aec89a0 2023-10-05T19:29:34.916 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.910+0000 7fae8763c700 1 -- 172.21.15.167:0/3844986762 <== osd.1 v2:172.21.15.167:6800/30035 28 ==== osd_op_reply(78 zonegroups_names.default [read 0~46 out=46b] v0'0 uv2 ondisk = 0) v8 ==== 168+0+46 (crc 0 0 0) 0x7fae7802f080 con 0x56000aec89a0 2023-10-05T19:29:34.916 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.910+0000 7fae89306e40 20 rados_obj.operate() r=0 bl.length=46 2023-10-05T19:29:34.916 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.910+0000 7fae89306e40 20 rados->read ofs=0 len=0 2023-10-05T19:29:34.916 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.910+0000 7fae89306e40 1 -- 172.21.15.167:0/3844986762 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:79 5.c 5:30d299a5:::zonegroup_info.4f1f719d-9648-417b-b10c-e5d939825419:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32) v8 -- 0x56000aecdd20 con 0x7fae7000f8d0 2023-10-05T19:29:34.917 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.914+0000 7fae8763c700 1 -- 172.21.15.167:0/3844986762 <== osd.0 v2:172.21.15.118:6800/27690 20 ==== osd_op_reply(79 zonegroup_info.4f1f719d-9648-417b-b10c-e5d939825419 [read 0~362 out=362b] v0'0 uv1 ondisk = 0) v8 ==== 195+0+362 (crc 0 0 0) 0x7fae7802f080 con 0x7fae7000f8d0 2023-10-05T19:29:34.917 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.914+0000 7fae89306e40 20 rados_obj.operate() r=0 bl.length=362 2023-10-05T19:29:34.917 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.914+0000 7fae89306e40 20 rados->read ofs=0 len=0 2023-10-05T19:29:34.917 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.914+0000 7fae89306e40 1 -- 172.21.15.167:0/3844986762 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:80 5.3 5:c0ad8cbd:::region_map:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32) v8 -- 0x56000aed2910 con 0x7fae7000f8d0 2023-10-05T19:29:34.918 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.914+0000 7fae8763c700 1 -- 172.21.15.167:0/3844986762 <== osd.0 v2:172.21.15.118:6800/27690 21 ==== osd_op_reply(80 region_map [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 ==== 154+0+0 (crc 0 0 0) 0x7fae7802f080 con 0x7fae7000f8d0 2023-10-05T19:29:34.918 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.914+0000 7fae89306e40 20 rados_obj.operate() r=-2 bl.length=0 2023-10-05T19:29:34.918 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.914+0000 7fae89306e40 20 zone default found 2023-10-05T19:29:34.918 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.914+0000 7fae89306e40 4 Realm: () 2023-10-05T19:29:34.918 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.914+0000 7fae89306e40 4 ZoneGroup: default (4f1f719d-9648-417b-b10c-e5d939825419) 2023-10-05T19:29:34.919 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.914+0000 7fae89306e40 4 Zone: default (9a9df965-6612-45d4-a45f-6e3c889bacc2) 2023-10-05T19:29:34.919 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.914+0000 7fae89306e40 10 cannot find current period zonegroup using local zonegroup configuration 2023-10-05T19:29:34.919 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.914+0000 7fae89306e40 20 zonegroup default 2023-10-05T19:29:34.919 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.914+0000 7fae89306e40 20 rados->read ofs=0 len=0 2023-10-05T19:29:34.919 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.914+0000 7fae89306e40 1 -- 172.21.15.167:0/3844986762 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:81 5.3 5:c52100b6:::period_config.default:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32) v8 -- 0x56000aed2910 con 0x7fae7000f8d0 2023-10-05T19:29:34.920 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.914+0000 7fae8763c700 1 -- 172.21.15.167:0/3844986762 <== osd.0 v2:172.21.15.118:6800/27690 22 ==== osd_op_reply(81 period_config.default [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 ==== 165+0+0 (crc 0 0 0) 0x7fae7802f080 con 0x7fae7000f8d0 2023-10-05T19:29:34.920 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.914+0000 7fae89306e40 20 rados_obj.operate() r=-2 bl.length=0 2023-10-05T19:29:34.920 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.914+0000 7fae89306e40 20 rados->read ofs=0 len=0 2023-10-05T19:29:34.920 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.914+0000 7fae89306e40 1 -- 172.21.15.167:0/3844986762 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:82 6.2 6:4347d321:::bucket.sync-source-hints.:head [call version.read in=11b,read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32) v8 -- 0x56000aed4290 con 0x56000aeceb60 2023-10-05T19:29:34.920 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.914+0000 7fae86e3b700 1 -- 172.21.15.167:0/3844986762 <== osd.3 v2:172.21.15.167:6808/30036 20 ==== osd_op_reply(82 bucket.sync-source-hints. [call,read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 ==== 211+0+0 (crc 0 0 0) 0x7fae8002b070 con 0x56000aeceb60 2023-10-05T19:29:34.920 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.914+0000 7fae89306e40 20 rados_obj.operate() r=-2 bl.length=0 2023-10-05T19:29:34.920 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.914+0000 7fae89306e40 20 rados->read ofs=0 len=0 2023-10-05T19:29:34.921 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.914+0000 7fae89306e40 1 -- 172.21.15.167:0/3844986762 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:83 6.b 6:d467b91b:::bucket.sync-target-hints.:head [call version.read in=11b,read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32) v8 -- 0x56000aed4640 con 0x56000aeceb60 2023-10-05T19:29:34.921 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.914+0000 7fae86e3b700 1 -- 172.21.15.167:0/3844986762 <== osd.3 v2:172.21.15.167:6808/30036 21 ==== osd_op_reply(83 bucket.sync-target-hints. [call,read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 ==== 211+0+0 (crc 0 0 0) 0x7fae8002b070 con 0x56000aeceb60 2023-10-05T19:29:34.921 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.918+0000 7fae89306e40 20 rados_obj.operate() r=-2 bl.length=0 2023-10-05T19:29:34.921 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.918+0000 7fae89306e40 20 started sync module instance, tier type = 2023-10-05T19:29:34.922 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.918+0000 7fae89306e40 20 started zone id=9a9df965-6612-45d4-a45f-6e3c889bacc2 (name=default) with tier type = 2023-10-05T19:29:34.922 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.918+0000 7fae89306e40 1 -- 172.21.15.167:0/3844986762 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:84 7.1f 7:f95f44c2:::notify.0:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e32) v8 -- 0x56000aed3950 con 0x56000ae95d10 2023-10-05T19:29:34.923 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.918+0000 7fae8663a700 1 -- 172.21.15.167:0/3844986762 <== osd.2 v2:172.21.15.167:6816/30037 13 ==== osd_op_reply(84 notify.0 [create] v32'243 uv243 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7fae7000bc70 con 0x56000ae95d10 2023-10-05T19:29:34.923 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.918+0000 7fae89306e40 1 -- 172.21.15.167:0/3844986762 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:85 7.1f 7:f95f44c2:::notify.0:head [watch watch cookie 94558183306336] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e32) v8 -- 0x56000aed49f0 con 0x56000ae95d10 2023-10-05T19:29:34.923 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.918+0000 7fae89306e40 1 -- 172.21.15.167:0/3844986762 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:86 7.0 7:05bf5b68:::notify.1:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e32) v8 -- 0x56000aed52d0 con 0x56000aec89a0 2023-10-05T19:29:34.926 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.922+0000 7fae8663a700 1 -- 172.21.15.167:0/3844986762 <== osd.2 v2:172.21.15.167:6816/30037 14 ==== osd_op_reply(85 notify.0 [watch watch cookie 94558183306336] v32'244 uv243 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7fae7000bc70 con 0x56000ae95d10 2023-10-05T19:29:34.927 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.922+0000 7fae8763c700 1 -- 172.21.15.167:0/3844986762 <== osd.1 v2:172.21.15.167:6800/30035 29 ==== osd_op_reply(86 notify.1 [create] v32'243 uv243 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7fae7802f080 con 0x56000aec89a0 2023-10-05T19:29:34.927 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.922+0000 7fae89306e40 1 -- 172.21.15.167:0/3844986762 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:87 7.0 7:05bf5b68:::notify.1:head [watch watch cookie 94558183308624] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e32) v8 -- 0x56000aed56d0 con 0x56000aec89a0 2023-10-05T19:29:34.927 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.922+0000 7fae89306e40 1 -- 172.21.15.167:0/3844986762 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:88 7.15 7:a93a5511:::notify.2:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e32) v8 -- 0x56000aed5f70 con 0x56000aeceb60 2023-10-05T19:29:34.930 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.926+0000 7fae8763c700 1 -- 172.21.15.167:0/3844986762 <== osd.1 v2:172.21.15.167:6800/30035 30 ==== osd_op_reply(87 notify.1 [watch watch cookie 94558183308624] v32'244 uv243 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7fae7802f080 con 0x56000aec89a0 2023-10-05T19:29:34.930 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.926+0000 7fae86e3b700 1 -- 172.21.15.167:0/3844986762 <== osd.3 v2:172.21.15.167:6808/30036 22 ==== osd_op_reply(88 notify.2 [create] v32'243 uv243 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7fae8002b070 con 0x56000aeceb60 2023-10-05T19:29:34.930 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.926+0000 7fae89306e40 1 -- 172.21.15.167:0/3844986762 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:89 7.15 7:a93a5511:::notify.2:head [watch watch cookie 94558183315152] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e32) v8 -- 0x56000aed4ec0 con 0x56000aeceb60 2023-10-05T19:29:34.930 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.926+0000 7fae89306e40 1 -- 172.21.15.167:0/3844986762 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:90 7.e 7:7759931f:::notify.3:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e32) v8 -- 0x56000aed6d00 con 0x56000ae95d10 2023-10-05T19:29:34.933 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.930+0000 7fae8663a700 1 -- 172.21.15.167:0/3844986762 <== osd.2 v2:172.21.15.167:6816/30037 15 ==== osd_op_reply(90 notify.3 [create] v32'243 uv243 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7fae7000bc70 con 0x56000ae95d10 2023-10-05T19:29:34.933 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.930+0000 7fae86e3b700 1 -- 172.21.15.167:0/3844986762 <== osd.3 v2:172.21.15.167:6808/30036 23 ==== osd_op_reply(89 notify.2 [watch watch cookie 94558183315152] v32'244 uv243 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7fae8002b070 con 0x56000aeceb60 2023-10-05T19:29:34.933 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.930+0000 7fae89306e40 1 -- 172.21.15.167:0/3844986762 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:91 7.e 7:7759931f:::notify.3:head [watch watch cookie 94558183319568] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e32) v8 -- 0x56000aed70b0 con 0x56000ae95d10 2023-10-05T19:29:34.933 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.930+0000 7fae89306e40 1 -- 172.21.15.167:0/3844986762 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:92 7.d 7:b4812045:::notify.4:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e32) v8 -- 0x56000aed7b30 con 0x56000aec89a0 2023-10-05T19:29:34.936 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.930+0000 7fae8663a700 1 -- 172.21.15.167:0/3844986762 <== osd.2 v2:172.21.15.167:6816/30037 16 ==== osd_op_reply(91 notify.3 [watch watch cookie 94558183319568] v32'244 uv243 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7fae7000bc70 con 0x56000ae95d10 2023-10-05T19:29:34.936 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.930+0000 7fae8763c700 1 -- 172.21.15.167:0/3844986762 <== osd.1 v2:172.21.15.167:6800/30035 31 ==== osd_op_reply(92 notify.4 [create] v32'243 uv243 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7fae7802f080 con 0x56000aec89a0 2023-10-05T19:29:34.936 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.930+0000 7fae89306e40 1 -- 172.21.15.167:0/3844986762 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:93 7.d 7:b4812045:::notify.4:head [watch watch cookie 94558183318304] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e32) v8 -- 0x56000aed7f30 con 0x56000aec89a0 2023-10-05T19:29:34.936 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.930+0000 7fae89306e40 1 -- 172.21.15.167:0/3844986762 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:94 7.3 7:c609908c:::notify.5:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e32) v8 -- 0x56000aed8940 con 0x7fae7000f8d0 2023-10-05T19:29:34.938 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.934+0000 7fae8763c700 1 -- 172.21.15.167:0/3844986762 <== osd.1 v2:172.21.15.167:6800/30035 32 ==== osd_op_reply(93 notify.4 [watch watch cookie 94558183318304] v32'244 uv243 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7fae7802f080 con 0x56000aec89a0 2023-10-05T19:29:34.939 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.934+0000 7fae8763c700 1 -- 172.21.15.167:0/3844986762 <== osd.0 v2:172.21.15.118:6800/27690 23 ==== osd_op_reply(94 notify.5 [create] v32'243 uv243 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7fae7802f080 con 0x7fae7000f8d0 2023-10-05T19:29:34.939 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.934+0000 7fae89306e40 1 -- 172.21.15.167:0/3844986762 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:95 7.3 7:c609908c:::notify.5:head [watch watch cookie 94558183323136] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e32) v8 -- 0x56000aed8940 con 0x7fae7000f8d0 2023-10-05T19:29:34.939 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.934+0000 7fae89306e40 1 -- 172.21.15.167:0/3844986762 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:96 7.14 7:2b04a3e9:::notify.6:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e32) v8 -- 0x56000aed91e0 con 0x7fae7000f8d0 2023-10-05T19:29:34.941 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.938+0000 7fae8763c700 1 -- 172.21.15.167:0/3844986762 <== osd.0 v2:172.21.15.118:6800/27690 24 ==== osd_op_reply(96 notify.6 [create] v32'243 uv243 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7fae7802f080 con 0x7fae7000f8d0 2023-10-05T19:29:34.941 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.938+0000 7fae89306e40 1 -- 172.21.15.167:0/3844986762 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:97 7.14 7:2b04a3e9:::notify.6:head [watch watch cookie 94558183328624] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e32) v8 -- 0x56000aed95c0 con 0x7fae7000f8d0 2023-10-05T19:29:34.941 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.938+0000 7fae89306e40 1 -- 172.21.15.167:0/3844986762 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:98 7.9 7:93e5b521:::notify.7:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e32) v8 -- 0x56000aeda120 con 0x7fae7000f8d0 2023-10-05T19:29:34.942 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.938+0000 7fae8763c700 1 -- 172.21.15.167:0/3844986762 <== osd.0 v2:172.21.15.118:6800/27690 25 ==== osd_op_reply(95 notify.5 [watch watch cookie 94558183323136] v32'244 uv243 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7fae7802f080 con 0x7fae7000f8d0 2023-10-05T19:29:34.943 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.938+0000 7fae8763c700 1 -- 172.21.15.167:0/3844986762 <== osd.0 v2:172.21.15.118:6800/27690 26 ==== osd_op_reply(97 notify.6 [watch watch cookie 94558183328624] v32'244 uv243 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7fae7802f080 con 0x7fae7000f8d0 2023-10-05T19:29:34.944 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.938+0000 7fae8763c700 1 -- 172.21.15.167:0/3844986762 <== osd.0 v2:172.21.15.118:6800/27690 27 ==== osd_op_reply(98 notify.7 [create] v32'243 uv243 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7fae7802f080 con 0x7fae7000f8d0 2023-10-05T19:29:34.944 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.938+0000 7fae89306e40 1 -- 172.21.15.167:0/3844986762 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:99 7.9 7:93e5b521:::notify.7:head [watch watch cookie 94558183326800] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e32) v8 -- 0x56000aed9bf0 con 0x7fae7000f8d0 2023-10-05T19:29:34.944 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.938+0000 7fae89306e40 20 add_watcher() i=0 2023-10-05T19:29:34.944 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.938+0000 7fae89306e40 20 add_watcher() i=1 2023-10-05T19:29:34.945 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.938+0000 7fae89306e40 20 add_watcher() i=2 2023-10-05T19:29:34.945 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.938+0000 7fae89306e40 20 add_watcher() i=3 2023-10-05T19:29:34.945 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.938+0000 7fae89306e40 20 add_watcher() i=4 2023-10-05T19:29:34.945 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.938+0000 7fae89306e40 20 add_watcher() i=5 2023-10-05T19:29:34.945 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.938+0000 7fae89306e40 20 add_watcher() i=6 2023-10-05T19:29:34.947 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.942+0000 7fae8763c700 1 -- 172.21.15.167:0/3844986762 <== osd.0 v2:172.21.15.118:6800/27690 28 ==== osd_op_reply(99 notify.7 [watch watch cookie 94558183326800] v32'244 uv243 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7fae7802f080 con 0x7fae7000f8d0 2023-10-05T19:29:34.947 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.942+0000 7fae89306e40 20 add_watcher() i=7 2023-10-05T19:29:34.947 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.942+0000 7fae89306e40 2 all 8 watchers are set, enabling cache 2023-10-05T19:29:34.947 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.942+0000 7fae89306e40 1 -- 172.21.15.167:0/3844986762 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:100 6.4 6:22d26bf9:::data_loggenerations_metadata:head [call version.check_conds in=74b,call version.read in=11b,read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32) v8 -- 0x56000aedd110 con 0x56000aec89a0 2023-10-05T19:29:34.947 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.942+0000 7fae8763c700 1 -- 172.21.15.167:0/3844986762 <== osd.1 v2:172.21.15.167:6800/30035 33 ==== osd_op_reply(100 data_loggenerations_metadata [call,call out=48b,read 0~28 out=28b] v0'0 uv1 ondisk = 0) v8 ==== 256+0+76 (crc 0 0 0) 0x7fae7802f080 con 0x56000aec89a0 2023-10-05T19:29:34.947 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.942+0000 7fae89306e40 1 -- 172.21.15.167:0/3844986762 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:101 6.4 6:22d26bf9:::data_loggenerations_metadata:head [watch watch cookie 94558183332848] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e32) v8 -- 0x56000aeda530 con 0x56000aec89a0 2023-10-05T19:29:34.950 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.946+0000 7fae8763c700 1 -- 172.21.15.167:0/3844986762 <== osd.1 v2:172.21.15.167:6800/30035 34 ==== osd_op_reply(101 data_loggenerations_metadata [watch watch cookie 94558183332848] v32'235 uv1 ondisk = 0) v8 ==== 172+0+0 (crc 0 0 0) 0x7fae7802f080 con 0x56000aec89a0 2023-10-05T19:29:34.950 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.946+0000 7fadd3fff700 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2023-10-05T19:29:34.950 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.946+0000 7fae89306e40 20 check_secure_mon_conn(): auth registy supported: methods=[2] modes=[2,1] 2023-10-05T19:29:34.950 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.946+0000 7fae89306e40 20 check_secure_mon_conn(): mode 1 is insecure 2023-10-05T19:29:34.951 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.946+0000 7fae89306e40 1 Processor -- start 2023-10-05T19:29:34.951 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.946+0000 7fae89306e40 1 -- start start 2023-10-05T19:29:34.951 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.946+0000 7fae89306e40 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x56000aee7580 0x56000aee7950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-05T19:29:34.951 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.946+0000 7fae89306e40 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x56000aee7e90 con 0x56000aee7580 2023-10-05T19:29:34.952 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.946+0000 7fae8663a700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x56000aee7580 0x56000aee7950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-05T19:29:34.952 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.946+0000 7fae8663a700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x56000aee7580 0x56000aee7950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.118:3300/0 says I am v2:172.21.15.167:56770/0 (socket says 172.21.15.167:56770) 2023-10-05T19:29:34.952 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.946+0000 7fae8663a700 1 -- 172.21.15.167:0/3919134041 learned_addr learned my addr 172.21.15.167:0/3919134041 (peer_addr_for_me v2:172.21.15.167:0/0) 2023-10-05T19:29:34.952 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.946+0000 7fae8663a700 1 -- 172.21.15.167:0/3919134041 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x56000aee7fd0 con 0x56000aee7580 2023-10-05T19:29:34.952 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.950+0000 7fae8663a700 1 --2- 172.21.15.167:0/3919134041 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x56000aee7580 0x56000aee7950 secure :-1 s=READY pgs=624 cs=0 l=1 rev1=1 crypto rx=0x7fae70000fb0 tx=0x7fae70011000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=aeb757e052fa08cd server_cookie=0 in_seq=0 out_seq=0 2023-10-05T19:29:34.953 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.950+0000 7fadd37fe700 1 -- 172.21.15.167:0/3919134041 <== mon.0 v2:172.21.15.118:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fae70010e20 con 0x56000aee7580 2023-10-05T19:29:34.953 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.950+0000 7fadd37fe700 1 -- 172.21.15.167:0/3919134041 <== mon.0 v2:172.21.15.118:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fae70026040 con 0x56000aee7580 2023-10-05T19:29:34.953 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.950+0000 7fadd37fe700 1 -- 172.21.15.167:0/3919134041 <== mon.0 v2:172.21.15.118:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fae7000ed60 con 0x56000aee7580 2023-10-05T19:29:34.954 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.950+0000 7fae89306e40 1 -- 172.21.15.167:0/3919134041 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x56000aee7580 msgr2=0x56000aee7950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-05T19:29:34.954 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.950+0000 7fae89306e40 1 --2- 172.21.15.167:0/3919134041 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x56000aee7580 0x56000aee7950 secure :-1 s=READY pgs=624 cs=0 l=1 rev1=1 crypto rx=0x7fae70000fb0 tx=0x7fae70011000 comp rx=0 tx=0).stop 2023-10-05T19:29:34.955 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.950+0000 7fae89306e40 1 -- 172.21.15.167:0/3919134041 shutdown_connections 2023-10-05T19:29:34.955 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.950+0000 7fae89306e40 1 --2- 172.21.15.167:0/3919134041 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x56000aee7580 0x56000aee7950 unknown :-1 s=CLOSED pgs=624 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-05T19:29:34.955 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.950+0000 7fae89306e40 1 -- 172.21.15.167:0/3919134041 >> 172.21.15.167:0/3919134041 conn(0x56000aee2940 msgr2=0x56000aee0a90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-05T19:29:34.955 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.950+0000 7fae89306e40 1 -- 172.21.15.167:0/3919134041 shutdown_connections 2023-10-05T19:29:34.955 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.950+0000 7fae89306e40 1 -- 172.21.15.167:0/3919134041 wait complete. 2023-10-05T19:29:34.956 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.950+0000 7fae89306e40 1 Processor -- start 2023-10-05T19:29:34.956 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.950+0000 7fae89306e40 1 -- start start 2023-10-05T19:29:34.956 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.950+0000 7fae89306e40 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x56000aee7580 0x56000aeed110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-05T19:29:34.956 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.950+0000 7fae89306e40 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x56000aeee110 con 0x56000aee7580 2023-10-05T19:29:34.956 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.950+0000 7fae8663a700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x56000aee7580 0x56000aeed110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-05T19:29:34.956 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.950+0000 7fae8663a700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x56000aee7580 0x56000aeed110 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.118:3300/0 says I am v2:172.21.15.167:56776/0 (socket says 172.21.15.167:56776) 2023-10-05T19:29:34.957 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.950+0000 7fae8663a700 1 -- 172.21.15.167:0/947572296 learned_addr learned my addr 172.21.15.167:0/947572296 (peer_addr_for_me v2:172.21.15.167:0/0) 2023-10-05T19:29:34.957 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.954+0000 7fae8663a700 1 -- 172.21.15.167:0/947572296 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fae7000a040 con 0x56000aee7580 2023-10-05T19:29:34.958 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.954+0000 7fae8663a700 1 --2- 172.21.15.167:0/947572296 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x56000aee7580 0x56000aeed110 secure :-1 s=READY pgs=625 cs=0 l=1 rev1=1 crypto rx=0x7fae70003010 tx=0x7fae70031070 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-05T19:29:34.958 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.954+0000 7fadd1ffb700 1 -- 172.21.15.167:0/947572296 <== mon.0 v2:172.21.15.118:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fae70044070 con 0x56000aee7580 2023-10-05T19:29:34.958 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.954+0000 7fae89306e40 1 -- 172.21.15.167:0/947572296 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x56000aeed710 con 0x56000aee7580 2023-10-05T19:29:34.958 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.954+0000 7fadd1ffb700 1 -- 172.21.15.167:0/947572296 <== mon.0 v2:172.21.15.118:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fae7003c460 con 0x56000aee7580 2023-10-05T19:29:34.958 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.954+0000 7fadd1ffb700 1 -- 172.21.15.167:0/947572296 <== mon.0 v2:172.21.15.118:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fae7003c610 con 0x56000aee7580 2023-10-05T19:29:34.958 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.954+0000 7fae89306e40 1 -- 172.21.15.167:0/947572296 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x56000aeedb50 con 0x56000aee7580 2023-10-05T19:29:34.959 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.954+0000 7fadd1ffb700 1 -- 172.21.15.167:0/947572296 <== mon.0 v2:172.21.15.118:3300/0 4 ==== mgrmap(e 5) v1 ==== 40329+0+0 (secure 0 0 0) 0x7fae7002e410 con 0x56000aee7580 2023-10-05T19:29:34.959 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.954+0000 7fadd1ffb700 1 --2- 172.21.15.167:0/947572296 >> [v2:172.21.15.167:6824/29889,v1:172.21.15.167:6825/29889] conn(0x7fadc4030bb0 0x7fadc4033060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-05T19:29:34.959 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.954+0000 7fadd1ffb700 1 -- 172.21.15.167:0/947572296 <== mon.0 v2:172.21.15.118:3300/0 5 ==== osd_map(32..32 src has 1..32) v4 ==== 6260+0+0 (secure 0 0 0) 0x7fae70018040 con 0x56000aee7580 2023-10-05T19:29:34.960 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.954+0000 7fae8763c700 1 --2- 172.21.15.167:0/947572296 >> [v2:172.21.15.167:6824/29889,v1:172.21.15.167:6825/29889] conn(0x7fadc4030bb0 0x7fadc4033060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-05T19:29:34.960 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.954+0000 7fae89306e40 5 note: GC not initialized 2023-10-05T19:29:34.960 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.958+0000 7fae8763c700 1 --2- 172.21.15.167:0/947572296 >> [v2:172.21.15.167:6824/29889,v1:172.21.15.167:6825/29889] conn(0x7fadc4030bb0 0x7fadc4033060 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x56000aed9310 tx=0x7fae7804a060 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-05T19:29:34.962 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.958+0000 7fadcaffd700 20 reqs_thread_entry: start 2023-10-05T19:29:34.963 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.958+0000 7fadc8ff9700 20 reqs_thread_entry: start 2023-10-05T19:29:34.963 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.958+0000 7fae89306e40 20 init_complete bucket index max shards: 11 2023-10-05T19:29:34.963 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.958+0000 7fae89306e40 10 rgw notify: Started notification manager with: 1 workers 2023-10-05T19:29:34.964 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.958+0000 7fadb6ffd700 20 rgw notify: INFO: next queues processing will happen at: Thu Oct 5 19:30:05 2023 2023-10-05T19:29:34.964 INFO:teuthology.orchestra.run.smithi167.stderr: 2023-10-05T19:29:34.964 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.958+0000 7fadb67fc700 20 reqs_thread_entry: start 2023-10-05T19:29:34.964 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.958+0000 7fae89306e40 20 get_system_obj_state: rctx=0x7fffd9fa8e48 obj=default.rgw.meta:users.uid:bar state=0x56000ad700b0 s->prefetch_data=0 2023-10-05T19:29:34.964 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.958+0000 7fae89306e40 10 cache get: name=default.rgw.meta+users.uid+bar : miss 2023-10-05T19:29:34.964 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.958+0000 7fae89306e40 1 -- 172.21.15.167:0/3844986762 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:102 8.e 8:718f6a54:users.uid::bar:head [call version.read in=11b,getxattrs,stat] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32) v8 -- 0x56000af55980 con 0x7fae7000f8d0 2023-10-05T19:29:34.965 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.962+0000 7fae8763c700 1 -- 172.21.15.167:0/3844986762 <== osd.0 v2:172.21.15.118:6800/27690 29 ==== osd_op_reply(102 bar [call out=48b,getxattrs out=75b,stat out=16b] v0'0 uv3 ondisk = 0) v8 ==== 231+0+139 (crc 0 0 0) 0x7fae7802f080 con 0x7fae7000f8d0 2023-10-05T19:29:34.965 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.962+0000 7fae89306e40 10 cache put: name=default.rgw.meta+users.uid+bar info.flags=0x16 2023-10-05T19:29:34.965 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.962+0000 7fae89306e40 10 adding default.rgw.meta+users.uid+bar to cache LRU end 2023-10-05T19:29:34.965 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.962+0000 7fae89306e40 10 updating xattr: name=ceph.objclass.version bl.length()=42 2023-10-05T19:29:34.965 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.962+0000 7fae89306e40 20 get_system_obj_state: s->obj_tag was set empty 2023-10-05T19:29:34.965 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.962+0000 7fae89306e40 20 Read xattr: user.rgw.idtag 2023-10-05T19:29:34.965 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.962+0000 7fae89306e40 10 cache get: name=default.rgw.meta+users.uid+bar : type miss (requested=0x13, cached=0x16) 2023-10-05T19:29:34.965 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.962+0000 7fae89306e40 20 rados->read ofs=0 len=0 2023-10-05T19:29:34.966 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.962+0000 7fae89306e40 1 -- 172.21.15.167:0/3844986762 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:103 8.e 8:718f6a54:users.uid::bar:head [call version.check_conds in=122b,call version.read in=11b,read 0~0,getxattrs] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32) v8 -- 0x56000af58410 con 0x7fae7000f8d0 2023-10-05T19:29:34.966 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.962+0000 7fae8763c700 1 -- 172.21.15.167:0/3844986762 <== osd.0 v2:172.21.15.118:6800/27690 30 ==== osd_op_reply(103 bar [call,call out=48b,read 0~346 out=346b,getxattrs out=75b] v0'0 uv3 ondisk = 0) v8 ==== 273+0+469 (crc 0 0 0) 0x7fae7802f080 con 0x7fae7000f8d0 2023-10-05T19:29:34.966 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.962+0000 7fae89306e40 20 rados_obj.operate() r=0 bl.length=346 2023-10-05T19:29:34.967 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.962+0000 7fae89306e40 10 cache put: name=default.rgw.meta+users.uid+bar info.flags=0x13 2023-10-05T19:29:34.967 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.962+0000 7fae89306e40 10 moving default.rgw.meta+users.uid+bar to cache LRU end 2023-10-05T19:29:34.967 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.962+0000 7fae89306e40 10 updating xattr: name=ceph.objclass.version bl.length()=42 2023-10-05T19:29:34.967 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.962+0000 7fae89306e40 1 -- 172.21.15.167:0/3844986762 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:104 8.4 8:267425a8:users.uid::bud:head [create,call version.set in=58b,writefull 0~181 in=181b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e32) v8 -- 0x56000af59a60 con 0x56000ae95d10 2023-10-05T19:29:34.970 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.966+0000 7fae8663a700 1 -- 172.21.15.167:0/3844986762 <== osd.2 v2:172.21.15.167:6816/30037 17 ==== osd_op_reply(104 bud [create,call,writefull 0~181] v32'1 uv1 ondisk = 0) v8 ==== 231+0+0 (crc 0 0 0) 0x7fae7000bc70 con 0x56000ae95d10 2023-10-05T19:29:34.970 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.966+0000 7fae89306e40 10 cache put: name=default.rgw.meta+users.uid+bud info.flags=0x17 2023-10-05T19:29:34.970 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.966+0000 7fae89306e40 10 adding default.rgw.meta+users.uid+bud to cache LRU end 2023-10-05T19:29:34.970 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.966+0000 7fae89306e40 10 distributing notification oid=default.rgw.control:notify.5 cni=[op: 0, obj: default.rgw.meta:users.uid:bud, ofs0, ns] 2023-10-05T19:29:34.970 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.966+0000 7fae89306e40 1 -- 172.21.15.167:0/3844986762 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:105 7.3 7:c609908c:::notify.5:head [notify cookie 94558183851360 in=369b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32) v8 -- 0x56000af5a730 con 0x7fae7000f8d0 2023-10-05T19:29:34.971 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.966+0000 7fae8763c700 1 -- 172.21.15.167:0/3844986762 <== osd.0 v2:172.21.15.118:6800/27690 31 ==== watch-notify(notify (1) cookie 94558183323136 notify 137438953505 ret 0) v3 ==== 399+0+0 (crc 0 0 0) 0x7fae7803d040 con 0x7fae7000f8d0 2023-10-05T19:29:34.971 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.966+0000 7fae84e37700 10 rgw watcher librados: RGWWatcher::handle_notify() notify_id 137438953505 cookie 94558183323136 notifier 4713 bl.length()=357 2023-10-05T19:29:34.971 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.966+0000 7fae84e37700 10 rgw watcher librados: cache put: name=default.rgw.meta+users.uid+bud info.flags=0x17 2023-10-05T19:29:34.971 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.966+0000 7fae84e37700 10 rgw watcher librados: moving default.rgw.meta+users.uid+bud to cache LRU end 2023-10-05T19:29:34.971 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.966+0000 7fae84e37700 1 -- 172.21.15.167:0/3844986762 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:106 7.3 7:c609908c:::notify.5:head [notify-ack in=20b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32) v8 -- 0x7fae6c003d90 con 0x7fae7000f8d0 2023-10-05T19:29:34.971 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.966+0000 7fae8763c700 1 -- 172.21.15.167:0/3844986762 <== osd.0 v2:172.21.15.118:6800/27690 32 ==== osd_op_reply(105 notify.5 [notify cookie 94558183851360 out=8b] v0'0 uv243 ondisk = 0) v8 ==== 152+0+8 (crc 0 0 0) 0x7fae7802f080 con 0x7fae7000f8d0 2023-10-05T19:29:34.972 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.966+0000 7fae8763c700 1 -- 172.21.15.167:0/3844986762 <== osd.0 v2:172.21.15.118:6800/27690 33 ==== osd_op_reply(106 notify.5 [notify-ack] v0'0 uv243 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7fae7802f080 con 0x7fae7000f8d0 2023-10-05T19:29:34.972 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.966+0000 7fae8763c700 1 -- 172.21.15.167:0/3844986762 <== osd.0 v2:172.21.15.118:6800/27690 34 ==== watch-notify(notify_complete (2) cookie 94558183851360 notify 137438953505 ret 0) v3 ==== 42+0+48 (crc 0 0 0) 0x7fae78002bd0 con 0x7fae7000f8d0 2023-10-05T19:29:34.972 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.966+0000 7fae89306e40 1 -- 172.21.15.167:0/3844986762 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:107 8.5 8:a00000ee:users.uid::bar.buckets:head [call user.list_buckets in=34b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32) v8 -- 0x56000af58560 con 0x56000aec89a0 2023-10-05T19:29:34.973 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.970+0000 7fae8763c700 1 -- 172.21.15.167:0/3844986762 <== osd.1 v2:172.21.15.167:6800/30035 35 ==== osd_op_reply(107 bar.buckets [call out=184b] v0'0 uv6 ondisk = 0) v8 ==== 155+0+184 (crc 0 0 0) 0x7fae7802f080 con 0x56000aec89a0 2023-10-05T19:29:34.973 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.970+0000 7fae89306e40 20 get_system_obj_state: rctx=0x7fffd9fa8db0 obj=default.rgw.meta:root:.bucket.meta.myfoo6:9a9df965-6612-45d4-a45f-6e3c889bacc2.4223.7 state=0x56000ad700b0 s->prefetch_data=0 2023-10-05T19:29:34.973 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.970+0000 7fae89306e40 10 cache get: name=default.rgw.meta+root+.bucket.meta.myfoo6:9a9df965-6612-45d4-a45f-6e3c889bacc2.4223.7 : miss 2023-10-05T19:29:34.973 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.970+0000 7fae89306e40 1 -- 172.21.15.167:0/3844986762 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:108 8.1f 8:fe511ebf:root::.bucket.meta.myfoo6%3a9a9df965-6612-45d4-a45f-6e3c889bacc2.4223.7:head [call version.read in=11b,getxattrs,stat] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32) v8 -- 0x56000af57ee0 con 0x56000ae95d10 2023-10-05T19:29:34.974 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.970+0000 7fae8663a700 1 -- 172.21.15.167:0/3844986762 <== osd.2 v2:172.21.15.167:6816/30037 18 ==== osd_op_reply(108 .bucket.meta.myfoo6:9a9df965-6612-45d4-a45f-6e3c889bacc2.4223.7 [call out=48b,getxattrs out=218b,stat out=16b] v0'0 uv1 ondisk = 0) v8 ==== 291+0+282 (crc 0 0 0) 0x7fae7000bc70 con 0x56000ae95d10 2023-10-05T19:29:34.974 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.970+0000 7fae89306e40 10 cache put: name=default.rgw.meta+root+.bucket.meta.myfoo6:9a9df965-6612-45d4-a45f-6e3c889bacc2.4223.7 info.flags=0x16 2023-10-05T19:29:34.974 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.970+0000 7fae89306e40 10 adding default.rgw.meta+root+.bucket.meta.myfoo6:9a9df965-6612-45d4-a45f-6e3c889bacc2.4223.7 to cache LRU end 2023-10-05T19:29:34.974 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.970+0000 7fae89306e40 10 updating xattr: name=ceph.objclass.version bl.length()=42 2023-10-05T19:29:34.975 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.970+0000 7fae89306e40 10 updating xattr: name=user.rgw.acl bl.length()=123 2023-10-05T19:29:34.975 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.970+0000 7fae89306e40 20 get_system_obj_state: s->obj_tag was set empty 2023-10-05T19:29:34.975 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.970+0000 7fae89306e40 20 Read xattr: user.rgw.acl 2023-10-05T19:29:34.975 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.970+0000 7fae89306e40 20 Read xattr: user.rgw.idtag 2023-10-05T19:29:34.975 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.970+0000 7fae89306e40 10 cache get: name=default.rgw.meta+root+.bucket.meta.myfoo6:9a9df965-6612-45d4-a45f-6e3c889bacc2.4223.7 : type miss (requested=0x13, cached=0x16) 2023-10-05T19:29:34.975 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.970+0000 7fae89306e40 20 rados->read ofs=0 len=0 2023-10-05T19:29:34.975 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.970+0000 7fae89306e40 1 -- 172.21.15.167:0/3844986762 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:109 8.1f 8:fe511ebf:root::.bucket.meta.myfoo6%3a9a9df965-6612-45d4-a45f-6e3c889bacc2.4223.7:head [call version.check_conds in=122b,call version.read in=11b,read 0~0,getxattrs] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32) v8 -- 0x56000af5c070 con 0x56000ae95d10 2023-10-05T19:29:34.976 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.970+0000 7fae8663a700 1 -- 172.21.15.167:0/3844986762 <== osd.2 v2:172.21.15.167:6816/30037 19 ==== osd_op_reply(109 .bucket.meta.myfoo6:9a9df965-6612-45d4-a45f-6e3c889bacc2.4223.7 [call,call out=48b,read 0~353 out=353b,getxattrs out=218b] v0'0 uv1 ondisk = 0) v8 ==== 333+0+619 (crc 0 0 0) 0x7fae7000bc70 con 0x56000ae95d10 2023-10-05T19:29:34.976 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.970+0000 7fae89306e40 20 rados_obj.operate() r=0 bl.length=353 2023-10-05T19:29:34.976 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.970+0000 7fae89306e40 10 cache put: name=default.rgw.meta+root+.bucket.meta.myfoo6:9a9df965-6612-45d4-a45f-6e3c889bacc2.4223.7 info.flags=0x13 2023-10-05T19:29:34.976 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.970+0000 7fae89306e40 10 moving default.rgw.meta+root+.bucket.meta.myfoo6:9a9df965-6612-45d4-a45f-6e3c889bacc2.4223.7 to cache LRU end 2023-10-05T19:29:34.976 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.970+0000 7fae89306e40 10 updating xattr: name=ceph.objclass.version bl.length()=42 2023-10-05T19:29:34.976 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.970+0000 7fae89306e40 10 updating xattr: name=user.rgw.acl bl.length()=123 2023-10-05T19:29:34.976 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.970+0000 7fae89306e40 10 chain_cache_entry: cache_locator=default.rgw.meta+root+.bucket.meta.myfoo6:9a9df965-6612-45d4-a45f-6e3c889bacc2.4223.7 2023-10-05T19:29:34.976 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.970+0000 7fae89306e40 1 -- 172.21.15.167:0/3844986762 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:110 8.1f 8:fe511ebf:root::.bucket.meta.myfoo6%3a9a9df965-6612-45d4-a45f-6e3c889bacc2.4223.7:head [delete,create,call version.check_conds in=122b,call version.set in=58b,writefull 0~353 in=353b,setxattr user.rgw.acl (117) in=129b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e32) v8 -- 0x56000af5bfe0 con 0x56000ae95d10 2023-10-05T19:29:34.979 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.974+0000 7fae8663a700 1 -- 172.21.15.167:0/3844986762 <== osd.2 v2:172.21.15.167:6816/30037 20 ==== osd_op_reply(110 .bucket.meta.myfoo6:9a9df965-6612-45d4-a45f-6e3c889bacc2.4223.7 [delete,create,call,call,writefull 0~353,setxattr (117)] v32'2 uv2 ondisk = 0) v8 ==== 417+0+0 (crc 0 0 0) 0x7fae7000bc70 con 0x56000ae95d10 2023-10-05T19:29:34.979 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.974+0000 7fae89306e40 10 cache put: name=default.rgw.meta+root+.bucket.meta.myfoo6:9a9df965-6612-45d4-a45f-6e3c889bacc2.4223.7 info.flags=0x17 2023-10-05T19:29:34.979 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.974+0000 7fae89306e40 10 moving default.rgw.meta+root+.bucket.meta.myfoo6:9a9df965-6612-45d4-a45f-6e3c889bacc2.4223.7 to cache LRU end 2023-10-05T19:29:34.979 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.974+0000 7fae89306e40 10 updating xattr: name=user.rgw.acl bl.length()=117 2023-10-05T19:29:34.979 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.974+0000 7fae89306e40 10 distributing notification oid=default.rgw.control:notify.2 cni=[op: 0, obj: default.rgw.meta:root:.bucket.meta.myfoo6:9a9df965-6612-45d4-a45f-6e3c889bacc2.4223.7, ofs0, ns] 2023-10-05T19:29:34.979 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.974+0000 7fae89306e40 1 -- 172.21.15.167:0/3844986762 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:111 7.15 7:a93a5511:::notify.2:head [notify cookie 94558183866144 in=733b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32) v8 -- 0x56000af5e9c0 con 0x56000aeceb60 2023-10-05T19:29:34.980 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.974+0000 7fae86e3b700 1 -- 172.21.15.167:0/3844986762 <== osd.3 v2:172.21.15.167:6808/30036 24 ==== watch-notify(notify (1) cookie 94558183315152 notify 137438953488 ret 0) v3 ==== 763+0+0 (crc 0 0 0) 0x56000aeb83d0 con 0x56000aeceb60 2023-10-05T19:29:34.980 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.974+0000 7fae7ffff700 10 rgw watcher librados: RGWWatcher::handle_notify() notify_id 137438953488 cookie 94558183315152 notifier 4713 bl.length()=721 2023-10-05T19:29:34.980 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.974+0000 7fae7ffff700 10 rgw watcher librados: cache put: name=default.rgw.meta+root+.bucket.meta.myfoo6:9a9df965-6612-45d4-a45f-6e3c889bacc2.4223.7 info.flags=0x17 2023-10-05T19:29:34.980 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.974+0000 7fae7ffff700 10 rgw watcher librados: moving default.rgw.meta+root+.bucket.meta.myfoo6:9a9df965-6612-45d4-a45f-6e3c889bacc2.4223.7 to cache LRU end 2023-10-05T19:29:34.980 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.974+0000 7fae7ffff700 10 rgw watcher librados: updating xattr: name=user.rgw.acl bl.length()=117 2023-10-05T19:29:34.980 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.974+0000 7fae7ffff700 1 -- 172.21.15.167:0/3844986762 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:112 7.15 7:a93a5511:::notify.2:head [notify-ack in=20b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32) v8 -- 0x7fadd8003520 con 0x56000aeceb60 2023-10-05T19:29:34.980 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.974+0000 7fae86e3b700 1 -- 172.21.15.167:0/3844986762 <== osd.3 v2:172.21.15.167:6808/30036 25 ==== osd_op_reply(111 notify.2 [notify cookie 94558183866144 out=8b] v0'0 uv243 ondisk = 0) v8 ==== 152+0+8 (crc 0 0 0) 0x7fae8002b070 con 0x56000aeceb60 2023-10-05T19:29:34.981 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.978+0000 7fae86e3b700 1 -- 172.21.15.167:0/3844986762 <== osd.3 v2:172.21.15.167:6808/30036 26 ==== watch-notify(notify_complete (2) cookie 94558183866144 notify 137438953488 ret 0) v3 ==== 42+0+48 (crc 0 0 0) 0x56000aeba010 con 0x56000aeceb60 2023-10-05T19:29:34.981 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.978+0000 7fae89306e40 20 handle_bi_update(): bucket=:myfoo6[9a9df965-6612-45d4-a45f-6e3c889bacc2.4223.7]): orig_sources= new_sources= 2023-10-05T19:29:34.981 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.978+0000 7fae89306e40 20 handle_bi_update(): bucket=:myfoo6[9a9df965-6612-45d4-a45f-6e3c889bacc2.4223.7]): potential sources added=[] removed=[] 2023-10-05T19:29:34.981 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.978+0000 7fae89306e40 20 handle_bi_update(): bucket=:myfoo6[9a9df965-6612-45d4-a45f-6e3c889bacc2.4223.7]): orig_dests= new_dests= 2023-10-05T19:29:34.981 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.978+0000 7fae89306e40 20 handle_bi_update(): bucket=:myfoo6[9a9df965-6612-45d4-a45f-6e3c889bacc2.4223.7]): potential dests added=[] removed=[] 2023-10-05T19:29:34.981 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.978+0000 7fae86e3b700 1 -- 172.21.15.167:0/3844986762 <== osd.3 v2:172.21.15.167:6808/30036 27 ==== osd_op_reply(112 notify.2 [notify-ack] v0'0 uv243 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7fae8002b070 con 0x56000aeceb60 2023-10-05T19:29:34.982 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.978+0000 7fae89306e40 1 -- 172.21.15.167:0/3844986762 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:113 8.5 8:a00000ee:users.uid::bar.buckets:head [call user.remove_bucket in=59b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e32) v8 -- 0x56000af568f0 con 0x56000aec89a0 2023-10-05T19:29:34.984 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.978+0000 7fae8763c700 1 -- 172.21.15.167:0/3844986762 <== osd.1 v2:172.21.15.167:6800/30035 36 ==== osd_op_reply(113 bar.buckets [call] v32'7 uv7 ondisk = 0) v8 ==== 155+0+0 (crc 0 0 0) 0x7fae7802f080 con 0x56000aec89a0 2023-10-05T19:29:34.984 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.978+0000 7fae89306e40 20 get_system_obj_state: rctx=0x7fffd9fa8668 obj=default.rgw.meta:root:myfoo6 state=0x56000ad700b0 s->prefetch_data=0 2023-10-05T19:29:34.984 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.978+0000 7fae89306e40 10 cache get: name=default.rgw.meta+root+myfoo6 : miss 2023-10-05T19:29:34.984 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.978+0000 7fae89306e40 1 -- 172.21.15.167:0/3844986762 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:114 8.18 8:1944a257:root::myfoo6:head [call version.read in=11b,getxattrs,stat] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32) v8 -- 0x56000af568f0 con 0x56000aec89a0 2023-10-05T19:29:34.985 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.978+0000 7fae8763c700 1 -- 172.21.15.167:0/3844986762 <== osd.1 v2:172.21.15.167:6800/30035 37 ==== osd_op_reply(114 myfoo6 [call out=48b,getxattrs out=75b,stat out=16b] v0'0 uv16 ondisk = 0) v8 ==== 234+0+139 (crc 0 0 0) 0x7fae780670a0 con 0x56000aec89a0 2023-10-05T19:29:34.985 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.978+0000 7fae89306e40 10 cache put: name=default.rgw.meta+root+myfoo6 info.flags=0x16 2023-10-05T19:29:34.985 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.978+0000 7fae89306e40 10 adding default.rgw.meta+root+myfoo6 to cache LRU end 2023-10-05T19:29:34.985 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.978+0000 7fae89306e40 10 updating xattr: name=ceph.objclass.version bl.length()=42 2023-10-05T19:29:34.985 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.978+0000 7fae89306e40 20 get_system_obj_state: s->obj_tag was set empty 2023-10-05T19:29:34.985 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.978+0000 7fae89306e40 20 Read xattr: user.rgw.idtag 2023-10-05T19:29:34.985 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.978+0000 7fae89306e40 10 cache get: name=default.rgw.meta+root+myfoo6 : type miss (requested=0x13, cached=0x16) 2023-10-05T19:29:34.985 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.978+0000 7fae89306e40 20 rados->read ofs=0 len=0 2023-10-05T19:29:34.985 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.978+0000 7fae89306e40 1 -- 172.21.15.167:0/3844986762 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:115 8.18 8:1944a257:root::myfoo6:head [call version.check_conds in=122b,call version.read in=11b,read 0~0,getxattrs] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32) v8 -- 0x56000af568f0 con 0x56000aec89a0 2023-10-05T19:29:34.986 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.982+0000 7fae8763c700 1 -- 172.21.15.167:0/3844986762 <== osd.1 v2:172.21.15.167:6800/30035 38 ==== osd_op_reply(115 myfoo6 [call,call out=48b,read 0~166 out=166b,getxattrs out=75b] v0'0 uv16 ondisk = 0) v8 ==== 276+0+289 (crc 0 0 0) 0x7fae780670a0 con 0x56000aec89a0 2023-10-05T19:29:34.986 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.982+0000 7fae89306e40 20 rados_obj.operate() r=0 bl.length=166 2023-10-05T19:29:34.986 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.982+0000 7fae89306e40 10 cache put: name=default.rgw.meta+root+myfoo6 info.flags=0x13 2023-10-05T19:29:34.987 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.982+0000 7fae89306e40 10 moving default.rgw.meta+root+myfoo6 to cache LRU end 2023-10-05T19:29:34.987 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.982+0000 7fae89306e40 10 updating xattr: name=ceph.objclass.version bl.length()=42 2023-10-05T19:29:34.987 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.982+0000 7fae89306e40 1 -- 172.21.15.167:0/3844986762 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:116 8.18 8:1944a257:root::myfoo6:head [delete,create,call version.check_conds in=122b,call version.set in=58b,writefull 0~166 in=166b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e32) v8 -- 0x56000af62500 con 0x56000aec89a0 2023-10-05T19:29:34.988 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.982+0000 7fae8763c700 1 -- 172.21.15.167:0/3844986762 <== osd.1 v2:172.21.15.167:6800/30035 39 ==== osd_op_reply(116 myfoo6 [delete,create,call,call,writefull 0~166] v32'17 uv17 ondisk = 0) v8 ==== 318+0+0 (crc 0 0 0) 0x7fae780670a0 con 0x56000aec89a0 2023-10-05T19:29:34.989 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.982+0000 7fae89306e40 10 cache put: name=default.rgw.meta+root+myfoo6 info.flags=0x17 2023-10-05T19:29:34.989 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.982+0000 7fae89306e40 10 moving default.rgw.meta+root+myfoo6 to cache LRU end 2023-10-05T19:29:34.989 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.982+0000 7fae89306e40 10 distributing notification oid=default.rgw.control:notify.3 cni=[op: 0, obj: default.rgw.meta:root:myfoo6, ofs0, ns] 2023-10-05T19:29:34.989 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.986+0000 7fae89306e40 1 -- 172.21.15.167:0/3844986762 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:117 7.e 7:7759931f:::notify.3:head [notify cookie 94558183851248 in=352b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32) v8 -- 0x56000af58e30 con 0x56000ae95d10 2023-10-05T19:29:34.990 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.986+0000 7fae8663a700 1 -- 172.21.15.167:0/3844986762 <== osd.2 v2:172.21.15.167:6816/30037 21 ==== watch-notify(notify (1) cookie 94558183319568 notify 137438953487 ret 0) v3 ==== 382+0+0 (crc 0 0 0) 0x56000aeedb50 con 0x56000ae95d10 2023-10-05T19:29:34.990 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.986+0000 7fae84e37700 10 rgw watcher librados: RGWWatcher::handle_notify() notify_id 137438953487 cookie 94558183319568 notifier 4713 bl.length()=340 2023-10-05T19:29:34.990 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.986+0000 7fae84e37700 10 rgw watcher librados: cache put: name=default.rgw.meta+root+myfoo6 info.flags=0x17 2023-10-05T19:29:34.990 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.986+0000 7fae84e37700 10 rgw watcher librados: moving default.rgw.meta+root+myfoo6 to cache LRU end 2023-10-05T19:29:34.990 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.986+0000 7fae84e37700 1 -- 172.21.15.167:0/3844986762 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:118 7.e 7:7759931f:::notify.3:head [notify-ack in=20b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32) v8 -- 0x7fae6c003d90 con 0x56000ae95d10 2023-10-05T19:29:34.990 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.986+0000 7fae8663a700 1 -- 172.21.15.167:0/3844986762 <== osd.2 v2:172.21.15.167:6816/30037 22 ==== osd_op_reply(117 notify.3 [notify cookie 94558183851248 out=8b] v0'0 uv243 ondisk = 0) v8 ==== 152+0+8 (crc 0 0 0) 0x7fae7000bc70 con 0x56000ae95d10 2023-10-05T19:29:34.990 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.986+0000 7fae8663a700 1 -- 172.21.15.167:0/3844986762 <== osd.2 v2:172.21.15.167:6816/30037 23 ==== osd_op_reply(118 notify.3 [notify-ack] v0'0 uv243 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7fae7000bc70 con 0x56000ae95d10 2023-10-05T19:29:34.991 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.986+0000 7fae8663a700 1 -- 172.21.15.167:0/3844986762 <== osd.2 v2:172.21.15.167:6816/30037 24 ==== watch-notify(notify_complete (2) cookie 94558183851248 notify 137438953487 ret 0) v3 ==== 42+0+48 (crc 0 0 0) 0x56000aeed710 con 0x56000ae95d10 2023-10-05T19:29:34.991 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.986+0000 7fae89306e40 1 -- 172.21.15.167:0/3844986762 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:119 8.19 8:99adfd5d:users.uid::bud.buckets:head [call user.set_buckets_info in=208b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e32) v8 -- 0x56000af5d9b0 con 0x56000aec89a0 2023-10-05T19:29:34.993 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.990+0000 7fae8763c700 1 -- 172.21.15.167:0/3844986762 <== osd.1 v2:172.21.15.167:6800/30035 40 ==== osd_op_reply(119 bud.buckets [call] v32'3 uv3 ondisk = 0) v8 ==== 155+0+0 (crc 0 0 0) 0x7fae780670a0 con 0x56000aec89a0 2023-10-05T19:29:34.993 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.990+0000 7fae89306e40 1 -- 172.21.15.167:0/3844986762 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:120 8.18 8:1944a257:root::myfoo6:head [delete,create,call version.inc in=38b,writefull 0~166 in=166b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e32) v8 -- 0x56000af5f4d0 con 0x56000aec89a0 2023-10-05T19:29:34.995 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.990+0000 7fae8763c700 1 -- 172.21.15.167:0/3844986762 <== osd.1 v2:172.21.15.167:6800/30035 41 ==== osd_op_reply(120 myfoo6 [delete,create,call,writefull 0~166] v32'18 uv18 ondisk = 0) v8 ==== 276+0+0 (crc 0 0 0) 0x7fae780670a0 con 0x56000aec89a0 2023-10-05T19:29:34.996 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.990+0000 7fae89306e40 10 cache put: name=default.rgw.meta+root+myfoo6 info.flags=0x7 2023-10-05T19:29:34.996 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.990+0000 7fae89306e40 10 moving default.rgw.meta+root+myfoo6 to cache LRU end 2023-10-05T19:29:34.996 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.990+0000 7fae89306e40 10 distributing notification oid=default.rgw.control:notify.3 cni=[op: 0, obj: default.rgw.meta:root:myfoo6, ofs0, ns] 2023-10-05T19:29:34.996 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.990+0000 7fae89306e40 1 -- 172.21.15.167:0/3844986762 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:121 7.e 7:7759931f:::notify.3:head [notify cookie 94558183871680 in=328b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32) v8 -- 0x56000af5de00 con 0x56000ae95d10 2023-10-05T19:29:34.996 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.990+0000 7fae8663a700 1 -- 172.21.15.167:0/3844986762 <== osd.2 v2:172.21.15.167:6816/30037 25 ==== watch-notify(notify (1) cookie 94558183319568 notify 137438953488 ret 0) v3 ==== 358+0+0 (crc 0 0 0) 0x56000aeed710 con 0x56000ae95d10 2023-10-05T19:29:34.997 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.990+0000 7fae7ffff700 10 rgw watcher librados: RGWWatcher::handle_notify() notify_id 137438953488 cookie 94558183319568 notifier 4713 bl.length()=316 2023-10-05T19:29:34.997 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.990+0000 7fae7ffff700 10 rgw watcher librados: cache put: name=default.rgw.meta+root+myfoo6 info.flags=0x7 2023-10-05T19:29:34.997 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.990+0000 7fae7ffff700 10 rgw watcher librados: moving default.rgw.meta+root+myfoo6 to cache LRU end 2023-10-05T19:29:34.997 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.990+0000 7fae7ffff700 1 -- 172.21.15.167:0/3844986762 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:122 7.e 7:7759931f:::notify.3:head [notify-ack in=20b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32) v8 -- 0x7fadd80038f0 con 0x56000ae95d10 2023-10-05T19:29:34.997 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.990+0000 7fae8663a700 1 -- 172.21.15.167:0/3844986762 <== osd.2 v2:172.21.15.167:6816/30037 26 ==== osd_op_reply(121 notify.3 [notify cookie 94558183871680 out=8b] v0'0 uv243 ondisk = 0) v8 ==== 152+0+8 (crc 0 0 0) 0x7fae7000bc70 con 0x56000ae95d10 2023-10-05T19:29:34.998 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.994+0000 7fae8663a700 1 -- 172.21.15.167:0/3844986762 <== osd.2 v2:172.21.15.167:6816/30037 27 ==== osd_op_reply(122 notify.3 [notify-ack] v0'0 uv243 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7fae7000bc70 con 0x56000ae95d10 2023-10-05T19:29:34.998 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.994+0000 7fae8663a700 1 -- 172.21.15.167:0/3844986762 <== osd.2 v2:172.21.15.167:6816/30037 28 ==== watch-notify(notify_complete (2) cookie 94558183871680 notify 137438953488 ret 0) v3 ==== 42+0+48 (crc 0 0 0) 0x7fae7000a040 con 0x56000ae95d10 2023-10-05T19:29:34.998 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.994+0000 7fae89306e40 10 int RGWRados::cls_bucket_list_unordered(const DoutPrefixProvider*, RGWBucketInfo&, int, const rgw_obj_index_key&, const string&, uint32_t, bool, std::vector&, bool*, rgw_obj_index_key*, optional_yield, RGWBucketListNameFilter) :myfoo6[9a9df965-6612-45d4-a45f-6e3c889bacc2.4223.7]) start_after="", prefix="", shard_id=-1", num_entries=1100, list_versions=1, force_check_filter is unset 2023-10-05T19:29:34.998 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.994+0000 7fae89306e40 1 -- 172.21.15.167:0/3844986762 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:123 4.c 4:3398558f:::.dir.9a9df965-6612-45d4-a45f-6e3c889bacc2.4223.7.0:head [call rgw.bucket_list in=47b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32) v8 -- 0x56000af5d9b0 con 0x56000ae95d10 2023-10-05T19:29:34.998 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.994+0000 7fae8663a700 1 -- 172.21.15.167:0/3844986762 <== osd.2 v2:172.21.15.167:6816/30037 29 ==== osd_op_reply(123 .dir.9a9df965-6612-45d4-a45f-6e3c889bacc2.4223.7.0 [call out=85b] v0'0 uv1 ondisk = 0) v8 ==== 194+0+85 (crc 0 0 0) 0x7fae7000bc70 con 0x56000ae95d10 2023-10-05T19:29:34.998 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.994+0000 7fae89306e40 1 -- 172.21.15.167:0/3844986762 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:124 4.39 4:9d7aa60a:::.dir.9a9df965-6612-45d4-a45f-6e3c889bacc2.4223.7.1:head [call rgw.bucket_list in=47b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32) v8 -- 0x56000af5ddf0 con 0x56000aeceb60 2023-10-05T19:29:34.999 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.994+0000 7fae86e3b700 1 -- 172.21.15.167:0/3844986762 <== osd.3 v2:172.21.15.167:6808/30036 28 ==== osd_op_reply(124 .dir.9a9df965-6612-45d4-a45f-6e3c889bacc2.4223.7.1 [call out=85b] v0'0 uv3 ondisk = 0) v8 ==== 194+0+85 (crc 0 0 0) 0x7fae8002b070 con 0x56000aeceb60 2023-10-05T19:29:34.999 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.994+0000 7fae89306e40 1 -- 172.21.15.167:0/3844986762 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:125 4.29 4:94bf6966:::.dir.9a9df965-6612-45d4-a45f-6e3c889bacc2.4223.7.2:head [call rgw.bucket_list in=47b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32) v8 -- 0x56000af5ddf0 con 0x56000aeceb60 2023-10-05T19:29:35.000 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.994+0000 7fae86e3b700 1 -- 172.21.15.167:0/3844986762 <== osd.3 v2:172.21.15.167:6808/30036 29 ==== osd_op_reply(125 .dir.9a9df965-6612-45d4-a45f-6e3c889bacc2.4223.7.2 [call out=85b] v0'0 uv13 ondisk = 0) v8 ==== 194+0+85 (crc 0 0 0) 0x7fae8002b070 con 0x56000aeceb60 2023-10-05T19:29:35.000 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.994+0000 7fae89306e40 1 -- 172.21.15.167:0/3844986762 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:126 4.1f 4:f8b953cb:::.dir.9a9df965-6612-45d4-a45f-6e3c889bacc2.4223.7.3:head [call rgw.bucket_list in=47b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32) v8 -- 0x56000af5ddf0 con 0x56000aeceb60 2023-10-05T19:29:35.000 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.994+0000 7fae86e3b700 1 -- 172.21.15.167:0/3844986762 <== osd.3 v2:172.21.15.167:6808/30036 30 ==== osd_op_reply(126 .dir.9a9df965-6612-45d4-a45f-6e3c889bacc2.4223.7.3 [call out=85b] v0'0 uv4 ondisk = 0) v8 ==== 194+0+85 (crc 0 0 0) 0x7fae8002b070 con 0x56000aeceb60 2023-10-05T19:29:35.000 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.998+0000 7fae89306e40 1 -- 172.21.15.167:0/3844986762 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:127 4.30 4:0f0fb17a:::.dir.9a9df965-6612-45d4-a45f-6e3c889bacc2.4223.7.4:head [call rgw.bucket_list in=47b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32) v8 -- 0x56000af5ddf0 con 0x56000aeceb60 2023-10-05T19:29:35.001 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.998+0000 7fae86e3b700 1 -- 172.21.15.167:0/3844986762 <== osd.3 v2:172.21.15.167:6808/30036 31 ==== osd_op_reply(127 .dir.9a9df965-6612-45d4-a45f-6e3c889bacc2.4223.7.4 [call out=85b] v0'0 uv1 ondisk = 0) v8 ==== 194+0+85 (crc 0 0 0) 0x7fae8002b070 con 0x56000aeceb60 2023-10-05T19:29:35.001 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.998+0000 7fae89306e40 1 -- 172.21.15.167:0/3844986762 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:128 4.1c 4:38fdb684:::.dir.9a9df965-6612-45d4-a45f-6e3c889bacc2.4223.7.5:head [call rgw.bucket_list in=47b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32) v8 -- 0x56000af5ddf0 con 0x7fae7000f8d0 2023-10-05T19:29:35.002 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.998+0000 7fae8763c700 1 -- 172.21.15.167:0/3844986762 <== osd.0 v2:172.21.15.118:6800/27690 35 ==== osd_op_reply(128 .dir.9a9df965-6612-45d4-a45f-6e3c889bacc2.4223.7.5 [call out=85b] v0'0 uv1 ondisk = 0) v8 ==== 194+0+85 (crc 0 0 0) 0x7fae780670a0 con 0x7fae7000f8d0 2023-10-05T19:29:35.002 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.998+0000 7fae89306e40 1 -- 172.21.15.167:0/3844986762 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:129 4.30 4:0eea655d:::.dir.9a9df965-6612-45d4-a45f-6e3c889bacc2.4223.7.6:head [call rgw.bucket_list in=47b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32) v8 -- 0x56000af5ddf0 con 0x56000aeceb60 2023-10-05T19:29:35.003 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.998+0000 7fae86e3b700 1 -- 172.21.15.167:0/3844986762 <== osd.3 v2:172.21.15.167:6808/30036 32 ==== osd_op_reply(129 .dir.9a9df965-6612-45d4-a45f-6e3c889bacc2.4223.7.6 [call out=85b] v0'0 uv2 ondisk = 0) v8 ==== 194+0+85 (crc 0 0 0) 0x7fae8002b070 con 0x56000aeceb60 2023-10-05T19:29:35.003 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.998+0000 7fae89306e40 1 -- 172.21.15.167:0/3844986762 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:130 4.1f 4:f8bb09cf:::.dir.9a9df965-6612-45d4-a45f-6e3c889bacc2.4223.7.7:head [call rgw.bucket_list in=47b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32) v8 -- 0x56000af5ddf0 con 0x56000aeceb60 2023-10-05T19:29:35.004 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.998+0000 7fae86e3b700 1 -- 172.21.15.167:0/3844986762 <== osd.3 v2:172.21.15.167:6808/30036 33 ==== osd_op_reply(130 .dir.9a9df965-6612-45d4-a45f-6e3c889bacc2.4223.7.7 [call out=85b] v0'0 uv5 ondisk = 0) v8 ==== 194+0+85 (crc 0 0 0) 0x7fae8002b070 con 0x56000aeceb60 2023-10-05T19:29:35.004 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:34.998+0000 7fae89306e40 1 -- 172.21.15.167:0/3844986762 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:131 4.15 4:aaa2eb30:::.dir.9a9df965-6612-45d4-a45f-6e3c889bacc2.4223.7.8:head [call rgw.bucket_list in=47b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32) v8 -- 0x56000af5ddf0 con 0x56000aeceb60 2023-10-05T19:29:35.004 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:35.002+0000 7fae86e3b700 1 -- 172.21.15.167:0/3844986762 <== osd.3 v2:172.21.15.167:6808/30036 34 ==== osd_op_reply(131 .dir.9a9df965-6612-45d4-a45f-6e3c889bacc2.4223.7.8 [call out=85b] v0'0 uv1 ondisk = 0) v8 ==== 194+0+85 (crc 0 0 0) 0x7fae8002b070 con 0x56000aeceb60 2023-10-05T19:29:35.004 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:35.002+0000 7fae89306e40 1 -- 172.21.15.167:0/3844986762 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:132 4.3a 4:5edbdf2e:::.dir.9a9df965-6612-45d4-a45f-6e3c889bacc2.4223.7.9:head [call rgw.bucket_list in=47b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32) v8 -- 0x56000af5ddf0 con 0x56000aeceb60 2023-10-05T19:29:35.005 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:35.002+0000 7fae86e3b700 1 -- 172.21.15.167:0/3844986762 <== osd.3 v2:172.21.15.167:6808/30036 35 ==== osd_op_reply(132 .dir.9a9df965-6612-45d4-a45f-6e3c889bacc2.4223.7.9 [call out=378b] v0'0 uv3 ondisk = 0) v8 ==== 194+0+378 (crc 0 0 0) 0x7fae8002b070 con 0x56000aeceb60 2023-10-05T19:29:35.005 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:35.002+0000 7fae89306e40 10 cls_bucket_list_unordered: got thirteen 2023-10-05T19:29:35.005 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:35.002+0000 7fae89306e40 1 -- 172.21.15.167:0/3844986762 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:133 4.33 4:cf2dd822:::.dir.9a9df965-6612-45d4-a45f-6e3c889bacc2.4223.7.10:head [call rgw.bucket_list in=47b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32) v8 -- 0x56000af5ddf0 con 0x7fae7000f8d0 2023-10-05T19:29:35.006 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:35.002+0000 7fae8763c700 1 -- 172.21.15.167:0/3844986762 <== osd.0 v2:172.21.15.118:6800/27690 36 ==== osd_op_reply(133 .dir.9a9df965-6612-45d4-a45f-6e3c889bacc2.4223.7.10 [call out=85b] v0'0 uv6 ondisk = 0) v8 ==== 195+0+85 (crc 0 0 0) 0x7fae780670a0 con 0x7fae7000f8d0 2023-10-05T19:29:35.006 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:35.002+0000 7fae89306e40 20 get_obj_state: rctx=0x7fffd9fa86c0 obj=myfoo6:thirteen state=0x56000af58698 s->prefetch_data=0 2023-10-05T19:29:35.007 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:35.002+0000 7fae89306e40 1 -- 172.21.15.167:0/3844986762 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:134 3.17s0 3:ebcd1212:::9a9df965-6612-45d4-a45f-6e3c889bacc2.4223.7_thirteen:head [getxattrs,stat] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32) v8 -- 0x56000af5ddf0 con 0x56000aec89a0 2023-10-05T19:29:35.007 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:35.002+0000 7fae8763c700 1 -- 172.21.15.167:0/3844986762 <== osd.1 v2:172.21.15.167:6800/30035 42 ==== osd_op_reply(134 9a9df965-6612-45d4-a45f-6e3c889bacc2.4223.7_thirteen [getxattrs out=856b,stat out=16b] v0'0 uv1 ondisk = 0) v8 ==== 238+0+872 (crc 0 0 0) 0x7fae780670a0 con 0x56000aec89a0 2023-10-05T19:29:35.008 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:35.002+0000 7fae89306e40 10 manifest: total_size = 8 2023-10-05T19:29:35.008 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:35.002+0000 7fae89306e40 20 get_obj_state: setting s->obj_tag to 9a9df965-6612-45d4-a45f-6e3c889bacc2.4221.10764886682478415245 2023-10-05T19:29:35.008 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:35.002+0000 7fae89306e40 20 Read xattr rgw_rados: user.rgw.acl 2023-10-05T19:29:35.008 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:35.002+0000 7fae89306e40 20 Read xattr rgw_rados: user.rgw.content_type 2023-10-05T19:29:35.008 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:35.002+0000 7fae89306e40 20 Read xattr rgw_rados: user.rgw.etag 2023-10-05T19:29:35.008 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:35.002+0000 7fae89306e40 20 Read xattr rgw_rados: user.rgw.idtag 2023-10-05T19:29:35.008 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:35.002+0000 7fae89306e40 20 Read xattr rgw_rados: user.rgw.manifest 2023-10-05T19:29:35.008 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:35.002+0000 7fae89306e40 20 Read xattr rgw_rados: user.rgw.pg_ver 2023-10-05T19:29:35.008 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:35.002+0000 7fae89306e40 20 Read xattr rgw_rados: user.rgw.source_zone 2023-10-05T19:29:35.009 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:35.002+0000 7fae89306e40 20 Read xattr rgw_rados: user.rgw.tail_tag 2023-10-05T19:29:35.009 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:35.002+0000 7fae89306e40 20 get_obj_state: rctx=0x7fffd9fa86c0 obj=myfoo6:thirteen state=0x56000af58698 s->prefetch_data=0 2023-10-05T19:29:35.009 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:35.002+0000 7fae89306e40 20 get_system_obj_state: rctx=0x7fffd9fa6d90 obj=default.rgw.meta:root:.bucket.meta.myfoo6:9a9df965-6612-45d4-a45f-6e3c889bacc2.4223.7 state=0x56000ad700b0 s->prefetch_data=0 2023-10-05T19:29:35.009 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:35.002+0000 7fae89306e40 10 cache get: name=default.rgw.meta+root+.bucket.meta.myfoo6:9a9df965-6612-45d4-a45f-6e3c889bacc2.4223.7 : hit (requested=0x16, cached=0x17) 2023-10-05T19:29:35.009 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:35.002+0000 7fae89306e40 20 get_system_obj_state: s->obj_tag was set empty 2023-10-05T19:29:35.009 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:35.002+0000 7fae89306e40 20 Read xattr: user.rgw.acl 2023-10-05T19:29:35.009 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:35.002+0000 7fae89306e40 20 Read xattr: user.rgw.idtag 2023-10-05T19:29:35.009 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:35.002+0000 7fae89306e40 10 cache get: name=default.rgw.meta+root+.bucket.meta.myfoo6:9a9df965-6612-45d4-a45f-6e3c889bacc2.4223.7 : hit (requested=0x13, cached=0x17) 2023-10-05T19:29:35.009 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:35.002+0000 7fae89306e40 10 chain_cache_entry: cache_locator=default.rgw.meta+root+.bucket.meta.myfoo6:9a9df965-6612-45d4-a45f-6e3c889bacc2.4223.7 2023-10-05T19:29:35.009 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:35.002+0000 7fae89306e40 20 bucket index object: default.rgw.buckets.index:.dir.9a9df965-6612-45d4-a45f-6e3c889bacc2.4223.7.9 2023-10-05T19:29:35.009 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:35.002+0000 7fae89306e40 1 -- 172.21.15.167:0/3844986762 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:135 4.3a 4:5edbdf2e:::.dir.9a9df965-6612-45d4-a45f-6e3c889bacc2.4223.7.9:head [stat,call rgw.guard_bucket_resharding in=36b,call rgw.bucket_prepare_op in=187b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e32) v8 -- 0x56000af660d0 con 0x56000aeceb60 2023-10-05T19:29:35.010 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:35.006+0000 7fae86e3b700 1 -- 172.21.15.167:0/3844986762 <== osd.3 v2:172.21.15.167:6808/30036 36 ==== osd_op_reply(135 .dir.9a9df965-6612-45d4-a45f-6e3c889bacc2.4223.7.9 [stat,call,call] v32'4 uv4 ondisk = 0) v8 ==== 278+0+0 (crc 0 0 0) 0x7fae8002b070 con 0x56000aeceb60 2023-10-05T19:29:35.010 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:35.006+0000 7fae89306e40 1 -- 172.21.15.167:0/3844986762 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:136 3.17s0 3:ebcd1212:::9a9df965-6612-45d4-a45f-6e3c889bacc2.4223.7_thirteen:head [cmpxattr user.rgw.idtag (63) op 1 mode 1 in=77b,setxattr user.rgw.acl (117) in=129b,setxattr user.rgw.idtag (33) in=47b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e32) v8 -- 0x56000af645a0 con 0x56000aec89a0 2023-10-05T19:29:35.014 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:35.010+0000 7fae8763c700 1 -- 172.21.15.167:0/3844986762 <== osd.1 v2:172.21.15.167:6800/30035 43 ==== osd_op_reply(136 9a9df965-6612-45d4-a45f-6e3c889bacc2.4223.7_thirteen [cmpxattr (63) op 1 mode 1,setxattr (117),setxattr (33)] v32'2 uv2 ondisk = 0) v8 ==== 280+0+0 (crc 0 0 0) 0x7fae780670a0 con 0x56000aec89a0 2023-10-05T19:29:35.014 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:35.010+0000 7fae89306e40 1 -- 172.21.15.167:0/3844986762 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:137 4.3a 4:5edbdf2e:::.dir.9a9df965-6612-45d4-a45f-6e3c889bacc2.4223.7.9:head [stat,call rgw.guard_bucket_resharding in=36b,call rgw.bucket_complete_op in=267b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e32) v8 -- 0x56000af65fa0 con 0x56000aeceb60 2023-10-05T19:29:35.014 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:35.010+0000 7fae89306e40 10 cache get: name=default.rgw.log++bucket.sync-source-hints.myfoo6 : miss 2023-10-05T19:29:35.014 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:35.010+0000 7fae89306e40 20 rados->read ofs=0 len=0 2023-10-05T19:29:35.014 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:35.010+0000 7fae89306e40 1 -- 172.21.15.167:0/3844986762 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:138 6.9 6:936e6ff6:::bucket.sync-source-hints.myfoo6:head [call version.read in=11b,read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32) v8 -- 0x56000af684b0 con 0x56000aeceb60 2023-10-05T19:29:35.015 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:35.010+0000 7fae86e3b700 1 -- 172.21.15.167:0/3844986762 <== osd.3 v2:172.21.15.167:6808/30036 37 ==== osd_op_reply(138 bucket.sync-source-hints.myfoo6 [call,read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 ==== 217+0+0 (crc 0 0 0) 0x7fae8002b070 con 0x56000aeceb60 2023-10-05T19:29:35.015 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:35.010+0000 7fae89306e40 20 rados_obj.operate() r=-2 bl.length=0 2023-10-05T19:29:35.015 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:35.010+0000 7fae89306e40 10 cache put: name=default.rgw.log++bucket.sync-source-hints.myfoo6 info.flags=0x0 2023-10-05T19:29:35.015 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:35.010+0000 7fae89306e40 10 adding default.rgw.log++bucket.sync-source-hints.myfoo6 to cache LRU end 2023-10-05T19:29:35.015 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:35.010+0000 7fae89306e40 10 cache get: name=default.rgw.log++bucket.sync-target-hints.myfoo6 : miss 2023-10-05T19:29:35.015 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:35.010+0000 7fae89306e40 20 rados->read ofs=0 len=0 2023-10-05T19:29:35.015 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:35.010+0000 7fae89306e40 1 -- 172.21.15.167:0/3844986762 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:139 6.19 6:9e3df8ab:::bucket.sync-target-hints.myfoo6:head [call version.read in=11b,read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32) v8 -- 0x56000af684b0 con 0x56000aec89a0 2023-10-05T19:29:35.016 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:35.010+0000 7fae8763c700 1 -- 172.21.15.167:0/3844986762 <== osd.1 v2:172.21.15.167:6800/30035 44 ==== osd_op_reply(139 bucket.sync-target-hints.myfoo6 [call,read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 ==== 217+0+0 (crc 0 0 0) 0x7fae780670a0 con 0x56000aec89a0 2023-10-05T19:29:35.016 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:35.010+0000 7fae89306e40 20 rados_obj.operate() r=-2 bl.length=0 2023-10-05T19:29:35.016 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:35.010+0000 7fae89306e40 10 cache put: name=default.rgw.log++bucket.sync-target-hints.myfoo6 info.flags=0x0 2023-10-05T19:29:35.017 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:35.010+0000 7fae89306e40 10 adding default.rgw.log++bucket.sync-target-hints.myfoo6 to cache LRU end 2023-10-05T19:29:35.017 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:35.010+0000 7fae89306e40 10 chain_cache_entry: cache_locator= 2023-10-05T19:29:35.017 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:35.010+0000 7fae89306e40 20 chain_cache_entry: couldn't find cache locator 2023-10-05T19:29:35.017 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:35.010+0000 7fae89306e40 20 couldn't put bucket_sync_policy cache entry, might have raced with data changes 2023-10-05T19:29:35.017 INFO:teuthology.orchestra.run.smithi167.stderr:1 objects processed in :myfoo6[9a9df965-6612-45d4-a45f-6e3c889bacc2.4223.7]). Next marker thirteen 2023-10-05T19:29:35.017 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:35.010+0000 7fae89306e40 1 -- 172.21.15.167:0/3844986762 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:140 8.4 8:267425a8:users.uid::bud:head [delete,create,call version.set in=58b,writefull 0~346 in=346b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e32) v8 -- 0x56000af584b0 con 0x56000ae95d10 2023-10-05T19:29:35.017 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:35.014+0000 7fae86e3b700 1 -- 172.21.15.167:0/3844986762 <== osd.3 v2:172.21.15.167:6808/30036 38 ==== osd_op_reply(137 .dir.9a9df965-6612-45d4-a45f-6e3c889bacc2.4223.7.9 [stat,call,call] v32'5 uv5 ondisk = 0) v8 ==== 278+0+0 (crc 0 0 0) 0x7fae8002b070 con 0x56000aeceb60 2023-10-05T19:29:35.018 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:35.014+0000 7fae84e37700 20 handle_completion(): completion ok for obj=thirteen 2023-10-05T19:29:35.019 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:35.014+0000 7fae8663a700 1 -- 172.21.15.167:0/3844986762 <== osd.2 v2:172.21.15.167:6816/30037 30 ==== osd_op_reply(140 bud [delete,create,call,writefull 0~346] v32'2 uv2 ondisk = 0) v8 ==== 273+0+0 (crc 0 0 0) 0x7fae7000bc70 con 0x56000ae95d10 2023-10-05T19:29:35.019 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:35.014+0000 7fae89306e40 10 cache put: name=default.rgw.meta+users.uid+bud info.flags=0x17 2023-10-05T19:29:35.019 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:35.014+0000 7fae89306e40 10 moving default.rgw.meta+users.uid+bud to cache LRU end 2023-10-05T19:29:35.019 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:35.014+0000 7fae89306e40 10 distributing notification oid=default.rgw.control:notify.5 cni=[op: 0, obj: default.rgw.meta:users.uid:bud, ofs0, ns] 2023-10-05T19:29:35.019 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:35.014+0000 7fae89306e40 1 -- 172.21.15.167:0/3844986762 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:141 7.3 7:c609908c:::notify.5:head [notify cookie 94558183852752 in=534b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32) v8 -- 0x56000af584b0 con 0x7fae7000f8d0 2023-10-05T19:29:35.020 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:35.014+0000 7fae8763c700 1 -- 172.21.15.167:0/3844986762 <== osd.0 v2:172.21.15.118:6800/27690 37 ==== watch-notify(notify (1) cookie 94558183323136 notify 137438953506 ret 0) v3 ==== 564+0+0 (crc 0 0 0) 0x7fae78002bd0 con 0x7fae7000f8d0 2023-10-05T19:29:35.020 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:35.014+0000 7fae7ffff700 10 rgw watcher librados: RGWWatcher::handle_notify() notify_id 137438953506 cookie 94558183323136 notifier 4713 bl.length()=522 2023-10-05T19:29:35.020 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:35.014+0000 7fae7ffff700 10 rgw watcher librados: cache put: name=default.rgw.meta+users.uid+bud info.flags=0x17 2023-10-05T19:29:35.020 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:35.014+0000 7fae7ffff700 10 rgw watcher librados: moving default.rgw.meta+users.uid+bud to cache LRU end 2023-10-05T19:29:35.020 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:35.014+0000 7fae7ffff700 1 -- 172.21.15.167:0/3844986762 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:142 7.3 7:c609908c:::notify.5:head [notify-ack in=20b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32) v8 -- 0x7fadd8003580 con 0x7fae7000f8d0 2023-10-05T19:29:35.020 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:35.014+0000 7fae8763c700 1 -- 172.21.15.167:0/3844986762 <== osd.0 v2:172.21.15.118:6800/27690 38 ==== osd_op_reply(141 notify.5 [notify cookie 94558183852752 out=8b] v0'0 uv243 ondisk = 0) v8 ==== 152+0+8 (crc 0 0 0) 0x7fae780670a0 con 0x7fae7000f8d0 2023-10-05T19:29:35.021 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:35.014+0000 7fae8763c700 1 -- 172.21.15.167:0/3844986762 <== osd.0 v2:172.21.15.118:6800/27690 39 ==== osd_op_reply(142 notify.5 [notify-ack] v0'0 uv243 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7fae780670a0 con 0x7fae7000f8d0 2023-10-05T19:29:35.021 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:35.018+0000 7fae8763c700 1 -- 172.21.15.167:0/3844986762 <== osd.0 v2:172.21.15.118:6800/27690 40 ==== watch-notify(notify_complete (2) cookie 94558183852752 notify 137438953506 ret 0) v3 ==== 42+0+48 (crc 0 0 0) 0x7fae780338e0 con 0x7fae7000f8d0 2023-10-05T19:29:35.021 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:35.018+0000 7fae89306e40 1 -- 172.21.15.167:0/3844986762 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:143 8.a 8:50c8eddb:users.keys::NX5QOQKC6BH2IDN8HC7A:head [delete,create,writefull 0~7 in=7b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e32) v8 -- 0x56000af584b0 con 0x56000aec89a0 2023-10-05T19:29:35.023 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:35.018+0000 7fae8763c700 1 -- 172.21.15.167:0/3844986762 <== osd.1 v2:172.21.15.167:6800/30035 45 ==== osd_op_reply(143 NX5QOQKC6BH2IDN8HC7A [delete,create,writefull 0~7] v32'2 uv2 ondisk = 0) v8 ==== 248+0+0 (crc 0 0 0) 0x7fae780670a0 con 0x56000aec89a0 2023-10-05T19:29:35.023 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:35.018+0000 7fae89306e40 10 cache put: name=default.rgw.meta+users.keys+NX5QOQKC6BH2IDN8HC7A info.flags=0x7 2023-10-05T19:29:35.023 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:35.018+0000 7fae89306e40 10 adding default.rgw.meta+users.keys+NX5QOQKC6BH2IDN8HC7A to cache LRU end 2023-10-05T19:29:35.023 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:35.018+0000 7fae89306e40 10 distributing notification oid=default.rgw.control:notify.5 cni=[op: 0, obj: default.rgw.meta:users.keys:NX5QOQKC6BH2IDN8HC7A, ofs0, ns] 2023-10-05T19:29:35.024 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:35.018+0000 7fae89306e40 1 -- 172.21.15.167:0/3844986762 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:144 7.3 7:c609908c:::notify.5:head [notify cookie 94558183852752 in=189b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32) v8 -- 0x56000af584b0 con 0x7fae7000f8d0 2023-10-05T19:29:35.024 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:35.018+0000 7fae8763c700 1 -- 172.21.15.167:0/3844986762 <== osd.0 v2:172.21.15.118:6800/27690 41 ==== watch-notify(notify (1) cookie 94558183323136 notify 137438953507 ret 0) v3 ==== 219+0+0 (crc 0 0 0) 0x7fae780338e0 con 0x7fae7000f8d0 2023-10-05T19:29:35.024 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:35.018+0000 7fae84e37700 10 rgw watcher librados: RGWWatcher::handle_notify() notify_id 137438953507 cookie 94558183323136 notifier 4713 bl.length()=177 2023-10-05T19:29:35.024 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:35.018+0000 7fae84e37700 10 rgw watcher librados: cache put: name=default.rgw.meta+users.keys+NX5QOQKC6BH2IDN8HC7A info.flags=0x7 2023-10-05T19:29:35.025 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:35.018+0000 7fae84e37700 10 rgw watcher librados: moving default.rgw.meta+users.keys+NX5QOQKC6BH2IDN8HC7A to cache LRU end 2023-10-05T19:29:35.025 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:35.018+0000 7fae8763c700 1 -- 172.21.15.167:0/3844986762 <== osd.0 v2:172.21.15.118:6800/27690 42 ==== osd_op_reply(144 notify.5 [notify cookie 94558183852752 out=8b] v0'0 uv243 ondisk = 0) v8 ==== 152+0+8 (crc 0 0 0) 0x7fae780670a0 con 0x7fae7000f8d0 2023-10-05T19:29:35.025 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:35.018+0000 7fae84e37700 1 -- 172.21.15.167:0/3844986762 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:145 7.3 7:c609908c:::notify.5:head [notify-ack in=20b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32) v8 -- 0x7fae6c003d90 con 0x7fae7000f8d0 2023-10-05T19:29:35.025 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:35.022+0000 7fae8763c700 1 -- 172.21.15.167:0/3844986762 <== osd.0 v2:172.21.15.118:6800/27690 43 ==== watch-notify(notify_complete (2) cookie 94558183852752 notify 137438953507 ret 0) v3 ==== 42+0+48 (crc 0 0 0) 0x7fae78033a70 con 0x7fae7000f8d0 2023-10-05T19:29:35.026 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:35.022+0000 7fae8763c700 1 -- 172.21.15.167:0/3844986762 <== osd.0 v2:172.21.15.118:6800/27690 44 ==== osd_op_reply(145 notify.5 [notify-ack] v0'0 uv243 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7fae780670a0 con 0x7fae7000f8d0 2023-10-05T19:29:35.026 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:35.022+0000 7fae89306e40 10 removing user index: bar 2023-10-05T19:29:35.026 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:35.022+0000 7fae89306e40 10 removing default.rgw.meta+users.uid+bar from cache 2023-10-05T19:29:35.026 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:35.022+0000 7fae89306e40 10 distributing notification oid=default.rgw.control:notify.7 cni=[op: 1, obj: default.rgw.meta:users.uid:bar, ofs0, ns] 2023-10-05T19:29:35.026 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:35.022+0000 7fae89306e40 1 -- 172.21.15.167:0/3844986762 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:146 7.9 7:93e5b521:::notify.7:head [notify cookie 94558183852752 in=164b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32) v8 -- 0x56000af584b0 con 0x7fae7000f8d0 2023-10-05T19:29:35.027 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:35.022+0000 7fae8763c700 1 -- 172.21.15.167:0/3844986762 <== osd.0 v2:172.21.15.118:6800/27690 45 ==== watch-notify(notify (1) cookie 94558183326800 notify 137438953508 ret 0) v3 ==== 194+0+0 (crc 0 0 0) 0x7fae78033a70 con 0x7fae7000f8d0 2023-10-05T19:29:35.027 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:35.022+0000 7fae7ffff700 10 rgw watcher librados: RGWWatcher::handle_notify() notify_id 137438953508 cookie 94558183326800 notifier 4713 bl.length()=152 2023-10-05T19:29:35.027 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:35.022+0000 7fae7ffff700 1 -- 172.21.15.167:0/3844986762 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:147 7.9 7:93e5b521:::notify.7:head [notify-ack in=20b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32) v8 -- 0x7fadd8003580 con 0x7fae7000f8d0 2023-10-05T19:29:35.027 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:35.022+0000 7fae8763c700 1 -- 172.21.15.167:0/3844986762 <== osd.0 v2:172.21.15.118:6800/27690 46 ==== osd_op_reply(146 notify.7 [notify cookie 94558183852752 out=8b] v0'0 uv243 ondisk = 0) v8 ==== 152+0+8 (crc 0 0 0) 0x7fae780670a0 con 0x7fae7000f8d0 2023-10-05T19:29:35.027 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:35.022+0000 7fae8763c700 1 -- 172.21.15.167:0/3844986762 <== osd.0 v2:172.21.15.118:6800/27690 47 ==== osd_op_reply(147 notify.7 [notify-ack] v0'0 uv243 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7fae780670a0 con 0x7fae7000f8d0 2023-10-05T19:29:35.028 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:35.022+0000 7fae8763c700 1 -- 172.21.15.167:0/3844986762 <== osd.0 v2:172.21.15.118:6800/27690 48 ==== watch-notify(notify_complete (2) cookie 94558183852752 notify 137438953508 ret 0) v3 ==== 42+0+48 (crc 0 0 0) 0x7fae78033ca0 con 0x7fae7000f8d0 2023-10-05T19:29:35.028 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:35.022+0000 7fae89306e40 1 -- 172.21.15.167:0/3844986762 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:148 8.e 8:718f6a54:users.uid::bar:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e32) v8 -- 0x56000af584b0 con 0x7fae7000f8d0 2023-10-05T19:29:35.030 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:35.026+0000 7fae8763c700 1 -- 172.21.15.167:0/3844986762 <== osd.0 v2:172.21.15.118:6800/27690 49 ==== osd_op_reply(148 bar [delete] v32'4 uv4 ondisk = 0) v8 ==== 147+0+0 (crc 0 0 0) 0x7fae780670a0 con 0x7fae7000f8d0 2023-10-05T19:29:35.030 INFO:teuthology.orchestra.run.smithi167.stdout:{ 2023-10-05T19:29:35.030 INFO:teuthology.orchestra.run.smithi167.stdout: "user_id": "bud", 2023-10-05T19:29:35.030 INFO:teuthology.orchestra.run.smithi167.stdout: "display_name": "Bar", 2023-10-05T19:29:35.031 INFO:teuthology.orchestra.run.smithi167.stdout: "email": "", 2023-10-05T19:29:35.031 INFO:teuthology.orchestra.run.smithi167.stdout: "suspended": 0, 2023-10-05T19:29:35.031 INFO:teuthology.orchestra.run.smithi167.stdout: "max_buckets": 4, 2023-10-05T19:29:35.031 INFO:teuthology.orchestra.run.smithi167.stdout: "subusers": [], 2023-10-05T19:29:35.031 INFO:teuthology.orchestra.run.smithi167.stdout: "keys": [ 2023-10-05T19:29:35.031 INFO:teuthology.orchestra.run.smithi167.stdout: { 2023-10-05T19:29:35.031 INFO:teuthology.orchestra.run.smithi167.stdout: "user": "bud", 2023-10-05T19:29:35.031 INFO:teuthology.orchestra.run.smithi167.stdout: "access_key": "NX5QOQKC6BH2IDN8HC7A", 2023-10-05T19:29:35.031 INFO:teuthology.orchestra.run.smithi167.stdout: "secret_key": "LnEsqNNqZIpkzauboDcLXLcYaWwLQ3Kop0zAnKIn" 2023-10-05T19:29:35.031 INFO:teuthology.orchestra.run.smithi167.stdout: } 2023-10-05T19:29:35.031 INFO:teuthology.orchestra.run.smithi167.stdout: ], 2023-10-05T19:29:35.032 INFO:teuthology.orchestra.run.smithi167.stdout: "swift_keys": [], 2023-10-05T19:29:35.032 INFO:teuthology.orchestra.run.smithi167.stdout: "caps": [], 2023-10-05T19:29:35.032 INFO:teuthology.orchestra.run.smithi167.stdout: "op_mask": "read, write, delete", 2023-10-05T19:29:35.032 INFO:teuthology.orchestra.run.smithi167.stdout: "default_placement": "", 2023-10-05T19:29:35.032 INFO:teuthology.orchestra.run.smithi167.stdout: "default_storage_class": "", 2023-10-05T19:29:35.032 INFO:teuthology.orchestra.run.smithi167.stdout: "placement_tags": [], 2023-10-05T19:29:35.032 INFO:teuthology.orchestra.run.smithi167.stdout: "bucket_quota": { 2023-10-05T19:29:35.032 INFO:teuthology.orchestra.run.smithi167.stdout: "enabled": false, 2023-10-05T19:29:35.032 INFO:teuthology.orchestra.run.smithi167.stdout: "check_on_raw": false, 2023-10-05T19:29:35.032 INFO:teuthology.orchestra.run.smithi167.stdout: "max_size": -1, 2023-10-05T19:29:35.032 INFO:teuthology.orchestra.run.smithi167.stdout: "max_size_kb": 0, 2023-10-05T19:29:35.032 INFO:teuthology.orchestra.run.smithi167.stdout: "max_objects": -1 2023-10-05T19:29:35.033 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2023-10-05T19:29:35.033 INFO:teuthology.orchestra.run.smithi167.stdout: "user_quota": { 2023-10-05T19:29:35.033 INFO:teuthology.orchestra.run.smithi167.stdout: "enabled": false, 2023-10-05T19:29:35.033 INFO:teuthology.orchestra.run.smithi167.stdout: "check_on_raw": false, 2023-10-05T19:29:35.033 INFO:teuthology.orchestra.run.smithi167.stdout: "max_size": -1, 2023-10-05T19:29:35.033 INFO:teuthology.orchestra.run.smithi167.stdout: "max_size_kb": 0, 2023-10-05T19:29:35.033 INFO:teuthology.orchestra.run.smithi167.stdout: "max_objects": -1 2023-10-05T19:29:35.033 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2023-10-05T19:29:35.033 INFO:teuthology.orchestra.run.smithi167.stdout: "temp_url_keys": [], 2023-10-05T19:29:35.033 INFO:teuthology.orchestra.run.smithi167.stdout: "type": "rgw", 2023-10-05T19:29:35.033 INFO:teuthology.orchestra.run.smithi167.stdout: "mfa_ids": [] 2023-10-05T19:29:35.033 INFO:teuthology.orchestra.run.smithi167.stdout:} 2023-10-05T19:29:35.034 INFO:teuthology.orchestra.run.smithi167.stdout: 2023-10-05T19:29:35.034 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:35.026+0000 7fae89306e40 1 -- 172.21.15.167:0/3844986762 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:149 7.1f 7:f95f44c2:::notify.0:head [watch unwatch cookie 94558183306336] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e32) v8 -- 0x56000af584b0 con 0x56000ae95d10 2023-10-05T19:29:35.034 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:35.030+0000 7fae8663a700 1 -- 172.21.15.167:0/3844986762 <== osd.2 v2:172.21.15.167:6816/30037 31 ==== osd_op_reply(149 notify.0 [watch unwatch cookie 94558183306336] v32'245 uv243 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7fae7000bc70 con 0x56000ae95d10 2023-10-05T19:29:35.034 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:35.030+0000 7fae89306e40 20 remove_watcher() i=0 2023-10-05T19:29:35.034 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:35.030+0000 7fae89306e40 2 removed watcher, disabling cache 2023-10-05T19:29:35.035 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:35.030+0000 7fae89306e40 1 -- 172.21.15.167:0/3844986762 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:150 7.0 7:05bf5b68:::notify.1:head [watch unwatch cookie 94558183308624] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e32) v8 -- 0x56000af584b0 con 0x56000aec89a0 2023-10-05T19:29:35.036 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:35.030+0000 7fae8763c700 1 -- 172.21.15.167:0/3844986762 <== osd.1 v2:172.21.15.167:6800/30035 46 ==== osd_op_reply(150 notify.1 [watch unwatch cookie 94558183308624] v32'245 uv243 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7fae780670a0 con 0x56000aec89a0 2023-10-05T19:29:35.036 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:35.030+0000 7fae89306e40 20 remove_watcher() i=1 2023-10-05T19:29:35.036 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:35.030+0000 7fae89306e40 1 -- 172.21.15.167:0/3844986762 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:151 7.15 7:a93a5511:::notify.2:head [watch unwatch cookie 94558183315152] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e32) v8 -- 0x56000af584b0 con 0x56000aeceb60 2023-10-05T19:29:35.039 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:35.034+0000 7fae86e3b700 1 -- 172.21.15.167:0/3844986762 <== osd.3 v2:172.21.15.167:6808/30036 39 ==== osd_op_reply(151 notify.2 [watch unwatch cookie 94558183315152] v32'245 uv243 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7fae8002b070 con 0x56000aeceb60 2023-10-05T19:29:35.039 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:35.034+0000 7fae89306e40 20 remove_watcher() i=2 2023-10-05T19:29:35.039 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:35.034+0000 7fae89306e40 1 -- 172.21.15.167:0/3844986762 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:152 7.e 7:7759931f:::notify.3:head [watch unwatch cookie 94558183319568] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e32) v8 -- 0x56000af584b0 con 0x56000ae95d10 2023-10-05T19:29:35.041 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:35.038+0000 7fae8663a700 1 -- 172.21.15.167:0/3844986762 <== osd.2 v2:172.21.15.167:6816/30037 32 ==== osd_op_reply(152 notify.3 [watch unwatch cookie 94558183319568] v32'245 uv243 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7fae7000bc70 con 0x56000ae95d10 2023-10-05T19:29:35.041 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:35.038+0000 7fae89306e40 20 remove_watcher() i=3 2023-10-05T19:29:35.041 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:35.038+0000 7fae89306e40 1 -- 172.21.15.167:0/3844986762 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:153 7.d 7:b4812045:::notify.4:head [watch unwatch cookie 94558183318304] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e32) v8 -- 0x56000af584b0 con 0x56000aec89a0 2023-10-05T19:29:35.044 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:35.038+0000 7fae8763c700 1 -- 172.21.15.167:0/3844986762 <== osd.1 v2:172.21.15.167:6800/30035 47 ==== osd_op_reply(153 notify.4 [watch unwatch cookie 94558183318304] v32'245 uv243 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7fae780670a0 con 0x56000aec89a0 2023-10-05T19:29:35.044 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:35.038+0000 7fae89306e40 20 remove_watcher() i=4 2023-10-05T19:29:35.044 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:35.038+0000 7fae89306e40 1 -- 172.21.15.167:0/3844986762 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:154 7.3 7:c609908c:::notify.5:head [watch unwatch cookie 94558183323136] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e32) v8 -- 0x56000af584b0 con 0x7fae7000f8d0 2023-10-05T19:29:35.047 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:35.042+0000 7fae8763c700 1 -- 172.21.15.167:0/3844986762 <== osd.0 v2:172.21.15.118:6800/27690 50 ==== osd_op_reply(154 notify.5 [watch unwatch cookie 94558183323136] v32'245 uv243 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7fae780670a0 con 0x7fae7000f8d0 2023-10-05T19:29:35.047 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:35.042+0000 7fae89306e40 20 remove_watcher() i=5 2023-10-05T19:29:35.047 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:35.042+0000 7fae89306e40 1 -- 172.21.15.167:0/3844986762 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:155 7.14 7:2b04a3e9:::notify.6:head [watch unwatch cookie 94558183328624] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e32) v8 -- 0x56000af584b0 con 0x7fae7000f8d0 2023-10-05T19:29:35.050 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:35.046+0000 7fae8763c700 1 -- 172.21.15.167:0/3844986762 <== osd.0 v2:172.21.15.118:6800/27690 51 ==== osd_op_reply(155 notify.6 [watch unwatch cookie 94558183328624] v32'245 uv243 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7fae780670a0 con 0x7fae7000f8d0 2023-10-05T19:29:35.050 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:35.046+0000 7fae89306e40 20 remove_watcher() i=6 2023-10-05T19:29:35.050 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:35.046+0000 7fae89306e40 1 -- 172.21.15.167:0/3844986762 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:156 7.9 7:93e5b521:::notify.7:head [watch unwatch cookie 94558183326800] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e32) v8 -- 0x56000af584b0 con 0x7fae7000f8d0 2023-10-05T19:29:35.052 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:35.050+0000 7fae8763c700 1 -- 172.21.15.167:0/3844986762 <== osd.0 v2:172.21.15.118:6800/27690 52 ==== osd_op_reply(156 notify.7 [watch unwatch cookie 94558183326800] v32'245 uv243 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7fae780670a0 con 0x7fae7000f8d0 2023-10-05T19:29:35.053 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:35.050+0000 7fae89306e40 20 remove_watcher() i=7 2023-10-05T19:29:35.054 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:35.050+0000 7fadd3fff700 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2023-10-05T19:29:35.054 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:35.050+0000 7fae89306e40 1 -- 172.21.15.167:0/3844986762 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:157 6.4 6:22d26bf9:::data_loggenerations_metadata:head [watch unwatch cookie 94558183332848] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e32) v8 -- 0x56000af584b0 con 0x56000aec89a0 2023-10-05T19:29:35.057 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:35.054+0000 7fae8763c700 1 -- 172.21.15.167:0/3844986762 <== osd.1 v2:172.21.15.167:6800/30035 48 ==== osd_op_reply(157 data_loggenerations_metadata [watch unwatch cookie 94558183332848] v32'236 uv1 ondisk = 0) v8 ==== 172+0+0 (crc 0 0 0) 0x7fae780670a0 con 0x56000aec89a0 2023-10-05T19:29:35.058 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:35.054+0000 7fae89306e40 1 -- 172.21.15.167:0/3844986762 >> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] conn(0x7fae7000f8d0 msgr2=0x7fae7000e360 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-05T19:29:35.058 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:35.054+0000 7fae89306e40 1 --2- 172.21.15.167:0/3844986762 >> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] conn(0x7fae7000f8d0 0x7fae7000e360 crc :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-05T19:29:35.058 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:35.054+0000 7fae89306e40 1 -- 172.21.15.167:0/3844986762 >> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] conn(0x56000aec89a0 msgr2=0x56000aecad80 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-05T19:29:35.058 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:35.054+0000 7fae89306e40 1 --2- 172.21.15.167:0/3844986762 >> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] conn(0x56000aec89a0 0x56000aecad80 crc :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-05T19:29:35.058 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:35.054+0000 7fae89306e40 1 -- 172.21.15.167:0/3844986762 >> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] conn(0x56000ae95d10 msgr2=0x7fae80025400 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-05T19:29:35.058 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:35.054+0000 7fae89306e40 1 --2- 172.21.15.167:0/3844986762 >> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] conn(0x56000ae95d10 0x7fae80025400 crc :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-05T19:29:35.058 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:35.054+0000 7fae89306e40 1 -- 172.21.15.167:0/3844986762 >> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] conn(0x56000aeceb60 msgr2=0x56000aed0f40 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-05T19:29:35.058 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:35.054+0000 7fae89306e40 1 --2- 172.21.15.167:0/3844986762 >> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] conn(0x56000aeceb60 0x56000aed0f40 crc :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-05T19:29:35.058 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:35.054+0000 7fae89306e40 1 -- 172.21.15.167:0/3844986762 >> [v2:172.21.15.167:6824/29889,v1:172.21.15.167:6825/29889] conn(0x7fae68030ca0 msgr2=0x7fae68033150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-05T19:29:35.058 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:35.054+0000 7fae89306e40 1 --2- 172.21.15.167:0/3844986762 >> [v2:172.21.15.167:6824/29889,v1:172.21.15.167:6825/29889] conn(0x7fae68030ca0 0x7fae68033150 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7fae7000b2b0 tx=0x7fae70001490 comp rx=0 tx=0).stop 2023-10-05T19:29:35.059 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:35.054+0000 7fae8763c700 1 -- 172.21.15.167:0/3844986762 reap_dead start 2023-10-05T19:29:35.059 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:35.054+0000 7fae89306e40 1 -- 172.21.15.167:0/3844986762 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x56000ae96620 msgr2=0x56000aeb9ad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-05T19:29:35.059 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:35.054+0000 7fae89306e40 1 --2- 172.21.15.167:0/3844986762 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x56000ae96620 0x56000aeb9ad0 secure :-1 s=READY pgs=623 cs=0 l=1 rev1=1 crypto rx=0x7fae8001e040 tx=0x7fae8001c000 comp rx=0 tx=0).stop 2023-10-05T19:29:35.060 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:35.054+0000 7fae89306e40 1 -- 172.21.15.167:0/3844986762 shutdown_connections 2023-10-05T19:29:35.060 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:35.054+0000 7fae89306e40 1 --2- 172.21.15.167:0/3844986762 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x56000ae96620 0x56000aeb9ad0 unknown :-1 s=CLOSED pgs=623 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-05T19:29:35.060 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:35.054+0000 7fae89306e40 1 -- 172.21.15.167:0/3844986762 >> 172.21.15.167:0/3844986762 conn(0x56000ae91730 msgr2=0x56000aeaf470 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-05T19:29:35.060 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:35.054+0000 7fae89306e40 1 -- 172.21.15.167:0/3844986762 shutdown_connections 2023-10-05T19:29:35.060 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:35.054+0000 7fae89306e40 1 -- 172.21.15.167:0/3844986762 wait complete. 2023-10-05T19:29:35.060 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:35.054+0000 7fae89306e40 1 -- 172.21.15.167:0/947572296 >> [v2:172.21.15.167:6824/29889,v1:172.21.15.167:6825/29889] conn(0x7fadc4030bb0 msgr2=0x7fadc4033060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-05T19:29:35.061 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:35.054+0000 7fae89306e40 1 --2- 172.21.15.167:0/947572296 >> [v2:172.21.15.167:6824/29889,v1:172.21.15.167:6825/29889] conn(0x7fadc4030bb0 0x7fadc4033060 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x56000aed9310 tx=0x7fae7804a060 comp rx=0 tx=0).stop 2023-10-05T19:29:35.061 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:35.054+0000 7fae89306e40 1 -- 172.21.15.167:0/947572296 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x56000aee7580 msgr2=0x56000aeed110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-05T19:29:35.061 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:35.054+0000 7fae89306e40 1 --2- 172.21.15.167:0/947572296 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x56000aee7580 0x56000aeed110 secure :-1 s=READY pgs=625 cs=0 l=1 rev1=1 crypto rx=0x7fae70003010 tx=0x7fae70031070 comp rx=0 tx=0).stop 2023-10-05T19:29:35.061 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:35.054+0000 7fae89306e40 1 -- 172.21.15.167:0/947572296 shutdown_connections 2023-10-05T19:29:35.061 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:35.054+0000 7fae89306e40 1 --2- 172.21.15.167:0/947572296 >> [v2:172.21.15.167:6824/29889,v1:172.21.15.167:6825/29889] conn(0x7fadc4030bb0 0x7fadc4033060 unknown :-1 s=CLOSED pgs=215 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-05T19:29:35.061 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:35.054+0000 7fae89306e40 1 --2- 172.21.15.167:0/947572296 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x56000aee7580 0x56000aeed110 unknown :-1 s=CLOSED pgs=625 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-05T19:29:35.061 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:35.054+0000 7fae89306e40 1 -- 172.21.15.167:0/947572296 >> 172.21.15.167:0/947572296 conn(0x56000aee2940 msgr2=0x56000aee09e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-05T19:29:35.061 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:35.054+0000 7fae89306e40 1 -- 172.21.15.167:0/947572296 shutdown_connections 2023-10-05T19:29:35.061 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:35.054+0000 7fae89306e40 1 -- 172.21.15.167:0/947572296 wait complete. 2023-10-05T19:29:35.065 DEBUG:tasks.util.rgw: json result: {'user_id': 'bud', 'display_name': 'Bar', 'email': '', 'suspended': 0, 'max_buckets': 4, 'subusers': [], 'keys': [{'user': 'bud', 'access_key': 'NX5QOQKC6BH2IDN8HC7A', 'secret_key': 'LnEsqNNqZIpkzauboDcLXLcYaWwLQ3Kop0zAnKIn'}], 'swift_keys': [], 'caps': [], 'op_mask': 'read, write, delete', 'default_placement': '', 'default_storage_class': '', 'placement_tags': [], 'bucket_quota': {'enabled': False, 'check_on_raw': False, 'max_size': -1, 'max_size_kb': 0, 'max_objects': -1}, 'user_quota': {'enabled': False, 'check_on_raw': False, 'max_size': -1, 'max_size_kb': 0, 'max_objects': -1}, 'temp_url_keys': [], 'type': 'rgw', 'mfa_ids': []} 2023-10-05T19:29:40.077 INFO:tasks.radosgw_admin:RL: get_obj myfoo6 bud bytes_out=0 bytes_in=0 failed=False 2023-10-05T19:29:40.078 INFO:tasks.radosgw_admin:RL: get_obj myfoo6 bud bytes_out=0 bytes_in=8 failed=False 2023-10-05T19:29:40.078 INFO:tasks.util.rgw:rgwadmin: client.0 : ['user', 'create', '--uid', 'fud', '--display-name', 'Fud', '--access-key', 'p5YnriCv1nAtykxBrupQ', '--secret', 'Q8Tk6Q/27hfbFSYdSkPtUqhqx1GgzvpXa4WARozh', '--max-buckets', '4'] 2023-10-05T19:29:40.078 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'ceph', 'user', 'create', '--uid', 'fud', '--display-name', 'Fud', '--access-key', 'p5YnriCv1nAtykxBrupQ', '--secret', 'Q8Tk6Q/27hfbFSYdSkPtUqhqx1GgzvpXa4WARozh', '--max-buckets', '4'] 2023-10-05T19:29:40.078 DEBUG:teuthology.orchestra.run.smithi167:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster ceph user create --uid fud --display-name Fud --access-key p5YnriCv1nAtykxBrupQ --secret Q8Tk6Q/27hfbFSYdSkPtUqhqx1GgzvpXa4WARozh --max-buckets 4 2023-10-05T19:29:40.123 INFO:teuthology.orchestra.run.smithi167.stderr:ignoring --setuser ceph since I am not root 2023-10-05T19:29:40.124 INFO:teuthology.orchestra.run.smithi167.stderr:ignoring --setgroup ceph since I am not root 2023-10-05T19:29:40.124 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.118+0000 7f8c12b0ee40 1 Processor -- start 2023-10-05T19:29:40.125 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.122+0000 7f8c12b0ee40 1 -- start start 2023-10-05T19:29:40.125 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.122+0000 7f8c12b0ee40 1 --2- >> v2:172.21.15.118:3300/0 conn(0x55f80c58a450 0x55f80c589000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-05T19:29:40.125 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.122+0000 7f8c12b0ee40 1 -- --> v1:172.21.15.118:6789/0 -- auth(proto 0 26 bytes epoch 0) v1 -- 0x55f80c4d6bd0 con 0x55f80c58a820 2023-10-05T19:29:40.125 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.122+0000 7f8c12b0ee40 1 -- --> v2:172.21.15.118:3300/0 -- mon_getmap magic: 0 v1 -- 0x55f80c493ee0 con 0x55f80c58a450 2023-10-05T19:29:40.126 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.122+0000 7f8c10643700 1 --2- >> v2:172.21.15.118:3300/0 conn(0x55f80c58a450 0x55f80c589000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-05T19:29:40.126 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.122+0000 7f8c10643700 1 --2- >> v2:172.21.15.118:3300/0 conn(0x55f80c58a450 0x55f80c589000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.118:3300/0 says I am v2:172.21.15.167:56786/0 (socket says 172.21.15.167:56786) 2023-10-05T19:29:40.126 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.122+0000 7f8c0fe42700 1 --1- >> v1:172.21.15.118:6789/0 conn(0x55f80c58a820 0x55f80c589540 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.118:6789/0 says I am v1:172.21.15.167:33330/0 (socket says 172.21.15.167:33330) 2023-10-05T19:29:40.127 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.122+0000 7f8c10643700 1 -- 172.21.15.167:0/422912269 learned_addr learned my addr 172.21.15.167:0/422912269 (peer_addr_for_me v2:172.21.15.167:0/0) 2023-10-05T19:29:40.127 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.122+0000 7f8c10643700 1 -- 172.21.15.167:0/422912269 >> v1:172.21.15.118:6789/0 conn(0x55f80c58a820 legacy=0x55f80c589540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-05T19:29:40.127 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.122+0000 7f8c10643700 1 -- 172.21.15.167:0/422912269 --> v2:172.21.15.118:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55f80c589d10 con 0x55f80c58a450 2023-10-05T19:29:40.127 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.122+0000 7f8c10643700 1 --2- 172.21.15.167:0/422912269 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55f80c58a450 0x55f80c589000 secure :-1 s=READY pgs=627 cs=0 l=1 rev1=1 crypto rx=0x7f8c0800a5c0 tx=0x7f8c08005e00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7cfd2739ac4398d server_cookie=0 in_seq=0 out_seq=0 2023-10-05T19:29:40.129 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.122+0000 7f8c0f641700 1 -- 172.21.15.167:0/422912269 <== mon.0 v2:172.21.15.118:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f8c0800e510 con 0x55f80c58a450 2023-10-05T19:29:40.129 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.122+0000 7f8c0f641700 1 -- 172.21.15.167:0/422912269 <== mon.0 v2:172.21.15.118:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8c0800e670 con 0x55f80c58a450 2023-10-05T19:29:40.129 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.122+0000 7f8c0f641700 1 -- 172.21.15.167:0/422912269 <== mon.0 v2:172.21.15.118:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f8c0800e920 con 0x55f80c58a450 2023-10-05T19:29:40.129 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.122+0000 7f8c12b0ee40 1 -- 172.21.15.167:0/422912269 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55f80c58a450 msgr2=0x55f80c589000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-05T19:29:40.130 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.122+0000 7f8c12b0ee40 1 --2- 172.21.15.167:0/422912269 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55f80c58a450 0x55f80c589000 secure :-1 s=READY pgs=627 cs=0 l=1 rev1=1 crypto rx=0x7f8c0800a5c0 tx=0x7f8c08005e00 comp rx=0 tx=0).stop 2023-10-05T19:29:40.130 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.122+0000 7f8c12b0ee40 1 -- 172.21.15.167:0/422912269 shutdown_connections 2023-10-05T19:29:40.130 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.122+0000 7f8c12b0ee40 1 --2- 172.21.15.167:0/422912269 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55f80c58a450 0x55f80c589000 unknown :-1 s=CLOSED pgs=627 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-05T19:29:40.130 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.122+0000 7f8c12b0ee40 1 -- 172.21.15.167:0/422912269 >> 172.21.15.167:0/422912269 conn(0x55f80c584760 msgr2=0x55f80c586b70 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-05T19:29:40.130 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.122+0000 7f8c12b0ee40 1 -- 172.21.15.167:0/422912269 shutdown_connections 2023-10-05T19:29:40.130 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.122+0000 7f8c12b0ee40 1 -- 172.21.15.167:0/422912269 wait complete. 2023-10-05T19:29:40.130 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.126+0000 7f8c12b0ee40 1 Processor -- start 2023-10-05T19:29:40.130 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.126+0000 7f8c12b0ee40 1 -- start start 2023-10-05T19:29:40.131 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.126+0000 7f8c12b0ee40 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55f80c58a450 0x55f80c5ab260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-05T19:29:40.131 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.126+0000 7f8c12b0ee40 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f80c457e30 con 0x55f80c58a450 2023-10-05T19:29:40.131 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.126+0000 7f8c10643700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55f80c58a450 0x55f80c5ab260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-05T19:29:40.131 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.126+0000 7f8c10643700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55f80c58a450 0x55f80c5ab260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.118:3300/0 says I am v2:172.21.15.167:56790/0 (socket says 172.21.15.167:56790) 2023-10-05T19:29:40.131 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.126+0000 7f8c10643700 1 -- 172.21.15.167:0/3794103068 learned_addr learned my addr 172.21.15.167:0/3794103068 (peer_addr_for_me v2:172.21.15.167:0/0) 2023-10-05T19:29:40.134 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.126+0000 7f8c10643700 1 -- 172.21.15.167:0/3794103068 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8c0800a180 con 0x55f80c58a450 2023-10-05T19:29:40.134 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.126+0000 7f8c10643700 1 --2- 172.21.15.167:0/3794103068 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55f80c58a450 0x55f80c5ab260 secure :-1 s=READY pgs=628 cs=0 l=1 rev1=1 crypto rx=0x55f80c589ce0 tx=0x7f8c08013000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b40113bc156bae2c server_cookie=0 in_seq=0 out_seq=0 2023-10-05T19:29:40.134 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.126+0000 7f8c0e63f700 1 -- 172.21.15.167:0/3794103068 <== mon.0 v2:172.21.15.118:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f8c080032f0 con 0x55f80c58a450 2023-10-05T19:29:40.134 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.126+0000 7f8c0e63f700 1 -- 172.21.15.167:0/3794103068 <== mon.0 v2:172.21.15.118:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8c0800e7e0 con 0x55f80c58a450 2023-10-05T19:29:40.134 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.126+0000 7f8c0e63f700 1 -- 172.21.15.167:0/3794103068 <== mon.0 v2:172.21.15.118:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f8c0800ead0 con 0x55f80c58a450 2023-10-05T19:29:40.134 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.126+0000 7f8c12b0ee40 1 -- 172.21.15.167:0/3794103068 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55f80c58a450 msgr2=0x55f80c5ab260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-05T19:29:40.135 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.126+0000 7f8c12b0ee40 1 --2- 172.21.15.167:0/3794103068 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55f80c58a450 0x55f80c5ab260 secure :-1 s=READY pgs=628 cs=0 l=1 rev1=1 crypto rx=0x55f80c589ce0 tx=0x7f8c08013000 comp rx=0 tx=0).stop 2023-10-05T19:29:40.135 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.126+0000 7f8c12b0ee40 1 -- 172.21.15.167:0/3794103068 shutdown_connections 2023-10-05T19:29:40.135 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.126+0000 7f8c12b0ee40 1 --2- 172.21.15.167:0/3794103068 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55f80c58a450 0x55f80c5ab260 unknown :-1 s=CLOSED pgs=628 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-05T19:29:40.135 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.126+0000 7f8c12b0ee40 1 -- 172.21.15.167:0/3794103068 >> 172.21.15.167:0/3794103068 conn(0x55f80c584760 msgr2=0x55f80c5a6e20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-05T19:29:40.135 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.126+0000 7f8c12b0ee40 1 -- 172.21.15.167:0/3794103068 shutdown_connections 2023-10-05T19:29:40.135 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.126+0000 7f8c12b0ee40 1 -- 172.21.15.167:0/3794103068 wait complete. 2023-10-05T19:29:40.135 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.126+0000 7f8c12b0ee40 1 Processor -- start 2023-10-05T19:29:40.135 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.126+0000 7f8c12b0ee40 1 -- start start 2023-10-05T19:29:40.135 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.126+0000 7f8c12b0ee40 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55f80c58a820 0x55f80c5b2000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-05T19:29:40.136 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.126+0000 7f8c12b0ee40 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f80c42aed0 con 0x55f80c58a820 2023-10-05T19:29:40.136 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.126+0000 7f8c10643700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55f80c58a820 0x55f80c5b2000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-05T19:29:40.136 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.126+0000 7f8c10643700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55f80c58a820 0x55f80c5b2000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.118:3300/0 says I am v2:172.21.15.167:56798/0 (socket says 172.21.15.167:56798) 2023-10-05T19:29:40.136 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.126+0000 7f8c10643700 1 -- 172.21.15.167:0/4016842607 learned_addr learned my addr 172.21.15.167:0/4016842607 (peer_addr_for_me v2:172.21.15.167:0/0) 2023-10-05T19:29:40.136 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.130+0000 7f8c10643700 1 -- 172.21.15.167:0/4016842607 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55f80c589d10 con 0x55f80c58a820 2023-10-05T19:29:40.136 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.130+0000 7f8c10643700 1 --2- 172.21.15.167:0/4016842607 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55f80c58a820 0x55f80c5b2000 secure :-1 s=READY pgs=629 cs=0 l=1 rev1=1 crypto rx=0x7f8c0801e040 tx=0x7f8c0801c000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-05T19:29:40.136 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.130+0000 7f8c0ce3c700 1 -- 172.21.15.167:0/4016842607 <== mon.0 v2:172.21.15.118:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f8c0801cb50 con 0x55f80c58a820 2023-10-05T19:29:40.136 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.130+0000 7f8c0ce3c700 1 -- 172.21.15.167:0/4016842607 <== mon.0 v2:172.21.15.118:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8c0801ccb0 con 0x55f80c58a820 2023-10-05T19:29:40.136 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.130+0000 7f8c12b0ee40 1 -- 172.21.15.167:0/4016842607 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x55f80c5b2540 con 0x55f80c58a820 2023-10-05T19:29:40.137 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.130+0000 7f8c0ce3c700 1 -- 172.21.15.167:0/4016842607 <== mon.0 v2:172.21.15.118:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f8c08010440 con 0x55f80c58a820 2023-10-05T19:29:40.137 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.130+0000 7f8c12b0ee40 1 -- 172.21.15.167:0/4016842607 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55f80c5b2980 con 0x55f80c58a820 2023-10-05T19:29:40.141 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.130+0000 7f8c12b0ee40 20 rados->read ofs=0 len=0 2023-10-05T19:29:40.141 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.130+0000 7f8c0ce3c700 1 -- 172.21.15.167:0/4016842607 <== mon.0 v2:172.21.15.118:3300/0 4 ==== mgrmap(e 5) v1 ==== 40329+0+0 (secure 0 0 0) 0x7f8c080105a0 con 0x55f80c58a820 2023-10-05T19:29:40.141 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.130+0000 7f8c0ce3c700 1 --2- 172.21.15.167:0/4016842607 >> [v2:172.21.15.167:6824/29889,v1:172.21.15.167:6825/29889] conn(0x7f8bfc030c00 0x7f8bfc0330b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-05T19:29:40.141 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.130+0000 7f8c0ce3c700 1 -- 172.21.15.167:0/4016842607 <== mon.0 v2:172.21.15.118:3300/0 5 ==== osd_map(32..32 src has 1..32) v4 ==== 6260+0+0 (secure 0 0 0) 0x7f8c08046520 con 0x55f80c58a820 2023-10-05T19:29:40.141 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.130+0000 7f8c0fe42700 1 --2- 172.21.15.167:0/4016842607 >> [v2:172.21.15.167:6824/29889,v1:172.21.15.167:6825/29889] conn(0x7f8bfc030c00 0x7f8bfc0330b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-05T19:29:40.141 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.130+0000 7f8c12b0ee40 1 --2- 172.21.15.167:0/4016842607 >> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] conn(0x55f80c5bba60 0x55f80c5bde40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-05T19:29:40.141 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.130+0000 7f8c12b0ee40 1 -- 172.21.15.167:0/4016842607 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:1 5.12 5:49953fa1:::default.realm:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32) v8 -- 0x55f80c5be540 con 0x55f80c5bba60 2023-10-05T19:29:40.141 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.130+0000 7f8c10e44700 1 --2- 172.21.15.167:0/4016842607 >> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] conn(0x55f80c5bba60 0x55f80c5bde40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-05T19:29:40.142 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.134+0000 7f8c0fe42700 1 --2- 172.21.15.167:0/4016842607 >> [v2:172.21.15.167:6824/29889,v1:172.21.15.167:6825/29889] conn(0x7f8bfc030c00 0x7f8bfc0330b0 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7f8c0400b300 tx=0x7f8c04001470 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-05T19:29:40.142 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.134+0000 7f8c10e44700 1 --2- 172.21.15.167:0/4016842607 >> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] conn(0x55f80c5bba60 0x55f80c5bde40 crc :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-05T19:29:40.142 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.134+0000 7f8c10e44700 1 -- 172.21.15.167:0/4016842607 <== osd.1 v2:172.21.15.167:6800/30035 1 ==== osd_op_reply(1 default.realm [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 ==== 157+0+0 (crc 0 0 0) 0x7f8c0000a500 con 0x55f80c5bba60 2023-10-05T19:29:40.142 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.134+0000 7f8c12b0ee40 20 rados_obj.operate() r=-2 bl.length=0 2023-10-05T19:29:40.142 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.134+0000 7f8c12b0ee40 20 realm 2023-10-05T19:29:40.142 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.134+0000 7f8c12b0ee40 20 rados->read ofs=0 len=0 2023-10-05T19:29:40.142 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.134+0000 7f8c12b0ee40 1 -- 172.21.15.167:0/4016842607 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:2 5.12 5:49953fa1:::default.realm:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32) v8 -- 0x55f80c5bb0d0 con 0x55f80c5bba60 2023-10-05T19:29:40.142 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.134+0000 7f8c10e44700 1 -- 172.21.15.167:0/4016842607 <== osd.1 v2:172.21.15.167:6800/30035 2 ==== osd_op_reply(2 default.realm [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 ==== 157+0+0 (crc 0 0 0) 0x7f8c0000a500 con 0x55f80c5bba60 2023-10-05T19:29:40.142 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.134+0000 7f8c12b0ee40 20 rados_obj.operate() r=-2 bl.length=0 2023-10-05T19:29:40.143 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.134+0000 7f8c12b0ee40 4 RGWPeriod::init failed to init realm id : (2) No such file or directory 2023-10-05T19:29:40.143 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.134+0000 7f8c12b0ee40 20 rados->read ofs=0 len=0 2023-10-05T19:29:40.143 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.134+0000 7f8c12b0ee40 1 -- 172.21.15.167:0/4016842607 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:3 5.12 5:49953fa1:::default.realm:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32) v8 -- 0x55f80c5bf020 con 0x55f80c5bba60 2023-10-05T19:29:40.143 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.134+0000 7f8c10e44700 1 -- 172.21.15.167:0/4016842607 <== osd.1 v2:172.21.15.167:6800/30035 3 ==== osd_op_reply(3 default.realm [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 ==== 157+0+0 (crc 0 0 0) 0x7f8c0000a500 con 0x55f80c5bba60 2023-10-05T19:29:40.143 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.134+0000 7f8c12b0ee40 20 rados_obj.operate() r=-2 bl.length=0 2023-10-05T19:29:40.143 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.134+0000 7f8c12b0ee40 20 rados->read ofs=0 len=0 2023-10-05T19:29:40.143 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.134+0000 7f8c12b0ee40 1 -- 172.21.15.167:0/4016842607 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:4 5.f 5:f43b6ece:::zone_names.default:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32) v8 -- 0x55f80c5bec30 con 0x55f80c5bba60 2023-10-05T19:29:40.143 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.134+0000 7f8c10e44700 1 -- 172.21.15.167:0/4016842607 <== osd.1 v2:172.21.15.167:6800/30035 4 ==== osd_op_reply(4 zone_names.default [read 0~46 out=46b] v0'0 uv1 ondisk = 0) v8 ==== 162+0+46 (crc 0 0 0) 0x7f8c0000a500 con 0x55f80c5bba60 2023-10-05T19:29:40.143 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.134+0000 7f8c12b0ee40 20 rados_obj.operate() r=0 bl.length=46 2023-10-05T19:29:40.143 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.134+0000 7f8c12b0ee40 20 rados->read ofs=0 len=0 2023-10-05T19:29:40.144 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.134+0000 7f8c12b0ee40 1 -- 172.21.15.167:0/4016842607 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:5 5.1d 5:b8374eb4:::zone_info.9a9df965-6612-45d4-a45f-6e3c889bacc2:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32) v8 -- 0x55f80c5bfeb0 con 0x55f80c5bba60 2023-10-05T19:29:40.144 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.134+0000 7f8c10e44700 1 -- 172.21.15.167:0/4016842607 <== osd.1 v2:172.21.15.167:6800/30035 5 ==== osd_op_reply(5 zone_info.9a9df965-6612-45d4-a45f-6e3c889bacc2 [read 0~874 out=874b] v0'0 uv2 ondisk = 0) v8 ==== 190+0+874 (crc 0 0 0) 0x7f8c0000a500 con 0x55f80c5bba60 2023-10-05T19:29:40.144 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.134+0000 7f8c12b0ee40 20 rados_obj.operate() r=0 bl.length=874 2023-10-05T19:29:40.144 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.134+0000 7f8c12b0ee40 20 searching for the correct realm 2023-10-05T19:29:40.144 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.134+0000 7f8c12b0ee40 1 --2- 172.21.15.167:0/4016842607 >> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] conn(0x55f80c5c1c70 0x55f80c5c4050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-05T19:29:40.144 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.134+0000 7f8c12b0ee40 1 -- 172.21.15.167:0/4016842607 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:6 5.0 5:00000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x55f80c5c4590 con 0x55f80c5c1c70 2023-10-05T19:29:40.144 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.134+0000 7f8c10643700 1 --2- 172.21.15.167:0/4016842607 >> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] conn(0x55f80c5c1c70 0x55f80c5c4050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-05T19:29:40.144 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.134+0000 7f8c10643700 1 --2- 172.21.15.167:0/4016842607 >> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] conn(0x55f80c5c1c70 0x55f80c5c4050 crc :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-05T19:29:40.146 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.138+0000 7f8c10643700 1 -- 172.21.15.167:0/4016842607 <== osd.3 v2:172.21.15.167:6808/30036 1 ==== osd_op_reply(6 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f8c0802b070 con 0x55f80c5c1c70 2023-10-05T19:29:40.146 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.138+0000 7f8c10643700 1 --2- 172.21.15.167:0/4016842607 >> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] conn(0x7f8c08044c30 0x7f8c08010850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-05T19:29:40.146 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.138+0000 7f8c10643700 1 -- 172.21.15.167:0/4016842607 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:7 5.10 5:08000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x55f80c5c4590 con 0x7f8c08044c30 2023-10-05T19:29:40.147 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.138+0000 7f8c0fe42700 1 --2- 172.21.15.167:0/4016842607 >> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] conn(0x7f8c08044c30 0x7f8c08010850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-05T19:29:40.147 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.138+0000 7f8c0fe42700 1 --2- 172.21.15.167:0/4016842607 >> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] conn(0x7f8c08044c30 0x7f8c08010850 crc :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-05T19:29:40.147 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.138+0000 7f8c0fe42700 1 -- 172.21.15.167:0/4016842607 <== osd.2 v2:172.21.15.167:6816/30037 1 ==== osd_op_reply(7 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f8c0400f560 con 0x7f8c08044c30 2023-10-05T19:29:40.147 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.138+0000 7f8c0fe42700 1 --2- 172.21.15.167:0/4016842607 >> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] conn(0x7f8c0400e8c0 0x7f8c0400d310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-05T19:29:40.147 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.138+0000 7f8c0fe42700 1 -- 172.21.15.167:0/4016842607 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:8 5.8 5:10000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x55f80c5c4590 con 0x7f8c0400e8c0 2023-10-05T19:29:40.147 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.138+0000 7f8c10e44700 1 --2- 172.21.15.167:0/4016842607 >> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] conn(0x7f8c0400e8c0 0x7f8c0400d310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-05T19:29:40.147 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.138+0000 7f8c10e44700 1 --2- 172.21.15.167:0/4016842607 >> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] conn(0x7f8c0400e8c0 0x7f8c0400d310 crc :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-05T19:29:40.147 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.142+0000 7f8c10e44700 1 -- 172.21.15.167:0/4016842607 <== osd.0 v2:172.21.15.118:6800/27690 1 ==== osd_op_reply(8 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f8c00014040 con 0x7f8c0400e8c0 2023-10-05T19:29:40.147 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.142+0000 7f8c10e44700 1 -- 172.21.15.167:0/4016842607 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:9 5.18 5:18000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x55f80c5c4590 con 0x55f80c5bba60 2023-10-05T19:29:40.148 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.142+0000 7f8c10e44700 1 -- 172.21.15.167:0/4016842607 <== osd.1 v2:172.21.15.167:6800/30035 6 ==== osd_op_reply(9 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f8c00014040 con 0x55f80c5bba60 2023-10-05T19:29:40.148 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.142+0000 7f8c10e44700 1 -- 172.21.15.167:0/4016842607 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:10 5.4 5:20000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x55f80c5c4590 con 0x55f80c5bba60 2023-10-05T19:29:40.148 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.142+0000 7f8c10e44700 1 -- 172.21.15.167:0/4016842607 <== osd.1 v2:172.21.15.167:6800/30035 7 ==== osd_op_reply(10 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f8c00014040 con 0x55f80c5bba60 2023-10-05T19:29:40.148 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.142+0000 7f8c10e44700 1 -- 172.21.15.167:0/4016842607 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:11 5.14 5:28000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x55f80c5c4590 con 0x7f8c0400e8c0 2023-10-05T19:29:40.149 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.142+0000 7f8c10e44700 1 -- 172.21.15.167:0/4016842607 <== osd.0 v2:172.21.15.118:6800/27690 2 ==== osd_op_reply(11 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f8c00014040 con 0x7f8c0400e8c0 2023-10-05T19:29:40.149 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.142+0000 7f8c10e44700 1 -- 172.21.15.167:0/4016842607 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:12 5.c 5:30000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x55f80c5c4590 con 0x7f8c0400e8c0 2023-10-05T19:29:40.149 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.142+0000 7f8c10e44700 1 -- 172.21.15.167:0/4016842607 <== osd.0 v2:172.21.15.118:6800/27690 3 ==== osd_op_reply(12 [pgnls start_epoch 32 out=112b] v23'1 uv1 ondisk = 1) v8 ==== 144+0+112 (crc 0 0 0) 0x7f8c00014040 con 0x7f8c0400e8c0 2023-10-05T19:29:40.150 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.142+0000 7f8c10e44700 1 -- 172.21.15.167:0/4016842607 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:13 5.1c 5:38000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x55f80c5c4590 con 0x7f8c08044c30 2023-10-05T19:29:40.150 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.142+0000 7f8c0fe42700 1 -- 172.21.15.167:0/4016842607 <== osd.2 v2:172.21.15.167:6816/30037 2 ==== osd_op_reply(13 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f8c0400f560 con 0x7f8c08044c30 2023-10-05T19:29:40.150 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.142+0000 7f8c0fe42700 1 -- 172.21.15.167:0/4016842607 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:14 5.2 5:40000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x55f80c5c4590 con 0x55f80c5c1c70 2023-10-05T19:29:40.150 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.146+0000 7f8c10643700 1 -- 172.21.15.167:0/4016842607 <== osd.3 v2:172.21.15.167:6808/30036 2 ==== osd_op_reply(14 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f8c0802b070 con 0x55f80c5c1c70 2023-10-05T19:29:40.150 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.146+0000 7f8c10643700 1 -- 172.21.15.167:0/4016842607 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:15 5.12 5:48000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x55f80c5c4590 con 0x55f80c5bba60 2023-10-05T19:29:40.151 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.146+0000 7f8c10e44700 1 -- 172.21.15.167:0/4016842607 <== osd.1 v2:172.21.15.167:6800/30035 8 ==== osd_op_reply(15 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f8c00014040 con 0x55f80c5bba60 2023-10-05T19:29:40.151 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.146+0000 7f8c10e44700 1 -- 172.21.15.167:0/4016842607 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:16 5.a 5:50000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x55f80c5c4590 con 0x7f8c0400e8c0 2023-10-05T19:29:40.151 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.146+0000 7f8c10e44700 1 -- 172.21.15.167:0/4016842607 <== osd.0 v2:172.21.15.118:6800/27690 4 ==== osd_op_reply(16 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f8c00022040 con 0x7f8c0400e8c0 2023-10-05T19:29:40.151 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.146+0000 7f8c10e44700 1 -- 172.21.15.167:0/4016842607 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:17 5.1a 5:58000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x55f80c5c4590 con 0x55f80c5bba60 2023-10-05T19:29:40.152 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.146+0000 7f8c10e44700 1 -- 172.21.15.167:0/4016842607 <== osd.1 v2:172.21.15.167:6800/30035 9 ==== osd_op_reply(17 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f8c00022040 con 0x55f80c5bba60 2023-10-05T19:29:40.152 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.146+0000 7f8c10e44700 1 -- 172.21.15.167:0/4016842607 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:18 5.6 5:60000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x55f80c5c4590 con 0x7f8c08044c30 2023-10-05T19:29:40.152 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.146+0000 7f8c0fe42700 1 -- 172.21.15.167:0/4016842607 <== osd.2 v2:172.21.15.167:6816/30037 3 ==== osd_op_reply(18 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f8c0400f560 con 0x7f8c08044c30 2023-10-05T19:29:40.152 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.146+0000 7f8c0fe42700 1 -- 172.21.15.167:0/4016842607 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:19 5.16 5:68000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x7f8c0401f040 con 0x55f80c5c1c70 2023-10-05T19:29:40.153 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.150+0000 7f8c10643700 1 -- 172.21.15.167:0/4016842607 <== osd.3 v2:172.21.15.167:6808/30036 3 ==== osd_op_reply(19 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f8c0802b070 con 0x55f80c5c1c70 2023-10-05T19:29:40.153 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.150+0000 7f8c10643700 1 -- 172.21.15.167:0/4016842607 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:20 5.e 5:70000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x7f8c0401f040 con 0x55f80c5c1c70 2023-10-05T19:29:40.153 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.150+0000 7f8c10643700 1 -- 172.21.15.167:0/4016842607 <== osd.3 v2:172.21.15.167:6808/30036 4 ==== osd_op_reply(20 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f8c0802b070 con 0x55f80c5c1c70 2023-10-05T19:29:40.153 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.150+0000 7f8c10643700 1 -- 172.21.15.167:0/4016842607 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:21 5.1e 5:78000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x7f8c0401f040 con 0x7f8c0400e8c0 2023-10-05T19:29:40.154 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.150+0000 7f8c10e44700 1 -- 172.21.15.167:0/4016842607 <== osd.0 v2:172.21.15.118:6800/27690 5 ==== osd_op_reply(21 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f8c00022040 con 0x7f8c0400e8c0 2023-10-05T19:29:40.154 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.150+0000 7f8c10e44700 1 -- 172.21.15.167:0/4016842607 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:22 5.1 5:80000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x7f8c0401f040 con 0x55f80c5c1c70 2023-10-05T19:29:40.155 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.150+0000 7f8c10643700 1 -- 172.21.15.167:0/4016842607 <== osd.3 v2:172.21.15.167:6808/30036 5 ==== osd_op_reply(22 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f8c0802b070 con 0x55f80c5c1c70 2023-10-05T19:29:40.155 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.150+0000 7f8c10643700 1 -- 172.21.15.167:0/4016842607 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:23 5.11 5:88000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x7f8c0401f040 con 0x55f80c5c1c70 2023-10-05T19:29:40.155 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.150+0000 7f8c10643700 1 -- 172.21.15.167:0/4016842607 <== osd.3 v2:172.21.15.167:6808/30036 6 ==== osd_op_reply(23 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f8c0802b070 con 0x55f80c5c1c70 2023-10-05T19:29:40.156 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.150+0000 7f8c10643700 1 -- 172.21.15.167:0/4016842607 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:24 5.9 5:90000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x7f8c0401f040 con 0x55f80c5bba60 2023-10-05T19:29:40.156 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.150+0000 7f8c10e44700 1 -- 172.21.15.167:0/4016842607 <== osd.1 v2:172.21.15.167:6800/30035 10 ==== osd_op_reply(24 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f8c00022040 con 0x55f80c5bba60 2023-10-05T19:29:40.156 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.150+0000 7f8c10e44700 1 -- 172.21.15.167:0/4016842607 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:25 5.19 5:98000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x7f8c0401f040 con 0x55f80c5bba60 2023-10-05T19:29:40.157 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.154+0000 7f8c10e44700 1 -- 172.21.15.167:0/4016842607 <== osd.1 v2:172.21.15.167:6800/30035 11 ==== osd_op_reply(25 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f8c00022040 con 0x55f80c5bba60 2023-10-05T19:29:40.157 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.154+0000 7f8c10e44700 1 -- 172.21.15.167:0/4016842607 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:26 5.5 5:a0000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x7f8c0401f040 con 0x7f8c0400e8c0 2023-10-05T19:29:40.158 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.154+0000 7f8c10e44700 1 -- 172.21.15.167:0/4016842607 <== osd.0 v2:172.21.15.118:6800/27690 6 ==== osd_op_reply(26 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f8c00022040 con 0x7f8c0400e8c0 2023-10-05T19:29:40.158 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.154+0000 7f8c10e44700 1 -- 172.21.15.167:0/4016842607 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:27 5.15 5:a8000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x7f8c0401f040 con 0x55f80c5c1c70 2023-10-05T19:29:40.158 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.154+0000 7f8c10643700 1 -- 172.21.15.167:0/4016842607 <== osd.3 v2:172.21.15.167:6808/30036 7 ==== osd_op_reply(27 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f8c0802b070 con 0x55f80c5c1c70 2023-10-05T19:29:40.159 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.154+0000 7f8c10643700 1 -- 172.21.15.167:0/4016842607 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:28 5.d 5:b0000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x7f8c0401f040 con 0x7f8c08044c30 2023-10-05T19:29:40.160 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.154+0000 7f8c0fe42700 1 -- 172.21.15.167:0/4016842607 <== osd.2 v2:172.21.15.167:6816/30037 4 ==== osd_op_reply(28 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f8c0400f560 con 0x7f8c08044c30 2023-10-05T19:29:40.160 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.154+0000 7f8c0fe42700 1 -- 172.21.15.167:0/4016842607 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:29 5.1d 5:b8000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x7f8c0401f040 con 0x55f80c5bba60 2023-10-05T19:29:40.160 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.154+0000 7f8c10e44700 1 -- 172.21.15.167:0/4016842607 <== osd.1 v2:172.21.15.167:6800/30035 12 ==== osd_op_reply(29 [pgnls start_epoch 32 out=107b] v30'2 uv2 ondisk = 1) v8 ==== 144+0+107 (crc 0 0 0) 0x7f8c00022040 con 0x55f80c5bba60 2023-10-05T19:29:40.160 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.158+0000 7f8c10e44700 1 -- 172.21.15.167:0/4016842607 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:30 5.3 5:c0000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x7f8c0401f040 con 0x7f8c0400e8c0 2023-10-05T19:29:40.161 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.158+0000 7f8c10e44700 1 -- 172.21.15.167:0/4016842607 <== osd.0 v2:172.21.15.118:6800/27690 7 ==== osd_op_reply(30 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f8c00022040 con 0x7f8c0400e8c0 2023-10-05T19:29:40.161 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.158+0000 7f8c10e44700 1 -- 172.21.15.167:0/4016842607 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:31 5.13 5:c8000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x7f8c0401f040 con 0x55f80c5c1c70 2023-10-05T19:29:40.162 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.158+0000 7f8c10643700 1 -- 172.21.15.167:0/4016842607 <== osd.3 v2:172.21.15.167:6808/30036 8 ==== osd_op_reply(31 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f8c0802b070 con 0x55f80c5c1c70 2023-10-05T19:29:40.162 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.158+0000 7f8c10643700 1 -- 172.21.15.167:0/4016842607 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:32 5.b 5:d0000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x7f8c0401f040 con 0x7f8c0400e8c0 2023-10-05T19:29:40.163 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.158+0000 7f8c10e44700 1 -- 172.21.15.167:0/4016842607 <== osd.0 v2:172.21.15.118:6800/27690 8 ==== osd_op_reply(32 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f8c00022040 con 0x7f8c0400e8c0 2023-10-05T19:29:40.163 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.158+0000 7f8c10e44700 1 -- 172.21.15.167:0/4016842607 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:33 5.1b 5:d8000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x7f8c0401f040 con 0x7f8c08044c30 2023-10-05T19:29:40.164 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.158+0000 7f8c0fe42700 1 -- 172.21.15.167:0/4016842607 <== osd.2 v2:172.21.15.167:6816/30037 5 ==== osd_op_reply(33 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f8c0400f560 con 0x7f8c08044c30 2023-10-05T19:29:40.164 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.158+0000 7f8c0fe42700 1 -- 172.21.15.167:0/4016842607 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:34 5.7 5:e0000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x7f8c0401f040 con 0x55f80c5c1c70 2023-10-05T19:29:40.165 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.162+0000 7f8c10643700 1 -- 172.21.15.167:0/4016842607 <== osd.3 v2:172.21.15.167:6808/30036 9 ==== osd_op_reply(34 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f8c0802b070 con 0x55f80c5c1c70 2023-10-05T19:29:40.165 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.162+0000 7f8c10643700 1 -- 172.21.15.167:0/4016842607 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:35 5.17 5:e8000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x7f8c0401f040 con 0x7f8c0400e8c0 2023-10-05T19:29:40.166 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.162+0000 7f8c10e44700 1 -- 172.21.15.167:0/4016842607 <== osd.0 v2:172.21.15.118:6800/27690 9 ==== osd_op_reply(35 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f8c00022040 con 0x7f8c0400e8c0 2023-10-05T19:29:40.166 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.162+0000 7f8c10e44700 1 -- 172.21.15.167:0/4016842607 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:36 5.f 5:f0000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x7f8c0401f040 con 0x55f80c5bba60 2023-10-05T19:29:40.167 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.162+0000 7f8c10e44700 1 -- 172.21.15.167:0/4016842607 <== osd.1 v2:172.21.15.167:6800/30035 13 ==== osd_op_reply(36 [pgnls start_epoch 32 out=115b] v23'2 uv2 ondisk = 1) v8 ==== 144+0+115 (crc 0 0 0) 0x7f8c0002f080 con 0x55f80c5bba60 2023-10-05T19:29:40.167 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.162+0000 7f8c10e44700 1 -- 172.21.15.167:0/4016842607 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:37 5.1f 5:f8000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x7f8c0401f040 con 0x7f8c08044c30 2023-10-05T19:29:40.168 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.162+0000 7f8c0fe42700 1 -- 172.21.15.167:0/4016842607 <== osd.2 v2:172.21.15.167:6816/30037 6 ==== osd_op_reply(37 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f8c0400f560 con 0x7f8c08044c30 2023-10-05T19:29:40.168 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.162+0000 7f8c12b0ee40 20 RGWRados::pool_iterate: got zonegroup_info.4f1f719d-9648-417b-b10c-e5d939825419 2023-10-05T19:29:40.168 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.162+0000 7f8c12b0ee40 20 RGWRados::pool_iterate: got zone_info.9a9df965-6612-45d4-a45f-6e3c889bacc2 2023-10-05T19:29:40.168 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.162+0000 7f8c12b0ee40 20 RGWRados::pool_iterate: got zone_names.default 2023-10-05T19:29:40.168 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.162+0000 7f8c12b0ee40 20 RGWRados::pool_iterate: got zonegroups_names.default 2023-10-05T19:29:40.168 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.162+0000 7f8c12b0ee40 20 rados->read ofs=0 len=0 2023-10-05T19:29:40.168 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.162+0000 7f8c12b0ee40 1 -- 172.21.15.167:0/4016842607 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:38 5.12 5:49953fa1:::default.realm:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32) v8 -- 0x55f80c5c0a20 con 0x55f80c5bba60 2023-10-05T19:29:40.169 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.162+0000 7f8c10e44700 1 -- 172.21.15.167:0/4016842607 <== osd.1 v2:172.21.15.167:6800/30035 14 ==== osd_op_reply(38 default.realm [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 ==== 157+0+0 (crc 0 0 0) 0x7f8c0002f080 con 0x55f80c5bba60 2023-10-05T19:29:40.170 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.162+0000 7f8c12b0ee40 20 rados_obj.operate() r=-2 bl.length=0 2023-10-05T19:29:40.170 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.162+0000 7f8c12b0ee40 20 rados->read ofs=0 len=0 2023-10-05T19:29:40.170 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.162+0000 7f8c12b0ee40 1 -- 172.21.15.167:0/4016842607 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:39 5.f 5:f4c53578:::zonegroups_names.default:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32) v8 -- 0x55f80c5c1310 con 0x55f80c5bba60 2023-10-05T19:29:40.170 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.166+0000 7f8c10e44700 1 -- 172.21.15.167:0/4016842607 <== osd.1 v2:172.21.15.167:6800/30035 15 ==== osd_op_reply(39 zonegroups_names.default [read 0~46 out=46b] v0'0 uv2 ondisk = 0) v8 ==== 168+0+46 (crc 0 0 0) 0x7f8c0002f080 con 0x55f80c5bba60 2023-10-05T19:29:40.170 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.166+0000 7f8c12b0ee40 20 rados_obj.operate() r=0 bl.length=46 2023-10-05T19:29:40.170 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.166+0000 7f8c12b0ee40 20 rados->read ofs=0 len=0 2023-10-05T19:29:40.170 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.166+0000 7f8c12b0ee40 1 -- 172.21.15.167:0/4016842607 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:40 5.c 5:30d299a5:::zonegroup_info.4f1f719d-9648-417b-b10c-e5d939825419:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32) v8 -- 0x55f80c5c49d0 con 0x7f8c0400e8c0 2023-10-05T19:29:40.172 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.166+0000 7f8c10e44700 1 -- 172.21.15.167:0/4016842607 <== osd.0 v2:172.21.15.118:6800/27690 10 ==== osd_op_reply(40 zonegroup_info.4f1f719d-9648-417b-b10c-e5d939825419 [read 0~362 out=362b] v0'0 uv1 ondisk = 0) v8 ==== 195+0+362 (crc 0 0 0) 0x7f8c0002f080 con 0x7f8c0400e8c0 2023-10-05T19:29:40.172 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.166+0000 7f8c12b0ee40 20 rados_obj.operate() r=0 bl.length=362 2023-10-05T19:29:40.172 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.166+0000 7f8c12b0ee40 20 rados->read ofs=0 len=0 2023-10-05T19:29:40.172 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.166+0000 7f8c12b0ee40 1 -- 172.21.15.167:0/4016842607 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:41 5.13 5:cbbf6c60:::converted:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32) v8 -- 0x55f80c5c49d0 con 0x55f80c5c1c70 2023-10-05T19:29:40.172 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.166+0000 7f8c10643700 1 -- 172.21.15.167:0/4016842607 <== osd.3 v2:172.21.15.167:6808/30036 10 ==== osd_op_reply(41 converted [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 ==== 153+0+0 (crc 0 0 0) 0x7f8c0802b070 con 0x55f80c5c1c70 2023-10-05T19:29:40.172 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.166+0000 7f8c12b0ee40 20 rados_obj.operate() r=-2 bl.length=0 2023-10-05T19:29:40.172 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.166+0000 7f8c12b0ee40 20 rados->read ofs=0 len=0 2023-10-05T19:29:40.172 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.166+0000 7f8c12b0ee40 1 -- 172.21.15.167:0/4016842607 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:42 5.12 5:49953fa1:::default.realm:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32) v8 -- 0x55f80c5c4d80 con 0x55f80c5bba60 2023-10-05T19:29:40.173 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.166+0000 7f8c10e44700 1 -- 172.21.15.167:0/4016842607 <== osd.1 v2:172.21.15.167:6800/30035 16 ==== osd_op_reply(42 default.realm [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 ==== 157+0+0 (crc 0 0 0) 0x7f8c0002f080 con 0x55f80c5bba60 2023-10-05T19:29:40.173 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.166+0000 7f8c12b0ee40 20 rados_obj.operate() r=-2 bl.length=0 2023-10-05T19:29:40.173 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.166+0000 7f8c12b0ee40 20 rados->read ofs=0 len=0 2023-10-05T19:29:40.173 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.166+0000 7f8c12b0ee40 1 -- 172.21.15.167:0/4016842607 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:43 5.f 5:f4c53578:::zonegroups_names.default:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32) v8 -- 0x55f80c5c5180 con 0x55f80c5bba60 2023-10-05T19:29:40.174 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.166+0000 7f8c10e44700 1 -- 172.21.15.167:0/4016842607 <== osd.1 v2:172.21.15.167:6800/30035 17 ==== osd_op_reply(43 zonegroups_names.default [read 0~46 out=46b] v0'0 uv2 ondisk = 0) v8 ==== 168+0+46 (crc 0 0 0) 0x7f8c0002f080 con 0x55f80c5bba60 2023-10-05T19:29:40.174 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.166+0000 7f8c12b0ee40 20 rados_obj.operate() r=0 bl.length=46 2023-10-05T19:29:40.174 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.166+0000 7f8c12b0ee40 1 -- 172.21.15.167:0/4016842607 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:44 5.0 5:00000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x55f80c5c6190 con 0x55f80c5c1c70 2023-10-05T19:29:40.174 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.170+0000 7f8c10643700 1 -- 172.21.15.167:0/4016842607 <== osd.3 v2:172.21.15.167:6808/30036 11 ==== osd_op_reply(44 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f8c0802b070 con 0x55f80c5c1c70 2023-10-05T19:29:40.174 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.170+0000 7f8c10643700 1 -- 172.21.15.167:0/4016842607 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:45 5.10 5:08000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x55f80c5c6190 con 0x7f8c08044c30 2023-10-05T19:29:40.175 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.170+0000 7f8c0fe42700 1 -- 172.21.15.167:0/4016842607 <== osd.2 v2:172.21.15.167:6816/30037 7 ==== osd_op_reply(45 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f8c0400f560 con 0x7f8c08044c30 2023-10-05T19:29:40.175 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.170+0000 7f8c0fe42700 1 -- 172.21.15.167:0/4016842607 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:46 5.8 5:10000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x55f80c5c6190 con 0x7f8c0400e8c0 2023-10-05T19:29:40.175 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.170+0000 7f8c10e44700 1 -- 172.21.15.167:0/4016842607 <== osd.0 v2:172.21.15.118:6800/27690 11 ==== osd_op_reply(46 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f8c0002f080 con 0x7f8c0400e8c0 2023-10-05T19:29:40.176 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.170+0000 7f8c10e44700 1 -- 172.21.15.167:0/4016842607 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:47 5.18 5:18000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x55f80c5c1310 con 0x55f80c5bba60 2023-10-05T19:29:40.176 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.170+0000 7f8c10e44700 1 -- 172.21.15.167:0/4016842607 <== osd.1 v2:172.21.15.167:6800/30035 18 ==== osd_op_reply(47 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f8c0002f080 con 0x55f80c5bba60 2023-10-05T19:29:40.176 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.170+0000 7f8c10e44700 1 -- 172.21.15.167:0/4016842607 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:48 5.4 5:20000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x55f80c5c1310 con 0x55f80c5bba60 2023-10-05T19:29:40.177 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.174+0000 7f8c10e44700 1 -- 172.21.15.167:0/4016842607 <== osd.1 v2:172.21.15.167:6800/30035 19 ==== osd_op_reply(48 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f8c0002f080 con 0x55f80c5bba60 2023-10-05T19:29:40.177 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.174+0000 7f8c10e44700 1 -- 172.21.15.167:0/4016842607 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:49 5.14 5:28000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x55f80c5c1310 con 0x7f8c0400e8c0 2023-10-05T19:29:40.178 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.174+0000 7f8c10e44700 1 -- 172.21.15.167:0/4016842607 <== osd.0 v2:172.21.15.118:6800/27690 12 ==== osd_op_reply(49 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f8c0002f080 con 0x7f8c0400e8c0 2023-10-05T19:29:40.178 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.174+0000 7f8c10e44700 1 -- 172.21.15.167:0/4016842607 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:50 5.c 5:30000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x55f80c5c1310 con 0x7f8c0400e8c0 2023-10-05T19:29:40.179 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.174+0000 7f8c10e44700 1 -- 172.21.15.167:0/4016842607 <== osd.0 v2:172.21.15.118:6800/27690 13 ==== osd_op_reply(50 [pgnls start_epoch 32 out=112b] v23'1 uv1 ondisk = 1) v8 ==== 144+0+112 (crc 0 0 0) 0x7f8c0002f080 con 0x7f8c0400e8c0 2023-10-05T19:29:40.179 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.174+0000 7f8c10e44700 1 -- 172.21.15.167:0/4016842607 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:51 5.1c 5:38000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x55f80c5c1310 con 0x7f8c08044c30 2023-10-05T19:29:40.180 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.174+0000 7f8c0fe42700 1 -- 172.21.15.167:0/4016842607 <== osd.2 v2:172.21.15.167:6816/30037 8 ==== osd_op_reply(51 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f8c0400f560 con 0x7f8c08044c30 2023-10-05T19:29:40.180 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.174+0000 7f8c0fe42700 1 -- 172.21.15.167:0/4016842607 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:52 5.2 5:40000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x55f80c5c1310 con 0x55f80c5c1c70 2023-10-05T19:29:40.180 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.178+0000 7f8c10643700 1 -- 172.21.15.167:0/4016842607 <== osd.3 v2:172.21.15.167:6808/30036 12 ==== osd_op_reply(52 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f8c0802b070 con 0x55f80c5c1c70 2023-10-05T19:29:40.181 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.178+0000 7f8c10643700 1 -- 172.21.15.167:0/4016842607 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:53 5.12 5:48000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x55f80c5c1310 con 0x55f80c5bba60 2023-10-05T19:29:40.181 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.178+0000 7f8c10e44700 1 -- 172.21.15.167:0/4016842607 <== osd.1 v2:172.21.15.167:6800/30035 20 ==== osd_op_reply(53 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f8c0002f080 con 0x55f80c5bba60 2023-10-05T19:29:40.181 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.178+0000 7f8c10e44700 1 -- 172.21.15.167:0/4016842607 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:54 5.a 5:50000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x55f80c5c1310 con 0x7f8c0400e8c0 2023-10-05T19:29:40.182 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.178+0000 7f8c10e44700 1 -- 172.21.15.167:0/4016842607 <== osd.0 v2:172.21.15.118:6800/27690 14 ==== osd_op_reply(54 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f8c0002f080 con 0x7f8c0400e8c0 2023-10-05T19:29:40.183 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.178+0000 7f8c10e44700 1 -- 172.21.15.167:0/4016842607 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:55 5.1a 5:58000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x55f80c5c1310 con 0x55f80c5bba60 2023-10-05T19:29:40.183 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.178+0000 7f8c10e44700 1 -- 172.21.15.167:0/4016842607 <== osd.1 v2:172.21.15.167:6800/30035 21 ==== osd_op_reply(55 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f8c0002f080 con 0x55f80c5bba60 2023-10-05T19:29:40.183 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.178+0000 7f8c10e44700 1 -- 172.21.15.167:0/4016842607 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:56 5.6 5:60000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x55f80c5c1310 con 0x7f8c08044c30 2023-10-05T19:29:40.184 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.178+0000 7f8c0fe42700 1 -- 172.21.15.167:0/4016842607 <== osd.2 v2:172.21.15.167:6816/30037 9 ==== osd_op_reply(56 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f8c0400f560 con 0x7f8c08044c30 2023-10-05T19:29:40.184 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.178+0000 7f8c0fe42700 1 -- 172.21.15.167:0/4016842607 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:57 5.16 5:68000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x55f80c5c1310 con 0x55f80c5c1c70 2023-10-05T19:29:40.185 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.182+0000 7f8c10643700 1 -- 172.21.15.167:0/4016842607 <== osd.3 v2:172.21.15.167:6808/30036 13 ==== osd_op_reply(57 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f8c0802b070 con 0x55f80c5c1c70 2023-10-05T19:29:40.185 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.182+0000 7f8c10643700 1 -- 172.21.15.167:0/4016842607 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:58 5.e 5:70000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x55f80c5c1310 con 0x55f80c5c1c70 2023-10-05T19:29:40.186 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.182+0000 7f8c10643700 1 -- 172.21.15.167:0/4016842607 <== osd.3 v2:172.21.15.167:6808/30036 14 ==== osd_op_reply(58 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f8c0802b070 con 0x55f80c5c1c70 2023-10-05T19:29:40.186 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.182+0000 7f8c10643700 1 -- 172.21.15.167:0/4016842607 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:59 5.1e 5:78000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x55f80c5c1310 con 0x7f8c0400e8c0 2023-10-05T19:29:40.187 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.182+0000 7f8c10e44700 1 -- 172.21.15.167:0/4016842607 <== osd.0 v2:172.21.15.118:6800/27690 15 ==== osd_op_reply(59 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f8c0002f080 con 0x7f8c0400e8c0 2023-10-05T19:29:40.187 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.182+0000 7f8c10e44700 1 -- 172.21.15.167:0/4016842607 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:60 5.1 5:80000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x55f80c5c1310 con 0x55f80c5c1c70 2023-10-05T19:29:40.187 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.182+0000 7f8c10643700 1 -- 172.21.15.167:0/4016842607 <== osd.3 v2:172.21.15.167:6808/30036 15 ==== osd_op_reply(60 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f8c0802b070 con 0x55f80c5c1c70 2023-10-05T19:29:40.188 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.182+0000 7f8c10643700 1 -- 172.21.15.167:0/4016842607 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:61 5.11 5:88000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x55f80c5c1310 con 0x55f80c5c1c70 2023-10-05T19:29:40.188 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.182+0000 7f8c10643700 1 -- 172.21.15.167:0/4016842607 <== osd.3 v2:172.21.15.167:6808/30036 16 ==== osd_op_reply(61 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f8c0802b070 con 0x55f80c5c1c70 2023-10-05T19:29:40.188 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.182+0000 7f8c10643700 1 -- 172.21.15.167:0/4016842607 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:62 5.9 5:90000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x55f80c5c1310 con 0x55f80c5bba60 2023-10-05T19:29:40.189 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.186+0000 7f8c10e44700 1 -- 172.21.15.167:0/4016842607 <== osd.1 v2:172.21.15.167:6800/30035 22 ==== osd_op_reply(62 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f8c0002f080 con 0x55f80c5bba60 2023-10-05T19:29:40.189 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.186+0000 7f8c10e44700 1 -- 172.21.15.167:0/4016842607 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:63 5.19 5:98000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x55f80c5c1310 con 0x55f80c5bba60 2023-10-05T19:29:40.190 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.186+0000 7f8c10e44700 1 -- 172.21.15.167:0/4016842607 <== osd.1 v2:172.21.15.167:6800/30035 23 ==== osd_op_reply(63 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f8c0002f080 con 0x55f80c5bba60 2023-10-05T19:29:40.190 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.186+0000 7f8c10e44700 1 -- 172.21.15.167:0/4016842607 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:64 5.5 5:a0000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x55f80c5c1310 con 0x7f8c0400e8c0 2023-10-05T19:29:40.191 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.186+0000 7f8c10e44700 1 -- 172.21.15.167:0/4016842607 <== osd.0 v2:172.21.15.118:6800/27690 16 ==== osd_op_reply(64 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f8c0002f080 con 0x7f8c0400e8c0 2023-10-05T19:29:40.191 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.186+0000 7f8c10e44700 1 -- 172.21.15.167:0/4016842607 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:65 5.15 5:a8000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x55f80c5c1310 con 0x55f80c5c1c70 2023-10-05T19:29:40.191 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.186+0000 7f8c10643700 1 -- 172.21.15.167:0/4016842607 <== osd.3 v2:172.21.15.167:6808/30036 17 ==== osd_op_reply(65 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f8c0802b070 con 0x55f80c5c1c70 2023-10-05T19:29:40.192 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.186+0000 7f8c10643700 1 -- 172.21.15.167:0/4016842607 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:66 5.d 5:b0000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x55f80c5c1310 con 0x7f8c08044c30 2023-10-05T19:29:40.192 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.190+0000 7f8c0fe42700 1 -- 172.21.15.167:0/4016842607 <== osd.2 v2:172.21.15.167:6816/30037 10 ==== osd_op_reply(66 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f8c0400f560 con 0x7f8c08044c30 2023-10-05T19:29:40.193 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.190+0000 7f8c0fe42700 1 -- 172.21.15.167:0/4016842607 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:67 5.1d 5:b8000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x55f80c5c1310 con 0x55f80c5bba60 2023-10-05T19:29:40.193 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.190+0000 7f8c10e44700 1 -- 172.21.15.167:0/4016842607 <== osd.1 v2:172.21.15.167:6800/30035 24 ==== osd_op_reply(67 [pgnls start_epoch 32 out=107b] v30'2 uv2 ondisk = 1) v8 ==== 144+0+107 (crc 0 0 0) 0x7f8c0002f080 con 0x55f80c5bba60 2023-10-05T19:29:40.193 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.190+0000 7f8c10e44700 1 -- 172.21.15.167:0/4016842607 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:68 5.3 5:c0000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x55f80c5c1310 con 0x7f8c0400e8c0 2023-10-05T19:29:40.194 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.190+0000 7f8c10e44700 1 -- 172.21.15.167:0/4016842607 <== osd.0 v2:172.21.15.118:6800/27690 17 ==== osd_op_reply(68 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f8c0002f080 con 0x7f8c0400e8c0 2023-10-05T19:29:40.195 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.190+0000 7f8c10e44700 1 -- 172.21.15.167:0/4016842607 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:69 5.13 5:c8000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x55f80c5c1310 con 0x55f80c5c1c70 2023-10-05T19:29:40.195 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.190+0000 7f8c10643700 1 -- 172.21.15.167:0/4016842607 <== osd.3 v2:172.21.15.167:6808/30036 18 ==== osd_op_reply(69 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f8c0802b070 con 0x55f80c5c1c70 2023-10-05T19:29:40.195 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.190+0000 7f8c10643700 1 -- 172.21.15.167:0/4016842607 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:70 5.b 5:d0000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x55f80c5c1310 con 0x7f8c0400e8c0 2023-10-05T19:29:40.196 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.190+0000 7f8c10e44700 1 -- 172.21.15.167:0/4016842607 <== osd.0 v2:172.21.15.118:6800/27690 18 ==== osd_op_reply(70 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f8c0002f080 con 0x7f8c0400e8c0 2023-10-05T19:29:40.196 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.190+0000 7f8c10e44700 1 -- 172.21.15.167:0/4016842607 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:71 5.1b 5:d8000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x55f80c5c1310 con 0x7f8c08044c30 2023-10-05T19:29:40.197 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.194+0000 7f8c0fe42700 1 -- 172.21.15.167:0/4016842607 <== osd.2 v2:172.21.15.167:6816/30037 11 ==== osd_op_reply(71 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f8c0400f560 con 0x7f8c08044c30 2023-10-05T19:29:40.197 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.194+0000 7f8c0fe42700 1 -- 172.21.15.167:0/4016842607 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:72 5.7 5:e0000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x55f80c5c1310 con 0x55f80c5c1c70 2023-10-05T19:29:40.198 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.194+0000 7f8c10643700 1 -- 172.21.15.167:0/4016842607 <== osd.3 v2:172.21.15.167:6808/30036 19 ==== osd_op_reply(72 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f8c0802b070 con 0x55f80c5c1c70 2023-10-05T19:29:40.198 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.194+0000 7f8c10643700 1 -- 172.21.15.167:0/4016842607 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:73 5.17 5:e8000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x55f80c5c1310 con 0x7f8c0400e8c0 2023-10-05T19:29:40.199 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.194+0000 7f8c10e44700 1 -- 172.21.15.167:0/4016842607 <== osd.0 v2:172.21.15.118:6800/27690 19 ==== osd_op_reply(73 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f8c0002f080 con 0x7f8c0400e8c0 2023-10-05T19:29:40.199 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.194+0000 7f8c10e44700 1 -- 172.21.15.167:0/4016842607 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:74 5.f 5:f0000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x55f80c5c1310 con 0x55f80c5bba60 2023-10-05T19:29:40.200 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.194+0000 7f8c10e44700 1 -- 172.21.15.167:0/4016842607 <== osd.1 v2:172.21.15.167:6800/30035 25 ==== osd_op_reply(74 [pgnls start_epoch 32 out=115b] v23'2 uv2 ondisk = 1) v8 ==== 144+0+115 (crc 0 0 0) 0x7f8c0002f080 con 0x55f80c5bba60 2023-10-05T19:29:40.200 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.194+0000 7f8c10e44700 1 -- 172.21.15.167:0/4016842607 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:75 5.1f 5:f8000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x55f80c5c1310 con 0x7f8c08044c30 2023-10-05T19:29:40.201 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.198+0000 7f8c0fe42700 1 -- 172.21.15.167:0/4016842607 <== osd.2 v2:172.21.15.167:6816/30037 12 ==== osd_op_reply(75 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f8c0400f560 con 0x7f8c08044c30 2023-10-05T19:29:40.201 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.198+0000 7f8c12b0ee40 20 RGWRados::pool_iterate: got zonegroup_info.4f1f719d-9648-417b-b10c-e5d939825419 2023-10-05T19:29:40.201 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.198+0000 7f8c12b0ee40 20 RGWRados::pool_iterate: got zone_info.9a9df965-6612-45d4-a45f-6e3c889bacc2 2023-10-05T19:29:40.201 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.198+0000 7f8c12b0ee40 20 RGWRados::pool_iterate: got zone_names.default 2023-10-05T19:29:40.201 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.198+0000 7f8c12b0ee40 20 RGWRados::pool_iterate: got zonegroups_names.default 2023-10-05T19:29:40.201 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.198+0000 7f8c12b0ee40 20 rados->read ofs=0 len=0 2023-10-05T19:29:40.202 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.198+0000 7f8c12b0ee40 1 -- 172.21.15.167:0/4016842607 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:76 5.f 5:f43b6ece:::zone_names.default:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32) v8 -- 0x55f80c5c5180 con 0x55f80c5bba60 2023-10-05T19:29:40.202 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.198+0000 7f8c10e44700 1 -- 172.21.15.167:0/4016842607 <== osd.1 v2:172.21.15.167:6800/30035 26 ==== osd_op_reply(76 zone_names.default [read 0~46 out=46b] v0'0 uv1 ondisk = 0) v8 ==== 162+0+46 (crc 0 0 0) 0x7f8c0002f080 con 0x55f80c5bba60 2023-10-05T19:29:40.202 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.198+0000 7f8c12b0ee40 20 rados_obj.operate() r=0 bl.length=46 2023-10-05T19:29:40.202 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.198+0000 7f8c12b0ee40 20 rados->read ofs=0 len=0 2023-10-05T19:29:40.203 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.198+0000 7f8c12b0ee40 1 -- 172.21.15.167:0/4016842607 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:77 5.1d 5:b8374eb4:::zone_info.9a9df965-6612-45d4-a45f-6e3c889bacc2:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32) v8 -- 0x55f80c5c5530 con 0x55f80c5bba60 2023-10-05T19:29:40.203 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.198+0000 7f8c10e44700 1 -- 172.21.15.167:0/4016842607 <== osd.1 v2:172.21.15.167:6800/30035 27 ==== osd_op_reply(77 zone_info.9a9df965-6612-45d4-a45f-6e3c889bacc2 [read 0~874 out=874b] v0'0 uv2 ondisk = 0) v8 ==== 190+0+874 (crc 0 0 0) 0x7f8c0002f080 con 0x55f80c5bba60 2023-10-05T19:29:40.203 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.198+0000 7f8c12b0ee40 20 rados_obj.operate() r=0 bl.length=874 2023-10-05T19:29:40.203 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.198+0000 7f8c12b0ee40 20 rados->read ofs=0 len=0 2023-10-05T19:29:40.203 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.198+0000 7f8c12b0ee40 1 -- 172.21.15.167:0/4016842607 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:78 5.f 5:f4c53578:::zonegroups_names.default:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32) v8 -- 0x55f80c5c0e80 con 0x55f80c5bba60 2023-10-05T19:29:40.204 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.198+0000 7f8c10e44700 1 -- 172.21.15.167:0/4016842607 <== osd.1 v2:172.21.15.167:6800/30035 28 ==== osd_op_reply(78 zonegroups_names.default [read 0~46 out=46b] v0'0 uv2 ondisk = 0) v8 ==== 168+0+46 (crc 0 0 0) 0x7f8c0002f080 con 0x55f80c5bba60 2023-10-05T19:29:40.204 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.198+0000 7f8c12b0ee40 20 rados_obj.operate() r=0 bl.length=46 2023-10-05T19:29:40.205 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.198+0000 7f8c12b0ee40 20 rados->read ofs=0 len=0 2023-10-05T19:29:40.205 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.198+0000 7f8c12b0ee40 1 -- 172.21.15.167:0/4016842607 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:79 5.c 5:30d299a5:::zonegroup_info.4f1f719d-9648-417b-b10c-e5d939825419:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32) v8 -- 0x55f80c5c0e60 con 0x7f8c0400e8c0 2023-10-05T19:29:40.205 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.202+0000 7f8c10e44700 1 -- 172.21.15.167:0/4016842607 <== osd.0 v2:172.21.15.118:6800/27690 20 ==== osd_op_reply(79 zonegroup_info.4f1f719d-9648-417b-b10c-e5d939825419 [read 0~362 out=362b] v0'0 uv1 ondisk = 0) v8 ==== 195+0+362 (crc 0 0 0) 0x7f8c0002f080 con 0x7f8c0400e8c0 2023-10-05T19:29:40.205 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.202+0000 7f8c12b0ee40 20 rados_obj.operate() r=0 bl.length=362 2023-10-05T19:29:40.205 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.202+0000 7f8c12b0ee40 20 rados->read ofs=0 len=0 2023-10-05T19:29:40.205 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.202+0000 7f8c12b0ee40 1 -- 172.21.15.167:0/4016842607 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:80 5.3 5:c0ad8cbd:::region_map:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32) v8 -- 0x55f80c5c5a70 con 0x7f8c0400e8c0 2023-10-05T19:29:40.206 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.202+0000 7f8c10e44700 1 -- 172.21.15.167:0/4016842607 <== osd.0 v2:172.21.15.118:6800/27690 21 ==== osd_op_reply(80 region_map [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 ==== 154+0+0 (crc 0 0 0) 0x7f8c0002f080 con 0x7f8c0400e8c0 2023-10-05T19:29:40.206 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.202+0000 7f8c12b0ee40 20 rados_obj.operate() r=-2 bl.length=0 2023-10-05T19:29:40.206 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.202+0000 7f8c12b0ee40 20 zone default found 2023-10-05T19:29:40.206 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.202+0000 7f8c12b0ee40 4 Realm: () 2023-10-05T19:29:40.206 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.202+0000 7f8c12b0ee40 4 ZoneGroup: default (4f1f719d-9648-417b-b10c-e5d939825419) 2023-10-05T19:29:40.206 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.202+0000 7f8c12b0ee40 4 Zone: default (9a9df965-6612-45d4-a45f-6e3c889bacc2) 2023-10-05T19:29:40.206 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.202+0000 7f8c12b0ee40 10 cannot find current period zonegroup using local zonegroup configuration 2023-10-05T19:29:40.207 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.202+0000 7f8c12b0ee40 20 zonegroup default 2023-10-05T19:29:40.207 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.202+0000 7f8c12b0ee40 20 rados->read ofs=0 len=0 2023-10-05T19:29:40.207 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.202+0000 7f8c12b0ee40 1 -- 172.21.15.167:0/4016842607 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:81 5.3 5:c52100b6:::period_config.default:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32) v8 -- 0x55f80c5c5a70 con 0x7f8c0400e8c0 2023-10-05T19:29:40.208 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.202+0000 7f8c10e44700 1 -- 172.21.15.167:0/4016842607 <== osd.0 v2:172.21.15.118:6800/27690 22 ==== osd_op_reply(81 period_config.default [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 ==== 165+0+0 (crc 0 0 0) 0x7f8c0002f080 con 0x7f8c0400e8c0 2023-10-05T19:29:40.208 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.202+0000 7f8c12b0ee40 20 rados_obj.operate() r=-2 bl.length=0 2023-10-05T19:29:40.208 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.202+0000 7f8c12b0ee40 20 rados->read ofs=0 len=0 2023-10-05T19:29:40.208 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.202+0000 7f8c12b0ee40 1 -- 172.21.15.167:0/4016842607 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:82 6.2 6:4347d321:::bucket.sync-source-hints.:head [call version.read in=11b,read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32) v8 -- 0x55f80c5c73f0 con 0x55f80c5c1c70 2023-10-05T19:29:40.209 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.202+0000 7f8c10643700 1 -- 172.21.15.167:0/4016842607 <== osd.3 v2:172.21.15.167:6808/30036 20 ==== osd_op_reply(82 bucket.sync-source-hints. [call,read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 ==== 211+0+0 (crc 0 0 0) 0x7f8c0802b070 con 0x55f80c5c1c70 2023-10-05T19:29:40.209 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.202+0000 7f8c12b0ee40 20 rados_obj.operate() r=-2 bl.length=0 2023-10-05T19:29:40.209 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.202+0000 7f8c12b0ee40 20 rados->read ofs=0 len=0 2023-10-05T19:29:40.209 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.202+0000 7f8c12b0ee40 1 -- 172.21.15.167:0/4016842607 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:83 6.b 6:d467b91b:::bucket.sync-target-hints.:head [call version.read in=11b,read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32) v8 -- 0x55f80c5c77a0 con 0x55f80c5c1c70 2023-10-05T19:29:40.209 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.202+0000 7f8c10643700 1 -- 172.21.15.167:0/4016842607 <== osd.3 v2:172.21.15.167:6808/30036 21 ==== osd_op_reply(83 bucket.sync-target-hints. [call,read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 ==== 211+0+0 (crc 0 0 0) 0x7f8c0802b070 con 0x55f80c5c1c70 2023-10-05T19:29:40.209 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.202+0000 7f8c12b0ee40 20 rados_obj.operate() r=-2 bl.length=0 2023-10-05T19:29:40.209 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.202+0000 7f8c12b0ee40 20 started sync module instance, tier type = 2023-10-05T19:29:40.209 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.202+0000 7f8c12b0ee40 20 started zone id=9a9df965-6612-45d4-a45f-6e3c889bacc2 (name=default) with tier type = 2023-10-05T19:29:40.209 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.202+0000 7f8c12b0ee40 1 -- 172.21.15.167:0/4016842607 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:84 7.1f 7:f95f44c2:::notify.0:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e32) v8 -- 0x55f80c5c6ab0 con 0x7f8c08044c30 2023-10-05T19:29:40.211 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.206+0000 7f8c0fe42700 1 -- 172.21.15.167:0/4016842607 <== osd.2 v2:172.21.15.167:6816/30037 13 ==== osd_op_reply(84 notify.0 [create] v32'246 uv246 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f8c0400f560 con 0x7f8c08044c30 2023-10-05T19:29:40.212 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.206+0000 7f8c12b0ee40 1 -- 172.21.15.167:0/4016842607 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:85 7.1f 7:f95f44c2:::notify.0:head [watch watch cookie 94523847628080] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e32) v8 -- 0x55f80c5c7b50 con 0x7f8c08044c30 2023-10-05T19:29:40.212 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.206+0000 7f8c12b0ee40 1 -- 172.21.15.167:0/4016842607 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:86 7.0 7:05bf5b68:::notify.1:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e32) v8 -- 0x55f80c5c83f0 con 0x55f80c5bba60 2023-10-05T19:29:40.215 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.210+0000 7f8c0fe42700 1 -- 172.21.15.167:0/4016842607 <== osd.2 v2:172.21.15.167:6816/30037 14 ==== osd_op_reply(85 notify.0 [watch watch cookie 94523847628080] v32'247 uv246 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f8c0400f560 con 0x7f8c08044c30 2023-10-05T19:29:40.215 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.210+0000 7f8c10e44700 1 -- 172.21.15.167:0/4016842607 <== osd.1 v2:172.21.15.167:6800/30035 29 ==== osd_op_reply(86 notify.1 [create] v32'246 uv246 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f8c0002f080 con 0x55f80c5bba60 2023-10-05T19:29:40.215 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.210+0000 7f8c12b0ee40 1 -- 172.21.15.167:0/4016842607 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:87 7.0 7:05bf5b68:::notify.1:head [watch watch cookie 94523847638784] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e32) v8 -- 0x55f80c5c8840 con 0x55f80c5bba60 2023-10-05T19:29:40.215 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.210+0000 7f8c12b0ee40 1 -- 172.21.15.167:0/4016842607 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:88 7.15 7:a93a5511:::notify.2:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e32) v8 -- 0x55f80c5c8840 con 0x55f80c5c1c70 2023-10-05T19:29:40.218 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.214+0000 7f8c10e44700 1 -- 172.21.15.167:0/4016842607 <== osd.1 v2:172.21.15.167:6800/30035 30 ==== osd_op_reply(87 notify.1 [watch watch cookie 94523847638784] v32'247 uv246 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f8c00050040 con 0x55f80c5bba60 2023-10-05T19:29:40.218 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.214+0000 7f8c10643700 1 -- 172.21.15.167:0/4016842607 <== osd.3 v2:172.21.15.167:6808/30036 22 ==== osd_op_reply(88 notify.2 [create] v32'246 uv246 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f8c0802b070 con 0x55f80c5c1c70 2023-10-05T19:29:40.219 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.214+0000 7f8c12b0ee40 1 -- 172.21.15.167:0/4016842607 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:89 7.15 7:a93a5511:::notify.2:head [watch watch cookie 94523847639888] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e32) v8 -- 0x55f80c5c90e0 con 0x55f80c5c1c70 2023-10-05T19:29:40.219 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.214+0000 7f8c12b0ee40 1 -- 172.21.15.167:0/4016842607 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:90 7.e 7:7759931f:::notify.3:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e32) v8 -- 0x55f80c5c9bc0 con 0x7f8c08044c30 2023-10-05T19:29:40.221 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.218+0000 7f8c0fe42700 1 -- 172.21.15.167:0/4016842607 <== osd.2 v2:172.21.15.167:6816/30037 15 ==== osd_op_reply(90 notify.3 [create] v32'246 uv246 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f8c0400f560 con 0x7f8c08044c30 2023-10-05T19:29:40.221 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.218+0000 7f8c12b0ee40 1 -- 172.21.15.167:0/4016842607 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:91 7.e 7:7759931f:::notify.3:head [watch watch cookie 94523847647168] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e32) v8 -- 0x55f80c5ca900 con 0x7f8c08044c30 2023-10-05T19:29:40.221 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.218+0000 7f8c12b0ee40 1 -- 172.21.15.167:0/4016842607 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:92 7.d 7:b4812045:::notify.4:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e32) v8 -- 0x55f80c5cb1e0 con 0x55f80c5bba60 2023-10-05T19:29:40.221 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.218+0000 7f8c10643700 1 -- 172.21.15.167:0/4016842607 <== osd.3 v2:172.21.15.167:6808/30036 23 ==== osd_op_reply(89 notify.2 [watch watch cookie 94523847639888] v32'247 uv246 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f8c0802b070 con 0x55f80c5c1c70 2023-10-05T19:29:40.223 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.218+0000 7f8c0fe42700 1 -- 172.21.15.167:0/4016842607 <== osd.2 v2:172.21.15.167:6816/30037 16 ==== osd_op_reply(91 notify.3 [watch watch cookie 94523847647168] v32'247 uv246 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f8c0400f560 con 0x7f8c08044c30 2023-10-05T19:29:40.224 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.218+0000 7f8c10e44700 1 -- 172.21.15.167:0/4016842607 <== osd.1 v2:172.21.15.167:6800/30035 31 ==== osd_op_reply(92 notify.4 [create] v32'246 uv246 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f8c00050040 con 0x55f80c5bba60 2023-10-05T19:29:40.224 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.218+0000 7f8c12b0ee40 1 -- 172.21.15.167:0/4016842607 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:93 7.d 7:b4812045:::notify.4:head [watch watch cookie 94523847648272] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e32) v8 -- 0x55f80c5cacb0 con 0x55f80c5bba60 2023-10-05T19:29:40.224 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.218+0000 7f8c12b0ee40 1 -- 172.21.15.167:0/4016842607 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:94 7.3 7:c609908c:::notify.5:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e32) v8 -- 0x55f80c5cbad0 con 0x7f8c0400e8c0 2023-10-05T19:29:40.226 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.222+0000 7f8c10e44700 1 -- 172.21.15.167:0/4016842607 <== osd.1 v2:172.21.15.167:6800/30035 32 ==== osd_op_reply(93 notify.4 [watch watch cookie 94523847648272] v32'247 uv246 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f8c00050040 con 0x55f80c5bba60 2023-10-05T19:29:40.227 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.222+0000 7f8c10e44700 1 -- 172.21.15.167:0/4016842607 <== osd.0 v2:172.21.15.118:6800/27690 23 ==== osd_op_reply(94 notify.5 [create] v32'246 uv246 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f8c00050040 con 0x7f8c0400e8c0 2023-10-05T19:29:40.227 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.222+0000 7f8c12b0ee40 1 -- 172.21.15.167:0/4016842607 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:95 7.3 7:c609908c:::notify.5:head [watch watch cookie 94523847642096] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e32) v8 -- 0x55f80c5cba80 con 0x7f8c0400e8c0 2023-10-05T19:29:40.227 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.222+0000 7f8c12b0ee40 1 -- 172.21.15.167:0/4016842607 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:96 7.14 7:2b04a3e9:::notify.6:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e32) v8 -- 0x55f80c5cc490 con 0x7f8c0400e8c0 2023-10-05T19:29:40.229 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.226+0000 7f8c10e44700 1 -- 172.21.15.167:0/4016842607 <== osd.0 v2:172.21.15.118:6800/27690 24 ==== osd_op_reply(95 notify.5 [watch watch cookie 94523847642096] v32'247 uv246 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f8c00050040 con 0x7f8c0400e8c0 2023-10-05T19:29:40.229 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.226+0000 7f8c10e44700 1 -- 172.21.15.167:0/4016842607 <== osd.0 v2:172.21.15.118:6800/27690 25 ==== osd_op_reply(96 notify.6 [create] v32'246 uv246 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f8c00050040 con 0x7f8c0400e8c0 2023-10-05T19:29:40.229 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.226+0000 7f8c12b0ee40 1 -- 172.21.15.167:0/4016842607 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:97 7.14 7:2b04a3e9:::notify.6:head [watch watch cookie 94523847653040] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e32) v8 -- 0x55f80c5cc490 con 0x7f8c0400e8c0 2023-10-05T19:29:40.230 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.226+0000 7f8c12b0ee40 1 -- 172.21.15.167:0/4016842607 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:98 7.9 7:93e5b521:::notify.7:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e32) v8 -- 0x55f80c5ccea0 con 0x7f8c0400e8c0 2023-10-05T19:29:40.232 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.226+0000 7f8c10e44700 1 -- 172.21.15.167:0/4016842607 <== osd.0 v2:172.21.15.118:6800/27690 26 ==== osd_op_reply(98 notify.7 [create] v32'246 uv246 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f8c00050040 con 0x7f8c0400e8c0 2023-10-05T19:29:40.232 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.226+0000 7f8c12b0ee40 1 -- 172.21.15.167:0/4016842607 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:99 7.9 7:93e5b521:::notify.7:head [watch watch cookie 94523847657904] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e32) v8 -- 0x55f80c5cd2f0 con 0x7f8c0400e8c0 2023-10-05T19:29:40.232 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.226+0000 7f8c12b0ee40 20 add_watcher() i=0 2023-10-05T19:29:40.233 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.226+0000 7f8c12b0ee40 20 add_watcher() i=1 2023-10-05T19:29:40.233 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.226+0000 7f8c12b0ee40 20 add_watcher() i=2 2023-10-05T19:29:40.233 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.226+0000 7f8c12b0ee40 20 add_watcher() i=3 2023-10-05T19:29:40.233 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.226+0000 7f8c12b0ee40 20 add_watcher() i=4 2023-10-05T19:29:40.233 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.226+0000 7f8c12b0ee40 20 add_watcher() i=5 2023-10-05T19:29:40.233 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.226+0000 7f8c10e44700 1 -- 172.21.15.167:0/4016842607 <== osd.0 v2:172.21.15.118:6800/27690 27 ==== osd_op_reply(97 notify.6 [watch watch cookie 94523847653040] v32'247 uv246 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f8c00050040 con 0x7f8c0400e8c0 2023-10-05T19:29:40.233 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.226+0000 7f8c12b0ee40 20 add_watcher() i=6 2023-10-05T19:29:40.235 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.230+0000 7f8c10e44700 1 -- 172.21.15.167:0/4016842607 <== osd.0 v2:172.21.15.118:6800/27690 28 ==== osd_op_reply(99 notify.7 [watch watch cookie 94523847657904] v32'247 uv246 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f8c00050040 con 0x7f8c0400e8c0 2023-10-05T19:29:40.235 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.230+0000 7f8c12b0ee40 20 add_watcher() i=7 2023-10-05T19:29:40.235 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.230+0000 7f8c12b0ee40 2 all 8 watchers are set, enabling cache 2023-10-05T19:29:40.235 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.230+0000 7f8c12b0ee40 1 -- 172.21.15.167:0/4016842607 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:100 6.4 6:22d26bf9:::data_loggenerations_metadata:head [call version.check_conds in=74b,call version.read in=11b,read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32) v8 -- 0x55f80c5d0320 con 0x55f80c5bba60 2023-10-05T19:29:40.235 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.230+0000 7f8c10e44700 1 -- 172.21.15.167:0/4016842607 <== osd.1 v2:172.21.15.167:6800/30035 33 ==== osd_op_reply(100 data_loggenerations_metadata [call,call out=48b,read 0~28 out=28b] v0'0 uv1 ondisk = 0) v8 ==== 256+0+76 (crc 0 0 0) 0x7f8c00050040 con 0x55f80c5bba60 2023-10-05T19:29:40.235 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.230+0000 7f8c12b0ee40 1 -- 172.21.15.167:0/4016842607 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:101 6.4 6:22d26bf9:::data_loggenerations_metadata:head [watch watch cookie 94523847659008] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e32) v8 -- 0x55f80c5cd740 con 0x55f80c5bba60 2023-10-05T19:29:40.237 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.234+0000 7f8c10e44700 1 -- 172.21.15.167:0/4016842607 <== osd.1 v2:172.21.15.167:6800/30035 34 ==== osd_op_reply(101 data_loggenerations_metadata [watch watch cookie 94523847659008] v32'237 uv1 ondisk = 0) v8 ==== 172+0+0 (crc 0 0 0) 0x7f8c00050040 con 0x55f80c5bba60 2023-10-05T19:29:40.238 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.234+0000 7f8be1fe3700 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2023-10-05T19:29:40.238 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.234+0000 7f8c12b0ee40 20 check_secure_mon_conn(): auth registy supported: methods=[2] modes=[2,1] 2023-10-05T19:29:40.238 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.234+0000 7f8c12b0ee40 20 check_secure_mon_conn(): mode 1 is insecure 2023-10-05T19:29:40.238 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.234+0000 7f8c12b0ee40 1 Processor -- start 2023-10-05T19:29:40.238 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.234+0000 7f8c12b0ee40 1 -- start start 2023-10-05T19:29:40.239 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.234+0000 7f8c12b0ee40 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55f80c5da860 0x55f80c5dac30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-05T19:29:40.240 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.234+0000 7f8c12b0ee40 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f80c5db170 con 0x55f80c5da860 2023-10-05T19:29:40.240 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.234+0000 7f8c0fe42700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55f80c5da860 0x55f80c5dac30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-05T19:29:40.240 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.234+0000 7f8c0fe42700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55f80c5da860 0x55f80c5dac30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.118:3300/0 says I am v2:172.21.15.167:56804/0 (socket says 172.21.15.167:56804) 2023-10-05T19:29:40.240 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.234+0000 7f8c0fe42700 1 -- 172.21.15.167:0/2055815489 learned_addr learned my addr 172.21.15.167:0/2055815489 (peer_addr_for_me v2:172.21.15.167:0/0) 2023-10-05T19:29:40.240 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.234+0000 7f8c0fe42700 1 -- 172.21.15.167:0/2055815489 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55f80c5db2b0 con 0x55f80c5da860 2023-10-05T19:29:40.240 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.234+0000 7f8c0fe42700 1 --2- 172.21.15.167:0/2055815489 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55f80c5da860 0x55f80c5dac30 secure :-1 s=READY pgs=630 cs=0 l=1 rev1=1 crypto rx=0x7f8c04000fb0 tx=0x7f8c04006000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f06bdc2b30db6e2d server_cookie=0 in_seq=0 out_seq=0 2023-10-05T19:29:40.241 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.234+0000 7f8be17e2700 1 -- 172.21.15.167:0/2055815489 <== mon.0 v2:172.21.15.118:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f8c0400db40 con 0x55f80c5da860 2023-10-05T19:29:40.241 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.234+0000 7f8be17e2700 1 -- 172.21.15.167:0/2055815489 <== mon.0 v2:172.21.15.118:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8c0400dca0 con 0x55f80c5da860 2023-10-05T19:29:40.242 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.234+0000 7f8be17e2700 1 -- 172.21.15.167:0/2055815489 <== mon.0 v2:172.21.15.118:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f8c0400de10 con 0x55f80c5da860 2023-10-05T19:29:40.242 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.238+0000 7f8c12b0ee40 1 -- 172.21.15.167:0/2055815489 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55f80c5da860 msgr2=0x55f80c5dac30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-05T19:29:40.242 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.238+0000 7f8c12b0ee40 1 --2- 172.21.15.167:0/2055815489 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55f80c5da860 0x55f80c5dac30 secure :-1 s=READY pgs=630 cs=0 l=1 rev1=1 crypto rx=0x7f8c04000fb0 tx=0x7f8c04006000 comp rx=0 tx=0).stop 2023-10-05T19:29:40.242 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.238+0000 7f8c12b0ee40 1 -- 172.21.15.167:0/2055815489 shutdown_connections 2023-10-05T19:29:40.242 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.238+0000 7f8c12b0ee40 1 --2- 172.21.15.167:0/2055815489 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55f80c5da860 0x55f80c5dac30 unknown :-1 s=CLOSED pgs=630 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-05T19:29:40.242 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.238+0000 7f8c12b0ee40 1 -- 172.21.15.167:0/2055815489 >> 172.21.15.167:0/2055815489 conn(0x55f80c5d5c20 msgr2=0x55f80c5d3d70 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-05T19:29:40.242 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.238+0000 7f8c12b0ee40 1 -- 172.21.15.167:0/2055815489 shutdown_connections 2023-10-05T19:29:40.242 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.238+0000 7f8c12b0ee40 1 -- 172.21.15.167:0/2055815489 wait complete. 2023-10-05T19:29:40.244 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.238+0000 7f8c12b0ee40 1 Processor -- start 2023-10-05T19:29:40.244 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.238+0000 7f8c12b0ee40 1 -- start start 2023-10-05T19:29:40.244 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.238+0000 7f8c12b0ee40 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55f80c5da860 0x55f80c5e03b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-05T19:29:40.244 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.238+0000 7f8c12b0ee40 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f80c5e13b0 con 0x55f80c5da860 2023-10-05T19:29:40.244 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.238+0000 7f8c0fe42700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55f80c5da860 0x55f80c5e03b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-05T19:29:40.244 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.238+0000 7f8c0fe42700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55f80c5da860 0x55f80c5e03b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.118:3300/0 says I am v2:172.21.15.167:56814/0 (socket says 172.21.15.167:56814) 2023-10-05T19:29:40.244 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.238+0000 7f8c0fe42700 1 -- 172.21.15.167:0/919496971 learned_addr learned my addr 172.21.15.167:0/919496971 (peer_addr_for_me v2:172.21.15.167:0/0) 2023-10-05T19:29:40.244 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.238+0000 7f8c0fe42700 1 -- 172.21.15.167:0/919496971 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8c04006070 con 0x55f80c5da860 2023-10-05T19:29:40.245 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.238+0000 7f8c0fe42700 1 --2- 172.21.15.167:0/919496971 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55f80c5da860 0x55f80c5e03b0 secure :-1 s=READY pgs=631 cs=0 l=1 rev1=1 crypto rx=0x7f8c0400efd0 tx=0x7f8c04032000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-05T19:29:40.245 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.238+0000 7f8b5b7fe700 1 -- 172.21.15.167:0/919496971 <== mon.0 v2:172.21.15.118:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f8c0400de10 con 0x55f80c5da860 2023-10-05T19:29:40.245 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.238+0000 7f8b5b7fe700 1 -- 172.21.15.167:0/919496971 <== mon.0 v2:172.21.15.118:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8c0403dd40 con 0x55f80c5da860 2023-10-05T19:29:40.246 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.238+0000 7f8c12b0ee40 1 -- 172.21.15.167:0/919496971 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x55f80c5e09b0 con 0x55f80c5da860 2023-10-05T19:29:40.246 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.238+0000 7f8b5b7fe700 1 -- 172.21.15.167:0/919496971 <== mon.0 v2:172.21.15.118:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f8c04047470 con 0x55f80c5da860 2023-10-05T19:29:40.246 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.242+0000 7f8c12b0ee40 1 -- 172.21.15.167:0/919496971 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55f80c5e0df0 con 0x55f80c5da860 2023-10-05T19:29:40.246 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.242+0000 7f8b5b7fe700 1 -- 172.21.15.167:0/919496971 <== mon.0 v2:172.21.15.118:3300/0 4 ==== mgrmap(e 5) v1 ==== 40329+0+0 (secure 0 0 0) 0x7f8c0401a040 con 0x55f80c5da860 2023-10-05T19:29:40.247 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.242+0000 7f8b5b7fe700 1 --2- 172.21.15.167:0/919496971 >> [v2:172.21.15.167:6824/29889,v1:172.21.15.167:6825/29889] conn(0x7f8b50030c00 0x7f8b500330b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-05T19:29:40.247 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.242+0000 7f8b5b7fe700 1 -- 172.21.15.167:0/919496971 <== mon.0 v2:172.21.15.118:3300/0 5 ==== osd_map(32..32 src has 1..32) v4 ==== 6260+0+0 (secure 0 0 0) 0x7f8c04023110 con 0x55f80c5da860 2023-10-05T19:29:40.247 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.242+0000 7f8c10e44700 1 --2- 172.21.15.167:0/919496971 >> [v2:172.21.15.167:6824/29889,v1:172.21.15.167:6825/29889] conn(0x7f8b50030c00 0x7f8b500330b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-05T19:29:40.247 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.242+0000 7f8c12b0ee40 5 note: GC not initialized 2023-10-05T19:29:40.247 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.242+0000 7f8c10e44700 1 --2- 172.21.15.167:0/919496971 >> [v2:172.21.15.167:6824/29889,v1:172.21.15.167:6825/29889] conn(0x7f8b50030c00 0x7f8b500330b0 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x55f80c5cc280 tx=0x7f8c00000f90 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-05T19:29:40.249 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.246+0000 7f8b58ff9700 20 reqs_thread_entry: start 2023-10-05T19:29:40.250 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.246+0000 7f8b4e7fc700 20 reqs_thread_entry: start 2023-10-05T19:29:40.250 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.246+0000 7f8c12b0ee40 20 init_complete bucket index max shards: 11 2023-10-05T19:29:40.250 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.246+0000 7f8c12b0ee40 10 rgw notify: Started notification manager with: 1 workers 2023-10-05T19:29:40.251 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.246+0000 7f8b3bfff700 20 reqs_thread_entry: start 2023-10-05T19:29:40.251 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.246+0000 7f8c12b0ee40 20 get_system_obj_state: rctx=0x7ffd9da66fa8 obj=default.rgw.meta:users.uid:fud state=0x55f80c4630b0 s->prefetch_data=0 2023-10-05T19:29:40.251 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.246+0000 7f8c12b0ee40 10 cache get: name=default.rgw.meta+users.uid+fud : miss 2023-10-05T19:29:40.251 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.246+0000 7f8c12b0ee40 1 -- 172.21.15.167:0/4016842607 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:102 8.e 8:75f48285:users.uid::fud:head [call version.read in=11b,getxattrs,stat] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32) v8 -- 0x55f80c648b60 con 0x7f8c0400e8c0 2023-10-05T19:29:40.251 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.246+0000 7f8b4cff9700 20 rgw notify: INFO: next queues processing will happen at: Thu Oct 5 19:30:10 2023 2023-10-05T19:29:40.251 INFO:teuthology.orchestra.run.smithi167.stderr: 2023-10-05T19:29:40.252 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.246+0000 7f8c10e44700 1 -- 172.21.15.167:0/4016842607 <== osd.0 v2:172.21.15.118:6800/27690 29 ==== osd_op_reply(102 fud [call,getxattrs,stat] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 ==== 231+0+0 (crc 0 0 0) 0x7f8c00050040 con 0x7f8c0400e8c0 2023-10-05T19:29:40.253 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.246+0000 7f8c12b0ee40 10 cache put: name=default.rgw.meta+users.uid+fud info.flags=0x0 2023-10-05T19:29:40.253 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.246+0000 7f8c12b0ee40 10 adding default.rgw.meta+users.uid+fud to cache LRU end 2023-10-05T19:29:40.253 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.246+0000 7f8c12b0ee40 20 get_system_obj_state: rctx=0x7ffd9da66c98 obj=default.rgw.meta:users.keys:p5YnriCv1nAtykxBrupQ state=0x55f80c4630b0 s->prefetch_data=0 2023-10-05T19:29:40.253 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.246+0000 7f8c12b0ee40 10 cache get: name=default.rgw.meta+users.keys+p5YnriCv1nAtykxBrupQ : miss 2023-10-05T19:29:40.253 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.246+0000 7f8c12b0ee40 1 -- 172.21.15.167:0/4016842607 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:103 8.13 8:cc695d75:users.keys::p5YnriCv1nAtykxBrupQ:head [getxattrs,stat] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32) v8 -- 0x55f80c648b60 con 0x7f8c0400e8c0 2023-10-05T19:29:40.253 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.246+0000 7f8c10e44700 1 -- 172.21.15.167:0/4016842607 <== osd.0 v2:172.21.15.118:6800/27690 30 ==== osd_op_reply(103 p5YnriCv1nAtykxBrupQ [getxattrs,stat] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 ==== 206+0+0 (crc 0 0 0) 0x7f8c00050040 con 0x7f8c0400e8c0 2023-10-05T19:29:40.253 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.246+0000 7f8c12b0ee40 10 cache put: name=default.rgw.meta+users.keys+p5YnriCv1nAtykxBrupQ info.flags=0x0 2023-10-05T19:29:40.253 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.246+0000 7f8c12b0ee40 10 adding default.rgw.meta+users.keys+p5YnriCv1nAtykxBrupQ to cache LRU end 2023-10-05T19:29:40.253 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.246+0000 7f8c12b0ee40 20 get_system_obj_state: rctx=0x7ffd9da66538 obj=default.rgw.meta:users.keys:p5YnriCv1nAtykxBrupQ state=0x55f80c4630b0 s->prefetch_data=0 2023-10-05T19:29:40.254 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.246+0000 7f8c12b0ee40 10 cache get: name=default.rgw.meta+users.keys+p5YnriCv1nAtykxBrupQ : hit (negative entry) 2023-10-05T19:29:40.254 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.246+0000 7f8c12b0ee40 20 get_system_obj_state: rctx=0x7ffd9da66bd8 obj=default.rgw.meta:users.keys:p5YnriCv1nAtykxBrupQ state=0x55f80c4630b0 s->prefetch_data=0 2023-10-05T19:29:40.254 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.246+0000 7f8c12b0ee40 10 cache get: name=default.rgw.meta+users.keys+p5YnriCv1nAtykxBrupQ : hit (negative entry) 2023-10-05T19:29:40.254 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.246+0000 7f8c12b0ee40 1 -- 172.21.15.167:0/4016842607 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:104 8.e 8:75f48285:users.uid::fud:head [delete,create,call version.set in=58b,writefull 0~346 in=346b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e32) v8 -- 0x55f80c64b100 con 0x7f8c0400e8c0 2023-10-05T19:29:40.256 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.250+0000 7f8c10e44700 1 -- 172.21.15.167:0/4016842607 <== osd.0 v2:172.21.15.118:6800/27690 31 ==== osd_op_reply(104 fud [delete,create,call,writefull 0~346] v32'5 uv5 ondisk = 0) v8 ==== 273+0+0 (crc 0 0 0) 0x7f8c00050040 con 0x7f8c0400e8c0 2023-10-05T19:29:40.256 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.250+0000 7f8c12b0ee40 10 cache put: name=default.rgw.meta+users.uid+fud info.flags=0x17 2023-10-05T19:29:40.256 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.250+0000 7f8c12b0ee40 10 moving default.rgw.meta+users.uid+fud to cache LRU end 2023-10-05T19:29:40.256 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.250+0000 7f8c12b0ee40 10 distributing notification oid=default.rgw.control:notify.5 cni=[op: 0, obj: default.rgw.meta:users.uid:fud, ofs0, ns] 2023-10-05T19:29:40.256 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.250+0000 7f8c12b0ee40 1 -- 172.21.15.167:0/4016842607 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:105 7.3 7:c609908c:::notify.5:head [notify cookie 94523848171760 in=534b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32) v8 -- 0x55f80c64aab0 con 0x7f8c0400e8c0 2023-10-05T19:29:40.257 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.254+0000 7f8c10e44700 1 -- 172.21.15.167:0/4016842607 <== osd.0 v2:172.21.15.118:6800/27690 32 ==== watch-notify(notify (1) cookie 94523847642096 notify 137438953509 ret 0) v3 ==== 564+0+0 (crc 0 0 0) 0x7f8c0003d040 con 0x7f8c0400e8c0 2023-10-05T19:29:40.258 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.254+0000 7f8c0e63f700 10 rgw watcher librados: RGWWatcher::handle_notify() notify_id 137438953509 cookie 94523847642096 notifier 4719 bl.length()=522 2023-10-05T19:29:40.258 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.254+0000 7f8c0e63f700 10 rgw watcher librados: cache put: name=default.rgw.meta+users.uid+fud info.flags=0x17 2023-10-05T19:29:40.258 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.254+0000 7f8c0e63f700 10 rgw watcher librados: moving default.rgw.meta+users.uid+fud to cache LRU end 2023-10-05T19:29:40.258 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.254+0000 7f8c10e44700 1 -- 172.21.15.167:0/4016842607 <== osd.0 v2:172.21.15.118:6800/27690 33 ==== osd_op_reply(105 notify.5 [notify cookie 94523848171760 out=8b] v0'0 uv246 ondisk = 0) v8 ==== 152+0+8 (crc 0 0 0) 0x7f8c00050040 con 0x7f8c0400e8c0 2023-10-05T19:29:40.258 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.254+0000 7f8c0e63f700 1 -- 172.21.15.167:0/4016842607 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:106 7.3 7:c609908c:::notify.5:head [notify-ack in=20b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32) v8 -- 0x7f8bf0003d90 con 0x7f8c0400e8c0 2023-10-05T19:29:40.258 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.254+0000 7f8c10e44700 1 -- 172.21.15.167:0/4016842607 <== osd.0 v2:172.21.15.118:6800/27690 34 ==== osd_op_reply(106 notify.5 [notify-ack] v0'0 uv246 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f8c00050040 con 0x7f8c0400e8c0 2023-10-05T19:29:40.258 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.254+0000 7f8c10e44700 1 -- 172.21.15.167:0/4016842607 <== osd.0 v2:172.21.15.118:6800/27690 35 ==== watch-notify(notify_complete (2) cookie 94523848171760 notify 137438953509 ret 0) v3 ==== 42+0+48 (crc 0 0 0) 0x7f8c000334d0 con 0x7f8c0400e8c0 2023-10-05T19:29:40.258 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.254+0000 7f8c12b0ee40 1 -- 172.21.15.167:0/4016842607 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:107 8.13 8:cc695d75:users.keys::p5YnriCv1nAtykxBrupQ:head [delete,create,writefull 0~7 in=7b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e32) v8 -- 0x55f80c6490e0 con 0x7f8c0400e8c0 2023-10-05T19:29:40.261 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.258+0000 7f8c10e44700 1 -- 172.21.15.167:0/4016842607 <== osd.0 v2:172.21.15.118:6800/27690 36 ==== osd_op_reply(107 p5YnriCv1nAtykxBrupQ [delete,create,writefull 0~7] v32'12 uv12 ondisk = 0) v8 ==== 248+0+0 (crc 0 0 0) 0x7f8c00050040 con 0x7f8c0400e8c0 2023-10-05T19:29:40.261 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.258+0000 7f8c12b0ee40 10 cache put: name=default.rgw.meta+users.keys+p5YnriCv1nAtykxBrupQ info.flags=0x7 2023-10-05T19:29:40.261 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.258+0000 7f8c12b0ee40 10 moving default.rgw.meta+users.keys+p5YnriCv1nAtykxBrupQ to cache LRU end 2023-10-05T19:29:40.261 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.258+0000 7f8c12b0ee40 10 distributing notification oid=default.rgw.control:notify.2 cni=[op: 0, obj: default.rgw.meta:users.keys:p5YnriCv1nAtykxBrupQ, ofs0, ns] 2023-10-05T19:29:40.261 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.258+0000 7f8c12b0ee40 1 -- 172.21.15.167:0/4016842607 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:108 7.15 7:a93a5511:::notify.2:head [notify cookie 94523848175344 in=189b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32) v8 -- 0x55f80c6490e0 con 0x55f80c5c1c70 2023-10-05T19:29:40.262 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.258+0000 7f8c10643700 1 -- 172.21.15.167:0/4016842607 <== osd.3 v2:172.21.15.167:6808/30036 24 ==== watch-notify(notify (1) cookie 94523847639888 notify 137438953489 ret 0) v3 ==== 219+0+0 (crc 0 0 0) 0x55f80c5b2980 con 0x55f80c5c1c70 2023-10-05T19:29:40.262 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.258+0000 7f8c0e63f700 10 rgw watcher librados: RGWWatcher::handle_notify() notify_id 137438953489 cookie 94523847639888 notifier 4719 bl.length()=177 2023-10-05T19:29:40.263 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.258+0000 7f8c0e63f700 10 rgw watcher librados: cache put: name=default.rgw.meta+users.keys+p5YnriCv1nAtykxBrupQ info.flags=0x7 2023-10-05T19:29:40.263 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.258+0000 7f8c0e63f700 10 rgw watcher librados: moving default.rgw.meta+users.keys+p5YnriCv1nAtykxBrupQ to cache LRU end 2023-10-05T19:29:40.263 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.258+0000 7f8c0e63f700 1 -- 172.21.15.167:0/4016842607 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:109 7.15 7:a93a5511:::notify.2:head [notify-ack in=20b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32) v8 -- 0x7f8bf0003d90 con 0x55f80c5c1c70 2023-10-05T19:29:40.263 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.258+0000 7f8c10643700 1 -- 172.21.15.167:0/4016842607 <== osd.3 v2:172.21.15.167:6808/30036 25 ==== osd_op_reply(108 notify.2 [notify cookie 94523848175344 out=8b] v0'0 uv246 ondisk = 0) v8 ==== 152+0+8 (crc 0 0 0) 0x7f8c0802b070 con 0x55f80c5c1c70 2023-10-05T19:29:40.263 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.258+0000 7f8c10643700 1 -- 172.21.15.167:0/4016842607 <== osd.3 v2:172.21.15.167:6808/30036 26 ==== osd_op_reply(109 notify.2 [notify-ack] v0'0 uv246 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f8c0802b070 con 0x55f80c5c1c70 2023-10-05T19:29:40.263 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.258+0000 7f8c10643700 1 -- 172.21.15.167:0/4016842607 <== osd.3 v2:172.21.15.167:6808/30036 27 ==== watch-notify(notify_complete (2) cookie 94523848175344 notify 137438953489 ret 0) v3 ==== 42+0+48 (crc 0 0 0) 0x55f80c5b2540 con 0x55f80c5c1c70 2023-10-05T19:29:40.263 INFO:teuthology.orchestra.run.smithi167.stdout:{ 2023-10-05T19:29:40.264 INFO:teuthology.orchestra.run.smithi167.stdout: "user_id": "fud", 2023-10-05T19:29:40.264 INFO:teuthology.orchestra.run.smithi167.stdout: "display_name": "Fud", 2023-10-05T19:29:40.264 INFO:teuthology.orchestra.run.smithi167.stdout: "email": "", 2023-10-05T19:29:40.264 INFO:teuthology.orchestra.run.smithi167.stdout: "suspended": 0, 2023-10-05T19:29:40.264 INFO:teuthology.orchestra.run.smithi167.stdout: "max_buckets": 4, 2023-10-05T19:29:40.264 INFO:teuthology.orchestra.run.smithi167.stdout: "subusers": [], 2023-10-05T19:29:40.264 INFO:teuthology.orchestra.run.smithi167.stdout: "keys": [ 2023-10-05T19:29:40.264 INFO:teuthology.orchestra.run.smithi167.stdout: { 2023-10-05T19:29:40.264 INFO:teuthology.orchestra.run.smithi167.stdout: "user": "fud", 2023-10-05T19:29:40.265 INFO:teuthology.orchestra.run.smithi167.stdout: "access_key": "p5YnriCv1nAtykxBrupQ", 2023-10-05T19:29:40.265 INFO:teuthology.orchestra.run.smithi167.stdout: "secret_key": "Q8Tk6Q/27hfbFSYdSkPtUqhqx1GgzvpXa4WARozh" 2023-10-05T19:29:40.265 INFO:teuthology.orchestra.run.smithi167.stdout: } 2023-10-05T19:29:40.265 INFO:teuthology.orchestra.run.smithi167.stdout: ], 2023-10-05T19:29:40.265 INFO:teuthology.orchestra.run.smithi167.stdout: "swift_keys": [], 2023-10-05T19:29:40.265 INFO:teuthology.orchestra.run.smithi167.stdout: "caps": [], 2023-10-05T19:29:40.265 INFO:teuthology.orchestra.run.smithi167.stdout: "op_mask": "read, write, delete", 2023-10-05T19:29:40.265 INFO:teuthology.orchestra.run.smithi167.stdout: "default_placement": "", 2023-10-05T19:29:40.265 INFO:teuthology.orchestra.run.smithi167.stdout: "default_storage_class": "", 2023-10-05T19:29:40.265 INFO:teuthology.orchestra.run.smithi167.stdout: "placement_tags": [], 2023-10-05T19:29:40.266 INFO:teuthology.orchestra.run.smithi167.stdout: "bucket_quota": { 2023-10-05T19:29:40.266 INFO:teuthology.orchestra.run.smithi167.stdout: "enabled": false, 2023-10-05T19:29:40.266 INFO:teuthology.orchestra.run.smithi167.stdout: "check_on_raw": false, 2023-10-05T19:29:40.266 INFO:teuthology.orchestra.run.smithi167.stdout: "max_size": -1, 2023-10-05T19:29:40.266 INFO:teuthology.orchestra.run.smithi167.stdout: "max_size_kb": 0, 2023-10-05T19:29:40.266 INFO:teuthology.orchestra.run.smithi167.stdout: "max_objects": -1 2023-10-05T19:29:40.266 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2023-10-05T19:29:40.266 INFO:teuthology.orchestra.run.smithi167.stdout: "user_quota": { 2023-10-05T19:29:40.266 INFO:teuthology.orchestra.run.smithi167.stdout: "enabled": false, 2023-10-05T19:29:40.266 INFO:teuthology.orchestra.run.smithi167.stdout: "check_on_raw": false, 2023-10-05T19:29:40.267 INFO:teuthology.orchestra.run.smithi167.stdout: "max_size": -1, 2023-10-05T19:29:40.267 INFO:teuthology.orchestra.run.smithi167.stdout: "max_size_kb": 0, 2023-10-05T19:29:40.267 INFO:teuthology.orchestra.run.smithi167.stdout: "max_objects": -1 2023-10-05T19:29:40.267 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2023-10-05T19:29:40.267 INFO:teuthology.orchestra.run.smithi167.stdout: "temp_url_keys": [], 2023-10-05T19:29:40.267 INFO:teuthology.orchestra.run.smithi167.stdout: "type": "rgw", 2023-10-05T19:29:40.267 INFO:teuthology.orchestra.run.smithi167.stdout: "mfa_ids": [] 2023-10-05T19:29:40.267 INFO:teuthology.orchestra.run.smithi167.stdout:} 2023-10-05T19:29:40.267 INFO:teuthology.orchestra.run.smithi167.stdout: 2023-10-05T19:29:40.268 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.258+0000 7f8c12b0ee40 1 -- 172.21.15.167:0/4016842607 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:110 7.1f 7:f95f44c2:::notify.0:head [watch unwatch cookie 94523847628080] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e32) v8 -- 0x55f80c5a1b00 con 0x7f8c08044c30 2023-10-05T19:29:40.268 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.262+0000 7f8c0fe42700 1 -- 172.21.15.167:0/4016842607 <== osd.2 v2:172.21.15.167:6816/30037 17 ==== osd_op_reply(110 notify.0 [watch unwatch cookie 94523847628080] v32'248 uv246 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f8c0400f560 con 0x7f8c08044c30 2023-10-05T19:29:40.268 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.262+0000 7f8c12b0ee40 20 remove_watcher() i=0 2023-10-05T19:29:40.268 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.262+0000 7f8c12b0ee40 2 removed watcher, disabling cache 2023-10-05T19:29:40.268 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.262+0000 7f8c12b0ee40 1 -- 172.21.15.167:0/4016842607 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:111 7.0 7:05bf5b68:::notify.1:head [watch unwatch cookie 94523847638784] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e32) v8 -- 0x55f80c5a1b00 con 0x55f80c5bba60 2023-10-05T19:29:40.270 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.266+0000 7f8c10e44700 1 -- 172.21.15.167:0/4016842607 <== osd.1 v2:172.21.15.167:6800/30035 35 ==== osd_op_reply(111 notify.1 [watch unwatch cookie 94523847638784] v32'248 uv246 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f8c00050040 con 0x55f80c5bba60 2023-10-05T19:29:40.270 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.266+0000 7f8c12b0ee40 20 remove_watcher() i=1 2023-10-05T19:29:40.270 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.266+0000 7f8c12b0ee40 1 -- 172.21.15.167:0/4016842607 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:112 7.15 7:a93a5511:::notify.2:head [watch unwatch cookie 94523847639888] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e32) v8 -- 0x55f80c5a1b00 con 0x55f80c5c1c70 2023-10-05T19:29:40.273 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.270+0000 7f8c10643700 1 -- 172.21.15.167:0/4016842607 <== osd.3 v2:172.21.15.167:6808/30036 28 ==== osd_op_reply(112 notify.2 [watch unwatch cookie 94523847639888] v32'248 uv246 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f8c0802b070 con 0x55f80c5c1c70 2023-10-05T19:29:40.273 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.270+0000 7f8c12b0ee40 20 remove_watcher() i=2 2023-10-05T19:29:40.273 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.270+0000 7f8c12b0ee40 1 -- 172.21.15.167:0/4016842607 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:113 7.e 7:7759931f:::notify.3:head [watch unwatch cookie 94523847647168] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e32) v8 -- 0x55f80c5c83f0 con 0x7f8c08044c30 2023-10-05T19:29:40.276 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.270+0000 7f8c0fe42700 1 -- 172.21.15.167:0/4016842607 <== osd.2 v2:172.21.15.167:6816/30037 18 ==== osd_op_reply(113 notify.3 [watch unwatch cookie 94523847647168] v32'248 uv246 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f8c0400f560 con 0x7f8c08044c30 2023-10-05T19:29:40.276 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.270+0000 7f8c12b0ee40 20 remove_watcher() i=3 2023-10-05T19:29:40.276 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.270+0000 7f8c12b0ee40 1 -- 172.21.15.167:0/4016842607 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:114 7.d 7:b4812045:::notify.4:head [watch unwatch cookie 94523847648272] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e32) v8 -- 0x55f80c5c83f0 con 0x55f80c5bba60 2023-10-05T19:29:40.278 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.274+0000 7f8c10e44700 1 -- 172.21.15.167:0/4016842607 <== osd.1 v2:172.21.15.167:6800/30035 36 ==== osd_op_reply(114 notify.4 [watch unwatch cookie 94523847648272] v32'248 uv246 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f8c00050040 con 0x55f80c5bba60 2023-10-05T19:29:40.279 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.274+0000 7f8c12b0ee40 20 remove_watcher() i=4 2023-10-05T19:29:40.279 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.274+0000 7f8c12b0ee40 1 -- 172.21.15.167:0/4016842607 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:115 7.3 7:c609908c:::notify.5:head [watch unwatch cookie 94523847642096] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e32) v8 -- 0x55f80c5c83f0 con 0x7f8c0400e8c0 2023-10-05T19:29:40.282 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.278+0000 7f8c10e44700 1 -- 172.21.15.167:0/4016842607 <== osd.0 v2:172.21.15.118:6800/27690 37 ==== osd_op_reply(115 notify.5 [watch unwatch cookie 94523847642096] v32'248 uv246 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f8c00050040 con 0x7f8c0400e8c0 2023-10-05T19:29:40.282 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.278+0000 7f8c12b0ee40 20 remove_watcher() i=5 2023-10-05T19:29:40.282 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.278+0000 7f8c12b0ee40 1 -- 172.21.15.167:0/4016842607 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:116 7.14 7:2b04a3e9:::notify.6:head [watch unwatch cookie 94523847653040] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e32) v8 -- 0x55f80c5c83f0 con 0x7f8c0400e8c0 2023-10-05T19:29:40.284 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.282+0000 7f8c10e44700 1 -- 172.21.15.167:0/4016842607 <== osd.0 v2:172.21.15.118:6800/27690 38 ==== osd_op_reply(116 notify.6 [watch unwatch cookie 94523847653040] v32'248 uv246 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f8c00050040 con 0x7f8c0400e8c0 2023-10-05T19:29:40.285 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.282+0000 7f8c12b0ee40 20 remove_watcher() i=6 2023-10-05T19:29:40.285 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.282+0000 7f8c12b0ee40 1 -- 172.21.15.167:0/4016842607 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:117 7.9 7:93e5b521:::notify.7:head [watch unwatch cookie 94523847657904] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e32) v8 -- 0x55f80c5a1b00 con 0x7f8c0400e8c0 2023-10-05T19:29:40.287 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.282+0000 7f8c10e44700 1 -- 172.21.15.167:0/4016842607 <== osd.0 v2:172.21.15.118:6800/27690 39 ==== osd_op_reply(117 notify.7 [watch unwatch cookie 94523847657904] v32'248 uv246 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f8c00050040 con 0x7f8c0400e8c0 2023-10-05T19:29:40.287 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.282+0000 7f8c12b0ee40 20 remove_watcher() i=7 2023-10-05T19:29:40.289 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.286+0000 7f8be1fe3700 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2023-10-05T19:29:40.289 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.286+0000 7f8c12b0ee40 1 -- 172.21.15.167:0/4016842607 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:118 6.4 6:22d26bf9:::data_loggenerations_metadata:head [watch unwatch cookie 94523847659008] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e32) v8 -- 0x55f80c5a1b00 con 0x55f80c5bba60 2023-10-05T19:29:40.292 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.286+0000 7f8c10e44700 1 -- 172.21.15.167:0/4016842607 <== osd.1 v2:172.21.15.167:6800/30035 37 ==== osd_op_reply(118 data_loggenerations_metadata [watch unwatch cookie 94523847659008] v32'238 uv1 ondisk = 0) v8 ==== 172+0+0 (crc 0 0 0) 0x7f8c00050040 con 0x55f80c5bba60 2023-10-05T19:29:40.292 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.286+0000 7f8c12b0ee40 1 -- 172.21.15.167:0/4016842607 >> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] conn(0x7f8c0400e8c0 msgr2=0x7f8c0400d310 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-05T19:29:40.293 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.286+0000 7f8c12b0ee40 1 --2- 172.21.15.167:0/4016842607 >> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] conn(0x7f8c0400e8c0 0x7f8c0400d310 crc :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-05T19:29:40.293 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.286+0000 7f8c12b0ee40 1 -- 172.21.15.167:0/4016842607 >> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] conn(0x55f80c5bba60 msgr2=0x55f80c5bde40 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-05T19:29:40.293 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.286+0000 7f8c12b0ee40 1 --2- 172.21.15.167:0/4016842607 >> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] conn(0x55f80c5bba60 0x55f80c5bde40 crc :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-05T19:29:40.293 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.286+0000 7f8c12b0ee40 1 -- 172.21.15.167:0/4016842607 >> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] conn(0x7f8c08044c30 msgr2=0x7f8c08010850 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-05T19:29:40.293 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.286+0000 7f8c12b0ee40 1 --2- 172.21.15.167:0/4016842607 >> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] conn(0x7f8c08044c30 0x7f8c08010850 crc :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-05T19:29:40.293 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.286+0000 7f8c12b0ee40 1 -- 172.21.15.167:0/4016842607 >> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] conn(0x55f80c5c1c70 msgr2=0x55f80c5c4050 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-05T19:29:40.293 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.286+0000 7f8c12b0ee40 1 --2- 172.21.15.167:0/4016842607 >> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] conn(0x55f80c5c1c70 0x55f80c5c4050 crc :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-05T19:29:40.293 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.286+0000 7f8c12b0ee40 1 -- 172.21.15.167:0/4016842607 >> [v2:172.21.15.167:6824/29889,v1:172.21.15.167:6825/29889] conn(0x7f8bfc030c00 msgr2=0x7f8bfc0330b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-05T19:29:40.294 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.286+0000 7f8c12b0ee40 1 --2- 172.21.15.167:0/4016842607 >> [v2:172.21.15.167:6824/29889,v1:172.21.15.167:6825/29889] conn(0x7f8bfc030c00 0x7f8bfc0330b0 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7f8c0400b300 tx=0x7f8c04001470 comp rx=0 tx=0).stop 2023-10-05T19:29:40.294 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.286+0000 7f8c10e44700 1 -- 172.21.15.167:0/4016842607 reap_dead start 2023-10-05T19:29:40.294 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.286+0000 7f8c12b0ee40 1 -- 172.21.15.167:0/4016842607 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55f80c58a820 msgr2=0x55f80c5b2000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-05T19:29:40.294 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.286+0000 7f8c12b0ee40 1 --2- 172.21.15.167:0/4016842607 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55f80c58a820 0x55f80c5b2000 secure :-1 s=READY pgs=629 cs=0 l=1 rev1=1 crypto rx=0x7f8c0801e040 tx=0x7f8c0801c000 comp rx=0 tx=0).stop 2023-10-05T19:29:40.294 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.286+0000 7f8c12b0ee40 1 -- 172.21.15.167:0/4016842607 shutdown_connections 2023-10-05T19:29:40.294 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.286+0000 7f8c12b0ee40 1 --2- 172.21.15.167:0/4016842607 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55f80c58a820 0x55f80c5b2000 unknown :-1 s=CLOSED pgs=629 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-05T19:29:40.294 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.286+0000 7f8c12b0ee40 1 -- 172.21.15.167:0/4016842607 >> 172.21.15.167:0/4016842607 conn(0x55f80c584760 msgr2=0x55f80c5a58a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-05T19:29:40.294 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.286+0000 7f8c12b0ee40 1 -- 172.21.15.167:0/4016842607 shutdown_connections 2023-10-05T19:29:40.294 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.286+0000 7f8c12b0ee40 1 -- 172.21.15.167:0/4016842607 wait complete. 2023-10-05T19:29:40.295 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.290+0000 7f8c12b0ee40 1 -- 172.21.15.167:0/919496971 >> [v2:172.21.15.167:6824/29889,v1:172.21.15.167:6825/29889] conn(0x7f8b50030c00 msgr2=0x7f8b500330b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-05T19:29:40.296 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.290+0000 7f8c12b0ee40 1 --2- 172.21.15.167:0/919496971 >> [v2:172.21.15.167:6824/29889,v1:172.21.15.167:6825/29889] conn(0x7f8b50030c00 0x7f8b500330b0 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x55f80c5cc280 tx=0x7f8c00000f90 comp rx=0 tx=0).stop 2023-10-05T19:29:40.296 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.290+0000 7f8c12b0ee40 1 -- 172.21.15.167:0/919496971 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55f80c5da860 msgr2=0x55f80c5e03b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-05T19:29:40.296 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.290+0000 7f8c12b0ee40 1 --2- 172.21.15.167:0/919496971 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55f80c5da860 0x55f80c5e03b0 secure :-1 s=READY pgs=631 cs=0 l=1 rev1=1 crypto rx=0x7f8c0400efd0 tx=0x7f8c04032000 comp rx=0 tx=0).stop 2023-10-05T19:29:40.296 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.290+0000 7f8c12b0ee40 1 -- 172.21.15.167:0/919496971 shutdown_connections 2023-10-05T19:29:40.296 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.290+0000 7f8c12b0ee40 1 --2- 172.21.15.167:0/919496971 >> [v2:172.21.15.167:6824/29889,v1:172.21.15.167:6825/29889] conn(0x7f8b50030c00 0x7f8b500330b0 unknown :-1 s=CLOSED pgs=217 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-05T19:29:40.296 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.290+0000 7f8c12b0ee40 1 --2- 172.21.15.167:0/919496971 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55f80c5da860 0x55f80c5e03b0 unknown :-1 s=CLOSED pgs=631 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-05T19:29:40.296 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.290+0000 7f8c12b0ee40 1 -- 172.21.15.167:0/919496971 >> 172.21.15.167:0/919496971 conn(0x55f80c5d5c20 msgr2=0x55f80c5d3cc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-05T19:29:40.296 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.290+0000 7f8c12b0ee40 1 -- 172.21.15.167:0/919496971 shutdown_connections 2023-10-05T19:29:40.296 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.290+0000 7f8c12b0ee40 1 -- 172.21.15.167:0/919496971 wait complete. 2023-10-05T19:29:40.299 DEBUG:tasks.util.rgw: json result: {'user_id': 'fud', 'display_name': 'Fud', 'email': '', 'suspended': 0, 'max_buckets': 4, 'subusers': [], 'keys': [{'user': 'fud', 'access_key': 'p5YnriCv1nAtykxBrupQ', 'secret_key': 'Q8Tk6Q/27hfbFSYdSkPtUqhqx1GgzvpXa4WARozh'}], 'swift_keys': [], 'caps': [], 'op_mask': 'read, write, delete', 'default_placement': '', 'default_storage_class': '', 'placement_tags': [], 'bucket_quota': {'enabled': False, 'check_on_raw': False, 'max_size': -1, 'max_size_kb': 0, 'max_objects': -1}, 'user_quota': {'enabled': False, 'check_on_raw': False, 'max_size': -1, 'max_size_kb': 0, 'max_objects': -1}, 'temp_url_keys': [], 'type': 'rgw', 'mfa_ids': []} 2023-10-05T19:29:40.323 INFO:tasks.radosgw_admin:RL: create_bucket myfoo7 fud bytes_out=0 bytes_in=0 failed=False 2023-10-05T19:29:40.334 INFO:tasks.radosgw_admin:RL: put_obj myfoo7 fud bytes_out=8 bytes_in=0 failed=False 2023-10-05T19:29:40.334 INFO:tasks.util.rgw:rgwadmin: client.0 : ['user', 'rename', '--uid', 'bud', '--new-uid', 'fud'] 2023-10-05T19:29:40.335 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'ceph', 'user', 'rename', '--uid', 'bud', '--new-uid', 'fud'] 2023-10-05T19:29:40.335 DEBUG:teuthology.orchestra.run.smithi167:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster ceph user rename --uid bud --new-uid fud 2023-10-05T19:29:40.380 INFO:teuthology.orchestra.run.smithi167.stderr:ignoring --setuser ceph since I am not root 2023-10-05T19:29:40.380 INFO:teuthology.orchestra.run.smithi167.stderr:ignoring --setgroup ceph since I am not root 2023-10-05T19:29:40.381 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.378+0000 7f41893f1e40 1 Processor -- start 2023-10-05T19:29:40.381 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.378+0000 7f41893f1e40 1 -- start start 2023-10-05T19:29:40.381 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.378+0000 7f41893f1e40 1 --2- >> v2:172.21.15.118:3300/0 conn(0x560d8c398020 0x560d8c3983f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-05T19:29:40.381 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.378+0000 7f41893f1e40 1 -- --> v1:172.21.15.118:6789/0 -- auth(proto 0 26 bytes epoch 0) v1 -- 0x560d8c2e2bd0 con 0x560d8c398930 2023-10-05T19:29:40.381 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.378+0000 7f41893f1e40 1 -- --> v2:172.21.15.118:3300/0 -- mon_getmap magic: 0 v1 -- 0x560d8c29fee0 con 0x560d8c398020 2023-10-05T19:29:40.382 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.378+0000 7f4186f26700 1 --2- >> v2:172.21.15.118:3300/0 conn(0x560d8c398020 0x560d8c3983f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-05T19:29:40.382 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.378+0000 7f4186725700 1 --1- >> v1:172.21.15.118:6789/0 conn(0x560d8c398930 0x560d8c3993d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.118:6789/0 says I am v1:172.21.15.167:33340/0 (socket says 172.21.15.167:33340) 2023-10-05T19:29:40.382 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.378+0000 7f4186725700 1 -- 172.21.15.167:0/205418733 learned_addr learned my addr 172.21.15.167:0/205418733 (peer_addr_for_me v1:172.21.15.167:0/0) 2023-10-05T19:29:40.382 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.378+0000 7f4186f26700 1 -- 172.21.15.167:0/205418733 >> v1:172.21.15.118:6789/0 conn(0x560d8c398930 legacy=0x560d8c3993d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-05T19:29:40.383 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.378+0000 7f4186f26700 1 -- 172.21.15.167:0/205418733 --> v2:172.21.15.118:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x560d8c399ae0 con 0x560d8c398020 2023-10-05T19:29:40.383 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.378+0000 7f4186f26700 1 --2- 172.21.15.167:0/205418733 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x560d8c398020 0x560d8c3983f0 secure :-1 s=READY pgs=633 cs=0 l=1 rev1=1 crypto rx=0x7f418000a520 tx=0x7f4180005d40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ef1794032d921940 server_cookie=0 in_seq=0 out_seq=0 2023-10-05T19:29:40.383 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.378+0000 7f4185f24700 1 -- 172.21.15.167:0/205418733 <== mon.0 v2:172.21.15.118:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f418000f590 con 0x560d8c398020 2023-10-05T19:29:40.383 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.378+0000 7f4185f24700 1 -- 172.21.15.167:0/205418733 <== mon.0 v2:172.21.15.118:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f418000f6f0 con 0x560d8c398020 2023-10-05T19:29:40.383 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.378+0000 7f4185f24700 1 -- 172.21.15.167:0/205418733 <== mon.0 v2:172.21.15.118:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f418000f880 con 0x560d8c398020 2023-10-05T19:29:40.384 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.378+0000 7f41893f1e40 1 -- 172.21.15.167:0/205418733 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x560d8c398020 msgr2=0x560d8c3983f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-05T19:29:40.385 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.378+0000 7f41893f1e40 1 --2- 172.21.15.167:0/205418733 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x560d8c398020 0x560d8c3983f0 secure :-1 s=READY pgs=633 cs=0 l=1 rev1=1 crypto rx=0x7f418000a520 tx=0x7f4180005d40 comp rx=0 tx=0).stop 2023-10-05T19:29:40.385 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.378+0000 7f41893f1e40 1 -- 172.21.15.167:0/205418733 shutdown_connections 2023-10-05T19:29:40.385 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.378+0000 7f41893f1e40 1 --2- 172.21.15.167:0/205418733 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x560d8c398020 0x560d8c3983f0 unknown :-1 s=CLOSED pgs=633 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-05T19:29:40.385 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.378+0000 7f41893f1e40 1 -- 172.21.15.167:0/205418733 >> 172.21.15.167:0/205418733 conn(0x560d8c30b870 msgr2=0x560d8c30bc70 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-05T19:29:40.385 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.378+0000 7f41893f1e40 1 -- 172.21.15.167:0/205418733 shutdown_connections 2023-10-05T19:29:40.385 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.378+0000 7f41893f1e40 1 -- 172.21.15.167:0/205418733 wait complete. 2023-10-05T19:29:40.386 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.382+0000 7f41893f1e40 1 Processor -- start 2023-10-05T19:29:40.387 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.382+0000 7f41893f1e40 1 -- start start 2023-10-05T19:29:40.387 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.382+0000 7f41893f1e40 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x560d8c398020 0x560d8c30ce50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-05T19:29:40.387 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.382+0000 7f41893f1e40 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x560d8c263e30 con 0x560d8c398020 2023-10-05T19:29:40.387 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.382+0000 7f4186f26700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x560d8c398020 0x560d8c30ce50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-05T19:29:40.387 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.382+0000 7f4186f26700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x560d8c398020 0x560d8c30ce50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.118:3300/0 says I am v2:172.21.15.167:56824/0 (socket says 172.21.15.167:56824) 2023-10-05T19:29:40.387 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.382+0000 7f4186f26700 1 -- 172.21.15.167:0/3924705742 learned_addr learned my addr 172.21.15.167:0/3924705742 (peer_addr_for_me v2:172.21.15.167:0/0) 2023-10-05T19:29:40.387 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.382+0000 7f4186f26700 1 -- 172.21.15.167:0/3924705742 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f418000a0e0 con 0x560d8c398020 2023-10-05T19:29:40.389 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.382+0000 7f4186f26700 1 --2- 172.21.15.167:0/3924705742 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x560d8c398020 0x560d8c30ce50 secure :-1 s=READY pgs=634 cs=0 l=1 rev1=1 crypto rx=0x7f418000a470 tx=0x7f4180014000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b907d9c580b7ce93 server_cookie=0 in_seq=0 out_seq=0 2023-10-05T19:29:40.389 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.382+0000 7f4184f22700 1 -- 172.21.15.167:0/3924705742 <== mon.0 v2:172.21.15.118:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f418000f860 con 0x560d8c398020 2023-10-05T19:29:40.389 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.382+0000 7f4184f22700 1 -- 172.21.15.167:0/3924705742 <== mon.0 v2:172.21.15.118:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f418000f9e0 con 0x560d8c398020 2023-10-05T19:29:40.389 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.382+0000 7f4184f22700 1 -- 172.21.15.167:0/3924705742 <== mon.0 v2:172.21.15.118:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f418000fb90 con 0x560d8c398020 2023-10-05T19:29:40.389 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.382+0000 7f41893f1e40 1 -- 172.21.15.167:0/3924705742 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x560d8c398020 msgr2=0x560d8c30ce50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-05T19:29:40.389 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.382+0000 7f41893f1e40 1 --2- 172.21.15.167:0/3924705742 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x560d8c398020 0x560d8c30ce50 secure :-1 s=READY pgs=634 cs=0 l=1 rev1=1 crypto rx=0x7f418000a470 tx=0x7f4180014000 comp rx=0 tx=0).stop 2023-10-05T19:29:40.390 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.382+0000 7f41893f1e40 1 -- 172.21.15.167:0/3924705742 shutdown_connections 2023-10-05T19:29:40.390 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.382+0000 7f41893f1e40 1 --2- 172.21.15.167:0/3924705742 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x560d8c398020 0x560d8c30ce50 unknown :-1 s=CLOSED pgs=634 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-05T19:29:40.390 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.382+0000 7f41893f1e40 1 -- 172.21.15.167:0/3924705742 >> 172.21.15.167:0/3924705742 conn(0x560d8c30b870 msgr2=0x560d8c3b2c80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-05T19:29:40.390 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.382+0000 7f41893f1e40 1 -- 172.21.15.167:0/3924705742 shutdown_connections 2023-10-05T19:29:40.390 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.386+0000 7f41893f1e40 1 -- 172.21.15.167:0/3924705742 wait complete. 2023-10-05T19:29:40.390 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.386+0000 7f41893f1e40 1 Processor -- start 2023-10-05T19:29:40.390 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.386+0000 7f41893f1e40 1 -- start start 2023-10-05T19:29:40.392 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.386+0000 7f41893f1e40 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x560d8c398020 0x560d8c3c20b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-05T19:29:40.392 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.386+0000 7f41893f1e40 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x560d8c236ed0 con 0x560d8c398020 2023-10-05T19:29:40.392 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.386+0000 7f4186f26700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x560d8c398020 0x560d8c3c20b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-05T19:29:40.392 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.386+0000 7f4186f26700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x560d8c398020 0x560d8c3c20b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.118:3300/0 says I am v2:172.21.15.167:56830/0 (socket says 172.21.15.167:56830) 2023-10-05T19:29:40.392 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.386+0000 7f4186f26700 1 -- 172.21.15.167:0/1416833216 learned_addr learned my addr 172.21.15.167:0/1416833216 (peer_addr_for_me v2:172.21.15.167:0/0) 2023-10-05T19:29:40.392 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.386+0000 7f4186f26700 1 -- 172.21.15.167:0/1416833216 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x560d8c399ae0 con 0x560d8c398020 2023-10-05T19:29:40.392 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.386+0000 7f4186f26700 1 --2- 172.21.15.167:0/1416833216 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x560d8c398020 0x560d8c3c20b0 secure :-1 s=READY pgs=635 cs=0 l=1 rev1=1 crypto rx=0x7f418000a470 tx=0x7f418001d000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-05T19:29:40.392 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.386+0000 7f4176ffd700 1 -- 172.21.15.167:0/1416833216 <== mon.0 v2:172.21.15.118:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f418001ce00 con 0x560d8c398020 2023-10-05T19:29:40.393 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.386+0000 7f41893f1e40 1 -- 172.21.15.167:0/1416833216 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x560d8c3c25f0 con 0x560d8c398020 2023-10-05T19:29:40.393 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.386+0000 7f4176ffd700 1 -- 172.21.15.167:0/1416833216 <== mon.0 v2:172.21.15.118:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4180010630 con 0x560d8c398020 2023-10-05T19:29:40.393 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.386+0000 7f4176ffd700 1 -- 172.21.15.167:0/1416833216 <== mon.0 v2:172.21.15.118:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f4180010820 con 0x560d8c398020 2023-10-05T19:29:40.393 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.386+0000 7f41893f1e40 1 -- 172.21.15.167:0/1416833216 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x560d8c3c2a30 con 0x560d8c398020 2023-10-05T19:29:40.394 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.386+0000 7f41893f1e40 20 rados->read ofs=0 len=0 2023-10-05T19:29:40.394 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.390+0000 7f4176ffd700 1 -- 172.21.15.167:0/1416833216 <== mon.0 v2:172.21.15.118:3300/0 4 ==== mgrmap(e 5) v1 ==== 40329+0+0 (secure 0 0 0) 0x7f4180010a90 con 0x560d8c398020 2023-10-05T19:29:40.395 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.390+0000 7f4176ffd700 1 --2- 172.21.15.167:0/1416833216 >> [v2:172.21.15.167:6824/29889,v1:172.21.15.167:6825/29889] conn(0x7f416c030c00 0x7f416c0330b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-05T19:29:40.395 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.390+0000 7f4176ffd700 1 -- 172.21.15.167:0/1416833216 <== mon.0 v2:172.21.15.118:3300/0 5 ==== osd_map(32..32 src has 1..32) v4 ==== 6260+0+0 (secure 0 0 0) 0x7f418002b070 con 0x560d8c398020 2023-10-05T19:29:40.395 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.390+0000 7f4186725700 1 --2- 172.21.15.167:0/1416833216 >> [v2:172.21.15.167:6824/29889,v1:172.21.15.167:6825/29889] conn(0x7f416c030c00 0x7f416c0330b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-05T19:29:40.395 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.390+0000 7f41893f1e40 1 --2- 172.21.15.167:0/1416833216 >> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] conn(0x560d8c398930 0x560d8c3cdba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-05T19:29:40.395 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.390+0000 7f41893f1e40 1 -- 172.21.15.167:0/1416833216 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:1 5.12 5:49953fa1:::default.realm:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32) v8 -- 0x560d8c3ce210 con 0x560d8c398930 2023-10-05T19:29:40.395 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.390+0000 7f4187727700 1 --2- 172.21.15.167:0/1416833216 >> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] conn(0x560d8c398930 0x560d8c3cdba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-05T19:29:40.395 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.390+0000 7f4186725700 1 --2- 172.21.15.167:0/1416833216 >> [v2:172.21.15.167:6824/29889,v1:172.21.15.167:6825/29889] conn(0x7f416c030c00 0x7f416c0330b0 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7f417c001e90 tx=0x7f417c0013b0 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-05T19:29:40.397 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.390+0000 7f4187727700 1 --2- 172.21.15.167:0/1416833216 >> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] conn(0x560d8c398930 0x560d8c3cdba0 crc :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-05T19:29:40.397 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.390+0000 7f4187727700 1 -- 172.21.15.167:0/1416833216 <== osd.1 v2:172.21.15.167:6800/30035 1 ==== osd_op_reply(1 default.realm [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 ==== 157+0+0 (crc 0 0 0) 0x7f417800a500 con 0x560d8c398930 2023-10-05T19:29:40.397 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.390+0000 7f41893f1e40 20 rados_obj.operate() r=-2 bl.length=0 2023-10-05T19:29:40.397 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.390+0000 7f41893f1e40 20 realm 2023-10-05T19:29:40.397 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.390+0000 7f41893f1e40 20 rados->read ofs=0 len=0 2023-10-05T19:29:40.397 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.390+0000 7f41893f1e40 1 -- 172.21.15.167:0/1416833216 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:2 5.12 5:49953fa1:::default.realm:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32) v8 -- 0x560d8c3cb200 con 0x560d8c398930 2023-10-05T19:29:40.398 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.390+0000 7f4187727700 1 -- 172.21.15.167:0/1416833216 <== osd.1 v2:172.21.15.167:6800/30035 2 ==== osd_op_reply(2 default.realm [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 ==== 157+0+0 (crc 0 0 0) 0x7f417800a500 con 0x560d8c398930 2023-10-05T19:29:40.398 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.390+0000 7f41893f1e40 20 rados_obj.operate() r=-2 bl.length=0 2023-10-05T19:29:40.398 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.390+0000 7f41893f1e40 4 RGWPeriod::init failed to init realm id : (2) No such file or directory 2023-10-05T19:29:40.398 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.390+0000 7f41893f1e40 20 rados->read ofs=0 len=0 2023-10-05T19:29:40.398 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.390+0000 7f41893f1e40 1 -- 172.21.15.167:0/1416833216 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:3 5.12 5:49953fa1:::default.realm:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32) v8 -- 0x560d8c3cecf0 con 0x560d8c398930 2023-10-05T19:29:40.398 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.394+0000 7f4187727700 1 -- 172.21.15.167:0/1416833216 <== osd.1 v2:172.21.15.167:6800/30035 3 ==== osd_op_reply(3 default.realm [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 ==== 157+0+0 (crc 0 0 0) 0x7f417800a500 con 0x560d8c398930 2023-10-05T19:29:40.398 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.394+0000 7f41893f1e40 20 rados_obj.operate() r=-2 bl.length=0 2023-10-05T19:29:40.398 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.394+0000 7f41893f1e40 20 rados->read ofs=0 len=0 2023-10-05T19:29:40.399 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.394+0000 7f41893f1e40 1 -- 172.21.15.167:0/1416833216 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:4 5.f 5:f43b6ece:::zone_names.default:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32) v8 -- 0x560d8c3ce900 con 0x560d8c398930 2023-10-05T19:29:40.400 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.394+0000 7f4187727700 1 -- 172.21.15.167:0/1416833216 <== osd.1 v2:172.21.15.167:6800/30035 4 ==== osd_op_reply(4 zone_names.default [read 0~46 out=46b] v0'0 uv1 ondisk = 0) v8 ==== 162+0+46 (crc 0 0 0) 0x7f417800a500 con 0x560d8c398930 2023-10-05T19:29:40.400 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.394+0000 7f41893f1e40 20 rados_obj.operate() r=0 bl.length=46 2023-10-05T19:29:40.401 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.394+0000 7f41893f1e40 20 rados->read ofs=0 len=0 2023-10-05T19:29:40.401 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.394+0000 7f41893f1e40 1 -- 172.21.15.167:0/1416833216 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:5 5.1d 5:b8374eb4:::zone_info.9a9df965-6612-45d4-a45f-6e3c889bacc2:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32) v8 -- 0x560d8c3cfb80 con 0x560d8c398930 2023-10-05T19:29:40.401 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.394+0000 7f4187727700 1 -- 172.21.15.167:0/1416833216 <== osd.1 v2:172.21.15.167:6800/30035 5 ==== osd_op_reply(5 zone_info.9a9df965-6612-45d4-a45f-6e3c889bacc2 [read 0~874 out=874b] v0'0 uv2 ondisk = 0) v8 ==== 190+0+874 (crc 0 0 0) 0x7f417800a500 con 0x560d8c398930 2023-10-05T19:29:40.401 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.394+0000 7f41893f1e40 20 rados_obj.operate() r=0 bl.length=874 2023-10-05T19:29:40.401 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.394+0000 7f41893f1e40 20 searching for the correct realm 2023-10-05T19:29:40.401 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.394+0000 7f41893f1e40 1 --2- 172.21.15.167:0/1416833216 >> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] conn(0x560d8c3d1940 0x560d8c3d3d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-05T19:29:40.401 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.394+0000 7f41893f1e40 1 -- 172.21.15.167:0/1416833216 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:6 5.0 5:00000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x560d8c3d4260 con 0x560d8c3d1940 2023-10-05T19:29:40.401 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.394+0000 7f4186f26700 1 --2- 172.21.15.167:0/1416833216 >> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] conn(0x560d8c3d1940 0x560d8c3d3d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-05T19:29:40.401 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.394+0000 7f4186f26700 1 --2- 172.21.15.167:0/1416833216 >> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] conn(0x560d8c3d1940 0x560d8c3d3d20 crc :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-05T19:29:40.402 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.394+0000 7f4186f26700 1 -- 172.21.15.167:0/1416833216 <== osd.3 v2:172.21.15.167:6808/30036 1 ==== osd_op_reply(6 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f418003adb0 con 0x560d8c3d1940 2023-10-05T19:29:40.402 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.394+0000 7f4186f26700 1 --2- 172.21.15.167:0/1416833216 >> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] conn(0x7f418001a3c0 0x7f418001b3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-05T19:29:40.402 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.394+0000 7f4186f26700 1 -- 172.21.15.167:0/1416833216 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:7 5.10 5:08000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x560d8c3d4260 con 0x7f418001a3c0 2023-10-05T19:29:40.402 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.398+0000 7f4186725700 1 --2- 172.21.15.167:0/1416833216 >> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] conn(0x7f418001a3c0 0x7f418001b3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-05T19:29:40.403 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.398+0000 7f4186725700 1 --2- 172.21.15.167:0/1416833216 >> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] conn(0x7f418001a3c0 0x7f418001b3c0 crc :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-05T19:29:40.403 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.398+0000 7f4186725700 1 -- 172.21.15.167:0/1416833216 <== osd.2 v2:172.21.15.167:6816/30037 1 ==== osd_op_reply(7 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f417c006070 con 0x7f418001a3c0 2023-10-05T19:29:40.403 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.398+0000 7f4186725700 1 --2- 172.21.15.167:0/1416833216 >> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] conn(0x7f417c00d8e0 0x7f417c00c360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-05T19:29:40.403 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.398+0000 7f4186725700 1 -- 172.21.15.167:0/1416833216 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:8 5.8 5:10000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x560d8c3d4260 con 0x7f417c00d8e0 2023-10-05T19:29:40.403 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.398+0000 7f4187727700 1 --2- 172.21.15.167:0/1416833216 >> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] conn(0x7f417c00d8e0 0x7f417c00c360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-05T19:29:40.404 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.398+0000 7f4187727700 1 --2- 172.21.15.167:0/1416833216 >> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] conn(0x7f417c00d8e0 0x7f417c00c360 crc :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-05T19:29:40.404 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.398+0000 7f4187727700 1 -- 172.21.15.167:0/1416833216 <== osd.0 v2:172.21.15.118:6800/27690 1 ==== osd_op_reply(8 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f4178014040 con 0x7f417c00d8e0 2023-10-05T19:29:40.404 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.398+0000 7f4187727700 1 -- 172.21.15.167:0/1416833216 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:9 5.18 5:18000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x560d8c3d4260 con 0x560d8c398930 2023-10-05T19:29:40.404 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.402+0000 7f4187727700 1 -- 172.21.15.167:0/1416833216 <== osd.1 v2:172.21.15.167:6800/30035 6 ==== osd_op_reply(9 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f4178014040 con 0x560d8c398930 2023-10-05T19:29:40.405 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.402+0000 7f4187727700 1 -- 172.21.15.167:0/1416833216 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:10 5.4 5:20000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x560d8c3d4260 con 0x560d8c398930 2023-10-05T19:29:40.405 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.402+0000 7f4187727700 1 -- 172.21.15.167:0/1416833216 <== osd.1 v2:172.21.15.167:6800/30035 7 ==== osd_op_reply(10 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f4178014040 con 0x560d8c398930 2023-10-05T19:29:40.405 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.402+0000 7f4187727700 1 -- 172.21.15.167:0/1416833216 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:11 5.14 5:28000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x560d8c3d4260 con 0x7f417c00d8e0 2023-10-05T19:29:40.406 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.402+0000 7f4187727700 1 -- 172.21.15.167:0/1416833216 <== osd.0 v2:172.21.15.118:6800/27690 2 ==== osd_op_reply(11 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f4178014040 con 0x7f417c00d8e0 2023-10-05T19:29:40.406 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.402+0000 7f4187727700 1 -- 172.21.15.167:0/1416833216 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:12 5.c 5:30000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x560d8c3d4260 con 0x7f417c00d8e0 2023-10-05T19:29:40.407 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.402+0000 7f4187727700 1 -- 172.21.15.167:0/1416833216 <== osd.0 v2:172.21.15.118:6800/27690 3 ==== osd_op_reply(12 [pgnls start_epoch 32 out=112b] v23'1 uv1 ondisk = 1) v8 ==== 144+0+112 (crc 0 0 0) 0x7f4178014040 con 0x7f417c00d8e0 2023-10-05T19:29:40.407 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.402+0000 7f4187727700 1 -- 172.21.15.167:0/1416833216 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:13 5.1c 5:38000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x560d8c3d4260 con 0x7f418001a3c0 2023-10-05T19:29:40.407 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.402+0000 7f4186725700 1 -- 172.21.15.167:0/1416833216 <== osd.2 v2:172.21.15.167:6816/30037 2 ==== osd_op_reply(13 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f417c006070 con 0x7f418001a3c0 2023-10-05T19:29:40.408 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.402+0000 7f4186725700 1 -- 172.21.15.167:0/1416833216 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:14 5.2 5:40000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x560d8c3d4260 con 0x560d8c3d1940 2023-10-05T19:29:40.408 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.402+0000 7f4186f26700 1 -- 172.21.15.167:0/1416833216 <== osd.3 v2:172.21.15.167:6808/30036 2 ==== osd_op_reply(14 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f418003adb0 con 0x560d8c3d1940 2023-10-05T19:29:40.408 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.402+0000 7f4186f26700 1 -- 172.21.15.167:0/1416833216 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:15 5.12 5:48000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x560d8c3d4260 con 0x560d8c398930 2023-10-05T19:29:40.409 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.406+0000 7f4187727700 1 -- 172.21.15.167:0/1416833216 <== osd.1 v2:172.21.15.167:6800/30035 8 ==== osd_op_reply(15 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f4178014040 con 0x560d8c398930 2023-10-05T19:29:40.409 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.406+0000 7f4187727700 1 -- 172.21.15.167:0/1416833216 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:16 5.a 5:50000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x560d8c3d4260 con 0x7f417c00d8e0 2023-10-05T19:29:40.410 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.406+0000 7f4187727700 1 -- 172.21.15.167:0/1416833216 <== osd.0 v2:172.21.15.118:6800/27690 4 ==== osd_op_reply(16 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f4178022040 con 0x7f417c00d8e0 2023-10-05T19:29:40.410 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.406+0000 7f4187727700 1 -- 172.21.15.167:0/1416833216 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:17 5.1a 5:58000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x560d8c3d4260 con 0x560d8c398930 2023-10-05T19:29:40.410 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.406+0000 7f4187727700 1 -- 172.21.15.167:0/1416833216 <== osd.1 v2:172.21.15.167:6800/30035 9 ==== osd_op_reply(17 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f4178022040 con 0x560d8c398930 2023-10-05T19:29:40.411 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.406+0000 7f4187727700 1 -- 172.21.15.167:0/1416833216 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:18 5.6 5:60000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x560d8c3d4260 con 0x7f418001a3c0 2023-10-05T19:29:40.411 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.406+0000 7f4186725700 1 -- 172.21.15.167:0/1416833216 <== osd.2 v2:172.21.15.167:6816/30037 3 ==== osd_op_reply(18 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f417c006070 con 0x7f418001a3c0 2023-10-05T19:29:40.411 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.406+0000 7f4186725700 1 -- 172.21.15.167:0/1416833216 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:19 5.16 5:68000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x560d8c3d4260 con 0x560d8c3d1940 2023-10-05T19:29:40.412 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.406+0000 7f4186f26700 1 -- 172.21.15.167:0/1416833216 <== osd.3 v2:172.21.15.167:6808/30036 3 ==== osd_op_reply(19 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f418003adb0 con 0x560d8c3d1940 2023-10-05T19:29:40.412 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.406+0000 7f4186f26700 1 -- 172.21.15.167:0/1416833216 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:20 5.e 5:70000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x560d8c3d4260 con 0x560d8c3d1940 2023-10-05T19:29:40.413 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.410+0000 7f4186f26700 1 -- 172.21.15.167:0/1416833216 <== osd.3 v2:172.21.15.167:6808/30036 4 ==== osd_op_reply(20 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f418003adb0 con 0x560d8c3d1940 2023-10-05T19:29:40.413 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.410+0000 7f4186f26700 1 -- 172.21.15.167:0/1416833216 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:21 5.1e 5:78000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x560d8c3d4260 con 0x7f417c00d8e0 2023-10-05T19:29:40.414 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.410+0000 7f4187727700 1 -- 172.21.15.167:0/1416833216 <== osd.0 v2:172.21.15.118:6800/27690 5 ==== osd_op_reply(21 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f4178022040 con 0x7f417c00d8e0 2023-10-05T19:29:40.414 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.410+0000 7f4187727700 1 -- 172.21.15.167:0/1416833216 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:22 5.1 5:80000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x560d8c3d4260 con 0x560d8c3d1940 2023-10-05T19:29:40.414 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.410+0000 7f4186f26700 1 -- 172.21.15.167:0/1416833216 <== osd.3 v2:172.21.15.167:6808/30036 5 ==== osd_op_reply(22 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f418003adb0 con 0x560d8c3d1940 2023-10-05T19:29:40.415 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.410+0000 7f4186f26700 1 -- 172.21.15.167:0/1416833216 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:23 5.11 5:88000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x560d8c3d4260 con 0x560d8c3d1940 2023-10-05T19:29:40.415 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.410+0000 7f4186f26700 1 -- 172.21.15.167:0/1416833216 <== osd.3 v2:172.21.15.167:6808/30036 6 ==== osd_op_reply(23 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f418003adb0 con 0x560d8c3d1940 2023-10-05T19:29:40.415 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.410+0000 7f4186f26700 1 -- 172.21.15.167:0/1416833216 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:24 5.9 5:90000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x560d8c3d4260 con 0x560d8c398930 2023-10-05T19:29:40.416 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.410+0000 7f4187727700 1 -- 172.21.15.167:0/1416833216 <== osd.1 v2:172.21.15.167:6800/30035 10 ==== osd_op_reply(24 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f4178022040 con 0x560d8c398930 2023-10-05T19:29:40.416 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.410+0000 7f4187727700 1 -- 172.21.15.167:0/1416833216 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:25 5.19 5:98000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x560d8c3d4260 con 0x560d8c398930 2023-10-05T19:29:40.417 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.414+0000 7f4187727700 1 -- 172.21.15.167:0/1416833216 <== osd.1 v2:172.21.15.167:6800/30035 11 ==== osd_op_reply(25 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f4178022040 con 0x560d8c398930 2023-10-05T19:29:40.417 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.414+0000 7f4187727700 1 -- 172.21.15.167:0/1416833216 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:26 5.5 5:a0000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x560d8c3d4260 con 0x7f417c00d8e0 2023-10-05T19:29:40.418 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.414+0000 7f4187727700 1 -- 172.21.15.167:0/1416833216 <== osd.0 v2:172.21.15.118:6800/27690 6 ==== osd_op_reply(26 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f4178022040 con 0x7f417c00d8e0 2023-10-05T19:29:40.418 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.414+0000 7f4187727700 1 -- 172.21.15.167:0/1416833216 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:27 5.15 5:a8000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x560d8c3d4260 con 0x560d8c3d1940 2023-10-05T19:29:40.419 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.414+0000 7f4186f26700 1 -- 172.21.15.167:0/1416833216 <== osd.3 v2:172.21.15.167:6808/30036 7 ==== osd_op_reply(27 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f418003adb0 con 0x560d8c3d1940 2023-10-05T19:29:40.419 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.414+0000 7f4186f26700 1 -- 172.21.15.167:0/1416833216 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:28 5.d 5:b0000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x560d8c3d4260 con 0x7f418001a3c0 2023-10-05T19:29:40.419 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.414+0000 7f4186725700 1 -- 172.21.15.167:0/1416833216 <== osd.2 v2:172.21.15.167:6816/30037 4 ==== osd_op_reply(28 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f417c006070 con 0x7f418001a3c0 2023-10-05T19:29:40.419 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.414+0000 7f4186725700 1 -- 172.21.15.167:0/1416833216 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:29 5.1d 5:b8000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x560d8c3d4260 con 0x560d8c398930 2023-10-05T19:29:40.420 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.414+0000 7f4187727700 1 -- 172.21.15.167:0/1416833216 <== osd.1 v2:172.21.15.167:6800/30035 12 ==== osd_op_reply(29 [pgnls start_epoch 32 out=107b] v30'2 uv2 ondisk = 1) v8 ==== 144+0+107 (crc 0 0 0) 0x7f4178022040 con 0x560d8c398930 2023-10-05T19:29:40.420 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.414+0000 7f4187727700 1 -- 172.21.15.167:0/1416833216 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:30 5.3 5:c0000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x560d8c3d4260 con 0x7f417c00d8e0 2023-10-05T19:29:40.421 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.418+0000 7f4187727700 1 -- 172.21.15.167:0/1416833216 <== osd.0 v2:172.21.15.118:6800/27690 7 ==== osd_op_reply(30 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f4178022040 con 0x7f417c00d8e0 2023-10-05T19:29:40.421 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.418+0000 7f4187727700 1 -- 172.21.15.167:0/1416833216 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:31 5.13 5:c8000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x560d8c3d4260 con 0x560d8c3d1940 2023-10-05T19:29:40.422 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.418+0000 7f4186f26700 1 -- 172.21.15.167:0/1416833216 <== osd.3 v2:172.21.15.167:6808/30036 8 ==== osd_op_reply(31 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f418003adb0 con 0x560d8c3d1940 2023-10-05T19:29:40.422 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.418+0000 7f4186f26700 1 -- 172.21.15.167:0/1416833216 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:32 5.b 5:d0000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x560d8c3d4260 con 0x7f417c00d8e0 2023-10-05T19:29:40.423 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.418+0000 7f4187727700 1 -- 172.21.15.167:0/1416833216 <== osd.0 v2:172.21.15.118:6800/27690 8 ==== osd_op_reply(32 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f4178022040 con 0x7f417c00d8e0 2023-10-05T19:29:40.423 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.418+0000 7f4187727700 1 -- 172.21.15.167:0/1416833216 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:33 5.1b 5:d8000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x560d8c3d4260 con 0x7f418001a3c0 2023-10-05T19:29:40.424 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.418+0000 7f4186725700 1 -- 172.21.15.167:0/1416833216 <== osd.2 v2:172.21.15.167:6816/30037 5 ==== osd_op_reply(33 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f417c006070 con 0x7f418001a3c0 2023-10-05T19:29:40.424 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.418+0000 7f4186725700 1 -- 172.21.15.167:0/1416833216 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:34 5.7 5:e0000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x560d8c3d4260 con 0x560d8c3d1940 2023-10-05T19:29:40.424 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.422+0000 7f4186f26700 1 -- 172.21.15.167:0/1416833216 <== osd.3 v2:172.21.15.167:6808/30036 9 ==== osd_op_reply(34 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f418003adb0 con 0x560d8c3d1940 2023-10-05T19:29:40.425 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.422+0000 7f4186f26700 1 -- 172.21.15.167:0/1416833216 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:35 5.17 5:e8000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x560d8c3d4260 con 0x7f417c00d8e0 2023-10-05T19:29:40.425 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.422+0000 7f4187727700 1 -- 172.21.15.167:0/1416833216 <== osd.0 v2:172.21.15.118:6800/27690 9 ==== osd_op_reply(35 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f4178022040 con 0x7f417c00d8e0 2023-10-05T19:29:40.426 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.422+0000 7f4187727700 1 -- 172.21.15.167:0/1416833216 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:36 5.f 5:f0000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x560d8c3d4260 con 0x560d8c398930 2023-10-05T19:29:40.426 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.422+0000 7f4187727700 1 -- 172.21.15.167:0/1416833216 <== osd.1 v2:172.21.15.167:6800/30035 13 ==== osd_op_reply(36 [pgnls start_epoch 32 out=115b] v23'2 uv2 ondisk = 1) v8 ==== 144+0+115 (crc 0 0 0) 0x7f417802f080 con 0x560d8c398930 2023-10-05T19:29:40.427 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.422+0000 7f4187727700 1 -- 172.21.15.167:0/1416833216 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:37 5.1f 5:f8000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x560d8c3d4260 con 0x7f418001a3c0 2023-10-05T19:29:40.427 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.422+0000 7f4186725700 1 -- 172.21.15.167:0/1416833216 <== osd.2 v2:172.21.15.167:6816/30037 6 ==== osd_op_reply(37 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f417c006070 con 0x7f418001a3c0 2023-10-05T19:29:40.428 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.422+0000 7f41893f1e40 20 RGWRados::pool_iterate: got zonegroup_info.4f1f719d-9648-417b-b10c-e5d939825419 2023-10-05T19:29:40.428 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.422+0000 7f41893f1e40 20 RGWRados::pool_iterate: got zone_info.9a9df965-6612-45d4-a45f-6e3c889bacc2 2023-10-05T19:29:40.428 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.422+0000 7f41893f1e40 20 RGWRados::pool_iterate: got zone_names.default 2023-10-05T19:29:40.428 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.422+0000 7f41893f1e40 20 RGWRados::pool_iterate: got zonegroups_names.default 2023-10-05T19:29:40.428 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.422+0000 7f41893f1e40 20 rados->read ofs=0 len=0 2023-10-05T19:29:40.428 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.422+0000 7f41893f1e40 1 -- 172.21.15.167:0/1416833216 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:38 5.12 5:49953fa1:::default.realm:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32) v8 -- 0x560d8c3d06f0 con 0x560d8c398930 2023-10-05T19:29:40.429 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.422+0000 7f4187727700 1 -- 172.21.15.167:0/1416833216 <== osd.1 v2:172.21.15.167:6800/30035 14 ==== osd_op_reply(38 default.realm [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 ==== 157+0+0 (crc 0 0 0) 0x7f417802f080 con 0x560d8c398930 2023-10-05T19:29:40.429 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.422+0000 7f41893f1e40 20 rados_obj.operate() r=-2 bl.length=0 2023-10-05T19:29:40.430 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.422+0000 7f41893f1e40 20 rados->read ofs=0 len=0 2023-10-05T19:29:40.430 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.422+0000 7f41893f1e40 1 -- 172.21.15.167:0/1416833216 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:39 5.f 5:f4c53578:::zonegroups_names.default:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32) v8 -- 0x560d8c3d0fe0 con 0x560d8c398930 2023-10-05T19:29:40.430 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.426+0000 7f4187727700 1 -- 172.21.15.167:0/1416833216 <== osd.1 v2:172.21.15.167:6800/30035 15 ==== osd_op_reply(39 zonegroups_names.default [read 0~46 out=46b] v0'0 uv2 ondisk = 0) v8 ==== 168+0+46 (crc 0 0 0) 0x7f417802f080 con 0x560d8c398930 2023-10-05T19:29:40.430 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.426+0000 7f41893f1e40 20 rados_obj.operate() r=0 bl.length=46 2023-10-05T19:29:40.430 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.426+0000 7f41893f1e40 20 rados->read ofs=0 len=0 2023-10-05T19:29:40.430 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.426+0000 7f41893f1e40 1 -- 172.21.15.167:0/1416833216 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:40 5.c 5:30d299a5:::zonegroup_info.4f1f719d-9648-417b-b10c-e5d939825419:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32) v8 -- 0x560d8c3d46d0 con 0x7f417c00d8e0 2023-10-05T19:29:40.432 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.426+0000 7f4187727700 1 -- 172.21.15.167:0/1416833216 <== osd.0 v2:172.21.15.118:6800/27690 10 ==== osd_op_reply(40 zonegroup_info.4f1f719d-9648-417b-b10c-e5d939825419 [read 0~362 out=362b] v0'0 uv1 ondisk = 0) v8 ==== 195+0+362 (crc 0 0 0) 0x7f417802f080 con 0x7f417c00d8e0 2023-10-05T19:29:40.432 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.426+0000 7f41893f1e40 20 rados_obj.operate() r=0 bl.length=362 2023-10-05T19:29:40.432 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.426+0000 7f41893f1e40 20 rados->read ofs=0 len=0 2023-10-05T19:29:40.432 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.426+0000 7f41893f1e40 1 -- 172.21.15.167:0/1416833216 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:41 5.13 5:cbbf6c60:::converted:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32) v8 -- 0x560d8c3d46d0 con 0x560d8c3d1940 2023-10-05T19:29:40.432 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.426+0000 7f4186f26700 1 -- 172.21.15.167:0/1416833216 <== osd.3 v2:172.21.15.167:6808/30036 10 ==== osd_op_reply(41 converted [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 ==== 153+0+0 (crc 0 0 0) 0x7f418003adb0 con 0x560d8c3d1940 2023-10-05T19:29:40.432 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.426+0000 7f41893f1e40 20 rados_obj.operate() r=-2 bl.length=0 2023-10-05T19:29:40.432 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.426+0000 7f41893f1e40 20 rados->read ofs=0 len=0 2023-10-05T19:29:40.432 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.426+0000 7f41893f1e40 1 -- 172.21.15.167:0/1416833216 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:42 5.12 5:49953fa1:::default.realm:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32) v8 -- 0x560d8c3d4a80 con 0x560d8c398930 2023-10-05T19:29:40.433 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.426+0000 7f4187727700 1 -- 172.21.15.167:0/1416833216 <== osd.1 v2:172.21.15.167:6800/30035 16 ==== osd_op_reply(42 default.realm [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 ==== 157+0+0 (crc 0 0 0) 0x7f417802f080 con 0x560d8c398930 2023-10-05T19:29:40.433 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.426+0000 7f41893f1e40 20 rados_obj.operate() r=-2 bl.length=0 2023-10-05T19:29:40.433 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.426+0000 7f41893f1e40 20 rados->read ofs=0 len=0 2023-10-05T19:29:40.433 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.426+0000 7f41893f1e40 1 -- 172.21.15.167:0/1416833216 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:43 5.f 5:f4c53578:::zonegroups_names.default:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32) v8 -- 0x560d8c3d4eb0 con 0x560d8c398930 2023-10-05T19:29:40.434 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.426+0000 7f4187727700 1 -- 172.21.15.167:0/1416833216 <== osd.1 v2:172.21.15.167:6800/30035 17 ==== osd_op_reply(43 zonegroups_names.default [read 0~46 out=46b] v0'0 uv2 ondisk = 0) v8 ==== 168+0+46 (crc 0 0 0) 0x7f417802f080 con 0x560d8c398930 2023-10-05T19:29:40.434 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.426+0000 7f41893f1e40 20 rados_obj.operate() r=0 bl.length=46 2023-10-05T19:29:40.434 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.430+0000 7f41893f1e40 1 -- 172.21.15.167:0/1416833216 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:44 5.0 5:00000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x560d8c3d5ec0 con 0x560d8c3d1940 2023-10-05T19:29:40.434 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.430+0000 7f4186f26700 1 -- 172.21.15.167:0/1416833216 <== osd.3 v2:172.21.15.167:6808/30036 11 ==== osd_op_reply(44 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f418003adb0 con 0x560d8c3d1940 2023-10-05T19:29:40.434 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.430+0000 7f4186f26700 1 -- 172.21.15.167:0/1416833216 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:45 5.10 5:08000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x560d8c3d5ec0 con 0x7f418001a3c0 2023-10-05T19:29:40.435 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.430+0000 7f4186725700 1 -- 172.21.15.167:0/1416833216 <== osd.2 v2:172.21.15.167:6816/30037 7 ==== osd_op_reply(45 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f417c006070 con 0x7f418001a3c0 2023-10-05T19:29:40.435 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.430+0000 7f4186725700 1 -- 172.21.15.167:0/1416833216 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:46 5.8 5:10000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x560d8c3d5ec0 con 0x7f417c00d8e0 2023-10-05T19:29:40.436 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.430+0000 7f4187727700 1 -- 172.21.15.167:0/1416833216 <== osd.0 v2:172.21.15.118:6800/27690 11 ==== osd_op_reply(46 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f417802f080 con 0x7f417c00d8e0 2023-10-05T19:29:40.436 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.430+0000 7f4187727700 1 -- 172.21.15.167:0/1416833216 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:47 5.18 5:18000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x560d8c3d0fe0 con 0x560d8c398930 2023-10-05T19:29:40.437 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.434+0000 7f4187727700 1 -- 172.21.15.167:0/1416833216 <== osd.1 v2:172.21.15.167:6800/30035 18 ==== osd_op_reply(47 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f417802f080 con 0x560d8c398930 2023-10-05T19:29:40.437 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.434+0000 7f4187727700 1 -- 172.21.15.167:0/1416833216 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:48 5.4 5:20000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x560d8c3d0fe0 con 0x560d8c398930 2023-10-05T19:29:40.437 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.434+0000 7f4187727700 1 -- 172.21.15.167:0/1416833216 <== osd.1 v2:172.21.15.167:6800/30035 19 ==== osd_op_reply(48 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f417802f080 con 0x560d8c398930 2023-10-05T19:29:40.438 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.434+0000 7f4187727700 1 -- 172.21.15.167:0/1416833216 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:49 5.14 5:28000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x560d8c3d0fe0 con 0x7f417c00d8e0 2023-10-05T19:29:40.439 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.434+0000 7f4187727700 1 -- 172.21.15.167:0/1416833216 <== osd.0 v2:172.21.15.118:6800/27690 12 ==== osd_op_reply(49 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f417802f080 con 0x7f417c00d8e0 2023-10-05T19:29:40.439 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.434+0000 7f4187727700 1 -- 172.21.15.167:0/1416833216 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:50 5.c 5:30000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x560d8c3d0fe0 con 0x7f417c00d8e0 2023-10-05T19:29:40.440 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.434+0000 7f4187727700 1 -- 172.21.15.167:0/1416833216 <== osd.0 v2:172.21.15.118:6800/27690 13 ==== osd_op_reply(50 [pgnls start_epoch 32 out=112b] v23'1 uv1 ondisk = 1) v8 ==== 144+0+112 (crc 0 0 0) 0x7f417802f080 con 0x7f417c00d8e0 2023-10-05T19:29:40.440 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.434+0000 7f4187727700 1 -- 172.21.15.167:0/1416833216 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:51 5.1c 5:38000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x560d8c3d0fe0 con 0x7f418001a3c0 2023-10-05T19:29:40.441 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.438+0000 7f4186725700 1 -- 172.21.15.167:0/1416833216 <== osd.2 v2:172.21.15.167:6816/30037 8 ==== osd_op_reply(51 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f417c006070 con 0x7f418001a3c0 2023-10-05T19:29:40.441 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.438+0000 7f4186725700 1 -- 172.21.15.167:0/1416833216 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:52 5.2 5:40000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x560d8c3d0fe0 con 0x560d8c3d1940 2023-10-05T19:29:40.442 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.438+0000 7f4186f26700 1 -- 172.21.15.167:0/1416833216 <== osd.3 v2:172.21.15.167:6808/30036 12 ==== osd_op_reply(52 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f418003adb0 con 0x560d8c3d1940 2023-10-05T19:29:40.442 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.438+0000 7f4186f26700 1 -- 172.21.15.167:0/1416833216 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:53 5.12 5:48000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x560d8c3d0fe0 con 0x560d8c398930 2023-10-05T19:29:40.443 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.438+0000 7f4187727700 1 -- 172.21.15.167:0/1416833216 <== osd.1 v2:172.21.15.167:6800/30035 20 ==== osd_op_reply(53 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f417802f080 con 0x560d8c398930 2023-10-05T19:29:40.443 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.438+0000 7f4187727700 1 -- 172.21.15.167:0/1416833216 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:54 5.a 5:50000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x560d8c3d0fe0 con 0x7f417c00d8e0 2023-10-05T19:29:40.444 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.438+0000 7f4187727700 1 -- 172.21.15.167:0/1416833216 <== osd.0 v2:172.21.15.118:6800/27690 14 ==== osd_op_reply(54 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f417802f080 con 0x7f417c00d8e0 2023-10-05T19:29:40.444 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.438+0000 7f4187727700 1 -- 172.21.15.167:0/1416833216 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:55 5.1a 5:58000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x560d8c3d0fe0 con 0x560d8c398930 2023-10-05T19:29:40.444 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.442+0000 7f4187727700 1 -- 172.21.15.167:0/1416833216 <== osd.1 v2:172.21.15.167:6800/30035 21 ==== osd_op_reply(55 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f417802f080 con 0x560d8c398930 2023-10-05T19:29:40.445 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.442+0000 7f4187727700 1 -- 172.21.15.167:0/1416833216 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:56 5.6 5:60000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x560d8c3d0fe0 con 0x7f418001a3c0 2023-10-05T19:29:40.446 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.442+0000 7f4186725700 1 -- 172.21.15.167:0/1416833216 <== osd.2 v2:172.21.15.167:6816/30037 9 ==== osd_op_reply(56 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f417c006070 con 0x7f418001a3c0 2023-10-05T19:29:40.446 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.442+0000 7f4186725700 1 -- 172.21.15.167:0/1416833216 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:57 5.16 5:68000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x560d8c3d0fe0 con 0x560d8c3d1940 2023-10-05T19:29:40.446 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.442+0000 7f4186f26700 1 -- 172.21.15.167:0/1416833216 <== osd.3 v2:172.21.15.167:6808/30036 13 ==== osd_op_reply(57 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f418003adb0 con 0x560d8c3d1940 2023-10-05T19:29:40.447 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.442+0000 7f4186f26700 1 -- 172.21.15.167:0/1416833216 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:58 5.e 5:70000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x560d8c3d0fe0 con 0x560d8c3d1940 2023-10-05T19:29:40.447 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.442+0000 7f4186f26700 1 -- 172.21.15.167:0/1416833216 <== osd.3 v2:172.21.15.167:6808/30036 14 ==== osd_op_reply(58 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f418003adb0 con 0x560d8c3d1940 2023-10-05T19:29:40.448 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.442+0000 7f4186f26700 1 -- 172.21.15.167:0/1416833216 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:59 5.1e 5:78000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x560d8c3d0fe0 con 0x7f417c00d8e0 2023-10-05T19:29:40.449 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.446+0000 7f4187727700 1 -- 172.21.15.167:0/1416833216 <== osd.0 v2:172.21.15.118:6800/27690 15 ==== osd_op_reply(59 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f417802f080 con 0x7f417c00d8e0 2023-10-05T19:29:40.449 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.446+0000 7f4187727700 1 -- 172.21.15.167:0/1416833216 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:60 5.1 5:80000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x560d8c3d0fe0 con 0x560d8c3d1940 2023-10-05T19:29:40.450 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.446+0000 7f4186f26700 1 -- 172.21.15.167:0/1416833216 <== osd.3 v2:172.21.15.167:6808/30036 15 ==== osd_op_reply(60 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f418003adb0 con 0x560d8c3d1940 2023-10-05T19:29:40.450 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.446+0000 7f4186f26700 1 -- 172.21.15.167:0/1416833216 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:61 5.11 5:88000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x560d8c3d0fe0 con 0x560d8c3d1940 2023-10-05T19:29:40.450 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.446+0000 7f4186f26700 1 -- 172.21.15.167:0/1416833216 <== osd.3 v2:172.21.15.167:6808/30036 16 ==== osd_op_reply(61 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f418003adb0 con 0x560d8c3d1940 2023-10-05T19:29:40.451 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.446+0000 7f4186f26700 1 -- 172.21.15.167:0/1416833216 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:62 5.9 5:90000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x560d8c3d0fe0 con 0x560d8c398930 2023-10-05T19:29:40.452 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.446+0000 7f4187727700 1 -- 172.21.15.167:0/1416833216 <== osd.1 v2:172.21.15.167:6800/30035 22 ==== osd_op_reply(62 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f417802f080 con 0x560d8c398930 2023-10-05T19:29:40.452 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.446+0000 7f4187727700 1 -- 172.21.15.167:0/1416833216 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:63 5.19 5:98000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x560d8c3d0fe0 con 0x560d8c398930 2023-10-05T19:29:40.453 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.450+0000 7f4187727700 1 -- 172.21.15.167:0/1416833216 <== osd.1 v2:172.21.15.167:6800/30035 23 ==== osd_op_reply(63 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f417802f080 con 0x560d8c398930 2023-10-05T19:29:40.453 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.450+0000 7f4187727700 1 -- 172.21.15.167:0/1416833216 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:64 5.5 5:a0000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x560d8c3d0fe0 con 0x7f417c00d8e0 2023-10-05T19:29:40.454 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.450+0000 7f4187727700 1 -- 172.21.15.167:0/1416833216 <== osd.0 v2:172.21.15.118:6800/27690 16 ==== osd_op_reply(64 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f417802f080 con 0x7f417c00d8e0 2023-10-05T19:29:40.454 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.450+0000 7f4187727700 1 -- 172.21.15.167:0/1416833216 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:65 5.15 5:a8000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x560d8c3d0fe0 con 0x560d8c3d1940 2023-10-05T19:29:40.455 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.450+0000 7f4186f26700 1 -- 172.21.15.167:0/1416833216 <== osd.3 v2:172.21.15.167:6808/30036 17 ==== osd_op_reply(65 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f418003adb0 con 0x560d8c3d1940 2023-10-05T19:29:40.455 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.450+0000 7f4186f26700 1 -- 172.21.15.167:0/1416833216 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:66 5.d 5:b0000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x560d8c3d0fe0 con 0x7f418001a3c0 2023-10-05T19:29:40.456 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.450+0000 7f4186725700 1 -- 172.21.15.167:0/1416833216 <== osd.2 v2:172.21.15.167:6816/30037 10 ==== osd_op_reply(66 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f417c006070 con 0x7f418001a3c0 2023-10-05T19:29:40.456 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.450+0000 7f4186725700 1 -- 172.21.15.167:0/1416833216 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:67 5.1d 5:b8000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x560d8c3d0fe0 con 0x560d8c398930 2023-10-05T19:29:40.456 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.454+0000 7f4187727700 1 -- 172.21.15.167:0/1416833216 <== osd.1 v2:172.21.15.167:6800/30035 24 ==== osd_op_reply(67 [pgnls start_epoch 32 out=107b] v30'2 uv2 ondisk = 1) v8 ==== 144+0+107 (crc 0 0 0) 0x7f417802f080 con 0x560d8c398930 2023-10-05T19:29:40.457 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.454+0000 7f4187727700 1 -- 172.21.15.167:0/1416833216 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:68 5.3 5:c0000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x560d8c3d0fe0 con 0x7f417c00d8e0 2023-10-05T19:29:40.458 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.454+0000 7f4187727700 1 -- 172.21.15.167:0/1416833216 <== osd.0 v2:172.21.15.118:6800/27690 17 ==== osd_op_reply(68 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f417802f080 con 0x7f417c00d8e0 2023-10-05T19:29:40.458 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.454+0000 7f4187727700 1 -- 172.21.15.167:0/1416833216 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:69 5.13 5:c8000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x560d8c3d0fe0 con 0x560d8c3d1940 2023-10-05T19:29:40.459 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.454+0000 7f4186f26700 1 -- 172.21.15.167:0/1416833216 <== osd.3 v2:172.21.15.167:6808/30036 18 ==== osd_op_reply(69 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f418003adb0 con 0x560d8c3d1940 2023-10-05T19:29:40.459 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.454+0000 7f4186f26700 1 -- 172.21.15.167:0/1416833216 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:70 5.b 5:d0000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x560d8c3d0fe0 con 0x7f417c00d8e0 2023-10-05T19:29:40.459 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.454+0000 7f4187727700 1 -- 172.21.15.167:0/1416833216 <== osd.0 v2:172.21.15.118:6800/27690 18 ==== osd_op_reply(70 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f417802f080 con 0x7f417c00d8e0 2023-10-05T19:29:40.460 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.454+0000 7f4187727700 1 -- 172.21.15.167:0/1416833216 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:71 5.1b 5:d8000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x560d8c3d0fe0 con 0x7f418001a3c0 2023-10-05T19:29:40.460 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.458+0000 7f4186725700 1 -- 172.21.15.167:0/1416833216 <== osd.2 v2:172.21.15.167:6816/30037 11 ==== osd_op_reply(71 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f417c006070 con 0x7f418001a3c0 2023-10-05T19:29:40.461 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.458+0000 7f4186725700 1 -- 172.21.15.167:0/1416833216 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:72 5.7 5:e0000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x560d8c3d0fe0 con 0x560d8c3d1940 2023-10-05T19:29:40.461 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.458+0000 7f4186f26700 1 -- 172.21.15.167:0/1416833216 <== osd.3 v2:172.21.15.167:6808/30036 19 ==== osd_op_reply(72 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f418003adb0 con 0x560d8c3d1940 2023-10-05T19:29:40.461 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.458+0000 7f4186f26700 1 -- 172.21.15.167:0/1416833216 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:73 5.17 5:e8000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x560d8c3d0fe0 con 0x7f417c00d8e0 2023-10-05T19:29:40.462 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.458+0000 7f4187727700 1 -- 172.21.15.167:0/1416833216 <== osd.0 v2:172.21.15.118:6800/27690 19 ==== osd_op_reply(73 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f417802f080 con 0x7f417c00d8e0 2023-10-05T19:29:40.463 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.458+0000 7f4187727700 1 -- 172.21.15.167:0/1416833216 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:74 5.f 5:f0000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x560d8c3d0fe0 con 0x560d8c398930 2023-10-05T19:29:40.464 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.458+0000 7f4187727700 1 -- 172.21.15.167:0/1416833216 <== osd.1 v2:172.21.15.167:6800/30035 25 ==== osd_op_reply(74 [pgnls start_epoch 32 out=115b] v23'2 uv2 ondisk = 1) v8 ==== 144+0+115 (crc 0 0 0) 0x7f417802f080 con 0x560d8c398930 2023-10-05T19:29:40.464 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.458+0000 7f4187727700 1 -- 172.21.15.167:0/1416833216 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:75 5.1f 5:f8000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x560d8c3d0fe0 con 0x7f418001a3c0 2023-10-05T19:29:40.465 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.462+0000 7f4186725700 1 -- 172.21.15.167:0/1416833216 <== osd.2 v2:172.21.15.167:6816/30037 12 ==== osd_op_reply(75 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f417c006070 con 0x7f418001a3c0 2023-10-05T19:29:40.465 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.462+0000 7f41893f1e40 20 RGWRados::pool_iterate: got zonegroup_info.4f1f719d-9648-417b-b10c-e5d939825419 2023-10-05T19:29:40.465 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.462+0000 7f41893f1e40 20 RGWRados::pool_iterate: got zone_info.9a9df965-6612-45d4-a45f-6e3c889bacc2 2023-10-05T19:29:40.465 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.462+0000 7f41893f1e40 20 RGWRados::pool_iterate: got zone_names.default 2023-10-05T19:29:40.465 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.462+0000 7f41893f1e40 20 RGWRados::pool_iterate: got zonegroups_names.default 2023-10-05T19:29:40.465 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.462+0000 7f41893f1e40 20 rados->read ofs=0 len=0 2023-10-05T19:29:40.465 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.462+0000 7f41893f1e40 1 -- 172.21.15.167:0/1416833216 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:76 5.f 5:f43b6ece:::zone_names.default:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32) v8 -- 0x560d8c3d4eb0 con 0x560d8c398930 2023-10-05T19:29:40.466 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.462+0000 7f4187727700 1 -- 172.21.15.167:0/1416833216 <== osd.1 v2:172.21.15.167:6800/30035 26 ==== osd_op_reply(76 zone_names.default [read 0~46 out=46b] v0'0 uv1 ondisk = 0) v8 ==== 162+0+46 (crc 0 0 0) 0x7f417802f080 con 0x560d8c398930 2023-10-05T19:29:40.467 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.462+0000 7f41893f1e40 20 rados_obj.operate() r=0 bl.length=46 2023-10-05T19:29:40.467 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.462+0000 7f41893f1e40 20 rados->read ofs=0 len=0 2023-10-05T19:29:40.467 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.462+0000 7f41893f1e40 1 -- 172.21.15.167:0/1416833216 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:77 5.1d 5:b8374eb4:::zone_info.9a9df965-6612-45d4-a45f-6e3c889bacc2:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32) v8 -- 0x560d8c3d5260 con 0x560d8c398930 2023-10-05T19:29:40.467 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.462+0000 7f4187727700 1 -- 172.21.15.167:0/1416833216 <== osd.1 v2:172.21.15.167:6800/30035 27 ==== osd_op_reply(77 zone_info.9a9df965-6612-45d4-a45f-6e3c889bacc2 [read 0~874 out=874b] v0'0 uv2 ondisk = 0) v8 ==== 190+0+874 (crc 0 0 0) 0x7f417802f080 con 0x560d8c398930 2023-10-05T19:29:40.467 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.462+0000 7f41893f1e40 20 rados_obj.operate() r=0 bl.length=874 2023-10-05T19:29:40.467 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.462+0000 7f41893f1e40 20 rados->read ofs=0 len=0 2023-10-05T19:29:40.467 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.462+0000 7f41893f1e40 1 -- 172.21.15.167:0/1416833216 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:78 5.f 5:f4c53578:::zonegroups_names.default:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32) v8 -- 0x560d8c3d52c0 con 0x560d8c398930 2023-10-05T19:29:40.468 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.462+0000 7f4187727700 1 -- 172.21.15.167:0/1416833216 <== osd.1 v2:172.21.15.167:6800/30035 28 ==== osd_op_reply(78 zonegroups_names.default [read 0~46 out=46b] v0'0 uv2 ondisk = 0) v8 ==== 168+0+46 (crc 0 0 0) 0x7f417802f080 con 0x560d8c398930 2023-10-05T19:29:40.468 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.462+0000 7f41893f1e40 20 rados_obj.operate() r=0 bl.length=46 2023-10-05T19:29:40.468 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.462+0000 7f41893f1e40 20 rados->read ofs=0 len=0 2023-10-05T19:29:40.468 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.462+0000 7f41893f1e40 1 -- 172.21.15.167:0/1416833216 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:79 5.c 5:30d299a5:::zonegroup_info.4f1f719d-9648-417b-b10c-e5d939825419:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32) v8 -- 0x560d8c3d5290 con 0x7f417c00d8e0 2023-10-05T19:29:40.469 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.466+0000 7f4187727700 1 -- 172.21.15.167:0/1416833216 <== osd.0 v2:172.21.15.118:6800/27690 20 ==== osd_op_reply(79 zonegroup_info.4f1f719d-9648-417b-b10c-e5d939825419 [read 0~362 out=362b] v0'0 uv1 ondisk = 0) v8 ==== 195+0+362 (crc 0 0 0) 0x7f417802f080 con 0x7f417c00d8e0 2023-10-05T19:29:40.470 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.466+0000 7f41893f1e40 20 rados_obj.operate() r=0 bl.length=362 2023-10-05T19:29:40.470 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.466+0000 7f41893f1e40 20 rados->read ofs=0 len=0 2023-10-05T19:29:40.470 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.466+0000 7f41893f1e40 1 -- 172.21.15.167:0/1416833216 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:80 5.3 5:c0ad8cbd:::region_map:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32) v8 -- 0x560d8c3d56d0 con 0x7f417c00d8e0 2023-10-05T19:29:40.470 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.466+0000 7f4187727700 1 -- 172.21.15.167:0/1416833216 <== osd.0 v2:172.21.15.118:6800/27690 21 ==== osd_op_reply(80 region_map [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 ==== 154+0+0 (crc 0 0 0) 0x7f417802f080 con 0x7f417c00d8e0 2023-10-05T19:29:40.470 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.466+0000 7f41893f1e40 20 rados_obj.operate() r=-2 bl.length=0 2023-10-05T19:29:40.470 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.466+0000 7f41893f1e40 20 zone default found 2023-10-05T19:29:40.470 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.466+0000 7f41893f1e40 4 Realm: () 2023-10-05T19:29:40.470 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.466+0000 7f41893f1e40 4 ZoneGroup: default (4f1f719d-9648-417b-b10c-e5d939825419) 2023-10-05T19:29:40.471 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.466+0000 7f41893f1e40 4 Zone: default (9a9df965-6612-45d4-a45f-6e3c889bacc2) 2023-10-05T19:29:40.471 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.466+0000 7f41893f1e40 10 cannot find current period zonegroup using local zonegroup configuration 2023-10-05T19:29:40.471 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.466+0000 7f41893f1e40 20 zonegroup default 2023-10-05T19:29:40.471 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.466+0000 7f41893f1e40 20 rados->read ofs=0 len=0 2023-10-05T19:29:40.471 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.466+0000 7f41893f1e40 1 -- 172.21.15.167:0/1416833216 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:81 5.3 5:c52100b6:::period_config.default:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32) v8 -- 0x560d8c3d56d0 con 0x7f417c00d8e0 2023-10-05T19:29:40.472 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.466+0000 7f4187727700 1 -- 172.21.15.167:0/1416833216 <== osd.0 v2:172.21.15.118:6800/27690 22 ==== osd_op_reply(81 period_config.default [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 ==== 165+0+0 (crc 0 0 0) 0x7f417802f080 con 0x7f417c00d8e0 2023-10-05T19:29:40.473 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.466+0000 7f41893f1e40 20 rados_obj.operate() r=-2 bl.length=0 2023-10-05T19:29:40.473 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.466+0000 7f41893f1e40 20 rados->read ofs=0 len=0 2023-10-05T19:29:40.473 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.466+0000 7f41893f1e40 1 -- 172.21.15.167:0/1416833216 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:82 6.2 6:4347d321:::bucket.sync-source-hints.:head [call version.read in=11b,read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32) v8 -- 0x560d8c3d6e30 con 0x560d8c3d1940 2023-10-05T19:29:40.473 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.466+0000 7f4186f26700 1 -- 172.21.15.167:0/1416833216 <== osd.3 v2:172.21.15.167:6808/30036 20 ==== osd_op_reply(82 bucket.sync-source-hints. [call,read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 ==== 211+0+0 (crc 0 0 0) 0x7f418003adb0 con 0x560d8c3d1940 2023-10-05T19:29:40.473 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.466+0000 7f41893f1e40 20 rados_obj.operate() r=-2 bl.length=0 2023-10-05T19:29:40.473 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.466+0000 7f41893f1e40 20 rados->read ofs=0 len=0 2023-10-05T19:29:40.473 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.466+0000 7f41893f1e40 1 -- 172.21.15.167:0/1416833216 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:83 6.b 6:d467b91b:::bucket.sync-target-hints.:head [call version.read in=11b,read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32) v8 -- 0x560d8c3d71e0 con 0x560d8c3d1940 2023-10-05T19:29:40.473 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.466+0000 7f4186f26700 1 -- 172.21.15.167:0/1416833216 <== osd.3 v2:172.21.15.167:6808/30036 21 ==== osd_op_reply(83 bucket.sync-target-hints. [call,read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 ==== 211+0+0 (crc 0 0 0) 0x7f418005f040 con 0x560d8c3d1940 2023-10-05T19:29:40.473 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.466+0000 7f41893f1e40 20 rados_obj.operate() r=-2 bl.length=0 2023-10-05T19:29:40.474 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.466+0000 7f41893f1e40 20 started sync module instance, tier type = 2023-10-05T19:29:40.474 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.466+0000 7f41893f1e40 20 started zone id=9a9df965-6612-45d4-a45f-6e3c889bacc2 (name=default) with tier type = 2023-10-05T19:29:40.474 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.466+0000 7f41893f1e40 1 -- 172.21.15.167:0/1416833216 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:84 7.1f 7:f95f44c2:::notify.0:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e32) v8 -- 0x560d8c3d64f0 con 0x7f418001a3c0 2023-10-05T19:29:40.475 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.470+0000 7f4186725700 1 -- 172.21.15.167:0/1416833216 <== osd.2 v2:172.21.15.167:6816/30037 13 ==== osd_op_reply(84 notify.0 [create] v32'249 uv249 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f417c006070 con 0x7f418001a3c0 2023-10-05T19:29:40.476 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.470+0000 7f41893f1e40 1 -- 172.21.15.167:0/1416833216 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:85 7.1f 7:f95f44c2:::notify.0:head [watch watch cookie 94616187398432] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e32) v8 -- 0x560d8c3d0c10 con 0x7f418001a3c0 2023-10-05T19:29:40.476 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.470+0000 7f41893f1e40 1 -- 172.21.15.167:0/1416833216 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:86 7.0 7:05bf5b68:::notify.1:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e32) v8 -- 0x560d8c3d7fb0 con 0x560d8c398930 2023-10-05T19:29:40.478 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.474+0000 7f4186725700 1 -- 172.21.15.167:0/1416833216 <== osd.2 v2:172.21.15.167:6816/30037 14 ==== osd_op_reply(85 notify.0 [watch watch cookie 94616187398432] v32'250 uv249 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f417c006070 con 0x7f418001a3c0 2023-10-05T19:29:40.479 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.474+0000 7f4187727700 1 -- 172.21.15.167:0/1416833216 <== osd.1 v2:172.21.15.167:6800/30035 29 ==== osd_op_reply(86 notify.1 [create] v32'249 uv249 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f417802f080 con 0x560d8c398930 2023-10-05T19:29:40.479 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.474+0000 7f41893f1e40 1 -- 172.21.15.167:0/1416833216 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:87 7.0 7:05bf5b68:::notify.1:head [watch watch cookie 94616187401616] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e32) v8 -- 0x560d8c3d83b0 con 0x560d8c398930 2023-10-05T19:29:40.479 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.474+0000 7f41893f1e40 1 -- 172.21.15.167:0/1416833216 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:88 7.15 7:a93a5511:::notify.2:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e32) v8 -- 0x560d8c3d8c50 con 0x560d8c3d1940 2023-10-05T19:29:40.482 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.478+0000 7f4186f26700 1 -- 172.21.15.167:0/1416833216 <== osd.3 v2:172.21.15.167:6808/30036 22 ==== osd_op_reply(88 notify.2 [create] v32'249 uv249 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f418005f040 con 0x560d8c3d1940 2023-10-05T19:29:40.482 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.478+0000 7f41893f1e40 1 -- 172.21.15.167:0/1416833216 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:89 7.15 7:a93a5511:::notify.2:head [watch watch cookie 94616187405520] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e32) v8 -- 0x560d8c3d94f0 con 0x560d8c3d1940 2023-10-05T19:29:40.482 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.478+0000 7f41893f1e40 1 -- 172.21.15.167:0/1416833216 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:90 7.e 7:7759931f:::notify.3:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e32) v8 -- 0x560d8c3d9d90 con 0x7f418001a3c0 2023-10-05T19:29:40.483 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.478+0000 7f4187727700 1 -- 172.21.15.167:0/1416833216 <== osd.1 v2:172.21.15.167:6800/30035 30 ==== osd_op_reply(87 notify.1 [watch watch cookie 94616187401616] v32'250 uv249 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f4178050040 con 0x560d8c398930 2023-10-05T19:29:40.484 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.478+0000 7f4186725700 1 -- 172.21.15.167:0/1416833216 <== osd.2 v2:172.21.15.167:6816/30037 15 ==== osd_op_reply(90 notify.3 [create] v32'249 uv249 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f417c006070 con 0x7f418001a3c0 2023-10-05T19:29:40.484 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.478+0000 7f41893f1e40 1 -- 172.21.15.167:0/1416833216 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:91 7.e 7:7759931f:::notify.3:head [watch watch cookie 94616187402720] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e32) v8 -- 0x560d8c3d99c0 con 0x7f418001a3c0 2023-10-05T19:29:40.484 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.478+0000 7f41893f1e40 1 -- 172.21.15.167:0/1416833216 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:92 7.d 7:b4812045:::notify.4:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e32) v8 -- 0x560d8c3dae10 con 0x560d8c398930 2023-10-05T19:29:40.485 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.482+0000 7f4186f26700 1 -- 172.21.15.167:0/1416833216 <== osd.3 v2:172.21.15.167:6808/30036 23 ==== osd_op_reply(89 notify.2 [watch watch cookie 94616187405520] v32'250 uv249 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f418005f040 con 0x560d8c3d1940 2023-10-05T19:29:40.486 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.482+0000 7f4186725700 1 -- 172.21.15.167:0/1416833216 <== osd.2 v2:172.21.15.167:6816/30037 16 ==== osd_op_reply(91 notify.3 [watch watch cookie 94616187402720] v32'250 uv249 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f417c006070 con 0x7f418001a3c0 2023-10-05T19:29:40.487 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.482+0000 7f4187727700 1 -- 172.21.15.167:0/1416833216 <== osd.1 v2:172.21.15.167:6800/30035 31 ==== osd_op_reply(92 notify.4 [create] v32'249 uv249 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f4178050040 con 0x560d8c398930 2023-10-05T19:29:40.487 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.482+0000 7f41893f1e40 1 -- 172.21.15.167:0/1416833216 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:93 7.d 7:b4812045:::notify.4:head [watch watch cookie 94616187412800] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e32) v8 -- 0x560d8c3da8e0 con 0x560d8c398930 2023-10-05T19:29:40.487 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.482+0000 7f41893f1e40 1 -- 172.21.15.167:0/1416833216 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:94 7.3 7:c609908c:::notify.5:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e32) v8 -- 0x560d8c3db700 con 0x7f417c00d8e0 2023-10-05T19:29:40.490 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.486+0000 7f4187727700 1 -- 172.21.15.167:0/1416833216 <== osd.0 v2:172.21.15.118:6800/27690 23 ==== osd_op_reply(94 notify.5 [create] v32'249 uv249 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f4178050040 con 0x7f417c00d8e0 2023-10-05T19:29:40.490 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.486+0000 7f41893f1e40 1 -- 172.21.15.167:0/1416833216 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:95 7.3 7:c609908c:::notify.5:head [watch watch cookie 94616187417024] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e32) v8 -- 0x560d8c3dbb00 con 0x7f417c00d8e0 2023-10-05T19:29:40.490 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.486+0000 7f41893f1e40 1 -- 172.21.15.167:0/1416833216 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:96 7.14 7:2b04a3e9:::notify.6:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e32) v8 -- 0x560d8c3dc550 con 0x7f417c00d8e0 2023-10-05T19:29:40.490 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.486+0000 7f4187727700 1 -- 172.21.15.167:0/1416833216 <== osd.1 v2:172.21.15.167:6800/30035 32 ==== osd_op_reply(93 notify.4 [watch watch cookie 94616187412800] v32'250 uv249 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f4178050040 con 0x560d8c398930 2023-10-05T19:29:40.493 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.490+0000 7f4187727700 1 -- 172.21.15.167:0/1416833216 <== osd.0 v2:172.21.15.118:6800/27690 24 ==== osd_op_reply(95 notify.5 [watch watch cookie 94616187417024] v32'250 uv249 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f4178050040 con 0x7f417c00d8e0 2023-10-05T19:29:40.493 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.490+0000 7f4187727700 1 -- 172.21.15.167:0/1416833216 <== osd.0 v2:172.21.15.118:6800/27690 25 ==== osd_op_reply(96 notify.6 [create] v32'249 uv249 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f4178050040 con 0x7f417c00d8e0 2023-10-05T19:29:40.494 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.490+0000 7f41893f1e40 1 -- 172.21.15.167:0/1416833216 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:97 7.14 7:2b04a3e9:::notify.6:head [watch watch cookie 94616187408384] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e32) v8 -- 0x560d8c3dbb00 con 0x7f417c00d8e0 2023-10-05T19:29:40.494 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.490+0000 7f41893f1e40 1 -- 172.21.15.167:0/1416833216 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:98 7.9 7:93e5b521:::notify.7:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e32) v8 -- 0x560d8c3dc950 con 0x7f417c00d8e0 2023-10-05T19:29:40.496 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.490+0000 7f4187727700 1 -- 172.21.15.167:0/1416833216 <== osd.0 v2:172.21.15.118:6800/27690 26 ==== osd_op_reply(97 notify.6 [watch watch cookie 94616187408384] v32'250 uv249 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f4178050040 con 0x7f417c00d8e0 2023-10-05T19:29:40.496 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.490+0000 7f4187727700 1 -- 172.21.15.167:0/1416833216 <== osd.0 v2:172.21.15.118:6800/27690 27 ==== osd_op_reply(98 notify.7 [create] v32'249 uv249 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f4178050040 con 0x7f417c00d8e0 2023-10-05T19:29:40.497 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.490+0000 7f41893f1e40 1 -- 172.21.15.167:0/1416833216 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:99 7.9 7:93e5b521:::notify.7:head [watch watch cookie 94616187421728] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e32) v8 -- 0x560d8c3dbb00 con 0x7f417c00d8e0 2023-10-05T19:29:40.497 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.490+0000 7f41893f1e40 20 add_watcher() i=0 2023-10-05T19:29:40.497 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.490+0000 7f41893f1e40 20 add_watcher() i=1 2023-10-05T19:29:40.497 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.490+0000 7f41893f1e40 20 add_watcher() i=2 2023-10-05T19:29:40.497 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.490+0000 7f41893f1e40 20 add_watcher() i=3 2023-10-05T19:29:40.497 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.490+0000 7f41893f1e40 20 add_watcher() i=4 2023-10-05T19:29:40.497 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.490+0000 7f41893f1e40 20 add_watcher() i=5 2023-10-05T19:29:40.497 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.490+0000 7f41893f1e40 20 add_watcher() i=6 2023-10-05T19:29:40.498 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.494+0000 7f4187727700 1 -- 172.21.15.167:0/1416833216 <== osd.0 v2:172.21.15.118:6800/27690 28 ==== osd_op_reply(99 notify.7 [watch watch cookie 94616187421728] v32'250 uv249 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f4178050040 con 0x7f417c00d8e0 2023-10-05T19:29:40.499 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.494+0000 7f41893f1e40 20 add_watcher() i=7 2023-10-05T19:29:40.499 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.494+0000 7f41893f1e40 2 all 8 watchers are set, enabling cache 2023-10-05T19:29:40.499 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.494+0000 7f41893f1e40 1 -- 172.21.15.167:0/1416833216 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:100 6.4 6:22d26bf9:::data_loggenerations_metadata:head [call version.check_conds in=74b,call version.read in=11b,read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32) v8 -- 0x560d8c3dfcf0 con 0x560d8c398930 2023-10-05T19:29:40.499 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.494+0000 7f4187727700 1 -- 172.21.15.167:0/1416833216 <== osd.1 v2:172.21.15.167:6800/30035 33 ==== osd_op_reply(100 data_loggenerations_metadata [call,call out=48b,read 0~28 out=28b] v0'0 uv1 ondisk = 0) v8 ==== 256+0+76 (crc 0 0 0) 0x7f4178050040 con 0x560d8c398930 2023-10-05T19:29:40.500 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.494+0000 7f41893f1e40 1 -- 172.21.15.167:0/1416833216 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:101 6.4 6:22d26bf9:::data_loggenerations_metadata:head [watch watch cookie 94616187418592] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e32) v8 -- 0x560d8c3dd1b0 con 0x560d8c398930 2023-10-05T19:29:40.502 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.498+0000 7f4187727700 1 -- 172.21.15.167:0/1416833216 <== osd.1 v2:172.21.15.167:6800/30035 34 ==== osd_op_reply(101 data_loggenerations_metadata [watch watch cookie 94616187418592] v32'239 uv1 ondisk = 0) v8 ==== 172+0+0 (crc 0 0 0) 0x7f4178050040 con 0x560d8c398930 2023-10-05T19:29:40.502 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.498+0000 7f4157fe7700 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2023-10-05T19:29:40.502 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.498+0000 7f41893f1e40 20 check_secure_mon_conn(): auth registy supported: methods=[2] modes=[2,1] 2023-10-05T19:29:40.502 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.498+0000 7f41893f1e40 20 check_secure_mon_conn(): mode 1 is insecure 2023-10-05T19:29:40.503 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.498+0000 7f41893f1e40 1 Processor -- start 2023-10-05T19:29:40.503 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.498+0000 7f41893f1e40 1 -- start start 2023-10-05T19:29:40.503 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.498+0000 7f41893f1e40 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x560d8c3ea3b0 0x560d8c3ea780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-05T19:29:40.503 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.498+0000 7f41893f1e40 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x560d8c3eacc0 con 0x560d8c3ea3b0 2023-10-05T19:29:40.503 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.498+0000 7f4186725700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x560d8c3ea3b0 0x560d8c3ea780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-05T19:29:40.504 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.498+0000 7f4186725700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x560d8c3ea3b0 0x560d8c3ea780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.118:3300/0 says I am v2:172.21.15.167:56832/0 (socket says 172.21.15.167:56832) 2023-10-05T19:29:40.504 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.498+0000 7f4186725700 1 -- 172.21.15.167:0/201499680 learned_addr learned my addr 172.21.15.167:0/201499680 (peer_addr_for_me v2:172.21.15.167:0/0) 2023-10-05T19:29:40.505 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.498+0000 7f4186725700 1 -- 172.21.15.167:0/201499680 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x560d8c3eae00 con 0x560d8c3ea3b0 2023-10-05T19:29:40.505 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.498+0000 7f4186725700 1 --2- 172.21.15.167:0/201499680 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x560d8c3ea3b0 0x560d8c3ea780 secure :-1 s=READY pgs=636 cs=0 l=1 rev1=1 crypto rx=0x560d8c3cf0a0 tx=0x7f417c006000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c57c84bd4dee70b1 server_cookie=0 in_seq=0 out_seq=0 2023-10-05T19:29:40.505 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.498+0000 7f41577e6700 1 -- 172.21.15.167:0/201499680 <== mon.0 v2:172.21.15.118:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f417c023040 con 0x560d8c3ea3b0 2023-10-05T19:29:40.505 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.502+0000 7f41577e6700 1 -- 172.21.15.167:0/201499680 <== mon.0 v2:172.21.15.118:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f417c024040 con 0x560d8c3ea3b0 2023-10-05T19:29:40.505 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.502+0000 7f41577e6700 1 -- 172.21.15.167:0/201499680 <== mon.0 v2:172.21.15.118:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f417c013040 con 0x560d8c3ea3b0 2023-10-05T19:29:40.506 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.502+0000 7f41893f1e40 1 -- 172.21.15.167:0/201499680 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x560d8c3ea3b0 msgr2=0x560d8c3ea780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-05T19:29:40.506 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.502+0000 7f41893f1e40 1 --2- 172.21.15.167:0/201499680 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x560d8c3ea3b0 0x560d8c3ea780 secure :-1 s=READY pgs=636 cs=0 l=1 rev1=1 crypto rx=0x560d8c3cf0a0 tx=0x7f417c006000 comp rx=0 tx=0).stop 2023-10-05T19:29:40.506 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.502+0000 7f41893f1e40 1 -- 172.21.15.167:0/201499680 shutdown_connections 2023-10-05T19:29:40.506 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.502+0000 7f41893f1e40 1 --2- 172.21.15.167:0/201499680 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x560d8c3ea3b0 0x560d8c3ea780 unknown :-1 s=CLOSED pgs=636 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-05T19:29:40.507 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.502+0000 7f41893f1e40 1 -- 172.21.15.167:0/201499680 >> 172.21.15.167:0/201499680 conn(0x560d8c3e5770 msgr2=0x560d8c3e38c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-05T19:29:40.507 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.502+0000 7f41893f1e40 1 -- 172.21.15.167:0/201499680 shutdown_connections 2023-10-05T19:29:40.507 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.502+0000 7f41893f1e40 1 -- 172.21.15.167:0/201499680 wait complete. 2023-10-05T19:29:40.507 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.502+0000 7f41893f1e40 1 Processor -- start 2023-10-05T19:29:40.507 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.502+0000 7f41893f1e40 1 -- start start 2023-10-05T19:29:40.508 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.502+0000 7f41893f1e40 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x560d8c3ea3b0 0x560d8c3eff10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-05T19:29:40.508 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.502+0000 7f41893f1e40 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x560d8c3f0f10 con 0x560d8c3ea3b0 2023-10-05T19:29:40.508 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.502+0000 7f4186725700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x560d8c3ea3b0 0x560d8c3eff10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-05T19:29:40.508 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.502+0000 7f4186725700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x560d8c3ea3b0 0x560d8c3eff10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.118:3300/0 says I am v2:172.21.15.167:56846/0 (socket says 172.21.15.167:56846) 2023-10-05T19:29:40.508 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.502+0000 7f4186725700 1 -- 172.21.15.167:0/2303268290 learned_addr learned my addr 172.21.15.167:0/2303268290 (peer_addr_for_me v2:172.21.15.167:0/0) 2023-10-05T19:29:40.508 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.502+0000 7f4186725700 1 -- 172.21.15.167:0/2303268290 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x560d8c2e2bd0 con 0x560d8c3ea3b0 2023-10-05T19:29:40.509 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.502+0000 7f4186725700 1 --2- 172.21.15.167:0/2303268290 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x560d8c3ea3b0 0x560d8c3eff10 secure :-1 s=READY pgs=637 cs=0 l=1 rev1=1 crypto rx=0x7f417c000fd0 tx=0x7f417c030000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-05T19:29:40.509 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.506+0000 7f4155fe3700 1 -- 172.21.15.167:0/2303268290 <== mon.0 v2:172.21.15.118:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f417c023040 con 0x560d8c3ea3b0 2023-10-05T19:29:40.510 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.506+0000 7f4155fe3700 1 -- 172.21.15.167:0/2303268290 <== mon.0 v2:172.21.15.118:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f417c03a420 con 0x560d8c3ea3b0 2023-10-05T19:29:40.510 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.506+0000 7f41893f1e40 1 -- 172.21.15.167:0/2303268290 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x560d8c3f0510 con 0x560d8c3ea3b0 2023-10-05T19:29:40.510 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.506+0000 7f4155fe3700 1 -- 172.21.15.167:0/2303268290 <== mon.0 v2:172.21.15.118:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f417c03a590 con 0x560d8c3ea3b0 2023-10-05T19:29:40.510 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.506+0000 7f41893f1e40 1 -- 172.21.15.167:0/2303268290 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x560d8c3f0950 con 0x560d8c3ea3b0 2023-10-05T19:29:40.511 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.506+0000 7f4155fe3700 1 -- 172.21.15.167:0/2303268290 <== mon.0 v2:172.21.15.118:3300/0 4 ==== mgrmap(e 5) v1 ==== 40329+0+0 (secure 0 0 0) 0x7f417c03a6f0 con 0x560d8c3ea3b0 2023-10-05T19:29:40.511 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.506+0000 7f4155fe3700 1 --2- 172.21.15.167:0/2303268290 >> [v2:172.21.15.167:6824/29889,v1:172.21.15.167:6825/29889] conn(0x7f40cc030c50 0x7f40cc033100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-05T19:29:40.511 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.506+0000 7f4155fe3700 1 -- 172.21.15.167:0/2303268290 <== mon.0 v2:172.21.15.118:3300/0 5 ==== osd_map(32..32 src has 1..32) v4 ==== 6260+0+0 (secure 0 0 0) 0x7f417c04fd30 con 0x560d8c3ea3b0 2023-10-05T19:29:40.511 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.506+0000 7f4187727700 1 --2- 172.21.15.167:0/2303268290 >> [v2:172.21.15.167:6824/29889,v1:172.21.15.167:6825/29889] conn(0x7f40cc030c50 0x7f40cc033100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-05T19:29:40.511 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.506+0000 7f41893f1e40 5 note: GC not initialized 2023-10-05T19:29:40.511 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.506+0000 7f4187727700 1 --2- 172.21.15.167:0/2303268290 >> [v2:172.21.15.167:6824/29889,v1:172.21.15.167:6825/29889] conn(0x7f40cc030c50 0x7f40cc033100 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x560d8c3dbd30 tx=0x7f4178000f90 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-05T19:29:40.513 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.510+0000 7f40c6ffd700 20 reqs_thread_entry: start 2023-10-05T19:29:40.513 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.510+0000 7f41893f1e40 20 init_complete bucket index max shards: 11 2023-10-05T19:29:40.514 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.510+0000 7f40c4ff9700 20 reqs_thread_entry: start 2023-10-05T19:29:40.515 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.510+0000 7f41893f1e40 10 rgw notify: Started notification manager with: 1 workers 2023-10-05T19:29:40.515 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.510+0000 7f40b6ffd700 20 rgw notify: INFO: next queues processing will happen at: Thu Oct 5 19:30:10 2023 2023-10-05T19:29:40.515 INFO:teuthology.orchestra.run.smithi167.stderr: 2023-10-05T19:29:40.515 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.510+0000 7f40b67fc700 20 reqs_thread_entry: start 2023-10-05T19:29:40.515 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.510+0000 7f41893f1e40 20 get_system_obj_state: rctx=0x7fffc7975e58 obj=default.rgw.meta:users.uid:bud state=0x560d8c26f0b0 s->prefetch_data=0 2023-10-05T19:29:40.515 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.510+0000 7f41893f1e40 10 cache get: name=default.rgw.meta+users.uid+bud : miss 2023-10-05T19:29:40.515 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.510+0000 7f41893f1e40 1 -- 172.21.15.167:0/1416833216 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:102 8.4 8:267425a8:users.uid::bud:head [call version.read in=11b,getxattrs,stat] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32) v8 -- 0x560d8c458810 con 0x7f418001a3c0 2023-10-05T19:29:40.516 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.510+0000 7f4186725700 1 -- 172.21.15.167:0/1416833216 <== osd.2 v2:172.21.15.167:6816/30037 17 ==== osd_op_reply(102 bud [call out=48b,getxattrs out=75b,stat out=16b] v0'0 uv2 ondisk = 0) v8 ==== 231+0+139 (crc 0 0 0) 0x7f417c006070 con 0x7f418001a3c0 2023-10-05T19:29:40.517 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.510+0000 7f41893f1e40 10 cache put: name=default.rgw.meta+users.uid+bud info.flags=0x16 2023-10-05T19:29:40.517 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.510+0000 7f41893f1e40 10 adding default.rgw.meta+users.uid+bud to cache LRU end 2023-10-05T19:29:40.517 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.510+0000 7f41893f1e40 10 updating xattr: name=ceph.objclass.version bl.length()=42 2023-10-05T19:29:40.517 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.510+0000 7f41893f1e40 20 get_system_obj_state: s->obj_tag was set empty 2023-10-05T19:29:40.517 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.510+0000 7f41893f1e40 20 Read xattr: user.rgw.idtag 2023-10-05T19:29:40.517 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.510+0000 7f41893f1e40 10 cache get: name=default.rgw.meta+users.uid+bud : type miss (requested=0x13, cached=0x16) 2023-10-05T19:29:40.517 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.510+0000 7f41893f1e40 20 rados->read ofs=0 len=0 2023-10-05T19:29:40.517 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.510+0000 7f41893f1e40 1 -- 172.21.15.167:0/1416833216 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:103 8.4 8:267425a8:users.uid::bud:head [call version.check_conds in=122b,call version.read in=11b,read 0~0,getxattrs] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32) v8 -- 0x560d8c45b2a0 con 0x7f418001a3c0 2023-10-05T19:29:40.517 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.510+0000 7f4186725700 1 -- 172.21.15.167:0/1416833216 <== osd.2 v2:172.21.15.167:6816/30037 18 ==== osd_op_reply(103 bud [call,call out=48b,read 0~346 out=346b,getxattrs out=75b] v0'0 uv2 ondisk = 0) v8 ==== 273+0+469 (crc 0 0 0) 0x7f417c006070 con 0x7f418001a3c0 2023-10-05T19:29:40.517 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.510+0000 7f41893f1e40 20 rados_obj.operate() r=0 bl.length=346 2023-10-05T19:29:40.518 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.510+0000 7f41893f1e40 10 cache put: name=default.rgw.meta+users.uid+bud info.flags=0x13 2023-10-05T19:29:40.518 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.510+0000 7f41893f1e40 10 moving default.rgw.meta+users.uid+bud to cache LRU end 2023-10-05T19:29:40.518 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.510+0000 7f41893f1e40 10 updating xattr: name=ceph.objclass.version bl.length()=42 2023-10-05T19:29:40.518 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.514+0000 7f41893f1e40 1 -- 172.21.15.167:0/1416833216 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:104 8.e 8:75f48285:users.uid::fud:head [create,call version.set in=58b,writefull 0~181 in=181b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e32) v8 -- 0x560d8c45cdc0 con 0x7f417c00d8e0 2023-10-05T19:29:40.519 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.514+0000 7f4187727700 1 -- 172.21.15.167:0/1416833216 <== osd.0 v2:172.21.15.118:6800/27690 29 ==== osd_op_reply(104 fud [create,call,writefull 0~181] v32'6 uv5 ondisk = -17 ((17) File exists)) v8 ==== 231+0+0 (crc 0 0 0) 0x7f4178050040 con 0x7f417c00d8e0 2023-10-05T19:29:40.520 INFO:teuthology.orchestra.run.smithi167.stderr:could not rename user: unable to rename user, user name given by --new-uid already exists. to overwrite this user, add --yes-i-really-mean-it 2023-10-05T19:29:40.520 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.514+0000 7f41893f1e40 1 -- 172.21.15.167:0/1416833216 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:105 7.1f 7:f95f44c2:::notify.0:head [watch unwatch cookie 94616187398432] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e32) v8 -- 0x560d8c3ad9a0 con 0x7f418001a3c0 2023-10-05T19:29:40.523 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.518+0000 7f4186725700 1 -- 172.21.15.167:0/1416833216 <== osd.2 v2:172.21.15.167:6816/30037 19 ==== osd_op_reply(105 notify.0 [watch unwatch cookie 94616187398432] v32'251 uv249 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f417c006070 con 0x7f418001a3c0 2023-10-05T19:29:40.523 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.518+0000 7f41893f1e40 20 remove_watcher() i=0 2023-10-05T19:29:40.523 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.518+0000 7f41893f1e40 2 removed watcher, disabling cache 2023-10-05T19:29:40.524 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.518+0000 7f41893f1e40 1 -- 172.21.15.167:0/1416833216 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:106 7.0 7:05bf5b68:::notify.1:head [watch unwatch cookie 94616187401616] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e32) v8 -- 0x560d8c3ad9a0 con 0x560d8c398930 2023-10-05T19:29:40.526 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.522+0000 7f4187727700 1 -- 172.21.15.167:0/1416833216 <== osd.1 v2:172.21.15.167:6800/30035 35 ==== osd_op_reply(106 notify.1 [watch unwatch cookie 94616187401616] v32'251 uv249 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f4178050040 con 0x560d8c398930 2023-10-05T19:29:40.526 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.522+0000 7f41893f1e40 20 remove_watcher() i=1 2023-10-05T19:29:40.526 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.522+0000 7f41893f1e40 1 -- 172.21.15.167:0/1416833216 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:107 7.15 7:a93a5511:::notify.2:head [watch unwatch cookie 94616187405520] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e32) v8 -- 0x560d8c3ad9a0 con 0x560d8c3d1940 2023-10-05T19:29:40.529 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.526+0000 7f4186f26700 1 -- 172.21.15.167:0/1416833216 <== osd.3 v2:172.21.15.167:6808/30036 24 ==== osd_op_reply(107 notify.2 [watch unwatch cookie 94616187405520] v32'251 uv249 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f418005f040 con 0x560d8c3d1940 2023-10-05T19:29:40.529 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.526+0000 7f41893f1e40 20 remove_watcher() i=2 2023-10-05T19:29:40.530 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.526+0000 7f41893f1e40 1 -- 172.21.15.167:0/1416833216 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:108 7.e 7:7759931f:::notify.3:head [watch unwatch cookie 94616187402720] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e32) v8 -- 0x560d8c3d7590 con 0x7f418001a3c0 2023-10-05T19:29:40.532 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.526+0000 7f4186725700 1 -- 172.21.15.167:0/1416833216 <== osd.2 v2:172.21.15.167:6816/30037 20 ==== osd_op_reply(108 notify.3 [watch unwatch cookie 94616187402720] v32'251 uv249 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f417c006070 con 0x7f418001a3c0 2023-10-05T19:29:40.533 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.530+0000 7f41893f1e40 20 remove_watcher() i=3 2023-10-05T19:29:40.533 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.530+0000 7f41893f1e40 1 -- 172.21.15.167:0/1416833216 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:109 7.d 7:b4812045:::notify.4:head [watch unwatch cookie 94616187412800] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e32) v8 -- 0x560d8c3d7a80 con 0x560d8c398930 2023-10-05T19:29:40.536 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.530+0000 7f4187727700 1 -- 172.21.15.167:0/1416833216 <== osd.1 v2:172.21.15.167:6800/30035 36 ==== osd_op_reply(109 notify.4 [watch unwatch cookie 94616187412800] v32'251 uv249 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f4178050040 con 0x560d8c398930 2023-10-05T19:29:40.536 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.530+0000 7f41893f1e40 20 remove_watcher() i=4 2023-10-05T19:29:40.536 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.530+0000 7f41893f1e40 1 -- 172.21.15.167:0/1416833216 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:110 7.3 7:c609908c:::notify.5:head [watch unwatch cookie 94616187417024] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e32) v8 -- 0x560d8c3d7a80 con 0x7f417c00d8e0 2023-10-05T19:29:40.539 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.534+0000 7f4187727700 1 -- 172.21.15.167:0/1416833216 <== osd.0 v2:172.21.15.118:6800/27690 30 ==== osd_op_reply(110 notify.5 [watch unwatch cookie 94616187417024] v32'251 uv249 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f4178050040 con 0x7f417c00d8e0 2023-10-05T19:29:40.539 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.534+0000 7f41893f1e40 20 remove_watcher() i=5 2023-10-05T19:29:40.539 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.534+0000 7f41893f1e40 1 -- 172.21.15.167:0/1416833216 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:111 7.14 7:2b04a3e9:::notify.6:head [watch unwatch cookie 94616187408384] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e32) v8 -- 0x560d8c3d7a80 con 0x7f417c00d8e0 2023-10-05T19:29:40.542 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.538+0000 7f4187727700 1 -- 172.21.15.167:0/1416833216 <== osd.0 v2:172.21.15.118:6800/27690 31 ==== osd_op_reply(111 notify.6 [watch unwatch cookie 94616187408384] v32'251 uv249 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f4178050040 con 0x7f417c00d8e0 2023-10-05T19:29:40.542 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.538+0000 7f41893f1e40 20 remove_watcher() i=6 2023-10-05T19:29:40.542 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.538+0000 7f41893f1e40 1 -- 172.21.15.167:0/1416833216 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:112 7.9 7:93e5b521:::notify.7:head [watch unwatch cookie 94616187421728] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e32) v8 -- 0x560d8c3d7a80 con 0x7f417c00d8e0 2023-10-05T19:29:40.545 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.542+0000 7f4187727700 1 -- 172.21.15.167:0/1416833216 <== osd.0 v2:172.21.15.118:6800/27690 32 ==== osd_op_reply(112 notify.7 [watch unwatch cookie 94616187421728] v32'251 uv249 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f4178050040 con 0x7f417c00d8e0 2023-10-05T19:29:40.545 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.542+0000 7f41893f1e40 20 remove_watcher() i=7 2023-10-05T19:29:40.546 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.542+0000 7f4157fe7700 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2023-10-05T19:29:40.547 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.542+0000 7f41893f1e40 1 -- 172.21.15.167:0/1416833216 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:113 6.4 6:22d26bf9:::data_loggenerations_metadata:head [watch unwatch cookie 94616187418592] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e32) v8 -- 0x560d8c3d7a80 con 0x560d8c398930 2023-10-05T19:29:40.551 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.546+0000 7f4187727700 1 -- 172.21.15.167:0/1416833216 <== osd.1 v2:172.21.15.167:6800/30035 37 ==== osd_op_reply(113 data_loggenerations_metadata [watch unwatch cookie 94616187418592] v32'240 uv1 ondisk = 0) v8 ==== 172+0+0 (crc 0 0 0) 0x7f4178050040 con 0x560d8c398930 2023-10-05T19:29:40.551 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.546+0000 7f41893f1e40 1 -- 172.21.15.167:0/1416833216 >> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] conn(0x7f417c00d8e0 msgr2=0x7f417c00c360 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-05T19:29:40.552 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.546+0000 7f41893f1e40 1 --2- 172.21.15.167:0/1416833216 >> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] conn(0x7f417c00d8e0 0x7f417c00c360 crc :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-05T19:29:40.552 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.546+0000 7f41893f1e40 1 -- 172.21.15.167:0/1416833216 >> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] conn(0x560d8c398930 msgr2=0x560d8c3cdba0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-05T19:29:40.552 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.546+0000 7f41893f1e40 1 --2- 172.21.15.167:0/1416833216 >> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] conn(0x560d8c398930 0x560d8c3cdba0 crc :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-05T19:29:40.552 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.546+0000 7f41893f1e40 1 -- 172.21.15.167:0/1416833216 >> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] conn(0x7f418001a3c0 msgr2=0x7f418001b3c0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-05T19:29:40.552 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.546+0000 7f41893f1e40 1 --2- 172.21.15.167:0/1416833216 >> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] conn(0x7f418001a3c0 0x7f418001b3c0 crc :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-05T19:29:40.552 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.546+0000 7f41893f1e40 1 -- 172.21.15.167:0/1416833216 >> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] conn(0x560d8c3d1940 msgr2=0x560d8c3d3d20 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-05T19:29:40.552 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.546+0000 7f41893f1e40 1 --2- 172.21.15.167:0/1416833216 >> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] conn(0x560d8c3d1940 0x560d8c3d3d20 crc :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-05T19:29:40.552 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.546+0000 7f41893f1e40 1 -- 172.21.15.167:0/1416833216 >> [v2:172.21.15.167:6824/29889,v1:172.21.15.167:6825/29889] conn(0x7f416c030c00 msgr2=0x7f416c0330b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-05T19:29:40.552 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.546+0000 7f41893f1e40 1 --2- 172.21.15.167:0/1416833216 >> [v2:172.21.15.167:6824/29889,v1:172.21.15.167:6825/29889] conn(0x7f416c030c00 0x7f416c0330b0 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7f417c001e90 tx=0x7f417c0013b0 comp rx=0 tx=0).stop 2023-10-05T19:29:40.553 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.546+0000 7f41893f1e40 1 -- 172.21.15.167:0/1416833216 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x560d8c398020 msgr2=0x560d8c3c20b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-05T19:29:40.553 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.546+0000 7f4187727700 1 -- 172.21.15.167:0/1416833216 reap_dead start 2023-10-05T19:29:40.553 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.546+0000 7f41893f1e40 1 --2- 172.21.15.167:0/1416833216 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x560d8c398020 0x560d8c3c20b0 secure :-1 s=READY pgs=635 cs=0 l=1 rev1=1 crypto rx=0x7f418000a470 tx=0x7f418001d000 comp rx=0 tx=0).stop 2023-10-05T19:29:40.553 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.546+0000 7f41893f1e40 1 -- 172.21.15.167:0/1416833216 shutdown_connections 2023-10-05T19:29:40.553 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.546+0000 7f41893f1e40 1 --2- 172.21.15.167:0/1416833216 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x560d8c398020 0x560d8c3c20b0 unknown :-1 s=CLOSED pgs=635 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-05T19:29:40.553 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.546+0000 7f41893f1e40 1 -- 172.21.15.167:0/1416833216 >> 172.21.15.167:0/1416833216 conn(0x560d8c30b870 msgr2=0x560d8c3b1700 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-05T19:29:40.553 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.546+0000 7f41893f1e40 1 -- 172.21.15.167:0/1416833216 shutdown_connections 2023-10-05T19:29:40.553 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.546+0000 7f41893f1e40 1 -- 172.21.15.167:0/1416833216 wait complete. 2023-10-05T19:29:40.553 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.546+0000 7f41893f1e40 1 -- 172.21.15.167:0/2303268290 >> [v2:172.21.15.167:6824/29889,v1:172.21.15.167:6825/29889] conn(0x7f40cc030c50 msgr2=0x7f40cc033100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-05T19:29:40.553 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.546+0000 7f41893f1e40 1 --2- 172.21.15.167:0/2303268290 >> [v2:172.21.15.167:6824/29889,v1:172.21.15.167:6825/29889] conn(0x7f40cc030c50 0x7f40cc033100 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x560d8c3dbd30 tx=0x7f4178000f90 comp rx=0 tx=0).stop 2023-10-05T19:29:40.554 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.546+0000 7f41893f1e40 1 -- 172.21.15.167:0/2303268290 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x560d8c3ea3b0 msgr2=0x560d8c3eff10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-05T19:29:40.554 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.546+0000 7f41893f1e40 1 --2- 172.21.15.167:0/2303268290 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x560d8c3ea3b0 0x560d8c3eff10 secure :-1 s=READY pgs=637 cs=0 l=1 rev1=1 crypto rx=0x7f417c000fd0 tx=0x7f417c030000 comp rx=0 tx=0).stop 2023-10-05T19:29:40.554 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.546+0000 7f41893f1e40 1 -- 172.21.15.167:0/2303268290 shutdown_connections 2023-10-05T19:29:40.554 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.546+0000 7f41893f1e40 1 --2- 172.21.15.167:0/2303268290 >> [v2:172.21.15.167:6824/29889,v1:172.21.15.167:6825/29889] conn(0x7f40cc030c50 0x7f40cc033100 unknown :-1 s=CLOSED pgs=219 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-05T19:29:40.554 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.546+0000 7f41893f1e40 1 --2- 172.21.15.167:0/2303268290 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x560d8c3ea3b0 0x560d8c3eff10 unknown :-1 s=CLOSED pgs=637 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-05T19:29:40.554 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.546+0000 7f41893f1e40 1 -- 172.21.15.167:0/2303268290 >> 172.21.15.167:0/2303268290 conn(0x560d8c3e5770 msgr2=0x560d8c3e3810 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-05T19:29:40.554 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.546+0000 7f41893f1e40 1 -- 172.21.15.167:0/2303268290 shutdown_connections 2023-10-05T19:29:40.554 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.546+0000 7f41893f1e40 1 -- 172.21.15.167:0/2303268290 wait complete. 2023-10-05T19:29:40.557 DEBUG:teuthology.orchestra.run:got remote process result: 17 2023-10-05T19:29:40.567 INFO:tasks.radosgw_admin:RL: get_obj myfoo6 bud bytes_out=0 bytes_in=0 failed=False 2023-10-05T19:29:40.567 INFO:tasks.radosgw_admin:RL: get_obj myfoo6 bud bytes_out=0 bytes_in=8 failed=False 2023-10-05T19:29:40.574 INFO:tasks.radosgw_admin:RL: get_obj myfoo7 fud bytes_out=0 bytes_in=0 failed=False 2023-10-05T19:29:40.574 INFO:tasks.radosgw_admin:RL: get_obj myfoo7 fud bytes_out=0 bytes_in=8 failed=False 2023-10-05T19:29:40.574 INFO:tasks.util.rgw:rgwadmin: client.0 : ['user', 'rm', '--uid', 'bud', '--purge-data'] 2023-10-05T19:29:40.574 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'ceph', 'user', 'rm', '--uid', 'bud', '--purge-data'] 2023-10-05T19:29:40.575 DEBUG:teuthology.orchestra.run.smithi167:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster ceph user rm --uid bud --purge-data 2023-10-05T19:29:40.611 INFO:teuthology.orchestra.run.smithi167.stderr:ignoring --setuser ceph since I am not root 2023-10-05T19:29:40.611 INFO:teuthology.orchestra.run.smithi167.stderr:ignoring --setgroup ceph since I am not root 2023-10-05T19:29:40.612 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.606+0000 7fd5eb271e40 1 Processor -- start 2023-10-05T19:29:40.612 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.606+0000 7fd5eb271e40 1 -- start start 2023-10-05T19:29:40.612 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.606+0000 7fd5eb271e40 1 --2- >> v2:172.21.15.118:3300/0 conn(0x560321190420 0x56032118efd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-05T19:29:40.612 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.606+0000 7fd5eb271e40 1 -- --> v1:172.21.15.118:6789/0 -- auth(proto 0 26 bytes epoch 0) v1 -- 0x5603210dcbd0 con 0x5603211907f0 2023-10-05T19:29:40.612 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.606+0000 7fd5eb271e40 1 -- --> v2:172.21.15.118:3300/0 -- mon_getmap magic: 0 v1 -- 0x560321099ee0 con 0x560321190420 2023-10-05T19:29:40.614 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.610+0000 7fd5e8da6700 1 --2- >> v2:172.21.15.118:3300/0 conn(0x560321190420 0x56032118efd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-05T19:29:40.614 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.610+0000 7fd5e8da6700 1 --2- >> v2:172.21.15.118:3300/0 conn(0x560321190420 0x56032118efd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.118:3300/0 says I am v2:172.21.15.167:56856/0 (socket says 172.21.15.167:56856) 2023-10-05T19:29:40.614 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.610+0000 7fd5e8da6700 1 -- 172.21.15.167:0/2477219644 learned_addr learned my addr 172.21.15.167:0/2477219644 (peer_addr_for_me v2:172.21.15.167:0/0) 2023-10-05T19:29:40.614 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.610+0000 7fd5e85a5700 1 --1- 172.21.15.167:0/2477219644 >> v1:172.21.15.118:6789/0 conn(0x5603211907f0 0x56032118f510 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.118:6789/0 says I am v1:172.21.15.167:33342/0 (socket says 172.21.15.167:33342) 2023-10-05T19:29:40.614 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.610+0000 7fd5e8da6700 1 -- 172.21.15.167:0/2477219644 >> v1:172.21.15.118:6789/0 conn(0x5603211907f0 legacy=0x56032118f510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-05T19:29:40.614 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.610+0000 7fd5e8da6700 1 -- 172.21.15.167:0/2477219644 --> v2:172.21.15.118:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x56032118fce0 con 0x560321190420 2023-10-05T19:29:40.614 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.610+0000 7fd5e8da6700 1 --2- 172.21.15.167:0/2477219644 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x560321190420 0x56032118efd0 secure :-1 s=READY pgs=639 cs=0 l=1 rev1=1 crypto rx=0x7fd5d800a5c0 tx=0x7fd5d8005e00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d3fb183ec402b66 server_cookie=0 in_seq=0 out_seq=0 2023-10-05T19:29:40.615 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.610+0000 7fd5e7da4700 1 -- 172.21.15.167:0/2477219644 <== mon.0 v2:172.21.15.118:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fd5d800e510 con 0x560321190420 2023-10-05T19:29:40.615 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.610+0000 7fd5e7da4700 1 -- 172.21.15.167:0/2477219644 <== mon.0 v2:172.21.15.118:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd5d800e670 con 0x560321190420 2023-10-05T19:29:40.615 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.610+0000 7fd5e7da4700 1 -- 172.21.15.167:0/2477219644 <== mon.0 v2:172.21.15.118:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fd5d800e920 con 0x560321190420 2023-10-05T19:29:40.617 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.610+0000 7fd5eb271e40 1 -- 172.21.15.167:0/2477219644 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x560321190420 msgr2=0x56032118efd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-05T19:29:40.617 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.610+0000 7fd5eb271e40 1 --2- 172.21.15.167:0/2477219644 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x560321190420 0x56032118efd0 secure :-1 s=READY pgs=639 cs=0 l=1 rev1=1 crypto rx=0x7fd5d800a5c0 tx=0x7fd5d8005e00 comp rx=0 tx=0).stop 2023-10-05T19:29:40.617 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.610+0000 7fd5eb271e40 1 -- 172.21.15.167:0/2477219644 shutdown_connections 2023-10-05T19:29:40.617 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.610+0000 7fd5eb271e40 1 --2- 172.21.15.167:0/2477219644 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x560321190420 0x56032118efd0 unknown :-1 s=CLOSED pgs=639 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-05T19:29:40.617 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.610+0000 7fd5eb271e40 1 -- 172.21.15.167:0/2477219644 >> 172.21.15.167:0/2477219644 conn(0x56032118a730 msgr2=0x56032118cb40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-05T19:29:40.617 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.610+0000 7fd5eb271e40 1 -- 172.21.15.167:0/2477219644 shutdown_connections 2023-10-05T19:29:40.617 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.610+0000 7fd5eb271e40 1 -- 172.21.15.167:0/2477219644 wait complete. 2023-10-05T19:29:40.617 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.610+0000 7fd5eb271e40 1 Processor -- start 2023-10-05T19:29:40.618 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.610+0000 7fd5eb271e40 1 -- start start 2023-10-05T19:29:40.618 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.610+0000 7fd5eb271e40 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x560321190420 0x5603211b1160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-05T19:29:40.618 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.610+0000 7fd5eb271e40 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x56032105de30 con 0x560321190420 2023-10-05T19:29:40.618 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.610+0000 7fd5e8da6700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x560321190420 0x5603211b1160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-05T19:29:40.618 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.610+0000 7fd5e8da6700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x560321190420 0x5603211b1160 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.118:3300/0 says I am v2:172.21.15.167:56864/0 (socket says 172.21.15.167:56864) 2023-10-05T19:29:40.618 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.610+0000 7fd5e8da6700 1 -- 172.21.15.167:0/3151790149 learned_addr learned my addr 172.21.15.167:0/3151790149 (peer_addr_for_me v2:172.21.15.167:0/0) 2023-10-05T19:29:40.618 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.614+0000 7fd5e8da6700 1 -- 172.21.15.167:0/3151790149 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd5d800a180 con 0x560321190420 2023-10-05T19:29:40.618 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.614+0000 7fd5e8da6700 1 --2- 172.21.15.167:0/3151790149 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x560321190420 0x5603211b1160 secure :-1 s=READY pgs=640 cs=0 l=1 rev1=1 crypto rx=0x56032118fcb0 tx=0x7fd5d8013000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=79f4ca35043d57a9 server_cookie=0 in_seq=0 out_seq=0 2023-10-05T19:29:40.621 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.614+0000 7fd5e6da2700 1 -- 172.21.15.167:0/3151790149 <== mon.0 v2:172.21.15.118:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fd5d80032f0 con 0x560321190420 2023-10-05T19:29:40.621 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.614+0000 7fd5e6da2700 1 -- 172.21.15.167:0/3151790149 <== mon.0 v2:172.21.15.118:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd5d800e7e0 con 0x560321190420 2023-10-05T19:29:40.621 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.614+0000 7fd5e6da2700 1 -- 172.21.15.167:0/3151790149 <== mon.0 v2:172.21.15.118:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fd5d800ead0 con 0x560321190420 2023-10-05T19:29:40.622 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.614+0000 7fd5eb271e40 1 -- 172.21.15.167:0/3151790149 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x560321190420 msgr2=0x5603211b1160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-05T19:29:40.622 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.614+0000 7fd5eb271e40 1 --2- 172.21.15.167:0/3151790149 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x560321190420 0x5603211b1160 secure :-1 s=READY pgs=640 cs=0 l=1 rev1=1 crypto rx=0x56032118fcb0 tx=0x7fd5d8013000 comp rx=0 tx=0).stop 2023-10-05T19:29:40.622 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.614+0000 7fd5eb271e40 1 -- 172.21.15.167:0/3151790149 shutdown_connections 2023-10-05T19:29:40.622 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.614+0000 7fd5eb271e40 1 --2- 172.21.15.167:0/3151790149 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x560321190420 0x5603211b1160 unknown :-1 s=CLOSED pgs=640 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-05T19:29:40.622 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.614+0000 7fd5eb271e40 1 -- 172.21.15.167:0/3151790149 >> 172.21.15.167:0/3151790149 conn(0x56032118a730 msgr2=0x5603211acd20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-05T19:29:40.622 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.614+0000 7fd5eb271e40 1 -- 172.21.15.167:0/3151790149 shutdown_connections 2023-10-05T19:29:40.622 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.614+0000 7fd5eb271e40 1 -- 172.21.15.167:0/3151790149 wait complete. 2023-10-05T19:29:40.622 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.614+0000 7fd5eb271e40 1 Processor -- start 2023-10-05T19:29:40.622 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.614+0000 7fd5eb271e40 1 -- start start 2023-10-05T19:29:40.623 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.614+0000 7fd5eb271e40 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x560321190420 0x5603211b7d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-05T19:29:40.623 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.614+0000 7fd5eb271e40 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x560321030ed0 con 0x560321190420 2023-10-05T19:29:40.623 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.614+0000 7fd5e8da6700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x560321190420 0x5603211b7d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-05T19:29:40.623 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.614+0000 7fd5e8da6700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x560321190420 0x5603211b7d90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.118:3300/0 says I am v2:172.21.15.167:56880/0 (socket says 172.21.15.167:56880) 2023-10-05T19:29:40.623 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.614+0000 7fd5e8da6700 1 -- 172.21.15.167:0/3205430407 learned_addr learned my addr 172.21.15.167:0/3205430407 (peer_addr_for_me v2:172.21.15.167:0/0) 2023-10-05T19:29:40.623 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.614+0000 7fd5e8da6700 1 -- 172.21.15.167:0/3205430407 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x56032118fce0 con 0x560321190420 2023-10-05T19:29:40.623 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.618+0000 7fd5e8da6700 1 --2- 172.21.15.167:0/3205430407 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x560321190420 0x5603211b7d90 secure :-1 s=READY pgs=641 cs=0 l=1 rev1=1 crypto rx=0x7fd5d8005860 tx=0x7fd5d801c000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-05T19:29:40.623 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.618+0000 7fd5e559f700 1 -- 172.21.15.167:0/3205430407 <== mon.0 v2:172.21.15.118:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fd5d80109f0 con 0x560321190420 2023-10-05T19:29:40.624 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.618+0000 7fd5e559f700 1 -- 172.21.15.167:0/3205430407 <== mon.0 v2:172.21.15.118:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd5d8010b70 con 0x560321190420 2023-10-05T19:29:40.624 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.618+0000 7fd5eb271e40 1 -- 172.21.15.167:0/3205430407 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x5603211b82d0 con 0x560321190420 2023-10-05T19:29:40.624 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.618+0000 7fd5e559f700 1 -- 172.21.15.167:0/3205430407 <== mon.0 v2:172.21.15.118:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fd5d8010e60 con 0x560321190420 2023-10-05T19:29:40.624 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.618+0000 7fd5eb271e40 1 -- 172.21.15.167:0/3205430407 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x5603211b8710 con 0x560321190420 2023-10-05T19:29:40.626 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.618+0000 7fd5eb271e40 20 rados->read ofs=0 len=0 2023-10-05T19:29:40.627 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.618+0000 7fd5e559f700 1 -- 172.21.15.167:0/3205430407 <== mon.0 v2:172.21.15.118:3300/0 4 ==== mgrmap(e 5) v1 ==== 40329+0+0 (secure 0 0 0) 0x7fd5d801c430 con 0x560321190420 2023-10-05T19:29:40.627 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.618+0000 7fd5e559f700 1 --2- 172.21.15.167:0/3205430407 >> [v2:172.21.15.167:6824/29889,v1:172.21.15.167:6825/29889] conn(0x7fd5c8030ca0 0x7fd5c8033150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-05T19:29:40.627 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.618+0000 7fd5e559f700 1 -- 172.21.15.167:0/3205430407 <== mon.0 v2:172.21.15.118:3300/0 5 ==== osd_map(32..32 src has 1..32) v4 ==== 6260+0+0 (secure 0 0 0) 0x7fd5d8046390 con 0x560321190420 2023-10-05T19:29:40.627 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.618+0000 7fd5e85a5700 1 --2- 172.21.15.167:0/3205430407 >> [v2:172.21.15.167:6824/29889,v1:172.21.15.167:6825/29889] conn(0x7fd5c8030ca0 0x7fd5c8033150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-05T19:29:40.627 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.618+0000 7fd5eb271e40 1 --2- 172.21.15.167:0/3205430407 >> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] conn(0x5603211907f0 0x5603211c3820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-05T19:29:40.627 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.618+0000 7fd5eb271e40 1 -- 172.21.15.167:0/3205430407 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:1 5.12 5:49953fa1:::default.realm:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32) v8 -- 0x5603211c3f20 con 0x5603211907f0 2023-10-05T19:29:40.627 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.618+0000 7fd5e95a7700 1 --2- 172.21.15.167:0/3205430407 >> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] conn(0x5603211907f0 0x5603211c3820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-05T19:29:40.627 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.622+0000 7fd5e85a5700 1 --2- 172.21.15.167:0/3205430407 >> [v2:172.21.15.167:6824/29889,v1:172.21.15.167:6825/29889] conn(0x7fd5c8030ca0 0x7fd5c8033150 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7fd5d000b300 tx=0x7fd5d0001430 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-05T19:29:40.628 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.622+0000 7fd5e95a7700 1 --2- 172.21.15.167:0/3205430407 >> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] conn(0x5603211907f0 0x5603211c3820 crc :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-05T19:29:40.628 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.622+0000 7fd5e95a7700 1 -- 172.21.15.167:0/3205430407 <== osd.1 v2:172.21.15.167:6800/30035 1 ==== osd_op_reply(1 default.realm [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 ==== 157+0+0 (crc 0 0 0) 0x7fd5e000a500 con 0x5603211907f0 2023-10-05T19:29:40.628 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.622+0000 7fd5eb271e40 20 rados_obj.operate() r=-2 bl.length=0 2023-10-05T19:29:40.628 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.622+0000 7fd5eb271e40 20 realm 2023-10-05T19:29:40.628 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.622+0000 7fd5eb271e40 20 rados->read ofs=0 len=0 2023-10-05T19:29:40.628 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.622+0000 7fd5eb271e40 1 -- 172.21.15.167:0/3205430407 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:2 5.12 5:49953fa1:::default.realm:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32) v8 -- 0x5603211c0e80 con 0x5603211907f0 2023-10-05T19:29:40.628 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.622+0000 7fd5e95a7700 1 -- 172.21.15.167:0/3205430407 <== osd.1 v2:172.21.15.167:6800/30035 2 ==== osd_op_reply(2 default.realm [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 ==== 157+0+0 (crc 0 0 0) 0x7fd5e000a500 con 0x5603211907f0 2023-10-05T19:29:40.628 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.622+0000 7fd5eb271e40 20 rados_obj.operate() r=-2 bl.length=0 2023-10-05T19:29:40.628 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.622+0000 7fd5eb271e40 4 RGWPeriod::init failed to init realm id : (2) No such file or directory 2023-10-05T19:29:40.628 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.622+0000 7fd5eb271e40 20 rados->read ofs=0 len=0 2023-10-05T19:29:40.629 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.622+0000 7fd5eb271e40 1 -- 172.21.15.167:0/3205430407 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:3 5.12 5:49953fa1:::default.realm:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32) v8 -- 0x5603211c4a00 con 0x5603211907f0 2023-10-05T19:29:40.629 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.622+0000 7fd5e95a7700 1 -- 172.21.15.167:0/3205430407 <== osd.1 v2:172.21.15.167:6800/30035 3 ==== osd_op_reply(3 default.realm [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 ==== 157+0+0 (crc 0 0 0) 0x7fd5e000a500 con 0x5603211907f0 2023-10-05T19:29:40.629 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.622+0000 7fd5eb271e40 20 rados_obj.operate() r=-2 bl.length=0 2023-10-05T19:29:40.629 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.622+0000 7fd5eb271e40 20 rados->read ofs=0 len=0 2023-10-05T19:29:40.629 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.622+0000 7fd5eb271e40 1 -- 172.21.15.167:0/3205430407 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:4 5.f 5:f43b6ece:::zone_names.default:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32) v8 -- 0x5603211c4610 con 0x5603211907f0 2023-10-05T19:29:40.631 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.622+0000 7fd5e95a7700 1 -- 172.21.15.167:0/3205430407 <== osd.1 v2:172.21.15.167:6800/30035 4 ==== osd_op_reply(4 zone_names.default [read 0~46 out=46b] v0'0 uv1 ondisk = 0) v8 ==== 162+0+46 (crc 0 0 0) 0x7fd5e000a500 con 0x5603211907f0 2023-10-05T19:29:40.631 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.622+0000 7fd5eb271e40 20 rados_obj.operate() r=0 bl.length=46 2023-10-05T19:29:40.631 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.622+0000 7fd5eb271e40 20 rados->read ofs=0 len=0 2023-10-05T19:29:40.631 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.622+0000 7fd5eb271e40 1 -- 172.21.15.167:0/3205430407 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:5 5.1d 5:b8374eb4:::zone_info.9a9df965-6612-45d4-a45f-6e3c889bacc2:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32) v8 -- 0x5603211c5890 con 0x5603211907f0 2023-10-05T19:29:40.631 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.622+0000 7fd5e95a7700 1 -- 172.21.15.167:0/3205430407 <== osd.1 v2:172.21.15.167:6800/30035 5 ==== osd_op_reply(5 zone_info.9a9df965-6612-45d4-a45f-6e3c889bacc2 [read 0~874 out=874b] v0'0 uv2 ondisk = 0) v8 ==== 190+0+874 (crc 0 0 0) 0x7fd5e000a500 con 0x5603211907f0 2023-10-05T19:29:40.631 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.622+0000 7fd5eb271e40 20 rados_obj.operate() r=0 bl.length=874 2023-10-05T19:29:40.631 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.622+0000 7fd5eb271e40 20 searching for the correct realm 2023-10-05T19:29:40.631 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.622+0000 7fd5eb271e40 1 --2- 172.21.15.167:0/3205430407 >> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] conn(0x5603211c7650 0x5603211c9a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-05T19:29:40.632 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.622+0000 7fd5eb271e40 1 -- 172.21.15.167:0/3205430407 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:6 5.0 5:00000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x5603211c9f70 con 0x5603211c7650 2023-10-05T19:29:40.632 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.626+0000 7fd5e8da6700 1 --2- 172.21.15.167:0/3205430407 >> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] conn(0x5603211c7650 0x5603211c9a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-05T19:29:40.632 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.626+0000 7fd5e8da6700 1 --2- 172.21.15.167:0/3205430407 >> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] conn(0x5603211c7650 0x5603211c9a30 crc :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-05T19:29:40.632 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.626+0000 7fd5e8da6700 1 -- 172.21.15.167:0/3205430407 <== osd.3 v2:172.21.15.167:6808/30036 1 ==== osd_op_reply(6 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fd5d802b070 con 0x5603211c7650 2023-10-05T19:29:40.632 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.626+0000 7fd5e8da6700 1 --2- 172.21.15.167:0/3205430407 >> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] conn(0x7fd5d8010070 0x7fd5d8044aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-05T19:29:40.632 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.626+0000 7fd5e8da6700 1 -- 172.21.15.167:0/3205430407 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:7 5.10 5:08000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x5603211c9f70 con 0x7fd5d8010070 2023-10-05T19:29:40.632 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.626+0000 7fd5e85a5700 1 --2- 172.21.15.167:0/3205430407 >> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] conn(0x7fd5d8010070 0x7fd5d8044aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-05T19:29:40.633 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.626+0000 7fd5e85a5700 1 --2- 172.21.15.167:0/3205430407 >> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] conn(0x7fd5d8010070 0x7fd5d8044aa0 crc :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-05T19:29:40.634 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.626+0000 7fd5e85a5700 1 -- 172.21.15.167:0/3205430407 <== osd.2 v2:172.21.15.167:6816/30037 1 ==== osd_op_reply(7 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fd5d000f560 con 0x7fd5d8010070 2023-10-05T19:29:40.634 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.626+0000 7fd5e85a5700 1 --2- 172.21.15.167:0/3205430407 >> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] conn(0x7fd5d000e8c0 0x7fd5d000d310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-05T19:29:40.634 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.626+0000 7fd5e85a5700 1 -- 172.21.15.167:0/3205430407 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:8 5.8 5:10000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x5603211c9f70 con 0x7fd5d000e8c0 2023-10-05T19:29:40.634 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.626+0000 7fd5e95a7700 1 --2- 172.21.15.167:0/3205430407 >> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] conn(0x7fd5d000e8c0 0x7fd5d000d310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-05T19:29:40.634 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.630+0000 7fd5e95a7700 1 --2- 172.21.15.167:0/3205430407 >> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] conn(0x7fd5d000e8c0 0x7fd5d000d310 crc :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-05T19:29:40.634 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.630+0000 7fd5e95a7700 1 -- 172.21.15.167:0/3205430407 <== osd.0 v2:172.21.15.118:6800/27690 1 ==== osd_op_reply(8 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fd5e0014040 con 0x7fd5d000e8c0 2023-10-05T19:29:40.634 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.630+0000 7fd5e95a7700 1 -- 172.21.15.167:0/3205430407 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:9 5.18 5:18000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x5603211c9f70 con 0x5603211907f0 2023-10-05T19:29:40.635 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.630+0000 7fd5e95a7700 1 -- 172.21.15.167:0/3205430407 <== osd.1 v2:172.21.15.167:6800/30035 6 ==== osd_op_reply(9 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fd5e0014040 con 0x5603211907f0 2023-10-05T19:29:40.635 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.630+0000 7fd5e95a7700 1 -- 172.21.15.167:0/3205430407 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:10 5.4 5:20000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x5603211c9f70 con 0x5603211907f0 2023-10-05T19:29:40.635 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.630+0000 7fd5e95a7700 1 -- 172.21.15.167:0/3205430407 <== osd.1 v2:172.21.15.167:6800/30035 7 ==== osd_op_reply(10 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fd5e0014040 con 0x5603211907f0 2023-10-05T19:29:40.635 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.630+0000 7fd5e95a7700 1 -- 172.21.15.167:0/3205430407 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:11 5.14 5:28000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x5603211c9f70 con 0x7fd5d000e8c0 2023-10-05T19:29:40.636 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.630+0000 7fd5e95a7700 1 -- 172.21.15.167:0/3205430407 <== osd.0 v2:172.21.15.118:6800/27690 2 ==== osd_op_reply(11 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fd5e0014040 con 0x7fd5d000e8c0 2023-10-05T19:29:40.636 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.630+0000 7fd5e95a7700 1 -- 172.21.15.167:0/3205430407 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:12 5.c 5:30000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x5603211c9f70 con 0x7fd5d000e8c0 2023-10-05T19:29:40.637 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.634+0000 7fd5e95a7700 1 -- 172.21.15.167:0/3205430407 <== osd.0 v2:172.21.15.118:6800/27690 3 ==== osd_op_reply(12 [pgnls start_epoch 32 out=112b] v23'1 uv1 ondisk = 1) v8 ==== 144+0+112 (crc 0 0 0) 0x7fd5e0014040 con 0x7fd5d000e8c0 2023-10-05T19:29:40.637 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.634+0000 7fd5e95a7700 1 -- 172.21.15.167:0/3205430407 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:13 5.1c 5:38000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x5603211c9f70 con 0x7fd5d8010070 2023-10-05T19:29:40.637 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.634+0000 7fd5e85a5700 1 -- 172.21.15.167:0/3205430407 <== osd.2 v2:172.21.15.167:6816/30037 2 ==== osd_op_reply(13 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fd5d000f560 con 0x7fd5d8010070 2023-10-05T19:29:40.637 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.634+0000 7fd5e85a5700 1 -- 172.21.15.167:0/3205430407 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:14 5.2 5:40000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x5603211c9f70 con 0x5603211c7650 2023-10-05T19:29:40.638 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.634+0000 7fd5e8da6700 1 -- 172.21.15.167:0/3205430407 <== osd.3 v2:172.21.15.167:6808/30036 2 ==== osd_op_reply(14 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fd5d802b070 con 0x5603211c7650 2023-10-05T19:29:40.638 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.634+0000 7fd5e8da6700 1 -- 172.21.15.167:0/3205430407 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:15 5.12 5:48000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x5603211c9f70 con 0x5603211907f0 2023-10-05T19:29:40.639 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.634+0000 7fd5e95a7700 1 -- 172.21.15.167:0/3205430407 <== osd.1 v2:172.21.15.167:6800/30035 8 ==== osd_op_reply(15 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fd5e0014040 con 0x5603211907f0 2023-10-05T19:29:40.639 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.634+0000 7fd5e95a7700 1 -- 172.21.15.167:0/3205430407 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:16 5.a 5:50000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x5603211c9f70 con 0x7fd5d000e8c0 2023-10-05T19:29:40.640 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.634+0000 7fd5e95a7700 1 -- 172.21.15.167:0/3205430407 <== osd.0 v2:172.21.15.118:6800/27690 4 ==== osd_op_reply(16 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fd5e0022040 con 0x7fd5d000e8c0 2023-10-05T19:29:40.640 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.634+0000 7fd5e95a7700 1 -- 172.21.15.167:0/3205430407 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:17 5.1a 5:58000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x5603211c9f70 con 0x5603211907f0 2023-10-05T19:29:40.640 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.638+0000 7fd5e95a7700 1 -- 172.21.15.167:0/3205430407 <== osd.1 v2:172.21.15.167:6800/30035 9 ==== osd_op_reply(17 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fd5e0022040 con 0x5603211907f0 2023-10-05T19:29:40.641 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.638+0000 7fd5e95a7700 1 -- 172.21.15.167:0/3205430407 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:18 5.6 5:60000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x5603211c9f70 con 0x7fd5d8010070 2023-10-05T19:29:40.641 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.638+0000 7fd5e85a5700 1 -- 172.21.15.167:0/3205430407 <== osd.2 v2:172.21.15.167:6816/30037 3 ==== osd_op_reply(18 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fd5d000f560 con 0x7fd5d8010070 2023-10-05T19:29:40.641 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.638+0000 7fd5e85a5700 1 -- 172.21.15.167:0/3205430407 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:19 5.16 5:68000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x7fd5d001f040 con 0x5603211c7650 2023-10-05T19:29:40.642 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.638+0000 7fd5e8da6700 1 -- 172.21.15.167:0/3205430407 <== osd.3 v2:172.21.15.167:6808/30036 3 ==== osd_op_reply(19 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fd5d802b070 con 0x5603211c7650 2023-10-05T19:29:40.642 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.638+0000 7fd5e8da6700 1 -- 172.21.15.167:0/3205430407 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:20 5.e 5:70000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x7fd5d001f040 con 0x5603211c7650 2023-10-05T19:29:40.643 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.638+0000 7fd5e8da6700 1 -- 172.21.15.167:0/3205430407 <== osd.3 v2:172.21.15.167:6808/30036 4 ==== osd_op_reply(20 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fd5d802b070 con 0x5603211c7650 2023-10-05T19:29:40.643 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.638+0000 7fd5e8da6700 1 -- 172.21.15.167:0/3205430407 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:21 5.1e 5:78000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x7fd5d001f040 con 0x7fd5d000e8c0 2023-10-05T19:29:40.644 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.638+0000 7fd5e95a7700 1 -- 172.21.15.167:0/3205430407 <== osd.0 v2:172.21.15.118:6800/27690 5 ==== osd_op_reply(21 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fd5e0022040 con 0x7fd5d000e8c0 2023-10-05T19:29:40.644 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.638+0000 7fd5e95a7700 1 -- 172.21.15.167:0/3205430407 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:22 5.1 5:80000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x7fd5d001f040 con 0x5603211c7650 2023-10-05T19:29:40.644 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.638+0000 7fd5e8da6700 1 -- 172.21.15.167:0/3205430407 <== osd.3 v2:172.21.15.167:6808/30036 5 ==== osd_op_reply(22 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fd5d802b070 con 0x5603211c7650 2023-10-05T19:29:40.644 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.642+0000 7fd5e8da6700 1 -- 172.21.15.167:0/3205430407 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:23 5.11 5:88000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x7fd5d001f040 con 0x5603211c7650 2023-10-05T19:29:40.645 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.642+0000 7fd5e8da6700 1 -- 172.21.15.167:0/3205430407 <== osd.3 v2:172.21.15.167:6808/30036 6 ==== osd_op_reply(23 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fd5d802b070 con 0x5603211c7650 2023-10-05T19:29:40.645 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.642+0000 7fd5e8da6700 1 -- 172.21.15.167:0/3205430407 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:24 5.9 5:90000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x7fd5d001f040 con 0x5603211907f0 2023-10-05T19:29:40.646 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.642+0000 7fd5e95a7700 1 -- 172.21.15.167:0/3205430407 <== osd.1 v2:172.21.15.167:6800/30035 10 ==== osd_op_reply(24 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fd5e0022040 con 0x5603211907f0 2023-10-05T19:29:40.646 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.642+0000 7fd5e95a7700 1 -- 172.21.15.167:0/3205430407 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:25 5.19 5:98000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x7fd5d001f040 con 0x5603211907f0 2023-10-05T19:29:40.646 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.642+0000 7fd5e95a7700 1 -- 172.21.15.167:0/3205430407 <== osd.1 v2:172.21.15.167:6800/30035 11 ==== osd_op_reply(25 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fd5e0022040 con 0x5603211907f0 2023-10-05T19:29:40.647 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.642+0000 7fd5e95a7700 1 -- 172.21.15.167:0/3205430407 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:26 5.5 5:a0000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x7fd5d001f040 con 0x7fd5d000e8c0 2023-10-05T19:29:40.647 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.642+0000 7fd5e95a7700 1 -- 172.21.15.167:0/3205430407 <== osd.0 v2:172.21.15.118:6800/27690 6 ==== osd_op_reply(26 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fd5e0022040 con 0x7fd5d000e8c0 2023-10-05T19:29:40.648 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.642+0000 7fd5e95a7700 1 -- 172.21.15.167:0/3205430407 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:27 5.15 5:a8000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x7fd5d001f040 con 0x5603211c7650 2023-10-05T19:29:40.648 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.642+0000 7fd5e8da6700 1 -- 172.21.15.167:0/3205430407 <== osd.3 v2:172.21.15.167:6808/30036 7 ==== osd_op_reply(27 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fd5d802b070 con 0x5603211c7650 2023-10-05T19:29:40.648 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.646+0000 7fd5e8da6700 1 -- 172.21.15.167:0/3205430407 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:28 5.d 5:b0000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x7fd5d001f040 con 0x7fd5d8010070 2023-10-05T19:29:40.649 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.646+0000 7fd5e85a5700 1 -- 172.21.15.167:0/3205430407 <== osd.2 v2:172.21.15.167:6816/30037 4 ==== osd_op_reply(28 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fd5d000f560 con 0x7fd5d8010070 2023-10-05T19:29:40.649 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.646+0000 7fd5e85a5700 1 -- 172.21.15.167:0/3205430407 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:29 5.1d 5:b8000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x7fd5d001f040 con 0x5603211907f0 2023-10-05T19:29:40.650 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.646+0000 7fd5e95a7700 1 -- 172.21.15.167:0/3205430407 <== osd.1 v2:172.21.15.167:6800/30035 12 ==== osd_op_reply(29 [pgnls start_epoch 32 out=107b] v30'2 uv2 ondisk = 1) v8 ==== 144+0+107 (crc 0 0 0) 0x7fd5e0022040 con 0x5603211907f0 2023-10-05T19:29:40.650 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.646+0000 7fd5e95a7700 1 -- 172.21.15.167:0/3205430407 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:30 5.3 5:c0000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x7fd5d001f040 con 0x7fd5d000e8c0 2023-10-05T19:29:40.651 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.646+0000 7fd5e95a7700 1 -- 172.21.15.167:0/3205430407 <== osd.0 v2:172.21.15.118:6800/27690 7 ==== osd_op_reply(30 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fd5e0022040 con 0x7fd5d000e8c0 2023-10-05T19:29:40.651 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.646+0000 7fd5e95a7700 1 -- 172.21.15.167:0/3205430407 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:31 5.13 5:c8000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x7fd5d001f040 con 0x5603211c7650 2023-10-05T19:29:40.652 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.646+0000 7fd5e8da6700 1 -- 172.21.15.167:0/3205430407 <== osd.3 v2:172.21.15.167:6808/30036 8 ==== osd_op_reply(31 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fd5d802b070 con 0x5603211c7650 2023-10-05T19:29:40.652 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.646+0000 7fd5e8da6700 1 -- 172.21.15.167:0/3205430407 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:32 5.b 5:d0000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x7fd5d001f040 con 0x7fd5d000e8c0 2023-10-05T19:29:40.653 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.650+0000 7fd5e95a7700 1 -- 172.21.15.167:0/3205430407 <== osd.0 v2:172.21.15.118:6800/27690 8 ==== osd_op_reply(32 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fd5e0022040 con 0x7fd5d000e8c0 2023-10-05T19:29:40.653 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.650+0000 7fd5e95a7700 1 -- 172.21.15.167:0/3205430407 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:33 5.1b 5:d8000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x7fd5d001f040 con 0x7fd5d8010070 2023-10-05T19:29:40.654 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.650+0000 7fd5e85a5700 1 -- 172.21.15.167:0/3205430407 <== osd.2 v2:172.21.15.167:6816/30037 5 ==== osd_op_reply(33 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fd5d000f560 con 0x7fd5d8010070 2023-10-05T19:29:40.654 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.650+0000 7fd5e85a5700 1 -- 172.21.15.167:0/3205430407 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:34 5.7 5:e0000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x7fd5d001f040 con 0x5603211c7650 2023-10-05T19:29:40.654 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.650+0000 7fd5e8da6700 1 -- 172.21.15.167:0/3205430407 <== osd.3 v2:172.21.15.167:6808/30036 9 ==== osd_op_reply(34 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fd5d802b070 con 0x5603211c7650 2023-10-05T19:29:40.655 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.650+0000 7fd5e8da6700 1 -- 172.21.15.167:0/3205430407 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:35 5.17 5:e8000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x7fd5d001f040 con 0x7fd5d000e8c0 2023-10-05T19:29:40.656 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.650+0000 7fd5e95a7700 1 -- 172.21.15.167:0/3205430407 <== osd.0 v2:172.21.15.118:6800/27690 9 ==== osd_op_reply(35 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fd5e0022040 con 0x7fd5d000e8c0 2023-10-05T19:29:40.656 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.650+0000 7fd5e95a7700 1 -- 172.21.15.167:0/3205430407 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:36 5.f 5:f0000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x7fd5d001f040 con 0x5603211907f0 2023-10-05T19:29:40.657 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.654+0000 7fd5e95a7700 1 -- 172.21.15.167:0/3205430407 <== osd.1 v2:172.21.15.167:6800/30035 13 ==== osd_op_reply(36 [pgnls start_epoch 32 out=115b] v23'2 uv2 ondisk = 1) v8 ==== 144+0+115 (crc 0 0 0) 0x7fd5e002f080 con 0x5603211907f0 2023-10-05T19:29:40.657 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.654+0000 7fd5e95a7700 1 -- 172.21.15.167:0/3205430407 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:37 5.1f 5:f8000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x7fd5d001f040 con 0x7fd5d8010070 2023-10-05T19:29:40.657 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.654+0000 7fd5e85a5700 1 -- 172.21.15.167:0/3205430407 <== osd.2 v2:172.21.15.167:6816/30037 6 ==== osd_op_reply(37 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fd5d000f560 con 0x7fd5d8010070 2023-10-05T19:29:40.658 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.654+0000 7fd5eb271e40 20 RGWRados::pool_iterate: got zonegroup_info.4f1f719d-9648-417b-b10c-e5d939825419 2023-10-05T19:29:40.658 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.654+0000 7fd5eb271e40 20 RGWRados::pool_iterate: got zone_info.9a9df965-6612-45d4-a45f-6e3c889bacc2 2023-10-05T19:29:40.658 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.654+0000 7fd5eb271e40 20 RGWRados::pool_iterate: got zone_names.default 2023-10-05T19:29:40.658 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.654+0000 7fd5eb271e40 20 RGWRados::pool_iterate: got zonegroups_names.default 2023-10-05T19:29:40.658 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.654+0000 7fd5eb271e40 20 rados->read ofs=0 len=0 2023-10-05T19:29:40.658 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.654+0000 7fd5eb271e40 1 -- 172.21.15.167:0/3205430407 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:38 5.12 5:49953fa1:::default.realm:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32) v8 -- 0x5603211c6400 con 0x5603211907f0 2023-10-05T19:29:40.659 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.654+0000 7fd5e95a7700 1 -- 172.21.15.167:0/3205430407 <== osd.1 v2:172.21.15.167:6800/30035 14 ==== osd_op_reply(38 default.realm [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 ==== 157+0+0 (crc 0 0 0) 0x7fd5e002f080 con 0x5603211907f0 2023-10-05T19:29:40.659 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.654+0000 7fd5eb271e40 20 rados_obj.operate() r=-2 bl.length=0 2023-10-05T19:29:40.659 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.654+0000 7fd5eb271e40 20 rados->read ofs=0 len=0 2023-10-05T19:29:40.659 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.654+0000 7fd5eb271e40 1 -- 172.21.15.167:0/3205430407 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:39 5.f 5:f4c53578:::zonegroups_names.default:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32) v8 -- 0x5603211c6cf0 con 0x5603211907f0 2023-10-05T19:29:40.659 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.654+0000 7fd5e95a7700 1 -- 172.21.15.167:0/3205430407 <== osd.1 v2:172.21.15.167:6800/30035 15 ==== osd_op_reply(39 zonegroups_names.default [read 0~46 out=46b] v0'0 uv2 ondisk = 0) v8 ==== 168+0+46 (crc 0 0 0) 0x7fd5e002f080 con 0x5603211907f0 2023-10-05T19:29:40.660 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.654+0000 7fd5eb271e40 20 rados_obj.operate() r=0 bl.length=46 2023-10-05T19:29:40.660 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.654+0000 7fd5eb271e40 20 rados->read ofs=0 len=0 2023-10-05T19:29:40.660 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.654+0000 7fd5eb271e40 1 -- 172.21.15.167:0/3205430407 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:40 5.c 5:30d299a5:::zonegroup_info.4f1f719d-9648-417b-b10c-e5d939825419:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32) v8 -- 0x5603211ca3e0 con 0x7fd5d000e8c0 2023-10-05T19:29:40.661 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.654+0000 7fd5e95a7700 1 -- 172.21.15.167:0/3205430407 <== osd.0 v2:172.21.15.118:6800/27690 10 ==== osd_op_reply(40 zonegroup_info.4f1f719d-9648-417b-b10c-e5d939825419 [read 0~362 out=362b] v0'0 uv1 ondisk = 0) v8 ==== 195+0+362 (crc 0 0 0) 0x7fd5e002f080 con 0x7fd5d000e8c0 2023-10-05T19:29:40.661 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.654+0000 7fd5eb271e40 20 rados_obj.operate() r=0 bl.length=362 2023-10-05T19:29:40.661 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.654+0000 7fd5eb271e40 20 rados->read ofs=0 len=0 2023-10-05T19:29:40.661 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.654+0000 7fd5eb271e40 1 -- 172.21.15.167:0/3205430407 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:41 5.13 5:cbbf6c60:::converted:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32) v8 -- 0x5603211ca3e0 con 0x5603211c7650 2023-10-05T19:29:40.661 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.658+0000 7fd5e8da6700 1 -- 172.21.15.167:0/3205430407 <== osd.3 v2:172.21.15.167:6808/30036 10 ==== osd_op_reply(41 converted [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 ==== 153+0+0 (crc 0 0 0) 0x7fd5d802b070 con 0x5603211c7650 2023-10-05T19:29:40.661 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.658+0000 7fd5eb271e40 20 rados_obj.operate() r=-2 bl.length=0 2023-10-05T19:29:40.661 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.658+0000 7fd5eb271e40 20 rados->read ofs=0 len=0 2023-10-05T19:29:40.661 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.658+0000 7fd5eb271e40 1 -- 172.21.15.167:0/3205430407 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:42 5.12 5:49953fa1:::default.realm:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32) v8 -- 0x5603211ca790 con 0x5603211907f0 2023-10-05T19:29:40.662 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.658+0000 7fd5e95a7700 1 -- 172.21.15.167:0/3205430407 <== osd.1 v2:172.21.15.167:6800/30035 16 ==== osd_op_reply(42 default.realm [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 ==== 157+0+0 (crc 0 0 0) 0x7fd5e002f080 con 0x5603211907f0 2023-10-05T19:29:40.662 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.658+0000 7fd5eb271e40 20 rados_obj.operate() r=-2 bl.length=0 2023-10-05T19:29:40.662 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.658+0000 7fd5eb271e40 20 rados->read ofs=0 len=0 2023-10-05T19:29:40.662 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.658+0000 7fd5eb271e40 1 -- 172.21.15.167:0/3205430407 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:43 5.f 5:f4c53578:::zonegroups_names.default:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32) v8 -- 0x5603211cab90 con 0x5603211907f0 2023-10-05T19:29:40.663 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.658+0000 7fd5e95a7700 1 -- 172.21.15.167:0/3205430407 <== osd.1 v2:172.21.15.167:6800/30035 17 ==== osd_op_reply(43 zonegroups_names.default [read 0~46 out=46b] v0'0 uv2 ondisk = 0) v8 ==== 168+0+46 (crc 0 0 0) 0x7fd5e002f080 con 0x5603211907f0 2023-10-05T19:29:40.663 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.658+0000 7fd5eb271e40 20 rados_obj.operate() r=0 bl.length=46 2023-10-05T19:29:40.663 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.658+0000 7fd5eb271e40 1 -- 172.21.15.167:0/3205430407 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:44 5.0 5:00000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x5603211cbba0 con 0x5603211c7650 2023-10-05T19:29:40.663 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.658+0000 7fd5e8da6700 1 -- 172.21.15.167:0/3205430407 <== osd.3 v2:172.21.15.167:6808/30036 11 ==== osd_op_reply(44 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fd5d802b070 con 0x5603211c7650 2023-10-05T19:29:40.664 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.658+0000 7fd5e8da6700 1 -- 172.21.15.167:0/3205430407 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:45 5.10 5:08000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x5603211cbba0 con 0x7fd5d8010070 2023-10-05T19:29:40.664 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.662+0000 7fd5e85a5700 1 -- 172.21.15.167:0/3205430407 <== osd.2 v2:172.21.15.167:6816/30037 7 ==== osd_op_reply(45 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fd5d000f560 con 0x7fd5d8010070 2023-10-05T19:29:40.665 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.662+0000 7fd5e85a5700 1 -- 172.21.15.167:0/3205430407 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:46 5.8 5:10000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x5603211cbba0 con 0x7fd5d000e8c0 2023-10-05T19:29:40.665 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.662+0000 7fd5e95a7700 1 -- 172.21.15.167:0/3205430407 <== osd.0 v2:172.21.15.118:6800/27690 11 ==== osd_op_reply(46 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fd5e002f080 con 0x7fd5d000e8c0 2023-10-05T19:29:40.666 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.662+0000 7fd5e95a7700 1 -- 172.21.15.167:0/3205430407 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:47 5.18 5:18000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x5603211c6cf0 con 0x5603211907f0 2023-10-05T19:29:40.666 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.662+0000 7fd5e95a7700 1 -- 172.21.15.167:0/3205430407 <== osd.1 v2:172.21.15.167:6800/30035 18 ==== osd_op_reply(47 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fd5e002f080 con 0x5603211907f0 2023-10-05T19:29:40.666 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.662+0000 7fd5e95a7700 1 -- 172.21.15.167:0/3205430407 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:48 5.4 5:20000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x5603211c6cf0 con 0x5603211907f0 2023-10-05T19:29:40.667 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.662+0000 7fd5e95a7700 1 -- 172.21.15.167:0/3205430407 <== osd.1 v2:172.21.15.167:6800/30035 19 ==== osd_op_reply(48 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fd5e002f080 con 0x5603211907f0 2023-10-05T19:29:40.667 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.662+0000 7fd5e95a7700 1 -- 172.21.15.167:0/3205430407 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:49 5.14 5:28000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x5603211c6cf0 con 0x7fd5d000e8c0 2023-10-05T19:29:40.668 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.662+0000 7fd5e95a7700 1 -- 172.21.15.167:0/3205430407 <== osd.0 v2:172.21.15.118:6800/27690 12 ==== osd_op_reply(49 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fd5e002f080 con 0x7fd5d000e8c0 2023-10-05T19:29:40.668 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.666+0000 7fd5e95a7700 1 -- 172.21.15.167:0/3205430407 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:50 5.c 5:30000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x5603211c6cf0 con 0x7fd5d000e8c0 2023-10-05T19:29:40.669 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.666+0000 7fd5e95a7700 1 -- 172.21.15.167:0/3205430407 <== osd.0 v2:172.21.15.118:6800/27690 13 ==== osd_op_reply(50 [pgnls start_epoch 32 out=112b] v23'1 uv1 ondisk = 1) v8 ==== 144+0+112 (crc 0 0 0) 0x7fd5e002f080 con 0x7fd5d000e8c0 2023-10-05T19:29:40.669 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.666+0000 7fd5e95a7700 1 -- 172.21.15.167:0/3205430407 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:51 5.1c 5:38000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x5603211c6cf0 con 0x7fd5d8010070 2023-10-05T19:29:40.670 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.666+0000 7fd5e85a5700 1 -- 172.21.15.167:0/3205430407 <== osd.2 v2:172.21.15.167:6816/30037 8 ==== osd_op_reply(51 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fd5d000f560 con 0x7fd5d8010070 2023-10-05T19:29:40.670 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.666+0000 7fd5e85a5700 1 -- 172.21.15.167:0/3205430407 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:52 5.2 5:40000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x5603211c6cf0 con 0x5603211c7650 2023-10-05T19:29:40.671 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.666+0000 7fd5e8da6700 1 -- 172.21.15.167:0/3205430407 <== osd.3 v2:172.21.15.167:6808/30036 12 ==== osd_op_reply(52 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fd5d802b070 con 0x5603211c7650 2023-10-05T19:29:40.671 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.666+0000 7fd5e8da6700 1 -- 172.21.15.167:0/3205430407 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:53 5.12 5:48000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x5603211c6cf0 con 0x5603211907f0 2023-10-05T19:29:40.672 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.666+0000 7fd5e95a7700 1 -- 172.21.15.167:0/3205430407 <== osd.1 v2:172.21.15.167:6800/30035 20 ==== osd_op_reply(53 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fd5e002f080 con 0x5603211907f0 2023-10-05T19:29:40.672 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.666+0000 7fd5e95a7700 1 -- 172.21.15.167:0/3205430407 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:54 5.a 5:50000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x5603211c6cf0 con 0x7fd5d000e8c0 2023-10-05T19:29:40.673 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.670+0000 7fd5e95a7700 1 -- 172.21.15.167:0/3205430407 <== osd.0 v2:172.21.15.118:6800/27690 14 ==== osd_op_reply(54 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fd5e002f080 con 0x7fd5d000e8c0 2023-10-05T19:29:40.673 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.670+0000 7fd5e95a7700 1 -- 172.21.15.167:0/3205430407 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:55 5.1a 5:58000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x5603211c6cf0 con 0x5603211907f0 2023-10-05T19:29:40.674 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.670+0000 7fd5e95a7700 1 -- 172.21.15.167:0/3205430407 <== osd.1 v2:172.21.15.167:6800/30035 21 ==== osd_op_reply(55 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fd5e002f080 con 0x5603211907f0 2023-10-05T19:29:40.674 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.670+0000 7fd5e95a7700 1 -- 172.21.15.167:0/3205430407 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:56 5.6 5:60000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x5603211c6cf0 con 0x7fd5d8010070 2023-10-05T19:29:40.675 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.670+0000 7fd5e85a5700 1 -- 172.21.15.167:0/3205430407 <== osd.2 v2:172.21.15.167:6816/30037 9 ==== osd_op_reply(56 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fd5d000f560 con 0x7fd5d8010070 2023-10-05T19:29:40.675 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.670+0000 7fd5e85a5700 1 -- 172.21.15.167:0/3205430407 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:57 5.16 5:68000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x5603211c6cf0 con 0x5603211c7650 2023-10-05T19:29:40.676 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.670+0000 7fd5e8da6700 1 -- 172.21.15.167:0/3205430407 <== osd.3 v2:172.21.15.167:6808/30036 13 ==== osd_op_reply(57 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fd5d802b070 con 0x5603211c7650 2023-10-05T19:29:40.676 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.670+0000 7fd5e8da6700 1 -- 172.21.15.167:0/3205430407 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:58 5.e 5:70000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x5603211c6cf0 con 0x5603211c7650 2023-10-05T19:29:40.676 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.674+0000 7fd5e8da6700 1 -- 172.21.15.167:0/3205430407 <== osd.3 v2:172.21.15.167:6808/30036 14 ==== osd_op_reply(58 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fd5d802b070 con 0x5603211c7650 2023-10-05T19:29:40.676 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.674+0000 7fd5e8da6700 1 -- 172.21.15.167:0/3205430407 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:59 5.1e 5:78000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x5603211c6cf0 con 0x7fd5d000e8c0 2023-10-05T19:29:40.677 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.674+0000 7fd5e95a7700 1 -- 172.21.15.167:0/3205430407 <== osd.0 v2:172.21.15.118:6800/27690 15 ==== osd_op_reply(59 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fd5e002f080 con 0x7fd5d000e8c0 2023-10-05T19:29:40.678 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.674+0000 7fd5e95a7700 1 -- 172.21.15.167:0/3205430407 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:60 5.1 5:80000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x5603211c6cf0 con 0x5603211c7650 2023-10-05T19:29:40.678 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.674+0000 7fd5e8da6700 1 -- 172.21.15.167:0/3205430407 <== osd.3 v2:172.21.15.167:6808/30036 15 ==== osd_op_reply(60 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fd5d802b070 con 0x5603211c7650 2023-10-05T19:29:40.678 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.674+0000 7fd5e8da6700 1 -- 172.21.15.167:0/3205430407 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:61 5.11 5:88000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x5603211c6cf0 con 0x5603211c7650 2023-10-05T19:29:40.679 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.674+0000 7fd5e8da6700 1 -- 172.21.15.167:0/3205430407 <== osd.3 v2:172.21.15.167:6808/30036 16 ==== osd_op_reply(61 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fd5d802b070 con 0x5603211c7650 2023-10-05T19:29:40.679 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.674+0000 7fd5e8da6700 1 -- 172.21.15.167:0/3205430407 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:62 5.9 5:90000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x5603211c6cf0 con 0x5603211907f0 2023-10-05T19:29:40.680 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.674+0000 7fd5e95a7700 1 -- 172.21.15.167:0/3205430407 <== osd.1 v2:172.21.15.167:6800/30035 22 ==== osd_op_reply(62 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fd5e002f080 con 0x5603211907f0 2023-10-05T19:29:40.680 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.674+0000 7fd5e95a7700 1 -- 172.21.15.167:0/3205430407 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:63 5.19 5:98000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x5603211c6cf0 con 0x5603211907f0 2023-10-05T19:29:40.681 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.678+0000 7fd5e95a7700 1 -- 172.21.15.167:0/3205430407 <== osd.1 v2:172.21.15.167:6800/30035 23 ==== osd_op_reply(63 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fd5e002f080 con 0x5603211907f0 2023-10-05T19:29:40.681 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.678+0000 7fd5e95a7700 1 -- 172.21.15.167:0/3205430407 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:64 5.5 5:a0000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x5603211c6cf0 con 0x7fd5d000e8c0 2023-10-05T19:29:40.682 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.678+0000 7fd5e95a7700 1 -- 172.21.15.167:0/3205430407 <== osd.0 v2:172.21.15.118:6800/27690 16 ==== osd_op_reply(64 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fd5e002f080 con 0x7fd5d000e8c0 2023-10-05T19:29:40.682 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.678+0000 7fd5e95a7700 1 -- 172.21.15.167:0/3205430407 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:65 5.15 5:a8000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x5603211c6cf0 con 0x5603211c7650 2023-10-05T19:29:40.683 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.678+0000 7fd5e8da6700 1 -- 172.21.15.167:0/3205430407 <== osd.3 v2:172.21.15.167:6808/30036 17 ==== osd_op_reply(65 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fd5d802b070 con 0x5603211c7650 2023-10-05T19:29:40.683 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.678+0000 7fd5e8da6700 1 -- 172.21.15.167:0/3205430407 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:66 5.d 5:b0000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x5603211c6cf0 con 0x7fd5d8010070 2023-10-05T19:29:40.684 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.678+0000 7fd5e85a5700 1 -- 172.21.15.167:0/3205430407 <== osd.2 v2:172.21.15.167:6816/30037 10 ==== osd_op_reply(66 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fd5d000f560 con 0x7fd5d8010070 2023-10-05T19:29:40.684 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.678+0000 7fd5e85a5700 1 -- 172.21.15.167:0/3205430407 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:67 5.1d 5:b8000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x5603211c6cf0 con 0x5603211907f0 2023-10-05T19:29:40.685 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.682+0000 7fd5e95a7700 1 -- 172.21.15.167:0/3205430407 <== osd.1 v2:172.21.15.167:6800/30035 24 ==== osd_op_reply(67 [pgnls start_epoch 32 out=107b] v30'2 uv2 ondisk = 1) v8 ==== 144+0+107 (crc 0 0 0) 0x7fd5e002f080 con 0x5603211907f0 2023-10-05T19:29:40.685 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.682+0000 7fd5e95a7700 1 -- 172.21.15.167:0/3205430407 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:68 5.3 5:c0000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x5603211c6cf0 con 0x7fd5d000e8c0 2023-10-05T19:29:40.686 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.682+0000 7fd5e95a7700 1 -- 172.21.15.167:0/3205430407 <== osd.0 v2:172.21.15.118:6800/27690 17 ==== osd_op_reply(68 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fd5e002f080 con 0x7fd5d000e8c0 2023-10-05T19:29:40.686 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.682+0000 7fd5e95a7700 1 -- 172.21.15.167:0/3205430407 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:69 5.13 5:c8000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x5603211c6cf0 con 0x5603211c7650 2023-10-05T19:29:40.687 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.682+0000 7fd5e8da6700 1 -- 172.21.15.167:0/3205430407 <== osd.3 v2:172.21.15.167:6808/30036 18 ==== osd_op_reply(69 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fd5d802b070 con 0x5603211c7650 2023-10-05T19:29:40.687 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.682+0000 7fd5e8da6700 1 -- 172.21.15.167:0/3205430407 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:70 5.b 5:d0000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x5603211c6cf0 con 0x7fd5d000e8c0 2023-10-05T19:29:40.688 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.682+0000 7fd5e95a7700 1 -- 172.21.15.167:0/3205430407 <== osd.0 v2:172.21.15.118:6800/27690 18 ==== osd_op_reply(70 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fd5e002f080 con 0x7fd5d000e8c0 2023-10-05T19:29:40.688 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.682+0000 7fd5e95a7700 1 -- 172.21.15.167:0/3205430407 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:71 5.1b 5:d8000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x5603211c6cf0 con 0x7fd5d8010070 2023-10-05T19:29:40.689 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.686+0000 7fd5e85a5700 1 -- 172.21.15.167:0/3205430407 <== osd.2 v2:172.21.15.167:6816/30037 11 ==== osd_op_reply(71 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fd5d000f560 con 0x7fd5d8010070 2023-10-05T19:29:40.689 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.686+0000 7fd5e85a5700 1 -- 172.21.15.167:0/3205430407 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:72 5.7 5:e0000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x5603211c6cf0 con 0x5603211c7650 2023-10-05T19:29:40.689 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.686+0000 7fd5e8da6700 1 -- 172.21.15.167:0/3205430407 <== osd.3 v2:172.21.15.167:6808/30036 19 ==== osd_op_reply(72 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fd5d802b070 con 0x5603211c7650 2023-10-05T19:29:40.690 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.686+0000 7fd5e8da6700 1 -- 172.21.15.167:0/3205430407 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:73 5.17 5:e8000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x5603211c6cf0 con 0x7fd5d000e8c0 2023-10-05T19:29:40.691 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.686+0000 7fd5e95a7700 1 -- 172.21.15.167:0/3205430407 <== osd.0 v2:172.21.15.118:6800/27690 19 ==== osd_op_reply(73 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fd5e002f080 con 0x7fd5d000e8c0 2023-10-05T19:29:40.691 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.686+0000 7fd5e95a7700 1 -- 172.21.15.167:0/3205430407 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:74 5.f 5:f0000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x5603211c6cf0 con 0x5603211907f0 2023-10-05T19:29:40.691 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.686+0000 7fd5e95a7700 1 -- 172.21.15.167:0/3205430407 <== osd.1 v2:172.21.15.167:6800/30035 25 ==== osd_op_reply(74 [pgnls start_epoch 32 out=115b] v23'2 uv2 ondisk = 1) v8 ==== 144+0+115 (crc 0 0 0) 0x7fd5e002f080 con 0x5603211907f0 2023-10-05T19:29:40.692 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.686+0000 7fd5e95a7700 1 -- 172.21.15.167:0/3205430407 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:75 5.1f 5:f8000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x5603211c6cf0 con 0x7fd5d8010070 2023-10-05T19:29:40.693 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.690+0000 7fd5e85a5700 1 -- 172.21.15.167:0/3205430407 <== osd.2 v2:172.21.15.167:6816/30037 12 ==== osd_op_reply(75 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fd5d000f560 con 0x7fd5d8010070 2023-10-05T19:29:40.693 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.690+0000 7fd5eb271e40 20 RGWRados::pool_iterate: got zonegroup_info.4f1f719d-9648-417b-b10c-e5d939825419 2023-10-05T19:29:40.693 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.690+0000 7fd5eb271e40 20 RGWRados::pool_iterate: got zone_info.9a9df965-6612-45d4-a45f-6e3c889bacc2 2023-10-05T19:29:40.693 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.690+0000 7fd5eb271e40 20 RGWRados::pool_iterate: got zone_names.default 2023-10-05T19:29:40.693 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.690+0000 7fd5eb271e40 20 RGWRados::pool_iterate: got zonegroups_names.default 2023-10-05T19:29:40.693 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.690+0000 7fd5eb271e40 20 rados->read ofs=0 len=0 2023-10-05T19:29:40.693 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.690+0000 7fd5eb271e40 1 -- 172.21.15.167:0/3205430407 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:76 5.f 5:f43b6ece:::zone_names.default:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32) v8 -- 0x5603211cab90 con 0x5603211907f0 2023-10-05T19:29:40.694 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.690+0000 7fd5e95a7700 1 -- 172.21.15.167:0/3205430407 <== osd.1 v2:172.21.15.167:6800/30035 26 ==== osd_op_reply(76 zone_names.default [read 0~46 out=46b] v0'0 uv1 ondisk = 0) v8 ==== 162+0+46 (crc 0 0 0) 0x7fd5e002f080 con 0x5603211907f0 2023-10-05T19:29:40.694 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.690+0000 7fd5eb271e40 20 rados_obj.operate() r=0 bl.length=46 2023-10-05T19:29:40.694 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.690+0000 7fd5eb271e40 20 rados->read ofs=0 len=0 2023-10-05T19:29:40.694 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.690+0000 7fd5eb271e40 1 -- 172.21.15.167:0/3205430407 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:77 5.1d 5:b8374eb4:::zone_info.9a9df965-6612-45d4-a45f-6e3c889bacc2:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32) v8 -- 0x5603211caf40 con 0x5603211907f0 2023-10-05T19:29:40.694 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.690+0000 7fd5e95a7700 1 -- 172.21.15.167:0/3205430407 <== osd.1 v2:172.21.15.167:6800/30035 27 ==== osd_op_reply(77 zone_info.9a9df965-6612-45d4-a45f-6e3c889bacc2 [read 0~874 out=874b] v0'0 uv2 ondisk = 0) v8 ==== 190+0+874 (crc 0 0 0) 0x7fd5e002f080 con 0x5603211907f0 2023-10-05T19:29:40.694 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.690+0000 7fd5eb271e40 20 rados_obj.operate() r=0 bl.length=874 2023-10-05T19:29:40.695 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.690+0000 7fd5eb271e40 20 rados->read ofs=0 len=0 2023-10-05T19:29:40.695 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.690+0000 7fd5eb271e40 1 -- 172.21.15.167:0/3205430407 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:78 5.f 5:f4c53578:::zonegroups_names.default:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32) v8 -- 0x5603211c6860 con 0x5603211907f0 2023-10-05T19:29:40.696 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.690+0000 7fd5e95a7700 1 -- 172.21.15.167:0/3205430407 <== osd.1 v2:172.21.15.167:6800/30035 28 ==== osd_op_reply(78 zonegroups_names.default [read 0~46 out=46b] v0'0 uv2 ondisk = 0) v8 ==== 168+0+46 (crc 0 0 0) 0x7fd5e002f080 con 0x5603211907f0 2023-10-05T19:29:40.696 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.690+0000 7fd5eb271e40 20 rados_obj.operate() r=0 bl.length=46 2023-10-05T19:29:40.696 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.690+0000 7fd5eb271e40 20 rados->read ofs=0 len=0 2023-10-05T19:29:40.696 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.690+0000 7fd5eb271e40 1 -- 172.21.15.167:0/3205430407 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:79 5.c 5:30d299a5:::zonegroup_info.4f1f719d-9648-417b-b10c-e5d939825419:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32) v8 -- 0x5603211c6840 con 0x7fd5d000e8c0 2023-10-05T19:29:40.696 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.690+0000 7fd5e95a7700 1 -- 172.21.15.167:0/3205430407 <== osd.0 v2:172.21.15.118:6800/27690 20 ==== osd_op_reply(79 zonegroup_info.4f1f719d-9648-417b-b10c-e5d939825419 [read 0~362 out=362b] v0'0 uv1 ondisk = 0) v8 ==== 195+0+362 (crc 0 0 0) 0x7fd5e002f080 con 0x7fd5d000e8c0 2023-10-05T19:29:40.696 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.690+0000 7fd5eb271e40 20 rados_obj.operate() r=0 bl.length=362 2023-10-05T19:29:40.696 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.690+0000 7fd5eb271e40 20 rados->read ofs=0 len=0 2023-10-05T19:29:40.696 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.690+0000 7fd5eb271e40 1 -- 172.21.15.167:0/3205430407 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:80 5.3 5:c0ad8cbd:::region_map:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32) v8 -- 0x5603211cb480 con 0x7fd5d000e8c0 2023-10-05T19:29:40.697 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.694+0000 7fd5e95a7700 1 -- 172.21.15.167:0/3205430407 <== osd.0 v2:172.21.15.118:6800/27690 21 ==== osd_op_reply(80 region_map [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 ==== 154+0+0 (crc 0 0 0) 0x7fd5e002f080 con 0x7fd5d000e8c0 2023-10-05T19:29:40.697 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.694+0000 7fd5eb271e40 20 rados_obj.operate() r=-2 bl.length=0 2023-10-05T19:29:40.697 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.694+0000 7fd5eb271e40 20 zone default found 2023-10-05T19:29:40.697 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.694+0000 7fd5eb271e40 4 Realm: () 2023-10-05T19:29:40.698 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.694+0000 7fd5eb271e40 4 ZoneGroup: default (4f1f719d-9648-417b-b10c-e5d939825419) 2023-10-05T19:29:40.698 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.694+0000 7fd5eb271e40 4 Zone: default (9a9df965-6612-45d4-a45f-6e3c889bacc2) 2023-10-05T19:29:40.698 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.694+0000 7fd5eb271e40 10 cannot find current period zonegroup using local zonegroup configuration 2023-10-05T19:29:40.698 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.694+0000 7fd5eb271e40 20 zonegroup default 2023-10-05T19:29:40.698 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.694+0000 7fd5eb271e40 20 rados->read ofs=0 len=0 2023-10-05T19:29:40.698 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.694+0000 7fd5eb271e40 1 -- 172.21.15.167:0/3205430407 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:81 5.3 5:c52100b6:::period_config.default:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32) v8 -- 0x5603211cb480 con 0x7fd5d000e8c0 2023-10-05T19:29:40.699 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.694+0000 7fd5e95a7700 1 -- 172.21.15.167:0/3205430407 <== osd.0 v2:172.21.15.118:6800/27690 22 ==== osd_op_reply(81 period_config.default [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 ==== 165+0+0 (crc 0 0 0) 0x7fd5e002f080 con 0x7fd5d000e8c0 2023-10-05T19:29:40.699 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.694+0000 7fd5eb271e40 20 rados_obj.operate() r=-2 bl.length=0 2023-10-05T19:29:40.699 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.694+0000 7fd5eb271e40 20 rados->read ofs=0 len=0 2023-10-05T19:29:40.699 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.694+0000 7fd5eb271e40 1 -- 172.21.15.167:0/3205430407 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:82 6.2 6:4347d321:::bucket.sync-source-hints.:head [call version.read in=11b,read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32) v8 -- 0x5603211ccdb0 con 0x5603211c7650 2023-10-05T19:29:40.699 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.694+0000 7fd5e8da6700 1 -- 172.21.15.167:0/3205430407 <== osd.3 v2:172.21.15.167:6808/30036 20 ==== osd_op_reply(82 bucket.sync-source-hints. [call,read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 ==== 211+0+0 (crc 0 0 0) 0x7fd5d802b070 con 0x5603211c7650 2023-10-05T19:29:40.699 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.694+0000 7fd5eb271e40 20 rados_obj.operate() r=-2 bl.length=0 2023-10-05T19:29:40.699 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.694+0000 7fd5eb271e40 20 rados->read ofs=0 len=0 2023-10-05T19:29:40.700 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.694+0000 7fd5eb271e40 1 -- 172.21.15.167:0/3205430407 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:83 6.b 6:d467b91b:::bucket.sync-target-hints.:head [call version.read in=11b,read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32) v8 -- 0x5603211cd160 con 0x5603211c7650 2023-10-05T19:29:40.700 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.694+0000 7fd5e8da6700 1 -- 172.21.15.167:0/3205430407 <== osd.3 v2:172.21.15.167:6808/30036 21 ==== osd_op_reply(83 bucket.sync-target-hints. [call,read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 ==== 211+0+0 (crc 0 0 0) 0x7fd5d802b070 con 0x5603211c7650 2023-10-05T19:29:40.700 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.694+0000 7fd5eb271e40 20 rados_obj.operate() r=-2 bl.length=0 2023-10-05T19:29:40.700 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.694+0000 7fd5eb271e40 20 started sync module instance, tier type = 2023-10-05T19:29:40.700 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.694+0000 7fd5eb271e40 20 started zone id=9a9df965-6612-45d4-a45f-6e3c889bacc2 (name=default) with tier type = 2023-10-05T19:29:40.701 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.694+0000 7fd5eb271e40 1 -- 172.21.15.167:0/3205430407 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:84 7.1f 7:f95f44c2:::notify.0:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e32) v8 -- 0x5603211cc470 con 0x7fd5d8010070 2023-10-05T19:29:40.702 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.698+0000 7fd5e85a5700 1 -- 172.21.15.167:0/3205430407 <== osd.2 v2:172.21.15.167:6816/30037 13 ==== osd_op_reply(84 notify.0 [create] v32'252 uv252 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7fd5d000f560 con 0x7fd5d8010070 2023-10-05T19:29:40.702 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.698+0000 7fd5eb271e40 1 -- 172.21.15.167:0/3205430407 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:85 7.1f 7:f95f44c2:::notify.0:head [watch watch cookie 94571440422784] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e32) v8 -- 0x5603211cd510 con 0x7fd5d8010070 2023-10-05T19:29:40.703 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.698+0000 7fd5eb271e40 1 -- 172.21.15.167:0/3205430407 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:86 7.0 7:05bf5b68:::notify.1:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e32) v8 -- 0x5603211cddb0 con 0x5603211907f0 2023-10-05T19:29:40.705 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.702+0000 7fd5e95a7700 1 -- 172.21.15.167:0/3205430407 <== osd.1 v2:172.21.15.167:6800/30035 29 ==== osd_op_reply(86 notify.1 [create] v32'252 uv252 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7fd5e002f080 con 0x5603211907f0 2023-10-05T19:29:40.706 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.702+0000 7fd5e85a5700 1 -- 172.21.15.167:0/3205430407 <== osd.2 v2:172.21.15.167:6816/30037 14 ==== osd_op_reply(85 notify.0 [watch watch cookie 94571440422784] v32'253 uv252 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7fd5d000f560 con 0x7fd5d8010070 2023-10-05T19:29:40.706 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.702+0000 7fd5eb271e40 1 -- 172.21.15.167:0/3205430407 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:87 7.0 7:05bf5b68:::notify.1:head [watch watch cookie 94571440429536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e32) v8 -- 0x5603211ce360 con 0x5603211907f0 2023-10-05T19:29:40.706 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.702+0000 7fd5eb271e40 1 -- 172.21.15.167:0/3205430407 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:88 7.15 7:a93a5511:::notify.2:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e32) v8 -- 0x5603211ce850 con 0x5603211c7650 2023-10-05T19:29:40.708 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.702+0000 7fd5e95a7700 1 -- 172.21.15.167:0/3205430407 <== osd.1 v2:172.21.15.167:6800/30035 30 ==== osd_op_reply(87 notify.1 [watch watch cookie 94571440429536] v32'253 uv252 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7fd5e0050040 con 0x5603211907f0 2023-10-05T19:29:40.709 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.706+0000 7fd5e8da6700 1 -- 172.21.15.167:0/3205430407 <== osd.3 v2:172.21.15.167:6808/30036 22 ==== osd_op_reply(88 notify.2 [create] v32'252 uv252 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7fd5d802b070 con 0x5603211c7650 2023-10-05T19:29:40.709 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.706+0000 7fd5eb271e40 1 -- 172.21.15.167:0/3205430407 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:89 7.15 7:a93a5511:::notify.2:head [watch watch cookie 94571440425072] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e32) v8 -- 0x5603211cec50 con 0x5603211c7650 2023-10-05T19:29:40.709 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.706+0000 7fd5eb271e40 1 -- 172.21.15.167:0/3205430407 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:90 7.e 7:7759931f:::notify.3:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e32) v8 -- 0x5603211cf4f0 con 0x7fd5d8010070 2023-10-05T19:29:40.711 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.706+0000 7fd5e85a5700 1 -- 172.21.15.167:0/3205430407 <== osd.2 v2:172.21.15.167:6816/30037 15 ==== osd_op_reply(90 notify.3 [create] v32'252 uv252 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7fd5d000f560 con 0x7fd5d8010070 2023-10-05T19:29:40.712 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.706+0000 7fd5eb271e40 1 -- 172.21.15.167:0/3205430407 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:91 7.e 7:7759931f:::notify.3:head [watch watch cookie 94571440437408] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e32) v8 -- 0x5603211d01e0 con 0x7fd5d8010070 2023-10-05T19:29:40.712 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.706+0000 7fd5e8da6700 1 -- 172.21.15.167:0/3205430407 <== osd.3 v2:172.21.15.167:6808/30036 23 ==== osd_op_reply(89 notify.2 [watch watch cookie 94571440425072] v32'253 uv252 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7fd5d802b070 con 0x5603211c7650 2023-10-05T19:29:40.712 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.706+0000 7fd5eb271e40 1 -- 172.21.15.167:0/3205430407 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:92 7.d 7:b4812045:::notify.4:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e32) v8 -- 0x5603211d0ac0 con 0x5603211907f0 2023-10-05T19:29:40.714 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.710+0000 7fd5e85a5700 1 -- 172.21.15.167:0/3205430407 <== osd.2 v2:172.21.15.167:6816/30037 16 ==== osd_op_reply(91 notify.3 [watch watch cookie 94571440437408] v32'253 uv252 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7fd5d000f560 con 0x7fd5d8010070 2023-10-05T19:29:40.714 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.710+0000 7fd5e95a7700 1 -- 172.21.15.167:0/3205430407 <== osd.1 v2:172.21.15.167:6800/30035 31 ==== osd_op_reply(92 notify.4 [create] v32'252 uv252 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7fd5e0050040 con 0x5603211907f0 2023-10-05T19:29:40.714 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.710+0000 7fd5eb271e40 1 -- 172.21.15.167:0/3205430407 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:93 7.d 7:b4812045:::notify.4:head [watch watch cookie 94571440438512] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e32) v8 -- 0x5603211d07c0 con 0x5603211907f0 2023-10-05T19:29:40.715 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.710+0000 7fd5eb271e40 1 -- 172.21.15.167:0/3205430407 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:94 7.3 7:c609908c:::notify.5:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e32) v8 -- 0x5603211d1240 con 0x7fd5d000e8c0 2023-10-05T19:29:40.717 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.714+0000 7fd5e95a7700 1 -- 172.21.15.167:0/3205430407 <== osd.1 v2:172.21.15.167:6800/30035 32 ==== osd_op_reply(93 notify.4 [watch watch cookie 94571440438512] v32'253 uv252 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7fd5e0050040 con 0x5603211907f0 2023-10-05T19:29:40.717 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.714+0000 7fd5e95a7700 1 -- 172.21.15.167:0/3205430407 <== osd.0 v2:172.21.15.118:6800/27690 23 ==== osd_op_reply(94 notify.5 [create] v32'252 uv252 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7fd5e0050040 con 0x7fd5d000e8c0 2023-10-05T19:29:40.718 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.714+0000 7fd5eb271e40 1 -- 172.21.15.167:0/3205430407 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:95 7.3 7:c609908c:::notify.5:head [watch watch cookie 94571440430640] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e32) v8 -- 0x5603211d1200 con 0x7fd5d000e8c0 2023-10-05T19:29:40.718 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.714+0000 7fd5eb271e40 1 -- 172.21.15.167:0/3205430407 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:96 7.14 7:2b04a3e9:::notify.6:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e32) v8 -- 0x5603211d1ca0 con 0x7fd5d000e8c0 2023-10-05T19:29:40.720 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.714+0000 7fd5e95a7700 1 -- 172.21.15.167:0/3205430407 <== osd.0 v2:172.21.15.118:6800/27690 24 ==== osd_op_reply(96 notify.6 [create] v32'252 uv252 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7fd5e0050040 con 0x7fd5d000e8c0 2023-10-05T19:29:40.721 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.718+0000 7fd5eb271e40 1 -- 172.21.15.167:0/3205430407 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:97 7.14 7:2b04a3e9:::notify.6:head [watch watch cookie 94571440445360] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e32) v8 -- 0x5603211d20f0 con 0x7fd5d000e8c0 2023-10-05T19:29:40.721 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.718+0000 7fd5eb271e40 1 -- 172.21.15.167:0/3205430407 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:98 7.9 7:93e5b521:::notify.7:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e32) v8 -- 0x5603211d2b40 con 0x7fd5d000e8c0 2023-10-05T19:29:40.721 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.718+0000 7fd5e95a7700 1 -- 172.21.15.167:0/3205430407 <== osd.0 v2:172.21.15.118:6800/27690 25 ==== osd_op_reply(95 notify.5 [watch watch cookie 94571440430640] v32'253 uv252 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7fd5e0050040 con 0x7fd5d000e8c0 2023-10-05T19:29:40.723 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.718+0000 7fd5e95a7700 1 -- 172.21.15.167:0/3205430407 <== osd.0 v2:172.21.15.118:6800/27690 26 ==== osd_op_reply(98 notify.7 [create] v32'252 uv252 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7fd5e0050040 con 0x7fd5d000e8c0 2023-10-05T19:29:40.723 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.718+0000 7fd5eb271e40 1 -- 172.21.15.167:0/3205430407 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:99 7.9 7:93e5b521:::notify.7:head [watch watch cookie 94571440442640] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e32) v8 -- 0x5603211d2b00 con 0x7fd5d000e8c0 2023-10-05T19:29:40.723 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.718+0000 7fd5eb271e40 20 add_watcher() i=0 2023-10-05T19:29:40.724 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.718+0000 7fd5eb271e40 20 add_watcher() i=1 2023-10-05T19:29:40.724 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.718+0000 7fd5eb271e40 20 add_watcher() i=2 2023-10-05T19:29:40.724 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.718+0000 7fd5eb271e40 20 add_watcher() i=3 2023-10-05T19:29:40.724 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.718+0000 7fd5eb271e40 20 add_watcher() i=4 2023-10-05T19:29:40.724 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.718+0000 7fd5eb271e40 20 add_watcher() i=5 2023-10-05T19:29:40.724 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.718+0000 7fd5e95a7700 1 -- 172.21.15.167:0/3205430407 <== osd.0 v2:172.21.15.118:6800/27690 27 ==== osd_op_reply(97 notify.6 [watch watch cookie 94571440445360] v32'253 uv252 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7fd5e0050040 con 0x7fd5d000e8c0 2023-10-05T19:29:40.724 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.718+0000 7fd5eb271e40 20 add_watcher() i=6 2023-10-05T19:29:40.726 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.722+0000 7fd5e95a7700 1 -- 172.21.15.167:0/3205430407 <== osd.0 v2:172.21.15.118:6800/27690 28 ==== osd_op_reply(99 notify.7 [watch watch cookie 94571440442640] v32'253 uv252 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7fd5e0050040 con 0x7fd5d000e8c0 2023-10-05T19:29:40.726 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.722+0000 7fd5eb271e40 20 add_watcher() i=7 2023-10-05T19:29:40.726 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.722+0000 7fd5eb271e40 2 all 8 watchers are set, enabling cache 2023-10-05T19:29:40.726 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.722+0000 7fd5eb271e40 1 -- 172.21.15.167:0/3205430407 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:100 6.4 6:22d26bf9:::data_loggenerations_metadata:head [call version.check_conds in=74b,call version.read in=11b,read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32) v8 -- 0x5603211d5b30 con 0x5603211907f0 2023-10-05T19:29:40.726 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.722+0000 7fd5e95a7700 1 -- 172.21.15.167:0/3205430407 <== osd.1 v2:172.21.15.167:6800/30035 33 ==== osd_op_reply(100 data_loggenerations_metadata [call,call out=48b,read 0~28 out=28b] v0'0 uv1 ondisk = 0) v8 ==== 256+0+76 (crc 0 0 0) 0x7fd5e0050040 con 0x5603211907f0 2023-10-05T19:29:40.727 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.722+0000 7fd5eb271e40 1 -- 172.21.15.167:0/3205430407 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:101 6.4 6:22d26bf9:::data_loggenerations_metadata:head [watch watch cookie 94571440449040] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e32) v8 -- 0x5603211d2f50 con 0x5603211907f0 2023-10-05T19:29:40.729 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.726+0000 7fd5e95a7700 1 -- 172.21.15.167:0/3205430407 <== osd.1 v2:172.21.15.167:6800/30035 34 ==== osd_op_reply(101 data_loggenerations_metadata [watch watch cookie 94571440449040] v32'241 uv1 ondisk = 0) v8 ==== 172+0+0 (crc 0 0 0) 0x7fd5e0050040 con 0x5603211907f0 2023-10-05T19:29:40.729 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.726+0000 7fd5ba7e4700 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2023-10-05T19:29:40.730 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.726+0000 7fd5eb271e40 20 check_secure_mon_conn(): auth registy supported: methods=[2] modes=[2,1] 2023-10-05T19:29:40.730 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.726+0000 7fd5eb271e40 20 check_secure_mon_conn(): mode 1 is insecure 2023-10-05T19:29:40.730 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.726+0000 7fd5eb271e40 1 Processor -- start 2023-10-05T19:29:40.730 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.726+0000 7fd5eb271e40 1 -- start start 2023-10-05T19:29:40.731 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.726+0000 7fd5eb271e40 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5603211e0000 0x5603211e03d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-05T19:29:40.731 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.726+0000 7fd5eb271e40 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x5603211e0910 con 0x5603211e0000 2023-10-05T19:29:40.731 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.726+0000 7fd5e85a5700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5603211e0000 0x5603211e03d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-05T19:29:40.731 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.726+0000 7fd5e85a5700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5603211e0000 0x5603211e03d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.118:3300/0 says I am v2:172.21.15.167:56890/0 (socket says 172.21.15.167:56890) 2023-10-05T19:29:40.731 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.726+0000 7fd5e85a5700 1 -- 172.21.15.167:0/2551587515 learned_addr learned my addr 172.21.15.167:0/2551587515 (peer_addr_for_me v2:172.21.15.167:0/0) 2023-10-05T19:29:40.732 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.726+0000 7fd5e85a5700 1 -- 172.21.15.167:0/2551587515 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5603211e0a50 con 0x5603211e0000 2023-10-05T19:29:40.732 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.726+0000 7fd5e85a5700 1 --2- 172.21.15.167:0/2551587515 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5603211e0000 0x5603211e03d0 secure :-1 s=READY pgs=642 cs=0 l=1 rev1=1 crypto rx=0x7fd5d0000fb0 tx=0x7fd5d0006000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=eee88cda0f500c36 server_cookie=0 in_seq=0 out_seq=0 2023-10-05T19:29:40.732 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.726+0000 7fd5b9fe3700 1 -- 172.21.15.167:0/2551587515 <== mon.0 v2:172.21.15.118:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fd5d000dac0 con 0x5603211e0000 2023-10-05T19:29:40.732 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.726+0000 7fd5b9fe3700 1 -- 172.21.15.167:0/2551587515 <== mon.0 v2:172.21.15.118:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd5d000dc20 con 0x5603211e0000 2023-10-05T19:29:40.732 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.726+0000 7fd5b9fe3700 1 -- 172.21.15.167:0/2551587515 <== mon.0 v2:172.21.15.118:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fd5d000ddb0 con 0x5603211e0000 2023-10-05T19:29:40.734 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.730+0000 7fd5eb271e40 1 -- 172.21.15.167:0/2551587515 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5603211e0000 msgr2=0x5603211e03d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-05T19:29:40.734 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.730+0000 7fd5eb271e40 1 --2- 172.21.15.167:0/2551587515 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5603211e0000 0x5603211e03d0 secure :-1 s=READY pgs=642 cs=0 l=1 rev1=1 crypto rx=0x7fd5d0000fb0 tx=0x7fd5d0006000 comp rx=0 tx=0).stop 2023-10-05T19:29:40.734 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.730+0000 7fd5eb271e40 1 -- 172.21.15.167:0/2551587515 shutdown_connections 2023-10-05T19:29:40.734 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.730+0000 7fd5eb271e40 1 --2- 172.21.15.167:0/2551587515 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5603211e0000 0x5603211e03d0 unknown :-1 s=CLOSED pgs=642 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-05T19:29:40.734 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.730+0000 7fd5eb271e40 1 -- 172.21.15.167:0/2551587515 >> 172.21.15.167:0/2551587515 conn(0x5603211db3c0 msgr2=0x5603211d9510 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-05T19:29:40.734 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.730+0000 7fd5eb271e40 1 -- 172.21.15.167:0/2551587515 shutdown_connections 2023-10-05T19:29:40.734 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.730+0000 7fd5eb271e40 1 -- 172.21.15.167:0/2551587515 wait complete. 2023-10-05T19:29:40.735 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.730+0000 7fd5eb271e40 1 Processor -- start 2023-10-05T19:29:40.735 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.730+0000 7fd5eb271e40 1 -- start start 2023-10-05T19:29:40.735 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.730+0000 7fd5eb271e40 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5603211e0000 0x5603211e5b40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-05T19:29:40.735 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.730+0000 7fd5eb271e40 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x5603211e6b40 con 0x5603211e0000 2023-10-05T19:29:40.735 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.730+0000 7fd5e85a5700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5603211e0000 0x5603211e5b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-05T19:29:40.736 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.730+0000 7fd5e85a5700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5603211e0000 0x5603211e5b40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.118:3300/0 says I am v2:172.21.15.167:56896/0 (socket says 172.21.15.167:56896) 2023-10-05T19:29:40.736 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.730+0000 7fd5e85a5700 1 -- 172.21.15.167:0/2343252391 learned_addr learned my addr 172.21.15.167:0/2343252391 (peer_addr_for_me v2:172.21.15.167:0/0) 2023-10-05T19:29:40.736 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.730+0000 7fd5e85a5700 1 -- 172.21.15.167:0/2343252391 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd5d0006070 con 0x5603211e0000 2023-10-05T19:29:40.736 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.730+0000 7fd5e85a5700 1 --2- 172.21.15.167:0/2343252391 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5603211e0000 0x5603211e5b40 secure :-1 s=READY pgs=643 cs=0 l=1 rev1=1 crypto rx=0x7fd5d0000fb0 tx=0x7fd5d0032000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-05T19:29:40.737 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.734+0000 7fd52bfff700 1 -- 172.21.15.167:0/2343252391 <== mon.0 v2:172.21.15.118:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fd5d003dba0 con 0x5603211e0000 2023-10-05T19:29:40.737 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.734+0000 7fd52bfff700 1 -- 172.21.15.167:0/2343252391 <== mon.0 v2:172.21.15.118:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd5d003dd00 con 0x5603211e0000 2023-10-05T19:29:40.737 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.734+0000 7fd5eb271e40 1 -- 172.21.15.167:0/2343252391 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x5603211e6140 con 0x5603211e0000 2023-10-05T19:29:40.737 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.734+0000 7fd52bfff700 1 -- 172.21.15.167:0/2343252391 <== mon.0 v2:172.21.15.118:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fd5d003de70 con 0x5603211e0000 2023-10-05T19:29:40.737 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.734+0000 7fd5eb271e40 1 -- 172.21.15.167:0/2343252391 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x5603211e6580 con 0x5603211e0000 2023-10-05T19:29:40.738 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.734+0000 7fd52bfff700 1 -- 172.21.15.167:0/2343252391 <== mon.0 v2:172.21.15.118:3300/0 4 ==== mgrmap(e 5) v1 ==== 40329+0+0 (secure 0 0 0) 0x7fd5d001a040 con 0x5603211e0000 2023-10-05T19:29:40.739 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.734+0000 7fd52bfff700 1 --2- 172.21.15.167:0/2343252391 >> [v2:172.21.15.167:6824/29889,v1:172.21.15.167:6825/29889] conn(0x7fd530030c00 0x7fd5300330b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-05T19:29:40.739 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.734+0000 7fd52bfff700 1 -- 172.21.15.167:0/2343252391 <== mon.0 v2:172.21.15.118:3300/0 5 ==== osd_map(32..32 src has 1..32) v4 ==== 6260+0+0 (secure 0 0 0) 0x7fd5d0048d60 con 0x5603211e0000 2023-10-05T19:29:40.739 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.734+0000 7fd5e95a7700 1 --2- 172.21.15.167:0/2343252391 >> [v2:172.21.15.167:6824/29889,v1:172.21.15.167:6825/29889] conn(0x7fd530030c00 0x7fd5300330b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-05T19:29:40.740 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.734+0000 7fd5eb271e40 1 --2- 172.21.15.167:0/2343252391 >> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] conn(0x5603211f0730 0x5603211f2b40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-05T19:29:40.740 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.734+0000 7fd5eb271e40 1 -- 172.21.15.167:0/2343252391 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:1 6.19 6:9a6dd8c0:gc::gc.0:head [create,call version.check_conds in=74b,call rgw_gc.rgw_gc_queue_init in=45b,call version.set in=34b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e32) v8 -- 0x5603211f3190 con 0x5603211f0730 2023-10-05T19:29:40.740 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.734+0000 7fd5e8da6700 1 --2- 172.21.15.167:0/2343252391 >> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] conn(0x5603211f0730 0x5603211f2b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-05T19:29:40.740 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.734+0000 7fd5e95a7700 1 --2- 172.21.15.167:0/2343252391 >> [v2:172.21.15.167:6824/29889,v1:172.21.15.167:6825/29889] conn(0x7fd530030c00 0x7fd5300330b0 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x5603211d1e70 tx=0x7fd5e0000f90 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-05T19:29:40.740 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.738+0000 7fd5e8da6700 1 --2- 172.21.15.167:0/2343252391 >> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] conn(0x5603211f0730 0x5603211f2b40 crc :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-05T19:29:40.743 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.738+0000 7fd5e8da6700 1 -- 172.21.15.167:0/2343252391 <== osd.1 v2:172.21.15.167:6800/30035 1 ==== osd_op_reply(1 gc.0 [create,call,call,call] v32'38 uv33 ondisk = -125 ((125) Operation canceled)) v8 ==== 274+0+0 (crc 0 0 0) 0x7fd5d802b070 con 0x5603211f0730 2023-10-05T19:29:40.743 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.738+0000 7fd5eb271e40 1 --2- 172.21.15.167:0/2343252391 >> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] conn(0x5603211ec370 0x5603211f6800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-05T19:29:40.743 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.738+0000 7fd5eb271e40 1 -- 172.21.15.167:0/2343252391 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:2 6.18 6:1e85421f:gc::gc.1:head [create,call version.check_conds in=74b,call rgw_gc.rgw_gc_queue_init in=45b,call version.set in=34b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e32) v8 -- 0x5603211f6d40 con 0x5603211ec370 2023-10-05T19:29:40.744 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.738+0000 7fd5e85a5700 1 --2- 172.21.15.167:0/2343252391 >> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] conn(0x5603211ec370 0x5603211f6800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-05T19:29:40.744 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.738+0000 7fd5e85a5700 1 --2- 172.21.15.167:0/2343252391 >> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] conn(0x5603211ec370 0x5603211f6800 crc :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-05T19:29:40.747 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.742+0000 7fd5e85a5700 1 -- 172.21.15.167:0/2343252391 <== osd.0 v2:172.21.15.118:6800/27690 1 ==== osd_op_reply(2 gc.1 [create,call,call,call] v32'30 uv25 ondisk = -125 ((125) Operation canceled)) v8 ==== 274+0+0 (crc 0 0 0) 0x7fd5d000f560 con 0x5603211ec370 2023-10-05T19:29:40.747 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.742+0000 7fd5eb271e40 1 --2- 172.21.15.167:0/2343252391 >> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] conn(0x5603211f7a10 0x5603211f9e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-05T19:29:40.747 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.742+0000 7fd5eb271e40 1 -- 172.21.15.167:0/2343252391 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:3 6.b 6:d242335b:gc::gc.2:head [create,call version.check_conds in=74b,call rgw_gc.rgw_gc_queue_init in=45b,call version.set in=34b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e32) v8 -- 0x5603211fa550 con 0x5603211f7a10 2023-10-05T19:29:40.747 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.742+0000 7fd5e95a7700 1 --2- 172.21.15.167:0/2343252391 >> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] conn(0x5603211f7a10 0x5603211f9e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-05T19:29:40.748 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.742+0000 7fd5e95a7700 1 --2- 172.21.15.167:0/2343252391 >> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] conn(0x5603211f7a10 0x5603211f9e20 crc :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-05T19:29:40.751 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.746+0000 7fd5e95a7700 1 -- 172.21.15.167:0/2343252391 <== osd.3 v2:172.21.15.167:6808/30036 1 ==== osd_op_reply(3 gc.2 [create,call,call,call] v32'26 uv21 ondisk = -125 ((125) Operation canceled)) v8 ==== 274+0+0 (crc 0 0 0) 0x7fd5e0050040 con 0x5603211f7a10 2023-10-05T19:29:40.751 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.746+0000 7fd5eb271e40 1 -- 172.21.15.167:0/2343252391 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:4 6.4 6:2166d8e1:gc::gc.3:head [create,call version.check_conds in=74b,call rgw_gc.rgw_gc_queue_init in=45b,call version.set in=34b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e32) v8 -- 0x5603211faba0 con 0x5603211f0730 2023-10-05T19:29:40.754 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.750+0000 7fd5e8da6700 1 -- 172.21.15.167:0/2343252391 <== osd.1 v2:172.21.15.167:6800/30035 2 ==== osd_op_reply(4 gc.3 [create,call,call,call] v32'242 uv184 ondisk = -125 ((125) Operation canceled)) v8 ==== 274+0+0 (crc 0 0 0) 0x7fd5d802b070 con 0x5603211f0730 2023-10-05T19:29:40.754 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.750+0000 7fd5eb271e40 1 --2- 172.21.15.167:0/2343252391 >> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] conn(0x5603211fb840 0x5603211fdc20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-05T19:29:40.754 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.750+0000 7fd5eb271e40 1 -- 172.21.15.167:0/2343252391 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:5 6.3 6:c60da066:gc::gc.4:head [create,call version.check_conds in=74b,call rgw_gc.rgw_gc_queue_init in=45b,call version.set in=34b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e32) v8 -- 0x5603211fe2d0 con 0x5603211fb840 2023-10-05T19:29:40.754 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.750+0000 7fd5e8da6700 1 --2- 172.21.15.167:0/2343252391 >> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] conn(0x5603211fb840 0x5603211fdc20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-05T19:29:40.755 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.750+0000 7fd5e8da6700 1 --2- 172.21.15.167:0/2343252391 >> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] conn(0x5603211fb840 0x5603211fdc20 crc :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-05T19:29:40.757 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.754+0000 7fd5e8da6700 1 -- 172.21.15.167:0/2343252391 <== osd.2 v2:172.21.15.167:6816/30037 1 ==== osd_op_reply(5 gc.4 [create,call,call,call] v32'68 uv51 ondisk = -125 ((125) Operation canceled)) v8 ==== 274+0+0 (crc 0 0 0) 0x7fd5d802b070 con 0x5603211fb840 2023-10-05T19:29:40.758 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.754+0000 7fd5eb271e40 1 -- 172.21.15.167:0/2343252391 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:6 6.9 6:954be8d2:gc::gc.5:head [create,call version.check_conds in=74b,call rgw_gc.rgw_gc_queue_init in=45b,call version.set in=34b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e32) v8 -- 0x5603211fe8f0 con 0x5603211f7a10 2023-10-05T19:29:40.760 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.754+0000 7fd5e95a7700 1 -- 172.21.15.167:0/2343252391 <== osd.3 v2:172.21.15.167:6808/30036 2 ==== osd_op_reply(6 gc.5 [create,call,call,call] v32'48 uv37 ondisk = -125 ((125) Operation canceled)) v8 ==== 274+0+0 (crc 0 0 0) 0x7fd5e0050040 con 0x5603211f7a10 2023-10-05T19:29:40.760 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.754+0000 7fd5eb271e40 1 -- 172.21.15.167:0/2343252391 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:7 6.e 6:748722ff:gc::gc.6:head [create,call version.check_conds in=74b,call rgw_gc.rgw_gc_queue_init in=45b,call version.set in=34b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e32) v8 -- 0x5603211fef10 con 0x5603211f0730 2023-10-05T19:29:40.763 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.758+0000 7fd5e8da6700 1 -- 172.21.15.167:0/2343252391 <== osd.1 v2:172.21.15.167:6800/30035 3 ==== osd_op_reply(7 gc.6 [create,call,call,call] v32'28 uv23 ondisk = -125 ((125) Operation canceled)) v8 ==== 274+0+0 (crc 0 0 0) 0x7fd5d802b070 con 0x5603211f0730 2023-10-05T19:29:40.763 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.758+0000 7fd5eb271e40 1 -- 172.21.15.167:0/2343252391 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:8 6.1d 6:beaa4be4:gc::gc.7:head [create,call version.check_conds in=74b,call rgw_gc.rgw_gc_queue_init in=45b,call version.set in=34b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e32) v8 -- 0x5603211fef10 con 0x5603211ec370 2023-10-05T19:29:40.766 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.762+0000 7fd5e85a5700 1 -- 172.21.15.167:0/2343252391 <== osd.0 v2:172.21.15.118:6800/27690 2 ==== osd_op_reply(8 gc.7 [create,call,call,call] v32'45 uv34 ondisk = -125 ((125) Operation canceled)) v8 ==== 274+0+0 (crc 0 0 0) 0x7fd5d000f560 con 0x5603211ec370 2023-10-05T19:29:40.766 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.762+0000 7fd5eb271e40 1 -- 172.21.15.167:0/2343252391 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:9 6.12 6:4f78e9a2:gc::gc.8:head [create,call version.check_conds in=74b,call rgw_gc.rgw_gc_queue_init in=45b,call version.set in=34b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e32) v8 -- 0x5603211ff530 con 0x5603211fb840 2023-10-05T19:29:40.769 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.766+0000 7fd5e8da6700 1 -- 172.21.15.167:0/2343252391 <== osd.2 v2:172.21.15.167:6816/30037 2 ==== osd_op_reply(9 gc.8 [create,call,call,call] v32'26 uv21 ondisk = -125 ((125) Operation canceled)) v8 ==== 274+0+0 (crc 0 0 0) 0x7fd5d802b070 con 0x5603211fb840 2023-10-05T19:29:40.769 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.766+0000 7fd5eb271e40 1 -- 172.21.15.167:0/2343252391 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:10 6.1c 6:3e426232:gc::gc.9:head [create,call version.check_conds in=74b,call rgw_gc.rgw_gc_queue_init in=45b,call version.set in=34b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e32) v8 -- 0x5603211ff530 con 0x5603211f7a10 2023-10-05T19:29:40.772 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.766+0000 7fd5e95a7700 1 -- 172.21.15.167:0/2343252391 <== osd.3 v2:172.21.15.167:6808/30036 3 ==== osd_op_reply(10 gc.9 [create,call,call,call] v32'47 uv39 ondisk = -125 ((125) Operation canceled)) v8 ==== 274+0+0 (crc 0 0 0) 0x7fd5e0050040 con 0x5603211f7a10 2023-10-05T19:29:40.772 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.766+0000 7fd5eb271e40 1 -- 172.21.15.167:0/2343252391 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:11 6.3 6:c31f41d7:gc::gc.10:head [create,call version.check_conds in=74b,call rgw_gc.rgw_gc_queue_init in=45b,call version.set in=34b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e32) v8 -- 0x5603211ffa20 con 0x5603211fb840 2023-10-05T19:29:40.775 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.770+0000 7fd5e8da6700 1 -- 172.21.15.167:0/2343252391 <== osd.2 v2:172.21.15.167:6816/30037 3 ==== osd_op_reply(11 gc.10 [create,call,call,call] v32'69 uv53 ondisk = -125 ((125) Operation canceled)) v8 ==== 275+0+0 (crc 0 0 0) 0x7fd5d802b070 con 0x5603211fb840 2023-10-05T19:29:40.775 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.770+0000 7fd5eb271e40 1 -- 172.21.15.167:0/2343252391 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:12 6.13 6:c8333aab:gc::gc.11:head [create,call version.check_conds in=74b,call rgw_gc.rgw_gc_queue_init in=45b,call version.set in=34b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e32) v8 -- 0x5603211ffaf0 con 0x5603211ec370 2023-10-05T19:29:40.778 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.774+0000 7fd5e85a5700 1 -- 172.21.15.167:0/2343252391 <== osd.0 v2:172.21.15.118:6800/27690 3 ==== osd_op_reply(12 gc.11 [create,call,call,call] v32'31 uv25 ondisk = -125 ((125) Operation canceled)) v8 ==== 275+0+0 (crc 0 0 0) 0x7fd5d000f560 con 0x5603211ec370 2023-10-05T19:29:40.778 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.774+0000 7fd5eb271e40 1 -- 172.21.15.167:0/2343252391 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:13 6.15 6:ab569f81:gc::gc.12:head [create,call version.check_conds in=74b,call rgw_gc.rgw_gc_queue_init in=45b,call version.set in=34b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e32) v8 -- 0x560321200190 con 0x5603211fb840 2023-10-05T19:29:40.781 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.778+0000 7fd5e8da6700 1 -- 172.21.15.167:0/2343252391 <== osd.2 v2:172.21.15.167:6816/30037 4 ==== osd_op_reply(13 gc.12 [create,call,call,call] v32'26 uv21 ondisk = -125 ((125) Operation canceled)) v8 ==== 275+0+0 (crc 0 0 0) 0x7fd5d802b070 con 0x5603211fb840 2023-10-05T19:29:40.781 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.778+0000 7fd5eb271e40 1 -- 172.21.15.167:0/2343252391 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:14 6.0 6:0667bb13:gc::gc.13:head [create,call version.check_conds in=74b,call rgw_gc.rgw_gc_queue_init in=45b,call version.set in=34b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e32) v8 -- 0x560321200190 con 0x5603211ec370 2023-10-05T19:29:40.783 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.778+0000 7fd5e85a5700 1 -- 172.21.15.167:0/2343252391 <== osd.0 v2:172.21.15.118:6800/27690 4 ==== osd_op_reply(14 gc.13 [create,call,call,call] v32'51 uv40 ondisk = -125 ((125) Operation canceled)) v8 ==== 275+0+0 (crc 0 0 0) 0x7fd5d000f560 con 0x5603211ec370 2023-10-05T19:29:40.784 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.778+0000 7fd5eb271e40 1 -- 172.21.15.167:0/2343252391 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:15 6.4 6:227f63b5:gc::gc.14:head [create,call version.check_conds in=74b,call rgw_gc.rgw_gc_queue_init in=45b,call version.set in=34b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e32) v8 -- 0x5603212001c0 con 0x5603211f0730 2023-10-05T19:29:40.786 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.782+0000 7fd5e8da6700 1 -- 172.21.15.167:0/2343252391 <== osd.1 v2:172.21.15.167:6800/30035 4 ==== osd_op_reply(15 gc.14 [create,call,call,call] v32'243 uv186 ondisk = -125 ((125) Operation canceled)) v8 ==== 275+0+0 (crc 0 0 0) 0x7fd5d802b070 con 0x5603211f0730 2023-10-05T19:29:40.787 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.782+0000 7fd5eb271e40 1 -- 172.21.15.167:0/2343252391 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:16 6.7 6:e07a6548:gc::gc.15:head [create,call version.check_conds in=74b,call rgw_gc.rgw_gc_queue_init in=45b,call version.set in=34b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e32) v8 -- 0x560321200250 con 0x5603211f7a10 2023-10-05T19:29:40.790 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.786+0000 7fd5e95a7700 1 -- 172.21.15.167:0/2343252391 <== osd.3 v2:172.21.15.167:6808/30036 4 ==== osd_op_reply(16 gc.15 [create,call,call,call] v32'65 uv45 ondisk = -125 ((125) Operation canceled)) v8 ==== 275+0+0 (crc 0 0 0) 0x7fd5e0050040 con 0x5603211f7a10 2023-10-05T19:29:40.790 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.786+0000 7fd5eb271e40 1 -- 172.21.15.167:0/2343252391 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:17 6.3 6:c623b173:gc::gc.16:head [create,call version.check_conds in=74b,call rgw_gc.rgw_gc_queue_init in=45b,call version.set in=34b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e32) v8 -- 0x5603212002e0 con 0x5603211fb840 2023-10-05T19:29:40.793 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.790+0000 7fd5e8da6700 1 -- 172.21.15.167:0/2343252391 <== osd.2 v2:172.21.15.167:6816/30037 5 ==== osd_op_reply(17 gc.16 [create,call,call,call] v32'70 uv55 ondisk = -125 ((125) Operation canceled)) v8 ==== 275+0+0 (crc 0 0 0) 0x7fd5d802b070 con 0x5603211fb840 2023-10-05T19:29:40.793 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.790+0000 7fd5eb271e40 1 -- 172.21.15.167:0/2343252391 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:18 6.16 6:6849fd06:gc::gc.17:head [create,call version.check_conds in=74b,call rgw_gc.rgw_gc_queue_init in=45b,call version.set in=34b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e32) v8 -- 0x5603212002e0 con 0x5603211f7a10 2023-10-05T19:29:40.795 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.790+0000 7fd5e95a7700 1 -- 172.21.15.167:0/2343252391 <== osd.3 v2:172.21.15.167:6808/30036 5 ==== osd_op_reply(18 gc.17 [create,call,call,call] v32'26 uv21 ondisk = -125 ((125) Operation canceled)) v8 ==== 275+0+0 (crc 0 0 0) 0x7fd5e0050040 con 0x5603211f7a10 2023-10-05T19:29:40.795 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.790+0000 7fd5eb271e40 1 -- 172.21.15.167:0/2343252391 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:19 6.d 6:b019ea83:gc::gc.18:head [create,call version.check_conds in=74b,call rgw_gc.rgw_gc_queue_init in=45b,call version.set in=34b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e32) v8 -- 0x560321200370 con 0x5603211f0730 2023-10-05T19:29:40.798 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.794+0000 7fd5e8da6700 1 -- 172.21.15.167:0/2343252391 <== osd.1 v2:172.21.15.167:6800/30035 5 ==== osd_op_reply(19 gc.18 [create,call,call,call] v32'34 uv29 ondisk = -125 ((125) Operation canceled)) v8 ==== 275+0+0 (crc 0 0 0) 0x7fd5d802b070 con 0x5603211f0730 2023-10-05T19:29:40.798 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.794+0000 7fd5eb271e40 1 -- 172.21.15.167:0/2343252391 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:20 6.4 6:24c8e1ae:gc::gc.19:head [create,call version.check_conds in=74b,call rgw_gc.rgw_gc_queue_init in=45b,call version.set in=34b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e32) v8 -- 0x560321200370 con 0x5603211f0730 2023-10-05T19:29:40.800 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.798+0000 7fd5e8da6700 1 -- 172.21.15.167:0/2343252391 <== osd.1 v2:172.21.15.167:6800/30035 6 ==== osd_op_reply(20 gc.19 [create,call,call,call] v32'244 uv189 ondisk = -125 ((125) Operation canceled)) v8 ==== 275+0+0 (crc 0 0 0) 0x7fd5d802b070 con 0x5603211f0730 2023-10-05T19:29:40.800 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.798+0000 7fd5eb271e40 1 -- 172.21.15.167:0/2343252391 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:21 6.7 6:e622f76a:gc::gc.20:head [create,call version.check_conds in=74b,call rgw_gc.rgw_gc_queue_init in=45b,call version.set in=34b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e32) v8 -- 0x560321200370 con 0x5603211f7a10 2023-10-05T19:29:40.803 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.798+0000 7fd5e95a7700 1 -- 172.21.15.167:0/2343252391 <== osd.3 v2:172.21.15.167:6808/30036 6 ==== osd_op_reply(21 gc.20 [create,call,call,call] v32'66 uv47 ondisk = -125 ((125) Operation canceled)) v8 ==== 275+0+0 (crc 0 0 0) 0x7fd5e0050040 con 0x5603211f7a10 2023-10-05T19:29:40.803 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.798+0000 7fd5eb271e40 1 -- 172.21.15.167:0/2343252391 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:22 6.a 6:5127e642:gc::gc.21:head [create,call version.check_conds in=74b,call rgw_gc.rgw_gc_queue_init in=45b,call version.set in=34b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e32) v8 -- 0x560321200370 con 0x5603211fb840 2023-10-05T19:29:40.806 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.802+0000 7fd5e8da6700 1 -- 172.21.15.167:0/2343252391 <== osd.2 v2:172.21.15.167:6816/30037 6 ==== osd_op_reply(22 gc.21 [create,call,call,call] v32'36 uv31 ondisk = -125 ((125) Operation canceled)) v8 ==== 275+0+0 (crc 0 0 0) 0x7fd5d802b070 con 0x5603211fb840 2023-10-05T19:29:40.806 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.802+0000 7fd5eb271e40 1 -- 172.21.15.167:0/2343252391 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:23 6.11 6:8eb99c53:gc::gc.22:head [create,call version.check_conds in=74b,call rgw_gc.rgw_gc_queue_init in=45b,call version.set in=34b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e32) v8 -- 0x560321200370 con 0x5603211f7a10 2023-10-05T19:29:40.809 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.806+0000 7fd5e95a7700 1 -- 172.21.15.167:0/2343252391 <== osd.3 v2:172.21.15.167:6808/30036 7 ==== osd_op_reply(23 gc.22 [create,call,call,call] v32'32 uv23 ondisk = -125 ((125) Operation canceled)) v8 ==== 275+0+0 (crc 0 0 0) 0x7fd5e0050040 con 0x5603211f7a10 2023-10-05T19:29:40.809 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.806+0000 7fd5eb271e40 1 -- 172.21.15.167:0/2343252391 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:24 6.8 6:17d7e8ed:gc::gc.23:head [create,call version.check_conds in=74b,call rgw_gc.rgw_gc_queue_init in=45b,call version.set in=34b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e32) v8 -- 0x560321200370 con 0x5603211f7a10 2023-10-05T19:29:40.813 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.810+0000 7fd5e95a7700 1 -- 172.21.15.167:0/2343252391 <== osd.3 v2:172.21.15.167:6808/30036 8 ==== osd_op_reply(24 gc.23 [create,call,call,call] v32'39 uv33 ondisk = -125 ((125) Operation canceled)) v8 ==== 275+0+0 (crc 0 0 0) 0x7fd5e0050040 con 0x5603211f7a10 2023-10-05T19:29:40.813 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.810+0000 7fd5eb271e40 1 -- 172.21.15.167:0/2343252391 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:25 6.1b 6:db9d130d:gc::gc.24:head [create,call version.check_conds in=74b,call rgw_gc.rgw_gc_queue_init in=45b,call version.set in=34b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e32) v8 -- 0x560321200370 con 0x5603211f7a10 2023-10-05T19:29:40.815 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.810+0000 7fd5e95a7700 1 -- 172.21.15.167:0/2343252391 <== osd.3 v2:172.21.15.167:6808/30036 9 ==== osd_op_reply(25 gc.24 [create,call,call,call] v32'22 uv17 ondisk = -125 ((125) Operation canceled)) v8 ==== 275+0+0 (crc 0 0 0) 0x7fd5e0050040 con 0x5603211f7a10 2023-10-05T19:29:40.816 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.810+0000 7fd5eb271e40 1 -- 172.21.15.167:0/2343252391 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:26 6.9 6:921ecdb1:gc::gc.25:head [create,call version.check_conds in=74b,call rgw_gc.rgw_gc_queue_init in=45b,call version.set in=34b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e32) v8 -- 0x560321200370 con 0x5603211f7a10 2023-10-05T19:29:40.818 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.814+0000 7fd5e95a7700 1 -- 172.21.15.167:0/2343252391 <== osd.3 v2:172.21.15.167:6808/30036 10 ==== osd_op_reply(26 gc.25 [create,call,call,call] v32'49 uv39 ondisk = -125 ((125) Operation canceled)) v8 ==== 275+0+0 (crc 0 0 0) 0x7fd5e0050040 con 0x5603211f7a10 2023-10-05T19:29:40.818 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.814+0000 7fd5eb271e40 1 -- 172.21.15.167:0/2343252391 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:27 6.1f 6:fde4dd55:gc::gc.26:head [create,call version.check_conds in=74b,call rgw_gc.rgw_gc_queue_init in=45b,call version.set in=34b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e32) v8 -- 0x560321200370 con 0x5603211ec370 2023-10-05T19:29:40.821 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.818+0000 7fd5e85a5700 1 -- 172.21.15.167:0/2343252391 <== osd.0 v2:172.21.15.118:6800/27690 5 ==== osd_op_reply(27 gc.26 [create,call,call,call] v32'39 uv23 ondisk = -125 ((125) Operation canceled)) v8 ==== 275+0+0 (crc 0 0 0) 0x7fd5d000f560 con 0x5603211ec370 2023-10-05T19:29:40.821 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.818+0000 7fd5eb271e40 1 -- 172.21.15.167:0/2343252391 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:28 6.1a 6:5909e671:gc::gc.27:head [create,call version.check_conds in=74b,call rgw_gc.rgw_gc_queue_init in=45b,call version.set in=34b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e32) v8 -- 0x560321200370 con 0x5603211f7a10 2023-10-05T19:29:40.822 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.818+0000 7fd5e95a7700 1 -- 172.21.15.167:0/2343252391 <== osd.3 v2:172.21.15.167:6808/30036 11 ==== osd_op_reply(28 gc.27 [create,call,call,call] v32'26 uv21 ondisk = -125 ((125) Operation canceled)) v8 ==== 275+0+0 (crc 0 0 0) 0x7fd5e0050040 con 0x5603211f7a10 2023-10-05T19:29:40.822 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.818+0000 7fd5eb271e40 1 -- 172.21.15.167:0/2343252391 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:29 6.1d 6:b88e3271:gc::gc.28:head [create,call version.check_conds in=74b,call rgw_gc.rgw_gc_queue_init in=45b,call version.set in=34b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e32) v8 -- 0x560321200370 con 0x5603211ec370 2023-10-05T19:29:40.825 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.822+0000 7fd5e85a5700 1 -- 172.21.15.167:0/2343252391 <== osd.0 v2:172.21.15.118:6800/27690 6 ==== osd_op_reply(29 gc.28 [create,call,call,call] v32'46 uv36 ondisk = -125 ((125) Operation canceled)) v8 ==== 275+0+0 (crc 0 0 0) 0x7fd5d000f560 con 0x5603211ec370 2023-10-05T19:29:40.825 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.822+0000 7fd5eb271e40 1 -- 172.21.15.167:0/2343252391 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:30 6.7 6:e47c9322:gc::gc.29:head [create,call version.check_conds in=74b,call rgw_gc.rgw_gc_queue_init in=45b,call version.set in=34b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e32) v8 -- 0x560321200370 con 0x5603211f7a10 2023-10-05T19:29:40.827 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.822+0000 7fd5e95a7700 1 -- 172.21.15.167:0/2343252391 <== osd.3 v2:172.21.15.167:6808/30036 12 ==== osd_op_reply(30 gc.29 [create,call,call,call] v32'67 uv49 ondisk = -125 ((125) Operation canceled)) v8 ==== 275+0+0 (crc 0 0 0) 0x7fd5e0050040 con 0x5603211f7a10 2023-10-05T19:29:40.827 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.822+0000 7fd5eb271e40 1 -- 172.21.15.167:0/2343252391 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:31 6.0 6:013d6c33:gc::gc.30:head [create,call version.check_conds in=74b,call rgw_gc.rgw_gc_queue_init in=45b,call version.set in=34b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e32) v8 -- 0x560321200370 con 0x5603211ec370 2023-10-05T19:29:40.830 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.826+0000 7fd5e85a5700 1 -- 172.21.15.167:0/2343252391 <== osd.0 v2:172.21.15.118:6800/27690 7 ==== osd_op_reply(31 gc.30 [create,call,call,call] v32'52 uv42 ondisk = -125 ((125) Operation canceled)) v8 ==== 275+0+0 (crc 0 0 0) 0x7fd5d000f560 con 0x5603211ec370 2023-10-05T19:29:40.830 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.826+0000 7fd5eb271e40 1 -- 172.21.15.167:0/2343252391 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:32 6.f 6:f24cb4a8:gc::gc.31:head [create,call version.check_conds in=74b,call rgw_gc.rgw_gc_queue_init in=45b,call version.set in=34b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e32) v8 -- 0x560321200370 con 0x5603211fb840 2023-10-05T19:29:40.832 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.826+0000 7fd5e8da6700 1 -- 172.21.15.167:0/2343252391 <== osd.2 v2:172.21.15.167:6816/30037 7 ==== osd_op_reply(32 gc.31 [create,call,call,call] v32'32 uv27 ondisk = -125 ((125) Operation canceled)) v8 ==== 275+0+0 (crc 0 0 0) 0x7fd5d802b070 con 0x5603211fb840 2023-10-05T19:29:40.834 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.830+0000 7fd5297fa700 20 reqs_thread_entry: start 2023-10-05T19:29:40.835 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.830+0000 7fd522ffd700 20 reqs_thread_entry: start 2023-10-05T19:29:40.835 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.830+0000 7fd5eb271e40 20 init_complete bucket index max shards: 11 2023-10-05T19:29:40.835 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.830+0000 7fd5eb271e40 10 rgw notify: Started notification manager with: 1 workers 2023-10-05T19:29:40.836 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.830+0000 7fd5217fa700 20 rgw notify: INFO: next queues processing will happen at: Thu Oct 5 19:30:11 2023 2023-10-05T19:29:40.836 INFO:teuthology.orchestra.run.smithi167.stderr: 2023-10-05T19:29:40.836 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.830+0000 7fd520ff9700 20 reqs_thread_entry: start 2023-10-05T19:29:40.836 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.830+0000 7fd5eb271e40 20 get_system_obj_state: rctx=0x7ffeedd66f98 obj=default.rgw.meta:users.uid:bud state=0x5603210690b0 s->prefetch_data=0 2023-10-05T19:29:40.836 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.830+0000 7fd5eb271e40 10 cache get: name=default.rgw.meta+users.uid+bud : miss 2023-10-05T19:29:40.836 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.830+0000 7fd5eb271e40 1 -- 172.21.15.167:0/3205430407 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:102 8.4 8:267425a8:users.uid::bud:head [call version.read in=11b,getxattrs,stat] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32) v8 -- 0x56032125ee50 con 0x7fd5d8010070 2023-10-05T19:29:40.838 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.830+0000 7fd5e85a5700 1 -- 172.21.15.167:0/3205430407 <== osd.2 v2:172.21.15.167:6816/30037 17 ==== osd_op_reply(102 bud [call out=48b,getxattrs out=75b,stat out=16b] v0'0 uv2 ondisk = 0) v8 ==== 231+0+139 (crc 0 0 0) 0x7fd5d000f560 con 0x7fd5d8010070 2023-10-05T19:29:40.838 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.830+0000 7fd5eb271e40 10 cache put: name=default.rgw.meta+users.uid+bud info.flags=0x16 2023-10-05T19:29:40.838 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.830+0000 7fd5eb271e40 10 adding default.rgw.meta+users.uid+bud to cache LRU end 2023-10-05T19:29:40.838 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.830+0000 7fd5eb271e40 10 updating xattr: name=ceph.objclass.version bl.length()=42 2023-10-05T19:29:40.838 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.830+0000 7fd5eb271e40 20 get_system_obj_state: s->obj_tag was set empty 2023-10-05T19:29:40.838 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.830+0000 7fd5eb271e40 20 Read xattr: user.rgw.idtag 2023-10-05T19:29:40.838 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.830+0000 7fd5eb271e40 10 cache get: name=default.rgw.meta+users.uid+bud : type miss (requested=0x13, cached=0x16) 2023-10-05T19:29:40.839 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.830+0000 7fd5eb271e40 20 rados->read ofs=0 len=0 2023-10-05T19:29:40.841 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.830+0000 7fd5eb271e40 1 -- 172.21.15.167:0/3205430407 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:103 8.4 8:267425a8:users.uid::bud:head [call version.check_conds in=122b,call version.read in=11b,read 0~0,getxattrs] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32) v8 -- 0x5603212618e0 con 0x7fd5d8010070 2023-10-05T19:29:40.841 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.834+0000 7fd5e85a5700 1 -- 172.21.15.167:0/3205430407 <== osd.2 v2:172.21.15.167:6816/30037 18 ==== osd_op_reply(103 bud [call,call out=48b,read 0~346 out=346b,getxattrs out=75b] v0'0 uv2 ondisk = 0) v8 ==== 273+0+469 (crc 0 0 0) 0x7fd5d000f560 con 0x7fd5d8010070 2023-10-05T19:29:40.841 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.834+0000 7fd5eb271e40 20 rados_obj.operate() r=0 bl.length=346 2023-10-05T19:29:40.841 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.834+0000 7fd5eb271e40 10 cache put: name=default.rgw.meta+users.uid+bud info.flags=0x13 2023-10-05T19:29:40.841 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.834+0000 7fd5eb271e40 10 moving default.rgw.meta+users.uid+bud to cache LRU end 2023-10-05T19:29:40.841 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.834+0000 7fd5eb271e40 10 updating xattr: name=ceph.objclass.version bl.length()=42 2023-10-05T19:29:40.841 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.834+0000 7fd5eb271e40 1 -- 172.21.15.167:0/3205430407 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:104 8.19 8:99adfd5d:users.uid::bud.buckets:head [call user.list_buckets in=34b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32) v8 -- 0x5603212618e0 con 0x5603211907f0 2023-10-05T19:29:40.842 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.834+0000 7fd5e95a7700 1 -- 172.21.15.167:0/3205430407 <== osd.1 v2:172.21.15.167:6800/30035 35 ==== osd_op_reply(104 bud.buckets [call out=184b] v0'0 uv3 ondisk = 0) v8 ==== 155+0+184 (crc 0 0 0) 0x7fd5e0050040 con 0x5603211907f0 2023-10-05T19:29:40.842 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.834+0000 7fd5eb271e40 20 get_system_obj_state: rctx=0x7ffeedd66c40 obj=default.rgw.meta:root:.bucket.meta.myfoo6:9a9df965-6612-45d4-a45f-6e3c889bacc2.4223.7 state=0x5603210690b0 s->prefetch_data=0 2023-10-05T19:29:40.842 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.834+0000 7fd5eb271e40 10 cache get: name=default.rgw.meta+root+.bucket.meta.myfoo6:9a9df965-6612-45d4-a45f-6e3c889bacc2.4223.7 : miss 2023-10-05T19:29:40.842 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.834+0000 7fd5eb271e40 1 -- 172.21.15.167:0/3205430407 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:105 8.1f 8:fe511ebf:root::.bucket.meta.myfoo6%3a9a9df965-6612-45d4-a45f-6e3c889bacc2.4223.7:head [call version.read in=11b,getxattrs,stat] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32) v8 -- 0x5603212610f0 con 0x7fd5d8010070 2023-10-05T19:29:40.842 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.834+0000 7fd5e85a5700 1 -- 172.21.15.167:0/3205430407 <== osd.2 v2:172.21.15.167:6816/30037 19 ==== osd_op_reply(105 .bucket.meta.myfoo6:9a9df965-6612-45d4-a45f-6e3c889bacc2.4223.7 [call out=48b,getxattrs out=212b,stat out=16b] v0'0 uv2 ondisk = 0) v8 ==== 291+0+276 (crc 0 0 0) 0x7fd5d000f560 con 0x7fd5d8010070 2023-10-05T19:29:40.842 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.834+0000 7fd5eb271e40 10 cache put: name=default.rgw.meta+root+.bucket.meta.myfoo6:9a9df965-6612-45d4-a45f-6e3c889bacc2.4223.7 info.flags=0x16 2023-10-05T19:29:40.842 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.834+0000 7fd5eb271e40 10 adding default.rgw.meta+root+.bucket.meta.myfoo6:9a9df965-6612-45d4-a45f-6e3c889bacc2.4223.7 to cache LRU end 2023-10-05T19:29:40.842 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.834+0000 7fd5eb271e40 10 updating xattr: name=ceph.objclass.version bl.length()=42 2023-10-05T19:29:40.842 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.834+0000 7fd5eb271e40 10 updating xattr: name=user.rgw.acl bl.length()=117 2023-10-05T19:29:40.843 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.834+0000 7fd5eb271e40 20 get_system_obj_state: s->obj_tag was set empty 2023-10-05T19:29:40.843 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.834+0000 7fd5eb271e40 20 Read xattr: user.rgw.acl 2023-10-05T19:29:40.843 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.834+0000 7fd5eb271e40 20 Read xattr: user.rgw.idtag 2023-10-05T19:29:40.843 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.834+0000 7fd5eb271e40 10 cache get: name=default.rgw.meta+root+.bucket.meta.myfoo6:9a9df965-6612-45d4-a45f-6e3c889bacc2.4223.7 : type miss (requested=0x13, cached=0x16) 2023-10-05T19:29:40.843 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.834+0000 7fd5eb271e40 20 rados->read ofs=0 len=0 2023-10-05T19:29:40.843 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.834+0000 7fd5eb271e40 1 -- 172.21.15.167:0/3205430407 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:106 8.1f 8:fe511ebf:root::.bucket.meta.myfoo6%3a9a9df965-6612-45d4-a45f-6e3c889bacc2.4223.7:head [call version.check_conds in=122b,call version.read in=11b,read 0~0,getxattrs] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32) v8 -- 0x560321263a20 con 0x7fd5d8010070 2023-10-05T19:29:40.845 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.834+0000 7fd5e85a5700 1 -- 172.21.15.167:0/3205430407 <== osd.2 v2:172.21.15.167:6816/30037 20 ==== osd_op_reply(106 .bucket.meta.myfoo6:9a9df965-6612-45d4-a45f-6e3c889bacc2.4223.7 [call,call out=48b,read 0~353 out=353b,getxattrs out=212b] v0'0 uv2 ondisk = 0) v8 ==== 333+0+613 (crc 0 0 0) 0x7fd5d000f560 con 0x7fd5d8010070 2023-10-05T19:29:40.845 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.834+0000 7fd5eb271e40 20 rados_obj.operate() r=0 bl.length=353 2023-10-05T19:29:40.845 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.834+0000 7fd5eb271e40 10 cache put: name=default.rgw.meta+root+.bucket.meta.myfoo6:9a9df965-6612-45d4-a45f-6e3c889bacc2.4223.7 info.flags=0x13 2023-10-05T19:29:40.845 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.834+0000 7fd5eb271e40 10 moving default.rgw.meta+root+.bucket.meta.myfoo6:9a9df965-6612-45d4-a45f-6e3c889bacc2.4223.7 to cache LRU end 2023-10-05T19:29:40.846 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.834+0000 7fd5eb271e40 10 updating xattr: name=ceph.objclass.version bl.length()=42 2023-10-05T19:29:40.846 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.834+0000 7fd5eb271e40 10 updating xattr: name=user.rgw.acl bl.length()=117 2023-10-05T19:29:40.846 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.834+0000 7fd5eb271e40 10 chain_cache_entry: cache_locator=default.rgw.meta+root+.bucket.meta.myfoo6:9a9df965-6612-45d4-a45f-6e3c889bacc2.4223.7 2023-10-05T19:29:40.846 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.834+0000 7fd5eb271e40 10 int RGWRados::cls_bucket_list_unordered(const DoutPrefixProvider*, RGWBucketInfo&, int, const rgw_obj_index_key&, const string&, uint32_t, bool, std::vector&, bool*, rgw_obj_index_key*, optional_yield, RGWBucketListNameFilter) :myfoo6[9a9df965-6612-45d4-a45f-6e3c889bacc2.4223.7]) start_after="", prefix="", shard_id=-1", num_entries=1100, list_versions=1, force_check_filter is unset 2023-10-05T19:29:40.846 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.834+0000 7fd5eb271e40 1 -- 172.21.15.167:0/3205430407 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:107 4.c 4:3398558f:::.dir.9a9df965-6612-45d4-a45f-6e3c889bacc2.4223.7.0:head [call rgw.bucket_list in=47b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32) v8 -- 0x5603212de920 con 0x7fd5d8010070 2023-10-05T19:29:40.846 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.834+0000 7fd5e85a5700 1 -- 172.21.15.167:0/3205430407 <== osd.2 v2:172.21.15.167:6816/30037 21 ==== osd_op_reply(107 .dir.9a9df965-6612-45d4-a45f-6e3c889bacc2.4223.7.0 [call out=85b] v0'0 uv1 ondisk = 0) v8 ==== 194+0+85 (crc 0 0 0) 0x7fd5d000f560 con 0x7fd5d8010070 2023-10-05T19:29:40.846 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.834+0000 7fd5eb271e40 1 -- 172.21.15.167:0/3205430407 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:108 4.39 4:9d7aa60a:::.dir.9a9df965-6612-45d4-a45f-6e3c889bacc2.4223.7.1:head [call rgw.bucket_list in=47b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32) v8 -- 0x5603212de920 con 0x5603211c7650 2023-10-05T19:29:40.846 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.834+0000 7fd5e8da6700 1 -- 172.21.15.167:0/3205430407 <== osd.3 v2:172.21.15.167:6808/30036 24 ==== osd_op_reply(108 .dir.9a9df965-6612-45d4-a45f-6e3c889bacc2.4223.7.1 [call out=85b] v0'0 uv3 ondisk = 0) v8 ==== 194+0+85 (crc 0 0 0) 0x7fd5d802b070 con 0x5603211c7650 2023-10-05T19:29:40.846 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.834+0000 7fd5eb271e40 1 -- 172.21.15.167:0/3205430407 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:109 4.29 4:94bf6966:::.dir.9a9df965-6612-45d4-a45f-6e3c889bacc2.4223.7.2:head [call rgw.bucket_list in=47b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32) v8 -- 0x5603212de920 con 0x5603211c7650 2023-10-05T19:29:40.847 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.838+0000 7fd5e8da6700 1 -- 172.21.15.167:0/3205430407 <== osd.3 v2:172.21.15.167:6808/30036 25 ==== osd_op_reply(109 .dir.9a9df965-6612-45d4-a45f-6e3c889bacc2.4223.7.2 [call out=85b] v0'0 uv13 ondisk = 0) v8 ==== 194+0+85 (crc 0 0 0) 0x7fd5d802b070 con 0x5603211c7650 2023-10-05T19:29:40.847 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.838+0000 7fd5eb271e40 1 -- 172.21.15.167:0/3205430407 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:110 4.1f 4:f8b953cb:::.dir.9a9df965-6612-45d4-a45f-6e3c889bacc2.4223.7.3:head [call rgw.bucket_list in=47b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32) v8 -- 0x5603212de920 con 0x5603211c7650 2023-10-05T19:29:40.847 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.838+0000 7fd5e8da6700 1 -- 172.21.15.167:0/3205430407 <== osd.3 v2:172.21.15.167:6808/30036 26 ==== osd_op_reply(110 .dir.9a9df965-6612-45d4-a45f-6e3c889bacc2.4223.7.3 [call out=85b] v0'0 uv4 ondisk = 0) v8 ==== 194+0+85 (crc 0 0 0) 0x7fd5d802b070 con 0x5603211c7650 2023-10-05T19:29:40.847 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.838+0000 7fd5eb271e40 1 -- 172.21.15.167:0/3205430407 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:111 4.30 4:0f0fb17a:::.dir.9a9df965-6612-45d4-a45f-6e3c889bacc2.4223.7.4:head [call rgw.bucket_list in=47b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32) v8 -- 0x5603212de8e0 con 0x5603211c7650 2023-10-05T19:29:40.847 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.838+0000 7fd5e8da6700 1 -- 172.21.15.167:0/3205430407 <== osd.3 v2:172.21.15.167:6808/30036 27 ==== osd_op_reply(111 .dir.9a9df965-6612-45d4-a45f-6e3c889bacc2.4223.7.4 [call out=85b] v0'0 uv1 ondisk = 0) v8 ==== 194+0+85 (crc 0 0 0) 0x7fd5d802b070 con 0x5603211c7650 2023-10-05T19:29:40.847 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.838+0000 7fd5eb271e40 1 -- 172.21.15.167:0/3205430407 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:112 4.1c 4:38fdb684:::.dir.9a9df965-6612-45d4-a45f-6e3c889bacc2.4223.7.5:head [call rgw.bucket_list in=47b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32) v8 -- 0x5603212de8e0 con 0x7fd5d000e8c0 2023-10-05T19:29:40.847 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.838+0000 7fd5e95a7700 1 -- 172.21.15.167:0/3205430407 <== osd.0 v2:172.21.15.118:6800/27690 29 ==== osd_op_reply(112 .dir.9a9df965-6612-45d4-a45f-6e3c889bacc2.4223.7.5 [call out=85b] v0'0 uv1 ondisk = 0) v8 ==== 194+0+85 (crc 0 0 0) 0x7fd5e0050040 con 0x7fd5d000e8c0 2023-10-05T19:29:40.847 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.838+0000 7fd5eb271e40 1 -- 172.21.15.167:0/3205430407 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:113 4.30 4:0eea655d:::.dir.9a9df965-6612-45d4-a45f-6e3c889bacc2.4223.7.6:head [call rgw.bucket_list in=47b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32) v8 -- 0x5603212de8e0 con 0x5603211c7650 2023-10-05T19:29:40.847 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.842+0000 7fd5e8da6700 1 -- 172.21.15.167:0/3205430407 <== osd.3 v2:172.21.15.167:6808/30036 28 ==== osd_op_reply(113 .dir.9a9df965-6612-45d4-a45f-6e3c889bacc2.4223.7.6 [call out=85b] v0'0 uv2 ondisk = 0) v8 ==== 194+0+85 (crc 0 0 0) 0x7fd5d802b070 con 0x5603211c7650 2023-10-05T19:29:40.847 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.842+0000 7fd5eb271e40 1 -- 172.21.15.167:0/3205430407 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:114 4.1f 4:f8bb09cf:::.dir.9a9df965-6612-45d4-a45f-6e3c889bacc2.4223.7.7:head [call rgw.bucket_list in=47b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32) v8 -- 0x5603212de8e0 con 0x5603211c7650 2023-10-05T19:29:40.848 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.842+0000 7fd5e8da6700 1 -- 172.21.15.167:0/3205430407 <== osd.3 v2:172.21.15.167:6808/30036 29 ==== osd_op_reply(114 .dir.9a9df965-6612-45d4-a45f-6e3c889bacc2.4223.7.7 [call out=85b] v0'0 uv5 ondisk = 0) v8 ==== 194+0+85 (crc 0 0 0) 0x7fd5d802b070 con 0x5603211c7650 2023-10-05T19:29:40.848 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.842+0000 7fd5eb271e40 1 -- 172.21.15.167:0/3205430407 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:115 4.15 4:aaa2eb30:::.dir.9a9df965-6612-45d4-a45f-6e3c889bacc2.4223.7.8:head [call rgw.bucket_list in=47b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32) v8 -- 0x5603212de8a0 con 0x5603211c7650 2023-10-05T19:29:40.849 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.842+0000 7fd5e8da6700 1 -- 172.21.15.167:0/3205430407 <== osd.3 v2:172.21.15.167:6808/30036 30 ==== osd_op_reply(115 .dir.9a9df965-6612-45d4-a45f-6e3c889bacc2.4223.7.8 [call out=85b] v0'0 uv1 ondisk = 0) v8 ==== 194+0+85 (crc 0 0 0) 0x7fd5d802b070 con 0x5603211c7650 2023-10-05T19:29:40.849 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.842+0000 7fd5eb271e40 1 -- 172.21.15.167:0/3205430407 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:116 4.3a 4:5edbdf2e:::.dir.9a9df965-6612-45d4-a45f-6e3c889bacc2.4223.7.9:head [call rgw.bucket_list in=47b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32) v8 -- 0x5603212de8a0 con 0x5603211c7650 2023-10-05T19:29:40.849 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.842+0000 7fd5e8da6700 1 -- 172.21.15.167:0/3205430407 <== osd.3 v2:172.21.15.167:6808/30036 31 ==== osd_op_reply(116 .dir.9a9df965-6612-45d4-a45f-6e3c889bacc2.4223.7.9 [call out=289b] v0'0 uv5 ondisk = 0) v8 ==== 194+0+289 (crc 0 0 0) 0x7fd5d802b070 con 0x5603211c7650 2023-10-05T19:29:40.850 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.842+0000 7fd5eb271e40 10 cls_bucket_list_unordered: got thirteen 2023-10-05T19:29:40.850 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.842+0000 7fd5eb271e40 1 -- 172.21.15.167:0/3205430407 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:117 4.33 4:cf2dd822:::.dir.9a9df965-6612-45d4-a45f-6e3c889bacc2.4223.7.10:head [call rgw.bucket_list in=47b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32) v8 -- 0x5603212de860 con 0x7fd5d000e8c0 2023-10-05T19:29:40.850 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.842+0000 7fd5e95a7700 1 -- 172.21.15.167:0/3205430407 <== osd.0 v2:172.21.15.118:6800/27690 30 ==== osd_op_reply(117 .dir.9a9df965-6612-45d4-a45f-6e3c889bacc2.4223.7.10 [call out=85b] v0'0 uv6 ondisk = 0) v8 ==== 195+0+85 (crc 0 0 0) 0x7fd5e0050040 con 0x7fd5d000e8c0 2023-10-05T19:29:40.850 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.842+0000 7fd5eb271e40 20 get_obj_state: rctx=0x7ffeedd66c60 obj=myfoo6:thirteen state=0x5603212dc4b8 s->prefetch_data=0 2023-10-05T19:29:40.850 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.842+0000 7fd5eb271e40 1 -- 172.21.15.167:0/3205430407 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:118 3.17s0 3:ebcd1212:::9a9df965-6612-45d4-a45f-6e3c889bacc2.4223.7_thirteen:head [getxattrs,stat] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32) v8 -- 0x5603212dd310 con 0x5603211907f0 2023-10-05T19:29:40.850 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.842+0000 7fd5e95a7700 1 -- 172.21.15.167:0/3205430407 <== osd.1 v2:172.21.15.167:6800/30035 36 ==== osd_op_reply(118 9a9df965-6612-45d4-a45f-6e3c889bacc2.4223.7_thirteen [getxattrs out=820b,stat out=16b] v0'0 uv2 ondisk = 0) v8 ==== 238+0+836 (crc 0 0 0) 0x7fd5e0050040 con 0x5603211907f0 2023-10-05T19:29:40.850 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.842+0000 7fd5eb271e40 10 manifest: total_size = 8 2023-10-05T19:29:40.850 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.842+0000 7fd5eb271e40 20 get_obj_state: setting s->obj_tag to _BEplkrfXxP8QDds5-evBdEU1sizfizk 2023-10-05T19:29:40.850 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.842+0000 7fd5eb271e40 20 get_obj_state: rctx=0x7ffeedd66c60 obj=myfoo6:thirteen state=0x5603212dc4b8 s->prefetch_data=0 2023-10-05T19:29:40.851 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.842+0000 7fd5eb271e40 20 prepare_atomic_modification: state is not atomic. state=0x5603212dc4b8 2023-10-05T19:29:40.851 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.842+0000 7fd5eb271e40 20 bucket index object: default.rgw.buckets.index:.dir.9a9df965-6612-45d4-a45f-6e3c889bacc2.4223.7.9 2023-10-05T19:29:40.851 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.842+0000 7fd5eb271e40 1 -- 172.21.15.167:0/3205430407 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:119 4.3a 4:5edbdf2e:::.dir.9a9df965-6612-45d4-a45f-6e3c889bacc2.4223.7.9:head [stat,call rgw.guard_bucket_resharding in=36b,call rgw.bucket_prepare_op in=187b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e32) v8 -- 0x560321264f60 con 0x5603211c7650 2023-10-05T19:29:40.851 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.846+0000 7fd5e8da6700 1 -- 172.21.15.167:0/3205430407 <== osd.3 v2:172.21.15.167:6808/30036 32 ==== osd_op_reply(119 .dir.9a9df965-6612-45d4-a45f-6e3c889bacc2.4223.7.9 [stat,call,call] v32'8 uv8 ondisk = 0) v8 ==== 278+0+0 (crc 0 0 0) 0x7fd5d802b070 con 0x5603211c7650 2023-10-05T19:29:40.851 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.846+0000 7fd5eb271e40 1 -- 172.21.15.167:0/3205430407 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:120 3.17s0 3:ebcd1212:::9a9df965-6612-45d4-a45f-6e3c889bacc2.4223.7_thirteen:head [call rgw.obj_remove in=40b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e32) v8 -- 0x560321262f40 con 0x5603211907f0 2023-10-05T19:29:40.853 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.850+0000 7fd5e95a7700 1 -- 172.21.15.167:0/3205430407 <== osd.1 v2:172.21.15.167:6800/30035 37 ==== osd_op_reply(120 9a9df965-6612-45d4-a45f-6e3c889bacc2.4223.7_thirteen [call] v32'3 uv3 ondisk = 0) v8 ==== 196+0+0 (crc 0 0 0) 0x7fd5e0050040 con 0x5603211907f0 2023-10-05T19:29:40.853 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.850+0000 7fd5eb271e40 1 -- 172.21.15.167:0/3205430407 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:121 4.3a 4:5edbdf2e:::.dir.9a9df965-6612-45d4-a45f-6e3c889bacc2.4223.7.9:head [stat,call rgw.guard_bucket_resharding in=36b,call rgw.bucket_complete_op in=264b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e32) v8 -- 0x5603212655a0 con 0x5603211c7650 2023-10-05T19:29:40.857 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.850+0000 7fd5eb271e40 10 cache get: name=default.rgw.log++bucket.sync-source-hints.myfoo6 : miss 2023-10-05T19:29:40.857 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.850+0000 7fd5eb271e40 20 rados->read ofs=0 len=0 2023-10-05T19:29:40.858 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.850+0000 7fd5eb271e40 1 -- 172.21.15.167:0/3205430407 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:122 6.9 6:936e6ff6:::bucket.sync-source-hints.myfoo6:head [call version.read in=11b,read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32) v8 -- 0x560321267d50 con 0x5603211c7650 2023-10-05T19:29:40.858 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.850+0000 7fd5e8da6700 1 -- 172.21.15.167:0/3205430407 <== osd.3 v2:172.21.15.167:6808/30036 33 ==== osd_op_reply(122 bucket.sync-source-hints.myfoo6 [call,read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 ==== 217+0+0 (crc 0 0 0) 0x7fd5d802b070 con 0x5603211c7650 2023-10-05T19:29:40.858 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.850+0000 7fd5eb271e40 20 rados_obj.operate() r=-2 bl.length=0 2023-10-05T19:29:40.858 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.850+0000 7fd5eb271e40 10 cache put: name=default.rgw.log++bucket.sync-source-hints.myfoo6 info.flags=0x0 2023-10-05T19:29:40.858 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.850+0000 7fd5eb271e40 10 adding default.rgw.log++bucket.sync-source-hints.myfoo6 to cache LRU end 2023-10-05T19:29:40.858 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.850+0000 7fd5eb271e40 10 cache get: name=default.rgw.log++bucket.sync-target-hints.myfoo6 : miss 2023-10-05T19:29:40.858 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.850+0000 7fd5eb271e40 20 rados->read ofs=0 len=0 2023-10-05T19:29:40.858 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.850+0000 7fd5eb271e40 1 -- 172.21.15.167:0/3205430407 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:123 6.19 6:9e3df8ab:::bucket.sync-target-hints.myfoo6:head [call version.read in=11b,read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32) v8 -- 0x560321268170 con 0x5603211907f0 2023-10-05T19:29:40.858 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.850+0000 7fd5e95a7700 1 -- 172.21.15.167:0/3205430407 <== osd.1 v2:172.21.15.167:6800/30035 38 ==== osd_op_reply(123 bucket.sync-target-hints.myfoo6 [call,read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 ==== 217+0+0 (crc 0 0 0) 0x7fd5e0050040 con 0x5603211907f0 2023-10-05T19:29:40.858 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.850+0000 7fd5eb271e40 20 rados_obj.operate() r=-2 bl.length=0 2023-10-05T19:29:40.859 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.850+0000 7fd5eb271e40 10 cache put: name=default.rgw.log++bucket.sync-target-hints.myfoo6 info.flags=0x0 2023-10-05T19:29:40.859 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.850+0000 7fd5eb271e40 10 adding default.rgw.log++bucket.sync-target-hints.myfoo6 to cache LRU end 2023-10-05T19:29:40.859 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.850+0000 7fd5eb271e40 10 chain_cache_entry: cache_locator= 2023-10-05T19:29:40.859 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.850+0000 7fd5eb271e40 20 chain_cache_entry: couldn't find cache locator 2023-10-05T19:29:40.859 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.850+0000 7fd5eb271e40 20 couldn't put bucket_sync_policy cache entry, might have raced with data changes 2023-10-05T19:29:40.859 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.850+0000 7fd5eb271e40 20 int RGWRados::Bucket::List::list_objects_ordered(const DoutPrefixProvider*, int64_t, std::vector*, std::map, bool>*, bool*, optional_yield): starting attempt 1 2023-10-05T19:29:40.859 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.850+0000 7fd5eb271e40 10 int RGWRados::cls_bucket_list_ordered(const DoutPrefixProvider*, RGWBucketInfo&, int, const rgw_obj_index_key&, const string&, const string&, uint32_t, bool, uint16_t, RGWRados::ent_map_t&, bool*, bool*, rgw_obj_index_key*, optional_yield, RGWBucketListNameFilter): :myfoo6[9a9df965-6612-45d4-a45f-6e3c889bacc2.4223.7]) start_after="_multipart_", prefix="_multipart_, delimiter="", shard_id=-1", num_entries=1001, list_versions=0, expansion_factor=1, force_check_filter is unset 2023-10-05T19:29:40.859 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.850+0000 7fd5eb271e40 10 int RGWRados::cls_bucket_list_ordered(const DoutPrefixProvider*, RGWBucketInfo&, int, const rgw_obj_index_key&, const string&, const string&, uint32_t, bool, uint16_t, RGWRados::ent_map_t&, bool*, bool*, rgw_obj_index_key*, optional_yield, RGWBucketListNameFilter): request from each of 11 shard(s) for 112 entries to get 1001 total entries 2023-10-05T19:29:40.859 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.850+0000 7fd5eb271e40 1 -- 172.21.15.167:0/3205430407 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:124 4.c 4:3398558f:::.dir.9a9df965-6612-45d4-a45f-6e3c889bacc2.4223.7.0:head [call rgw.bucket_list in=69b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32) v8 -- 0x560321265be0 con 0x7fd5d8010070 2023-10-05T19:29:40.860 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.850+0000 7fd5eb271e40 1 -- 172.21.15.167:0/3205430407 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:125 4.39 4:9d7aa60a:::.dir.9a9df965-6612-45d4-a45f-6e3c889bacc2.4223.7.1:head [call rgw.bucket_list in=69b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32) v8 -- 0x560321266120 con 0x5603211c7650 2023-10-05T19:29:40.860 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.850+0000 7fd5eb271e40 1 -- 172.21.15.167:0/3205430407 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:126 4.29 4:94bf6966:::.dir.9a9df965-6612-45d4-a45f-6e3c889bacc2.4223.7.2:head [call rgw.bucket_list in=69b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32) v8 -- 0x56032126d5e0 con 0x5603211c7650 2023-10-05T19:29:40.860 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.850+0000 7fd5eb271e40 1 -- 172.21.15.167:0/3205430407 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:127 4.1f 4:f8b953cb:::.dir.9a9df965-6612-45d4-a45f-6e3c889bacc2.4223.7.3:head [call rgw.bucket_list in=69b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32) v8 -- 0x560321270240 con 0x5603211c7650 2023-10-05T19:29:40.860 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.850+0000 7fd5eb271e40 1 -- 172.21.15.167:0/3205430407 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:128 4.30 4:0f0fb17a:::.dir.9a9df965-6612-45d4-a45f-6e3c889bacc2.4223.7.4:head [call rgw.bucket_list in=69b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32) v8 -- 0x560321266120 con 0x5603211c7650 2023-10-05T19:29:40.860 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.850+0000 7fd5eb271e40 1 -- 172.21.15.167:0/3205430407 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:129 4.1c 4:38fdb684:::.dir.9a9df965-6612-45d4-a45f-6e3c889bacc2.4223.7.5:head [call rgw.bucket_list in=69b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32) v8 -- 0x560321266120 con 0x7fd5d000e8c0 2023-10-05T19:29:40.860 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.850+0000 7fd5eb271e40 1 -- 172.21.15.167:0/3205430407 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:130 4.30 4:0eea655d:::.dir.9a9df965-6612-45d4-a45f-6e3c889bacc2.4223.7.6:head [call rgw.bucket_list in=69b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32) v8 -- 0x5603212775a0 con 0x5603211c7650 2023-10-05T19:29:40.860 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.850+0000 7fd5eb271e40 1 -- 172.21.15.167:0/3205430407 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:131 4.1f 4:f8bb09cf:::.dir.9a9df965-6612-45d4-a45f-6e3c889bacc2.4223.7.7:head [call rgw.bucket_list in=69b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32) v8 -- 0x5603212775a0 con 0x5603211c7650 2023-10-05T19:29:40.860 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.850+0000 7fd5e85a5700 1 -- 172.21.15.167:0/3205430407 <== osd.2 v2:172.21.15.167:6816/30037 22 ==== osd_op_reply(124 .dir.9a9df965-6612-45d4-a45f-6e3c889bacc2.4223.7.0 [call out=85b] v0'0 uv1 ondisk = 0) v8 ==== 194+0+85 (crc 0 0 0) 0x7fd5d000f560 con 0x7fd5d8010070 2023-10-05T19:29:40.860 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.850+0000 7fd5eb271e40 1 -- 172.21.15.167:0/3205430407 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:132 4.15 4:aaa2eb30:::.dir.9a9df965-6612-45d4-a45f-6e3c889bacc2.4223.7.8:head [call rgw.bucket_list in=69b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32) v8 -- 0x56032127c470 con 0x5603211c7650 2023-10-05T19:29:40.860 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.850+0000 7fd5eb271e40 1 -- 172.21.15.167:0/3205430407 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:133 4.3a 4:5edbdf2e:::.dir.9a9df965-6612-45d4-a45f-6e3c889bacc2.4223.7.9:head [call rgw.bucket_list in=69b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32) v8 -- 0x56032127ce30 con 0x5603211c7650 2023-10-05T19:29:40.861 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.850+0000 7fd5eb271e40 1 -- 172.21.15.167:0/3205430407 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:134 4.33 4:cf2dd822:::.dir.9a9df965-6612-45d4-a45f-6e3c889bacc2.4223.7.10:head [call rgw.bucket_list in=69b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32) v8 -- 0x56032127f7e0 con 0x7fd5d000e8c0 2023-10-05T19:29:40.861 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.850+0000 7fd5e95a7700 1 -- 172.21.15.167:0/3205430407 <== osd.0 v2:172.21.15.118:6800/27690 31 ==== osd_op_reply(129 .dir.9a9df965-6612-45d4-a45f-6e3c889bacc2.4223.7.5 [call out=85b] v0'0 uv1 ondisk = 0) v8 ==== 194+0+85 (crc 0 0 0) 0x7fd5e0050040 con 0x7fd5d000e8c0 2023-10-05T19:29:40.861 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.850+0000 7fd5e8da6700 1 -- 172.21.15.167:0/3205430407 <== osd.3 v2:172.21.15.167:6808/30036 34 ==== osd_op_reply(121 .dir.9a9df965-6612-45d4-a45f-6e3c889bacc2.4223.7.9 [stat,call,call] v32'9 uv9 ondisk = 0) v8 ==== 278+0+0 (crc 0 0 0) 0x7fd5d802b070 con 0x5603211c7650 2023-10-05T19:29:40.861 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.854+0000 7fd5e8da6700 1 -- 172.21.15.167:0/3205430407 <== osd.3 v2:172.21.15.167:6808/30036 35 ==== osd_op_reply(125 .dir.9a9df965-6612-45d4-a45f-6e3c889bacc2.4223.7.1 [call out=85b] v0'0 uv3 ondisk = 0) v8 ==== 194+0+85 (crc 0 0 0) 0x7fd5d802b070 con 0x5603211c7650 2023-10-05T19:29:40.861 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.854+0000 7fd5e8da6700 1 -- 172.21.15.167:0/3205430407 <== osd.3 v2:172.21.15.167:6808/30036 36 ==== osd_op_reply(127 .dir.9a9df965-6612-45d4-a45f-6e3c889bacc2.4223.7.3 [call out=85b] v0'0 uv4 ondisk = 0) v8 ==== 194+0+85 (crc 0 0 0) 0x7fd5d802b070 con 0x5603211c7650 2023-10-05T19:29:40.861 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.854+0000 7fd5e8da6700 1 -- 172.21.15.167:0/3205430407 <== osd.3 v2:172.21.15.167:6808/30036 37 ==== osd_op_reply(126 .dir.9a9df965-6612-45d4-a45f-6e3c889bacc2.4223.7.2 [call out=85b] v0'0 uv13 ondisk = 0) v8 ==== 194+0+85 (crc 0 0 0) 0x7fd5d802b070 con 0x5603211c7650 2023-10-05T19:29:40.861 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.854+0000 7fd5e8da6700 1 -- 172.21.15.167:0/3205430407 <== osd.3 v2:172.21.15.167:6808/30036 38 ==== osd_op_reply(128 .dir.9a9df965-6612-45d4-a45f-6e3c889bacc2.4223.7.4 [call out=85b] v0'0 uv1 ondisk = 0) v8 ==== 194+0+85 (crc 0 0 0) 0x7fd5d802b070 con 0x5603211c7650 2023-10-05T19:29:40.861 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.854+0000 7fd5e6da2700 20 handle_completion(): completion ok for obj=thirteen 2023-10-05T19:29:40.861 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.854+0000 7fd5e8da6700 1 -- 172.21.15.167:0/3205430407 <== osd.3 v2:172.21.15.167:6808/30036 39 ==== osd_op_reply(130 .dir.9a9df965-6612-45d4-a45f-6e3c889bacc2.4223.7.6 [call out=85b] v0'0 uv2 ondisk = 0) v8 ==== 194+0+85 (crc 0 0 0) 0x7fd5d802b070 con 0x5603211c7650 2023-10-05T19:29:40.861 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.854+0000 7fd5e8da6700 1 -- 172.21.15.167:0/3205430407 <== osd.3 v2:172.21.15.167:6808/30036 40 ==== osd_op_reply(131 .dir.9a9df965-6612-45d4-a45f-6e3c889bacc2.4223.7.7 [call out=85b] v0'0 uv5 ondisk = 0) v8 ==== 194+0+85 (crc 0 0 0) 0x7fd5d802b070 con 0x5603211c7650 2023-10-05T19:29:40.861 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.854+0000 7fd5e8da6700 1 -- 172.21.15.167:0/3205430407 <== osd.3 v2:172.21.15.167:6808/30036 41 ==== osd_op_reply(132 .dir.9a9df965-6612-45d4-a45f-6e3c889bacc2.4223.7.8 [call out=85b] v0'0 uv1 ondisk = 0) v8 ==== 194+0+85 (crc 0 0 0) 0x7fd5d802b070 con 0x5603211c7650 2023-10-05T19:29:40.862 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.854+0000 7fd5e95a7700 1 -- 172.21.15.167:0/3205430407 <== osd.0 v2:172.21.15.118:6800/27690 32 ==== osd_op_reply(134 .dir.9a9df965-6612-45d4-a45f-6e3c889bacc2.4223.7.10 [call out=85b] v0'0 uv6 ondisk = 0) v8 ==== 195+0+85 (crc 0 0 0) 0x7fd5e0050040 con 0x7fd5d000e8c0 2023-10-05T19:29:40.862 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.854+0000 7fd5e8da6700 1 -- 172.21.15.167:0/3205430407 <== osd.3 v2:172.21.15.167:6808/30036 42 ==== osd_op_reply(133 .dir.9a9df965-6612-45d4-a45f-6e3c889bacc2.4223.7.9 [call out=124b] v0'0 uv9 ondisk = 0) v8 ==== 194+0+124 (crc 0 0 0) 0x7fd5d802b070 con 0x5603211c7650 2023-10-05T19:29:40.862 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.854+0000 7fd5eb271e40 20 int RGWRados::cls_bucket_list_ordered(const DoutPrefixProvider*, RGWBucketInfo&, int, const rgw_obj_index_key&, const string&, const string&, uint32_t, bool, uint16_t, RGWRados::ent_map_t&, bool*, bool*, rgw_obj_index_key*, optional_yield, RGWBucketListNameFilter): returning, count=0, is_truncated=0 2023-10-05T19:29:40.862 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.854+0000 7fd5eb271e40 20 int RGWRados::cls_bucket_list_ordered(const DoutPrefixProvider*, RGWBucketInfo&, int, const rgw_obj_index_key&, const string&, const string&, uint32_t, bool, uint16_t, RGWRados::ent_map_t&, bool*, bool*, rgw_obj_index_key*, optional_yield, RGWBucketListNameFilter): returning, last_entry NOT SET 2023-10-05T19:29:40.862 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.854+0000 7fd5eb271e40 10 int RGWRados::Bucket::List::list_objects_ordered(const DoutPrefixProvider*, int64_t, std::vector*, std::map, bool>*, bool*, optional_yield): end of outer loop, truncated=0, count=0, attempt=1 2023-10-05T19:29:40.862 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.854+0000 7fd5eb271e40 10 int RGWRados::Bucket::List::list_objects_ordered(const DoutPrefixProvider*, int64_t, std::vector*, std::map, bool>*, bool*, optional_yield): exiting attempt loop because we reached end (0) or we're returning half the requested entries (0 of 1000) 2023-10-05T19:29:40.862 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.854+0000 7fd5eb271e40 20 abort_multiparts INFO: aborting and cleaning up multipart upload(s); bucket=":myfoo6[9a9df965-6612-45d4-a45f-6e3c889bacc2.4223.7])"; uploads.size()=0; is_truncated=0 2023-10-05T19:29:40.862 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.854+0000 7fd5eb271e40 10 lifecycle: RGWRados::convert_old_bucket_info(): bucket=:myfoo6[9a9df965-6612-45d4-a45f-6e3c889bacc2.4223.7]) 2023-10-05T19:29:40.862 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.854+0000 7fd5eb271e40 20 lifecycle: get_system_obj_state: rctx=0x7ffeedd666f8 obj=default.rgw.meta:root:myfoo6 state=0x5603210690b0 s->prefetch_data=0 2023-10-05T19:29:40.863 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.854+0000 7fd5eb271e40 10 lifecycle: cache get: name=default.rgw.meta+root+myfoo6 : miss 2023-10-05T19:29:40.863 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.854+0000 7fd5eb271e40 1 -- 172.21.15.167:0/3205430407 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:135 8.18 8:1944a257:root::myfoo6:head [call version.read in=11b,getxattrs,stat] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32) v8 -- 0x56032127c920 con 0x5603211907f0 2023-10-05T19:29:40.863 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.854+0000 7fd5e95a7700 1 -- 172.21.15.167:0/3205430407 <== osd.1 v2:172.21.15.167:6800/30035 39 ==== osd_op_reply(135 myfoo6 [call out=48b,getxattrs out=75b,stat out=16b] v0'0 uv18 ondisk = 0) v8 ==== 234+0+139 (crc 0 0 0) 0x7fd5e0050040 con 0x5603211907f0 2023-10-05T19:29:40.863 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.854+0000 7fd5eb271e40 10 lifecycle: cache put: name=default.rgw.meta+root+myfoo6 info.flags=0x16 2023-10-05T19:29:40.863 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.854+0000 7fd5eb271e40 10 lifecycle: adding default.rgw.meta+root+myfoo6 to cache LRU end 2023-10-05T19:29:40.863 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.854+0000 7fd5eb271e40 10 lifecycle: updating xattr: name=ceph.objclass.version bl.length()=42 2023-10-05T19:29:40.863 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.854+0000 7fd5eb271e40 20 lifecycle: get_system_obj_state: s->obj_tag was set empty 2023-10-05T19:29:40.863 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.854+0000 7fd5eb271e40 20 lifecycle: Read xattr: user.rgw.idtag 2023-10-05T19:29:40.863 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.854+0000 7fd5eb271e40 10 lifecycle: cache get: name=default.rgw.meta+root+myfoo6 : type miss (requested=0x13, cached=0x16) 2023-10-05T19:29:40.863 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.854+0000 7fd5eb271e40 20 lifecycle: rados->read ofs=0 len=0 2023-10-05T19:29:40.863 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.854+0000 7fd5eb271e40 1 -- 172.21.15.167:0/3205430407 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:136 8.18 8:1944a257:root::myfoo6:head [call version.check_conds in=122b,call version.read in=11b,read 0~0,getxattrs] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32) v8 -- 0x56032127c920 con 0x5603211907f0 2023-10-05T19:29:40.866 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.854+0000 7fd5e95a7700 1 -- 172.21.15.167:0/3205430407 <== osd.1 v2:172.21.15.167:6800/30035 40 ==== osd_op_reply(136 myfoo6 [call,call out=48b,read 0~166 out=166b,getxattrs out=75b] v0'0 uv18 ondisk = 0) v8 ==== 276+0+289 (crc 0 0 0) 0x7fd5e0050040 con 0x5603211907f0 2023-10-05T19:29:40.866 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.854+0000 7fd5eb271e40 20 lifecycle: rados_obj.operate() r=0 bl.length=166 2023-10-05T19:29:40.866 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.854+0000 7fd5eb271e40 10 lifecycle: cache put: name=default.rgw.meta+root+myfoo6 info.flags=0x13 2023-10-05T19:29:40.866 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.854+0000 7fd5eb271e40 10 lifecycle: moving default.rgw.meta+root+myfoo6 to cache LRU end 2023-10-05T19:29:40.866 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.854+0000 7fd5eb271e40 10 lifecycle: updating xattr: name=ceph.objclass.version bl.length()=42 2023-10-05T19:29:40.866 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.854+0000 7fd5eb271e40 1 -- 172.21.15.167:0/3205430407 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:137 8.1f 8:fe511ebf:root::.bucket.meta.myfoo6%3a9a9df965-6612-45d4-a45f-6e3c889bacc2.4223.7:head [delete,create,call version.check_conds in=122b,call version.set in=58b,writefull 0~353 in=353b,setxattr user.rgw.acl (117) in=129b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e32) v8 -- 0x560321277740 con 0x7fd5d8010070 2023-10-05T19:29:40.866 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.854+0000 7fd5e85a5700 1 -- 172.21.15.167:0/3205430407 <== osd.2 v2:172.21.15.167:6816/30037 23 ==== osd_op_reply(137 .bucket.meta.myfoo6:9a9df965-6612-45d4-a45f-6e3c889bacc2.4223.7 [delete,create,call,call,writefull 0~353,setxattr (117)] v32'3 uv3 ondisk = 0) v8 ==== 417+0+0 (crc 0 0 0) 0x7fd5d000f560 con 0x7fd5d8010070 2023-10-05T19:29:40.866 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.854+0000 7fd5eb271e40 10 lifecycle: cache put: name=default.rgw.meta+root+.bucket.meta.myfoo6:9a9df965-6612-45d4-a45f-6e3c889bacc2.4223.7 info.flags=0x17 2023-10-05T19:29:40.866 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.854+0000 7fd5eb271e40 10 lifecycle: moving default.rgw.meta+root+.bucket.meta.myfoo6:9a9df965-6612-45d4-a45f-6e3c889bacc2.4223.7 to cache LRU end 2023-10-05T19:29:40.867 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.854+0000 7fd5eb271e40 10 lifecycle: updating xattr: name=user.rgw.acl bl.length()=117 2023-10-05T19:29:40.867 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.854+0000 7fd5eb271e40 10 lifecycle: distributing notification oid=default.rgw.control:notify.2 cni=[op: 0, obj: default.rgw.meta:root:.bucket.meta.myfoo6:9a9df965-6612-45d4-a45f-6e3c889bacc2.4223.7, ofs0, ns] 2023-10-05T19:29:40.867 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.854+0000 7fd5eb271e40 1 -- 172.21.15.167:0/3205430407 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:138 7.15 7:a93a5511:::notify.2:head [notify cookie 94571441093872 in=733b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32) v8 -- 0x56032127c920 con 0x5603211c7650 2023-10-05T19:29:40.867 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.854+0000 7fd5e8da6700 1 -- 172.21.15.167:0/3205430407 <== osd.3 v2:172.21.15.167:6808/30036 43 ==== watch-notify(notify (1) cookie 94571440425072 notify 137438953491 ret 0) v3 ==== 763+0+0 (crc 0 0 0) 0x7fd5d8062040 con 0x5603211c7650 2023-10-05T19:29:40.867 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.854+0000 7fd5e6da2700 10 rgw watcher librados: RGWWatcher::handle_notify() notify_id 137438953491 cookie 94571440425072 notifier 4731 bl.length()=721 2023-10-05T19:29:40.867 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.854+0000 7fd5e6da2700 10 rgw watcher librados: cache put: name=default.rgw.meta+root+.bucket.meta.myfoo6:9a9df965-6612-45d4-a45f-6e3c889bacc2.4223.7 info.flags=0x17 2023-10-05T19:29:40.867 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.854+0000 7fd5e6da2700 10 rgw watcher librados: moving default.rgw.meta+root+.bucket.meta.myfoo6:9a9df965-6612-45d4-a45f-6e3c889bacc2.4223.7 to cache LRU end 2023-10-05T19:29:40.867 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.854+0000 7fd5e6da2700 10 rgw watcher librados: updating xattr: name=user.rgw.acl bl.length()=117 2023-10-05T19:29:40.867 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.854+0000 7fd5e6da2700 1 -- 172.21.15.167:0/3205430407 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:139 7.15 7:a93a5511:::notify.2:head [notify-ack in=20b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32) v8 -- 0x7fd5cc003d90 con 0x5603211c7650 2023-10-05T19:29:40.867 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.854+0000 7fd5e8da6700 1 -- 172.21.15.167:0/3205430407 <== osd.3 v2:172.21.15.167:6808/30036 44 ==== osd_op_reply(138 notify.2 [notify cookie 94571441093872 out=8b] v0'0 uv252 ondisk = 0) v8 ==== 152+0+8 (crc 0 0 0) 0x7fd5d802b070 con 0x5603211c7650 2023-10-05T19:29:40.868 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.854+0000 7fd5e8da6700 1 -- 172.21.15.167:0/3205430407 <== osd.3 v2:172.21.15.167:6808/30036 45 ==== osd_op_reply(139 notify.2 [notify-ack] v0'0 uv252 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7fd5d802b070 con 0x5603211c7650 2023-10-05T19:29:40.868 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.858+0000 7fd5e8da6700 1 -- 172.21.15.167:0/3205430407 <== osd.3 v2:172.21.15.167:6808/30036 46 ==== watch-notify(notify_complete (2) cookie 94571441093872 notify 137438953491 ret 0) v3 ==== 42+0+48 (crc 0 0 0) 0x5603211b8710 con 0x5603211c7650 2023-10-05T19:29:40.868 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.858+0000 7fd5eb271e40 20 lifecycle: handle_bi_update(): bucket=:myfoo6[9a9df965-6612-45d4-a45f-6e3c889bacc2.4223.7]): orig_sources= new_sources= 2023-10-05T19:29:40.868 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.858+0000 7fd5eb271e40 20 lifecycle: handle_bi_update(): bucket=:myfoo6[9a9df965-6612-45d4-a45f-6e3c889bacc2.4223.7]): potential sources added=[] removed=[] 2023-10-05T19:29:40.868 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.858+0000 7fd5eb271e40 20 lifecycle: handle_bi_update(): bucket=:myfoo6[9a9df965-6612-45d4-a45f-6e3c889bacc2.4223.7]): orig_dests= new_dests= 2023-10-05T19:29:40.868 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.858+0000 7fd5eb271e40 20 lifecycle: handle_bi_update(): bucket=:myfoo6[9a9df965-6612-45d4-a45f-6e3c889bacc2.4223.7]): potential dests added=[] removed=[] 2023-10-05T19:29:40.868 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.858+0000 7fd5eb271e40 1 -- 172.21.15.167:0/2343252391 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:33 6.1f 6:ffe20264:lc::lc.0:head [call lock.lock in=65b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e32) v8 -- 0x56032127c920 con 0x5603211ec370 2023-10-05T19:29:40.868 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.858+0000 7fd5e85a5700 1 -- 172.21.15.167:0/2343252391 <== osd.0 v2:172.21.15.118:6800/27690 8 ==== osd_op_reply(33 lc.0 [call] v32'40 uv40 ondisk = 0) v8 ==== 148+0+0 (crc 0 0 0) 0x7fd5d000f560 con 0x5603211ec370 2023-10-05T19:29:40.868 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.858+0000 7fd5eb271e40 1 -- 172.21.15.167:0/2343252391 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:34 6.1f 6:ffe20264:lc::lc.0:head [call rgw.lc_rm_entry in=93b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32) v8 -- 0x56032127c920 con 0x5603211ec370 2023-10-05T19:29:40.868 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.862+0000 7fd5e85a5700 1 -- 172.21.15.167:0/2343252391 <== osd.0 v2:172.21.15.118:6800/27690 9 ==== osd_op_reply(34 lc.0 [call] v32'41 uv41 ondisk = 0) v8 ==== 148+0+0 (crc 0 0 0) 0x7fd5d000f560 con 0x5603211ec370 2023-10-05T19:29:40.868 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.862+0000 7fd5eb271e40 1 -- 172.21.15.167:0/2343252391 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:35 6.1f 6:ffe20264:lc::lc.0:head [call lock.unlock in=49b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e32) v8 -- 0x56032127c920 con 0x5603211ec370 2023-10-05T19:29:40.872 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.862+0000 7fd5e85a5700 1 -- 172.21.15.167:0/2343252391 <== osd.0 v2:172.21.15.118:6800/27690 10 ==== osd_op_reply(35 lc.0 [call] v32'42 uv42 ondisk = 0) v8 ==== 148+0+0 (crc 0 0 0) 0x7fd5d000f560 con 0x5603211ec370 2023-10-05T19:29:40.872 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.862+0000 7fd5eb271e40 1 -- 172.21.15.167:0/3205430407 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:140 4.c 4:3398558f:::.dir.9a9df965-6612-45d4-a45f-6e3c889bacc2.4223.7.0:head [call rgw.bucket_list in=47b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32) v8 -- 0x56032127c920 con 0x7fd5d8010070 2023-10-05T19:29:40.872 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.862+0000 7fd5eb271e40 1 -- 172.21.15.167:0/3205430407 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:141 4.39 4:9d7aa60a:::.dir.9a9df965-6612-45d4-a45f-6e3c889bacc2.4223.7.1:head [call rgw.bucket_list in=47b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32) v8 -- 0x560321277740 con 0x5603211c7650 2023-10-05T19:29:40.872 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.862+0000 7fd5eb271e40 1 -- 172.21.15.167:0/3205430407 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:142 4.29 4:94bf6966:::.dir.9a9df965-6612-45d4-a45f-6e3c889bacc2.4223.7.2:head [call rgw.bucket_list in=47b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32) v8 -- 0x56032127c920 con 0x5603211c7650 2023-10-05T19:29:40.872 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.862+0000 7fd5eb271e40 1 -- 172.21.15.167:0/3205430407 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:143 4.1f 4:f8b953cb:::.dir.9a9df965-6612-45d4-a45f-6e3c889bacc2.4223.7.3:head [call rgw.bucket_list in=47b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32) v8 -- 0x560321264400 con 0x5603211c7650 2023-10-05T19:29:40.872 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.862+0000 7fd5eb271e40 1 -- 172.21.15.167:0/3205430407 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:144 4.30 4:0f0fb17a:::.dir.9a9df965-6612-45d4-a45f-6e3c889bacc2.4223.7.4:head [call rgw.bucket_list in=47b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32) v8 -- 0x560321277740 con 0x5603211c7650 2023-10-05T19:29:40.872 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.862+0000 7fd5eb271e40 1 -- 172.21.15.167:0/3205430407 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:145 4.1c 4:38fdb684:::.dir.9a9df965-6612-45d4-a45f-6e3c889bacc2.4223.7.5:head [call rgw.bucket_list in=47b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32) v8 -- 0x56032127c920 con 0x7fd5d000e8c0 2023-10-05T19:29:40.872 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.862+0000 7fd5eb271e40 1 -- 172.21.15.167:0/3205430407 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:146 4.30 4:0eea655d:::.dir.9a9df965-6612-45d4-a45f-6e3c889bacc2.4223.7.6:head [call rgw.bucket_list in=47b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32) v8 -- 0x560321264400 con 0x5603211c7650 2023-10-05T19:29:40.873 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.862+0000 7fd5eb271e40 1 -- 172.21.15.167:0/3205430407 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:147 4.1f 4:f8bb09cf:::.dir.9a9df965-6612-45d4-a45f-6e3c889bacc2.4223.7.7:head [call rgw.bucket_list in=47b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32) v8 -- 0x560321277740 con 0x5603211c7650 2023-10-05T19:29:40.873 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.862+0000 7fd5eb271e40 1 -- 172.21.15.167:0/3205430407 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:148 4.15 4:aaa2eb30:::.dir.9a9df965-6612-45d4-a45f-6e3c889bacc2.4223.7.8:head [call rgw.bucket_list in=47b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32) v8 -- 0x56032127c920 con 0x5603211c7650 2023-10-05T19:29:40.873 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.862+0000 7fd5eb271e40 1 -- 172.21.15.167:0/3205430407 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:149 4.3a 4:5edbdf2e:::.dir.9a9df965-6612-45d4-a45f-6e3c889bacc2.4223.7.9:head [call rgw.bucket_list in=47b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32) v8 -- 0x560321264400 con 0x5603211c7650 2023-10-05T19:29:40.873 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.862+0000 7fd5eb271e40 1 -- 172.21.15.167:0/3205430407 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:150 4.33 4:cf2dd822:::.dir.9a9df965-6612-45d4-a45f-6e3c889bacc2.4223.7.10:head [call rgw.bucket_list in=47b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32) v8 -- 0x560321277740 con 0x7fd5d000e8c0 2023-10-05T19:29:40.873 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.862+0000 7fd5e85a5700 1 -- 172.21.15.167:0/3205430407 <== osd.2 v2:172.21.15.167:6816/30037 24 ==== osd_op_reply(140 .dir.9a9df965-6612-45d4-a45f-6e3c889bacc2.4223.7.0 [call out=85b] v0'0 uv1 ondisk = 0) v8 ==== 194+0+85 (crc 0 0 0) 0x7fd5d000f560 con 0x7fd5d8010070 2023-10-05T19:29:40.873 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.862+0000 7fd5e8da6700 1 -- 172.21.15.167:0/3205430407 <== osd.3 v2:172.21.15.167:6808/30036 47 ==== osd_op_reply(141 .dir.9a9df965-6612-45d4-a45f-6e3c889bacc2.4223.7.1 [call out=85b] v0'0 uv3 ondisk = 0) v8 ==== 194+0+85 (crc 0 0 0) 0x7fd5d8075040 con 0x5603211c7650 2023-10-05T19:29:40.873 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.862+0000 7fd5e8da6700 1 -- 172.21.15.167:0/3205430407 <== osd.3 v2:172.21.15.167:6808/30036 48 ==== osd_op_reply(143 .dir.9a9df965-6612-45d4-a45f-6e3c889bacc2.4223.7.3 [call out=85b] v0'0 uv4 ondisk = 0) v8 ==== 194+0+85 (crc 0 0 0) 0x7fd5d8075040 con 0x5603211c7650 2023-10-05T19:29:40.873 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.862+0000 7fd5e95a7700 1 -- 172.21.15.167:0/3205430407 <== osd.0 v2:172.21.15.118:6800/27690 33 ==== osd_op_reply(145 .dir.9a9df965-6612-45d4-a45f-6e3c889bacc2.4223.7.5 [call out=85b] v0'0 uv1 ondisk = 0) v8 ==== 194+0+85 (crc 0 0 0) 0x7fd5e0050040 con 0x7fd5d000e8c0 2023-10-05T19:29:40.873 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.862+0000 7fd5e8da6700 1 -- 172.21.15.167:0/3205430407 <== osd.3 v2:172.21.15.167:6808/30036 49 ==== osd_op_reply(142 .dir.9a9df965-6612-45d4-a45f-6e3c889bacc2.4223.7.2 [call out=85b] v0'0 uv13 ondisk = 0) v8 ==== 194+0+85 (crc 0 0 0) 0x7fd5d8075040 con 0x5603211c7650 2023-10-05T19:29:40.873 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.862+0000 7fd5e8da6700 1 -- 172.21.15.167:0/3205430407 <== osd.3 v2:172.21.15.167:6808/30036 50 ==== osd_op_reply(149 .dir.9a9df965-6612-45d4-a45f-6e3c889bacc2.4223.7.9 [call out=124b] v0'0 uv9 ondisk = 0) v8 ==== 194+0+124 (crc 0 0 0) 0x7fd5d8075040 con 0x5603211c7650 2023-10-05T19:29:40.873 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.862+0000 7fd5e8da6700 1 -- 172.21.15.167:0/3205430407 <== osd.3 v2:172.21.15.167:6808/30036 51 ==== osd_op_reply(144 .dir.9a9df965-6612-45d4-a45f-6e3c889bacc2.4223.7.4 [call out=85b] v0'0 uv1 ondisk = 0) v8 ==== 194+0+85 (crc 0 0 0) 0x7fd5d8075040 con 0x5603211c7650 2023-10-05T19:29:40.874 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.862+0000 7fd5e8da6700 1 -- 172.21.15.167:0/3205430407 <== osd.3 v2:172.21.15.167:6808/30036 52 ==== osd_op_reply(148 .dir.9a9df965-6612-45d4-a45f-6e3c889bacc2.4223.7.8 [call out=85b] v0'0 uv1 ondisk = 0) v8 ==== 194+0+85 (crc 0 0 0) 0x7fd5d8075040 con 0x5603211c7650 2023-10-05T19:29:40.874 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.862+0000 7fd5e8da6700 1 -- 172.21.15.167:0/3205430407 <== osd.3 v2:172.21.15.167:6808/30036 53 ==== osd_op_reply(147 .dir.9a9df965-6612-45d4-a45f-6e3c889bacc2.4223.7.7 [call out=85b] v0'0 uv5 ondisk = 0) v8 ==== 194+0+85 (crc 0 0 0) 0x7fd5d8075040 con 0x5603211c7650 2023-10-05T19:29:40.874 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.862+0000 7fd5e8da6700 1 -- 172.21.15.167:0/3205430407 <== osd.3 v2:172.21.15.167:6808/30036 54 ==== osd_op_reply(146 .dir.9a9df965-6612-45d4-a45f-6e3c889bacc2.4223.7.6 [call out=85b] v0'0 uv2 ondisk = 0) v8 ==== 194+0+85 (crc 0 0 0) 0x7fd5d8075040 con 0x5603211c7650 2023-10-05T19:29:40.874 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.862+0000 7fd5e95a7700 1 -- 172.21.15.167:0/3205430407 <== osd.0 v2:172.21.15.118:6800/27690 34 ==== osd_op_reply(150 .dir.9a9df965-6612-45d4-a45f-6e3c889bacc2.4223.7.10 [call out=85b] v0'0 uv6 ondisk = 0) v8 ==== 195+0+85 (crc 0 0 0) 0x7fd5e0050040 con 0x7fd5d000e8c0 2023-10-05T19:29:40.874 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.866+0000 7fd5eb271e40 1 -- 172.21.15.167:0/3205430407 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:151 8.19 8:99adfd5d:users.uid::bud.buckets:head [call user.set_buckets_info in=208b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e32) v8 -- 0x56032127c920 con 0x5603211907f0 2023-10-05T19:29:40.874 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.866+0000 7fd5e95a7700 1 -- 172.21.15.167:0/3205430407 <== osd.1 v2:172.21.15.167:6800/30035 41 ==== osd_op_reply(151 bud.buckets [call] v32'4 uv4 ondisk = 0) v8 ==== 155+0+0 (crc 0 0 0) 0x7fd5e0050040 con 0x5603211907f0 2023-10-05T19:29:40.874 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.866+0000 7fd5eb271e40 20 get_system_obj_state: rctx=0x7ffeedd66178 obj=default.rgw.meta:root:myfoo6 state=0x5603210690b0 s->prefetch_data=0 2023-10-05T19:29:40.874 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.866+0000 7fd5eb271e40 10 cache get: name=default.rgw.meta+root+myfoo6 : type miss (requested=0x16, cached=0x13) 2023-10-05T19:29:40.874 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.866+0000 7fd5eb271e40 1 -- 172.21.15.167:0/3205430407 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:152 8.18 8:1944a257:root::myfoo6:head [call version.read in=11b,getxattrs,stat] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32) v8 -- 0x560321277740 con 0x5603211907f0 2023-10-05T19:29:40.874 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.866+0000 7fd5e95a7700 1 -- 172.21.15.167:0/3205430407 <== osd.1 v2:172.21.15.167:6800/30035 42 ==== osd_op_reply(152 myfoo6 [call out=48b,getxattrs out=75b,stat out=16b] v0'0 uv18 ondisk = 0) v8 ==== 234+0+139 (crc 0 0 0) 0x7fd5e0050040 con 0x5603211907f0 2023-10-05T19:29:40.875 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.866+0000 7fd5eb271e40 10 cache put: name=default.rgw.meta+root+myfoo6 info.flags=0x16 2023-10-05T19:29:40.875 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.866+0000 7fd5eb271e40 10 moving default.rgw.meta+root+myfoo6 to cache LRU end 2023-10-05T19:29:40.875 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.866+0000 7fd5eb271e40 10 updating xattr: name=ceph.objclass.version bl.length()=42 2023-10-05T19:29:40.875 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.866+0000 7fd5eb271e40 20 get_system_obj_state: s->obj_tag was set empty 2023-10-05T19:29:40.875 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.866+0000 7fd5eb271e40 10 cache get: name=default.rgw.meta+root+myfoo6 : hit (requested=0x11, cached=0x17) 2023-10-05T19:29:40.875 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.866+0000 7fd5eb271e40 10 removing default.rgw.meta+root+myfoo6 from cache 2023-10-05T19:29:40.875 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.866+0000 7fd5eb271e40 10 distributing notification oid=default.rgw.control:notify.3 cni=[op: 1, obj: default.rgw.meta:root:myfoo6, ofs0, ns] 2023-10-05T19:29:40.875 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.866+0000 7fd5eb271e40 1 -- 172.21.15.167:0/3205430407 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:153 7.e 7:7759931f:::notify.3:head [notify cookie 94571441093872 in=162b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32) v8 -- 0x56032127c920 con 0x7fd5d8010070 2023-10-05T19:29:40.875 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.866+0000 7fd5e85a5700 1 -- 172.21.15.167:0/3205430407 <== osd.2 v2:172.21.15.167:6816/30037 25 ==== watch-notify(notify (1) cookie 94571440437408 notify 137438953490 ret 0) v3 ==== 192+0+0 (crc 0 0 0) 0x7fd5d0064040 con 0x7fd5d8010070 2023-10-05T19:29:40.875 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.866+0000 7fd5e6da2700 10 rgw watcher librados: RGWWatcher::handle_notify() notify_id 137438953490 cookie 94571440437408 notifier 4731 bl.length()=150 2023-10-05T19:29:40.875 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.866+0000 7fd5e6da2700 1 -- 172.21.15.167:0/3205430407 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:154 7.e 7:7759931f:::notify.3:head [notify-ack in=20b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32) v8 -- 0x7fd5cc003d90 con 0x7fd5d8010070 2023-10-05T19:29:40.875 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.866+0000 7fd5e85a5700 1 -- 172.21.15.167:0/3205430407 <== osd.2 v2:172.21.15.167:6816/30037 26 ==== osd_op_reply(153 notify.3 [notify cookie 94571441093872 out=8b] v0'0 uv252 ondisk = 0) v8 ==== 152+0+8 (crc 0 0 0) 0x7fd5d000f560 con 0x7fd5d8010070 2023-10-05T19:29:40.876 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.866+0000 7fd5e85a5700 1 -- 172.21.15.167:0/3205430407 <== osd.2 v2:172.21.15.167:6816/30037 27 ==== osd_op_reply(154 notify.3 [notify-ack] v0'0 uv252 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7fd5d000f560 con 0x7fd5d8010070 2023-10-05T19:29:40.876 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.866+0000 7fd5e85a5700 1 -- 172.21.15.167:0/3205430407 <== osd.2 v2:172.21.15.167:6816/30037 28 ==== watch-notify(notify_complete (2) cookie 94571441093872 notify 137438953490 ret 0) v3 ==== 42+0+48 (crc 0 0 0) 0x5603211e6580 con 0x7fd5d8010070 2023-10-05T19:29:40.876 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.866+0000 7fd5eb271e40 1 -- 172.21.15.167:0/3205430407 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:155 8.18 8:1944a257:root::myfoo6:head [call version.check_conds in=122b,call version.set in=58b,delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e32) v8 -- 0x560321277740 con 0x5603211907f0 2023-10-05T19:29:40.878 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.870+0000 7fd5e95a7700 1 -- 172.21.15.167:0/3205430407 <== osd.1 v2:172.21.15.167:6800/30035 43 ==== osd_op_reply(155 myfoo6 [call,call,delete] v32'19 uv19 ondisk = 0) v8 ==== 234+0+0 (crc 0 0 0) 0x7fd5e0050040 con 0x5603211907f0 2023-10-05T19:29:40.878 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.870+0000 7fd5eb271e40 10 removing default.rgw.meta+root+.bucket.meta.myfoo6:9a9df965-6612-45d4-a45f-6e3c889bacc2.4223.7 from cache 2023-10-05T19:29:40.878 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.870+0000 7fd5eb271e40 10 distributing notification oid=default.rgw.control:notify.2 cni=[op: 1, obj: default.rgw.meta:root:.bucket.meta.myfoo6:9a9df965-6612-45d4-a45f-6e3c889bacc2.4223.7, ofs0, ns] 2023-10-05T19:29:40.878 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.870+0000 7fd5eb271e40 1 -- 172.21.15.167:0/3205430407 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:156 7.15 7:a93a5511:::notify.2:head [notify cookie 94571441093872 in=219b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32) v8 -- 0x56032127c920 con 0x5603211c7650 2023-10-05T19:29:40.878 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.870+0000 7fd5e8da6700 1 -- 172.21.15.167:0/3205430407 <== osd.3 v2:172.21.15.167:6808/30036 55 ==== watch-notify(notify (1) cookie 94571440425072 notify 137438953492 ret 0) v3 ==== 249+0+0 (crc 0 0 0) 0x5603211b8710 con 0x5603211c7650 2023-10-05T19:29:40.878 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.870+0000 7fd5e65a1700 10 rgw watcher librados: RGWWatcher::handle_notify() notify_id 137438953492 cookie 94571440425072 notifier 4731 bl.length()=207 2023-10-05T19:29:40.879 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.870+0000 7fd5e8da6700 1 -- 172.21.15.167:0/3205430407 <== osd.3 v2:172.21.15.167:6808/30036 56 ==== osd_op_reply(156 notify.2 [notify cookie 94571441093872 out=8b] v0'0 uv252 ondisk = 0) v8 ==== 152+0+8 (crc 0 0 0) 0x7fd5d8075040 con 0x5603211c7650 2023-10-05T19:29:40.879 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.870+0000 7fd5e65a1700 1 -- 172.21.15.167:0/3205430407 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:157 7.15 7:a93a5511:::notify.2:head [notify-ack in=20b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32) v8 -- 0x7fd538003340 con 0x5603211c7650 2023-10-05T19:29:40.879 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.870+0000 7fd5e8da6700 1 -- 172.21.15.167:0/3205430407 <== osd.3 v2:172.21.15.167:6808/30036 57 ==== osd_op_reply(157 notify.2 [notify-ack] v0'0 uv252 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7fd5d8075040 con 0x5603211c7650 2023-10-05T19:29:40.879 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.870+0000 7fd5e8da6700 1 -- 172.21.15.167:0/3205430407 <== osd.3 v2:172.21.15.167:6808/30036 58 ==== watch-notify(notify_complete (2) cookie 94571441093872 notify 137438953492 ret 0) v3 ==== 42+0+48 (crc 0 0 0) 0x5603211b82d0 con 0x5603211c7650 2023-10-05T19:29:40.879 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.870+0000 7fd5eb271e40 1 -- 172.21.15.167:0/3205430407 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:158 8.1f 8:fe511ebf:root::.bucket.meta.myfoo6%3a9a9df965-6612-45d4-a45f-6e3c889bacc2.4223.7:head [call version.check_conds in=122b,call version.inc in=38b,delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e32) v8 -- 0x560321277740 con 0x7fd5d8010070 2023-10-05T19:29:40.879 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.874+0000 7fd5e85a5700 1 -- 172.21.15.167:0/3205430407 <== osd.2 v2:172.21.15.167:6816/30037 29 ==== osd_op_reply(158 .bucket.meta.myfoo6:9a9df965-6612-45d4-a45f-6e3c889bacc2.4223.7 [call,call,delete] v32'4 uv4 ondisk = 0) v8 ==== 291+0+0 (crc 0 0 0) 0x7fd5d000f560 con 0x7fd5d8010070 2023-10-05T19:29:40.879 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.874+0000 7fd5eb271e40 1 -- 172.21.15.167:0/3205430407 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:159 4.c 4:3398558f:::.dir.9a9df965-6612-45d4-a45f-6e3c889bacc2.4223.7.0:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e32) v8 -- 0x56032127c920 con 0x7fd5d8010070 2023-10-05T19:29:40.879 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.874+0000 7fd5eb271e40 1 -- 172.21.15.167:0/3205430407 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:160 4.39 4:9d7aa60a:::.dir.9a9df965-6612-45d4-a45f-6e3c889bacc2.4223.7.1:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e32) v8 -- 0x560321277740 con 0x5603211c7650 2023-10-05T19:29:40.879 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.874+0000 7fd5eb271e40 1 -- 172.21.15.167:0/3205430407 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:161 4.29 4:94bf6966:::.dir.9a9df965-6612-45d4-a45f-6e3c889bacc2.4223.7.2:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e32) v8 -- 0x5603212606a0 con 0x5603211c7650 2023-10-05T19:29:40.879 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.874+0000 7fd5eb271e40 1 -- 172.21.15.167:0/3205430407 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:162 4.1f 4:f8b953cb:::.dir.9a9df965-6612-45d4-a45f-6e3c889bacc2.4223.7.3:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e32) v8 -- 0x560321266a60 con 0x5603211c7650 2023-10-05T19:29:40.879 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.874+0000 7fd5eb271e40 1 -- 172.21.15.167:0/3205430407 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:163 4.30 4:0f0fb17a:::.dir.9a9df965-6612-45d4-a45f-6e3c889bacc2.4223.7.4:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e32) v8 -- 0x56032127c920 con 0x5603211c7650 2023-10-05T19:29:40.880 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.874+0000 7fd5eb271e40 1 -- 172.21.15.167:0/3205430407 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:164 4.1c 4:38fdb684:::.dir.9a9df965-6612-45d4-a45f-6e3c889bacc2.4223.7.5:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e32) v8 -- 0x560321277740 con 0x7fd5d000e8c0 2023-10-05T19:29:40.880 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.874+0000 7fd5eb271e40 1 -- 172.21.15.167:0/3205430407 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:165 4.30 4:0eea655d:::.dir.9a9df965-6612-45d4-a45f-6e3c889bacc2.4223.7.6:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e32) v8 -- 0x5603212606a0 con 0x5603211c7650 2023-10-05T19:29:40.880 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.874+0000 7fd5eb271e40 1 -- 172.21.15.167:0/3205430407 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:166 4.1f 4:f8bb09cf:::.dir.9a9df965-6612-45d4-a45f-6e3c889bacc2.4223.7.7:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e32) v8 -- 0x560321266a60 con 0x5603211c7650 2023-10-05T19:29:40.880 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.874+0000 7fd5eb271e40 1 -- 172.21.15.167:0/3205430407 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:167 4.15 4:aaa2eb30:::.dir.9a9df965-6612-45d4-a45f-6e3c889bacc2.4223.7.8:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e32) v8 -- 0x56032127c920 con 0x5603211c7650 2023-10-05T19:29:40.880 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.874+0000 7fd5eb271e40 1 -- 172.21.15.167:0/3205430407 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:168 4.3a 4:5edbdf2e:::.dir.9a9df965-6612-45d4-a45f-6e3c889bacc2.4223.7.9:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e32) v8 -- 0x560321277740 con 0x5603211c7650 2023-10-05T19:29:40.880 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.874+0000 7fd5eb271e40 1 -- 172.21.15.167:0/3205430407 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:169 4.33 4:cf2dd822:::.dir.9a9df965-6612-45d4-a45f-6e3c889bacc2.4223.7.10:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e32) v8 -- 0x56032126e800 con 0x7fd5d000e8c0 2023-10-05T19:29:40.880 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.874+0000 7fd5e95a7700 1 -- 172.21.15.167:0/3205430407 <== osd.0 v2:172.21.15.118:6800/27690 35 ==== osd_op_reply(164 .dir.9a9df965-6612-45d4-a45f-6e3c889bacc2.4223.7.5 [delete] v32'2 uv2 ondisk = 0) v8 ==== 194+0+0 (crc 0 0 0) 0x7fd5e0050040 con 0x7fd5d000e8c0 2023-10-05T19:29:40.881 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.874+0000 7fd5e8da6700 1 -- 172.21.15.167:0/3205430407 <== osd.3 v2:172.21.15.167:6808/30036 59 ==== osd_op_reply(160 .dir.9a9df965-6612-45d4-a45f-6e3c889bacc2.4223.7.1 [delete] v32'4 uv4 ondisk = 0) v8 ==== 194+0+0 (crc 0 0 0) 0x7fd5d8075040 con 0x5603211c7650 2023-10-05T19:29:40.882 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.878+0000 7fd5e8da6700 1 -- 172.21.15.167:0/3205430407 <== osd.3 v2:172.21.15.167:6808/30036 60 ==== osd_op_reply(163 .dir.9a9df965-6612-45d4-a45f-6e3c889bacc2.4223.7.4 [delete] v32'3 uv3 ondisk = 0) v8 ==== 194+0+0 (crc 0 0 0) 0x7fd5d8075040 con 0x5603211c7650 2023-10-05T19:29:40.882 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.878+0000 7fd5e8da6700 1 -- 172.21.15.167:0/3205430407 <== osd.3 v2:172.21.15.167:6808/30036 61 ==== osd_op_reply(161 .dir.9a9df965-6612-45d4-a45f-6e3c889bacc2.4223.7.2 [delete] v32'14 uv14 ondisk = 0) v8 ==== 194+0+0 (crc 0 0 0) 0x7fd5d8075040 con 0x5603211c7650 2023-10-05T19:29:40.882 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.878+0000 7fd5e8da6700 1 -- 172.21.15.167:0/3205430407 <== osd.3 v2:172.21.15.167:6808/30036 62 ==== osd_op_reply(162 .dir.9a9df965-6612-45d4-a45f-6e3c889bacc2.4223.7.3 [delete] v32'6 uv6 ondisk = 0) v8 ==== 194+0+0 (crc 0 0 0) 0x7fd5d8075040 con 0x5603211c7650 2023-10-05T19:29:40.882 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.878+0000 7fd5e8da6700 1 -- 172.21.15.167:0/3205430407 <== osd.3 v2:172.21.15.167:6808/30036 63 ==== osd_op_reply(165 .dir.9a9df965-6612-45d4-a45f-6e3c889bacc2.4223.7.6 [delete] v32'4 uv4 ondisk = 0) v8 ==== 194+0+0 (crc 0 0 0) 0x7fd5d8075040 con 0x5603211c7650 2023-10-05T19:29:40.882 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.878+0000 7fd5e85a5700 1 -- 172.21.15.167:0/3205430407 <== osd.2 v2:172.21.15.167:6816/30037 30 ==== osd_op_reply(159 .dir.9a9df965-6612-45d4-a45f-6e3c889bacc2.4223.7.0 [delete] v32'2 uv2 ondisk = 0) v8 ==== 194+0+0 (crc 0 0 0) 0x7fd5d000f560 con 0x7fd5d8010070 2023-10-05T19:29:40.882 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.878+0000 7fd5e95a7700 1 -- 172.21.15.167:0/3205430407 <== osd.0 v2:172.21.15.118:6800/27690 36 ==== osd_op_reply(169 .dir.9a9df965-6612-45d4-a45f-6e3c889bacc2.4223.7.10 [delete] v32'7 uv7 ondisk = 0) v8 ==== 195+0+0 (crc 0 0 0) 0x7fd5e0050040 con 0x7fd5d000e8c0 2023-10-05T19:29:40.883 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.878+0000 7fd5e8da6700 1 -- 172.21.15.167:0/3205430407 <== osd.3 v2:172.21.15.167:6808/30036 64 ==== osd_op_reply(167 .dir.9a9df965-6612-45d4-a45f-6e3c889bacc2.4223.7.8 [delete] v32'2 uv2 ondisk = 0) v8 ==== 194+0+0 (crc 0 0 0) 0x7fd5d8075040 con 0x5603211c7650 2023-10-05T19:29:40.883 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.878+0000 7fd5e8da6700 1 -- 172.21.15.167:0/3205430407 <== osd.3 v2:172.21.15.167:6808/30036 65 ==== osd_op_reply(168 .dir.9a9df965-6612-45d4-a45f-6e3c889bacc2.4223.7.9 [delete] v32'10 uv10 ondisk = 0) v8 ==== 194+0+0 (crc 0 0 0) 0x7fd5d8075040 con 0x5603211c7650 2023-10-05T19:29:40.883 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.878+0000 7fd5e8da6700 1 -- 172.21.15.167:0/3205430407 <== osd.3 v2:172.21.15.167:6808/30036 66 ==== osd_op_reply(166 .dir.9a9df965-6612-45d4-a45f-6e3c889bacc2.4223.7.7 [delete] v32'7 uv7 ondisk = 0) v8 ==== 194+0+0 (crc 0 0 0) 0x7fd5d8075040 con 0x5603211c7650 2023-10-05T19:29:40.884 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.878+0000 7fd5eb271e40 1 -- 172.21.15.167:0/3205430407 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:170 6.18 6:1ff1ff30:::pubsub..bucket.myfoo6%2f9a9df965-6612-45d4-a45f-6e3c889bacc2.4223.7:head [getxattrs,stat] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32) v8 -- 0x56032127c920 con 0x7fd5d000e8c0 2023-10-05T19:29:40.884 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.878+0000 7fd5e95a7700 1 -- 172.21.15.167:0/3205430407 <== osd.0 v2:172.21.15.118:6800/27690 37 ==== osd_op_reply(170 pubsub..bucket.myfoo6/9a9df965-6612-45d4-a45f-6e3c889bacc2.4223.7 [getxattrs,stat] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 ==== 251+0+0 (crc 0 0 0) 0x7fd5e0050040 con 0x7fd5d000e8c0 2023-10-05T19:29:40.884 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.878+0000 7fd5eb271e40 10 removing default.rgw.log++pubsub..bucket.myfoo6/9a9df965-6612-45d4-a45f-6e3c889bacc2.4223.7 from cache 2023-10-05T19:29:40.884 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.878+0000 7fd5eb271e40 10 distributing notification oid=default.rgw.control:notify.1 cni=[op: 1, obj: default.rgw.log:pubsub..bucket.myfoo6/9a9df965-6612-45d4-a45f-6e3c889bacc2.4223.7, ofs0, ns] 2023-10-05T19:29:40.884 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.878+0000 7fd5eb271e40 1 -- 172.21.15.167:0/3205430407 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:171 7.0 7:05bf5b68:::notify.1:head [notify cookie 94571441093872 in=216b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32) v8 -- 0x560321277740 con 0x5603211907f0 2023-10-05T19:29:40.885 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.878+0000 7fd5e95a7700 1 -- 172.21.15.167:0/3205430407 <== osd.1 v2:172.21.15.167:6800/30035 44 ==== watch-notify(notify (1) cookie 94571440429536 notify 137438953493 ret 0) v3 ==== 246+0+0 (crc 0 0 0) 0x7fd5e0075040 con 0x5603211907f0 2023-10-05T19:29:40.885 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.878+0000 7fd5e6da2700 10 rgw watcher librados: RGWWatcher::handle_notify() notify_id 137438953493 cookie 94571440429536 notifier 4731 bl.length()=204 2023-10-05T19:29:40.885 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.878+0000 7fd5e6da2700 1 -- 172.21.15.167:0/3205430407 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:172 7.0 7:05bf5b68:::notify.1:head [notify-ack in=20b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32) v8 -- 0x7fd5cc003d90 con 0x5603211907f0 2023-10-05T19:29:40.885 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.878+0000 7fd5e95a7700 1 -- 172.21.15.167:0/3205430407 <== osd.1 v2:172.21.15.167:6800/30035 45 ==== osd_op_reply(171 notify.1 [notify cookie 94571441093872 out=8b] v0'0 uv252 ondisk = 0) v8 ==== 152+0+8 (crc 0 0 0) 0x7fd5e0050040 con 0x5603211907f0 2023-10-05T19:29:40.885 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.882+0000 7fd5e95a7700 1 -- 172.21.15.167:0/3205430407 <== osd.1 v2:172.21.15.167:6800/30035 46 ==== watch-notify(notify_complete (2) cookie 94571441093872 notify 137438953493 ret 0) v3 ==== 42+0+48 (crc 0 0 0) 0x7fd5e003d040 con 0x5603211907f0 2023-10-05T19:29:40.885 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.882+0000 7fd5e95a7700 1 -- 172.21.15.167:0/3205430407 <== osd.1 v2:172.21.15.167:6800/30035 47 ==== osd_op_reply(172 notify.1 [notify-ack] v0'0 uv252 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7fd5e0050040 con 0x5603211907f0 2023-10-05T19:29:40.885 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.882+0000 7fd5eb271e40 1 -- 172.21.15.167:0/3205430407 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:173 6.18 6:1ff1ff30:::pubsub..bucket.myfoo6%2f9a9df965-6612-45d4-a45f-6e3c889bacc2.4223.7:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e32) v8 -- 0x56032127c920 con 0x7fd5d000e8c0 2023-10-05T19:29:40.886 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.882+0000 7fd5e95a7700 1 -- 172.21.15.167:0/3205430407 <== osd.0 v2:172.21.15.118:6800/27690 38 ==== osd_op_reply(173 pubsub..bucket.myfoo6/9a9df965-6612-45d4-a45f-6e3c889bacc2.4223.7 [delete] v32'31 uv25 ondisk = -2 ((2) No such file or directory)) v8 ==== 209+0+0 (crc 0 0 0) 0x7fd5e0050040 con 0x7fd5d000e8c0 2023-10-05T19:29:40.887 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.882+0000 7fd5eb271e40 1 -- 172.21.15.167:0/3205430407 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:174 8.19 8:99adfd5d:users.uid::bud.buckets:head [call user.remove_bucket in=59b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e32) v8 -- 0x56032127c920 con 0x5603211907f0 2023-10-05T19:29:40.888 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.886+0000 7fd5e95a7700 1 -- 172.21.15.167:0/3205430407 <== osd.1 v2:172.21.15.167:6800/30035 48 ==== osd_op_reply(174 bud.buckets [call] v32'5 uv5 ondisk = 0) v8 ==== 155+0+0 (crc 0 0 0) 0x7fd5e0050040 con 0x5603211907f0 2023-10-05T19:29:40.889 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.886+0000 7fd5eb271e40 10 removing key index: NX5QOQKC6BH2IDN8HC7A 2023-10-05T19:29:40.889 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.886+0000 7fd5eb271e40 10 distributing notification oid=default.rgw.control:notify.5 cni=[op: 1, obj: default.rgw.meta:users.keys:NX5QOQKC6BH2IDN8HC7A, ofs0, ns] 2023-10-05T19:29:40.889 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.886+0000 7fd5eb271e40 1 -- 172.21.15.167:0/3205430407 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:175 7.3 7:c609908c:::notify.5:head [notify cookie 94571441093872 in=182b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32) v8 -- 0x56032127c920 con 0x7fd5d000e8c0 2023-10-05T19:29:40.889 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.886+0000 7fd5e95a7700 1 -- 172.21.15.167:0/3205430407 <== osd.0 v2:172.21.15.118:6800/27690 39 ==== watch-notify(notify (1) cookie 94571440430640 notify 137438953510 ret 0) v3 ==== 212+0+0 (crc 0 0 0) 0x7fd5e003d040 con 0x7fd5d000e8c0 2023-10-05T19:29:40.890 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.886+0000 7fd5e95a7700 1 -- 172.21.15.167:0/3205430407 <== osd.0 v2:172.21.15.118:6800/27690 40 ==== osd_op_reply(175 notify.5 [notify cookie 94571441093872 out=8b] v0'0 uv252 ondisk = 0) v8 ==== 152+0+8 (crc 0 0 0) 0x7fd5e0050040 con 0x7fd5d000e8c0 2023-10-05T19:29:40.890 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.886+0000 7fd5e65a1700 10 rgw watcher librados: RGWWatcher::handle_notify() notify_id 137438953510 cookie 94571440430640 notifier 4731 bl.length()=170 2023-10-05T19:29:40.890 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.886+0000 7fd5e65a1700 1 -- 172.21.15.167:0/3205430407 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:176 7.3 7:c609908c:::notify.5:head [notify-ack in=20b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32) v8 -- 0x7fd538003300 con 0x7fd5d000e8c0 2023-10-05T19:29:40.890 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.886+0000 7fd5e95a7700 1 -- 172.21.15.167:0/3205430407 <== osd.0 v2:172.21.15.118:6800/27690 41 ==== osd_op_reply(176 notify.5 [notify-ack] v0'0 uv252 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7fd5e0050040 con 0x7fd5d000e8c0 2023-10-05T19:29:40.890 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.886+0000 7fd5e95a7700 1 -- 172.21.15.167:0/3205430407 <== osd.0 v2:172.21.15.118:6800/27690 42 ==== watch-notify(notify_complete (2) cookie 94571441093872 notify 137438953510 ret 0) v3 ==== 42+0+48 (crc 0 0 0) 0x7fd5e0033790 con 0x7fd5d000e8c0 2023-10-05T19:29:40.891 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.886+0000 7fd5eb271e40 1 -- 172.21.15.167:0/3205430407 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:177 8.a 8:50c8eddb:users.keys::NX5QOQKC6BH2IDN8HC7A:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e32) v8 -- 0x56032127c920 con 0x5603211907f0 2023-10-05T19:29:40.892 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.886+0000 7fd5e95a7700 1 -- 172.21.15.167:0/3205430407 <== osd.1 v2:172.21.15.167:6800/30035 49 ==== osd_op_reply(177 NX5QOQKC6BH2IDN8HC7A [delete] v32'3 uv3 ondisk = 0) v8 ==== 164+0+0 (crc 0 0 0) 0x7fd5e0050040 con 0x5603211907f0 2023-10-05T19:29:40.892 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.886+0000 7fd5eb271e40 10 removing email index: 2023-10-05T19:29:40.892 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.886+0000 7fd5eb271e40 10 removing user buckets index 2023-10-05T19:29:40.892 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.886+0000 7fd5eb271e40 10 distributing notification oid=default.rgw.control:notify.6 cni=[op: 1, obj: default.rgw.meta:users.uid:bud.buckets, ofs0, ns] 2023-10-05T19:29:40.893 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.886+0000 7fd5eb271e40 1 -- 172.21.15.167:0/3205430407 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:178 7.14 7:2b04a3e9:::notify.6:head [notify cookie 94571441093872 in=172b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32) v8 -- 0x56032127c920 con 0x7fd5d000e8c0 2023-10-05T19:29:40.893 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.890+0000 7fd5e95a7700 1 -- 172.21.15.167:0/3205430407 <== osd.0 v2:172.21.15.118:6800/27690 43 ==== watch-notify(notify (1) cookie 94571440445360 notify 137438953511 ret 0) v3 ==== 202+0+0 (crc 0 0 0) 0x7fd5e0033790 con 0x7fd5d000e8c0 2023-10-05T19:29:40.893 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.890+0000 7fd5e65a1700 10 rgw watcher librados: RGWWatcher::handle_notify() notify_id 137438953511 cookie 94571440445360 notifier 4731 bl.length()=160 2023-10-05T19:29:40.893 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.890+0000 7fd5e95a7700 1 -- 172.21.15.167:0/3205430407 <== osd.0 v2:172.21.15.118:6800/27690 44 ==== osd_op_reply(178 notify.6 [notify cookie 94571441093872 out=8b] v0'0 uv252 ondisk = 0) v8 ==== 152+0+8 (crc 0 0 0) 0x7fd5e0050040 con 0x7fd5d000e8c0 2023-10-05T19:29:40.894 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.890+0000 7fd5e65a1700 1 -- 172.21.15.167:0/3205430407 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:179 7.14 7:2b04a3e9:::notify.6:head [notify-ack in=20b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32) v8 -- 0x7fd538003300 con 0x7fd5d000e8c0 2023-10-05T19:29:40.894 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.890+0000 7fd5e95a7700 1 -- 172.21.15.167:0/3205430407 <== osd.0 v2:172.21.15.118:6800/27690 45 ==== watch-notify(notify_complete (2) cookie 94571441093872 notify 137438953511 ret 0) v3 ==== 42+0+48 (crc 0 0 0) 0x7fd5e0033b00 con 0x7fd5d000e8c0 2023-10-05T19:29:40.894 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.890+0000 7fd5e95a7700 1 -- 172.21.15.167:0/3205430407 <== osd.0 v2:172.21.15.118:6800/27690 46 ==== osd_op_reply(179 notify.6 [notify-ack] v0'0 uv252 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7fd5e0050040 con 0x7fd5d000e8c0 2023-10-05T19:29:40.894 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.890+0000 7fd5eb271e40 1 -- 172.21.15.167:0/3205430407 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:180 8.19 8:99adfd5d:users.uid::bud.buckets:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e32) v8 -- 0x56032127c920 con 0x5603211907f0 2023-10-05T19:29:40.896 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.890+0000 7fd5e95a7700 1 -- 172.21.15.167:0/3205430407 <== osd.1 v2:172.21.15.167:6800/30035 50 ==== osd_op_reply(180 bud.buckets [delete] v32'6 uv6 ondisk = 0) v8 ==== 155+0+0 (crc 0 0 0) 0x7fd5e0050040 con 0x5603211907f0 2023-10-05T19:29:40.896 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.890+0000 7fd5eb271e40 10 removing user index: bud 2023-10-05T19:29:40.896 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.890+0000 7fd5eb271e40 10 removing default.rgw.meta+users.uid+bud from cache 2023-10-05T19:29:40.897 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.890+0000 7fd5eb271e40 10 distributing notification oid=default.rgw.control:notify.5 cni=[op: 1, obj: default.rgw.meta:users.uid:bud, ofs0, ns] 2023-10-05T19:29:40.897 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.890+0000 7fd5eb271e40 1 -- 172.21.15.167:0/3205430407 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:181 7.3 7:c609908c:::notify.5:head [notify cookie 94571441093872 in=164b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32) v8 -- 0x56032127c920 con 0x7fd5d000e8c0 2023-10-05T19:29:40.897 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.894+0000 7fd5e95a7700 1 -- 172.21.15.167:0/3205430407 <== osd.0 v2:172.21.15.118:6800/27690 47 ==== watch-notify(notify (1) cookie 94571440430640 notify 137438953512 ret 0) v3 ==== 194+0+0 (crc 0 0 0) 0x7fd5e0033b00 con 0x7fd5d000e8c0 2023-10-05T19:29:40.897 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.894+0000 7fd5e65a1700 10 rgw watcher librados: RGWWatcher::handle_notify() notify_id 137438953512 cookie 94571440430640 notifier 4731 bl.length()=152 2023-10-05T19:29:40.897 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.894+0000 7fd5e95a7700 1 -- 172.21.15.167:0/3205430407 <== osd.0 v2:172.21.15.118:6800/27690 48 ==== osd_op_reply(181 notify.5 [notify cookie 94571441093872 out=8b] v0'0 uv252 ondisk = 0) v8 ==== 152+0+8 (crc 0 0 0) 0x7fd5e0050040 con 0x7fd5d000e8c0 2023-10-05T19:29:40.898 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.894+0000 7fd5e65a1700 1 -- 172.21.15.167:0/3205430407 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:182 7.3 7:c609908c:::notify.5:head [notify-ack in=20b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32) v8 -- 0x7fd538003300 con 0x7fd5d000e8c0 2023-10-05T19:29:40.898 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.894+0000 7fd5e95a7700 1 -- 172.21.15.167:0/3205430407 <== osd.0 v2:172.21.15.118:6800/27690 49 ==== osd_op_reply(182 notify.5 [notify-ack] v0'0 uv252 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7fd5e0050040 con 0x7fd5d000e8c0 2023-10-05T19:29:40.898 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.894+0000 7fd5e95a7700 1 -- 172.21.15.167:0/3205430407 <== osd.0 v2:172.21.15.118:6800/27690 50 ==== watch-notify(notify_complete (2) cookie 94571441093872 notify 137438953512 ret 0) v3 ==== 42+0+48 (crc 0 0 0) 0x7fd5e0033d30 con 0x7fd5d000e8c0 2023-10-05T19:29:40.898 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.894+0000 7fd5eb271e40 1 -- 172.21.15.167:0/3205430407 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:183 8.4 8:267425a8:users.uid::bud:head [call version.inc in=38b,delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e32) v8 -- 0x56032127c920 con 0x7fd5d8010070 2023-10-05T19:29:40.900 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.898+0000 7fd5e85a5700 1 -- 172.21.15.167:0/3205430407 <== osd.2 v2:172.21.15.167:6816/30037 31 ==== osd_op_reply(183 bud [call,delete] v32'3 uv3 ondisk = 0) v8 ==== 189+0+0 (crc 0 0 0) 0x7fd5d000f560 con 0x7fd5d8010070 2023-10-05T19:29:40.901 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.898+0000 7fd5eb271e40 1 -- 172.21.15.167:0/3205430407 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:184 7.1f 7:f95f44c2:::notify.0:head [watch unwatch cookie 94571440422784] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e32) v8 -- 0x56032127c920 con 0x7fd5d8010070 2023-10-05T19:29:40.904 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.898+0000 7fd5e85a5700 1 -- 172.21.15.167:0/3205430407 <== osd.2 v2:172.21.15.167:6816/30037 32 ==== osd_op_reply(184 notify.0 [watch unwatch cookie 94571440422784] v32'254 uv252 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7fd5d000f560 con 0x7fd5d8010070 2023-10-05T19:29:40.904 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.898+0000 7fd5eb271e40 20 remove_watcher() i=0 2023-10-05T19:29:40.904 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.898+0000 7fd5eb271e40 2 removed watcher, disabling cache 2023-10-05T19:29:40.904 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.898+0000 7fd5eb271e40 1 -- 172.21.15.167:0/3205430407 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:185 7.0 7:05bf5b68:::notify.1:head [watch unwatch cookie 94571440429536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e32) v8 -- 0x56032127c920 con 0x5603211907f0 2023-10-05T19:29:40.906 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.902+0000 7fd5e95a7700 1 -- 172.21.15.167:0/3205430407 <== osd.1 v2:172.21.15.167:6800/30035 51 ==== osd_op_reply(185 notify.1 [watch unwatch cookie 94571440429536] v32'254 uv252 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7fd5e0050040 con 0x5603211907f0 2023-10-05T19:29:40.906 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.902+0000 7fd5eb271e40 20 remove_watcher() i=1 2023-10-05T19:29:40.906 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.902+0000 7fd5eb271e40 1 -- 172.21.15.167:0/3205430407 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:186 7.15 7:a93a5511:::notify.2:head [watch unwatch cookie 94571440425072] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e32) v8 -- 0x56032127c920 con 0x5603211c7650 2023-10-05T19:29:40.909 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.906+0000 7fd5e8da6700 1 -- 172.21.15.167:0/3205430407 <== osd.3 v2:172.21.15.167:6808/30036 67 ==== osd_op_reply(186 notify.2 [watch unwatch cookie 94571440425072] v32'254 uv252 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7fd5d8075040 con 0x5603211c7650 2023-10-05T19:29:40.909 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.906+0000 7fd5eb271e40 20 remove_watcher() i=2 2023-10-05T19:29:40.909 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.906+0000 7fd5eb271e40 1 -- 172.21.15.167:0/3205430407 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:187 7.e 7:7759931f:::notify.3:head [watch unwatch cookie 94571440437408] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e32) v8 -- 0x56032127c920 con 0x7fd5d8010070 2023-10-05T19:29:40.911 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.906+0000 7fd5e85a5700 1 -- 172.21.15.167:0/3205430407 <== osd.2 v2:172.21.15.167:6816/30037 33 ==== osd_op_reply(187 notify.3 [watch unwatch cookie 94571440437408] v32'254 uv252 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7fd5d000f560 con 0x7fd5d8010070 2023-10-05T19:29:40.911 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.906+0000 7fd5eb271e40 20 remove_watcher() i=3 2023-10-05T19:29:40.911 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.906+0000 7fd5eb271e40 1 -- 172.21.15.167:0/3205430407 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:188 7.d 7:b4812045:::notify.4:head [watch unwatch cookie 94571440438512] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e32) v8 -- 0x56032127c920 con 0x5603211907f0 2023-10-05T19:29:40.914 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.910+0000 7fd5e95a7700 1 -- 172.21.15.167:0/3205430407 <== osd.1 v2:172.21.15.167:6800/30035 52 ==== osd_op_reply(188 notify.4 [watch unwatch cookie 94571440438512] v32'254 uv252 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7fd5e0050040 con 0x5603211907f0 2023-10-05T19:29:40.914 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.910+0000 7fd5eb271e40 20 remove_watcher() i=4 2023-10-05T19:29:40.914 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.910+0000 7fd5eb271e40 1 -- 172.21.15.167:0/3205430407 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:189 7.3 7:c609908c:::notify.5:head [watch unwatch cookie 94571440430640] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e32) v8 -- 0x56032127c920 con 0x7fd5d000e8c0 2023-10-05T19:29:40.917 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.914+0000 7fd5e95a7700 1 -- 172.21.15.167:0/3205430407 <== osd.0 v2:172.21.15.118:6800/27690 51 ==== osd_op_reply(189 notify.5 [watch unwatch cookie 94571440430640] v32'254 uv252 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7fd5e0050040 con 0x7fd5d000e8c0 2023-10-05T19:29:40.917 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.914+0000 7fd5eb271e40 20 remove_watcher() i=5 2023-10-05T19:29:40.917 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.914+0000 7fd5eb271e40 1 -- 172.21.15.167:0/3205430407 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:190 7.14 7:2b04a3e9:::notify.6:head [watch unwatch cookie 94571440445360] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e32) v8 -- 0x56032127c920 con 0x7fd5d000e8c0 2023-10-05T19:29:40.920 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.914+0000 7fd5e95a7700 1 -- 172.21.15.167:0/3205430407 <== osd.0 v2:172.21.15.118:6800/27690 52 ==== osd_op_reply(190 notify.6 [watch unwatch cookie 94571440445360] v32'254 uv252 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7fd5e0050040 con 0x7fd5d000e8c0 2023-10-05T19:29:40.920 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.914+0000 7fd5eb271e40 20 remove_watcher() i=6 2023-10-05T19:29:40.920 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.914+0000 7fd5eb271e40 1 -- 172.21.15.167:0/3205430407 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:191 7.9 7:93e5b521:::notify.7:head [watch unwatch cookie 94571440442640] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e32) v8 -- 0x56032127c920 con 0x7fd5d000e8c0 2023-10-05T19:29:40.923 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.918+0000 7fd5e95a7700 1 -- 172.21.15.167:0/3205430407 <== osd.0 v2:172.21.15.118:6800/27690 53 ==== osd_op_reply(191 notify.7 [watch unwatch cookie 94571440442640] v32'254 uv252 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7fd5e0050040 con 0x7fd5d000e8c0 2023-10-05T19:29:40.923 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.918+0000 7fd5eb271e40 20 remove_watcher() i=7 2023-10-05T19:29:40.925 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.922+0000 7fd5ba7e4700 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2023-10-05T19:29:40.925 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.922+0000 7fd5eb271e40 1 -- 172.21.15.167:0/3205430407 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:192 6.4 6:22d26bf9:::data_loggenerations_metadata:head [watch unwatch cookie 94571440449040] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e32) v8 -- 0x56032127c920 con 0x5603211907f0 2023-10-05T19:29:40.928 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.922+0000 7fd5e95a7700 1 -- 172.21.15.167:0/3205430407 <== osd.1 v2:172.21.15.167:6800/30035 53 ==== osd_op_reply(192 data_loggenerations_metadata [watch unwatch cookie 94571440449040] v32'245 uv1 ondisk = 0) v8 ==== 172+0+0 (crc 0 0 0) 0x7fd5e0050040 con 0x5603211907f0 2023-10-05T19:29:40.928 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.922+0000 7fd5eb271e40 1 -- 172.21.15.167:0/3205430407 >> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] conn(0x7fd5d000e8c0 msgr2=0x7fd5d000d310 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-05T19:29:40.929 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.922+0000 7fd5eb271e40 1 --2- 172.21.15.167:0/3205430407 >> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] conn(0x7fd5d000e8c0 0x7fd5d000d310 crc :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-05T19:29:40.929 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.922+0000 7fd5eb271e40 1 -- 172.21.15.167:0/3205430407 >> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] conn(0x5603211907f0 msgr2=0x5603211c3820 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-05T19:29:40.929 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.922+0000 7fd5eb271e40 1 --2- 172.21.15.167:0/3205430407 >> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] conn(0x5603211907f0 0x5603211c3820 crc :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-05T19:29:40.929 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.922+0000 7fd5eb271e40 1 -- 172.21.15.167:0/3205430407 >> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] conn(0x7fd5d8010070 msgr2=0x7fd5d8044aa0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-05T19:29:40.929 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.922+0000 7fd5eb271e40 1 --2- 172.21.15.167:0/3205430407 >> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] conn(0x7fd5d8010070 0x7fd5d8044aa0 crc :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-05T19:29:40.929 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.922+0000 7fd5eb271e40 1 -- 172.21.15.167:0/3205430407 >> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] conn(0x5603211c7650 msgr2=0x5603211c9a30 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-05T19:29:40.929 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.922+0000 7fd5eb271e40 1 --2- 172.21.15.167:0/3205430407 >> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] conn(0x5603211c7650 0x5603211c9a30 crc :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-05T19:29:40.929 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.922+0000 7fd5eb271e40 1 -- 172.21.15.167:0/3205430407 >> [v2:172.21.15.167:6824/29889,v1:172.21.15.167:6825/29889] conn(0x7fd5c8030ca0 msgr2=0x7fd5c8033150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-05T19:29:40.929 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.922+0000 7fd5eb271e40 1 --2- 172.21.15.167:0/3205430407 >> [v2:172.21.15.167:6824/29889,v1:172.21.15.167:6825/29889] conn(0x7fd5c8030ca0 0x7fd5c8033150 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7fd5d000b300 tx=0x7fd5d0001430 comp rx=0 tx=0).stop 2023-10-05T19:29:40.929 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.922+0000 7fd5e95a7700 1 -- 172.21.15.167:0/3205430407 reap_dead start 2023-10-05T19:29:40.929 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.922+0000 7fd5eb271e40 1 -- 172.21.15.167:0/3205430407 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x560321190420 msgr2=0x5603211b7d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-05T19:29:40.930 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.922+0000 7fd5eb271e40 1 --2- 172.21.15.167:0/3205430407 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x560321190420 0x5603211b7d90 secure :-1 s=READY pgs=641 cs=0 l=1 rev1=1 crypto rx=0x7fd5d8005860 tx=0x7fd5d801c000 comp rx=0 tx=0).stop 2023-10-05T19:29:40.930 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.922+0000 7fd5eb271e40 1 -- 172.21.15.167:0/3205430407 shutdown_connections 2023-10-05T19:29:40.930 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.922+0000 7fd5eb271e40 1 --2- 172.21.15.167:0/3205430407 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x560321190420 0x5603211b7d90 unknown :-1 s=CLOSED pgs=641 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-05T19:29:40.930 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.922+0000 7fd5eb271e40 1 -- 172.21.15.167:0/3205430407 >> 172.21.15.167:0/3205430407 conn(0x56032118a730 msgr2=0x5603211ab7a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-05T19:29:40.930 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.922+0000 7fd5eb271e40 1 -- 172.21.15.167:0/3205430407 shutdown_connections 2023-10-05T19:29:40.930 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.922+0000 7fd5eb271e40 1 -- 172.21.15.167:0/3205430407 wait complete. 2023-10-05T19:29:40.931 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.926+0000 7fd5eb271e40 1 -- 172.21.15.167:0/2343252391 >> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] conn(0x5603211ec370 msgr2=0x5603211f6800 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-05T19:29:40.931 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.926+0000 7fd5eb271e40 1 --2- 172.21.15.167:0/2343252391 >> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] conn(0x5603211ec370 0x5603211f6800 crc :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-05T19:29:40.932 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.926+0000 7fd5eb271e40 1 -- 172.21.15.167:0/2343252391 >> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] conn(0x5603211f0730 msgr2=0x5603211f2b40 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-05T19:29:40.932 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.926+0000 7fd5eb271e40 1 --2- 172.21.15.167:0/2343252391 >> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] conn(0x5603211f0730 0x5603211f2b40 crc :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-05T19:29:40.932 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.926+0000 7fd5eb271e40 1 -- 172.21.15.167:0/2343252391 >> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] conn(0x5603211fb840 msgr2=0x5603211fdc20 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-05T19:29:40.932 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.926+0000 7fd5eb271e40 1 --2- 172.21.15.167:0/2343252391 >> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] conn(0x5603211fb840 0x5603211fdc20 crc :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-05T19:29:40.932 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.926+0000 7fd5eb271e40 1 -- 172.21.15.167:0/2343252391 >> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] conn(0x5603211f7a10 msgr2=0x5603211f9e20 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-05T19:29:40.932 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.926+0000 7fd5eb271e40 1 --2- 172.21.15.167:0/2343252391 >> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] conn(0x5603211f7a10 0x5603211f9e20 crc :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-05T19:29:40.932 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.926+0000 7fd5eb271e40 1 -- 172.21.15.167:0/2343252391 >> [v2:172.21.15.167:6824/29889,v1:172.21.15.167:6825/29889] conn(0x7fd530030c00 msgr2=0x7fd5300330b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-05T19:29:40.932 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.926+0000 7fd5eb271e40 1 --2- 172.21.15.167:0/2343252391 >> [v2:172.21.15.167:6824/29889,v1:172.21.15.167:6825/29889] conn(0x7fd530030c00 0x7fd5300330b0 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x5603211d1e70 tx=0x7fd5e0000f90 comp rx=0 tx=0).stop 2023-10-05T19:29:40.932 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.926+0000 7fd5eb271e40 1 -- 172.21.15.167:0/2343252391 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5603211e0000 msgr2=0x5603211e5b40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-05T19:29:40.933 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.926+0000 7fd5e8da6700 1 -- 172.21.15.167:0/2343252391 reap_dead start 2023-10-05T19:29:40.933 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.926+0000 7fd5eb271e40 1 --2- 172.21.15.167:0/2343252391 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5603211e0000 0x5603211e5b40 secure :-1 s=READY pgs=643 cs=0 l=1 rev1=1 crypto rx=0x7fd5d0000fb0 tx=0x7fd5d0032000 comp rx=0 tx=0).stop 2023-10-05T19:29:40.933 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.926+0000 7fd5eb271e40 1 -- 172.21.15.167:0/2343252391 shutdown_connections 2023-10-05T19:29:40.933 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.926+0000 7fd5eb271e40 1 --2- 172.21.15.167:0/2343252391 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5603211e0000 0x5603211e5b40 unknown :-1 s=CLOSED pgs=643 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-05T19:29:40.933 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.926+0000 7fd5eb271e40 1 -- 172.21.15.167:0/2343252391 >> 172.21.15.167:0/2343252391 conn(0x5603211db3c0 msgr2=0x5603211d9460 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-05T19:29:40.933 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.926+0000 7fd5eb271e40 1 -- 172.21.15.167:0/2343252391 shutdown_connections 2023-10-05T19:29:40.933 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.926+0000 7fd5eb271e40 1 -- 172.21.15.167:0/2343252391 wait complete. 2023-10-05T19:29:40.935 INFO:tasks.util.rgw:rgwadmin: client.0 : ['user', 'rm', '--uid', 'fud', '--purge-data'] 2023-10-05T19:29:40.935 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'ceph', 'user', 'rm', '--uid', 'fud', '--purge-data'] 2023-10-05T19:29:40.935 DEBUG:teuthology.orchestra.run.smithi167:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster ceph user rm --uid fud --purge-data 2023-10-05T19:29:40.975 INFO:teuthology.orchestra.run.smithi167.stderr:ignoring --setuser ceph since I am not root 2023-10-05T19:29:40.975 INFO:teuthology.orchestra.run.smithi167.stderr:ignoring --setgroup ceph since I am not root 2023-10-05T19:29:40.976 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.970+0000 7f371296be40 1 Processor -- start 2023-10-05T19:29:40.976 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.970+0000 7f371296be40 1 -- start start 2023-10-05T19:29:40.977 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.970+0000 7f371296be40 1 --2- >> v2:172.21.15.118:3300/0 conn(0x55e8134147f0 0x55e8134136e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-05T19:29:40.977 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.970+0000 7f371296be40 1 -- --> v1:172.21.15.118:6789/0 -- auth(proto 0 26 bytes epoch 0) v1 -- 0x55e813360bd0 con 0x55e813414420 2023-10-05T19:29:40.977 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.970+0000 7f371296be40 1 -- --> v2:172.21.15.118:3300/0 -- mon_getmap magic: 0 v1 -- 0x55e81331dee0 con 0x55e8134147f0 2023-10-05T19:29:40.977 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.974+0000 7f370fc9f700 1 --2- >> v2:172.21.15.118:3300/0 conn(0x55e8134147f0 0x55e8134136e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-05T19:29:40.977 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.974+0000 7f37104a0700 1 --1- >> v1:172.21.15.118:6789/0 conn(0x55e813414420 0x55e813412fd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.118:6789/0 says I am v1:172.21.15.167:33352/0 (socket says 172.21.15.167:33352) 2023-10-05T19:29:40.977 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.974+0000 7f37104a0700 1 -- 172.21.15.167:0/2898639619 learned_addr learned my addr 172.21.15.167:0/2898639619 (peer_addr_for_me v1:172.21.15.167:0/0) 2023-10-05T19:29:40.978 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.974+0000 7f370fc9f700 1 -- 172.21.15.167:0/2898639619 >> v1:172.21.15.118:6789/0 conn(0x55e813414420 legacy=0x55e813412fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-05T19:29:40.978 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.974+0000 7f370fc9f700 1 -- 172.21.15.167:0/2898639619 --> v2:172.21.15.118:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55e813413ce0 con 0x55e8134147f0 2023-10-05T19:29:40.978 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.974+0000 7f370fc9f700 1 --2- 172.21.15.167:0/2898639619 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55e8134147f0 0x55e8134136e0 secure :-1 s=READY pgs=645 cs=0 l=1 rev1=1 crypto rx=0x7f36f800a5d0 tx=0x7f36f8005a50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=586ffabd5daed497 server_cookie=0 in_seq=0 out_seq=0 2023-10-05T19:29:40.978 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.974+0000 7f370f49e700 1 -- 172.21.15.167:0/2898639619 <== mon.0 v2:172.21.15.118:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f36f80086a0 con 0x55e8134147f0 2023-10-05T19:29:40.978 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.974+0000 7f370f49e700 1 -- 172.21.15.167:0/2898639619 <== mon.0 v2:172.21.15.118:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f36f8008800 con 0x55e8134147f0 2023-10-05T19:29:40.979 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.974+0000 7f370f49e700 1 -- 172.21.15.167:0/2898639619 <== mon.0 v2:172.21.15.118:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f36f8008ad0 con 0x55e8134147f0 2023-10-05T19:29:40.979 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.974+0000 7f371296be40 1 -- 172.21.15.167:0/2898639619 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55e8134147f0 msgr2=0x55e8134136e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-05T19:29:40.979 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.974+0000 7f371296be40 1 --2- 172.21.15.167:0/2898639619 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55e8134147f0 0x55e8134136e0 secure :-1 s=READY pgs=645 cs=0 l=1 rev1=1 crypto rx=0x7f36f800a5d0 tx=0x7f36f8005a50 comp rx=0 tx=0).stop 2023-10-05T19:29:40.980 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.974+0000 7f371296be40 1 -- 172.21.15.167:0/2898639619 shutdown_connections 2023-10-05T19:29:40.980 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.974+0000 7f371296be40 1 --2- 172.21.15.167:0/2898639619 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55e8134147f0 0x55e8134136e0 unknown :-1 s=CLOSED pgs=645 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-05T19:29:40.980 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.974+0000 7f371296be40 1 -- 172.21.15.167:0/2898639619 >> 172.21.15.167:0/2898639619 conn(0x55e81340e730 msgr2=0x55e813410b40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-05T19:29:40.980 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.974+0000 7f371296be40 1 -- 172.21.15.167:0/2898639619 shutdown_connections 2023-10-05T19:29:40.980 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.974+0000 7f371296be40 1 -- 172.21.15.167:0/2898639619 wait complete. 2023-10-05T19:29:40.981 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.974+0000 7f371296be40 1 Processor -- start 2023-10-05T19:29:40.981 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.974+0000 7f371296be40 1 -- start start 2023-10-05T19:29:40.981 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.974+0000 7f371296be40 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55e813414420 0x55e813435170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-05T19:29:40.981 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.974+0000 7f371296be40 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e8132e1e30 con 0x55e813414420 2023-10-05T19:29:40.981 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.978+0000 7f37104a0700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55e813414420 0x55e813435170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-05T19:29:40.981 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.978+0000 7f37104a0700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55e813414420 0x55e813435170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.118:3300/0 says I am v2:172.21.15.167:56906/0 (socket says 172.21.15.167:56906) 2023-10-05T19:29:40.981 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.978+0000 7f37104a0700 1 -- 172.21.15.167:0/1472594258 learned_addr learned my addr 172.21.15.167:0/1472594258 (peer_addr_for_me v2:172.21.15.167:0/0) 2023-10-05T19:29:40.981 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.978+0000 7f37104a0700 1 -- 172.21.15.167:0/1472594258 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f36f8005cf0 con 0x55e813414420 2023-10-05T19:29:40.982 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.978+0000 7f37104a0700 1 --2- 172.21.15.167:0/1472594258 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55e813414420 0x55e813435170 secure :-1 s=READY pgs=646 cs=0 l=1 rev1=1 crypto rx=0x7f3700001c30 tx=0x7f37000017e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c2c40c8c1f1a9cbd server_cookie=0 in_seq=0 out_seq=0 2023-10-05T19:29:40.983 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.978+0000 7f370e49c700 1 -- 172.21.15.167:0/1472594258 <== mon.0 v2:172.21.15.118:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f370000bbf0 con 0x55e813414420 2023-10-05T19:29:40.983 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.978+0000 7f370e49c700 1 -- 172.21.15.167:0/1472594258 <== mon.0 v2:172.21.15.118:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f370000bd50 con 0x55e813414420 2023-10-05T19:29:40.984 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.978+0000 7f370e49c700 1 -- 172.21.15.167:0/1472594258 <== mon.0 v2:172.21.15.118:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f370000e470 con 0x55e813414420 2023-10-05T19:29:40.984 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.978+0000 7f371296be40 1 -- 172.21.15.167:0/1472594258 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55e813414420 msgr2=0x55e813435170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-05T19:29:40.984 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.978+0000 7f371296be40 1 --2- 172.21.15.167:0/1472594258 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55e813414420 0x55e813435170 secure :-1 s=READY pgs=646 cs=0 l=1 rev1=1 crypto rx=0x7f3700001c30 tx=0x7f37000017e0 comp rx=0 tx=0).stop 2023-10-05T19:29:40.984 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.978+0000 7f371296be40 1 -- 172.21.15.167:0/1472594258 shutdown_connections 2023-10-05T19:29:40.984 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.978+0000 7f371296be40 1 --2- 172.21.15.167:0/1472594258 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55e813414420 0x55e813435170 unknown :-1 s=CLOSED pgs=646 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-05T19:29:40.984 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.978+0000 7f371296be40 1 -- 172.21.15.167:0/1472594258 >> 172.21.15.167:0/1472594258 conn(0x55e81340e730 msgr2=0x55e813430d30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-05T19:29:40.984 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.978+0000 7f371296be40 1 -- 172.21.15.167:0/1472594258 shutdown_connections 2023-10-05T19:29:40.984 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.978+0000 7f371296be40 1 -- 172.21.15.167:0/1472594258 wait complete. 2023-10-05T19:29:40.985 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.978+0000 7f371296be40 1 Processor -- start 2023-10-05T19:29:40.985 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.982+0000 7f371296be40 1 -- start start 2023-10-05T19:29:40.985 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.982+0000 7f371296be40 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55e813414420 0x55e81343bf40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-05T19:29:40.985 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.982+0000 7f371296be40 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e8132b4ed0 con 0x55e813414420 2023-10-05T19:29:40.985 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.982+0000 7f37104a0700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55e813414420 0x55e81343bf40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-05T19:29:40.986 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.982+0000 7f37104a0700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55e813414420 0x55e81343bf40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.118:3300/0 says I am v2:172.21.15.167:56912/0 (socket says 172.21.15.167:56912) 2023-10-05T19:29:40.986 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.982+0000 7f37104a0700 1 -- 172.21.15.167:0/2673647112 learned_addr learned my addr 172.21.15.167:0/2673647112 (peer_addr_for_me v2:172.21.15.167:0/0) 2023-10-05T19:29:40.986 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.982+0000 7f37104a0700 1 -- 172.21.15.167:0/2673647112 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55e813360bd0 con 0x55e813414420 2023-10-05T19:29:40.986 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.982+0000 7f37104a0700 1 --2- 172.21.15.167:0/2673647112 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55e813414420 0x55e81343bf40 secure :-1 s=READY pgs=647 cs=0 l=1 rev1=1 crypto rx=0x7f3700000c00 tx=0x7f3700016000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-05T19:29:40.987 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.982+0000 7f370cc99700 1 -- 172.21.15.167:0/2673647112 <== mon.0 v2:172.21.15.118:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f370001f070 con 0x55e813414420 2023-10-05T19:29:40.987 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.982+0000 7f370cc99700 1 -- 172.21.15.167:0/2673647112 <== mon.0 v2:172.21.15.118:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3700019690 con 0x55e813414420 2023-10-05T19:29:40.987 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.982+0000 7f370cc99700 1 -- 172.21.15.167:0/2673647112 <== mon.0 v2:172.21.15.118:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f3700019840 con 0x55e813414420 2023-10-05T19:29:40.987 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.982+0000 7f371296be40 1 -- 172.21.15.167:0/2673647112 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x55e81343c480 con 0x55e813414420 2023-10-05T19:29:40.987 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.982+0000 7f371296be40 1 -- 172.21.15.167:0/2673647112 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55e81343c8c0 con 0x55e813414420 2023-10-05T19:29:40.989 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.986+0000 7f370cc99700 1 -- 172.21.15.167:0/2673647112 <== mon.0 v2:172.21.15.118:3300/0 4 ==== mgrmap(e 5) v1 ==== 40329+0+0 (secure 0 0 0) 0x7f3700022020 con 0x55e813414420 2023-10-05T19:29:40.989 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.986+0000 7f371296be40 20 rados->read ofs=0 len=0 2023-10-05T19:29:40.990 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.986+0000 7f370cc99700 1 --2- 172.21.15.167:0/2673647112 >> [v2:172.21.15.167:6824/29889,v1:172.21.15.167:6825/29889] conn(0x7f36f0030c40 0x7f36f00330f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-05T19:29:40.990 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.986+0000 7f370cc99700 1 -- 172.21.15.167:0/2673647112 <== mon.0 v2:172.21.15.118:3300/0 5 ==== osd_map(32..32 src has 1..32) v4 ==== 6260+0+0 (secure 0 0 0) 0x7f3700044f00 con 0x55e813414420 2023-10-05T19:29:40.990 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.986+0000 7f370fc9f700 1 --2- 172.21.15.167:0/2673647112 >> [v2:172.21.15.167:6824/29889,v1:172.21.15.167:6825/29889] conn(0x7f36f0030c40 0x7f36f00330f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-05T19:29:40.990 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.986+0000 7f371296be40 1 --2- 172.21.15.167:0/2673647112 >> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] conn(0x55e8134147f0 0x55e8134479a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-05T19:29:40.991 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.986+0000 7f371296be40 1 -- 172.21.15.167:0/2673647112 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:1 5.12 5:49953fa1:::default.realm:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32) v8 -- 0x55e8134480a0 con 0x55e8134147f0 2023-10-05T19:29:40.991 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.986+0000 7f3710ca1700 1 --2- 172.21.15.167:0/2673647112 >> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] conn(0x55e8134147f0 0x55e8134479a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-05T19:29:40.991 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.986+0000 7f370fc9f700 1 --2- 172.21.15.167:0/2673647112 >> [v2:172.21.15.167:6824/29889,v1:172.21.15.167:6825/29889] conn(0x7f36f0030c40 0x7f36f00330f0 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7f36f8005a90 tx=0x7f36f8015000 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-05T19:29:40.992 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.986+0000 7f3710ca1700 1 --2- 172.21.15.167:0/2673647112 >> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] conn(0x55e8134147f0 0x55e8134479a0 crc :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-05T19:29:40.992 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.986+0000 7f3710ca1700 1 -- 172.21.15.167:0/2673647112 <== osd.1 v2:172.21.15.167:6800/30035 1 ==== osd_op_reply(1 default.realm [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 ==== 157+0+0 (crc 0 0 0) 0x7f370800a5a0 con 0x55e8134147f0 2023-10-05T19:29:40.992 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.986+0000 7f371296be40 20 rados_obj.operate() r=-2 bl.length=0 2023-10-05T19:29:40.992 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.986+0000 7f371296be40 20 realm 2023-10-05T19:29:40.992 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.986+0000 7f371296be40 20 rados->read ofs=0 len=0 2023-10-05T19:29:40.992 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.986+0000 7f371296be40 1 -- 172.21.15.167:0/2673647112 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:2 5.12 5:49953fa1:::default.realm:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32) v8 -- 0x55e813445000 con 0x55e8134147f0 2023-10-05T19:29:40.994 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.986+0000 7f3710ca1700 1 -- 172.21.15.167:0/2673647112 <== osd.1 v2:172.21.15.167:6800/30035 2 ==== osd_op_reply(2 default.realm [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 ==== 157+0+0 (crc 0 0 0) 0x7f370800a5a0 con 0x55e8134147f0 2023-10-05T19:29:40.994 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.986+0000 7f371296be40 20 rados_obj.operate() r=-2 bl.length=0 2023-10-05T19:29:40.994 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.986+0000 7f371296be40 4 RGWPeriod::init failed to init realm id : (2) No such file or directory 2023-10-05T19:29:40.994 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.986+0000 7f371296be40 20 rados->read ofs=0 len=0 2023-10-05T19:29:40.994 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.990+0000 7f371296be40 1 -- 172.21.15.167:0/2673647112 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:3 5.12 5:49953fa1:::default.realm:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32) v8 -- 0x55e813448b80 con 0x55e8134147f0 2023-10-05T19:29:40.994 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.990+0000 7f3710ca1700 1 -- 172.21.15.167:0/2673647112 <== osd.1 v2:172.21.15.167:6800/30035 3 ==== osd_op_reply(3 default.realm [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 ==== 157+0+0 (crc 0 0 0) 0x7f370800a5a0 con 0x55e8134147f0 2023-10-05T19:29:40.994 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.990+0000 7f371296be40 20 rados_obj.operate() r=-2 bl.length=0 2023-10-05T19:29:40.994 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.990+0000 7f371296be40 20 rados->read ofs=0 len=0 2023-10-05T19:29:40.994 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.990+0000 7f371296be40 1 -- 172.21.15.167:0/2673647112 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:4 5.f 5:f43b6ece:::zone_names.default:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32) v8 -- 0x55e813448790 con 0x55e8134147f0 2023-10-05T19:29:40.994 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.990+0000 7f3710ca1700 1 -- 172.21.15.167:0/2673647112 <== osd.1 v2:172.21.15.167:6800/30035 4 ==== osd_op_reply(4 zone_names.default [read 0~46 out=46b] v0'0 uv1 ondisk = 0) v8 ==== 162+0+46 (crc 0 0 0) 0x7f370800a5a0 con 0x55e8134147f0 2023-10-05T19:29:40.995 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.990+0000 7f371296be40 20 rados_obj.operate() r=0 bl.length=46 2023-10-05T19:29:40.995 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.990+0000 7f371296be40 20 rados->read ofs=0 len=0 2023-10-05T19:29:40.995 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.990+0000 7f371296be40 1 -- 172.21.15.167:0/2673647112 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:5 5.1d 5:b8374eb4:::zone_info.9a9df965-6612-45d4-a45f-6e3c889bacc2:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32) v8 -- 0x55e813449a10 con 0x55e8134147f0 2023-10-05T19:29:40.996 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.990+0000 7f3710ca1700 1 -- 172.21.15.167:0/2673647112 <== osd.1 v2:172.21.15.167:6800/30035 5 ==== osd_op_reply(5 zone_info.9a9df965-6612-45d4-a45f-6e3c889bacc2 [read 0~874 out=874b] v0'0 uv2 ondisk = 0) v8 ==== 190+0+874 (crc 0 0 0) 0x7f370800a5a0 con 0x55e8134147f0 2023-10-05T19:29:40.996 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.990+0000 7f371296be40 20 rados_obj.operate() r=0 bl.length=874 2023-10-05T19:29:40.996 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.990+0000 7f371296be40 20 searching for the correct realm 2023-10-05T19:29:40.996 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.990+0000 7f371296be40 1 --2- 172.21.15.167:0/2673647112 >> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] conn(0x55e81344b7d0 0x55e81344dbb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-05T19:29:40.996 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.990+0000 7f371296be40 1 -- 172.21.15.167:0/2673647112 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:6 5.0 5:00000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x55e81344e0f0 con 0x55e81344b7d0 2023-10-05T19:29:40.996 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.990+0000 7f37104a0700 1 --2- 172.21.15.167:0/2673647112 >> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] conn(0x55e81344b7d0 0x55e81344dbb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-05T19:29:40.996 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.990+0000 7f37104a0700 1 --2- 172.21.15.167:0/2673647112 >> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] conn(0x55e81344b7d0 0x55e81344dbb0 crc :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-05T19:29:40.997 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.990+0000 7f37104a0700 1 -- 172.21.15.167:0/2673647112 <== osd.3 v2:172.21.15.167:6808/30036 1 ==== osd_op_reply(6 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f370000d7f0 con 0x55e81344b7d0 2023-10-05T19:29:40.997 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.990+0000 7f37104a0700 1 --2- 172.21.15.167:0/2673647112 >> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] conn(0x7f3700044040 0x7f37000420e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-05T19:29:40.997 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.994+0000 7f37104a0700 1 -- 172.21.15.167:0/2673647112 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:7 5.10 5:08000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x55e81344e0f0 con 0x7f3700044040 2023-10-05T19:29:40.997 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.994+0000 7f370fc9f700 1 --2- 172.21.15.167:0/2673647112 >> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] conn(0x7f3700044040 0x7f37000420e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-05T19:29:40.997 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.994+0000 7f370fc9f700 1 --2- 172.21.15.167:0/2673647112 >> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] conn(0x7f3700044040 0x7f37000420e0 crc :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-05T19:29:40.998 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.994+0000 7f370fc9f700 1 -- 172.21.15.167:0/2673647112 <== osd.2 v2:172.21.15.167:6816/30037 1 ==== osd_op_reply(7 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f36f8008ae0 con 0x7f3700044040 2023-10-05T19:29:40.998 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.994+0000 7f370fc9f700 1 --2- 172.21.15.167:0/2673647112 >> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] conn(0x7f36f8007030 0x7f36f801c920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-05T19:29:40.998 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.994+0000 7f370fc9f700 1 -- 172.21.15.167:0/2673647112 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:8 5.8 5:10000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x55e81344e0f0 con 0x7f36f8007030 2023-10-05T19:29:40.998 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.994+0000 7f3710ca1700 1 --2- 172.21.15.167:0/2673647112 >> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] conn(0x7f36f8007030 0x7f36f801c920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-05T19:29:40.999 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.994+0000 7f3710ca1700 1 --2- 172.21.15.167:0/2673647112 >> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] conn(0x7f36f8007030 0x7f36f801c920 crc :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-05T19:29:41.000 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.994+0000 7f3710ca1700 1 -- 172.21.15.167:0/2673647112 <== osd.0 v2:172.21.15.118:6800/27690 1 ==== osd_op_reply(8 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f370800a5a0 con 0x7f36f8007030 2023-10-05T19:29:41.000 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.994+0000 7f3710ca1700 1 -- 172.21.15.167:0/2673647112 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:9 5.18 5:18000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x55e81344e0f0 con 0x55e8134147f0 2023-10-05T19:29:41.001 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.998+0000 7f3710ca1700 1 -- 172.21.15.167:0/2673647112 <== osd.1 v2:172.21.15.167:6800/30035 6 ==== osd_op_reply(9 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f370800a5a0 con 0x55e8134147f0 2023-10-05T19:29:41.001 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.998+0000 7f3710ca1700 1 -- 172.21.15.167:0/2673647112 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:10 5.4 5:20000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x55e81344e0f0 con 0x55e8134147f0 2023-10-05T19:29:41.001 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.998+0000 7f3710ca1700 1 -- 172.21.15.167:0/2673647112 <== osd.1 v2:172.21.15.167:6800/30035 7 ==== osd_op_reply(10 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f370800a5a0 con 0x55e8134147f0 2023-10-05T19:29:41.002 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.998+0000 7f3710ca1700 1 -- 172.21.15.167:0/2673647112 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:11 5.14 5:28000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x55e81344e0f0 con 0x7f36f8007030 2023-10-05T19:29:41.002 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.998+0000 7f3710ca1700 1 -- 172.21.15.167:0/2673647112 <== osd.0 v2:172.21.15.118:6800/27690 2 ==== osd_op_reply(11 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f370800a5a0 con 0x7f36f8007030 2023-10-05T19:29:41.002 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.998+0000 7f3710ca1700 1 -- 172.21.15.167:0/2673647112 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:12 5.c 5:30000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x55e81344e0f0 con 0x7f36f8007030 2023-10-05T19:29:41.003 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.998+0000 7f3710ca1700 1 -- 172.21.15.167:0/2673647112 <== osd.0 v2:172.21.15.118:6800/27690 3 ==== osd_op_reply(12 [pgnls start_epoch 32 out=112b] v23'1 uv1 ondisk = 1) v8 ==== 144+0+112 (crc 0 0 0) 0x7f370800a5a0 con 0x7f36f8007030 2023-10-05T19:29:41.003 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.998+0000 7f3710ca1700 1 -- 172.21.15.167:0/2673647112 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:13 5.1c 5:38000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x55e81344e0f0 con 0x7f3700044040 2023-10-05T19:29:41.004 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.998+0000 7f370fc9f700 1 -- 172.21.15.167:0/2673647112 <== osd.2 v2:172.21.15.167:6816/30037 2 ==== osd_op_reply(13 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f36f8008ae0 con 0x7f3700044040 2023-10-05T19:29:41.004 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:40.998+0000 7f370fc9f700 1 -- 172.21.15.167:0/2673647112 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:14 5.2 5:40000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x55e81344e0f0 con 0x55e81344b7d0 2023-10-05T19:29:41.005 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:41.002+0000 7f37104a0700 1 -- 172.21.15.167:0/2673647112 <== osd.3 v2:172.21.15.167:6808/30036 2 ==== osd_op_reply(14 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f370000d7f0 con 0x55e81344b7d0 2023-10-05T19:29:41.005 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:41.002+0000 7f37104a0700 1 -- 172.21.15.167:0/2673647112 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:15 5.12 5:48000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x55e81344e0f0 con 0x55e8134147f0 2023-10-05T19:29:41.005 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:41.002+0000 7f3710ca1700 1 -- 172.21.15.167:0/2673647112 <== osd.1 v2:172.21.15.167:6800/30035 8 ==== osd_op_reply(15 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f370800a5a0 con 0x55e8134147f0 2023-10-05T19:29:41.005 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:41.002+0000 7f3710ca1700 1 -- 172.21.15.167:0/2673647112 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:16 5.a 5:50000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x55e81344e0f0 con 0x7f36f8007030 2023-10-05T19:29:41.006 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:41.002+0000 7f3710ca1700 1 -- 172.21.15.167:0/2673647112 <== osd.0 v2:172.21.15.118:6800/27690 4 ==== osd_op_reply(16 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f370800a5a0 con 0x7f36f8007030 2023-10-05T19:29:41.006 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:41.002+0000 7f3710ca1700 1 -- 172.21.15.167:0/2673647112 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:17 5.1a 5:58000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x55e81344e0f0 con 0x55e8134147f0 2023-10-05T19:29:41.007 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:41.002+0000 7f3710ca1700 1 -- 172.21.15.167:0/2673647112 <== osd.1 v2:172.21.15.167:6800/30035 9 ==== osd_op_reply(17 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f370800a5a0 con 0x55e8134147f0 2023-10-05T19:29:41.007 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:41.002+0000 7f3710ca1700 1 -- 172.21.15.167:0/2673647112 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:18 5.6 5:60000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x55e81344e0f0 con 0x7f3700044040 2023-10-05T19:29:41.008 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:41.002+0000 7f370fc9f700 1 -- 172.21.15.167:0/2673647112 <== osd.2 v2:172.21.15.167:6816/30037 3 ==== osd_op_reply(18 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f36f8008ae0 con 0x7f3700044040 2023-10-05T19:29:41.008 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:41.002+0000 7f370fc9f700 1 -- 172.21.15.167:0/2673647112 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:19 5.16 5:68000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x55e81344e0f0 con 0x55e81344b7d0 2023-10-05T19:29:41.009 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:41.006+0000 7f37104a0700 1 -- 172.21.15.167:0/2673647112 <== osd.3 v2:172.21.15.167:6808/30036 3 ==== osd_op_reply(19 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f370000d7f0 con 0x55e81344b7d0 2023-10-05T19:29:41.009 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:41.006+0000 7f37104a0700 1 -- 172.21.15.167:0/2673647112 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:20 5.e 5:70000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x55e81344e0f0 con 0x55e81344b7d0 2023-10-05T19:29:41.009 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:41.006+0000 7f37104a0700 1 -- 172.21.15.167:0/2673647112 <== osd.3 v2:172.21.15.167:6808/30036 4 ==== osd_op_reply(20 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f370000d7f0 con 0x55e81344b7d0 2023-10-05T19:29:41.010 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:41.006+0000 7f37104a0700 1 -- 172.21.15.167:0/2673647112 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:21 5.1e 5:78000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x55e81344e0f0 con 0x7f36f8007030 2023-10-05T19:29:41.011 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:41.006+0000 7f3710ca1700 1 -- 172.21.15.167:0/2673647112 <== osd.0 v2:172.21.15.118:6800/27690 5 ==== osd_op_reply(21 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f370800a5a0 con 0x7f36f8007030 2023-10-05T19:29:41.011 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:41.006+0000 7f3710ca1700 1 -- 172.21.15.167:0/2673647112 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:22 5.1 5:80000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x55e81344e0f0 con 0x55e81344b7d0 2023-10-05T19:29:41.011 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:41.006+0000 7f37104a0700 1 -- 172.21.15.167:0/2673647112 <== osd.3 v2:172.21.15.167:6808/30036 5 ==== osd_op_reply(22 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f370000d7f0 con 0x55e81344b7d0 2023-10-05T19:29:41.011 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:41.006+0000 7f37104a0700 1 -- 172.21.15.167:0/2673647112 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:23 5.11 5:88000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x55e81344e0f0 con 0x55e81344b7d0 2023-10-05T19:29:41.012 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:41.006+0000 7f37104a0700 1 -- 172.21.15.167:0/2673647112 <== osd.3 v2:172.21.15.167:6808/30036 6 ==== osd_op_reply(23 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f370000d7f0 con 0x55e81344b7d0 2023-10-05T19:29:41.012 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:41.006+0000 7f37104a0700 1 -- 172.21.15.167:0/2673647112 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:24 5.9 5:90000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x55e81344e0f0 con 0x55e8134147f0 2023-10-05T19:29:41.013 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:41.010+0000 7f3710ca1700 1 -- 172.21.15.167:0/2673647112 <== osd.1 v2:172.21.15.167:6800/30035 10 ==== osd_op_reply(24 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f370800a5a0 con 0x55e8134147f0 2023-10-05T19:29:41.013 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:41.010+0000 7f3710ca1700 1 -- 172.21.15.167:0/2673647112 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:25 5.19 5:98000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x55e81344e0f0 con 0x55e8134147f0 2023-10-05T19:29:41.013 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:41.010+0000 7f3710ca1700 1 -- 172.21.15.167:0/2673647112 <== osd.1 v2:172.21.15.167:6800/30035 11 ==== osd_op_reply(25 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f370800a5a0 con 0x55e8134147f0 2023-10-05T19:29:41.014 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:41.010+0000 7f3710ca1700 1 -- 172.21.15.167:0/2673647112 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:26 5.5 5:a0000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x55e81344e0f0 con 0x7f36f8007030 2023-10-05T19:29:41.015 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:41.010+0000 7f3710ca1700 1 -- 172.21.15.167:0/2673647112 <== osd.0 v2:172.21.15.118:6800/27690 6 ==== osd_op_reply(26 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f370800a5a0 con 0x7f36f8007030 2023-10-05T19:29:41.015 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:41.010+0000 7f3710ca1700 1 -- 172.21.15.167:0/2673647112 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:27 5.15 5:a8000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x55e81344e0f0 con 0x55e81344b7d0 2023-10-05T19:29:41.015 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:41.010+0000 7f37104a0700 1 -- 172.21.15.167:0/2673647112 <== osd.3 v2:172.21.15.167:6808/30036 7 ==== osd_op_reply(27 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f370000d7f0 con 0x55e81344b7d0 2023-10-05T19:29:41.015 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:41.010+0000 7f37104a0700 1 -- 172.21.15.167:0/2673647112 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:28 5.d 5:b0000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x55e81344e0f0 con 0x7f3700044040 2023-10-05T19:29:41.016 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:41.010+0000 7f370fc9f700 1 -- 172.21.15.167:0/2673647112 <== osd.2 v2:172.21.15.167:6816/30037 4 ==== osd_op_reply(28 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f36f8008ae0 con 0x7f3700044040 2023-10-05T19:29:41.016 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:41.010+0000 7f370fc9f700 1 -- 172.21.15.167:0/2673647112 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:29 5.1d 5:b8000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x55e81344e0f0 con 0x55e8134147f0 2023-10-05T19:29:41.017 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:41.014+0000 7f3710ca1700 1 -- 172.21.15.167:0/2673647112 <== osd.1 v2:172.21.15.167:6800/30035 12 ==== osd_op_reply(29 [pgnls start_epoch 32 out=107b] v30'2 uv2 ondisk = 1) v8 ==== 144+0+107 (crc 0 0 0) 0x7f370800a5a0 con 0x55e8134147f0 2023-10-05T19:29:41.017 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:41.014+0000 7f3710ca1700 1 -- 172.21.15.167:0/2673647112 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:30 5.3 5:c0000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x55e81344e0f0 con 0x7f36f8007030 2023-10-05T19:29:41.018 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:41.014+0000 7f3710ca1700 1 -- 172.21.15.167:0/2673647112 <== osd.0 v2:172.21.15.118:6800/27690 7 ==== osd_op_reply(30 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f370800a5a0 con 0x7f36f8007030 2023-10-05T19:29:41.018 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:41.014+0000 7f3710ca1700 1 -- 172.21.15.167:0/2673647112 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:31 5.13 5:c8000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x55e81344e0f0 con 0x55e81344b7d0 2023-10-05T19:29:41.019 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:41.014+0000 7f37104a0700 1 -- 172.21.15.167:0/2673647112 <== osd.3 v2:172.21.15.167:6808/30036 8 ==== osd_op_reply(31 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f370000d7f0 con 0x55e81344b7d0 2023-10-05T19:29:41.019 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:41.014+0000 7f37104a0700 1 -- 172.21.15.167:0/2673647112 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:32 5.b 5:d0000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x55e81344e0f0 con 0x7f36f8007030 2023-10-05T19:29:41.020 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:41.014+0000 7f3710ca1700 1 -- 172.21.15.167:0/2673647112 <== osd.0 v2:172.21.15.118:6800/27690 8 ==== osd_op_reply(32 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f370800a5a0 con 0x7f36f8007030 2023-10-05T19:29:41.020 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:41.014+0000 7f3710ca1700 1 -- 172.21.15.167:0/2673647112 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:33 5.1b 5:d8000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x55e81344e0f0 con 0x7f3700044040 2023-10-05T19:29:41.021 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:41.018+0000 7f370fc9f700 1 -- 172.21.15.167:0/2673647112 <== osd.2 v2:172.21.15.167:6816/30037 5 ==== osd_op_reply(33 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f36f8008ae0 con 0x7f3700044040 2023-10-05T19:29:41.021 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:41.018+0000 7f370fc9f700 1 -- 172.21.15.167:0/2673647112 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:34 5.7 5:e0000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x55e81344e0f0 con 0x55e81344b7d0 2023-10-05T19:29:41.021 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:41.018+0000 7f37104a0700 1 -- 172.21.15.167:0/2673647112 <== osd.3 v2:172.21.15.167:6808/30036 9 ==== osd_op_reply(34 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f370000d7f0 con 0x55e81344b7d0 2023-10-05T19:29:41.021 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:41.018+0000 7f37104a0700 1 -- 172.21.15.167:0/2673647112 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:35 5.17 5:e8000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x55e81344e0f0 con 0x7f36f8007030 2023-10-05T19:29:41.022 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:41.018+0000 7f3710ca1700 1 -- 172.21.15.167:0/2673647112 <== osd.0 v2:172.21.15.118:6800/27690 9 ==== osd_op_reply(35 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f370800a5a0 con 0x7f36f8007030 2023-10-05T19:29:41.023 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:41.018+0000 7f3710ca1700 1 -- 172.21.15.167:0/2673647112 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:36 5.f 5:f0000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x55e81344e0f0 con 0x55e8134147f0 2023-10-05T19:29:41.023 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:41.018+0000 7f3710ca1700 1 -- 172.21.15.167:0/2673647112 <== osd.1 v2:172.21.15.167:6800/30035 13 ==== osd_op_reply(36 [pgnls start_epoch 32 out=115b] v23'2 uv2 ondisk = 1) v8 ==== 144+0+115 (crc 0 0 0) 0x7f370800a5a0 con 0x55e8134147f0 2023-10-05T19:29:41.023 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:41.018+0000 7f3710ca1700 1 -- 172.21.15.167:0/2673647112 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:37 5.1f 5:f8000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x55e81344e0f0 con 0x7f3700044040 2023-10-05T19:29:41.024 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:41.018+0000 7f370fc9f700 1 -- 172.21.15.167:0/2673647112 <== osd.2 v2:172.21.15.167:6816/30037 6 ==== osd_op_reply(37 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f36f8008ae0 con 0x7f3700044040 2023-10-05T19:29:41.024 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:41.018+0000 7f371296be40 20 RGWRados::pool_iterate: got zonegroup_info.4f1f719d-9648-417b-b10c-e5d939825419 2023-10-05T19:29:41.025 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:41.022+0000 7f371296be40 20 RGWRados::pool_iterate: got zone_info.9a9df965-6612-45d4-a45f-6e3c889bacc2 2023-10-05T19:29:41.025 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:41.022+0000 7f371296be40 20 RGWRados::pool_iterate: got zone_names.default 2023-10-05T19:29:41.025 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:41.022+0000 7f371296be40 20 RGWRados::pool_iterate: got zonegroups_names.default 2023-10-05T19:29:41.025 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:41.022+0000 7f371296be40 20 rados->read ofs=0 len=0 2023-10-05T19:29:41.025 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:41.022+0000 7f371296be40 1 -- 172.21.15.167:0/2673647112 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:38 5.12 5:49953fa1:::default.realm:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32) v8 -- 0x55e81344a520 con 0x55e8134147f0 2023-10-05T19:29:41.026 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:41.022+0000 7f3710ca1700 1 -- 172.21.15.167:0/2673647112 <== osd.1 v2:172.21.15.167:6800/30035 14 ==== osd_op_reply(38 default.realm [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 ==== 157+0+0 (crc 0 0 0) 0x7f370800a5a0 con 0x55e8134147f0 2023-10-05T19:29:41.026 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:41.022+0000 7f371296be40 20 rados_obj.operate() r=-2 bl.length=0 2023-10-05T19:29:41.026 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:41.022+0000 7f371296be40 20 rados->read ofs=0 len=0 2023-10-05T19:29:41.026 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:41.022+0000 7f371296be40 1 -- 172.21.15.167:0/2673647112 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:39 5.f 5:f4c53578:::zonegroups_names.default:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32) v8 -- 0x55e81344ae10 con 0x55e8134147f0 2023-10-05T19:29:41.026 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:41.022+0000 7f3710ca1700 1 -- 172.21.15.167:0/2673647112 <== osd.1 v2:172.21.15.167:6800/30035 15 ==== osd_op_reply(39 zonegroups_names.default [read 0~46 out=46b] v0'0 uv2 ondisk = 0) v8 ==== 168+0+46 (crc 0 0 0) 0x7f370800a5a0 con 0x55e8134147f0 2023-10-05T19:29:41.026 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:41.022+0000 7f371296be40 20 rados_obj.operate() r=0 bl.length=46 2023-10-05T19:29:41.026 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:41.022+0000 7f371296be40 20 rados->read ofs=0 len=0 2023-10-05T19:29:41.027 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:41.022+0000 7f371296be40 1 -- 172.21.15.167:0/2673647112 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:40 5.c 5:30d299a5:::zonegroup_info.4f1f719d-9648-417b-b10c-e5d939825419:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32) v8 -- 0x55e81344ae10 con 0x7f36f8007030 2023-10-05T19:29:41.028 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:41.022+0000 7f3710ca1700 1 -- 172.21.15.167:0/2673647112 <== osd.0 v2:172.21.15.118:6800/27690 10 ==== osd_op_reply(40 zonegroup_info.4f1f719d-9648-417b-b10c-e5d939825419 [read 0~362 out=362b] v0'0 uv1 ondisk = 0) v8 ==== 195+0+362 (crc 0 0 0) 0x7f370800a5a0 con 0x7f36f8007030 2023-10-05T19:29:41.028 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:41.022+0000 7f371296be40 20 rados_obj.operate() r=0 bl.length=362 2023-10-05T19:29:41.028 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:41.022+0000 7f371296be40 20 rados->read ofs=0 len=0 2023-10-05T19:29:41.028 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:41.022+0000 7f371296be40 1 -- 172.21.15.167:0/2673647112 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:41 5.13 5:cbbf6c60:::converted:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32) v8 -- 0x55e81344ae10 con 0x55e81344b7d0 2023-10-05T19:29:41.028 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:41.022+0000 7f37104a0700 1 -- 172.21.15.167:0/2673647112 <== osd.3 v2:172.21.15.167:6808/30036 10 ==== osd_op_reply(41 converted [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 ==== 153+0+0 (crc 0 0 0) 0x7f370000d7f0 con 0x55e81344b7d0 2023-10-05T19:29:41.028 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:41.022+0000 7f371296be40 20 rados_obj.operate() r=-2 bl.length=0 2023-10-05T19:29:41.028 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:41.022+0000 7f371296be40 20 rados->read ofs=0 len=0 2023-10-05T19:29:41.028 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:41.022+0000 7f371296be40 1 -- 172.21.15.167:0/2673647112 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:42 5.12 5:49953fa1:::default.realm:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32) v8 -- 0x55e81344e560 con 0x55e8134147f0 2023-10-05T19:29:41.029 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:41.022+0000 7f3710ca1700 1 -- 172.21.15.167:0/2673647112 <== osd.1 v2:172.21.15.167:6800/30035 16 ==== osd_op_reply(42 default.realm [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 ==== 157+0+0 (crc 0 0 0) 0x7f370800a5a0 con 0x55e8134147f0 2023-10-05T19:29:41.029 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:41.022+0000 7f371296be40 20 rados_obj.operate() r=-2 bl.length=0 2023-10-05T19:29:41.029 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:41.022+0000 7f371296be40 20 rados->read ofs=0 len=0 2023-10-05T19:29:41.029 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:41.022+0000 7f371296be40 1 -- 172.21.15.167:0/2673647112 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:43 5.f 5:f4c53578:::zonegroups_names.default:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32) v8 -- 0x55e81344aa10 con 0x55e8134147f0 2023-10-05T19:29:41.029 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:41.026+0000 7f3710ca1700 1 -- 172.21.15.167:0/2673647112 <== osd.1 v2:172.21.15.167:6800/30035 17 ==== osd_op_reply(43 zonegroups_names.default [read 0~46 out=46b] v0'0 uv2 ondisk = 0) v8 ==== 168+0+46 (crc 0 0 0) 0x7f370800a5a0 con 0x55e8134147f0 2023-10-05T19:29:41.029 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:41.026+0000 7f371296be40 20 rados_obj.operate() r=0 bl.length=46 2023-10-05T19:29:41.030 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:41.026+0000 7f371296be40 1 -- 172.21.15.167:0/2673647112 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:44 5.0 5:00000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x55e81344a9e0 con 0x55e81344b7d0 2023-10-05T19:29:41.030 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:41.026+0000 7f37104a0700 1 -- 172.21.15.167:0/2673647112 <== osd.3 v2:172.21.15.167:6808/30036 11 ==== osd_op_reply(44 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f370000d7f0 con 0x55e81344b7d0 2023-10-05T19:29:41.030 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:41.026+0000 7f37104a0700 1 -- 172.21.15.167:0/2673647112 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:45 5.10 5:08000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x55e81344a9e0 con 0x7f3700044040 2023-10-05T19:29:41.031 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:41.026+0000 7f370fc9f700 1 -- 172.21.15.167:0/2673647112 <== osd.2 v2:172.21.15.167:6816/30037 7 ==== osd_op_reply(45 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f36f8008ae0 con 0x7f3700044040 2023-10-05T19:29:41.031 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:41.026+0000 7f370fc9f700 1 -- 172.21.15.167:0/2673647112 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:46 5.8 5:10000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x55e81344a9e0 con 0x7f36f8007030 2023-10-05T19:29:41.032 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:41.026+0000 7f3710ca1700 1 -- 172.21.15.167:0/2673647112 <== osd.0 v2:172.21.15.118:6800/27690 11 ==== osd_op_reply(46 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f370800a5a0 con 0x7f36f8007030 2023-10-05T19:29:41.032 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:41.026+0000 7f3710ca1700 1 -- 172.21.15.167:0/2673647112 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:47 5.18 5:18000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x55e81344a520 con 0x55e8134147f0 2023-10-05T19:29:41.033 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:41.030+0000 7f3710ca1700 1 -- 172.21.15.167:0/2673647112 <== osd.1 v2:172.21.15.167:6800/30035 18 ==== osd_op_reply(47 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f370800a5a0 con 0x55e8134147f0 2023-10-05T19:29:41.033 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:41.030+0000 7f3710ca1700 1 -- 172.21.15.167:0/2673647112 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:48 5.4 5:20000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x55e81344a520 con 0x55e8134147f0 2023-10-05T19:29:41.034 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:41.030+0000 7f3710ca1700 1 -- 172.21.15.167:0/2673647112 <== osd.1 v2:172.21.15.167:6800/30035 19 ==== osd_op_reply(48 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f370800a5a0 con 0x55e8134147f0 2023-10-05T19:29:41.034 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:41.030+0000 7f3710ca1700 1 -- 172.21.15.167:0/2673647112 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:49 5.14 5:28000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x55e81344a520 con 0x7f36f8007030 2023-10-05T19:29:41.035 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:41.030+0000 7f3710ca1700 1 -- 172.21.15.167:0/2673647112 <== osd.0 v2:172.21.15.118:6800/27690 12 ==== osd_op_reply(49 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f370800a5a0 con 0x7f36f8007030 2023-10-05T19:29:41.035 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:41.030+0000 7f3710ca1700 1 -- 172.21.15.167:0/2673647112 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:50 5.c 5:30000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x55e81344a520 con 0x7f36f8007030 2023-10-05T19:29:41.036 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:41.030+0000 7f3710ca1700 1 -- 172.21.15.167:0/2673647112 <== osd.0 v2:172.21.15.118:6800/27690 13 ==== osd_op_reply(50 [pgnls start_epoch 32 out=112b] v23'1 uv1 ondisk = 1) v8 ==== 144+0+112 (crc 0 0 0) 0x7f370800a5a0 con 0x7f36f8007030 2023-10-05T19:29:41.036 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:41.030+0000 7f3710ca1700 1 -- 172.21.15.167:0/2673647112 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:51 5.1c 5:38000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x55e81344a520 con 0x7f3700044040 2023-10-05T19:29:41.037 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:41.034+0000 7f370fc9f700 1 -- 172.21.15.167:0/2673647112 <== osd.2 v2:172.21.15.167:6816/30037 8 ==== osd_op_reply(51 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f36f8008ae0 con 0x7f3700044040 2023-10-05T19:29:41.037 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:41.034+0000 7f370fc9f700 1 -- 172.21.15.167:0/2673647112 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:52 5.2 5:40000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x55e81344a520 con 0x55e81344b7d0 2023-10-05T19:29:41.037 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:41.034+0000 7f37104a0700 1 -- 172.21.15.167:0/2673647112 <== osd.3 v2:172.21.15.167:6808/30036 12 ==== osd_op_reply(52 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f370000d7f0 con 0x55e81344b7d0 2023-10-05T19:29:41.038 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:41.034+0000 7f37104a0700 1 -- 172.21.15.167:0/2673647112 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:53 5.12 5:48000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x55e81344a520 con 0x55e8134147f0 2023-10-05T19:29:41.038 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:41.034+0000 7f3710ca1700 1 -- 172.21.15.167:0/2673647112 <== osd.1 v2:172.21.15.167:6800/30035 20 ==== osd_op_reply(53 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f370800a5a0 con 0x55e8134147f0 2023-10-05T19:29:41.039 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:41.034+0000 7f3710ca1700 1 -- 172.21.15.167:0/2673647112 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:54 5.a 5:50000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x55e81344a520 con 0x7f36f8007030 2023-10-05T19:29:41.039 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:41.034+0000 7f3710ca1700 1 -- 172.21.15.167:0/2673647112 <== osd.0 v2:172.21.15.118:6800/27690 14 ==== osd_op_reply(54 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f370800a5a0 con 0x7f36f8007030 2023-10-05T19:29:41.039 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:41.034+0000 7f3710ca1700 1 -- 172.21.15.167:0/2673647112 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:55 5.1a 5:58000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x55e81344a520 con 0x55e8134147f0 2023-10-05T19:29:41.040 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:41.034+0000 7f3710ca1700 1 -- 172.21.15.167:0/2673647112 <== osd.1 v2:172.21.15.167:6800/30035 21 ==== osd_op_reply(55 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f370800a5a0 con 0x55e8134147f0 2023-10-05T19:29:41.040 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:41.034+0000 7f3710ca1700 1 -- 172.21.15.167:0/2673647112 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:56 5.6 5:60000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x55e81344a520 con 0x7f3700044040 2023-10-05T19:29:41.041 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:41.038+0000 7f370fc9f700 1 -- 172.21.15.167:0/2673647112 <== osd.2 v2:172.21.15.167:6816/30037 9 ==== osd_op_reply(56 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f36f8008ae0 con 0x7f3700044040 2023-10-05T19:29:41.041 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:41.038+0000 7f370fc9f700 1 -- 172.21.15.167:0/2673647112 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:57 5.16 5:68000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x55e81344a520 con 0x55e81344b7d0 2023-10-05T19:29:41.042 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:41.038+0000 7f37104a0700 1 -- 172.21.15.167:0/2673647112 <== osd.3 v2:172.21.15.167:6808/30036 13 ==== osd_op_reply(57 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f370000d7f0 con 0x55e81344b7d0 2023-10-05T19:29:41.042 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:41.038+0000 7f37104a0700 1 -- 172.21.15.167:0/2673647112 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:58 5.e 5:70000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x55e81344a520 con 0x55e81344b7d0 2023-10-05T19:29:41.043 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:41.038+0000 7f37104a0700 1 -- 172.21.15.167:0/2673647112 <== osd.3 v2:172.21.15.167:6808/30036 14 ==== osd_op_reply(58 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f370000d7f0 con 0x55e81344b7d0 2023-10-05T19:29:41.043 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:41.038+0000 7f37104a0700 1 -- 172.21.15.167:0/2673647112 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:59 5.1e 5:78000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x55e81344a520 con 0x7f36f8007030 2023-10-05T19:29:41.045 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:41.038+0000 7f3710ca1700 1 -- 172.21.15.167:0/2673647112 <== osd.0 v2:172.21.15.118:6800/27690 15 ==== osd_op_reply(59 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f370803f040 con 0x7f36f8007030 2023-10-05T19:29:41.046 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:41.038+0000 7f3710ca1700 1 -- 172.21.15.167:0/2673647112 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:60 5.1 5:80000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x55e81344a520 con 0x55e81344b7d0 2023-10-05T19:29:41.046 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:41.042+0000 7f37104a0700 1 -- 172.21.15.167:0/2673647112 <== osd.3 v2:172.21.15.167:6808/30036 15 ==== osd_op_reply(60 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f370000d7f0 con 0x55e81344b7d0 2023-10-05T19:29:41.046 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:41.042+0000 7f37104a0700 1 -- 172.21.15.167:0/2673647112 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:61 5.11 5:88000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x55e81344a520 con 0x55e81344b7d0 2023-10-05T19:29:41.046 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:41.042+0000 7f37104a0700 1 -- 172.21.15.167:0/2673647112 <== osd.3 v2:172.21.15.167:6808/30036 16 ==== osd_op_reply(61 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f370000d7f0 con 0x55e81344b7d0 2023-10-05T19:29:41.046 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:41.042+0000 7f37104a0700 1 -- 172.21.15.167:0/2673647112 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:62 5.9 5:90000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x55e81344a520 con 0x55e8134147f0 2023-10-05T19:29:41.047 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:41.042+0000 7f3710ca1700 1 -- 172.21.15.167:0/2673647112 <== osd.1 v2:172.21.15.167:6800/30035 22 ==== osd_op_reply(62 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f370803f040 con 0x55e8134147f0 2023-10-05T19:29:41.047 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:41.042+0000 7f3710ca1700 1 -- 172.21.15.167:0/2673647112 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:63 5.19 5:98000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x55e81344a520 con 0x55e8134147f0 2023-10-05T19:29:41.047 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:41.042+0000 7f3710ca1700 1 -- 172.21.15.167:0/2673647112 <== osd.1 v2:172.21.15.167:6800/30035 23 ==== osd_op_reply(63 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f370803f040 con 0x55e8134147f0 2023-10-05T19:29:41.047 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:41.042+0000 7f3710ca1700 1 -- 172.21.15.167:0/2673647112 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:64 5.5 5:a0000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x55e81344a520 con 0x7f36f8007030 2023-10-05T19:29:41.048 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:41.042+0000 7f3710ca1700 1 -- 172.21.15.167:0/2673647112 <== osd.0 v2:172.21.15.118:6800/27690 16 ==== osd_op_reply(64 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f370803f040 con 0x7f36f8007030 2023-10-05T19:29:41.048 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:41.042+0000 7f3710ca1700 1 -- 172.21.15.167:0/2673647112 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:65 5.15 5:a8000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x55e81344a520 con 0x55e81344b7d0 2023-10-05T19:29:41.049 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:41.046+0000 7f37104a0700 1 -- 172.21.15.167:0/2673647112 <== osd.3 v2:172.21.15.167:6808/30036 17 ==== osd_op_reply(65 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f370000d7f0 con 0x55e81344b7d0 2023-10-05T19:29:41.049 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:41.046+0000 7f37104a0700 1 -- 172.21.15.167:0/2673647112 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:66 5.d 5:b0000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x55e81344a520 con 0x7f3700044040 2023-10-05T19:29:41.050 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:41.046+0000 7f370fc9f700 1 -- 172.21.15.167:0/2673647112 <== osd.2 v2:172.21.15.167:6816/30037 10 ==== osd_op_reply(66 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f36f8008ae0 con 0x7f3700044040 2023-10-05T19:29:41.050 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:41.046+0000 7f370fc9f700 1 -- 172.21.15.167:0/2673647112 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:67 5.1d 5:b8000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x55e81344a520 con 0x55e8134147f0 2023-10-05T19:29:41.050 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:41.046+0000 7f3710ca1700 1 -- 172.21.15.167:0/2673647112 <== osd.1 v2:172.21.15.167:6800/30035 24 ==== osd_op_reply(67 [pgnls start_epoch 32 out=107b] v30'2 uv2 ondisk = 1) v8 ==== 144+0+107 (crc 0 0 0) 0x7f370803f040 con 0x55e8134147f0 2023-10-05T19:29:41.050 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:41.046+0000 7f3710ca1700 1 -- 172.21.15.167:0/2673647112 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:68 5.3 5:c0000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x55e81344a520 con 0x7f36f8007030 2023-10-05T19:29:41.052 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:41.046+0000 7f3710ca1700 1 -- 172.21.15.167:0/2673647112 <== osd.0 v2:172.21.15.118:6800/27690 17 ==== osd_op_reply(68 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f370803f040 con 0x7f36f8007030 2023-10-05T19:29:41.052 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:41.046+0000 7f3710ca1700 1 -- 172.21.15.167:0/2673647112 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:69 5.13 5:c8000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x55e81344a520 con 0x55e81344b7d0 2023-10-05T19:29:41.052 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:41.050+0000 7f37104a0700 1 -- 172.21.15.167:0/2673647112 <== osd.3 v2:172.21.15.167:6808/30036 18 ==== osd_op_reply(69 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f370000d7f0 con 0x55e81344b7d0 2023-10-05T19:29:41.052 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:41.050+0000 7f37104a0700 1 -- 172.21.15.167:0/2673647112 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:70 5.b 5:d0000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x55e81344a520 con 0x7f36f8007030 2023-10-05T19:29:41.053 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:41.050+0000 7f3710ca1700 1 -- 172.21.15.167:0/2673647112 <== osd.0 v2:172.21.15.118:6800/27690 18 ==== osd_op_reply(70 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f370803f040 con 0x7f36f8007030 2023-10-05T19:29:41.054 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:41.050+0000 7f3710ca1700 1 -- 172.21.15.167:0/2673647112 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:71 5.1b 5:d8000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x55e81344a520 con 0x7f3700044040 2023-10-05T19:29:41.054 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:41.050+0000 7f370fc9f700 1 -- 172.21.15.167:0/2673647112 <== osd.2 v2:172.21.15.167:6816/30037 11 ==== osd_op_reply(71 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f36f8008ae0 con 0x7f3700044040 2023-10-05T19:29:41.054 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:41.050+0000 7f370fc9f700 1 -- 172.21.15.167:0/2673647112 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:72 5.7 5:e0000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x55e81344a520 con 0x55e81344b7d0 2023-10-05T19:29:41.055 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:41.050+0000 7f37104a0700 1 -- 172.21.15.167:0/2673647112 <== osd.3 v2:172.21.15.167:6808/30036 19 ==== osd_op_reply(72 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f370000d7f0 con 0x55e81344b7d0 2023-10-05T19:29:41.055 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:41.050+0000 7f37104a0700 1 -- 172.21.15.167:0/2673647112 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:73 5.17 5:e8000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x55e81344a520 con 0x7f36f8007030 2023-10-05T19:29:41.056 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:41.050+0000 7f3710ca1700 1 -- 172.21.15.167:0/2673647112 <== osd.0 v2:172.21.15.118:6800/27690 19 ==== osd_op_reply(73 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f370803f040 con 0x7f36f8007030 2023-10-05T19:29:41.056 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:41.050+0000 7f3710ca1700 1 -- 172.21.15.167:0/2673647112 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:74 5.f 5:f0000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x55e81344a520 con 0x55e8134147f0 2023-10-05T19:29:41.057 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:41.054+0000 7f3710ca1700 1 -- 172.21.15.167:0/2673647112 <== osd.1 v2:172.21.15.167:6800/30035 25 ==== osd_op_reply(74 [pgnls start_epoch 32 out=115b] v23'2 uv2 ondisk = 1) v8 ==== 144+0+115 (crc 0 0 0) 0x7f370803f040 con 0x55e8134147f0 2023-10-05T19:29:41.057 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:41.054+0000 7f3710ca1700 1 -- 172.21.15.167:0/2673647112 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:75 5.1f 5:f8000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x55e81344a520 con 0x7f3700044040 2023-10-05T19:29:41.058 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:41.054+0000 7f370fc9f700 1 -- 172.21.15.167:0/2673647112 <== osd.2 v2:172.21.15.167:6816/30037 12 ==== osd_op_reply(75 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f36f8008ae0 con 0x7f3700044040 2023-10-05T19:29:41.058 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:41.054+0000 7f371296be40 20 RGWRados::pool_iterate: got zonegroup_info.4f1f719d-9648-417b-b10c-e5d939825419 2023-10-05T19:29:41.058 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:41.054+0000 7f371296be40 20 RGWRados::pool_iterate: got zone_info.9a9df965-6612-45d4-a45f-6e3c889bacc2 2023-10-05T19:29:41.058 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:41.054+0000 7f371296be40 20 RGWRados::pool_iterate: got zone_names.default 2023-10-05T19:29:41.059 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:41.054+0000 7f371296be40 20 RGWRados::pool_iterate: got zonegroups_names.default 2023-10-05T19:29:41.059 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:41.054+0000 7f371296be40 20 rados->read ofs=0 len=0 2023-10-05T19:29:41.059 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:41.054+0000 7f371296be40 1 -- 172.21.15.167:0/2673647112 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:76 5.f 5:f43b6ece:::zone_names.default:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32) v8 -- 0x55e81344a9e0 con 0x55e8134147f0 2023-10-05T19:29:41.060 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:41.054+0000 7f3710ca1700 1 -- 172.21.15.167:0/2673647112 <== osd.1 v2:172.21.15.167:6800/30035 26 ==== osd_op_reply(76 zone_names.default [read 0~46 out=46b] v0'0 uv1 ondisk = 0) v8 ==== 162+0+46 (crc 0 0 0) 0x7f370803f040 con 0x55e8134147f0 2023-10-05T19:29:41.060 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:41.054+0000 7f371296be40 20 rados_obj.operate() r=0 bl.length=46 2023-10-05T19:29:41.060 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:41.054+0000 7f371296be40 20 rados->read ofs=0 len=0 2023-10-05T19:29:41.060 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:41.054+0000 7f371296be40 1 -- 172.21.15.167:0/2673647112 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:77 5.1d 5:b8374eb4:::zone_info.9a9df965-6612-45d4-a45f-6e3c889bacc2:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32) v8 -- 0x55e81344eaa0 con 0x55e8134147f0 2023-10-05T19:29:41.060 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:41.054+0000 7f3710ca1700 1 -- 172.21.15.167:0/2673647112 <== osd.1 v2:172.21.15.167:6800/30035 27 ==== osd_op_reply(77 zone_info.9a9df965-6612-45d4-a45f-6e3c889bacc2 [read 0~874 out=874b] v0'0 uv2 ondisk = 0) v8 ==== 190+0+874 (crc 0 0 0) 0x7f370803f040 con 0x55e8134147f0 2023-10-05T19:29:41.060 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:41.054+0000 7f371296be40 20 rados_obj.operate() r=0 bl.length=874 2023-10-05T19:29:41.060 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:41.054+0000 7f371296be40 20 rados->read ofs=0 len=0 2023-10-05T19:29:41.060 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:41.054+0000 7f371296be40 1 -- 172.21.15.167:0/2673647112 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:78 5.f 5:f4c53578:::zonegroups_names.default:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32) v8 -- 0x55e81344ef90 con 0x55e8134147f0 2023-10-05T19:29:41.061 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:41.054+0000 7f3710ca1700 1 -- 172.21.15.167:0/2673647112 <== osd.1 v2:172.21.15.167:6800/30035 28 ==== osd_op_reply(78 zonegroups_names.default [read 0~46 out=46b] v0'0 uv2 ondisk = 0) v8 ==== 168+0+46 (crc 0 0 0) 0x7f370803f040 con 0x55e8134147f0 2023-10-05T19:29:41.061 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:41.058+0000 7f371296be40 20 rados_obj.operate() r=0 bl.length=46 2023-10-05T19:29:41.061 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:41.058+0000 7f371296be40 20 rados->read ofs=0 len=0 2023-10-05T19:29:41.061 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:41.058+0000 7f371296be40 1 -- 172.21.15.167:0/2673647112 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:79 5.c 5:30d299a5:::zonegroup_info.4f1f719d-9648-417b-b10c-e5d939825419:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32) v8 -- 0x55e81344ef90 con 0x7f36f8007030 2023-10-05T19:29:41.062 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:41.058+0000 7f3710ca1700 1 -- 172.21.15.167:0/2673647112 <== osd.0 v2:172.21.15.118:6800/27690 20 ==== osd_op_reply(79 zonegroup_info.4f1f719d-9648-417b-b10c-e5d939825419 [read 0~362 out=362b] v0'0 uv1 ondisk = 0) v8 ==== 195+0+362 (crc 0 0 0) 0x7f370803f040 con 0x7f36f8007030 2023-10-05T19:29:41.062 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:41.058+0000 7f371296be40 20 rados_obj.operate() r=0 bl.length=362 2023-10-05T19:29:41.062 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:41.058+0000 7f371296be40 20 rados->read ofs=0 len=0 2023-10-05T19:29:41.063 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:41.058+0000 7f371296be40 1 -- 172.21.15.167:0/2673647112 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:80 5.3 5:c0ad8cbd:::region_map:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32) v8 -- 0x55e81344ef90 con 0x7f36f8007030 2023-10-05T19:29:41.063 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:41.058+0000 7f3710ca1700 1 -- 172.21.15.167:0/2673647112 <== osd.0 v2:172.21.15.118:6800/27690 21 ==== osd_op_reply(80 region_map [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 ==== 154+0+0 (crc 0 0 0) 0x7f370803f040 con 0x7f36f8007030 2023-10-05T19:29:41.063 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:41.058+0000 7f371296be40 20 rados_obj.operate() r=-2 bl.length=0 2023-10-05T19:29:41.063 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:41.058+0000 7f371296be40 20 zone default found 2023-10-05T19:29:41.063 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:41.058+0000 7f371296be40 4 Realm: () 2023-10-05T19:29:41.063 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:41.058+0000 7f371296be40 4 ZoneGroup: default (4f1f719d-9648-417b-b10c-e5d939825419) 2023-10-05T19:29:41.063 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:41.058+0000 7f371296be40 4 Zone: default (9a9df965-6612-45d4-a45f-6e3c889bacc2) 2023-10-05T19:29:41.063 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:41.058+0000 7f371296be40 10 cannot find current period zonegroup using local zonegroup configuration 2023-10-05T19:29:41.063 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:41.058+0000 7f371296be40 20 zonegroup default 2023-10-05T19:29:41.063 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:41.058+0000 7f371296be40 20 rados->read ofs=0 len=0 2023-10-05T19:29:41.064 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:41.058+0000 7f371296be40 1 -- 172.21.15.167:0/2673647112 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:81 5.3 5:c52100b6:::period_config.default:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32) v8 -- 0x55e81344ef90 con 0x7f36f8007030 2023-10-05T19:29:41.065 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:41.058+0000 7f3710ca1700 1 -- 172.21.15.167:0/2673647112 <== osd.0 v2:172.21.15.118:6800/27690 22 ==== osd_op_reply(81 period_config.default [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 ==== 165+0+0 (crc 0 0 0) 0x7f370803f040 con 0x7f36f8007030 2023-10-05T19:29:41.065 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:41.058+0000 7f371296be40 20 rados_obj.operate() r=-2 bl.length=0 2023-10-05T19:29:41.065 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:41.058+0000 7f371296be40 20 rados->read ofs=0 len=0 2023-10-05T19:29:41.065 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:41.058+0000 7f371296be40 1 -- 172.21.15.167:0/2673647112 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:82 6.2 6:4347d321:::bucket.sync-source-hints.:head [call version.read in=11b,read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32) v8 -- 0x55e813450910 con 0x55e81344b7d0 2023-10-05T19:29:41.065 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:41.058+0000 7f37104a0700 1 -- 172.21.15.167:0/2673647112 <== osd.3 v2:172.21.15.167:6808/30036 20 ==== osd_op_reply(82 bucket.sync-source-hints. [call,read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 ==== 211+0+0 (crc 0 0 0) 0x7f370000d7f0 con 0x55e81344b7d0 2023-10-05T19:29:41.065 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:41.058+0000 7f371296be40 20 rados_obj.operate() r=-2 bl.length=0 2023-10-05T19:29:41.065 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:41.058+0000 7f371296be40 20 rados->read ofs=0 len=0 2023-10-05T19:29:41.066 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:41.058+0000 7f371296be40 1 -- 172.21.15.167:0/2673647112 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:83 6.b 6:d467b91b:::bucket.sync-target-hints.:head [call version.read in=11b,read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32) v8 -- 0x55e813450cc0 con 0x55e81344b7d0 2023-10-05T19:29:41.066 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:41.062+0000 7f37104a0700 1 -- 172.21.15.167:0/2673647112 <== osd.3 v2:172.21.15.167:6808/30036 21 ==== osd_op_reply(83 bucket.sync-target-hints. [call,read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 ==== 211+0+0 (crc 0 0 0) 0x7f370000d7f0 con 0x55e81344b7d0 2023-10-05T19:29:41.066 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:41.062+0000 7f371296be40 20 rados_obj.operate() r=-2 bl.length=0 2023-10-05T19:29:41.066 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:41.062+0000 7f371296be40 20 started sync module instance, tier type = 2023-10-05T19:29:41.066 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:41.062+0000 7f371296be40 20 started zone id=9a9df965-6612-45d4-a45f-6e3c889bacc2 (name=default) with tier type = 2023-10-05T19:29:41.066 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:41.062+0000 7f371296be40 1 -- 172.21.15.167:0/2673647112 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:84 7.1f 7:f95f44c2:::notify.0:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e32) v8 -- 0x55e81344ffd0 con 0x7f3700044040 2023-10-05T19:29:41.068 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:41.062+0000 7f370fc9f700 1 -- 172.21.15.167:0/2673647112 <== osd.2 v2:172.21.15.167:6816/30037 13 ==== osd_op_reply(84 notify.0 [create] v32'255 uv255 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f36f8008ae0 con 0x7f3700044040 2023-10-05T19:29:41.068 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:41.062+0000 7f371296be40 1 -- 172.21.15.167:0/2673647112 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:85 7.1f 7:f95f44c2:::notify.0:head [watch watch cookie 94455244063616] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e32) v8 -- 0x55e81344fc00 con 0x7f3700044040 2023-10-05T19:29:41.068 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:41.062+0000 7f371296be40 1 -- 172.21.15.167:0/2673647112 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:86 7.0 7:05bf5b68:::notify.1:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e32) v8 -- 0x55e813451a50 con 0x55e8134147f0 2023-10-05T19:29:41.070 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:41.066+0000 7f370fc9f700 1 -- 172.21.15.167:0/2673647112 <== osd.2 v2:172.21.15.167:6816/30037 14 ==== osd_op_reply(85 notify.0 [watch watch cookie 94455244063616] v32'256 uv255 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f36f8008ae0 con 0x7f3700044040 2023-10-05T19:29:41.071 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:41.066+0000 7f3710ca1700 1 -- 172.21.15.167:0/2673647112 <== osd.1 v2:172.21.15.167:6800/30035 29 ==== osd_op_reply(86 notify.1 [create] v32'255 uv255 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f3708052040 con 0x55e8134147f0 2023-10-05T19:29:41.071 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:41.066+0000 7f371296be40 1 -- 172.21.15.167:0/2673647112 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:87 7.0 7:05bf5b68:::notify.1:head [watch watch cookie 94455244067216] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e32) v8 -- 0x55e813451b10 con 0x55e8134147f0 2023-10-05T19:29:41.071 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:41.066+0000 7f371296be40 1 -- 172.21.15.167:0/2673647112 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:88 7.15 7:a93a5511:::notify.2:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e32) v8 -- 0x55e813452530 con 0x55e81344b7d0 2023-10-05T19:29:41.074 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:41.070+0000 7f3710ca1700 1 -- 172.21.15.167:0/2673647112 <== osd.1 v2:172.21.15.167:6800/30035 30 ==== osd_op_reply(87 notify.1 [watch watch cookie 94455244067216] v32'256 uv255 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f3708052040 con 0x55e8134147f0 2023-10-05T19:29:41.075 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:41.070+0000 7f37104a0700 1 -- 172.21.15.167:0/2673647112 <== osd.3 v2:172.21.15.167:6808/30036 22 ==== osd_op_reply(88 notify.2 [create] v32'255 uv255 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f370000d7f0 con 0x55e81344b7d0 2023-10-05T19:29:41.075 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:41.070+0000 7f371296be40 1 -- 172.21.15.167:0/2673647112 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:89 7.15 7:a93a5511:::notify.2:head [watch watch cookie 94455244068320] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e32) v8 -- 0x55e813452930 con 0x55e81344b7d0 2023-10-05T19:29:41.075 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:41.070+0000 7f371296be40 1 -- 172.21.15.167:0/2673647112 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:90 7.e 7:7759931f:::notify.3:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e32) v8 -- 0x55e813453210 con 0x7f3700044040 2023-10-05T19:29:41.078 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:41.074+0000 7f370fc9f700 1 -- 172.21.15.167:0/2673647112 <== osd.2 v2:172.21.15.167:6816/30037 15 ==== osd_op_reply(90 notify.3 [create] v32'255 uv255 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f36f8008ae0 con 0x7f3700044040 2023-10-05T19:29:41.078 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:41.074+0000 7f371296be40 1 -- 172.21.15.167:0/2673647112 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:91 7.e 7:7759931f:::notify.3:head [watch watch cookie 94455244076560] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e32) v8 -- 0x55e813453f50 con 0x7f3700044040 2023-10-05T19:29:41.078 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:41.074+0000 7f37104a0700 1 -- 172.21.15.167:0/2673647112 <== osd.3 v2:172.21.15.167:6808/30036 23 ==== osd_op_reply(89 notify.2 [watch watch cookie 94455244068320] v32'256 uv255 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f370000d7f0 con 0x55e81344b7d0 2023-10-05T19:29:41.078 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:41.074+0000 7f371296be40 1 -- 172.21.15.167:0/2673647112 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:92 7.d 7:b4812045:::notify.4:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e32) v8 -- 0x55e813454960 con 0x55e8134147f0 2023-10-05T19:29:41.081 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:41.078+0000 7f370fc9f700 1 -- 172.21.15.167:0/2673647112 <== osd.2 v2:172.21.15.167:6816/30037 16 ==== osd_op_reply(91 notify.3 [watch watch cookie 94455244076560] v32'256 uv255 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f36f8008ae0 con 0x7f3700044040 2023-10-05T19:29:41.081 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:41.078+0000 7f3710ca1700 1 -- 172.21.15.167:0/2673647112 <== osd.1 v2:172.21.15.167:6800/30035 31 ==== osd_op_reply(92 notify.4 [create] v32'255 uv255 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f3708052040 con 0x55e8134147f0 2023-10-05T19:29:41.081 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:41.078+0000 7f371296be40 1 -- 172.21.15.167:0/2673647112 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:93 7.d 7:b4812045:::notify.4:head [watch watch cookie 94455244070912] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e32) v8 -- 0x55e813453b70 con 0x55e8134147f0 2023-10-05T19:29:41.081 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:41.078+0000 7f371296be40 1 -- 172.21.15.167:0/2673647112 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:94 7.3 7:c609908c:::notify.5:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e32) v8 -- 0x55e813455030 con 0x7f36f8007030 2023-10-05T19:29:41.084 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:41.078+0000 7f3710ca1700 1 -- 172.21.15.167:0/2673647112 <== osd.0 v2:172.21.15.118:6800/27690 23 ==== osd_op_reply(94 notify.5 [create] v32'255 uv255 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f3708052040 con 0x7f36f8007030 2023-10-05T19:29:41.084 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:41.078+0000 7f371296be40 1 -- 172.21.15.167:0/2673647112 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:95 7.3 7:c609908c:::notify.5:head [watch watch cookie 94455244081920] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e32) v8 -- 0x55e813454f50 con 0x7f36f8007030 2023-10-05T19:29:41.084 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:41.078+0000 7f371296be40 1 -- 172.21.15.167:0/2673647112 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:96 7.14 7:2b04a3e9:::notify.6:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e32) v8 -- 0x55e813455f80 con 0x7f36f8007030 2023-10-05T19:29:41.084 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:41.078+0000 7f3710ca1700 1 -- 172.21.15.167:0/2673647112 <== osd.1 v2:172.21.15.167:6800/30035 32 ==== osd_op_reply(93 notify.4 [watch watch cookie 94455244070912] v32'256 uv255 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f3708052040 con 0x55e8134147f0 2023-10-05T19:29:41.086 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:41.082+0000 7f3710ca1700 1 -- 172.21.15.167:0/2673647112 <== osd.0 v2:172.21.15.118:6800/27690 24 ==== osd_op_reply(95 notify.5 [watch watch cookie 94455244081920] v32'256 uv255 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f3708052040 con 0x7f36f8007030 2023-10-05T19:29:41.087 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:41.082+0000 7f3710ca1700 1 -- 172.21.15.167:0/2673647112 <== osd.0 v2:172.21.15.118:6800/27690 25 ==== osd_op_reply(96 notify.6 [create] v32'255 uv255 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f3708052040 con 0x7f36f8007030 2023-10-05T19:29:41.087 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:41.082+0000 7f371296be40 1 -- 172.21.15.167:0/2673647112 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:97 7.14 7:2b04a3e9:::notify.6:head [watch watch cookie 94455244080240] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e32) v8 -- 0x55e813455a90 con 0x7f36f8007030 2023-10-05T19:29:41.087 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:41.082+0000 7f371296be40 1 -- 172.21.15.167:0/2673647112 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:98 7.9 7:93e5b521:::notify.7:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e32) v8 -- 0x55e813456870 con 0x7f36f8007030 2023-10-05T19:29:41.090 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:41.086+0000 7f3710ca1700 1 -- 172.21.15.167:0/2673647112 <== osd.0 v2:172.21.15.118:6800/27690 26 ==== osd_op_reply(97 notify.6 [watch watch cookie 94455244080240] v32'256 uv255 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f3708052040 con 0x7f36f8007030 2023-10-05T19:29:41.090 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:41.086+0000 7f3710ca1700 1 -- 172.21.15.167:0/2673647112 <== osd.0 v2:172.21.15.118:6800/27690 27 ==== osd_op_reply(98 notify.7 [create] v32'255 uv255 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f3708052040 con 0x7f36f8007030 2023-10-05T19:29:41.091 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:41.086+0000 7f371296be40 1 -- 172.21.15.167:0/2673647112 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:99 7.9 7:93e5b521:::notify.7:head [watch watch cookie 94455244084192] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e32) v8 -- 0x55e813455a90 con 0x7f36f8007030 2023-10-05T19:29:41.091 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:41.086+0000 7f371296be40 20 add_watcher() i=0 2023-10-05T19:29:41.091 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:41.086+0000 7f371296be40 20 add_watcher() i=1 2023-10-05T19:29:41.091 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:41.086+0000 7f371296be40 20 add_watcher() i=2 2023-10-05T19:29:41.091 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:41.086+0000 7f371296be40 20 add_watcher() i=3 2023-10-05T19:29:41.091 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:41.086+0000 7f371296be40 20 add_watcher() i=4 2023-10-05T19:29:41.091 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:41.086+0000 7f371296be40 20 add_watcher() i=5 2023-10-05T19:29:41.091 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:41.086+0000 7f371296be40 20 add_watcher() i=6 2023-10-05T19:29:41.092 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:41.090+0000 7f3710ca1700 1 -- 172.21.15.167:0/2673647112 <== osd.0 v2:172.21.15.118:6800/27690 28 ==== osd_op_reply(99 notify.7 [watch watch cookie 94455244084192] v32'256 uv255 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f3708052040 con 0x7f36f8007030 2023-10-05T19:29:41.093 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:41.090+0000 7f371296be40 20 add_watcher() i=7 2023-10-05T19:29:41.093 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:41.090+0000 7f371296be40 2 all 8 watchers are set, enabling cache 2023-10-05T19:29:41.093 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:41.090+0000 7f371296be40 1 -- 172.21.15.167:0/2673647112 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:100 6.4 6:22d26bf9:::data_loggenerations_metadata:head [call version.check_conds in=74b,call version.read in=11b,read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32) v8 -- 0x55e8134596d0 con 0x55e8134147f0 2023-10-05T19:29:41.093 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:41.090+0000 7f3710ca1700 1 -- 172.21.15.167:0/2673647112 <== osd.1 v2:172.21.15.167:6800/30035 33 ==== osd_op_reply(100 data_loggenerations_metadata [call,call out=48b,read 0~28 out=28b] v0'0 uv1 ondisk = 0) v8 ==== 256+0+76 (crc 0 0 0) 0x7f3708052040 con 0x55e8134147f0 2023-10-05T19:29:41.093 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:41.090+0000 7f371296be40 1 -- 172.21.15.167:0/2673647112 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:101 6.4 6:22d26bf9:::data_loggenerations_metadata:head [watch watch cookie 94455244087728] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e32) v8 -- 0x55e813456b90 con 0x55e8134147f0 2023-10-05T19:29:41.096 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:41.090+0000 7f3710ca1700 1 -- 172.21.15.167:0/2673647112 <== osd.1 v2:172.21.15.167:6800/30035 34 ==== osd_op_reply(101 data_loggenerations_metadata [watch watch cookie 94455244087728] v32'246 uv1 ondisk = 0) v8 ==== 172+0+0 (crc 0 0 0) 0x7f3708052040 con 0x55e8134147f0 2023-10-05T19:29:41.096 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:41.090+0000 7f36e1fe3700 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2023-10-05T19:29:41.096 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:41.090+0000 7f371296be40 20 check_secure_mon_conn(): auth registy supported: methods=[2] modes=[2,1] 2023-10-05T19:29:41.096 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:41.090+0000 7f371296be40 20 check_secure_mon_conn(): mode 1 is insecure 2023-10-05T19:29:41.097 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:41.094+0000 7f371296be40 1 Processor -- start 2023-10-05T19:29:41.097 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:41.094+0000 7f371296be40 1 -- start start 2023-10-05T19:29:41.097 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:41.094+0000 7f371296be40 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55e813463df0 0x55e8134641c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-05T19:29:41.098 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:41.094+0000 7f371296be40 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e813464700 con 0x55e813463df0 2023-10-05T19:29:41.098 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:41.094+0000 7f370fc9f700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55e813463df0 0x55e8134641c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-05T19:29:41.098 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:41.094+0000 7f370fc9f700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55e813463df0 0x55e8134641c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.118:3300/0 says I am v2:172.21.15.167:56916/0 (socket says 172.21.15.167:56916) 2023-10-05T19:29:41.098 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:41.094+0000 7f370fc9f700 1 -- 172.21.15.167:0/2597043961 learned_addr learned my addr 172.21.15.167:0/2597043961 (peer_addr_for_me v2:172.21.15.167:0/0) 2023-10-05T19:29:41.098 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:41.094+0000 7f370fc9f700 1 -- 172.21.15.167:0/2597043961 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55e813464840 con 0x55e813463df0 2023-10-05T19:29:41.098 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:41.094+0000 7f370fc9f700 1 --2- 172.21.15.167:0/2597043961 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55e813463df0 0x55e8134641c0 secure :-1 s=READY pgs=648 cs=0 l=1 rev1=1 crypto rx=0x55e813413cb0 tx=0x7f36f8012000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8d4281335d590fd4 server_cookie=0 in_seq=0 out_seq=0 2023-10-05T19:29:41.099 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:41.094+0000 7f36e17e2700 1 -- 172.21.15.167:0/2597043961 <== mon.0 v2:172.21.15.118:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f36f8012c90 con 0x55e813463df0 2023-10-05T19:29:41.099 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:41.094+0000 7f36e17e2700 1 -- 172.21.15.167:0/2597043961 <== mon.0 v2:172.21.15.118:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f36f8012e10 con 0x55e813463df0 2023-10-05T19:29:41.099 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:41.094+0000 7f36e17e2700 1 -- 172.21.15.167:0/2597043961 <== mon.0 v2:172.21.15.118:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f36f802a8a0 con 0x55e813463df0 2023-10-05T19:29:41.100 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:41.094+0000 7f371296be40 1 -- 172.21.15.167:0/2597043961 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55e813463df0 msgr2=0x55e8134641c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-05T19:29:41.101 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:41.094+0000 7f371296be40 1 --2- 172.21.15.167:0/2597043961 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55e813463df0 0x55e8134641c0 secure :-1 s=READY pgs=648 cs=0 l=1 rev1=1 crypto rx=0x55e813413cb0 tx=0x7f36f8012000 comp rx=0 tx=0).stop 2023-10-05T19:29:41.101 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:41.098+0000 7f371296be40 1 -- 172.21.15.167:0/2597043961 shutdown_connections 2023-10-05T19:29:41.101 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:41.098+0000 7f371296be40 1 --2- 172.21.15.167:0/2597043961 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55e813463df0 0x55e8134641c0 unknown :-1 s=CLOSED pgs=648 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-05T19:29:41.101 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:41.098+0000 7f371296be40 1 -- 172.21.15.167:0/2597043961 >> 172.21.15.167:0/2597043961 conn(0x55e81345f1b0 msgr2=0x55e81345d300 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-05T19:29:41.101 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:41.098+0000 7f371296be40 1 -- 172.21.15.167:0/2597043961 shutdown_connections 2023-10-05T19:29:41.101 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:41.098+0000 7f371296be40 1 -- 172.21.15.167:0/2597043961 wait complete. 2023-10-05T19:29:41.102 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:41.098+0000 7f371296be40 1 Processor -- start 2023-10-05T19:29:41.102 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:41.098+0000 7f371296be40 1 -- start start 2023-10-05T19:29:41.102 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:41.098+0000 7f371296be40 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55e813463df0 0x55e813469920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-05T19:29:41.102 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:41.098+0000 7f371296be40 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e81346a920 con 0x55e813463df0 2023-10-05T19:29:41.103 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:41.098+0000 7f370fc9f700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55e813463df0 0x55e813469920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-05T19:29:41.103 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:41.098+0000 7f370fc9f700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55e813463df0 0x55e813469920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.118:3300/0 says I am v2:172.21.15.167:56930/0 (socket says 172.21.15.167:56930) 2023-10-05T19:29:41.103 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:41.098+0000 7f370fc9f700 1 -- 172.21.15.167:0/2022125644 learned_addr learned my addr 172.21.15.167:0/2022125644 (peer_addr_for_me v2:172.21.15.167:0/0) 2023-10-05T19:29:41.103 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:41.098+0000 7f370fc9f700 1 -- 172.21.15.167:0/2022125644 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55e813413ce0 con 0x55e813463df0 2023-10-05T19:29:41.104 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:41.098+0000 7f370fc9f700 1 --2- 172.21.15.167:0/2022125644 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55e813463df0 0x55e813469920 secure :-1 s=READY pgs=649 cs=0 l=1 rev1=1 crypto rx=0x55e81345de70 tx=0x7f36f802f000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-05T19:29:41.104 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:41.098+0000 7f365b7fe700 1 -- 172.21.15.167:0/2022125644 <== mon.0 v2:172.21.15.118:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f36f803e950 con 0x55e813463df0 2023-10-05T19:29:41.104 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:41.098+0000 7f365b7fe700 1 -- 172.21.15.167:0/2022125644 <== mon.0 v2:172.21.15.118:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f36f803eab0 con 0x55e813463df0 2023-10-05T19:29:41.104 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:41.098+0000 7f371296be40 1 -- 172.21.15.167:0/2022125644 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x55e813469f20 con 0x55e813463df0 2023-10-05T19:29:41.104 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:41.098+0000 7f365b7fe700 1 -- 172.21.15.167:0/2022125644 <== mon.0 v2:172.21.15.118:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f36f803ec60 con 0x55e813463df0 2023-10-05T19:29:41.104 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:41.098+0000 7f371296be40 1 -- 172.21.15.167:0/2022125644 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55e81346a3c0 con 0x55e813463df0 2023-10-05T19:29:41.105 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:41.102+0000 7f365b7fe700 1 -- 172.21.15.167:0/2022125644 <== mon.0 v2:172.21.15.118:3300/0 4 ==== mgrmap(e 5) v1 ==== 40329+0+0 (secure 0 0 0) 0x7f36f8046070 con 0x55e813463df0 2023-10-05T19:29:41.106 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:41.102+0000 7f365b7fe700 1 --2- 172.21.15.167:0/2022125644 >> [v2:172.21.15.167:6824/29889,v1:172.21.15.167:6825/29889] conn(0x7f3650030c50 0x7f3650033100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-05T19:29:41.106 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:41.102+0000 7f365b7fe700 1 -- 172.21.15.167:0/2022125644 <== mon.0 v2:172.21.15.118:3300/0 5 ==== osd_map(32..32 src has 1..32) v4 ==== 6260+0+0 (secure 0 0 0) 0x7f36f802f070 con 0x55e813463df0 2023-10-05T19:29:41.106 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:41.102+0000 7f3710ca1700 1 --2- 172.21.15.167:0/2022125644 >> [v2:172.21.15.167:6824/29889,v1:172.21.15.167:6825/29889] conn(0x7f3650030c50 0x7f3650033100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-05T19:29:41.106 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:41.102+0000 7f371296be40 1 --2- 172.21.15.167:0/2022125644 >> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] conn(0x55e813474510 0x55e813476920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-05T19:29:41.107 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:41.102+0000 7f371296be40 1 -- 172.21.15.167:0/2022125644 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:1 6.19 6:9a6dd8c0:gc::gc.0:head [create,call version.check_conds in=74b,call rgw_gc.rgw_gc_queue_init in=45b,call version.set in=34b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e32) v8 -- 0x55e813476f70 con 0x55e813474510 2023-10-05T19:29:41.107 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:41.102+0000 7f37104a0700 1 --2- 172.21.15.167:0/2022125644 >> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] conn(0x55e813474510 0x55e813476920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-05T19:29:41.107 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:41.102+0000 7f3710ca1700 1 --2- 172.21.15.167:0/2022125644 >> [v2:172.21.15.167:6824/29889,v1:172.21.15.167:6825/29889] conn(0x7f3650030c50 0x7f3650033100 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x55e813456af0 tx=0x7f370800aec0 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-05T19:29:41.107 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:41.102+0000 7f37104a0700 1 --2- 172.21.15.167:0/2022125644 >> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] conn(0x55e813474510 0x55e813476920 crc :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-05T19:29:41.110 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:41.106+0000 7f37104a0700 1 -- 172.21.15.167:0/2022125644 <== osd.1 v2:172.21.15.167:6800/30035 1 ==== osd_op_reply(1 gc.0 [create,call,call,call] v32'39 uv33 ondisk = -125 ((125) Operation canceled)) v8 ==== 274+0+0 (crc 0 0 0) 0x7f370000d7f0 con 0x55e813474510 2023-10-05T19:29:41.110 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:41.106+0000 7f371296be40 1 --2- 172.21.15.167:0/2022125644 >> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] conn(0x55e813470150 0x55e81347a5e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-05T19:29:41.110 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:41.106+0000 7f371296be40 1 -- 172.21.15.167:0/2022125644 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:2 6.18 6:1e85421f:gc::gc.1:head [create,call version.check_conds in=74b,call rgw_gc.rgw_gc_queue_init in=45b,call version.set in=34b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e32) v8 -- 0x55e81347ab20 con 0x55e813470150 2023-10-05T19:29:41.110 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:41.106+0000 7f370fc9f700 1 --2- 172.21.15.167:0/2022125644 >> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] conn(0x55e813470150 0x55e81347a5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-05T19:29:41.111 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:41.106+0000 7f370fc9f700 1 --2- 172.21.15.167:0/2022125644 >> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] conn(0x55e813470150 0x55e81347a5e0 crc :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-05T19:29:41.114 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:41.110+0000 7f370fc9f700 1 -- 172.21.15.167:0/2022125644 <== osd.0 v2:172.21.15.118:6800/27690 1 ==== osd_op_reply(2 gc.1 [create,call,call,call] v32'32 uv25 ondisk = -125 ((125) Operation canceled)) v8 ==== 274+0+0 (crc 0 0 0) 0x7f36f8032040 con 0x55e813470150 2023-10-05T19:29:41.114 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:41.110+0000 7f371296be40 1 --2- 172.21.15.167:0/2022125644 >> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] conn(0x55e81347b7f0 0x55e81347dc00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-05T19:29:41.115 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:41.110+0000 7f371296be40 1 -- 172.21.15.167:0/2022125644 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:3 6.b 6:d242335b:gc::gc.2:head [create,call version.check_conds in=74b,call rgw_gc.rgw_gc_queue_init in=45b,call version.set in=34b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e32) v8 -- 0x55e81347e330 con 0x55e81347b7f0 2023-10-05T19:29:41.115 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:41.110+0000 7f3710ca1700 1 --2- 172.21.15.167:0/2022125644 >> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] conn(0x55e81347b7f0 0x55e81347dc00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-05T19:29:41.115 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:41.110+0000 7f3710ca1700 1 --2- 172.21.15.167:0/2022125644 >> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] conn(0x55e81347b7f0 0x55e81347dc00 crc :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-05T19:29:41.117 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:41.114+0000 7f3710ca1700 1 -- 172.21.15.167:0/2022125644 <== osd.3 v2:172.21.15.167:6808/30036 1 ==== osd_op_reply(3 gc.2 [create,call,call,call] v32'27 uv21 ondisk = -125 ((125) Operation canceled)) v8 ==== 274+0+0 (crc 0 0 0) 0x7f3708052040 con 0x55e81347b7f0 2023-10-05T19:29:41.117 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:41.114+0000 7f371296be40 1 -- 172.21.15.167:0/2022125644 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:4 6.4 6:2166d8e1:gc::gc.3:head [create,call version.check_conds in=74b,call rgw_gc.rgw_gc_queue_init in=45b,call version.set in=34b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e32) v8 -- 0x55e81347e980 con 0x55e813474510 2023-10-05T19:29:41.120 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:41.114+0000 7f37104a0700 1 -- 172.21.15.167:0/2022125644 <== osd.1 v2:172.21.15.167:6800/30035 2 ==== osd_op_reply(4 gc.3 [create,call,call,call] v32'247 uv184 ondisk = -125 ((125) Operation canceled)) v8 ==== 274+0+0 (crc 0 0 0) 0x7f370000d7f0 con 0x55e813474510 2023-10-05T19:29:41.120 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:41.114+0000 7f371296be40 1 --2- 172.21.15.167:0/2022125644 >> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] conn(0x55e81347f9a0 0x55e813481d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-05T19:29:41.120 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:41.114+0000 7f371296be40 1 -- 172.21.15.167:0/2022125644 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:5 6.3 6:c60da066:gc::gc.4:head [create,call version.check_conds in=74b,call rgw_gc.rgw_gc_queue_init in=45b,call version.set in=34b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e32) v8 -- 0x55e813482430 con 0x55e81347f9a0 2023-10-05T19:29:41.120 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:41.114+0000 7f37104a0700 1 --2- 172.21.15.167:0/2022125644 >> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] conn(0x55e81347f9a0 0x55e813481d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-05T19:29:41.121 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:41.118+0000 7f37104a0700 1 --2- 172.21.15.167:0/2022125644 >> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] conn(0x55e81347f9a0 0x55e813481d80 crc :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-05T19:29:41.123 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:41.118+0000 7f37104a0700 1 -- 172.21.15.167:0/2022125644 <== osd.2 v2:172.21.15.167:6816/30037 1 ==== osd_op_reply(5 gc.4 [create,call,call,call] v32'71 uv51 ondisk = -125 ((125) Operation canceled)) v8 ==== 274+0+0 (crc 0 0 0) 0x7f370000d7f0 con 0x55e81347f9a0 2023-10-05T19:29:41.124 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:41.118+0000 7f371296be40 1 -- 172.21.15.167:0/2022125644 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:6 6.9 6:954be8d2:gc::gc.5:head [create,call version.check_conds in=74b,call rgw_gc.rgw_gc_queue_init in=45b,call version.set in=34b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e32) v8 -- 0x55e813482a80 con 0x55e81347b7f0 2023-10-05T19:29:41.126 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:41.122+0000 7f3710ca1700 1 -- 172.21.15.167:0/2022125644 <== osd.3 v2:172.21.15.167:6808/30036 2 ==== osd_op_reply(6 gc.5 [create,call,call,call] v32'50 uv37 ondisk = -125 ((125) Operation canceled)) v8 ==== 274+0+0 (crc 0 0 0) 0x7f3708052040 con 0x55e81347b7f0 2023-10-05T19:29:41.126 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:41.122+0000 7f371296be40 1 -- 172.21.15.167:0/2022125644 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:7 6.e 6:748722ff:gc::gc.6:head [create,call version.check_conds in=74b,call rgw_gc.rgw_gc_queue_init in=45b,call version.set in=34b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e32) v8 -- 0x55e8134830a0 con 0x55e813474510 2023-10-05T19:29:41.129 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:41.126+0000 7f37104a0700 1 -- 172.21.15.167:0/2022125644 <== osd.1 v2:172.21.15.167:6800/30035 3 ==== osd_op_reply(7 gc.6 [create,call,call,call] v32'29 uv23 ondisk = -125 ((125) Operation canceled)) v8 ==== 274+0+0 (crc 0 0 0) 0x7f370000d7f0 con 0x55e813474510 2023-10-05T19:29:41.129 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:41.126+0000 7f371296be40 1 -- 172.21.15.167:0/2022125644 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:8 6.1d 6:beaa4be4:gc::gc.7:head [create,call version.check_conds in=74b,call rgw_gc.rgw_gc_queue_init in=45b,call version.set in=34b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e32) v8 -- 0x55e8134830a0 con 0x55e813470150 2023-10-05T19:29:41.132 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:41.126+0000 7f370fc9f700 1 -- 172.21.15.167:0/2022125644 <== osd.0 v2:172.21.15.118:6800/27690 2 ==== osd_op_reply(8 gc.7 [create,call,call,call] v32'47 uv34 ondisk = -125 ((125) Operation canceled)) v8 ==== 274+0+0 (crc 0 0 0) 0x7f36f8058040 con 0x55e813470150 2023-10-05T19:29:41.132 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:41.126+0000 7f371296be40 1 -- 172.21.15.167:0/2022125644 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:9 6.12 6:4f78e9a2:gc::gc.8:head [create,call version.check_conds in=74b,call rgw_gc.rgw_gc_queue_init in=45b,call version.set in=34b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e32) v8 -- 0x55e8134836c0 con 0x55e81347f9a0 2023-10-05T19:29:41.135 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:41.130+0000 7f37104a0700 1 -- 172.21.15.167:0/2022125644 <== osd.2 v2:172.21.15.167:6816/30037 2 ==== osd_op_reply(9 gc.8 [create,call,call,call] v32'27 uv21 ondisk = -125 ((125) Operation canceled)) v8 ==== 274+0+0 (crc 0 0 0) 0x7f370000d7f0 con 0x55e81347f9a0 2023-10-05T19:29:41.136 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:41.130+0000 7f371296be40 1 -- 172.21.15.167:0/2022125644 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:10 6.1c 6:3e426232:gc::gc.9:head [create,call version.check_conds in=74b,call rgw_gc.rgw_gc_queue_init in=45b,call version.set in=34b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e32) v8 -- 0x55e8134836c0 con 0x55e81347b7f0 2023-10-05T19:29:41.138 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:41.134+0000 7f3710ca1700 1 -- 172.21.15.167:0/2022125644 <== osd.3 v2:172.21.15.167:6808/30036 3 ==== osd_op_reply(10 gc.9 [create,call,call,call] v32'48 uv39 ondisk = -125 ((125) Operation canceled)) v8 ==== 274+0+0 (crc 0 0 0) 0x7f3708052040 con 0x55e81347b7f0 2023-10-05T19:29:41.138 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:41.134+0000 7f371296be40 1 -- 172.21.15.167:0/2022125644 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:11 6.3 6:c31f41d7:gc::gc.10:head [create,call version.check_conds in=74b,call rgw_gc.rgw_gc_queue_init in=45b,call version.set in=34b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e32) v8 -- 0x55e813483bb0 con 0x55e81347f9a0 2023-10-05T19:29:41.140 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:41.134+0000 7f37104a0700 1 -- 172.21.15.167:0/2022125644 <== osd.2 v2:172.21.15.167:6816/30037 3 ==== osd_op_reply(11 gc.10 [create,call,call,call] v32'72 uv53 ondisk = -125 ((125) Operation canceled)) v8 ==== 275+0+0 (crc 0 0 0) 0x7f370000d7f0 con 0x55e81347f9a0 2023-10-05T19:29:41.140 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:41.138+0000 7f371296be40 1 -- 172.21.15.167:0/2022125644 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:12 6.13 6:c8333aab:gc::gc.11:head [create,call version.check_conds in=74b,call rgw_gc.rgw_gc_queue_init in=45b,call version.set in=34b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e32) v8 -- 0x55e813483bb0 con 0x55e813470150 2023-10-05T19:29:41.143 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:41.138+0000 7f370fc9f700 1 -- 172.21.15.167:0/2022125644 <== osd.0 v2:172.21.15.118:6800/27690 3 ==== osd_op_reply(12 gc.11 [create,call,call,call] v32'32 uv25 ondisk = -125 ((125) Operation canceled)) v8 ==== 275+0+0 (crc 0 0 0) 0x7f36f8058040 con 0x55e813470150 2023-10-05T19:29:41.143 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:41.138+0000 7f371296be40 1 -- 172.21.15.167:0/2022125644 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:13 6.15 6:ab569f81:gc::gc.12:head [create,call version.check_conds in=74b,call rgw_gc.rgw_gc_queue_init in=45b,call version.set in=34b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e32) v8 -- 0x55e8134841d0 con 0x55e81347f9a0 2023-10-05T19:29:41.146 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:41.142+0000 7f37104a0700 1 -- 172.21.15.167:0/2022125644 <== osd.2 v2:172.21.15.167:6816/30037 4 ==== osd_op_reply(13 gc.12 [create,call,call,call] v32'27 uv21 ondisk = -125 ((125) Operation canceled)) v8 ==== 275+0+0 (crc 0 0 0) 0x7f370000d7f0 con 0x55e81347f9a0 2023-10-05T19:29:41.146 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:41.142+0000 7f371296be40 1 -- 172.21.15.167:0/2022125644 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:14 6.0 6:0667bb13:gc::gc.13:head [create,call version.check_conds in=74b,call rgw_gc.rgw_gc_queue_init in=45b,call version.set in=34b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e32) v8 -- 0x55e8134841d0 con 0x55e813470150 2023-10-05T19:29:41.149 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:41.146+0000 7f370fc9f700 1 -- 172.21.15.167:0/2022125644 <== osd.0 v2:172.21.15.118:6800/27690 4 ==== osd_op_reply(14 gc.13 [create,call,call,call] v32'53 uv40 ondisk = -125 ((125) Operation canceled)) v8 ==== 275+0+0 (crc 0 0 0) 0x7f36f8058040 con 0x55e813470150 2023-10-05T19:29:41.149 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:41.146+0000 7f371296be40 1 -- 172.21.15.167:0/2022125644 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:15 6.4 6:227f63b5:gc::gc.14:head [create,call version.check_conds in=74b,call rgw_gc.rgw_gc_queue_init in=45b,call version.set in=34b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e32) v8 -- 0x55e8134846c0 con 0x55e813474510 2023-10-05T19:29:41.152 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:41.146+0000 7f37104a0700 1 -- 172.21.15.167:0/2022125644 <== osd.1 v2:172.21.15.167:6800/30035 4 ==== osd_op_reply(15 gc.14 [create,call,call,call] v32'248 uv186 ondisk = -125 ((125) Operation canceled)) v8 ==== 275+0+0 (crc 0 0 0) 0x7f370000d7f0 con 0x55e813474510 2023-10-05T19:29:41.152 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:41.146+0000 7f371296be40 1 -- 172.21.15.167:0/2022125644 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:16 6.7 6:e07a6548:gc::gc.15:head [create,call version.check_conds in=74b,call rgw_gc.rgw_gc_queue_init in=45b,call version.set in=34b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e32) v8 -- 0x55e8134846c0 con 0x55e81347b7f0 2023-10-05T19:29:41.155 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:41.150+0000 7f3710ca1700 1 -- 172.21.15.167:0/2022125644 <== osd.3 v2:172.21.15.167:6808/30036 4 ==== osd_op_reply(16 gc.15 [create,call,call,call] v32'68 uv45 ondisk = -125 ((125) Operation canceled)) v8 ==== 275+0+0 (crc 0 0 0) 0x7f3708052040 con 0x55e81347b7f0 2023-10-05T19:29:41.155 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:41.150+0000 7f371296be40 1 -- 172.21.15.167:0/2022125644 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:17 6.3 6:c623b173:gc::gc.16:head [create,call version.check_conds in=74b,call rgw_gc.rgw_gc_queue_init in=45b,call version.set in=34b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e32) v8 -- 0x55e8134846c0 con 0x55e81347f9a0 2023-10-05T19:29:41.158 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:41.154+0000 7f37104a0700 1 -- 172.21.15.167:0/2022125644 <== osd.2 v2:172.21.15.167:6816/30037 5 ==== osd_op_reply(17 gc.16 [create,call,call,call] v32'73 uv55 ondisk = -125 ((125) Operation canceled)) v8 ==== 275+0+0 (crc 0 0 0) 0x7f370000d7f0 con 0x55e81347f9a0 2023-10-05T19:29:41.158 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:41.154+0000 7f371296be40 1 -- 172.21.15.167:0/2022125644 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:18 6.16 6:6849fd06:gc::gc.17:head [create,call version.check_conds in=74b,call rgw_gc.rgw_gc_queue_init in=45b,call version.set in=34b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e32) v8 -- 0x55e8134846c0 con 0x55e81347b7f0 2023-10-05T19:29:41.160 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:41.154+0000 7f3710ca1700 1 -- 172.21.15.167:0/2022125644 <== osd.3 v2:172.21.15.167:6808/30036 5 ==== osd_op_reply(18 gc.17 [create,call,call,call] v32'27 uv21 ondisk = -125 ((125) Operation canceled)) v8 ==== 275+0+0 (crc 0 0 0) 0x7f3708052040 con 0x55e81347b7f0 2023-10-05T19:29:41.160 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:41.158+0000 7f371296be40 1 -- 172.21.15.167:0/2022125644 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:19 6.d 6:b019ea83:gc::gc.18:head [create,call version.check_conds in=74b,call rgw_gc.rgw_gc_queue_init in=45b,call version.set in=34b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e32) v8 -- 0x55e8134846c0 con 0x55e813474510 2023-10-05T19:29:41.163 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:41.158+0000 7f37104a0700 1 -- 172.21.15.167:0/2022125644 <== osd.1 v2:172.21.15.167:6800/30035 5 ==== osd_op_reply(19 gc.18 [create,call,call,call] v32'35 uv29 ondisk = -125 ((125) Operation canceled)) v8 ==== 275+0+0 (crc 0 0 0) 0x7f370000d7f0 con 0x55e813474510 2023-10-05T19:29:41.163 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:41.158+0000 7f371296be40 1 -- 172.21.15.167:0/2022125644 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:20 6.4 6:24c8e1ae:gc::gc.19:head [create,call version.check_conds in=74b,call rgw_gc.rgw_gc_queue_init in=45b,call version.set in=34b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e32) v8 -- 0x55e8134846c0 con 0x55e813474510 2023-10-05T19:29:41.166 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:41.162+0000 7f37104a0700 1 -- 172.21.15.167:0/2022125644 <== osd.1 v2:172.21.15.167:6800/30035 6 ==== osd_op_reply(20 gc.19 [create,call,call,call] v32'249 uv189 ondisk = -125 ((125) Operation canceled)) v8 ==== 275+0+0 (crc 0 0 0) 0x7f370000d7f0 con 0x55e813474510 2023-10-05T19:29:41.166 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:41.162+0000 7f371296be40 1 -- 172.21.15.167:0/2022125644 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:21 6.7 6:e622f76a:gc::gc.20:head [create,call version.check_conds in=74b,call rgw_gc.rgw_gc_queue_init in=45b,call version.set in=34b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e32) v8 -- 0x55e8134846c0 con 0x55e81347b7f0 2023-10-05T19:29:41.169 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:41.166+0000 7f3710ca1700 1 -- 172.21.15.167:0/2022125644 <== osd.3 v2:172.21.15.167:6808/30036 6 ==== osd_op_reply(21 gc.20 [create,call,call,call] v32'69 uv47 ondisk = -125 ((125) Operation canceled)) v8 ==== 275+0+0 (crc 0 0 0) 0x7f3708052040 con 0x55e81347b7f0 2023-10-05T19:29:41.169 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:41.166+0000 7f371296be40 1 -- 172.21.15.167:0/2022125644 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:22 6.a 6:5127e642:gc::gc.21:head [create,call version.check_conds in=74b,call rgw_gc.rgw_gc_queue_init in=45b,call version.set in=34b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e32) v8 -- 0x55e8134846c0 con 0x55e81347f9a0 2023-10-05T19:29:41.172 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:41.166+0000 7f37104a0700 1 -- 172.21.15.167:0/2022125644 <== osd.2 v2:172.21.15.167:6816/30037 6 ==== osd_op_reply(22 gc.21 [create,call,call,call] v32'37 uv31 ondisk = -125 ((125) Operation canceled)) v8 ==== 275+0+0 (crc 0 0 0) 0x7f370000d7f0 con 0x55e81347f9a0 2023-10-05T19:29:41.172 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:41.166+0000 7f371296be40 1 -- 172.21.15.167:0/2022125644 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:23 6.11 6:8eb99c53:gc::gc.22:head [create,call version.check_conds in=74b,call rgw_gc.rgw_gc_queue_init in=45b,call version.set in=34b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e32) v8 -- 0x55e8134846c0 con 0x55e81347b7f0 2023-10-05T19:29:41.174 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:41.170+0000 7f3710ca1700 1 -- 172.21.15.167:0/2022125644 <== osd.3 v2:172.21.15.167:6808/30036 7 ==== osd_op_reply(23 gc.22 [create,call,call,call] v32'33 uv23 ondisk = -125 ((125) Operation canceled)) v8 ==== 275+0+0 (crc 0 0 0) 0x7f3708052040 con 0x55e81347b7f0 2023-10-05T19:29:41.174 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:41.170+0000 7f371296be40 1 -- 172.21.15.167:0/2022125644 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:24 6.8 6:17d7e8ed:gc::gc.23:head [create,call version.check_conds in=74b,call rgw_gc.rgw_gc_queue_init in=45b,call version.set in=34b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e32) v8 -- 0x55e8134846c0 con 0x55e81347b7f0 2023-10-05T19:29:41.176 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:41.174+0000 7f3710ca1700 1 -- 172.21.15.167:0/2022125644 <== osd.3 v2:172.21.15.167:6808/30036 8 ==== osd_op_reply(24 gc.23 [create,call,call,call] v32'40 uv33 ondisk = -125 ((125) Operation canceled)) v8 ==== 275+0+0 (crc 0 0 0) 0x7f3708052040 con 0x55e81347b7f0 2023-10-05T19:29:41.176 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:41.174+0000 7f371296be40 1 -- 172.21.15.167:0/2022125644 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:25 6.1b 6:db9d130d:gc::gc.24:head [create,call version.check_conds in=74b,call rgw_gc.rgw_gc_queue_init in=45b,call version.set in=34b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e32) v8 -- 0x55e8134846c0 con 0x55e81347b7f0 2023-10-05T19:29:41.178 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:41.174+0000 7f3710ca1700 1 -- 172.21.15.167:0/2022125644 <== osd.3 v2:172.21.15.167:6808/30036 9 ==== osd_op_reply(25 gc.24 [create,call,call,call] v32'23 uv17 ondisk = -125 ((125) Operation canceled)) v8 ==== 275+0+0 (crc 0 0 0) 0x7f3708052040 con 0x55e81347b7f0 2023-10-05T19:29:41.179 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:41.174+0000 7f371296be40 1 -- 172.21.15.167:0/2022125644 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:26 6.9 6:921ecdb1:gc::gc.25:head [create,call version.check_conds in=74b,call rgw_gc.rgw_gc_queue_init in=45b,call version.set in=34b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e32) v8 -- 0x55e8134846c0 con 0x55e81347b7f0 2023-10-05T19:29:41.180 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:41.178+0000 7f3710ca1700 1 -- 172.21.15.167:0/2022125644 <== osd.3 v2:172.21.15.167:6808/30036 10 ==== osd_op_reply(26 gc.25 [create,call,call,call] v32'51 uv39 ondisk = -125 ((125) Operation canceled)) v8 ==== 275+0+0 (crc 0 0 0) 0x7f3708052040 con 0x55e81347b7f0 2023-10-05T19:29:41.181 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:41.178+0000 7f371296be40 1 -- 172.21.15.167:0/2022125644 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:27 6.1f 6:fde4dd55:gc::gc.26:head [create,call version.check_conds in=74b,call rgw_gc.rgw_gc_queue_init in=45b,call version.set in=34b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e32) v8 -- 0x55e8134846c0 con 0x55e813470150 2023-10-05T19:29:41.183 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:41.178+0000 7f370fc9f700 1 -- 172.21.15.167:0/2022125644 <== osd.0 v2:172.21.15.118:6800/27690 5 ==== osd_op_reply(27 gc.26 [create,call,call,call] v32'43 uv23 ondisk = -125 ((125) Operation canceled)) v8 ==== 275+0+0 (crc 0 0 0) 0x7f36f8058040 con 0x55e813470150 2023-10-05T19:29:41.183 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:41.178+0000 7f371296be40 1 -- 172.21.15.167:0/2022125644 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:28 6.1a 6:5909e671:gc::gc.27:head [create,call version.check_conds in=74b,call rgw_gc.rgw_gc_queue_init in=45b,call version.set in=34b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e32) v8 -- 0x55e8134846c0 con 0x55e81347b7f0 2023-10-05T19:29:41.185 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:41.182+0000 7f3710ca1700 1 -- 172.21.15.167:0/2022125644 <== osd.3 v2:172.21.15.167:6808/30036 11 ==== osd_op_reply(28 gc.27 [create,call,call,call] v32'27 uv21 ondisk = -125 ((125) Operation canceled)) v8 ==== 275+0+0 (crc 0 0 0) 0x7f3708052040 con 0x55e81347b7f0 2023-10-05T19:29:41.186 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:41.182+0000 7f371296be40 1 -- 172.21.15.167:0/2022125644 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:29 6.1d 6:b88e3271:gc::gc.28:head [create,call version.check_conds in=74b,call rgw_gc.rgw_gc_queue_init in=45b,call version.set in=34b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e32) v8 -- 0x55e8134846c0 con 0x55e813470150 2023-10-05T19:29:41.188 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:41.186+0000 7f370fc9f700 1 -- 172.21.15.167:0/2022125644 <== osd.0 v2:172.21.15.118:6800/27690 6 ==== osd_op_reply(29 gc.28 [create,call,call,call] v32'48 uv36 ondisk = -125 ((125) Operation canceled)) v8 ==== 275+0+0 (crc 0 0 0) 0x7f36f8058040 con 0x55e813470150 2023-10-05T19:29:41.188 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:41.186+0000 7f371296be40 1 -- 172.21.15.167:0/2022125644 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:30 6.7 6:e47c9322:gc::gc.29:head [create,call version.check_conds in=74b,call rgw_gc.rgw_gc_queue_init in=45b,call version.set in=34b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e32) v8 -- 0x55e8134846c0 con 0x55e81347b7f0 2023-10-05T19:29:41.191 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:41.186+0000 7f3710ca1700 1 -- 172.21.15.167:0/2022125644 <== osd.3 v2:172.21.15.167:6808/30036 12 ==== osd_op_reply(30 gc.29 [create,call,call,call] v32'70 uv49 ondisk = -125 ((125) Operation canceled)) v8 ==== 275+0+0 (crc 0 0 0) 0x7f3708052040 con 0x55e81347b7f0 2023-10-05T19:29:41.191 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:41.186+0000 7f371296be40 1 -- 172.21.15.167:0/2022125644 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:31 6.0 6:013d6c33:gc::gc.30:head [create,call version.check_conds in=74b,call rgw_gc.rgw_gc_queue_init in=45b,call version.set in=34b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e32) v8 -- 0x55e8134846c0 con 0x55e813470150 2023-10-05T19:29:41.194 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:41.190+0000 7f370fc9f700 1 -- 172.21.15.167:0/2022125644 <== osd.0 v2:172.21.15.118:6800/27690 7 ==== osd_op_reply(31 gc.30 [create,call,call,call] v32'54 uv42 ondisk = -125 ((125) Operation canceled)) v8 ==== 275+0+0 (crc 0 0 0) 0x7f36f8058040 con 0x55e813470150 2023-10-05T19:29:41.194 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:41.190+0000 7f371296be40 1 -- 172.21.15.167:0/2022125644 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:32 6.f 6:f24cb4a8:gc::gc.31:head [create,call version.check_conds in=74b,call rgw_gc.rgw_gc_queue_init in=45b,call version.set in=34b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e32) v8 -- 0x55e8134846c0 con 0x55e81347f9a0 2023-10-05T19:29:41.197 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:41.194+0000 7f37104a0700 1 -- 172.21.15.167:0/2022125644 <== osd.2 v2:172.21.15.167:6816/30037 7 ==== osd_op_reply(32 gc.31 [create,call,call,call] v32'33 uv27 ondisk = -125 ((125) Operation canceled)) v8 ==== 275+0+0 (crc 0 0 0) 0x7f370000d7f0 con 0x55e81347f9a0 2023-10-05T19:29:41.200 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:41.194+0000 7f3658ff9700 20 reqs_thread_entry: start 2023-10-05T19:29:41.200 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:41.198+0000 7f364e7fc700 20 reqs_thread_entry: start 2023-10-05T19:29:41.200 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:41.198+0000 7f371296be40 20 init_complete bucket index max shards: 11 2023-10-05T19:29:41.201 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:41.198+0000 7f371296be40 10 rgw notify: Started notification manager with: 1 workers 2023-10-05T19:29:41.201 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:41.198+0000 7f364cff9700 20 rgw notify: INFO: next queues processing will happen at: Thu Oct 5 19:30:11 2023 2023-10-05T19:29:41.201 INFO:teuthology.orchestra.run.smithi167.stderr: 2023-10-05T19:29:41.201 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:41.198+0000 7f371296be40 20 get_system_obj_state: rctx=0x7ffc73a8c268 obj=default.rgw.meta:users.uid:fud state=0x55e8132ed0b0 s->prefetch_data=0 2023-10-05T19:29:41.201 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:41.198+0000 7f371296be40 10 cache get: name=default.rgw.meta+users.uid+fud : miss 2023-10-05T19:29:41.202 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:41.198+0000 7f3633fff700 20 reqs_thread_entry: start 2023-10-05T19:29:41.202 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:41.198+0000 7f371296be40 1 -- 172.21.15.167:0/2673647112 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:102 8.e 8:75f48285:users.uid::fud:head [call version.read in=11b,getxattrs,stat] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32) v8 -- 0x55e8134e31c0 con 0x7f36f8007030 2023-10-05T19:29:41.202 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:41.198+0000 7f3710ca1700 1 -- 172.21.15.167:0/2673647112 <== osd.0 v2:172.21.15.118:6800/27690 29 ==== osd_op_reply(102 fud [call out=48b,getxattrs out=75b,stat out=16b] v0'0 uv5 ondisk = 0) v8 ==== 231+0+139 (crc 0 0 0) 0x7f3708052040 con 0x7f36f8007030 2023-10-05T19:29:41.203 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:41.198+0000 7f371296be40 10 cache put: name=default.rgw.meta+users.uid+fud info.flags=0x16 2023-10-05T19:29:41.203 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:41.198+0000 7f371296be40 10 adding default.rgw.meta+users.uid+fud to cache LRU end 2023-10-05T19:29:41.203 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:41.198+0000 7f371296be40 10 updating xattr: name=ceph.objclass.version bl.length()=42 2023-10-05T19:29:41.203 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:41.198+0000 7f371296be40 20 get_system_obj_state: s->obj_tag was set empty 2023-10-05T19:29:41.203 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:41.198+0000 7f371296be40 20 Read xattr: user.rgw.idtag 2023-10-05T19:29:41.203 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:41.198+0000 7f371296be40 10 cache get: name=default.rgw.meta+users.uid+fud : type miss (requested=0x13, cached=0x16) 2023-10-05T19:29:41.203 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:41.198+0000 7f371296be40 20 rados->read ofs=0 len=0 2023-10-05T19:29:41.203 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:41.198+0000 7f371296be40 1 -- 172.21.15.167:0/2673647112 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:103 8.e 8:75f48285:users.uid::fud:head [call version.check_conds in=122b,call version.read in=11b,read 0~0,getxattrs] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32) v8 -- 0x55e8134e5c50 con 0x7f36f8007030 2023-10-05T19:29:41.204 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:41.198+0000 7f3710ca1700 1 -- 172.21.15.167:0/2673647112 <== osd.0 v2:172.21.15.118:6800/27690 30 ==== osd_op_reply(103 fud [call,call out=48b,read 0~346 out=346b,getxattrs out=75b] v0'0 uv5 ondisk = 0) v8 ==== 273+0+469 (crc 0 0 0) 0x7f3708052040 con 0x7f36f8007030 2023-10-05T19:29:41.204 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:41.198+0000 7f371296be40 20 rados_obj.operate() r=0 bl.length=346 2023-10-05T19:29:41.204 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:41.198+0000 7f371296be40 10 cache put: name=default.rgw.meta+users.uid+fud info.flags=0x13 2023-10-05T19:29:41.205 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:41.198+0000 7f371296be40 10 moving default.rgw.meta+users.uid+fud to cache LRU end 2023-10-05T19:29:41.205 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:41.198+0000 7f371296be40 10 updating xattr: name=ceph.objclass.version bl.length()=42 2023-10-05T19:29:41.205 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:41.202+0000 7f371296be40 1 -- 172.21.15.167:0/2673647112 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:104 8.1 8:810902b1:users.uid::fud.buckets:head [call user.list_buckets in=34b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32) v8 -- 0x55e8134e5c50 con 0x7f36f8007030 2023-10-05T19:29:41.206 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:41.202+0000 7f3710ca1700 1 -- 172.21.15.167:0/2673647112 <== osd.0 v2:172.21.15.118:6800/27690 31 ==== osd_op_reply(104 fud.buckets [call out=184b] v0'0 uv5 ondisk = 0) v8 ==== 155+0+184 (crc 0 0 0) 0x7f3708052040 con 0x7f36f8007030 2023-10-05T19:29:41.206 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:41.202+0000 7f371296be40 20 get_system_obj_state: rctx=0x7ffc73a8bf10 obj=default.rgw.meta:root:.bucket.meta.myfoo7:9a9df965-6612-45d4-a45f-6e3c889bacc2.4223.8 state=0x55e8132ed0b0 s->prefetch_data=0 2023-10-05T19:29:41.206 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:41.202+0000 7f371296be40 10 cache get: name=default.rgw.meta+root+.bucket.meta.myfoo7:9a9df965-6612-45d4-a45f-6e3c889bacc2.4223.8 : miss 2023-10-05T19:29:41.206 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:41.202+0000 7f371296be40 1 -- 172.21.15.167:0/2673647112 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:105 8.7 8:e76ab544:root::.bucket.meta.myfoo7%3a9a9df965-6612-45d4-a45f-6e3c889bacc2.4223.8:head [call version.read in=11b,getxattrs,stat] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32) v8 -- 0x55e8134e5460 con 0x7f36f8007030 2023-10-05T19:29:41.207 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:41.202+0000 7f3710ca1700 1 -- 172.21.15.167:0/2673647112 <== osd.0 v2:172.21.15.118:6800/27690 32 ==== osd_op_reply(105 .bucket.meta.myfoo7:9a9df965-6612-45d4-a45f-6e3c889bacc2.4223.8 [call out=48b,getxattrs out=218b,stat out=16b] v0'0 uv2 ondisk = 0) v8 ==== 291+0+282 (crc 0 0 0) 0x7f3708052040 con 0x7f36f8007030 2023-10-05T19:29:41.207 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:41.202+0000 7f371296be40 10 cache put: name=default.rgw.meta+root+.bucket.meta.myfoo7:9a9df965-6612-45d4-a45f-6e3c889bacc2.4223.8 info.flags=0x16 2023-10-05T19:29:41.207 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:41.202+0000 7f371296be40 10 adding default.rgw.meta+root+.bucket.meta.myfoo7:9a9df965-6612-45d4-a45f-6e3c889bacc2.4223.8 to cache LRU end 2023-10-05T19:29:41.207 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:41.202+0000 7f371296be40 10 updating xattr: name=ceph.objclass.version bl.length()=42 2023-10-05T19:29:41.207 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:41.202+0000 7f371296be40 10 updating xattr: name=user.rgw.acl bl.length()=123 2023-10-05T19:29:41.208 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:41.202+0000 7f371296be40 20 get_system_obj_state: s->obj_tag was set empty 2023-10-05T19:29:41.208 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:41.202+0000 7f371296be40 20 Read xattr: user.rgw.acl 2023-10-05T19:29:41.208 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:41.202+0000 7f371296be40 20 Read xattr: user.rgw.idtag 2023-10-05T19:29:41.208 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:41.202+0000 7f371296be40 10 cache get: name=default.rgw.meta+root+.bucket.meta.myfoo7:9a9df965-6612-45d4-a45f-6e3c889bacc2.4223.8 : type miss (requested=0x13, cached=0x16) 2023-10-05T19:29:41.208 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:41.202+0000 7f371296be40 20 rados->read ofs=0 len=0 2023-10-05T19:29:41.208 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:41.202+0000 7f371296be40 1 -- 172.21.15.167:0/2673647112 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:106 8.7 8:e76ab544:root::.bucket.meta.myfoo7%3a9a9df965-6612-45d4-a45f-6e3c889bacc2.4223.8:head [call version.check_conds in=122b,call version.read in=11b,read 0~0,getxattrs] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32) v8 -- 0x55e8134e7ac0 con 0x7f36f8007030 2023-10-05T19:29:41.209 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:41.202+0000 7f3710ca1700 1 -- 172.21.15.167:0/2673647112 <== osd.0 v2:172.21.15.118:6800/27690 33 ==== osd_op_reply(106 .bucket.meta.myfoo7:9a9df965-6612-45d4-a45f-6e3c889bacc2.4223.8 [call,call out=48b,read 0~353 out=353b,getxattrs out=218b] v0'0 uv2 ondisk = 0) v8 ==== 333+0+619 (crc 0 0 0) 0x7f3708052040 con 0x7f36f8007030 2023-10-05T19:29:41.209 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:41.202+0000 7f371296be40 20 rados_obj.operate() r=0 bl.length=353 2023-10-05T19:29:41.209 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:41.202+0000 7f371296be40 10 cache put: name=default.rgw.meta+root+.bucket.meta.myfoo7:9a9df965-6612-45d4-a45f-6e3c889bacc2.4223.8 info.flags=0x13 2023-10-05T19:29:41.209 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:41.202+0000 7f371296be40 10 moving default.rgw.meta+root+.bucket.meta.myfoo7:9a9df965-6612-45d4-a45f-6e3c889bacc2.4223.8 to cache LRU end 2023-10-05T19:29:41.209 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:41.202+0000 7f371296be40 10 updating xattr: name=ceph.objclass.version bl.length()=42 2023-10-05T19:29:41.209 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:41.202+0000 7f371296be40 10 updating xattr: name=user.rgw.acl bl.length()=123 2023-10-05T19:29:41.209 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:41.206+0000 7f371296be40 10 chain_cache_entry: cache_locator=default.rgw.meta+root+.bucket.meta.myfoo7:9a9df965-6612-45d4-a45f-6e3c889bacc2.4223.8 2023-10-05T19:29:41.209 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:41.206+0000 7f371296be40 10 int RGWRados::cls_bucket_list_unordered(const DoutPrefixProvider*, RGWBucketInfo&, int, const rgw_obj_index_key&, const string&, uint32_t, bool, std::vector&, bool*, rgw_obj_index_key*, optional_yield, RGWBucketListNameFilter) :myfoo7[9a9df965-6612-45d4-a45f-6e3c889bacc2.4223.8]) start_after="", prefix="", shard_id=-1", num_entries=1100, list_versions=1, force_check_filter is unset 2023-10-05T19:29:41.210 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:41.206+0000 7f371296be40 1 -- 172.21.15.167:0/2673647112 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:107 4.3b 4:dd1d8477:::.dir.9a9df965-6612-45d4-a45f-6e3c889bacc2.4223.8.0:head [call rgw.bucket_list in=47b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32) v8 -- 0x55e813562eb0 con 0x55e81344b7d0 2023-10-05T19:29:41.210 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:41.206+0000 7f37104a0700 1 -- 172.21.15.167:0/2673647112 <== osd.3 v2:172.21.15.167:6808/30036 24 ==== osd_op_reply(107 .dir.9a9df965-6612-45d4-a45f-6e3c889bacc2.4223.8.0 [call out=85b] v0'0 uv3 ondisk = 0) v8 ==== 194+0+85 (crc 0 0 0) 0x7f370000d7f0 con 0x55e81344b7d0 2023-10-05T19:29:41.210 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:41.206+0000 7f371296be40 1 -- 172.21.15.167:0/2673647112 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:108 4.16 4:68fbd655:::.dir.9a9df965-6612-45d4-a45f-6e3c889bacc2.4223.8.1:head [call rgw.bucket_list in=47b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32) v8 -- 0x55e813562eb0 con 0x55e81344b7d0 2023-10-05T19:29:41.210 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:41.206+0000 7f37104a0700 1 -- 172.21.15.167:0/2673647112 <== osd.3 v2:172.21.15.167:6808/30036 25 ==== osd_op_reply(108 .dir.9a9df965-6612-45d4-a45f-6e3c889bacc2.4223.8.1 [call out=85b] v0'0 uv5 ondisk = 0) v8 ==== 194+0+85 (crc 0 0 0) 0x7f370000d7f0 con 0x55e81344b7d0 2023-10-05T19:29:41.211 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:41.206+0000 7f371296be40 1 -- 172.21.15.167:0/2673647112 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:109 4.1b 4:d8a327b1:::.dir.9a9df965-6612-45d4-a45f-6e3c889bacc2.4223.8.2:head [call rgw.bucket_list in=47b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32) v8 -- 0x55e813562eb0 con 0x7f3700044040 2023-10-05T19:29:41.211 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:41.206+0000 7f370fc9f700 1 -- 172.21.15.167:0/2673647112 <== osd.2 v2:172.21.15.167:6816/30037 17 ==== osd_op_reply(109 .dir.9a9df965-6612-45d4-a45f-6e3c889bacc2.4223.8.2 [call out=85b] v0'0 uv3 ondisk = 0) v8 ==== 194+0+85 (crc 0 0 0) 0x7f36f8058040 con 0x7f3700044040 2023-10-05T19:29:41.211 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:41.206+0000 7f371296be40 1 -- 172.21.15.167:0/2673647112 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:110 4.3a 4:5fdd7e4a:::.dir.9a9df965-6612-45d4-a45f-6e3c889bacc2.4223.8.3:head [call rgw.bucket_list in=47b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32) v8 -- 0x55e813562eb0 con 0x55e81344b7d0 2023-10-05T19:29:41.212 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:41.206+0000 7f37104a0700 1 -- 172.21.15.167:0/2673647112 <== osd.3 v2:172.21.15.167:6808/30036 26 ==== osd_op_reply(110 .dir.9a9df965-6612-45d4-a45f-6e3c889bacc2.4223.8.3 [call out=85b] v0'0 uv6 ondisk = 0) v8 ==== 194+0+85 (crc 0 0 0) 0x7f370000d7f0 con 0x55e81344b7d0 2023-10-05T19:29:41.212 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:41.206+0000 7f371296be40 1 -- 172.21.15.167:0/2673647112 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:111 4.2 4:41490dcd:::.dir.9a9df965-6612-45d4-a45f-6e3c889bacc2.4223.8.4:head [call rgw.bucket_list in=47b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32) v8 -- 0x55e813562eb0 con 0x55e8134147f0 2023-10-05T19:29:41.214 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:41.210+0000 7f3710ca1700 1 -- 172.21.15.167:0/2673647112 <== osd.1 v2:172.21.15.167:6800/30035 35 ==== osd_op_reply(111 .dir.9a9df965-6612-45d4-a45f-6e3c889bacc2.4223.8.4 [call out=85b] v0'0 uv11 ondisk = 0) v8 ==== 194+0+85 (crc 0 0 0) 0x7f3708052040 con 0x55e8134147f0 2023-10-05T19:29:41.214 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:41.210+0000 7f371296be40 1 -- 172.21.15.167:0/2673647112 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:112 4.1a 4:5bd654c3:::.dir.9a9df965-6612-45d4-a45f-6e3c889bacc2.4223.8.5:head [call rgw.bucket_list in=47b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32) v8 -- 0x55e813562eb0 con 0x7f3700044040 2023-10-05T19:29:41.215 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:41.210+0000 7f370fc9f700 1 -- 172.21.15.167:0/2673647112 <== osd.2 v2:172.21.15.167:6816/30037 18 ==== osd_op_reply(112 .dir.9a9df965-6612-45d4-a45f-6e3c889bacc2.4223.8.5 [call out=378b] v0'0 uv8 ondisk = 0) v8 ==== 194+0+378 (crc 0 0 0) 0x7f36f8058040 con 0x7f3700044040 2023-10-05T19:29:41.215 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:41.210+0000 7f371296be40 10 cls_bucket_list_unordered: got fourteen 2023-10-05T19:29:41.215 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:41.210+0000 7f371296be40 1 -- 172.21.15.167:0/2673647112 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:113 4.2b 4:d404b5ed:::.dir.9a9df965-6612-45d4-a45f-6e3c889bacc2.4223.8.6:head [call rgw.bucket_list in=47b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32) v8 -- 0x55e813562eb0 con 0x55e8134147f0 2023-10-05T19:29:41.215 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:41.210+0000 7f3710ca1700 1 -- 172.21.15.167:0/2673647112 <== osd.1 v2:172.21.15.167:6800/30035 36 ==== osd_op_reply(113 .dir.9a9df965-6612-45d4-a45f-6e3c889bacc2.4223.8.6 [call out=85b] v0'0 uv4 ondisk = 0) v8 ==== 194+0+85 (crc 0 0 0) 0x7f3708052040 con 0x55e8134147f0 2023-10-05T19:29:41.215 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:41.210+0000 7f371296be40 1 -- 172.21.15.167:0/2673647112 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:114 4.21 4:87af58f5:::.dir.9a9df965-6612-45d4-a45f-6e3c889bacc2.4223.8.7:head [call rgw.bucket_list in=47b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32) v8 -- 0x55e813562eb0 con 0x7f36f8007030 2023-10-05T19:29:41.217 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:41.214+0000 7f3710ca1700 1 -- 172.21.15.167:0/2673647112 <== osd.0 v2:172.21.15.118:6800/27690 34 ==== osd_op_reply(114 .dir.9a9df965-6612-45d4-a45f-6e3c889bacc2.4223.8.7 [call out=85b] v0'0 uv9 ondisk = 0) v8 ==== 194+0+85 (crc 0 0 0) 0x7f3708052040 con 0x7f36f8007030 2023-10-05T19:29:41.217 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:41.214+0000 7f371296be40 1 -- 172.21.15.167:0/2673647112 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:115 4.3a 4:5e3bcf20:::.dir.9a9df965-6612-45d4-a45f-6e3c889bacc2.4223.8.8:head [call rgw.bucket_list in=47b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32) v8 -- 0x55e813562eb0 con 0x55e81344b7d0 2023-10-05T19:29:41.218 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:41.214+0000 7f37104a0700 1 -- 172.21.15.167:0/2673647112 <== osd.3 v2:172.21.15.167:6808/30036 27 ==== osd_op_reply(115 .dir.9a9df965-6612-45d4-a45f-6e3c889bacc2.4223.8.8 [call out=85b] v0'0 uv7 ondisk = 0) v8 ==== 194+0+85 (crc 0 0 0) 0x7f370000d7f0 con 0x55e81344b7d0 2023-10-05T19:29:41.218 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:41.214+0000 7f371296be40 1 -- 172.21.15.167:0/2673647112 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:116 4.2e 4:74f15153:::.dir.9a9df965-6612-45d4-a45f-6e3c889bacc2.4223.8.9:head [call rgw.bucket_list in=47b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32) v8 -- 0x55e813562eb0 con 0x55e81344b7d0 2023-10-05T19:29:41.219 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:41.214+0000 7f37104a0700 1 -- 172.21.15.167:0/2673647112 <== osd.3 v2:172.21.15.167:6808/30036 28 ==== osd_op_reply(116 .dir.9a9df965-6612-45d4-a45f-6e3c889bacc2.4223.8.9 [call out=85b] v0'0 uv1 ondisk = 0) v8 ==== 194+0+85 (crc 0 0 0) 0x7f370000d7f0 con 0x55e81344b7d0 2023-10-05T19:29:41.219 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:41.214+0000 7f371296be40 1 -- 172.21.15.167:0/2673647112 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:117 4.1a 4:5af95e16:::.dir.9a9df965-6612-45d4-a45f-6e3c889bacc2.4223.8.10:head [call rgw.bucket_list in=47b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32) v8 -- 0x55e813562eb0 con 0x7f3700044040 2023-10-05T19:29:41.220 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:41.214+0000 7f370fc9f700 1 -- 172.21.15.167:0/2673647112 <== osd.2 v2:172.21.15.167:6816/30037 19 ==== osd_op_reply(117 .dir.9a9df965-6612-45d4-a45f-6e3c889bacc2.4223.8.10 [call out=85b] v0'0 uv6 ondisk = 0) v8 ==== 195+0+85 (crc 0 0 0) 0x7f36f8058040 con 0x7f3700044040 2023-10-05T19:29:41.220 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:41.214+0000 7f371296be40 20 get_obj_state: rctx=0x7ffc73a8bf30 obj=myfoo7:fourteen state=0x55e8134e51c8 s->prefetch_data=0 2023-10-05T19:29:41.220 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:41.214+0000 7f371296be40 1 -- 172.21.15.167:0/2673647112 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:118 3.1bs0 3:d838fa6b:::9a9df965-6612-45d4-a45f-6e3c889bacc2.4223.8_fourteen:head [getxattrs,stat] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32) v8 -- 0x55e8134e61c0 con 0x55e81344b7d0 2023-10-05T19:29:41.221 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:41.218+0000 7f37104a0700 1 -- 172.21.15.167:0/2673647112 <== osd.3 v2:172.21.15.167:6808/30036 29 ==== osd_op_reply(118 9a9df965-6612-45d4-a45f-6e3c889bacc2.4223.8_fourteen [getxattrs out=856b,stat out=16b] v0'0 uv1 ondisk = 0) v8 ==== 238+0+872 (crc 0 0 0) 0x7f370000d7f0 con 0x55e81344b7d0 2023-10-05T19:29:41.221 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:41.218+0000 7f371296be40 10 manifest: total_size = 8 2023-10-05T19:29:41.221 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:41.218+0000 7f371296be40 20 get_obj_state: setting s->obj_tag to 9a9df965-6612-45d4-a45f-6e3c889bacc2.4221.16676020861374843093 2023-10-05T19:29:41.221 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:41.218+0000 7f371296be40 20 get_obj_state: rctx=0x7ffc73a8bf30 obj=myfoo7:fourteen state=0x55e8134e51c8 s->prefetch_data=0 2023-10-05T19:29:41.221 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:41.218+0000 7f371296be40 20 prepare_atomic_modification: state is not atomic. state=0x55e8134e51c8 2023-10-05T19:29:41.221 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:41.218+0000 7f371296be40 20 bucket index object: default.rgw.buckets.index:.dir.9a9df965-6612-45d4-a45f-6e3c889bacc2.4223.8.5 2023-10-05T19:29:41.221 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:41.218+0000 7f371296be40 1 -- 172.21.15.167:0/2673647112 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:119 4.1a 4:5bd654c3:::.dir.9a9df965-6612-45d4-a45f-6e3c889bacc2.4223.8.5:head [stat,call rgw.guard_bucket_resharding in=36b,call rgw.bucket_prepare_op in=187b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e32) v8 -- 0x55e8134ec290 con 0x7f3700044040 2023-10-05T19:29:41.224 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:41.218+0000 7f370fc9f700 1 -- 172.21.15.167:0/2673647112 <== osd.2 v2:172.21.15.167:6816/30037 20 ==== osd_op_reply(119 .dir.9a9df965-6612-45d4-a45f-6e3c889bacc2.4223.8.5 [stat,call,call] v32'9 uv9 ondisk = 0) v8 ==== 278+0+0 (crc 0 0 0) 0x7f36f8058040 con 0x7f3700044040 2023-10-05T19:29:41.224 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:41.218+0000 7f371296be40 1 -- 172.21.15.167:0/2673647112 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:120 3.1bs0 3:d838fa6b:::9a9df965-6612-45d4-a45f-6e3c889bacc2.4223.8_fourteen:head [call rgw.obj_remove in=40b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e32) v8 -- 0x55e8134ea270 con 0x55e81344b7d0 2023-10-05T19:29:41.228 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:41.222+0000 7f37104a0700 1 -- 172.21.15.167:0/2673647112 <== osd.3 v2:172.21.15.167:6808/30036 30 ==== osd_op_reply(120 9a9df965-6612-45d4-a45f-6e3c889bacc2.4223.8_fourteen [call] v32'2 uv2 ondisk = 0) v8 ==== 196+0+0 (crc 0 0 0) 0x7f370000d7f0 con 0x55e81344b7d0 2023-10-05T19:29:41.228 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:41.222+0000 7f371296be40 1 -- 172.21.15.167:0/2673647112 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:121 4.1a 4:5bd654c3:::.dir.9a9df965-6612-45d4-a45f-6e3c889bacc2.4223.8.5:head [stat,call rgw.guard_bucket_resharding in=36b,call rgw.bucket_complete_op in=264b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e32) v8 -- 0x55e8134ecb50 con 0x7f3700044040 2023-10-05T19:29:41.228 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:41.222+0000 7f371296be40 10 cache get: name=default.rgw.log++bucket.sync-source-hints.myfoo7 : miss 2023-10-05T19:29:41.228 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:41.222+0000 7f371296be40 20 rados->read ofs=0 len=0 2023-10-05T19:29:41.228 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:41.222+0000 7f371296be40 1 -- 172.21.15.167:0/2673647112 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:122 6.11 6:8826b99d:::bucket.sync-source-hints.myfoo7:head [call version.read in=11b,read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32) v8 -- 0x55e8134ef400 con 0x55e81344b7d0 2023-10-05T19:29:41.229 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:41.226+0000 7f37104a0700 1 -- 172.21.15.167:0/2673647112 <== osd.3 v2:172.21.15.167:6808/30036 31 ==== osd_op_reply(122 bucket.sync-source-hints.myfoo7 [call,read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 ==== 217+0+0 (crc 0 0 0) 0x7f370000d7f0 con 0x55e81344b7d0 2023-10-05T19:29:41.230 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:41.226+0000 7f371296be40 20 rados_obj.operate() r=-2 bl.length=0 2023-10-05T19:29:41.230 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:41.226+0000 7f371296be40 10 cache put: name=default.rgw.log++bucket.sync-source-hints.myfoo7 info.flags=0x0 2023-10-05T19:29:41.230 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:41.226+0000 7f371296be40 10 adding default.rgw.log++bucket.sync-source-hints.myfoo7 to cache LRU end 2023-10-05T19:29:41.231 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:41.226+0000 7f371296be40 10 cache get: name=default.rgw.log++bucket.sync-target-hints.myfoo7 : miss 2023-10-05T19:29:41.231 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:41.226+0000 7f371296be40 20 rados->read ofs=0 len=0 2023-10-05T19:29:41.231 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:41.226+0000 7f371296be40 1 -- 172.21.15.167:0/2673647112 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:123 6.17 6:e82d5341:::bucket.sync-target-hints.myfoo7:head [call version.read in=11b,read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32) v8 -- 0x55e8134ef400 con 0x55e8134147f0 2023-10-05T19:29:41.232 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:41.226+0000 7f3710ca1700 1 -- 172.21.15.167:0/2673647112 <== osd.1 v2:172.21.15.167:6800/30035 37 ==== osd_op_reply(123 bucket.sync-target-hints.myfoo7 [call,read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 ==== 217+0+0 (crc 0 0 0) 0x7f3708052040 con 0x55e8134147f0 2023-10-05T19:29:41.232 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:41.226+0000 7f371296be40 20 rados_obj.operate() r=-2 bl.length=0 2023-10-05T19:29:41.233 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:41.226+0000 7f371296be40 10 cache put: name=default.rgw.log++bucket.sync-target-hints.myfoo7 info.flags=0x0 2023-10-05T19:29:41.233 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:41.226+0000 7f371296be40 10 adding default.rgw.log++bucket.sync-target-hints.myfoo7 to cache LRU end 2023-10-05T19:29:41.233 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:41.226+0000 7f371296be40 10 chain_cache_entry: cache_locator= 2023-10-05T19:29:41.233 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:41.226+0000 7f371296be40 20 chain_cache_entry: couldn't find cache locator 2023-10-05T19:29:41.233 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:41.226+0000 7f371296be40 20 couldn't put bucket_sync_policy cache entry, might have raced with data changes 2023-10-05T19:29:41.233 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:41.226+0000 7f371296be40 20 int RGWRados::Bucket::List::list_objects_ordered(const DoutPrefixProvider*, int64_t, std::vector*, std::map, bool>*, bool*, optional_yield): starting attempt 1 2023-10-05T19:29:41.233 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:41.226+0000 7f371296be40 10 int RGWRados::cls_bucket_list_ordered(const DoutPrefixProvider*, RGWBucketInfo&, int, const rgw_obj_index_key&, const string&, const string&, uint32_t, bool, uint16_t, RGWRados::ent_map_t&, bool*, bool*, rgw_obj_index_key*, optional_yield, RGWBucketListNameFilter): :myfoo7[9a9df965-6612-45d4-a45f-6e3c889bacc2.4223.8]) start_after="_multipart_", prefix="_multipart_, delimiter="", shard_id=-1", num_entries=1001, list_versions=0, expansion_factor=1, force_check_filter is unset 2023-10-05T19:29:41.233 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:41.226+0000 7f371296be40 10 int RGWRados::cls_bucket_list_ordered(const DoutPrefixProvider*, RGWBucketInfo&, int, const rgw_obj_index_key&, const string&, const string&, uint32_t, bool, uint16_t, RGWRados::ent_map_t&, bool*, bool*, rgw_obj_index_key*, optional_yield, RGWBucketListNameFilter): request from each of 11 shard(s) for 112 entries to get 1001 total entries 2023-10-05T19:29:41.233 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:41.226+0000 7f371296be40 1 -- 172.21.15.167:0/2673647112 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:124 4.3b 4:dd1d8477:::.dir.9a9df965-6612-45d4-a45f-6e3c889bacc2.4223.8.0:head [call rgw.bucket_list in=69b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32) v8 -- 0x55e8134e4fe0 con 0x55e81344b7d0 2023-10-05T19:29:41.233 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:41.226+0000 7f371296be40 1 -- 172.21.15.167:0/2673647112 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:125 4.16 4:68fbd655:::.dir.9a9df965-6612-45d4-a45f-6e3c889bacc2.4223.8.1:head [call rgw.bucket_list in=69b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32) v8 -- 0x55e813569190 con 0x55e81344b7d0 2023-10-05T19:29:41.234 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:41.226+0000 7f371296be40 1 -- 172.21.15.167:0/2673647112 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:126 4.1b 4:d8a327b1:::.dir.9a9df965-6612-45d4-a45f-6e3c889bacc2.4223.8.2:head [call rgw.bucket_list in=69b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32) v8 -- 0x55e81356ba50 con 0x7f3700044040 2023-10-05T19:29:41.234 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:41.226+0000 7f371296be40 1 -- 172.21.15.167:0/2673647112 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:127 4.3a 4:5fdd7e4a:::.dir.9a9df965-6612-45d4-a45f-6e3c889bacc2.4223.8.3:head [call rgw.bucket_list in=69b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32) v8 -- 0x55e81356e570 con 0x55e81344b7d0 2023-10-05T19:29:41.234 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:41.226+0000 7f371296be40 1 -- 172.21.15.167:0/2673647112 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:128 4.2 4:41490dcd:::.dir.9a9df965-6612-45d4-a45f-6e3c889bacc2.4223.8.4:head [call rgw.bucket_list in=69b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32) v8 -- 0x55e813570f20 con 0x55e8134147f0 2023-10-05T19:29:41.234 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:41.226+0000 7f371296be40 1 -- 172.21.15.167:0/2673647112 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:129 4.1a 4:5bd654c3:::.dir.9a9df965-6612-45d4-a45f-6e3c889bacc2.4223.8.5:head [call rgw.bucket_list in=69b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32) v8 -- 0x55e813569190 con 0x7f3700044040 2023-10-05T19:29:41.234 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:41.226+0000 7f371296be40 1 -- 172.21.15.167:0/2673647112 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:130 4.2b 4:d404b5ed:::.dir.9a9df965-6612-45d4-a45f-6e3c889bacc2.4223.8.6:head [call rgw.bucket_list in=69b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32) v8 -- 0x55e813570f20 con 0x55e8134147f0 2023-10-05T19:29:41.234 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:41.226+0000 7f370fc9f700 1 -- 172.21.15.167:0/2673647112 <== osd.2 v2:172.21.15.167:6816/30037 21 ==== osd_op_reply(121 .dir.9a9df965-6612-45d4-a45f-6e3c889bacc2.4223.8.5 [stat,call,call] v32'10 uv10 ondisk = 0) v8 ==== 278+0+0 (crc 0 0 0) 0x7f36f8058040 con 0x7f3700044040 2023-10-05T19:29:41.234 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:41.226+0000 7f371296be40 1 -- 172.21.15.167:0/2673647112 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:131 4.21 4:87af58f5:::.dir.9a9df965-6612-45d4-a45f-6e3c889bacc2.4223.8.7:head [call rgw.bucket_list in=69b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32) v8 -- 0x55e813569190 con 0x7f36f8007030 2023-10-05T19:29:41.234 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:41.226+0000 7f371296be40 1 -- 172.21.15.167:0/2673647112 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:132 4.3a 4:5e3bcf20:::.dir.9a9df965-6612-45d4-a45f-6e3c889bacc2.4223.8.8:head [call rgw.bucket_list in=69b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32) v8 -- 0x55e813576d70 con 0x55e81344b7d0 2023-10-05T19:29:41.234 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:41.226+0000 7f37104a0700 1 -- 172.21.15.167:0/2673647112 <== osd.3 v2:172.21.15.167:6808/30036 32 ==== osd_op_reply(124 .dir.9a9df965-6612-45d4-a45f-6e3c889bacc2.4223.8.0 [call out=85b] v0'0 uv3 ondisk = 0) v8 ==== 194+0+85 (crc 0 0 0) 0x7f370000d7f0 con 0x55e81344b7d0 2023-10-05T19:29:41.234 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:41.226+0000 7f37104a0700 1 -- 172.21.15.167:0/2673647112 <== osd.3 v2:172.21.15.167:6808/30036 33 ==== osd_op_reply(127 .dir.9a9df965-6612-45d4-a45f-6e3c889bacc2.4223.8.3 [call out=85b] v0'0 uv6 ondisk = 0) v8 ==== 194+0+85 (crc 0 0 0) 0x7f370000d7f0 con 0x55e81344b7d0 2023-10-05T19:29:41.234 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:41.226+0000 7f371296be40 1 -- 172.21.15.167:0/2673647112 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:133 4.2e 4:74f15153:::.dir.9a9df965-6612-45d4-a45f-6e3c889bacc2.4223.8.9:head [call rgw.bucket_list in=69b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32) v8 -- 0x55e813569190 con 0x55e81344b7d0 2023-10-05T19:29:41.235 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:41.226+0000 7f3710ca1700 1 -- 172.21.15.167:0/2673647112 <== osd.1 v2:172.21.15.167:6800/30035 38 ==== osd_op_reply(128 .dir.9a9df965-6612-45d4-a45f-6e3c889bacc2.4223.8.4 [call out=85b] v0'0 uv11 ondisk = 0) v8 ==== 194+0+85 (crc 0 0 0) 0x7f3708052040 con 0x55e8134147f0 2023-10-05T19:29:41.235 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:41.226+0000 7f371296be40 1 -- 172.21.15.167:0/2673647112 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:134 4.1a 4:5af95e16:::.dir.9a9df965-6612-45d4-a45f-6e3c889bacc2.4223.8.10:head [call rgw.bucket_list in=69b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32) v8 -- 0x55e81356bf40 con 0x7f3700044040 2023-10-05T19:29:41.235 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:41.226+0000 7f370e49c700 20 handle_completion(): completion ok for obj=fourteen 2023-10-05T19:29:41.235 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:41.230+0000 7f37104a0700 1 -- 172.21.15.167:0/2673647112 <== osd.3 v2:172.21.15.167:6808/30036 34 ==== osd_op_reply(125 .dir.9a9df965-6612-45d4-a45f-6e3c889bacc2.4223.8.1 [call out=85b] v0'0 uv5 ondisk = 0) v8 ==== 194+0+85 (crc 0 0 0) 0x7f370000d7f0 con 0x55e81344b7d0 2023-10-05T19:29:41.236 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:41.230+0000 7f37104a0700 1 -- 172.21.15.167:0/2673647112 <== osd.3 v2:172.21.15.167:6808/30036 35 ==== osd_op_reply(132 .dir.9a9df965-6612-45d4-a45f-6e3c889bacc2.4223.8.8 [call out=85b] v0'0 uv7 ondisk = 0) v8 ==== 194+0+85 (crc 0 0 0) 0x7f370000d7f0 con 0x55e81344b7d0 2023-10-05T19:29:41.236 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:41.230+0000 7f3710ca1700 1 -- 172.21.15.167:0/2673647112 <== osd.1 v2:172.21.15.167:6800/30035 39 ==== osd_op_reply(130 .dir.9a9df965-6612-45d4-a45f-6e3c889bacc2.4223.8.6 [call out=85b] v0'0 uv4 ondisk = 0) v8 ==== 194+0+85 (crc 0 0 0) 0x7f3708052040 con 0x55e8134147f0 2023-10-05T19:29:41.236 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:41.230+0000 7f370fc9f700 1 -- 172.21.15.167:0/2673647112 <== osd.2 v2:172.21.15.167:6816/30037 22 ==== osd_op_reply(129 .dir.9a9df965-6612-45d4-a45f-6e3c889bacc2.4223.8.5 [call out=124b] v0'0 uv10 ondisk = 0) v8 ==== 194+0+124 (crc 0 0 0) 0x7f36f8058040 con 0x7f3700044040 2023-10-05T19:29:41.237 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:41.230+0000 7f370fc9f700 1 -- 172.21.15.167:0/2673647112 <== osd.2 v2:172.21.15.167:6816/30037 23 ==== osd_op_reply(126 .dir.9a9df965-6612-45d4-a45f-6e3c889bacc2.4223.8.2 [call out=85b] v0'0 uv3 ondisk = 0) v8 ==== 194+0+85 (crc 0 0 0) 0x7f36f8058040 con 0x7f3700044040 2023-10-05T19:29:41.237 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:41.230+0000 7f37104a0700 1 -- 172.21.15.167:0/2673647112 <== osd.3 v2:172.21.15.167:6808/30036 36 ==== osd_op_reply(133 .dir.9a9df965-6612-45d4-a45f-6e3c889bacc2.4223.8.9 [call out=85b] v0'0 uv1 ondisk = 0) v8 ==== 194+0+85 (crc 0 0 0) 0x7f370000d7f0 con 0x55e81344b7d0 2023-10-05T19:29:41.237 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:41.230+0000 7f3710ca1700 1 -- 172.21.15.167:0/2673647112 <== osd.0 v2:172.21.15.118:6800/27690 35 ==== osd_op_reply(131 .dir.9a9df965-6612-45d4-a45f-6e3c889bacc2.4223.8.7 [call out=85b] v0'0 uv9 ondisk = 0) v8 ==== 194+0+85 (crc 0 0 0) 0x7f3708052040 con 0x7f36f8007030 2023-10-05T19:29:41.237 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:41.230+0000 7f370fc9f700 1 -- 172.21.15.167:0/2673647112 <== osd.2 v2:172.21.15.167:6816/30037 24 ==== osd_op_reply(134 .dir.9a9df965-6612-45d4-a45f-6e3c889bacc2.4223.8.10 [call out=85b] v0'0 uv6 ondisk = 0) v8 ==== 195+0+85 (crc 0 0 0) 0x7f36f8058040 con 0x7f3700044040 2023-10-05T19:29:41.237 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:41.230+0000 7f371296be40 20 int RGWRados::cls_bucket_list_ordered(const DoutPrefixProvider*, RGWBucketInfo&, int, const rgw_obj_index_key&, const string&, const string&, uint32_t, bool, uint16_t, RGWRados::ent_map_t&, bool*, bool*, rgw_obj_index_key*, optional_yield, RGWBucketListNameFilter): returning, count=0, is_truncated=0 2023-10-05T19:29:41.237 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:41.230+0000 7f371296be40 20 int RGWRados::cls_bucket_list_ordered(const DoutPrefixProvider*, RGWBucketInfo&, int, const rgw_obj_index_key&, const string&, const string&, uint32_t, bool, uint16_t, RGWRados::ent_map_t&, bool*, bool*, rgw_obj_index_key*, optional_yield, RGWBucketListNameFilter): returning, last_entry NOT SET 2023-10-05T19:29:41.237 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:41.230+0000 7f371296be40 10 int RGWRados::Bucket::List::list_objects_ordered(const DoutPrefixProvider*, int64_t, std::vector*, std::map, bool>*, bool*, optional_yield): end of outer loop, truncated=0, count=0, attempt=1 2023-10-05T19:29:41.237 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:41.230+0000 7f371296be40 10 int RGWRados::Bucket::List::list_objects_ordered(const DoutPrefixProvider*, int64_t, std::vector*, std::map, bool>*, bool*, optional_yield): exiting attempt loop because we reached end (0) or we're returning half the requested entries (0 of 1000) 2023-10-05T19:29:41.237 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:41.230+0000 7f371296be40 20 abort_multiparts INFO: aborting and cleaning up multipart upload(s); bucket=":myfoo7[9a9df965-6612-45d4-a45f-6e3c889bacc2.4223.8])"; uploads.size()=0; is_truncated=0 2023-10-05T19:29:41.237 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:41.230+0000 7f371296be40 10 lifecycle: RGWRados::convert_old_bucket_info(): bucket=:myfoo7[9a9df965-6612-45d4-a45f-6e3c889bacc2.4223.8]) 2023-10-05T19:29:41.237 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:41.230+0000 7f371296be40 20 lifecycle: get_system_obj_state: rctx=0x7ffc73a8b9c8 obj=default.rgw.meta:root:myfoo7 state=0x55e8132ed0b0 s->prefetch_data=0 2023-10-05T19:29:41.238 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:41.230+0000 7f371296be40 10 lifecycle: cache get: name=default.rgw.meta+root+myfoo7 : miss 2023-10-05T19:29:41.238 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:41.230+0000 7f371296be40 1 -- 172.21.15.167:0/2673647112 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:135 8.9 8:938c738d:root::myfoo7:head [call version.read in=11b,getxattrs,stat] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32) v8 -- 0x55e8134e5390 con 0x7f3700044040 2023-10-05T19:29:41.238 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:41.230+0000 7f370fc9f700 1 -- 172.21.15.167:0/2673647112 <== osd.2 v2:172.21.15.167:6816/30037 25 ==== osd_op_reply(135 myfoo7 [call out=48b,getxattrs out=75b,stat out=16b] v0'0 uv45 ondisk = 0) v8 ==== 234+0+139 (crc 0 0 0) 0x7f36f8058040 con 0x7f3700044040 2023-10-05T19:29:41.238 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:41.230+0000 7f371296be40 10 lifecycle: cache put: name=default.rgw.meta+root+myfoo7 info.flags=0x16 2023-10-05T19:29:41.238 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:41.230+0000 7f371296be40 10 lifecycle: adding default.rgw.meta+root+myfoo7 to cache LRU end 2023-10-05T19:29:41.238 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:41.230+0000 7f371296be40 10 lifecycle: updating xattr: name=ceph.objclass.version bl.length()=42 2023-10-05T19:29:41.238 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:41.230+0000 7f371296be40 20 lifecycle: get_system_obj_state: s->obj_tag was set empty 2023-10-05T19:29:41.238 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:41.230+0000 7f371296be40 20 lifecycle: Read xattr: user.rgw.idtag 2023-10-05T19:29:41.238 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:41.230+0000 7f371296be40 10 lifecycle: cache get: name=default.rgw.meta+root+myfoo7 : type miss (requested=0x13, cached=0x16) 2023-10-05T19:29:41.238 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:41.230+0000 7f371296be40 20 lifecycle: rados->read ofs=0 len=0 2023-10-05T19:29:41.238 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:41.230+0000 7f371296be40 1 -- 172.21.15.167:0/2673647112 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:136 8.9 8:938c738d:root::myfoo7:head [call version.check_conds in=122b,call version.read in=11b,read 0~0,getxattrs] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32) v8 -- 0x55e8134eba70 con 0x7f3700044040 2023-10-05T19:29:41.238 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:41.230+0000 7f370fc9f700 1 -- 172.21.15.167:0/2673647112 <== osd.2 v2:172.21.15.167:6816/30037 26 ==== osd_op_reply(136 myfoo7 [call,call out=48b,read 0~166 out=166b,getxattrs out=75b] v0'0 uv45 ondisk = 0) v8 ==== 276+0+289 (crc 0 0 0) 0x7f36f8058040 con 0x7f3700044040 2023-10-05T19:29:41.239 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:41.230+0000 7f371296be40 20 lifecycle: rados_obj.operate() r=0 bl.length=166 2023-10-05T19:29:41.239 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:41.230+0000 7f371296be40 10 lifecycle: cache put: name=default.rgw.meta+root+myfoo7 info.flags=0x13 2023-10-05T19:29:41.239 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:41.230+0000 7f371296be40 10 lifecycle: moving default.rgw.meta+root+myfoo7 to cache LRU end 2023-10-05T19:29:41.239 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:41.230+0000 7f371296be40 10 lifecycle: updating xattr: name=ceph.objclass.version bl.length()=42 2023-10-05T19:29:41.239 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:41.230+0000 7f371296be40 1 -- 172.21.15.167:0/2673647112 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:137 8.7 8:e76ab544:root::.bucket.meta.myfoo7%3a9a9df965-6612-45d4-a45f-6e3c889bacc2.4223.8:head [delete,create,call version.check_conds in=122b,call version.set in=58b,writefull 0~353 in=353b,setxattr user.rgw.acl (123) in=135b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e32) v8 -- 0x55e8134eba70 con 0x7f36f8007030 2023-10-05T19:29:41.240 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:41.234+0000 7f3710ca1700 1 -- 172.21.15.167:0/2673647112 <== osd.0 v2:172.21.15.118:6800/27690 36 ==== osd_op_reply(137 .bucket.meta.myfoo7:9a9df965-6612-45d4-a45f-6e3c889bacc2.4223.8 [delete,create,call,call,writefull 0~353,setxattr (123)] v32'3 uv3 ondisk = 0) v8 ==== 417+0+0 (crc 0 0 0) 0x7f3708052040 con 0x7f36f8007030 2023-10-05T19:29:41.240 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:41.234+0000 7f371296be40 10 lifecycle: cache put: name=default.rgw.meta+root+.bucket.meta.myfoo7:9a9df965-6612-45d4-a45f-6e3c889bacc2.4223.8 info.flags=0x17 2023-10-05T19:29:41.240 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:41.234+0000 7f371296be40 10 lifecycle: moving default.rgw.meta+root+.bucket.meta.myfoo7:9a9df965-6612-45d4-a45f-6e3c889bacc2.4223.8 to cache LRU end 2023-10-05T19:29:41.241 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:41.234+0000 7f371296be40 10 lifecycle: updating xattr: name=user.rgw.acl bl.length()=123 2023-10-05T19:29:41.241 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:41.234+0000 7f371296be40 10 lifecycle: distributing notification oid=default.rgw.control:notify.2 cni=[op: 0, obj: default.rgw.meta:root:.bucket.meta.myfoo7:9a9df965-6612-45d4-a45f-6e3c889bacc2.4223.8, ofs0, ns] 2023-10-05T19:29:41.241 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:41.234+0000 7f371296be40 1 -- 172.21.15.167:0/2673647112 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:138 7.15 7:a93a5511:::notify.2:head [notify cookie 94455244707680 in=739b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32) v8 -- 0x55e8134eba70 con 0x55e81344b7d0 2023-10-05T19:29:41.241 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:41.234+0000 7f37104a0700 1 -- 172.21.15.167:0/2673647112 <== osd.3 v2:172.21.15.167:6808/30036 37 ==== watch-notify(notify (1) cookie 94455244068320 notify 137438953493 ret 0) v3 ==== 769+0+0 (crc 0 0 0) 0x7f3700071040 con 0x55e81344b7d0 2023-10-05T19:29:41.241 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:41.234+0000 7f370e49c700 10 rgw watcher librados: RGWWatcher::handle_notify() notify_id 137438953493 cookie 94455244068320 notifier 4737 bl.length()=727 2023-10-05T19:29:41.241 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:41.234+0000 7f370e49c700 10 rgw watcher librados: cache put: name=default.rgw.meta+root+.bucket.meta.myfoo7:9a9df965-6612-45d4-a45f-6e3c889bacc2.4223.8 info.flags=0x17 2023-10-05T19:29:41.241 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:41.234+0000 7f370e49c700 10 rgw watcher librados: moving default.rgw.meta+root+.bucket.meta.myfoo7:9a9df965-6612-45d4-a45f-6e3c889bacc2.4223.8 to cache LRU end 2023-10-05T19:29:41.241 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:41.234+0000 7f370e49c700 10 rgw watcher librados: updating xattr: name=user.rgw.acl bl.length()=123 2023-10-05T19:29:41.241 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:41.234+0000 7f370e49c700 1 -- 172.21.15.167:0/2673647112 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:139 7.15 7:a93a5511:::notify.2:head [notify-ack in=20b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32) v8 -- 0x7f36f4003d90 con 0x55e81344b7d0 2023-10-05T19:29:41.241 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:41.234+0000 7f37104a0700 1 -- 172.21.15.167:0/2673647112 <== osd.3 v2:172.21.15.167:6808/30036 38 ==== osd_op_reply(138 notify.2 [notify cookie 94455244707680 out=8b] v0'0 uv255 ondisk = 0) v8 ==== 152+0+8 (crc 0 0 0) 0x7f370000d7f0 con 0x55e81344b7d0 2023-10-05T19:29:41.241 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:41.234+0000 7f37104a0700 1 -- 172.21.15.167:0/2673647112 <== osd.3 v2:172.21.15.167:6808/30036 39 ==== osd_op_reply(139 notify.2 [notify-ack] v0'0 uv255 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f370000d7f0 con 0x55e81344b7d0 2023-10-05T19:29:41.242 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:41.234+0000 7f37104a0700 1 -- 172.21.15.167:0/2673647112 <== osd.3 v2:172.21.15.167:6808/30036 40 ==== watch-notify(notify_complete (2) cookie 94455244707680 notify 137438953493 ret 0) v3 ==== 42+0+48 (crc 0 0 0) 0x55e81343c8c0 con 0x55e81344b7d0 2023-10-05T19:29:41.242 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:41.234+0000 7f371296be40 20 lifecycle: handle_bi_update(): bucket=:myfoo7[9a9df965-6612-45d4-a45f-6e3c889bacc2.4223.8]): orig_sources= new_sources= 2023-10-05T19:29:41.242 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:41.234+0000 7f371296be40 20 lifecycle: handle_bi_update(): bucket=:myfoo7[9a9df965-6612-45d4-a45f-6e3c889bacc2.4223.8]): potential sources added=[] removed=[] 2023-10-05T19:29:41.242 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:41.234+0000 7f371296be40 20 lifecycle: handle_bi_update(): bucket=:myfoo7[9a9df965-6612-45d4-a45f-6e3c889bacc2.4223.8]): orig_dests= new_dests= 2023-10-05T19:29:41.242 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:41.234+0000 7f371296be40 20 lifecycle: handle_bi_update(): bucket=:myfoo7[9a9df965-6612-45d4-a45f-6e3c889bacc2.4223.8]): potential dests added=[] removed=[] 2023-10-05T19:29:41.242 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:41.234+0000 7f371296be40 1 -- 172.21.15.167:0/2022125644 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:33 6.1a 6:5be42343:lc::lc.11:head [call lock.lock in=65b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e32) v8 -- 0x55e8134ed660 con 0x55e81347b7f0 2023-10-05T19:29:41.243 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:41.238+0000 7f3710ca1700 1 -- 172.21.15.167:0/2022125644 <== osd.3 v2:172.21.15.167:6808/30036 13 ==== osd_op_reply(33 lc.11 [call] v32'28 uv28 ondisk = 0) v8 ==== 149+0+0 (crc 0 0 0) 0x7f3708052040 con 0x55e81347b7f0 2023-10-05T19:29:41.243 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:41.238+0000 7f371296be40 1 -- 172.21.15.167:0/2022125644 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:34 6.1a 6:5be42343:lc::lc.11:head [call rgw.lc_rm_entry in=93b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32) v8 -- 0x55e8134ed660 con 0x55e81347b7f0 2023-10-05T19:29:41.244 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:41.238+0000 7f3710ca1700 1 -- 172.21.15.167:0/2022125644 <== osd.3 v2:172.21.15.167:6808/30036 14 ==== osd_op_reply(34 lc.11 [call] v32'29 uv29 ondisk = 0) v8 ==== 149+0+0 (crc 0 0 0) 0x7f3708052040 con 0x55e81347b7f0 2023-10-05T19:29:41.244 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:41.242+0000 7f371296be40 1 -- 172.21.15.167:0/2022125644 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:35 6.1a 6:5be42343:lc::lc.11:head [call lock.unlock in=49b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e32) v8 -- 0x55e8134ed660 con 0x55e81347b7f0 2023-10-05T19:29:41.247 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:41.242+0000 7f3710ca1700 1 -- 172.21.15.167:0/2022125644 <== osd.3 v2:172.21.15.167:6808/30036 15 ==== osd_op_reply(35 lc.11 [call] v32'30 uv30 ondisk = 0) v8 ==== 149+0+0 (crc 0 0 0) 0x7f3708052040 con 0x55e81347b7f0 2023-10-05T19:29:41.247 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:41.242+0000 7f371296be40 1 -- 172.21.15.167:0/2673647112 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:140 4.3b 4:dd1d8477:::.dir.9a9df965-6612-45d4-a45f-6e3c889bacc2.4223.8.0:head [call rgw.bucket_list in=47b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32) v8 -- 0x55e8134ed660 con 0x55e81344b7d0 2023-10-05T19:29:41.249 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:41.242+0000 7f371296be40 1 -- 172.21.15.167:0/2673647112 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:141 4.16 4:68fbd655:::.dir.9a9df965-6612-45d4-a45f-6e3c889bacc2.4223.8.1:head [call rgw.bucket_list in=47b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32) v8 -- 0x55e8134e78a0 con 0x55e81344b7d0 2023-10-05T19:29:41.249 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:41.242+0000 7f371296be40 1 -- 172.21.15.167:0/2673647112 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:142 4.1b 4:d8a327b1:::.dir.9a9df965-6612-45d4-a45f-6e3c889bacc2.4223.8.2:head [call rgw.bucket_list in=47b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32) v8 -- 0x55e8134e8740 con 0x7f3700044040 2023-10-05T19:29:41.249 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:41.242+0000 7f371296be40 1 -- 172.21.15.167:0/2673647112 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:143 4.3a 4:5fdd7e4a:::.dir.9a9df965-6612-45d4-a45f-6e3c889bacc2.4223.8.3:head [call rgw.bucket_list in=47b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32) v8 -- 0x55e8134ea080 con 0x55e81344b7d0 2023-10-05T19:29:41.249 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:41.242+0000 7f371296be40 1 -- 172.21.15.167:0/2673647112 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:144 4.2 4:41490dcd:::.dir.9a9df965-6612-45d4-a45f-6e3c889bacc2.4223.8.4:head [call rgw.bucket_list in=47b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32) v8 -- 0x55e8134ed660 con 0x55e8134147f0 2023-10-05T19:29:41.249 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:41.242+0000 7f371296be40 1 -- 172.21.15.167:0/2673647112 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:145 4.1a 4:5bd654c3:::.dir.9a9df965-6612-45d4-a45f-6e3c889bacc2.4223.8.5:head [call rgw.bucket_list in=47b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32) v8 -- 0x55e8134e8740 con 0x7f3700044040 2023-10-05T19:29:41.249 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:41.242+0000 7f371296be40 1 -- 172.21.15.167:0/2673647112 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:146 4.2b 4:d404b5ed:::.dir.9a9df965-6612-45d4-a45f-6e3c889bacc2.4223.8.6:head [call rgw.bucket_list in=47b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32) v8 -- 0x55e8134ed660 con 0x55e8134147f0 2023-10-05T19:29:41.249 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:41.242+0000 7f37104a0700 1 -- 172.21.15.167:0/2673647112 <== osd.3 v2:172.21.15.167:6808/30036 41 ==== osd_op_reply(143 .dir.9a9df965-6612-45d4-a45f-6e3c889bacc2.4223.8.3 [call out=85b] v0'0 uv6 ondisk = 0) v8 ==== 194+0+85 (crc 0 0 0) 0x7f370000d7f0 con 0x55e81344b7d0 2023-10-05T19:29:41.250 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:41.242+0000 7f371296be40 1 -- 172.21.15.167:0/2673647112 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:147 4.21 4:87af58f5:::.dir.9a9df965-6612-45d4-a45f-6e3c889bacc2.4223.8.7:head [call rgw.bucket_list in=47b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32) v8 -- 0x55e8134e8740 con 0x7f36f8007030 2023-10-05T19:29:41.250 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:41.242+0000 7f371296be40 1 -- 172.21.15.167:0/2673647112 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:148 4.3a 4:5e3bcf20:::.dir.9a9df965-6612-45d4-a45f-6e3c889bacc2.4223.8.8:head [call rgw.bucket_list in=47b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32) v8 -- 0x55e8134ed660 con 0x55e81344b7d0 2023-10-05T19:29:41.251 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:41.242+0000 7f37104a0700 1 -- 172.21.15.167:0/2673647112 <== osd.3 v2:172.21.15.167:6808/30036 42 ==== osd_op_reply(141 .dir.9a9df965-6612-45d4-a45f-6e3c889bacc2.4223.8.1 [call out=85b] v0'0 uv5 ondisk = 0) v8 ==== 194+0+85 (crc 0 0 0) 0x7f370000d7f0 con 0x55e81344b7d0 2023-10-05T19:29:41.251 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:41.242+0000 7f370fc9f700 1 -- 172.21.15.167:0/2673647112 <== osd.2 v2:172.21.15.167:6816/30037 27 ==== osd_op_reply(142 .dir.9a9df965-6612-45d4-a45f-6e3c889bacc2.4223.8.2 [call out=85b] v0'0 uv3 ondisk = 0) v8 ==== 194+0+85 (crc 0 0 0) 0x7f36f8058040 con 0x7f3700044040 2023-10-05T19:29:41.251 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:41.242+0000 7f3710ca1700 1 -- 172.21.15.167:0/2673647112 <== osd.1 v2:172.21.15.167:6800/30035 40 ==== osd_op_reply(144 .dir.9a9df965-6612-45d4-a45f-6e3c889bacc2.4223.8.4 [call out=85b] v0'0 uv11 ondisk = 0) v8 ==== 194+0+85 (crc 0 0 0) 0x7f3708052040 con 0x55e8134147f0 2023-10-05T19:29:41.251 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:41.242+0000 7f37104a0700 1 -- 172.21.15.167:0/2673647112 <== osd.3 v2:172.21.15.167:6808/30036 43 ==== osd_op_reply(140 .dir.9a9df965-6612-45d4-a45f-6e3c889bacc2.4223.8.0 [call out=85b] v0'0 uv3 ondisk = 0) v8 ==== 194+0+85 (crc 0 0 0) 0x7f370000d7f0 con 0x55e81344b7d0 2023-10-05T19:29:41.251 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:41.242+0000 7f371296be40 1 -- 172.21.15.167:0/2673647112 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:149 4.2e 4:74f15153:::.dir.9a9df965-6612-45d4-a45f-6e3c889bacc2.4223.8.9:head [call rgw.bucket_list in=47b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32) v8 -- 0x55e8134ed660 con 0x55e81344b7d0 2023-10-05T19:29:41.251 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:41.242+0000 7f371296be40 1 -- 172.21.15.167:0/2673647112 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:150 4.1a 4:5af95e16:::.dir.9a9df965-6612-45d4-a45f-6e3c889bacc2.4223.8.10:head [call rgw.bucket_list in=47b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32) v8 -- 0x55e8134ea080 con 0x7f3700044040 2023-10-05T19:29:41.251 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:41.242+0000 7f370fc9f700 1 -- 172.21.15.167:0/2673647112 <== osd.2 v2:172.21.15.167:6816/30037 28 ==== osd_op_reply(145 .dir.9a9df965-6612-45d4-a45f-6e3c889bacc2.4223.8.5 [call out=124b] v0'0 uv10 ondisk = 0) v8 ==== 194+0+124 (crc 0 0 0) 0x7f36f8058040 con 0x7f3700044040 2023-10-05T19:29:41.251 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:41.242+0000 7f3710ca1700 1 -- 172.21.15.167:0/2673647112 <== osd.1 v2:172.21.15.167:6800/30035 41 ==== osd_op_reply(146 .dir.9a9df965-6612-45d4-a45f-6e3c889bacc2.4223.8.6 [call out=85b] v0'0 uv4 ondisk = 0) v8 ==== 194+0+85 (crc 0 0 0) 0x7f3708052040 con 0x55e8134147f0 2023-10-05T19:29:41.251 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:41.242+0000 7f37104a0700 1 -- 172.21.15.167:0/2673647112 <== osd.3 v2:172.21.15.167:6808/30036 44 ==== osd_op_reply(148 .dir.9a9df965-6612-45d4-a45f-6e3c889bacc2.4223.8.8 [call out=85b] v0'0 uv7 ondisk = 0) v8 ==== 194+0+85 (crc 0 0 0) 0x7f370000d7f0 con 0x55e81344b7d0 2023-10-05T19:29:41.251 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:41.242+0000 7f37104a0700 1 -- 172.21.15.167:0/2673647112 <== osd.3 v2:172.21.15.167:6808/30036 45 ==== osd_op_reply(149 .dir.9a9df965-6612-45d4-a45f-6e3c889bacc2.4223.8.9 [call out=85b] v0'0 uv1 ondisk = 0) v8 ==== 194+0+85 (crc 0 0 0) 0x7f370000d7f0 con 0x55e81344b7d0 2023-10-05T19:29:41.251 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:41.242+0000 7f3710ca1700 1 -- 172.21.15.167:0/2673647112 <== osd.0 v2:172.21.15.118:6800/27690 37 ==== osd_op_reply(147 .dir.9a9df965-6612-45d4-a45f-6e3c889bacc2.4223.8.7 [call out=85b] v0'0 uv9 ondisk = 0) v8 ==== 194+0+85 (crc 0 0 0) 0x7f3708052040 con 0x7f36f8007030 2023-10-05T19:29:41.252 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:41.242+0000 7f370fc9f700 1 -- 172.21.15.167:0/2673647112 <== osd.2 v2:172.21.15.167:6816/30037 29 ==== osd_op_reply(150 .dir.9a9df965-6612-45d4-a45f-6e3c889bacc2.4223.8.10 [call out=85b] v0'0 uv6 ondisk = 0) v8 ==== 195+0+85 (crc 0 0 0) 0x7f36f8058040 con 0x7f3700044040 2023-10-05T19:29:41.252 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:41.246+0000 7f371296be40 1 -- 172.21.15.167:0/2673647112 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:151 8.1 8:810902b1:users.uid::fud.buckets:head [call user.set_buckets_info in=208b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e32) v8 -- 0x55e8134ed660 con 0x7f36f8007030 2023-10-05T19:29:41.253 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:41.246+0000 7f3710ca1700 1 -- 172.21.15.167:0/2673647112 <== osd.0 v2:172.21.15.118:6800/27690 38 ==== osd_op_reply(151 fud.buckets [call] v32'6 uv6 ondisk = 0) v8 ==== 155+0+0 (crc 0 0 0) 0x7f3708052040 con 0x7f36f8007030 2023-10-05T19:29:41.254 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:41.246+0000 7f371296be40 20 get_system_obj_state: rctx=0x7ffc73a8b448 obj=default.rgw.meta:root:myfoo7 state=0x55e8132ed0b0 s->prefetch_data=0 2023-10-05T19:29:41.254 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:41.246+0000 7f371296be40 10 cache get: name=default.rgw.meta+root+myfoo7 : type miss (requested=0x16, cached=0x13) 2023-10-05T19:29:41.254 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:41.246+0000 7f371296be40 1 -- 172.21.15.167:0/2673647112 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:152 8.9 8:938c738d:root::myfoo7:head [call version.read in=11b,getxattrs,stat] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32) v8 -- 0x55e8134ed660 con 0x7f3700044040 2023-10-05T19:29:41.254 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:41.246+0000 7f370fc9f700 1 -- 172.21.15.167:0/2673647112 <== osd.2 v2:172.21.15.167:6816/30037 30 ==== osd_op_reply(152 myfoo7 [call out=48b,getxattrs out=75b,stat out=16b] v0'0 uv45 ondisk = 0) v8 ==== 234+0+139 (crc 0 0 0) 0x7f36f8058040 con 0x7f3700044040 2023-10-05T19:29:41.254 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:41.246+0000 7f371296be40 10 cache put: name=default.rgw.meta+root+myfoo7 info.flags=0x16 2023-10-05T19:29:41.254 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:41.246+0000 7f371296be40 10 moving default.rgw.meta+root+myfoo7 to cache LRU end 2023-10-05T19:29:41.254 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:41.246+0000 7f371296be40 10 updating xattr: name=ceph.objclass.version bl.length()=42 2023-10-05T19:29:41.254 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:41.246+0000 7f371296be40 20 get_system_obj_state: s->obj_tag was set empty 2023-10-05T19:29:41.254 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:41.246+0000 7f371296be40 10 cache get: name=default.rgw.meta+root+myfoo7 : hit (requested=0x11, cached=0x17) 2023-10-05T19:29:41.254 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:41.246+0000 7f371296be40 10 removing default.rgw.meta+root+myfoo7 from cache 2023-10-05T19:29:41.254 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:41.246+0000 7f371296be40 10 distributing notification oid=default.rgw.control:notify.3 cni=[op: 1, obj: default.rgw.meta:root:myfoo7, ofs0, ns] 2023-10-05T19:29:41.255 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:41.246+0000 7f371296be40 1 -- 172.21.15.167:0/2673647112 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:153 7.e 7:7759931f:::notify.3:head [notify cookie 94455244683424 in=162b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32) v8 -- 0x55e8134ed660 con 0x7f3700044040 2023-10-05T19:29:41.255 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:41.250+0000 7f370fc9f700 1 -- 172.21.15.167:0/2673647112 <== osd.2 v2:172.21.15.167:6816/30037 31 ==== watch-notify(notify (1) cookie 94455244076560 notify 137438953491 ret 0) v3 ==== 192+0+0 (crc 0 0 0) 0x7f36f805f000 con 0x7f3700044040 2023-10-05T19:29:41.255 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:41.250+0000 7f370dc9b700 10 rgw watcher librados: RGWWatcher::handle_notify() notify_id 137438953491 cookie 94455244076560 notifier 4737 bl.length()=150 2023-10-05T19:29:41.255 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:41.250+0000 7f370fc9f700 1 -- 172.21.15.167:0/2673647112 <== osd.2 v2:172.21.15.167:6816/30037 32 ==== osd_op_reply(153 notify.3 [notify cookie 94455244683424 out=8b] v0'0 uv255 ondisk = 0) v8 ==== 152+0+8 (crc 0 0 0) 0x7f36f8058040 con 0x7f3700044040 2023-10-05T19:29:41.255 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:41.250+0000 7f370dc9b700 1 -- 172.21.15.167:0/2673647112 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:154 7.e 7:7759931f:::notify.3:head [notify-ack in=20b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32) v8 -- 0x7f3660003290 con 0x7f3700044040 2023-10-05T19:29:41.255 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:41.250+0000 7f370fc9f700 1 -- 172.21.15.167:0/2673647112 <== osd.2 v2:172.21.15.167:6816/30037 33 ==== osd_op_reply(154 notify.3 [notify-ack] v0'0 uv255 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f36f8058040 con 0x7f3700044040 2023-10-05T19:29:41.255 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:41.250+0000 7f370fc9f700 1 -- 172.21.15.167:0/2673647112 <== osd.2 v2:172.21.15.167:6816/30037 34 ==== watch-notify(notify_complete (2) cookie 94455244683424 notify 137438953491 ret 0) v3 ==== 42+0+48 (crc 0 0 0) 0x55e81346a3c0 con 0x7f3700044040 2023-10-05T19:29:41.255 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:41.250+0000 7f371296be40 1 -- 172.21.15.167:0/2673647112 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:155 8.9 8:938c738d:root::myfoo7:head [call version.check_conds in=122b,call version.set in=58b,delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e32) v8 -- 0x55e8134ed660 con 0x7f3700044040 2023-10-05T19:29:41.257 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:41.250+0000 7f370fc9f700 1 -- 172.21.15.167:0/2673647112 <== osd.2 v2:172.21.15.167:6816/30037 35 ==== osd_op_reply(155 myfoo7 [call,call,delete] v32'46 uv46 ondisk = 0) v8 ==== 234+0+0 (crc 0 0 0) 0x7f36f8058040 con 0x7f3700044040 2023-10-05T19:29:41.257 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:41.250+0000 7f371296be40 10 removing default.rgw.meta+root+.bucket.meta.myfoo7:9a9df965-6612-45d4-a45f-6e3c889bacc2.4223.8 from cache 2023-10-05T19:29:41.257 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:41.254+0000 7f371296be40 10 distributing notification oid=default.rgw.control:notify.2 cni=[op: 1, obj: default.rgw.meta:root:.bucket.meta.myfoo7:9a9df965-6612-45d4-a45f-6e3c889bacc2.4223.8, ofs0, ns] 2023-10-05T19:29:41.257 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:41.254+0000 7f371296be40 1 -- 172.21.15.167:0/2673647112 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:156 7.15 7:a93a5511:::notify.2:head [notify cookie 94455244683424 in=219b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32) v8 -- 0x55e8134ed660 con 0x55e81344b7d0 2023-10-05T19:29:41.257 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:41.254+0000 7f37104a0700 1 -- 172.21.15.167:0/2673647112 <== osd.3 v2:172.21.15.167:6808/30036 46 ==== watch-notify(notify (1) cookie 94455244068320 notify 137438953494 ret 0) v3 ==== 249+0+0 (crc 0 0 0) 0x55e81343c8c0 con 0x55e81344b7d0 2023-10-05T19:29:41.257 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:41.254+0000 7f370e49c700 10 rgw watcher librados: RGWWatcher::handle_notify() notify_id 137438953494 cookie 94455244068320 notifier 4737 bl.length()=207 2023-10-05T19:29:41.257 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:41.254+0000 7f370e49c700 1 -- 172.21.15.167:0/2673647112 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:157 7.15 7:a93a5511:::notify.2:head [notify-ack in=20b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32) v8 -- 0x7f36f4003d90 con 0x55e81344b7d0 2023-10-05T19:29:41.257 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:41.254+0000 7f37104a0700 1 -- 172.21.15.167:0/2673647112 <== osd.3 v2:172.21.15.167:6808/30036 47 ==== osd_op_reply(156 notify.2 [notify cookie 94455244683424 out=8b] v0'0 uv255 ondisk = 0) v8 ==== 152+0+8 (crc 0 0 0) 0x7f370000d7f0 con 0x55e81344b7d0 2023-10-05T19:29:41.258 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:41.254+0000 7f37104a0700 1 -- 172.21.15.167:0/2673647112 <== osd.3 v2:172.21.15.167:6808/30036 48 ==== osd_op_reply(157 notify.2 [notify-ack] v0'0 uv255 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f370000d7f0 con 0x55e81344b7d0 2023-10-05T19:29:41.258 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:41.254+0000 7f37104a0700 1 -- 172.21.15.167:0/2673647112 <== osd.3 v2:172.21.15.167:6808/30036 49 ==== watch-notify(notify_complete (2) cookie 94455244683424 notify 137438953494 ret 0) v3 ==== 42+0+48 (crc 0 0 0) 0x55e81343c480 con 0x55e81344b7d0 2023-10-05T19:29:41.258 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:41.254+0000 7f371296be40 1 -- 172.21.15.167:0/2673647112 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:158 8.7 8:e76ab544:root::.bucket.meta.myfoo7%3a9a9df965-6612-45d4-a45f-6e3c889bacc2.4223.8:head [call version.check_conds in=122b,call version.inc in=38b,delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e32) v8 -- 0x55e8134ed660 con 0x7f36f8007030 2023-10-05T19:29:41.261 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:41.258+0000 7f3710ca1700 1 -- 172.21.15.167:0/2673647112 <== osd.0 v2:172.21.15.118:6800/27690 39 ==== osd_op_reply(158 .bucket.meta.myfoo7:9a9df965-6612-45d4-a45f-6e3c889bacc2.4223.8 [call,call,delete] v32'4 uv4 ondisk = 0) v8 ==== 291+0+0 (crc 0 0 0) 0x7f3708052040 con 0x7f36f8007030 2023-10-05T19:29:41.261 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:41.258+0000 7f371296be40 1 -- 172.21.15.167:0/2673647112 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:159 4.3b 4:dd1d8477:::.dir.9a9df965-6612-45d4-a45f-6e3c889bacc2.4223.8.0:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e32) v8 -- 0x55e8134ed660 con 0x55e81344b7d0 2023-10-05T19:29:41.261 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:41.258+0000 7f371296be40 1 -- 172.21.15.167:0/2673647112 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:160 4.16 4:68fbd655:::.dir.9a9df965-6612-45d4-a45f-6e3c889bacc2.4223.8.1:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e32) v8 -- 0x55e8134eba70 con 0x55e81344b7d0 2023-10-05T19:29:41.261 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:41.258+0000 7f371296be40 1 -- 172.21.15.167:0/2673647112 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:161 4.1b 4:d8a327b1:::.dir.9a9df965-6612-45d4-a45f-6e3c889bacc2.4223.8.2:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e32) v8 -- 0x55e8134ead90 con 0x7f3700044040 2023-10-05T19:29:41.261 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:41.258+0000 7f371296be40 1 -- 172.21.15.167:0/2673647112 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:162 4.3a 4:5fdd7e4a:::.dir.9a9df965-6612-45d4-a45f-6e3c889bacc2.4223.8.3:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e32) v8 -- 0x55e8134edce0 con 0x55e81344b7d0 2023-10-05T19:29:41.261 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:41.258+0000 7f371296be40 1 -- 172.21.15.167:0/2673647112 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:163 4.2 4:41490dcd:::.dir.9a9df965-6612-45d4-a45f-6e3c889bacc2.4223.8.4:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e32) v8 -- 0x55e8134ee600 con 0x55e8134147f0 2023-10-05T19:29:41.262 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:41.258+0000 7f371296be40 1 -- 172.21.15.167:0/2673647112 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:164 4.1a 4:5bd654c3:::.dir.9a9df965-6612-45d4-a45f-6e3c889bacc2.4223.8.5:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e32) v8 -- 0x55e8134ed660 con 0x7f3700044040 2023-10-05T19:29:41.262 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:41.258+0000 7f371296be40 1 -- 172.21.15.167:0/2673647112 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:165 4.2b 4:d404b5ed:::.dir.9a9df965-6612-45d4-a45f-6e3c889bacc2.4223.8.6:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e32) v8 -- 0x55e8134eba70 con 0x55e8134147f0 2023-10-05T19:29:41.262 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:41.258+0000 7f371296be40 1 -- 172.21.15.167:0/2673647112 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:166 4.21 4:87af58f5:::.dir.9a9df965-6612-45d4-a45f-6e3c889bacc2.4223.8.7:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e32) v8 -- 0x55e8134edce0 con 0x7f36f8007030 2023-10-05T19:29:41.262 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:41.258+0000 7f371296be40 1 -- 172.21.15.167:0/2673647112 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:167 4.3a 4:5e3bcf20:::.dir.9a9df965-6612-45d4-a45f-6e3c889bacc2.4223.8.8:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e32) v8 -- 0x55e8134ee600 con 0x55e81344b7d0 2023-10-05T19:29:41.262 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:41.258+0000 7f371296be40 1 -- 172.21.15.167:0/2673647112 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:168 4.2e 4:74f15153:::.dir.9a9df965-6612-45d4-a45f-6e3c889bacc2.4223.8.9:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e32) v8 -- 0x55e8134ed660 con 0x55e81344b7d0 2023-10-05T19:29:41.262 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:41.258+0000 7f371296be40 1 -- 172.21.15.167:0/2673647112 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:169 4.1a 4:5af95e16:::.dir.9a9df965-6612-45d4-a45f-6e3c889bacc2.4223.8.10:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e32) v8 -- 0x55e8134eba70 con 0x7f3700044040 2023-10-05T19:29:41.264 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:41.258+0000 7f370fc9f700 1 -- 172.21.15.167:0/2673647112 <== osd.2 v2:172.21.15.167:6816/30037 36 ==== osd_op_reply(161 .dir.9a9df965-6612-45d4-a45f-6e3c889bacc2.4223.8.2 [delete] v32'4 uv4 ondisk = 0) v8 ==== 194+0+0 (crc 0 0 0) 0x7f36f8058040 con 0x7f3700044040 2023-10-05T19:29:41.265 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:41.262+0000 7f37104a0700 1 -- 172.21.15.167:0/2673647112 <== osd.3 v2:172.21.15.167:6808/30036 50 ==== osd_op_reply(160 .dir.9a9df965-6612-45d4-a45f-6e3c889bacc2.4223.8.1 [delete] v32'6 uv6 ondisk = 0) v8 ==== 194+0+0 (crc 0 0 0) 0x7f370000d7f0 con 0x55e81344b7d0 2023-10-05T19:29:41.266 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:41.262+0000 7f37104a0700 1 -- 172.21.15.167:0/2673647112 <== osd.3 v2:172.21.15.167:6808/30036 51 ==== osd_op_reply(159 .dir.9a9df965-6612-45d4-a45f-6e3c889bacc2.4223.8.0 [delete] v32'4 uv4 ondisk = 0) v8 ==== 194+0+0 (crc 0 0 0) 0x7f370000d7f0 con 0x55e81344b7d0 2023-10-05T19:29:41.266 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:41.262+0000 7f3710ca1700 1 -- 172.21.15.167:0/2673647112 <== osd.1 v2:172.21.15.167:6800/30035 42 ==== osd_op_reply(163 .dir.9a9df965-6612-45d4-a45f-6e3c889bacc2.4223.8.4 [delete] v32'12 uv12 ondisk = 0) v8 ==== 194+0+0 (crc 0 0 0) 0x7f3708052040 con 0x55e8134147f0 2023-10-05T19:29:41.266 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:41.262+0000 7f3710ca1700 1 -- 172.21.15.167:0/2673647112 <== osd.1 v2:172.21.15.167:6800/30035 43 ==== osd_op_reply(165 .dir.9a9df965-6612-45d4-a45f-6e3c889bacc2.4223.8.6 [delete] v32'5 uv5 ondisk = 0) v8 ==== 194+0+0 (crc 0 0 0) 0x7f3708052040 con 0x55e8134147f0 2023-10-05T19:29:41.267 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:41.262+0000 7f370fc9f700 1 -- 172.21.15.167:0/2673647112 <== osd.2 v2:172.21.15.167:6816/30037 37 ==== osd_op_reply(164 .dir.9a9df965-6612-45d4-a45f-6e3c889bacc2.4223.8.5 [delete] v32'11 uv11 ondisk = 0) v8 ==== 194+0+0 (crc 0 0 0) 0x7f36f8058040 con 0x7f3700044040 2023-10-05T19:29:41.267 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:41.262+0000 7f3710ca1700 1 -- 172.21.15.167:0/2673647112 <== osd.0 v2:172.21.15.118:6800/27690 40 ==== osd_op_reply(166 .dir.9a9df965-6612-45d4-a45f-6e3c889bacc2.4223.8.7 [delete] v32'10 uv10 ondisk = 0) v8 ==== 194+0+0 (crc 0 0 0) 0x7f3708052040 con 0x7f36f8007030 2023-10-05T19:29:41.268 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:41.262+0000 7f370fc9f700 1 -- 172.21.15.167:0/2673647112 <== osd.2 v2:172.21.15.167:6816/30037 38 ==== osd_op_reply(169 .dir.9a9df965-6612-45d4-a45f-6e3c889bacc2.4223.8.10 [delete] v32'12 uv12 ondisk = 0) v8 ==== 195+0+0 (crc 0 0 0) 0x7f36f8058040 con 0x7f3700044040 2023-10-05T19:29:41.269 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:41.262+0000 7f37104a0700 1 -- 172.21.15.167:0/2673647112 <== osd.3 v2:172.21.15.167:6808/30036 52 ==== osd_op_reply(162 .dir.9a9df965-6612-45d4-a45f-6e3c889bacc2.4223.8.3 [delete] v32'11 uv11 ondisk = 0) v8 ==== 194+0+0 (crc 0 0 0) 0x7f370000d7f0 con 0x55e81344b7d0 2023-10-05T19:29:41.269 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:41.266+0000 7f37104a0700 1 -- 172.21.15.167:0/2673647112 <== osd.3 v2:172.21.15.167:6808/30036 53 ==== osd_op_reply(168 .dir.9a9df965-6612-45d4-a45f-6e3c889bacc2.4223.8.9 [delete] v32'2 uv2 ondisk = 0) v8 ==== 194+0+0 (crc 0 0 0) 0x7f370000d7f0 con 0x55e81344b7d0 2023-10-05T19:29:41.269 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:41.266+0000 7f37104a0700 1 -- 172.21.15.167:0/2673647112 <== osd.3 v2:172.21.15.167:6808/30036 54 ==== osd_op_reply(167 .dir.9a9df965-6612-45d4-a45f-6e3c889bacc2.4223.8.8 [delete] v32'12 uv12 ondisk = 0) v8 ==== 194+0+0 (crc 0 0 0) 0x7f370000d7f0 con 0x55e81344b7d0 2023-10-05T19:29:41.269 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:41.266+0000 7f371296be40 1 -- 172.21.15.167:0/2673647112 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:170 6.17 6:e9b09e78:::pubsub..bucket.myfoo7%2f9a9df965-6612-45d4-a45f-6e3c889bacc2.4223.8:head [getxattrs,stat] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32) v8 -- 0x55e8134edce0 con 0x55e8134147f0 2023-10-05T19:29:41.269 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:41.266+0000 7f3710ca1700 1 -- 172.21.15.167:0/2673647112 <== osd.1 v2:172.21.15.167:6800/30035 44 ==== osd_op_reply(170 pubsub..bucket.myfoo7/9a9df965-6612-45d4-a45f-6e3c889bacc2.4223.8 [getxattrs,stat] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 ==== 251+0+0 (crc 0 0 0) 0x7f3708052040 con 0x55e8134147f0 2023-10-05T19:29:41.269 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:41.266+0000 7f371296be40 10 removing default.rgw.log++pubsub..bucket.myfoo7/9a9df965-6612-45d4-a45f-6e3c889bacc2.4223.8 from cache 2023-10-05T19:29:41.269 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:41.266+0000 7f371296be40 10 distributing notification oid=default.rgw.control:notify.1 cni=[op: 1, obj: default.rgw.log:pubsub..bucket.myfoo7/9a9df965-6612-45d4-a45f-6e3c889bacc2.4223.8, ofs0, ns] 2023-10-05T19:29:41.270 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:41.266+0000 7f371296be40 1 -- 172.21.15.167:0/2673647112 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:171 7.0 7:05bf5b68:::notify.1:head [notify cookie 94455245225440 in=216b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32) v8 -- 0x55e8134ed660 con 0x55e8134147f0 2023-10-05T19:29:41.270 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:41.266+0000 7f3710ca1700 1 -- 172.21.15.167:0/2673647112 <== osd.1 v2:172.21.15.167:6800/30035 45 ==== watch-notify(notify (1) cookie 94455244067216 notify 137438953494 ret 0) v3 ==== 246+0+0 (crc 0 0 0) 0x7f3708086040 con 0x55e8134147f0 2023-10-05T19:29:41.271 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:41.266+0000 7f370e49c700 10 rgw watcher librados: RGWWatcher::handle_notify() notify_id 137438953494 cookie 94455244067216 notifier 4737 bl.length()=204 2023-10-05T19:29:41.271 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:41.266+0000 7f3710ca1700 1 -- 172.21.15.167:0/2673647112 <== osd.1 v2:172.21.15.167:6800/30035 46 ==== osd_op_reply(171 notify.1 [notify cookie 94455245225440 out=8b] v0'0 uv255 ondisk = 0) v8 ==== 152+0+8 (crc 0 0 0) 0x7f3708052040 con 0x55e8134147f0 2023-10-05T19:29:41.271 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:41.266+0000 7f370e49c700 1 -- 172.21.15.167:0/2673647112 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:172 7.0 7:05bf5b68:::notify.1:head [notify-ack in=20b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32) v8 -- 0x7f36f4003d90 con 0x55e8134147f0 2023-10-05T19:29:41.271 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:41.266+0000 7f3710ca1700 1 -- 172.21.15.167:0/2673647112 <== osd.1 v2:172.21.15.167:6800/30035 47 ==== osd_op_reply(172 notify.1 [notify-ack] v0'0 uv255 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f3708052040 con 0x55e8134147f0 2023-10-05T19:29:41.271 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:41.266+0000 7f3710ca1700 1 -- 172.21.15.167:0/2673647112 <== osd.1 v2:172.21.15.167:6800/30035 48 ==== watch-notify(notify_complete (2) cookie 94455245225440 notify 137438953494 ret 0) v3 ==== 42+0+48 (crc 0 0 0) 0x7f370804e380 con 0x55e8134147f0 2023-10-05T19:29:41.271 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:41.266+0000 7f371296be40 1 -- 172.21.15.167:0/2673647112 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:173 6.17 6:e9b09e78:::pubsub..bucket.myfoo7%2f9a9df965-6612-45d4-a45f-6e3c889bacc2.4223.8:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e32) v8 -- 0x55e8134ed660 con 0x55e8134147f0 2023-10-05T19:29:41.272 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:41.266+0000 7f3710ca1700 1 -- 172.21.15.167:0/2673647112 <== osd.1 v2:172.21.15.167:6800/30035 49 ==== osd_op_reply(173 pubsub..bucket.myfoo7/9a9df965-6612-45d4-a45f-6e3c889bacc2.4223.8 [delete] v32'19 uv18 ondisk = -2 ((2) No such file or directory)) v8 ==== 209+0+0 (crc 0 0 0) 0x7f3708052040 con 0x55e8134147f0 2023-10-05T19:29:41.272 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:41.266+0000 7f371296be40 1 -- 172.21.15.167:0/2673647112 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:174 8.1 8:810902b1:users.uid::fud.buckets:head [call user.remove_bucket in=59b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e32) v8 -- 0x55e8134ed660 con 0x7f36f8007030 2023-10-05T19:29:41.274 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:41.270+0000 7f3710ca1700 1 -- 172.21.15.167:0/2673647112 <== osd.0 v2:172.21.15.118:6800/27690 41 ==== osd_op_reply(174 fud.buckets [call] v32'7 uv7 ondisk = 0) v8 ==== 155+0+0 (crc 0 0 0) 0x7f3708052040 con 0x7f36f8007030 2023-10-05T19:29:41.275 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:41.270+0000 7f371296be40 10 removing key index: p5YnriCv1nAtykxBrupQ 2023-10-05T19:29:41.275 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:41.270+0000 7f371296be40 10 distributing notification oid=default.rgw.control:notify.2 cni=[op: 1, obj: default.rgw.meta:users.keys:p5YnriCv1nAtykxBrupQ, ofs0, ns] 2023-10-05T19:29:41.275 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:41.270+0000 7f371296be40 1 -- 172.21.15.167:0/2673647112 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:175 7.15 7:a93a5511:::notify.2:head [notify cookie 94455245225440 in=182b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32) v8 -- 0x55e8134ed660 con 0x55e81344b7d0 2023-10-05T19:29:41.276 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:41.270+0000 7f37104a0700 1 -- 172.21.15.167:0/2673647112 <== osd.3 v2:172.21.15.167:6808/30036 55 ==== watch-notify(notify (1) cookie 94455244068320 notify 137438953495 ret 0) v3 ==== 212+0+0 (crc 0 0 0) 0x55e81343c480 con 0x55e81344b7d0 2023-10-05T19:29:41.276 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:41.270+0000 7f370dc9b700 10 rgw watcher librados: RGWWatcher::handle_notify() notify_id 137438953495 cookie 94455244068320 notifier 4737 bl.length()=170 2023-10-05T19:29:41.276 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:41.270+0000 7f37104a0700 1 -- 172.21.15.167:0/2673647112 <== osd.3 v2:172.21.15.167:6808/30036 56 ==== osd_op_reply(175 notify.2 [notify cookie 94455245225440 out=8b] v0'0 uv255 ondisk = 0) v8 ==== 152+0+8 (crc 0 0 0) 0x7f370000d7f0 con 0x55e81344b7d0 2023-10-05T19:29:41.276 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:41.270+0000 7f370dc9b700 1 -- 172.21.15.167:0/2673647112 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:176 7.15 7:a93a5511:::notify.2:head [notify-ack in=20b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32) v8 -- 0x7f3660003270 con 0x55e81344b7d0 2023-10-05T19:29:41.276 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:41.270+0000 7f37104a0700 1 -- 172.21.15.167:0/2673647112 <== osd.3 v2:172.21.15.167:6808/30036 57 ==== osd_op_reply(176 notify.2 [notify-ack] v0'0 uv255 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f370000d7f0 con 0x55e81344b7d0 2023-10-05T19:29:41.276 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:41.270+0000 7f37104a0700 1 -- 172.21.15.167:0/2673647112 <== osd.3 v2:172.21.15.167:6808/30036 58 ==== watch-notify(notify_complete (2) cookie 94455245225440 notify 137438953495 ret 0) v3 ==== 42+0+48 (crc 0 0 0) 0x55e813360bd0 con 0x55e81344b7d0 2023-10-05T19:29:41.276 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:41.270+0000 7f371296be40 1 -- 172.21.15.167:0/2673647112 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:177 8.13 8:cc695d75:users.keys::p5YnriCv1nAtykxBrupQ:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e32) v8 -- 0x55e8134ed660 con 0x7f36f8007030 2023-10-05T19:29:41.278 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:41.274+0000 7f3710ca1700 1 -- 172.21.15.167:0/2673647112 <== osd.0 v2:172.21.15.118:6800/27690 42 ==== osd_op_reply(177 p5YnriCv1nAtykxBrupQ [delete] v32'13 uv13 ondisk = 0) v8 ==== 164+0+0 (crc 0 0 0) 0x7f3708052040 con 0x7f36f8007030 2023-10-05T19:29:41.278 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:41.274+0000 7f371296be40 10 removing email index: 2023-10-05T19:29:41.278 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:41.274+0000 7f371296be40 10 removing user buckets index 2023-10-05T19:29:41.279 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:41.274+0000 7f371296be40 10 distributing notification oid=default.rgw.control:notify.6 cni=[op: 1, obj: default.rgw.meta:users.uid:fud.buckets, ofs0, ns] 2023-10-05T19:29:41.279 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:41.274+0000 7f371296be40 1 -- 172.21.15.167:0/2673647112 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:178 7.14 7:2b04a3e9:::notify.6:head [notify cookie 94455245225440 in=172b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32) v8 -- 0x55e8134ed660 con 0x7f36f8007030 2023-10-05T19:29:41.279 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:41.274+0000 7f3710ca1700 1 -- 172.21.15.167:0/2673647112 <== osd.0 v2:172.21.15.118:6800/27690 43 ==== watch-notify(notify (1) cookie 94455244080240 notify 137438953513 ret 0) v3 ==== 202+0+0 (crc 0 0 0) 0x7f370804e380 con 0x7f36f8007030 2023-10-05T19:29:41.279 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:41.274+0000 7f370e49c700 10 rgw watcher librados: RGWWatcher::handle_notify() notify_id 137438953513 cookie 94455244080240 notifier 4737 bl.length()=160 2023-10-05T19:29:41.279 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:41.274+0000 7f3710ca1700 1 -- 172.21.15.167:0/2673647112 <== osd.0 v2:172.21.15.118:6800/27690 44 ==== osd_op_reply(178 notify.6 [notify cookie 94455245225440 out=8b] v0'0 uv255 ondisk = 0) v8 ==== 152+0+8 (crc 0 0 0) 0x7f3708052040 con 0x7f36f8007030 2023-10-05T19:29:41.280 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:41.274+0000 7f370e49c700 1 -- 172.21.15.167:0/2673647112 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:179 7.14 7:2b04a3e9:::notify.6:head [notify-ack in=20b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32) v8 -- 0x7f36f4003d90 con 0x7f36f8007030 2023-10-05T19:29:41.280 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:41.274+0000 7f3710ca1700 1 -- 172.21.15.167:0/2673647112 <== osd.0 v2:172.21.15.118:6800/27690 45 ==== osd_op_reply(179 notify.6 [notify-ack] v0'0 uv255 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f3708052040 con 0x7f36f8007030 2023-10-05T19:29:41.281 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:41.278+0000 7f3710ca1700 1 -- 172.21.15.167:0/2673647112 <== osd.0 v2:172.21.15.118:6800/27690 46 ==== watch-notify(notify_complete (2) cookie 94455245225440 notify 137438953513 ret 0) v3 ==== 42+0+48 (crc 0 0 0) 0x7f3708036d80 con 0x7f36f8007030 2023-10-05T19:29:41.281 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:41.278+0000 7f371296be40 1 -- 172.21.15.167:0/2673647112 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:180 8.1 8:810902b1:users.uid::fud.buckets:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e32) v8 -- 0x55e8134ed660 con 0x7f36f8007030 2023-10-05T19:29:41.283 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:41.278+0000 7f3710ca1700 1 -- 172.21.15.167:0/2673647112 <== osd.0 v2:172.21.15.118:6800/27690 47 ==== osd_op_reply(180 fud.buckets [delete] v32'8 uv8 ondisk = 0) v8 ==== 155+0+0 (crc 0 0 0) 0x7f3708052040 con 0x7f36f8007030 2023-10-05T19:29:41.283 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:41.278+0000 7f371296be40 10 removing user index: fud 2023-10-05T19:29:41.283 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:41.278+0000 7f371296be40 10 removing default.rgw.meta+users.uid+fud from cache 2023-10-05T19:29:41.283 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:41.278+0000 7f371296be40 10 distributing notification oid=default.rgw.control:notify.5 cni=[op: 1, obj: default.rgw.meta:users.uid:fud, ofs0, ns] 2023-10-05T19:29:41.284 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:41.278+0000 7f371296be40 1 -- 172.21.15.167:0/2673647112 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:181 7.3 7:c609908c:::notify.5:head [notify cookie 94455245225440 in=164b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32) v8 -- 0x55e8134ed660 con 0x7f36f8007030 2023-10-05T19:29:41.284 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:41.278+0000 7f3710ca1700 1 -- 172.21.15.167:0/2673647112 <== osd.0 v2:172.21.15.118:6800/27690 48 ==== watch-notify(notify (1) cookie 94455244081920 notify 137438953514 ret 0) v3 ==== 194+0+0 (crc 0 0 0) 0x7f3708036d80 con 0x7f36f8007030 2023-10-05T19:29:41.284 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:41.278+0000 7f370e49c700 10 rgw watcher librados: RGWWatcher::handle_notify() notify_id 137438953514 cookie 94455244081920 notifier 4737 bl.length()=152 2023-10-05T19:29:41.284 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:41.278+0000 7f3710ca1700 1 -- 172.21.15.167:0/2673647112 <== osd.0 v2:172.21.15.118:6800/27690 49 ==== osd_op_reply(181 notify.5 [notify cookie 94455245225440 out=8b] v0'0 uv255 ondisk = 0) v8 ==== 152+0+8 (crc 0 0 0) 0x7f3708052040 con 0x7f36f8007030 2023-10-05T19:29:41.284 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:41.278+0000 7f370e49c700 1 -- 172.21.15.167:0/2673647112 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:182 7.3 7:c609908c:::notify.5:head [notify-ack in=20b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32) v8 -- 0x7f36f4003d90 con 0x7f36f8007030 2023-10-05T19:29:41.285 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:41.282+0000 7f3710ca1700 1 -- 172.21.15.167:0/2673647112 <== osd.0 v2:172.21.15.118:6800/27690 50 ==== osd_op_reply(182 notify.5 [notify-ack] v0'0 uv255 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f3708052040 con 0x7f36f8007030 2023-10-05T19:29:41.285 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:41.282+0000 7f3710ca1700 1 -- 172.21.15.167:0/2673647112 <== osd.0 v2:172.21.15.118:6800/27690 51 ==== watch-notify(notify_complete (2) cookie 94455245225440 notify 137438953514 ret 0) v3 ==== 42+0+48 (crc 0 0 0) 0x7f3708070530 con 0x7f36f8007030 2023-10-05T19:29:41.285 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:41.282+0000 7f371296be40 1 -- 172.21.15.167:0/2673647112 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:183 8.e 8:75f48285:users.uid::fud:head [call version.inc in=38b,delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e32) v8 -- 0x55e8134ed660 con 0x7f36f8007030 2023-10-05T19:29:41.288 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:41.286+0000 7f3710ca1700 1 -- 172.21.15.167:0/2673647112 <== osd.0 v2:172.21.15.118:6800/27690 52 ==== osd_op_reply(183 fud [call,delete] v32'7 uv7 ondisk = 0) v8 ==== 189+0+0 (crc 0 0 0) 0x7f3708052040 con 0x7f36f8007030 2023-10-05T19:29:41.289 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:41.286+0000 7f371296be40 1 -- 172.21.15.167:0/2673647112 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:184 7.1f 7:f95f44c2:::notify.0:head [watch unwatch cookie 94455244063616] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e32) v8 -- 0x55e8134ed660 con 0x7f3700044040 2023-10-05T19:29:41.292 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:41.286+0000 7f370fc9f700 1 -- 172.21.15.167:0/2673647112 <== osd.2 v2:172.21.15.167:6816/30037 39 ==== osd_op_reply(184 notify.0 [watch unwatch cookie 94455244063616] v32'257 uv255 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f36f8058040 con 0x7f3700044040 2023-10-05T19:29:41.292 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:41.286+0000 7f371296be40 20 remove_watcher() i=0 2023-10-05T19:29:41.292 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:41.286+0000 7f371296be40 2 removed watcher, disabling cache 2023-10-05T19:29:41.292 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:41.286+0000 7f371296be40 1 -- 172.21.15.167:0/2673647112 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:185 7.0 7:05bf5b68:::notify.1:head [watch unwatch cookie 94455244067216] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e32) v8 -- 0x55e81342ba90 con 0x55e8134147f0 2023-10-05T19:29:41.295 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:41.290+0000 7f3710ca1700 1 -- 172.21.15.167:0/2673647112 <== osd.1 v2:172.21.15.167:6800/30035 50 ==== osd_op_reply(185 notify.1 [watch unwatch cookie 94455244067216] v32'257 uv255 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f3708052040 con 0x55e8134147f0 2023-10-05T19:29:41.295 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:41.290+0000 7f371296be40 20 remove_watcher() i=1 2023-10-05T19:29:41.295 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:41.290+0000 7f371296be40 1 -- 172.21.15.167:0/2673647112 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:186 7.15 7:a93a5511:::notify.2:head [watch unwatch cookie 94455244068320] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e32) v8 -- 0x55e8134ed660 con 0x55e81344b7d0 2023-10-05T19:29:41.297 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:41.294+0000 7f37104a0700 1 -- 172.21.15.167:0/2673647112 <== osd.3 v2:172.21.15.167:6808/30036 59 ==== osd_op_reply(186 notify.2 [watch unwatch cookie 94455244068320] v32'257 uv255 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f370000d7f0 con 0x55e81344b7d0 2023-10-05T19:29:41.298 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:41.294+0000 7f371296be40 20 remove_watcher() i=2 2023-10-05T19:29:41.298 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:41.294+0000 7f371296be40 1 -- 172.21.15.167:0/2673647112 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:187 7.e 7:7759931f:::notify.3:head [watch unwatch cookie 94455244076560] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e32) v8 -- 0x55e81342ba90 con 0x7f3700044040 2023-10-05T19:29:41.300 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:41.298+0000 7f370fc9f700 1 -- 172.21.15.167:0/2673647112 <== osd.2 v2:172.21.15.167:6816/30037 40 ==== osd_op_reply(187 notify.3 [watch unwatch cookie 94455244076560] v32'257 uv255 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f36f8058040 con 0x7f3700044040 2023-10-05T19:29:41.301 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:41.298+0000 7f371296be40 20 remove_watcher() i=3 2023-10-05T19:29:41.301 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:41.298+0000 7f371296be40 1 -- 172.21.15.167:0/2673647112 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:188 7.d 7:b4812045:::notify.4:head [watch unwatch cookie 94455244070912] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e32) v8 -- 0x55e8134ed660 con 0x55e8134147f0 2023-10-05T19:29:41.303 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:41.298+0000 7f3710ca1700 1 -- 172.21.15.167:0/2673647112 <== osd.1 v2:172.21.15.167:6800/30035 51 ==== osd_op_reply(188 notify.4 [watch unwatch cookie 94455244070912] v32'257 uv255 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f3708052040 con 0x55e8134147f0 2023-10-05T19:29:41.303 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:41.298+0000 7f371296be40 20 remove_watcher() i=4 2023-10-05T19:29:41.304 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:41.298+0000 7f371296be40 1 -- 172.21.15.167:0/2673647112 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:189 7.3 7:c609908c:::notify.5:head [watch unwatch cookie 94455244081920] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e32) v8 -- 0x55e81342ba90 con 0x7f36f8007030 2023-10-05T19:29:41.306 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:41.302+0000 7f3710ca1700 1 -- 172.21.15.167:0/2673647112 <== osd.0 v2:172.21.15.118:6800/27690 53 ==== osd_op_reply(189 notify.5 [watch unwatch cookie 94455244081920] v32'257 uv255 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f3708052040 con 0x7f36f8007030 2023-10-05T19:29:41.306 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:41.302+0000 7f371296be40 20 remove_watcher() i=5 2023-10-05T19:29:41.306 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:41.302+0000 7f371296be40 1 -- 172.21.15.167:0/2673647112 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:190 7.14 7:2b04a3e9:::notify.6:head [watch unwatch cookie 94455244080240] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e32) v8 -- 0x55e8134ed660 con 0x7f36f8007030 2023-10-05T19:29:41.309 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:41.306+0000 7f3710ca1700 1 -- 172.21.15.167:0/2673647112 <== osd.0 v2:172.21.15.118:6800/27690 54 ==== osd_op_reply(190 notify.6 [watch unwatch cookie 94455244080240] v32'257 uv255 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f3708052040 con 0x7f36f8007030 2023-10-05T19:29:41.309 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:41.306+0000 7f371296be40 20 remove_watcher() i=6 2023-10-05T19:29:41.309 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:41.306+0000 7f371296be40 1 -- 172.21.15.167:0/2673647112 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:191 7.9 7:93e5b521:::notify.7:head [watch unwatch cookie 94455244084192] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e32) v8 -- 0x55e81342ba90 con 0x7f36f8007030 2023-10-05T19:29:41.312 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:41.306+0000 7f3710ca1700 1 -- 172.21.15.167:0/2673647112 <== osd.0 v2:172.21.15.118:6800/27690 55 ==== osd_op_reply(191 notify.7 [watch unwatch cookie 94455244084192] v32'257 uv255 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f3708052040 con 0x7f36f8007030 2023-10-05T19:29:41.312 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:41.306+0000 7f371296be40 20 remove_watcher() i=7 2023-10-05T19:29:41.314 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:41.310+0000 7f36e1fe3700 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2023-10-05T19:29:41.314 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:41.310+0000 7f371296be40 1 -- 172.21.15.167:0/2673647112 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:192 6.4 6:22d26bf9:::data_loggenerations_metadata:head [watch unwatch cookie 94455244087728] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e32) v8 -- 0x55e8134a2700 con 0x55e8134147f0 2023-10-05T19:29:41.318 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:41.314+0000 7f3710ca1700 1 -- 172.21.15.167:0/2673647112 <== osd.1 v2:172.21.15.167:6800/30035 52 ==== osd_op_reply(192 data_loggenerations_metadata [watch unwatch cookie 94455244087728] v32'250 uv1 ondisk = 0) v8 ==== 172+0+0 (crc 0 0 0) 0x7f3708052040 con 0x55e8134147f0 2023-10-05T19:29:41.318 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:41.314+0000 7f371296be40 1 -- 172.21.15.167:0/2673647112 >> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] conn(0x7f36f8007030 msgr2=0x7f36f801c920 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-05T19:29:41.318 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:41.314+0000 7f371296be40 1 --2- 172.21.15.167:0/2673647112 >> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] conn(0x7f36f8007030 0x7f36f801c920 crc :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-05T19:29:41.318 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:41.314+0000 7f371296be40 1 -- 172.21.15.167:0/2673647112 >> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] conn(0x55e8134147f0 msgr2=0x55e8134479a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-05T19:29:41.318 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:41.314+0000 7f371296be40 1 --2- 172.21.15.167:0/2673647112 >> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] conn(0x55e8134147f0 0x55e8134479a0 crc :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-05T19:29:41.318 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:41.314+0000 7f371296be40 1 -- 172.21.15.167:0/2673647112 >> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] conn(0x7f3700044040 msgr2=0x7f37000420e0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-05T19:29:41.318 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:41.314+0000 7f371296be40 1 --2- 172.21.15.167:0/2673647112 >> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] conn(0x7f3700044040 0x7f37000420e0 crc :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-05T19:29:41.318 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:41.314+0000 7f371296be40 1 -- 172.21.15.167:0/2673647112 >> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] conn(0x55e81344b7d0 msgr2=0x55e81344dbb0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-05T19:29:41.319 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:41.314+0000 7f371296be40 1 --2- 172.21.15.167:0/2673647112 >> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] conn(0x55e81344b7d0 0x55e81344dbb0 crc :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-05T19:29:41.319 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:41.314+0000 7f371296be40 1 -- 172.21.15.167:0/2673647112 >> [v2:172.21.15.167:6824/29889,v1:172.21.15.167:6825/29889] conn(0x7f36f0030c40 msgr2=0x7f36f00330f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-05T19:29:41.319 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:41.314+0000 7f371296be40 1 --2- 172.21.15.167:0/2673647112 >> [v2:172.21.15.167:6824/29889,v1:172.21.15.167:6825/29889] conn(0x7f36f0030c40 0x7f36f00330f0 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7f36f8005a90 tx=0x7f36f8015000 comp rx=0 tx=0).stop 2023-10-05T19:29:41.319 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:41.314+0000 7f3710ca1700 1 -- 172.21.15.167:0/2673647112 reap_dead start 2023-10-05T19:29:41.319 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:41.314+0000 7f371296be40 1 -- 172.21.15.167:0/2673647112 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55e813414420 msgr2=0x55e81343bf40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-05T19:29:41.319 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:41.314+0000 7f371296be40 1 --2- 172.21.15.167:0/2673647112 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55e813414420 0x55e81343bf40 secure :-1 s=READY pgs=647 cs=0 l=1 rev1=1 crypto rx=0x7f3700000c00 tx=0x7f3700016000 comp rx=0 tx=0).stop 2023-10-05T19:29:41.319 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:41.314+0000 7f371296be40 1 -- 172.21.15.167:0/2673647112 shutdown_connections 2023-10-05T19:29:41.319 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:41.314+0000 7f371296be40 1 --2- 172.21.15.167:0/2673647112 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55e813414420 0x55e81343bf40 unknown :-1 s=CLOSED pgs=647 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-05T19:29:41.319 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:41.314+0000 7f371296be40 1 -- 172.21.15.167:0/2673647112 >> 172.21.15.167:0/2673647112 conn(0x55e81340e730 msgr2=0x55e81342f7b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-05T19:29:41.319 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:41.314+0000 7f371296be40 1 -- 172.21.15.167:0/2673647112 shutdown_connections 2023-10-05T19:29:41.319 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:41.314+0000 7f371296be40 1 -- 172.21.15.167:0/2673647112 wait complete. 2023-10-05T19:29:41.320 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:41.314+0000 7f371296be40 1 -- 172.21.15.167:0/2022125644 >> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] conn(0x55e813470150 msgr2=0x55e81347a5e0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-05T19:29:41.321 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:41.314+0000 7f371296be40 1 --2- 172.21.15.167:0/2022125644 >> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] conn(0x55e813470150 0x55e81347a5e0 crc :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-05T19:29:41.321 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:41.314+0000 7f371296be40 1 -- 172.21.15.167:0/2022125644 >> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] conn(0x55e813474510 msgr2=0x55e813476920 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-05T19:29:41.321 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:41.314+0000 7f371296be40 1 --2- 172.21.15.167:0/2022125644 >> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] conn(0x55e813474510 0x55e813476920 crc :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-05T19:29:41.321 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:41.314+0000 7f371296be40 1 -- 172.21.15.167:0/2022125644 >> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] conn(0x55e81347f9a0 msgr2=0x55e813481d80 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-05T19:29:41.321 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:41.314+0000 7f371296be40 1 --2- 172.21.15.167:0/2022125644 >> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] conn(0x55e81347f9a0 0x55e813481d80 crc :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-05T19:29:41.321 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:41.314+0000 7f371296be40 1 -- 172.21.15.167:0/2022125644 >> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] conn(0x55e81347b7f0 msgr2=0x55e81347dc00 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-05T19:29:41.321 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:41.314+0000 7f371296be40 1 --2- 172.21.15.167:0/2022125644 >> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] conn(0x55e81347b7f0 0x55e81347dc00 crc :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-05T19:29:41.321 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:41.314+0000 7f371296be40 1 -- 172.21.15.167:0/2022125644 >> [v2:172.21.15.167:6824/29889,v1:172.21.15.167:6825/29889] conn(0x7f3650030c50 msgr2=0x7f3650033100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-05T19:29:41.321 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:41.314+0000 7f371296be40 1 --2- 172.21.15.167:0/2022125644 >> [v2:172.21.15.167:6824/29889,v1:172.21.15.167:6825/29889] conn(0x7f3650030c50 0x7f3650033100 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x55e813456af0 tx=0x7f370800aec0 comp rx=0 tx=0).stop 2023-10-05T19:29:41.321 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:41.314+0000 7f37104a0700 1 -- 172.21.15.167:0/2022125644 reap_dead start 2023-10-05T19:29:41.321 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:41.314+0000 7f371296be40 1 -- 172.21.15.167:0/2022125644 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55e813463df0 msgr2=0x55e813469920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-05T19:29:41.322 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:41.314+0000 7f371296be40 1 --2- 172.21.15.167:0/2022125644 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55e813463df0 0x55e813469920 secure :-1 s=READY pgs=649 cs=0 l=1 rev1=1 crypto rx=0x55e81345de70 tx=0x7f36f802f000 comp rx=0 tx=0).stop 2023-10-05T19:29:41.322 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:41.314+0000 7f371296be40 1 -- 172.21.15.167:0/2022125644 shutdown_connections 2023-10-05T19:29:41.322 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:41.314+0000 7f371296be40 1 --2- 172.21.15.167:0/2022125644 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55e813463df0 0x55e813469920 unknown :-1 s=CLOSED pgs=649 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-05T19:29:41.322 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:41.314+0000 7f371296be40 1 -- 172.21.15.167:0/2022125644 >> 172.21.15.167:0/2022125644 conn(0x55e81345f1b0 msgr2=0x55e81345d250 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-05T19:29:41.322 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:41.314+0000 7f371296be40 1 -- 172.21.15.167:0/2022125644 shutdown_connections 2023-10-05T19:29:41.322 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:29:41.314+0000 7f371296be40 1 -- 172.21.15.167:0/2022125644 wait complete. 2023-10-05T19:30:21.327 INFO:tasks.util.rgw:rgwadmin: client.0 : ['usage', 'trim', '--uid', 'foo'] 2023-10-05T19:30:21.328 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'ceph', 'usage', 'trim', '--uid', 'foo'] 2023-10-05T19:30:21.328 DEBUG:teuthology.orchestra.run.smithi167:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster ceph usage trim --uid foo 2023-10-05T19:30:21.371 INFO:teuthology.orchestra.run.smithi167.stderr:ignoring --setuser ceph since I am not root 2023-10-05T19:30:21.371 INFO:teuthology.orchestra.run.smithi167.stderr:ignoring --setgroup ceph since I am not root 2023-10-05T19:30:21.372 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:21.366+0000 7f727fff8e40 1 Processor -- start 2023-10-05T19:30:21.372 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:21.366+0000 7f727fff8e40 1 -- start start 2023-10-05T19:30:21.372 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:21.370+0000 7f727fff8e40 1 --2- >> v2:172.21.15.118:3300/0 conn(0x556f6de882f0 0x556f6de8a6d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-05T19:30:21.372 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:21.370+0000 7f727fff8e40 1 -- --> v1:172.21.15.118:6789/0 -- auth(proto 0 26 bytes epoch 0) v1 -- 0x556f6ddd0bd0 con 0x556f6ddf83d0 2023-10-05T19:30:21.372 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:21.370+0000 7f727fff8e40 1 -- --> v2:172.21.15.118:3300/0 -- mon_getmap magic: 0 v1 -- 0x556f6dd8dee0 con 0x556f6de882f0 2023-10-05T19:30:21.373 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:21.370+0000 7f727d32c700 1 --2- >> v2:172.21.15.118:3300/0 conn(0x556f6de882f0 0x556f6de8a6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-05T19:30:21.373 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:21.370+0000 7f727d32c700 1 --2- >> v2:172.21.15.118:3300/0 conn(0x556f6de882f0 0x556f6de8a6d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.118:3300/0 says I am v2:172.21.15.167:32886/0 (socket says 172.21.15.167:32886) 2023-10-05T19:30:21.373 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:21.370+0000 7f727d32c700 1 -- 172.21.15.167:0/3896728496 learned_addr learned my addr 172.21.15.167:0/3896728496 (peer_addr_for_me v2:172.21.15.167:0/0) 2023-10-05T19:30:21.373 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:21.370+0000 7f727db2d700 1 --1- 172.21.15.167:0/3896728496 >> v1:172.21.15.118:6789/0 conn(0x556f6ddf83d0 0x556f6de86bd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.118:6789/0 says I am v1:172.21.15.167:50778/0 (socket says 172.21.15.167:50778) 2023-10-05T19:30:21.373 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:21.370+0000 7f727d32c700 1 -- 172.21.15.167:0/3896728496 >> v1:172.21.15.118:6789/0 conn(0x556f6ddf83d0 legacy=0x556f6de86bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-05T19:30:21.373 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:21.370+0000 7f727d32c700 1 -- 172.21.15.167:0/3896728496 --> v2:172.21.15.118:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x556f6de8bc20 con 0x556f6de882f0 2023-10-05T19:30:21.374 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:21.370+0000 7f727d32c700 1 --2- 172.21.15.167:0/3896728496 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x556f6de882f0 0x556f6de8a6d0 secure :-1 s=READY pgs=651 cs=0 l=1 rev1=1 crypto rx=0x7f726800a5c0 tx=0x7f7268005e00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f48fe87c5594b67c server_cookie=0 in_seq=0 out_seq=0 2023-10-05T19:30:21.374 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:21.370+0000 7f727cb2b700 1 -- 172.21.15.167:0/3896728496 <== mon.0 v2:172.21.15.118:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f726800e510 con 0x556f6de882f0 2023-10-05T19:30:21.374 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:21.370+0000 7f727cb2b700 1 -- 172.21.15.167:0/3896728496 <== mon.0 v2:172.21.15.118:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f726800e670 con 0x556f6de882f0 2023-10-05T19:30:21.374 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:21.370+0000 7f727cb2b700 1 -- 172.21.15.167:0/3896728496 <== mon.0 v2:172.21.15.118:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f726800e920 con 0x556f6de882f0 2023-10-05T19:30:21.375 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:21.370+0000 7f727fff8e40 1 -- 172.21.15.167:0/3896728496 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x556f6de882f0 msgr2=0x556f6de8a6d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-05T19:30:21.375 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:21.370+0000 7f727fff8e40 1 --2- 172.21.15.167:0/3896728496 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x556f6de882f0 0x556f6de8a6d0 secure :-1 s=READY pgs=651 cs=0 l=1 rev1=1 crypto rx=0x7f726800a5c0 tx=0x7f7268005e00 comp rx=0 tx=0).stop 2023-10-05T19:30:21.375 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:21.370+0000 7f727fff8e40 1 -- 172.21.15.167:0/3896728496 shutdown_connections 2023-10-05T19:30:21.375 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:21.370+0000 7f727fff8e40 1 --2- 172.21.15.167:0/3896728496 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x556f6de882f0 0x556f6de8a6d0 unknown :-1 s=CLOSED pgs=651 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-05T19:30:21.375 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:21.370+0000 7f727fff8e40 1 -- 172.21.15.167:0/3896728496 >> 172.21.15.167:0/3896728496 conn(0x556f6de7e720 msgr2=0x556f6de80b30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-05T19:30:21.376 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:21.370+0000 7f727fff8e40 1 -- 172.21.15.167:0/3896728496 shutdown_connections 2023-10-05T19:30:21.376 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:21.370+0000 7f727fff8e40 1 -- 172.21.15.167:0/3896728496 wait complete. 2023-10-05T19:30:21.376 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:21.370+0000 7f727fff8e40 1 Processor -- start 2023-10-05T19:30:21.376 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:21.370+0000 7f727fff8e40 1 -- start start 2023-10-05T19:30:21.377 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:21.370+0000 7f727fff8e40 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x556f6ddf83d0 0x556f6de84150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-05T19:30:21.377 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:21.370+0000 7f727fff8e40 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x556f6dd51e30 con 0x556f6ddf83d0 2023-10-05T19:30:21.377 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:21.374+0000 7f727db2d700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x556f6ddf83d0 0x556f6de84150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-05T19:30:21.377 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:21.374+0000 7f727db2d700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x556f6ddf83d0 0x556f6de84150 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.118:3300/0 says I am v2:172.21.15.167:32902/0 (socket says 172.21.15.167:32902) 2023-10-05T19:30:21.377 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:21.374+0000 7f727db2d700 1 -- 172.21.15.167:0/1208160722 learned_addr learned my addr 172.21.15.167:0/1208160722 (peer_addr_for_me v2:172.21.15.167:0/0) 2023-10-05T19:30:21.377 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:21.374+0000 7f727db2d700 1 -- 172.21.15.167:0/1208160722 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f726800a180 con 0x556f6ddf83d0 2023-10-05T19:30:21.377 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:21.374+0000 7f727db2d700 1 --2- 172.21.15.167:0/1208160722 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x556f6ddf83d0 0x556f6de84150 secure :-1 s=READY pgs=652 cs=0 l=1 rev1=1 crypto rx=0x7f7270001cf0 tx=0x7f72700018a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ef986b8d411c05a8 server_cookie=0 in_seq=0 out_seq=0 2023-10-05T19:30:21.379 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:21.374+0000 7f72777fe700 1 -- 172.21.15.167:0/1208160722 <== mon.0 v2:172.21.15.118:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f7270014070 con 0x556f6ddf83d0 2023-10-05T19:30:21.379 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:21.374+0000 7f72777fe700 1 -- 172.21.15.167:0/1208160722 <== mon.0 v2:172.21.15.118:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f727000be10 con 0x556f6ddf83d0 2023-10-05T19:30:21.379 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:21.374+0000 7f72777fe700 1 -- 172.21.15.167:0/1208160722 <== mon.0 v2:172.21.15.118:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f727000f590 con 0x556f6ddf83d0 2023-10-05T19:30:21.379 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:21.374+0000 7f727fff8e40 1 -- 172.21.15.167:0/1208160722 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x556f6ddf83d0 msgr2=0x556f6de84150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-05T19:30:21.379 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:21.374+0000 7f727fff8e40 1 --2- 172.21.15.167:0/1208160722 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x556f6ddf83d0 0x556f6de84150 secure :-1 s=READY pgs=652 cs=0 l=1 rev1=1 crypto rx=0x7f7270001cf0 tx=0x7f72700018a0 comp rx=0 tx=0).stop 2023-10-05T19:30:21.379 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:21.374+0000 7f727fff8e40 1 -- 172.21.15.167:0/1208160722 shutdown_connections 2023-10-05T19:30:21.379 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:21.374+0000 7f727fff8e40 1 --2- 172.21.15.167:0/1208160722 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x556f6ddf83d0 0x556f6de84150 unknown :-1 s=CLOSED pgs=652 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-05T19:30:21.379 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:21.374+0000 7f727fff8e40 1 -- 172.21.15.167:0/1208160722 >> 172.21.15.167:0/1208160722 conn(0x556f6de7e720 msgr2=0x556f6de9eb50 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-05T19:30:21.379 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:21.374+0000 7f727fff8e40 1 -- 172.21.15.167:0/1208160722 shutdown_connections 2023-10-05T19:30:21.379 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:21.374+0000 7f727fff8e40 1 -- 172.21.15.167:0/1208160722 wait complete. 2023-10-05T19:30:21.380 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:21.374+0000 7f727fff8e40 1 Processor -- start 2023-10-05T19:30:21.380 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:21.374+0000 7f727fff8e40 1 -- start start 2023-10-05T19:30:21.380 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:21.374+0000 7f727fff8e40 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x556f6ddf83d0 0x556f6deabf60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-05T19:30:21.380 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:21.374+0000 7f727fff8e40 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x556f6dd24ed0 con 0x556f6ddf83d0 2023-10-05T19:30:21.381 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:21.374+0000 7f727db2d700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x556f6ddf83d0 0x556f6deabf60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-05T19:30:21.381 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:21.374+0000 7f727db2d700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x556f6ddf83d0 0x556f6deabf60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.118:3300/0 says I am v2:172.21.15.167:32910/0 (socket says 172.21.15.167:32910) 2023-10-05T19:30:21.381 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:21.374+0000 7f727db2d700 1 -- 172.21.15.167:0/164415728 learned_addr learned my addr 172.21.15.167:0/164415728 (peer_addr_for_me v2:172.21.15.167:0/0) 2023-10-05T19:30:21.381 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:21.378+0000 7f727db2d700 1 -- 172.21.15.167:0/164415728 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x556f6ddd0bd0 con 0x556f6ddf83d0 2023-10-05T19:30:21.381 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:21.378+0000 7f727db2d700 1 --2- 172.21.15.167:0/164415728 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x556f6ddf83d0 0x556f6deabf60 secure :-1 s=READY pgs=653 cs=0 l=1 rev1=1 crypto rx=0x7f72700018e0 tx=0x7f7270016000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-05T19:30:21.381 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:21.378+0000 7f7275ffb700 1 -- 172.21.15.167:0/164415728 <== mon.0 v2:172.21.15.118:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f727001c740 con 0x556f6ddf83d0 2023-10-05T19:30:21.381 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:21.378+0000 7f727fff8e40 1 -- 172.21.15.167:0/164415728 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x556f6deac4a0 con 0x556f6ddf83d0 2023-10-05T19:30:21.382 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:21.378+0000 7f7275ffb700 1 -- 172.21.15.167:0/164415728 <== mon.0 v2:172.21.15.118:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f727001c8a0 con 0x556f6ddf83d0 2023-10-05T19:30:21.382 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:21.378+0000 7f7275ffb700 1 -- 172.21.15.167:0/164415728 <== mon.0 v2:172.21.15.118:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f727001cb90 con 0x556f6ddf83d0 2023-10-05T19:30:21.382 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:21.378+0000 7f727fff8e40 1 -- 172.21.15.167:0/164415728 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x556f6deac8e0 con 0x556f6ddf83d0 2023-10-05T19:30:21.382 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:21.378+0000 7f727fff8e40 20 rados->read ofs=0 len=0 2023-10-05T19:30:21.383 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:21.378+0000 7f7275ffb700 1 -- 172.21.15.167:0/164415728 <== mon.0 v2:172.21.15.118:3300/0 4 ==== mgrmap(e 5) v1 ==== 40329+0+0 (secure 0 0 0) 0x7f7270023020 con 0x556f6ddf83d0 2023-10-05T19:30:21.383 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:21.378+0000 7f7275ffb700 1 --2- 172.21.15.167:0/164415728 >> [v2:172.21.15.167:6824/29889,v1:172.21.15.167:6825/29889] conn(0x7f7264030c00 0x7f72640330b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-05T19:30:21.383 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:21.378+0000 7f7275ffb700 1 -- 172.21.15.167:0/164415728 <== mon.0 v2:172.21.15.118:3300/0 5 ==== osd_map(32..32 src has 1..32) v4 ==== 6260+0+0 (secure 0 0 0) 0x7f7270008030 con 0x556f6ddf83d0 2023-10-05T19:30:21.383 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:21.378+0000 7f727d32c700 1 --2- 172.21.15.167:0/164415728 >> [v2:172.21.15.167:6824/29889,v1:172.21.15.167:6825/29889] conn(0x7f7264030c00 0x7f72640330b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-05T19:30:21.384 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:21.378+0000 7f727fff8e40 1 --2- 172.21.15.167:0/164415728 >> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] conn(0x556f6de882f0 0x556f6deb7b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-05T19:30:21.384 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:21.378+0000 7f727fff8e40 1 -- 172.21.15.167:0/164415728 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:1 5.12 5:49953fa1:::default.realm:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32) v8 -- 0x556f6deb8170 con 0x556f6de882f0 2023-10-05T19:30:21.384 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:21.378+0000 7f727e32e700 1 --2- 172.21.15.167:0/164415728 >> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] conn(0x556f6de882f0 0x556f6deb7b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-05T19:30:21.384 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:21.382+0000 7f727d32c700 1 --2- 172.21.15.167:0/164415728 >> [v2:172.21.15.167:6824/29889,v1:172.21.15.167:6825/29889] conn(0x7f7264030c00 0x7f72640330b0 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7f7268000f80 tx=0x7f7268013000 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-05T19:30:21.384 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:21.382+0000 7f727e32e700 1 --2- 172.21.15.167:0/164415728 >> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] conn(0x556f6de882f0 0x556f6deb7b10 crc :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-05T19:30:21.385 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:21.382+0000 7f727e32e700 1 -- 172.21.15.167:0/164415728 <== osd.1 v2:172.21.15.167:6800/30035 1 ==== osd_op_reply(1 default.realm [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 ==== 157+0+0 (crc 0 0 0) 0x7f727800a500 con 0x556f6de882f0 2023-10-05T19:30:21.385 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:21.382+0000 7f727fff8e40 20 rados_obj.operate() r=-2 bl.length=0 2023-10-05T19:30:21.385 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:21.382+0000 7f727fff8e40 20 realm 2023-10-05T19:30:21.385 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:21.382+0000 7f727fff8e40 20 rados->read ofs=0 len=0 2023-10-05T19:30:21.385 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:21.382+0000 7f727fff8e40 1 -- 172.21.15.167:0/164415728 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:2 5.12 5:49953fa1:::default.realm:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32) v8 -- 0x556f6deb5170 con 0x556f6de882f0 2023-10-05T19:30:21.387 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:21.382+0000 7f727e32e700 1 -- 172.21.15.167:0/164415728 <== osd.1 v2:172.21.15.167:6800/30035 2 ==== osd_op_reply(2 default.realm [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 ==== 157+0+0 (crc 0 0 0) 0x7f727800a500 con 0x556f6de882f0 2023-10-05T19:30:21.387 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:21.382+0000 7f727fff8e40 20 rados_obj.operate() r=-2 bl.length=0 2023-10-05T19:30:21.387 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:21.382+0000 7f727fff8e40 4 RGWPeriod::init failed to init realm id : (2) No such file or directory 2023-10-05T19:30:21.387 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:21.382+0000 7f727fff8e40 20 rados->read ofs=0 len=0 2023-10-05T19:30:21.387 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:21.382+0000 7f727fff8e40 1 -- 172.21.15.167:0/164415728 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:3 5.12 5:49953fa1:::default.realm:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32) v8 -- 0x556f6deb8c50 con 0x556f6de882f0 2023-10-05T19:30:21.387 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:21.382+0000 7f727e32e700 1 -- 172.21.15.167:0/164415728 <== osd.1 v2:172.21.15.167:6800/30035 3 ==== osd_op_reply(3 default.realm [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 ==== 157+0+0 (crc 0 0 0) 0x7f727800a500 con 0x556f6de882f0 2023-10-05T19:30:21.387 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:21.382+0000 7f727fff8e40 20 rados_obj.operate() r=-2 bl.length=0 2023-10-05T19:30:21.387 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:21.382+0000 7f727fff8e40 20 rados->read ofs=0 len=0 2023-10-05T19:30:21.387 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:21.382+0000 7f727fff8e40 1 -- 172.21.15.167:0/164415728 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:4 5.f 5:f43b6ece:::zone_names.default:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32) v8 -- 0x556f6deb8860 con 0x556f6de882f0 2023-10-05T19:30:21.388 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:21.382+0000 7f727e32e700 1 -- 172.21.15.167:0/164415728 <== osd.1 v2:172.21.15.167:6800/30035 4 ==== osd_op_reply(4 zone_names.default [read 0~46 out=46b] v0'0 uv1 ondisk = 0) v8 ==== 162+0+46 (crc 0 0 0) 0x7f727800a500 con 0x556f6de882f0 2023-10-05T19:30:21.388 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:21.382+0000 7f727fff8e40 20 rados_obj.operate() r=0 bl.length=46 2023-10-05T19:30:21.388 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:21.382+0000 7f727fff8e40 20 rados->read ofs=0 len=0 2023-10-05T19:30:21.388 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:21.382+0000 7f727fff8e40 1 -- 172.21.15.167:0/164415728 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:5 5.1d 5:b8374eb4:::zone_info.9a9df965-6612-45d4-a45f-6e3c889bacc2:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32) v8 -- 0x556f6deb9ae0 con 0x556f6de882f0 2023-10-05T19:30:21.389 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:21.382+0000 7f727e32e700 1 -- 172.21.15.167:0/164415728 <== osd.1 v2:172.21.15.167:6800/30035 5 ==== osd_op_reply(5 zone_info.9a9df965-6612-45d4-a45f-6e3c889bacc2 [read 0~874 out=874b] v0'0 uv2 ondisk = 0) v8 ==== 190+0+874 (crc 0 0 0) 0x7f727800a500 con 0x556f6de882f0 2023-10-05T19:30:21.389 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:21.382+0000 7f727fff8e40 20 rados_obj.operate() r=0 bl.length=874 2023-10-05T19:30:21.389 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:21.386+0000 7f727fff8e40 20 searching for the correct realm 2023-10-05T19:30:21.389 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:21.386+0000 7f727fff8e40 1 --2- 172.21.15.167:0/164415728 >> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] conn(0x556f6debb8a0 0x556f6debdc80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-05T19:30:21.389 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:21.386+0000 7f727fff8e40 1 -- 172.21.15.167:0/164415728 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:6 5.0 5:00000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x556f6debe1c0 con 0x556f6debb8a0 2023-10-05T19:30:21.389 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:21.386+0000 7f727db2d700 1 --2- 172.21.15.167:0/164415728 >> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] conn(0x556f6debb8a0 0x556f6debdc80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-05T19:30:21.389 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:21.386+0000 7f727db2d700 1 --2- 172.21.15.167:0/164415728 >> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] conn(0x556f6debb8a0 0x556f6debdc80 crc :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-05T19:30:21.390 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:21.386+0000 7f727db2d700 1 -- 172.21.15.167:0/164415728 <== osd.3 v2:172.21.15.167:6808/30036 1 ==== osd_op_reply(6 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f7270016070 con 0x556f6debb8a0 2023-10-05T19:30:21.390 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:21.386+0000 7f727db2d700 1 --2- 172.21.15.167:0/164415728 >> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] conn(0x7f727001a070 0x7f7270042210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-05T19:30:21.390 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:21.386+0000 7f727db2d700 1 -- 172.21.15.167:0/164415728 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:7 5.10 5:08000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x556f6debe1c0 con 0x7f727001a070 2023-10-05T19:30:21.390 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:21.386+0000 7f727d32c700 1 --2- 172.21.15.167:0/164415728 >> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] conn(0x7f727001a070 0x7f7270042210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-05T19:30:21.390 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:21.386+0000 7f727d32c700 1 --2- 172.21.15.167:0/164415728 >> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] conn(0x7f727001a070 0x7f7270042210 crc :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-05T19:30:21.391 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:21.386+0000 7f727d32c700 1 -- 172.21.15.167:0/164415728 <== osd.2 v2:172.21.15.167:6816/30037 1 ==== osd_op_reply(7 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f7268017050 con 0x7f727001a070 2023-10-05T19:30:21.391 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:21.386+0000 7f727d32c700 1 --2- 172.21.15.167:0/164415728 >> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] conn(0x7f7268010070 0x7f726801c430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-05T19:30:21.391 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:21.386+0000 7f727d32c700 1 -- 172.21.15.167:0/164415728 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:8 5.8 5:10000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x556f6debe1c0 con 0x7f7268010070 2023-10-05T19:30:21.391 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:21.386+0000 7f727e32e700 1 --2- 172.21.15.167:0/164415728 >> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] conn(0x7f7268010070 0x7f726801c430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-05T19:30:21.392 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:21.390+0000 7f727e32e700 1 --2- 172.21.15.167:0/164415728 >> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] conn(0x7f7268010070 0x7f726801c430 crc :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-05T19:30:21.393 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:21.390+0000 7f727e32e700 1 -- 172.21.15.167:0/164415728 <== osd.0 v2:172.21.15.118:6800/27690 1 ==== osd_op_reply(8 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f7278014040 con 0x7f7268010070 2023-10-05T19:30:21.393 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:21.390+0000 7f727e32e700 1 -- 172.21.15.167:0/164415728 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:9 5.18 5:18000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x556f6debe1c0 con 0x556f6de882f0 2023-10-05T19:30:21.393 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:21.390+0000 7f727e32e700 1 -- 172.21.15.167:0/164415728 <== osd.1 v2:172.21.15.167:6800/30035 6 ==== osd_op_reply(9 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f7278014040 con 0x556f6de882f0 2023-10-05T19:30:21.393 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:21.390+0000 7f727e32e700 1 -- 172.21.15.167:0/164415728 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:10 5.4 5:20000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x556f6debe1c0 con 0x556f6de882f0 2023-10-05T19:30:21.394 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:21.390+0000 7f727e32e700 1 -- 172.21.15.167:0/164415728 <== osd.1 v2:172.21.15.167:6800/30035 7 ==== osd_op_reply(10 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f7278014040 con 0x556f6de882f0 2023-10-05T19:30:21.394 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:21.390+0000 7f727e32e700 1 -- 172.21.15.167:0/164415728 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:11 5.14 5:28000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x556f6debe1c0 con 0x7f7268010070 2023-10-05T19:30:21.395 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:21.390+0000 7f727e32e700 1 -- 172.21.15.167:0/164415728 <== osd.0 v2:172.21.15.118:6800/27690 2 ==== osd_op_reply(11 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f7278014040 con 0x7f7268010070 2023-10-05T19:30:21.395 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:21.390+0000 7f727e32e700 1 -- 172.21.15.167:0/164415728 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:12 5.c 5:30000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x556f6debe1c0 con 0x7f7268010070 2023-10-05T19:30:21.396 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:21.394+0000 7f727e32e700 1 -- 172.21.15.167:0/164415728 <== osd.0 v2:172.21.15.118:6800/27690 3 ==== osd_op_reply(12 [pgnls start_epoch 32 out=112b] v23'1 uv1 ondisk = 1) v8 ==== 144+0+112 (crc 0 0 0) 0x7f7278014040 con 0x7f7268010070 2023-10-05T19:30:21.396 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:21.394+0000 7f727e32e700 1 -- 172.21.15.167:0/164415728 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:13 5.1c 5:38000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x556f6debe1c0 con 0x7f727001a070 2023-10-05T19:30:21.396 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:21.394+0000 7f727d32c700 1 -- 172.21.15.167:0/164415728 <== osd.2 v2:172.21.15.167:6816/30037 2 ==== osd_op_reply(13 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f7268017050 con 0x7f727001a070 2023-10-05T19:30:21.396 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:21.394+0000 7f727d32c700 1 -- 172.21.15.167:0/164415728 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:14 5.2 5:40000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x556f6debe1c0 con 0x556f6debb8a0 2023-10-05T19:30:21.397 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:21.394+0000 7f727db2d700 1 -- 172.21.15.167:0/164415728 <== osd.3 v2:172.21.15.167:6808/30036 2 ==== osd_op_reply(14 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f7270016070 con 0x556f6debb8a0 2023-10-05T19:30:21.397 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:21.394+0000 7f727db2d700 1 -- 172.21.15.167:0/164415728 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:15 5.12 5:48000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x556f6debe1c0 con 0x556f6de882f0 2023-10-05T19:30:21.398 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:21.394+0000 7f727e32e700 1 -- 172.21.15.167:0/164415728 <== osd.1 v2:172.21.15.167:6800/30035 8 ==== osd_op_reply(15 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f7278014040 con 0x556f6de882f0 2023-10-05T19:30:21.398 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:21.394+0000 7f727e32e700 1 -- 172.21.15.167:0/164415728 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:16 5.a 5:50000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x556f6debe1c0 con 0x7f7268010070 2023-10-05T19:30:21.399 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:21.394+0000 7f727e32e700 1 -- 172.21.15.167:0/164415728 <== osd.0 v2:172.21.15.118:6800/27690 4 ==== osd_op_reply(16 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f7278022040 con 0x7f7268010070 2023-10-05T19:30:21.399 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:21.394+0000 7f727e32e700 1 -- 172.21.15.167:0/164415728 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:17 5.1a 5:58000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x556f6debe1c0 con 0x556f6de882f0 2023-10-05T19:30:21.400 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:21.398+0000 7f727e32e700 1 -- 172.21.15.167:0/164415728 <== osd.1 v2:172.21.15.167:6800/30035 9 ==== osd_op_reply(17 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f7278022040 con 0x556f6de882f0 2023-10-05T19:30:21.400 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:21.398+0000 7f727e32e700 1 -- 172.21.15.167:0/164415728 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:18 5.6 5:60000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x556f6debe1c0 con 0x7f727001a070 2023-10-05T19:30:21.401 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:21.398+0000 7f727d32c700 1 -- 172.21.15.167:0/164415728 <== osd.2 v2:172.21.15.167:6816/30037 3 ==== osd_op_reply(18 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f7268017050 con 0x7f727001a070 2023-10-05T19:30:21.401 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:21.398+0000 7f727d32c700 1 -- 172.21.15.167:0/164415728 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:19 5.16 5:68000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x556f6debe1c0 con 0x556f6debb8a0 2023-10-05T19:30:21.401 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:21.398+0000 7f727db2d700 1 -- 172.21.15.167:0/164415728 <== osd.3 v2:172.21.15.167:6808/30036 3 ==== osd_op_reply(19 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f7270016070 con 0x556f6debb8a0 2023-10-05T19:30:21.402 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:21.398+0000 7f727db2d700 1 -- 172.21.15.167:0/164415728 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:20 5.e 5:70000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x556f6debe1c0 con 0x556f6debb8a0 2023-10-05T19:30:21.403 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:21.398+0000 7f727db2d700 1 -- 172.21.15.167:0/164415728 <== osd.3 v2:172.21.15.167:6808/30036 4 ==== osd_op_reply(20 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f7270016070 con 0x556f6debb8a0 2023-10-05T19:30:21.403 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:21.398+0000 7f727db2d700 1 -- 172.21.15.167:0/164415728 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:21 5.1e 5:78000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x556f6debe1c0 con 0x7f7268010070 2023-10-05T19:30:21.404 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:21.398+0000 7f727e32e700 1 -- 172.21.15.167:0/164415728 <== osd.0 v2:172.21.15.118:6800/27690 5 ==== osd_op_reply(21 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f7278022040 con 0x7f7268010070 2023-10-05T19:30:21.404 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:21.398+0000 7f727e32e700 1 -- 172.21.15.167:0/164415728 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:22 5.1 5:80000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x556f6debe1c0 con 0x556f6debb8a0 2023-10-05T19:30:21.405 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:21.402+0000 7f727db2d700 1 -- 172.21.15.167:0/164415728 <== osd.3 v2:172.21.15.167:6808/30036 5 ==== osd_op_reply(22 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f7270016070 con 0x556f6debb8a0 2023-10-05T19:30:21.405 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:21.402+0000 7f727db2d700 1 -- 172.21.15.167:0/164415728 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:23 5.11 5:88000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x556f6debe1c0 con 0x556f6debb8a0 2023-10-05T19:30:21.405 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:21.402+0000 7f727db2d700 1 -- 172.21.15.167:0/164415728 <== osd.3 v2:172.21.15.167:6808/30036 6 ==== osd_op_reply(23 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f7270016070 con 0x556f6debb8a0 2023-10-05T19:30:21.405 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:21.402+0000 7f727db2d700 1 -- 172.21.15.167:0/164415728 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:24 5.9 5:90000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x556f6debe1c0 con 0x556f6de882f0 2023-10-05T19:30:21.406 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:21.402+0000 7f727e32e700 1 -- 172.21.15.167:0/164415728 <== osd.1 v2:172.21.15.167:6800/30035 10 ==== osd_op_reply(24 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f7278022040 con 0x556f6de882f0 2023-10-05T19:30:21.406 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:21.402+0000 7f727e32e700 1 -- 172.21.15.167:0/164415728 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:25 5.19 5:98000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x556f6debe1c0 con 0x556f6de882f0 2023-10-05T19:30:21.406 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:21.402+0000 7f727e32e700 1 -- 172.21.15.167:0/164415728 <== osd.1 v2:172.21.15.167:6800/30035 11 ==== osd_op_reply(25 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f7278022040 con 0x556f6de882f0 2023-10-05T19:30:21.407 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:21.402+0000 7f727e32e700 1 -- 172.21.15.167:0/164415728 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:26 5.5 5:a0000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x556f6debe1c0 con 0x7f7268010070 2023-10-05T19:30:21.408 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:21.406+0000 7f727e32e700 1 -- 172.21.15.167:0/164415728 <== osd.0 v2:172.21.15.118:6800/27690 6 ==== osd_op_reply(26 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f7278022040 con 0x7f7268010070 2023-10-05T19:30:21.408 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:21.406+0000 7f727e32e700 1 -- 172.21.15.167:0/164415728 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:27 5.15 5:a8000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x556f6debe1c0 con 0x556f6debb8a0 2023-10-05T19:30:21.408 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:21.406+0000 7f727db2d700 1 -- 172.21.15.167:0/164415728 <== osd.3 v2:172.21.15.167:6808/30036 7 ==== osd_op_reply(27 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f7270016070 con 0x556f6debb8a0 2023-10-05T19:30:21.409 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:21.406+0000 7f727db2d700 1 -- 172.21.15.167:0/164415728 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:28 5.d 5:b0000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x556f6debe1c0 con 0x7f727001a070 2023-10-05T19:30:21.409 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:21.406+0000 7f727d32c700 1 -- 172.21.15.167:0/164415728 <== osd.2 v2:172.21.15.167:6816/30037 4 ==== osd_op_reply(28 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f7268017050 con 0x7f727001a070 2023-10-05T19:30:21.409 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:21.406+0000 7f727d32c700 1 -- 172.21.15.167:0/164415728 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:29 5.1d 5:b8000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x556f6debe1c0 con 0x556f6de882f0 2023-10-05T19:30:21.410 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:21.406+0000 7f727e32e700 1 -- 172.21.15.167:0/164415728 <== osd.1 v2:172.21.15.167:6800/30035 12 ==== osd_op_reply(29 [pgnls start_epoch 32 out=107b] v30'2 uv2 ondisk = 1) v8 ==== 144+0+107 (crc 0 0 0) 0x7f7278022040 con 0x556f6de882f0 2023-10-05T19:30:21.410 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:21.406+0000 7f727e32e700 1 -- 172.21.15.167:0/164415728 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:30 5.3 5:c0000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x556f6debe1c0 con 0x7f7268010070 2023-10-05T19:30:21.411 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:21.406+0000 7f727e32e700 1 -- 172.21.15.167:0/164415728 <== osd.0 v2:172.21.15.118:6800/27690 7 ==== osd_op_reply(30 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f7278022040 con 0x7f7268010070 2023-10-05T19:30:21.411 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:21.406+0000 7f727e32e700 1 -- 172.21.15.167:0/164415728 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:31 5.13 5:c8000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x556f6debe1c0 con 0x556f6debb8a0 2023-10-05T19:30:21.412 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:21.410+0000 7f727db2d700 1 -- 172.21.15.167:0/164415728 <== osd.3 v2:172.21.15.167:6808/30036 8 ==== osd_op_reply(31 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f7270016070 con 0x556f6debb8a0 2023-10-05T19:30:21.412 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:21.410+0000 7f727db2d700 1 -- 172.21.15.167:0/164415728 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:32 5.b 5:d0000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x556f6debe1c0 con 0x7f7268010070 2023-10-05T19:30:21.413 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:21.410+0000 7f727e32e700 1 -- 172.21.15.167:0/164415728 <== osd.0 v2:172.21.15.118:6800/27690 8 ==== osd_op_reply(32 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f7278022040 con 0x7f7268010070 2023-10-05T19:30:21.413 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:21.410+0000 7f727e32e700 1 -- 172.21.15.167:0/164415728 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:33 5.1b 5:d8000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x556f6debe1c0 con 0x7f727001a070 2023-10-05T19:30:21.414 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:21.410+0000 7f727d32c700 1 -- 172.21.15.167:0/164415728 <== osd.2 v2:172.21.15.167:6816/30037 5 ==== osd_op_reply(33 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f7268017050 con 0x7f727001a070 2023-10-05T19:30:21.414 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:21.410+0000 7f727d32c700 1 -- 172.21.15.167:0/164415728 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:34 5.7 5:e0000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x556f6debe1c0 con 0x556f6debb8a0 2023-10-05T19:30:21.415 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:21.410+0000 7f727db2d700 1 -- 172.21.15.167:0/164415728 <== osd.3 v2:172.21.15.167:6808/30036 9 ==== osd_op_reply(34 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f7270016070 con 0x556f6debb8a0 2023-10-05T19:30:21.415 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:21.410+0000 7f727db2d700 1 -- 172.21.15.167:0/164415728 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:35 5.17 5:e8000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x556f6debe1c0 con 0x7f7268010070 2023-10-05T19:30:21.416 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:21.414+0000 7f727e32e700 1 -- 172.21.15.167:0/164415728 <== osd.0 v2:172.21.15.118:6800/27690 9 ==== osd_op_reply(35 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f7278022040 con 0x7f7268010070 2023-10-05T19:30:21.416 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:21.414+0000 7f727e32e700 1 -- 172.21.15.167:0/164415728 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:36 5.f 5:f0000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x556f6debe1c0 con 0x556f6de882f0 2023-10-05T19:30:21.417 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:21.414+0000 7f727e32e700 1 -- 172.21.15.167:0/164415728 <== osd.1 v2:172.21.15.167:6800/30035 13 ==== osd_op_reply(36 [pgnls start_epoch 32 out=115b] v23'2 uv2 ondisk = 1) v8 ==== 144+0+115 (crc 0 0 0) 0x7f727802f080 con 0x556f6de882f0 2023-10-05T19:30:21.417 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:21.414+0000 7f727e32e700 1 -- 172.21.15.167:0/164415728 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:37 5.1f 5:f8000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x556f6debe1c0 con 0x7f727001a070 2023-10-05T19:30:21.418 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:21.414+0000 7f727d32c700 1 -- 172.21.15.167:0/164415728 <== osd.2 v2:172.21.15.167:6816/30037 6 ==== osd_op_reply(37 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f7268017050 con 0x7f727001a070 2023-10-05T19:30:21.418 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:21.414+0000 7f727fff8e40 20 RGWRados::pool_iterate: got zonegroup_info.4f1f719d-9648-417b-b10c-e5d939825419 2023-10-05T19:30:21.418 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:21.414+0000 7f727fff8e40 20 RGWRados::pool_iterate: got zone_info.9a9df965-6612-45d4-a45f-6e3c889bacc2 2023-10-05T19:30:21.418 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:21.414+0000 7f727fff8e40 20 RGWRados::pool_iterate: got zone_names.default 2023-10-05T19:30:21.418 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:21.414+0000 7f727fff8e40 20 RGWRados::pool_iterate: got zonegroups_names.default 2023-10-05T19:30:21.419 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:21.414+0000 7f727fff8e40 20 rados->read ofs=0 len=0 2023-10-05T19:30:21.419 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:21.414+0000 7f727fff8e40 1 -- 172.21.15.167:0/164415728 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:38 5.12 5:49953fa1:::default.realm:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32) v8 -- 0x556f6deba650 con 0x556f6de882f0 2023-10-05T19:30:21.420 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:21.414+0000 7f727e32e700 1 -- 172.21.15.167:0/164415728 <== osd.1 v2:172.21.15.167:6800/30035 14 ==== osd_op_reply(38 default.realm [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 ==== 157+0+0 (crc 0 0 0) 0x7f727802f080 con 0x556f6de882f0 2023-10-05T19:30:21.420 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:21.414+0000 7f727fff8e40 20 rados_obj.operate() r=-2 bl.length=0 2023-10-05T19:30:21.420 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:21.414+0000 7f727fff8e40 20 rados->read ofs=0 len=0 2023-10-05T19:30:21.420 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:21.414+0000 7f727fff8e40 1 -- 172.21.15.167:0/164415728 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:39 5.f 5:f4c53578:::zonegroups_names.default:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32) v8 -- 0x556f6debaf40 con 0x556f6de882f0 2023-10-05T19:30:21.420 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:21.414+0000 7f727e32e700 1 -- 172.21.15.167:0/164415728 <== osd.1 v2:172.21.15.167:6800/30035 15 ==== osd_op_reply(39 zonegroups_names.default [read 0~46 out=46b] v0'0 uv2 ondisk = 0) v8 ==== 168+0+46 (crc 0 0 0) 0x7f727802f080 con 0x556f6de882f0 2023-10-05T19:30:21.420 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:21.414+0000 7f727fff8e40 20 rados_obj.operate() r=0 bl.length=46 2023-10-05T19:30:21.420 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:21.418+0000 7f727fff8e40 20 rados->read ofs=0 len=0 2023-10-05T19:30:21.420 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:21.418+0000 7f727fff8e40 1 -- 172.21.15.167:0/164415728 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:40 5.c 5:30d299a5:::zonegroup_info.4f1f719d-9648-417b-b10c-e5d939825419:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32) v8 -- 0x556f6debe600 con 0x7f7268010070 2023-10-05T19:30:21.421 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:21.418+0000 7f727e32e700 1 -- 172.21.15.167:0/164415728 <== osd.0 v2:172.21.15.118:6800/27690 10 ==== osd_op_reply(40 zonegroup_info.4f1f719d-9648-417b-b10c-e5d939825419 [read 0~362 out=362b] v0'0 uv1 ondisk = 0) v8 ==== 195+0+362 (crc 0 0 0) 0x7f727802f080 con 0x7f7268010070 2023-10-05T19:30:21.421 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:21.418+0000 7f727fff8e40 20 rados_obj.operate() r=0 bl.length=362 2023-10-05T19:30:21.421 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:21.418+0000 7f727fff8e40 20 rados->read ofs=0 len=0 2023-10-05T19:30:21.421 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:21.418+0000 7f727fff8e40 1 -- 172.21.15.167:0/164415728 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:41 5.13 5:cbbf6c60:::converted:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32) v8 -- 0x556f6debe600 con 0x556f6debb8a0 2023-10-05T19:30:21.422 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:21.418+0000 7f727db2d700 1 -- 172.21.15.167:0/164415728 <== osd.3 v2:172.21.15.167:6808/30036 10 ==== osd_op_reply(41 converted [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 ==== 153+0+0 (crc 0 0 0) 0x7f7270016070 con 0x556f6debb8a0 2023-10-05T19:30:21.422 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:21.418+0000 7f727fff8e40 20 rados_obj.operate() r=-2 bl.length=0 2023-10-05T19:30:21.422 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:21.418+0000 7f727fff8e40 20 rados->read ofs=0 len=0 2023-10-05T19:30:21.422 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:21.418+0000 7f727fff8e40 1 -- 172.21.15.167:0/164415728 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:42 5.12 5:49953fa1:::default.realm:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32) v8 -- 0x556f6debe9b0 con 0x556f6de882f0 2023-10-05T19:30:21.424 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:21.418+0000 7f727e32e700 1 -- 172.21.15.167:0/164415728 <== osd.1 v2:172.21.15.167:6800/30035 16 ==== osd_op_reply(42 default.realm [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 ==== 157+0+0 (crc 0 0 0) 0x7f727802f080 con 0x556f6de882f0 2023-10-05T19:30:21.424 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:21.418+0000 7f727fff8e40 20 rados_obj.operate() r=-2 bl.length=0 2023-10-05T19:30:21.424 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:21.418+0000 7f727fff8e40 20 rados->read ofs=0 len=0 2023-10-05T19:30:21.424 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:21.418+0000 7f727fff8e40 1 -- 172.21.15.167:0/164415728 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:43 5.f 5:f4c53578:::zonegroups_names.default:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32) v8 -- 0x556f6debede0 con 0x556f6de882f0 2023-10-05T19:30:21.424 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:21.418+0000 7f727e32e700 1 -- 172.21.15.167:0/164415728 <== osd.1 v2:172.21.15.167:6800/30035 17 ==== osd_op_reply(43 zonegroups_names.default [read 0~46 out=46b] v0'0 uv2 ondisk = 0) v8 ==== 168+0+46 (crc 0 0 0) 0x7f727802f080 con 0x556f6de882f0 2023-10-05T19:30:21.424 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:21.418+0000 7f727fff8e40 20 rados_obj.operate() r=0 bl.length=46 2023-10-05T19:30:21.424 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:21.418+0000 7f727fff8e40 1 -- 172.21.15.167:0/164415728 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:44 5.0 5:00000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x556f6debfdf0 con 0x556f6debb8a0 2023-10-05T19:30:21.425 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:21.422+0000 7f727db2d700 1 -- 172.21.15.167:0/164415728 <== osd.3 v2:172.21.15.167:6808/30036 11 ==== osd_op_reply(44 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f7270016070 con 0x556f6debb8a0 2023-10-05T19:30:21.425 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:21.422+0000 7f727db2d700 1 -- 172.21.15.167:0/164415728 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:45 5.10 5:08000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x556f6debfdf0 con 0x7f727001a070 2023-10-05T19:30:21.426 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:21.422+0000 7f727d32c700 1 -- 172.21.15.167:0/164415728 <== osd.2 v2:172.21.15.167:6816/30037 7 ==== osd_op_reply(45 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f7268017050 con 0x7f727001a070 2023-10-05T19:30:21.426 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:21.422+0000 7f727d32c700 1 -- 172.21.15.167:0/164415728 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:46 5.8 5:10000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x556f6debfdf0 con 0x7f7268010070 2023-10-05T19:30:21.427 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:21.422+0000 7f727e32e700 1 -- 172.21.15.167:0/164415728 <== osd.0 v2:172.21.15.118:6800/27690 11 ==== osd_op_reply(46 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f727802f080 con 0x7f7268010070 2023-10-05T19:30:21.427 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:21.422+0000 7f727e32e700 1 -- 172.21.15.167:0/164415728 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:47 5.18 5:18000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x556f6debaf40 con 0x556f6de882f0 2023-10-05T19:30:21.428 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:21.426+0000 7f727e32e700 1 -- 172.21.15.167:0/164415728 <== osd.1 v2:172.21.15.167:6800/30035 18 ==== osd_op_reply(47 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f727802f080 con 0x556f6de882f0 2023-10-05T19:30:21.428 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:21.426+0000 7f727e32e700 1 -- 172.21.15.167:0/164415728 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:48 5.4 5:20000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x556f6debaf40 con 0x556f6de882f0 2023-10-05T19:30:21.428 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:21.426+0000 7f727e32e700 1 -- 172.21.15.167:0/164415728 <== osd.1 v2:172.21.15.167:6800/30035 19 ==== osd_op_reply(48 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f727802f080 con 0x556f6de882f0 2023-10-05T19:30:21.429 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:21.426+0000 7f727e32e700 1 -- 172.21.15.167:0/164415728 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:49 5.14 5:28000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x556f6debaf40 con 0x7f7268010070 2023-10-05T19:30:21.430 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:21.426+0000 7f727e32e700 1 -- 172.21.15.167:0/164415728 <== osd.0 v2:172.21.15.118:6800/27690 12 ==== osd_op_reply(49 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f727802f080 con 0x7f7268010070 2023-10-05T19:30:21.430 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:21.426+0000 7f727e32e700 1 -- 172.21.15.167:0/164415728 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:50 5.c 5:30000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x556f6debaf40 con 0x7f7268010070 2023-10-05T19:30:21.431 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:21.426+0000 7f727e32e700 1 -- 172.21.15.167:0/164415728 <== osd.0 v2:172.21.15.118:6800/27690 13 ==== osd_op_reply(50 [pgnls start_epoch 32 out=112b] v23'1 uv1 ondisk = 1) v8 ==== 144+0+112 (crc 0 0 0) 0x7f727802f080 con 0x7f7268010070 2023-10-05T19:30:21.431 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:21.426+0000 7f727e32e700 1 -- 172.21.15.167:0/164415728 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:51 5.1c 5:38000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x556f6debaf40 con 0x7f727001a070 2023-10-05T19:30:21.432 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:21.430+0000 7f727d32c700 1 -- 172.21.15.167:0/164415728 <== osd.2 v2:172.21.15.167:6816/30037 8 ==== osd_op_reply(51 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f7268017050 con 0x7f727001a070 2023-10-05T19:30:21.432 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:21.430+0000 7f727d32c700 1 -- 172.21.15.167:0/164415728 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:52 5.2 5:40000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x556f6debaf40 con 0x556f6debb8a0 2023-10-05T19:30:21.432 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:21.430+0000 7f727db2d700 1 -- 172.21.15.167:0/164415728 <== osd.3 v2:172.21.15.167:6808/30036 12 ==== osd_op_reply(52 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f7270016070 con 0x556f6debb8a0 2023-10-05T19:30:21.433 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:21.430+0000 7f727db2d700 1 -- 172.21.15.167:0/164415728 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:53 5.12 5:48000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x556f6debaf40 con 0x556f6de882f0 2023-10-05T19:30:21.433 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:21.430+0000 7f727e32e700 1 -- 172.21.15.167:0/164415728 <== osd.1 v2:172.21.15.167:6800/30035 20 ==== osd_op_reply(53 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f727802f080 con 0x556f6de882f0 2023-10-05T19:30:21.434 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:21.430+0000 7f727e32e700 1 -- 172.21.15.167:0/164415728 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:54 5.a 5:50000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x556f6debaf40 con 0x7f7268010070 2023-10-05T19:30:21.434 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:21.430+0000 7f727e32e700 1 -- 172.21.15.167:0/164415728 <== osd.0 v2:172.21.15.118:6800/27690 14 ==== osd_op_reply(54 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f727802f080 con 0x7f7268010070 2023-10-05T19:30:21.435 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:21.430+0000 7f727e32e700 1 -- 172.21.15.167:0/164415728 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:55 5.1a 5:58000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x556f6debaf40 con 0x556f6de882f0 2023-10-05T19:30:21.435 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:21.430+0000 7f727e32e700 1 -- 172.21.15.167:0/164415728 <== osd.1 v2:172.21.15.167:6800/30035 21 ==== osd_op_reply(55 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f727802f080 con 0x556f6de882f0 2023-10-05T19:30:21.436 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:21.430+0000 7f727e32e700 1 -- 172.21.15.167:0/164415728 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:56 5.6 5:60000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x556f6debaf40 con 0x7f727001a070 2023-10-05T19:30:21.436 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:21.434+0000 7f727d32c700 1 -- 172.21.15.167:0/164415728 <== osd.2 v2:172.21.15.167:6816/30037 9 ==== osd_op_reply(56 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f7268017050 con 0x7f727001a070 2023-10-05T19:30:21.436 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:21.434+0000 7f727d32c700 1 -- 172.21.15.167:0/164415728 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:57 5.16 5:68000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x556f6debaf40 con 0x556f6debb8a0 2023-10-05T19:30:21.437 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:21.434+0000 7f727db2d700 1 -- 172.21.15.167:0/164415728 <== osd.3 v2:172.21.15.167:6808/30036 13 ==== osd_op_reply(57 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f7270016070 con 0x556f6debb8a0 2023-10-05T19:30:21.437 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:21.434+0000 7f727db2d700 1 -- 172.21.15.167:0/164415728 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:58 5.e 5:70000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x556f6debaf40 con 0x556f6debb8a0 2023-10-05T19:30:21.438 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:21.434+0000 7f727db2d700 1 -- 172.21.15.167:0/164415728 <== osd.3 v2:172.21.15.167:6808/30036 14 ==== osd_op_reply(58 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f7270016070 con 0x556f6debb8a0 2023-10-05T19:30:21.438 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:21.434+0000 7f727db2d700 1 -- 172.21.15.167:0/164415728 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:59 5.1e 5:78000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x556f6debaf40 con 0x7f7268010070 2023-10-05T19:30:21.439 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:21.434+0000 7f727e32e700 1 -- 172.21.15.167:0/164415728 <== osd.0 v2:172.21.15.118:6800/27690 15 ==== osd_op_reply(59 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f727802f080 con 0x7f7268010070 2023-10-05T19:30:21.439 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:21.434+0000 7f727e32e700 1 -- 172.21.15.167:0/164415728 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:60 5.1 5:80000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x556f6debaf40 con 0x556f6debb8a0 2023-10-05T19:30:21.440 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:21.438+0000 7f727db2d700 1 -- 172.21.15.167:0/164415728 <== osd.3 v2:172.21.15.167:6808/30036 15 ==== osd_op_reply(60 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f7270016070 con 0x556f6debb8a0 2023-10-05T19:30:21.440 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:21.438+0000 7f727db2d700 1 -- 172.21.15.167:0/164415728 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:61 5.11 5:88000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x556f6debaf40 con 0x556f6debb8a0 2023-10-05T19:30:21.441 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:21.438+0000 7f727db2d700 1 -- 172.21.15.167:0/164415728 <== osd.3 v2:172.21.15.167:6808/30036 16 ==== osd_op_reply(61 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f7270016070 con 0x556f6debb8a0 2023-10-05T19:30:21.441 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:21.438+0000 7f727db2d700 1 -- 172.21.15.167:0/164415728 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:62 5.9 5:90000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x556f6debaf40 con 0x556f6de882f0 2023-10-05T19:30:21.442 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:21.438+0000 7f727e32e700 1 -- 172.21.15.167:0/164415728 <== osd.1 v2:172.21.15.167:6800/30035 22 ==== osd_op_reply(62 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f727802f080 con 0x556f6de882f0 2023-10-05T19:30:21.442 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:21.438+0000 7f727e32e700 1 -- 172.21.15.167:0/164415728 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:63 5.19 5:98000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x556f6debaf40 con 0x556f6de882f0 2023-10-05T19:30:21.442 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:21.438+0000 7f727e32e700 1 -- 172.21.15.167:0/164415728 <== osd.1 v2:172.21.15.167:6800/30035 23 ==== osd_op_reply(63 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f727802f080 con 0x556f6de882f0 2023-10-05T19:30:21.443 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:21.438+0000 7f727e32e700 1 -- 172.21.15.167:0/164415728 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:64 5.5 5:a0000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x556f6debaf40 con 0x7f7268010070 2023-10-05T19:30:21.443 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:21.442+0000 7f727e32e700 1 -- 172.21.15.167:0/164415728 <== osd.0 v2:172.21.15.118:6800/27690 16 ==== osd_op_reply(64 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f727802f080 con 0x7f7268010070 2023-10-05T19:30:21.444 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:21.442+0000 7f727e32e700 1 -- 172.21.15.167:0/164415728 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:65 5.15 5:a8000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x556f6debaf40 con 0x556f6debb8a0 2023-10-05T19:30:21.444 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:21.442+0000 7f727db2d700 1 -- 172.21.15.167:0/164415728 <== osd.3 v2:172.21.15.167:6808/30036 17 ==== osd_op_reply(65 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f7270016070 con 0x556f6debb8a0 2023-10-05T19:30:21.444 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:21.442+0000 7f727db2d700 1 -- 172.21.15.167:0/164415728 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:66 5.d 5:b0000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x556f6debaf40 con 0x7f727001a070 2023-10-05T19:30:21.445 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:21.442+0000 7f727d32c700 1 -- 172.21.15.167:0/164415728 <== osd.2 v2:172.21.15.167:6816/30037 10 ==== osd_op_reply(66 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f7268017050 con 0x7f727001a070 2023-10-05T19:30:21.445 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:21.442+0000 7f727d32c700 1 -- 172.21.15.167:0/164415728 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:67 5.1d 5:b8000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x556f6debaf40 con 0x556f6de882f0 2023-10-05T19:30:21.446 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:21.442+0000 7f727e32e700 1 -- 172.21.15.167:0/164415728 <== osd.1 v2:172.21.15.167:6800/30035 24 ==== osd_op_reply(67 [pgnls start_epoch 32 out=107b] v30'2 uv2 ondisk = 1) v8 ==== 144+0+107 (crc 0 0 0) 0x7f727802f080 con 0x556f6de882f0 2023-10-05T19:30:21.446 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:21.442+0000 7f727e32e700 1 -- 172.21.15.167:0/164415728 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:68 5.3 5:c0000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x556f6debaf40 con 0x7f7268010070 2023-10-05T19:30:21.450 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:21.442+0000 7f727e32e700 1 -- 172.21.15.167:0/164415728 <== osd.0 v2:172.21.15.118:6800/27690 17 ==== osd_op_reply(68 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f727802f080 con 0x7f7268010070 2023-10-05T19:30:21.450 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:21.442+0000 7f727e32e700 1 -- 172.21.15.167:0/164415728 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:69 5.13 5:c8000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x556f6debaf40 con 0x556f6debb8a0 2023-10-05T19:30:21.450 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:21.446+0000 7f727db2d700 1 -- 172.21.15.167:0/164415728 <== osd.3 v2:172.21.15.167:6808/30036 18 ==== osd_op_reply(69 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f7270016070 con 0x556f6debb8a0 2023-10-05T19:30:21.450 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:21.446+0000 7f727db2d700 1 -- 172.21.15.167:0/164415728 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:70 5.b 5:d0000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x556f6debaf40 con 0x7f7268010070 2023-10-05T19:30:21.450 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:21.446+0000 7f727e32e700 1 -- 172.21.15.167:0/164415728 <== osd.0 v2:172.21.15.118:6800/27690 18 ==== osd_op_reply(70 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f727802f080 con 0x7f7268010070 2023-10-05T19:30:21.450 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:21.446+0000 7f727e32e700 1 -- 172.21.15.167:0/164415728 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:71 5.1b 5:d8000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x556f6debaf40 con 0x7f727001a070 2023-10-05T19:30:21.450 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:21.446+0000 7f727d32c700 1 -- 172.21.15.167:0/164415728 <== osd.2 v2:172.21.15.167:6816/30037 11 ==== osd_op_reply(71 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f7268017050 con 0x7f727001a070 2023-10-05T19:30:21.451 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:21.446+0000 7f727d32c700 1 -- 172.21.15.167:0/164415728 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:72 5.7 5:e0000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x556f6debaf40 con 0x556f6debb8a0 2023-10-05T19:30:21.451 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:21.446+0000 7f727db2d700 1 -- 172.21.15.167:0/164415728 <== osd.3 v2:172.21.15.167:6808/30036 19 ==== osd_op_reply(72 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f7270016070 con 0x556f6debb8a0 2023-10-05T19:30:21.451 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:21.446+0000 7f727db2d700 1 -- 172.21.15.167:0/164415728 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:73 5.17 5:e8000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x556f6debaf40 con 0x7f7268010070 2023-10-05T19:30:21.452 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:21.450+0000 7f727e32e700 1 -- 172.21.15.167:0/164415728 <== osd.0 v2:172.21.15.118:6800/27690 19 ==== osd_op_reply(73 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f727802f080 con 0x7f7268010070 2023-10-05T19:30:21.452 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:21.450+0000 7f727e32e700 1 -- 172.21.15.167:0/164415728 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:74 5.f 5:f0000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x556f6debaf40 con 0x556f6de882f0 2023-10-05T19:30:21.453 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:21.450+0000 7f727e32e700 1 -- 172.21.15.167:0/164415728 <== osd.1 v2:172.21.15.167:6800/30035 25 ==== osd_op_reply(74 [pgnls start_epoch 32 out=115b] v23'2 uv2 ondisk = 1) v8 ==== 144+0+115 (crc 0 0 0) 0x7f727802f080 con 0x556f6de882f0 2023-10-05T19:30:21.453 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:21.450+0000 7f727e32e700 1 -- 172.21.15.167:0/164415728 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:75 5.1f 5:f8000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x556f6debaf40 con 0x7f727001a070 2023-10-05T19:30:21.454 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:21.450+0000 7f727d32c700 1 -- 172.21.15.167:0/164415728 <== osd.2 v2:172.21.15.167:6816/30037 12 ==== osd_op_reply(75 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f7268017050 con 0x7f727001a070 2023-10-05T19:30:21.454 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:21.450+0000 7f727fff8e40 20 RGWRados::pool_iterate: got zonegroup_info.4f1f719d-9648-417b-b10c-e5d939825419 2023-10-05T19:30:21.454 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:21.450+0000 7f727fff8e40 20 RGWRados::pool_iterate: got zone_info.9a9df965-6612-45d4-a45f-6e3c889bacc2 2023-10-05T19:30:21.454 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:21.450+0000 7f727fff8e40 20 RGWRados::pool_iterate: got zone_names.default 2023-10-05T19:30:21.454 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:21.450+0000 7f727fff8e40 20 RGWRados::pool_iterate: got zonegroups_names.default 2023-10-05T19:30:21.455 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:21.450+0000 7f727fff8e40 20 rados->read ofs=0 len=0 2023-10-05T19:30:21.455 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:21.450+0000 7f727fff8e40 1 -- 172.21.15.167:0/164415728 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:76 5.f 5:f43b6ece:::zone_names.default:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32) v8 -- 0x556f6debede0 con 0x556f6de882f0 2023-10-05T19:30:21.456 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:21.450+0000 7f727e32e700 1 -- 172.21.15.167:0/164415728 <== osd.1 v2:172.21.15.167:6800/30035 26 ==== osd_op_reply(76 zone_names.default [read 0~46 out=46b] v0'0 uv1 ondisk = 0) v8 ==== 162+0+46 (crc 0 0 0) 0x7f727802f080 con 0x556f6de882f0 2023-10-05T19:30:21.456 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:21.450+0000 7f727fff8e40 20 rados_obj.operate() r=0 bl.length=46 2023-10-05T19:30:21.456 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:21.450+0000 7f727fff8e40 20 rados->read ofs=0 len=0 2023-10-05T19:30:21.456 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:21.450+0000 7f727fff8e40 1 -- 172.21.15.167:0/164415728 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:77 5.1d 5:b8374eb4:::zone_info.9a9df965-6612-45d4-a45f-6e3c889bacc2:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32) v8 -- 0x556f6debf190 con 0x556f6de882f0 2023-10-05T19:30:21.456 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:21.450+0000 7f727e32e700 1 -- 172.21.15.167:0/164415728 <== osd.1 v2:172.21.15.167:6800/30035 27 ==== osd_op_reply(77 zone_info.9a9df965-6612-45d4-a45f-6e3c889bacc2 [read 0~874 out=874b] v0'0 uv2 ondisk = 0) v8 ==== 190+0+874 (crc 0 0 0) 0x7f727802f080 con 0x556f6de882f0 2023-10-05T19:30:21.456 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:21.454+0000 7f727fff8e40 20 rados_obj.operate() r=0 bl.length=874 2023-10-05T19:30:21.456 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:21.454+0000 7f727fff8e40 20 rados->read ofs=0 len=0 2023-10-05T19:30:21.456 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:21.454+0000 7f727fff8e40 1 -- 172.21.15.167:0/164415728 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:78 5.f 5:f4c53578:::zonegroups_names.default:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32) v8 -- 0x556f6debf190 con 0x556f6de882f0 2023-10-05T19:30:21.457 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:21.454+0000 7f727e32e700 1 -- 172.21.15.167:0/164415728 <== osd.1 v2:172.21.15.167:6800/30035 28 ==== osd_op_reply(78 zonegroups_names.default [read 0~46 out=46b] v0'0 uv2 ondisk = 0) v8 ==== 168+0+46 (crc 0 0 0) 0x7f727802f080 con 0x556f6de882f0 2023-10-05T19:30:21.457 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:21.454+0000 7f727fff8e40 20 rados_obj.operate() r=0 bl.length=46 2023-10-05T19:30:21.457 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:21.454+0000 7f727fff8e40 20 rados->read ofs=0 len=0 2023-10-05T19:30:21.457 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:21.454+0000 7f727fff8e40 1 -- 172.21.15.167:0/164415728 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:79 5.c 5:30d299a5:::zonegroup_info.4f1f719d-9648-417b-b10c-e5d939825419:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32) v8 -- 0x556f6debf190 con 0x7f7268010070 2023-10-05T19:30:21.458 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:21.454+0000 7f727e32e700 1 -- 172.21.15.167:0/164415728 <== osd.0 v2:172.21.15.118:6800/27690 20 ==== osd_op_reply(79 zonegroup_info.4f1f719d-9648-417b-b10c-e5d939825419 [read 0~362 out=362b] v0'0 uv1 ondisk = 0) v8 ==== 195+0+362 (crc 0 0 0) 0x7f727802f080 con 0x7f7268010070 2023-10-05T19:30:21.458 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:21.454+0000 7f727fff8e40 20 rados_obj.operate() r=0 bl.length=362 2023-10-05T19:30:21.458 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:21.454+0000 7f727fff8e40 20 rados->read ofs=0 len=0 2023-10-05T19:30:21.458 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:21.454+0000 7f727fff8e40 1 -- 172.21.15.167:0/164415728 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:80 5.3 5:c0ad8cbd:::region_map:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32) v8 -- 0x556f6debf5e0 con 0x7f7268010070 2023-10-05T19:30:21.459 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:21.454+0000 7f727e32e700 1 -- 172.21.15.167:0/164415728 <== osd.0 v2:172.21.15.118:6800/27690 21 ==== osd_op_reply(80 region_map [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 ==== 154+0+0 (crc 0 0 0) 0x7f727802f080 con 0x7f7268010070 2023-10-05T19:30:21.459 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:21.454+0000 7f727fff8e40 20 rados_obj.operate() r=-2 bl.length=0 2023-10-05T19:30:21.459 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:21.454+0000 7f727fff8e40 20 zone default found 2023-10-05T19:30:21.459 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:21.454+0000 7f727fff8e40 4 Realm: () 2023-10-05T19:30:21.460 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:21.454+0000 7f727fff8e40 4 ZoneGroup: default (4f1f719d-9648-417b-b10c-e5d939825419) 2023-10-05T19:30:21.460 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:21.454+0000 7f727fff8e40 4 Zone: default (9a9df965-6612-45d4-a45f-6e3c889bacc2) 2023-10-05T19:30:21.460 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:21.454+0000 7f727fff8e40 10 cannot find current period zonegroup using local zonegroup configuration 2023-10-05T19:30:21.460 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:21.454+0000 7f727fff8e40 20 zonegroup default 2023-10-05T19:30:21.460 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:21.454+0000 7f727fff8e40 20 rados->read ofs=0 len=0 2023-10-05T19:30:21.460 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:21.454+0000 7f727fff8e40 1 -- 172.21.15.167:0/164415728 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:81 5.3 5:c52100b6:::period_config.default:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32) v8 -- 0x556f6debf5e0 con 0x7f7268010070 2023-10-05T19:30:21.460 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:21.454+0000 7f727e32e700 1 -- 172.21.15.167:0/164415728 <== osd.0 v2:172.21.15.118:6800/27690 22 ==== osd_op_reply(81 period_config.default [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 ==== 165+0+0 (crc 0 0 0) 0x7f7278050040 con 0x7f7268010070 2023-10-05T19:30:21.460 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:21.454+0000 7f727fff8e40 20 rados_obj.operate() r=-2 bl.length=0 2023-10-05T19:30:21.460 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:21.454+0000 7f727fff8e40 20 rados->read ofs=0 len=0 2023-10-05T19:30:21.460 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:21.454+0000 7f727fff8e40 1 -- 172.21.15.167:0/164415728 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:82 6.2 6:4347d321:::bucket.sync-source-hints.:head [call version.read in=11b,read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32) v8 -- 0x556f6dec0d30 con 0x556f6debb8a0 2023-10-05T19:30:21.461 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:21.458+0000 7f727db2d700 1 -- 172.21.15.167:0/164415728 <== osd.3 v2:172.21.15.167:6808/30036 20 ==== osd_op_reply(82 bucket.sync-source-hints. [call,read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 ==== 211+0+0 (crc 0 0 0) 0x7f7270016070 con 0x556f6debb8a0 2023-10-05T19:30:21.462 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:21.458+0000 7f727fff8e40 20 rados_obj.operate() r=-2 bl.length=0 2023-10-05T19:30:21.462 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:21.458+0000 7f727fff8e40 20 rados->read ofs=0 len=0 2023-10-05T19:30:21.462 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:21.458+0000 7f727fff8e40 1 -- 172.21.15.167:0/164415728 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:83 6.b 6:d467b91b:::bucket.sync-target-hints.:head [call version.read in=11b,read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32) v8 -- 0x556f6dec10e0 con 0x556f6debb8a0 2023-10-05T19:30:21.462 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:21.458+0000 7f727db2d700 1 -- 172.21.15.167:0/164415728 <== osd.3 v2:172.21.15.167:6808/30036 21 ==== osd_op_reply(83 bucket.sync-target-hints. [call,read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 ==== 211+0+0 (crc 0 0 0) 0x7f7270016070 con 0x556f6debb8a0 2023-10-05T19:30:21.462 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:21.458+0000 7f727fff8e40 20 rados_obj.operate() r=-2 bl.length=0 2023-10-05T19:30:21.462 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:21.458+0000 7f727fff8e40 20 started sync module instance, tier type = 2023-10-05T19:30:21.462 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:21.458+0000 7f727fff8e40 20 started zone id=9a9df965-6612-45d4-a45f-6e3c889bacc2 (name=default) with tier type = 2023-10-05T19:30:21.462 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:21.458+0000 7f727fff8e40 1 -- 172.21.15.167:0/164415728 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:84 7.1f 7:f95f44c2:::notify.0:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e32) v8 -- 0x556f6dec03f0 con 0x7f727001a070 2023-10-05T19:30:21.464 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:21.462+0000 7f727d32c700 1 -- 172.21.15.167:0/164415728 <== osd.2 v2:172.21.15.167:6816/30037 13 ==== osd_op_reply(84 notify.0 [create] v32'258 uv258 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f7268017050 con 0x7f727001a070 2023-10-05T19:30:21.464 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:21.462+0000 7f727fff8e40 1 -- 172.21.15.167:0/164415728 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:85 7.1f 7:f95f44c2:::notify.0:head [watch watch cookie 93937073915808] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e32) v8 -- 0x556f6debab70 con 0x7f727001a070 2023-10-05T19:30:21.465 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:21.462+0000 7f727fff8e40 1 -- 172.21.15.167:0/164415728 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:86 7.0 7:05bf5b68:::notify.1:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e32) v8 -- 0x556f6dec1ea0 con 0x556f6de882f0 2023-10-05T19:30:21.468 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:21.466+0000 7f727e32e700 1 -- 172.21.15.167:0/164415728 <== osd.1 v2:172.21.15.167:6800/30035 29 ==== osd_op_reply(86 notify.1 [create] v32'258 uv258 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f7278050040 con 0x556f6de882f0 2023-10-05T19:30:21.468 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:21.466+0000 7f727d32c700 1 -- 172.21.15.167:0/164415728 <== osd.2 v2:172.21.15.167:6816/30037 14 ==== osd_op_reply(85 notify.0 [watch watch cookie 93937073915808] v32'259 uv258 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f7268017050 con 0x7f727001a070 2023-10-05T19:30:21.468 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:21.466+0000 7f727fff8e40 1 -- 172.21.15.167:0/164415728 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:87 7.0 7:05bf5b68:::notify.1:head [watch watch cookie 93937073920432] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e32) v8 -- 0x556f6dec1e00 con 0x556f6de882f0 2023-10-05T19:30:21.468 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:21.466+0000 7f727fff8e40 1 -- 172.21.15.167:0/164415728 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:88 7.15 7:a93a5511:::notify.2:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e32) v8 -- 0x556f6dec2c80 con 0x556f6debb8a0 2023-10-05T19:30:21.472 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:21.466+0000 7f727e32e700 1 -- 172.21.15.167:0/164415728 <== osd.1 v2:172.21.15.167:6800/30035 30 ==== osd_op_reply(87 notify.1 [watch watch cookie 93937073920432] v32'259 uv258 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f7278050040 con 0x556f6de882f0 2023-10-05T19:30:21.472 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:21.470+0000 7f727db2d700 1 -- 172.21.15.167:0/164415728 <== osd.3 v2:172.21.15.167:6808/30036 22 ==== osd_op_reply(88 notify.2 [create] v32'258 uv258 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f7270016070 con 0x556f6debb8a0 2023-10-05T19:30:21.472 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:21.470+0000 7f727fff8e40 1 -- 172.21.15.167:0/164415728 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:89 7.15 7:a93a5511:::notify.2:head [watch watch cookie 93937073922720] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e32) v8 -- 0x556f6dec1e00 con 0x556f6debb8a0 2023-10-05T19:30:21.472 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:21.470+0000 7f727fff8e40 1 -- 172.21.15.167:0/164415728 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:90 7.e 7:7759931f:::notify.3:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e32) v8 -- 0x556f6dec36f0 con 0x7f727001a070 2023-10-05T19:30:21.474 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:21.470+0000 7f727d32c700 1 -- 172.21.15.167:0/164415728 <== osd.2 v2:172.21.15.167:6816/30037 15 ==== osd_op_reply(90 notify.3 [create] v32'258 uv258 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f7268017050 con 0x7f727001a070 2023-10-05T19:30:21.474 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:21.470+0000 7f727fff8e40 1 -- 172.21.15.167:0/164415728 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:91 7.e 7:7759931f:::notify.3:head [watch watch cookie 93937073928944] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e32) v8 -- 0x556f6dec4430 con 0x7f727001a070 2023-10-05T19:30:21.474 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:21.470+0000 7f727fff8e40 1 -- 172.21.15.167:0/164415728 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:92 7.d 7:b4812045:::notify.4:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e32) v8 -- 0x556f6dec4e40 con 0x556f6de882f0 2023-10-05T19:30:21.475 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:21.470+0000 7f727db2d700 1 -- 172.21.15.167:0/164415728 <== osd.3 v2:172.21.15.167:6808/30036 23 ==== osd_op_reply(89 notify.2 [watch watch cookie 93937073922720] v32'259 uv258 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f7270016070 con 0x556f6debb8a0 2023-10-05T19:30:21.476 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:21.474+0000 7f727d32c700 1 -- 172.21.15.167:0/164415728 <== osd.2 v2:172.21.15.167:6816/30037 16 ==== osd_op_reply(91 notify.3 [watch watch cookie 93937073928944] v32'259 uv258 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f7268017050 con 0x7f727001a070 2023-10-05T19:30:21.477 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:21.474+0000 7f727e32e700 1 -- 172.21.15.167:0/164415728 <== osd.1 v2:172.21.15.167:6800/30035 31 ==== osd_op_reply(92 notify.4 [create] v32'258 uv258 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f7278050040 con 0x556f6de882f0 2023-10-05T19:30:21.477 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:21.474+0000 7f727fff8e40 1 -- 172.21.15.167:0/164415728 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:93 7.d 7:b4812045:::notify.4:head [watch watch cookie 93937073923824] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e32) v8 -- 0x556f6dec4e40 con 0x556f6de882f0 2023-10-05T19:30:21.477 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:21.474+0000 7f727fff8e40 1 -- 172.21.15.167:0/164415728 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:94 7.3 7:c609908c:::notify.5:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e32) v8 -- 0x556f6dec56e0 con 0x7f7268010070 2023-10-05T19:30:21.479 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:21.474+0000 7f727e32e700 1 -- 172.21.15.167:0/164415728 <== osd.1 v2:172.21.15.167:6800/30035 32 ==== osd_op_reply(93 notify.4 [watch watch cookie 93937073923824] v32'259 uv258 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f7278050040 con 0x556f6de882f0 2023-10-05T19:30:21.480 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:21.478+0000 7f727e32e700 1 -- 172.21.15.167:0/164415728 <== osd.0 v2:172.21.15.118:6800/27690 23 ==== osd_op_reply(94 notify.5 [create] v32'258 uv258 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f7278050040 con 0x7f7268010070 2023-10-05T19:30:21.480 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:21.478+0000 7f727fff8e40 1 -- 172.21.15.167:0/164415728 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:95 7.3 7:c609908c:::notify.5:head [watch watch cookie 93937073932896] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e32) v8 -- 0x556f6dec53a0 con 0x7f7268010070 2023-10-05T19:30:21.480 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:21.478+0000 7f727fff8e40 1 -- 172.21.15.167:0/164415728 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:96 7.14 7:2b04a3e9:::notify.6:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e32) v8 -- 0x556f6dec5f30 con 0x7f7268010070 2023-10-05T19:30:21.483 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:21.478+0000 7f727e32e700 1 -- 172.21.15.167:0/164415728 <== osd.0 v2:172.21.15.118:6800/27690 24 ==== osd_op_reply(95 notify.5 [watch watch cookie 93937073932896] v32'259 uv258 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f7278050040 con 0x7f7268010070 2023-10-05T19:30:21.484 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:21.478+0000 7f727e32e700 1 -- 172.21.15.167:0/164415728 <== osd.0 v2:172.21.15.118:6800/27690 25 ==== osd_op_reply(96 notify.6 [create] v32'258 uv258 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f7278050040 con 0x7f7268010070 2023-10-05T19:30:21.484 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:21.478+0000 7f727fff8e40 1 -- 172.21.15.167:0/164415728 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:97 7.14 7:2b04a3e9:::notify.6:head [watch watch cookie 93937073936896] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e32) v8 -- 0x556f6dec53a0 con 0x7f7268010070 2023-10-05T19:30:21.484 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:21.478+0000 7f727fff8e40 1 -- 172.21.15.167:0/164415728 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:98 7.9 7:93e5b521:::notify.7:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e32) v8 -- 0x556f6dec68d0 con 0x7f7268010070 2023-10-05T19:30:21.485 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:21.482+0000 7f727e32e700 1 -- 172.21.15.167:0/164415728 <== osd.0 v2:172.21.15.118:6800/27690 26 ==== osd_op_reply(97 notify.6 [watch watch cookie 93937073936896] v32'259 uv258 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f7278050040 con 0x7f7268010070 2023-10-05T19:30:21.485 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:21.482+0000 7f727e32e700 1 -- 172.21.15.167:0/164415728 <== osd.0 v2:172.21.15.118:6800/27690 27 ==== osd_op_reply(98 notify.7 [create] v32'258 uv258 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f7278050040 con 0x7f7268010070 2023-10-05T19:30:21.486 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:21.482+0000 7f727fff8e40 1 -- 172.21.15.167:0/164415728 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:99 7.9 7:93e5b521:::notify.7:head [watch watch cookie 93937073934000] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e32) v8 -- 0x556f6dec5300 con 0x7f7268010070 2023-10-05T19:30:21.486 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:21.482+0000 7f727fff8e40 20 add_watcher() i=0 2023-10-05T19:30:21.486 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:21.482+0000 7f727fff8e40 20 add_watcher() i=1 2023-10-05T19:30:21.486 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:21.482+0000 7f727fff8e40 20 add_watcher() i=2 2023-10-05T19:30:21.486 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:21.482+0000 7f727fff8e40 20 add_watcher() i=3 2023-10-05T19:30:21.486 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:21.482+0000 7f727fff8e40 20 add_watcher() i=4 2023-10-05T19:30:21.486 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:21.482+0000 7f727fff8e40 20 add_watcher() i=5 2023-10-05T19:30:21.486 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:21.482+0000 7f727fff8e40 20 add_watcher() i=6 2023-10-05T19:30:21.487 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:21.482+0000 7f727e32e700 1 -- 172.21.15.167:0/164415728 <== osd.0 v2:172.21.15.118:6800/27690 28 ==== osd_op_reply(99 notify.7 [watch watch cookie 93937073934000] v32'259 uv258 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f7278050040 con 0x7f7268010070 2023-10-05T19:30:21.488 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:21.482+0000 7f727fff8e40 20 add_watcher() i=7 2023-10-05T19:30:21.488 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:21.482+0000 7f727fff8e40 2 all 8 watchers are set, enabling cache 2023-10-05T19:30:21.488 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:21.482+0000 7f727fff8e40 1 -- 172.21.15.167:0/164415728 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:100 6.4 6:22d26bf9:::data_loggenerations_metadata:head [call version.check_conds in=74b,call version.read in=11b,read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32) v8 -- 0x556f6dec9c20 con 0x556f6de882f0 2023-10-05T19:30:21.488 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:21.486+0000 7f727e32e700 1 -- 172.21.15.167:0/164415728 <== osd.1 v2:172.21.15.167:6800/30035 33 ==== osd_op_reply(100 data_loggenerations_metadata [call,call out=48b,read 0~28 out=28b] v0'0 uv1 ondisk = 0) v8 ==== 256+0+76 (crc 0 0 0) 0x7f7278050040 con 0x556f6de882f0 2023-10-05T19:30:21.488 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:21.486+0000 7f727fff8e40 1 -- 172.21.15.167:0/164415728 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:101 6.4 6:22d26bf9:::data_loggenerations_metadata:head [watch watch cookie 93937073939712] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e32) v8 -- 0x556f6dec70e0 con 0x556f6de882f0 2023-10-05T19:30:21.491 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:21.486+0000 7f727e32e700 1 -- 172.21.15.167:0/164415728 <== osd.1 v2:172.21.15.167:6800/30035 34 ==== osd_op_reply(101 data_loggenerations_metadata [watch watch cookie 93937073939712] v32'251 uv1 ondisk = 0) v8 ==== 172+0+0 (crc 0 0 0) 0x7f7278050040 con 0x556f6de882f0 2023-10-05T19:30:21.491 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:21.486+0000 7f724e7ec700 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2023-10-05T19:30:21.491 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:21.486+0000 7f727fff8e40 20 check_secure_mon_conn(): auth registy supported: methods=[2] modes=[2,1] 2023-10-05T19:30:21.491 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:21.486+0000 7f727fff8e40 20 check_secure_mon_conn(): mode 1 is insecure 2023-10-05T19:30:21.492 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:21.490+0000 7f727fff8e40 1 Processor -- start 2023-10-05T19:30:21.492 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:21.490+0000 7f727fff8e40 1 -- start start 2023-10-05T19:30:21.492 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:21.490+0000 7f727fff8e40 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x556f6ded42b0 0x556f6ded4680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-05T19:30:21.492 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:21.490+0000 7f727fff8e40 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x556f6ded4bc0 con 0x556f6ded42b0 2023-10-05T19:30:21.493 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:21.490+0000 7f727d32c700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x556f6ded42b0 0x556f6ded4680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-05T19:30:21.493 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:21.490+0000 7f727d32c700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x556f6ded42b0 0x556f6ded4680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.118:3300/0 says I am v2:172.21.15.167:32912/0 (socket says 172.21.15.167:32912) 2023-10-05T19:30:21.493 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:21.490+0000 7f727d32c700 1 -- 172.21.15.167:0/706356877 learned_addr learned my addr 172.21.15.167:0/706356877 (peer_addr_for_me v2:172.21.15.167:0/0) 2023-10-05T19:30:21.493 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:21.490+0000 7f727d32c700 1 -- 172.21.15.167:0/706356877 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x556f6ded4d00 con 0x556f6ded42b0 2023-10-05T19:30:21.493 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:21.490+0000 7f727d32c700 1 --2- 172.21.15.167:0/706356877 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x556f6ded42b0 0x556f6ded4680 secure :-1 s=READY pgs=654 cs=0 l=1 rev1=1 crypto rx=0x7f7268003000 tx=0x7f7268010000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bd04ed2928cb9993 server_cookie=0 in_seq=0 out_seq=0 2023-10-05T19:30:21.494 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:21.490+0000 7f724dfeb700 1 -- 172.21.15.167:0/706356877 <== mon.0 v2:172.21.15.118:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f726801c9f0 con 0x556f6ded42b0 2023-10-05T19:30:21.494 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:21.490+0000 7f724dfeb700 1 -- 172.21.15.167:0/706356877 <== mon.0 v2:172.21.15.118:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f726801cb70 con 0x556f6ded42b0 2023-10-05T19:30:21.494 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:21.490+0000 7f724dfeb700 1 -- 172.21.15.167:0/706356877 <== mon.0 v2:172.21.15.118:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f726801cd60 con 0x556f6ded42b0 2023-10-05T19:30:21.495 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:21.490+0000 7f727fff8e40 1 -- 172.21.15.167:0/706356877 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x556f6ded42b0 msgr2=0x556f6ded4680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-05T19:30:21.495 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:21.490+0000 7f727fff8e40 1 --2- 172.21.15.167:0/706356877 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x556f6ded42b0 0x556f6ded4680 secure :-1 s=READY pgs=654 cs=0 l=1 rev1=1 crypto rx=0x7f7268003000 tx=0x7f7268010000 comp rx=0 tx=0).stop 2023-10-05T19:30:21.495 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:21.490+0000 7f727fff8e40 1 -- 172.21.15.167:0/706356877 shutdown_connections 2023-10-05T19:30:21.495 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:21.490+0000 7f727fff8e40 1 --2- 172.21.15.167:0/706356877 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x556f6ded42b0 0x556f6ded4680 unknown :-1 s=CLOSED pgs=654 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-05T19:30:21.496 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:21.490+0000 7f727fff8e40 1 -- 172.21.15.167:0/706356877 >> 172.21.15.167:0/706356877 conn(0x556f6decf670 msgr2=0x556f6decd7c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-05T19:30:21.496 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:21.490+0000 7f727fff8e40 1 -- 172.21.15.167:0/706356877 shutdown_connections 2023-10-05T19:30:21.496 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:21.490+0000 7f727fff8e40 1 -- 172.21.15.167:0/706356877 wait complete. 2023-10-05T19:30:21.497 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:21.494+0000 7f727fff8e40 1 Processor -- start 2023-10-05T19:30:21.497 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:21.494+0000 7f727fff8e40 1 -- start start 2023-10-05T19:30:21.497 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:21.494+0000 7f727fff8e40 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x556f6ded42b0 0x556f6ded9da0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-05T19:30:21.497 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:21.494+0000 7f727fff8e40 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x556f6dedada0 con 0x556f6ded42b0 2023-10-05T19:30:21.497 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:21.494+0000 7f727d32c700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x556f6ded42b0 0x556f6ded9da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-05T19:30:21.497 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:21.494+0000 7f727d32c700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x556f6ded42b0 0x556f6ded9da0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.118:3300/0 says I am v2:172.21.15.167:32926/0 (socket says 172.21.15.167:32926) 2023-10-05T19:30:21.497 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:21.494+0000 7f727d32c700 1 -- 172.21.15.167:0/2258697426 learned_addr learned my addr 172.21.15.167:0/2258697426 (peer_addr_for_me v2:172.21.15.167:0/0) 2023-10-05T19:30:21.498 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:21.494+0000 7f727d32c700 1 -- 172.21.15.167:0/2258697426 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x556f6de8bc20 con 0x556f6ded42b0 2023-10-05T19:30:21.498 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:21.494+0000 7f727d32c700 1 --2- 172.21.15.167:0/2258697426 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x556f6ded42b0 0x556f6ded9da0 secure :-1 s=READY pgs=655 cs=0 l=1 rev1=1 crypto rx=0x7f726800efd0 tx=0x7f726802f000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-05T19:30:21.498 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:21.494+0000 7f71bffff700 1 -- 172.21.15.167:0/2258697426 <== mon.0 v2:172.21.15.118:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f726802ab20 con 0x556f6ded42b0 2023-10-05T19:30:21.499 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:21.494+0000 7f71bffff700 1 -- 172.21.15.167:0/2258697426 <== mon.0 v2:172.21.15.118:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f726802ac80 con 0x556f6ded42b0 2023-10-05T19:30:21.499 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:21.494+0000 7f727fff8e40 1 -- 172.21.15.167:0/2258697426 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x556f6deda3a0 con 0x556f6ded42b0 2023-10-05T19:30:21.499 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:21.494+0000 7f71bffff700 1 -- 172.21.15.167:0/2258697426 <== mon.0 v2:172.21.15.118:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f726802ae30 con 0x556f6ded42b0 2023-10-05T19:30:21.499 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:21.494+0000 7f727fff8e40 1 -- 172.21.15.167:0/2258697426 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x556f6deda840 con 0x556f6ded42b0 2023-10-05T19:30:21.499 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:21.494+0000 7f71bffff700 1 -- 172.21.15.167:0/2258697426 <== mon.0 v2:172.21.15.118:3300/0 4 ==== mgrmap(e 5) v1 ==== 40329+0+0 (secure 0 0 0) 0x7f7268050440 con 0x556f6ded42b0 2023-10-05T19:30:21.500 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:21.498+0000 7f71bffff700 1 --2- 172.21.15.167:0/2258697426 >> [v2:172.21.15.167:6824/29889,v1:172.21.15.167:6825/29889] conn(0x7f71c4030c00 0x7f71c40330b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-05T19:30:21.500 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:21.498+0000 7f71bffff700 1 -- 172.21.15.167:0/2258697426 <== mon.0 v2:172.21.15.118:3300/0 5 ==== osd_map(32..32 src has 1..32) v4 ==== 6260+0+0 (secure 0 0 0) 0x7f726802f070 con 0x556f6ded42b0 2023-10-05T19:30:21.500 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:21.498+0000 7f727e32e700 1 --2- 172.21.15.167:0/2258697426 >> [v2:172.21.15.167:6824/29889,v1:172.21.15.167:6825/29889] conn(0x7f71c4030c00 0x7f71c40330b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-05T19:30:21.500 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:21.498+0000 7f727fff8e40 5 note: GC not initialized 2023-10-05T19:30:21.501 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:21.498+0000 7f727e32e700 1 --2- 172.21.15.167:0/2258697426 >> [v2:172.21.15.167:6824/29889,v1:172.21.15.167:6825/29889] conn(0x7f71c4030c00 0x7f71c40330b0 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x556f6dec55b0 tx=0x7f7278000f90 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-05T19:30:21.502 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:21.498+0000 7f71bd7fa700 20 reqs_thread_entry: start 2023-10-05T19:30:21.503 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:21.498+0000 7f71b6ffd700 20 reqs_thread_entry: start 2023-10-05T19:30:21.503 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:21.498+0000 7f727fff8e40 20 init_complete bucket index max shards: 11 2023-10-05T19:30:21.504 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:21.498+0000 7f727fff8e40 10 rgw notify: Started notification manager with: 1 workers 2023-10-05T19:30:21.504 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:21.498+0000 7f71b57fa700 20 rgw notify: INFO: next queues processing will happen at: Thu Oct 5 19:30:51 2023 2023-10-05T19:30:21.504 INFO:teuthology.orchestra.run.smithi167.stderr: 2023-10-05T19:30:21.504 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:21.502+0000 7f71b4ff9700 20 reqs_thread_entry: start 2023-10-05T19:30:21.504 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:21.502+0000 7f727fff8e40 20 get_system_obj_state: rctx=0x7ffc0e89f298 obj=default.rgw.meta:users.uid:foo state=0x556f6dd5d0b0 s->prefetch_data=0 2023-10-05T19:30:21.504 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:21.502+0000 7f727fff8e40 10 cache get: name=default.rgw.meta+users.uid+foo : miss 2023-10-05T19:30:21.504 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:21.502+0000 7f727fff8e40 1 -- 172.21.15.167:0/164415728 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:102 8.18 8:1e6ca5ed:users.uid::foo:head [call version.read in=11b,getxattrs,stat] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32) v8 -- 0x556f6df424f0 con 0x556f6de882f0 2023-10-05T19:30:21.505 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:21.502+0000 7f727e32e700 1 -- 172.21.15.167:0/164415728 <== osd.1 v2:172.21.15.167:6800/30035 35 ==== osd_op_reply(102 foo [call out=48b,getxattrs out=75b,stat out=16b] v0'0 uv15 ondisk = 0) v8 ==== 231+0+139 (crc 0 0 0) 0x7f7278050040 con 0x556f6de882f0 2023-10-05T19:30:21.506 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:21.502+0000 7f727fff8e40 10 cache put: name=default.rgw.meta+users.uid+foo info.flags=0x16 2023-10-05T19:30:21.506 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:21.502+0000 7f727fff8e40 10 adding default.rgw.meta+users.uid+foo to cache LRU end 2023-10-05T19:30:21.506 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:21.502+0000 7f727fff8e40 10 updating xattr: name=ceph.objclass.version bl.length()=42 2023-10-05T19:30:21.506 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:21.502+0000 7f727fff8e40 20 get_system_obj_state: s->obj_tag was set empty 2023-10-05T19:30:21.506 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:21.502+0000 7f727fff8e40 20 Read xattr: user.rgw.idtag 2023-10-05T19:30:21.506 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:21.502+0000 7f727fff8e40 10 cache get: name=default.rgw.meta+users.uid+foo : type miss (requested=0x13, cached=0x16) 2023-10-05T19:30:21.506 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:21.502+0000 7f727fff8e40 20 rados->read ofs=0 len=0 2023-10-05T19:30:21.506 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:21.502+0000 7f727fff8e40 1 -- 172.21.15.167:0/164415728 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:103 8.18 8:1e6ca5ed:users.uid::foo:head [call version.check_conds in=122b,call version.read in=11b,read 0~0,getxattrs] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32) v8 -- 0x556f6df44f80 con 0x556f6de882f0 2023-10-05T19:30:21.506 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:21.502+0000 7f727e32e700 1 -- 172.21.15.167:0/164415728 <== osd.1 v2:172.21.15.167:6800/30035 36 ==== osd_op_reply(103 foo [call,call out=48b,read 0~357 out=357b,getxattrs out=75b] v0'0 uv15 ondisk = 0) v8 ==== 273+0+480 (crc 0 0 0) 0x7f7278050040 con 0x556f6de882f0 2023-10-05T19:30:21.506 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:21.502+0000 7f727fff8e40 20 rados_obj.operate() r=0 bl.length=357 2023-10-05T19:30:21.507 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:21.502+0000 7f727fff8e40 10 cache put: name=default.rgw.meta+users.uid+foo info.flags=0x13 2023-10-05T19:30:21.507 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:21.502+0000 7f727fff8e40 10 moving default.rgw.meta+users.uid+foo to cache LRU end 2023-10-05T19:30:21.507 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:21.502+0000 7f727fff8e40 10 updating xattr: name=ceph.objclass.version bl.length()=42 2023-10-05T19:30:21.507 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:21.502+0000 7f727fff8e40 1 -- 172.21.15.167:0/164415728 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:104 6.13 6:cea599e4:usage::usage.10:head [call rgw.user_usage_log_trim in=55b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e32) v8 -- 0x556f6df44f80 con 0x7f7268010070 2023-10-05T19:30:21.509 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:21.506+0000 7f727e32e700 1 -- 172.21.15.167:0/164415728 <== osd.0 v2:172.21.15.118:6800/27690 29 ==== osd_op_reply(104 usage.10 [call] v32'33 uv33 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f7278050040 con 0x7f7268010070 2023-10-05T19:30:21.509 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:21.506+0000 7f727fff8e40 1 -- 172.21.15.167:0/164415728 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:105 6.13 6:cea599e4:usage::usage.10:head [call rgw.user_usage_log_trim in=55b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e32) v8 -- 0x556f6df44f80 con 0x7f7268010070 2023-10-05T19:30:21.512 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:21.510+0000 7f727e32e700 1 -- 172.21.15.167:0/164415728 <== osd.0 v2:172.21.15.118:6800/27690 30 ==== osd_op_reply(105 usage.10 [call] v32'34 uv33 ondisk = -61 ((61) No data available)) v8 ==== 152+0+0 (crc 0 0 0) 0x7f7278050040 con 0x7f7268010070 2023-10-05T19:30:21.512 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:21.510+0000 7f727fff8e40 1 -- 172.21.15.167:0/164415728 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:106 7.1f 7:f95f44c2:::notify.0:head [watch unwatch cookie 93937073915808] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e32) v8 -- 0x556f6df008e0 con 0x7f727001a070 2023-10-05T19:30:21.515 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:21.510+0000 7f727d32c700 1 -- 172.21.15.167:0/164415728 <== osd.2 v2:172.21.15.167:6816/30037 17 ==== osd_op_reply(106 notify.0 [watch unwatch cookie 93937073915808] v32'260 uv258 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f7268032040 con 0x7f727001a070 2023-10-05T19:30:21.516 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:21.510+0000 7f727fff8e40 20 remove_watcher() i=0 2023-10-05T19:30:21.516 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:21.510+0000 7f727fff8e40 2 removed watcher, disabling cache 2023-10-05T19:30:21.516 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:21.510+0000 7f727fff8e40 1 -- 172.21.15.167:0/164415728 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:107 7.0 7:05bf5b68:::notify.1:head [watch unwatch cookie 93937073920432] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e32) v8 -- 0x556f6dec07a0 con 0x556f6de882f0 2023-10-05T19:30:21.518 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:21.514+0000 7f727e32e700 1 -- 172.21.15.167:0/164415728 <== osd.1 v2:172.21.15.167:6800/30035 37 ==== osd_op_reply(107 notify.1 [watch unwatch cookie 93937073920432] v32'260 uv258 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f7278050040 con 0x556f6de882f0 2023-10-05T19:30:21.518 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:21.514+0000 7f727fff8e40 20 remove_watcher() i=1 2023-10-05T19:30:21.518 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:21.514+0000 7f727fff8e40 1 -- 172.21.15.167:0/164415728 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:108 7.15 7:a93a5511:::notify.2:head [watch unwatch cookie 93937073922720] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e32) v8 -- 0x556f6dec07a0 con 0x556f6debb8a0 2023-10-05T19:30:21.521 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:21.518+0000 7f727db2d700 1 -- 172.21.15.167:0/164415728 <== osd.3 v2:172.21.15.167:6808/30036 24 ==== osd_op_reply(108 notify.2 [watch unwatch cookie 93937073922720] v32'260 uv258 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f7270016070 con 0x556f6debb8a0 2023-10-05T19:30:21.521 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:21.518+0000 7f727fff8e40 20 remove_watcher() i=2 2023-10-05T19:30:21.521 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:21.518+0000 7f727fff8e40 1 -- 172.21.15.167:0/164415728 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:109 7.e 7:7759931f:::notify.3:head [watch unwatch cookie 93937073928944] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e32) v8 -- 0x556f6dec22a0 con 0x7f727001a070 2023-10-05T19:30:21.524 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:21.522+0000 7f727d32c700 1 -- 172.21.15.167:0/164415728 <== osd.2 v2:172.21.15.167:6816/30037 18 ==== osd_op_reply(109 notify.3 [watch unwatch cookie 93937073928944] v32'260 uv258 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f7268032040 con 0x7f727001a070 2023-10-05T19:30:21.524 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:21.522+0000 7f727fff8e40 20 remove_watcher() i=3 2023-10-05T19:30:21.525 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:21.522+0000 7f727fff8e40 1 -- 172.21.15.167:0/164415728 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:110 7.d 7:b4812045:::notify.4:head [watch unwatch cookie 93937073923824] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e32) v8 -- 0x556f6dec3af0 con 0x556f6de882f0 2023-10-05T19:30:21.527 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:21.522+0000 7f727e32e700 1 -- 172.21.15.167:0/164415728 <== osd.1 v2:172.21.15.167:6800/30035 38 ==== osd_op_reply(110 notify.4 [watch unwatch cookie 93937073923824] v32'260 uv258 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f7278050040 con 0x556f6de882f0 2023-10-05T19:30:21.527 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:21.522+0000 7f727fff8e40 20 remove_watcher() i=4 2023-10-05T19:30:21.528 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:21.522+0000 7f727fff8e40 1 -- 172.21.15.167:0/164415728 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:111 7.3 7:c609908c:::notify.5:head [watch unwatch cookie 93937073932896] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e32) v8 -- 0x556f6dec3af0 con 0x7f7268010070 2023-10-05T19:30:21.531 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:21.526+0000 7f727e32e700 1 -- 172.21.15.167:0/164415728 <== osd.0 v2:172.21.15.118:6800/27690 31 ==== osd_op_reply(111 notify.5 [watch unwatch cookie 93937073932896] v32'260 uv258 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f7278050040 con 0x7f7268010070 2023-10-05T19:30:21.531 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:21.526+0000 7f727fff8e40 20 remove_watcher() i=5 2023-10-05T19:30:21.531 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:21.526+0000 7f727fff8e40 1 -- 172.21.15.167:0/164415728 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:112 7.14 7:2b04a3e9:::notify.6:head [watch unwatch cookie 93937073936896] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e32) v8 -- 0x556f6dec4a60 con 0x7f7268010070 2023-10-05T19:30:21.534 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:21.530+0000 7f727e32e700 1 -- 172.21.15.167:0/164415728 <== osd.0 v2:172.21.15.118:6800/27690 32 ==== osd_op_reply(112 notify.6 [watch unwatch cookie 93937073936896] v32'260 uv258 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f7278050040 con 0x7f7268010070 2023-10-05T19:30:21.534 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:21.530+0000 7f727fff8e40 20 remove_watcher() i=6 2023-10-05T19:30:21.534 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:21.530+0000 7f727fff8e40 1 -- 172.21.15.167:0/164415728 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:113 7.9 7:93e5b521:::notify.7:head [watch unwatch cookie 93937073934000] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e32) v8 -- 0x556f6dec4a60 con 0x7f7268010070 2023-10-05T19:30:21.537 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:21.534+0000 7f727e32e700 1 -- 172.21.15.167:0/164415728 <== osd.0 v2:172.21.15.118:6800/27690 33 ==== osd_op_reply(113 notify.7 [watch unwatch cookie 93937073934000] v32'260 uv258 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f7278050040 con 0x7f7268010070 2023-10-05T19:30:21.537 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:21.534+0000 7f727fff8e40 20 remove_watcher() i=7 2023-10-05T19:30:21.539 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:21.534+0000 7f724e7ec700 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2023-10-05T19:30:21.539 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:21.534+0000 7f727fff8e40 1 -- 172.21.15.167:0/164415728 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:114 6.4 6:22d26bf9:::data_loggenerations_metadata:head [watch unwatch cookie 93937073939712] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e32) v8 -- 0x556f6dec4a60 con 0x556f6de882f0 2023-10-05T19:30:21.543 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:21.538+0000 7f727e32e700 1 -- 172.21.15.167:0/164415728 <== osd.1 v2:172.21.15.167:6800/30035 39 ==== osd_op_reply(114 data_loggenerations_metadata [watch unwatch cookie 93937073939712] v32'252 uv1 ondisk = 0) v8 ==== 172+0+0 (crc 0 0 0) 0x7f7278050040 con 0x556f6de882f0 2023-10-05T19:30:21.543 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:21.538+0000 7f727fff8e40 1 -- 172.21.15.167:0/164415728 >> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] conn(0x7f7268010070 msgr2=0x7f726801c430 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-05T19:30:21.543 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:21.538+0000 7f727fff8e40 1 --2- 172.21.15.167:0/164415728 >> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] conn(0x7f7268010070 0x7f726801c430 crc :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-05T19:30:21.543 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:21.538+0000 7f727fff8e40 1 -- 172.21.15.167:0/164415728 >> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] conn(0x556f6de882f0 msgr2=0x556f6deb7b10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-05T19:30:21.544 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:21.538+0000 7f727fff8e40 1 --2- 172.21.15.167:0/164415728 >> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] conn(0x556f6de882f0 0x556f6deb7b10 crc :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-05T19:30:21.544 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:21.538+0000 7f727fff8e40 1 -- 172.21.15.167:0/164415728 >> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] conn(0x7f727001a070 msgr2=0x7f7270042210 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-05T19:30:21.544 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:21.538+0000 7f727fff8e40 1 --2- 172.21.15.167:0/164415728 >> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] conn(0x7f727001a070 0x7f7270042210 crc :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-05T19:30:21.544 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:21.538+0000 7f727fff8e40 1 -- 172.21.15.167:0/164415728 >> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] conn(0x556f6debb8a0 msgr2=0x556f6debdc80 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-05T19:30:21.544 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:21.538+0000 7f727fff8e40 1 --2- 172.21.15.167:0/164415728 >> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] conn(0x556f6debb8a0 0x556f6debdc80 crc :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-05T19:30:21.544 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:21.538+0000 7f727fff8e40 1 -- 172.21.15.167:0/164415728 >> [v2:172.21.15.167:6824/29889,v1:172.21.15.167:6825/29889] conn(0x7f7264030c00 msgr2=0x7f72640330b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-05T19:30:21.544 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:21.538+0000 7f727fff8e40 1 --2- 172.21.15.167:0/164415728 >> [v2:172.21.15.167:6824/29889,v1:172.21.15.167:6825/29889] conn(0x7f7264030c00 0x7f72640330b0 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7f7268000f80 tx=0x7f7268013000 comp rx=0 tx=0).stop 2023-10-05T19:30:21.544 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:21.538+0000 7f727e32e700 1 -- 172.21.15.167:0/164415728 reap_dead start 2023-10-05T19:30:21.544 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:21.538+0000 7f727fff8e40 1 -- 172.21.15.167:0/164415728 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x556f6ddf83d0 msgr2=0x556f6deabf60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-05T19:30:21.544 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:21.538+0000 7f727fff8e40 1 --2- 172.21.15.167:0/164415728 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x556f6ddf83d0 0x556f6deabf60 secure :-1 s=READY pgs=653 cs=0 l=1 rev1=1 crypto rx=0x7f72700018e0 tx=0x7f7270016000 comp rx=0 tx=0).stop 2023-10-05T19:30:21.545 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:21.538+0000 7f727fff8e40 1 -- 172.21.15.167:0/164415728 shutdown_connections 2023-10-05T19:30:21.545 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:21.538+0000 7f727fff8e40 1 --2- 172.21.15.167:0/164415728 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x556f6ddf83d0 0x556f6deabf60 unknown :-1 s=CLOSED pgs=653 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-05T19:30:21.545 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:21.538+0000 7f727fff8e40 1 -- 172.21.15.167:0/164415728 >> 172.21.15.167:0/164415728 conn(0x556f6de7e720 msgr2=0x556f6de9d5d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-05T19:30:21.545 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:21.538+0000 7f727fff8e40 1 -- 172.21.15.167:0/164415728 shutdown_connections 2023-10-05T19:30:21.545 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:21.538+0000 7f727fff8e40 1 -- 172.21.15.167:0/164415728 wait complete. 2023-10-05T19:30:21.546 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:21.542+0000 7f727fff8e40 1 -- 172.21.15.167:0/2258697426 >> [v2:172.21.15.167:6824/29889,v1:172.21.15.167:6825/29889] conn(0x7f71c4030c00 msgr2=0x7f71c40330b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-05T19:30:21.546 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:21.542+0000 7f727fff8e40 1 --2- 172.21.15.167:0/2258697426 >> [v2:172.21.15.167:6824/29889,v1:172.21.15.167:6825/29889] conn(0x7f71c4030c00 0x7f71c40330b0 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x556f6dec55b0 tx=0x7f7278000f90 comp rx=0 tx=0).stop 2023-10-05T19:30:21.547 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:21.542+0000 7f727fff8e40 1 -- 172.21.15.167:0/2258697426 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x556f6ded42b0 msgr2=0x556f6ded9da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-05T19:30:21.547 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:21.542+0000 7f727fff8e40 1 --2- 172.21.15.167:0/2258697426 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x556f6ded42b0 0x556f6ded9da0 secure :-1 s=READY pgs=655 cs=0 l=1 rev1=1 crypto rx=0x7f726800efd0 tx=0x7f726802f000 comp rx=0 tx=0).stop 2023-10-05T19:30:21.547 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:21.542+0000 7f727fff8e40 1 -- 172.21.15.167:0/2258697426 shutdown_connections 2023-10-05T19:30:21.547 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:21.542+0000 7f727fff8e40 1 --2- 172.21.15.167:0/2258697426 >> [v2:172.21.15.167:6824/29889,v1:172.21.15.167:6825/29889] conn(0x7f71c4030c00 0x7f71c40330b0 unknown :-1 s=CLOSED pgs=225 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-05T19:30:21.547 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:21.542+0000 7f727fff8e40 1 --2- 172.21.15.167:0/2258697426 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x556f6ded42b0 0x556f6ded9da0 unknown :-1 s=CLOSED pgs=655 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-05T19:30:21.547 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:21.542+0000 7f727fff8e40 1 -- 172.21.15.167:0/2258697426 >> 172.21.15.167:0/2258697426 conn(0x556f6decf670 msgr2=0x556f6decd710 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-05T19:30:21.547 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:21.542+0000 7f727fff8e40 1 -- 172.21.15.167:0/2258697426 shutdown_connections 2023-10-05T19:30:21.547 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:21.542+0000 7f727fff8e40 1 -- 172.21.15.167:0/2258697426 wait complete. 2023-10-05T19:30:21.550 INFO:tasks.util.rgw:rgwadmin: client.0 : ['usage', 'show', '--uid', 'foo'] 2023-10-05T19:30:21.550 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'ceph', 'usage', 'show', '--uid', 'foo'] 2023-10-05T19:30:21.550 DEBUG:teuthology.orchestra.run.smithi167:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster ceph usage show --uid foo 2023-10-05T19:30:21.591 INFO:teuthology.orchestra.run.smithi167.stderr:ignoring --setuser ceph since I am not root 2023-10-05T19:30:21.592 INFO:teuthology.orchestra.run.smithi167.stderr:ignoring --setgroup ceph since I am not root 2023-10-05T19:30:21.592 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:21.590+0000 7f296e044e40 1 Processor -- start 2023-10-05T19:30:21.592 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:21.590+0000 7f296e044e40 1 -- start start 2023-10-05T19:30:21.593 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:21.590+0000 7f296e044e40 1 --2- >> v2:172.21.15.118:3300/0 conn(0x55afb287d5f0 0x55afb287f9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-05T19:30:21.593 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:21.590+0000 7f296e044e40 1 -- --> v1:172.21.15.118:6789/0 -- auth(proto 0 26 bytes epoch 0) v1 -- 0x55afb27c3bd0 con 0x55afb2879010 2023-10-05T19:30:21.593 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:21.590+0000 7f296e044e40 1 -- --> v2:172.21.15.118:3300/0 -- mon_getmap magic: 0 v1 -- 0x55afb2780ee0 con 0x55afb287d5f0 2023-10-05T19:30:21.593 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:21.590+0000 7f296b378700 1 --2- >> v2:172.21.15.118:3300/0 conn(0x55afb287d5f0 0x55afb287f9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-05T19:30:21.593 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:21.590+0000 7f296b378700 1 --2- >> v2:172.21.15.118:3300/0 conn(0x55afb287d5f0 0x55afb287f9d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.118:3300/0 says I am v2:172.21.15.167:32930/0 (socket says 172.21.15.167:32930) 2023-10-05T19:30:21.594 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:21.590+0000 7f296b378700 1 -- 172.21.15.167:0/3071637523 learned_addr learned my addr 172.21.15.167:0/3071637523 (peer_addr_for_me v2:172.21.15.167:0/0) 2023-10-05T19:30:21.594 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:21.590+0000 7f296b378700 1 -- 172.21.15.167:0/3071637523 >> v1:172.21.15.118:6789/0 conn(0x55afb2879010 legacy=0x55afb28793e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-05T19:30:21.594 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:21.590+0000 7f296b378700 1 -- 172.21.15.167:0/3071637523 --> v2:172.21.15.118:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55afb2879c20 con 0x55afb287d5f0 2023-10-05T19:30:21.594 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:21.590+0000 7f296b378700 1 --2- 172.21.15.167:0/3071637523 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55afb287d5f0 0x55afb287f9d0 secure :-1 s=READY pgs=657 cs=0 l=1 rev1=1 crypto rx=0x7f295400a5c0 tx=0x7f2954005e00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b5c1066050120105 server_cookie=0 in_seq=0 out_seq=0 2023-10-05T19:30:21.595 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:21.590+0000 7f296ab77700 1 -- 172.21.15.167:0/3071637523 <== mon.0 v2:172.21.15.118:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f295400e510 con 0x55afb287d5f0 2023-10-05T19:30:21.595 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:21.590+0000 7f296ab77700 1 -- 172.21.15.167:0/3071637523 <== mon.0 v2:172.21.15.118:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f295400e670 con 0x55afb287d5f0 2023-10-05T19:30:21.595 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:21.590+0000 7f296ab77700 1 -- 172.21.15.167:0/3071637523 <== mon.0 v2:172.21.15.118:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f295400e920 con 0x55afb287d5f0 2023-10-05T19:30:21.600 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:21.590+0000 7f296e044e40 1 -- 172.21.15.167:0/3071637523 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55afb287d5f0 msgr2=0x55afb287f9d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-05T19:30:21.600 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:21.590+0000 7f296e044e40 1 --2- 172.21.15.167:0/3071637523 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55afb287d5f0 0x55afb287f9d0 secure :-1 s=READY pgs=657 cs=0 l=1 rev1=1 crypto rx=0x7f295400a5c0 tx=0x7f2954005e00 comp rx=0 tx=0).stop 2023-10-05T19:30:21.601 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:21.590+0000 7f296e044e40 1 -- 172.21.15.167:0/3071637523 shutdown_connections 2023-10-05T19:30:21.601 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:21.590+0000 7f296e044e40 1 --2- 172.21.15.167:0/3071637523 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55afb287d5f0 0x55afb287f9d0 unknown :-1 s=CLOSED pgs=657 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-05T19:30:21.601 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:21.590+0000 7f296e044e40 1 -- 172.21.15.167:0/3071637523 >> 172.21.15.167:0/3071637523 conn(0x55afb27ec860 msgr2=0x55afb27ecc60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-05T19:30:21.601 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:21.590+0000 7f296e044e40 1 -- 172.21.15.167:0/3071637523 shutdown_connections 2023-10-05T19:30:21.601 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:21.590+0000 7f296e044e40 1 -- 172.21.15.167:0/3071637523 wait complete. 2023-10-05T19:30:21.601 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:21.594+0000 7f296e044e40 1 Processor -- start 2023-10-05T19:30:21.601 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:21.594+0000 7f296e044e40 1 -- start start 2023-10-05T19:30:21.601 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:21.594+0000 7f296e044e40 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55afb2879010 0x55afb2898190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-05T19:30:21.601 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:21.594+0000 7f296e044e40 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x55afb2744e30 con 0x55afb2879010 2023-10-05T19:30:21.602 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:21.594+0000 7f296bb79700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55afb2879010 0x55afb2898190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-05T19:30:21.602 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:21.594+0000 7f296bb79700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55afb2879010 0x55afb2898190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.118:3300/0 says I am v2:172.21.15.167:32944/0 (socket says 172.21.15.167:32944) 2023-10-05T19:30:21.602 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:21.594+0000 7f296bb79700 1 -- 172.21.15.167:0/4157512443 learned_addr learned my addr 172.21.15.167:0/4157512443 (peer_addr_for_me v2:172.21.15.167:0/0) 2023-10-05T19:30:21.602 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:21.594+0000 7f296bb79700 1 -- 172.21.15.167:0/4157512443 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f295400a180 con 0x55afb2879010 2023-10-05T19:30:21.602 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:21.594+0000 7f296bb79700 1 --2- 172.21.15.167:0/4157512443 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55afb2879010 0x55afb2898190 secure :-1 s=READY pgs=658 cs=0 l=1 rev1=1 crypto rx=0x7f295c001d30 tx=0x7f295c001900 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f3920c722499baee server_cookie=0 in_seq=0 out_seq=0 2023-10-05T19:30:21.602 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:21.594+0000 7f2969b75700 1 -- 172.21.15.167:0/4157512443 <== mon.0 v2:172.21.15.118:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f295c014070 con 0x55afb2879010 2023-10-05T19:30:21.602 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:21.594+0000 7f2969b75700 1 -- 172.21.15.167:0/4157512443 <== mon.0 v2:172.21.15.118:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f295c008070 con 0x55afb2879010 2023-10-05T19:30:21.602 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:21.594+0000 7f2969b75700 1 -- 172.21.15.167:0/4157512443 <== mon.0 v2:172.21.15.118:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f295c00bd30 con 0x55afb2879010 2023-10-05T19:30:21.602 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:21.594+0000 7f296e044e40 1 -- 172.21.15.167:0/4157512443 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55afb2879010 msgr2=0x55afb2898190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-05T19:30:21.602 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:21.594+0000 7f296e044e40 1 --2- 172.21.15.167:0/4157512443 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55afb2879010 0x55afb2898190 secure :-1 s=READY pgs=658 cs=0 l=1 rev1=1 crypto rx=0x7f295c001d30 tx=0x7f295c001900 comp rx=0 tx=0).stop 2023-10-05T19:30:21.603 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:21.594+0000 7f296e044e40 1 -- 172.21.15.167:0/4157512443 shutdown_connections 2023-10-05T19:30:21.603 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:21.594+0000 7f296e044e40 1 --2- 172.21.15.167:0/4157512443 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55afb2879010 0x55afb2898190 unknown :-1 s=CLOSED pgs=658 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-05T19:30:21.603 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:21.594+0000 7f296e044e40 1 -- 172.21.15.167:0/4157512443 >> 172.21.15.167:0/4157512443 conn(0x55afb27ec860 msgr2=0x55afb2893d50 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-05T19:30:21.603 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:21.594+0000 7f296e044e40 1 -- 172.21.15.167:0/4157512443 shutdown_connections 2023-10-05T19:30:21.603 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:21.594+0000 7f296e044e40 1 -- 172.21.15.167:0/4157512443 wait complete. 2023-10-05T19:30:21.603 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:21.594+0000 7f296e044e40 1 Processor -- start 2023-10-05T19:30:21.603 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:21.594+0000 7f296e044e40 1 -- start start 2023-10-05T19:30:21.603 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:21.594+0000 7f296e044e40 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55afb2879010 0x55afb289ef60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-05T19:30:21.603 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:21.594+0000 7f296e044e40 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x55afb2717ed0 con 0x55afb2879010 2023-10-05T19:30:21.604 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:21.598+0000 7f296bb79700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55afb2879010 0x55afb289ef60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-05T19:30:21.604 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:21.598+0000 7f296bb79700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55afb2879010 0x55afb289ef60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.118:3300/0 says I am v2:172.21.15.167:32954/0 (socket says 172.21.15.167:32954) 2023-10-05T19:30:21.604 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:21.598+0000 7f296bb79700 1 -- 172.21.15.167:0/1098790227 learned_addr learned my addr 172.21.15.167:0/1098790227 (peer_addr_for_me v2:172.21.15.167:0/0) 2023-10-05T19:30:21.606 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:21.598+0000 7f296bb79700 1 -- 172.21.15.167:0/1098790227 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f295c00a040 con 0x55afb2879010 2023-10-05T19:30:21.606 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:21.598+0000 7f296bb79700 1 --2- 172.21.15.167:0/1098790227 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55afb2879010 0x55afb289ef60 secure :-1 s=READY pgs=659 cs=0 l=1 rev1=1 crypto rx=0x55afb2894870 tx=0x7f295c016000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-05T19:30:21.606 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:21.598+0000 7f2963fff700 1 -- 172.21.15.167:0/1098790227 <== mon.0 v2:172.21.15.118:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f295c00bee0 con 0x55afb2879010 2023-10-05T19:30:21.606 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:21.598+0000 7f2963fff700 1 -- 172.21.15.167:0/1098790227 <== mon.0 v2:172.21.15.118:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f295c021070 con 0x55afb2879010 2023-10-05T19:30:21.607 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:21.598+0000 7f296e044e40 1 -- 172.21.15.167:0/1098790227 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x55afb289f4a0 con 0x55afb2879010 2023-10-05T19:30:21.607 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:21.598+0000 7f2963fff700 1 -- 172.21.15.167:0/1098790227 <== mon.0 v2:172.21.15.118:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f295c01a600 con 0x55afb2879010 2023-10-05T19:30:21.607 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:21.598+0000 7f296e044e40 1 -- 172.21.15.167:0/1098790227 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55afb289f940 con 0x55afb2879010 2023-10-05T19:30:21.607 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:21.602+0000 7f2963fff700 1 -- 172.21.15.167:0/1098790227 <== mon.0 v2:172.21.15.118:3300/0 4 ==== mgrmap(e 5) v1 ==== 40329+0+0 (secure 0 0 0) 0x7f295c01a800 con 0x55afb2879010 2023-10-05T19:30:21.607 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:21.602+0000 7f2963fff700 1 --2- 172.21.15.167:0/1098790227 >> [v2:172.21.15.167:6824/29889,v1:172.21.15.167:6825/29889] conn(0x7f294c030c00 0x7f294c0330b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-05T19:30:21.607 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:21.602+0000 7f296e044e40 20 rados->read ofs=0 len=0 2023-10-05T19:30:21.607 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:21.602+0000 7f2963fff700 1 -- 172.21.15.167:0/1098790227 <== mon.0 v2:172.21.15.118:3300/0 5 ==== osd_map(32..32 src has 1..32) v4 ==== 6260+0+0 (secure 0 0 0) 0x7f295c044ee0 con 0x55afb2879010 2023-10-05T19:30:21.607 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:21.602+0000 7f296b378700 1 --2- 172.21.15.167:0/1098790227 >> [v2:172.21.15.167:6824/29889,v1:172.21.15.167:6825/29889] conn(0x7f294c030c00 0x7f294c0330b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-05T19:30:21.607 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:21.602+0000 7f296e044e40 1 --2- 172.21.15.167:0/1098790227 >> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] conn(0x55afb287d5f0 0x55afb28aaa50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-05T19:30:21.607 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:21.602+0000 7f296e044e40 1 -- 172.21.15.167:0/1098790227 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:1 5.12 5:49953fa1:::default.realm:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32) v8 -- 0x55afb28ab100 con 0x55afb287d5f0 2023-10-05T19:30:21.608 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:21.602+0000 7f296c37a700 1 --2- 172.21.15.167:0/1098790227 >> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] conn(0x55afb287d5f0 0x55afb28aaa50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-05T19:30:21.608 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:21.602+0000 7f296b378700 1 --2- 172.21.15.167:0/1098790227 >> [v2:172.21.15.167:6824/29889,v1:172.21.15.167:6825/29889] conn(0x7f294c030c00 0x7f294c0330b0 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7f2954000f80 tx=0x7f2954013000 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-05T19:30:21.608 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:21.602+0000 7f296c37a700 1 --2- 172.21.15.167:0/1098790227 >> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] conn(0x55afb287d5f0 0x55afb28aaa50 crc :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-05T19:30:21.608 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:21.602+0000 7f296c37a700 1 -- 172.21.15.167:0/1098790227 <== osd.1 v2:172.21.15.167:6800/30035 1 ==== osd_op_reply(1 default.realm [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 ==== 157+0+0 (crc 0 0 0) 0x7f296400a500 con 0x55afb287d5f0 2023-10-05T19:30:21.608 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:21.602+0000 7f296e044e40 20 rados_obj.operate() r=-2 bl.length=0 2023-10-05T19:30:21.608 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:21.602+0000 7f296e044e40 20 realm 2023-10-05T19:30:21.608 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:21.602+0000 7f296e044e40 20 rados->read ofs=0 len=0 2023-10-05T19:30:21.608 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:21.602+0000 7f296e044e40 1 -- 172.21.15.167:0/1098790227 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:2 5.12 5:49953fa1:::default.realm:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32) v8 -- 0x55afb28a80b0 con 0x55afb287d5f0 2023-10-05T19:30:21.610 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:21.602+0000 7f296c37a700 1 -- 172.21.15.167:0/1098790227 <== osd.1 v2:172.21.15.167:6800/30035 2 ==== osd_op_reply(2 default.realm [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 ==== 157+0+0 (crc 0 0 0) 0x7f296400a500 con 0x55afb287d5f0 2023-10-05T19:30:21.611 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:21.602+0000 7f296e044e40 20 rados_obj.operate() r=-2 bl.length=0 2023-10-05T19:30:21.611 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:21.602+0000 7f296e044e40 4 RGWPeriod::init failed to init realm id : (2) No such file or directory 2023-10-05T19:30:21.611 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:21.602+0000 7f296e044e40 20 rados->read ofs=0 len=0 2023-10-05T19:30:21.611 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:21.602+0000 7f296e044e40 1 -- 172.21.15.167:0/1098790227 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:3 5.12 5:49953fa1:::default.realm:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32) v8 -- 0x55afb28abbe0 con 0x55afb287d5f0 2023-10-05T19:30:21.611 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:21.602+0000 7f296c37a700 1 -- 172.21.15.167:0/1098790227 <== osd.1 v2:172.21.15.167:6800/30035 3 ==== osd_op_reply(3 default.realm [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 ==== 157+0+0 (crc 0 0 0) 0x7f296400a500 con 0x55afb287d5f0 2023-10-05T19:30:21.611 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:21.602+0000 7f296e044e40 20 rados_obj.operate() r=-2 bl.length=0 2023-10-05T19:30:21.611 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:21.602+0000 7f296e044e40 20 rados->read ofs=0 len=0 2023-10-05T19:30:21.611 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:21.602+0000 7f296e044e40 1 -- 172.21.15.167:0/1098790227 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:4 5.f 5:f43b6ece:::zone_names.default:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32) v8 -- 0x55afb28ab7c0 con 0x55afb287d5f0 2023-10-05T19:30:21.611 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:21.606+0000 7f296c37a700 1 -- 172.21.15.167:0/1098790227 <== osd.1 v2:172.21.15.167:6800/30035 4 ==== osd_op_reply(4 zone_names.default [read 0~46 out=46b] v0'0 uv1 ondisk = 0) v8 ==== 162+0+46 (crc 0 0 0) 0x7f296400a500 con 0x55afb287d5f0 2023-10-05T19:30:21.612 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:21.606+0000 7f296e044e40 20 rados_obj.operate() r=0 bl.length=46 2023-10-05T19:30:21.612 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:21.606+0000 7f296e044e40 20 rados->read ofs=0 len=0 2023-10-05T19:30:21.612 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:21.606+0000 7f296e044e40 1 -- 172.21.15.167:0/1098790227 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:5 5.1d 5:b8374eb4:::zone_info.9a9df965-6612-45d4-a45f-6e3c889bacc2:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32) v8 -- 0x55afb28ac650 con 0x55afb287d5f0 2023-10-05T19:30:21.612 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:21.606+0000 7f296c37a700 1 -- 172.21.15.167:0/1098790227 <== osd.1 v2:172.21.15.167:6800/30035 5 ==== osd_op_reply(5 zone_info.9a9df965-6612-45d4-a45f-6e3c889bacc2 [read 0~874 out=874b] v0'0 uv2 ondisk = 0) v8 ==== 190+0+874 (crc 0 0 0) 0x7f296400a500 con 0x55afb287d5f0 2023-10-05T19:30:21.612 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:21.606+0000 7f296e044e40 20 rados_obj.operate() r=0 bl.length=874 2023-10-05T19:30:21.612 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:21.606+0000 7f296e044e40 20 searching for the correct realm 2023-10-05T19:30:21.612 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:21.606+0000 7f296e044e40 1 --2- 172.21.15.167:0/1098790227 >> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] conn(0x55afb28ae4a0 0x55afb28b0880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-05T19:30:21.612 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:21.606+0000 7f296e044e40 1 -- 172.21.15.167:0/1098790227 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:6 5.0 5:00000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x55afb28b0dc0 con 0x55afb28ae4a0 2023-10-05T19:30:21.612 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:21.606+0000 7f296bb79700 1 --2- 172.21.15.167:0/1098790227 >> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] conn(0x55afb28ae4a0 0x55afb28b0880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-05T19:30:21.612 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:21.606+0000 7f296bb79700 1 --2- 172.21.15.167:0/1098790227 >> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] conn(0x55afb28ae4a0 0x55afb28b0880 crc :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-05T19:30:21.614 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:21.606+0000 7f296bb79700 1 -- 172.21.15.167:0/1098790227 <== osd.3 v2:172.21.15.167:6808/30036 1 ==== osd_op_reply(6 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f295c016070 con 0x55afb28ae4a0 2023-10-05T19:30:21.614 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:21.606+0000 7f296bb79700 1 --2- 172.21.15.167:0/1098790227 >> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] conn(0x7f295c0176e0 0x7f295c0420e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-05T19:30:21.614 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:21.606+0000 7f296bb79700 1 -- 172.21.15.167:0/1098790227 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:7 5.10 5:08000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x55afb28b0dc0 con 0x7f295c0176e0 2023-10-05T19:30:21.614 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:21.610+0000 7f296b378700 1 --2- 172.21.15.167:0/1098790227 >> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] conn(0x7f295c0176e0 0x7f295c0420e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-05T19:30:21.614 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:21.610+0000 7f296b378700 1 --2- 172.21.15.167:0/1098790227 >> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] conn(0x7f295c0176e0 0x7f295c0420e0 crc :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-05T19:30:21.614 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:21.610+0000 7f296b378700 1 -- 172.21.15.167:0/1098790227 <== osd.2 v2:172.21.15.167:6816/30037 1 ==== osd_op_reply(7 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f2954017050 con 0x7f295c0176e0 2023-10-05T19:30:21.614 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:21.610+0000 7f296b378700 1 --2- 172.21.15.167:0/1098790227 >> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] conn(0x7f2954010070 0x7f295401c430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-05T19:30:21.615 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:21.610+0000 7f296b378700 1 -- 172.21.15.167:0/1098790227 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:8 5.8 5:10000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x55afb28b0dc0 con 0x7f2954010070 2023-10-05T19:30:21.615 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:21.610+0000 7f296c37a700 1 --2- 172.21.15.167:0/1098790227 >> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] conn(0x7f2954010070 0x7f295401c430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-05T19:30:21.616 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:21.610+0000 7f296c37a700 1 --2- 172.21.15.167:0/1098790227 >> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] conn(0x7f2954010070 0x7f295401c430 crc :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-05T19:30:21.616 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:21.610+0000 7f296c37a700 1 -- 172.21.15.167:0/1098790227 <== osd.0 v2:172.21.15.118:6800/27690 1 ==== osd_op_reply(8 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f2964014040 con 0x7f2954010070 2023-10-05T19:30:21.616 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:21.610+0000 7f296c37a700 1 -- 172.21.15.167:0/1098790227 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:9 5.18 5:18000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x55afb28b0dc0 con 0x55afb287d5f0 2023-10-05T19:30:21.616 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:21.614+0000 7f296c37a700 1 -- 172.21.15.167:0/1098790227 <== osd.1 v2:172.21.15.167:6800/30035 6 ==== osd_op_reply(9 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f2964014040 con 0x55afb287d5f0 2023-10-05T19:30:21.616 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:21.614+0000 7f296c37a700 1 -- 172.21.15.167:0/1098790227 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:10 5.4 5:20000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x55afb28b0dc0 con 0x55afb287d5f0 2023-10-05T19:30:21.617 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:21.614+0000 7f296c37a700 1 -- 172.21.15.167:0/1098790227 <== osd.1 v2:172.21.15.167:6800/30035 7 ==== osd_op_reply(10 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f2964014040 con 0x55afb287d5f0 2023-10-05T19:30:21.617 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:21.614+0000 7f296c37a700 1 -- 172.21.15.167:0/1098790227 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:11 5.14 5:28000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x55afb28b0dc0 con 0x7f2954010070 2023-10-05T19:30:21.617 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:21.614+0000 7f296c37a700 1 -- 172.21.15.167:0/1098790227 <== osd.0 v2:172.21.15.118:6800/27690 2 ==== osd_op_reply(11 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f2964014040 con 0x7f2954010070 2023-10-05T19:30:21.617 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:21.614+0000 7f296c37a700 1 -- 172.21.15.167:0/1098790227 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:12 5.c 5:30000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x55afb28b0dc0 con 0x7f2954010070 2023-10-05T19:30:21.618 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:21.614+0000 7f296c37a700 1 -- 172.21.15.167:0/1098790227 <== osd.0 v2:172.21.15.118:6800/27690 3 ==== osd_op_reply(12 [pgnls start_epoch 32 out=112b] v23'1 uv1 ondisk = 1) v8 ==== 144+0+112 (crc 0 0 0) 0x7f2964014040 con 0x7f2954010070 2023-10-05T19:30:21.618 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:21.614+0000 7f296c37a700 1 -- 172.21.15.167:0/1098790227 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:13 5.1c 5:38000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x55afb28b0dc0 con 0x7f295c0176e0 2023-10-05T19:30:21.619 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:21.614+0000 7f296b378700 1 -- 172.21.15.167:0/1098790227 <== osd.2 v2:172.21.15.167:6816/30037 2 ==== osd_op_reply(13 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f2954017050 con 0x7f295c0176e0 2023-10-05T19:30:21.619 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:21.614+0000 7f296b378700 1 -- 172.21.15.167:0/1098790227 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:14 5.2 5:40000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x55afb28b0dc0 con 0x55afb28ae4a0 2023-10-05T19:30:21.620 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:21.618+0000 7f296bb79700 1 -- 172.21.15.167:0/1098790227 <== osd.3 v2:172.21.15.167:6808/30036 2 ==== osd_op_reply(14 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f295c016070 con 0x55afb28ae4a0 2023-10-05T19:30:21.620 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:21.618+0000 7f296bb79700 1 -- 172.21.15.167:0/1098790227 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:15 5.12 5:48000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x55afb28b0dc0 con 0x55afb287d5f0 2023-10-05T19:30:21.620 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:21.618+0000 7f296c37a700 1 -- 172.21.15.167:0/1098790227 <== osd.1 v2:172.21.15.167:6800/30035 8 ==== osd_op_reply(15 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f2964014040 con 0x55afb287d5f0 2023-10-05T19:30:21.620 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:21.618+0000 7f296c37a700 1 -- 172.21.15.167:0/1098790227 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:16 5.a 5:50000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x55afb28b0dc0 con 0x7f2954010070 2023-10-05T19:30:21.621 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:21.618+0000 7f296c37a700 1 -- 172.21.15.167:0/1098790227 <== osd.0 v2:172.21.15.118:6800/27690 4 ==== osd_op_reply(16 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f2964022040 con 0x7f2954010070 2023-10-05T19:30:21.621 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:21.618+0000 7f296c37a700 1 -- 172.21.15.167:0/1098790227 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:17 5.1a 5:58000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x55afb28b0dc0 con 0x55afb287d5f0 2023-10-05T19:30:21.622 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:21.618+0000 7f296c37a700 1 -- 172.21.15.167:0/1098790227 <== osd.1 v2:172.21.15.167:6800/30035 9 ==== osd_op_reply(17 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f2964022040 con 0x55afb287d5f0 2023-10-05T19:30:21.622 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:21.618+0000 7f296c37a700 1 -- 172.21.15.167:0/1098790227 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:18 5.6 5:60000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x55afb28b0dc0 con 0x7f295c0176e0 2023-10-05T19:30:21.622 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:21.618+0000 7f296b378700 1 -- 172.21.15.167:0/1098790227 <== osd.2 v2:172.21.15.167:6816/30037 3 ==== osd_op_reply(18 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f2954017050 con 0x7f295c0176e0 2023-10-05T19:30:21.623 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:21.618+0000 7f296b378700 1 -- 172.21.15.167:0/1098790227 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:19 5.16 5:68000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x55afb28b0dc0 con 0x55afb28ae4a0 2023-10-05T19:30:21.624 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:21.622+0000 7f296bb79700 1 -- 172.21.15.167:0/1098790227 <== osd.3 v2:172.21.15.167:6808/30036 3 ==== osd_op_reply(19 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f295c016070 con 0x55afb28ae4a0 2023-10-05T19:30:21.624 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:21.622+0000 7f296bb79700 1 -- 172.21.15.167:0/1098790227 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:20 5.e 5:70000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x55afb28b0dc0 con 0x55afb28ae4a0 2023-10-05T19:30:21.624 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:21.622+0000 7f296bb79700 1 -- 172.21.15.167:0/1098790227 <== osd.3 v2:172.21.15.167:6808/30036 4 ==== osd_op_reply(20 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f295c016070 con 0x55afb28ae4a0 2023-10-05T19:30:21.624 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:21.622+0000 7f296bb79700 1 -- 172.21.15.167:0/1098790227 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:21 5.1e 5:78000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x55afb28b0dc0 con 0x7f2954010070 2023-10-05T19:30:21.626 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:21.622+0000 7f296c37a700 1 -- 172.21.15.167:0/1098790227 <== osd.0 v2:172.21.15.118:6800/27690 5 ==== osd_op_reply(21 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f2964022040 con 0x7f2954010070 2023-10-05T19:30:21.626 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:21.622+0000 7f296c37a700 1 -- 172.21.15.167:0/1098790227 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:22 5.1 5:80000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x55afb28b0dc0 con 0x55afb28ae4a0 2023-10-05T19:30:21.626 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:21.622+0000 7f296bb79700 1 -- 172.21.15.167:0/1098790227 <== osd.3 v2:172.21.15.167:6808/30036 5 ==== osd_op_reply(22 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f295c039040 con 0x55afb28ae4a0 2023-10-05T19:30:21.627 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:21.622+0000 7f296bb79700 1 -- 172.21.15.167:0/1098790227 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:23 5.11 5:88000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x55afb28b0dc0 con 0x55afb28ae4a0 2023-10-05T19:30:21.627 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:21.622+0000 7f296bb79700 1 -- 172.21.15.167:0/1098790227 <== osd.3 v2:172.21.15.167:6808/30036 6 ==== osd_op_reply(23 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f295c039040 con 0x55afb28ae4a0 2023-10-05T19:30:21.627 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:21.622+0000 7f296bb79700 1 -- 172.21.15.167:0/1098790227 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:24 5.9 5:90000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x7f295c03d040 con 0x55afb287d5f0 2023-10-05T19:30:21.628 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:21.626+0000 7f296c37a700 1 -- 172.21.15.167:0/1098790227 <== osd.1 v2:172.21.15.167:6800/30035 10 ==== osd_op_reply(24 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f2964022040 con 0x55afb287d5f0 2023-10-05T19:30:21.628 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:21.626+0000 7f296c37a700 1 -- 172.21.15.167:0/1098790227 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:25 5.19 5:98000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x7f295c03d040 con 0x55afb287d5f0 2023-10-05T19:30:21.628 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:21.626+0000 7f296c37a700 1 -- 172.21.15.167:0/1098790227 <== osd.1 v2:172.21.15.167:6800/30035 11 ==== osd_op_reply(25 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f2964022040 con 0x55afb287d5f0 2023-10-05T19:30:21.629 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:21.626+0000 7f296c37a700 1 -- 172.21.15.167:0/1098790227 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:26 5.5 5:a0000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x7f295c03d040 con 0x7f2954010070 2023-10-05T19:30:21.629 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:21.626+0000 7f296c37a700 1 -- 172.21.15.167:0/1098790227 <== osd.0 v2:172.21.15.118:6800/27690 6 ==== osd_op_reply(26 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f2964022040 con 0x7f2954010070 2023-10-05T19:30:21.630 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:21.626+0000 7f296c37a700 1 -- 172.21.15.167:0/1098790227 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:27 5.15 5:a8000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x7f295c03d040 con 0x55afb28ae4a0 2023-10-05T19:30:21.630 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:21.626+0000 7f296bb79700 1 -- 172.21.15.167:0/1098790227 <== osd.3 v2:172.21.15.167:6808/30036 7 ==== osd_op_reply(27 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f295c039040 con 0x55afb28ae4a0 2023-10-05T19:30:21.630 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:21.626+0000 7f296bb79700 1 -- 172.21.15.167:0/1098790227 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:28 5.d 5:b0000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x7f295c03d040 con 0x7f295c0176e0 2023-10-05T19:30:21.631 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:21.626+0000 7f296b378700 1 -- 172.21.15.167:0/1098790227 <== osd.2 v2:172.21.15.167:6816/30037 4 ==== osd_op_reply(28 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f2954017050 con 0x7f295c0176e0 2023-10-05T19:30:21.631 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:21.626+0000 7f296b378700 1 -- 172.21.15.167:0/1098790227 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:29 5.1d 5:b8000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x7f295c03d040 con 0x55afb287d5f0 2023-10-05T19:30:21.632 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:21.630+0000 7f296c37a700 1 -- 172.21.15.167:0/1098790227 <== osd.1 v2:172.21.15.167:6800/30035 12 ==== osd_op_reply(29 [pgnls start_epoch 32 out=107b] v30'2 uv2 ondisk = 1) v8 ==== 144+0+107 (crc 0 0 0) 0x7f2964022040 con 0x55afb287d5f0 2023-10-05T19:30:21.632 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:21.630+0000 7f296c37a700 1 -- 172.21.15.167:0/1098790227 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:30 5.3 5:c0000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x7f295c03d040 con 0x7f2954010070 2023-10-05T19:30:21.633 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:21.630+0000 7f296c37a700 1 -- 172.21.15.167:0/1098790227 <== osd.0 v2:172.21.15.118:6800/27690 7 ==== osd_op_reply(30 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f2964022040 con 0x7f2954010070 2023-10-05T19:30:21.633 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:21.630+0000 7f296c37a700 1 -- 172.21.15.167:0/1098790227 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:31 5.13 5:c8000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x7f295c03d040 con 0x55afb28ae4a0 2023-10-05T19:30:21.634 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:21.630+0000 7f296bb79700 1 -- 172.21.15.167:0/1098790227 <== osd.3 v2:172.21.15.167:6808/30036 8 ==== osd_op_reply(31 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f295c039040 con 0x55afb28ae4a0 2023-10-05T19:30:21.634 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:21.630+0000 7f296bb79700 1 -- 172.21.15.167:0/1098790227 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:32 5.b 5:d0000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x7f295c03d040 con 0x7f2954010070 2023-10-05T19:30:21.634 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:21.630+0000 7f296c37a700 1 -- 172.21.15.167:0/1098790227 <== osd.0 v2:172.21.15.118:6800/27690 8 ==== osd_op_reply(32 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f2964022040 con 0x7f2954010070 2023-10-05T19:30:21.635 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:21.630+0000 7f296c37a700 1 -- 172.21.15.167:0/1098790227 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:33 5.1b 5:d8000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x7f295c03d040 con 0x7f295c0176e0 2023-10-05T19:30:21.636 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:21.630+0000 7f296b378700 1 -- 172.21.15.167:0/1098790227 <== osd.2 v2:172.21.15.167:6816/30037 5 ==== osd_op_reply(33 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f2954017050 con 0x7f295c0176e0 2023-10-05T19:30:21.636 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:21.634+0000 7f296b378700 1 -- 172.21.15.167:0/1098790227 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:34 5.7 5:e0000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x7f295c03d040 con 0x55afb28ae4a0 2023-10-05T19:30:21.636 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:21.634+0000 7f296bb79700 1 -- 172.21.15.167:0/1098790227 <== osd.3 v2:172.21.15.167:6808/30036 9 ==== osd_op_reply(34 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f295c039040 con 0x55afb28ae4a0 2023-10-05T19:30:21.636 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:21.634+0000 7f296bb79700 1 -- 172.21.15.167:0/1098790227 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:35 5.17 5:e8000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x7f295c03d040 con 0x7f2954010070 2023-10-05T19:30:21.637 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:21.634+0000 7f296c37a700 1 -- 172.21.15.167:0/1098790227 <== osd.0 v2:172.21.15.118:6800/27690 9 ==== osd_op_reply(35 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f2964022040 con 0x7f2954010070 2023-10-05T19:30:21.638 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:21.634+0000 7f296c37a700 1 -- 172.21.15.167:0/1098790227 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:36 5.f 5:f0000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x7f295c03d040 con 0x55afb287d5f0 2023-10-05T19:30:21.638 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:21.634+0000 7f296c37a700 1 -- 172.21.15.167:0/1098790227 <== osd.1 v2:172.21.15.167:6800/30035 13 ==== osd_op_reply(36 [pgnls start_epoch 32 out=115b] v23'2 uv2 ondisk = 1) v8 ==== 144+0+115 (crc 0 0 0) 0x7f296402f080 con 0x55afb287d5f0 2023-10-05T19:30:21.638 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:21.634+0000 7f296c37a700 1 -- 172.21.15.167:0/1098790227 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:37 5.1f 5:f8000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x7f295c03d040 con 0x7f295c0176e0 2023-10-05T19:30:21.639 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:21.634+0000 7f296b378700 1 -- 172.21.15.167:0/1098790227 <== osd.2 v2:172.21.15.167:6816/30037 6 ==== osd_op_reply(37 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f2954017050 con 0x7f295c0176e0 2023-10-05T19:30:21.639 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:21.634+0000 7f296e044e40 20 RGWRados::pool_iterate: got zonegroup_info.4f1f719d-9648-417b-b10c-e5d939825419 2023-10-05T19:30:21.639 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:21.634+0000 7f296e044e40 20 RGWRados::pool_iterate: got zone_info.9a9df965-6612-45d4-a45f-6e3c889bacc2 2023-10-05T19:30:21.640 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:21.634+0000 7f296e044e40 20 RGWRados::pool_iterate: got zone_names.default 2023-10-05T19:30:21.640 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:21.634+0000 7f296e044e40 20 RGWRados::pool_iterate: got zonegroups_names.default 2023-10-05T19:30:21.640 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:21.634+0000 7f296e044e40 20 rados->read ofs=0 len=0 2023-10-05T19:30:21.640 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:21.634+0000 7f296e044e40 1 -- 172.21.15.167:0/1098790227 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:38 5.12 5:49953fa1:::default.realm:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32) v8 -- 0x55afb28ad250 con 0x55afb287d5f0 2023-10-05T19:30:21.641 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:21.638+0000 7f296c37a700 1 -- 172.21.15.167:0/1098790227 <== osd.1 v2:172.21.15.167:6800/30035 14 ==== osd_op_reply(38 default.realm [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 ==== 157+0+0 (crc 0 0 0) 0x7f296402f080 con 0x55afb287d5f0 2023-10-05T19:30:21.641 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:21.638+0000 7f296e044e40 20 rados_obj.operate() r=-2 bl.length=0 2023-10-05T19:30:21.641 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:21.638+0000 7f296e044e40 20 rados->read ofs=0 len=0 2023-10-05T19:30:21.641 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:21.638+0000 7f296e044e40 1 -- 172.21.15.167:0/1098790227 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:39 5.f 5:f4c53578:::zonegroups_names.default:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32) v8 -- 0x55afb28adbd0 con 0x55afb287d5f0 2023-10-05T19:30:21.641 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:21.638+0000 7f296c37a700 1 -- 172.21.15.167:0/1098790227 <== osd.1 v2:172.21.15.167:6800/30035 15 ==== osd_op_reply(39 zonegroups_names.default [read 0~46 out=46b] v0'0 uv2 ondisk = 0) v8 ==== 168+0+46 (crc 0 0 0) 0x7f296402f080 con 0x55afb287d5f0 2023-10-05T19:30:21.641 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:21.638+0000 7f296e044e40 20 rados_obj.operate() r=0 bl.length=46 2023-10-05T19:30:21.641 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:21.638+0000 7f296e044e40 20 rados->read ofs=0 len=0 2023-10-05T19:30:21.641 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:21.638+0000 7f296e044e40 1 -- 172.21.15.167:0/1098790227 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:40 5.c 5:30d299a5:::zonegroup_info.4f1f719d-9648-417b-b10c-e5d939825419:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32) v8 -- 0x55afb28ace40 con 0x7f2954010070 2023-10-05T19:30:21.642 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:21.638+0000 7f296c37a700 1 -- 172.21.15.167:0/1098790227 <== osd.0 v2:172.21.15.118:6800/27690 10 ==== osd_op_reply(40 zonegroup_info.4f1f719d-9648-417b-b10c-e5d939825419 [read 0~362 out=362b] v0'0 uv1 ondisk = 0) v8 ==== 195+0+362 (crc 0 0 0) 0x7f296402f080 con 0x7f2954010070 2023-10-05T19:30:21.642 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:21.638+0000 7f296e044e40 20 rados_obj.operate() r=0 bl.length=362 2023-10-05T19:30:21.642 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:21.638+0000 7f296e044e40 20 rados->read ofs=0 len=0 2023-10-05T19:30:21.642 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:21.638+0000 7f296e044e40 1 -- 172.21.15.167:0/1098790227 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:41 5.13 5:cbbf6c60:::converted:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32) v8 -- 0x55afb28b16b0 con 0x55afb28ae4a0 2023-10-05T19:30:21.643 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:21.638+0000 7f296bb79700 1 -- 172.21.15.167:0/1098790227 <== osd.3 v2:172.21.15.167:6808/30036 10 ==== osd_op_reply(41 converted [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 ==== 153+0+0 (crc 0 0 0) 0x7f295c039040 con 0x55afb28ae4a0 2023-10-05T19:30:21.643 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:21.638+0000 7f296e044e40 20 rados_obj.operate() r=-2 bl.length=0 2023-10-05T19:30:21.643 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:21.638+0000 7f296e044e40 20 rados->read ofs=0 len=0 2023-10-05T19:30:21.643 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:21.638+0000 7f296e044e40 1 -- 172.21.15.167:0/1098790227 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:42 5.12 5:49953fa1:::default.realm:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32) v8 -- 0x55afb28b1a60 con 0x55afb287d5f0 2023-10-05T19:30:21.644 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:21.638+0000 7f296c37a700 1 -- 172.21.15.167:0/1098790227 <== osd.1 v2:172.21.15.167:6800/30035 16 ==== osd_op_reply(42 default.realm [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 ==== 157+0+0 (crc 0 0 0) 0x7f296402f080 con 0x55afb287d5f0 2023-10-05T19:30:21.644 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:21.638+0000 7f296e044e40 20 rados_obj.operate() r=-2 bl.length=0 2023-10-05T19:30:21.644 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:21.638+0000 7f296e044e40 20 rados->read ofs=0 len=0 2023-10-05T19:30:21.644 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:21.638+0000 7f296e044e40 1 -- 172.21.15.167:0/1098790227 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:43 5.f 5:f4c53578:::zonegroups_names.default:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32) v8 -- 0x55afb28b1a60 con 0x55afb287d5f0 2023-10-05T19:30:21.644 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:21.642+0000 7f296c37a700 1 -- 172.21.15.167:0/1098790227 <== osd.1 v2:172.21.15.167:6800/30035 17 ==== osd_op_reply(43 zonegroups_names.default [read 0~46 out=46b] v0'0 uv2 ondisk = 0) v8 ==== 168+0+46 (crc 0 0 0) 0x7f296402f080 con 0x55afb287d5f0 2023-10-05T19:30:21.644 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:21.642+0000 7f296e044e40 20 rados_obj.operate() r=0 bl.length=46 2023-10-05T19:30:21.645 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:21.642+0000 7f296e044e40 1 -- 172.21.15.167:0/1098790227 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:44 5.0 5:00000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x55afb28b2a70 con 0x55afb28ae4a0 2023-10-05T19:30:21.645 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:21.642+0000 7f296bb79700 1 -- 172.21.15.167:0/1098790227 <== osd.3 v2:172.21.15.167:6808/30036 11 ==== osd_op_reply(44 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f295c039040 con 0x55afb28ae4a0 2023-10-05T19:30:21.646 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:21.642+0000 7f296bb79700 1 -- 172.21.15.167:0/1098790227 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:45 5.10 5:08000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x55afb28b2a70 con 0x7f295c0176e0 2023-10-05T19:30:21.646 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:21.642+0000 7f296b378700 1 -- 172.21.15.167:0/1098790227 <== osd.2 v2:172.21.15.167:6816/30037 7 ==== osd_op_reply(45 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f2954017050 con 0x7f295c0176e0 2023-10-05T19:30:21.646 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:21.642+0000 7f296b378700 1 -- 172.21.15.167:0/1098790227 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:46 5.8 5:10000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x55afb28b2a70 con 0x7f2954010070 2023-10-05T19:30:21.648 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:21.646+0000 7f296c37a700 1 -- 172.21.15.167:0/1098790227 <== osd.0 v2:172.21.15.118:6800/27690 11 ==== osd_op_reply(46 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f296402f080 con 0x7f2954010070 2023-10-05T19:30:21.648 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:21.646+0000 7f296c37a700 1 -- 172.21.15.167:0/1098790227 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:47 5.18 5:18000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x55afb28ace40 con 0x55afb287d5f0 2023-10-05T19:30:21.648 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:21.646+0000 7f296c37a700 1 -- 172.21.15.167:0/1098790227 <== osd.1 v2:172.21.15.167:6800/30035 18 ==== osd_op_reply(47 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f296402f080 con 0x55afb287d5f0 2023-10-05T19:30:21.648 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:21.646+0000 7f296c37a700 1 -- 172.21.15.167:0/1098790227 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:48 5.4 5:20000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x55afb28ace40 con 0x55afb287d5f0 2023-10-05T19:30:21.649 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:21.646+0000 7f296c37a700 1 -- 172.21.15.167:0/1098790227 <== osd.1 v2:172.21.15.167:6800/30035 19 ==== osd_op_reply(48 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f296402f080 con 0x55afb287d5f0 2023-10-05T19:30:21.649 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:21.646+0000 7f296c37a700 1 -- 172.21.15.167:0/1098790227 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:49 5.14 5:28000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x55afb28ace40 con 0x7f2954010070 2023-10-05T19:30:21.650 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:21.646+0000 7f296c37a700 1 -- 172.21.15.167:0/1098790227 <== osd.0 v2:172.21.15.118:6800/27690 12 ==== osd_op_reply(49 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f296402f080 con 0x7f2954010070 2023-10-05T19:30:21.650 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:21.646+0000 7f296c37a700 1 -- 172.21.15.167:0/1098790227 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:50 5.c 5:30000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x55afb28ace40 con 0x7f2954010070 2023-10-05T19:30:21.651 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:21.646+0000 7f296c37a700 1 -- 172.21.15.167:0/1098790227 <== osd.0 v2:172.21.15.118:6800/27690 13 ==== osd_op_reply(50 [pgnls start_epoch 32 out=112b] v23'1 uv1 ondisk = 1) v8 ==== 144+0+112 (crc 0 0 0) 0x7f296402f080 con 0x7f2954010070 2023-10-05T19:30:21.651 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:21.646+0000 7f296c37a700 1 -- 172.21.15.167:0/1098790227 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:51 5.1c 5:38000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x55afb28ace40 con 0x7f295c0176e0 2023-10-05T19:30:21.652 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:21.650+0000 7f296b378700 1 -- 172.21.15.167:0/1098790227 <== osd.2 v2:172.21.15.167:6816/30037 8 ==== osd_op_reply(51 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f2954017050 con 0x7f295c0176e0 2023-10-05T19:30:21.652 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:21.650+0000 7f296b378700 1 -- 172.21.15.167:0/1098790227 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:52 5.2 5:40000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x55afb28ace40 con 0x55afb28ae4a0 2023-10-05T19:30:21.653 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:21.650+0000 7f296bb79700 1 -- 172.21.15.167:0/1098790227 <== osd.3 v2:172.21.15.167:6808/30036 12 ==== osd_op_reply(52 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f295c039040 con 0x55afb28ae4a0 2023-10-05T19:30:21.653 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:21.650+0000 7f296bb79700 1 -- 172.21.15.167:0/1098790227 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:53 5.12 5:48000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x55afb28ace40 con 0x55afb287d5f0 2023-10-05T19:30:21.654 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:21.650+0000 7f296c37a700 1 -- 172.21.15.167:0/1098790227 <== osd.1 v2:172.21.15.167:6800/30035 20 ==== osd_op_reply(53 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f296402f080 con 0x55afb287d5f0 2023-10-05T19:30:21.654 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:21.650+0000 7f296c37a700 1 -- 172.21.15.167:0/1098790227 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:54 5.a 5:50000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x55afb28ace40 con 0x7f2954010070 2023-10-05T19:30:21.655 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:21.650+0000 7f296c37a700 1 -- 172.21.15.167:0/1098790227 <== osd.0 v2:172.21.15.118:6800/27690 14 ==== osd_op_reply(54 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f296402f080 con 0x7f2954010070 2023-10-05T19:30:21.655 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:21.650+0000 7f296c37a700 1 -- 172.21.15.167:0/1098790227 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:55 5.1a 5:58000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x55afb28ace40 con 0x55afb287d5f0 2023-10-05T19:30:21.656 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:21.654+0000 7f296c37a700 1 -- 172.21.15.167:0/1098790227 <== osd.1 v2:172.21.15.167:6800/30035 21 ==== osd_op_reply(55 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f296402f080 con 0x55afb287d5f0 2023-10-05T19:30:21.656 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:21.654+0000 7f296c37a700 1 -- 172.21.15.167:0/1098790227 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:56 5.6 5:60000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x55afb28ace40 con 0x7f295c0176e0 2023-10-05T19:30:21.657 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:21.654+0000 7f296b378700 1 -- 172.21.15.167:0/1098790227 <== osd.2 v2:172.21.15.167:6816/30037 9 ==== osd_op_reply(56 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f2954017050 con 0x7f295c0176e0 2023-10-05T19:30:21.657 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:21.654+0000 7f296b378700 1 -- 172.21.15.167:0/1098790227 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:57 5.16 5:68000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x55afb28ace40 con 0x55afb28ae4a0 2023-10-05T19:30:21.658 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:21.654+0000 7f296bb79700 1 -- 172.21.15.167:0/1098790227 <== osd.3 v2:172.21.15.167:6808/30036 13 ==== osd_op_reply(57 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f295c039040 con 0x55afb28ae4a0 2023-10-05T19:30:21.658 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:21.654+0000 7f296bb79700 1 -- 172.21.15.167:0/1098790227 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:58 5.e 5:70000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x55afb28ace40 con 0x55afb28ae4a0 2023-10-05T19:30:21.659 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:21.654+0000 7f296bb79700 1 -- 172.21.15.167:0/1098790227 <== osd.3 v2:172.21.15.167:6808/30036 14 ==== osd_op_reply(58 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f295c039040 con 0x55afb28ae4a0 2023-10-05T19:30:21.659 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:21.654+0000 7f296bb79700 1 -- 172.21.15.167:0/1098790227 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:59 5.1e 5:78000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x55afb28ace40 con 0x7f2954010070 2023-10-05T19:30:21.660 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:21.658+0000 7f296c37a700 1 -- 172.21.15.167:0/1098790227 <== osd.0 v2:172.21.15.118:6800/27690 15 ==== osd_op_reply(59 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f296402f080 con 0x7f2954010070 2023-10-05T19:30:21.660 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:21.658+0000 7f296c37a700 1 -- 172.21.15.167:0/1098790227 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:60 5.1 5:80000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x55afb28ace40 con 0x55afb28ae4a0 2023-10-05T19:30:21.660 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:21.658+0000 7f296bb79700 1 -- 172.21.15.167:0/1098790227 <== osd.3 v2:172.21.15.167:6808/30036 15 ==== osd_op_reply(60 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f295c039040 con 0x55afb28ae4a0 2023-10-05T19:30:21.661 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:21.658+0000 7f296bb79700 1 -- 172.21.15.167:0/1098790227 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:61 5.11 5:88000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x55afb28ace40 con 0x55afb28ae4a0 2023-10-05T19:30:21.662 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:21.658+0000 7f296bb79700 1 -- 172.21.15.167:0/1098790227 <== osd.3 v2:172.21.15.167:6808/30036 16 ==== osd_op_reply(61 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f295c039040 con 0x55afb28ae4a0 2023-10-05T19:30:21.662 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:21.658+0000 7f296bb79700 1 -- 172.21.15.167:0/1098790227 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:62 5.9 5:90000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x55afb28ace40 con 0x55afb287d5f0 2023-10-05T19:30:21.662 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:21.658+0000 7f296c37a700 1 -- 172.21.15.167:0/1098790227 <== osd.1 v2:172.21.15.167:6800/30035 22 ==== osd_op_reply(62 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f296402f080 con 0x55afb287d5f0 2023-10-05T19:30:21.662 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:21.658+0000 7f296c37a700 1 -- 172.21.15.167:0/1098790227 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:63 5.19 5:98000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x55afb28ace40 con 0x55afb287d5f0 2023-10-05T19:30:21.663 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:21.658+0000 7f296c37a700 1 -- 172.21.15.167:0/1098790227 <== osd.1 v2:172.21.15.167:6800/30035 23 ==== osd_op_reply(63 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f296402f080 con 0x55afb287d5f0 2023-10-05T19:30:21.663 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:21.658+0000 7f296c37a700 1 -- 172.21.15.167:0/1098790227 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:64 5.5 5:a0000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x55afb28ace40 con 0x7f2954010070 2023-10-05T19:30:21.664 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:21.662+0000 7f296c37a700 1 -- 172.21.15.167:0/1098790227 <== osd.0 v2:172.21.15.118:6800/27690 16 ==== osd_op_reply(64 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f296402f080 con 0x7f2954010070 2023-10-05T19:30:21.664 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:21.662+0000 7f296c37a700 1 -- 172.21.15.167:0/1098790227 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:65 5.15 5:a8000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x55afb28ace40 con 0x55afb28ae4a0 2023-10-05T19:30:21.665 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:21.662+0000 7f296bb79700 1 -- 172.21.15.167:0/1098790227 <== osd.3 v2:172.21.15.167:6808/30036 17 ==== osd_op_reply(65 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f295c039040 con 0x55afb28ae4a0 2023-10-05T19:30:21.665 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:21.662+0000 7f296bb79700 1 -- 172.21.15.167:0/1098790227 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:66 5.d 5:b0000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x55afb28ace40 con 0x7f295c0176e0 2023-10-05T19:30:21.666 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:21.662+0000 7f296b378700 1 -- 172.21.15.167:0/1098790227 <== osd.2 v2:172.21.15.167:6816/30037 10 ==== osd_op_reply(66 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f2954017050 con 0x7f295c0176e0 2023-10-05T19:30:21.666 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:21.662+0000 7f296b378700 1 -- 172.21.15.167:0/1098790227 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:67 5.1d 5:b8000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x55afb28ace40 con 0x55afb287d5f0 2023-10-05T19:30:21.667 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:21.662+0000 7f296c37a700 1 -- 172.21.15.167:0/1098790227 <== osd.1 v2:172.21.15.167:6800/30035 24 ==== osd_op_reply(67 [pgnls start_epoch 32 out=107b] v30'2 uv2 ondisk = 1) v8 ==== 144+0+107 (crc 0 0 0) 0x7f296402f080 con 0x55afb287d5f0 2023-10-05T19:30:21.667 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:21.662+0000 7f296c37a700 1 -- 172.21.15.167:0/1098790227 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:68 5.3 5:c0000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x55afb28ace40 con 0x7f2954010070 2023-10-05T19:30:21.668 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:21.666+0000 7f296c37a700 1 -- 172.21.15.167:0/1098790227 <== osd.0 v2:172.21.15.118:6800/27690 17 ==== osd_op_reply(68 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f296402f080 con 0x7f2954010070 2023-10-05T19:30:21.668 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:21.666+0000 7f296c37a700 1 -- 172.21.15.167:0/1098790227 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:69 5.13 5:c8000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x55afb28ace40 con 0x55afb28ae4a0 2023-10-05T19:30:21.669 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:21.666+0000 7f296bb79700 1 -- 172.21.15.167:0/1098790227 <== osd.3 v2:172.21.15.167:6808/30036 18 ==== osd_op_reply(69 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f295c039040 con 0x55afb28ae4a0 2023-10-05T19:30:21.669 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:21.666+0000 7f296bb79700 1 -- 172.21.15.167:0/1098790227 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:70 5.b 5:d0000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x55afb28ace40 con 0x7f2954010070 2023-10-05T19:30:21.670 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:21.666+0000 7f296c37a700 1 -- 172.21.15.167:0/1098790227 <== osd.0 v2:172.21.15.118:6800/27690 18 ==== osd_op_reply(70 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f296402f080 con 0x7f2954010070 2023-10-05T19:30:21.670 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:21.666+0000 7f296c37a700 1 -- 172.21.15.167:0/1098790227 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:71 5.1b 5:d8000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x55afb28ace40 con 0x7f295c0176e0 2023-10-05T19:30:21.671 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:21.666+0000 7f296b378700 1 -- 172.21.15.167:0/1098790227 <== osd.2 v2:172.21.15.167:6816/30037 11 ==== osd_op_reply(71 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f2954017050 con 0x7f295c0176e0 2023-10-05T19:30:21.671 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:21.666+0000 7f296b378700 1 -- 172.21.15.167:0/1098790227 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:72 5.7 5:e0000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x55afb28ace40 con 0x55afb28ae4a0 2023-10-05T19:30:21.672 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:21.670+0000 7f296bb79700 1 -- 172.21.15.167:0/1098790227 <== osd.3 v2:172.21.15.167:6808/30036 19 ==== osd_op_reply(72 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f295c039040 con 0x55afb28ae4a0 2023-10-05T19:30:21.672 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:21.670+0000 7f296bb79700 1 -- 172.21.15.167:0/1098790227 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:73 5.17 5:e8000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x55afb28ace40 con 0x7f2954010070 2023-10-05T19:30:21.673 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:21.670+0000 7f296c37a700 1 -- 172.21.15.167:0/1098790227 <== osd.0 v2:172.21.15.118:6800/27690 19 ==== osd_op_reply(73 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f296402f080 con 0x7f2954010070 2023-10-05T19:30:21.673 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:21.670+0000 7f296c37a700 1 -- 172.21.15.167:0/1098790227 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:74 5.f 5:f0000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x55afb28ace40 con 0x55afb287d5f0 2023-10-05T19:30:21.674 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:21.670+0000 7f296c37a700 1 -- 172.21.15.167:0/1098790227 <== osd.1 v2:172.21.15.167:6800/30035 25 ==== osd_op_reply(74 [pgnls start_epoch 32 out=115b] v23'2 uv2 ondisk = 1) v8 ==== 144+0+115 (crc 0 0 0) 0x7f296402f080 con 0x55afb287d5f0 2023-10-05T19:30:21.674 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:21.670+0000 7f296c37a700 1 -- 172.21.15.167:0/1098790227 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:75 5.1f 5:f8000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x55afb28ace40 con 0x7f295c0176e0 2023-10-05T19:30:21.675 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:21.670+0000 7f296b378700 1 -- 172.21.15.167:0/1098790227 <== osd.2 v2:172.21.15.167:6816/30037 12 ==== osd_op_reply(75 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f2954017050 con 0x7f295c0176e0 2023-10-05T19:30:21.675 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:21.670+0000 7f296e044e40 20 RGWRados::pool_iterate: got zonegroup_info.4f1f719d-9648-417b-b10c-e5d939825419 2023-10-05T19:30:21.675 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:21.670+0000 7f296e044e40 20 RGWRados::pool_iterate: got zone_info.9a9df965-6612-45d4-a45f-6e3c889bacc2 2023-10-05T19:30:21.675 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:21.670+0000 7f296e044e40 20 RGWRados::pool_iterate: got zone_names.default 2023-10-05T19:30:21.675 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:21.670+0000 7f296e044e40 20 RGWRados::pool_iterate: got zonegroups_names.default 2023-10-05T19:30:21.675 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:21.670+0000 7f296e044e40 20 rados->read ofs=0 len=0 2023-10-05T19:30:21.675 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:21.670+0000 7f296e044e40 1 -- 172.21.15.167:0/1098790227 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:76 5.f 5:f43b6ece:::zone_names.default:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32) v8 -- 0x55afb28b1a60 con 0x55afb287d5f0 2023-10-05T19:30:21.676 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:21.674+0000 7f296c37a700 1 -- 172.21.15.167:0/1098790227 <== osd.1 v2:172.21.15.167:6800/30035 26 ==== osd_op_reply(76 zone_names.default [read 0~46 out=46b] v0'0 uv1 ondisk = 0) v8 ==== 162+0+46 (crc 0 0 0) 0x7f296402f080 con 0x55afb287d5f0 2023-10-05T19:30:21.676 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:21.674+0000 7f296e044e40 20 rados_obj.operate() r=0 bl.length=46 2023-10-05T19:30:21.676 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:21.674+0000 7f296e044e40 20 rados->read ofs=0 len=0 2023-10-05T19:30:21.677 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:21.674+0000 7f296e044e40 1 -- 172.21.15.167:0/1098790227 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:77 5.1d 5:b8374eb4:::zone_info.9a9df965-6612-45d4-a45f-6e3c889bacc2:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32) v8 -- 0x55afb28b1e10 con 0x55afb287d5f0 2023-10-05T19:30:21.677 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:21.674+0000 7f296c37a700 1 -- 172.21.15.167:0/1098790227 <== osd.1 v2:172.21.15.167:6800/30035 27 ==== osd_op_reply(77 zone_info.9a9df965-6612-45d4-a45f-6e3c889bacc2 [read 0~874 out=874b] v0'0 uv2 ondisk = 0) v8 ==== 190+0+874 (crc 0 0 0) 0x7f296402f080 con 0x55afb287d5f0 2023-10-05T19:30:21.677 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:21.674+0000 7f296e044e40 20 rados_obj.operate() r=0 bl.length=874 2023-10-05T19:30:21.677 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:21.674+0000 7f296e044e40 20 rados->read ofs=0 len=0 2023-10-05T19:30:21.677 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:21.674+0000 7f296e044e40 1 -- 172.21.15.167:0/1098790227 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:78 5.f 5:f4c53578:::zonegroups_names.default:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32) v8 -- 0x55afb28b1250 con 0x55afb287d5f0 2023-10-05T19:30:21.677 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:21.674+0000 7f296c37a700 1 -- 172.21.15.167:0/1098790227 <== osd.1 v2:172.21.15.167:6800/30035 28 ==== osd_op_reply(78 zonegroups_names.default [read 0~46 out=46b] v0'0 uv2 ondisk = 0) v8 ==== 168+0+46 (crc 0 0 0) 0x7f296402f080 con 0x55afb287d5f0 2023-10-05T19:30:21.677 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:21.674+0000 7f296e044e40 20 rados_obj.operate() r=0 bl.length=46 2023-10-05T19:30:21.678 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:21.674+0000 7f296e044e40 20 rados->read ofs=0 len=0 2023-10-05T19:30:21.678 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:21.674+0000 7f296e044e40 1 -- 172.21.15.167:0/1098790227 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:79 5.c 5:30d299a5:::zonegroup_info.4f1f719d-9648-417b-b10c-e5d939825419:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32) v8 -- 0x55afb28b2300 con 0x7f2954010070 2023-10-05T19:30:21.678 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:21.674+0000 7f296c37a700 1 -- 172.21.15.167:0/1098790227 <== osd.0 v2:172.21.15.118:6800/27690 20 ==== osd_op_reply(79 zonegroup_info.4f1f719d-9648-417b-b10c-e5d939825419 [read 0~362 out=362b] v0'0 uv1 ondisk = 0) v8 ==== 195+0+362 (crc 0 0 0) 0x7f296402f080 con 0x7f2954010070 2023-10-05T19:30:21.678 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:21.674+0000 7f296e044e40 20 rados_obj.operate() r=0 bl.length=362 2023-10-05T19:30:21.678 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:21.674+0000 7f296e044e40 20 rados->read ofs=0 len=0 2023-10-05T19:30:21.678 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:21.674+0000 7f296e044e40 1 -- 172.21.15.167:0/1098790227 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:80 5.3 5:c0ad8cbd:::region_map:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32) v8 -- 0x55afb28b2300 con 0x7f2954010070 2023-10-05T19:30:21.679 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:21.674+0000 7f296c37a700 1 -- 172.21.15.167:0/1098790227 <== osd.0 v2:172.21.15.118:6800/27690 21 ==== osd_op_reply(80 region_map [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 ==== 154+0+0 (crc 0 0 0) 0x7f2964050040 con 0x7f2954010070 2023-10-05T19:30:21.679 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:21.674+0000 7f296e044e40 20 rados_obj.operate() r=-2 bl.length=0 2023-10-05T19:30:21.679 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:21.674+0000 7f296e044e40 20 zone default found 2023-10-05T19:30:21.680 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:21.674+0000 7f296e044e40 4 Realm: () 2023-10-05T19:30:21.680 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:21.674+0000 7f296e044e40 4 ZoneGroup: default (4f1f719d-9648-417b-b10c-e5d939825419) 2023-10-05T19:30:21.680 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:21.674+0000 7f296e044e40 4 Zone: default (9a9df965-6612-45d4-a45f-6e3c889bacc2) 2023-10-05T19:30:21.680 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:21.674+0000 7f296e044e40 10 cannot find current period zonegroup using local zonegroup configuration 2023-10-05T19:30:21.680 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:21.674+0000 7f296e044e40 20 zonegroup default 2023-10-05T19:30:21.680 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:21.674+0000 7f296e044e40 20 rados->read ofs=0 len=0 2023-10-05T19:30:21.680 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:21.674+0000 7f296e044e40 1 -- 172.21.15.167:0/1098790227 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:81 5.3 5:c52100b6:::period_config.default:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32) v8 -- 0x55afb28b2300 con 0x7f2954010070 2023-10-05T19:30:21.681 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:21.678+0000 7f296c37a700 1 -- 172.21.15.167:0/1098790227 <== osd.0 v2:172.21.15.118:6800/27690 22 ==== osd_op_reply(81 period_config.default [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 ==== 165+0+0 (crc 0 0 0) 0x7f2964050040 con 0x7f2954010070 2023-10-05T19:30:21.681 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:21.678+0000 7f296e044e40 20 rados_obj.operate() r=-2 bl.length=0 2023-10-05T19:30:21.681 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:21.678+0000 7f296e044e40 20 rados->read ofs=0 len=0 2023-10-05T19:30:21.681 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:21.678+0000 7f296e044e40 1 -- 172.21.15.167:0/1098790227 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:82 6.2 6:4347d321:::bucket.sync-source-hints.:head [call version.read in=11b,read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32) v8 -- 0x55afb28b3c80 con 0x55afb28ae4a0 2023-10-05T19:30:21.682 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:21.678+0000 7f296bb79700 1 -- 172.21.15.167:0/1098790227 <== osd.3 v2:172.21.15.167:6808/30036 20 ==== osd_op_reply(82 bucket.sync-source-hints. [call,read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 ==== 211+0+0 (crc 0 0 0) 0x7f295c039040 con 0x55afb28ae4a0 2023-10-05T19:30:21.682 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:21.678+0000 7f296e044e40 20 rados_obj.operate() r=-2 bl.length=0 2023-10-05T19:30:21.682 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:21.678+0000 7f296e044e40 20 rados->read ofs=0 len=0 2023-10-05T19:30:21.682 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:21.678+0000 7f296e044e40 1 -- 172.21.15.167:0/1098790227 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:83 6.b 6:d467b91b:::bucket.sync-target-hints.:head [call version.read in=11b,read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32) v8 -- 0x55afb28b4030 con 0x55afb28ae4a0 2023-10-05T19:30:21.683 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:21.678+0000 7f296bb79700 1 -- 172.21.15.167:0/1098790227 <== osd.3 v2:172.21.15.167:6808/30036 21 ==== osd_op_reply(83 bucket.sync-target-hints. [call,read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 ==== 211+0+0 (crc 0 0 0) 0x7f295c039040 con 0x55afb28ae4a0 2023-10-05T19:30:21.683 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:21.678+0000 7f296e044e40 20 rados_obj.operate() r=-2 bl.length=0 2023-10-05T19:30:21.683 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:21.678+0000 7f296e044e40 20 started sync module instance, tier type = 2023-10-05T19:30:21.684 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:21.678+0000 7f296e044e40 20 started zone id=9a9df965-6612-45d4-a45f-6e3c889bacc2 (name=default) with tier type = 2023-10-05T19:30:21.684 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:21.678+0000 7f296e044e40 1 -- 172.21.15.167:0/1098790227 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:84 7.1f 7:f95f44c2:::notify.0:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e32) v8 -- 0x55afb28b3340 con 0x7f295c0176e0 2023-10-05T19:30:21.685 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:21.682+0000 7f296b378700 1 -- 172.21.15.167:0/1098790227 <== osd.2 v2:172.21.15.167:6816/30037 13 ==== osd_op_reply(84 notify.0 [create] v32'261 uv261 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f2954017050 con 0x7f295c0176e0 2023-10-05T19:30:21.685 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:21.682+0000 7f296e044e40 1 -- 172.21.15.167:0/1098790227 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:85 7.1f 7:f95f44c2:::notify.0:head [watch watch cookie 94213103103568] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e32) v8 -- 0x55afb28b43e0 con 0x7f295c0176e0 2023-10-05T19:30:21.685 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:21.682+0000 7f296e044e40 1 -- 172.21.15.167:0/1098790227 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:86 7.0 7:05bf5b68:::notify.1:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e32) v8 -- 0x55afb28b4ce0 con 0x55afb287d5f0 2023-10-05T19:30:21.687 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:21.682+0000 7f296b378700 1 -- 172.21.15.167:0/1098790227 <== osd.2 v2:172.21.15.167:6816/30037 14 ==== osd_op_reply(85 notify.0 [watch watch cookie 94213103103568] v32'262 uv261 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f2954017050 con 0x7f295c0176e0 2023-10-05T19:30:21.688 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:21.686+0000 7f296c37a700 1 -- 172.21.15.167:0/1098790227 <== osd.1 v2:172.21.15.167:6800/30035 29 ==== osd_op_reply(86 notify.1 [create] v32'261 uv261 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f2964050040 con 0x55afb287d5f0 2023-10-05T19:30:21.688 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:21.686+0000 7f296e044e40 1 -- 172.21.15.167:0/1098790227 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:87 7.0 7:05bf5b68:::notify.1:head [watch watch cookie 94213103110128] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e32) v8 -- 0x55afb28b4c40 con 0x55afb287d5f0 2023-10-05T19:30:21.689 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:21.686+0000 7f296e044e40 1 -- 172.21.15.167:0/1098790227 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:88 7.15 7:a93a5511:::notify.2:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e32) v8 -- 0x55afb28b5ac0 con 0x55afb28ae4a0 2023-10-05T19:30:21.691 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:21.686+0000 7f296c37a700 1 -- 172.21.15.167:0/1098790227 <== osd.1 v2:172.21.15.167:6800/30035 30 ==== osd_op_reply(87 notify.1 [watch watch cookie 94213103110128] v32'262 uv261 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f2964050040 con 0x55afb287d5f0 2023-10-05T19:30:21.692 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:21.690+0000 7f296bb79700 1 -- 172.21.15.167:0/1098790227 <== osd.3 v2:172.21.15.167:6808/30036 22 ==== osd_op_reply(88 notify.2 [create] v32'261 uv261 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f295c039040 con 0x55afb28ae4a0 2023-10-05T19:30:21.692 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:21.690+0000 7f296e044e40 1 -- 172.21.15.167:0/1098790227 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:89 7.15 7:a93a5511:::notify.2:head [watch watch cookie 94213103112416] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e32) v8 -- 0x55afb28b4c40 con 0x55afb28ae4a0 2023-10-05T19:30:21.692 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:21.690+0000 7f296e044e40 1 -- 172.21.15.167:0/1098790227 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:90 7.e 7:7759931f:::notify.3:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e32) v8 -- 0x55afb28b65d0 con 0x7f295c0176e0 2023-10-05T19:30:21.694 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:21.690+0000 7f296b378700 1 -- 172.21.15.167:0/1098790227 <== osd.2 v2:172.21.15.167:6816/30037 15 ==== osd_op_reply(90 notify.3 [create] v32'261 uv261 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f2954017050 con 0x7f295c0176e0 2023-10-05T19:30:21.695 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:21.690+0000 7f296e044e40 1 -- 172.21.15.167:0/1098790227 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:91 7.e 7:7759931f:::notify.3:head [watch watch cookie 94213103118800] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e32) v8 -- 0x55afb28b7310 con 0x7f295c0176e0 2023-10-05T19:30:21.695 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:21.690+0000 7f296e044e40 1 -- 172.21.15.167:0/1098790227 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:92 7.d 7:b4812045:::notify.4:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e32) v8 -- 0x55afb28b7bf0 con 0x55afb287d5f0 2023-10-05T19:30:21.695 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:21.690+0000 7f296bb79700 1 -- 172.21.15.167:0/1098790227 <== osd.3 v2:172.21.15.167:6808/30036 23 ==== osd_op_reply(89 notify.2 [watch watch cookie 94213103112416] v32'262 uv261 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f295c039040 con 0x55afb28ae4a0 2023-10-05T19:30:21.696 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:21.694+0000 7f296b378700 1 -- 172.21.15.167:0/1098790227 <== osd.2 v2:172.21.15.167:6816/30037 16 ==== osd_op_reply(91 notify.3 [watch watch cookie 94213103118800] v32'262 uv261 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f2954017050 con 0x7f295c0176e0 2023-10-05T19:30:21.698 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:21.694+0000 7f296c37a700 1 -- 172.21.15.167:0/1098790227 <== osd.1 v2:172.21.15.167:6800/30035 31 ==== osd_op_reply(92 notify.4 [create] v32'261 uv261 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f2964050040 con 0x55afb287d5f0 2023-10-05T19:30:21.698 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:21.694+0000 7f296e044e40 1 -- 172.21.15.167:0/1098790227 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:93 7.d 7:b4812045:::notify.4:head [watch watch cookie 94213103119904] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e32) v8 -- 0x55afb28b76c0 con 0x55afb287d5f0 2023-10-05T19:30:21.698 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:21.694+0000 7f296e044e40 1 -- 172.21.15.167:0/1098790227 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:94 7.3 7:c609908c:::notify.5:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e32) v8 -- 0x55afb28b84e0 con 0x7f2954010070 2023-10-05T19:30:21.701 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:21.698+0000 7f296c37a700 1 -- 172.21.15.167:0/1098790227 <== osd.0 v2:172.21.15.118:6800/27690 23 ==== osd_op_reply(94 notify.5 [create] v32'261 uv261 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f2964050040 con 0x7f2954010070 2023-10-05T19:30:21.701 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:21.698+0000 7f296e044e40 1 -- 172.21.15.167:0/1098790227 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:95 7.3 7:c609908c:::notify.5:head [watch watch cookie 94213103124384] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e32) v8 -- 0x55afb28b88e0 con 0x7f2954010070 2023-10-05T19:30:21.701 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:21.698+0000 7f296e044e40 1 -- 172.21.15.167:0/1098790227 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:96 7.14 7:2b04a3e9:::notify.6:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e32) v8 -- 0x55afb28b92f0 con 0x7f2954010070 2023-10-05T19:30:21.701 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:21.698+0000 7f296c37a700 1 -- 172.21.15.167:0/1098790227 <== osd.1 v2:172.21.15.167:6800/30035 32 ==== osd_op_reply(93 notify.4 [watch watch cookie 94213103119904] v32'262 uv261 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f2964050040 con 0x55afb287d5f0 2023-10-05T19:30:21.703 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:21.698+0000 7f296c37a700 1 -- 172.21.15.167:0/1098790227 <== osd.0 v2:172.21.15.118:6800/27690 24 ==== osd_op_reply(95 notify.5 [watch watch cookie 94213103124384] v32'262 uv261 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f2964050040 con 0x7f2954010070 2023-10-05T19:30:21.704 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:21.702+0000 7f296c37a700 1 -- 172.21.15.167:0/1098790227 <== osd.0 v2:172.21.15.118:6800/27690 25 ==== osd_op_reply(96 notify.6 [create] v32'261 uv261 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f2964050040 con 0x7f2954010070 2023-10-05T19:30:21.704 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:21.702+0000 7f296e044e40 1 -- 172.21.15.167:0/1098790227 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:97 7.14 7:2b04a3e9:::notify.6:head [watch watch cookie 94213103125488] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e32) v8 -- 0x55afb28b8840 con 0x7f2954010070 2023-10-05T19:30:21.704 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:21.702+0000 7f296e044e40 1 -- 172.21.15.167:0/1098790227 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:98 7.9 7:93e5b521:::notify.7:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e32) v8 -- 0x55afb28b97e0 con 0x7f2954010070 2023-10-05T19:30:21.707 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:21.702+0000 7f296c37a700 1 -- 172.21.15.167:0/1098790227 <== osd.0 v2:172.21.15.118:6800/27690 26 ==== osd_op_reply(98 notify.7 [create] v32'261 uv261 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f2964050040 con 0x7f2954010070 2023-10-05T19:30:21.707 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:21.702+0000 7f296e044e40 1 -- 172.21.15.167:0/1098790227 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:99 7.9 7:93e5b521:::notify.7:head [watch watch cookie 94213103129328] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e32) v8 -- 0x55afb28b9c30 con 0x7f2954010070 2023-10-05T19:30:21.707 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:21.702+0000 7f296e044e40 20 add_watcher() i=0 2023-10-05T19:30:21.707 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:21.702+0000 7f296e044e40 20 add_watcher() i=1 2023-10-05T19:30:21.707 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:21.702+0000 7f296e044e40 20 add_watcher() i=2 2023-10-05T19:30:21.707 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:21.702+0000 7f296e044e40 20 add_watcher() i=3 2023-10-05T19:30:21.707 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:21.702+0000 7f296e044e40 20 add_watcher() i=4 2023-10-05T19:30:21.708 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:21.702+0000 7f296e044e40 20 add_watcher() i=5 2023-10-05T19:30:21.708 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:21.702+0000 7f296c37a700 1 -- 172.21.15.167:0/1098790227 <== osd.0 v2:172.21.15.118:6800/27690 27 ==== osd_op_reply(97 notify.6 [watch watch cookie 94213103125488] v32'262 uv261 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f2964050040 con 0x7f2954010070 2023-10-05T19:30:21.708 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:21.702+0000 7f296e044e40 20 add_watcher() i=6 2023-10-05T19:30:21.709 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:21.706+0000 7f296c37a700 1 -- 172.21.15.167:0/1098790227 <== osd.0 v2:172.21.15.118:6800/27690 28 ==== osd_op_reply(99 notify.7 [watch watch cookie 94213103129328] v32'262 uv261 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f2964050040 con 0x7f2954010070 2023-10-05T19:30:21.709 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:21.706+0000 7f296e044e40 20 add_watcher() i=7 2023-10-05T19:30:21.709 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:21.706+0000 7f296e044e40 2 all 8 watchers are set, enabling cache 2023-10-05T19:30:21.709 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:21.706+0000 7f296e044e40 1 -- 172.21.15.167:0/1098790227 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:100 6.4 6:22d26bf9:::data_loggenerations_metadata:head [call version.check_conds in=74b,call version.read in=11b,read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32) v8 -- 0x55afb28bcc60 con 0x55afb287d5f0 2023-10-05T19:30:21.710 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:21.706+0000 7f296c37a700 1 -- 172.21.15.167:0/1098790227 <== osd.1 v2:172.21.15.167:6800/30035 33 ==== osd_op_reply(100 data_loggenerations_metadata [call,call out=48b,read 0~28 out=28b] v0'0 uv1 ondisk = 0) v8 ==== 256+0+76 (crc 0 0 0) 0x7f2964050040 con 0x55afb287d5f0 2023-10-05T19:30:21.710 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:21.706+0000 7f296e044e40 1 -- 172.21.15.167:0/1098790227 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:101 6.4 6:22d26bf9:::data_loggenerations_metadata:head [watch watch cookie 94213103130432] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e32) v8 -- 0x55afb28ba080 con 0x55afb287d5f0 2023-10-05T19:30:21.712 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:21.710+0000 7f296c37a700 1 -- 172.21.15.167:0/1098790227 <== osd.1 v2:172.21.15.167:6800/30035 34 ==== osd_op_reply(101 data_loggenerations_metadata [watch watch cookie 94213103130432] v32'253 uv1 ondisk = 0) v8 ==== 172+0+0 (crc 0 0 0) 0x7f2964050040 con 0x55afb287d5f0 2023-10-05T19:30:21.712 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:21.710+0000 7f293d7e2700 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2023-10-05T19:30:21.712 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:21.710+0000 7f296e044e40 20 check_secure_mon_conn(): auth registy supported: methods=[2] modes=[2,1] 2023-10-05T19:30:21.712 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:21.710+0000 7f296e044e40 20 check_secure_mon_conn(): mode 1 is insecure 2023-10-05T19:30:21.713 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:21.710+0000 7f296e044e40 1 Processor -- start 2023-10-05T19:30:21.713 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:21.710+0000 7f296e044e40 1 -- start start 2023-10-05T19:30:21.713 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:21.710+0000 7f296e044e40 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55afb28c6e20 0x55afb28c71f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-05T19:30:21.713 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:21.710+0000 7f296e044e40 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x55afb28c7730 con 0x55afb28c6e20 2023-10-05T19:30:21.714 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:21.710+0000 7f296b378700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55afb28c6e20 0x55afb28c71f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-05T19:30:21.714 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:21.710+0000 7f296b378700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55afb28c6e20 0x55afb28c71f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.118:3300/0 says I am v2:172.21.15.167:32956/0 (socket says 172.21.15.167:32956) 2023-10-05T19:30:21.714 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:21.710+0000 7f296b378700 1 -- 172.21.15.167:0/2641990574 learned_addr learned my addr 172.21.15.167:0/2641990574 (peer_addr_for_me v2:172.21.15.167:0/0) 2023-10-05T19:30:21.714 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:21.710+0000 7f296b378700 1 -- 172.21.15.167:0/2641990574 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55afb28c7870 con 0x55afb28c6e20 2023-10-05T19:30:21.714 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:21.710+0000 7f296b378700 1 --2- 172.21.15.167:0/2641990574 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55afb28c6e20 0x55afb28c71f0 secure :-1 s=READY pgs=660 cs=0 l=1 rev1=1 crypto rx=0x7f2954003000 tx=0x7f2954010000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6a65889c36430679 server_cookie=0 in_seq=0 out_seq=0 2023-10-05T19:30:21.715 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:21.710+0000 7f293cfe1700 1 -- 172.21.15.167:0/2641990574 <== mon.0 v2:172.21.15.118:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f295401c9f0 con 0x55afb28c6e20 2023-10-05T19:30:21.715 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:21.710+0000 7f293cfe1700 1 -- 172.21.15.167:0/2641990574 <== mon.0 v2:172.21.15.118:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f295401cb70 con 0x55afb28c6e20 2023-10-05T19:30:21.715 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:21.710+0000 7f293cfe1700 1 -- 172.21.15.167:0/2641990574 <== mon.0 v2:172.21.15.118:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f295401cd60 con 0x55afb28c6e20 2023-10-05T19:30:21.716 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:21.714+0000 7f296e044e40 1 -- 172.21.15.167:0/2641990574 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55afb28c6e20 msgr2=0x55afb28c71f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-05T19:30:21.716 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:21.714+0000 7f296e044e40 1 --2- 172.21.15.167:0/2641990574 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55afb28c6e20 0x55afb28c71f0 secure :-1 s=READY pgs=660 cs=0 l=1 rev1=1 crypto rx=0x7f2954003000 tx=0x7f2954010000 comp rx=0 tx=0).stop 2023-10-05T19:30:21.716 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:21.714+0000 7f296e044e40 1 -- 172.21.15.167:0/2641990574 shutdown_connections 2023-10-05T19:30:21.717 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:21.714+0000 7f296e044e40 1 --2- 172.21.15.167:0/2641990574 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55afb28c6e20 0x55afb28c71f0 unknown :-1 s=CLOSED pgs=660 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-05T19:30:21.717 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:21.714+0000 7f296e044e40 1 -- 172.21.15.167:0/2641990574 >> 172.21.15.167:0/2641990574 conn(0x55afb28c21e0 msgr2=0x55afb28c0390 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-05T19:30:21.717 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:21.714+0000 7f296e044e40 1 -- 172.21.15.167:0/2641990574 shutdown_connections 2023-10-05T19:30:21.717 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:21.714+0000 7f296e044e40 1 -- 172.21.15.167:0/2641990574 wait complete. 2023-10-05T19:30:21.717 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:21.714+0000 7f296e044e40 1 Processor -- start 2023-10-05T19:30:21.718 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:21.714+0000 7f296e044e40 1 -- start start 2023-10-05T19:30:21.718 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:21.714+0000 7f296e044e40 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55afb28c6e20 0x55afb28cc8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-05T19:30:21.718 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:21.714+0000 7f296e044e40 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x55afb28cd8f0 con 0x55afb28c6e20 2023-10-05T19:30:21.718 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:21.714+0000 7f296b378700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55afb28c6e20 0x55afb28cc8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-05T19:30:21.719 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:21.714+0000 7f296b378700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55afb28c6e20 0x55afb28cc8f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.118:3300/0 says I am v2:172.21.15.167:32966/0 (socket says 172.21.15.167:32966) 2023-10-05T19:30:21.719 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:21.714+0000 7f296b378700 1 -- 172.21.15.167:0/2989941766 learned_addr learned my addr 172.21.15.167:0/2989941766 (peer_addr_for_me v2:172.21.15.167:0/0) 2023-10-05T19:30:21.719 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:21.714+0000 7f296b378700 1 -- 172.21.15.167:0/2989941766 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55afb2879c20 con 0x55afb28c6e20 2023-10-05T19:30:21.719 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:21.714+0000 7f296b378700 1 --2- 172.21.15.167:0/2989941766 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55afb28c6e20 0x55afb28cc8f0 secure :-1 s=READY pgs=661 cs=0 l=1 rev1=1 crypto rx=0x55afb28c0f00 tx=0x7f295402f000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-05T19:30:21.720 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:21.714+0000 7f28b6ffd700 1 -- 172.21.15.167:0/2989941766 <== mon.0 v2:172.21.15.118:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f2954035400 con 0x55afb28c6e20 2023-10-05T19:30:21.720 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:21.714+0000 7f28b6ffd700 1 -- 172.21.15.167:0/2989941766 <== mon.0 v2:172.21.15.118:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2954035580 con 0x55afb28c6e20 2023-10-05T19:30:21.720 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:21.714+0000 7f296e044e40 1 -- 172.21.15.167:0/2989941766 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x55afb28ccef0 con 0x55afb28c6e20 2023-10-05T19:30:21.720 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:21.714+0000 7f28b6ffd700 1 -- 172.21.15.167:0/2989941766 <== mon.0 v2:172.21.15.118:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f2954035770 con 0x55afb28c6e20 2023-10-05T19:30:21.720 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:21.714+0000 7f296e044e40 1 -- 172.21.15.167:0/2989941766 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55afb28cd330 con 0x55afb28c6e20 2023-10-05T19:30:21.721 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:21.718+0000 7f28b6ffd700 1 -- 172.21.15.167:0/2989941766 <== mon.0 v2:172.21.15.118:3300/0 4 ==== mgrmap(e 5) v1 ==== 40329+0+0 (secure 0 0 0) 0x7f29540358d0 con 0x55afb28c6e20 2023-10-05T19:30:21.721 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:21.718+0000 7f28b6ffd700 1 --2- 172.21.15.167:0/2989941766 >> [v2:172.21.15.167:6824/29889,v1:172.21.15.167:6825/29889] conn(0x7f28ac030c50 0x7f28ac033100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-05T19:30:21.721 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:21.718+0000 7f28b6ffd700 1 -- 172.21.15.167:0/2989941766 <== mon.0 v2:172.21.15.118:3300/0 5 ==== osd_map(32..32 src has 1..32) v4 ==== 6260+0+0 (secure 0 0 0) 0x7f295402f070 con 0x55afb28c6e20 2023-10-05T19:30:21.721 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:21.718+0000 7f296c37a700 1 --2- 172.21.15.167:0/2989941766 >> [v2:172.21.15.167:6824/29889,v1:172.21.15.167:6825/29889] conn(0x7f28ac030c50 0x7f28ac033100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-05T19:30:21.721 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:21.718+0000 7f296e044e40 5 note: GC not initialized 2023-10-05T19:30:21.722 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:21.718+0000 7f296c37a700 1 --2- 172.21.15.167:0/2989941766 >> [v2:172.21.15.167:6824/29889,v1:172.21.15.167:6825/29889] conn(0x7f28ac030c50 0x7f28ac033100 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x55afb28b9070 tx=0x7f2964000f90 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-05T19:30:21.723 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:21.718+0000 7f28abfff700 20 reqs_thread_entry: start 2023-10-05T19:30:21.724 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:21.722+0000 7f28a9ffb700 20 reqs_thread_entry: start 2023-10-05T19:30:21.724 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:21.722+0000 7f296e044e40 20 init_complete bucket index max shards: 11 2023-10-05T19:30:21.725 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:21.722+0000 7f296e044e40 10 rgw notify: Started notification manager with: 1 workers 2023-10-05T19:30:21.725 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:21.722+0000 7f28977fe700 20 reqs_thread_entry: start 2023-10-05T19:30:21.725 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:21.722+0000 7f296e044e40 20 get_system_obj_state: rctx=0x7ffdada8d838 obj=default.rgw.meta:users.uid:foo state=0x55afb27500b0 s->prefetch_data=0 2023-10-05T19:30:21.725 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:21.722+0000 7f296e044e40 1 -- 172.21.15.167:0/1098790227 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:102 8.18 8:1e6ca5ed:users.uid::foo:head [call version.read in=11b,getxattrs,stat] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32) v8 -- 0x55afb29350c0 con 0x55afb287d5f0 2023-10-05T19:30:21.725 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:21.722+0000 7f2897fff700 20 rgw notify: INFO: next queues processing will happen at: Thu Oct 5 19:30:52 2023 2023-10-05T19:30:21.725 INFO:teuthology.orchestra.run.smithi167.stderr: 2023-10-05T19:30:21.726 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:21.722+0000 7f296c37a700 1 -- 172.21.15.167:0/1098790227 <== osd.1 v2:172.21.15.167:6800/30035 35 ==== osd_op_reply(102 foo [call out=48b,getxattrs out=75b,stat out=16b] v0'0 uv15 ondisk = 0) v8 ==== 231+0+139 (crc 0 0 0) 0x7f2964050040 con 0x55afb287d5f0 2023-10-05T19:30:21.726 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:21.722+0000 7f296e044e40 20 get_system_obj_state: s->obj_tag was set empty 2023-10-05T19:30:21.726 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:21.722+0000 7f296e044e40 20 Read xattr: user.rgw.idtag 2023-10-05T19:30:21.726 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:21.722+0000 7f296e044e40 20 rados->read ofs=0 len=0 2023-10-05T19:30:21.727 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:21.722+0000 7f296e044e40 1 -- 172.21.15.167:0/1098790227 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:103 8.18 8:1e6ca5ed:users.uid::foo:head [call version.check_conds in=122b,call version.read in=11b,read 0~0,getxattrs] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32) v8 -- 0x55afb2937b50 con 0x55afb287d5f0 2023-10-05T19:30:21.727 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:21.722+0000 7f296c37a700 1 -- 172.21.15.167:0/1098790227 <== osd.1 v2:172.21.15.167:6800/30035 36 ==== osd_op_reply(103 foo [call,call out=48b,read 0~357 out=357b,getxattrs out=75b] v0'0 uv15 ondisk = 0) v8 ==== 273+0+480 (crc 0 0 0) 0x7f2964050040 con 0x55afb287d5f0 2023-10-05T19:30:21.727 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:21.722+0000 7f296e044e40 20 rados_obj.operate() r=0 bl.length=357 2023-10-05T19:30:21.727 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:21.722+0000 7f296e044e40 1 -- 172.21.15.167:0/1098790227 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:104 6.13 6:cea599e4:usage::usage.10:head [call rgw.user_usage_log_read in=63b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32) v8 -- 0x55afb2936b40 con 0x7f2954010070 2023-10-05T19:30:21.728 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:21.726+0000 7f296c37a700 1 -- 172.21.15.167:0/1098790227 <== osd.0 v2:172.21.15.118:6800/27690 29 ==== osd_op_reply(104 usage.10 [call out=15b] v0'0 uv33 ondisk = 0) v8 ==== 152+0+15 (crc 0 0 0) 0x7f2964050040 con 0x7f2954010070 2023-10-05T19:30:21.728 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:21.726+0000 7f296e044e40 1 -- 172.21.15.167:0/1098790227 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:105 7.1f 7:f95f44c2:::notify.0:head [watch unwatch cookie 94213103103568] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e32) v8 -- 0x55afb28f34b0 con 0x7f295c0176e0 2023-10-05T19:30:21.731 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:21.726+0000 7f296b378700 1 -- 172.21.15.167:0/1098790227 <== osd.2 v2:172.21.15.167:6816/30037 17 ==== osd_op_reply(105 notify.0 [watch unwatch cookie 94213103103568] v32'263 uv261 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f2954032040 con 0x7f295c0176e0 2023-10-05T19:30:21.731 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:21.726+0000 7f296e044e40 20 remove_watcher() i=0 2023-10-05T19:30:21.732 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:21.726+0000 7f296e044e40 2 removed watcher, disabling cache 2023-10-05T19:30:21.732 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:21.726+0000 7f296e044e40 1 -- 172.21.15.167:0/1098790227 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:106 7.0 7:05bf5b68:::notify.1:head [watch unwatch cookie 94213103110128] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e32) v8 -- 0x55afb28b2e50 con 0x55afb287d5f0 2023-10-05T19:30:21.734 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:21.730+0000 7f296c37a700 1 -- 172.21.15.167:0/1098790227 <== osd.1 v2:172.21.15.167:6800/30035 37 ==== osd_op_reply(106 notify.1 [watch unwatch cookie 94213103110128] v32'263 uv261 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f2964050040 con 0x55afb287d5f0 2023-10-05T19:30:21.734 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:21.730+0000 7f296e044e40 20 remove_watcher() i=1 2023-10-05T19:30:21.734 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:21.730+0000 7f296e044e40 1 -- 172.21.15.167:0/1098790227 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:107 7.15 7:a93a5511:::notify.2:head [watch unwatch cookie 94213103112416] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e32) v8 -- 0x55afb28b2e50 con 0x55afb28ae4a0 2023-10-05T19:30:21.738 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:21.734+0000 7f296bb79700 1 -- 172.21.15.167:0/1098790227 <== osd.3 v2:172.21.15.167:6808/30036 24 ==== osd_op_reply(107 notify.2 [watch unwatch cookie 94213103112416] v32'263 uv261 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f295c039040 con 0x55afb28ae4a0 2023-10-05T19:30:21.738 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:21.734+0000 7f296e044e40 20 remove_watcher() i=2 2023-10-05T19:30:21.738 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:21.734+0000 7f296e044e40 1 -- 172.21.15.167:0/1098790227 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:108 7.e 7:7759931f:::notify.3:head [watch unwatch cookie 94213103118800] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e32) v8 -- 0x55afb28b50e0 con 0x7f295c0176e0 2023-10-05T19:30:21.740 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:21.738+0000 7f296b378700 1 -- 172.21.15.167:0/1098790227 <== osd.2 v2:172.21.15.167:6816/30037 18 ==== osd_op_reply(108 notify.3 [watch unwatch cookie 94213103118800] v32'263 uv261 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f2954032040 con 0x7f295c0176e0 2023-10-05T19:30:21.740 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:21.738+0000 7f296e044e40 20 remove_watcher() i=3 2023-10-05T19:30:21.740 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:21.738+0000 7f296e044e40 1 -- 172.21.15.167:0/1098790227 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:109 7.d 7:b4812045:::notify.4:head [watch unwatch cookie 94213103119904] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e32) v8 -- 0x55afb28b2e50 con 0x55afb287d5f0 2023-10-05T19:30:21.742 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:21.738+0000 7f296c37a700 1 -- 172.21.15.167:0/1098790227 <== osd.1 v2:172.21.15.167:6800/30035 38 ==== osd_op_reply(109 notify.4 [watch unwatch cookie 94213103119904] v32'263 uv261 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f2964050040 con 0x55afb287d5f0 2023-10-05T19:30:21.743 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:21.738+0000 7f296e044e40 20 remove_watcher() i=4 2023-10-05T19:30:21.743 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:21.738+0000 7f296e044e40 1 -- 172.21.15.167:0/1098790227 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:110 7.3 7:c609908c:::notify.5:head [watch unwatch cookie 94213103124384] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e32) v8 -- 0x55afb28b50e0 con 0x7f2954010070 2023-10-05T19:30:21.746 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:21.742+0000 7f296c37a700 1 -- 172.21.15.167:0/1098790227 <== osd.0 v2:172.21.15.118:6800/27690 30 ==== osd_op_reply(110 notify.5 [watch unwatch cookie 94213103124384] v32'263 uv261 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f2964050040 con 0x7f2954010070 2023-10-05T19:30:21.746 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:21.742+0000 7f296e044e40 20 remove_watcher() i=5 2023-10-05T19:30:21.746 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:21.742+0000 7f296e044e40 1 -- 172.21.15.167:0/1098790227 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:111 7.14 7:2b04a3e9:::notify.6:head [watch unwatch cookie 94213103125488] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e32) v8 -- 0x55afb28b2e50 con 0x7f2954010070 2023-10-05T19:30:21.749 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:21.746+0000 7f296c37a700 1 -- 172.21.15.167:0/1098790227 <== osd.0 v2:172.21.15.118:6800/27690 31 ==== osd_op_reply(111 notify.6 [watch unwatch cookie 94213103125488] v32'263 uv261 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f2964050040 con 0x7f2954010070 2023-10-05T19:30:21.749 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:21.746+0000 7f296e044e40 20 remove_watcher() i=6 2023-10-05T19:30:21.749 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:21.746+0000 7f296e044e40 1 -- 172.21.15.167:0/1098790227 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:112 7.9 7:93e5b521:::notify.7:head [watch unwatch cookie 94213103129328] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e32) v8 -- 0x55afb28b50e0 con 0x7f2954010070 2023-10-05T19:30:21.752 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:21.750+0000 7f296c37a700 1 -- 172.21.15.167:0/1098790227 <== osd.0 v2:172.21.15.118:6800/27690 32 ==== osd_op_reply(112 notify.7 [watch unwatch cookie 94213103129328] v32'263 uv261 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f2964050040 con 0x7f2954010070 2023-10-05T19:30:21.752 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:21.750+0000 7f296e044e40 20 remove_watcher() i=7 2023-10-05T19:30:21.754 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:21.750+0000 7f293d7e2700 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2023-10-05T19:30:21.754 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:21.750+0000 7f296e044e40 1 -- 172.21.15.167:0/1098790227 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:113 6.4 6:22d26bf9:::data_loggenerations_metadata:head [watch unwatch cookie 94213103130432] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e32) v8 -- 0x55afb28b50e0 con 0x55afb287d5f0 2023-10-05T19:30:21.758 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:21.754+0000 7f296c37a700 1 -- 172.21.15.167:0/1098790227 <== osd.1 v2:172.21.15.167:6800/30035 39 ==== osd_op_reply(113 data_loggenerations_metadata [watch unwatch cookie 94213103130432] v32'254 uv1 ondisk = 0) v8 ==== 172+0+0 (crc 0 0 0) 0x7f2964050040 con 0x55afb287d5f0 2023-10-05T19:30:21.759 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:21.754+0000 7f296e044e40 1 -- 172.21.15.167:0/1098790227 >> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] conn(0x7f2954010070 msgr2=0x7f295401c430 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-05T19:30:21.759 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:21.754+0000 7f296e044e40 1 --2- 172.21.15.167:0/1098790227 >> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] conn(0x7f2954010070 0x7f295401c430 crc :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-05T19:30:21.759 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:21.754+0000 7f296e044e40 1 -- 172.21.15.167:0/1098790227 >> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] conn(0x55afb287d5f0 msgr2=0x55afb28aaa50 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-05T19:30:21.759 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:21.754+0000 7f296e044e40 1 --2- 172.21.15.167:0/1098790227 >> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] conn(0x55afb287d5f0 0x55afb28aaa50 crc :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-05T19:30:21.759 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:21.754+0000 7f296e044e40 1 -- 172.21.15.167:0/1098790227 >> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] conn(0x7f295c0176e0 msgr2=0x7f295c0420e0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-05T19:30:21.759 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:21.754+0000 7f296e044e40 1 --2- 172.21.15.167:0/1098790227 >> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] conn(0x7f295c0176e0 0x7f295c0420e0 crc :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-05T19:30:21.759 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:21.754+0000 7f296e044e40 1 -- 172.21.15.167:0/1098790227 >> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] conn(0x55afb28ae4a0 msgr2=0x55afb28b0880 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-05T19:30:21.759 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:21.754+0000 7f296e044e40 1 --2- 172.21.15.167:0/1098790227 >> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] conn(0x55afb28ae4a0 0x55afb28b0880 crc :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-05T19:30:21.760 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:21.754+0000 7f296e044e40 1 -- 172.21.15.167:0/1098790227 >> [v2:172.21.15.167:6824/29889,v1:172.21.15.167:6825/29889] conn(0x7f294c030c00 msgr2=0x7f294c0330b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-05T19:30:21.760 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:21.754+0000 7f296e044e40 1 --2- 172.21.15.167:0/1098790227 >> [v2:172.21.15.167:6824/29889,v1:172.21.15.167:6825/29889] conn(0x7f294c030c00 0x7f294c0330b0 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7f2954000f80 tx=0x7f2954013000 comp rx=0 tx=0).stop 2023-10-05T19:30:21.760 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:21.754+0000 7f296c37a700 1 -- 172.21.15.167:0/1098790227 reap_dead start 2023-10-05T19:30:21.760 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:21.754+0000 7f296e044e40 1 -- 172.21.15.167:0/1098790227 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55afb2879010 msgr2=0x55afb289ef60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-05T19:30:21.760 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:21.754+0000 7f296e044e40 1 --2- 172.21.15.167:0/1098790227 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55afb2879010 0x55afb289ef60 secure :-1 s=READY pgs=659 cs=0 l=1 rev1=1 crypto rx=0x55afb2894870 tx=0x7f295c016000 comp rx=0 tx=0).stop 2023-10-05T19:30:21.760 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:21.754+0000 7f296e044e40 1 -- 172.21.15.167:0/1098790227 shutdown_connections 2023-10-05T19:30:21.760 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:21.754+0000 7f296e044e40 1 --2- 172.21.15.167:0/1098790227 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55afb2879010 0x55afb289ef60 unknown :-1 s=CLOSED pgs=659 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-05T19:30:21.760 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:21.754+0000 7f296e044e40 1 -- 172.21.15.167:0/1098790227 >> 172.21.15.167:0/1098790227 conn(0x55afb27ec860 msgr2=0x55afb28927d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-05T19:30:21.760 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:21.754+0000 7f296e044e40 1 -- 172.21.15.167:0/1098790227 shutdown_connections 2023-10-05T19:30:21.761 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:21.754+0000 7f296e044e40 1 -- 172.21.15.167:0/1098790227 wait complete. 2023-10-05T19:30:21.762 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:21.754+0000 7f296e044e40 1 -- 172.21.15.167:0/2989941766 >> [v2:172.21.15.167:6824/29889,v1:172.21.15.167:6825/29889] conn(0x7f28ac030c50 msgr2=0x7f28ac033100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-05T19:30:21.762 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:21.754+0000 7f296e044e40 1 --2- 172.21.15.167:0/2989941766 >> [v2:172.21.15.167:6824/29889,v1:172.21.15.167:6825/29889] conn(0x7f28ac030c50 0x7f28ac033100 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x55afb28b9070 tx=0x7f2964000f90 comp rx=0 tx=0).stop 2023-10-05T19:30:21.762 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:21.754+0000 7f296e044e40 1 -- 172.21.15.167:0/2989941766 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55afb28c6e20 msgr2=0x55afb28cc8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-05T19:30:21.762 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:21.754+0000 7f296e044e40 1 --2- 172.21.15.167:0/2989941766 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55afb28c6e20 0x55afb28cc8f0 secure :-1 s=READY pgs=661 cs=0 l=1 rev1=1 crypto rx=0x55afb28c0f00 tx=0x7f295402f000 comp rx=0 tx=0).stop 2023-10-05T19:30:21.763 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:21.754+0000 7f296e044e40 1 -- 172.21.15.167:0/2989941766 shutdown_connections 2023-10-05T19:30:21.763 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:21.754+0000 7f296e044e40 1 --2- 172.21.15.167:0/2989941766 >> [v2:172.21.15.167:6824/29889,v1:172.21.15.167:6825/29889] conn(0x7f28ac030c50 0x7f28ac033100 unknown :-1 s=CLOSED pgs=227 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-05T19:30:21.763 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:21.754+0000 7f296e044e40 1 --2- 172.21.15.167:0/2989941766 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55afb28c6e20 0x55afb28cc8f0 unknown :-1 s=CLOSED pgs=661 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-05T19:30:21.763 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:21.754+0000 7f296e044e40 1 -- 172.21.15.167:0/2989941766 >> 172.21.15.167:0/2989941766 conn(0x55afb28c21e0 msgr2=0x55afb28c02e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-05T19:30:21.763 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:21.754+0000 7f296e044e40 1 -- 172.21.15.167:0/2989941766 shutdown_connections 2023-10-05T19:30:21.763 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:21.758+0000 7f296e044e40 1 -- 172.21.15.167:0/2989941766 wait complete. 2023-10-05T19:30:21.764 INFO:teuthology.orchestra.run.smithi167.stdout:{ 2023-10-05T19:30:21.764 INFO:teuthology.orchestra.run.smithi167.stdout: "entries": [], 2023-10-05T19:30:21.764 INFO:teuthology.orchestra.run.smithi167.stdout: "summary": [] 2023-10-05T19:30:21.764 INFO:teuthology.orchestra.run.smithi167.stdout:} 2023-10-05T19:30:21.767 DEBUG:tasks.util.rgw: json result: {'entries': [], 'summary': []} 2023-10-05T19:30:21.768 INFO:tasks.util.rgw:rgwadmin: client.0 : ['user', 'rm', '--uid', 'foo', '--purge-data'] 2023-10-05T19:30:21.768 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'ceph', 'user', 'rm', '--uid', 'foo', '--purge-data'] 2023-10-05T19:30:21.768 DEBUG:teuthology.orchestra.run.smithi167:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster ceph user rm --uid foo --purge-data 2023-10-05T19:30:21.856 INFO:teuthology.orchestra.run.smithi167.stderr:ignoring --setuser ceph since I am not root 2023-10-05T19:30:21.856 INFO:teuthology.orchestra.run.smithi167.stderr:ignoring --setgroup ceph since I am not root 2023-10-05T19:30:21.857 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:21.854+0000 7fd29bc01e40 1 Processor -- start 2023-10-05T19:30:21.857 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:21.854+0000 7fd29bc01e40 1 -- start start 2023-10-05T19:30:21.857 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:21.854+0000 7fd29bc01e40 1 --2- >> v2:172.21.15.118:3300/0 conn(0x5570998f1d10 0x5570998f20e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-05T19:30:21.857 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:21.854+0000 7fd29bc01e40 1 -- --> v1:172.21.15.118:6789/0 -- auth(proto 0 26 bytes epoch 0) v1 -- 0x55709983fbd0 con 0x5570998f2620 2023-10-05T19:30:21.857 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:21.854+0000 7fd29bc01e40 1 -- --> v2:172.21.15.118:3300/0 -- mon_getmap magic: 0 v1 -- 0x5570997fcee0 con 0x5570998f1d10 2023-10-05T19:30:21.858 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:21.854+0000 7fd299736700 1 --2- >> v2:172.21.15.118:3300/0 conn(0x5570998f1d10 0x5570998f20e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-05T19:30:21.858 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:21.854+0000 7fd298f35700 1 --1- >> v1:172.21.15.118:6789/0 conn(0x5570998f2620 0x5570998f30c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.118:6789/0 says I am v1:172.21.15.167:50796/0 (socket says 172.21.15.167:50796) 2023-10-05T19:30:21.858 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:21.854+0000 7fd298f35700 1 -- 172.21.15.167:0/2761410398 learned_addr learned my addr 172.21.15.167:0/2761410398 (peer_addr_for_me v1:172.21.15.167:0/0) 2023-10-05T19:30:21.859 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:21.854+0000 7fd299736700 1 -- 172.21.15.167:0/2761410398 >> v1:172.21.15.118:6789/0 conn(0x5570998f2620 legacy=0x5570998f30c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-05T19:30:21.859 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:21.854+0000 7fd299736700 1 -- 172.21.15.167:0/2761410398 --> v2:172.21.15.118:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5570998f37d0 con 0x5570998f1d10 2023-10-05T19:30:21.859 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:21.854+0000 7fd299736700 1 --2- 172.21.15.167:0/2761410398 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5570998f1d10 0x5570998f20e0 secure :-1 s=READY pgs=663 cs=0 l=1 rev1=1 crypto rx=0x7fd28c00a5d0 tx=0x7fd28c005a50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4920b9f4629f2fc9 server_cookie=0 in_seq=0 out_seq=0 2023-10-05T19:30:21.859 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:21.854+0000 7fd293fff700 1 -- 172.21.15.167:0/2761410398 <== mon.0 v2:172.21.15.118:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fd28c0086a0 con 0x5570998f1d10 2023-10-05T19:30:21.859 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:21.854+0000 7fd293fff700 1 -- 172.21.15.167:0/2761410398 <== mon.0 v2:172.21.15.118:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd28c008800 con 0x5570998f1d10 2023-10-05T19:30:21.860 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:21.854+0000 7fd293fff700 1 -- 172.21.15.167:0/2761410398 <== mon.0 v2:172.21.15.118:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fd28c008ad0 con 0x5570998f1d10 2023-10-05T19:30:21.864 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:21.858+0000 7fd29bc01e40 1 -- 172.21.15.167:0/2761410398 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5570998f1d10 msgr2=0x5570998f20e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-05T19:30:21.864 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:21.858+0000 7fd29bc01e40 1 --2- 172.21.15.167:0/2761410398 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5570998f1d10 0x5570998f20e0 secure :-1 s=READY pgs=663 cs=0 l=1 rev1=1 crypto rx=0x7fd28c00a5d0 tx=0x7fd28c005a50 comp rx=0 tx=0).stop 2023-10-05T19:30:21.864 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:21.858+0000 7fd29bc01e40 1 -- 172.21.15.167:0/2761410398 shutdown_connections 2023-10-05T19:30:21.865 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:21.858+0000 7fd29bc01e40 1 --2- 172.21.15.167:0/2761410398 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5570998f1d10 0x5570998f20e0 unknown :-1 s=CLOSED pgs=663 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-05T19:30:21.865 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:21.858+0000 7fd29bc01e40 1 -- 172.21.15.167:0/2761410398 >> 172.21.15.167:0/2761410398 conn(0x5570998ed730 msgr2=0x5570998efb40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-05T19:30:21.865 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:21.858+0000 7fd29bc01e40 1 -- 172.21.15.167:0/2761410398 shutdown_connections 2023-10-05T19:30:21.865 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:21.858+0000 7fd29bc01e40 1 -- 172.21.15.167:0/2761410398 wait complete. 2023-10-05T19:30:21.865 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:21.858+0000 7fd29bc01e40 1 Processor -- start 2023-10-05T19:30:21.865 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:21.858+0000 7fd29bc01e40 1 -- start start 2023-10-05T19:30:21.865 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:21.858+0000 7fd29bc01e40 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5570998f1d10 0x557099916a30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-05T19:30:21.865 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:21.858+0000 7fd29bc01e40 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x5570997c0e30 con 0x5570998f1d10 2023-10-05T19:30:21.865 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:21.858+0000 7fd299736700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5570998f1d10 0x557099916a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-05T19:30:21.865 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:21.858+0000 7fd299736700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5570998f1d10 0x557099916a30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.118:3300/0 says I am v2:172.21.15.167:32992/0 (socket says 172.21.15.167:32992) 2023-10-05T19:30:21.865 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:21.858+0000 7fd299736700 1 -- 172.21.15.167:0/2634294516 learned_addr learned my addr 172.21.15.167:0/2634294516 (peer_addr_for_me v2:172.21.15.167:0/0) 2023-10-05T19:30:21.866 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:21.858+0000 7fd299736700 1 -- 172.21.15.167:0/2634294516 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd28c005cf0 con 0x5570998f1d10 2023-10-05T19:30:21.866 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:21.858+0000 7fd299736700 1 --2- 172.21.15.167:0/2634294516 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5570998f1d10 0x557099916a30 secure :-1 s=READY pgs=664 cs=0 l=1 rev1=1 crypto rx=0x7fd28c00a520 tx=0x7fd28c015000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=21b744c7872c9e0d server_cookie=0 in_seq=0 out_seq=0 2023-10-05T19:30:21.866 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:21.858+0000 7fd292ffd700 1 -- 172.21.15.167:0/2634294516 <== mon.0 v2:172.21.15.118:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fd28c008a40 con 0x5570998f1d10 2023-10-05T19:30:21.866 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:21.858+0000 7fd292ffd700 1 -- 172.21.15.167:0/2634294516 <== mon.0 v2:172.21.15.118:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd28c008ba0 con 0x5570998f1d10 2023-10-05T19:30:21.866 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:21.858+0000 7fd292ffd700 1 -- 172.21.15.167:0/2634294516 <== mon.0 v2:172.21.15.118:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fd28c008e90 con 0x5570998f1d10 2023-10-05T19:30:21.866 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:21.858+0000 7fd29bc01e40 1 -- 172.21.15.167:0/2634294516 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5570998f1d10 msgr2=0x557099916a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-05T19:30:21.866 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:21.858+0000 7fd29bc01e40 1 --2- 172.21.15.167:0/2634294516 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5570998f1d10 0x557099916a30 secure :-1 s=READY pgs=664 cs=0 l=1 rev1=1 crypto rx=0x7fd28c00a520 tx=0x7fd28c015000 comp rx=0 tx=0).stop 2023-10-05T19:30:21.866 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:21.858+0000 7fd29bc01e40 1 -- 172.21.15.167:0/2634294516 shutdown_connections 2023-10-05T19:30:21.866 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:21.858+0000 7fd29bc01e40 1 --2- 172.21.15.167:0/2634294516 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5570998f1d10 0x557099916a30 unknown :-1 s=CLOSED pgs=664 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-05T19:30:21.866 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:21.858+0000 7fd29bc01e40 1 -- 172.21.15.167:0/2634294516 >> 172.21.15.167:0/2634294516 conn(0x5570998ed730 msgr2=0x557099866e10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-05T19:30:21.866 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:21.862+0000 7fd29bc01e40 1 -- 172.21.15.167:0/2634294516 shutdown_connections 2023-10-05T19:30:21.867 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:21.862+0000 7fd29bc01e40 1 -- 172.21.15.167:0/2634294516 wait complete. 2023-10-05T19:30:21.867 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:21.862+0000 7fd29bc01e40 1 Processor -- start 2023-10-05T19:30:21.867 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:21.862+0000 7fd29bc01e40 1 -- start start 2023-10-05T19:30:21.867 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:21.862+0000 7fd29bc01e40 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5570998f1d10 0x55709991ac60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-05T19:30:21.867 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:21.862+0000 7fd29bc01e40 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x557099793ed0 con 0x5570998f1d10 2023-10-05T19:30:21.868 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:21.862+0000 7fd299736700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5570998f1d10 0x55709991ac60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-05T19:30:21.868 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:21.862+0000 7fd299736700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5570998f1d10 0x55709991ac60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.118:3300/0 says I am v2:172.21.15.167:32998/0 (socket says 172.21.15.167:32998) 2023-10-05T19:30:21.868 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:21.862+0000 7fd299736700 1 -- 172.21.15.167:0/444809523 learned_addr learned my addr 172.21.15.167:0/444809523 (peer_addr_for_me v2:172.21.15.167:0/0) 2023-10-05T19:30:21.868 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:21.862+0000 7fd299736700 1 -- 172.21.15.167:0/444809523 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5570998f37d0 con 0x5570998f1d10 2023-10-05T19:30:21.869 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:21.862+0000 7fd299736700 1 --2- 172.21.15.167:0/444809523 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5570998f1d10 0x55709991ac60 secure :-1 s=READY pgs=665 cs=0 l=1 rev1=1 crypto rx=0x7fd28c01e040 tx=0x7fd28c003050 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-05T19:30:21.869 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:21.862+0000 7fd2917fa700 1 -- 172.21.15.167:0/444809523 <== mon.0 v2:172.21.15.118:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fd28c012c00 con 0x5570998f1d10 2023-10-05T19:30:21.869 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:21.862+0000 7fd2917fa700 1 -- 172.21.15.167:0/444809523 <== mon.0 v2:172.21.15.118:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd28c02a070 con 0x5570998f1d10 2023-10-05T19:30:21.869 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:21.862+0000 7fd29bc01e40 1 -- 172.21.15.167:0/444809523 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x55709991b1a0 con 0x5570998f1d10 2023-10-05T19:30:21.869 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:21.862+0000 7fd2917fa700 1 -- 172.21.15.167:0/444809523 <== mon.0 v2:172.21.15.118:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fd28c031480 con 0x5570998f1d10 2023-10-05T19:30:21.869 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:21.862+0000 7fd29bc01e40 1 -- 172.21.15.167:0/444809523 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55709991b640 con 0x5570998f1d10 2023-10-05T19:30:21.869 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:21.866+0000 7fd2917fa700 1 -- 172.21.15.167:0/444809523 <== mon.0 v2:172.21.15.118:3300/0 4 ==== mgrmap(e 5) v1 ==== 40329+0+0 (secure 0 0 0) 0x7fd28c019070 con 0x5570998f1d10 2023-10-05T19:30:21.869 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:21.866+0000 7fd2917fa700 1 --2- 172.21.15.167:0/444809523 >> [v2:172.21.15.167:6824/29889,v1:172.21.15.167:6825/29889] conn(0x7fd27c030bf0 0x7fd27c0330a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-05T19:30:21.870 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:21.866+0000 7fd2917fa700 1 -- 172.21.15.167:0/444809523 <== mon.0 v2:172.21.15.118:3300/0 5 ==== osd_map(32..32 src has 1..32) v4 ==== 6260+0+0 (secure 0 0 0) 0x7fd28c02c070 con 0x5570998f1d10 2023-10-05T19:30:21.870 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:21.866+0000 7fd298f35700 1 --2- 172.21.15.167:0/444809523 >> [v2:172.21.15.167:6824/29889,v1:172.21.15.167:6825/29889] conn(0x7fd27c030bf0 0x7fd27c0330a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-05T19:30:21.870 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:21.866+0000 7fd29bc01e40 20 rados->read ofs=0 len=0 2023-10-05T19:30:21.871 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:21.866+0000 7fd29bc01e40 1 --2- 172.21.15.167:0/444809523 >> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] conn(0x5570998f2620 0x557099926710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-05T19:30:21.871 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:21.866+0000 7fd29bc01e40 1 -- 172.21.15.167:0/444809523 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:1 5.12 5:49953fa1:::default.realm:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32) v8 -- 0x557099926e40 con 0x5570998f2620 2023-10-05T19:30:21.871 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:21.866+0000 7fd299f37700 1 --2- 172.21.15.167:0/444809523 >> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] conn(0x5570998f2620 0x557099926710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-05T19:30:21.871 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:21.866+0000 7fd298f35700 1 --2- 172.21.15.167:0/444809523 >> [v2:172.21.15.167:6824/29889,v1:172.21.15.167:6825/29889] conn(0x7fd27c030bf0 0x7fd27c0330a0 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7fd2840016f0 tx=0x7fd2840067f0 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-05T19:30:21.871 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:21.866+0000 7fd299f37700 1 --2- 172.21.15.167:0/444809523 >> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] conn(0x5570998f2620 0x557099926710 crc :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-05T19:30:21.872 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:21.866+0000 7fd299f37700 1 -- 172.21.15.167:0/444809523 <== osd.1 v2:172.21.15.167:6800/30035 1 ==== osd_op_reply(1 default.realm [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 ==== 157+0+0 (crc 0 0 0) 0x7fd29400a500 con 0x5570998f2620 2023-10-05T19:30:21.872 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:21.866+0000 7fd29bc01e40 20 rados_obj.operate() r=-2 bl.length=0 2023-10-05T19:30:21.872 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:21.866+0000 7fd29bc01e40 20 realm 2023-10-05T19:30:21.872 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:21.866+0000 7fd29bc01e40 20 rados->read ofs=0 len=0 2023-10-05T19:30:21.873 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:21.866+0000 7fd29bc01e40 1 -- 172.21.15.167:0/444809523 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:2 5.12 5:49953fa1:::default.realm:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32) v8 -- 0x557099923d40 con 0x5570998f2620 2023-10-05T19:30:21.873 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:21.866+0000 7fd299f37700 1 -- 172.21.15.167:0/444809523 <== osd.1 v2:172.21.15.167:6800/30035 2 ==== osd_op_reply(2 default.realm [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 ==== 157+0+0 (crc 0 0 0) 0x7fd29400a500 con 0x5570998f2620 2023-10-05T19:30:21.873 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:21.866+0000 7fd29bc01e40 20 rados_obj.operate() r=-2 bl.length=0 2023-10-05T19:30:21.873 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:21.866+0000 7fd29bc01e40 4 RGWPeriod::init failed to init realm id : (2) No such file or directory 2023-10-05T19:30:21.873 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:21.866+0000 7fd29bc01e40 20 rados->read ofs=0 len=0 2023-10-05T19:30:21.873 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:21.866+0000 7fd29bc01e40 1 -- 172.21.15.167:0/444809523 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:3 5.12 5:49953fa1:::default.realm:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32) v8 -- 0x557099927970 con 0x5570998f2620 2023-10-05T19:30:21.873 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:21.870+0000 7fd299f37700 1 -- 172.21.15.167:0/444809523 <== osd.1 v2:172.21.15.167:6800/30035 3 ==== osd_op_reply(3 default.realm [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 ==== 157+0+0 (crc 0 0 0) 0x7fd29400a500 con 0x5570998f2620 2023-10-05T19:30:21.873 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:21.870+0000 7fd29bc01e40 20 rados_obj.operate() r=-2 bl.length=0 2023-10-05T19:30:21.873 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:21.870+0000 7fd29bc01e40 20 rados->read ofs=0 len=0 2023-10-05T19:30:21.873 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:21.870+0000 7fd29bc01e40 1 -- 172.21.15.167:0/444809523 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:4 5.f 5:f43b6ece:::zone_names.default:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32) v8 -- 0x557099927580 con 0x5570998f2620 2023-10-05T19:30:21.875 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:21.870+0000 7fd299f37700 1 -- 172.21.15.167:0/444809523 <== osd.1 v2:172.21.15.167:6800/30035 4 ==== osd_op_reply(4 zone_names.default [read 0~46 out=46b] v0'0 uv1 ondisk = 0) v8 ==== 162+0+46 (crc 0 0 0) 0x7fd29400a500 con 0x5570998f2620 2023-10-05T19:30:21.875 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:21.870+0000 7fd29bc01e40 20 rados_obj.operate() r=0 bl.length=46 2023-10-05T19:30:21.875 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:21.870+0000 7fd29bc01e40 20 rados->read ofs=0 len=0 2023-10-05T19:30:21.875 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:21.870+0000 7fd29bc01e40 1 -- 172.21.15.167:0/444809523 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:5 5.1d 5:b8374eb4:::zone_info.9a9df965-6612-45d4-a45f-6e3c889bacc2:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32) v8 -- 0x557099928800 con 0x5570998f2620 2023-10-05T19:30:21.876 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:21.870+0000 7fd299f37700 1 -- 172.21.15.167:0/444809523 <== osd.1 v2:172.21.15.167:6800/30035 5 ==== osd_op_reply(5 zone_info.9a9df965-6612-45d4-a45f-6e3c889bacc2 [read 0~874 out=874b] v0'0 uv2 ondisk = 0) v8 ==== 190+0+874 (crc 0 0 0) 0x7fd29400a500 con 0x5570998f2620 2023-10-05T19:30:21.876 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:21.870+0000 7fd29bc01e40 20 rados_obj.operate() r=0 bl.length=874 2023-10-05T19:30:21.876 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:21.870+0000 7fd29bc01e40 20 searching for the correct realm 2023-10-05T19:30:21.876 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:21.870+0000 7fd29bc01e40 1 --2- 172.21.15.167:0/444809523 >> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] conn(0x55709992a5c0 0x55709992c9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-05T19:30:21.876 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:21.870+0000 7fd29bc01e40 1 -- 172.21.15.167:0/444809523 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:6 5.0 5:00000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x55709992cee0 con 0x55709992a5c0 2023-10-05T19:30:21.876 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:21.870+0000 7fd299736700 1 --2- 172.21.15.167:0/444809523 >> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] conn(0x55709992a5c0 0x55709992c9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-05T19:30:21.876 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:21.870+0000 7fd299736700 1 --2- 172.21.15.167:0/444809523 >> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] conn(0x55709992a5c0 0x55709992c9a0 crc :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-05T19:30:21.876 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:21.870+0000 7fd299736700 1 -- 172.21.15.167:0/444809523 <== osd.3 v2:172.21.15.167:6808/30036 1 ==== osd_op_reply(6 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fd28c022020 con 0x55709992a5c0 2023-10-05T19:30:21.876 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:21.870+0000 7fd299736700 1 --2- 172.21.15.167:0/444809523 >> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] conn(0x7fd28c007030 0x7fd28c026880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-05T19:30:21.876 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:21.870+0000 7fd299736700 1 -- 172.21.15.167:0/444809523 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:7 5.10 5:08000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x55709992cee0 con 0x7fd28c007030 2023-10-05T19:30:21.876 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:21.870+0000 7fd298f35700 1 --2- 172.21.15.167:0/444809523 >> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] conn(0x7fd28c007030 0x7fd28c026880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-05T19:30:21.877 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:21.874+0000 7fd298f35700 1 --2- 172.21.15.167:0/444809523 >> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] conn(0x7fd28c007030 0x7fd28c026880 crc :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-05T19:30:21.877 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:21.874+0000 7fd298f35700 1 -- 172.21.15.167:0/444809523 <== osd.2 v2:172.21.15.167:6816/30037 1 ==== osd_op_reply(7 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fd2840060d0 con 0x7fd28c007030 2023-10-05T19:30:21.877 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:21.874+0000 7fd298f35700 1 --2- 172.21.15.167:0/444809523 >> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] conn(0x7fd284011040 0x7fd284014950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-05T19:30:21.877 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:21.874+0000 7fd298f35700 1 -- 172.21.15.167:0/444809523 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:8 5.8 5:10000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x55709992cee0 con 0x7fd284011040 2023-10-05T19:30:21.878 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:21.874+0000 7fd299f37700 1 --2- 172.21.15.167:0/444809523 >> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] conn(0x7fd284011040 0x7fd284014950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-05T19:30:21.878 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:21.874+0000 7fd299f37700 1 --2- 172.21.15.167:0/444809523 >> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] conn(0x7fd284011040 0x7fd284014950 crc :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-05T19:30:21.878 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:21.874+0000 7fd299f37700 1 -- 172.21.15.167:0/444809523 <== osd.0 v2:172.21.15.118:6800/27690 1 ==== osd_op_reply(8 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fd294014040 con 0x7fd284011040 2023-10-05T19:30:21.878 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:21.874+0000 7fd299f37700 1 -- 172.21.15.167:0/444809523 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:9 5.18 5:18000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x55709992cee0 con 0x5570998f2620 2023-10-05T19:30:21.879 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:21.874+0000 7fd299f37700 1 -- 172.21.15.167:0/444809523 <== osd.1 v2:172.21.15.167:6800/30035 6 ==== osd_op_reply(9 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fd294014040 con 0x5570998f2620 2023-10-05T19:30:21.879 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:21.874+0000 7fd299f37700 1 -- 172.21.15.167:0/444809523 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:10 5.4 5:20000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x55709992cee0 con 0x5570998f2620 2023-10-05T19:30:21.880 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:21.874+0000 7fd299f37700 1 -- 172.21.15.167:0/444809523 <== osd.1 v2:172.21.15.167:6800/30035 7 ==== osd_op_reply(10 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fd294014040 con 0x5570998f2620 2023-10-05T19:30:21.880 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:21.878+0000 7fd299f37700 1 -- 172.21.15.167:0/444809523 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:11 5.14 5:28000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x55709992cee0 con 0x7fd284011040 2023-10-05T19:30:21.880 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:21.878+0000 7fd299f37700 1 -- 172.21.15.167:0/444809523 <== osd.0 v2:172.21.15.118:6800/27690 2 ==== osd_op_reply(11 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fd294014040 con 0x7fd284011040 2023-10-05T19:30:21.881 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:21.878+0000 7fd299f37700 1 -- 172.21.15.167:0/444809523 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:12 5.c 5:30000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x55709992cee0 con 0x7fd284011040 2023-10-05T19:30:21.881 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:21.878+0000 7fd299f37700 1 -- 172.21.15.167:0/444809523 <== osd.0 v2:172.21.15.118:6800/27690 3 ==== osd_op_reply(12 [pgnls start_epoch 32 out=112b] v23'1 uv1 ondisk = 1) v8 ==== 144+0+112 (crc 0 0 0) 0x7fd294014040 con 0x7fd284011040 2023-10-05T19:30:21.881 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:21.878+0000 7fd299f37700 1 -- 172.21.15.167:0/444809523 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:13 5.1c 5:38000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x55709992cee0 con 0x7fd28c007030 2023-10-05T19:30:21.882 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:21.878+0000 7fd298f35700 1 -- 172.21.15.167:0/444809523 <== osd.2 v2:172.21.15.167:6816/30037 2 ==== osd_op_reply(13 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fd284004bd0 con 0x7fd28c007030 2023-10-05T19:30:21.882 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:21.878+0000 7fd298f35700 1 -- 172.21.15.167:0/444809523 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:14 5.2 5:40000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x55709992cee0 con 0x55709992a5c0 2023-10-05T19:30:21.882 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:21.878+0000 7fd299736700 1 -- 172.21.15.167:0/444809523 <== osd.3 v2:172.21.15.167:6808/30036 2 ==== osd_op_reply(14 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fd28c022020 con 0x55709992a5c0 2023-10-05T19:30:21.883 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:21.878+0000 7fd299736700 1 -- 172.21.15.167:0/444809523 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:15 5.12 5:48000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x55709992cee0 con 0x5570998f2620 2023-10-05T19:30:21.883 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:21.878+0000 7fd299f37700 1 -- 172.21.15.167:0/444809523 <== osd.1 v2:172.21.15.167:6800/30035 8 ==== osd_op_reply(15 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fd294014040 con 0x5570998f2620 2023-10-05T19:30:21.883 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:21.878+0000 7fd299f37700 1 -- 172.21.15.167:0/444809523 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:16 5.a 5:50000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x55709992cee0 con 0x7fd284011040 2023-10-05T19:30:21.884 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:21.882+0000 7fd299f37700 1 -- 172.21.15.167:0/444809523 <== osd.0 v2:172.21.15.118:6800/27690 4 ==== osd_op_reply(16 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fd294022040 con 0x7fd284011040 2023-10-05T19:30:21.884 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:21.882+0000 7fd299f37700 1 -- 172.21.15.167:0/444809523 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:17 5.1a 5:58000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x55709992cee0 con 0x5570998f2620 2023-10-05T19:30:21.885 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:21.882+0000 7fd299f37700 1 -- 172.21.15.167:0/444809523 <== osd.1 v2:172.21.15.167:6800/30035 9 ==== osd_op_reply(17 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fd294022040 con 0x5570998f2620 2023-10-05T19:30:21.885 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:21.882+0000 7fd299f37700 1 -- 172.21.15.167:0/444809523 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:18 5.6 5:60000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x55709992cee0 con 0x7fd28c007030 2023-10-05T19:30:21.885 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:21.882+0000 7fd298f35700 1 -- 172.21.15.167:0/444809523 <== osd.2 v2:172.21.15.167:6816/30037 3 ==== osd_op_reply(18 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fd284004bd0 con 0x7fd28c007030 2023-10-05T19:30:21.886 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:21.882+0000 7fd298f35700 1 -- 172.21.15.167:0/444809523 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:19 5.16 5:68000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x55709992cee0 con 0x55709992a5c0 2023-10-05T19:30:21.886 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:21.882+0000 7fd299736700 1 -- 172.21.15.167:0/444809523 <== osd.3 v2:172.21.15.167:6808/30036 3 ==== osd_op_reply(19 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fd28c022020 con 0x55709992a5c0 2023-10-05T19:30:21.886 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:21.882+0000 7fd299736700 1 -- 172.21.15.167:0/444809523 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:20 5.e 5:70000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x55709992cee0 con 0x55709992a5c0 2023-10-05T19:30:21.887 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:21.882+0000 7fd299736700 1 -- 172.21.15.167:0/444809523 <== osd.3 v2:172.21.15.167:6808/30036 4 ==== osd_op_reply(20 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fd28c022020 con 0x55709992a5c0 2023-10-05T19:30:21.887 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:21.882+0000 7fd299736700 1 -- 172.21.15.167:0/444809523 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:21 5.1e 5:78000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x55709992cee0 con 0x7fd284011040 2023-10-05T19:30:21.887 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:21.882+0000 7fd299f37700 1 -- 172.21.15.167:0/444809523 <== osd.0 v2:172.21.15.118:6800/27690 5 ==== osd_op_reply(21 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fd294022040 con 0x7fd284011040 2023-10-05T19:30:21.888 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:21.882+0000 7fd299f37700 1 -- 172.21.15.167:0/444809523 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:22 5.1 5:80000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x55709992cee0 con 0x55709992a5c0 2023-10-05T19:30:21.888 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:21.886+0000 7fd299736700 1 -- 172.21.15.167:0/444809523 <== osd.3 v2:172.21.15.167:6808/30036 5 ==== osd_op_reply(22 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fd28c022020 con 0x55709992a5c0 2023-10-05T19:30:21.888 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:21.886+0000 7fd299736700 1 -- 172.21.15.167:0/444809523 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:23 5.11 5:88000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x55709992cee0 con 0x55709992a5c0 2023-10-05T19:30:21.889 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:21.886+0000 7fd299736700 1 -- 172.21.15.167:0/444809523 <== osd.3 v2:172.21.15.167:6808/30036 6 ==== osd_op_reply(23 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fd28c022020 con 0x55709992a5c0 2023-10-05T19:30:21.889 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:21.886+0000 7fd299736700 1 -- 172.21.15.167:0/444809523 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:24 5.9 5:90000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x55709992cee0 con 0x5570998f2620 2023-10-05T19:30:21.889 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:21.886+0000 7fd299f37700 1 -- 172.21.15.167:0/444809523 <== osd.1 v2:172.21.15.167:6800/30035 10 ==== osd_op_reply(24 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fd294022040 con 0x5570998f2620 2023-10-05T19:30:21.890 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:21.886+0000 7fd299f37700 1 -- 172.21.15.167:0/444809523 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:25 5.19 5:98000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x55709992cee0 con 0x5570998f2620 2023-10-05T19:30:21.890 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:21.886+0000 7fd299f37700 1 -- 172.21.15.167:0/444809523 <== osd.1 v2:172.21.15.167:6800/30035 11 ==== osd_op_reply(25 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fd294022040 con 0x5570998f2620 2023-10-05T19:30:21.890 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:21.886+0000 7fd299f37700 1 -- 172.21.15.167:0/444809523 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:26 5.5 5:a0000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x55709992cee0 con 0x7fd284011040 2023-10-05T19:30:21.891 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:21.886+0000 7fd299f37700 1 -- 172.21.15.167:0/444809523 <== osd.0 v2:172.21.15.118:6800/27690 6 ==== osd_op_reply(26 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fd294022040 con 0x7fd284011040 2023-10-05T19:30:21.891 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:21.886+0000 7fd299f37700 1 -- 172.21.15.167:0/444809523 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:27 5.15 5:a8000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x55709992cee0 con 0x55709992a5c0 2023-10-05T19:30:21.892 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:21.890+0000 7fd299736700 1 -- 172.21.15.167:0/444809523 <== osd.3 v2:172.21.15.167:6808/30036 7 ==== osd_op_reply(27 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fd28c022020 con 0x55709992a5c0 2023-10-05T19:30:21.892 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:21.890+0000 7fd299736700 1 -- 172.21.15.167:0/444809523 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:28 5.d 5:b0000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x55709992cee0 con 0x7fd28c007030 2023-10-05T19:30:21.893 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:21.890+0000 7fd298f35700 1 -- 172.21.15.167:0/444809523 <== osd.2 v2:172.21.15.167:6816/30037 4 ==== osd_op_reply(28 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fd284004bd0 con 0x7fd28c007030 2023-10-05T19:30:21.893 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:21.890+0000 7fd298f35700 1 -- 172.21.15.167:0/444809523 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:29 5.1d 5:b8000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x55709992cee0 con 0x5570998f2620 2023-10-05T19:30:21.893 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:21.890+0000 7fd299f37700 1 -- 172.21.15.167:0/444809523 <== osd.1 v2:172.21.15.167:6800/30035 12 ==== osd_op_reply(29 [pgnls start_epoch 32 out=107b] v30'2 uv2 ondisk = 1) v8 ==== 144+0+107 (crc 0 0 0) 0x7fd294022040 con 0x5570998f2620 2023-10-05T19:30:21.894 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:21.890+0000 7fd299f37700 1 -- 172.21.15.167:0/444809523 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:30 5.3 5:c0000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x55709992cee0 con 0x7fd284011040 2023-10-05T19:30:21.894 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:21.890+0000 7fd299f37700 1 -- 172.21.15.167:0/444809523 <== osd.0 v2:172.21.15.118:6800/27690 7 ==== osd_op_reply(30 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fd294022040 con 0x7fd284011040 2023-10-05T19:30:21.894 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:21.890+0000 7fd299f37700 1 -- 172.21.15.167:0/444809523 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:31 5.13 5:c8000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x55709992cee0 con 0x55709992a5c0 2023-10-05T19:30:21.895 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:21.890+0000 7fd299736700 1 -- 172.21.15.167:0/444809523 <== osd.3 v2:172.21.15.167:6808/30036 8 ==== osd_op_reply(31 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fd28c022020 con 0x55709992a5c0 2023-10-05T19:30:21.895 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:21.890+0000 7fd299736700 1 -- 172.21.15.167:0/444809523 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:32 5.b 5:d0000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x55709992cee0 con 0x7fd284011040 2023-10-05T19:30:21.896 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:21.894+0000 7fd299f37700 1 -- 172.21.15.167:0/444809523 <== osd.0 v2:172.21.15.118:6800/27690 8 ==== osd_op_reply(32 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fd294022040 con 0x7fd284011040 2023-10-05T19:30:21.896 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:21.894+0000 7fd299f37700 1 -- 172.21.15.167:0/444809523 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:33 5.1b 5:d8000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x55709992cee0 con 0x7fd28c007030 2023-10-05T19:30:21.897 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:21.894+0000 7fd298f35700 1 -- 172.21.15.167:0/444809523 <== osd.2 v2:172.21.15.167:6816/30037 5 ==== osd_op_reply(33 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fd284004bd0 con 0x7fd28c007030 2023-10-05T19:30:21.897 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:21.894+0000 7fd298f35700 1 -- 172.21.15.167:0/444809523 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:34 5.7 5:e0000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x55709992cee0 con 0x55709992a5c0 2023-10-05T19:30:21.898 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:21.894+0000 7fd299736700 1 -- 172.21.15.167:0/444809523 <== osd.3 v2:172.21.15.167:6808/30036 9 ==== osd_op_reply(34 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fd28c022020 con 0x55709992a5c0 2023-10-05T19:30:21.898 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:21.894+0000 7fd299736700 1 -- 172.21.15.167:0/444809523 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:35 5.17 5:e8000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x55709992cee0 con 0x7fd284011040 2023-10-05T19:30:21.899 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:21.894+0000 7fd299f37700 1 -- 172.21.15.167:0/444809523 <== osd.0 v2:172.21.15.118:6800/27690 9 ==== osd_op_reply(35 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fd294022040 con 0x7fd284011040 2023-10-05T19:30:21.899 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:21.894+0000 7fd299f37700 1 -- 172.21.15.167:0/444809523 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:36 5.f 5:f0000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x55709992cee0 con 0x5570998f2620 2023-10-05T19:30:21.900 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:21.898+0000 7fd299f37700 1 -- 172.21.15.167:0/444809523 <== osd.1 v2:172.21.15.167:6800/30035 13 ==== osd_op_reply(36 [pgnls start_epoch 32 out=115b] v23'2 uv2 ondisk = 1) v8 ==== 144+0+115 (crc 0 0 0) 0x7fd29402f080 con 0x5570998f2620 2023-10-05T19:30:21.900 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:21.898+0000 7fd299f37700 1 -- 172.21.15.167:0/444809523 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:37 5.1f 5:f8000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x55709992cee0 con 0x7fd28c007030 2023-10-05T19:30:21.901 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:21.898+0000 7fd298f35700 1 -- 172.21.15.167:0/444809523 <== osd.2 v2:172.21.15.167:6816/30037 6 ==== osd_op_reply(37 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fd284004bd0 con 0x7fd28c007030 2023-10-05T19:30:21.901 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:21.898+0000 7fd29bc01e40 20 RGWRados::pool_iterate: got zonegroup_info.4f1f719d-9648-417b-b10c-e5d939825419 2023-10-05T19:30:21.901 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:21.898+0000 7fd29bc01e40 20 RGWRados::pool_iterate: got zone_info.9a9df965-6612-45d4-a45f-6e3c889bacc2 2023-10-05T19:30:21.901 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:21.898+0000 7fd29bc01e40 20 RGWRados::pool_iterate: got zone_names.default 2023-10-05T19:30:21.901 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:21.898+0000 7fd29bc01e40 20 RGWRados::pool_iterate: got zonegroups_names.default 2023-10-05T19:30:21.901 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:21.898+0000 7fd29bc01e40 20 rados->read ofs=0 len=0 2023-10-05T19:30:21.901 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:21.898+0000 7fd29bc01e40 1 -- 172.21.15.167:0/444809523 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:38 5.12 5:49953fa1:::default.realm:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32) v8 -- 0x557099929370 con 0x5570998f2620 2023-10-05T19:30:21.902 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:21.898+0000 7fd299f37700 1 -- 172.21.15.167:0/444809523 <== osd.1 v2:172.21.15.167:6800/30035 14 ==== osd_op_reply(38 default.realm [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 ==== 157+0+0 (crc 0 0 0) 0x7fd29402f080 con 0x5570998f2620 2023-10-05T19:30:21.902 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:21.898+0000 7fd29bc01e40 20 rados_obj.operate() r=-2 bl.length=0 2023-10-05T19:30:21.902 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:21.898+0000 7fd29bc01e40 20 rados->read ofs=0 len=0 2023-10-05T19:30:21.903 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:21.898+0000 7fd29bc01e40 1 -- 172.21.15.167:0/444809523 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:39 5.f 5:f4c53578:::zonegroups_names.default:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32) v8 -- 0x557099929c60 con 0x5570998f2620 2023-10-05T19:30:21.903 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:21.898+0000 7fd299f37700 1 -- 172.21.15.167:0/444809523 <== osd.1 v2:172.21.15.167:6800/30035 15 ==== osd_op_reply(39 zonegroups_names.default [read 0~46 out=46b] v0'0 uv2 ondisk = 0) v8 ==== 168+0+46 (crc 0 0 0) 0x7fd29402f080 con 0x5570998f2620 2023-10-05T19:30:21.903 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:21.898+0000 7fd29bc01e40 20 rados_obj.operate() r=0 bl.length=46 2023-10-05T19:30:21.903 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:21.898+0000 7fd29bc01e40 20 rados->read ofs=0 len=0 2023-10-05T19:30:21.903 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:21.898+0000 7fd29bc01e40 1 -- 172.21.15.167:0/444809523 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:40 5.c 5:30d299a5:::zonegroup_info.4f1f719d-9648-417b-b10c-e5d939825419:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32) v8 -- 0x55709992d320 con 0x7fd284011040 2023-10-05T19:30:21.904 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:21.898+0000 7fd299f37700 1 -- 172.21.15.167:0/444809523 <== osd.0 v2:172.21.15.118:6800/27690 10 ==== osd_op_reply(40 zonegroup_info.4f1f719d-9648-417b-b10c-e5d939825419 [read 0~362 out=362b] v0'0 uv1 ondisk = 0) v8 ==== 195+0+362 (crc 0 0 0) 0x7fd29402f080 con 0x7fd284011040 2023-10-05T19:30:21.904 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:21.898+0000 7fd29bc01e40 20 rados_obj.operate() r=0 bl.length=362 2023-10-05T19:30:21.904 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:21.898+0000 7fd29bc01e40 20 rados->read ofs=0 len=0 2023-10-05T19:30:21.904 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:21.898+0000 7fd29bc01e40 1 -- 172.21.15.167:0/444809523 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:41 5.13 5:cbbf6c60:::converted:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32) v8 -- 0x55709992d320 con 0x55709992a5c0 2023-10-05T19:30:21.904 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:21.902+0000 7fd299736700 1 -- 172.21.15.167:0/444809523 <== osd.3 v2:172.21.15.167:6808/30036 10 ==== osd_op_reply(41 converted [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 ==== 153+0+0 (crc 0 0 0) 0x7fd28c022020 con 0x55709992a5c0 2023-10-05T19:30:21.904 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:21.902+0000 7fd29bc01e40 20 rados_obj.operate() r=-2 bl.length=0 2023-10-05T19:30:21.905 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:21.902+0000 7fd29bc01e40 20 rados->read ofs=0 len=0 2023-10-05T19:30:21.905 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:21.902+0000 7fd29bc01e40 1 -- 172.21.15.167:0/444809523 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:42 5.12 5:49953fa1:::default.realm:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32) v8 -- 0x55709992d6d0 con 0x5570998f2620 2023-10-05T19:30:21.906 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:21.902+0000 7fd299f37700 1 -- 172.21.15.167:0/444809523 <== osd.1 v2:172.21.15.167:6800/30035 16 ==== osd_op_reply(42 default.realm [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 ==== 157+0+0 (crc 0 0 0) 0x7fd29402f080 con 0x5570998f2620 2023-10-05T19:30:21.906 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:21.902+0000 7fd29bc01e40 20 rados_obj.operate() r=-2 bl.length=0 2023-10-05T19:30:21.906 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:21.902+0000 7fd29bc01e40 20 rados->read ofs=0 len=0 2023-10-05T19:30:21.906 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:21.902+0000 7fd29bc01e40 1 -- 172.21.15.167:0/444809523 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:43 5.f 5:f4c53578:::zonegroups_names.default:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32) v8 -- 0x55709992db00 con 0x5570998f2620 2023-10-05T19:30:21.906 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:21.902+0000 7fd299f37700 1 -- 172.21.15.167:0/444809523 <== osd.1 v2:172.21.15.167:6800/30035 17 ==== osd_op_reply(43 zonegroups_names.default [read 0~46 out=46b] v0'0 uv2 ondisk = 0) v8 ==== 168+0+46 (crc 0 0 0) 0x7fd29402f080 con 0x5570998f2620 2023-10-05T19:30:21.906 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:21.902+0000 7fd29bc01e40 20 rados_obj.operate() r=0 bl.length=46 2023-10-05T19:30:21.906 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:21.902+0000 7fd29bc01e40 1 -- 172.21.15.167:0/444809523 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:44 5.0 5:00000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x55709992eb10 con 0x55709992a5c0 2023-10-05T19:30:21.907 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:21.902+0000 7fd299736700 1 -- 172.21.15.167:0/444809523 <== osd.3 v2:172.21.15.167:6808/30036 11 ==== osd_op_reply(44 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fd28c022020 con 0x55709992a5c0 2023-10-05T19:30:21.907 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:21.902+0000 7fd299736700 1 -- 172.21.15.167:0/444809523 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:45 5.10 5:08000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x55709992eb10 con 0x7fd28c007030 2023-10-05T19:30:21.907 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:21.902+0000 7fd298f35700 1 -- 172.21.15.167:0/444809523 <== osd.2 v2:172.21.15.167:6816/30037 7 ==== osd_op_reply(45 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fd284004bd0 con 0x7fd28c007030 2023-10-05T19:30:21.908 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:21.906+0000 7fd298f35700 1 -- 172.21.15.167:0/444809523 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:46 5.8 5:10000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x55709992eb10 con 0x7fd284011040 2023-10-05T19:30:21.908 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:21.906+0000 7fd299f37700 1 -- 172.21.15.167:0/444809523 <== osd.0 v2:172.21.15.118:6800/27690 11 ==== osd_op_reply(46 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fd29402f080 con 0x7fd284011040 2023-10-05T19:30:21.909 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:21.906+0000 7fd299f37700 1 -- 172.21.15.167:0/444809523 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:47 5.18 5:18000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x557099929c60 con 0x5570998f2620 2023-10-05T19:30:21.909 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:21.906+0000 7fd299f37700 1 -- 172.21.15.167:0/444809523 <== osd.1 v2:172.21.15.167:6800/30035 18 ==== osd_op_reply(47 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fd29402f080 con 0x5570998f2620 2023-10-05T19:30:21.909 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:21.906+0000 7fd299f37700 1 -- 172.21.15.167:0/444809523 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:48 5.4 5:20000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x557099929c60 con 0x5570998f2620 2023-10-05T19:30:21.910 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:21.906+0000 7fd299f37700 1 -- 172.21.15.167:0/444809523 <== osd.1 v2:172.21.15.167:6800/30035 19 ==== osd_op_reply(48 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fd29402f080 con 0x5570998f2620 2023-10-05T19:30:21.910 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:21.906+0000 7fd299f37700 1 -- 172.21.15.167:0/444809523 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:49 5.14 5:28000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x557099929c60 con 0x7fd284011040 2023-10-05T19:30:21.911 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:21.906+0000 7fd299f37700 1 -- 172.21.15.167:0/444809523 <== osd.0 v2:172.21.15.118:6800/27690 12 ==== osd_op_reply(49 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fd29402f080 con 0x7fd284011040 2023-10-05T19:30:21.911 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:21.906+0000 7fd299f37700 1 -- 172.21.15.167:0/444809523 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:50 5.c 5:30000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x557099929c60 con 0x7fd284011040 2023-10-05T19:30:21.912 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:21.910+0000 7fd299f37700 1 -- 172.21.15.167:0/444809523 <== osd.0 v2:172.21.15.118:6800/27690 13 ==== osd_op_reply(50 [pgnls start_epoch 32 out=112b] v23'1 uv1 ondisk = 1) v8 ==== 144+0+112 (crc 0 0 0) 0x7fd29402f080 con 0x7fd284011040 2023-10-05T19:30:21.912 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:21.910+0000 7fd299f37700 1 -- 172.21.15.167:0/444809523 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:51 5.1c 5:38000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x557099929c60 con 0x7fd28c007030 2023-10-05T19:30:21.913 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:21.910+0000 7fd298f35700 1 -- 172.21.15.167:0/444809523 <== osd.2 v2:172.21.15.167:6816/30037 8 ==== osd_op_reply(51 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fd284004bd0 con 0x7fd28c007030 2023-10-05T19:30:21.913 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:21.910+0000 7fd298f35700 1 -- 172.21.15.167:0/444809523 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:52 5.2 5:40000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x557099929c60 con 0x55709992a5c0 2023-10-05T19:30:21.914 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:21.910+0000 7fd299736700 1 -- 172.21.15.167:0/444809523 <== osd.3 v2:172.21.15.167:6808/30036 12 ==== osd_op_reply(52 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fd28c022020 con 0x55709992a5c0 2023-10-05T19:30:21.914 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:21.910+0000 7fd299736700 1 -- 172.21.15.167:0/444809523 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:53 5.12 5:48000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x557099929c60 con 0x5570998f2620 2023-10-05T19:30:21.914 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:21.910+0000 7fd299f37700 1 -- 172.21.15.167:0/444809523 <== osd.1 v2:172.21.15.167:6800/30035 20 ==== osd_op_reply(53 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fd29402f080 con 0x5570998f2620 2023-10-05T19:30:21.914 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:21.910+0000 7fd299f37700 1 -- 172.21.15.167:0/444809523 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:54 5.a 5:50000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x557099929c60 con 0x7fd284011040 2023-10-05T19:30:21.915 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:21.910+0000 7fd299f37700 1 -- 172.21.15.167:0/444809523 <== osd.0 v2:172.21.15.118:6800/27690 14 ==== osd_op_reply(54 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fd29403d040 con 0x7fd284011040 2023-10-05T19:30:21.915 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:21.910+0000 7fd299f37700 1 -- 172.21.15.167:0/444809523 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:55 5.1a 5:58000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x557099929c60 con 0x5570998f2620 2023-10-05T19:30:21.916 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:21.914+0000 7fd299f37700 1 -- 172.21.15.167:0/444809523 <== osd.1 v2:172.21.15.167:6800/30035 21 ==== osd_op_reply(55 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fd29403d040 con 0x5570998f2620 2023-10-05T19:30:21.916 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:21.914+0000 7fd299f37700 1 -- 172.21.15.167:0/444809523 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:56 5.6 5:60000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x557099929c60 con 0x7fd28c007030 2023-10-05T19:30:21.917 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:21.914+0000 7fd298f35700 1 -- 172.21.15.167:0/444809523 <== osd.2 v2:172.21.15.167:6816/30037 9 ==== osd_op_reply(56 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fd284004bd0 con 0x7fd28c007030 2023-10-05T19:30:21.917 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:21.914+0000 7fd298f35700 1 -- 172.21.15.167:0/444809523 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:57 5.16 5:68000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x557099929c60 con 0x55709992a5c0 2023-10-05T19:30:21.918 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:21.914+0000 7fd299736700 1 -- 172.21.15.167:0/444809523 <== osd.3 v2:172.21.15.167:6808/30036 13 ==== osd_op_reply(57 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fd28c022020 con 0x55709992a5c0 2023-10-05T19:30:21.918 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:21.914+0000 7fd299736700 1 -- 172.21.15.167:0/444809523 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:58 5.e 5:70000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x557099929c60 con 0x55709992a5c0 2023-10-05T19:30:21.919 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:21.914+0000 7fd299736700 1 -- 172.21.15.167:0/444809523 <== osd.3 v2:172.21.15.167:6808/30036 14 ==== osd_op_reply(58 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fd28c022020 con 0x55709992a5c0 2023-10-05T19:30:21.919 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:21.914+0000 7fd299736700 1 -- 172.21.15.167:0/444809523 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:59 5.1e 5:78000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x557099929c60 con 0x7fd284011040 2023-10-05T19:30:21.920 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:21.918+0000 7fd299f37700 1 -- 172.21.15.167:0/444809523 <== osd.0 v2:172.21.15.118:6800/27690 15 ==== osd_op_reply(59 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fd29403d040 con 0x7fd284011040 2023-10-05T19:30:21.920 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:21.918+0000 7fd299f37700 1 -- 172.21.15.167:0/444809523 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:60 5.1 5:80000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x557099929c60 con 0x55709992a5c0 2023-10-05T19:30:21.921 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:21.918+0000 7fd299736700 1 -- 172.21.15.167:0/444809523 <== osd.3 v2:172.21.15.167:6808/30036 15 ==== osd_op_reply(60 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fd28c022020 con 0x55709992a5c0 2023-10-05T19:30:21.921 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:21.918+0000 7fd299736700 1 -- 172.21.15.167:0/444809523 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:61 5.11 5:88000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x557099929c60 con 0x55709992a5c0 2023-10-05T19:30:21.921 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:21.918+0000 7fd299736700 1 -- 172.21.15.167:0/444809523 <== osd.3 v2:172.21.15.167:6808/30036 16 ==== osd_op_reply(61 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fd28c022020 con 0x55709992a5c0 2023-10-05T19:30:21.921 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:21.918+0000 7fd299736700 1 -- 172.21.15.167:0/444809523 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:62 5.9 5:90000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x557099929c60 con 0x5570998f2620 2023-10-05T19:30:21.922 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:21.918+0000 7fd299f37700 1 -- 172.21.15.167:0/444809523 <== osd.1 v2:172.21.15.167:6800/30035 22 ==== osd_op_reply(62 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fd29403d040 con 0x5570998f2620 2023-10-05T19:30:21.922 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:21.918+0000 7fd299f37700 1 -- 172.21.15.167:0/444809523 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:63 5.19 5:98000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x557099929c60 con 0x5570998f2620 2023-10-05T19:30:21.923 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:21.918+0000 7fd299f37700 1 -- 172.21.15.167:0/444809523 <== osd.1 v2:172.21.15.167:6800/30035 23 ==== osd_op_reply(63 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fd29403d040 con 0x5570998f2620 2023-10-05T19:30:21.923 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:21.918+0000 7fd299f37700 1 -- 172.21.15.167:0/444809523 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:64 5.5 5:a0000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x557099929c60 con 0x7fd284011040 2023-10-05T19:30:21.924 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:21.922+0000 7fd299f37700 1 -- 172.21.15.167:0/444809523 <== osd.0 v2:172.21.15.118:6800/27690 16 ==== osd_op_reply(64 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fd29403d040 con 0x7fd284011040 2023-10-05T19:30:21.924 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:21.922+0000 7fd299f37700 1 -- 172.21.15.167:0/444809523 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:65 5.15 5:a8000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x557099929c60 con 0x55709992a5c0 2023-10-05T19:30:21.925 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:21.922+0000 7fd299736700 1 -- 172.21.15.167:0/444809523 <== osd.3 v2:172.21.15.167:6808/30036 17 ==== osd_op_reply(65 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fd28c022020 con 0x55709992a5c0 2023-10-05T19:30:21.925 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:21.922+0000 7fd299736700 1 -- 172.21.15.167:0/444809523 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:66 5.d 5:b0000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x557099929c60 con 0x7fd28c007030 2023-10-05T19:30:21.926 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:21.922+0000 7fd298f35700 1 -- 172.21.15.167:0/444809523 <== osd.2 v2:172.21.15.167:6816/30037 10 ==== osd_op_reply(66 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fd284004bd0 con 0x7fd28c007030 2023-10-05T19:30:21.926 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:21.922+0000 7fd298f35700 1 -- 172.21.15.167:0/444809523 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:67 5.1d 5:b8000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x557099929c60 con 0x5570998f2620 2023-10-05T19:30:21.927 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:21.922+0000 7fd299f37700 1 -- 172.21.15.167:0/444809523 <== osd.1 v2:172.21.15.167:6800/30035 24 ==== osd_op_reply(67 [pgnls start_epoch 32 out=107b] v30'2 uv2 ondisk = 1) v8 ==== 144+0+107 (crc 0 0 0) 0x7fd29403d040 con 0x5570998f2620 2023-10-05T19:30:21.927 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:21.922+0000 7fd299f37700 1 -- 172.21.15.167:0/444809523 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:68 5.3 5:c0000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x557099929c60 con 0x7fd284011040 2023-10-05T19:30:21.928 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:21.926+0000 7fd299f37700 1 -- 172.21.15.167:0/444809523 <== osd.0 v2:172.21.15.118:6800/27690 17 ==== osd_op_reply(68 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fd29403d040 con 0x7fd284011040 2023-10-05T19:30:22.336 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:21.926+0000 7fd299f37700 1 -- 172.21.15.167:0/444809523 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:69 5.13 5:c8000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x557099929c60 con 0x55709992a5c0 2023-10-05T19:30:22.398 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:21.926+0000 7fd299736700 1 -- 172.21.15.167:0/444809523 <== osd.3 v2:172.21.15.167:6808/30036 18 ==== osd_op_reply(69 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fd28c022020 con 0x55709992a5c0 2023-10-05T19:30:22.399 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:21.926+0000 7fd299736700 1 -- 172.21.15.167:0/444809523 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:70 5.b 5:d0000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x557099929c60 con 0x7fd284011040 2023-10-05T19:30:22.399 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:21.926+0000 7fd299f37700 1 -- 172.21.15.167:0/444809523 <== osd.0 v2:172.21.15.118:6800/27690 18 ==== osd_op_reply(70 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fd29403d040 con 0x7fd284011040 2023-10-05T19:30:22.399 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:21.926+0000 7fd299f37700 1 -- 172.21.15.167:0/444809523 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:71 5.1b 5:d8000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x557099929c60 con 0x7fd28c007030 2023-10-05T19:30:22.399 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:21.926+0000 7fd298f35700 1 -- 172.21.15.167:0/444809523 <== osd.2 v2:172.21.15.167:6816/30037 11 ==== osd_op_reply(71 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fd284004bd0 con 0x7fd28c007030 2023-10-05T19:30:22.399 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:21.926+0000 7fd298f35700 1 -- 172.21.15.167:0/444809523 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:72 5.7 5:e0000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x557099929c60 con 0x55709992a5c0 2023-10-05T19:30:22.399 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:21.930+0000 7fd299736700 1 -- 172.21.15.167:0/444809523 <== osd.3 v2:172.21.15.167:6808/30036 19 ==== osd_op_reply(72 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fd28c022020 con 0x55709992a5c0 2023-10-05T19:30:22.399 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:21.930+0000 7fd299736700 1 -- 172.21.15.167:0/444809523 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:73 5.17 5:e8000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x557099929c60 con 0x7fd284011040 2023-10-05T19:30:22.399 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:21.930+0000 7fd299f37700 1 -- 172.21.15.167:0/444809523 <== osd.0 v2:172.21.15.118:6800/27690 19 ==== osd_op_reply(73 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fd29403d040 con 0x7fd284011040 2023-10-05T19:30:22.399 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:21.930+0000 7fd299f37700 1 -- 172.21.15.167:0/444809523 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:74 5.f 5:f0000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x557099929c60 con 0x5570998f2620 2023-10-05T19:30:22.399 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:21.930+0000 7fd299f37700 1 -- 172.21.15.167:0/444809523 <== osd.1 v2:172.21.15.167:6800/30035 25 ==== osd_op_reply(74 [pgnls start_epoch 32 out=115b] v23'2 uv2 ondisk = 1) v8 ==== 144+0+115 (crc 0 0 0) 0x7fd29403d040 con 0x5570998f2620 2023-10-05T19:30:22.399 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:21.930+0000 7fd299f37700 1 -- 172.21.15.167:0/444809523 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:75 5.1f 5:f8000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x557099929c60 con 0x7fd28c007030 2023-10-05T19:30:22.400 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:21.930+0000 7fd298f35700 1 -- 172.21.15.167:0/444809523 <== osd.2 v2:172.21.15.167:6816/30037 12 ==== osd_op_reply(75 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fd284004bd0 con 0x7fd28c007030 2023-10-05T19:30:22.400 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:21.930+0000 7fd29bc01e40 20 RGWRados::pool_iterate: got zonegroup_info.4f1f719d-9648-417b-b10c-e5d939825419 2023-10-05T19:30:22.400 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:21.930+0000 7fd29bc01e40 20 RGWRados::pool_iterate: got zone_info.9a9df965-6612-45d4-a45f-6e3c889bacc2 2023-10-05T19:30:22.400 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:21.930+0000 7fd29bc01e40 20 RGWRados::pool_iterate: got zone_names.default 2023-10-05T19:30:22.400 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:21.930+0000 7fd29bc01e40 20 RGWRados::pool_iterate: got zonegroups_names.default 2023-10-05T19:30:22.400 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:21.930+0000 7fd29bc01e40 20 rados->read ofs=0 len=0 2023-10-05T19:30:22.400 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:21.930+0000 7fd29bc01e40 1 -- 172.21.15.167:0/444809523 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:76 5.f 5:f43b6ece:::zone_names.default:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32) v8 -- 0x55709992db00 con 0x5570998f2620 2023-10-05T19:30:22.400 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:21.934+0000 7fd299f37700 1 -- 172.21.15.167:0/444809523 <== osd.1 v2:172.21.15.167:6800/30035 26 ==== osd_op_reply(76 zone_names.default [read 0~46 out=46b] v0'0 uv1 ondisk = 0) v8 ==== 162+0+46 (crc 0 0 0) 0x7fd29403d040 con 0x5570998f2620 2023-10-05T19:30:22.400 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:21.934+0000 7fd29bc01e40 20 rados_obj.operate() r=0 bl.length=46 2023-10-05T19:30:22.400 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:21.934+0000 7fd29bc01e40 20 rados->read ofs=0 len=0 2023-10-05T19:30:22.400 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:21.934+0000 7fd29bc01e40 1 -- 172.21.15.167:0/444809523 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:77 5.1d 5:b8374eb4:::zone_info.9a9df965-6612-45d4-a45f-6e3c889bacc2:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32) v8 -- 0x55709992deb0 con 0x5570998f2620 2023-10-05T19:30:22.401 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:21.934+0000 7fd299f37700 1 -- 172.21.15.167:0/444809523 <== osd.1 v2:172.21.15.167:6800/30035 27 ==== osd_op_reply(77 zone_info.9a9df965-6612-45d4-a45f-6e3c889bacc2 [read 0~874 out=874b] v0'0 uv2 ondisk = 0) v8 ==== 190+0+874 (crc 0 0 0) 0x7fd29403d040 con 0x5570998f2620 2023-10-05T19:30:22.401 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:21.934+0000 7fd29bc01e40 20 rados_obj.operate() r=0 bl.length=874 2023-10-05T19:30:22.401 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:21.934+0000 7fd29bc01e40 20 rados->read ofs=0 len=0 2023-10-05T19:30:22.401 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:21.934+0000 7fd29bc01e40 1 -- 172.21.15.167:0/444809523 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:78 5.f 5:f4c53578:::zonegroups_names.default:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32) v8 -- 0x55709992dee0 con 0x5570998f2620 2023-10-05T19:30:22.401 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:21.934+0000 7fd299f37700 1 -- 172.21.15.167:0/444809523 <== osd.1 v2:172.21.15.167:6800/30035 28 ==== osd_op_reply(78 zonegroups_names.default [read 0~46 out=46b] v0'0 uv2 ondisk = 0) v8 ==== 168+0+46 (crc 0 0 0) 0x7fd29403d040 con 0x5570998f2620 2023-10-05T19:30:22.401 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:21.934+0000 7fd29bc01e40 20 rados_obj.operate() r=0 bl.length=46 2023-10-05T19:30:22.401 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:21.934+0000 7fd29bc01e40 20 rados->read ofs=0 len=0 2023-10-05T19:30:22.401 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:21.934+0000 7fd29bc01e40 1 -- 172.21.15.167:0/444809523 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:79 5.c 5:30d299a5:::zonegroup_info.4f1f719d-9648-417b-b10c-e5d939825419:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32) v8 -- 0x55709992e310 con 0x7fd284011040 2023-10-05T19:30:22.401 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:21.934+0000 7fd299f37700 1 -- 172.21.15.167:0/444809523 <== osd.0 v2:172.21.15.118:6800/27690 20 ==== osd_op_reply(79 zonegroup_info.4f1f719d-9648-417b-b10c-e5d939825419 [read 0~362 out=362b] v0'0 uv1 ondisk = 0) v8 ==== 195+0+362 (crc 0 0 0) 0x7fd29403d040 con 0x7fd284011040 2023-10-05T19:30:22.401 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:21.934+0000 7fd29bc01e40 20 rados_obj.operate() r=0 bl.length=362 2023-10-05T19:30:22.401 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:21.934+0000 7fd29bc01e40 20 rados->read ofs=0 len=0 2023-10-05T19:30:22.401 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:21.934+0000 7fd29bc01e40 1 -- 172.21.15.167:0/444809523 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:80 5.3 5:c0ad8cbd:::region_map:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32) v8 -- 0x55709992e310 con 0x7fd284011040 2023-10-05T19:30:22.402 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:21.934+0000 7fd299f37700 1 -- 172.21.15.167:0/444809523 <== osd.0 v2:172.21.15.118:6800/27690 21 ==== osd_op_reply(80 region_map [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 ==== 154+0+0 (crc 0 0 0) 0x7fd29403d040 con 0x7fd284011040 2023-10-05T19:30:22.402 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:21.934+0000 7fd29bc01e40 20 rados_obj.operate() r=-2 bl.length=0 2023-10-05T19:30:22.402 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:21.934+0000 7fd29bc01e40 20 zone default found 2023-10-05T19:30:22.402 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:21.934+0000 7fd29bc01e40 4 Realm: () 2023-10-05T19:30:22.402 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:21.934+0000 7fd29bc01e40 4 ZoneGroup: default (4f1f719d-9648-417b-b10c-e5d939825419) 2023-10-05T19:30:22.402 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:21.934+0000 7fd29bc01e40 4 Zone: default (9a9df965-6612-45d4-a45f-6e3c889bacc2) 2023-10-05T19:30:22.402 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:21.934+0000 7fd29bc01e40 10 cannot find current period zonegroup using local zonegroup configuration 2023-10-05T19:30:22.402 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:21.934+0000 7fd29bc01e40 20 zonegroup default 2023-10-05T19:30:22.402 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:21.934+0000 7fd29bc01e40 20 rados->read ofs=0 len=0 2023-10-05T19:30:22.402 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:21.934+0000 7fd29bc01e40 1 -- 172.21.15.167:0/444809523 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:81 5.3 5:c52100b6:::period_config.default:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32) v8 -- 0x55709992e1e0 con 0x7fd284011040 2023-10-05T19:30:22.402 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:21.938+0000 7fd299f37700 1 -- 172.21.15.167:0/444809523 <== osd.0 v2:172.21.15.118:6800/27690 22 ==== osd_op_reply(81 period_config.default [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 ==== 165+0+0 (crc 0 0 0) 0x7fd29403d040 con 0x7fd284011040 2023-10-05T19:30:22.403 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:21.938+0000 7fd29bc01e40 20 rados_obj.operate() r=-2 bl.length=0 2023-10-05T19:30:22.403 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:21.938+0000 7fd29bc01e40 20 rados->read ofs=0 len=0 2023-10-05T19:30:22.403 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:21.938+0000 7fd29bc01e40 1 -- 172.21.15.167:0/444809523 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:82 6.2 6:4347d321:::bucket.sync-source-hints.:head [call version.read in=11b,read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32) v8 -- 0x55709992fb60 con 0x55709992a5c0 2023-10-05T19:30:22.403 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:21.938+0000 7fd299736700 1 -- 172.21.15.167:0/444809523 <== osd.3 v2:172.21.15.167:6808/30036 20 ==== osd_op_reply(82 bucket.sync-source-hints. [call,read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 ==== 211+0+0 (crc 0 0 0) 0x7fd28c022020 con 0x55709992a5c0 2023-10-05T19:30:22.403 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:21.938+0000 7fd29bc01e40 20 rados_obj.operate() r=-2 bl.length=0 2023-10-05T19:30:22.403 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:21.938+0000 7fd29bc01e40 20 rados->read ofs=0 len=0 2023-10-05T19:30:22.403 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:21.938+0000 7fd29bc01e40 1 -- 172.21.15.167:0/444809523 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:83 6.b 6:d467b91b:::bucket.sync-target-hints.:head [call version.read in=11b,read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32) v8 -- 0x55709992ff10 con 0x55709992a5c0 2023-10-05T19:30:22.403 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:21.938+0000 7fd299736700 1 -- 172.21.15.167:0/444809523 <== osd.3 v2:172.21.15.167:6808/30036 21 ==== osd_op_reply(83 bucket.sync-target-hints. [call,read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 ==== 211+0+0 (crc 0 0 0) 0x7fd28c022020 con 0x55709992a5c0 2023-10-05T19:30:22.403 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:21.938+0000 7fd29bc01e40 20 rados_obj.operate() r=-2 bl.length=0 2023-10-05T19:30:22.403 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:21.938+0000 7fd29bc01e40 20 started sync module instance, tier type = 2023-10-05T19:30:22.403 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:21.938+0000 7fd29bc01e40 20 started zone id=9a9df965-6612-45d4-a45f-6e3c889bacc2 (name=default) with tier type = 2023-10-05T19:30:22.403 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:21.938+0000 7fd29bc01e40 1 -- 172.21.15.167:0/444809523 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:84 7.1f 7:f95f44c2:::notify.0:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e32) v8 -- 0x55709992f220 con 0x7fd28c007030 2023-10-05T19:30:22.404 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:21.942+0000 7fd298f35700 1 -- 172.21.15.167:0/444809523 <== osd.2 v2:172.21.15.167:6816/30037 13 ==== osd_op_reply(84 notify.0 [create] v32'264 uv264 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7fd28402f040 con 0x7fd28c007030 2023-10-05T19:30:22.404 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:21.942+0000 7fd29bc01e40 1 -- 172.21.15.167:0/444809523 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:85 7.1f 7:f95f44c2:::notify.0:head [watch watch cookie 93942101243424] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e32) v8 -- 0x557099929890 con 0x7fd28c007030 2023-10-05T19:30:22.404 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:21.942+0000 7fd29bc01e40 1 -- 172.21.15.167:0/444809523 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:86 7.0 7:05bf5b68:::notify.1:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e32) v8 -- 0x557099930ca0 con 0x5570998f2620 2023-10-05T19:30:22.404 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:21.946+0000 7fd298f35700 1 -- 172.21.15.167:0/444809523 <== osd.2 v2:172.21.15.167:6816/30037 14 ==== osd_op_reply(85 notify.0 [watch watch cookie 93942101243424] v32'265 uv264 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7fd28402f040 con 0x7fd28c007030 2023-10-05T19:30:22.404 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:21.946+0000 7fd299f37700 1 -- 172.21.15.167:0/444809523 <== osd.1 v2:172.21.15.167:6800/30035 29 ==== osd_op_reply(86 notify.1 [create] v32'264 uv264 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7fd29403d040 con 0x5570998f2620 2023-10-05T19:30:22.404 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:21.946+0000 7fd29bc01e40 1 -- 172.21.15.167:0/444809523 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:87 7.0 7:05bf5b68:::notify.1:head [watch watch cookie 93942101246944] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e32) v8 -- 0x5570999310a0 con 0x5570998f2620 2023-10-05T19:30:22.404 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:21.946+0000 7fd29bc01e40 1 -- 172.21.15.167:0/444809523 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:88 7.15 7:a93a5511:::notify.2:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e32) v8 -- 0x557099931940 con 0x55709992a5c0 2023-10-05T19:30:22.404 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:21.946+0000 7fd299f37700 1 -- 172.21.15.167:0/444809523 <== osd.1 v2:172.21.15.167:6800/30035 30 ==== osd_op_reply(87 notify.1 [watch watch cookie 93942101246944] v32'265 uv264 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7fd29403d040 con 0x5570998f2620 2023-10-05T19:30:22.404 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:21.950+0000 7fd299736700 1 -- 172.21.15.167:0/444809523 <== osd.3 v2:172.21.15.167:6808/30036 22 ==== osd_op_reply(88 notify.2 [create] v32'264 uv264 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7fd28c022020 con 0x55709992a5c0 2023-10-05T19:30:22.404 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:21.950+0000 7fd29bc01e40 1 -- 172.21.15.167:0/444809523 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:89 7.15 7:a93a5511:::notify.2:head [watch watch cookie 93942101248048] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e32) v8 -- 0x557099931590 con 0x55709992a5c0 2023-10-05T19:30:22.404 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:21.950+0000 7fd29bc01e40 1 -- 172.21.15.167:0/444809523 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:90 7.e 7:7759931f:::notify.3:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e32) v8 -- 0x5570999323f0 con 0x7fd28c007030 2023-10-05T19:30:22.405 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:21.950+0000 7fd299736700 1 -- 172.21.15.167:0/444809523 <== osd.3 v2:172.21.15.167:6808/30036 23 ==== osd_op_reply(89 notify.2 [watch watch cookie 93942101248048] v32'265 uv264 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7fd28c022020 con 0x55709992a5c0 2023-10-05T19:30:22.405 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:21.950+0000 7fd298f35700 1 -- 172.21.15.167:0/444809523 <== osd.2 v2:172.21.15.167:6816/30037 15 ==== osd_op_reply(90 notify.3 [create] v32'264 uv264 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7fd28402f040 con 0x7fd28c007030 2023-10-05T19:30:22.405 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:21.950+0000 7fd29bc01e40 1 -- 172.21.15.167:0/444809523 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:91 7.e 7:7759931f:::notify.3:head [watch watch cookie 93942101250208] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e32) v8 -- 0x557099931fe0 con 0x7fd28c007030 2023-10-05T19:30:22.405 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:21.950+0000 7fd29bc01e40 1 -- 172.21.15.167:0/444809523 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:92 7.d 7:b4812045:::notify.4:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e32) v8 -- 0x557099933490 con 0x5570998f2620 2023-10-05T19:30:22.405 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:21.954+0000 7fd298f35700 1 -- 172.21.15.167:0/444809523 <== osd.2 v2:172.21.15.167:6816/30037 16 ==== osd_op_reply(91 notify.3 [watch watch cookie 93942101250208] v32'265 uv264 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7fd28402f040 con 0x7fd28c007030 2023-10-05T19:30:22.405 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:21.954+0000 7fd299f37700 1 -- 172.21.15.167:0/444809523 <== osd.1 v2:172.21.15.167:6800/30035 31 ==== osd_op_reply(92 notify.4 [create] v32'264 uv264 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7fd29403d040 con 0x5570998f2620 2023-10-05T19:30:22.405 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:21.954+0000 7fd29bc01e40 1 -- 172.21.15.167:0/444809523 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:93 7.d 7:b4812045:::notify.4:head [watch watch cookie 93942101256176] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e32) v8 -- 0x557099933890 con 0x5570998f2620 2023-10-05T19:30:22.405 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:21.954+0000 7fd29bc01e40 1 -- 172.21.15.167:0/444809523 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:94 7.3 7:c609908c:::notify.5:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e32) v8 -- 0x5570999342a0 con 0x7fd284011040 2023-10-05T19:30:22.405 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:21.958+0000 7fd299f37700 1 -- 172.21.15.167:0/444809523 <== osd.1 v2:172.21.15.167:6800/30035 32 ==== osd_op_reply(93 notify.4 [watch watch cookie 93942101256176] v32'265 uv264 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7fd29403d040 con 0x5570998f2620 2023-10-05T19:30:22.405 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:21.958+0000 7fd299f37700 1 -- 172.21.15.167:0/444809523 <== osd.0 v2:172.21.15.118:6800/27690 23 ==== osd_op_reply(94 notify.5 [create] v32'264 uv264 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7fd29403d040 con 0x7fd284011040 2023-10-05T19:30:22.405 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:21.958+0000 7fd29bc01e40 1 -- 172.21.15.167:0/444809523 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:95 7.3 7:c609908c:::notify.5:head [watch watch cookie 93942101258080] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e32) v8 -- 0x5570999342a0 con 0x7fd284011040 2023-10-05T19:30:22.406 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:21.958+0000 7fd29bc01e40 1 -- 172.21.15.167:0/444809523 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:96 7.14 7:2b04a3e9:::notify.6:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e32) v8 -- 0x557099934d00 con 0x7fd284011040 2023-10-05T19:30:22.406 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:21.962+0000 7fd299f37700 1 -- 172.21.15.167:0/444809523 <== osd.0 v2:172.21.15.118:6800/27690 24 ==== osd_op_reply(96 notify.6 [create] v32'264 uv264 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7fd29403d040 con 0x7fd284011040 2023-10-05T19:30:22.406 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:21.962+0000 7fd29bc01e40 1 -- 172.21.15.167:0/444809523 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:97 7.14 7:2b04a3e9:::notify.6:head [watch watch cookie 93942101264400] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e32) v8 -- 0x557099935150 con 0x7fd284011040 2023-10-05T19:30:22.406 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:21.962+0000 7fd29bc01e40 1 -- 172.21.15.167:0/444809523 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:98 7.9 7:93e5b521:::notify.7:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e32) v8 -- 0x557099935b60 con 0x7fd284011040 2023-10-05T19:30:22.406 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:21.962+0000 7fd299f37700 1 -- 172.21.15.167:0/444809523 <== osd.0 v2:172.21.15.118:6800/27690 25 ==== osd_op_reply(95 notify.5 [watch watch cookie 93942101258080] v32'265 uv264 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7fd29403d040 con 0x7fd284011040 2023-10-05T19:30:22.406 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:21.962+0000 7fd299f37700 1 -- 172.21.15.167:0/444809523 <== osd.0 v2:172.21.15.118:6800/27690 26 ==== osd_op_reply(97 notify.6 [watch watch cookie 93942101264400] v32'265 uv264 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7fd29403d040 con 0x7fd284011040 2023-10-05T19:30:22.406 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:21.962+0000 7fd299f37700 1 -- 172.21.15.167:0/444809523 <== osd.0 v2:172.21.15.118:6800/27690 27 ==== osd_op_reply(98 notify.7 [create] v32'264 uv264 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7fd29403d040 con 0x7fd284011040 2023-10-05T19:30:22.406 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:21.962+0000 7fd29bc01e40 1 -- 172.21.15.167:0/444809523 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:99 7.9 7:93e5b521:::notify.7:head [watch watch cookie 93942101265504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e32) v8 -- 0x5570999350b0 con 0x7fd284011040 2023-10-05T19:30:22.406 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:21.962+0000 7fd29bc01e40 20 add_watcher() i=0 2023-10-05T19:30:22.406 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:21.962+0000 7fd29bc01e40 20 add_watcher() i=1 2023-10-05T19:30:22.406 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:21.962+0000 7fd29bc01e40 20 add_watcher() i=2 2023-10-05T19:30:22.406 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:21.962+0000 7fd29bc01e40 20 add_watcher() i=3 2023-10-05T19:30:22.407 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:21.962+0000 7fd29bc01e40 20 add_watcher() i=4 2023-10-05T19:30:22.407 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:21.962+0000 7fd29bc01e40 20 add_watcher() i=5 2023-10-05T19:30:22.407 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:21.962+0000 7fd29bc01e40 20 add_watcher() i=6 2023-10-05T19:30:22.407 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:21.966+0000 7fd299f37700 1 -- 172.21.15.167:0/444809523 <== osd.0 v2:172.21.15.118:6800/27690 28 ==== osd_op_reply(99 notify.7 [watch watch cookie 93942101265504] v32'265 uv264 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7fd29403d040 con 0x7fd284011040 2023-10-05T19:30:22.407 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:21.966+0000 7fd29bc01e40 20 add_watcher() i=7 2023-10-05T19:30:22.407 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:21.966+0000 7fd29bc01e40 2 all 8 watchers are set, enabling cache 2023-10-05T19:30:22.407 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:21.966+0000 7fd29bc01e40 1 -- 172.21.15.167:0/444809523 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:100 6.4 6:22d26bf9:::data_loggenerations_metadata:head [call version.check_conds in=74b,call version.read in=11b,read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32) v8 -- 0x557099938b90 con 0x5570998f2620 2023-10-05T19:30:22.407 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:21.966+0000 7fd299f37700 1 -- 172.21.15.167:0/444809523 <== osd.1 v2:172.21.15.167:6800/30035 33 ==== osd_op_reply(100 data_loggenerations_metadata [call,call out=48b,read 0~28 out=28b] v0'0 uv1 ondisk = 0) v8 ==== 256+0+76 (crc 0 0 0) 0x7fd29403d040 con 0x5570998f2620 2023-10-05T19:30:22.407 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:21.966+0000 7fd29bc01e40 1 -- 172.21.15.167:0/444809523 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:101 6.4 6:22d26bf9:::data_loggenerations_metadata:head [watch watch cookie 93942101268080] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e32) v8 -- 0x557099935fb0 con 0x5570998f2620 2023-10-05T19:30:22.407 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:21.970+0000 7fd299f37700 1 -- 172.21.15.167:0/444809523 <== osd.1 v2:172.21.15.167:6800/30035 34 ==== osd_op_reply(101 data_loggenerations_metadata [watch watch cookie 93942101268080] v32'255 uv1 ondisk = 0) v8 ==== 172+0+0 (crc 0 0 0) 0x7fd29403d040 con 0x5570998f2620 2023-10-05T19:30:22.407 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:21.970+0000 7fd26a7e4700 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2023-10-05T19:30:22.407 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:21.970+0000 7fd29bc01e40 20 check_secure_mon_conn(): auth registy supported: methods=[2] modes=[2,1] 2023-10-05T19:30:22.408 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:21.970+0000 7fd29bc01e40 20 check_secure_mon_conn(): mode 1 is insecure 2023-10-05T19:30:22.408 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:21.970+0000 7fd29bc01e40 1 Processor -- start 2023-10-05T19:30:22.408 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:21.970+0000 7fd29bc01e40 1 -- start start 2023-10-05T19:30:22.408 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:21.970+0000 7fd29bc01e40 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5570999430a0 0x557099943470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-05T19:30:22.408 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:21.970+0000 7fd29bc01e40 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x5570999439b0 con 0x5570999430a0 2023-10-05T19:30:22.408 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:21.970+0000 7fd298f35700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5570999430a0 0x557099943470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-05T19:30:22.408 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:21.970+0000 7fd298f35700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5570999430a0 0x557099943470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.118:3300/0 says I am v2:172.21.15.167:33008/0 (socket says 172.21.15.167:33008) 2023-10-05T19:30:22.408 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:21.970+0000 7fd298f35700 1 -- 172.21.15.167:0/2927792302 learned_addr learned my addr 172.21.15.167:0/2927792302 (peer_addr_for_me v2:172.21.15.167:0/0) 2023-10-05T19:30:22.408 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:21.970+0000 7fd298f35700 1 -- 172.21.15.167:0/2927792302 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x557099943af0 con 0x5570999430a0 2023-10-05T19:30:22.408 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:21.970+0000 7fd298f35700 1 --2- 172.21.15.167:0/2927792302 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5570999430a0 0x557099943470 secure :-1 s=READY pgs=666 cs=0 l=1 rev1=1 crypto rx=0x557099927d20 tx=0x7fd284006000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=592c3576662479d2 server_cookie=0 in_seq=0 out_seq=0 2023-10-05T19:30:22.408 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:21.970+0000 7fd269fe3700 1 -- 172.21.15.167:0/2927792302 <== mon.0 v2:172.21.15.118:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fd28402c040 con 0x5570999430a0 2023-10-05T19:30:22.409 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:21.970+0000 7fd269fe3700 1 -- 172.21.15.167:0/2927792302 <== mon.0 v2:172.21.15.118:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd28402d040 con 0x5570999430a0 2023-10-05T19:30:22.409 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:21.970+0000 7fd269fe3700 1 -- 172.21.15.167:0/2927792302 <== mon.0 v2:172.21.15.118:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fd284028ca0 con 0x5570999430a0 2023-10-05T19:30:22.409 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:21.974+0000 7fd29bc01e40 1 -- 172.21.15.167:0/2927792302 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5570999430a0 msgr2=0x557099943470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-05T19:30:22.409 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:21.974+0000 7fd29bc01e40 1 --2- 172.21.15.167:0/2927792302 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5570999430a0 0x557099943470 secure :-1 s=READY pgs=666 cs=0 l=1 rev1=1 crypto rx=0x557099927d20 tx=0x7fd284006000 comp rx=0 tx=0).stop 2023-10-05T19:30:22.409 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:21.974+0000 7fd29bc01e40 1 -- 172.21.15.167:0/2927792302 shutdown_connections 2023-10-05T19:30:22.409 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:21.974+0000 7fd29bc01e40 1 --2- 172.21.15.167:0/2927792302 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5570999430a0 0x557099943470 unknown :-1 s=CLOSED pgs=666 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-05T19:30:22.409 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:21.974+0000 7fd29bc01e40 1 -- 172.21.15.167:0/2927792302 >> 172.21.15.167:0/2927792302 conn(0x55709993e460 msgr2=0x55709993c5b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-05T19:30:22.409 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:21.974+0000 7fd29bc01e40 1 -- 172.21.15.167:0/2927792302 shutdown_connections 2023-10-05T19:30:22.409 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:21.974+0000 7fd29bc01e40 1 -- 172.21.15.167:0/2927792302 wait complete. 2023-10-05T19:30:22.409 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:21.974+0000 7fd29bc01e40 1 Processor -- start 2023-10-05T19:30:22.409 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:21.974+0000 7fd29bc01e40 1 -- start start 2023-10-05T19:30:22.410 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:21.974+0000 7fd29bc01e40 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5570999430a0 0x557099948c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-05T19:30:22.410 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:21.974+0000 7fd29bc01e40 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x557099949c90 con 0x5570999430a0 2023-10-05T19:30:22.410 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:21.974+0000 7fd298f35700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5570999430a0 0x557099948c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-05T19:30:22.410 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:21.974+0000 7fd298f35700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5570999430a0 0x557099948c90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.118:3300/0 says I am v2:172.21.15.167:33018/0 (socket says 172.21.15.167:33018) 2023-10-05T19:30:22.410 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:21.974+0000 7fd298f35700 1 -- 172.21.15.167:0/832409700 learned_addr learned my addr 172.21.15.167:0/832409700 (peer_addr_for_me v2:172.21.15.167:0/0) 2023-10-05T19:30:22.410 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:21.974+0000 7fd298f35700 1 -- 172.21.15.167:0/832409700 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55709983fbd0 con 0x5570999430a0 2023-10-05T19:30:22.410 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:21.974+0000 7fd298f35700 1 --2- 172.21.15.167:0/832409700 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5570999430a0 0x557099948c90 secure :-1 s=READY pgs=667 cs=0 l=1 rev1=1 crypto rx=0x7fd284041040 tx=0x7fd28403c000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-05T19:30:22.410 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:21.974+0000 7fd1dbfff700 1 -- 172.21.15.167:0/832409700 <== mon.0 v2:172.21.15.118:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fd28402c070 con 0x5570999430a0 2023-10-05T19:30:22.410 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:21.974+0000 7fd1dbfff700 1 -- 172.21.15.167:0/832409700 <== mon.0 v2:172.21.15.118:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd28401c040 con 0x5570999430a0 2023-10-05T19:30:22.410 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:21.974+0000 7fd29bc01e40 1 -- 172.21.15.167:0/832409700 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x557099949290 con 0x5570999430a0 2023-10-05T19:30:22.410 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:21.974+0000 7fd1dbfff700 1 -- 172.21.15.167:0/832409700 <== mon.0 v2:172.21.15.118:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fd284048410 con 0x5570999430a0 2023-10-05T19:30:22.411 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:21.974+0000 7fd29bc01e40 1 -- 172.21.15.167:0/832409700 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x5570999496d0 con 0x5570999430a0 2023-10-05T19:30:22.411 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:21.978+0000 7fd1dbfff700 1 -- 172.21.15.167:0/832409700 <== mon.0 v2:172.21.15.118:3300/0 4 ==== mgrmap(e 5) v1 ==== 40329+0+0 (secure 0 0 0) 0x7fd284045050 con 0x5570999430a0 2023-10-05T19:30:22.411 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:21.978+0000 7fd1dbfff700 1 --2- 172.21.15.167:0/832409700 >> [v2:172.21.15.167:6824/29889,v1:172.21.15.167:6825/29889] conn(0x7fd1e0030ca0 0x7fd1e0033150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-05T19:30:22.411 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:21.978+0000 7fd1dbfff700 1 -- 172.21.15.167:0/832409700 <== mon.0 v2:172.21.15.118:3300/0 5 ==== osd_map(32..32 src has 1..32) v4 ==== 6260+0+0 (secure 0 0 0) 0x7fd28400b0e0 con 0x5570999430a0 2023-10-05T19:30:22.411 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:21.978+0000 7fd299f37700 1 --2- 172.21.15.167:0/832409700 >> [v2:172.21.15.167:6824/29889,v1:172.21.15.167:6825/29889] conn(0x7fd1e0030ca0 0x7fd1e0033150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-05T19:30:22.411 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:21.978+0000 7fd29bc01e40 1 --2- 172.21.15.167:0/832409700 >> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] conn(0x557099953820 0x557099955c30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-05T19:30:22.411 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:21.978+0000 7fd29bc01e40 1 -- 172.21.15.167:0/832409700 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:1 6.19 6:9a6dd8c0:gc::gc.0:head [create,call version.check_conds in=74b,call rgw_gc.rgw_gc_queue_init in=45b,call version.set in=34b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e32) v8 -- 0x557099956280 con 0x557099953820 2023-10-05T19:30:22.411 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:21.978+0000 7fd299736700 1 --2- 172.21.15.167:0/832409700 >> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] conn(0x557099953820 0x557099955c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-05T19:30:22.411 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:21.978+0000 7fd299f37700 1 --2- 172.21.15.167:0/832409700 >> [v2:172.21.15.167:6824/29889,v1:172.21.15.167:6825/29889] conn(0x7fd1e0030ca0 0x7fd1e0033150 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x557099935230 tx=0x7fd29400e040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-05T19:30:22.411 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:21.978+0000 7fd299736700 1 --2- 172.21.15.167:0/832409700 >> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] conn(0x557099953820 0x557099955c30 crc :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-05T19:30:22.411 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:21.982+0000 7fd299736700 1 -- 172.21.15.167:0/832409700 <== osd.1 v2:172.21.15.167:6800/30035 1 ==== osd_op_reply(1 gc.0 [create,call,call,call] v32'40 uv33 ondisk = -125 ((125) Operation canceled)) v8 ==== 274+0+0 (crc 0 0 0) 0x7fd28c022020 con 0x557099953820 2023-10-05T19:30:22.411 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:21.982+0000 7fd29bc01e40 1 --2- 172.21.15.167:0/832409700 >> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] conn(0x557099956f50 0x557099959330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-05T19:30:22.412 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:21.982+0000 7fd29bc01e40 1 -- 172.21.15.167:0/832409700 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:2 6.18 6:1e85421f:gc::gc.1:head [create,call version.check_conds in=74b,call rgw_gc.rgw_gc_queue_init in=45b,call version.set in=34b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e32) v8 -- 0x557099959a60 con 0x557099956f50 2023-10-05T19:30:22.412 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:21.982+0000 7fd298f35700 1 --2- 172.21.15.167:0/832409700 >> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] conn(0x557099956f50 0x557099959330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-05T19:30:22.412 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:21.982+0000 7fd298f35700 1 --2- 172.21.15.167:0/832409700 >> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] conn(0x557099956f50 0x557099959330 crc :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-05T19:30:22.412 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:21.986+0000 7fd298f35700 1 -- 172.21.15.167:0/832409700 <== osd.0 v2:172.21.15.118:6800/27690 1 ==== osd_op_reply(2 gc.1 [create,call,call,call] v32'33 uv25 ondisk = -125 ((125) Operation canceled)) v8 ==== 274+0+0 (crc 0 0 0) 0x7fd28402f040 con 0x557099956f50 2023-10-05T19:30:22.412 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:21.986+0000 7fd29bc01e40 1 --2- 172.21.15.167:0/832409700 >> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] conn(0x55709994f4c0 0x55709995d0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-05T19:30:22.412 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:21.986+0000 7fd29bc01e40 1 -- 172.21.15.167:0/832409700 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:3 6.b 6:d242335b:gc::gc.2:head [create,call version.check_conds in=74b,call rgw_gc.rgw_gc_queue_init in=45b,call version.set in=34b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e32) v8 -- 0x55709995d610 con 0x55709994f4c0 2023-10-05T19:30:22.412 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:21.986+0000 7fd299f37700 1 --2- 172.21.15.167:0/832409700 >> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] conn(0x55709994f4c0 0x55709995d0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-05T19:30:22.412 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:21.986+0000 7fd299f37700 1 --2- 172.21.15.167:0/832409700 >> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] conn(0x55709994f4c0 0x55709995d0d0 crc :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-05T19:30:22.412 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:21.990+0000 7fd299f37700 1 -- 172.21.15.167:0/832409700 <== osd.3 v2:172.21.15.167:6808/30036 1 ==== osd_op_reply(3 gc.2 [create,call,call,call] v32'28 uv21 ondisk = -125 ((125) Operation canceled)) v8 ==== 274+0+0 (crc 0 0 0) 0x7fd29403d040 con 0x55709994f4c0 2023-10-05T19:30:22.412 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:21.990+0000 7fd29bc01e40 1 -- 172.21.15.167:0/832409700 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:4 6.4 6:2166d8e1:gc::gc.3:head [create,call version.check_conds in=74b,call rgw_gc.rgw_gc_queue_init in=45b,call version.set in=34b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e32) v8 -- 0x55709995dc60 con 0x557099953820 2023-10-05T19:30:22.412 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:21.990+0000 7fd299736700 1 -- 172.21.15.167:0/832409700 <== osd.1 v2:172.21.15.167:6800/30035 2 ==== osd_op_reply(4 gc.3 [create,call,call,call] v32'256 uv184 ondisk = -125 ((125) Operation canceled)) v8 ==== 274+0+0 (crc 0 0 0) 0x7fd28c022020 con 0x557099953820 2023-10-05T19:30:22.412 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:21.990+0000 7fd29bc01e40 1 --2- 172.21.15.167:0/832409700 >> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] conn(0x55709995e900 0x557099960ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-05T19:30:22.413 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:21.990+0000 7fd29bc01e40 1 -- 172.21.15.167:0/832409700 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:5 6.3 6:c60da066:gc::gc.4:head [create,call version.check_conds in=74b,call rgw_gc.rgw_gc_queue_init in=45b,call version.set in=34b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e32) v8 -- 0x557099961390 con 0x55709995e900 2023-10-05T19:30:22.413 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:21.990+0000 7fd299736700 1 --2- 172.21.15.167:0/832409700 >> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] conn(0x55709995e900 0x557099960ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-05T19:30:22.413 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:21.994+0000 7fd299736700 1 --2- 172.21.15.167:0/832409700 >> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] conn(0x55709995e900 0x557099960ce0 crc :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-05T19:30:22.413 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:21.994+0000 7fd299736700 1 -- 172.21.15.167:0/832409700 <== osd.2 v2:172.21.15.167:6816/30037 1 ==== osd_op_reply(5 gc.4 [create,call,call,call] v32'74 uv51 ondisk = -125 ((125) Operation canceled)) v8 ==== 274+0+0 (crc 0 0 0) 0x7fd28c022020 con 0x55709995e900 2023-10-05T19:30:22.413 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:21.994+0000 7fd29bc01e40 1 -- 172.21.15.167:0/832409700 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:6 6.9 6:954be8d2:gc::gc.5:head [create,call version.check_conds in=74b,call rgw_gc.rgw_gc_queue_init in=45b,call version.set in=34b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e32) v8 -- 0x5570999619b0 con 0x55709994f4c0 2023-10-05T19:30:22.413 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:21.998+0000 7fd299f37700 1 -- 172.21.15.167:0/832409700 <== osd.3 v2:172.21.15.167:6808/30036 2 ==== osd_op_reply(6 gc.5 [create,call,call,call] v32'52 uv37 ondisk = -125 ((125) Operation canceled)) v8 ==== 274+0+0 (crc 0 0 0) 0x7fd29403d040 con 0x55709994f4c0 2023-10-05T19:30:22.413 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:21.998+0000 7fd29bc01e40 1 -- 172.21.15.167:0/832409700 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:7 6.e 6:748722ff:gc::gc.6:head [create,call version.check_conds in=74b,call rgw_gc.rgw_gc_queue_init in=45b,call version.set in=34b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e32) v8 -- 0x557099961fd0 con 0x557099953820 2023-10-05T19:30:22.413 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:21.998+0000 7fd299736700 1 -- 172.21.15.167:0/832409700 <== osd.1 v2:172.21.15.167:6800/30035 3 ==== osd_op_reply(7 gc.6 [create,call,call,call] v32'30 uv23 ondisk = -125 ((125) Operation canceled)) v8 ==== 274+0+0 (crc 0 0 0) 0x7fd28c022020 con 0x557099953820 2023-10-05T19:30:22.413 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:21.998+0000 7fd29bc01e40 1 -- 172.21.15.167:0/832409700 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:8 6.1d 6:beaa4be4:gc::gc.7:head [create,call version.check_conds in=74b,call rgw_gc.rgw_gc_queue_init in=45b,call version.set in=34b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e32) v8 -- 0x5570999624c0 con 0x557099956f50 2023-10-05T19:30:22.413 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.002+0000 7fd298f35700 1 -- 172.21.15.167:0/832409700 <== osd.0 v2:172.21.15.118:6800/27690 2 ==== osd_op_reply(8 gc.7 [create,call,call,call] v32'49 uv34 ondisk = -125 ((125) Operation canceled)) v8 ==== 274+0+0 (crc 0 0 0) 0x7fd28402f040 con 0x557099956f50 2023-10-05T19:30:22.413 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.002+0000 7fd29bc01e40 1 -- 172.21.15.167:0/832409700 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:9 6.12 6:4f78e9a2:gc::gc.8:head [create,call version.check_conds in=74b,call rgw_gc.rgw_gc_queue_init in=45b,call version.set in=34b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e32) v8 -- 0x5570999523d0 con 0x55709995e900 2023-10-05T19:30:22.413 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.006+0000 7fd299736700 1 -- 172.21.15.167:0/832409700 <== osd.2 v2:172.21.15.167:6816/30037 2 ==== osd_op_reply(9 gc.8 [create,call,call,call] v32'28 uv21 ondisk = -125 ((125) Operation canceled)) v8 ==== 274+0+0 (crc 0 0 0) 0x7fd28c022020 con 0x55709995e900 2023-10-05T19:30:22.414 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.006+0000 7fd29bc01e40 1 -- 172.21.15.167:0/832409700 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:10 6.1c 6:3e426232:gc::gc.9:head [create,call version.check_conds in=74b,call rgw_gc.rgw_gc_queue_init in=45b,call version.set in=34b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e32) v8 -- 0x5570999523d0 con 0x55709994f4c0 2023-10-05T19:30:22.414 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.006+0000 7fd299f37700 1 -- 172.21.15.167:0/832409700 <== osd.3 v2:172.21.15.167:6808/30036 3 ==== osd_op_reply(10 gc.9 [create,call,call,call] v32'49 uv39 ondisk = -125 ((125) Operation canceled)) v8 ==== 274+0+0 (crc 0 0 0) 0x7fd29403d040 con 0x55709994f4c0 2023-10-05T19:30:22.414 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.006+0000 7fd29bc01e40 1 -- 172.21.15.167:0/832409700 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:11 6.3 6:c31f41d7:gc::gc.10:head [create,call version.check_conds in=74b,call rgw_gc.rgw_gc_queue_init in=45b,call version.set in=34b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e32) v8 -- 0x557099952640 con 0x55709995e900 2023-10-05T19:30:22.414 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.010+0000 7fd299736700 1 -- 172.21.15.167:0/832409700 <== osd.2 v2:172.21.15.167:6816/30037 3 ==== osd_op_reply(11 gc.10 [create,call,call,call] v32'75 uv53 ondisk = -125 ((125) Operation canceled)) v8 ==== 275+0+0 (crc 0 0 0) 0x7fd28c022020 con 0x55709995e900 2023-10-05T19:30:22.414 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.010+0000 7fd29bc01e40 1 -- 172.21.15.167:0/832409700 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:12 6.13 6:c8333aab:gc::gc.11:head [create,call version.check_conds in=74b,call rgw_gc.rgw_gc_queue_init in=45b,call version.set in=34b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e32) v8 -- 0x557099952640 con 0x557099956f50 2023-10-05T19:30:22.414 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.014+0000 7fd298f35700 1 -- 172.21.15.167:0/832409700 <== osd.0 v2:172.21.15.118:6800/27690 3 ==== osd_op_reply(12 gc.11 [create,call,call,call] v32'35 uv25 ondisk = -125 ((125) Operation canceled)) v8 ==== 275+0+0 (crc 0 0 0) 0x7fd28402f040 con 0x557099956f50 2023-10-05T19:30:22.414 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.014+0000 7fd29bc01e40 1 -- 172.21.15.167:0/832409700 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:13 6.15 6:ab569f81:gc::gc.12:head [create,call version.check_conds in=74b,call rgw_gc.rgw_gc_queue_init in=45b,call version.set in=34b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e32) v8 -- 0x557099962fe0 con 0x55709995e900 2023-10-05T19:30:22.414 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.018+0000 7fd299736700 1 -- 172.21.15.167:0/832409700 <== osd.2 v2:172.21.15.167:6816/30037 4 ==== osd_op_reply(13 gc.12 [create,call,call,call] v32'28 uv21 ondisk = -125 ((125) Operation canceled)) v8 ==== 275+0+0 (crc 0 0 0) 0x7fd28c022020 con 0x55709995e900 2023-10-05T19:30:22.414 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.018+0000 7fd29bc01e40 1 -- 172.21.15.167:0/832409700 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:14 6.0 6:0667bb13:gc::gc.13:head [create,call version.check_conds in=74b,call rgw_gc.rgw_gc_queue_init in=45b,call version.set in=34b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e32) v8 -- 0x557099962fe0 con 0x557099956f50 2023-10-05T19:30:22.414 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.018+0000 7fd298f35700 1 -- 172.21.15.167:0/832409700 <== osd.0 v2:172.21.15.118:6800/27690 4 ==== osd_op_reply(14 gc.13 [create,call,call,call] v32'55 uv40 ondisk = -125 ((125) Operation canceled)) v8 ==== 275+0+0 (crc 0 0 0) 0x7fd28402f040 con 0x557099956f50 2023-10-05T19:30:22.414 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.018+0000 7fd29bc01e40 1 -- 172.21.15.167:0/832409700 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:15 6.4 6:227f63b5:gc::gc.14:head [create,call version.check_conds in=74b,call rgw_gc.rgw_gc_queue_init in=45b,call version.set in=34b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e32) v8 -- 0x5570999634d0 con 0x557099953820 2023-10-05T19:30:22.415 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.022+0000 7fd299736700 1 -- 172.21.15.167:0/832409700 <== osd.1 v2:172.21.15.167:6800/30035 4 ==== osd_op_reply(15 gc.14 [create,call,call,call] v32'257 uv186 ondisk = -125 ((125) Operation canceled)) v8 ==== 275+0+0 (crc 0 0 0) 0x7fd28c022020 con 0x557099953820 2023-10-05T19:30:22.415 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.022+0000 7fd29bc01e40 1 -- 172.21.15.167:0/832409700 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:16 6.7 6:e07a6548:gc::gc.15:head [create,call version.check_conds in=74b,call rgw_gc.rgw_gc_queue_init in=45b,call version.set in=34b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e32) v8 -- 0x5570999634d0 con 0x55709994f4c0 2023-10-05T19:30:22.415 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.026+0000 7fd299f37700 1 -- 172.21.15.167:0/832409700 <== osd.3 v2:172.21.15.167:6808/30036 4 ==== osd_op_reply(16 gc.15 [create,call,call,call] v32'71 uv45 ondisk = -125 ((125) Operation canceled)) v8 ==== 275+0+0 (crc 0 0 0) 0x7fd29403d040 con 0x55709994f4c0 2023-10-05T19:30:22.415 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.026+0000 7fd29bc01e40 1 -- 172.21.15.167:0/832409700 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:17 6.3 6:c623b173:gc::gc.16:head [create,call version.check_conds in=74b,call rgw_gc.rgw_gc_queue_init in=45b,call version.set in=34b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e32) v8 -- 0x5570999634d0 con 0x55709995e900 2023-10-05T19:30:22.415 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.030+0000 7fd299736700 1 -- 172.21.15.167:0/832409700 <== osd.2 v2:172.21.15.167:6816/30037 5 ==== osd_op_reply(17 gc.16 [create,call,call,call] v32'76 uv55 ondisk = -125 ((125) Operation canceled)) v8 ==== 275+0+0 (crc 0 0 0) 0x7fd28c022020 con 0x55709995e900 2023-10-05T19:30:22.415 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.030+0000 7fd29bc01e40 1 -- 172.21.15.167:0/832409700 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:18 6.16 6:6849fd06:gc::gc.17:head [create,call version.check_conds in=74b,call rgw_gc.rgw_gc_queue_init in=45b,call version.set in=34b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e32) v8 -- 0x5570999634d0 con 0x55709994f4c0 2023-10-05T19:30:22.415 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.030+0000 7fd299f37700 1 -- 172.21.15.167:0/832409700 <== osd.3 v2:172.21.15.167:6808/30036 5 ==== osd_op_reply(18 gc.17 [create,call,call,call] v32'28 uv21 ondisk = -125 ((125) Operation canceled)) v8 ==== 275+0+0 (crc 0 0 0) 0x7fd29403d040 con 0x55709994f4c0 2023-10-05T19:30:22.415 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.030+0000 7fd29bc01e40 1 -- 172.21.15.167:0/832409700 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:19 6.d 6:b019ea83:gc::gc.18:head [create,call version.check_conds in=74b,call rgw_gc.rgw_gc_queue_init in=45b,call version.set in=34b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e32) v8 -- 0x5570999634d0 con 0x557099953820 2023-10-05T19:30:22.415 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.034+0000 7fd299736700 1 -- 172.21.15.167:0/832409700 <== osd.1 v2:172.21.15.167:6800/30035 5 ==== osd_op_reply(19 gc.18 [create,call,call,call] v32'36 uv29 ondisk = -125 ((125) Operation canceled)) v8 ==== 275+0+0 (crc 0 0 0) 0x7fd28c022020 con 0x557099953820 2023-10-05T19:30:22.415 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.034+0000 7fd29bc01e40 1 -- 172.21.15.167:0/832409700 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:20 6.4 6:24c8e1ae:gc::gc.19:head [create,call version.check_conds in=74b,call rgw_gc.rgw_gc_queue_init in=45b,call version.set in=34b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e32) v8 -- 0x5570999634d0 con 0x557099953820 2023-10-05T19:30:22.415 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.034+0000 7fd299736700 1 -- 172.21.15.167:0/832409700 <== osd.1 v2:172.21.15.167:6800/30035 6 ==== osd_op_reply(20 gc.19 [create,call,call,call] v32'258 uv189 ondisk = -125 ((125) Operation canceled)) v8 ==== 275+0+0 (crc 0 0 0) 0x7fd28c022020 con 0x557099953820 2023-10-05T19:30:22.416 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.038+0000 7fd29bc01e40 1 -- 172.21.15.167:0/832409700 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:21 6.7 6:e622f76a:gc::gc.20:head [create,call version.check_conds in=74b,call rgw_gc.rgw_gc_queue_init in=45b,call version.set in=34b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e32) v8 -- 0x5570999634d0 con 0x55709994f4c0 2023-10-05T19:30:22.416 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.038+0000 7fd299f37700 1 -- 172.21.15.167:0/832409700 <== osd.3 v2:172.21.15.167:6808/30036 6 ==== osd_op_reply(21 gc.20 [create,call,call,call] v32'72 uv47 ondisk = -125 ((125) Operation canceled)) v8 ==== 275+0+0 (crc 0 0 0) 0x7fd29403d040 con 0x55709994f4c0 2023-10-05T19:30:22.416 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.038+0000 7fd29bc01e40 1 -- 172.21.15.167:0/832409700 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:22 6.a 6:5127e642:gc::gc.21:head [create,call version.check_conds in=74b,call rgw_gc.rgw_gc_queue_init in=45b,call version.set in=34b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e32) v8 -- 0x5570999634d0 con 0x55709995e900 2023-10-05T19:30:22.416 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.042+0000 7fd299736700 1 -- 172.21.15.167:0/832409700 <== osd.2 v2:172.21.15.167:6816/30037 6 ==== osd_op_reply(22 gc.21 [create,call,call,call] v32'38 uv31 ondisk = -125 ((125) Operation canceled)) v8 ==== 275+0+0 (crc 0 0 0) 0x7fd28c022020 con 0x55709995e900 2023-10-05T19:30:22.416 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.042+0000 7fd29bc01e40 1 -- 172.21.15.167:0/832409700 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:23 6.11 6:8eb99c53:gc::gc.22:head [create,call version.check_conds in=74b,call rgw_gc.rgw_gc_queue_init in=45b,call version.set in=34b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e32) v8 -- 0x5570999634d0 con 0x55709994f4c0 2023-10-05T19:30:22.416 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.042+0000 7fd299f37700 1 -- 172.21.15.167:0/832409700 <== osd.3 v2:172.21.15.167:6808/30036 7 ==== osd_op_reply(23 gc.22 [create,call,call,call] v32'34 uv23 ondisk = -125 ((125) Operation canceled)) v8 ==== 275+0+0 (crc 0 0 0) 0x7fd29403d040 con 0x55709994f4c0 2023-10-05T19:30:22.416 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.042+0000 7fd29bc01e40 1 -- 172.21.15.167:0/832409700 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:24 6.8 6:17d7e8ed:gc::gc.23:head [create,call version.check_conds in=74b,call rgw_gc.rgw_gc_queue_init in=45b,call version.set in=34b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e32) v8 -- 0x5570999634d0 con 0x55709994f4c0 2023-10-05T19:30:22.416 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.046+0000 7fd299f37700 1 -- 172.21.15.167:0/832409700 <== osd.3 v2:172.21.15.167:6808/30036 8 ==== osd_op_reply(24 gc.23 [create,call,call,call] v32'41 uv33 ondisk = -125 ((125) Operation canceled)) v8 ==== 275+0+0 (crc 0 0 0) 0x7fd29403d040 con 0x55709994f4c0 2023-10-05T19:30:22.416 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.046+0000 7fd29bc01e40 1 -- 172.21.15.167:0/832409700 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:25 6.1b 6:db9d130d:gc::gc.24:head [create,call version.check_conds in=74b,call rgw_gc.rgw_gc_queue_init in=45b,call version.set in=34b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e32) v8 -- 0x5570999634d0 con 0x55709994f4c0 2023-10-05T19:30:22.416 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.050+0000 7fd299f37700 1 -- 172.21.15.167:0/832409700 <== osd.3 v2:172.21.15.167:6808/30036 9 ==== osd_op_reply(25 gc.24 [create,call,call,call] v32'24 uv17 ondisk = -125 ((125) Operation canceled)) v8 ==== 275+0+0 (crc 0 0 0) 0x7fd29403d040 con 0x55709994f4c0 2023-10-05T19:30:22.416 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.050+0000 7fd29bc01e40 1 -- 172.21.15.167:0/832409700 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:26 6.9 6:921ecdb1:gc::gc.25:head [create,call version.check_conds in=74b,call rgw_gc.rgw_gc_queue_init in=45b,call version.set in=34b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e32) v8 -- 0x5570999634d0 con 0x55709994f4c0 2023-10-05T19:30:22.417 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.050+0000 7fd299f37700 1 -- 172.21.15.167:0/832409700 <== osd.3 v2:172.21.15.167:6808/30036 10 ==== osd_op_reply(26 gc.25 [create,call,call,call] v32'53 uv39 ondisk = -125 ((125) Operation canceled)) v8 ==== 275+0+0 (crc 0 0 0) 0x7fd29403d040 con 0x55709994f4c0 2023-10-05T19:30:22.417 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.050+0000 7fd29bc01e40 1 -- 172.21.15.167:0/832409700 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:27 6.1f 6:fde4dd55:gc::gc.26:head [create,call version.check_conds in=74b,call rgw_gc.rgw_gc_queue_init in=45b,call version.set in=34b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e32) v8 -- 0x5570999634d0 con 0x557099956f50 2023-10-05T19:30:22.417 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.054+0000 7fd298f35700 1 -- 172.21.15.167:0/832409700 <== osd.0 v2:172.21.15.118:6800/27690 5 ==== osd_op_reply(27 gc.26 [create,call,call,call] v32'44 uv23 ondisk = -125 ((125) Operation canceled)) v8 ==== 275+0+0 (crc 0 0 0) 0x7fd28402f040 con 0x557099956f50 2023-10-05T19:30:22.417 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.054+0000 7fd29bc01e40 1 -- 172.21.15.167:0/832409700 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:28 6.1a 6:5909e671:gc::gc.27:head [create,call version.check_conds in=74b,call rgw_gc.rgw_gc_queue_init in=45b,call version.set in=34b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e32) v8 -- 0x5570999634d0 con 0x55709994f4c0 2023-10-05T19:30:22.417 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.058+0000 7fd299f37700 1 -- 172.21.15.167:0/832409700 <== osd.3 v2:172.21.15.167:6808/30036 11 ==== osd_op_reply(28 gc.27 [create,call,call,call] v32'32 uv21 ondisk = -125 ((125) Operation canceled)) v8 ==== 275+0+0 (crc 0 0 0) 0x7fd29403d040 con 0x55709994f4c0 2023-10-05T19:30:22.417 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.058+0000 7fd29bc01e40 1 -- 172.21.15.167:0/832409700 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:29 6.1d 6:b88e3271:gc::gc.28:head [create,call version.check_conds in=74b,call rgw_gc.rgw_gc_queue_init in=45b,call version.set in=34b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e32) v8 -- 0x5570999634d0 con 0x557099956f50 2023-10-05T19:30:22.417 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.058+0000 7fd298f35700 1 -- 172.21.15.167:0/832409700 <== osd.0 v2:172.21.15.118:6800/27690 6 ==== osd_op_reply(29 gc.28 [create,call,call,call] v32'50 uv36 ondisk = -125 ((125) Operation canceled)) v8 ==== 275+0+0 (crc 0 0 0) 0x7fd28402f040 con 0x557099956f50 2023-10-05T19:30:22.417 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.058+0000 7fd29bc01e40 1 -- 172.21.15.167:0/832409700 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:30 6.7 6:e47c9322:gc::gc.29:head [create,call version.check_conds in=74b,call rgw_gc.rgw_gc_queue_init in=45b,call version.set in=34b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e32) v8 -- 0x5570999634d0 con 0x55709994f4c0 2023-10-05T19:30:22.417 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.062+0000 7fd299f37700 1 -- 172.21.15.167:0/832409700 <== osd.3 v2:172.21.15.167:6808/30036 12 ==== osd_op_reply(30 gc.29 [create,call,call,call] v32'73 uv49 ondisk = -125 ((125) Operation canceled)) v8 ==== 275+0+0 (crc 0 0 0) 0x7fd29403d040 con 0x55709994f4c0 2023-10-05T19:30:22.417 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.062+0000 7fd29bc01e40 1 -- 172.21.15.167:0/832409700 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:31 6.0 6:013d6c33:gc::gc.30:head [create,call version.check_conds in=74b,call rgw_gc.rgw_gc_queue_init in=45b,call version.set in=34b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e32) v8 -- 0x5570999634d0 con 0x557099956f50 2023-10-05T19:30:22.417 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.066+0000 7fd298f35700 1 -- 172.21.15.167:0/832409700 <== osd.0 v2:172.21.15.118:6800/27690 7 ==== osd_op_reply(31 gc.30 [create,call,call,call] v32'56 uv42 ondisk = -125 ((125) Operation canceled)) v8 ==== 275+0+0 (crc 0 0 0) 0x7fd28402f040 con 0x557099956f50 2023-10-05T19:30:22.418 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.066+0000 7fd29bc01e40 1 -- 172.21.15.167:0/832409700 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:32 6.f 6:f24cb4a8:gc::gc.31:head [create,call version.check_conds in=74b,call rgw_gc.rgw_gc_queue_init in=45b,call version.set in=34b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e32) v8 -- 0x5570999634d0 con 0x55709995e900 2023-10-05T19:30:22.418 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.070+0000 7fd299736700 1 -- 172.21.15.167:0/832409700 <== osd.2 v2:172.21.15.167:6816/30037 7 ==== osd_op_reply(32 gc.31 [create,call,call,call] v32'34 uv27 ondisk = -125 ((125) Operation canceled)) v8 ==== 275+0+0 (crc 0 0 0) 0x7fd28c022020 con 0x55709995e900 2023-10-05T19:30:22.418 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.070+0000 7fd1d97fa700 20 reqs_thread_entry: start 2023-10-05T19:30:22.418 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.074+0000 7fd29bc01e40 20 init_complete bucket index max shards: 11 2023-10-05T19:30:22.418 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.074+0000 7fd1d2ffd700 20 reqs_thread_entry: start 2023-10-05T19:30:22.418 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.074+0000 7fd29bc01e40 10 rgw notify: Started notification manager with: 1 workers 2023-10-05T19:30:22.418 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.074+0000 7fd1d17fa700 20 rgw notify: INFO: next queues processing will happen at: Thu Oct 5 19:30:52 2023 2023-10-05T19:30:22.418 INFO:teuthology.orchestra.run.smithi167.stderr: 2023-10-05T19:30:22.418 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.074+0000 7fd1d0ff9700 20 reqs_thread_entry: start 2023-10-05T19:30:22.418 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.074+0000 7fd29bc01e40 20 get_system_obj_state: rctx=0x7fff991e43a8 obj=default.rgw.meta:users.uid:foo state=0x5570997cc0b0 s->prefetch_data=0 2023-10-05T19:30:22.418 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.074+0000 7fd29bc01e40 10 cache get: name=default.rgw.meta+users.uid+foo : miss 2023-10-05T19:30:22.419 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.074+0000 7fd29bc01e40 1 -- 172.21.15.167:0/444809523 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:102 8.18 8:1e6ca5ed:users.uid::foo:head [call version.read in=11b,getxattrs,stat] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32) v8 -- 0x5570999c1ec0 con 0x5570998f2620 2023-10-05T19:30:22.419 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.074+0000 7fd299f37700 1 -- 172.21.15.167:0/444809523 <== osd.1 v2:172.21.15.167:6800/30035 35 ==== osd_op_reply(102 foo [call out=48b,getxattrs out=75b,stat out=16b] v0'0 uv15 ondisk = 0) v8 ==== 231+0+139 (crc 0 0 0) 0x7fd29403d040 con 0x5570998f2620 2023-10-05T19:30:22.419 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.074+0000 7fd29bc01e40 10 cache put: name=default.rgw.meta+users.uid+foo info.flags=0x16 2023-10-05T19:30:22.419 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.074+0000 7fd29bc01e40 10 adding default.rgw.meta+users.uid+foo to cache LRU end 2023-10-05T19:30:22.419 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.074+0000 7fd29bc01e40 10 updating xattr: name=ceph.objclass.version bl.length()=42 2023-10-05T19:30:22.419 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.074+0000 7fd29bc01e40 20 get_system_obj_state: s->obj_tag was set empty 2023-10-05T19:30:22.419 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.074+0000 7fd29bc01e40 20 Read xattr: user.rgw.idtag 2023-10-05T19:30:22.419 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.074+0000 7fd29bc01e40 10 cache get: name=default.rgw.meta+users.uid+foo : type miss (requested=0x13, cached=0x16) 2023-10-05T19:30:22.419 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.074+0000 7fd29bc01e40 20 rados->read ofs=0 len=0 2023-10-05T19:30:22.419 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.074+0000 7fd29bc01e40 1 -- 172.21.15.167:0/444809523 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:103 8.18 8:1e6ca5ed:users.uid::foo:head [call version.check_conds in=122b,call version.read in=11b,read 0~0,getxattrs] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32) v8 -- 0x5570999c4950 con 0x5570998f2620 2023-10-05T19:30:22.419 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.074+0000 7fd299f37700 1 -- 172.21.15.167:0/444809523 <== osd.1 v2:172.21.15.167:6800/30035 36 ==== osd_op_reply(103 foo [call,call out=48b,read 0~357 out=357b,getxattrs out=75b] v0'0 uv15 ondisk = 0) v8 ==== 273+0+480 (crc 0 0 0) 0x7fd29403d040 con 0x5570998f2620 2023-10-05T19:30:22.419 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.074+0000 7fd29bc01e40 20 rados_obj.operate() r=0 bl.length=357 2023-10-05T19:30:22.420 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.074+0000 7fd29bc01e40 10 cache put: name=default.rgw.meta+users.uid+foo info.flags=0x13 2023-10-05T19:30:22.420 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.074+0000 7fd29bc01e40 10 moving default.rgw.meta+users.uid+foo to cache LRU end 2023-10-05T19:30:22.420 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.074+0000 7fd29bc01e40 10 updating xattr: name=ceph.objclass.version bl.length()=42 2023-10-05T19:30:22.420 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.074+0000 7fd29bc01e40 1 -- 172.21.15.167:0/444809523 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:104 8.9 8:96d98480:users.uid::foo.buckets:head [call user.list_buckets in=34b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32) v8 -- 0x5570999c4950 con 0x7fd28c007030 2023-10-05T19:30:22.420 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.078+0000 7fd298f35700 1 -- 172.21.15.167:0/444809523 <== osd.2 v2:172.21.15.167:6816/30037 17 ==== osd_op_reply(104 foo.buckets [call out=183b] v0'0 uv44 ondisk = 0) v8 ==== 155+0+183 (crc 0 0 0) 0x7fd28402f040 con 0x7fd28c007030 2023-10-05T19:30:22.420 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.078+0000 7fd29bc01e40 20 get_system_obj_state: rctx=0x7fff991e4050 obj=default.rgw.meta:root:.bucket.meta.myfoo:9a9df965-6612-45d4-a45f-6e3c889bacc2.4223.6 state=0x5570997cc0b0 s->prefetch_data=0 2023-10-05T19:30:22.420 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.078+0000 7fd29bc01e40 10 cache get: name=default.rgw.meta+root+.bucket.meta.myfoo:9a9df965-6612-45d4-a45f-6e3c889bacc2.4223.6 : miss 2023-10-05T19:30:22.420 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.078+0000 7fd29bc01e40 1 -- 172.21.15.167:0/444809523 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:105 8.b 8:d4944359:root::.bucket.meta.myfoo%3a9a9df965-6612-45d4-a45f-6e3c889bacc2.4223.6:head [call version.read in=11b,getxattrs,stat] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32) v8 -- 0x5570999c4160 con 0x55709992a5c0 2023-10-05T19:30:22.420 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.078+0000 7fd299736700 1 -- 172.21.15.167:0/444809523 <== osd.3 v2:172.21.15.167:6808/30036 24 ==== osd_op_reply(105 .bucket.meta.myfoo:9a9df965-6612-45d4-a45f-6e3c889bacc2.4223.6 [call out=48b,getxattrs out=218b,stat out=16b] v0'0 uv4 ondisk = 0) v8 ==== 290+0+282 (crc 0 0 0) 0x7fd28c022020 con 0x55709992a5c0 2023-10-05T19:30:22.420 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.078+0000 7fd29bc01e40 10 cache put: name=default.rgw.meta+root+.bucket.meta.myfoo:9a9df965-6612-45d4-a45f-6e3c889bacc2.4223.6 info.flags=0x16 2023-10-05T19:30:22.420 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.078+0000 7fd29bc01e40 10 adding default.rgw.meta+root+.bucket.meta.myfoo:9a9df965-6612-45d4-a45f-6e3c889bacc2.4223.6 to cache LRU end 2023-10-05T19:30:22.420 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.078+0000 7fd29bc01e40 10 updating xattr: name=ceph.objclass.version bl.length()=42 2023-10-05T19:30:22.421 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.078+0000 7fd29bc01e40 10 updating xattr: name=user.rgw.acl bl.length()=123 2023-10-05T19:30:22.421 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.078+0000 7fd29bc01e40 20 get_system_obj_state: s->obj_tag was set empty 2023-10-05T19:30:22.421 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.078+0000 7fd29bc01e40 20 Read xattr: user.rgw.acl 2023-10-05T19:30:22.421 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.078+0000 7fd29bc01e40 20 Read xattr: user.rgw.idtag 2023-10-05T19:30:22.421 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.078+0000 7fd29bc01e40 10 cache get: name=default.rgw.meta+root+.bucket.meta.myfoo:9a9df965-6612-45d4-a45f-6e3c889bacc2.4223.6 : type miss (requested=0x13, cached=0x16) 2023-10-05T19:30:22.421 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.078+0000 7fd29bc01e40 20 rados->read ofs=0 len=0 2023-10-05T19:30:22.421 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.078+0000 7fd29bc01e40 1 -- 172.21.15.167:0/444809523 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:106 8.b 8:d4944359:root::.bucket.meta.myfoo%3a9a9df965-6612-45d4-a45f-6e3c889bacc2.4223.6:head [call version.check_conds in=122b,call version.read in=11b,read 0~0,getxattrs] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32) v8 -- 0x5570999c6820 con 0x55709992a5c0 2023-10-05T19:30:22.421 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.078+0000 7fd299736700 1 -- 172.21.15.167:0/444809523 <== osd.3 v2:172.21.15.167:6808/30036 25 ==== osd_op_reply(106 .bucket.meta.myfoo:9a9df965-6612-45d4-a45f-6e3c889bacc2.4223.6 [call,call out=48b,read 0~352 out=352b,getxattrs out=218b] v0'0 uv4 ondisk = 0) v8 ==== 332+0+618 (crc 0 0 0) 0x7fd28c022020 con 0x55709992a5c0 2023-10-05T19:30:22.421 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.078+0000 7fd29bc01e40 20 rados_obj.operate() r=0 bl.length=352 2023-10-05T19:30:22.421 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.078+0000 7fd29bc01e40 10 cache put: name=default.rgw.meta+root+.bucket.meta.myfoo:9a9df965-6612-45d4-a45f-6e3c889bacc2.4223.6 info.flags=0x13 2023-10-05T19:30:22.421 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.078+0000 7fd29bc01e40 10 moving default.rgw.meta+root+.bucket.meta.myfoo:9a9df965-6612-45d4-a45f-6e3c889bacc2.4223.6 to cache LRU end 2023-10-05T19:30:22.421 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.078+0000 7fd29bc01e40 10 updating xattr: name=ceph.objclass.version bl.length()=42 2023-10-05T19:30:22.422 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.078+0000 7fd29bc01e40 10 updating xattr: name=user.rgw.acl bl.length()=123 2023-10-05T19:30:22.422 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.078+0000 7fd29bc01e40 10 chain_cache_entry: cache_locator=default.rgw.meta+root+.bucket.meta.myfoo:9a9df965-6612-45d4-a45f-6e3c889bacc2.4223.6 2023-10-05T19:30:22.422 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.078+0000 7fd29bc01e40 10 int RGWRados::cls_bucket_list_unordered(const DoutPrefixProvider*, RGWBucketInfo&, int, const rgw_obj_index_key&, const string&, uint32_t, bool, std::vector&, bool*, rgw_obj_index_key*, optional_yield, RGWBucketListNameFilter) :myfoo[9a9df965-6612-45d4-a45f-6e3c889bacc2.4223.6]) start_after="", prefix="", shard_id=-1", num_entries=1100, list_versions=1, force_check_filter is unset 2023-10-05T19:30:22.422 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.078+0000 7fd29bc01e40 1 -- 172.21.15.167:0/444809523 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:107 4.3 4:c2471d59:::.dir.9a9df965-6612-45d4-a45f-6e3c889bacc2.4223.6.0:head [call rgw.bucket_list in=47b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32) v8 -- 0x557099a41c30 con 0x5570998f2620 2023-10-05T19:30:22.422 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.078+0000 7fd299f37700 1 -- 172.21.15.167:0/444809523 <== osd.1 v2:172.21.15.167:6800/30035 37 ==== osd_op_reply(107 .dir.9a9df965-6612-45d4-a45f-6e3c889bacc2.4223.6.0 [call out=85b] v0'0 uv1 ondisk = 0) v8 ==== 194+0+85 (crc 0 0 0) 0x7fd29403d040 con 0x5570998f2620 2023-10-05T19:30:22.422 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.078+0000 7fd29bc01e40 1 -- 172.21.15.167:0/444809523 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:108 4.1f 4:fac78d74:::.dir.9a9df965-6612-45d4-a45f-6e3c889bacc2.4223.6.1:head [call rgw.bucket_list in=47b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32) v8 -- 0x557099a41c30 con 0x55709992a5c0 2023-10-05T19:30:22.422 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.082+0000 7fd299736700 1 -- 172.21.15.167:0/444809523 <== osd.3 v2:172.21.15.167:6808/30036 26 ==== osd_op_reply(108 .dir.9a9df965-6612-45d4-a45f-6e3c889bacc2.4223.6.1 [call out=85b] v0'0 uv3 ondisk = 0) v8 ==== 194+0+85 (crc 0 0 0) 0x7fd28c022020 con 0x55709992a5c0 2023-10-05T19:30:22.422 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.082+0000 7fd29bc01e40 1 -- 172.21.15.167:0/444809523 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:109 4.3 4:c00d743b:::.dir.9a9df965-6612-45d4-a45f-6e3c889bacc2.4223.6.2:head [call rgw.bucket_list in=47b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32) v8 -- 0x557099a41c30 con 0x5570998f2620 2023-10-05T19:30:22.422 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.082+0000 7fd299f37700 1 -- 172.21.15.167:0/444809523 <== osd.1 v2:172.21.15.167:6800/30035 38 ==== osd_op_reply(109 .dir.9a9df965-6612-45d4-a45f-6e3c889bacc2.4223.6.2 [call out=85b] v0'0 uv2 ondisk = 0) v8 ==== 194+0+85 (crc 0 0 0) 0x7fd29403d040 con 0x5570998f2620 2023-10-05T19:30:22.422 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.082+0000 7fd29bc01e40 1 -- 172.21.15.167:0/444809523 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:110 4.2b 4:d72453a1:::.dir.9a9df965-6612-45d4-a45f-6e3c889bacc2.4223.6.3:head [call rgw.bucket_list in=47b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32) v8 -- 0x557099a41c30 con 0x5570998f2620 2023-10-05T19:30:22.422 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.082+0000 7fd299f37700 1 -- 172.21.15.167:0/444809523 <== osd.1 v2:172.21.15.167:6800/30035 39 ==== osd_op_reply(110 .dir.9a9df965-6612-45d4-a45f-6e3c889bacc2.4223.6.3 [call out=85b] v0'0 uv3 ondisk = 0) v8 ==== 194+0+85 (crc 0 0 0) 0x7fd29403d040 con 0x5570998f2620 2023-10-05T19:30:22.422 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.082+0000 7fd29bc01e40 1 -- 172.21.15.167:0/444809523 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:111 4.2a 4:576abad7:::.dir.9a9df965-6612-45d4-a45f-6e3c889bacc2.4223.6.4:head [call rgw.bucket_list in=47b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32) v8 -- 0x557099a41c30 con 0x5570998f2620 2023-10-05T19:30:22.423 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.082+0000 7fd299f37700 1 -- 172.21.15.167:0/444809523 <== osd.1 v2:172.21.15.167:6800/30035 40 ==== osd_op_reply(111 .dir.9a9df965-6612-45d4-a45f-6e3c889bacc2.4223.6.4 [call out=425b] v0'0 uv4 ondisk = 0) v8 ==== 194+0+425 (crc 0 0 0) 0x7fd29403d040 con 0x5570998f2620 2023-10-05T19:30:22.423 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.082+0000 7fd29bc01e40 10 cls_bucket_list_unordered: got 15f6f8dc036519d7fe15b39338f6e5db 2023-10-05T19:30:22.423 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.082+0000 7fd29bc01e40 1 -- 172.21.15.167:0/444809523 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:112 4.2d 4:b45bfada:::.dir.9a9df965-6612-45d4-a45f-6e3c889bacc2.4223.6.5:head [call rgw.bucket_list in=47b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32) v8 -- 0x557099a41c30 con 0x7fd284011040 2023-10-05T19:30:22.423 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.086+0000 7fd299f37700 1 -- 172.21.15.167:0/444809523 <== osd.0 v2:172.21.15.118:6800/27690 29 ==== osd_op_reply(112 .dir.9a9df965-6612-45d4-a45f-6e3c889bacc2.4223.6.5 [call out=85b] v0'0 uv7 ondisk = 0) v8 ==== 194+0+85 (crc 0 0 0) 0x7fd29403d040 con 0x7fd284011040 2023-10-05T19:30:22.423 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.086+0000 7fd29bc01e40 1 -- 172.21.15.167:0/444809523 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:113 4.33 4:ceb97e7a:::.dir.9a9df965-6612-45d4-a45f-6e3c889bacc2.4223.6.6:head [call rgw.bucket_list in=47b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32) v8 -- 0x557099a41c30 con 0x7fd284011040 2023-10-05T19:30:22.423 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.086+0000 7fd299f37700 1 -- 172.21.15.167:0/444809523 <== osd.0 v2:172.21.15.118:6800/27690 30 ==== osd_op_reply(113 .dir.9a9df965-6612-45d4-a45f-6e3c889bacc2.4223.6.6 [call out=85b] v0'0 uv5 ondisk = 0) v8 ==== 194+0+85 (crc 0 0 0) 0x7fd29403d040 con 0x7fd284011040 2023-10-05T19:30:22.423 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.086+0000 7fd29bc01e40 1 -- 172.21.15.167:0/444809523 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:114 4.5 4:a170829b:::.dir.9a9df965-6612-45d4-a45f-6e3c889bacc2.4223.6.7:head [call rgw.bucket_list in=47b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32) v8 -- 0x557099a41c30 con 0x55709992a5c0 2023-10-05T19:30:22.423 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.086+0000 7fd299736700 1 -- 172.21.15.167:0/444809523 <== osd.3 v2:172.21.15.167:6808/30036 27 ==== osd_op_reply(114 .dir.9a9df965-6612-45d4-a45f-6e3c889bacc2.4223.6.7 [call out=85b] v0'0 uv3 ondisk = 0) v8 ==== 194+0+85 (crc 0 0 0) 0x7fd28c022020 con 0x55709992a5c0 2023-10-05T19:30:22.423 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.086+0000 7fd29bc01e40 1 -- 172.21.15.167:0/444809523 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:115 4.8 4:13a6ed6b:::.dir.9a9df965-6612-45d4-a45f-6e3c889bacc2.4223.6.8:head [call rgw.bucket_list in=47b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32) v8 -- 0x557099a41c30 con 0x55709992a5c0 2023-10-05T19:30:22.423 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.086+0000 7fd299736700 1 -- 172.21.15.167:0/444809523 <== osd.3 v2:172.21.15.167:6808/30036 28 ==== osd_op_reply(115 .dir.9a9df965-6612-45d4-a45f-6e3c889bacc2.4223.6.8 [call out=85b] v0'0 uv7 ondisk = 0) v8 ==== 194+0+85 (crc 0 0 0) 0x7fd28c022020 con 0x55709992a5c0 2023-10-05T19:30:22.423 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.086+0000 7fd29bc01e40 1 -- 172.21.15.167:0/444809523 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:116 4.7 4:e2c155b5:::.dir.9a9df965-6612-45d4-a45f-6e3c889bacc2.4223.6.9:head [call rgw.bucket_list in=47b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32) v8 -- 0x557099a41c30 con 0x7fd284011040 2023-10-05T19:30:22.424 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.090+0000 7fd299f37700 1 -- 172.21.15.167:0/444809523 <== osd.0 v2:172.21.15.118:6800/27690 31 ==== osd_op_reply(116 .dir.9a9df965-6612-45d4-a45f-6e3c889bacc2.4223.6.9 [call out=85b] v0'0 uv1 ondisk = 0) v8 ==== 194+0+85 (crc 0 0 0) 0x7fd29403d040 con 0x7fd284011040 2023-10-05T19:30:22.424 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.090+0000 7fd29bc01e40 1 -- 172.21.15.167:0/444809523 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:117 4.2a 4:55a3e9a6:::.dir.9a9df965-6612-45d4-a45f-6e3c889bacc2.4223.6.10:head [call rgw.bucket_list in=47b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32) v8 -- 0x557099a41c30 con 0x5570998f2620 2023-10-05T19:30:22.424 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.090+0000 7fd299f37700 1 -- 172.21.15.167:0/444809523 <== osd.1 v2:172.21.15.167:6800/30035 41 ==== osd_op_reply(117 .dir.9a9df965-6612-45d4-a45f-6e3c889bacc2.4223.6.10 [call out=85b] v0'0 uv2 ondisk = 0) v8 ==== 195+0+85 (crc 0 0 0) 0x7fd29403d040 con 0x5570998f2620 2023-10-05T19:30:22.424 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.090+0000 7fd29bc01e40 20 get_obj_state: rctx=0x7fff991e4070 obj=myfoo:15f6f8dc036519d7fe15b39338f6e5db state=0x5570999c3ec8 s->prefetch_data=0 2023-10-05T19:30:22.424 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.090+0000 7fd29bc01e40 1 -- 172.21.15.167:0/444809523 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:118 3.8s0 3:101585a7:::9a9df965-6612-45d4-a45f-6e3c889bacc2.4223.6_15f6f8dc036519d7fe15b39338f6e5db:head [getxattrs,stat] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32) v8 -- 0x5570999c4f20 con 0x55709992a5c0 2023-10-05T19:30:22.424 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.090+0000 7fd299736700 1 -- 172.21.15.167:0/444809523 <== osd.3 v2:172.21.15.167:6808/30036 29 ==== osd_op_reply(118 9a9df965-6612-45d4-a45f-6e3c889bacc2.4223.6_15f6f8dc036519d7fe15b39338f6e5db [getxattrs out=877b,stat out=16b] v0'0 uv1 ondisk = 0) v8 ==== 262+0+893 (crc 0 0 0) 0x7fd28c022020 con 0x55709992a5c0 2023-10-05T19:30:22.424 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.090+0000 7fd29bc01e40 10 manifest: total_size = 6 2023-10-05T19:30:22.424 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.090+0000 7fd29bc01e40 20 get_obj_state: setting s->obj_tag to 9a9df965-6612-45d4-a45f-6e3c889bacc2.4221.5260909277933780099 2023-10-05T19:30:22.424 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.090+0000 7fd29bc01e40 20 get_obj_state: rctx=0x7fff991e4070 obj=myfoo:15f6f8dc036519d7fe15b39338f6e5db state=0x5570999c3ec8 s->prefetch_data=0 2023-10-05T19:30:22.424 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.090+0000 7fd29bc01e40 20 prepare_atomic_modification: state is not atomic. state=0x5570999c3ec8 2023-10-05T19:30:22.424 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.090+0000 7fd29bc01e40 20 bucket index object: default.rgw.buckets.index:.dir.9a9df965-6612-45d4-a45f-6e3c889bacc2.4223.6.4 2023-10-05T19:30:22.424 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.090+0000 7fd29bc01e40 1 -- 172.21.15.167:0/444809523 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:119 4.2a 4:576abad7:::.dir.9a9df965-6612-45d4-a45f-6e3c889bacc2.4223.6.4:head [stat,call rgw.guard_bucket_resharding in=36b,call rgw.bucket_prepare_op in=210b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e32) v8 -- 0x5570999caff0 con 0x5570998f2620 2023-10-05T19:30:22.424 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.094+0000 7fd299f37700 1 -- 172.21.15.167:0/444809523 <== osd.1 v2:172.21.15.167:6800/30035 42 ==== osd_op_reply(119 .dir.9a9df965-6612-45d4-a45f-6e3c889bacc2.4223.6.4 [stat,call,call] v32'5 uv5 ondisk = 0) v8 ==== 278+0+0 (crc 0 0 0) 0x7fd29403d040 con 0x5570998f2620 2023-10-05T19:30:22.425 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.094+0000 7fd29bc01e40 1 -- 172.21.15.167:0/444809523 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:120 3.8s0 3:101585a7:::9a9df965-6612-45d4-a45f-6e3c889bacc2.4223.6_15f6f8dc036519d7fe15b39338f6e5db:head [call rgw.obj_remove in=40b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e32) v8 -- 0x5570999c8fd0 con 0x55709992a5c0 2023-10-05T19:30:22.425 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.098+0000 7fd299736700 1 -- 172.21.15.167:0/444809523 <== osd.3 v2:172.21.15.167:6808/30036 30 ==== osd_op_reply(120 9a9df965-6612-45d4-a45f-6e3c889bacc2.4223.6_15f6f8dc036519d7fe15b39338f6e5db [call] v32'2 uv2 ondisk = 0) v8 ==== 220+0+0 (crc 0 0 0) 0x7fd28c022020 con 0x55709992a5c0 2023-10-05T19:30:22.425 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.098+0000 7fd29bc01e40 1 -- 172.21.15.167:0/444809523 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:121 4.2a 4:576abad7:::.dir.9a9df965-6612-45d4-a45f-6e3c889bacc2.4223.6.4:head [stat,call rgw.guard_bucket_resharding in=36b,call rgw.bucket_complete_op in=287b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e32) v8 -- 0x5570999cb650 con 0x5570998f2620 2023-10-05T19:30:22.425 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.098+0000 7fd29bc01e40 10 cache get: name=default.rgw.log++bucket.sync-source-hints.myfoo : miss 2023-10-05T19:30:22.425 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.098+0000 7fd29bc01e40 20 rados->read ofs=0 len=0 2023-10-05T19:30:22.425 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.098+0000 7fd29bc01e40 1 -- 172.21.15.167:0/444809523 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:122 6.18 6:1df25bda:::bucket.sync-source-hints.myfoo:head [call version.read in=11b,read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32) v8 -- 0x5570999cb650 con 0x7fd284011040 2023-10-05T19:30:22.425 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.098+0000 7fd299f37700 1 -- 172.21.15.167:0/444809523 <== osd.0 v2:172.21.15.118:6800/27690 32 ==== osd_op_reply(122 bucket.sync-source-hints.myfoo [call,read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 ==== 216+0+0 (crc 0 0 0) 0x7fd29403d040 con 0x7fd284011040 2023-10-05T19:30:22.425 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.098+0000 7fd29bc01e40 20 rados_obj.operate() r=-2 bl.length=0 2023-10-05T19:30:22.425 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.098+0000 7fd29bc01e40 10 cache put: name=default.rgw.log++bucket.sync-source-hints.myfoo info.flags=0x0 2023-10-05T19:30:22.425 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.098+0000 7fd29bc01e40 10 adding default.rgw.log++bucket.sync-source-hints.myfoo to cache LRU end 2023-10-05T19:30:22.425 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.098+0000 7fd29bc01e40 10 cache get: name=default.rgw.log++bucket.sync-target-hints.myfoo : miss 2023-10-05T19:30:22.426 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.098+0000 7fd29bc01e40 20 rados->read ofs=0 len=0 2023-10-05T19:30:22.426 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.098+0000 7fd29bc01e40 1 -- 172.21.15.167:0/444809523 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:123 6.4 6:26e4c4bc:::bucket.sync-target-hints.myfoo:head [call version.read in=11b,read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32) v8 -- 0x5570999cb650 con 0x5570998f2620 2023-10-05T19:30:22.426 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.098+0000 7fd299f37700 1 -- 172.21.15.167:0/444809523 <== osd.1 v2:172.21.15.167:6800/30035 43 ==== osd_op_reply(123 bucket.sync-target-hints.myfoo [call,read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 ==== 216+0+0 (crc 0 0 0) 0x7fd29403d040 con 0x5570998f2620 2023-10-05T19:30:22.426 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.098+0000 7fd29bc01e40 20 rados_obj.operate() r=-2 bl.length=0 2023-10-05T19:30:22.426 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.098+0000 7fd29bc01e40 10 cache put: name=default.rgw.log++bucket.sync-target-hints.myfoo info.flags=0x0 2023-10-05T19:30:22.426 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.098+0000 7fd29bc01e40 10 adding default.rgw.log++bucket.sync-target-hints.myfoo to cache LRU end 2023-10-05T19:30:22.426 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.098+0000 7fd29bc01e40 10 chain_cache_entry: cache_locator= 2023-10-05T19:30:22.426 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.098+0000 7fd29bc01e40 20 chain_cache_entry: couldn't find cache locator 2023-10-05T19:30:22.426 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.098+0000 7fd29bc01e40 20 couldn't put bucket_sync_policy cache entry, might have raced with data changes 2023-10-05T19:30:22.426 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.098+0000 7fd29bc01e40 20 int RGWRados::Bucket::List::list_objects_ordered(const DoutPrefixProvider*, int64_t, std::vector*, std::map, bool>*, bool*, optional_yield): starting attempt 1 2023-10-05T19:30:22.426 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.098+0000 7fd29bc01e40 10 int RGWRados::cls_bucket_list_ordered(const DoutPrefixProvider*, RGWBucketInfo&, int, const rgw_obj_index_key&, const string&, const string&, uint32_t, bool, uint16_t, RGWRados::ent_map_t&, bool*, bool*, rgw_obj_index_key*, optional_yield, RGWBucketListNameFilter): :myfoo[9a9df965-6612-45d4-a45f-6e3c889bacc2.4223.6]) start_after="_multipart_", prefix="_multipart_, delimiter="", shard_id=-1", num_entries=1001, list_versions=0, expansion_factor=1, force_check_filter is unset 2023-10-05T19:30:22.426 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.098+0000 7fd29bc01e40 10 int RGWRados::cls_bucket_list_ordered(const DoutPrefixProvider*, RGWBucketInfo&, int, const rgw_obj_index_key&, const string&, const string&, uint32_t, bool, uint16_t, RGWRados::ent_map_t&, bool*, bool*, rgw_obj_index_key*, optional_yield, RGWBucketListNameFilter): request from each of 11 shard(s) for 112 entries to get 1001 total entries 2023-10-05T19:30:22.427 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.098+0000 7fd29bc01e40 1 -- 172.21.15.167:0/444809523 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:124 4.3 4:c2471d59:::.dir.9a9df965-6612-45d4-a45f-6e3c889bacc2.4223.6.0:head [call rgw.bucket_list in=69b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32) v8 -- 0x5570999cc4a0 con 0x5570998f2620 2023-10-05T19:30:22.427 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.098+0000 7fd29bc01e40 1 -- 172.21.15.167:0/444809523 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:125 4.1f 4:fac78d74:::.dir.9a9df965-6612-45d4-a45f-6e3c889bacc2.4223.6.1:head [call rgw.bucket_list in=69b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32) v8 -- 0x5570999c41d0 con 0x55709992a5c0 2023-10-05T19:30:22.427 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.098+0000 7fd29bc01e40 1 -- 172.21.15.167:0/444809523 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:126 4.3 4:c00d743b:::.dir.9a9df965-6612-45d4-a45f-6e3c889bacc2.4223.6.2:head [call rgw.bucket_list in=69b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32) v8 -- 0x5570999cc4a0 con 0x5570998f2620 2023-10-05T19:30:22.427 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.098+0000 7fd29bc01e40 1 -- 172.21.15.167:0/444809523 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:127 4.2b 4:d72453a1:::.dir.9a9df965-6612-45d4-a45f-6e3c889bacc2.4223.6.3:head [call rgw.bucket_list in=69b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32) v8 -- 0x5570999c41d0 con 0x5570998f2620 2023-10-05T19:30:22.427 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.098+0000 7fd29bc01e40 1 -- 172.21.15.167:0/444809523 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:128 4.2a 4:576abad7:::.dir.9a9df965-6612-45d4-a45f-6e3c889bacc2.4223.6.4:head [call rgw.bucket_list in=69b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32) v8 -- 0x5570999d80a0 con 0x5570998f2620 2023-10-05T19:30:22.427 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.098+0000 7fd29bc01e40 1 -- 172.21.15.167:0/444809523 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:129 4.2d 4:b45bfada:::.dir.9a9df965-6612-45d4-a45f-6e3c889bacc2.4223.6.5:head [call rgw.bucket_list in=69b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32) v8 -- 0x5570999daa00 con 0x7fd284011040 2023-10-05T19:30:22.427 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.098+0000 7fd29bc01e40 1 -- 172.21.15.167:0/444809523 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:130 4.33 4:ceb97e7a:::.dir.9a9df965-6612-45d4-a45f-6e3c889bacc2.4223.6.6:head [call rgw.bucket_list in=69b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32) v8 -- 0x5570999dd300 con 0x7fd284011040 2023-10-05T19:30:22.427 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.098+0000 7fd29bc01e40 1 -- 172.21.15.167:0/444809523 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:131 4.5 4:a170829b:::.dir.9a9df965-6612-45d4-a45f-6e3c889bacc2.4223.6.7:head [call rgw.bucket_list in=69b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32) v8 -- 0x5570999dfc40 con 0x55709992a5c0 2023-10-05T19:30:22.427 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.102+0000 7fd29bc01e40 1 -- 172.21.15.167:0/444809523 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:132 4.8 4:13a6ed6b:::.dir.9a9df965-6612-45d4-a45f-6e3c889bacc2.4223.6.8:head [call rgw.bucket_list in=69b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32) v8 -- 0x5570999e2a90 con 0x55709992a5c0 2023-10-05T19:30:22.427 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.102+0000 7fd29bc01e40 1 -- 172.21.15.167:0/444809523 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:133 4.7 4:e2c155b5:::.dir.9a9df965-6612-45d4-a45f-6e3c889bacc2.4223.6.9:head [call rgw.bucket_list in=69b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32) v8 -- 0x5570999e5540 con 0x7fd284011040 2023-10-05T19:30:22.427 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.102+0000 7fd29bc01e40 1 -- 172.21.15.167:0/444809523 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:134 4.2a 4:55a3e9a6:::.dir.9a9df965-6612-45d4-a45f-6e3c889bacc2.4223.6.10:head [call rgw.bucket_list in=69b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32) v8 -- 0x5570999e83c0 con 0x5570998f2620 2023-10-05T19:30:22.428 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.102+0000 7fd299f37700 1 -- 172.21.15.167:0/444809523 <== osd.1 v2:172.21.15.167:6800/30035 44 ==== osd_op_reply(124 .dir.9a9df965-6612-45d4-a45f-6e3c889bacc2.4223.6.0 [call out=85b] v0'0 uv1 ondisk = 0) v8 ==== 194+0+85 (crc 0 0 0) 0x7fd29403d040 con 0x5570998f2620 2023-10-05T19:30:22.428 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.102+0000 7fd299f37700 1 -- 172.21.15.167:0/444809523 <== osd.1 v2:172.21.15.167:6800/30035 45 ==== osd_op_reply(121 .dir.9a9df965-6612-45d4-a45f-6e3c889bacc2.4223.6.4 [stat,call,call] v32'6 uv6 ondisk = 0) v8 ==== 278+0+0 (crc 0 0 0) 0x7fd29403d040 con 0x5570998f2620 2023-10-05T19:30:22.428 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.102+0000 7fd2927fc700 20 handle_completion(): completion ok for obj=15f6f8dc036519d7fe15b39338f6e5db 2023-10-05T19:30:22.428 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.102+0000 7fd299736700 1 -- 172.21.15.167:0/444809523 <== osd.3 v2:172.21.15.167:6808/30036 31 ==== osd_op_reply(125 .dir.9a9df965-6612-45d4-a45f-6e3c889bacc2.4223.6.1 [call out=85b] v0'0 uv3 ondisk = 0) v8 ==== 194+0+85 (crc 0 0 0) 0x7fd28c022020 con 0x55709992a5c0 2023-10-05T19:30:22.428 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.102+0000 7fd299736700 1 -- 172.21.15.167:0/444809523 <== osd.3 v2:172.21.15.167:6808/30036 32 ==== osd_op_reply(131 .dir.9a9df965-6612-45d4-a45f-6e3c889bacc2.4223.6.7 [call out=85b] v0'0 uv3 ondisk = 0) v8 ==== 194+0+85 (crc 0 0 0) 0x7fd28c022020 con 0x55709992a5c0 2023-10-05T19:30:22.428 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.102+0000 7fd299f37700 1 -- 172.21.15.167:0/444809523 <== osd.1 v2:172.21.15.167:6800/30035 46 ==== osd_op_reply(127 .dir.9a9df965-6612-45d4-a45f-6e3c889bacc2.4223.6.3 [call out=85b] v0'0 uv3 ondisk = 0) v8 ==== 194+0+85 (crc 0 0 0) 0x7fd29403d040 con 0x5570998f2620 2023-10-05T19:30:22.428 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.102+0000 7fd299f37700 1 -- 172.21.15.167:0/444809523 <== osd.1 v2:172.21.15.167:6800/30035 47 ==== osd_op_reply(128 .dir.9a9df965-6612-45d4-a45f-6e3c889bacc2.4223.6.4 [call out=124b] v0'0 uv6 ondisk = 0) v8 ==== 194+0+124 (crc 0 0 0) 0x7fd29403d040 con 0x5570998f2620 2023-10-05T19:30:22.428 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.102+0000 7fd299f37700 1 -- 172.21.15.167:0/444809523 <== osd.0 v2:172.21.15.118:6800/27690 33 ==== osd_op_reply(130 .dir.9a9df965-6612-45d4-a45f-6e3c889bacc2.4223.6.6 [call out=85b] v0'0 uv5 ondisk = 0) v8 ==== 194+0+85 (crc 0 0 0) 0x7fd29403d040 con 0x7fd284011040 2023-10-05T19:30:22.428 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.102+0000 7fd299f37700 1 -- 172.21.15.167:0/444809523 <== osd.0 v2:172.21.15.118:6800/27690 34 ==== osd_op_reply(129 .dir.9a9df965-6612-45d4-a45f-6e3c889bacc2.4223.6.5 [call out=85b] v0'0 uv7 ondisk = 0) v8 ==== 194+0+85 (crc 0 0 0) 0x7fd29403d040 con 0x7fd284011040 2023-10-05T19:30:22.428 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.102+0000 7fd299736700 1 -- 172.21.15.167:0/444809523 <== osd.3 v2:172.21.15.167:6808/30036 33 ==== osd_op_reply(132 .dir.9a9df965-6612-45d4-a45f-6e3c889bacc2.4223.6.8 [call out=85b] v0'0 uv7 ondisk = 0) v8 ==== 194+0+85 (crc 0 0 0) 0x7fd28c022020 con 0x55709992a5c0 2023-10-05T19:30:22.428 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.102+0000 7fd299f37700 1 -- 172.21.15.167:0/444809523 <== osd.0 v2:172.21.15.118:6800/27690 35 ==== osd_op_reply(133 .dir.9a9df965-6612-45d4-a45f-6e3c889bacc2.4223.6.9 [call out=85b] v0'0 uv1 ondisk = 0) v8 ==== 194+0+85 (crc 0 0 0) 0x7fd29403d040 con 0x7fd284011040 2023-10-05T19:30:22.428 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.102+0000 7fd299f37700 1 -- 172.21.15.167:0/444809523 <== osd.1 v2:172.21.15.167:6800/30035 48 ==== osd_op_reply(134 .dir.9a9df965-6612-45d4-a45f-6e3c889bacc2.4223.6.10 [call out=85b] v0'0 uv2 ondisk = 0) v8 ==== 195+0+85 (crc 0 0 0) 0x7fd29403d040 con 0x5570998f2620 2023-10-05T19:30:22.429 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.102+0000 7fd299f37700 1 -- 172.21.15.167:0/444809523 <== osd.1 v2:172.21.15.167:6800/30035 49 ==== osd_op_reply(126 .dir.9a9df965-6612-45d4-a45f-6e3c889bacc2.4223.6.2 [call out=85b] v0'0 uv2 ondisk = 0) v8 ==== 194+0+85 (crc 0 0 0) 0x7fd29403d040 con 0x5570998f2620 2023-10-05T19:30:22.429 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.102+0000 7fd29bc01e40 20 int RGWRados::cls_bucket_list_ordered(const DoutPrefixProvider*, RGWBucketInfo&, int, const rgw_obj_index_key&, const string&, const string&, uint32_t, bool, uint16_t, RGWRados::ent_map_t&, bool*, bool*, rgw_obj_index_key*, optional_yield, RGWBucketListNameFilter): returning, count=0, is_truncated=0 2023-10-05T19:30:22.429 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.102+0000 7fd29bc01e40 20 int RGWRados::cls_bucket_list_ordered(const DoutPrefixProvider*, RGWBucketInfo&, int, const rgw_obj_index_key&, const string&, const string&, uint32_t, bool, uint16_t, RGWRados::ent_map_t&, bool*, bool*, rgw_obj_index_key*, optional_yield, RGWBucketListNameFilter): returning, last_entry NOT SET 2023-10-05T19:30:22.429 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.102+0000 7fd29bc01e40 10 int RGWRados::Bucket::List::list_objects_ordered(const DoutPrefixProvider*, int64_t, std::vector*, std::map, bool>*, bool*, optional_yield): end of outer loop, truncated=0, count=0, attempt=1 2023-10-05T19:30:22.429 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.102+0000 7fd29bc01e40 10 int RGWRados::Bucket::List::list_objects_ordered(const DoutPrefixProvider*, int64_t, std::vector*, std::map, bool>*, bool*, optional_yield): exiting attempt loop because we reached end (0) or we're returning half the requested entries (0 of 1000) 2023-10-05T19:30:22.429 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.102+0000 7fd29bc01e40 20 abort_multiparts INFO: aborting and cleaning up multipart upload(s); bucket=":myfoo[9a9df965-6612-45d4-a45f-6e3c889bacc2.4223.6])"; uploads.size()=0; is_truncated=0 2023-10-05T19:30:22.429 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.102+0000 7fd29bc01e40 10 lifecycle: RGWRados::convert_old_bucket_info(): bucket=:myfoo[9a9df965-6612-45d4-a45f-6e3c889bacc2.4223.6]) 2023-10-05T19:30:22.429 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.102+0000 7fd29bc01e40 20 lifecycle: get_system_obj_state: rctx=0x7fff991e3b08 obj=default.rgw.meta:root:myfoo state=0x5570997cc0b0 s->prefetch_data=0 2023-10-05T19:30:22.429 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.102+0000 7fd29bc01e40 10 lifecycle: cache get: name=default.rgw.meta+root+myfoo : miss 2023-10-05T19:30:22.429 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.102+0000 7fd29bc01e40 1 -- 172.21.15.167:0/444809523 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:135 8.9 8:91b71841:root::myfoo:head [call version.read in=11b,getxattrs,stat] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32) v8 -- 0x5570999e83c0 con 0x7fd28c007030 2023-10-05T19:30:22.429 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.102+0000 7fd298f35700 1 -- 172.21.15.167:0/444809523 <== osd.2 v2:172.21.15.167:6816/30037 18 ==== osd_op_reply(135 myfoo [call out=48b,getxattrs out=75b,stat out=16b] v0'0 uv42 ondisk = 0) v8 ==== 233+0+139 (crc 0 0 0) 0x7fd28402f040 con 0x7fd28c007030 2023-10-05T19:30:22.429 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.102+0000 7fd29bc01e40 10 lifecycle: cache put: name=default.rgw.meta+root+myfoo info.flags=0x16 2023-10-05T19:30:22.430 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.102+0000 7fd29bc01e40 10 lifecycle: adding default.rgw.meta+root+myfoo to cache LRU end 2023-10-05T19:30:22.430 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.102+0000 7fd29bc01e40 10 lifecycle: updating xattr: name=ceph.objclass.version bl.length()=42 2023-10-05T19:30:22.430 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.102+0000 7fd29bc01e40 20 lifecycle: get_system_obj_state: s->obj_tag was set empty 2023-10-05T19:30:22.430 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.102+0000 7fd29bc01e40 20 lifecycle: Read xattr: user.rgw.idtag 2023-10-05T19:30:22.430 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.102+0000 7fd29bc01e40 10 lifecycle: cache get: name=default.rgw.meta+root+myfoo : type miss (requested=0x13, cached=0x16) 2023-10-05T19:30:22.430 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.102+0000 7fd29bc01e40 20 lifecycle: rados->read ofs=0 len=0 2023-10-05T19:30:22.430 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.102+0000 7fd29bc01e40 1 -- 172.21.15.167:0/444809523 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:136 8.9 8:91b71841:root::myfoo:head [call version.check_conds in=122b,call version.read in=11b,read 0~0,getxattrs] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32) v8 -- 0x5570999c3ce0 con 0x7fd28c007030 2023-10-05T19:30:22.430 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.102+0000 7fd298f35700 1 -- 172.21.15.167:0/444809523 <== osd.2 v2:172.21.15.167:6816/30037 19 ==== osd_op_reply(136 myfoo [call,call out=48b,read 0~165 out=165b,getxattrs out=75b] v0'0 uv42 ondisk = 0) v8 ==== 275+0+288 (crc 0 0 0) 0x7fd28402f040 con 0x7fd28c007030 2023-10-05T19:30:22.430 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.102+0000 7fd29bc01e40 20 lifecycle: rados_obj.operate() r=0 bl.length=165 2023-10-05T19:30:22.430 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.102+0000 7fd29bc01e40 10 lifecycle: cache put: name=default.rgw.meta+root+myfoo info.flags=0x13 2023-10-05T19:30:22.430 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.102+0000 7fd29bc01e40 10 lifecycle: moving default.rgw.meta+root+myfoo to cache LRU end 2023-10-05T19:30:22.431 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.102+0000 7fd29bc01e40 10 lifecycle: updating xattr: name=ceph.objclass.version bl.length()=42 2023-10-05T19:30:22.431 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.102+0000 7fd29bc01e40 1 -- 172.21.15.167:0/444809523 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:137 8.b 8:d4944359:root::.bucket.meta.myfoo%3a9a9df965-6612-45d4-a45f-6e3c889bacc2.4223.6:head [delete,create,call version.check_conds in=122b,call version.set in=58b,writefull 0~352 in=352b,setxattr user.rgw.acl (123) in=135b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e32) v8 -- 0x5570999d7b50 con 0x55709992a5c0 2023-10-05T19:30:22.431 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.106+0000 7fd299736700 1 -- 172.21.15.167:0/444809523 <== osd.3 v2:172.21.15.167:6808/30036 34 ==== osd_op_reply(137 .bucket.meta.myfoo:9a9df965-6612-45d4-a45f-6e3c889bacc2.4223.6 [delete,create,call,call,writefull 0~352,setxattr (123)] v32'5 uv5 ondisk = 0) v8 ==== 416+0+0 (crc 0 0 0) 0x7fd28c022020 con 0x55709992a5c0 2023-10-05T19:30:22.431 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.106+0000 7fd29bc01e40 10 lifecycle: cache put: name=default.rgw.meta+root+.bucket.meta.myfoo:9a9df965-6612-45d4-a45f-6e3c889bacc2.4223.6 info.flags=0x17 2023-10-05T19:30:22.431 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.106+0000 7fd29bc01e40 10 lifecycle: moving default.rgw.meta+root+.bucket.meta.myfoo:9a9df965-6612-45d4-a45f-6e3c889bacc2.4223.6 to cache LRU end 2023-10-05T19:30:22.431 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.106+0000 7fd29bc01e40 10 lifecycle: updating xattr: name=user.rgw.acl bl.length()=123 2023-10-05T19:30:22.431 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.106+0000 7fd29bc01e40 10 lifecycle: distributing notification oid=default.rgw.control:notify.7 cni=[op: 0, obj: default.rgw.meta:root:.bucket.meta.myfoo:9a9df965-6612-45d4-a45f-6e3c889bacc2.4223.6, ofs0, ns] 2023-10-05T19:30:22.431 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.106+0000 7fd29bc01e40 1 -- 172.21.15.167:0/444809523 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:138 7.9 7:93e5b521:::notify.7:head [notify cookie 93942101932880 in=737b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32) v8 -- 0x5570999d84b0 con 0x7fd284011040 2023-10-05T19:30:22.431 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.106+0000 7fd299f37700 1 -- 172.21.15.167:0/444809523 <== osd.0 v2:172.21.15.118:6800/27690 36 ==== watch-notify(notify (1) cookie 93942101265504 notify 137438953515 ret 0) v3 ==== 767+0+0 (crc 0 0 0) 0x7fd294061700 con 0x7fd284011040 2023-10-05T19:30:22.431 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.106+0000 7fd292ffd700 10 rgw watcher librados: RGWWatcher::handle_notify() notify_id 137438953515 cookie 93942101265504 notifier 4755 bl.length()=725 2023-10-05T19:30:22.431 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.106+0000 7fd292ffd700 10 rgw watcher librados: cache put: name=default.rgw.meta+root+.bucket.meta.myfoo:9a9df965-6612-45d4-a45f-6e3c889bacc2.4223.6 info.flags=0x17 2023-10-05T19:30:22.431 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.106+0000 7fd292ffd700 10 rgw watcher librados: moving default.rgw.meta+root+.bucket.meta.myfoo:9a9df965-6612-45d4-a45f-6e3c889bacc2.4223.6 to cache LRU end 2023-10-05T19:30:22.432 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.106+0000 7fd292ffd700 10 rgw watcher librados: updating xattr: name=user.rgw.acl bl.length()=123 2023-10-05T19:30:22.432 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.106+0000 7fd292ffd700 1 -- 172.21.15.167:0/444809523 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:139 7.9 7:93e5b521:::notify.7:head [notify-ack in=20b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32) v8 -- 0x7fd278003d90 con 0x7fd284011040 2023-10-05T19:30:22.432 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.106+0000 7fd299f37700 1 -- 172.21.15.167:0/444809523 <== osd.0 v2:172.21.15.118:6800/27690 37 ==== osd_op_reply(138 notify.7 [notify cookie 93942101932880 out=8b] v0'0 uv264 ondisk = 0) v8 ==== 152+0+8 (crc 0 0 0) 0x7fd29403d040 con 0x7fd284011040 2023-10-05T19:30:22.432 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.106+0000 7fd299f37700 1 -- 172.21.15.167:0/444809523 <== osd.0 v2:172.21.15.118:6800/27690 38 ==== osd_op_reply(139 notify.7 [notify-ack] v0'0 uv264 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7fd29403d040 con 0x7fd284011040 2023-10-05T19:30:22.432 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.106+0000 7fd299f37700 1 -- 172.21.15.167:0/444809523 <== osd.0 v2:172.21.15.118:6800/27690 39 ==== watch-notify(notify_complete (2) cookie 93942101932880 notify 137438953515 ret 0) v3 ==== 42+0+48 (crc 0 0 0) 0x7fd294086040 con 0x7fd284011040 2023-10-05T19:30:22.432 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.106+0000 7fd29bc01e40 20 lifecycle: handle_bi_update(): bucket=:myfoo[9a9df965-6612-45d4-a45f-6e3c889bacc2.4223.6]): orig_sources= new_sources= 2023-10-05T19:30:22.432 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.106+0000 7fd29bc01e40 20 lifecycle: handle_bi_update(): bucket=:myfoo[9a9df965-6612-45d4-a45f-6e3c889bacc2.4223.6]): potential sources added=[] removed=[] 2023-10-05T19:30:22.432 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.106+0000 7fd29bc01e40 20 lifecycle: handle_bi_update(): bucket=:myfoo[9a9df965-6612-45d4-a45f-6e3c889bacc2.4223.6]): orig_dests= new_dests= 2023-10-05T19:30:22.432 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.106+0000 7fd29bc01e40 20 lifecycle: handle_bi_update(): bucket=:myfoo[9a9df965-6612-45d4-a45f-6e3c889bacc2.4223.6]): potential dests added=[] removed=[] 2023-10-05T19:30:22.432 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.106+0000 7fd29bc01e40 1 -- 172.21.15.167:0/832409700 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:33 6.14 6:2f3f5e88:lc::lc.17:head [call lock.lock in=65b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e32) v8 -- 0x5570999c6790 con 0x55709995e900 2023-10-05T19:30:22.432 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.110+0000 7fd299736700 1 -- 172.21.15.167:0/832409700 <== osd.2 v2:172.21.15.167:6816/30037 8 ==== osd_op_reply(33 lc.17 [call] v32'14 uv14 ondisk = 0) v8 ==== 149+0+0 (crc 0 0 0) 0x7fd28c022020 con 0x55709995e900 2023-10-05T19:30:22.432 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.110+0000 7fd29bc01e40 1 -- 172.21.15.167:0/832409700 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:34 6.14 6:2f3f5e88:lc::lc.17:head [call rgw.lc_rm_entry in=92b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32) v8 -- 0x5570999c6790 con 0x55709995e900 2023-10-05T19:30:22.433 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.110+0000 7fd299736700 1 -- 172.21.15.167:0/832409700 <== osd.2 v2:172.21.15.167:6816/30037 9 ==== osd_op_reply(34 lc.17 [call] v32'15 uv15 ondisk = 0) v8 ==== 149+0+0 (crc 0 0 0) 0x7fd28c022020 con 0x55709995e900 2023-10-05T19:30:22.433 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.110+0000 7fd29bc01e40 1 -- 172.21.15.167:0/832409700 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:35 6.14 6:2f3f5e88:lc::lc.17:head [call lock.unlock in=49b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e32) v8 -- 0x5570999c6790 con 0x55709995e900 2023-10-05T19:30:22.433 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.114+0000 7fd299736700 1 -- 172.21.15.167:0/832409700 <== osd.2 v2:172.21.15.167:6816/30037 10 ==== osd_op_reply(35 lc.17 [call] v32'16 uv16 ondisk = 0) v8 ==== 149+0+0 (crc 0 0 0) 0x7fd28c022020 con 0x55709995e900 2023-10-05T19:30:22.433 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.114+0000 7fd29bc01e40 1 -- 172.21.15.167:0/444809523 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:140 4.3 4:c2471d59:::.dir.9a9df965-6612-45d4-a45f-6e3c889bacc2.4223.6.0:head [call rgw.bucket_list in=47b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32) v8 -- 0x5570999c6790 con 0x5570998f2620 2023-10-05T19:30:22.433 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.114+0000 7fd29bc01e40 1 -- 172.21.15.167:0/444809523 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:141 4.1f 4:fac78d74:::.dir.9a9df965-6612-45d4-a45f-6e3c889bacc2.4223.6.1:head [call rgw.bucket_list in=47b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32) v8 -- 0x5570999d6ee0 con 0x55709992a5c0 2023-10-05T19:30:22.433 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.114+0000 7fd29bc01e40 1 -- 172.21.15.167:0/444809523 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:142 4.3 4:c00d743b:::.dir.9a9df965-6612-45d4-a45f-6e3c889bacc2.4223.6.2:head [call rgw.bucket_list in=47b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32) v8 -- 0x5570999d72d0 con 0x5570998f2620 2023-10-05T19:30:22.433 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.114+0000 7fd29bc01e40 1 -- 172.21.15.167:0/444809523 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:143 4.2b 4:d72453a1:::.dir.9a9df965-6612-45d4-a45f-6e3c889bacc2.4223.6.3:head [call rgw.bucket_list in=47b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32) v8 -- 0x5570999c6790 con 0x5570998f2620 2023-10-05T19:30:22.433 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.114+0000 7fd29bc01e40 1 -- 172.21.15.167:0/444809523 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:144 4.2a 4:576abad7:::.dir.9a9df965-6612-45d4-a45f-6e3c889bacc2.4223.6.4:head [call rgw.bucket_list in=47b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32) v8 -- 0x5570999df510 con 0x5570998f2620 2023-10-05T19:30:22.433 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.114+0000 7fd29bc01e40 1 -- 172.21.15.167:0/444809523 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:145 4.2d 4:b45bfada:::.dir.9a9df965-6612-45d4-a45f-6e3c889bacc2.4223.6.5:head [call rgw.bucket_list in=47b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32) v8 -- 0x5570999c6790 con 0x7fd284011040 2023-10-05T19:30:22.433 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.114+0000 7fd29bc01e40 1 -- 172.21.15.167:0/444809523 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:146 4.33 4:ceb97e7a:::.dir.9a9df965-6612-45d4-a45f-6e3c889bacc2.4223.6.6:head [call rgw.bucket_list in=47b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32) v8 -- 0x5570999e0280 con 0x7fd284011040 2023-10-05T19:30:22.433 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.114+0000 7fd29bc01e40 1 -- 172.21.15.167:0/444809523 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:147 4.5 4:a170829b:::.dir.9a9df965-6612-45d4-a45f-6e3c889bacc2.4223.6.7:head [call rgw.bucket_list in=47b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32) v8 -- 0x5570999e2c30 con 0x55709992a5c0 2023-10-05T19:30:22.433 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.114+0000 7fd29bc01e40 1 -- 172.21.15.167:0/444809523 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:148 4.8 4:13a6ed6b:::.dir.9a9df965-6612-45d4-a45f-6e3c889bacc2.4223.6.8:head [call rgw.bucket_list in=47b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32) v8 -- 0x5570999e5630 con 0x55709992a5c0 2023-10-05T19:30:22.434 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.114+0000 7fd29bc01e40 1 -- 172.21.15.167:0/444809523 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:149 4.7 4:e2c155b5:::.dir.9a9df965-6612-45d4-a45f-6e3c889bacc2.4223.6.9:head [call rgw.bucket_list in=47b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32) v8 -- 0x5570999d0360 con 0x7fd284011040 2023-10-05T19:30:22.434 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.114+0000 7fd29bc01e40 1 -- 172.21.15.167:0/444809523 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:150 4.2a 4:55a3e9a6:::.dir.9a9df965-6612-45d4-a45f-6e3c889bacc2.4223.6.10:head [call rgw.bucket_list in=47b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32) v8 -- 0x5570999d2d50 con 0x5570998f2620 2023-10-05T19:30:22.434 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.114+0000 7fd299736700 1 -- 172.21.15.167:0/444809523 <== osd.3 v2:172.21.15.167:6808/30036 35 ==== osd_op_reply(141 .dir.9a9df965-6612-45d4-a45f-6e3c889bacc2.4223.6.1 [call out=85b] v0'0 uv3 ondisk = 0) v8 ==== 194+0+85 (crc 0 0 0) 0x7fd28c07a040 con 0x55709992a5c0 2023-10-05T19:30:22.434 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.114+0000 7fd299736700 1 -- 172.21.15.167:0/444809523 <== osd.3 v2:172.21.15.167:6808/30036 36 ==== osd_op_reply(148 .dir.9a9df965-6612-45d4-a45f-6e3c889bacc2.4223.6.8 [call out=85b] v0'0 uv7 ondisk = 0) v8 ==== 194+0+85 (crc 0 0 0) 0x7fd28c07a040 con 0x55709992a5c0 2023-10-05T19:30:22.434 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.114+0000 7fd299f37700 1 -- 172.21.15.167:0/444809523 <== osd.1 v2:172.21.15.167:6800/30035 50 ==== osd_op_reply(140 .dir.9a9df965-6612-45d4-a45f-6e3c889bacc2.4223.6.0 [call out=85b] v0'0 uv1 ondisk = 0) v8 ==== 194+0+85 (crc 0 0 0) 0x7fd29403d040 con 0x5570998f2620 2023-10-05T19:30:22.434 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.114+0000 7fd299736700 1 -- 172.21.15.167:0/444809523 <== osd.3 v2:172.21.15.167:6808/30036 37 ==== osd_op_reply(147 .dir.9a9df965-6612-45d4-a45f-6e3c889bacc2.4223.6.7 [call out=85b] v0'0 uv3 ondisk = 0) v8 ==== 194+0+85 (crc 0 0 0) 0x7fd28c07a040 con 0x55709992a5c0 2023-10-05T19:30:22.434 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.114+0000 7fd299f37700 1 -- 172.21.15.167:0/444809523 <== osd.1 v2:172.21.15.167:6800/30035 51 ==== osd_op_reply(143 .dir.9a9df965-6612-45d4-a45f-6e3c889bacc2.4223.6.3 [call out=85b] v0'0 uv3 ondisk = 0) v8 ==== 194+0+85 (crc 0 0 0) 0x7fd29403d040 con 0x5570998f2620 2023-10-05T19:30:22.434 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.114+0000 7fd299f37700 1 -- 172.21.15.167:0/444809523 <== osd.1 v2:172.21.15.167:6800/30035 52 ==== osd_op_reply(142 .dir.9a9df965-6612-45d4-a45f-6e3c889bacc2.4223.6.2 [call out=85b] v0'0 uv2 ondisk = 0) v8 ==== 194+0+85 (crc 0 0 0) 0x7fd29403d040 con 0x5570998f2620 2023-10-05T19:30:22.434 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.114+0000 7fd299f37700 1 -- 172.21.15.167:0/444809523 <== osd.1 v2:172.21.15.167:6800/30035 53 ==== osd_op_reply(144 .dir.9a9df965-6612-45d4-a45f-6e3c889bacc2.4223.6.4 [call out=124b] v0'0 uv6 ondisk = 0) v8 ==== 194+0+124 (crc 0 0 0) 0x7fd29403d040 con 0x5570998f2620 2023-10-05T19:30:22.434 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.114+0000 7fd299f37700 1 -- 172.21.15.167:0/444809523 <== osd.1 v2:172.21.15.167:6800/30035 54 ==== osd_op_reply(150 .dir.9a9df965-6612-45d4-a45f-6e3c889bacc2.4223.6.10 [call out=85b] v0'0 uv2 ondisk = 0) v8 ==== 195+0+85 (crc 0 0 0) 0x7fd29403d040 con 0x5570998f2620 2023-10-05T19:30:22.434 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.114+0000 7fd299f37700 1 -- 172.21.15.167:0/444809523 <== osd.0 v2:172.21.15.118:6800/27690 40 ==== osd_op_reply(146 .dir.9a9df965-6612-45d4-a45f-6e3c889bacc2.4223.6.6 [call out=85b] v0'0 uv5 ondisk = 0) v8 ==== 194+0+85 (crc 0 0 0) 0x7fd29403d040 con 0x7fd284011040 2023-10-05T19:30:22.435 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.114+0000 7fd299f37700 1 -- 172.21.15.167:0/444809523 <== osd.0 v2:172.21.15.118:6800/27690 41 ==== osd_op_reply(149 .dir.9a9df965-6612-45d4-a45f-6e3c889bacc2.4223.6.9 [call out=85b] v0'0 uv1 ondisk = 0) v8 ==== 194+0+85 (crc 0 0 0) 0x7fd29403d040 con 0x7fd284011040 2023-10-05T19:30:22.435 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.114+0000 7fd299f37700 1 -- 172.21.15.167:0/444809523 <== osd.0 v2:172.21.15.118:6800/27690 42 ==== osd_op_reply(145 .dir.9a9df965-6612-45d4-a45f-6e3c889bacc2.4223.6.5 [call out=85b] v0'0 uv7 ondisk = 0) v8 ==== 194+0+85 (crc 0 0 0) 0x7fd29403d040 con 0x7fd284011040 2023-10-05T19:30:22.435 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.118+0000 7fd29bc01e40 1 -- 172.21.15.167:0/444809523 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:151 8.9 8:96d98480:users.uid::foo.buckets:head [call user.set_buckets_info in=207b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e32) v8 -- 0x5570999e0140 con 0x7fd28c007030 2023-10-05T19:30:22.435 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.118+0000 7fd298f35700 1 -- 172.21.15.167:0/444809523 <== osd.2 v2:172.21.15.167:6816/30037 20 ==== osd_op_reply(151 foo.buckets [call] v32'47 uv47 ondisk = 0) v8 ==== 155+0+0 (crc 0 0 0) 0x7fd28402f040 con 0x7fd28c007030 2023-10-05T19:30:22.435 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.118+0000 7fd29bc01e40 20 get_system_obj_state: rctx=0x7fff991e3588 obj=default.rgw.meta:root:myfoo state=0x5570997cc0b0 s->prefetch_data=0 2023-10-05T19:30:22.435 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.118+0000 7fd29bc01e40 10 cache get: name=default.rgw.meta+root+myfoo : type miss (requested=0x16, cached=0x13) 2023-10-05T19:30:22.435 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.118+0000 7fd29bc01e40 1 -- 172.21.15.167:0/444809523 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:152 8.9 8:91b71841:root::myfoo:head [call version.read in=11b,getxattrs,stat] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32) v8 -- 0x5570999e0140 con 0x7fd28c007030 2023-10-05T19:30:22.435 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.118+0000 7fd298f35700 1 -- 172.21.15.167:0/444809523 <== osd.2 v2:172.21.15.167:6816/30037 21 ==== osd_op_reply(152 myfoo [call out=48b,getxattrs out=75b,stat out=16b] v0'0 uv42 ondisk = 0) v8 ==== 233+0+139 (crc 0 0 0) 0x7fd28402f040 con 0x7fd28c007030 2023-10-05T19:30:22.435 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.118+0000 7fd29bc01e40 10 cache put: name=default.rgw.meta+root+myfoo info.flags=0x16 2023-10-05T19:30:22.435 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.118+0000 7fd29bc01e40 10 moving default.rgw.meta+root+myfoo to cache LRU end 2023-10-05T19:30:22.435 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.118+0000 7fd29bc01e40 10 updating xattr: name=ceph.objclass.version bl.length()=42 2023-10-05T19:30:22.435 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.118+0000 7fd29bc01e40 20 get_system_obj_state: s->obj_tag was set empty 2023-10-05T19:30:22.436 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.118+0000 7fd29bc01e40 10 cache get: name=default.rgw.meta+root+myfoo : hit (requested=0x11, cached=0x17) 2023-10-05T19:30:22.436 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.118+0000 7fd29bc01e40 10 removing default.rgw.meta+root+myfoo from cache 2023-10-05T19:30:22.436 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.118+0000 7fd29bc01e40 10 distributing notification oid=default.rgw.control:notify.6 cni=[op: 1, obj: default.rgw.meta:root:myfoo, ofs0, ns] 2023-10-05T19:30:22.436 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.118+0000 7fd29bc01e40 1 -- 172.21.15.167:0/444809523 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:153 7.14 7:2b04a3e9:::notify.6:head [notify cookie 93942101932880 in=161b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32) v8 -- 0x5570999c6770 con 0x7fd284011040 2023-10-05T19:30:22.436 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.122+0000 7fd299f37700 1 -- 172.21.15.167:0/444809523 <== osd.0 v2:172.21.15.118:6800/27690 43 ==== watch-notify(notify (1) cookie 93942101264400 notify 137438953516 ret 0) v3 ==== 191+0+0 (crc 0 0 0) 0x7fd294086040 con 0x7fd284011040 2023-10-05T19:30:22.436 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.122+0000 7fd292ffd700 10 rgw watcher librados: RGWWatcher::handle_notify() notify_id 137438953516 cookie 93942101264400 notifier 4755 bl.length()=149 2023-10-05T19:30:22.436 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.122+0000 7fd299f37700 1 -- 172.21.15.167:0/444809523 <== osd.0 v2:172.21.15.118:6800/27690 44 ==== osd_op_reply(153 notify.6 [notify cookie 93942101932880 out=8b] v0'0 uv264 ondisk = 0) v8 ==== 152+0+8 (crc 0 0 0) 0x7fd29403d040 con 0x7fd284011040 2023-10-05T19:30:22.436 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.122+0000 7fd292ffd700 1 -- 172.21.15.167:0/444809523 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:154 7.14 7:2b04a3e9:::notify.6:head [notify-ack in=20b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32) v8 -- 0x7fd278003d90 con 0x7fd284011040 2023-10-05T19:30:22.436 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.122+0000 7fd299f37700 1 -- 172.21.15.167:0/444809523 <== osd.0 v2:172.21.15.118:6800/27690 45 ==== osd_op_reply(154 notify.6 [notify-ack] v0'0 uv264 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7fd29403d040 con 0x7fd284011040 2023-10-05T19:30:22.436 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.122+0000 7fd299f37700 1 -- 172.21.15.167:0/444809523 <== osd.0 v2:172.21.15.118:6800/27690 46 ==== watch-notify(notify_complete (2) cookie 93942101932880 notify 137438953516 ret 0) v3 ==== 42+0+48 (crc 0 0 0) 0x7fd294061890 con 0x7fd284011040 2023-10-05T19:30:22.436 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.122+0000 7fd29bc01e40 1 -- 172.21.15.167:0/444809523 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:155 8.9 8:91b71841:root::myfoo:head [call version.check_conds in=122b,call version.set in=58b,delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e32) v8 -- 0x5570999e0140 con 0x7fd28c007030 2023-10-05T19:30:22.437 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.126+0000 7fd298f35700 1 -- 172.21.15.167:0/444809523 <== osd.2 v2:172.21.15.167:6816/30037 22 ==== osd_op_reply(155 myfoo [call,call,delete] v32'48 uv48 ondisk = 0) v8 ==== 233+0+0 (crc 0 0 0) 0x7fd28402f040 con 0x7fd28c007030 2023-10-05T19:30:22.437 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.126+0000 7fd29bc01e40 10 removing default.rgw.meta+root+.bucket.meta.myfoo:9a9df965-6612-45d4-a45f-6e3c889bacc2.4223.6 from cache 2023-10-05T19:30:22.437 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.126+0000 7fd29bc01e40 10 distributing notification oid=default.rgw.control:notify.7 cni=[op: 1, obj: default.rgw.meta:root:.bucket.meta.myfoo:9a9df965-6612-45d4-a45f-6e3c889bacc2.4223.6, ofs0, ns] 2023-10-05T19:30:22.437 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.126+0000 7fd29bc01e40 1 -- 172.21.15.167:0/444809523 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:156 7.9 7:93e5b521:::notify.7:head [notify cookie 93942101932880 in=218b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32) v8 -- 0x5570999c6980 con 0x7fd284011040 2023-10-05T19:30:22.437 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.126+0000 7fd299f37700 1 -- 172.21.15.167:0/444809523 <== osd.0 v2:172.21.15.118:6800/27690 47 ==== watch-notify(notify (1) cookie 93942101265504 notify 137438953517 ret 0) v3 ==== 248+0+0 (crc 0 0 0) 0x7fd294061890 con 0x7fd284011040 2023-10-05T19:30:22.437 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.126+0000 7fd292ffd700 10 rgw watcher librados: RGWWatcher::handle_notify() notify_id 137438953517 cookie 93942101265504 notifier 4755 bl.length()=206 2023-10-05T19:30:22.437 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.126+0000 7fd292ffd700 1 -- 172.21.15.167:0/444809523 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:157 7.9 7:93e5b521:::notify.7:head [notify-ack in=20b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32) v8 -- 0x7fd278003d90 con 0x7fd284011040 2023-10-05T19:30:22.437 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.126+0000 7fd299f37700 1 -- 172.21.15.167:0/444809523 <== osd.0 v2:172.21.15.118:6800/27690 48 ==== osd_op_reply(156 notify.7 [notify cookie 93942101932880 out=8b] v0'0 uv264 ondisk = 0) v8 ==== 152+0+8 (crc 0 0 0) 0x7fd29403d040 con 0x7fd284011040 2023-10-05T19:30:22.437 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.126+0000 7fd299f37700 1 -- 172.21.15.167:0/444809523 <== osd.0 v2:172.21.15.118:6800/27690 49 ==== osd_op_reply(157 notify.7 [notify-ack] v0'0 uv264 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7fd29403d040 con 0x7fd284011040 2023-10-05T19:30:22.437 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.126+0000 7fd299f37700 1 -- 172.21.15.167:0/444809523 <== osd.0 v2:172.21.15.118:6800/27690 50 ==== watch-notify(notify_complete (2) cookie 93942101932880 notify 137438953517 ret 0) v3 ==== 42+0+48 (crc 0 0 0) 0x7fd294061a20 con 0x7fd284011040 2023-10-05T19:30:22.437 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.126+0000 7fd29bc01e40 1 -- 172.21.15.167:0/444809523 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:158 8.b 8:d4944359:root::.bucket.meta.myfoo%3a9a9df965-6612-45d4-a45f-6e3c889bacc2.4223.6:head [call version.check_conds in=122b,call version.inc in=38b,delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e32) v8 -- 0x5570999e0140 con 0x55709992a5c0 2023-10-05T19:30:22.437 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.130+0000 7fd299736700 1 -- 172.21.15.167:0/444809523 <== osd.3 v2:172.21.15.167:6808/30036 38 ==== osd_op_reply(158 .bucket.meta.myfoo:9a9df965-6612-45d4-a45f-6e3c889bacc2.4223.6 [call,call,delete] v32'6 uv6 ondisk = 0) v8 ==== 290+0+0 (crc 0 0 0) 0x7fd28c07a040 con 0x55709992a5c0 2023-10-05T19:30:22.438 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.130+0000 7fd29bc01e40 1 -- 172.21.15.167:0/444809523 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:159 4.3 4:c2471d59:::.dir.9a9df965-6612-45d4-a45f-6e3c889bacc2.4223.6.0:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e32) v8 -- 0x5570999e0050 con 0x5570998f2620 2023-10-05T19:30:22.438 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.130+0000 7fd29bc01e40 1 -- 172.21.15.167:0/444809523 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:160 4.1f 4:fac78d74:::.dir.9a9df965-6612-45d4-a45f-6e3c889bacc2.4223.6.1:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e32) v8 -- 0x5570999c31c0 con 0x55709992a5c0 2023-10-05T19:30:22.438 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.130+0000 7fd29bc01e40 1 -- 172.21.15.167:0/444809523 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:161 4.3 4:c00d743b:::.dir.9a9df965-6612-45d4-a45f-6e3c889bacc2.4223.6.2:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e32) v8 -- 0x5570999ccaa0 con 0x5570998f2620 2023-10-05T19:30:22.438 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.130+0000 7fd29bc01e40 1 -- 172.21.15.167:0/444809523 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:162 4.2b 4:d72453a1:::.dir.9a9df965-6612-45d4-a45f-6e3c889bacc2.4223.6.3:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e32) v8 -- 0x5570999cce90 con 0x5570998f2620 2023-10-05T19:30:22.438 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.130+0000 7fd29bc01e40 1 -- 172.21.15.167:0/444809523 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:163 4.2a 4:576abad7:::.dir.9a9df965-6612-45d4-a45f-6e3c889bacc2.4223.6.4:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e32) v8 -- 0x5570999c31c0 con 0x5570998f2620 2023-10-05T19:30:22.438 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.130+0000 7fd29bc01e40 1 -- 172.21.15.167:0/444809523 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:164 4.2d 4:b45bfada:::.dir.9a9df965-6612-45d4-a45f-6e3c889bacc2.4223.6.5:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e32) v8 -- 0x5570999ccaa0 con 0x7fd284011040 2023-10-05T19:30:22.438 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.130+0000 7fd29bc01e40 1 -- 172.21.15.167:0/444809523 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:165 4.33 4:ceb97e7a:::.dir.9a9df965-6612-45d4-a45f-6e3c889bacc2.4223.6.6:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e32) v8 -- 0x5570999cd730 con 0x7fd284011040 2023-10-05T19:30:22.438 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.130+0000 7fd29bc01e40 1 -- 172.21.15.167:0/444809523 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:166 4.5 4:a170829b:::.dir.9a9df965-6612-45d4-a45f-6e3c889bacc2.4223.6.7:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e32) v8 -- 0x5570999cce90 con 0x55709992a5c0 2023-10-05T19:30:22.438 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.130+0000 7fd29bc01e40 1 -- 172.21.15.167:0/444809523 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:167 4.8 4:13a6ed6b:::.dir.9a9df965-6612-45d4-a45f-6e3c889bacc2.4223.6.8:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e32) v8 -- 0x5570999ccaa0 con 0x55709992a5c0 2023-10-05T19:30:22.438 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.130+0000 7fd29bc01e40 1 -- 172.21.15.167:0/444809523 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:168 4.7 4:e2c155b5:::.dir.9a9df965-6612-45d4-a45f-6e3c889bacc2.4223.6.9:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e32) v8 -- 0x5570999c31c0 con 0x7fd284011040 2023-10-05T19:30:22.438 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.130+0000 7fd29bc01e40 1 -- 172.21.15.167:0/444809523 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:169 4.2a 4:55a3e9a6:::.dir.9a9df965-6612-45d4-a45f-6e3c889bacc2.4223.6.10:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e32) v8 -- 0x5570999cd730 con 0x5570998f2620 2023-10-05T19:30:22.439 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.134+0000 7fd299f37700 1 -- 172.21.15.167:0/444809523 <== osd.1 v2:172.21.15.167:6800/30035 55 ==== osd_op_reply(163 .dir.9a9df965-6612-45d4-a45f-6e3c889bacc2.4223.6.4 [delete] v32'7 uv7 ondisk = 0) v8 ==== 194+0+0 (crc 0 0 0) 0x7fd29403d040 con 0x5570998f2620 2023-10-05T19:30:22.439 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.134+0000 7fd299f37700 1 -- 172.21.15.167:0/444809523 <== osd.1 v2:172.21.15.167:6800/30035 56 ==== osd_op_reply(159 .dir.9a9df965-6612-45d4-a45f-6e3c889bacc2.4223.6.0 [delete] v32'3 uv3 ondisk = 0) v8 ==== 194+0+0 (crc 0 0 0) 0x7fd29403d040 con 0x5570998f2620 2023-10-05T19:30:22.439 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.134+0000 7fd299f37700 1 -- 172.21.15.167:0/444809523 <== osd.1 v2:172.21.15.167:6800/30035 57 ==== osd_op_reply(169 .dir.9a9df965-6612-45d4-a45f-6e3c889bacc2.4223.6.10 [delete] v32'8 uv8 ondisk = 0) v8 ==== 195+0+0 (crc 0 0 0) 0x7fd29403d040 con 0x5570998f2620 2023-10-05T19:30:22.439 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.134+0000 7fd299f37700 1 -- 172.21.15.167:0/444809523 <== osd.0 v2:172.21.15.118:6800/27690 51 ==== osd_op_reply(164 .dir.9a9df965-6612-45d4-a45f-6e3c889bacc2.4223.6.5 [delete] v32'8 uv8 ondisk = 0) v8 ==== 194+0+0 (crc 0 0 0) 0x7fd29403d040 con 0x7fd284011040 2023-10-05T19:30:22.439 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.134+0000 7fd299736700 1 -- 172.21.15.167:0/444809523 <== osd.3 v2:172.21.15.167:6808/30036 39 ==== osd_op_reply(160 .dir.9a9df965-6612-45d4-a45f-6e3c889bacc2.4223.6.1 [delete] v32'8 uv8 ondisk = 0) v8 ==== 194+0+0 (crc 0 0 0) 0x7fd28c07a040 con 0x55709992a5c0 2023-10-05T19:30:22.439 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.134+0000 7fd299736700 1 -- 172.21.15.167:0/444809523 <== osd.3 v2:172.21.15.167:6808/30036 40 ==== osd_op_reply(167 .dir.9a9df965-6612-45d4-a45f-6e3c889bacc2.4223.6.8 [delete] v32'8 uv8 ondisk = 0) v8 ==== 194+0+0 (crc 0 0 0) 0x7fd28c07a040 con 0x55709992a5c0 2023-10-05T19:30:22.439 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.134+0000 7fd299f37700 1 -- 172.21.15.167:0/444809523 <== osd.0 v2:172.21.15.118:6800/27690 52 ==== osd_op_reply(168 .dir.9a9df965-6612-45d4-a45f-6e3c889bacc2.4223.6.9 [delete] v32'2 uv2 ondisk = 0) v8 ==== 194+0+0 (crc 0 0 0) 0x7fd29403d040 con 0x7fd284011040 2023-10-05T19:30:22.439 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.134+0000 7fd299736700 1 -- 172.21.15.167:0/444809523 <== osd.3 v2:172.21.15.167:6808/30036 41 ==== osd_op_reply(166 .dir.9a9df965-6612-45d4-a45f-6e3c889bacc2.4223.6.7 [delete] v32'4 uv4 ondisk = 0) v8 ==== 194+0+0 (crc 0 0 0) 0x7fd28c07a040 con 0x55709992a5c0 2023-10-05T19:30:22.439 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.138+0000 7fd299f37700 1 -- 172.21.15.167:0/444809523 <== osd.1 v2:172.21.15.167:6800/30035 58 ==== osd_op_reply(162 .dir.9a9df965-6612-45d4-a45f-6e3c889bacc2.4223.6.3 [delete] v32'6 uv6 ondisk = 0) v8 ==== 194+0+0 (crc 0 0 0) 0x7fd29403d040 con 0x5570998f2620 2023-10-05T19:30:22.439 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.138+0000 7fd299f37700 1 -- 172.21.15.167:0/444809523 <== osd.1 v2:172.21.15.167:6800/30035 59 ==== osd_op_reply(161 .dir.9a9df965-6612-45d4-a45f-6e3c889bacc2.4223.6.2 [delete] v32'4 uv4 ondisk = 0) v8 ==== 194+0+0 (crc 0 0 0) 0x7fd29403d040 con 0x5570998f2620 2023-10-05T19:30:22.439 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.138+0000 7fd299f37700 1 -- 172.21.15.167:0/444809523 <== osd.0 v2:172.21.15.118:6800/27690 53 ==== osd_op_reply(165 .dir.9a9df965-6612-45d4-a45f-6e3c889bacc2.4223.6.6 [delete] v32'8 uv8 ondisk = 0) v8 ==== 194+0+0 (crc 0 0 0) 0x7fd29403d040 con 0x7fd284011040 2023-10-05T19:30:22.439 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.138+0000 7fd29bc01e40 1 -- 172.21.15.167:0/444809523 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:170 6.1a 6:58522b81:::pubsub..bucket.myfoo%2f9a9df965-6612-45d4-a45f-6e3c889bacc2.4223.6:head [getxattrs,stat] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32) v8 -- 0x5570999e0050 con 0x55709992a5c0 2023-10-05T19:30:22.440 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.138+0000 7fd299736700 1 -- 172.21.15.167:0/444809523 <== osd.3 v2:172.21.15.167:6808/30036 42 ==== osd_op_reply(170 pubsub..bucket.myfoo/9a9df965-6612-45d4-a45f-6e3c889bacc2.4223.6 [getxattrs,stat] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 ==== 250+0+0 (crc 0 0 0) 0x7fd28c07a040 con 0x55709992a5c0 2023-10-05T19:30:22.440 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.138+0000 7fd29bc01e40 10 removing default.rgw.log++pubsub..bucket.myfoo/9a9df965-6612-45d4-a45f-6e3c889bacc2.4223.6 from cache 2023-10-05T19:30:22.440 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.138+0000 7fd29bc01e40 10 distributing notification oid=default.rgw.control:notify.6 cni=[op: 1, obj: default.rgw.log:pubsub..bucket.myfoo/9a9df965-6612-45d4-a45f-6e3c889bacc2.4223.6, ofs0, ns] 2023-10-05T19:30:22.440 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.138+0000 7fd29bc01e40 1 -- 172.21.15.167:0/444809523 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:171 7.14 7:2b04a3e9:::notify.6:head [notify cookie 93942101932608 in=215b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32) v8 -- 0x5570999d8370 con 0x7fd284011040 2023-10-05T19:30:22.440 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.138+0000 7fd299f37700 1 -- 172.21.15.167:0/444809523 <== osd.0 v2:172.21.15.118:6800/27690 54 ==== watch-notify(notify (1) cookie 93942101264400 notify 137438953518 ret 0) v3 ==== 245+0+0 (crc 0 0 0) 0x7fd294061a20 con 0x7fd284011040 2023-10-05T19:30:22.440 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.138+0000 7fd2927fc700 10 rgw watcher librados: RGWWatcher::handle_notify() notify_id 137438953518 cookie 93942101264400 notifier 4755 bl.length()=203 2023-10-05T19:30:22.440 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.138+0000 7fd299f37700 1 -- 172.21.15.167:0/444809523 <== osd.0 v2:172.21.15.118:6800/27690 55 ==== osd_op_reply(171 notify.6 [notify cookie 93942101932608 out=8b] v0'0 uv264 ondisk = 0) v8 ==== 152+0+8 (crc 0 0 0) 0x7fd29403d040 con 0x7fd284011040 2023-10-05T19:30:22.440 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.138+0000 7fd2927fc700 1 -- 172.21.15.167:0/444809523 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:172 7.14 7:2b04a3e9:::notify.6:head [notify-ack in=20b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32) v8 -- 0x7fd1e80032e0 con 0x7fd284011040 2023-10-05T19:30:22.440 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.138+0000 7fd299f37700 1 -- 172.21.15.167:0/444809523 <== osd.0 v2:172.21.15.118:6800/27690 56 ==== watch-notify(notify_complete (2) cookie 93942101932608 notify 137438953518 ret 0) v3 ==== 42+0+48 (crc 0 0 0) 0x7fd294061db0 con 0x7fd284011040 2023-10-05T19:30:22.440 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.138+0000 7fd299f37700 1 -- 172.21.15.167:0/444809523 <== osd.0 v2:172.21.15.118:6800/27690 57 ==== osd_op_reply(172 notify.6 [notify-ack] v0'0 uv264 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7fd29403d040 con 0x7fd284011040 2023-10-05T19:30:22.440 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.138+0000 7fd29bc01e40 1 -- 172.21.15.167:0/444809523 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:173 6.1a 6:58522b81:::pubsub..bucket.myfoo%2f9a9df965-6612-45d4-a45f-6e3c889bacc2.4223.6:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e32) v8 -- 0x5570999d7a40 con 0x55709992a5c0 2023-10-05T19:30:22.440 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.142+0000 7fd299736700 1 -- 172.21.15.167:0/444809523 <== osd.3 v2:172.21.15.167:6808/30036 43 ==== osd_op_reply(173 pubsub..bucket.myfoo/9a9df965-6612-45d4-a45f-6e3c889bacc2.4223.6 [delete] v32'33 uv31 ondisk = -2 ((2) No such file or directory)) v8 ==== 208+0+0 (crc 0 0 0) 0x7fd28c07a040 con 0x55709992a5c0 2023-10-05T19:30:22.441 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.142+0000 7fd29bc01e40 1 -- 172.21.15.167:0/444809523 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:174 8.9 8:96d98480:users.uid::foo.buckets:head [call user.remove_bucket in=58b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e32) v8 -- 0x5570999d7a40 con 0x7fd28c007030 2023-10-05T19:30:22.441 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.142+0000 7fd298f35700 1 -- 172.21.15.167:0/444809523 <== osd.2 v2:172.21.15.167:6816/30037 23 ==== osd_op_reply(174 foo.buckets [call] v32'49 uv49 ondisk = 0) v8 ==== 155+0+0 (crc 0 0 0) 0x7fd28402f040 con 0x7fd28c007030 2023-10-05T19:30:22.441 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.142+0000 7fd29bc01e40 10 removing key index: 9te6NH5mcdcq0Tc5i8i1 2023-10-05T19:30:22.441 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.142+0000 7fd29bc01e40 10 distributing notification oid=default.rgw.control:notify.3 cni=[op: 1, obj: default.rgw.meta:users.keys:9te6NH5mcdcq0Tc5i8i1, ofs0, ns] 2023-10-05T19:30:22.441 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.142+0000 7fd29bc01e40 1 -- 172.21.15.167:0/444809523 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:175 7.e 7:7759931f:::notify.3:head [notify cookie 93942101966928 in=182b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32) v8 -- 0x5570999d8370 con 0x7fd28c007030 2023-10-05T19:30:22.441 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.146+0000 7fd298f35700 1 -- 172.21.15.167:0/444809523 <== osd.2 v2:172.21.15.167:6816/30037 24 ==== watch-notify(notify (1) cookie 93942101250208 notify 137438953492 ret 0) v3 ==== 212+0+0 (crc 0 0 0) 0x5570999496d0 con 0x7fd28c007030 2023-10-05T19:30:22.441 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.146+0000 7fd2927fc700 10 rgw watcher librados: RGWWatcher::handle_notify() notify_id 137438953492 cookie 93942101250208 notifier 4755 bl.length()=170 2023-10-05T19:30:22.441 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.146+0000 7fd2927fc700 1 -- 172.21.15.167:0/444809523 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:176 7.e 7:7759931f:::notify.3:head [notify-ack in=20b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32) v8 -- 0x7fd1e80032e0 con 0x7fd28c007030 2023-10-05T19:30:22.441 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.146+0000 7fd298f35700 1 -- 172.21.15.167:0/444809523 <== osd.2 v2:172.21.15.167:6816/30037 25 ==== osd_op_reply(175 notify.3 [notify cookie 93942101966928 out=8b] v0'0 uv264 ondisk = 0) v8 ==== 152+0+8 (crc 0 0 0) 0x7fd28402f040 con 0x7fd28c007030 2023-10-05T19:30:22.441 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.146+0000 7fd298f35700 1 -- 172.21.15.167:0/444809523 <== osd.2 v2:172.21.15.167:6816/30037 26 ==== osd_op_reply(176 notify.3 [notify-ack] v0'0 uv264 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7fd28402f040 con 0x7fd28c007030 2023-10-05T19:30:22.441 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.146+0000 7fd298f35700 1 -- 172.21.15.167:0/444809523 <== osd.2 v2:172.21.15.167:6816/30037 27 ==== watch-notify(notify_complete (2) cookie 93942101966928 notify 137438953492 ret 0) v3 ==== 42+0+48 (crc 0 0 0) 0x7fd284055070 con 0x7fd28c007030 2023-10-05T19:30:22.441 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.146+0000 7fd29bc01e40 1 -- 172.21.15.167:0/444809523 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:177 8.7 8:e4e0e270:users.keys::9te6NH5mcdcq0Tc5i8i1:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e32) v8 -- 0x5570999d7a40 con 0x7fd284011040 2023-10-05T19:30:22.442 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.146+0000 7fd299f37700 1 -- 172.21.15.167:0/444809523 <== osd.0 v2:172.21.15.118:6800/27690 58 ==== osd_op_reply(177 9te6NH5mcdcq0Tc5i8i1 [delete] v32'5 uv5 ondisk = 0) v8 ==== 164+0+0 (crc 0 0 0) 0x7fd29403d040 con 0x7fd284011040 2023-10-05T19:30:22.442 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.146+0000 7fd29bc01e40 10 removing email index: foo@foo.com 2023-10-05T19:30:22.442 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.146+0000 7fd29bc01e40 10 distributing notification oid=default.rgw.control:notify.3 cni=[op: 1, obj: default.rgw.meta:users.email:foo@foo.com, ofs0, ns] 2023-10-05T19:30:22.442 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.146+0000 7fd29bc01e40 1 -- 172.21.15.167:0/444809523 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:178 7.e 7:7759931f:::notify.3:head [notify cookie 93942101966928 in=174b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32) v8 -- 0x5570999d8370 con 0x7fd28c007030 2023-10-05T19:30:22.442 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.150+0000 7fd298f35700 1 -- 172.21.15.167:0/444809523 <== osd.2 v2:172.21.15.167:6816/30037 28 ==== watch-notify(notify (1) cookie 93942101250208 notify 137438953493 ret 0) v3 ==== 204+0+0 (crc 0 0 0) 0x7fd284055070 con 0x7fd28c007030 2023-10-05T19:30:22.442 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.150+0000 7fd292ffd700 10 rgw watcher librados: RGWWatcher::handle_notify() notify_id 137438953493 cookie 93942101250208 notifier 4755 bl.length()=162 2023-10-05T19:30:22.442 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.150+0000 7fd292ffd700 1 -- 172.21.15.167:0/444809523 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:179 7.e 7:7759931f:::notify.3:head [notify-ack in=20b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32) v8 -- 0x7fd278003d90 con 0x7fd28c007030 2023-10-05T19:30:22.442 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.150+0000 7fd298f35700 1 -- 172.21.15.167:0/444809523 <== osd.2 v2:172.21.15.167:6816/30037 29 ==== osd_op_reply(178 notify.3 [notify cookie 93942101966928 out=8b] v0'0 uv264 ondisk = 0) v8 ==== 152+0+8 (crc 0 0 0) 0x7fd28402f040 con 0x7fd28c007030 2023-10-05T19:30:22.442 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.150+0000 7fd298f35700 1 -- 172.21.15.167:0/444809523 <== osd.2 v2:172.21.15.167:6816/30037 30 ==== watch-notify(notify_complete (2) cookie 93942101966928 notify 137438953493 ret 0) v3 ==== 42+0+48 (crc 0 0 0) 0x557099949290 con 0x7fd28c007030 2023-10-05T19:30:22.442 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.150+0000 7fd298f35700 1 -- 172.21.15.167:0/444809523 <== osd.2 v2:172.21.15.167:6816/30037 31 ==== osd_op_reply(179 notify.3 [notify-ack] v0'0 uv264 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7fd28402f040 con 0x7fd28c007030 2023-10-05T19:30:22.442 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.150+0000 7fd29bc01e40 1 -- 172.21.15.167:0/444809523 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:180 8.3 8:c162d5d1:users.email::foo@foo.com:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e32) v8 -- 0x5570999c2cd0 con 0x5570998f2620 2023-10-05T19:30:22.443 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.150+0000 7fd299f37700 1 -- 172.21.15.167:0/444809523 <== osd.1 v2:172.21.15.167:6800/30035 60 ==== osd_op_reply(180 foo@foo.com [delete] v32'2 uv2 ondisk = 0) v8 ==== 155+0+0 (crc 0 0 0) 0x7fd29403d040 con 0x5570998f2620 2023-10-05T19:30:22.443 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.150+0000 7fd29bc01e40 10 removing user buckets index 2023-10-05T19:30:22.443 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.150+0000 7fd29bc01e40 10 distributing notification oid=default.rgw.control:notify.5 cni=[op: 1, obj: default.rgw.meta:users.uid:foo.buckets, ofs0, ns] 2023-10-05T19:30:22.443 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.150+0000 7fd29bc01e40 1 -- 172.21.15.167:0/444809523 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:181 7.3 7:c609908c:::notify.5:head [notify cookie 93942101932608 in=172b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32) v8 -- 0x5570999d8860 con 0x7fd284011040 2023-10-05T19:30:22.443 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.154+0000 7fd299f37700 1 -- 172.21.15.167:0/444809523 <== osd.0 v2:172.21.15.118:6800/27690 59 ==== watch-notify(notify (1) cookie 93942101258080 notify 137438953519 ret 0) v3 ==== 202+0+0 (crc 0 0 0) 0x7fd294061db0 con 0x7fd284011040 2023-10-05T19:30:22.443 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.154+0000 7fd2927fc700 10 rgw watcher librados: RGWWatcher::handle_notify() notify_id 137438953519 cookie 93942101258080 notifier 4755 bl.length()=160 2023-10-05T19:30:22.443 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.154+0000 7fd299f37700 1 -- 172.21.15.167:0/444809523 <== osd.0 v2:172.21.15.118:6800/27690 60 ==== osd_op_reply(181 notify.5 [notify cookie 93942101932608 out=8b] v0'0 uv264 ondisk = 0) v8 ==== 152+0+8 (crc 0 0 0) 0x7fd29403d040 con 0x7fd284011040 2023-10-05T19:30:22.443 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.154+0000 7fd2927fc700 1 -- 172.21.15.167:0/444809523 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:182 7.3 7:c609908c:::notify.5:head [notify-ack in=20b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32) v8 -- 0x7fd1e80032e0 con 0x7fd284011040 2023-10-05T19:30:22.443 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.154+0000 7fd299f37700 1 -- 172.21.15.167:0/444809523 <== osd.0 v2:172.21.15.118:6800/27690 61 ==== osd_op_reply(182 notify.5 [notify-ack] v0'0 uv264 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7fd29403d040 con 0x7fd284011040 2023-10-05T19:30:22.443 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.154+0000 7fd299f37700 1 -- 172.21.15.167:0/444809523 <== osd.0 v2:172.21.15.118:6800/27690 62 ==== watch-notify(notify_complete (2) cookie 93942101932608 notify 137438953519 ret 0) v3 ==== 42+0+48 (crc 0 0 0) 0x7fd2940335b0 con 0x7fd284011040 2023-10-05T19:30:22.443 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.154+0000 7fd29bc01e40 1 -- 172.21.15.167:0/444809523 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:183 8.9 8:96d98480:users.uid::foo.buckets:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e32) v8 -- 0x5570999c2cd0 con 0x7fd28c007030 2023-10-05T19:30:22.443 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.158+0000 7fd298f35700 1 -- 172.21.15.167:0/444809523 <== osd.2 v2:172.21.15.167:6816/30037 32 ==== osd_op_reply(183 foo.buckets [delete] v32'50 uv50 ondisk = 0) v8 ==== 155+0+0 (crc 0 0 0) 0x7fd28402f040 con 0x7fd28c007030 2023-10-05T19:30:22.444 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.158+0000 7fd29bc01e40 10 removing user index: foo 2023-10-05T19:30:22.444 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.158+0000 7fd29bc01e40 10 removing default.rgw.meta+users.uid+foo from cache 2023-10-05T19:30:22.444 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.158+0000 7fd29bc01e40 10 distributing notification oid=default.rgw.control:notify.4 cni=[op: 1, obj: default.rgw.meta:users.uid:foo, ofs0, ns] 2023-10-05T19:30:22.444 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.158+0000 7fd29bc01e40 1 -- 172.21.15.167:0/444809523 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:184 7.d 7:b4812045:::notify.4:head [notify cookie 93942101932608 in=164b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32) v8 -- 0x5570999ccf90 con 0x5570998f2620 2023-10-05T19:30:22.444 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.158+0000 7fd299f37700 1 -- 172.21.15.167:0/444809523 <== osd.1 v2:172.21.15.167:6800/30035 61 ==== watch-notify(notify (1) cookie 93942101256176 notify 137438953495 ret 0) v3 ==== 194+0+0 (crc 0 0 0) 0x7fd2940335b0 con 0x5570998f2620 2023-10-05T19:30:22.444 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.158+0000 7fd292ffd700 10 rgw watcher librados: RGWWatcher::handle_notify() notify_id 137438953495 cookie 93942101256176 notifier 4755 bl.length()=152 2023-10-05T19:30:22.444 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.158+0000 7fd292ffd700 1 -- 172.21.15.167:0/444809523 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:185 7.d 7:b4812045:::notify.4:head [notify-ack in=20b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32) v8 -- 0x7fd278003d90 con 0x5570998f2620 2023-10-05T19:30:22.444 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.158+0000 7fd299f37700 1 -- 172.21.15.167:0/444809523 <== osd.1 v2:172.21.15.167:6800/30035 62 ==== osd_op_reply(184 notify.4 [notify cookie 93942101932608 out=8b] v0'0 uv264 ondisk = 0) v8 ==== 152+0+8 (crc 0 0 0) 0x7fd29403d040 con 0x5570998f2620 2023-10-05T19:30:22.444 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.158+0000 7fd299f37700 1 -- 172.21.15.167:0/444809523 <== osd.1 v2:172.21.15.167:6800/30035 63 ==== osd_op_reply(185 notify.4 [notify-ack] v0'0 uv264 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7fd29403d040 con 0x5570998f2620 2023-10-05T19:30:22.444 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.158+0000 7fd299f37700 1 -- 172.21.15.167:0/444809523 <== osd.1 v2:172.21.15.167:6800/30035 64 ==== watch-notify(notify_complete (2) cookie 93942101932608 notify 137438953495 ret 0) v3 ==== 42+0+48 (crc 0 0 0) 0x7fd2940337e0 con 0x5570998f2620 2023-10-05T19:30:22.444 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.158+0000 7fd29bc01e40 1 -- 172.21.15.167:0/444809523 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:186 8.18 8:1e6ca5ed:users.uid::foo:head [call version.inc in=38b,delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e32) v8 -- 0x5570999ccaa0 con 0x5570998f2620 2023-10-05T19:30:22.444 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.162+0000 7fd299f37700 1 -- 172.21.15.167:0/444809523 <== osd.1 v2:172.21.15.167:6800/30035 65 ==== osd_op_reply(186 foo [call,delete] v32'20 uv20 ondisk = 0) v8 ==== 189+0+0 (crc 0 0 0) 0x7fd29403d040 con 0x5570998f2620 2023-10-05T19:30:22.445 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.162+0000 7fd29bc01e40 1 -- 172.21.15.167:0/444809523 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:187 7.1f 7:f95f44c2:::notify.0:head [watch unwatch cookie 93942101243424] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e32) v8 -- 0x557099907670 con 0x7fd28c007030 2023-10-05T19:30:22.445 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.162+0000 7fd298f35700 1 -- 172.21.15.167:0/444809523 <== osd.2 v2:172.21.15.167:6816/30037 33 ==== osd_op_reply(187 notify.0 [watch unwatch cookie 93942101243424] v32'266 uv264 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7fd28402f040 con 0x7fd28c007030 2023-10-05T19:30:22.445 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.162+0000 7fd29bc01e40 20 remove_watcher() i=0 2023-10-05T19:30:22.445 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.162+0000 7fd29bc01e40 2 removed watcher, disabling cache 2023-10-05T19:30:22.445 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.162+0000 7fd29bc01e40 1 -- 172.21.15.167:0/444809523 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:188 7.0 7:05bf5b68:::notify.1:head [watch unwatch cookie 93942101246944] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e32) v8 -- 0x557099907670 con 0x5570998f2620 2023-10-05T19:30:22.445 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.166+0000 7fd299f37700 1 -- 172.21.15.167:0/444809523 <== osd.1 v2:172.21.15.167:6800/30035 66 ==== osd_op_reply(188 notify.1 [watch unwatch cookie 93942101246944] v32'266 uv264 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7fd29403d040 con 0x5570998f2620 2023-10-05T19:30:22.445 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.166+0000 7fd29bc01e40 20 remove_watcher() i=1 2023-10-05T19:30:22.445 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.166+0000 7fd29bc01e40 1 -- 172.21.15.167:0/444809523 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:189 7.15 7:a93a5511:::notify.2:head [watch unwatch cookie 93942101248048] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e32) v8 -- 0x557099907670 con 0x55709992a5c0 2023-10-05T19:30:22.445 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.170+0000 7fd299736700 1 -- 172.21.15.167:0/444809523 <== osd.3 v2:172.21.15.167:6808/30036 44 ==== osd_op_reply(189 notify.2 [watch unwatch cookie 93942101248048] v32'266 uv264 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7fd28c07a040 con 0x55709992a5c0 2023-10-05T19:30:22.445 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.170+0000 7fd29bc01e40 20 remove_watcher() i=2 2023-10-05T19:30:22.445 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.170+0000 7fd29bc01e40 1 -- 172.21.15.167:0/444809523 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:190 7.e 7:7759931f:::notify.3:head [watch unwatch cookie 93942101250208] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e32) v8 -- 0x557099907670 con 0x7fd28c007030 2023-10-05T19:30:22.445 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.170+0000 7fd298f35700 1 -- 172.21.15.167:0/444809523 <== osd.2 v2:172.21.15.167:6816/30037 34 ==== osd_op_reply(190 notify.3 [watch unwatch cookie 93942101250208] v32'266 uv264 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7fd28402f040 con 0x7fd28c007030 2023-10-05T19:30:22.446 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.174+0000 7fd29bc01e40 20 remove_watcher() i=3 2023-10-05T19:30:22.446 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.174+0000 7fd29bc01e40 1 -- 172.21.15.167:0/444809523 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:191 7.d 7:b4812045:::notify.4:head [watch unwatch cookie 93942101256176] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e32) v8 -- 0x557099907670 con 0x5570998f2620 2023-10-05T19:30:22.446 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.174+0000 7fd299f37700 1 -- 172.21.15.167:0/444809523 <== osd.1 v2:172.21.15.167:6800/30035 67 ==== osd_op_reply(191 notify.4 [watch unwatch cookie 93942101256176] v32'266 uv264 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7fd29403d040 con 0x5570998f2620 2023-10-05T19:30:22.446 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.174+0000 7fd29bc01e40 20 remove_watcher() i=4 2023-10-05T19:30:22.446 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.174+0000 7fd29bc01e40 1 -- 172.21.15.167:0/444809523 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:192 7.3 7:c609908c:::notify.5:head [watch unwatch cookie 93942101258080] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e32) v8 -- 0x557099907670 con 0x7fd284011040 2023-10-05T19:30:22.446 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.178+0000 7fd299f37700 1 -- 172.21.15.167:0/444809523 <== osd.0 v2:172.21.15.118:6800/27690 63 ==== osd_op_reply(192 notify.5 [watch unwatch cookie 93942101258080] v32'266 uv264 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7fd29403d040 con 0x7fd284011040 2023-10-05T19:30:22.446 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.178+0000 7fd29bc01e40 20 remove_watcher() i=5 2023-10-05T19:30:22.446 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.178+0000 7fd29bc01e40 1 -- 172.21.15.167:0/444809523 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:193 7.14 7:2b04a3e9:::notify.6:head [watch unwatch cookie 93942101264400] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e32) v8 -- 0x557099907670 con 0x7fd284011040 2023-10-05T19:30:22.446 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.182+0000 7fd299f37700 1 -- 172.21.15.167:0/444809523 <== osd.0 v2:172.21.15.118:6800/27690 64 ==== osd_op_reply(193 notify.6 [watch unwatch cookie 93942101264400] v32'266 uv264 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7fd29403d040 con 0x7fd284011040 2023-10-05T19:30:22.446 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.182+0000 7fd29bc01e40 20 remove_watcher() i=6 2023-10-05T19:30:22.446 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.182+0000 7fd29bc01e40 1 -- 172.21.15.167:0/444809523 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:194 7.9 7:93e5b521:::notify.7:head [watch unwatch cookie 93942101265504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e32) v8 -- 0x557099907670 con 0x7fd284011040 2023-10-05T19:30:22.447 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.186+0000 7fd299f37700 1 -- 172.21.15.167:0/444809523 <== osd.0 v2:172.21.15.118:6800/27690 65 ==== osd_op_reply(194 notify.7 [watch unwatch cookie 93942101265504] v32'266 uv264 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7fd29403d040 con 0x7fd284011040 2023-10-05T19:30:22.447 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.186+0000 7fd29bc01e40 20 remove_watcher() i=7 2023-10-05T19:30:22.447 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.186+0000 7fd26a7e4700 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2023-10-05T19:30:22.447 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.186+0000 7fd29bc01e40 1 -- 172.21.15.167:0/444809523 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:195 6.4 6:22d26bf9:::data_loggenerations_metadata:head [watch unwatch cookie 93942101268080] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e32) v8 -- 0x557099907670 con 0x5570998f2620 2023-10-05T19:30:22.447 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.190+0000 7fd299f37700 1 -- 172.21.15.167:0/444809523 <== osd.1 v2:172.21.15.167:6800/30035 68 ==== osd_op_reply(195 data_loggenerations_metadata [watch unwatch cookie 93942101268080] v32'259 uv1 ondisk = 0) v8 ==== 172+0+0 (crc 0 0 0) 0x7fd29403d040 con 0x5570998f2620 2023-10-05T19:30:22.447 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.190+0000 7fd29bc01e40 1 -- 172.21.15.167:0/444809523 >> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] conn(0x7fd284011040 msgr2=0x7fd284014950 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-05T19:30:22.447 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.190+0000 7fd29bc01e40 1 --2- 172.21.15.167:0/444809523 >> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] conn(0x7fd284011040 0x7fd284014950 crc :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-05T19:30:22.447 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.190+0000 7fd29bc01e40 1 -- 172.21.15.167:0/444809523 >> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] conn(0x5570998f2620 msgr2=0x557099926710 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-05T19:30:22.447 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.190+0000 7fd29bc01e40 1 --2- 172.21.15.167:0/444809523 >> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] conn(0x5570998f2620 0x557099926710 crc :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-05T19:30:22.447 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.190+0000 7fd29bc01e40 1 -- 172.21.15.167:0/444809523 >> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] conn(0x7fd28c007030 msgr2=0x7fd28c026880 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-05T19:30:22.447 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.190+0000 7fd29bc01e40 1 --2- 172.21.15.167:0/444809523 >> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] conn(0x7fd28c007030 0x7fd28c026880 crc :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-05T19:30:22.447 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.190+0000 7fd29bc01e40 1 -- 172.21.15.167:0/444809523 >> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] conn(0x55709992a5c0 msgr2=0x55709992c9a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-05T19:30:22.448 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.190+0000 7fd29bc01e40 1 --2- 172.21.15.167:0/444809523 >> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] conn(0x55709992a5c0 0x55709992c9a0 crc :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-05T19:30:22.448 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.190+0000 7fd29bc01e40 1 -- 172.21.15.167:0/444809523 >> [v2:172.21.15.167:6824/29889,v1:172.21.15.167:6825/29889] conn(0x7fd27c030bf0 msgr2=0x7fd27c0330a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-05T19:30:22.448 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.190+0000 7fd29bc01e40 1 --2- 172.21.15.167:0/444809523 >> [v2:172.21.15.167:6824/29889,v1:172.21.15.167:6825/29889] conn(0x7fd27c030bf0 0x7fd27c0330a0 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7fd2840016f0 tx=0x7fd2840067f0 comp rx=0 tx=0).stop 2023-10-05T19:30:22.448 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.190+0000 7fd29bc01e40 1 -- 172.21.15.167:0/444809523 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5570998f1d10 msgr2=0x55709991ac60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-05T19:30:22.448 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.190+0000 7fd299f37700 1 -- 172.21.15.167:0/444809523 reap_dead start 2023-10-05T19:30:22.448 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.190+0000 7fd29bc01e40 1 --2- 172.21.15.167:0/444809523 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5570998f1d10 0x55709991ac60 secure :-1 s=READY pgs=665 cs=0 l=1 rev1=1 crypto rx=0x7fd28c01e040 tx=0x7fd28c003050 comp rx=0 tx=0).stop 2023-10-05T19:30:22.448 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.190+0000 7fd29bc01e40 1 -- 172.21.15.167:0/444809523 shutdown_connections 2023-10-05T19:30:22.448 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.190+0000 7fd29bc01e40 1 --2- 172.21.15.167:0/444809523 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5570998f1d10 0x55709991ac60 unknown :-1 s=CLOSED pgs=665 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-05T19:30:22.448 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.190+0000 7fd29bc01e40 1 -- 172.21.15.167:0/444809523 >> 172.21.15.167:0/444809523 conn(0x5570998ed730 msgr2=0x55709990b3d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-05T19:30:22.448 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.190+0000 7fd29bc01e40 1 -- 172.21.15.167:0/444809523 shutdown_connections 2023-10-05T19:30:22.448 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.190+0000 7fd29bc01e40 1 -- 172.21.15.167:0/444809523 wait complete. 2023-10-05T19:30:22.448 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.190+0000 7fd29bc01e40 1 -- 172.21.15.167:0/832409700 >> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] conn(0x557099956f50 msgr2=0x557099959330 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-05T19:30:22.449 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.190+0000 7fd29bc01e40 1 --2- 172.21.15.167:0/832409700 >> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] conn(0x557099956f50 0x557099959330 crc :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-05T19:30:22.449 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.190+0000 7fd29bc01e40 1 -- 172.21.15.167:0/832409700 >> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] conn(0x557099953820 msgr2=0x557099955c30 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-05T19:30:22.449 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.190+0000 7fd29bc01e40 1 --2- 172.21.15.167:0/832409700 >> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] conn(0x557099953820 0x557099955c30 crc :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-05T19:30:22.449 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.190+0000 7fd29bc01e40 1 -- 172.21.15.167:0/832409700 >> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] conn(0x55709995e900 msgr2=0x557099960ce0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-05T19:30:22.449 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.190+0000 7fd29bc01e40 1 --2- 172.21.15.167:0/832409700 >> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] conn(0x55709995e900 0x557099960ce0 crc :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-05T19:30:22.449 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.190+0000 7fd29bc01e40 1 -- 172.21.15.167:0/832409700 >> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] conn(0x55709994f4c0 msgr2=0x55709995d0d0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-05T19:30:22.449 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.190+0000 7fd29bc01e40 1 --2- 172.21.15.167:0/832409700 >> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] conn(0x55709994f4c0 0x55709995d0d0 crc :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-05T19:30:22.449 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.190+0000 7fd29bc01e40 1 -- 172.21.15.167:0/832409700 >> [v2:172.21.15.167:6824/29889,v1:172.21.15.167:6825/29889] conn(0x7fd1e0030ca0 msgr2=0x7fd1e0033150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-05T19:30:22.449 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.190+0000 7fd29bc01e40 1 --2- 172.21.15.167:0/832409700 >> [v2:172.21.15.167:6824/29889,v1:172.21.15.167:6825/29889] conn(0x7fd1e0030ca0 0x7fd1e0033150 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x557099935230 tx=0x7fd29400e040 comp rx=0 tx=0).stop 2023-10-05T19:30:22.449 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.190+0000 7fd29bc01e40 1 -- 172.21.15.167:0/832409700 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5570999430a0 msgr2=0x557099948c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-05T19:30:22.449 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.190+0000 7fd29bc01e40 1 --2- 172.21.15.167:0/832409700 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5570999430a0 0x557099948c90 secure :-1 s=READY pgs=667 cs=0 l=1 rev1=1 crypto rx=0x7fd284041040 tx=0x7fd28403c000 comp rx=0 tx=0).stop 2023-10-05T19:30:22.449 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.190+0000 7fd299736700 1 -- 172.21.15.167:0/832409700 reap_dead start 2023-10-05T19:30:22.450 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.190+0000 7fd29bc01e40 1 -- 172.21.15.167:0/832409700 shutdown_connections 2023-10-05T19:30:22.450 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.190+0000 7fd29bc01e40 1 --2- 172.21.15.167:0/832409700 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5570999430a0 0x557099948c90 unknown :-1 s=CLOSED pgs=667 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-05T19:30:22.450 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.190+0000 7fd29bc01e40 1 -- 172.21.15.167:0/832409700 >> 172.21.15.167:0/832409700 conn(0x55709993e460 msgr2=0x55709993c500 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-05T19:30:22.450 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.190+0000 7fd29bc01e40 1 -- 172.21.15.167:0/832409700 shutdown_connections 2023-10-05T19:30:22.450 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.190+0000 7fd29bc01e40 1 -- 172.21.15.167:0/832409700 wait complete. 2023-10-05T19:30:22.450 INFO:tasks.util.rgw:rgwadmin: client.0 : ['user', 'info', '--uid', 'foo'] 2023-10-05T19:30:22.450 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'ceph', 'user', 'info', '--uid', 'foo'] 2023-10-05T19:30:22.450 DEBUG:teuthology.orchestra.run.smithi167:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster ceph user info --uid foo 2023-10-05T19:30:22.490 INFO:teuthology.orchestra.run.smithi167.stderr:ignoring --setuser ceph since I am not root 2023-10-05T19:30:22.490 INFO:teuthology.orchestra.run.smithi167.stderr:ignoring --setgroup ceph since I am not root 2023-10-05T19:30:22.490 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.486+0000 7f4e4fb42e40 1 Processor -- start 2023-10-05T19:30:22.491 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.486+0000 7f4e4fb42e40 1 -- start start 2023-10-05T19:30:22.491 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.486+0000 7f4e4fb42e40 1 --2- >> v2:172.21.15.118:3300/0 conn(0x55b374311d00 0x55b3743120d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-05T19:30:22.491 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.486+0000 7f4e4fb42e40 1 -- --> v1:172.21.15.118:6789/0 -- auth(proto 0 26 bytes epoch 0) v1 -- 0x55b37425fbd0 con 0x55b374312610 2023-10-05T19:30:22.491 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.486+0000 7f4e4fb42e40 1 -- --> v2:172.21.15.118:3300/0 -- mon_getmap magic: 0 v1 -- 0x55b37421cee0 con 0x55b374311d00 2023-10-05T19:30:22.492 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.486+0000 7f4e4d677700 1 --2- >> v2:172.21.15.118:3300/0 conn(0x55b374311d00 0x55b3743120d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-05T19:30:22.492 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.486+0000 7f4e4d677700 1 --2- >> v2:172.21.15.118:3300/0 conn(0x55b374311d00 0x55b3743120d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.118:3300/0 says I am v2:172.21.15.167:33026/0 (socket says 172.21.15.167:33026) 2023-10-05T19:30:22.492 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.486+0000 7f4e4d677700 1 -- 172.21.15.167:0/4166306826 learned_addr learned my addr 172.21.15.167:0/4166306826 (peer_addr_for_me v2:172.21.15.167:0/0) 2023-10-05T19:30:22.492 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.486+0000 7f4e4ce76700 1 --1- 172.21.15.167:0/4166306826 >> v1:172.21.15.118:6789/0 conn(0x55b374312610 0x55b3743130b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.118:6789/0 says I am v1:172.21.15.167:50800/0 (socket says 172.21.15.167:50800) 2023-10-05T19:30:22.492 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.490+0000 7f4e4d677700 1 -- 172.21.15.167:0/4166306826 >> v1:172.21.15.118:6789/0 conn(0x55b374312610 legacy=0x55b3743130b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-05T19:30:22.492 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.490+0000 7f4e4d677700 1 -- 172.21.15.167:0/4166306826 --> v2:172.21.15.118:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55b3743137c0 con 0x55b374311d00 2023-10-05T19:30:22.492 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.490+0000 7f4e4d677700 1 --2- 172.21.15.167:0/4166306826 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55b374311d00 0x55b3743120d0 secure :-1 s=READY pgs=669 cs=0 l=1 rev1=1 crypto rx=0x7f4e4000a5c0 tx=0x7f4e40005e00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=21650f565e101669 server_cookie=0 in_seq=0 out_seq=0 2023-10-05T19:30:22.493 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.490+0000 7f4e47fff700 1 -- 172.21.15.167:0/4166306826 <== mon.0 v2:172.21.15.118:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f4e4000e510 con 0x55b374311d00 2023-10-05T19:30:22.493 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.490+0000 7f4e47fff700 1 -- 172.21.15.167:0/4166306826 <== mon.0 v2:172.21.15.118:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4e4000e670 con 0x55b374311d00 2023-10-05T19:30:22.493 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.490+0000 7f4e47fff700 1 -- 172.21.15.167:0/4166306826 <== mon.0 v2:172.21.15.118:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f4e4000e920 con 0x55b374311d00 2023-10-05T19:30:22.494 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.490+0000 7f4e4fb42e40 1 -- 172.21.15.167:0/4166306826 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55b374311d00 msgr2=0x55b3743120d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-05T19:30:22.494 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.490+0000 7f4e4fb42e40 1 --2- 172.21.15.167:0/4166306826 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55b374311d00 0x55b3743120d0 secure :-1 s=READY pgs=669 cs=0 l=1 rev1=1 crypto rx=0x7f4e4000a5c0 tx=0x7f4e40005e00 comp rx=0 tx=0).stop 2023-10-05T19:30:22.494 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.490+0000 7f4e4fb42e40 1 -- 172.21.15.167:0/4166306826 shutdown_connections 2023-10-05T19:30:22.494 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.490+0000 7f4e4fb42e40 1 --2- 172.21.15.167:0/4166306826 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55b374311d00 0x55b3743120d0 unknown :-1 s=CLOSED pgs=669 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-05T19:30:22.494 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.490+0000 7f4e4fb42e40 1 -- 172.21.15.167:0/4166306826 >> 172.21.15.167:0/4166306826 conn(0x55b37430d720 msgr2=0x55b37430fb30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-05T19:30:22.495 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.490+0000 7f4e4fb42e40 1 -- 172.21.15.167:0/4166306826 shutdown_connections 2023-10-05T19:30:22.495 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.490+0000 7f4e4fb42e40 1 -- 172.21.15.167:0/4166306826 wait complete. 2023-10-05T19:30:22.495 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.490+0000 7f4e4fb42e40 1 Processor -- start 2023-10-05T19:30:22.496 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.490+0000 7f4e4fb42e40 1 -- start start 2023-10-05T19:30:22.496 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.490+0000 7f4e4fb42e40 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55b374311d00 0x55b374336ac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-05T19:30:22.496 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.490+0000 7f4e4fb42e40 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b3741e0e30 con 0x55b374311d00 2023-10-05T19:30:22.496 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.490+0000 7f4e4d677700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55b374311d00 0x55b374336ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-05T19:30:22.496 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.494+0000 7f4e4d677700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55b374311d00 0x55b374336ac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.118:3300/0 says I am v2:172.21.15.167:33040/0 (socket says 172.21.15.167:33040) 2023-10-05T19:30:22.496 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.494+0000 7f4e4d677700 1 -- 172.21.15.167:0/2339606781 learned_addr learned my addr 172.21.15.167:0/2339606781 (peer_addr_for_me v2:172.21.15.167:0/0) 2023-10-05T19:30:22.497 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.494+0000 7f4e4d677700 1 -- 172.21.15.167:0/2339606781 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4e4000a180 con 0x55b374311d00 2023-10-05T19:30:22.497 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.494+0000 7f4e4d677700 1 --2- 172.21.15.167:0/2339606781 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55b374311d00 0x55b374336ac0 secure :-1 s=READY pgs=670 cs=0 l=1 rev1=1 crypto rx=0x7f4e4000ad10 tx=0x7f4e40013000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4d5df72172a0ea24 server_cookie=0 in_seq=0 out_seq=0 2023-10-05T19:30:22.497 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.494+0000 7f4e46ffd700 1 -- 172.21.15.167:0/2339606781 <== mon.0 v2:172.21.15.118:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f4e400032f0 con 0x55b374311d00 2023-10-05T19:30:22.498 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.494+0000 7f4e46ffd700 1 -- 172.21.15.167:0/2339606781 <== mon.0 v2:172.21.15.118:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4e4000e7e0 con 0x55b374311d00 2023-10-05T19:30:22.498 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.494+0000 7f4e46ffd700 1 -- 172.21.15.167:0/2339606781 <== mon.0 v2:172.21.15.118:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f4e4000eab0 con 0x55b374311d00 2023-10-05T19:30:22.498 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.494+0000 7f4e4fb42e40 1 -- 172.21.15.167:0/2339606781 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55b374311d00 msgr2=0x55b374336ac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-05T19:30:22.498 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.494+0000 7f4e4fb42e40 1 --2- 172.21.15.167:0/2339606781 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55b374311d00 0x55b374336ac0 secure :-1 s=READY pgs=670 cs=0 l=1 rev1=1 crypto rx=0x7f4e4000ad10 tx=0x7f4e40013000 comp rx=0 tx=0).stop 2023-10-05T19:30:22.498 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.494+0000 7f4e4fb42e40 1 -- 172.21.15.167:0/2339606781 shutdown_connections 2023-10-05T19:30:22.498 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.494+0000 7f4e4fb42e40 1 --2- 172.21.15.167:0/2339606781 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55b374311d00 0x55b374336ac0 unknown :-1 s=CLOSED pgs=670 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-05T19:30:22.498 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.494+0000 7f4e4fb42e40 1 -- 172.21.15.167:0/2339606781 >> 172.21.15.167:0/2339606781 conn(0x55b37430d720 msgr2=0x55b374286e00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-05T19:30:22.498 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.494+0000 7f4e4fb42e40 1 -- 172.21.15.167:0/2339606781 shutdown_connections 2023-10-05T19:30:22.498 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.494+0000 7f4e4fb42e40 1 -- 172.21.15.167:0/2339606781 wait complete. 2023-10-05T19:30:22.499 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.494+0000 7f4e4fb42e40 1 Processor -- start 2023-10-05T19:30:22.500 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.494+0000 7f4e4fb42e40 1 -- start start 2023-10-05T19:30:22.500 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.494+0000 7f4e4fb42e40 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55b374312610 0x55b37433aec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-05T19:30:22.500 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.494+0000 7f4e4fb42e40 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b3741b3ed0 con 0x55b374312610 2023-10-05T19:30:22.500 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.494+0000 7f4e4d677700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55b374312610 0x55b37433aec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-05T19:30:22.500 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.494+0000 7f4e4d677700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55b374312610 0x55b37433aec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.118:3300/0 says I am v2:172.21.15.167:33052/0 (socket says 172.21.15.167:33052) 2023-10-05T19:30:22.500 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.494+0000 7f4e4d677700 1 -- 172.21.15.167:0/2792530994 learned_addr learned my addr 172.21.15.167:0/2792530994 (peer_addr_for_me v2:172.21.15.167:0/0) 2023-10-05T19:30:22.500 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.498+0000 7f4e4d677700 1 -- 172.21.15.167:0/2792530994 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55b3743137c0 con 0x55b374312610 2023-10-05T19:30:22.501 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.498+0000 7f4e4d677700 1 --2- 172.21.15.167:0/2792530994 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55b374312610 0x55b37433aec0 secure :-1 s=READY pgs=671 cs=0 l=1 rev1=1 crypto rx=0x7f4e4000a5c0 tx=0x7f4e4001c000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-05T19:30:22.501 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.498+0000 7f4e457fa700 1 -- 172.21.15.167:0/2792530994 <== mon.0 v2:172.21.15.118:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f4e4001cb50 con 0x55b374312610 2023-10-05T19:30:22.501 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.498+0000 7f4e457fa700 1 -- 172.21.15.167:0/2792530994 <== mon.0 v2:172.21.15.118:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4e4001ccb0 con 0x55b374312610 2023-10-05T19:30:22.501 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.498+0000 7f4e4fb42e40 1 -- 172.21.15.167:0/2792530994 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x55b37433b400 con 0x55b374312610 2023-10-05T19:30:22.501 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.498+0000 7f4e457fa700 1 -- 172.21.15.167:0/2792530994 <== mon.0 v2:172.21.15.118:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f4e40010440 con 0x55b374312610 2023-10-05T19:30:22.501 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.498+0000 7f4e4fb42e40 1 -- 172.21.15.167:0/2792530994 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55b37433b840 con 0x55b374312610 2023-10-05T19:30:22.503 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.498+0000 7f4e457fa700 1 -- 172.21.15.167:0/2792530994 <== mon.0 v2:172.21.15.118:3300/0 4 ==== mgrmap(e 5) v1 ==== 40329+0+0 (secure 0 0 0) 0x7f4e400105c0 con 0x55b374312610 2023-10-05T19:30:22.503 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.498+0000 7f4e4fb42e40 20 rados->read ofs=0 len=0 2023-10-05T19:30:22.503 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.498+0000 7f4e457fa700 1 --2- 172.21.15.167:0/2792530994 >> [v2:172.21.15.167:6824/29889,v1:172.21.15.167:6825/29889] conn(0x7f4e30030c50 0x7f4e30033100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-05T19:30:22.503 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.498+0000 7f4e457fa700 1 -- 172.21.15.167:0/2792530994 <== mon.0 v2:172.21.15.118:3300/0 5 ==== osd_map(32..32 src has 1..32) v4 ==== 6260+0+0 (secure 0 0 0) 0x7f4e40046430 con 0x55b374312610 2023-10-05T19:30:22.504 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.498+0000 7f4e4ce76700 1 --2- 172.21.15.167:0/2792530994 >> [v2:172.21.15.167:6824/29889,v1:172.21.15.167:6825/29889] conn(0x7f4e30030c50 0x7f4e30033100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-05T19:30:22.504 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.498+0000 7f4e4fb42e40 1 --2- 172.21.15.167:0/2792530994 >> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] conn(0x55b374344940 0x55b374346d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-05T19:30:22.504 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.498+0000 7f4e4fb42e40 1 -- 172.21.15.167:0/2792530994 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:1 5.12 5:49953fa1:::default.realm:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32) v8 -- 0x55b3743473d0 con 0x55b374344940 2023-10-05T19:30:22.504 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.498+0000 7f4e4de78700 1 --2- 172.21.15.167:0/2792530994 >> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] conn(0x55b374344940 0x55b374346d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-05T19:30:22.505 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.502+0000 7f4e4ce76700 1 --2- 172.21.15.167:0/2792530994 >> [v2:172.21.15.167:6824/29889,v1:172.21.15.167:6825/29889] conn(0x7f4e30030c50 0x7f4e30033100 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7f4e3800b300 tx=0x7f4e38001470 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-05T19:30:22.505 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.502+0000 7f4e4de78700 1 --2- 172.21.15.167:0/2792530994 >> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] conn(0x55b374344940 0x55b374346d20 crc :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-05T19:30:22.505 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.502+0000 7f4e4de78700 1 -- 172.21.15.167:0/2792530994 <== osd.1 v2:172.21.15.167:6800/30035 1 ==== osd_op_reply(1 default.realm [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 ==== 157+0+0 (crc 0 0 0) 0x7f4e4800a500 con 0x55b374344940 2023-10-05T19:30:22.505 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.502+0000 7f4e4fb42e40 20 rados_obj.operate() r=-2 bl.length=0 2023-10-05T19:30:22.505 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.502+0000 7f4e4fb42e40 20 realm 2023-10-05T19:30:22.505 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.502+0000 7f4e4fb42e40 20 rados->read ofs=0 len=0 2023-10-05T19:30:22.505 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.502+0000 7f4e4fb42e40 1 -- 172.21.15.167:0/2792530994 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:2 5.12 5:49953fa1:::default.realm:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32) v8 -- 0x55b374343fb0 con 0x55b374344940 2023-10-05T19:30:22.507 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.502+0000 7f4e4de78700 1 -- 172.21.15.167:0/2792530994 <== osd.1 v2:172.21.15.167:6800/30035 2 ==== osd_op_reply(2 default.realm [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 ==== 157+0+0 (crc 0 0 0) 0x7f4e4800a500 con 0x55b374344940 2023-10-05T19:30:22.507 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.502+0000 7f4e4fb42e40 20 rados_obj.operate() r=-2 bl.length=0 2023-10-05T19:30:22.507 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.502+0000 7f4e4fb42e40 4 RGWPeriod::init failed to init realm id : (2) No such file or directory 2023-10-05T19:30:22.507 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.502+0000 7f4e4fb42e40 20 rados->read ofs=0 len=0 2023-10-05T19:30:22.507 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.502+0000 7f4e4fb42e40 1 -- 172.21.15.167:0/2792530994 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:3 5.12 5:49953fa1:::default.realm:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32) v8 -- 0x55b374347eb0 con 0x55b374344940 2023-10-05T19:30:22.508 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.502+0000 7f4e4de78700 1 -- 172.21.15.167:0/2792530994 <== osd.1 v2:172.21.15.167:6800/30035 3 ==== osd_op_reply(3 default.realm [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 ==== 157+0+0 (crc 0 0 0) 0x7f4e4800a500 con 0x55b374344940 2023-10-05T19:30:22.508 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.502+0000 7f4e4fb42e40 20 rados_obj.operate() r=-2 bl.length=0 2023-10-05T19:30:22.508 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.502+0000 7f4e4fb42e40 20 rados->read ofs=0 len=0 2023-10-05T19:30:22.508 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.502+0000 7f4e4fb42e40 1 -- 172.21.15.167:0/2792530994 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:4 5.f 5:f43b6ece:::zone_names.default:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32) v8 -- 0x55b374347a90 con 0x55b374344940 2023-10-05T19:30:22.508 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.502+0000 7f4e4de78700 1 -- 172.21.15.167:0/2792530994 <== osd.1 v2:172.21.15.167:6800/30035 4 ==== osd_op_reply(4 zone_names.default [read 0~46 out=46b] v0'0 uv1 ondisk = 0) v8 ==== 162+0+46 (crc 0 0 0) 0x7f4e4800a500 con 0x55b374344940 2023-10-05T19:30:22.508 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.502+0000 7f4e4fb42e40 20 rados_obj.operate() r=0 bl.length=46 2023-10-05T19:30:22.508 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.502+0000 7f4e4fb42e40 20 rados->read ofs=0 len=0 2023-10-05T19:30:22.508 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.502+0000 7f4e4fb42e40 1 -- 172.21.15.167:0/2792530994 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:5 5.1d 5:b8374eb4:::zone_info.9a9df965-6612-45d4-a45f-6e3c889bacc2:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32) v8 -- 0x55b374348920 con 0x55b374344940 2023-10-05T19:30:22.508 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.502+0000 7f4e4de78700 1 -- 172.21.15.167:0/2792530994 <== osd.1 v2:172.21.15.167:6800/30035 5 ==== osd_op_reply(5 zone_info.9a9df965-6612-45d4-a45f-6e3c889bacc2 [read 0~874 out=874b] v0'0 uv2 ondisk = 0) v8 ==== 190+0+874 (crc 0 0 0) 0x7f4e4800a500 con 0x55b374344940 2023-10-05T19:30:22.508 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.502+0000 7f4e4fb42e40 20 rados_obj.operate() r=0 bl.length=874 2023-10-05T19:30:22.508 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.502+0000 7f4e4fb42e40 20 searching for the correct realm 2023-10-05T19:30:22.508 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.502+0000 7f4e4fb42e40 1 --2- 172.21.15.167:0/2792530994 >> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] conn(0x55b37434a770 0x55b37434cb50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-05T19:30:22.509 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.502+0000 7f4e4fb42e40 1 -- 172.21.15.167:0/2792530994 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:6 5.0 5:00000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x55b37434d090 con 0x55b37434a770 2023-10-05T19:30:22.509 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.502+0000 7f4e4d677700 1 --2- 172.21.15.167:0/2792530994 >> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] conn(0x55b37434a770 0x55b37434cb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-05T19:30:22.509 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.506+0000 7f4e4d677700 1 --2- 172.21.15.167:0/2792530994 >> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] conn(0x55b37434a770 0x55b37434cb50 crc :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-05T19:30:22.510 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.506+0000 7f4e4d677700 1 -- 172.21.15.167:0/2792530994 <== osd.3 v2:172.21.15.167:6808/30036 1 ==== osd_op_reply(6 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f4e4002b070 con 0x55b37434a770 2023-10-05T19:30:22.510 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.506+0000 7f4e4d677700 1 --2- 172.21.15.167:0/2792530994 >> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] conn(0x55b374311d00 0x7f4e400373f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-05T19:30:22.510 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.506+0000 7f4e4d677700 1 -- 172.21.15.167:0/2792530994 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:7 5.10 5:08000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x55b37434d090 con 0x55b374311d00 2023-10-05T19:30:22.510 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.506+0000 7f4e4ce76700 1 --2- 172.21.15.167:0/2792530994 >> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] conn(0x55b374311d00 0x7f4e400373f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-05T19:30:22.510 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.506+0000 7f4e4ce76700 1 --2- 172.21.15.167:0/2792530994 >> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] conn(0x55b374311d00 0x7f4e400373f0 crc :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-05T19:30:22.511 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.506+0000 7f4e4ce76700 1 -- 172.21.15.167:0/2792530994 <== osd.2 v2:172.21.15.167:6816/30037 1 ==== osd_op_reply(7 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f4e3800f560 con 0x55b374311d00 2023-10-05T19:30:22.511 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.506+0000 7f4e4ce76700 1 --2- 172.21.15.167:0/2792530994 >> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] conn(0x7f4e3800e8c0 0x7f4e3800d310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-05T19:30:22.511 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.506+0000 7f4e4ce76700 1 -- 172.21.15.167:0/2792530994 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:8 5.8 5:10000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x55b37434d090 con 0x7f4e3800e8c0 2023-10-05T19:30:22.511 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.506+0000 7f4e4de78700 1 --2- 172.21.15.167:0/2792530994 >> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] conn(0x7f4e3800e8c0 0x7f4e3800d310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-05T19:30:22.511 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.506+0000 7f4e4de78700 1 --2- 172.21.15.167:0/2792530994 >> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] conn(0x7f4e3800e8c0 0x7f4e3800d310 crc :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-05T19:30:22.512 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.510+0000 7f4e4de78700 1 -- 172.21.15.167:0/2792530994 <== osd.0 v2:172.21.15.118:6800/27690 1 ==== osd_op_reply(8 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f4e48014040 con 0x7f4e3800e8c0 2023-10-05T19:30:22.512 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.510+0000 7f4e4de78700 1 -- 172.21.15.167:0/2792530994 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:9 5.18 5:18000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x55b37434d090 con 0x55b374344940 2023-10-05T19:30:22.513 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.510+0000 7f4e4de78700 1 -- 172.21.15.167:0/2792530994 <== osd.1 v2:172.21.15.167:6800/30035 6 ==== osd_op_reply(9 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f4e48014040 con 0x55b374344940 2023-10-05T19:30:22.513 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.510+0000 7f4e4de78700 1 -- 172.21.15.167:0/2792530994 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:10 5.4 5:20000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x55b37434d090 con 0x55b374344940 2023-10-05T19:30:22.514 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.510+0000 7f4e4de78700 1 -- 172.21.15.167:0/2792530994 <== osd.1 v2:172.21.15.167:6800/30035 7 ==== osd_op_reply(10 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f4e48014040 con 0x55b374344940 2023-10-05T19:30:22.514 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.510+0000 7f4e4de78700 1 -- 172.21.15.167:0/2792530994 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:11 5.14 5:28000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x55b37434d090 con 0x7f4e3800e8c0 2023-10-05T19:30:22.515 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.510+0000 7f4e4de78700 1 -- 172.21.15.167:0/2792530994 <== osd.0 v2:172.21.15.118:6800/27690 2 ==== osd_op_reply(11 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f4e48014040 con 0x7f4e3800e8c0 2023-10-05T19:30:22.515 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.510+0000 7f4e4de78700 1 -- 172.21.15.167:0/2792530994 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:12 5.c 5:30000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x55b37434d090 con 0x7f4e3800e8c0 2023-10-05T19:30:22.516 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.514+0000 7f4e4de78700 1 -- 172.21.15.167:0/2792530994 <== osd.0 v2:172.21.15.118:6800/27690 3 ==== osd_op_reply(12 [pgnls start_epoch 32 out=112b] v23'1 uv1 ondisk = 1) v8 ==== 144+0+112 (crc 0 0 0) 0x7f4e48014040 con 0x7f4e3800e8c0 2023-10-05T19:30:22.516 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.514+0000 7f4e4de78700 1 -- 172.21.15.167:0/2792530994 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:13 5.1c 5:38000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x55b37434d090 con 0x55b374311d00 2023-10-05T19:30:22.517 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.514+0000 7f4e4ce76700 1 -- 172.21.15.167:0/2792530994 <== osd.2 v2:172.21.15.167:6816/30037 2 ==== osd_op_reply(13 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f4e3800f560 con 0x55b374311d00 2023-10-05T19:30:22.517 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.514+0000 7f4e4ce76700 1 -- 172.21.15.167:0/2792530994 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:14 5.2 5:40000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x55b37434d090 con 0x55b37434a770 2023-10-05T19:30:22.517 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.514+0000 7f4e4d677700 1 -- 172.21.15.167:0/2792530994 <== osd.3 v2:172.21.15.167:6808/30036 2 ==== osd_op_reply(14 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f4e4002b070 con 0x55b37434a770 2023-10-05T19:30:22.517 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.514+0000 7f4e4d677700 1 -- 172.21.15.167:0/2792530994 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:15 5.12 5:48000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x55b37434d090 con 0x55b374344940 2023-10-05T19:30:22.518 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.514+0000 7f4e4de78700 1 -- 172.21.15.167:0/2792530994 <== osd.1 v2:172.21.15.167:6800/30035 8 ==== osd_op_reply(15 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f4e48014040 con 0x55b374344940 2023-10-05T19:30:22.518 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.514+0000 7f4e4de78700 1 -- 172.21.15.167:0/2792530994 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:16 5.a 5:50000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x55b37434d090 con 0x7f4e3800e8c0 2023-10-05T19:30:22.519 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.514+0000 7f4e4de78700 1 -- 172.21.15.167:0/2792530994 <== osd.0 v2:172.21.15.118:6800/27690 4 ==== osd_op_reply(16 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f4e48022040 con 0x7f4e3800e8c0 2023-10-05T19:30:22.519 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.514+0000 7f4e4de78700 1 -- 172.21.15.167:0/2792530994 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:17 5.1a 5:58000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x55b37434d090 con 0x55b374344940 2023-10-05T19:30:22.520 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.518+0000 7f4e4de78700 1 -- 172.21.15.167:0/2792530994 <== osd.1 v2:172.21.15.167:6800/30035 9 ==== osd_op_reply(17 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f4e48022040 con 0x55b374344940 2023-10-05T19:30:22.520 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.518+0000 7f4e4de78700 1 -- 172.21.15.167:0/2792530994 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:18 5.6 5:60000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x55b37434d090 con 0x55b374311d00 2023-10-05T19:30:22.521 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.518+0000 7f4e4ce76700 1 -- 172.21.15.167:0/2792530994 <== osd.2 v2:172.21.15.167:6816/30037 3 ==== osd_op_reply(18 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f4e3800f560 con 0x55b374311d00 2023-10-05T19:30:22.521 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.518+0000 7f4e4ce76700 1 -- 172.21.15.167:0/2792530994 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:19 5.16 5:68000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x7f4e3801f040 con 0x55b37434a770 2023-10-05T19:30:22.521 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.518+0000 7f4e4d677700 1 -- 172.21.15.167:0/2792530994 <== osd.3 v2:172.21.15.167:6808/30036 3 ==== osd_op_reply(19 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f4e4002b070 con 0x55b37434a770 2023-10-05T19:30:22.522 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.518+0000 7f4e4d677700 1 -- 172.21.15.167:0/2792530994 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:20 5.e 5:70000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x7f4e3801f040 con 0x55b37434a770 2023-10-05T19:30:22.522 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.518+0000 7f4e4d677700 1 -- 172.21.15.167:0/2792530994 <== osd.3 v2:172.21.15.167:6808/30036 4 ==== osd_op_reply(20 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f4e4002b070 con 0x55b37434a770 2023-10-05T19:30:22.522 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.518+0000 7f4e4d677700 1 -- 172.21.15.167:0/2792530994 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:21 5.1e 5:78000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x7f4e3801f040 con 0x7f4e3800e8c0 2023-10-05T19:30:22.523 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.518+0000 7f4e4de78700 1 -- 172.21.15.167:0/2792530994 <== osd.0 v2:172.21.15.118:6800/27690 5 ==== osd_op_reply(21 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f4e48022040 con 0x7f4e3800e8c0 2023-10-05T19:30:22.523 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.518+0000 7f4e4de78700 1 -- 172.21.15.167:0/2792530994 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:22 5.1 5:80000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x7f4e3801f040 con 0x55b37434a770 2023-10-05T19:30:22.524 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.522+0000 7f4e4d677700 1 -- 172.21.15.167:0/2792530994 <== osd.3 v2:172.21.15.167:6808/30036 5 ==== osd_op_reply(22 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f4e4002b070 con 0x55b37434a770 2023-10-05T19:30:22.524 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.522+0000 7f4e4d677700 1 -- 172.21.15.167:0/2792530994 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:23 5.11 5:88000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x7f4e3801f040 con 0x55b37434a770 2023-10-05T19:30:22.524 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.522+0000 7f4e4d677700 1 -- 172.21.15.167:0/2792530994 <== osd.3 v2:172.21.15.167:6808/30036 6 ==== osd_op_reply(23 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f4e4002b070 con 0x55b37434a770 2023-10-05T19:30:22.525 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.522+0000 7f4e4d677700 1 -- 172.21.15.167:0/2792530994 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:24 5.9 5:90000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x7f4e3801f040 con 0x55b374344940 2023-10-05T19:30:22.525 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.522+0000 7f4e4de78700 1 -- 172.21.15.167:0/2792530994 <== osd.1 v2:172.21.15.167:6800/30035 10 ==== osd_op_reply(24 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f4e48022040 con 0x55b374344940 2023-10-05T19:30:22.525 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.522+0000 7f4e4de78700 1 -- 172.21.15.167:0/2792530994 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:25 5.19 5:98000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x7f4e3801f040 con 0x55b374344940 2023-10-05T19:30:22.526 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.522+0000 7f4e4de78700 1 -- 172.21.15.167:0/2792530994 <== osd.1 v2:172.21.15.167:6800/30035 11 ==== osd_op_reply(25 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f4e48022040 con 0x55b374344940 2023-10-05T19:30:22.526 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.522+0000 7f4e4de78700 1 -- 172.21.15.167:0/2792530994 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:26 5.5 5:a0000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x7f4e3801f040 con 0x7f4e3800e8c0 2023-10-05T19:30:22.527 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.522+0000 7f4e4de78700 1 -- 172.21.15.167:0/2792530994 <== osd.0 v2:172.21.15.118:6800/27690 6 ==== osd_op_reply(26 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f4e48022040 con 0x7f4e3800e8c0 2023-10-05T19:30:22.527 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.522+0000 7f4e4de78700 1 -- 172.21.15.167:0/2792530994 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:27 5.15 5:a8000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x7f4e3801f040 con 0x55b37434a770 2023-10-05T19:30:22.528 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.526+0000 7f4e4d677700 1 -- 172.21.15.167:0/2792530994 <== osd.3 v2:172.21.15.167:6808/30036 7 ==== osd_op_reply(27 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f4e4002b070 con 0x55b37434a770 2023-10-05T19:30:22.528 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.526+0000 7f4e4d677700 1 -- 172.21.15.167:0/2792530994 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:28 5.d 5:b0000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x7f4e3801f040 con 0x55b374311d00 2023-10-05T19:30:22.529 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.526+0000 7f4e4ce76700 1 -- 172.21.15.167:0/2792530994 <== osd.2 v2:172.21.15.167:6816/30037 4 ==== osd_op_reply(28 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f4e3800f560 con 0x55b374311d00 2023-10-05T19:30:22.529 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.526+0000 7f4e4ce76700 1 -- 172.21.15.167:0/2792530994 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:29 5.1d 5:b8000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x7f4e3801f040 con 0x55b374344940 2023-10-05T19:30:22.529 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.526+0000 7f4e4de78700 1 -- 172.21.15.167:0/2792530994 <== osd.1 v2:172.21.15.167:6800/30035 12 ==== osd_op_reply(29 [pgnls start_epoch 32 out=107b] v30'2 uv2 ondisk = 1) v8 ==== 144+0+107 (crc 0 0 0) 0x7f4e48022040 con 0x55b374344940 2023-10-05T19:30:22.529 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.526+0000 7f4e4de78700 1 -- 172.21.15.167:0/2792530994 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:30 5.3 5:c0000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x7f4e3801f040 con 0x7f4e3800e8c0 2023-10-05T19:30:22.531 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.526+0000 7f4e4de78700 1 -- 172.21.15.167:0/2792530994 <== osd.0 v2:172.21.15.118:6800/27690 7 ==== osd_op_reply(30 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f4e48022040 con 0x7f4e3800e8c0 2023-10-05T19:30:22.531 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.526+0000 7f4e4de78700 1 -- 172.21.15.167:0/2792530994 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:31 5.13 5:c8000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x7f4e3801f040 con 0x55b37434a770 2023-10-05T19:30:22.531 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.526+0000 7f4e4d677700 1 -- 172.21.15.167:0/2792530994 <== osd.3 v2:172.21.15.167:6808/30036 8 ==== osd_op_reply(31 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f4e4002b070 con 0x55b37434a770 2023-10-05T19:30:22.532 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.526+0000 7f4e4d677700 1 -- 172.21.15.167:0/2792530994 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:32 5.b 5:d0000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x7f4e3801f040 con 0x7f4e3800e8c0 2023-10-05T19:30:22.533 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.530+0000 7f4e4de78700 1 -- 172.21.15.167:0/2792530994 <== osd.0 v2:172.21.15.118:6800/27690 8 ==== osd_op_reply(32 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f4e48022040 con 0x7f4e3800e8c0 2023-10-05T19:30:22.533 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.530+0000 7f4e4de78700 1 -- 172.21.15.167:0/2792530994 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:33 5.1b 5:d8000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x7f4e3801f040 con 0x55b374311d00 2023-10-05T19:30:22.533 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.530+0000 7f4e4ce76700 1 -- 172.21.15.167:0/2792530994 <== osd.2 v2:172.21.15.167:6816/30037 5 ==== osd_op_reply(33 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f4e3800f560 con 0x55b374311d00 2023-10-05T19:30:22.534 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.530+0000 7f4e4ce76700 1 -- 172.21.15.167:0/2792530994 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:34 5.7 5:e0000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x7f4e3801f040 con 0x55b37434a770 2023-10-05T19:30:22.534 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.530+0000 7f4e4d677700 1 -- 172.21.15.167:0/2792530994 <== osd.3 v2:172.21.15.167:6808/30036 9 ==== osd_op_reply(34 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f4e4002b070 con 0x55b37434a770 2023-10-05T19:30:22.535 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.530+0000 7f4e4d677700 1 -- 172.21.15.167:0/2792530994 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:35 5.17 5:e8000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x7f4e3801f040 con 0x7f4e3800e8c0 2023-10-05T19:30:22.536 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.534+0000 7f4e4de78700 1 -- 172.21.15.167:0/2792530994 <== osd.0 v2:172.21.15.118:6800/27690 9 ==== osd_op_reply(35 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f4e48022040 con 0x7f4e3800e8c0 2023-10-05T19:30:22.536 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.534+0000 7f4e4de78700 1 -- 172.21.15.167:0/2792530994 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:36 5.f 5:f0000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x7f4e3801f040 con 0x55b374344940 2023-10-05T19:30:22.537 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.534+0000 7f4e4de78700 1 -- 172.21.15.167:0/2792530994 <== osd.1 v2:172.21.15.167:6800/30035 13 ==== osd_op_reply(36 [pgnls start_epoch 32 out=115b] v23'2 uv2 ondisk = 1) v8 ==== 144+0+115 (crc 0 0 0) 0x7f4e4802f080 con 0x55b374344940 2023-10-05T19:30:22.537 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.534+0000 7f4e4de78700 1 -- 172.21.15.167:0/2792530994 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:37 5.1f 5:f8000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x7f4e3801f040 con 0x55b374311d00 2023-10-05T19:30:22.538 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.534+0000 7f4e4ce76700 1 -- 172.21.15.167:0/2792530994 <== osd.2 v2:172.21.15.167:6816/30037 6 ==== osd_op_reply(37 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f4e3800f560 con 0x55b374311d00 2023-10-05T19:30:22.538 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.534+0000 7f4e4fb42e40 20 RGWRados::pool_iterate: got zonegroup_info.4f1f719d-9648-417b-b10c-e5d939825419 2023-10-05T19:30:22.538 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.534+0000 7f4e4fb42e40 20 RGWRados::pool_iterate: got zone_info.9a9df965-6612-45d4-a45f-6e3c889bacc2 2023-10-05T19:30:22.538 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.534+0000 7f4e4fb42e40 20 RGWRados::pool_iterate: got zone_names.default 2023-10-05T19:30:22.538 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.534+0000 7f4e4fb42e40 20 RGWRados::pool_iterate: got zonegroups_names.default 2023-10-05T19:30:22.538 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.534+0000 7f4e4fb42e40 20 rados->read ofs=0 len=0 2023-10-05T19:30:22.538 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.534+0000 7f4e4fb42e40 1 -- 172.21.15.167:0/2792530994 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:38 5.12 5:49953fa1:::default.realm:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32) v8 -- 0x55b374349520 con 0x55b374344940 2023-10-05T19:30:22.539 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.534+0000 7f4e4de78700 1 -- 172.21.15.167:0/2792530994 <== osd.1 v2:172.21.15.167:6800/30035 14 ==== osd_op_reply(38 default.realm [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 ==== 157+0+0 (crc 0 0 0) 0x7f4e4802f080 con 0x55b374344940 2023-10-05T19:30:22.539 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.534+0000 7f4e4fb42e40 20 rados_obj.operate() r=-2 bl.length=0 2023-10-05T19:30:22.539 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.534+0000 7f4e4fb42e40 20 rados->read ofs=0 len=0 2023-10-05T19:30:22.539 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.534+0000 7f4e4fb42e40 1 -- 172.21.15.167:0/2792530994 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:39 5.f 5:f4c53578:::zonegroups_names.default:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32) v8 -- 0x55b374349ea0 con 0x55b374344940 2023-10-05T19:30:22.539 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.534+0000 7f4e4de78700 1 -- 172.21.15.167:0/2792530994 <== osd.1 v2:172.21.15.167:6800/30035 15 ==== osd_op_reply(39 zonegroups_names.default [read 0~46 out=46b] v0'0 uv2 ondisk = 0) v8 ==== 168+0+46 (crc 0 0 0) 0x7f4e4802f080 con 0x55b374344940 2023-10-05T19:30:22.539 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.534+0000 7f4e4fb42e40 20 rados_obj.operate() r=0 bl.length=46 2023-10-05T19:30:22.540 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.534+0000 7f4e4fb42e40 20 rados->read ofs=0 len=0 2023-10-05T19:30:22.540 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.534+0000 7f4e4fb42e40 1 -- 172.21.15.167:0/2792530994 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:40 5.c 5:30d299a5:::zonegroup_info.4f1f719d-9648-417b-b10c-e5d939825419:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32) v8 -- 0x55b374349110 con 0x7f4e3800e8c0 2023-10-05T19:30:22.540 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.538+0000 7f4e4de78700 1 -- 172.21.15.167:0/2792530994 <== osd.0 v2:172.21.15.118:6800/27690 10 ==== osd_op_reply(40 zonegroup_info.4f1f719d-9648-417b-b10c-e5d939825419 [read 0~362 out=362b] v0'0 uv1 ondisk = 0) v8 ==== 195+0+362 (crc 0 0 0) 0x7f4e4802f080 con 0x7f4e3800e8c0 2023-10-05T19:30:22.540 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.538+0000 7f4e4fb42e40 20 rados_obj.operate() r=0 bl.length=362 2023-10-05T19:30:22.540 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.538+0000 7f4e4fb42e40 20 rados->read ofs=0 len=0 2023-10-05T19:30:22.540 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.538+0000 7f4e4fb42e40 1 -- 172.21.15.167:0/2792530994 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:41 5.13 5:cbbf6c60:::converted:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32) v8 -- 0x55b37434d930 con 0x55b37434a770 2023-10-05T19:30:22.541 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.538+0000 7f4e4d677700 1 -- 172.21.15.167:0/2792530994 <== osd.3 v2:172.21.15.167:6808/30036 10 ==== osd_op_reply(41 converted [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 ==== 153+0+0 (crc 0 0 0) 0x7f4e4002b070 con 0x55b37434a770 2023-10-05T19:30:22.541 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.538+0000 7f4e4fb42e40 20 rados_obj.operate() r=-2 bl.length=0 2023-10-05T19:30:22.541 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.538+0000 7f4e4fb42e40 20 rados->read ofs=0 len=0 2023-10-05T19:30:22.541 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.538+0000 7f4e4fb42e40 1 -- 172.21.15.167:0/2792530994 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:42 5.12 5:49953fa1:::default.realm:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32) v8 -- 0x55b37434dce0 con 0x55b374344940 2023-10-05T19:30:22.542 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.538+0000 7f4e4de78700 1 -- 172.21.15.167:0/2792530994 <== osd.1 v2:172.21.15.167:6800/30035 16 ==== osd_op_reply(42 default.realm [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 ==== 157+0+0 (crc 0 0 0) 0x7f4e4802f080 con 0x55b374344940 2023-10-05T19:30:22.542 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.538+0000 7f4e4fb42e40 20 rados_obj.operate() r=-2 bl.length=0 2023-10-05T19:30:22.542 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.538+0000 7f4e4fb42e40 20 rados->read ofs=0 len=0 2023-10-05T19:30:22.542 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.538+0000 7f4e4fb42e40 1 -- 172.21.15.167:0/2792530994 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:43 5.f 5:f4c53578:::zonegroups_names.default:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32) v8 -- 0x55b37434dce0 con 0x55b374344940 2023-10-05T19:30:22.542 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.538+0000 7f4e4de78700 1 -- 172.21.15.167:0/2792530994 <== osd.1 v2:172.21.15.167:6800/30035 17 ==== osd_op_reply(43 zonegroups_names.default [read 0~46 out=46b] v0'0 uv2 ondisk = 0) v8 ==== 168+0+46 (crc 0 0 0) 0x7f4e4802f080 con 0x55b374344940 2023-10-05T19:30:22.543 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.538+0000 7f4e4fb42e40 20 rados_obj.operate() r=0 bl.length=46 2023-10-05T19:30:22.543 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.538+0000 7f4e4fb42e40 1 -- 172.21.15.167:0/2792530994 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:44 5.0 5:00000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x55b37434ecf0 con 0x55b37434a770 2023-10-05T19:30:22.544 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.542+0000 7f4e4d677700 1 -- 172.21.15.167:0/2792530994 <== osd.3 v2:172.21.15.167:6808/30036 11 ==== osd_op_reply(44 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f4e4002b070 con 0x55b37434a770 2023-10-05T19:30:22.544 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.542+0000 7f4e4d677700 1 -- 172.21.15.167:0/2792530994 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:45 5.10 5:08000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x55b37434ecf0 con 0x55b374311d00 2023-10-05T19:30:22.545 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.542+0000 7f4e4ce76700 1 -- 172.21.15.167:0/2792530994 <== osd.2 v2:172.21.15.167:6816/30037 7 ==== osd_op_reply(45 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f4e3800f560 con 0x55b374311d00 2023-10-05T19:30:22.545 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.542+0000 7f4e4ce76700 1 -- 172.21.15.167:0/2792530994 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:46 5.8 5:10000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x55b37434ecf0 con 0x7f4e3800e8c0 2023-10-05T19:30:22.546 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.542+0000 7f4e4de78700 1 -- 172.21.15.167:0/2792530994 <== osd.0 v2:172.21.15.118:6800/27690 11 ==== osd_op_reply(46 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f4e4802f080 con 0x7f4e3800e8c0 2023-10-05T19:30:22.546 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.542+0000 7f4e4de78700 1 -- 172.21.15.167:0/2792530994 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:47 5.18 5:18000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x55b374349110 con 0x55b374344940 2023-10-05T19:30:22.546 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.542+0000 7f4e4de78700 1 -- 172.21.15.167:0/2792530994 <== osd.1 v2:172.21.15.167:6800/30035 18 ==== osd_op_reply(47 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f4e4802f080 con 0x55b374344940 2023-10-05T19:30:22.547 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.542+0000 7f4e4de78700 1 -- 172.21.15.167:0/2792530994 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:48 5.4 5:20000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x55b374349110 con 0x55b374344940 2023-10-05T19:30:22.547 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.542+0000 7f4e4de78700 1 -- 172.21.15.167:0/2792530994 <== osd.1 v2:172.21.15.167:6800/30035 19 ==== osd_op_reply(48 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f4e4802f080 con 0x55b374344940 2023-10-05T19:30:22.547 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.542+0000 7f4e4de78700 1 -- 172.21.15.167:0/2792530994 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:49 5.14 5:28000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x55b374349110 con 0x7f4e3800e8c0 2023-10-05T19:30:22.548 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.546+0000 7f4e4de78700 1 -- 172.21.15.167:0/2792530994 <== osd.0 v2:172.21.15.118:6800/27690 12 ==== osd_op_reply(49 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f4e4802f080 con 0x7f4e3800e8c0 2023-10-05T19:30:22.549 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.546+0000 7f4e4de78700 1 -- 172.21.15.167:0/2792530994 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:50 5.c 5:30000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x55b374349110 con 0x7f4e3800e8c0 2023-10-05T19:30:22.550 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.546+0000 7f4e4de78700 1 -- 172.21.15.167:0/2792530994 <== osd.0 v2:172.21.15.118:6800/27690 13 ==== osd_op_reply(50 [pgnls start_epoch 32 out=112b] v23'1 uv1 ondisk = 1) v8 ==== 144+0+112 (crc 0 0 0) 0x7f4e4802f080 con 0x7f4e3800e8c0 2023-10-05T19:30:22.550 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.546+0000 7f4e4de78700 1 -- 172.21.15.167:0/2792530994 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:51 5.1c 5:38000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x55b374349110 con 0x55b374311d00 2023-10-05T19:30:22.551 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.546+0000 7f4e4ce76700 1 -- 172.21.15.167:0/2792530994 <== osd.2 v2:172.21.15.167:6816/30037 8 ==== osd_op_reply(51 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f4e3800f560 con 0x55b374311d00 2023-10-05T19:30:22.551 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.546+0000 7f4e4ce76700 1 -- 172.21.15.167:0/2792530994 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:52 5.2 5:40000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x55b374349110 con 0x55b37434a770 2023-10-05T19:30:22.552 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.550+0000 7f4e4d677700 1 -- 172.21.15.167:0/2792530994 <== osd.3 v2:172.21.15.167:6808/30036 12 ==== osd_op_reply(52 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f4e4002b070 con 0x55b37434a770 2023-10-05T19:30:22.552 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.550+0000 7f4e4d677700 1 -- 172.21.15.167:0/2792530994 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:53 5.12 5:48000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x55b374349110 con 0x55b374344940 2023-10-05T19:30:22.553 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.550+0000 7f4e4de78700 1 -- 172.21.15.167:0/2792530994 <== osd.1 v2:172.21.15.167:6800/30035 20 ==== osd_op_reply(53 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f4e4802f080 con 0x55b374344940 2023-10-05T19:30:22.553 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.550+0000 7f4e4de78700 1 -- 172.21.15.167:0/2792530994 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:54 5.a 5:50000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x55b374349110 con 0x7f4e3800e8c0 2023-10-05T19:30:22.554 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.550+0000 7f4e4de78700 1 -- 172.21.15.167:0/2792530994 <== osd.0 v2:172.21.15.118:6800/27690 14 ==== osd_op_reply(54 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f4e4802f080 con 0x7f4e3800e8c0 2023-10-05T19:30:22.554 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.550+0000 7f4e4de78700 1 -- 172.21.15.167:0/2792530994 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:55 5.1a 5:58000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x55b374349110 con 0x55b374344940 2023-10-05T19:30:22.554 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.550+0000 7f4e4de78700 1 -- 172.21.15.167:0/2792530994 <== osd.1 v2:172.21.15.167:6800/30035 21 ==== osd_op_reply(55 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f4e4802f080 con 0x55b374344940 2023-10-05T19:30:22.554 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.550+0000 7f4e4de78700 1 -- 172.21.15.167:0/2792530994 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:56 5.6 5:60000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x55b374349110 con 0x55b374311d00 2023-10-05T19:30:22.555 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.550+0000 7f4e4ce76700 1 -- 172.21.15.167:0/2792530994 <== osd.2 v2:172.21.15.167:6816/30037 9 ==== osd_op_reply(56 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f4e3800f560 con 0x55b374311d00 2023-10-05T19:30:22.555 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.550+0000 7f4e4ce76700 1 -- 172.21.15.167:0/2792530994 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:57 5.16 5:68000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x55b374349110 con 0x55b37434a770 2023-10-05T19:30:22.556 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.554+0000 7f4e4d677700 1 -- 172.21.15.167:0/2792530994 <== osd.3 v2:172.21.15.167:6808/30036 13 ==== osd_op_reply(57 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f4e4002b070 con 0x55b37434a770 2023-10-05T19:30:22.556 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.554+0000 7f4e4d677700 1 -- 172.21.15.167:0/2792530994 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:58 5.e 5:70000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x55b374349110 con 0x55b37434a770 2023-10-05T19:30:22.557 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.554+0000 7f4e4d677700 1 -- 172.21.15.167:0/2792530994 <== osd.3 v2:172.21.15.167:6808/30036 14 ==== osd_op_reply(58 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f4e4002b070 con 0x55b37434a770 2023-10-05T19:30:22.557 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.554+0000 7f4e4d677700 1 -- 172.21.15.167:0/2792530994 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:59 5.1e 5:78000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x55b374349110 con 0x7f4e3800e8c0 2023-10-05T19:30:22.558 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.554+0000 7f4e4de78700 1 -- 172.21.15.167:0/2792530994 <== osd.0 v2:172.21.15.118:6800/27690 15 ==== osd_op_reply(59 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f4e4802f080 con 0x7f4e3800e8c0 2023-10-05T19:30:22.558 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.554+0000 7f4e4de78700 1 -- 172.21.15.167:0/2792530994 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:60 5.1 5:80000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x55b374349110 con 0x55b37434a770 2023-10-05T19:30:22.559 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.554+0000 7f4e4d677700 1 -- 172.21.15.167:0/2792530994 <== osd.3 v2:172.21.15.167:6808/30036 15 ==== osd_op_reply(60 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f4e4002b070 con 0x55b37434a770 2023-10-05T19:30:22.559 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.554+0000 7f4e4d677700 1 -- 172.21.15.167:0/2792530994 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:61 5.11 5:88000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x55b374349110 con 0x55b37434a770 2023-10-05T19:30:22.560 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.558+0000 7f4e4d677700 1 -- 172.21.15.167:0/2792530994 <== osd.3 v2:172.21.15.167:6808/30036 16 ==== osd_op_reply(61 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f4e4002b070 con 0x55b37434a770 2023-10-05T19:30:22.560 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.558+0000 7f4e4d677700 1 -- 172.21.15.167:0/2792530994 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:62 5.9 5:90000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x55b374349110 con 0x55b374344940 2023-10-05T19:30:22.560 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.558+0000 7f4e4de78700 1 -- 172.21.15.167:0/2792530994 <== osd.1 v2:172.21.15.167:6800/30035 22 ==== osd_op_reply(62 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f4e4802f080 con 0x55b374344940 2023-10-05T19:30:22.560 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.558+0000 7f4e4de78700 1 -- 172.21.15.167:0/2792530994 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:63 5.19 5:98000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x55b374349110 con 0x55b374344940 2023-10-05T19:30:22.561 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.558+0000 7f4e4de78700 1 -- 172.21.15.167:0/2792530994 <== osd.1 v2:172.21.15.167:6800/30035 23 ==== osd_op_reply(63 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f4e4802f080 con 0x55b374344940 2023-10-05T19:30:22.561 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.558+0000 7f4e4de78700 1 -- 172.21.15.167:0/2792530994 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:64 5.5 5:a0000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x55b374349110 con 0x7f4e3800e8c0 2023-10-05T19:30:22.562 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.558+0000 7f4e4de78700 1 -- 172.21.15.167:0/2792530994 <== osd.0 v2:172.21.15.118:6800/27690 16 ==== osd_op_reply(64 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f4e4802f080 con 0x7f4e3800e8c0 2023-10-05T19:30:22.562 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.558+0000 7f4e4de78700 1 -- 172.21.15.167:0/2792530994 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:65 5.15 5:a8000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x55b374349110 con 0x55b37434a770 2023-10-05T19:30:22.563 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.558+0000 7f4e4d677700 1 -- 172.21.15.167:0/2792530994 <== osd.3 v2:172.21.15.167:6808/30036 17 ==== osd_op_reply(65 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f4e4002b070 con 0x55b37434a770 2023-10-05T19:30:22.563 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.558+0000 7f4e4d677700 1 -- 172.21.15.167:0/2792530994 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:66 5.d 5:b0000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x55b374349110 con 0x55b374311d00 2023-10-05T19:30:22.564 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.562+0000 7f4e4ce76700 1 -- 172.21.15.167:0/2792530994 <== osd.2 v2:172.21.15.167:6816/30037 10 ==== osd_op_reply(66 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f4e3800f560 con 0x55b374311d00 2023-10-05T19:30:22.564 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.562+0000 7f4e4ce76700 1 -- 172.21.15.167:0/2792530994 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:67 5.1d 5:b8000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x55b374349110 con 0x55b374344940 2023-10-05T19:30:22.565 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.562+0000 7f4e4de78700 1 -- 172.21.15.167:0/2792530994 <== osd.1 v2:172.21.15.167:6800/30035 24 ==== osd_op_reply(67 [pgnls start_epoch 32 out=107b] v30'2 uv2 ondisk = 1) v8 ==== 144+0+107 (crc 0 0 0) 0x7f4e4802f080 con 0x55b374344940 2023-10-05T19:30:22.565 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.562+0000 7f4e4de78700 1 -- 172.21.15.167:0/2792530994 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:68 5.3 5:c0000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x55b374349110 con 0x7f4e3800e8c0 2023-10-05T19:30:22.566 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.562+0000 7f4e4de78700 1 -- 172.21.15.167:0/2792530994 <== osd.0 v2:172.21.15.118:6800/27690 17 ==== osd_op_reply(68 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f4e4802f080 con 0x7f4e3800e8c0 2023-10-05T19:30:22.567 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.562+0000 7f4e4de78700 1 -- 172.21.15.167:0/2792530994 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:69 5.13 5:c8000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x55b374349110 con 0x55b37434a770 2023-10-05T19:30:22.567 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.562+0000 7f4e4d677700 1 -- 172.21.15.167:0/2792530994 <== osd.3 v2:172.21.15.167:6808/30036 18 ==== osd_op_reply(69 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f4e4002b070 con 0x55b37434a770 2023-10-05T19:30:22.567 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.562+0000 7f4e4d677700 1 -- 172.21.15.167:0/2792530994 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:70 5.b 5:d0000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x55b374349110 con 0x7f4e3800e8c0 2023-10-05T19:30:22.568 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.566+0000 7f4e4de78700 1 -- 172.21.15.167:0/2792530994 <== osd.0 v2:172.21.15.118:6800/27690 18 ==== osd_op_reply(70 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f4e4802f080 con 0x7f4e3800e8c0 2023-10-05T19:30:22.568 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.566+0000 7f4e4de78700 1 -- 172.21.15.167:0/2792530994 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:71 5.1b 5:d8000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x55b374349110 con 0x55b374311d00 2023-10-05T19:30:22.569 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.566+0000 7f4e4ce76700 1 -- 172.21.15.167:0/2792530994 <== osd.2 v2:172.21.15.167:6816/30037 11 ==== osd_op_reply(71 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f4e3800f560 con 0x55b374311d00 2023-10-05T19:30:22.569 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.566+0000 7f4e4ce76700 1 -- 172.21.15.167:0/2792530994 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:72 5.7 5:e0000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x55b374349110 con 0x55b37434a770 2023-10-05T19:30:22.570 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.566+0000 7f4e4d677700 1 -- 172.21.15.167:0/2792530994 <== osd.3 v2:172.21.15.167:6808/30036 19 ==== osd_op_reply(72 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f4e4002b070 con 0x55b37434a770 2023-10-05T19:30:22.570 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.566+0000 7f4e4d677700 1 -- 172.21.15.167:0/2792530994 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:73 5.17 5:e8000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x55b374349110 con 0x7f4e3800e8c0 2023-10-05T19:30:22.571 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.566+0000 7f4e4de78700 1 -- 172.21.15.167:0/2792530994 <== osd.0 v2:172.21.15.118:6800/27690 19 ==== osd_op_reply(73 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f4e4802f080 con 0x7f4e3800e8c0 2023-10-05T19:30:22.571 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.566+0000 7f4e4de78700 1 -- 172.21.15.167:0/2792530994 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:74 5.f 5:f0000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x55b374349110 con 0x55b374344940 2023-10-05T19:30:22.572 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.570+0000 7f4e4de78700 1 -- 172.21.15.167:0/2792530994 <== osd.1 v2:172.21.15.167:6800/30035 25 ==== osd_op_reply(74 [pgnls start_epoch 32 out=115b] v23'2 uv2 ondisk = 1) v8 ==== 144+0+115 (crc 0 0 0) 0x7f4e4802f080 con 0x55b374344940 2023-10-05T19:30:22.572 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.570+0000 7f4e4de78700 1 -- 172.21.15.167:0/2792530994 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:75 5.1f 5:f8000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x55b374349110 con 0x55b374311d00 2023-10-05T19:30:22.573 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.570+0000 7f4e4ce76700 1 -- 172.21.15.167:0/2792530994 <== osd.2 v2:172.21.15.167:6816/30037 12 ==== osd_op_reply(75 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f4e3800f560 con 0x55b374311d00 2023-10-05T19:30:22.573 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.570+0000 7f4e4fb42e40 20 RGWRados::pool_iterate: got zonegroup_info.4f1f719d-9648-417b-b10c-e5d939825419 2023-10-05T19:30:22.573 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.570+0000 7f4e4fb42e40 20 RGWRados::pool_iterate: got zone_info.9a9df965-6612-45d4-a45f-6e3c889bacc2 2023-10-05T19:30:22.573 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.570+0000 7f4e4fb42e40 20 RGWRados::pool_iterate: got zone_names.default 2023-10-05T19:30:22.574 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.570+0000 7f4e4fb42e40 20 RGWRados::pool_iterate: got zonegroups_names.default 2023-10-05T19:30:22.574 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.570+0000 7f4e4fb42e40 20 rados->read ofs=0 len=0 2023-10-05T19:30:22.574 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.570+0000 7f4e4fb42e40 1 -- 172.21.15.167:0/2792530994 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:76 5.f 5:f43b6ece:::zone_names.default:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32) v8 -- 0x55b37434dce0 con 0x55b374344940 2023-10-05T19:30:22.575 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.570+0000 7f4e4de78700 1 -- 172.21.15.167:0/2792530994 <== osd.1 v2:172.21.15.167:6800/30035 26 ==== osd_op_reply(76 zone_names.default [read 0~46 out=46b] v0'0 uv1 ondisk = 0) v8 ==== 162+0+46 (crc 0 0 0) 0x7f4e4802f080 con 0x55b374344940 2023-10-05T19:30:22.575 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.570+0000 7f4e4fb42e40 20 rados_obj.operate() r=0 bl.length=46 2023-10-05T19:30:22.575 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.570+0000 7f4e4fb42e40 20 rados->read ofs=0 len=0 2023-10-05T19:30:22.575 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.570+0000 7f4e4fb42e40 1 -- 172.21.15.167:0/2792530994 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:77 5.1d 5:b8374eb4:::zone_info.9a9df965-6612-45d4-a45f-6e3c889bacc2:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32) v8 -- 0x55b37434e090 con 0x55b374344940 2023-10-05T19:30:22.575 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.570+0000 7f4e4de78700 1 -- 172.21.15.167:0/2792530994 <== osd.1 v2:172.21.15.167:6800/30035 27 ==== osd_op_reply(77 zone_info.9a9df965-6612-45d4-a45f-6e3c889bacc2 [read 0~874 out=874b] v0'0 uv2 ondisk = 0) v8 ==== 190+0+874 (crc 0 0 0) 0x7f4e4802f080 con 0x55b374344940 2023-10-05T19:30:22.575 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.570+0000 7f4e4fb42e40 20 rados_obj.operate() r=0 bl.length=874 2023-10-05T19:30:22.575 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.570+0000 7f4e4fb42e40 20 rados->read ofs=0 len=0 2023-10-05T19:30:22.575 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.570+0000 7f4e4fb42e40 1 -- 172.21.15.167:0/2792530994 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:78 5.f 5:f4c53578:::zonegroups_names.default:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32) v8 -- 0x55b37434e580 con 0x55b374344940 2023-10-05T19:30:22.576 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.570+0000 7f4e4de78700 1 -- 172.21.15.167:0/2792530994 <== osd.1 v2:172.21.15.167:6800/30035 28 ==== osd_op_reply(78 zonegroups_names.default [read 0~46 out=46b] v0'0 uv2 ondisk = 0) v8 ==== 168+0+46 (crc 0 0 0) 0x7f4e4802f080 con 0x55b374344940 2023-10-05T19:30:22.576 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.570+0000 7f4e4fb42e40 20 rados_obj.operate() r=0 bl.length=46 2023-10-05T19:30:22.576 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.570+0000 7f4e4fb42e40 20 rados->read ofs=0 len=0 2023-10-05T19:30:22.576 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.570+0000 7f4e4fb42e40 1 -- 172.21.15.167:0/2792530994 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:79 5.c 5:30d299a5:::zonegroup_info.4f1f719d-9648-417b-b10c-e5d939825419:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32) v8 -- 0x55b37434e580 con 0x7f4e3800e8c0 2023-10-05T19:30:22.577 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.574+0000 7f4e4de78700 1 -- 172.21.15.167:0/2792530994 <== osd.0 v2:172.21.15.118:6800/27690 20 ==== osd_op_reply(79 zonegroup_info.4f1f719d-9648-417b-b10c-e5d939825419 [read 0~362 out=362b] v0'0 uv1 ondisk = 0) v8 ==== 195+0+362 (crc 0 0 0) 0x7f4e4802f080 con 0x7f4e3800e8c0 2023-10-05T19:30:22.577 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.574+0000 7f4e4fb42e40 20 rados_obj.operate() r=0 bl.length=362 2023-10-05T19:30:22.577 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.574+0000 7f4e4fb42e40 20 rados->read ofs=0 len=0 2023-10-05T19:30:22.577 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.574+0000 7f4e4fb42e40 1 -- 172.21.15.167:0/2792530994 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:80 5.3 5:c0ad8cbd:::region_map:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32) v8 -- 0x55b37434e580 con 0x7f4e3800e8c0 2023-10-05T19:30:22.578 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.574+0000 7f4e4de78700 1 -- 172.21.15.167:0/2792530994 <== osd.0 v2:172.21.15.118:6800/27690 21 ==== osd_op_reply(80 region_map [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 ==== 154+0+0 (crc 0 0 0) 0x7f4e48050040 con 0x7f4e3800e8c0 2023-10-05T19:30:22.578 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.574+0000 7f4e4fb42e40 20 rados_obj.operate() r=-2 bl.length=0 2023-10-05T19:30:22.578 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.574+0000 7f4e4fb42e40 20 zone default found 2023-10-05T19:30:22.579 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.574+0000 7f4e4fb42e40 4 Realm: () 2023-10-05T19:30:22.579 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.574+0000 7f4e4fb42e40 4 ZoneGroup: default (4f1f719d-9648-417b-b10c-e5d939825419) 2023-10-05T19:30:22.579 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.574+0000 7f4e4fb42e40 4 Zone: default (9a9df965-6612-45d4-a45f-6e3c889bacc2) 2023-10-05T19:30:22.579 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.574+0000 7f4e4fb42e40 10 cannot find current period zonegroup using local zonegroup configuration 2023-10-05T19:30:22.579 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.574+0000 7f4e4fb42e40 20 zonegroup default 2023-10-05T19:30:22.579 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.574+0000 7f4e4fb42e40 20 rados->read ofs=0 len=0 2023-10-05T19:30:22.579 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.574+0000 7f4e4fb42e40 1 -- 172.21.15.167:0/2792530994 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:81 5.3 5:c52100b6:::period_config.default:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32) v8 -- 0x55b37434e580 con 0x7f4e3800e8c0 2023-10-05T19:30:22.580 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.574+0000 7f4e4de78700 1 -- 172.21.15.167:0/2792530994 <== osd.0 v2:172.21.15.118:6800/27690 22 ==== osd_op_reply(81 period_config.default [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 ==== 165+0+0 (crc 0 0 0) 0x7f4e48050040 con 0x7f4e3800e8c0 2023-10-05T19:30:22.580 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.574+0000 7f4e4fb42e40 20 rados_obj.operate() r=-2 bl.length=0 2023-10-05T19:30:22.580 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.574+0000 7f4e4fb42e40 20 rados->read ofs=0 len=0 2023-10-05T19:30:22.581 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.574+0000 7f4e4fb42e40 1 -- 172.21.15.167:0/2792530994 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:82 6.2 6:4347d321:::bucket.sync-source-hints.:head [call version.read in=11b,read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32) v8 -- 0x55b37434ff00 con 0x55b37434a770 2023-10-05T19:30:22.581 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.574+0000 7f4e4d677700 1 -- 172.21.15.167:0/2792530994 <== osd.3 v2:172.21.15.167:6808/30036 20 ==== osd_op_reply(82 bucket.sync-source-hints. [call,read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 ==== 211+0+0 (crc 0 0 0) 0x7f4e4002b070 con 0x55b37434a770 2023-10-05T19:30:22.581 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.574+0000 7f4e4fb42e40 20 rados_obj.operate() r=-2 bl.length=0 2023-10-05T19:30:22.581 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.574+0000 7f4e4fb42e40 20 rados->read ofs=0 len=0 2023-10-05T19:30:22.581 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.574+0000 7f4e4fb42e40 1 -- 172.21.15.167:0/2792530994 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:83 6.b 6:d467b91b:::bucket.sync-target-hints.:head [call version.read in=11b,read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32) v8 -- 0x55b3743502b0 con 0x55b37434a770 2023-10-05T19:30:22.581 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.578+0000 7f4e4d677700 1 -- 172.21.15.167:0/2792530994 <== osd.3 v2:172.21.15.167:6808/30036 21 ==== osd_op_reply(83 bucket.sync-target-hints. [call,read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 ==== 211+0+0 (crc 0 0 0) 0x7f4e4002b070 con 0x55b37434a770 2023-10-05T19:30:22.581 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.578+0000 7f4e4fb42e40 20 rados_obj.operate() r=-2 bl.length=0 2023-10-05T19:30:22.581 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.578+0000 7f4e4fb42e40 20 started sync module instance, tier type = 2023-10-05T19:30:22.581 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.578+0000 7f4e4fb42e40 20 started zone id=9a9df965-6612-45d4-a45f-6e3c889bacc2 (name=default) with tier type = 2023-10-05T19:30:22.581 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.578+0000 7f4e4fb42e40 1 -- 172.21.15.167:0/2792530994 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:84 7.1f 7:f95f44c2:::notify.0:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e32) v8 -- 0x55b37434f5c0 con 0x55b374311d00 2023-10-05T19:30:22.583 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.578+0000 7f4e4ce76700 1 -- 172.21.15.167:0/2792530994 <== osd.2 v2:172.21.15.167:6816/30037 13 ==== osd_op_reply(84 notify.0 [create] v32'267 uv267 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f4e3800f560 con 0x55b374311d00 2023-10-05T19:30:22.584 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.578+0000 7f4e4fb42e40 1 -- 172.21.15.167:0/2792530994 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:85 7.1f 7:f95f44c2:::notify.0:head [watch watch cookie 94229237135808] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e32) v8 -- 0x55b37434f1f0 con 0x55b374311d00 2023-10-05T19:30:22.584 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.578+0000 7f4e4fb42e40 1 -- 172.21.15.167:0/2792530994 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:86 7.0 7:05bf5b68:::notify.1:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e32) v8 -- 0x55b3743510b0 con 0x55b374344940 2023-10-05T19:30:22.587 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.582+0000 7f4e4de78700 1 -- 172.21.15.167:0/2792530994 <== osd.1 v2:172.21.15.167:6800/30035 29 ==== osd_op_reply(86 notify.1 [create] v32'267 uv267 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f4e48050040 con 0x55b374344940 2023-10-05T19:30:22.587 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.582+0000 7f4e4ce76700 1 -- 172.21.15.167:0/2792530994 <== osd.2 v2:172.21.15.167:6816/30037 14 ==== osd_op_reply(85 notify.0 [watch watch cookie 94229237135808] v32'268 uv267 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f4e3800f560 con 0x55b374311d00 2023-10-05T19:30:22.587 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.582+0000 7f4e4fb42e40 1 -- 172.21.15.167:0/2792530994 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:87 7.0 7:05bf5b68:::notify.1:head [watch watch cookie 94229237140352] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e32) v8 -- 0x55b3743514b0 con 0x55b374344940 2023-10-05T19:30:22.587 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.582+0000 7f4e4fb42e40 1 -- 172.21.15.167:0/2792530994 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:88 7.15 7:a93a5511:::notify.2:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e32) v8 -- 0x55b374351d50 con 0x55b37434a770 2023-10-05T19:30:22.590 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.586+0000 7f4e4d677700 1 -- 172.21.15.167:0/2792530994 <== osd.3 v2:172.21.15.167:6808/30036 22 ==== osd_op_reply(88 notify.2 [create] v32'267 uv267 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f4e4002b070 con 0x55b37434a770 2023-10-05T19:30:22.590 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.586+0000 7f4e4fb42e40 1 -- 172.21.15.167:0/2792530994 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:89 7.15 7:a93a5511:::notify.2:head [watch watch cookie 94229237142992] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e32) v8 -- 0x55b3743525f0 con 0x55b37434a770 2023-10-05T19:30:22.591 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.586+0000 7f4e4fb42e40 1 -- 172.21.15.167:0/2792530994 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:90 7.e 7:7759931f:::notify.3:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e32) v8 -- 0x55b374352e90 con 0x55b374311d00 2023-10-05T19:30:22.591 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.586+0000 7f4e4de78700 1 -- 172.21.15.167:0/2792530994 <== osd.1 v2:172.21.15.167:6800/30035 30 ==== osd_op_reply(87 notify.1 [watch watch cookie 94229237140352] v32'268 uv267 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f4e48050040 con 0x55b374344940 2023-10-05T19:30:22.593 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.590+0000 7f4e4ce76700 1 -- 172.21.15.167:0/2792530994 <== osd.2 v2:172.21.15.167:6816/30037 15 ==== osd_op_reply(90 notify.3 [create] v32'267 uv267 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f4e3800f560 con 0x55b374311d00 2023-10-05T19:30:22.593 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.590+0000 7f4e4fb42e40 1 -- 172.21.15.167:0/2792530994 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:91 7.e 7:7759931f:::notify.3:head [watch watch cookie 94229237139040] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e32) v8 -- 0x55b374352ac0 con 0x55b374311d00 2023-10-05T19:30:22.593 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.590+0000 7f4e4fb42e40 1 -- 172.21.15.167:0/2792530994 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:92 7.d 7:b4812045:::notify.4:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e32) v8 -- 0x55b374353f30 con 0x55b374344940 2023-10-05T19:30:22.593 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.590+0000 7f4e4d677700 1 -- 172.21.15.167:0/2792530994 <== osd.3 v2:172.21.15.167:6808/30036 23 ==== osd_op_reply(89 notify.2 [watch watch cookie 94229237142992] v32'268 uv267 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f4e4002b070 con 0x55b37434a770 2023-10-05T19:30:22.594 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.590+0000 7f4e4ce76700 1 -- 172.21.15.167:0/2792530994 <== osd.2 v2:172.21.15.167:6816/30037 16 ==== osd_op_reply(91 notify.3 [watch watch cookie 94229237139040] v32'268 uv267 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f4e3800f560 con 0x55b374311d00 2023-10-05T19:30:22.597 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.594+0000 7f4e4de78700 1 -- 172.21.15.167:0/2792530994 <== osd.1 v2:172.21.15.167:6800/30035 31 ==== osd_op_reply(92 notify.4 [create] v32'267 uv267 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f4e48050040 con 0x55b374344940 2023-10-05T19:30:22.597 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.594+0000 7f4e4fb42e40 1 -- 172.21.15.167:0/2792530994 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:93 7.d 7:b4812045:::notify.4:head [watch watch cookie 94229237150352] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e32) v8 -- 0x55b374353a00 con 0x55b374344940 2023-10-05T19:30:22.597 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.594+0000 7f4e4fb42e40 1 -- 172.21.15.167:0/2792530994 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:94 7.3 7:c609908c:::notify.5:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e32) v8 -- 0x55b374354820 con 0x7f4e3800e8c0 2023-10-05T19:30:22.599 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.594+0000 7f4e4de78700 1 -- 172.21.15.167:0/2792530994 <== osd.1 v2:172.21.15.167:6800/30035 32 ==== osd_op_reply(93 notify.4 [watch watch cookie 94229237150352] v32'268 uv267 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f4e48050040 con 0x55b374344940 2023-10-05T19:30:22.600 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.598+0000 7f4e4de78700 1 -- 172.21.15.167:0/2792530994 <== osd.0 v2:172.21.15.118:6800/27690 23 ==== osd_op_reply(94 notify.5 [create] v32'267 uv267 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f4e48050040 con 0x7f4e3800e8c0 2023-10-05T19:30:22.600 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.598+0000 7f4e4fb42e40 1 -- 172.21.15.167:0/2792530994 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:95 7.3 7:c609908c:::notify.5:head [watch watch cookie 94229237145856] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e32) v8 -- 0x55b3743547d0 con 0x7f4e3800e8c0 2023-10-05T19:30:22.600 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.598+0000 7f4e4fb42e40 1 -- 172.21.15.167:0/2792530994 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:96 7.14 7:2b04a3e9:::notify.6:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e32) v8 -- 0x55b3743551e0 con 0x7f4e3800e8c0 2023-10-05T19:30:22.602 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.598+0000 7f4e4de78700 1 -- 172.21.15.167:0/2792530994 <== osd.0 v2:172.21.15.118:6800/27690 24 ==== osd_op_reply(95 notify.5 [watch watch cookie 94229237145856] v32'268 uv267 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f4e48050040 con 0x7f4e3800e8c0 2023-10-05T19:30:22.603 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.598+0000 7f4e4de78700 1 -- 172.21.15.167:0/2792530994 <== osd.0 v2:172.21.15.118:6800/27690 25 ==== osd_op_reply(96 notify.6 [create] v32'267 uv267 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f4e48050040 con 0x7f4e3800e8c0 2023-10-05T19:30:22.603 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.598+0000 7f4e4fb42e40 1 -- 172.21.15.167:0/2792530994 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:97 7.14 7:2b04a3e9:::notify.6:head [watch watch cookie 94229237154816] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e32) v8 -- 0x55b3743551e0 con 0x7f4e3800e8c0 2023-10-05T19:30:22.603 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.598+0000 7f4e4fb42e40 1 -- 172.21.15.167:0/2792530994 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:98 7.9 7:93e5b521:::notify.7:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e32) v8 -- 0x55b374355a80 con 0x7f4e3800e8c0 2023-10-05T19:30:22.605 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.602+0000 7f4e4de78700 1 -- 172.21.15.167:0/2792530994 <== osd.0 v2:172.21.15.118:6800/27690 26 ==== osd_op_reply(97 notify.6 [watch watch cookie 94229237154816] v32'268 uv267 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f4e48050040 con 0x7f4e3800e8c0 2023-10-05T19:30:22.605 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.602+0000 7f4e4de78700 1 -- 172.21.15.167:0/2792530994 <== osd.0 v2:172.21.15.118:6800/27690 27 ==== osd_op_reply(98 notify.7 [create] v32'267 uv267 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f4e48050040 con 0x7f4e3800e8c0 2023-10-05T19:30:22.605 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.602+0000 7f4e4fb42e40 1 -- 172.21.15.167:0/2792530994 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:99 7.9 7:93e5b521:::notify.7:head [watch watch cookie 94229237157664] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e32) v8 -- 0x55b374355860 con 0x7f4e3800e8c0 2023-10-05T19:30:22.606 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.602+0000 7f4e4fb42e40 20 add_watcher() i=0 2023-10-05T19:30:22.606 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.602+0000 7f4e4fb42e40 20 add_watcher() i=1 2023-10-05T19:30:22.606 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.602+0000 7f4e4fb42e40 20 add_watcher() i=2 2023-10-05T19:30:22.606 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.602+0000 7f4e4fb42e40 20 add_watcher() i=3 2023-10-05T19:30:22.606 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.602+0000 7f4e4fb42e40 20 add_watcher() i=4 2023-10-05T19:30:22.606 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.602+0000 7f4e4fb42e40 20 add_watcher() i=5 2023-10-05T19:30:22.606 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.602+0000 7f4e4fb42e40 20 add_watcher() i=6 2023-10-05T19:30:22.607 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.602+0000 7f4e4de78700 1 -- 172.21.15.167:0/2792530994 <== osd.0 v2:172.21.15.118:6800/27690 28 ==== osd_op_reply(99 notify.7 [watch watch cookie 94229237157664] v32'268 uv267 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f4e48050040 con 0x7f4e3800e8c0 2023-10-05T19:30:22.607 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.602+0000 7f4e4fb42e40 20 add_watcher() i=7 2023-10-05T19:30:22.607 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.602+0000 7f4e4fb42e40 2 all 8 watchers are set, enabling cache 2023-10-05T19:30:22.607 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.602+0000 7f4e4fb42e40 1 -- 172.21.15.167:0/2792530994 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:100 6.4 6:22d26bf9:::data_loggenerations_metadata:head [call version.check_conds in=74b,call version.read in=11b,read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32) v8 -- 0x55b374358de0 con 0x55b374344940 2023-10-05T19:30:22.608 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.606+0000 7f4e4de78700 1 -- 172.21.15.167:0/2792530994 <== osd.1 v2:172.21.15.167:6800/30035 33 ==== osd_op_reply(100 data_loggenerations_metadata [call,call out=48b,read 0~28 out=28b] v0'0 uv1 ondisk = 0) v8 ==== 256+0+76 (crc 0 0 0) 0x7f4e48050040 con 0x55b374344940 2023-10-05T19:30:22.608 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.606+0000 7f4e4fb42e40 1 -- 172.21.15.167:0/2792530994 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:101 6.4 6:22d26bf9:::data_loggenerations_metadata:head [watch watch cookie 94229237159232] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e32) v8 -- 0x55b3743562a0 con 0x55b374344940 2023-10-05T19:30:22.611 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.606+0000 7f4e4de78700 1 -- 172.21.15.167:0/2792530994 <== osd.1 v2:172.21.15.167:6800/30035 34 ==== osd_op_reply(101 data_loggenerations_metadata [watch watch cookie 94229237159232] v32'260 uv1 ondisk = 0) v8 ==== 172+0+0 (crc 0 0 0) 0x7f4e48050040 con 0x55b374344940 2023-10-05T19:30:22.611 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.606+0000 7f4e1e7e4700 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2023-10-05T19:30:22.611 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.606+0000 7f4e4fb42e40 20 check_secure_mon_conn(): auth registy supported: methods=[2] modes=[2,1] 2023-10-05T19:30:22.611 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.606+0000 7f4e4fb42e40 20 check_secure_mon_conn(): mode 1 is insecure 2023-10-05T19:30:22.611 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.606+0000 7f4e4fb42e40 1 Processor -- start 2023-10-05T19:30:22.612 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.606+0000 7f4e4fb42e40 1 -- start start 2023-10-05T19:30:22.612 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.610+0000 7f4e4fb42e40 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55b3743631c0 0x55b374363590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-05T19:30:22.612 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.610+0000 7f4e4fb42e40 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b374363ad0 con 0x55b3743631c0 2023-10-05T19:30:22.613 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.610+0000 7f4e4ce76700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55b3743631c0 0x55b374363590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-05T19:30:22.613 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.610+0000 7f4e4ce76700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55b3743631c0 0x55b374363590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.118:3300/0 says I am v2:172.21.15.167:33054/0 (socket says 172.21.15.167:33054) 2023-10-05T19:30:22.613 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.610+0000 7f4e4ce76700 1 -- 172.21.15.167:0/3089295066 learned_addr learned my addr 172.21.15.167:0/3089295066 (peer_addr_for_me v2:172.21.15.167:0/0) 2023-10-05T19:30:22.613 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.610+0000 7f4e4ce76700 1 -- 172.21.15.167:0/3089295066 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55b374363c10 con 0x55b3743631c0 2023-10-05T19:30:22.614 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.610+0000 7f4e4ce76700 1 --2- 172.21.15.167:0/3089295066 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55b3743631c0 0x55b374363590 secure :-1 s=READY pgs=672 cs=0 l=1 rev1=1 crypto rx=0x55b3743484c0 tx=0x7f4e38006000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cae8345679eac520 server_cookie=0 in_seq=0 out_seq=0 2023-10-05T19:30:22.614 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.610+0000 7f4e1dfe3700 1 -- 172.21.15.167:0/3089295066 <== mon.0 v2:172.21.15.118:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f4e3800dac0 con 0x55b3743631c0 2023-10-05T19:30:22.614 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.610+0000 7f4e1dfe3700 1 -- 172.21.15.167:0/3089295066 <== mon.0 v2:172.21.15.118:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4e3800dc40 con 0x55b3743631c0 2023-10-05T19:30:22.614 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.610+0000 7f4e1dfe3700 1 -- 172.21.15.167:0/3089295066 <== mon.0 v2:172.21.15.118:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f4e3800ddf0 con 0x55b3743631c0 2023-10-05T19:30:22.615 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.610+0000 7f4e4fb42e40 1 -- 172.21.15.167:0/3089295066 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55b3743631c0 msgr2=0x55b374363590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-05T19:30:22.615 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.610+0000 7f4e4fb42e40 1 --2- 172.21.15.167:0/3089295066 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55b3743631c0 0x55b374363590 secure :-1 s=READY pgs=672 cs=0 l=1 rev1=1 crypto rx=0x55b3743484c0 tx=0x7f4e38006000 comp rx=0 tx=0).stop 2023-10-05T19:30:22.615 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.610+0000 7f4e4fb42e40 1 -- 172.21.15.167:0/3089295066 shutdown_connections 2023-10-05T19:30:22.615 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.610+0000 7f4e4fb42e40 1 --2- 172.21.15.167:0/3089295066 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55b3743631c0 0x55b374363590 unknown :-1 s=CLOSED pgs=672 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-05T19:30:22.615 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.610+0000 7f4e4fb42e40 1 -- 172.21.15.167:0/3089295066 >> 172.21.15.167:0/3089295066 conn(0x55b37435e580 msgr2=0x55b37435c730 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-05T19:30:22.615 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.610+0000 7f4e4fb42e40 1 -- 172.21.15.167:0/3089295066 shutdown_connections 2023-10-05T19:30:22.615 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.610+0000 7f4e4fb42e40 1 -- 172.21.15.167:0/3089295066 wait complete. 2023-10-05T19:30:22.616 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.610+0000 7f4e4fb42e40 1 Processor -- start 2023-10-05T19:30:22.616 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.614+0000 7f4e4fb42e40 1 -- start start 2023-10-05T19:30:22.617 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.614+0000 7f4e4fb42e40 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55b3743631c0 0x55b374368be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-05T19:30:22.617 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.614+0000 7f4e4fb42e40 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b374369be0 con 0x55b3743631c0 2023-10-05T19:30:22.617 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.614+0000 7f4e4ce76700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55b3743631c0 0x55b374368be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-05T19:30:22.617 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.614+0000 7f4e4ce76700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55b3743631c0 0x55b374368be0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.118:3300/0 says I am v2:172.21.15.167:33066/0 (socket says 172.21.15.167:33066) 2023-10-05T19:30:22.617 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.614+0000 7f4e4ce76700 1 -- 172.21.15.167:0/786787770 learned_addr learned my addr 172.21.15.167:0/786787770 (peer_addr_for_me v2:172.21.15.167:0/0) 2023-10-05T19:30:22.618 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.614+0000 7f4e4ce76700 1 -- 172.21.15.167:0/786787770 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4e38006070 con 0x55b3743631c0 2023-10-05T19:30:22.618 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.614+0000 7f4e4ce76700 1 --2- 172.21.15.167:0/786787770 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55b3743631c0 0x55b374368be0 secure :-1 s=READY pgs=673 cs=0 l=1 rev1=1 crypto rx=0x7f4e3800f4b0 tx=0x7f4e38032000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-05T19:30:22.618 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.614+0000 7f4d8ffff700 1 -- 172.21.15.167:0/786787770 <== mon.0 v2:172.21.15.118:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f4e3803db60 con 0x55b3743631c0 2023-10-05T19:30:22.618 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.614+0000 7f4d8ffff700 1 -- 172.21.15.167:0/786787770 <== mon.0 v2:172.21.15.118:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4e3803dcc0 con 0x55b3743631c0 2023-10-05T19:30:22.618 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.614+0000 7f4e4fb42e40 1 -- 172.21.15.167:0/786787770 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x55b3743691e0 con 0x55b3743631c0 2023-10-05T19:30:22.618 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.614+0000 7f4d8ffff700 1 -- 172.21.15.167:0/786787770 <== mon.0 v2:172.21.15.118:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f4e3803de70 con 0x55b3743631c0 2023-10-05T19:30:22.619 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.614+0000 7f4e4fb42e40 1 -- 172.21.15.167:0/786787770 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55b374369620 con 0x55b3743631c0 2023-10-05T19:30:22.619 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.614+0000 7f4d8ffff700 1 -- 172.21.15.167:0/786787770 <== mon.0 v2:172.21.15.118:3300/0 4 ==== mgrmap(e 5) v1 ==== 40329+0+0 (secure 0 0 0) 0x7f4e3801a040 con 0x55b3743631c0 2023-10-05T19:30:22.620 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.614+0000 7f4d8ffff700 1 --2- 172.21.15.167:0/786787770 >> [v2:172.21.15.167:6824/29889,v1:172.21.15.167:6825/29889] conn(0x7f4d94030bb0 0x7f4d94033060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-05T19:30:22.620 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.618+0000 7f4d8ffff700 1 -- 172.21.15.167:0/786787770 <== mon.0 v2:172.21.15.118:3300/0 5 ==== osd_map(32..32 src has 1..32) v4 ==== 6260+0+0 (secure 0 0 0) 0x7f4e38048d60 con 0x55b3743631c0 2023-10-05T19:30:22.620 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.618+0000 7f4e4de78700 1 --2- 172.21.15.167:0/786787770 >> [v2:172.21.15.167:6824/29889,v1:172.21.15.167:6825/29889] conn(0x7f4d94030bb0 0x7f4d94033060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-05T19:30:22.620 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.618+0000 7f4e4fb42e40 5 note: GC not initialized 2023-10-05T19:30:22.620 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.618+0000 7f4e4de78700 1 --2- 172.21.15.167:0/786787770 >> [v2:172.21.15.167:6824/29889,v1:172.21.15.167:6825/29889] conn(0x7f4d94030bb0 0x7f4d94033060 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x55b374355ae0 tx=0x7f4e48000f90 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-05T19:30:22.622 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.618+0000 7f4d8d7fa700 20 reqs_thread_entry: start 2023-10-05T19:30:22.622 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.618+0000 7f4d86ffd700 20 reqs_thread_entry: start 2023-10-05T19:30:22.622 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.618+0000 7f4e4fb42e40 20 init_complete bucket index max shards: 11 2023-10-05T19:30:22.623 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.618+0000 7f4e4fb42e40 10 rgw notify: Started notification manager with: 1 workers 2023-10-05T19:30:22.623 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.618+0000 7f4d857fa700 20 rgw notify: INFO: next queues processing will happen at: Thu Oct 5 19:30:52 2023 2023-10-05T19:30:22.623 INFO:teuthology.orchestra.run.smithi167.stderr: 2023-10-05T19:30:22.624 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.618+0000 7f4d84ff9700 20 reqs_thread_entry: start 2023-10-05T19:30:22.624 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.618+0000 7f4e4fb42e40 20 get_system_obj_state: rctx=0x7fffeb2688a8 obj=default.rgw.meta:users.uid:foo state=0x55b3741ec0b0 s->prefetch_data=0 2023-10-05T19:30:22.624 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.618+0000 7f4e4fb42e40 1 -- 172.21.15.167:0/2792530994 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:102 8.18 8:1e6ca5ed:users.uid::foo:head [call version.read in=11b,getxattrs,stat] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32) v8 -- 0x55b3743d13f0 con 0x55b374344940 2023-10-05T19:30:22.624 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.622+0000 7f4e4de78700 1 -- 172.21.15.167:0/2792530994 <== osd.1 v2:172.21.15.167:6800/30035 35 ==== osd_op_reply(102 foo [call,getxattrs,stat] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 ==== 231+0+0 (crc 0 0 0) 0x7f4e48050040 con 0x55b374344940 2023-10-05T19:30:22.624 INFO:teuthology.orchestra.run.smithi167.stderr:could not fetch user info: no user info saved 2023-10-05T19:30:22.625 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.622+0000 7f4e4fb42e40 1 -- 172.21.15.167:0/2792530994 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:103 7.1f 7:f95f44c2:::notify.0:head [watch unwatch cookie 94229237135808] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e32) v8 -- 0x55b3743d13f0 con 0x55b374311d00 2023-10-05T19:30:22.627 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.622+0000 7f4e4ce76700 1 -- 172.21.15.167:0/2792530994 <== osd.2 v2:172.21.15.167:6816/30037 17 ==== osd_op_reply(103 notify.0 [watch unwatch cookie 94229237135808] v32'269 uv267 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f4e3800f560 con 0x55b374311d00 2023-10-05T19:30:22.628 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.626+0000 7f4e4fb42e40 20 remove_watcher() i=0 2023-10-05T19:30:22.628 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.626+0000 7f4e4fb42e40 2 removed watcher, disabling cache 2023-10-05T19:30:22.628 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.626+0000 7f4e4fb42e40 1 -- 172.21.15.167:0/2792530994 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:104 7.0 7:05bf5b68:::notify.1:head [watch unwatch cookie 94229237140352] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e32) v8 -- 0x55b3743d13f0 con 0x55b374344940 2023-10-05T19:30:22.630 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.626+0000 7f4e4de78700 1 -- 172.21.15.167:0/2792530994 <== osd.1 v2:172.21.15.167:6800/30035 36 ==== osd_op_reply(104 notify.1 [watch unwatch cookie 94229237140352] v32'269 uv267 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f4e48050040 con 0x55b374344940 2023-10-05T19:30:22.631 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.626+0000 7f4e4fb42e40 20 remove_watcher() i=1 2023-10-05T19:30:22.631 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.626+0000 7f4e4fb42e40 1 -- 172.21.15.167:0/2792530994 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:105 7.15 7:a93a5511:::notify.2:head [watch unwatch cookie 94229237142992] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e32) v8 -- 0x55b3743d13f0 con 0x55b37434a770 2023-10-05T19:30:22.634 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.630+0000 7f4e4d677700 1 -- 172.21.15.167:0/2792530994 <== osd.3 v2:172.21.15.167:6808/30036 24 ==== osd_op_reply(105 notify.2 [watch unwatch cookie 94229237142992] v32'269 uv267 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f4e4002b070 con 0x55b37434a770 2023-10-05T19:30:22.634 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.630+0000 7f4e4fb42e40 20 remove_watcher() i=2 2023-10-05T19:30:22.634 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.630+0000 7f4e4fb42e40 1 -- 172.21.15.167:0/2792530994 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:106 7.e 7:7759931f:::notify.3:head [watch unwatch cookie 94229237139040] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e32) v8 -- 0x55b3743515d0 con 0x55b374311d00 2023-10-05T19:30:22.636 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.634+0000 7f4e4ce76700 1 -- 172.21.15.167:0/2792530994 <== osd.2 v2:172.21.15.167:6816/30037 18 ==== osd_op_reply(106 notify.3 [watch unwatch cookie 94229237139040] v32'269 uv267 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f4e3800f560 con 0x55b374311d00 2023-10-05T19:30:22.636 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.634+0000 7f4e4fb42e40 20 remove_watcher() i=3 2023-10-05T19:30:22.636 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.634+0000 7f4e4fb42e40 1 -- 172.21.15.167:0/2792530994 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:107 7.d 7:b4812045:::notify.4:head [watch unwatch cookie 94229237150352] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e32) v8 -- 0x55b374350660 con 0x55b374344940 2023-10-05T19:30:22.639 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.634+0000 7f4e4de78700 1 -- 172.21.15.167:0/2792530994 <== osd.1 v2:172.21.15.167:6800/30035 37 ==== osd_op_reply(107 notify.4 [watch unwatch cookie 94229237150352] v32'269 uv267 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f4e48050040 con 0x55b374344940 2023-10-05T19:30:22.639 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.634+0000 7f4e4fb42e40 20 remove_watcher() i=4 2023-10-05T19:30:22.639 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.634+0000 7f4e4fb42e40 1 -- 172.21.15.167:0/2792530994 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:108 7.3 7:c609908c:::notify.5:head [watch unwatch cookie 94229237145856] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e32) v8 -- 0x55b3743515d0 con 0x7f4e3800e8c0 2023-10-05T19:30:22.642 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.638+0000 7f4e4de78700 1 -- 172.21.15.167:0/2792530994 <== osd.0 v2:172.21.15.118:6800/27690 29 ==== osd_op_reply(108 notify.5 [watch unwatch cookie 94229237145856] v32'269 uv267 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f4e48050040 con 0x7f4e3800e8c0 2023-10-05T19:30:22.642 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.638+0000 7f4e4fb42e40 20 remove_watcher() i=5 2023-10-05T19:30:22.642 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.638+0000 7f4e4fb42e40 1 -- 172.21.15.167:0/2792530994 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:109 7.14 7:2b04a3e9:::notify.6:head [watch unwatch cookie 94229237154816] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e32) v8 -- 0x55b374353290 con 0x7f4e3800e8c0 2023-10-05T19:30:22.644 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.642+0000 7f4e4de78700 1 -- 172.21.15.167:0/2792530994 <== osd.0 v2:172.21.15.118:6800/27690 30 ==== osd_op_reply(109 notify.6 [watch unwatch cookie 94229237154816] v32'269 uv267 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f4e48050040 con 0x7f4e3800e8c0 2023-10-05T19:30:22.645 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.642+0000 7f4e4fb42e40 20 remove_watcher() i=6 2023-10-05T19:30:22.645 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.642+0000 7f4e4fb42e40 1 -- 172.21.15.167:0/2792530994 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:110 7.9 7:93e5b521:::notify.7:head [watch unwatch cookie 94229237157664] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e32) v8 -- 0x55b374350660 con 0x7f4e3800e8c0 2023-10-05T19:30:22.647 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.642+0000 7f4e4de78700 1 -- 172.21.15.167:0/2792530994 <== osd.0 v2:172.21.15.118:6800/27690 31 ==== osd_op_reply(110 notify.7 [watch unwatch cookie 94229237157664] v32'269 uv267 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f4e48050040 con 0x7f4e3800e8c0 2023-10-05T19:30:22.648 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.642+0000 7f4e4fb42e40 20 remove_watcher() i=7 2023-10-05T19:30:22.649 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.646+0000 7f4e1e7e4700 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2023-10-05T19:30:22.649 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.646+0000 7f4e4fb42e40 1 -- 172.21.15.167:0/2792530994 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:111 6.4 6:22d26bf9:::data_loggenerations_metadata:head [watch unwatch cookie 94229237159232] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e32) v8 -- 0x55b374390770 con 0x55b374344940 2023-10-05T19:30:22.653 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.650+0000 7f4e4de78700 1 -- 172.21.15.167:0/2792530994 <== osd.1 v2:172.21.15.167:6800/30035 38 ==== osd_op_reply(111 data_loggenerations_metadata [watch unwatch cookie 94229237159232] v32'261 uv1 ondisk = 0) v8 ==== 172+0+0 (crc 0 0 0) 0x7f4e48050040 con 0x55b374344940 2023-10-05T19:30:22.653 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.650+0000 7f4e4fb42e40 1 -- 172.21.15.167:0/2792530994 >> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] conn(0x7f4e3800e8c0 msgr2=0x7f4e3800d310 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-05T19:30:22.653 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.650+0000 7f4e4fb42e40 1 --2- 172.21.15.167:0/2792530994 >> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] conn(0x7f4e3800e8c0 0x7f4e3800d310 crc :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-05T19:30:22.653 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.650+0000 7f4e4fb42e40 1 -- 172.21.15.167:0/2792530994 >> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] conn(0x55b374344940 msgr2=0x55b374346d20 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-05T19:30:22.654 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.650+0000 7f4e4fb42e40 1 --2- 172.21.15.167:0/2792530994 >> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] conn(0x55b374344940 0x55b374346d20 crc :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-05T19:30:22.654 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.650+0000 7f4e4fb42e40 1 -- 172.21.15.167:0/2792530994 >> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] conn(0x55b374311d00 msgr2=0x7f4e400373f0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-05T19:30:22.654 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.650+0000 7f4e4fb42e40 1 --2- 172.21.15.167:0/2792530994 >> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] conn(0x55b374311d00 0x7f4e400373f0 crc :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-05T19:30:22.654 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.650+0000 7f4e4fb42e40 1 -- 172.21.15.167:0/2792530994 >> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] conn(0x55b37434a770 msgr2=0x55b37434cb50 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-05T19:30:22.654 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.650+0000 7f4e4fb42e40 1 --2- 172.21.15.167:0/2792530994 >> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] conn(0x55b37434a770 0x55b37434cb50 crc :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-05T19:30:22.654 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.650+0000 7f4e4fb42e40 1 -- 172.21.15.167:0/2792530994 >> [v2:172.21.15.167:6824/29889,v1:172.21.15.167:6825/29889] conn(0x7f4e30030c50 msgr2=0x7f4e30033100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-05T19:30:22.654 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.650+0000 7f4e4fb42e40 1 --2- 172.21.15.167:0/2792530994 >> [v2:172.21.15.167:6824/29889,v1:172.21.15.167:6825/29889] conn(0x7f4e30030c50 0x7f4e30033100 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7f4e3800b300 tx=0x7f4e38001470 comp rx=0 tx=0).stop 2023-10-05T19:30:22.654 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.650+0000 7f4e4fb42e40 1 -- 172.21.15.167:0/2792530994 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55b374312610 msgr2=0x55b37433aec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-05T19:30:22.654 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.650+0000 7f4e4de78700 1 -- 172.21.15.167:0/2792530994 reap_dead start 2023-10-05T19:30:22.654 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.650+0000 7f4e4fb42e40 1 --2- 172.21.15.167:0/2792530994 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55b374312610 0x55b37433aec0 secure :-1 s=READY pgs=671 cs=0 l=1 rev1=1 crypto rx=0x7f4e4000a5c0 tx=0x7f4e4001c000 comp rx=0 tx=0).stop 2023-10-05T19:30:22.654 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.650+0000 7f4e4fb42e40 1 -- 172.21.15.167:0/2792530994 shutdown_connections 2023-10-05T19:30:22.655 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.650+0000 7f4e4fb42e40 1 --2- 172.21.15.167:0/2792530994 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55b374312610 0x55b37433aec0 unknown :-1 s=CLOSED pgs=671 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-05T19:30:22.655 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.650+0000 7f4e4fb42e40 1 -- 172.21.15.167:0/2792530994 >> 172.21.15.167:0/2792530994 conn(0x55b37430d720 msgr2=0x55b37432b460 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-05T19:30:22.655 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.650+0000 7f4e4fb42e40 1 -- 172.21.15.167:0/2792530994 shutdown_connections 2023-10-05T19:30:22.655 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.650+0000 7f4e4fb42e40 1 -- 172.21.15.167:0/2792530994 wait complete. 2023-10-05T19:30:22.656 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.650+0000 7f4e4fb42e40 1 -- 172.21.15.167:0/786787770 >> [v2:172.21.15.167:6824/29889,v1:172.21.15.167:6825/29889] conn(0x7f4d94030bb0 msgr2=0x7f4d94033060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-05T19:30:22.656 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.650+0000 7f4e4fb42e40 1 --2- 172.21.15.167:0/786787770 >> [v2:172.21.15.167:6824/29889,v1:172.21.15.167:6825/29889] conn(0x7f4d94030bb0 0x7f4d94033060 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x55b374355ae0 tx=0x7f4e48000f90 comp rx=0 tx=0).stop 2023-10-05T19:30:22.656 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.650+0000 7f4e4fb42e40 1 -- 172.21.15.167:0/786787770 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55b3743631c0 msgr2=0x55b374368be0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-05T19:30:22.656 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.650+0000 7f4e4fb42e40 1 --2- 172.21.15.167:0/786787770 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55b3743631c0 0x55b374368be0 secure :-1 s=READY pgs=673 cs=0 l=1 rev1=1 crypto rx=0x7f4e3800f4b0 tx=0x7f4e38032000 comp rx=0 tx=0).stop 2023-10-05T19:30:22.656 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.650+0000 7f4e4fb42e40 1 -- 172.21.15.167:0/786787770 shutdown_connections 2023-10-05T19:30:22.656 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.650+0000 7f4e4fb42e40 1 --2- 172.21.15.167:0/786787770 >> [v2:172.21.15.167:6824/29889,v1:172.21.15.167:6825/29889] conn(0x7f4d94030bb0 0x7f4d94033060 unknown :-1 s=CLOSED pgs=231 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-05T19:30:22.656 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.650+0000 7f4e4fb42e40 1 --2- 172.21.15.167:0/786787770 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55b3743631c0 0x55b374368be0 unknown :-1 s=CLOSED pgs=673 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-05T19:30:22.657 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.650+0000 7f4e4fb42e40 1 -- 172.21.15.167:0/786787770 >> 172.21.15.167:0/786787770 conn(0x55b37435e580 msgr2=0x55b37435c680 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-05T19:30:22.657 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.650+0000 7f4e4fb42e40 1 -- 172.21.15.167:0/786787770 shutdown_connections 2023-10-05T19:30:22.657 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.650+0000 7f4e4fb42e40 1 -- 172.21.15.167:0/786787770 wait complete. 2023-10-05T19:30:22.659 DEBUG:teuthology.orchestra.run:got remote process result: 22 2023-10-05T19:30:22.660 INFO:tasks.util.rgw:rgwadmin: client.0 : ['zone', 'get', '--rgw-zone', 'default'] 2023-10-05T19:30:22.660 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'ceph', 'zone', 'get', '--rgw-zone', 'default'] 2023-10-05T19:30:22.660 DEBUG:teuthology.orchestra.run.smithi167:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster ceph zone get --rgw-zone default 2023-10-05T19:30:22.700 INFO:teuthology.orchestra.run.smithi167.stderr:ignoring --setuser ceph since I am not root 2023-10-05T19:30:22.701 INFO:teuthology.orchestra.run.smithi167.stderr:ignoring --setgroup ceph since I am not root 2023-10-05T19:30:22.701 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.698+0000 7fc1ad445e40 1 Processor -- start 2023-10-05T19:30:22.701 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.698+0000 7fc1ad445e40 1 -- start start 2023-10-05T19:30:22.702 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.698+0000 7fc1ad445e40 1 --2- >> v2:172.21.15.118:3300/0 conn(0x55c2f5f9b070 0x55c2f5f99650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-05T19:30:22.702 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.698+0000 7fc1ad445e40 1 -- --> v1:172.21.15.118:6789/0 -- auth(proto 0 26 bytes epoch 0) v1 -- 0x55c2f5f71bd0 con 0x55c2f5f99b90 2023-10-05T19:30:22.702 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.698+0000 7fc1ad445e40 1 -- --> v2:172.21.15.118:3300/0 -- mon_getmap magic: 0 v1 -- 0x55c2f5f2eee0 con 0x55c2f5f9b070 2023-10-05T19:30:22.703 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.698+0000 7fc1aaf7a700 1 --2- >> v2:172.21.15.118:3300/0 conn(0x55c2f5f9b070 0x55c2f5f99650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-05T19:30:22.703 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.698+0000 7fc1aaf7a700 1 --2- >> v2:172.21.15.118:3300/0 conn(0x55c2f5f9b070 0x55c2f5f99650 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.118:3300/0 says I am v2:172.21.15.167:33082/0 (socket says 172.21.15.167:33082) 2023-10-05T19:30:22.703 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.698+0000 7fc1aa779700 1 --1- >> v1:172.21.15.118:6789/0 conn(0x55c2f5f99b90 0x55c2f602cfb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.118:6789/0 says I am v1:172.21.15.167:50816/0 (socket says 172.21.15.167:50816) 2023-10-05T19:30:22.703 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.698+0000 7fc1aaf7a700 1 -- 172.21.15.167:0/3386247027 learned_addr learned my addr 172.21.15.167:0/3386247027 (peer_addr_for_me v2:172.21.15.167:0/0) 2023-10-05T19:30:22.703 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.698+0000 7fc1aaf7a700 1 -- 172.21.15.167:0/3386247027 >> v1:172.21.15.118:6789/0 conn(0x55c2f5f99b90 legacy=0x55c2f602cfb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-05T19:30:22.703 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.698+0000 7fc1aaf7a700 1 -- 172.21.15.167:0/3386247027 --> v2:172.21.15.118:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55c2f5f9b500 con 0x55c2f5f9b070 2023-10-05T19:30:22.704 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.698+0000 7fc1aaf7a700 1 --2- 172.21.15.167:0/3386247027 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55c2f5f9b070 0x55c2f5f99650 secure :-1 s=READY pgs=675 cs=0 l=1 rev1=1 crypto rx=0x7fc19c00a650 tx=0x7fc19c005b10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e8210367fe4cf44e server_cookie=0 in_seq=0 out_seq=0 2023-10-05T19:30:22.704 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.702+0000 7fc1a9f78700 1 -- 172.21.15.167:0/3386247027 <== mon.0 v2:172.21.15.118:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fc19c008760 con 0x55c2f5f9b070 2023-10-05T19:30:22.704 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.702+0000 7fc1a9f78700 1 -- 172.21.15.167:0/3386247027 <== mon.0 v2:172.21.15.118:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc19c0088c0 con 0x55c2f5f9b070 2023-10-05T19:30:22.704 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.702+0000 7fc1a9f78700 1 -- 172.21.15.167:0/3386247027 <== mon.0 v2:172.21.15.118:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fc19c008bb0 con 0x55c2f5f9b070 2023-10-05T19:30:22.705 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.702+0000 7fc1ad445e40 1 -- 172.21.15.167:0/3386247027 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55c2f5f9b070 msgr2=0x55c2f5f99650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-05T19:30:22.705 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.702+0000 7fc1ad445e40 1 --2- 172.21.15.167:0/3386247027 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55c2f5f9b070 0x55c2f5f99650 secure :-1 s=READY pgs=675 cs=0 l=1 rev1=1 crypto rx=0x7fc19c00a650 tx=0x7fc19c005b10 comp rx=0 tx=0).stop 2023-10-05T19:30:22.705 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.702+0000 7fc1ad445e40 1 -- 172.21.15.167:0/3386247027 shutdown_connections 2023-10-05T19:30:22.705 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.702+0000 7fc1ad445e40 1 --2- 172.21.15.167:0/3386247027 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55c2f5f9b070 0x55c2f5f99650 unknown :-1 s=CLOSED pgs=675 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-05T19:30:22.705 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.702+0000 7fc1ad445e40 1 -- 172.21.15.167:0/3386247027 >> 172.21.15.167:0/3386247027 conn(0x55c2f60208b0 msgr2=0x55c2f6022cc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-05T19:30:22.705 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.702+0000 7fc1ad445e40 1 -- 172.21.15.167:0/3386247027 shutdown_connections 2023-10-05T19:30:22.705 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.702+0000 7fc1ad445e40 1 -- 172.21.15.167:0/3386247027 wait complete. 2023-10-05T19:30:22.706 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.702+0000 7fc1ad445e40 1 Processor -- start 2023-10-05T19:30:22.706 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.702+0000 7fc1ad445e40 1 -- start start 2023-10-05T19:30:22.707 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.702+0000 7fc1ad445e40 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55c2f5f99b90 0x55c2f60460f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-05T19:30:22.707 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.702+0000 7fc1ad445e40 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c2f5ef2e30 con 0x55c2f5f99b90 2023-10-05T19:30:22.707 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.702+0000 7fc1aaf7a700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55c2f5f99b90 0x55c2f60460f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-05T19:30:22.707 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.702+0000 7fc1aaf7a700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55c2f5f99b90 0x55c2f60460f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.118:3300/0 says I am v2:172.21.15.167:33094/0 (socket says 172.21.15.167:33094) 2023-10-05T19:30:22.707 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.702+0000 7fc1aaf7a700 1 -- 172.21.15.167:0/766861558 learned_addr learned my addr 172.21.15.167:0/766861558 (peer_addr_for_me v2:172.21.15.167:0/0) 2023-10-05T19:30:22.707 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.702+0000 7fc1aaf7a700 1 -- 172.21.15.167:0/766861558 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc19c005db0 con 0x55c2f5f99b90 2023-10-05T19:30:22.708 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.702+0000 7fc1aaf7a700 1 --2- 172.21.15.167:0/766861558 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55c2f5f99b90 0x55c2f60460f0 secure :-1 s=READY pgs=676 cs=0 l=1 rev1=1 crypto rx=0x7fc19c005b50 tx=0x7fc19c015000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=71694b9faa24372e server_cookie=0 in_seq=0 out_seq=0 2023-10-05T19:30:22.708 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.702+0000 7fc1a8f76700 1 -- 172.21.15.167:0/766861558 <== mon.0 v2:172.21.15.118:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fc19c005460 con 0x55c2f5f99b90 2023-10-05T19:30:22.708 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.702+0000 7fc1a8f76700 1 -- 172.21.15.167:0/766861558 <== mon.0 v2:172.21.15.118:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc19c008a30 con 0x55c2f5f99b90 2023-10-05T19:30:22.708 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.706+0000 7fc1a8f76700 1 -- 172.21.15.167:0/766861558 <== mon.0 v2:172.21.15.118:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fc19c008d00 con 0x55c2f5f99b90 2023-10-05T19:30:22.708 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.706+0000 7fc1ad445e40 1 -- 172.21.15.167:0/766861558 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55c2f5f99b90 msgr2=0x55c2f60460f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-05T19:30:22.709 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.706+0000 7fc1ad445e40 1 --2- 172.21.15.167:0/766861558 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55c2f5f99b90 0x55c2f60460f0 secure :-1 s=READY pgs=676 cs=0 l=1 rev1=1 crypto rx=0x7fc19c005b50 tx=0x7fc19c015000 comp rx=0 tx=0).stop 2023-10-05T19:30:22.709 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.706+0000 7fc1ad445e40 1 -- 172.21.15.167:0/766861558 shutdown_connections 2023-10-05T19:30:22.709 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.706+0000 7fc1ad445e40 1 --2- 172.21.15.167:0/766861558 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55c2f5f99b90 0x55c2f60460f0 unknown :-1 s=CLOSED pgs=676 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-05T19:30:22.709 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.706+0000 7fc1ad445e40 1 -- 172.21.15.167:0/766861558 >> 172.21.15.167:0/766861558 conn(0x55c2f60208b0 msgr2=0x55c2f6041cb0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-05T19:30:22.709 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.706+0000 7fc1ad445e40 1 -- 172.21.15.167:0/766861558 shutdown_connections 2023-10-05T19:30:22.709 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.706+0000 7fc1ad445e40 1 -- 172.21.15.167:0/766861558 wait complete. 2023-10-05T19:30:22.710 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.706+0000 7fc1ad445e40 1 Processor -- start 2023-10-05T19:30:22.710 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.706+0000 7fc1ad445e40 1 -- start start 2023-10-05T19:30:22.710 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.706+0000 7fc1ad445e40 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55c2f5f99b90 0x55c2f604cd50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-05T19:30:22.710 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.706+0000 7fc1ad445e40 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c2f5ec5ed0 con 0x55c2f5f99b90 2023-10-05T19:30:22.710 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.706+0000 7fc1aaf7a700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55c2f5f99b90 0x55c2f604cd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-05T19:30:22.710 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.706+0000 7fc1aaf7a700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55c2f5f99b90 0x55c2f604cd50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.118:3300/0 says I am v2:172.21.15.167:33106/0 (socket says 172.21.15.167:33106) 2023-10-05T19:30:22.710 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.706+0000 7fc1aaf7a700 1 -- 172.21.15.167:0/644066034 learned_addr learned my addr 172.21.15.167:0/644066034 (peer_addr_for_me v2:172.21.15.167:0/0) 2023-10-05T19:30:22.711 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.706+0000 7fc1aaf7a700 1 -- 172.21.15.167:0/644066034 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55c2f5f9b500 con 0x55c2f5f99b90 2023-10-05T19:30:22.711 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.706+0000 7fc1aaf7a700 1 --2- 172.21.15.167:0/644066034 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55c2f5f99b90 0x55c2f604cd50 secure :-1 s=READY pgs=677 cs=0 l=1 rev1=1 crypto rx=0x7fc19c01f040 tx=0x7fc19c01c000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-05T19:30:22.711 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.706+0000 7fc19affd700 1 -- 172.21.15.167:0/644066034 <== mon.0 v2:172.21.15.118:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fc19c012450 con 0x55c2f5f99b90 2023-10-05T19:30:22.712 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.706+0000 7fc1ad445e40 1 -- 172.21.15.167:0/644066034 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x55c2f604d290 con 0x55c2f5f99b90 2023-10-05T19:30:22.712 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.706+0000 7fc19affd700 1 -- 172.21.15.167:0/644066034 <== mon.0 v2:172.21.15.118:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc19c0125b0 con 0x55c2f5f99b90 2023-10-05T19:30:22.712 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.706+0000 7fc1ad445e40 1 -- 172.21.15.167:0/644066034 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55c2f604d730 con 0x55c2f5f99b90 2023-10-05T19:30:22.712 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.706+0000 7fc19affd700 1 -- 172.21.15.167:0/644066034 <== mon.0 v2:172.21.15.118:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fc19c012760 con 0x55c2f5f99b90 2023-10-05T19:30:22.713 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.710+0000 7fc1ad445e40 20 check_secure_mon_conn(): auth registy supported: methods=[2] modes=[2,1] 2023-10-05T19:30:22.713 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.710+0000 7fc1ad445e40 20 check_secure_mon_conn(): mode 1 is insecure 2023-10-05T19:30:22.713 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.710+0000 7fc1ad445e40 1 Processor -- start 2023-10-05T19:30:22.713 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.710+0000 7fc1ad445e40 1 -- start start 2023-10-05T19:30:22.713 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.710+0000 7fc1ad445e40 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55c2f605e490 0x55c2f605e860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-05T19:30:22.713 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.710+0000 7fc1ad445e40 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c2f605eda0 con 0x55c2f605e490 2023-10-05T19:30:22.713 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.710+0000 7fc19affd700 1 -- 172.21.15.167:0/644066034 <== mon.0 v2:172.21.15.118:3300/0 4 ==== mgrmap(e 5) v1 ==== 40329+0+0 (secure 0 0 0) 0x7fc19c019070 con 0x55c2f5f99b90 2023-10-05T19:30:22.714 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.710+0000 7fc1ab77b700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55c2f605e490 0x55c2f605e860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-05T19:30:22.715 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.710+0000 7fc1ab77b700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55c2f605e490 0x55c2f605e860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.118:3300/0 says I am v2:172.21.15.167:33110/0 (socket says 172.21.15.167:33110) 2023-10-05T19:30:22.715 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.710+0000 7fc1ab77b700 1 -- 172.21.15.167:0/2107446111 learned_addr learned my addr 172.21.15.167:0/2107446111 (peer_addr_for_me v2:172.21.15.167:0/0) 2023-10-05T19:30:22.715 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.710+0000 7fc19affd700 1 --2- 172.21.15.167:0/644066034 >> [v2:172.21.15.167:6824/29889,v1:172.21.15.167:6825/29889] conn(0x7fc190030c50 0x7fc190033100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-05T19:30:22.715 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.710+0000 7fc1ab77b700 1 -- 172.21.15.167:0/2107446111 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55c2f605eee0 con 0x55c2f605e490 2023-10-05T19:30:22.715 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.710+0000 7fc19affd700 1 -- 172.21.15.167:0/644066034 <== mon.0 v2:172.21.15.118:3300/0 5 ==== osd_map(32..32 src has 1..32) v4 ==== 6260+0+0 (secure 0 0 0) 0x7fc19c0464e0 con 0x55c2f5f99b90 2023-10-05T19:30:22.715 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.710+0000 7fc1aaf7a700 1 --2- 172.21.15.167:0/644066034 >> [v2:172.21.15.167:6824/29889,v1:172.21.15.167:6825/29889] conn(0x7fc190030c50 0x7fc190033100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-05T19:30:22.715 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.710+0000 7fc1ab77b700 1 --2- 172.21.15.167:0/2107446111 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55c2f605e490 0x55c2f605e860 secure :-1 s=READY pgs=678 cs=0 l=1 rev1=1 crypto rx=0x7fc1a400a3b0 tx=0x7fc1a4005cc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=624c5a9bfc71b22b server_cookie=0 in_seq=0 out_seq=0 2023-10-05T19:30:22.715 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.710+0000 7fc17bfe7700 1 -- 172.21.15.167:0/2107446111 <== mon.0 v2:172.21.15.118:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fc1a4001520 con 0x55c2f605e490 2023-10-05T19:30:22.715 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.710+0000 7fc17bfe7700 1 -- 172.21.15.167:0/2107446111 <== mon.0 v2:172.21.15.118:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc1a4001680 con 0x55c2f605e490 2023-10-05T19:30:22.715 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.710+0000 7fc17bfe7700 1 -- 172.21.15.167:0/2107446111 <== mon.0 v2:172.21.15.118:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fc1a4001830 con 0x55c2f605e490 2023-10-05T19:30:22.717 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.710+0000 7fc1ad445e40 1 -- 172.21.15.167:0/2107446111 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55c2f605e490 msgr2=0x55c2f605e860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-05T19:30:22.717 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.710+0000 7fc1ad445e40 1 --2- 172.21.15.167:0/2107446111 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55c2f605e490 0x55c2f605e860 secure :-1 s=READY pgs=678 cs=0 l=1 rev1=1 crypto rx=0x7fc1a400a3b0 tx=0x7fc1a4005cc0 comp rx=0 tx=0).stop 2023-10-05T19:30:22.717 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.710+0000 7fc1aaf7a700 1 --2- 172.21.15.167:0/644066034 >> [v2:172.21.15.167:6824/29889,v1:172.21.15.167:6825/29889] conn(0x7fc190030c50 0x7fc190033100 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7fc19c0447e0 tx=0x7fc19c044710 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-05T19:30:22.717 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.710+0000 7fc1ad445e40 1 -- 172.21.15.167:0/2107446111 shutdown_connections 2023-10-05T19:30:22.717 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.710+0000 7fc1ad445e40 1 --2- 172.21.15.167:0/2107446111 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55c2f605e490 0x55c2f605e860 unknown :-1 s=CLOSED pgs=678 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-05T19:30:22.717 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.710+0000 7fc1ad445e40 1 -- 172.21.15.167:0/2107446111 >> 172.21.15.167:0/2107446111 conn(0x55c2f5f9b070 msgr2=0x55c2f605b1e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-05T19:30:22.717 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.710+0000 7fc1ad445e40 1 -- 172.21.15.167:0/2107446111 shutdown_connections 2023-10-05T19:30:22.718 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.710+0000 7fc1ad445e40 1 -- 172.21.15.167:0/2107446111 wait complete. 2023-10-05T19:30:22.718 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.714+0000 7fc1ad445e40 1 Processor -- start 2023-10-05T19:30:22.718 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.714+0000 7fc1ad445e40 1 -- start start 2023-10-05T19:30:22.718 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.714+0000 7fc1ad445e40 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55c2f605e490 0x55c2f6065610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-05T19:30:22.718 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.714+0000 7fc1ad445e40 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c2f6066610 con 0x55c2f605e490 2023-10-05T19:30:22.718 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.714+0000 7fc1ab77b700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55c2f605e490 0x55c2f6065610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-05T19:30:22.718 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.714+0000 7fc1ab77b700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55c2f605e490 0x55c2f6065610 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.118:3300/0 says I am v2:172.21.15.167:33120/0 (socket says 172.21.15.167:33120) 2023-10-05T19:30:22.718 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.714+0000 7fc1ab77b700 1 -- 172.21.15.167:0/905352282 learned_addr learned my addr 172.21.15.167:0/905352282 (peer_addr_for_me v2:172.21.15.167:0/0) 2023-10-05T19:30:22.718 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.714+0000 7fc1ab77b700 1 -- 172.21.15.167:0/905352282 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc1a400a0e0 con 0x55c2f605e490 2023-10-05T19:30:22.719 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.714+0000 7fc1ab77b700 1 --2- 172.21.15.167:0/905352282 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55c2f605e490 0x55c2f6065610 secure :-1 s=READY pgs=679 cs=0 l=1 rev1=1 crypto rx=0x7fc1a4000f80 tx=0x7fc1a4014000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-05T19:30:22.719 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.714+0000 7fc17a7e4700 1 -- 172.21.15.167:0/905352282 <== mon.0 v2:172.21.15.118:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fc1a400aec0 con 0x55c2f605e490 2023-10-05T19:30:22.719 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.714+0000 7fc17a7e4700 1 -- 172.21.15.167:0/905352282 <== mon.0 v2:172.21.15.118:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc1a4010ba0 con 0x55c2f605e490 2023-10-05T19:30:22.719 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.714+0000 7fc1ad445e40 1 -- 172.21.15.167:0/905352282 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x55c2f6065c10 con 0x55c2f605e490 2023-10-05T19:30:22.719 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.714+0000 7fc17a7e4700 1 -- 172.21.15.167:0/905352282 <== mon.0 v2:172.21.15.118:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fc1a4010e70 con 0x55c2f605e490 2023-10-05T19:30:22.720 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.714+0000 7fc1ad445e40 1 -- 172.21.15.167:0/905352282 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55c2f60660d0 con 0x55c2f605e490 2023-10-05T19:30:22.720 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.714+0000 7fc17a7e4700 1 -- 172.21.15.167:0/905352282 <== mon.0 v2:172.21.15.118:3300/0 4 ==== mgrmap(e 5) v1 ==== 40329+0+0 (secure 0 0 0) 0x7fc1a4023020 con 0x55c2f605e490 2023-10-05T19:30:22.721 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.714+0000 7fc17a7e4700 1 --2- 172.21.15.167:0/905352282 >> [v2:172.21.15.167:6824/29889,v1:172.21.15.167:6825/29889] conn(0x7fc0f8030c50 0x7fc0f8033100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-05T19:30:22.721 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.718+0000 7fc17a7e4700 1 -- 172.21.15.167:0/905352282 <== mon.0 v2:172.21.15.118:3300/0 5 ==== osd_map(32..32 src has 1..32) v4 ==== 6260+0+0 (secure 0 0 0) 0x7fc1a4045a90 con 0x55c2f605e490 2023-10-05T19:30:22.721 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.718+0000 7fc0f3fff700 20 reqs_thread_entry: start 2023-10-05T19:30:22.721 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.718+0000 7fc1ad445e40 20 try_to_resolve_local_entities(): before: realm_id= realm_name= zonegroup_id= zonegroup_name= zone_id= zone_name=default 2023-10-05T19:30:22.721 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.718+0000 7fc1ad445e40 20 rados->read ofs=0 len=0 2023-10-05T19:30:22.721 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.718+0000 7fc1ad445e40 1 --2- 172.21.15.167:0/644066034 >> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] conn(0x55c2f60833d0 0x55c2f6083870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-05T19:30:22.721 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.718+0000 7fc1ad445e40 1 -- 172.21.15.167:0/644066034 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:1 5.f 5:f43b6ece:::zone_names.default:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32) v8 -- 0x55c2f6096910 con 0x55c2f60833d0 2023-10-05T19:30:22.721 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.718+0000 7fc1aa779700 1 --2- 172.21.15.167:0/905352282 >> [v2:172.21.15.167:6824/29889,v1:172.21.15.167:6825/29889] conn(0x7fc0f8030c50 0x7fc0f8033100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-05T19:30:22.722 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.718+0000 7fc1ab77b700 1 --2- 172.21.15.167:0/644066034 >> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] conn(0x55c2f60833d0 0x55c2f6083870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-05T19:30:22.722 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.718+0000 7fc1aa779700 1 --2- 172.21.15.167:0/905352282 >> [v2:172.21.15.167:6824/29889,v1:172.21.15.167:6825/29889] conn(0x7fc0f8030c50 0x7fc0f8033100 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7fc1a000b230 tx=0x7fc1a00013f0 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-05T19:30:22.722 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.718+0000 7fc1ab77b700 1 --2- 172.21.15.167:0/644066034 >> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] conn(0x55c2f60833d0 0x55c2f6083870 crc :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-05T19:30:22.723 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.718+0000 7fc1ab77b700 1 -- 172.21.15.167:0/644066034 <== osd.1 v2:172.21.15.167:6800/30035 1 ==== osd_op_reply(1 zone_names.default [read 0~46 out=46b] v0'0 uv1 ondisk = 0) v8 ==== 162+0+46 (crc 0 0 0) 0x7fc1a401cce0 con 0x55c2f60833d0 2023-10-05T19:30:22.723 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.718+0000 7fc1ad445e40 20 rados_obj.operate() r=0 bl.length=46 2023-10-05T19:30:22.723 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.718+0000 7fc1ad445e40 20 rados->read ofs=0 len=0 2023-10-05T19:30:22.723 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.718+0000 7fc1ad445e40 1 -- 172.21.15.167:0/644066034 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:2 5.1d 5:b8374eb4:::zone_info.9a9df965-6612-45d4-a45f-6e3c889bacc2:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32) v8 -- 0x55c2f6096cc0 con 0x55c2f60833d0 2023-10-05T19:30:22.723 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.718+0000 7fc1ab77b700 1 -- 172.21.15.167:0/644066034 <== osd.1 v2:172.21.15.167:6800/30035 2 ==== osd_op_reply(2 zone_info.9a9df965-6612-45d4-a45f-6e3c889bacc2 [read 0~874 out=874b] v0'0 uv2 ondisk = 0) v8 ==== 190+0+874 (crc 0 0 0) 0x7fc1a401cce0 con 0x55c2f60833d0 2023-10-05T19:30:22.723 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.718+0000 7fc1ad445e40 20 rados_obj.operate() r=0 bl.length=874 2023-10-05T19:30:22.723 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.718+0000 7fc1ad445e40 1 --2- 172.21.15.167:0/644066034 >> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] conn(0x55c2f6098590 0x55c2f609a970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-05T19:30:22.723 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.718+0000 7fc1ad445e40 1 -- 172.21.15.167:0/644066034 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:3 5.0 5:00000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x55c2f609aeb0 con 0x55c2f6098590 2023-10-05T19:30:22.724 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.718+0000 7fc1aaf7a700 1 --2- 172.21.15.167:0/644066034 >> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] conn(0x55c2f6098590 0x55c2f609a970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-05T19:30:22.724 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.718+0000 7fc1aaf7a700 1 --2- 172.21.15.167:0/644066034 >> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] conn(0x55c2f6098590 0x55c2f609a970 crc :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-05T19:30:22.724 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.722+0000 7fc1aaf7a700 1 -- 172.21.15.167:0/644066034 <== osd.3 v2:172.21.15.167:6808/30036 1 ==== osd_op_reply(3 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fc19c0443d0 con 0x55c2f6098590 2023-10-05T19:30:22.724 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.722+0000 7fc1aaf7a700 1 --2- 172.21.15.167:0/644066034 >> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] conn(0x7fc19c007030 0x7fc19c02e810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-05T19:30:22.725 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.722+0000 7fc1aaf7a700 1 -- 172.21.15.167:0/644066034 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:4 5.10 5:08000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x55c2f609aeb0 con 0x7fc19c007030 2023-10-05T19:30:22.725 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.722+0000 7fc1aa779700 1 --2- 172.21.15.167:0/644066034 >> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] conn(0x7fc19c007030 0x7fc19c02e810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-05T19:30:22.725 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.722+0000 7fc1aa779700 1 --2- 172.21.15.167:0/644066034 >> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] conn(0x7fc19c007030 0x7fc19c02e810 crc :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-05T19:30:22.726 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.722+0000 7fc1aa779700 1 -- 172.21.15.167:0/644066034 <== osd.2 v2:172.21.15.167:6816/30037 1 ==== osd_op_reply(4 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fc1a000f600 con 0x7fc19c007030 2023-10-05T19:30:22.726 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.722+0000 7fc1aa779700 1 --2- 172.21.15.167:0/644066034 >> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] conn(0x7fc1a000e8c0 0x7fc1a000d310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-05T19:30:22.726 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.722+0000 7fc1aa779700 1 -- 172.21.15.167:0/644066034 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:5 5.8 5:10000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x55c2f609aeb0 con 0x7fc1a000e8c0 2023-10-05T19:30:22.726 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.722+0000 7fc1ab77b700 1 --2- 172.21.15.167:0/644066034 >> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] conn(0x7fc1a000e8c0 0x7fc1a000d310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-05T19:30:22.727 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.722+0000 7fc1ab77b700 1 --2- 172.21.15.167:0/644066034 >> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] conn(0x7fc1a000e8c0 0x7fc1a000d310 crc :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-05T19:30:22.728 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.726+0000 7fc1ab77b700 1 -- 172.21.15.167:0/644066034 <== osd.0 v2:172.21.15.118:6800/27690 1 ==== osd_op_reply(5 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fc1a401cce0 con 0x7fc1a000e8c0 2023-10-05T19:30:22.728 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.726+0000 7fc1ab77b700 1 -- 172.21.15.167:0/644066034 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:6 5.18 5:18000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x55c2f609aeb0 con 0x55c2f60833d0 2023-10-05T19:30:22.728 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.726+0000 7fc1ab77b700 1 -- 172.21.15.167:0/644066034 <== osd.1 v2:172.21.15.167:6800/30035 3 ==== osd_op_reply(6 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fc1a401cce0 con 0x55c2f60833d0 2023-10-05T19:30:22.728 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.726+0000 7fc1ab77b700 1 -- 172.21.15.167:0/644066034 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:7 5.4 5:20000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x55c2f609aeb0 con 0x55c2f60833d0 2023-10-05T19:30:22.729 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.726+0000 7fc1ab77b700 1 -- 172.21.15.167:0/644066034 <== osd.1 v2:172.21.15.167:6800/30035 4 ==== osd_op_reply(7 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fc1a401cce0 con 0x55c2f60833d0 2023-10-05T19:30:22.729 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.726+0000 7fc1ab77b700 1 -- 172.21.15.167:0/644066034 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:8 5.14 5:28000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x55c2f609aeb0 con 0x7fc1a000e8c0 2023-10-05T19:30:22.730 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.726+0000 7fc1ab77b700 1 -- 172.21.15.167:0/644066034 <== osd.0 v2:172.21.15.118:6800/27690 2 ==== osd_op_reply(8 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fc1a401cce0 con 0x7fc1a000e8c0 2023-10-05T19:30:22.730 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.726+0000 7fc1ab77b700 1 -- 172.21.15.167:0/644066034 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:9 5.c 5:30000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x55c2f609aeb0 con 0x7fc1a000e8c0 2023-10-05T19:30:22.731 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.726+0000 7fc1ab77b700 1 -- 172.21.15.167:0/644066034 <== osd.0 v2:172.21.15.118:6800/27690 3 ==== osd_op_reply(9 [pgnls start_epoch 32 out=112b] v23'1 uv1 ondisk = 1) v8 ==== 144+0+112 (crc 0 0 0) 0x7fc1a401cce0 con 0x7fc1a000e8c0 2023-10-05T19:30:22.731 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.726+0000 7fc1ab77b700 1 -- 172.21.15.167:0/644066034 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:10 5.1c 5:38000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x55c2f609aeb0 con 0x7fc19c007030 2023-10-05T19:30:22.732 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.730+0000 7fc1aa779700 1 -- 172.21.15.167:0/644066034 <== osd.2 v2:172.21.15.167:6816/30037 2 ==== osd_op_reply(10 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fc1a000f600 con 0x7fc19c007030 2023-10-05T19:30:22.732 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.730+0000 7fc1aa779700 1 -- 172.21.15.167:0/644066034 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:11 5.2 5:40000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x55c2f609aeb0 con 0x55c2f6098590 2023-10-05T19:30:22.733 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.730+0000 7fc1aaf7a700 1 -- 172.21.15.167:0/644066034 <== osd.3 v2:172.21.15.167:6808/30036 2 ==== osd_op_reply(11 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fc19c0443d0 con 0x55c2f6098590 2023-10-05T19:30:22.733 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.730+0000 7fc1aaf7a700 1 -- 172.21.15.167:0/644066034 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:12 5.12 5:48000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x55c2f609aeb0 con 0x55c2f60833d0 2023-10-05T19:30:22.734 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.730+0000 7fc1ab77b700 1 -- 172.21.15.167:0/644066034 <== osd.1 v2:172.21.15.167:6800/30035 5 ==== osd_op_reply(12 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fc1a401cce0 con 0x55c2f60833d0 2023-10-05T19:30:22.734 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.730+0000 7fc1ab77b700 1 -- 172.21.15.167:0/644066034 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:13 5.a 5:50000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x55c2f609aeb0 con 0x7fc1a000e8c0 2023-10-05T19:30:22.735 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.730+0000 7fc1ab77b700 1 -- 172.21.15.167:0/644066034 <== osd.0 v2:172.21.15.118:6800/27690 4 ==== osd_op_reply(13 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fc1a401cce0 con 0x7fc1a000e8c0 2023-10-05T19:30:22.735 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.730+0000 7fc1ab77b700 1 -- 172.21.15.167:0/644066034 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:14 5.1a 5:58000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x55c2f609aeb0 con 0x55c2f60833d0 2023-10-05T19:30:22.735 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.734+0000 7fc1ab77b700 1 -- 172.21.15.167:0/644066034 <== osd.1 v2:172.21.15.167:6800/30035 6 ==== osd_op_reply(14 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fc1a401cce0 con 0x55c2f60833d0 2023-10-05T19:30:22.736 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.734+0000 7fc1ab77b700 1 -- 172.21.15.167:0/644066034 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:15 5.6 5:60000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x55c2f609aeb0 con 0x7fc19c007030 2023-10-05T19:30:22.736 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.734+0000 7fc1aa779700 1 -- 172.21.15.167:0/644066034 <== osd.2 v2:172.21.15.167:6816/30037 3 ==== osd_op_reply(15 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fc1a000f600 con 0x7fc19c007030 2023-10-05T19:30:22.736 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.734+0000 7fc1aa779700 1 -- 172.21.15.167:0/644066034 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:16 5.16 5:68000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x7fc1a001f040 con 0x55c2f6098590 2023-10-05T19:30:22.737 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.734+0000 7fc1aaf7a700 1 -- 172.21.15.167:0/644066034 <== osd.3 v2:172.21.15.167:6808/30036 3 ==== osd_op_reply(16 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fc19c0443d0 con 0x55c2f6098590 2023-10-05T19:30:22.737 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.734+0000 7fc1aaf7a700 1 -- 172.21.15.167:0/644066034 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:17 5.e 5:70000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x7fc1a001f040 con 0x55c2f6098590 2023-10-05T19:30:22.738 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.734+0000 7fc1aaf7a700 1 -- 172.21.15.167:0/644066034 <== osd.3 v2:172.21.15.167:6808/30036 4 ==== osd_op_reply(17 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fc19c0443d0 con 0x55c2f6098590 2023-10-05T19:30:22.738 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.734+0000 7fc1aaf7a700 1 -- 172.21.15.167:0/644066034 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:18 5.1e 5:78000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x7fc1a001f040 con 0x7fc1a000e8c0 2023-10-05T19:30:22.739 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.734+0000 7fc1ab77b700 1 -- 172.21.15.167:0/644066034 <== osd.0 v2:172.21.15.118:6800/27690 5 ==== osd_op_reply(18 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fc1a401cce0 con 0x7fc1a000e8c0 2023-10-05T19:30:22.739 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.734+0000 7fc1ab77b700 1 -- 172.21.15.167:0/644066034 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:19 5.1 5:80000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x7fc1a001f040 con 0x55c2f6098590 2023-10-05T19:30:22.740 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.738+0000 7fc1aaf7a700 1 -- 172.21.15.167:0/644066034 <== osd.3 v2:172.21.15.167:6808/30036 5 ==== osd_op_reply(19 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fc19c0443d0 con 0x55c2f6098590 2023-10-05T19:30:22.740 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.738+0000 7fc1aaf7a700 1 -- 172.21.15.167:0/644066034 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:20 5.11 5:88000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x7fc1a001f040 con 0x55c2f6098590 2023-10-05T19:30:22.740 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.738+0000 7fc1aaf7a700 1 -- 172.21.15.167:0/644066034 <== osd.3 v2:172.21.15.167:6808/30036 6 ==== osd_op_reply(20 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fc19c0443d0 con 0x55c2f6098590 2023-10-05T19:30:22.741 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.738+0000 7fc1aaf7a700 1 -- 172.21.15.167:0/644066034 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:21 5.9 5:90000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x7fc1a001f040 con 0x55c2f60833d0 2023-10-05T19:30:22.741 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.738+0000 7fc1ab77b700 1 -- 172.21.15.167:0/644066034 <== osd.1 v2:172.21.15.167:6800/30035 7 ==== osd_op_reply(21 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fc1a401cce0 con 0x55c2f60833d0 2023-10-05T19:30:22.742 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.738+0000 7fc1ab77b700 1 -- 172.21.15.167:0/644066034 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:22 5.19 5:98000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x7fc1a001f040 con 0x55c2f60833d0 2023-10-05T19:30:22.742 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.738+0000 7fc1ab77b700 1 -- 172.21.15.167:0/644066034 <== osd.1 v2:172.21.15.167:6800/30035 8 ==== osd_op_reply(22 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fc1a401cce0 con 0x55c2f60833d0 2023-10-05T19:30:22.742 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.738+0000 7fc1ab77b700 1 -- 172.21.15.167:0/644066034 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:23 5.5 5:a0000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x7fc1a001f040 con 0x7fc1a000e8c0 2023-10-05T19:30:22.743 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.738+0000 7fc1ab77b700 1 -- 172.21.15.167:0/644066034 <== osd.0 v2:172.21.15.118:6800/27690 6 ==== osd_op_reply(23 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fc1a401cce0 con 0x7fc1a000e8c0 2023-10-05T19:30:22.743 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.738+0000 7fc1ab77b700 1 -- 172.21.15.167:0/644066034 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:24 5.15 5:a8000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x7fc1a001f040 con 0x55c2f6098590 2023-10-05T19:30:22.744 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.742+0000 7fc1aaf7a700 1 -- 172.21.15.167:0/644066034 <== osd.3 v2:172.21.15.167:6808/30036 7 ==== osd_op_reply(24 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fc19c0443d0 con 0x55c2f6098590 2023-10-05T19:30:22.744 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.742+0000 7fc1aaf7a700 1 -- 172.21.15.167:0/644066034 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:25 5.d 5:b0000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x7fc1a001f040 con 0x7fc19c007030 2023-10-05T19:30:22.745 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.742+0000 7fc1aa779700 1 -- 172.21.15.167:0/644066034 <== osd.2 v2:172.21.15.167:6816/30037 4 ==== osd_op_reply(25 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fc1a000f600 con 0x7fc19c007030 2023-10-05T19:30:22.745 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.742+0000 7fc1aa779700 1 -- 172.21.15.167:0/644066034 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:26 5.1d 5:b8000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x7fc1a001f040 con 0x55c2f60833d0 2023-10-05T19:30:22.746 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.742+0000 7fc1ab77b700 1 -- 172.21.15.167:0/644066034 <== osd.1 v2:172.21.15.167:6800/30035 9 ==== osd_op_reply(26 [pgnls start_epoch 32 out=107b] v30'2 uv2 ondisk = 1) v8 ==== 144+0+107 (crc 0 0 0) 0x7fc1a401cce0 con 0x55c2f60833d0 2023-10-05T19:30:22.746 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.742+0000 7fc1ab77b700 1 -- 172.21.15.167:0/644066034 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:27 5.3 5:c0000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x7fc1a001f040 con 0x7fc1a000e8c0 2023-10-05T19:30:22.747 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.742+0000 7fc1ab77b700 1 -- 172.21.15.167:0/644066034 <== osd.0 v2:172.21.15.118:6800/27690 7 ==== osd_op_reply(27 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fc1a401cce0 con 0x7fc1a000e8c0 2023-10-05T19:30:22.747 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.742+0000 7fc1ab77b700 1 -- 172.21.15.167:0/644066034 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:28 5.13 5:c8000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x7fc1a001f040 con 0x55c2f6098590 2023-10-05T19:30:22.748 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.746+0000 7fc1aaf7a700 1 -- 172.21.15.167:0/644066034 <== osd.3 v2:172.21.15.167:6808/30036 8 ==== osd_op_reply(28 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fc19c0443d0 con 0x55c2f6098590 2023-10-05T19:30:22.748 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.746+0000 7fc1aaf7a700 1 -- 172.21.15.167:0/644066034 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:29 5.b 5:d0000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x7fc1a001f040 con 0x7fc1a000e8c0 2023-10-05T19:30:22.748 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.746+0000 7fc1ab77b700 1 -- 172.21.15.167:0/644066034 <== osd.0 v2:172.21.15.118:6800/27690 8 ==== osd_op_reply(29 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fc1a401cce0 con 0x7fc1a000e8c0 2023-10-05T19:30:22.749 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.746+0000 7fc1ab77b700 1 -- 172.21.15.167:0/644066034 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:30 5.1b 5:d8000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x7fc1a001f040 con 0x7fc19c007030 2023-10-05T19:30:22.749 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.746+0000 7fc1aa779700 1 -- 172.21.15.167:0/644066034 <== osd.2 v2:172.21.15.167:6816/30037 5 ==== osd_op_reply(30 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fc1a000f600 con 0x7fc19c007030 2023-10-05T19:30:22.750 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.746+0000 7fc1aa779700 1 -- 172.21.15.167:0/644066034 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:31 5.7 5:e0000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x7fc1a001f040 con 0x55c2f6098590 2023-10-05T19:30:22.750 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.746+0000 7fc1aaf7a700 1 -- 172.21.15.167:0/644066034 <== osd.3 v2:172.21.15.167:6808/30036 9 ==== osd_op_reply(31 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fc19c0443d0 con 0x55c2f6098590 2023-10-05T19:30:22.750 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.746+0000 7fc1aaf7a700 1 -- 172.21.15.167:0/644066034 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:32 5.17 5:e8000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x7fc1a001f040 con 0x7fc1a000e8c0 2023-10-05T19:30:22.751 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.746+0000 7fc1ab77b700 1 -- 172.21.15.167:0/644066034 <== osd.0 v2:172.21.15.118:6800/27690 9 ==== osd_op_reply(32 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fc1a401cce0 con 0x7fc1a000e8c0 2023-10-05T19:30:22.751 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.746+0000 7fc1ab77b700 1 -- 172.21.15.167:0/644066034 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:33 5.f 5:f0000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x7fc1a001f040 con 0x55c2f60833d0 2023-10-05T19:30:22.752 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.750+0000 7fc1ab77b700 1 -- 172.21.15.167:0/644066034 <== osd.1 v2:172.21.15.167:6800/30035 10 ==== osd_op_reply(33 [pgnls start_epoch 32 out=115b] v23'2 uv2 ondisk = 1) v8 ==== 144+0+115 (crc 0 0 0) 0x7fc1a401cce0 con 0x55c2f60833d0 2023-10-05T19:30:22.752 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.750+0000 7fc1ab77b700 1 -- 172.21.15.167:0/644066034 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:34 5.1f 5:f8000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x7fc1a001f040 con 0x7fc19c007030 2023-10-05T19:30:22.753 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.750+0000 7fc1aa779700 1 -- 172.21.15.167:0/644066034 <== osd.2 v2:172.21.15.167:6816/30037 6 ==== osd_op_reply(34 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fc1a000f600 con 0x7fc19c007030 2023-10-05T19:30:22.753 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.750+0000 7fc1ad445e40 20 RGWRados::pool_iterate: got zonegroup_info.4f1f719d-9648-417b-b10c-e5d939825419 2023-10-05T19:30:22.753 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.750+0000 7fc1ad445e40 20 RGWRados::pool_iterate: got zone_info.9a9df965-6612-45d4-a45f-6e3c889bacc2 2023-10-05T19:30:22.753 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.750+0000 7fc1ad445e40 20 RGWRados::pool_iterate: got zone_names.default 2023-10-05T19:30:22.754 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.750+0000 7fc1ad445e40 20 RGWRados::pool_iterate: got zonegroups_names.default 2023-10-05T19:30:22.754 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.750+0000 7fc1ad445e40 20 rados->read ofs=0 len=0 2023-10-05T19:30:22.754 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.750+0000 7fc1ad445e40 1 -- 172.21.15.167:0/644066034 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:35 5.1d 5:b8374eb4:::zone_info.9a9df965-6612-45d4-a45f-6e3c889bacc2:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32) v8 -- 0x55c2f6097070 con 0x55c2f60833d0 2023-10-05T19:30:22.754 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.750+0000 7fc1ab77b700 1 -- 172.21.15.167:0/644066034 <== osd.1 v2:172.21.15.167:6800/30035 11 ==== osd_op_reply(35 zone_info.9a9df965-6612-45d4-a45f-6e3c889bacc2 [read 0~874 out=874b] v0'0 uv2 ondisk = 0) v8 ==== 190+0+874 (crc 0 0 0) 0x7fc1a401cce0 con 0x55c2f60833d0 2023-10-05T19:30:22.754 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.750+0000 7fc1ad445e40 20 rados_obj.operate() r=0 bl.length=874 2023-10-05T19:30:22.756 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.754+0000 7fc1ad445e40 1 -- 172.21.15.167:0/644066034 >> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] conn(0x7fc1a000e8c0 msgr2=0x7fc1a000d310 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-05T19:30:22.756 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.754+0000 7fc1ad445e40 1 --2- 172.21.15.167:0/644066034 >> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] conn(0x7fc1a000e8c0 0x7fc1a000d310 crc :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-05T19:30:22.756 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.754+0000 7fc1ad445e40 1 -- 172.21.15.167:0/644066034 >> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] conn(0x55c2f60833d0 msgr2=0x55c2f6083870 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-05T19:30:22.757 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.754+0000 7fc1ad445e40 1 --2- 172.21.15.167:0/644066034 >> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] conn(0x55c2f60833d0 0x55c2f6083870 crc :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-05T19:30:22.757 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.754+0000 7fc1ad445e40 1 -- 172.21.15.167:0/644066034 >> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] conn(0x7fc19c007030 msgr2=0x7fc19c02e810 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-05T19:30:22.757 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.754+0000 7fc1ad445e40 1 --2- 172.21.15.167:0/644066034 >> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] conn(0x7fc19c007030 0x7fc19c02e810 crc :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-05T19:30:22.757 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.754+0000 7fc1ad445e40 1 -- 172.21.15.167:0/644066034 >> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] conn(0x55c2f6098590 msgr2=0x55c2f609a970 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-05T19:30:22.757 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.754+0000 7fc1ad445e40 1 --2- 172.21.15.167:0/644066034 >> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] conn(0x55c2f6098590 0x55c2f609a970 crc :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-05T19:30:22.757 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.754+0000 7fc1ad445e40 1 -- 172.21.15.167:0/644066034 >> [v2:172.21.15.167:6824/29889,v1:172.21.15.167:6825/29889] conn(0x7fc190030c50 msgr2=0x7fc190033100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-05T19:30:22.757 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.754+0000 7fc1ad445e40 1 --2- 172.21.15.167:0/644066034 >> [v2:172.21.15.167:6824/29889,v1:172.21.15.167:6825/29889] conn(0x7fc190030c50 0x7fc190033100 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7fc19c0447e0 tx=0x7fc19c044710 comp rx=0 tx=0).stop 2023-10-05T19:30:22.757 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.754+0000 7fc1ad445e40 1 -- 172.21.15.167:0/644066034 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55c2f5f99b90 msgr2=0x55c2f604cd50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-05T19:30:22.757 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.754+0000 7fc1ab77b700 1 -- 172.21.15.167:0/644066034 reap_dead start 2023-10-05T19:30:22.757 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.754+0000 7fc1ad445e40 1 --2- 172.21.15.167:0/644066034 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55c2f5f99b90 0x55c2f604cd50 secure :-1 s=READY pgs=677 cs=0 l=1 rev1=1 crypto rx=0x7fc19c01f040 tx=0x7fc19c01c000 comp rx=0 tx=0).stop 2023-10-05T19:30:22.757 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.754+0000 7fc1ad445e40 1 -- 172.21.15.167:0/644066034 shutdown_connections 2023-10-05T19:30:22.757 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.754+0000 7fc1ad445e40 1 --2- 172.21.15.167:0/644066034 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55c2f5f99b90 0x55c2f604cd50 unknown :-1 s=CLOSED pgs=677 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-05T19:30:22.758 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.754+0000 7fc1ad445e40 1 -- 172.21.15.167:0/644066034 >> 172.21.15.167:0/644066034 conn(0x55c2f60208b0 msgr2=0x55c2f6040730 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-05T19:30:22.758 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.754+0000 7fc1ad445e40 1 -- 172.21.15.167:0/644066034 shutdown_connections 2023-10-05T19:30:22.758 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.754+0000 7fc1ad445e40 1 -- 172.21.15.167:0/644066034 wait complete. 2023-10-05T19:30:22.759 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.754+0000 7fc1ad445e40 1 -- 172.21.15.167:0/905352282 >> [v2:172.21.15.167:6824/29889,v1:172.21.15.167:6825/29889] conn(0x7fc0f8030c50 msgr2=0x7fc0f8033100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-05T19:30:22.759 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.754+0000 7fc1ad445e40 1 --2- 172.21.15.167:0/905352282 >> [v2:172.21.15.167:6824/29889,v1:172.21.15.167:6825/29889] conn(0x7fc0f8030c50 0x7fc0f8033100 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7fc1a000b230 tx=0x7fc1a00013f0 comp rx=0 tx=0).stop 2023-10-05T19:30:22.759 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.754+0000 7fc1ad445e40 1 -- 172.21.15.167:0/905352282 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55c2f605e490 msgr2=0x55c2f6065610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-05T19:30:22.759 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.754+0000 7fc1ad445e40 1 --2- 172.21.15.167:0/905352282 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55c2f605e490 0x55c2f6065610 secure :-1 s=READY pgs=679 cs=0 l=1 rev1=1 crypto rx=0x7fc1a4000f80 tx=0x7fc1a4014000 comp rx=0 tx=0).stop 2023-10-05T19:30:22.759 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.754+0000 7fc1ad445e40 1 -- 172.21.15.167:0/905352282 shutdown_connections 2023-10-05T19:30:22.759 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.754+0000 7fc1ad445e40 1 --2- 172.21.15.167:0/905352282 >> [v2:172.21.15.167:6824/29889,v1:172.21.15.167:6825/29889] conn(0x7fc0f8030c50 0x7fc0f8033100 unknown :-1 s=CLOSED pgs=233 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-05T19:30:22.759 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.754+0000 7fc1ad445e40 1 --2- 172.21.15.167:0/905352282 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55c2f605e490 0x55c2f6065610 unknown :-1 s=CLOSED pgs=679 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-05T19:30:22.759 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.754+0000 7fc1ad445e40 1 -- 172.21.15.167:0/905352282 >> 172.21.15.167:0/905352282 conn(0x55c2f5f9b070 msgr2=0x55c2f605ae20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-05T19:30:22.760 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.754+0000 7fc1ad445e40 1 -- 172.21.15.167:0/905352282 shutdown_connections 2023-10-05T19:30:22.760 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.754+0000 7fc1ad445e40 1 -- 172.21.15.167:0/905352282 wait complete. 2023-10-05T19:30:22.760 INFO:teuthology.orchestra.run.smithi167.stdout:{ 2023-10-05T19:30:22.760 INFO:teuthology.orchestra.run.smithi167.stdout: "id": "9a9df965-6612-45d4-a45f-6e3c889bacc2", 2023-10-05T19:30:22.760 INFO:teuthology.orchestra.run.smithi167.stdout: "name": "default", 2023-10-05T19:30:22.760 INFO:teuthology.orchestra.run.smithi167.stdout: "domain_root": "default.rgw.meta:root", 2023-10-05T19:30:22.761 INFO:teuthology.orchestra.run.smithi167.stdout: "control_pool": "default.rgw.control", 2023-10-05T19:30:22.761 INFO:teuthology.orchestra.run.smithi167.stdout: "gc_pool": "default.rgw.log:gc", 2023-10-05T19:30:22.761 INFO:teuthology.orchestra.run.smithi167.stdout: "lc_pool": "default.rgw.log:lc", 2023-10-05T19:30:22.761 INFO:teuthology.orchestra.run.smithi167.stdout: "log_pool": "default.rgw.log", 2023-10-05T19:30:22.761 INFO:teuthology.orchestra.run.smithi167.stdout: "intent_log_pool": "default.rgw.log:intent", 2023-10-05T19:30:22.761 INFO:teuthology.orchestra.run.smithi167.stdout: "usage_log_pool": "default.rgw.log:usage", 2023-10-05T19:30:22.761 INFO:teuthology.orchestra.run.smithi167.stdout: "roles_pool": "default.rgw.meta:roles", 2023-10-05T19:30:22.761 INFO:teuthology.orchestra.run.smithi167.stdout: "reshard_pool": "default.rgw.log:reshard", 2023-10-05T19:30:22.761 INFO:teuthology.orchestra.run.smithi167.stdout: "user_keys_pool": "default.rgw.meta:users.keys", 2023-10-05T19:30:22.761 INFO:teuthology.orchestra.run.smithi167.stdout: "user_email_pool": "default.rgw.meta:users.email", 2023-10-05T19:30:22.761 INFO:teuthology.orchestra.run.smithi167.stdout: "user_swift_pool": "default.rgw.meta:users.swift", 2023-10-05T19:30:22.761 INFO:teuthology.orchestra.run.smithi167.stdout: "user_uid_pool": "default.rgw.meta:users.uid", 2023-10-05T19:30:22.762 INFO:teuthology.orchestra.run.smithi167.stdout: "otp_pool": "default.rgw.otp", 2023-10-05T19:30:22.762 INFO:teuthology.orchestra.run.smithi167.stdout: "system_key": { 2023-10-05T19:30:22.762 INFO:teuthology.orchestra.run.smithi167.stdout: "access_key": "", 2023-10-05T19:30:22.762 INFO:teuthology.orchestra.run.smithi167.stdout: "secret_key": "" 2023-10-05T19:30:22.762 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2023-10-05T19:30:22.762 INFO:teuthology.orchestra.run.smithi167.stdout: "placement_pools": [ 2023-10-05T19:30:22.762 INFO:teuthology.orchestra.run.smithi167.stdout: { 2023-10-05T19:30:22.762 INFO:teuthology.orchestra.run.smithi167.stdout: "key": "default-placement", 2023-10-05T19:30:22.762 INFO:teuthology.orchestra.run.smithi167.stdout: "val": { 2023-10-05T19:30:22.762 INFO:teuthology.orchestra.run.smithi167.stdout: "index_pool": "default.rgw.buckets.index", 2023-10-05T19:30:22.762 INFO:teuthology.orchestra.run.smithi167.stdout: "storage_classes": { 2023-10-05T19:30:22.762 INFO:teuthology.orchestra.run.smithi167.stdout: "STANDARD": { 2023-10-05T19:30:22.763 INFO:teuthology.orchestra.run.smithi167.stdout: "data_pool": "default.rgw.buckets.data" 2023-10-05T19:30:22.763 INFO:teuthology.orchestra.run.smithi167.stdout: } 2023-10-05T19:30:22.763 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2023-10-05T19:30:22.763 INFO:teuthology.orchestra.run.smithi167.stdout: "data_extra_pool": "default.rgw.buckets.non-ec", 2023-10-05T19:30:22.763 INFO:teuthology.orchestra.run.smithi167.stdout: "index_type": 0, 2023-10-05T19:30:22.763 INFO:teuthology.orchestra.run.smithi167.stdout: "inline_data": "true" 2023-10-05T19:30:22.763 INFO:teuthology.orchestra.run.smithi167.stdout: } 2023-10-05T19:30:22.763 INFO:teuthology.orchestra.run.smithi167.stdout: } 2023-10-05T19:30:22.763 INFO:teuthology.orchestra.run.smithi167.stdout: ], 2023-10-05T19:30:22.763 INFO:teuthology.orchestra.run.smithi167.stdout: "realm_id": "", 2023-10-05T19:30:22.763 INFO:teuthology.orchestra.run.smithi167.stdout: "notif_pool": "default.rgw.log:notif" 2023-10-05T19:30:22.763 INFO:teuthology.orchestra.run.smithi167.stdout:} 2023-10-05T19:30:22.764 DEBUG:tasks.util.rgw: json result: {'id': '9a9df965-6612-45d4-a45f-6e3c889bacc2', 'name': 'default', 'domain_root': 'default.rgw.meta:root', 'control_pool': 'default.rgw.control', 'gc_pool': 'default.rgw.log:gc', 'lc_pool': 'default.rgw.log:lc', 'log_pool': 'default.rgw.log', 'intent_log_pool': 'default.rgw.log:intent', 'usage_log_pool': 'default.rgw.log:usage', 'roles_pool': 'default.rgw.meta:roles', 'reshard_pool': 'default.rgw.log:reshard', 'user_keys_pool': 'default.rgw.meta:users.keys', 'user_email_pool': 'default.rgw.meta:users.email', 'user_swift_pool': 'default.rgw.meta:users.swift', 'user_uid_pool': 'default.rgw.meta:users.uid', 'otp_pool': 'default.rgw.otp', 'system_key': {'access_key': '', 'secret_key': ''}, 'placement_pools': [{'key': 'default-placement', 'val': {'index_pool': 'default.rgw.buckets.index', 'storage_classes': {'STANDARD': {'data_pool': 'default.rgw.buckets.data'}}, 'data_extra_pool': 'default.rgw.buckets.non-ec', 'index_type': 0, 'inline_data': 'true'}}], 'realm_id': '', 'notif_pool': 'default.rgw.log:notif'} 2023-10-05T19:30:22.765 INFO:tasks.util.rgw:rgwadmin: client.0 : ['zone', 'set'] 2023-10-05T19:30:22.765 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'ceph', 'zone', 'set'] 2023-10-05T19:30:22.765 DEBUG:teuthology.orchestra.run.smithi167:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster ceph zone set 2023-10-05T19:30:22.843 INFO:teuthology.orchestra.run.smithi167.stderr:ignoring --setuser ceph since I am not root 2023-10-05T19:30:22.843 INFO:teuthology.orchestra.run.smithi167.stderr:ignoring --setgroup ceph since I am not root 2023-10-05T19:30:22.844 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.842+0000 7f6c637e1e40 1 Processor -- start 2023-10-05T19:30:22.844 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.842+0000 7f6c637e1e40 1 -- start start 2023-10-05T19:30:22.845 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.842+0000 7f6c637e1e40 1 --2- >> v2:172.21.15.118:3300/0 conn(0x561a03cd45c0 0x561a03cd69a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-05T19:30:22.845 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.842+0000 7f6c637e1e40 1 -- --> v1:172.21.15.118:6789/0 -- auth(proto 0 26 bytes epoch 0) v1 -- 0x561a03c1ac00 con 0x561a03ccffe0 2023-10-05T19:30:22.845 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.842+0000 7f6c637e1e40 1 -- --> v2:172.21.15.118:3300/0 -- mon_getmap magic: 0 v1 -- 0x561a03bd7ee0 con 0x561a03cd45c0 2023-10-05T19:30:22.845 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.842+0000 7f6c60b15700 1 --2- >> v2:172.21.15.118:3300/0 conn(0x561a03cd45c0 0x561a03cd69a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-05T19:30:22.845 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.842+0000 7f6c60b15700 1 --2- >> v2:172.21.15.118:3300/0 conn(0x561a03cd45c0 0x561a03cd69a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.118:3300/0 says I am v2:172.21.15.167:33136/0 (socket says 172.21.15.167:33136) 2023-10-05T19:30:22.845 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.842+0000 7f6c61316700 1 --1- >> v1:172.21.15.118:6789/0 conn(0x561a03ccffe0 0x561a03cd03b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.118:6789/0 says I am v1:172.21.15.167:50824/0 (socket says 172.21.15.167:50824) 2023-10-05T19:30:22.846 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.842+0000 7f6c60b15700 1 -- 172.21.15.167:0/668408623 learned_addr learned my addr 172.21.15.167:0/668408623 (peer_addr_for_me v2:172.21.15.167:0/0) 2023-10-05T19:30:22.847 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.842+0000 7f6c60b15700 1 -- 172.21.15.167:0/668408623 >> v1:172.21.15.118:6789/0 conn(0x561a03ccffe0 legacy=0x561a03cd03b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-05T19:30:22.847 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.842+0000 7f6c60b15700 1 -- 172.21.15.167:0/668408623 --> v2:172.21.15.118:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x561a03cd0bf0 con 0x561a03cd45c0 2023-10-05T19:30:22.847 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.842+0000 7f6c60b15700 1 --2- 172.21.15.167:0/668408623 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x561a03cd45c0 0x561a03cd69a0 secure :-1 s=READY pgs=681 cs=0 l=1 rev1=1 crypto rx=0x7f6c4c00a5c0 tx=0x7f6c4c005e00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=55023cce37c49cfd server_cookie=0 in_seq=0 out_seq=0 2023-10-05T19:30:22.847 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.842+0000 7f6c5bfff700 1 -- 172.21.15.167:0/668408623 <== mon.0 v2:172.21.15.118:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f6c4c00e510 con 0x561a03cd45c0 2023-10-05T19:30:22.847 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.842+0000 7f6c5bfff700 1 -- 172.21.15.167:0/668408623 <== mon.0 v2:172.21.15.118:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6c4c00e670 con 0x561a03cd45c0 2023-10-05T19:30:22.847 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.842+0000 7f6c5bfff700 1 -- 172.21.15.167:0/668408623 <== mon.0 v2:172.21.15.118:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f6c4c00e920 con 0x561a03cd45c0 2023-10-05T19:30:22.847 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.842+0000 7f6c637e1e40 1 -- 172.21.15.167:0/668408623 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x561a03cd45c0 msgr2=0x561a03cd69a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-05T19:30:22.847 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.842+0000 7f6c637e1e40 1 --2- 172.21.15.167:0/668408623 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x561a03cd45c0 0x561a03cd69a0 secure :-1 s=READY pgs=681 cs=0 l=1 rev1=1 crypto rx=0x7f6c4c00a5c0 tx=0x7f6c4c005e00 comp rx=0 tx=0).stop 2023-10-05T19:30:22.848 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.842+0000 7f6c637e1e40 1 -- 172.21.15.167:0/668408623 shutdown_connections 2023-10-05T19:30:22.848 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.842+0000 7f6c637e1e40 1 --2- 172.21.15.167:0/668408623 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x561a03cd45c0 0x561a03cd69a0 unknown :-1 s=CLOSED pgs=681 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-05T19:30:22.848 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.842+0000 7f6c637e1e40 1 -- 172.21.15.167:0/668408623 >> 172.21.15.167:0/668408623 conn(0x561a03c43830 msgr2=0x561a03c43c30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-05T19:30:22.848 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.842+0000 7f6c637e1e40 1 -- 172.21.15.167:0/668408623 shutdown_connections 2023-10-05T19:30:22.848 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.842+0000 7f6c637e1e40 1 -- 172.21.15.167:0/668408623 wait complete. 2023-10-05T19:30:22.849 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.842+0000 7f6c637e1e40 1 Processor -- start 2023-10-05T19:30:22.849 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.846+0000 7f6c637e1e40 1 -- start start 2023-10-05T19:30:22.849 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.846+0000 7f6c637e1e40 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x561a03ccffe0 0x561a03cef160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-05T19:30:22.849 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.846+0000 7f6c637e1e40 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x561a03b9be30 con 0x561a03ccffe0 2023-10-05T19:30:22.850 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.846+0000 7f6c61316700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x561a03ccffe0 0x561a03cef160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-05T19:30:22.850 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.846+0000 7f6c61316700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x561a03ccffe0 0x561a03cef160 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.118:3300/0 says I am v2:172.21.15.167:33144/0 (socket says 172.21.15.167:33144) 2023-10-05T19:30:22.850 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.846+0000 7f6c61316700 1 -- 172.21.15.167:0/2495004252 learned_addr learned my addr 172.21.15.167:0/2495004252 (peer_addr_for_me v2:172.21.15.167:0/0) 2023-10-05T19:30:22.850 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.846+0000 7f6c61316700 1 -- 172.21.15.167:0/2495004252 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6c4c00a180 con 0x561a03ccffe0 2023-10-05T19:30:22.850 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.846+0000 7f6c61316700 1 --2- 172.21.15.167:0/2495004252 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x561a03ccffe0 0x561a03cef160 secure :-1 s=READY pgs=682 cs=0 l=1 rev1=1 crypto rx=0x7f6c54001cf0 tx=0x7f6c540018a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3c385cef3fda2ccd server_cookie=0 in_seq=0 out_seq=0 2023-10-05T19:30:22.850 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.846+0000 7f6c5affd700 1 -- 172.21.15.167:0/2495004252 <== mon.0 v2:172.21.15.118:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f6c54014070 con 0x561a03ccffe0 2023-10-05T19:30:22.850 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.846+0000 7f6c5affd700 1 -- 172.21.15.167:0/2495004252 <== mon.0 v2:172.21.15.118:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6c5400be10 con 0x561a03ccffe0 2023-10-05T19:30:22.850 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.846+0000 7f6c5affd700 1 -- 172.21.15.167:0/2495004252 <== mon.0 v2:172.21.15.118:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f6c5400f590 con 0x561a03ccffe0 2023-10-05T19:30:22.851 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.846+0000 7f6c637e1e40 1 -- 172.21.15.167:0/2495004252 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x561a03ccffe0 msgr2=0x561a03cef160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-05T19:30:22.852 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.846+0000 7f6c637e1e40 1 --2- 172.21.15.167:0/2495004252 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x561a03ccffe0 0x561a03cef160 secure :-1 s=READY pgs=682 cs=0 l=1 rev1=1 crypto rx=0x7f6c54001cf0 tx=0x7f6c540018a0 comp rx=0 tx=0).stop 2023-10-05T19:30:22.852 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.846+0000 7f6c637e1e40 1 -- 172.21.15.167:0/2495004252 shutdown_connections 2023-10-05T19:30:22.852 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.846+0000 7f6c637e1e40 1 --2- 172.21.15.167:0/2495004252 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x561a03ccffe0 0x561a03cef160 unknown :-1 s=CLOSED pgs=682 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-05T19:30:22.852 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.846+0000 7f6c637e1e40 1 -- 172.21.15.167:0/2495004252 >> 172.21.15.167:0/2495004252 conn(0x561a03c43830 msgr2=0x561a03cead20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-05T19:30:22.852 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.846+0000 7f6c637e1e40 1 -- 172.21.15.167:0/2495004252 shutdown_connections 2023-10-05T19:30:22.852 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.846+0000 7f6c637e1e40 1 -- 172.21.15.167:0/2495004252 wait complete. 2023-10-05T19:30:22.852 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.846+0000 7f6c637e1e40 1 Processor -- start 2023-10-05T19:30:22.852 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.846+0000 7f6c637e1e40 1 -- start start 2023-10-05T19:30:22.852 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.846+0000 7f6c637e1e40 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x561a03ccffe0 0x561a03cf5f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-05T19:30:22.852 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.846+0000 7f6c637e1e40 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x561a03b6eed0 con 0x561a03ccffe0 2023-10-05T19:30:22.852 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.846+0000 7f6c61316700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x561a03ccffe0 0x561a03cf5f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-05T19:30:22.853 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.846+0000 7f6c61316700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x561a03ccffe0 0x561a03cf5f30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.118:3300/0 says I am v2:172.21.15.167:33158/0 (socket says 172.21.15.167:33158) 2023-10-05T19:30:22.853 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.846+0000 7f6c61316700 1 -- 172.21.15.167:0/2430872828 learned_addr learned my addr 172.21.15.167:0/2430872828 (peer_addr_for_me v2:172.21.15.167:0/0) 2023-10-05T19:30:22.853 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.850+0000 7f6c61316700 1 -- 172.21.15.167:0/2430872828 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x561a03c1ac00 con 0x561a03ccffe0 2023-10-05T19:30:22.854 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.850+0000 7f6c61316700 1 --2- 172.21.15.167:0/2430872828 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x561a03ccffe0 0x561a03cf5f30 secure :-1 s=READY pgs=683 cs=0 l=1 rev1=1 crypto rx=0x7f6c540018e0 tx=0x7f6c54016000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-05T19:30:22.854 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.850+0000 7f6c597fa700 1 -- 172.21.15.167:0/2430872828 <== mon.0 v2:172.21.15.118:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f6c5401c670 con 0x561a03ccffe0 2023-10-05T19:30:22.854 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.850+0000 7f6c597fa700 1 -- 172.21.15.167:0/2430872828 <== mon.0 v2:172.21.15.118:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6c5401c7f0 con 0x561a03ccffe0 2023-10-05T19:30:22.854 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.850+0000 7f6c637e1e40 1 -- 172.21.15.167:0/2430872828 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x561a03cf6470 con 0x561a03ccffe0 2023-10-05T19:30:22.854 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.850+0000 7f6c597fa700 1 -- 172.21.15.167:0/2430872828 <== mon.0 v2:172.21.15.118:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f6c5401cae0 con 0x561a03ccffe0 2023-10-05T19:30:22.854 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.850+0000 7f6c637e1e40 1 -- 172.21.15.167:0/2430872828 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x561a03cf68b0 con 0x561a03ccffe0 2023-10-05T19:30:22.854 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.850+0000 7f6c597fa700 1 -- 172.21.15.167:0/2430872828 <== mon.0 v2:172.21.15.118:3300/0 4 ==== mgrmap(e 5) v1 ==== 40329+0+0 (secure 0 0 0) 0x7f6c54023020 con 0x561a03ccffe0 2023-10-05T19:30:22.855 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.850+0000 7f6c597fa700 1 --2- 172.21.15.167:0/2430872828 >> [v2:172.21.15.167:6824/29889,v1:172.21.15.167:6825/29889] conn(0x7f6c48030c00 0x7f6c480330b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-05T19:30:22.855 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.850+0000 7f6c597fa700 1 -- 172.21.15.167:0/2430872828 <== mon.0 v2:172.21.15.118:3300/0 5 ==== osd_map(32..32 src has 1..32) v4 ==== 6260+0+0 (secure 0 0 0) 0x7f6c54008030 con 0x561a03ccffe0 2023-10-05T19:30:22.855 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.850+0000 7f6c60b15700 1 --2- 172.21.15.167:0/2430872828 >> [v2:172.21.15.167:6824/29889,v1:172.21.15.167:6825/29889] conn(0x7f6c48030c00 0x7f6c480330b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-05T19:30:22.856 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.850+0000 7f6c637e1e40 20 check_secure_mon_conn(): auth registy supported: methods=[2] modes=[2,1] 2023-10-05T19:30:22.856 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.850+0000 7f6c637e1e40 20 check_secure_mon_conn(): mode 1 is insecure 2023-10-05T19:30:22.856 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.850+0000 7f6c637e1e40 1 Processor -- start 2023-10-05T19:30:22.856 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.850+0000 7f6c637e1e40 1 -- start start 2023-10-05T19:30:22.856 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.850+0000 7f6c60b15700 1 --2- 172.21.15.167:0/2430872828 >> [v2:172.21.15.167:6824/29889,v1:172.21.15.167:6825/29889] conn(0x7f6c48030c00 0x7f6c480330b0 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7f6c4c000f80 tx=0x7f6c4c013000 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-05T19:30:22.857 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.854+0000 7f6c637e1e40 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x561a03d07590 0x561a03d07960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-05T19:30:22.857 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.854+0000 7f6c637e1e40 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x561a03d07ea0 con 0x561a03d07590 2023-10-05T19:30:22.857 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.854+0000 7f6c61316700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x561a03d07590 0x561a03d07960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-05T19:30:22.857 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.854+0000 7f6c61316700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x561a03d07590 0x561a03d07960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.118:3300/0 says I am v2:172.21.15.167:33168/0 (socket says 172.21.15.167:33168) 2023-10-05T19:30:22.857 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.854+0000 7f6c61316700 1 -- 172.21.15.167:0/1798768987 learned_addr learned my addr 172.21.15.167:0/1798768987 (peer_addr_for_me v2:172.21.15.167:0/0) 2023-10-05T19:30:22.857 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.854+0000 7f6c61316700 1 -- 172.21.15.167:0/1798768987 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x561a03d07fe0 con 0x561a03d07590 2023-10-05T19:30:22.857 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.854+0000 7f6c61316700 1 --2- 172.21.15.167:0/1798768987 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x561a03d07590 0x561a03d07960 secure :-1 s=READY pgs=684 cs=0 l=1 rev1=1 crypto rx=0x7f6c5401dfd0 tx=0x7f6c54043620 comp rx=0 tx=0).ready entity=mon.0 client_cookie=14e92fd9dee4ba1e server_cookie=0 in_seq=0 out_seq=0 2023-10-05T19:30:22.858 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.854+0000 7f6c31feb700 1 -- 172.21.15.167:0/1798768987 <== mon.0 v2:172.21.15.118:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f6c540437d0 con 0x561a03d07590 2023-10-05T19:30:22.858 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.854+0000 7f6c31feb700 1 -- 172.21.15.167:0/1798768987 <== mon.0 v2:172.21.15.118:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6c54025c40 con 0x561a03d07590 2023-10-05T19:30:22.858 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.854+0000 7f6c31feb700 1 -- 172.21.15.167:0/1798768987 <== mon.0 v2:172.21.15.118:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f6c54045230 con 0x561a03d07590 2023-10-05T19:30:22.858 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.854+0000 7f6c637e1e40 1 -- 172.21.15.167:0/1798768987 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x561a03d07590 msgr2=0x561a03d07960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-05T19:30:22.858 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.854+0000 7f6c637e1e40 1 --2- 172.21.15.167:0/1798768987 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x561a03d07590 0x561a03d07960 secure :-1 s=READY pgs=684 cs=0 l=1 rev1=1 crypto rx=0x7f6c5401dfd0 tx=0x7f6c54043620 comp rx=0 tx=0).stop 2023-10-05T19:30:22.860 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.854+0000 7f6c637e1e40 1 -- 172.21.15.167:0/1798768987 shutdown_connections 2023-10-05T19:30:22.860 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.854+0000 7f6c637e1e40 1 --2- 172.21.15.167:0/1798768987 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x561a03d07590 0x561a03d07960 unknown :-1 s=CLOSED pgs=684 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-05T19:30:22.860 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.854+0000 7f6c637e1e40 1 -- 172.21.15.167:0/1798768987 >> 172.21.15.167:0/1798768987 conn(0x561a03cd45c0 msgr2=0x561a03d042e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-05T19:30:22.860 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.854+0000 7f6c637e1e40 1 -- 172.21.15.167:0/1798768987 shutdown_connections 2023-10-05T19:30:22.860 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.854+0000 7f6c637e1e40 1 -- 172.21.15.167:0/1798768987 wait complete. 2023-10-05T19:30:22.860 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.854+0000 7f6c637e1e40 1 Processor -- start 2023-10-05T19:30:22.860 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.854+0000 7f6c637e1e40 1 -- start start 2023-10-05T19:30:22.860 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.854+0000 7f6c637e1e40 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x561a03d07590 0x561a03d0e850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-05T19:30:22.860 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.854+0000 7f6c637e1e40 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x561a03d0f850 con 0x561a03d07590 2023-10-05T19:30:22.860 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.858+0000 7f6c61316700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x561a03d07590 0x561a03d0e850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-05T19:30:22.860 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.858+0000 7f6c61316700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x561a03d07590 0x561a03d0e850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.118:3300/0 says I am v2:172.21.15.167:33176/0 (socket says 172.21.15.167:33176) 2023-10-05T19:30:22.861 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.858+0000 7f6c61316700 1 -- 172.21.15.167:0/3050533416 learned_addr learned my addr 172.21.15.167:0/3050533416 (peer_addr_for_me v2:172.21.15.167:0/0) 2023-10-05T19:30:22.861 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.858+0000 7f6c61316700 1 -- 172.21.15.167:0/3050533416 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x561a03cf68b0 con 0x561a03d07590 2023-10-05T19:30:22.861 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.858+0000 7f6c61316700 1 --2- 172.21.15.167:0/3050533416 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x561a03d07590 0x561a03d0e850 secure :-1 s=READY pgs=685 cs=0 l=1 rev1=1 crypto rx=0x7f6c5401cfd0 tx=0x7f6c54036000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-05T19:30:22.862 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.858+0000 7f6babfff700 1 -- 172.21.15.167:0/3050533416 <== mon.0 v2:172.21.15.118:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f6c54043ae0 con 0x561a03d07590 2023-10-05T19:30:22.862 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.858+0000 7f6c637e1e40 1 -- 172.21.15.167:0/3050533416 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x561a03d0ee50 con 0x561a03d07590 2023-10-05T19:30:22.862 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.858+0000 7f6babfff700 1 -- 172.21.15.167:0/3050533416 <== mon.0 v2:172.21.15.118:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6c54043c40 con 0x561a03d07590 2023-10-05T19:30:22.862 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.858+0000 7f6babfff700 1 -- 172.21.15.167:0/3050533416 <== mon.0 v2:172.21.15.118:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f6c540253d0 con 0x561a03d07590 2023-10-05T19:30:22.862 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.858+0000 7f6c637e1e40 1 -- 172.21.15.167:0/3050533416 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x561a03d0f2b0 con 0x561a03d07590 2023-10-05T19:30:22.863 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.858+0000 7f6babfff700 1 -- 172.21.15.167:0/3050533416 <== mon.0 v2:172.21.15.118:3300/0 4 ==== mgrmap(e 5) v1 ==== 40329+0+0 (secure 0 0 0) 0x7f6c54025530 con 0x561a03d07590 2023-10-05T19:30:22.863 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.858+0000 7f6ba9ffb700 20 reqs_thread_entry: start 2023-10-05T19:30:22.864 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.858+0000 7f6c637e1e40 20 try_to_resolve_local_entities(): before: realm_id= realm_name= zonegroup_id= zonegroup_name= zone_id= zone_name= 2023-10-05T19:30:22.864 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.858+0000 7f6c637e1e40 20 rados->read ofs=0 len=0 2023-10-05T19:30:22.864 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.858+0000 7f6c637e1e40 1 --2- 172.21.15.167:0/2430872828 >> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] conn(0x561a03d2c570 0x561a03d2c9f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-05T19:30:22.864 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.858+0000 7f6c637e1e40 1 -- 172.21.15.167:0/2430872828 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:1 5.12 5:49953fa1:::default.realm:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32) v8 -- 0x561a03d2cf30 con 0x561a03d2c570 2023-10-05T19:30:22.864 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.862+0000 7f6babfff700 1 --2- 172.21.15.167:0/3050533416 >> [v2:172.21.15.167:6824/29889,v1:172.21.15.167:6825/29889] conn(0x7f6bb0030c00 0x7f6bb00330b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-05T19:30:22.864 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.862+0000 7f6babfff700 1 -- 172.21.15.167:0/3050533416 <== mon.0 v2:172.21.15.118:3300/0 5 ==== osd_map(32..32 src has 1..32) v4 ==== 6260+0+0 (secure 0 0 0) 0x7f6c54014070 con 0x561a03d07590 2023-10-05T19:30:22.864 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.862+0000 7f6c60b15700 1 --2- 172.21.15.167:0/2430872828 >> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] conn(0x561a03d2c570 0x561a03d2c9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-05T19:30:22.865 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.862+0000 7f6c61b17700 1 --2- 172.21.15.167:0/3050533416 >> [v2:172.21.15.167:6824/29889,v1:172.21.15.167:6825/29889] conn(0x7f6bb0030c00 0x7f6bb00330b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-05T19:30:22.865 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.862+0000 7f6c61b17700 1 --2- 172.21.15.167:0/3050533416 >> [v2:172.21.15.167:6824/29889,v1:172.21.15.167:6825/29889] conn(0x7f6bb0030c00 0x7f6bb00330b0 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7f6c5c00a370 tx=0x7f6c5c005500 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-05T19:30:22.865 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.862+0000 7f6c60b15700 1 --2- 172.21.15.167:0/2430872828 >> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] conn(0x561a03d2c570 0x561a03d2c9f0 crc :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-05T19:30:22.865 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.862+0000 7f6c60b15700 1 -- 172.21.15.167:0/2430872828 <== osd.1 v2:172.21.15.167:6800/30035 1 ==== osd_op_reply(1 default.realm [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 ==== 157+0+0 (crc 0 0 0) 0x7f6c4c017050 con 0x561a03d2c570 2023-10-05T19:30:22.866 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.862+0000 7f6c637e1e40 20 rados_obj.operate() r=-2 bl.length=0 2023-10-05T19:30:22.866 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.862+0000 7f6c637e1e40 20 rados->read ofs=0 len=0 2023-10-05T19:30:22.866 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.862+0000 7f6c637e1e40 1 -- 172.21.15.167:0/2430872828 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:2 5.f 5:f43b6ece:::zone_names.default:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32) v8 -- 0x561a03d3fab0 con 0x561a03d2c570 2023-10-05T19:30:22.867 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.862+0000 7f6c60b15700 1 -- 172.21.15.167:0/2430872828 <== osd.1 v2:172.21.15.167:6800/30035 2 ==== osd_op_reply(2 zone_names.default [read 0~46 out=46b] v0'0 uv1 ondisk = 0) v8 ==== 162+0+46 (crc 0 0 0) 0x7f6c4c017050 con 0x561a03d2c570 2023-10-05T19:30:22.867 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.862+0000 7f6c637e1e40 20 rados_obj.operate() r=0 bl.length=46 2023-10-05T19:30:22.867 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.862+0000 7f6c637e1e40 20 rados->read ofs=0 len=0 2023-10-05T19:30:22.867 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.862+0000 7f6c637e1e40 1 -- 172.21.15.167:0/2430872828 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:3 5.1d 5:b8374eb4:::zone_info.9a9df965-6612-45d4-a45f-6e3c889bacc2:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32) v8 -- 0x561a03d2ed50 con 0x561a03d2c570 2023-10-05T19:30:22.867 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.862+0000 7f6c60b15700 1 -- 172.21.15.167:0/2430872828 <== osd.1 v2:172.21.15.167:6800/30035 3 ==== osd_op_reply(3 zone_info.9a9df965-6612-45d4-a45f-6e3c889bacc2 [read 0~874 out=874b] v0'0 uv2 ondisk = 0) v8 ==== 190+0+874 (crc 0 0 0) 0x7f6c4c017050 con 0x561a03d2c570 2023-10-05T19:30:22.867 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.862+0000 7f6c637e1e40 20 rados_obj.operate() r=0 bl.length=874 2023-10-05T19:30:22.867 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.862+0000 7f6c637e1e40 1 --2- 172.21.15.167:0/2430872828 >> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] conn(0x561a03d41380 0x561a03d43760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-05T19:30:22.868 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.862+0000 7f6c637e1e40 1 -- 172.21.15.167:0/2430872828 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:4 5.0 5:00000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x561a03d43ca0 con 0x561a03d41380 2023-10-05T19:30:22.868 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.862+0000 7f6c61316700 1 --2- 172.21.15.167:0/2430872828 >> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] conn(0x561a03d41380 0x561a03d43760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-05T19:30:22.868 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.866+0000 7f6c61316700 1 --2- 172.21.15.167:0/2430872828 >> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] conn(0x561a03d41380 0x561a03d43760 crc :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-05T19:30:22.869 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.866+0000 7f6c61316700 1 -- 172.21.15.167:0/2430872828 <== osd.3 v2:172.21.15.167:6808/30036 1 ==== osd_op_reply(4 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f6c54016070 con 0x561a03d41380 2023-10-05T19:30:22.869 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.866+0000 7f6c61316700 1 --2- 172.21.15.167:0/2430872828 >> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] conn(0x7f6c5401a070 0x7f6c54044050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-05T19:30:22.869 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.866+0000 7f6c61316700 1 -- 172.21.15.167:0/2430872828 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:5 5.10 5:08000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x561a03d43ca0 con 0x7f6c5401a070 2023-10-05T19:30:22.870 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.866+0000 7f6c60b15700 1 --2- 172.21.15.167:0/2430872828 >> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] conn(0x7f6c5401a070 0x7f6c54044050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-05T19:30:22.870 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.866+0000 7f6c60b15700 1 --2- 172.21.15.167:0/2430872828 >> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] conn(0x7f6c5401a070 0x7f6c54044050 crc :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-05T19:30:22.871 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.866+0000 7f6c60b15700 1 -- 172.21.15.167:0/2430872828 <== osd.2 v2:172.21.15.167:6816/30037 1 ==== osd_op_reply(5 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f6c4c017050 con 0x7f6c5401a070 2023-10-05T19:30:22.871 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.866+0000 7f6c60b15700 1 --2- 172.21.15.167:0/2430872828 >> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] conn(0x7f6c4c010070 0x7f6c4c028940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-05T19:30:22.871 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.866+0000 7f6c60b15700 1 -- 172.21.15.167:0/2430872828 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:6 5.8 5:10000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x561a03d43ca0 con 0x7f6c4c010070 2023-10-05T19:30:22.872 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.870+0000 7f6c61b17700 1 --2- 172.21.15.167:0/2430872828 >> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] conn(0x7f6c4c010070 0x7f6c4c028940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-05T19:30:22.872 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.870+0000 7f6c61b17700 1 --2- 172.21.15.167:0/2430872828 >> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] conn(0x7f6c4c010070 0x7f6c4c028940 crc :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-05T19:30:22.873 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.870+0000 7f6c61b17700 1 -- 172.21.15.167:0/2430872828 <== osd.0 v2:172.21.15.118:6800/27690 1 ==== osd_op_reply(6 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f6c5c011040 con 0x7f6c4c010070 2023-10-05T19:30:22.873 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.870+0000 7f6c61b17700 1 -- 172.21.15.167:0/2430872828 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:7 5.18 5:18000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x561a03d43ca0 con 0x561a03d2c570 2023-10-05T19:30:22.874 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.870+0000 7f6c60b15700 1 -- 172.21.15.167:0/2430872828 <== osd.1 v2:172.21.15.167:6800/30035 4 ==== osd_op_reply(7 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f6c4c034040 con 0x561a03d2c570 2023-10-05T19:30:22.874 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.870+0000 7f6c60b15700 1 -- 172.21.15.167:0/2430872828 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:8 5.4 5:20000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x561a03d43ca0 con 0x561a03d2c570 2023-10-05T19:30:22.875 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.870+0000 7f6c60b15700 1 -- 172.21.15.167:0/2430872828 <== osd.1 v2:172.21.15.167:6800/30035 5 ==== osd_op_reply(8 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f6c4c034040 con 0x561a03d2c570 2023-10-05T19:30:22.875 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.870+0000 7f6c60b15700 1 -- 172.21.15.167:0/2430872828 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:9 5.14 5:28000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x561a03d43ca0 con 0x7f6c4c010070 2023-10-05T19:30:22.875 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.870+0000 7f6c61b17700 1 -- 172.21.15.167:0/2430872828 <== osd.0 v2:172.21.15.118:6800/27690 2 ==== osd_op_reply(9 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f6c5c011040 con 0x7f6c4c010070 2023-10-05T19:30:22.875 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.870+0000 7f6c61b17700 1 -- 172.21.15.167:0/2430872828 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:10 5.c 5:30000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x561a03d43ca0 con 0x7f6c4c010070 2023-10-05T19:30:22.876 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.874+0000 7f6c61b17700 1 -- 172.21.15.167:0/2430872828 <== osd.0 v2:172.21.15.118:6800/27690 3 ==== osd_op_reply(10 [pgnls start_epoch 32 out=112b] v23'1 uv1 ondisk = 1) v8 ==== 144+0+112 (crc 0 0 0) 0x7f6c5c011040 con 0x7f6c4c010070 2023-10-05T19:30:22.876 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.874+0000 7f6c61b17700 1 -- 172.21.15.167:0/2430872828 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:11 5.1c 5:38000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x561a03d43ca0 con 0x7f6c5401a070 2023-10-05T19:30:22.877 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.874+0000 7f6c60b15700 1 -- 172.21.15.167:0/2430872828 <== osd.2 v2:172.21.15.167:6816/30037 2 ==== osd_op_reply(11 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f6c4c034040 con 0x7f6c5401a070 2023-10-05T19:30:22.878 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.874+0000 7f6c60b15700 1 -- 172.21.15.167:0/2430872828 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:12 5.2 5:40000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x561a03d43ca0 con 0x561a03d41380 2023-10-05T19:30:22.878 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.874+0000 7f6c61316700 1 -- 172.21.15.167:0/2430872828 <== osd.3 v2:172.21.15.167:6808/30036 2 ==== osd_op_reply(12 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f6c54056040 con 0x561a03d41380 2023-10-05T19:30:22.878 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.874+0000 7f6c61316700 1 -- 172.21.15.167:0/2430872828 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:13 5.12 5:48000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x561a03d43ca0 con 0x561a03d2c570 2023-10-05T19:30:22.878 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.874+0000 7f6c60b15700 1 -- 172.21.15.167:0/2430872828 <== osd.1 v2:172.21.15.167:6800/30035 6 ==== osd_op_reply(13 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f6c4c034040 con 0x561a03d2c570 2023-10-05T19:30:22.878 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.874+0000 7f6c60b15700 1 -- 172.21.15.167:0/2430872828 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:14 5.a 5:50000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x561a03d43ca0 con 0x7f6c4c010070 2023-10-05T19:30:22.879 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.874+0000 7f6c61b17700 1 -- 172.21.15.167:0/2430872828 <== osd.0 v2:172.21.15.118:6800/27690 4 ==== osd_op_reply(14 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f6c5c011040 con 0x7f6c4c010070 2023-10-05T19:30:22.879 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.874+0000 7f6c61b17700 1 -- 172.21.15.167:0/2430872828 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:15 5.1a 5:58000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x561a03d43ca0 con 0x561a03d2c570 2023-10-05T19:30:22.880 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.878+0000 7f6c60b15700 1 -- 172.21.15.167:0/2430872828 <== osd.1 v2:172.21.15.167:6800/30035 7 ==== osd_op_reply(15 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f6c4c034040 con 0x561a03d2c570 2023-10-05T19:30:22.880 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.878+0000 7f6c60b15700 1 -- 172.21.15.167:0/2430872828 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:16 5.6 5:60000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x561a03d43ca0 con 0x7f6c5401a070 2023-10-05T19:30:22.880 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.878+0000 7f6c60b15700 1 -- 172.21.15.167:0/2430872828 <== osd.2 v2:172.21.15.167:6816/30037 3 ==== osd_op_reply(16 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f6c4c034040 con 0x7f6c5401a070 2023-10-05T19:30:22.880 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.878+0000 7f6c60b15700 1 -- 172.21.15.167:0/2430872828 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:17 5.16 5:68000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x561a03d43ca0 con 0x561a03d41380 2023-10-05T19:30:22.881 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.878+0000 7f6c61316700 1 -- 172.21.15.167:0/2430872828 <== osd.3 v2:172.21.15.167:6808/30036 3 ==== osd_op_reply(17 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f6c54056040 con 0x561a03d41380 2023-10-05T19:30:22.881 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.878+0000 7f6c61316700 1 -- 172.21.15.167:0/2430872828 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:18 5.e 5:70000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x561a03d43ca0 con 0x561a03d41380 2023-10-05T19:30:22.881 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.878+0000 7f6c61316700 1 -- 172.21.15.167:0/2430872828 <== osd.3 v2:172.21.15.167:6808/30036 4 ==== osd_op_reply(18 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f6c54056040 con 0x561a03d41380 2023-10-05T19:30:22.881 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.878+0000 7f6c61316700 1 -- 172.21.15.167:0/2430872828 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:19 5.1e 5:78000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x561a03d43ca0 con 0x7f6c4c010070 2023-10-05T19:30:22.882 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.878+0000 7f6c61b17700 1 -- 172.21.15.167:0/2430872828 <== osd.0 v2:172.21.15.118:6800/27690 5 ==== osd_op_reply(19 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f6c5c011040 con 0x7f6c4c010070 2023-10-05T19:30:22.882 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.878+0000 7f6c61b17700 1 -- 172.21.15.167:0/2430872828 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:20 5.1 5:80000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x561a03d43ca0 con 0x561a03d41380 2023-10-05T19:30:22.883 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.878+0000 7f6c61316700 1 -- 172.21.15.167:0/2430872828 <== osd.3 v2:172.21.15.167:6808/30036 5 ==== osd_op_reply(20 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f6c54056040 con 0x561a03d41380 2023-10-05T19:30:22.883 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.878+0000 7f6c61316700 1 -- 172.21.15.167:0/2430872828 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:21 5.11 5:88000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x561a03d43ca0 con 0x561a03d41380 2023-10-05T19:30:22.883 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.882+0000 7f6c61316700 1 -- 172.21.15.167:0/2430872828 <== osd.3 v2:172.21.15.167:6808/30036 6 ==== osd_op_reply(21 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f6c54056040 con 0x561a03d41380 2023-10-05T19:30:22.884 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.882+0000 7f6c61316700 1 -- 172.21.15.167:0/2430872828 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:22 5.9 5:90000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x561a03d43ca0 con 0x561a03d2c570 2023-10-05T19:30:22.884 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.882+0000 7f6c60b15700 1 -- 172.21.15.167:0/2430872828 <== osd.1 v2:172.21.15.167:6800/30035 8 ==== osd_op_reply(22 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f6c4c034040 con 0x561a03d2c570 2023-10-05T19:30:22.884 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.882+0000 7f6c60b15700 1 -- 172.21.15.167:0/2430872828 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:23 5.19 5:98000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x561a03d43ca0 con 0x561a03d2c570 2023-10-05T19:30:22.885 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.882+0000 7f6c60b15700 1 -- 172.21.15.167:0/2430872828 <== osd.1 v2:172.21.15.167:6800/30035 9 ==== osd_op_reply(23 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f6c4c034040 con 0x561a03d2c570 2023-10-05T19:30:22.885 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.882+0000 7f6c60b15700 1 -- 172.21.15.167:0/2430872828 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:24 5.5 5:a0000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x561a03d43ca0 con 0x7f6c4c010070 2023-10-05T19:30:22.886 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.882+0000 7f6c61b17700 1 -- 172.21.15.167:0/2430872828 <== osd.0 v2:172.21.15.118:6800/27690 6 ==== osd_op_reply(24 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f6c5c011040 con 0x7f6c4c010070 2023-10-05T19:30:22.886 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.882+0000 7f6c61b17700 1 -- 172.21.15.167:0/2430872828 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:25 5.15 5:a8000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x561a03d43ca0 con 0x561a03d41380 2023-10-05T19:30:22.886 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.882+0000 7f6c61316700 1 -- 172.21.15.167:0/2430872828 <== osd.3 v2:172.21.15.167:6808/30036 7 ==== osd_op_reply(25 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f6c54056040 con 0x561a03d41380 2023-10-05T19:30:22.886 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.882+0000 7f6c61316700 1 -- 172.21.15.167:0/2430872828 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:26 5.d 5:b0000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x561a03d43ca0 con 0x7f6c5401a070 2023-10-05T19:30:22.888 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.882+0000 7f6c60b15700 1 -- 172.21.15.167:0/2430872828 <== osd.2 v2:172.21.15.167:6816/30037 4 ==== osd_op_reply(26 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f6c4c034040 con 0x7f6c5401a070 2023-10-05T19:30:22.888 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.882+0000 7f6c60b15700 1 -- 172.21.15.167:0/2430872828 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:27 5.1d 5:b8000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x561a03d43ca0 con 0x561a03d2c570 2023-10-05T19:30:22.888 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.886+0000 7f6c60b15700 1 -- 172.21.15.167:0/2430872828 <== osd.1 v2:172.21.15.167:6800/30035 10 ==== osd_op_reply(27 [pgnls start_epoch 32 out=107b] v30'2 uv2 ondisk = 1) v8 ==== 144+0+107 (crc 0 0 0) 0x7f6c4c034040 con 0x561a03d2c570 2023-10-05T19:30:22.888 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.886+0000 7f6c60b15700 1 -- 172.21.15.167:0/2430872828 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:28 5.3 5:c0000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x561a03d43ca0 con 0x7f6c4c010070 2023-10-05T19:30:22.889 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.886+0000 7f6c61b17700 1 -- 172.21.15.167:0/2430872828 <== osd.0 v2:172.21.15.118:6800/27690 7 ==== osd_op_reply(28 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f6c5c011040 con 0x7f6c4c010070 2023-10-05T19:30:22.889 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.886+0000 7f6c61b17700 1 -- 172.21.15.167:0/2430872828 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:29 5.13 5:c8000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x561a03d43ca0 con 0x561a03d41380 2023-10-05T19:30:22.889 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.886+0000 7f6c61316700 1 -- 172.21.15.167:0/2430872828 <== osd.3 v2:172.21.15.167:6808/30036 8 ==== osd_op_reply(29 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f6c54056040 con 0x561a03d41380 2023-10-05T19:30:22.889 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.886+0000 7f6c61316700 1 -- 172.21.15.167:0/2430872828 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:30 5.b 5:d0000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x561a03d43ca0 con 0x7f6c4c010070 2023-10-05T19:30:22.893 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.886+0000 7f6c61b17700 1 -- 172.21.15.167:0/2430872828 <== osd.0 v2:172.21.15.118:6800/27690 8 ==== osd_op_reply(30 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f6c5c011040 con 0x7f6c4c010070 2023-10-05T19:30:22.893 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.886+0000 7f6c61b17700 1 -- 172.21.15.167:0/2430872828 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:31 5.1b 5:d8000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x561a03d43ca0 con 0x7f6c5401a070 2023-10-05T19:30:22.895 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.886+0000 7f6c60b15700 1 -- 172.21.15.167:0/2430872828 <== osd.2 v2:172.21.15.167:6816/30037 5 ==== osd_op_reply(31 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f6c4c034040 con 0x7f6c5401a070 2023-10-05T19:30:22.895 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.886+0000 7f6c60b15700 1 -- 172.21.15.167:0/2430872828 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:32 5.7 5:e0000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x561a03d43ca0 con 0x561a03d41380 2023-10-05T19:30:22.895 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.890+0000 7f6c61316700 1 -- 172.21.15.167:0/2430872828 <== osd.3 v2:172.21.15.167:6808/30036 9 ==== osd_op_reply(32 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f6c54056040 con 0x561a03d41380 2023-10-05T19:30:22.895 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.890+0000 7f6c61316700 1 -- 172.21.15.167:0/2430872828 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:33 5.17 5:e8000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x561a03d43ca0 con 0x7f6c4c010070 2023-10-05T19:30:22.895 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.890+0000 7f6c61b17700 1 -- 172.21.15.167:0/2430872828 <== osd.0 v2:172.21.15.118:6800/27690 9 ==== osd_op_reply(33 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f6c5c011040 con 0x7f6c4c010070 2023-10-05T19:30:22.896 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.890+0000 7f6c61b17700 1 -- 172.21.15.167:0/2430872828 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:34 5.f 5:f0000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x561a03d43ca0 con 0x561a03d2c570 2023-10-05T19:30:22.896 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.890+0000 7f6c60b15700 1 -- 172.21.15.167:0/2430872828 <== osd.1 v2:172.21.15.167:6800/30035 11 ==== osd_op_reply(34 [pgnls start_epoch 32 out=115b] v23'2 uv2 ondisk = 1) v8 ==== 144+0+115 (crc 0 0 0) 0x7f6c4c034040 con 0x561a03d2c570 2023-10-05T19:30:22.896 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.890+0000 7f6c60b15700 1 -- 172.21.15.167:0/2430872828 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:35 5.1f 5:f8000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x561a03d43ca0 con 0x7f6c5401a070 2023-10-05T19:30:22.896 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.890+0000 7f6c60b15700 1 -- 172.21.15.167:0/2430872828 <== osd.2 v2:172.21.15.167:6816/30037 6 ==== osd_op_reply(35 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f6c4c034040 con 0x7f6c5401a070 2023-10-05T19:30:22.896 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.890+0000 7f6c637e1e40 20 RGWRados::pool_iterate: got zonegroup_info.4f1f719d-9648-417b-b10c-e5d939825419 2023-10-05T19:30:22.896 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.890+0000 7f6c637e1e40 20 RGWRados::pool_iterate: got zone_info.9a9df965-6612-45d4-a45f-6e3c889bacc2 2023-10-05T19:30:22.896 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.890+0000 7f6c637e1e40 20 RGWRados::pool_iterate: got zone_names.default 2023-10-05T19:30:22.896 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.890+0000 7f6c637e1e40 20 RGWRados::pool_iterate: got zonegroups_names.default 2023-10-05T19:30:22.896 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.890+0000 7f6c637e1e40 20 rados->read ofs=0 len=0 2023-10-05T19:30:22.896 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.890+0000 7f6c637e1e40 1 -- 172.21.15.167:0/2430872828 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:36 5.f 5:f43b6ece:::zone_names.default:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32) v8 -- 0x561a03d3fe60 con 0x561a03d2c570 2023-10-05T19:30:22.897 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.890+0000 7f6c60b15700 1 -- 172.21.15.167:0/2430872828 <== osd.1 v2:172.21.15.167:6800/30035 12 ==== osd_op_reply(36 zone_names.default [read 0~46 out=46b] v0'0 uv1 ondisk = 0) v8 ==== 162+0+46 (crc 0 0 0) 0x7f6c4c034040 con 0x561a03d2c570 2023-10-05T19:30:22.898 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.890+0000 7f6c637e1e40 20 rados_obj.operate() r=0 bl.length=46 2023-10-05T19:30:22.898 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.890+0000 7f6c637e1e40 20 rados->read ofs=0 len=0 2023-10-05T19:30:22.898 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.890+0000 7f6c637e1e40 1 -- 172.21.15.167:0/2430872828 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:37 5.1d 5:b8374eb4:::zone_info.9a9df965-6612-45d4-a45f-6e3c889bacc2:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32) v8 -- 0x561a03d40260 con 0x561a03d2c570 2023-10-05T19:30:22.898 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.894+0000 7f6c60b15700 1 -- 172.21.15.167:0/2430872828 <== osd.1 v2:172.21.15.167:6800/30035 13 ==== osd_op_reply(37 zone_info.9a9df965-6612-45d4-a45f-6e3c889bacc2 [read 0~874 out=874b] v0'0 uv2 ondisk = 0) v8 ==== 190+0+874 (crc 0 0 0) 0x7f6c4c034040 con 0x561a03d2c570 2023-10-05T19:30:22.898 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.894+0000 7f6c637e1e40 20 rados_obj.operate() r=0 bl.length=874 2023-10-05T19:30:22.898 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.894+0000 7f6c637e1e40 20 rados->read ofs=0 len=0 2023-10-05T19:30:22.898 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.894+0000 7f6c637e1e40 1 -- 172.21.15.167:0/2430872828 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:38 5.12 5:49953fa1:::default.realm:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32) v8 -- 0x561a03d2e1e0 con 0x561a03d2c570 2023-10-05T19:30:22.898 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.894+0000 7f6c60b15700 1 -- 172.21.15.167:0/2430872828 <== osd.1 v2:172.21.15.167:6800/30035 14 ==== osd_op_reply(38 default.realm [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 ==== 157+0+0 (crc 0 0 0) 0x7f6c4c034040 con 0x561a03d2c570 2023-10-05T19:30:22.898 INFO:teuthology.orchestra.run.smithi167.stderr:zone id 9a9df965-6612-45d4-a45f-6e3c889bacc22023-10-05T19:30:22.894+0000 7f6c637e1e40 20 rados_obj.operate() r=-2 bl.length=0 2023-10-05T19:30:22.899 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.894+0000 7f6c637e1e40 1 -- 172.21.15.167:0/2430872828 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:39 5.0 5:00000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x561a03d47fc0 con 0x561a03d41380 2023-10-05T19:30:22.899 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.894+0000 7f6c61316700 1 -- 172.21.15.167:0/2430872828 <== osd.3 v2:172.21.15.167:6808/30036 10 ==== osd_op_reply(39 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f6c54056040 con 0x561a03d41380 2023-10-05T19:30:22.899 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.894+0000 7f6c61316700 1 -- 172.21.15.167:0/2430872828 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:40 5.10 5:08000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x561a03d47fc0 con 0x7f6c5401a070 2023-10-05T19:30:22.899 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.894+0000 7f6c60b15700 1 -- 172.21.15.167:0/2430872828 <== osd.2 v2:172.21.15.167:6816/30037 7 ==== osd_op_reply(40 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f6c4c034040 con 0x7f6c5401a070 2023-10-05T19:30:22.900 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.894+0000 7f6c60b15700 1 -- 172.21.15.167:0/2430872828 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:41 5.8 5:10000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x7f6c4c044040 con 0x7f6c4c010070 2023-10-05T19:30:22.900 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.898+0000 7f6c61b17700 1 -- 172.21.15.167:0/2430872828 <== osd.0 v2:172.21.15.118:6800/27690 10 ==== osd_op_reply(41 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f6c5c011040 con 0x7f6c4c010070 2023-10-05T19:30:22.900 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.898+0000 7f6c61b17700 1 -- 172.21.15.167:0/2430872828 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:42 5.18 5:18000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x7f6c4c044040 con 0x561a03d2c570 2023-10-05T19:30:22.901 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.898+0000 7f6c60b15700 1 -- 172.21.15.167:0/2430872828 <== osd.1 v2:172.21.15.167:6800/30035 15 ==== osd_op_reply(42 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f6c4c034040 con 0x561a03d2c570 2023-10-05T19:30:22.901 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.898+0000 7f6c60b15700 1 -- 172.21.15.167:0/2430872828 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:43 5.4 5:20000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x7f6c4c044040 con 0x561a03d2c570 2023-10-05T19:30:22.902 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.898+0000 7f6c60b15700 1 -- 172.21.15.167:0/2430872828 <== osd.1 v2:172.21.15.167:6800/30035 16 ==== osd_op_reply(43 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f6c4c034040 con 0x561a03d2c570 2023-10-05T19:30:22.902 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.898+0000 7f6c60b15700 1 -- 172.21.15.167:0/2430872828 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:44 5.14 5:28000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x7f6c4c044040 con 0x7f6c4c010070 2023-10-05T19:30:22.903 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.898+0000 7f6c61b17700 1 -- 172.21.15.167:0/2430872828 <== osd.0 v2:172.21.15.118:6800/27690 11 ==== osd_op_reply(44 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f6c5c011040 con 0x7f6c4c010070 2023-10-05T19:30:22.903 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.898+0000 7f6c61b17700 1 -- 172.21.15.167:0/2430872828 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:45 5.c 5:30000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x7f6c4c044040 con 0x7f6c4c010070 2023-10-05T19:30:22.904 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.902+0000 7f6c61b17700 1 -- 172.21.15.167:0/2430872828 <== osd.0 v2:172.21.15.118:6800/27690 12 ==== osd_op_reply(45 [pgnls start_epoch 32 out=112b] v23'1 uv1 ondisk = 1) v8 ==== 144+0+112 (crc 0 0 0) 0x7f6c5c011040 con 0x7f6c4c010070 2023-10-05T19:30:22.904 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.902+0000 7f6c61b17700 1 -- 172.21.15.167:0/2430872828 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:46 5.1c 5:38000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x7f6c4c044040 con 0x7f6c5401a070 2023-10-05T19:30:22.905 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.902+0000 7f6c60b15700 1 -- 172.21.15.167:0/2430872828 <== osd.2 v2:172.21.15.167:6816/30037 8 ==== osd_op_reply(46 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f6c4c034040 con 0x7f6c5401a070 2023-10-05T19:30:22.905 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.902+0000 7f6c60b15700 1 -- 172.21.15.167:0/2430872828 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:47 5.2 5:40000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x7f6c4c044040 con 0x561a03d41380 2023-10-05T19:30:22.906 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.902+0000 7f6c61316700 1 -- 172.21.15.167:0/2430872828 <== osd.3 v2:172.21.15.167:6808/30036 11 ==== osd_op_reply(47 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f6c54056040 con 0x561a03d41380 2023-10-05T19:30:22.906 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.902+0000 7f6c61316700 1 -- 172.21.15.167:0/2430872828 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:48 5.12 5:48000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x7f6c4c044040 con 0x561a03d2c570 2023-10-05T19:30:22.906 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.902+0000 7f6c60b15700 1 -- 172.21.15.167:0/2430872828 <== osd.1 v2:172.21.15.167:6800/30035 17 ==== osd_op_reply(48 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f6c4c034040 con 0x561a03d2c570 2023-10-05T19:30:22.907 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.902+0000 7f6c60b15700 1 -- 172.21.15.167:0/2430872828 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:49 5.a 5:50000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x7f6c4c044040 con 0x7f6c4c010070 2023-10-05T19:30:22.908 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.906+0000 7f6c61b17700 1 -- 172.21.15.167:0/2430872828 <== osd.0 v2:172.21.15.118:6800/27690 13 ==== osd_op_reply(49 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f6c5c011040 con 0x7f6c4c010070 2023-10-05T19:30:22.908 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.906+0000 7f6c61b17700 1 -- 172.21.15.167:0/2430872828 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:50 5.1a 5:58000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x7f6c4c044040 con 0x561a03d2c570 2023-10-05T19:30:22.908 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.906+0000 7f6c60b15700 1 -- 172.21.15.167:0/2430872828 <== osd.1 v2:172.21.15.167:6800/30035 18 ==== osd_op_reply(50 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f6c4c034040 con 0x561a03d2c570 2023-10-05T19:30:22.908 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.906+0000 7f6c60b15700 1 -- 172.21.15.167:0/2430872828 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:51 5.6 5:60000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x7f6c4c044040 con 0x7f6c5401a070 2023-10-05T19:30:22.909 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.906+0000 7f6c60b15700 1 -- 172.21.15.167:0/2430872828 <== osd.2 v2:172.21.15.167:6816/30037 9 ==== osd_op_reply(51 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f6c4c034040 con 0x7f6c5401a070 2023-10-05T19:30:22.910 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.906+0000 7f6c60b15700 1 -- 172.21.15.167:0/2430872828 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:52 5.16 5:68000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x7f6c4c044040 con 0x561a03d41380 2023-10-05T19:30:22.910 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.906+0000 7f6c61316700 1 -- 172.21.15.167:0/2430872828 <== osd.3 v2:172.21.15.167:6808/30036 12 ==== osd_op_reply(52 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f6c54056040 con 0x561a03d41380 2023-10-05T19:30:22.910 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.906+0000 7f6c61316700 1 -- 172.21.15.167:0/2430872828 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:53 5.e 5:70000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x7f6c4c044040 con 0x561a03d41380 2023-10-05T19:30:22.911 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.906+0000 7f6c61316700 1 -- 172.21.15.167:0/2430872828 <== osd.3 v2:172.21.15.167:6808/30036 13 ==== osd_op_reply(53 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f6c54056040 con 0x561a03d41380 2023-10-05T19:30:22.911 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.906+0000 7f6c61316700 1 -- 172.21.15.167:0/2430872828 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:54 5.1e 5:78000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x7f6c4c044040 con 0x7f6c4c010070 2023-10-05T19:30:22.912 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.910+0000 7f6c61b17700 1 -- 172.21.15.167:0/2430872828 <== osd.0 v2:172.21.15.118:6800/27690 14 ==== osd_op_reply(54 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f6c5c011040 con 0x7f6c4c010070 2023-10-05T19:30:22.912 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.910+0000 7f6c61b17700 1 -- 172.21.15.167:0/2430872828 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:55 5.1 5:80000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x7f6c4c044040 con 0x561a03d41380 2023-10-05T19:30:22.913 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.910+0000 7f6c61316700 1 -- 172.21.15.167:0/2430872828 <== osd.3 v2:172.21.15.167:6808/30036 14 ==== osd_op_reply(55 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f6c54056040 con 0x561a03d41380 2023-10-05T19:30:22.913 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.910+0000 7f6c61316700 1 -- 172.21.15.167:0/2430872828 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:56 5.11 5:88000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x7f6c4c044040 con 0x561a03d41380 2023-10-05T19:30:22.914 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.910+0000 7f6c61316700 1 -- 172.21.15.167:0/2430872828 <== osd.3 v2:172.21.15.167:6808/30036 15 ==== osd_op_reply(56 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f6c54056040 con 0x561a03d41380 2023-10-05T19:30:22.914 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.910+0000 7f6c61316700 1 -- 172.21.15.167:0/2430872828 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:57 5.9 5:90000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x7f6c4c044040 con 0x561a03d2c570 2023-10-05T19:30:22.914 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.910+0000 7f6c60b15700 1 -- 172.21.15.167:0/2430872828 <== osd.1 v2:172.21.15.167:6800/30035 19 ==== osd_op_reply(57 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f6c4c034040 con 0x561a03d2c570 2023-10-05T19:30:22.914 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.910+0000 7f6c60b15700 1 -- 172.21.15.167:0/2430872828 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:58 5.19 5:98000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x7f6c4c044040 con 0x561a03d2c570 2023-10-05T19:30:22.915 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.910+0000 7f6c60b15700 1 -- 172.21.15.167:0/2430872828 <== osd.1 v2:172.21.15.167:6800/30035 20 ==== osd_op_reply(58 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f6c4c034040 con 0x561a03d2c570 2023-10-05T19:30:22.915 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.910+0000 7f6c60b15700 1 -- 172.21.15.167:0/2430872828 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:59 5.5 5:a0000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x7f6c4c044040 con 0x7f6c4c010070 2023-10-05T19:30:22.916 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.914+0000 7f6c61b17700 1 -- 172.21.15.167:0/2430872828 <== osd.0 v2:172.21.15.118:6800/27690 15 ==== osd_op_reply(59 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f6c5c011040 con 0x7f6c4c010070 2023-10-05T19:30:22.916 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.914+0000 7f6c61b17700 1 -- 172.21.15.167:0/2430872828 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:60 5.15 5:a8000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x7f6c4c044040 con 0x561a03d41380 2023-10-05T19:30:22.917 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.914+0000 7f6c61316700 1 -- 172.21.15.167:0/2430872828 <== osd.3 v2:172.21.15.167:6808/30036 16 ==== osd_op_reply(60 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f6c54056040 con 0x561a03d41380 2023-10-05T19:30:22.917 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.914+0000 7f6c61316700 1 -- 172.21.15.167:0/2430872828 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:61 5.d 5:b0000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x7f6c4c044040 con 0x7f6c5401a070 2023-10-05T19:30:22.918 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.914+0000 7f6c60b15700 1 -- 172.21.15.167:0/2430872828 <== osd.2 v2:172.21.15.167:6816/30037 10 ==== osd_op_reply(61 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f6c4c034040 con 0x7f6c5401a070 2023-10-05T19:30:22.918 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.914+0000 7f6c60b15700 1 -- 172.21.15.167:0/2430872828 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:62 5.1d 5:b8000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x7f6c4c044040 con 0x561a03d2c570 2023-10-05T19:30:22.919 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.914+0000 7f6c60b15700 1 -- 172.21.15.167:0/2430872828 <== osd.1 v2:172.21.15.167:6800/30035 21 ==== osd_op_reply(62 [pgnls start_epoch 32 out=107b] v30'2 uv2 ondisk = 1) v8 ==== 144+0+107 (crc 0 0 0) 0x7f6c4c034040 con 0x561a03d2c570 2023-10-05T19:30:22.919 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.914+0000 7f6c60b15700 1 -- 172.21.15.167:0/2430872828 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:63 5.3 5:c0000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x7f6c4c044040 con 0x7f6c4c010070 2023-10-05T19:30:22.920 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.918+0000 7f6c61b17700 1 -- 172.21.15.167:0/2430872828 <== osd.0 v2:172.21.15.118:6800/27690 16 ==== osd_op_reply(63 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f6c5c011040 con 0x7f6c4c010070 2023-10-05T19:30:22.920 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.918+0000 7f6c61b17700 1 -- 172.21.15.167:0/2430872828 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:64 5.13 5:c8000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x7f6c4c044040 con 0x561a03d41380 2023-10-05T19:30:22.921 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.918+0000 7f6c61316700 1 -- 172.21.15.167:0/2430872828 <== osd.3 v2:172.21.15.167:6808/30036 17 ==== osd_op_reply(64 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f6c54056040 con 0x561a03d41380 2023-10-05T19:30:22.921 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.918+0000 7f6c61316700 1 -- 172.21.15.167:0/2430872828 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:65 5.b 5:d0000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x7f6c4c044040 con 0x7f6c4c010070 2023-10-05T19:30:22.922 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.918+0000 7f6c61b17700 1 -- 172.21.15.167:0/2430872828 <== osd.0 v2:172.21.15.118:6800/27690 17 ==== osd_op_reply(65 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f6c5c011040 con 0x7f6c4c010070 2023-10-05T19:30:22.922 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.918+0000 7f6c61b17700 1 -- 172.21.15.167:0/2430872828 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:66 5.1b 5:d8000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x7f6c4c044040 con 0x7f6c5401a070 2023-10-05T19:30:22.923 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.918+0000 7f6c60b15700 1 -- 172.21.15.167:0/2430872828 <== osd.2 v2:172.21.15.167:6816/30037 11 ==== osd_op_reply(66 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f6c4c034040 con 0x7f6c5401a070 2023-10-05T19:30:22.923 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.918+0000 7f6c60b15700 1 -- 172.21.15.167:0/2430872828 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:67 5.7 5:e0000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x7f6c4c044040 con 0x561a03d41380 2023-10-05T19:30:22.925 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.922+0000 7f6c61316700 1 -- 172.21.15.167:0/2430872828 <== osd.3 v2:172.21.15.167:6808/30036 18 ==== osd_op_reply(67 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f6c54056040 con 0x561a03d41380 2023-10-05T19:30:22.925 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.922+0000 7f6c61316700 1 -- 172.21.15.167:0/2430872828 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:68 5.17 5:e8000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x7f6c4c044040 con 0x7f6c4c010070 2023-10-05T19:30:22.925 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.922+0000 7f6c61b17700 1 -- 172.21.15.167:0/2430872828 <== osd.0 v2:172.21.15.118:6800/27690 18 ==== osd_op_reply(68 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f6c5c011040 con 0x7f6c4c010070 2023-10-05T19:30:22.925 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.922+0000 7f6c61b17700 1 -- 172.21.15.167:0/2430872828 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:69 5.f 5:f0000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x7f6c4c044040 con 0x561a03d2c570 2023-10-05T19:30:22.926 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.922+0000 7f6c60b15700 1 -- 172.21.15.167:0/2430872828 <== osd.1 v2:172.21.15.167:6800/30035 22 ==== osd_op_reply(69 [pgnls start_epoch 32 out=115b] v23'2 uv2 ondisk = 1) v8 ==== 144+0+115 (crc 0 0 0) 0x7f6c4c034040 con 0x561a03d2c570 2023-10-05T19:30:22.926 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.922+0000 7f6c60b15700 1 -- 172.21.15.167:0/2430872828 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:70 5.1f 5:f8000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x7f6c4c044040 con 0x7f6c5401a070 2023-10-05T19:30:22.927 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.922+0000 7f6c60b15700 1 -- 172.21.15.167:0/2430872828 <== osd.2 v2:172.21.15.167:6816/30037 12 ==== osd_op_reply(70 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f6c4c034040 con 0x7f6c5401a070 2023-10-05T19:30:22.927 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.922+0000 7f6c637e1e40 20 RGWRados::pool_iterate: got zonegroup_info.4f1f719d-9648-417b-b10c-e5d939825419 2023-10-05T19:30:22.927 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.922+0000 7f6c637e1e40 20 RGWRados::pool_iterate: got zone_info.9a9df965-6612-45d4-a45f-6e3c889bacc2 2023-10-05T19:30:22.927 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.922+0000 7f6c637e1e40 20 RGWRados::pool_iterate: got zone_names.default 2023-10-05T19:30:22.927 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.922+0000 7f6c637e1e40 20 RGWRados::pool_iterate: got zonegroups_names.default 2023-10-05T19:30:22.927 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.922+0000 7f6c637e1e40 20 rados->read ofs=0 len=0 2023-10-05T19:30:22.927 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.922+0000 7f6c637e1e40 1 -- 172.21.15.167:0/2430872828 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:71 5.f 5:f43b6ece:::zone_names.default:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32) v8 -- 0x561a03d2e1e0 con 0x561a03d2c570 2023-10-05T19:30:22.928 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.922+0000 7f6c60b15700 1 -- 172.21.15.167:0/2430872828 <== osd.1 v2:172.21.15.167:6800/30035 23 ==== osd_op_reply(71 zone_names.default [read 0~46 out=46b] v0'0 uv1 ondisk = 0) v8 ==== 162+0+46 (crc 0 0 0) 0x7f6c4c034040 con 0x561a03d2c570 2023-10-05T19:30:22.928 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.922+0000 7f6c637e1e40 20 rados_obj.operate() r=0 bl.length=46 2023-10-05T19:30:22.928 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.926+0000 7f6c637e1e40 20 rados->read ofs=0 len=0 2023-10-05T19:30:22.928 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.926+0000 7f6c637e1e40 1 -- 172.21.15.167:0/2430872828 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:72 5.1d 5:b8374eb4:::zone_info.9a9df965-6612-45d4-a45f-6e3c889bacc2:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32) v8 -- 0x561a03d2e1e0 con 0x561a03d2c570 2023-10-05T19:30:22.929 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.926+0000 7f6c60b15700 1 -- 172.21.15.167:0/2430872828 <== osd.1 v2:172.21.15.167:6800/30035 24 ==== osd_op_reply(72 zone_info.9a9df965-6612-45d4-a45f-6e3c889bacc2 [read 0~874 out=874b] v0'0 uv2 ondisk = 0) v8 ==== 190+0+874 (crc 0 0 0) 0x7f6c4c034040 con 0x561a03d2c570 2023-10-05T19:30:22.929 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.926+0000 7f6c637e1e40 20 rados_obj.operate() r=0 bl.length=874 2023-10-05T19:30:22.929 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.926+0000 7f6c637e1e40 1 -- 172.21.15.167:0/2430872828 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:73 5.1d 5:b8374eb4:::zone_info.9a9df965-6612-45d4-a45f-6e3c889bacc2:head [delete,create,writefull 0~1036 in=1036b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e32) v8 -- 0x561a03d47fc0 con 0x561a03d2c570 2023-10-05T19:30:22.932 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.930+0000 7f6c60b15700 1 -- 172.21.15.167:0/2430872828 <== osd.1 v2:172.21.15.167:6800/30035 25 ==== osd_op_reply(73 zone_info.9a9df965-6612-45d4-a45f-6e3c889bacc2 [delete,create,writefull 0~1036] v32'3 uv3 ondisk = 0) v8 ==== 274+0+0 (crc 0 0 0) 0x7f6c4c034040 con 0x561a03d2c570 2023-10-05T19:30:22.932 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.930+0000 7f6c637e1e40 1 -- 172.21.15.167:0/2430872828 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:74 5.f 5:f43b6ece:::zone_names.default:head [delete,create,writefull 0~46 in=46b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e32) v8 -- 0x561a03d47fc0 con 0x561a03d2c570 2023-10-05T19:30:22.935 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.930+0000 7f6c60b15700 1 -- 172.21.15.167:0/2430872828 <== osd.1 v2:172.21.15.167:6800/30035 26 ==== osd_op_reply(74 zone_names.default [delete,create,writefull 0~46] v32'3 uv3 ondisk = 0) v8 ==== 246+0+0 (crc 0 0 0) 0x7f6c4c034040 con 0x561a03d2c570 2023-10-05T19:30:22.938 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.934+0000 7f6c637e1e40 1 -- 172.21.15.167:0/2430872828 >> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] conn(0x7f6c4c010070 msgr2=0x7f6c4c028940 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-05T19:30:22.938 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.934+0000 7f6c637e1e40 1 --2- 172.21.15.167:0/2430872828 >> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] conn(0x7f6c4c010070 0x7f6c4c028940 crc :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-05T19:30:22.938 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.934+0000 7f6c637e1e40 1 -- 172.21.15.167:0/2430872828 >> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] conn(0x561a03d2c570 msgr2=0x561a03d2c9f0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-05T19:30:22.938 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.934+0000 7f6c637e1e40 1 --2- 172.21.15.167:0/2430872828 >> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] conn(0x561a03d2c570 0x561a03d2c9f0 crc :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-05T19:30:22.938 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.934+0000 7f6c637e1e40 1 -- 172.21.15.167:0/2430872828 >> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] conn(0x7f6c5401a070 msgr2=0x7f6c54044050 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-05T19:30:22.938 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.934+0000 7f6c637e1e40 1 --2- 172.21.15.167:0/2430872828 >> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] conn(0x7f6c5401a070 0x7f6c54044050 crc :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-05T19:30:22.939 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.934+0000 7f6c637e1e40 1 -- 172.21.15.167:0/2430872828 >> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] conn(0x561a03d41380 msgr2=0x561a03d43760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-05T19:30:22.939 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.934+0000 7f6c637e1e40 1 --2- 172.21.15.167:0/2430872828 >> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] conn(0x561a03d41380 0x561a03d43760 crc :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-05T19:30:22.939 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.934+0000 7f6c637e1e40 1 -- 172.21.15.167:0/2430872828 >> [v2:172.21.15.167:6824/29889,v1:172.21.15.167:6825/29889] conn(0x7f6c48030c00 msgr2=0x7f6c480330b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-05T19:30:22.939 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.934+0000 7f6c637e1e40 1 --2- 172.21.15.167:0/2430872828 >> [v2:172.21.15.167:6824/29889,v1:172.21.15.167:6825/29889] conn(0x7f6c48030c00 0x7f6c480330b0 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7f6c4c000f80 tx=0x7f6c4c013000 comp rx=0 tx=0).stop 2023-10-05T19:30:22.939 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.934+0000 7f6c637e1e40 1 -- 172.21.15.167:0/2430872828 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x561a03ccffe0 msgr2=0x561a03cf5f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-05T19:30:22.939 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.934+0000 7f6c637e1e40 1 --2- 172.21.15.167:0/2430872828 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x561a03ccffe0 0x561a03cf5f30 secure :-1 s=READY pgs=683 cs=0 l=1 rev1=1 crypto rx=0x7f6c540018e0 tx=0x7f6c54016000 comp rx=0 tx=0).stop 2023-10-05T19:30:22.939 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.934+0000 7f6c61b17700 1 -- 172.21.15.167:0/2430872828 reap_dead start 2023-10-05T19:30:22.940 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.934+0000 7f6c637e1e40 1 -- 172.21.15.167:0/2430872828 shutdown_connections 2023-10-05T19:30:22.940 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.934+0000 7f6c637e1e40 1 -- 172.21.15.167:0/2430872828 >> 172.21.15.167:0/2430872828 conn(0x561a03c43830 msgr2=0x561a03ce97a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-05T19:30:22.940 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.934+0000 7f6c637e1e40 1 -- 172.21.15.167:0/2430872828 shutdown_connections 2023-10-05T19:30:22.940 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.934+0000 7f6c637e1e40 1 -- 172.21.15.167:0/2430872828 wait complete. 2023-10-05T19:30:22.941 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.934+0000 7f6c637e1e40 1 -- 172.21.15.167:0/3050533416 >> [v2:172.21.15.167:6824/29889,v1:172.21.15.167:6825/29889] conn(0x7f6bb0030c00 msgr2=0x7f6bb00330b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-05T19:30:22.941 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.934+0000 7f6c637e1e40 1 --2- 172.21.15.167:0/3050533416 >> [v2:172.21.15.167:6824/29889,v1:172.21.15.167:6825/29889] conn(0x7f6bb0030c00 0x7f6bb00330b0 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7f6c5c00a370 tx=0x7f6c5c005500 comp rx=0 tx=0).stop 2023-10-05T19:30:22.941 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.934+0000 7f6c637e1e40 1 -- 172.21.15.167:0/3050533416 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x561a03d07590 msgr2=0x561a03d0e850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-05T19:30:22.941 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.934+0000 7f6c637e1e40 1 --2- 172.21.15.167:0/3050533416 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x561a03d07590 0x561a03d0e850 secure :-1 s=READY pgs=685 cs=0 l=1 rev1=1 crypto rx=0x7f6c5401cfd0 tx=0x7f6c54036000 comp rx=0 tx=0).stop 2023-10-05T19:30:22.941 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.934+0000 7f6c637e1e40 1 -- 172.21.15.167:0/3050533416 shutdown_connections 2023-10-05T19:30:22.941 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.934+0000 7f6c637e1e40 1 --2- 172.21.15.167:0/3050533416 >> [v2:172.21.15.167:6824/29889,v1:172.21.15.167:6825/29889] conn(0x7f6bb0030c00 0x7f6bb00330b0 unknown :-1 s=CLOSED pgs=235 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-05T19:30:22.941 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.934+0000 7f6c637e1e40 1 --2- 172.21.15.167:0/3050533416 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x561a03d07590 0x561a03d0e850 unknown :-1 s=CLOSED pgs=685 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-05T19:30:22.941 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.934+0000 7f6c637e1e40 1 -- 172.21.15.167:0/3050533416 >> 172.21.15.167:0/3050533416 conn(0x561a03cd45c0 msgr2=0x561a03d03f20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-05T19:30:22.942 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.934+0000 7f6c637e1e40 1 -- 172.21.15.167:0/3050533416 shutdown_connections 2023-10-05T19:30:22.942 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.934+0000 7f6c637e1e40 1 -- 172.21.15.167:0/3050533416 wait complete. 2023-10-05T19:30:22.942 INFO:teuthology.orchestra.run.smithi167.stdout:{ 2023-10-05T19:30:22.942 INFO:teuthology.orchestra.run.smithi167.stdout: "id": "9a9df965-6612-45d4-a45f-6e3c889bacc2", 2023-10-05T19:30:22.942 INFO:teuthology.orchestra.run.smithi167.stdout: "name": "default", 2023-10-05T19:30:22.942 INFO:teuthology.orchestra.run.smithi167.stdout: "domain_root": "default.rgw.meta:root", 2023-10-05T19:30:22.942 INFO:teuthology.orchestra.run.smithi167.stdout: "control_pool": "default.rgw.control", 2023-10-05T19:30:22.942 INFO:teuthology.orchestra.run.smithi167.stdout: "gc_pool": "default.rgw.log:gc", 2023-10-05T19:30:22.943 INFO:teuthology.orchestra.run.smithi167.stdout: "lc_pool": "default.rgw.log:lc", 2023-10-05T19:30:22.943 INFO:teuthology.orchestra.run.smithi167.stdout: "log_pool": "default.rgw.log", 2023-10-05T19:30:22.943 INFO:teuthology.orchestra.run.smithi167.stdout: "intent_log_pool": "default.rgw.log:intent", 2023-10-05T19:30:22.943 INFO:teuthology.orchestra.run.smithi167.stdout: "usage_log_pool": "default.rgw.log:usage", 2023-10-05T19:30:22.943 INFO:teuthology.orchestra.run.smithi167.stdout: "roles_pool": "default.rgw.meta:roles", 2023-10-05T19:30:22.943 INFO:teuthology.orchestra.run.smithi167.stdout: "reshard_pool": "default.rgw.log:reshard", 2023-10-05T19:30:22.943 INFO:teuthology.orchestra.run.smithi167.stdout: "user_keys_pool": "default.rgw.meta:users.keys", 2023-10-05T19:30:22.943 INFO:teuthology.orchestra.run.smithi167.stdout: "user_email_pool": "default.rgw.meta:users.email", 2023-10-05T19:30:22.943 INFO:teuthology.orchestra.run.smithi167.stdout: "user_swift_pool": "default.rgw.meta:users.swift", 2023-10-05T19:30:22.943 INFO:teuthology.orchestra.run.smithi167.stdout: "user_uid_pool": "default.rgw.meta:users.uid", 2023-10-05T19:30:22.943 INFO:teuthology.orchestra.run.smithi167.stdout: "otp_pool": "default.rgw.otp", 2023-10-05T19:30:22.943 INFO:teuthology.orchestra.run.smithi167.stdout: "system_key": { 2023-10-05T19:30:22.944 INFO:teuthology.orchestra.run.smithi167.stdout: "access_key": "", 2023-10-05T19:30:22.944 INFO:teuthology.orchestra.run.smithi167.stdout: "secret_key": "" 2023-10-05T19:30:22.944 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2023-10-05T19:30:22.944 INFO:teuthology.orchestra.run.smithi167.stdout: "placement_pools": [ 2023-10-05T19:30:22.944 INFO:teuthology.orchestra.run.smithi167.stdout: { 2023-10-05T19:30:22.944 INFO:teuthology.orchestra.run.smithi167.stdout: "key": "default-placement", 2023-10-05T19:30:22.944 INFO:teuthology.orchestra.run.smithi167.stdout: "val": { 2023-10-05T19:30:22.944 INFO:teuthology.orchestra.run.smithi167.stdout: "index_pool": "default.rgw.buckets.index", 2023-10-05T19:30:22.944 INFO:teuthology.orchestra.run.smithi167.stdout: "storage_classes": { 2023-10-05T19:30:22.945 INFO:teuthology.orchestra.run.smithi167.stdout: "STANDARD": { 2023-10-05T19:30:22.945 INFO:teuthology.orchestra.run.smithi167.stdout: "data_pool": "default.rgw.buckets.data" 2023-10-05T19:30:22.945 INFO:teuthology.orchestra.run.smithi167.stdout: } 2023-10-05T19:30:22.945 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2023-10-05T19:30:22.945 INFO:teuthology.orchestra.run.smithi167.stdout: "data_extra_pool": "default.rgw.buckets.non-ec", 2023-10-05T19:30:22.946 INFO:teuthology.orchestra.run.smithi167.stdout: "index_type": 0, 2023-10-05T19:30:22.946 INFO:teuthology.orchestra.run.smithi167.stdout: "inline_data": "true" 2023-10-05T19:30:22.946 INFO:teuthology.orchestra.run.smithi167.stdout: } 2023-10-05T19:30:22.946 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2023-10-05T19:30:22.946 INFO:teuthology.orchestra.run.smithi167.stdout: { 2023-10-05T19:30:22.946 INFO:teuthology.orchestra.run.smithi167.stdout: "key": "new-placement", 2023-10-05T19:30:22.946 INFO:teuthology.orchestra.run.smithi167.stdout: "val": { 2023-10-05T19:30:22.946 INFO:teuthology.orchestra.run.smithi167.stdout: "index_pool": ".rgw.buckets.index.2", 2023-10-05T19:30:22.947 INFO:teuthology.orchestra.run.smithi167.stdout: "storage_classes": { 2023-10-05T19:30:22.947 INFO:teuthology.orchestra.run.smithi167.stdout: "STANDARD": { 2023-10-05T19:30:22.947 INFO:teuthology.orchestra.run.smithi167.stdout: "data_pool": ".rgw.buckets.2" 2023-10-05T19:30:22.947 INFO:teuthology.orchestra.run.smithi167.stdout: } 2023-10-05T19:30:22.947 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2023-10-05T19:30:22.947 INFO:teuthology.orchestra.run.smithi167.stdout: "data_extra_pool": "default.rgw.buckets.non-ec", 2023-10-05T19:30:22.947 INFO:teuthology.orchestra.run.smithi167.stdout: "index_type": 0, 2023-10-05T19:30:22.947 INFO:teuthology.orchestra.run.smithi167.stdout: "inline_data": "false" 2023-10-05T19:30:22.947 INFO:teuthology.orchestra.run.smithi167.stdout: } 2023-10-05T19:30:22.947 INFO:teuthology.orchestra.run.smithi167.stdout: } 2023-10-05T19:30:22.947 INFO:teuthology.orchestra.run.smithi167.stdout: ], 2023-10-05T19:30:22.947 INFO:teuthology.orchestra.run.smithi167.stdout: "realm_id": "", 2023-10-05T19:30:22.948 INFO:teuthology.orchestra.run.smithi167.stdout: "notif_pool": "default.rgw.log:notif" 2023-10-05T19:30:22.948 INFO:teuthology.orchestra.run.smithi167.stdout:} 2023-10-05T19:30:22.948 DEBUG:tasks.util.rgw: json result: {'id': '9a9df965-6612-45d4-a45f-6e3c889bacc2', 'name': 'default', 'domain_root': 'default.rgw.meta:root', 'control_pool': 'default.rgw.control', 'gc_pool': 'default.rgw.log:gc', 'lc_pool': 'default.rgw.log:lc', 'log_pool': 'default.rgw.log', 'intent_log_pool': 'default.rgw.log:intent', 'usage_log_pool': 'default.rgw.log:usage', 'roles_pool': 'default.rgw.meta:roles', 'reshard_pool': 'default.rgw.log:reshard', 'user_keys_pool': 'default.rgw.meta:users.keys', 'user_email_pool': 'default.rgw.meta:users.email', 'user_swift_pool': 'default.rgw.meta:users.swift', 'user_uid_pool': 'default.rgw.meta:users.uid', 'otp_pool': 'default.rgw.otp', 'system_key': {'access_key': '', 'secret_key': ''}, 'placement_pools': [{'key': 'default-placement', 'val': {'index_pool': 'default.rgw.buckets.index', 'storage_classes': {'STANDARD': {'data_pool': 'default.rgw.buckets.data'}}, 'data_extra_pool': 'default.rgw.buckets.non-ec', 'index_type': 0, 'inline_data': 'true'}}, {'key': 'new-placement', 'val': {'index_pool': '.rgw.buckets.index.2', 'storage_classes': {'STANDARD': {'data_pool': '.rgw.buckets.2'}}, 'data_extra_pool': 'default.rgw.buckets.non-ec', 'index_type': 0, 'inline_data': 'false'}}], 'realm_id': '', 'notif_pool': 'default.rgw.log:notif'} 2023-10-05T19:30:22.949 INFO:tasks.util.rgw:rgwadmin: client.0 : ['zone', 'get'] 2023-10-05T19:30:22.949 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'ceph', 'zone', 'get'] 2023-10-05T19:30:22.950 DEBUG:teuthology.orchestra.run.smithi167:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster ceph zone get 2023-10-05T19:30:22.986 INFO:teuthology.orchestra.run.smithi167.stderr:ignoring --setuser ceph since I am not root 2023-10-05T19:30:22.986 INFO:teuthology.orchestra.run.smithi167.stderr:ignoring --setgroup ceph since I am not root 2023-10-05T19:30:22.987 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.982+0000 7fd0ed711e40 1 Processor -- start 2023-10-05T19:30:22.987 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.982+0000 7fd0ed711e40 1 -- start start 2023-10-05T19:30:22.987 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.982+0000 7fd0ed711e40 1 --2- >> v2:172.21.15.118:3300/0 conn(0x5570aa2065c0 0x5570aa2089a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-05T19:30:22.988 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.982+0000 7fd0ed711e40 1 -- --> v1:172.21.15.118:6789/0 -- auth(proto 0 26 bytes epoch 0) v1 -- 0x5570aa14cc00 con 0x5570aa201fe0 2023-10-05T19:30:22.988 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.982+0000 7fd0ed711e40 1 -- --> v2:172.21.15.118:3300/0 -- mon_getmap magic: 0 v1 -- 0x5570aa109ee0 con 0x5570aa2065c0 2023-10-05T19:30:22.988 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.986+0000 7fd0eaa45700 1 --2- >> v2:172.21.15.118:3300/0 conn(0x5570aa2065c0 0x5570aa2089a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-05T19:30:22.988 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.986+0000 7fd0eb246700 1 --1- >> v1:172.21.15.118:6789/0 conn(0x5570aa201fe0 0x5570aa2023b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.118:6789/0 says I am v1:172.21.15.167:50832/0 (socket says 172.21.15.167:50832) 2023-10-05T19:30:22.988 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.986+0000 7fd0eb246700 1 -- 172.21.15.167:0/337680611 learned_addr learned my addr 172.21.15.167:0/337680611 (peer_addr_for_me v1:172.21.15.167:0/0) 2023-10-05T19:30:22.989 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.986+0000 7fd0eaa45700 1 -- 172.21.15.167:0/337680611 >> v1:172.21.15.118:6789/0 conn(0x5570aa201fe0 legacy=0x5570aa2023b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-05T19:30:22.989 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.986+0000 7fd0ea244700 1 -- 172.21.15.167:0/337680611 <== mon.0 v1:172.21.15.118:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3516362341 0 0) 0x5570aa14cc00 con 0x5570aa201fe0 2023-10-05T19:30:22.989 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.986+0000 7fd0eaa45700 1 -- 172.21.15.167:0/337680611 --> v2:172.21.15.118:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5570aa202bf0 con 0x5570aa2065c0 2023-10-05T19:30:22.989 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.986+0000 7fd0eaa45700 1 --2- 172.21.15.167:0/337680611 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5570aa2065c0 0x5570aa2089a0 secure :-1 s=READY pgs=687 cs=0 l=1 rev1=1 crypto rx=0x7fd0d400a520 tx=0x7fd0d4005d40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e7cb0bdf67c88d01 server_cookie=0 in_seq=0 out_seq=0 2023-10-05T19:30:22.990 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.986+0000 7fd0ea244700 1 -- 172.21.15.167:0/337680611 <== mon.0 v2:172.21.15.118:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fd0d400f590 con 0x5570aa2065c0 2023-10-05T19:30:22.990 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.986+0000 7fd0ea244700 1 -- 172.21.15.167:0/337680611 <== mon.0 v2:172.21.15.118:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd0d400f6f0 con 0x5570aa2065c0 2023-10-05T19:30:22.990 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.986+0000 7fd0ea244700 1 -- 172.21.15.167:0/337680611 <== mon.0 v2:172.21.15.118:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fd0d400f880 con 0x5570aa2065c0 2023-10-05T19:30:22.991 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.986+0000 7fd0ed711e40 1 -- 172.21.15.167:0/337680611 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5570aa2065c0 msgr2=0x5570aa2089a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-05T19:30:22.991 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.986+0000 7fd0ed711e40 1 --2- 172.21.15.167:0/337680611 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5570aa2065c0 0x5570aa2089a0 secure :-1 s=READY pgs=687 cs=0 l=1 rev1=1 crypto rx=0x7fd0d400a520 tx=0x7fd0d4005d40 comp rx=0 tx=0).stop 2023-10-05T19:30:22.991 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.986+0000 7fd0ed711e40 1 -- 172.21.15.167:0/337680611 shutdown_connections 2023-10-05T19:30:22.991 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.986+0000 7fd0ed711e40 1 --2- 172.21.15.167:0/337680611 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5570aa2065c0 0x5570aa2089a0 unknown :-1 s=CLOSED pgs=687 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-05T19:30:22.991 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.986+0000 7fd0ed711e40 1 -- 172.21.15.167:0/337680611 >> 172.21.15.167:0/337680611 conn(0x5570aa175830 msgr2=0x5570aa175c30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-05T19:30:22.991 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.986+0000 7fd0ed711e40 1 -- 172.21.15.167:0/337680611 shutdown_connections 2023-10-05T19:30:22.991 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.986+0000 7fd0ed711e40 1 -- 172.21.15.167:0/337680611 wait complete. 2023-10-05T19:30:22.992 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.986+0000 7fd0ed711e40 1 Processor -- start 2023-10-05T19:30:22.992 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.986+0000 7fd0ed711e40 1 -- start start 2023-10-05T19:30:22.992 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.986+0000 7fd0ed711e40 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5570aa201fe0 0x5570aa221010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-05T19:30:22.992 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.986+0000 7fd0ed711e40 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x5570aa0cde30 con 0x5570aa201fe0 2023-10-05T19:30:22.992 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.990+0000 7fd0eb246700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5570aa201fe0 0x5570aa221010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-05T19:30:22.992 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.990+0000 7fd0eb246700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5570aa201fe0 0x5570aa221010 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.118:3300/0 says I am v2:172.21.15.167:33194/0 (socket says 172.21.15.167:33194) 2023-10-05T19:30:22.992 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.990+0000 7fd0eb246700 1 -- 172.21.15.167:0/205394083 learned_addr learned my addr 172.21.15.167:0/205394083 (peer_addr_for_me v2:172.21.15.167:0/0) 2023-10-05T19:30:22.993 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.990+0000 7fd0eb246700 1 -- 172.21.15.167:0/205394083 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd0d400a0e0 con 0x5570aa201fe0 2023-10-05T19:30:22.993 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.990+0000 7fd0eb246700 1 --2- 172.21.15.167:0/205394083 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5570aa201fe0 0x5570aa221010 secure :-1 s=READY pgs=688 cs=0 l=1 rev1=1 crypto rx=0x7fd0dc001dc0 tx=0x7fd0dc001790 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ee127dae47ac24a2 server_cookie=0 in_seq=0 out_seq=0 2023-10-05T19:30:22.993 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.990+0000 7fd0e9242700 1 -- 172.21.15.167:0/205394083 <== mon.0 v2:172.21.15.118:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fd0dc00bce0 con 0x5570aa201fe0 2023-10-05T19:30:22.993 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.990+0000 7fd0e9242700 1 -- 172.21.15.167:0/205394083 <== mon.0 v2:172.21.15.118:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd0dc00be40 con 0x5570aa201fe0 2023-10-05T19:30:22.994 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.990+0000 7fd0e9242700 1 -- 172.21.15.167:0/205394083 <== mon.0 v2:172.21.15.118:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fd0dc00e5d0 con 0x5570aa201fe0 2023-10-05T19:30:22.994 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.990+0000 7fd0ed711e40 1 -- 172.21.15.167:0/205394083 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5570aa201fe0 msgr2=0x5570aa221010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-05T19:30:22.995 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.990+0000 7fd0ed711e40 1 --2- 172.21.15.167:0/205394083 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5570aa201fe0 0x5570aa221010 secure :-1 s=READY pgs=688 cs=0 l=1 rev1=1 crypto rx=0x7fd0dc001dc0 tx=0x7fd0dc001790 comp rx=0 tx=0).stop 2023-10-05T19:30:22.995 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.990+0000 7fd0ed711e40 1 -- 172.21.15.167:0/205394083 shutdown_connections 2023-10-05T19:30:22.995 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.990+0000 7fd0ed711e40 1 --2- 172.21.15.167:0/205394083 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5570aa201fe0 0x5570aa221010 unknown :-1 s=CLOSED pgs=688 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-05T19:30:22.995 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.990+0000 7fd0ed711e40 1 -- 172.21.15.167:0/205394083 >> 172.21.15.167:0/205394083 conn(0x5570aa175830 msgr2=0x5570aa21cbd0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-05T19:30:22.995 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.990+0000 7fd0ed711e40 1 -- 172.21.15.167:0/205394083 shutdown_connections 2023-10-05T19:30:22.995 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.990+0000 7fd0ed711e40 1 -- 172.21.15.167:0/205394083 wait complete. 2023-10-05T19:30:22.995 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.990+0000 7fd0ed711e40 1 Processor -- start 2023-10-05T19:30:22.995 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.990+0000 7fd0ed711e40 1 -- start start 2023-10-05T19:30:22.995 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.990+0000 7fd0ed711e40 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5570aa227f30 0x5570aa228300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-05T19:30:22.995 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.990+0000 7fd0ed711e40 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x5570aa0a0ed0 con 0x5570aa227f30 2023-10-05T19:30:22.996 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.990+0000 7fd0eb246700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5570aa227f30 0x5570aa228300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-05T19:30:22.997 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.990+0000 7fd0eb246700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5570aa227f30 0x5570aa228300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.118:3300/0 says I am v2:172.21.15.167:33196/0 (socket says 172.21.15.167:33196) 2023-10-05T19:30:22.997 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.990+0000 7fd0eb246700 1 -- 172.21.15.167:0/1564520155 learned_addr learned my addr 172.21.15.167:0/1564520155 (peer_addr_for_me v2:172.21.15.167:0/0) 2023-10-05T19:30:22.997 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.994+0000 7fd0eb246700 1 -- 172.21.15.167:0/1564520155 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd0dc001a70 con 0x5570aa227f30 2023-10-05T19:30:22.997 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.994+0000 7fd0eb246700 1 --2- 172.21.15.167:0/1564520155 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5570aa227f30 0x5570aa228300 secure :-1 s=READY pgs=689 cs=0 l=1 rev1=1 crypto rx=0x7fd0dc000c00 tx=0x7fd0dc016000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-05T19:30:22.997 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.994+0000 7fd0e37fe700 1 -- 172.21.15.167:0/1564520155 <== mon.0 v2:172.21.15.118:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fd0dc01f070 con 0x5570aa227f30 2023-10-05T19:30:22.997 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.994+0000 7fd0e37fe700 1 -- 172.21.15.167:0/1564520155 <== mon.0 v2:172.21.15.118:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd0dc01a2e0 con 0x5570aa227f30 2023-10-05T19:30:22.997 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.994+0000 7fd0ed711e40 1 -- 172.21.15.167:0/1564520155 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x5570aa228840 con 0x5570aa227f30 2023-10-05T19:30:22.997 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.994+0000 7fd0e37fe700 1 -- 172.21.15.167:0/1564520155 <== mon.0 v2:172.21.15.118:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fd0dc01a7c0 con 0x5570aa227f30 2023-10-05T19:30:22.997 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.994+0000 7fd0ed711e40 1 -- 172.21.15.167:0/1564520155 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x5570aa22b450 con 0x5570aa227f30 2023-10-05T19:30:22.998 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.994+0000 7fd0ed711e40 20 check_secure_mon_conn(): auth registy supported: methods=[2] modes=[2,1] 2023-10-05T19:30:22.998 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.994+0000 7fd0ed711e40 20 check_secure_mon_conn(): mode 1 is insecure 2023-10-05T19:30:22.998 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.994+0000 7fd0ed711e40 1 Processor -- start 2023-10-05T19:30:22.998 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.994+0000 7fd0e37fe700 1 -- 172.21.15.167:0/1564520155 <== mon.0 v2:172.21.15.118:3300/0 4 ==== mgrmap(e 5) v1 ==== 40329+0+0 (secure 0 0 0) 0x7fd0dc01a920 con 0x5570aa227f30 2023-10-05T19:30:22.999 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.994+0000 7fd0ed711e40 1 -- start start 2023-10-05T19:30:22.999 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.994+0000 7fd0ed711e40 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5570aa23b3e0 0x5570aa23b7b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-05T19:30:22.999 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.994+0000 7fd0ed711e40 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x5570aa23bcf0 con 0x5570aa23b3e0 2023-10-05T19:30:23.000 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.994+0000 7fd0eba47700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5570aa23b3e0 0x5570aa23b7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-05T19:30:23.000 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.994+0000 7fd0eba47700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5570aa23b3e0 0x5570aa23b7b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.118:3300/0 says I am v2:172.21.15.167:33204/0 (socket says 172.21.15.167:33204) 2023-10-05T19:30:23.000 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.994+0000 7fd0e37fe700 1 --2- 172.21.15.167:0/1564520155 >> [v2:172.21.15.167:6824/29889,v1:172.21.15.167:6825/29889] conn(0x7fd0d0030c50 0x7fd0d0033100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-05T19:30:23.000 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.994+0000 7fd0eba47700 1 -- 172.21.15.167:0/2097080358 learned_addr learned my addr 172.21.15.167:0/2097080358 (peer_addr_for_me v2:172.21.15.167:0/0) 2023-10-05T19:30:23.000 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.994+0000 7fd0e37fe700 1 -- 172.21.15.167:0/1564520155 <== mon.0 v2:172.21.15.118:3300/0 5 ==== osd_map(32..32 src has 1..32) v4 ==== 6260+0+0 (secure 0 0 0) 0x7fd0dc045220 con 0x5570aa227f30 2023-10-05T19:30:23.000 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.994+0000 7fd0eb246700 1 --2- 172.21.15.167:0/1564520155 >> [v2:172.21.15.167:6824/29889,v1:172.21.15.167:6825/29889] conn(0x7fd0d0030c50 0x7fd0d0033100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-05T19:30:23.000 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.994+0000 7fd0eba47700 1 -- 172.21.15.167:0/2097080358 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5570aa23be30 con 0x5570aa23b3e0 2023-10-05T19:30:23.001 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.998+0000 7fd0eba47700 1 --2- 172.21.15.167:0/2097080358 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5570aa23b3e0 0x5570aa23b7b0 secure :-1 s=READY pgs=690 cs=0 l=1 rev1=1 crypto rx=0x7fd0e400a320 tx=0x7fd0e40059d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7a0f2783099b3a50 server_cookie=0 in_seq=0 out_seq=0 2023-10-05T19:30:23.001 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.998+0000 7fd0eb246700 1 --2- 172.21.15.167:0/1564520155 >> [v2:172.21.15.167:6824/29889,v1:172.21.15.167:6825/29889] conn(0x7fd0d0030c50 0x7fd0d0033100 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7fd0dc017020 tx=0x7fd0dc043490 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-05T19:30:23.001 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.998+0000 7fd0bbfef700 1 -- 172.21.15.167:0/2097080358 <== mon.0 v2:172.21.15.118:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fd0e400ae30 con 0x5570aa23b3e0 2023-10-05T19:30:23.001 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.998+0000 7fd0bbfef700 1 -- 172.21.15.167:0/2097080358 <== mon.0 v2:172.21.15.118:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd0e400c610 con 0x5570aa23b3e0 2023-10-05T19:30:23.001 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.998+0000 7fd0bbfef700 1 -- 172.21.15.167:0/2097080358 <== mon.0 v2:172.21.15.118:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fd0e400c7c0 con 0x5570aa23b3e0 2023-10-05T19:30:23.002 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.998+0000 7fd0ed711e40 1 -- 172.21.15.167:0/2097080358 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5570aa23b3e0 msgr2=0x5570aa23b7b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-05T19:30:23.002 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.998+0000 7fd0ed711e40 1 --2- 172.21.15.167:0/2097080358 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5570aa23b3e0 0x5570aa23b7b0 secure :-1 s=READY pgs=690 cs=0 l=1 rev1=1 crypto rx=0x7fd0e400a320 tx=0x7fd0e40059d0 comp rx=0 tx=0).stop 2023-10-05T19:30:23.002 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.998+0000 7fd0ed711e40 1 -- 172.21.15.167:0/2097080358 shutdown_connections 2023-10-05T19:30:23.002 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.998+0000 7fd0ed711e40 1 --2- 172.21.15.167:0/2097080358 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5570aa23b3e0 0x5570aa23b7b0 unknown :-1 s=CLOSED pgs=690 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-05T19:30:23.002 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.998+0000 7fd0ed711e40 1 -- 172.21.15.167:0/2097080358 >> 172.21.15.167:0/2097080358 conn(0x5570aa2367a0 msgr2=0x5570aa2349e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-05T19:30:23.002 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.998+0000 7fd0ed711e40 1 -- 172.21.15.167:0/2097080358 shutdown_connections 2023-10-05T19:30:23.003 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.998+0000 7fd0ed711e40 1 -- 172.21.15.167:0/2097080358 wait complete. 2023-10-05T19:30:23.003 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.998+0000 7fd0ed711e40 1 Processor -- start 2023-10-05T19:30:23.003 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.998+0000 7fd0ed711e40 1 -- start start 2023-10-05T19:30:23.003 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.998+0000 7fd0ed711e40 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5570aa23b3e0 0x5570aa240f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-05T19:30:23.003 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.998+0000 7fd0ed711e40 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x5570aa241f80 con 0x5570aa23b3e0 2023-10-05T19:30:23.003 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.998+0000 7fd0eba47700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5570aa23b3e0 0x5570aa240f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-05T19:30:23.003 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.998+0000 7fd0eba47700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5570aa23b3e0 0x5570aa240f80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.118:3300/0 says I am v2:172.21.15.167:33216/0 (socket says 172.21.15.167:33216) 2023-10-05T19:30:23.003 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.998+0000 7fd0eba47700 1 -- 172.21.15.167:0/614670754 learned_addr learned my addr 172.21.15.167:0/614670754 (peer_addr_for_me v2:172.21.15.167:0/0) 2023-10-05T19:30:23.003 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.998+0000 7fd0eba47700 1 -- 172.21.15.167:0/614670754 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd0e4005c90 con 0x5570aa23b3e0 2023-10-05T19:30:23.004 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.998+0000 7fd0eba47700 1 --2- 172.21.15.167:0/614670754 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5570aa23b3e0 0x5570aa240f80 secure :-1 s=READY pgs=691 cs=0 l=1 rev1=1 crypto rx=0x5570aa235550 tx=0x7fd0e4013000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-05T19:30:23.004 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.998+0000 7fd0ba7ec700 1 -- 172.21.15.167:0/614670754 <== mon.0 v2:172.21.15.118:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fd0e4015780 con 0x5570aa23b3e0 2023-10-05T19:30:23.004 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.998+0000 7fd0ba7ec700 1 -- 172.21.15.167:0/614670754 <== mon.0 v2:172.21.15.118:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd0e40158e0 con 0x5570aa23b3e0 2023-10-05T19:30:23.004 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.998+0000 7fd0ed711e40 1 -- 172.21.15.167:0/614670754 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x5570aa241580 con 0x5570aa23b3e0 2023-10-05T19:30:23.004 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.998+0000 7fd0ba7ec700 1 -- 172.21.15.167:0/614670754 <== mon.0 v2:172.21.15.118:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fd0e4015bd0 con 0x5570aa23b3e0 2023-10-05T19:30:23.004 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:22.998+0000 7fd0ed711e40 1 -- 172.21.15.167:0/614670754 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x5570aa241a40 con 0x5570aa23b3e0 2023-10-05T19:30:23.005 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:23.002+0000 7fd033fff700 20 reqs_thread_entry: start 2023-10-05T19:30:23.005 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:23.002+0000 7fd0ed711e40 20 try_to_resolve_local_entities(): before: realm_id= realm_name= zonegroup_id= zonegroup_name= zone_id= zone_name= 2023-10-05T19:30:23.006 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:23.002+0000 7fd0ed711e40 20 rados->read ofs=0 len=0 2023-10-05T19:30:23.006 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:23.002+0000 7fd0ed711e40 1 --2- 172.21.15.167:0/1564520155 >> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] conn(0x5570aa25ed20 0x5570aa25f1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-05T19:30:23.006 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:23.002+0000 7fd0ed711e40 1 -- 172.21.15.167:0/1564520155 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:1 5.12 5:49953fa1:::default.realm:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32) v8 -- 0x5570aa25f6e0 con 0x5570aa25ed20 2023-10-05T19:30:23.006 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:23.002+0000 7fd0ba7ec700 1 -- 172.21.15.167:0/614670754 <== mon.0 v2:172.21.15.118:3300/0 4 ==== mgrmap(e 5) v1 ==== 40329+0+0 (secure 0 0 0) 0x7fd0e4007050 con 0x5570aa23b3e0 2023-10-05T19:30:23.006 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:23.002+0000 7fd0eaa45700 1 --2- 172.21.15.167:0/1564520155 >> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] conn(0x5570aa25ed20 0x5570aa25f1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-05T19:30:23.006 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:23.002+0000 7fd0ba7ec700 1 --2- 172.21.15.167:0/614670754 >> [v2:172.21.15.167:6824/29889,v1:172.21.15.167:6825/29889] conn(0x7fd038030c00 0x7fd0380330b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-05T19:30:23.006 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:23.002+0000 7fd0ba7ec700 1 -- 172.21.15.167:0/614670754 <== mon.0 v2:172.21.15.118:3300/0 5 ==== osd_map(32..32 src has 1..32) v4 ==== 6260+0+0 (secure 0 0 0) 0x7fd0e40442e0 con 0x5570aa23b3e0 2023-10-05T19:30:23.006 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:23.002+0000 7fd0eaa45700 1 --2- 172.21.15.167:0/1564520155 >> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] conn(0x5570aa25ed20 0x5570aa25f1a0 crc :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-05T19:30:23.006 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:23.002+0000 7fd0eba47700 1 --2- 172.21.15.167:0/614670754 >> [v2:172.21.15.167:6824/29889,v1:172.21.15.167:6825/29889] conn(0x7fd038030c00 0x7fd0380330b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-05T19:30:23.007 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:23.002+0000 7fd0eaa45700 1 -- 172.21.15.167:0/1564520155 <== osd.1 v2:172.21.15.167:6800/30035 1 ==== osd_op_reply(1 default.realm [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 ==== 157+0+0 (crc 0 0 0) 0x7fd0d400f860 con 0x5570aa25ed20 2023-10-05T19:30:23.007 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:23.002+0000 7fd0ed711e40 20 rados_obj.operate() r=-2 bl.length=0 2023-10-05T19:30:23.007 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:23.002+0000 7fd0ed711e40 20 rados->read ofs=0 len=0 2023-10-05T19:30:23.007 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:23.002+0000 7fd0ed711e40 1 -- 172.21.15.167:0/1564520155 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:2 5.f 5:f43b6ece:::zone_names.default:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32) v8 -- 0x5570aa272260 con 0x5570aa25ed20 2023-10-05T19:30:23.007 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:23.002+0000 7fd0eba47700 1 --2- 172.21.15.167:0/614670754 >> [v2:172.21.15.167:6824/29889,v1:172.21.15.167:6825/29889] conn(0x7fd038030c00 0x7fd0380330b0 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7fd0e400af50 tx=0x7fd0e40424d0 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-05T19:30:23.010 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:23.006+0000 7fd0eaa45700 1 -- 172.21.15.167:0/1564520155 <== osd.1 v2:172.21.15.167:6800/30035 2 ==== osd_op_reply(2 zone_names.default [read 0~46 out=46b] v0'0 uv3 ondisk = 0) v8 ==== 162+0+46 (crc 0 0 0) 0x7fd0d400f860 con 0x5570aa25ed20 2023-10-05T19:30:23.010 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:23.006+0000 7fd0ed711e40 20 rados_obj.operate() r=0 bl.length=46 2023-10-05T19:30:23.010 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:23.006+0000 7fd0ed711e40 20 rados->read ofs=0 len=0 2023-10-05T19:30:23.010 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:23.006+0000 7fd0ed711e40 1 -- 172.21.15.167:0/1564520155 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:3 5.1d 5:b8374eb4:::zone_info.9a9df965-6612-45d4-a45f-6e3c889bacc2:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32) v8 -- 0x5570aa261500 con 0x5570aa25ed20 2023-10-05T19:30:23.011 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:23.006+0000 7fd0eaa45700 1 -- 172.21.15.167:0/1564520155 <== osd.1 v2:172.21.15.167:6800/30035 3 ==== osd_op_reply(3 zone_info.9a9df965-6612-45d4-a45f-6e3c889bacc2 [read 0~1036 out=1036b] v0'0 uv3 ondisk = 0) v8 ==== 190+0+1036 (crc 0 0 0) 0x7fd0d400f860 con 0x5570aa25ed20 2023-10-05T19:30:23.011 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:23.006+0000 7fd0ed711e40 20 rados_obj.operate() r=0 bl.length=1036 2023-10-05T19:30:23.011 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:23.006+0000 7fd0ed711e40 1 --2- 172.21.15.167:0/1564520155 >> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] conn(0x5570aa273b30 0x5570aa275f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-05T19:30:23.011 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:23.006+0000 7fd0ed711e40 1 -- 172.21.15.167:0/1564520155 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:4 5.0 5:00000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x5570aa276450 con 0x5570aa273b30 2023-10-05T19:30:23.011 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:23.006+0000 7fd0eb246700 1 --2- 172.21.15.167:0/1564520155 >> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] conn(0x5570aa273b30 0x5570aa275f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-05T19:30:23.011 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:23.006+0000 7fd0eb246700 1 --2- 172.21.15.167:0/1564520155 >> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] conn(0x5570aa273b30 0x5570aa275f10 crc :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-05T19:30:23.012 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:23.010+0000 7fd0eb246700 1 -- 172.21.15.167:0/1564520155 <== osd.3 v2:172.21.15.167:6808/30036 1 ==== osd_op_reply(4 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fd0dc014070 con 0x5570aa273b30 2023-10-05T19:30:23.012 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:23.010+0000 7fd0eb246700 1 --2- 172.21.15.167:0/1564520155 >> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] conn(0x7fd0dc037040 0x7fd0dc044650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-05T19:30:23.012 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:23.010+0000 7fd0eb246700 1 -- 172.21.15.167:0/1564520155 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:5 5.10 5:08000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x5570aa276450 con 0x7fd0dc037040 2023-10-05T19:30:23.013 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:23.010+0000 7fd0eaa45700 1 --2- 172.21.15.167:0/1564520155 >> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] conn(0x7fd0dc037040 0x7fd0dc044650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-05T19:30:23.013 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:23.010+0000 7fd0eaa45700 1 --2- 172.21.15.167:0/1564520155 >> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] conn(0x7fd0dc037040 0x7fd0dc044650 crc :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-05T19:30:23.014 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:23.010+0000 7fd0eaa45700 1 -- 172.21.15.167:0/1564520155 <== osd.2 v2:172.21.15.167:6816/30037 1 ==== osd_op_reply(5 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fd0d400f860 con 0x7fd0dc037040 2023-10-05T19:30:23.014 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:23.010+0000 7fd0eaa45700 1 --2- 172.21.15.167:0/1564520155 >> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] conn(0x5570aa2065c0 0x7fd0d4010420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-05T19:30:23.014 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:23.010+0000 7fd0eaa45700 1 -- 172.21.15.167:0/1564520155 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:6 5.8 5:10000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x5570aa276450 con 0x5570aa2065c0 2023-10-05T19:30:23.014 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:23.010+0000 7fd0eba47700 1 --2- 172.21.15.167:0/1564520155 >> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] conn(0x5570aa2065c0 0x7fd0d4010420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-05T19:30:23.015 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:23.010+0000 7fd0eba47700 1 --2- 172.21.15.167:0/1564520155 >> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] conn(0x5570aa2065c0 0x7fd0d4010420 crc :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-05T19:30:23.016 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:23.014+0000 7fd0eba47700 1 -- 172.21.15.167:0/1564520155 <== osd.0 v2:172.21.15.118:6800/27690 1 ==== osd_op_reply(6 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fd0e4042240 con 0x5570aa2065c0 2023-10-05T19:30:23.016 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:23.014+0000 7fd0eba47700 1 -- 172.21.15.167:0/1564520155 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:7 5.18 5:18000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x5570aa276450 con 0x5570aa25ed20 2023-10-05T19:30:23.017 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:23.014+0000 7fd0eaa45700 1 -- 172.21.15.167:0/1564520155 <== osd.1 v2:172.21.15.167:6800/30035 4 ==== osd_op_reply(7 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fd0d400f860 con 0x5570aa25ed20 2023-10-05T19:30:23.017 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:23.014+0000 7fd0eaa45700 1 -- 172.21.15.167:0/1564520155 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:8 5.4 5:20000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x5570aa276450 con 0x5570aa25ed20 2023-10-05T19:30:23.018 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:23.014+0000 7fd0eaa45700 1 -- 172.21.15.167:0/1564520155 <== osd.1 v2:172.21.15.167:6800/30035 5 ==== osd_op_reply(8 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fd0d400f860 con 0x5570aa25ed20 2023-10-05T19:30:23.018 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:23.014+0000 7fd0eaa45700 1 -- 172.21.15.167:0/1564520155 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:9 5.14 5:28000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x5570aa276450 con 0x5570aa2065c0 2023-10-05T19:30:23.019 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:23.014+0000 7fd0eba47700 1 -- 172.21.15.167:0/1564520155 <== osd.0 v2:172.21.15.118:6800/27690 2 ==== osd_op_reply(9 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fd0e4042240 con 0x5570aa2065c0 2023-10-05T19:30:23.019 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:23.014+0000 7fd0eba47700 1 -- 172.21.15.167:0/1564520155 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:10 5.c 5:30000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x5570aa276450 con 0x5570aa2065c0 2023-10-05T19:30:23.020 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:23.018+0000 7fd0eba47700 1 -- 172.21.15.167:0/1564520155 <== osd.0 v2:172.21.15.118:6800/27690 3 ==== osd_op_reply(10 [pgnls start_epoch 32 out=112b] v23'1 uv1 ondisk = 1) v8 ==== 144+0+112 (crc 0 0 0) 0x7fd0e4042240 con 0x5570aa2065c0 2023-10-05T19:30:23.020 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:23.018+0000 7fd0eba47700 1 -- 172.21.15.167:0/1564520155 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:11 5.1c 5:38000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x5570aa276450 con 0x7fd0dc037040 2023-10-05T19:30:23.021 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:23.018+0000 7fd0eaa45700 1 -- 172.21.15.167:0/1564520155 <== osd.2 v2:172.21.15.167:6816/30037 2 ==== osd_op_reply(11 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fd0d400f860 con 0x7fd0dc037040 2023-10-05T19:30:23.021 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:23.018+0000 7fd0eaa45700 1 -- 172.21.15.167:0/1564520155 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:12 5.2 5:40000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x5570aa276450 con 0x5570aa273b30 2023-10-05T19:30:23.021 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:23.018+0000 7fd0eb246700 1 -- 172.21.15.167:0/1564520155 <== osd.3 v2:172.21.15.167:6808/30036 2 ==== osd_op_reply(12 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fd0dc042040 con 0x5570aa273b30 2023-10-05T19:30:23.022 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:23.018+0000 7fd0eb246700 1 -- 172.21.15.167:0/1564520155 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:13 5.12 5:48000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x5570aa276450 con 0x5570aa25ed20 2023-10-05T19:30:23.022 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:23.018+0000 7fd0eaa45700 1 -- 172.21.15.167:0/1564520155 <== osd.1 v2:172.21.15.167:6800/30035 6 ==== osd_op_reply(13 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fd0d400f860 con 0x5570aa25ed20 2023-10-05T19:30:23.022 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:23.018+0000 7fd0eaa45700 1 -- 172.21.15.167:0/1564520155 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:14 5.a 5:50000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x5570aa276450 con 0x5570aa2065c0 2023-10-05T19:30:23.023 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:23.018+0000 7fd0eba47700 1 -- 172.21.15.167:0/1564520155 <== osd.0 v2:172.21.15.118:6800/27690 4 ==== osd_op_reply(14 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fd0e4042240 con 0x5570aa2065c0 2023-10-05T19:30:23.023 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:23.018+0000 7fd0eba47700 1 -- 172.21.15.167:0/1564520155 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:15 5.1a 5:58000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x5570aa276450 con 0x5570aa25ed20 2023-10-05T19:30:23.024 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:23.022+0000 7fd0eaa45700 1 -- 172.21.15.167:0/1564520155 <== osd.1 v2:172.21.15.167:6800/30035 7 ==== osd_op_reply(15 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fd0d400f860 con 0x5570aa25ed20 2023-10-05T19:30:23.024 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:23.022+0000 7fd0eaa45700 1 -- 172.21.15.167:0/1564520155 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:16 5.6 5:60000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x5570aa276450 con 0x7fd0dc037040 2023-10-05T19:30:23.025 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:23.022+0000 7fd0eaa45700 1 -- 172.21.15.167:0/1564520155 <== osd.2 v2:172.21.15.167:6816/30037 3 ==== osd_op_reply(16 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fd0d400f860 con 0x7fd0dc037040 2023-10-05T19:30:23.025 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:23.022+0000 7fd0eaa45700 1 -- 172.21.15.167:0/1564520155 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:17 5.16 5:68000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x5570aa276450 con 0x5570aa273b30 2023-10-05T19:30:23.026 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:23.022+0000 7fd0eb246700 1 -- 172.21.15.167:0/1564520155 <== osd.3 v2:172.21.15.167:6808/30036 3 ==== osd_op_reply(17 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fd0dc042040 con 0x5570aa273b30 2023-10-05T19:30:23.026 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:23.022+0000 7fd0eb246700 1 -- 172.21.15.167:0/1564520155 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:18 5.e 5:70000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x5570aa276450 con 0x5570aa273b30 2023-10-05T19:30:23.027 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:23.022+0000 7fd0eb246700 1 -- 172.21.15.167:0/1564520155 <== osd.3 v2:172.21.15.167:6808/30036 4 ==== osd_op_reply(18 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fd0dc042040 con 0x5570aa273b30 2023-10-05T19:30:23.027 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:23.022+0000 7fd0eb246700 1 -- 172.21.15.167:0/1564520155 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:19 5.1e 5:78000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x5570aa276450 con 0x5570aa2065c0 2023-10-05T19:30:23.028 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:23.026+0000 7fd0eba47700 1 -- 172.21.15.167:0/1564520155 <== osd.0 v2:172.21.15.118:6800/27690 5 ==== osd_op_reply(19 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fd0e4042240 con 0x5570aa2065c0 2023-10-05T19:30:23.028 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:23.026+0000 7fd0eba47700 1 -- 172.21.15.167:0/1564520155 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:20 5.1 5:80000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x5570aa276450 con 0x5570aa273b30 2023-10-05T19:30:23.029 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:23.026+0000 7fd0eb246700 1 -- 172.21.15.167:0/1564520155 <== osd.3 v2:172.21.15.167:6808/30036 5 ==== osd_op_reply(20 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fd0dc042040 con 0x5570aa273b30 2023-10-05T19:30:23.029 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:23.026+0000 7fd0eb246700 1 -- 172.21.15.167:0/1564520155 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:21 5.11 5:88000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x5570aa276450 con 0x5570aa273b30 2023-10-05T19:30:23.029 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:23.026+0000 7fd0eb246700 1 -- 172.21.15.167:0/1564520155 <== osd.3 v2:172.21.15.167:6808/30036 6 ==== osd_op_reply(21 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fd0dc042040 con 0x5570aa273b30 2023-10-05T19:30:23.029 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:23.026+0000 7fd0eb246700 1 -- 172.21.15.167:0/1564520155 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:22 5.9 5:90000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x5570aa276450 con 0x5570aa25ed20 2023-10-05T19:30:23.030 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:23.026+0000 7fd0eaa45700 1 -- 172.21.15.167:0/1564520155 <== osd.1 v2:172.21.15.167:6800/30035 8 ==== osd_op_reply(22 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fd0d400f860 con 0x5570aa25ed20 2023-10-05T19:30:23.030 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:23.026+0000 7fd0eaa45700 1 -- 172.21.15.167:0/1564520155 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:23 5.19 5:98000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x5570aa276450 con 0x5570aa25ed20 2023-10-05T19:30:23.031 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:23.026+0000 7fd0eaa45700 1 -- 172.21.15.167:0/1564520155 <== osd.1 v2:172.21.15.167:6800/30035 9 ==== osd_op_reply(23 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fd0d400f860 con 0x5570aa25ed20 2023-10-05T19:30:23.031 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:23.026+0000 7fd0eaa45700 1 -- 172.21.15.167:0/1564520155 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:24 5.5 5:a0000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x5570aa276450 con 0x5570aa2065c0 2023-10-05T19:30:23.032 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:23.030+0000 7fd0eba47700 1 -- 172.21.15.167:0/1564520155 <== osd.0 v2:172.21.15.118:6800/27690 6 ==== osd_op_reply(24 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fd0e4042240 con 0x5570aa2065c0 2023-10-05T19:30:23.032 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:23.030+0000 7fd0eba47700 1 -- 172.21.15.167:0/1564520155 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:25 5.15 5:a8000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x5570aa276450 con 0x5570aa273b30 2023-10-05T19:30:23.033 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:23.030+0000 7fd0eb246700 1 -- 172.21.15.167:0/1564520155 <== osd.3 v2:172.21.15.167:6808/30036 7 ==== osd_op_reply(25 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fd0dc042040 con 0x5570aa273b30 2023-10-05T19:30:23.033 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:23.030+0000 7fd0eb246700 1 -- 172.21.15.167:0/1564520155 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:26 5.d 5:b0000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x5570aa276450 con 0x7fd0dc037040 2023-10-05T19:30:23.034 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:23.030+0000 7fd0eaa45700 1 -- 172.21.15.167:0/1564520155 <== osd.2 v2:172.21.15.167:6816/30037 4 ==== osd_op_reply(26 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fd0d400f860 con 0x7fd0dc037040 2023-10-05T19:30:23.034 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:23.030+0000 7fd0eaa45700 1 -- 172.21.15.167:0/1564520155 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:27 5.1d 5:b8000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x5570aa276450 con 0x5570aa25ed20 2023-10-05T19:30:23.035 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:23.030+0000 7fd0eaa45700 1 -- 172.21.15.167:0/1564520155 <== osd.1 v2:172.21.15.167:6800/30035 10 ==== osd_op_reply(27 [pgnls start_epoch 32 out=107b] v32'3 uv3 ondisk = 1) v8 ==== 144+0+107 (crc 0 0 0) 0x7fd0d400f860 con 0x5570aa25ed20 2023-10-05T19:30:23.035 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:23.030+0000 7fd0eaa45700 1 -- 172.21.15.167:0/1564520155 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:28 5.3 5:c0000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x5570aa276450 con 0x5570aa2065c0 2023-10-05T19:30:23.036 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:23.034+0000 7fd0eba47700 1 -- 172.21.15.167:0/1564520155 <== osd.0 v2:172.21.15.118:6800/27690 7 ==== osd_op_reply(28 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fd0e4042240 con 0x5570aa2065c0 2023-10-05T19:30:23.036 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:23.034+0000 7fd0eba47700 1 -- 172.21.15.167:0/1564520155 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:29 5.13 5:c8000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x5570aa276450 con 0x5570aa273b30 2023-10-05T19:30:23.036 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:23.034+0000 7fd0eb246700 1 -- 172.21.15.167:0/1564520155 <== osd.3 v2:172.21.15.167:6808/30036 8 ==== osd_op_reply(29 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fd0dc042040 con 0x5570aa273b30 2023-10-05T19:30:23.036 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:23.034+0000 7fd0eb246700 1 -- 172.21.15.167:0/1564520155 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:30 5.b 5:d0000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x5570aa276450 con 0x5570aa2065c0 2023-10-05T19:30:23.037 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:23.034+0000 7fd0eba47700 1 -- 172.21.15.167:0/1564520155 <== osd.0 v2:172.21.15.118:6800/27690 8 ==== osd_op_reply(30 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fd0e4042240 con 0x5570aa2065c0 2023-10-05T19:30:23.037 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:23.034+0000 7fd0eba47700 1 -- 172.21.15.167:0/1564520155 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:31 5.1b 5:d8000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x5570aa276450 con 0x7fd0dc037040 2023-10-05T19:30:23.038 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:23.034+0000 7fd0eaa45700 1 -- 172.21.15.167:0/1564520155 <== osd.2 v2:172.21.15.167:6816/30037 5 ==== osd_op_reply(31 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fd0d400f860 con 0x7fd0dc037040 2023-10-05T19:30:23.038 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:23.034+0000 7fd0eaa45700 1 -- 172.21.15.167:0/1564520155 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:32 5.7 5:e0000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x5570aa276450 con 0x5570aa273b30 2023-10-05T19:30:23.039 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:23.034+0000 7fd0eb246700 1 -- 172.21.15.167:0/1564520155 <== osd.3 v2:172.21.15.167:6808/30036 9 ==== osd_op_reply(32 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fd0dc042040 con 0x5570aa273b30 2023-10-05T19:30:23.039 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:23.034+0000 7fd0eb246700 1 -- 172.21.15.167:0/1564520155 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:33 5.17 5:e8000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x5570aa276450 con 0x5570aa2065c0 2023-10-05T19:30:23.040 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:23.038+0000 7fd0eba47700 1 -- 172.21.15.167:0/1564520155 <== osd.0 v2:172.21.15.118:6800/27690 9 ==== osd_op_reply(33 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fd0e4042240 con 0x5570aa2065c0 2023-10-05T19:30:23.040 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:23.038+0000 7fd0eba47700 1 -- 172.21.15.167:0/1564520155 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:34 5.f 5:f0000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x5570aa276450 con 0x5570aa25ed20 2023-10-05T19:30:23.041 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:23.038+0000 7fd0eaa45700 1 -- 172.21.15.167:0/1564520155 <== osd.1 v2:172.21.15.167:6800/30035 11 ==== osd_op_reply(34 [pgnls start_epoch 32 out=115b] v32'3 uv3 ondisk = 1) v8 ==== 144+0+115 (crc 0 0 0) 0x7fd0d400f860 con 0x5570aa25ed20 2023-10-05T19:30:23.041 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:23.038+0000 7fd0eaa45700 1 -- 172.21.15.167:0/1564520155 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:35 5.1f 5:f8000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x5570aa276450 con 0x7fd0dc037040 2023-10-05T19:30:23.042 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:23.038+0000 7fd0eaa45700 1 -- 172.21.15.167:0/1564520155 <== osd.2 v2:172.21.15.167:6816/30037 6 ==== osd_op_reply(35 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7fd0d400f860 con 0x7fd0dc037040 2023-10-05T19:30:23.042 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:23.038+0000 7fd0ed711e40 20 RGWRados::pool_iterate: got zonegroup_info.4f1f719d-9648-417b-b10c-e5d939825419 2023-10-05T19:30:23.043 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:23.038+0000 7fd0ed711e40 20 RGWRados::pool_iterate: got zone_info.9a9df965-6612-45d4-a45f-6e3c889bacc2 2023-10-05T19:30:23.043 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:23.038+0000 7fd0ed711e40 20 RGWRados::pool_iterate: got zone_names.default 2023-10-05T19:30:23.043 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:23.038+0000 7fd0ed711e40 20 RGWRados::pool_iterate: got zonegroups_names.default 2023-10-05T19:30:23.043 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:23.038+0000 7fd0ed711e40 20 rados->read ofs=0 len=0 2023-10-05T19:30:23.043 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:23.038+0000 7fd0ed711e40 1 -- 172.21.15.167:0/1564520155 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:36 5.1d 5:b8374eb4:::zone_info.9a9df965-6612-45d4-a45f-6e3c889bacc2:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32) v8 -- 0x5570aa272610 con 0x5570aa25ed20 2023-10-05T19:30:23.043 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:23.038+0000 7fd0eaa45700 1 -- 172.21.15.167:0/1564520155 <== osd.1 v2:172.21.15.167:6800/30035 12 ==== osd_op_reply(36 zone_info.9a9df965-6612-45d4-a45f-6e3c889bacc2 [read 0~1036 out=1036b] v0'0 uv3 ondisk = 0) v8 ==== 190+0+1036 (crc 0 0 0) 0x7fd0d400f860 con 0x5570aa25ed20 2023-10-05T19:30:23.043 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:23.038+0000 7fd0ed711e40 20 rados_obj.operate() r=0 bl.length=1036 2023-10-05T19:30:23.045 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:23.042+0000 7fd0ed711e40 1 -- 172.21.15.167:0/1564520155 >> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] conn(0x5570aa2065c0 msgr2=0x7fd0d4010420 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-05T19:30:23.045 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:23.042+0000 7fd0ed711e40 1 --2- 172.21.15.167:0/1564520155 >> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] conn(0x5570aa2065c0 0x7fd0d4010420 crc :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-05T19:30:23.046 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:23.042+0000 7fd0ed711e40 1 -- 172.21.15.167:0/1564520155 >> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] conn(0x5570aa25ed20 msgr2=0x5570aa25f1a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-05T19:30:23.046 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:23.042+0000 7fd0ed711e40 1 --2- 172.21.15.167:0/1564520155 >> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] conn(0x5570aa25ed20 0x5570aa25f1a0 crc :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-05T19:30:23.046 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:23.042+0000 7fd0ed711e40 1 -- 172.21.15.167:0/1564520155 >> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] conn(0x7fd0dc037040 msgr2=0x7fd0dc044650 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-05T19:30:23.046 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:23.042+0000 7fd0ed711e40 1 --2- 172.21.15.167:0/1564520155 >> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] conn(0x7fd0dc037040 0x7fd0dc044650 crc :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-05T19:30:23.046 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:23.042+0000 7fd0ed711e40 1 -- 172.21.15.167:0/1564520155 >> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] conn(0x5570aa273b30 msgr2=0x5570aa275f10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-05T19:30:23.046 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:23.042+0000 7fd0ed711e40 1 --2- 172.21.15.167:0/1564520155 >> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] conn(0x5570aa273b30 0x5570aa275f10 crc :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-05T19:30:23.046 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:23.042+0000 7fd0ed711e40 1 -- 172.21.15.167:0/1564520155 >> [v2:172.21.15.167:6824/29889,v1:172.21.15.167:6825/29889] conn(0x7fd0d0030c50 msgr2=0x7fd0d0033100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-05T19:30:23.046 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:23.042+0000 7fd0ed711e40 1 --2- 172.21.15.167:0/1564520155 >> [v2:172.21.15.167:6824/29889,v1:172.21.15.167:6825/29889] conn(0x7fd0d0030c50 0x7fd0d0033100 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7fd0dc017020 tx=0x7fd0dc043490 comp rx=0 tx=0).stop 2023-10-05T19:30:23.046 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:23.042+0000 7fd0ed711e40 1 -- 172.21.15.167:0/1564520155 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5570aa227f30 msgr2=0x5570aa228300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-05T19:30:23.046 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:23.042+0000 7fd0ed711e40 1 --2- 172.21.15.167:0/1564520155 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5570aa227f30 0x5570aa228300 secure :-1 s=READY pgs=689 cs=0 l=1 rev1=1 crypto rx=0x7fd0dc000c00 tx=0x7fd0dc016000 comp rx=0 tx=0).stop 2023-10-05T19:30:23.047 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:23.042+0000 7fd0eba47700 1 -- 172.21.15.167:0/1564520155 reap_dead start 2023-10-05T19:30:23.047 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:23.042+0000 7fd0ed711e40 1 -- 172.21.15.167:0/1564520155 shutdown_connections 2023-10-05T19:30:23.047 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:23.042+0000 7fd0ed711e40 1 -- 172.21.15.167:0/1564520155 >> 172.21.15.167:0/1564520155 conn(0x5570aa175830 msgr2=0x5570aa21b650 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-05T19:30:23.047 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:23.042+0000 7fd0ed711e40 1 -- 172.21.15.167:0/1564520155 shutdown_connections 2023-10-05T19:30:23.047 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:23.042+0000 7fd0ed711e40 1 -- 172.21.15.167:0/1564520155 wait complete. 2023-10-05T19:30:23.048 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:23.042+0000 7fd0ed711e40 1 -- 172.21.15.167:0/614670754 >> [v2:172.21.15.167:6824/29889,v1:172.21.15.167:6825/29889] conn(0x7fd038030c00 msgr2=0x7fd0380330b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-05T19:30:23.048 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:23.042+0000 7fd0ed711e40 1 --2- 172.21.15.167:0/614670754 >> [v2:172.21.15.167:6824/29889,v1:172.21.15.167:6825/29889] conn(0x7fd038030c00 0x7fd0380330b0 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7fd0e400af50 tx=0x7fd0e40424d0 comp rx=0 tx=0).stop 2023-10-05T19:30:23.048 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:23.042+0000 7fd0ed711e40 1 -- 172.21.15.167:0/614670754 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5570aa23b3e0 msgr2=0x5570aa240f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-05T19:30:23.048 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:23.042+0000 7fd0ed711e40 1 --2- 172.21.15.167:0/614670754 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5570aa23b3e0 0x5570aa240f80 secure :-1 s=READY pgs=691 cs=0 l=1 rev1=1 crypto rx=0x5570aa235550 tx=0x7fd0e4013000 comp rx=0 tx=0).stop 2023-10-05T19:30:23.048 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:23.042+0000 7fd0ed711e40 1 -- 172.21.15.167:0/614670754 shutdown_connections 2023-10-05T19:30:23.048 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:23.042+0000 7fd0ed711e40 1 --2- 172.21.15.167:0/614670754 >> [v2:172.21.15.167:6824/29889,v1:172.21.15.167:6825/29889] conn(0x7fd038030c00 0x7fd0380330b0 unknown :-1 s=CLOSED pgs=237 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-05T19:30:23.049 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:23.042+0000 7fd0ed711e40 1 --2- 172.21.15.167:0/614670754 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5570aa23b3e0 0x5570aa240f80 unknown :-1 s=CLOSED pgs=691 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-05T19:30:23.049 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:23.042+0000 7fd0ed711e40 1 -- 172.21.15.167:0/614670754 >> 172.21.15.167:0/614670754 conn(0x5570aa2367a0 msgr2=0x5570aa234920 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-05T19:30:23.049 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:23.042+0000 7fd0ed711e40 1 -- 172.21.15.167:0/614670754 shutdown_connections 2023-10-05T19:30:23.049 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:23.042+0000 7fd0ed711e40 1 -- 172.21.15.167:0/614670754 wait complete. 2023-10-05T19:30:23.050 INFO:teuthology.orchestra.run.smithi167.stdout:{ 2023-10-05T19:30:23.050 INFO:teuthology.orchestra.run.smithi167.stdout: "id": "9a9df965-6612-45d4-a45f-6e3c889bacc2", 2023-10-05T19:30:23.050 INFO:teuthology.orchestra.run.smithi167.stdout: "name": "default", 2023-10-05T19:30:23.050 INFO:teuthology.orchestra.run.smithi167.stdout: "domain_root": "default.rgw.meta:root", 2023-10-05T19:30:23.050 INFO:teuthology.orchestra.run.smithi167.stdout: "control_pool": "default.rgw.control", 2023-10-05T19:30:23.050 INFO:teuthology.orchestra.run.smithi167.stdout: "gc_pool": "default.rgw.log:gc", 2023-10-05T19:30:23.050 INFO:teuthology.orchestra.run.smithi167.stdout: "lc_pool": "default.rgw.log:lc", 2023-10-05T19:30:23.050 INFO:teuthology.orchestra.run.smithi167.stdout: "log_pool": "default.rgw.log", 2023-10-05T19:30:23.050 INFO:teuthology.orchestra.run.smithi167.stdout: "intent_log_pool": "default.rgw.log:intent", 2023-10-05T19:30:23.050 INFO:teuthology.orchestra.run.smithi167.stdout: "usage_log_pool": "default.rgw.log:usage", 2023-10-05T19:30:23.051 INFO:teuthology.orchestra.run.smithi167.stdout: "roles_pool": "default.rgw.meta:roles", 2023-10-05T19:30:23.051 INFO:teuthology.orchestra.run.smithi167.stdout: "reshard_pool": "default.rgw.log:reshard", 2023-10-05T19:30:23.051 INFO:teuthology.orchestra.run.smithi167.stdout: "user_keys_pool": "default.rgw.meta:users.keys", 2023-10-05T19:30:23.051 INFO:teuthology.orchestra.run.smithi167.stdout: "user_email_pool": "default.rgw.meta:users.email", 2023-10-05T19:30:23.051 INFO:teuthology.orchestra.run.smithi167.stdout: "user_swift_pool": "default.rgw.meta:users.swift", 2023-10-05T19:30:23.051 INFO:teuthology.orchestra.run.smithi167.stdout: "user_uid_pool": "default.rgw.meta:users.uid", 2023-10-05T19:30:23.051 INFO:teuthology.orchestra.run.smithi167.stdout: "otp_pool": "default.rgw.otp", 2023-10-05T19:30:23.051 INFO:teuthology.orchestra.run.smithi167.stdout: "system_key": { 2023-10-05T19:30:23.051 INFO:teuthology.orchestra.run.smithi167.stdout: "access_key": "", 2023-10-05T19:30:23.051 INFO:teuthology.orchestra.run.smithi167.stdout: "secret_key": "" 2023-10-05T19:30:23.052 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2023-10-05T19:30:23.052 INFO:teuthology.orchestra.run.smithi167.stdout: "placement_pools": [ 2023-10-05T19:30:23.052 INFO:teuthology.orchestra.run.smithi167.stdout: { 2023-10-05T19:30:23.052 INFO:teuthology.orchestra.run.smithi167.stdout: "key": "default-placement", 2023-10-05T19:30:23.052 INFO:teuthology.orchestra.run.smithi167.stdout: "val": { 2023-10-05T19:30:23.052 INFO:teuthology.orchestra.run.smithi167.stdout: "index_pool": "default.rgw.buckets.index", 2023-10-05T19:30:23.052 INFO:teuthology.orchestra.run.smithi167.stdout: "storage_classes": { 2023-10-05T19:30:23.052 INFO:teuthology.orchestra.run.smithi167.stdout: "STANDARD": { 2023-10-05T19:30:23.052 INFO:teuthology.orchestra.run.smithi167.stdout: "data_pool": "default.rgw.buckets.data" 2023-10-05T19:30:23.052 INFO:teuthology.orchestra.run.smithi167.stdout: } 2023-10-05T19:30:23.052 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2023-10-05T19:30:23.052 INFO:teuthology.orchestra.run.smithi167.stdout: "data_extra_pool": "default.rgw.buckets.non-ec", 2023-10-05T19:30:23.053 INFO:teuthology.orchestra.run.smithi167.stdout: "index_type": 0, 2023-10-05T19:30:23.053 INFO:teuthology.orchestra.run.smithi167.stdout: "inline_data": "true" 2023-10-05T19:30:23.053 INFO:teuthology.orchestra.run.smithi167.stdout: } 2023-10-05T19:30:23.053 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2023-10-05T19:30:23.053 INFO:teuthology.orchestra.run.smithi167.stdout: { 2023-10-05T19:30:23.053 INFO:teuthology.orchestra.run.smithi167.stdout: "key": "new-placement", 2023-10-05T19:30:23.053 INFO:teuthology.orchestra.run.smithi167.stdout: "val": { 2023-10-05T19:30:23.053 INFO:teuthology.orchestra.run.smithi167.stdout: "index_pool": ".rgw.buckets.index.2", 2023-10-05T19:30:23.053 INFO:teuthology.orchestra.run.smithi167.stdout: "storage_classes": { 2023-10-05T19:30:23.053 INFO:teuthology.orchestra.run.smithi167.stdout: "STANDARD": { 2023-10-05T19:30:23.053 INFO:teuthology.orchestra.run.smithi167.stdout: "data_pool": ".rgw.buckets.2" 2023-10-05T19:30:23.054 INFO:teuthology.orchestra.run.smithi167.stdout: } 2023-10-05T19:30:23.054 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2023-10-05T19:30:23.054 INFO:teuthology.orchestra.run.smithi167.stdout: "data_extra_pool": "default.rgw.buckets.non-ec", 2023-10-05T19:30:23.054 INFO:teuthology.orchestra.run.smithi167.stdout: "index_type": 0, 2023-10-05T19:30:23.054 INFO:teuthology.orchestra.run.smithi167.stdout: "inline_data": "false" 2023-10-05T19:30:23.054 INFO:teuthology.orchestra.run.smithi167.stdout: } 2023-10-05T19:30:23.054 INFO:teuthology.orchestra.run.smithi167.stdout: } 2023-10-05T19:30:23.054 INFO:teuthology.orchestra.run.smithi167.stdout: ], 2023-10-05T19:30:23.054 INFO:teuthology.orchestra.run.smithi167.stdout: "realm_id": "", 2023-10-05T19:30:23.054 INFO:teuthology.orchestra.run.smithi167.stdout: "notif_pool": "default.rgw.log:notif" 2023-10-05T19:30:23.055 INFO:teuthology.orchestra.run.smithi167.stdout:} 2023-10-05T19:30:23.055 DEBUG:tasks.util.rgw: json result: {'id': '9a9df965-6612-45d4-a45f-6e3c889bacc2', 'name': 'default', 'domain_root': 'default.rgw.meta:root', 'control_pool': 'default.rgw.control', 'gc_pool': 'default.rgw.log:gc', 'lc_pool': 'default.rgw.log:lc', 'log_pool': 'default.rgw.log', 'intent_log_pool': 'default.rgw.log:intent', 'usage_log_pool': 'default.rgw.log:usage', 'roles_pool': 'default.rgw.meta:roles', 'reshard_pool': 'default.rgw.log:reshard', 'user_keys_pool': 'default.rgw.meta:users.keys', 'user_email_pool': 'default.rgw.meta:users.email', 'user_swift_pool': 'default.rgw.meta:users.swift', 'user_uid_pool': 'default.rgw.meta:users.uid', 'otp_pool': 'default.rgw.otp', 'system_key': {'access_key': '', 'secret_key': ''}, 'placement_pools': [{'key': 'default-placement', 'val': {'index_pool': 'default.rgw.buckets.index', 'storage_classes': {'STANDARD': {'data_pool': 'default.rgw.buckets.data'}}, 'data_extra_pool': 'default.rgw.buckets.non-ec', 'index_type': 0, 'inline_data': 'true'}}, {'key': 'new-placement', 'val': {'index_pool': '.rgw.buckets.index.2', 'storage_classes': {'STANDARD': {'data_pool': '.rgw.buckets.2'}}, 'data_extra_pool': 'default.rgw.buckets.non-ec', 'index_type': 0, 'inline_data': 'false'}}], 'realm_id': '', 'notif_pool': 'default.rgw.log:notif'} 2023-10-05T19:30:23.056 INFO:tasks.util.rgw:rgwadmin: client.0 : ['zone', 'placement', 'rm', '--rgw-zone', 'default', '--placement-id', 'new-placement'] 2023-10-05T19:30:23.056 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'ceph', 'zone', 'placement', 'rm', '--rgw-zone', 'default', '--placement-id', 'new-placement'] 2023-10-05T19:30:23.056 DEBUG:teuthology.orchestra.run.smithi167:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster ceph zone placement rm --rgw-zone default --placement-id new-placement 2023-10-05T19:30:23.098 INFO:teuthology.orchestra.run.smithi167.stderr:ignoring --setuser ceph since I am not root 2023-10-05T19:30:23.098 INFO:teuthology.orchestra.run.smithi167.stderr:ignoring --setgroup ceph since I am not root 2023-10-05T19:30:23.099 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:23.094+0000 7f89244f6e40 1 Processor -- start 2023-10-05T19:30:23.099 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:23.094+0000 7f89244f6e40 1 -- start start 2023-10-05T19:30:23.099 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:23.094+0000 7f89244f6e40 1 --2- >> v2:172.21.15.118:3300/0 conn(0x55a787f9a610 0x55a787f9c9f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-05T19:30:23.099 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:23.094+0000 7f89244f6e40 1 -- --> v1:172.21.15.118:6789/0 -- auth(proto 0 26 bytes epoch 0) v1 -- 0x55a787ee0bd0 con 0x55a787f96080 2023-10-05T19:30:23.099 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:23.094+0000 7f89244f6e40 1 -- --> v2:172.21.15.118:3300/0 -- mon_getmap magic: 0 v1 -- 0x55a787e9dee0 con 0x55a787f9a610 2023-10-05T19:30:23.103 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:23.098+0000 7f892182a700 1 --2- >> v2:172.21.15.118:3300/0 conn(0x55a787f9a610 0x55a787f9c9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-05T19:30:23.103 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:23.098+0000 7f892202b700 1 --1- >> v1:172.21.15.118:6789/0 conn(0x55a787f96080 0x55a787f96450 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.118:6789/0 says I am v1:172.21.15.167:50838/0 (socket says 172.21.15.167:50838) 2023-10-05T19:30:23.103 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:23.098+0000 7f892202b700 1 -- 172.21.15.167:0/355481765 learned_addr learned my addr 172.21.15.167:0/355481765 (peer_addr_for_me v1:172.21.15.167:0/0) 2023-10-05T19:30:23.104 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:23.098+0000 7f8921029700 1 -- 172.21.15.167:0/355481765 <== mon.0 v1:172.21.15.118:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4146079574 0 0) 0x55a787ee0bd0 con 0x55a787f96080 2023-10-05T19:30:23.104 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:23.098+0000 7f8921029700 1 -- 172.21.15.167:0/355481765 --> v1:172.21.15.118:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8910001e10 con 0x55a787f96080 2023-10-05T19:30:23.104 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:23.098+0000 7f892182a700 1 -- 172.21.15.167:0/355481765 >> v1:172.21.15.118:6789/0 conn(0x55a787f96080 legacy=0x55a787f96450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-05T19:30:23.104 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:23.098+0000 7f892182a700 1 -- 172.21.15.167:0/355481765 --> v2:172.21.15.118:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55a787f96c90 con 0x55a787f9a610 2023-10-05T19:30:23.104 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:23.098+0000 7f892182a700 1 --2- 172.21.15.167:0/355481765 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55a787f9a610 0x55a787f9c9f0 secure :-1 s=READY pgs=693 cs=0 l=1 rev1=1 crypto rx=0x7f890c00a5d0 tx=0x7f890c005a50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6726e4cd3854bc4f server_cookie=0 in_seq=0 out_seq=0 2023-10-05T19:30:23.104 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:23.098+0000 7f8921029700 1 -- 172.21.15.167:0/355481765 <== mon.0 v2:172.21.15.118:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f890c007610 con 0x55a787f9a610 2023-10-05T19:30:23.104 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:23.098+0000 7f8921029700 1 -- 172.21.15.167:0/355481765 <== mon.0 v2:172.21.15.118:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f890c007770 con 0x55a787f9a610 2023-10-05T19:30:23.104 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:23.098+0000 7f8921029700 1 -- 172.21.15.167:0/355481765 <== mon.0 v2:172.21.15.118:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f890c007a20 con 0x55a787f9a610 2023-10-05T19:30:23.104 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:23.098+0000 7f89244f6e40 1 -- 172.21.15.167:0/355481765 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55a787f9a610 msgr2=0x55a787f9c9f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-05T19:30:23.105 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:23.098+0000 7f89244f6e40 1 --2- 172.21.15.167:0/355481765 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55a787f9a610 0x55a787f9c9f0 secure :-1 s=READY pgs=693 cs=0 l=1 rev1=1 crypto rx=0x7f890c00a5d0 tx=0x7f890c005a50 comp rx=0 tx=0).stop 2023-10-05T19:30:23.105 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:23.098+0000 7f89244f6e40 1 -- 172.21.15.167:0/355481765 shutdown_connections 2023-10-05T19:30:23.105 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:23.098+0000 7f89244f6e40 1 --2- 172.21.15.167:0/355481765 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55a787f9a610 0x55a787f9c9f0 unknown :-1 s=CLOSED pgs=693 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-05T19:30:23.105 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:23.098+0000 7f89244f6e40 1 -- 172.21.15.167:0/355481765 >> 172.21.15.167:0/355481765 conn(0x55a787f09880 msgr2=0x55a787f09c80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-05T19:30:23.105 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:23.098+0000 7f89244f6e40 1 -- 172.21.15.167:0/355481765 shutdown_connections 2023-10-05T19:30:23.105 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:23.098+0000 7f89244f6e40 1 -- 172.21.15.167:0/355481765 wait complete. 2023-10-05T19:30:23.105 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:23.098+0000 7f89244f6e40 1 Processor -- start 2023-10-05T19:30:23.105 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:23.098+0000 7f89244f6e40 1 -- start start 2023-10-05T19:30:23.105 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:23.098+0000 7f89244f6e40 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55a787f96080 0x55a787fb4fe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-05T19:30:23.105 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:23.098+0000 7f89244f6e40 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a787e61e30 con 0x55a787f96080 2023-10-05T19:30:23.107 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:23.102+0000 7f892202b700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55a787f96080 0x55a787fb4fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-05T19:30:23.107 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:23.102+0000 7f892202b700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55a787f96080 0x55a787fb4fe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.118:3300/0 says I am v2:172.21.15.167:33226/0 (socket says 172.21.15.167:33226) 2023-10-05T19:30:23.108 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:23.102+0000 7f892202b700 1 -- 172.21.15.167:0/2144927244 learned_addr learned my addr 172.21.15.167:0/2144927244 (peer_addr_for_me v2:172.21.15.167:0/0) 2023-10-05T19:30:23.108 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:23.102+0000 7f892202b700 1 -- 172.21.15.167:0/2144927244 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f890c005cf0 con 0x55a787f96080 2023-10-05T19:30:23.108 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:23.102+0000 7f892202b700 1 --2- 172.21.15.167:0/2144927244 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55a787f96080 0x55a787fb4fe0 secure :-1 s=READY pgs=694 cs=0 l=1 rev1=1 crypto rx=0x7f8914001c30 tx=0x7f8914001810 comp rx=0 tx=0).ready entity=mon.0 client_cookie=331acce17c975567 server_cookie=0 in_seq=0 out_seq=0 2023-10-05T19:30:23.108 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:23.102+0000 7f891bfff700 1 -- 172.21.15.167:0/2144927244 <== mon.0 v2:172.21.15.118:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f891400bbf0 con 0x55a787f96080 2023-10-05T19:30:23.108 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:23.102+0000 7f891bfff700 1 -- 172.21.15.167:0/2144927244 <== mon.0 v2:172.21.15.118:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f891400bd50 con 0x55a787f96080 2023-10-05T19:30:23.108 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:23.102+0000 7f891bfff700 1 -- 172.21.15.167:0/2144927244 <== mon.0 v2:172.21.15.118:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f891400e470 con 0x55a787f96080 2023-10-05T19:30:23.108 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:23.102+0000 7f89244f6e40 1 -- 172.21.15.167:0/2144927244 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55a787f96080 msgr2=0x55a787fb4fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-05T19:30:23.108 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:23.102+0000 7f89244f6e40 1 --2- 172.21.15.167:0/2144927244 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55a787f96080 0x55a787fb4fe0 secure :-1 s=READY pgs=694 cs=0 l=1 rev1=1 crypto rx=0x7f8914001c30 tx=0x7f8914001810 comp rx=0 tx=0).stop 2023-10-05T19:30:23.108 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:23.102+0000 7f89244f6e40 1 -- 172.21.15.167:0/2144927244 shutdown_connections 2023-10-05T19:30:23.108 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:23.102+0000 7f89244f6e40 1 --2- 172.21.15.167:0/2144927244 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55a787f96080 0x55a787fb4fe0 unknown :-1 s=CLOSED pgs=694 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-05T19:30:23.109 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:23.102+0000 7f89244f6e40 1 -- 172.21.15.167:0/2144927244 >> 172.21.15.167:0/2144927244 conn(0x55a787f09880 msgr2=0x55a787fb0b20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-05T19:30:23.109 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:23.102+0000 7f89244f6e40 1 -- 172.21.15.167:0/2144927244 shutdown_connections 2023-10-05T19:30:23.109 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:23.102+0000 7f89244f6e40 1 -- 172.21.15.167:0/2144927244 wait complete. 2023-10-05T19:30:23.109 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:23.102+0000 7f89244f6e40 1 Processor -- start 2023-10-05T19:30:23.109 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:23.102+0000 7f89244f6e40 1 -- start start 2023-10-05T19:30:23.109 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:23.102+0000 7f89244f6e40 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55a787f96080 0x55a787fbbda0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-05T19:30:23.109 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:23.102+0000 7f89244f6e40 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a787e34ed0 con 0x55a787f96080 2023-10-05T19:30:23.109 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:23.102+0000 7f892202b700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55a787f96080 0x55a787fbbda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-05T19:30:23.109 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:23.102+0000 7f892202b700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55a787f96080 0x55a787fbbda0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.118:3300/0 says I am v2:172.21.15.167:33228/0 (socket says 172.21.15.167:33228) 2023-10-05T19:30:23.109 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:23.102+0000 7f892202b700 1 -- 172.21.15.167:0/4194990805 learned_addr learned my addr 172.21.15.167:0/4194990805 (peer_addr_for_me v2:172.21.15.167:0/0) 2023-10-05T19:30:23.111 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:23.106+0000 7f892202b700 1 -- 172.21.15.167:0/4194990805 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8910001e10 con 0x55a787f96080 2023-10-05T19:30:23.111 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:23.106+0000 7f892202b700 1 --2- 172.21.15.167:0/4194990805 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55a787f96080 0x55a787fbbda0 secure :-1 s=READY pgs=695 cs=0 l=1 rev1=1 crypto rx=0x7f891400b960 tx=0x7f8914014000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-05T19:30:23.111 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:23.106+0000 7f891a7fc700 1 -- 172.21.15.167:0/4194990805 <== mon.0 v2:172.21.15.118:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f891401f070 con 0x55a787f96080 2023-10-05T19:30:23.111 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:23.106+0000 7f891a7fc700 1 -- 172.21.15.167:0/4194990805 <== mon.0 v2:172.21.15.118:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f891401a2e0 con 0x55a787f96080 2023-10-05T19:30:23.112 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:23.106+0000 7f89244f6e40 1 -- 172.21.15.167:0/4194990805 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x55a787fbc2e0 con 0x55a787f96080 2023-10-05T19:30:23.112 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:23.106+0000 7f891a7fc700 1 -- 172.21.15.167:0/4194990805 <== mon.0 v2:172.21.15.118:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f891400e470 con 0x55a787f96080 2023-10-05T19:30:23.112 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:23.106+0000 7f89244f6e40 1 -- 172.21.15.167:0/4194990805 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55a787fbc720 con 0x55a787f96080 2023-10-05T19:30:23.112 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:23.106+0000 7f89244f6e40 20 check_secure_mon_conn(): auth registy supported: methods=[2] modes=[2,1] 2023-10-05T19:30:23.112 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:23.106+0000 7f89244f6e40 20 check_secure_mon_conn(): mode 1 is insecure 2023-10-05T19:30:23.112 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:23.106+0000 7f891a7fc700 1 -- 172.21.15.167:0/4194990805 <== mon.0 v2:172.21.15.118:3300/0 4 ==== mgrmap(e 5) v1 ==== 40329+0+0 (secure 0 0 0) 0x7f891401aa10 con 0x55a787f96080 2023-10-05T19:30:23.112 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:23.106+0000 7f89244f6e40 1 Processor -- start 2023-10-05T19:30:23.112 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:23.106+0000 7f89244f6e40 1 -- start start 2023-10-05T19:30:23.112 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:23.106+0000 7f89244f6e40 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55a787fcd4d0 0x55a787fcd8a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-05T19:30:23.112 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:23.106+0000 7f89244f6e40 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a787fcdde0 con 0x55a787fcd4d0 2023-10-05T19:30:23.112 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:23.106+0000 7f891a7fc700 1 --2- 172.21.15.167:0/4194990805 >> [v2:172.21.15.167:6824/29889,v1:172.21.15.167:6825/29889] conn(0x7f8904030c40 0x7f89040330f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-05T19:30:23.113 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:23.106+0000 7f891a7fc700 1 -- 172.21.15.167:0/4194990805 <== mon.0 v2:172.21.15.118:3300/0 5 ==== osd_map(32..32 src has 1..32) v4 ==== 6260+0+0 (secure 0 0 0) 0x7f8914045280 con 0x55a787f96080 2023-10-05T19:30:23.113 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:23.106+0000 7f892282c700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55a787fcd4d0 0x55a787fcd8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-05T19:30:23.113 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:23.106+0000 7f892202b700 1 --2- 172.21.15.167:0/4194990805 >> [v2:172.21.15.167:6824/29889,v1:172.21.15.167:6825/29889] conn(0x7f8904030c40 0x7f89040330f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-05T19:30:23.113 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:23.106+0000 7f892282c700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55a787fcd4d0 0x55a787fcd8a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.118:3300/0 says I am v2:172.21.15.167:33242/0 (socket says 172.21.15.167:33242) 2023-10-05T19:30:23.113 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:23.106+0000 7f892282c700 1 -- 172.21.15.167:0/3854315503 learned_addr learned my addr 172.21.15.167:0/3854315503 (peer_addr_for_me v2:172.21.15.167:0/0) 2023-10-05T19:30:23.115 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:23.110+0000 7f892282c700 1 -- 172.21.15.167:0/3854315503 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55a787fcdf20 con 0x55a787fcd4d0 2023-10-05T19:30:23.115 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:23.110+0000 7f892282c700 1 --2- 172.21.15.167:0/3854315503 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55a787fcd4d0 0x55a787fcd8a0 secure :-1 s=READY pgs=696 cs=0 l=1 rev1=1 crypto rx=0x7f891c00a3b0 tx=0x7f891c005cc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5b7f6483a343fe9c server_cookie=0 in_seq=0 out_seq=0 2023-10-05T19:30:23.115 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:23.110+0000 7f892202b700 1 --2- 172.21.15.167:0/4194990805 >> [v2:172.21.15.167:6824/29889,v1:172.21.15.167:6825/29889] conn(0x7f8904030c40 0x7f89040330f0 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7f891402dfd0 tx=0x7f8914043570 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-05T19:30:23.115 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:23.110+0000 7f88f3fdf700 1 -- 172.21.15.167:0/3854315503 <== mon.0 v2:172.21.15.118:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f891c001520 con 0x55a787fcd4d0 2023-10-05T19:30:23.115 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:23.110+0000 7f88f3fdf700 1 -- 172.21.15.167:0/3854315503 <== mon.0 v2:172.21.15.118:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f891c001680 con 0x55a787fcd4d0 2023-10-05T19:30:23.115 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:23.110+0000 7f88f3fdf700 1 -- 172.21.15.167:0/3854315503 <== mon.0 v2:172.21.15.118:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f891c001830 con 0x55a787fcd4d0 2023-10-05T19:30:23.115 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:23.110+0000 7f89244f6e40 1 -- 172.21.15.167:0/3854315503 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55a787fcd4d0 msgr2=0x55a787fcd8a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-05T19:30:23.115 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:23.110+0000 7f89244f6e40 1 --2- 172.21.15.167:0/3854315503 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55a787fcd4d0 0x55a787fcd8a0 secure :-1 s=READY pgs=696 cs=0 l=1 rev1=1 crypto rx=0x7f891c00a3b0 tx=0x7f891c005cc0 comp rx=0 tx=0).stop 2023-10-05T19:30:23.115 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:23.110+0000 7f89244f6e40 1 -- 172.21.15.167:0/3854315503 shutdown_connections 2023-10-05T19:30:23.116 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:23.110+0000 7f89244f6e40 1 --2- 172.21.15.167:0/3854315503 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55a787fcd4d0 0x55a787fcd8a0 unknown :-1 s=CLOSED pgs=696 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-05T19:30:23.116 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:23.110+0000 7f89244f6e40 1 -- 172.21.15.167:0/3854315503 >> 172.21.15.167:0/3854315503 conn(0x55a787f9a610 msgr2=0x55a787fca220 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-05T19:30:23.116 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:23.110+0000 7f89244f6e40 1 -- 172.21.15.167:0/3854315503 shutdown_connections 2023-10-05T19:30:23.116 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:23.110+0000 7f89244f6e40 1 -- 172.21.15.167:0/3854315503 wait complete. 2023-10-05T19:30:23.116 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:23.110+0000 7f89244f6e40 1 Processor -- start 2023-10-05T19:30:23.116 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:23.110+0000 7f89244f6e40 1 -- start start 2023-10-05T19:30:23.117 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:23.110+0000 7f89244f6e40 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55a787fcd4d0 0x55a787fd4610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-05T19:30:23.117 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:23.110+0000 7f89244f6e40 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a787fd5610 con 0x55a787fcd4d0 2023-10-05T19:30:23.117 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:23.110+0000 7f892282c700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55a787fcd4d0 0x55a787fd4610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-05T19:30:23.118 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:23.110+0000 7f892282c700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55a787fcd4d0 0x55a787fd4610 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.118:3300/0 says I am v2:172.21.15.167:33256/0 (socket says 172.21.15.167:33256) 2023-10-05T19:30:23.118 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:23.110+0000 7f892282c700 1 -- 172.21.15.167:0/2976557353 learned_addr learned my addr 172.21.15.167:0/2976557353 (peer_addr_for_me v2:172.21.15.167:0/0) 2023-10-05T19:30:23.118 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:23.114+0000 7f892282c700 1 -- 172.21.15.167:0/2976557353 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f891c00a0e0 con 0x55a787fcd4d0 2023-10-05T19:30:23.118 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:23.114+0000 7f892282c700 1 --2- 172.21.15.167:0/2976557353 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55a787fcd4d0 0x55a787fd4610 secure :-1 s=READY pgs=697 cs=0 l=1 rev1=1 crypto rx=0x7f891c00ab00 tx=0x7f891c014000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-05T19:30:23.118 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:23.114+0000 7f88f27dc700 1 -- 172.21.15.167:0/2976557353 <== mon.0 v2:172.21.15.118:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f891c001520 con 0x55a787fcd4d0 2023-10-05T19:30:23.118 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:23.114+0000 7f88f27dc700 1 -- 172.21.15.167:0/2976557353 <== mon.0 v2:172.21.15.118:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f891c00aec0 con 0x55a787fcd4d0 2023-10-05T19:30:23.118 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:23.114+0000 7f89244f6e40 1 -- 172.21.15.167:0/2976557353 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x55a787fd4c10 con 0x55a787fcd4d0 2023-10-05T19:30:23.118 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:23.114+0000 7f88f27dc700 1 -- 172.21.15.167:0/2976557353 <== mon.0 v2:172.21.15.118:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f891c005830 con 0x55a787fcd4d0 2023-10-05T19:30:23.118 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:23.114+0000 7f89244f6e40 1 -- 172.21.15.167:0/2976557353 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55a787fd50d0 con 0x55a787fcd4d0 2023-10-05T19:30:23.119 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:23.114+0000 7f88f27dc700 1 -- 172.21.15.167:0/2976557353 <== mon.0 v2:172.21.15.118:3300/0 4 ==== mgrmap(e 5) v1 ==== 40329+0+0 (secure 0 0 0) 0x7f891c023020 con 0x55a787fcd4d0 2023-10-05T19:30:23.119 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:23.114+0000 7f88f27dc700 1 --2- 172.21.15.167:0/2976557353 >> [v2:172.21.15.167:6824/29889,v1:172.21.15.167:6825/29889] conn(0x7f8870030c50 0x7f8870033100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-05T19:30:23.119 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:23.114+0000 7f88f27dc700 1 -- 172.21.15.167:0/2976557353 <== mon.0 v2:172.21.15.118:3300/0 5 ==== osd_map(32..32 src has 1..32) v4 ==== 6260+0+0 (secure 0 0 0) 0x7f891c045820 con 0x55a787fcd4d0 2023-10-05T19:30:23.119 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:23.114+0000 7f892182a700 1 --2- 172.21.15.167:0/2976557353 >> [v2:172.21.15.167:6824/29889,v1:172.21.15.167:6825/29889] conn(0x7f8870030c50 0x7f8870033100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-05T19:30:23.120 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:23.118+0000 7f892182a700 1 --2- 172.21.15.167:0/2976557353 >> [v2:172.21.15.167:6824/29889,v1:172.21.15.167:6825/29889] conn(0x7f8870030c50 0x7f8870033100 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7f890c005cc0 tx=0x7f890c014000 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-05T19:30:23.120 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:23.118+0000 7f886bfff700 20 reqs_thread_entry: start 2023-10-05T19:30:23.121 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:23.118+0000 7f89244f6e40 20 try_to_resolve_local_entities(): before: realm_id= realm_name= zonegroup_id= zonegroup_name= zone_id= zone_name=default 2023-10-05T19:30:23.121 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:23.118+0000 7f89244f6e40 20 rados->read ofs=0 len=0 2023-10-05T19:30:23.121 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:23.118+0000 7f89244f6e40 1 --2- 172.21.15.167:0/4194990805 >> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] conn(0x55a787ff22f0 0x55a787ff2760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-05T19:30:23.121 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:23.118+0000 7f89244f6e40 1 -- 172.21.15.167:0/4194990805 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:1 5.f 5:f43b6ece:::zone_names.default:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32) v8 -- 0x55a787ff2ca0 con 0x55a787ff22f0 2023-10-05T19:30:23.121 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:23.118+0000 7f892282c700 1 --2- 172.21.15.167:0/4194990805 >> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] conn(0x55a787ff22f0 0x55a787ff2760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-05T19:30:23.121 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:23.118+0000 7f892282c700 1 --2- 172.21.15.167:0/4194990805 >> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] conn(0x55a787ff22f0 0x55a787ff2760 crc :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-05T19:30:23.122 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:23.118+0000 7f892282c700 1 -- 172.21.15.167:0/4194990805 <== osd.1 v2:172.21.15.167:6800/30035 1 ==== osd_op_reply(1 zone_names.default [read 0~46 out=46b] v0'0 uv3 ondisk = 0) v8 ==== 162+0+46 (crc 0 0 0) 0x7f891c01cce0 con 0x55a787ff22f0 2023-10-05T19:30:23.122 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:23.118+0000 7f89244f6e40 20 rados_obj.operate() r=0 bl.length=46 2023-10-05T19:30:23.122 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:23.118+0000 7f89244f6e40 20 rados->read ofs=0 len=0 2023-10-05T19:30:23.122 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:23.118+0000 7f89244f6e40 1 -- 172.21.15.167:0/4194990805 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:2 5.1d 5:b8374eb4:::zone_info.9a9df965-6612-45d4-a45f-6e3c889bacc2:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32) v8 -- 0x55a788005810 con 0x55a787ff22f0 2023-10-05T19:30:23.123 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:23.118+0000 7f892282c700 1 -- 172.21.15.167:0/4194990805 <== osd.1 v2:172.21.15.167:6800/30035 2 ==== osd_op_reply(2 zone_info.9a9df965-6612-45d4-a45f-6e3c889bacc2 [read 0~1036 out=1036b] v0'0 uv3 ondisk = 0) v8 ==== 190+0+1036 (crc 0 0 0) 0x7f891c01cce0 con 0x55a787ff22f0 2023-10-05T19:30:23.123 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:23.118+0000 7f89244f6e40 20 rados_obj.operate() r=0 bl.length=1036 2023-10-05T19:30:23.123 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:23.118+0000 7f89244f6e40 1 --2- 172.21.15.167:0/4194990805 >> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] conn(0x55a7880070e0 0x55a7880094c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-05T19:30:23.123 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:23.118+0000 7f89244f6e40 1 -- 172.21.15.167:0/4194990805 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:3 5.0 5:00000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x55a788009a00 con 0x55a7880070e0 2023-10-05T19:30:23.124 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:23.118+0000 7f892202b700 1 --2- 172.21.15.167:0/4194990805 >> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] conn(0x55a7880070e0 0x55a7880094c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-05T19:30:23.124 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:23.118+0000 7f892202b700 1 --2- 172.21.15.167:0/4194990805 >> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] conn(0x55a7880070e0 0x55a7880094c0 crc :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-05T19:30:23.125 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:23.122+0000 7f892202b700 1 -- 172.21.15.167:0/4194990805 <== osd.3 v2:172.21.15.167:6808/30036 1 ==== osd_op_reply(3 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f8914014070 con 0x55a7880070e0 2023-10-05T19:30:23.125 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:23.122+0000 7f892202b700 1 --2- 172.21.15.167:0/4194990805 >> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] conn(0x7f891402d020 0x7f8914044730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-05T19:30:23.125 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:23.122+0000 7f892202b700 1 -- 172.21.15.167:0/4194990805 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:4 5.10 5:08000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x55a788009a00 con 0x7f891402d020 2023-10-05T19:30:23.125 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:23.122+0000 7f892182a700 1 --2- 172.21.15.167:0/4194990805 >> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] conn(0x7f891402d020 0x7f8914044730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-05T19:30:23.126 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:23.122+0000 7f892182a700 1 --2- 172.21.15.167:0/4194990805 >> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] conn(0x7f891402d020 0x7f8914044730 crc :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-05T19:30:23.126 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:23.122+0000 7f892182a700 1 -- 172.21.15.167:0/4194990805 <== osd.2 v2:172.21.15.167:6816/30037 1 ==== osd_op_reply(4 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f890c007c80 con 0x7f891402d020 2023-10-05T19:30:23.126 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:23.122+0000 7f892182a700 1 --2- 172.21.15.167:0/4194990805 >> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] conn(0x7f890c007030 0x7f890c01d430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-05T19:30:23.127 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:23.122+0000 7f892182a700 1 -- 172.21.15.167:0/4194990805 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:5 5.8 5:10000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x55a788009a00 con 0x7f890c007030 2023-10-05T19:30:23.127 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:23.122+0000 7f892282c700 1 --2- 172.21.15.167:0/4194990805 >> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] conn(0x7f890c007030 0x7f890c01d430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-05T19:30:23.127 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:23.122+0000 7f892282c700 1 --2- 172.21.15.167:0/4194990805 >> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] conn(0x7f890c007030 0x7f890c01d430 crc :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-05T19:30:23.128 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:23.126+0000 7f892282c700 1 -- 172.21.15.167:0/4194990805 <== osd.0 v2:172.21.15.118:6800/27690 1 ==== osd_op_reply(5 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f891c01cce0 con 0x7f890c007030 2023-10-05T19:30:23.128 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:23.126+0000 7f892282c700 1 -- 172.21.15.167:0/4194990805 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:6 5.18 5:18000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x55a788009a00 con 0x55a787ff22f0 2023-10-05T19:30:23.129 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:23.126+0000 7f892282c700 1 -- 172.21.15.167:0/4194990805 <== osd.1 v2:172.21.15.167:6800/30035 3 ==== osd_op_reply(6 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f891c01cce0 con 0x55a787ff22f0 2023-10-05T19:30:23.129 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:23.126+0000 7f892282c700 1 -- 172.21.15.167:0/4194990805 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:7 5.4 5:20000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x55a788009a00 con 0x55a787ff22f0 2023-10-05T19:30:23.129 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:23.126+0000 7f892282c700 1 -- 172.21.15.167:0/4194990805 <== osd.1 v2:172.21.15.167:6800/30035 4 ==== osd_op_reply(7 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f891c01cce0 con 0x55a787ff22f0 2023-10-05T19:30:23.129 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:23.126+0000 7f892282c700 1 -- 172.21.15.167:0/4194990805 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:8 5.14 5:28000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x55a788009a00 con 0x7f890c007030 2023-10-05T19:30:23.130 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:23.126+0000 7f892282c700 1 -- 172.21.15.167:0/4194990805 <== osd.0 v2:172.21.15.118:6800/27690 2 ==== osd_op_reply(8 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f891c01cce0 con 0x7f890c007030 2023-10-05T19:30:23.130 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:23.126+0000 7f892282c700 1 -- 172.21.15.167:0/4194990805 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:9 5.c 5:30000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x55a788009a00 con 0x7f890c007030 2023-10-05T19:30:23.131 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:23.126+0000 7f892282c700 1 -- 172.21.15.167:0/4194990805 <== osd.0 v2:172.21.15.118:6800/27690 3 ==== osd_op_reply(9 [pgnls start_epoch 32 out=112b] v23'1 uv1 ondisk = 1) v8 ==== 144+0+112 (crc 0 0 0) 0x7f891c01cce0 con 0x7f890c007030 2023-10-05T19:30:23.131 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:23.126+0000 7f892282c700 1 -- 172.21.15.167:0/4194990805 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:10 5.1c 5:38000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x55a788009a00 con 0x7f891402d020 2023-10-05T19:30:23.132 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:23.130+0000 7f892182a700 1 -- 172.21.15.167:0/4194990805 <== osd.2 v2:172.21.15.167:6816/30037 2 ==== osd_op_reply(10 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f890c029040 con 0x7f891402d020 2023-10-05T19:30:23.132 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:23.130+0000 7f892182a700 1 -- 172.21.15.167:0/4194990805 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:11 5.2 5:40000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x55a788009a00 con 0x55a7880070e0 2023-10-05T19:30:23.132 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:23.130+0000 7f892202b700 1 -- 172.21.15.167:0/4194990805 <== osd.3 v2:172.21.15.167:6808/30036 2 ==== osd_op_reply(11 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f8914014070 con 0x55a7880070e0 2023-10-05T19:30:23.132 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:23.130+0000 7f892202b700 1 -- 172.21.15.167:0/4194990805 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:12 5.12 5:48000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x55a788009a00 con 0x55a787ff22f0 2023-10-05T19:30:23.133 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:23.130+0000 7f892282c700 1 -- 172.21.15.167:0/4194990805 <== osd.1 v2:172.21.15.167:6800/30035 5 ==== osd_op_reply(12 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f891c01cce0 con 0x55a787ff22f0 2023-10-05T19:30:23.133 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:23.130+0000 7f892282c700 1 -- 172.21.15.167:0/4194990805 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:13 5.a 5:50000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x55a788009a00 con 0x7f890c007030 2023-10-05T19:30:23.134 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:23.130+0000 7f892282c700 1 -- 172.21.15.167:0/4194990805 <== osd.0 v2:172.21.15.118:6800/27690 4 ==== osd_op_reply(13 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f891c01cce0 con 0x7f890c007030 2023-10-05T19:30:23.134 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:23.130+0000 7f892282c700 1 -- 172.21.15.167:0/4194990805 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:14 5.1a 5:58000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x55a788009a00 con 0x55a787ff22f0 2023-10-05T19:30:23.135 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:23.130+0000 7f892282c700 1 -- 172.21.15.167:0/4194990805 <== osd.1 v2:172.21.15.167:6800/30035 6 ==== osd_op_reply(14 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f891c01cce0 con 0x55a787ff22f0 2023-10-05T19:30:23.135 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:23.130+0000 7f892282c700 1 -- 172.21.15.167:0/4194990805 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:15 5.6 5:60000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x55a788009a00 con 0x7f891402d020 2023-10-05T19:30:23.136 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:23.134+0000 7f892182a700 1 -- 172.21.15.167:0/4194990805 <== osd.2 v2:172.21.15.167:6816/30037 3 ==== osd_op_reply(15 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f890c029040 con 0x7f891402d020 2023-10-05T19:30:23.136 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:23.134+0000 7f892182a700 1 -- 172.21.15.167:0/4194990805 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:16 5.16 5:68000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x55a788009a00 con 0x55a7880070e0 2023-10-05T19:30:23.136 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:23.134+0000 7f892202b700 1 -- 172.21.15.167:0/4194990805 <== osd.3 v2:172.21.15.167:6808/30036 3 ==== osd_op_reply(16 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f8914014070 con 0x55a7880070e0 2023-10-05T19:30:23.136 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:23.134+0000 7f892202b700 1 -- 172.21.15.167:0/4194990805 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:17 5.e 5:70000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x55a788009a00 con 0x55a7880070e0 2023-10-05T19:30:23.137 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:23.134+0000 7f892202b700 1 -- 172.21.15.167:0/4194990805 <== osd.3 v2:172.21.15.167:6808/30036 4 ==== osd_op_reply(17 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f8914014070 con 0x55a7880070e0 2023-10-05T19:30:23.137 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:23.134+0000 7f892202b700 1 -- 172.21.15.167:0/4194990805 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:18 5.1e 5:78000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x55a788009a00 con 0x7f890c007030 2023-10-05T19:30:23.138 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:23.134+0000 7f892282c700 1 -- 172.21.15.167:0/4194990805 <== osd.0 v2:172.21.15.118:6800/27690 5 ==== osd_op_reply(18 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f891c01cce0 con 0x7f890c007030 2023-10-05T19:30:23.138 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:23.134+0000 7f892282c700 1 -- 172.21.15.167:0/4194990805 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:19 5.1 5:80000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x55a788009a00 con 0x55a7880070e0 2023-10-05T19:30:23.139 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:23.134+0000 7f892202b700 1 -- 172.21.15.167:0/4194990805 <== osd.3 v2:172.21.15.167:6808/30036 5 ==== osd_op_reply(19 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f8914014070 con 0x55a7880070e0 2023-10-05T19:30:23.139 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:23.134+0000 7f892202b700 1 -- 172.21.15.167:0/4194990805 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:20 5.11 5:88000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x55a788009a00 con 0x55a7880070e0 2023-10-05T19:30:23.139 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:23.134+0000 7f892202b700 1 -- 172.21.15.167:0/4194990805 <== osd.3 v2:172.21.15.167:6808/30036 6 ==== osd_op_reply(20 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f8914014070 con 0x55a7880070e0 2023-10-05T19:30:23.140 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:23.134+0000 7f892202b700 1 -- 172.21.15.167:0/4194990805 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:21 5.9 5:90000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x55a788009a00 con 0x55a787ff22f0 2023-10-05T19:30:23.140 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:23.138+0000 7f892282c700 1 -- 172.21.15.167:0/4194990805 <== osd.1 v2:172.21.15.167:6800/30035 7 ==== osd_op_reply(21 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f891c01cce0 con 0x55a787ff22f0 2023-10-05T19:30:23.140 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:23.138+0000 7f892282c700 1 -- 172.21.15.167:0/4194990805 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:22 5.19 5:98000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x55a788009a00 con 0x55a787ff22f0 2023-10-05T19:30:23.141 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:23.138+0000 7f892282c700 1 -- 172.21.15.167:0/4194990805 <== osd.1 v2:172.21.15.167:6800/30035 8 ==== osd_op_reply(22 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f891c01cce0 con 0x55a787ff22f0 2023-10-05T19:30:23.141 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:23.138+0000 7f892282c700 1 -- 172.21.15.167:0/4194990805 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:23 5.5 5:a0000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x55a788009a00 con 0x7f890c007030 2023-10-05T19:30:23.142 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:23.138+0000 7f892282c700 1 -- 172.21.15.167:0/4194990805 <== osd.0 v2:172.21.15.118:6800/27690 6 ==== osd_op_reply(23 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f891c01cce0 con 0x7f890c007030 2023-10-05T19:30:23.142 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:23.138+0000 7f892282c700 1 -- 172.21.15.167:0/4194990805 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:24 5.15 5:a8000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x55a788009a00 con 0x55a7880070e0 2023-10-05T19:30:23.143 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:23.138+0000 7f892202b700 1 -- 172.21.15.167:0/4194990805 <== osd.3 v2:172.21.15.167:6808/30036 7 ==== osd_op_reply(24 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f8914014070 con 0x55a7880070e0 2023-10-05T19:30:23.143 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:23.138+0000 7f892202b700 1 -- 172.21.15.167:0/4194990805 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:25 5.d 5:b0000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x55a788009a00 con 0x7f891402d020 2023-10-05T19:30:23.143 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:23.138+0000 7f892182a700 1 -- 172.21.15.167:0/4194990805 <== osd.2 v2:172.21.15.167:6816/30037 4 ==== osd_op_reply(25 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f890c029040 con 0x7f891402d020 2023-10-05T19:30:23.143 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:23.138+0000 7f892182a700 1 -- 172.21.15.167:0/4194990805 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:26 5.1d 5:b8000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x55a788009a00 con 0x55a787ff22f0 2023-10-05T19:30:23.144 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:23.142+0000 7f892282c700 1 -- 172.21.15.167:0/4194990805 <== osd.1 v2:172.21.15.167:6800/30035 9 ==== osd_op_reply(26 [pgnls start_epoch 32 out=107b] v32'3 uv3 ondisk = 1) v8 ==== 144+0+107 (crc 0 0 0) 0x7f891c01cce0 con 0x55a787ff22f0 2023-10-05T19:30:23.144 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:23.142+0000 7f892282c700 1 -- 172.21.15.167:0/4194990805 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:27 5.3 5:c0000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x55a788009a00 con 0x7f890c007030 2023-10-05T19:30:23.145 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:23.142+0000 7f892282c700 1 -- 172.21.15.167:0/4194990805 <== osd.0 v2:172.21.15.118:6800/27690 7 ==== osd_op_reply(27 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f891c01cce0 con 0x7f890c007030 2023-10-05T19:30:23.145 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:23.142+0000 7f892282c700 1 -- 172.21.15.167:0/4194990805 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:28 5.13 5:c8000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x55a788009a00 con 0x55a7880070e0 2023-10-05T19:30:23.146 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:23.142+0000 7f892202b700 1 -- 172.21.15.167:0/4194990805 <== osd.3 v2:172.21.15.167:6808/30036 8 ==== osd_op_reply(28 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f8914014070 con 0x55a7880070e0 2023-10-05T19:30:23.146 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:23.142+0000 7f892202b700 1 -- 172.21.15.167:0/4194990805 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:29 5.b 5:d0000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x55a788009a00 con 0x7f890c007030 2023-10-05T19:30:23.147 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:23.142+0000 7f892282c700 1 -- 172.21.15.167:0/4194990805 <== osd.0 v2:172.21.15.118:6800/27690 8 ==== osd_op_reply(29 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f891c01cce0 con 0x7f890c007030 2023-10-05T19:30:23.147 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:23.142+0000 7f892282c700 1 -- 172.21.15.167:0/4194990805 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:30 5.1b 5:d8000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x55a788009a00 con 0x7f891402d020 2023-10-05T19:30:23.148 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:23.146+0000 7f892182a700 1 -- 172.21.15.167:0/4194990805 <== osd.2 v2:172.21.15.167:6816/30037 5 ==== osd_op_reply(30 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f890c029040 con 0x7f891402d020 2023-10-05T19:30:23.148 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:23.146+0000 7f892182a700 1 -- 172.21.15.167:0/4194990805 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:31 5.7 5:e0000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x55a788009a00 con 0x55a7880070e0 2023-10-05T19:30:23.148 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:23.146+0000 7f892202b700 1 -- 172.21.15.167:0/4194990805 <== osd.3 v2:172.21.15.167:6808/30036 9 ==== osd_op_reply(31 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f8914014070 con 0x55a7880070e0 2023-10-05T19:30:23.148 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:23.146+0000 7f892202b700 1 -- 172.21.15.167:0/4194990805 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:32 5.17 5:e8000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x55a788009a00 con 0x7f890c007030 2023-10-05T19:30:23.149 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:23.146+0000 7f892282c700 1 -- 172.21.15.167:0/4194990805 <== osd.0 v2:172.21.15.118:6800/27690 9 ==== osd_op_reply(32 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f891c01cce0 con 0x7f890c007030 2023-10-05T19:30:23.150 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:23.146+0000 7f892282c700 1 -- 172.21.15.167:0/4194990805 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:33 5.f 5:f0000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x55a788009a00 con 0x55a787ff22f0 2023-10-05T19:30:23.150 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:23.146+0000 7f892282c700 1 -- 172.21.15.167:0/4194990805 <== osd.1 v2:172.21.15.167:6800/30035 10 ==== osd_op_reply(33 [pgnls start_epoch 32 out=115b] v32'3 uv3 ondisk = 1) v8 ==== 144+0+115 (crc 0 0 0) 0x7f891c01cce0 con 0x55a787ff22f0 2023-10-05T19:30:23.150 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:23.146+0000 7f892282c700 1 -- 172.21.15.167:0/4194990805 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:34 5.1f 5:f8000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x55a788009a00 con 0x7f891402d020 2023-10-05T19:30:23.151 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:23.146+0000 7f892182a700 1 -- 172.21.15.167:0/4194990805 <== osd.2 v2:172.21.15.167:6816/30037 6 ==== osd_op_reply(34 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f890c029040 con 0x7f891402d020 2023-10-05T19:30:23.151 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:23.146+0000 7f89244f6e40 20 RGWRados::pool_iterate: got zonegroup_info.4f1f719d-9648-417b-b10c-e5d939825419 2023-10-05T19:30:23.151 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:23.146+0000 7f89244f6e40 20 RGWRados::pool_iterate: got zone_info.9a9df965-6612-45d4-a45f-6e3c889bacc2 2023-10-05T19:30:23.151 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:23.146+0000 7f89244f6e40 20 RGWRados::pool_iterate: got zone_names.default 2023-10-05T19:30:23.152 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:23.146+0000 7f89244f6e40 20 RGWRados::pool_iterate: got zonegroups_names.default 2023-10-05T19:30:23.152 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:23.146+0000 7f89244f6e40 20 rados->read ofs=0 len=0 2023-10-05T19:30:23.152 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:23.146+0000 7f89244f6e40 1 -- 172.21.15.167:0/4194990805 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:35 5.1d 5:b8374eb4:::zone_info.9a9df965-6612-45d4-a45f-6e3c889bacc2:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32) v8 -- 0x55a788005bc0 con 0x55a787ff22f0 2023-10-05T19:30:23.152 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:23.150+0000 7f892282c700 1 -- 172.21.15.167:0/4194990805 <== osd.1 v2:172.21.15.167:6800/30035 11 ==== osd_op_reply(35 zone_info.9a9df965-6612-45d4-a45f-6e3c889bacc2 [read 0~1036 out=1036b] v0'0 uv3 ondisk = 0) v8 ==== 190+0+1036 (crc 0 0 0) 0x7f891c01cce0 con 0x55a787ff22f0 2023-10-05T19:30:23.152 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:23.150+0000 7f89244f6e40 20 rados_obj.operate() r=0 bl.length=1036 2023-10-05T19:30:23.152 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:23.150+0000 7f89244f6e40 1 -- 172.21.15.167:0/4194990805 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:36 5.1d 5:b8374eb4:::zone_info.9a9df965-6612-45d4-a45f-6e3c889bacc2:head [delete,create,writefull 0~874 in=874b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e32) v8 -- 0x55a788006000 con 0x55a787ff22f0 2023-10-05T19:30:23.155 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:23.150+0000 7f892282c700 1 -- 172.21.15.167:0/4194990805 <== osd.1 v2:172.21.15.167:6800/30035 12 ==== osd_op_reply(36 zone_info.9a9df965-6612-45d4-a45f-6e3c889bacc2 [delete,create,writefull 0~874] v32'4 uv4 ondisk = 0) v8 ==== 274+0+0 (crc 0 0 0) 0x7f891c01cce0 con 0x55a787ff22f0 2023-10-05T19:30:23.158 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:23.154+0000 7f89244f6e40 1 -- 172.21.15.167:0/4194990805 >> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] conn(0x7f890c007030 msgr2=0x7f890c01d430 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-05T19:30:23.158 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:23.154+0000 7f89244f6e40 1 --2- 172.21.15.167:0/4194990805 >> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] conn(0x7f890c007030 0x7f890c01d430 crc :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-05T19:30:23.158 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:23.154+0000 7f89244f6e40 1 -- 172.21.15.167:0/4194990805 >> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] conn(0x55a787ff22f0 msgr2=0x55a787ff2760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-05T19:30:23.158 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:23.154+0000 7f89244f6e40 1 --2- 172.21.15.167:0/4194990805 >> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] conn(0x55a787ff22f0 0x55a787ff2760 crc :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-05T19:30:23.158 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:23.154+0000 7f89244f6e40 1 -- 172.21.15.167:0/4194990805 >> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] conn(0x7f891402d020 msgr2=0x7f8914044730 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-05T19:30:23.158 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:23.154+0000 7f89244f6e40 1 --2- 172.21.15.167:0/4194990805 >> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] conn(0x7f891402d020 0x7f8914044730 crc :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-05T19:30:23.158 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:23.154+0000 7f89244f6e40 1 -- 172.21.15.167:0/4194990805 >> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] conn(0x55a7880070e0 msgr2=0x55a7880094c0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-05T19:30:23.158 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:23.154+0000 7f89244f6e40 1 --2- 172.21.15.167:0/4194990805 >> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] conn(0x55a7880070e0 0x55a7880094c0 crc :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-05T19:30:23.158 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:23.154+0000 7f89244f6e40 1 -- 172.21.15.167:0/4194990805 >> [v2:172.21.15.167:6824/29889,v1:172.21.15.167:6825/29889] conn(0x7f8904030c40 msgr2=0x7f89040330f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-05T19:30:23.159 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:23.154+0000 7f89244f6e40 1 --2- 172.21.15.167:0/4194990805 >> [v2:172.21.15.167:6824/29889,v1:172.21.15.167:6825/29889] conn(0x7f8904030c40 0x7f89040330f0 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7f891402dfd0 tx=0x7f8914043570 comp rx=0 tx=0).stop 2023-10-05T19:30:23.159 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:23.154+0000 7f89244f6e40 1 -- 172.21.15.167:0/4194990805 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55a787f96080 msgr2=0x55a787fbbda0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-05T19:30:23.159 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:23.154+0000 7f89244f6e40 1 --2- 172.21.15.167:0/4194990805 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55a787f96080 0x55a787fbbda0 secure :-1 s=READY pgs=695 cs=0 l=1 rev1=1 crypto rx=0x7f891400b960 tx=0x7f8914014000 comp rx=0 tx=0).stop 2023-10-05T19:30:23.159 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:23.154+0000 7f892282c700 1 -- 172.21.15.167:0/4194990805 reap_dead start 2023-10-05T19:30:23.159 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:23.154+0000 7f89244f6e40 1 -- 172.21.15.167:0/4194990805 shutdown_connections 2023-10-05T19:30:23.159 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:23.154+0000 7f89244f6e40 1 -- 172.21.15.167:0/4194990805 >> 172.21.15.167:0/4194990805 conn(0x55a787f09880 msgr2=0x55a787faf5a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-05T19:30:23.159 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:23.154+0000 7f89244f6e40 1 -- 172.21.15.167:0/4194990805 shutdown_connections 2023-10-05T19:30:23.159 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:23.154+0000 7f89244f6e40 1 -- 172.21.15.167:0/4194990805 wait complete. 2023-10-05T19:30:23.160 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:23.154+0000 7f89244f6e40 1 -- 172.21.15.167:0/2976557353 >> [v2:172.21.15.167:6824/29889,v1:172.21.15.167:6825/29889] conn(0x7f8870030c50 msgr2=0x7f8870033100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-05T19:30:23.160 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:23.154+0000 7f89244f6e40 1 --2- 172.21.15.167:0/2976557353 >> [v2:172.21.15.167:6824/29889,v1:172.21.15.167:6825/29889] conn(0x7f8870030c50 0x7f8870033100 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7f890c005cc0 tx=0x7f890c014000 comp rx=0 tx=0).stop 2023-10-05T19:30:23.160 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:23.154+0000 7f89244f6e40 1 -- 172.21.15.167:0/2976557353 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55a787fcd4d0 msgr2=0x55a787fd4610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-05T19:30:23.161 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:23.154+0000 7f89244f6e40 1 --2- 172.21.15.167:0/2976557353 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55a787fcd4d0 0x55a787fd4610 secure :-1 s=READY pgs=697 cs=0 l=1 rev1=1 crypto rx=0x7f891c00ab00 tx=0x7f891c014000 comp rx=0 tx=0).stop 2023-10-05T19:30:23.161 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:23.154+0000 7f89244f6e40 1 -- 172.21.15.167:0/2976557353 shutdown_connections 2023-10-05T19:30:23.161 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:23.154+0000 7f89244f6e40 1 --2- 172.21.15.167:0/2976557353 >> [v2:172.21.15.167:6824/29889,v1:172.21.15.167:6825/29889] conn(0x7f8870030c50 0x7f8870033100 unknown :-1 s=CLOSED pgs=239 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-05T19:30:23.161 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:23.154+0000 7f89244f6e40 1 --2- 172.21.15.167:0/2976557353 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55a787fcd4d0 0x55a787fd4610 unknown :-1 s=CLOSED pgs=697 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-05T19:30:23.161 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:23.154+0000 7f89244f6e40 1 -- 172.21.15.167:0/2976557353 >> 172.21.15.167:0/2976557353 conn(0x55a787f9a610 msgr2=0x55a787fc9e60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-05T19:30:23.161 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:23.154+0000 7f89244f6e40 1 -- 172.21.15.167:0/2976557353 shutdown_connections 2023-10-05T19:30:23.161 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:23.154+0000 7f89244f6e40 1 -- 172.21.15.167:0/2976557353 wait complete. 2023-10-05T19:30:23.162 INFO:teuthology.orchestra.run.smithi167.stdout:{ 2023-10-05T19:30:23.162 INFO:teuthology.orchestra.run.smithi167.stdout: "id": "9a9df965-6612-45d4-a45f-6e3c889bacc2", 2023-10-05T19:30:23.162 INFO:teuthology.orchestra.run.smithi167.stdout: "name": "default", 2023-10-05T19:30:23.162 INFO:teuthology.orchestra.run.smithi167.stdout: "domain_root": "default.rgw.meta:root", 2023-10-05T19:30:23.162 INFO:teuthology.orchestra.run.smithi167.stdout: "control_pool": "default.rgw.control", 2023-10-05T19:30:23.162 INFO:teuthology.orchestra.run.smithi167.stdout: "gc_pool": "default.rgw.log:gc", 2023-10-05T19:30:23.162 INFO:teuthology.orchestra.run.smithi167.stdout: "lc_pool": "default.rgw.log:lc", 2023-10-05T19:30:23.163 INFO:teuthology.orchestra.run.smithi167.stdout: "log_pool": "default.rgw.log", 2023-10-05T19:30:23.163 INFO:teuthology.orchestra.run.smithi167.stdout: "intent_log_pool": "default.rgw.log:intent", 2023-10-05T19:30:23.163 INFO:teuthology.orchestra.run.smithi167.stdout: "usage_log_pool": "default.rgw.log:usage", 2023-10-05T19:30:23.163 INFO:teuthology.orchestra.run.smithi167.stdout: "roles_pool": "default.rgw.meta:roles", 2023-10-05T19:30:23.163 INFO:teuthology.orchestra.run.smithi167.stdout: "reshard_pool": "default.rgw.log:reshard", 2023-10-05T19:30:23.163 INFO:teuthology.orchestra.run.smithi167.stdout: "user_keys_pool": "default.rgw.meta:users.keys", 2023-10-05T19:30:23.163 INFO:teuthology.orchestra.run.smithi167.stdout: "user_email_pool": "default.rgw.meta:users.email", 2023-10-05T19:30:23.163 INFO:teuthology.orchestra.run.smithi167.stdout: "user_swift_pool": "default.rgw.meta:users.swift", 2023-10-05T19:30:23.163 INFO:teuthology.orchestra.run.smithi167.stdout: "user_uid_pool": "default.rgw.meta:users.uid", 2023-10-05T19:30:23.163 INFO:teuthology.orchestra.run.smithi167.stdout: "otp_pool": "default.rgw.otp", 2023-10-05T19:30:23.163 INFO:teuthology.orchestra.run.smithi167.stdout: "system_key": { 2023-10-05T19:30:23.163 INFO:teuthology.orchestra.run.smithi167.stdout: "access_key": "", 2023-10-05T19:30:23.164 INFO:teuthology.orchestra.run.smithi167.stdout: "secret_key": "" 2023-10-05T19:30:23.164 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2023-10-05T19:30:23.164 INFO:teuthology.orchestra.run.smithi167.stdout: "placement_pools": [ 2023-10-05T19:30:23.164 INFO:teuthology.orchestra.run.smithi167.stdout: { 2023-10-05T19:30:23.164 INFO:teuthology.orchestra.run.smithi167.stdout: "key": "default-placement", 2023-10-05T19:30:23.164 INFO:teuthology.orchestra.run.smithi167.stdout: "val": { 2023-10-05T19:30:23.164 INFO:teuthology.orchestra.run.smithi167.stdout: "index_pool": "default.rgw.buckets.index", 2023-10-05T19:30:23.164 INFO:teuthology.orchestra.run.smithi167.stdout: "storage_classes": { 2023-10-05T19:30:23.164 INFO:teuthology.orchestra.run.smithi167.stdout: "STANDARD": { 2023-10-05T19:30:23.164 INFO:teuthology.orchestra.run.smithi167.stdout: "data_pool": "default.rgw.buckets.data" 2023-10-05T19:30:23.164 INFO:teuthology.orchestra.run.smithi167.stdout: } 2023-10-05T19:30:23.164 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2023-10-05T19:30:23.165 INFO:teuthology.orchestra.run.smithi167.stdout: "data_extra_pool": "default.rgw.buckets.non-ec", 2023-10-05T19:30:23.165 INFO:teuthology.orchestra.run.smithi167.stdout: "index_type": 0, 2023-10-05T19:30:23.165 INFO:teuthology.orchestra.run.smithi167.stdout: "inline_data": "true" 2023-10-05T19:30:23.165 INFO:teuthology.orchestra.run.smithi167.stdout: } 2023-10-05T19:30:23.165 INFO:teuthology.orchestra.run.smithi167.stdout: } 2023-10-05T19:30:23.165 INFO:teuthology.orchestra.run.smithi167.stdout: ], 2023-10-05T19:30:23.165 INFO:teuthology.orchestra.run.smithi167.stdout: "realm_id": "", 2023-10-05T19:30:23.165 INFO:teuthology.orchestra.run.smithi167.stdout: "notif_pool": "default.rgw.log:notif" 2023-10-05T19:30:23.165 INFO:teuthology.orchestra.run.smithi167.stdout:} 2023-10-05T19:30:23.166 DEBUG:tasks.util.rgw: json result: {'id': '9a9df965-6612-45d4-a45f-6e3c889bacc2', 'name': 'default', 'domain_root': 'default.rgw.meta:root', 'control_pool': 'default.rgw.control', 'gc_pool': 'default.rgw.log:gc', 'lc_pool': 'default.rgw.log:lc', 'log_pool': 'default.rgw.log', 'intent_log_pool': 'default.rgw.log:intent', 'usage_log_pool': 'default.rgw.log:usage', 'roles_pool': 'default.rgw.meta:roles', 'reshard_pool': 'default.rgw.log:reshard', 'user_keys_pool': 'default.rgw.meta:users.keys', 'user_email_pool': 'default.rgw.meta:users.email', 'user_swift_pool': 'default.rgw.meta:users.swift', 'user_uid_pool': 'default.rgw.meta:users.uid', 'otp_pool': 'default.rgw.otp', 'system_key': {'access_key': '', 'secret_key': ''}, 'placement_pools': [{'key': 'default-placement', 'val': {'index_pool': 'default.rgw.buckets.index', 'storage_classes': {'STANDARD': {'data_pool': 'default.rgw.buckets.data'}}, 'data_extra_pool': 'default.rgw.buckets.non-ec', 'index_type': 0, 'inline_data': 'true'}}], 'realm_id': '', 'notif_pool': 'default.rgw.log:notif'} 2023-10-05T19:30:23.166 INFO:tasks.util.rgw:rgwadmin: client.0 : ['zonegroup', 'get'] 2023-10-05T19:30:23.166 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'ceph', 'zonegroup', 'get'] 2023-10-05T19:30:23.166 DEBUG:teuthology.orchestra.run.smithi167:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster ceph zonegroup get 2023-10-05T19:30:23.244 INFO:teuthology.orchestra.run.smithi167.stderr:ignoring --setuser ceph since I am not root 2023-10-05T19:30:23.244 INFO:teuthology.orchestra.run.smithi167.stderr:ignoring --setgroup ceph since I am not root 2023-10-05T19:30:23.245 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:23.242+0000 7f29b8e4de40 1 Processor -- start 2023-10-05T19:30:23.245 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:23.242+0000 7f29b8e4de40 1 -- start start 2023-10-05T19:30:23.245 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:23.242+0000 7f29b8e4de40 1 --2- >> v2:172.21.15.118:3300/0 conn(0x564a407e1ff0 0x564a407e23c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-05T19:30:23.245 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:23.242+0000 7f29b8e4de40 1 -- --> v1:172.21.15.118:6789/0 -- auth(proto 0 26 bytes epoch 0) v1 -- 0x564a4072cc00 con 0x564a407e2900 2023-10-05T19:30:23.245 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:23.242+0000 7f29b8e4de40 1 -- --> v2:172.21.15.118:3300/0 -- mon_getmap magic: 0 v1 -- 0x564a406e9ee0 con 0x564a407e1ff0 2023-10-05T19:30:23.246 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:23.242+0000 7f29b6982700 1 --2- >> v2:172.21.15.118:3300/0 conn(0x564a407e1ff0 0x564a407e23c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-05T19:30:23.246 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:23.242+0000 7f29b6181700 1 --1- >> v1:172.21.15.118:6789/0 conn(0x564a407e2900 0x564a407e33a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.118:6789/0 says I am v1:172.21.15.167:50848/0 (socket says 172.21.15.167:50848) 2023-10-05T19:30:23.246 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:23.242+0000 7f29b6181700 1 -- 172.21.15.167:0/3726920701 learned_addr learned my addr 172.21.15.167:0/3726920701 (peer_addr_for_me v1:172.21.15.167:0/0) 2023-10-05T19:30:23.247 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:23.242+0000 7f29b5980700 1 -- 172.21.15.167:0/3726920701 <== mon.0 v1:172.21.15.118:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3955415416 0 0) 0x564a4072cc00 con 0x564a407e2900 2023-10-05T19:30:23.247 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:23.242+0000 7f29b5980700 1 -- 172.21.15.167:0/3726920701 --> v1:172.21.15.118:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f29a4001e10 con 0x564a407e2900 2023-10-05T19:30:23.247 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:23.242+0000 7f29b6982700 1 -- 172.21.15.167:0/3726920701 >> v1:172.21.15.118:6789/0 conn(0x564a407e2900 legacy=0x564a407e33a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-05T19:30:23.247 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:23.242+0000 7f29b6982700 1 -- 172.21.15.167:0/3726920701 --> v2:172.21.15.118:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x564a407e3ab0 con 0x564a407e1ff0 2023-10-05T19:30:23.248 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:23.242+0000 7f29b6982700 1 --2- 172.21.15.167:0/3726920701 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x564a407e1ff0 0x564a407e23c0 secure :-1 s=READY pgs=699 cs=0 l=1 rev1=1 crypto rx=0x7f29b000a520 tx=0x7f29b0005d40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=24611c65359a8ccf server_cookie=0 in_seq=0 out_seq=0 2023-10-05T19:30:23.248 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:23.242+0000 7f29b5980700 1 -- 172.21.15.167:0/3726920701 <== mon.0 v2:172.21.15.118:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f29b000f590 con 0x564a407e1ff0 2023-10-05T19:30:23.248 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:23.242+0000 7f29b5980700 1 -- 172.21.15.167:0/3726920701 <== mon.0 v2:172.21.15.118:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f29b000f6f0 con 0x564a407e1ff0 2023-10-05T19:30:23.248 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:23.242+0000 7f29b5980700 1 -- 172.21.15.167:0/3726920701 <== mon.0 v2:172.21.15.118:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f29b000f860 con 0x564a407e1ff0 2023-10-05T19:30:23.249 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:23.246+0000 7f29b8e4de40 1 -- 172.21.15.167:0/3726920701 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x564a407e1ff0 msgr2=0x564a407e23c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-05T19:30:23.249 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:23.246+0000 7f29b8e4de40 1 --2- 172.21.15.167:0/3726920701 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x564a407e1ff0 0x564a407e23c0 secure :-1 s=READY pgs=699 cs=0 l=1 rev1=1 crypto rx=0x7f29b000a520 tx=0x7f29b0005d40 comp rx=0 tx=0).stop 2023-10-05T19:30:23.249 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:23.246+0000 7f29b8e4de40 1 -- 172.21.15.167:0/3726920701 shutdown_connections 2023-10-05T19:30:23.249 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:23.246+0000 7f29b8e4de40 1 --2- 172.21.15.167:0/3726920701 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x564a407e1ff0 0x564a407e23c0 unknown :-1 s=CLOSED pgs=699 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-05T19:30:23.249 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:23.246+0000 7f29b8e4de40 1 -- 172.21.15.167:0/3726920701 >> 172.21.15.167:0/3726920701 conn(0x564a40755840 msgr2=0x564a407e04e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-05T19:30:23.249 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:23.246+0000 7f29b8e4de40 1 -- 172.21.15.167:0/3726920701 shutdown_connections 2023-10-05T19:30:23.249 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:23.246+0000 7f29b8e4de40 1 -- 172.21.15.167:0/3726920701 wait complete. 2023-10-05T19:30:23.250 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:23.246+0000 7f29b8e4de40 1 Processor -- start 2023-10-05T19:30:23.250 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:23.246+0000 7f29b8e4de40 1 -- start start 2023-10-05T19:30:23.250 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:23.246+0000 7f29b8e4de40 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x564a407e1ff0 0x564a40800e60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-05T19:30:23.250 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:23.246+0000 7f29b8e4de40 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x564a406ade30 con 0x564a407e1ff0 2023-10-05T19:30:23.251 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:23.246+0000 7f29b6982700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x564a407e1ff0 0x564a40800e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-05T19:30:23.251 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:23.246+0000 7f29b6982700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x564a407e1ff0 0x564a40800e60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.118:3300/0 says I am v2:172.21.15.167:33278/0 (socket says 172.21.15.167:33278) 2023-10-05T19:30:23.251 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:23.246+0000 7f29b6982700 1 -- 172.21.15.167:0/2114753728 learned_addr learned my addr 172.21.15.167:0/2114753728 (peer_addr_for_me v2:172.21.15.167:0/0) 2023-10-05T19:30:23.251 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:23.246+0000 7f29b6982700 1 -- 172.21.15.167:0/2114753728 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f29b000a0e0 con 0x564a407e1ff0 2023-10-05T19:30:23.251 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:23.246+0000 7f29b6982700 1 --2- 172.21.15.167:0/2114753728 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x564a407e1ff0 0x564a40800e60 secure :-1 s=READY pgs=700 cs=0 l=1 rev1=1 crypto rx=0x7f29b000a470 tx=0x7f29b0014000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=64a1adc3d2a9fb2 server_cookie=0 in_seq=0 out_seq=0 2023-10-05T19:30:23.252 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:23.246+0000 7f29b497e700 1 -- 172.21.15.167:0/2114753728 <== mon.0 v2:172.21.15.118:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f29b000f860 con 0x564a407e1ff0 2023-10-05T19:30:23.252 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:23.246+0000 7f29b497e700 1 -- 172.21.15.167:0/2114753728 <== mon.0 v2:172.21.15.118:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f29b000f9c0 con 0x564a407e1ff0 2023-10-05T19:30:23.252 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:23.246+0000 7f29b497e700 1 -- 172.21.15.167:0/2114753728 <== mon.0 v2:172.21.15.118:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f29b000fb90 con 0x564a407e1ff0 2023-10-05T19:30:23.253 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:23.246+0000 7f29b8e4de40 1 -- 172.21.15.167:0/2114753728 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x564a407e1ff0 msgr2=0x564a40800e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-05T19:30:23.253 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:23.246+0000 7f29b8e4de40 1 --2- 172.21.15.167:0/2114753728 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x564a407e1ff0 0x564a40800e60 secure :-1 s=READY pgs=700 cs=0 l=1 rev1=1 crypto rx=0x7f29b000a470 tx=0x7f29b0014000 comp rx=0 tx=0).stop 2023-10-05T19:30:23.253 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:23.246+0000 7f29b8e4de40 1 -- 172.21.15.167:0/2114753728 shutdown_connections 2023-10-05T19:30:23.253 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:23.246+0000 7f29b8e4de40 1 --2- 172.21.15.167:0/2114753728 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x564a407e1ff0 0x564a40800e60 unknown :-1 s=CLOSED pgs=700 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-05T19:30:23.253 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:23.246+0000 7f29b8e4de40 1 -- 172.21.15.167:0/2114753728 >> 172.21.15.167:0/2114753728 conn(0x564a40755840 msgr2=0x564a407fca20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-05T19:30:23.253 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:23.246+0000 7f29b8e4de40 1 -- 172.21.15.167:0/2114753728 shutdown_connections 2023-10-05T19:30:23.253 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:23.250+0000 7f29b8e4de40 1 -- 172.21.15.167:0/2114753728 wait complete. 2023-10-05T19:30:23.253 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:23.250+0000 7f29b8e4de40 1 Processor -- start 2023-10-05T19:30:23.253 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:23.250+0000 7f29b8e4de40 1 -- start start 2023-10-05T19:30:23.253 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:23.250+0000 7f29b8e4de40 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x564a407e1ff0 0x564a40807ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-05T19:30:23.253 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:23.250+0000 7f29b8e4de40 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x564a40680ed0 con 0x564a407e1ff0 2023-10-05T19:30:23.254 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:23.250+0000 7f29b6982700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x564a407e1ff0 0x564a40807ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-05T19:30:23.255 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:23.250+0000 7f29b6982700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x564a407e1ff0 0x564a40807ac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.118:3300/0 says I am v2:172.21.15.167:33290/0 (socket says 172.21.15.167:33290) 2023-10-05T19:30:23.255 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:23.250+0000 7f29b6982700 1 -- 172.21.15.167:0/1496643883 learned_addr learned my addr 172.21.15.167:0/1496643883 (peer_addr_for_me v2:172.21.15.167:0/0) 2023-10-05T19:30:23.255 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:23.250+0000 7f29b6982700 1 -- 172.21.15.167:0/1496643883 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x564a407e3ab0 con 0x564a407e1ff0 2023-10-05T19:30:23.255 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:23.250+0000 7f29b6982700 1 --2- 172.21.15.167:0/1496643883 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x564a407e1ff0 0x564a40807ac0 secure :-1 s=READY pgs=701 cs=0 l=1 rev1=1 crypto rx=0x7f29b000adc0 tx=0x7f29b001d000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-05T19:30:23.255 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:23.250+0000 7f29aeffd700 1 -- 172.21.15.167:0/1496643883 <== mon.0 v2:172.21.15.118:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f29b002f430 con 0x564a407e1ff0 2023-10-05T19:30:23.255 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:23.250+0000 7f29aeffd700 1 -- 172.21.15.167:0/1496643883 <== mon.0 v2:172.21.15.118:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f29b002f5b0 con 0x564a407e1ff0 2023-10-05T19:30:23.255 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:23.250+0000 7f29b8e4de40 1 -- 172.21.15.167:0/1496643883 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x564a40808000 con 0x564a407e1ff0 2023-10-05T19:30:23.255 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:23.250+0000 7f29aeffd700 1 -- 172.21.15.167:0/1496643883 <== mon.0 v2:172.21.15.118:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f29b002f760 con 0x564a407e1ff0 2023-10-05T19:30:23.255 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:23.250+0000 7f29b8e4de40 1 -- 172.21.15.167:0/1496643883 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x564a408084a0 con 0x564a407e1ff0 2023-10-05T19:30:23.256 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:23.254+0000 7f29b8e4de40 20 check_secure_mon_conn(): auth registy supported: methods=[2] modes=[2,1] 2023-10-05T19:30:23.256 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:23.254+0000 7f29b8e4de40 20 check_secure_mon_conn(): mode 1 is insecure 2023-10-05T19:30:23.256 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:23.254+0000 7f29b8e4de40 1 Processor -- start 2023-10-05T19:30:23.256 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:23.254+0000 7f29aeffd700 1 -- 172.21.15.167:0/1496643883 <== mon.0 v2:172.21.15.118:3300/0 4 ==== mgrmap(e 5) v1 ==== 40329+0+0 (secure 0 0 0) 0x7f29b0010420 con 0x564a407e1ff0 2023-10-05T19:30:23.256 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:23.254+0000 7f29b8e4de40 1 -- start start 2023-10-05T19:30:23.257 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:23.254+0000 7f29b8e4de40 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x564a40819200 0x564a408195d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-05T19:30:23.258 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:23.254+0000 7f29b8e4de40 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x564a40819b10 con 0x564a40819200 2023-10-05T19:30:23.258 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:23.254+0000 7f29aeffd700 1 --2- 172.21.15.167:0/1496643883 >> [v2:172.21.15.167:6824/29889,v1:172.21.15.167:6825/29889] conn(0x7f2998030c50 0x7f2998033100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-05T19:30:23.258 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:23.254+0000 7f29aeffd700 1 -- 172.21.15.167:0/1496643883 <== mon.0 v2:172.21.15.118:3300/0 5 ==== osd_map(32..32 src has 1..32) v4 ==== 6260+0+0 (secure 0 0 0) 0x7f29b002b070 con 0x564a407e1ff0 2023-10-05T19:30:23.258 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:23.254+0000 7f29b7183700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x564a40819200 0x564a408195d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-05T19:30:23.258 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:23.254+0000 7f29b6982700 1 --2- 172.21.15.167:0/1496643883 >> [v2:172.21.15.167:6824/29889,v1:172.21.15.167:6825/29889] conn(0x7f2998030c50 0x7f2998033100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-05T19:30:23.258 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:23.254+0000 7f29b7183700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x564a40819200 0x564a408195d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.118:3300/0 says I am v2:172.21.15.167:33292/0 (socket says 172.21.15.167:33292) 2023-10-05T19:30:23.258 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:23.254+0000 7f29b7183700 1 -- 172.21.15.167:0/340643843 learned_addr learned my addr 172.21.15.167:0/340643843 (peer_addr_for_me v2:172.21.15.167:0/0) 2023-10-05T19:30:23.258 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:23.254+0000 7f29b6982700 1 --2- 172.21.15.167:0/1496643883 >> [v2:172.21.15.167:6824/29889,v1:172.21.15.167:6825/29889] conn(0x7f2998030c50 0x7f2998033100 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7f29b003a130 tx=0x7f29b003a000 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-05T19:30:23.258 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:23.254+0000 7f29b7183700 1 -- 172.21.15.167:0/340643843 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x564a40819c50 con 0x564a40819200 2023-10-05T19:30:23.259 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:23.254+0000 7f29b7183700 1 --2- 172.21.15.167:0/340643843 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x564a40819200 0x564a408195d0 secure :-1 s=READY pgs=702 cs=0 l=1 rev1=1 crypto rx=0x7f29a800a320 tx=0x7f29a80059d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=96dbc851130bc0b3 server_cookie=0 in_seq=0 out_seq=0 2023-10-05T19:30:23.259 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:23.254+0000 7f2953fff700 1 -- 172.21.15.167:0/340643843 <== mon.0 v2:172.21.15.118:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f29a800ae30 con 0x564a40819200 2023-10-05T19:30:23.260 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:23.254+0000 7f2953fff700 1 -- 172.21.15.167:0/340643843 <== mon.0 v2:172.21.15.118:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f29a800c610 con 0x564a40819200 2023-10-05T19:30:23.260 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:23.254+0000 7f2953fff700 1 -- 172.21.15.167:0/340643843 <== mon.0 v2:172.21.15.118:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f29a800c7c0 con 0x564a40819200 2023-10-05T19:30:23.260 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:23.254+0000 7f29b8e4de40 1 -- 172.21.15.167:0/340643843 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x564a40819200 msgr2=0x564a408195d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-05T19:30:23.260 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:23.254+0000 7f29b8e4de40 1 --2- 172.21.15.167:0/340643843 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x564a40819200 0x564a408195d0 secure :-1 s=READY pgs=702 cs=0 l=1 rev1=1 crypto rx=0x7f29a800a320 tx=0x7f29a80059d0 comp rx=0 tx=0).stop 2023-10-05T19:30:23.260 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:23.254+0000 7f29b8e4de40 1 -- 172.21.15.167:0/340643843 shutdown_connections 2023-10-05T19:30:23.260 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:23.254+0000 7f29b8e4de40 1 --2- 172.21.15.167:0/340643843 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x564a40819200 0x564a408195d0 unknown :-1 s=CLOSED pgs=702 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-05T19:30:23.260 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:23.254+0000 7f29b8e4de40 1 -- 172.21.15.167:0/340643843 >> 172.21.15.167:0/340643843 conn(0x564a407e2900 msgr2=0x564a40815f50 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-05T19:30:23.260 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:23.254+0000 7f29b8e4de40 1 -- 172.21.15.167:0/340643843 shutdown_connections 2023-10-05T19:30:23.260 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:23.254+0000 7f29b8e4de40 1 -- 172.21.15.167:0/340643843 wait complete. 2023-10-05T19:30:23.261 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:23.258+0000 7f29b8e4de40 1 Processor -- start 2023-10-05T19:30:23.261 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:23.258+0000 7f29b8e4de40 1 -- start start 2023-10-05T19:30:23.261 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:23.258+0000 7f29b8e4de40 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x564a40819200 0x564a408203e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-05T19:30:23.261 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:23.258+0000 7f29b8e4de40 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x564a408213e0 con 0x564a40819200 2023-10-05T19:30:23.262 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:23.258+0000 7f29b7183700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x564a40819200 0x564a408203e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-05T19:30:23.262 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:23.258+0000 7f29b7183700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x564a40819200 0x564a408203e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.118:3300/0 says I am v2:172.21.15.167:33298/0 (socket says 172.21.15.167:33298) 2023-10-05T19:30:23.262 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:23.258+0000 7f29b7183700 1 -- 172.21.15.167:0/2550600892 learned_addr learned my addr 172.21.15.167:0/2550600892 (peer_addr_for_me v2:172.21.15.167:0/0) 2023-10-05T19:30:23.262 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:23.258+0000 7f29b7183700 1 -- 172.21.15.167:0/2550600892 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f29a8005c90 con 0x564a40819200 2023-10-05T19:30:23.263 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:23.258+0000 7f29b7183700 1 --2- 172.21.15.167:0/2550600892 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x564a40819200 0x564a408203e0 secure :-1 s=READY pgs=703 cs=0 l=1 rev1=1 crypto rx=0x7f29a800aa70 tx=0x7f29a8013000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-05T19:30:23.263 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:23.258+0000 7f29527fc700 1 -- 172.21.15.167:0/2550600892 <== mon.0 v2:172.21.15.118:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f29a8015660 con 0x564a40819200 2023-10-05T19:30:23.263 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:23.258+0000 7f29b8e4de40 1 -- 172.21.15.167:0/2550600892 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x564a408209e0 con 0x564a40819200 2023-10-05T19:30:23.263 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:23.258+0000 7f29527fc700 1 -- 172.21.15.167:0/2550600892 <== mon.0 v2:172.21.15.118:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f29a80157c0 con 0x564a40819200 2023-10-05T19:30:23.263 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:23.258+0000 7f29527fc700 1 -- 172.21.15.167:0/2550600892 <== mon.0 v2:172.21.15.118:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f29a8015a70 con 0x564a40819200 2023-10-05T19:30:23.263 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:23.258+0000 7f29b8e4de40 1 -- 172.21.15.167:0/2550600892 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x564a40820ea0 con 0x564a40819200 2023-10-05T19:30:23.264 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:23.262+0000 7f29b8e4de40 20 try_to_resolve_local_entities(): before: realm_id= realm_name= zonegroup_id= zonegroup_name= zone_id= zone_name= 2023-10-05T19:30:23.264 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:23.262+0000 7f28fffff700 20 reqs_thread_entry: start 2023-10-05T19:30:23.264 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:23.262+0000 7f29b8e4de40 20 rados->read ofs=0 len=0 2023-10-05T19:30:23.265 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:23.262+0000 7f29b8e4de40 1 --2- 172.21.15.167:0/1496643883 >> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] conn(0x564a4083e1e0 0x564a4083e660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-05T19:30:23.265 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:23.262+0000 7f29b8e4de40 1 -- 172.21.15.167:0/1496643883 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:1 5.12 5:49953fa1:::default.realm:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32) v8 -- 0x564a4083eba0 con 0x564a4083e1e0 2023-10-05T19:30:23.265 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:23.262+0000 7f29527fc700 1 -- 172.21.15.167:0/2550600892 <== mon.0 v2:172.21.15.118:3300/0 4 ==== mgrmap(e 5) v1 ==== 40329+0+0 (secure 0 0 0) 0x7f29a8007050 con 0x564a40819200 2023-10-05T19:30:23.265 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:23.262+0000 7f29b6181700 1 --2- 172.21.15.167:0/1496643883 >> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] conn(0x564a4083e1e0 0x564a4083e660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-05T19:30:23.265 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:23.262+0000 7f29527fc700 1 --2- 172.21.15.167:0/2550600892 >> [v2:172.21.15.167:6824/29889,v1:172.21.15.167:6825/29889] conn(0x7f2904030c00 0x7f29040330b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-05T19:30:23.265 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:23.262+0000 7f29527fc700 1 -- 172.21.15.167:0/2550600892 <== mon.0 v2:172.21.15.118:3300/0 5 ==== osd_map(32..32 src has 1..32) v4 ==== 6260+0+0 (secure 0 0 0) 0x7f29a80441c0 con 0x564a40819200 2023-10-05T19:30:23.265 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:23.262+0000 7f29b7183700 1 --2- 172.21.15.167:0/2550600892 >> [v2:172.21.15.167:6824/29889,v1:172.21.15.167:6825/29889] conn(0x7f2904030c00 0x7f29040330b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-05T19:30:23.266 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:23.262+0000 7f29b6181700 1 --2- 172.21.15.167:0/1496643883 >> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] conn(0x564a4083e1e0 0x564a4083e660 crc :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-05T19:30:23.266 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:23.262+0000 7f29b7183700 1 --2- 172.21.15.167:0/2550600892 >> [v2:172.21.15.167:6824/29889,v1:172.21.15.167:6825/29889] conn(0x7f2904030c00 0x7f29040330b0 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7f29a800bb80 tx=0x7f29a8042410 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-05T19:30:23.266 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:23.262+0000 7f29b6181700 1 -- 172.21.15.167:0/1496643883 <== osd.1 v2:172.21.15.167:6800/30035 1 ==== osd_op_reply(1 default.realm [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 ==== 157+0+0 (crc 0 0 0) 0x7f29a000b4d0 con 0x564a4083e1e0 2023-10-05T19:30:23.266 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:23.262+0000 7f29b8e4de40 20 rados_obj.operate() r=-2 bl.length=0 2023-10-05T19:30:23.266 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:23.262+0000 7f29b8e4de40 20 rados->read ofs=0 len=0 2023-10-05T19:30:23.267 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:23.262+0000 7f29b8e4de40 1 -- 172.21.15.167:0/1496643883 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:2 5.f 5:f43b6ece:::zone_names.default:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32) v8 -- 0x564a40851720 con 0x564a4083e1e0 2023-10-05T19:30:23.267 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:23.262+0000 7f29b6181700 1 -- 172.21.15.167:0/1496643883 <== osd.1 v2:172.21.15.167:6800/30035 2 ==== osd_op_reply(2 zone_names.default [read 0~46 out=46b] v0'0 uv3 ondisk = 0) v8 ==== 162+0+46 (crc 0 0 0) 0x7f29a000b4d0 con 0x564a4083e1e0 2023-10-05T19:30:23.267 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:23.262+0000 7f29b8e4de40 20 rados_obj.operate() r=0 bl.length=46 2023-10-05T19:30:23.267 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:23.262+0000 7f29b8e4de40 20 rados->read ofs=0 len=0 2023-10-05T19:30:23.267 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:23.262+0000 7f29b8e4de40 1 -- 172.21.15.167:0/1496643883 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:3 5.1d 5:b8374eb4:::zone_info.9a9df965-6612-45d4-a45f-6e3c889bacc2:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32) v8 -- 0x564a408409c0 con 0x564a4083e1e0 2023-10-05T19:30:23.268 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:23.262+0000 7f29b6181700 1 -- 172.21.15.167:0/1496643883 <== osd.1 v2:172.21.15.167:6800/30035 3 ==== osd_op_reply(3 zone_info.9a9df965-6612-45d4-a45f-6e3c889bacc2 [read 0~874 out=874b] v0'0 uv4 ondisk = 0) v8 ==== 190+0+874 (crc 0 0 0) 0x7f29a000b4d0 con 0x564a4083e1e0 2023-10-05T19:30:23.268 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:23.262+0000 7f29b8e4de40 20 rados_obj.operate() r=0 bl.length=874 2023-10-05T19:30:23.268 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:23.262+0000 7f29b8e4de40 1 --2- 172.21.15.167:0/1496643883 >> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] conn(0x564a40852ff0 0x564a408553d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-05T19:30:23.268 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:23.262+0000 7f29b8e4de40 1 -- 172.21.15.167:0/1496643883 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:4 5.0 5:00000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x564a40855910 con 0x564a40852ff0 2023-10-05T19:30:23.268 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:23.262+0000 7f29b6982700 1 --2- 172.21.15.167:0/1496643883 >> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] conn(0x564a40852ff0 0x564a408553d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-05T19:30:23.268 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:23.266+0000 7f29b6982700 1 --2- 172.21.15.167:0/1496643883 >> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] conn(0x564a40852ff0 0x564a408553d0 crc :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-05T19:30:23.269 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:23.266+0000 7f29b6982700 1 -- 172.21.15.167:0/1496643883 <== osd.3 v2:172.21.15.167:6808/30036 1 ==== osd_op_reply(4 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f29b0024400 con 0x564a40852ff0 2023-10-05T19:30:23.269 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:23.266+0000 7f29b6982700 1 --2- 172.21.15.167:0/1496643883 >> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] conn(0x7f29b00383b0 0x7f29b002f8c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-05T19:30:23.269 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:23.266+0000 7f29b6982700 1 -- 172.21.15.167:0/1496643883 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:5 5.10 5:08000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x564a40855910 con 0x7f29b00383b0 2023-10-05T19:30:23.269 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:23.266+0000 7f29b6181700 1 --2- 172.21.15.167:0/1496643883 >> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] conn(0x7f29b00383b0 0x7f29b002f8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-05T19:30:23.270 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:23.266+0000 7f29b6181700 1 --2- 172.21.15.167:0/1496643883 >> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] conn(0x7f29b00383b0 0x7f29b002f8c0 crc :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-05T19:30:23.271 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:23.266+0000 7f29b6181700 1 -- 172.21.15.167:0/1496643883 <== osd.2 v2:172.21.15.167:6816/30037 1 ==== osd_op_reply(5 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f29a000b4d0 con 0x7f29b00383b0 2023-10-05T19:30:23.271 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:23.266+0000 7f29b6181700 1 --2- 172.21.15.167:0/1496643883 >> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] conn(0x7f29a0012040 0x7f29a0013420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-05T19:30:23.271 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:23.266+0000 7f29b6181700 1 -- 172.21.15.167:0/1496643883 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:6 5.8 5:10000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x564a40855910 con 0x7f29a0012040 2023-10-05T19:30:23.271 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:23.266+0000 7f29b7183700 1 --2- 172.21.15.167:0/1496643883 >> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] conn(0x7f29a0012040 0x7f29a0013420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-05T19:30:23.272 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:23.270+0000 7f29b7183700 1 --2- 172.21.15.167:0/1496643883 >> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] conn(0x7f29a0012040 0x7f29a0013420 crc :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-05T19:30:23.273 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:23.270+0000 7f29b7183700 1 -- 172.21.15.167:0/1496643883 <== osd.0 v2:172.21.15.118:6800/27690 1 ==== osd_op_reply(6 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f29a8008c80 con 0x7f29a0012040 2023-10-05T19:30:23.273 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:23.270+0000 7f29b7183700 1 -- 172.21.15.167:0/1496643883 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:7 5.18 5:18000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x564a40855910 con 0x564a4083e1e0 2023-10-05T19:30:23.273 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:23.270+0000 7f29b6181700 1 -- 172.21.15.167:0/1496643883 <== osd.1 v2:172.21.15.167:6800/30035 4 ==== osd_op_reply(7 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f29a000b4d0 con 0x564a4083e1e0 2023-10-05T19:30:23.273 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:23.270+0000 7f29b6181700 1 -- 172.21.15.167:0/1496643883 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:8 5.4 5:20000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x564a40855910 con 0x564a4083e1e0 2023-10-05T19:30:23.274 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:23.270+0000 7f29b6181700 1 -- 172.21.15.167:0/1496643883 <== osd.1 v2:172.21.15.167:6800/30035 5 ==== osd_op_reply(8 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f29a000b4d0 con 0x564a4083e1e0 2023-10-05T19:30:23.274 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:23.270+0000 7f29b6181700 1 -- 172.21.15.167:0/1496643883 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:9 5.14 5:28000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x564a40855910 con 0x7f29a0012040 2023-10-05T19:30:23.275 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:23.270+0000 7f29b7183700 1 -- 172.21.15.167:0/1496643883 <== osd.0 v2:172.21.15.118:6800/27690 2 ==== osd_op_reply(9 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f29a8008c80 con 0x7f29a0012040 2023-10-05T19:30:23.275 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:23.270+0000 7f29b7183700 1 -- 172.21.15.167:0/1496643883 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:10 5.c 5:30000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x564a40855910 con 0x7f29a0012040 2023-10-05T19:30:23.275 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:23.270+0000 7f29b7183700 1 -- 172.21.15.167:0/1496643883 <== osd.0 v2:172.21.15.118:6800/27690 3 ==== osd_op_reply(10 [pgnls start_epoch 32 out=112b] v23'1 uv1 ondisk = 1) v8 ==== 144+0+112 (crc 0 0 0) 0x7f29a8008c80 con 0x7f29a0012040 2023-10-05T19:30:23.275 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:23.274+0000 7f29b7183700 1 -- 172.21.15.167:0/1496643883 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:11 5.1c 5:38000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x564a40855910 con 0x7f29b00383b0 2023-10-05T19:30:23.276 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:23.274+0000 7f29b6181700 1 -- 172.21.15.167:0/1496643883 <== osd.2 v2:172.21.15.167:6816/30037 2 ==== osd_op_reply(11 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f29a000b4d0 con 0x7f29b00383b0 2023-10-05T19:30:23.276 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:23.274+0000 7f29b6181700 1 -- 172.21.15.167:0/1496643883 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:12 5.2 5:40000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x564a40855910 con 0x564a40852ff0 2023-10-05T19:30:23.277 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:23.274+0000 7f29b6982700 1 -- 172.21.15.167:0/1496643883 <== osd.3 v2:172.21.15.167:6808/30036 2 ==== osd_op_reply(12 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f29b0024400 con 0x564a40852ff0 2023-10-05T19:30:23.277 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:23.274+0000 7f29b6982700 1 -- 172.21.15.167:0/1496643883 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:13 5.12 5:48000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x564a40855910 con 0x564a4083e1e0 2023-10-05T19:30:23.277 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:23.274+0000 7f29b6181700 1 -- 172.21.15.167:0/1496643883 <== osd.1 v2:172.21.15.167:6800/30035 6 ==== osd_op_reply(13 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f29a000b4d0 con 0x564a4083e1e0 2023-10-05T19:30:23.278 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:23.274+0000 7f29b6181700 1 -- 172.21.15.167:0/1496643883 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:14 5.a 5:50000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x564a40855910 con 0x7f29a0012040 2023-10-05T19:30:23.278 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:23.274+0000 7f29b7183700 1 -- 172.21.15.167:0/1496643883 <== osd.0 v2:172.21.15.118:6800/27690 4 ==== osd_op_reply(14 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f29a8008c80 con 0x7f29a0012040 2023-10-05T19:30:23.279 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:23.274+0000 7f29b7183700 1 -- 172.21.15.167:0/1496643883 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:15 5.1a 5:58000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x564a40855910 con 0x564a4083e1e0 2023-10-05T19:30:23.279 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:23.274+0000 7f29b6181700 1 -- 172.21.15.167:0/1496643883 <== osd.1 v2:172.21.15.167:6800/30035 7 ==== osd_op_reply(15 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f29a000b4d0 con 0x564a4083e1e0 2023-10-05T19:30:23.279 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:23.274+0000 7f29b6181700 1 -- 172.21.15.167:0/1496643883 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:16 5.6 5:60000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x564a40855910 con 0x7f29b00383b0 2023-10-05T19:30:23.280 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:23.278+0000 7f29b6181700 1 -- 172.21.15.167:0/1496643883 <== osd.2 v2:172.21.15.167:6816/30037 3 ==== osd_op_reply(16 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f29a000b4d0 con 0x7f29b00383b0 2023-10-05T19:30:23.280 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:23.278+0000 7f29b6181700 1 -- 172.21.15.167:0/1496643883 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:17 5.16 5:68000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x564a40855910 con 0x564a40852ff0 2023-10-05T19:30:23.281 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:23.278+0000 7f29b6982700 1 -- 172.21.15.167:0/1496643883 <== osd.3 v2:172.21.15.167:6808/30036 3 ==== osd_op_reply(17 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f29b0024400 con 0x564a40852ff0 2023-10-05T19:30:23.281 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:23.278+0000 7f29b6982700 1 -- 172.21.15.167:0/1496643883 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:18 5.e 5:70000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x564a40855910 con 0x564a40852ff0 2023-10-05T19:30:23.282 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:23.278+0000 7f29b6982700 1 -- 172.21.15.167:0/1496643883 <== osd.3 v2:172.21.15.167:6808/30036 4 ==== osd_op_reply(18 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f29b0024400 con 0x564a40852ff0 2023-10-05T19:30:23.282 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:23.278+0000 7f29b6982700 1 -- 172.21.15.167:0/1496643883 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:19 5.1e 5:78000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x564a40855910 con 0x7f29a0012040 2023-10-05T19:30:23.283 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:23.278+0000 7f29b7183700 1 -- 172.21.15.167:0/1496643883 <== osd.0 v2:172.21.15.118:6800/27690 5 ==== osd_op_reply(19 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f29a8008c80 con 0x7f29a0012040 2023-10-05T19:30:23.283 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:23.278+0000 7f29b7183700 1 -- 172.21.15.167:0/1496643883 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:20 5.1 5:80000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x564a40855910 con 0x564a40852ff0 2023-10-05T19:30:23.283 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:23.278+0000 7f29b6982700 1 -- 172.21.15.167:0/1496643883 <== osd.3 v2:172.21.15.167:6808/30036 5 ==== osd_op_reply(20 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f29b0024400 con 0x564a40852ff0 2023-10-05T19:30:23.284 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:23.282+0000 7f29b6982700 1 -- 172.21.15.167:0/1496643883 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:21 5.11 5:88000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x564a40855910 con 0x564a40852ff0 2023-10-05T19:30:23.284 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:23.282+0000 7f29b6982700 1 -- 172.21.15.167:0/1496643883 <== osd.3 v2:172.21.15.167:6808/30036 6 ==== osd_op_reply(21 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f29b0024400 con 0x564a40852ff0 2023-10-05T19:30:23.284 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:23.282+0000 7f29b6982700 1 -- 172.21.15.167:0/1496643883 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:22 5.9 5:90000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x564a40855910 con 0x564a4083e1e0 2023-10-05T19:30:23.285 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:23.282+0000 7f29b6181700 1 -- 172.21.15.167:0/1496643883 <== osd.1 v2:172.21.15.167:6800/30035 8 ==== osd_op_reply(22 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f29a000b4d0 con 0x564a4083e1e0 2023-10-05T19:30:23.285 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:23.282+0000 7f29b6181700 1 -- 172.21.15.167:0/1496643883 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:23 5.19 5:98000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x564a40855910 con 0x564a4083e1e0 2023-10-05T19:30:23.286 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:23.282+0000 7f29b6181700 1 -- 172.21.15.167:0/1496643883 <== osd.1 v2:172.21.15.167:6800/30035 9 ==== osd_op_reply(23 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f29a000b4d0 con 0x564a4083e1e0 2023-10-05T19:30:23.286 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:23.282+0000 7f29b6181700 1 -- 172.21.15.167:0/1496643883 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:24 5.5 5:a0000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x564a40855910 con 0x7f29a0012040 2023-10-05T19:30:23.287 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:23.282+0000 7f29b7183700 1 -- 172.21.15.167:0/1496643883 <== osd.0 v2:172.21.15.118:6800/27690 6 ==== osd_op_reply(24 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f29a8008c80 con 0x7f29a0012040 2023-10-05T19:30:23.287 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:23.282+0000 7f29b7183700 1 -- 172.21.15.167:0/1496643883 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:25 5.15 5:a8000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x564a40855910 con 0x564a40852ff0 2023-10-05T19:30:23.288 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:23.286+0000 7f29b6982700 1 -- 172.21.15.167:0/1496643883 <== osd.3 v2:172.21.15.167:6808/30036 7 ==== osd_op_reply(25 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f29b0024400 con 0x564a40852ff0 2023-10-05T19:30:23.288 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:23.286+0000 7f29b6982700 1 -- 172.21.15.167:0/1496643883 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:26 5.d 5:b0000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x564a40855910 con 0x7f29b00383b0 2023-10-05T19:30:23.289 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:23.286+0000 7f29b6181700 1 -- 172.21.15.167:0/1496643883 <== osd.2 v2:172.21.15.167:6816/30037 4 ==== osd_op_reply(26 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f29a000b4d0 con 0x7f29b00383b0 2023-10-05T19:30:23.289 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:23.286+0000 7f29b6181700 1 -- 172.21.15.167:0/1496643883 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:27 5.1d 5:b8000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x564a40855910 con 0x564a4083e1e0 2023-10-05T19:30:23.289 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:23.286+0000 7f29b6181700 1 -- 172.21.15.167:0/1496643883 <== osd.1 v2:172.21.15.167:6800/30035 10 ==== osd_op_reply(27 [pgnls start_epoch 32 out=107b] v32'4 uv4 ondisk = 1) v8 ==== 144+0+107 (crc 0 0 0) 0x7f29a000b4d0 con 0x564a4083e1e0 2023-10-05T19:30:23.289 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:23.286+0000 7f29b6181700 1 -- 172.21.15.167:0/1496643883 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:28 5.3 5:c0000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x564a40855910 con 0x7f29a0012040 2023-10-05T19:30:23.290 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:23.286+0000 7f29b7183700 1 -- 172.21.15.167:0/1496643883 <== osd.0 v2:172.21.15.118:6800/27690 7 ==== osd_op_reply(28 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f29a8008c80 con 0x7f29a0012040 2023-10-05T19:30:23.290 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:23.286+0000 7f29b7183700 1 -- 172.21.15.167:0/1496643883 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:29 5.13 5:c8000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x564a40855910 con 0x564a40852ff0 2023-10-05T19:30:23.291 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:23.286+0000 7f29b6982700 1 -- 172.21.15.167:0/1496643883 <== osd.3 v2:172.21.15.167:6808/30036 8 ==== osd_op_reply(29 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f29b0024400 con 0x564a40852ff0 2023-10-05T19:30:23.291 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:23.286+0000 7f29b6982700 1 -- 172.21.15.167:0/1496643883 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:30 5.b 5:d0000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x564a40855910 con 0x7f29a0012040 2023-10-05T19:30:23.292 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:23.290+0000 7f29b7183700 1 -- 172.21.15.167:0/1496643883 <== osd.0 v2:172.21.15.118:6800/27690 8 ==== osd_op_reply(30 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f29a8008c80 con 0x7f29a0012040 2023-10-05T19:30:23.292 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:23.290+0000 7f29b7183700 1 -- 172.21.15.167:0/1496643883 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:31 5.1b 5:d8000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x564a40855910 con 0x7f29b00383b0 2023-10-05T19:30:23.293 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:23.290+0000 7f29b6181700 1 -- 172.21.15.167:0/1496643883 <== osd.2 v2:172.21.15.167:6816/30037 5 ==== osd_op_reply(31 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f29a000b4d0 con 0x7f29b00383b0 2023-10-05T19:30:23.293 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:23.290+0000 7f29b6181700 1 -- 172.21.15.167:0/1496643883 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:32 5.7 5:e0000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x564a40855910 con 0x564a40852ff0 2023-10-05T19:30:23.294 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:23.290+0000 7f29b6982700 1 -- 172.21.15.167:0/1496643883 <== osd.3 v2:172.21.15.167:6808/30036 9 ==== osd_op_reply(32 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f29b0024400 con 0x564a40852ff0 2023-10-05T19:30:23.294 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:23.290+0000 7f29b6982700 1 -- 172.21.15.167:0/1496643883 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:33 5.17 5:e8000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x564a40855910 con 0x7f29a0012040 2023-10-05T19:30:23.295 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:23.290+0000 7f29b7183700 1 -- 172.21.15.167:0/1496643883 <== osd.0 v2:172.21.15.118:6800/27690 9 ==== osd_op_reply(33 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f29a8008c80 con 0x7f29a0012040 2023-10-05T19:30:23.295 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:23.290+0000 7f29b7183700 1 -- 172.21.15.167:0/1496643883 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:34 5.f 5:f0000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x564a40855910 con 0x564a4083e1e0 2023-10-05T19:30:23.295 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:23.294+0000 7f29b6181700 1 -- 172.21.15.167:0/1496643883 <== osd.1 v2:172.21.15.167:6800/30035 11 ==== osd_op_reply(34 [pgnls start_epoch 32 out=115b] v32'3 uv3 ondisk = 1) v8 ==== 144+0+115 (crc 0 0 0) 0x7f29a000b4d0 con 0x564a4083e1e0 2023-10-05T19:30:23.296 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:23.294+0000 7f29b6181700 1 -- 172.21.15.167:0/1496643883 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:35 5.1f 5:f8000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x564a40855910 con 0x7f29b00383b0 2023-10-05T19:30:23.296 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:23.294+0000 7f29b6181700 1 -- 172.21.15.167:0/1496643883 <== osd.2 v2:172.21.15.167:6816/30037 6 ==== osd_op_reply(35 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f29a000b4d0 con 0x7f29b00383b0 2023-10-05T19:30:23.297 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:23.294+0000 7f29b8e4de40 20 RGWRados::pool_iterate: got zonegroup_info.4f1f719d-9648-417b-b10c-e5d939825419 2023-10-05T19:30:23.297 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:23.294+0000 7f29b8e4de40 20 RGWRados::pool_iterate: got zone_info.9a9df965-6612-45d4-a45f-6e3c889bacc2 2023-10-05T19:30:23.297 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:23.294+0000 7f29b8e4de40 20 RGWRados::pool_iterate: got zone_names.default 2023-10-05T19:30:23.297 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:23.294+0000 7f29b8e4de40 20 RGWRados::pool_iterate: got zonegroups_names.default 2023-10-05T19:30:23.297 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:23.294+0000 7f29b8e4de40 20 rados->read ofs=0 len=0 2023-10-05T19:30:23.297 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:23.294+0000 7f29b8e4de40 1 -- 172.21.15.167:0/1496643883 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:36 5.12 5:49953fa1:::default.realm:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32) v8 -- 0x564a40851ad0 con 0x564a4083e1e0 2023-10-05T19:30:23.298 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:23.294+0000 7f29b6181700 1 -- 172.21.15.167:0/1496643883 <== osd.1 v2:172.21.15.167:6800/30035 12 ==== osd_op_reply(36 default.realm [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 ==== 157+0+0 (crc 0 0 0) 0x7f29a002f040 con 0x564a4083e1e0 2023-10-05T19:30:23.298 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:23.294+0000 7f29b8e4de40 20 rados_obj.operate() r=-2 bl.length=0 2023-10-05T19:30:23.298 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:23.294+0000 7f29b8e4de40 20 rados->read ofs=0 len=0 2023-10-05T19:30:23.298 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:23.294+0000 7f29b8e4de40 1 -- 172.21.15.167:0/1496643883 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:37 5.f 5:f4c53578:::zonegroups_names.default:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32) v8 -- 0x564a40851e80 con 0x564a4083e1e0 2023-10-05T19:30:23.298 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:23.294+0000 7f29b6181700 1 -- 172.21.15.167:0/1496643883 <== osd.1 v2:172.21.15.167:6800/30035 13 ==== osd_op_reply(37 zonegroups_names.default [read 0~46 out=46b] v0'0 uv2 ondisk = 0) v8 ==== 168+0+46 (crc 0 0 0) 0x7f29a002f040 con 0x564a4083e1e0 2023-10-05T19:30:23.298 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:23.294+0000 7f29b8e4de40 20 rados_obj.operate() r=0 bl.length=46 2023-10-05T19:30:23.298 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:23.294+0000 7f29b8e4de40 20 rados->read ofs=0 len=0 2023-10-05T19:30:23.299 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:23.294+0000 7f29b8e4de40 1 -- 172.21.15.167:0/1496643883 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:38 5.c 5:30d299a5:::zonegroup_info.4f1f719d-9648-417b-b10c-e5d939825419:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32) v8 -- 0x564a40852230 con 0x7f29a0012040 2023-10-05T19:30:23.299 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:23.294+0000 7f29b7183700 1 -- 172.21.15.167:0/1496643883 <== osd.0 v2:172.21.15.118:6800/27690 10 ==== osd_op_reply(38 zonegroup_info.4f1f719d-9648-417b-b10c-e5d939825419 [read 0~362 out=362b] v0'0 uv1 ondisk = 0) v8 ==== 195+0+362 (crc 0 0 0) 0x7f29a8008c80 con 0x7f29a0012040 2023-10-05T19:30:23.299 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:23.294+0000 7f29b8e4de40 20 rados_obj.operate() r=0 bl.length=362 2023-10-05T19:30:23.301 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:23.298+0000 7f29b8e4de40 1 -- 172.21.15.167:0/1496643883 >> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] conn(0x7f29a0012040 msgr2=0x7f29a0013420 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-05T19:30:23.301 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:23.298+0000 7f29b8e4de40 1 --2- 172.21.15.167:0/1496643883 >> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] conn(0x7f29a0012040 0x7f29a0013420 crc :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-05T19:30:23.301 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:23.298+0000 7f29b8e4de40 1 -- 172.21.15.167:0/1496643883 >> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] conn(0x564a4083e1e0 msgr2=0x564a4083e660 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-05T19:30:23.301 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:23.298+0000 7f29b8e4de40 1 --2- 172.21.15.167:0/1496643883 >> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] conn(0x564a4083e1e0 0x564a4083e660 crc :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-05T19:30:23.301 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:23.298+0000 7f29b8e4de40 1 -- 172.21.15.167:0/1496643883 >> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] conn(0x7f29b00383b0 msgr2=0x7f29b002f8c0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-05T19:30:23.301 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:23.298+0000 7f29b8e4de40 1 --2- 172.21.15.167:0/1496643883 >> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] conn(0x7f29b00383b0 0x7f29b002f8c0 crc :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-05T19:30:23.302 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:23.298+0000 7f29b8e4de40 1 -- 172.21.15.167:0/1496643883 >> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] conn(0x564a40852ff0 msgr2=0x564a408553d0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-05T19:30:23.302 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:23.298+0000 7f29b8e4de40 1 --2- 172.21.15.167:0/1496643883 >> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] conn(0x564a40852ff0 0x564a408553d0 crc :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-05T19:30:23.302 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:23.298+0000 7f29b8e4de40 1 -- 172.21.15.167:0/1496643883 >> [v2:172.21.15.167:6824/29889,v1:172.21.15.167:6825/29889] conn(0x7f2998030c50 msgr2=0x7f2998033100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-05T19:30:23.302 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:23.298+0000 7f29b8e4de40 1 --2- 172.21.15.167:0/1496643883 >> [v2:172.21.15.167:6824/29889,v1:172.21.15.167:6825/29889] conn(0x7f2998030c50 0x7f2998033100 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7f29b003a130 tx=0x7f29b003a000 comp rx=0 tx=0).stop 2023-10-05T19:30:23.302 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:23.298+0000 7f29b8e4de40 1 -- 172.21.15.167:0/1496643883 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x564a407e1ff0 msgr2=0x564a40807ac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-05T19:30:23.302 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:23.298+0000 7f29b7183700 1 -- 172.21.15.167:0/1496643883 reap_dead start 2023-10-05T19:30:23.302 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:23.298+0000 7f29b8e4de40 1 --2- 172.21.15.167:0/1496643883 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x564a407e1ff0 0x564a40807ac0 secure :-1 s=READY pgs=701 cs=0 l=1 rev1=1 crypto rx=0x7f29b000adc0 tx=0x7f29b001d000 comp rx=0 tx=0).stop 2023-10-05T19:30:23.302 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:23.298+0000 7f29b8e4de40 1 -- 172.21.15.167:0/1496643883 shutdown_connections 2023-10-05T19:30:23.302 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:23.298+0000 7f29b8e4de40 1 --2- 172.21.15.167:0/1496643883 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x564a407e1ff0 0x564a40807ac0 unknown :-1 s=CLOSED pgs=701 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-05T19:30:23.302 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:23.298+0000 7f29b8e4de40 1 -- 172.21.15.167:0/1496643883 >> 172.21.15.167:0/1496643883 conn(0x564a40755840 msgr2=0x564a407fb4a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-05T19:30:23.302 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:23.298+0000 7f29b8e4de40 1 -- 172.21.15.167:0/1496643883 shutdown_connections 2023-10-05T19:30:23.303 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:23.298+0000 7f29b8e4de40 1 -- 172.21.15.167:0/1496643883 wait complete. 2023-10-05T19:30:23.303 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:23.298+0000 7f29b8e4de40 1 -- 172.21.15.167:0/2550600892 >> [v2:172.21.15.167:6824/29889,v1:172.21.15.167:6825/29889] conn(0x7f2904030c00 msgr2=0x7f29040330b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-05T19:30:23.303 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:23.298+0000 7f29b8e4de40 1 --2- 172.21.15.167:0/2550600892 >> [v2:172.21.15.167:6824/29889,v1:172.21.15.167:6825/29889] conn(0x7f2904030c00 0x7f29040330b0 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7f29a800bb80 tx=0x7f29a8042410 comp rx=0 tx=0).stop 2023-10-05T19:30:23.304 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:23.298+0000 7f29b8e4de40 1 -- 172.21.15.167:0/2550600892 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x564a40819200 msgr2=0x564a408203e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-05T19:30:23.304 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:23.298+0000 7f29b8e4de40 1 --2- 172.21.15.167:0/2550600892 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x564a40819200 0x564a408203e0 secure :-1 s=READY pgs=703 cs=0 l=1 rev1=1 crypto rx=0x7f29a800aa70 tx=0x7f29a8013000 comp rx=0 tx=0).stop 2023-10-05T19:30:23.304 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:23.298+0000 7f29b8e4de40 1 -- 172.21.15.167:0/2550600892 shutdown_connections 2023-10-05T19:30:23.304 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:23.298+0000 7f29b8e4de40 1 --2- 172.21.15.167:0/2550600892 >> [v2:172.21.15.167:6824/29889,v1:172.21.15.167:6825/29889] conn(0x7f2904030c00 0x7f29040330b0 unknown :-1 s=CLOSED pgs=241 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-05T19:30:23.304 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:23.298+0000 7f29b8e4de40 1 --2- 172.21.15.167:0/2550600892 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x564a40819200 0x564a408203e0 unknown :-1 s=CLOSED pgs=703 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-05T19:30:23.304 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:23.298+0000 7f29b8e4de40 1 -- 172.21.15.167:0/2550600892 >> 172.21.15.167:0/2550600892 conn(0x564a407e2900 msgr2=0x564a40815b90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-05T19:30:23.304 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:23.298+0000 7f29b8e4de40 1 -- 172.21.15.167:0/2550600892 shutdown_connections 2023-10-05T19:30:23.304 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:23.298+0000 7f29b8e4de40 1 -- 172.21.15.167:0/2550600892 wait complete. 2023-10-05T19:30:23.305 INFO:teuthology.orchestra.run.smithi167.stdout:{ 2023-10-05T19:30:23.305 INFO:teuthology.orchestra.run.smithi167.stdout: "id": "4f1f719d-9648-417b-b10c-e5d939825419", 2023-10-05T19:30:23.305 INFO:teuthology.orchestra.run.smithi167.stdout: "name": "default", 2023-10-05T19:30:23.305 INFO:teuthology.orchestra.run.smithi167.stdout: "api_name": "default", 2023-10-05T19:30:23.305 INFO:teuthology.orchestra.run.smithi167.stdout: "is_master": "true", 2023-10-05T19:30:23.305 INFO:teuthology.orchestra.run.smithi167.stdout: "endpoints": [], 2023-10-05T19:30:23.305 INFO:teuthology.orchestra.run.smithi167.stdout: "hostnames": [], 2023-10-05T19:30:23.305 INFO:teuthology.orchestra.run.smithi167.stdout: "hostnames_s3website": [], 2023-10-05T19:30:23.305 INFO:teuthology.orchestra.run.smithi167.stdout: "master_zone": "9a9df965-6612-45d4-a45f-6e3c889bacc2", 2023-10-05T19:30:23.305 INFO:teuthology.orchestra.run.smithi167.stdout: "zones": [ 2023-10-05T19:30:23.305 INFO:teuthology.orchestra.run.smithi167.stdout: { 2023-10-05T19:30:23.306 INFO:teuthology.orchestra.run.smithi167.stdout: "id": "9a9df965-6612-45d4-a45f-6e3c889bacc2", 2023-10-05T19:30:23.306 INFO:teuthology.orchestra.run.smithi167.stdout: "name": "default", 2023-10-05T19:30:23.306 INFO:teuthology.orchestra.run.smithi167.stdout: "endpoints": [], 2023-10-05T19:30:23.306 INFO:teuthology.orchestra.run.smithi167.stdout: "log_meta": "false", 2023-10-05T19:30:23.306 INFO:teuthology.orchestra.run.smithi167.stdout: "log_data": "false", 2023-10-05T19:30:23.306 INFO:teuthology.orchestra.run.smithi167.stdout: "bucket_index_max_shards": 11, 2023-10-05T19:30:23.306 INFO:teuthology.orchestra.run.smithi167.stdout: "read_only": "false", 2023-10-05T19:30:23.306 INFO:teuthology.orchestra.run.smithi167.stdout: "tier_type": "", 2023-10-05T19:30:23.306 INFO:teuthology.orchestra.run.smithi167.stdout: "sync_from_all": "true", 2023-10-05T19:30:23.306 INFO:teuthology.orchestra.run.smithi167.stdout: "sync_from": [], 2023-10-05T19:30:23.306 INFO:teuthology.orchestra.run.smithi167.stdout: "redirect_zone": "" 2023-10-05T19:30:23.306 INFO:teuthology.orchestra.run.smithi167.stdout: } 2023-10-05T19:30:23.306 INFO:teuthology.orchestra.run.smithi167.stdout: ], 2023-10-05T19:30:23.307 INFO:teuthology.orchestra.run.smithi167.stdout: "placement_targets": [ 2023-10-05T19:30:23.307 INFO:teuthology.orchestra.run.smithi167.stdout: { 2023-10-05T19:30:23.307 INFO:teuthology.orchestra.run.smithi167.stdout: "name": "default-placement", 2023-10-05T19:30:23.309 INFO:teuthology.orchestra.run.smithi167.stdout: "tags": [], 2023-10-05T19:30:23.309 INFO:teuthology.orchestra.run.smithi167.stdout: "storage_classes": [ 2023-10-05T19:30:23.309 INFO:teuthology.orchestra.run.smithi167.stdout: "STANDARD" 2023-10-05T19:30:23.309 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2023-10-05T19:30:23.309 INFO:teuthology.orchestra.run.smithi167.stdout: } 2023-10-05T19:30:23.309 INFO:teuthology.orchestra.run.smithi167.stdout: ], 2023-10-05T19:30:23.309 INFO:teuthology.orchestra.run.smithi167.stdout: "default_placement": "default-placement", 2023-10-05T19:30:23.309 INFO:teuthology.orchestra.run.smithi167.stdout: "realm_id": "", 2023-10-05T19:30:23.310 INFO:teuthology.orchestra.run.smithi167.stdout: "sync_policy": { 2023-10-05T19:30:23.310 INFO:teuthology.orchestra.run.smithi167.stdout: "groups": [] 2023-10-05T19:30:23.310 INFO:teuthology.orchestra.run.smithi167.stdout: } 2023-10-05T19:30:23.310 INFO:teuthology.orchestra.run.smithi167.stdout:} 2023-10-05T19:30:23.311 DEBUG:tasks.util.rgw: json result: {'id': '4f1f719d-9648-417b-b10c-e5d939825419', 'name': 'default', 'api_name': 'default', 'is_master': 'true', 'endpoints': [], 'hostnames': [], 'hostnames_s3website': [], 'master_zone': '9a9df965-6612-45d4-a45f-6e3c889bacc2', 'zones': [{'id': '9a9df965-6612-45d4-a45f-6e3c889bacc2', 'name': 'default', 'endpoints': [], 'log_meta': 'false', 'log_data': 'false', 'bucket_index_max_shards': 11, 'read_only': 'false', 'tier_type': '', 'sync_from_all': 'true', 'sync_from': [], 'redirect_zone': ''}], 'placement_targets': [{'name': 'default-placement', 'tags': [], 'storage_classes': ['STANDARD']}], 'default_placement': 'default-placement', 'realm_id': '', 'sync_policy': {'groups': []}} 2023-10-05T19:30:23.313 INFO:teuthology.run_tasks:Running task radosgw-admin-rest... 2023-10-05T19:30:23.329 INFO:tasks.util.rgw:rgwadmin: client.0 : ['user', 'create', '--uid', 'ada', '--display-name', 'Ms. Admin User', '--access-key', 'MH1WC2XQ1S8UISFDZC8W', '--secret', 'dQyrTPA0s248YeN5bBv4ukvKU0kh54LWWywkrpoG', '--max-buckets', '0', '--caps', 'users=read, write; usage=read, write; buckets=read, write; zone=read, write; info=read;ratelimit=read, write'] 2023-10-05T19:30:23.329 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'ceph', 'user', 'create', '--uid', 'ada', '--display-name', 'Ms. Admin User', '--access-key', 'MH1WC2XQ1S8UISFDZC8W', '--secret', 'dQyrTPA0s248YeN5bBv4ukvKU0kh54LWWywkrpoG', '--max-buckets', '0', '--caps', 'users=read, write; usage=read, write; buckets=read, write; zone=read, write; info=read;ratelimit=read, write'] 2023-10-05T19:30:23.329 DEBUG:teuthology.orchestra.run.smithi167:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster ceph user create --uid ada --display-name 'Ms. Admin User' --access-key MH1WC2XQ1S8UISFDZC8W --secret dQyrTPA0s248YeN5bBv4ukvKU0kh54LWWywkrpoG --max-buckets 0 --caps 'users=read, write; usage=read, write; buckets=read, write; zone=read, write; info=read;ratelimit=read, write' 2023-10-05T19:30:23.381 INFO:teuthology.orchestra.run.smithi167.stderr:ignoring --setuser ceph since I am not root 2023-10-05T19:30:23.382 INFO:teuthology.orchestra.run.smithi167.stderr:ignoring --setgroup ceph since I am not root 2023-10-05T19:30:23.382 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:23.378+0000 7ffa94847e40 1 Processor -- start 2023-10-05T19:30:23.383 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:23.378+0000 7ffa94847e40 1 -- start start 2023-10-05T19:30:23.383 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:23.378+0000 7ffa94847e40 1 --2- >> v2:172.21.15.118:3300/0 conn(0x5638b1cdafc0 0x5638b1cdb390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-05T19:30:23.383 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:23.378+0000 7ffa94847e40 1 -- --> v1:172.21.15.118:6789/0 -- auth(proto 0 26 bytes epoch 0) v1 -- 0x5638b1c25bd0 con 0x5638b1cdb8d0 2023-10-05T19:30:23.383 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:23.378+0000 7ffa94847e40 1 -- --> v2:172.21.15.118:3300/0 -- mon_getmap magic: 0 v1 -- 0x5638b1be2ee0 con 0x5638b1cdafc0 2023-10-05T19:30:23.384 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:23.378+0000 7ffa9237c700 1 --2- >> v2:172.21.15.118:3300/0 conn(0x5638b1cdafc0 0x5638b1cdb390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-05T19:30:23.384 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:23.378+0000 7ffa91b7b700 1 --1- >> v1:172.21.15.118:6789/0 conn(0x5638b1cdb8d0 0x5638b1cdc370 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.118:6789/0 says I am v1:172.21.15.167:50852/0 (socket says 172.21.15.167:50852) 2023-10-05T19:30:23.384 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:23.378+0000 7ffa91b7b700 1 -- 172.21.15.167:0/3120786719 learned_addr learned my addr 172.21.15.167:0/3120786719 (peer_addr_for_me v1:172.21.15.167:0/0) 2023-10-05T19:30:23.384 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:23.382+0000 7ffa9237c700 1 -- 172.21.15.167:0/3120786719 >> v1:172.21.15.118:6789/0 conn(0x5638b1cdb8d0 legacy=0x5638b1cdc370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-05T19:30:23.384 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:23.382+0000 7ffa9237c700 1 -- 172.21.15.167:0/3120786719 --> v2:172.21.15.118:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5638b1cdca80 con 0x5638b1cdafc0 2023-10-05T19:30:23.385 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:23.382+0000 7ffa9237c700 1 --2- 172.21.15.167:0/3120786719 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5638b1cdafc0 0x5638b1cdb390 secure :-1 s=READY pgs=705 cs=0 l=1 rev1=1 crypto rx=0x7ffa8400a5d0 tx=0x7ffa84005a50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6c7ce05a42e89f8a server_cookie=0 in_seq=0 out_seq=0 2023-10-05T19:30:23.385 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:23.382+0000 7ffa9137a700 1 -- 172.21.15.167:0/3120786719 <== mon.0 v2:172.21.15.118:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7ffa840086a0 con 0x5638b1cdafc0 2023-10-05T19:30:23.385 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:23.382+0000 7ffa9137a700 1 -- 172.21.15.167:0/3120786719 <== mon.0 v2:172.21.15.118:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ffa84008800 con 0x5638b1cdafc0 2023-10-05T19:30:23.386 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:23.382+0000 7ffa9137a700 1 -- 172.21.15.167:0/3120786719 <== mon.0 v2:172.21.15.118:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7ffa84008ad0 con 0x5638b1cdafc0 2023-10-05T19:30:23.386 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:23.382+0000 7ffa94847e40 1 -- 172.21.15.167:0/3120786719 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5638b1cdafc0 msgr2=0x5638b1cdb390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-05T19:30:23.386 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:23.382+0000 7ffa94847e40 1 --2- 172.21.15.167:0/3120786719 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5638b1cdafc0 0x5638b1cdb390 secure :-1 s=READY pgs=705 cs=0 l=1 rev1=1 crypto rx=0x7ffa8400a5d0 tx=0x7ffa84005a50 comp rx=0 tx=0).stop 2023-10-05T19:30:23.386 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:23.382+0000 7ffa94847e40 1 -- 172.21.15.167:0/3120786719 shutdown_connections 2023-10-05T19:30:23.386 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:23.382+0000 7ffa94847e40 1 --2- 172.21.15.167:0/3120786719 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5638b1cdafc0 0x5638b1cdb390 unknown :-1 s=CLOSED pgs=705 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-05T19:30:23.386 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:23.382+0000 7ffa94847e40 1 -- 172.21.15.167:0/3120786719 >> 172.21.15.167:0/3120786719 conn(0x5638b1c4e810 msgr2=0x5638b1cd94b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-05T19:30:23.386 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:23.382+0000 7ffa94847e40 1 -- 172.21.15.167:0/3120786719 shutdown_connections 2023-10-05T19:30:23.386 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:23.382+0000 7ffa94847e40 1 -- 172.21.15.167:0/3120786719 wait complete. 2023-10-05T19:30:23.387 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:23.382+0000 7ffa94847e40 1 Processor -- start 2023-10-05T19:30:23.387 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:23.382+0000 7ffa94847e40 1 -- start start 2023-10-05T19:30:23.387 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:23.382+0000 7ffa94847e40 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5638b1cdafc0 0x5638b1cfa160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-05T19:30:23.388 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:23.382+0000 7ffa94847e40 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x5638b1ba6e30 con 0x5638b1cdafc0 2023-10-05T19:30:23.388 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:23.382+0000 7ffa9237c700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5638b1cdafc0 0x5638b1cfa160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-05T19:30:23.388 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:23.382+0000 7ffa9237c700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5638b1cdafc0 0x5638b1cfa160 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.118:3300/0 says I am v2:172.21.15.167:33316/0 (socket says 172.21.15.167:33316) 2023-10-05T19:30:23.388 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:23.382+0000 7ffa9237c700 1 -- 172.21.15.167:0/2721147509 learned_addr learned my addr 172.21.15.167:0/2721147509 (peer_addr_for_me v2:172.21.15.167:0/0) 2023-10-05T19:30:23.388 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:23.382+0000 7ffa9237c700 1 -- 172.21.15.167:0/2721147509 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffa84005cf0 con 0x5638b1cdafc0 2023-10-05T19:30:23.388 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:23.386+0000 7ffa9237c700 1 --2- 172.21.15.167:0/2721147509 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5638b1cdafc0 0x5638b1cfa160 secure :-1 s=READY pgs=706 cs=0 l=1 rev1=1 crypto rx=0x7ffa8400a520 tx=0x7ffa84015000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=508a2d9e7d4aa487 server_cookie=0 in_seq=0 out_seq=0 2023-10-05T19:30:23.389 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:23.386+0000 7ffa8bfff700 1 -- 172.21.15.167:0/2721147509 <== mon.0 v2:172.21.15.118:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7ffa84008970 con 0x5638b1cdafc0 2023-10-05T19:30:23.389 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:23.386+0000 7ffa8bfff700 1 -- 172.21.15.167:0/2721147509 <== mon.0 v2:172.21.15.118:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ffa84008ad0 con 0x5638b1cdafc0 2023-10-05T19:30:23.389 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:23.386+0000 7ffa8bfff700 1 -- 172.21.15.167:0/2721147509 <== mon.0 v2:172.21.15.118:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7ffa84008dc0 con 0x5638b1cdafc0 2023-10-05T19:30:23.390 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:23.386+0000 7ffa94847e40 1 -- 172.21.15.167:0/2721147509 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5638b1cdafc0 msgr2=0x5638b1cfa160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-05T19:30:23.390 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:23.386+0000 7ffa94847e40 1 --2- 172.21.15.167:0/2721147509 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5638b1cdafc0 0x5638b1cfa160 secure :-1 s=READY pgs=706 cs=0 l=1 rev1=1 crypto rx=0x7ffa8400a520 tx=0x7ffa84015000 comp rx=0 tx=0).stop 2023-10-05T19:30:23.390 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:23.386+0000 7ffa94847e40 1 -- 172.21.15.167:0/2721147509 shutdown_connections 2023-10-05T19:30:23.390 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:23.386+0000 7ffa94847e40 1 --2- 172.21.15.167:0/2721147509 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5638b1cdafc0 0x5638b1cfa160 unknown :-1 s=CLOSED pgs=706 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-05T19:30:23.390 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:23.386+0000 7ffa94847e40 1 -- 172.21.15.167:0/2721147509 >> 172.21.15.167:0/2721147509 conn(0x5638b1c4e810 msgr2=0x5638b1cf5ca0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-05T19:30:23.390 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:23.386+0000 7ffa94847e40 1 -- 172.21.15.167:0/2721147509 shutdown_connections 2023-10-05T19:30:23.390 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:23.386+0000 7ffa94847e40 1 -- 172.21.15.167:0/2721147509 wait complete. 2023-10-05T19:30:23.391 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:23.386+0000 7ffa94847e40 1 Processor -- start 2023-10-05T19:30:23.391 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:23.386+0000 7ffa94847e40 1 -- start start 2023-10-05T19:30:23.391 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:23.386+0000 7ffa94847e40 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5638b1cdafc0 0x5638b1d00d70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-05T19:30:23.391 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:23.386+0000 7ffa94847e40 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x5638b1b79ed0 con 0x5638b1cdafc0 2023-10-05T19:30:23.392 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:23.386+0000 7ffa9237c700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5638b1cdafc0 0x5638b1d00d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-05T19:30:23.392 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:23.386+0000 7ffa9237c700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5638b1cdafc0 0x5638b1d00d70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.118:3300/0 says I am v2:172.21.15.167:33324/0 (socket says 172.21.15.167:33324) 2023-10-05T19:30:23.392 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:23.386+0000 7ffa9237c700 1 -- 172.21.15.167:0/2027390457 learned_addr learned my addr 172.21.15.167:0/2027390457 (peer_addr_for_me v2:172.21.15.167:0/0) 2023-10-05T19:30:23.392 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:23.390+0000 7ffa9237c700 1 -- 172.21.15.167:0/2027390457 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5638b1cdca80 con 0x5638b1cdafc0 2023-10-05T19:30:23.392 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:23.390+0000 7ffa9237c700 1 --2- 172.21.15.167:0/2027390457 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5638b1cdafc0 0x5638b1d00d70 secure :-1 s=READY pgs=707 cs=0 l=1 rev1=1 crypto rx=0x7ffa84005540 tx=0x7ffa84003050 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-05T19:30:23.393 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:23.390+0000 7ffa8a7fc700 1 -- 172.21.15.167:0/2027390457 <== mon.0 v2:172.21.15.118:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7ffa84012bf0 con 0x5638b1cdafc0 2023-10-05T19:30:23.393 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:23.390+0000 7ffa8a7fc700 1 -- 172.21.15.167:0/2027390457 <== mon.0 v2:172.21.15.118:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ffa8402a070 con 0x5638b1cdafc0 2023-10-05T19:30:23.393 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:23.390+0000 7ffa94847e40 1 -- 172.21.15.167:0/2027390457 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x5638b1d012b0 con 0x5638b1cdafc0 2023-10-05T19:30:23.393 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:23.390+0000 7ffa8a7fc700 1 -- 172.21.15.167:0/2027390457 <== mon.0 v2:172.21.15.118:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7ffa84012e90 con 0x5638b1cdafc0 2023-10-05T19:30:23.393 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:23.390+0000 7ffa94847e40 1 -- 172.21.15.167:0/2027390457 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x5638b1d01750 con 0x5638b1cdafc0 2023-10-05T19:30:23.394 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:23.390+0000 7ffa8a7fc700 1 -- 172.21.15.167:0/2027390457 <== mon.0 v2:172.21.15.118:3300/0 4 ==== mgrmap(e 5) v1 ==== 40329+0+0 (secure 0 0 0) 0x7ffa84019070 con 0x5638b1cdafc0 2023-10-05T19:30:23.395 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:23.390+0000 7ffa8a7fc700 1 --2- 172.21.15.167:0/2027390457 >> [v2:172.21.15.167:6824/29889,v1:172.21.15.167:6825/29889] conn(0x7ffa74030c50 0x7ffa74033100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-05T19:30:23.395 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:23.390+0000 7ffa94847e40 20 rados->read ofs=0 len=0 2023-10-05T19:30:23.395 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:23.390+0000 7ffa8a7fc700 1 -- 172.21.15.167:0/2027390457 <== mon.0 v2:172.21.15.118:3300/0 5 ==== osd_map(32..32 src has 1..32) v4 ==== 6260+0+0 (secure 0 0 0) 0x7ffa8402c070 con 0x5638b1cdafc0 2023-10-05T19:30:23.395 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:23.390+0000 7ffa91b7b700 1 --2- 172.21.15.167:0/2027390457 >> [v2:172.21.15.167:6824/29889,v1:172.21.15.167:6825/29889] conn(0x7ffa74030c50 0x7ffa74033100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-05T19:30:23.396 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:23.390+0000 7ffa94847e40 1 --2- 172.21.15.167:0/2027390457 >> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] conn(0x5638b1cdb8d0 0x5638b1d0c890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-05T19:30:23.396 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:23.390+0000 7ffa94847e40 1 -- 172.21.15.167:0/2027390457 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:1 5.12 5:49953fa1:::default.realm:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32) v8 -- 0x5638b1d0cf00 con 0x5638b1cdb8d0 2023-10-05T19:30:23.396 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:23.394+0000 7ffa92b7d700 1 --2- 172.21.15.167:0/2027390457 >> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] conn(0x5638b1cdb8d0 0x5638b1d0c890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-05T19:30:23.396 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:23.394+0000 7ffa91b7b700 1 --2- 172.21.15.167:0/2027390457 >> [v2:172.21.15.167:6824/29889,v1:172.21.15.167:6825/29889] conn(0x7ffa74030c50 0x7ffa74033100 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7ffa7c001e90 tx=0x7ffa7c0013b0 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-05T19:30:23.396 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:23.394+0000 7ffa92b7d700 1 --2- 172.21.15.167:0/2027390457 >> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] conn(0x5638b1cdb8d0 0x5638b1d0c890 crc :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-05T19:30:23.398 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:23.394+0000 7ffa92b7d700 1 -- 172.21.15.167:0/2027390457 <== osd.1 v2:172.21.15.167:6800/30035 1 ==== osd_op_reply(1 default.realm [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 ==== 157+0+0 (crc 0 0 0) 0x7ffa8c00a5a0 con 0x5638b1cdb8d0 2023-10-05T19:30:23.398 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:23.394+0000 7ffa94847e40 20 rados_obj.operate() r=-2 bl.length=0 2023-10-05T19:30:23.398 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:23.394+0000 7ffa94847e40 20 realm 2023-10-05T19:30:23.398 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:23.394+0000 7ffa94847e40 20 rados->read ofs=0 len=0 2023-10-05T19:30:23.398 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:23.394+0000 7ffa94847e40 1 -- 172.21.15.167:0/2027390457 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:2 5.12 5:49953fa1:::default.realm:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32) v8 -- 0x5638b1d09ef0 con 0x5638b1cdb8d0 2023-10-05T19:30:23.398 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:23.394+0000 7ffa92b7d700 1 -- 172.21.15.167:0/2027390457 <== osd.1 v2:172.21.15.167:6800/30035 2 ==== osd_op_reply(2 default.realm [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 ==== 157+0+0 (crc 0 0 0) 0x7ffa8c00a5a0 con 0x5638b1cdb8d0 2023-10-05T19:30:23.398 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:23.394+0000 7ffa94847e40 20 rados_obj.operate() r=-2 bl.length=0 2023-10-05T19:30:23.398 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:23.394+0000 7ffa94847e40 4 RGWPeriod::init failed to init realm id : (2) No such file or directory 2023-10-05T19:30:23.399 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:23.394+0000 7ffa94847e40 20 rados->read ofs=0 len=0 2023-10-05T19:30:23.399 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:23.394+0000 7ffa94847e40 1 -- 172.21.15.167:0/2027390457 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:3 5.12 5:49953fa1:::default.realm:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32) v8 -- 0x5638b1d0d9e0 con 0x5638b1cdb8d0 2023-10-05T19:30:23.399 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:23.394+0000 7ffa92b7d700 1 -- 172.21.15.167:0/2027390457 <== osd.1 v2:172.21.15.167:6800/30035 3 ==== osd_op_reply(3 default.realm [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 ==== 157+0+0 (crc 0 0 0) 0x7ffa8c00a5a0 con 0x5638b1cdb8d0 2023-10-05T19:30:23.399 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:23.394+0000 7ffa94847e40 20 rados_obj.operate() r=-2 bl.length=0 2023-10-05T19:30:23.399 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:23.394+0000 7ffa94847e40 20 rados->read ofs=0 len=0 2023-10-05T19:30:23.399 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:23.394+0000 7ffa94847e40 1 -- 172.21.15.167:0/2027390457 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:4 5.f 5:f43b6ece:::zone_names.default:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32) v8 -- 0x5638b1d0d5f0 con 0x5638b1cdb8d0 2023-10-05T19:30:23.400 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:23.394+0000 7ffa92b7d700 1 -- 172.21.15.167:0/2027390457 <== osd.1 v2:172.21.15.167:6800/30035 4 ==== osd_op_reply(4 zone_names.default [read 0~46 out=46b] v0'0 uv3 ondisk = 0) v8 ==== 162+0+46 (crc 0 0 0) 0x7ffa8c00a5a0 con 0x5638b1cdb8d0 2023-10-05T19:30:23.400 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:23.394+0000 7ffa94847e40 20 rados_obj.operate() r=0 bl.length=46 2023-10-05T19:30:23.400 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:23.394+0000 7ffa94847e40 20 rados->read ofs=0 len=0 2023-10-05T19:30:23.401 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:23.394+0000 7ffa94847e40 1 -- 172.21.15.167:0/2027390457 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:5 5.1d 5:b8374eb4:::zone_info.9a9df965-6612-45d4-a45f-6e3c889bacc2:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32) v8 -- 0x5638b1d0e870 con 0x5638b1cdb8d0 2023-10-05T19:30:23.401 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:23.394+0000 7ffa92b7d700 1 -- 172.21.15.167:0/2027390457 <== osd.1 v2:172.21.15.167:6800/30035 5 ==== osd_op_reply(5 zone_info.9a9df965-6612-45d4-a45f-6e3c889bacc2 [read 0~874 out=874b] v0'0 uv4 ondisk = 0) v8 ==== 190+0+874 (crc 0 0 0) 0x7ffa8c00a5a0 con 0x5638b1cdb8d0 2023-10-05T19:30:23.401 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:23.394+0000 7ffa94847e40 20 rados_obj.operate() r=0 bl.length=874 2023-10-05T19:30:23.401 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:23.394+0000 7ffa94847e40 20 searching for the correct realm 2023-10-05T19:30:23.401 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:23.394+0000 7ffa94847e40 1 --2- 172.21.15.167:0/2027390457 >> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] conn(0x5638b1d10630 0x5638b1d12a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-05T19:30:23.401 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:23.394+0000 7ffa94847e40 1 -- 172.21.15.167:0/2027390457 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:6 5.0 5:00000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x5638b1d12f50 con 0x5638b1d10630 2023-10-05T19:30:23.401 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:23.394+0000 7ffa9237c700 1 --2- 172.21.15.167:0/2027390457 >> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] conn(0x5638b1d10630 0x5638b1d12a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-05T19:30:23.401 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:23.398+0000 7ffa9237c700 1 --2- 172.21.15.167:0/2027390457 >> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] conn(0x5638b1d10630 0x5638b1d12a10 crc :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-05T19:30:23.402 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:23.398+0000 7ffa9237c700 1 -- 172.21.15.167:0/2027390457 <== osd.3 v2:172.21.15.167:6808/30036 1 ==== osd_op_reply(6 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7ffa84022020 con 0x5638b1d10630 2023-10-05T19:30:23.402 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:23.398+0000 7ffa9237c700 1 --2- 172.21.15.167:0/2027390457 >> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] conn(0x7ffa84007030 0x7ffa84000ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-05T19:30:23.402 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:23.398+0000 7ffa9237c700 1 -- 172.21.15.167:0/2027390457 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:7 5.10 5:08000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x5638b1d12f50 con 0x7ffa84007030 2023-10-05T19:30:23.403 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:23.398+0000 7ffa91b7b700 1 --2- 172.21.15.167:0/2027390457 >> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] conn(0x7ffa84007030 0x7ffa84000ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-05T19:30:23.403 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:23.398+0000 7ffa91b7b700 1 --2- 172.21.15.167:0/2027390457 >> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] conn(0x7ffa84007030 0x7ffa84000ed0 crc :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-05T19:30:23.403 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:23.398+0000 7ffa91b7b700 1 -- 172.21.15.167:0/2027390457 <== osd.2 v2:172.21.15.167:6816/30037 1 ==== osd_op_reply(7 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7ffa7c006070 con 0x7ffa84007030 2023-10-05T19:30:23.403 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:23.398+0000 7ffa91b7b700 1 --2- 172.21.15.167:0/2027390457 >> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] conn(0x7ffa7c00d8e0 0x7ffa7c00c360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-05T19:30:23.403 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:23.398+0000 7ffa91b7b700 1 -- 172.21.15.167:0/2027390457 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:8 5.8 5:10000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x5638b1d12f50 con 0x7ffa7c00d8e0 2023-10-05T19:30:23.404 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:23.398+0000 7ffa92b7d700 1 --2- 172.21.15.167:0/2027390457 >> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] conn(0x7ffa7c00d8e0 0x7ffa7c00c360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-05T19:30:23.404 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:23.402+0000 7ffa92b7d700 1 --2- 172.21.15.167:0/2027390457 >> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] conn(0x7ffa7c00d8e0 0x7ffa7c00c360 crc :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-05T19:30:23.404 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:23.402+0000 7ffa92b7d700 1 -- 172.21.15.167:0/2027390457 <== osd.0 v2:172.21.15.118:6800/27690 1 ==== osd_op_reply(8 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7ffa8c00a5a0 con 0x7ffa7c00d8e0 2023-10-05T19:30:23.404 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:23.402+0000 7ffa92b7d700 1 -- 172.21.15.167:0/2027390457 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:9 5.18 5:18000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x5638b1d12f50 con 0x5638b1cdb8d0 2023-10-05T19:30:23.405 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:23.402+0000 7ffa92b7d700 1 -- 172.21.15.167:0/2027390457 <== osd.1 v2:172.21.15.167:6800/30035 6 ==== osd_op_reply(9 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7ffa8c00a5a0 con 0x5638b1cdb8d0 2023-10-05T19:30:23.405 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:23.402+0000 7ffa92b7d700 1 -- 172.21.15.167:0/2027390457 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:10 5.4 5:20000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x5638b1d12f50 con 0x5638b1cdb8d0 2023-10-05T19:30:23.406 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:23.402+0000 7ffa92b7d700 1 -- 172.21.15.167:0/2027390457 <== osd.1 v2:172.21.15.167:6800/30035 7 ==== osd_op_reply(10 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7ffa8c00a5a0 con 0x5638b1cdb8d0 2023-10-05T19:30:23.406 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:23.402+0000 7ffa92b7d700 1 -- 172.21.15.167:0/2027390457 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:11 5.14 5:28000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x5638b1d12f50 con 0x7ffa7c00d8e0 2023-10-05T19:30:23.406 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:23.402+0000 7ffa92b7d700 1 -- 172.21.15.167:0/2027390457 <== osd.0 v2:172.21.15.118:6800/27690 2 ==== osd_op_reply(11 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7ffa8c00a5a0 con 0x7ffa7c00d8e0 2023-10-05T19:30:23.406 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:23.402+0000 7ffa92b7d700 1 -- 172.21.15.167:0/2027390457 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:12 5.c 5:30000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x5638b1d12f50 con 0x7ffa7c00d8e0 2023-10-05T19:30:23.407 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:23.402+0000 7ffa92b7d700 1 -- 172.21.15.167:0/2027390457 <== osd.0 v2:172.21.15.118:6800/27690 3 ==== osd_op_reply(12 [pgnls start_epoch 32 out=112b] v23'1 uv1 ondisk = 1) v8 ==== 144+0+112 (crc 0 0 0) 0x7ffa8c00a5a0 con 0x7ffa7c00d8e0 2023-10-05T19:30:23.407 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:23.402+0000 7ffa92b7d700 1 -- 172.21.15.167:0/2027390457 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:13 5.1c 5:38000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x5638b1d12f50 con 0x7ffa84007030 2023-10-05T19:30:23.408 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:23.406+0000 7ffa91b7b700 1 -- 172.21.15.167:0/2027390457 <== osd.2 v2:172.21.15.167:6816/30037 2 ==== osd_op_reply(13 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7ffa7c006070 con 0x7ffa84007030 2023-10-05T19:30:23.408 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:23.406+0000 7ffa91b7b700 1 -- 172.21.15.167:0/2027390457 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:14 5.2 5:40000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x5638b1d12f50 con 0x5638b1d10630 2023-10-05T19:30:23.409 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:23.406+0000 7ffa9237c700 1 -- 172.21.15.167:0/2027390457 <== osd.3 v2:172.21.15.167:6808/30036 2 ==== osd_op_reply(14 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7ffa84022020 con 0x5638b1d10630 2023-10-05T19:30:23.409 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:23.406+0000 7ffa9237c700 1 -- 172.21.15.167:0/2027390457 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:15 5.12 5:48000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x5638b1d12f50 con 0x5638b1cdb8d0 2023-10-05T19:30:23.409 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:23.406+0000 7ffa92b7d700 1 -- 172.21.15.167:0/2027390457 <== osd.1 v2:172.21.15.167:6800/30035 8 ==== osd_op_reply(15 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7ffa8c00a5a0 con 0x5638b1cdb8d0 2023-10-05T19:30:23.410 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:23.406+0000 7ffa92b7d700 1 -- 172.21.15.167:0/2027390457 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:16 5.a 5:50000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x5638b1d12f50 con 0x7ffa7c00d8e0 2023-10-05T19:30:23.411 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:23.406+0000 7ffa92b7d700 1 -- 172.21.15.167:0/2027390457 <== osd.0 v2:172.21.15.118:6800/27690 4 ==== osd_op_reply(16 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7ffa8c00a5a0 con 0x7ffa7c00d8e0 2023-10-05T19:30:23.411 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:23.406+0000 7ffa92b7d700 1 -- 172.21.15.167:0/2027390457 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:17 5.1a 5:58000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x5638b1d12f50 con 0x5638b1cdb8d0 2023-10-05T19:30:23.411 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:23.406+0000 7ffa92b7d700 1 -- 172.21.15.167:0/2027390457 <== osd.1 v2:172.21.15.167:6800/30035 9 ==== osd_op_reply(17 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7ffa8c00a5a0 con 0x5638b1cdb8d0 2023-10-05T19:30:23.411 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:23.406+0000 7ffa92b7d700 1 -- 172.21.15.167:0/2027390457 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:18 5.6 5:60000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x5638b1d12f50 con 0x7ffa84007030 2023-10-05T19:30:23.412 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:23.410+0000 7ffa91b7b700 1 -- 172.21.15.167:0/2027390457 <== osd.2 v2:172.21.15.167:6816/30037 3 ==== osd_op_reply(18 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7ffa7c006070 con 0x7ffa84007030 2023-10-05T19:30:23.412 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:23.410+0000 7ffa91b7b700 1 -- 172.21.15.167:0/2027390457 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:19 5.16 5:68000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x5638b1d12f50 con 0x5638b1d10630 2023-10-05T19:30:23.413 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:23.410+0000 7ffa9237c700 1 -- 172.21.15.167:0/2027390457 <== osd.3 v2:172.21.15.167:6808/30036 3 ==== osd_op_reply(19 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7ffa84022020 con 0x5638b1d10630 2023-10-05T19:30:23.413 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:23.410+0000 7ffa9237c700 1 -- 172.21.15.167:0/2027390457 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:20 5.e 5:70000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x5638b1d12f50 con 0x5638b1d10630 2023-10-05T19:30:23.413 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:23.410+0000 7ffa9237c700 1 -- 172.21.15.167:0/2027390457 <== osd.3 v2:172.21.15.167:6808/30036 4 ==== osd_op_reply(20 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7ffa84022020 con 0x5638b1d10630 2023-10-05T19:30:23.413 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:23.410+0000 7ffa9237c700 1 -- 172.21.15.167:0/2027390457 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:21 5.1e 5:78000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x5638b1d12f50 con 0x7ffa7c00d8e0 2023-10-05T19:30:23.414 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:23.410+0000 7ffa92b7d700 1 -- 172.21.15.167:0/2027390457 <== osd.0 v2:172.21.15.118:6800/27690 5 ==== osd_op_reply(21 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7ffa8c00a5a0 con 0x7ffa7c00d8e0 2023-10-05T19:30:23.415 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:23.410+0000 7ffa92b7d700 1 -- 172.21.15.167:0/2027390457 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:22 5.1 5:80000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x5638b1d12f50 con 0x5638b1d10630 2023-10-05T19:30:23.415 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:23.410+0000 7ffa9237c700 1 -- 172.21.15.167:0/2027390457 <== osd.3 v2:172.21.15.167:6808/30036 5 ==== osd_op_reply(22 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7ffa84022020 con 0x5638b1d10630 2023-10-05T19:30:23.415 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:23.410+0000 7ffa9237c700 1 -- 172.21.15.167:0/2027390457 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:23 5.11 5:88000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x5638b1d12f50 con 0x5638b1d10630 2023-10-05T19:30:23.416 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:23.414+0000 7ffa9237c700 1 -- 172.21.15.167:0/2027390457 <== osd.3 v2:172.21.15.167:6808/30036 6 ==== osd_op_reply(23 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7ffa84022020 con 0x5638b1d10630 2023-10-05T19:30:23.416 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:23.414+0000 7ffa9237c700 1 -- 172.21.15.167:0/2027390457 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:24 5.9 5:90000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x5638b1d12f50 con 0x5638b1cdb8d0 2023-10-05T19:30:23.417 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:23.414+0000 7ffa92b7d700 1 -- 172.21.15.167:0/2027390457 <== osd.1 v2:172.21.15.167:6800/30035 10 ==== osd_op_reply(24 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7ffa8c00a5a0 con 0x5638b1cdb8d0 2023-10-05T19:30:23.417 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:23.414+0000 7ffa92b7d700 1 -- 172.21.15.167:0/2027390457 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:25 5.19 5:98000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x5638b1d12f50 con 0x5638b1cdb8d0 2023-10-05T19:30:23.417 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:23.414+0000 7ffa92b7d700 1 -- 172.21.15.167:0/2027390457 <== osd.1 v2:172.21.15.167:6800/30035 11 ==== osd_op_reply(25 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7ffa8c00a5a0 con 0x5638b1cdb8d0 2023-10-05T19:30:23.417 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:23.414+0000 7ffa92b7d700 1 -- 172.21.15.167:0/2027390457 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:26 5.5 5:a0000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x5638b1d12f50 con 0x7ffa7c00d8e0 2023-10-05T19:30:23.418 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:23.414+0000 7ffa92b7d700 1 -- 172.21.15.167:0/2027390457 <== osd.0 v2:172.21.15.118:6800/27690 6 ==== osd_op_reply(26 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7ffa8c00a5a0 con 0x7ffa7c00d8e0 2023-10-05T19:30:23.418 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:23.414+0000 7ffa92b7d700 1 -- 172.21.15.167:0/2027390457 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:27 5.15 5:a8000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x5638b1d12f50 con 0x5638b1d10630 2023-10-05T19:30:23.419 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:23.414+0000 7ffa9237c700 1 -- 172.21.15.167:0/2027390457 <== osd.3 v2:172.21.15.167:6808/30036 7 ==== osd_op_reply(27 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7ffa84022020 con 0x5638b1d10630 2023-10-05T19:30:23.420 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:23.418+0000 7ffa9237c700 1 -- 172.21.15.167:0/2027390457 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:28 5.d 5:b0000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x5638b1d12f50 con 0x7ffa84007030 2023-10-05T19:30:23.420 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:23.418+0000 7ffa91b7b700 1 -- 172.21.15.167:0/2027390457 <== osd.2 v2:172.21.15.167:6816/30037 4 ==== osd_op_reply(28 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7ffa7c006070 con 0x7ffa84007030 2023-10-05T19:30:23.420 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:23.418+0000 7ffa91b7b700 1 -- 172.21.15.167:0/2027390457 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:29 5.1d 5:b8000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x5638b1d12f50 con 0x5638b1cdb8d0 2023-10-05T19:30:23.421 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:23.418+0000 7ffa92b7d700 1 -- 172.21.15.167:0/2027390457 <== osd.1 v2:172.21.15.167:6800/30035 12 ==== osd_op_reply(29 [pgnls start_epoch 32 out=107b] v32'4 uv4 ondisk = 1) v8 ==== 144+0+107 (crc 0 0 0) 0x7ffa8c00a5a0 con 0x5638b1cdb8d0 2023-10-05T19:30:23.421 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:23.418+0000 7ffa92b7d700 1 -- 172.21.15.167:0/2027390457 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:30 5.3 5:c0000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x5638b1d12f50 con 0x7ffa7c00d8e0 2023-10-05T19:30:23.422 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:23.418+0000 7ffa92b7d700 1 -- 172.21.15.167:0/2027390457 <== osd.0 v2:172.21.15.118:6800/27690 7 ==== osd_op_reply(30 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7ffa8c00a5a0 con 0x7ffa7c00d8e0 2023-10-05T19:30:23.422 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:23.418+0000 7ffa92b7d700 1 -- 172.21.15.167:0/2027390457 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:31 5.13 5:c8000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x5638b1d12f50 con 0x5638b1d10630 2023-10-05T19:30:23.423 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:23.418+0000 7ffa9237c700 1 -- 172.21.15.167:0/2027390457 <== osd.3 v2:172.21.15.167:6808/30036 8 ==== osd_op_reply(31 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7ffa84022020 con 0x5638b1d10630 2023-10-05T19:30:23.423 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:23.418+0000 7ffa9237c700 1 -- 172.21.15.167:0/2027390457 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:32 5.b 5:d0000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x5638b1d12f50 con 0x7ffa7c00d8e0 2023-10-05T19:30:23.424 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:23.422+0000 7ffa92b7d700 1 -- 172.21.15.167:0/2027390457 <== osd.0 v2:172.21.15.118:6800/27690 8 ==== osd_op_reply(32 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7ffa8c00a5a0 con 0x7ffa7c00d8e0 2023-10-05T19:30:23.424 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:23.422+0000 7ffa92b7d700 1 -- 172.21.15.167:0/2027390457 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:33 5.1b 5:d8000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x5638b1d12f50 con 0x7ffa84007030 2023-10-05T19:30:23.425 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:23.422+0000 7ffa91b7b700 1 -- 172.21.15.167:0/2027390457 <== osd.2 v2:172.21.15.167:6816/30037 5 ==== osd_op_reply(33 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7ffa7c006070 con 0x7ffa84007030 2023-10-05T19:30:23.425 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:23.422+0000 7ffa91b7b700 1 -- 172.21.15.167:0/2027390457 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:34 5.7 5:e0000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x5638b1d12f50 con 0x5638b1d10630 2023-10-05T19:30:23.426 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:23.422+0000 7ffa9237c700 1 -- 172.21.15.167:0/2027390457 <== osd.3 v2:172.21.15.167:6808/30036 9 ==== osd_op_reply(34 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7ffa84022020 con 0x5638b1d10630 2023-10-05T19:30:23.426 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:23.422+0000 7ffa9237c700 1 -- 172.21.15.167:0/2027390457 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:35 5.17 5:e8000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x5638b1d12f50 con 0x7ffa7c00d8e0 2023-10-05T19:30:23.427 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:23.422+0000 7ffa92b7d700 1 -- 172.21.15.167:0/2027390457 <== osd.0 v2:172.21.15.118:6800/27690 9 ==== osd_op_reply(35 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7ffa8c00a5a0 con 0x7ffa7c00d8e0 2023-10-05T19:30:23.427 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:23.422+0000 7ffa92b7d700 1 -- 172.21.15.167:0/2027390457 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:36 5.f 5:f0000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x5638b1d12f50 con 0x5638b1cdb8d0 2023-10-05T19:30:23.428 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:23.426+0000 7ffa92b7d700 1 -- 172.21.15.167:0/2027390457 <== osd.1 v2:172.21.15.167:6800/30035 13 ==== osd_op_reply(36 [pgnls start_epoch 32 out=115b] v32'3 uv3 ondisk = 1) v8 ==== 144+0+115 (crc 0 0 0) 0x7ffa8c00a5a0 con 0x5638b1cdb8d0 2023-10-05T19:30:23.428 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:23.426+0000 7ffa92b7d700 1 -- 172.21.15.167:0/2027390457 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:37 5.1f 5:f8000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x5638b1d12f50 con 0x7ffa84007030 2023-10-05T19:30:23.429 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:23.426+0000 7ffa91b7b700 1 -- 172.21.15.167:0/2027390457 <== osd.2 v2:172.21.15.167:6816/30037 6 ==== osd_op_reply(37 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7ffa7c006070 con 0x7ffa84007030 2023-10-05T19:30:23.429 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:23.426+0000 7ffa94847e40 20 RGWRados::pool_iterate: got zonegroup_info.4f1f719d-9648-417b-b10c-e5d939825419 2023-10-05T19:30:23.429 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:23.426+0000 7ffa94847e40 20 RGWRados::pool_iterate: got zone_info.9a9df965-6612-45d4-a45f-6e3c889bacc2 2023-10-05T19:30:23.429 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:23.426+0000 7ffa94847e40 20 RGWRados::pool_iterate: got zone_names.default 2023-10-05T19:30:23.429 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:23.426+0000 7ffa94847e40 20 RGWRados::pool_iterate: got zonegroups_names.default 2023-10-05T19:30:23.429 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:23.426+0000 7ffa94847e40 20 rados->read ofs=0 len=0 2023-10-05T19:30:23.429 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:23.426+0000 7ffa94847e40 1 -- 172.21.15.167:0/2027390457 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:38 5.12 5:49953fa1:::default.realm:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32) v8 -- 0x5638b1d0f3e0 con 0x5638b1cdb8d0 2023-10-05T19:30:23.430 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:23.426+0000 7ffa92b7d700 1 -- 172.21.15.167:0/2027390457 <== osd.1 v2:172.21.15.167:6800/30035 14 ==== osd_op_reply(38 default.realm [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 ==== 157+0+0 (crc 0 0 0) 0x7ffa8c00a5a0 con 0x5638b1cdb8d0 2023-10-05T19:30:23.430 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:23.426+0000 7ffa94847e40 20 rados_obj.operate() r=-2 bl.length=0 2023-10-05T19:30:23.430 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:23.426+0000 7ffa94847e40 20 rados->read ofs=0 len=0 2023-10-05T19:30:23.430 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:23.426+0000 7ffa94847e40 1 -- 172.21.15.167:0/2027390457 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:39 5.f 5:f4c53578:::zonegroups_names.default:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32) v8 -- 0x5638b1d0fcd0 con 0x5638b1cdb8d0 2023-10-05T19:30:23.431 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:23.426+0000 7ffa92b7d700 1 -- 172.21.15.167:0/2027390457 <== osd.1 v2:172.21.15.167:6800/30035 15 ==== osd_op_reply(39 zonegroups_names.default [read 0~46 out=46b] v0'0 uv2 ondisk = 0) v8 ==== 168+0+46 (crc 0 0 0) 0x7ffa8c00a5a0 con 0x5638b1cdb8d0 2023-10-05T19:30:23.431 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:23.426+0000 7ffa94847e40 20 rados_obj.operate() r=0 bl.length=46 2023-10-05T19:30:23.431 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:23.426+0000 7ffa94847e40 20 rados->read ofs=0 len=0 2023-10-05T19:30:23.431 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:23.426+0000 7ffa94847e40 1 -- 172.21.15.167:0/2027390457 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:40 5.c 5:30d299a5:::zonegroup_info.4f1f719d-9648-417b-b10c-e5d939825419:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32) v8 -- 0x5638b1d0fcd0 con 0x7ffa7c00d8e0 2023-10-05T19:30:23.431 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:23.426+0000 7ffa92b7d700 1 -- 172.21.15.167:0/2027390457 <== osd.0 v2:172.21.15.118:6800/27690 10 ==== osd_op_reply(40 zonegroup_info.4f1f719d-9648-417b-b10c-e5d939825419 [read 0~362 out=362b] v0'0 uv1 ondisk = 0) v8 ==== 195+0+362 (crc 0 0 0) 0x7ffa8c00a5a0 con 0x7ffa7c00d8e0 2023-10-05T19:30:23.431 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:23.426+0000 7ffa94847e40 20 rados_obj.operate() r=0 bl.length=362 2023-10-05T19:30:23.432 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:23.426+0000 7ffa94847e40 20 rados->read ofs=0 len=0 2023-10-05T19:30:23.432 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:23.426+0000 7ffa94847e40 1 -- 172.21.15.167:0/2027390457 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:41 5.13 5:cbbf6c60:::converted:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32) v8 -- 0x5638b1d0fcd0 con 0x5638b1d10630 2023-10-05T19:30:23.432 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:23.430+0000 7ffa9237c700 1 -- 172.21.15.167:0/2027390457 <== osd.3 v2:172.21.15.167:6808/30036 10 ==== osd_op_reply(41 converted [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 ==== 153+0+0 (crc 0 0 0) 0x7ffa84022020 con 0x5638b1d10630 2023-10-05T19:30:23.432 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:23.430+0000 7ffa94847e40 20 rados_obj.operate() r=-2 bl.length=0 2023-10-05T19:30:23.433 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:23.430+0000 7ffa94847e40 20 rados->read ofs=0 len=0 2023-10-05T19:30:23.433 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:23.430+0000 7ffa94847e40 1 -- 172.21.15.167:0/2027390457 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:42 5.12 5:49953fa1:::default.realm:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32) v8 -- 0x5638b1d133c0 con 0x5638b1cdb8d0 2023-10-05T19:30:23.433 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:23.430+0000 7ffa92b7d700 1 -- 172.21.15.167:0/2027390457 <== osd.1 v2:172.21.15.167:6800/30035 16 ==== osd_op_reply(42 default.realm [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 ==== 157+0+0 (crc 0 0 0) 0x7ffa8c00a5a0 con 0x5638b1cdb8d0 2023-10-05T19:30:23.433 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:23.430+0000 7ffa94847e40 20 rados_obj.operate() r=-2 bl.length=0 2023-10-05T19:30:23.433 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:23.430+0000 7ffa94847e40 20 rados->read ofs=0 len=0 2023-10-05T19:30:23.433 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:23.430+0000 7ffa94847e40 1 -- 172.21.15.167:0/2027390457 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:43 5.f 5:f4c53578:::zonegroups_names.default:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32) v8 -- 0x5638b1d137c0 con 0x5638b1cdb8d0 2023-10-05T19:30:23.433 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:23.430+0000 7ffa92b7d700 1 -- 172.21.15.167:0/2027390457 <== osd.1 v2:172.21.15.167:6800/30035 17 ==== osd_op_reply(43 zonegroups_names.default [read 0~46 out=46b] v0'0 uv2 ondisk = 0) v8 ==== 168+0+46 (crc 0 0 0) 0x7ffa8c00a5a0 con 0x5638b1cdb8d0 2023-10-05T19:30:23.434 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:23.430+0000 7ffa94847e40 20 rados_obj.operate() r=0 bl.length=46 2023-10-05T19:30:23.434 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:23.430+0000 7ffa94847e40 1 -- 172.21.15.167:0/2027390457 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:44 5.0 5:00000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x5638b1d147d0 con 0x5638b1d10630 2023-10-05T19:30:23.435 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:23.430+0000 7ffa9237c700 1 -- 172.21.15.167:0/2027390457 <== osd.3 v2:172.21.15.167:6808/30036 11 ==== osd_op_reply(44 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7ffa84022020 con 0x5638b1d10630 2023-10-05T19:30:23.435 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:23.430+0000 7ffa9237c700 1 -- 172.21.15.167:0/2027390457 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:45 5.10 5:08000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x5638b1d147d0 con 0x7ffa84007030 2023-10-05T19:30:23.435 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:23.434+0000 7ffa91b7b700 1 -- 172.21.15.167:0/2027390457 <== osd.2 v2:172.21.15.167:6816/30037 7 ==== osd_op_reply(45 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7ffa7c006070 con 0x7ffa84007030 2023-10-05T19:30:23.436 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:23.434+0000 7ffa91b7b700 1 -- 172.21.15.167:0/2027390457 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:46 5.8 5:10000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x5638b1d147d0 con 0x7ffa7c00d8e0 2023-10-05T19:30:23.437 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:23.434+0000 7ffa92b7d700 1 -- 172.21.15.167:0/2027390457 <== osd.0 v2:172.21.15.118:6800/27690 11 ==== osd_op_reply(46 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7ffa8c00a5a0 con 0x7ffa7c00d8e0 2023-10-05T19:30:23.437 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:23.434+0000 7ffa92b7d700 1 -- 172.21.15.167:0/2027390457 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:47 5.18 5:18000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x5638b1d0f3e0 con 0x5638b1cdb8d0 2023-10-05T19:30:23.437 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:23.434+0000 7ffa92b7d700 1 -- 172.21.15.167:0/2027390457 <== osd.1 v2:172.21.15.167:6800/30035 18 ==== osd_op_reply(47 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7ffa8c00a5a0 con 0x5638b1cdb8d0 2023-10-05T19:30:23.438 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:23.434+0000 7ffa92b7d700 1 -- 172.21.15.167:0/2027390457 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:48 5.4 5:20000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x5638b1d0f3e0 con 0x5638b1cdb8d0 2023-10-05T19:30:23.438 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:23.434+0000 7ffa92b7d700 1 -- 172.21.15.167:0/2027390457 <== osd.1 v2:172.21.15.167:6800/30035 19 ==== osd_op_reply(48 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7ffa8c00a5a0 con 0x5638b1cdb8d0 2023-10-05T19:30:23.438 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:23.434+0000 7ffa92b7d700 1 -- 172.21.15.167:0/2027390457 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:49 5.14 5:28000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x5638b1d0f3e0 con 0x7ffa7c00d8e0 2023-10-05T19:30:23.439 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:23.434+0000 7ffa92b7d700 1 -- 172.21.15.167:0/2027390457 <== osd.0 v2:172.21.15.118:6800/27690 12 ==== osd_op_reply(49 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7ffa8c00a5a0 con 0x7ffa7c00d8e0 2023-10-05T19:30:23.439 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:23.434+0000 7ffa92b7d700 1 -- 172.21.15.167:0/2027390457 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:50 5.c 5:30000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x5638b1d0f3e0 con 0x7ffa7c00d8e0 2023-10-05T19:30:23.440 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:23.438+0000 7ffa92b7d700 1 -- 172.21.15.167:0/2027390457 <== osd.0 v2:172.21.15.118:6800/27690 13 ==== osd_op_reply(50 [pgnls start_epoch 32 out=112b] v23'1 uv1 ondisk = 1) v8 ==== 144+0+112 (crc 0 0 0) 0x7ffa8c00a5a0 con 0x7ffa7c00d8e0 2023-10-05T19:30:23.440 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:23.438+0000 7ffa92b7d700 1 -- 172.21.15.167:0/2027390457 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:51 5.1c 5:38000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x5638b1d0f3e0 con 0x7ffa84007030 2023-10-05T19:30:23.441 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:23.438+0000 7ffa91b7b700 1 -- 172.21.15.167:0/2027390457 <== osd.2 v2:172.21.15.167:6816/30037 8 ==== osd_op_reply(51 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7ffa7c006070 con 0x7ffa84007030 2023-10-05T19:30:23.441 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:23.438+0000 7ffa91b7b700 1 -- 172.21.15.167:0/2027390457 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:52 5.2 5:40000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x5638b1d0f3e0 con 0x5638b1d10630 2023-10-05T19:30:23.442 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:23.438+0000 7ffa9237c700 1 -- 172.21.15.167:0/2027390457 <== osd.3 v2:172.21.15.167:6808/30036 12 ==== osd_op_reply(52 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7ffa84022020 con 0x5638b1d10630 2023-10-05T19:30:23.442 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:23.438+0000 7ffa9237c700 1 -- 172.21.15.167:0/2027390457 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:53 5.12 5:48000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x5638b1d0f3e0 con 0x5638b1cdb8d0 2023-10-05T19:30:23.443 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:23.438+0000 7ffa92b7d700 1 -- 172.21.15.167:0/2027390457 <== osd.1 v2:172.21.15.167:6800/30035 20 ==== osd_op_reply(53 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7ffa8c00a5a0 con 0x5638b1cdb8d0 2023-10-05T19:30:23.443 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:23.438+0000 7ffa92b7d700 1 -- 172.21.15.167:0/2027390457 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:54 5.a 5:50000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x5638b1d0f3e0 con 0x7ffa7c00d8e0 2023-10-05T19:30:23.444 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:23.442+0000 7ffa92b7d700 1 -- 172.21.15.167:0/2027390457 <== osd.0 v2:172.21.15.118:6800/27690 14 ==== osd_op_reply(54 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7ffa8c00a5a0 con 0x7ffa7c00d8e0 2023-10-05T19:30:23.444 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:23.442+0000 7ffa92b7d700 1 -- 172.21.15.167:0/2027390457 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:55 5.1a 5:58000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x5638b1d0f3e0 con 0x5638b1cdb8d0 2023-10-05T19:30:23.445 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:23.442+0000 7ffa92b7d700 1 -- 172.21.15.167:0/2027390457 <== osd.1 v2:172.21.15.167:6800/30035 21 ==== osd_op_reply(55 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7ffa8c00a5a0 con 0x5638b1cdb8d0 2023-10-05T19:30:23.445 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:23.442+0000 7ffa92b7d700 1 -- 172.21.15.167:0/2027390457 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:56 5.6 5:60000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x5638b1d0f3e0 con 0x7ffa84007030 2023-10-05T19:30:23.446 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:23.442+0000 7ffa91b7b700 1 -- 172.21.15.167:0/2027390457 <== osd.2 v2:172.21.15.167:6816/30037 9 ==== osd_op_reply(56 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7ffa7c006070 con 0x7ffa84007030 2023-10-05T19:30:23.446 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:23.442+0000 7ffa91b7b700 1 -- 172.21.15.167:0/2027390457 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:57 5.16 5:68000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x5638b1d0f3e0 con 0x5638b1d10630 2023-10-05T19:30:23.447 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:23.442+0000 7ffa9237c700 1 -- 172.21.15.167:0/2027390457 <== osd.3 v2:172.21.15.167:6808/30036 13 ==== osd_op_reply(57 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7ffa84022020 con 0x5638b1d10630 2023-10-05T19:30:23.447 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:23.442+0000 7ffa9237c700 1 -- 172.21.15.167:0/2027390457 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:58 5.e 5:70000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x5638b1d0f3e0 con 0x5638b1d10630 2023-10-05T19:30:23.447 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:23.446+0000 7ffa9237c700 1 -- 172.21.15.167:0/2027390457 <== osd.3 v2:172.21.15.167:6808/30036 14 ==== osd_op_reply(58 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7ffa84022020 con 0x5638b1d10630 2023-10-05T19:30:23.448 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:23.446+0000 7ffa9237c700 1 -- 172.21.15.167:0/2027390457 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:59 5.1e 5:78000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x5638b1d0f3e0 con 0x7ffa7c00d8e0 2023-10-05T19:30:23.448 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:23.446+0000 7ffa92b7d700 1 -- 172.21.15.167:0/2027390457 <== osd.0 v2:172.21.15.118:6800/27690 15 ==== osd_op_reply(59 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7ffa8c03f040 con 0x7ffa7c00d8e0 2023-10-05T19:30:23.449 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:23.446+0000 7ffa92b7d700 1 -- 172.21.15.167:0/2027390457 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:60 5.1 5:80000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x5638b1d0f3e0 con 0x5638b1d10630 2023-10-05T19:30:23.450 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:23.446+0000 7ffa9237c700 1 -- 172.21.15.167:0/2027390457 <== osd.3 v2:172.21.15.167:6808/30036 15 ==== osd_op_reply(60 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7ffa84022020 con 0x5638b1d10630 2023-10-05T19:30:23.450 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:23.446+0000 7ffa9237c700 1 -- 172.21.15.167:0/2027390457 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:61 5.11 5:88000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x5638b1d0f3e0 con 0x5638b1d10630 2023-10-05T19:30:23.450 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:23.446+0000 7ffa9237c700 1 -- 172.21.15.167:0/2027390457 <== osd.3 v2:172.21.15.167:6808/30036 16 ==== osd_op_reply(61 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7ffa84022020 con 0x5638b1d10630 2023-10-05T19:30:23.450 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:23.446+0000 7ffa9237c700 1 -- 172.21.15.167:0/2027390457 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:62 5.9 5:90000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x5638b1d0f3e0 con 0x5638b1cdb8d0 2023-10-05T19:30:23.451 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:23.446+0000 7ffa92b7d700 1 -- 172.21.15.167:0/2027390457 <== osd.1 v2:172.21.15.167:6800/30035 22 ==== osd_op_reply(62 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7ffa8c03f040 con 0x5638b1cdb8d0 2023-10-05T19:30:23.451 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:23.446+0000 7ffa92b7d700 1 -- 172.21.15.167:0/2027390457 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:63 5.19 5:98000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x5638b1d0f3e0 con 0x5638b1cdb8d0 2023-10-05T19:30:23.452 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:23.450+0000 7ffa92b7d700 1 -- 172.21.15.167:0/2027390457 <== osd.1 v2:172.21.15.167:6800/30035 23 ==== osd_op_reply(63 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7ffa8c03f040 con 0x5638b1cdb8d0 2023-10-05T19:30:23.452 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:23.450+0000 7ffa92b7d700 1 -- 172.21.15.167:0/2027390457 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:64 5.5 5:a0000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x5638b1d0f3e0 con 0x7ffa7c00d8e0 2023-10-05T19:30:23.453 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:23.450+0000 7ffa92b7d700 1 -- 172.21.15.167:0/2027390457 <== osd.0 v2:172.21.15.118:6800/27690 16 ==== osd_op_reply(64 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7ffa8c03f040 con 0x7ffa7c00d8e0 2023-10-05T19:30:23.453 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:23.450+0000 7ffa92b7d700 1 -- 172.21.15.167:0/2027390457 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:65 5.15 5:a8000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x5638b1d0f3e0 con 0x5638b1d10630 2023-10-05T19:30:23.454 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:23.450+0000 7ffa9237c700 1 -- 172.21.15.167:0/2027390457 <== osd.3 v2:172.21.15.167:6808/30036 17 ==== osd_op_reply(65 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7ffa84022020 con 0x5638b1d10630 2023-10-05T19:30:23.454 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:23.450+0000 7ffa9237c700 1 -- 172.21.15.167:0/2027390457 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:66 5.d 5:b0000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x5638b1d0f3e0 con 0x7ffa84007030 2023-10-05T19:30:23.455 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:23.450+0000 7ffa91b7b700 1 -- 172.21.15.167:0/2027390457 <== osd.2 v2:172.21.15.167:6816/30037 10 ==== osd_op_reply(66 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7ffa7c006070 con 0x7ffa84007030 2023-10-05T19:30:23.455 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:23.450+0000 7ffa91b7b700 1 -- 172.21.15.167:0/2027390457 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:67 5.1d 5:b8000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x5638b1d0f3e0 con 0x5638b1cdb8d0 2023-10-05T19:30:23.456 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:23.454+0000 7ffa92b7d700 1 -- 172.21.15.167:0/2027390457 <== osd.1 v2:172.21.15.167:6800/30035 24 ==== osd_op_reply(67 [pgnls start_epoch 32 out=107b] v32'4 uv4 ondisk = 1) v8 ==== 144+0+107 (crc 0 0 0) 0x7ffa8c03f040 con 0x5638b1cdb8d0 2023-10-05T19:30:23.456 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:23.454+0000 7ffa92b7d700 1 -- 172.21.15.167:0/2027390457 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:68 5.3 5:c0000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x5638b1d0f3e0 con 0x7ffa7c00d8e0 2023-10-05T19:30:23.456 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:23.454+0000 7ffa92b7d700 1 -- 172.21.15.167:0/2027390457 <== osd.0 v2:172.21.15.118:6800/27690 17 ==== osd_op_reply(68 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7ffa8c03f040 con 0x7ffa7c00d8e0 2023-10-05T19:30:23.457 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:23.454+0000 7ffa92b7d700 1 -- 172.21.15.167:0/2027390457 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:69 5.13 5:c8000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x5638b1d0f3e0 con 0x5638b1d10630 2023-10-05T19:30:23.457 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:23.454+0000 7ffa9237c700 1 -- 172.21.15.167:0/2027390457 <== osd.3 v2:172.21.15.167:6808/30036 18 ==== osd_op_reply(69 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7ffa84022020 con 0x5638b1d10630 2023-10-05T19:30:23.458 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:23.454+0000 7ffa9237c700 1 -- 172.21.15.167:0/2027390457 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:70 5.b 5:d0000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x5638b1d0f3e0 con 0x7ffa7c00d8e0 2023-10-05T19:30:23.458 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:23.454+0000 7ffa92b7d700 1 -- 172.21.15.167:0/2027390457 <== osd.0 v2:172.21.15.118:6800/27690 18 ==== osd_op_reply(70 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7ffa8c03f040 con 0x7ffa7c00d8e0 2023-10-05T19:30:23.458 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:23.454+0000 7ffa92b7d700 1 -- 172.21.15.167:0/2027390457 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:71 5.1b 5:d8000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x5638b1d0f3e0 con 0x7ffa84007030 2023-10-05T19:30:23.459 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:23.454+0000 7ffa91b7b700 1 -- 172.21.15.167:0/2027390457 <== osd.2 v2:172.21.15.167:6816/30037 11 ==== osd_op_reply(71 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7ffa7c006070 con 0x7ffa84007030 2023-10-05T19:30:23.459 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:23.454+0000 7ffa91b7b700 1 -- 172.21.15.167:0/2027390457 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:72 5.7 5:e0000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x5638b1d0f3e0 con 0x5638b1d10630 2023-10-05T19:30:23.460 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:23.458+0000 7ffa9237c700 1 -- 172.21.15.167:0/2027390457 <== osd.3 v2:172.21.15.167:6808/30036 19 ==== osd_op_reply(72 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7ffa84022020 con 0x5638b1d10630 2023-10-05T19:30:23.460 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:23.458+0000 7ffa9237c700 1 -- 172.21.15.167:0/2027390457 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:73 5.17 5:e8000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x5638b1d0f3e0 con 0x7ffa7c00d8e0 2023-10-05T19:30:23.461 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:23.458+0000 7ffa92b7d700 1 -- 172.21.15.167:0/2027390457 <== osd.0 v2:172.21.15.118:6800/27690 19 ==== osd_op_reply(73 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7ffa8c03f040 con 0x7ffa7c00d8e0 2023-10-05T19:30:23.461 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:23.458+0000 7ffa92b7d700 1 -- 172.21.15.167:0/2027390457 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:74 5.f 5:f0000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x5638b1d0f3e0 con 0x5638b1cdb8d0 2023-10-05T19:30:23.462 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:23.458+0000 7ffa92b7d700 1 -- 172.21.15.167:0/2027390457 <== osd.1 v2:172.21.15.167:6800/30035 25 ==== osd_op_reply(74 [pgnls start_epoch 32 out=115b] v32'3 uv3 ondisk = 1) v8 ==== 144+0+115 (crc 0 0 0) 0x7ffa8c03f040 con 0x5638b1cdb8d0 2023-10-05T19:30:23.462 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:23.458+0000 7ffa92b7d700 1 -- 172.21.15.167:0/2027390457 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:75 5.1f 5:f8000000::::head [pgnls start_epoch 32 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e32) v8 -- 0x5638b1d0f3e0 con 0x7ffa84007030 2023-10-05T19:30:23.463 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:23.458+0000 7ffa91b7b700 1 -- 172.21.15.167:0/2027390457 <== osd.2 v2:172.21.15.167:6816/30037 12 ==== osd_op_reply(75 [pgnls start_epoch 32 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7ffa7c006070 con 0x7ffa84007030 2023-10-05T19:30:23.463 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:23.458+0000 7ffa94847e40 20 RGWRados::pool_iterate: got zonegroup_info.4f1f719d-9648-417b-b10c-e5d939825419 2023-10-05T19:30:23.463 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:23.458+0000 7ffa94847e40 20 RGWRados::pool_iterate: got zone_info.9a9df965-6612-45d4-a45f-6e3c889bacc2 2023-10-05T19:30:23.463 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:23.458+0000 7ffa94847e40 20 RGWRados::pool_iterate: got zone_names.default 2023-10-05T19:30:23.463 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:23.458+0000 7ffa94847e40 20 RGWRados::pool_iterate: got zonegroups_names.default 2023-10-05T19:30:23.463 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:23.458+0000 7ffa94847e40 20 rados->read ofs=0 len=0 2023-10-05T19:30:23.463 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:23.458+0000 7ffa94847e40 1 -- 172.21.15.167:0/2027390457 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:76 5.f 5:f43b6ece:::zone_names.default:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32) v8 -- 0x5638b1d137c0 con 0x5638b1cdb8d0 2023-10-05T19:30:23.464 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:23.462+0000 7ffa92b7d700 1 -- 172.21.15.167:0/2027390457 <== osd.1 v2:172.21.15.167:6800/30035 26 ==== osd_op_reply(76 zone_names.default [read 0~46 out=46b] v0'0 uv3 ondisk = 0) v8 ==== 162+0+46 (crc 0 0 0) 0x7ffa8c03f040 con 0x5638b1cdb8d0 2023-10-05T19:30:23.464 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:23.462+0000 7ffa94847e40 20 rados_obj.operate() r=0 bl.length=46 2023-10-05T19:30:23.464 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:23.462+0000 7ffa94847e40 20 rados->read ofs=0 len=0 2023-10-05T19:30:23.464 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:23.462+0000 7ffa94847e40 1 -- 172.21.15.167:0/2027390457 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:77 5.1d 5:b8374eb4:::zone_info.9a9df965-6612-45d4-a45f-6e3c889bacc2:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32) v8 -- 0x5638b1d13b70 con 0x5638b1cdb8d0 2023-10-05T19:30:23.465 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:23.462+0000 7ffa92b7d700 1 -- 172.21.15.167:0/2027390457 <== osd.1 v2:172.21.15.167:6800/30035 27 ==== osd_op_reply(77 zone_info.9a9df965-6612-45d4-a45f-6e3c889bacc2 [read 0~874 out=874b] v0'0 uv4 ondisk = 0) v8 ==== 190+0+874 (crc 0 0 0) 0x7ffa8c03f040 con 0x5638b1cdb8d0 2023-10-05T19:30:23.465 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:23.462+0000 7ffa94847e40 20 rados_obj.operate() r=0 bl.length=874 2023-10-05T19:30:23.466 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:23.462+0000 7ffa94847e40 20 rados->read ofs=0 len=0 2023-10-05T19:30:23.466 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:23.462+0000 7ffa94847e40 1 -- 172.21.15.167:0/2027390457 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:78 5.f 5:f4c53578:::zonegroups_names.default:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32) v8 -- 0x5638b1d13ba0 con 0x5638b1cdb8d0 2023-10-05T19:30:23.466 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:23.462+0000 7ffa92b7d700 1 -- 172.21.15.167:0/2027390457 <== osd.1 v2:172.21.15.167:6800/30035 28 ==== osd_op_reply(78 zonegroups_names.default [read 0~46 out=46b] v0'0 uv2 ondisk = 0) v8 ==== 168+0+46 (crc 0 0 0) 0x7ffa8c03f040 con 0x5638b1cdb8d0 2023-10-05T19:30:23.466 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:23.462+0000 7ffa94847e40 20 rados_obj.operate() r=0 bl.length=46 2023-10-05T19:30:23.466 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:23.462+0000 7ffa94847e40 20 rados->read ofs=0 len=0 2023-10-05T19:30:23.466 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:23.462+0000 7ffa94847e40 1 -- 172.21.15.167:0/2027390457 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:79 5.c 5:30d299a5:::zonegroup_info.4f1f719d-9648-417b-b10c-e5d939825419:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32) v8 -- 0x5638b1d13fd0 con 0x7ffa7c00d8e0 2023-10-05T19:30:23.467 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:23.462+0000 7ffa92b7d700 1 -- 172.21.15.167:0/2027390457 <== osd.0 v2:172.21.15.118:6800/27690 20 ==== osd_op_reply(79 zonegroup_info.4f1f719d-9648-417b-b10c-e5d939825419 [read 0~362 out=362b] v0'0 uv1 ondisk = 0) v8 ==== 195+0+362 (crc 0 0 0) 0x7ffa8c03f040 con 0x7ffa7c00d8e0 2023-10-05T19:30:23.467 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:23.462+0000 7ffa94847e40 20 rados_obj.operate() r=0 bl.length=362 2023-10-05T19:30:23.467 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:23.462+0000 7ffa94847e40 20 rados->read ofs=0 len=0 2023-10-05T19:30:23.467 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:23.462+0000 7ffa94847e40 1 -- 172.21.15.167:0/2027390457 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:80 5.3 5:c0ad8cbd:::region_map:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32) v8 -- 0x5638b1d13fd0 con 0x7ffa7c00d8e0 2023-10-05T19:30:23.467 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:23.462+0000 7ffa92b7d700 1 -- 172.21.15.167:0/2027390457 <== osd.0 v2:172.21.15.118:6800/27690 21 ==== osd_op_reply(80 region_map [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 ==== 154+0+0 (crc 0 0 0) 0x7ffa8c03f040 con 0x7ffa7c00d8e0 2023-10-05T19:30:23.468 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:23.462+0000 7ffa94847e40 20 rados_obj.operate() r=-2 bl.length=0 2023-10-05T19:30:23.468 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:23.462+0000 7ffa94847e40 20 zone default found 2023-10-05T19:30:23.468 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:23.462+0000 7ffa94847e40 4 Realm: () 2023-10-05T19:30:23.468 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:23.462+0000 7ffa94847e40 4 ZoneGroup: default (4f1f719d-9648-417b-b10c-e5d939825419) 2023-10-05T19:30:23.468 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:23.462+0000 7ffa94847e40 4 Zone: default (9a9df965-6612-45d4-a45f-6e3c889bacc2) 2023-10-05T19:30:23.468 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:23.462+0000 7ffa94847e40 10 cannot find current period zonegroup using local zonegroup configuration 2023-10-05T19:30:23.468 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:23.462+0000 7ffa94847e40 20 zonegroup default 2023-10-05T19:30:23.468 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:23.462+0000 7ffa94847e40 20 rados->read ofs=0 len=0 2023-10-05T19:30:23.468 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:23.462+0000 7ffa94847e40 1 -- 172.21.15.167:0/2027390457 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:81 5.3 5:c52100b6:::period_config.default:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32) v8 -- 0x5638b1d13ea0 con 0x7ffa7c00d8e0 2023-10-05T19:30:23.469 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:23.466+0000 7ffa92b7d700 1 -- 172.21.15.167:0/2027390457 <== osd.0 v2:172.21.15.118:6800/27690 22 ==== osd_op_reply(81 period_config.default [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 ==== 165+0+0 (crc 0 0 0) 0x7ffa8c03f040 con 0x7ffa7c00d8e0 2023-10-05T19:30:23.469 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:23.466+0000 7ffa94847e40 20 rados_obj.operate() r=-2 bl.length=0 2023-10-05T19:30:23.469 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:23.466+0000 7ffa94847e40 20 rados->read ofs=0 len=0 2023-10-05T19:30:23.469 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:23.466+0000 7ffa94847e40 1 -- 172.21.15.167:0/2027390457 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:82 6.2 6:4347d321:::bucket.sync-source-hints.:head [call version.read in=11b,read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32) v8 -- 0x5638b1d15820 con 0x5638b1d10630 2023-10-05T19:30:23.469 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:23.466+0000 7ffa9237c700 1 -- 172.21.15.167:0/2027390457 <== osd.3 v2:172.21.15.167:6808/30036 20 ==== osd_op_reply(82 bucket.sync-source-hints. [call,read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 ==== 211+0+0 (crc 0 0 0) 0x7ffa84022020 con 0x5638b1d10630 2023-10-05T19:30:23.470 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:23.466+0000 7ffa94847e40 20 rados_obj.operate() r=-2 bl.length=0 2023-10-05T19:30:23.470 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:23.466+0000 7ffa94847e40 20 rados->read ofs=0 len=0 2023-10-05T19:30:23.470 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:23.466+0000 7ffa94847e40 1 -- 172.21.15.167:0/2027390457 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:83 6.b 6:d467b91b:::bucket.sync-target-hints.:head [call version.read in=11b,read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32) v8 -- 0x5638b1d15bd0 con 0x5638b1d10630 2023-10-05T19:30:23.470 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:23.466+0000 7ffa9237c700 1 -- 172.21.15.167:0/2027390457 <== osd.3 v2:172.21.15.167:6808/30036 21 ==== osd_op_reply(83 bucket.sync-target-hints. [call,read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 ==== 211+0+0 (crc 0 0 0) 0x7ffa84022020 con 0x5638b1d10630 2023-10-05T19:30:23.471 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:23.466+0000 7ffa94847e40 20 rados_obj.operate() r=-2 bl.length=0 2023-10-05T19:30:23.471 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:23.466+0000 7ffa94847e40 20 started sync module instance, tier type = 2023-10-05T19:30:23.471 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:23.466+0000 7ffa94847e40 20 started zone id=9a9df965-6612-45d4-a45f-6e3c889bacc2 (name=default) with tier type = 2023-10-05T19:30:23.471 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:23.466+0000 7ffa94847e40 1 -- 172.21.15.167:0/2027390457 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:84 7.1f 7:f95f44c2:::notify.0:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e32) v8 -- 0x5638b1d14ee0 con 0x7ffa84007030 2023-10-05T19:30:23.473 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:23.470+0000 7ffa91b7b700 1 -- 172.21.15.167:0/2027390457 <== osd.2 v2:172.21.15.167:6816/30037 13 ==== osd_op_reply(84 notify.0 [create] v32'270 uv270 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7ffa7c006070 con 0x7ffa84007030 2023-10-05T19:30:23.473 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:23.470+0000 7ffa94847e40 1 -- 172.21.15.167:0/2027390457 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:85 7.1f 7:f95f44c2:::notify.0:head [watch watch cookie 94801501442704] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e32) v8 -- 0x5638b1d0f900 con 0x7ffa84007030 2023-10-05T19:30:23.473 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:23.470+0000 7ffa94847e40 1 -- 172.21.15.167:0/2027390457 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:86 7.0 7:05bf5b68:::notify.1:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e32) v8 -- 0x5638b1d16960 con 0x5638b1cdb8d0 2023-10-05T19:30:23.476 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:23.474+0000 7ffa91b7b700 1 -- 172.21.15.167:0/2027390457 <== osd.2 v2:172.21.15.167:6816/30037 14 ==== osd_op_reply(85 notify.0 [watch watch cookie 94801501442704] v32'271 uv270 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7ffa7c006070 con 0x7ffa84007030 2023-10-05T19:30:23.476 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:23.474+0000 7ffa92b7d700 1 -- 172.21.15.167:0/2027390457 <== osd.1 v2:172.21.15.167:6800/30035 29 ==== osd_op_reply(86 notify.1 [create] v32'270 uv270 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7ffa8c052040 con 0x5638b1cdb8d0 2023-10-05T19:30:23.476 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:23.474+0000 7ffa94847e40 1 -- 172.21.15.167:0/2027390457 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:87 7.0 7:05bf5b68:::notify.1:head [watch watch cookie 94801501446304] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e32) v8 -- 0x5638b1d169e0 con 0x5638b1cdb8d0 2023-10-05T19:30:23.476 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:23.474+0000 7ffa94847e40 1 -- 172.21.15.167:0/2027390457 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:88 7.15 7:a93a5511:::notify.2:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e32) v8 -- 0x5638b1d17400 con 0x5638b1d10630 2023-10-05T19:30:23.479 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:23.474+0000 7ffa92b7d700 1 -- 172.21.15.167:0/2027390457 <== osd.1 v2:172.21.15.167:6800/30035 30 ==== osd_op_reply(87 notify.1 [watch watch cookie 94801501446304] v32'271 uv270 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7ffa8c052040 con 0x5638b1cdb8d0 2023-10-05T19:30:23.480 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:23.478+0000 7ffa9237c700 1 -- 172.21.15.167:0/2027390457 <== osd.3 v2:172.21.15.167:6808/30036 22 ==== osd_op_reply(88 notify.2 [create] v32'270 uv270 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7ffa84022020 con 0x5638b1d10630 2023-10-05T19:30:23.480 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:23.478+0000 7ffa94847e40 1 -- 172.21.15.167:0/2027390457 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:89 7.15 7:a93a5511:::notify.2:head [watch watch cookie 94801501447408] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e32) v8 -- 0x5638b1d16940 con 0x5638b1d10630 2023-10-05T19:30:23.480 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:23.478+0000 7ffa94847e40 1 -- 172.21.15.167:0/2027390457 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:90 7.e 7:7759931f:::notify.3:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e32) v8 -- 0x5638b1d181e0 con 0x7ffa84007030 2023-10-05T19:30:23.483 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:23.478+0000 7ffa9237c700 1 -- 172.21.15.167:0/2027390457 <== osd.3 v2:172.21.15.167:6808/30036 23 ==== osd_op_reply(89 notify.2 [watch watch cookie 94801501447408] v32'271 uv270 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7ffa84022020 con 0x5638b1d10630 2023-10-05T19:30:23.483 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:23.478+0000 7ffa91b7b700 1 -- 172.21.15.167:0/2027390457 <== osd.2 v2:172.21.15.167:6816/30037 15 ==== osd_op_reply(90 notify.3 [create] v32'270 uv270 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7ffa7c006070 con 0x7ffa84007030 2023-10-05T19:30:23.483 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:23.478+0000 7ffa94847e40 1 -- 172.21.15.167:0/2027390457 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:91 7.e 7:7759931f:::notify.3:head [watch watch cookie 94801501452576] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e32) v8 -- 0x5638b1d17d70 con 0x7ffa84007030 2023-10-05T19:30:23.483 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:23.478+0000 7ffa94847e40 1 -- 172.21.15.167:0/2027390457 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:92 7.d 7:b4812045:::notify.4:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e32) v8 -- 0x5638b1d19140 con 0x5638b1cdb8d0 2023-10-05T19:30:23.485 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:23.482+0000 7ffa91b7b700 1 -- 172.21.15.167:0/2027390457 <== osd.2 v2:172.21.15.167:6816/30037 16 ==== osd_op_reply(91 notify.3 [watch watch cookie 94801501452576] v32'271 uv270 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7ffa7c006070 con 0x7ffa84007030 2023-10-05T19:30:23.486 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:23.482+0000 7ffa92b7d700 1 -- 172.21.15.167:0/2027390457 <== osd.1 v2:172.21.15.167:6800/30035 31 ==== osd_op_reply(92 notify.4 [create] v32'270 uv270 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7ffa8c052040 con 0x5638b1cdb8d0 2023-10-05T19:30:23.486 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:23.482+0000 7ffa94847e40 1 -- 172.21.15.167:0/2027390457 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:93 7.d 7:b4812045:::notify.4:head [watch watch cookie 94801501457488] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e32) v8 -- 0x5638b1d19590 con 0x5638b1cdb8d0 2023-10-05T19:30:23.486 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:23.482+0000 7ffa94847e40 1 -- 172.21.15.167:0/2027390457 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:94 7.3 7:c609908c:::notify.5:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e32) v8 -- 0x5638b1d19fa0 con 0x7ffa7c00d8e0 2023-10-05T19:30:23.488 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:23.486+0000 7ffa92b7d700 1 -- 172.21.15.167:0/2027390457 <== osd.0 v2:172.21.15.118:6800/27690 23 ==== osd_op_reply(94 notify.5 [create] v32'270 uv270 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7ffa8c052040 con 0x7ffa7c00d8e0 2023-10-05T19:30:23.489 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:23.486+0000 7ffa94847e40 1 -- 172.21.15.167:0/2027390457 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:95 7.3 7:c609908c:::notify.5:head [watch watch cookie 94801501461168] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e32) v8 -- 0x5638b1d1a3f0 con 0x7ffa7c00d8e0 2023-10-05T19:30:23.489 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:23.486+0000 7ffa94847e40 1 -- 172.21.15.167:0/2027390457 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:96 7.14 7:2b04a3e9:::notify.6:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e32) v8 -- 0x5638b1d1ae00 con 0x7ffa7c00d8e0 2023-10-05T19:30:23.489 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:23.486+0000 7ffa92b7d700 1 -- 172.21.15.167:0/2027390457 <== osd.1 v2:172.21.15.167:6800/30035 32 ==== osd_op_reply(93 notify.4 [watch watch cookie 94801501457488] v32'271 uv270 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7ffa8c052040 con 0x5638b1cdb8d0 2023-10-05T19:30:23.492 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:23.490+0000 7ffa92b7d700 1 -- 172.21.15.167:0/2027390457 <== osd.0 v2:172.21.15.118:6800/27690 24 ==== osd_op_reply(96 notify.6 [create] v32'270 uv270 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7ffa8c052040 con 0x7ffa7c00d8e0 2023-10-05T19:30:23.492 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:23.490+0000 7ffa94847e40 1 -- 172.21.15.167:0/2027390457 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:97 7.14 7:2b04a3e9:::notify.6:head [watch watch cookie 94801501458592] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e32) v8 -- 0x5638b1d1ae00 con 0x7ffa7c00d8e0 2023-10-05T19:30:23.492 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:23.490+0000 7ffa94847e40 1 -- 172.21.15.167:0/2027390457 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:98 7.9 7:93e5b521:::notify.7:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e32) v8 -- 0x5638b1d1b810 con 0x7ffa7c00d8e0 2023-10-05T19:30:23.492 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:23.490+0000 7ffa92b7d700 1 -- 172.21.15.167:0/2027390457 <== osd.0 v2:172.21.15.118:6800/27690 25 ==== osd_op_reply(95 notify.5 [watch watch cookie 94801501461168] v32'271 uv270 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7ffa8c052040 con 0x7ffa7c00d8e0 2023-10-05T19:30:23.494 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:23.490+0000 7ffa92b7d700 1 -- 172.21.15.167:0/2027390457 <== osd.0 v2:172.21.15.118:6800/27690 26 ==== osd_op_reply(97 notify.6 [watch watch cookie 94801501458592] v32'271 uv270 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7ffa8c052040 con 0x7ffa7c00d8e0 2023-10-05T19:30:23.494 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:23.490+0000 7ffa92b7d700 1 -- 172.21.15.167:0/2027390457 <== osd.0 v2:172.21.15.118:6800/27690 27 ==== osd_op_reply(98 notify.7 [create] v32'270 uv270 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7ffa8c052040 con 0x7ffa7c00d8e0 2023-10-05T19:30:23.495 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:23.490+0000 7ffa94847e40 1 -- 172.21.15.167:0/2027390457 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:99 7.9 7:93e5b521:::notify.7:head [watch watch cookie 94801501467424] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e32) v8 -- 0x5638b1d1ae00 con 0x7ffa7c00d8e0 2023-10-05T19:30:23.495 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:23.490+0000 7ffa94847e40 20 add_watcher() i=0 2023-10-05T19:30:23.495 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:23.490+0000 7ffa94847e40 20 add_watcher() i=1 2023-10-05T19:30:23.495 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:23.490+0000 7ffa94847e40 20 add_watcher() i=2 2023-10-05T19:30:23.495 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:23.490+0000 7ffa94847e40 20 add_watcher() i=3 2023-10-05T19:30:23.495 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:23.490+0000 7ffa94847e40 20 add_watcher() i=4 2023-10-05T19:30:23.495 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:23.490+0000 7ffa94847e40 20 add_watcher() i=5 2023-10-05T19:30:23.495 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:23.490+0000 7ffa94847e40 20 add_watcher() i=6 2023-10-05T19:30:23.497 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:23.494+0000 7ffa92b7d700 1 -- 172.21.15.167:0/2027390457 <== osd.0 v2:172.21.15.118:6800/27690 28 ==== osd_op_reply(99 notify.7 [watch watch cookie 94801501467424] v32'271 uv270 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7ffa8c052040 con 0x7ffa7c00d8e0 2023-10-05T19:30:23.497 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:23.494+0000 7ffa94847e40 20 add_watcher() i=7 2023-10-05T19:30:23.497 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:23.494+0000 7ffa94847e40 2 all 8 watchers are set, enabling cache 2023-10-05T19:30:23.497 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:23.494+0000 7ffa94847e40 1 -- 172.21.15.167:0/2027390457 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:100 6.4 6:22d26bf9:::data_loggenerations_metadata:head [call version.check_conds in=74b,call version.read in=11b,read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32) v8 -- 0x5638b1d1b770 con 0x5638b1cdb8d0 2023-10-05T19:30:23.498 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:23.494+0000 7ffa92b7d700 1 -- 172.21.15.167:0/2027390457 <== osd.1 v2:172.21.15.167:6800/30035 33 ==== osd_op_reply(100 data_loggenerations_metadata [call,call out=48b,read 0~28 out=28b] v0'0 uv1 ondisk = 0) v8 ==== 256+0+76 (crc 0 0 0) 0x7ffa8c052040 con 0x5638b1cdb8d0 2023-10-05T19:30:23.498 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:23.494+0000 7ffa94847e40 1 -- 172.21.15.167:0/2027390457 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:101 6.4 6:22d26bf9:::data_loggenerations_metadata:head [watch watch cookie 94801501464848] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e32) v8 -- 0x5638b1d1bc60 con 0x5638b1cdb8d0 2023-10-05T19:30:23.501 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:23.498+0000 7ffa92b7d700 1 -- 172.21.15.167:0/2027390457 <== osd.1 v2:172.21.15.167:6800/30035 34 ==== osd_op_reply(101 data_loggenerations_metadata [watch watch cookie 94801501464848] v32'262 uv1 ondisk = 0) v8 ==== 172+0+0 (crc 0 0 0) 0x7ffa8c052040 con 0x5638b1cdb8d0 2023-10-05T19:30:23.501 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:23.498+0000 7ffa63fdf700 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2023-10-05T19:30:23.501 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:23.498+0000 7ffa94847e40 20 check_secure_mon_conn(): auth registy supported: methods=[2] modes=[2,1] 2023-10-05T19:30:23.501 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:23.498+0000 7ffa94847e40 20 check_secure_mon_conn(): mode 1 is insecure 2023-10-05T19:30:23.502 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:23.498+0000 7ffa94847e40 1 Processor -- start 2023-10-05T19:30:23.502 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:23.498+0000 7ffa94847e40 1 -- start start 2023-10-05T19:30:23.502 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:23.498+0000 7ffa94847e40 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5638b1d28ca0 0x5638b1d29070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-05T19:30:23.502 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:23.498+0000 7ffa94847e40 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x5638b1d295b0 con 0x5638b1d28ca0 2023-10-05T19:30:23.503 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:23.498+0000 7ffa91b7b700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5638b1d28ca0 0x5638b1d29070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-05T19:30:23.503 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:23.498+0000 7ffa91b7b700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5638b1d28ca0 0x5638b1d29070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.118:3300/0 says I am v2:172.21.15.167:33330/0 (socket says 172.21.15.167:33330) 2023-10-05T19:30:23.503 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:23.498+0000 7ffa91b7b700 1 -- 172.21.15.167:0/3111673451 learned_addr learned my addr 172.21.15.167:0/3111673451 (peer_addr_for_me v2:172.21.15.167:0/0) 2023-10-05T19:30:23.503 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:23.498+0000 7ffa91b7b700 1 -- 172.21.15.167:0/3111673451 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5638b1d296f0 con 0x5638b1d28ca0 2023-10-05T19:30:23.504 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:23.498+0000 7ffa91b7b700 1 --2- 172.21.15.167:0/3111673451 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5638b1d28ca0 0x5638b1d29070 secure :-1 s=READY pgs=708 cs=0 l=1 rev1=1 crypto rx=0x7ffa8400a520 tx=0x7ffa7c006000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7fdf63d2c6a99f67 server_cookie=0 in_seq=0 out_seq=0 2023-10-05T19:30:23.504 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:23.502+0000 7ffa637de700 1 -- 172.21.15.167:0/3111673451 <== mon.0 v2:172.21.15.118:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7ffa7c023040 con 0x5638b1d28ca0 2023-10-05T19:30:23.504 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:23.502+0000 7ffa637de700 1 -- 172.21.15.167:0/3111673451 <== mon.0 v2:172.21.15.118:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ffa7c024040 con 0x5638b1d28ca0 2023-10-05T19:30:23.504 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:23.502+0000 7ffa637de700 1 -- 172.21.15.167:0/3111673451 <== mon.0 v2:172.21.15.118:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7ffa7c013040 con 0x5638b1d28ca0 2023-10-05T19:30:23.505 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:23.502+0000 7ffa94847e40 1 -- 172.21.15.167:0/3111673451 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5638b1d28ca0 msgr2=0x5638b1d29070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-05T19:30:23.505 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:23.502+0000 7ffa94847e40 1 --2- 172.21.15.167:0/3111673451 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5638b1d28ca0 0x5638b1d29070 secure :-1 s=READY pgs=708 cs=0 l=1 rev1=1 crypto rx=0x7ffa8400a520 tx=0x7ffa7c006000 comp rx=0 tx=0).stop 2023-10-05T19:30:23.505 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:23.502+0000 7ffa94847e40 1 -- 172.21.15.167:0/3111673451 shutdown_connections 2023-10-05T19:30:23.505 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:23.502+0000 7ffa94847e40 1 --2- 172.21.15.167:0/3111673451 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5638b1d28ca0 0x5638b1d29070 unknown :-1 s=CLOSED pgs=708 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-05T19:30:23.505 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:23.502+0000 7ffa94847e40 1 -- 172.21.15.167:0/3111673451 >> 172.21.15.167:0/3111673451 conn(0x5638b1d24060 msgr2=0x5638b1d221b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-05T19:30:23.506 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:23.502+0000 7ffa94847e40 1 -- 172.21.15.167:0/3111673451 shutdown_connections 2023-10-05T19:30:23.506 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:23.502+0000 7ffa94847e40 1 -- 172.21.15.167:0/3111673451 wait complete. 2023-10-05T19:30:23.506 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:23.502+0000 7ffa94847e40 1 Processor -- start 2023-10-05T19:30:23.507 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:23.502+0000 7ffa94847e40 1 -- start start 2023-10-05T19:30:23.507 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:23.502+0000 7ffa94847e40 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5638b1d28ca0 0x5638b1d2e800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-05T19:30:23.507 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:23.502+0000 7ffa94847e40 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x5638b1d2f800 con 0x5638b1d28ca0 2023-10-05T19:30:23.507 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:23.502+0000 7ffa91b7b700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5638b1d28ca0 0x5638b1d2e800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-05T19:30:23.507 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:23.502+0000 7ffa91b7b700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5638b1d28ca0 0x5638b1d2e800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.118:3300/0 says I am v2:172.21.15.167:33336/0 (socket says 172.21.15.167:33336) 2023-10-05T19:30:23.507 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:23.502+0000 7ffa91b7b700 1 -- 172.21.15.167:0/2522232662 learned_addr learned my addr 172.21.15.167:0/2522232662 (peer_addr_for_me v2:172.21.15.167:0/0) 2023-10-05T19:30:23.508 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:23.502+0000 7ffa91b7b700 1 -- 172.21.15.167:0/2522232662 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5638b1c25bd0 con 0x5638b1d28ca0 2023-10-05T19:30:23.508 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:23.506+0000 7ffa91b7b700 1 --2- 172.21.15.167:0/2522232662 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5638b1d28ca0 0x5638b1d2e800 secure :-1 s=READY pgs=709 cs=0 l=1 rev1=1 crypto rx=0x7ffa7c000fd0 tx=0x7ffa7c030000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-05T19:30:23.508 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:23.506+0000 7ffa61fdb700 1 -- 172.21.15.167:0/2522232662 <== mon.0 v2:172.21.15.118:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7ffa7c023040 con 0x5638b1d28ca0 2023-10-05T19:30:23.508 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:23.506+0000 7ffa94847e40 1 -- 172.21.15.167:0/2522232662 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x5638b1d2ee00 con 0x5638b1d28ca0 2023-10-05T19:30:23.508 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:23.506+0000 7ffa61fdb700 1 -- 172.21.15.167:0/2522232662 <== mon.0 v2:172.21.15.118:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ffa7c03a420 con 0x5638b1d28ca0 2023-10-05T19:30:23.509 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:23.506+0000 7ffa61fdb700 1 -- 172.21.15.167:0/2522232662 <== mon.0 v2:172.21.15.118:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7ffa7c03a5b0 con 0x5638b1d28ca0 2023-10-05T19:30:23.509 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:23.506+0000 7ffa94847e40 1 -- 172.21.15.167:0/2522232662 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x5638b1d2f240 con 0x5638b1d28ca0 2023-10-05T19:30:23.510 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:23.506+0000 7ffa61fdb700 1 -- 172.21.15.167:0/2522232662 <== mon.0 v2:172.21.15.118:3300/0 4 ==== mgrmap(e 5) v1 ==== 40329+0+0 (secure 0 0 0) 0x7ffa7c03a710 con 0x5638b1d28ca0 2023-10-05T19:30:23.510 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:23.506+0000 7ffa61fdb700 1 --2- 172.21.15.167:0/2522232662 >> [v2:172.21.15.167:6824/29889,v1:172.21.15.167:6825/29889] conn(0x7ff9d8030c00 0x7ff9d80330b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-05T19:30:23.510 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:23.506+0000 7ffa61fdb700 1 -- 172.21.15.167:0/2522232662 <== mon.0 v2:172.21.15.118:3300/0 5 ==== osd_map(32..32 src has 1..32) v4 ==== 6260+0+0 (secure 0 0 0) 0x7ffa7c04fd30 con 0x5638b1d28ca0 2023-10-05T19:30:23.511 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:23.506+0000 7ffa92b7d700 1 --2- 172.21.15.167:0/2522232662 >> [v2:172.21.15.167:6824/29889,v1:172.21.15.167:6825/29889] conn(0x7ff9d8030c00 0x7ff9d80330b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-05T19:30:23.511 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:23.506+0000 7ffa94847e40 5 note: GC not initialized 2023-10-05T19:30:23.511 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:23.506+0000 7ffa92b7d700 1 --2- 172.21.15.167:0/2522232662 >> [v2:172.21.15.167:6824/29889,v1:172.21.15.167:6825/29889] conn(0x7ff9d8030c00 0x7ff9d80330b0 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x5638b1d1aed0 tx=0x7ffa8c036580 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-05T19:30:23.514 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:23.510+0000 7ff9d2ffd700 20 reqs_thread_entry: start 2023-10-05T19:30:23.514 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:23.510+0000 7ff9d0ff9700 20 reqs_thread_entry: start 2023-10-05T19:30:23.514 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:23.510+0000 7ffa94847e40 20 init_complete bucket index max shards: 11 2023-10-05T19:30:23.515 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:23.510+0000 7ffa94847e40 10 rgw notify: Started notification manager with: 1 workers 2023-10-05T19:30:23.515 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:23.510+0000 7ff9c2ffd700 20 rgw notify: INFO: next queues processing will happen at: Thu Oct 5 19:30:53 2023 2023-10-05T19:30:23.515 INFO:teuthology.orchestra.run.smithi167.stderr: 2023-10-05T19:30:23.515 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:23.510+0000 7ff9c27fc700 20 reqs_thread_entry: start 2023-10-05T19:30:23.515 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:23.510+0000 7ffa94847e40 20 get_system_obj_state: rctx=0x7ffce7471238 obj=default.rgw.meta:users.uid:ada state=0x5638b1bb20b0 s->prefetch_data=0 2023-10-05T19:30:23.515 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:23.510+0000 7ffa94847e40 10 cache get: name=default.rgw.meta+users.uid+ada : miss 2023-10-05T19:30:23.515 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:23.510+0000 7ffa94847e40 1 -- 172.21.15.167:0/2027390457 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:102 8.7 8:e1d59a8c:users.uid::ada:head [call version.read in=11b,getxattrs,stat] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32) v8 -- 0x5638b1d97110 con 0x7ffa7c00d8e0 2023-10-05T19:30:23.518 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:23.514+0000 7ffa92b7d700 1 -- 172.21.15.167:0/2027390457 <== osd.0 v2:172.21.15.118:6800/27690 29 ==== osd_op_reply(102 ada [call,getxattrs,stat] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 ==== 231+0+0 (crc 0 0 0) 0x7ffa8c052040 con 0x7ffa7c00d8e0 2023-10-05T19:30:23.519 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:23.514+0000 7ffa94847e40 10 cache put: name=default.rgw.meta+users.uid+ada info.flags=0x0 2023-10-05T19:30:23.519 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:23.514+0000 7ffa94847e40 10 adding default.rgw.meta+users.uid+ada to cache LRU end 2023-10-05T19:30:23.519 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:23.514+0000 7ffa94847e40 20 get_system_obj_state: rctx=0x7ffce7470f28 obj=default.rgw.meta:users.keys:MH1WC2XQ1S8UISFDZC8W state=0x5638b1bb20b0 s->prefetch_data=0 2023-10-05T19:30:23.519 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:23.514+0000 7ffa94847e40 10 cache get: name=default.rgw.meta+users.keys+MH1WC2XQ1S8UISFDZC8W : miss 2023-10-05T19:30:23.519 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:23.514+0000 7ffa94847e40 1 -- 172.21.15.167:0/2027390457 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:103 8.18 8:1f190392:users.keys::MH1WC2XQ1S8UISFDZC8W:head [getxattrs,stat] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32) v8 -- 0x5638b1d97110 con 0x5638b1cdb8d0 2023-10-05T19:30:23.519 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:23.514+0000 7ffa92b7d700 1 -- 172.21.15.167:0/2027390457 <== osd.1 v2:172.21.15.167:6800/30035 35 ==== osd_op_reply(103 MH1WC2XQ1S8UISFDZC8W [getxattrs,stat] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 ==== 206+0+0 (crc 0 0 0) 0x7ffa8c052040 con 0x5638b1cdb8d0 2023-10-05T19:30:23.519 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:23.514+0000 7ffa94847e40 10 cache put: name=default.rgw.meta+users.keys+MH1WC2XQ1S8UISFDZC8W info.flags=0x0 2023-10-05T19:30:23.519 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:23.514+0000 7ffa94847e40 10 adding default.rgw.meta+users.keys+MH1WC2XQ1S8UISFDZC8W to cache LRU end 2023-10-05T19:30:23.519 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:23.514+0000 7ffa94847e40 20 get_system_obj_state: rctx=0x7ffce74707c8 obj=default.rgw.meta:users.keys:MH1WC2XQ1S8UISFDZC8W state=0x5638b1bb20b0 s->prefetch_data=0 2023-10-05T19:30:23.519 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:23.514+0000 7ffa94847e40 10 cache get: name=default.rgw.meta+users.keys+MH1WC2XQ1S8UISFDZC8W : hit (negative entry) 2023-10-05T19:30:23.520 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:23.514+0000 7ffa94847e40 20 get_system_obj_state: rctx=0x7ffce7470e68 obj=default.rgw.meta:users.keys:MH1WC2XQ1S8UISFDZC8W state=0x5638b1bb20b0 s->prefetch_data=0 2023-10-05T19:30:23.520 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:23.514+0000 7ffa94847e40 10 cache get: name=default.rgw.meta+users.keys+MH1WC2XQ1S8UISFDZC8W : hit (negative entry) 2023-10-05T19:30:23.520 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:23.514+0000 7ffa94847e40 1 -- 172.21.15.167:0/2027390457 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:104 8.7 8:e1d59a8c:users.uid::ada:head [delete,create,call version.set in=58b,writefull 0~439 in=439b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e32) v8 -- 0x5638b1d99ba0 con 0x7ffa7c00d8e0 2023-10-05T19:30:23.521 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:23.518+0000 7ffa92b7d700 1 -- 172.21.15.167:0/2027390457 <== osd.0 v2:172.21.15.118:6800/27690 30 ==== osd_op_reply(104 ada [delete,create,call,writefull 0~439] v32'6 uv6 ondisk = 0) v8 ==== 273+0+0 (crc 0 0 0) 0x7ffa8c052040 con 0x7ffa7c00d8e0 2023-10-05T19:30:23.521 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:23.518+0000 7ffa94847e40 10 cache put: name=default.rgw.meta+users.uid+ada info.flags=0x17 2023-10-05T19:30:23.521 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:23.518+0000 7ffa94847e40 10 moving default.rgw.meta+users.uid+ada to cache LRU end 2023-10-05T19:30:23.521 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:23.518+0000 7ffa94847e40 10 distributing notification oid=default.rgw.control:notify.4 cni=[op: 0, obj: default.rgw.meta:users.uid:ada, ofs0, ns] 2023-10-05T19:30:23.521 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:23.518+0000 7ffa94847e40 1 -- 172.21.15.167:0/2027390457 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:105 7.d 7:b4812045:::notify.4:head [notify cookie 94801501980320 in=627b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32) v8 -- 0x5638b1d9a6b0 con 0x5638b1cdb8d0 2023-10-05T19:30:23.522 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:23.518+0000 7ffa92b7d700 1 -- 172.21.15.167:0/2027390457 <== osd.1 v2:172.21.15.167:6800/30035 36 ==== watch-notify(notify (1) cookie 94801501457488 notify 137438953496 ret 0) v3 ==== 657+0+0 (crc 0 0 0) 0x7ffa8c04e380 con 0x5638b1cdb8d0 2023-10-05T19:30:23.522 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:23.518+0000 7ffa8bfff700 10 rgw watcher librados: RGWWatcher::handle_notify() notify_id 137438953496 cookie 94801501457488 notifier 4797 bl.length()=615 2023-10-05T19:30:23.522 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:23.518+0000 7ffa8bfff700 10 rgw watcher librados: cache put: name=default.rgw.meta+users.uid+ada info.flags=0x17 2023-10-05T19:30:23.522 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:23.518+0000 7ffa8bfff700 10 rgw watcher librados: moving default.rgw.meta+users.uid+ada to cache LRU end 2023-10-05T19:30:23.522 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:23.518+0000 7ffa92b7d700 1 -- 172.21.15.167:0/2027390457 <== osd.1 v2:172.21.15.167:6800/30035 37 ==== osd_op_reply(105 notify.4 [notify cookie 94801501980320 out=8b] v0'0 uv270 ondisk = 0) v8 ==== 152+0+8 (crc 0 0 0) 0x7ffa8c052040 con 0x5638b1cdb8d0 2023-10-05T19:30:23.522 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:23.518+0000 7ffa8bfff700 1 -- 172.21.15.167:0/2027390457 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:106 7.d 7:b4812045:::notify.4:head [notify-ack in=20b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32) v8 -- 0x7ffa78003d90 con 0x5638b1cdb8d0 2023-10-05T19:30:23.523 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:23.518+0000 7ffa92b7d700 1 -- 172.21.15.167:0/2027390457 <== osd.1 v2:172.21.15.167:6800/30035 38 ==== watch-notify(notify_complete (2) cookie 94801501980320 notify 137438953496 ret 0) v3 ==== 42+0+48 (crc 0 0 0) 0x7ffa8c036bb0 con 0x5638b1cdb8d0 2023-10-05T19:30:23.523 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:23.518+0000 7ffa92b7d700 1 -- 172.21.15.167:0/2027390457 <== osd.1 v2:172.21.15.167:6800/30035 39 ==== osd_op_reply(106 notify.4 [notify-ack] v0'0 uv270 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7ffa8c052040 con 0x5638b1cdb8d0 2023-10-05T19:30:23.523 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:23.518+0000 7ffa94847e40 1 -- 172.21.15.167:0/2027390457 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:107 8.18 8:1f190392:users.keys::MH1WC2XQ1S8UISFDZC8W:head [delete,create,writefull 0~7 in=7b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e32) v8 -- 0x5638b1d9a6b0 con 0x5638b1cdb8d0 2023-10-05T19:30:23.525 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:23.522+0000 7ffa92b7d700 1 -- 172.21.15.167:0/2027390457 <== osd.1 v2:172.21.15.167:6800/30035 40 ==== osd_op_reply(107 MH1WC2XQ1S8UISFDZC8W [delete,create,writefull 0~7] v32'21 uv21 ondisk = 0) v8 ==== 248+0+0 (crc 0 0 0) 0x7ffa8c052040 con 0x5638b1cdb8d0 2023-10-05T19:30:23.526 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:23.522+0000 7ffa94847e40 10 cache put: name=default.rgw.meta+users.keys+MH1WC2XQ1S8UISFDZC8W info.flags=0x7 2023-10-05T19:30:23.526 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:23.522+0000 7ffa94847e40 10 moving default.rgw.meta+users.keys+MH1WC2XQ1S8UISFDZC8W to cache LRU end 2023-10-05T19:30:23.526 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:23.522+0000 7ffa94847e40 10 distributing notification oid=default.rgw.control:notify.2 cni=[op: 0, obj: default.rgw.meta:users.keys:MH1WC2XQ1S8UISFDZC8W, ofs0, ns] 2023-10-05T19:30:23.526 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:23.522+0000 7ffa94847e40 1 -- 172.21.15.167:0/2027390457 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:108 7.15 7:a93a5511:::notify.2:head [notify cookie 94801501987264 in=189b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32) v8 -- 0x5638b1d9a610 con 0x5638b1d10630 2023-10-05T19:30:23.527 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:23.522+0000 7ffa9237c700 1 -- 172.21.15.167:0/2027390457 <== osd.3 v2:172.21.15.167:6808/30036 24 ==== watch-notify(notify (1) cookie 94801501447408 notify 137438953496 ret 0) v3 ==== 219+0+0 (crc 0 0 0) 0x5638b1d01750 con 0x5638b1d10630 2023-10-05T19:30:23.527 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:23.522+0000 7ffa8b7fe700 10 rgw watcher librados: RGWWatcher::handle_notify() notify_id 137438953496 cookie 94801501447408 notifier 4797 bl.length()=177 2023-10-05T19:30:23.527 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:23.522+0000 7ffa8b7fe700 10 rgw watcher librados: cache put: name=default.rgw.meta+users.keys+MH1WC2XQ1S8UISFDZC8W info.flags=0x7 2023-10-05T19:30:23.527 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:23.522+0000 7ffa8b7fe700 10 rgw watcher librados: moving default.rgw.meta+users.keys+MH1WC2XQ1S8UISFDZC8W to cache LRU end 2023-10-05T19:30:23.527 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:23.522+0000 7ffa8b7fe700 1 -- 172.21.15.167:0/2027390457 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:109 7.15 7:a93a5511:::notify.2:head [notify-ack in=20b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32) v8 -- 0x7ff9e0003440 con 0x5638b1d10630 2023-10-05T19:30:23.527 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:23.522+0000 7ffa9237c700 1 -- 172.21.15.167:0/2027390457 <== osd.3 v2:172.21.15.167:6808/30036 25 ==== osd_op_reply(108 notify.2 [notify cookie 94801501987264 out=8b] v0'0 uv270 ondisk = 0) v8 ==== 152+0+8 (crc 0 0 0) 0x7ffa84022020 con 0x5638b1d10630 2023-10-05T19:30:23.528 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:23.522+0000 7ffa9237c700 1 -- 172.21.15.167:0/2027390457 <== osd.3 v2:172.21.15.167:6808/30036 26 ==== watch-notify(notify_complete (2) cookie 94801501987264 notify 137438953496 ret 0) v3 ==== 42+0+48 (crc 0 0 0) 0x5638b1d012b0 con 0x5638b1d10630 2023-10-05T19:30:23.528 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:23.522+0000 7ffa9237c700 1 -- 172.21.15.167:0/2027390457 <== osd.3 v2:172.21.15.167:6808/30036 27 ==== osd_op_reply(109 notify.2 [notify-ack] v0'0 uv270 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7ffa84022020 con 0x5638b1d10630 2023-10-05T19:30:23.528 INFO:teuthology.orchestra.run.smithi167.stdout:{ 2023-10-05T19:30:23.528 INFO:teuthology.orchestra.run.smithi167.stdout: "user_id": "ada", 2023-10-05T19:30:23.528 INFO:teuthology.orchestra.run.smithi167.stdout: "display_name": "Ms. Admin User", 2023-10-05T19:30:23.528 INFO:teuthology.orchestra.run.smithi167.stdout: "email": "", 2023-10-05T19:30:23.528 INFO:teuthology.orchestra.run.smithi167.stdout: "suspended": 0, 2023-10-05T19:30:23.528 INFO:teuthology.orchestra.run.smithi167.stdout: "max_buckets": 0, 2023-10-05T19:30:23.529 INFO:teuthology.orchestra.run.smithi167.stdout: "subusers": [], 2023-10-05T19:30:23.529 INFO:teuthology.orchestra.run.smithi167.stdout: "keys": [ 2023-10-05T19:30:23.529 INFO:teuthology.orchestra.run.smithi167.stdout: { 2023-10-05T19:30:23.529 INFO:teuthology.orchestra.run.smithi167.stdout: "user": "ada", 2023-10-05T19:30:23.529 INFO:teuthology.orchestra.run.smithi167.stdout: "access_key": "MH1WC2XQ1S8UISFDZC8W", 2023-10-05T19:30:23.529 INFO:teuthology.orchestra.run.smithi167.stdout: "secret_key": "dQyrTPA0s248YeN5bBv4ukvKU0kh54LWWywkrpoG" 2023-10-05T19:30:23.529 INFO:teuthology.orchestra.run.smithi167.stdout: } 2023-10-05T19:30:23.529 INFO:teuthology.orchestra.run.smithi167.stdout: ], 2023-10-05T19:30:23.529 INFO:teuthology.orchestra.run.smithi167.stdout: "swift_keys": [], 2023-10-05T19:30:23.529 INFO:teuthology.orchestra.run.smithi167.stdout: "caps": [ 2023-10-05T19:30:23.529 INFO:teuthology.orchestra.run.smithi167.stdout: { 2023-10-05T19:30:23.529 INFO:teuthology.orchestra.run.smithi167.stdout: "type": "buckets", 2023-10-05T19:30:23.530 INFO:teuthology.orchestra.run.smithi167.stdout: "perm": "*" 2023-10-05T19:30:23.530 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2023-10-05T19:30:23.530 INFO:teuthology.orchestra.run.smithi167.stdout: { 2023-10-05T19:30:23.530 INFO:teuthology.orchestra.run.smithi167.stdout: "type": "info", 2023-10-05T19:30:23.530 INFO:teuthology.orchestra.run.smithi167.stdout: "perm": "read" 2023-10-05T19:30:23.530 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2023-10-05T19:30:23.530 INFO:teuthology.orchestra.run.smithi167.stdout: { 2023-10-05T19:30:23.530 INFO:teuthology.orchestra.run.smithi167.stdout: "type": "ratelimit", 2023-10-05T19:30:23.530 INFO:teuthology.orchestra.run.smithi167.stdout: "perm": "*" 2023-10-05T19:30:23.530 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2023-10-05T19:30:23.530 INFO:teuthology.orchestra.run.smithi167.stdout: { 2023-10-05T19:30:23.530 INFO:teuthology.orchestra.run.smithi167.stdout: "type": "usage", 2023-10-05T19:30:23.531 INFO:teuthology.orchestra.run.smithi167.stdout: "perm": "*" 2023-10-05T19:30:23.531 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2023-10-05T19:30:23.531 INFO:teuthology.orchestra.run.smithi167.stdout: { 2023-10-05T19:30:23.531 INFO:teuthology.orchestra.run.smithi167.stdout: "type": "users", 2023-10-05T19:30:23.531 INFO:teuthology.orchestra.run.smithi167.stdout: "perm": "*" 2023-10-05T19:30:23.531 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2023-10-05T19:30:23.531 INFO:teuthology.orchestra.run.smithi167.stdout: { 2023-10-05T19:30:23.531 INFO:teuthology.orchestra.run.smithi167.stdout: "type": "zone", 2023-10-05T19:30:23.531 INFO:teuthology.orchestra.run.smithi167.stdout: "perm": "*" 2023-10-05T19:30:23.531 INFO:teuthology.orchestra.run.smithi167.stdout: } 2023-10-05T19:30:23.531 INFO:teuthology.orchestra.run.smithi167.stdout: ], 2023-10-05T19:30:23.531 INFO:teuthology.orchestra.run.smithi167.stdout: "op_mask": "read, write, delete", 2023-10-05T19:30:23.532 INFO:teuthology.orchestra.run.smithi167.stdout: "default_placement": "", 2023-10-05T19:30:23.532 INFO:teuthology.orchestra.run.smithi167.stdout: "default_storage_class": "", 2023-10-05T19:30:23.532 INFO:teuthology.orchestra.run.smithi167.stdout: "placement_tags": [], 2023-10-05T19:30:23.532 INFO:teuthology.orchestra.run.smithi167.stdout: "bucket_quota": { 2023-10-05T19:30:23.532 INFO:teuthology.orchestra.run.smithi167.stdout: "enabled": false, 2023-10-05T19:30:23.532 INFO:teuthology.orchestra.run.smithi167.stdout: "check_on_raw": false, 2023-10-05T19:30:23.532 INFO:teuthology.orchestra.run.smithi167.stdout: "max_size": -1, 2023-10-05T19:30:23.532 INFO:teuthology.orchestra.run.smithi167.stdout: "max_size_kb": 0, 2023-10-05T19:30:23.532 INFO:teuthology.orchestra.run.smithi167.stdout: "max_objects": -1 2023-10-05T19:30:23.532 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2023-10-05T19:30:23.532 INFO:teuthology.orchestra.run.smithi167.stdout: "user_quota": { 2023-10-05T19:30:23.532 INFO:teuthology.orchestra.run.smithi167.stdout: "enabled": false, 2023-10-05T19:30:23.532 INFO:teuthology.orchestra.run.smithi167.stdout: "check_on_raw": false, 2023-10-05T19:30:23.533 INFO:teuthology.orchestra.run.smithi167.stdout: "max_size": -1, 2023-10-05T19:30:23.533 INFO:teuthology.orchestra.run.smithi167.stdout: "max_size_kb": 0, 2023-10-05T19:30:23.533 INFO:teuthology.orchestra.run.smithi167.stdout: "max_objects": -1 2023-10-05T19:30:23.533 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2023-10-05T19:30:23.533 INFO:teuthology.orchestra.run.smithi167.stdout: "temp_url_keys": [], 2023-10-05T19:30:23.533 INFO:teuthology.orchestra.run.smithi167.stdout: "type": "rgw", 2023-10-05T19:30:23.533 INFO:teuthology.orchestra.run.smithi167.stdout: "mfa_ids": [] 2023-10-05T19:30:23.533 INFO:teuthology.orchestra.run.smithi167.stdout:} 2023-10-05T19:30:23.533 INFO:teuthology.orchestra.run.smithi167.stdout: 2023-10-05T19:30:23.534 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:23.526+0000 7ffa94847e40 1 -- 172.21.15.167:0/2027390457 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:110 7.1f 7:f95f44c2:::notify.0:head [watch unwatch cookie 94801501442704] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e32) v8 -- 0x5638b1cf0940 con 0x7ffa84007030 2023-10-05T19:30:23.534 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:23.526+0000 7ffa91b7b700 1 -- 172.21.15.167:0/2027390457 <== osd.2 v2:172.21.15.167:6816/30037 17 ==== osd_op_reply(110 notify.0 [watch unwatch cookie 94801501442704] v32'272 uv270 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7ffa7c006070 con 0x7ffa84007030 2023-10-05T19:30:23.534 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:23.526+0000 7ffa94847e40 20 remove_watcher() i=0 2023-10-05T19:30:23.534 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:23.526+0000 7ffa94847e40 2 removed watcher, disabling cache 2023-10-05T19:30:23.535 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:23.526+0000 7ffa94847e40 1 -- 172.21.15.167:0/2027390457 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:111 7.0 7:05bf5b68:::notify.1:head [watch unwatch cookie 94801501446304] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e32) v8 -- 0x5638b1d15290 con 0x5638b1cdb8d0 2023-10-05T19:30:23.535 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:23.530+0000 7ffa92b7d700 1 -- 172.21.15.167:0/2027390457 <== osd.1 v2:172.21.15.167:6800/30035 41 ==== osd_op_reply(111 notify.1 [watch unwatch cookie 94801501446304] v32'272 uv270 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7ffa8c052040 con 0x5638b1cdb8d0 2023-10-05T19:30:23.535 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:23.530+0000 7ffa94847e40 20 remove_watcher() i=1 2023-10-05T19:30:23.535 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:23.530+0000 7ffa94847e40 1 -- 172.21.15.167:0/2027390457 --> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] -- osd_op(unknown.0.0:112 7.15 7:a93a5511:::notify.2:head [watch unwatch cookie 94801501447408] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e32) v8 -- 0x5638b1d15290 con 0x5638b1d10630 2023-10-05T19:30:23.535 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:23.530+0000 7ffa9237c700 1 -- 172.21.15.167:0/2027390457 <== osd.3 v2:172.21.15.167:6808/30036 28 ==== osd_op_reply(112 notify.2 [watch unwatch cookie 94801501447408] v32'272 uv270 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7ffa84022020 con 0x5638b1d10630 2023-10-05T19:30:23.536 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:23.534+0000 7ffa94847e40 20 remove_watcher() i=2 2023-10-05T19:30:23.536 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:23.534+0000 7ffa94847e40 1 -- 172.21.15.167:0/2027390457 --> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] -- osd_op(unknown.0.0:113 7.e 7:7759931f:::notify.3:head [watch unwatch cookie 94801501452576] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e32) v8 -- 0x5638b1d15290 con 0x7ffa84007030 2023-10-05T19:30:23.538 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:23.534+0000 7ffa91b7b700 1 -- 172.21.15.167:0/2027390457 <== osd.2 v2:172.21.15.167:6816/30037 18 ==== osd_op_reply(113 notify.3 [watch unwatch cookie 94801501452576] v32'272 uv270 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7ffa7c006070 con 0x7ffa84007030 2023-10-05T19:30:23.538 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:23.534+0000 7ffa94847e40 20 remove_watcher() i=3 2023-10-05T19:30:23.538 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:23.534+0000 7ffa94847e40 1 -- 172.21.15.167:0/2027390457 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:114 7.d 7:b4812045:::notify.4:head [watch unwatch cookie 94801501457488] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e32) v8 -- 0x5638b1d15290 con 0x5638b1cdb8d0 2023-10-05T19:30:23.541 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:23.538+0000 7ffa92b7d700 1 -- 172.21.15.167:0/2027390457 <== osd.1 v2:172.21.15.167:6800/30035 42 ==== osd_op_reply(114 notify.4 [watch unwatch cookie 94801501457488] v32'272 uv270 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7ffa8c052040 con 0x5638b1cdb8d0 2023-10-05T19:30:23.541 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:23.538+0000 7ffa94847e40 20 remove_watcher() i=4 2023-10-05T19:30:23.541 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:23.538+0000 7ffa94847e40 1 -- 172.21.15.167:0/2027390457 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:115 7.3 7:c609908c:::notify.5:head [watch unwatch cookie 94801501461168] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e32) v8 -- 0x5638b1d18c50 con 0x7ffa7c00d8e0 2023-10-05T19:30:23.544 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:23.538+0000 7ffa92b7d700 1 -- 172.21.15.167:0/2027390457 <== osd.0 v2:172.21.15.118:6800/27690 31 ==== osd_op_reply(115 notify.5 [watch unwatch cookie 94801501461168] v32'272 uv270 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7ffa8c052040 con 0x7ffa7c00d8e0 2023-10-05T19:30:23.544 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:23.542+0000 7ffa94847e40 20 remove_watcher() i=5 2023-10-05T19:30:23.544 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:23.542+0000 7ffa94847e40 1 -- 172.21.15.167:0/2027390457 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:116 7.14 7:2b04a3e9:::notify.6:head [watch unwatch cookie 94801501458592] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e32) v8 -- 0x5638b1d15290 con 0x7ffa7c00d8e0 2023-10-05T19:30:23.547 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:23.542+0000 7ffa92b7d700 1 -- 172.21.15.167:0/2027390457 <== osd.0 v2:172.21.15.118:6800/27690 32 ==== osd_op_reply(116 notify.6 [watch unwatch cookie 94801501458592] v32'272 uv270 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7ffa8c052040 con 0x7ffa7c00d8e0 2023-10-05T19:30:23.547 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:23.542+0000 7ffa94847e40 20 remove_watcher() i=6 2023-10-05T19:30:23.547 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:23.542+0000 7ffa94847e40 1 -- 172.21.15.167:0/2027390457 --> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] -- osd_op(unknown.0.0:117 7.9 7:93e5b521:::notify.7:head [watch unwatch cookie 94801501467424] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e32) v8 -- 0x5638b1d19ab0 con 0x7ffa7c00d8e0 2023-10-05T19:30:23.550 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:23.546+0000 7ffa92b7d700 1 -- 172.21.15.167:0/2027390457 <== osd.0 v2:172.21.15.118:6800/27690 33 ==== osd_op_reply(117 notify.7 [watch unwatch cookie 94801501467424] v32'272 uv270 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7ffa8c052040 con 0x7ffa7c00d8e0 2023-10-05T19:30:23.550 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:23.546+0000 7ffa94847e40 20 remove_watcher() i=7 2023-10-05T19:30:23.551 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:23.546+0000 7ffa63fdf700 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2023-10-05T19:30:23.552 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:23.550+0000 7ffa94847e40 1 -- 172.21.15.167:0/2027390457 --> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] -- osd_op(unknown.0.0:118 6.4 6:22d26bf9:::data_loggenerations_metadata:head [watch unwatch cookie 94801501464848] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e32) v8 -- 0x5638b1d15290 con 0x5638b1cdb8d0 2023-10-05T19:30:23.555 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:23.550+0000 7ffa92b7d700 1 -- 172.21.15.167:0/2027390457 <== osd.1 v2:172.21.15.167:6800/30035 43 ==== osd_op_reply(118 data_loggenerations_metadata [watch unwatch cookie 94801501464848] v32'263 uv1 ondisk = 0) v8 ==== 172+0+0 (crc 0 0 0) 0x7ffa8c052040 con 0x5638b1cdb8d0 2023-10-05T19:30:23.555 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:23.550+0000 7ffa94847e40 1 -- 172.21.15.167:0/2027390457 >> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] conn(0x7ffa7c00d8e0 msgr2=0x7ffa7c00c360 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-05T19:30:23.555 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:23.550+0000 7ffa94847e40 1 --2- 172.21.15.167:0/2027390457 >> [v2:172.21.15.118:6800/27690,v1:172.21.15.118:6801/27690] conn(0x7ffa7c00d8e0 0x7ffa7c00c360 crc :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-05T19:30:23.555 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:23.550+0000 7ffa94847e40 1 -- 172.21.15.167:0/2027390457 >> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] conn(0x5638b1cdb8d0 msgr2=0x5638b1d0c890 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-05T19:30:23.555 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:23.550+0000 7ffa94847e40 1 --2- 172.21.15.167:0/2027390457 >> [v2:172.21.15.167:6800/30035,v1:172.21.15.167:6801/30035] conn(0x5638b1cdb8d0 0x5638b1d0c890 crc :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-05T19:30:23.556 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:23.550+0000 7ffa94847e40 1 -- 172.21.15.167:0/2027390457 >> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] conn(0x7ffa84007030 msgr2=0x7ffa84000ed0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-05T19:30:23.556 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:23.550+0000 7ffa94847e40 1 --2- 172.21.15.167:0/2027390457 >> [v2:172.21.15.167:6816/30037,v1:172.21.15.167:6817/30037] conn(0x7ffa84007030 0x7ffa84000ed0 crc :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-05T19:30:23.556 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:23.550+0000 7ffa94847e40 1 -- 172.21.15.167:0/2027390457 >> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] conn(0x5638b1d10630 msgr2=0x5638b1d12a10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-05T19:30:23.556 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:23.550+0000 7ffa94847e40 1 --2- 172.21.15.167:0/2027390457 >> [v2:172.21.15.167:6808/30036,v1:172.21.15.167:6809/30036] conn(0x5638b1d10630 0x5638b1d12a10 crc :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-05T19:30:23.556 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:23.550+0000 7ffa94847e40 1 -- 172.21.15.167:0/2027390457 >> [v2:172.21.15.167:6824/29889,v1:172.21.15.167:6825/29889] conn(0x7ffa74030c50 msgr2=0x7ffa74033100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-05T19:30:23.556 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:23.550+0000 7ffa94847e40 1 --2- 172.21.15.167:0/2027390457 >> [v2:172.21.15.167:6824/29889,v1:172.21.15.167:6825/29889] conn(0x7ffa74030c50 0x7ffa74033100 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7ffa7c001e90 tx=0x7ffa7c0013b0 comp rx=0 tx=0).stop 2023-10-05T19:30:23.556 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:23.550+0000 7ffa92b7d700 1 -- 172.21.15.167:0/2027390457 reap_dead start 2023-10-05T19:30:23.556 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:23.550+0000 7ffa94847e40 1 -- 172.21.15.167:0/2027390457 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5638b1cdafc0 msgr2=0x5638b1d00d70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-05T19:30:23.556 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:23.550+0000 7ffa94847e40 1 --2- 172.21.15.167:0/2027390457 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5638b1cdafc0 0x5638b1d00d70 secure :-1 s=READY pgs=707 cs=0 l=1 rev1=1 crypto rx=0x7ffa84005540 tx=0x7ffa84003050 comp rx=0 tx=0).stop 2023-10-05T19:30:23.556 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:23.550+0000 7ffa94847e40 1 -- 172.21.15.167:0/2027390457 shutdown_connections 2023-10-05T19:30:23.556 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:23.550+0000 7ffa94847e40 1 --2- 172.21.15.167:0/2027390457 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5638b1cdafc0 0x5638b1d00d70 unknown :-1 s=CLOSED pgs=707 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-05T19:30:23.557 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:23.550+0000 7ffa94847e40 1 -- 172.21.15.167:0/2027390457 >> 172.21.15.167:0/2027390457 conn(0x5638b1c4e810 msgr2=0x5638b1cf4720 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-05T19:30:23.557 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:23.550+0000 7ffa94847e40 1 -- 172.21.15.167:0/2027390457 shutdown_connections 2023-10-05T19:30:23.557 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:23.550+0000 7ffa94847e40 1 -- 172.21.15.167:0/2027390457 wait complete. 2023-10-05T19:30:23.558 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:23.554+0000 7ffa94847e40 1 -- 172.21.15.167:0/2522232662 >> [v2:172.21.15.167:6824/29889,v1:172.21.15.167:6825/29889] conn(0x7ff9d8030c00 msgr2=0x7ff9d80330b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-05T19:30:23.558 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:23.554+0000 7ffa94847e40 1 --2- 172.21.15.167:0/2522232662 >> [v2:172.21.15.167:6824/29889,v1:172.21.15.167:6825/29889] conn(0x7ff9d8030c00 0x7ff9d80330b0 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x5638b1d1aed0 tx=0x7ffa8c036580 comp rx=0 tx=0).stop 2023-10-05T19:30:23.558 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:23.554+0000 7ffa94847e40 1 -- 172.21.15.167:0/2522232662 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5638b1d28ca0 msgr2=0x5638b1d2e800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-05T19:30:23.558 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:23.554+0000 7ffa94847e40 1 --2- 172.21.15.167:0/2522232662 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5638b1d28ca0 0x5638b1d2e800 secure :-1 s=READY pgs=709 cs=0 l=1 rev1=1 crypto rx=0x7ffa7c000fd0 tx=0x7ffa7c030000 comp rx=0 tx=0).stop 2023-10-05T19:30:23.558 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:23.554+0000 7ffa94847e40 1 -- 172.21.15.167:0/2522232662 shutdown_connections 2023-10-05T19:30:23.558 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:23.554+0000 7ffa94847e40 1 --2- 172.21.15.167:0/2522232662 >> [v2:172.21.15.167:6824/29889,v1:172.21.15.167:6825/29889] conn(0x7ff9d8030c00 0x7ff9d80330b0 unknown :-1 s=CLOSED pgs=243 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-05T19:30:23.558 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:23.554+0000 7ffa94847e40 1 --2- 172.21.15.167:0/2522232662 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5638b1d28ca0 0x5638b1d2e800 unknown :-1 s=CLOSED pgs=709 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-05T19:30:23.558 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:23.554+0000 7ffa94847e40 1 -- 172.21.15.167:0/2522232662 >> 172.21.15.167:0/2522232662 conn(0x5638b1d24060 msgr2=0x5638b1d22100 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-05T19:30:23.559 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:23.554+0000 7ffa94847e40 1 -- 172.21.15.167:0/2522232662 shutdown_connections 2023-10-05T19:30:23.559 INFO:teuthology.orchestra.run.smithi167.stderr:2023-10-05T19:30:23.554+0000 7ffa94847e40 1 -- 172.21.15.167:0/2522232662 wait complete. 2023-10-05T19:30:23.562 DEBUG:tasks.util.rgw: json result: {'user_id': 'ada', 'display_name': 'Ms. Admin User', 'email': '', 'suspended': 0, 'max_buckets': 0, 'subusers': [], 'keys': [{'user': 'ada', 'access_key': 'MH1WC2XQ1S8UISFDZC8W', 'secret_key': 'dQyrTPA0s248YeN5bBv4ukvKU0kh54LWWywkrpoG'}], 'swift_keys': [], 'caps': [{'type': 'buckets', 'perm': '*'}, {'type': 'info', 'perm': 'read'}, {'type': 'ratelimit', 'perm': '*'}, {'type': 'usage', 'perm': '*'}, {'type': 'users', 'perm': '*'}, {'type': 'zone', 'perm': '*'}], 'op_mask': 'read, write, delete', 'default_placement': '', 'default_storage_class': '', 'placement_tags': [], 'bucket_quota': {'enabled': False, 'check_on_raw': False, 'max_size': -1, 'max_size_kb': 0, 'max_objects': -1}, 'user_quota': {'enabled': False, 'check_on_raw': False, 'max_size': -1, 'max_size_kb': 0, 'max_objects': -1}, 'temp_url_keys': [], 'type': 'rgw', 'mfa_ids': []} 2023-10-05T19:30:23.562 ERROR:root:{'user_id': 'ada', 'display_name': 'Ms. Admin User', 'email': '', 'suspended': 0, 'max_buckets': 0, 'subusers': [], 'keys': [{'user': 'ada', 'access_key': 'MH1WC2XQ1S8UISFDZC8W', 'secret_key': 'dQyrTPA0s248YeN5bBv4ukvKU0kh54LWWywkrpoG'}], 'swift_keys': [], 'caps': [{'type': 'buckets', 'perm': '*'}, {'type': 'info', 'perm': 'read'}, {'type': 'ratelimit', 'perm': '*'}, {'type': 'usage', 'perm': '*'}, {'type': 'users', 'perm': '*'}, {'type': 'zone', 'perm': '*'}], 'op_mask': 'read, write, delete', 'default_placement': '', 'default_storage_class': '', 'placement_tags': [], 'bucket_quota': {'enabled': False, 'check_on_raw': False, 'max_size': -1, 'max_size_kb': 0, 'max_objects': -1}, 'user_quota': {'enabled': False, 'check_on_raw': False, 'max_size': -1, 'max_size_kb': 0, 'max_objects': -1}, 'temp_url_keys': [], 'type': 'rgw', 'mfa_ids': []} 2023-10-05T19:30:23.562 ERROR:root:0 2023-10-05T19:30:23.563 INFO:tasks.radosgw_admin_rest:radosgw-admin-rest: ['user', 'info'] {'uid': 'foo'} 2023-10-05T19:30:23.569 INFO:tasks.radosgw_admin_rest: json result: {'Code': 'NoSuchUser', 'RequestId': 'tx00000dc28a327ef4013e0-00651f0ecf-107d-default', 'HostId': '107d-default-default'} 2023-10-05T19:30:23.570 INFO:tasks.radosgw_admin_rest:radosgw-admin-rest: ['user', 'create'] {'uid': 'foo', 'display-name': 'Foo', 'email': 'foo@foo.com', 'access-key': '9te6NH5mcdcq0Tc5i8i1', 'secret-key': 'Ny4IOauQoL18Gp2zM7lC1vLmoawgqcYP/YGcWfXu', 'max-buckets': '4'} 2023-10-05T19:30:23.592 INFO:tasks.radosgw_admin_rest: json result: {'tenant': '', 'user_id': 'foo', 'display_name': 'Foo', 'email': 'foo@foo.com', 'suspended': 0, 'max_buckets': 4, 'subusers': [], 'keys': [{'user': 'foo', 'access_key': '9te6NH5mcdcq0Tc5i8i1', 'secret_key': 'Ny4IOauQoL18Gp2zM7lC1vLmoawgqcYP/YGcWfXu'}], 'swift_keys': [], 'caps': [], 'op_mask': 'read, write, delete', 'system': 'false', 'admin': 'false', 'default_placement': '', 'default_storage_class': '', 'placement_tags': [], 'bucket_quota': {'enabled': False, 'check_on_raw': False, 'max_size': -1, 'max_size_kb': 0, 'max_objects': -1}, 'user_quota': {'enabled': False, 'check_on_raw': False, 'max_size': -1, 'max_size_kb': 0, 'max_objects': -1}, 'temp_url_keys': [], 'type': 'rgw', 'mfa_ids': []} 2023-10-05T19:30:23.592 INFO:tasks.radosgw_admin_rest:radosgw-admin-rest: ['user', 'list'] {'list': '', 'max-entries': 0} 2023-10-05T19:30:23.627 INFO:tasks.radosgw_admin_rest: json result: {'keys': [], 'truncated': True, 'count': 0, 'marker': '8:1e6ca5ed:users.uid::foo:head'} 2023-10-05T19:30:23.627 INFO:tasks.radosgw_admin_rest:radosgw-admin-rest: ['user', 'list'] {'list': '', 'max-entries': 1} 2023-10-05T19:30:23.663 INFO:tasks.radosgw_admin_rest: json result: {'keys': ['foo'], 'truncated': True, 'count': 1, 'marker': '8:a00000ee:users.uid::bar.buckets:head'} 2023-10-05T19:30:23.663 INFO:tasks.radosgw_admin_rest:radosgw-admin-rest: ['user', 'list'] {'list': '', 'max-entries': 1, 'marker': '8:a00000ee:users.uid::bar.buckets:head'} 2023-10-05T19:30:23.680 INFO:tasks.radosgw_admin_rest: json result: {'keys': ['ada'], 'truncated': False, 'count': 1} 2023-10-05T19:30:23.680 INFO:tasks.radosgw_admin_rest:radosgw-admin-rest: ['user', 'info'] {'uid': 'foo'} 2023-10-05T19:30:23.684 INFO:tasks.radosgw_admin_rest: json result: {'tenant': '', 'user_id': 'foo', 'display_name': 'Foo', 'email': 'foo@foo.com', 'suspended': 0, 'max_buckets': 4, 'subusers': [], 'keys': [{'user': 'foo', 'access_key': '9te6NH5mcdcq0Tc5i8i1', 'secret_key': 'Ny4IOauQoL18Gp2zM7lC1vLmoawgqcYP/YGcWfXu'}], 'swift_keys': [], 'caps': [], 'op_mask': 'read, write, delete', 'system': 'false', 'admin': 'false', 'default_placement': '', 'default_storage_class': '', 'placement_tags': [], 'bucket_quota': {'enabled': False, 'check_on_raw': False, 'max_size': -1, 'max_size_kb': 0, 'max_objects': -1}, 'user_quota': {'enabled': False, 'check_on_raw': False, 'max_size': -1, 'max_size_kb': 0, 'max_objects': -1}, 'temp_url_keys': [], 'type': 'rgw', 'mfa_ids': []} 2023-10-05T19:30:23.684 INFO:tasks.radosgw_admin_rest:radosgw-admin-rest: ['user', 'info'] {'access-key': '9te6NH5mcdcq0Tc5i8i1', 'uid': 'uid_not_exist'} 2023-10-05T19:30:23.690 INFO:tasks.radosgw_admin_rest: json result: {'tenant': '', 'user_id': 'foo', 'display_name': 'Foo', 'email': 'foo@foo.com', 'suspended': 0, 'max_buckets': 4, 'subusers': [], 'keys': [{'user': 'foo', 'access_key': '9te6NH5mcdcq0Tc5i8i1', 'secret_key': 'Ny4IOauQoL18Gp2zM7lC1vLmoawgqcYP/YGcWfXu'}], 'swift_keys': [], 'caps': [], 'op_mask': 'read, write, delete', 'system': 'false', 'admin': 'false', 'default_placement': '', 'default_storage_class': '', 'placement_tags': [], 'bucket_quota': {'enabled': False, 'check_on_raw': False, 'max_size': -1, 'max_size_kb': 0, 'max_objects': -1}, 'user_quota': {'enabled': False, 'check_on_raw': False, 'max_size': -1, 'max_size_kb': 0, 'max_objects': -1}, 'temp_url_keys': [], 'type': 'rgw', 'mfa_ids': []} 2023-10-05T19:30:23.690 INFO:tasks.radosgw_admin_rest:radosgw-admin-rest: ['user', 'modify'] {'uid': 'foo', 'suspended': True} 2023-10-05T19:30:23.700 INFO:tasks.radosgw_admin_rest: json result: {'tenant': '', 'user_id': 'foo', 'display_name': 'Foo', 'email': 'foo@foo.com', 'suspended': 1, 'max_buckets': 4, 'subusers': [], 'keys': [{'user': 'foo', 'access_key': '9te6NH5mcdcq0Tc5i8i1', 'secret_key': 'Ny4IOauQoL18Gp2zM7lC1vLmoawgqcYP/YGcWfXu'}], 'swift_keys': [], 'caps': [], 'op_mask': 'read, write, delete', 'system': 'false', 'admin': 'false', 'default_placement': '', 'default_storage_class': '', 'placement_tags': [], 'bucket_quota': {'enabled': False, 'check_on_raw': False, 'max_size': -1, 'max_size_kb': 0, 'max_objects': -1}, 'user_quota': {'enabled': False, 'check_on_raw': False, 'max_size': -1, 'max_size_kb': 0, 'max_objects': -1}, 'temp_url_keys': [], 'type': 'rgw', 'mfa_ids': []} 2023-10-05T19:30:23.701 INFO:tasks.radosgw_admin_rest:radosgw-admin-rest: ['user', 'info'] {'uid': 'foo'} 2023-10-05T19:30:23.705 INFO:tasks.radosgw_admin_rest: json result: {'tenant': '', 'user_id': 'foo', 'display_name': 'Foo', 'email': 'foo@foo.com', 'suspended': 1, 'max_buckets': 4, 'subusers': [], 'keys': [{'user': 'foo', 'access_key': '9te6NH5mcdcq0Tc5i8i1', 'secret_key': 'Ny4IOauQoL18Gp2zM7lC1vLmoawgqcYP/YGcWfXu'}], 'swift_keys': [], 'caps': [], 'op_mask': 'read, write, delete', 'system': 'false', 'admin': 'false', 'default_placement': '', 'default_storage_class': '', 'placement_tags': [], 'bucket_quota': {'enabled': False, 'check_on_raw': False, 'max_size': -1, 'max_size_kb': 0, 'max_objects': -1}, 'user_quota': {'enabled': False, 'check_on_raw': False, 'max_size': -1, 'max_size_kb': 0, 'max_objects': -1}, 'temp_url_keys': [], 'type': 'rgw', 'mfa_ids': []} 2023-10-05T19:30:23.705 INFO:tasks.radosgw_admin_rest:radosgw-admin-rest: ['user', 'modify'] {'uid': 'foo', 'suspended': 'false'} 2023-10-05T19:30:23.714 INFO:tasks.radosgw_admin_rest: json result: {'tenant': '', 'user_id': 'foo', 'display_name': 'Foo', 'email': 'foo@foo.com', 'suspended': 0, 'max_buckets': 4, 'subusers': [], 'keys': [{'user': 'foo', 'access_key': '9te6NH5mcdcq0Tc5i8i1', 'secret_key': 'Ny4IOauQoL18Gp2zM7lC1vLmoawgqcYP/YGcWfXu'}], 'swift_keys': [], 'caps': [], 'op_mask': 'read, write, delete', 'system': 'false', 'admin': 'false', 'default_placement': '', 'default_storage_class': '', 'placement_tags': [], 'bucket_quota': {'enabled': False, 'check_on_raw': False, 'max_size': -1, 'max_size_kb': 0, 'max_objects': -1}, 'user_quota': {'enabled': False, 'check_on_raw': False, 'max_size': -1, 'max_size_kb': 0, 'max_objects': -1}, 'temp_url_keys': [], 'type': 'rgw', 'mfa_ids': []} 2023-10-05T19:30:23.714 INFO:tasks.radosgw_admin_rest:radosgw-admin-rest: ['user', 'info'] {'uid': 'foo'} 2023-10-05T19:30:23.718 INFO:tasks.radosgw_admin_rest: json result: {'tenant': '', 'user_id': 'foo', 'display_name': 'Foo', 'email': 'foo@foo.com', 'suspended': 0, 'max_buckets': 4, 'subusers': [], 'keys': [{'user': 'foo', 'access_key': '9te6NH5mcdcq0Tc5i8i1', 'secret_key': 'Ny4IOauQoL18Gp2zM7lC1vLmoawgqcYP/YGcWfXu'}], 'swift_keys': [], 'caps': [], 'op_mask': 'read, write, delete', 'system': 'false', 'admin': 'false', 'default_placement': '', 'default_storage_class': '', 'placement_tags': [], 'bucket_quota': {'enabled': False, 'check_on_raw': False, 'max_size': -1, 'max_size_kb': 0, 'max_objects': -1}, 'user_quota': {'enabled': False, 'check_on_raw': False, 'max_size': -1, 'max_size_kb': 0, 'max_objects': -1}, 'temp_url_keys': [], 'type': 'rgw', 'mfa_ids': []} 2023-10-05T19:30:23.719 INFO:tasks.radosgw_admin_rest:radosgw-admin-rest: ['key', 'create'] {'uid': 'foo', 'access-key': 'p5YnriCv1nAtykxBrupQ', 'secret-key': 'Q8Tk6Q/27hfbFSYdSkPtUqhqx1GgzvpXa4WARozh'} 2023-10-05T19:30:23.736 INFO:tasks.radosgw_admin_rest: json result: [{'user': 'foo', 'access_key': '9te6NH5mcdcq0Tc5i8i1', 'secret_key': 'Ny4IOauQoL18Gp2zM7lC1vLmoawgqcYP/YGcWfXu'}, {'user': 'foo', 'access_key': 'p5YnriCv1nAtykxBrupQ', 'secret_key': 'Q8Tk6Q/27hfbFSYdSkPtUqhqx1GgzvpXa4WARozh'}] 2023-10-05T19:30:23.736 INFO:tasks.radosgw_admin_rest:radosgw-admin-rest: ['user', 'info'] {'uid': 'foo'} 2023-10-05T19:30:23.741 INFO:tasks.radosgw_admin_rest: json result: {'tenant': '', 'user_id': 'foo', 'display_name': 'Foo', 'email': 'foo@foo.com', 'suspended': 0, 'max_buckets': 4, 'subusers': [], 'keys': [{'user': 'foo', 'access_key': '9te6NH5mcdcq0Tc5i8i1', 'secret_key': 'Ny4IOauQoL18Gp2zM7lC1vLmoawgqcYP/YGcWfXu'}, {'user': 'foo', 'access_key': 'p5YnriCv1nAtykxBrupQ', 'secret_key': 'Q8Tk6Q/27hfbFSYdSkPtUqhqx1GgzvpXa4WARozh'}], 'swift_keys': [], 'caps': [], 'op_mask': 'read, write, delete', 'system': 'false', 'admin': 'false', 'default_placement': '', 'default_storage_class': '', 'placement_tags': [], 'bucket_quota': {'enabled': False, 'check_on_raw': False, 'max_size': -1, 'max_size_kb': 0, 'max_objects': -1}, 'user_quota': {'enabled': False, 'check_on_raw': False, 'max_size': -1, 'max_size_kb': 0, 'max_objects': -1}, 'temp_url_keys': [], 'type': 'rgw', 'mfa_ids': []} 2023-10-05T19:30:23.741 INFO:tasks.radosgw_admin_rest:radosgw-admin-rest: ['key', 'rm'] {'uid': 'foo', 'access-key': 'p5YnriCv1nAtykxBrupQ'} 2023-10-05T19:30:23.755 INFO:tasks.radosgw_admin_rest: empty result 2023-10-05T19:30:23.755 INFO:tasks.radosgw_admin_rest:radosgw-admin-rest: ['user', 'info'] {'uid': 'foo'} 2023-10-05T19:30:23.760 INFO:tasks.radosgw_admin_rest: json result: {'tenant': '', 'user_id': 'foo', 'display_name': 'Foo', 'email': 'foo@foo.com', 'suspended': 0, 'max_buckets': 4, 'subusers': [], 'keys': [{'user': 'foo', 'access_key': '9te6NH5mcdcq0Tc5i8i1', 'secret_key': 'Ny4IOauQoL18Gp2zM7lC1vLmoawgqcYP/YGcWfXu'}], 'swift_keys': [], 'caps': [], 'op_mask': 'read, write, delete', 'system': 'false', 'admin': 'false', 'default_placement': '', 'default_storage_class': '', 'placement_tags': [], 'bucket_quota': {'enabled': False, 'check_on_raw': False, 'max_size': -1, 'max_size_kb': 0, 'max_objects': -1}, 'user_quota': {'enabled': False, 'check_on_raw': False, 'max_size': -1, 'max_size_kb': 0, 'max_objects': -1}, 'temp_url_keys': [], 'type': 'rgw', 'mfa_ids': []} 2023-10-05T19:30:23.760 INFO:tasks.radosgw_admin_rest:radosgw-admin-rest: ['subuser', 'create'] {'subuser': 'foo:foo1', 'secret-key': 'gpS2G9RREMrnbqlp29PP2D36kgPR1tm72n5fPYfL', 'key-type': 'swift'} 2023-10-05T19:30:23.776 INFO:tasks.radosgw_admin_rest: json result: [{'id': 'foo:foo1', 'permissions': ''}] 2023-10-05T19:30:23.776 INFO:tasks.radosgw_admin_rest:radosgw-admin-rest: ['user', 'info'] {'uid': 'foo'} 2023-10-05T19:30:23.780 INFO:tasks.radosgw_admin_rest: json result: {'tenant': '', 'user_id': 'foo', 'display_name': 'Foo', 'email': 'foo@foo.com', 'suspended': 0, 'max_buckets': 4, 'subusers': [{'id': 'foo:foo1', 'permissions': ''}], 'keys': [{'user': 'foo', 'access_key': '9te6NH5mcdcq0Tc5i8i1', 'secret_key': 'Ny4IOauQoL18Gp2zM7lC1vLmoawgqcYP/YGcWfXu'}], 'swift_keys': [{'user': 'foo:foo1', 'secret_key': 'gpS2G9RREMrnbqlp29PP2D36kgPR1tm72n5fPYfL'}], 'caps': [], 'op_mask': 'read, write, delete', 'system': 'false', 'admin': 'false', 'default_placement': '', 'default_storage_class': '', 'placement_tags': [], 'bucket_quota': {'enabled': False, 'check_on_raw': False, 'max_size': -1, 'max_size_kb': 0, 'max_objects': -1}, 'user_quota': {'enabled': False, 'check_on_raw': False, 'max_size': -1, 'max_size_kb': 0, 'max_objects': -1}, 'temp_url_keys': [], 'type': 'rgw', 'mfa_ids': []} 2023-10-05T19:30:23.781 INFO:tasks.radosgw_admin_rest:radosgw-admin-rest: ['subuser', 'create'] {'subuser': 'foo:foo2', 'secret-key': 'ri2VJQcKSYATOY6uaDUX7pxgkW+W1YmC6OCxPHwy', 'key-type': 'swift'} 2023-10-05T19:30:23.796 INFO:tasks.radosgw_admin_rest: json result: [{'id': 'foo:foo1', 'permissions': ''}, {'id': 'foo:foo2', 'permissions': ''}] 2023-10-05T19:30:23.797 INFO:tasks.radosgw_admin_rest:radosgw-admin-rest: ['user', 'info'] {'uid': 'foo'} 2023-10-05T19:30:23.800 INFO:tasks.radosgw_admin_rest: json result: {'tenant': '', 'user_id': 'foo', 'display_name': 'Foo', 'email': 'foo@foo.com', 'suspended': 0, 'max_buckets': 4, 'subusers': [{'id': 'foo:foo1', 'permissions': ''}, {'id': 'foo:foo2', 'permissions': ''}], 'keys': [{'user': 'foo', 'access_key': '9te6NH5mcdcq0Tc5i8i1', 'secret_key': 'Ny4IOauQoL18Gp2zM7lC1vLmoawgqcYP/YGcWfXu'}], 'swift_keys': [{'user': 'foo:foo1', 'secret_key': 'gpS2G9RREMrnbqlp29PP2D36kgPR1tm72n5fPYfL'}, {'user': 'foo:foo2', 'secret_key': 'ri2VJQcKSYATOY6uaDUX7pxgkW+W1YmC6OCxPHwy'}], 'caps': [], 'op_mask': 'read, write, delete', 'system': 'false', 'admin': 'false', 'default_placement': '', 'default_storage_class': '', 'placement_tags': [], 'bucket_quota': {'enabled': False, 'check_on_raw': False, 'max_size': -1, 'max_size_kb': 0, 'max_objects': -1}, 'user_quota': {'enabled': False, 'check_on_raw': False, 'max_size': -1, 'max_size_kb': 0, 'max_objects': -1}, 'temp_url_keys': [], 'type': 'rgw', 'mfa_ids': []} 2023-10-05T19:30:23.800 INFO:tasks.radosgw_admin_rest:radosgw-admin-rest: ['key', 'rm'] {'subuser': 'foo:foo1', 'key-type': 'swift'} 2023-10-05T19:30:23.813 INFO:tasks.radosgw_admin_rest: empty result 2023-10-05T19:30:23.813 INFO:tasks.radosgw_admin_rest:radosgw-admin-rest: ['user', 'info'] {'uid': 'foo'} 2023-10-05T19:30:23.818 INFO:tasks.radosgw_admin_rest: json result: {'tenant': '', 'user_id': 'foo', 'display_name': 'Foo', 'email': 'foo@foo.com', 'suspended': 0, 'max_buckets': 4, 'subusers': [{'id': 'foo:foo1', 'permissions': ''}, {'id': 'foo:foo2', 'permissions': ''}], 'keys': [{'user': 'foo', 'access_key': '9te6NH5mcdcq0Tc5i8i1', 'secret_key': 'Ny4IOauQoL18Gp2zM7lC1vLmoawgqcYP/YGcWfXu'}], 'swift_keys': [{'user': 'foo:foo2', 'secret_key': 'ri2VJQcKSYATOY6uaDUX7pxgkW+W1YmC6OCxPHwy'}], 'caps': [], 'op_mask': 'read, write, delete', 'system': 'false', 'admin': 'false', 'default_placement': '', 'default_storage_class': '', 'placement_tags': [], 'bucket_quota': {'enabled': False, 'check_on_raw': False, 'max_size': -1, 'max_size_kb': 0, 'max_objects': -1}, 'user_quota': {'enabled': False, 'check_on_raw': False, 'max_size': -1, 'max_size_kb': 0, 'max_objects': -1}, 'temp_url_keys': [], 'type': 'rgw', 'mfa_ids': []} 2023-10-05T19:30:23.818 INFO:tasks.radosgw_admin_rest:radosgw-admin-rest: ['subuser', 'rm'] {'subuser': 'foo:foo1'} 2023-10-05T19:30:23.827 INFO:tasks.radosgw_admin_rest: empty result 2023-10-05T19:30:23.827 INFO:tasks.radosgw_admin_rest:radosgw-admin-rest: ['user', 'info'] {'uid': 'foo'} 2023-10-05T19:30:23.831 INFO:tasks.radosgw_admin_rest: json result: {'tenant': '', 'user_id': 'foo', 'display_name': 'Foo', 'email': 'foo@foo.com', 'suspended': 0, 'max_buckets': 4, 'subusers': [{'id': 'foo:foo2', 'permissions': ''}], 'keys': [{'user': 'foo', 'access_key': '9te6NH5mcdcq0Tc5i8i1', 'secret_key': 'Ny4IOauQoL18Gp2zM7lC1vLmoawgqcYP/YGcWfXu'}], 'swift_keys': [{'user': 'foo:foo2', 'secret_key': 'ri2VJQcKSYATOY6uaDUX7pxgkW+W1YmC6OCxPHwy'}], 'caps': [], 'op_mask': 'read, write, delete', 'system': 'false', 'admin': 'false', 'default_placement': '', 'default_storage_class': '', 'placement_tags': [], 'bucket_quota': {'enabled': False, 'check_on_raw': False, 'max_size': -1, 'max_size_kb': 0, 'max_objects': -1}, 'user_quota': {'enabled': False, 'check_on_raw': False, 'max_size': -1, 'max_size_kb': 0, 'max_objects': -1}, 'temp_url_keys': [], 'type': 'rgw', 'mfa_ids': []} 2023-10-05T19:30:23.832 INFO:tasks.radosgw_admin_rest:radosgw-admin-rest: ['subuser', 'rm'] {'subuser': 'foo:foo2', 'key-type': 'swift', '{purge-keys': True} 2023-10-05T19:30:23.846 INFO:tasks.radosgw_admin_rest: empty result 2023-10-05T19:30:23.846 INFO:tasks.radosgw_admin_rest:radosgw-admin-rest: ['user', 'info'] {'uid': 'foo'} 2023-10-05T19:30:23.850 INFO:tasks.radosgw_admin_rest: json result: {'tenant': '', 'user_id': 'foo', 'display_name': 'Foo', 'email': 'foo@foo.com', 'suspended': 0, 'max_buckets': 4, 'subusers': [], 'keys': [{'user': 'foo', 'access_key': '9te6NH5mcdcq0Tc5i8i1', 'secret_key': 'Ny4IOauQoL18Gp2zM7lC1vLmoawgqcYP/YGcWfXu'}], 'swift_keys': [], 'caps': [], 'op_mask': 'read, write, delete', 'system': 'false', 'admin': 'false', 'default_placement': '', 'default_storage_class': '', 'placement_tags': [], 'bucket_quota': {'enabled': False, 'check_on_raw': False, 'max_size': -1, 'max_size_kb': 0, 'max_objects': -1}, 'user_quota': {'enabled': False, 'check_on_raw': False, 'max_size': -1, 'max_size_kb': 0, 'max_objects': -1}, 'temp_url_keys': [], 'type': 'rgw', 'mfa_ids': []} 2023-10-05T19:30:23.851 INFO:tasks.radosgw_admin_rest:radosgw-admin-rest: ['bucket', 'info'] {'uid': 'foo'} 2023-10-05T19:30:23.857 INFO:tasks.radosgw_admin_rest: json result: [] 2023-10-05T19:30:23.858 INFO:tasks.radosgw_admin_rest:radosgw-admin-rest: ['bucket', 'info'] {'uid': 'foo', 'stats': True} 2023-10-05T19:30:23.863 INFO:tasks.radosgw_admin_rest: json result: [] 2023-10-05T19:30:23.886 INFO:tasks.radosgw_admin_rest:radosgw-admin-rest: ['bucket', 'info'] {'uid': 'foo'} 2023-10-05T19:30:23.892 INFO:tasks.radosgw_admin_rest: json result: ['myfoo'] 2023-10-05T19:30:23.892 INFO:tasks.radosgw_admin_rest:radosgw-admin-rest: ['bucket', 'info'] {'bucket': 'myfoo', 'stats': True} 2023-10-05T19:30:23.898 INFO:tasks.radosgw_admin_rest: json result: {'bucket': 'myfoo', 'num_shards': 11, 'tenant': '', 'zonegroup': '4f1f719d-9648-417b-b10c-e5d939825419', 'placement_rule': 'default-placement', 'explicit_placement': {'data_pool': '', 'data_extra_pool': '', 'index_pool': ''}, 'id': '9a9df965-6612-45d4-a45f-6e3c889bacc2.4223.9', 'marker': '9a9df965-6612-45d4-a45f-6e3c889bacc2.4223.9', 'index_type': 'Normal', 'owner': 'foo', 'ver': '0#1,1#1,2#1,3#1,4#1,5#1,6#1,7#1,8#1,9#1,10#1', 'master_ver': '0#0,1#0,2#0,3#0,4#0,5#0,6#0,7#0,8#0,9#0,10#0', 'mtime': '2023-10-05T19:30:23.875081Z', 'creation_time': '2023-10-05T19:30:23.868235Z', 'max_marker': '0#,1#,2#,3#,4#,5#,6#,7#,8#,9#,10#', 'usage': {}, 'bucket_quota': {'enabled': False, 'check_on_raw': False, 'max_size': -1, 'max_size_kb': 0, 'max_objects': -1}} 2023-10-05T19:30:23.899 INFO:tasks.radosgw_admin_rest:radosgw-admin-rest: ['bucket', 'info'] {'uid': 'foo', 'stats': True} 2023-10-05T19:30:23.906 INFO:tasks.radosgw_admin_rest: json result: [{'bucket': 'myfoo', 'num_shards': 11, 'tenant': '', 'zonegroup': '4f1f719d-9648-417b-b10c-e5d939825419', 'placement_rule': 'default-placement', 'explicit_placement': {'data_pool': '', 'data_extra_pool': '', 'index_pool': ''}, 'id': '9a9df965-6612-45d4-a45f-6e3c889bacc2.4223.9', 'marker': '9a9df965-6612-45d4-a45f-6e3c889bacc2.4223.9', 'index_type': 'Normal', 'owner': 'foo', 'ver': '0#1,1#1,2#1,3#1,4#1,5#1,6#1,7#1,8#1,9#1,10#1', 'master_ver': '0#0,1#0,2#0,3#0,4#0,5#0,6#0,7#0,8#0,9#0,10#0', 'mtime': '2023-10-05T19:30:23.875081Z', 'creation_time': '2023-10-05T19:30:23.868235Z', 'max_marker': '0#,1#,2#,3#,4#,5#,6#,7#,8#,9#,10#', 'usage': {}, 'bucket_quota': {'enabled': False, 'check_on_raw': False, 'max_size': -1, 'max_size_kb': 0, 'max_objects': -1}}] 2023-10-05T19:30:23.917 INFO:tasks.radosgw_admin_rest:radosgw-admin-rest: ['bucket', 'info'] {'bucket': 'myfoo', 'stats': True} 2023-10-05T19:30:23.925 INFO:tasks.radosgw_admin_rest: json result: {'bucket': 'myfoo', 'num_shards': 11, 'tenant': '', 'zonegroup': '4f1f719d-9648-417b-b10c-e5d939825419', 'placement_rule': 'default-placement', 'explicit_placement': {'data_pool': '', 'data_extra_pool': '', 'index_pool': ''}, 'id': '9a9df965-6612-45d4-a45f-6e3c889bacc2.4223.9', 'marker': '9a9df965-6612-45d4-a45f-6e3c889bacc2.4223.9', 'index_type': 'Normal', 'owner': 'foo', 'ver': '0#1,1#1,2#1,3#1,4#1,5#2,6#1,7#1,8#1,9#1,10#1', 'master_ver': '0#0,1#0,2#0,3#0,4#0,5#0,6#0,7#0,8#0,9#0,10#0', 'mtime': '2023-10-05T19:30:23.875081Z', 'creation_time': '2023-10-05T19:30:23.868235Z', 'max_marker': '0#,1#,2#,3#,4#,5#,6#,7#,8#,9#,10#', 'usage': {'rgw.main': {'size': 3, 'size_actual': 4096, 'size_utilized': 3, 'size_kb': 1, 'size_kb_actual': 4, 'size_kb_utilized': 1, 'num_objects': 1}}, 'bucket_quota': {'enabled': False, 'check_on_raw': False, 'max_size': -1, 'max_size_kb': 0, 'max_objects': -1}} 2023-10-05T19:30:23.926 INFO:tasks.radosgw_admin_rest:radosgw-admin-rest: ['bucket', 'info'] {'bucket': 'doesnotexist'} 2023-10-05T19:30:23.932 INFO:tasks.radosgw_admin_rest: json result: {'Code': 'NoSuchBucket', 'RequestId': 'tx000003d35198edd6e526a-00651f0ecf-107d-default', 'HostId': '107d-default-default'} 2023-10-05T19:30:23.942 INFO:tasks.radosgw_admin_rest:radosgw-admin-rest: ['bucket', 'unlink'] {'uid': 'foo', 'bucket': 'myfoo'} 2023-10-05T19:30:23.955 INFO:tasks.radosgw_admin_rest: empty result 2023-10-05T19:30:23.956 INFO:tasks.radosgw_admin_rest:radosgw-admin-rest: ['user', 'create'] {'uid': 'fud', 'display-name': 'Fud', 'access-key': 'p5YnriCv1nAtykxBrupQ', 'secret-key': 'Q8Tk6Q/27hfbFSYdSkPtUqhqx1GgzvpXa4WARozh', 'max-buckets': '1'} 2023-10-05T19:30:23.977 INFO:tasks.radosgw_admin_rest: json result: {'tenant': '', 'user_id': 'fud', 'display_name': 'Fud', 'email': '', 'suspended': 0, 'max_buckets': 1, 'subusers': [], 'keys': [{'user': 'fud', 'access_key': 'p5YnriCv1nAtykxBrupQ', 'secret_key': 'Q8Tk6Q/27hfbFSYdSkPtUqhqx1GgzvpXa4WARozh'}], 'swift_keys': [], 'caps': [], 'op_mask': 'read, write, delete', 'system': 'false', 'admin': 'false', 'default_placement': '', 'default_storage_class': '', 'placement_tags': [], 'bucket_quota': {'enabled': False, 'check_on_raw': False, 'max_size': -1, 'max_size_kb': 0, 'max_objects': -1}, 'user_quota': {'enabled': False, 'check_on_raw': False, 'max_size': -1, 'max_size_kb': 0, 'max_objects': -1}, 'temp_url_keys': [], 'type': 'rgw', 'mfa_ids': []} 2023-10-05T19:30:24.033 INFO:tasks.radosgw_admin_rest:radosgw-admin-rest: ['bucket', 'link'] {'uid': 'fud', 'bucket': 'myfoo', 'bucket-id': '9a9df965-6612-45d4-a45f-6e3c889bacc2.4223.9'} 2023-10-05T19:30:24.053 INFO:tasks.radosgw_admin_rest: empty result 2023-10-05T19:30:24.057 INFO:tasks.radosgw_admin_rest:radosgw-admin-rest: ['bucket', 'link'] {'uid': 'foo', 'bucket': 'myfoo', 'bucket-id': '9a9df965-6612-45d4-a45f-6e3c889bacc2.4223.9'} 2023-10-05T19:30:24.079 INFO:tasks.radosgw_admin_rest: empty result 2023-10-05T19:30:24.080 INFO:tasks.radosgw_admin_rest:radosgw-admin-rest: ['user', 'rm'] {'uid': 'fud'} 2023-10-05T19:30:24.102 INFO:tasks.radosgw_admin_rest: empty result 2023-10-05T19:30:24.116 INFO:tasks.radosgw_admin_rest:radosgw-admin-rest: ['object', 'rm'] {'bucket': 'myfoo', 'object': 'four'} 2023-10-05T19:30:24.128 INFO:tasks.radosgw_admin_rest: empty result 2023-10-05T19:30:24.129 INFO:tasks.radosgw_admin_rest:radosgw-admin-rest: ['bucket', 'info'] {'bucket': 'myfoo', 'stats': True} 2023-10-05T19:30:24.135 INFO:tasks.radosgw_admin_rest: json result: {'bucket': 'myfoo', 'num_shards': 11, 'tenant': '', 'zonegroup': '4f1f719d-9648-417b-b10c-e5d939825419', 'placement_rule': 'default-placement', 'explicit_placement': {'data_pool': '', 'data_extra_pool': '', 'index_pool': ''}, 'id': '9a9df965-6612-45d4-a45f-6e3c889bacc2.4223.9', 'marker': '9a9df965-6612-45d4-a45f-6e3c889bacc2.4223.9', 'index_type': 'Normal', 'owner': 'foo', 'ver': '0#1,1#1,2#1,3#1,4#1,5#3,6#3,7#1,8#3,9#1,10#1', 'master_ver': '0#0,1#0,2#0,3#0,4#0,5#0,6#0,7#0,8#0,9#0,10#0', 'mtime': '2023-10-05T19:30:24.064557Z', 'creation_time': '2023-10-05T19:30:23.868235Z', 'max_marker': '0#,1#,2#,3#,4#,5#,6#,7#,8#,9#,10#', 'usage': {'rgw.main': {'size': 0, 'size_actual': 0, 'size_utilized': 0, 'size_kb': 0, 'size_kb_actual': 0, 'size_kb_utilized': 0, 'num_objects': 0}}, 'bucket_quota': {'enabled': False, 'check_on_raw': False, 'max_size': -1, 'max_size_kb': 0, 'max_objects': -1}} 2023-10-05T19:31:24.253 INFO:tasks.radosgw_admin_rest:radosgw-admin-rest: ['usage', 'show'] {'categories': 'delete_obj'} 2023-10-05T19:31:24.288 INFO:tasks.radosgw_admin_rest: json result: {'entries': [{'user': 'anonymous', 'buckets': [{'bucket': '', 'time': '2023-10-05T19:00:00.000000Z', 'epoch': 1696532400, 'owner': 'anonymous', 'categories': []}]}, {'user': 'bar', 'buckets': [{'bucket': 'myfoo6', 'time': '2023-10-05T19:00:00.000000Z', 'epoch': 1696532400, 'owner': 'bar', 'categories': []}]}, {'user': 'bud', 'buckets': [{'bucket': 'myfoo6', 'time': '2023-10-05T19:00:00.000000Z', 'epoch': 1696532400, 'owner': 'bud', 'categories': []}]}, {'user': 'foo', 'buckets': [{'bucket': 'myfoo', 'time': '2023-10-05T19:00:00.000000Z', 'epoch': 1696532400, 'owner': 'foo', 'categories': [{'category': 'delete_obj', 'bytes_sent': 0, 'bytes_received': 0, 'ops': 2, 'successful_ops': 2}]}, {'bucket': 'useless-bucket', 'time': '2023-10-05T19:00:00.000000Z', 'epoch': 1696532400, 'owner': 'foo', 'categories': [{'category': 'delete_obj', 'bytes_sent': 0, 'bytes_received': 0, 'ops': 1, 'successful_ops': 1}]}]}, {'user': 'fud', 'buckets': [{'bucket': 'myfoo', 'time': '2023-10-05T19:00:00.000000Z', 'epoch': 1696532400, 'owner': 'fud', 'categories': []}, {'bucket': 'myfoo7', 'time': '2023-10-05T19:00:00.000000Z', 'epoch': 1696532400, 'owner': 'fud', 'categories': []}]}], 'summary': [{'user': 'anonymous', 'categories': [], 'total': {'bytes_sent': 0, 'bytes_received': 0, 'ops': 0, 'successful_ops': 0}}, {'user': 'bar', 'categories': [], 'total': {'bytes_sent': 0, 'bytes_received': 0, 'ops': 0, 'successful_ops': 0}}, {'user': 'bud', 'categories': [], 'total': {'bytes_sent': 0, 'bytes_received': 0, 'ops': 0, 'successful_ops': 0}}, {'user': 'foo', 'categories': [{'category': 'delete_obj', 'bytes_sent': 0, 'bytes_received': 0, 'ops': 3, 'successful_ops': 3}], 'total': {'bytes_sent': 0, 'bytes_received': 0, 'ops': 3, 'successful_ops': 3}}, {'user': 'fud', 'categories': [], 'total': {'bytes_sent': 0, 'bytes_received': 0, 'ops': 0, 'successful_ops': 0}}]} 2023-10-05T19:31:24.289 INFO:tasks.radosgw_admin_rest:radosgw-admin-rest: ['usage', 'show'] None 2023-10-05T19:31:24.321 INFO:tasks.radosgw_admin_rest: json result: {'entries': [{'user': 'anonymous', 'buckets': [{'bucket': '', 'time': '2023-10-05T19:00:00.000000Z', 'epoch': 1696532400, 'owner': 'anonymous', 'categories': [{'category': 'list_buckets', 'bytes_sent': 231, 'bytes_received': 0, 'ops': 1, 'successful_ops': 1}]}]}, {'user': 'bar', 'buckets': [{'bucket': 'myfoo6', 'time': '2023-10-05T19:00:00.000000Z', 'epoch': 1696532400, 'owner': 'bar', 'categories': [{'category': 'create_bucket', 'bytes_sent': 0, 'bytes_received': 0, 'ops': 1, 'successful_ops': 1}, {'category': 'put_obj', 'bytes_sent': 0, 'bytes_received': 8, 'ops': 1, 'successful_ops': 1}]}]}, {'user': 'bud', 'buckets': [{'bucket': 'myfoo6', 'time': '2023-10-05T19:00:00.000000Z', 'epoch': 1696532400, 'owner': 'bud', 'categories': [{'category': 'get_obj', 'bytes_sent': 16, 'bytes_received': 0, 'ops': 2, 'successful_ops': 2}, {'category': 'stat_bucket', 'bytes_sent': 0, 'bytes_received': 0, 'ops': 2, 'successful_ops': 2}]}]}, {'user': 'foo', 'buckets': [{'bucket': 'myfoo', 'time': '2023-10-05T19:00:00.000000Z', 'epoch': 1696532400, 'owner': 'foo', 'categories': [{'category': 'create_bucket', 'bytes_sent': 0, 'bytes_received': 0, 'ops': 1, 'successful_ops': 1}, {'category': 'delete_obj', 'bytes_sent': 0, 'bytes_received': 0, 'ops': 2, 'successful_ops': 2}, {'category': 'put_obj', 'bytes_sent': 0, 'bytes_received': 10, 'ops': 3, 'successful_ops': 3}]}, {'bucket': 'useless-bucket', 'time': '2023-10-05T19:00:00.000000Z', 'epoch': 1696532400, 'owner': 'foo', 'categories': [{'category': 'create_bucket', 'bytes_sent': 0, 'bytes_received': 0, 'ops': 1, 'successful_ops': 1}, {'category': 'delete_bucket', 'bytes_sent': 0, 'bytes_received': 0, 'ops': 1, 'successful_ops': 1}, {'category': 'delete_obj', 'bytes_sent': 0, 'bytes_received': 0, 'ops': 1, 'successful_ops': 1}, {'category': 'put_obj', 'bytes_sent': 0, 'bytes_received': 14, 'ops': 1, 'successful_ops': 1}]}]}, {'user': 'fud', 'buckets': [{'bucket': 'myfoo', 'time': '2023-10-05T19:00:00.000000Z', 'epoch': 1696532400, 'owner': 'fud', 'categories': [{'category': 'put_obj', 'bytes_sent': 215, 'bytes_received': 0, 'ops': 1, 'successful_ops': 0}]}, {'bucket': 'myfoo7', 'time': '2023-10-05T19:00:00.000000Z', 'epoch': 1696532400, 'owner': 'fud', 'categories': [{'category': 'create_bucket', 'bytes_sent': 0, 'bytes_received': 0, 'ops': 1, 'successful_ops': 1}, {'category': 'get_obj', 'bytes_sent': 8, 'bytes_received': 0, 'ops': 1, 'successful_ops': 1}, {'category': 'put_obj', 'bytes_sent': 0, 'bytes_received': 8, 'ops': 1, 'successful_ops': 1}, {'category': 'stat_bucket', 'bytes_sent': 0, 'bytes_received': 0, 'ops': 1, 'successful_ops': 1}]}]}], 'summary': [{'user': 'anonymous', 'categories': [{'category': 'list_buckets', 'bytes_sent': 231, 'bytes_received': 0, 'ops': 1, 'successful_ops': 1}], 'total': {'bytes_sent': 231, 'bytes_received': 0, 'ops': 1, 'successful_ops': 1}}, {'user': 'bar', 'categories': [{'category': 'create_bucket', 'bytes_sent': 0, 'bytes_received': 0, 'ops': 1, 'successful_ops': 1}, {'category': 'put_obj', 'bytes_sent': 0, 'bytes_received': 8, 'ops': 1, 'successful_ops': 1}], 'total': {'bytes_sent': 0, 'bytes_received': 8, 'ops': 2, 'successful_ops': 2}}, {'user': 'bud', 'categories': [{'category': 'get_obj', 'bytes_sent': 16, 'bytes_received': 0, 'ops': 2, 'successful_ops': 2}, {'category': 'stat_bucket', 'bytes_sent': 0, 'bytes_received': 0, 'ops': 2, 'successful_ops': 2}], 'total': {'bytes_sent': 16, 'bytes_received': 0, 'ops': 4, 'successful_ops': 4}}, {'user': 'foo', 'categories': [{'category': 'create_bucket', 'bytes_sent': 0, 'bytes_received': 0, 'ops': 2, 'successful_ops': 2}, {'category': 'delete_bucket', 'bytes_sent': 0, 'bytes_received': 0, 'ops': 1, 'successful_ops': 1}, {'category': 'delete_obj', 'bytes_sent': 0, 'bytes_received': 0, 'ops': 3, 'successful_ops': 3}, {'category': 'put_obj', 'bytes_sent': 0, 'bytes_received': 24, 'ops': 4, 'successful_ops': 4}], 'total': {'bytes_sent': 0, 'bytes_received': 24, 'ops': 10, 'successful_ops': 10}}, {'user': 'fud', 'categories': [{'category': 'create_bucket', 'bytes_sent': 0, 'bytes_received': 0, 'ops': 1, 'successful_ops': 1}, {'category': 'get_obj', 'bytes_sent': 8, 'bytes_received': 0, 'ops': 1, 'successful_ops': 1}, {'category': 'put_obj', 'bytes_sent': 215, 'bytes_received': 8, 'ops': 2, 'successful_ops': 1}, {'category': 'stat_bucket', 'bytes_sent': 0, 'bytes_received': 0, 'ops': 1, 'successful_ops': 1}], 'total': {'bytes_sent': 223, 'bytes_received': 8, 'ops': 5, 'successful_ops': 4}}]} 2023-10-05T19:31:24.321 INFO:tasks.radosgw_admin_rest:radosgw-admin-rest: ['usage', 'show'] {'uid': 'foo'} 2023-10-05T19:31:24.327 INFO:tasks.radosgw_admin_rest: json result: {'entries': [{'user': 'foo', 'buckets': [{'bucket': 'myfoo', 'time': '2023-10-05T19:00:00.000000Z', 'epoch': 1696532400, 'owner': 'foo', 'categories': [{'category': 'create_bucket', 'bytes_sent': 0, 'bytes_received': 0, 'ops': 1, 'successful_ops': 1}, {'category': 'delete_obj', 'bytes_sent': 0, 'bytes_received': 0, 'ops': 2, 'successful_ops': 2}, {'category': 'put_obj', 'bytes_sent': 0, 'bytes_received': 10, 'ops': 3, 'successful_ops': 3}]}, {'bucket': 'useless-bucket', 'time': '2023-10-05T19:00:00.000000Z', 'epoch': 1696532400, 'owner': 'foo', 'categories': [{'category': 'create_bucket', 'bytes_sent': 0, 'bytes_received': 0, 'ops': 1, 'successful_ops': 1}, {'category': 'delete_bucket', 'bytes_sent': 0, 'bytes_received': 0, 'ops': 1, 'successful_ops': 1}, {'category': 'delete_obj', 'bytes_sent': 0, 'bytes_received': 0, 'ops': 1, 'successful_ops': 1}, {'category': 'put_obj', 'bytes_sent': 0, 'bytes_received': 14, 'ops': 1, 'successful_ops': 1}]}]}], 'summary': [{'user': 'foo', 'categories': [{'category': 'create_bucket', 'bytes_sent': 0, 'bytes_received': 0, 'ops': 2, 'successful_ops': 2}, {'category': 'delete_bucket', 'bytes_sent': 0, 'bytes_received': 0, 'ops': 1, 'successful_ops': 1}, {'category': 'delete_obj', 'bytes_sent': 0, 'bytes_received': 0, 'ops': 3, 'successful_ops': 3}, {'category': 'put_obj', 'bytes_sent': 0, 'bytes_received': 24, 'ops': 4, 'successful_ops': 4}], 'total': {'bytes_sent': 0, 'bytes_received': 24, 'ops': 10, 'successful_ops': 10}}]} 2023-10-05T19:31:24.327 INFO:tasks.radosgw_admin_rest:radosgw-admin-rest: ['usage', 'show'] {'uid': 'foo', 'categories': 'create_bucket'} 2023-10-05T19:31:24.333 INFO:tasks.radosgw_admin_rest: json result: {'entries': [{'user': 'foo', 'buckets': [{'bucket': 'myfoo', 'time': '2023-10-05T19:00:00.000000Z', 'epoch': 1696532400, 'owner': 'foo', 'categories': [{'category': 'create_bucket', 'bytes_sent': 0, 'bytes_received': 0, 'ops': 1, 'successful_ops': 1}]}, {'bucket': 'useless-bucket', 'time': '2023-10-05T19:00:00.000000Z', 'epoch': 1696532400, 'owner': 'foo', 'categories': [{'category': 'create_bucket', 'bytes_sent': 0, 'bytes_received': 0, 'ops': 1, 'successful_ops': 1}]}]}], 'summary': [{'user': 'foo', 'categories': [{'category': 'create_bucket', 'bytes_sent': 0, 'bytes_received': 0, 'ops': 2, 'successful_ops': 2}], 'total': {'bytes_sent': 0, 'bytes_received': 0, 'ops': 2, 'successful_ops': 2}}]} 2023-10-05T19:31:24.334 INFO:tasks.radosgw_admin_rest:radosgw-admin-rest: ['usage', 'show'] {'uid': 'foo', 'categories': 'put_obj'} 2023-10-05T19:31:24.339 INFO:tasks.radosgw_admin_rest: json result: {'entries': [{'user': 'foo', 'buckets': [{'bucket': 'myfoo', 'time': '2023-10-05T19:00:00.000000Z', 'epoch': 1696532400, 'owner': 'foo', 'categories': [{'category': 'put_obj', 'bytes_sent': 0, 'bytes_received': 10, 'ops': 3, 'successful_ops': 3}]}, {'bucket': 'useless-bucket', 'time': '2023-10-05T19:00:00.000000Z', 'epoch': 1696532400, 'owner': 'foo', 'categories': [{'category': 'put_obj', 'bytes_sent': 0, 'bytes_received': 14, 'ops': 1, 'successful_ops': 1}]}]}], 'summary': [{'user': 'foo', 'categories': [{'category': 'put_obj', 'bytes_sent': 0, 'bytes_received': 24, 'ops': 4, 'successful_ops': 4}], 'total': {'bytes_sent': 0, 'bytes_received': 24, 'ops': 4, 'successful_ops': 4}}]} 2023-10-05T19:31:24.339 INFO:tasks.radosgw_admin_rest:radosgw-admin-rest: ['usage', 'show'] {'uid': 'foo', 'categories': 'delete_obj'} 2023-10-05T19:31:24.345 INFO:tasks.radosgw_admin_rest: json result: {'entries': [{'user': 'foo', 'buckets': [{'bucket': 'myfoo', 'time': '2023-10-05T19:00:00.000000Z', 'epoch': 1696532400, 'owner': 'foo', 'categories': [{'category': 'delete_obj', 'bytes_sent': 0, 'bytes_received': 0, 'ops': 2, 'successful_ops': 2}]}, {'bucket': 'useless-bucket', 'time': '2023-10-05T19:00:00.000000Z', 'epoch': 1696532400, 'owner': 'foo', 'categories': [{'category': 'delete_obj', 'bytes_sent': 0, 'bytes_received': 0, 'ops': 1, 'successful_ops': 1}]}]}], 'summary': [{'user': 'foo', 'categories': [{'category': 'delete_obj', 'bytes_sent': 0, 'bytes_received': 0, 'ops': 3, 'successful_ops': 3}], 'total': {'bytes_sent': 0, 'bytes_received': 0, 'ops': 3, 'successful_ops': 3}}]} 2023-10-05T19:31:24.345 INFO:tasks.radosgw_admin_rest:radosgw-admin-rest: ['usage', 'show'] {'uid': 'foo', 'categories': 'delete_bucket'} 2023-10-05T19:31:24.351 INFO:tasks.radosgw_admin_rest: json result: {'entries': [{'user': 'foo', 'buckets': [{'bucket': 'myfoo', 'time': '2023-10-05T19:00:00.000000Z', 'epoch': 1696532400, 'owner': 'foo', 'categories': []}, {'bucket': 'useless-bucket', 'time': '2023-10-05T19:00:00.000000Z', 'epoch': 1696532400, 'owner': 'foo', 'categories': [{'category': 'delete_bucket', 'bytes_sent': 0, 'bytes_received': 0, 'ops': 1, 'successful_ops': 1}]}]}], 'summary': [{'user': 'foo', 'categories': [{'category': 'delete_bucket', 'bytes_sent': 0, 'bytes_received': 0, 'ops': 1, 'successful_ops': 1}], 'total': {'bytes_sent': 0, 'bytes_received': 0, 'ops': 1, 'successful_ops': 1}}]} 2023-10-05T19:31:24.352 INFO:tasks.radosgw_admin_rest:radosgw-admin-rest: ['usage', 'trim'] {'uid': 'foo'} 2023-10-05T19:31:24.363 INFO:tasks.radosgw_admin_rest: empty result 2023-10-05T19:31:24.364 INFO:tasks.radosgw_admin_rest:radosgw-admin-rest: ['usage', 'show'] {'uid': 'foo'} 2023-10-05T19:31:24.369 INFO:tasks.radosgw_admin_rest: json result: {'entries': [], 'summary': []} 2023-10-05T19:31:24.369 INFO:tasks.radosgw_admin_rest:radosgw-admin-rest: ['user', 'modify'] {'uid': 'foo', 'suspended': True} 2023-10-05T19:31:24.386 INFO:tasks.radosgw_admin_rest: json result: {'tenant': '', 'user_id': 'foo', 'display_name': 'Foo', 'email': 'foo@foo.com', 'suspended': 1, 'max_buckets': 4, 'subusers': [], 'keys': [{'user': 'foo', 'access_key': '9te6NH5mcdcq0Tc5i8i1', 'secret_key': 'Ny4IOauQoL18Gp2zM7lC1vLmoawgqcYP/YGcWfXu'}], 'swift_keys': [], 'caps': [], 'op_mask': 'read, write, delete', 'system': 'false', 'admin': 'false', 'default_placement': '', 'default_storage_class': '', 'placement_tags': [], 'bucket_quota': {'enabled': False, 'check_on_raw': False, 'max_size': -1, 'max_size_kb': 0, 'max_objects': -1}, 'user_quota': {'enabled': False, 'check_on_raw': False, 'max_size': -1, 'max_size_kb': 0, 'max_objects': -1}, 'temp_url_keys': [], 'type': 'rgw', 'mfa_ids': []} 2023-10-05T19:31:24.390 INFO:tasks.radosgw_admin_rest:radosgw-admin-rest: ['user', 'modify'] {'uid': 'foo', 'suspended': 'false'} 2023-10-05T19:31:24.406 INFO:tasks.radosgw_admin_rest: json result: {'tenant': '', 'user_id': 'foo', 'display_name': 'Foo', 'email': 'foo@foo.com', 'suspended': 0, 'max_buckets': 4, 'subusers': [], 'keys': [{'user': 'foo', 'access_key': '9te6NH5mcdcq0Tc5i8i1', 'secret_key': 'Ny4IOauQoL18Gp2zM7lC1vLmoawgqcYP/YGcWfXu'}], 'swift_keys': [], 'caps': [], 'op_mask': 'read, write, delete', 'system': 'false', 'admin': 'false', 'default_placement': '', 'default_storage_class': '', 'placement_tags': [], 'bucket_quota': {'enabled': False, 'check_on_raw': False, 'max_size': -1, 'max_size_kb': 0, 'max_objects': -1}, 'user_quota': {'enabled': False, 'check_on_raw': False, 'max_size': -1, 'max_size_kb': 0, 'max_objects': -1}, 'temp_url_keys': [], 'type': 'rgw', 'mfa_ids': []} 2023-10-05T19:31:24.668 INFO:tasks.radosgw_admin_rest:radosgw-admin-rest: ['user', 'rm'] {'uid': 'foo'} 2023-10-05T19:31:24.673 INFO:tasks.radosgw_admin_rest: json result: {'Code': 'BucketAlreadyExists', 'RequestId': 'tx000000bfd97713bcce79c-00651f0f0c-107d-default', 'HostId': '107d-default-default'} 2023-10-05T19:31:24.897 INFO:tasks.radosgw_admin_rest:radosgw-admin-rest: ['policy', 'show'] {'bucket': 'myfoo', 'object': b'bb3aec0fdcdbc2974890f805c585d432'} 2023-10-05T19:31:24.903 INFO:tasks.radosgw_admin_rest: json result: {'acl': {'acl_user_map': [{'user': 'foo', 'acl': 15}], 'acl_group_map': [], 'grant_map': [{'id': 'foo', 'grant': {'type': {'type': 0}, 'id': 'foo', 'email': '', 'permission': {'flags': 15}, 'name': 'Foo', 'group': 0, 'url_spec': ''}}]}, 'owner': {'id': 'foo', 'display_name': 'Foo'}} 2023-10-05T19:31:24.912 INFO:tasks.radosgw_admin_rest:radosgw-admin-rest: ['policy', 'show'] {'bucket': 'myfoo', 'object': b'bb3aec0fdcdbc2974890f805c585d432'} 2023-10-05T19:31:24.918 INFO:tasks.radosgw_admin_rest: json result: {'acl': {'acl_user_map': [{'user': 'foo', 'acl': 15}], 'acl_group_map': [{'group': 1, 'acl': 1}], 'grant_map': [{'id': '', 'grant': {'type': {'type': 2}, 'id': '', 'email': '', 'permission': {'flags': 1}, 'name': '', 'group': 1, 'url_spec': ''}}, {'id': 'foo', 'grant': {'type': {'type': 0}, 'id': 'foo', 'email': '', 'permission': {'flags': 15}, 'name': 'Foo', 'group': 0, 'url_spec': ''}}]}, 'owner': {'id': 'foo', 'display_name': 'Foo'}} 2023-10-05T19:31:25.031 INFO:tasks.radosgw_admin_rest:radosgw-admin-rest: ['bucket', 'rm'] {'bucket': 'myfoo', 'purge-objects': True} 2023-10-05T19:31:25.128 INFO:tasks.radosgw_admin_rest: empty result 2023-10-05T19:31:25.128 INFO:tasks.radosgw_admin_rest:radosgw-admin-rest: ['caps', 'add'] {'uid': 'foo', 'user-caps': 'usage=read'} 2023-10-05T19:31:25.136 INFO:tasks.radosgw_admin_rest: json result: [{'type': 'usage', 'perm': 'read'}] 2023-10-05T19:31:25.136 INFO:tasks.radosgw_admin_rest:radosgw-admin-rest: ['caps', 'rm'] {'uid': 'foo', 'user-caps': 'usage=read'} 2023-10-05T19:31:25.145 INFO:tasks.radosgw_admin_rest: json result: [] 2023-10-05T19:31:25.166 INFO:tasks.radosgw_admin_rest:radosgw-admin-rest: ['user', 'rm'] {'uid': 'foo'} 2023-10-05T19:31:25.171 INFO:tasks.radosgw_admin_rest: json result: {'Code': 'BucketAlreadyExists', 'RequestId': 'tx00000463e4936ad795be6-00651f0f0d-107d-default', 'HostId': '107d-default-default'} 2023-10-05T19:31:25.188 INFO:tasks.radosgw_admin_rest:radosgw-admin-rest: ['user', 'rm'] {'uid': 'foo', 'purge-data': True} 2023-10-05T19:31:25.265 INFO:tasks.radosgw_admin_rest: empty result 2023-10-05T19:31:25.265 INFO:tasks.radosgw_admin_rest:radosgw-admin-rest: ['user', 'info'] {'uid': 'foo'} 2023-10-05T19:31:25.270 INFO:tasks.radosgw_admin_rest: json result: {'Code': 'NoSuchUser', 'RequestId': 'tx00000a5fa7c1727ea5b16-00651f0f0d-107d-default', 'HostId': '107d-default-default'} 2023-10-05T19:31:25.271 INFO:tasks.radosgw_admin_rest:radosgw-admin-rest: ['info', ''] None 2023-10-05T19:31:25.274 INFO:tasks.radosgw_admin_rest: json result: {'info': {'storage_backends': [{'name': 'rados', 'cluster_id': '8c4bd0d1-cbcb-4713-96b7-dcb14bcaf584'}]}} 2023-10-05T19:31:25.275 INFO:tasks.radosgw_admin_rest:radosgw-admin-rest: ['user', 'create'] {'uid': 'ratelimit_user', 'display-name': 'Foo', 'email': 'foo@foo.com', 'access-key': '9te6NH5mcdcq0Tc5i8i1', 'secret-key': 'Ny4IOauQoL18Gp2zM7lC1vLmoawgqcYP/YGcWfXu', 'max-buckets': '1000'} 2023-10-05T19:31:25.297 INFO:tasks.radosgw_admin_rest: json result: {'tenant': '', 'user_id': 'ratelimit_user', 'display_name': 'Foo', 'email': 'foo@foo.com', 'suspended': 0, 'max_buckets': 1000, 'subusers': [], 'keys': [{'user': 'ratelimit_user', 'access_key': '9te6NH5mcdcq0Tc5i8i1', 'secret_key': 'Ny4IOauQoL18Gp2zM7lC1vLmoawgqcYP/YGcWfXu'}], 'swift_keys': [], 'caps': [], 'op_mask': 'read, write, delete', 'system': 'false', 'admin': 'false', 'default_placement': '', 'default_storage_class': '', 'placement_tags': [], 'bucket_quota': {'enabled': False, 'check_on_raw': False, 'max_size': -1, 'max_size_kb': 0, 'max_objects': -1}, 'user_quota': {'enabled': False, 'check_on_raw': False, 'max_size': -1, 'max_size_kb': 0, 'max_objects': -1}, 'temp_url_keys': [], 'type': 'rgw', 'mfa_ids': []} 2023-10-05T19:31:25.297 INFO:tasks.radosgw_admin_rest:radosgw-admin-rest: ['ratelimit', 'info'] {'ratelimit-scope': 'user', 'uid': 'ratelimit_user'} 2023-10-05T19:31:25.301 INFO:tasks.radosgw_admin_rest: json result: {'user_ratelimit': {'max_read_ops': 0, 'max_write_ops': 0, 'max_read_bytes': 0, 'max_write_bytes': 0, 'enabled': False}} 2023-10-05T19:31:25.301 INFO:tasks.radosgw_admin_rest:radosgw-admin-rest: ['ratelimit', 'info'] {'ratelimit-scope': 'user', 'uid': 'ratelimit_userstring'} 2023-10-05T19:31:25.307 INFO:tasks.radosgw_admin_rest: json result: {'Code': 'NoSuchKey', 'RequestId': 'tx00000b10e72001285b6cd-00651f0f0d-107d-default', 'HostId': '107d-default-default'} 2023-10-05T19:31:25.307 INFO:tasks.radosgw_admin_rest:radosgw-admin-rest: ['ratelimit', 'info'] {'ratelimit-scope': 'user'} 2023-10-05T19:31:25.311 INFO:tasks.radosgw_admin_rest: json result: {'Code': 'InvalidArgument', 'RequestId': 'tx000009ffc0672b62925e3-00651f0f0d-107d-default', 'HostId': '107d-default-default'} 2023-10-05T19:31:25.332 INFO:tasks.radosgw_admin_rest:radosgw-admin-rest: ['ratelimit', 'info'] {'ratelimit-scope': 'bucket', 'bucket': 'ratelimitbucket'} 2023-10-05T19:31:25.336 INFO:tasks.radosgw_admin_rest: json result: {'bucket_ratelimit': {'max_read_ops': 0, 'max_write_ops': 0, 'max_read_bytes': 0, 'max_write_bytes': 0, 'enabled': False}} 2023-10-05T19:31:25.336 INFO:tasks.radosgw_admin_rest:radosgw-admin-rest: ['ratelimit', 'info'] {'ratelimit-scope': 'bucket', 'bucket': 'ratelimitbucketstring'} 2023-10-05T19:31:25.340 INFO:tasks.radosgw_admin_rest: json result: {'Code': 'NoSuchKey', 'RequestId': 'tx00000328404b5a892f413-00651f0f0d-107d-default', 'HostId': '107d-default-default'} 2023-10-05T19:31:25.340 INFO:tasks.radosgw_admin_rest:radosgw-admin-rest: ['ratelimit', 'info'] {'ratelimit-scope': 'bucket'} 2023-10-05T19:31:25.344 INFO:tasks.radosgw_admin_rest: json result: {'Code': 'InvalidArgument', 'RequestId': 'tx00000801d2204f0d671c0-00651f0f0d-107d-default', 'HostId': '107d-default-default'} 2023-10-05T19:31:25.345 INFO:tasks.radosgw_admin_rest:radosgw-admin-rest: ['ratelimit', 'info'] {'global': 'true'} 2023-10-05T19:31:25.349 INFO:tasks.radosgw_admin_rest: json result: {'bucket_ratelimit': {'max_read_ops': 0, 'max_write_ops': 0, 'max_read_bytes': 0, 'max_write_bytes': 0, 'enabled': False}, 'user_ratelimit': {'max_read_ops': 0, 'max_write_ops': 0, 'max_read_bytes': 0, 'max_write_bytes': 0, 'enabled': False}, 'anonymous_ratelimit': {'max_read_ops': 0, 'max_write_ops': 0, 'max_read_bytes': 0, 'max_write_bytes': 0, 'enabled': False}} 2023-10-05T19:31:25.350 INFO:tasks.radosgw_admin_rest:radosgw-admin-rest: ['ratelimit', 'modify'] {'ratelimit-scope': 'user', 'uid': 'ratelimit_userstring', 'enabled': 'true'} 2023-10-05T19:31:25.353 INFO:tasks.radosgw_admin_rest: json result: {'Code': 'NoSuchKey', 'RequestId': 'tx0000005a01054089ffbb3-00651f0f0d-107d-default', 'HostId': '107d-default-default'} 2023-10-05T19:31:25.354 INFO:tasks.radosgw_admin_rest:radosgw-admin-rest: ['ratelimit', 'modify'] {'ratelimit-scope': 'user'} 2023-10-05T19:31:25.358 INFO:tasks.radosgw_admin_rest: json result: {'Code': 'InvalidArgument', 'RequestId': 'tx00000218c539b01592183-00651f0f0d-107d-default', 'HostId': '107d-default-default'} 2023-10-05T19:31:25.358 INFO:tasks.radosgw_admin_rest:radosgw-admin-rest: ['ratelimit', 'modify'] {'ratelimit-scope': 'bucket', 'bucket': 'ratelimitbucketstring', 'enabled': 'true'} 2023-10-05T19:31:25.362 INFO:tasks.radosgw_admin_rest: json result: {'Code': 'NoSuchKey', 'RequestId': 'tx00000b04fa58638bc2cc5-00651f0f0d-107d-default', 'HostId': '107d-default-default'} 2023-10-05T19:31:25.362 INFO:tasks.radosgw_admin_rest:radosgw-admin-rest: ['ratelimit', 'modify'] {'ratelimit-scope': 'bucket', 'enabled': 'true'} 2023-10-05T19:31:25.366 INFO:tasks.radosgw_admin_rest: json result: {'Code': 'InvalidArgument', 'RequestId': 'tx00000ccef4569adb26fb1-00651f0f0d-107d-default', 'HostId': '107d-default-default'} 2023-10-05T19:31:25.366 INFO:tasks.radosgw_admin_rest:radosgw-admin-rest: ['ratelimit', 'modify'] {'ratelimit-scope': 'user', 'uid': 'ratelimit_user', 'enabled': 'true', 'max-read-bytes': '2'} 2023-10-05T19:31:25.385 INFO:tasks.radosgw_admin_rest: empty result 2023-10-05T19:31:25.385 INFO:tasks.radosgw_admin_rest:radosgw-admin-rest: ['ratelimit', 'info'] {'ratelimit-scope': 'user', 'uid': 'ratelimit_user'} 2023-10-05T19:31:25.390 INFO:tasks.radosgw_admin_rest: json result: {'user_ratelimit': {'max_read_ops': 0, 'max_write_ops': 0, 'max_read_bytes': 2, 'max_write_bytes': 0, 'enabled': True}} 2023-10-05T19:31:25.390 INFO:tasks.radosgw_admin_rest:radosgw-admin-rest: ['ratelimit', 'modify'] {'ratelimit-scope': 'bucket', 'bucket': 'ratelimitbucket', 'enabled': 'true', 'max-write-bytes': '2'} 2023-10-05T19:31:25.401 INFO:tasks.radosgw_admin_rest: empty result 2023-10-05T19:31:25.401 INFO:tasks.radosgw_admin_rest:radosgw-admin-rest: ['ratelimit', 'info'] {'ratelimit-scope': 'bucket', 'bucket': 'ratelimitbucket'} 2023-10-05T19:31:25.406 INFO:tasks.radosgw_admin_rest: json result: {'bucket_ratelimit': {'max_read_ops': 0, 'max_write_ops': 0, 'max_read_bytes': 0, 'max_write_bytes': 2, 'enabled': True}} 2023-10-05T19:31:25.406 INFO:tasks.radosgw_admin_rest:radosgw-admin-rest: ['ratelimit', 'modify'] {'ratelimit-scope': 'bucket', 'global': 'true', 'enabled': 'true'} 2023-10-05T19:31:25.416 INFO:tasks.radosgw_admin_rest: empty result 2023-10-05T19:31:25.420 DEBUG:teuthology.run_tasks:Unwinding manager rgw 2023-10-05T19:31:25.430 DEBUG:tasks.rgw.client.0:waiting for process to exit 2023-10-05T19:31:25.430 INFO:teuthology.orchestra.run:waiting for 300 2023-10-05T19:31:25.431 INFO:tasks.rgw.client.0.smithi167.stdout:2023-10-05T19:31:25.425+0000 7f1226e0d700 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term radosgw --rgw-frontends beast port=80 -n client.0 --cluster ceph -k /etc/ceph/ceph.client.0.keyring --log-file /var/log/ceph/rgw.ceph.client.0.log --rgw_ops_log_socket_path /home/ubuntu/cephtest/rgw.opslog.ceph.client.0.sock --foreground (PID: 34536) UID: 0 2023-10-05T19:31:25.431 INFO:tasks.rgw.client.0.smithi167.stdout:2023-10-05T19:31:25.425+0000 7f122b2ede40 -1 shutting down 2023-10-05T19:31:29.279 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.rgw.client.0 is failed for ~0s 2023-10-05T19:31:31.531 INFO:tasks.rgw.client.0:Stopped 2023-10-05T19:31:31.532 DEBUG:teuthology.orchestra.run.smithi167:> rm -f /home/ubuntu/cephtest/rgw.opslog.ceph.client.0.sock 2023-10-05T19:31:31.539 DEBUG:teuthology.orchestra.run.smithi167:> sudo rm -f /etc/ceph/vault-root-token 2023-10-05T19:31:31.555 DEBUG:teuthology.run_tasks:Unwinding manager ceph 2023-10-05T19:31:31.565 INFO:tasks.ceph.ceph_manager.ceph:Canceling any pending splits or merges... 2023-10-05T19:31:31.565 DEBUG:teuthology.orchestra.run.smithi118:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-05T19:31:31.679 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-05T19:31:31.670+0000 7fb81df21700 1 Processor -- start 2023-10-05T19:31:31.680 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-05T19:31:31.670+0000 7fb81df21700 1 -- start start 2023-10-05T19:31:31.680 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-05T19:31:31.670+0000 7fb81df21700 1 --2- >> v2:172.21.15.118:3300/0 conn(0x7fb81812dda0 0x7fb81812e170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-05T19:31:31.680 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-05T19:31:31.670+0000 7fb81df21700 1 -- --> v1:172.21.15.118:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb81806b850 con 0x7fb81806cc50 2023-10-05T19:31:31.680 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-05T19:31:31.670+0000 7fb81df21700 1 -- --> v2:172.21.15.118:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb81806b100 con 0x7fb81812dda0 2023-10-05T19:31:31.680 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-05T19:31:31.670+0000 7fb81cf1f700 1 --2- >> v2:172.21.15.118:3300/0 conn(0x7fb81812dda0 0x7fb81812e170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-05T19:31:31.681 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-05T19:31:31.670+0000 7fb81cf1f700 1 --2- >> v2:172.21.15.118:3300/0 conn(0x7fb81812dda0 0x7fb81812e170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.118:3300/0 says I am v2:172.21.15.118:55290/0 (socket says 172.21.15.118:55290) 2023-10-05T19:31:31.681 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-05T19:31:31.670+0000 7fb817fff700 1 --1- >> v1:172.21.15.118:6789/0 conn(0x7fb81806cc50 0x7fb81806d020 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.118:6789/0 says I am v1:172.21.15.118:34956/0 (socket says 172.21.15.118:34956) 2023-10-05T19:31:31.681 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-05T19:31:31.670+0000 7fb81cf1f700 1 -- 172.21.15.118:0/1786422523 learned_addr learned my addr 172.21.15.118:0/1786422523 (peer_addr_for_me v2:172.21.15.118:0/0) 2023-10-05T19:31:31.681 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-05T19:31:31.674+0000 7fb81cf1f700 1 -- 172.21.15.118:0/1786422523 >> v1:172.21.15.118:6789/0 conn(0x7fb81806cc50 legacy=0x7fb81806d020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-05T19:31:31.681 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-05T19:31:31.674+0000 7fb8177fe700 1 -- 172.21.15.118:0/1786422523 <== mon.0 v1:172.21.15.118:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1750675473 0 0) 0x7fb81806b850 con 0x7fb81806cc50 2023-10-05T19:31:31.682 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-05T19:31:31.674+0000 7fb81cf1f700 1 -- 172.21.15.118:0/1786422523 --> v2:172.21.15.118:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb81812e6b0 con 0x7fb81812dda0 2023-10-05T19:31:31.682 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-05T19:31:31.674+0000 7fb81cf1f700 1 --2- 172.21.15.118:0/1786422523 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb81812dda0 0x7fb81812e170 secure :-1 s=READY pgs=711 cs=0 l=1 rev1=1 crypto rx=0x7fb808009930 tx=0x7fb808009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c649bd9d8d5bc3e9 server_cookie=0 in_seq=0 out_seq=0 2023-10-05T19:31:31.682 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-05T19:31:31.674+0000 7fb8177fe700 1 -- 172.21.15.118:0/1786422523 <== mon.0 v2:172.21.15.118:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fb80800b860 con 0x7fb81812dda0 2023-10-05T19:31:31.682 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-05T19:31:31.674+0000 7fb8177fe700 1 -- 172.21.15.118:0/1786422523 <== mon.0 v2:172.21.15.118:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb80800b9c0 con 0x7fb81812dda0 2023-10-05T19:31:31.682 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-05T19:31:31.674+0000 7fb8177fe700 1 -- 172.21.15.118:0/1786422523 <== mon.0 v2:172.21.15.118:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fb80800bb50 con 0x7fb81812dda0 2023-10-05T19:31:31.683 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-05T19:31:31.674+0000 7fb81df21700 1 -- 172.21.15.118:0/1786422523 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb81812dda0 msgr2=0x7fb81812e170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-05T19:31:31.683 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-05T19:31:31.674+0000 7fb81df21700 1 --2- 172.21.15.118:0/1786422523 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb81812dda0 0x7fb81812e170 secure :-1 s=READY pgs=711 cs=0 l=1 rev1=1 crypto rx=0x7fb808009930 tx=0x7fb808009250 comp rx=0 tx=0).stop 2023-10-05T19:31:31.683 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-05T19:31:31.674+0000 7fb81df21700 1 -- 172.21.15.118:0/1786422523 shutdown_connections 2023-10-05T19:31:31.683 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-05T19:31:31.674+0000 7fb81df21700 1 --2- 172.21.15.118:0/1786422523 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb81812dda0 0x7fb81812e170 unknown :-1 s=CLOSED pgs=711 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-05T19:31:31.683 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-05T19:31:31.674+0000 7fb81df21700 1 -- 172.21.15.118:0/1786422523 >> 172.21.15.118:0/1786422523 conn(0x7fb81809a530 msgr2=0x7fb81809a930 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-05T19:31:31.683 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-05T19:31:31.674+0000 7fb81df21700 1 -- 172.21.15.118:0/1786422523 shutdown_connections 2023-10-05T19:31:31.684 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-05T19:31:31.674+0000 7fb81df21700 1 -- 172.21.15.118:0/1786422523 wait complete. 2023-10-05T19:31:31.684 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-05T19:31:31.674+0000 7fb81df21700 1 Processor -- start 2023-10-05T19:31:31.684 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-05T19:31:31.674+0000 7fb81df21700 1 -- start start 2023-10-05T19:31:31.688 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-05T19:31:31.674+0000 7fb81df21700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb81806cc50 0x7fb81813cbb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-05T19:31:31.688 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-05T19:31:31.674+0000 7fb81df21700 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb81806ee50 con 0x7fb81806cc50 2023-10-05T19:31:31.688 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-05T19:31:31.674+0000 7fb81cf1f700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb81806cc50 0x7fb81813cbb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-05T19:31:31.688 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-05T19:31:31.674+0000 7fb81cf1f700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb81806cc50 0x7fb81813cbb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.118:3300/0 says I am v2:172.21.15.118:55302/0 (socket says 172.21.15.118:55302) 2023-10-05T19:31:31.688 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-05T19:31:31.674+0000 7fb81cf1f700 1 -- 172.21.15.118:0/1050988487 learned_addr learned my addr 172.21.15.118:0/1050988487 (peer_addr_for_me v2:172.21.15.118:0/0) 2023-10-05T19:31:31.689 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-05T19:31:31.678+0000 7fb81cf1f700 1 -- 172.21.15.118:0/1050988487 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb8080094f0 con 0x7fb81806cc50 2023-10-05T19:31:31.689 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-05T19:31:31.678+0000 7fb81cf1f700 1 --2- 172.21.15.118:0/1050988487 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb81806cc50 0x7fb81813cbb0 secure :-1 s=READY pgs=712 cs=0 l=1 rev1=1 crypto rx=0x7fb808009fd0 tx=0x7fb808016000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-05T19:31:31.689 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-05T19:31:31.678+0000 7fb814ff9700 1 -- 172.21.15.118:0/1050988487 <== mon.0 v2:172.21.15.118:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fb80800bd00 con 0x7fb81806cc50 2023-10-05T19:31:31.689 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-05T19:31:31.678+0000 7fb814ff9700 1 -- 172.21.15.118:0/1050988487 <== mon.0 v2:172.21.15.118:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb808020070 con 0x7fb81806cc50 2023-10-05T19:31:31.689 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-05T19:31:31.678+0000 7fb81df21700 1 -- 172.21.15.118:0/1050988487 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb818123800 con 0x7fb81806cc50 2023-10-05T19:31:31.689 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-05T19:31:31.678+0000 7fb814ff9700 1 -- 172.21.15.118:0/1050988487 <== mon.0 v2:172.21.15.118:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fb80801b450 con 0x7fb81806cc50 2023-10-05T19:31:31.690 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-05T19:31:31.678+0000 7fb81df21700 1 -- 172.21.15.118:0/1050988487 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb818121d50 con 0x7fb81806cc50 2023-10-05T19:31:31.690 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-05T19:31:31.678+0000 7fb81df21700 1 -- 172.21.15.118:0/1050988487 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb81805ecb0 con 0x7fb81806cc50 2023-10-05T19:31:31.690 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-05T19:31:31.678+0000 7fb814ff9700 1 -- 172.21.15.118:0/1050988487 <== mon.0 v2:172.21.15.118:3300/0 4 ==== mgrmap(e 5) v1 ==== 40329+0+0 (secure 0 0 0) 0x7fb808022030 con 0x7fb81806cc50 2023-10-05T19:31:31.690 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-05T19:31:31.678+0000 7fb814ff9700 1 --2- 172.21.15.118:0/1050988487 >> [v2:172.21.15.167:6824/29889,v1:172.21.15.167:6825/29889] conn(0x7fb7f80323e0 0x7fb7f8034890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-05T19:31:31.690 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-05T19:31:31.678+0000 7fb814ff9700 1 -- 172.21.15.118:0/1050988487 <== mon.0 v2:172.21.15.118:3300/0 5 ==== osd_map(32..32 src has 1..32) v4 ==== 6260+0+0 (secure 0 0 0) 0x7fb808046270 con 0x7fb81806cc50 2023-10-05T19:31:31.691 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-05T19:31:31.678+0000 7fb817fff700 1 --2- 172.21.15.118:0/1050988487 >> [v2:172.21.15.167:6824/29889,v1:172.21.15.167:6825/29889] conn(0x7fb7f80323e0 0x7fb7f8034890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-05T19:31:31.691 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-05T19:31:31.682+0000 7fb817fff700 1 --2- 172.21.15.118:0/1050988487 >> [v2:172.21.15.167:6824/29889,v1:172.21.15.167:6825/29889] conn(0x7fb7f80323e0 0x7fb7f8034890 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7fb8000011e0 tx=0x7fb8000079b0 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-05T19:31:31.691 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-05T19:31:31.686+0000 7fb814ff9700 1 -- 172.21.15.118:0/1050988487 <== mon.0 v2:172.21.15.118:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166807 (secure 0 0 0) 0x7fb80801b5b0 con 0x7fb81806cc50 2023-10-05T19:31:31.853 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-05T19:31:31.850+0000 7fb81df21700 1 -- 172.21.15.118:0/1050988487 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fb81812d950 con 0x7fb81806cc50 2023-10-05T19:31:31.855 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-05T19:31:31.850+0000 7fb814ff9700 1 -- 172.21.15.118:0/1050988487 <== mon.0 v2:172.21.15.118:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v32) v1 ==== 74+0+18788 (secure 0 0 0) 0x7fb8080193f0 con 0x7fb81806cc50 2023-10-05T19:31:31.855 INFO:teuthology.orchestra.run.smithi118.stdout: 2023-10-05T19:31:31.856 INFO:teuthology.orchestra.run.smithi118.stdout:{"epoch":32,"fsid":"8c4bd0d1-cbcb-4713-96b7-dcb14bcaf584","created":"2023-10-05T19:27:40.755769+0000","modified":"2023-10-05T19:28:46.775986+0000","last_up_change":"2023-10-05T19:27:50.196733+0000","last_in_change":"2023-10-05T19:27:41.811702+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":8,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-05T19:27:50.980612+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"12","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}},{"pool":2,"pool_name":"rbd","create_time":"2023-10-05T19:27:56.423831+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"15","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":3,"pool_name":"default.rgw.buckets.data","create_time":"2023-10-05T19:28:02.811297+0000","flags":1,"flags_names":"hashpspool","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"client.0","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":12288,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rgw":{}}},{"pool":4,"pool_name":"default.rgw.buckets.index","create_time":"2023-10-05T19:28:05.160536+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rgw":{}}},{"pool":5,"pool_name":".rgw.root","create_time":"2023-10-05T19:28:06.942969+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"24","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rgw":{}}},{"pool":6,"pool_name":"default.rgw.log","create_time":"2023-10-05T19:28:08.916250+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rgw":{}}},{"pool":7,"pool_name":"default.rgw.control","create_time":"2023-10-05T19:28:10.863179+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"28","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rgw":{}}},{"pool":8,"pool_name":"default.rgw.meta","create_time":"2023-10-05T19:28:13.021822+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"30","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4},"application_metadata":{"rgw":{}}}],"osds":[{"osd":0,"uuid":"61ba9225-834e-4288-b391-7ffa540f7334","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":28,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6800","nonce":27690},{"type":"v1","addr":"172.21.15.118:6801","nonce":27690}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6802","nonce":27690},{"type":"v1","addr":"172.21.15.118:6803","nonce":27690}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6806","nonce":27690},{"type":"v1","addr":"172.21.15.118:6807","nonce":27690}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6804","nonce":27690},{"type":"v1","addr":"172.21.15.118:6805","nonce":27690}]},"public_addr":"172.21.15.118:6801/27690","cluster_addr":"172.21.15.118:6803/27690","heartbeat_back_addr":"172.21.15.118:6807/27690","heartbeat_front_addr":"172.21.15.118:6805/27690","state":["exists","up"]},{"osd":1,"uuid":"15a3631e-9f20-4b33-9ee3-39eb73e13709","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":31,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6800","nonce":30035},{"type":"v1","addr":"172.21.15.167:6801","nonce":30035}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6802","nonce":30035},{"type":"v1","addr":"172.21.15.167:6803","nonce":30035}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6806","nonce":30035},{"type":"v1","addr":"172.21.15.167:6807","nonce":30035}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6804","nonce":30035},{"type":"v1","addr":"172.21.15.167:6805","nonce":30035}]},"public_addr":"172.21.15.167:6801/30035","cluster_addr":"172.21.15.167:6803/30035","heartbeat_back_addr":"172.21.15.167:6807/30035","heartbeat_front_addr":"172.21.15.167:6805/30035","state":["exists","up"]},{"osd":2,"uuid":"5712158a-931c-4ce7-adf0-5c9fbb59d521","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":31,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6816","nonce":30037},{"type":"v1","addr":"172.21.15.167:6817","nonce":30037}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6818","nonce":30037},{"type":"v1","addr":"172.21.15.167:6819","nonce":30037}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6822","nonce":30037},{"type":"v1","addr":"172.21.15.167:6823","nonce":30037}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6820","nonce":30037},{"type":"v1","addr":"172.21.15.167:6821","nonce":30037}]},"public_addr":"172.21.15.167:6817/30037","cluster_addr":"172.21.15.167:6819/30037","heartbeat_back_addr":"172.21.15.167:6823/30037","heartbeat_front_addr":"172.21.15.167:6821/30037","state":["exists","up"]},{"osd":3,"uuid":"f838ed4e-05a7-49ef-ad9d-8f7dd7be30f1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":31,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6808","nonce":30036},{"type":"v1","addr":"172.21.15.167:6809","nonce":30036}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6810","nonce":30036},{"type":"v1","addr":"172.21.15.167:6811","nonce":30036}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6814","nonce":30036},{"type":"v1","addr":"172.21.15.167:6815","nonce":30036}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.167:6812","nonce":30036},{"type":"v1","addr":"172.21.15.167:6813","nonce":30036}]},"public_addr":"172.21.15.167:6809/30036","cluster_addr":"172.21.15.167:6811/30036","heartbeat_back_addr":"172.21.15.167:6815/30036","heartbeat_front_addr":"172.21.15.167:6813/30036","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-05T19:27:47.190191+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-05T19:27:47.302630+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-05T19:27:47.374216+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-05T19:27:47.350455+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"4.c","mappings":[{"from":1,"to":2}]},{"pgid":"4.14","mappings":[{"from":3,"to":2}]},{"pgid":"4.19","mappings":[{"from":3,"to":2}]},{"pgid":"4.3f","mappings":[{"from":3,"to":0}]},{"pgid":"5.2","mappings":[{"from":0,"to":2}]},{"pgid":"5.1a","mappings":[{"from":0,"to":2}]},{"pgid":"5.1d","mappings":[{"from":0,"to":2}]}],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"client.0":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"3","m":"1","name":"testprofile","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-05T19:31:31.858 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-05T19:31:31.854+0000 7fb81df21700 1 -- 172.21.15.118:0/1050988487 >> [v2:172.21.15.167:6824/29889,v1:172.21.15.167:6825/29889] conn(0x7fb7f80323e0 msgr2=0x7fb7f8034890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-05T19:31:31.858 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-05T19:31:31.854+0000 7fb81df21700 1 --2- 172.21.15.118:0/1050988487 >> [v2:172.21.15.167:6824/29889,v1:172.21.15.167:6825/29889] conn(0x7fb7f80323e0 0x7fb7f8034890 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7fb8000011e0 tx=0x7fb8000079b0 comp rx=0 tx=0).stop 2023-10-05T19:31:31.859 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-05T19:31:31.854+0000 7fb81df21700 1 -- 172.21.15.118:0/1050988487 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb81806cc50 msgr2=0x7fb81813cbb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-05T19:31:31.859 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-05T19:31:31.854+0000 7fb81df21700 1 --2- 172.21.15.118:0/1050988487 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb81806cc50 0x7fb81813cbb0 secure :-1 s=READY pgs=712 cs=0 l=1 rev1=1 crypto rx=0x7fb808009fd0 tx=0x7fb808016000 comp rx=0 tx=0).stop 2023-10-05T19:31:31.859 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-05T19:31:31.854+0000 7fb81df21700 1 -- 172.21.15.118:0/1050988487 shutdown_connections 2023-10-05T19:31:31.859 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-05T19:31:31.854+0000 7fb81df21700 1 --2- 172.21.15.118:0/1050988487 >> [v2:172.21.15.167:6824/29889,v1:172.21.15.167:6825/29889] conn(0x7fb7f80323e0 0x7fb7f8034890 unknown :-1 s=CLOSED pgs=244 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-05T19:31:31.859 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-05T19:31:31.854+0000 7fb81df21700 1 --2- 172.21.15.118:0/1050988487 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb81806cc50 0x7fb81813cbb0 unknown :-1 s=CLOSED pgs=712 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-05T19:31:31.859 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-05T19:31:31.854+0000 7fb81df21700 1 -- 172.21.15.118:0/1050988487 >> 172.21.15.118:0/1050988487 conn(0x7fb81809a530 msgr2=0x7fb818110b90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-05T19:31:31.859 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-05T19:31:31.854+0000 7fb81df21700 1 -- 172.21.15.118:0/1050988487 shutdown_connections 2023-10-05T19:31:31.859 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-05T19:31:31.854+0000 7fb81df21700 1 -- 172.21.15.118:0/1050988487 wait complete. 2023-10-05T19:31:31.873 DEBUG:teuthology.orchestra.run.smithi118:> sudo ceph --cluster ceph config set global mon_health_to_clog false 2023-10-05T19:31:31.982 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-05T19:31:31.978+0000 7f5222e0e700 1 Processor -- start 2023-10-05T19:31:31.982 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-05T19:31:31.978+0000 7f5222e0e700 1 -- start start 2023-10-05T19:31:31.983 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-05T19:31:31.978+0000 7f5222e0e700 1 --2- >> v2:172.21.15.118:3300/0 conn(0x7f521c069dd0 0x7f521c06a1a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-05T19:31:31.983 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-05T19:31:31.978+0000 7f5222e0e700 1 -- --> v1:172.21.15.118:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f521c06c190 con 0x7f521c06a6e0 2023-10-05T19:31:31.983 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-05T19:31:31.978+0000 7f5222e0e700 1 -- --> v2:172.21.15.118:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f521c06b2f0 con 0x7f521c069dd0 2023-10-05T19:31:31.983 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-05T19:31:31.978+0000 7f5221e0c700 1 --2- >> v2:172.21.15.118:3300/0 conn(0x7f521c069dd0 0x7f521c06a1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-05T19:31:31.983 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-05T19:31:31.978+0000 7f522160b700 1 --1- >> v1:172.21.15.118:6789/0 conn(0x7f521c06a6e0 0x7f521c129c40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.118:6789/0 says I am v1:172.21.15.118:34966/0 (socket says 172.21.15.118:34966) 2023-10-05T19:31:31.983 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-05T19:31:31.978+0000 7f522160b700 1 -- 172.21.15.118:0/2658598033 learned_addr learned my addr 172.21.15.118:0/2658598033 (peer_addr_for_me v1:172.21.15.118:0/0) 2023-10-05T19:31:31.984 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-05T19:31:31.978+0000 7f5220e0a700 1 -- 172.21.15.118:0/2658598033 <== mon.0 v1:172.21.15.118:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4044509919 0 0) 0x7f521c06c190 con 0x7f521c06a6e0 2023-10-05T19:31:31.984 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-05T19:31:31.978+0000 7f5221e0c700 1 -- 172.21.15.118:0/2658598033 >> v1:172.21.15.118:6789/0 conn(0x7f521c06a6e0 legacy=0x7f521c129c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-05T19:31:31.984 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-05T19:31:31.978+0000 7f5221e0c700 1 -- 172.21.15.118:0/2658598033 --> v2:172.21.15.118:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f521c0952c0 con 0x7f521c069dd0 2023-10-05T19:31:31.985 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-05T19:31:31.978+0000 7f5221e0c700 1 --2- 172.21.15.118:0/2658598033 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f521c069dd0 0x7f521c06a1a0 secure :-1 s=READY pgs=714 cs=0 l=1 rev1=1 crypto rx=0x7f5218009850 tx=0x7f5218009120 comp rx=0 tx=0).ready entity=mon.0 client_cookie=feb713574fd53534 server_cookie=0 in_seq=0 out_seq=0 2023-10-05T19:31:31.985 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-05T19:31:31.978+0000 7f5220e0a700 1 -- 172.21.15.118:0/2658598033 <== mon.0 v2:172.21.15.118:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f5218005c10 con 0x7f521c069dd0 2023-10-05T19:31:31.985 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-05T19:31:31.982+0000 7f5220e0a700 1 -- 172.21.15.118:0/2658598033 <== mon.0 v2:172.21.15.118:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5218005d70 con 0x7f521c069dd0 2023-10-05T19:31:31.985 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-05T19:31:31.982+0000 7f5220e0a700 1 -- 172.21.15.118:0/2658598033 <== mon.0 v2:172.21.15.118:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f5218006190 con 0x7f521c069dd0 2023-10-05T19:31:31.986 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-05T19:31:31.982+0000 7f5222e0e700 1 -- 172.21.15.118:0/2658598033 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f521c069dd0 msgr2=0x7f521c06a1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-05T19:31:31.986 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-05T19:31:31.982+0000 7f5222e0e700 1 --2- 172.21.15.118:0/2658598033 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f521c069dd0 0x7f521c06a1a0 secure :-1 s=READY pgs=714 cs=0 l=1 rev1=1 crypto rx=0x7f5218009850 tx=0x7f5218009120 comp rx=0 tx=0).stop 2023-10-05T19:31:31.986 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-05T19:31:31.982+0000 7f5222e0e700 1 -- 172.21.15.118:0/2658598033 shutdown_connections 2023-10-05T19:31:31.986 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-05T19:31:31.982+0000 7f5222e0e700 1 --2- 172.21.15.118:0/2658598033 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f521c069dd0 0x7f521c06a1a0 unknown :-1 s=CLOSED pgs=714 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-05T19:31:31.986 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-05T19:31:31.982+0000 7f5222e0e700 1 -- 172.21.15.118:0/2658598033 >> 172.21.15.118:0/2658598033 conn(0x7f521c095760 msgr2=0x7f521c095b60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-05T19:31:31.986 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-05T19:31:31.982+0000 7f5222e0e700 1 -- 172.21.15.118:0/2658598033 shutdown_connections 2023-10-05T19:31:31.986 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-05T19:31:31.982+0000 7f5222e0e700 1 -- 172.21.15.118:0/2658598033 wait complete. 2023-10-05T19:31:31.987 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-05T19:31:31.982+0000 7f5222e0e700 1 Processor -- start 2023-10-05T19:31:31.987 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-05T19:31:31.982+0000 7f5222e0e700 1 -- start start 2023-10-05T19:31:31.988 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-05T19:31:31.982+0000 7f5222e0e700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f521c069dd0 0x7f521c135cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-05T19:31:31.988 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-05T19:31:31.982+0000 7f5222e0e700 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f521c133740 con 0x7f521c069dd0 2023-10-05T19:31:31.988 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-05T19:31:31.982+0000 7f5221e0c700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f521c069dd0 0x7f521c135cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-05T19:31:31.988 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-05T19:31:31.982+0000 7f5221e0c700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f521c069dd0 0x7f521c135cc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.118:3300/0 says I am v2:172.21.15.118:55312/0 (socket says 172.21.15.118:55312) 2023-10-05T19:31:31.989 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-05T19:31:31.982+0000 7f5221e0c700 1 -- 172.21.15.118:0/2148316955 learned_addr learned my addr 172.21.15.118:0/2148316955 (peer_addr_for_me v2:172.21.15.118:0/0) 2023-10-05T19:31:31.990 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-05T19:31:31.986+0000 7f5221e0c700 1 -- 172.21.15.118:0/2148316955 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f52180093c0 con 0x7f521c069dd0 2023-10-05T19:31:31.990 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-05T19:31:31.986+0000 7f5221e0c700 1 --2- 172.21.15.118:0/2148316955 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f521c069dd0 0x7f521c135cc0 secure :-1 s=READY pgs=715 cs=0 l=1 rev1=1 crypto rx=0x7f5218009390 tx=0x7f5218015000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-05T19:31:31.990 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-05T19:31:31.986+0000 7f520dffb700 1 -- 172.21.15.118:0/2148316955 <== mon.0 v2:172.21.15.118:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f521800b9e0 con 0x7f521c069dd0 2023-10-05T19:31:31.990 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-05T19:31:31.986+0000 7f520dffb700 1 -- 172.21.15.118:0/2148316955 <== mon.0 v2:172.21.15.118:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f521800bb40 con 0x7f521c069dd0 2023-10-05T19:31:31.990 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-05T19:31:31.986+0000 7f5222e0e700 1 -- 172.21.15.118:0/2148316955 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f521c092e80 con 0x7f521c069dd0 2023-10-05T19:31:31.990 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-05T19:31:31.986+0000 7f520dffb700 1 -- 172.21.15.118:0/2148316955 <== mon.0 v2:172.21.15.118:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f521800bdf0 con 0x7f521c069dd0 2023-10-05T19:31:31.990 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-05T19:31:31.986+0000 7f5222e0e700 1 -- 172.21.15.118:0/2148316955 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f521c091990 con 0x7f521c069dd0 2023-10-05T19:31:31.991 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-05T19:31:31.986+0000 7f520dffb700 1 -- 172.21.15.118:0/2148316955 <== mon.0 v2:172.21.15.118:3300/0 4 ==== mgrmap(e 5) v1 ==== 40329+0+0 (secure 0 0 0) 0x7f5218023030 con 0x7f521c069dd0 2023-10-05T19:31:31.991 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-05T19:31:31.986+0000 7f520dffb700 1 --2- 172.21.15.118:0/2148316955 >> [v2:172.21.15.167:6824/29889,v1:172.21.15.167:6825/29889] conn(0x7f52000323e0 0x7f5200034890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-05T19:31:31.991 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-05T19:31:31.986+0000 7f520dffb700 1 -- 172.21.15.118:0/2148316955 <== mon.0 v2:172.21.15.118:3300/0 5 ==== osd_map(32..32 src has 1..32) v4 ==== 6260+0+0 (secure 0 0 0) 0x7f5218046620 con 0x7f521c069dd0 2023-10-05T19:31:31.992 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-05T19:31:31.986+0000 7f5222e0e700 1 -- 172.21.15.118:0/2148316955 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f521c05ecb0 con 0x7f521c069dd0 2023-10-05T19:31:31.992 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-05T19:31:31.986+0000 7f522160b700 1 --2- 172.21.15.118:0/2148316955 >> [v2:172.21.15.167:6824/29889,v1:172.21.15.167:6825/29889] conn(0x7f52000323e0 0x7f5200034890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-05T19:31:31.992 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-05T19:31:31.990+0000 7f522160b700 1 --2- 172.21.15.118:0/2148316955 >> [v2:172.21.15.167:6824/29889,v1:172.21.15.167:6825/29889] conn(0x7f52000323e0 0x7f5200034890 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7f52100011e0 tx=0x7f52100079d0 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-05T19:31:31.999 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-05T19:31:31.994+0000 7f520dffb700 1 -- 172.21.15.118:0/2148316955 <== mon.0 v2:172.21.15.118:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+166807 (secure 0 0 0) 0x7f521801d430 con 0x7f521c069dd0 2023-10-05T19:31:32.159 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-05T19:31:32.154+0000 7f5222e0e700 1 -- 172.21.15.118:0/2148316955 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_command([{prefix=config set, name=mon_health_to_clog}] v 0) v1 -- 0x7f521c06be80 con 0x7f521c069dd0 2023-10-05T19:31:32.163 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-05T19:31:32.158+0000 7f520dffb700 1 -- 172.21.15.118:0/2148316955 <== mon.0 v2:172.21.15.118:3300/0 7 ==== config(1 keys) v1 ==== 35+0+0 (secure 0 0 0) 0x7f521801d430 con 0x7f521c069dd0 2023-10-05T19:31:32.163 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-05T19:31:32.158+0000 7f520dffb700 1 -- 172.21.15.118:0/2148316955 <== mon.0 v2:172.21.15.118:3300/0 8 ==== mon_command_ack([{prefix=config set, name=mon_health_to_clog}]=0 v6)=0 v6) v1 ==== 123+0+0 (secure 0 0 0) 0x7f5218010380 con 0x7f521c069dd0 2023-10-05T19:31:32.166 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-05T19:31:32.162+0000 7f5222e0e700 1 -- 172.21.15.118:0/2148316955 >> [v2:172.21.15.167:6824/29889,v1:172.21.15.167:6825/29889] conn(0x7f52000323e0 msgr2=0x7f5200034890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-05T19:31:32.166 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-05T19:31:32.162+0000 7f5222e0e700 1 --2- 172.21.15.118:0/2148316955 >> [v2:172.21.15.167:6824/29889,v1:172.21.15.167:6825/29889] conn(0x7f52000323e0 0x7f5200034890 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7f52100011e0 tx=0x7f52100079d0 comp rx=0 tx=0).stop 2023-10-05T19:31:32.166 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-05T19:31:32.162+0000 7f5222e0e700 1 -- 172.21.15.118:0/2148316955 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f521c069dd0 msgr2=0x7f521c135cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-05T19:31:32.166 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-05T19:31:32.162+0000 7f5222e0e700 1 --2- 172.21.15.118:0/2148316955 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f521c069dd0 0x7f521c135cc0 secure :-1 s=READY pgs=715 cs=0 l=1 rev1=1 crypto rx=0x7f5218009390 tx=0x7f5218015000 comp rx=0 tx=0).stop 2023-10-05T19:31:32.166 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-05T19:31:32.162+0000 7f5222e0e700 1 -- 172.21.15.118:0/2148316955 shutdown_connections 2023-10-05T19:31:32.166 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-05T19:31:32.162+0000 7f5222e0e700 1 --2- 172.21.15.118:0/2148316955 >> [v2:172.21.15.167:6824/29889,v1:172.21.15.167:6825/29889] conn(0x7f52000323e0 0x7f5200034890 unknown :-1 s=CLOSED pgs=245 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-05T19:31:32.166 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-05T19:31:32.162+0000 7f5222e0e700 1 --2- 172.21.15.118:0/2148316955 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f521c069dd0 0x7f521c135cc0 unknown :-1 s=CLOSED pgs=715 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-05T19:31:32.166 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-05T19:31:32.162+0000 7f5222e0e700 1 -- 172.21.15.118:0/2148316955 >> 172.21.15.118:0/2148316955 conn(0x7f521c095760 msgr2=0x7f521c06cf90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-05T19:31:32.167 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-05T19:31:32.162+0000 7f5222e0e700 1 -- 172.21.15.118:0/2148316955 shutdown_connections 2023-10-05T19:31:32.167 INFO:teuthology.orchestra.run.smithi118.stderr:2023-10-05T19:31:32.162+0000 7f5222e0e700 1 -- 172.21.15.118:0/2148316955 wait complete. 2023-10-05T19:31:32.176 INFO:teuthology.misc:Shutting down mds daemons... 2023-10-05T19:31:32.177 INFO:teuthology.misc:Shutting down osd daemons... 2023-10-05T19:31:32.177 DEBUG:tasks.ceph.osd.0:waiting for process to exit 2023-10-05T19:31:32.178 INFO:teuthology.orchestra.run:waiting for 300 2023-10-05T19:31:32.270 INFO:tasks.ceph.osd.0:Stopped 2023-10-05T19:31:32.271 DEBUG:tasks.ceph.osd.1:waiting for process to exit 2023-10-05T19:31:32.271 INFO:teuthology.orchestra.run:waiting for 300 2023-10-05T19:31:32.350 INFO:tasks.ceph.osd.1:Stopped 2023-10-05T19:31:32.351 DEBUG:tasks.ceph.osd.2:waiting for process to exit 2023-10-05T19:31:32.351 INFO:teuthology.orchestra.run:waiting for 300 2023-10-05T19:31:32.424 INFO:tasks.ceph.osd.2:Stopped 2023-10-05T19:31:32.424 DEBUG:tasks.ceph.osd.3:waiting for process to exit 2023-10-05T19:31:32.424 INFO:teuthology.orchestra.run:waiting for 300 2023-10-05T19:31:32.510 INFO:tasks.ceph.osd.3:Stopped 2023-10-05T19:31:32.510 INFO:teuthology.misc:Shutting down mgr daemons... 2023-10-05T19:31:32.510 DEBUG:tasks.ceph.mgr.x:waiting for process to exit 2023-10-05T19:31:32.510 INFO:teuthology.orchestra.run:waiting for 300 2023-10-05T19:31:32.538 INFO:tasks.ceph.mgr.x:Stopped 2023-10-05T19:31:32.538 INFO:teuthology.misc:Shutting down mon daemons... 2023-10-05T19:31:32.538 DEBUG:tasks.ceph.mon.a:waiting for process to exit 2023-10-05T19:31:32.538 INFO:teuthology.orchestra.run:waiting for 300 2023-10-05T19:31:32.559 INFO:tasks.ceph.mon.a:Stopped 2023-10-05T19:31:32.559 INFO:tasks.ceph:Checking cluster log for badness... 2023-10-05T19:31:32.560 DEBUG:teuthology.orchestra.run.smithi118:> sudo egrep '\[ERR\]|\[WRN\]|\[SEC\]' /var/log/ceph/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v '\(PG_AVAILABILITY\)' | egrep -v '\(PG_DEGRADED\)' | head -n 1 2023-10-05T19:31:32.577 INFO:teuthology.orchestra.run.smithi118.stdout:1696534077.7401268 mon.a (mon.0) 113 : cluster [WRN] Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2023-10-05T19:31:32.578 WARNING:tasks.ceph:Found errors (ERR|WRN|SEC) in cluster log 2023-10-05T19:31:32.578 DEBUG:teuthology.orchestra.run.smithi118:> sudo egrep '\[SEC\]' /var/log/ceph/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v '\(PG_AVAILABILITY\)' | egrep -v '\(PG_DEGRADED\)' | head -n 1 2023-10-05T19:31:32.594 DEBUG:teuthology.orchestra.run.smithi118:> sudo egrep '\[ERR\]' /var/log/ceph/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v '\(PG_AVAILABILITY\)' | egrep -v '\(PG_DEGRADED\)' | head -n 1 2023-10-05T19:31:32.654 DEBUG:teuthology.orchestra.run.smithi118:> sudo egrep '\[WRN\]' /var/log/ceph/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v '\(PG_AVAILABILITY\)' | egrep -v '\(PG_DEGRADED\)' | head -n 1 2023-10-05T19:31:32.670 INFO:teuthology.orchestra.run.smithi118.stdout:1696534077.7401268 mon.a (mon.0) 113 : cluster [WRN] Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2023-10-05T19:31:32.671 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-0 on ubuntu@smithi118.front.sepia.ceph.com 2023-10-05T19:31:32.671 DEBUG:teuthology.orchestra.run.smithi118:> sync && sudo umount -f /var/lib/ceph/osd/ceph-0 2023-10-05T19:31:32.934 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-1 on ubuntu@smithi167.front.sepia.ceph.com 2023-10-05T19:31:32.934 DEBUG:teuthology.orchestra.run.smithi167:> sync && sudo umount -f /var/lib/ceph/osd/ceph-1 2023-10-05T19:31:33.263 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-2 on ubuntu@smithi167.front.sepia.ceph.com 2023-10-05T19:31:33.263 DEBUG:teuthology.orchestra.run.smithi167:> sync && sudo umount -f /var/lib/ceph/osd/ceph-2 2023-10-05T19:31:33.541 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-3 on ubuntu@smithi167.front.sepia.ceph.com 2023-10-05T19:31:33.542 DEBUG:teuthology.orchestra.run.smithi167:> sync && sudo umount -f /var/lib/ceph/osd/ceph-3 2023-10-05T19:31:33.733 INFO:tasks.ceph:Archiving mon data... 2023-10-05T19:31:33.734 DEBUG:teuthology.misc:Transferring archived files from smithi118:/var/lib/ceph/mon/ceph-a to /home/teuthworker/archive/yuriw-2023-10-05_14:30:58-rgw-wip-yuri2-testing-2023-10-04-1333-quincy-distro-default-smithi/7411744/data/mon.a.tgz 2023-10-05T19:31:33.734 DEBUG:teuthology.orchestra.run.smithi118:> mktemp 2023-10-05T19:31:33.741 INFO:teuthology.orchestra.run.smithi118.stdout:/tmp/tmp.3OTV5KbTbp 2023-10-05T19:31:33.741 DEBUG:teuthology.orchestra.run.smithi118:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-a -- . > /tmp/tmp.3OTV5KbTbp 2023-10-05T19:31:33.852 DEBUG:teuthology.orchestra.run.smithi118:> sudo chmod 0666 /tmp/tmp.3OTV5KbTbp 2023-10-05T19:31:33.872 DEBUG:teuthology.orchestra.remote:smithi118:/tmp/tmp.3OTV5KbTbp is 213KB 2023-10-05T19:31:33.934 DEBUG:teuthology.orchestra.run.smithi118:> rm -fr /tmp/tmp.3OTV5KbTbp 2023-10-05T19:31:33.941 INFO:tasks.ceph:Cleaning ceph cluster... 2023-10-05T19:31:33.941 DEBUG:teuthology.orchestra.run.smithi118:> sudo rm -rf -- /etc/ceph/ceph.conf /etc/ceph/ceph.keyring /home/ubuntu/cephtest/ceph.data /home/ubuntu/cephtest/ceph.monmap /home/ubuntu/cephtest/../*.pid 2023-10-05T19:31:33.943 DEBUG:teuthology.orchestra.run.smithi167:> sudo rm -rf -- /etc/ceph/ceph.conf /etc/ceph/ceph.keyring /home/ubuntu/cephtest/ceph.data /home/ubuntu/cephtest/ceph.monmap /home/ubuntu/cephtest/../*.pid 2023-10-05T19:31:33.959 INFO:tasks.ceph:Checking for errors in any valgrind logs... 2023-10-05T19:31:33.959 DEBUG:teuthology.orchestra.run.smithi118:> sudo zgrep '' /var/log/ceph/valgrind/* /dev/null | sort | uniq 2023-10-05T19:31:33.961 DEBUG:teuthology.orchestra.run.smithi167:> sudo zgrep '' /var/log/ceph/valgrind/* /dev/null | sort | uniq 2023-10-05T19:31:33.983 INFO:teuthology.orchestra.run.smithi118.stderr:gzip: /var/log/ceph/valgrind/*.gz: No such file or directory 2023-10-05T19:31:33.986 INFO:teuthology.orchestra.run.smithi167.stderr:gzip: /var/log/ceph/valgrind/*.gz: No such file or directory 2023-10-05T19:31:33.992 INFO:tasks.ceph:Archiving crash dumps... 2023-10-05T19:31:33.994 DEBUG:teuthology.misc:Transferring archived files from smithi118:/var/lib/ceph/crash to /home/teuthworker/archive/yuriw-2023-10-05_14:30:58-rgw-wip-yuri2-testing-2023-10-04-1333-quincy-distro-default-smithi/7411744/remote/smithi118/crash 2023-10-05T19:31:33.995 DEBUG:teuthology.orchestra.run.smithi118:> sudo tar c -f - -C /var/lib/ceph/crash -- . 2023-10-05T19:31:34.047 DEBUG:teuthology.misc:Transferring archived files from smithi167:/var/lib/ceph/crash to /home/teuthworker/archive/yuriw-2023-10-05_14:30:58-rgw-wip-yuri2-testing-2023-10-04-1333-quincy-distro-default-smithi/7411744/remote/smithi167/crash 2023-10-05T19:31:34.048 DEBUG:teuthology.orchestra.run.smithi167:> sudo tar c -f - -C /var/lib/ceph/crash -- . 2023-10-05T19:31:34.065 INFO:tasks.ceph:Compressing logs... 2023-10-05T19:31:34.065 DEBUG:teuthology.orchestra.run.smithi118:> sudo find /var/log/ceph -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2023-10-05T19:31:34.068 DEBUG:teuthology.orchestra.run.smithi167:> sudo find /var/log/ceph -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2023-10-05T19:31:34.280 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.rgw.client.0 has been restored 2023-10-05T19:31:36.793 INFO:tasks.ceph:Archiving logs... 2023-10-05T19:31:36.793 DEBUG:teuthology.misc:Transferring archived files from smithi118:/var/log/ceph to /home/teuthworker/archive/yuriw-2023-10-05_14:30:58-rgw-wip-yuri2-testing-2023-10-04-1333-quincy-distro-default-smithi/7411744/remote/smithi118/log 2023-10-05T19:31:36.795 DEBUG:teuthology.orchestra.run.smithi118:> sudo tar c -f - -C /var/log/ceph -- . 2023-10-05T19:31:36.965 DEBUG:teuthology.misc:Transferring archived files from smithi167:/var/log/ceph to /home/teuthworker/archive/yuriw-2023-10-05_14:30:58-rgw-wip-yuri2-testing-2023-10-04-1333-quincy-distro-default-smithi/7411744/remote/smithi167/log 2023-10-05T19:31:36.966 DEBUG:teuthology.orchestra.run.smithi167:> sudo tar c -f - -C /var/log/ceph -- . 2023-10-05T19:31:37.397 DEBUG:teuthology.run_tasks:Unwinding manager install 2023-10-05T19:31:37.474 INFO:teuthology.task.install.util:Removing shipped files: /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer... 2023-10-05T19:31:37.474 DEBUG:teuthology.orchestra.run.smithi118:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2023-10-05T19:31:37.476 DEBUG:teuthology.orchestra.run.smithi167:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2023-10-05T19:31:37.544 DEBUG:teuthology.run_tasks:Unwinding manager clock 2023-10-05T19:31:37.556 INFO:teuthology.task.clock:Checking final clock skew... 2023-10-05T19:31:37.556 DEBUG:teuthology.orchestra.run.smithi118:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2023-10-05T19:31:37.559 DEBUG:teuthology.orchestra.run.smithi167:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2023-10-05T19:31:37.578 INFO:teuthology.orchestra.run.smithi118.stdout: remote refid st t when poll reach delay offset jitter 2023-10-05T19:31:37.578 INFO:teuthology.orchestra.run.smithi118.stdout:============================================================================== 2023-10-05T19:31:37.578 INFO:teuthology.orchestra.run.smithi118.stdout: hv01.front.sepi 44.4.53.2 2 u 37 64 77 0.076 2.214 0.140 2023-10-05T19:31:37.578 INFO:teuthology.orchestra.run.smithi118.stdout: hv02.front.sepi 63.231.80.2 3 u 48 64 77 0.067 1.421 0.140 2023-10-05T19:31:37.578 INFO:teuthology.orchestra.run.smithi118.stdout:*hv03.front.sepi 74.6.168.72 3 u 48 64 77 0.098 0.172 0.092 2023-10-05T19:31:37.578 INFO:teuthology.orchestra.run.smithi118.stdout: hv04.front.sepi .INIT. 16 u - 64 0 0.000 0.000 0.000 2023-10-05T19:31:37.582 INFO:teuthology.orchestra.run.smithi167.stdout: remote refid st t when poll reach delay offset jitter 2023-10-05T19:31:37.583 INFO:teuthology.orchestra.run.smithi167.stdout:============================================================================== 2023-10-05T19:31:37.583 INFO:teuthology.orchestra.run.smithi167.stdout:*hv01.front.sepi 44.4.53.2 2 u 37 64 77 0.088 2.700 1.204 2023-10-05T19:31:37.583 INFO:teuthology.orchestra.run.smithi167.stdout:+hv02.front.sepi 63.231.80.2 3 u 41 64 77 0.071 -0.255 1.862 2023-10-05T19:31:37.583 INFO:teuthology.orchestra.run.smithi167.stdout:+hv03.front.sepi 74.6.168.72 3 u 40 64 77 0.083 1.003 1.491 2023-10-05T19:31:37.583 INFO:teuthology.orchestra.run.smithi167.stdout: hv04.front.sepi .INIT. 16 u - 64 0 0.000 0.000 0.000 2023-10-05T19:31:37.584 DEBUG:teuthology.run_tasks:Unwinding manager ansible.cephlab 2023-10-05T19:31:37.596 INFO:teuthology.task.ansible:Skipping ansible cleanup... 2023-10-05T19:31:37.597 DEBUG:teuthology.run_tasks:Unwinding manager selinux 2023-10-05T19:31:37.624 DEBUG:teuthology.run_tasks:Unwinding manager pcp 2023-10-05T19:31:37.661 DEBUG:teuthology.run_tasks:Unwinding manager internal.timer 2023-10-05T19:31:37.695 INFO:teuthology.task.internal:Duration was 718.452000 seconds 2023-10-05T19:31:37.697 DEBUG:teuthology.run_tasks:Unwinding manager internal.syslog 2023-10-05T19:31:37.727 INFO:teuthology.task.internal.syslog:Shutting down syslog monitoring... 2023-10-05T19:31:37.727 DEBUG:teuthology.orchestra.run.smithi118:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2023-10-05T19:31:37.730 DEBUG:teuthology.orchestra.run.smithi167:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2023-10-05T19:31:37.780 INFO:teuthology.task.internal.syslog:Checking logs for errors... 2023-10-05T19:31:37.780 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi118.front.sepia.ceph.com 2023-10-05T19:31:37.780 DEBUG:teuthology.orchestra.run.smithi118:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2023-10-05T19:31:37.792 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi167.front.sepia.ceph.com 2023-10-05T19:31:37.792 DEBUG:teuthology.orchestra.run.smithi167:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2023-10-05T19:31:37.806 INFO:teuthology.task.internal.syslog:Compressing syslogs... 2023-10-05T19:31:37.806 DEBUG:teuthology.orchestra.run.smithi118:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2023-10-05T19:31:37.808 DEBUG:teuthology.orchestra.run.smithi167:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2023-10-05T19:31:37.840 INFO:teuthology.task.internal.syslog:Gathering journactl -b0... 2023-10-05T19:31:37.841 DEBUG:teuthology.orchestra.run.smithi118:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2023-10-05T19:31:37.842 DEBUG:teuthology.orchestra.run.smithi167:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2023-10-05T19:31:37.947 DEBUG:teuthology.run_tasks:Unwinding manager internal.sudo 2023-10-05T19:31:37.959 INFO:teuthology.task.internal:Restoring /etc/sudoers... 2023-10-05T19:31:37.959 DEBUG:teuthology.orchestra.run.smithi118:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2023-10-05T19:31:37.975 DEBUG:teuthology.orchestra.run.smithi167:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2023-10-05T19:31:37.992 DEBUG:teuthology.run_tasks:Unwinding manager internal.coredump 2023-10-05T19:31:38.001 DEBUG:teuthology.orchestra.run.smithi118:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2023-10-05T19:31:38.004 DEBUG:teuthology.orchestra.run.smithi167:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2023-10-05T19:31:38.042 INFO:teuthology.orchestra.run.smithi118.stdout:kernel.core_pattern = core 2023-10-05T19:31:38.054 INFO:teuthology.orchestra.run.smithi167.stdout:kernel.core_pattern = core 2023-10-05T19:31:38.089 DEBUG:teuthology.orchestra.run.smithi118:> test -e /home/ubuntu/cephtest/archive/coredump 2023-10-05T19:31:38.094 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-10-05T19:31:38.095 DEBUG:teuthology.orchestra.run.smithi167:> test -e /home/ubuntu/cephtest/archive/coredump 2023-10-05T19:31:38.136 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-10-05T19:31:38.137 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive 2023-10-05T19:31:38.148 INFO:teuthology.task.internal:Transferring archived files... 2023-10-05T19:31:38.149 DEBUG:teuthology.misc:Transferring archived files from smithi118:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/yuriw-2023-10-05_14:30:58-rgw-wip-yuri2-testing-2023-10-04-1333-quincy-distro-default-smithi/7411744/remote/smithi118 2023-10-05T19:31:38.149 DEBUG:teuthology.orchestra.run.smithi118:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2023-10-05T19:31:38.170 DEBUG:teuthology.misc:Transferring archived files from smithi167:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/yuriw-2023-10-05_14:30:58-rgw-wip-yuri2-testing-2023-10-04-1333-quincy-distro-default-smithi/7411744/remote/smithi167 2023-10-05T19:31:38.170 DEBUG:teuthology.orchestra.run.smithi167:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2023-10-05T19:31:38.193 INFO:teuthology.task.internal:Removing archive directory... 2023-10-05T19:31:38.193 DEBUG:teuthology.orchestra.run.smithi118:> rm -rf -- /home/ubuntu/cephtest/archive 2023-10-05T19:31:38.195 DEBUG:teuthology.orchestra.run.smithi167:> rm -rf -- /home/ubuntu/cephtest/archive 2023-10-05T19:31:38.202 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive_upload 2023-10-05T19:31:38.218 INFO:teuthology.task.internal:Not uploading archives. 2023-10-05T19:31:38.218 DEBUG:teuthology.run_tasks:Unwinding manager internal.base 2023-10-05T19:31:38.235 INFO:teuthology.task.internal:Tidying up after the test... 2023-10-05T19:31:38.235 DEBUG:teuthology.orchestra.run.smithi118:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2023-10-05T19:31:38.237 DEBUG:teuthology.orchestra.run.smithi167:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2023-10-05T19:31:38.243 INFO:teuthology.orchestra.run.smithi118.stdout: 924112 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Oct 5 19:31 /home/ubuntu/cephtest 2023-10-05T19:31:38.250 INFO:teuthology.orchestra.run.smithi167.stdout: 924112 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Oct 5 19:31 /home/ubuntu/cephtest 2023-10-05T19:31:38.253 DEBUG:teuthology.run_tasks:Unwinding manager kernel 2023-10-05T19:31:38.267 DEBUG:teuthology.run_tasks:Unwinding manager console_log 2023-10-05T19:31:38.335 INFO:teuthology.nuke:Checking targets against current locks 2023-10-05T19:31:38.362 DEBUG:teuthology.nuke:shortname: smithi118 2023-10-05T19:31:38.363 INFO:teuthology.task.internal.check_lock:Checking locks... 2023-10-05T19:31:38.384 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi118.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/yuriw-2023-10-05_14:30:58-rgw-wip-yuri2-testing-2023-10-04-1333-quincy-distro-default-smithi/7411744', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'ubuntu', 'os_version': '20.04', 'arch': 'x86_64', 'locked': True, 'locked_since': '2023-10-05 19:09:12.712230', 'locked_by': 'scheduled_yuriw@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBFyT/r7xuKfmg50w0OdQebr4A/ptJo1Zk6DyZNZNZM/q5rW+sznh+wvHqeT3e4YtCdZLaXyvMms8o8u8H5nXOYM='} 2023-10-05T19:31:38.387 DEBUG:teuthology.nuke:shortname: smithi167 2023-10-05T19:31:38.388 INFO:teuthology.task.internal.check_lock:Checking locks... 2023-10-05T19:31:38.410 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi167.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/yuriw-2023-10-05_14:30:58-rgw-wip-yuri2-testing-2023-10-04-1333-quincy-distro-default-smithi/7411744', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'ubuntu', 'os_version': '20.04', 'arch': 'x86_64', 'locked': True, 'locked_since': '2023-10-05 19:09:12.710580', 'locked_by': 'scheduled_yuriw@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBFyT/r7xuKfmg50w0OdQebr4A/ptJo1Zk6DyZNZNZM/q5rW+sznh+wvHqeT3e4YtCdZLaXyvMms8o8u8H5nXOYM='} 2023-10-05T19:31:38.440 INFO:teuthology.orchestra.console.smithi118:Power off 2023-10-05T19:31:38.441 DEBUG:teuthology.orchestra.console.smithi118:pexpect command: ipmitool -H smithi118.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2023-10-05T19:31:38.466 DEBUG:teuthology.orchestra.console.smithi118:power off output: Chassis Power Control: Down/Off 2023-10-05T19:31:38.466 DEBUG:teuthology.orchestra.console.smithi118:pexpect command: ipmitool -H smithi118.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-10-05T19:31:38.482 INFO:teuthology.orchestra.console.smithi167:Power off 2023-10-05T19:31:38.482 DEBUG:teuthology.orchestra.console.smithi167:pexpect command: ipmitool -H smithi167.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2023-10-05T19:31:38.492 DEBUG:teuthology.orchestra.console.smithi118:check power output: Chassis Power is on 2023-10-05T19:31:38.506 DEBUG:teuthology.orchestra.console.smithi167:power off output: Chassis Power Control: Down/Off 2023-10-05T19:31:38.506 DEBUG:teuthology.orchestra.console.smithi167:pexpect command: ipmitool -H smithi167.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-10-05T19:31:38.531 DEBUG:teuthology.orchestra.console.smithi167:check power output: Chassis Power is on 2023-10-05T19:31:42.493 DEBUG:teuthology.orchestra.console.smithi118:pexpect command: ipmitool -H smithi118.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-10-05T19:31:42.532 DEBUG:teuthology.orchestra.console.smithi167:pexpect command: ipmitool -H smithi167.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-10-05T19:31:42.604 DEBUG:teuthology.orchestra.console.smithi118:check power output: Chassis Power is on 2023-10-05T19:31:42.642 DEBUG:teuthology.orchestra.console.smithi167:check power output: Chassis Power is on 2023-10-05T19:31:46.606 DEBUG:teuthology.orchestra.console.smithi118:pexpect command: ipmitool -H smithi118.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-10-05T19:31:46.643 DEBUG:teuthology.orchestra.console.smithi167:pexpect command: ipmitool -H smithi167.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-10-05T19:31:46.720 DEBUG:teuthology.orchestra.console.smithi118:check power output: Chassis Power is on 2023-10-05T19:31:46.754 DEBUG:teuthology.orchestra.console.smithi167:check power output: Chassis Power is on 2023-10-05T19:31:50.722 DEBUG:teuthology.orchestra.console.smithi118:pexpect command: ipmitool -H smithi118.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-10-05T19:31:50.755 DEBUG:teuthology.orchestra.console.smithi167:pexpect command: ipmitool -H smithi167.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-10-05T19:31:50.834 DEBUG:teuthology.orchestra.console.smithi118:check power output: Chassis Power is off 2023-10-05T19:31:50.865 DEBUG:teuthology.orchestra.console.smithi167:check power output: Chassis Power is off 2023-10-05T19:31:50.935 INFO:teuthology.orchestra.console.smithi118:Power off completed 2023-10-05T19:31:50.966 INFO:teuthology.orchestra.console.smithi167:Power off completed 2023-10-05T19:31:51.070 INFO:teuthology.run:Summary data: description: rgw/singleton/{all/radosgw-admin frontend/beast ignore-pg-availability objectstore/bluestore-bitmap overrides rgw_pool_type/ec-profile supported-random-distro$/{ubuntu_latest}} duration: 718.4519999027252 failure_reason: '"1696534077.7401268 mon.a (mon.0) 113 : cluster [WRN] Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED)" in cluster log' flavor: default owner: scheduled_yuriw@teuthology success: false 2023-10-05T19:31:51.070 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2023-10-05T19:31:51.140 INFO:teuthology.run:FAIL